2022-05-13T22:46:08.815 INFO:root:teuthology version: 1.1.1.dev333+gb018fa01 2022-05-13T22:46:08.817 DEBUG:teuthology.run:Teuthology command: teuthology --archive /home/teuthworker/archive/yuriw-2022-05-12_22:06:41-fs-wip-yuri-testing-2022-05-10-1027-quincy-distro-default-smithi/6830937 --name yuriw-2022-05-12_22:06:41-fs-wip-yuri-testing-2022-05-10-1027-quincy-distro-default-smithi --description fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/{crc} objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/no standby-replay tasks/{0-check-counter workunit/suites/fsstress} wsync/{no}} --verbose --owner scheduled_yuriw@teuthology -- /home/teuthworker/archive/yuriw-2022-05-12_22:06:41-fs-wip-yuri-testing-2022-05-10-1027-quincy-distro-default-smithi/6830937/orig.config.yaml 2022-05-13T22:46:08.874 DEBUG:teuthology.report:Pushing job info to http://paddles.front.sepia.ceph.com/ 2022-05-13T22:46:08.950 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/yuriw-2022-05-12_22:06:41-fs-wip-yuri-testing-2022-05-10-1027-quincy-distro-default-smithi/6830937 branch: wip-yuri-testing-2022-05-10-1027-quincy description: fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/{crc} objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/no standby-replay tasks/{0-check-counter workunit/suites/fsstress} wsync/{no}} email: ceph-qa@ceph.io first_in_suite: false job_id: '6830937' kernel: client: branch: testing kdb: true sha1: distro last_in_suite: false log-rotate: ceph-mds: 10G ceph-osd: 10G machine_type: smithi name: yuriw-2022-05-12_22:06:41-fs-wip-yuri-testing-2022-05-10-1027-quincy-distro-default-smithi nuke-on-error: true openstack: - volumes: count: 4 size: 20 - machine: disk: 200 os_type: rhel os_version: '8.4' overrides: admin_socket: branch: wip-yuri-testing-2022-05-10-1027-quincy ceph: cephfs: max_mds: 5 session_timeout: 300 standby_replay: true conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 rados mon op timeout: 900 rados osd op timeout: 900 global: ms die on skipped message: false mds: debug mds: 20 debug ms: 1 mds bal fragment size max: 10000 mds bal merge size: 5 mds bal split bits: 3 mds bal split size: 100 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon op complaint time: 120 osd: bdev async discard: true bdev enable discard: true bluestore allocator: bitmap bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug ms: 1 debug osd: 20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op complaint time: 180 osd shutdown pgref assert: true osd_max_omap_entries_per_request: 10 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - overall HEALTH_ - \(FS_DEGRADED\) - \(MDS_FAILED\) - \(MDS_DEGRADED\) - \(FS_WITH_FAILED_MDS\) - \(MDS_DAMAGE\) - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(FS_INLINE_DATA_DEPRECATED\) - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding log-whitelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) sha1: 23b0658391ce94ca40799469eec00c28b298816c ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: osd default pool size: 2 osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs check-counter: counters: mds: - mds.exported - mds.imported install: ceph: flavor: default sha1: 23b0658391ce94ca40799469eec00c28b298816c kclient: mntopts: - ms_mode=crc - nowsync syntax: v2 selinux: whitelist: - scontext=system_u:system_r:logrotate_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-yuri-testing-2022-05-10-1027-quincy sha1: 23b0658391ce94ca40799469eec00c28b298816c owner: scheduled_yuriw@teuthology priority: 79 repo: https://github.com/ceph/ceph-ci.git roles: - - mon.a - mgr.x - mds.a - mds.d - mds.g - mds.j - osd.0 - osd.3 - osd.6 - osd.9 - client.0 - - mon.b - mgr.y - mds.b - mds.e - mds.h - mds.k - osd.1 - osd.4 - osd.7 - osd.10 - - mon.c - mgr.z - mds.c - mds.f - mds.i - mds.l - osd.2 - osd.5 - osd.8 - osd.11 sha1: 23b0658391ce94ca40799469eec00c28b298816c sleep_before_teardown: 0 suite: fs suite_branch: wip-yuri-testing-2022-05-10-1027-quincy suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_23b0658391ce94ca40799469eec00c28b298816c/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: 23b0658391ce94ca40799469eec00c28b298816c targets: smithi002.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDJCRJH+22ib3W4xM4ZWpplmVD8g5uJM45T2MJNaSCVH7+/beTyG2ULZaLyFLm6PVaqSAkceWqx5LY2sFx4cg/LnOoeMZb3lRKMk97Z/wPJIgRd643O3csq2+y5f3onVXCv6I+DO99s5PdZhRNdX6L4tXN6cSMMizDqRcGOhyceepEi/iWaLtXXu3FUi0k0xemi9IEKbSopFyW5UwSZgaOUA4dPoZI+b+RpvdjY1ubmfSsHA8fD8h2LQvk7hv9XVaV/JAiKqrVwYSmbh2R/+gbSWO5Rp4LBv2/Rs26BxGG8/eHOU0rLM98wPXVhCsuXDx5KUumc+XCTPB0dxavGbh2EE8IUtJJeSMsFvKMcAqQgy1kask4UelvMDZAOoZJTsWIl4gkQ9aY6svjT9x8+GdSIkxdY9+ryeczXL3QBSJXsCyJEM+ruwK21HD/OyNXQN51lAaDtwjjPiM8Ybd1BF/qtcYWmde+pu2LmYfk15WZPZFoQA+fAfpdxNd7CZzCGLbc= smithi047.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDC7GVeUzB9EY9JZUnFEjA5BKACAdLEvfPpJ3jNrvSvjwo/ZbOuqomQaD9NFx4aUnVoxO+2qzFCXHcaqYAUX9cHd4pvajYYJRiZiv2qtfM9h3bpPAhibIYOCk8x3Y35gB/QS2vtNqSpwbvOq+mqnZAles6Exr2RDwaZvrzwh9NIDZyPLu5RFSAdfFQIv+pI7aiPh42jk20TrLjRE1vzoDWNzHRY0S3UD61pkCilfMgnhlviIuN8rLtCEGVPdGf9kKF/t7sTm3CtkU4RvcZ/pOGGg0JVQBhI/n4x43lWULepvvPKYUogIP771LIRHhZCwT46pirpgEY9ZW/r33UbMqA3TmFUssNH79FrVJx006InKmc1tsb/GKn6OaEspCPSonyVJO/BvPvqykgxdZ0Gy0YiDGufmn0fqnerlOBodYk4KrxHDyMcXUyZWwLmd0cRg1q37N9MYD/i7X7r2T0uRFg++ZvLnp4aEW8eDSBCnN7lUDVe72X51Jb/uZ5dJEIgYW8= smithi115.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDUepPK3mzf/7dyOYSMoic9jRkFukPVBYkSzsTvq6qbI9uJ6Z+xoWcBoXdWS5lvz22TERdt7f4uQ7dhXix9NxFfIYPZgnPVzLjm6/0+qt5wHYzIqXbajCWl3GC1gPMVu8M1yFOiR4vxwCmQ2SyozhJnLq8568YcnO4y2keWsFFUqz4f2x5RQsVEkS1cY4Ahu8RWDw0xJdYW4clg2kE58RMALA/gYWw8lMGUsV9dqhXT0878GnJFFDJ6Um5AXN6j0ybjP/Feq9MV8zSDDHoN0rdVd9MCpsUbfVJQC4mUyQYjzOEbIlaMnRjqMvC+rGxzGeKzYe3ON8Yw8EHVSeLPKnmGXlaSzI9QZa0lTKRRXKgSV9ixAs5YZEd5VSqlYJVXia90mtKzUTLrbZGdGfDHcap8PAXFY9/VHYkMfV22H9Kyexpk6kr+mU1tzdYqxYZ30UxvvZJQg/F70lkRa8FP0ZjdTfOWj8YSr4+zDZixiyX2qmryCk+iL8/zVJ6eR1+w/Os= tasks: - pexec: all: - sudo cp /etc/containers/registries.conf /etc/containers/registries.conf.backup - sudo dnf -y module reset container-tools - sudo dnf -y module install container-tools:rhel8 - sudo cp /etc/containers/registries.conf.backup /etc/containers/registries.conf - install: extra_packages: deb: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror rpm: - python3-cephfs - cephfs-top - cephfs-mirror extra_system_packages: deb: - bison - flex - libelf-dev - libssl-dev - network-manager - iproute2 - util-linux - dump - indent - libaio-dev - libtool-bin - uuid-dev - xfslibs-dev rpm: - bison - flex - elfutils-libelf-devel - openssl-devel - NetworkManager - iproute - util-linux - libacl-devel - libaio-devel - libattr-devel - libtool - libuuid-devel - xfsdump - xfsprogs - xfsprogs-devel - libaio-devel - libtool - libuuid-devel - xfsprogs-devel - cephadm: roleless: false - cephadm.shell: mon.a: - ceph orch status - ceph orch ps - ceph orch ls - ceph orch host ls - ceph orch device ls - cephadm.shell: mon.a: - ceph fs dump - ceph osd dump - fs.ready: timeout: 300 - kclient: null - check-counter: {} - workunit: clients: all: - suites/fsstress.sh timeout: 6h teuthology_branch: master teuthology_sha1: b018fa018d3f885cbc018f839b42536477653f84 timestamp: 2022-05-12_22:06:41 tube: smithi user: yuriw verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2086 2022-05-13T22:46:08.951 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_23b0658391ce94ca40799469eec00c28b298816c/qa; will attempt to use it 2022-05-13T22:46:08.952 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_23b0658391ce94ca40799469eec00c28b298816c/qa/tasks 2022-05-13T22:46:08.952 INFO:teuthology.run_tasks:Running task internal.check_packages... 2022-05-13T22:46:08.954 INFO:teuthology.task.internal:Checking packages... 2022-05-13T22:46:08.974 INFO:teuthology.task.internal:Checking packages for os_type 'rhel', flavor 'default' and ceph hash '23b0658391ce94ca40799469eec00c28b298816c' 2022-05-13T22:46:08.975 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2022-05-13T22:46:08.975 INFO:teuthology.packaging:ref: None 2022-05-13T22:46:08.976 INFO:teuthology.packaging:tag: None 2022-05-13T22:46:08.976 INFO:teuthology.packaging:branch: wip-yuri-testing-2022-05-10-1027-quincy 2022-05-13T22:46:08.976 INFO:teuthology.packaging:sha1: 23b0658391ce94ca40799469eec00c28b298816c 2022-05-13T22:46:08.977 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=wip-yuri-testing-2022-05-10-1027-quincy 2022-05-13T22:46:09.148 INFO:teuthology.task.internal:Found packages for ceph version 17.2.0-188.g23b06583 2022-05-13T22:46:09.149 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2022-05-13T22:46:09.155 INFO:teuthology.task.internal:no buildpackages task found 2022-05-13T22:46:09.155 INFO:teuthology.run_tasks:Running task internal.save_config... 2022-05-13T22:46:09.204 INFO:teuthology.task.internal:Saving configuration 2022-05-13T22:46:09.226 INFO:teuthology.run_tasks:Running task internal.check_lock... 2022-05-13T22:46:09.240 INFO:teuthology.task.internal.check_lock:Checking locks... 2022-05-13T22:46:09.267 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi002.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2022-05-12_22:06:41-fs-wip-yuri-testing-2022-05-10-1027-quincy-distro-default-smithi/6830937', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.4', 'arch': 'x86_64', 'locked': True, 'locked_since': '2022-05-13 22:41:03.919671', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBOCLPu/hwRgAetRZLYrNhE57u0+sJlnRtJ9bHkHxsBEJwUldq+0d9lpMwkT17de6uuSs4Lty5k07W7cCQG1S7oY='} 2022-05-13T22:46:09.293 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi047.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2022-05-12_22:06:41-fs-wip-yuri-testing-2022-05-10-1027-quincy-distro-default-smithi/6830937', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.4', 'arch': 'x86_64', 'locked': True, 'locked_since': '2022-05-13 22:41:03.917893', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLhlxLQqpMY1dLYeEybe0xRKjNBDYD0iFISphIE4bZHdSMiV3GVA+9MUwV/VPo3Q/71xyiBbsbLaoCGcMtwQDK8='} 2022-05-13T22:46:09.325 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi115.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2022-05-12_22:06:41-fs-wip-yuri-testing-2022-05-10-1027-quincy-distro-default-smithi/6830937', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.4', 'arch': 'x86_64', 'locked': True, 'locked_since': '2022-05-13 22:41:03.920881', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJTDDM9oZ5gy16lma9HQNDIfpteKiVuv0iv72maNgGv5eiDjL4/ZGvwjtzHSkywRHP1KBcvLK/TP3nAsTs9sGjE='} 2022-05-13T22:46:09.326 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2022-05-13T22:46:09.333 INFO:teuthology.task.internal:roles: ubuntu@smithi002.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.d', 'mds.g', 'mds.j', 'osd.0', 'osd.3', 'osd.6', 'osd.9', 'client.0'] 2022-05-13T22:46:09.333 INFO:teuthology.task.internal:roles: ubuntu@smithi047.front.sepia.ceph.com - ['mon.b', 'mgr.y', 'mds.b', 'mds.e', 'mds.h', 'mds.k', 'osd.1', 'osd.4', 'osd.7', 'osd.10'] 2022-05-13T22:46:09.333 INFO:teuthology.task.internal:roles: ubuntu@smithi115.front.sepia.ceph.com - ['mon.c', 'mgr.z', 'mds.c', 'mds.f', 'mds.i', 'mds.l', 'osd.2', 'osd.5', 'osd.8', 'osd.11'] 2022-05-13T22:46:09.334 INFO:teuthology.run_tasks:Running task console_log... 2022-05-13T22:46:09.450 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fd0b063de18>, signals=[15]) 2022-05-13T22:46:09.451 INFO:teuthology.run_tasks:Running task internal.connect... 2022-05-13T22:46:09.457 INFO:teuthology.task.internal:Opening connections... 2022-05-13T22:46:09.458 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi002.front.sepia.ceph.com 2022-05-13T22:46:09.460 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi002.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-05-13T22:46:09.542 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi047.front.sepia.ceph.com 2022-05-13T22:46:09.543 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi047.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-05-13T22:46:09.619 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi115.front.sepia.ceph.com 2022-05-13T22:46:09.620 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi115.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-05-13T22:46:09.715 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2022-05-13T22:46:09.722 DEBUG:teuthology.orchestra.run.smithi002:> uname -m 2022-05-13T22:46:09.742 INFO:teuthology.orchestra.run.smithi002.stdout:x86_64 2022-05-13T22:46:09.743 DEBUG:teuthology.orchestra.run.smithi002:> cat /etc/os-release 2022-05-13T22:46:09.799 INFO:teuthology.orchestra.run.smithi002.stdout:NAME="Red Hat Enterprise Linux" 2022-05-13T22:46:09.799 INFO:teuthology.orchestra.run.smithi002.stdout:VERSION="8.4 (Ootpa)" 2022-05-13T22:46:09.800 INFO:teuthology.orchestra.run.smithi002.stdout:ID="rhel" 2022-05-13T22:46:09.800 INFO:teuthology.orchestra.run.smithi002.stdout:ID_LIKE="fedora" 2022-05-13T22:46:09.800 INFO:teuthology.orchestra.run.smithi002.stdout:VERSION_ID="8.4" 2022-05-13T22:46:09.800 INFO:teuthology.orchestra.run.smithi002.stdout:PLATFORM_ID="platform:el8" 2022-05-13T22:46:09.801 INFO:teuthology.orchestra.run.smithi002.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.4 (Ootpa)" 2022-05-13T22:46:09.801 INFO:teuthology.orchestra.run.smithi002.stdout:ANSI_COLOR="0;31" 2022-05-13T22:46:09.801 INFO:teuthology.orchestra.run.smithi002.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8.4:GA" 2022-05-13T22:46:09.801 INFO:teuthology.orchestra.run.smithi002.stdout:HOME_URL="https://www.redhat.com/" 2022-05-13T22:46:09.801 INFO:teuthology.orchestra.run.smithi002.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/" 2022-05-13T22:46:09.802 INFO:teuthology.orchestra.run.smithi002.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2022-05-13T22:46:09.802 INFO:teuthology.orchestra.run.smithi002.stdout: 2022-05-13T22:46:09.802 INFO:teuthology.orchestra.run.smithi002.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" 2022-05-13T22:46:09.802 INFO:teuthology.orchestra.run.smithi002.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.4 2022-05-13T22:46:09.802 INFO:teuthology.orchestra.run.smithi002.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2022-05-13T22:46:09.803 INFO:teuthology.orchestra.run.smithi002.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.4" 2022-05-13T22:46:09.804 INFO:teuthology.lock.ops:Updating smithi002.front.sepia.ceph.com on lock server 2022-05-13T22:46:09.823 DEBUG:teuthology.orchestra.run.smithi047:> uname -m 2022-05-13T22:46:09.841 INFO:teuthology.orchestra.run.smithi047.stdout:x86_64 2022-05-13T22:46:09.842 DEBUG:teuthology.orchestra.run.smithi047:> cat /etc/os-release 2022-05-13T22:46:09.899 INFO:teuthology.orchestra.run.smithi047.stdout:NAME="Red Hat Enterprise Linux" 2022-05-13T22:46:09.899 INFO:teuthology.orchestra.run.smithi047.stdout:VERSION="8.4 (Ootpa)" 2022-05-13T22:46:09.900 INFO:teuthology.orchestra.run.smithi047.stdout:ID="rhel" 2022-05-13T22:46:09.900 INFO:teuthology.orchestra.run.smithi047.stdout:ID_LIKE="fedora" 2022-05-13T22:46:09.900 INFO:teuthology.orchestra.run.smithi047.stdout:VERSION_ID="8.4" 2022-05-13T22:46:09.900 INFO:teuthology.orchestra.run.smithi047.stdout:PLATFORM_ID="platform:el8" 2022-05-13T22:46:09.900 INFO:teuthology.orchestra.run.smithi047.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.4 (Ootpa)" 2022-05-13T22:46:09.901 INFO:teuthology.orchestra.run.smithi047.stdout:ANSI_COLOR="0;31" 2022-05-13T22:46:09.901 INFO:teuthology.orchestra.run.smithi047.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8.4:GA" 2022-05-13T22:46:09.901 INFO:teuthology.orchestra.run.smithi047.stdout:HOME_URL="https://www.redhat.com/" 2022-05-13T22:46:09.901 INFO:teuthology.orchestra.run.smithi047.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/" 2022-05-13T22:46:09.901 INFO:teuthology.orchestra.run.smithi047.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2022-05-13T22:46:09.902 INFO:teuthology.orchestra.run.smithi047.stdout: 2022-05-13T22:46:09.902 INFO:teuthology.orchestra.run.smithi047.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" 2022-05-13T22:46:09.902 INFO:teuthology.orchestra.run.smithi047.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.4 2022-05-13T22:46:09.902 INFO:teuthology.orchestra.run.smithi047.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2022-05-13T22:46:09.903 INFO:teuthology.orchestra.run.smithi047.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.4" 2022-05-13T22:46:09.903 INFO:teuthology.lock.ops:Updating smithi047.front.sepia.ceph.com on lock server 2022-05-13T22:46:09.921 DEBUG:teuthology.orchestra.run.smithi115:> uname -m 2022-05-13T22:46:09.938 INFO:teuthology.orchestra.run.smithi115.stdout:x86_64 2022-05-13T22:46:09.938 DEBUG:teuthology.orchestra.run.smithi115:> cat /etc/os-release 2022-05-13T22:46:09.994 INFO:teuthology.orchestra.run.smithi115.stdout:NAME="Red Hat Enterprise Linux" 2022-05-13T22:46:09.995 INFO:teuthology.orchestra.run.smithi115.stdout:VERSION="8.4 (Ootpa)" 2022-05-13T22:46:09.995 INFO:teuthology.orchestra.run.smithi115.stdout:ID="rhel" 2022-05-13T22:46:09.995 INFO:teuthology.orchestra.run.smithi115.stdout:ID_LIKE="fedora" 2022-05-13T22:46:09.996 INFO:teuthology.orchestra.run.smithi115.stdout:VERSION_ID="8.4" 2022-05-13T22:46:09.996 INFO:teuthology.orchestra.run.smithi115.stdout:PLATFORM_ID="platform:el8" 2022-05-13T22:46:09.996 INFO:teuthology.orchestra.run.smithi115.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.4 (Ootpa)" 2022-05-13T22:46:09.996 INFO:teuthology.orchestra.run.smithi115.stdout:ANSI_COLOR="0;31" 2022-05-13T22:46:09.996 INFO:teuthology.orchestra.run.smithi115.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8.4:GA" 2022-05-13T22:46:09.997 INFO:teuthology.orchestra.run.smithi115.stdout:HOME_URL="https://www.redhat.com/" 2022-05-13T22:46:09.997 INFO:teuthology.orchestra.run.smithi115.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/" 2022-05-13T22:46:09.997 INFO:teuthology.orchestra.run.smithi115.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2022-05-13T22:46:09.997 INFO:teuthology.orchestra.run.smithi115.stdout: 2022-05-13T22:46:09.998 INFO:teuthology.orchestra.run.smithi115.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" 2022-05-13T22:46:09.998 INFO:teuthology.orchestra.run.smithi115.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.4 2022-05-13T22:46:09.998 INFO:teuthology.orchestra.run.smithi115.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2022-05-13T22:46:09.998 INFO:teuthology.orchestra.run.smithi115.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.4" 2022-05-13T22:46:09.999 INFO:teuthology.lock.ops:Updating smithi115.front.sepia.ceph.com on lock server 2022-05-13T22:46:10.017 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2022-05-13T22:46:10.030 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2022-05-13T22:46:10.067 INFO:teuthology.task.internal:Checking for old test directory... 2022-05-13T22:46:10.068 DEBUG:teuthology.orchestra.run.smithi002:> test '!' -e /home/ubuntu/cephtest 2022-05-13T22:46:10.072 DEBUG:teuthology.orchestra.run.smithi047:> test '!' -e /home/ubuntu/cephtest 2022-05-13T22:46:10.075 DEBUG:teuthology.orchestra.run.smithi115:> test '!' -e /home/ubuntu/cephtest 2022-05-13T22:46:10.091 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2022-05-13T22:46:10.096 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2022-05-13T22:46:10.097 DEBUG:teuthology.orchestra.run.smithi002:> test -z $(ls -A /var/lib/ceph) 2022-05-13T22:46:10.129 DEBUG:teuthology.orchestra.run.smithi047:> test -z $(ls -A /var/lib/ceph) 2022-05-13T22:46:10.133 DEBUG:teuthology.orchestra.run.smithi115:> test -z $(ls -A /var/lib/ceph) 2022-05-13T22:46:10.161 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2022-05-13T22:46:10.248 INFO:teuthology.run_tasks:Running task kernel... 2022-05-13T22:46:10.263 INFO:teuthology.task.kernel:normalize config orig: {'client': {'branch': 'testing'}, 'kdb': True, 'sha1': 'distro'} 2022-05-13T22:46:10.264 INFO:teuthology.task.kernel:normalize config final: {'client.0': {'branch': 'testing'}} 2022-05-13T22:46:10.265 INFO:teuthology.task.kernel:config {'client.0': {'branch': 'testing'}}, timeout 300 2022-05-13T22:46:10.265 DEBUG:teuthology.orchestra.run.smithi002:> test -f /run/.containerenv -o -f /.dockerenv 2022-05-13T22:46:10.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-05-13T22:46:10.283 DEBUG:teuthology.orchestra.run.smithi002:> sudo lsb_release -is 2022-05-13T22:46:10.398 INFO:teuthology.orchestra.run.smithi002.stdout:RedHatEnterprise 2022-05-13T22:46:10.399 DEBUG:teuthology.misc:System to be installed: RedHatEnterprise 2022-05-13T22:46:10.399 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=centos%2F8%2Fx86_64&ref=testing 2022-05-13T22:46:10.717 DEBUG:teuthology.task.kernel:sha1 for client.0 is b850759e370d7607466610dcd560ae8317630693 2022-05-13T22:46:10.718 INFO:teuthology.task.kernel:Checking kernel version of client.0, want "b850759e370d7607466610dcd560ae8317630693"... 2022-05-13T22:46:10.718 DEBUG:teuthology.orchestra.run.smithi002:> uname -r 2022-05-13T22:46:10.736 INFO:teuthology.orchestra.run.smithi002.stdout:4.18.0-348.23.1.el8_5.x86_64 2022-05-13T22:46:10.737 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-348.23.1.el8_5.x86_64 vs b850759e370d7607466610dcd560ae8317630693 2022-05-13T22:46:10.737 DEBUG:teuthology.task.kernel:failed to parse current kernel version 2022-05-13T22:46:10.738 DEBUG:teuthology.orchestra.run.smithi002:> sudo yum upgrade -y linux-firmware 2022-05-13T22:46:12.073 INFO:teuthology.orchestra.run.smithi002.stdout:Updating Subscription Management repositories. 2022-05-13T22:46:12.074 INFO:teuthology.orchestra.run.smithi002.stdout:Unable to read consumer identity 2022-05-13T22:46:12.168 INFO:teuthology.orchestra.run.smithi002.stdout: 2022-05-13T22:46:12.169 INFO:teuthology.orchestra.run.smithi002.stdout:This system is not registered to Red Hat Subscription Management. You can use subscription-manager to register. 2022-05-13T22:46:12.169 INFO:teuthology.orchestra.run.smithi002.stdout: 2022-05-13T22:46:13.701 INFO:teuthology.orchestra.run.smithi002.stdout:Copr repo for python3-asyncssh owned by ceph 29 kB/s | 3.5 kB 00:00 2022-05-13T22:46:14.705 INFO:teuthology.orchestra.run.smithi002.stdout:Extra Packages for Enterprise Linux 11 MB/s | 11 MB 00:00 2022-05-13T22:46:17.162 INFO:teuthology.orchestra.run.smithi002.stdout:lab-extras 4.1 MB/s | 24 kB 00:00 2022-05-13T22:46:17.966 INFO:teuthology.orchestra.run.smithi002.stdout:Dependencies resolved. 2022-05-13T22:46:17.967 INFO:teuthology.orchestra.run.smithi002.stdout:Nothing to do. 2022-05-13T22:46:17.967 INFO:teuthology.orchestra.run.smithi002.stdout:Complete! 2022-05-13T22:46:18.092 INFO:teuthology.task.kernel:Downloading kernel b850759e370d7607466610dcd560ae8317630693 on client.0... 2022-05-13T22:46:18.093 DEBUG:teuthology.orchestra.run.smithi002:> sudo lsb_release -is 2022-05-13T22:46:18.172 INFO:teuthology.orchestra.run.smithi002.stdout:RedHatEnterprise 2022-05-13T22:46:18.173 DEBUG:teuthology.misc:System to be installed: RedHatEnterprise 2022-05-13T22:46:18.174 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=centos%2F8%2Fx86_64&sha1=b850759e370d7607466610dcd560ae8317630693 2022-05-13T22:46:18.383 INFO:teuthology.task.kernel:fetching, builder baseurl is https://1.chacra.ceph.com/r/kernel/testing/b850759e370d7607466610dcd560ae8317630693/centos/8/flavors/default/x86_64/ 2022-05-13T22:46:18.384 DEBUG:teuthology.orchestra.run.smithi002:> rm -f /tmp/kernel.x86_64.rpm && echo kernel-5.18.0_rc6_ceph_gb850759e370d-1.x86_64.rpm | wget -nv -O /tmp/kernel.x86_64.rpm --base=https://1.chacra.ceph.com/r/kernel/testing/b850759e370d7607466610dcd560ae8317630693/centos/8/flavors/default/x86_64/ --input-file=- 2022-05-13T22:46:18.386 DEBUG:teuthology.task.kernel:Waiting for download/copy to ubuntu@smithi002.front.sepia.ceph.com to complete... 2022-05-13T22:46:34.759 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13 22:46:34 URL:https://1.chacra.ceph.com/r/kernel/testing/b850759e370d7607466610dcd560ae8317630693/centos/8/flavors/default/x86_64/kernel-5.18.0_rc6_ceph_gb850759e370d-1.x86_64.rpm [45223780/45223780] -> "/tmp/kernel.x86_64.rpm" [1] 2022-05-13T22:46:34.759 INFO:teuthology.orchestra.run.smithi002.stderr:FINISHED --2022-05-13 22:46:34-- 2022-05-13T22:46:34.760 INFO:teuthology.orchestra.run.smithi002.stderr:Total wall clock time: 16s 2022-05-13T22:46:34.760 INFO:teuthology.orchestra.run.smithi002.stderr:Downloaded: 1 files, 43M in 16s (2.71 MB/s) 2022-05-13T22:46:34.762 INFO:teuthology.task.kernel:Installing kernel b850759e370d7607466610dcd560ae8317630693 on client.0... 2022-05-13T22:46:34.762 DEBUG:teuthology.orchestra.run.smithi002:> sudo rpm -ivh --oldpackage --replacefiles --replacepkgs /tmp/kernel.x86_64.rpm 2022-05-13T22:46:35.330 INFO:teuthology.orchestra.run.smithi002.stdout:Verifying... ######################################## 2022-05-13T22:46:35.453 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing... ######################################## 2022-05-13T22:46:35.579 INFO:teuthology.orchestra.run.smithi002.stdout:Updating / installing... 2022-05-13T22:46:38.880 INFO:teuthology.orchestra.run.smithi002.stdout:kernel-5.18.0_rc6_ceph_gb850759e370d-1######################################## 2022-05-13T22:46:51.262 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi002.front.sepia.ceph.com, path=/tmp/kernel.x86_64.rpm, version=None) 2022-05-13T22:46:51.264 DEBUG:teuthology.orchestra.run.smithi002:> rpm -qlp /tmp/kernel.x86_64.rpm 2022-05-13T22:46:51.321 INFO:teuthology.orchestra.run.smithi002.stdout:/boot/System.map-5.18.0-rc6-ceph-gb850759e370d 2022-05-13T22:46:51.322 INFO:teuthology.orchestra.run.smithi002.stdout:/boot/config-5.18.0-rc6-ceph-gb850759e370d 2022-05-13T22:46:51.322 INFO:teuthology.orchestra.run.smithi002.stdout:/boot/vmlinux-5.18.0-rc6-ceph-gb850759e370d.bz2 2022-05-13T22:46:51.322 INFO:teuthology.orchestra.run.smithi002.stdout:/boot/vmlinuz-5.18.0-rc6-ceph-gb850759e370d 2022-05-13T22:46:51.323 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d 2022-05-13T22:46:51.323 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel 2022-05-13T22:46:51.323 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/arch 2022-05-13T22:46:51.323 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/arch/x86 2022-05-13T22:46:51.324 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/arch/x86/crypto 2022-05-13T22:46:51.324 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/arch/x86/crypto/blowfish-x86_64.ko 2022-05-13T22:46:51.324 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/arch/x86/crypto/camellia-aesni-avx-x86_64.ko 2022-05-13T22:46:51.324 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/arch/x86/crypto/camellia-aesni-avx2.ko 2022-05-13T22:46:51.324 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/arch/x86/crypto/camellia-x86_64.ko 2022-05-13T22:46:51.325 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/arch/x86/crypto/cast5-avx-x86_64.ko 2022-05-13T22:46:51.325 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/arch/x86/crypto/cast6-avx-x86_64.ko 2022-05-13T22:46:51.325 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/arch/x86/crypto/crc32-pclmul.ko 2022-05-13T22:46:51.325 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/arch/x86/crypto/crc32c-intel.ko 2022-05-13T22:46:51.326 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/arch/x86/crypto/crct10dif-pclmul.ko 2022-05-13T22:46:51.326 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/arch/x86/crypto/des3_ede-x86_64.ko 2022-05-13T22:46:51.326 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/arch/x86/crypto/ghash-clmulni-intel.ko 2022-05-13T22:46:51.326 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/arch/x86/crypto/serpent-avx-x86_64.ko 2022-05-13T22:46:51.326 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/arch/x86/crypto/serpent-avx2.ko 2022-05-13T22:46:51.327 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/arch/x86/crypto/serpent-sse2-x86_64.ko 2022-05-13T22:46:51.327 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/arch/x86/crypto/sha1-ssse3.ko 2022-05-13T22:46:51.327 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/arch/x86/crypto/sha256-ssse3.ko 2022-05-13T22:46:51.327 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/arch/x86/crypto/sha512-ssse3.ko 2022-05-13T22:46:51.328 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/arch/x86/crypto/twofish-avx-x86_64.ko 2022-05-13T22:46:51.328 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/arch/x86/crypto/twofish-x86_64-3way.ko 2022-05-13T22:46:51.328 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/arch/x86/crypto/twofish-x86_64.ko 2022-05-13T22:46:51.328 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/arch/x86/kernel 2022-05-13T22:46:51.328 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/arch/x86/kernel/cpu 2022-05-13T22:46:51.329 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/arch/x86/kernel/cpu/mce 2022-05-13T22:46:51.329 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/arch/x86/kernel/cpu/mce/mce-inject.ko 2022-05-13T22:46:51.329 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/arch/x86/kvm 2022-05-13T22:46:51.329 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/arch/x86/kvm/kvm-amd.ko 2022-05-13T22:46:51.329 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/arch/x86/kvm/kvm-intel.ko 2022-05-13T22:46:51.330 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/arch/x86/kvm/kvm.ko 2022-05-13T22:46:51.330 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/crypto 2022-05-13T22:46:51.330 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/crypto/ansi_cprng.ko 2022-05-13T22:46:51.330 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/crypto/anubis.ko 2022-05-13T22:46:51.330 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/crypto/arc4.ko 2022-05-13T22:46:51.331 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/crypto/async_tx 2022-05-13T22:46:51.331 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/crypto/async_tx/async_memcpy.ko 2022-05-13T22:46:51.331 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/crypto/async_tx/async_pq.ko 2022-05-13T22:46:51.331 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/crypto/async_tx/async_raid6_recov.ko 2022-05-13T22:46:51.332 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/crypto/async_tx/async_tx.ko 2022-05-13T22:46:51.332 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/crypto/async_tx/async_xor.ko 2022-05-13T22:46:51.332 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/crypto/async_tx/raid6test.ko 2022-05-13T22:46:51.332 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/crypto/authenc.ko 2022-05-13T22:46:51.333 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/crypto/authencesn.ko 2022-05-13T22:46:51.333 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/crypto/blake2b_generic.ko 2022-05-13T22:46:51.333 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/crypto/blowfish_common.ko 2022-05-13T22:46:51.333 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/crypto/blowfish_generic.ko 2022-05-13T22:46:51.333 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/crypto/camellia_generic.ko 2022-05-13T22:46:51.334 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/crypto/cast5_generic.ko 2022-05-13T22:46:51.334 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/crypto/cast6_generic.ko 2022-05-13T22:46:51.334 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/crypto/cast_common.ko 2022-05-13T22:46:51.334 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/crypto/ccm.ko 2022-05-13T22:46:51.334 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/crypto/cmac.ko 2022-05-13T22:46:51.335 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/crypto/crc32_generic.ko 2022-05-13T22:46:51.335 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/crypto/crypto_engine.ko 2022-05-13T22:46:51.335 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/crypto/crypto_user.ko 2022-05-13T22:46:51.335 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/crypto/cts.ko 2022-05-13T22:46:51.335 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/crypto/des_generic.ko 2022-05-13T22:46:51.336 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/crypto/dh_generic.ko 2022-05-13T22:46:51.336 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/crypto/echainiv.ko 2022-05-13T22:46:51.336 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/crypto/essiv.ko 2022-05-13T22:46:51.336 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/crypto/fcrypt.ko 2022-05-13T22:46:51.337 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/crypto/khazad.ko 2022-05-13T22:46:51.337 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/crypto/lz4.ko 2022-05-13T22:46:51.337 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/crypto/lz4hc.ko 2022-05-13T22:46:51.337 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/crypto/md4.ko 2022-05-13T22:46:51.337 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/crypto/michael_mic.ko 2022-05-13T22:46:51.338 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/crypto/pcbc.ko 2022-05-13T22:46:51.338 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/crypto/pcrypt.ko 2022-05-13T22:46:51.338 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/crypto/rmd160.ko 2022-05-13T22:46:51.338 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/crypto/seed.ko 2022-05-13T22:46:51.338 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/crypto/serpent_generic.ko 2022-05-13T22:46:51.339 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/crypto/tcrypt.ko 2022-05-13T22:46:51.339 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/crypto/tea.ko 2022-05-13T22:46:51.339 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/crypto/twofish_common.ko 2022-05-13T22:46:51.339 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/crypto/twofish_generic.ko 2022-05-13T22:46:51.339 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/crypto/vmac.ko 2022-05-13T22:46:51.340 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/crypto/wp512.ko 2022-05-13T22:46:51.340 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/crypto/xcbc.ko 2022-05-13T22:46:51.340 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/crypto/xor.ko 2022-05-13T22:46:51.340 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/crypto/xxhash_generic.ko 2022-05-13T22:46:51.341 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers 2022-05-13T22:46:51.341 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/acpi 2022-05-13T22:46:51.341 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/acpi/acpi_ipmi.ko 2022-05-13T22:46:51.341 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/acpi/acpi_pad.ko 2022-05-13T22:46:51.341 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/acpi/custom_method.ko 2022-05-13T22:46:51.342 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/acpi/ec_sys.ko 2022-05-13T22:46:51.342 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/acpi/platform_profile.ko 2022-05-13T22:46:51.342 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/acpi/sbs.ko 2022-05-13T22:46:51.342 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/acpi/sbshc.ko 2022-05-13T22:46:51.342 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/acpi/video.ko 2022-05-13T22:46:51.343 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/ata 2022-05-13T22:46:51.343 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/ata/acard-ahci.ko 2022-05-13T22:46:51.343 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/ata/ahci_platform.ko 2022-05-13T22:46:51.343 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/ata/ata_generic.ko 2022-05-13T22:46:51.343 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/ata/libahci_platform.ko 2022-05-13T22:46:51.344 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/ata/pata_acpi.ko 2022-05-13T22:46:51.344 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/ata/pata_ali.ko 2022-05-13T22:46:51.344 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/ata/pata_amd.ko 2022-05-13T22:46:51.344 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/ata/pata_artop.ko 2022-05-13T22:46:51.344 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/ata/pata_atiixp.ko 2022-05-13T22:46:51.345 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/ata/pata_atp867x.ko 2022-05-13T22:46:51.345 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/ata/pata_cmd640.ko 2022-05-13T22:46:51.345 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/ata/pata_cmd64x.ko 2022-05-13T22:46:51.345 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/ata/pata_cypress.ko 2022-05-13T22:46:51.346 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/ata/pata_efar.ko 2022-05-13T22:46:51.346 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/ata/pata_hpt366.ko 2022-05-13T22:46:51.346 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/ata/pata_hpt37x.ko 2022-05-13T22:46:51.346 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/ata/pata_hpt3x2n.ko 2022-05-13T22:46:51.346 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/ata/pata_hpt3x3.ko 2022-05-13T22:46:51.347 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/ata/pata_it8213.ko 2022-05-13T22:46:51.347 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/ata/pata_it821x.ko 2022-05-13T22:46:51.347 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/ata/pata_jmicron.ko 2022-05-13T22:46:51.347 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/ata/pata_marvell.ko 2022-05-13T22:46:51.347 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/ata/pata_mpiix.ko 2022-05-13T22:46:51.348 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/ata/pata_netcell.ko 2022-05-13T22:46:51.348 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/ata/pata_ninja32.ko 2022-05-13T22:46:51.348 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/ata/pata_ns87410.ko 2022-05-13T22:46:51.348 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/ata/pata_ns87415.ko 2022-05-13T22:46:51.349 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/ata/pata_oldpiix.ko 2022-05-13T22:46:51.349 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/ata/pata_opti.ko 2022-05-13T22:46:51.349 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/ata/pata_optidma.ko 2022-05-13T22:46:51.349 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/ata/pata_pcmcia.ko 2022-05-13T22:46:51.349 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/ata/pata_pdc2027x.ko 2022-05-13T22:46:51.350 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/ata/pata_pdc202xx_old.ko 2022-05-13T22:46:51.350 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/ata/pata_piccolo.ko 2022-05-13T22:46:51.350 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/ata/pata_rdc.ko 2022-05-13T22:46:51.350 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/ata/pata_sch.ko 2022-05-13T22:46:51.350 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/ata/pata_serverworks.ko 2022-05-13T22:46:51.351 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/ata/pata_sil680.ko 2022-05-13T22:46:51.351 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/ata/pata_sis.ko 2022-05-13T22:46:51.351 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/ata/pata_sl82c105.ko 2022-05-13T22:46:51.351 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/ata/pata_triflex.ko 2022-05-13T22:46:51.351 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/ata/pata_via.ko 2022-05-13T22:46:51.352 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/ata/pdc_adma.ko 2022-05-13T22:46:51.352 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/ata/sata_inic162x.ko 2022-05-13T22:46:51.352 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/ata/sata_mv.ko 2022-05-13T22:46:51.352 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/ata/sata_nv.ko 2022-05-13T22:46:51.352 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/ata/sata_promise.ko 2022-05-13T22:46:51.353 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/ata/sata_qstor.ko 2022-05-13T22:46:51.353 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/ata/sata_sil.ko 2022-05-13T22:46:51.353 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/ata/sata_sil24.ko 2022-05-13T22:46:51.353 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/ata/sata_sis.ko 2022-05-13T22:46:51.353 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/ata/sata_svw.ko 2022-05-13T22:46:51.354 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/ata/sata_sx4.ko 2022-05-13T22:46:51.354 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/ata/sata_uli.ko 2022-05-13T22:46:51.354 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/ata/sata_via.ko 2022-05-13T22:46:51.354 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/ata/sata_vsc.ko 2022-05-13T22:46:51.355 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/auxdisplay 2022-05-13T22:46:51.355 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/auxdisplay/ks0108.ko 2022-05-13T22:46:51.355 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/base 2022-05-13T22:46:51.355 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/base/regmap 2022-05-13T22:46:51.355 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/base/regmap/regmap-i2c.ko 2022-05-13T22:46:51.356 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/bcma 2022-05-13T22:46:51.356 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/bcma/bcma.ko 2022-05-13T22:46:51.356 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/block 2022-05-13T22:46:51.356 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/block/aoe 2022-05-13T22:46:51.356 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/block/aoe/aoe.ko 2022-05-13T22:46:51.357 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/block/brd.ko 2022-05-13T22:46:51.357 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/block/drbd 2022-05-13T22:46:51.357 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/block/drbd/drbd.ko 2022-05-13T22:46:51.358 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/block/floppy.ko 2022-05-13T22:46:51.358 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/block/loop.ko 2022-05-13T22:46:51.358 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/block/mtip32xx 2022-05-13T22:46:51.358 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/block/mtip32xx/mtip32xx.ko 2022-05-13T22:46:51.358 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/block/nbd.ko 2022-05-13T22:46:51.359 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/block/null_blk 2022-05-13T22:46:51.359 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/block/null_blk/null_blk.ko 2022-05-13T22:46:51.359 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/block/pktcdvd.ko 2022-05-13T22:46:51.359 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/block/rbd.ko 2022-05-13T22:46:51.359 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/block/sx8.ko 2022-05-13T22:46:51.360 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/block/virtio_blk.ko 2022-05-13T22:46:51.360 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/block/xen-blkback 2022-05-13T22:46:51.360 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/block/xen-blkback/xen-blkback.ko 2022-05-13T22:46:51.360 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/block/xen-blkfront.ko 2022-05-13T22:46:51.360 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/block/zram 2022-05-13T22:46:51.361 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/block/zram/zram.ko 2022-05-13T22:46:51.361 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/char 2022-05-13T22:46:51.361 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/char/hangcheck-timer.ko 2022-05-13T22:46:51.361 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/char/hw_random 2022-05-13T22:46:51.362 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/char/hw_random/amd-rng.ko 2022-05-13T22:46:51.362 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/char/hw_random/intel-rng.ko 2022-05-13T22:46:51.362 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/char/hw_random/timeriomem-rng.ko 2022-05-13T22:46:51.362 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/char/hw_random/via-rng.ko 2022-05-13T22:46:51.362 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/char/hw_random/virtio-rng.ko 2022-05-13T22:46:51.363 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/char/ipmi 2022-05-13T22:46:51.363 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/char/ipmi/ipmi_devintf.ko 2022-05-13T22:46:51.363 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/char/ipmi/ipmi_msghandler.ko 2022-05-13T22:46:51.363 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/char/ipmi/ipmi_poweroff.ko 2022-05-13T22:46:51.363 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/char/ipmi/ipmi_si.ko 2022-05-13T22:46:51.364 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/char/ipmi/ipmi_ssif.ko 2022-05-13T22:46:51.364 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/char/ipmi/ipmi_watchdog.ko 2022-05-13T22:46:51.364 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/char/lp.ko 2022-05-13T22:46:51.364 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/char/mwave 2022-05-13T22:46:51.364 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/char/mwave/mwave.ko 2022-05-13T22:46:51.365 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/char/pcmcia 2022-05-13T22:46:51.365 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/char/pcmcia/cm4000_cs.ko 2022-05-13T22:46:51.365 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/char/pcmcia/cm4040_cs.ko 2022-05-13T22:46:51.366 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/char/ppdev.ko 2022-05-13T22:46:51.366 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/char/tlclk.ko 2022-05-13T22:46:51.366 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/char/tpm 2022-05-13T22:46:51.366 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/char/tpm/tpm.ko 2022-05-13T22:46:51.367 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/char/tpm/tpm_atmel.ko 2022-05-13T22:46:51.367 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/char/tpm/tpm_infineon.ko 2022-05-13T22:46:51.367 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/char/tpm/tpm_nsc.ko 2022-05-13T22:46:51.367 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/char/tpm/tpm_tis.ko 2022-05-13T22:46:51.367 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/char/tpm/tpm_tis_core.ko 2022-05-13T22:46:51.368 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/char/uv_mmtimer.ko 2022-05-13T22:46:51.368 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/char/virtio_console.ko 2022-05-13T22:46:51.368 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/cpufreq 2022-05-13T22:46:51.368 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/cpufreq/acpi-cpufreq.ko 2022-05-13T22:46:51.368 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/cpufreq/amd_freq_sensitivity.ko 2022-05-13T22:46:51.369 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/cpufreq/p4-clockmod.ko 2022-05-13T22:46:51.369 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/cpufreq/pcc-cpufreq.ko 2022-05-13T22:46:51.369 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/cpufreq/powernow-k8.ko 2022-05-13T22:46:51.369 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/cpufreq/speedstep-lib.ko 2022-05-13T22:46:51.370 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/crypto 2022-05-13T22:46:51.370 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/crypto/ccp 2022-05-13T22:46:51.370 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/crypto/ccp/ccp-crypto.ko 2022-05-13T22:46:51.370 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/crypto/ccp/ccp.ko 2022-05-13T22:46:51.370 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/crypto/padlock-aes.ko 2022-05-13T22:46:51.371 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/crypto/padlock-sha.ko 2022-05-13T22:46:51.371 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/crypto/qat 2022-05-13T22:46:51.371 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/crypto/qat/qat_common 2022-05-13T22:46:51.371 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/crypto/qat/qat_common/intel_qat.ko 2022-05-13T22:46:51.371 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/crypto/qat/qat_dh895xcc 2022-05-13T22:46:51.372 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/crypto/qat/qat_dh895xcc/qat_dh895xcc.ko 2022-05-13T22:46:51.372 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/crypto/virtio 2022-05-13T22:46:51.372 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/crypto/virtio/virtio_crypto.ko 2022-05-13T22:46:51.372 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/dca 2022-05-13T22:46:51.373 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/dca/dca.ko 2022-05-13T22:46:51.373 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/devfreq 2022-05-13T22:46:51.373 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/devfreq/governor_simpleondemand.ko 2022-05-13T22:46:51.373 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/dma 2022-05-13T22:46:51.373 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/dma/dw 2022-05-13T22:46:51.374 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/dma/dw/dw_dmac.ko 2022-05-13T22:46:51.374 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/dma/ioat 2022-05-13T22:46:51.374 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/dma/ioat/ioatdma.ko 2022-05-13T22:46:51.375 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/edac 2022-05-13T22:46:51.375 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/edac/amd64_edac.ko 2022-05-13T22:46:51.375 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/edac/e752x_edac.ko 2022-05-13T22:46:51.375 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/edac/edac_mce_amd.ko 2022-05-13T22:46:51.375 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/edac/i3000_edac.ko 2022-05-13T22:46:51.376 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/edac/i3200_edac.ko 2022-05-13T22:46:51.376 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/edac/i5000_edac.ko 2022-05-13T22:46:51.376 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/edac/i5100_edac.ko 2022-05-13T22:46:51.376 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/edac/i5400_edac.ko 2022-05-13T22:46:51.377 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/edac/i7300_edac.ko 2022-05-13T22:46:51.377 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/edac/i7core_edac.ko 2022-05-13T22:46:51.377 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/edac/i82975x_edac.ko 2022-05-13T22:46:51.377 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/edac/ie31200_edac.ko 2022-05-13T22:46:51.377 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/edac/sb_edac.ko 2022-05-13T22:46:51.378 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/edac/x38_edac.ko 2022-05-13T22:46:51.378 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/firmware 2022-05-13T22:46:51.378 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/firmware/edd.ko 2022-05-13T22:46:51.378 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/firmware/iscsi_ibft.ko 2022-05-13T22:46:51.385 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/gpio 2022-05-13T22:46:51.385 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/gpio/gpio-ich.ko 2022-05-13T22:46:51.385 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/gpio/gpio-viperboard.ko 2022-05-13T22:46:51.386 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/hid 2022-05-13T22:46:51.386 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/hid/hid-appleir.ko 2022-05-13T22:46:51.386 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/hid/hid-aureal.ko 2022-05-13T22:46:51.386 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/hid/hid-axff.ko 2022-05-13T22:46:51.387 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/hid/hid-dr.ko 2022-05-13T22:46:51.387 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/hid/hid-elecom.ko 2022-05-13T22:46:51.387 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/hid/hid-elo.ko 2022-05-13T22:46:51.387 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/hid/hid-emsff.ko 2022-05-13T22:46:51.387 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/hid/hid-gaff.ko 2022-05-13T22:46:51.388 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/hid/hid-gt683r.ko 2022-05-13T22:46:51.388 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/hid/hid-gyration.ko 2022-05-13T22:46:51.388 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/hid/hid-holtek-kbd.ko 2022-05-13T22:46:51.388 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/hid/hid-holtek-mouse.ko 2022-05-13T22:46:51.389 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/hid/hid-holtekff.ko 2022-05-13T22:46:51.389 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/hid/hid-hyperv.ko 2022-05-13T22:46:51.389 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/hid/hid-icade.ko 2022-05-13T22:46:51.389 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/hid/hid-keytouch.ko 2022-05-13T22:46:51.389 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/hid/hid-kye.ko 2022-05-13T22:46:51.390 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/hid/hid-lcpower.ko 2022-05-13T22:46:51.390 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/hid/hid-led.ko 2022-05-13T22:46:51.390 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/hid/hid-lenovo.ko 2022-05-13T22:46:51.390 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/hid/hid-logitech-dj.ko 2022-05-13T22:46:51.390 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/hid/hid-logitech-hidpp.ko 2022-05-13T22:46:51.391 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/hid/hid-multitouch.ko 2022-05-13T22:46:51.391 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/hid/hid-ortek.ko 2022-05-13T22:46:51.391 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/hid/hid-penmount.ko 2022-05-13T22:46:51.392 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/hid/hid-petalynx.ko 2022-05-13T22:46:51.392 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/hid/hid-picolcd.ko 2022-05-13T22:46:51.392 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/hid/hid-pl.ko 2022-05-13T22:46:51.392 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/hid/hid-plantronics.ko 2022-05-13T22:46:51.393 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/hid/hid-primax.ko 2022-05-13T22:46:51.393 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/hid/hid-rmi.ko 2022-05-13T22:46:51.393 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/hid/hid-roccat-arvo.ko 2022-05-13T22:46:51.393 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/hid/hid-roccat-common.ko 2022-05-13T22:46:51.393 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/hid/hid-roccat-isku.ko 2022-05-13T22:46:51.394 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/hid/hid-roccat-kone.ko 2022-05-13T22:46:51.394 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/hid/hid-roccat-koneplus.ko 2022-05-13T22:46:51.394 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/hid/hid-roccat-konepure.ko 2022-05-13T22:46:51.394 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/hid/hid-roccat-kovaplus.ko 2022-05-13T22:46:51.395 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/hid/hid-roccat-lua.ko 2022-05-13T22:46:51.395 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/hid/hid-roccat-pyra.ko 2022-05-13T22:46:51.395 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/hid/hid-roccat-ryos.ko 2022-05-13T22:46:51.395 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/hid/hid-roccat-savu.ko 2022-05-13T22:46:51.395 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/hid/hid-roccat.ko 2022-05-13T22:46:51.396 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/hid/hid-saitek.ko 2022-05-13T22:46:51.396 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/hid/hid-samsung.ko 2022-05-13T22:46:51.396 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/hid/hid-sensor-hub.ko 2022-05-13T22:46:51.396 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/hid/hid-sjoy.ko 2022-05-13T22:46:51.396 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/hid/hid-sony.ko 2022-05-13T22:46:51.397 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/hid/hid-speedlink.ko 2022-05-13T22:46:51.397 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/hid/hid-steelseries.ko 2022-05-13T22:46:51.397 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/hid/hid-sunplus.ko 2022-05-13T22:46:51.397 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/hid/hid-thrustmaster.ko 2022-05-13T22:46:51.397 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/hid/hid-tivo.ko 2022-05-13T22:46:51.398 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/hid/hid-tmff.ko 2022-05-13T22:46:51.398 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/hid/hid-topseed.ko 2022-05-13T22:46:51.398 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/hid/hid-twinhan.ko 2022-05-13T22:46:51.398 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/hid/hid-uclogic.ko 2022-05-13T22:46:51.399 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/hid/hid-waltop.ko 2022-05-13T22:46:51.399 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/hid/hid-wiimote.ko 2022-05-13T22:46:51.399 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/hid/hid-xinmo.ko 2022-05-13T22:46:51.399 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/hid/hid-zpff.ko 2022-05-13T22:46:51.399 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/hid/hid-zydacron.ko 2022-05-13T22:46:51.400 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/hid/uhid.ko 2022-05-13T22:46:51.400 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/hid/wacom.ko 2022-05-13T22:46:51.400 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/hv 2022-05-13T22:46:51.401 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/hv/hv_balloon.ko 2022-05-13T22:46:51.401 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/hv/hv_utils.ko 2022-05-13T22:46:51.401 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/hv/hv_vmbus.ko 2022-05-13T22:46:51.401 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/hwmon 2022-05-13T22:46:51.402 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/hwmon/abituguru.ko 2022-05-13T22:46:51.402 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/hwmon/abituguru3.ko 2022-05-13T22:46:51.402 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/hwmon/acpi_power_meter.ko 2022-05-13T22:46:51.402 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/hwmon/ad7414.ko 2022-05-13T22:46:51.402 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/hwmon/ad7418.ko 2022-05-13T22:46:51.403 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/hwmon/adc128d818.ko 2022-05-13T22:46:51.403 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/hwmon/adm1021.ko 2022-05-13T22:46:51.403 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/hwmon/adm1025.ko 2022-05-13T22:46:51.403 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/hwmon/adm1026.ko 2022-05-13T22:46:51.404 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/hwmon/adm1029.ko 2022-05-13T22:46:51.404 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/hwmon/adm1031.ko 2022-05-13T22:46:51.404 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/hwmon/adm9240.ko 2022-05-13T22:46:51.404 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/hwmon/ads7828.ko 2022-05-13T22:46:51.404 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/hwmon/adt7410.ko 2022-05-13T22:46:51.405 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/hwmon/adt7411.ko 2022-05-13T22:46:51.405 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/hwmon/adt7462.ko 2022-05-13T22:46:51.405 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/hwmon/adt7470.ko 2022-05-13T22:46:51.405 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/hwmon/adt7475.ko 2022-05-13T22:46:51.405 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/hwmon/adt7x10.ko 2022-05-13T22:46:51.406 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/hwmon/amc6821.ko 2022-05-13T22:46:51.406 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/hwmon/applesmc.ko 2022-05-13T22:46:51.406 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/hwmon/asb100.ko 2022-05-13T22:46:51.406 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/hwmon/asc7621.ko 2022-05-13T22:46:51.407 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/hwmon/asus_atk0110.ko 2022-05-13T22:46:51.407 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/hwmon/atxp1.ko 2022-05-13T22:46:51.407 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/hwmon/coretemp.ko 2022-05-13T22:46:51.407 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/hwmon/dell-smm-hwmon.ko 2022-05-13T22:46:51.407 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/hwmon/dme1737.ko 2022-05-13T22:46:51.408 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/hwmon/ds1621.ko 2022-05-13T22:46:51.408 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/hwmon/ds620.ko 2022-05-13T22:46:51.408 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/hwmon/emc1403.ko 2022-05-13T22:46:51.408 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/hwmon/emc6w201.ko 2022-05-13T22:46:51.409 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/hwmon/f71805f.ko 2022-05-13T22:46:51.409 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/hwmon/f71882fg.ko 2022-05-13T22:46:51.409 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/hwmon/f75375s.ko 2022-05-13T22:46:51.410 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/hwmon/fam15h_power.ko 2022-05-13T22:46:51.410 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/hwmon/fschmd.ko 2022-05-13T22:46:51.410 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/hwmon/g760a.ko 2022-05-13T22:46:51.410 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/hwmon/g762.ko 2022-05-13T22:46:51.410 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/hwmon/gl518sm.ko 2022-05-13T22:46:51.411 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/hwmon/gl520sm.ko 2022-05-13T22:46:51.411 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/hwmon/hwmon-vid.ko 2022-05-13T22:46:51.411 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/hwmon/i5500_temp.ko 2022-05-13T22:46:51.411 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/hwmon/i5k_amb.ko 2022-05-13T22:46:51.411 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/hwmon/ibmaem.ko 2022-05-13T22:46:51.412 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/hwmon/ibmpex.ko 2022-05-13T22:46:51.412 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/hwmon/ina209.ko 2022-05-13T22:46:51.412 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/hwmon/ina2xx.ko 2022-05-13T22:46:51.412 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/hwmon/it87.ko 2022-05-13T22:46:51.412 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/hwmon/k10temp.ko 2022-05-13T22:46:51.413 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/hwmon/k8temp.ko 2022-05-13T22:46:51.413 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/hwmon/lineage-pem.ko 2022-05-13T22:46:51.413 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/hwmon/lm63.ko 2022-05-13T22:46:51.413 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/hwmon/lm73.ko 2022-05-13T22:46:51.414 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/hwmon/lm75.ko 2022-05-13T22:46:51.414 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/hwmon/lm77.ko 2022-05-13T22:46:51.414 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/hwmon/lm78.ko 2022-05-13T22:46:51.414 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/hwmon/lm80.ko 2022-05-13T22:46:51.414 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/hwmon/lm83.ko 2022-05-13T22:46:51.415 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/hwmon/lm85.ko 2022-05-13T22:46:51.415 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/hwmon/lm87.ko 2022-05-13T22:46:51.415 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/hwmon/lm90.ko 2022-05-13T22:46:51.415 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/hwmon/lm92.ko 2022-05-13T22:46:51.415 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/hwmon/lm93.ko 2022-05-13T22:46:51.416 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/hwmon/lm95234.ko 2022-05-13T22:46:51.416 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/hwmon/lm95241.ko 2022-05-13T22:46:51.416 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/hwmon/lm95245.ko 2022-05-13T22:46:51.416 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/hwmon/ltc2945.ko 2022-05-13T22:46:51.417 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/hwmon/ltc4151.ko 2022-05-13T22:46:51.417 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/hwmon/ltc4215.ko 2022-05-13T22:46:51.417 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/hwmon/ltc4222.ko 2022-05-13T22:46:51.417 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/hwmon/ltc4245.ko 2022-05-13T22:46:51.417 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/hwmon/ltc4260.ko 2022-05-13T22:46:51.418 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/hwmon/ltc4261.ko 2022-05-13T22:46:51.418 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/hwmon/max16065.ko 2022-05-13T22:46:51.418 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/hwmon/max1619.ko 2022-05-13T22:46:51.418 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/hwmon/max1668.ko 2022-05-13T22:46:51.419 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/hwmon/max197.ko 2022-05-13T22:46:51.419 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/hwmon/max6639.ko 2022-05-13T22:46:51.419 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/hwmon/max6642.ko 2022-05-13T22:46:51.419 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/hwmon/max6650.ko 2022-05-13T22:46:51.419 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/hwmon/max6697.ko 2022-05-13T22:46:51.420 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/hwmon/mcp3021.ko 2022-05-13T22:46:51.420 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/hwmon/nct6683.ko 2022-05-13T22:46:51.420 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/hwmon/nct6775.ko 2022-05-13T22:46:51.420 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/hwmon/nct7802.ko 2022-05-13T22:46:51.420 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/hwmon/pc87360.ko 2022-05-13T22:46:51.421 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/hwmon/pc87427.ko 2022-05-13T22:46:51.421 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/hwmon/pcf8591.ko 2022-05-13T22:46:51.421 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/hwmon/pmbus 2022-05-13T22:46:51.421 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/hwmon/pmbus/adm1275.ko 2022-05-13T22:46:51.422 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/hwmon/pmbus/lm25066.ko 2022-05-13T22:46:51.422 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/hwmon/pmbus/ltc2978.ko 2022-05-13T22:46:51.422 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/hwmon/pmbus/max16064.ko 2022-05-13T22:46:51.422 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/hwmon/pmbus/max34440.ko 2022-05-13T22:46:51.422 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/hwmon/pmbus/max8688.ko 2022-05-13T22:46:51.423 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/hwmon/pmbus/pmbus.ko 2022-05-13T22:46:51.423 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/hwmon/pmbus/pmbus_core.ko 2022-05-13T22:46:51.423 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/hwmon/pmbus/tps40422.ko 2022-05-13T22:46:51.423 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/hwmon/pmbus/ucd9000.ko 2022-05-13T22:46:51.423 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/hwmon/pmbus/ucd9200.ko 2022-05-13T22:46:51.424 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/hwmon/pmbus/zl6100.ko 2022-05-13T22:46:51.424 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/hwmon/powr1220.ko 2022-05-13T22:46:51.424 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/hwmon/sch5627.ko 2022-05-13T22:46:51.424 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/hwmon/sch5636.ko 2022-05-13T22:46:51.424 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/hwmon/sch56xx-common.ko 2022-05-13T22:46:51.425 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/hwmon/sht15.ko 2022-05-13T22:46:51.425 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/hwmon/sht21.ko 2022-05-13T22:46:51.425 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/hwmon/shtc1.ko 2022-05-13T22:46:51.425 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/hwmon/sis5595.ko 2022-05-13T22:46:51.425 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/hwmon/smsc47b397.ko 2022-05-13T22:46:51.426 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/hwmon/smsc47m1.ko 2022-05-13T22:46:51.426 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/hwmon/smsc47m192.ko 2022-05-13T22:46:51.426 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/hwmon/thmc50.ko 2022-05-13T22:46:51.426 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/hwmon/tmp102.ko 2022-05-13T22:46:51.426 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/hwmon/tmp103.ko 2022-05-13T22:46:51.427 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/hwmon/tmp401.ko 2022-05-13T22:46:51.427 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/hwmon/tmp421.ko 2022-05-13T22:46:51.427 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/hwmon/via-cputemp.ko 2022-05-13T22:46:51.427 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/hwmon/via686a.ko 2022-05-13T22:46:51.428 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/hwmon/vt1211.ko 2022-05-13T22:46:51.428 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/hwmon/vt8231.ko 2022-05-13T22:46:51.428 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/hwmon/w83627ehf.ko 2022-05-13T22:46:51.428 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/hwmon/w83627hf.ko 2022-05-13T22:46:51.428 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/hwmon/w83781d.ko 2022-05-13T22:46:51.429 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/hwmon/w83791d.ko 2022-05-13T22:46:51.429 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/hwmon/w83792d.ko 2022-05-13T22:46:51.429 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/hwmon/w83793.ko 2022-05-13T22:46:51.429 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/hwmon/w83795.ko 2022-05-13T22:46:51.429 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/hwmon/w83l785ts.ko 2022-05-13T22:46:51.430 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/hwmon/w83l786ng.ko 2022-05-13T22:46:51.430 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/i2c 2022-05-13T22:46:51.430 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/i2c/algos 2022-05-13T22:46:51.430 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/i2c/algos/i2c-algo-bit.ko 2022-05-13T22:46:51.430 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/i2c/algos/i2c-algo-pca.ko 2022-05-13T22:46:51.431 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/i2c/busses 2022-05-13T22:46:51.431 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/i2c/busses/i2c-amd756-s4882.ko 2022-05-13T22:46:51.431 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/i2c/busses/i2c-amd756.ko 2022-05-13T22:46:51.431 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/i2c/busses/i2c-amd8111.ko 2022-05-13T22:46:51.432 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/i2c/busses/i2c-ccgx-ucsi.ko 2022-05-13T22:46:51.432 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/i2c/busses/i2c-designware-core.ko 2022-05-13T22:46:51.432 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/i2c/busses/i2c-designware-pci.ko 2022-05-13T22:46:51.432 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/i2c/busses/i2c-designware-platform.ko 2022-05-13T22:46:51.432 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/i2c/busses/i2c-diolan-u2c.ko 2022-05-13T22:46:51.433 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/i2c/busses/i2c-i801.ko 2022-05-13T22:46:51.433 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/i2c/busses/i2c-isch.ko 2022-05-13T22:46:51.433 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/i2c/busses/i2c-ismt.ko 2022-05-13T22:46:51.433 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/i2c/busses/i2c-nforce2-s4985.ko 2022-05-13T22:46:51.433 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/i2c/busses/i2c-nforce2.ko 2022-05-13T22:46:51.434 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/i2c/busses/i2c-parport.ko 2022-05-13T22:46:51.434 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/i2c/busses/i2c-pca-platform.ko 2022-05-13T22:46:51.434 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/i2c/busses/i2c-piix4.ko 2022-05-13T22:46:51.434 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/i2c/busses/i2c-scmi.ko 2022-05-13T22:46:51.434 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/i2c/busses/i2c-simtec.ko 2022-05-13T22:46:51.435 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/i2c/busses/i2c-sis96x.ko 2022-05-13T22:46:51.435 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/i2c/busses/i2c-tiny-usb.ko 2022-05-13T22:46:51.435 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/i2c/busses/i2c-via.ko 2022-05-13T22:46:51.435 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/i2c/busses/i2c-viapro.ko 2022-05-13T22:46:51.436 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/i2c/busses/i2c-viperboard.ko 2022-05-13T22:46:51.436 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/i2c/i2c-dev.ko 2022-05-13T22:46:51.436 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/i2c/i2c-mux.ko 2022-05-13T22:46:51.436 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/i2c/i2c-slave-eeprom.ko 2022-05-13T22:46:51.436 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/i2c/i2c-smbus.ko 2022-05-13T22:46:51.437 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/i2c/i2c-stub.ko 2022-05-13T22:46:51.437 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/infiniband 2022-05-13T22:46:51.437 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/infiniband/core 2022-05-13T22:46:51.437 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/infiniband/core/ib_cm.ko 2022-05-13T22:46:51.437 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/infiniband/core/ib_core.ko 2022-05-13T22:46:51.438 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/infiniband/core/ib_umad.ko 2022-05-13T22:46:51.438 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/infiniband/core/ib_uverbs.ko 2022-05-13T22:46:51.438 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/infiniband/core/iw_cm.ko 2022-05-13T22:46:51.438 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/infiniband/core/rdma_cm.ko 2022-05-13T22:46:51.438 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/infiniband/core/rdma_ucm.ko 2022-05-13T22:46:51.439 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/infiniband/hw 2022-05-13T22:46:51.439 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/infiniband/hw/cxgb4 2022-05-13T22:46:51.439 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/infiniband/hw/cxgb4/iw_cxgb4.ko 2022-05-13T22:46:51.439 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/infiniband/hw/mlx4 2022-05-13T22:46:51.440 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/infiniband/hw/mlx4/mlx4_ib.ko 2022-05-13T22:46:51.440 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/infiniband/hw/mlx5 2022-05-13T22:46:51.440 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/infiniband/hw/mlx5/mlx5_ib.ko 2022-05-13T22:46:51.440 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/infiniband/hw/mthca 2022-05-13T22:46:51.440 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/infiniband/hw/mthca/ib_mthca.ko 2022-05-13T22:46:51.441 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/infiniband/ulp 2022-05-13T22:46:51.441 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/infiniband/ulp/ipoib 2022-05-13T22:46:51.441 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/infiniband/ulp/ipoib/ib_ipoib.ko 2022-05-13T22:46:51.441 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/infiniband/ulp/iser 2022-05-13T22:46:51.441 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/infiniband/ulp/iser/ib_iser.ko 2022-05-13T22:46:51.442 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/infiniband/ulp/isert 2022-05-13T22:46:51.442 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/infiniband/ulp/isert/ib_isert.ko 2022-05-13T22:46:51.442 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/infiniband/ulp/srp 2022-05-13T22:46:51.442 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/infiniband/ulp/srp/ib_srp.ko 2022-05-13T22:46:51.442 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/infiniband/ulp/srpt 2022-05-13T22:46:51.443 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/infiniband/ulp/srpt/ib_srpt.ko 2022-05-13T22:46:51.443 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/input 2022-05-13T22:46:51.443 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/input/gameport 2022-05-13T22:46:51.443 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/input/gameport/emu10k1-gp.ko 2022-05-13T22:46:51.443 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/input/gameport/fm801-gp.ko 2022-05-13T22:46:51.444 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/input/gameport/gameport.ko 2022-05-13T22:46:51.444 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/input/gameport/lightning.ko 2022-05-13T22:46:51.444 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/input/gameport/ns558.ko 2022-05-13T22:46:51.445 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/input/joydev.ko 2022-05-13T22:46:51.445 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/input/keyboard 2022-05-13T22:46:51.445 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/input/keyboard/gpio_keys.ko 2022-05-13T22:46:51.445 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/input/mouse 2022-05-13T22:46:51.445 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/input/mouse/appletouch.ko 2022-05-13T22:46:51.446 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/input/mouse/bcm5974.ko 2022-05-13T22:46:51.446 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/input/mouse/cyapatp.ko 2022-05-13T22:46:51.446 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/input/mouse/elan_i2c.ko 2022-05-13T22:46:51.446 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/input/mouse/sermouse.ko 2022-05-13T22:46:51.447 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/input/mouse/synaptics_i2c.ko 2022-05-13T22:46:51.447 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/input/mouse/synaptics_usb.ko 2022-05-13T22:46:51.447 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/input/mouse/vsxxxaa.ko 2022-05-13T22:46:51.447 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/input/rmi4 2022-05-13T22:46:51.447 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/input/rmi4/rmi_core.ko 2022-05-13T22:46:51.448 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/input/serio 2022-05-13T22:46:51.448 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/input/serio/altera_ps2.ko 2022-05-13T22:46:51.448 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/input/serio/arc_ps2.ko 2022-05-13T22:46:51.448 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/input/serio/hyperv-keyboard.ko 2022-05-13T22:46:51.448 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/input/serio/serio_raw.ko 2022-05-13T22:46:51.449 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/input/sparse-keymap.ko 2022-05-13T22:46:51.449 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/iommu 2022-05-13T22:46:51.449 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/iommu/amd 2022-05-13T22:46:51.449 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/iommu/amd/iommu_v2.ko 2022-05-13T22:46:51.450 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/leds 2022-05-13T22:46:51.450 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/leds/leds-blinkm.ko 2022-05-13T22:46:51.450 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/leds/leds-clevo-mail.ko 2022-05-13T22:46:51.450 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/leds/leds-lm3530.ko 2022-05-13T22:46:51.450 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/leds/leds-lp3944.ko 2022-05-13T22:46:51.451 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/leds/leds-ss4200.ko 2022-05-13T22:46:51.451 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/leds/trigger 2022-05-13T22:46:51.451 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/leds/trigger/ledtrig-audio.ko 2022-05-13T22:46:51.451 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/leds/trigger/ledtrig-backlight.ko 2022-05-13T22:46:51.451 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/leds/trigger/ledtrig-camera.ko 2022-05-13T22:46:51.452 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/leds/trigger/ledtrig-default-on.ko 2022-05-13T22:46:51.452 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/leds/trigger/ledtrig-gpio.ko 2022-05-13T22:46:51.452 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/leds/trigger/ledtrig-heartbeat.ko 2022-05-13T22:46:51.453 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/leds/trigger/ledtrig-oneshot.ko 2022-05-13T22:46:51.453 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/leds/trigger/ledtrig-timer.ko 2022-05-13T22:46:51.453 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/leds/trigger/ledtrig-transient.ko 2022-05-13T22:46:51.453 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/md 2022-05-13T22:46:51.454 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/md/bcache 2022-05-13T22:46:51.454 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/md/bcache/bcache.ko 2022-05-13T22:46:51.454 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/md/dm-bio-prison.ko 2022-05-13T22:46:51.454 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/md/dm-cache-smq.ko 2022-05-13T22:46:51.454 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/md/dm-cache.ko 2022-05-13T22:46:51.455 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/md/dm-crypt.ko 2022-05-13T22:46:51.455 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/md/dm-delay.ko 2022-05-13T22:46:51.455 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/md/dm-flakey.ko 2022-05-13T22:46:51.455 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/md/dm-log-userspace.ko 2022-05-13T22:46:51.456 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/md/dm-multipath.ko 2022-05-13T22:46:51.456 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/md/dm-queue-length.ko 2022-05-13T22:46:51.456 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/md/dm-raid.ko 2022-05-13T22:46:51.456 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/md/dm-round-robin.ko 2022-05-13T22:46:51.456 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/md/dm-service-time.ko 2022-05-13T22:46:51.457 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/md/dm-switch.ko 2022-05-13T22:46:51.457 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/md/dm-thin-pool.ko 2022-05-13T22:46:51.457 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/md/dm-verity.ko 2022-05-13T22:46:51.457 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/md/faulty.ko 2022-05-13T22:46:51.457 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/md/linear.ko 2022-05-13T22:46:51.458 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/md/multipath.ko 2022-05-13T22:46:51.458 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/md/persistent-data 2022-05-13T22:46:51.458 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/md/persistent-data/dm-persistent-data.ko 2022-05-13T22:46:51.458 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/md/raid0.ko 2022-05-13T22:46:51.458 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/md/raid1.ko 2022-05-13T22:46:51.459 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/md/raid10.ko 2022-05-13T22:46:51.459 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/md/raid456.ko 2022-05-13T22:46:51.459 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/message 2022-05-13T22:46:51.459 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/message/fusion 2022-05-13T22:46:51.459 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/message/fusion/mptbase.ko 2022-05-13T22:46:51.460 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/message/fusion/mptctl.ko 2022-05-13T22:46:51.460 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/message/fusion/mptfc.ko 2022-05-13T22:46:51.460 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/message/fusion/mptlan.ko 2022-05-13T22:46:51.460 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/message/fusion/mptsas.ko 2022-05-13T22:46:51.460 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/message/fusion/mptscsih.ko 2022-05-13T22:46:51.461 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/message/fusion/mptspi.ko 2022-05-13T22:46:51.461 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/mfd 2022-05-13T22:46:51.461 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/mfd/lpc_ich.ko 2022-05-13T22:46:51.462 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/mfd/lpc_sch.ko 2022-05-13T22:46:51.462 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/mfd/mfd-core.ko 2022-05-13T22:46:51.462 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/mfd/sm501.ko 2022-05-13T22:46:51.462 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/mfd/viperboard.ko 2022-05-13T22:46:51.463 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/mfd/vx855.ko 2022-05-13T22:46:51.463 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/mfd/wl1273-core.ko 2022-05-13T22:46:51.463 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/misc 2022-05-13T22:46:51.463 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/misc/altera-stapl 2022-05-13T22:46:51.463 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/misc/altera-stapl/altera-stapl.ko 2022-05-13T22:46:51.464 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/misc/apds9802als.ko 2022-05-13T22:46:51.464 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/misc/apds990x.ko 2022-05-13T22:46:51.464 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/misc/bh1770glc.ko 2022-05-13T22:46:51.464 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/misc/cb710 2022-05-13T22:46:51.464 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/misc/cb710/cb710.ko 2022-05-13T22:46:51.465 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/misc/eeprom 2022-05-13T22:46:51.465 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/misc/eeprom/at24.ko 2022-05-13T22:46:51.465 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/misc/eeprom/eeprom.ko 2022-05-13T22:46:51.465 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/misc/eeprom/eeprom_93cx6.ko 2022-05-13T22:46:51.466 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/misc/eeprom/max6875.ko 2022-05-13T22:46:51.466 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/misc/enclosure.ko 2022-05-13T22:46:51.466 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/misc/hpilo.ko 2022-05-13T22:46:51.466 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/misc/isl29003.ko 2022-05-13T22:46:51.466 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/misc/isl29020.ko 2022-05-13T22:46:51.467 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/misc/lis3lv02d 2022-05-13T22:46:51.467 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/misc/lis3lv02d/lis3lv02d.ko 2022-05-13T22:46:51.467 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/misc/lis3lv02d/lis3lv02d_i2c.ko 2022-05-13T22:46:51.467 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/misc/mei 2022-05-13T22:46:51.468 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/misc/mei/mei-me.ko 2022-05-13T22:46:51.468 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/misc/mei/mei-txe.ko 2022-05-13T22:46:51.468 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/misc/mei/mei.ko 2022-05-13T22:46:51.468 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/misc/sgi-gru 2022-05-13T22:46:51.468 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/misc/sgi-gru/gru.ko 2022-05-13T22:46:51.469 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/misc/sgi-xp 2022-05-13T22:46:51.469 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/misc/sgi-xp/xp.ko 2022-05-13T22:46:51.469 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/misc/sgi-xp/xpc.ko 2022-05-13T22:46:51.469 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/misc/sgi-xp/xpnet.ko 2022-05-13T22:46:51.470 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/misc/tifm_7xx1.ko 2022-05-13T22:46:51.470 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/misc/tifm_core.ko 2022-05-13T22:46:51.470 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/misc/tsl2550.ko 2022-05-13T22:46:51.470 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/misc/vmw_balloon.ko 2022-05-13T22:46:51.471 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/misc/vmw_vmci 2022-05-13T22:46:51.471 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/misc/vmw_vmci/vmw_vmci.ko 2022-05-13T22:46:51.471 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/net 2022-05-13T22:46:51.471 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/net/bonding 2022-05-13T22:46:51.471 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/net/bonding/bonding.ko 2022-05-13T22:46:51.472 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/net/dsa 2022-05-13T22:46:51.472 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/net/dsa/b53 2022-05-13T22:46:51.472 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/net/dsa/b53/b53_common.ko 2022-05-13T22:46:51.472 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/net/dsa/bcm-sf2.ko 2022-05-13T22:46:51.473 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/net/dsa/mv88e6060.ko 2022-05-13T22:46:51.473 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/net/dsa/mv88e6xxx 2022-05-13T22:46:51.473 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/net/dsa/mv88e6xxx/mv88e6xxx.ko 2022-05-13T22:46:51.473 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/net/dummy.ko 2022-05-13T22:46:51.473 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/net/eql.ko 2022-05-13T22:46:51.474 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/net/ethernet 2022-05-13T22:46:51.474 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/net/ethernet/3com 2022-05-13T22:46:51.474 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/net/ethernet/3com/3c574_cs.ko 2022-05-13T22:46:51.474 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/net/ethernet/3com/3c589_cs.ko 2022-05-13T22:46:51.474 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/net/ethernet/3com/3c59x.ko 2022-05-13T22:46:51.475 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/net/ethernet/3com/typhoon.ko 2022-05-13T22:46:51.475 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/net/ethernet/8390 2022-05-13T22:46:51.475 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/net/ethernet/8390/8390.ko 2022-05-13T22:46:51.475 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/net/ethernet/8390/axnet_cs.ko 2022-05-13T22:46:51.476 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/net/ethernet/8390/ne2k-pci.ko 2022-05-13T22:46:51.476 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/net/ethernet/8390/pcnet_cs.ko 2022-05-13T22:46:51.476 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/net/ethernet/adaptec 2022-05-13T22:46:51.476 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/net/ethernet/adaptec/starfire.ko 2022-05-13T22:46:51.476 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/net/ethernet/agere 2022-05-13T22:46:51.477 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/net/ethernet/agere/et131x.ko 2022-05-13T22:46:51.477 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/net/ethernet/alteon 2022-05-13T22:46:51.477 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/net/ethernet/alteon/acenic.ko 2022-05-13T22:46:51.477 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/net/ethernet/altera 2022-05-13T22:46:51.477 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/net/ethernet/altera/altera_tse.ko 2022-05-13T22:46:51.478 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/net/ethernet/amd 2022-05-13T22:46:51.478 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/net/ethernet/amd/amd8111e.ko 2022-05-13T22:46:51.478 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/net/ethernet/amd/nmclan_cs.ko 2022-05-13T22:46:51.478 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/net/ethernet/amd/pcnet32.ko 2022-05-13T22:46:51.479 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/net/ethernet/atheros 2022-05-13T22:46:51.479 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/net/ethernet/atheros/alx 2022-05-13T22:46:51.479 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/net/ethernet/atheros/alx/alx.ko 2022-05-13T22:46:51.479 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/net/ethernet/atheros/atl1c 2022-05-13T22:46:51.480 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/net/ethernet/atheros/atl1c/atl1c.ko 2022-05-13T22:46:51.480 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/net/ethernet/atheros/atl1e 2022-05-13T22:46:51.480 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/net/ethernet/atheros/atl1e/atl1e.ko 2022-05-13T22:46:51.480 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/net/ethernet/atheros/atlx 2022-05-13T22:46:51.480 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/net/ethernet/atheros/atlx/atl1.ko 2022-05-13T22:46:51.481 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/net/ethernet/atheros/atlx/atl2.ko 2022-05-13T22:46:51.481 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/net/ethernet/broadcom 2022-05-13T22:46:51.481 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/net/ethernet/broadcom/b44.ko 2022-05-13T22:46:51.481 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/net/ethernet/broadcom/bnx2.ko 2022-05-13T22:46:51.482 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/net/ethernet/broadcom/bnx2x 2022-05-13T22:46:51.482 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/net/ethernet/broadcom/bnx2x/bnx2x.ko 2022-05-13T22:46:51.482 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/net/ethernet/broadcom/cnic.ko 2022-05-13T22:46:51.482 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/net/ethernet/broadcom/genet 2022-05-13T22:46:51.482 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/net/ethernet/broadcom/genet/genet.ko 2022-05-13T22:46:51.483 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/net/ethernet/broadcom/tg3.ko 2022-05-13T22:46:51.483 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/net/ethernet/brocade 2022-05-13T22:46:51.483 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/net/ethernet/brocade/bna 2022-05-13T22:46:51.483 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/net/ethernet/brocade/bna/bna.ko 2022-05-13T22:46:51.483 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/net/ethernet/chelsio 2022-05-13T22:46:51.484 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/net/ethernet/chelsio/cxgb 2022-05-13T22:46:51.484 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/net/ethernet/chelsio/cxgb/cxgb.ko 2022-05-13T22:46:51.484 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/net/ethernet/chelsio/cxgb3 2022-05-13T22:46:51.484 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/net/ethernet/chelsio/cxgb3/cxgb3.ko 2022-05-13T22:46:51.484 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/net/ethernet/chelsio/cxgb4 2022-05-13T22:46:51.485 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/net/ethernet/chelsio/cxgb4/cxgb4.ko 2022-05-13T22:46:51.485 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/net/ethernet/chelsio/cxgb4vf 2022-05-13T22:46:51.485 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/net/ethernet/chelsio/cxgb4vf/cxgb4vf.ko 2022-05-13T22:46:51.485 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/net/ethernet/chelsio/libcxgb 2022-05-13T22:46:51.485 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/net/ethernet/chelsio/libcxgb/libcxgb.ko 2022-05-13T22:46:51.486 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/net/ethernet/cisco 2022-05-13T22:46:51.486 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/net/ethernet/cisco/enic 2022-05-13T22:46:51.486 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/net/ethernet/cisco/enic/enic.ko 2022-05-13T22:46:51.486 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/net/ethernet/dec 2022-05-13T22:46:51.487 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/net/ethernet/dec/tulip 2022-05-13T22:46:51.487 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/net/ethernet/dec/tulip/de2104x.ko 2022-05-13T22:46:51.487 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/net/ethernet/dec/tulip/de4x5.ko 2022-05-13T22:46:51.487 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/net/ethernet/dec/tulip/dmfe.ko 2022-05-13T22:46:51.488 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/net/ethernet/dec/tulip/tulip.ko 2022-05-13T22:46:51.488 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/net/ethernet/dec/tulip/uli526x.ko 2022-05-13T22:46:51.488 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/net/ethernet/dec/tulip/winbond-840.ko 2022-05-13T22:46:51.488 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/net/ethernet/dec/tulip/xircom_cb.ko 2022-05-13T22:46:51.488 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/net/ethernet/dlink 2022-05-13T22:46:51.489 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/net/ethernet/dlink/dl2k.ko 2022-05-13T22:46:51.489 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/net/ethernet/dlink/sundance.ko 2022-05-13T22:46:51.489 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/net/ethernet/dnet.ko 2022-05-13T22:46:51.489 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/net/ethernet/emulex 2022-05-13T22:46:51.490 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/net/ethernet/emulex/benet 2022-05-13T22:46:51.490 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/net/ethernet/emulex/benet/be2net.ko 2022-05-13T22:46:51.490 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/net/ethernet/ethoc.ko 2022-05-13T22:46:51.490 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/net/ethernet/fealnx.ko 2022-05-13T22:46:51.490 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/net/ethernet/intel 2022-05-13T22:46:51.491 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/net/ethernet/intel/e100.ko 2022-05-13T22:46:51.491 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/net/ethernet/intel/e1000 2022-05-13T22:46:51.491 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/net/ethernet/intel/e1000/e1000.ko 2022-05-13T22:46:51.491 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/net/ethernet/intel/e1000e 2022-05-13T22:46:51.491 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/net/ethernet/intel/e1000e/e1000e.ko 2022-05-13T22:46:51.492 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/net/ethernet/intel/fm10k 2022-05-13T22:46:51.492 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/net/ethernet/intel/fm10k/fm10k.ko 2022-05-13T22:46:51.492 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/net/ethernet/intel/i40e 2022-05-13T22:46:51.492 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/net/ethernet/intel/i40e/i40e.ko 2022-05-13T22:46:51.492 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/net/ethernet/intel/iavf 2022-05-13T22:46:51.493 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/net/ethernet/intel/iavf/iavf.ko 2022-05-13T22:46:51.493 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/net/ethernet/intel/igb 2022-05-13T22:46:51.493 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/net/ethernet/intel/igb/igb.ko 2022-05-13T22:46:51.493 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/net/ethernet/intel/igbvf 2022-05-13T22:46:51.494 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/net/ethernet/intel/igbvf/igbvf.ko 2022-05-13T22:46:51.494 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/net/ethernet/intel/ixgb 2022-05-13T22:46:51.494 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/net/ethernet/intel/ixgb/ixgb.ko 2022-05-13T22:46:51.494 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/net/ethernet/intel/ixgbe 2022-05-13T22:46:51.494 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/net/ethernet/intel/ixgbe/ixgbe.ko 2022-05-13T22:46:51.495 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/net/ethernet/intel/ixgbevf 2022-05-13T22:46:51.495 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/net/ethernet/intel/ixgbevf/ixgbevf.ko 2022-05-13T22:46:51.495 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/net/ethernet/jme.ko 2022-05-13T22:46:51.495 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/net/ethernet/marvell 2022-05-13T22:46:51.495 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/net/ethernet/marvell/mvmdio.ko 2022-05-13T22:46:51.496 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/net/ethernet/marvell/skge.ko 2022-05-13T22:46:51.496 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/net/ethernet/marvell/sky2.ko 2022-05-13T22:46:51.496 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/net/ethernet/mellanox 2022-05-13T22:46:51.496 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/net/ethernet/mellanox/mlx4 2022-05-13T22:46:51.496 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_core.ko 2022-05-13T22:46:51.497 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_en.ko 2022-05-13T22:46:51.497 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/net/ethernet/mellanox/mlx5 2022-05-13T22:46:51.497 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/net/ethernet/mellanox/mlx5/core 2022-05-13T22:46:51.497 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/net/ethernet/mellanox/mlx5/core/mlx5_core.ko 2022-05-13T22:46:51.497 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/net/ethernet/micrel 2022-05-13T22:46:51.498 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/net/ethernet/micrel/ksz884x.ko 2022-05-13T22:46:51.498 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/net/ethernet/myricom 2022-05-13T22:46:51.498 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/net/ethernet/myricom/myri10ge 2022-05-13T22:46:51.498 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/net/ethernet/myricom/myri10ge/myri10ge.ko 2022-05-13T22:46:51.498 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/net/ethernet/natsemi 2022-05-13T22:46:51.499 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/net/ethernet/natsemi/natsemi.ko 2022-05-13T22:46:51.499 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/net/ethernet/natsemi/ns83820.ko 2022-05-13T22:46:51.499 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/net/ethernet/neterion 2022-05-13T22:46:51.499 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/net/ethernet/neterion/s2io.ko 2022-05-13T22:46:51.500 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/net/ethernet/neterion/vxge 2022-05-13T22:46:51.500 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/net/ethernet/neterion/vxge/vxge.ko 2022-05-13T22:46:51.500 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/net/ethernet/nvidia 2022-05-13T22:46:51.500 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/net/ethernet/nvidia/forcedeth.ko 2022-05-13T22:46:51.500 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/net/ethernet/packetengines 2022-05-13T22:46:51.501 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/net/ethernet/packetengines/hamachi.ko 2022-05-13T22:46:51.501 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/net/ethernet/packetengines/yellowfin.ko 2022-05-13T22:46:51.501 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/net/ethernet/qlogic 2022-05-13T22:46:51.501 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/net/ethernet/qlogic/netxen 2022-05-13T22:46:51.501 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/net/ethernet/qlogic/netxen/netxen_nic.ko 2022-05-13T22:46:51.502 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/net/ethernet/qlogic/qla3xxx.ko 2022-05-13T22:46:51.502 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/net/ethernet/qlogic/qlcnic 2022-05-13T22:46:51.502 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/net/ethernet/qlogic/qlcnic/qlcnic.ko 2022-05-13T22:46:51.502 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/net/ethernet/rdc 2022-05-13T22:46:51.502 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/net/ethernet/rdc/r6040.ko 2022-05-13T22:46:51.503 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/net/ethernet/realtek 2022-05-13T22:46:51.503 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/net/ethernet/realtek/8139cp.ko 2022-05-13T22:46:51.503 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/net/ethernet/realtek/8139too.ko 2022-05-13T22:46:51.503 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/net/ethernet/realtek/atp.ko 2022-05-13T22:46:51.503 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/net/ethernet/realtek/r8169.ko 2022-05-13T22:46:51.504 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/net/ethernet/rocker 2022-05-13T22:46:51.504 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/net/ethernet/rocker/rocker.ko 2022-05-13T22:46:51.504 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/net/ethernet/sfc 2022-05-13T22:46:51.504 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/net/ethernet/sfc/sfc.ko 2022-05-13T22:46:51.505 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/net/ethernet/silan 2022-05-13T22:46:51.505 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/net/ethernet/silan/sc92031.ko 2022-05-13T22:46:51.505 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/net/ethernet/sis 2022-05-13T22:46:51.505 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/net/ethernet/sis/sis190.ko 2022-05-13T22:46:51.505 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/net/ethernet/sis/sis900.ko 2022-05-13T22:46:51.506 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/net/ethernet/smsc 2022-05-13T22:46:51.506 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/net/ethernet/smsc/epic100.ko 2022-05-13T22:46:51.506 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/net/ethernet/smsc/smc91c92_cs.ko 2022-05-13T22:46:51.506 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/net/ethernet/smsc/smsc911x.ko 2022-05-13T22:46:51.506 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/net/ethernet/smsc/smsc9420.ko 2022-05-13T22:46:51.507 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/net/ethernet/stmicro 2022-05-13T22:46:51.507 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/net/ethernet/stmicro/stmmac 2022-05-13T22:46:51.507 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/net/ethernet/stmicro/stmmac/dwmac-intel.ko 2022-05-13T22:46:51.507 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/net/ethernet/stmicro/stmmac/stmmac.ko 2022-05-13T22:46:51.507 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/net/ethernet/sun 2022-05-13T22:46:51.508 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/net/ethernet/sun/cassini.ko 2022-05-13T22:46:51.508 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/net/ethernet/sun/niu.ko 2022-05-13T22:46:51.508 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/net/ethernet/sun/sungem.ko 2022-05-13T22:46:51.508 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/net/ethernet/sun/sunhme.ko 2022-05-13T22:46:51.508 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/net/ethernet/tehuti 2022-05-13T22:46:51.509 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/net/ethernet/tehuti/tehuti.ko 2022-05-13T22:46:51.509 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/net/ethernet/ti 2022-05-13T22:46:51.509 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/net/ethernet/ti/tlan.ko 2022-05-13T22:46:51.509 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/net/ethernet/via 2022-05-13T22:46:51.510 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/net/ethernet/via/via-rhine.ko 2022-05-13T22:46:51.510 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/net/ethernet/via/via-velocity.ko 2022-05-13T22:46:51.510 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/net/ethernet/wiznet 2022-05-13T22:46:51.510 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/net/ethernet/wiznet/w5100.ko 2022-05-13T22:46:51.511 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/net/ethernet/wiznet/w5300.ko 2022-05-13T22:46:51.511 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/net/ethernet/xircom 2022-05-13T22:46:51.511 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/net/ethernet/xircom/xirc2ps_cs.ko 2022-05-13T22:46:51.511 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/net/geneve.ko 2022-05-13T22:46:51.512 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/net/hyperv 2022-05-13T22:46:51.512 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/net/hyperv/hv_netvsc.ko 2022-05-13T22:46:51.512 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/net/ieee802154 2022-05-13T22:46:51.512 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/net/ieee802154/fakelb.ko 2022-05-13T22:46:51.512 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/net/ifb.ko 2022-05-13T22:46:51.513 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/net/ipvlan 2022-05-13T22:46:51.513 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/net/ipvlan/ipvlan.ko 2022-05-13T22:46:51.513 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/net/macvlan.ko 2022-05-13T22:46:51.514 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/net/macvtap.ko 2022-05-13T22:46:51.514 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/net/mdio 2022-05-13T22:46:51.514 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/net/mdio.ko 2022-05-13T22:46:51.514 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/net/mdio/mdio-bcm-unimac.ko 2022-05-13T22:46:51.515 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/net/mdio/mdio-bitbang.ko 2022-05-13T22:46:51.515 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/net/mii.ko 2022-05-13T22:46:51.515 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/net/net_failover.ko 2022-05-13T22:46:51.515 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/net/netconsole.ko 2022-05-13T22:46:51.516 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/net/nlmon.ko 2022-05-13T22:46:51.516 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/net/pcs 2022-05-13T22:46:51.516 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/net/pcs/pcs_xpcs.ko 2022-05-13T22:46:51.516 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/net/phy 2022-05-13T22:46:51.516 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/net/phy/amd.ko 2022-05-13T22:46:51.517 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/net/phy/bcm-phy-lib.ko 2022-05-13T22:46:51.517 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/net/phy/bcm7xxx.ko 2022-05-13T22:46:51.517 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/net/phy/bcm87xx.ko 2022-05-13T22:46:51.517 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/net/phy/broadcom.ko 2022-05-13T22:46:51.517 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/net/phy/cicada.ko 2022-05-13T22:46:51.518 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/net/phy/davicom.ko 2022-05-13T22:46:51.518 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/net/phy/dp83640.ko 2022-05-13T22:46:51.518 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/net/phy/et1011c.ko 2022-05-13T22:46:51.518 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/net/phy/icplus.ko 2022-05-13T22:46:51.518 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/net/phy/lxt.ko 2022-05-13T22:46:51.519 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/net/phy/marvell.ko 2022-05-13T22:46:51.519 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/net/phy/micrel.ko 2022-05-13T22:46:51.519 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/net/phy/national.ko 2022-05-13T22:46:51.519 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/net/phy/phylink.ko 2022-05-13T22:46:51.519 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/net/phy/qsemi.ko 2022-05-13T22:46:51.520 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/net/phy/realtek.ko 2022-05-13T22:46:51.520 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/net/phy/smsc.ko 2022-05-13T22:46:51.520 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/net/phy/ste10Xp.ko 2022-05-13T22:46:51.520 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/net/phy/vitesse.ko 2022-05-13T22:46:51.520 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/net/ppp 2022-05-13T22:46:51.521 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/net/ppp/bsd_comp.ko 2022-05-13T22:46:51.521 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/net/ppp/ppp_async.ko 2022-05-13T22:46:51.521 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/net/ppp/ppp_deflate.ko 2022-05-13T22:46:51.521 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/net/ppp/ppp_generic.ko 2022-05-13T22:46:51.522 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/net/ppp/ppp_mppe.ko 2022-05-13T22:46:51.522 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/net/ppp/ppp_synctty.ko 2022-05-13T22:46:51.522 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/net/ppp/pppoe.ko 2022-05-13T22:46:51.522 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/net/ppp/pppox.ko 2022-05-13T22:46:51.522 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/net/ppp/pptp.ko 2022-05-13T22:46:51.523 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/net/slip 2022-05-13T22:46:51.523 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/net/slip/slhc.ko 2022-05-13T22:46:51.523 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/net/slip/slip.ko 2022-05-13T22:46:51.523 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/net/sungem_phy.ko 2022-05-13T22:46:51.523 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/net/tap.ko 2022-05-13T22:46:51.524 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/net/team 2022-05-13T22:46:51.524 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/net/team/team.ko 2022-05-13T22:46:51.524 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/net/team/team_mode_activebackup.ko 2022-05-13T22:46:51.524 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/net/team/team_mode_broadcast.ko 2022-05-13T22:46:51.524 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/net/team/team_mode_loadbalance.ko 2022-05-13T22:46:51.525 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/net/team/team_mode_random.ko 2022-05-13T22:46:51.525 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/net/team/team_mode_roundrobin.ko 2022-05-13T22:46:51.525 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/net/tun.ko 2022-05-13T22:46:51.525 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/net/veth.ko 2022-05-13T22:46:51.525 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/net/virtio_net.ko 2022-05-13T22:46:51.526 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/net/vmxnet3 2022-05-13T22:46:51.526 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/net/vmxnet3/vmxnet3.ko 2022-05-13T22:46:51.526 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/net/vxlan 2022-05-13T22:46:51.526 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/net/vxlan/vxlan.ko 2022-05-13T22:46:51.526 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/net/xen-netback 2022-05-13T22:46:51.527 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/net/xen-netback/xen-netback.ko 2022-05-13T22:46:51.527 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/net/xen-netfront.ko 2022-05-13T22:46:51.527 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/ntb 2022-05-13T22:46:51.527 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/ntb/ntb.ko 2022-05-13T22:46:51.527 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/nvme 2022-05-13T22:46:51.528 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/nvme/host 2022-05-13T22:46:51.528 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/nvme/host/nvme-core.ko 2022-05-13T22:46:51.528 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/nvme/host/nvme.ko 2022-05-13T22:46:51.528 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/parport 2022-05-13T22:46:51.528 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/parport/parport.ko 2022-05-13T22:46:51.529 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/parport/parport_cs.ko 2022-05-13T22:46:51.529 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/parport/parport_pc.ko 2022-05-13T22:46:51.529 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/parport/parport_serial.ko 2022-05-13T22:46:51.529 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/pci 2022-05-13T22:46:51.530 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/pci/hotplug 2022-05-13T22:46:51.530 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/pci/hotplug/acpiphp_ibm.ko 2022-05-13T22:46:51.530 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/pci/pcie 2022-05-13T22:46:51.530 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/pci/pcie/aer_inject.ko 2022-05-13T22:46:51.530 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/pci/xen-pcifront.ko 2022-05-13T22:46:51.531 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/pcmcia 2022-05-13T22:46:51.531 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/pcmcia/i82092.ko 2022-05-13T22:46:51.531 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/pcmcia/pd6729.ko 2022-05-13T22:46:51.531 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/pcmcia/yenta_socket.ko 2022-05-13T22:46:51.531 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/pinctrl 2022-05-13T22:46:51.532 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/pinctrl/intel 2022-05-13T22:46:51.532 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/pinctrl/intel/pinctrl-cherryview.ko 2022-05-13T22:46:51.532 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/platform 2022-05-13T22:46:51.532 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/platform/x86 2022-05-13T22:46:51.532 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/platform/x86/acer-wmi.ko 2022-05-13T22:46:51.533 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/platform/x86/acerhdf.ko 2022-05-13T22:46:51.533 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/platform/x86/apple-gmux.ko 2022-05-13T22:46:51.533 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/platform/x86/asus-laptop.ko 2022-05-13T22:46:51.533 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/platform/x86/asus-nb-wmi.ko 2022-05-13T22:46:51.534 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/platform/x86/asus-wmi.ko 2022-05-13T22:46:51.534 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/platform/x86/classmate-laptop.ko 2022-05-13T22:46:51.534 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/platform/x86/eeepc-laptop.ko 2022-05-13T22:46:51.534 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/platform/x86/eeepc-wmi.ko 2022-05-13T22:46:51.534 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/platform/x86/fujitsu-laptop.ko 2022-05-13T22:46:51.535 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/platform/x86/fujitsu-tablet.ko 2022-05-13T22:46:51.535 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/platform/x86/hdaps.ko 2022-05-13T22:46:51.535 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/platform/x86/hp-wmi.ko 2022-05-13T22:46:51.535 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/platform/x86/hp_accel.ko 2022-05-13T22:46:51.535 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/platform/x86/intel 2022-05-13T22:46:51.536 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/platform/x86/intel/intel-rst.ko 2022-05-13T22:46:51.536 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/platform/x86/intel_ips.ko 2022-05-13T22:46:51.536 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/platform/x86/msi-wmi.ko 2022-05-13T22:46:51.536 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/platform/x86/mxm-wmi.ko 2022-05-13T22:46:51.536 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/platform/x86/panasonic-laptop.ko 2022-05-13T22:46:51.537 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/platform/x86/samsung-laptop.ko 2022-05-13T22:46:51.537 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/platform/x86/samsung-q10.ko 2022-05-13T22:46:51.537 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/platform/x86/topstar-laptop.ko 2022-05-13T22:46:51.537 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/platform/x86/toshiba_bluetooth.ko 2022-05-13T22:46:51.537 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/platform/x86/toshiba_haps.ko 2022-05-13T22:46:51.538 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/platform/x86/wmi-bmof.ko 2022-05-13T22:46:51.538 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/platform/x86/wmi.ko 2022-05-13T22:46:51.538 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/powercap 2022-05-13T22:46:51.538 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/powercap/intel_rapl_common.ko 2022-05-13T22:46:51.539 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/powercap/intel_rapl_msr.ko 2022-05-13T22:46:51.539 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/pps 2022-05-13T22:46:51.539 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/pps/clients 2022-05-13T22:46:51.539 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/pps/clients/pps-gpio.ko 2022-05-13T22:46:51.539 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/pps/clients/pps-ldisc.ko 2022-05-13T22:46:51.540 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/pps/clients/pps_parport.ko 2022-05-13T22:46:51.540 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/pps/pps_core.ko 2022-05-13T22:46:51.540 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/ptp 2022-05-13T22:46:51.540 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/ptp/ptp.ko 2022-05-13T22:46:51.540 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/ptp/ptp_kvm.ko 2022-05-13T22:46:51.541 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/rtc 2022-05-13T22:46:51.541 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/rtc/rtc-bq32k.ko 2022-05-13T22:46:51.541 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/rtc/rtc-bq4802.ko 2022-05-13T22:46:51.541 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/rtc/rtc-ds1286.ko 2022-05-13T22:46:51.541 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/rtc/rtc-ds1307.ko 2022-05-13T22:46:51.542 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/rtc/rtc-ds1374.ko 2022-05-13T22:46:51.542 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/rtc/rtc-ds1511.ko 2022-05-13T22:46:51.542 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/rtc/rtc-ds1553.ko 2022-05-13T22:46:51.542 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/rtc/rtc-ds1672.ko 2022-05-13T22:46:51.542 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/rtc/rtc-ds1742.ko 2022-05-13T22:46:51.543 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/rtc/rtc-ds2404.ko 2022-05-13T22:46:51.543 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/rtc/rtc-ds3232.ko 2022-05-13T22:46:51.543 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/rtc/rtc-em3027.ko 2022-05-13T22:46:51.543 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/rtc/rtc-fm3130.ko 2022-05-13T22:46:51.543 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/rtc/rtc-isl12022.ko 2022-05-13T22:46:51.544 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/rtc/rtc-isl1208.ko 2022-05-13T22:46:51.544 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/rtc/rtc-m41t80.ko 2022-05-13T22:46:51.544 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/rtc/rtc-m48t35.ko 2022-05-13T22:46:51.544 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/rtc/rtc-m48t59.ko 2022-05-13T22:46:51.544 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/rtc/rtc-max6900.ko 2022-05-13T22:46:51.545 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/rtc/rtc-msm6242.ko 2022-05-13T22:46:51.545 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/rtc/rtc-pcf2127.ko 2022-05-13T22:46:51.545 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/rtc/rtc-pcf85063.ko 2022-05-13T22:46:51.545 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/rtc/rtc-pcf8523.ko 2022-05-13T22:46:51.546 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/rtc/rtc-pcf8563.ko 2022-05-13T22:46:51.546 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/rtc/rtc-pcf8583.ko 2022-05-13T22:46:51.546 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/rtc/rtc-rp5c01.ko 2022-05-13T22:46:51.546 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/rtc/rtc-rs5c372.ko 2022-05-13T22:46:51.546 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/rtc/rtc-rv3029c2.ko 2022-05-13T22:46:51.547 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/rtc/rtc-rx8025.ko 2022-05-13T22:46:51.547 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/rtc/rtc-rx8581.ko 2022-05-13T22:46:51.547 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/rtc/rtc-stk17ta8.ko 2022-05-13T22:46:51.547 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/rtc/rtc-v3020.ko 2022-05-13T22:46:51.547 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/rtc/rtc-x1205.ko 2022-05-13T22:46:51.548 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/scsi 2022-05-13T22:46:51.548 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/scsi/3w-9xxx.ko 2022-05-13T22:46:51.548 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/scsi/3w-sas.ko 2022-05-13T22:46:51.548 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/scsi/3w-xxxx.ko 2022-05-13T22:46:51.548 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/scsi/BusLogic.ko 2022-05-13T22:46:51.549 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/scsi/a100u2w.ko 2022-05-13T22:46:51.549 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/scsi/aacraid 2022-05-13T22:46:51.549 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/scsi/aacraid/aacraid.ko 2022-05-13T22:46:51.549 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/scsi/advansys.ko 2022-05-13T22:46:51.550 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/scsi/aic7xxx 2022-05-13T22:46:51.550 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/scsi/aic7xxx/aic79xx.ko 2022-05-13T22:46:51.550 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/scsi/aic7xxx/aic7xxx.ko 2022-05-13T22:46:51.550 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/scsi/am53c974.ko 2022-05-13T22:46:51.550 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/scsi/arcmsr 2022-05-13T22:46:51.551 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/scsi/arcmsr/arcmsr.ko 2022-05-13T22:46:51.551 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/scsi/atp870u.ko 2022-05-13T22:46:51.551 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/scsi/be2iscsi 2022-05-13T22:46:51.551 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/scsi/be2iscsi/be2iscsi.ko 2022-05-13T22:46:51.551 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/scsi/bfa 2022-05-13T22:46:51.552 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/scsi/bfa/bfa.ko 2022-05-13T22:46:51.552 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/scsi/bnx2fc 2022-05-13T22:46:51.552 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/scsi/bnx2fc/bnx2fc.ko 2022-05-13T22:46:51.552 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/scsi/bnx2i 2022-05-13T22:46:51.552 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/scsi/bnx2i/bnx2i.ko 2022-05-13T22:46:51.553 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/scsi/ch.ko 2022-05-13T22:46:51.553 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/scsi/csiostor 2022-05-13T22:46:51.553 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/scsi/csiostor/csiostor.ko 2022-05-13T22:46:51.553 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/scsi/cxgbi 2022-05-13T22:46:51.553 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/scsi/cxgbi/cxgb3i 2022-05-13T22:46:51.554 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/scsi/cxgbi/cxgb3i/cxgb3i.ko 2022-05-13T22:46:51.554 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/scsi/cxgbi/cxgb4i 2022-05-13T22:46:51.554 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/scsi/cxgbi/cxgb4i/cxgb4i.ko 2022-05-13T22:46:51.554 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/scsi/cxgbi/libcxgbi.ko 2022-05-13T22:46:51.554 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/scsi/dc395x.ko 2022-05-13T22:46:51.555 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/scsi/device_handler 2022-05-13T22:46:51.555 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/scsi/device_handler/scsi_dh_alua.ko 2022-05-13T22:46:51.555 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/scsi/device_handler/scsi_dh_emc.ko 2022-05-13T22:46:51.555 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/scsi/device_handler/scsi_dh_hp_sw.ko 2022-05-13T22:46:51.556 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/scsi/device_handler/scsi_dh_rdac.ko 2022-05-13T22:46:51.556 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/scsi/dmx3191d.ko 2022-05-13T22:46:51.556 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/scsi/esas2r 2022-05-13T22:46:51.556 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/scsi/esas2r/esas2r.ko 2022-05-13T22:46:51.556 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/scsi/esp_scsi.ko 2022-05-13T22:46:51.557 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/scsi/fcoe 2022-05-13T22:46:51.557 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/scsi/fcoe/fcoe.ko 2022-05-13T22:46:51.557 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/scsi/fcoe/libfcoe.ko 2022-05-13T22:46:51.557 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/scsi/fnic 2022-05-13T22:46:51.557 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/scsi/fnic/fnic.ko 2022-05-13T22:46:51.558 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/scsi/hpsa.ko 2022-05-13T22:46:51.558 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/scsi/hptiop.ko 2022-05-13T22:46:51.558 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/scsi/hv_storvsc.ko 2022-05-13T22:46:51.558 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/scsi/initio.ko 2022-05-13T22:46:51.558 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/scsi/ipr.ko 2022-05-13T22:46:51.559 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/scsi/ips.ko 2022-05-13T22:46:51.559 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/scsi/isci 2022-05-13T22:46:51.559 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/scsi/isci/isci.ko 2022-05-13T22:46:51.559 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/scsi/iscsi_boot_sysfs.ko 2022-05-13T22:46:51.560 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/scsi/iscsi_tcp.ko 2022-05-13T22:46:51.560 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/scsi/libfc 2022-05-13T22:46:51.560 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/scsi/libfc/libfc.ko 2022-05-13T22:46:51.560 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/scsi/libiscsi.ko 2022-05-13T22:46:51.560 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/scsi/libiscsi_tcp.ko 2022-05-13T22:46:51.561 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/scsi/libsas 2022-05-13T22:46:51.561 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/scsi/libsas/libsas.ko 2022-05-13T22:46:51.561 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/scsi/lpfc 2022-05-13T22:46:51.561 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/scsi/lpfc/lpfc.ko 2022-05-13T22:46:51.561 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/scsi/megaraid 2022-05-13T22:46:51.562 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/scsi/megaraid.ko 2022-05-13T22:46:51.562 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/scsi/megaraid/megaraid_mbox.ko 2022-05-13T22:46:51.562 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/scsi/megaraid/megaraid_mm.ko 2022-05-13T22:46:51.562 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/scsi/megaraid/megaraid_sas.ko 2022-05-13T22:46:51.562 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/scsi/mpt3sas 2022-05-13T22:46:51.563 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/scsi/mpt3sas/mpt3sas.ko 2022-05-13T22:46:51.563 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/scsi/mvsas 2022-05-13T22:46:51.563 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/scsi/mvsas/mvsas.ko 2022-05-13T22:46:51.563 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/scsi/mvumi.ko 2022-05-13T22:46:51.563 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/scsi/pm8001 2022-05-13T22:46:51.564 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/scsi/pm8001/pm80xx.ko 2022-05-13T22:46:51.564 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/scsi/pmcraid.ko 2022-05-13T22:46:51.564 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/scsi/qla1280.ko 2022-05-13T22:46:51.564 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/scsi/qla2xxx 2022-05-13T22:46:51.564 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/scsi/qla2xxx/qla2xxx.ko 2022-05-13T22:46:51.565 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/scsi/qla2xxx/tcm_qla2xxx.ko 2022-05-13T22:46:51.565 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/scsi/qla4xxx 2022-05-13T22:46:51.565 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/scsi/qla4xxx/qla4xxx.ko 2022-05-13T22:46:51.565 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/scsi/raid_class.ko 2022-05-13T22:46:51.566 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/scsi/scsi_debug.ko 2022-05-13T22:46:51.566 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/scsi/scsi_transport_fc.ko 2022-05-13T22:46:51.566 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/scsi/scsi_transport_iscsi.ko 2022-05-13T22:46:51.566 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/scsi/scsi_transport_sas.ko 2022-05-13T22:46:51.566 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/scsi/scsi_transport_spi.ko 2022-05-13T22:46:51.567 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/scsi/scsi_transport_srp.ko 2022-05-13T22:46:51.567 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/scsi/ses.ko 2022-05-13T22:46:51.567 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/scsi/st.ko 2022-05-13T22:46:51.567 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/scsi/stex.ko 2022-05-13T22:46:51.567 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/scsi/sym53c8xx_2 2022-05-13T22:46:51.568 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/scsi/sym53c8xx_2/sym53c8xx.ko 2022-05-13T22:46:51.568 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/scsi/ufs 2022-05-13T22:46:51.568 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/scsi/ufs/ufshcd-core.ko 2022-05-13T22:46:51.568 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/scsi/ufs/ufshcd-pci.ko 2022-05-13T22:46:51.568 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/scsi/virtio_scsi.ko 2022-05-13T22:46:51.569 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/scsi/vmw_pvscsi.ko 2022-05-13T22:46:51.569 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/scsi/wd719x.ko 2022-05-13T22:46:51.569 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/ssb 2022-05-13T22:46:51.569 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/ssb/ssb.ko 2022-05-13T22:46:51.569 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/target 2022-05-13T22:46:51.570 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/target/iscsi 2022-05-13T22:46:51.570 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/target/iscsi/iscsi_target_mod.ko 2022-05-13T22:46:51.570 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/target/loopback 2022-05-13T22:46:51.570 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/target/loopback/tcm_loop.ko 2022-05-13T22:46:51.570 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/target/target_core_file.ko 2022-05-13T22:46:51.571 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/target/target_core_iblock.ko 2022-05-13T22:46:51.571 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/target/target_core_mod.ko 2022-05-13T22:46:51.571 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/target/target_core_pscsi.ko 2022-05-13T22:46:51.571 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/target/target_core_user.ko 2022-05-13T22:46:51.572 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/target/tcm_fc 2022-05-13T22:46:51.572 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/target/tcm_fc/tcm_fc.ko 2022-05-13T22:46:51.572 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/thermal 2022-05-13T22:46:51.572 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/thermal/intel 2022-05-13T22:46:51.572 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/thermal/intel/int340x_thermal 2022-05-13T22:46:51.573 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/thermal/intel/int340x_thermal/acpi_thermal_rel.ko 2022-05-13T22:46:51.573 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/thermal/intel/int340x_thermal/int3400_thermal.ko 2022-05-13T22:46:51.573 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/thermal/intel/int340x_thermal/int3401_thermal.ko 2022-05-13T22:46:51.573 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/thermal/intel/int340x_thermal/int3402_thermal.ko 2022-05-13T22:46:51.573 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/thermal/intel/int340x_thermal/int3403_thermal.ko 2022-05-13T22:46:51.574 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/thermal/intel/int340x_thermal/int340x_thermal_zone.ko 2022-05-13T22:46:51.574 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device.ko 2022-05-13T22:46:51.574 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci.ko 2022-05-13T22:46:51.574 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci_legacy.ko 2022-05-13T22:46:51.574 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_mbox.ko 2022-05-13T22:46:51.575 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rapl.ko 2022-05-13T22:46:51.575 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rfim.ko 2022-05-13T22:46:51.575 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/thermal/intel/intel_soc_dts_iosf.ko 2022-05-13T22:46:51.575 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/thermal/intel/intel_soc_dts_thermal.ko 2022-05-13T22:46:51.576 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/thermal/intel/x86_pkg_temp_thermal.ko 2022-05-13T22:46:51.576 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/tty 2022-05-13T22:46:51.576 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/tty/ipwireless 2022-05-13T22:46:51.576 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/tty/ipwireless/ipwireless.ko 2022-05-13T22:46:51.576 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/tty/n_gsm.ko 2022-05-13T22:46:51.577 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/tty/n_hdlc.ko 2022-05-13T22:46:51.577 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/tty/nozomi.ko 2022-05-13T22:46:51.577 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/tty/serial 2022-05-13T22:46:51.577 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/tty/serial/8250 2022-05-13T22:46:51.577 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/tty/serial/8250/serial_cs.ko 2022-05-13T22:46:51.578 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/tty/serial/arc_uart.ko 2022-05-13T22:46:51.578 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/tty/serial/jsm 2022-05-13T22:46:51.578 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/tty/serial/jsm/jsm.ko 2022-05-13T22:46:51.578 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/tty/synclink_gt.ko 2022-05-13T22:46:51.578 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/uio 2022-05-13T22:46:51.579 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/uio/uio.ko 2022-05-13T22:46:51.579 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/uio/uio_aec.ko 2022-05-13T22:46:51.579 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/uio/uio_cif.ko 2022-05-13T22:46:51.579 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/uio/uio_pci_generic.ko 2022-05-13T22:46:51.579 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/uio/uio_sercos3.ko 2022-05-13T22:46:51.580 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/usb 2022-05-13T22:46:51.580 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/usb/class 2022-05-13T22:46:51.580 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/usb/class/cdc-acm.ko 2022-05-13T22:46:51.580 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/usb/class/cdc-wdm.ko 2022-05-13T22:46:51.581 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/usb/class/usblp.ko 2022-05-13T22:46:51.581 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/usb/class/usbtmc.ko 2022-05-13T22:46:51.581 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/usb/host 2022-05-13T22:46:51.581 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/usb/host/sl811-hcd.ko 2022-05-13T22:46:51.581 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/usb/host/u132-hcd.ko 2022-05-13T22:46:51.582 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/usb/image 2022-05-13T22:46:51.582 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/usb/image/mdc800.ko 2022-05-13T22:46:51.582 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/usb/image/microtek.ko 2022-05-13T22:46:51.582 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/usb/misc 2022-05-13T22:46:51.583 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/usb/misc/adutux.ko 2022-05-13T22:46:51.583 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/usb/misc/appledisplay.ko 2022-05-13T22:46:51.583 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/usb/misc/emi26.ko 2022-05-13T22:46:51.583 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/usb/misc/emi62.ko 2022-05-13T22:46:51.583 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/usb/misc/ezusb.ko 2022-05-13T22:46:51.584 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/usb/misc/ftdi-elan.ko 2022-05-13T22:46:51.584 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/usb/misc/idmouse.ko 2022-05-13T22:46:51.584 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/usb/misc/iowarrior.ko 2022-05-13T22:46:51.584 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/usb/misc/isight_firmware.ko 2022-05-13T22:46:51.584 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/usb/misc/ldusb.ko 2022-05-13T22:46:51.585 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/usb/misc/legousbtower.ko 2022-05-13T22:46:51.585 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/usb/misc/sisusbvga 2022-05-13T22:46:51.585 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/usb/misc/sisusbvga/sisusbvga.ko 2022-05-13T22:46:51.585 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/usb/misc/trancevibrator.ko 2022-05-13T22:46:51.585 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/usb/misc/usb3503.ko 2022-05-13T22:46:51.586 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/usb/misc/usblcd.ko 2022-05-13T22:46:51.586 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/usb/misc/usbsevseg.ko 2022-05-13T22:46:51.586 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/usb/misc/uss720.ko 2022-05-13T22:46:51.586 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/usb/misc/yurex.ko 2022-05-13T22:46:51.587 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/usb/phy 2022-05-13T22:46:51.587 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/usb/phy/phy-generic.ko 2022-05-13T22:46:51.587 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/usb/serial 2022-05-13T22:46:51.587 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/usb/serial/aircable.ko 2022-05-13T22:46:51.587 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/usb/serial/ark3116.ko 2022-05-13T22:46:51.588 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/usb/serial/belkin_sa.ko 2022-05-13T22:46:51.588 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/usb/serial/ch341.ko 2022-05-13T22:46:51.588 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/usb/serial/cp210x.ko 2022-05-13T22:46:51.588 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/usb/serial/cyberjack.ko 2022-05-13T22:46:51.588 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/usb/serial/cypress_m8.ko 2022-05-13T22:46:51.589 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/usb/serial/digi_acceleport.ko 2022-05-13T22:46:51.589 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/usb/serial/empeg.ko 2022-05-13T22:46:51.589 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/usb/serial/ftdi_sio.ko 2022-05-13T22:46:51.589 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/usb/serial/garmin_gps.ko 2022-05-13T22:46:51.590 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/usb/serial/io_edgeport.ko 2022-05-13T22:46:51.590 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/usb/serial/io_ti.ko 2022-05-13T22:46:51.590 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/usb/serial/ipaq.ko 2022-05-13T22:46:51.590 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/usb/serial/ipw.ko 2022-05-13T22:46:51.590 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/usb/serial/ir-usb.ko 2022-05-13T22:46:51.591 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/usb/serial/iuu_phoenix.ko 2022-05-13T22:46:51.591 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/usb/serial/keyspan.ko 2022-05-13T22:46:51.591 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/usb/serial/keyspan_pda.ko 2022-05-13T22:46:51.591 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/usb/serial/kl5kusb105.ko 2022-05-13T22:46:51.591 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/usb/serial/kobil_sct.ko 2022-05-13T22:46:51.592 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/usb/serial/mct_u232.ko 2022-05-13T22:46:51.592 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/usb/serial/mos7720.ko 2022-05-13T22:46:51.592 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/usb/serial/mos7840.ko 2022-05-13T22:46:51.592 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/usb/serial/navman.ko 2022-05-13T22:46:51.592 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/usb/serial/omninet.ko 2022-05-13T22:46:51.593 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/usb/serial/opticon.ko 2022-05-13T22:46:51.593 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/usb/serial/option.ko 2022-05-13T22:46:51.593 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/usb/serial/oti6858.ko 2022-05-13T22:46:51.593 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/usb/serial/pl2303.ko 2022-05-13T22:46:51.593 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/usb/serial/qcaux.ko 2022-05-13T22:46:51.594 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/usb/serial/qcserial.ko 2022-05-13T22:46:51.594 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/usb/serial/quatech2.ko 2022-05-13T22:46:51.594 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/usb/serial/safe_serial.ko 2022-05-13T22:46:51.594 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/usb/serial/sierra.ko 2022-05-13T22:46:51.594 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/usb/serial/spcp8x5.ko 2022-05-13T22:46:51.595 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/usb/serial/ssu100.ko 2022-05-13T22:46:51.595 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/usb/serial/symbolserial.ko 2022-05-13T22:46:51.595 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/usb/serial/ti_usb_3410_5052.ko 2022-05-13T22:46:51.595 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/usb/serial/usb-serial-simple.ko 2022-05-13T22:46:51.595 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/usb/serial/usb_debug.ko 2022-05-13T22:46:51.596 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/usb/serial/usb_wwan.ko 2022-05-13T22:46:51.596 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/usb/serial/visor.ko 2022-05-13T22:46:51.596 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/usb/serial/whiteheat.ko 2022-05-13T22:46:51.596 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/usb/serial/xsens_mt.ko 2022-05-13T22:46:51.596 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/usb/storage 2022-05-13T22:46:51.597 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/usb/storage/uas.ko 2022-05-13T22:46:51.597 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/usb/storage/ums-alauda.ko 2022-05-13T22:46:51.597 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/usb/storage/ums-cypress.ko 2022-05-13T22:46:51.597 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/usb/storage/ums-datafab.ko 2022-05-13T22:46:51.598 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/usb/storage/ums-eneub6250.ko 2022-05-13T22:46:51.598 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/usb/storage/ums-freecom.ko 2022-05-13T22:46:51.598 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/usb/storage/ums-isd200.ko 2022-05-13T22:46:51.598 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/usb/storage/ums-jumpshot.ko 2022-05-13T22:46:51.598 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/usb/storage/ums-karma.ko 2022-05-13T22:46:51.599 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/usb/storage/ums-onetouch.ko 2022-05-13T22:46:51.599 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/usb/storage/ums-realtek.ko 2022-05-13T22:46:51.599 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/usb/storage/ums-sddr09.ko 2022-05-13T22:46:51.599 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/usb/storage/ums-sddr55.ko 2022-05-13T22:46:51.600 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/usb/storage/ums-usbat.ko 2022-05-13T22:46:51.600 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/usb/storage/usb-storage.ko 2022-05-13T22:46:51.600 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/usb/usbip 2022-05-13T22:46:51.600 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/usb/usbip/usbip-core.ko 2022-05-13T22:46:51.600 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/usb/usbip/usbip-host.ko 2022-05-13T22:46:51.601 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/usb/usbip/vhci-hcd.ko 2022-05-13T22:46:51.601 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/vfio 2022-05-13T22:46:51.601 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/vfio/pci 2022-05-13T22:46:51.601 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/vfio/pci/vfio-pci-core.ko 2022-05-13T22:46:51.601 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/vfio/pci/vfio-pci.ko 2022-05-13T22:46:51.602 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/vfio/vfio.ko 2022-05-13T22:46:51.602 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/vfio/vfio_iommu_type1.ko 2022-05-13T22:46:51.602 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/vfio/vfio_virqfd.ko 2022-05-13T22:46:51.602 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/vhost 2022-05-13T22:46:51.602 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/vhost/vhost.ko 2022-05-13T22:46:51.603 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/vhost/vhost_iotlb.ko 2022-05-13T22:46:51.603 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/vhost/vhost_net.ko 2022-05-13T22:46:51.603 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/vhost/vhost_scsi.ko 2022-05-13T22:46:51.603 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/video 2022-05-13T22:46:51.603 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/video/backlight 2022-05-13T22:46:51.604 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/video/backlight/apple_bl.ko 2022-05-13T22:46:51.604 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/video/backlight/lcd.ko 2022-05-13T22:46:51.604 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/video/backlight/platform_lcd.ko 2022-05-13T22:46:51.604 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/virtio 2022-05-13T22:46:51.605 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/virtio/virtio.ko 2022-05-13T22:46:51.605 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/virtio/virtio_balloon.ko 2022-05-13T22:46:51.605 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/virtio/virtio_mmio.ko 2022-05-13T22:46:51.605 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/virtio/virtio_pci.ko 2022-05-13T22:46:51.605 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/virtio/virtio_pci_legacy_dev.ko 2022-05-13T22:46:51.606 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/virtio/virtio_pci_modern_dev.ko 2022-05-13T22:46:51.606 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/virtio/virtio_ring.ko 2022-05-13T22:46:51.606 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/w1 2022-05-13T22:46:51.606 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/w1/masters 2022-05-13T22:46:51.606 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/w1/masters/ds1wm.ko 2022-05-13T22:46:51.607 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/w1/masters/ds2482.ko 2022-05-13T22:46:51.607 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/w1/masters/ds2490.ko 2022-05-13T22:46:51.607 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/w1/slaves 2022-05-13T22:46:51.607 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/w1/slaves/w1_ds2406.ko 2022-05-13T22:46:51.607 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/w1/slaves/w1_ds2408.ko 2022-05-13T22:46:51.608 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/w1/slaves/w1_ds2413.ko 2022-05-13T22:46:51.608 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/w1/slaves/w1_ds2423.ko 2022-05-13T22:46:51.608 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/w1/slaves/w1_ds2431.ko 2022-05-13T22:46:51.608 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/w1/slaves/w1_ds2433.ko 2022-05-13T22:46:51.608 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/w1/slaves/w1_ds2780.ko 2022-05-13T22:46:51.609 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/w1/slaves/w1_ds2781.ko 2022-05-13T22:46:51.609 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/w1/slaves/w1_ds28e04.ko 2022-05-13T22:46:51.609 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/w1/slaves/w1_smem.ko 2022-05-13T22:46:51.609 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/w1/slaves/w1_therm.ko 2022-05-13T22:46:51.610 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/w1/wire.ko 2022-05-13T22:46:51.610 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/watchdog 2022-05-13T22:46:51.610 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/watchdog/alim1535_wdt.ko 2022-05-13T22:46:51.610 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/watchdog/alim7101_wdt.ko 2022-05-13T22:46:51.610 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/watchdog/f71808e_wdt.ko 2022-05-13T22:46:51.611 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/watchdog/hpwdt.ko 2022-05-13T22:46:51.611 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/watchdog/i6300esb.ko 2022-05-13T22:46:51.611 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/watchdog/iTCO_vendor_support.ko 2022-05-13T22:46:51.611 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/watchdog/iTCO_wdt.ko 2022-05-13T22:46:51.611 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/watchdog/ib700wdt.ko 2022-05-13T22:46:51.612 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/watchdog/ibmasr.ko 2022-05-13T22:46:51.612 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/watchdog/ie6xx_wdt.ko 2022-05-13T22:46:51.612 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/watchdog/it8712f_wdt.ko 2022-05-13T22:46:51.612 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/watchdog/it87_wdt.ko 2022-05-13T22:46:51.612 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/watchdog/machzwd.ko 2022-05-13T22:46:51.613 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/watchdog/nv_tco.ko 2022-05-13T22:46:51.613 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/watchdog/pcwd_pci.ko 2022-05-13T22:46:51.613 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/watchdog/pcwd_usb.ko 2022-05-13T22:46:51.613 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/watchdog/sbc_fitpc2_wdt.ko 2022-05-13T22:46:51.614 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/watchdog/sch311x_wdt.ko 2022-05-13T22:46:51.614 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/watchdog/softdog.ko 2022-05-13T22:46:51.614 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/watchdog/sp5100_tco.ko 2022-05-13T22:46:51.614 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/watchdog/via_wdt.ko 2022-05-13T22:46:51.614 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/watchdog/w83627hf_wdt.ko 2022-05-13T22:46:51.615 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/watchdog/w83877f_wdt.ko 2022-05-13T22:46:51.615 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/watchdog/w83977f_wdt.ko 2022-05-13T22:46:51.615 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/watchdog/wdt_pci.ko 2022-05-13T22:46:51.615 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/watchdog/xen_wdt.ko 2022-05-13T22:46:51.615 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/xen 2022-05-13T22:46:51.616 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/xen/xen-acpi-processor.ko 2022-05-13T22:46:51.616 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/xen/xen-evtchn.ko 2022-05-13T22:46:51.616 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/xen/xen-gntalloc.ko 2022-05-13T22:46:51.616 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/xen/xen-gntdev.ko 2022-05-13T22:46:51.616 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/xen/xen-pciback 2022-05-13T22:46:51.617 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/xen/xen-pciback/xen-pciback.ko 2022-05-13T22:46:51.617 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/xen/xen-privcmd.ko 2022-05-13T22:46:51.617 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/xen/xenfs 2022-05-13T22:46:51.617 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/drivers/xen/xenfs/xenfs.ko 2022-05-13T22:46:51.618 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/fs 2022-05-13T22:46:51.618 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/fs/9p 2022-05-13T22:46:51.618 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/fs/9p/9p.ko 2022-05-13T22:46:51.618 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/fs/affs 2022-05-13T22:46:51.618 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/fs/affs/affs.ko 2022-05-13T22:46:51.619 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/fs/befs 2022-05-13T22:46:51.619 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/fs/befs/befs.ko 2022-05-13T22:46:51.619 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/fs/binfmt_misc.ko 2022-05-13T22:46:51.619 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/fs/btrfs 2022-05-13T22:46:51.619 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/fs/btrfs/btrfs.ko 2022-05-13T22:46:51.620 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/fs/cachefiles 2022-05-13T22:46:51.620 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/fs/cachefiles/cachefiles.ko 2022-05-13T22:46:51.620 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/fs/ceph 2022-05-13T22:46:51.620 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/fs/ceph/ceph.ko 2022-05-13T22:46:51.620 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/fs/cifs 2022-05-13T22:46:51.621 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/fs/cifs/cifs.ko 2022-05-13T22:46:51.621 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/fs/coda 2022-05-13T22:46:51.621 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/fs/coda/coda.ko 2022-05-13T22:46:51.621 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/fs/cramfs 2022-05-13T22:46:51.621 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/fs/cramfs/cramfs.ko 2022-05-13T22:46:51.622 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/fs/dlm 2022-05-13T22:46:51.622 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/fs/dlm/dlm.ko 2022-05-13T22:46:51.622 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/fs/ecryptfs 2022-05-13T22:46:51.622 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/fs/ecryptfs/ecryptfs.ko 2022-05-13T22:46:51.622 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/fs/f2fs 2022-05-13T22:46:51.623 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/fs/f2fs/f2fs.ko 2022-05-13T22:46:51.623 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/fs/fat 2022-05-13T22:46:51.623 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/fs/fat/fat.ko 2022-05-13T22:46:51.623 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/fs/fat/msdos.ko 2022-05-13T22:46:51.623 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/fs/fat/vfat.ko 2022-05-13T22:46:51.624 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/fs/fscache 2022-05-13T22:46:51.624 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/fs/fscache/fscache.ko 2022-05-13T22:46:51.624 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/fs/fuse 2022-05-13T22:46:51.624 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/fs/fuse/cuse.ko 2022-05-13T22:46:51.625 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/fs/fuse/fuse.ko 2022-05-13T22:46:51.625 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/fs/gfs2 2022-05-13T22:46:51.625 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/fs/gfs2/gfs2.ko 2022-05-13T22:46:51.625 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/fs/hfs 2022-05-13T22:46:51.625 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/fs/hfs/hfs.ko 2022-05-13T22:46:51.626 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/fs/hfsplus 2022-05-13T22:46:51.626 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/fs/hfsplus/hfsplus.ko 2022-05-13T22:46:51.626 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/fs/isofs 2022-05-13T22:46:51.626 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/fs/isofs/isofs.ko 2022-05-13T22:46:51.626 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/fs/jfs 2022-05-13T22:46:51.627 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/fs/jfs/jfs.ko 2022-05-13T22:46:51.627 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/fs/lockd 2022-05-13T22:46:51.627 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/fs/lockd/lockd.ko 2022-05-13T22:46:51.627 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/fs/minix 2022-05-13T22:46:51.627 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/fs/minix/minix.ko 2022-05-13T22:46:51.628 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/fs/netfs 2022-05-13T22:46:51.628 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/fs/netfs/netfs.ko 2022-05-13T22:46:51.628 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/fs/nfs 2022-05-13T22:46:51.628 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/fs/nfs/blocklayout 2022-05-13T22:46:51.628 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/fs/nfs/blocklayout/blocklayoutdriver.ko 2022-05-13T22:46:51.629 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/fs/nfs/filelayout 2022-05-13T22:46:51.629 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/fs/nfs/filelayout/nfs_layout_nfsv41_files.ko 2022-05-13T22:46:51.629 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/fs/nfs/flexfilelayout 2022-05-13T22:46:51.629 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/fs/nfs/flexfilelayout/nfs_layout_flexfiles.ko 2022-05-13T22:46:51.630 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/fs/nfs/nfs.ko 2022-05-13T22:46:51.630 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/fs/nfs/nfsv3.ko 2022-05-13T22:46:51.630 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/fs/nfs/nfsv4.ko 2022-05-13T22:46:51.630 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/fs/nfs_common 2022-05-13T22:46:51.630 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/fs/nfs_common/grace.ko 2022-05-13T22:46:51.631 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/fs/nfs_common/nfs_acl.ko 2022-05-13T22:46:51.631 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/fs/nfsd 2022-05-13T22:46:51.631 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/fs/nfsd/nfsd.ko 2022-05-13T22:46:51.631 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/fs/nilfs2 2022-05-13T22:46:51.631 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/fs/nilfs2/nilfs2.ko 2022-05-13T22:46:51.632 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/fs/nls 2022-05-13T22:46:51.632 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/fs/nls/mac-celtic.ko 2022-05-13T22:46:51.632 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/fs/nls/mac-centeuro.ko 2022-05-13T22:46:51.632 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/fs/nls/mac-croatian.ko 2022-05-13T22:46:51.632 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/fs/nls/mac-cyrillic.ko 2022-05-13T22:46:51.633 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/fs/nls/mac-gaelic.ko 2022-05-13T22:46:51.633 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/fs/nls/mac-greek.ko 2022-05-13T22:46:51.633 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/fs/nls/mac-iceland.ko 2022-05-13T22:46:51.633 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/fs/nls/mac-inuit.ko 2022-05-13T22:46:51.633 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/fs/nls/mac-roman.ko 2022-05-13T22:46:51.634 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/fs/nls/mac-romanian.ko 2022-05-13T22:46:51.634 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/fs/nls/mac-turkish.ko 2022-05-13T22:46:51.634 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/fs/nls/nls_cp1250.ko 2022-05-13T22:46:51.634 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/fs/nls/nls_cp1251.ko 2022-05-13T22:46:51.635 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/fs/nls/nls_cp1255.ko 2022-05-13T22:46:51.635 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/fs/nls/nls_cp737.ko 2022-05-13T22:46:51.635 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/fs/nls/nls_cp775.ko 2022-05-13T22:46:51.635 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/fs/nls/nls_cp850.ko 2022-05-13T22:46:51.635 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/fs/nls/nls_cp852.ko 2022-05-13T22:46:51.636 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/fs/nls/nls_cp855.ko 2022-05-13T22:46:51.636 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/fs/nls/nls_cp857.ko 2022-05-13T22:46:51.636 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/fs/nls/nls_cp860.ko 2022-05-13T22:46:51.636 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/fs/nls/nls_cp861.ko 2022-05-13T22:46:51.636 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/fs/nls/nls_cp862.ko 2022-05-13T22:46:51.637 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/fs/nls/nls_cp863.ko 2022-05-13T22:46:51.637 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/fs/nls/nls_cp864.ko 2022-05-13T22:46:51.637 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/fs/nls/nls_cp865.ko 2022-05-13T22:46:51.637 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/fs/nls/nls_cp866.ko 2022-05-13T22:46:51.638 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/fs/nls/nls_cp869.ko 2022-05-13T22:46:51.638 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/fs/nls/nls_cp874.ko 2022-05-13T22:46:51.638 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/fs/nls/nls_cp932.ko 2022-05-13T22:46:51.638 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/fs/nls/nls_cp936.ko 2022-05-13T22:46:51.638 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/fs/nls/nls_cp949.ko 2022-05-13T22:46:51.639 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/fs/nls/nls_cp950.ko 2022-05-13T22:46:51.639 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/fs/nls/nls_euc-jp.ko 2022-05-13T22:46:51.639 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/fs/nls/nls_iso8859-1.ko 2022-05-13T22:46:51.639 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/fs/nls/nls_iso8859-13.ko 2022-05-13T22:46:51.639 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/fs/nls/nls_iso8859-14.ko 2022-05-13T22:46:51.640 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/fs/nls/nls_iso8859-15.ko 2022-05-13T22:46:51.640 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/fs/nls/nls_iso8859-2.ko 2022-05-13T22:46:51.640 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/fs/nls/nls_iso8859-3.ko 2022-05-13T22:46:51.640 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/fs/nls/nls_iso8859-4.ko 2022-05-13T22:46:51.640 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/fs/nls/nls_iso8859-5.ko 2022-05-13T22:46:51.641 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/fs/nls/nls_iso8859-6.ko 2022-05-13T22:46:51.641 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/fs/nls/nls_iso8859-7.ko 2022-05-13T22:46:51.641 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/fs/nls/nls_iso8859-9.ko 2022-05-13T22:46:51.641 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/fs/nls/nls_koi8-r.ko 2022-05-13T22:46:51.641 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/fs/nls/nls_koi8-ru.ko 2022-05-13T22:46:51.642 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/fs/nls/nls_koi8-u.ko 2022-05-13T22:46:51.642 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/fs/nls/nls_utf8.ko 2022-05-13T22:46:51.642 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/fs/ocfs2 2022-05-13T22:46:51.642 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/fs/ocfs2/cluster 2022-05-13T22:46:51.642 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/fs/ocfs2/cluster/ocfs2_nodemanager.ko 2022-05-13T22:46:51.643 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/fs/ocfs2/dlm 2022-05-13T22:46:51.643 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/fs/ocfs2/dlm/ocfs2_dlm.ko 2022-05-13T22:46:51.643 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/fs/ocfs2/dlmfs 2022-05-13T22:46:51.643 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/fs/ocfs2/dlmfs/ocfs2_dlmfs.ko 2022-05-13T22:46:51.643 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/fs/ocfs2/ocfs2.ko 2022-05-13T22:46:51.644 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/fs/ocfs2/ocfs2_stack_o2cb.ko 2022-05-13T22:46:51.644 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/fs/ocfs2/ocfs2_stack_user.ko 2022-05-13T22:46:51.644 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/fs/ocfs2/ocfs2_stackglue.ko 2022-05-13T22:46:51.644 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/fs/overlayfs 2022-05-13T22:46:51.644 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/fs/overlayfs/overlay.ko 2022-05-13T22:46:51.645 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/fs/pstore 2022-05-13T22:46:51.645 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/fs/pstore/ramoops.ko 2022-05-13T22:46:51.645 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/fs/reiserfs 2022-05-13T22:46:51.645 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/fs/reiserfs/reiserfs.ko 2022-05-13T22:46:51.646 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/fs/romfs 2022-05-13T22:46:51.646 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/fs/romfs/romfs.ko 2022-05-13T22:46:51.646 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/fs/smbfs_common 2022-05-13T22:46:51.646 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/fs/smbfs_common/cifs_arc4.ko 2022-05-13T22:46:51.646 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/fs/smbfs_common/cifs_md4.ko 2022-05-13T22:46:51.647 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/fs/squashfs 2022-05-13T22:46:51.647 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/fs/squashfs/squashfs.ko 2022-05-13T22:46:51.647 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/fs/sysv 2022-05-13T22:46:51.647 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/fs/sysv/sysv.ko 2022-05-13T22:46:51.647 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/fs/udf 2022-05-13T22:46:51.648 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/fs/udf/udf.ko 2022-05-13T22:46:51.648 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/fs/ufs 2022-05-13T22:46:51.648 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/fs/ufs/ufs.ko 2022-05-13T22:46:51.648 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/fs/xfs 2022-05-13T22:46:51.648 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/fs/xfs/xfs.ko 2022-05-13T22:46:51.649 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/kernel 2022-05-13T22:46:51.649 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/kernel/rcu 2022-05-13T22:46:51.649 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/kernel/rcu/rcutorture.ko 2022-05-13T22:46:51.649 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/kernel/torture.ko 2022-05-13T22:46:51.650 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/kernel/trace 2022-05-13T22:46:51.650 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/kernel/trace/ring_buffer_benchmark.ko 2022-05-13T22:46:51.650 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/lib 2022-05-13T22:46:51.650 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/lib/asn1_encoder.ko 2022-05-13T22:46:51.650 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/lib/crc-itu-t.ko 2022-05-13T22:46:51.651 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/lib/crc8.ko 2022-05-13T22:46:51.651 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/lib/crypto 2022-05-13T22:46:51.651 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/lib/crypto/libarc4.ko 2022-05-13T22:46:51.651 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/lib/crypto/libdes.ko 2022-05-13T22:46:51.651 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/lib/libcrc32c.ko 2022-05-13T22:46:51.652 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/lib/lru_cache.ko 2022-05-13T22:46:51.652 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/lib/lz4 2022-05-13T22:46:51.652 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/lib/lz4/lz4_compress.ko 2022-05-13T22:46:51.652 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/lib/lz4/lz4hc_compress.ko 2022-05-13T22:46:51.653 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/lib/math 2022-05-13T22:46:51.653 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/lib/math/cordic.ko 2022-05-13T22:46:51.653 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/lib/raid6 2022-05-13T22:46:51.653 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/lib/raid6/raid6_pq.ko 2022-05-13T22:46:51.653 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/lib/reed_solomon 2022-05-13T22:46:51.654 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/lib/reed_solomon/reed_solomon.ko 2022-05-13T22:46:51.654 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/lib/ts_bm.ko 2022-05-13T22:46:51.654 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/lib/ts_fsm.ko 2022-05-13T22:46:51.654 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/lib/ts_kmp.ko 2022-05-13T22:46:51.655 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/lib/zstd 2022-05-13T22:46:51.655 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/lib/zstd/zstd_compress.ko 2022-05-13T22:46:51.655 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/mm 2022-05-13T22:46:51.655 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/mm/hwpoison-inject.ko 2022-05-13T22:46:51.655 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net 2022-05-13T22:46:51.656 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/6lowpan 2022-05-13T22:46:51.656 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/6lowpan/6lowpan.ko 2022-05-13T22:46:51.656 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/6lowpan/nhc_dest.ko 2022-05-13T22:46:51.656 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/6lowpan/nhc_fragment.ko 2022-05-13T22:46:51.656 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/6lowpan/nhc_hop.ko 2022-05-13T22:46:51.657 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/6lowpan/nhc_ipv6.ko 2022-05-13T22:46:51.657 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/6lowpan/nhc_mobility.ko 2022-05-13T22:46:51.657 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/6lowpan/nhc_routing.ko 2022-05-13T22:46:51.657 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/6lowpan/nhc_udp.ko 2022-05-13T22:46:51.658 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/802 2022-05-13T22:46:51.658 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/802/garp.ko 2022-05-13T22:46:51.658 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/802/mrp.ko 2022-05-13T22:46:51.658 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/802/p8022.ko 2022-05-13T22:46:51.658 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/802/psnap.ko 2022-05-13T22:46:51.659 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/802/stp.ko 2022-05-13T22:46:51.659 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/8021q 2022-05-13T22:46:51.659 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/8021q/8021q.ko 2022-05-13T22:46:51.659 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/9p 2022-05-13T22:46:51.659 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/9p/9pnet.ko 2022-05-13T22:46:51.660 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/9p/9pnet_fd.ko 2022-05-13T22:46:51.660 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/9p/9pnet_rdma.ko 2022-05-13T22:46:51.660 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/9p/9pnet_virtio.ko 2022-05-13T22:46:51.660 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/batman-adv 2022-05-13T22:46:51.661 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/batman-adv/batman-adv.ko 2022-05-13T22:46:51.661 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/bridge 2022-05-13T22:46:51.661 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/bridge/br_netfilter.ko 2022-05-13T22:46:51.661 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/bridge/bridge.ko 2022-05-13T22:46:51.661 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/bridge/netfilter 2022-05-13T22:46:51.662 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/bridge/netfilter/ebt_802_3.ko 2022-05-13T22:46:51.662 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/bridge/netfilter/ebt_among.ko 2022-05-13T22:46:51.662 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/bridge/netfilter/ebt_arp.ko 2022-05-13T22:46:51.662 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/bridge/netfilter/ebt_arpreply.ko 2022-05-13T22:46:51.662 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/bridge/netfilter/ebt_dnat.ko 2022-05-13T22:46:51.663 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/bridge/netfilter/ebt_ip.ko 2022-05-13T22:46:51.663 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/bridge/netfilter/ebt_ip6.ko 2022-05-13T22:46:51.663 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/bridge/netfilter/ebt_limit.ko 2022-05-13T22:46:51.663 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/bridge/netfilter/ebt_log.ko 2022-05-13T22:46:51.663 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/bridge/netfilter/ebt_mark.ko 2022-05-13T22:46:51.664 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/bridge/netfilter/ebt_mark_m.ko 2022-05-13T22:46:51.664 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/bridge/netfilter/ebt_nflog.ko 2022-05-13T22:46:51.664 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/bridge/netfilter/ebt_pkttype.ko 2022-05-13T22:46:51.664 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/bridge/netfilter/ebt_redirect.ko 2022-05-13T22:46:51.665 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/bridge/netfilter/ebt_snat.ko 2022-05-13T22:46:51.665 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/bridge/netfilter/ebt_stp.ko 2022-05-13T22:46:51.665 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/bridge/netfilter/ebt_vlan.ko 2022-05-13T22:46:51.665 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/bridge/netfilter/ebtable_broute.ko 2022-05-13T22:46:51.665 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/bridge/netfilter/ebtable_filter.ko 2022-05-13T22:46:51.666 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/bridge/netfilter/ebtable_nat.ko 2022-05-13T22:46:51.666 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/bridge/netfilter/ebtables.ko 2022-05-13T22:46:51.666 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/bridge/netfilter/nf_conntrack_bridge.ko 2022-05-13T22:46:51.666 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/bridge/netfilter/nft_meta_bridge.ko 2022-05-13T22:46:51.666 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/bridge/netfilter/nft_reject_bridge.ko 2022-05-13T22:46:51.667 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/ceph 2022-05-13T22:46:51.667 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/ceph/libceph.ko 2022-05-13T22:46:51.667 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/core 2022-05-13T22:46:51.667 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/core/failover.ko 2022-05-13T22:46:51.667 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/core/pktgen.ko 2022-05-13T22:46:51.668 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/dccp 2022-05-13T22:46:51.668 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/dccp/dccp.ko 2022-05-13T22:46:51.668 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/dccp/dccp_diag.ko 2022-05-13T22:46:51.668 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/dccp/dccp_ipv4.ko 2022-05-13T22:46:51.669 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/dccp/dccp_ipv6.ko 2022-05-13T22:46:51.669 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/dns_resolver 2022-05-13T22:46:51.669 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/dns_resolver/dns_resolver.ko 2022-05-13T22:46:51.669 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/dsa 2022-05-13T22:46:51.670 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/dsa/dsa_core.ko 2022-05-13T22:46:51.670 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/dsa/tag_brcm.ko 2022-05-13T22:46:51.670 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/dsa/tag_dsa.ko 2022-05-13T22:46:51.671 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/dsa/tag_trailer.ko 2022-05-13T22:46:51.671 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/ieee802154 2022-05-13T22:46:51.671 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/ieee802154/6lowpan 2022-05-13T22:46:51.671 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/ieee802154/6lowpan/ieee802154_6lowpan.ko 2022-05-13T22:46:51.671 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/ieee802154/ieee802154.ko 2022-05-13T22:46:51.672 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/ieee802154/ieee802154_socket.ko 2022-05-13T22:46:51.672 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/ipv4 2022-05-13T22:46:51.672 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/ipv4/ah4.ko 2022-05-13T22:46:51.672 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/ipv4/esp4.ko 2022-05-13T22:46:51.673 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/ipv4/fou.ko 2022-05-13T22:46:51.673 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/ipv4/gre.ko 2022-05-13T22:46:51.673 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/ipv4/inet_diag.ko 2022-05-13T22:46:51.673 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/ipv4/ip_gre.ko 2022-05-13T22:46:51.674 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/ipv4/ip_tunnel.ko 2022-05-13T22:46:51.674 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/ipv4/ip_vti.ko 2022-05-13T22:46:51.674 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/ipv4/ipcomp.ko 2022-05-13T22:46:51.674 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/ipv4/ipip.ko 2022-05-13T22:46:51.675 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/ipv4/netfilter 2022-05-13T22:46:51.675 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/ipv4/netfilter/arp_tables.ko 2022-05-13T22:46:51.675 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/ipv4/netfilter/arpt_mangle.ko 2022-05-13T22:46:51.676 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/ipv4/netfilter/arptable_filter.ko 2022-05-13T22:46:51.676 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/ipv4/netfilter/ipt_CLUSTERIP.ko 2022-05-13T22:46:51.676 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/ipv4/netfilter/ipt_ECN.ko 2022-05-13T22:46:51.677 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/ipv4/netfilter/ipt_SYNPROXY.ko 2022-05-13T22:46:51.677 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/ipv4/netfilter/ipt_ah.ko 2022-05-13T22:46:51.677 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/ipv4/netfilter/ipt_rpfilter.ko 2022-05-13T22:46:51.686 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/ipv4/netfilter/iptable_mangle.ko 2022-05-13T22:46:51.686 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/ipv4/netfilter/iptable_nat.ko 2022-05-13T22:46:51.686 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/ipv4/netfilter/iptable_raw.ko 2022-05-13T22:46:51.687 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/ipv4/netfilter/iptable_security.ko 2022-05-13T22:46:51.687 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/ipv4/netfilter/nf_defrag_ipv4.ko 2022-05-13T22:46:51.687 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/ipv4/netfilter/nf_dup_ipv4.ko 2022-05-13T22:46:51.688 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/ipv4/netfilter/nf_nat_h323.ko 2022-05-13T22:46:51.688 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/ipv4/netfilter/nf_nat_pptp.ko 2022-05-13T22:46:51.689 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/ipv4/netfilter/nf_nat_snmp_basic.ko 2022-05-13T22:46:51.689 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/ipv4/netfilter/nf_socket_ipv4.ko 2022-05-13T22:46:51.689 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/ipv4/netfilter/nf_tproxy_ipv4.ko 2022-05-13T22:46:51.690 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/ipv4/netfilter/nft_dup_ipv4.ko 2022-05-13T22:46:51.690 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/ipv4/netfilter/nft_fib_ipv4.ko 2022-05-13T22:46:51.690 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/ipv4/netfilter/nft_reject_ipv4.ko 2022-05-13T22:46:51.691 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/ipv4/tcp_bic.ko 2022-05-13T22:46:51.691 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/ipv4/tcp_dctcp.ko 2022-05-13T22:46:51.691 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/ipv4/tcp_diag.ko 2022-05-13T22:46:51.692 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/ipv4/tcp_highspeed.ko 2022-05-13T22:46:51.692 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/ipv4/tcp_htcp.ko 2022-05-13T22:46:51.693 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/ipv4/tcp_hybla.ko 2022-05-13T22:46:51.693 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/ipv4/tcp_illinois.ko 2022-05-13T22:46:51.693 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/ipv4/tcp_lp.ko 2022-05-13T22:46:51.694 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/ipv4/tcp_scalable.ko 2022-05-13T22:46:51.694 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/ipv4/tcp_vegas.ko 2022-05-13T22:46:51.694 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/ipv4/tcp_veno.ko 2022-05-13T22:46:51.695 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/ipv4/tcp_westwood.ko 2022-05-13T22:46:51.695 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/ipv4/tcp_yeah.ko 2022-05-13T22:46:51.695 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/ipv4/tunnel4.ko 2022-05-13T22:46:51.696 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/ipv4/udp_diag.ko 2022-05-13T22:46:51.697 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/ipv4/udp_tunnel.ko 2022-05-13T22:46:51.697 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/ipv4/xfrm4_tunnel.ko 2022-05-13T22:46:51.697 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/ipv6 2022-05-13T22:46:51.698 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/ipv6/ah6.ko 2022-05-13T22:46:51.698 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/ipv6/esp6.ko 2022-05-13T22:46:51.699 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/ipv6/fou6.ko 2022-05-13T22:46:51.699 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/ipv6/ip6_tunnel.ko 2022-05-13T22:46:51.699 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/ipv6/ip6_udp_tunnel.ko 2022-05-13T22:46:51.700 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/ipv6/ip6_vti.ko 2022-05-13T22:46:51.700 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/ipv6/ipcomp6.ko 2022-05-13T22:46:51.701 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/ipv6/netfilter 2022-05-13T22:46:51.701 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/ipv6/netfilter/ip6_tables.ko 2022-05-13T22:46:51.701 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/ipv6/netfilter/ip6t_REJECT.ko 2022-05-13T22:46:51.702 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/ipv6/netfilter/ip6t_SYNPROXY.ko 2022-05-13T22:46:51.702 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/ipv6/netfilter/ip6t_ah.ko 2022-05-13T22:46:51.703 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/ipv6/netfilter/ip6t_eui64.ko 2022-05-13T22:46:51.703 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/ipv6/netfilter/ip6t_frag.ko 2022-05-13T22:46:51.703 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/ipv6/netfilter/ip6t_hbh.ko 2022-05-13T22:46:51.704 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/ipv6/netfilter/ip6t_ipv6header.ko 2022-05-13T22:46:51.704 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/ipv6/netfilter/ip6t_mh.ko 2022-05-13T22:46:51.704 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/ipv6/netfilter/ip6t_rpfilter.ko 2022-05-13T22:46:51.705 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/ipv6/netfilter/ip6t_rt.ko 2022-05-13T22:46:51.705 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/ipv6/netfilter/ip6table_filter.ko 2022-05-13T22:46:51.706 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/ipv6/netfilter/ip6table_mangle.ko 2022-05-13T22:46:51.706 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/ipv6/netfilter/ip6table_nat.ko 2022-05-13T22:46:51.706 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/ipv6/netfilter/ip6table_raw.ko 2022-05-13T22:46:51.706 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/ipv6/netfilter/ip6table_security.ko 2022-05-13T22:46:51.707 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/ipv6/netfilter/nf_defrag_ipv6.ko 2022-05-13T22:46:51.707 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/ipv6/netfilter/nf_dup_ipv6.ko 2022-05-13T22:46:51.708 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/ipv6/netfilter/nf_reject_ipv6.ko 2022-05-13T22:46:51.708 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/ipv6/netfilter/nf_socket_ipv6.ko 2022-05-13T22:46:51.709 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/ipv6/netfilter/nf_tproxy_ipv6.ko 2022-05-13T22:46:51.709 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/ipv6/netfilter/nft_dup_ipv6.ko 2022-05-13T22:46:51.709 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/ipv6/netfilter/nft_fib_ipv6.ko 2022-05-13T22:46:51.710 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/ipv6/netfilter/nft_reject_ipv6.ko 2022-05-13T22:46:51.710 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/ipv6/sit.ko 2022-05-13T22:46:51.710 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/ipv6/tunnel6.ko 2022-05-13T22:46:51.711 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/ipv6/xfrm6_tunnel.ko 2022-05-13T22:46:51.711 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/key 2022-05-13T22:46:51.711 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/key/af_key.ko 2022-05-13T22:46:51.712 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/l2tp 2022-05-13T22:46:51.712 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/l2tp/l2tp_core.ko 2022-05-13T22:46:51.712 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/l2tp/l2tp_debugfs.ko 2022-05-13T22:46:51.713 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/l2tp/l2tp_eth.ko 2022-05-13T22:46:51.713 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/l2tp/l2tp_ip.ko 2022-05-13T22:46:51.713 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/l2tp/l2tp_ip6.ko 2022-05-13T22:46:51.714 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/l2tp/l2tp_netlink.ko 2022-05-13T22:46:51.714 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/l2tp/l2tp_ppp.ko 2022-05-13T22:46:51.715 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/llc 2022-05-13T22:46:51.715 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/llc/llc.ko 2022-05-13T22:46:51.715 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/mac802154 2022-05-13T22:46:51.716 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/mac802154/mac802154.ko 2022-05-13T22:46:51.716 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/mpls 2022-05-13T22:46:51.716 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/mpls/mpls_gso.ko 2022-05-13T22:46:51.716 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/netfilter 2022-05-13T22:46:51.717 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/netfilter/ipset 2022-05-13T22:46:51.717 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/netfilter/ipset/ip_set.ko 2022-05-13T22:46:51.717 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/netfilter/ipset/ip_set_bitmap_ip.ko 2022-05-13T22:46:51.717 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/netfilter/ipset/ip_set_bitmap_ipmac.ko 2022-05-13T22:46:51.718 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/netfilter/ipset/ip_set_bitmap_port.ko 2022-05-13T22:46:51.718 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/netfilter/ipset/ip_set_hash_ip.ko 2022-05-13T22:46:51.718 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/netfilter/ipset/ip_set_hash_ipmark.ko 2022-05-13T22:46:51.718 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/netfilter/ipset/ip_set_hash_ipport.ko 2022-05-13T22:46:51.719 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/netfilter/ipset/ip_set_hash_ipportip.ko 2022-05-13T22:46:51.719 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/netfilter/ipset/ip_set_hash_ipportnet.ko 2022-05-13T22:46:51.719 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/netfilter/ipset/ip_set_hash_mac.ko 2022-05-13T22:46:51.719 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/netfilter/ipset/ip_set_hash_net.ko 2022-05-13T22:46:51.719 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/netfilter/ipset/ip_set_hash_netiface.ko 2022-05-13T22:46:51.720 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/netfilter/ipset/ip_set_hash_netnet.ko 2022-05-13T22:46:51.720 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/netfilter/ipset/ip_set_hash_netport.ko 2022-05-13T22:46:51.720 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/netfilter/ipset/ip_set_hash_netportnet.ko 2022-05-13T22:46:51.720 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/netfilter/ipset/ip_set_list_set.ko 2022-05-13T22:46:51.720 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/netfilter/ipvs 2022-05-13T22:46:51.721 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/netfilter/ipvs/ip_vs.ko 2022-05-13T22:46:51.721 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/netfilter/ipvs/ip_vs_dh.ko 2022-05-13T22:46:51.721 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/netfilter/ipvs/ip_vs_fo.ko 2022-05-13T22:46:51.722 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/netfilter/ipvs/ip_vs_ftp.ko 2022-05-13T22:46:51.722 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/netfilter/ipvs/ip_vs_lblc.ko 2022-05-13T22:46:51.722 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/netfilter/ipvs/ip_vs_lblcr.ko 2022-05-13T22:46:51.722 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/netfilter/ipvs/ip_vs_lc.ko 2022-05-13T22:46:51.723 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/netfilter/ipvs/ip_vs_nq.ko 2022-05-13T22:46:51.723 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/netfilter/ipvs/ip_vs_pe_sip.ko 2022-05-13T22:46:51.723 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/netfilter/ipvs/ip_vs_rr.ko 2022-05-13T22:46:51.723 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/netfilter/ipvs/ip_vs_sed.ko 2022-05-13T22:46:51.723 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/netfilter/ipvs/ip_vs_sh.ko 2022-05-13T22:46:51.724 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/netfilter/ipvs/ip_vs_wlc.ko 2022-05-13T22:46:51.724 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/netfilter/ipvs/ip_vs_wrr.ko 2022-05-13T22:46:51.724 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/netfilter/nf_conncount.ko 2022-05-13T22:46:51.724 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/netfilter/nf_conntrack.ko 2022-05-13T22:46:51.725 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/netfilter/nf_conntrack_amanda.ko 2022-05-13T22:46:51.725 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/netfilter/nf_conntrack_broadcast.ko 2022-05-13T22:46:51.725 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/netfilter/nf_conntrack_ftp.ko 2022-05-13T22:46:51.725 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/netfilter/nf_conntrack_h323.ko 2022-05-13T22:46:51.726 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/netfilter/nf_conntrack_irc.ko 2022-05-13T22:46:51.726 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/netfilter/nf_conntrack_netbios_ns.ko 2022-05-13T22:46:51.726 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/netfilter/nf_conntrack_netlink.ko 2022-05-13T22:46:51.726 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/netfilter/nf_conntrack_pptp.ko 2022-05-13T22:46:51.726 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/netfilter/nf_conntrack_sane.ko 2022-05-13T22:46:51.727 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/netfilter/nf_conntrack_sip.ko 2022-05-13T22:46:51.727 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/netfilter/nf_conntrack_snmp.ko 2022-05-13T22:46:51.727 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/netfilter/nf_conntrack_tftp.ko 2022-05-13T22:46:51.727 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/netfilter/nf_dup_netdev.ko 2022-05-13T22:46:51.728 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/netfilter/nf_flow_table.ko 2022-05-13T22:46:51.728 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/netfilter/nf_flow_table_inet.ko 2022-05-13T22:46:51.728 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/netfilter/nf_log_syslog.ko 2022-05-13T22:46:51.728 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/netfilter/nf_nat.ko 2022-05-13T22:46:51.728 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/netfilter/nf_nat_amanda.ko 2022-05-13T22:46:51.729 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/netfilter/nf_nat_ftp.ko 2022-05-13T22:46:51.729 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/netfilter/nf_nat_irc.ko 2022-05-13T22:46:51.729 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/netfilter/nf_nat_sip.ko 2022-05-13T22:46:51.729 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/netfilter/nf_nat_tftp.ko 2022-05-13T22:46:51.730 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/netfilter/nf_synproxy_core.ko 2022-05-13T22:46:51.730 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/netfilter/nf_tables.ko 2022-05-13T22:46:51.730 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/netfilter/nfnetlink.ko 2022-05-13T22:46:51.730 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/netfilter/nfnetlink_acct.ko 2022-05-13T22:46:51.731 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/netfilter/nfnetlink_log.ko 2022-05-13T22:46:51.731 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/netfilter/nfnetlink_osf.ko 2022-05-13T22:46:51.731 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/netfilter/nfnetlink_queue.ko 2022-05-13T22:46:51.731 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/netfilter/nft_chain_nat.ko 2022-05-13T22:46:51.732 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/netfilter/nft_compat.ko 2022-05-13T22:46:51.732 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/netfilter/nft_ct.ko 2022-05-13T22:46:51.732 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/netfilter/nft_dup_netdev.ko 2022-05-13T22:46:51.732 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/netfilter/nft_fib.ko 2022-05-13T22:46:51.733 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/netfilter/nft_fwd_netdev.ko 2022-05-13T22:46:51.733 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/netfilter/nft_hash.ko 2022-05-13T22:46:51.733 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/netfilter/nft_limit.ko 2022-05-13T22:46:51.733 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/netfilter/nft_log.ko 2022-05-13T22:46:51.733 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/netfilter/nft_masq.ko 2022-05-13T22:46:51.734 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/netfilter/nft_nat.ko 2022-05-13T22:46:51.734 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/netfilter/nft_numgen.ko 2022-05-13T22:46:51.734 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/netfilter/nft_objref.ko 2022-05-13T22:46:51.734 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/netfilter/nft_queue.ko 2022-05-13T22:46:51.734 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/netfilter/nft_quota.ko 2022-05-13T22:46:51.735 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/netfilter/nft_redir.ko 2022-05-13T22:46:51.735 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/netfilter/nft_reject.ko 2022-05-13T22:46:51.735 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/netfilter/nft_reject_inet.ko 2022-05-13T22:46:51.735 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/netfilter/nft_socket.ko 2022-05-13T22:46:51.735 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/netfilter/nft_tunnel.ko 2022-05-13T22:46:51.736 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/netfilter/xt_AUDIT.ko 2022-05-13T22:46:51.736 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/netfilter/xt_CHECKSUM.ko 2022-05-13T22:46:51.736 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/netfilter/xt_CLASSIFY.ko 2022-05-13T22:46:51.736 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/netfilter/xt_CONNSECMARK.ko 2022-05-13T22:46:51.736 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/netfilter/xt_CT.ko 2022-05-13T22:46:51.737 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/netfilter/xt_DSCP.ko 2022-05-13T22:46:51.737 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/netfilter/xt_HL.ko 2022-05-13T22:46:51.737 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/netfilter/xt_HMARK.ko 2022-05-13T22:46:51.737 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/netfilter/xt_IDLETIMER.ko 2022-05-13T22:46:51.738 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/netfilter/xt_LED.ko 2022-05-13T22:46:51.738 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/netfilter/xt_LOG.ko 2022-05-13T22:46:51.738 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/netfilter/xt_MASQUERADE.ko 2022-05-13T22:46:51.738 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/netfilter/xt_NETMAP.ko 2022-05-13T22:46:51.738 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/netfilter/xt_NFLOG.ko 2022-05-13T22:46:51.739 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/netfilter/xt_NFQUEUE.ko 2022-05-13T22:46:51.739 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/netfilter/xt_RATEEST.ko 2022-05-13T22:46:51.739 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/netfilter/xt_REDIRECT.ko 2022-05-13T22:46:51.739 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/netfilter/xt_SECMARK.ko 2022-05-13T22:46:51.739 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/netfilter/xt_TCPMSS.ko 2022-05-13T22:46:51.740 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/netfilter/xt_TCPOPTSTRIP.ko 2022-05-13T22:46:51.740 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/netfilter/xt_TEE.ko 2022-05-13T22:46:51.740 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/netfilter/xt_TPROXY.ko 2022-05-13T22:46:51.740 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/netfilter/xt_TRACE.ko 2022-05-13T22:46:51.740 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/netfilter/xt_addrtype.ko 2022-05-13T22:46:51.741 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/netfilter/xt_bpf.ko 2022-05-13T22:46:51.741 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/netfilter/xt_cgroup.ko 2022-05-13T22:46:51.741 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/netfilter/xt_cluster.ko 2022-05-13T22:46:51.741 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/netfilter/xt_comment.ko 2022-05-13T22:46:51.742 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/netfilter/xt_connbytes.ko 2022-05-13T22:46:51.742 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/netfilter/xt_connlabel.ko 2022-05-13T22:46:51.742 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/netfilter/xt_connlimit.ko 2022-05-13T22:46:51.743 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/netfilter/xt_connmark.ko 2022-05-13T22:46:51.743 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/netfilter/xt_conntrack.ko 2022-05-13T22:46:51.743 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/netfilter/xt_cpu.ko 2022-05-13T22:46:51.743 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/netfilter/xt_dccp.ko 2022-05-13T22:46:51.743 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/netfilter/xt_devgroup.ko 2022-05-13T22:46:51.744 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/netfilter/xt_dscp.ko 2022-05-13T22:46:51.744 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/netfilter/xt_ecn.ko 2022-05-13T22:46:51.744 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/netfilter/xt_esp.ko 2022-05-13T22:46:51.744 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/netfilter/xt_hashlimit.ko 2022-05-13T22:46:51.745 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/netfilter/xt_helper.ko 2022-05-13T22:46:51.745 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/netfilter/xt_hl.ko 2022-05-13T22:46:51.745 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/netfilter/xt_ipcomp.ko 2022-05-13T22:46:51.745 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/netfilter/xt_iprange.ko 2022-05-13T22:46:51.745 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/netfilter/xt_ipvs.ko 2022-05-13T22:46:51.746 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/netfilter/xt_l2tp.ko 2022-05-13T22:46:51.746 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/netfilter/xt_length.ko 2022-05-13T22:46:51.746 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/netfilter/xt_limit.ko 2022-05-13T22:46:51.746 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/netfilter/xt_mac.ko 2022-05-13T22:46:51.746 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/netfilter/xt_mark.ko 2022-05-13T22:46:51.747 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/netfilter/xt_multiport.ko 2022-05-13T22:46:51.747 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/netfilter/xt_nat.ko 2022-05-13T22:46:51.747 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/netfilter/xt_nfacct.ko 2022-05-13T22:46:51.747 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/netfilter/xt_osf.ko 2022-05-13T22:46:51.747 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/netfilter/xt_owner.ko 2022-05-13T22:46:51.748 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/netfilter/xt_physdev.ko 2022-05-13T22:46:51.748 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/netfilter/xt_pkttype.ko 2022-05-13T22:46:51.748 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/netfilter/xt_policy.ko 2022-05-13T22:46:51.748 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/netfilter/xt_quota.ko 2022-05-13T22:46:51.748 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/netfilter/xt_rateest.ko 2022-05-13T22:46:51.749 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/netfilter/xt_realm.ko 2022-05-13T22:46:51.749 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/netfilter/xt_recent.ko 2022-05-13T22:46:51.749 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/netfilter/xt_sctp.ko 2022-05-13T22:46:51.749 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/netfilter/xt_set.ko 2022-05-13T22:46:51.749 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/netfilter/xt_socket.ko 2022-05-13T22:46:51.750 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/netfilter/xt_state.ko 2022-05-13T22:46:51.750 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/netfilter/xt_statistic.ko 2022-05-13T22:46:51.750 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/netfilter/xt_string.ko 2022-05-13T22:46:51.750 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/netfilter/xt_tcpmss.ko 2022-05-13T22:46:51.751 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/netfilter/xt_time.ko 2022-05-13T22:46:51.751 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/netfilter/xt_u32.ko 2022-05-13T22:46:51.751 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/netlink 2022-05-13T22:46:51.751 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/netlink/netlink_diag.ko 2022-05-13T22:46:51.751 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/nsh 2022-05-13T22:46:51.752 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/nsh/nsh.ko 2022-05-13T22:46:51.752 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/openvswitch 2022-05-13T22:46:51.752 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/openvswitch/openvswitch.ko 2022-05-13T22:46:51.752 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/openvswitch/vport-geneve.ko 2022-05-13T22:46:51.752 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/openvswitch/vport-gre.ko 2022-05-13T22:46:51.753 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/openvswitch/vport-vxlan.ko 2022-05-13T22:46:51.753 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/packet 2022-05-13T22:46:51.753 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/packet/af_packet_diag.ko 2022-05-13T22:46:51.753 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/rds 2022-05-13T22:46:51.753 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/rds/rds.ko 2022-05-13T22:46:51.754 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/rds/rds_rdma.ko 2022-05-13T22:46:51.754 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/rds/rds_tcp.ko 2022-05-13T22:46:51.754 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/sched 2022-05-13T22:46:51.754 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/sched/act_csum.ko 2022-05-13T22:46:51.755 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/sched/act_gact.ko 2022-05-13T22:46:51.755 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/sched/act_ipt.ko 2022-05-13T22:46:51.755 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/sched/act_mirred.ko 2022-05-13T22:46:51.755 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/sched/act_nat.ko 2022-05-13T22:46:51.755 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/sched/act_pedit.ko 2022-05-13T22:46:51.756 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/sched/act_police.ko 2022-05-13T22:46:51.756 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/sched/act_simple.ko 2022-05-13T22:46:51.756 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/sched/act_skbedit.ko 2022-05-13T22:46:51.756 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/sched/act_vlan.ko 2022-05-13T22:46:51.757 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/sched/cls_basic.ko 2022-05-13T22:46:51.757 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/sched/cls_bpf.ko 2022-05-13T22:46:51.757 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/sched/cls_flow.ko 2022-05-13T22:46:51.757 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/sched/cls_fw.ko 2022-05-13T22:46:51.758 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/sched/cls_route.ko 2022-05-13T22:46:51.758 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/sched/cls_rsvp.ko 2022-05-13T22:46:51.758 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/sched/cls_rsvp6.ko 2022-05-13T22:46:51.758 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/sched/cls_tcindex.ko 2022-05-13T22:46:51.758 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/sched/cls_u32.ko 2022-05-13T22:46:51.759 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/sched/em_cmp.ko 2022-05-13T22:46:51.759 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/sched/em_ipset.ko 2022-05-13T22:46:51.759 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/sched/em_meta.ko 2022-05-13T22:46:51.759 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/sched/em_nbyte.ko 2022-05-13T22:46:51.759 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/sched/em_text.ko 2022-05-13T22:46:51.760 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/sched/em_u32.ko 2022-05-13T22:46:51.760 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/sched/sch_cbq.ko 2022-05-13T22:46:51.760 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/sched/sch_choke.ko 2022-05-13T22:46:51.760 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/sched/sch_codel.ko 2022-05-13T22:46:51.760 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/sched/sch_drr.ko 2022-05-13T22:46:51.761 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/sched/sch_dsmark.ko 2022-05-13T22:46:51.761 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/sched/sch_fq.ko 2022-05-13T22:46:51.761 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/sched/sch_gred.ko 2022-05-13T22:46:51.761 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/sched/sch_hfsc.ko 2022-05-13T22:46:51.762 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/sched/sch_hhf.ko 2022-05-13T22:46:51.762 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/sched/sch_htb.ko 2022-05-13T22:46:51.762 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/sched/sch_ingress.ko 2022-05-13T22:46:51.762 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/sched/sch_mqprio.ko 2022-05-13T22:46:51.762 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/sched/sch_multiq.ko 2022-05-13T22:46:51.763 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/sched/sch_netem.ko 2022-05-13T22:46:51.763 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/sched/sch_pie.ko 2022-05-13T22:46:51.763 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/sched/sch_plug.ko 2022-05-13T22:46:51.763 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/sched/sch_prio.ko 2022-05-13T22:46:51.763 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/sched/sch_qfq.ko 2022-05-13T22:46:51.764 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/sched/sch_red.ko 2022-05-13T22:46:51.764 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/sched/sch_sfb.ko 2022-05-13T22:46:51.764 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/sched/sch_sfq.ko 2022-05-13T22:46:51.764 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/sched/sch_tbf.ko 2022-05-13T22:46:51.764 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/sched/sch_teql.ko 2022-05-13T22:46:51.765 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/sctp 2022-05-13T22:46:51.765 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/sctp/sctp.ko 2022-05-13T22:46:51.765 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/sctp/sctp_diag.ko 2022-05-13T22:46:51.765 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/sunrpc 2022-05-13T22:46:51.765 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/sunrpc/auth_gss 2022-05-13T22:46:51.766 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/sunrpc/auth_gss/auth_rpcgss.ko 2022-05-13T22:46:51.766 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/sunrpc/auth_gss/rpcsec_gss_krb5.ko 2022-05-13T22:46:51.766 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/sunrpc/sunrpc.ko 2022-05-13T22:46:51.766 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/sunrpc/xprtrdma 2022-05-13T22:46:51.766 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/sunrpc/xprtrdma/rpcrdma.ko 2022-05-13T22:46:51.767 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/tipc 2022-05-13T22:46:51.767 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/tipc/diag.ko 2022-05-13T22:46:51.767 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/tipc/tipc.ko 2022-05-13T22:46:51.767 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/unix 2022-05-13T22:46:51.768 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/unix/unix_diag.ko 2022-05-13T22:46:51.768 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/vmw_vsock 2022-05-13T22:46:51.768 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/vmw_vsock/vmw_vsock_virtio_transport_common.ko 2022-05-13T22:46:51.768 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/vmw_vsock/vmw_vsock_vmci_transport.ko 2022-05-13T22:46:51.768 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/vmw_vsock/vsock.ko 2022-05-13T22:46:51.769 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/vmw_vsock/vsock_diag.ko 2022-05-13T22:46:51.769 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/vmw_vsock/vsock_loopback.ko 2022-05-13T22:46:51.769 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/xfrm 2022-05-13T22:46:51.769 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/net/xfrm/xfrm_ipcomp.ko 2022-05-13T22:46:51.769 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/security 2022-05-13T22:46:51.770 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/security/keys 2022-05-13T22:46:51.770 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/security/keys/encrypted-keys 2022-05-13T22:46:51.770 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/security/keys/encrypted-keys/encrypted-keys.ko 2022-05-13T22:46:51.770 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/security/keys/trusted-keys 2022-05-13T22:46:51.770 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/security/keys/trusted-keys/trusted.ko 2022-05-13T22:46:51.771 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/virt 2022-05-13T22:46:51.771 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/virt/lib 2022-05-13T22:46:51.771 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/kernel/virt/lib/irqbypass.ko 2022-05-13T22:46:51.771 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/modules.alias 2022-05-13T22:46:51.771 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/modules.alias.bin 2022-05-13T22:46:51.772 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/modules.builtin 2022-05-13T22:46:51.772 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/modules.builtin.bin 2022-05-13T22:46:51.772 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/modules.builtin.modinfo 2022-05-13T22:46:51.772 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/modules.dep 2022-05-13T22:46:51.772 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/modules.dep.bin 2022-05-13T22:46:51.773 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/modules.devname 2022-05-13T22:46:51.773 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/modules.order 2022-05-13T22:46:51.773 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/modules.softdep 2022-05-13T22:46:51.773 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/modules.symbols 2022-05-13T22:46:51.773 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/5.18.0-rc6-ceph-gb850759e370d/modules.symbols.bin 2022-05-13T22:46:51.774 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/lib/.build-id 2022-05-13T22:46:51.774 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/lib/.build-id/df 2022-05-13T22:46:51.774 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/lib/.build-id/df/c7d577461eb3bc1a44b0f476c58eb0c5e266f8 2022-05-13T22:46:51.775 DEBUG:teuthology.task.kernel:get_image_version: 5.18.0-rc6-ceph-gb850759e370d 2022-05-13T22:46:51.776 DEBUG:teuthology.orchestra.run.smithi002:> rpm --scripts -qp /tmp/kernel.x86_64.rpm 2022-05-13T22:46:51.801 INFO:teuthology.orchestra.run.smithi002.stdout:postinstall scriptlet (using /bin/sh): 2022-05-13T22:46:51.801 INFO:teuthology.orchestra.run.smithi002.stdout:if [ -x /sbin/installkernel -a -r /boot/vmlinuz-5.18.0-rc6-ceph-gb850759e370d -a -r /boot/System.map-5.18.0-rc6-ceph-gb850759e370d ]; then 2022-05-13T22:46:51.801 INFO:teuthology.orchestra.run.smithi002.stdout:cp /boot/vmlinuz-5.18.0-rc6-ceph-gb850759e370d /boot/.vmlinuz-5.18.0-rc6-ceph-gb850759e370d-rpm 2022-05-13T22:46:51.802 INFO:teuthology.orchestra.run.smithi002.stdout:cp /boot/System.map-5.18.0-rc6-ceph-gb850759e370d /boot/.System.map-5.18.0-rc6-ceph-gb850759e370d-rpm 2022-05-13T22:46:51.802 INFO:teuthology.orchestra.run.smithi002.stdout:rm -f /boot/vmlinuz-5.18.0-rc6-ceph-gb850759e370d /boot/System.map-5.18.0-rc6-ceph-gb850759e370d 2022-05-13T22:46:51.802 INFO:teuthology.orchestra.run.smithi002.stdout:/sbin/installkernel 5.18.0-rc6-ceph-gb850759e370d /boot/.vmlinuz-5.18.0-rc6-ceph-gb850759e370d-rpm /boot/.System.map-5.18.0-rc6-ceph-gb850759e370d-rpm 2022-05-13T22:46:51.803 INFO:teuthology.orchestra.run.smithi002.stdout:rm -f /boot/.vmlinuz-5.18.0-rc6-ceph-gb850759e370d-rpm /boot/.System.map-5.18.0-rc6-ceph-gb850759e370d-rpm 2022-05-13T22:46:51.803 INFO:teuthology.orchestra.run.smithi002.stdout:fi 2022-05-13T22:46:51.803 INFO:teuthology.orchestra.run.smithi002.stdout:preuninstall scriptlet (using /bin/sh): 2022-05-13T22:46:51.803 INFO:teuthology.orchestra.run.smithi002.stdout:if [ -x /sbin/new-kernel-pkg ]; then 2022-05-13T22:46:51.803 INFO:teuthology.orchestra.run.smithi002.stdout:new-kernel-pkg --remove 5.18.0-rc6-ceph-gb850759e370d --rminitrd --initrdfile=/boot/initramfs-5.18.0-rc6-ceph-gb850759e370d.img 2022-05-13T22:46:51.804 INFO:teuthology.orchestra.run.smithi002.stdout:elif [ -x /usr/bin/kernel-install ]; then 2022-05-13T22:46:51.804 INFO:teuthology.orchestra.run.smithi002.stdout:kernel-install remove 5.18.0-rc6-ceph-gb850759e370d 2022-05-13T22:46:51.804 INFO:teuthology.orchestra.run.smithi002.stdout:fi 2022-05-13T22:46:51.804 INFO:teuthology.orchestra.run.smithi002.stdout:postuninstall scriptlet (using /bin/sh): 2022-05-13T22:46:51.804 INFO:teuthology.orchestra.run.smithi002.stdout:if [ -x /sbin/update-bootloader ]; then 2022-05-13T22:46:51.805 INFO:teuthology.orchestra.run.smithi002.stdout:/sbin/update-bootloader --remove 5.18.0-rc6-ceph-gb850759e370d 2022-05-13T22:46:51.805 INFO:teuthology.orchestra.run.smithi002.stdout:fi 2022-05-13T22:46:51.806 DEBUG:teuthology.orchestra.run.smithi002:> sudo rpm -qi grub2-tools 2022-05-13T22:46:51.882 INFO:teuthology.orchestra.run.smithi002.stdout:Name : grub2-tools 2022-05-13T22:46:51.882 INFO:teuthology.orchestra.run.smithi002.stdout:Epoch : 1 2022-05-13T22:46:51.883 INFO:teuthology.orchestra.run.smithi002.stdout:Version : 2.02 2022-05-13T22:46:51.883 INFO:teuthology.orchestra.run.smithi002.stdout:Release : 99.el8 2022-05-13T22:46:51.883 INFO:teuthology.orchestra.run.smithi002.stdout:Architecture: x86_64 2022-05-13T22:46:51.883 INFO:teuthology.orchestra.run.smithi002.stdout:Install Date: Sun 08 May 2022 12:09:30 PM UTC 2022-05-13T22:46:51.884 INFO:teuthology.orchestra.run.smithi002.stdout:Group : System Environment/Base 2022-05-13T22:46:51.884 INFO:teuthology.orchestra.run.smithi002.stdout:Size : 9264713 2022-05-13T22:46:51.884 INFO:teuthology.orchestra.run.smithi002.stdout:License : GPLv3+ 2022-05-13T22:46:51.884 INFO:teuthology.orchestra.run.smithi002.stdout:Signature : RSA/SHA256, Tue 02 Mar 2021 08:11:27 PM UTC, Key ID 199e2f91fd431d51 2022-05-13T22:46:51.885 INFO:teuthology.orchestra.run.smithi002.stdout:Source RPM : grub2-2.02-99.el8.src.rpm 2022-05-13T22:46:51.885 INFO:teuthology.orchestra.run.smithi002.stdout:Build Date : Thu 25 Feb 2021 04:59:31 PM UTC 2022-05-13T22:46:51.885 INFO:teuthology.orchestra.run.smithi002.stdout:Build Host : x86-vm-07.build.eng.bos.redhat.com 2022-05-13T22:46:51.885 INFO:teuthology.orchestra.run.smithi002.stdout:Relocations : (not relocatable) 2022-05-13T22:46:51.885 INFO:teuthology.orchestra.run.smithi002.stdout:Packager : Red Hat, Inc. 2022-05-13T22:46:51.886 INFO:teuthology.orchestra.run.smithi002.stdout:Vendor : Red Hat, Inc. 2022-05-13T22:46:51.886 INFO:teuthology.orchestra.run.smithi002.stdout:URL : http://www.gnu.org/software/grub/ 2022-05-13T22:46:51.886 INFO:teuthology.orchestra.run.smithi002.stdout:Summary : Support tools for GRUB. 2022-05-13T22:46:51.886 INFO:teuthology.orchestra.run.smithi002.stdout:Description : 2022-05-13T22:46:51.887 INFO:teuthology.orchestra.run.smithi002.stdout: 2022-05-13T22:46:51.887 INFO:teuthology.orchestra.run.smithi002.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2022-05-13T22:46:51.887 INFO:teuthology.orchestra.run.smithi002.stdout:customizable bootloader with modular architecture. It supports a rich 2022-05-13T22:46:51.887 INFO:teuthology.orchestra.run.smithi002.stdout:variety of kernel formats, file systems, computer architectures and 2022-05-13T22:46:51.887 INFO:teuthology.orchestra.run.smithi002.stdout:hardware devices. 2022-05-13T22:46:51.888 INFO:teuthology.orchestra.run.smithi002.stdout: 2022-05-13T22:46:51.888 INFO:teuthology.orchestra.run.smithi002.stdout:This subpackage provides tools for support of all platforms. 2022-05-13T22:46:51.889 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2022-05-13T22:46:51.889 INFO:teuthology.task.kernel:Updating grub on smithi002 to boot 5.18.0-rc6-ceph-gb850759e370d 2022-05-13T22:46:51.890 DEBUG:teuthology.orchestra.run.smithi002:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2022-05-13T22:46:53.956 INFO:teuthology.orchestra.run.smithi002.stderr:Generating grub configuration file ... 2022-05-13T22:46:56.037 INFO:teuthology.orchestra.run.smithi002.stderr:done 2022-05-13T22:46:56.040 DEBUG:teuthology.orchestra.run.smithi002:> mktemp 2022-05-13T22:46:56.056 INFO:teuthology.orchestra.run.smithi002.stdout:/tmp/tmp.ua1EcDcK8X 2022-05-13T22:46:56.056 DEBUG:teuthology.orchestra.run.smithi002:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.ua1EcDcK8X 2022-05-13T22:46:56.125 DEBUG:teuthology.orchestra.run.smithi002:> sudo chmod 0666 /tmp/tmp.ua1EcDcK8X 2022-05-13T22:46:56.288 DEBUG:teuthology.orchestra.remote:smithi002:/tmp/tmp.ua1EcDcK8X is 6KB 2022-05-13T22:46:56.337 DEBUG:teuthology.orchestra.run.smithi002:> rm -fr /tmp/tmp.ua1EcDcK8X 2022-05-13T22:46:56.352 DEBUG:teuthology.orchestra.run.smithi002:> sudo /bin/ls /boot/loader/entries || true 2022-05-13T22:46:56.416 INFO:teuthology.orchestra.run.smithi002.stdout:28be16ddee7c46bab2727ec8a3fb5401-0-rescue.conf 2022-05-13T22:46:56.417 INFO:teuthology.orchestra.run.smithi002.stdout:28be16ddee7c46bab2727ec8a3fb5401-4.18.0-305.el8.x86_64.conf 2022-05-13T22:46:56.417 INFO:teuthology.orchestra.run.smithi002.stdout:28be16ddee7c46bab2727ec8a3fb5401-4.18.0-348.23.1.el8_5.x86_64.conf 2022-05-13T22:46:56.417 INFO:teuthology.orchestra.run.smithi002.stdout:28be16ddee7c46bab2727ec8a3fb5401-5.18.0-rc6-ceph-gb850759e370d.conf 2022-05-13T22:46:56.419 DEBUG:teuthology.orchestra.run.smithi002:> sudo grub2-set-default 28be16ddee7c46bab2727ec8a3fb5401-5.18.0-rc6-ceph-gb850759e370d 2022-05-13T22:46:56.576 DEBUG:teuthology.orchestra.run.smithi002:> sudo shutdown -r now 2022-05-13T22:47:26.579 INFO:teuthology.task.kernel:Checking client client.0 for new kernel version... 2022-05-13T22:47:26.580 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi002.front.sepia.ceph.com' 2022-05-13T22:47:26.581 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi002.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-05-13T22:48:26.583 DEBUG:teuthology.orchestra.remote:timed out 2022-05-13T22:48:56.593 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi002.front.sepia.ceph.com' 2022-05-13T22:48:56.594 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi002.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-05-13T22:48:57.004 DEBUG:teuthology.orchestra.run.smithi002:> true 2022-05-13T22:48:57.429 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi002.front.sepia.ceph.com' 2022-05-13T22:48:57.430 INFO:teuthology.task.kernel:Checking kernel version of client.0, want "5.18.0-rc6-ceph-gb850759e370d"... 2022-05-13T22:48:57.430 DEBUG:teuthology.orchestra.run.smithi002:> uname -r 2022-05-13T22:48:57.480 INFO:teuthology.orchestra.run.smithi002.stdout:5.18.0-rc6-ceph-gb850759e370d 2022-05-13T22:48:57.481 DEBUG:teuthology.task.kernel:current kernel version is 5.18.0-rc6-ceph-gb850759e370d vs 5.18.0-rc6-ceph-gb850759e370d 2022-05-13T22:48:57.481 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2022-05-13T22:48:57.481 DEBUG:teuthology.task.kernel:Distro of this test job: rhel 2022-05-13T22:48:58.483 DEBUG:teuthology.parallel:result is None 2022-05-13T22:48:58.484 INFO:teuthology.run_tasks:Running task internal.base... 2022-05-13T22:48:58.494 INFO:teuthology.task.internal:Creating test directory... 2022-05-13T22:48:58.495 DEBUG:teuthology.orchestra.run.smithi002:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2022-05-13T22:48:58.502 DEBUG:teuthology.orchestra.run.smithi047:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2022-05-13T22:48:58.506 DEBUG:teuthology.orchestra.run.smithi115:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2022-05-13T22:48:58.548 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2022-05-13T22:48:58.581 INFO:teuthology.run_tasks:Running task internal.archive... 2022-05-13T22:48:58.611 INFO:teuthology.task.internal:Creating archive directory... 2022-05-13T22:48:58.613 DEBUG:teuthology.orchestra.run.smithi002:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2022-05-13T22:48:58.619 DEBUG:teuthology.orchestra.run.smithi047:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2022-05-13T22:48:58.622 DEBUG:teuthology.orchestra.run.smithi115:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2022-05-13T22:48:58.693 INFO:teuthology.run_tasks:Running task internal.coredump... 2022-05-13T22:48:58.705 INFO:teuthology.task.internal:Enabling coredump saving... 2022-05-13T22:48:58.707 DEBUG:teuthology.orchestra.run.smithi047:> test -f /run/.containerenv -o -f /.dockerenv 2022-05-13T22:48:58.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-05-13T22:48:58.726 DEBUG:teuthology.orchestra.run.smithi115:> test -f /run/.containerenv -o -f /.dockerenv 2022-05-13T22:48:58.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-05-13T22:48:58.743 DEBUG:teuthology.orchestra.run.smithi002:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2022-05-13T22:48:58.750 DEBUG:teuthology.orchestra.run.smithi047:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2022-05-13T22:48:58.768 DEBUG:teuthology.orchestra.run.smithi115:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2022-05-13T22:48:58.812 INFO:teuthology.orchestra.run.smithi047.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2022-05-13T22:48:58.823 INFO:teuthology.orchestra.run.smithi115.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2022-05-13T22:48:58.832 INFO:teuthology.orchestra.run.smithi047.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2022-05-13T22:48:58.837 INFO:teuthology.orchestra.run.smithi115.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2022-05-13T22:48:58.869 INFO:teuthology.orchestra.run.smithi002.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2022-05-13T22:48:58.908 INFO:teuthology.orchestra.run.smithi002.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2022-05-13T22:48:58.913 INFO:teuthology.run_tasks:Running task internal.sudo... 2022-05-13T22:48:58.920 INFO:teuthology.task.internal:Configuring sudo... 2022-05-13T22:48:58.921 DEBUG:teuthology.orchestra.run.smithi002:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2022-05-13T22:48:58.926 DEBUG:teuthology.orchestra.run.smithi047:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2022-05-13T22:48:58.928 DEBUG:teuthology.orchestra.run.smithi115:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2022-05-13T22:48:59.001 INFO:teuthology.run_tasks:Running task internal.syslog... 2022-05-13T22:48:59.010 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2022-05-13T22:48:59.011 DEBUG:teuthology.orchestra.run.smithi002:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2022-05-13T22:48:59.016 DEBUG:teuthology.orchestra.run.smithi047:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2022-05-13T22:48:59.018 DEBUG:teuthology.orchestra.run.smithi115:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2022-05-13T22:48:59.059 DEBUG:teuthology.orchestra.run.smithi002:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2022-05-13T22:48:59.138 DEBUG:teuthology.orchestra.run.smithi002:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2022-05-13T22:48:59.228 DEBUG:teuthology.orchestra.run.smithi002:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2022-05-13T22:48:59.308 DEBUG:teuthology.orchestra.run.smithi002:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2022-05-13T22:48:59.386 DEBUG:teuthology.orchestra.run.smithi002:> set -ex 2022-05-13T22:48:59.386 DEBUG:teuthology.orchestra.run.smithi002:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2022-05-13T22:48:59.464 DEBUG:teuthology.orchestra.run.smithi047:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2022-05-13T22:48:59.515 DEBUG:teuthology.orchestra.run.smithi047:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2022-05-13T22:48:59.566 DEBUG:teuthology.orchestra.run.smithi047:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2022-05-13T22:48:59.657 DEBUG:teuthology.orchestra.run.smithi047:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2022-05-13T22:48:59.698 DEBUG:teuthology.orchestra.run.smithi047:> set -ex 2022-05-13T22:48:59.700 DEBUG:teuthology.orchestra.run.smithi047:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2022-05-13T22:48:59.767 DEBUG:teuthology.orchestra.run.smithi115:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2022-05-13T22:48:59.814 DEBUG:teuthology.orchestra.run.smithi115:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2022-05-13T22:48:59.862 DEBUG:teuthology.orchestra.run.smithi115:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2022-05-13T22:48:59.948 DEBUG:teuthology.orchestra.run.smithi115:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2022-05-13T22:48:59.988 DEBUG:teuthology.orchestra.run.smithi115:> set -ex 2022-05-13T22:48:59.989 DEBUG:teuthology.orchestra.run.smithi115:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2022-05-13T22:49:00.054 DEBUG:teuthology.orchestra.run.smithi002:> sudo service rsyslog restart 2022-05-13T22:49:00.059 DEBUG:teuthology.orchestra.run.smithi047:> sudo service rsyslog restart 2022-05-13T22:49:00.062 DEBUG:teuthology.orchestra.run.smithi115:> sudo service rsyslog restart 2022-05-13T22:49:00.119 INFO:teuthology.orchestra.run.smithi047.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2022-05-13T22:49:00.136 INFO:teuthology.orchestra.run.smithi115.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2022-05-13T22:49:00.193 INFO:teuthology.orchestra.run.smithi002.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2022-05-13T22:49:00.654 INFO:teuthology.run_tasks:Running task internal.timer... 2022-05-13T22:49:00.662 INFO:teuthology.task.internal:Starting timer... 2022-05-13T22:49:00.663 INFO:teuthology.run_tasks:Running task pcp... 2022-05-13T22:49:00.696 INFO:teuthology.run_tasks:Running task selinux... 2022-05-13T22:49:00.720 DEBUG:teuthology.task:Applying overrides for task selinux: {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']} 2022-05-13T22:49:00.721 DEBUG:teuthology.orchestra.run.smithi002:> sudo service auditd rotate 2022-05-13T22:49:00.881 INFO:teuthology.orchestra.run.smithi002.stdout:Rotating logs: [ OK ] 2022-05-13T22:49:00.887 DEBUG:teuthology.orchestra.run.smithi047:> sudo service auditd rotate 2022-05-13T22:49:00.949 INFO:teuthology.orchestra.run.smithi047.stdout:Rotating logs: [ OK ] 2022-05-13T22:49:00.950 DEBUG:teuthology.orchestra.run.smithi115:> sudo service auditd rotate 2022-05-13T22:49:01.008 INFO:teuthology.orchestra.run.smithi115.stdout:Rotating logs: [ OK ] 2022-05-13T22:49:01.010 DEBUG:teuthology.task.selinux:Getting current SELinux state 2022-05-13T22:49:01.011 DEBUG:teuthology.orchestra.run.smithi002:> /usr/sbin/getenforce 2022-05-13T22:49:01.068 INFO:teuthology.orchestra.run.smithi002.stdout:Permissive 2022-05-13T22:49:01.069 DEBUG:teuthology.orchestra.run.smithi047:> /usr/sbin/getenforce 2022-05-13T22:49:01.104 INFO:teuthology.orchestra.run.smithi047.stdout:Permissive 2022-05-13T22:49:01.105 DEBUG:teuthology.orchestra.run.smithi115:> /usr/sbin/getenforce 2022-05-13T22:49:01.129 INFO:teuthology.orchestra.run.smithi115.stdout:Permissive 2022-05-13T22:49:01.130 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi002.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi047.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi115.front.sepia.ceph.com': 'permissive'} 2022-05-13T22:49:01.130 DEBUG:teuthology.orchestra.run.smithi002:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av '\(comm="dmidecode"\|chronyd.service\|name="cephtest"\|scontext=system_u:system_r:nrpe_t:s0\|scontext=system_u:system_r:pcp_pmlogger_t\|scontext=system_u:system_r:pcp_pmcd_t:s0\|comm="rhsmd"\|scontext=system_u:system_r:syslogd_t:s0\|tcontext=system_u:system_r:nrpe_t:s0\|comm="updatedb"\|comm="smartd"\|comm="rhsmcertd-worke"\|comm="setroubleshootd"\|comm="rpm"\|tcontext=system_u:object_r:container_runtime_exec_t:s0\|comm="ksmtuned"\|comm="sssd"\|comm="sss_cache"\|context=system_u:system_r:NetworkManager_dispatcher_t:s0\|scontext=system_u:system_r:logrotate_t:s0\)' 2022-05-13T22:49:01.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-05-13T22:49:01.212 DEBUG:teuthology.orchestra.run.smithi047:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av '\(comm="dmidecode"\|chronyd.service\|name="cephtest"\|scontext=system_u:system_r:nrpe_t:s0\|scontext=system_u:system_r:pcp_pmlogger_t\|scontext=system_u:system_r:pcp_pmcd_t:s0\|comm="rhsmd"\|scontext=system_u:system_r:syslogd_t:s0\|tcontext=system_u:system_r:nrpe_t:s0\|comm="updatedb"\|comm="smartd"\|comm="rhsmcertd-worke"\|comm="setroubleshootd"\|comm="rpm"\|tcontext=system_u:object_r:container_runtime_exec_t:s0\|comm="ksmtuned"\|comm="sssd"\|comm="sss_cache"\|context=system_u:system_r:NetworkManager_dispatcher_t:s0\|scontext=system_u:system_r:logrotate_t:s0\)' 2022-05-13T22:49:01.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-05-13T22:49:01.239 DEBUG:teuthology.orchestra.run.smithi115:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av '\(comm="dmidecode"\|chronyd.service\|name="cephtest"\|scontext=system_u:system_r:nrpe_t:s0\|scontext=system_u:system_r:pcp_pmlogger_t\|scontext=system_u:system_r:pcp_pmcd_t:s0\|comm="rhsmd"\|scontext=system_u:system_r:syslogd_t:s0\|tcontext=system_u:system_r:nrpe_t:s0\|comm="updatedb"\|comm="smartd"\|comm="rhsmcertd-worke"\|comm="setroubleshootd"\|comm="rpm"\|tcontext=system_u:object_r:container_runtime_exec_t:s0\|comm="ksmtuned"\|comm="sssd"\|comm="sss_cache"\|context=system_u:system_r:NetworkManager_dispatcher_t:s0\|scontext=system_u:system_r:logrotate_t:s0\)' 2022-05-13T22:49:01.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-05-13T22:49:01.264 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2022-05-13T22:49:01.265 DEBUG:teuthology.orchestra.run.smithi002:> sudo /usr/sbin/setenforce permissive 2022-05-13T22:49:01.344 DEBUG:teuthology.orchestra.run.smithi047:> sudo /usr/sbin/setenforce permissive 2022-05-13T22:49:01.370 DEBUG:teuthology.orchestra.run.smithi115:> sudo /usr/sbin/setenforce permissive 2022-05-13T22:49:01.408 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2022-05-13T22:49:01.424 INFO:teuthology.repo_utils:/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master was just updated or references a specific commit; assuming it is current 2022-05-13T22:49:01.424 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master to origin/master 2022-05-13T22:49:01.445 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2022-05-13T22:49:01.447 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi002.front.sepia.ceph.com,smithi047.front.sepia.ceph.com,smithi115.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/cephlab.yml 2022-05-13T23:00:55.663 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi002.front.sepia.ceph.com'), Remote(name='ubuntu@smithi047.front.sepia.ceph.com'), Remote(name='ubuntu@smithi115.front.sepia.ceph.com')] 2022-05-13T23:00:55.666 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi002.front.sepia.ceph.com' 2022-05-13T23:00:55.668 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi002.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-05-13T23:00:55.781 DEBUG:teuthology.orchestra.run.smithi002:> true 2022-05-13T23:00:55.894 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi047.front.sepia.ceph.com' 2022-05-13T23:00:55.895 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi047.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-05-13T23:00:55.978 DEBUG:teuthology.orchestra.run.smithi047:> true 2022-05-13T23:00:56.051 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi115.front.sepia.ceph.com' 2022-05-13T23:00:56.051 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi115.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-05-13T23:00:56.134 DEBUG:teuthology.orchestra.run.smithi115:> true 2022-05-13T23:00:56.202 INFO:teuthology.run_tasks:Running task clock... 2022-05-13T23:00:56.220 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2022-05-13T23:00:56.221 INFO:teuthology.orchestra.run:Running command with timeout 360 2022-05-13T23:00:56.224 DEBUG:teuthology.orchestra.run.smithi002:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2022-05-13T23:00:56.230 INFO:teuthology.orchestra.run:Running command with timeout 360 2022-05-13T23:00:56.231 DEBUG:teuthology.orchestra.run.smithi047:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2022-05-13T23:00:56.233 INFO:teuthology.orchestra.run:Running command with timeout 360 2022-05-13T23:00:56.234 DEBUG:teuthology.orchestra.run.smithi115:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2022-05-13T23:00:56.265 INFO:teuthology.orchestra.run.smithi047.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2022-05-13T23:00:56.283 INFO:teuthology.orchestra.run.smithi047.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2022-05-13T23:00:56.284 INFO:teuthology.orchestra.run.smithi115.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2022-05-13T23:00:56.297 INFO:teuthology.orchestra.run.smithi115.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2022-05-13T23:00:56.314 INFO:teuthology.orchestra.run.smithi002.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2022-05-13T23:00:56.323 INFO:teuthology.orchestra.run.smithi047.stderr:sudo: ntpd: command not found 2022-05-13T23:00:56.329 INFO:teuthology.orchestra.run.smithi115.stderr:sudo: ntpd: command not found 2022-05-13T23:00:56.334 INFO:teuthology.orchestra.run.smithi047.stdout:506 Cannot talk to daemon 2022-05-13T23:00:56.339 INFO:teuthology.orchestra.run.smithi115.stdout:506 Cannot talk to daemon 2022-05-13T23:00:56.350 INFO:teuthology.orchestra.run.smithi047.stderr:Failed to start ntp.service: Unit ntp.service not found. 2022-05-13T23:00:56.353 INFO:teuthology.orchestra.run.smithi115.stderr:Failed to start ntp.service: Unit ntp.service not found. 2022-05-13T23:00:56.358 INFO:teuthology.orchestra.run.smithi002.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2022-05-13T23:00:56.365 INFO:teuthology.orchestra.run.smithi047.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2022-05-13T23:00:56.367 INFO:teuthology.orchestra.run.smithi115.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2022-05-13T23:00:56.429 INFO:teuthology.orchestra.run.smithi115.stderr:bash: ntpq: command not found 2022-05-13T23:00:56.432 INFO:teuthology.orchestra.run.smithi115.stdout:210 Number of sources = 4 2022-05-13T23:00:56.432 INFO:teuthology.orchestra.run.smithi115.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2022-05-13T23:00:56.432 INFO:teuthology.orchestra.run.smithi115.stdout:=============================================================================== 2022-05-13T23:00:56.432 INFO:teuthology.orchestra.run.smithi115.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2022-05-13T23:00:56.433 INFO:teuthology.orchestra.run.smithi115.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2022-05-13T23:00:56.433 INFO:teuthology.orchestra.run.smithi115.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2022-05-13T23:00:56.433 INFO:teuthology.orchestra.run.smithi115.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2022-05-13T23:00:56.434 INFO:teuthology.orchestra.run.smithi047.stderr:bash: ntpq: command not found 2022-05-13T23:00:56.436 INFO:teuthology.orchestra.run.smithi047.stdout:210 Number of sources = 4 2022-05-13T23:00:56.436 INFO:teuthology.orchestra.run.smithi047.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2022-05-13T23:00:56.436 INFO:teuthology.orchestra.run.smithi047.stdout:=============================================================================== 2022-05-13T23:00:56.437 INFO:teuthology.orchestra.run.smithi047.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2022-05-13T23:00:56.437 INFO:teuthology.orchestra.run.smithi047.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2022-05-13T23:00:56.437 INFO:teuthology.orchestra.run.smithi047.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2022-05-13T23:00:56.437 INFO:teuthology.orchestra.run.smithi047.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2022-05-13T23:00:56.438 INFO:teuthology.orchestra.run.smithi002.stderr:sudo: ntpd: command not found 2022-05-13T23:00:56.468 INFO:teuthology.orchestra.run.smithi002.stdout:506 Cannot talk to daemon 2022-05-13T23:00:56.509 INFO:teuthology.orchestra.run.smithi002.stderr:Failed to start ntp.service: Unit ntp.service not found. 2022-05-13T23:00:56.552 INFO:teuthology.orchestra.run.smithi002.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2022-05-13T23:00:56.646 INFO:teuthology.orchestra.run.smithi002.stderr:bash: ntpq: command not found 2022-05-13T23:00:56.652 INFO:teuthology.orchestra.run.smithi002.stdout:210 Number of sources = 4 2022-05-13T23:00:56.652 INFO:teuthology.orchestra.run.smithi002.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2022-05-13T23:00:56.652 INFO:teuthology.orchestra.run.smithi002.stdout:=============================================================================== 2022-05-13T23:00:56.653 INFO:teuthology.orchestra.run.smithi002.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2022-05-13T23:00:56.653 INFO:teuthology.orchestra.run.smithi002.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2022-05-13T23:00:56.653 INFO:teuthology.orchestra.run.smithi002.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2022-05-13T23:00:56.653 INFO:teuthology.orchestra.run.smithi002.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2022-05-13T23:00:56.655 INFO:teuthology.run_tasks:Running task pexec... 2022-05-13T23:00:56.670 INFO:teuthology.task.pexec:Executing custom commands... 2022-05-13T23:00:56.672 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi002.front.sepia.ceph.com 2022-05-13T23:00:56.672 DEBUG:teuthology.orchestra.run.smithi002:> TESTDIR=/home/ubuntu/cephtest bash -s 2022-05-13T23:00:56.673 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi047.front.sepia.ceph.com 2022-05-13T23:00:56.673 DEBUG:teuthology.orchestra.run.smithi047:> TESTDIR=/home/ubuntu/cephtest bash -s 2022-05-13T23:00:56.674 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi115.front.sepia.ceph.com 2022-05-13T23:00:56.674 DEBUG:teuthology.orchestra.run.smithi115:> TESTDIR=/home/ubuntu/cephtest bash -s 2022-05-13T23:00:57.032 INFO:teuthology.orchestra.run.smithi115.stdout:Updating Subscription Management repositories. 2022-05-13T23:00:57.054 INFO:teuthology.orchestra.run.smithi047.stdout:Updating Subscription Management repositories. 2022-05-13T23:00:57.219 INFO:teuthology.orchestra.run.smithi002.stdout:Updating Subscription Management repositories. 2022-05-13T23:00:57.621 INFO:teuthology.orchestra.run.smithi115.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 80 kB/s | 2.4 kB 00:00 2022-05-13T23:00:57.702 INFO:teuthology.orchestra.run.smithi047.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 62 kB/s | 2.4 kB 00:00 2022-05-13T23:00:57.755 INFO:teuthology.orchestra.run.smithi115.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 93 kB/s | 2.8 kB 00:00 2022-05-13T23:00:57.855 INFO:teuthology.orchestra.run.smithi047.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 65 kB/s | 2.8 kB 00:00 2022-05-13T23:00:57.941 INFO:teuthology.orchestra.run.smithi002.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 75 kB/s | 2.8 kB 00:00 2022-05-13T23:00:58.176 INFO:teuthology.orchestra.run.smithi002.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 61 kB/s | 2.4 kB 00:00 2022-05-13T23:00:59.942 INFO:teuthology.orchestra.run.smithi115.stdout:Dependencies resolved. 2022-05-13T23:00:59.943 INFO:teuthology.orchestra.run.smithi115.stdout:================================================================================ 2022-05-13T23:00:59.943 INFO:teuthology.orchestra.run.smithi115.stdout: Package Architecture Version Repository Size 2022-05-13T23:00:59.943 INFO:teuthology.orchestra.run.smithi115.stdout:================================================================================ 2022-05-13T23:00:59.943 INFO:teuthology.orchestra.run.smithi115.stdout:Resetting modules: 2022-05-13T23:00:59.943 INFO:teuthology.orchestra.run.smithi115.stdout: container-tools 2022-05-13T23:00:59.944 INFO:teuthology.orchestra.run.smithi115.stdout: 2022-05-13T23:00:59.944 INFO:teuthology.orchestra.run.smithi115.stdout:Transaction Summary 2022-05-13T23:00:59.944 INFO:teuthology.orchestra.run.smithi115.stdout:================================================================================ 2022-05-13T23:00:59.944 INFO:teuthology.orchestra.run.smithi115.stdout: 2022-05-13T23:01:00.052 INFO:teuthology.orchestra.run.smithi047.stdout:Dependencies resolved. 2022-05-13T23:01:00.053 INFO:teuthology.orchestra.run.smithi047.stdout:================================================================================ 2022-05-13T23:01:00.053 INFO:teuthology.orchestra.run.smithi047.stdout: Package Architecture Version Repository Size 2022-05-13T23:01:00.053 INFO:teuthology.orchestra.run.smithi047.stdout:================================================================================ 2022-05-13T23:01:00.054 INFO:teuthology.orchestra.run.smithi047.stdout:Resetting modules: 2022-05-13T23:01:00.054 INFO:teuthology.orchestra.run.smithi047.stdout: container-tools 2022-05-13T23:01:00.054 INFO:teuthology.orchestra.run.smithi047.stdout: 2022-05-13T23:01:00.055 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction Summary 2022-05-13T23:01:00.055 INFO:teuthology.orchestra.run.smithi047.stdout:================================================================================ 2022-05-13T23:01:00.055 INFO:teuthology.orchestra.run.smithi047.stdout: 2022-05-13T23:01:00.517 INFO:teuthology.orchestra.run.smithi002.stdout:Dependencies resolved. 2022-05-13T23:01:00.518 INFO:teuthology.orchestra.run.smithi002.stdout:================================================================================ 2022-05-13T23:01:00.518 INFO:teuthology.orchestra.run.smithi002.stdout: Package Architecture Version Repository Size 2022-05-13T23:01:00.519 INFO:teuthology.orchestra.run.smithi002.stdout:================================================================================ 2022-05-13T23:01:00.519 INFO:teuthology.orchestra.run.smithi002.stdout:Resetting modules: 2022-05-13T23:01:00.519 INFO:teuthology.orchestra.run.smithi002.stdout: container-tools 2022-05-13T23:01:00.519 INFO:teuthology.orchestra.run.smithi002.stdout: 2022-05-13T23:01:00.520 INFO:teuthology.orchestra.run.smithi002.stdout:Transaction Summary 2022-05-13T23:01:00.520 INFO:teuthology.orchestra.run.smithi002.stdout:================================================================================ 2022-05-13T23:01:00.520 INFO:teuthology.orchestra.run.smithi002.stdout: 2022-05-13T23:01:13.787 INFO:teuthology.orchestra.run.smithi115.stdout:Last metadata expiration check: 0:00:06 ago on Fri 13 May 2022 10:55:44 PM UTC. 2022-05-13T23:01:14.777 INFO:teuthology.orchestra.run.smithi047.stdout:Last metadata expiration check: 0:00:07 ago on Fri 13 May 2022 10:56:28 PM UTC. 2022-05-13T23:01:16.345 INFO:teuthology.orchestra.run.smithi002.stdout:Last metadata expiration check: 0:00:07 ago on Fri 13 May 2022 10:58:58 PM UTC. 2022-05-13T23:01:16.970 INFO:teuthology.orchestra.run.smithi115.stdout:Complete! 2022-05-13T23:01:17.375 INFO:teuthology.orchestra.run.smithi115.stdout:Updating Subscription Management repositories. 2022-05-13T23:01:17.945 INFO:teuthology.orchestra.run.smithi115.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 76 kB/s | 2.4 kB 00:00 2022-05-13T23:01:18.075 INFO:teuthology.orchestra.run.smithi047.stdout:Complete! 2022-05-13T23:01:18.546 INFO:teuthology.orchestra.run.smithi047.stdout:Updating Subscription Management repositories. 2022-05-13T23:01:19.160 INFO:teuthology.orchestra.run.smithi047.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 60 kB/s | 2.4 kB 00:00 2022-05-13T23:01:19.830 INFO:teuthology.orchestra.run.smithi002.stdout:Complete! 2022-05-13T23:01:20.523 INFO:teuthology.orchestra.run.smithi002.stdout:Updating Subscription Management repositories. 2022-05-13T23:01:21.226 INFO:teuthology.orchestra.run.smithi002.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 72 kB/s | 2.8 kB 00:00 2022-05-13T23:01:23.383 INFO:teuthology.orchestra.run.smithi115.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 92 kB/s | 2.8 kB 00:00 2022-05-13T23:01:24.883 INFO:teuthology.orchestra.run.smithi047.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 71 kB/s | 2.8 kB 00:00 2022-05-13T23:01:26.992 INFO:teuthology.orchestra.run.smithi002.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 62 kB/s | 2.4 kB 00:00 2022-05-13T23:01:28.582 INFO:teuthology.orchestra.run.smithi115.stdout:Last metadata expiration check: 0:00:05 ago on Fri 13 May 2022 10:54:27 PM UTC. 2022-05-13T23:01:30.464 INFO:teuthology.orchestra.run.smithi047.stdout:Last metadata expiration check: 0:00:05 ago on Fri 13 May 2022 10:55:06 PM UTC. 2022-05-13T23:01:31.380 INFO:teuthology.orchestra.run.smithi115.stdout:Dependencies resolved. 2022-05-13T23:01:31.381 INFO:teuthology.orchestra.run.smithi115.stdout:========================================================================================================= 2022-05-13T23:01:31.381 INFO:teuthology.orchestra.run.smithi115.stdout: Package Arch Version Repository Size 2022-05-13T23:01:31.382 INFO:teuthology.orchestra.run.smithi115.stdout:========================================================================================================= 2022-05-13T23:01:31.384 INFO:teuthology.orchestra.run.smithi115.stdout:Installing group/module packages: 2022-05-13T23:01:31.384 INFO:teuthology.orchestra.run.smithi115.stdout: buildah x86_64 1.21.4-2.module+el8.4.0+12050+ef972f71 rhel-8-for-x86_64-appstream-rpms 8.2 M 2022-05-13T23:01:31.384 INFO:teuthology.orchestra.run.smithi115.stdout: cockpit-podman noarch 32-2.module+el8.4.0+11990+22932769 rhel-8-for-x86_64-appstream-rpms 410 k 2022-05-13T23:01:31.385 INFO:teuthology.orchestra.run.smithi115.stdout: crun x86_64 0.20.1-1.module+el8.4.0+11822+6cc1e7d7 rhel-8-for-x86_64-appstream-rpms 192 k 2022-05-13T23:01:31.385 INFO:teuthology.orchestra.run.smithi115.stdout: skopeo x86_64 1:1.3.1-5.module+el8.4.0+11990+22932769 rhel-8-for-x86_64-appstream-rpms 7.0 M 2022-05-13T23:01:31.385 INFO:teuthology.orchestra.run.smithi115.stdout: toolbox noarch 0.0.8-1.module+el8.4.0+11822+6cc1e7d7 rhel-8-for-x86_64-appstream-rpms 16 k 2022-05-13T23:01:31.385 INFO:teuthology.orchestra.run.smithi115.stdout: udica noarch 0.2.4-2.module+el8.4.0+11822+6cc1e7d7 rhel-8-for-x86_64-appstream-rpms 51 k 2022-05-13T23:01:31.385 INFO:teuthology.orchestra.run.smithi115.stdout:Installing dependencies: 2022-05-13T23:01:31.386 INFO:teuthology.orchestra.run.smithi115.stdout: yajl x86_64 2.1.0-10.el8 rhel-8-for-x86_64-appstream-rpms 41 k 2022-05-13T23:01:31.386 INFO:teuthology.orchestra.run.smithi115.stdout:Installing module profiles: 2022-05-13T23:01:31.386 INFO:teuthology.orchestra.run.smithi115.stdout: container-tools/common 2022-05-13T23:01:31.387 INFO:teuthology.orchestra.run.smithi115.stdout: 2022-05-13T23:01:31.387 INFO:teuthology.orchestra.run.smithi115.stdout:Enabling module streams: 2022-05-13T23:01:31.387 INFO:teuthology.orchestra.run.smithi115.stdout: container-tools rhel8 2022-05-13T23:01:31.387 INFO:teuthology.orchestra.run.smithi115.stdout: 2022-05-13T23:01:31.388 INFO:teuthology.orchestra.run.smithi115.stdout:Transaction Summary 2022-05-13T23:01:31.388 INFO:teuthology.orchestra.run.smithi115.stdout:========================================================================================================= 2022-05-13T23:01:31.388 INFO:teuthology.orchestra.run.smithi115.stdout:Install 7 Packages 2022-05-13T23:01:31.388 INFO:teuthology.orchestra.run.smithi115.stdout: 2022-05-13T23:01:31.389 INFO:teuthology.orchestra.run.smithi115.stdout:Total download size: 16 M 2022-05-13T23:01:31.389 INFO:teuthology.orchestra.run.smithi115.stdout:Installed size: 57 M 2022-05-13T23:01:31.390 INFO:teuthology.orchestra.run.smithi115.stdout:Downloading Packages: 2022-05-13T23:01:31.431 INFO:teuthology.orchestra.run.smithi115.stdout:(1/7): toolbox-0.0.8-1.module+el8.4.0+11822+6cc 334 kB/s | 16 kB 00:00 2022-05-13T23:01:31.438 INFO:teuthology.orchestra.run.smithi115.stdout:(2/7): yajl-2.1.0-10.el8.x86_64.rpm 744 kB/s | 41 kB 00:00 2022-05-13T23:01:31.439 INFO:teuthology.orchestra.run.smithi115.stdout:(3/7): crun-0.20.1-1.module+el8.4.0+11822+6cc1e 3.3 MB/s | 192 kB 00:00 2022-05-13T23:01:31.467 INFO:teuthology.orchestra.run.smithi115.stdout:(4/7): udica-0.2.4-2.module+el8.4.0+11822+6cc1e 1.7 MB/s | 51 kB 00:00 2022-05-13T23:01:31.477 INFO:teuthology.orchestra.run.smithi115.stdout:(5/7): cockpit-podman-32-2.module+el8.4.0+11990 12 MB/s | 410 kB 00:00 2022-05-13T23:01:31.519 INFO:teuthology.orchestra.run.smithi115.stdout:(6/7): skopeo-1.3.1-5.module+el8.4.0+11990+2293 80 MB/s | 7.0 MB 00:00 2022-05-13T23:01:31.553 INFO:teuthology.orchestra.run.smithi115.stdout:(7/7): buildah-1.21.4-2.module+el8.4.0+12050+ef 95 MB/s | 8.2 MB 00:00 2022-05-13T23:01:31.553 INFO:teuthology.orchestra.run.smithi115.stdout:-------------------------------------------------------------------------------- 2022-05-13T23:01:31.554 INFO:teuthology.orchestra.run.smithi115.stdout:Total 93 MB/s | 16 MB 00:00 2022-05-13T23:01:31.560 INFO:teuthology.orchestra.run.smithi115.stdout:Running transaction check 2022-05-13T23:01:31.576 INFO:teuthology.orchestra.run.smithi115.stdout:Transaction check succeeded. 2022-05-13T23:01:31.576 INFO:teuthology.orchestra.run.smithi115.stdout:Running transaction test 2022-05-13T23:01:31.741 INFO:teuthology.orchestra.run.smithi115.stdout:Transaction test succeeded. 2022-05-13T23:01:31.745 INFO:teuthology.orchestra.run.smithi115.stdout:Running transaction 2022-05-13T23:01:32.156 INFO:teuthology.orchestra.run.smithi115.stdout: Preparing : 1/1 2022-05-13T23:01:32.385 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : yajl-2.1.0-10.el8.x86_64 1/7 2022-05-13T23:01:33.212 INFO:teuthology.orchestra.run.smithi002.stdout:Last metadata expiration check: 0:00:05 ago on Fri 13 May 2022 10:57:28 PM UTC. 2022-05-13T23:01:33.272 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : crun-0.20.1-1.module+el8.4.0+11822+6cc1e7d7.x86_64 2/7 2022-05-13T23:01:33.445 INFO:teuthology.orchestra.run.smithi047.stdout:Dependencies resolved. 2022-05-13T23:01:33.446 INFO:teuthology.orchestra.run.smithi047.stdout:========================================================================================================= 2022-05-13T23:01:33.446 INFO:teuthology.orchestra.run.smithi047.stdout: Package Arch Version Repository Size 2022-05-13T23:01:33.446 INFO:teuthology.orchestra.run.smithi047.stdout:========================================================================================================= 2022-05-13T23:01:33.446 INFO:teuthology.orchestra.run.smithi047.stdout:Installing group/module packages: 2022-05-13T23:01:33.447 INFO:teuthology.orchestra.run.smithi047.stdout: buildah x86_64 1.21.4-2.module+el8.4.0+12050+ef972f71 rhel-8-for-x86_64-appstream-rpms 8.2 M 2022-05-13T23:01:33.447 INFO:teuthology.orchestra.run.smithi047.stdout: cockpit-podman noarch 32-2.module+el8.4.0+11990+22932769 rhel-8-for-x86_64-appstream-rpms 410 k 2022-05-13T23:01:33.447 INFO:teuthology.orchestra.run.smithi047.stdout: crun x86_64 0.20.1-1.module+el8.4.0+11822+6cc1e7d7 rhel-8-for-x86_64-appstream-rpms 192 k 2022-05-13T23:01:33.447 INFO:teuthology.orchestra.run.smithi047.stdout: skopeo x86_64 1:1.3.1-5.module+el8.4.0+11990+22932769 rhel-8-for-x86_64-appstream-rpms 7.0 M 2022-05-13T23:01:33.448 INFO:teuthology.orchestra.run.smithi047.stdout: toolbox noarch 0.0.8-1.module+el8.4.0+11822+6cc1e7d7 rhel-8-for-x86_64-appstream-rpms 16 k 2022-05-13T23:01:33.448 INFO:teuthology.orchestra.run.smithi047.stdout: udica noarch 0.2.4-2.module+el8.4.0+11822+6cc1e7d7 rhel-8-for-x86_64-appstream-rpms 51 k 2022-05-13T23:01:33.448 INFO:teuthology.orchestra.run.smithi047.stdout:Installing dependencies: 2022-05-13T23:01:33.448 INFO:teuthology.orchestra.run.smithi047.stdout: yajl x86_64 2.1.0-10.el8 rhel-8-for-x86_64-appstream-rpms 41 k 2022-05-13T23:01:33.448 INFO:teuthology.orchestra.run.smithi047.stdout:Installing module profiles: 2022-05-13T23:01:33.449 INFO:teuthology.orchestra.run.smithi047.stdout: container-tools/common 2022-05-13T23:01:33.449 INFO:teuthology.orchestra.run.smithi047.stdout: 2022-05-13T23:01:33.449 INFO:teuthology.orchestra.run.smithi047.stdout:Enabling module streams: 2022-05-13T23:01:33.449 INFO:teuthology.orchestra.run.smithi047.stdout: container-tools rhel8 2022-05-13T23:01:33.450 INFO:teuthology.orchestra.run.smithi047.stdout: 2022-05-13T23:01:33.450 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction Summary 2022-05-13T23:01:33.450 INFO:teuthology.orchestra.run.smithi047.stdout:========================================================================================================= 2022-05-13T23:01:33.450 INFO:teuthology.orchestra.run.smithi047.stdout:Install 7 Packages 2022-05-13T23:01:33.450 INFO:teuthology.orchestra.run.smithi047.stdout: 2022-05-13T23:01:33.451 INFO:teuthology.orchestra.run.smithi047.stdout:Total download size: 16 M 2022-05-13T23:01:33.452 INFO:teuthology.orchestra.run.smithi047.stdout:Installed size: 57 M 2022-05-13T23:01:33.452 INFO:teuthology.orchestra.run.smithi047.stdout:Downloading Packages: 2022-05-13T23:01:33.477 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : buildah-1.21.4-2.module+el8.4.0+12050+ef972f71.x86_6 3/7 2022-05-13T23:01:33.520 INFO:teuthology.orchestra.run.smithi047.stdout:(1/7): toolbox-0.0.8-1.module+el8.4.0+11822+6cc 221 kB/s | 16 kB 00:00 2022-05-13T23:01:33.521 INFO:teuthology.orchestra.run.smithi047.stdout:(2/7): crun-0.20.1-1.module+el8.4.0+11822+6cc1e 2.6 MB/s | 192 kB 00:00 2022-05-13T23:01:33.536 INFO:teuthology.orchestra.run.smithi047.stdout:(3/7): yajl-2.1.0-10.el8.x86_64.rpm 460 kB/s | 41 kB 00:00 2022-05-13T23:01:33.559 INFO:teuthology.orchestra.run.smithi047.stdout:(4/7): udica-0.2.4-2.module+el8.4.0+11822+6cc1e 1.3 MB/s | 51 kB 00:00 2022-05-13T23:01:33.588 INFO:teuthology.orchestra.run.smithi047.stdout:(5/7): cockpit-podman-32-2.module+el8.4.0+11990 8.1 MB/s | 410 kB 00:00 2022-05-13T23:01:33.625 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : cockpit-podman-32-2.module+el8.4.0+11990+22932769.no 4/7 2022-05-13T23:01:33.635 INFO:teuthology.orchestra.run.smithi047.stdout:(6/7): skopeo-1.3.1-5.module+el8.4.0+11990+2293 61 MB/s | 7.0 MB 00:00 2022-05-13T23:01:33.668 INFO:teuthology.orchestra.run.smithi047.stdout:(7/7): buildah-1.21.4-2.module+el8.4.0+12050+ef 76 MB/s | 8.2 MB 00:00 2022-05-13T23:01:33.668 INFO:teuthology.orchestra.run.smithi047.stdout:-------------------------------------------------------------------------------- 2022-05-13T23:01:33.669 INFO:teuthology.orchestra.run.smithi047.stdout:Total 72 MB/s | 16 MB 00:00 2022-05-13T23:01:33.674 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction check 2022-05-13T23:01:33.693 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction check succeeded. 2022-05-13T23:01:33.694 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction test 2022-05-13T23:01:33.871 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction test succeeded. 2022-05-13T23:01:33.874 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction 2022-05-13T23:01:34.300 INFO:teuthology.orchestra.run.smithi047.stdout: Preparing : 1/1 2022-05-13T23:01:34.430 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : udica-0.2.4-2.module+el8.4.0+11822+6cc1e7d7.noarch 5/7 2022-05-13T23:01:34.484 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : yajl-2.1.0-10.el8.x86_64 1/7 2022-05-13T23:01:34.636 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : skopeo-1:1.3.1-5.module+el8.4.0+11990+22932769.x86_6 6/7 2022-05-13T23:01:34.831 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : toolbox-0.0.8-1.module+el8.4.0+11822+6cc1e7d7.noarch 7/7 2022-05-13T23:01:35.063 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: toolbox-0.0.8-1.module+el8.4.0+11822+6cc1e7d7.noarch 7/7 2022-05-13T23:01:35.064 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : yajl-2.1.0-10.el8.x86_64 1/7 2022-05-13T23:01:35.064 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : toolbox-0.0.8-1.module+el8.4.0+11822+6cc1e7d7.noarch 2/7 2022-05-13T23:01:35.064 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : crun-0.20.1-1.module+el8.4.0+11822+6cc1e7d7.x86_64 3/7 2022-05-13T23:01:35.064 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : skopeo-1:1.3.1-5.module+el8.4.0+11990+22932769.x86_6 4/7 2022-05-13T23:01:35.065 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : udica-0.2.4-2.module+el8.4.0+11822+6cc1e7d7.noarch 5/7 2022-05-13T23:01:35.065 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : cockpit-podman-32-2.module+el8.4.0+11990+22932769.no 6/7 2022-05-13T23:01:35.385 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : buildah-1.21.4-2.module+el8.4.0+12050+ef972f71.x86_6 7/7 2022-05-13T23:01:35.385 INFO:teuthology.orchestra.run.smithi115.stdout:Installed products updated. 2022-05-13T23:01:35.406 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : crun-0.20.1-1.module+el8.4.0+11822+6cc1e7d7.x86_64 2/7 2022-05-13T23:01:35.636 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : buildah-1.21.4-2.module+el8.4.0+12050+ef972f71.x86_6 3/7 2022-05-13T23:01:35.857 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : cockpit-podman-32-2.module+el8.4.0+11990+22932769.no 4/7 2022-05-13T23:01:36.377 INFO:teuthology.orchestra.run.smithi002.stdout:Dependencies resolved. 2022-05-13T23:01:36.380 INFO:teuthology.orchestra.run.smithi002.stdout:========================================================================================================= 2022-05-13T23:01:36.380 INFO:teuthology.orchestra.run.smithi002.stdout: Package Arch Version Repository Size 2022-05-13T23:01:36.380 INFO:teuthology.orchestra.run.smithi002.stdout:========================================================================================================= 2022-05-13T23:01:36.381 INFO:teuthology.orchestra.run.smithi002.stdout:Installing group/module packages: 2022-05-13T23:01:36.385 INFO:teuthology.orchestra.run.smithi002.stdout: buildah x86_64 1.21.4-2.module+el8.4.0+12050+ef972f71 rhel-8-for-x86_64-appstream-rpms 8.2 M 2022-05-13T23:01:36.385 INFO:teuthology.orchestra.run.smithi002.stdout: cockpit-podman noarch 32-2.module+el8.4.0+11990+22932769 rhel-8-for-x86_64-appstream-rpms 410 k 2022-05-13T23:01:36.386 INFO:teuthology.orchestra.run.smithi002.stdout: crun x86_64 0.20.1-1.module+el8.4.0+11822+6cc1e7d7 rhel-8-for-x86_64-appstream-rpms 192 k 2022-05-13T23:01:36.386 INFO:teuthology.orchestra.run.smithi002.stdout: skopeo x86_64 1:1.3.1-5.module+el8.4.0+11990+22932769 rhel-8-for-x86_64-appstream-rpms 7.0 M 2022-05-13T23:01:36.387 INFO:teuthology.orchestra.run.smithi002.stdout: toolbox noarch 0.0.8-1.module+el8.4.0+11822+6cc1e7d7 rhel-8-for-x86_64-appstream-rpms 16 k 2022-05-13T23:01:36.387 INFO:teuthology.orchestra.run.smithi002.stdout: udica noarch 0.2.4-2.module+el8.4.0+11822+6cc1e7d7 rhel-8-for-x86_64-appstream-rpms 51 k 2022-05-13T23:01:36.387 INFO:teuthology.orchestra.run.smithi002.stdout:Installing dependencies: 2022-05-13T23:01:36.388 INFO:teuthology.orchestra.run.smithi002.stdout: yajl x86_64 2.1.0-10.el8 rhel-8-for-x86_64-appstream-rpms 41 k 2022-05-13T23:01:36.388 INFO:teuthology.orchestra.run.smithi002.stdout:Installing module profiles: 2022-05-13T23:01:36.389 INFO:teuthology.orchestra.run.smithi002.stdout: container-tools/common 2022-05-13T23:01:36.389 INFO:teuthology.orchestra.run.smithi002.stdout: 2022-05-13T23:01:36.389 INFO:teuthology.orchestra.run.smithi002.stdout:Enabling module streams: 2022-05-13T23:01:36.390 INFO:teuthology.orchestra.run.smithi002.stdout: container-tools rhel8 2022-05-13T23:01:36.390 INFO:teuthology.orchestra.run.smithi002.stdout: 2022-05-13T23:01:36.390 INFO:teuthology.orchestra.run.smithi002.stdout:Transaction Summary 2022-05-13T23:01:36.390 INFO:teuthology.orchestra.run.smithi002.stdout:========================================================================================================= 2022-05-13T23:01:36.390 INFO:teuthology.orchestra.run.smithi002.stdout:Install 7 Packages 2022-05-13T23:01:36.391 INFO:teuthology.orchestra.run.smithi002.stdout: 2022-05-13T23:01:36.391 INFO:teuthology.orchestra.run.smithi002.stdout:Total download size: 16 M 2022-05-13T23:01:36.391 INFO:teuthology.orchestra.run.smithi002.stdout:Installed size: 57 M 2022-05-13T23:01:36.391 INFO:teuthology.orchestra.run.smithi002.stdout:Downloading Packages: 2022-05-13T23:01:36.456 INFO:teuthology.orchestra.run.smithi002.stdout:(1/7): toolbox-0.0.8-1.module+el8.4.0+11822+6cc 214 kB/s | 16 kB 00:00 2022-05-13T23:01:36.467 INFO:teuthology.orchestra.run.smithi002.stdout:(2/7): yajl-2.1.0-10.el8.x86_64.rpm 477 kB/s | 41 kB 00:00 2022-05-13T23:01:36.470 INFO:teuthology.orchestra.run.smithi002.stdout:(3/7): crun-0.20.1-1.module+el8.4.0+11822+6cc1e 2.1 MB/s | 192 kB 00:00 2022-05-13T23:01:36.509 INFO:teuthology.orchestra.run.smithi002.stdout:(4/7): udica-0.2.4-2.module+el8.4.0+11822+6cc1e 1.2 MB/s | 51 kB 00:00 2022-05-13T23:01:36.537 INFO:teuthology.orchestra.run.smithi002.stdout:(5/7): cockpit-podman-32-2.module+el8.4.0+11990 6.6 MB/s | 410 kB 00:00 2022-05-13T23:01:36.663 INFO:teuthology.orchestra.run.smithi002.stdout:(6/7): skopeo-1.3.1-5.module+el8.4.0+11990+2293 34 MB/s | 7.0 MB 00:00 2022-05-13T23:01:36.709 INFO:teuthology.orchestra.run.smithi002.stdout:(7/7): buildah-1.21.4-2.module+el8.4.0+12050+ef 41 MB/s | 8.2 MB 00:00 2022-05-13T23:01:36.710 INFO:teuthology.orchestra.run.smithi002.stdout:-------------------------------------------------------------------------------- 2022-05-13T23:01:36.710 INFO:teuthology.orchestra.run.smithi002.stdout:Total 48 MB/s | 16 MB 00:00 2022-05-13T23:01:36.721 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : udica-0.2.4-2.module+el8.4.0+11822+6cc1e7d7.noarch 5/7 2022-05-13T23:01:36.723 INFO:teuthology.orchestra.run.smithi002.stdout:Running transaction check 2022-05-13T23:01:36.747 INFO:teuthology.orchestra.run.smithi002.stdout:Transaction check succeeded. 2022-05-13T23:01:36.747 INFO:teuthology.orchestra.run.smithi002.stdout:Running transaction test 2022-05-13T23:01:36.915 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : skopeo-1:1.3.1-5.module+el8.4.0+11990+22932769.x86_6 6/7 2022-05-13T23:01:37.110 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : toolbox-0.0.8-1.module+el8.4.0+11822+6cc1e7d7.noarch 7/7 2022-05-13T23:01:37.311 INFO:teuthology.orchestra.run.smithi002.stdout:Transaction test succeeded. 2022-05-13T23:01:37.316 INFO:teuthology.orchestra.run.smithi002.stdout:Running transaction 2022-05-13T23:01:37.361 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: toolbox-0.0.8-1.module+el8.4.0+11822+6cc1e7d7.noarch 7/7 2022-05-13T23:01:37.361 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : yajl-2.1.0-10.el8.x86_64 1/7 2022-05-13T23:01:37.362 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : toolbox-0.0.8-1.module+el8.4.0+11822+6cc1e7d7.noarch 2/7 2022-05-13T23:01:37.362 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : crun-0.20.1-1.module+el8.4.0+11822+6cc1e7d7.x86_64 3/7 2022-05-13T23:01:37.362 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : skopeo-1:1.3.1-5.module+el8.4.0+11990+22932769.x86_6 4/7 2022-05-13T23:01:37.362 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : udica-0.2.4-2.module+el8.4.0+11822+6cc1e7d7.noarch 5/7 2022-05-13T23:01:37.362 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : cockpit-podman-32-2.module+el8.4.0+11990+22932769.no 6/7 2022-05-13T23:01:37.721 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : buildah-1.21.4-2.module+el8.4.0+12050+ef972f71.x86_6 7/7 2022-05-13T23:01:37.722 INFO:teuthology.orchestra.run.smithi047.stdout:Installed products updated. 2022-05-13T23:01:37.782 INFO:teuthology.orchestra.run.smithi002.stdout: Preparing : 1/1 2022-05-13T23:01:37.998 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : yajl-2.1.0-10.el8.x86_64 1/7 2022-05-13T23:01:39.045 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : crun-0.20.1-1.module+el8.4.0+11822+6cc1e7d7.x86_64 2/7 2022-05-13T23:01:39.322 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : buildah-1.21.4-2.module+el8.4.0+12050+ef972f71.x86_6 3/7 2022-05-13T23:01:39.507 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : cockpit-podman-32-2.module+el8.4.0+11990+22932769.no 4/7 2022-05-13T23:01:40.453 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : udica-0.2.4-2.module+el8.4.0+11822+6cc1e7d7.noarch 5/7 2022-05-13T23:01:40.642 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : skopeo-1:1.3.1-5.module+el8.4.0+11990+22932769.x86_6 6/7 2022-05-13T23:01:40.830 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : toolbox-0.0.8-1.module+el8.4.0+11822+6cc1e7d7.noarch 7/7 2022-05-13T23:01:41.261 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: toolbox-0.0.8-1.module+el8.4.0+11822+6cc1e7d7.noarch 7/7 2022-05-13T23:01:41.261 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : yajl-2.1.0-10.el8.x86_64 1/7 2022-05-13T23:01:41.262 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : toolbox-0.0.8-1.module+el8.4.0+11822+6cc1e7d7.noarch 2/7 2022-05-13T23:01:41.262 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : crun-0.20.1-1.module+el8.4.0+11822+6cc1e7d7.x86_64 3/7 2022-05-13T23:01:41.262 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : skopeo-1:1.3.1-5.module+el8.4.0+11990+22932769.x86_6 4/7 2022-05-13T23:01:41.262 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : udica-0.2.4-2.module+el8.4.0+11822+6cc1e7d7.noarch 5/7 2022-05-13T23:01:41.262 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : cockpit-podman-32-2.module+el8.4.0+11990+22932769.no 6/7 2022-05-13T23:01:41.649 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : buildah-1.21.4-2.module+el8.4.0+12050+ef972f71.x86_6 7/7 2022-05-13T23:01:41.649 INFO:teuthology.orchestra.run.smithi002.stdout:Installed products updated. 2022-05-13T23:01:49.421 INFO:teuthology.orchestra.run.smithi115.stdout:Last metadata expiration check: 0:00:07 ago on Fri 13 May 2022 11:01:07 PM UTC. 2022-05-13T23:01:52.743 INFO:teuthology.orchestra.run.smithi047.stdout:Last metadata expiration check: 0:00:07 ago on Fri 13 May 2022 11:01:07 PM UTC. 2022-05-13T23:01:52.751 INFO:teuthology.orchestra.run.smithi115.stdout: 2022-05-13T23:01:52.752 INFO:teuthology.orchestra.run.smithi115.stdout:Installed: 2022-05-13T23:01:52.752 INFO:teuthology.orchestra.run.smithi115.stdout: buildah-1.21.4-2.module+el8.4.0+12050+ef972f71.x86_64 2022-05-13T23:01:52.753 INFO:teuthology.orchestra.run.smithi115.stdout: cockpit-podman-32-2.module+el8.4.0+11990+22932769.noarch 2022-05-13T23:01:52.753 INFO:teuthology.orchestra.run.smithi115.stdout: crun-0.20.1-1.module+el8.4.0+11822+6cc1e7d7.x86_64 2022-05-13T23:01:52.754 INFO:teuthology.orchestra.run.smithi115.stdout: skopeo-1:1.3.1-5.module+el8.4.0+11990+22932769.x86_64 2022-05-13T23:01:52.754 INFO:teuthology.orchestra.run.smithi115.stdout: toolbox-0.0.8-1.module+el8.4.0+11822+6cc1e7d7.noarch 2022-05-13T23:01:52.755 INFO:teuthology.orchestra.run.smithi115.stdout: udica-0.2.4-2.module+el8.4.0+11822+6cc1e7d7.noarch 2022-05-13T23:01:52.755 INFO:teuthology.orchestra.run.smithi115.stdout: yajl-2.1.0-10.el8.x86_64 2022-05-13T23:01:52.755 INFO:teuthology.orchestra.run.smithi115.stdout: 2022-05-13T23:01:52.756 INFO:teuthology.orchestra.run.smithi115.stdout:Complete! 2022-05-13T23:01:52.878 DEBUG:teuthology.parallel:result is None 2022-05-13T23:01:56.158 INFO:teuthology.orchestra.run.smithi047.stdout: 2022-05-13T23:01:56.158 INFO:teuthology.orchestra.run.smithi047.stdout:Installed: 2022-05-13T23:01:56.158 INFO:teuthology.orchestra.run.smithi047.stdout: buildah-1.21.4-2.module+el8.4.0+12050+ef972f71.x86_64 2022-05-13T23:01:56.158 INFO:teuthology.orchestra.run.smithi047.stdout: cockpit-podman-32-2.module+el8.4.0+11990+22932769.noarch 2022-05-13T23:01:56.159 INFO:teuthology.orchestra.run.smithi047.stdout: crun-0.20.1-1.module+el8.4.0+11822+6cc1e7d7.x86_64 2022-05-13T23:01:56.159 INFO:teuthology.orchestra.run.smithi047.stdout: skopeo-1:1.3.1-5.module+el8.4.0+11990+22932769.x86_64 2022-05-13T23:01:56.159 INFO:teuthology.orchestra.run.smithi047.stdout: toolbox-0.0.8-1.module+el8.4.0+11822+6cc1e7d7.noarch 2022-05-13T23:01:56.159 INFO:teuthology.orchestra.run.smithi047.stdout: udica-0.2.4-2.module+el8.4.0+11822+6cc1e7d7.noarch 2022-05-13T23:01:56.160 INFO:teuthology.orchestra.run.smithi047.stdout: yajl-2.1.0-10.el8.x86_64 2022-05-13T23:01:56.160 INFO:teuthology.orchestra.run.smithi047.stdout: 2022-05-13T23:01:56.160 INFO:teuthology.orchestra.run.smithi047.stdout:Complete! 2022-05-13T23:01:56.286 DEBUG:teuthology.parallel:result is None 2022-05-13T23:01:57.634 INFO:teuthology.orchestra.run.smithi002.stdout:Last metadata expiration check: 0:00:07 ago on Fri 13 May 2022 11:01:08 PM UTC. 2022-05-13T23:02:01.252 INFO:teuthology.orchestra.run.smithi002.stdout: 2022-05-13T23:02:01.256 INFO:teuthology.orchestra.run.smithi002.stdout:Installed: 2022-05-13T23:02:01.257 INFO:teuthology.orchestra.run.smithi002.stdout: buildah-1.21.4-2.module+el8.4.0+12050+ef972f71.x86_64 2022-05-13T23:02:01.257 INFO:teuthology.orchestra.run.smithi002.stdout: cockpit-podman-32-2.module+el8.4.0+11990+22932769.noarch 2022-05-13T23:02:01.257 INFO:teuthology.orchestra.run.smithi002.stdout: crun-0.20.1-1.module+el8.4.0+11822+6cc1e7d7.x86_64 2022-05-13T23:02:01.258 INFO:teuthology.orchestra.run.smithi002.stdout: skopeo-1:1.3.1-5.module+el8.4.0+11990+22932769.x86_64 2022-05-13T23:02:01.258 INFO:teuthology.orchestra.run.smithi002.stdout: toolbox-0.0.8-1.module+el8.4.0+11822+6cc1e7d7.noarch 2022-05-13T23:02:01.258 INFO:teuthology.orchestra.run.smithi002.stdout: udica-0.2.4-2.module+el8.4.0+11822+6cc1e7d7.noarch 2022-05-13T23:02:01.258 INFO:teuthology.orchestra.run.smithi002.stdout: yajl-2.1.0-10.el8.x86_64 2022-05-13T23:02:01.259 INFO:teuthology.orchestra.run.smithi002.stdout: 2022-05-13T23:02:01.259 INFO:teuthology.orchestra.run.smithi002.stdout:Complete! 2022-05-13T23:02:01.589 DEBUG:teuthology.parallel:result is None 2022-05-13T23:02:01.590 INFO:teuthology.run_tasks:Running task install... 2022-05-13T23:02:01.604 DEBUG:teuthology.task.install:project ceph 2022-05-13T23:02:01.605 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '23b0658391ce94ca40799469eec00c28b298816c'}} 2022-05-13T23:02:01.607 DEBUG:teuthology.task.install:config {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel']}, 'flavor': 'default', 'sha1': '23b0658391ce94ca40799469eec00c28b298816c'} 2022-05-13T23:02:01.607 INFO:teuthology.task.install:Using flavor: default 2022-05-13T23:02:01.615 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2022-05-13T23:02:01.616 INFO:teuthology.task.install:extra packages: {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']} 2022-05-13T23:02:01.617 DEBUG:teuthology.orchestra.run.smithi002:> sudo lsb_release -is 2022-05-13T23:02:01.753 INFO:teuthology.orchestra.run.smithi002.stdout:RedHatEnterprise 2022-05-13T23:02:01.766 DEBUG:teuthology.misc:System to be installed: RedHatEnterprise 2022-05-13T23:02:01.767 DEBUG:teuthology.orchestra.run.smithi047:> sudo lsb_release -is 2022-05-13T23:02:01.768 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel']}, 'extras': None, 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '23b0658391ce94ca40799469eec00c28b298816c', 'tag': None, 'wait_for_package': False} 2022-05-13T23:02:01.769 DEBUG:teuthology.orchestra.run.smithi002:> sudo lsb_release -is 2022-05-13T23:02:01.824 INFO:teuthology.orchestra.run.smithi047.stdout:RedHatEnterprise 2022-05-13T23:02:01.827 DEBUG:teuthology.misc:System to be installed: RedHatEnterprise 2022-05-13T23:02:01.827 DEBUG:teuthology.orchestra.run.smithi115:> sudo lsb_release -is 2022-05-13T23:02:01.834 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel']}, 'extras': None, 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '23b0658391ce94ca40799469eec00c28b298816c', 'tag': None, 'wait_for_package': False} 2022-05-13T23:02:01.835 DEBUG:teuthology.orchestra.run.smithi047:> sudo lsb_release -is 2022-05-13T23:02:01.881 INFO:teuthology.orchestra.run.smithi115.stdout:RedHatEnterprise 2022-05-13T23:02:01.883 DEBUG:teuthology.misc:System to be installed: RedHatEnterprise 2022-05-13T23:02:01.883 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel']}, 'extras': None, 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '23b0658391ce94ca40799469eec00c28b298816c', 'tag': None, 'wait_for_package': False} 2022-05-13T23:02:01.884 DEBUG:teuthology.orchestra.run.smithi115:> sudo lsb_release -is 2022-05-13T23:02:01.912 INFO:teuthology.orchestra.run.smithi047.stdout:RedHatEnterprise 2022-05-13T23:02:01.917 INFO:teuthology.orchestra.run.smithi002.stdout:RedHatEnterprise 2022-05-13T23:02:01.918 DEBUG:teuthology.misc:System to be installed: RedHatEnterprise 2022-05-13T23:02:01.919 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=23b0658391ce94ca40799469eec00c28b298816c 2022-05-13T23:02:01.926 DEBUG:teuthology.misc:System to be installed: RedHatEnterprise 2022-05-13T23:02:01.927 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=23b0658391ce94ca40799469eec00c28b298816c 2022-05-13T23:02:01.931 INFO:teuthology.orchestra.run.smithi115.stdout:RedHatEnterprise 2022-05-13T23:02:01.931 DEBUG:teuthology.misc:System to be installed: RedHatEnterprise 2022-05-13T23:02:01.932 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=23b0658391ce94ca40799469eec00c28b298816c 2022-05-13T23:02:02.078 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/wip-yuri-testing-2022-05-10-1027-quincy/23b0658391ce94ca40799469eec00c28b298816c/centos/8/flavors/default/ 2022-05-13T23:02:02.079 INFO:teuthology.task.install.rpm:Package version is 17.2.0-188.g23b06583 2022-05-13T23:02:02.093 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/wip-yuri-testing-2022-05-10-1027-quincy/23b0658391ce94ca40799469eec00c28b298816c/centos/8/flavors/default/ 2022-05-13T23:02:02.093 INFO:teuthology.task.install.rpm:Package version is 17.2.0-188.g23b06583 2022-05-13T23:02:02.107 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/wip-yuri-testing-2022-05-10-1027-quincy/23b0658391ce94ca40799469eec00c28b298816c/centos/8/flavors/default/ 2022-05-13T23:02:02.108 INFO:teuthology.task.install.rpm:Package version is 17.2.0-188.g23b06583 2022-05-13T23:02:02.206 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/wip-yuri-testing-2022-05-10-1027-quincy/23b0658391ce94ca40799469eec00c28b298816c/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-yuri-testing-2022-05-10-1027-quincy/23b0658391ce94ca40799469eec00c28b298816c/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-yuri-testing-2022-05-10-1027-quincy/23b0658391ce94ca40799469eec00c28b298816c/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2022-05-13T23:02:02.207 DEBUG:teuthology.orchestra.run.smithi047:> set -ex 2022-05-13T23:02:02.208 DEBUG:teuthology.orchestra.run.smithi047:> sudo dd of=/etc/yum.repos.d/ceph.repo 2022-05-13T23:02:02.212 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/wip-yuri-testing-2022-05-10-1027-quincy/23b0658391ce94ca40799469eec00c28b298816c/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-yuri-testing-2022-05-10-1027-quincy/23b0658391ce94ca40799469eec00c28b298816c/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-yuri-testing-2022-05-10-1027-quincy/23b0658391ce94ca40799469eec00c28b298816c/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2022-05-13T23:02:02.213 DEBUG:teuthology.orchestra.run.smithi115:> set -ex 2022-05-13T23:02:02.213 DEBUG:teuthology.orchestra.run.smithi115:> sudo dd of=/etc/yum.repos.d/ceph.repo 2022-05-13T23:02:02.226 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/wip-yuri-testing-2022-05-10-1027-quincy/23b0658391ce94ca40799469eec00c28b298816c/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-yuri-testing-2022-05-10-1027-quincy/23b0658391ce94ca40799469eec00c28b298816c/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-yuri-testing-2022-05-10-1027-quincy/23b0658391ce94ca40799469eec00c28b298816c/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2022-05-13T23:02:02.226 DEBUG:teuthology.orchestra.run.smithi002:> set -ex 2022-05-13T23:02:02.227 DEBUG:teuthology.orchestra.run.smithi002:> sudo dd of=/etc/yum.repos.d/ceph.repo 2022-05-13T23:02:02.244 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel on remote rpm x86_64 2022-05-13T23:02:02.244 DEBUG:teuthology.orchestra.run.smithi047:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/23b0658391ce94ca40799469eec00c28b298816c/;g' /etc/yum.repos.d/ceph.repo ; fi 2022-05-13T23:02:02.246 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel on remote rpm x86_64 2022-05-13T23:02:02.247 DEBUG:teuthology.orchestra.run.smithi115:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/23b0658391ce94ca40799469eec00c28b298816c/;g' /etc/yum.repos.d/ceph.repo ; fi 2022-05-13T23:02:02.306 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel on remote rpm x86_64 2022-05-13T23:02:02.306 DEBUG:teuthology.orchestra.run.smithi002:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/23b0658391ce94ca40799469eec00c28b298816c/;g' /etc/yum.repos.d/ceph.repo ; fi 2022-05-13T23:02:02.314 DEBUG:teuthology.orchestra.run.smithi047:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2022-05-13T23:02:02.316 DEBUG:teuthology.orchestra.run.smithi115:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2022-05-13T23:02:02.394 DEBUG:teuthology.orchestra.run.smithi047:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2022-05-13T23:02:02.396 DEBUG:teuthology.orchestra.run.smithi115:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2022-05-13T23:02:02.427 DEBUG:teuthology.orchestra.run.smithi002:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2022-05-13T23:02:02.461 INFO:teuthology.orchestra.run.smithi047.stdout:check_obsoletes = 1 2022-05-13T23:02:02.464 INFO:teuthology.orchestra.run.smithi115.stdout:check_obsoletes = 1 2022-05-13T23:02:02.464 DEBUG:teuthology.orchestra.run.smithi047:> sudo yum clean all 2022-05-13T23:02:02.465 DEBUG:teuthology.orchestra.run.smithi115:> sudo yum clean all 2022-05-13T23:02:02.540 DEBUG:teuthology.orchestra.run.smithi002:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2022-05-13T23:02:02.617 INFO:teuthology.orchestra.run.smithi002.stdout:check_obsoletes = 1 2022-05-13T23:02:02.622 DEBUG:teuthology.orchestra.run.smithi002:> sudo yum clean all 2022-05-13T23:02:02.821 INFO:teuthology.orchestra.run.smithi115.stdout:Updating Subscription Management repositories. 2022-05-13T23:02:02.842 INFO:teuthology.orchestra.run.smithi047.stdout:Updating Subscription Management repositories. 2022-05-13T23:02:03.115 INFO:teuthology.orchestra.run.smithi002.stdout:Updating Subscription Management repositories. 2022-05-13T23:02:03.115 INFO:teuthology.orchestra.run.smithi115.stdout:54 files removed 2022-05-13T23:02:03.159 DEBUG:teuthology.orchestra.run.smithi115:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel 2022-05-13T23:02:03.169 INFO:teuthology.orchestra.run.smithi047.stdout:54 files removed 2022-05-13T23:02:03.214 DEBUG:teuthology.orchestra.run.smithi047:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel 2022-05-13T23:02:03.473 INFO:teuthology.orchestra.run.smithi115.stdout:Updating Subscription Management repositories. 2022-05-13T23:02:03.548 INFO:teuthology.orchestra.run.smithi047.stdout:Updating Subscription Management repositories. 2022-05-13T23:02:03.569 INFO:teuthology.orchestra.run.smithi002.stdout:54 files removed 2022-05-13T23:02:03.633 DEBUG:teuthology.orchestra.run.smithi002:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel 2022-05-13T23:02:04.121 INFO:teuthology.orchestra.run.smithi115.stdout:ceph packages for x86_64 323 kB/s | 78 kB 00:00 2022-05-13T23:02:04.123 INFO:teuthology.orchestra.run.smithi002.stdout:Updating Subscription Management repositories. 2022-05-13T23:02:04.228 INFO:teuthology.orchestra.run.smithi047.stdout:ceph packages for x86_64 324 kB/s | 78 kB 00:00 2022-05-13T23:02:04.354 INFO:teuthology.orchestra.run.smithi115.stdout:ceph noarch packages 46 kB/s | 9.1 kB 00:00 2022-05-13T23:02:04.466 INFO:teuthology.orchestra.run.smithi047.stdout:ceph noarch packages 45 kB/s | 9.1 kB 00:00 2022-05-13T23:02:04.577 INFO:teuthology.orchestra.run.smithi115.stdout:ceph source packages 7.2 kB/s | 1.4 kB 00:00 2022-05-13T23:02:04.688 INFO:teuthology.orchestra.run.smithi115.stdout:Copr repo for python3-asyncssh owned by ceph 39 kB/s | 3.5 kB 00:00 2022-05-13T23:02:04.691 INFO:teuthology.orchestra.run.smithi047.stdout:ceph source packages 7.1 kB/s | 1.4 kB 00:00 2022-05-13T23:02:04.812 INFO:teuthology.orchestra.run.smithi047.stdout:Copr repo for python3-asyncssh owned by ceph 36 kB/s | 3.5 kB 00:00 2022-05-13T23:02:04.913 INFO:teuthology.orchestra.run.smithi002.stdout:ceph packages for x86_64 320 kB/s | 78 kB 00:00 2022-05-13T23:02:05.157 INFO:teuthology.orchestra.run.smithi002.stdout:ceph noarch packages 45 kB/s | 9.1 kB 00:00 2022-05-13T23:02:05.386 INFO:teuthology.orchestra.run.smithi002.stdout:ceph source packages 7.1 kB/s | 1.4 kB 00:00 2022-05-13T23:02:05.494 INFO:teuthology.orchestra.run.smithi002.stdout:Copr repo for python3-asyncssh owned by ceph 42 kB/s | 3.5 kB 00:00 2022-05-13T23:02:05.509 INFO:teuthology.orchestra.run.smithi115.stdout:Extra Packages for Enterprise Linux 14 MB/s | 11 MB 00:00 2022-05-13T23:02:05.612 INFO:teuthology.orchestra.run.smithi047.stdout:Extra Packages for Enterprise Linux 14 MB/s | 11 MB 00:00 2022-05-13T23:02:06.304 INFO:teuthology.orchestra.run.smithi002.stdout:Extra Packages for Enterprise Linux 14 MB/s | 11 MB 00:00 2022-05-13T23:02:07.761 INFO:teuthology.orchestra.run.smithi115.stdout:lab-extras 4.2 MB/s | 24 kB 00:00 2022-05-13T23:02:07.965 INFO:teuthology.orchestra.run.smithi047.stdout:lab-extras 4.3 MB/s | 24 kB 00:00 2022-05-13T23:02:08.092 INFO:teuthology.orchestra.run.smithi115.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 143 MB/s | 38 MB 00:00 2022-05-13T23:02:08.364 INFO:teuthology.orchestra.run.smithi047.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 116 MB/s | 38 MB 00:00 2022-05-13T23:02:08.940 INFO:teuthology.orchestra.run.smithi002.stdout:lab-extras 4.0 MB/s | 24 kB 00:00 2022-05-13T23:02:09.501 INFO:teuthology.orchestra.run.smithi002.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 68 MB/s | 33 MB 00:00 2022-05-13T23:02:13.535 INFO:teuthology.orchestra.run.smithi115.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 129 MB/s | 33 MB 00:00 2022-05-13T23:02:14.172 INFO:teuthology.orchestra.run.smithi047.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 102 MB/s | 33 MB 00:00 2022-05-13T23:02:15.784 INFO:teuthology.orchestra.run.smithi002.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 73 MB/s | 38 MB 00:00 2022-05-13T23:02:21.360 INFO:teuthology.orchestra.run.smithi115.stdout:Package librados2-1:12.2.7-9.el8.x86_64 is already installed. 2022-05-13T23:02:21.361 INFO:teuthology.orchestra.run.smithi115.stdout:Package librbd1-1:12.2.7-9.el8.x86_64 is already installed. 2022-05-13T23:02:21.371 INFO:teuthology.orchestra.run.smithi115.stdout:Package NetworkManager-1:1.30.0-7.el8.x86_64 is already installed. 2022-05-13T23:02:21.372 INFO:teuthology.orchestra.run.smithi115.stdout:Package iproute-5.9.0-4.el8.x86_64 is already installed. 2022-05-13T23:02:21.373 INFO:teuthology.orchestra.run.smithi115.stdout:Package util-linux-2.32.1-27.el8.x86_64 is already installed. 2022-05-13T23:02:21.374 INFO:teuthology.orchestra.run.smithi115.stdout:Package libacl-devel-2.2.53-1.el8.x86_64 is already installed. 2022-05-13T23:02:21.376 INFO:teuthology.orchestra.run.smithi115.stdout:Package libattr-devel-2.4.48-3.el8.x86_64 is already installed. 2022-05-13T23:02:21.378 INFO:teuthology.orchestra.run.smithi115.stdout:Package xfsdump-3.1.8-2.el8.x86_64 is already installed. 2022-05-13T23:02:21.379 INFO:teuthology.orchestra.run.smithi115.stdout:Package xfsprogs-5.0.0-8.el8.x86_64 is already installed. 2022-05-13T23:02:21.831 INFO:teuthology.orchestra.run.smithi115.stdout:Dependencies resolved. 2022-05-13T23:02:21.844 INFO:teuthology.orchestra.run.smithi115.stdout:====================================================================================================================== 2022-05-13T23:02:21.845 INFO:teuthology.orchestra.run.smithi115.stdout: Package Arch Version Repository Size 2022-05-13T23:02:21.846 INFO:teuthology.orchestra.run.smithi115.stdout:====================================================================================================================== 2022-05-13T23:02:21.846 INFO:teuthology.orchestra.run.smithi115.stdout:Installing: 2022-05-13T23:02:21.846 INFO:teuthology.orchestra.run.smithi115.stdout: bison x86_64 3.0.4-10.el8 rhel-8-for-x86_64-appstream-rpms 688 k 2022-05-13T23:02:21.846 INFO:teuthology.orchestra.run.smithi115.stdout: ceph x86_64 2:17.2.0-188.g23b06583.el8 ceph 6.4 k 2022-05-13T23:02:21.847 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-base x86_64 2:17.2.0-188.g23b06583.el8 ceph 5.3 M 2022-05-13T23:02:21.847 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-fuse x86_64 2:17.2.0-188.g23b06583.el8 ceph 876 k 2022-05-13T23:02:21.847 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-immutable-object-cache x86_64 2:17.2.0-188.g23b06583.el8 ceph 169 k 2022-05-13T23:02:21.847 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-mgr x86_64 2:17.2.0-188.g23b06583.el8 ceph 1.5 M 2022-05-13T23:02:21.848 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-mgr-cephadm noarch 2:17.2.0-188.g23b06583.el8 ceph-noarch 105 k 2022-05-13T23:02:21.848 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-mgr-dashboard noarch 2:17.2.0-188.g23b06583.el8 ceph-noarch 1.6 M 2022-05-13T23:02:21.848 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-mgr-diskprediction-local noarch 2:17.2.0-188.g23b06583.el8 ceph-noarch 7.4 M 2022-05-13T23:02:21.848 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-mgr-rook noarch 2:17.2.0-188.g23b06583.el8 ceph-noarch 47 k 2022-05-13T23:02:21.848 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-radosgw x86_64 2:17.2.0-188.g23b06583.el8 ceph 16 M 2022-05-13T23:02:21.849 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-test x86_64 2:17.2.0-188.g23b06583.el8 ceph 52 M 2022-05-13T23:02:21.849 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-volume noarch 2:17.2.0-188.g23b06583.el8 ceph-noarch 259 k 2022-05-13T23:02:21.849 INFO:teuthology.orchestra.run.smithi115.stdout: cephadm noarch 2:17.2.0-188.g23b06583.el8 ceph-noarch 75 k 2022-05-13T23:02:21.849 INFO:teuthology.orchestra.run.smithi115.stdout: cephfs-mirror x86_64 2:17.2.0-188.g23b06583.el8 ceph 272 k 2022-05-13T23:02:21.850 INFO:teuthology.orchestra.run.smithi115.stdout: cephfs-top noarch 2:17.2.0-188.g23b06583.el8 ceph-noarch 14 k 2022-05-13T23:02:21.850 INFO:teuthology.orchestra.run.smithi115.stdout: elfutils-libelf-devel x86_64 0.182-3.el8 rhel-8-for-x86_64-baseos-rpms 59 k 2022-05-13T23:02:21.850 INFO:teuthology.orchestra.run.smithi115.stdout: flex x86_64 2.6.1-9.el8 rhel-8-for-x86_64-appstream-rpms 320 k 2022-05-13T23:02:21.850 INFO:teuthology.orchestra.run.smithi115.stdout: libaio-devel x86_64 0.3.112-1.el8 rhel-8-for-x86_64-baseos-rpms 19 k 2022-05-13T23:02:21.850 INFO:teuthology.orchestra.run.smithi115.stdout: libcephfs-devel x86_64 2:17.2.0-188.g23b06583.el8 ceph 24 k 2022-05-13T23:02:21.851 INFO:teuthology.orchestra.run.smithi115.stdout: libcephfs2 x86_64 2:17.2.0-188.g23b06583.el8 ceph 790 k 2022-05-13T23:02:21.851 INFO:teuthology.orchestra.run.smithi115.stdout: librados-devel x86_64 2:17.2.0-188.g23b06583.el8 ceph 125 k 2022-05-13T23:02:21.851 INFO:teuthology.orchestra.run.smithi115.stdout: libtool x86_64 2.4.6-25.el8 rhel-8-for-x86_64-appstream-rpms 709 k 2022-05-13T23:02:21.851 INFO:teuthology.orchestra.run.smithi115.stdout: libuuid-devel x86_64 2.32.1-27.el8 rhel-8-for-x86_64-baseos-rpms 97 k 2022-05-13T23:02:21.852 INFO:teuthology.orchestra.run.smithi115.stdout: openssl-devel x86_64 1:1.1.1g-15.el8_3 rhel-8-for-x86_64-baseos-rpms 2.3 M 2022-05-13T23:02:21.852 INFO:teuthology.orchestra.run.smithi115.stdout: python3-cephfs x86_64 2:17.2.0-188.g23b06583.el8 ceph 178 k 2022-05-13T23:02:21.852 INFO:teuthology.orchestra.run.smithi115.stdout: python3-rados x86_64 2:17.2.0-188.g23b06583.el8 ceph 367 k 2022-05-13T23:02:21.852 INFO:teuthology.orchestra.run.smithi115.stdout: python3-rbd x86_64 2:17.2.0-188.g23b06583.el8 ceph 346 k 2022-05-13T23:02:21.853 INFO:teuthology.orchestra.run.smithi115.stdout: python3-rgw x86_64 2:17.2.0-188.g23b06583.el8 ceph 109 k 2022-05-13T23:02:21.853 INFO:teuthology.orchestra.run.smithi115.stdout: rbd-fuse x86_64 2:17.2.0-188.g23b06583.el8 ceph 93 k 2022-05-13T23:02:21.853 INFO:teuthology.orchestra.run.smithi115.stdout: rbd-mirror x86_64 2:17.2.0-188.g23b06583.el8 ceph 3.8 M 2022-05-13T23:02:21.853 INFO:teuthology.orchestra.run.smithi115.stdout: rbd-nbd x86_64 2:17.2.0-188.g23b06583.el8 ceph 175 k 2022-05-13T23:02:21.854 INFO:teuthology.orchestra.run.smithi115.stdout: xfsprogs-devel x86_64 5.0.0-8.el8 rhel-8-for-x86_64-baseos-rpms 70 k 2022-05-13T23:02:21.854 INFO:teuthology.orchestra.run.smithi115.stdout:Upgrading: 2022-05-13T23:02:21.854 INFO:teuthology.orchestra.run.smithi115.stdout: NetworkManager x86_64 1:1.30.0-13.el8_4 rhel-8-for-x86_64-baseos-rpms 2.6 M 2022-05-13T23:02:21.854 INFO:teuthology.orchestra.run.smithi115.stdout: NetworkManager-libnm x86_64 1:1.30.0-13.el8_4 rhel-8-for-x86_64-baseos-rpms 1.8 M 2022-05-13T23:02:21.854 INFO:teuthology.orchestra.run.smithi115.stdout: NetworkManager-team x86_64 1:1.30.0-13.el8_4 rhel-8-for-x86_64-baseos-rpms 146 k 2022-05-13T23:02:21.855 INFO:teuthology.orchestra.run.smithi115.stdout: NetworkManager-tui x86_64 1:1.30.0-13.el8_4 rhel-8-for-x86_64-baseos-rpms 329 k 2022-05-13T23:02:21.855 INFO:teuthology.orchestra.run.smithi115.stdout: librados2 x86_64 2:17.2.0-188.g23b06583.el8 ceph 3.4 M 2022-05-13T23:02:21.855 INFO:teuthology.orchestra.run.smithi115.stdout: librbd1 x86_64 2:17.2.0-188.g23b06583.el8 ceph 3.9 M 2022-05-13T23:02:21.855 INFO:teuthology.orchestra.run.smithi115.stdout:Installing dependencies: 2022-05-13T23:02:21.856 INFO:teuthology.orchestra.run.smithi115.stdout: automake noarch 1.16.1-7.el8 rhel-8-for-x86_64-appstream-rpms 713 k 2022-05-13T23:02:21.856 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-common x86_64 2:17.2.0-188.g23b06583.el8 ceph 23 M 2022-05-13T23:02:21.856 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-grafana-dashboards noarch 2:17.2.0-188.g23b06583.el8 ceph-noarch 23 k 2022-05-13T23:02:21.856 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-mds x86_64 2:17.2.0-188.g23b06583.el8 ceph 2.3 M 2022-05-13T23:02:21.856 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-mgr-modules-core noarch 2:17.2.0-188.g23b06583.el8 ceph-noarch 228 k 2022-05-13T23:02:21.857 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-mon x86_64 2:17.2.0-188.g23b06583.el8 ceph 3.9 M 2022-05-13T23:02:21.857 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-osd x86_64 2:17.2.0-188.g23b06583.el8 ceph 17 M 2022-05-13T23:02:21.857 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-prometheus-alerts noarch 2:17.2.0-188.g23b06583.el8 ceph-noarch 15 k 2022-05-13T23:02:21.857 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-selinux x86_64 2:17.2.0-188.g23b06583.el8 ceph 24 k 2022-05-13T23:02:21.858 INFO:teuthology.orchestra.run.smithi115.stdout: fuse x86_64 2.9.7-12.el8 rhel-8-for-x86_64-baseos-rpms 83 k 2022-05-13T23:02:21.858 INFO:teuthology.orchestra.run.smithi115.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2022-05-13T23:02:21.858 INFO:teuthology.orchestra.run.smithi115.stdout: jq x86_64 1.5-12.el8 rhel-8-for-x86_64-appstream-rpms 161 k 2022-05-13T23:02:21.858 INFO:teuthology.orchestra.run.smithi115.stdout: keyutils-libs-devel x86_64 1.5.10-6.el8 rhel-8-for-x86_64-baseos-rpms 48 k 2022-05-13T23:02:21.858 INFO:teuthology.orchestra.run.smithi115.stdout: krb5-devel x86_64 1.18.2-8.3.el8_4 rhel-8-for-x86_64-baseos-rpms 559 k 2022-05-13T23:02:21.859 INFO:teuthology.orchestra.run.smithi115.stdout: libcephsqlite x86_64 2:17.2.0-188.g23b06583.el8 ceph 170 k 2022-05-13T23:02:21.859 INFO:teuthology.orchestra.run.smithi115.stdout: libcom_err-devel x86_64 1.45.6-1.el8 rhel-8-for-x86_64-baseos-rpms 38 k 2022-05-13T23:02:21.859 INFO:teuthology.orchestra.run.smithi115.stdout: libgfortran x86_64 8.4.1-1.el8 rhel-8-for-x86_64-baseos-rpms 639 k 2022-05-13T23:02:21.859 INFO:teuthology.orchestra.run.smithi115.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2022-05-13T23:02:21.860 INFO:teuthology.orchestra.run.smithi115.stdout: libquadmath x86_64 8.4.1-1.el8 rhel-8-for-x86_64-baseos-rpms 169 k 2022-05-13T23:02:21.860 INFO:teuthology.orchestra.run.smithi115.stdout: librabbitmq x86_64 0.9.0-3.el8 rhel-8-for-x86_64-baseos-rpms 47 k 2022-05-13T23:02:21.860 INFO:teuthology.orchestra.run.smithi115.stdout: libradosstriper1 x86_64 2:17.2.0-188.g23b06583.el8 ceph 483 k 2022-05-13T23:02:21.860 INFO:teuthology.orchestra.run.smithi115.stdout: librdkafka x86_64 0.11.4-1.el8 rhel-8-for-x86_64-appstream-rpms 353 k 2022-05-13T23:02:21.861 INFO:teuthology.orchestra.run.smithi115.stdout: librgw2 x86_64 2:17.2.0-188.g23b06583.el8 ceph 5.3 M 2022-05-13T23:02:21.861 INFO:teuthology.orchestra.run.smithi115.stdout: libselinux-devel x86_64 2.9-5.el8 rhel-8-for-x86_64-baseos-rpms 200 k 2022-05-13T23:02:21.861 INFO:teuthology.orchestra.run.smithi115.stdout: libsepol-devel x86_64 2.9-2.el8 rhel-8-for-x86_64-baseos-rpms 86 k 2022-05-13T23:02:21.861 INFO:teuthology.orchestra.run.smithi115.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2022-05-13T23:02:21.862 INFO:teuthology.orchestra.run.smithi115.stdout: libverto-devel x86_64 0.3.0-5.el8 rhel-8-for-x86_64-baseos-rpms 18 k 2022-05-13T23:02:21.862 INFO:teuthology.orchestra.run.smithi115.stdout: lttng-ust x86_64 2.8.1-11.el8 rhel-8-for-x86_64-appstream-rpms 259 k 2022-05-13T23:02:21.862 INFO:teuthology.orchestra.run.smithi115.stdout: oniguruma x86_64 6.8.2-2.el8 rhel-8-for-x86_64-appstream-rpms 187 k 2022-05-13T23:02:21.862 INFO:teuthology.orchestra.run.smithi115.stdout: openblas x86_64 0.3.12-1.el8 rhel-8-for-x86_64-appstream-rpms 4.6 M 2022-05-13T23:02:21.863 INFO:teuthology.orchestra.run.smithi115.stdout: openblas-threads x86_64 0.3.12-1.el8 rhel-8-for-x86_64-appstream-rpms 4.7 M 2022-05-13T23:02:21.863 INFO:teuthology.orchestra.run.smithi115.stdout: pcre2-devel x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 605 k 2022-05-13T23:02:21.863 INFO:teuthology.orchestra.run.smithi115.stdout: pcre2-utf16 x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 229 k 2022-05-13T23:02:21.863 INFO:teuthology.orchestra.run.smithi115.stdout: pcre2-utf32 x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 220 k 2022-05-13T23:02:21.863 INFO:teuthology.orchestra.run.smithi115.stdout: python3-asyncssh noarch 2.7.0-2.el8 epel 476 k 2022-05-13T23:02:21.864 INFO:teuthology.orchestra.run.smithi115.stdout: python3-babel noarch 2.5.1-5.el8 rhel-8-for-x86_64-appstream-rpms 4.8 M 2022-05-13T23:02:21.864 INFO:teuthology.orchestra.run.smithi115.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2022-05-13T23:02:21.864 INFO:teuthology.orchestra.run.smithi115.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2022-05-13T23:02:21.864 INFO:teuthology.orchestra.run.smithi115.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2022-05-13T23:02:21.864 INFO:teuthology.orchestra.run.smithi115.stdout: python3-ceph-argparse x86_64 2:17.2.0-188.g23b06583.el8 ceph 45 k 2022-05-13T23:02:21.865 INFO:teuthology.orchestra.run.smithi115.stdout: python3-ceph-common x86_64 2:17.2.0-188.g23b06583.el8 ceph 106 k 2022-05-13T23:02:21.865 INFO:teuthology.orchestra.run.smithi115.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2022-05-13T23:02:21.865 INFO:teuthology.orchestra.run.smithi115.stdout: python3-cffi x86_64 1.11.5-5.el8 rhel-8-for-x86_64-baseos-rpms 238 k 2022-05-13T23:02:21.865 INFO:teuthology.orchestra.run.smithi115.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2022-05-13T23:02:21.866 INFO:teuthology.orchestra.run.smithi115.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2022-05-13T23:02:21.866 INFO:teuthology.orchestra.run.smithi115.stdout: python3-cryptography x86_64 3.2.1-4.el8 rhel-8-for-x86_64-baseos-rpms 559 k 2022-05-13T23:02:21.866 INFO:teuthology.orchestra.run.smithi115.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2022-05-13T23:02:21.866 INFO:teuthology.orchestra.run.smithi115.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2022-05-13T23:02:21.866 INFO:teuthology.orchestra.run.smithi115.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2022-05-13T23:02:21.867 INFO:teuthology.orchestra.run.smithi115.stdout: python3-jinja2 noarch 2.10.1-2.el8_0 rhel-8-for-x86_64-appstream-rpms 538 k 2022-05-13T23:02:21.867 INFO:teuthology.orchestra.run.smithi115.stdout: python3-jsonpatch noarch 1.21-2.el8 rhel-8-for-x86_64-appstream-rpms 27 k 2022-05-13T23:02:21.867 INFO:teuthology.orchestra.run.smithi115.stdout: python3-jsonpointer noarch 1.10-11.el8 rhel-8-for-x86_64-appstream-rpms 20 k 2022-05-13T23:02:21.867 INFO:teuthology.orchestra.run.smithi115.stdout: python3-jwt noarch 1.6.1-2.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2022-05-13T23:02:21.867 INFO:teuthology.orchestra.run.smithi115.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2022-05-13T23:02:21.868 INFO:teuthology.orchestra.run.smithi115.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2022-05-13T23:02:21.868 INFO:teuthology.orchestra.run.smithi115.stdout: python3-mako noarch 1.0.6-13.el8 rhel-8-for-x86_64-appstream-rpms 157 k 2022-05-13T23:02:21.868 INFO:teuthology.orchestra.run.smithi115.stdout: python3-markupsafe x86_64 0.23-19.el8 rhel-8-for-x86_64-appstream-rpms 39 k 2022-05-13T23:02:21.869 INFO:teuthology.orchestra.run.smithi115.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2022-05-13T23:02:21.869 INFO:teuthology.orchestra.run.smithi115.stdout: python3-natsort noarch 7.1.1-2.el8 epel 60 k 2022-05-13T23:02:21.869 INFO:teuthology.orchestra.run.smithi115.stdout: python3-numpy x86_64 1:1.14.3-9.el8 rhel-8-for-x86_64-appstream-rpms 3.7 M 2022-05-13T23:02:21.869 INFO:teuthology.orchestra.run.smithi115.stdout: python3-numpy-f2py x86_64 1:1.14.3-9.el8 rhel-8-for-x86_64-appstream-rpms 225 k 2022-05-13T23:02:21.869 INFO:teuthology.orchestra.run.smithi115.stdout: python3-oauthlib noarch 2.1.0-1.el8 rhel-8-for-x86_64-baseos-rpms 155 k 2022-05-13T23:02:21.870 INFO:teuthology.orchestra.run.smithi115.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2022-05-13T23:02:21.870 INFO:teuthology.orchestra.run.smithi115.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2022-05-13T23:02:21.870 INFO:teuthology.orchestra.run.smithi115.stdout: python3-prettytable noarch 0.7.2-14.el8 rhel-8-for-x86_64-appstream-rpms 44 k 2022-05-13T23:02:21.870 INFO:teuthology.orchestra.run.smithi115.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 rhel-8-for-x86_64-appstream-rpms 103 k 2022-05-13T23:02:21.871 INFO:teuthology.orchestra.run.smithi115.stdout: python3-pyasn1 noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 126 k 2022-05-13T23:02:21.871 INFO:teuthology.orchestra.run.smithi115.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 110 k 2022-05-13T23:02:21.871 INFO:teuthology.orchestra.run.smithi115.stdout: python3-pycparser noarch 2.14-14.el8 rhel-8-for-x86_64-baseos-rpms 109 k 2022-05-13T23:02:21.871 INFO:teuthology.orchestra.run.smithi115.stdout: python3-pytz noarch 2017.2-9.el8 rhel-8-for-x86_64-appstream-rpms 54 k 2022-05-13T23:02:21.871 INFO:teuthology.orchestra.run.smithi115.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2022-05-13T23:02:21.872 INFO:teuthology.orchestra.run.smithi115.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2022-05-13T23:02:21.872 INFO:teuthology.orchestra.run.smithi115.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2022-05-13T23:02:21.872 INFO:teuthology.orchestra.run.smithi115.stdout: python3-rsa noarch 4.8-1.el8 epel 61 k 2022-05-13T23:02:21.872 INFO:teuthology.orchestra.run.smithi115.stdout: python3-scipy x86_64 1.0.0-20.module+el8.1.0+3334+5cb623d7 rhel-8-for-x86_64-appstream-rpms 15 M 2022-05-13T23:02:21.873 INFO:teuthology.orchestra.run.smithi115.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2022-05-13T23:02:21.873 INFO:teuthology.orchestra.run.smithi115.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2022-05-13T23:02:21.873 INFO:teuthology.orchestra.run.smithi115.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2022-05-13T23:02:21.873 INFO:teuthology.orchestra.run.smithi115.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2022-05-13T23:02:21.873 INFO:teuthology.orchestra.run.smithi115.stdout: python3-waitress noarch 1.2.1-2.el8.1 epel 241 k 2022-05-13T23:02:21.874 INFO:teuthology.orchestra.run.smithi115.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2022-05-13T23:02:21.874 INFO:teuthology.orchestra.run.smithi115.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2022-05-13T23:02:21.874 INFO:teuthology.orchestra.run.smithi115.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2022-05-13T23:02:21.874 INFO:teuthology.orchestra.run.smithi115.stdout: python3-werkzeug noarch 0.12.2-4.el8 rhel-8-for-x86_64-appstream-rpms 457 k 2022-05-13T23:02:21.875 INFO:teuthology.orchestra.run.smithi115.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2022-05-13T23:02:21.875 INFO:teuthology.orchestra.run.smithi115.stdout: socat x86_64 1.7.3.3-2.el8 rhel-8-for-x86_64-appstream-rpms 302 k 2022-05-13T23:02:21.875 INFO:teuthology.orchestra.run.smithi115.stdout: thrift x86_64 0.13.0-2.el8 epel 1.7 M 2022-05-13T23:02:21.875 INFO:teuthology.orchestra.run.smithi115.stdout: xmlstarlet x86_64 1.6.1-11.el8 epel 69 k 2022-05-13T23:02:21.875 INFO:teuthology.orchestra.run.smithi115.stdout: zlib-devel x86_64 1.2.11-17.el8 rhel-8-for-x86_64-baseos-rpms 58 k 2022-05-13T23:02:21.876 INFO:teuthology.orchestra.run.smithi115.stdout: 2022-05-13T23:02:21.876 INFO:teuthology.orchestra.run.smithi115.stdout:Transaction Summary 2022-05-13T23:02:21.876 INFO:teuthology.orchestra.run.smithi115.stdout:====================================================================================================================== 2022-05-13T23:02:21.876 INFO:teuthology.orchestra.run.smithi115.stdout:Install 122 Packages 2022-05-13T23:02:21.877 INFO:teuthology.orchestra.run.smithi115.stdout:Upgrade 6 Packages 2022-05-13T23:02:21.877 INFO:teuthology.orchestra.run.smithi115.stdout: 2022-05-13T23:02:21.878 INFO:teuthology.orchestra.run.smithi115.stdout:Total download size: 207 M 2022-05-13T23:02:21.878 INFO:teuthology.orchestra.run.smithi115.stdout:Downloading Packages: 2022-05-13T23:02:21.966 INFO:teuthology.orchestra.run.smithi115.stdout:(1/128): ceph-17.2.0-188.g23b06583.el8.x86_64.r 60 kB/s | 6.4 kB 00:00 2022-05-13T23:02:22.148 INFO:teuthology.orchestra.run.smithi115.stdout:(2/128): ceph-fuse-17.2.0-188.g23b06583.el8.x86 4.7 MB/s | 876 kB 00:00 2022-05-13T23:02:22.174 INFO:teuthology.orchestra.run.smithi115.stdout:(3/128): ceph-immutable-object-cache-17.2.0-188 6.7 MB/s | 169 kB 00:00 2022-05-13T23:02:22.422 INFO:teuthology.orchestra.run.smithi047.stdout:Package librados2-1:12.2.7-9.el8.x86_64 is already installed. 2022-05-13T23:02:22.423 INFO:teuthology.orchestra.run.smithi047.stdout:Package librbd1-1:12.2.7-9.el8.x86_64 is already installed. 2022-05-13T23:02:22.436 INFO:teuthology.orchestra.run.smithi047.stdout:Package NetworkManager-1:1.30.0-7.el8.x86_64 is already installed. 2022-05-13T23:02:22.436 INFO:teuthology.orchestra.run.smithi047.stdout:Package iproute-5.9.0-4.el8.x86_64 is already installed. 2022-05-13T23:02:22.436 INFO:teuthology.orchestra.run.smithi047.stdout:Package util-linux-2.32.1-27.el8.x86_64 is already installed. 2022-05-13T23:02:22.438 INFO:teuthology.orchestra.run.smithi047.stdout:Package libacl-devel-2.2.53-1.el8.x86_64 is already installed. 2022-05-13T23:02:22.439 INFO:teuthology.orchestra.run.smithi047.stdout:Package libattr-devel-2.4.48-3.el8.x86_64 is already installed. 2022-05-13T23:02:22.440 INFO:teuthology.orchestra.run.smithi115.stdout:(4/128): ceph-mds-17.2.0-188.g23b06583.el8.x86_ 8.5 MB/s | 2.3 MB 00:00 2022-05-13T23:02:22.441 INFO:teuthology.orchestra.run.smithi047.stdout:Package xfsdump-3.1.8-2.el8.x86_64 is already installed. 2022-05-13T23:02:22.442 INFO:teuthology.orchestra.run.smithi047.stdout:Package xfsprogs-5.0.0-8.el8.x86_64 is already installed. 2022-05-13T23:02:22.551 INFO:teuthology.orchestra.run.smithi115.stdout:(5/128): ceph-base-17.2.0-188.g23b06583.el8.x86 7.6 MB/s | 5.3 MB 00:00 2022-05-13T23:02:22.618 INFO:teuthology.orchestra.run.smithi115.stdout:(6/128): ceph-mgr-17.2.0-188.g23b06583.el8.x86_ 8.7 MB/s | 1.5 MB 00:00 2022-05-13T23:02:22.921 INFO:teuthology.orchestra.run.smithi047.stdout:Dependencies resolved. 2022-05-13T23:02:22.934 INFO:teuthology.orchestra.run.smithi047.stdout:====================================================================================================================== 2022-05-13T23:02:22.934 INFO:teuthology.orchestra.run.smithi047.stdout: Package Arch Version Repository Size 2022-05-13T23:02:22.935 INFO:teuthology.orchestra.run.smithi047.stdout:====================================================================================================================== 2022-05-13T23:02:22.935 INFO:teuthology.orchestra.run.smithi047.stdout:Installing: 2022-05-13T23:02:22.935 INFO:teuthology.orchestra.run.smithi047.stdout: bison x86_64 3.0.4-10.el8 rhel-8-for-x86_64-appstream-rpms 688 k 2022-05-13T23:02:22.936 INFO:teuthology.orchestra.run.smithi047.stdout: ceph x86_64 2:17.2.0-188.g23b06583.el8 ceph 6.4 k 2022-05-13T23:02:22.936 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-base x86_64 2:17.2.0-188.g23b06583.el8 ceph 5.3 M 2022-05-13T23:02:22.936 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-fuse x86_64 2:17.2.0-188.g23b06583.el8 ceph 876 k 2022-05-13T23:02:22.936 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-immutable-object-cache x86_64 2:17.2.0-188.g23b06583.el8 ceph 169 k 2022-05-13T23:02:22.937 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-mgr x86_64 2:17.2.0-188.g23b06583.el8 ceph 1.5 M 2022-05-13T23:02:22.937 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-mgr-cephadm noarch 2:17.2.0-188.g23b06583.el8 ceph-noarch 105 k 2022-05-13T23:02:22.937 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-mgr-dashboard noarch 2:17.2.0-188.g23b06583.el8 ceph-noarch 1.6 M 2022-05-13T23:02:22.937 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-mgr-diskprediction-local noarch 2:17.2.0-188.g23b06583.el8 ceph-noarch 7.4 M 2022-05-13T23:02:22.938 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-mgr-rook noarch 2:17.2.0-188.g23b06583.el8 ceph-noarch 47 k 2022-05-13T23:02:22.938 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-radosgw x86_64 2:17.2.0-188.g23b06583.el8 ceph 16 M 2022-05-13T23:02:22.938 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-test x86_64 2:17.2.0-188.g23b06583.el8 ceph 52 M 2022-05-13T23:02:22.940 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-volume noarch 2:17.2.0-188.g23b06583.el8 ceph-noarch 259 k 2022-05-13T23:02:22.940 INFO:teuthology.orchestra.run.smithi047.stdout: cephadm noarch 2:17.2.0-188.g23b06583.el8 ceph-noarch 75 k 2022-05-13T23:02:22.940 INFO:teuthology.orchestra.run.smithi047.stdout: cephfs-mirror x86_64 2:17.2.0-188.g23b06583.el8 ceph 272 k 2022-05-13T23:02:22.941 INFO:teuthology.orchestra.run.smithi047.stdout: cephfs-top noarch 2:17.2.0-188.g23b06583.el8 ceph-noarch 14 k 2022-05-13T23:02:22.941 INFO:teuthology.orchestra.run.smithi047.stdout: elfutils-libelf-devel x86_64 0.182-3.el8 rhel-8-for-x86_64-baseos-rpms 59 k 2022-05-13T23:02:22.942 INFO:teuthology.orchestra.run.smithi047.stdout: flex x86_64 2.6.1-9.el8 rhel-8-for-x86_64-appstream-rpms 320 k 2022-05-13T23:02:22.942 INFO:teuthology.orchestra.run.smithi047.stdout: libaio-devel x86_64 0.3.112-1.el8 rhel-8-for-x86_64-baseos-rpms 19 k 2022-05-13T23:02:22.942 INFO:teuthology.orchestra.run.smithi047.stdout: libcephfs-devel x86_64 2:17.2.0-188.g23b06583.el8 ceph 24 k 2022-05-13T23:02:22.942 INFO:teuthology.orchestra.run.smithi047.stdout: libcephfs2 x86_64 2:17.2.0-188.g23b06583.el8 ceph 790 k 2022-05-13T23:02:22.943 INFO:teuthology.orchestra.run.smithi047.stdout: librados-devel x86_64 2:17.2.0-188.g23b06583.el8 ceph 125 k 2022-05-13T23:02:22.943 INFO:teuthology.orchestra.run.smithi047.stdout: libtool x86_64 2.4.6-25.el8 rhel-8-for-x86_64-appstream-rpms 709 k 2022-05-13T23:02:22.943 INFO:teuthology.orchestra.run.smithi047.stdout: libuuid-devel x86_64 2.32.1-27.el8 rhel-8-for-x86_64-baseos-rpms 97 k 2022-05-13T23:02:22.944 INFO:teuthology.orchestra.run.smithi047.stdout: openssl-devel x86_64 1:1.1.1g-15.el8_3 rhel-8-for-x86_64-baseos-rpms 2.3 M 2022-05-13T23:02:22.944 INFO:teuthology.orchestra.run.smithi047.stdout: python3-cephfs x86_64 2:17.2.0-188.g23b06583.el8 ceph 178 k 2022-05-13T23:02:22.944 INFO:teuthology.orchestra.run.smithi047.stdout: python3-rados x86_64 2:17.2.0-188.g23b06583.el8 ceph 367 k 2022-05-13T23:02:22.944 INFO:teuthology.orchestra.run.smithi047.stdout: python3-rbd x86_64 2:17.2.0-188.g23b06583.el8 ceph 346 k 2022-05-13T23:02:22.945 INFO:teuthology.orchestra.run.smithi047.stdout: python3-rgw x86_64 2:17.2.0-188.g23b06583.el8 ceph 109 k 2022-05-13T23:02:22.945 INFO:teuthology.orchestra.run.smithi047.stdout: rbd-fuse x86_64 2:17.2.0-188.g23b06583.el8 ceph 93 k 2022-05-13T23:02:22.946 INFO:teuthology.orchestra.run.smithi047.stdout: rbd-mirror x86_64 2:17.2.0-188.g23b06583.el8 ceph 3.8 M 2022-05-13T23:02:22.946 INFO:teuthology.orchestra.run.smithi047.stdout: rbd-nbd x86_64 2:17.2.0-188.g23b06583.el8 ceph 175 k 2022-05-13T23:02:22.947 INFO:teuthology.orchestra.run.smithi047.stdout: xfsprogs-devel x86_64 5.0.0-8.el8 rhel-8-for-x86_64-baseos-rpms 70 k 2022-05-13T23:02:22.947 INFO:teuthology.orchestra.run.smithi047.stdout:Upgrading: 2022-05-13T23:02:22.947 INFO:teuthology.orchestra.run.smithi047.stdout: NetworkManager x86_64 1:1.30.0-13.el8_4 rhel-8-for-x86_64-baseos-rpms 2.6 M 2022-05-13T23:02:22.947 INFO:teuthology.orchestra.run.smithi047.stdout: NetworkManager-libnm x86_64 1:1.30.0-13.el8_4 rhel-8-for-x86_64-baseos-rpms 1.8 M 2022-05-13T23:02:22.948 INFO:teuthology.orchestra.run.smithi047.stdout: NetworkManager-team x86_64 1:1.30.0-13.el8_4 rhel-8-for-x86_64-baseos-rpms 146 k 2022-05-13T23:02:22.948 INFO:teuthology.orchestra.run.smithi047.stdout: NetworkManager-tui x86_64 1:1.30.0-13.el8_4 rhel-8-for-x86_64-baseos-rpms 329 k 2022-05-13T23:02:22.948 INFO:teuthology.orchestra.run.smithi047.stdout: librados2 x86_64 2:17.2.0-188.g23b06583.el8 ceph 3.4 M 2022-05-13T23:02:22.949 INFO:teuthology.orchestra.run.smithi047.stdout: librbd1 x86_64 2:17.2.0-188.g23b06583.el8 ceph 3.9 M 2022-05-13T23:02:22.950 INFO:teuthology.orchestra.run.smithi047.stdout:Installing dependencies: 2022-05-13T23:02:22.950 INFO:teuthology.orchestra.run.smithi047.stdout: automake noarch 1.16.1-7.el8 rhel-8-for-x86_64-appstream-rpms 713 k 2022-05-13T23:02:22.950 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-common x86_64 2:17.2.0-188.g23b06583.el8 ceph 23 M 2022-05-13T23:02:22.950 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-grafana-dashboards noarch 2:17.2.0-188.g23b06583.el8 ceph-noarch 23 k 2022-05-13T23:02:22.951 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-mds x86_64 2:17.2.0-188.g23b06583.el8 ceph 2.3 M 2022-05-13T23:02:22.951 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-mgr-modules-core noarch 2:17.2.0-188.g23b06583.el8 ceph-noarch 228 k 2022-05-13T23:02:22.951 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-mon x86_64 2:17.2.0-188.g23b06583.el8 ceph 3.9 M 2022-05-13T23:02:22.951 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-osd x86_64 2:17.2.0-188.g23b06583.el8 ceph 17 M 2022-05-13T23:02:22.952 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-prometheus-alerts noarch 2:17.2.0-188.g23b06583.el8 ceph-noarch 15 k 2022-05-13T23:02:22.952 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-selinux x86_64 2:17.2.0-188.g23b06583.el8 ceph 24 k 2022-05-13T23:02:22.952 INFO:teuthology.orchestra.run.smithi047.stdout: fuse x86_64 2.9.7-12.el8 rhel-8-for-x86_64-baseos-rpms 83 k 2022-05-13T23:02:22.953 INFO:teuthology.orchestra.run.smithi047.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2022-05-13T23:02:22.953 INFO:teuthology.orchestra.run.smithi047.stdout: jq x86_64 1.5-12.el8 rhel-8-for-x86_64-appstream-rpms 161 k 2022-05-13T23:02:22.954 INFO:teuthology.orchestra.run.smithi047.stdout: keyutils-libs-devel x86_64 1.5.10-6.el8 rhel-8-for-x86_64-baseos-rpms 48 k 2022-05-13T23:02:22.955 INFO:teuthology.orchestra.run.smithi047.stdout: krb5-devel x86_64 1.18.2-8.3.el8_4 rhel-8-for-x86_64-baseos-rpms 559 k 2022-05-13T23:02:22.955 INFO:teuthology.orchestra.run.smithi047.stdout: libcephsqlite x86_64 2:17.2.0-188.g23b06583.el8 ceph 170 k 2022-05-13T23:02:22.955 INFO:teuthology.orchestra.run.smithi047.stdout: libcom_err-devel x86_64 1.45.6-1.el8 rhel-8-for-x86_64-baseos-rpms 38 k 2022-05-13T23:02:22.955 INFO:teuthology.orchestra.run.smithi047.stdout: libgfortran x86_64 8.4.1-1.el8 rhel-8-for-x86_64-baseos-rpms 639 k 2022-05-13T23:02:22.956 INFO:teuthology.orchestra.run.smithi047.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2022-05-13T23:02:22.956 INFO:teuthology.orchestra.run.smithi047.stdout: libquadmath x86_64 8.4.1-1.el8 rhel-8-for-x86_64-baseos-rpms 169 k 2022-05-13T23:02:22.956 INFO:teuthology.orchestra.run.smithi047.stdout: librabbitmq x86_64 0.9.0-3.el8 rhel-8-for-x86_64-baseos-rpms 47 k 2022-05-13T23:02:22.957 INFO:teuthology.orchestra.run.smithi047.stdout: libradosstriper1 x86_64 2:17.2.0-188.g23b06583.el8 ceph 483 k 2022-05-13T23:02:22.957 INFO:teuthology.orchestra.run.smithi047.stdout: librdkafka x86_64 0.11.4-1.el8 rhel-8-for-x86_64-appstream-rpms 353 k 2022-05-13T23:02:22.957 INFO:teuthology.orchestra.run.smithi047.stdout: librgw2 x86_64 2:17.2.0-188.g23b06583.el8 ceph 5.3 M 2022-05-13T23:02:22.957 INFO:teuthology.orchestra.run.smithi047.stdout: libselinux-devel x86_64 2.9-5.el8 rhel-8-for-x86_64-baseos-rpms 200 k 2022-05-13T23:02:22.958 INFO:teuthology.orchestra.run.smithi047.stdout: libsepol-devel x86_64 2.9-2.el8 rhel-8-for-x86_64-baseos-rpms 86 k 2022-05-13T23:02:22.958 INFO:teuthology.orchestra.run.smithi047.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2022-05-13T23:02:22.958 INFO:teuthology.orchestra.run.smithi047.stdout: libverto-devel x86_64 0.3.0-5.el8 rhel-8-for-x86_64-baseos-rpms 18 k 2022-05-13T23:02:22.958 INFO:teuthology.orchestra.run.smithi047.stdout: lttng-ust x86_64 2.8.1-11.el8 rhel-8-for-x86_64-appstream-rpms 259 k 2022-05-13T23:02:22.958 INFO:teuthology.orchestra.run.smithi047.stdout: oniguruma x86_64 6.8.2-2.el8 rhel-8-for-x86_64-appstream-rpms 187 k 2022-05-13T23:02:22.959 INFO:teuthology.orchestra.run.smithi047.stdout: openblas x86_64 0.3.12-1.el8 rhel-8-for-x86_64-appstream-rpms 4.6 M 2022-05-13T23:02:22.959 INFO:teuthology.orchestra.run.smithi047.stdout: openblas-threads x86_64 0.3.12-1.el8 rhel-8-for-x86_64-appstream-rpms 4.7 M 2022-05-13T23:02:22.959 INFO:teuthology.orchestra.run.smithi047.stdout: pcre2-devel x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 605 k 2022-05-13T23:02:22.959 INFO:teuthology.orchestra.run.smithi047.stdout: pcre2-utf16 x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 229 k 2022-05-13T23:02:22.960 INFO:teuthology.orchestra.run.smithi047.stdout: pcre2-utf32 x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 220 k 2022-05-13T23:02:22.960 INFO:teuthology.orchestra.run.smithi047.stdout: python3-asyncssh noarch 2.7.0-2.el8 epel 476 k 2022-05-13T23:02:22.960 INFO:teuthology.orchestra.run.smithi047.stdout: python3-babel noarch 2.5.1-5.el8 rhel-8-for-x86_64-appstream-rpms 4.8 M 2022-05-13T23:02:22.960 INFO:teuthology.orchestra.run.smithi047.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2022-05-13T23:02:22.960 INFO:teuthology.orchestra.run.smithi047.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2022-05-13T23:02:22.961 INFO:teuthology.orchestra.run.smithi047.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2022-05-13T23:02:22.961 INFO:teuthology.orchestra.run.smithi047.stdout: python3-ceph-argparse x86_64 2:17.2.0-188.g23b06583.el8 ceph 45 k 2022-05-13T23:02:22.961 INFO:teuthology.orchestra.run.smithi047.stdout: python3-ceph-common x86_64 2:17.2.0-188.g23b06583.el8 ceph 106 k 2022-05-13T23:02:22.961 INFO:teuthology.orchestra.run.smithi047.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2022-05-13T23:02:22.962 INFO:teuthology.orchestra.run.smithi047.stdout: python3-cffi x86_64 1.11.5-5.el8 rhel-8-for-x86_64-baseos-rpms 238 k 2022-05-13T23:02:22.962 INFO:teuthology.orchestra.run.smithi047.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2022-05-13T23:02:22.962 INFO:teuthology.orchestra.run.smithi047.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2022-05-13T23:02:22.962 INFO:teuthology.orchestra.run.smithi047.stdout: python3-cryptography x86_64 3.2.1-4.el8 rhel-8-for-x86_64-baseos-rpms 559 k 2022-05-13T23:02:22.962 INFO:teuthology.orchestra.run.smithi047.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2022-05-13T23:02:22.963 INFO:teuthology.orchestra.run.smithi047.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2022-05-13T23:02:22.963 INFO:teuthology.orchestra.run.smithi047.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2022-05-13T23:02:22.963 INFO:teuthology.orchestra.run.smithi047.stdout: python3-jinja2 noarch 2.10.1-2.el8_0 rhel-8-for-x86_64-appstream-rpms 538 k 2022-05-13T23:02:22.963 INFO:teuthology.orchestra.run.smithi047.stdout: python3-jsonpatch noarch 1.21-2.el8 rhel-8-for-x86_64-appstream-rpms 27 k 2022-05-13T23:02:22.964 INFO:teuthology.orchestra.run.smithi047.stdout: python3-jsonpointer noarch 1.10-11.el8 rhel-8-for-x86_64-appstream-rpms 20 k 2022-05-13T23:02:22.965 INFO:teuthology.orchestra.run.smithi047.stdout: python3-jwt noarch 1.6.1-2.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2022-05-13T23:02:22.966 INFO:teuthology.orchestra.run.smithi047.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2022-05-13T23:02:22.966 INFO:teuthology.orchestra.run.smithi047.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2022-05-13T23:02:22.966 INFO:teuthology.orchestra.run.smithi047.stdout: python3-mako noarch 1.0.6-13.el8 rhel-8-for-x86_64-appstream-rpms 157 k 2022-05-13T23:02:22.967 INFO:teuthology.orchestra.run.smithi047.stdout: python3-markupsafe x86_64 0.23-19.el8 rhel-8-for-x86_64-appstream-rpms 39 k 2022-05-13T23:02:22.967 INFO:teuthology.orchestra.run.smithi047.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2022-05-13T23:02:22.968 INFO:teuthology.orchestra.run.smithi047.stdout: python3-natsort noarch 7.1.1-2.el8 epel 60 k 2022-05-13T23:02:22.969 INFO:teuthology.orchestra.run.smithi047.stdout: python3-numpy x86_64 1:1.14.3-9.el8 rhel-8-for-x86_64-appstream-rpms 3.7 M 2022-05-13T23:02:22.970 INFO:teuthology.orchestra.run.smithi047.stdout: python3-numpy-f2py x86_64 1:1.14.3-9.el8 rhel-8-for-x86_64-appstream-rpms 225 k 2022-05-13T23:02:22.970 INFO:teuthology.orchestra.run.smithi047.stdout: python3-oauthlib noarch 2.1.0-1.el8 rhel-8-for-x86_64-baseos-rpms 155 k 2022-05-13T23:02:22.970 INFO:teuthology.orchestra.run.smithi047.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2022-05-13T23:02:22.971 INFO:teuthology.orchestra.run.smithi047.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2022-05-13T23:02:22.971 INFO:teuthology.orchestra.run.smithi047.stdout: python3-prettytable noarch 0.7.2-14.el8 rhel-8-for-x86_64-appstream-rpms 44 k 2022-05-13T23:02:22.971 INFO:teuthology.orchestra.run.smithi047.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 rhel-8-for-x86_64-appstream-rpms 103 k 2022-05-13T23:02:22.971 INFO:teuthology.orchestra.run.smithi047.stdout: python3-pyasn1 noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 126 k 2022-05-13T23:02:22.972 INFO:teuthology.orchestra.run.smithi047.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 110 k 2022-05-13T23:02:22.972 INFO:teuthology.orchestra.run.smithi047.stdout: python3-pycparser noarch 2.14-14.el8 rhel-8-for-x86_64-baseos-rpms 109 k 2022-05-13T23:02:22.972 INFO:teuthology.orchestra.run.smithi047.stdout: python3-pytz noarch 2017.2-9.el8 rhel-8-for-x86_64-appstream-rpms 54 k 2022-05-13T23:02:22.972 INFO:teuthology.orchestra.run.smithi047.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2022-05-13T23:02:22.973 INFO:teuthology.orchestra.run.smithi047.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2022-05-13T23:02:22.973 INFO:teuthology.orchestra.run.smithi047.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2022-05-13T23:02:22.973 INFO:teuthology.orchestra.run.smithi047.stdout: python3-rsa noarch 4.8-1.el8 epel 61 k 2022-05-13T23:02:22.973 INFO:teuthology.orchestra.run.smithi047.stdout: python3-scipy x86_64 1.0.0-20.module+el8.1.0+3334+5cb623d7 rhel-8-for-x86_64-appstream-rpms 15 M 2022-05-13T23:02:22.974 INFO:teuthology.orchestra.run.smithi047.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2022-05-13T23:02:22.974 INFO:teuthology.orchestra.run.smithi047.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2022-05-13T23:02:22.974 INFO:teuthology.orchestra.run.smithi047.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2022-05-13T23:02:22.974 INFO:teuthology.orchestra.run.smithi047.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2022-05-13T23:02:22.975 INFO:teuthology.orchestra.run.smithi047.stdout: python3-waitress noarch 1.2.1-2.el8.1 epel 241 k 2022-05-13T23:02:22.975 INFO:teuthology.orchestra.run.smithi047.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2022-05-13T23:02:22.975 INFO:teuthology.orchestra.run.smithi047.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2022-05-13T23:02:22.975 INFO:teuthology.orchestra.run.smithi047.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2022-05-13T23:02:22.976 INFO:teuthology.orchestra.run.smithi047.stdout: python3-werkzeug noarch 0.12.2-4.el8 rhel-8-for-x86_64-appstream-rpms 457 k 2022-05-13T23:02:22.976 INFO:teuthology.orchestra.run.smithi047.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2022-05-13T23:02:22.976 INFO:teuthology.orchestra.run.smithi047.stdout: socat x86_64 1.7.3.3-2.el8 rhel-8-for-x86_64-appstream-rpms 302 k 2022-05-13T23:02:22.976 INFO:teuthology.orchestra.run.smithi047.stdout: thrift x86_64 0.13.0-2.el8 epel 1.7 M 2022-05-13T23:02:22.976 INFO:teuthology.orchestra.run.smithi047.stdout: xmlstarlet x86_64 1.6.1-11.el8 epel 69 k 2022-05-13T23:02:22.977 INFO:teuthology.orchestra.run.smithi047.stdout: zlib-devel x86_64 1.2.11-17.el8 rhel-8-for-x86_64-baseos-rpms 58 k 2022-05-13T23:02:22.977 INFO:teuthology.orchestra.run.smithi047.stdout: 2022-05-13T23:02:22.977 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction Summary 2022-05-13T23:02:22.977 INFO:teuthology.orchestra.run.smithi047.stdout:====================================================================================================================== 2022-05-13T23:02:22.978 INFO:teuthology.orchestra.run.smithi047.stdout:Install 122 Packages 2022-05-13T23:02:22.978 INFO:teuthology.orchestra.run.smithi047.stdout:Upgrade 6 Packages 2022-05-13T23:02:22.978 INFO:teuthology.orchestra.run.smithi047.stdout: 2022-05-13T23:02:22.979 INFO:teuthology.orchestra.run.smithi115.stdout:(7/128): ceph-mon-17.2.0-188.g23b06583.el8.x86_ 9.8 MB/s | 3.9 MB 00:00 2022-05-13T23:02:22.979 INFO:teuthology.orchestra.run.smithi047.stdout:Total download size: 207 M 2022-05-13T23:02:22.980 INFO:teuthology.orchestra.run.smithi047.stdout:Downloading Packages: 2022-05-13T23:02:23.229 INFO:teuthology.orchestra.run.smithi047.stdout:(1/128): ceph-17.2.0-188.g23b06583.el8.x86_64.r 23 kB/s | 6.4 kB 00:00 2022-05-13T23:02:24.402 INFO:teuthology.orchestra.run.smithi115.stdout:(8/128): ceph-common-17.2.0-188.g23b06583.el8.x 9.1 MB/s | 23 MB 00:02 2022-05-13T23:02:24.454 INFO:teuthology.orchestra.run.smithi115.stdout:(9/128): ceph-selinux-17.2.0-188.g23b06583.el8. 467 kB/s | 24 kB 00:00 2022-05-13T23:02:24.676 INFO:teuthology.orchestra.run.smithi115.stdout:(10/128): ceph-radosgw-17.2.0-188.g23b06583.el8 9.3 MB/s | 16 MB 00:01 2022-05-13T23:02:24.743 INFO:teuthology.orchestra.run.smithi047.stdout:(2/128): ceph-fuse-17.2.0-188.g23b06583.el8.x86 579 kB/s | 876 kB 00:01 2022-05-13T23:02:24.803 INFO:teuthology.orchestra.run.smithi115.stdout:(11/128): ceph-osd-17.2.0-188.g23b06583.el8.x86 7.7 MB/s | 17 MB 00:02 2022-05-13T23:02:24.805 INFO:teuthology.orchestra.run.smithi115.stdout:(12/128): cephfs-mirror-17.2.0-188.g23b06583.el 2.1 MB/s | 272 kB 00:00 2022-05-13T23:02:24.829 INFO:teuthology.orchestra.run.smithi115.stdout:(13/128): libcephfs-devel-17.2.0-188.g23b06583. 939 kB/s | 24 kB 00:00 2022-05-13T23:02:24.882 INFO:teuthology.orchestra.run.smithi047.stdout:(3/128): ceph-immutable-object-cache-17.2.0-188 1.2 MB/s | 169 kB 00:00 2022-05-13T23:02:24.890 INFO:teuthology.orchestra.run.smithi115.stdout:(14/128): libcephsqlite-17.2.0-188.g23b06583.el 2.8 MB/s | 170 kB 00:00 2022-05-13T23:02:24.896 INFO:teuthology.orchestra.run.smithi115.stdout:(15/128): libcephfs2-17.2.0-188.g23b06583.el8.x 8.6 MB/s | 790 kB 00:00 2022-05-13T23:02:24.928 INFO:teuthology.orchestra.run.smithi115.stdout:(16/128): librados-devel-17.2.0-188.g23b06583.e 3.2 MB/s | 125 kB 00:00 2022-05-13T23:02:24.949 INFO:teuthology.orchestra.run.smithi115.stdout:(17/128): libradosstriper1-17.2.0-188.g23b06583 8.9 MB/s | 483 kB 00:00 2022-05-13T23:02:24.999 INFO:teuthology.orchestra.run.smithi115.stdout:(18/128): python3-ceph-argparse-17.2.0-188.g23b 899 kB/s | 45 kB 00:00 2022-05-13T23:02:25.054 INFO:teuthology.orchestra.run.smithi115.stdout:(19/128): python3-ceph-common-17.2.0-188.g23b06 1.9 MB/s | 106 kB 00:00 2022-05-13T23:02:25.112 INFO:teuthology.orchestra.run.smithi115.stdout:(20/128): python3-cephfs-17.2.0-188.g23b06583.e 3.0 MB/s | 178 kB 00:00 2022-05-13T23:02:25.145 INFO:teuthology.orchestra.run.smithi002.stdout:Package librados2-1:12.2.7-9.el8.x86_64 is already installed. 2022-05-13T23:02:25.146 INFO:teuthology.orchestra.run.smithi002.stdout:Package librbd1-1:12.2.7-9.el8.x86_64 is already installed. 2022-05-13T23:02:25.157 INFO:teuthology.orchestra.run.smithi002.stdout:Package NetworkManager-1:1.30.0-7.el8.x86_64 is already installed. 2022-05-13T23:02:25.158 INFO:teuthology.orchestra.run.smithi002.stdout:Package iproute-5.9.0-4.el8.x86_64 is already installed. 2022-05-13T23:02:25.159 INFO:teuthology.orchestra.run.smithi002.stdout:Package util-linux-2.32.1-27.el8.x86_64 is already installed. 2022-05-13T23:02:25.160 INFO:teuthology.orchestra.run.smithi002.stdout:Package libacl-devel-2.2.53-1.el8.x86_64 is already installed. 2022-05-13T23:02:25.161 INFO:teuthology.orchestra.run.smithi002.stdout:Package libattr-devel-2.4.48-3.el8.x86_64 is already installed. 2022-05-13T23:02:25.164 INFO:teuthology.orchestra.run.smithi002.stdout:Package xfsdump-3.1.8-2.el8.x86_64 is already installed. 2022-05-13T23:02:25.165 INFO:teuthology.orchestra.run.smithi002.stdout:Package xfsprogs-5.0.0-8.el8.x86_64 is already installed. 2022-05-13T23:02:25.177 INFO:teuthology.orchestra.run.smithi115.stdout:(21/128): python3-rados-17.2.0-188.g23b06583.el 5.5 MB/s | 367 kB 00:00 2022-05-13T23:02:25.242 INFO:teuthology.orchestra.run.smithi115.stdout:(22/128): python3-rbd-17.2.0-188.g23b06583.el8. 5.2 MB/s | 346 kB 00:00 2022-05-13T23:02:25.299 INFO:teuthology.orchestra.run.smithi115.stdout:(23/128): python3-rgw-17.2.0-188.g23b06583.el8. 1.9 MB/s | 109 kB 00:00 2022-05-13T23:02:25.355 INFO:teuthology.orchestra.run.smithi115.stdout:(24/128): rbd-fuse-17.2.0-188.g23b06583.el8.x86 1.6 MB/s | 93 kB 00:00 2022-05-13T23:02:25.544 INFO:teuthology.orchestra.run.smithi115.stdout:(25/128): librgw2-17.2.0-188.g23b06583.el8.x86_ 8.6 MB/s | 5.3 MB 00:00 2022-05-13T23:02:25.608 INFO:teuthology.orchestra.run.smithi115.stdout:(26/128): rbd-nbd-17.2.0-188.g23b06583.el8.x86_ 2.7 MB/s | 175 kB 00:00 2022-05-13T23:02:25.662 INFO:teuthology.orchestra.run.smithi002.stdout:Dependencies resolved. 2022-05-13T23:02:25.668 INFO:teuthology.orchestra.run.smithi115.stdout:(27/128): ceph-grafana-dashboards-17.2.0-188.g2 383 kB/s | 23 kB 00:00 2022-05-13T23:02:25.675 INFO:teuthology.orchestra.run.smithi002.stdout:====================================================================================================================== 2022-05-13T23:02:25.676 INFO:teuthology.orchestra.run.smithi002.stdout: Package Arch Version Repository Size 2022-05-13T23:02:25.676 INFO:teuthology.orchestra.run.smithi002.stdout:====================================================================================================================== 2022-05-13T23:02:25.676 INFO:teuthology.orchestra.run.smithi002.stdout:Installing: 2022-05-13T23:02:25.677 INFO:teuthology.orchestra.run.smithi002.stdout: bison x86_64 3.0.4-10.el8 rhel-8-for-x86_64-appstream-rpms 688 k 2022-05-13T23:02:25.677 INFO:teuthology.orchestra.run.smithi002.stdout: ceph x86_64 2:17.2.0-188.g23b06583.el8 ceph 6.4 k 2022-05-13T23:02:25.677 INFO:teuthology.orchestra.run.smithi002.stdout: ceph-base x86_64 2:17.2.0-188.g23b06583.el8 ceph 5.3 M 2022-05-13T23:02:25.677 INFO:teuthology.orchestra.run.smithi002.stdout: ceph-fuse x86_64 2:17.2.0-188.g23b06583.el8 ceph 876 k 2022-05-13T23:02:25.677 INFO:teuthology.orchestra.run.smithi002.stdout: ceph-immutable-object-cache x86_64 2:17.2.0-188.g23b06583.el8 ceph 169 k 2022-05-13T23:02:25.678 INFO:teuthology.orchestra.run.smithi002.stdout: ceph-mgr x86_64 2:17.2.0-188.g23b06583.el8 ceph 1.5 M 2022-05-13T23:02:25.678 INFO:teuthology.orchestra.run.smithi002.stdout: ceph-mgr-cephadm noarch 2:17.2.0-188.g23b06583.el8 ceph-noarch 105 k 2022-05-13T23:02:25.678 INFO:teuthology.orchestra.run.smithi002.stdout: ceph-mgr-dashboard noarch 2:17.2.0-188.g23b06583.el8 ceph-noarch 1.6 M 2022-05-13T23:02:25.678 INFO:teuthology.orchestra.run.smithi002.stdout: ceph-mgr-diskprediction-local noarch 2:17.2.0-188.g23b06583.el8 ceph-noarch 7.4 M 2022-05-13T23:02:25.679 INFO:teuthology.orchestra.run.smithi002.stdout: ceph-mgr-rook noarch 2:17.2.0-188.g23b06583.el8 ceph-noarch 47 k 2022-05-13T23:02:25.679 INFO:teuthology.orchestra.run.smithi002.stdout: ceph-radosgw x86_64 2:17.2.0-188.g23b06583.el8 ceph 16 M 2022-05-13T23:02:25.679 INFO:teuthology.orchestra.run.smithi002.stdout: ceph-test x86_64 2:17.2.0-188.g23b06583.el8 ceph 52 M 2022-05-13T23:02:25.679 INFO:teuthology.orchestra.run.smithi002.stdout: ceph-volume noarch 2:17.2.0-188.g23b06583.el8 ceph-noarch 259 k 2022-05-13T23:02:25.679 INFO:teuthology.orchestra.run.smithi002.stdout: cephadm noarch 2:17.2.0-188.g23b06583.el8 ceph-noarch 75 k 2022-05-13T23:02:25.680 INFO:teuthology.orchestra.run.smithi002.stdout: cephfs-mirror x86_64 2:17.2.0-188.g23b06583.el8 ceph 272 k 2022-05-13T23:02:25.680 INFO:teuthology.orchestra.run.smithi002.stdout: cephfs-top noarch 2:17.2.0-188.g23b06583.el8 ceph-noarch 14 k 2022-05-13T23:02:25.680 INFO:teuthology.orchestra.run.smithi002.stdout: elfutils-libelf-devel x86_64 0.182-3.el8 rhel-8-for-x86_64-baseos-rpms 59 k 2022-05-13T23:02:25.680 INFO:teuthology.orchestra.run.smithi002.stdout: flex x86_64 2.6.1-9.el8 rhel-8-for-x86_64-appstream-rpms 320 k 2022-05-13T23:02:25.680 INFO:teuthology.orchestra.run.smithi002.stdout: libaio-devel x86_64 0.3.112-1.el8 rhel-8-for-x86_64-baseos-rpms 19 k 2022-05-13T23:02:25.681 INFO:teuthology.orchestra.run.smithi002.stdout: libcephfs-devel x86_64 2:17.2.0-188.g23b06583.el8 ceph 24 k 2022-05-13T23:02:25.681 INFO:teuthology.orchestra.run.smithi002.stdout: libcephfs2 x86_64 2:17.2.0-188.g23b06583.el8 ceph 790 k 2022-05-13T23:02:25.685 INFO:teuthology.orchestra.run.smithi002.stdout: librados-devel x86_64 2:17.2.0-188.g23b06583.el8 ceph 125 k 2022-05-13T23:02:25.686 INFO:teuthology.orchestra.run.smithi002.stdout: libtool x86_64 2.4.6-25.el8 rhel-8-for-x86_64-appstream-rpms 709 k 2022-05-13T23:02:25.686 INFO:teuthology.orchestra.run.smithi002.stdout: libuuid-devel x86_64 2.32.1-27.el8 rhel-8-for-x86_64-baseos-rpms 97 k 2022-05-13T23:02:25.686 INFO:teuthology.orchestra.run.smithi002.stdout: openssl-devel x86_64 1:1.1.1g-15.el8_3 rhel-8-for-x86_64-baseos-rpms 2.3 M 2022-05-13T23:02:25.686 INFO:teuthology.orchestra.run.smithi002.stdout: python3-cephfs x86_64 2:17.2.0-188.g23b06583.el8 ceph 178 k 2022-05-13T23:02:25.687 INFO:teuthology.orchestra.run.smithi002.stdout: python3-rados x86_64 2:17.2.0-188.g23b06583.el8 ceph 367 k 2022-05-13T23:02:25.687 INFO:teuthology.orchestra.run.smithi002.stdout: python3-rbd x86_64 2:17.2.0-188.g23b06583.el8 ceph 346 k 2022-05-13T23:02:25.687 INFO:teuthology.orchestra.run.smithi002.stdout: python3-rgw x86_64 2:17.2.0-188.g23b06583.el8 ceph 109 k 2022-05-13T23:02:25.687 INFO:teuthology.orchestra.run.smithi002.stdout: rbd-fuse x86_64 2:17.2.0-188.g23b06583.el8 ceph 93 k 2022-05-13T23:02:25.689 INFO:teuthology.orchestra.run.smithi002.stdout: rbd-mirror x86_64 2:17.2.0-188.g23b06583.el8 ceph 3.8 M 2022-05-13T23:02:25.690 INFO:teuthology.orchestra.run.smithi002.stdout: rbd-nbd x86_64 2:17.2.0-188.g23b06583.el8 ceph 175 k 2022-05-13T23:02:25.690 INFO:teuthology.orchestra.run.smithi002.stdout: xfsprogs-devel x86_64 5.0.0-8.el8 rhel-8-for-x86_64-baseos-rpms 70 k 2022-05-13T23:02:25.690 INFO:teuthology.orchestra.run.smithi002.stdout:Upgrading: 2022-05-13T23:02:25.690 INFO:teuthology.orchestra.run.smithi002.stdout: NetworkManager x86_64 1:1.30.0-13.el8_4 rhel-8-for-x86_64-baseos-rpms 2.6 M 2022-05-13T23:02:25.691 INFO:teuthology.orchestra.run.smithi002.stdout: NetworkManager-libnm x86_64 1:1.30.0-13.el8_4 rhel-8-for-x86_64-baseos-rpms 1.8 M 2022-05-13T23:02:25.691 INFO:teuthology.orchestra.run.smithi002.stdout: NetworkManager-team x86_64 1:1.30.0-13.el8_4 rhel-8-for-x86_64-baseos-rpms 146 k 2022-05-13T23:02:25.691 INFO:teuthology.orchestra.run.smithi002.stdout: NetworkManager-tui x86_64 1:1.30.0-13.el8_4 rhel-8-for-x86_64-baseos-rpms 329 k 2022-05-13T23:02:25.691 INFO:teuthology.orchestra.run.smithi002.stdout: librados2 x86_64 2:17.2.0-188.g23b06583.el8 ceph 3.4 M 2022-05-13T23:02:25.691 INFO:teuthology.orchestra.run.smithi002.stdout: librbd1 x86_64 2:17.2.0-188.g23b06583.el8 ceph 3.9 M 2022-05-13T23:02:25.692 INFO:teuthology.orchestra.run.smithi002.stdout:Installing dependencies: 2022-05-13T23:02:25.692 INFO:teuthology.orchestra.run.smithi002.stdout: automake noarch 1.16.1-7.el8 rhel-8-for-x86_64-appstream-rpms 713 k 2022-05-13T23:02:25.692 INFO:teuthology.orchestra.run.smithi002.stdout: ceph-common x86_64 2:17.2.0-188.g23b06583.el8 ceph 23 M 2022-05-13T23:02:25.692 INFO:teuthology.orchestra.run.smithi002.stdout: ceph-grafana-dashboards noarch 2:17.2.0-188.g23b06583.el8 ceph-noarch 23 k 2022-05-13T23:02:25.692 INFO:teuthology.orchestra.run.smithi002.stdout: ceph-mds x86_64 2:17.2.0-188.g23b06583.el8 ceph 2.3 M 2022-05-13T23:02:25.693 INFO:teuthology.orchestra.run.smithi002.stdout: ceph-mgr-modules-core noarch 2:17.2.0-188.g23b06583.el8 ceph-noarch 228 k 2022-05-13T23:02:25.694 INFO:teuthology.orchestra.run.smithi002.stdout: ceph-mon x86_64 2:17.2.0-188.g23b06583.el8 ceph 3.9 M 2022-05-13T23:02:25.694 INFO:teuthology.orchestra.run.smithi002.stdout: ceph-osd x86_64 2:17.2.0-188.g23b06583.el8 ceph 17 M 2022-05-13T23:02:25.694 INFO:teuthology.orchestra.run.smithi002.stdout: ceph-prometheus-alerts noarch 2:17.2.0-188.g23b06583.el8 ceph-noarch 15 k 2022-05-13T23:02:25.694 INFO:teuthology.orchestra.run.smithi002.stdout: ceph-selinux x86_64 2:17.2.0-188.g23b06583.el8 ceph 24 k 2022-05-13T23:02:25.695 INFO:teuthology.orchestra.run.smithi002.stdout: fuse x86_64 2.9.7-12.el8 rhel-8-for-x86_64-baseos-rpms 83 k 2022-05-13T23:02:25.695 INFO:teuthology.orchestra.run.smithi002.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2022-05-13T23:02:25.695 INFO:teuthology.orchestra.run.smithi002.stdout: jq x86_64 1.5-12.el8 rhel-8-for-x86_64-appstream-rpms 161 k 2022-05-13T23:02:25.695 INFO:teuthology.orchestra.run.smithi002.stdout: keyutils-libs-devel x86_64 1.5.10-6.el8 rhel-8-for-x86_64-baseos-rpms 48 k 2022-05-13T23:02:25.695 INFO:teuthology.orchestra.run.smithi002.stdout: krb5-devel x86_64 1.18.2-8.3.el8_4 rhel-8-for-x86_64-baseos-rpms 559 k 2022-05-13T23:02:25.696 INFO:teuthology.orchestra.run.smithi002.stdout: libcephsqlite x86_64 2:17.2.0-188.g23b06583.el8 ceph 170 k 2022-05-13T23:02:25.696 INFO:teuthology.orchestra.run.smithi002.stdout: libcom_err-devel x86_64 1.45.6-1.el8 rhel-8-for-x86_64-baseos-rpms 38 k 2022-05-13T23:02:25.696 INFO:teuthology.orchestra.run.smithi002.stdout: libgfortran x86_64 8.4.1-1.el8 rhel-8-for-x86_64-baseos-rpms 639 k 2022-05-13T23:02:25.696 INFO:teuthology.orchestra.run.smithi002.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2022-05-13T23:02:25.696 INFO:teuthology.orchestra.run.smithi002.stdout: libquadmath x86_64 8.4.1-1.el8 rhel-8-for-x86_64-baseos-rpms 169 k 2022-05-13T23:02:25.697 INFO:teuthology.orchestra.run.smithi002.stdout: librabbitmq x86_64 0.9.0-3.el8 rhel-8-for-x86_64-baseos-rpms 47 k 2022-05-13T23:02:25.697 INFO:teuthology.orchestra.run.smithi002.stdout: libradosstriper1 x86_64 2:17.2.0-188.g23b06583.el8 ceph 483 k 2022-05-13T23:02:25.697 INFO:teuthology.orchestra.run.smithi002.stdout: librdkafka x86_64 0.11.4-1.el8 rhel-8-for-x86_64-appstream-rpms 353 k 2022-05-13T23:02:25.697 INFO:teuthology.orchestra.run.smithi002.stdout: librgw2 x86_64 2:17.2.0-188.g23b06583.el8 ceph 5.3 M 2022-05-13T23:02:25.698 INFO:teuthology.orchestra.run.smithi002.stdout: libselinux-devel x86_64 2.9-5.el8 rhel-8-for-x86_64-baseos-rpms 200 k 2022-05-13T23:02:25.698 INFO:teuthology.orchestra.run.smithi002.stdout: libsepol-devel x86_64 2.9-2.el8 rhel-8-for-x86_64-baseos-rpms 86 k 2022-05-13T23:02:25.698 INFO:teuthology.orchestra.run.smithi002.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2022-05-13T23:02:25.698 INFO:teuthology.orchestra.run.smithi002.stdout: libverto-devel x86_64 0.3.0-5.el8 rhel-8-for-x86_64-baseos-rpms 18 k 2022-05-13T23:02:25.699 INFO:teuthology.orchestra.run.smithi002.stdout: lttng-ust x86_64 2.8.1-11.el8 rhel-8-for-x86_64-appstream-rpms 259 k 2022-05-13T23:02:25.699 INFO:teuthology.orchestra.run.smithi002.stdout: oniguruma x86_64 6.8.2-2.el8 rhel-8-for-x86_64-appstream-rpms 187 k 2022-05-13T23:02:25.699 INFO:teuthology.orchestra.run.smithi002.stdout: openblas x86_64 0.3.12-1.el8 rhel-8-for-x86_64-appstream-rpms 4.6 M 2022-05-13T23:02:25.700 INFO:teuthology.orchestra.run.smithi002.stdout: openblas-threads x86_64 0.3.12-1.el8 rhel-8-for-x86_64-appstream-rpms 4.7 M 2022-05-13T23:02:25.700 INFO:teuthology.orchestra.run.smithi002.stdout: pcre2-devel x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 605 k 2022-05-13T23:02:25.700 INFO:teuthology.orchestra.run.smithi002.stdout: pcre2-utf16 x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 229 k 2022-05-13T23:02:25.700 INFO:teuthology.orchestra.run.smithi002.stdout: pcre2-utf32 x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 220 k 2022-05-13T23:02:25.701 INFO:teuthology.orchestra.run.smithi002.stdout: python3-asyncssh noarch 2.7.0-2.el8 epel 476 k 2022-05-13T23:02:25.701 INFO:teuthology.orchestra.run.smithi002.stdout: python3-babel noarch 2.5.1-5.el8 rhel-8-for-x86_64-appstream-rpms 4.8 M 2022-05-13T23:02:25.701 INFO:teuthology.orchestra.run.smithi002.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2022-05-13T23:02:25.702 INFO:teuthology.orchestra.run.smithi002.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2022-05-13T23:02:25.702 INFO:teuthology.orchestra.run.smithi002.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2022-05-13T23:02:25.702 INFO:teuthology.orchestra.run.smithi002.stdout: python3-ceph-argparse x86_64 2:17.2.0-188.g23b06583.el8 ceph 45 k 2022-05-13T23:02:25.702 INFO:teuthology.orchestra.run.smithi002.stdout: python3-ceph-common x86_64 2:17.2.0-188.g23b06583.el8 ceph 106 k 2022-05-13T23:02:25.703 INFO:teuthology.orchestra.run.smithi002.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2022-05-13T23:02:25.703 INFO:teuthology.orchestra.run.smithi002.stdout: python3-cffi x86_64 1.11.5-5.el8 rhel-8-for-x86_64-baseos-rpms 238 k 2022-05-13T23:02:25.703 INFO:teuthology.orchestra.run.smithi002.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2022-05-13T23:02:25.703 INFO:teuthology.orchestra.run.smithi002.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2022-05-13T23:02:25.704 INFO:teuthology.orchestra.run.smithi002.stdout: python3-cryptography x86_64 3.2.1-4.el8 rhel-8-for-x86_64-baseos-rpms 559 k 2022-05-13T23:02:25.704 INFO:teuthology.orchestra.run.smithi002.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2022-05-13T23:02:25.704 INFO:teuthology.orchestra.run.smithi002.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2022-05-13T23:02:25.704 INFO:teuthology.orchestra.run.smithi002.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2022-05-13T23:02:25.704 INFO:teuthology.orchestra.run.smithi002.stdout: python3-jinja2 noarch 2.10.1-2.el8_0 rhel-8-for-x86_64-appstream-rpms 538 k 2022-05-13T23:02:25.705 INFO:teuthology.orchestra.run.smithi002.stdout: python3-jsonpatch noarch 1.21-2.el8 rhel-8-for-x86_64-appstream-rpms 27 k 2022-05-13T23:02:25.705 INFO:teuthology.orchestra.run.smithi002.stdout: python3-jsonpointer noarch 1.10-11.el8 rhel-8-for-x86_64-appstream-rpms 20 k 2022-05-13T23:02:25.705 INFO:teuthology.orchestra.run.smithi002.stdout: python3-jwt noarch 1.6.1-2.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2022-05-13T23:02:25.705 INFO:teuthology.orchestra.run.smithi002.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2022-05-13T23:02:25.705 INFO:teuthology.orchestra.run.smithi002.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2022-05-13T23:02:25.706 INFO:teuthology.orchestra.run.smithi002.stdout: python3-mako noarch 1.0.6-13.el8 rhel-8-for-x86_64-appstream-rpms 157 k 2022-05-13T23:02:25.706 INFO:teuthology.orchestra.run.smithi002.stdout: python3-markupsafe x86_64 0.23-19.el8 rhel-8-for-x86_64-appstream-rpms 39 k 2022-05-13T23:02:25.706 INFO:teuthology.orchestra.run.smithi002.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2022-05-13T23:02:25.706 INFO:teuthology.orchestra.run.smithi002.stdout: python3-natsort noarch 7.1.1-2.el8 epel 60 k 2022-05-13T23:02:25.707 INFO:teuthology.orchestra.run.smithi002.stdout: python3-numpy x86_64 1:1.14.3-9.el8 rhel-8-for-x86_64-appstream-rpms 3.7 M 2022-05-13T23:02:25.707 INFO:teuthology.orchestra.run.smithi002.stdout: python3-numpy-f2py x86_64 1:1.14.3-9.el8 rhel-8-for-x86_64-appstream-rpms 225 k 2022-05-13T23:02:25.707 INFO:teuthology.orchestra.run.smithi002.stdout: python3-oauthlib noarch 2.1.0-1.el8 rhel-8-for-x86_64-baseos-rpms 155 k 2022-05-13T23:02:25.707 INFO:teuthology.orchestra.run.smithi002.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2022-05-13T23:02:25.708 INFO:teuthology.orchestra.run.smithi002.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2022-05-13T23:02:25.708 INFO:teuthology.orchestra.run.smithi002.stdout: python3-prettytable noarch 0.7.2-14.el8 rhel-8-for-x86_64-appstream-rpms 44 k 2022-05-13T23:02:25.708 INFO:teuthology.orchestra.run.smithi002.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 rhel-8-for-x86_64-appstream-rpms 103 k 2022-05-13T23:02:25.708 INFO:teuthology.orchestra.run.smithi002.stdout: python3-pyasn1 noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 126 k 2022-05-13T23:02:25.709 INFO:teuthology.orchestra.run.smithi002.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 110 k 2022-05-13T23:02:25.709 INFO:teuthology.orchestra.run.smithi002.stdout: python3-pycparser noarch 2.14-14.el8 rhel-8-for-x86_64-baseos-rpms 109 k 2022-05-13T23:02:25.709 INFO:teuthology.orchestra.run.smithi002.stdout: python3-pytz noarch 2017.2-9.el8 rhel-8-for-x86_64-appstream-rpms 54 k 2022-05-13T23:02:25.709 INFO:teuthology.orchestra.run.smithi002.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2022-05-13T23:02:25.710 INFO:teuthology.orchestra.run.smithi002.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2022-05-13T23:02:25.710 INFO:teuthology.orchestra.run.smithi002.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2022-05-13T23:02:25.710 INFO:teuthology.orchestra.run.smithi002.stdout: python3-rsa noarch 4.8-1.el8 epel 61 k 2022-05-13T23:02:25.710 INFO:teuthology.orchestra.run.smithi002.stdout: python3-scipy x86_64 1.0.0-20.module+el8.1.0+3334+5cb623d7 rhel-8-for-x86_64-appstream-rpms 15 M 2022-05-13T23:02:25.711 INFO:teuthology.orchestra.run.smithi002.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2022-05-13T23:02:25.711 INFO:teuthology.orchestra.run.smithi002.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2022-05-13T23:02:25.711 INFO:teuthology.orchestra.run.smithi002.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2022-05-13T23:02:25.711 INFO:teuthology.orchestra.run.smithi002.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2022-05-13T23:02:25.712 INFO:teuthology.orchestra.run.smithi002.stdout: python3-waitress noarch 1.2.1-2.el8.1 epel 241 k 2022-05-13T23:02:25.712 INFO:teuthology.orchestra.run.smithi002.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2022-05-13T23:02:25.712 INFO:teuthology.orchestra.run.smithi002.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2022-05-13T23:02:25.712 INFO:teuthology.orchestra.run.smithi002.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2022-05-13T23:02:25.712 INFO:teuthology.orchestra.run.smithi002.stdout: python3-werkzeug noarch 0.12.2-4.el8 rhel-8-for-x86_64-appstream-rpms 457 k 2022-05-13T23:02:25.712 INFO:teuthology.orchestra.run.smithi002.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2022-05-13T23:02:25.713 INFO:teuthology.orchestra.run.smithi002.stdout: socat x86_64 1.7.3.3-2.el8 rhel-8-for-x86_64-appstream-rpms 302 k 2022-05-13T23:02:25.713 INFO:teuthology.orchestra.run.smithi002.stdout: thrift x86_64 0.13.0-2.el8 epel 1.7 M 2022-05-13T23:02:25.713 INFO:teuthology.orchestra.run.smithi002.stdout: xmlstarlet x86_64 1.6.1-11.el8 epel 69 k 2022-05-13T23:02:25.713 INFO:teuthology.orchestra.run.smithi002.stdout: zlib-devel x86_64 1.2.11-17.el8 rhel-8-for-x86_64-baseos-rpms 58 k 2022-05-13T23:02:25.714 INFO:teuthology.orchestra.run.smithi002.stdout: 2022-05-13T23:02:25.714 INFO:teuthology.orchestra.run.smithi002.stdout:Transaction Summary 2022-05-13T23:02:25.714 INFO:teuthology.orchestra.run.smithi002.stdout:====================================================================================================================== 2022-05-13T23:02:25.714 INFO:teuthology.orchestra.run.smithi002.stdout:Install 122 Packages 2022-05-13T23:02:25.714 INFO:teuthology.orchestra.run.smithi002.stdout:Upgrade 6 Packages 2022-05-13T23:02:25.715 INFO:teuthology.orchestra.run.smithi002.stdout: 2022-05-13T23:02:25.715 INFO:teuthology.orchestra.run.smithi002.stdout:Total download size: 207 M 2022-05-13T23:02:25.716 INFO:teuthology.orchestra.run.smithi002.stdout:Downloading Packages: 2022-05-13T23:02:25.729 INFO:teuthology.orchestra.run.smithi115.stdout:(28/128): ceph-mgr-cephadm-17.2.0-188.g23b06583 1.7 MB/s | 105 kB 00:00 2022-05-13T23:02:25.747 INFO:teuthology.orchestra.run.smithi115.stdout:(29/128): rbd-mirror-17.2.0-188.g23b06583.el8.x 9.6 MB/s | 3.8 MB 00:00 2022-05-13T23:02:25.947 INFO:teuthology.orchestra.run.smithi115.stdout:(30/128): ceph-mgr-dashboard-17.2.0-188.g23b065 7.5 MB/s | 1.6 MB 00:00 2022-05-13T23:02:25.990 INFO:teuthology.orchestra.run.smithi002.stdout:(1/128): ceph-17.2.0-188.g23b06583.el8.x86_64.r 23 kB/s | 6.4 kB 00:00 2022-05-13T23:02:26.016 INFO:teuthology.orchestra.run.smithi115.stdout:(31/128): ceph-mgr-modules-core-17.2.0-188.g23b 3.2 MB/s | 228 kB 00:00 2022-05-13T23:02:26.080 INFO:teuthology.orchestra.run.smithi115.stdout:(32/128): ceph-mgr-rook-17.2.0-188.g23b06583.el 741 kB/s | 47 kB 00:00 2022-05-13T23:02:26.145 INFO:teuthology.orchestra.run.smithi115.stdout:(33/128): ceph-prometheus-alerts-17.2.0-188.g23 226 kB/s | 15 kB 00:00 2022-05-13T23:02:26.224 INFO:teuthology.orchestra.run.smithi115.stdout:(34/128): ceph-volume-17.2.0-188.g23b06583.el8. 3.2 MB/s | 259 kB 00:00 2022-05-13T23:02:26.300 INFO:teuthology.orchestra.run.smithi115.stdout:(35/128): cephadm-17.2.0-188.g23b06583.el8.noar 988 kB/s | 75 kB 00:00 2022-05-13T23:02:26.377 INFO:teuthology.orchestra.run.smithi115.stdout:(36/128): cephfs-top-17.2.0-188.g23b06583.el8.n 183 kB/s | 14 kB 00:00 2022-05-13T23:02:26.548 INFO:teuthology.orchestra.run.smithi115.stdout:(37/128): gperftools-libs-2.7-9.el8.x86_64.rpm 1.8 MB/s | 306 kB 00:00 2022-05-13T23:02:26.549 INFO:teuthology.orchestra.run.smithi002.stdout:(2/128): ceph-fuse-17.2.0-188.g23b06583.el8.x86 1.5 MB/s | 876 kB 00:00 2022-05-13T23:02:26.577 INFO:teuthology.orchestra.run.smithi115.stdout:(38/128): liboath-2.6.2-3.el8.x86_64.rpm 2.0 MB/s | 59 kB 00:00 2022-05-13T23:02:26.604 INFO:teuthology.orchestra.run.smithi115.stdout:(39/128): ceph-mgr-diskprediction-local-17.2.0- 8.7 MB/s | 7.4 MB 00:00 2022-05-13T23:02:26.604 INFO:teuthology.orchestra.run.smithi002.stdout:(3/128): ceph-immutable-object-cache-17.2.0-188 3.0 MB/s | 169 kB 00:00 2022-05-13T23:02:26.606 INFO:teuthology.orchestra.run.smithi115.stdout:(40/128): libunwind-1.3.1-3.el8.x86_64.rpm 2.5 MB/s | 75 kB 00:00 2022-05-13T23:02:26.636 INFO:teuthology.orchestra.run.smithi115.stdout:(41/128): python3-bcrypt-3.1.6-2.el8.1.x86_64.r 1.4 MB/s | 44 kB 00:00 2022-05-13T23:02:26.667 INFO:teuthology.orchestra.run.smithi115.stdout:(42/128): python3-beautifulsoup4-4.6.3-2.el8.1. 6.0 MB/s | 185 kB 00:00 2022-05-13T23:02:26.695 INFO:teuthology.orchestra.run.smithi115.stdout:(43/128): python3-cachetools-3.1.1-4.el8.noarch 1.1 MB/s | 33 kB 00:00 2022-05-13T23:02:26.724 INFO:teuthology.orchestra.run.smithi115.stdout:(44/128): python3-certifi-2018.10.15-7.el8.noar 576 kB/s | 16 kB 00:00 2022-05-13T23:02:26.754 INFO:teuthology.orchestra.run.smithi115.stdout:(45/128): python3-cheroot-8.5.2-1.el8.noarch.rp 5.6 MB/s | 173 kB 00:00 2022-05-13T23:02:26.800 INFO:teuthology.orchestra.run.smithi115.stdout:(46/128): python3-asyncssh-2.7.0-2.el8.noarch.r 2.4 MB/s | 476 kB 00:00 2022-05-13T23:02:26.813 INFO:teuthology.orchestra.run.smithi115.stdout:(47/128): python3-cherrypy-18.4.0-1.el8.noarch. 6.4 MB/s | 384 kB 00:00 2022-05-13T23:02:26.829 INFO:teuthology.orchestra.run.smithi115.stdout:(48/128): python3-google-auth-1.1.1-10.el8.noar 2.8 MB/s | 81 kB 00:00 2022-05-13T23:02:26.841 INFO:teuthology.orchestra.run.smithi115.stdout:(49/128): python3-jaraco-6.2-6.el8.noarch.rpm 392 kB/s | 11 kB 00:00 2022-05-13T23:02:26.857 INFO:teuthology.orchestra.run.smithi115.stdout:(50/128): python3-jaraco-functools-2.0-4.el8.no 635 kB/s | 18 kB 00:00 2022-05-13T23:02:26.885 INFO:teuthology.orchestra.run.smithi115.stdout:(51/128): python3-logutils-0.3.5-11.el8.noarch. 1.7 MB/s | 49 kB 00:00 2022-05-13T23:02:26.914 INFO:teuthology.orchestra.run.smithi115.stdout:(52/128): python3-more-itertools-7.2.0-3.el8.no 2.0 MB/s | 59 kB 00:00 2022-05-13T23:02:26.942 INFO:teuthology.orchestra.run.smithi115.stdout:(53/128): python3-natsort-7.1.1-2.el8.noarch.rp 2.1 MB/s | 60 kB 00:00 2022-05-13T23:02:26.986 INFO:teuthology.orchestra.run.smithi115.stdout:(54/128): python3-kubernetes-11.0.0-6.el8.noarc 8.1 MB/s | 1.2 MB 00:00 2022-05-13T23:02:26.997 INFO:teuthology.orchestra.run.smithi115.stdout:(55/128): python3-pecan-1.3.2-9.el8.noarch.rpm 5.0 MB/s | 283 kB 00:00 2022-05-13T23:02:27.014 INFO:teuthology.orchestra.run.smithi115.stdout:(56/128): python3-portend-2.6-1.el8.noarch.rpm 546 kB/s | 16 kB 00:00 2022-05-13T23:02:27.025 INFO:teuthology.orchestra.run.smithi115.stdout:(57/128): python3-repoze-lru-0.7-6.el8.noarch.r 1.2 MB/s | 34 kB 00:00 2022-05-13T23:02:27.045 INFO:teuthology.orchestra.run.smithi115.stdout:(58/128): python3-routes-2.4.1-12.el8.noarch.rp 6.2 MB/s | 196 kB 00:00 2022-05-13T23:02:27.053 INFO:teuthology.orchestra.run.smithi115.stdout:(59/128): python3-rsa-4.8-1.el8.noarch.rpm 2.1 MB/s | 61 kB 00:00 2022-05-13T23:02:27.074 INFO:teuthology.orchestra.run.smithi115.stdout:(60/128): python3-simplegeneric-0.8.1-17.el8.no 654 kB/s | 19 kB 00:00 2022-05-13T23:02:27.081 INFO:teuthology.orchestra.run.smithi115.stdout:(61/128): python3-singledispatch-3.4.0.3-18.el8 865 kB/s | 24 kB 00:00 2022-05-13T23:02:27.103 INFO:teuthology.orchestra.run.smithi115.stdout:(62/128): python3-tempora-1.14.1-5.el8.noarch.r 1.0 MB/s | 29 kB 00:00 2022-05-13T23:02:27.111 INFO:teuthology.orchestra.run.smithi002.stdout:(4/128): ceph-mds-17.2.0-188.g23b06583.el8.x86_ 4.5 MB/s | 2.3 MB 00:00 2022-05-13T23:02:27.112 INFO:teuthology.orchestra.run.smithi115.stdout:(63/128): python3-trustme-0.6.0-4.el8.noarch.rp 898 kB/s | 27 kB 00:00 2022-05-13T23:02:27.134 INFO:teuthology.orchestra.run.smithi115.stdout:(64/128): python3-waitress-1.2.1-2.el8.1.noarch 7.6 MB/s | 241 kB 00:00 2022-05-13T23:02:27.145 INFO:teuthology.orchestra.run.smithi115.stdout:(65/128): python3-webob-1.8.5-1.el8.1.noarch.rp 7.5 MB/s | 251 kB 00:00 2022-05-13T23:02:27.170 INFO:teuthology.orchestra.run.smithi115.stdout:(66/128): python3-websocket-client-0.56.0-5.el8 1.7 MB/s | 61 kB 00:00 2022-05-13T23:02:27.176 INFO:teuthology.orchestra.run.smithi115.stdout:(67/128): python3-webtest-2.0.33-1.el8.noarch.r 2.7 MB/s | 85 kB 00:00 2022-05-13T23:02:27.199 INFO:teuthology.orchestra.run.smithi115.stdout:(68/128): python3-zc-lockfile-2.0-2.el8.noarch. 809 kB/s | 23 kB 00:00 2022-05-13T23:02:27.228 INFO:teuthology.orchestra.run.smithi115.stdout:(69/128): xmlstarlet-1.6.1-11.el8.x86_64.rpm 2.3 MB/s | 69 kB 00:00 2022-05-13T23:02:27.259 INFO:teuthology.orchestra.run.smithi115.stdout:(70/128): python3-jwt-1.6.1-2.el8.noarch.rpm 1.4 MB/s | 43 kB 00:00 2022-05-13T23:02:27.279 INFO:teuthology.orchestra.run.smithi115.stdout:(71/128): libaio-devel-0.3.112-1.el8.x86_64.rpm 913 kB/s | 19 kB 00:00 2022-05-13T23:02:27.299 INFO:teuthology.orchestra.run.smithi115.stdout:(72/128): python3-requests-oauthlib-1.0.0-1.el8 2.2 MB/s | 43 kB 00:00 2022-05-13T23:02:27.319 INFO:teuthology.orchestra.run.smithi115.stdout:(73/128): libquadmath-8.4.1-1.el8.x86_64.rpm 8.3 MB/s | 169 kB 00:00 2022-05-13T23:02:27.340 INFO:teuthology.orchestra.run.smithi115.stdout:(74/128): python3-cffi-1.11.5-5.el8.x86_64.rpm 11 MB/s | 238 kB 00:00 2022-05-13T23:02:27.363 INFO:teuthology.orchestra.run.smithi115.stdout:(75/128): libgfortran-8.4.1-1.el8.x86_64.rpm 28 MB/s | 639 kB 00:00 2022-05-13T23:02:27.371 INFO:teuthology.orchestra.run.smithi047.stdout:(4/128): ceph-mds-17.2.0-188.g23b06583.el8.x86_ 939 kB/s | 2.3 MB 00:02 2022-05-13T23:02:27.379 INFO:teuthology.orchestra.run.smithi115.stdout:(76/128): thrift-0.13.0-2.el8.x86_64.rpm 8.5 MB/s | 1.7 MB 00:00 2022-05-13T23:02:27.381 INFO:teuthology.orchestra.run.smithi115.stdout:(77/128): elfutils-libelf-devel-0.182-3.el8.x86 3.0 MB/s | 59 kB 00:00 2022-05-13T23:02:27.407 INFO:teuthology.orchestra.run.smithi002.stdout:(5/128): ceph-mgr-17.2.0-188.g23b06583.el8.x86_ 5.2 MB/s | 1.5 MB 00:00 2022-05-13T23:02:27.410 INFO:teuthology.orchestra.run.smithi115.stdout:(78/128): zlib-devel-1.2.11-17.el8.x86_64.rpm 2.0 MB/s | 58 kB 00:00 2022-05-13T23:02:27.418 INFO:teuthology.orchestra.run.smithi115.stdout:(79/128): libselinux-devel-2.9-5.el8.x86_64.rpm 5.1 MB/s | 200 kB 00:00 2022-05-13T23:02:27.431 INFO:teuthology.orchestra.run.smithi115.stdout:(80/128): pcre2-utf16-10.32-2.el8.x86_64.rpm 11 MB/s | 229 kB 00:00 2022-05-13T23:02:27.443 INFO:teuthology.orchestra.run.smithi115.stdout:(81/128): librabbitmq-0.9.0-3.el8.x86_64.rpm 1.9 MB/s | 47 kB 00:00 2022-05-13T23:02:27.455 INFO:teuthology.orchestra.run.smithi115.stdout:(82/128): libverto-devel-0.3.0-5.el8.x86_64.rpm 775 kB/s | 18 kB 00:00 2022-05-13T23:02:27.467 INFO:teuthology.orchestra.run.smithi115.stdout:(83/128): libsepol-devel-2.9-2.el8.x86_64.rpm 3.5 MB/s | 86 kB 00:00 2022-05-13T23:02:27.479 INFO:teuthology.orchestra.run.smithi115.stdout:(84/128): libuuid-devel-2.32.1-27.el8.x86_64.rp 3.9 MB/s | 97 kB 00:00 2022-05-13T23:02:27.496 INFO:teuthology.orchestra.run.smithi115.stdout:(85/128): python3-oauthlib-2.1.0-1.el8.noarch.r 5.3 MB/s | 155 kB 00:00 2022-05-13T23:02:27.521 INFO:teuthology.orchestra.run.smithi115.stdout:(86/128): openssl-devel-1.1.1g-15.el8_3.x86_64. 56 MB/s | 2.3 MB 00:00 2022-05-13T23:02:27.521 INFO:teuthology.orchestra.run.smithi115.stdout:(87/128): fuse-2.9.7-12.el8.x86_64.rpm 3.1 MB/s | 83 kB 00:00 2022-05-13T23:02:27.545 INFO:teuthology.orchestra.run.smithi115.stdout:(88/128): python3-pycparser-2.14-14.el8.noarch. 4.4 MB/s | 109 kB 00:00 2022-05-13T23:02:27.555 INFO:teuthology.orchestra.run.smithi115.stdout:(89/128): pcre2-devel-10.32-2.el8.x86_64.rpm 18 MB/s | 605 kB 00:00 2022-05-13T23:02:27.574 INFO:teuthology.orchestra.run.smithi115.stdout:(90/128): python3-cryptography-3.2.1-4.el8.x86_ 19 MB/s | 559 kB 00:00 2022-05-13T23:02:27.581 INFO:teuthology.orchestra.run.smithi115.stdout:(91/128): xfsprogs-devel-5.0.0-8.el8.x86_64.rpm 3.1 MB/s | 70 kB 00:00 2022-05-13T23:02:27.600 INFO:teuthology.orchestra.run.smithi115.stdout:(92/128): pcre2-utf32-10.32-2.el8.x86_64.rpm 8.5 MB/s | 220 kB 00:00 2022-05-13T23:02:27.607 INFO:teuthology.orchestra.run.smithi115.stdout:(93/128): keyutils-libs-devel-1.5.10-6.el8.x86_ 1.9 MB/s | 48 kB 00:00 2022-05-13T23:02:27.625 INFO:teuthology.orchestra.run.smithi115.stdout:(94/128): libcom_err-devel-1.45.6-1.el8.x86_64. 1.5 MB/s | 38 kB 00:00 2022-05-13T23:02:27.626 INFO:teuthology.orchestra.run.smithi047.stdout:(5/128): ceph-base-17.2.0-188.g23b06583.el8.x86 1.1 MB/s | 5.3 MB 00:04 2022-05-13T23:02:27.635 INFO:teuthology.orchestra.run.smithi115.stdout:(95/128): krb5-devel-1.18.2-8.3.el8_4.x86_64.rp 20 MB/s | 559 kB 00:00 2022-05-13T23:02:27.648 INFO:teuthology.orchestra.run.smithi115.stdout:(96/128): python3-mako-1.0.6-13.el8.noarch.rpm 6.9 MB/s | 157 kB 00:00 2022-05-13T23:02:27.661 INFO:teuthology.orchestra.run.smithi115.stdout:(97/128): flex-2.6.1-9.el8.x86_64.rpm 14 MB/s | 320 kB 00:00 2022-05-13T23:02:27.674 INFO:teuthology.orchestra.run.smithi115.stdout:(98/128): jq-1.5-12.el8.x86_64.rpm 7.2 MB/s | 161 kB 00:00 2022-05-13T23:02:27.685 INFO:teuthology.orchestra.run.smithi115.stdout:(99/128): python3-jsonpatch-1.21-2.el8.noarch.r 1.3 MB/s | 27 kB 00:00 2022-05-13T23:02:27.704 INFO:teuthology.orchestra.run.smithi115.stdout:(100/128): librdkafka-0.11.4-1.el8.x86_64.rpm 14 MB/s | 353 kB 00:00 2022-05-13T23:02:27.713 INFO:teuthology.orchestra.run.smithi115.stdout:(101/128): python3-jinja2-2.10.1-2.el8_0.noarch 19 MB/s | 538 kB 00:00 2022-05-13T23:02:27.735 INFO:teuthology.orchestra.run.smithi115.stdout:(102/128): automake-1.16.1-7.el8.noarch.rpm 22 MB/s | 713 kB 00:00 2022-05-13T23:02:27.750 INFO:teuthology.orchestra.run.smithi115.stdout:(103/128): libtool-2.4.6-25.el8.x86_64.rpm 21 MB/s | 709 kB 00:00 2022-05-13T23:02:27.771 INFO:teuthology.orchestra.run.smithi115.stdout:(104/128): python3-pyasn1-modules-0.3.7-6.el8.n 5.1 MB/s | 110 kB 00:00 2022-05-13T23:02:27.794 INFO:teuthology.orchestra.run.smithi115.stdout:(105/128): python3-numpy-1.14.3-9.el8.x86_64.rp 64 MB/s | 3.7 MB 00:00 2022-05-13T23:02:27.795 INFO:teuthology.orchestra.run.smithi115.stdout:(106/128): python3-jsonpointer-1.10-11.el8.noar 880 kB/s | 20 kB 00:00 2022-05-13T23:02:27.816 INFO:teuthology.orchestra.run.smithi115.stdout:(107/128): python3-pytz-2017.2-9.el8.noarch.rpm 2.4 MB/s | 54 kB 00:00 2022-05-13T23:02:27.824 INFO:teuthology.orchestra.run.smithi115.stdout:(108/128): oniguruma-6.8.2-2.el8.x86_64.rpm 6.2 MB/s | 187 kB 00:00 2022-05-13T23:02:27.841 INFO:teuthology.orchestra.run.smithi115.stdout:(109/128): python3-pyasn1-0.3.7-6.el8.noarch.rp 5.1 MB/s | 126 kB 00:00 2022-05-13T23:02:27.866 INFO:teuthology.orchestra.run.smithi115.stdout:(110/128): python3-pyOpenSSL-19.0.0-1.el8.noarc 4.8 MB/s | 103 kB 00:00 2022-05-13T23:02:27.894 INFO:teuthology.orchestra.run.smithi115.stdout:(111/128): lttng-ust-2.8.1-11.el8.x86_64.rpm 12 MB/s | 259 kB 00:00 2022-05-13T23:02:27.941 INFO:teuthology.orchestra.run.smithi115.stdout:(112/128): openblas-threads-0.3.12-1.el8.x86_64 85 MB/s | 4.7 MB 00:00 2022-05-13T23:02:28.005 INFO:teuthology.orchestra.run.smithi002.stdout:(6/128): ceph-mon-17.2.0-188.g23b06583.el8.x86_ 6.5 MB/s | 3.9 MB 00:00 2022-05-13T23:02:28.056 INFO:teuthology.orchestra.run.smithi115.stdout:(113/128): python3-scipy-1.0.0-20.module+el8.1. 63 MB/s | 15 MB 00:00 2022-05-13T23:02:28.080 INFO:teuthology.orchestra.run.smithi115.stdout:(114/128): python3-werkzeug-0.12.2-4.el8.noarch 20 MB/s | 457 kB 00:00 2022-05-13T23:02:28.108 INFO:teuthology.orchestra.run.smithi115.stdout:(115/128): python3-babel-2.5.1-5.el8.noarch.rpm 29 MB/s | 4.8 MB 00:00 2022-05-13T23:02:28.109 INFO:teuthology.orchestra.run.smithi115.stdout:(116/128): python3-markupsafe-0.23-19.el8.x86_6 1.3 MB/s | 39 kB 00:00 2022-05-13T23:02:28.131 INFO:teuthology.orchestra.run.smithi115.stdout:(117/128): python3-prettytable-0.7.2-14.el8.noa 1.9 MB/s | 44 kB 00:00 2022-05-13T23:02:28.140 INFO:teuthology.orchestra.run.smithi115.stdout:(118/128): socat-1.7.3.3-2.el8.x86_64.rpm 9.6 MB/s | 302 kB 00:00 2022-05-13T23:02:28.172 INFO:teuthology.orchestra.run.smithi115.stdout:(119/128): bison-3.0.4-10.el8.x86_64.rpm 22 MB/s | 688 kB 00:00 2022-05-13T23:02:28.189 INFO:teuthology.orchestra.run.smithi002.stdout:(7/128): ceph-base-17.2.0-188.g23b06583.el8.x86 2.1 MB/s | 5.3 MB 00:02 2022-05-13T23:02:28.196 INFO:teuthology.orchestra.run.smithi115.stdout:(120/128): openblas-0.3.12-1.el8.x86_64.rpm 71 MB/s | 4.6 MB 00:00 2022-05-13T23:02:28.199 INFO:teuthology.orchestra.run.smithi115.stdout:(121/128): python3-numpy-f2py-1.14.3-9.el8.x86_ 8.3 MB/s | 225 kB 00:00 2022-05-13T23:02:28.682 INFO:teuthology.orchestra.run.smithi047.stdout:(6/128): ceph-mgr-17.2.0-188.g23b06583.el8.x86_ 1.2 MB/s | 1.5 MB 00:01 2022-05-13T23:02:28.999 INFO:teuthology.orchestra.run.smithi115.stdout:(122/128): ceph-test-17.2.0-188.g23b06583.el8.x 11 MB/s | 52 MB 00:04 2022-05-13T23:02:29.020 INFO:teuthology.orchestra.run.smithi115.stdout:(123/128): NetworkManager-team-1.30.0-13.el8_4. 6.6 MB/s | 146 kB 00:00 2022-05-13T23:02:29.052 INFO:teuthology.orchestra.run.smithi115.stdout:(124/128): NetworkManager-libnm-1.30.0-13.el8_4 57 MB/s | 1.8 MB 00:00 2022-05-13T23:02:29.075 INFO:teuthology.orchestra.run.smithi115.stdout:(125/128): NetworkManager-tui-1.30.0-13.el8_4.x 14 MB/s | 329 kB 00:00 2022-05-13T23:02:29.112 INFO:teuthology.orchestra.run.smithi115.stdout:(126/128): NetworkManager-1.30.0-13.el8_4.x86_6 71 MB/s | 2.6 MB 00:00 2022-05-13T23:02:29.568 INFO:teuthology.orchestra.run.smithi002.stdout:(8/128): ceph-osd-17.2.0-188.g23b06583.el8.x86_ 11 MB/s | 17 MB 00:01 2022-05-13T23:02:29.594 INFO:teuthology.orchestra.run.smithi002.stdout:(9/128): ceph-selinux-17.2.0-188.g23b06583.el8. 955 kB/s | 24 kB 00:00 2022-05-13T23:02:29.597 INFO:teuthology.orchestra.run.smithi047.stdout:(7/128): ceph-mon-17.2.0-188.g23b06583.el8.x86_ 2.0 MB/s | 3.9 MB 00:01 2022-05-13T23:02:30.312 INFO:teuthology.orchestra.run.smithi115.stdout:(127/128): librbd1-17.2.0-188.g23b06583.el8.x86 1.8 MB/s | 3.9 MB 00:02 2022-05-13T23:02:30.345 INFO:teuthology.orchestra.run.smithi115.stdout:(128/128): librados2-17.2.0-188.g23b06583.el8.x 1.6 MB/s | 3.4 MB 00:02 2022-05-13T23:02:30.349 INFO:teuthology.orchestra.run.smithi115.stdout:-------------------------------------------------------------------------------- 2022-05-13T23:02:30.350 INFO:teuthology.orchestra.run.smithi115.stdout:Total 24 MB/s | 207 MB 00:08 2022-05-13T23:02:30.409 INFO:teuthology.orchestra.run.smithi115.stdout:Running transaction check 2022-05-13T23:02:30.534 INFO:teuthology.orchestra.run.smithi115.stdout:Transaction check succeeded. 2022-05-13T23:02:30.535 INFO:teuthology.orchestra.run.smithi115.stdout:Running transaction test 2022-05-13T23:02:31.625 INFO:teuthology.orchestra.run.smithi115.stdout:Transaction test succeeded. 2022-05-13T23:02:31.627 INFO:teuthology.orchestra.run.smithi115.stdout:Running transaction 2022-05-13T23:02:33.282 INFO:teuthology.orchestra.run.smithi002.stdout:(10/128): ceph-common-17.2.0-188.g23b06583.el8. 3.0 MB/s | 23 MB 00:07 2022-05-13T23:02:33.394 INFO:teuthology.orchestra.run.smithi002.stdout:(11/128): cephfs-mirror-17.2.0-188.g23b06583.el 2.4 MB/s | 272 kB 00:00 2022-05-13T23:02:33.442 INFO:teuthology.orchestra.run.smithi002.stdout:(12/128): libcephfs-devel-17.2.0-188.g23b06583. 514 kB/s | 24 kB 00:00 2022-05-13T23:02:33.775 INFO:teuthology.orchestra.run.smithi002.stdout:(13/128): ceph-test-17.2.0-188.g23b06583.el8.x8 12 MB/s | 52 MB 00:04 2022-05-13T23:02:33.784 INFO:teuthology.orchestra.run.smithi002.stdout:(14/128): libcephfs2-17.2.0-188.g23b06583.el8.x 2.3 MB/s | 790 kB 00:00 2022-05-13T23:02:33.846 INFO:teuthology.orchestra.run.smithi002.stdout:(15/128): ceph-radosgw-17.2.0-188.g23b06583.el8 2.8 MB/s | 16 MB 00:05 2022-05-13T23:02:33.850 INFO:teuthology.orchestra.run.smithi002.stdout:(16/128): librados-devel-17.2.0-188.g23b06583.e 1.9 MB/s | 125 kB 00:00 2022-05-13T23:02:33.851 INFO:teuthology.orchestra.run.smithi002.stdout:(17/128): libcephsqlite-17.2.0-188.g23b06583.el 2.2 MB/s | 170 kB 00:00 2022-05-13T23:02:33.902 INFO:teuthology.orchestra.run.smithi002.stdout:(18/128): python3-ceph-argparse-17.2.0-188.g23b 882 kB/s | 45 kB 00:00 2022-05-13T23:02:33.950 INFO:teuthology.orchestra.run.smithi002.stdout:(19/128): python3-ceph-common-17.2.0-188.g23b06 2.1 MB/s | 106 kB 00:00 2022-05-13T23:02:34.000 INFO:teuthology.orchestra.run.smithi002.stdout:(20/128): python3-cephfs-17.2.0-188.g23b06583.e 3.5 MB/s | 178 kB 00:00 2022-05-13T23:02:34.026 INFO:teuthology.orchestra.run.smithi002.stdout:(21/128): libradosstriper1-17.2.0-188.g23b06583 2.6 MB/s | 483 kB 00:00 2022-05-13T23:02:34.058 INFO:teuthology.orchestra.run.smithi002.stdout:(22/128): python3-rados-17.2.0-188.g23b06583.el 6.9 MB/s | 367 kB 00:00 2022-05-13T23:02:34.098 INFO:teuthology.orchestra.run.smithi002.stdout:(23/128): python3-rgw-17.2.0-188.g23b06583.el8. 2.3 MB/s | 109 kB 00:00 2022-05-13T23:02:34.145 INFO:teuthology.orchestra.run.smithi002.stdout:(24/128): rbd-fuse-17.2.0-188.g23b06583.el8.x86 1.9 MB/s | 93 kB 00:00 2022-05-13T23:02:34.157 INFO:teuthology.orchestra.run.smithi002.stdout:(25/128): python3-rbd-17.2.0-188.g23b06583.el8. 2.6 MB/s | 346 kB 00:00 2022-05-13T23:02:34.178 INFO:teuthology.orchestra.run.smithi047.stdout:(8/128): ceph-common-17.2.0-188.g23b06583.el8.x 2.1 MB/s | 23 MB 00:11 2022-05-13T23:02:34.230 INFO:teuthology.orchestra.run.smithi047.stdout:(9/128): ceph-selinux-17.2.0-188.g23b06583.el8. 480 kB/s | 24 kB 00:00 2022-05-13T23:02:34.265 INFO:teuthology.orchestra.run.smithi002.stdout:(26/128): rbd-nbd-17.2.0-188.g23b06583.el8.x86_ 1.6 MB/s | 175 kB 00:00 2022-05-13T23:02:34.321 INFO:teuthology.orchestra.run.smithi002.stdout:(27/128): ceph-grafana-dashboards-17.2.0-188.g2 402 kB/s | 23 kB 00:00 2022-05-13T23:02:34.382 INFO:teuthology.orchestra.run.smithi002.stdout:(28/128): ceph-mgr-cephadm-17.2.0-188.g23b06583 1.7 MB/s | 105 kB 00:00 2022-05-13T23:02:34.558 INFO:teuthology.orchestra.run.smithi002.stdout:(29/128): rbd-mirror-17.2.0-188.g23b06583.el8.x 9.2 MB/s | 3.8 MB 00:00 2022-05-13T23:02:34.682 INFO:teuthology.orchestra.run.smithi047.stdout:(10/128): ceph-osd-17.2.0-188.g23b06583.el8.x86 2.8 MB/s | 17 MB 00:05 2022-05-13T23:02:34.794 INFO:teuthology.orchestra.run.smithi047.stdout:(11/128): ceph-radosgw-17.2.0-188.g23b06583.el8 3.1 MB/s | 16 MB 00:05 2022-05-13T23:02:34.797 INFO:teuthology.orchestra.run.smithi047.stdout:(12/128): cephfs-mirror-17.2.0-188.g23b06583.el 2.3 MB/s | 272 kB 00:00 2022-05-13T23:02:34.845 INFO:teuthology.orchestra.run.smithi047.stdout:(13/128): libcephfs-devel-17.2.0-188.g23b06583. 479 kB/s | 24 kB 00:00 2022-05-13T23:02:35.003 INFO:teuthology.orchestra.run.smithi047.stdout:(14/128): libcephfs2-17.2.0-188.g23b06583.el8.x 3.7 MB/s | 790 kB 00:00 2022-05-13T23:02:35.010 INFO:teuthology.orchestra.run.smithi002.stdout:(30/128): ceph-mgr-dashboard-17.2.0-188.g23b065 2.6 MB/s | 1.6 MB 00:00 2022-05-13T23:02:35.041 INFO:teuthology.orchestra.run.smithi047.stdout:(15/128): libcephsqlite-17.2.0-188.g23b06583.el 867 kB/s | 170 kB 00:00 2022-05-13T23:02:35.045 INFO:teuthology.orchestra.run.smithi047.stdout:(16/128): librados-devel-17.2.0-188.g23b06583.e 2.9 MB/s | 125 kB 00:00 2022-05-13T23:02:35.099 INFO:teuthology.orchestra.run.smithi002.stdout:(31/128): ceph-mgr-modules-core-17.2.0-188.g23b 2.5 MB/s | 228 kB 00:00 2022-05-13T23:02:35.162 INFO:teuthology.orchestra.run.smithi002.stdout:(32/128): librgw2-17.2.0-188.g23b06583.el8.x86_ 4.1 MB/s | 5.3 MB 00:01 2022-05-13T23:02:35.162 INFO:teuthology.orchestra.run.smithi002.stdout:(33/128): ceph-mgr-rook-17.2.0-188.g23b06583.el 733 kB/s | 47 kB 00:00 2022-05-13T23:02:35.208 INFO:teuthology.orchestra.run.smithi002.stdout:(34/128): ceph-prometheus-alerts-17.2.0-188.g23 317 kB/s | 15 kB 00:00 2022-05-13T23:02:35.215 INFO:teuthology.orchestra.run.smithi047.stdout:(17/128): libradosstriper1-17.2.0-188.g23b06583 2.7 MB/s | 483 kB 00:00 2022-05-13T23:02:35.252 INFO:teuthology.orchestra.run.smithi002.stdout:(35/128): cephadm-17.2.0-188.g23b06583.el8.noar 1.7 MB/s | 75 kB 00:00 2022-05-13T23:02:35.260 INFO:teuthology.orchestra.run.smithi047.stdout:(18/128): python3-ceph-argparse-17.2.0-188.g23b 1.0 MB/s | 45 kB 00:00 2022-05-13T23:02:35.302 INFO:teuthology.orchestra.run.smithi047.stdout:(19/128): python3-ceph-common-17.2.0-188.g23b06 2.4 MB/s | 106 kB 00:00 2022-05-13T23:02:35.320 INFO:teuthology.orchestra.run.smithi002.stdout:(36/128): ceph-mgr-diskprediction-local-17.2.0- 9.7 MB/s | 7.4 MB 00:00 2022-05-13T23:02:35.321 INFO:teuthology.orchestra.run.smithi002.stdout:(37/128): cephfs-top-17.2.0-188.g23b06583.el8.n 206 kB/s | 14 kB 00:00 2022-05-13T23:02:35.322 INFO:teuthology.orchestra.run.smithi002.stdout:(38/128): ceph-volume-17.2.0-188.g23b06583.el8. 1.6 MB/s | 259 kB 00:00 2022-05-13T23:02:35.378 INFO:teuthology.orchestra.run.smithi047.stdout:(20/128): python3-cephfs-17.2.0-188.g23b06583.e 2.3 MB/s | 178 kB 00:00 2022-05-13T23:02:35.488 INFO:teuthology.orchestra.run.smithi047.stdout:(21/128): python3-rados-17.2.0-188.g23b06583.el 3.3 MB/s | 367 kB 00:00 2022-05-13T23:02:35.489 INFO:teuthology.orchestra.run.smithi002.stdout:(39/128): libunwind-1.3.1-3.el8.x86_64.rpm 454 kB/s | 75 kB 00:00 2022-05-13T23:02:35.491 INFO:teuthology.orchestra.run.smithi002.stdout:(40/128): liboath-2.6.2-3.el8.x86_64.rpm 345 kB/s | 59 kB 00:00 2022-05-13T23:02:35.538 INFO:teuthology.orchestra.run.smithi002.stdout:(41/128): python3-bcrypt-3.1.6-2.el8.1.x86_64.r 947 kB/s | 44 kB 00:00 2022-05-13T23:02:35.578 INFO:teuthology.orchestra.run.smithi002.stdout:(42/128): gperftools-libs-2.7-9.el8.x86_64.rpm 1.2 MB/s | 306 kB 00:00 2022-05-13T23:02:35.601 INFO:teuthology.orchestra.run.smithi047.stdout:(22/128): python3-rbd-17.2.0-188.g23b06583.el8. 3.0 MB/s | 346 kB 00:00 2022-05-13T23:02:35.618 INFO:teuthology.orchestra.run.smithi002.stdout:(43/128): python3-cachetools-3.1.1-4.el8.noarch 801 kB/s | 33 kB 00:00 2022-05-13T23:02:35.633 INFO:teuthology.orchestra.run.smithi002.stdout:(44/128): python3-beautifulsoup4-4.6.3-2.el8.1. 1.9 MB/s | 185 kB 00:00 2022-05-13T23:02:35.642 INFO:teuthology.orchestra.run.smithi047.stdout:(23/128): python3-rgw-17.2.0-188.g23b06583.el8. 2.6 MB/s | 109 kB 00:00 2022-05-13T23:02:35.661 INFO:teuthology.orchestra.run.smithi002.stdout:(45/128): python3-certifi-2018.10.15-7.el8.noar 377 kB/s | 16 kB 00:00 2022-05-13T23:02:35.683 INFO:teuthology.orchestra.run.smithi047.stdout:(24/128): rbd-fuse-17.2.0-188.g23b06583.el8.x86 2.3 MB/s | 93 kB 00:00 2022-05-13T23:02:35.704 INFO:teuthology.orchestra.run.smithi002.stdout:(46/128): python3-asyncssh-2.7.0-2.el8.noarch.r 2.2 MB/s | 476 kB 00:00 2022-05-13T23:02:35.722 INFO:teuthology.orchestra.run.smithi002.stdout:(47/128): python3-cheroot-8.5.2-1.el8.noarch.rp 1.9 MB/s | 173 kB 00:00 2022-05-13T23:02:35.750 INFO:teuthology.orchestra.run.smithi002.stdout:(48/128): python3-google-auth-1.1.1-10.el8.noar 1.8 MB/s | 81 kB 00:00 2022-05-13T23:02:35.769 INFO:teuthology.orchestra.run.smithi002.stdout:(49/128): python3-jaraco-6.2-6.el8.noarch.rpm 241 kB/s | 11 kB 00:00 2022-05-13T23:02:35.794 INFO:teuthology.orchestra.run.smithi002.stdout:(50/128): python3-jaraco-functools-2.0-4.el8.no 399 kB/s | 18 kB 00:00 2022-05-13T23:02:35.836 INFO:teuthology.orchestra.run.smithi002.stdout:(51/128): python3-cherrypy-18.4.0-1.el8.noarch. 2.1 MB/s | 384 kB 00:00 2022-05-13T23:02:35.840 INFO:teuthology.orchestra.run.smithi002.stdout:(52/128): python3-logutils-0.3.5-11.el8.noarch. 1.0 MB/s | 49 kB 00:00 2022-05-13T23:02:35.883 INFO:teuthology.orchestra.run.smithi002.stdout:(53/128): python3-more-itertools-7.2.0-3.el8.no 1.2 MB/s | 59 kB 00:00 2022-05-13T23:02:35.888 INFO:teuthology.orchestra.run.smithi002.stdout:(54/128): python3-natsort-7.1.1-2.el8.noarch.rp 1.2 MB/s | 60 kB 00:00 2022-05-13T23:02:35.936 INFO:teuthology.orchestra.run.smithi002.stdout:(55/128): python3-portend-2.6-1.el8.noarch.rpm 328 kB/s | 16 kB 00:00 2022-05-13T23:02:35.979 INFO:teuthology.orchestra.run.smithi047.stdout:(25/128): librgw2-17.2.0-188.g23b06583.el8.x86_ 5.7 MB/s | 5.3 MB 00:00 2022-05-13T23:02:35.983 INFO:teuthology.orchestra.run.smithi002.stdout:(56/128): python3-repoze-lru-0.7-6.el8.noarch.r 718 kB/s | 34 kB 00:00 2022-05-13T23:02:36.018 INFO:teuthology.orchestra.run.smithi002.stdout:(57/128): python3-pecan-1.3.2-9.el8.noarch.rpm 2.1 MB/s | 283 kB 00:00 2022-05-13T23:02:36.023 INFO:teuthology.orchestra.run.smithi047.stdout:(26/128): rbd-nbd-17.2.0-188.g23b06583.el8.x86_ 3.9 MB/s | 175 kB 00:00 2022-05-13T23:02:36.069 INFO:teuthology.orchestra.run.smithi002.stdout:(58/128): python3-rsa-4.8-1.el8.noarch.rpm 1.3 MB/s | 61 kB 00:00 2022-05-13T23:02:36.070 INFO:teuthology.orchestra.run.smithi047.stdout:(27/128): ceph-grafana-dashboards-17.2.0-188.g2 546 kB/s | 23 kB 00:00 2022-05-13T23:02:36.073 INFO:teuthology.orchestra.run.smithi002.stdout:(59/128): python3-routes-2.4.1-12.el8.noarch.rp 2.1 MB/s | 196 kB 00:00 2022-05-13T23:02:36.107 INFO:teuthology.orchestra.run.smithi047.stdout:(28/128): ceph-mgr-cephadm-17.2.0-188.g23b06583 2.5 MB/s | 105 kB 00:00 2022-05-13T23:02:36.110 INFO:teuthology.orchestra.run.smithi002.stdout:(60/128): python3-simplegeneric-0.8.1-17.el8.no 410 kB/s | 19 kB 00:00 2022-05-13T23:02:36.119 INFO:teuthology.orchestra.run.smithi002.stdout:(61/128): python3-singledispatch-3.4.0.3-18.el8 520 kB/s | 24 kB 00:00 2022-05-13T23:02:36.158 INFO:teuthology.orchestra.run.smithi002.stdout:(62/128): python3-tempora-1.14.1-5.el8.noarch.r 618 kB/s | 29 kB 00:00 2022-05-13T23:02:36.166 INFO:teuthology.orchestra.run.smithi002.stdout:(63/128): python3-trustme-0.6.0-4.el8.noarch.rp 598 kB/s | 27 kB 00:00 2022-05-13T23:02:36.184 INFO:teuthology.orchestra.run.smithi002.stdout:(64/128): python3-kubernetes-11.0.0-6.el8.noarc 2.8 MB/s | 1.2 MB 00:00 2022-05-13T23:02:36.236 INFO:teuthology.orchestra.run.smithi002.stdout:(65/128): python3-websocket-client-0.56.0-5.el8 1.2 MB/s | 61 kB 00:00 2022-05-13T23:02:36.253 INFO:teuthology.orchestra.run.smithi002.stdout:(66/128): python3-waitress-1.2.1-2.el8.1.noarch 2.5 MB/s | 241 kB 00:00 2022-05-13T23:02:36.260 INFO:teuthology.orchestra.run.smithi002.stdout:(67/128): python3-webob-1.8.5-1.el8.1.noarch.rp 2.6 MB/s | 251 kB 00:00 2022-05-13T23:02:36.287 INFO:teuthology.orchestra.run.smithi002.stdout:(68/128): python3-webtest-2.0.33-1.el8.noarch.r 1.6 MB/s | 85 kB 00:00 2022-05-13T23:02:36.300 INFO:teuthology.orchestra.run.smithi002.stdout:(69/128): python3-zc-lockfile-2.0-2.el8.noarch. 489 kB/s | 23 kB 00:00 2022-05-13T23:02:36.336 INFO:teuthology.orchestra.run.smithi002.stdout:(70/128): xmlstarlet-1.6.1-11.el8.x86_64.rpm 1.4 MB/s | 69 kB 00:00 2022-05-13T23:02:36.349 INFO:teuthology.orchestra.run.smithi002.stdout:(71/128): python3-mako-1.0.6-13.el8.noarch.rpm 3.1 MB/s | 157 kB 00:00 2022-05-13T23:02:36.386 INFO:teuthology.orchestra.run.smithi002.stdout:(72/128): flex-2.6.1-9.el8.x86_64.rpm 6.3 MB/s | 320 kB 00:00 2022-05-13T23:02:36.398 INFO:teuthology.orchestra.run.smithi002.stdout:(73/128): jq-1.5-12.el8.x86_64.rpm 3.8 MB/s | 161 kB 00:00 2022-05-13T23:02:36.401 INFO:teuthology.orchestra.run.smithi047.stdout:(29/128): ceph-mgr-dashboard-17.2.0-188.g23b065 5.5 MB/s | 1.6 MB 00:00 2022-05-13T23:02:36.417 INFO:teuthology.orchestra.run.smithi002.stdout:(74/128): python3-jsonpatch-1.21-2.el8.noarch.r 898 kB/s | 27 kB 00:00 2022-05-13T23:02:36.443 INFO:teuthology.orchestra.run.smithi002.stdout:(75/128): librdkafka-0.11.4-1.el8.x86_64.rpm 9.1 MB/s | 353 kB 00:00 2022-05-13T23:02:36.466 INFO:teuthology.orchestra.run.smithi002.stdout:(76/128): python3-jinja2-2.10.1-2.el8_0.noarch. 10 MB/s | 538 kB 00:00 2022-05-13T23:02:36.491 INFO:teuthology.orchestra.run.smithi002.stdout:(77/128): automake-1.16.1-7.el8.noarch.rpm 15 MB/s | 713 kB 00:00 2022-05-13T23:02:36.515 INFO:teuthology.orchestra.run.smithi002.stdout:(78/128): libtool-2.4.6-25.el8.x86_64.rpm 14 MB/s | 709 kB 00:00 2022-05-13T23:02:36.544 INFO:teuthology.orchestra.run.smithi002.stdout:(79/128): python3-pyasn1-modules-0.3.7-6.el8.no 4.0 MB/s | 110 kB 00:00 2022-05-13T23:02:36.570 INFO:teuthology.orchestra.run.smithi002.stdout:(80/128): python3-jsonpointer-1.10-11.el8.noarc 782 kB/s | 20 kB 00:00 2022-05-13T23:02:36.589 INFO:teuthology.orchestra.run.smithi047.stdout:(30/128): rbd-mirror-17.2.0-188.g23b06583.el8.x 4.2 MB/s | 3.8 MB 00:00 2022-05-13T23:02:36.609 INFO:teuthology.orchestra.run.smithi002.stdout:(81/128): python3-numpy-1.14.3-9.el8.x86_64.rpm 32 MB/s | 3.7 MB 00:00 2022-05-13T23:02:36.609 INFO:teuthology.orchestra.run.smithi002.stdout:(82/128): python3-pytz-2017.2-9.el8.noarch.rpm 1.3 MB/s | 54 kB 00:00 2022-05-13T23:02:36.618 INFO:teuthology.orchestra.run.smithi115.stdout: Preparing : 1/1 2022-05-13T23:02:36.643 INFO:teuthology.orchestra.run.smithi002.stdout:(83/128): oniguruma-6.8.2-2.el8.x86_64.rpm 5.4 MB/s | 187 kB 00:00 2022-05-13T23:02:36.646 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: liboath-2.6.2-3.el8.x86_64 1/1 2022-05-13T23:02:36.654 INFO:teuthology.orchestra.run.smithi002.stdout:(84/128): python3-pyasn1-0.3.7-6.el8.noarch.rpm 2.8 MB/s | 126 kB 00:00 2022-05-13T23:02:36.675 INFO:teuthology.orchestra.run.smithi047.stdout:(31/128): ceph-mgr-modules-core-17.2.0-188.g23b 2.6 MB/s | 228 kB 00:00 2022-05-13T23:02:36.688 INFO:teuthology.orchestra.run.smithi002.stdout:(85/128): python3-pyOpenSSL-19.0.0-1.el8.noarch 3.7 MB/s | 103 kB 00:00 2022-05-13T23:02:36.716 INFO:teuthology.orchestra.run.smithi047.stdout:(32/128): ceph-mgr-rook-17.2.0-188.g23b06583.el 1.1 MB/s | 47 kB 00:00 2022-05-13T23:02:36.720 INFO:teuthology.orchestra.run.smithi002.stdout:(86/128): lttng-ust-2.8.1-11.el8.x86_64.rpm 8.6 MB/s | 259 kB 00:00 2022-05-13T23:02:36.757 INFO:teuthology.orchestra.run.smithi047.stdout:(33/128): ceph-prometheus-alerts-17.2.0-188.g23 354 kB/s | 15 kB 00:00 2022-05-13T23:02:36.838 INFO:teuthology.orchestra.run.smithi002.stdout:(87/128): openblas-threads-0.3.12-1.el8.x86_64. 40 MB/s | 4.7 MB 00:00 2022-05-13T23:02:36.840 INFO:teuthology.orchestra.run.smithi047.stdout:(34/128): ceph-volume-17.2.0-188.g23b06583.el8. 3.0 MB/s | 259 kB 00:00 2022-05-13T23:02:36.845 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : liboath-2.6.2-3.el8.x86_64 1/134 2022-05-13T23:02:36.883 INFO:teuthology.orchestra.run.smithi047.stdout:(35/128): cephadm-17.2.0-188.g23b06583.el8.noar 1.7 MB/s | 75 kB 00:00 2022-05-13T23:02:36.902 INFO:teuthology.orchestra.run.smithi002.stdout:(88/128): thrift-0.13.0-2.el8.x86_64.rpm 2.7 MB/s | 1.7 MB 00:00 2022-05-13T23:02:36.924 INFO:teuthology.orchestra.run.smithi047.stdout:(36/128): cephfs-top-17.2.0-188.g23b06583.el8.n 354 kB/s | 14 kB 00:00 2022-05-13T23:02:36.950 INFO:teuthology.orchestra.run.smithi002.stdout:(89/128): python3-werkzeug-0.12.2-4.el8.noarch. 9.4 MB/s | 457 kB 00:00 2022-05-13T23:02:36.978 INFO:teuthology.orchestra.run.smithi002.stdout:(90/128): python3-markupsafe-0.23-19.el8.x86_64 1.4 MB/s | 39 kB 00:00 2022-05-13T23:02:37.043 INFO:teuthology.orchestra.run.smithi002.stdout:(91/128): python3-scipy-1.0.0-20.module+el8.1.0 36 MB/s | 15 MB 00:00 2022-05-13T23:02:37.052 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 2/134 2022-05-13T23:02:37.060 INFO:teuthology.orchestra.run.smithi002.stdout:(92/128): python3-prettytable-0.7.2-14.el8.noar 533 kB/s | 44 kB 00:00 2022-05-13T23:02:37.080 INFO:teuthology.orchestra.run.smithi002.stdout:(93/128): python3-babel-2.5.1-5.el8.noarch.rpm 20 MB/s | 4.8 MB 00:00 2022-05-13T23:02:37.112 INFO:teuthology.orchestra.run.smithi002.stdout:(94/128): socat-1.7.3.3-2.el8.x86_64.rpm 4.3 MB/s | 302 kB 00:00 2022-05-13T23:02:37.149 INFO:teuthology.orchestra.run.smithi002.stdout:(95/128): bison-3.0.4-10.el8.x86_64.rpm 12 MB/s | 688 kB 00:00 2022-05-13T23:02:37.152 INFO:teuthology.orchestra.run.smithi002.stdout:(96/128): python3-numpy-f2py-1.14.3-9.el8.x86_6 6.6 MB/s | 225 kB 00:00 2022-05-13T23:02:37.185 INFO:teuthology.orchestra.run.smithi002.stdout:(97/128): python3-jwt-1.6.1-2.el8.noarch.rpm 1.2 MB/s | 43 kB 00:00 2022-05-13T23:02:37.195 INFO:teuthology.orchestra.run.smithi002.stdout:(98/128): libaio-devel-0.3.112-1.el8.x86_64.rpm 528 kB/s | 19 kB 00:00 2022-05-13T23:02:37.198 INFO:teuthology.orchestra.run.smithi047.stdout:(37/128): gperftools-libs-2.7-9.el8.x86_64.rpm 1.1 MB/s | 306 kB 00:00 2022-05-13T23:02:37.211 INFO:teuthology.orchestra.run.smithi002.stdout:(99/128): python3-requests-oauthlib-1.0.0-1.el8 1.6 MB/s | 43 kB 00:00 2022-05-13T23:02:37.238 INFO:teuthology.orchestra.run.smithi002.stdout:(100/128): libquadmath-8.4.1-1.el8.x86_64.rpm 3.9 MB/s | 169 kB 00:00 2022-05-13T23:02:37.244 INFO:teuthology.orchestra.run.smithi047.stdout:(38/128): liboath-2.6.2-3.el8.x86_64.rpm 1.3 MB/s | 59 kB 00:00 2022-05-13T23:02:37.259 INFO:teuthology.orchestra.run.smithi002.stdout:(101/128): python3-cffi-1.11.5-5.el8.x86_64.rpm 4.9 MB/s | 238 kB 00:00 2022-05-13T23:02:37.290 INFO:teuthology.orchestra.run.smithi002.stdout:(102/128): libgfortran-8.4.1-1.el8.x86_64.rpm 12 MB/s | 639 kB 00:00 2022-05-13T23:02:37.292 INFO:teuthology.orchestra.run.smithi047.stdout:(39/128): libunwind-1.3.1-3.el8.x86_64.rpm 1.5 MB/s | 75 kB 00:00 2022-05-13T23:02:37.292 INFO:teuthology.orchestra.run.smithi002.stdout:(103/128): elfutils-libelf-devel-0.182-3.el8.x8 1.8 MB/s | 59 kB 00:00 2022-05-13T23:02:37.335 INFO:teuthology.orchestra.run.smithi002.stdout:(104/128): openblas-0.3.12-1.el8.x86_64.rpm 17 MB/s | 4.6 MB 00:00 2022-05-13T23:02:37.344 INFO:teuthology.orchestra.run.smithi002.stdout:(105/128): libselinux-devel-2.9-5.el8.x86_64.rp 3.6 MB/s | 200 kB 00:00 2022-05-13T23:02:37.353 INFO:teuthology.orchestra.run.smithi002.stdout:(106/128): zlib-devel-1.2.11-17.el8.x86_64.rpm 957 kB/s | 58 kB 00:00 2022-05-13T23:02:37.385 INFO:teuthology.orchestra.run.smithi002.stdout:(107/128): pcre2-utf16-10.32-2.el8.x86_64.rpm 5.2 MB/s | 229 kB 00:00 2022-05-13T23:02:37.393 INFO:teuthology.orchestra.run.smithi002.stdout:(108/128): librabbitmq-0.9.0-3.el8.x86_64.rpm 1.1 MB/s | 47 kB 00:00 2022-05-13T23:02:37.408 INFO:teuthology.orchestra.run.smithi002.stdout:(109/128): libverto-devel-0.3.0-5.el8.x86_64.rp 323 kB/s | 18 kB 00:00 2022-05-13T23:02:37.431 INFO:teuthology.orchestra.run.smithi002.stdout:(110/128): libsepol-devel-2.9-2.el8.x86_64.rpm 2.1 MB/s | 86 kB 00:00 2022-05-13T23:02:37.433 INFO:teuthology.orchestra.run.smithi002.stdout:(111/128): libuuid-devel-2.32.1-27.el8.x86_64.r 2.4 MB/s | 97 kB 00:00 2022-05-13T23:02:37.439 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 2/134 2022-05-13T23:02:37.449 INFO:teuthology.orchestra.run.smithi002.stdout:(112/128): python3-oauthlib-2.1.0-1.el8.noarch. 3.7 MB/s | 155 kB 00:00 2022-05-13T23:02:37.480 INFO:teuthology.orchestra.run.smithi047.stdout:(40/128): python3-asyncssh-2.7.0-2.el8.noarch.r 2.5 MB/s | 476 kB 00:00 2022-05-13T23:02:37.483 INFO:teuthology.orchestra.run.smithi002.stdout:(113/128): fuse-2.9.7-12.el8.x86_64.rpm 2.2 MB/s | 83 kB 00:00 2022-05-13T23:02:37.502 INFO:teuthology.orchestra.run.smithi002.stdout:(114/128): python3-pycparser-2.14-14.el8.noarch 2.7 MB/s | 109 kB 00:00 2022-05-13T23:02:37.528 INFO:teuthology.orchestra.run.smithi047.stdout:(41/128): python3-bcrypt-3.1.6-2.el8.1.x86_64.r 915 kB/s | 44 kB 00:00 2022-05-13T23:02:37.537 INFO:teuthology.orchestra.run.smithi002.stdout:(115/128): pcre2-devel-10.32-2.el8.x86_64.rpm 13 MB/s | 605 kB 00:00 2022-05-13T23:02:37.554 INFO:teuthology.orchestra.run.smithi047.stdout:(42/128): ceph-mgr-diskprediction-local-17.2.0- 6.4 MB/s | 7.4 MB 00:01 2022-05-13T23:02:37.561 INFO:teuthology.orchestra.run.smithi002.stdout:(116/128): python3-cryptography-3.2.1-4.el8.x86 11 MB/s | 559 kB 00:00 2022-05-13T23:02:37.563 INFO:teuthology.orchestra.run.smithi002.stdout:(117/128): xfsprogs-devel-5.0.0-8.el8.x86_64.rp 2.6 MB/s | 70 kB 00:00 2022-05-13T23:02:37.579 INFO:teuthology.orchestra.run.smithi002.stdout:(118/128): openssl-devel-1.1.1g-15.el8_3.x86_64 16 MB/s | 2.3 MB 00:00 2022-05-13T23:02:37.615 INFO:teuthology.orchestra.run.smithi047.stdout:(43/128): python3-beautifulsoup4-4.6.3-2.el8.1. 2.1 MB/s | 185 kB 00:00 2022-05-13T23:02:37.619 INFO:teuthology.orchestra.run.smithi002.stdout:(119/128): pcre2-utf32-10.32-2.el8.x86_64.rpm 3.8 MB/s | 220 kB 00:00 2022-05-13T23:02:37.619 INFO:teuthology.orchestra.run.smithi002.stdout:(120/128): keyutils-libs-devel-1.5.10-6.el8.x86 976 kB/s | 48 kB 00:00 2022-05-13T23:02:37.630 INFO:teuthology.orchestra.run.smithi002.stdout:(121/128): libcom_err-devel-1.45.6-1.el8.x86_64 771 kB/s | 38 kB 00:00 2022-05-13T23:02:37.651 INFO:teuthology.orchestra.run.smithi002.stdout:(122/128): krb5-devel-1.18.2-8.3.el8_4.x86_64.r 18 MB/s | 559 kB 00:00 2022-05-13T23:02:37.678 INFO:teuthology.orchestra.run.smithi002.stdout:(123/128): NetworkManager-team-1.30.0-13.el8_4. 5.3 MB/s | 146 kB 00:00 2022-05-13T23:02:37.679 INFO:teuthology.orchestra.run.smithi047.stdout:(44/128): python3-certifi-2018.10.15-7.el8.noar 250 kB/s | 16 kB 00:00 2022-05-13T23:02:37.681 INFO:teuthology.orchestra.run.smithi047.stdout:(45/128): python3-cachetools-3.1.1-4.el8.noarch 260 kB/s | 33 kB 00:00 2022-05-13T23:02:37.685 INFO:teuthology.orchestra.run.smithi115.stdout: Upgrading : librados2-2:17.2.0-188.g23b06583.el8.x86_64 3/134 2022-05-13T23:02:37.725 INFO:teuthology.orchestra.run.smithi002.stdout:(124/128): NetworkManager-libnm-1.30.0-13.el8_4 38 MB/s | 1.8 MB 00:00 2022-05-13T23:02:37.755 INFO:teuthology.orchestra.run.smithi002.stdout:(125/128): NetworkManager-tui-1.30.0-13.el8_4.x 11 MB/s | 329 kB 00:00 2022-05-13T23:02:37.770 INFO:teuthology.orchestra.run.smithi047.stdout:(46/128): python3-cheroot-8.5.2-1.el8.noarch.rp 1.9 MB/s | 173 kB 00:00 2022-05-13T23:02:37.815 INFO:teuthology.orchestra.run.smithi002.stdout:(126/128): NetworkManager-1.30.0-13.el8_4.x86_6 43 MB/s | 2.6 MB 00:00 2022-05-13T23:02:37.818 INFO:teuthology.orchestra.run.smithi047.stdout:(47/128): python3-google-auth-1.1.1-10.el8.noar 1.7 MB/s | 81 kB 00:00 2022-05-13T23:02:37.869 INFO:teuthology.orchestra.run.smithi047.stdout:(48/128): python3-jaraco-6.2-6.el8.noarch.rpm 224 kB/s | 11 kB 00:00 2022-05-13T23:02:37.913 INFO:teuthology.orchestra.run.smithi047.stdout:(49/128): python3-cherrypy-18.4.0-1.el8.noarch. 1.7 MB/s | 384 kB 00:00 2022-05-13T23:02:37.918 INFO:teuthology.orchestra.run.smithi047.stdout:(50/128): python3-jaraco-functools-2.0-4.el8.no 354 kB/s | 18 kB 00:00 2022-05-13T23:02:37.970 INFO:teuthology.orchestra.run.smithi047.stdout:(51/128): python3-logutils-0.3.5-11.el8.noarch. 940 kB/s | 49 kB 00:00 2022-05-13T23:02:38.022 INFO:teuthology.orchestra.run.smithi047.stdout:(52/128): python3-more-itertools-7.2.0-3.el8.no 1.1 MB/s | 59 kB 00:00 2022-05-13T23:02:38.074 INFO:teuthology.orchestra.run.smithi047.stdout:(53/128): python3-natsort-7.1.1-2.el8.noarch.rp 1.1 MB/s | 60 kB 00:00 2022-05-13T23:02:38.151 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: librados2-2:17.2.0-188.g23b06583.el8.x86_64 3/134 2022-05-13T23:02:38.151 INFO:teuthology.orchestra.run.smithi002.stdout:(127/128): librados2-17.2.0-188.g23b06583.el8.x 6.5 MB/s | 3.4 MB 00:00 2022-05-13T23:02:38.177 INFO:teuthology.orchestra.run.smithi047.stdout:(54/128): python3-pecan-1.3.2-9.el8.noarch.rpm 2.7 MB/s | 283 kB 00:00 2022-05-13T23:02:38.223 INFO:teuthology.orchestra.run.smithi047.stdout:(55/128): python3-portend-2.6-1.el8.noarch.rpm 343 kB/s | 16 kB 00:00 2022-05-13T23:02:38.270 INFO:teuthology.orchestra.run.smithi047.stdout:(56/128): python3-repoze-lru-0.7-6.el8.noarch.r 732 kB/s | 34 kB 00:00 2022-05-13T23:02:38.343 INFO:teuthology.orchestra.run.smithi115.stdout: Upgrading : librbd1-2:17.2.0-188.g23b06583.el8.x86_64 4/134 2022-05-13T23:02:38.349 INFO:teuthology.orchestra.run.smithi047.stdout:(57/128): python3-kubernetes-11.0.0-6.el8.noarc 2.6 MB/s | 1.2 MB 00:00 2022-05-13T23:02:38.364 INFO:teuthology.orchestra.run.smithi047.stdout:(58/128): python3-routes-2.4.1-12.el8.noarch.rp 2.0 MB/s | 196 kB 00:00 2022-05-13T23:02:38.396 INFO:teuthology.orchestra.run.smithi047.stdout:(59/128): python3-rsa-4.8-1.el8.noarch.rpm 1.3 MB/s | 61 kB 00:00 2022-05-13T23:02:38.412 INFO:teuthology.orchestra.run.smithi047.stdout:(60/128): python3-simplegeneric-0.8.1-17.el8.no 395 kB/s | 19 kB 00:00 2022-05-13T23:02:38.443 INFO:teuthology.orchestra.run.smithi047.stdout:(61/128): python3-singledispatch-3.4.0.3-18.el8 512 kB/s | 24 kB 00:00 2022-05-13T23:02:38.458 INFO:teuthology.orchestra.run.smithi047.stdout:(62/128): python3-tempora-1.14.1-5.el8.noarch.r 630 kB/s | 29 kB 00:00 2022-05-13T23:02:38.460 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: librbd1-2:17.2.0-188.g23b06583.el8.x86_64 4/134 2022-05-13T23:02:38.490 INFO:teuthology.orchestra.run.smithi047.stdout:(63/128): python3-trustme-0.6.0-4.el8.noarch.rp 581 kB/s | 27 kB 00:00 2022-05-13T23:02:38.563 INFO:teuthology.orchestra.run.smithi047.stdout:(64/128): python3-waitress-1.2.1-2.el8.1.noarch 2.3 MB/s | 241 kB 00:00 2022-05-13T23:02:38.600 INFO:teuthology.orchestra.run.smithi047.stdout:(65/128): python3-webob-1.8.5-1.el8.1.noarch.rp 2.2 MB/s | 251 kB 00:00 2022-05-13T23:02:38.614 INFO:teuthology.orchestra.run.smithi047.stdout:(66/128): python3-websocket-client-0.56.0-5.el8 1.2 MB/s | 61 kB 00:00 2022-05-13T23:02:38.647 INFO:teuthology.orchestra.run.smithi047.stdout:(67/128): python3-webtest-2.0.33-1.el8.noarch.r 1.8 MB/s | 85 kB 00:00 2022-05-13T23:02:38.652 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : libcephfs2-2:17.2.0-188.g23b06583.el8.x86_64 5/134 2022-05-13T23:02:38.659 INFO:teuthology.orchestra.run.smithi047.stdout:(68/128): python3-zc-lockfile-2.0-2.el8.noarch. 504 kB/s | 23 kB 00:00 2022-05-13T23:02:38.708 INFO:teuthology.orchestra.run.smithi047.stdout:(69/128): xmlstarlet-1.6.1-11.el8.x86_64.rpm 1.4 MB/s | 69 kB 00:00 2022-05-13T23:02:38.746 INFO:teuthology.orchestra.run.smithi047.stdout:(70/128): python3-jwt-1.6.1-2.el8.noarch.rpm 1.1 MB/s | 43 kB 00:00 2022-05-13T23:02:38.755 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: libcephfs2-2:17.2.0-188.g23b06583.el8.x86_64 5/134 2022-05-13T23:02:38.772 INFO:teuthology.orchestra.run.smithi047.stdout:(71/128): libaio-devel-0.3.112-1.el8.x86_64.rpm 744 kB/s | 19 kB 00:00 2022-05-13T23:02:38.797 INFO:teuthology.orchestra.run.smithi047.stdout:(72/128): python3-requests-oauthlib-1.0.0-1.el8 1.7 MB/s | 43 kB 00:00 2022-05-13T23:02:38.823 INFO:teuthology.orchestra.run.smithi047.stdout:(73/128): libquadmath-8.4.1-1.el8.x86_64.rpm 6.5 MB/s | 169 kB 00:00 2022-05-13T23:02:38.850 INFO:teuthology.orchestra.run.smithi047.stdout:(74/128): python3-cffi-1.11.5-5.el8.x86_64.rpm 8.8 MB/s | 238 kB 00:00 2022-05-13T23:02:38.879 INFO:teuthology.orchestra.run.smithi047.stdout:(75/128): libgfortran-8.4.1-1.el8.x86_64.rpm 21 MB/s | 639 kB 00:00 2022-05-13T23:02:38.904 INFO:teuthology.orchestra.run.smithi047.stdout:(76/128): elfutils-libelf-devel-0.182-3.el8.x86 2.3 MB/s | 59 kB 00:00 2022-05-13T23:02:38.931 INFO:teuthology.orchestra.run.smithi047.stdout:(77/128): libselinux-devel-2.9-5.el8.x86_64.rpm 7.3 MB/s | 200 kB 00:00 2022-05-13T23:02:38.956 INFO:teuthology.orchestra.run.smithi047.stdout:(78/128): zlib-devel-1.2.11-17.el8.x86_64.rpm 2.3 MB/s | 58 kB 00:00 2022-05-13T23:02:38.983 INFO:teuthology.orchestra.run.smithi047.stdout:(79/128): pcre2-utf16-10.32-2.el8.x86_64.rpm 8.4 MB/s | 229 kB 00:00 2022-05-13T23:02:38.984 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-rados-2:17.2.0-188.g23b06583.el8.x86_64 6/134 2022-05-13T23:02:39.008 INFO:teuthology.orchestra.run.smithi047.stdout:(80/128): librabbitmq-0.9.0-3.el8.x86_64.rpm 1.8 MB/s | 47 kB 00:00 2022-05-13T23:02:39.032 INFO:teuthology.orchestra.run.smithi002.stdout:(128/128): librbd1-17.2.0-188.g23b06583.el8.x86 2.8 MB/s | 3.9 MB 00:01 2022-05-13T23:02:39.033 INFO:teuthology.orchestra.run.smithi047.stdout:(81/128): libverto-devel-0.3.0-5.el8.x86_64.rpm 723 kB/s | 18 kB 00:00 2022-05-13T23:02:39.036 INFO:teuthology.orchestra.run.smithi002.stdout:-------------------------------------------------------------------------------- 2022-05-13T23:02:39.036 INFO:teuthology.orchestra.run.smithi002.stdout:Total 16 MB/s | 207 MB 00:13 2022-05-13T23:02:39.058 INFO:teuthology.orchestra.run.smithi047.stdout:(82/128): libsepol-devel-2.9-2.el8.x86_64.rpm 3.4 MB/s | 86 kB 00:00 2022-05-13T23:02:39.083 INFO:teuthology.orchestra.run.smithi047.stdout:(83/128): libuuid-devel-2.32.1-27.el8.x86_64.rp 3.8 MB/s | 97 kB 00:00 2022-05-13T23:02:39.109 INFO:teuthology.orchestra.run.smithi047.stdout:(84/128): python3-oauthlib-2.1.0-1.el8.noarch.r 5.9 MB/s | 155 kB 00:00 2022-05-13T23:02:39.136 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : librdkafka-0.11.4-1.el8.x86_64 7/134 2022-05-13T23:02:39.149 INFO:teuthology.orchestra.run.smithi002.stdout:Running transaction check 2022-05-13T23:02:39.150 INFO:teuthology.orchestra.run.smithi047.stdout:(85/128): openssl-devel-1.1.1g-15.el8_3.x86_64. 56 MB/s | 2.3 MB 00:00 2022-05-13T23:02:39.174 INFO:teuthology.orchestra.run.smithi047.stdout:(86/128): thrift-0.13.0-2.el8.x86_64.rpm 3.3 MB/s | 1.7 MB 00:00 2022-05-13T23:02:39.175 INFO:teuthology.orchestra.run.smithi047.stdout:(87/128): fuse-2.9.7-12.el8.x86_64.rpm 3.3 MB/s | 83 kB 00:00 2022-05-13T23:02:39.220 INFO:teuthology.orchestra.run.smithi047.stdout:(88/128): pcre2-devel-10.32-2.el8.x86_64.rpm 13 MB/s | 605 kB 00:00 2022-05-13T23:02:39.230 INFO:teuthology.orchestra.run.smithi047.stdout:(89/128): python3-pycparser-2.14-14.el8.noarch. 1.9 MB/s | 109 kB 00:00 2022-05-13T23:02:39.235 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: librdkafka-0.11.4-1.el8.x86_64 7/134 2022-05-13T23:02:39.255 INFO:teuthology.orchestra.run.smithi047.stdout:(90/128): python3-cryptography-3.2.1-4.el8.x86_ 15 MB/s | 559 kB 00:00 2022-05-13T23:02:39.265 INFO:teuthology.orchestra.run.smithi047.stdout:(91/128): xfsprogs-devel-5.0.0-8.el8.x86_64.rpm 1.9 MB/s | 70 kB 00:00 2022-05-13T23:02:39.283 INFO:teuthology.orchestra.run.smithi047.stdout:(92/128): pcre2-utf32-10.32-2.el8.x86_64.rpm 8.0 MB/s | 220 kB 00:00 2022-05-13T23:02:39.299 INFO:teuthology.orchestra.run.smithi047.stdout:(93/128): keyutils-libs-devel-1.5.10-6.el8.x86_ 1.4 MB/s | 48 kB 00:00 2022-05-13T23:02:39.304 INFO:teuthology.orchestra.run.smithi002.stdout:Transaction check succeeded. 2022-05-13T23:02:39.304 INFO:teuthology.orchestra.run.smithi002.stdout:Running transaction test 2022-05-13T23:02:39.315 INFO:teuthology.orchestra.run.smithi047.stdout:(94/128): libcom_err-devel-1.45.6-1.el8.x86_64. 1.2 MB/s | 38 kB 00:00 2022-05-13T23:02:39.341 INFO:teuthology.orchestra.run.smithi047.stdout:(95/128): krb5-devel-1.18.2-8.3.el8_4.x86_64.rp 13 MB/s | 559 kB 00:00 2022-05-13T23:02:39.352 INFO:teuthology.orchestra.run.smithi047.stdout:(96/128): python3-mako-1.0.6-13.el8.noarch.rpm 4.2 MB/s | 157 kB 00:00 2022-05-13T23:02:39.370 INFO:teuthology.orchestra.run.smithi047.stdout:(97/128): flex-2.6.1-9.el8.x86_64.rpm 11 MB/s | 320 kB 00:00 2022-05-13T23:02:39.388 INFO:teuthology.orchestra.run.smithi047.stdout:(98/128): jq-1.5-12.el8.x86_64.rpm 4.4 MB/s | 161 kB 00:00 2022-05-13T23:02:39.404 INFO:teuthology.orchestra.run.smithi047.stdout:(99/128): python3-jsonpatch-1.21-2.el8.noarch.r 989 kB/s | 27 kB 00:00 2022-05-13T23:02:39.421 INFO:teuthology.orchestra.run.smithi047.stdout:(100/128): librdkafka-0.11.4-1.el8.x86_64.rpm 10 MB/s | 353 kB 00:00 2022-05-13T23:02:39.448 INFO:teuthology.orchestra.run.smithi047.stdout:(101/128): python3-jinja2-2.10.1-2.el8_0.noarch 12 MB/s | 538 kB 00:00 2022-05-13T23:02:39.469 INFO:teuthology.orchestra.run.smithi047.stdout:(102/128): automake-1.16.1-7.el8.noarch.rpm 17 MB/s | 713 kB 00:00 2022-05-13T23:02:39.490 INFO:teuthology.orchestra.run.smithi047.stdout:(103/128): libtool-2.4.6-25.el8.x86_64.rpm 17 MB/s | 709 kB 00:00 2022-05-13T23:02:39.518 INFO:teuthology.orchestra.run.smithi047.stdout:(104/128): python3-pyasn1-modules-0.3.7-6.el8.n 3.9 MB/s | 110 kB 00:00 2022-05-13T23:02:39.548 INFO:teuthology.orchestra.run.smithi047.stdout:(105/128): python3-numpy-1.14.3-9.el8.x86_64.rp 47 MB/s | 3.7 MB 00:00 2022-05-13T23:02:39.550 INFO:teuthology.orchestra.run.smithi047.stdout:(106/128): python3-jsonpointer-1.10-11.el8.noar 654 kB/s | 20 kB 00:00 2022-05-13T23:02:39.563 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : librabbitmq-0.9.0-3.el8.x86_64 8/134 2022-05-13T23:02:39.581 INFO:teuthology.orchestra.run.smithi047.stdout:(107/128): python3-pytz-2017.2-9.el8.noarch.rpm 1.7 MB/s | 54 kB 00:00 2022-05-13T23:02:39.591 INFO:teuthology.orchestra.run.smithi047.stdout:(108/128): oniguruma-6.8.2-2.el8.x86_64.rpm 4.4 MB/s | 187 kB 00:00 2022-05-13T23:02:39.607 INFO:teuthology.orchestra.run.smithi047.stdout:(109/128): python3-pyasn1-0.3.7-6.el8.noarch.rp 4.7 MB/s | 126 kB 00:00 2022-05-13T23:02:39.642 INFO:teuthology.orchestra.run.smithi047.stdout:(110/128): python3-pyOpenSSL-19.0.0-1.el8.noarc 2.9 MB/s | 103 kB 00:00 2022-05-13T23:02:39.669 INFO:teuthology.orchestra.run.smithi047.stdout:(111/128): lttng-ust-2.8.1-11.el8.x86_64.rpm 9.5 MB/s | 259 kB 00:00 2022-05-13T23:02:39.737 INFO:teuthology.orchestra.run.smithi047.stdout:(112/128): openblas-threads-0.3.12-1.el8.x86_64 70 MB/s | 4.7 MB 00:00 2022-05-13T23:02:39.860 INFO:teuthology.orchestra.run.smithi047.stdout:(113/128): python3-scipy-1.0.0-20.module+el8.1. 54 MB/s | 15 MB 00:00 2022-05-13T23:02:39.892 INFO:teuthology.orchestra.run.smithi047.stdout:(114/128): python3-werkzeug-0.12.2-4.el8.noarch 15 MB/s | 457 kB 00:00 2022-05-13T23:02:39.926 INFO:teuthology.orchestra.run.smithi047.stdout:(115/128): python3-babel-2.5.1-5.el8.noarch.rpm 25 MB/s | 4.8 MB 00:00 2022-05-13T23:02:39.927 INFO:teuthology.orchestra.run.smithi047.stdout:(116/128): python3-markupsafe-0.23-19.el8.x86_6 1.0 MB/s | 39 kB 00:00 2022-05-13T23:02:39.957 INFO:teuthology.orchestra.run.smithi047.stdout:(117/128): python3-prettytable-0.7.2-14.el8.noa 1.4 MB/s | 44 kB 00:00 2022-05-13T23:02:39.968 INFO:teuthology.orchestra.run.smithi047.stdout:(118/128): socat-1.7.3.3-2.el8.x86_64.rpm 7.1 MB/s | 302 kB 00:00 2022-05-13T23:02:40.008 INFO:teuthology.orchestra.run.smithi047.stdout:(119/128): bison-3.0.4-10.el8.x86_64.rpm 21 MB/s | 688 kB 00:00 2022-05-13T23:02:40.042 INFO:teuthology.orchestra.run.smithi047.stdout:(120/128): openblas-0.3.12-1.el8.x86_64.rpm 55 MB/s | 4.6 MB 00:00 2022-05-13T23:02:40.044 INFO:teuthology.orchestra.run.smithi047.stdout:(121/128): python3-numpy-f2py-1.14.3-9.el8.x86_ 6.0 MB/s | 225 kB 00:00 2022-05-13T23:02:40.250 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : thrift-0.13.0-2.el8.x86_64 9/134 2022-05-13T23:02:40.468 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : librgw2-2:17.2.0-188.g23b06583.el8.x86_64 10/134 2022-05-13T23:02:40.583 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: librgw2-2:17.2.0-188.g23b06583.el8.x86_64 10/134 2022-05-13T23:02:40.602 INFO:teuthology.orchestra.run.smithi047.stdout:(122/128): librados2-17.2.0-188.g23b06583.el8.x 6.1 MB/s | 3.4 MB 00:00 2022-05-13T23:02:40.629 INFO:teuthology.orchestra.run.smithi047.stdout:(123/128): NetworkManager-team-1.30.0-13.el8_4. 5.4 MB/s | 146 kB 00:00 2022-05-13T23:02:40.666 INFO:teuthology.orchestra.run.smithi047.stdout:(124/128): NetworkManager-libnm-1.30.0-13.el8_4 48 MB/s | 1.8 MB 00:00 2022-05-13T23:02:40.695 INFO:teuthology.orchestra.run.smithi047.stdout:(125/128): NetworkManager-tui-1.30.0-13.el8_4.x 12 MB/s | 329 kB 00:00 2022-05-13T23:02:40.739 INFO:teuthology.orchestra.run.smithi047.stdout:(126/128): NetworkManager-1.30.0-13.el8_4.x86_6 58 MB/s | 2.6 MB 00:00 2022-05-13T23:02:40.772 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 11/134 2022-05-13T23:02:40.928 INFO:teuthology.orchestra.run.smithi047.stdout:(127/128): librbd1-17.2.0-188.g23b06583.el8.x86 4.4 MB/s | 3.9 MB 00:00 2022-05-13T23:02:40.949 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 12/134 2022-05-13T23:02:41.039 INFO:teuthology.orchestra.run.smithi002.stdout:Transaction test succeeded. 2022-05-13T23:02:41.043 INFO:teuthology.orchestra.run.smithi002.stdout:Running transaction 2022-05-13T23:02:41.085 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : libcephsqlite-2:17.2.0-188.g23b06583.el8.x86_64 13/134 2022-05-13T23:02:41.212 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: libcephsqlite-2:17.2.0-188.g23b06583.el8.x86_64 13/134 2022-05-13T23:02:41.859 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : libradosstriper1-2:17.2.0-188.g23b06583.el8.x86_ 14/134 2022-05-13T23:02:42.118 INFO:teuthology.orchestra.run.smithi047.stdout:(128/128): ceph-test-17.2.0-188.g23b06583.el8.x 6.6 MB/s | 52 MB 00:07 2022-05-13T23:02:42.119 INFO:teuthology.orchestra.run.smithi047.stdout:-------------------------------------------------------------------------------- 2022-05-13T23:02:42.119 INFO:teuthology.orchestra.run.smithi047.stdout:Total 11 MB/s | 207 MB 00:19 2022-05-13T23:02:42.123 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: libradosstriper1-2:17.2.0-188.g23b06583.el8.x86_ 14/134 2022-05-13T23:02:42.186 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction check 2022-05-13T23:02:42.326 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction check succeeded. 2022-05-13T23:02:42.327 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction test 2022-05-13T23:02:42.426 INFO:teuthology.orchestra.run.smithi115.stdout: Upgrading : NetworkManager-libnm-1:1.30.0-13.el8_4.x86_64 15/134 2022-05-13T23:02:42.483 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: NetworkManager-libnm-1:1.30.0-13.el8_4.x86_64 15/134 2022-05-13T23:02:42.719 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: NetworkManager-1:1.30.0-13.el8_4.x86_64 16/134 2022-05-13T23:02:43.010 INFO:teuthology.orchestra.run.smithi115.stdout: Upgrading : NetworkManager-1:1.30.0-13.el8_4.x86_64 16/134 2022-05-13T23:02:43.343 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: NetworkManager-1:1.30.0-13.el8_4.x86_64 16/134 2022-05-13T23:02:43.479 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction test succeeded. 2022-05-13T23:02:43.483 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction 2022-05-13T23:02:43.617 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 17/134 2022-05-13T23:02:43.850 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 18/134 2022-05-13T23:02:44.065 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-pytz-2017.2-9.el8.noarch 19/134 2022-05-13T23:02:44.256 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : zlib-devel-1.2.11-17.el8.x86_64 20/134 2022-05-13T23:02:44.450 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : libquadmath-8.4.1-1.el8.x86_64 21/134 2022-05-13T23:02:44.571 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: libquadmath-8.4.1-1.el8.x86_64 21/134 2022-05-13T23:02:44.759 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : libgfortran-8.4.1-1.el8.x86_64 22/134 2022-05-13T23:02:45.307 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: libgfortran-8.4.1-1.el8.x86_64 22/134 2022-05-13T23:02:45.530 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : openblas-threads-0.3.12-1.el8.x86_64 23/134 2022-05-13T23:02:46.057 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: openblas-threads-0.3.12-1.el8.x86_64 23/134 2022-05-13T23:02:46.242 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : openblas-0.3.12-1.el8.x86_64 24/134 2022-05-13T23:02:46.459 INFO:teuthology.orchestra.run.smithi002.stdout: Preparing : 1/1 2022-05-13T23:02:46.535 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: liboath-2.6.2-3.el8.x86_64 1/1 2022-05-13T23:02:46.710 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : liboath-2.6.2-3.el8.x86_64 1/134 2022-05-13T23:02:46.995 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: openblas-0.3.12-1.el8.x86_64 24/134 2022-05-13T23:02:46.998 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : librabbitmq-0.9.0-3.el8.x86_64 2/134 2022-05-13T23:02:47.178 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 3/134 2022-05-13T23:02:47.640 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-numpy-1:1.14.3-9.el8.x86_64 25/134 2022-05-13T23:02:47.699 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 3/134 2022-05-13T23:02:47.944 INFO:teuthology.orchestra.run.smithi002.stdout: Upgrading : librados2-2:17.2.0-188.g23b06583.el8.x86_64 4/134 2022-05-13T23:02:47.945 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 26/134 2022-05-13T23:02:48.165 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-ceph-common-2:17.2.0-188.g23b06583.el8.x 27/134 2022-05-13T23:02:48.310 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-ceph-argparse-2:17.2.0-188.g23b06583.el8 28/134 2022-05-13T23:02:48.521 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: librados2-2:17.2.0-188.g23b06583.el8.x86_64 4/134 2022-05-13T23:02:48.590 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-cephfs-2:17.2.0-188.g23b06583.el8.x86_64 29/134 2022-05-13T23:02:48.703 INFO:teuthology.orchestra.run.smithi002.stdout: Upgrading : librbd1-2:17.2.0-188.g23b06583.el8.x86_64 5/134 2022-05-13T23:02:48.739 INFO:teuthology.orchestra.run.smithi047.stdout: Preparing : 1/1 2022-05-13T23:02:48.768 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: liboath-2.6.2-3.el8.x86_64 1/1 2022-05-13T23:02:48.858 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-numpy-f2py-1:1.14.3-9.el8.x86_64 30/134 2022-05-13T23:02:48.929 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: librbd1-2:17.2.0-188.g23b06583.el8.x86_64 5/134 2022-05-13T23:02:49.103 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : libcephfs2-2:17.2.0-188.g23b06583.el8.x86_64 6/134 2022-05-13T23:02:49.330 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : liboath-2.6.2-3.el8.x86_64 1/134 2022-05-13T23:02:49.536 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 2/134 2022-05-13T23:02:49.594 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: libcephfs2-2:17.2.0-188.g23b06583.el8.x86_64 6/134 2022-05-13T23:02:49.829 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-rados-2:17.2.0-188.g23b06583.el8.x86_64 7/134 2022-05-13T23:02:49.990 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 2/134 2022-05-13T23:02:50.003 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : librdkafka-0.11.4-1.el8.x86_64 8/134 2022-05-13T23:02:50.236 INFO:teuthology.orchestra.run.smithi047.stdout: Upgrading : librados2-2:17.2.0-188.g23b06583.el8.x86_64 3/134 2022-05-13T23:02:50.306 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: librdkafka-0.11.4-1.el8.x86_64 8/134 2022-05-13T23:02:50.708 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: librados2-2:17.2.0-188.g23b06583.el8.x86_64 3/134 2022-05-13T23:02:50.902 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-9.el8.x86_64 30/134 2022-05-13T23:02:50.919 INFO:teuthology.orchestra.run.smithi047.stdout: Upgrading : librbd1-2:17.2.0-188.g23b06583.el8.x86_64 4/134 2022-05-13T23:02:51.036 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: librbd1-2:17.2.0-188.g23b06583.el8.x86_64 4/134 2022-05-13T23:02:51.159 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : thrift-0.13.0-2.el8.x86_64 9/134 2022-05-13T23:02:51.252 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : libcephfs2-2:17.2.0-188.g23b06583.el8.x86_64 5/134 2022-05-13T23:02:51.344 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : librgw2-2:17.2.0-188.g23b06583.el8.x86_64 10/134 2022-05-13T23:02:51.374 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: libcephfs2-2:17.2.0-188.g23b06583.el8.x86_64 5/134 2022-05-13T23:02:51.532 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: librgw2-2:17.2.0-188.g23b06583.el8.x86_64 10/134 2022-05-13T23:02:51.646 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-rados-2:17.2.0-188.g23b06583.el8.x86_64 6/134 2022-05-13T23:02:51.753 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 11/134 2022-05-13T23:02:51.836 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : librdkafka-0.11.4-1.el8.x86_64 7/134 2022-05-13T23:02:51.907 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: librdkafka-0.11.4-1.el8.x86_64 7/134 2022-05-13T23:02:51.971 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 12/134 2022-05-13T23:02:52.038 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-scipy-1.0.0-20.module+el8.1.0+3334+5cb62 31/134 2022-05-13T23:02:52.119 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : libcephsqlite-2:17.2.0-188.g23b06583.el8.x86_64 13/134 2022-05-13T23:02:52.258 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : librabbitmq-0.9.0-3.el8.x86_64 8/134 2022-05-13T23:02:52.286 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: libcephsqlite-2:17.2.0-188.g23b06583.el8.x86_64 13/134 2022-05-13T23:02:52.390 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-babel-2.5.1-5.el8.noarch 32/134 2022-05-13T23:02:52.452 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : libradosstriper1-2:17.2.0-188.g23b06583.el8.x86_ 14/134 2022-05-13T23:02:52.948 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: libradosstriper1-2:17.2.0-188.g23b06583.el8.x86_ 14/134 2022-05-13T23:02:53.019 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : thrift-0.13.0-2.el8.x86_64 9/134 2022-05-13T23:02:53.154 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-jinja2-2.10.1-2.el8_0.noarch 33/134 2022-05-13T23:02:53.211 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : librgw2-2:17.2.0-188.g23b06583.el8.x86_64 10/134 2022-05-13T23:02:53.235 INFO:teuthology.orchestra.run.smithi002.stdout: Upgrading : NetworkManager-libnm-1:1.30.0-13.el8_4.x86_64 15/134 2022-05-13T23:02:53.357 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: NetworkManager-libnm-1:1.30.0-13.el8_4.x86_64 15/134 2022-05-13T23:02:53.359 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: librgw2-2:17.2.0-188.g23b06583.el8.x86_64 10/134 2022-05-13T23:02:53.504 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-mako-1.0.6-13.el8.noarch 34/134 2022-05-13T23:02:53.591 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 11/134 2022-05-13T23:02:53.718 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: NetworkManager-1:1.30.0-13.el8_4.x86_64 16/134 2022-05-13T23:02:53.743 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 12/134 2022-05-13T23:02:53.756 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-rsa-4.8-1.el8.noarch 35/134 2022-05-13T23:02:53.902 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : libcephsqlite-2:17.2.0-188.g23b06583.el8.x86_64 13/134 2022-05-13T23:02:53.953 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 36/134 2022-05-13T23:02:54.012 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: libcephsqlite-2:17.2.0-188.g23b06583.el8.x86_64 13/134 2022-05-13T23:02:54.028 INFO:teuthology.orchestra.run.smithi002.stdout: Upgrading : NetworkManager-1:1.30.0-13.el8_4.x86_64 16/134 2022-05-13T23:02:54.192 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-rgw-2:17.2.0-188.g23b06583.el8.x86_64 37/134 2022-05-13T23:02:54.354 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-rbd-2:17.2.0-188.g23b06583.el8.x86_64 38/134 2022-05-13T23:02:54.354 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : libradosstriper1-2:17.2.0-188.g23b06583.el8.x86_ 14/134 2022-05-13T23:02:54.588 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : librados-devel-2:17.2.0-188.g23b06583.el8.x86_64 39/134 2022-05-13T23:02:54.654 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: libradosstriper1-2:17.2.0-188.g23b06583.el8.x86_ 14/134 2022-05-13T23:02:54.833 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : socat-1.7.3.3-2.el8.x86_64 40/134 2022-05-13T23:02:54.868 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: NetworkManager-1:1.30.0-13.el8_4.x86_64 16/134 2022-05-13T23:02:54.885 INFO:teuthology.orchestra.run.smithi047.stdout: Upgrading : NetworkManager-libnm-1:1.30.0-13.el8_4.x86_64 15/134 2022-05-13T23:02:54.954 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: NetworkManager-libnm-1:1.30.0-13.el8_4.x86_64 15/134 2022-05-13T23:02:54.977 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 41/134 2022-05-13T23:02:55.173 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : oniguruma-6.8.2-2.el8.x86_64 42/134 2022-05-13T23:02:55.202 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : zlib-devel-1.2.11-17.el8.x86_64 17/134 2022-05-13T23:02:55.214 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: NetworkManager-1:1.30.0-13.el8_4.x86_64 16/134 2022-05-13T23:02:55.253 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: oniguruma-6.8.2-2.el8.x86_64 42/134 2022-05-13T23:02:55.368 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : libquadmath-8.4.1-1.el8.x86_64 18/134 2022-05-13T23:02:55.448 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : jq-1.5-12.el8.x86_64 43/134 2022-05-13T23:02:55.512 INFO:teuthology.orchestra.run.smithi047.stdout: Upgrading : NetworkManager-1:1.30.0-13.el8_4.x86_64 16/134 2022-05-13T23:02:55.575 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: libquadmath-8.4.1-1.el8.x86_64 18/134 2022-05-13T23:02:55.624 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 44/134 2022-05-13T23:02:55.744 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : libgfortran-8.4.1-1.el8.x86_64 19/134 2022-05-13T23:02:55.855 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 45/134 2022-05-13T23:02:55.856 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: NetworkManager-1:1.30.0-13.el8_4.x86_64 16/134 2022-05-13T23:02:56.037 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : automake-1.16.1-7.el8.noarch 46/134 2022-05-13T23:02:56.133 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 17/134 2022-05-13T23:02:56.229 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : libcom_err-devel-1.45.6-1.el8.x86_64 47/134 2022-05-13T23:02:56.369 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 18/134 2022-05-13T23:02:56.458 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: libgfortran-8.4.1-1.el8.x86_64 19/134 2022-05-13T23:02:56.459 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : keyutils-libs-devel-1.5.10-6.el8.x86_64 48/134 2022-05-13T23:02:56.575 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-pytz-2017.2-9.el8.noarch 19/134 2022-05-13T23:02:56.651 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : openblas-threads-0.3.12-1.el8.x86_64 20/134 2022-05-13T23:02:56.658 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : pcre2-utf32-10.32-2.el8.x86_64 49/134 2022-05-13T23:02:56.792 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : zlib-devel-1.2.11-17.el8.x86_64 20/134 2022-05-13T23:02:56.926 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 50/134 2022-05-13T23:02:56.960 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : libquadmath-8.4.1-1.el8.x86_64 21/134 2022-05-13T23:02:57.096 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: libquadmath-8.4.1-1.el8.x86_64 21/134 2022-05-13T23:02:57.286 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : libgfortran-8.4.1-1.el8.x86_64 22/134 2022-05-13T23:02:57.341 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: openblas-threads-0.3.12-1.el8.x86_64 20/134 2022-05-13T23:02:57.349 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-cffi-1.11.5-5.el8.x86_64 51/134 2022-05-13T23:02:57.577 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : openblas-0.3.12-1.el8.x86_64 21/134 2022-05-13T23:02:57.642 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-cryptography-3.2.1-4.el8.x86_64 52/134 2022-05-13T23:02:57.810 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 53/134 2022-05-13T23:02:57.855 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: libgfortran-8.4.1-1.el8.x86_64 22/134 2022-05-13T23:02:58.055 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 54/134 2022-05-13T23:02:58.084 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : openblas-threads-0.3.12-1.el8.x86_64 23/134 2022-05-13T23:02:58.321 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 55/134 2022-05-13T23:02:58.600 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: openblas-0.3.12-1.el8.x86_64 21/134 2022-05-13T23:02:58.665 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 56/134 2022-05-13T23:02:58.669 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: openblas-threads-0.3.12-1.el8.x86_64 23/134 2022-05-13T23:02:58.863 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-numpy-1:1.14.3-9.el8.x86_64 22/134 2022-05-13T23:02:58.881 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-asyncssh-2.7.0-2.el8.noarch 57/134 2022-05-13T23:02:59.010 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : openblas-0.3.12-1.el8.x86_64 24/134 2022-05-13T23:02:59.099 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 58/134 2022-05-13T23:02:59.175 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 23/134 2022-05-13T23:02:59.349 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 59/134 2022-05-13T23:02:59.407 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 24/134 2022-05-13T23:02:59.557 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : fuse-2.9.7-12.el8.x86_64 60/134 2022-05-13T23:02:59.669 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-pytz-2017.2-9.el8.noarch 25/134 2022-05-13T23:02:59.735 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : libuuid-devel-2.32.1-27.el8.x86_64 61/134 2022-05-13T23:02:59.785 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: openblas-0.3.12-1.el8.x86_64 24/134 2022-05-13T23:02:59.949 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 26/134 2022-05-13T23:02:59.958 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : libsepol-devel-2.9-2.el8.x86_64 62/134 2022-05-13T23:03:00.102 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-numpy-1:1.14.3-9.el8.x86_64 25/134 2022-05-13T23:03:00.135 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-ceph-common-2:17.2.0-188.g23b06583.el8.x 27/134 2022-05-13T23:03:00.136 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : libverto-devel-0.3.0-5.el8.x86_64 63/134 2022-05-13T23:03:00.282 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-ceph-argparse-2:17.2.0-188.g23b06583.el8 28/134 2022-05-13T23:03:00.402 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 26/134 2022-05-13T23:03:00.435 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : pcre2-utf16-10.32-2.el8.x86_64 64/134 2022-05-13T23:03:00.617 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-ceph-common-2:17.2.0-188.g23b06583.el8.x 27/134 2022-05-13T23:03:00.780 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : pcre2-devel-10.32-2.el8.x86_64 65/134 2022-05-13T23:03:00.812 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-ceph-argparse-2:17.2.0-188.g23b06583.el8 28/134 2022-05-13T23:03:01.072 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : libselinux-devel-2.9-5.el8.x86_64 66/134 2022-05-13T23:03:01.088 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-cephfs-2:17.2.0-188.g23b06583.el8.x86_64 29/134 2022-05-13T23:03:01.326 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-numpy-f2py-1:1.14.3-9.el8.x86_64 30/134 2022-05-13T23:03:01.340 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : krb5-devel-1.18.2-8.3.el8_4.x86_64 67/134 2022-05-13T23:03:01.504 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-cephfs-2:17.2.0-188.g23b06583.el8.x86_64 29/134 2022-05-13T23:03:01.588 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : xmlstarlet-1.6.1-11.el8.x86_64 68/134 2022-05-13T23:03:01.759 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 69/134 2022-05-13T23:03:01.881 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-babel-2.5.1-5.el8.noarch 30/134 2022-05-13T23:03:02.057 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 70/134 2022-05-13T23:03:02.243 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-jinja2-2.10.1-2.el8_0.noarch 31/134 2022-05-13T23:03:02.287 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-waitress-1.2.1-2.el8.1.noarch 71/134 2022-05-13T23:03:02.467 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 72/134 2022-05-13T23:03:02.581 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-mako-1.0.6-13.el8.noarch 32/134 2022-05-13T23:03:02.619 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 73/134 2022-05-13T23:03:02.784 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-numpy-f2py-1:1.14.3-9.el8.x86_64 33/134 2022-05-13T23:03:02.843 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 74/134 2022-05-13T23:03:03.035 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 75/134 2022-05-13T23:03:03.201 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-natsort-7.1.1-2.el8.noarch 76/134 2022-05-13T23:03:03.475 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 77/134 2022-05-13T23:03:03.513 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-9.el8.x86_64 30/134 2022-05-13T23:03:03.663 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 78/134 2022-05-13T23:03:03.910 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 79/134 2022-05-13T23:03:04.160 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 80/134 2022-05-13T23:03:04.362 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 81/134 2022-05-13T23:03:04.647 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-portend-2.6-1.el8.noarch 82/134 2022-05-13T23:03:04.675 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-scipy-1.0.0-20.module+el8.1.0+3334+5cb62 31/134 2022-05-13T23:03:04.906 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 83/134 2022-05-13T23:03:05.107 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 84/134 2022-05-13T23:03:05.194 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-babel-2.5.1-5.el8.noarch 32/134 2022-05-13T23:03:05.353 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 85/134 2022-05-13T23:03:05.472 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-jinja2-2.10.1-2.el8_0.noarch 33/134 2022-05-13T23:03:05.544 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-9.el8.x86_64 33/134 2022-05-13T23:03:05.927 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-mako-1.0.6-13.el8.noarch 34/134 2022-05-13T23:03:05.935 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-scipy-1.0.0-20.module+el8.1.0+3334+5cb62 34/134 2022-05-13T23:03:06.174 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-rsa-4.8-1.el8.noarch 35/134 2022-05-13T23:03:06.188 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-rsa-4.8-1.el8.noarch 35/134 2022-05-13T23:03:06.350 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 36/134 2022-05-13T23:03:06.380 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 36/134 2022-05-13T23:03:06.515 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 86/134 2022-05-13T23:03:06.600 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-rgw-2:17.2.0-188.g23b06583.el8.x86_64 37/134 2022-05-13T23:03:06.612 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-rgw-2:17.2.0-188.g23b06583.el8.x86_64 37/134 2022-05-13T23:03:06.775 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-rbd-2:17.2.0-188.g23b06583.el8.x86_64 38/134 2022-05-13T23:03:06.843 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 87/134 2022-05-13T23:03:06.849 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-rbd-2:17.2.0-188.g23b06583.el8.x86_64 38/134 2022-05-13T23:03:06.957 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : librados-devel-2:17.2.0-188.g23b06583.el8.x86_64 39/134 2022-05-13T23:03:07.103 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : librados-devel-2:17.2.0-188.g23b06583.el8.x86_64 39/134 2022-05-13T23:03:07.113 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 88/134 2022-05-13T23:03:07.120 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : libcom_err-devel-1.45.6-1.el8.x86_64 40/134 2022-05-13T23:03:07.283 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : keyutils-libs-devel-1.5.10-6.el8.x86_64 41/134 2022-05-13T23:03:07.354 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : socat-1.7.3.3-2.el8.x86_64 40/134 2022-05-13T23:03:07.463 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 89/134 2022-05-13T23:03:07.497 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : pcre2-utf32-10.32-2.el8.x86_64 42/134 2022-05-13T23:03:07.540 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 41/134 2022-05-13T23:03:07.675 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : oniguruma-6.8.2-2.el8.x86_64 42/134 2022-05-13T23:03:07.729 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 90/134 2022-05-13T23:03:07.735 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 43/134 2022-05-13T23:03:07.793 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: oniguruma-6.8.2-2.el8.x86_64 42/134 2022-05-13T23:03:07.928 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 91/134 2022-05-13T23:03:07.998 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : jq-1.5-12.el8.x86_64 43/134 2022-05-13T23:03:08.153 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 44/134 2022-05-13T23:03:08.215 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 92/134 2022-05-13T23:03:08.238 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-cffi-1.11.5-5.el8.x86_64 44/134 2022-05-13T23:03:08.450 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 45/134 2022-05-13T23:03:08.580 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-cryptography-3.2.1-4.el8.x86_64 45/134 2022-05-13T23:03:08.672 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : automake-1.16.1-7.el8.noarch 46/134 2022-05-13T23:03:08.797 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 46/134 2022-05-13T23:03:08.886 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : libcom_err-devel-1.45.6-1.el8.x86_64 47/134 2022-05-13T23:03:09.115 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : keyutils-libs-devel-1.5.10-6.el8.x86_64 48/134 2022-05-13T23:03:09.126 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 47/134 2022-05-13T23:03:09.342 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : pcre2-utf32-10.32-2.el8.x86_64 49/134 2022-05-13T23:03:09.384 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 48/134 2022-05-13T23:03:09.593 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 50/134 2022-05-13T23:03:09.778 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 49/134 2022-05-13T23:03:10.029 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-asyncssh-2.7.0-2.el8.noarch 50/134 2022-05-13T23:03:10.045 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-cffi-1.11.5-5.el8.x86_64 51/134 2022-05-13T23:03:10.215 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 51/134 2022-05-13T23:03:10.348 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-cryptography-3.2.1-4.el8.x86_64 52/134 2022-05-13T23:03:10.464 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 52/134 2022-05-13T23:03:10.623 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 53/134 2022-05-13T23:03:10.645 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : fuse-2.9.7-12.el8.x86_64 53/134 2022-05-13T23:03:10.855 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : libuuid-devel-2.32.1-27.el8.x86_64 54/134 2022-05-13T23:03:10.902 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 54/134 2022-05-13T23:03:11.054 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : libsepol-devel-2.9-2.el8.x86_64 55/134 2022-05-13T23:03:11.194 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 55/134 2022-05-13T23:03:11.210 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : libverto-devel-0.3.0-5.el8.x86_64 56/134 2022-05-13T23:03:11.532 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 56/134 2022-05-13T23:03:11.557 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : pcre2-utf16-10.32-2.el8.x86_64 57/134 2022-05-13T23:03:11.784 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-asyncssh-2.7.0-2.el8.noarch 57/134 2022-05-13T23:03:11.920 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : pcre2-devel-10.32-2.el8.x86_64 58/134 2022-05-13T23:03:11.994 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 58/134 2022-05-13T23:03:12.179 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : libselinux-devel-2.9-5.el8.x86_64 59/134 2022-05-13T23:03:12.278 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 59/134 2022-05-13T23:03:12.476 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : fuse-2.9.7-12.el8.x86_64 60/134 2022-05-13T23:03:12.515 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : krb5-devel-1.18.2-8.3.el8_4.x86_64 60/134 2022-05-13T23:03:12.665 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : libuuid-devel-2.32.1-27.el8.x86_64 61/134 2022-05-13T23:03:12.727 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : socat-1.7.3.3-2.el8.x86_64 61/134 2022-05-13T23:03:12.878 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : libsepol-devel-2.9-2.el8.x86_64 62/134 2022-05-13T23:03:12.977 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 62/134 2022-05-13T23:03:13.073 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : libverto-devel-0.3.0-5.el8.x86_64 63/134 2022-05-13T23:03:13.116 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : oniguruma-6.8.2-2.el8.x86_64 63/134 2022-05-13T23:03:13.147 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: ceph-common-2:17.2.0-188.g23b06583.el8.x86_64 93/134 2022-05-13T23:03:13.261 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: oniguruma-6.8.2-2.el8.x86_64 63/134 2022-05-13T23:03:13.370 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : ceph-common-2:17.2.0-188.g23b06583.el8.x86_64 93/134 2022-05-13T23:03:13.436 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : pcre2-utf16-10.32-2.el8.x86_64 64/134 2022-05-13T23:03:13.476 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : jq-1.5-12.el8.x86_64 64/134 2022-05-13T23:03:13.629 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 65/134 2022-05-13T23:03:13.785 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : pcre2-devel-10.32-2.el8.x86_64 65/134 2022-05-13T23:03:13.938 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: ceph-common-2:17.2.0-188.g23b06583.el8.x86_64 93/134 2022-05-13T23:03:13.941 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 66/134 2022-05-13T23:03:14.062 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : libselinux-devel-2.9-5.el8.x86_64 66/134 2022-05-13T23:03:14.189 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : automake-1.16.1-7.el8.noarch 67/134 2022-05-13T23:03:14.335 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : krb5-devel-1.18.2-8.3.el8_4.x86_64 67/134 2022-05-13T23:03:14.395 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : xmlstarlet-1.6.1-11.el8.x86_64 68/134 2022-05-13T23:03:14.592 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : xmlstarlet-1.6.1-11.el8.x86_64 68/134 2022-05-13T23:03:14.614 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 69/134 2022-05-13T23:03:14.783 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 69/134 2022-05-13T23:03:14.805 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : ceph-base-2:17.2.0-188.g23b06583.el8.x86_64 94/134 2022-05-13T23:03:14.915 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 70/134 2022-05-13T23:03:14.985 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: ceph-base-2:17.2.0-188.g23b06583.el8.x86_64 94/134 2022-05-13T23:03:15.082 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 70/134 2022-05-13T23:03:15.102 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : ceph-selinux-2:17.2.0-188.g23b06583.el8.x86_64 95/134 2022-05-13T23:03:15.134 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-waitress-1.2.1-2.el8.1.noarch 71/134 2022-05-13T23:03:15.283 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-waitress-1.2.1-2.el8.1.noarch 71/134 2022-05-13T23:03:15.289 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 72/134 2022-05-13T23:03:15.455 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 72/134 2022-05-13T23:03:15.475 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 73/134 2022-05-13T23:03:15.614 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 73/134 2022-05-13T23:03:15.700 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 74/134 2022-05-13T23:03:15.844 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 74/134 2022-05-13T23:03:15.947 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 75/134 2022-05-13T23:03:16.076 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 75/134 2022-05-13T23:03:16.146 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-natsort-7.1.1-2.el8.noarch 76/134 2022-05-13T23:03:16.292 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-natsort-7.1.1-2.el8.noarch 76/134 2022-05-13T23:03:16.329 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 77/134 2022-05-13T23:03:16.503 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 77/134 2022-05-13T23:03:16.517 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 78/134 2022-05-13T23:03:16.707 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 78/134 2022-05-13T23:03:16.762 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 79/134 2022-05-13T23:03:16.959 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 79/134 2022-05-13T23:03:17.027 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 80/134 2022-05-13T23:03:17.190 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 80/134 2022-05-13T23:03:17.200 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 81/134 2022-05-13T23:03:17.382 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 81/134 2022-05-13T23:03:17.472 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-portend-2.6-1.el8.noarch 82/134 2022-05-13T23:03:17.622 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-portend-2.6-1.el8.noarch 82/134 2022-05-13T23:03:17.702 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 83/134 2022-05-13T23:03:17.867 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 83/134 2022-05-13T23:03:17.881 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 84/134 2022-05-13T23:03:18.036 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 84/134 2022-05-13T23:03:18.129 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 85/134 2022-05-13T23:03:18.294 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 85/134 2022-05-13T23:03:19.511 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 86/134 2022-05-13T23:03:19.743 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 86/134 2022-05-13T23:03:20.022 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 87/134 2022-05-13T23:03:20.095 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 87/134 2022-05-13T23:03:20.268 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 88/134 2022-05-13T23:03:20.328 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 88/134 2022-05-13T23:03:20.693 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 89/134 2022-05-13T23:03:20.984 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 90/134 2022-05-13T23:03:21.210 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 89/134 2022-05-13T23:03:21.211 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 91/134 2022-05-13T23:03:21.466 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 92/134 2022-05-13T23:03:21.512 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 90/134 2022-05-13T23:03:21.751 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 91/134 2022-05-13T23:03:21.940 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 92/134 2022-05-13T23:03:26.512 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: ceph-common-2:17.2.0-188.g23b06583.el8.x86_64 93/134 2022-05-13T23:03:26.715 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : ceph-common-2:17.2.0-188.g23b06583.el8.x86_64 93/134 2022-05-13T23:03:27.322 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: ceph-common-2:17.2.0-188.g23b06583.el8.x86_64 93/134 2022-05-13T23:03:27.607 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : ceph-base-2:17.2.0-188.g23b06583.el8.x86_64 94/134 2022-05-13T23:03:27.702 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: ceph-base-2:17.2.0-188.g23b06583.el8.x86_64 94/134 2022-05-13T23:03:27.866 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : ceph-selinux-2:17.2.0-188.g23b06583.el8.x86_64 95/134 2022-05-13T23:03:29.598 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: ceph-common-2:17.2.0-188.g23b06583.el8.x86_64 93/134 2022-05-13T23:03:29.806 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : ceph-common-2:17.2.0-188.g23b06583.el8.x86_64 93/134 2022-05-13T23:03:30.602 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: ceph-common-2:17.2.0-188.g23b06583.el8.x86_64 93/134 2022-05-13T23:03:31.023 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : ceph-base-2:17.2.0-188.g23b06583.el8.x86_64 94/134 2022-05-13T23:03:31.722 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: ceph-base-2:17.2.0-188.g23b06583.el8.x86_64 94/134 2022-05-13T23:03:31.971 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : ceph-selinux-2:17.2.0-188.g23b06583.el8.x86_64 95/134 2022-05-13T23:03:33.003 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: ceph-selinux-2:17.2.0-188.g23b06583.el8.x86_64 95/134 2022-05-13T23:03:33.004 INFO:teuthology.orchestra.run.smithi115.stdout:skipping the directory /sys 2022-05-13T23:03:33.004 INFO:teuthology.orchestra.run.smithi115.stdout:skipping the directory /proc 2022-05-13T23:03:33.005 INFO:teuthology.orchestra.run.smithi115.stdout:skipping the directory /dev 2022-05-13T23:03:33.005 INFO:teuthology.orchestra.run.smithi115.stdout:skipping the directory /run 2022-05-13T23:03:33.005 INFO:teuthology.orchestra.run.smithi115.stdout:skipping the directory /mnt 2022-05-13T23:03:33.017 INFO:teuthology.orchestra.run.smithi115.stdout:skipping the directory /var/tmp 2022-05-13T23:03:33.018 INFO:teuthology.orchestra.run.smithi115.stdout:skipping the directory /home 2022-05-13T23:03:33.018 INFO:teuthology.orchestra.run.smithi115.stdout:skipping the directory /tmp 2022-05-13T23:03:33.018 INFO:teuthology.orchestra.run.smithi115.stdout:skipping the directory /dev 2022-05-13T23:03:33.018 INFO:teuthology.orchestra.run.smithi115.stdout: 2022-05-13T23:03:33.348 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : ceph-volume-2:17.2.0-188.g23b06583.el8.noarch 96/134 2022-05-13T23:03:34.830 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: ceph-volume-2:17.2.0-188.g23b06583.el8.noarch 96/134 2022-05-13T23:03:35.024 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : ceph-osd-2:17.2.0-188.g23b06583.el8.x86_64 97/134 2022-05-13T23:03:35.248 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: ceph-osd-2:17.2.0-188.g23b06583.el8.x86_64 97/134 2022-05-13T23:03:35.431 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : ceph-mds-2:17.2.0-188.g23b06583.el8.x86_64 98/134 2022-05-13T23:03:35.819 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: ceph-mds-2:17.2.0-188.g23b06583.el8.x86_64 98/134 2022-05-13T23:03:36.022 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : ceph-mon-2:17.2.0-188.g23b06583.el8.x86_64 99/134 2022-05-13T23:03:36.041 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: ceph-mon-2:17.2.0-188.g23b06583.el8.x86_64 99/134 2022-05-13T23:03:38.447 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: cephadm-2:17.2.0-188.g23b06583.el8.noarch 100/134 2022-05-13T23:03:38.743 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : cephadm-2:17.2.0-188.g23b06583.el8.noarch 100/134 2022-05-13T23:03:38.904 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : ceph-prometheus-alerts-2:17.2.0-188.g23b06583.el 101/134 2022-05-13T23:03:39.100 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : ceph-grafana-dashboards-2:17.2.0-188.g23b06583.e 102/134 2022-05-13T23:03:39.297 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : ceph-mgr-cephadm-2:17.2.0-188.g23b06583.el8.noar 103/134 2022-05-13T23:03:39.503 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: ceph-mgr-cephadm-2:17.2.0-188.g23b06583.el8.noar 103/134 2022-05-13T23:03:39.741 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : ceph-mgr-dashboard-2:17.2.0-188.g23b06583.el8.no 104/134 2022-05-13T23:03:40.705 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: ceph-mgr-dashboard-2:17.2.0-188.g23b06583.el8.no 104/134 2022-05-13T23:03:40.913 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : ceph-mgr-diskprediction-local-2:17.2.0-188.g23b0 105/134 2022-05-13T23:03:41.012 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:17.2.0-188.g23b0 105/134 2022-05-13T23:03:41.384 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : ceph-mgr-modules-core-2:17.2.0-188.g23b06583.el8 106/134 2022-05-13T23:03:41.548 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : ceph-mgr-2:17.2.0-188.g23b06583.el8.x86_64 107/134 2022-05-13T23:03:41.585 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: ceph-mgr-2:17.2.0-188.g23b06583.el8.x86_64 107/134 2022-05-13T23:03:41.771 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : ceph-mgr-rook-2:17.2.0-188.g23b06583.el8.noarch 108/134 2022-05-13T23:03:41.783 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: ceph-mgr-rook-2:17.2.0-188.g23b06583.el8.noarch 108/134 2022-05-13T23:03:43.304 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : ceph-2:17.2.0-188.g23b06583.el8.x86_64 109/134 2022-05-13T23:03:43.522 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : ceph-radosgw-2:17.2.0-188.g23b06583.el8.x86_64 110/134 2022-05-13T23:03:43.603 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: ceph-radosgw-2:17.2.0-188.g23b06583.el8.x86_64 110/134 2022-05-13T23:03:43.804 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : ceph-immutable-object-cache-2:17.2.0-188.g23b065 111/134 2022-05-13T23:03:43.850 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: ceph-immutable-object-cache-2:17.2.0-188.g23b065 111/134 2022-05-13T23:03:44.062 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : cephfs-mirror-2:17.2.0-188.g23b06583.el8.x86_64 112/134 2022-05-13T23:03:44.450 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: cephfs-mirror-2:17.2.0-188.g23b06583.el8.x86_64 112/134 2022-05-13T23:03:44.646 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : rbd-mirror-2:17.2.0-188.g23b06583.el8.x86_64 113/134 2022-05-13T23:03:46.267 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: ceph-selinux-2:17.2.0-188.g23b06583.el8.x86_64 95/134 2022-05-13T23:03:46.267 INFO:teuthology.orchestra.run.smithi047.stdout:skipping the directory /sys 2022-05-13T23:03:46.268 INFO:teuthology.orchestra.run.smithi047.stdout:skipping the directory /proc 2022-05-13T23:03:46.268 INFO:teuthology.orchestra.run.smithi047.stdout:skipping the directory /dev 2022-05-13T23:03:46.268 INFO:teuthology.orchestra.run.smithi047.stdout:skipping the directory /run 2022-05-13T23:03:46.269 INFO:teuthology.orchestra.run.smithi047.stdout:skipping the directory /mnt 2022-05-13T23:03:46.269 INFO:teuthology.orchestra.run.smithi047.stdout:skipping the directory /var/tmp 2022-05-13T23:03:46.269 INFO:teuthology.orchestra.run.smithi047.stdout:skipping the directory /home 2022-05-13T23:03:46.269 INFO:teuthology.orchestra.run.smithi047.stdout:skipping the directory /tmp 2022-05-13T23:03:46.270 INFO:teuthology.orchestra.run.smithi047.stdout:skipping the directory /dev 2022-05-13T23:03:46.270 INFO:teuthology.orchestra.run.smithi047.stdout: 2022-05-13T23:03:46.646 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : ceph-volume-2:17.2.0-188.g23b06583.el8.noarch 96/134 2022-05-13T23:03:48.234 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: ceph-volume-2:17.2.0-188.g23b06583.el8.noarch 96/134 2022-05-13T23:03:48.420 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : ceph-osd-2:17.2.0-188.g23b06583.el8.x86_64 97/134 2022-05-13T23:03:48.658 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: ceph-osd-2:17.2.0-188.g23b06583.el8.x86_64 97/134 2022-05-13T23:03:48.820 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : ceph-mds-2:17.2.0-188.g23b06583.el8.x86_64 98/134 2022-05-13T23:03:49.233 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: ceph-mds-2:17.2.0-188.g23b06583.el8.x86_64 98/134 2022-05-13T23:03:49.387 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : ceph-mon-2:17.2.0-188.g23b06583.el8.x86_64 99/134 2022-05-13T23:03:49.406 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: ceph-mon-2:17.2.0-188.g23b06583.el8.x86_64 99/134 2022-05-13T23:03:49.554 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: rbd-mirror-2:17.2.0-188.g23b06583.el8.x86_64 113/134 2022-05-13T23:03:50.493 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : ceph-test-2:17.2.0-188.g23b06583.el8.x86_64 114/134 2022-05-13T23:03:50.772 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : openssl-devel-1:1.1.1g-15.el8_3.x86_64 115/134 2022-05-13T23:03:51.070 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : xfsprogs-devel-5.0.0-8.el8.x86_64 116/134 2022-05-13T23:03:51.355 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : ceph-fuse-2:17.2.0-188.g23b06583.el8.x86_64 117/134 2022-05-13T23:03:51.560 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : libtool-2.4.6-25.el8.x86_64 118/134 2022-05-13T23:03:51.572 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: libtool-2.4.6-25.el8.x86_64 118/134 2022-05-13T23:03:51.723 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : libcephfs-devel-2:17.2.0-188.g23b06583.el8.x86_6 119/134 2022-05-13T23:03:51.905 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : elfutils-libelf-devel-0.182-3.el8.x86_64 120/134 2022-05-13T23:03:52.044 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: cephadm-2:17.2.0-188.g23b06583.el8.noarch 100/134 2022-05-13T23:03:52.064 INFO:teuthology.orchestra.run.smithi115.stdout: Upgrading : NetworkManager-team-1:1.30.0-13.el8_4.x86_64 121/134 2022-05-13T23:03:52.265 INFO:teuthology.orchestra.run.smithi115.stdout: Upgrading : NetworkManager-tui-1:1.30.0-13.el8_4.x86_64 122/134 2022-05-13T23:03:52.314 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : cephadm-2:17.2.0-188.g23b06583.el8.noarch 100/134 2022-05-13T23:03:52.466 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : cephfs-top-2:17.2.0-188.g23b06583.el8.noarch 123/134 2022-05-13T23:03:52.484 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : ceph-prometheus-alerts-2:17.2.0-188.g23b06583.el 101/134 2022-05-13T23:03:52.625 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : ceph-grafana-dashboards-2:17.2.0-188.g23b06583.e 102/134 2022-05-13T23:03:52.627 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : rbd-fuse-2:17.2.0-188.g23b06583.el8.x86_64 124/134 2022-05-13T23:03:52.811 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : ceph-mgr-cephadm-2:17.2.0-188.g23b06583.el8.noar 103/134 2022-05-13T23:03:52.919 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : rbd-nbd-2:17.2.0-188.g23b06583.el8.x86_64 125/134 2022-05-13T23:03:53.030 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: ceph-mgr-cephadm-2:17.2.0-188.g23b06583.el8.noar 103/134 2022-05-13T23:03:53.135 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : bison-3.0.4-10.el8.x86_64 126/134 2022-05-13T23:03:53.188 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: bison-3.0.4-10.el8.x86_64 126/134 2022-05-13T23:03:53.277 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : ceph-mgr-dashboard-2:17.2.0-188.g23b06583.el8.no 104/134 2022-05-13T23:03:53.344 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: ceph-selinux-2:17.2.0-188.g23b06583.el8.x86_64 95/134 2022-05-13T23:03:53.345 INFO:teuthology.orchestra.run.smithi002.stdout:skipping the directory /sys 2022-05-13T23:03:53.345 INFO:teuthology.orchestra.run.smithi002.stdout:skipping the directory /proc 2022-05-13T23:03:53.345 INFO:teuthology.orchestra.run.smithi002.stdout:skipping the directory /dev 2022-05-13T23:03:53.345 INFO:teuthology.orchestra.run.smithi002.stdout:skipping the directory /run 2022-05-13T23:03:53.346 INFO:teuthology.orchestra.run.smithi002.stdout:skipping the directory /mnt 2022-05-13T23:03:53.346 INFO:teuthology.orchestra.run.smithi002.stdout:skipping the directory /var/tmp 2022-05-13T23:03:53.346 INFO:teuthology.orchestra.run.smithi002.stdout:skipping the directory /home 2022-05-13T23:03:53.346 INFO:teuthology.orchestra.run.smithi002.stdout:skipping the directory /tmp 2022-05-13T23:03:53.347 INFO:teuthology.orchestra.run.smithi002.stdout:skipping the directory /dev 2022-05-13T23:03:53.347 INFO:teuthology.orchestra.run.smithi002.stdout: 2022-05-13T23:03:53.394 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : flex-2.6.1-9.el8.x86_64 127/134 2022-05-13T23:03:53.403 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: flex-2.6.1-9.el8.x86_64 127/134 2022-05-13T23:03:53.570 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : libaio-devel-0.3.112-1.el8.x86_64 128/134 2022-05-13T23:03:53.721 INFO:teuthology.orchestra.run.smithi115.stdout: Cleanup : NetworkManager-tui-1:1.30.0-7.el8.x86_64 129/134 2022-05-13T23:03:53.721 INFO:teuthology.orchestra.run.smithi115.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 130/134 2022-05-13T23:03:53.780 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : ceph-volume-2:17.2.0-188.g23b06583.el8.noarch 96/134 2022-05-13T23:03:53.954 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 130/134 2022-05-13T23:03:54.130 INFO:teuthology.orchestra.run.smithi115.stdout: Cleanup : NetworkManager-team-1:1.30.0-7.el8.x86_64 131/134 2022-05-13T23:03:54.139 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: NetworkManager-1:1.30.0-7.el8.x86_64 132/134 2022-05-13T23:03:54.139 INFO:teuthology.orchestra.run.smithi115.stdout: Cleanup : NetworkManager-1:1.30.0-7.el8.x86_64 132/134 2022-05-13T23:03:54.309 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: ceph-mgr-dashboard-2:17.2.0-188.g23b06583.el8.no 104/134 2022-05-13T23:03:54.469 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : ceph-mgr-diskprediction-local-2:17.2.0-188.g23b0 105/134 2022-05-13T23:03:54.581 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:17.2.0-188.g23b0 105/134 2022-05-13T23:03:54.588 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: NetworkManager-1:1.30.0-7.el8.x86_64 132/134 2022-05-13T23:03:54.588 INFO:teuthology.orchestra.run.smithi115.stdout: Cleanup : NetworkManager-libnm-1:1.30.0-7.el8.x86_64 133/134 2022-05-13T23:03:54.838 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: NetworkManager-libnm-1:1.30.0-7.el8.x86_64 133/134 2022-05-13T23:03:54.838 INFO:teuthology.orchestra.run.smithi115.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 134/134 2022-05-13T23:03:54.992 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : ceph-mgr-modules-core-2:17.2.0-188.g23b06583.el8 106/134 2022-05-13T23:03:55.258 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : ceph-mgr-2:17.2.0-188.g23b06583.el8.x86_64 107/134 2022-05-13T23:03:55.258 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: ceph-mgr-2:17.2.0-188.g23b06583.el8.x86_64 107/134 2022-05-13T23:03:55.403 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : ceph-mgr-rook-2:17.2.0-188.g23b06583.el8.noarch 108/134 2022-05-13T23:03:55.419 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: ceph-mgr-rook-2:17.2.0-188.g23b06583.el8.noarch 108/134 2022-05-13T23:03:55.648 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: ceph-volume-2:17.2.0-188.g23b06583.el8.noarch 96/134 2022-05-13T23:03:55.783 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 134/134 2022-05-13T23:03:55.783 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : ceph-2:17.2.0-188.g23b06583.el8.x86_64 1/134 2022-05-13T23:03:55.783 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : ceph-base-2:17.2.0-188.g23b06583.el8.x86_64 2/134 2022-05-13T23:03:55.784 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : ceph-common-2:17.2.0-188.g23b06583.el8.x86_64 3/134 2022-05-13T23:03:55.784 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : ceph-fuse-2:17.2.0-188.g23b06583.el8.x86_64 4/134 2022-05-13T23:03:55.784 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : ceph-immutable-object-cache-2:17.2.0-188.g23b065 5/134 2022-05-13T23:03:55.784 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : ceph-mds-2:17.2.0-188.g23b06583.el8.x86_64 6/134 2022-05-13T23:03:55.785 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : ceph-mgr-2:17.2.0-188.g23b06583.el8.x86_64 7/134 2022-05-13T23:03:55.785 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : ceph-mon-2:17.2.0-188.g23b06583.el8.x86_64 8/134 2022-05-13T23:03:55.785 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : ceph-osd-2:17.2.0-188.g23b06583.el8.x86_64 9/134 2022-05-13T23:03:55.785 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : ceph-radosgw-2:17.2.0-188.g23b06583.el8.x86_64 10/134 2022-05-13T23:03:55.785 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : ceph-selinux-2:17.2.0-188.g23b06583.el8.x86_64 11/134 2022-05-13T23:03:55.786 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : ceph-test-2:17.2.0-188.g23b06583.el8.x86_64 12/134 2022-05-13T23:03:55.786 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : cephfs-mirror-2:17.2.0-188.g23b06583.el8.x86_64 13/134 2022-05-13T23:03:55.786 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : libcephfs-devel-2:17.2.0-188.g23b06583.el8.x86_6 14/134 2022-05-13T23:03:55.786 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : libcephfs2-2:17.2.0-188.g23b06583.el8.x86_64 15/134 2022-05-13T23:03:55.787 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : libcephsqlite-2:17.2.0-188.g23b06583.el8.x86_64 16/134 2022-05-13T23:03:55.787 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : librados-devel-2:17.2.0-188.g23b06583.el8.x86_64 17/134 2022-05-13T23:03:55.787 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : libradosstriper1-2:17.2.0-188.g23b06583.el8.x86_ 18/134 2022-05-13T23:03:55.787 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : librgw2-2:17.2.0-188.g23b06583.el8.x86_64 19/134 2022-05-13T23:03:55.787 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-ceph-argparse-2:17.2.0-188.g23b06583.el8 20/134 2022-05-13T23:03:55.788 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-ceph-common-2:17.2.0-188.g23b06583.el8.x 21/134 2022-05-13T23:03:55.788 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-cephfs-2:17.2.0-188.g23b06583.el8.x86_64 22/134 2022-05-13T23:03:55.788 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-rados-2:17.2.0-188.g23b06583.el8.x86_64 23/134 2022-05-13T23:03:55.788 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-rbd-2:17.2.0-188.g23b06583.el8.x86_64 24/134 2022-05-13T23:03:55.788 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-rgw-2:17.2.0-188.g23b06583.el8.x86_64 25/134 2022-05-13T23:03:55.789 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : rbd-fuse-2:17.2.0-188.g23b06583.el8.x86_64 26/134 2022-05-13T23:03:55.790 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : rbd-mirror-2:17.2.0-188.g23b06583.el8.x86_64 27/134 2022-05-13T23:03:55.790 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : rbd-nbd-2:17.2.0-188.g23b06583.el8.x86_64 28/134 2022-05-13T23:03:55.790 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : ceph-grafana-dashboards-2:17.2.0-188.g23b06583.e 29/134 2022-05-13T23:03:55.791 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : ceph-mgr-cephadm-2:17.2.0-188.g23b06583.el8.noar 30/134 2022-05-13T23:03:55.791 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : ceph-mgr-dashboard-2:17.2.0-188.g23b06583.el8.no 31/134 2022-05-13T23:03:55.791 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : ceph-mgr-diskprediction-local-2:17.2.0-188.g23b0 32/134 2022-05-13T23:03:55.791 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : ceph-mgr-modules-core-2:17.2.0-188.g23b06583.el8 33/134 2022-05-13T23:03:55.791 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : ceph-mgr-rook-2:17.2.0-188.g23b06583.el8.noarch 34/134 2022-05-13T23:03:55.792 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : ceph-prometheus-alerts-2:17.2.0-188.g23b06583.el 35/134 2022-05-13T23:03:55.792 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : ceph-volume-2:17.2.0-188.g23b06583.el8.noarch 36/134 2022-05-13T23:03:55.792 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : cephadm-2:17.2.0-188.g23b06583.el8.noarch 37/134 2022-05-13T23:03:55.792 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : cephfs-top-2:17.2.0-188.g23b06583.el8.noarch 38/134 2022-05-13T23:03:55.793 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 39/134 2022-05-13T23:03:55.793 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 40/134 2022-05-13T23:03:55.793 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 41/134 2022-05-13T23:03:55.794 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-asyncssh-2.7.0-2.el8.noarch 42/134 2022-05-13T23:03:55.794 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 43/134 2022-05-13T23:03:55.794 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 44/134 2022-05-13T23:03:55.794 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 45/134 2022-05-13T23:03:55.795 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 46/134 2022-05-13T23:03:55.795 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 47/134 2022-05-13T23:03:55.795 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 48/134 2022-05-13T23:03:55.795 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 49/134 2022-05-13T23:03:55.796 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 50/134 2022-05-13T23:03:55.796 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 51/134 2022-05-13T23:03:55.796 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 52/134 2022-05-13T23:03:55.796 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 53/134 2022-05-13T23:03:55.797 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 54/134 2022-05-13T23:03:55.797 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-natsort-7.1.1-2.el8.noarch 55/134 2022-05-13T23:03:55.797 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 56/134 2022-05-13T23:03:55.797 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-portend-2.6-1.el8.noarch 57/134 2022-05-13T23:03:55.797 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 58/134 2022-05-13T23:03:55.798 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 59/134 2022-05-13T23:03:55.798 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-rsa-4.8-1.el8.noarch 60/134 2022-05-13T23:03:55.798 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 61/134 2022-05-13T23:03:55.798 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 62/134 2022-05-13T23:03:55.799 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 63/134 2022-05-13T23:03:55.799 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 64/134 2022-05-13T23:03:55.799 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-waitress-1.2.1-2.el8.1.noarch 65/134 2022-05-13T23:03:55.799 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 66/134 2022-05-13T23:03:55.799 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 67/134 2022-05-13T23:03:55.800 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 68/134 2022-05-13T23:03:55.800 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 69/134 2022-05-13T23:03:55.800 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : thrift-0.13.0-2.el8.x86_64 70/134 2022-05-13T23:03:55.800 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : xmlstarlet-1.6.1-11.el8.x86_64 71/134 2022-05-13T23:03:55.800 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 72/134 2022-05-13T23:03:55.801 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : libaio-devel-0.3.112-1.el8.x86_64 73/134 2022-05-13T23:03:55.801 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 74/134 2022-05-13T23:03:55.801 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : libquadmath-8.4.1-1.el8.x86_64 75/134 2022-05-13T23:03:55.801 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-cffi-1.11.5-5.el8.x86_64 76/134 2022-05-13T23:03:55.802 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : libgfortran-8.4.1-1.el8.x86_64 77/134 2022-05-13T23:03:55.802 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : elfutils-libelf-devel-0.182-3.el8.x86_64 78/134 2022-05-13T23:03:55.802 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : libselinux-devel-2.9-5.el8.x86_64 79/134 2022-05-13T23:03:55.802 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : zlib-devel-1.2.11-17.el8.x86_64 80/134 2022-05-13T23:03:55.802 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : pcre2-utf16-10.32-2.el8.x86_64 81/134 2022-05-13T23:03:55.803 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : librabbitmq-0.9.0-3.el8.x86_64 82/134 2022-05-13T23:03:55.803 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : libverto-devel-0.3.0-5.el8.x86_64 83/134 2022-05-13T23:03:55.803 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : libsepol-devel-2.9-2.el8.x86_64 84/134 2022-05-13T23:03:55.803 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : libuuid-devel-2.32.1-27.el8.x86_64 85/134 2022-05-13T23:03:55.804 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 86/134 2022-05-13T23:03:55.804 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : openssl-devel-1:1.1.1g-15.el8_3.x86_64 87/134 2022-05-13T23:03:55.804 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : fuse-2.9.7-12.el8.x86_64 88/134 2022-05-13T23:03:55.804 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 89/134 2022-05-13T23:03:55.804 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : pcre2-devel-10.32-2.el8.x86_64 90/134 2022-05-13T23:03:55.805 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-cryptography-3.2.1-4.el8.x86_64 91/134 2022-05-13T23:03:55.805 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : xfsprogs-devel-5.0.0-8.el8.x86_64 92/134 2022-05-13T23:03:55.805 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : pcre2-utf32-10.32-2.el8.x86_64 93/134 2022-05-13T23:03:55.805 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : keyutils-libs-devel-1.5.10-6.el8.x86_64 94/134 2022-05-13T23:03:55.806 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : libcom_err-devel-1.45.6-1.el8.x86_64 95/134 2022-05-13T23:03:55.806 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : krb5-devel-1.18.2-8.3.el8_4.x86_64 96/134 2022-05-13T23:03:55.806 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-mako-1.0.6-13.el8.noarch 97/134 2022-05-13T23:03:55.806 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : flex-2.6.1-9.el8.x86_64 98/134 2022-05-13T23:03:55.806 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : jq-1.5-12.el8.x86_64 99/134 2022-05-13T23:03:55.807 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 100/134 2022-05-13T23:03:55.807 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : librdkafka-0.11.4-1.el8.x86_64 101/134 2022-05-13T23:03:55.807 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-jinja2-2.10.1-2.el8_0.noarch 102/134 2022-05-13T23:03:55.807 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : automake-1.16.1-7.el8.noarch 103/134 2022-05-13T23:03:55.808 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : libtool-2.4.6-25.el8.x86_64 104/134 2022-05-13T23:03:55.808 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-numpy-1:1.14.3-9.el8.x86_64 105/134 2022-05-13T23:03:55.808 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 106/134 2022-05-13T23:03:55.808 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 107/134 2022-05-13T23:03:55.808 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-pytz-2017.2-9.el8.noarch 108/134 2022-05-13T23:03:55.809 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : oniguruma-6.8.2-2.el8.x86_64 109/134 2022-05-13T23:03:55.809 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 110/134 2022-05-13T23:03:55.809 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-scipy-1.0.0-20.module+el8.1.0+3334+5cb62 111/134 2022-05-13T23:03:55.809 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 112/134 2022-05-13T23:03:55.809 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 113/134 2022-05-13T23:03:55.810 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : openblas-threads-0.3.12-1.el8.x86_64 114/134 2022-05-13T23:03:55.810 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-babel-2.5.1-5.el8.noarch 115/134 2022-05-13T23:03:55.810 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 116/134 2022-05-13T23:03:55.810 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 117/134 2022-05-13T23:03:55.811 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 118/134 2022-05-13T23:03:55.811 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : socat-1.7.3.3-2.el8.x86_64 119/134 2022-05-13T23:03:55.811 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : openblas-0.3.12-1.el8.x86_64 120/134 2022-05-13T23:03:55.811 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : bison-3.0.4-10.el8.x86_64 121/134 2022-05-13T23:03:55.812 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-numpy-f2py-1:1.14.3-9.el8.x86_64 122/134 2022-05-13T23:03:55.812 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : librados2-2:17.2.0-188.g23b06583.el8.x86_64 123/134 2022-05-13T23:03:55.812 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 124/134 2022-05-13T23:03:55.812 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : librbd1-2:17.2.0-188.g23b06583.el8.x86_64 125/134 2022-05-13T23:03:55.813 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 126/134 2022-05-13T23:03:55.813 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : NetworkManager-team-1:1.30.0-13.el8_4.x86_64 127/134 2022-05-13T23:03:55.813 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : NetworkManager-team-1:1.30.0-7.el8.x86_64 128/134 2022-05-13T23:03:55.813 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : NetworkManager-libnm-1:1.30.0-13.el8_4.x86_64 129/134 2022-05-13T23:03:55.813 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : NetworkManager-libnm-1:1.30.0-7.el8.x86_64 130/134 2022-05-13T23:03:55.814 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : NetworkManager-tui-1:1.30.0-13.el8_4.x86_64 131/134 2022-05-13T23:03:55.814 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : NetworkManager-tui-1:1.30.0-7.el8.x86_64 132/134 2022-05-13T23:03:55.814 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : NetworkManager-1:1.30.0-13.el8_4.x86_64 133/134 2022-05-13T23:03:55.827 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : ceph-osd-2:17.2.0-188.g23b06583.el8.x86_64 97/134 2022-05-13T23:03:56.161 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: ceph-osd-2:17.2.0-188.g23b06583.el8.x86_64 97/134 2022-05-13T23:03:56.306 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : ceph-mds-2:17.2.0-188.g23b06583.el8.x86_64 98/134 2022-05-13T23:03:56.848 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: ceph-mds-2:17.2.0-188.g23b06583.el8.x86_64 98/134 2022-05-13T23:03:56.998 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : ceph-mon-2:17.2.0-188.g23b06583.el8.x86_64 99/134 2022-05-13T23:03:57.046 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : ceph-2:17.2.0-188.g23b06583.el8.x86_64 109/134 2022-05-13T23:03:57.066 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: ceph-mon-2:17.2.0-188.g23b06583.el8.x86_64 99/134 2022-05-13T23:03:57.236 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : ceph-radosgw-2:17.2.0-188.g23b06583.el8.x86_64 110/134 2022-05-13T23:03:57.349 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: ceph-radosgw-2:17.2.0-188.g23b06583.el8.x86_64 110/134 2022-05-13T23:03:57.407 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : NetworkManager-1:1.30.0-7.el8.x86_64 134/134 2022-05-13T23:03:57.408 INFO:teuthology.orchestra.run.smithi115.stdout:Installed products updated. 2022-05-13T23:03:57.553 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : ceph-immutable-object-cache-2:17.2.0-188.g23b065 111/134 2022-05-13T23:03:57.603 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: ceph-immutable-object-cache-2:17.2.0-188.g23b065 111/134 2022-05-13T23:03:57.802 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : cephfs-mirror-2:17.2.0-188.g23b06583.el8.x86_64 112/134 2022-05-13T23:03:58.217 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: cephfs-mirror-2:17.2.0-188.g23b06583.el8.x86_64 112/134 2022-05-13T23:03:58.394 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : rbd-mirror-2:17.2.0-188.g23b06583.el8.x86_64 113/134 2022-05-13T23:04:01.885 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: cephadm-2:17.2.0-188.g23b06583.el8.noarch 100/134 2022-05-13T23:04:02.055 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : cephadm-2:17.2.0-188.g23b06583.el8.noarch 100/134 2022-05-13T23:04:02.302 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : ceph-prometheus-alerts-2:17.2.0-188.g23b06583.el 101/134 2022-05-13T23:04:02.481 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : ceph-grafana-dashboards-2:17.2.0-188.g23b06583.e 102/134 2022-05-13T23:04:02.658 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : ceph-mgr-cephadm-2:17.2.0-188.g23b06583.el8.noar 103/134 2022-05-13T23:04:02.967 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: ceph-mgr-cephadm-2:17.2.0-188.g23b06583.el8.noar 103/134 2022-05-13T23:04:03.259 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : ceph-mgr-dashboard-2:17.2.0-188.g23b06583.el8.no 104/134 2022-05-13T23:04:03.655 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: rbd-mirror-2:17.2.0-188.g23b06583.el8.x86_64 113/134 2022-05-13T23:04:04.566 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: ceph-mgr-dashboard-2:17.2.0-188.g23b06583.el8.no 104/134 2022-05-13T23:04:04.679 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : ceph-test-2:17.2.0-188.g23b06583.el8.x86_64 114/134 2022-05-13T23:04:04.802 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : ceph-mgr-diskprediction-local-2:17.2.0-188.g23b0 105/134 2022-05-13T23:04:04.978 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:17.2.0-188.g23b0 105/134 2022-05-13T23:04:04.980 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : openssl-devel-1:1.1.1g-15.el8_3.x86_64 115/134 2022-05-13T23:04:05.300 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : xfsprogs-devel-5.0.0-8.el8.x86_64 116/134 2022-05-13T23:04:05.411 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : ceph-mgr-modules-core-2:17.2.0-188.g23b06583.el8 106/134 2022-05-13T23:04:05.559 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : ceph-mgr-2:17.2.0-188.g23b06583.el8.x86_64 107/134 2022-05-13T23:04:05.560 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : ceph-fuse-2:17.2.0-188.g23b06583.el8.x86_64 117/134 2022-05-13T23:04:05.661 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: ceph-mgr-2:17.2.0-188.g23b06583.el8.x86_64 107/134 2022-05-13T23:04:05.833 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : ceph-mgr-rook-2:17.2.0-188.g23b06583.el8.noarch 108/134 2022-05-13T23:04:05.834 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : libtool-2.4.6-25.el8.x86_64 118/134 2022-05-13T23:04:05.848 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: libtool-2.4.6-25.el8.x86_64 118/134 2022-05-13T23:04:05.887 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: ceph-mgr-rook-2:17.2.0-188.g23b06583.el8.noarch 108/134 2022-05-13T23:04:06.048 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : libcephfs-devel-2:17.2.0-188.g23b06583.el8.x86_6 119/134 2022-05-13T23:04:06.196 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : elfutils-libelf-devel-0.182-3.el8.x86_64 120/134 2022-05-13T23:04:06.382 INFO:teuthology.orchestra.run.smithi047.stdout: Upgrading : NetworkManager-team-1:1.30.0-13.el8_4.x86_64 121/134 2022-05-13T23:04:06.565 INFO:teuthology.orchestra.run.smithi047.stdout: Upgrading : NetworkManager-tui-1:1.30.0-13.el8_4.x86_64 122/134 2022-05-13T23:04:06.711 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : cephfs-top-2:17.2.0-188.g23b06583.el8.noarch 123/134 2022-05-13T23:04:06.919 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : rbd-fuse-2:17.2.0-188.g23b06583.el8.x86_64 124/134 2022-05-13T23:04:07.223 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : rbd-nbd-2:17.2.0-188.g23b06583.el8.x86_64 125/134 2022-05-13T23:04:07.426 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : bison-3.0.4-10.el8.x86_64 126/134 2022-05-13T23:04:07.482 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: bison-3.0.4-10.el8.x86_64 126/134 2022-05-13T23:04:07.676 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : flex-2.6.1-9.el8.x86_64 127/134 2022-05-13T23:04:07.686 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: flex-2.6.1-9.el8.x86_64 127/134 2022-05-13T23:04:07.760 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : ceph-2:17.2.0-188.g23b06583.el8.x86_64 109/134 2022-05-13T23:04:07.844 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : libaio-devel-0.3.112-1.el8.x86_64 128/134 2022-05-13T23:04:07.959 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : ceph-radosgw-2:17.2.0-188.g23b06583.el8.x86_64 110/134 2022-05-13T23:04:08.020 INFO:teuthology.orchestra.run.smithi047.stdout: Cleanup : NetworkManager-tui-1:1.30.0-7.el8.x86_64 129/134 2022-05-13T23:04:08.020 INFO:teuthology.orchestra.run.smithi047.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 130/134 2022-05-13T23:04:08.122 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: ceph-radosgw-2:17.2.0-188.g23b06583.el8.x86_64 110/134 2022-05-13T23:04:08.245 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 130/134 2022-05-13T23:04:08.316 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : ceph-immutable-object-cache-2:17.2.0-188.g23b065 111/134 2022-05-13T23:04:08.386 INFO:teuthology.orchestra.run.smithi047.stdout: Cleanup : NetworkManager-team-1:1.30.0-7.el8.x86_64 131/134 2022-05-13T23:04:08.397 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: NetworkManager-1:1.30.0-7.el8.x86_64 132/134 2022-05-13T23:04:08.397 INFO:teuthology.orchestra.run.smithi047.stdout: Cleanup : NetworkManager-1:1.30.0-7.el8.x86_64 132/134 2022-05-13T23:04:08.430 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: ceph-immutable-object-cache-2:17.2.0-188.g23b065 111/134 2022-05-13T23:04:08.583 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : cephfs-mirror-2:17.2.0-188.g23b06583.el8.x86_64 112/134 2022-05-13T23:04:08.880 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: NetworkManager-1:1.30.0-7.el8.x86_64 132/134 2022-05-13T23:04:08.881 INFO:teuthology.orchestra.run.smithi047.stdout: Cleanup : NetworkManager-libnm-1:1.30.0-7.el8.x86_64 133/134 2022-05-13T23:04:09.122 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: cephfs-mirror-2:17.2.0-188.g23b06583.el8.x86_64 112/134 2022-05-13T23:04:09.129 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: NetworkManager-libnm-1:1.30.0-7.el8.x86_64 133/134 2022-05-13T23:04:09.130 INFO:teuthology.orchestra.run.smithi047.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 134/134 2022-05-13T23:04:09.300 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : rbd-mirror-2:17.2.0-188.g23b06583.el8.x86_64 113/134 2022-05-13T23:04:10.065 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 134/134 2022-05-13T23:04:10.065 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : ceph-2:17.2.0-188.g23b06583.el8.x86_64 1/134 2022-05-13T23:04:10.066 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : ceph-base-2:17.2.0-188.g23b06583.el8.x86_64 2/134 2022-05-13T23:04:10.066 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : ceph-common-2:17.2.0-188.g23b06583.el8.x86_64 3/134 2022-05-13T23:04:10.066 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : ceph-fuse-2:17.2.0-188.g23b06583.el8.x86_64 4/134 2022-05-13T23:04:10.066 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : ceph-immutable-object-cache-2:17.2.0-188.g23b065 5/134 2022-05-13T23:04:10.067 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : ceph-mds-2:17.2.0-188.g23b06583.el8.x86_64 6/134 2022-05-13T23:04:10.067 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : ceph-mgr-2:17.2.0-188.g23b06583.el8.x86_64 7/134 2022-05-13T23:04:10.067 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : ceph-mon-2:17.2.0-188.g23b06583.el8.x86_64 8/134 2022-05-13T23:04:10.067 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : ceph-osd-2:17.2.0-188.g23b06583.el8.x86_64 9/134 2022-05-13T23:04:10.067 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : ceph-radosgw-2:17.2.0-188.g23b06583.el8.x86_64 10/134 2022-05-13T23:04:10.068 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : ceph-selinux-2:17.2.0-188.g23b06583.el8.x86_64 11/134 2022-05-13T23:04:10.068 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : ceph-test-2:17.2.0-188.g23b06583.el8.x86_64 12/134 2022-05-13T23:04:10.068 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : cephfs-mirror-2:17.2.0-188.g23b06583.el8.x86_64 13/134 2022-05-13T23:04:10.068 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : libcephfs-devel-2:17.2.0-188.g23b06583.el8.x86_6 14/134 2022-05-13T23:04:10.068 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : libcephfs2-2:17.2.0-188.g23b06583.el8.x86_64 15/134 2022-05-13T23:04:10.069 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : libcephsqlite-2:17.2.0-188.g23b06583.el8.x86_64 16/134 2022-05-13T23:04:10.069 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : librados-devel-2:17.2.0-188.g23b06583.el8.x86_64 17/134 2022-05-13T23:04:10.069 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : libradosstriper1-2:17.2.0-188.g23b06583.el8.x86_ 18/134 2022-05-13T23:04:10.069 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : librgw2-2:17.2.0-188.g23b06583.el8.x86_64 19/134 2022-05-13T23:04:10.070 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-ceph-argparse-2:17.2.0-188.g23b06583.el8 20/134 2022-05-13T23:04:10.070 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-ceph-common-2:17.2.0-188.g23b06583.el8.x 21/134 2022-05-13T23:04:10.070 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-cephfs-2:17.2.0-188.g23b06583.el8.x86_64 22/134 2022-05-13T23:04:10.070 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-rados-2:17.2.0-188.g23b06583.el8.x86_64 23/134 2022-05-13T23:04:10.070 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-rbd-2:17.2.0-188.g23b06583.el8.x86_64 24/134 2022-05-13T23:04:10.071 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-rgw-2:17.2.0-188.g23b06583.el8.x86_64 25/134 2022-05-13T23:04:10.071 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : rbd-fuse-2:17.2.0-188.g23b06583.el8.x86_64 26/134 2022-05-13T23:04:10.071 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : rbd-mirror-2:17.2.0-188.g23b06583.el8.x86_64 27/134 2022-05-13T23:04:10.071 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : rbd-nbd-2:17.2.0-188.g23b06583.el8.x86_64 28/134 2022-05-13T23:04:10.072 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : ceph-grafana-dashboards-2:17.2.0-188.g23b06583.e 29/134 2022-05-13T23:04:10.072 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : ceph-mgr-cephadm-2:17.2.0-188.g23b06583.el8.noar 30/134 2022-05-13T23:04:10.072 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : ceph-mgr-dashboard-2:17.2.0-188.g23b06583.el8.no 31/134 2022-05-13T23:04:10.072 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : ceph-mgr-diskprediction-local-2:17.2.0-188.g23b0 32/134 2022-05-13T23:04:10.072 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : ceph-mgr-modules-core-2:17.2.0-188.g23b06583.el8 33/134 2022-05-13T23:04:10.073 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : ceph-mgr-rook-2:17.2.0-188.g23b06583.el8.noarch 34/134 2022-05-13T23:04:10.073 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : ceph-prometheus-alerts-2:17.2.0-188.g23b06583.el 35/134 2022-05-13T23:04:10.073 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : ceph-volume-2:17.2.0-188.g23b06583.el8.noarch 36/134 2022-05-13T23:04:10.073 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : cephadm-2:17.2.0-188.g23b06583.el8.noarch 37/134 2022-05-13T23:04:10.073 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : cephfs-top-2:17.2.0-188.g23b06583.el8.noarch 38/134 2022-05-13T23:04:10.074 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 39/134 2022-05-13T23:04:10.074 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 40/134 2022-05-13T23:04:10.074 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 41/134 2022-05-13T23:04:10.074 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-asyncssh-2.7.0-2.el8.noarch 42/134 2022-05-13T23:04:10.075 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 43/134 2022-05-13T23:04:10.075 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 44/134 2022-05-13T23:04:10.075 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 45/134 2022-05-13T23:04:10.076 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 46/134 2022-05-13T23:04:10.076 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 47/134 2022-05-13T23:04:10.076 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 48/134 2022-05-13T23:04:10.076 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 49/134 2022-05-13T23:04:10.076 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 50/134 2022-05-13T23:04:10.077 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 51/134 2022-05-13T23:04:10.077 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 52/134 2022-05-13T23:04:10.077 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 53/134 2022-05-13T23:04:10.077 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 54/134 2022-05-13T23:04:10.078 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-natsort-7.1.1-2.el8.noarch 55/134 2022-05-13T23:04:10.078 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 56/134 2022-05-13T23:04:10.078 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-portend-2.6-1.el8.noarch 57/134 2022-05-13T23:04:10.078 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 58/134 2022-05-13T23:04:10.078 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 59/134 2022-05-13T23:04:10.079 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-rsa-4.8-1.el8.noarch 60/134 2022-05-13T23:04:10.079 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 61/134 2022-05-13T23:04:10.079 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 62/134 2022-05-13T23:04:10.079 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 63/134 2022-05-13T23:04:10.079 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 64/134 2022-05-13T23:04:10.080 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-waitress-1.2.1-2.el8.1.noarch 65/134 2022-05-13T23:04:10.080 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 66/134 2022-05-13T23:04:10.080 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 67/134 2022-05-13T23:04:10.080 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 68/134 2022-05-13T23:04:10.080 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 69/134 2022-05-13T23:04:10.081 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : thrift-0.13.0-2.el8.x86_64 70/134 2022-05-13T23:04:10.081 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : xmlstarlet-1.6.1-11.el8.x86_64 71/134 2022-05-13T23:04:10.081 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 72/134 2022-05-13T23:04:10.081 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : libaio-devel-0.3.112-1.el8.x86_64 73/134 2022-05-13T23:04:10.082 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 74/134 2022-05-13T23:04:10.082 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : libquadmath-8.4.1-1.el8.x86_64 75/134 2022-05-13T23:04:10.082 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-cffi-1.11.5-5.el8.x86_64 76/134 2022-05-13T23:04:10.083 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : libgfortran-8.4.1-1.el8.x86_64 77/134 2022-05-13T23:04:10.083 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : elfutils-libelf-devel-0.182-3.el8.x86_64 78/134 2022-05-13T23:04:10.083 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : libselinux-devel-2.9-5.el8.x86_64 79/134 2022-05-13T23:04:10.083 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : zlib-devel-1.2.11-17.el8.x86_64 80/134 2022-05-13T23:04:10.083 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : pcre2-utf16-10.32-2.el8.x86_64 81/134 2022-05-13T23:04:10.084 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : librabbitmq-0.9.0-3.el8.x86_64 82/134 2022-05-13T23:04:10.084 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : libverto-devel-0.3.0-5.el8.x86_64 83/134 2022-05-13T23:04:10.084 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : libsepol-devel-2.9-2.el8.x86_64 84/134 2022-05-13T23:04:10.084 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : libuuid-devel-2.32.1-27.el8.x86_64 85/134 2022-05-13T23:04:10.084 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 86/134 2022-05-13T23:04:10.085 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : openssl-devel-1:1.1.1g-15.el8_3.x86_64 87/134 2022-05-13T23:04:10.085 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : fuse-2.9.7-12.el8.x86_64 88/134 2022-05-13T23:04:10.085 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 89/134 2022-05-13T23:04:10.085 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : pcre2-devel-10.32-2.el8.x86_64 90/134 2022-05-13T23:04:10.086 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-cryptography-3.2.1-4.el8.x86_64 91/134 2022-05-13T23:04:10.086 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : xfsprogs-devel-5.0.0-8.el8.x86_64 92/134 2022-05-13T23:04:10.086 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : pcre2-utf32-10.32-2.el8.x86_64 93/134 2022-05-13T23:04:10.086 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : keyutils-libs-devel-1.5.10-6.el8.x86_64 94/134 2022-05-13T23:04:10.087 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : libcom_err-devel-1.45.6-1.el8.x86_64 95/134 2022-05-13T23:04:10.087 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : krb5-devel-1.18.2-8.3.el8_4.x86_64 96/134 2022-05-13T23:04:10.087 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-mako-1.0.6-13.el8.noarch 97/134 2022-05-13T23:04:10.087 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : flex-2.6.1-9.el8.x86_64 98/134 2022-05-13T23:04:10.088 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : jq-1.5-12.el8.x86_64 99/134 2022-05-13T23:04:10.088 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 100/134 2022-05-13T23:04:10.088 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : librdkafka-0.11.4-1.el8.x86_64 101/134 2022-05-13T23:04:10.088 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-jinja2-2.10.1-2.el8_0.noarch 102/134 2022-05-13T23:04:10.089 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : automake-1.16.1-7.el8.noarch 103/134 2022-05-13T23:04:10.089 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : libtool-2.4.6-25.el8.x86_64 104/134 2022-05-13T23:04:10.089 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-numpy-1:1.14.3-9.el8.x86_64 105/134 2022-05-13T23:04:10.089 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 106/134 2022-05-13T23:04:10.089 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 107/134 2022-05-13T23:04:10.090 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-pytz-2017.2-9.el8.noarch 108/134 2022-05-13T23:04:10.090 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : oniguruma-6.8.2-2.el8.x86_64 109/134 2022-05-13T23:04:10.090 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 110/134 2022-05-13T23:04:10.090 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-scipy-1.0.0-20.module+el8.1.0+3334+5cb62 111/134 2022-05-13T23:04:10.090 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 112/134 2022-05-13T23:04:10.091 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 113/134 2022-05-13T23:04:10.091 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : openblas-threads-0.3.12-1.el8.x86_64 114/134 2022-05-13T23:04:10.091 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-babel-2.5.1-5.el8.noarch 115/134 2022-05-13T23:04:10.092 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 116/134 2022-05-13T23:04:10.092 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 117/134 2022-05-13T23:04:10.092 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 118/134 2022-05-13T23:04:10.092 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : socat-1.7.3.3-2.el8.x86_64 119/134 2022-05-13T23:04:10.093 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : openblas-0.3.12-1.el8.x86_64 120/134 2022-05-13T23:04:10.093 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : bison-3.0.4-10.el8.x86_64 121/134 2022-05-13T23:04:10.093 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-numpy-f2py-1:1.14.3-9.el8.x86_64 122/134 2022-05-13T23:04:10.093 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : librados2-2:17.2.0-188.g23b06583.el8.x86_64 123/134 2022-05-13T23:04:10.094 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 124/134 2022-05-13T23:04:10.094 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : librbd1-2:17.2.0-188.g23b06583.el8.x86_64 125/134 2022-05-13T23:04:10.094 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 126/134 2022-05-13T23:04:10.094 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : NetworkManager-team-1:1.30.0-13.el8_4.x86_64 127/134 2022-05-13T23:04:10.094 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : NetworkManager-team-1:1.30.0-7.el8.x86_64 128/134 2022-05-13T23:04:10.095 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : NetworkManager-libnm-1:1.30.0-13.el8_4.x86_64 129/134 2022-05-13T23:04:10.095 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : NetworkManager-libnm-1:1.30.0-7.el8.x86_64 130/134 2022-05-13T23:04:10.095 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : NetworkManager-tui-1:1.30.0-13.el8_4.x86_64 131/134 2022-05-13T23:04:10.095 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : NetworkManager-tui-1:1.30.0-7.el8.x86_64 132/134 2022-05-13T23:04:10.095 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : NetworkManager-1:1.30.0-13.el8_4.x86_64 133/134 2022-05-13T23:04:11.756 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : NetworkManager-1:1.30.0-7.el8.x86_64 134/134 2022-05-13T23:04:11.757 INFO:teuthology.orchestra.run.smithi047.stdout:Installed products updated. 2022-05-13T23:04:12.162 INFO:teuthology.orchestra.run.smithi115.stdout:Last metadata expiration check: 0:00:07 ago on Fri 13 May 2022 11:04:05 PM UTC. 2022-05-13T23:04:15.509 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: rbd-mirror-2:17.2.0-188.g23b06583.el8.x86_64 113/134 2022-05-13T23:04:15.848 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : ceph-test-2:17.2.0-188.g23b06583.el8.x86_64 114/134 2022-05-13T23:04:16.033 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : libtool-2.4.6-25.el8.x86_64 115/134 2022-05-13T23:04:16.407 INFO:teuthology.orchestra.run.smithi115.stdout: 2022-05-13T23:04:16.407 INFO:teuthology.orchestra.run.smithi115.stdout:Upgraded: 2022-05-13T23:04:16.408 INFO:teuthology.orchestra.run.smithi115.stdout: NetworkManager-1:1.30.0-13.el8_4.x86_64 2022-05-13T23:04:16.408 INFO:teuthology.orchestra.run.smithi115.stdout: NetworkManager-libnm-1:1.30.0-13.el8_4.x86_64 2022-05-13T23:04:16.408 INFO:teuthology.orchestra.run.smithi115.stdout: NetworkManager-team-1:1.30.0-13.el8_4.x86_64 2022-05-13T23:04:16.408 INFO:teuthology.orchestra.run.smithi115.stdout: NetworkManager-tui-1:1.30.0-13.el8_4.x86_64 2022-05-13T23:04:16.408 INFO:teuthology.orchestra.run.smithi115.stdout: librados2-2:17.2.0-188.g23b06583.el8.x86_64 2022-05-13T23:04:16.409 INFO:teuthology.orchestra.run.smithi115.stdout: librbd1-2:17.2.0-188.g23b06583.el8.x86_64 2022-05-13T23:04:16.409 INFO:teuthology.orchestra.run.smithi115.stdout:Installed: 2022-05-13T23:04:16.409 INFO:teuthology.orchestra.run.smithi115.stdout: automake-1.16.1-7.el8.noarch 2022-05-13T23:04:16.409 INFO:teuthology.orchestra.run.smithi115.stdout: bison-3.0.4-10.el8.x86_64 2022-05-13T23:04:16.409 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-2:17.2.0-188.g23b06583.el8.x86_64 2022-05-13T23:04:16.410 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-base-2:17.2.0-188.g23b06583.el8.x86_64 2022-05-13T23:04:16.410 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-common-2:17.2.0-188.g23b06583.el8.x86_64 2022-05-13T23:04:16.410 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-fuse-2:17.2.0-188.g23b06583.el8.x86_64 2022-05-13T23:04:16.410 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-grafana-dashboards-2:17.2.0-188.g23b06583.el8.noarch 2022-05-13T23:04:16.411 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-immutable-object-cache-2:17.2.0-188.g23b06583.el8.x86_64 2022-05-13T23:04:16.411 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-mds-2:17.2.0-188.g23b06583.el8.x86_64 2022-05-13T23:04:16.411 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-mgr-2:17.2.0-188.g23b06583.el8.x86_64 2022-05-13T23:04:16.411 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-mgr-cephadm-2:17.2.0-188.g23b06583.el8.noarch 2022-05-13T23:04:16.411 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-mgr-dashboard-2:17.2.0-188.g23b06583.el8.noarch 2022-05-13T23:04:16.412 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-mgr-diskprediction-local-2:17.2.0-188.g23b06583.el8.noarch 2022-05-13T23:04:16.412 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-mgr-modules-core-2:17.2.0-188.g23b06583.el8.noarch 2022-05-13T23:04:16.412 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-mgr-rook-2:17.2.0-188.g23b06583.el8.noarch 2022-05-13T23:04:16.412 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-mon-2:17.2.0-188.g23b06583.el8.x86_64 2022-05-13T23:04:16.412 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-osd-2:17.2.0-188.g23b06583.el8.x86_64 2022-05-13T23:04:16.413 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-prometheus-alerts-2:17.2.0-188.g23b06583.el8.noarch 2022-05-13T23:04:16.413 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-radosgw-2:17.2.0-188.g23b06583.el8.x86_64 2022-05-13T23:04:16.413 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-selinux-2:17.2.0-188.g23b06583.el8.x86_64 2022-05-13T23:04:16.416 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-test-2:17.2.0-188.g23b06583.el8.x86_64 2022-05-13T23:04:16.416 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-volume-2:17.2.0-188.g23b06583.el8.noarch 2022-05-13T23:04:16.416 INFO:teuthology.orchestra.run.smithi115.stdout: cephadm-2:17.2.0-188.g23b06583.el8.noarch 2022-05-13T23:04:16.417 INFO:teuthology.orchestra.run.smithi115.stdout: cephfs-mirror-2:17.2.0-188.g23b06583.el8.x86_64 2022-05-13T23:04:16.417 INFO:teuthology.orchestra.run.smithi115.stdout: cephfs-top-2:17.2.0-188.g23b06583.el8.noarch 2022-05-13T23:04:16.417 INFO:teuthology.orchestra.run.smithi115.stdout: elfutils-libelf-devel-0.182-3.el8.x86_64 2022-05-13T23:04:16.417 INFO:teuthology.orchestra.run.smithi115.stdout: flex-2.6.1-9.el8.x86_64 2022-05-13T23:04:16.418 INFO:teuthology.orchestra.run.smithi115.stdout: fuse-2.9.7-12.el8.x86_64 2022-05-13T23:04:16.418 INFO:teuthology.orchestra.run.smithi115.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2022-05-13T23:04:16.418 INFO:teuthology.orchestra.run.smithi115.stdout: jq-1.5-12.el8.x86_64 2022-05-13T23:04:16.419 INFO:teuthology.orchestra.run.smithi115.stdout: keyutils-libs-devel-1.5.10-6.el8.x86_64 2022-05-13T23:04:16.419 INFO:teuthology.orchestra.run.smithi115.stdout: krb5-devel-1.18.2-8.3.el8_4.x86_64 2022-05-13T23:04:16.419 INFO:teuthology.orchestra.run.smithi115.stdout: libaio-devel-0.3.112-1.el8.x86_64 2022-05-13T23:04:16.419 INFO:teuthology.orchestra.run.smithi115.stdout: libcephfs-devel-2:17.2.0-188.g23b06583.el8.x86_64 2022-05-13T23:04:16.420 INFO:teuthology.orchestra.run.smithi115.stdout: libcephfs2-2:17.2.0-188.g23b06583.el8.x86_64 2022-05-13T23:04:16.420 INFO:teuthology.orchestra.run.smithi115.stdout: libcephsqlite-2:17.2.0-188.g23b06583.el8.x86_64 2022-05-13T23:04:16.420 INFO:teuthology.orchestra.run.smithi115.stdout: libcom_err-devel-1.45.6-1.el8.x86_64 2022-05-13T23:04:16.421 INFO:teuthology.orchestra.run.smithi115.stdout: libgfortran-8.4.1-1.el8.x86_64 2022-05-13T23:04:16.421 INFO:teuthology.orchestra.run.smithi115.stdout: liboath-2.6.2-3.el8.x86_64 2022-05-13T23:04:16.421 INFO:teuthology.orchestra.run.smithi115.stdout: libquadmath-8.4.1-1.el8.x86_64 2022-05-13T23:04:16.421 INFO:teuthology.orchestra.run.smithi115.stdout: librabbitmq-0.9.0-3.el8.x86_64 2022-05-13T23:04:16.422 INFO:teuthology.orchestra.run.smithi115.stdout: librados-devel-2:17.2.0-188.g23b06583.el8.x86_64 2022-05-13T23:04:16.422 INFO:teuthology.orchestra.run.smithi115.stdout: libradosstriper1-2:17.2.0-188.g23b06583.el8.x86_64 2022-05-13T23:04:16.423 INFO:teuthology.orchestra.run.smithi115.stdout: librdkafka-0.11.4-1.el8.x86_64 2022-05-13T23:04:16.423 INFO:teuthology.orchestra.run.smithi115.stdout: librgw2-2:17.2.0-188.g23b06583.el8.x86_64 2022-05-13T23:04:16.423 INFO:teuthology.orchestra.run.smithi115.stdout: libselinux-devel-2.9-5.el8.x86_64 2022-05-13T23:04:16.424 INFO:teuthology.orchestra.run.smithi115.stdout: libsepol-devel-2.9-2.el8.x86_64 2022-05-13T23:04:16.424 INFO:teuthology.orchestra.run.smithi115.stdout: libtool-2.4.6-25.el8.x86_64 2022-05-13T23:04:16.425 INFO:teuthology.orchestra.run.smithi115.stdout: libunwind-1.3.1-3.el8.x86_64 2022-05-13T23:04:16.425 INFO:teuthology.orchestra.run.smithi115.stdout: libuuid-devel-2.32.1-27.el8.x86_64 2022-05-13T23:04:16.426 INFO:teuthology.orchestra.run.smithi115.stdout: libverto-devel-0.3.0-5.el8.x86_64 2022-05-13T23:04:16.426 INFO:teuthology.orchestra.run.smithi115.stdout: lttng-ust-2.8.1-11.el8.x86_64 2022-05-13T23:04:16.426 INFO:teuthology.orchestra.run.smithi115.stdout: oniguruma-6.8.2-2.el8.x86_64 2022-05-13T23:04:16.427 INFO:teuthology.orchestra.run.smithi115.stdout: openblas-0.3.12-1.el8.x86_64 2022-05-13T23:04:16.427 INFO:teuthology.orchestra.run.smithi115.stdout: openblas-threads-0.3.12-1.el8.x86_64 2022-05-13T23:04:16.427 INFO:teuthology.orchestra.run.smithi115.stdout: openssl-devel-1:1.1.1g-15.el8_3.x86_64 2022-05-13T23:04:16.428 INFO:teuthology.orchestra.run.smithi115.stdout: pcre2-devel-10.32-2.el8.x86_64 2022-05-13T23:04:16.428 INFO:teuthology.orchestra.run.smithi115.stdout: pcre2-utf16-10.32-2.el8.x86_64 2022-05-13T23:04:16.428 INFO:teuthology.orchestra.run.smithi115.stdout: pcre2-utf32-10.32-2.el8.x86_64 2022-05-13T23:04:16.428 INFO:teuthology.orchestra.run.smithi115.stdout: python3-asyncssh-2.7.0-2.el8.noarch 2022-05-13T23:04:16.429 INFO:teuthology.orchestra.run.smithi115.stdout: python3-babel-2.5.1-5.el8.noarch 2022-05-13T23:04:16.429 INFO:teuthology.orchestra.run.smithi115.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2022-05-13T23:04:16.429 INFO:teuthology.orchestra.run.smithi115.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2022-05-13T23:04:16.429 INFO:teuthology.orchestra.run.smithi115.stdout: python3-cachetools-3.1.1-4.el8.noarch 2022-05-13T23:04:16.430 INFO:teuthology.orchestra.run.smithi115.stdout: python3-ceph-argparse-2:17.2.0-188.g23b06583.el8.x86_64 2022-05-13T23:04:16.430 INFO:teuthology.orchestra.run.smithi115.stdout: python3-ceph-common-2:17.2.0-188.g23b06583.el8.x86_64 2022-05-13T23:04:16.430 INFO:teuthology.orchestra.run.smithi115.stdout: python3-cephfs-2:17.2.0-188.g23b06583.el8.x86_64 2022-05-13T23:04:16.431 INFO:teuthology.orchestra.run.smithi115.stdout: python3-certifi-2018.10.15-7.el8.noarch 2022-05-13T23:04:16.431 INFO:teuthology.orchestra.run.smithi115.stdout: python3-cffi-1.11.5-5.el8.x86_64 2022-05-13T23:04:16.431 INFO:teuthology.orchestra.run.smithi115.stdout: python3-cheroot-8.5.2-1.el8.noarch 2022-05-13T23:04:16.432 INFO:teuthology.orchestra.run.smithi115.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2022-05-13T23:04:16.432 INFO:teuthology.orchestra.run.smithi115.stdout: python3-cryptography-3.2.1-4.el8.x86_64 2022-05-13T23:04:16.432 INFO:teuthology.orchestra.run.smithi115.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2022-05-13T23:04:16.432 INFO:teuthology.orchestra.run.smithi115.stdout: python3-jaraco-6.2-6.el8.noarch 2022-05-13T23:04:16.432 INFO:teuthology.orchestra.run.smithi115.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2022-05-13T23:04:16.433 INFO:teuthology.orchestra.run.smithi115.stdout: python3-jinja2-2.10.1-2.el8_0.noarch 2022-05-13T23:04:16.433 INFO:teuthology.orchestra.run.smithi115.stdout: python3-jsonpatch-1.21-2.el8.noarch 2022-05-13T23:04:16.433 INFO:teuthology.orchestra.run.smithi115.stdout: python3-jsonpointer-1.10-11.el8.noarch 2022-05-13T23:04:16.433 INFO:teuthology.orchestra.run.smithi115.stdout: python3-jwt-1.6.1-2.el8.noarch 2022-05-13T23:04:16.434 INFO:teuthology.orchestra.run.smithi115.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2022-05-13T23:04:16.434 INFO:teuthology.orchestra.run.smithi115.stdout: python3-logutils-0.3.5-11.el8.noarch 2022-05-13T23:04:16.434 INFO:teuthology.orchestra.run.smithi115.stdout: python3-mako-1.0.6-13.el8.noarch 2022-05-13T23:04:16.434 INFO:teuthology.orchestra.run.smithi115.stdout: python3-markupsafe-0.23-19.el8.x86_64 2022-05-13T23:04:16.435 INFO:teuthology.orchestra.run.smithi115.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2022-05-13T23:04:16.435 INFO:teuthology.orchestra.run.smithi115.stdout: python3-natsort-7.1.1-2.el8.noarch 2022-05-13T23:04:16.436 INFO:teuthology.orchestra.run.smithi115.stdout: python3-numpy-1:1.14.3-9.el8.x86_64 2022-05-13T23:04:16.436 INFO:teuthology.orchestra.run.smithi115.stdout: python3-numpy-f2py-1:1.14.3-9.el8.x86_64 2022-05-13T23:04:16.436 INFO:teuthology.orchestra.run.smithi115.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2022-05-13T23:04:16.436 INFO:teuthology.orchestra.run.smithi115.stdout: python3-pecan-1.3.2-9.el8.noarch 2022-05-13T23:04:16.437 INFO:teuthology.orchestra.run.smithi115.stdout: python3-portend-2.6-1.el8.noarch 2022-05-13T23:04:16.437 INFO:teuthology.orchestra.run.smithi115.stdout: python3-prettytable-0.7.2-14.el8.noarch 2022-05-13T23:04:16.437 INFO:teuthology.orchestra.run.smithi115.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2022-05-13T23:04:16.437 INFO:teuthology.orchestra.run.smithi115.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2022-05-13T23:04:16.438 INFO:teuthology.orchestra.run.smithi115.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2022-05-13T23:04:16.438 INFO:teuthology.orchestra.run.smithi115.stdout: python3-pycparser-2.14-14.el8.noarch 2022-05-13T23:04:16.439 INFO:teuthology.orchestra.run.smithi115.stdout: python3-pytz-2017.2-9.el8.noarch 2022-05-13T23:04:16.439 INFO:teuthology.orchestra.run.smithi115.stdout: python3-rados-2:17.2.0-188.g23b06583.el8.x86_64 2022-05-13T23:04:16.439 INFO:teuthology.orchestra.run.smithi115.stdout: python3-rbd-2:17.2.0-188.g23b06583.el8.x86_64 2022-05-13T23:04:16.439 INFO:teuthology.orchestra.run.smithi115.stdout: python3-repoze-lru-0.7-6.el8.noarch 2022-05-13T23:04:16.439 INFO:teuthology.orchestra.run.smithi115.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2022-05-13T23:04:16.440 INFO:teuthology.orchestra.run.smithi115.stdout: python3-rgw-2:17.2.0-188.g23b06583.el8.x86_64 2022-05-13T23:04:16.440 INFO:teuthology.orchestra.run.smithi115.stdout: python3-routes-2.4.1-12.el8.noarch 2022-05-13T23:04:16.440 INFO:teuthology.orchestra.run.smithi115.stdout: python3-rsa-4.8-1.el8.noarch 2022-05-13T23:04:16.440 INFO:teuthology.orchestra.run.smithi115.stdout: python3-scipy-1.0.0-20.module+el8.1.0+3334+5cb623d7.x86_64 2022-05-13T23:04:16.441 INFO:teuthology.orchestra.run.smithi115.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2022-05-13T23:04:16.441 INFO:teuthology.orchestra.run.smithi115.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2022-05-13T23:04:16.441 INFO:teuthology.orchestra.run.smithi115.stdout: python3-tempora-1.14.1-5.el8.noarch 2022-05-13T23:04:16.441 INFO:teuthology.orchestra.run.smithi115.stdout: python3-trustme-0.6.0-4.el8.noarch 2022-05-13T23:04:16.442 INFO:teuthology.orchestra.run.smithi115.stdout: python3-waitress-1.2.1-2.el8.1.noarch 2022-05-13T23:04:16.442 INFO:teuthology.orchestra.run.smithi115.stdout: python3-webob-1.8.5-1.el8.1.noarch 2022-05-13T23:04:16.442 INFO:teuthology.orchestra.run.smithi115.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2022-05-13T23:04:16.442 INFO:teuthology.orchestra.run.smithi115.stdout: python3-webtest-2.0.33-1.el8.noarch 2022-05-13T23:04:16.442 INFO:teuthology.orchestra.run.smithi115.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2022-05-13T23:04:16.443 INFO:teuthology.orchestra.run.smithi115.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2022-05-13T23:04:16.443 INFO:teuthology.orchestra.run.smithi115.stdout: rbd-fuse-2:17.2.0-188.g23b06583.el8.x86_64 2022-05-13T23:04:16.443 INFO:teuthology.orchestra.run.smithi115.stdout: rbd-mirror-2:17.2.0-188.g23b06583.el8.x86_64 2022-05-13T23:04:16.443 INFO:teuthology.orchestra.run.smithi115.stdout: rbd-nbd-2:17.2.0-188.g23b06583.el8.x86_64 2022-05-13T23:04:16.444 INFO:teuthology.orchestra.run.smithi115.stdout: socat-1.7.3.3-2.el8.x86_64 2022-05-13T23:04:16.444 INFO:teuthology.orchestra.run.smithi115.stdout: thrift-0.13.0-2.el8.x86_64 2022-05-13T23:04:16.444 INFO:teuthology.orchestra.run.smithi115.stdout: xfsprogs-devel-5.0.0-8.el8.x86_64 2022-05-13T23:04:16.444 INFO:teuthology.orchestra.run.smithi115.stdout: xmlstarlet-1.6.1-11.el8.x86_64 2022-05-13T23:04:16.445 INFO:teuthology.orchestra.run.smithi115.stdout: zlib-devel-1.2.11-17.el8.x86_64 2022-05-13T23:04:16.446 INFO:teuthology.orchestra.run.smithi115.stdout: 2022-05-13T23:04:16.446 INFO:teuthology.orchestra.run.smithi115.stdout:Complete! 2022-05-13T23:04:16.610 DEBUG:teuthology.parallel:result is None 2022-05-13T23:04:17.410 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: libtool-2.4.6-25.el8.x86_64 115/134 2022-05-13T23:04:17.681 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : openssl-devel-1:1.1.1g-15.el8_3.x86_64 116/134 2022-05-13T23:04:17.958 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : xfsprogs-devel-5.0.0-8.el8.x86_64 117/134 2022-05-13T23:04:18.137 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : ceph-fuse-2:17.2.0-188.g23b06583.el8.x86_64 118/134 2022-05-13T23:04:18.271 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : libcephfs-devel-2:17.2.0-188.g23b06583.el8.x86_6 119/134 2022-05-13T23:04:18.410 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : elfutils-libelf-devel-0.182-3.el8.x86_64 120/134 2022-05-13T23:04:18.615 INFO:teuthology.orchestra.run.smithi002.stdout: Upgrading : NetworkManager-team-1:1.30.0-13.el8_4.x86_64 121/134 2022-05-13T23:04:18.788 INFO:teuthology.orchestra.run.smithi002.stdout: Upgrading : NetworkManager-tui-1:1.30.0-13.el8_4.x86_64 122/134 2022-05-13T23:04:18.992 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : cephfs-top-2:17.2.0-188.g23b06583.el8.noarch 123/134 2022-05-13T23:04:19.162 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : rbd-fuse-2:17.2.0-188.g23b06583.el8.x86_64 124/134 2022-05-13T23:04:19.324 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : rbd-nbd-2:17.2.0-188.g23b06583.el8.x86_64 125/134 2022-05-13T23:04:19.622 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : libaio-devel-0.3.112-1.el8.x86_64 126/134 2022-05-13T23:04:19.822 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : bison-3.0.4-10.el8.x86_64 127/134 2022-05-13T23:04:19.952 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: bison-3.0.4-10.el8.x86_64 127/134 2022-05-13T23:04:20.122 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : flex-2.6.1-9.el8.x86_64 128/134 2022-05-13T23:04:20.167 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: flex-2.6.1-9.el8.x86_64 128/134 2022-05-13T23:04:20.310 INFO:teuthology.orchestra.run.smithi002.stdout: Cleanup : NetworkManager-tui-1:1.30.0-7.el8.x86_64 129/134 2022-05-13T23:04:20.311 INFO:teuthology.orchestra.run.smithi002.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 130/134 2022-05-13T23:04:20.590 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 130/134 2022-05-13T23:04:20.708 INFO:teuthology.orchestra.run.smithi002.stdout: Cleanup : NetworkManager-team-1:1.30.0-7.el8.x86_64 131/134 2022-05-13T23:04:20.752 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: NetworkManager-1:1.30.0-7.el8.x86_64 132/134 2022-05-13T23:04:20.752 INFO:teuthology.orchestra.run.smithi002.stdout: Cleanup : NetworkManager-1:1.30.0-7.el8.x86_64 132/134 2022-05-13T23:04:21.328 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: NetworkManager-1:1.30.0-7.el8.x86_64 132/134 2022-05-13T23:04:21.329 INFO:teuthology.orchestra.run.smithi002.stdout: Cleanup : NetworkManager-libnm-1:1.30.0-7.el8.x86_64 133/134 2022-05-13T23:04:21.657 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: NetworkManager-libnm-1:1.30.0-7.el8.x86_64 133/134 2022-05-13T23:04:21.657 INFO:teuthology.orchestra.run.smithi002.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 134/134 2022-05-13T23:04:22.362 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 134/134 2022-05-13T23:04:22.363 INFO:teuthology.orchestra.run.smithi002.stdout:Couldn't write '0' to 'kernel/yama/ptrace_scope', ignoring: No such file or directory 2022-05-13T23:04:22.363 INFO:teuthology.orchestra.run.smithi002.stdout: 2022-05-13T23:04:23.386 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : ceph-2:17.2.0-188.g23b06583.el8.x86_64 1/134 2022-05-13T23:04:23.386 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : ceph-base-2:17.2.0-188.g23b06583.el8.x86_64 2/134 2022-05-13T23:04:23.387 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : ceph-common-2:17.2.0-188.g23b06583.el8.x86_64 3/134 2022-05-13T23:04:23.388 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : ceph-fuse-2:17.2.0-188.g23b06583.el8.x86_64 4/134 2022-05-13T23:04:23.388 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : ceph-immutable-object-cache-2:17.2.0-188.g23b065 5/134 2022-05-13T23:04:23.388 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : ceph-mds-2:17.2.0-188.g23b06583.el8.x86_64 6/134 2022-05-13T23:04:23.388 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : ceph-mgr-2:17.2.0-188.g23b06583.el8.x86_64 7/134 2022-05-13T23:04:23.388 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : ceph-mon-2:17.2.0-188.g23b06583.el8.x86_64 8/134 2022-05-13T23:04:23.389 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : ceph-osd-2:17.2.0-188.g23b06583.el8.x86_64 9/134 2022-05-13T23:04:23.389 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : ceph-radosgw-2:17.2.0-188.g23b06583.el8.x86_64 10/134 2022-05-13T23:04:23.389 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : ceph-selinux-2:17.2.0-188.g23b06583.el8.x86_64 11/134 2022-05-13T23:04:23.389 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : ceph-test-2:17.2.0-188.g23b06583.el8.x86_64 12/134 2022-05-13T23:04:23.390 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : cephfs-mirror-2:17.2.0-188.g23b06583.el8.x86_64 13/134 2022-05-13T23:04:23.390 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : libcephfs-devel-2:17.2.0-188.g23b06583.el8.x86_6 14/134 2022-05-13T23:04:23.390 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : libcephfs2-2:17.2.0-188.g23b06583.el8.x86_64 15/134 2022-05-13T23:04:23.391 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : libcephsqlite-2:17.2.0-188.g23b06583.el8.x86_64 16/134 2022-05-13T23:04:23.392 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : librados-devel-2:17.2.0-188.g23b06583.el8.x86_64 17/134 2022-05-13T23:04:23.392 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : libradosstriper1-2:17.2.0-188.g23b06583.el8.x86_ 18/134 2022-05-13T23:04:23.392 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : librgw2-2:17.2.0-188.g23b06583.el8.x86_64 19/134 2022-05-13T23:04:23.392 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-ceph-argparse-2:17.2.0-188.g23b06583.el8 20/134 2022-05-13T23:04:23.392 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-ceph-common-2:17.2.0-188.g23b06583.el8.x 21/134 2022-05-13T23:04:23.393 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-cephfs-2:17.2.0-188.g23b06583.el8.x86_64 22/134 2022-05-13T23:04:23.393 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-rados-2:17.2.0-188.g23b06583.el8.x86_64 23/134 2022-05-13T23:04:23.393 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-rbd-2:17.2.0-188.g23b06583.el8.x86_64 24/134 2022-05-13T23:04:23.393 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-rgw-2:17.2.0-188.g23b06583.el8.x86_64 25/134 2022-05-13T23:04:23.394 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : rbd-fuse-2:17.2.0-188.g23b06583.el8.x86_64 26/134 2022-05-13T23:04:23.394 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : rbd-mirror-2:17.2.0-188.g23b06583.el8.x86_64 27/134 2022-05-13T23:04:23.394 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : rbd-nbd-2:17.2.0-188.g23b06583.el8.x86_64 28/134 2022-05-13T23:04:23.394 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : ceph-grafana-dashboards-2:17.2.0-188.g23b06583.e 29/134 2022-05-13T23:04:23.395 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : ceph-mgr-cephadm-2:17.2.0-188.g23b06583.el8.noar 30/134 2022-05-13T23:04:23.395 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : ceph-mgr-dashboard-2:17.2.0-188.g23b06583.el8.no 31/134 2022-05-13T23:04:23.395 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : ceph-mgr-diskprediction-local-2:17.2.0-188.g23b0 32/134 2022-05-13T23:04:23.395 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : ceph-mgr-modules-core-2:17.2.0-188.g23b06583.el8 33/134 2022-05-13T23:04:23.396 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : ceph-mgr-rook-2:17.2.0-188.g23b06583.el8.noarch 34/134 2022-05-13T23:04:23.396 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : ceph-prometheus-alerts-2:17.2.0-188.g23b06583.el 35/134 2022-05-13T23:04:23.396 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : ceph-volume-2:17.2.0-188.g23b06583.el8.noarch 36/134 2022-05-13T23:04:23.396 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : cephadm-2:17.2.0-188.g23b06583.el8.noarch 37/134 2022-05-13T23:04:23.397 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : cephfs-top-2:17.2.0-188.g23b06583.el8.noarch 38/134 2022-05-13T23:04:23.397 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 39/134 2022-05-13T23:04:23.397 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 40/134 2022-05-13T23:04:23.397 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 41/134 2022-05-13T23:04:23.406 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-asyncssh-2.7.0-2.el8.noarch 42/134 2022-05-13T23:04:23.406 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 43/134 2022-05-13T23:04:23.406 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 44/134 2022-05-13T23:04:23.407 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 45/134 2022-05-13T23:04:23.407 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 46/134 2022-05-13T23:04:23.407 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 47/134 2022-05-13T23:04:23.407 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 48/134 2022-05-13T23:04:23.408 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 49/134 2022-05-13T23:04:23.408 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 50/134 2022-05-13T23:04:23.408 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 51/134 2022-05-13T23:04:23.408 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 52/134 2022-05-13T23:04:23.409 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 53/134 2022-05-13T23:04:23.409 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 54/134 2022-05-13T23:04:23.409 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-natsort-7.1.1-2.el8.noarch 55/134 2022-05-13T23:04:23.410 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 56/134 2022-05-13T23:04:23.410 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-portend-2.6-1.el8.noarch 57/134 2022-05-13T23:04:23.410 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 58/134 2022-05-13T23:04:23.410 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 59/134 2022-05-13T23:04:23.410 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-rsa-4.8-1.el8.noarch 60/134 2022-05-13T23:04:23.411 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 61/134 2022-05-13T23:04:23.411 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 62/134 2022-05-13T23:04:23.411 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 63/134 2022-05-13T23:04:23.412 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 64/134 2022-05-13T23:04:23.412 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-waitress-1.2.1-2.el8.1.noarch 65/134 2022-05-13T23:04:23.412 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 66/134 2022-05-13T23:04:23.412 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 67/134 2022-05-13T23:04:23.413 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 68/134 2022-05-13T23:04:23.413 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 69/134 2022-05-13T23:04:23.413 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : thrift-0.13.0-2.el8.x86_64 70/134 2022-05-13T23:04:23.413 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : xmlstarlet-1.6.1-11.el8.x86_64 71/134 2022-05-13T23:04:23.414 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-mako-1.0.6-13.el8.noarch 72/134 2022-05-13T23:04:23.414 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : flex-2.6.1-9.el8.x86_64 73/134 2022-05-13T23:04:23.414 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : jq-1.5-12.el8.x86_64 74/134 2022-05-13T23:04:23.414 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 75/134 2022-05-13T23:04:23.414 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : librdkafka-0.11.4-1.el8.x86_64 76/134 2022-05-13T23:04:23.415 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-jinja2-2.10.1-2.el8_0.noarch 77/134 2022-05-13T23:04:23.415 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : automake-1.16.1-7.el8.noarch 78/134 2022-05-13T23:04:23.415 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : libtool-2.4.6-25.el8.x86_64 79/134 2022-05-13T23:04:23.415 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-numpy-1:1.14.3-9.el8.x86_64 80/134 2022-05-13T23:04:23.416 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 81/134 2022-05-13T23:04:23.416 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 82/134 2022-05-13T23:04:23.416 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-pytz-2017.2-9.el8.noarch 83/134 2022-05-13T23:04:23.416 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : oniguruma-6.8.2-2.el8.x86_64 84/134 2022-05-13T23:04:23.416 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 85/134 2022-05-13T23:04:23.417 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-scipy-1.0.0-20.module+el8.1.0+3334+5cb62 86/134 2022-05-13T23:04:23.417 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 87/134 2022-05-13T23:04:23.417 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 88/134 2022-05-13T23:04:23.417 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : openblas-threads-0.3.12-1.el8.x86_64 89/134 2022-05-13T23:04:23.418 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-babel-2.5.1-5.el8.noarch 90/134 2022-05-13T23:04:23.418 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 91/134 2022-05-13T23:04:23.418 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 92/134 2022-05-13T23:04:23.418 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 93/134 2022-05-13T23:04:23.418 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : socat-1.7.3.3-2.el8.x86_64 94/134 2022-05-13T23:04:23.419 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : openblas-0.3.12-1.el8.x86_64 95/134 2022-05-13T23:04:23.419 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : bison-3.0.4-10.el8.x86_64 96/134 2022-05-13T23:04:23.419 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-numpy-f2py-1:1.14.3-9.el8.x86_64 97/134 2022-05-13T23:04:23.419 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 98/134 2022-05-13T23:04:23.419 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : libaio-devel-0.3.112-1.el8.x86_64 99/134 2022-05-13T23:04:23.420 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 100/134 2022-05-13T23:04:23.420 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : libquadmath-8.4.1-1.el8.x86_64 101/134 2022-05-13T23:04:23.420 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-cffi-1.11.5-5.el8.x86_64 102/134 2022-05-13T23:04:23.420 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : libgfortran-8.4.1-1.el8.x86_64 103/134 2022-05-13T23:04:23.420 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : elfutils-libelf-devel-0.182-3.el8.x86_64 104/134 2022-05-13T23:04:23.421 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : libselinux-devel-2.9-5.el8.x86_64 105/134 2022-05-13T23:04:23.421 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : zlib-devel-1.2.11-17.el8.x86_64 106/134 2022-05-13T23:04:23.421 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : pcre2-utf16-10.32-2.el8.x86_64 107/134 2022-05-13T23:04:23.421 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : librabbitmq-0.9.0-3.el8.x86_64 108/134 2022-05-13T23:04:23.422 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : libverto-devel-0.3.0-5.el8.x86_64 109/134 2022-05-13T23:04:23.422 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : libsepol-devel-2.9-2.el8.x86_64 110/134 2022-05-13T23:04:23.422 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : libuuid-devel-2.32.1-27.el8.x86_64 111/134 2022-05-13T23:04:23.423 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 112/134 2022-05-13T23:04:23.423 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : openssl-devel-1:1.1.1g-15.el8_3.x86_64 113/134 2022-05-13T23:04:23.424 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : fuse-2.9.7-12.el8.x86_64 114/134 2022-05-13T23:04:23.424 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 115/134 2022-05-13T23:04:23.424 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : pcre2-devel-10.32-2.el8.x86_64 116/134 2022-05-13T23:04:23.424 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-cryptography-3.2.1-4.el8.x86_64 117/134 2022-05-13T23:04:23.425 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : xfsprogs-devel-5.0.0-8.el8.x86_64 118/134 2022-05-13T23:04:23.425 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : pcre2-utf32-10.32-2.el8.x86_64 119/134 2022-05-13T23:04:23.425 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : keyutils-libs-devel-1.5.10-6.el8.x86_64 120/134 2022-05-13T23:04:23.425 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : libcom_err-devel-1.45.6-1.el8.x86_64 121/134 2022-05-13T23:04:23.426 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : krb5-devel-1.18.2-8.3.el8_4.x86_64 122/134 2022-05-13T23:04:23.426 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : librados2-2:17.2.0-188.g23b06583.el8.x86_64 123/134 2022-05-13T23:04:23.426 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 124/134 2022-05-13T23:04:23.426 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : librbd1-2:17.2.0-188.g23b06583.el8.x86_64 125/134 2022-05-13T23:04:23.427 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 126/134 2022-05-13T23:04:23.427 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : NetworkManager-team-1:1.30.0-13.el8_4.x86_64 127/134 2022-05-13T23:04:23.427 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : NetworkManager-team-1:1.30.0-7.el8.x86_64 128/134 2022-05-13T23:04:23.428 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : NetworkManager-libnm-1:1.30.0-13.el8_4.x86_64 129/134 2022-05-13T23:04:23.428 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : NetworkManager-libnm-1:1.30.0-7.el8.x86_64 130/134 2022-05-13T23:04:23.428 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : NetworkManager-tui-1:1.30.0-13.el8_4.x86_64 131/134 2022-05-13T23:04:23.428 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : NetworkManager-tui-1:1.30.0-7.el8.x86_64 132/134 2022-05-13T23:04:23.429 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : NetworkManager-1:1.30.0-13.el8_4.x86_64 133/134 2022-05-13T23:04:25.006 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : NetworkManager-1:1.30.0-7.el8.x86_64 134/134 2022-05-13T23:04:25.006 INFO:teuthology.orchestra.run.smithi002.stdout:Installed products updated. 2022-05-13T23:04:27.264 INFO:teuthology.orchestra.run.smithi047.stdout:Last metadata expiration check: 0:00:07 ago on Fri 13 May 2022 11:04:19 PM UTC. 2022-05-13T23:04:31.410 INFO:teuthology.orchestra.run.smithi047.stdout: 2022-05-13T23:04:31.411 INFO:teuthology.orchestra.run.smithi047.stdout:Upgraded: 2022-05-13T23:04:31.411 INFO:teuthology.orchestra.run.smithi047.stdout: NetworkManager-1:1.30.0-13.el8_4.x86_64 2022-05-13T23:04:31.411 INFO:teuthology.orchestra.run.smithi047.stdout: NetworkManager-libnm-1:1.30.0-13.el8_4.x86_64 2022-05-13T23:04:31.412 INFO:teuthology.orchestra.run.smithi047.stdout: NetworkManager-team-1:1.30.0-13.el8_4.x86_64 2022-05-13T23:04:31.412 INFO:teuthology.orchestra.run.smithi047.stdout: NetworkManager-tui-1:1.30.0-13.el8_4.x86_64 2022-05-13T23:04:31.412 INFO:teuthology.orchestra.run.smithi047.stdout: librados2-2:17.2.0-188.g23b06583.el8.x86_64 2022-05-13T23:04:31.412 INFO:teuthology.orchestra.run.smithi047.stdout: librbd1-2:17.2.0-188.g23b06583.el8.x86_64 2022-05-13T23:04:31.413 INFO:teuthology.orchestra.run.smithi047.stdout:Installed: 2022-05-13T23:04:31.413 INFO:teuthology.orchestra.run.smithi047.stdout: automake-1.16.1-7.el8.noarch 2022-05-13T23:04:31.413 INFO:teuthology.orchestra.run.smithi047.stdout: bison-3.0.4-10.el8.x86_64 2022-05-13T23:04:31.413 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-2:17.2.0-188.g23b06583.el8.x86_64 2022-05-13T23:04:31.414 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-base-2:17.2.0-188.g23b06583.el8.x86_64 2022-05-13T23:04:31.414 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-common-2:17.2.0-188.g23b06583.el8.x86_64 2022-05-13T23:04:31.414 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-fuse-2:17.2.0-188.g23b06583.el8.x86_64 2022-05-13T23:04:31.415 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-grafana-dashboards-2:17.2.0-188.g23b06583.el8.noarch 2022-05-13T23:04:31.415 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-immutable-object-cache-2:17.2.0-188.g23b06583.el8.x86_64 2022-05-13T23:04:31.415 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-mds-2:17.2.0-188.g23b06583.el8.x86_64 2022-05-13T23:04:31.415 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-mgr-2:17.2.0-188.g23b06583.el8.x86_64 2022-05-13T23:04:31.416 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-mgr-cephadm-2:17.2.0-188.g23b06583.el8.noarch 2022-05-13T23:04:31.416 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-mgr-dashboard-2:17.2.0-188.g23b06583.el8.noarch 2022-05-13T23:04:31.416 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-mgr-diskprediction-local-2:17.2.0-188.g23b06583.el8.noarch 2022-05-13T23:04:31.417 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-mgr-modules-core-2:17.2.0-188.g23b06583.el8.noarch 2022-05-13T23:04:31.417 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-mgr-rook-2:17.2.0-188.g23b06583.el8.noarch 2022-05-13T23:04:31.417 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-mon-2:17.2.0-188.g23b06583.el8.x86_64 2022-05-13T23:04:31.418 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-osd-2:17.2.0-188.g23b06583.el8.x86_64 2022-05-13T23:04:31.418 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-prometheus-alerts-2:17.2.0-188.g23b06583.el8.noarch 2022-05-13T23:04:31.418 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-radosgw-2:17.2.0-188.g23b06583.el8.x86_64 2022-05-13T23:04:31.418 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-selinux-2:17.2.0-188.g23b06583.el8.x86_64 2022-05-13T23:04:31.419 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-test-2:17.2.0-188.g23b06583.el8.x86_64 2022-05-13T23:04:31.419 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-volume-2:17.2.0-188.g23b06583.el8.noarch 2022-05-13T23:04:31.419 INFO:teuthology.orchestra.run.smithi047.stdout: cephadm-2:17.2.0-188.g23b06583.el8.noarch 2022-05-13T23:04:31.419 INFO:teuthology.orchestra.run.smithi047.stdout: cephfs-mirror-2:17.2.0-188.g23b06583.el8.x86_64 2022-05-13T23:04:31.420 INFO:teuthology.orchestra.run.smithi047.stdout: cephfs-top-2:17.2.0-188.g23b06583.el8.noarch 2022-05-13T23:04:31.420 INFO:teuthology.orchestra.run.smithi047.stdout: elfutils-libelf-devel-0.182-3.el8.x86_64 2022-05-13T23:04:31.420 INFO:teuthology.orchestra.run.smithi047.stdout: flex-2.6.1-9.el8.x86_64 2022-05-13T23:04:31.421 INFO:teuthology.orchestra.run.smithi047.stdout: fuse-2.9.7-12.el8.x86_64 2022-05-13T23:04:31.421 INFO:teuthology.orchestra.run.smithi047.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2022-05-13T23:04:31.421 INFO:teuthology.orchestra.run.smithi047.stdout: jq-1.5-12.el8.x86_64 2022-05-13T23:04:31.421 INFO:teuthology.orchestra.run.smithi047.stdout: keyutils-libs-devel-1.5.10-6.el8.x86_64 2022-05-13T23:04:31.421 INFO:teuthology.orchestra.run.smithi047.stdout: krb5-devel-1.18.2-8.3.el8_4.x86_64 2022-05-13T23:04:31.422 INFO:teuthology.orchestra.run.smithi047.stdout: libaio-devel-0.3.112-1.el8.x86_64 2022-05-13T23:04:31.422 INFO:teuthology.orchestra.run.smithi047.stdout: libcephfs-devel-2:17.2.0-188.g23b06583.el8.x86_64 2022-05-13T23:04:31.422 INFO:teuthology.orchestra.run.smithi047.stdout: libcephfs2-2:17.2.0-188.g23b06583.el8.x86_64 2022-05-13T23:04:31.423 INFO:teuthology.orchestra.run.smithi047.stdout: libcephsqlite-2:17.2.0-188.g23b06583.el8.x86_64 2022-05-13T23:04:31.423 INFO:teuthology.orchestra.run.smithi047.stdout: libcom_err-devel-1.45.6-1.el8.x86_64 2022-05-13T23:04:31.423 INFO:teuthology.orchestra.run.smithi047.stdout: libgfortran-8.4.1-1.el8.x86_64 2022-05-13T23:04:31.423 INFO:teuthology.orchestra.run.smithi047.stdout: liboath-2.6.2-3.el8.x86_64 2022-05-13T23:04:31.424 INFO:teuthology.orchestra.run.smithi047.stdout: libquadmath-8.4.1-1.el8.x86_64 2022-05-13T23:04:31.424 INFO:teuthology.orchestra.run.smithi047.stdout: librabbitmq-0.9.0-3.el8.x86_64 2022-05-13T23:04:31.424 INFO:teuthology.orchestra.run.smithi047.stdout: librados-devel-2:17.2.0-188.g23b06583.el8.x86_64 2022-05-13T23:04:31.424 INFO:teuthology.orchestra.run.smithi047.stdout: libradosstriper1-2:17.2.0-188.g23b06583.el8.x86_64 2022-05-13T23:04:31.425 INFO:teuthology.orchestra.run.smithi047.stdout: librdkafka-0.11.4-1.el8.x86_64 2022-05-13T23:04:31.425 INFO:teuthology.orchestra.run.smithi047.stdout: librgw2-2:17.2.0-188.g23b06583.el8.x86_64 2022-05-13T23:04:31.425 INFO:teuthology.orchestra.run.smithi047.stdout: libselinux-devel-2.9-5.el8.x86_64 2022-05-13T23:04:31.425 INFO:teuthology.orchestra.run.smithi047.stdout: libsepol-devel-2.9-2.el8.x86_64 2022-05-13T23:04:31.426 INFO:teuthology.orchestra.run.smithi047.stdout: libtool-2.4.6-25.el8.x86_64 2022-05-13T23:04:31.426 INFO:teuthology.orchestra.run.smithi047.stdout: libunwind-1.3.1-3.el8.x86_64 2022-05-13T23:04:31.426 INFO:teuthology.orchestra.run.smithi047.stdout: libuuid-devel-2.32.1-27.el8.x86_64 2022-05-13T23:04:31.426 INFO:teuthology.orchestra.run.smithi047.stdout: libverto-devel-0.3.0-5.el8.x86_64 2022-05-13T23:04:31.427 INFO:teuthology.orchestra.run.smithi047.stdout: lttng-ust-2.8.1-11.el8.x86_64 2022-05-13T23:04:31.427 INFO:teuthology.orchestra.run.smithi047.stdout: oniguruma-6.8.2-2.el8.x86_64 2022-05-13T23:04:31.428 INFO:teuthology.orchestra.run.smithi047.stdout: openblas-0.3.12-1.el8.x86_64 2022-05-13T23:04:31.428 INFO:teuthology.orchestra.run.smithi047.stdout: openblas-threads-0.3.12-1.el8.x86_64 2022-05-13T23:04:31.428 INFO:teuthology.orchestra.run.smithi047.stdout: openssl-devel-1:1.1.1g-15.el8_3.x86_64 2022-05-13T23:04:31.428 INFO:teuthology.orchestra.run.smithi047.stdout: pcre2-devel-10.32-2.el8.x86_64 2022-05-13T23:04:31.428 INFO:teuthology.orchestra.run.smithi047.stdout: pcre2-utf16-10.32-2.el8.x86_64 2022-05-13T23:04:31.429 INFO:teuthology.orchestra.run.smithi047.stdout: pcre2-utf32-10.32-2.el8.x86_64 2022-05-13T23:04:31.429 INFO:teuthology.orchestra.run.smithi047.stdout: python3-asyncssh-2.7.0-2.el8.noarch 2022-05-13T23:04:31.429 INFO:teuthology.orchestra.run.smithi047.stdout: python3-babel-2.5.1-5.el8.noarch 2022-05-13T23:04:31.429 INFO:teuthology.orchestra.run.smithi047.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2022-05-13T23:04:31.430 INFO:teuthology.orchestra.run.smithi047.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2022-05-13T23:04:31.430 INFO:teuthology.orchestra.run.smithi047.stdout: python3-cachetools-3.1.1-4.el8.noarch 2022-05-13T23:04:31.430 INFO:teuthology.orchestra.run.smithi047.stdout: python3-ceph-argparse-2:17.2.0-188.g23b06583.el8.x86_64 2022-05-13T23:04:31.430 INFO:teuthology.orchestra.run.smithi047.stdout: python3-ceph-common-2:17.2.0-188.g23b06583.el8.x86_64 2022-05-13T23:04:31.430 INFO:teuthology.orchestra.run.smithi047.stdout: python3-cephfs-2:17.2.0-188.g23b06583.el8.x86_64 2022-05-13T23:04:31.431 INFO:teuthology.orchestra.run.smithi047.stdout: python3-certifi-2018.10.15-7.el8.noarch 2022-05-13T23:04:31.431 INFO:teuthology.orchestra.run.smithi047.stdout: python3-cffi-1.11.5-5.el8.x86_64 2022-05-13T23:04:31.431 INFO:teuthology.orchestra.run.smithi047.stdout: python3-cheroot-8.5.2-1.el8.noarch 2022-05-13T23:04:31.431 INFO:teuthology.orchestra.run.smithi047.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2022-05-13T23:04:31.432 INFO:teuthology.orchestra.run.smithi047.stdout: python3-cryptography-3.2.1-4.el8.x86_64 2022-05-13T23:04:31.432 INFO:teuthology.orchestra.run.smithi047.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2022-05-13T23:04:31.432 INFO:teuthology.orchestra.run.smithi047.stdout: python3-jaraco-6.2-6.el8.noarch 2022-05-13T23:04:31.432 INFO:teuthology.orchestra.run.smithi047.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2022-05-13T23:04:31.432 INFO:teuthology.orchestra.run.smithi047.stdout: python3-jinja2-2.10.1-2.el8_0.noarch 2022-05-13T23:04:31.433 INFO:teuthology.orchestra.run.smithi047.stdout: python3-jsonpatch-1.21-2.el8.noarch 2022-05-13T23:04:31.433 INFO:teuthology.orchestra.run.smithi047.stdout: python3-jsonpointer-1.10-11.el8.noarch 2022-05-13T23:04:31.433 INFO:teuthology.orchestra.run.smithi047.stdout: python3-jwt-1.6.1-2.el8.noarch 2022-05-13T23:04:31.433 INFO:teuthology.orchestra.run.smithi047.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2022-05-13T23:04:31.434 INFO:teuthology.orchestra.run.smithi047.stdout: python3-logutils-0.3.5-11.el8.noarch 2022-05-13T23:04:31.434 INFO:teuthology.orchestra.run.smithi047.stdout: python3-mako-1.0.6-13.el8.noarch 2022-05-13T23:04:31.434 INFO:teuthology.orchestra.run.smithi047.stdout: python3-markupsafe-0.23-19.el8.x86_64 2022-05-13T23:04:31.435 INFO:teuthology.orchestra.run.smithi047.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2022-05-13T23:04:31.435 INFO:teuthology.orchestra.run.smithi047.stdout: python3-natsort-7.1.1-2.el8.noarch 2022-05-13T23:04:31.435 INFO:teuthology.orchestra.run.smithi047.stdout: python3-numpy-1:1.14.3-9.el8.x86_64 2022-05-13T23:04:31.435 INFO:teuthology.orchestra.run.smithi047.stdout: python3-numpy-f2py-1:1.14.3-9.el8.x86_64 2022-05-13T23:04:31.436 INFO:teuthology.orchestra.run.smithi047.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2022-05-13T23:04:31.436 INFO:teuthology.orchestra.run.smithi047.stdout: python3-pecan-1.3.2-9.el8.noarch 2022-05-13T23:04:31.436 INFO:teuthology.orchestra.run.smithi047.stdout: python3-portend-2.6-1.el8.noarch 2022-05-13T23:04:31.437 INFO:teuthology.orchestra.run.smithi047.stdout: python3-prettytable-0.7.2-14.el8.noarch 2022-05-13T23:04:31.437 INFO:teuthology.orchestra.run.smithi047.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2022-05-13T23:04:31.438 INFO:teuthology.orchestra.run.smithi047.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2022-05-13T23:04:31.438 INFO:teuthology.orchestra.run.smithi047.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2022-05-13T23:04:31.438 INFO:teuthology.orchestra.run.smithi047.stdout: python3-pycparser-2.14-14.el8.noarch 2022-05-13T23:04:31.439 INFO:teuthology.orchestra.run.smithi047.stdout: python3-pytz-2017.2-9.el8.noarch 2022-05-13T23:04:31.439 INFO:teuthology.orchestra.run.smithi047.stdout: python3-rados-2:17.2.0-188.g23b06583.el8.x86_64 2022-05-13T23:04:31.439 INFO:teuthology.orchestra.run.smithi047.stdout: python3-rbd-2:17.2.0-188.g23b06583.el8.x86_64 2022-05-13T23:04:31.439 INFO:teuthology.orchestra.run.smithi047.stdout: python3-repoze-lru-0.7-6.el8.noarch 2022-05-13T23:04:31.440 INFO:teuthology.orchestra.run.smithi047.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2022-05-13T23:04:31.440 INFO:teuthology.orchestra.run.smithi047.stdout: python3-rgw-2:17.2.0-188.g23b06583.el8.x86_64 2022-05-13T23:04:31.440 INFO:teuthology.orchestra.run.smithi047.stdout: python3-routes-2.4.1-12.el8.noarch 2022-05-13T23:04:31.440 INFO:teuthology.orchestra.run.smithi047.stdout: python3-rsa-4.8-1.el8.noarch 2022-05-13T23:04:31.441 INFO:teuthology.orchestra.run.smithi047.stdout: python3-scipy-1.0.0-20.module+el8.1.0+3334+5cb623d7.x86_64 2022-05-13T23:04:31.441 INFO:teuthology.orchestra.run.smithi047.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2022-05-13T23:04:31.441 INFO:teuthology.orchestra.run.smithi047.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2022-05-13T23:04:31.441 INFO:teuthology.orchestra.run.smithi047.stdout: python3-tempora-1.14.1-5.el8.noarch 2022-05-13T23:04:31.441 INFO:teuthology.orchestra.run.smithi047.stdout: python3-trustme-0.6.0-4.el8.noarch 2022-05-13T23:04:31.442 INFO:teuthology.orchestra.run.smithi047.stdout: python3-waitress-1.2.1-2.el8.1.noarch 2022-05-13T23:04:31.442 INFO:teuthology.orchestra.run.smithi047.stdout: python3-webob-1.8.5-1.el8.1.noarch 2022-05-13T23:04:31.442 INFO:teuthology.orchestra.run.smithi047.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2022-05-13T23:04:31.445 INFO:teuthology.orchestra.run.smithi047.stdout: python3-webtest-2.0.33-1.el8.noarch 2022-05-13T23:04:31.445 INFO:teuthology.orchestra.run.smithi047.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2022-05-13T23:04:31.446 INFO:teuthology.orchestra.run.smithi047.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2022-05-13T23:04:31.446 INFO:teuthology.orchestra.run.smithi047.stdout: rbd-fuse-2:17.2.0-188.g23b06583.el8.x86_64 2022-05-13T23:04:31.446 INFO:teuthology.orchestra.run.smithi047.stdout: rbd-mirror-2:17.2.0-188.g23b06583.el8.x86_64 2022-05-13T23:04:31.446 INFO:teuthology.orchestra.run.smithi047.stdout: rbd-nbd-2:17.2.0-188.g23b06583.el8.x86_64 2022-05-13T23:04:31.447 INFO:teuthology.orchestra.run.smithi047.stdout: socat-1.7.3.3-2.el8.x86_64 2022-05-13T23:04:31.447 INFO:teuthology.orchestra.run.smithi047.stdout: thrift-0.13.0-2.el8.x86_64 2022-05-13T23:04:31.447 INFO:teuthology.orchestra.run.smithi047.stdout: xfsprogs-devel-5.0.0-8.el8.x86_64 2022-05-13T23:04:31.447 INFO:teuthology.orchestra.run.smithi047.stdout: xmlstarlet-1.6.1-11.el8.x86_64 2022-05-13T23:04:31.449 INFO:teuthology.orchestra.run.smithi047.stdout: zlib-devel-1.2.11-17.el8.x86_64 2022-05-13T23:04:31.449 INFO:teuthology.orchestra.run.smithi047.stdout: 2022-05-13T23:04:31.449 INFO:teuthology.orchestra.run.smithi047.stdout:Complete! 2022-05-13T23:04:31.568 DEBUG:teuthology.parallel:result is None 2022-05-13T23:04:42.274 INFO:teuthology.orchestra.run.smithi002.stdout:Last metadata expiration check: 0:00:08 ago on Fri 13 May 2022 11:04:34 PM UTC. 2022-05-13T23:04:46.739 INFO:teuthology.orchestra.run.smithi002.stdout: 2022-05-13T23:04:46.739 INFO:teuthology.orchestra.run.smithi002.stdout:Upgraded: 2022-05-13T23:04:46.739 INFO:teuthology.orchestra.run.smithi002.stdout: NetworkManager-1:1.30.0-13.el8_4.x86_64 2022-05-13T23:04:46.740 INFO:teuthology.orchestra.run.smithi002.stdout: NetworkManager-libnm-1:1.30.0-13.el8_4.x86_64 2022-05-13T23:04:46.740 INFO:teuthology.orchestra.run.smithi002.stdout: NetworkManager-team-1:1.30.0-13.el8_4.x86_64 2022-05-13T23:04:46.740 INFO:teuthology.orchestra.run.smithi002.stdout: NetworkManager-tui-1:1.30.0-13.el8_4.x86_64 2022-05-13T23:04:46.740 INFO:teuthology.orchestra.run.smithi002.stdout: librados2-2:17.2.0-188.g23b06583.el8.x86_64 2022-05-13T23:04:46.741 INFO:teuthology.orchestra.run.smithi002.stdout: librbd1-2:17.2.0-188.g23b06583.el8.x86_64 2022-05-13T23:04:46.741 INFO:teuthology.orchestra.run.smithi002.stdout:Installed: 2022-05-13T23:04:46.741 INFO:teuthology.orchestra.run.smithi002.stdout: automake-1.16.1-7.el8.noarch 2022-05-13T23:04:46.741 INFO:teuthology.orchestra.run.smithi002.stdout: bison-3.0.4-10.el8.x86_64 2022-05-13T23:04:46.742 INFO:teuthology.orchestra.run.smithi002.stdout: ceph-2:17.2.0-188.g23b06583.el8.x86_64 2022-05-13T23:04:46.742 INFO:teuthology.orchestra.run.smithi002.stdout: ceph-base-2:17.2.0-188.g23b06583.el8.x86_64 2022-05-13T23:04:46.742 INFO:teuthology.orchestra.run.smithi002.stdout: ceph-common-2:17.2.0-188.g23b06583.el8.x86_64 2022-05-13T23:04:46.743 INFO:teuthology.orchestra.run.smithi002.stdout: ceph-fuse-2:17.2.0-188.g23b06583.el8.x86_64 2022-05-13T23:04:46.743 INFO:teuthology.orchestra.run.smithi002.stdout: ceph-grafana-dashboards-2:17.2.0-188.g23b06583.el8.noarch 2022-05-13T23:04:46.743 INFO:teuthology.orchestra.run.smithi002.stdout: ceph-immutable-object-cache-2:17.2.0-188.g23b06583.el8.x86_64 2022-05-13T23:04:46.744 INFO:teuthology.orchestra.run.smithi002.stdout: ceph-mds-2:17.2.0-188.g23b06583.el8.x86_64 2022-05-13T23:04:46.744 INFO:teuthology.orchestra.run.smithi002.stdout: ceph-mgr-2:17.2.0-188.g23b06583.el8.x86_64 2022-05-13T23:04:46.746 INFO:teuthology.orchestra.run.smithi002.stdout: ceph-mgr-cephadm-2:17.2.0-188.g23b06583.el8.noarch 2022-05-13T23:04:46.746 INFO:teuthology.orchestra.run.smithi002.stdout: ceph-mgr-dashboard-2:17.2.0-188.g23b06583.el8.noarch 2022-05-13T23:04:46.746 INFO:teuthology.orchestra.run.smithi002.stdout: ceph-mgr-diskprediction-local-2:17.2.0-188.g23b06583.el8.noarch 2022-05-13T23:04:46.747 INFO:teuthology.orchestra.run.smithi002.stdout: ceph-mgr-modules-core-2:17.2.0-188.g23b06583.el8.noarch 2022-05-13T23:04:46.747 INFO:teuthology.orchestra.run.smithi002.stdout: ceph-mgr-rook-2:17.2.0-188.g23b06583.el8.noarch 2022-05-13T23:04:46.748 INFO:teuthology.orchestra.run.smithi002.stdout: ceph-mon-2:17.2.0-188.g23b06583.el8.x86_64 2022-05-13T23:04:46.748 INFO:teuthology.orchestra.run.smithi002.stdout: ceph-osd-2:17.2.0-188.g23b06583.el8.x86_64 2022-05-13T23:04:46.749 INFO:teuthology.orchestra.run.smithi002.stdout: ceph-prometheus-alerts-2:17.2.0-188.g23b06583.el8.noarch 2022-05-13T23:04:46.749 INFO:teuthology.orchestra.run.smithi002.stdout: ceph-radosgw-2:17.2.0-188.g23b06583.el8.x86_64 2022-05-13T23:04:46.750 INFO:teuthology.orchestra.run.smithi002.stdout: ceph-selinux-2:17.2.0-188.g23b06583.el8.x86_64 2022-05-13T23:04:46.750 INFO:teuthology.orchestra.run.smithi002.stdout: ceph-test-2:17.2.0-188.g23b06583.el8.x86_64 2022-05-13T23:04:46.750 INFO:teuthology.orchestra.run.smithi002.stdout: ceph-volume-2:17.2.0-188.g23b06583.el8.noarch 2022-05-13T23:04:46.750 INFO:teuthology.orchestra.run.smithi002.stdout: cephadm-2:17.2.0-188.g23b06583.el8.noarch 2022-05-13T23:04:46.751 INFO:teuthology.orchestra.run.smithi002.stdout: cephfs-mirror-2:17.2.0-188.g23b06583.el8.x86_64 2022-05-13T23:04:46.751 INFO:teuthology.orchestra.run.smithi002.stdout: cephfs-top-2:17.2.0-188.g23b06583.el8.noarch 2022-05-13T23:04:46.751 INFO:teuthology.orchestra.run.smithi002.stdout: elfutils-libelf-devel-0.182-3.el8.x86_64 2022-05-13T23:04:46.752 INFO:teuthology.orchestra.run.smithi002.stdout: flex-2.6.1-9.el8.x86_64 2022-05-13T23:04:46.752 INFO:teuthology.orchestra.run.smithi002.stdout: fuse-2.9.7-12.el8.x86_64 2022-05-13T23:04:46.752 INFO:teuthology.orchestra.run.smithi002.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2022-05-13T23:04:46.753 INFO:teuthology.orchestra.run.smithi002.stdout: jq-1.5-12.el8.x86_64 2022-05-13T23:04:46.753 INFO:teuthology.orchestra.run.smithi002.stdout: keyutils-libs-devel-1.5.10-6.el8.x86_64 2022-05-13T23:04:46.755 INFO:teuthology.orchestra.run.smithi002.stdout: krb5-devel-1.18.2-8.3.el8_4.x86_64 2022-05-13T23:04:46.756 INFO:teuthology.orchestra.run.smithi002.stdout: libaio-devel-0.3.112-1.el8.x86_64 2022-05-13T23:04:46.756 INFO:teuthology.orchestra.run.smithi002.stdout: libcephfs-devel-2:17.2.0-188.g23b06583.el8.x86_64 2022-05-13T23:04:46.756 INFO:teuthology.orchestra.run.smithi002.stdout: libcephfs2-2:17.2.0-188.g23b06583.el8.x86_64 2022-05-13T23:04:46.756 INFO:teuthology.orchestra.run.smithi002.stdout: libcephsqlite-2:17.2.0-188.g23b06583.el8.x86_64 2022-05-13T23:04:46.757 INFO:teuthology.orchestra.run.smithi002.stdout: libcom_err-devel-1.45.6-1.el8.x86_64 2022-05-13T23:04:46.757 INFO:teuthology.orchestra.run.smithi002.stdout: libgfortran-8.4.1-1.el8.x86_64 2022-05-13T23:04:46.757 INFO:teuthology.orchestra.run.smithi002.stdout: liboath-2.6.2-3.el8.x86_64 2022-05-13T23:04:46.757 INFO:teuthology.orchestra.run.smithi002.stdout: libquadmath-8.4.1-1.el8.x86_64 2022-05-13T23:04:46.758 INFO:teuthology.orchestra.run.smithi002.stdout: librabbitmq-0.9.0-3.el8.x86_64 2022-05-13T23:04:46.758 INFO:teuthology.orchestra.run.smithi002.stdout: librados-devel-2:17.2.0-188.g23b06583.el8.x86_64 2022-05-13T23:04:46.758 INFO:teuthology.orchestra.run.smithi002.stdout: libradosstriper1-2:17.2.0-188.g23b06583.el8.x86_64 2022-05-13T23:04:46.758 INFO:teuthology.orchestra.run.smithi002.stdout: librdkafka-0.11.4-1.el8.x86_64 2022-05-13T23:04:46.758 INFO:teuthology.orchestra.run.smithi002.stdout: librgw2-2:17.2.0-188.g23b06583.el8.x86_64 2022-05-13T23:04:46.759 INFO:teuthology.orchestra.run.smithi002.stdout: libselinux-devel-2.9-5.el8.x86_64 2022-05-13T23:04:46.759 INFO:teuthology.orchestra.run.smithi002.stdout: libsepol-devel-2.9-2.el8.x86_64 2022-05-13T23:04:46.759 INFO:teuthology.orchestra.run.smithi002.stdout: libtool-2.4.6-25.el8.x86_64 2022-05-13T23:04:46.759 INFO:teuthology.orchestra.run.smithi002.stdout: libunwind-1.3.1-3.el8.x86_64 2022-05-13T23:04:46.760 INFO:teuthology.orchestra.run.smithi002.stdout: libuuid-devel-2.32.1-27.el8.x86_64 2022-05-13T23:04:46.760 INFO:teuthology.orchestra.run.smithi002.stdout: libverto-devel-0.3.0-5.el8.x86_64 2022-05-13T23:04:46.760 INFO:teuthology.orchestra.run.smithi002.stdout: lttng-ust-2.8.1-11.el8.x86_64 2022-05-13T23:04:46.760 INFO:teuthology.orchestra.run.smithi002.stdout: oniguruma-6.8.2-2.el8.x86_64 2022-05-13T23:04:46.760 INFO:teuthology.orchestra.run.smithi002.stdout: openblas-0.3.12-1.el8.x86_64 2022-05-13T23:04:46.761 INFO:teuthology.orchestra.run.smithi002.stdout: openblas-threads-0.3.12-1.el8.x86_64 2022-05-13T23:04:46.761 INFO:teuthology.orchestra.run.smithi002.stdout: openssl-devel-1:1.1.1g-15.el8_3.x86_64 2022-05-13T23:04:46.761 INFO:teuthology.orchestra.run.smithi002.stdout: pcre2-devel-10.32-2.el8.x86_64 2022-05-13T23:04:46.761 INFO:teuthology.orchestra.run.smithi002.stdout: pcre2-utf16-10.32-2.el8.x86_64 2022-05-13T23:04:46.762 INFO:teuthology.orchestra.run.smithi002.stdout: pcre2-utf32-10.32-2.el8.x86_64 2022-05-13T23:04:46.762 INFO:teuthology.orchestra.run.smithi002.stdout: python3-asyncssh-2.7.0-2.el8.noarch 2022-05-13T23:04:46.762 INFO:teuthology.orchestra.run.smithi002.stdout: python3-babel-2.5.1-5.el8.noarch 2022-05-13T23:04:46.762 INFO:teuthology.orchestra.run.smithi002.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2022-05-13T23:04:46.762 INFO:teuthology.orchestra.run.smithi002.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2022-05-13T23:04:46.763 INFO:teuthology.orchestra.run.smithi002.stdout: python3-cachetools-3.1.1-4.el8.noarch 2022-05-13T23:04:46.763 INFO:teuthology.orchestra.run.smithi002.stdout: python3-ceph-argparse-2:17.2.0-188.g23b06583.el8.x86_64 2022-05-13T23:04:46.763 INFO:teuthology.orchestra.run.smithi002.stdout: python3-ceph-common-2:17.2.0-188.g23b06583.el8.x86_64 2022-05-13T23:04:46.763 INFO:teuthology.orchestra.run.smithi002.stdout: python3-cephfs-2:17.2.0-188.g23b06583.el8.x86_64 2022-05-13T23:04:46.764 INFO:teuthology.orchestra.run.smithi002.stdout: python3-certifi-2018.10.15-7.el8.noarch 2022-05-13T23:04:46.764 INFO:teuthology.orchestra.run.smithi002.stdout: python3-cffi-1.11.5-5.el8.x86_64 2022-05-13T23:04:46.764 INFO:teuthology.orchestra.run.smithi002.stdout: python3-cheroot-8.5.2-1.el8.noarch 2022-05-13T23:04:46.764 INFO:teuthology.orchestra.run.smithi002.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2022-05-13T23:04:46.764 INFO:teuthology.orchestra.run.smithi002.stdout: python3-cryptography-3.2.1-4.el8.x86_64 2022-05-13T23:04:46.765 INFO:teuthology.orchestra.run.smithi002.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2022-05-13T23:04:46.765 INFO:teuthology.orchestra.run.smithi002.stdout: python3-jaraco-6.2-6.el8.noarch 2022-05-13T23:04:46.765 INFO:teuthology.orchestra.run.smithi002.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2022-05-13T23:04:46.765 INFO:teuthology.orchestra.run.smithi002.stdout: python3-jinja2-2.10.1-2.el8_0.noarch 2022-05-13T23:04:46.766 INFO:teuthology.orchestra.run.smithi002.stdout: python3-jsonpatch-1.21-2.el8.noarch 2022-05-13T23:04:46.766 INFO:teuthology.orchestra.run.smithi002.stdout: python3-jsonpointer-1.10-11.el8.noarch 2022-05-13T23:04:46.766 INFO:teuthology.orchestra.run.smithi002.stdout: python3-jwt-1.6.1-2.el8.noarch 2022-05-13T23:04:46.766 INFO:teuthology.orchestra.run.smithi002.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2022-05-13T23:04:46.766 INFO:teuthology.orchestra.run.smithi002.stdout: python3-logutils-0.3.5-11.el8.noarch 2022-05-13T23:04:46.767 INFO:teuthology.orchestra.run.smithi002.stdout: python3-mako-1.0.6-13.el8.noarch 2022-05-13T23:04:46.767 INFO:teuthology.orchestra.run.smithi002.stdout: python3-markupsafe-0.23-19.el8.x86_64 2022-05-13T23:04:46.767 INFO:teuthology.orchestra.run.smithi002.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2022-05-13T23:04:46.767 INFO:teuthology.orchestra.run.smithi002.stdout: python3-natsort-7.1.1-2.el8.noarch 2022-05-13T23:04:46.768 INFO:teuthology.orchestra.run.smithi002.stdout: python3-numpy-1:1.14.3-9.el8.x86_64 2022-05-13T23:04:46.768 INFO:teuthology.orchestra.run.smithi002.stdout: python3-numpy-f2py-1:1.14.3-9.el8.x86_64 2022-05-13T23:04:46.768 INFO:teuthology.orchestra.run.smithi002.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2022-05-13T23:04:46.768 INFO:teuthology.orchestra.run.smithi002.stdout: python3-pecan-1.3.2-9.el8.noarch 2022-05-13T23:04:46.768 INFO:teuthology.orchestra.run.smithi002.stdout: python3-portend-2.6-1.el8.noarch 2022-05-13T23:04:46.769 INFO:teuthology.orchestra.run.smithi002.stdout: python3-prettytable-0.7.2-14.el8.noarch 2022-05-13T23:04:46.769 INFO:teuthology.orchestra.run.smithi002.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2022-05-13T23:04:46.769 INFO:teuthology.orchestra.run.smithi002.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2022-05-13T23:04:46.769 INFO:teuthology.orchestra.run.smithi002.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2022-05-13T23:04:46.770 INFO:teuthology.orchestra.run.smithi002.stdout: python3-pycparser-2.14-14.el8.noarch 2022-05-13T23:04:46.770 INFO:teuthology.orchestra.run.smithi002.stdout: python3-pytz-2017.2-9.el8.noarch 2022-05-13T23:04:46.771 INFO:teuthology.orchestra.run.smithi002.stdout: python3-rados-2:17.2.0-188.g23b06583.el8.x86_64 2022-05-13T23:04:46.772 INFO:teuthology.orchestra.run.smithi002.stdout: python3-rbd-2:17.2.0-188.g23b06583.el8.x86_64 2022-05-13T23:04:46.772 INFO:teuthology.orchestra.run.smithi002.stdout: python3-repoze-lru-0.7-6.el8.noarch 2022-05-13T23:04:46.772 INFO:teuthology.orchestra.run.smithi002.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2022-05-13T23:04:46.772 INFO:teuthology.orchestra.run.smithi002.stdout: python3-rgw-2:17.2.0-188.g23b06583.el8.x86_64 2022-05-13T23:04:46.773 INFO:teuthology.orchestra.run.smithi002.stdout: python3-routes-2.4.1-12.el8.noarch 2022-05-13T23:04:46.773 INFO:teuthology.orchestra.run.smithi002.stdout: python3-rsa-4.8-1.el8.noarch 2022-05-13T23:04:46.773 INFO:teuthology.orchestra.run.smithi002.stdout: python3-scipy-1.0.0-20.module+el8.1.0+3334+5cb623d7.x86_64 2022-05-13T23:04:46.773 INFO:teuthology.orchestra.run.smithi002.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2022-05-13T23:04:46.774 INFO:teuthology.orchestra.run.smithi002.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2022-05-13T23:04:46.774 INFO:teuthology.orchestra.run.smithi002.stdout: python3-tempora-1.14.1-5.el8.noarch 2022-05-13T23:04:46.774 INFO:teuthology.orchestra.run.smithi002.stdout: python3-trustme-0.6.0-4.el8.noarch 2022-05-13T23:04:46.774 INFO:teuthology.orchestra.run.smithi002.stdout: python3-waitress-1.2.1-2.el8.1.noarch 2022-05-13T23:04:46.775 INFO:teuthology.orchestra.run.smithi002.stdout: python3-webob-1.8.5-1.el8.1.noarch 2022-05-13T23:04:46.775 INFO:teuthology.orchestra.run.smithi002.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2022-05-13T23:04:46.775 INFO:teuthology.orchestra.run.smithi002.stdout: python3-webtest-2.0.33-1.el8.noarch 2022-05-13T23:04:46.776 INFO:teuthology.orchestra.run.smithi002.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2022-05-13T23:04:46.776 INFO:teuthology.orchestra.run.smithi002.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2022-05-13T23:04:46.776 INFO:teuthology.orchestra.run.smithi002.stdout: rbd-fuse-2:17.2.0-188.g23b06583.el8.x86_64 2022-05-13T23:04:46.776 INFO:teuthology.orchestra.run.smithi002.stdout: rbd-mirror-2:17.2.0-188.g23b06583.el8.x86_64 2022-05-13T23:04:46.777 INFO:teuthology.orchestra.run.smithi002.stdout: rbd-nbd-2:17.2.0-188.g23b06583.el8.x86_64 2022-05-13T23:04:46.777 INFO:teuthology.orchestra.run.smithi002.stdout: socat-1.7.3.3-2.el8.x86_64 2022-05-13T23:04:46.777 INFO:teuthology.orchestra.run.smithi002.stdout: thrift-0.13.0-2.el8.x86_64 2022-05-13T23:04:46.777 INFO:teuthology.orchestra.run.smithi002.stdout: xfsprogs-devel-5.0.0-8.el8.x86_64 2022-05-13T23:04:46.778 INFO:teuthology.orchestra.run.smithi002.stdout: xmlstarlet-1.6.1-11.el8.x86_64 2022-05-13T23:04:46.778 INFO:teuthology.orchestra.run.smithi002.stdout: zlib-devel-1.2.11-17.el8.x86_64 2022-05-13T23:04:46.779 INFO:teuthology.orchestra.run.smithi002.stdout: 2022-05-13T23:04:46.779 INFO:teuthology.orchestra.run.smithi002.stdout:Complete! 2022-05-13T23:04:47.156 DEBUG:teuthology.parallel:result is None 2022-05-13T23:04:47.158 DEBUG:teuthology.orchestra.run.smithi002:> sudo lsb_release -is 2022-05-13T23:04:47.315 INFO:teuthology.orchestra.run.smithi002.stdout:RedHatEnterprise 2022-05-13T23:04:47.319 DEBUG:teuthology.misc:System to be installed: RedHatEnterprise 2022-05-13T23:04:47.320 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=23b0658391ce94ca40799469eec00c28b298816c 2022-05-13T23:04:47.539 DEBUG:teuthology.orchestra.run.smithi002:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2022-05-13T23:04:47.599 INFO:teuthology.orchestra.run.smithi002.stdout:17.2.0-188.g23b06583.el8 2022-05-13T23:04:47.600 INFO:teuthology.packaging:The installed version of ceph is 17.2.0-188.g23b06583.el8 2022-05-13T23:04:47.601 INFO:teuthology.task.install:The correct ceph version 17.2.0-188.g23b06583 is installed. 2022-05-13T23:04:47.602 DEBUG:teuthology.orchestra.run.smithi047:> sudo lsb_release -is 2022-05-13T23:04:47.670 INFO:teuthology.orchestra.run.smithi047.stdout:RedHatEnterprise 2022-05-13T23:04:47.671 DEBUG:teuthology.misc:System to be installed: RedHatEnterprise 2022-05-13T23:04:47.671 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=23b0658391ce94ca40799469eec00c28b298816c 2022-05-13T23:04:47.856 DEBUG:teuthology.orchestra.run.smithi047:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2022-05-13T23:04:47.881 INFO:teuthology.orchestra.run.smithi047.stdout:17.2.0-188.g23b06583.el8 2022-05-13T23:04:47.882 INFO:teuthology.packaging:The installed version of ceph is 17.2.0-188.g23b06583.el8 2022-05-13T23:04:47.883 INFO:teuthology.task.install:The correct ceph version 17.2.0-188.g23b06583 is installed. 2022-05-13T23:04:47.883 DEBUG:teuthology.orchestra.run.smithi115:> sudo lsb_release -is 2022-05-13T23:04:47.944 INFO:teuthology.orchestra.run.smithi115.stdout:RedHatEnterprise 2022-05-13T23:04:47.946 DEBUG:teuthology.misc:System to be installed: RedHatEnterprise 2022-05-13T23:04:47.946 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=23b0658391ce94ca40799469eec00c28b298816c 2022-05-13T23:04:48.116 DEBUG:teuthology.orchestra.run.smithi115:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2022-05-13T23:04:48.140 INFO:teuthology.orchestra.run.smithi115.stdout:17.2.0-188.g23b06583.el8 2022-05-13T23:04:48.140 INFO:teuthology.packaging:The installed version of ceph is 17.2.0-188.g23b06583.el8 2022-05-13T23:04:48.141 INFO:teuthology.task.install:The correct ceph version 17.2.0-188.g23b06583 is installed. 2022-05-13T23:04:48.141 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2022-05-13T23:04:48.144 DEBUG:teuthology.orchestra.run.smithi002:> set -ex 2022-05-13T23:04:48.145 DEBUG:teuthology.orchestra.run.smithi002:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2022-05-13T23:04:48.240 DEBUG:teuthology.orchestra.run.smithi047:> set -ex 2022-05-13T23:04:48.240 DEBUG:teuthology.orchestra.run.smithi047:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2022-05-13T23:04:48.274 DEBUG:teuthology.orchestra.run.smithi115:> set -ex 2022-05-13T23:04:48.275 DEBUG:teuthology.orchestra.run.smithi115:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2022-05-13T23:04:48.305 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2022-05-13T23:04:48.307 DEBUG:teuthology.orchestra.run.smithi002:> set -ex 2022-05-13T23:04:48.307 DEBUG:teuthology.orchestra.run.smithi002:> sudo dd of=/usr/bin/daemon-helper 2022-05-13T23:04:48.387 DEBUG:teuthology.orchestra.run.smithi002:> sudo chmod a=rx -- /usr/bin/daemon-helper 2022-05-13T23:04:48.508 DEBUG:teuthology.orchestra.run.smithi047:> set -ex 2022-05-13T23:04:48.509 DEBUG:teuthology.orchestra.run.smithi047:> sudo dd of=/usr/bin/daemon-helper 2022-05-13T23:04:48.542 DEBUG:teuthology.orchestra.run.smithi047:> sudo chmod a=rx -- /usr/bin/daemon-helper 2022-05-13T23:04:48.614 DEBUG:teuthology.orchestra.run.smithi115:> set -ex 2022-05-13T23:04:48.615 DEBUG:teuthology.orchestra.run.smithi115:> sudo dd of=/usr/bin/daemon-helper 2022-05-13T23:04:48.647 DEBUG:teuthology.orchestra.run.smithi115:> sudo chmod a=rx -- /usr/bin/daemon-helper 2022-05-13T23:04:48.714 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2022-05-13T23:04:48.715 DEBUG:teuthology.orchestra.run.smithi002:> set -ex 2022-05-13T23:04:48.715 DEBUG:teuthology.orchestra.run.smithi002:> sudo dd of=/usr/bin/adjust-ulimits 2022-05-13T23:04:48.795 DEBUG:teuthology.orchestra.run.smithi002:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2022-05-13T23:04:48.915 DEBUG:teuthology.orchestra.run.smithi047:> set -ex 2022-05-13T23:04:48.916 DEBUG:teuthology.orchestra.run.smithi047:> sudo dd of=/usr/bin/adjust-ulimits 2022-05-13T23:04:48.947 DEBUG:teuthology.orchestra.run.smithi047:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2022-05-13T23:04:49.017 DEBUG:teuthology.orchestra.run.smithi115:> set -ex 2022-05-13T23:04:49.018 DEBUG:teuthology.orchestra.run.smithi115:> sudo dd of=/usr/bin/adjust-ulimits 2022-05-13T23:04:49.046 DEBUG:teuthology.orchestra.run.smithi115:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2022-05-13T23:04:49.115 INFO:teuthology.run_tasks:Running task cephadm... 2022-05-13T23:04:49.221 INFO:tasks.cephadm:Config: {'roleless': False, 'cephfs': {'max_mds': 5, 'session_timeout': 300, 'standby_replay': True}, 'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'ms die on skipped message': False}, 'mds': {'debug mds': 20, 'debug ms': 1, 'mds bal fragment size max': 10000, 'mds bal merge size': 5, 'mds bal split bits': 3, 'mds bal split size': 100, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180, 'osd shutdown pgref assert': True, 'osd_max_omap_entries_per_request': 10}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'overall HEALTH_', '\\(FS_DEGRADED\\)', '\\(MDS_FAILED\\)', '\\(MDS_DEGRADED\\)', '\\(FS_WITH_FAILED_MDS\\)', '\\(MDS_DAMAGE\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(FS_INLINE_DATA_DEPRECATED\\)', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding'], 'log-whitelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'sha1': '23b0658391ce94ca40799469eec00c28b298816c'} 2022-05-13T23:04:49.222 INFO:tasks.cephadm:Cluster image is quay.ceph.io/ceph-ci/ceph:23b0658391ce94ca40799469eec00c28b298816c 2022-05-13T23:04:49.223 INFO:tasks.cephadm:Cluster fsid is 16cf5f88-d311-11ec-be40-001a4aab830c 2022-05-13T23:04:49.224 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2022-05-13T23:04:49.224 INFO:tasks.cephadm:Monitor IPs: {'mon.a': '172.21.15.2', 'mon.b': '172.21.15.47', 'mon.c': '172.21.15.115'} 2022-05-13T23:04:49.225 INFO:tasks.cephadm:First mon is mon.a on smithi002 2022-05-13T23:04:49.225 INFO:tasks.cephadm:First mgr is x 2022-05-13T23:04:49.225 INFO:tasks.cephadm:Normalizing hostnames... 2022-05-13T23:04:49.226 DEBUG:teuthology.orchestra.run.smithi002:> sudo hostname $(hostname -s) 2022-05-13T23:04:49.311 DEBUG:teuthology.orchestra.run.smithi047:> sudo hostname $(hostname -s) 2022-05-13T23:04:49.345 DEBUG:teuthology.orchestra.run.smithi115:> sudo hostname $(hostname -s) 2022-05-13T23:04:49.375 INFO:tasks.cephadm:Downloading cephadm (repo https://github.com/ceph/ceph-ci.git ref 23b0658391ce94ca40799469eec00c28b298816c)... 2022-05-13T23:04:49.376 DEBUG:teuthology.orchestra.run.smithi002:> curl --silent https://raw.githubusercontent.com/ceph/ceph-ci/23b0658391ce94ca40799469eec00c28b298816c/src/cephadm/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2022-05-13T23:04:49.874 INFO:teuthology.orchestra.run.smithi002.stdout:-rw-rw-r--. 1 ubuntu ubuntu 340966 May 13 23:04 /home/ubuntu/cephtest/cephadm 2022-05-13T23:04:49.875 DEBUG:teuthology.orchestra.run.smithi047:> curl --silent https://raw.githubusercontent.com/ceph/ceph-ci/23b0658391ce94ca40799469eec00c28b298816c/src/cephadm/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2022-05-13T23:04:50.197 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-rw-r--. 1 ubuntu ubuntu 340966 May 13 23:04 /home/ubuntu/cephtest/cephadm 2022-05-13T23:04:50.198 DEBUG:teuthology.orchestra.run.smithi115:> curl --silent https://raw.githubusercontent.com/ceph/ceph-ci/23b0658391ce94ca40799469eec00c28b298816c/src/cephadm/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2022-05-13T23:04:50.511 INFO:teuthology.orchestra.run.smithi115.stdout:-rw-rw-r--. 1 ubuntu ubuntu 340966 May 13 23:04 /home/ubuntu/cephtest/cephadm 2022-05-13T23:04:50.511 DEBUG:teuthology.orchestra.run.smithi002:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2022-05-13T23:04:50.567 DEBUG:teuthology.orchestra.run.smithi047:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2022-05-13T23:04:50.587 DEBUG:teuthology.orchestra.run.smithi115:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2022-05-13T23:04:50.621 INFO:tasks.cephadm:Pulling image quay.ceph.io/ceph-ci/ceph:23b0658391ce94ca40799469eec00c28b298816c on all hosts... 2022-05-13T23:04:50.622 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:23b0658391ce94ca40799469eec00c28b298816c pull 2022-05-13T23:04:50.627 DEBUG:teuthology.orchestra.run.smithi047:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:23b0658391ce94ca40799469eec00c28b298816c pull 2022-05-13T23:04:50.630 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:23b0658391ce94ca40799469eec00c28b298816c pull 2022-05-13T23:04:52.183 INFO:teuthology.orchestra.run.smithi115.stderr:Pulling container image quay.ceph.io/ceph-ci/ceph:23b0658391ce94ca40799469eec00c28b298816c... 2022-05-13T23:04:52.415 INFO:teuthology.orchestra.run.smithi047.stderr:Pulling container image quay.ceph.io/ceph-ci/ceph:23b0658391ce94ca40799469eec00c28b298816c... 2022-05-13T23:04:53.848 INFO:teuthology.orchestra.run.smithi002.stderr:Pulling container image quay.ceph.io/ceph-ci/ceph:23b0658391ce94ca40799469eec00c28b298816c... 2022-05-13T23:05:29.880 INFO:teuthology.orchestra.run.smithi047.stdout:{ 2022-05-13T23:05:29.881 INFO:teuthology.orchestra.run.smithi047.stdout: "ceph_version": "ceph version 17.2.0-188-g23b06583 (23b0658391ce94ca40799469eec00c28b298816c) quincy (stable)", 2022-05-13T23:05:29.881 INFO:teuthology.orchestra.run.smithi047.stdout: "image_id": "efd555832784ba13b89797ad456a9720d13b97cb4a26d42d0a87e1565c9985c8", 2022-05-13T23:05:29.881 INFO:teuthology.orchestra.run.smithi047.stdout: "repo_digests": [ 2022-05-13T23:05:29.882 INFO:teuthology.orchestra.run.smithi047.stdout: "quay.ceph.io/ceph-ci/ceph@sha256:74192c9c61986c7cb5d84694d42bd0ae576dc263eba7dad9caf38503d4bf3ff2" 2022-05-13T23:05:29.882 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2022-05-13T23:05:29.882 INFO:teuthology.orchestra.run.smithi047.stdout:} 2022-05-13T23:05:30.655 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2022-05-13T23:05:30.655 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph_version": "ceph version 17.2.0-188-g23b06583 (23b0658391ce94ca40799469eec00c28b298816c) quincy (stable)", 2022-05-13T23:05:30.656 INFO:teuthology.orchestra.run.smithi115.stdout: "image_id": "efd555832784ba13b89797ad456a9720d13b97cb4a26d42d0a87e1565c9985c8", 2022-05-13T23:05:30.656 INFO:teuthology.orchestra.run.smithi115.stdout: "repo_digests": [ 2022-05-13T23:05:30.656 INFO:teuthology.orchestra.run.smithi115.stdout: "quay.ceph.io/ceph-ci/ceph@sha256:74192c9c61986c7cb5d84694d42bd0ae576dc263eba7dad9caf38503d4bf3ff2" 2022-05-13T23:05:30.656 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2022-05-13T23:05:30.656 INFO:teuthology.orchestra.run.smithi115.stdout:} 2022-05-13T23:05:48.706 INFO:teuthology.orchestra.run.smithi002.stdout:{ 2022-05-13T23:05:48.707 INFO:teuthology.orchestra.run.smithi002.stdout: "ceph_version": "ceph version 17.2.0-188-g23b06583 (23b0658391ce94ca40799469eec00c28b298816c) quincy (stable)", 2022-05-13T23:05:48.707 INFO:teuthology.orchestra.run.smithi002.stdout: "image_id": "efd555832784ba13b89797ad456a9720d13b97cb4a26d42d0a87e1565c9985c8", 2022-05-13T23:05:48.707 INFO:teuthology.orchestra.run.smithi002.stdout: "repo_digests": [ 2022-05-13T23:05:48.707 INFO:teuthology.orchestra.run.smithi002.stdout: "quay.ceph.io/ceph-ci/ceph@sha256:74192c9c61986c7cb5d84694d42bd0ae576dc263eba7dad9caf38503d4bf3ff2" 2022-05-13T23:05:48.708 INFO:teuthology.orchestra.run.smithi002.stdout: ] 2022-05-13T23:05:48.708 INFO:teuthology.orchestra.run.smithi002.stdout:} 2022-05-13T23:05:48.732 DEBUG:teuthology.orchestra.run.smithi002:> sudo mkdir -p /etc/ceph 2022-05-13T23:05:48.814 DEBUG:teuthology.orchestra.run.smithi047:> sudo mkdir -p /etc/ceph 2022-05-13T23:05:48.851 DEBUG:teuthology.orchestra.run.smithi115:> sudo mkdir -p /etc/ceph 2022-05-13T23:05:48.887 DEBUG:teuthology.orchestra.run.smithi002:> sudo chmod 777 /etc/ceph 2022-05-13T23:05:48.965 DEBUG:teuthology.orchestra.run.smithi047:> sudo chmod 777 /etc/ceph 2022-05-13T23:05:48.995 DEBUG:teuthology.orchestra.run.smithi115:> sudo chmod 777 /etc/ceph 2022-05-13T23:05:49.025 INFO:tasks.cephadm:Writing seed config... 2022-05-13T23:05:49.027 INFO:tasks.cephadm: override: [client] client mount timeout = 600 2022-05-13T23:05:49.027 INFO:tasks.cephadm: override: [client] debug client = 20 2022-05-13T23:05:49.027 INFO:tasks.cephadm: override: [client] debug ms = 1 2022-05-13T23:05:49.028 INFO:tasks.cephadm: override: [client] rados mon op timeout = 900 2022-05-13T23:05:49.028 INFO:tasks.cephadm: override: [client] rados osd op timeout = 900 2022-05-13T23:05:49.028 INFO:tasks.cephadm: override: [global] ms die on skipped message = False 2022-05-13T23:05:49.029 INFO:tasks.cephadm: override: [mds] debug mds = 20 2022-05-13T23:05:49.029 INFO:tasks.cephadm: override: [mds] debug ms = 1 2022-05-13T23:05:49.029 INFO:tasks.cephadm: override: [mds] mds bal fragment size max = 10000 2022-05-13T23:05:49.030 INFO:tasks.cephadm: override: [mds] mds bal merge size = 5 2022-05-13T23:05:49.030 INFO:tasks.cephadm: override: [mds] mds bal split bits = 3 2022-05-13T23:05:49.030 INFO:tasks.cephadm: override: [mds] mds bal split size = 100 2022-05-13T23:05:49.031 INFO:tasks.cephadm: override: [mds] mds debug frag = True 2022-05-13T23:05:49.031 INFO:tasks.cephadm: override: [mds] mds debug scatterstat = True 2022-05-13T23:05:49.031 INFO:tasks.cephadm: override: [mds] mds op complaint time = 180 2022-05-13T23:05:49.032 INFO:tasks.cephadm: override: [mds] mds verify scatter = True 2022-05-13T23:05:49.032 INFO:tasks.cephadm: override: [mds] osd op complaint time = 180 2022-05-13T23:05:49.032 INFO:tasks.cephadm: override: [mds] rados mon op timeout = 900 2022-05-13T23:05:49.033 INFO:tasks.cephadm: override: [mds] rados osd op timeout = 900 2022-05-13T23:05:49.033 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2022-05-13T23:05:49.033 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2022-05-13T23:05:49.034 INFO:tasks.cephadm: override: [mon] debug mon = 20 2022-05-13T23:05:49.034 INFO:tasks.cephadm: override: [mon] debug ms = 1 2022-05-13T23:05:49.034 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2022-05-13T23:05:49.035 INFO:tasks.cephadm: override: [mon] mon op complaint time = 120 2022-05-13T23:05:49.035 INFO:tasks.cephadm: override: [osd] bdev async discard = True 2022-05-13T23:05:49.035 INFO:tasks.cephadm: override: [osd] bdev enable discard = True 2022-05-13T23:05:49.035 INFO:tasks.cephadm: override: [osd] bluestore allocator = bitmap 2022-05-13T23:05:49.036 INFO:tasks.cephadm: override: [osd] bluestore block size = 96636764160 2022-05-13T23:05:49.036 INFO:tasks.cephadm: override: [osd] bluestore fsck on mount = True 2022-05-13T23:05:49.036 INFO:tasks.cephadm: override: [osd] debug bluefs = 1/20 2022-05-13T23:05:49.037 INFO:tasks.cephadm: override: [osd] debug bluestore = 1/20 2022-05-13T23:05:49.037 INFO:tasks.cephadm: override: [osd] debug ms = 1 2022-05-13T23:05:49.037 INFO:tasks.cephadm: override: [osd] debug osd = 20 2022-05-13T23:05:49.038 INFO:tasks.cephadm: override: [osd] debug rocksdb = 4/10 2022-05-13T23:05:49.038 INFO:tasks.cephadm: override: [osd] mon osd backfillfull_ratio = 0.85 2022-05-13T23:05:49.038 INFO:tasks.cephadm: override: [osd] mon osd full ratio = 0.9 2022-05-13T23:05:49.038 INFO:tasks.cephadm: override: [osd] mon osd nearfull ratio = 0.8 2022-05-13T23:05:49.039 INFO:tasks.cephadm: override: [osd] osd failsafe full ratio = 0.95 2022-05-13T23:05:49.039 INFO:tasks.cephadm: override: [osd] osd objectstore = bluestore 2022-05-13T23:05:49.039 INFO:tasks.cephadm: override: [osd] osd op complaint time = 180 2022-05-13T23:05:49.040 INFO:tasks.cephadm: override: [osd] osd shutdown pgref assert = True 2022-05-13T23:05:49.040 INFO:tasks.cephadm: override: [osd] osd_max_omap_entries_per_request = 10 2022-05-13T23:05:49.041 DEBUG:teuthology.orchestra.run.smithi002:> set -ex 2022-05-13T23:05:49.041 DEBUG:teuthology.orchestra.run.smithi002:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2022-05-13T23:05:49.091 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = 16cf5f88-d311-11ec-be40-001a4aab830c ms die on skipped message = False [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = True bdev debug aio = true osd sloppy crc = true bdev async discard = True bdev enable discard = True bluestore allocator = bitmap bluestore block size = 96636764160 bluestore fsck on mount = True debug bluefs = 1/20 debug bluestore = 1/20 debug ms = 1 debug osd = 20 debug rocksdb = 4/10 mon osd backfillfull_ratio = 0.85 mon osd full ratio = 0.9 mon osd nearfull ratio = 0.8 osd failsafe full ratio = 0.95 osd objectstore = bluestore osd op complaint time = 180 osd_max_omap_entries_per_request = 10 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 mon op complaint time = 120 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true [client] client mount timeout = 600 debug client = 20 debug ms = 1 rados mon op timeout = 900 rados osd op timeout = 900 [mds] debug mds = 20 debug ms = 1 mds bal fragment size max = 10000 mds bal merge size = 5 mds bal split bits = 3 mds bal split size = 100 mds debug frag = True mds debug scatterstat = True mds op complaint time = 180 mds verify scatter = True osd op complaint time = 180 rados mon op timeout = 900 rados osd op timeout = 900 2022-05-13T23:05:49.092 DEBUG:teuthology.orchestra.run.smithi002:mon.a> sudo journalctl -f -n 0 -u ceph-16cf5f88-d311-11ec-be40-001a4aab830c@mon.a.service 2022-05-13T23:05:49.137 DEBUG:teuthology.orchestra.run.smithi002:mgr.x> sudo journalctl -f -n 0 -u ceph-16cf5f88-d311-11ec-be40-001a4aab830c@mgr.x.service 2022-05-13T23:05:49.182 INFO:tasks.cephadm:Bootstrapping... 2022-05-13T23:05:49.183 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:23b0658391ce94ca40799469eec00c28b298816c -v bootstrap --fsid 16cf5f88-d311-11ec-be40-001a4aab830c --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id x --orphan-initial-daemons --skip-monitoring-stack --mon-ip 172.21.15.2 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2022-05-13T23:05:49.220 INFO:journalctl@ceph.mon.a.smithi002.stdout:-- Logs begin at Fri 2022-05-13 22:48:11 UTC. -- 2022-05-13T23:05:49.260 INFO:journalctl@ceph.mgr.x.smithi002.stdout:-- Logs begin at Fri 2022-05-13 22:48:11 UTC. -- 2022-05-13T23:05:49.668 INFO:teuthology.orchestra.run.smithi002.stderr:Specifying an fsid for your cluster offers no advantages and may increase the likelihood of fsid conflicts. 2022-05-13T23:05:49.672 INFO:teuthology.orchestra.run.smithi002.stdout:Verifying podman|docker is present... 2022-05-13T23:05:49.866 INFO:teuthology.orchestra.run.smithi002.stdout:Verifying lvm2 is present... 2022-05-13T23:05:49.925 INFO:teuthology.orchestra.run.smithi002.stdout:Verifying time synchronization is in place... 2022-05-13T23:05:49.941 INFO:teuthology.orchestra.run.smithi002.stdout:Unit chronyd.service is enabled and running 2022-05-13T23:05:49.941 INFO:teuthology.orchestra.run.smithi002.stdout:Repeating the final host check... 2022-05-13T23:05:50.128 INFO:teuthology.orchestra.run.smithi002.stdout:podman (/bin/podman) version 3.2.3 is present 2022-05-13T23:05:50.129 INFO:teuthology.orchestra.run.smithi002.stdout:systemctl is present 2022-05-13T23:05:50.129 INFO:teuthology.orchestra.run.smithi002.stdout:lvcreate is present 2022-05-13T23:05:50.204 INFO:teuthology.orchestra.run.smithi002.stdout:Unit chronyd.service is enabled and running 2022-05-13T23:05:50.204 INFO:teuthology.orchestra.run.smithi002.stdout:Host looks OK 2022-05-13T23:05:50.212 INFO:teuthology.orchestra.run.smithi002.stdout:Cluster fsid: 16cf5f88-d311-11ec-be40-001a4aab830c 2022-05-13T23:05:50.213 INFO:teuthology.orchestra.run.smithi002.stdout:Verifying IP 172.21.15.2 port 3300 ... 2022-05-13T23:05:50.213 INFO:teuthology.orchestra.run.smithi002.stdout:Verifying IP 172.21.15.2 port 6789 ... 2022-05-13T23:05:50.243 INFO:teuthology.orchestra.run.smithi002.stdout:Mon IP `172.21.15.2` is in CIDR network `172.21.0.0/20` 2022-05-13T23:05:50.244 INFO:teuthology.orchestra.run.smithi002.stdout:Mon IP `172.21.15.2` is in CIDR network `172.21.0.0/20` 2022-05-13T23:05:50.244 INFO:teuthology.orchestra.run.smithi002.stdout:Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2022-05-13T23:05:50.245 INFO:teuthology.orchestra.run.smithi002.stdout:Pulling container image quay.ceph.io/ceph-ci/ceph:23b0658391ce94ca40799469eec00c28b298816c... 2022-05-13T23:05:51.927 INFO:teuthology.orchestra.run.smithi002.stdout:Ceph version: ceph version 17.2.0-188-g23b06583 (23b0658391ce94ca40799469eec00c28b298816c) quincy (stable) 2022-05-13T23:05:51.927 INFO:teuthology.orchestra.run.smithi002.stdout:Extracting ceph user uid/gid from container image... 2022-05-13T23:05:52.819 INFO:teuthology.orchestra.run.smithi002.stdout:Creating initial keys... 2022-05-13T23:05:55.601 INFO:teuthology.orchestra.run.smithi002.stdout:Creating initial monmap... 2022-05-13T23:05:56.501 INFO:teuthology.orchestra.run.smithi002.stdout:Creating mon... 2022-05-13T23:05:59.573 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:05:59 smithi002 systemd[1]: Starting Ceph mon.a for 16cf5f88-d311-11ec-be40-001a4aab830c... 2022-05-13T23:06:00.821 INFO:teuthology.orchestra.run.smithi002.stdout:Waiting for mon to start... 2022-05-13T23:06:00.821 INFO:teuthology.orchestra.run.smithi002.stdout:Waiting for mon... 2022-05-13T23:06:01.072 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:06:00 smithi002 bash[70770]: 35c190f21c836335c210f9c2140d742c686e20aa3219fd31af9d23e685bbb16a 2022-05-13T23:06:01.073 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:06:00 smithi002 systemd[1]: Started Ceph mon.a for 16cf5f88-d311-11ec-be40-001a4aab830c. 2022-05-13T23:06:01.073 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:06:00 smithi002 ceph-mon[70982]: mkfs 16cf5f88-d311-11ec-be40-001a4aab830c 2022-05-13T23:06:01.073 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:06:00 smithi002 ceph-mon[70982]: mon.a is new leader, mons a in quorum (ranks 0) 2022-05-13T23:06:01.861 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:06:01 smithi002 ceph-mon[70982]: mon.a is new leader, mons a in quorum (ranks 0) 2022-05-13T23:06:01.862 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:06:01 smithi002 ceph-mon[70982]: monmap e1: 1 mons at {a=[v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0]} 2022-05-13T23:06:01.862 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:06:01 smithi002 ceph-mon[70982]: fsmap 2022-05-13T23:06:01.862 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:06:01 smithi002 ceph-mon[70982]: osdmap e1: 0 total, 0 up, 0 in 2022-05-13T23:06:01.862 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:06:01 smithi002 ceph-mon[70982]: mgrmap e1: no daemons active 2022-05-13T23:06:01.863 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:06:01 smithi002 ceph-mon[70982]: from='client.? 172.21.15.2:0/3947851125' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2022-05-13T23:06:02.123 INFO:teuthology.orchestra.run.smithi002.stdout:mon is available 2022-05-13T23:06:02.123 INFO:teuthology.orchestra.run.smithi002.stdout:Assimilating anything we can from ceph.conf... 2022-05-13T23:06:03.323 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:06:02 smithi002 ceph-mon[70982]: from='client.? 172.21.15.2:0/770579647' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2022-05-13T23:06:03.323 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:06:02 smithi002 ceph-mon[70982]: from='client.? 172.21.15.2:0/770579647' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2022-05-13T23:06:03.420 INFO:teuthology.orchestra.run.smithi002.stdout:Generating new minimal ceph.conf... 2022-05-13T23:06:04.460 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:06:04 smithi002 ceph-mon[70982]: from='client.? 172.21.15.2:0/1152659999' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-13T23:06:04.717 INFO:teuthology.orchestra.run.smithi002.stdout:Restarting the monitor... 2022-05-13T23:06:05.072 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:06:04 smithi002 systemd[1]: Stopping Ceph mon.a for 16cf5f88-d311-11ec-be40-001a4aab830c... 2022-05-13T23:06:05.073 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:06:04 smithi002 bash[71432]: Error: no container with name or ID "ceph-16cf5f88-d311-11ec-be40-001a4aab830c-mon.a" found: no such container 2022-05-13T23:06:05.572 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:06:05 smithi002 conmon[70958]: 2022-05-13T23:06:05.169+0000 7f1034de1700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2022-05-13T23:06:05.573 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:06:05 smithi002 conmon[70958]: 2022-05-13T23:06:05.169+0000 7f1034de1700 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2022-05-13T23:06:07.989 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:06:07 smithi002 bash[71432]: ceph-16cf5f88-d311-11ec-be40-001a4aab830c-mon-a 2022-05-13T23:06:08.334 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:06:07 smithi002 bash[71432]: Error: no container with name or ID "ceph-16cf5f88-d311-11ec-be40-001a4aab830c-mon.a" found: no such container 2022-05-13T23:06:08.385 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:06:08 smithi002 systemd[1]: ceph-16cf5f88-d311-11ec-be40-001a4aab830c@mon.a.service: Succeeded. 2022-05-13T23:06:08.387 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:06:08 smithi002 systemd[1]: Stopped Ceph mon.a for 16cf5f88-d311-11ec-be40-001a4aab830c. 2022-05-13T23:06:08.388 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:06:08 smithi002 systemd[1]: Starting Ceph mon.a for 16cf5f88-d311-11ec-be40-001a4aab830c... 2022-05-13T23:06:09.175 INFO:teuthology.orchestra.run.smithi002.stdout:Setting mon public_network to 172.21.0.0/20 2022-05-13T23:06:09.497 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:06:09 smithi002 ceph-mon[71816]: set uid:gid to 167:167 (ceph:ceph) 2022-05-13T23:06:09.498 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:06:09 smithi002 ceph-mon[71816]: ceph version 17.2.0-188-g23b06583 (23b0658391ce94ca40799469eec00c28b298816c) quincy (stable), process ceph-mon, pid 7 2022-05-13T23:06:09.498 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:06:09 smithi002 ceph-mon[71816]: pidfile_write: ignore empty --pid-file 2022-05-13T23:06:09.499 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:06:09 smithi002 ceph-mon[71816]: load: jerasure load: lrc 2022-05-13T23:06:09.499 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:06:09 smithi002 ceph-mon[71816]: rocksdb: RocksDB version: 6.15.5 2022-05-13T23:06:09.499 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:06:09 smithi002 ceph-mon[71816]: rocksdb: Git sha rocksdb_build_git_sha:@0@ 2022-05-13T23:06:09.499 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:06:09 smithi002 ceph-mon[71816]: rocksdb: Compile date May 12 2022 2022-05-13T23:06:09.499 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:06:09 smithi002 ceph-mon[71816]: rocksdb: DB SUMMARY 2022-05-13T23:06:09.500 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:06:09 smithi002 ceph-mon[71816]: rocksdb: DB Session ID: LL68RT1PL5QAV7MZYVZD 2022-05-13T23:06:09.500 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:06:09 smithi002 ceph-mon[71816]: rocksdb: CURRENT file: CURRENT 2022-05-13T23:06:09.500 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:06:09 smithi002 ceph-mon[71816]: rocksdb: IDENTITY file: IDENTITY 2022-05-13T23:06:09.500 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:06:09 smithi002 ceph-mon[71816]: rocksdb: MANIFEST file: MANIFEST-000009 size: 131 Bytes 2022-05-13T23:06:09.501 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:06:09 smithi002 ceph-mon[71816]: rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000008.sst 2022-05-13T23:06:09.501 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:06:09 smithi002 ceph-mon[71816]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000010.log size: 96465 ; 2022-05-13T23:06:09.501 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:06:09 smithi002 ceph-mon[71816]: rocksdb: Options.error_if_exists: 0 2022-05-13T23:06:09.501 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:06:09 smithi002 ceph-mon[71816]: rocksdb: Options.create_if_missing: 0 2022-05-13T23:06:09.502 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:06:09 smithi002 ceph-mon[71816]: rocksdb: Options.paranoid_checks: 1 2022-05-13T23:06:09.502 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:06:09 smithi002 ceph-mon[71816]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2022-05-13T23:06:09.502 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:06:09 smithi002 ceph-mon[71816]: rocksdb: Options.env: 0x5610a24c7860 2022-05-13T23:06:09.503 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:06:09 smithi002 ceph-mon[71816]: rocksdb: Options.fs: Posix File System 2022-05-13T23:06:09.503 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:06:09 smithi002 ceph-mon[71816]: rocksdb: Options.info_log: 0x5610a3fe7dc0 2022-05-13T23:06:09.503 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:06:09 smithi002 ceph-mon[71816]: rocksdb: Options.max_file_opening_threads: 16 2022-05-13T23:06:09.503 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:06:09 smithi002 ceph-mon[71816]: rocksdb: Options.statistics: (nil) 2022-05-13T23:06:09.504 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:06:09 smithi002 ceph-mon[71816]: rocksdb: Options.use_fsync: 0 2022-05-13T23:06:09.504 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:06:09 smithi002 ceph-mon[71816]: rocksdb: Options.max_log_file_size: 0 2022-05-13T23:06:09.505 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:06:09 smithi002 ceph-mon[71816]: rocksdb: Options.max_manifest_file_size: 1073741824 2022-05-13T23:06:09.505 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:06:09 smithi002 ceph-mon[71816]: rocksdb: Options.log_file_time_to_roll: 0 2022-05-13T23:06:09.505 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:06:09 smithi002 ceph-mon[71816]: rocksdb: Options.keep_log_file_num: 1000 2022-05-13T23:06:09.505 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:06:09 smithi002 ceph-mon[71816]: rocksdb: Options.recycle_log_file_num: 0 2022-05-13T23:06:09.505 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:06:09 smithi002 ceph-mon[71816]: rocksdb: Options.allow_fallocate: 1 2022-05-13T23:06:09.506 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:06:09 smithi002 ceph-mon[71816]: rocksdb: Options.allow_mmap_reads: 0 2022-05-13T23:06:09.506 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:06:09 smithi002 ceph-mon[71816]: rocksdb: Options.allow_mmap_writes: 0 2022-05-13T23:06:09.506 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:06:09 smithi002 ceph-mon[71816]: rocksdb: Options.use_direct_reads: 0 2022-05-13T23:06:09.506 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:06:09 smithi002 ceph-mon[71816]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2022-05-13T23:06:09.507 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:06:09 smithi002 ceph-mon[71816]: rocksdb: Options.create_missing_column_families: 0 2022-05-13T23:06:09.507 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:06:09 smithi002 ceph-mon[71816]: rocksdb: Options.db_log_dir: 2022-05-13T23:06:09.507 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:06:09 smithi002 ceph-mon[71816]: rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-a/store.db 2022-05-13T23:06:09.507 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:06:09 smithi002 ceph-mon[71816]: rocksdb: Options.table_cache_numshardbits: 6 2022-05-13T23:06:09.507 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:06:09 smithi002 ceph-mon[71816]: rocksdb: Options.WAL_ttl_seconds: 0 2022-05-13T23:06:09.508 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:06:09 smithi002 ceph-mon[71816]: rocksdb: Options.WAL_size_limit_MB: 0 2022-05-13T23:06:09.508 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:06:09 smithi002 ceph-mon[71816]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2022-05-13T23:06:09.508 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:06:09 smithi002 ceph-mon[71816]: rocksdb: Options.manifest_preallocation_size: 4194304 2022-05-13T23:06:09.508 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:06:09 smithi002 ceph-mon[71816]: rocksdb: Options.is_fd_close_on_exec: 1 2022-05-13T23:06:09.508 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:06:09 smithi002 ceph-mon[71816]: rocksdb: Options.advise_random_on_open: 1 2022-05-13T23:06:09.509 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:06:09 smithi002 ceph-mon[71816]: rocksdb: Options.db_write_buffer_size: 0 2022-05-13T23:06:09.509 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:06:09 smithi002 ceph-mon[71816]: rocksdb: Options.write_buffer_manager: 0x5610a40d8240 2022-05-13T23:06:09.510 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:06:09 smithi002 ceph-mon[71816]: rocksdb: Options.access_hint_on_compaction_start: 1 2022-05-13T23:06:09.510 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:06:09 smithi002 ceph-mon[71816]: rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2022-05-13T23:06:09.510 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:06:09 smithi002 ceph-mon[71816]: rocksdb: Options.random_access_max_buffer_size: 1048576 2022-05-13T23:06:09.510 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:06:09 smithi002 ceph-mon[71816]: rocksdb: Options.use_adaptive_mutex: 0 2022-05-13T23:06:09.511 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:06:09 smithi002 ceph-mon[71816]: rocksdb: Options.rate_limiter: (nil) 2022-05-13T23:06:09.511 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:06:09 smithi002 ceph-mon[71816]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2022-05-13T23:06:09.511 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:06:09 smithi002 ceph-mon[71816]: rocksdb: Options.wal_recovery_mode: 2 2022-05-13T23:06:09.511 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:06:09 smithi002 ceph-mon[71816]: rocksdb: Options.enable_thread_tracking: 0 2022-05-13T23:06:09.511 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:06:09 smithi002 ceph-mon[71816]: rocksdb: Options.enable_pipelined_write: 0 2022-05-13T23:06:09.512 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:06:09 smithi002 ceph-mon[71816]: rocksdb: Options.unordered_write: 0 2022-05-13T23:06:09.512 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:06:09 smithi002 ceph-mon[71816]: rocksdb: Options.allow_concurrent_memtable_write: 1 2022-05-13T23:06:09.512 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:06:09 smithi002 ceph-mon[71816]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2022-05-13T23:06:09.512 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:06:09 smithi002 ceph-mon[71816]: rocksdb: Options.write_thread_max_yield_usec: 100 2022-05-13T23:06:09.512 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:06:09 smithi002 ceph-mon[71816]: rocksdb: Options.write_thread_slow_yield_usec: 3 2022-05-13T23:06:09.513 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:06:09 smithi002 ceph-mon[71816]: rocksdb: Options.row_cache: None 2022-05-13T23:06:09.513 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:06:09 smithi002 ceph-mon[71816]: rocksdb: Options.wal_filter: None 2022-05-13T23:06:09.513 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:06:09 smithi002 ceph-mon[71816]: rocksdb: Options.avoid_flush_during_recovery: 0 2022-05-13T23:06:09.514 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:06:09 smithi002 ceph-mon[71816]: rocksdb: Options.allow_ingest_behind: 0 2022-05-13T23:06:09.514 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:06:09 smithi002 ceph-mon[71816]: rocksdb: Options.preserve_deletes: 0 2022-05-13T23:06:09.514 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:06:09 smithi002 ceph-mon[71816]: rocksdb: Options.two_write_queues: 0 2022-05-13T23:06:09.515 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:06:09 smithi002 ceph-mon[71816]: rocksdb: Options.manual_wal_flush: 0 2022-05-13T23:06:09.515 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:06:09 smithi002 ceph-mon[71816]: rocksdb: Options.atomic_flush: 0 2022-05-13T23:06:09.515 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:06:09 smithi002 ceph-mon[71816]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2022-05-13T23:06:09.516 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:06:09 smithi002 ceph-mon[71816]: rocksdb: Options.persist_stats_to_disk: 0 2022-05-13T23:06:09.516 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:06:09 smithi002 ceph-mon[71816]: rocksdb: Options.write_dbid_to_manifest: 0 2022-05-13T23:06:09.516 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:06:09 smithi002 ceph-mon[71816]: rocksdb: Options.log_readahead_size: 0 2022-05-13T23:06:09.516 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:06:09 smithi002 ceph-mon[71816]: rocksdb: Options.file_checksum_gen_factory: Unknown 2022-05-13T23:06:09.516 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:06:09 smithi002 ceph-mon[71816]: rocksdb: Options.best_efforts_recovery: 0 2022-05-13T23:06:09.517 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:06:09 smithi002 ceph-mon[71816]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2022-05-13T23:06:09.517 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:06:09 smithi002 ceph-mon[71816]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2022-05-13T23:06:09.517 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:06:09 smithi002 ceph-mon[71816]: rocksdb: Options.allow_data_in_errors: 0 2022-05-13T23:06:09.517 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:06:09 smithi002 ceph-mon[71816]: rocksdb: Options.db_host_id: __hostname__ 2022-05-13T23:06:09.517 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:06:09 smithi002 ceph-mon[71816]: rocksdb: Options.max_background_jobs: 2 2022-05-13T23:06:09.518 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:06:09 smithi002 ceph-mon[71816]: rocksdb: Options.max_background_compactions: -1 2022-05-13T23:06:09.518 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:06:09 smithi002 ceph-mon[71816]: rocksdb: Options.max_subcompactions: 1 2022-05-13T23:06:09.518 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:06:09 smithi002 ceph-mon[71816]: rocksdb: Options.avoid_flush_during_shutdown: 0 2022-05-13T23:06:09.518 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:06:09 smithi002 ceph-mon[71816]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2022-05-13T23:06:09.519 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:06:09 smithi002 ceph-mon[71816]: rocksdb: Options.delayed_write_rate : 16777216 2022-05-13T23:06:09.519 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:06:09 smithi002 bash[71606]: 36eab9452fd0abb91f4ef10b451718dc170093b2e87d84651e9e8954b8352bca 2022-05-13T23:06:09.519 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:06:09 smithi002 ceph-mon[71816]: rocksdb: Options.max_total_wal_size: 0 2022-05-13T23:06:09.520 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:06:09 smithi002 ceph-mon[71816]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2022-05-13T23:06:09.520 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:06:09 smithi002 ceph-mon[71816]: rocksdb: Options.stats_dump_period_sec: 600 2022-05-13T23:06:09.520 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:06:09 smithi002 ceph-mon[71816]: rocksdb: Options.stats_persist_period_sec: 600 2022-05-13T23:06:09.520 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:06:09 smithi002 ceph-mon[71816]: rocksdb: Options.stats_history_buffer_size: 1048576 2022-05-13T23:06:09.521 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:06:09 smithi002 ceph-mon[71816]: rocksdb: Options.max_open_files: -1 2022-05-13T23:06:09.521 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:06:09 smithi002 ceph-mon[71816]: rocksdb: Options.bytes_per_sync: 0 2022-05-13T23:06:09.521 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:06:09 smithi002 ceph-mon[71816]: rocksdb: Options.wal_bytes_per_sync: 0 2022-05-13T23:06:09.521 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:06:09 smithi002 ceph-mon[71816]: rocksdb: Options.strict_bytes_per_sync: 0 2022-05-13T23:06:09.521 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:06:09 smithi002 ceph-mon[71816]: rocksdb: Options.compaction_readahead_size: 0 2022-05-13T23:06:09.522 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:06:09 smithi002 ceph-mon[71816]: rocksdb: Options.max_background_flushes: -1 2022-05-13T23:06:09.522 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:06:09 smithi002 ceph-mon[71816]: rocksdb: Compression algorithms supported: 2022-05-13T23:06:09.522 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:06:09 smithi002 ceph-mon[71816]: rocksdb: kZSTDNotFinalCompression supported: 0 2022-05-13T23:06:09.522 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:06:09 smithi002 ceph-mon[71816]: rocksdb: kZSTD supported: 0 2022-05-13T23:06:09.523 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:06:09 smithi002 ceph-mon[71816]: rocksdb: kXpressCompression supported: 0 2022-05-13T23:06:09.523 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:06:09 smithi002 ceph-mon[71816]: rocksdb: kLZ4HCCompression supported: 1 2022-05-13T23:06:09.523 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:06:09 smithi002 ceph-mon[71816]: rocksdb: kLZ4Compression supported: 1 2022-05-13T23:06:09.523 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:06:09 smithi002 ceph-mon[71816]: rocksdb: kBZip2Compression supported: 0 2022-05-13T23:06:09.523 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:06:09 smithi002 ceph-mon[71816]: rocksdb: kZlibCompression supported: 1 2022-05-13T23:06:09.524 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:06:09 smithi002 ceph-mon[71816]: rocksdb: kSnappyCompression supported: 1 2022-05-13T23:06:09.524 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:06:09 smithi002 ceph-mon[71816]: rocksdb: Fast CRC32 supported: Supported on x86 2022-05-13T23:06:09.524 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:06:09 smithi002 ceph-mon[71816]: rocksdb: [db/version_set.cc:4725] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000009 2022-05-13T23:06:09.525 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:06:09 smithi002 ceph-mon[71816]: rocksdb: [db/column_family.cc:597] --------------- Options for column family [default]: 2022-05-13T23:06:09.525 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:06:09 smithi002 ceph-mon[71816]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2022-05-13T23:06:09.525 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:06:09 smithi002 ceph-mon[71816]: rocksdb: Options.merge_operator: 2022-05-13T23:06:09.525 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:06:09 smithi002 ceph-mon[71816]: rocksdb: Options.compaction_filter: None 2022-05-13T23:06:09.526 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:06:09 smithi002 ceph-mon[71816]: rocksdb: Options.compaction_filter_factory: None 2022-05-13T23:06:09.526 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:06:09 smithi002 ceph-mon[71816]: rocksdb: Options.sst_partitioner_factory: None 2022-05-13T23:06:09.527 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:06:09 smithi002 ceph-mon[71816]: rocksdb: Options.memtable_factory: SkipListFactory 2022-05-13T23:06:09.527 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:06:09 smithi002 ceph-mon[71816]: rocksdb: Options.table_factory: BlockBasedTable 2022-05-13T23:06:09.528 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:06:09 smithi002 ceph-mon[71816]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x5610a3fb3d10) 2022-05-13T23:06:09.528 INFO:journalctl@ceph.mon.a.smithi002.stdout: cache_index_and_filter_blocks: 1 2022-05-13T23:06:09.531 INFO:journalctl@ceph.mon.a.smithi002.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2022-05-13T23:06:09.531 INFO:journalctl@ceph.mon.a.smithi002.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2022-05-13T23:06:09.531 INFO:journalctl@ceph.mon.a.smithi002.stdout: pin_top_level_index_and_filter: 1 2022-05-13T23:06:09.531 INFO:journalctl@ceph.mon.a.smithi002.stdout: index_type: 0 2022-05-13T23:06:09.532 INFO:journalctl@ceph.mon.a.smithi002.stdout: data_block_index_type: 0 2022-05-13T23:06:09.532 INFO:journalctl@ceph.mon.a.smithi002.stdout: index_shortening: 1 2022-05-13T23:06:09.532 INFO:journalctl@ceph.mon.a.smithi002.stdout: data_block_hash_table_util_ratio: 0.750000 2022-05-13T23:06:09.532 INFO:journalctl@ceph.mon.a.smithi002.stdout: hash_index_allow_collision: 1 2022-05-13T23:06:09.532 INFO:journalctl@ceph.mon.a.smithi002.stdout: checksum: 1 2022-05-13T23:06:09.533 INFO:journalctl@ceph.mon.a.smithi002.stdout: no_block_cache: 0 2022-05-13T23:06:09.533 INFO:journalctl@ceph.mon.a.smithi002.stdout: block_cache: 0x5610a401e170 2022-05-13T23:06:09.533 INFO:journalctl@ceph.mon.a.smithi002.stdout: block_cache_name: BinnedLRUCache 2022-05-13T23:06:09.533 INFO:journalctl@ceph.mon.a.smithi002.stdout: block_cache_options: 2022-05-13T23:06:09.534 INFO:journalctl@ceph.mon.a.smithi002.stdout: capacity : 536870912 2022-05-13T23:06:09.534 INFO:journalctl@ceph.mon.a.smithi002.stdout: num_shard_bits : 4 2022-05-13T23:06:09.534 INFO:journalctl@ceph.mon.a.smithi002.stdout: strict_capacity_limit : 0 2022-05-13T23:06:09.534 INFO:journalctl@ceph.mon.a.smithi002.stdout: high_pri_pool_ratio: 0.000 2022-05-13T23:06:09.535 INFO:journalctl@ceph.mon.a.smithi002.stdout: block_cache_compressed: (nil) 2022-05-13T23:06:09.535 INFO:journalctl@ceph.mon.a.smithi002.stdout: persistent_cache: (nil) 2022-05-13T23:06:09.535 INFO:journalctl@ceph.mon.a.smithi002.stdout: block_size: 4096 2022-05-13T23:06:09.535 INFO:journalctl@ceph.mon.a.smithi002.stdout: block_size_deviation: 10 2022-05-13T23:06:09.535 INFO:journalctl@ceph.mon.a.smithi002.stdout: block_restart_interval: 16 2022-05-13T23:06:09.536 INFO:journalctl@ceph.mon.a.smithi002.stdout: index_block_restart_interval: 1 2022-05-13T23:06:09.536 INFO:journalctl@ceph.mon.a.smithi002.stdout: metadata_block_size: 4096 2022-05-13T23:06:09.536 INFO:journalctl@ceph.mon.a.smithi002.stdout: partition_filters: 0 2022-05-13T23:06:09.536 INFO:journalctl@ceph.mon.a.smithi002.stdout: use_delta_encoding: 1 2022-05-13T23:06:09.536 INFO:journalctl@ceph.mon.a.smithi002.stdout: filter_policy: rocksdb.BuiltinBloomFilter 2022-05-13T23:06:09.537 INFO:journalctl@ceph.mon.a.smithi002.stdout: whole_key_filtering: 1 2022-05-13T23:06:09.537 INFO:journalctl@ceph.mon.a.smithi002.stdout: verify_compression: 0 2022-05-13T23:06:09.537 INFO:journalctl@ceph.mon.a.smithi002.stdout: read_amp_bytes_per_bit: 0 2022-05-13T23:06:09.537 INFO:journalctl@ceph.mon.a.smithi002.stdout: format_version: 4 2022-05-13T23:06:09.537 INFO:journalctl@ceph.mon.a.smithi002.stdout: enable_index_compression: 1 2022-05-13T23:06:09.538 INFO:journalctl@ceph.mon.a.smithi002.stdout: block_align: 0 2022-05-13T23:06:09.538 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:06:09 smithi002 ceph-mon[71816]: rocksdb: Options.write_buffer_size: 33554432 2022-05-13T23:06:09.538 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:06:09 smithi002 ceph-mon[71816]: rocksdb: Options.max_write_buffer_number: 2 2022-05-13T23:06:09.538 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:06:09 smithi002 ceph-mon[71816]: rocksdb: Options.compression: NoCompression 2022-05-13T23:06:09.538 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:06:09 smithi002 ceph-mon[71816]: rocksdb: Options.bottommost_compression: Disabled 2022-05-13T23:06:09.539 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:06:09 smithi002 ceph-mon[71816]: rocksdb: Options.prefix_extractor: nullptr 2022-05-13T23:06:09.539 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:06:09 smithi002 ceph-mon[71816]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2022-05-13T23:06:09.539 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:06:09 smithi002 ceph-mon[71816]: rocksdb: Options.num_levels: 7 2022-05-13T23:06:09.539 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:06:09 smithi002 ceph-mon[71816]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2022-05-13T23:06:09.540 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:06:09 smithi002 ceph-mon[71816]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2022-05-13T23:06:09.540 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:06:09 smithi002 ceph-mon[71816]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2022-05-13T23:06:09.540 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:06:09 smithi002 ceph-mon[71816]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2022-05-13T23:06:09.540 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:06:09 smithi002 ceph-mon[71816]: rocksdb: Options.bottommost_compression_opts.level: 32767 2022-05-13T23:06:09.541 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:06:09 smithi002 ceph-mon[71816]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2022-05-13T23:06:09.541 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:06:09 smithi002 ceph-mon[71816]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2022-05-13T23:06:09.541 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:06:09 smithi002 ceph-mon[71816]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2022-05-13T23:06:09.542 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:06:09 smithi002 ceph-mon[71816]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2022-05-13T23:06:09.542 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:06:09 smithi002 ceph-mon[71816]: rocksdb: Options.bottommost_compression_opts.enabled: false 2022-05-13T23:06:09.542 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:06:09 smithi002 ceph-mon[71816]: rocksdb: Options.compression_opts.window_bits: -14 2022-05-13T23:06:09.542 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:06:09 smithi002 ceph-mon[71816]: rocksdb: Options.compression_opts.level: 32767 2022-05-13T23:06:09.542 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:06:09 smithi002 ceph-mon[71816]: rocksdb: Options.compression_opts.strategy: 0 2022-05-13T23:06:09.543 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:06:09 smithi002 ceph-mon[71816]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2022-05-13T23:06:09.543 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:06:09 smithi002 ceph-mon[71816]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2022-05-13T23:06:09.543 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:06:09 smithi002 ceph-mon[71816]: rocksdb: Options.compression_opts.parallel_threads: 1 2022-05-13T23:06:09.543 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:06:09 smithi002 ceph-mon[71816]: rocksdb: Options.compression_opts.enabled: false 2022-05-13T23:06:09.544 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:06:09 smithi002 ceph-mon[71816]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2022-05-13T23:06:09.544 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:06:09 smithi002 ceph-mon[71816]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2022-05-13T23:06:09.544 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:06:09 smithi002 ceph-mon[71816]: rocksdb: Options.level0_stop_writes_trigger: 36 2022-05-13T23:06:09.544 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:06:09 smithi002 ceph-mon[71816]: rocksdb: Options.target_file_size_base: 67108864 2022-05-13T23:06:09.544 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:06:09 smithi002 ceph-mon[71816]: rocksdb: Options.target_file_size_multiplier: 1 2022-05-13T23:06:09.545 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:06:09 smithi002 ceph-mon[71816]: rocksdb: Options.max_bytes_for_level_base: 268435456 2022-05-13T23:06:09.545 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:06:09 smithi002 ceph-mon[71816]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2022-05-13T23:06:09.545 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:06:09 smithi002 ceph-mon[71816]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2022-05-13T23:06:09.545 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:06:09 smithi002 ceph-mon[71816]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2022-05-13T23:06:09.546 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:06:09 smithi002 ceph-mon[71816]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2022-05-13T23:06:09.546 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:06:09 smithi002 ceph-mon[71816]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2022-05-13T23:06:09.546 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:06:09 smithi002 ceph-mon[71816]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2022-05-13T23:06:09.546 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:06:09 smithi002 ceph-mon[71816]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2022-05-13T23:06:09.546 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:06:09 smithi002 ceph-mon[71816]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2022-05-13T23:06:09.547 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:06:09 smithi002 ceph-mon[71816]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2022-05-13T23:06:09.547 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:06:09 smithi002 ceph-mon[71816]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2022-05-13T23:06:09.547 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:06:09 smithi002 ceph-mon[71816]: rocksdb: Options.max_compaction_bytes: 1677721600 2022-05-13T23:06:09.547 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:06:09 smithi002 ceph-mon[71816]: rocksdb: Options.arena_block_size: 4194304 2022-05-13T23:06:09.547 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:06:09 smithi002 ceph-mon[71816]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2022-05-13T23:06:09.548 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:06:09 smithi002 ceph-mon[71816]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2022-05-13T23:06:09.548 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:06:09 smithi002 ceph-mon[71816]: rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2022-05-13T23:06:09.548 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:06:09 smithi002 ceph-mon[71816]: rocksdb: Options.disable_auto_compactions: 0 2022-05-13T23:06:09.548 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:06:09 smithi002 ceph-mon[71816]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2022-05-13T23:06:09.548 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:06:09 smithi002 ceph-mon[71816]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2022-05-13T23:06:09.549 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:06:09 smithi002 ceph-mon[71816]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2022-05-13T23:06:09.549 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:06:09 smithi002 ceph-mon[71816]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2022-05-13T23:06:09.549 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:06:09 smithi002 ceph-mon[71816]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2022-05-13T23:06:09.549 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:06:09 smithi002 ceph-mon[71816]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2022-05-13T23:06:09.550 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:06:09 smithi002 ceph-mon[71816]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2022-05-13T23:06:09.550 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:06:09 smithi002 ceph-mon[71816]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2022-05-13T23:06:09.550 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:06:09 smithi002 ceph-mon[71816]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2022-05-13T23:06:09.550 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:06:09 smithi002 ceph-mon[71816]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2022-05-13T23:06:09.550 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:06:09 smithi002 ceph-mon[71816]: rocksdb: Options.table_properties_collectors: 2022-05-13T23:06:09.551 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:06:09 smithi002 ceph-mon[71816]: rocksdb: Options.inplace_update_support: 0 2022-05-13T23:06:09.551 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:06:09 smithi002 ceph-mon[71816]: rocksdb: Options.inplace_update_num_locks: 10000 2022-05-13T23:06:09.552 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:06:09 smithi002 ceph-mon[71816]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2022-05-13T23:06:09.552 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:06:09 smithi002 ceph-mon[71816]: rocksdb: Options.memtable_whole_key_filtering: 0 2022-05-13T23:06:09.552 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:06:09 smithi002 ceph-mon[71816]: rocksdb: Options.memtable_huge_page_size: 0 2022-05-13T23:06:09.552 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:06:09 smithi002 ceph-mon[71816]: rocksdb: Options.bloom_locality: 0 2022-05-13T23:06:09.552 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:06:09 smithi002 ceph-mon[71816]: rocksdb: Options.max_successive_merges: 0 2022-05-13T23:06:09.553 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:06:09 smithi002 ceph-mon[71816]: rocksdb: Options.optimize_filters_for_hits: 0 2022-05-13T23:06:09.553 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:06:09 smithi002 ceph-mon[71816]: rocksdb: Options.paranoid_file_checks: 0 2022-05-13T23:06:09.553 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:06:09 smithi002 ceph-mon[71816]: rocksdb: Options.force_consistency_checks: 1 2022-05-13T23:06:09.553 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:06:09 smithi002 ceph-mon[71816]: rocksdb: Options.report_bg_io_stats: 0 2022-05-13T23:06:09.554 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:06:09 smithi002 ceph-mon[71816]: rocksdb: Options.ttl: 2592000 2022-05-13T23:06:09.554 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:06:09 smithi002 ceph-mon[71816]: rocksdb: Options.periodic_compaction_seconds: 0 2022-05-13T23:06:09.554 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:06:09 smithi002 ceph-mon[71816]: rocksdb: Options.enable_blob_files: false 2022-05-13T23:06:09.554 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:06:09 smithi002 ceph-mon[71816]: rocksdb: Options.min_blob_size: 0 2022-05-13T23:06:09.554 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:06:09 smithi002 ceph-mon[71816]: rocksdb: Options.blob_file_size: 268435456 2022-05-13T23:06:09.555 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:06:09 smithi002 ceph-mon[71816]: rocksdb: Options.blob_compression_type: NoCompression 2022-05-13T23:06:09.555 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:06:09 smithi002 ceph-mon[71816]: rocksdb: Options.enable_blob_garbage_collection: false 2022-05-13T23:06:09.555 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:06:09 smithi002 ceph-mon[71816]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2022-05-13T23:06:09.555 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:06:09 smithi002 ceph-mon[71816]: rocksdb: [db/version_set.cc:4773] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000009 succeeded,manifest_file_number is 9, next_file_number is 11, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2022-05-13T23:06:09.555 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:06:09 smithi002 ceph-mon[71816]: rocksdb: [db/version_set.cc:4782] Column family [default] (ID 0), log number is 5 2022-05-13T23:06:09.556 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:06:09 smithi002 ceph-mon[71816]: rocksdb: [db/version_set.cc:4083] Creating manifest 13 2022-05-13T23:06:09.556 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:06:09 smithi002 ceph-mon[71816]: rocksdb: EVENT_LOG_v1 {"time_micros": 1652483169121806, "job": 1, "event": "recovery_started", "wal_files": [10]} 2022-05-13T23:06:09.556 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:06:09 smithi002 ceph-mon[71816]: rocksdb: [db/db_impl/db_impl_open.cc:847] Recovering log #10 mode 2 2022-05-13T23:06:09.556 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:06:09 smithi002 ceph-mon[71816]: rocksdb: [table/block_based/filter_policy.cc:996] Using legacy Bloom filter with high (20) bits/key. Dramatic filter space and/or accuracy improvement is available with format_version>=5. 2022-05-13T23:06:09.557 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:06:09 smithi002 ceph-mon[71816]: rocksdb: EVENT_LOG_v1 {"time_micros": 1652483169123636, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 14, "file_size": 91756, "file_checksum": "", "file_checksum_func_name": "Unknown", "table_properties": {"data_size": 89849, "index_size": 267, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 709, "raw_key_size": 13376, "raw_average_key_size": 49, "raw_value_size": 82528, "raw_average_value_size": 306, "num_data_blocks": 11, "num_entries": 269, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "rocksdb.BuiltinBloomFilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ", "creation_time": 1652483169, "oldest_key_time": 0, "file_creation_time": 0, "db_id": "4aedf19e-7050-42b5-a8e3-47cff7cfd09c", "db_session_id": "LL68RT1PL5QAV7MZYVZD"}} 2022-05-13T23:06:09.557 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:06:09 smithi002 ceph-mon[71816]: rocksdb: [db/version_set.cc:4083] Creating manifest 15 2022-05-13T23:06:09.557 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:06:09 smithi002 ceph-mon[71816]: rocksdb: EVENT_LOG_v1 {"time_micros": 1652483169124541, "job": 1, "event": "recovery_finished"} 2022-05-13T23:06:09.557 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:06:09 smithi002 ceph-mon[71816]: rocksdb: [file/delete_scheduler.cc:73] Deleted file /var/lib/ceph/mon/ceph-a/store.db/000010.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2022-05-13T23:06:09.557 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:06:09 smithi002 ceph-mon[71816]: rocksdb: [db/db_impl/db_impl_open.cc:1701] SstFileManager instance 0x5610a4004a80 2022-05-13T23:06:09.558 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:06:09 smithi002 ceph-mon[71816]: rocksdb: DB pointer 0x5610a4014000 2022-05-13T23:06:09.558 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:06:09 smithi002 ceph-mon[71816]: rocksdb: [db/db_impl/db_impl.cc:902] ------- DUMPING STATS ------- 2022-05-13T23:06:09.558 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:06:09 smithi002 ceph-mon[71816]: rocksdb: [db/db_impl/db_impl.cc:903] 2022-05-13T23:06:09.558 INFO:journalctl@ceph.mon.a.smithi002.stdout: ** DB Stats ** 2022-05-13T23:06:09.558 INFO:journalctl@ceph.mon.a.smithi002.stdout: Uptime(secs): 0.0 total, 0.0 interval 2022-05-13T23:06:09.559 INFO:journalctl@ceph.mon.a.smithi002.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2022-05-13T23:06:09.559 INFO:journalctl@ceph.mon.a.smithi002.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2022-05-13T23:06:09.559 INFO:journalctl@ceph.mon.a.smithi002.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2022-05-13T23:06:09.559 INFO:journalctl@ceph.mon.a.smithi002.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2022-05-13T23:06:09.560 INFO:journalctl@ceph.mon.a.smithi002.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2022-05-13T23:06:09.560 INFO:journalctl@ceph.mon.a.smithi002.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2022-05-13T23:06:09.560 INFO:journalctl@ceph.mon.a.smithi002.stdout: 2022-05-13T23:06:09.560 INFO:journalctl@ceph.mon.a.smithi002.stdout: ** Compaction Stats [default] ** 2022-05-13T23:06:09.560 INFO:journalctl@ceph.mon.a.smithi002.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2022-05-13T23:06:09.561 INFO:journalctl@ceph.mon.a.smithi002.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2022-05-13T23:06:09.561 INFO:journalctl@ceph.mon.a.smithi002.stdout: L0 2/0 91.37 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 67.8 0.00 0.00 1 0.001 0 0 2022-05-13T23:06:09.561 INFO:journalctl@ceph.mon.a.smithi002.stdout: Sum 2/0 91.37 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 67.8 0.00 0.00 1 0.001 0 0 2022-05-13T23:06:09.561 INFO:journalctl@ceph.mon.a.smithi002.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 67.8 0.00 0.00 1 0.001 0 0 2022-05-13T23:06:09.562 INFO:journalctl@ceph.mon.a.smithi002.stdout: 2022-05-13T23:06:09.562 INFO:journalctl@ceph.mon.a.smithi002.stdout: ** Compaction Stats [default] ** 2022-05-13T23:06:09.562 INFO:journalctl@ceph.mon.a.smithi002.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2022-05-13T23:06:09.563 INFO:journalctl@ceph.mon.a.smithi002.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2022-05-13T23:06:09.563 INFO:journalctl@ceph.mon.a.smithi002.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 67.8 0.00 0.00 1 0.001 0 0 2022-05-13T23:06:09.563 INFO:journalctl@ceph.mon.a.smithi002.stdout: Uptime(secs): 0.0 total, 0.0 interval 2022-05-13T23:06:09.563 INFO:journalctl@ceph.mon.a.smithi002.stdout: Flush(GB): cumulative 0.000, interval 0.000 2022-05-13T23:06:09.564 INFO:journalctl@ceph.mon.a.smithi002.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2022-05-13T23:06:09.564 INFO:journalctl@ceph.mon.a.smithi002.stdout: AddFile(Total Files): cumulative 0, interval 0 2022-05-13T23:06:09.564 INFO:journalctl@ceph.mon.a.smithi002.stdout: AddFile(L0 Files): cumulative 0, interval 0 2022-05-13T23:06:09.564 INFO:journalctl@ceph.mon.a.smithi002.stdout: AddFile(Keys): cumulative 0, interval 0 2022-05-13T23:06:09.565 INFO:journalctl@ceph.mon.a.smithi002.stdout: Cumulative compaction: 0.00 GB write, 11.19 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2022-05-13T23:06:09.565 INFO:journalctl@ceph.mon.a.smithi002.stdout: Interval compaction: 0.00 GB write, 11.19 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2022-05-13T23:06:09.565 INFO:journalctl@ceph.mon.a.smithi002.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2022-05-13T23:06:09.565 INFO:journalctl@ceph.mon.a.smithi002.stdout: 2022-05-13T23:06:09.565 INFO:journalctl@ceph.mon.a.smithi002.stdout: ** File Read Latency Histogram By Level [default] ** 2022-05-13T23:06:09.566 INFO:journalctl@ceph.mon.a.smithi002.stdout: 2022-05-13T23:06:09.566 INFO:journalctl@ceph.mon.a.smithi002.stdout: ** Compaction Stats [default] ** 2022-05-13T23:06:09.566 INFO:journalctl@ceph.mon.a.smithi002.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2022-05-13T23:06:09.566 INFO:journalctl@ceph.mon.a.smithi002.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2022-05-13T23:06:09.566 INFO:journalctl@ceph.mon.a.smithi002.stdout: L0 2/0 91.37 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 67.8 0.00 0.00 1 0.001 0 0 2022-05-13T23:06:09.567 INFO:journalctl@ceph.mon.a.smithi002.stdout: Sum 2/0 91.37 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 67.8 0.00 0.00 1 0.001 0 0 2022-05-13T23:06:09.567 INFO:journalctl@ceph.mon.a.smithi002.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2022-05-13T23:06:09.567 INFO:journalctl@ceph.mon.a.smithi002.stdout: 2022-05-13T23:06:09.567 INFO:journalctl@ceph.mon.a.smithi002.stdout: ** Compaction Stats [default] ** 2022-05-13T23:06:09.568 INFO:journalctl@ceph.mon.a.smithi002.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2022-05-13T23:06:09.568 INFO:journalctl@ceph.mon.a.smithi002.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2022-05-13T23:06:09.568 INFO:journalctl@ceph.mon.a.smithi002.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 67.8 0.00 0.00 1 0.001 0 0 2022-05-13T23:06:09.568 INFO:journalctl@ceph.mon.a.smithi002.stdout: Uptime(secs): 0.0 total, 0.0 interval 2022-05-13T23:06:09.569 INFO:journalctl@ceph.mon.a.smithi002.stdout: Flush(GB): cumulative 0.000, interval 0.000 2022-05-13T23:06:09.569 INFO:journalctl@ceph.mon.a.smithi002.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2022-05-13T23:06:09.569 INFO:journalctl@ceph.mon.a.smithi002.stdout: AddFile(Total Files): cumulative 0, interval 0 2022-05-13T23:06:09.569 INFO:journalctl@ceph.mon.a.smithi002.stdout: AddFile(L0 Files): cumulative 0, interval 0 2022-05-13T23:06:09.569 INFO:journalctl@ceph.mon.a.smithi002.stdout: AddFile(Keys): cumulative 0, interval 0 2022-05-13T23:06:09.570 INFO:journalctl@ceph.mon.a.smithi002.stdout: Cumulative compaction: 0.00 GB write, 11.14 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2022-05-13T23:06:09.570 INFO:journalctl@ceph.mon.a.smithi002.stdout: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2022-05-13T23:06:09.570 INFO:journalctl@ceph.mon.a.smithi002.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2022-05-13T23:06:09.570 INFO:journalctl@ceph.mon.a.smithi002.stdout: 2022-05-13T23:06:09.571 INFO:journalctl@ceph.mon.a.smithi002.stdout: ** File Read Latency Histogram By Level [default] ** 2022-05-13T23:06:09.571 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:06:09 smithi002 ceph-mon[71816]: starting mon.a rank 0 at public addrs [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] at bind addrs [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] mon_data /var/lib/ceph/mon/ceph-a fsid 16cf5f88-d311-11ec-be40-001a4aab830c 2022-05-13T23:06:09.571 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:06:09 smithi002 ceph-mon[71816]: mon.a@-1(???) e1 preinit fsid 16cf5f88-d311-11ec-be40-001a4aab830c 2022-05-13T23:06:09.571 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:06:09 smithi002 ceph-mon[71816]: mon.a@-1(???).mds e1 new map 2022-05-13T23:06:09.571 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:06:09 smithi002 ceph-mon[71816]: mon.a@-1(???).mds e1 print_map 2022-05-13T23:06:09.572 INFO:journalctl@ceph.mon.a.smithi002.stdout: e1 2022-05-13T23:06:09.572 INFO:journalctl@ceph.mon.a.smithi002.stdout: enable_multiple, ever_enabled_multiple: 1,1 2022-05-13T23:06:09.572 INFO:journalctl@ceph.mon.a.smithi002.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2022-05-13T23:06:09.573 INFO:journalctl@ceph.mon.a.smithi002.stdout: legacy client fscid: -1 2022-05-13T23:06:09.573 INFO:journalctl@ceph.mon.a.smithi002.stdout: 2022-05-13T23:06:09.573 INFO:journalctl@ceph.mon.a.smithi002.stdout: No filesystems configured 2022-05-13T23:06:09.573 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:06:09 smithi002 ceph-mon[71816]: mon.a@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2022-05-13T23:06:09.574 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:06:09 smithi002 ceph-mon[71816]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2022-05-13T23:06:09.574 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:06:09 smithi002 ceph-mon[71816]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2022-05-13T23:06:09.574 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:06:09 smithi002 ceph-mon[71816]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2022-05-13T23:06:09.574 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:06:09 smithi002 ceph-mon[71816]: mon.a@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2022-05-13T23:06:09.575 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:06:09 smithi002 systemd[1]: Started Ceph mon.a for 16cf5f88-d311-11ec-be40-001a4aab830c. 2022-05-13T23:06:09.575 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:06:09 smithi002 ceph-mon[71816]: mon.a is new leader, mons a in quorum (ranks 0) 2022-05-13T23:06:09.575 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:06:09 smithi002 ceph-mon[71816]: monmap e1: 1 mons at {a=[v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0]} 2022-05-13T23:06:09.575 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:06:09 smithi002 ceph-mon[71816]: fsmap 2022-05-13T23:06:09.575 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:06:09 smithi002 ceph-mon[71816]: osdmap e1: 0 total, 0 up, 0 in 2022-05-13T23:06:09.576 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:06:09 smithi002 ceph-mon[71816]: mgrmap e1: no daemons active 2022-05-13T23:06:10.476 INFO:teuthology.orchestra.run.smithi002.stdout:Wrote config to /etc/ceph/ceph.conf 2022-05-13T23:06:10.476 INFO:teuthology.orchestra.run.smithi002.stdout:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2022-05-13T23:06:10.476 INFO:teuthology.orchestra.run.smithi002.stdout:Creating mgr... 2022-05-13T23:06:10.477 INFO:teuthology.orchestra.run.smithi002.stdout:Verifying port 9283 ... 2022-05-13T23:06:11.230 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:06:10 smithi002 ceph-mon[71816]: from='client.? 172.21.15.2:0/3199405001' entity='client.admin' 2022-05-13T23:06:11.572 INFO:journalctl@ceph.mgr.x.smithi002.stdout:May 13 23:06:11 smithi002 systemd[1]: Starting Ceph mgr.x for 16cf5f88-d311-11ec-be40-001a4aab830c... 2022-05-13T23:06:12.423 INFO:teuthology.orchestra.run.smithi002.stdout:Waiting for mgr to start... 2022-05-13T23:06:12.424 INFO:teuthology.orchestra.run.smithi002.stdout:Waiting for mgr... 2022-05-13T23:06:12.572 INFO:journalctl@ceph.mgr.x.smithi002.stdout:May 13 23:06:12 smithi002 bash[72042]: 6418c2a5057e5f790df508504230d2412e9c89a9861515e39e846c516720f305 2022-05-13T23:06:12.572 INFO:journalctl@ceph.mgr.x.smithi002.stdout:May 13 23:06:12 smithi002 systemd[1]: Started Ceph mgr.x for 16cf5f88-d311-11ec-be40-001a4aab830c. 2022-05-13T23:06:13.073 INFO:journalctl@ceph.mgr.x.smithi002.stdout:May 13 23:06:12 smithi002 conmon[72230]: 2022-05-13T23:06:12.787+0000 7f1d25db9000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2022-05-13T23:06:13.073 INFO:journalctl@ceph.mgr.x.smithi002.stdout:May 13 23:06:12 smithi002 conmon[72230]: 2022-05-13T23:06:12.922+0000 7f1d25db9000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2022-05-13T23:06:13.529 INFO:journalctl@ceph.mgr.x.smithi002.stdout:May 13 23:06:13 smithi002 conmon[72230]: 2022-05-13T23:06:13.166+0000 7f1d25db9000 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2022-05-13T23:06:13.530 INFO:journalctl@ceph.mgr.x.smithi002.stdout:May 13 23:06:13 smithi002 conmon[72230]: 2022-05-13T23:06:13.317+0000 7f1d25db9000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2022-05-13T23:06:13.822 INFO:journalctl@ceph.mgr.x.smithi002.stdout:May 13 23:06:13 smithi002 conmon[72230]: 2022-05-13T23:06:13.456+0000 7f1d25db9000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2022-05-13T23:06:14.322 INFO:journalctl@ceph.mgr.x.smithi002.stdout:May 13 23:06:13 smithi002 conmon[72230]: 2022-05-13T23:06:13.996+0000 7f1d25db9000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2022-05-13T23:06:14.323 INFO:journalctl@ceph.mgr.x.smithi002.stdout:May 13 23:06:14 smithi002 conmon[72230]: 2022-05-13T23:06:14.163+0000 7f1d25db9000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2022-05-13T23:06:15.797 INFO:journalctl@ceph.mgr.x.smithi002.stdout:May 13 23:06:15 smithi002 conmon[72230]: 2022-05-13T23:06:15.370+0000 7f1d25db9000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2022-05-13T23:06:15.798 INFO:journalctl@ceph.mgr.x.smithi002.stdout:May 13 23:06:15 smithi002 conmon[72230]: 2022-05-13T23:06:15.500+0000 7f1d25db9000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2022-05-13T23:06:16.072 INFO:journalctl@ceph.mgr.x.smithi002.stdout:May 13 23:06:15 smithi002 conmon[72230]: 2022-05-13T23:06:15.796+0000 7f1d25db9000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2022-05-13T23:06:16.572 INFO:journalctl@ceph.mgr.x.smithi002.stdout:May 13 23:06:16 smithi002 conmon[72230]: 2022-05-13T23:06:16.174+0000 7f1d25db9000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2022-05-13T23:06:16.943 INFO:journalctl@ceph.mgr.x.smithi002.stdout:May 13 23:06:16 smithi002 conmon[72230]: 2022-05-13T23:06:16.437+0000 7f1d25db9000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2022-05-13T23:06:16.943 INFO:journalctl@ceph.mgr.x.smithi002.stdout:May 13 23:06:16 smithi002 conmon[72230]: 2022-05-13T23:06:16.663+0000 7f1d25db9000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2022-05-13T23:06:17.237 INFO:journalctl@ceph.mgr.x.smithi002.stdout:May 13 23:06:16 smithi002 conmon[72230]: 2022-05-13T23:06:16.941+0000 7f1d25db9000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2022-05-13T23:06:17.237 INFO:journalctl@ceph.mgr.x.smithi002.stdout:May 13 23:06:17 smithi002 conmon[72230]: 2022-05-13T23:06:17.095+0000 7f1d25db9000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2022-05-13T23:06:17.550 INFO:journalctl@ceph.mgr.x.smithi002.stdout:May 13 23:06:17 smithi002 conmon[72230]: 2022-05-13T23:06:17.235+0000 7f1d25db9000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2022-05-13T23:06:17.551 INFO:journalctl@ceph.mgr.x.smithi002.stdout:May 13 23:06:17 smithi002 conmon[72230]: 2022-05-13T23:06:17.391+0000 7f1d25db9000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2022-05-13T23:06:17.822 INFO:journalctl@ceph.mgr.x.smithi002.stdout:May 13 23:06:17 smithi002 conmon[72230]: 2022-05-13T23:06:17.549+0000 7f1d25db9000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2022-05-13T23:06:17.822 INFO:journalctl@ceph.mgr.x.smithi002.stdout:May 13 23:06:17 smithi002 conmon[72230]: 2022-05-13T23:06:17.705+0000 7f1d25db9000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2022-05-13T23:06:18.322 INFO:journalctl@ceph.mgr.x.smithi002.stdout:May 13 23:06:18 smithi002 conmon[72230]: 2022-05-13T23:06:18.020+0000 7f1d25db9000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2022-05-13T23:06:19.121 INFO:journalctl@ceph.mgr.x.smithi002.stdout:May 13 23:06:18 smithi002 conmon[72230]: 2022-05-13T23:06:18.826+0000 7f1d25db9000 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2022-05-13T23:06:19.371 INFO:journalctl@ceph.mgr.x.smithi002.stdout:May 13 23:06:19 smithi002 conmon[72230]: 2022-05-13T23:06:19.119+0000 7f1d25db9000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2022-05-13T23:06:20.072 INFO:journalctl@ceph.mgr.x.smithi002.stdout:May 13 23:06:19 smithi002 conmon[72230]: 2022-05-13T23:06:19.806+0000 7f1d25db9000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2022-05-13T23:06:20.073 INFO:journalctl@ceph.mgr.x.smithi002.stdout:May 13 23:06:19 smithi002 conmon[72230]: 2022-05-13T23:06:19.991+0000 7f1d25db9000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2022-05-13T23:06:20.357 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:06:20 smithi002 ceph-mon[71816]: from='client.? 172.21.15.2:0/3835390402' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2022-05-13T23:06:20.444 INFO:teuthology.orchestra.run.smithi002.stdout:mgr not available, waiting (1/15)... 2022-05-13T23:06:22.322 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:06:22 smithi002 ceph-mon[71816]: Activating manager daemon x 2022-05-13T23:06:22.323 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:06:22 smithi002 ceph-mon[71816]: mgrmap e2: x(active, starting, since 0.00431749s) 2022-05-13T23:06:22.323 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:06:22 smithi002 ceph-mon[71816]: from='mgr.14100 172.21.15.2:0/2219257689' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2022-05-13T23:06:22.323 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:06:22 smithi002 ceph-mon[71816]: from='mgr.14100 172.21.15.2:0/2219257689' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2022-05-13T23:06:22.324 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:06:22 smithi002 ceph-mon[71816]: from='mgr.14100 172.21.15.2:0/2219257689' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2022-05-13T23:06:22.324 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:06:22 smithi002 ceph-mon[71816]: from='mgr.14100 172.21.15.2:0/2219257689' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-05-13T23:06:22.324 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:06:22 smithi002 ceph-mon[71816]: from='mgr.14100 172.21.15.2:0/2219257689' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2022-05-13T23:06:22.324 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:06:22 smithi002 ceph-mon[71816]: Manager daemon x is now available 2022-05-13T23:06:22.325 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:06:22 smithi002 ceph-mon[71816]: from='mgr.14100 172.21.15.2:0/2219257689' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-05-13T23:06:22.325 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:06:22 smithi002 ceph-mon[71816]: from='mgr.14100 172.21.15.2:0/2219257689' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-05-13T23:06:23.322 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:06:23 smithi002 ceph-mon[71816]: from='mgr.14100 172.21.15.2:0/2219257689' entity='mgr.x' 2022-05-13T23:06:23.323 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:06:23 smithi002 ceph-mon[71816]: from='mgr.14100 172.21.15.2:0/2219257689' entity='mgr.x' 2022-05-13T23:06:23.323 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:06:23 smithi002 ceph-mon[71816]: from='mgr.14100 172.21.15.2:0/2219257689' entity='mgr.x' 2022-05-13T23:06:23.323 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:06:23 smithi002 ceph-mon[71816]: mgrmap e3: x(active, since 1.01047s) 2022-05-13T23:06:24.058 INFO:teuthology.orchestra.run.smithi002.stdout:mgr is available 2022-05-13T23:06:24.322 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:06:24 smithi002 ceph-mon[71816]: from='client.? 172.21.15.2:0/2735932582' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2022-05-13T23:06:25.322 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:06:25 smithi002 ceph-mon[71816]: mgrmap e4: x(active, since 2s) 2022-05-13T23:06:25.323 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:06:25 smithi002 ceph-mon[71816]: from='client.? 172.21.15.2:0/1553054934' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2022-05-13T23:06:25.565 INFO:teuthology.orchestra.run.smithi002.stdout:Enabling cephadm module... 2022-05-13T23:06:26.822 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:06:26 smithi002 ceph-mon[71816]: from='client.? 172.21.15.2:0/4235272439' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2022-05-13T23:06:26.823 INFO:journalctl@ceph.mgr.x.smithi002.stdout:May 13 23:06:26 smithi002 conmon[72230]: ignoring --setuser ceph since I am not root 2022-05-13T23:06:26.823 INFO:journalctl@ceph.mgr.x.smithi002.stdout:May 13 23:06:26 smithi002 conmon[72230]: ignoring --setgroup ceph since I am not root 2022-05-13T23:06:27.415 INFO:journalctl@ceph.mgr.x.smithi002.stdout:May 13 23:06:27 smithi002 conmon[72230]: 2022-05-13T23:06:27.187+0000 7fe77ddd2000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2022-05-13T23:06:27.416 INFO:journalctl@ceph.mgr.x.smithi002.stdout:May 13 23:06:27 smithi002 conmon[72230]: 2022-05-13T23:06:27.330+0000 7fe77ddd2000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2022-05-13T23:06:27.749 INFO:journalctl@ceph.mgr.x.smithi002.stdout:May 13 23:06:27 smithi002 conmon[72230]: 2022-05-13T23:06:27.585+0000 7fe77ddd2000 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2022-05-13T23:06:27.749 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:06:27 smithi002 ceph-mon[71816]: from='client.? 172.21.15.2:0/4235272439' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2022-05-13T23:06:27.750 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:06:27 smithi002 ceph-mon[71816]: mgrmap e5: x(active, since 4s) 2022-05-13T23:06:28.072 INFO:journalctl@ceph.mgr.x.smithi002.stdout:May 13 23:06:27 smithi002 conmon[72230]: 2022-05-13T23:06:27.746+0000 7fe77ddd2000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2022-05-13T23:06:28.072 INFO:journalctl@ceph.mgr.x.smithi002.stdout:May 13 23:06:27 smithi002 conmon[72230]: 2022-05-13T23:06:27.889+0000 7fe77ddd2000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2022-05-13T23:06:28.563 INFO:teuthology.orchestra.run.smithi002.stdout:Waiting for the mgr to restart... 2022-05-13T23:06:28.563 INFO:teuthology.orchestra.run.smithi002.stdout:Waiting for mgr epoch 5... 2022-05-13T23:06:28.614 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:06:28 smithi002 ceph-mon[71816]: from='client.? 172.21.15.2:0/1770078136' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2022-05-13T23:06:28.614 INFO:journalctl@ceph.mgr.x.smithi002.stdout:May 13 23:06:28 smithi002 conmon[72230]: 2022-05-13T23:06:28.438+0000 7fe77ddd2000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2022-05-13T23:06:28.975 INFO:journalctl@ceph.mgr.x.smithi002.stdout:May 13 23:06:28 smithi002 conmon[72230]: 2022-05-13T23:06:28.612+0000 7fe77ddd2000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2022-05-13T23:06:30.268 INFO:journalctl@ceph.mgr.x.smithi002.stdout:May 13 23:06:29 smithi002 conmon[72230]: 2022-05-13T23:06:29.836+0000 7fe77ddd2000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2022-05-13T23:06:30.268 INFO:journalctl@ceph.mgr.x.smithi002.stdout:May 13 23:06:29 smithi002 conmon[72230]: 2022-05-13T23:06:29.968+0000 7fe77ddd2000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2022-05-13T23:06:30.572 INFO:journalctl@ceph.mgr.x.smithi002.stdout:May 13 23:06:30 smithi002 conmon[72230]: 2022-05-13T23:06:30.266+0000 7fe77ddd2000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2022-05-13T23:06:30.915 INFO:journalctl@ceph.mgr.x.smithi002.stdout:May 13 23:06:30 smithi002 conmon[72230]: 2022-05-13T23:06:30.648+0000 7fe77ddd2000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2022-05-13T23:06:31.322 INFO:journalctl@ceph.mgr.x.smithi002.stdout:May 13 23:06:30 smithi002 conmon[72230]: 2022-05-13T23:06:30.913+0000 7fe77ddd2000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2022-05-13T23:06:31.323 INFO:journalctl@ceph.mgr.x.smithi002.stdout:May 13 23:06:31 smithi002 conmon[72230]: 2022-05-13T23:06:31.142+0000 7fe77ddd2000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2022-05-13T23:06:31.717 INFO:journalctl@ceph.mgr.x.smithi002.stdout:May 13 23:06:31 smithi002 conmon[72230]: 2022-05-13T23:06:31.422+0000 7fe77ddd2000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2022-05-13T23:06:31.717 INFO:journalctl@ceph.mgr.x.smithi002.stdout:May 13 23:06:31 smithi002 conmon[72230]: 2022-05-13T23:06:31.575+0000 7fe77ddd2000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2022-05-13T23:06:32.023 INFO:journalctl@ceph.mgr.x.smithi002.stdout:May 13 23:06:31 smithi002 conmon[72230]: 2022-05-13T23:06:31.715+0000 7fe77ddd2000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2022-05-13T23:06:32.023 INFO:journalctl@ceph.mgr.x.smithi002.stdout:May 13 23:06:31 smithi002 conmon[72230]: 2022-05-13T23:06:31.872+0000 7fe77ddd2000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2022-05-13T23:06:32.322 INFO:journalctl@ceph.mgr.x.smithi002.stdout:May 13 23:06:32 smithi002 conmon[72230]: 2022-05-13T23:06:32.021+0000 7fe77ddd2000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2022-05-13T23:06:32.322 INFO:journalctl@ceph.mgr.x.smithi002.stdout:May 13 23:06:32 smithi002 conmon[72230]: 2022-05-13T23:06:32.178+0000 7fe77ddd2000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2022-05-13T23:06:32.823 INFO:journalctl@ceph.mgr.x.smithi002.stdout:May 13 23:06:32 smithi002 conmon[72230]: 2022-05-13T23:06:32.492+0000 7fe77ddd2000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2022-05-13T23:06:33.537 INFO:journalctl@ceph.mgr.x.smithi002.stdout:May 13 23:06:33 smithi002 conmon[72230]: 2022-05-13T23:06:33.240+0000 7fe77ddd2000 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2022-05-13T23:06:33.822 INFO:journalctl@ceph.mgr.x.smithi002.stdout:May 13 23:06:33 smithi002 conmon[72230]: 2022-05-13T23:06:33.535+0000 7fe77ddd2000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2022-05-13T23:06:34.573 INFO:journalctl@ceph.mgr.x.smithi002.stdout:May 13 23:06:34 smithi002 conmon[72230]: 2022-05-13T23:06:34.203+0000 7fe77ddd2000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2022-05-13T23:06:34.574 INFO:journalctl@ceph.mgr.x.smithi002.stdout:May 13 23:06:34 smithi002 conmon[72230]: 2022-05-13T23:06:34.366+0000 7fe77ddd2000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2022-05-13T23:06:36.822 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:06:36 smithi002 ceph-mon[71816]: Active manager daemon x restarted 2022-05-13T23:06:36.823 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:06:36 smithi002 ceph-mon[71816]: Activating manager daemon x 2022-05-13T23:06:36.823 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:06:36 smithi002 ceph-mon[71816]: osdmap e2: 0 total, 0 up, 0 in 2022-05-13T23:06:37.655 INFO:journalctl@ceph.mgr.x.smithi002.stdout:May 13 23:06:37 smithi002 conmon[72230]: [13/May/2022:23:06:37] ENGINE Bus STARTING 2022-05-13T23:06:37.655 INFO:journalctl@ceph.mgr.x.smithi002.stdout:May 13 23:06:37 smithi002 conmon[72230]: [13/May/2022:23:06:37] ENGINE Serving on https://172.21.15.2:7150 2022-05-13T23:06:37.656 INFO:journalctl@ceph.mgr.x.smithi002.stdout:May 13 23:06:37 smithi002 conmon[72230]: [13/May/2022:23:06:37] ENGINE Bus STARTED 2022-05-13T23:06:37.656 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:06:37 smithi002 ceph-mon[71816]: mgrmap e6: x(active, starting, since 0.0535079s) 2022-05-13T23:06:37.656 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:06:37 smithi002 ceph-mon[71816]: from='mgr.14116 172.21.15.2:0/3171684691' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-05-13T23:06:37.656 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:06:37 smithi002 ceph-mon[71816]: from='mgr.14116 172.21.15.2:0/3171684691' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2022-05-13T23:06:37.657 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:06:37 smithi002 ceph-mon[71816]: from='mgr.14116 172.21.15.2:0/3171684691' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2022-05-13T23:06:37.657 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:06:37 smithi002 ceph-mon[71816]: from='mgr.14116 172.21.15.2:0/3171684691' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2022-05-13T23:06:37.657 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:06:37 smithi002 ceph-mon[71816]: from='mgr.14116 172.21.15.2:0/3171684691' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2022-05-13T23:06:37.657 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:06:37 smithi002 ceph-mon[71816]: Manager daemon x is now available 2022-05-13T23:06:37.658 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:06:37 smithi002 ceph-mon[71816]: from='mgr.14116 172.21.15.2:0/3171684691' entity='mgr.x' 2022-05-13T23:06:37.658 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:06:37 smithi002 ceph-mon[71816]: from='mgr.14116 172.21.15.2:0/3171684691' entity='mgr.x' 2022-05-13T23:06:37.658 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:06:37 smithi002 ceph-mon[71816]: from='mgr.14116 172.21.15.2:0/3171684691' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-13T23:06:37.658 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:06:37 smithi002 ceph-mon[71816]: from='mgr.14116 172.21.15.2:0/3171684691' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-13T23:06:37.658 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:06:37 smithi002 ceph-mon[71816]: from='mgr.14116 172.21.15.2:0/3171684691' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-13T23:06:37.659 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:06:37 smithi002 ceph-mon[71816]: from='mgr.14116 172.21.15.2:0/3171684691' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-05-13T23:06:37.659 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:06:37 smithi002 ceph-mon[71816]: from='mgr.14116 172.21.15.2:0/3171684691' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-05-13T23:06:37.870 INFO:teuthology.orchestra.run.smithi002.stdout:mgr epoch 5 is available 2022-05-13T23:06:37.871 INFO:teuthology.orchestra.run.smithi002.stdout:Setting orchestrator backend to cephadm... 2022-05-13T23:06:38.572 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:06:38 smithi002 ceph-mon[71816]: mgrmap e7: x(active, since 1.06335s) 2022-05-13T23:06:38.573 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:06:38 smithi002 ceph-mon[71816]: [13/May/2022:23:06:37] ENGINE Bus STARTING 2022-05-13T23:06:38.573 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:06:38 smithi002 ceph-mon[71816]: [13/May/2022:23:06:37] ENGINE Serving on https://172.21.15.2:7150 2022-05-13T23:06:38.573 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:06:38 smithi002 ceph-mon[71816]: [13/May/2022:23:06:37] ENGINE Bus STARTED 2022-05-13T23:06:38.574 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:06:38 smithi002 ceph-mon[71816]: from='mgr.14116 172.21.15.2:0/3171684691' entity='mgr.x' 2022-05-13T23:06:38.574 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:06:38 smithi002 ceph-mon[71816]: from='mgr.14116 172.21.15.2:0/3171684691' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-13T23:06:39.824 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:06:39 smithi002 ceph-mon[71816]: mgrmap e8: x(active, since 2s) 2022-05-13T23:06:39.824 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:06:39 smithi002 ceph-mon[71816]: from='mgr.14116 172.21.15.2:0/3171684691' entity='mgr.x' 2022-05-13T23:06:39.825 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:06:39 smithi002 ceph-mon[71816]: from='mgr.14116 172.21.15.2:0/3171684691' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-13T23:06:40.665 INFO:teuthology.orchestra.run.smithi002.stdout:Generating ssh key... 2022-05-13T23:06:40.694 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:06:40 smithi002 ceph-mon[71816]: from='client.14128 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2022-05-13T23:06:41.822 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:06:41 smithi002 ceph-mon[71816]: from='client.14130 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2022-05-13T23:06:42.121 INFO:journalctl@ceph.mgr.x.smithi002.stdout:May 13 23:06:41 smithi002 conmon[72230]: Generating public/private rsa key pair. 2022-05-13T23:06:42.121 INFO:journalctl@ceph.mgr.x.smithi002.stdout:May 13 23:06:41 smithi002 conmon[72230]: Your identification has been saved in /tmp/tmpc7kxp0t8/key. 2022-05-13T23:06:42.122 INFO:journalctl@ceph.mgr.x.smithi002.stdout:May 13 23:06:41 smithi002 conmon[72230]: Your public key has been saved in /tmp/tmpc7kxp0t8/key.pub. 2022-05-13T23:06:42.122 INFO:journalctl@ceph.mgr.x.smithi002.stdout:May 13 23:06:41 smithi002 conmon[72230]: The key fingerprint is: 2022-05-13T23:06:42.122 INFO:journalctl@ceph.mgr.x.smithi002.stdout:May 13 23:06:41 smithi002 conmon[72230]: SHA256:dRsQTAJJ5KfoVLI4iw7F43tGizckzLHuudmvMu5msDg ceph-16cf5f88-d311-11ec-be40-001a4aab830c 2022-05-13T23:06:42.122 INFO:journalctl@ceph.mgr.x.smithi002.stdout:May 13 23:06:41 smithi002 conmon[72230]: The key's randomart image is: 2022-05-13T23:06:42.123 INFO:journalctl@ceph.mgr.x.smithi002.stdout:May 13 23:06:41 smithi002 conmon[72230]: +---[RSA 3072]----+ 2022-05-13T23:06:42.123 INFO:journalctl@ceph.mgr.x.smithi002.stdout:May 13 23:06:41 smithi002 conmon[72230]: | o+o.o+. | 2022-05-13T23:06:42.123 INFO:journalctl@ceph.mgr.x.smithi002.stdout:May 13 23:06:41 smithi002 conmon[72230]: | .. ... | 2022-05-13T23:06:42.123 INFO:journalctl@ceph.mgr.x.smithi002.stdout:May 13 23:06:41 smithi002 conmon[72230]: | . o . . o | 2022-05-13T23:06:42.124 INFO:journalctl@ceph.mgr.x.smithi002.stdout:May 13 23:06:41 smithi002 conmon[72230]: | ... = o . . o | 2022-05-13T23:06:42.124 INFO:journalctl@ceph.mgr.x.smithi002.stdout:May 13 23:06:41 smithi002 conmon[72230]: | o*o+ . S . | 2022-05-13T23:06:42.124 INFO:journalctl@ceph.mgr.x.smithi002.stdout:May 13 23:06:41 smithi002 conmon[72230]: |.+=*o | 2022-05-13T23:06:42.124 INFO:journalctl@ceph.mgr.x.smithi002.stdout:May 13 23:06:41 smithi002 conmon[72230]: |++o=.. | 2022-05-13T23:06:42.124 INFO:journalctl@ceph.mgr.x.smithi002.stdout:May 13 23:06:41 smithi002 conmon[72230]: |E B=* | 2022-05-13T23:06:42.125 INFO:journalctl@ceph.mgr.x.smithi002.stdout:May 13 23:06:41 smithi002 conmon[72230]: | OOBo+. | 2022-05-13T23:06:42.125 INFO:journalctl@ceph.mgr.x.smithi002.stdout:May 13 23:06:41 smithi002 conmon[72230]: +----[SHA256]-----+ 2022-05-13T23:06:42.810 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:06:42 smithi002 ceph-mon[71816]: from='client.14132 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2022-05-13T23:06:42.810 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:06:42 smithi002 ceph-mon[71816]: Generating ssh key... 2022-05-13T23:06:42.811 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:06:42 smithi002 ceph-mon[71816]: from='mgr.14116 172.21.15.2:0/3171684691' entity='mgr.x' 2022-05-13T23:06:42.811 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:06:42 smithi002 ceph-mon[71816]: from='mgr.14116 172.21.15.2:0/3171684691' entity='mgr.x' 2022-05-13T23:06:43.769 INFO:teuthology.orchestra.run.smithi002.stdout:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2022-05-13T23:06:43.770 INFO:teuthology.orchestra.run.smithi002.stdout:Adding key to root@localhost authorized_keys... 2022-05-13T23:06:43.771 INFO:teuthology.orchestra.run.smithi002.stdout:Adding host smithi002... 2022-05-13T23:06:43.823 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:06:43 smithi002 ceph-mon[71816]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2022-05-13T23:06:45.326 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:06:45 smithi002 ceph-mon[71816]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi002", "addr": "172.21.15.2", "target": ["mon-mgr", ""]}]: dispatch 2022-05-13T23:06:46.572 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:06:46 smithi002 ceph-mon[71816]: Deploying cephadm binary to smithi002 2022-05-13T23:06:47.475 INFO:teuthology.orchestra.run.smithi002.stdout:Deploying unmanaged mon service... 2022-05-13T23:06:48.072 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:06:47 smithi002 ceph-mon[71816]: from='mgr.14116 172.21.15.2:0/3171684691' entity='mgr.x' 2022-05-13T23:06:48.073 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:06:47 smithi002 ceph-mon[71816]: Added host smithi002 2022-05-13T23:06:48.073 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:06:47 smithi002 ceph-mon[71816]: from='mgr.14116 172.21.15.2:0/3171684691' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-13T23:06:48.892 INFO:teuthology.orchestra.run.smithi002.stdout:Deploying unmanaged mgr service... 2022-05-13T23:06:49.401 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:06:49 smithi002 ceph-mon[71816]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2022-05-13T23:06:49.402 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:06:49 smithi002 ceph-mon[71816]: Saving service mon spec with placement count:5 2022-05-13T23:06:49.402 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:06:49 smithi002 ceph-mon[71816]: from='mgr.14116 172.21.15.2:0/3171684691' entity='mgr.x' 2022-05-13T23:06:51.130 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:06:50 smithi002 ceph-mon[71816]: from='mgr.14116 172.21.15.2:0/3171684691' entity='mgr.x' 2022-05-13T23:06:51.131 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:06:50 smithi002 ceph-mon[71816]: from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2022-05-13T23:06:51.131 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:06:50 smithi002 ceph-mon[71816]: Saving service mgr spec with placement count:2 2022-05-13T23:06:51.131 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:06:50 smithi002 ceph-mon[71816]: from='mgr.14116 172.21.15.2:0/3171684691' entity='mgr.x' 2022-05-13T23:06:51.132 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:06:50 smithi002 ceph-mon[71816]: from='mgr.14116 172.21.15.2:0/3171684691' entity='mgr.x' 2022-05-13T23:06:52.822 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:06:52 smithi002 ceph-mon[71816]: from='client.? 172.21.15.2:0/2402129278' entity='client.admin' 2022-05-13T23:06:53.649 INFO:teuthology.orchestra.run.smithi002.stdout:Enabling the dashboard module... 2022-05-13T23:06:54.224 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:06:54 smithi002 ceph-mon[71816]: from='client.? 172.21.15.2:0/574044750' entity='client.admin' 2022-05-13T23:06:54.224 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:06:54 smithi002 ceph-mon[71816]: from='mgr.14116 172.21.15.2:0/3171684691' entity='mgr.x' 2022-05-13T23:06:55.403 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:06:55 smithi002 ceph-mon[71816]: from='client.? 172.21.15.2:0/784699693' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2022-05-13T23:06:55.405 INFO:journalctl@ceph.mgr.x.smithi002.stdout:May 13 23:06:55 smithi002 conmon[72230]: ignoring --setuser ceph since I am not root 2022-05-13T23:06:55.405 INFO:journalctl@ceph.mgr.x.smithi002.stdout:May 13 23:06:55 smithi002 conmon[72230]: ignoring --setgroup ceph since I am not root 2022-05-13T23:06:55.983 INFO:journalctl@ceph.mgr.x.smithi002.stdout:May 13 23:06:55 smithi002 conmon[72230]: 2022-05-13T23:06:55.894+0000 7f845f0d9000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2022-05-13T23:06:55.984 INFO:journalctl@ceph.mgr.x.smithi002.stdout:May 13 23:06:55 smithi002 conmon[72230]: 2022-05-13T23:06:56.290 INFO:journalctl@ceph.mgr.x.smithi002.stdout:May 13 23:06:56 smithi002 conmon[72230]: 2022-05-13T23:06:56.037+0000 7f845f0d9000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2022-05-13T23:06:56.290 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:06:56 smithi002 ceph-mon[71816]: from='client.? 172.21.15.2:0/784699693' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2022-05-13T23:06:56.291 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:06:56 smithi002 ceph-mon[71816]: mgrmap e9: x(active, since 18s) 2022-05-13T23:06:56.572 INFO:journalctl@ceph.mgr.x.smithi002.stdout:May 13 23:06:56 smithi002 conmon[72230]: 2022-05-13T23:06:56.286+0000 7f845f0d9000 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2022-05-13T23:06:56.572 INFO:journalctl@ceph.mgr.x.smithi002.stdout:May 13 23:06:56 smithi002 conmon[72230]: 2022-05-13T23:06:56.445+0000 7f845f0d9000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2022-05-13T23:06:56.974 INFO:teuthology.orchestra.run.smithi002.stdout:Waiting for the mgr to restart... 2022-05-13T23:06:56.975 INFO:teuthology.orchestra.run.smithi002.stdout:Waiting for mgr epoch 9... 2022-05-13T23:06:56.975 INFO:journalctl@ceph.mgr.x.smithi002.stdout:May 13 23:06:56 smithi002 conmon[72230]: 2022-05-13T23:06:56.587+0000 7f845f0d9000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2022-05-13T23:06:57.296 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:06:57 smithi002 ceph-mon[71816]: from='client.? 172.21.15.2:0/1696696679' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2022-05-13T23:06:57.297 INFO:journalctl@ceph.mgr.x.smithi002.stdout:May 13 23:06:57 smithi002 conmon[72230]: 2022-05-13T23:06:57.155+0000 7f845f0d9000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2022-05-13T23:06:57.572 INFO:journalctl@ceph.mgr.x.smithi002.stdout:May 13 23:06:57 smithi002 conmon[72230]: 2022-05-13T23:06:57.328+0000 7f845f0d9000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2022-05-13T23:06:58.822 INFO:journalctl@ceph.mgr.x.smithi002.stdout:May 13 23:06:58 smithi002 conmon[72230]: 2022-05-13T23:06:58.528+0000 7f845f0d9000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2022-05-13T23:06:58.823 INFO:journalctl@ceph.mgr.x.smithi002.stdout:May 13 23:06:58 smithi002 conmon[72230]: 2022-05-13T23:06:58.659+0000 7f845f0d9000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2022-05-13T23:06:59.323 INFO:journalctl@ceph.mgr.x.smithi002.stdout:May 13 23:06:58 smithi002 conmon[72230]: 2022-05-13T23:06:58.956+0000 7f845f0d9000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2022-05-13T23:06:59.605 INFO:journalctl@ceph.mgr.x.smithi002.stdout:May 13 23:06:59 smithi002 conmon[72230]: 2022-05-13T23:06:59.337+0000 7f845f0d9000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2022-05-13T23:07:00.075 INFO:journalctl@ceph.mgr.x.smithi002.stdout:May 13 23:06:59 smithi002 conmon[72230]: 2022-05-13T23:06:59.601+0000 7f845f0d9000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2022-05-13T23:07:00.075 INFO:journalctl@ceph.mgr.x.smithi002.stdout:May 13 23:06:59 smithi002 conmon[72230]: 2022-05-13T23:06:59.829+0000 7f845f0d9000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2022-05-13T23:07:00.404 INFO:journalctl@ceph.mgr.x.smithi002.stdout:May 13 23:07:00 smithi002 conmon[72230]: 2022-05-13T23:07:00.108+0000 7f845f0d9000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2022-05-13T23:07:00.404 INFO:journalctl@ceph.mgr.x.smithi002.stdout:May 13 23:07:00 smithi002 conmon[72230]: 2022-05-13T23:07:00.261+0000 7f845f0d9000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2022-05-13T23:07:00.711 INFO:journalctl@ceph.mgr.x.smithi002.stdout:May 13 23:07:00 smithi002 conmon[72230]: 2022-05-13T23:07:00.401+0000 7f845f0d9000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2022-05-13T23:07:00.712 INFO:journalctl@ceph.mgr.x.smithi002.stdout:May 13 23:07:00 smithi002 conmon[72230]: 2022-05-13T23:07:00.558+0000 7f845f0d9000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2022-05-13T23:07:01.073 INFO:journalctl@ceph.mgr.x.smithi002.stdout:May 13 23:07:00 smithi002 conmon[72230]: 2022-05-13T23:07:00.708+0000 7f845f0d9000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2022-05-13T23:07:01.073 INFO:journalctl@ceph.mgr.x.smithi002.stdout:May 13 23:07:00 smithi002 conmon[72230]: 2022-05-13T23:07:00.865+0000 7f845f0d9000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2022-05-13T23:07:01.573 INFO:journalctl@ceph.mgr.x.smithi002.stdout:May 13 23:07:01 smithi002 conmon[72230]: 2022-05-13T23:07:01.179+0000 7f845f0d9000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2022-05-13T23:07:02.225 INFO:journalctl@ceph.mgr.x.smithi002.stdout:May 13 23:07:01 smithi002 conmon[72230]: 2022-05-13T23:07:01.928+0000 7f845f0d9000 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2022-05-13T23:07:02.574 INFO:journalctl@ceph.mgr.x.smithi002.stdout:May 13 23:07:02 smithi002 conmon[72230]: 2022-05-13T23:07:02.222+0000 7f845f0d9000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2022-05-13T23:07:03.326 INFO:journalctl@ceph.mgr.x.smithi002.stdout:May 13 23:07:02 smithi002 conmon[72230]: 2022-05-13T23:07:02.889+0000 7f845f0d9000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2022-05-13T23:07:03.327 INFO:journalctl@ceph.mgr.x.smithi002.stdout:May 13 23:07:03 smithi002 conmon[72230]: 2022-05-13T23:07:03.051+0000 7f845f0d9000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2022-05-13T23:07:06.322 INFO:journalctl@ceph.mgr.x.smithi002.stdout:May 13 23:07:06 smithi002 conmon[72230]: [13/May/2022:23:07:06] ENGINE Bus STARTING 2022-05-13T23:07:06.323 INFO:journalctl@ceph.mgr.x.smithi002.stdout:May 13 23:07:06 smithi002 conmon[72230]: [13/May/2022:23:07:06] ENGINE Serving on https://172.21.15.2:7150 2022-05-13T23:07:06.323 INFO:journalctl@ceph.mgr.x.smithi002.stdout:May 13 23:07:06 smithi002 conmon[72230]: [13/May/2022:23:07:06] ENGINE Bus STARTED 2022-05-13T23:07:06.323 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:07:05 smithi002 ceph-mon[71816]: Active manager daemon x restarted 2022-05-13T23:07:06.324 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:07:05 smithi002 ceph-mon[71816]: Activating manager daemon x 2022-05-13T23:07:06.324 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:07:05 smithi002 ceph-mon[71816]: osdmap e3: 0 total, 0 up, 0 in 2022-05-13T23:07:06.324 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:07:05 smithi002 ceph-mon[71816]: mgrmap e10: x(active, starting, since 0.161676s) 2022-05-13T23:07:06.324 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:07:05 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-05-13T23:07:06.325 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:07:05 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2022-05-13T23:07:06.325 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:07:05 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2022-05-13T23:07:06.325 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:07:05 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2022-05-13T23:07:06.326 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:07:05 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2022-05-13T23:07:06.326 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:07:05 smithi002 ceph-mon[71816]: Manager daemon x is now available 2022-05-13T23:07:06.326 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:07:05 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-13T23:07:06.326 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:07:05 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-13T23:07:06.326 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:07:05 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-05-13T23:07:06.327 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:07:05 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-05-13T23:07:06.585 INFO:teuthology.orchestra.run.smithi002.stdout:mgr epoch 9 is available 2022-05-13T23:07:06.585 INFO:teuthology.orchestra.run.smithi002.stdout:Generating a dashboard self-signed certificate... 2022-05-13T23:07:07.322 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:07:07 smithi002 ceph-mon[71816]: [13/May/2022:23:07:06] ENGINE Bus STARTING 2022-05-13T23:07:07.323 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:07:07 smithi002 ceph-mon[71816]: [13/May/2022:23:07:06] ENGINE Serving on https://172.21.15.2:7150 2022-05-13T23:07:07.323 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:07:07 smithi002 ceph-mon[71816]: [13/May/2022:23:07:06] ENGINE Bus STARTED 2022-05-13T23:07:07.323 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:07:07 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:07:07.324 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:07:07 smithi002 ceph-mon[71816]: mgrmap e11: x(active, since 1.16622s) 2022-05-13T23:07:07.324 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:07:07 smithi002 ceph-mon[71816]: from='client.14152 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2022-05-13T23:07:07.324 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:07:07 smithi002 ceph-mon[71816]: from='client.14152 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2022-05-13T23:07:08.297 INFO:teuthology.orchestra.run.smithi002.stdout:Creating initial admin user... 2022-05-13T23:07:08.322 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:07:08 smithi002 ceph-mon[71816]: mgrmap e12: x(active, since 2s) 2022-05-13T23:07:08.323 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:07:08 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:07:08.323 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:07:08 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:07:09.322 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:07:09 smithi002 ceph-mon[71816]: from='client.14160 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2022-05-13T23:07:09.323 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:07:09 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:07:09.996 INFO:teuthology.orchestra.run.smithi002.stdout:Fetching dashboard port number... 2022-05-13T23:07:10.364 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:07:10 smithi002 ceph-mon[71816]: from='client.14162 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2022-05-13T23:07:10.364 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:07:10 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:07:11.562 INFO:teuthology.orchestra.run.smithi002.stdout:Ceph Dashboard is now available at: 2022-05-13T23:07:11.563 INFO:teuthology.orchestra.run.smithi002.stdout: 2022-05-13T23:07:11.563 INFO:teuthology.orchestra.run.smithi002.stdout: URL: https://smithi002.front.sepia.ceph.com:8443/ 2022-05-13T23:07:11.565 INFO:teuthology.orchestra.run.smithi002.stdout: User: admin 2022-05-13T23:07:11.566 INFO:teuthology.orchestra.run.smithi002.stdout: Password: 0bqcvtrqp1 2022-05-13T23:07:11.566 INFO:teuthology.orchestra.run.smithi002.stdout: 2022-05-13T23:07:11.566 INFO:teuthology.orchestra.run.smithi002.stdout:Saving cluster configuration to /var/lib/ceph/16cf5f88-d311-11ec-be40-001a4aab830c/config directory 2022-05-13T23:07:11.567 INFO:teuthology.orchestra.run.smithi002.stdout:Enabling autotune for osd_memory_target 2022-05-13T23:07:11.823 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:07:11 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:07:11.824 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:07:11 smithi002 ceph-mon[71816]: from='client.? 172.21.15.2:0/3458638183' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2022-05-13T23:07:12.822 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:07:12 smithi002 ceph-mon[71816]: mgrmap e13: x(active, since 6s) 2022-05-13T23:07:14.766 INFO:teuthology.orchestra.run.smithi002.stdout:You can access the Ceph CLI as following in case of multi-cluster or non-default config: 2022-05-13T23:07:14.767 INFO:teuthology.orchestra.run.smithi002.stdout: 2022-05-13T23:07:14.767 INFO:teuthology.orchestra.run.smithi002.stdout: sudo /home/ubuntu/cephtest/cephadm shell --fsid 16cf5f88-d311-11ec-be40-001a4aab830c -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2022-05-13T23:07:14.767 INFO:teuthology.orchestra.run.smithi002.stdout: 2022-05-13T23:07:14.768 INFO:teuthology.orchestra.run.smithi002.stdout:Or, if you are only running a single cluster on this host: 2022-05-13T23:07:14.768 INFO:teuthology.orchestra.run.smithi002.stdout: 2022-05-13T23:07:14.768 INFO:teuthology.orchestra.run.smithi002.stdout: sudo /home/ubuntu/cephtest/cephadm shell 2022-05-13T23:07:14.768 INFO:teuthology.orchestra.run.smithi002.stdout: 2022-05-13T23:07:14.768 INFO:teuthology.orchestra.run.smithi002.stdout:Please consider enabling telemetry to help improve Ceph: 2022-05-13T23:07:14.769 INFO:teuthology.orchestra.run.smithi002.stdout: 2022-05-13T23:07:14.769 INFO:teuthology.orchestra.run.smithi002.stdout: ceph telemetry on 2022-05-13T23:07:14.769 INFO:teuthology.orchestra.run.smithi002.stdout: 2022-05-13T23:07:14.769 INFO:teuthology.orchestra.run.smithi002.stdout:For more information see: 2022-05-13T23:07:14.770 INFO:teuthology.orchestra.run.smithi002.stdout: 2022-05-13T23:07:14.770 INFO:teuthology.orchestra.run.smithi002.stdout: https://docs.ceph.com/docs/master/mgr/telemetry/ 2022-05-13T23:07:14.770 INFO:teuthology.orchestra.run.smithi002.stdout: 2022-05-13T23:07:14.771 INFO:teuthology.orchestra.run.smithi002.stdout:Bootstrap complete. 2022-05-13T23:07:14.838 INFO:tasks.cephadm:Fetching config... 2022-05-13T23:07:14.840 DEBUG:teuthology.orchestra.run.smithi002:> set -ex 2022-05-13T23:07:14.840 DEBUG:teuthology.orchestra.run.smithi002:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2022-05-13T23:07:14.931 INFO:tasks.cephadm:Fetching client.admin keyring... 2022-05-13T23:07:14.932 DEBUG:teuthology.orchestra.run.smithi002:> set -ex 2022-05-13T23:07:14.932 DEBUG:teuthology.orchestra.run.smithi002:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2022-05-13T23:07:14.984 INFO:tasks.cephadm:Fetching mon keyring... 2022-05-13T23:07:14.984 DEBUG:teuthology.orchestra.run.smithi002:> set -ex 2022-05-13T23:07:14.985 DEBUG:teuthology.orchestra.run.smithi002:> sudo dd if=/var/lib/ceph/16cf5f88-d311-11ec-be40-001a4aab830c/mon.a/keyring of=/dev/stdout 2022-05-13T23:07:15.106 INFO:tasks.cephadm:Fetching pub ssh key... 2022-05-13T23:07:15.106 DEBUG:teuthology.orchestra.run.smithi002:> set -ex 2022-05-13T23:07:15.107 DEBUG:teuthology.orchestra.run.smithi002:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2022-05-13T23:07:15.155 INFO:tasks.cephadm:Installing pub ssh key for root users... 2022-05-13T23:07:15.156 DEBUG:teuthology.orchestra.run.smithi002:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDAVN6Yya4KOFANgejNIItAtmrM+uWZkcoVehsZxGDOB8u+mDiJJBlFd64MRSsmKn1vhGwZ392qa4jZwwJc9ou199J5+dTjUls9XQmfmDfsfaoiq4FGGgzXxskjZD5lmO7eTHuZ+SG1fQgJGCI6SAmMws3paKdiNhxX3ViwIV2cOofnNJei7kA8idwtmEVCnGatj7L17EUcUbRXCz5LLjMXP7QA6tI7IV/sYeu/l15oUAz9cO9qgx1d0LKxNwP14NMTJYmgcU7s9gfZ44RZeUL2MJrmiD8GeANFUcadySNsCk8Br4SNbLuoKQbjnk4AI6Le2Co7LJWPWULwI5pKYU1ndv0qZ1He830XRCr4a5XF0B6GuepbdDM/DWxKiyIkk1cbhYBkueEBpVH30Kj4asNt79z8SfpuTdPVtWqj6nhICYOSN5iDtpAvfKwZkNFvuUP6wZexK5AxVSMP87Rv95WR3+yWi1BpJ0kT0myYpnPB0I86hPfmjHDB24PWF6eQDFM= ceph-16cf5f88-d311-11ec-be40-001a4aab830c' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2022-05-13T23:07:15.313 INFO:teuthology.orchestra.run.smithi002.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDAVN6Yya4KOFANgejNIItAtmrM+uWZkcoVehsZxGDOB8u+mDiJJBlFd64MRSsmKn1vhGwZ392qa4jZwwJc9ou199J5+dTjUls9XQmfmDfsfaoiq4FGGgzXxskjZD5lmO7eTHuZ+SG1fQgJGCI6SAmMws3paKdiNhxX3ViwIV2cOofnNJei7kA8idwtmEVCnGatj7L17EUcUbRXCz5LLjMXP7QA6tI7IV/sYeu/l15oUAz9cO9qgx1d0LKxNwP14NMTJYmgcU7s9gfZ44RZeUL2MJrmiD8GeANFUcadySNsCk8Br4SNbLuoKQbjnk4AI6Le2Co7LJWPWULwI5pKYU1ndv0qZ1He830XRCr4a5XF0B6GuepbdDM/DWxKiyIkk1cbhYBkueEBpVH30Kj4asNt79z8SfpuTdPVtWqj6nhICYOSN5iDtpAvfKwZkNFvuUP6wZexK5AxVSMP87Rv95WR3+yWi1BpJ0kT0myYpnPB0I86hPfmjHDB24PWF6eQDFM= ceph-16cf5f88-d311-11ec-be40-001a4aab830c 2022-05-13T23:07:15.349 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:07:15 smithi002 ceph-mon[71816]: from='client.? 172.21.15.2:0/3167898562' entity='client.admin' 2022-05-13T23:07:15.354 DEBUG:teuthology.orchestra.run.smithi047:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDAVN6Yya4KOFANgejNIItAtmrM+uWZkcoVehsZxGDOB8u+mDiJJBlFd64MRSsmKn1vhGwZ392qa4jZwwJc9ou199J5+dTjUls9XQmfmDfsfaoiq4FGGgzXxskjZD5lmO7eTHuZ+SG1fQgJGCI6SAmMws3paKdiNhxX3ViwIV2cOofnNJei7kA8idwtmEVCnGatj7L17EUcUbRXCz5LLjMXP7QA6tI7IV/sYeu/l15oUAz9cO9qgx1d0LKxNwP14NMTJYmgcU7s9gfZ44RZeUL2MJrmiD8GeANFUcadySNsCk8Br4SNbLuoKQbjnk4AI6Le2Co7LJWPWULwI5pKYU1ndv0qZ1He830XRCr4a5XF0B6GuepbdDM/DWxKiyIkk1cbhYBkueEBpVH30Kj4asNt79z8SfpuTdPVtWqj6nhICYOSN5iDtpAvfKwZkNFvuUP6wZexK5AxVSMP87Rv95WR3+yWi1BpJ0kT0myYpnPB0I86hPfmjHDB24PWF6eQDFM= ceph-16cf5f88-d311-11ec-be40-001a4aab830c' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2022-05-13T23:07:15.397 INFO:teuthology.orchestra.run.smithi047.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDAVN6Yya4KOFANgejNIItAtmrM+uWZkcoVehsZxGDOB8u+mDiJJBlFd64MRSsmKn1vhGwZ392qa4jZwwJc9ou199J5+dTjUls9XQmfmDfsfaoiq4FGGgzXxskjZD5lmO7eTHuZ+SG1fQgJGCI6SAmMws3paKdiNhxX3ViwIV2cOofnNJei7kA8idwtmEVCnGatj7L17EUcUbRXCz5LLjMXP7QA6tI7IV/sYeu/l15oUAz9cO9qgx1d0LKxNwP14NMTJYmgcU7s9gfZ44RZeUL2MJrmiD8GeANFUcadySNsCk8Br4SNbLuoKQbjnk4AI6Le2Co7LJWPWULwI5pKYU1ndv0qZ1He830XRCr4a5XF0B6GuepbdDM/DWxKiyIkk1cbhYBkueEBpVH30Kj4asNt79z8SfpuTdPVtWqj6nhICYOSN5iDtpAvfKwZkNFvuUP6wZexK5AxVSMP87Rv95WR3+yWi1BpJ0kT0myYpnPB0I86hPfmjHDB24PWF6eQDFM= ceph-16cf5f88-d311-11ec-be40-001a4aab830c 2022-05-13T23:07:15.411 DEBUG:teuthology.orchestra.run.smithi115:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDAVN6Yya4KOFANgejNIItAtmrM+uWZkcoVehsZxGDOB8u+mDiJJBlFd64MRSsmKn1vhGwZ392qa4jZwwJc9ou199J5+dTjUls9XQmfmDfsfaoiq4FGGgzXxskjZD5lmO7eTHuZ+SG1fQgJGCI6SAmMws3paKdiNhxX3ViwIV2cOofnNJei7kA8idwtmEVCnGatj7L17EUcUbRXCz5LLjMXP7QA6tI7IV/sYeu/l15oUAz9cO9qgx1d0LKxNwP14NMTJYmgcU7s9gfZ44RZeUL2MJrmiD8GeANFUcadySNsCk8Br4SNbLuoKQbjnk4AI6Le2Co7LJWPWULwI5pKYU1ndv0qZ1He830XRCr4a5XF0B6GuepbdDM/DWxKiyIkk1cbhYBkueEBpVH30Kj4asNt79z8SfpuTdPVtWqj6nhICYOSN5iDtpAvfKwZkNFvuUP6wZexK5AxVSMP87Rv95WR3+yWi1BpJ0kT0myYpnPB0I86hPfmjHDB24PWF6eQDFM= ceph-16cf5f88-d311-11ec-be40-001a4aab830c' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2022-05-13T23:07:15.452 INFO:teuthology.orchestra.run.smithi115.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDAVN6Yya4KOFANgejNIItAtmrM+uWZkcoVehsZxGDOB8u+mDiJJBlFd64MRSsmKn1vhGwZ392qa4jZwwJc9ou199J5+dTjUls9XQmfmDfsfaoiq4FGGgzXxskjZD5lmO7eTHuZ+SG1fQgJGCI6SAmMws3paKdiNhxX3ViwIV2cOofnNJei7kA8idwtmEVCnGatj7L17EUcUbRXCz5LLjMXP7QA6tI7IV/sYeu/l15oUAz9cO9qgx1d0LKxNwP14NMTJYmgcU7s9gfZ44RZeUL2MJrmiD8GeANFUcadySNsCk8Br4SNbLuoKQbjnk4AI6Le2Co7LJWPWULwI5pKYU1ndv0qZ1He830XRCr4a5XF0B6GuepbdDM/DWxKiyIkk1cbhYBkueEBpVH30Kj4asNt79z8SfpuTdPVtWqj6nhICYOSN5iDtpAvfKwZkNFvuUP6wZexK5AxVSMP87Rv95WR3+yWi1BpJ0kT0myYpnPB0I86hPfmjHDB24PWF6eQDFM= ceph-16cf5f88-d311-11ec-be40-001a4aab830c 2022-05-13T23:07:15.464 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:23b0658391ce94ca40799469eec00c28b298816c shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 16cf5f88-d311-11ec-be40-001a4aab830c -- ceph config set mgr mgr/cephadm/allow_ptrace true 2022-05-13T23:07:16.140 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/16cf5f88-d311-11ec-be40-001a4aab830c/mon.a/config 2022-05-13T23:07:18.565 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:07:18.560+0000 7fd0e8f0f700 1 -- 172.21.15.2:0/1300450081 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd0e40fd240 msgr2=0x7fd0e40fd650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:07:18.565 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:07:18.560+0000 7fd0e8f0f700 1 --2- 172.21.15.2:0/1300450081 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd0e40fd240 0x7fd0e40fd650 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fd0d4004770 tx=0x7fd0d4009cb0 comp rx=0 tx=0).stop 2022-05-13T23:07:18.566 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:07:18.560+0000 7fd0e8f0f700 1 -- 172.21.15.2:0/1300450081 shutdown_connections 2022-05-13T23:07:18.566 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:07:18.560+0000 7fd0e8f0f700 1 --2- 172.21.15.2:0/1300450081 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd0e40fd240 0x7fd0e40fd650 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:07:18.566 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:07:18.560+0000 7fd0e8f0f700 1 -- 172.21.15.2:0/1300450081 >> 172.21.15.2:0/1300450081 conn(0x7fd0e40f8af0 msgr2=0x7fd0e40faed0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-05-13T23:07:18.566 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:07:18.560+0000 7fd0e8f0f700 1 -- 172.21.15.2:0/1300450081 shutdown_connections 2022-05-13T23:07:18.567 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:07:18.560+0000 7fd0e8f0f700 1 -- 172.21.15.2:0/1300450081 wait complete. 2022-05-13T23:07:18.567 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:07:18.561+0000 7fd0e8f0f700 1 Processor -- start 2022-05-13T23:07:18.567 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:07:18.561+0000 7fd0e8f0f700 1 -- start start 2022-05-13T23:07:18.568 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:07:18.561+0000 7fd0e8f0f700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd0e40fd240 0x7fd0e410d340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:07:18.568 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:07:18.561+0000 7fd0e8f0f700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0e410d880 con 0x7fd0e40fd240 2022-05-13T23:07:18.569 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:07:18.562+0000 7fd0e259c700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd0e40fd240 0x7fd0e410d340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:07:18.569 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:07:18.562+0000 7fd0e259c700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd0e40fd240 0x7fd0e410d340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:43976/0 (socket says 172.21.15.2:43976) 2022-05-13T23:07:18.569 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:07:18.562+0000 7fd0e259c700 1 -- 172.21.15.2:0/350737345 learned_addr learned my addr 172.21.15.2:0/350737345 (peer_addr_for_me v2:172.21.15.2:0/0) 2022-05-13T23:07:18.570 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:07:18.562+0000 7fd0e259c700 1 -- 172.21.15.2:0/350737345 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0d4005040 con 0x7fd0e40fd240 2022-05-13T23:07:18.570 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:07:18.563+0000 7fd0e259c700 1 --2- 172.21.15.2:0/350737345 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd0e40fd240 0x7fd0e410d340 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fd0d4009810 tx=0x7fd0d4009cb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-05-13T23:07:18.570 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:07:18.563+0000 7fd0d2ffd700 1 -- 172.21.15.2:0/350737345 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd0d400e050 con 0x7fd0e40fd240 2022-05-13T23:07:18.570 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:07:18.563+0000 7fd0d2ffd700 1 -- 172.21.15.2:0/350737345 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1087+0+0 (secure 0 0 0) 0x7fd0d4013bc0 con 0x7fd0e40fd240 2022-05-13T23:07:18.571 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:07:18.563+0000 7fd0e8f0f700 1 -- 172.21.15.2:0/350737345 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd0e410da80 con 0x7fd0e40fd240 2022-05-13T23:07:18.571 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:07:18.563+0000 7fd0d2ffd700 1 -- 172.21.15.2:0/350737345 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd0d40123c0 con 0x7fd0e40fd240 2022-05-13T23:07:18.571 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:07:18.563+0000 7fd0e8f0f700 1 -- 172.21.15.2:0/350737345 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd0e410e720 con 0x7fd0e40fd240 2022-05-13T23:07:18.573 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:07:18.564+0000 7fd0e8f0f700 1 -- 172.21.15.2:0/350737345 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd0e405db10 con 0x7fd0e40fd240 2022-05-13T23:07:18.574 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:07:18.564+0000 7fd0d2ffd700 1 -- 172.21.15.2:0/350737345 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 13) v1 ==== 41898+0+0 (secure 0 0 0) 0x7fd0d402b070 con 0x7fd0e40fd240 2022-05-13T23:07:18.574 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:07:18.565+0000 7fd0d2ffd700 1 --2- 172.21.15.2:0/350737345 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7fd0cc034c50 0x7fd0cc037100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:07:18.574 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:07:18.565+0000 7fd0d2ffd700 1 -- 172.21.15.2:0/350737345 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7fd0d4047760 con 0x7fd0e40fd240 2022-05-13T23:07:18.575 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:07:18.565+0000 7fd0e1d9b700 1 --2- 172.21.15.2:0/350737345 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7fd0cc034c50 0x7fd0cc037100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:07:18.575 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:07:18.566+0000 7fd0e1d9b700 1 --2- 172.21.15.2:0/350737345 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7fd0cc034c50 0x7fd0cc037100 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fd0d8009780 tx=0x7fd0d8006cc0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-05-13T23:07:18.575 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:07:18.570+0000 7fd0d2ffd700 1 -- 172.21.15.2:0/350737345 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+165215 (secure 0 0 0) 0x7fd0d4022540 con 0x7fd0e40fd240 2022-05-13T23:07:18.761 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:07:18.759+0000 7fd0e8f0f700 1 -- 172.21.15.2:0/350737345 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/allow_ptrace}] v 0) v1 -- 0x7fd0e404a430 con 0x7fd0e40fd240 2022-05-13T23:07:18.764 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:07:18.762+0000 7fd0d2ffd700 1 -- 172.21.15.2:0/350737345 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/allow_ptrace}]=0 v9)=0 v9) v1 ==== 125+0+0 (secure 0 0 0) 0x7fd0d40192b0 con 0x7fd0e40fd240 2022-05-13T23:07:18.767 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:07:18.765+0000 7fd0e8f0f700 1 -- 172.21.15.2:0/350737345 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7fd0cc034c50 msgr2=0x7fd0cc037100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:07:18.767 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:07:18.765+0000 7fd0e8f0f700 1 --2- 172.21.15.2:0/350737345 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7fd0cc034c50 0x7fd0cc037100 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fd0d8009780 tx=0x7fd0d8006cc0 comp rx=0 tx=0).stop 2022-05-13T23:07:18.768 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:07:18.765+0000 7fd0e8f0f700 1 -- 172.21.15.2:0/350737345 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd0e40fd240 msgr2=0x7fd0e410d340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:07:18.768 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:07:18.765+0000 7fd0e8f0f700 1 --2- 172.21.15.2:0/350737345 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd0e40fd240 0x7fd0e410d340 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fd0d4009810 tx=0x7fd0d4009cb0 comp rx=0 tx=0).stop 2022-05-13T23:07:18.768 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:07:18.765+0000 7fd0e8f0f700 1 -- 172.21.15.2:0/350737345 shutdown_connections 2022-05-13T23:07:18.769 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:07:18.765+0000 7fd0e8f0f700 1 --2- 172.21.15.2:0/350737345 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7fd0cc034c50 0x7fd0cc037100 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:07:18.769 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:07:18.765+0000 7fd0e8f0f700 1 --2- 172.21.15.2:0/350737345 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd0e40fd240 0x7fd0e410d340 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:07:18.769 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:07:18.765+0000 7fd0e8f0f700 1 -- 172.21.15.2:0/350737345 >> 172.21.15.2:0/350737345 conn(0x7fd0e40f8af0 msgr2=0x7fd0e40fa7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-05-13T23:07:18.769 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:07:18.766+0000 7fd0e8f0f700 1 -- 172.21.15.2:0/350737345 shutdown_connections 2022-05-13T23:07:18.770 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:07:18.766+0000 7fd0e8f0f700 1 -- 172.21.15.2:0/350737345 wait complete. 2022-05-13T23:07:19.066 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:07:18 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:07:19.066 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:07:18 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi002", "name": "osd_memory_target"}]: dispatch 2022-05-13T23:07:19.067 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:07:18 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:07:19.067 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:07:18 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-13T23:07:19.067 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:07:18 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:07:19.067 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:07:18 smithi002 ceph-mon[71816]: from='client.? 172.21.15.2:0/350737345' entity='client.admin' 2022-05-13T23:07:19.067 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:07:18 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-13T23:07:19.068 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:07:18 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:07:19.339 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2022-05-13T23:07:19.339 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:23b0658391ce94ca40799469eec00c28b298816c shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 16cf5f88-d311-11ec-be40-001a4aab830c -- ceph orch client-keyring set client.admin '*' --mode 0755 2022-05-13T23:07:20.009 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/16cf5f88-d311-11ec-be40-001a4aab830c/mon.a/config 2022-05-13T23:07:21.860 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:07:21.857+0000 7f0f7ffff700 1 -- 172.21.15.2:0/2415951955 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0f800642f0 msgr2=0x7f0f800fda40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:07:21.860 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:07:21.857+0000 7f0f7ffff700 1 --2- 172.21.15.2:0/2415951955 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0f800642f0 0x7f0f800fda40 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f0f70004770 tx=0x7f0f70009cb0 comp rx=0 tx=0).stop 2022-05-13T23:07:21.861 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:07:21.857+0000 7f0f7ffff700 1 -- 172.21.15.2:0/2415951955 shutdown_connections 2022-05-13T23:07:21.862 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:07:21.857+0000 7f0f7ffff700 1 --2- 172.21.15.2:0/2415951955 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0f800642f0 0x7f0f800fda40 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:07:21.864 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:07:21.858+0000 7f0f7ffff700 1 -- 172.21.15.2:0/2415951955 >> 172.21.15.2:0/2415951955 conn(0x7f0f800f54b0 msgr2=0x7f0f800f78e0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-05-13T23:07:21.864 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:07:21.858+0000 7f0f7ffff700 1 -- 172.21.15.2:0/2415951955 shutdown_connections 2022-05-13T23:07:21.865 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:07:21.858+0000 7f0f7ffff700 1 -- 172.21.15.2:0/2415951955 wait complete. 2022-05-13T23:07:21.865 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:07:21.859+0000 7f0f7ffff700 1 Processor -- start 2022-05-13T23:07:21.865 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:07:21.859+0000 7f0f7ffff700 1 -- start start 2022-05-13T23:07:21.870 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:07:21.859+0000 7f0f7ffff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0f800642f0 0x7f0f8010d140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:07:21.870 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:07:21.859+0000 7f0f7ffff700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f8010d680 con 0x7f0f800642f0 2022-05-13T23:07:21.871 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:07:21.859+0000 7f0f7effd700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0f800642f0 0x7f0f8010d140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:07:21.871 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:07:21.859+0000 7f0f7effd700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0f800642f0 0x7f0f8010d140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:59212/0 (socket says 172.21.15.2:59212) 2022-05-13T23:07:21.872 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:07:21.860+0000 7f0f7effd700 1 -- 172.21.15.2:0/3933896881 learned_addr learned my addr 172.21.15.2:0/3933896881 (peer_addr_for_me v2:172.21.15.2:0/0) 2022-05-13T23:07:21.872 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:07:21.860+0000 7f0f7effd700 1 -- 172.21.15.2:0/3933896881 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f70005040 con 0x7f0f800642f0 2022-05-13T23:07:21.874 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:07:21.860+0000 7f0f7effd700 1 --2- 172.21.15.2:0/3933896881 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0f800642f0 0x7f0f8010d140 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f0f70009820 tx=0x7f0f70009cb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-05-13T23:07:21.874 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:07:21.860+0000 7f0f677fe700 1 -- 172.21.15.2:0/3933896881 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f0f7000e050 con 0x7f0f800642f0 2022-05-13T23:07:21.874 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:07:21.860+0000 7f0f677fe700 1 -- 172.21.15.2:0/3933896881 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1087+0+0 (secure 0 0 0) 0x7f0f70013b70 con 0x7f0f800642f0 2022-05-13T23:07:21.875 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:07:21.860+0000 7f0f677fe700 1 -- 172.21.15.2:0/3933896881 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f0f700123c0 con 0x7f0f800642f0 2022-05-13T23:07:21.875 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:07:21.860+0000 7f0f7ffff700 1 -- 172.21.15.2:0/3933896881 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0f8010d880 con 0x7f0f800642f0 2022-05-13T23:07:21.876 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:07:21.861+0000 7f0f7ffff700 1 -- 172.21.15.2:0/3933896881 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0f8010e520 con 0x7f0f800642f0 2022-05-13T23:07:21.876 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:07:21.862+0000 7f0f677fe700 1 -- 172.21.15.2:0/3933896881 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 13) v1 ==== 41898+0+0 (secure 0 0 0) 0x7f0f7002b070 con 0x7f0f800642f0 2022-05-13T23:07:21.877 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:07:21.862+0000 7f0f7ffff700 1 -- 172.21.15.2:0/3933896881 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0f8005db10 con 0x7f0f800642f0 2022-05-13T23:07:21.879 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:07:21.862+0000 7f0f677fe700 1 --2- 172.21.15.2:0/3933896881 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f0f68034ca0 0x7f0f68037150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:07:21.879 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:07:21.862+0000 7f0f677fe700 1 -- 172.21.15.2:0/3933896881 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f0f70047780 con 0x7f0f800642f0 2022-05-13T23:07:21.879 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:07:21.863+0000 7f0f7e7fc700 1 --2- 172.21.15.2:0/3933896881 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f0f68034ca0 0x7f0f68037150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:07:21.879 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:07:21.863+0000 7f0f7e7fc700 1 --2- 172.21.15.2:0/3933896881 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f0f68034ca0 0x7f0f68037150 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f0f74009780 tx=0x7f0f74006cc0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-05-13T23:07:21.880 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:07:21.868+0000 7f0f677fe700 1 -- 172.21.15.2:0/3933896881 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+165215 (secure 0 0 0) 0x7f0f7001b3f0 con 0x7f0f800642f0 2022-05-13T23:07:22.066 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:07:22.063+0000 7f0f7ffff700 1 -- 172.21.15.2:0/3933896881 --> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] -- mgr_command(tid 0: {"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}) v1 -- 0x7f0f800f9f50 con 0x7f0f68034ca0 2022-05-13T23:07:22.068 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:07:22.067+0000 7f0f677fe700 1 -- 172.21.15.2:0/3933896881 <== mgr.14148 v2:172.21.15.2:6800/2788339627 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f0f800f9f50 con 0x7f0f68034ca0 2022-05-13T23:07:22.071 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:07:22.069+0000 7f0f7ffff700 1 -- 172.21.15.2:0/3933896881 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f0f68034ca0 msgr2=0x7f0f68037150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:07:22.071 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:07:22.069+0000 7f0f7ffff700 1 --2- 172.21.15.2:0/3933896881 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f0f68034ca0 0x7f0f68037150 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f0f74009780 tx=0x7f0f74006cc0 comp rx=0 tx=0).stop 2022-05-13T23:07:22.071 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:07:22.069+0000 7f0f7ffff700 1 -- 172.21.15.2:0/3933896881 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0f800642f0 msgr2=0x7f0f8010d140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:07:22.072 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:07:22.069+0000 7f0f7ffff700 1 --2- 172.21.15.2:0/3933896881 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0f800642f0 0x7f0f8010d140 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f0f70009820 tx=0x7f0f70009cb0 comp rx=0 tx=0).stop 2022-05-13T23:07:22.072 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:07:22.069+0000 7f0f7ffff700 1 -- 172.21.15.2:0/3933896881 shutdown_connections 2022-05-13T23:07:22.072 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:07:22.069+0000 7f0f7ffff700 1 --2- 172.21.15.2:0/3933896881 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f0f68034ca0 0x7f0f68037150 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:07:22.072 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:07:22.069+0000 7f0f7ffff700 1 --2- 172.21.15.2:0/3933896881 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0f800642f0 0x7f0f8010d140 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:07:22.073 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:07:22.069+0000 7f0f7ffff700 1 -- 172.21.15.2:0/3933896881 >> 172.21.15.2:0/3933896881 conn(0x7f0f800f54b0 msgr2=0x7f0f800f9840 unknown :-1 s=STATE_NONE l=0).mark_down 2022-05-13T23:07:22.073 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:07:22.070+0000 7f0f7ffff700 1 -- 172.21.15.2:0/3933896881 shutdown_connections 2022-05-13T23:07:22.073 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:07:22.070+0000 7f0f7ffff700 1 -- 172.21.15.2:0/3933896881 wait complete. 2022-05-13T23:07:22.674 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi047 2022-05-13T23:07:22.675 DEBUG:teuthology.orchestra.run.smithi047:> set -ex 2022-05-13T23:07:22.676 DEBUG:teuthology.orchestra.run.smithi047:> dd of=/etc/ceph/ceph.conf 2022-05-13T23:07:22.697 DEBUG:teuthology.orchestra.run.smithi047:> set -ex 2022-05-13T23:07:22.699 DEBUG:teuthology.orchestra.run.smithi047:> dd of=/etc/ceph/ceph.client.admin.keyring 2022-05-13T23:07:22.754 INFO:tasks.cephadm:Adding host smithi047 to orchestrator... 2022-05-13T23:07:22.755 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:23b0658391ce94ca40799469eec00c28b298816c shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 16cf5f88-d311-11ec-be40-001a4aab830c -- ceph orch host add smithi047 2022-05-13T23:07:23.186 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:07:23 smithi002 ceph-mon[71816]: from='client.14172 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2022-05-13T23:07:23.187 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:07:23 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:07:23.187 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:07:23 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-13T23:07:23.187 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:07:23 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-13T23:07:23.188 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:07:23 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-13T23:07:23.188 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:07:23 smithi002 ceph-mon[71816]: Updating smithi002:/etc/ceph/ceph.conf 2022-05-13T23:07:23.447 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/16cf5f88-d311-11ec-be40-001a4aab830c/mon.a/config 2022-05-13T23:07:24.323 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:07:24 smithi002 ceph-mon[71816]: Updating smithi002:/etc/ceph/ceph.client.admin.keyring 2022-05-13T23:07:24.323 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:07:24 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:07:24.324 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:07:24 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:07:25.303 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:07:25.299+0000 7f84fad4c700 1 -- 172.21.15.2:0/127189988 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f84f4101020 msgr2=0x7f84f41013f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:07:25.303 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:07:25.299+0000 7f84fad4c700 1 --2- 172.21.15.2:0/127189988 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f84f4101020 0x7f84f41013f0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f84e8004770 tx=0x7f84e8009cb0 comp rx=0 tx=0).stop 2022-05-13T23:07:25.304 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:07:25.299+0000 7f84fad4c700 1 -- 172.21.15.2:0/127189988 shutdown_connections 2022-05-13T23:07:25.304 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:07:25.299+0000 7f84fad4c700 1 --2- 172.21.15.2:0/127189988 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f84f4101020 0x7f84f41013f0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:07:25.305 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:07:25.299+0000 7f84fad4c700 1 -- 172.21.15.2:0/127189988 >> 172.21.15.2:0/127189988 conn(0x7f84f40f6c60 msgr2=0x7f84f40f9070 unknown :-1 s=STATE_NONE l=0).mark_down 2022-05-13T23:07:25.305 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:07:25.299+0000 7f84fad4c700 1 -- 172.21.15.2:0/127189988 shutdown_connections 2022-05-13T23:07:25.306 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:07:25.299+0000 7f84fad4c700 1 -- 172.21.15.2:0/127189988 wait complete. 2022-05-13T23:07:25.306 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:07:25.300+0000 7f84fad4c700 1 Processor -- start 2022-05-13T23:07:25.307 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:07:25.300+0000 7f84fad4c700 1 -- start start 2022-05-13T23:07:25.307 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:07:25.300+0000 7f84fad4c700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f84f4101020 0x7f84f41133c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:07:25.309 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:07:25.300+0000 7f84fad4c700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84f4113900 con 0x7f84f4101020 2022-05-13T23:07:25.309 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:07:25.301+0000 7f84f8ae8700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f84f4101020 0x7f84f41133c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:07:25.309 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:07:25.301+0000 7f84f8ae8700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f84f4101020 0x7f84f41133c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:59220/0 (socket says 172.21.15.2:59220) 2022-05-13T23:07:25.310 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:07:25.301+0000 7f84f8ae8700 1 -- 172.21.15.2:0/2281217149 learned_addr learned my addr 172.21.15.2:0/2281217149 (peer_addr_for_me v2:172.21.15.2:0/0) 2022-05-13T23:07:25.310 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:07:25.301+0000 7f84f8ae8700 1 -- 172.21.15.2:0/2281217149 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84e8005040 con 0x7f84f4101020 2022-05-13T23:07:25.311 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:07:25.301+0000 7f84f8ae8700 1 --2- 172.21.15.2:0/2281217149 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f84f4101020 0x7f84f41133c0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f84e80097f0 tx=0x7f84e8009cb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-05-13T23:07:25.314 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:07:25.302+0000 7f84f17fa700 1 -- 172.21.15.2:0/2281217149 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f84e800e050 con 0x7f84f4101020 2022-05-13T23:07:25.315 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:07:25.302+0000 7f84f17fa700 1 -- 172.21.15.2:0/2281217149 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1087+0+0 (secure 0 0 0) 0x7f84e8013df0 con 0x7f84f4101020 2022-05-13T23:07:25.315 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:07:25.302+0000 7f84fad4c700 1 -- 172.21.15.2:0/2281217149 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f84f4113b00 con 0x7f84f4101020 2022-05-13T23:07:25.315 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:07:25.302+0000 7f84f17fa700 1 -- 172.21.15.2:0/2281217149 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f84e801bdc0 con 0x7f84f4101020 2022-05-13T23:07:25.315 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:07:25.302+0000 7f84fad4c700 1 -- 172.21.15.2:0/2281217149 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f84f4113ff0 con 0x7f84f4101020 2022-05-13T23:07:25.315 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:07:25.303+0000 7f84f17fa700 1 -- 172.21.15.2:0/2281217149 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 13) v1 ==== 41898+0+0 (secure 0 0 0) 0x7f84e802b070 con 0x7f84f4101020 2022-05-13T23:07:25.316 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:07:25.303+0000 7f84fad4c700 1 -- 172.21.15.2:0/2281217149 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f84f405db10 con 0x7f84f4101020 2022-05-13T23:07:25.316 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:07:25.303+0000 7f84f17fa700 1 --2- 172.21.15.2:0/2281217149 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f84e4034ca0 0x7f84e4037150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:07:25.316 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:07:25.303+0000 7f84f17fa700 1 -- 172.21.15.2:0/2281217149 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f84e80474b0 con 0x7f84f4101020 2022-05-13T23:07:25.316 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:07:25.304+0000 7f84f3fff700 1 --2- 172.21.15.2:0/2281217149 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f84e4034ca0 0x7f84e4037150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:07:25.317 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:07:25.304+0000 7f84f3fff700 1 --2- 172.21.15.2:0/2281217149 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f84e4034ca0 0x7f84e4037150 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f84e0009780 tx=0x7f84e0006cc0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-05-13T23:07:25.342 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:07:25.309+0000 7f84f17fa700 1 -- 172.21.15.2:0/2281217149 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+165215 (secure 0 0 0) 0x7f84e8022360 con 0x7f84f4101020 2022-05-13T23:07:25.506 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:07:25.505+0000 7f84fad4c700 1 -- 172.21.15.2:0/2281217149 --> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi047", "target": ["mon-mgr", ""]}) v1 -- 0x7f84f410eab0 con 0x7f84e4034ca0 2022-05-13T23:07:26.572 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:07:26 smithi002 ceph-mon[71816]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-05-13T23:07:26.643 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:07:26.641+0000 7f84f17fa700 1 -- 172.21.15.2:0/2281217149 <== mgr.14148 v2:172.21.15.2:6800/2788339627 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+48 (secure 0 0 0) 0x7f84f410eab0 con 0x7f84e4034ca0 2022-05-13T23:07:26.643 INFO:teuthology.orchestra.run.smithi002.stdout:Added host 'smithi047' with addr '172.21.15.47' 2022-05-13T23:07:26.646 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:07:26.644+0000 7f84fad4c700 1 -- 172.21.15.2:0/2281217149 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f84e4034ca0 msgr2=0x7f84e4037150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:07:26.646 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:07:26.644+0000 7f84fad4c700 1 --2- 172.21.15.2:0/2281217149 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f84e4034ca0 0x7f84e4037150 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f84e0009780 tx=0x7f84e0006cc0 comp rx=0 tx=0).stop 2022-05-13T23:07:26.647 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:07:26.644+0000 7f84fad4c700 1 -- 172.21.15.2:0/2281217149 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f84f4101020 msgr2=0x7f84f41133c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:07:26.647 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:07:26.644+0000 7f84fad4c700 1 --2- 172.21.15.2:0/2281217149 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f84f4101020 0x7f84f41133c0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f84e80097f0 tx=0x7f84e8009cb0 comp rx=0 tx=0).stop 2022-05-13T23:07:26.647 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:07:26.644+0000 7f84fad4c700 1 -- 172.21.15.2:0/2281217149 shutdown_connections 2022-05-13T23:07:26.647 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:07:26.644+0000 7f84fad4c700 1 --2- 172.21.15.2:0/2281217149 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f84e4034ca0 0x7f84e4037150 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:07:26.648 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:07:26.644+0000 7f84fad4c700 1 --2- 172.21.15.2:0/2281217149 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f84f4101020 0x7f84f41133c0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:07:26.648 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:07:26.644+0000 7f84fad4c700 1 -- 172.21.15.2:0/2281217149 >> 172.21.15.2:0/2281217149 conn(0x7f84f40f6c60 msgr2=0x7f84f40701b0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-05-13T23:07:26.648 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:07:26.645+0000 7f84fad4c700 1 -- 172.21.15.2:0/2281217149 shutdown_connections 2022-05-13T23:07:26.648 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:07:26.645+0000 7f84fad4c700 1 -- 172.21.15.2:0/2281217149 wait complete. 2022-05-13T23:07:27.202 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:07:27 smithi002 ceph-mon[71816]: from='client.14174 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi047", "target": ["mon-mgr", ""]}]: dispatch 2022-05-13T23:07:27.202 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:07:27 smithi002 ceph-mon[71816]: Deploying cephadm binary to smithi047 2022-05-13T23:07:27.203 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:07:27 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:07:27.203 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:07:27 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-13T23:07:27.203 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:07:27 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-13T23:07:27.203 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:07:27 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-13T23:07:27.203 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:07:27 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:07:27.220 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:23b0658391ce94ca40799469eec00c28b298816c shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 16cf5f88-d311-11ec-be40-001a4aab830c -- ceph orch host ls --format=json 2022-05-13T23:07:27.891 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/16cf5f88-d311-11ec-be40-001a4aab830c/mon.a/config 2022-05-13T23:07:28.416 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:07:28 smithi002 ceph-mon[71816]: Added host smithi047 2022-05-13T23:07:28.416 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:07:28 smithi002 ceph-mon[71816]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-05-13T23:07:28.416 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:07:28 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:07:29.688 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:07:29.683+0000 7f55d0320700 1 -- 172.21.15.2:0/1709460778 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f55c8103230 msgr2=0x7f55c8103600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:07:29.688 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:07:29.683+0000 7f55d0320700 1 --2- 172.21.15.2:0/1709460778 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f55c8103230 0x7f55c8103600 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f55c4004770 tx=0x7f55c4009cb0 comp rx=0 tx=0).stop 2022-05-13T23:07:29.689 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:07:29.683+0000 7f55d0320700 1 -- 172.21.15.2:0/1709460778 shutdown_connections 2022-05-13T23:07:29.689 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:07:29.683+0000 7f55d0320700 1 --2- 172.21.15.2:0/1709460778 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f55c8103230 0x7f55c8103600 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:07:29.690 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:07:29.683+0000 7f55d0320700 1 -- 172.21.15.2:0/1709460778 >> 172.21.15.2:0/1709460778 conn(0x7f55c80f9030 msgr2=0x7f55c80fb410 unknown :-1 s=STATE_NONE l=0).mark_down 2022-05-13T23:07:29.690 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:07:29.684+0000 7f55d0320700 1 -- 172.21.15.2:0/1709460778 shutdown_connections 2022-05-13T23:07:29.690 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:07:29.684+0000 7f55d0320700 1 -- 172.21.15.2:0/1709460778 wait complete. 2022-05-13T23:07:29.690 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:07:29.684+0000 7f55d0320700 1 Processor -- start 2022-05-13T23:07:29.691 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:07:29.685+0000 7f55d0320700 1 -- start start 2022-05-13T23:07:29.691 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:07:29.685+0000 7f55d0320700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f55c8103230 0x7f55c8113370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:07:29.691 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:07:29.685+0000 7f55d0320700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55c81138b0 con 0x7f55c8103230 2022-05-13T23:07:29.691 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:07:29.685+0000 7f55ce0bc700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f55c8103230 0x7f55c8113370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:07:29.692 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:07:29.686+0000 7f55ce0bc700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f55c8103230 0x7f55c8113370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:59224/0 (socket says 172.21.15.2:59224) 2022-05-13T23:07:29.692 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:07:29.686+0000 7f55ce0bc700 1 -- 172.21.15.2:0/825511931 learned_addr learned my addr 172.21.15.2:0/825511931 (peer_addr_for_me v2:172.21.15.2:0/0) 2022-05-13T23:07:29.692 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:07:29.686+0000 7f55ce0bc700 1 -- 172.21.15.2:0/825511931 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55c4005040 con 0x7f55c8103230 2022-05-13T23:07:29.693 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:07:29.686+0000 7f55ce0bc700 1 --2- 172.21.15.2:0/825511931 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f55c8103230 0x7f55c8113370 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f55c40097f0 tx=0x7f55c4009cb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-05-13T23:07:29.695 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:07:29.686+0000 7f55beffd700 1 -- 172.21.15.2:0/825511931 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f55c400e050 con 0x7f55c8103230 2022-05-13T23:07:29.695 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:07:29.687+0000 7f55d0320700 1 -- 172.21.15.2:0/825511931 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f55c8113ab0 con 0x7f55c8103230 2022-05-13T23:07:29.695 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:07:29.687+0000 7f55beffd700 1 -- 172.21.15.2:0/825511931 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1087+0+0 (secure 0 0 0) 0x7f55c4013df0 con 0x7f55c8103230 2022-05-13T23:07:29.695 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:07:29.687+0000 7f55beffd700 1 -- 172.21.15.2:0/825511931 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f55c401bdc0 con 0x7f55c8103230 2022-05-13T23:07:29.696 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:07:29.687+0000 7f55d0320700 1 -- 172.21.15.2:0/825511931 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f55c8113fa0 con 0x7f55c8103230 2022-05-13T23:07:29.696 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:07:29.688+0000 7f55beffd700 1 -- 172.21.15.2:0/825511931 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 13) v1 ==== 41898+0+0 (secure 0 0 0) 0x7f55c402b070 con 0x7f55c8103230 2022-05-13T23:07:29.696 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:07:29.688+0000 7f55d0320700 1 -- 172.21.15.2:0/825511931 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f55c805db10 con 0x7f55c8103230 2022-05-13T23:07:29.696 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:07:29.688+0000 7f55beffd700 1 --2- 172.21.15.2:0/825511931 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f55b4034c50 0x7f55b4037100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:07:29.696 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:07:29.688+0000 7f55beffd700 1 -- 172.21.15.2:0/825511931 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f55c4047510 con 0x7f55c8103230 2022-05-13T23:07:29.697 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:07:29.689+0000 7f55cd8bb700 1 --2- 172.21.15.2:0/825511931 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f55b4034c50 0x7f55b4037100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:07:29.697 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:07:29.689+0000 7f55cd8bb700 1 --2- 172.21.15.2:0/825511931 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f55b4034c50 0x7f55b4037100 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f55b8009780 tx=0x7f55b8006cc0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-05-13T23:07:29.697 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:07:29.693+0000 7f55beffd700 1 -- 172.21.15.2:0/825511931 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+165215 (secure 0 0 0) 0x7f55c4023b60 con 0x7f55c8103230 2022-05-13T23:07:29.822 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:07:29 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:07:29.823 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:07:29 smithi002 ceph-mon[71816]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-05-13T23:07:29.889 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:07:29.887+0000 7f55d0320700 1 -- 172.21.15.2:0/825511931 --> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f55c810eaa0 con 0x7f55b4034c50 2022-05-13T23:07:29.890 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:07:29.889+0000 7f55beffd700 1 -- 172.21.15.2:0/825511931 <== mgr.14148 v2:172.21.15.2:6800/2788339627 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+158 (secure 0 0 0) 0x7f55c810eaa0 con 0x7f55b4034c50 2022-05-13T23:07:29.890 INFO:teuthology.orchestra.run.smithi002.stdout: 2022-05-13T23:07:29.891 INFO:teuthology.orchestra.run.smithi002.stdout:[{"addr": "172.21.15.2", "hostname": "smithi002", "labels": [], "status": ""}, {"addr": "172.21.15.47", "hostname": "smithi047", "labels": [], "status": ""}] 2022-05-13T23:07:29.893 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:07:29.891+0000 7f55d0320700 1 -- 172.21.15.2:0/825511931 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f55b4034c50 msgr2=0x7f55b4037100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:07:29.893 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:07:29.891+0000 7f55d0320700 1 --2- 172.21.15.2:0/825511931 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f55b4034c50 0x7f55b4037100 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f55b8009780 tx=0x7f55b8006cc0 comp rx=0 tx=0).stop 2022-05-13T23:07:29.893 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:07:29.891+0000 7f55d0320700 1 -- 172.21.15.2:0/825511931 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f55c8103230 msgr2=0x7f55c8113370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:07:29.894 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:07:29.891+0000 7f55d0320700 1 --2- 172.21.15.2:0/825511931 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f55c8103230 0x7f55c8113370 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f55c40097f0 tx=0x7f55c4009cb0 comp rx=0 tx=0).stop 2022-05-13T23:07:29.894 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:07:29.891+0000 7f55d0320700 1 -- 172.21.15.2:0/825511931 shutdown_connections 2022-05-13T23:07:29.894 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:07:29.891+0000 7f55d0320700 1 --2- 172.21.15.2:0/825511931 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f55b4034c50 0x7f55b4037100 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:07:29.894 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:07:29.891+0000 7f55d0320700 1 --2- 172.21.15.2:0/825511931 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f55c8103230 0x7f55c8113370 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:07:29.895 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:07:29.891+0000 7f55d0320700 1 -- 172.21.15.2:0/825511931 >> 172.21.15.2:0/825511931 conn(0x7f55c80f9030 msgr2=0x7f55c80fbe90 unknown :-1 s=STATE_NONE l=0).mark_down 2022-05-13T23:07:29.895 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:07:29.891+0000 7f55d0320700 1 -- 172.21.15.2:0/825511931 shutdown_connections 2022-05-13T23:07:29.895 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:07:29.891+0000 7f55d0320700 1 -- 172.21.15.2:0/825511931 wait complete. 2022-05-13T23:07:30.445 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi115 2022-05-13T23:07:30.446 DEBUG:teuthology.orchestra.run.smithi115:> set -ex 2022-05-13T23:07:30.446 DEBUG:teuthology.orchestra.run.smithi115:> dd of=/etc/ceph/ceph.conf 2022-05-13T23:07:30.464 DEBUG:teuthology.orchestra.run.smithi115:> set -ex 2022-05-13T23:07:30.465 DEBUG:teuthology.orchestra.run.smithi115:> dd of=/etc/ceph/ceph.client.admin.keyring 2022-05-13T23:07:30.521 INFO:tasks.cephadm:Adding host smithi115 to orchestrator... 2022-05-13T23:07:30.522 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:23b0658391ce94ca40799469eec00c28b298816c shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 16cf5f88-d311-11ec-be40-001a4aab830c -- ceph orch host add smithi115 2022-05-13T23:07:31.191 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/16cf5f88-d311-11ec-be40-001a4aab830c/mon.a/config 2022-05-13T23:07:31.538 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:07:31 smithi002 ceph-mon[71816]: from='client.14176 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2022-05-13T23:07:32.324 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:07:32 smithi002 ceph-mon[71816]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-05-13T23:07:32.324 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:07:32 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:07:32.325 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:07:32 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:07:32.325 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:07:32 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi047", "name": "osd_memory_target"}]: dispatch 2022-05-13T23:07:32.326 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:07:32 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:07:33.013 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:07:33.011+0000 7f6600c21700 1 -- 172.21.15.2:0/3078310589 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f65fc101020 msgr2=0x7f65fc1013f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:07:33.014 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:07:33.011+0000 7f6600c21700 1 --2- 172.21.15.2:0/3078310589 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f65fc101020 0x7f65fc1013f0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f65ec004770 tx=0x7f65ec009cb0 comp rx=0 tx=0).stop 2022-05-13T23:07:33.014 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:07:33.011+0000 7f6600c21700 1 -- 172.21.15.2:0/3078310589 shutdown_connections 2022-05-13T23:07:33.015 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:07:33.011+0000 7f6600c21700 1 --2- 172.21.15.2:0/3078310589 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f65fc101020 0x7f65fc1013f0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:07:33.015 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:07:33.011+0000 7f6600c21700 1 -- 172.21.15.2:0/3078310589 >> 172.21.15.2:0/3078310589 conn(0x7f65fc0f6c60 msgr2=0x7f65fc0f9070 unknown :-1 s=STATE_NONE l=0).mark_down 2022-05-13T23:07:33.015 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:07:33.011+0000 7f6600c21700 1 -- 172.21.15.2:0/3078310589 shutdown_connections 2022-05-13T23:07:33.015 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:07:33.012+0000 7f6600c21700 1 -- 172.21.15.2:0/3078310589 wait complete. 2022-05-13T23:07:33.018 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:07:33.012+0000 7f6600c21700 1 Processor -- start 2022-05-13T23:07:33.018 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:07:33.013+0000 7f6600c21700 1 -- start start 2022-05-13T23:07:33.018 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:07:33.013+0000 7f6600c21700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f65fc101020 0x7f65fc06e7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:07:33.019 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:07:33.013+0000 7f6600c21700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65fc06ed20 con 0x7f65fc101020 2022-05-13T23:07:33.019 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:07:33.013+0000 7f65fa59c700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f65fc101020 0x7f65fc06e7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:07:33.019 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:07:33.013+0000 7f65fa59c700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f65fc101020 0x7f65fc06e7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:40042/0 (socket says 172.21.15.2:40042) 2022-05-13T23:07:33.019 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:07:33.013+0000 7f65fa59c700 1 -- 172.21.15.2:0/1898580801 learned_addr learned my addr 172.21.15.2:0/1898580801 (peer_addr_for_me v2:172.21.15.2:0/0) 2022-05-13T23:07:33.020 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:07:33.014+0000 7f65fa59c700 1 -- 172.21.15.2:0/1898580801 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65ec005040 con 0x7f65fc101020 2022-05-13T23:07:33.020 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:07:33.014+0000 7f65fa59c700 1 --2- 172.21.15.2:0/1898580801 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f65fc101020 0x7f65fc06e7e0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f65ec0097f0 tx=0x7f65ec009cb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-05-13T23:07:33.020 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:07:33.014+0000 7f65eaffd700 1 -- 172.21.15.2:0/1898580801 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f65ec00e050 con 0x7f65fc101020 2022-05-13T23:07:33.020 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:07:33.014+0000 7f6600c21700 1 -- 172.21.15.2:0/1898580801 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f65fc06ef20 con 0x7f65fc101020 2022-05-13T23:07:33.021 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:07:33.014+0000 7f65eaffd700 1 -- 172.21.15.2:0/1898580801 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1087+0+0 (secure 0 0 0) 0x7f65ec013df0 con 0x7f65fc101020 2022-05-13T23:07:33.021 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:07:33.014+0000 7f65eaffd700 1 -- 172.21.15.2:0/1898580801 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f65ec01bdc0 con 0x7f65fc101020 2022-05-13T23:07:33.021 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:07:33.014+0000 7f6600c21700 1 -- 172.21.15.2:0/1898580801 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f65fc06f410 con 0x7f65fc101020 2022-05-13T23:07:33.021 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:07:33.015+0000 7f6600c21700 1 -- 172.21.15.2:0/1898580801 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f65fc05db10 con 0x7f65fc101020 2022-05-13T23:07:33.022 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:07:33.016+0000 7f65eaffd700 1 -- 172.21.15.2:0/1898580801 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 13) v1 ==== 41898+0+0 (secure 0 0 0) 0x7f65ec02b070 con 0x7f65fc101020 2022-05-13T23:07:33.022 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:07:33.016+0000 7f65eaffd700 1 --2- 172.21.15.2:0/1898580801 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f65e4034c50 0x7f65e4037100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:07:33.022 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:07:33.016+0000 7f65eaffd700 1 -- 172.21.15.2:0/1898580801 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f65ec047510 con 0x7f65fc101020 2022-05-13T23:07:33.022 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:07:33.016+0000 7f65f9d9b700 1 --2- 172.21.15.2:0/1898580801 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f65e4034c50 0x7f65e4037100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:07:33.023 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:07:33.017+0000 7f65f9d9b700 1 --2- 172.21.15.2:0/1898580801 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f65e4034c50 0x7f65e4037100 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f65f0009780 tx=0x7f65f0006cc0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-05-13T23:07:33.023 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:07:33.021+0000 7f65eaffd700 1 -- 172.21.15.2:0/1898580801 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+165215 (secure 0 0 0) 0x7f65ec023b60 con 0x7f65fc101020 2022-05-13T23:07:33.219 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:07:33.217+0000 7f6600c21700 1 -- 172.21.15.2:0/1898580801 --> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi115", "target": ["mon-mgr", ""]}) v1 -- 0x7f65fc06ff90 con 0x7f65e4034c50 2022-05-13T23:07:34.314 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:07:34.312+0000 7f65eaffd700 1 -- 172.21.15.2:0/1898580801 <== mgr.14148 v2:172.21.15.2:6800/2788339627 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+49 (secure 0 0 0) 0x7f65fc06ff90 con 0x7f65e4034c50 2022-05-13T23:07:34.315 INFO:teuthology.orchestra.run.smithi002.stdout:Added host 'smithi115' with addr '172.21.15.115' 2022-05-13T23:07:34.317 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:07:34.315+0000 7f6600c21700 1 -- 172.21.15.2:0/1898580801 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f65e4034c50 msgr2=0x7f65e4037100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:07:34.317 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:07:34.315+0000 7f6600c21700 1 --2- 172.21.15.2:0/1898580801 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f65e4034c50 0x7f65e4037100 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f65f0009780 tx=0x7f65f0006cc0 comp rx=0 tx=0).stop 2022-05-13T23:07:34.317 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:07:34.315+0000 7f6600c21700 1 -- 172.21.15.2:0/1898580801 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f65fc101020 msgr2=0x7f65fc06e7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:07:34.318 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:07:34.315+0000 7f6600c21700 1 --2- 172.21.15.2:0/1898580801 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f65fc101020 0x7f65fc06e7e0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f65ec0097f0 tx=0x7f65ec009cb0 comp rx=0 tx=0).stop 2022-05-13T23:07:34.318 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:07:34.315+0000 7f6600c21700 1 -- 172.21.15.2:0/1898580801 shutdown_connections 2022-05-13T23:07:34.318 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:07:34.315+0000 7f6600c21700 1 --2- 172.21.15.2:0/1898580801 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f65e4034c50 0x7f65e4037100 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:07:34.318 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:07:34.315+0000 7f6600c21700 1 --2- 172.21.15.2:0/1898580801 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f65fc101020 0x7f65fc06e7e0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:07:34.319 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:07:34.315+0000 7f6600c21700 1 -- 172.21.15.2:0/1898580801 >> 172.21.15.2:0/1898580801 conn(0x7f65fc0f6c60 msgr2=0x7f65fc0f78a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-05-13T23:07:34.319 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:07:34.315+0000 7f6600c21700 1 -- 172.21.15.2:0/1898580801 shutdown_connections 2022-05-13T23:07:34.319 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:07:34.315+0000 7f6600c21700 1 -- 172.21.15.2:0/1898580801 wait complete. 2022-05-13T23:07:34.572 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:07:34 smithi002 ceph-mon[71816]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-05-13T23:07:34.572 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:07:34 smithi002 ceph-mon[71816]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi115", "target": ["mon-mgr", ""]}]: dispatch 2022-05-13T23:07:34.868 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:23b0658391ce94ca40799469eec00c28b298816c shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 16cf5f88-d311-11ec-be40-001a4aab830c -- ceph orch host ls --format=json 2022-05-13T23:07:35.543 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/16cf5f88-d311-11ec-be40-001a4aab830c/mon.a/config 2022-05-13T23:07:35.572 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:07:35 smithi002 ceph-mon[71816]: Deploying cephadm binary to smithi115 2022-05-13T23:07:35.573 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:07:35 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:07:35.573 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:07:35 smithi002 ceph-mon[71816]: Added host smithi115 2022-05-13T23:07:35.573 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:07:35 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-13T23:07:35.573 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:07:35 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-13T23:07:35.574 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:07:35 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-13T23:07:35.574 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:07:35 smithi002 ceph-mon[71816]: Updating smithi047:/etc/ceph/ceph.conf 2022-05-13T23:07:35.574 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:07:35 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:07:35.574 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:07:35 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:07:35.575 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:07:35 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:07:36.438 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:07:36 smithi002 ceph-mon[71816]: Updating smithi047:/etc/ceph/ceph.client.admin.keyring 2022-05-13T23:07:36.439 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:07:36 smithi002 ceph-mon[71816]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-05-13T23:07:36.439 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:07:36 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:07:37.335 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:07:37.332+0000 7fc0be07e700 1 -- 172.21.15.2:0/2287417832 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc0b806e840 msgr2=0x7fc0b806ec10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:07:37.336 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:07:37.332+0000 7fc0be07e700 1 --2- 172.21.15.2:0/2287417832 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc0b806e840 0x7fc0b806ec10 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fc0a8004770 tx=0x7fc0a8009cb0 comp rx=0 tx=0).stop 2022-05-13T23:07:37.337 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:07:37.332+0000 7fc0be07e700 1 -- 172.21.15.2:0/2287417832 shutdown_connections 2022-05-13T23:07:37.337 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:07:37.332+0000 7fc0be07e700 1 --2- 172.21.15.2:0/2287417832 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc0b806e840 0x7fc0b806ec10 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:07:37.337 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:07:37.332+0000 7fc0be07e700 1 -- 172.21.15.2:0/2287417832 >> 172.21.15.2:0/2287417832 conn(0x7fc0b80f6c40 msgr2=0x7fc0b80f9050 unknown :-1 s=STATE_NONE l=0).mark_down 2022-05-13T23:07:37.337 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:07:37.332+0000 7fc0be07e700 1 -- 172.21.15.2:0/2287417832 shutdown_connections 2022-05-13T23:07:37.338 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:07:37.332+0000 7fc0be07e700 1 -- 172.21.15.2:0/2287417832 wait complete. 2022-05-13T23:07:37.341 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:07:37.333+0000 7fc0be07e700 1 Processor -- start 2022-05-13T23:07:37.341 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:07:37.333+0000 7fc0be07e700 1 -- start start 2022-05-13T23:07:37.341 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:07:37.334+0000 7fc0be07e700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc0b806e840 0x7fc0b81133c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:07:37.341 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:07:37.334+0000 7fc0be07e700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0b8113900 con 0x7fc0b806e840 2022-05-13T23:07:37.342 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:07:37.334+0000 7fc0b77fe700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc0b806e840 0x7fc0b81133c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:07:37.342 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:07:37.334+0000 7fc0b77fe700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc0b806e840 0x7fc0b81133c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:40056/0 (socket says 172.21.15.2:40056) 2022-05-13T23:07:37.342 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:07:37.334+0000 7fc0b77fe700 1 -- 172.21.15.2:0/1307698483 learned_addr learned my addr 172.21.15.2:0/1307698483 (peer_addr_for_me v2:172.21.15.2:0/0) 2022-05-13T23:07:37.343 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:07:37.334+0000 7fc0b77fe700 1 -- 172.21.15.2:0/1307698483 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0a8005040 con 0x7fc0b806e840 2022-05-13T23:07:37.343 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:07:37.335+0000 7fc0b77fe700 1 --2- 172.21.15.2:0/1307698483 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc0b806e840 0x7fc0b81133c0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fc0a80097f0 tx=0x7fc0a8009cb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-05-13T23:07:37.343 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:07:37.335+0000 7fc09ffff700 1 -- 172.21.15.2:0/1307698483 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fc0a800e050 con 0x7fc0b806e840 2022-05-13T23:07:37.344 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:07:37.335+0000 7fc09ffff700 1 -- 172.21.15.2:0/1307698483 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1087+0+0 (secure 0 0 0) 0x7fc0a8013df0 con 0x7fc0b806e840 2022-05-13T23:07:37.344 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:07:37.335+0000 7fc0be07e700 1 -- 172.21.15.2:0/1307698483 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc0b8113b00 con 0x7fc0b806e840 2022-05-13T23:07:37.344 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:07:37.335+0000 7fc09ffff700 1 -- 172.21.15.2:0/1307698483 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fc0a801bdc0 con 0x7fc0b806e840 2022-05-13T23:07:37.344 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:07:37.335+0000 7fc0be07e700 1 -- 172.21.15.2:0/1307698483 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc0b8113f70 con 0x7fc0b806e840 2022-05-13T23:07:37.344 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:07:37.336+0000 7fc09ffff700 1 -- 172.21.15.2:0/1307698483 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 13) v1 ==== 41898+0+0 (secure 0 0 0) 0x7fc0a802b070 con 0x7fc0b806e840 2022-05-13T23:07:37.345 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:07:37.336+0000 7fc0be07e700 1 -- 172.21.15.2:0/1307698483 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc0b805db10 con 0x7fc0b806e840 2022-05-13T23:07:37.345 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:07:37.337+0000 7fc09ffff700 1 --2- 172.21.15.2:0/1307698483 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7fc0a0034ca0 0x7fc0a0037150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:07:37.345 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:07:37.337+0000 7fc09ffff700 1 -- 172.21.15.2:0/1307698483 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7fc0a8047480 con 0x7fc0b806e840 2022-05-13T23:07:37.345 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:07:37.337+0000 7fc0b6ffd700 1 --2- 172.21.15.2:0/1307698483 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7fc0a0034ca0 0x7fc0a0037150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:07:37.346 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:07:37.338+0000 7fc0b6ffd700 1 --2- 172.21.15.2:0/1307698483 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7fc0a0034ca0 0x7fc0a0037150 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fc0ac009780 tx=0x7fc0ac006cc0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-05-13T23:07:37.347 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:07:37.342+0000 7fc09ffff700 1 -- 172.21.15.2:0/1307698483 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+165215 (secure 0 0 0) 0x7fc0a8023d80 con 0x7fc0b806e840 2022-05-13T23:07:37.540 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:07:37.537+0000 7fc0be07e700 1 -- 172.21.15.2:0/1307698483 --> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fc0b810d2a0 con 0x7fc0a0034ca0 2022-05-13T23:07:37.541 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:07:37.539+0000 7fc09ffff700 1 -- 172.21.15.2:0/1307698483 <== mgr.14148 v2:172.21.15.2:6800/2788339627 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+238 (secure 0 0 0) 0x7fc0b810d2a0 con 0x7fc0a0034ca0 2022-05-13T23:07:37.541 INFO:teuthology.orchestra.run.smithi002.stdout: 2022-05-13T23:07:37.542 INFO:teuthology.orchestra.run.smithi002.stdout:[{"addr": "172.21.15.2", "hostname": "smithi002", "labels": [], "status": ""}, {"addr": "172.21.15.47", "hostname": "smithi047", "labels": [], "status": ""}, {"addr": "172.21.15.115", "hostname": "smithi115", "labels": [], "status": ""}] 2022-05-13T23:07:37.544 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:07:37.541+0000 7fc0be07e700 1 -- 172.21.15.2:0/1307698483 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7fc0a0034ca0 msgr2=0x7fc0a0037150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:07:37.544 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:07:37.541+0000 7fc0be07e700 1 --2- 172.21.15.2:0/1307698483 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7fc0a0034ca0 0x7fc0a0037150 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fc0ac009780 tx=0x7fc0ac006cc0 comp rx=0 tx=0).stop 2022-05-13T23:07:37.544 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:07:37.541+0000 7fc0be07e700 1 -- 172.21.15.2:0/1307698483 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc0b806e840 msgr2=0x7fc0b81133c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:07:37.545 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:07:37.541+0000 7fc0be07e700 1 --2- 172.21.15.2:0/1307698483 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc0b806e840 0x7fc0b81133c0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fc0a80097f0 tx=0x7fc0a8009cb0 comp rx=0 tx=0).stop 2022-05-13T23:07:37.545 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:07:37.541+0000 7fc0be07e700 1 -- 172.21.15.2:0/1307698483 shutdown_connections 2022-05-13T23:07:37.545 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:07:37.541+0000 7fc0be07e700 1 --2- 172.21.15.2:0/1307698483 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7fc0a0034ca0 0x7fc0a0037150 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:07:37.545 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:07:37.541+0000 7fc0be07e700 1 --2- 172.21.15.2:0/1307698483 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc0b806e840 0x7fc0b81133c0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:07:37.546 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:07:37.541+0000 7fc0be07e700 1 -- 172.21.15.2:0/1307698483 >> 172.21.15.2:0/1307698483 conn(0x7fc0b80f6c40 msgr2=0x7fc0b8101c00 unknown :-1 s=STATE_NONE l=0).mark_down 2022-05-13T23:07:37.546 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:07:37.541+0000 7fc0be07e700 1 -- 172.21.15.2:0/1307698483 shutdown_connections 2022-05-13T23:07:37.546 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:07:37.541+0000 7fc0be07e700 1 -- 172.21.15.2:0/1307698483 wait complete. 2022-05-13T23:07:38.121 INFO:tasks.cephadm:Setting crush tunables to default 2022-05-13T23:07:38.121 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:23b0658391ce94ca40799469eec00c28b298816c shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 16cf5f88-d311-11ec-be40-001a4aab830c -- ceph osd crush tunables default 2022-05-13T23:07:38.572 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:07:38 smithi002 ceph-mon[71816]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-05-13T23:07:38.791 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/16cf5f88-d311-11ec-be40-001a4aab830c/mon.a/config 2022-05-13T23:07:39.572 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:07:39 smithi002 ceph-mon[71816]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2022-05-13T23:07:39.573 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:07:39 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:07:39.573 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:07:39 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:07:39.573 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:07:39 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi115", "name": "osd_memory_target"}]: dispatch 2022-05-13T23:07:39.573 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:07:39 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:07:40.572 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:07:40 smithi002 ceph-mon[71816]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-05-13T23:07:40.594 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:07:40.591+0000 7f55c2ece700 1 -- 172.21.15.2:0/2112760702 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f55bc0fbe70 msgr2=0x7f55bc0fc240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:07:40.594 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:07:40.591+0000 7f55c2ece700 1 --2- 172.21.15.2:0/2112760702 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f55bc0fbe70 0x7f55bc0fc240 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f55b8004770 tx=0x7f55b8009cb0 comp rx=0 tx=0).stop 2022-05-13T23:07:40.595 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:07:40.591+0000 7f55c2ece700 1 -- 172.21.15.2:0/2112760702 shutdown_connections 2022-05-13T23:07:40.595 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:07:40.591+0000 7f55c2ece700 1 --2- 172.21.15.2:0/2112760702 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f55bc0fbe70 0x7f55bc0fc240 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:07:40.595 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:07:40.591+0000 7f55c2ece700 1 -- 172.21.15.2:0/2112760702 >> 172.21.15.2:0/2112760702 conn(0x7f55bc0f5990 msgr2=0x7f55bc0f7da0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-05-13T23:07:40.595 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:07:40.591+0000 7f55c2ece700 1 -- 172.21.15.2:0/2112760702 shutdown_connections 2022-05-13T23:07:40.596 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:07:40.591+0000 7f55c2ece700 1 -- 172.21.15.2:0/2112760702 wait complete. 2022-05-13T23:07:40.597 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:07:40.592+0000 7f55c2ece700 1 Processor -- start 2022-05-13T23:07:40.598 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:07:40.593+0000 7f55c2ece700 1 -- start start 2022-05-13T23:07:40.598 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:07:40.593+0000 7f55c2ece700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f55bc0fbe70 0x7f55bc1110c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:07:40.598 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:07:40.593+0000 7f55c2ece700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55bc111600 con 0x7f55bc0fbe70 2022-05-13T23:07:40.598 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:07:40.593+0000 7f55c1ecc700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f55bc0fbe70 0x7f55bc1110c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:07:40.599 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:07:40.593+0000 7f55c1ecc700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f55bc0fbe70 0x7f55bc1110c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:53484/0 (socket says 172.21.15.2:53484) 2022-05-13T23:07:40.599 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:07:40.593+0000 7f55c1ecc700 1 -- 172.21.15.2:0/3530583463 learned_addr learned my addr 172.21.15.2:0/3530583463 (peer_addr_for_me v2:172.21.15.2:0/0) 2022-05-13T23:07:40.599 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:07:40.594+0000 7f55c1ecc700 1 -- 172.21.15.2:0/3530583463 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55b8005040 con 0x7f55bc0fbe70 2022-05-13T23:07:40.599 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:07:40.594+0000 7f55c1ecc700 1 --2- 172.21.15.2:0/3530583463 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f55bc0fbe70 0x7f55bc1110c0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f55b80097f0 tx=0x7f55b8009cb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-05-13T23:07:40.599 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:07:40.594+0000 7f55b27fc700 1 -- 172.21.15.2:0/3530583463 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f55b800e050 con 0x7f55bc0fbe70 2022-05-13T23:07:40.600 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:07:40.594+0000 7f55b27fc700 1 -- 172.21.15.2:0/3530583463 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1087+0+0 (secure 0 0 0) 0x7f55b8013df0 con 0x7f55bc0fbe70 2022-05-13T23:07:40.600 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:07:40.594+0000 7f55b27fc700 1 -- 172.21.15.2:0/3530583463 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f55b801bdc0 con 0x7f55bc0fbe70 2022-05-13T23:07:40.600 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:07:40.594+0000 7f55c2ece700 1 -- 172.21.15.2:0/3530583463 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f55bc111800 con 0x7f55bc0fbe70 2022-05-13T23:07:40.600 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:07:40.595+0000 7f55c2ece700 1 -- 172.21.15.2:0/3530583463 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f55bc111c70 con 0x7f55bc0fbe70 2022-05-13T23:07:40.601 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:07:40.595+0000 7f55c2ece700 1 -- 172.21.15.2:0/3530583463 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f55bc05db10 con 0x7f55bc0fbe70 2022-05-13T23:07:40.602 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:07:40.596+0000 7f55b27fc700 1 -- 172.21.15.2:0/3530583463 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 13) v1 ==== 41898+0+0 (secure 0 0 0) 0x7f55b802b070 con 0x7f55bc0fbe70 2022-05-13T23:07:40.602 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:07:40.596+0000 7f55b27fc700 1 --2- 172.21.15.2:0/3530583463 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f55a8034ca0 0x7f55a8037150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:07:40.602 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:07:40.596+0000 7f55b27fc700 1 -- 172.21.15.2:0/3530583463 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f55b8047550 con 0x7f55bc0fbe70 2022-05-13T23:07:40.602 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:07:40.597+0000 7f55c16cb700 1 --2- 172.21.15.2:0/3530583463 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f55a8034ca0 0x7f55a8037150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:07:40.603 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:07:40.597+0000 7f55c16cb700 1 --2- 172.21.15.2:0/3530583463 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f55a8034ca0 0x7f55a8037150 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f55ac009780 tx=0x7f55ac006cc0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-05-13T23:07:40.603 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:07:40.601+0000 7f55b27fc700 1 -- 172.21.15.2:0/3530583463 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+165215 (secure 0 0 0) 0x7f55b8022360 con 0x7f55bc0fbe70 2022-05-13T23:07:40.801 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:07:40.799+0000 7f55c2ece700 1 -- 172.21.15.2:0/3530583463 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7f55bc10c6d0 con 0x7f55bc0fbe70 2022-05-13T23:07:41.323 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:07:41.320+0000 7f55b27fc700 1 -- 172.21.15.2:0/3530583463 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v4) v1 ==== 124+0+0 (secure 0 0 0) 0x7f55b801f070 con 0x7f55bc0fbe70 2022-05-13T23:07:41.326 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:07:41.323+0000 7f55c2ece700 1 -- 172.21.15.2:0/3530583463 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f55a8034ca0 msgr2=0x7f55a8037150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:07:41.326 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:07:41.323+0000 7f55c2ece700 1 --2- 172.21.15.2:0/3530583463 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f55a8034ca0 0x7f55a8037150 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f55ac009780 tx=0x7f55ac006cc0 comp rx=0 tx=0).stop 2022-05-13T23:07:41.327 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:07:41.323+0000 7f55c2ece700 1 -- 172.21.15.2:0/3530583463 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f55bc0fbe70 msgr2=0x7f55bc1110c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:07:41.327 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:07:41.323+0000 7f55c2ece700 1 --2- 172.21.15.2:0/3530583463 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f55bc0fbe70 0x7f55bc1110c0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f55b80097f0 tx=0x7f55b8009cb0 comp rx=0 tx=0).stop 2022-05-13T23:07:41.327 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:07:41.323+0000 7f55c2ece700 1 -- 172.21.15.2:0/3530583463 shutdown_connections 2022-05-13T23:07:41.327 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:07:41.323+0000 7f55c2ece700 1 --2- 172.21.15.2:0/3530583463 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f55a8034ca0 0x7f55a8037150 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:07:41.328 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:07:41.323+0000 7f55c2ece700 1 --2- 172.21.15.2:0/3530583463 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f55bc0fbe70 0x7f55bc1110c0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:07:41.328 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:07:41.323+0000 7f55c2ece700 1 -- 172.21.15.2:0/3530583463 >> 172.21.15.2:0/3530583463 conn(0x7f55bc0f5990 msgr2=0x7f55bc0ff6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-05-13T23:07:41.328 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:07:41.324+0000 7f55c2ece700 1 -- 172.21.15.2:0/3530583463 shutdown_connections 2022-05-13T23:07:41.328 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:07:41.324+0000 7f55c2ece700 1 -- 172.21.15.2:0/3530583463 wait complete. 2022-05-13T23:07:41.329 INFO:teuthology.orchestra.run.smithi002.stderr:adjusted tunables profile to default 2022-05-13T23:07:41.572 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:07:41 smithi002 ceph-mon[71816]: from='client.? 172.21.15.2:0/3530583463' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2022-05-13T23:07:41.852 INFO:tasks.cephadm:Adding mon.a on smithi002 2022-05-13T23:07:41.853 INFO:tasks.cephadm:Adding mon.b on smithi047 2022-05-13T23:07:41.853 INFO:tasks.cephadm:Adding mon.c on smithi115 2022-05-13T23:07:41.853 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:23b0658391ce94ca40799469eec00c28b298816c shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 16cf5f88-d311-11ec-be40-001a4aab830c -- ceph orch apply mon '3;smithi002:172.21.15.2=a;smithi047:172.21.15.47=b;smithi115:172.21.15.115=c' 2022-05-13T23:07:42.326 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /etc/ceph/ceph.conf 2022-05-13T23:07:42.469 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /etc/ceph/ceph.conf 2022-05-13T23:07:42.572 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:07:42 smithi002 ceph-mon[71816]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-05-13T23:07:42.572 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:07:42 smithi002 ceph-mon[71816]: from='client.? 172.21.15.2:0/3530583463' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2022-05-13T23:07:42.573 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:07:42 smithi002 ceph-mon[71816]: osdmap e4: 0 total, 0 up, 0 in 2022-05-13T23:07:43.791 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:07:43.788+0000 7fc3809df700 1 -- 172.21.15.115:0/371893278 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc37c0fb010 msgr2=0x7fc37c0fb420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:07:43.792 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:07:43.788+0000 7fc3809df700 1 --2- 172.21.15.115:0/371893278 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc37c0fb010 0x7fc37c0fb420 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fc36c004770 tx=0x7fc36c009cb0 comp rx=0 tx=0).stop 2022-05-13T23:07:43.793 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:07:43.788+0000 7fc3809df700 1 -- 172.21.15.115:0/371893278 shutdown_connections 2022-05-13T23:07:43.793 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:07:43.788+0000 7fc3809df700 1 --2- 172.21.15.115:0/371893278 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc37c0fb010 0x7fc37c0fb420 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:07:43.793 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:07:43.788+0000 7fc3809df700 1 -- 172.21.15.115:0/371893278 >> 172.21.15.115:0/371893278 conn(0x7fc37c0f6780 msgr2=0x7fc37c0f8b90 unknown :-1 s=STATE_NONE l=0).mark_down 2022-05-13T23:07:43.793 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:07:43.788+0000 7fc3809df700 1 -- 172.21.15.115:0/371893278 shutdown_connections 2022-05-13T23:07:43.794 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:07:43.788+0000 7fc3809df700 1 -- 172.21.15.115:0/371893278 wait complete. 2022-05-13T23:07:43.794 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:07:43.788+0000 7fc3809df700 1 Processor -- start 2022-05-13T23:07:43.794 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:07:43.788+0000 7fc3809df700 1 -- start start 2022-05-13T23:07:43.794 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:07:43.789+0000 7fc3809df700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc37c0fb010 0x7fc37c0703b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:07:43.794 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:07:43.789+0000 7fc3809df700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc37c06e830 con 0x7fc37c0fb010 2022-05-13T23:07:43.795 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:07:43.789+0000 7fc37a59c700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc37c0fb010 0x7fc37c0703b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:07:43.795 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:07:43.789+0000 7fc37a59c700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc37c0fb010 0x7fc37c0703b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.115:53072/0 (socket says 172.21.15.115:53072) 2022-05-13T23:07:43.795 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:07:43.789+0000 7fc37a59c700 1 -- 172.21.15.115:0/4023255909 learned_addr learned my addr 172.21.15.115:0/4023255909 (peer_addr_for_me v2:172.21.15.115:0/0) 2022-05-13T23:07:43.795 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:07:43.789+0000 7fc37a59c700 1 -- 172.21.15.115:0/4023255909 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc36c005040 con 0x7fc37c0fb010 2022-05-13T23:07:43.796 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:07:43.790+0000 7fc37a59c700 1 --2- 172.21.15.115:0/4023255909 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc37c0fb010 0x7fc37c0703b0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fc36c009810 tx=0x7fc36c009cb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-05-13T23:07:43.796 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:07:43.790+0000 7fc36affd700 1 -- 172.21.15.115:0/4023255909 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fc36c00e050 con 0x7fc37c0fb010 2022-05-13T23:07:43.796 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:07:43.790+0000 7fc3809df700 1 -- 172.21.15.115:0/4023255909 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc37c06ea30 con 0x7fc37c0fb010 2022-05-13T23:07:43.796 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:07:43.790+0000 7fc36affd700 1 -- 172.21.15.115:0/4023255909 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1087+0+0 (secure 0 0 0) 0x7fc36c013bc0 con 0x7fc37c0fb010 2022-05-13T23:07:43.797 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:07:43.790+0000 7fc36affd700 1 -- 172.21.15.115:0/4023255909 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fc36c0123c0 con 0x7fc37c0fb010 2022-05-13T23:07:43.797 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:07:43.790+0000 7fc3809df700 1 -- 172.21.15.115:0/4023255909 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc37c06ef20 con 0x7fc37c0fb010 2022-05-13T23:07:43.798 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:07:43.791+0000 7fc3809df700 1 -- 172.21.15.115:0/4023255909 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc37c04a430 con 0x7fc37c0fb010 2022-05-13T23:07:43.798 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:07:43.791+0000 7fc36affd700 1 -- 172.21.15.115:0/4023255909 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 13) v1 ==== 41898+0+0 (secure 0 0 0) 0x7fc36c02b070 con 0x7fc37c0fb010 2022-05-13T23:07:43.799 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:07:43.791+0000 7fc36affd700 1 --2- 172.21.15.115:0/4023255909 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7fc364034c50 0x7fc364037100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:07:43.799 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:07:43.791+0000 7fc36affd700 1 -- 172.21.15.115:0/4023255909 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7fc36c047760 con 0x7fc37c0fb010 2022-05-13T23:07:43.799 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:07:43.791+0000 7fc379d9b700 1 --2- 172.21.15.115:0/4023255909 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7fc364034c50 0x7fc364037100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:07:43.799 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:07:43.792+0000 7fc379d9b700 1 --2- 172.21.15.115:0/4023255909 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7fc364034c50 0x7fc364037100 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fc370009780 tx=0x7fc370006cc0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-05-13T23:07:43.799 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:07:43.796+0000 7fc36affd700 1 -- 172.21.15.115:0/4023255909 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+165215 (secure 0 0 0) 0x7fc36c022540 con 0x7fc37c0fb010 2022-05-13T23:07:43.976 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:07:43.974+0000 7fc3809df700 1 -- 172.21.15.115:0/4023255909 --> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi002:172.21.15.2=a;smithi047:172.21.15.47=b;smithi115:172.21.15.115=c", "target": ["mon-mgr", ""]}) v1 -- 0x7fc37c06f2d0 con 0x7fc364034c50 2022-05-13T23:07:43.983 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:07:43.982+0000 7fc36affd700 1 -- 172.21.15.115:0/4023255909 <== mgr.14148 v2:172.21.15.2:6800/2788339627 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7fc37c06f2d0 con 0x7fc364034c50 2022-05-13T23:07:43.984 INFO:teuthology.orchestra.run.smithi115.stdout:Scheduled mon update... 2022-05-13T23:07:43.986 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:07:43.983+0000 7fc3809df700 1 -- 172.21.15.115:0/4023255909 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7fc364034c50 msgr2=0x7fc364037100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:07:43.986 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:07:43.983+0000 7fc3809df700 1 --2- 172.21.15.115:0/4023255909 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7fc364034c50 0x7fc364037100 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fc370009780 tx=0x7fc370006cc0 comp rx=0 tx=0).stop 2022-05-13T23:07:43.987 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:07:43.983+0000 7fc3809df700 1 -- 172.21.15.115:0/4023255909 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc37c0fb010 msgr2=0x7fc37c0703b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:07:43.987 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:07:43.983+0000 7fc3809df700 1 --2- 172.21.15.115:0/4023255909 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc37c0fb010 0x7fc37c0703b0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fc36c009810 tx=0x7fc36c009cb0 comp rx=0 tx=0).stop 2022-05-13T23:07:43.987 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:07:43.983+0000 7fc3809df700 1 -- 172.21.15.115:0/4023255909 shutdown_connections 2022-05-13T23:07:43.988 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:07:43.983+0000 7fc3809df700 1 --2- 172.21.15.115:0/4023255909 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7fc364034c50 0x7fc364037100 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:07:43.988 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:07:43.983+0000 7fc3809df700 1 --2- 172.21.15.115:0/4023255909 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc37c0fb010 0x7fc37c0703b0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:07:43.989 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:07:43.983+0000 7fc3809df700 1 -- 172.21.15.115:0/4023255909 >> 172.21.15.115:0/4023255909 conn(0x7fc37c0f6780 msgr2=0x7fc37c0f7460 unknown :-1 s=STATE_NONE l=0).mark_down 2022-05-13T23:07:43.989 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:07:43.983+0000 7fc3809df700 1 -- 172.21.15.115:0/4023255909 shutdown_connections 2022-05-13T23:07:43.989 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:07:43.983+0000 7fc3809df700 1 -- 172.21.15.115:0/4023255909 wait complete. 2022-05-13T23:07:44.507 DEBUG:teuthology.orchestra.run.smithi047:mon.b> sudo journalctl -f -n 0 -u ceph-16cf5f88-d311-11ec-be40-001a4aab830c@mon.b.service 2022-05-13T23:07:44.512 DEBUG:teuthology.orchestra.run.smithi115:mon.c> sudo journalctl -f -n 0 -u ceph-16cf5f88-d311-11ec-be40-001a4aab830c@mon.c.service 2022-05-13T23:07:44.517 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2022-05-13T23:07:44.518 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:23b0658391ce94ca40799469eec00c28b298816c shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 16cf5f88-d311-11ec-be40-001a4aab830c -- ceph mon dump -f json 2022-05-13T23:07:44.540 INFO:journalctl@ceph.mon.c.smithi115.stdout:-- Logs begin at Fri 2022-05-13 22:45:29 UTC. -- 2022-05-13T23:07:44.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:-- Logs begin at Fri 2022-05-13 22:44:47 UTC. -- 2022-05-13T23:07:44.822 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:07:44 smithi002 ceph-mon[71816]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-05-13T23:07:44.823 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:07:44 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:07:44.823 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:07:44 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-13T23:07:44.823 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:07:44 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-13T23:07:44.824 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:07:44 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-13T23:07:44.997 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /etc/ceph/ceph.conf 2022-05-13T23:07:45.135 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /etc/ceph/ceph.conf 2022-05-13T23:07:45.822 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:07:45 smithi002 ceph-mon[71816]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi002:172.21.15.2=a;smithi047:172.21.15.47=b;smithi115:172.21.15.115=c", "target": ["mon-mgr", ""]}]: dispatch 2022-05-13T23:07:45.823 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:07:45 smithi002 ceph-mon[71816]: Saving service mon spec with placement smithi002:172.21.15.2=a;smithi047:172.21.15.47=b;smithi115:172.21.15.115=c;count:3 2022-05-13T23:07:45.823 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:07:45 smithi002 ceph-mon[71816]: Updating smithi115:/etc/ceph/ceph.conf 2022-05-13T23:07:45.823 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:07:45 smithi002 ceph-mon[71816]: Updating smithi115:/etc/ceph/ceph.client.admin.keyring 2022-05-13T23:07:45.824 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:07:45 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:07:45.824 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:07:45 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:07:45.824 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:07:45 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2022-05-13T23:07:45.824 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:07:45 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-13T23:07:46.397 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:07:46.391+0000 7f274910f700 1 -- 172.21.15.115:0/1300785379 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2744101020 msgr2=0x7f27441013f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:07:46.397 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:07:46.391+0000 7f274910f700 1 --2- 172.21.15.115:0/1300785379 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2744101020 0x7f27441013f0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f2734004770 tx=0x7f2734009cb0 comp rx=0 tx=0).stop 2022-05-13T23:07:46.397 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:07:46.391+0000 7f274910f700 1 -- 172.21.15.115:0/1300785379 shutdown_connections 2022-05-13T23:07:46.397 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:07:46.391+0000 7f274910f700 1 --2- 172.21.15.115:0/1300785379 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2744101020 0x7f27441013f0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:07:46.398 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:07:46.391+0000 7f274910f700 1 -- 172.21.15.115:0/1300785379 >> 172.21.15.115:0/1300785379 conn(0x7f27440f6c40 msgr2=0x7f27440f9050 unknown :-1 s=STATE_NONE l=0).mark_down 2022-05-13T23:07:46.398 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:07:46.391+0000 7f274910f700 1 -- 172.21.15.115:0/1300785379 shutdown_connections 2022-05-13T23:07:46.398 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:07:46.391+0000 7f274910f700 1 -- 172.21.15.115:0/1300785379 wait complete. 2022-05-13T23:07:46.398 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:07:46.392+0000 7f274910f700 1 Processor -- start 2022-05-13T23:07:46.399 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:07:46.392+0000 7f274910f700 1 -- start start 2022-05-13T23:07:46.399 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:07:46.392+0000 7f274910f700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2744101020 0x7f27441111e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:07:46.399 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:07:46.392+0000 7f274910f700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2744111720 con 0x7f2744101020 2022-05-13T23:07:46.399 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:07:46.393+0000 7f2742d9d700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2744101020 0x7f27441111e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:07:46.399 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:07:46.393+0000 7f2742d9d700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2744101020 0x7f27441111e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.115:53078/0 (socket says 172.21.15.115:53078) 2022-05-13T23:07:46.400 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:07:46.393+0000 7f2742d9d700 1 -- 172.21.15.115:0/250495708 learned_addr learned my addr 172.21.15.115:0/250495708 (peer_addr_for_me v2:172.21.15.115:0/0) 2022-05-13T23:07:46.400 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:07:46.393+0000 7f2742d9d700 1 -- 172.21.15.115:0/250495708 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2734005040 con 0x7f2744101020 2022-05-13T23:07:46.400 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:07:46.393+0000 7f2742d9d700 1 --2- 172.21.15.115:0/250495708 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2744101020 0x7f27441111e0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f27340097f0 tx=0x7f2734009cb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-05-13T23:07:46.400 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:07:46.394+0000 7f272b7fe700 1 -- 172.21.15.115:0/250495708 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f273400e050 con 0x7f2744101020 2022-05-13T23:07:46.401 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:07:46.394+0000 7f274910f700 1 -- 172.21.15.115:0/250495708 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2744111920 con 0x7f2744101020 2022-05-13T23:07:46.401 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:07:46.394+0000 7f272b7fe700 1 -- 172.21.15.115:0/250495708 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1087+0+0 (secure 0 0 0) 0x7f273401b3f0 con 0x7f2744101020 2022-05-13T23:07:46.401 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:07:46.394+0000 7f272b7fe700 1 -- 172.21.15.115:0/250495708 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f27340123c0 con 0x7f2744101020 2022-05-13T23:07:46.401 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:07:46.394+0000 7f274910f700 1 -- 172.21.15.115:0/250495708 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2744111e10 con 0x7f2744101020 2022-05-13T23:07:46.402 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:07:46.395+0000 7f272b7fe700 1 -- 172.21.15.115:0/250495708 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 13) v1 ==== 41898+0+0 (secure 0 0 0) 0x7f273402b070 con 0x7f2744101020 2022-05-13T23:07:46.402 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:07:46.395+0000 7f274910f700 1 -- 172.21.15.115:0/250495708 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f274405db10 con 0x7f2744101020 2022-05-13T23:07:46.402 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:07:46.395+0000 7f272b7fe700 1 --2- 172.21.15.115:0/250495708 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f272c034c50 0x7f272c037100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:07:46.402 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:07:46.395+0000 7f272b7fe700 1 -- 172.21.15.115:0/250495708 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f27340474b0 con 0x7f2744101020 2022-05-13T23:07:46.403 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:07:46.395+0000 7f274259c700 1 --2- 172.21.15.115:0/250495708 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f272c034c50 0x7f272c037100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:07:46.404 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:07:46.396+0000 7f274259c700 1 --2- 172.21.15.115:0/250495708 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f272c034c50 0x7f272c037100 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f2738009780 tx=0x7f2738006cc0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-05-13T23:07:46.405 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:07:46.400+0000 7f272b7fe700 1 -- 172.21.15.115:0/250495708 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+165215 (secure 0 0 0) 0x7f273401b560 con 0x7f2744101020 2022-05-13T23:07:46.637 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:07:46.634+0000 7f274910f700 1 -- 172.21.15.115:0/250495708 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f274410c940 con 0x7f2744101020 2022-05-13T23:07:46.638 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:07:46.635+0000 7f272b7fe700 1 -- 172.21.15.115:0/250495708 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+706 (secure 0 0 0) 0x7f273401f030 con 0x7f2744101020 2022-05-13T23:07:46.639 INFO:teuthology.orchestra.run.smithi115.stdout: 2022-05-13T23:07:46.639 INFO:teuthology.orchestra.run.smithi115.stdout:{"epoch":1,"fsid":"16cf5f88-d311-11ec-be40-001a4aab830c","modified":"2022-05-13T23:05:56.093353Z","created":"2022-05-13T23:05:56.093353Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:3300","nonce":0},{"type":"v1","addr":"172.21.15.2:6789","nonce":0}]},"addr":"172.21.15.2:6789/0","public_addr":"172.21.15.2:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2022-05-13T23:07:46.640 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:07:46.637+0000 7f274910f700 1 -- 172.21.15.115:0/250495708 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f272c034c50 msgr2=0x7f272c037100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:07:46.641 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:07:46.637+0000 7f274910f700 1 --2- 172.21.15.115:0/250495708 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f272c034c50 0x7f272c037100 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f2738009780 tx=0x7f2738006cc0 comp rx=0 tx=0).stop 2022-05-13T23:07:46.641 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:07:46.637+0000 7f274910f700 1 -- 172.21.15.115:0/250495708 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2744101020 msgr2=0x7f27441111e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:07:46.641 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:07:46.637+0000 7f274910f700 1 --2- 172.21.15.115:0/250495708 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2744101020 0x7f27441111e0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f27340097f0 tx=0x7f2734009cb0 comp rx=0 tx=0).stop 2022-05-13T23:07:46.641 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:07:46.637+0000 7f274910f700 1 -- 172.21.15.115:0/250495708 shutdown_connections 2022-05-13T23:07:46.642 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:07:46.637+0000 7f274910f700 1 --2- 172.21.15.115:0/250495708 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f272c034c50 0x7f272c037100 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:07:46.642 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:07:46.637+0000 7f274910f700 1 --2- 172.21.15.115:0/250495708 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2744101020 0x7f27441111e0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:07:46.642 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:07:46.637+0000 7f274910f700 1 -- 172.21.15.115:0/250495708 >> 172.21.15.115:0/250495708 conn(0x7f27440f6c40 msgr2=0x7f27440f7880 unknown :-1 s=STATE_NONE l=0).mark_down 2022-05-13T23:07:46.642 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:07:46.637+0000 7f274910f700 1 -- 172.21.15.115:0/250495708 shutdown_connections 2022-05-13T23:07:46.642 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:07:46.637+0000 7f274910f700 1 -- 172.21.15.115:0/250495708 wait complete. 2022-05-13T23:07:46.643 INFO:teuthology.orchestra.run.smithi115.stderr:dumped monmap epoch 1 2022-05-13T23:07:46.828 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:07:46 smithi002 ceph-mon[71816]: Deploying daemon mon.b on smithi047 2022-05-13T23:07:46.828 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:07:46 smithi002 ceph-mon[71816]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-05-13T23:07:47.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:07:47 smithi047 systemd[1]: Starting Ceph mon.b for 16cf5f88-d311-11ec-be40-001a4aab830c... 2022-05-13T23:07:47.826 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:07:47 smithi002 ceph-mon[71816]: from='client.? 172.21.15.115:0/250495708' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2022-05-13T23:07:48.062 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2022-05-13T23:07:48.063 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:23b0658391ce94ca40799469eec00c28b298816c shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 16cf5f88-d311-11ec-be40-001a4aab830c -- ceph mon dump -f json 2022-05-13T23:07:48.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:07:47 smithi047 ceph-mon[77147]: set uid:gid to 167:167 (ceph:ceph) 2022-05-13T23:07:48.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:07:47 smithi047 ceph-mon[77147]: ceph version 17.2.0-188-g23b06583 (23b0658391ce94ca40799469eec00c28b298816c) quincy (stable), process ceph-mon, pid 7 2022-05-13T23:07:48.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:07:47 smithi047 ceph-mon[77147]: pidfile_write: ignore empty --pid-file 2022-05-13T23:07:48.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:07:47 smithi047 bash[76938]: 01602766cbfc6eae19dc67d00720932b9fc7b928f40f0ec32d22ff9a7bf8a27d 2022-05-13T23:07:48.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:07:47 smithi047 ceph-mon[77147]: load: jerasure load: lrc 2022-05-13T23:07:48.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:07:47 smithi047 ceph-mon[77147]: rocksdb: RocksDB version: 6.15.5 2022-05-13T23:07:48.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:07:47 smithi047 ceph-mon[77147]: rocksdb: Git sha rocksdb_build_git_sha:@0@ 2022-05-13T23:07:48.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:07:47 smithi047 ceph-mon[77147]: rocksdb: Compile date May 12 2022 2022-05-13T23:07:48.295 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:07:47 smithi047 ceph-mon[77147]: rocksdb: DB SUMMARY 2022-05-13T23:07:48.295 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:07:47 smithi047 ceph-mon[77147]: rocksdb: DB Session ID: P9BX4N4B7CW4EMFE5GU5 2022-05-13T23:07:48.295 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:07:47 smithi047 ceph-mon[77147]: rocksdb: CURRENT file: CURRENT 2022-05-13T23:07:48.295 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:07:47 smithi047 ceph-mon[77147]: rocksdb: IDENTITY file: IDENTITY 2022-05-13T23:07:48.296 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:07:47 smithi047 ceph-mon[77147]: rocksdb: MANIFEST file: MANIFEST-000003 size: 57 Bytes 2022-05-13T23:07:48.296 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:07:47 smithi047 ceph-mon[77147]: rocksdb: SST files in /var/lib/ceph/mon/ceph-b/store.db dir, Total Num: 0, files: 2022-05-13T23:07:48.296 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:07:47 smithi047 ceph-mon[77147]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-b/store.db: 000004.log size: 511 ; 2022-05-13T23:07:48.296 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:07:47 smithi047 ceph-mon[77147]: rocksdb: Options.error_if_exists: 0 2022-05-13T23:07:48.297 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:07:47 smithi047 ceph-mon[77147]: rocksdb: Options.create_if_missing: 0 2022-05-13T23:07:48.297 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:07:47 smithi047 ceph-mon[77147]: rocksdb: Options.paranoid_checks: 1 2022-05-13T23:07:48.297 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:07:47 smithi047 ceph-mon[77147]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2022-05-13T23:07:48.297 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:07:47 smithi047 ceph-mon[77147]: rocksdb: Options.env: 0x55c4bef5d860 2022-05-13T23:07:48.298 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:07:47 smithi047 ceph-mon[77147]: rocksdb: Options.fs: Posix File System 2022-05-13T23:07:48.298 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:07:47 smithi047 ceph-mon[77147]: rocksdb: Options.info_log: 0x55c4c0ccfdc0 2022-05-13T23:07:48.298 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:07:47 smithi047 ceph-mon[77147]: rocksdb: Options.max_file_opening_threads: 16 2022-05-13T23:07:48.299 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:07:47 smithi047 ceph-mon[77147]: rocksdb: Options.statistics: (nil) 2022-05-13T23:07:48.299 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:07:47 smithi047 ceph-mon[77147]: rocksdb: Options.use_fsync: 0 2022-05-13T23:07:48.299 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:07:47 smithi047 ceph-mon[77147]: rocksdb: Options.max_log_file_size: 0 2022-05-13T23:07:48.299 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:07:47 smithi047 ceph-mon[77147]: rocksdb: Options.max_manifest_file_size: 1073741824 2022-05-13T23:07:48.300 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:07:47 smithi047 ceph-mon[77147]: rocksdb: Options.log_file_time_to_roll: 0 2022-05-13T23:07:48.300 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:07:47 smithi047 ceph-mon[77147]: rocksdb: Options.keep_log_file_num: 1000 2022-05-13T23:07:48.300 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:07:47 smithi047 ceph-mon[77147]: rocksdb: Options.recycle_log_file_num: 0 2022-05-13T23:07:48.300 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:07:47 smithi047 ceph-mon[77147]: rocksdb: Options.allow_fallocate: 1 2022-05-13T23:07:48.300 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:07:47 smithi047 ceph-mon[77147]: rocksdb: Options.allow_mmap_reads: 0 2022-05-13T23:07:48.301 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:07:47 smithi047 ceph-mon[77147]: rocksdb: Options.allow_mmap_writes: 0 2022-05-13T23:07:48.301 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:07:47 smithi047 ceph-mon[77147]: rocksdb: Options.use_direct_reads: 0 2022-05-13T23:07:48.301 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:07:47 smithi047 ceph-mon[77147]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2022-05-13T23:07:48.301 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:07:47 smithi047 ceph-mon[77147]: rocksdb: Options.create_missing_column_families: 0 2022-05-13T23:07:48.302 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:07:47 smithi047 ceph-mon[77147]: rocksdb: Options.db_log_dir: 2022-05-13T23:07:48.302 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:07:47 smithi047 ceph-mon[77147]: rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-b/store.db 2022-05-13T23:07:48.302 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:07:47 smithi047 ceph-mon[77147]: rocksdb: Options.table_cache_numshardbits: 6 2022-05-13T23:07:48.302 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:07:47 smithi047 ceph-mon[77147]: rocksdb: Options.WAL_ttl_seconds: 0 2022-05-13T23:07:48.303 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:07:47 smithi047 ceph-mon[77147]: rocksdb: Options.WAL_size_limit_MB: 0 2022-05-13T23:07:48.303 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:07:47 smithi047 ceph-mon[77147]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2022-05-13T23:07:48.303 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:07:47 smithi047 ceph-mon[77147]: rocksdb: Options.manifest_preallocation_size: 4194304 2022-05-13T23:07:48.303 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:07:47 smithi047 ceph-mon[77147]: rocksdb: Options.is_fd_close_on_exec: 1 2022-05-13T23:07:48.304 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:07:47 smithi047 ceph-mon[77147]: rocksdb: Options.advise_random_on_open: 1 2022-05-13T23:07:48.304 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:07:47 smithi047 ceph-mon[77147]: rocksdb: Options.db_write_buffer_size: 0 2022-05-13T23:07:48.304 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:07:47 smithi047 ceph-mon[77147]: rocksdb: Options.write_buffer_manager: 0x55c4c0dc0240 2022-05-13T23:07:48.304 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:07:47 smithi047 ceph-mon[77147]: rocksdb: Options.access_hint_on_compaction_start: 1 2022-05-13T23:07:48.304 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:07:47 smithi047 ceph-mon[77147]: rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2022-05-13T23:07:48.305 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:07:47 smithi047 ceph-mon[77147]: rocksdb: Options.random_access_max_buffer_size: 1048576 2022-05-13T23:07:48.305 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:07:47 smithi047 ceph-mon[77147]: rocksdb: Options.use_adaptive_mutex: 0 2022-05-13T23:07:48.305 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:07:47 smithi047 ceph-mon[77147]: rocksdb: Options.rate_limiter: (nil) 2022-05-13T23:07:48.305 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:07:47 smithi047 ceph-mon[77147]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2022-05-13T23:07:48.306 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:07:47 smithi047 ceph-mon[77147]: rocksdb: Options.wal_recovery_mode: 2 2022-05-13T23:07:48.306 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:07:47 smithi047 ceph-mon[77147]: rocksdb: Options.enable_thread_tracking: 0 2022-05-13T23:07:48.306 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:07:47 smithi047 ceph-mon[77147]: rocksdb: Options.enable_pipelined_write: 0 2022-05-13T23:07:48.306 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:07:47 smithi047 ceph-mon[77147]: rocksdb: Options.unordered_write: 0 2022-05-13T23:07:48.307 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:07:47 smithi047 ceph-mon[77147]: rocksdb: Options.allow_concurrent_memtable_write: 1 2022-05-13T23:07:48.307 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:07:47 smithi047 ceph-mon[77147]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2022-05-13T23:07:48.307 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:07:47 smithi047 ceph-mon[77147]: rocksdb: Options.write_thread_max_yield_usec: 100 2022-05-13T23:07:48.307 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:07:47 smithi047 ceph-mon[77147]: rocksdb: Options.write_thread_slow_yield_usec: 3 2022-05-13T23:07:48.307 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:07:47 smithi047 ceph-mon[77147]: rocksdb: Options.row_cache: None 2022-05-13T23:07:48.308 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:07:47 smithi047 ceph-mon[77147]: rocksdb: Options.wal_filter: None 2022-05-13T23:07:48.308 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:07:47 smithi047 ceph-mon[77147]: rocksdb: Options.avoid_flush_during_recovery: 0 2022-05-13T23:07:48.308 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:07:47 smithi047 ceph-mon[77147]: rocksdb: Options.allow_ingest_behind: 0 2022-05-13T23:07:48.308 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:07:47 smithi047 ceph-mon[77147]: rocksdb: Options.preserve_deletes: 0 2022-05-13T23:07:48.308 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:07:47 smithi047 ceph-mon[77147]: rocksdb: Options.two_write_queues: 0 2022-05-13T23:07:48.309 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:07:47 smithi047 ceph-mon[77147]: rocksdb: Options.manual_wal_flush: 0 2022-05-13T23:07:48.309 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:07:47 smithi047 ceph-mon[77147]: rocksdb: Options.atomic_flush: 0 2022-05-13T23:07:48.309 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:07:47 smithi047 ceph-mon[77147]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2022-05-13T23:07:48.309 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:07:47 smithi047 ceph-mon[77147]: rocksdb: Options.persist_stats_to_disk: 0 2022-05-13T23:07:48.310 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:07:47 smithi047 ceph-mon[77147]: rocksdb: Options.write_dbid_to_manifest: 0 2022-05-13T23:07:48.310 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:07:47 smithi047 ceph-mon[77147]: rocksdb: Options.log_readahead_size: 0 2022-05-13T23:07:48.310 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:07:47 smithi047 ceph-mon[77147]: rocksdb: Options.file_checksum_gen_factory: Unknown 2022-05-13T23:07:48.310 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:07:47 smithi047 ceph-mon[77147]: rocksdb: Options.best_efforts_recovery: 0 2022-05-13T23:07:48.310 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:07:47 smithi047 ceph-mon[77147]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2022-05-13T23:07:48.311 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:07:47 smithi047 ceph-mon[77147]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2022-05-13T23:07:48.311 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:07:47 smithi047 ceph-mon[77147]: rocksdb: Options.allow_data_in_errors: 0 2022-05-13T23:07:48.311 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:07:47 smithi047 ceph-mon[77147]: rocksdb: Options.db_host_id: __hostname__ 2022-05-13T23:07:48.311 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:07:47 smithi047 ceph-mon[77147]: rocksdb: Options.max_background_jobs: 2 2022-05-13T23:07:48.312 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:07:47 smithi047 ceph-mon[77147]: rocksdb: Options.max_background_compactions: -1 2022-05-13T23:07:48.312 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:07:47 smithi047 ceph-mon[77147]: rocksdb: Options.max_subcompactions: 1 2022-05-13T23:07:48.312 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:07:47 smithi047 ceph-mon[77147]: rocksdb: Options.avoid_flush_during_shutdown: 0 2022-05-13T23:07:48.312 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:07:47 smithi047 ceph-mon[77147]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2022-05-13T23:07:48.312 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:07:47 smithi047 ceph-mon[77147]: rocksdb: Options.delayed_write_rate : 16777216 2022-05-13T23:07:48.313 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:07:47 smithi047 ceph-mon[77147]: rocksdb: Options.max_total_wal_size: 0 2022-05-13T23:07:48.313 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:07:47 smithi047 ceph-mon[77147]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2022-05-13T23:07:48.313 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:07:47 smithi047 ceph-mon[77147]: rocksdb: Options.stats_dump_period_sec: 600 2022-05-13T23:07:48.314 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:07:47 smithi047 ceph-mon[77147]: rocksdb: Options.stats_persist_period_sec: 600 2022-05-13T23:07:48.314 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:07:47 smithi047 ceph-mon[77147]: rocksdb: Options.stats_history_buffer_size: 1048576 2022-05-13T23:07:48.314 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:07:47 smithi047 ceph-mon[77147]: rocksdb: Options.max_open_files: -1 2022-05-13T23:07:48.314 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:07:47 smithi047 ceph-mon[77147]: rocksdb: Options.bytes_per_sync: 0 2022-05-13T23:07:48.315 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:07:47 smithi047 ceph-mon[77147]: rocksdb: Options.wal_bytes_per_sync: 0 2022-05-13T23:07:48.315 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:07:47 smithi047 ceph-mon[77147]: rocksdb: Options.strict_bytes_per_sync: 0 2022-05-13T23:07:48.315 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:07:47 smithi047 ceph-mon[77147]: rocksdb: Options.compaction_readahead_size: 0 2022-05-13T23:07:48.315 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:07:47 smithi047 ceph-mon[77147]: rocksdb: Options.max_background_flushes: -1 2022-05-13T23:07:48.315 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:07:47 smithi047 ceph-mon[77147]: rocksdb: Compression algorithms supported: 2022-05-13T23:07:48.316 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:07:47 smithi047 ceph-mon[77147]: rocksdb: kZSTDNotFinalCompression supported: 0 2022-05-13T23:07:48.316 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:07:47 smithi047 ceph-mon[77147]: rocksdb: kZSTD supported: 0 2022-05-13T23:07:48.316 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:07:47 smithi047 ceph-mon[77147]: rocksdb: kXpressCompression supported: 0 2022-05-13T23:07:48.316 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:07:47 smithi047 ceph-mon[77147]: rocksdb: kLZ4HCCompression supported: 1 2022-05-13T23:07:48.317 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:07:47 smithi047 ceph-mon[77147]: rocksdb: kLZ4Compression supported: 1 2022-05-13T23:07:48.317 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:07:47 smithi047 ceph-mon[77147]: rocksdb: kBZip2Compression supported: 0 2022-05-13T23:07:48.317 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:07:47 smithi047 ceph-mon[77147]: rocksdb: kZlibCompression supported: 1 2022-05-13T23:07:48.317 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:07:47 smithi047 ceph-mon[77147]: rocksdb: kSnappyCompression supported: 1 2022-05-13T23:07:48.318 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:07:47 smithi047 ceph-mon[77147]: rocksdb: Fast CRC32 supported: Supported on x86 2022-05-13T23:07:48.318 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:07:47 smithi047 ceph-mon[77147]: rocksdb: [db/version_set.cc:4725] Recovering from manifest file: /var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000003 2022-05-13T23:07:48.318 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:07:47 smithi047 ceph-mon[77147]: rocksdb: [db/column_family.cc:597] --------------- Options for column family [default]: 2022-05-13T23:07:48.318 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:07:47 smithi047 ceph-mon[77147]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2022-05-13T23:07:48.319 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:07:47 smithi047 ceph-mon[77147]: rocksdb: Options.merge_operator: 2022-05-13T23:07:48.319 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:07:47 smithi047 ceph-mon[77147]: rocksdb: Options.compaction_filter: None 2022-05-13T23:07:48.319 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:07:47 smithi047 ceph-mon[77147]: rocksdb: Options.compaction_filter_factory: None 2022-05-13T23:07:48.319 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:07:47 smithi047 ceph-mon[77147]: rocksdb: Options.sst_partitioner_factory: None 2022-05-13T23:07:48.320 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:07:47 smithi047 ceph-mon[77147]: rocksdb: Options.memtable_factory: SkipListFactory 2022-05-13T23:07:48.320 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:07:47 smithi047 ceph-mon[77147]: rocksdb: Options.table_factory: BlockBasedTable 2022-05-13T23:07:48.320 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:07:47 smithi047 ceph-mon[77147]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55c4c0c9bd10) 2022-05-13T23:07:48.320 INFO:journalctl@ceph.mon.b.smithi047.stdout: cache_index_and_filter_blocks: 1 2022-05-13T23:07:48.321 INFO:journalctl@ceph.mon.b.smithi047.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2022-05-13T23:07:48.321 INFO:journalctl@ceph.mon.b.smithi047.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2022-05-13T23:07:48.321 INFO:journalctl@ceph.mon.b.smithi047.stdout: pin_top_level_index_and_filter: 1 2022-05-13T23:07:48.321 INFO:journalctl@ceph.mon.b.smithi047.stdout: index_type: 0 2022-05-13T23:07:48.322 INFO:journalctl@ceph.mon.b.smithi047.stdout: data_block_index_type: 0 2022-05-13T23:07:48.322 INFO:journalctl@ceph.mon.b.smithi047.stdout: index_shortening: 1 2022-05-13T23:07:48.322 INFO:journalctl@ceph.mon.b.smithi047.stdout: data_block_hash_table_util_ratio: 0.750000 2022-05-13T23:07:48.322 INFO:journalctl@ceph.mon.b.smithi047.stdout: hash_index_allow_collision: 1 2022-05-13T23:07:48.322 INFO:journalctl@ceph.mon.b.smithi047.stdout: checksum: 1 2022-05-13T23:07:48.323 INFO:journalctl@ceph.mon.b.smithi047.stdout: no_block_cache: 0 2022-05-13T23:07:48.323 INFO:journalctl@ceph.mon.b.smithi047.stdout: block_cache: 0x55c4c0d06170 2022-05-13T23:07:48.323 INFO:journalctl@ceph.mon.b.smithi047.stdout: block_cache_name: BinnedLRUCache 2022-05-13T23:07:48.323 INFO:journalctl@ceph.mon.b.smithi047.stdout: block_cache_options: 2022-05-13T23:07:48.324 INFO:journalctl@ceph.mon.b.smithi047.stdout: capacity : 536870912 2022-05-13T23:07:48.324 INFO:journalctl@ceph.mon.b.smithi047.stdout: num_shard_bits : 4 2022-05-13T23:07:48.324 INFO:journalctl@ceph.mon.b.smithi047.stdout: strict_capacity_limit : 0 2022-05-13T23:07:48.324 INFO:journalctl@ceph.mon.b.smithi047.stdout: high_pri_pool_ratio: 0.000 2022-05-13T23:07:48.324 INFO:journalctl@ceph.mon.b.smithi047.stdout: block_cache_compressed: (nil) 2022-05-13T23:07:48.325 INFO:journalctl@ceph.mon.b.smithi047.stdout: persistent_cache: (nil) 2022-05-13T23:07:48.325 INFO:journalctl@ceph.mon.b.smithi047.stdout: block_size: 4096 2022-05-13T23:07:48.325 INFO:journalctl@ceph.mon.b.smithi047.stdout: block_size_deviation: 10 2022-05-13T23:07:48.325 INFO:journalctl@ceph.mon.b.smithi047.stdout: block_restart_interval: 16 2022-05-13T23:07:48.326 INFO:journalctl@ceph.mon.b.smithi047.stdout: index_block_restart_interval: 1 2022-05-13T23:07:48.326 INFO:journalctl@ceph.mon.b.smithi047.stdout: metadata_block_size: 4096 2022-05-13T23:07:48.326 INFO:journalctl@ceph.mon.b.smithi047.stdout: partition_filters: 0 2022-05-13T23:07:48.326 INFO:journalctl@ceph.mon.b.smithi047.stdout: use_delta_encoding: 1 2022-05-13T23:07:48.327 INFO:journalctl@ceph.mon.b.smithi047.stdout: filter_policy: rocksdb.BuiltinBloomFilter 2022-05-13T23:07:48.328 INFO:journalctl@ceph.mon.b.smithi047.stdout: whole_key_filtering: 1 2022-05-13T23:07:48.328 INFO:journalctl@ceph.mon.b.smithi047.stdout: verify_compression: 0 2022-05-13T23:07:48.328 INFO:journalctl@ceph.mon.b.smithi047.stdout: read_amp_bytes_per_bit: 0 2022-05-13T23:07:48.329 INFO:journalctl@ceph.mon.b.smithi047.stdout: format_version: 4 2022-05-13T23:07:48.329 INFO:journalctl@ceph.mon.b.smithi047.stdout: enable_index_compression: 1 2022-05-13T23:07:48.329 INFO:journalctl@ceph.mon.b.smithi047.stdout: block_align: 0 2022-05-13T23:07:48.329 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:07:47 smithi047 ceph-mon[77147]: rocksdb: Options.write_buffer_size: 33554432 2022-05-13T23:07:48.330 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:07:47 smithi047 ceph-mon[77147]: rocksdb: Options.max_write_buffer_number: 2 2022-05-13T23:07:48.330 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:07:47 smithi047 ceph-mon[77147]: rocksdb: Options.compression: NoCompression 2022-05-13T23:07:48.330 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:07:47 smithi047 ceph-mon[77147]: rocksdb: Options.bottommost_compression: Disabled 2022-05-13T23:07:48.330 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:07:47 smithi047 ceph-mon[77147]: rocksdb: Options.prefix_extractor: nullptr 2022-05-13T23:07:48.330 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:07:47 smithi047 ceph-mon[77147]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2022-05-13T23:07:48.331 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:07:47 smithi047 ceph-mon[77147]: rocksdb: Options.num_levels: 7 2022-05-13T23:07:48.331 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:07:47 smithi047 ceph-mon[77147]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2022-05-13T23:07:48.331 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:07:47 smithi047 ceph-mon[77147]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2022-05-13T23:07:48.331 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:07:47 smithi047 ceph-mon[77147]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2022-05-13T23:07:48.331 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:07:47 smithi047 ceph-mon[77147]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2022-05-13T23:07:48.332 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:07:47 smithi047 ceph-mon[77147]: rocksdb: Options.bottommost_compression_opts.level: 32767 2022-05-13T23:07:48.332 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:07:47 smithi047 ceph-mon[77147]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2022-05-13T23:07:48.332 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:07:47 smithi047 ceph-mon[77147]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2022-05-13T23:07:48.332 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:07:47 smithi047 ceph-mon[77147]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2022-05-13T23:07:48.333 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:07:47 smithi047 ceph-mon[77147]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2022-05-13T23:07:48.333 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:07:47 smithi047 ceph-mon[77147]: rocksdb: Options.bottommost_compression_opts.enabled: false 2022-05-13T23:07:48.333 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:07:47 smithi047 ceph-mon[77147]: rocksdb: Options.compression_opts.window_bits: -14 2022-05-13T23:07:48.333 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:07:47 smithi047 ceph-mon[77147]: rocksdb: Options.compression_opts.level: 32767 2022-05-13T23:07:48.335 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:07:47 smithi047 ceph-mon[77147]: rocksdb: Options.compression_opts.strategy: 0 2022-05-13T23:07:48.335 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:07:47 smithi047 ceph-mon[77147]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2022-05-13T23:07:48.336 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:07:47 smithi047 ceph-mon[77147]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2022-05-13T23:07:48.336 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:07:47 smithi047 ceph-mon[77147]: rocksdb: Options.compression_opts.parallel_threads: 1 2022-05-13T23:07:48.336 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:07:47 smithi047 ceph-mon[77147]: rocksdb: Options.compression_opts.enabled: false 2022-05-13T23:07:48.337 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:07:47 smithi047 ceph-mon[77147]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2022-05-13T23:07:48.337 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:07:47 smithi047 ceph-mon[77147]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2022-05-13T23:07:48.337 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:07:47 smithi047 ceph-mon[77147]: rocksdb: Options.level0_stop_writes_trigger: 36 2022-05-13T23:07:48.338 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:07:47 smithi047 ceph-mon[77147]: rocksdb: Options.target_file_size_base: 67108864 2022-05-13T23:07:48.338 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:07:47 smithi047 ceph-mon[77147]: rocksdb: Options.target_file_size_multiplier: 1 2022-05-13T23:07:48.338 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:07:47 smithi047 ceph-mon[77147]: rocksdb: Options.max_bytes_for_level_base: 268435456 2022-05-13T23:07:48.338 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:07:47 smithi047 ceph-mon[77147]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2022-05-13T23:07:48.339 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:07:47 smithi047 ceph-mon[77147]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2022-05-13T23:07:48.339 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:07:47 smithi047 ceph-mon[77147]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2022-05-13T23:07:48.339 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:07:47 smithi047 ceph-mon[77147]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2022-05-13T23:07:48.339 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:07:47 smithi047 ceph-mon[77147]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2022-05-13T23:07:48.340 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:07:47 smithi047 ceph-mon[77147]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2022-05-13T23:07:48.340 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:07:47 smithi047 ceph-mon[77147]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2022-05-13T23:07:48.340 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:07:47 smithi047 ceph-mon[77147]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2022-05-13T23:07:48.340 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:07:47 smithi047 ceph-mon[77147]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2022-05-13T23:07:48.341 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:07:47 smithi047 ceph-mon[77147]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2022-05-13T23:07:48.341 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:07:47 smithi047 ceph-mon[77147]: rocksdb: Options.max_compaction_bytes: 1677721600 2022-05-13T23:07:48.341 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:07:47 smithi047 ceph-mon[77147]: rocksdb: Options.arena_block_size: 4194304 2022-05-13T23:07:48.341 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:07:47 smithi047 ceph-mon[77147]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2022-05-13T23:07:48.342 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:07:47 smithi047 ceph-mon[77147]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2022-05-13T23:07:48.342 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:07:47 smithi047 ceph-mon[77147]: rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2022-05-13T23:07:48.342 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:07:47 smithi047 ceph-mon[77147]: rocksdb: Options.disable_auto_compactions: 0 2022-05-13T23:07:48.342 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:07:47 smithi047 ceph-mon[77147]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2022-05-13T23:07:48.343 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:07:47 smithi047 ceph-mon[77147]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2022-05-13T23:07:48.343 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:07:47 smithi047 ceph-mon[77147]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2022-05-13T23:07:48.343 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:07:47 smithi047 ceph-mon[77147]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2022-05-13T23:07:48.343 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:07:47 smithi047 ceph-mon[77147]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2022-05-13T23:07:48.344 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:07:47 smithi047 ceph-mon[77147]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2022-05-13T23:07:48.344 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:07:47 smithi047 ceph-mon[77147]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2022-05-13T23:07:48.344 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:07:47 smithi047 ceph-mon[77147]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2022-05-13T23:07:48.344 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:07:47 smithi047 ceph-mon[77147]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2022-05-13T23:07:48.345 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:07:47 smithi047 ceph-mon[77147]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2022-05-13T23:07:48.345 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:07:47 smithi047 ceph-mon[77147]: rocksdb: Options.table_properties_collectors: 2022-05-13T23:07:48.345 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:07:47 smithi047 ceph-mon[77147]: rocksdb: Options.inplace_update_support: 0 2022-05-13T23:07:48.345 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:07:47 smithi047 ceph-mon[77147]: rocksdb: Options.inplace_update_num_locks: 10000 2022-05-13T23:07:48.346 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:07:47 smithi047 ceph-mon[77147]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2022-05-13T23:07:48.346 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:07:47 smithi047 ceph-mon[77147]: rocksdb: Options.memtable_whole_key_filtering: 0 2022-05-13T23:07:48.346 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:07:47 smithi047 ceph-mon[77147]: rocksdb: Options.memtable_huge_page_size: 0 2022-05-13T23:07:48.346 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:07:47 smithi047 ceph-mon[77147]: rocksdb: Options.bloom_locality: 0 2022-05-13T23:07:48.347 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:07:47 smithi047 ceph-mon[77147]: rocksdb: Options.max_successive_merges: 0 2022-05-13T23:07:48.347 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:07:47 smithi047 ceph-mon[77147]: rocksdb: Options.optimize_filters_for_hits: 0 2022-05-13T23:07:48.347 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:07:47 smithi047 ceph-mon[77147]: rocksdb: Options.paranoid_file_checks: 0 2022-05-13T23:07:48.347 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:07:47 smithi047 ceph-mon[77147]: rocksdb: Options.force_consistency_checks: 1 2022-05-13T23:07:48.347 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:07:47 smithi047 ceph-mon[77147]: rocksdb: Options.report_bg_io_stats: 0 2022-05-13T23:07:48.348 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:07:47 smithi047 ceph-mon[77147]: rocksdb: Options.ttl: 2592000 2022-05-13T23:07:48.348 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:07:47 smithi047 ceph-mon[77147]: rocksdb: Options.periodic_compaction_seconds: 0 2022-05-13T23:07:48.348 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:07:47 smithi047 ceph-mon[77147]: rocksdb: Options.enable_blob_files: false 2022-05-13T23:07:48.348 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:07:47 smithi047 ceph-mon[77147]: rocksdb: Options.min_blob_size: 0 2022-05-13T23:07:48.349 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:07:47 smithi047 ceph-mon[77147]: rocksdb: Options.blob_file_size: 268435456 2022-05-13T23:07:48.349 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:07:47 smithi047 ceph-mon[77147]: rocksdb: Options.blob_compression_type: NoCompression 2022-05-13T23:07:48.349 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:07:47 smithi047 ceph-mon[77147]: rocksdb: Options.enable_blob_garbage_collection: false 2022-05-13T23:07:48.349 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:07:47 smithi047 ceph-mon[77147]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2022-05-13T23:07:48.350 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:07:47 smithi047 ceph-mon[77147]: rocksdb: [db/version_set.cc:4773] Recovered from manifest file:/var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000003 succeeded,manifest_file_number is 3, next_file_number is 5, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2022-05-13T23:07:48.350 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:07:47 smithi047 ceph-mon[77147]: rocksdb: [db/version_set.cc:4782] Column family [default] (ID 0), log number is 0 2022-05-13T23:07:48.350 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:07:47 smithi047 ceph-mon[77147]: rocksdb: [db/version_set.cc:4083] Creating manifest 7 2022-05-13T23:07:48.350 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:07:47 smithi047 ceph-mon[77147]: rocksdb: EVENT_LOG_v1 {"time_micros": 1652483267885728, "job": 1, "event": "recovery_started", "wal_files": [4]} 2022-05-13T23:07:48.350 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:07:47 smithi047 ceph-mon[77147]: rocksdb: [db/db_impl/db_impl_open.cc:847] Recovering log #4 mode 2 2022-05-13T23:07:48.351 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:07:47 smithi047 ceph-mon[77147]: rocksdb: [table/block_based/filter_policy.cc:996] Using legacy Bloom filter with high (20) bits/key. Dramatic filter space and/or accuracy improvement is available with format_version>=5. 2022-05-13T23:07:48.351 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:07:47 smithi047 ceph-mon[77147]: rocksdb: EVENT_LOG_v1 {"time_micros": 1652483267885999, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1545, "file_checksum": "", "file_checksum_func_name": "Unknown", "table_properties": {"data_size": 523, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 401, "raw_average_value_size": 80, "num_data_blocks": 1, "num_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "rocksdb.BuiltinBloomFilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ", "creation_time": 1652483267, "oldest_key_time": 0, "file_creation_time": 0, "db_id": "9b9cef7f-9ba6-4873-92e9-5aa4c3018895", "db_session_id": "P9BX4N4B7CW4EMFE5GU5"}} 2022-05-13T23:07:48.351 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:07:47 smithi047 ceph-mon[77147]: rocksdb: [db/version_set.cc:4083] Creating manifest 9 2022-05-13T23:07:48.351 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:07:47 smithi047 ceph-mon[77147]: rocksdb: EVENT_LOG_v1 {"time_micros": 1652483267886327, "job": 1, "event": "recovery_finished"} 2022-05-13T23:07:48.352 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:07:47 smithi047 ceph-mon[77147]: rocksdb: [file/delete_scheduler.cc:73] Deleted file /var/lib/ceph/mon/ceph-b/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2022-05-13T23:07:48.352 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:07:47 smithi047 ceph-mon[77147]: rocksdb: [db/db_impl/db_impl_open.cc:1701] SstFileManager instance 0x55c4c0ceca80 2022-05-13T23:07:48.352 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:07:47 smithi047 ceph-mon[77147]: rocksdb: DB pointer 0x55c4c0cfc000 2022-05-13T23:07:48.352 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:07:47 smithi047 ceph-mon[77147]: rocksdb: [db/db_impl/db_impl.cc:902] ------- DUMPING STATS ------- 2022-05-13T23:07:48.353 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:07:47 smithi047 ceph-mon[77147]: rocksdb: [db/db_impl/db_impl.cc:903] 2022-05-13T23:07:48.353 INFO:journalctl@ceph.mon.b.smithi047.stdout: ** DB Stats ** 2022-05-13T23:07:48.353 INFO:journalctl@ceph.mon.b.smithi047.stdout: Uptime(secs): 0.0 total, 0.0 interval 2022-05-13T23:07:48.353 INFO:journalctl@ceph.mon.b.smithi047.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2022-05-13T23:07:48.354 INFO:journalctl@ceph.mon.b.smithi047.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2022-05-13T23:07:48.354 INFO:journalctl@ceph.mon.b.smithi047.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2022-05-13T23:07:48.354 INFO:journalctl@ceph.mon.b.smithi047.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2022-05-13T23:07:48.354 INFO:journalctl@ceph.mon.b.smithi047.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2022-05-13T23:07:48.355 INFO:journalctl@ceph.mon.b.smithi047.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2022-05-13T23:07:48.355 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-13T23:07:48.355 INFO:journalctl@ceph.mon.b.smithi047.stdout: ** Compaction Stats [default] ** 2022-05-13T23:07:48.355 INFO:journalctl@ceph.mon.b.smithi047.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2022-05-13T23:07:48.355 INFO:journalctl@ceph.mon.b.smithi047.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2022-05-13T23:07:48.356 INFO:journalctl@ceph.mon.b.smithi047.stdout: L0 1/0 1.51 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 5.7 0.00 0.00 1 0.000 0 0 2022-05-13T23:07:48.356 INFO:journalctl@ceph.mon.b.smithi047.stdout: Sum 1/0 1.51 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 5.7 0.00 0.00 1 0.000 0 0 2022-05-13T23:07:48.356 INFO:journalctl@ceph.mon.b.smithi047.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 5.7 0.00 0.00 1 0.000 0 0 2022-05-13T23:07:48.356 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-13T23:07:48.357 INFO:journalctl@ceph.mon.b.smithi047.stdout: ** Compaction Stats [default] ** 2022-05-13T23:07:48.357 INFO:journalctl@ceph.mon.b.smithi047.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2022-05-13T23:07:48.357 INFO:journalctl@ceph.mon.b.smithi047.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2022-05-13T23:07:48.357 INFO:journalctl@ceph.mon.b.smithi047.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 5.7 0.00 0.00 1 0.000 0 0 2022-05-13T23:07:48.357 INFO:journalctl@ceph.mon.b.smithi047.stdout: Uptime(secs): 0.0 total, 0.0 interval 2022-05-13T23:07:48.358 INFO:journalctl@ceph.mon.b.smithi047.stdout: Flush(GB): cumulative 0.000, interval 0.000 2022-05-13T23:07:48.358 INFO:journalctl@ceph.mon.b.smithi047.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2022-05-13T23:07:48.358 INFO:journalctl@ceph.mon.b.smithi047.stdout: AddFile(Total Files): cumulative 0, interval 0 2022-05-13T23:07:48.358 INFO:journalctl@ceph.mon.b.smithi047.stdout: AddFile(L0 Files): cumulative 0, interval 0 2022-05-13T23:07:48.359 INFO:journalctl@ceph.mon.b.smithi047.stdout: AddFile(Keys): cumulative 0, interval 0 2022-05-13T23:07:48.359 INFO:journalctl@ceph.mon.b.smithi047.stdout: Cumulative compaction: 0.00 GB write, 0.58 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2022-05-13T23:07:48.359 INFO:journalctl@ceph.mon.b.smithi047.stdout: Interval compaction: 0.00 GB write, 0.58 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2022-05-13T23:07:48.359 INFO:journalctl@ceph.mon.b.smithi047.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2022-05-13T23:07:48.360 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-13T23:07:48.360 INFO:journalctl@ceph.mon.b.smithi047.stdout: ** File Read Latency Histogram By Level [default] ** 2022-05-13T23:07:48.360 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-13T23:07:48.360 INFO:journalctl@ceph.mon.b.smithi047.stdout: ** Compaction Stats [default] ** 2022-05-13T23:07:48.360 INFO:journalctl@ceph.mon.b.smithi047.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2022-05-13T23:07:48.361 INFO:journalctl@ceph.mon.b.smithi047.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2022-05-13T23:07:48.361 INFO:journalctl@ceph.mon.b.smithi047.stdout: L0 1/0 1.51 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 5.7 0.00 0.00 1 0.000 0 0 2022-05-13T23:07:48.361 INFO:journalctl@ceph.mon.b.smithi047.stdout: Sum 1/0 1.51 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 5.7 0.00 0.00 1 0.000 0 0 2022-05-13T23:07:48.361 INFO:journalctl@ceph.mon.b.smithi047.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2022-05-13T23:07:48.362 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-13T23:07:48.362 INFO:journalctl@ceph.mon.b.smithi047.stdout: ** Compaction Stats [default] ** 2022-05-13T23:07:48.362 INFO:journalctl@ceph.mon.b.smithi047.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2022-05-13T23:07:48.362 INFO:journalctl@ceph.mon.b.smithi047.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2022-05-13T23:07:48.362 INFO:journalctl@ceph.mon.b.smithi047.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 5.7 0.00 0.00 1 0.000 0 0 2022-05-13T23:07:48.363 INFO:journalctl@ceph.mon.b.smithi047.stdout: Uptime(secs): 0.0 total, 0.0 interval 2022-05-13T23:07:48.363 INFO:journalctl@ceph.mon.b.smithi047.stdout: Flush(GB): cumulative 0.000, interval 0.000 2022-05-13T23:07:48.363 INFO:journalctl@ceph.mon.b.smithi047.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2022-05-13T23:07:48.363 INFO:journalctl@ceph.mon.b.smithi047.stdout: AddFile(Total Files): cumulative 0, interval 0 2022-05-13T23:07:48.363 INFO:journalctl@ceph.mon.b.smithi047.stdout: AddFile(L0 Files): cumulative 0, interval 0 2022-05-13T23:07:48.364 INFO:journalctl@ceph.mon.b.smithi047.stdout: AddFile(Keys): cumulative 0, interval 0 2022-05-13T23:07:48.364 INFO:journalctl@ceph.mon.b.smithi047.stdout: Cumulative compaction: 0.00 GB write, 0.58 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2022-05-13T23:07:48.364 INFO:journalctl@ceph.mon.b.smithi047.stdout: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2022-05-13T23:07:48.364 INFO:journalctl@ceph.mon.b.smithi047.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2022-05-13T23:07:48.365 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-13T23:07:48.365 INFO:journalctl@ceph.mon.b.smithi047.stdout: ** File Read Latency Histogram By Level [default] ** 2022-05-13T23:07:48.365 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:07:47 smithi047 ceph-mon[77147]: mon.b does not exist in monmap, will attempt to join an existing cluster 2022-05-13T23:07:48.365 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:07:47 smithi047 ceph-mon[77147]: using public_addr v2:172.21.15.47:0/0 -> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] 2022-05-13T23:07:48.365 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:07:47 smithi047 ceph-mon[77147]: starting mon.b rank -1 at public addrs [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] at bind addrs [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon_data /var/lib/ceph/mon/ceph-b fsid 16cf5f88-d311-11ec-be40-001a4aab830c 2022-05-13T23:07:48.366 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:07:47 smithi047 ceph-mon[77147]: mon.b@-1(???) e0 preinit fsid 16cf5f88-d311-11ec-be40-001a4aab830c 2022-05-13T23:07:48.366 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:07:47 smithi047 ceph-mon[77147]: mon.b@-1(synchronizing).mds e1 new map 2022-05-13T23:07:48.366 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:07:47 smithi047 ceph-mon[77147]: mon.b@-1(synchronizing).mds e1 print_map 2022-05-13T23:07:48.366 INFO:journalctl@ceph.mon.b.smithi047.stdout: e1 2022-05-13T23:07:48.366 INFO:journalctl@ceph.mon.b.smithi047.stdout: enable_multiple, ever_enabled_multiple: 1,1 2022-05-13T23:07:48.367 INFO:journalctl@ceph.mon.b.smithi047.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2022-05-13T23:07:48.367 INFO:journalctl@ceph.mon.b.smithi047.stdout: legacy client fscid: -1 2022-05-13T23:07:48.367 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-13T23:07:48.367 INFO:journalctl@ceph.mon.b.smithi047.stdout: No filesystems configured 2022-05-13T23:07:48.368 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:07:47 smithi047 ceph-mon[77147]: mon.b@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2022-05-13T23:07:48.368 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:07:47 smithi047 ceph-mon[77147]: mon.b@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2022-05-13T23:07:48.368 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:07:47 smithi047 ceph-mon[77147]: mon.b@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2022-05-13T23:07:48.368 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:07:47 smithi047 ceph-mon[77147]: mon.b@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2022-05-13T23:07:48.368 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:07:47 smithi047 ceph-mon[77147]: mon.b@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2022-05-13T23:07:48.369 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:07:47 smithi047 ceph-mon[77147]: mon.b@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2022-05-13T23:07:48.369 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:07:47 smithi047 ceph-mon[77147]: mon.b@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2022-05-13T23:07:48.369 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:07:47 smithi047 ceph-mon[77147]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2022-05-13T23:07:48.369 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:07:47 smithi047 ceph-mon[77147]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2022-05-13T23:07:48.369 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:07:47 smithi047 ceph-mon[77147]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2022-05-13T23:07:48.370 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:07:47 smithi047 ceph-mon[77147]: Deploying cephadm binary to smithi115 2022-05-13T23:07:48.370 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:07:47 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:07:48.370 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:07:47 smithi047 ceph-mon[77147]: Added host smithi115 2022-05-13T23:07:48.370 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:07:47 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-13T23:07:48.370 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:07:47 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-13T23:07:48.371 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:07:47 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-13T23:07:48.371 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:07:47 smithi047 ceph-mon[77147]: Updating smithi047:/etc/ceph/ceph.conf 2022-05-13T23:07:48.371 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:07:47 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:07:48.371 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:07:47 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:07:48.371 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:07:47 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:07:48.372 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:07:47 smithi047 ceph-mon[77147]: Updating smithi047:/etc/ceph/ceph.client.admin.keyring 2022-05-13T23:07:48.372 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:07:47 smithi047 ceph-mon[77147]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-05-13T23:07:48.372 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:07:47 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:07:48.372 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:07:47 smithi047 ceph-mon[77147]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-05-13T23:07:48.373 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:07:47 smithi047 ceph-mon[77147]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2022-05-13T23:07:48.373 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:07:47 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:07:48.373 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:07:47 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:07:48.373 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:07:47 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi115", "name": "osd_memory_target"}]: dispatch 2022-05-13T23:07:48.373 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:07:47 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:07:48.374 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:07:47 smithi047 ceph-mon[77147]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-05-13T23:07:48.374 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:07:47 smithi047 ceph-mon[77147]: from='client.? 172.21.15.2:0/3530583463' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2022-05-13T23:07:48.374 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:07:47 smithi047 ceph-mon[77147]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-05-13T23:07:48.374 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:07:47 smithi047 ceph-mon[77147]: from='client.? 172.21.15.2:0/3530583463' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2022-05-13T23:07:48.374 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:07:47 smithi047 ceph-mon[77147]: osdmap e4: 0 total, 0 up, 0 in 2022-05-13T23:07:48.375 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:07:47 smithi047 ceph-mon[77147]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-05-13T23:07:48.375 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:07:47 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:07:48.375 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:07:47 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-13T23:07:48.375 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:07:47 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-13T23:07:48.375 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:07:47 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-13T23:07:48.376 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:07:47 smithi047 ceph-mon[77147]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi002:172.21.15.2=a;smithi047:172.21.15.47=b;smithi115:172.21.15.115=c", "target": ["mon-mgr", ""]}]: dispatch 2022-05-13T23:07:48.376 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:07:47 smithi047 ceph-mon[77147]: Saving service mon spec with placement smithi002:172.21.15.2=a;smithi047:172.21.15.47=b;smithi115:172.21.15.115=c;count:3 2022-05-13T23:07:48.376 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:07:47 smithi047 ceph-mon[77147]: Updating smithi115:/etc/ceph/ceph.conf 2022-05-13T23:07:48.376 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:07:47 smithi047 ceph-mon[77147]: Updating smithi115:/etc/ceph/ceph.client.admin.keyring 2022-05-13T23:07:48.377 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:07:47 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:07:48.377 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:07:47 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:07:48.377 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:07:47 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2022-05-13T23:07:48.377 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:07:47 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-13T23:07:48.377 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:07:47 smithi047 ceph-mon[77147]: Deploying daemon mon.b on smithi047 2022-05-13T23:07:48.378 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:07:47 smithi047 ceph-mon[77147]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-05-13T23:07:48.378 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:07:47 smithi047 ceph-mon[77147]: from='client.? 172.21.15.115:0/250495708' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2022-05-13T23:07:48.378 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:07:47 smithi047 ceph-mon[77147]: mon.b@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2022-05-13T23:07:48.378 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:07:47 smithi047 ceph-mon[77147]: mon.b@-1(synchronizing) e1 handle_conf_change mon_allow_pool_delete 2022-05-13T23:07:48.378 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:07:47 smithi047 systemd[1]: Started Ceph mon.b for 16cf5f88-d311-11ec-be40-001a4aab830c. 2022-05-13T23:07:48.532 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /etc/ceph/ceph.conf 2022-05-13T23:07:48.677 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /etc/ceph/ceph.conf 2022-05-13T23:07:53.066 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:07:53.061+0000 7f6443e6a700 1 -- 172.21.15.115:0/2214996718 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6428003400 msgr2=0x7f6428005880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:07:53.066 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:07:53.061+0000 7f6443e6a700 1 --2- 172.21.15.115:0/2214996718 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6428003400 0x7f6428005880 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f643c0656f0 tx=0x7f643800b040 comp rx=0 tx=0).stop 2022-05-13T23:07:53.066 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:07:53.061+0000 7f6443e6a700 1 -- 172.21.15.115:0/2214996718 shutdown_connections 2022-05-13T23:07:53.067 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:07:53.061+0000 7f6443e6a700 1 --2- 172.21.15.115:0/2214996718 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6428003400 0x7f6428005880 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:07:53.067 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:07:53.061+0000 7f6443e6a700 1 -- 172.21.15.115:0/2214996718 >> 172.21.15.115:0/2214996718 conn(0x7f643c070f00 msgr2=0x7f643c071300 unknown :-1 s=STATE_NONE l=0).mark_down 2022-05-13T23:07:53.067 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:07:53.061+0000 7f6443e6a700 1 -- 172.21.15.115:0/2214996718 shutdown_connections 2022-05-13T23:07:53.067 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:07:53.061+0000 7f6443e6a700 1 -- 172.21.15.115:0/2214996718 wait complete. 2022-05-13T23:07:53.068 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:07:53.062+0000 7f6443e6a700 1 Processor -- start 2022-05-13T23:07:53.068 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:07:53.062+0000 7f6443e6a700 1 -- start start 2022-05-13T23:07:53.068 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:07:53.062+0000 7f6443e6a700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6428003400 0x7f643c10c380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:07:53.068 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:07:53.062+0000 7f6443e6a700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f643c10c8c0 0x7f643c115230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:07:53.069 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:07:53.062+0000 7f6443e6a700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f643c10cdc0 con 0x7f643c10c8c0 2022-05-13T23:07:53.069 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:07:53.062+0000 7f6443e6a700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f643c10cf00 con 0x7f6428003400 2022-05-13T23:07:53.069 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:07:53.062+0000 7f6441c06700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6428003400 0x7f643c10c380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:07:53.069 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:07:53.062+0000 7f6441c06700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6428003400 0x7f643c10c380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.115:39860/0 (socket says 172.21.15.115:39860) 2022-05-13T23:07:53.070 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:07:53.062+0000 7f6441405700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f643c10c8c0 0x7f643c115230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:07:53.070 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:07:53.062+0000 7f6441c06700 1 -- 172.21.15.115:0/718060714 learned_addr learned my addr 172.21.15.115:0/718060714 (peer_addr_for_me v2:172.21.15.115:0/0) 2022-05-13T23:07:53.071 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:07:53.062+0000 7f6441c06700 1 -- 172.21.15.115:0/718060714 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6428003400 msgr2=0x7f643c10c380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2022-05-13T23:07:53.071 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:07:53.063+0000 7f6441c06700 1 -- 172.21.15.115:0/718060714 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6428003400 msgr2=0x7f643c10c380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2022-05-13T23:07:53.072 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:07:53.063+0000 7f6441c06700 1 --2- 172.21.15.115:0/718060714 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6428003400 0x7f643c10c380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2022-05-13T23:07:53.072 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:07:53.063+0000 7f6441c06700 1 --2- 172.21.15.115:0/718060714 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6428003400 0x7f643c10c380 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2022-05-13T23:07:53.072 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:07:53.063+0000 7f6441405700 1 -- 172.21.15.115:0/718060714 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6428003400 msgr2=0x7f643c10c380 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2022-05-13T23:07:53.073 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:07:53.063+0000 7f6441405700 1 --2- 172.21.15.115:0/718060714 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6428003400 0x7f643c10c380 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:07:53.073 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:07:53.063+0000 7f6441405700 1 -- 172.21.15.115:0/718060714 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6438005040 con 0x7f643c10c8c0 2022-05-13T23:07:53.073 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:07:53.063+0000 7f6441405700 1 --2- 172.21.15.115:0/718060714 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f643c10c8c0 0x7f643c115230 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f642c00b460 tx=0x7f642c009060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-05-13T23:07:53.073 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:07:53.064+0000 7f64327fc700 1 -- 172.21.15.115:0/718060714 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 309+0+0 (secure 0 0 0) 0x7f642c0117e0 con 0x7f643c10c8c0 2022-05-13T23:07:53.074 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:07:53.064+0000 7f6443e6a700 1 -- 172.21.15.115:0/718060714 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f643c115770 con 0x7f643c10c8c0 2022-05-13T23:07:53.074 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:07:53.064+0000 7f64327fc700 1 -- 172.21.15.115:0/718060714 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1087+0+0 (secure 0 0 0) 0x7f642c011df0 con 0x7f643c10c8c0 2022-05-13T23:07:53.074 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:07:53.064+0000 7f64327fc700 1 -- 172.21.15.115:0/718060714 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 309+0+0 (secure 0 0 0) 0x7f642c010e00 con 0x7f643c10c8c0 2022-05-13T23:07:53.074 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:07:53.064+0000 7f6443e6a700 1 -- 172.21.15.115:0/718060714 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f643c115ce0 con 0x7f643c10c8c0 2022-05-13T23:07:53.076 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:07:53.065+0000 7f6443e6a700 1 -- 172.21.15.115:0/718060714 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f643c05db10 con 0x7f643c10c8c0 2022-05-13T23:07:53.076 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:07:53.065+0000 7f64327fc700 1 -- 172.21.15.115:0/718060714 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 13) v1 ==== 41898+0+0 (secure 0 0 0) 0x7f642c017020 con 0x7f643c10c8c0 2022-05-13T23:07:53.076 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:07:53.065+0000 7f64327fc700 1 --2- 172.21.15.115:0/718060714 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f6420034de0 0x7f6420037290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:07:53.077 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:07:53.065+0000 7f64327fc700 1 -- 172.21.15.115:0/718060714 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f642c046340 con 0x7f643c10c8c0 2022-05-13T23:07:53.077 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:07:53.066+0000 7f6441c06700 1 --2- 172.21.15.115:0/718060714 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f6420034de0 0x7f6420037290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:07:53.077 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:07:53.066+0000 7f6441c06700 1 --2- 172.21.15.115:0/718060714 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f6420034de0 0x7f6420037290 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f643800b010 tx=0x7f6438002040 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-05-13T23:07:53.077 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:07:53.070+0000 7f64327fc700 1 -- 172.21.15.115:0/718060714 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+165215 (secure 0 0 0) 0x7f642c010510 con 0x7f643c10c8c0 2022-05-13T23:07:53.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:07:52 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-05-13T23:07:53.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:07:52 smithi047 ceph-mon[77147]: mon.a calling monitor election 2022-05-13T23:07:53.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:07:52 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-05-13T23:07:53.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:07:52 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-05-13T23:07:53.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:07:52 smithi047 ceph-mon[77147]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-05-13T23:07:53.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:07:52 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-05-13T23:07:53.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:07:52 smithi047 ceph-mon[77147]: mon.b calling monitor election 2022-05-13T23:07:53.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:07:52 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-05-13T23:07:53.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:07:52 smithi047 ceph-mon[77147]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-05-13T23:07:53.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:07:52 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-05-13T23:07:53.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:07:52 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-05-13T23:07:53.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:07:52 smithi047 ceph-mon[77147]: mon.a is new leader, mons a,b in quorum (ranks 0,1) 2022-05-13T23:07:53.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:07:52 smithi047 ceph-mon[77147]: monmap e2: 2 mons at {a=[v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0],b=[v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0]} 2022-05-13T23:07:53.295 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:07:52 smithi047 ceph-mon[77147]: fsmap 2022-05-13T23:07:53.295 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:07:52 smithi047 ceph-mon[77147]: osdmap e4: 0 total, 0 up, 0 in 2022-05-13T23:07:53.295 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:07:52 smithi047 ceph-mon[77147]: mgrmap e13: x(active, since 47s) 2022-05-13T23:07:53.295 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:07:52 smithi047 ceph-mon[77147]: overall HEALTH_OK 2022-05-13T23:07:53.296 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:07:52 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:07:53.296 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:07:52 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2022-05-13T23:07:53.296 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:07:52 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-13T23:07:53.320 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:07:53.318+0000 7f6443e6a700 1 -- 172.21.15.115:0/718060714 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f643c115f60 con 0x7f643c10c8c0 2022-05-13T23:07:53.321 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:07:53.319+0000 7f64327fc700 1 -- 172.21.15.115:0/718060714 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 2 v2) v1 ==== 95+0+972 (secure 0 0 0) 0x7f642c0212f0 con 0x7f643c10c8c0 2022-05-13T23:07:53.322 INFO:teuthology.orchestra.run.smithi115.stdout: 2022-05-13T23:07:53.322 INFO:teuthology.orchestra.run.smithi115.stdout:{"epoch":2,"fsid":"16cf5f88-d311-11ec-be40-001a4aab830c","modified":"2022-05-13T23:07:47.910705Z","created":"2022-05-13T23:05:56.093353Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:3300","nonce":0},{"type":"v1","addr":"172.21.15.2:6789","nonce":0}]},"addr":"172.21.15.2:6789/0","public_addr":"172.21.15.2:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:3300","nonce":0},{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2022-05-13T23:07:53.323 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:07:52 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-05-13T23:07:53.323 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:07:52 smithi002 ceph-mon[71816]: mon.a calling monitor election 2022-05-13T23:07:53.323 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:07:52 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-05-13T23:07:53.323 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:07:52 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-05-13T23:07:53.324 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:07:52 smithi002 ceph-mon[71816]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-05-13T23:07:53.324 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:07:52 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-05-13T23:07:53.324 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:07:52 smithi002 ceph-mon[71816]: mon.b calling monitor election 2022-05-13T23:07:53.324 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:07:52 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-05-13T23:07:53.325 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:07:52 smithi002 ceph-mon[71816]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-05-13T23:07:53.325 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:07:52 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-05-13T23:07:53.325 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:07:52 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-05-13T23:07:53.325 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:07:52 smithi002 ceph-mon[71816]: mon.a is new leader, mons a,b in quorum (ranks 0,1) 2022-05-13T23:07:53.325 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:07:52 smithi002 ceph-mon[71816]: monmap e2: 2 mons at {a=[v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0],b=[v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0]} 2022-05-13T23:07:53.326 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:07:52 smithi002 ceph-mon[71816]: fsmap 2022-05-13T23:07:53.326 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:07:52 smithi002 ceph-mon[71816]: osdmap e4: 0 total, 0 up, 0 in 2022-05-13T23:07:53.326 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:07:52 smithi002 ceph-mon[71816]: mgrmap e13: x(active, since 47s) 2022-05-13T23:07:53.326 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:07:52 smithi002 ceph-mon[71816]: overall HEALTH_OK 2022-05-13T23:07:53.327 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:07:52 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:07:53.327 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:07:52 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2022-05-13T23:07:53.327 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:07:52 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-13T23:07:53.329 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:07:53.321+0000 7f6443e6a700 1 -- 172.21.15.115:0/718060714 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f6420034de0 msgr2=0x7f6420037290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:07:53.329 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:07:53.321+0000 7f6443e6a700 1 --2- 172.21.15.115:0/718060714 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f6420034de0 0x7f6420037290 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f643800b010 tx=0x7f6438002040 comp rx=0 tx=0).stop 2022-05-13T23:07:53.329 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:07:53.321+0000 7f6443e6a700 1 -- 172.21.15.115:0/718060714 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f643c10c8c0 msgr2=0x7f643c115230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:07:53.330 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:07:53.321+0000 7f6443e6a700 1 --2- 172.21.15.115:0/718060714 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f643c10c8c0 0x7f643c115230 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f642c00b460 tx=0x7f642c009060 comp rx=0 tx=0).stop 2022-05-13T23:07:53.330 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:07:53.321+0000 7f6443e6a700 1 -- 172.21.15.115:0/718060714 shutdown_connections 2022-05-13T23:07:53.330 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:07:53.321+0000 7f6443e6a700 1 --2- 172.21.15.115:0/718060714 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6428003400 0x7f643c10c380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:07:53.331 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:07:53.321+0000 7f6443e6a700 1 --2- 172.21.15.115:0/718060714 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f6420034de0 0x7f6420037290 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:07:53.331 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:07:53.322+0000 7f6443e6a700 1 --2- 172.21.15.115:0/718060714 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f643c10c8c0 0x7f643c115230 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:07:53.332 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:07:53.322+0000 7f6443e6a700 1 -- 172.21.15.115:0/718060714 >> 172.21.15.115:0/718060714 conn(0x7f643c070f00 msgr2=0x7f643c0fbb80 unknown :-1 s=STATE_NONE l=0).mark_down 2022-05-13T23:07:53.332 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:07:53.322+0000 7f6443e6a700 1 -- 172.21.15.115:0/718060714 shutdown_connections 2022-05-13T23:07:53.333 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:07:53.322+0000 7f6443e6a700 1 -- 172.21.15.115:0/718060714 wait complete. 2022-05-13T23:07:53.333 INFO:teuthology.orchestra.run.smithi115.stderr:dumped monmap epoch 2 2022-05-13T23:07:54.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:07:54 smithi047 ceph-mon[77147]: Deploying daemon mon.c on smithi115 2022-05-13T23:07:54.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:07:54 smithi047 ceph-mon[77147]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-05-13T23:07:54.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:07:54 smithi047 ceph-mon[77147]: from='client.? 172.21.15.115:0/718060714' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2022-05-13T23:07:54.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:07:54 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-05-13T23:07:54.322 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:07:54 smithi002 ceph-mon[71816]: Deploying daemon mon.c on smithi115 2022-05-13T23:07:54.322 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:07:54 smithi002 ceph-mon[71816]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-05-13T23:07:54.323 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:07:54 smithi002 ceph-mon[71816]: from='client.? 172.21.15.115:0/718060714' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2022-05-13T23:07:54.323 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:07:54 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-05-13T23:07:54.701 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2022-05-13T23:07:54.702 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:23b0658391ce94ca40799469eec00c28b298816c shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 16cf5f88-d311-11ec-be40-001a4aab830c -- ceph mon dump -f json 2022-05-13T23:07:55.359 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/16cf5f88-d311-11ec-be40-001a4aab830c/mon.c/config 2022-05-13T23:07:55.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:07:55 smithi115 systemd[1]: Starting Ceph mon.c for 16cf5f88-d311-11ec-be40-001a4aab830c... 2022-05-13T23:07:56.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:07:56 smithi047 ceph-mon[77147]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-05-13T23:07:56.572 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:07:56 smithi002 ceph-mon[71816]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-05-13T23:07:56.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:07:56 smithi115 ceph-mon[78593]: set uid:gid to 167:167 (ceph:ceph) 2022-05-13T23:07:56.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:07:56 smithi115 ceph-mon[78593]: ceph version 17.2.0-188-g23b06583 (23b0658391ce94ca40799469eec00c28b298816c) quincy (stable), process ceph-mon, pid 7 2022-05-13T23:07:56.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:07:56 smithi115 ceph-mon[78593]: pidfile_write: ignore empty --pid-file 2022-05-13T23:07:56.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:07:56 smithi115 ceph-mon[78593]: load: jerasure load: lrc 2022-05-13T23:07:56.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:07:56 smithi115 ceph-mon[78593]: rocksdb: RocksDB version: 6.15.5 2022-05-13T23:07:56.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:07:56 smithi115 ceph-mon[78593]: rocksdb: Git sha rocksdb_build_git_sha:@0@ 2022-05-13T23:07:56.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:07:56 smithi115 ceph-mon[78593]: rocksdb: Compile date May 12 2022 2022-05-13T23:07:56.862 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:07:56 smithi115 ceph-mon[78593]: rocksdb: DB SUMMARY 2022-05-13T23:07:56.862 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:07:56 smithi115 ceph-mon[78593]: rocksdb: DB Session ID: WY0HESN4QS60QXEQLCWJ 2022-05-13T23:07:56.862 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:07:56 smithi115 ceph-mon[78593]: rocksdb: CURRENT file: CURRENT 2022-05-13T23:07:56.862 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:07:56 smithi115 ceph-mon[78593]: rocksdb: IDENTITY file: IDENTITY 2022-05-13T23:07:56.863 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:07:56 smithi115 ceph-mon[78593]: rocksdb: MANIFEST file: MANIFEST-000003 size: 57 Bytes 2022-05-13T23:07:56.863 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:07:56 smithi115 ceph-mon[78593]: rocksdb: SST files in /var/lib/ceph/mon/ceph-c/store.db dir, Total Num: 0, files: 2022-05-13T23:07:56.863 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:07:56 smithi115 ceph-mon[78593]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-c/store.db: 000004.log size: 636 ; 2022-05-13T23:07:56.863 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:07:56 smithi115 ceph-mon[78593]: rocksdb: Options.error_if_exists: 0 2022-05-13T23:07:56.863 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:07:56 smithi115 ceph-mon[78593]: rocksdb: Options.create_if_missing: 0 2022-05-13T23:07:56.864 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:07:56 smithi115 ceph-mon[78593]: rocksdb: Options.paranoid_checks: 1 2022-05-13T23:07:56.864 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:07:56 smithi115 ceph-mon[78593]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2022-05-13T23:07:56.864 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:07:56 smithi115 ceph-mon[78593]: rocksdb: Options.env: 0x55a088c0f860 2022-05-13T23:07:56.864 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:07:56 smithi115 ceph-mon[78593]: rocksdb: Options.fs: Posix File System 2022-05-13T23:07:56.864 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:07:56 smithi115 ceph-mon[78593]: rocksdb: Options.info_log: 0x55a0899c7dc0 2022-05-13T23:07:56.865 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:07:56 smithi115 ceph-mon[78593]: rocksdb: Options.max_file_opening_threads: 16 2022-05-13T23:07:56.865 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:07:56 smithi115 ceph-mon[78593]: rocksdb: Options.statistics: (nil) 2022-05-13T23:07:56.865 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:07:56 smithi115 ceph-mon[78593]: rocksdb: Options.use_fsync: 0 2022-05-13T23:07:56.865 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:07:56 smithi115 ceph-mon[78593]: rocksdb: Options.max_log_file_size: 0 2022-05-13T23:07:56.866 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:07:56 smithi115 ceph-mon[78593]: rocksdb: Options.max_manifest_file_size: 1073741824 2022-05-13T23:07:56.866 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:07:56 smithi115 ceph-mon[78593]: rocksdb: Options.log_file_time_to_roll: 0 2022-05-13T23:07:56.866 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:07:56 smithi115 ceph-mon[78593]: rocksdb: Options.keep_log_file_num: 1000 2022-05-13T23:07:56.866 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:07:56 smithi115 ceph-mon[78593]: rocksdb: Options.recycle_log_file_num: 0 2022-05-13T23:07:56.867 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:07:56 smithi115 ceph-mon[78593]: rocksdb: Options.allow_fallocate: 1 2022-05-13T23:07:56.867 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:07:56 smithi115 ceph-mon[78593]: rocksdb: Options.allow_mmap_reads: 0 2022-05-13T23:07:56.867 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:07:56 smithi115 ceph-mon[78593]: rocksdb: Options.allow_mmap_writes: 0 2022-05-13T23:07:56.867 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:07:56 smithi115 ceph-mon[78593]: rocksdb: Options.use_direct_reads: 0 2022-05-13T23:07:56.867 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:07:56 smithi115 ceph-mon[78593]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2022-05-13T23:07:56.868 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:07:56 smithi115 ceph-mon[78593]: rocksdb: Options.create_missing_column_families: 0 2022-05-13T23:07:56.868 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:07:56 smithi115 ceph-mon[78593]: rocksdb: Options.db_log_dir: 2022-05-13T23:07:56.868 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:07:56 smithi115 ceph-mon[78593]: rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-c/store.db 2022-05-13T23:07:56.868 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:07:56 smithi115 ceph-mon[78593]: rocksdb: Options.table_cache_numshardbits: 6 2022-05-13T23:07:56.869 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:07:56 smithi115 ceph-mon[78593]: rocksdb: Options.WAL_ttl_seconds: 0 2022-05-13T23:07:56.869 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:07:56 smithi115 ceph-mon[78593]: rocksdb: Options.WAL_size_limit_MB: 0 2022-05-13T23:07:56.869 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:07:56 smithi115 ceph-mon[78593]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2022-05-13T23:07:56.869 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:07:56 smithi115 ceph-mon[78593]: rocksdb: Options.manifest_preallocation_size: 4194304 2022-05-13T23:07:56.870 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:07:56 smithi115 ceph-mon[78593]: rocksdb: Options.is_fd_close_on_exec: 1 2022-05-13T23:07:56.870 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:07:56 smithi115 ceph-mon[78593]: rocksdb: Options.advise_random_on_open: 1 2022-05-13T23:07:56.870 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:07:56 smithi115 ceph-mon[78593]: rocksdb: Options.db_write_buffer_size: 0 2022-05-13T23:07:56.870 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:07:56 smithi115 ceph-mon[78593]: rocksdb: Options.write_buffer_manager: 0x55a089ab81e0 2022-05-13T23:07:56.871 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:07:56 smithi115 ceph-mon[78593]: rocksdb: Options.access_hint_on_compaction_start: 1 2022-05-13T23:07:56.871 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:07:56 smithi115 ceph-mon[78593]: rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2022-05-13T23:07:56.871 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:07:56 smithi115 ceph-mon[78593]: rocksdb: Options.random_access_max_buffer_size: 1048576 2022-05-13T23:07:56.871 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:07:56 smithi115 ceph-mon[78593]: rocksdb: Options.use_adaptive_mutex: 0 2022-05-13T23:07:56.872 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:07:56 smithi115 ceph-mon[78593]: rocksdb: Options.rate_limiter: (nil) 2022-05-13T23:07:56.872 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:07:56 smithi115 ceph-mon[78593]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2022-05-13T23:07:56.872 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:07:56 smithi115 ceph-mon[78593]: rocksdb: Options.wal_recovery_mode: 2 2022-05-13T23:07:56.872 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:07:56 smithi115 ceph-mon[78593]: rocksdb: Options.enable_thread_tracking: 0 2022-05-13T23:07:56.873 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:07:56 smithi115 ceph-mon[78593]: rocksdb: Options.enable_pipelined_write: 0 2022-05-13T23:07:56.873 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:07:56 smithi115 ceph-mon[78593]: rocksdb: Options.unordered_write: 0 2022-05-13T23:07:56.873 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:07:56 smithi115 ceph-mon[78593]: rocksdb: Options.allow_concurrent_memtable_write: 1 2022-05-13T23:07:56.873 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:07:56 smithi115 ceph-mon[78593]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2022-05-13T23:07:56.874 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:07:56 smithi115 ceph-mon[78593]: rocksdb: Options.write_thread_max_yield_usec: 100 2022-05-13T23:07:56.874 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:07:56 smithi115 ceph-mon[78593]: rocksdb: Options.write_thread_slow_yield_usec: 3 2022-05-13T23:07:56.874 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:07:56 smithi115 ceph-mon[78593]: rocksdb: Options.row_cache: None 2022-05-13T23:07:56.874 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:07:56 smithi115 ceph-mon[78593]: rocksdb: Options.wal_filter: None 2022-05-13T23:07:56.874 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:07:56 smithi115 ceph-mon[78593]: rocksdb: Options.avoid_flush_during_recovery: 0 2022-05-13T23:07:56.875 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:07:56 smithi115 ceph-mon[78593]: rocksdb: Options.allow_ingest_behind: 0 2022-05-13T23:07:56.875 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:07:56 smithi115 ceph-mon[78593]: rocksdb: Options.preserve_deletes: 0 2022-05-13T23:07:56.875 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:07:56 smithi115 ceph-mon[78593]: rocksdb: Options.two_write_queues: 0 2022-05-13T23:07:56.875 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:07:56 smithi115 ceph-mon[78593]: rocksdb: Options.manual_wal_flush: 0 2022-05-13T23:07:56.876 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:07:56 smithi115 ceph-mon[78593]: rocksdb: Options.atomic_flush: 0 2022-05-13T23:07:56.876 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:07:56 smithi115 ceph-mon[78593]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2022-05-13T23:07:56.876 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:07:56 smithi115 ceph-mon[78593]: rocksdb: Options.persist_stats_to_disk: 0 2022-05-13T23:07:56.876 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:07:56 smithi115 ceph-mon[78593]: rocksdb: Options.write_dbid_to_manifest: 0 2022-05-13T23:07:56.876 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:07:56 smithi115 ceph-mon[78593]: rocksdb: Options.log_readahead_size: 0 2022-05-13T23:07:56.877 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:07:56 smithi115 ceph-mon[78593]: rocksdb: Options.file_checksum_gen_factory: Unknown 2022-05-13T23:07:56.877 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:07:56 smithi115 ceph-mon[78593]: rocksdb: Options.best_efforts_recovery: 0 2022-05-13T23:07:56.877 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:07:56 smithi115 ceph-mon[78593]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2022-05-13T23:07:56.877 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:07:56 smithi115 ceph-mon[78593]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2022-05-13T23:07:56.878 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:07:56 smithi115 ceph-mon[78593]: rocksdb: Options.allow_data_in_errors: 0 2022-05-13T23:07:56.878 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:07:56 smithi115 ceph-mon[78593]: rocksdb: Options.db_host_id: __hostname__ 2022-05-13T23:07:56.878 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:07:56 smithi115 ceph-mon[78593]: rocksdb: Options.max_background_jobs: 2 2022-05-13T23:07:56.878 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:07:56 smithi115 ceph-mon[78593]: rocksdb: Options.max_background_compactions: -1 2022-05-13T23:07:56.879 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:07:56 smithi115 ceph-mon[78593]: rocksdb: Options.max_subcompactions: 1 2022-05-13T23:07:56.879 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:07:56 smithi115 ceph-mon[78593]: rocksdb: Options.avoid_flush_during_shutdown: 0 2022-05-13T23:07:56.879 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:07:56 smithi115 ceph-mon[78593]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2022-05-13T23:07:56.879 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:07:56 smithi115 ceph-mon[78593]: rocksdb: Options.delayed_write_rate : 16777216 2022-05-13T23:07:56.880 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:07:56 smithi115 ceph-mon[78593]: rocksdb: Options.max_total_wal_size: 0 2022-05-13T23:07:56.880 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:07:56 smithi115 ceph-mon[78593]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2022-05-13T23:07:56.880 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:07:56 smithi115 ceph-mon[78593]: rocksdb: Options.stats_dump_period_sec: 600 2022-05-13T23:07:56.880 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:07:56 smithi115 ceph-mon[78593]: rocksdb: Options.stats_persist_period_sec: 600 2022-05-13T23:07:56.880 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:07:56 smithi115 ceph-mon[78593]: rocksdb: Options.stats_history_buffer_size: 1048576 2022-05-13T23:07:56.881 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:07:56 smithi115 ceph-mon[78593]: rocksdb: Options.max_open_files: -1 2022-05-13T23:07:56.881 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:07:56 smithi115 ceph-mon[78593]: rocksdb: Options.bytes_per_sync: 0 2022-05-13T23:07:56.881 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:07:56 smithi115 ceph-mon[78593]: rocksdb: Options.wal_bytes_per_sync: 0 2022-05-13T23:07:56.881 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:07:56 smithi115 ceph-mon[78593]: rocksdb: Options.strict_bytes_per_sync: 0 2022-05-13T23:07:56.882 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:07:56 smithi115 ceph-mon[78593]: rocksdb: Options.compaction_readahead_size: 0 2022-05-13T23:07:56.882 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:07:56 smithi115 ceph-mon[78593]: rocksdb: Options.max_background_flushes: -1 2022-05-13T23:07:56.882 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:07:56 smithi115 ceph-mon[78593]: rocksdb: Compression algorithms supported: 2022-05-13T23:07:56.882 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:07:56 smithi115 ceph-mon[78593]: rocksdb: kZSTDNotFinalCompression supported: 0 2022-05-13T23:07:56.883 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:07:56 smithi115 ceph-mon[78593]: rocksdb: kZSTD supported: 0 2022-05-13T23:07:56.883 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:07:56 smithi115 ceph-mon[78593]: rocksdb: kXpressCompression supported: 0 2022-05-13T23:07:56.883 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:07:56 smithi115 ceph-mon[78593]: rocksdb: kLZ4HCCompression supported: 1 2022-05-13T23:07:56.883 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:07:56 smithi115 ceph-mon[78593]: rocksdb: kLZ4Compression supported: 1 2022-05-13T23:07:56.883 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:07:56 smithi115 ceph-mon[78593]: rocksdb: kBZip2Compression supported: 0 2022-05-13T23:07:56.884 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:07:56 smithi115 ceph-mon[78593]: rocksdb: kZlibCompression supported: 1 2022-05-13T23:07:56.884 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:07:56 smithi115 ceph-mon[78593]: rocksdb: kSnappyCompression supported: 1 2022-05-13T23:07:56.884 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:07:56 smithi115 ceph-mon[78593]: rocksdb: Fast CRC32 supported: Supported on x86 2022-05-13T23:07:56.884 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:07:56 smithi115 ceph-mon[78593]: rocksdb: [db/version_set.cc:4725] Recovering from manifest file: /var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000003 2022-05-13T23:07:56.885 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:07:56 smithi115 ceph-mon[78593]: rocksdb: [db/column_family.cc:597] --------------- Options for column family [default]: 2022-05-13T23:07:56.885 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:07:56 smithi115 ceph-mon[78593]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2022-05-13T23:07:56.885 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:07:56 smithi115 ceph-mon[78593]: rocksdb: Options.merge_operator: 2022-05-13T23:07:56.885 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:07:56 smithi115 ceph-mon[78593]: rocksdb: Options.compaction_filter: None 2022-05-13T23:07:56.885 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:07:56 smithi115 ceph-mon[78593]: rocksdb: Options.compaction_filter_factory: None 2022-05-13T23:07:56.886 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:07:56 smithi115 ceph-mon[78593]: rocksdb: Options.sst_partitioner_factory: None 2022-05-13T23:07:56.886 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:07:56 smithi115 ceph-mon[78593]: rocksdb: Options.memtable_factory: SkipListFactory 2022-05-13T23:07:56.886 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:07:56 smithi115 ceph-mon[78593]: rocksdb: Options.table_factory: BlockBasedTable 2022-05-13T23:07:56.886 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:07:56 smithi115 ceph-mon[78593]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55a089993d10) 2022-05-13T23:07:56.887 INFO:journalctl@ceph.mon.c.smithi115.stdout: cache_index_and_filter_blocks: 1 2022-05-13T23:07:56.887 INFO:journalctl@ceph.mon.c.smithi115.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2022-05-13T23:07:56.887 INFO:journalctl@ceph.mon.c.smithi115.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2022-05-13T23:07:56.887 INFO:journalctl@ceph.mon.c.smithi115.stdout: pin_top_level_index_and_filter: 1 2022-05-13T23:07:56.887 INFO:journalctl@ceph.mon.c.smithi115.stdout: index_type: 0 2022-05-13T23:07:56.888 INFO:journalctl@ceph.mon.c.smithi115.stdout: data_block_index_type: 0 2022-05-13T23:07:56.888 INFO:journalctl@ceph.mon.c.smithi115.stdout: index_shortening: 1 2022-05-13T23:07:56.888 INFO:journalctl@ceph.mon.c.smithi115.stdout: data_block_hash_table_util_ratio: 0.750000 2022-05-13T23:07:56.888 INFO:journalctl@ceph.mon.c.smithi115.stdout: hash_index_allow_collision: 1 2022-05-13T23:07:56.889 INFO:journalctl@ceph.mon.c.smithi115.stdout: checksum: 1 2022-05-13T23:07:56.889 INFO:journalctl@ceph.mon.c.smithi115.stdout: no_block_cache: 0 2022-05-13T23:07:56.889 INFO:journalctl@ceph.mon.c.smithi115.stdout: block_cache: 0x55a0899fe170 2022-05-13T23:07:56.889 INFO:journalctl@ceph.mon.c.smithi115.stdout: block_cache_name: BinnedLRUCache 2022-05-13T23:07:56.889 INFO:journalctl@ceph.mon.c.smithi115.stdout: block_cache_options: 2022-05-13T23:07:56.890 INFO:journalctl@ceph.mon.c.smithi115.stdout: capacity : 536870912 2022-05-13T23:07:56.890 INFO:journalctl@ceph.mon.c.smithi115.stdout: num_shard_bits : 4 2022-05-13T23:07:56.890 INFO:journalctl@ceph.mon.c.smithi115.stdout: strict_capacity_limit : 0 2022-05-13T23:07:56.890 INFO:journalctl@ceph.mon.c.smithi115.stdout: high_pri_pool_ratio: 0.000 2022-05-13T23:07:56.890 INFO:journalctl@ceph.mon.c.smithi115.stdout: block_cache_compressed: (nil) 2022-05-13T23:07:56.891 INFO:journalctl@ceph.mon.c.smithi115.stdout: persistent_cache: (nil) 2022-05-13T23:07:56.891 INFO:journalctl@ceph.mon.c.smithi115.stdout: block_size: 4096 2022-05-13T23:07:56.891 INFO:journalctl@ceph.mon.c.smithi115.stdout: block_size_deviation: 10 2022-05-13T23:07:56.891 INFO:journalctl@ceph.mon.c.smithi115.stdout: block_restart_interval: 16 2022-05-13T23:07:56.891 INFO:journalctl@ceph.mon.c.smithi115.stdout: index_block_restart_interval: 1 2022-05-13T23:07:56.892 INFO:journalctl@ceph.mon.c.smithi115.stdout: metadata_block_size: 4096 2022-05-13T23:07:56.892 INFO:journalctl@ceph.mon.c.smithi115.stdout: partition_filters: 0 2022-05-13T23:07:56.892 INFO:journalctl@ceph.mon.c.smithi115.stdout: use_delta_encoding: 1 2022-05-13T23:07:56.892 INFO:journalctl@ceph.mon.c.smithi115.stdout: filter_policy: rocksdb.BuiltinBloomFilter 2022-05-13T23:07:56.893 INFO:journalctl@ceph.mon.c.smithi115.stdout: whole_key_filtering: 1 2022-05-13T23:07:56.893 INFO:journalctl@ceph.mon.c.smithi115.stdout: verify_compression: 0 2022-05-13T23:07:56.893 INFO:journalctl@ceph.mon.c.smithi115.stdout: read_amp_bytes_per_bit: 0 2022-05-13T23:07:56.893 INFO:journalctl@ceph.mon.c.smithi115.stdout: format_version: 4 2022-05-13T23:07:56.894 INFO:journalctl@ceph.mon.c.smithi115.stdout: enable_index_compression: 1 2022-05-13T23:07:56.894 INFO:journalctl@ceph.mon.c.smithi115.stdout: block_align: 0 2022-05-13T23:07:56.894 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:07:56 smithi115 ceph-mon[78593]: rocksdb: Options.write_buffer_size: 33554432 2022-05-13T23:07:56.894 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:07:56 smithi115 ceph-mon[78593]: rocksdb: Options.max_write_buffer_number: 2 2022-05-13T23:07:56.894 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:07:56 smithi115 ceph-mon[78593]: rocksdb: Options.compression: NoCompression 2022-05-13T23:07:56.895 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:07:56 smithi115 ceph-mon[78593]: rocksdb: Options.bottommost_compression: Disabled 2022-05-13T23:07:56.895 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:07:56 smithi115 ceph-mon[78593]: rocksdb: Options.prefix_extractor: nullptr 2022-05-13T23:07:56.895 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:07:56 smithi115 ceph-mon[78593]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2022-05-13T23:07:56.895 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:07:56 smithi115 ceph-mon[78593]: rocksdb: Options.num_levels: 7 2022-05-13T23:07:56.896 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:07:56 smithi115 ceph-mon[78593]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2022-05-13T23:07:56.896 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:07:56 smithi115 ceph-mon[78593]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2022-05-13T23:07:56.896 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:07:56 smithi115 ceph-mon[78593]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2022-05-13T23:07:56.896 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:07:56 smithi115 ceph-mon[78593]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2022-05-13T23:07:56.896 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:07:56 smithi115 ceph-mon[78593]: rocksdb: Options.bottommost_compression_opts.level: 32767 2022-05-13T23:07:56.897 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:07:56 smithi115 ceph-mon[78593]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2022-05-13T23:07:56.897 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:07:56 smithi115 ceph-mon[78593]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2022-05-13T23:07:56.897 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:07:56 smithi115 ceph-mon[78593]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2022-05-13T23:07:56.897 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:07:56 smithi115 ceph-mon[78593]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2022-05-13T23:07:56.898 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:07:56 smithi115 ceph-mon[78593]: rocksdb: Options.bottommost_compression_opts.enabled: false 2022-05-13T23:07:56.898 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:07:56 smithi115 ceph-mon[78593]: rocksdb: Options.compression_opts.window_bits: -14 2022-05-13T23:07:56.898 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:07:56 smithi115 ceph-mon[78593]: rocksdb: Options.compression_opts.level: 32767 2022-05-13T23:07:56.898 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:07:56 smithi115 ceph-mon[78593]: rocksdb: Options.compression_opts.strategy: 0 2022-05-13T23:07:56.900 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:07:56 smithi115 ceph-mon[78593]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2022-05-13T23:07:56.900 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:07:56 smithi115 ceph-mon[78593]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2022-05-13T23:07:56.900 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:07:56 smithi115 ceph-mon[78593]: rocksdb: Options.compression_opts.parallel_threads: 1 2022-05-13T23:07:56.901 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:07:56 smithi115 ceph-mon[78593]: rocksdb: Options.compression_opts.enabled: false 2022-05-13T23:07:56.901 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:07:56 smithi115 ceph-mon[78593]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2022-05-13T23:07:56.901 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:07:56 smithi115 ceph-mon[78593]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2022-05-13T23:07:56.901 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:07:56 smithi115 ceph-mon[78593]: rocksdb: Options.level0_stop_writes_trigger: 36 2022-05-13T23:07:56.902 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:07:56 smithi115 ceph-mon[78593]: rocksdb: Options.target_file_size_base: 67108864 2022-05-13T23:07:56.902 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:07:56 smithi115 ceph-mon[78593]: rocksdb: Options.target_file_size_multiplier: 1 2022-05-13T23:07:56.902 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:07:56 smithi115 ceph-mon[78593]: rocksdb: Options.max_bytes_for_level_base: 268435456 2022-05-13T23:07:56.902 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:07:56 smithi115 ceph-mon[78593]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2022-05-13T23:07:56.903 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:07:56 smithi115 ceph-mon[78593]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2022-05-13T23:07:56.903 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:07:56 smithi115 ceph-mon[78593]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2022-05-13T23:07:56.903 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:07:56 smithi115 ceph-mon[78593]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2022-05-13T23:07:56.903 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:07:56 smithi115 ceph-mon[78593]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2022-05-13T23:07:56.904 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:07:56 smithi115 ceph-mon[78593]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2022-05-13T23:07:56.904 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:07:56 smithi115 ceph-mon[78593]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2022-05-13T23:07:56.904 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:07:56 smithi115 ceph-mon[78593]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2022-05-13T23:07:56.904 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:07:56 smithi115 ceph-mon[78593]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2022-05-13T23:07:56.904 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:07:56 smithi115 ceph-mon[78593]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2022-05-13T23:07:56.905 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:07:56 smithi115 ceph-mon[78593]: rocksdb: Options.max_compaction_bytes: 1677721600 2022-05-13T23:07:56.905 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:07:56 smithi115 ceph-mon[78593]: rocksdb: Options.arena_block_size: 4194304 2022-05-13T23:07:56.905 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:07:56 smithi115 ceph-mon[78593]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2022-05-13T23:07:56.905 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:07:56 smithi115 ceph-mon[78593]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2022-05-13T23:07:56.906 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:07:56 smithi115 ceph-mon[78593]: rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2022-05-13T23:07:56.906 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:07:56 smithi115 ceph-mon[78593]: rocksdb: Options.disable_auto_compactions: 0 2022-05-13T23:07:56.906 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:07:56 smithi115 ceph-mon[78593]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2022-05-13T23:07:56.906 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:07:56 smithi115 ceph-mon[78593]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2022-05-13T23:07:56.906 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:07:56 smithi115 ceph-mon[78593]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2022-05-13T23:07:56.907 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:07:56 smithi115 ceph-mon[78593]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2022-05-13T23:07:56.907 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:07:56 smithi115 ceph-mon[78593]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2022-05-13T23:07:56.907 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:07:56 smithi115 ceph-mon[78593]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2022-05-13T23:07:56.907 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:07:56 smithi115 ceph-mon[78593]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2022-05-13T23:07:56.908 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:07:56 smithi115 ceph-mon[78593]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2022-05-13T23:07:56.908 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:07:56 smithi115 ceph-mon[78593]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2022-05-13T23:07:56.908 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:07:56 smithi115 ceph-mon[78593]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2022-05-13T23:07:56.908 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:07:56 smithi115 ceph-mon[78593]: rocksdb: Options.table_properties_collectors: 2022-05-13T23:07:56.908 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:07:56 smithi115 ceph-mon[78593]: rocksdb: Options.inplace_update_support: 0 2022-05-13T23:07:56.909 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:07:56 smithi115 ceph-mon[78593]: rocksdb: Options.inplace_update_num_locks: 10000 2022-05-13T23:07:56.909 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:07:56 smithi115 ceph-mon[78593]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2022-05-13T23:07:56.909 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:07:56 smithi115 ceph-mon[78593]: rocksdb: Options.memtable_whole_key_filtering: 0 2022-05-13T23:07:56.909 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:07:56 smithi115 ceph-mon[78593]: rocksdb: Options.memtable_huge_page_size: 0 2022-05-13T23:07:56.910 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:07:56 smithi115 ceph-mon[78593]: rocksdb: Options.bloom_locality: 0 2022-05-13T23:07:56.910 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:07:56 smithi115 ceph-mon[78593]: rocksdb: Options.max_successive_merges: 0 2022-05-13T23:07:56.910 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:07:56 smithi115 ceph-mon[78593]: rocksdb: Options.optimize_filters_for_hits: 0 2022-05-13T23:07:56.910 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:07:56 smithi115 ceph-mon[78593]: rocksdb: Options.paranoid_file_checks: 0 2022-05-13T23:07:56.911 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:07:56 smithi115 ceph-mon[78593]: rocksdb: Options.force_consistency_checks: 1 2022-05-13T23:07:56.911 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:07:56 smithi115 ceph-mon[78593]: rocksdb: Options.report_bg_io_stats: 0 2022-05-13T23:07:56.911 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:07:56 smithi115 ceph-mon[78593]: rocksdb: Options.ttl: 2592000 2022-05-13T23:07:56.911 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:07:56 smithi115 ceph-mon[78593]: rocksdb: Options.periodic_compaction_seconds: 0 2022-05-13T23:07:56.911 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:07:56 smithi115 ceph-mon[78593]: rocksdb: Options.enable_blob_files: false 2022-05-13T23:07:56.912 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:07:56 smithi115 ceph-mon[78593]: rocksdb: Options.min_blob_size: 0 2022-05-13T23:07:56.912 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:07:56 smithi115 ceph-mon[78593]: rocksdb: Options.blob_file_size: 268435456 2022-05-13T23:07:56.912 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:07:56 smithi115 ceph-mon[78593]: rocksdb: Options.blob_compression_type: NoCompression 2022-05-13T23:07:56.912 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:07:56 smithi115 ceph-mon[78593]: rocksdb: Options.enable_blob_garbage_collection: false 2022-05-13T23:07:56.912 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:07:56 smithi115 ceph-mon[78593]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2022-05-13T23:07:56.913 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:07:56 smithi115 ceph-mon[78593]: rocksdb: [db/version_set.cc:4773] Recovered from manifest file:/var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000003 succeeded,manifest_file_number is 3, next_file_number is 5, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2022-05-13T23:07:56.913 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:07:56 smithi115 ceph-mon[78593]: rocksdb: [db/version_set.cc:4782] Column family [default] (ID 0), log number is 0 2022-05-13T23:07:56.913 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:07:56 smithi115 ceph-mon[78593]: rocksdb: [db/version_set.cc:4083] Creating manifest 7 2022-05-13T23:07:56.913 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:07:56 smithi115 ceph-mon[78593]: rocksdb: EVENT_LOG_v1 {"time_micros": 1652483276729434, "job": 1, "event": "recovery_started", "wal_files": [4]} 2022-05-13T23:07:56.914 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:07:56 smithi115 ceph-mon[78593]: rocksdb: [db/db_impl/db_impl_open.cc:847] Recovering log #4 mode 2 2022-05-13T23:07:56.914 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:07:56 smithi115 ceph-mon[78593]: rocksdb: [table/block_based/filter_policy.cc:996] Using legacy Bloom filter with high (20) bits/key. Dramatic filter space and/or accuracy improvement is available with format_version>=5. 2022-05-13T23:07:56.914 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:07:56 smithi115 ceph-mon[78593]: rocksdb: EVENT_LOG_v1 {"time_micros": 1652483276729803, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1670, "file_checksum": "", "file_checksum_func_name": "Unknown", "table_properties": {"data_size": 648, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 526, "raw_average_value_size": 105, "num_data_blocks": 1, "num_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "rocksdb.BuiltinBloomFilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ", "creation_time": 1652483276, "oldest_key_time": 0, "file_creation_time": 0, "db_id": "a5e567f9-a485-4546-9657-c59ba50bf7dc", "db_session_id": "WY0HESN4QS60QXEQLCWJ"}} 2022-05-13T23:07:56.914 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:07:56 smithi115 ceph-mon[78593]: rocksdb: [db/version_set.cc:4083] Creating manifest 9 2022-05-13T23:07:56.915 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:07:56 smithi115 ceph-mon[78593]: rocksdb: EVENT_LOG_v1 {"time_micros": 1652483276730227, "job": 1, "event": "recovery_finished"} 2022-05-13T23:07:56.915 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:07:56 smithi115 ceph-mon[78593]: rocksdb: [file/delete_scheduler.cc:73] Deleted file /var/lib/ceph/mon/ceph-c/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2022-05-13T23:07:56.915 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:07:56 smithi115 ceph-mon[78593]: rocksdb: [db/db_impl/db_impl_open.cc:1701] SstFileManager instance 0x55a0899e4a80 2022-05-13T23:07:56.915 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:07:56 smithi115 ceph-mon[78593]: rocksdb: DB pointer 0x55a0899f4000 2022-05-13T23:07:56.915 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:07:56 smithi115 ceph-mon[78593]: rocksdb: [db/db_impl/db_impl.cc:902] ------- DUMPING STATS ------- 2022-05-13T23:07:56.916 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:07:56 smithi115 ceph-mon[78593]: rocksdb: [db/db_impl/db_impl.cc:903] 2022-05-13T23:07:56.916 INFO:journalctl@ceph.mon.c.smithi115.stdout: ** DB Stats ** 2022-05-13T23:07:56.916 INFO:journalctl@ceph.mon.c.smithi115.stdout: Uptime(secs): 0.0 total, 0.0 interval 2022-05-13T23:07:56.916 INFO:journalctl@ceph.mon.c.smithi115.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2022-05-13T23:07:56.917 INFO:journalctl@ceph.mon.c.smithi115.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2022-05-13T23:07:56.917 INFO:journalctl@ceph.mon.c.smithi115.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2022-05-13T23:07:56.917 INFO:journalctl@ceph.mon.c.smithi115.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2022-05-13T23:07:56.917 INFO:journalctl@ceph.mon.c.smithi115.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2022-05-13T23:07:56.918 INFO:journalctl@ceph.mon.c.smithi115.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2022-05-13T23:07:56.918 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-13T23:07:56.918 INFO:journalctl@ceph.mon.c.smithi115.stdout: ** Compaction Stats [default] ** 2022-05-13T23:07:56.918 INFO:journalctl@ceph.mon.c.smithi115.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2022-05-13T23:07:56.918 INFO:journalctl@ceph.mon.c.smithi115.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2022-05-13T23:07:56.919 INFO:journalctl@ceph.mon.c.smithi115.stdout: L0 1/0 1.63 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.6 0.00 0.00 1 0.000 0 0 2022-05-13T23:07:56.919 INFO:journalctl@ceph.mon.c.smithi115.stdout: Sum 1/0 1.63 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.6 0.00 0.00 1 0.000 0 0 2022-05-13T23:07:56.919 INFO:journalctl@ceph.mon.c.smithi115.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.6 0.00 0.00 1 0.000 0 0 2022-05-13T23:07:56.919 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-13T23:07:56.920 INFO:journalctl@ceph.mon.c.smithi115.stdout: ** Compaction Stats [default] ** 2022-05-13T23:07:56.920 INFO:journalctl@ceph.mon.c.smithi115.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2022-05-13T23:07:56.920 INFO:journalctl@ceph.mon.c.smithi115.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2022-05-13T23:07:56.920 INFO:journalctl@ceph.mon.c.smithi115.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 4.6 0.00 0.00 1 0.000 0 0 2022-05-13T23:07:56.921 INFO:journalctl@ceph.mon.c.smithi115.stdout: Uptime(secs): 0.0 total, 0.0 interval 2022-05-13T23:07:56.921 INFO:journalctl@ceph.mon.c.smithi115.stdout: Flush(GB): cumulative 0.000, interval 0.000 2022-05-13T23:07:56.921 INFO:journalctl@ceph.mon.c.smithi115.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2022-05-13T23:07:56.921 INFO:journalctl@ceph.mon.c.smithi115.stdout: AddFile(Total Files): cumulative 0, interval 0 2022-05-13T23:07:56.921 INFO:journalctl@ceph.mon.c.smithi115.stdout: AddFile(L0 Files): cumulative 0, interval 0 2022-05-13T23:07:56.922 INFO:journalctl@ceph.mon.c.smithi115.stdout: AddFile(Keys): cumulative 0, interval 0 2022-05-13T23:07:56.922 INFO:journalctl@ceph.mon.c.smithi115.stdout: Cumulative compaction: 0.00 GB write, 0.52 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2022-05-13T23:07:56.922 INFO:journalctl@ceph.mon.c.smithi115.stdout: Interval compaction: 0.00 GB write, 0.52 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2022-05-13T23:07:56.922 INFO:journalctl@ceph.mon.c.smithi115.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2022-05-13T23:07:56.923 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-13T23:07:56.923 INFO:journalctl@ceph.mon.c.smithi115.stdout: ** File Read Latency Histogram By Level [default] ** 2022-05-13T23:07:56.923 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-13T23:07:56.923 INFO:journalctl@ceph.mon.c.smithi115.stdout: ** Compaction Stats [default] ** 2022-05-13T23:07:56.924 INFO:journalctl@ceph.mon.c.smithi115.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2022-05-13T23:07:56.924 INFO:journalctl@ceph.mon.c.smithi115.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2022-05-13T23:07:56.924 INFO:journalctl@ceph.mon.c.smithi115.stdout: L0 1/0 1.63 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.6 0.00 0.00 1 0.000 0 0 2022-05-13T23:07:56.924 INFO:journalctl@ceph.mon.c.smithi115.stdout: Sum 1/0 1.63 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.6 0.00 0.00 1 0.000 0 0 2022-05-13T23:07:56.924 INFO:journalctl@ceph.mon.c.smithi115.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2022-05-13T23:07:56.925 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-13T23:07:56.925 INFO:journalctl@ceph.mon.c.smithi115.stdout: ** Compaction Stats [default] ** 2022-05-13T23:07:56.925 INFO:journalctl@ceph.mon.c.smithi115.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2022-05-13T23:07:56.925 INFO:journalctl@ceph.mon.c.smithi115.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2022-05-13T23:07:56.926 INFO:journalctl@ceph.mon.c.smithi115.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 4.6 0.00 0.00 1 0.000 0 0 2022-05-13T23:07:56.926 INFO:journalctl@ceph.mon.c.smithi115.stdout: Uptime(secs): 0.0 total, 0.0 interval 2022-05-13T23:07:56.926 INFO:journalctl@ceph.mon.c.smithi115.stdout: Flush(GB): cumulative 0.000, interval 0.000 2022-05-13T23:07:56.926 INFO:journalctl@ceph.mon.c.smithi115.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2022-05-13T23:07:56.927 INFO:journalctl@ceph.mon.c.smithi115.stdout: AddFile(Total Files): cumulative 0, interval 0 2022-05-13T23:07:56.927 INFO:journalctl@ceph.mon.c.smithi115.stdout: AddFile(L0 Files): cumulative 0, interval 0 2022-05-13T23:07:56.927 INFO:journalctl@ceph.mon.c.smithi115.stdout: AddFile(Keys): cumulative 0, interval 0 2022-05-13T23:07:56.927 INFO:journalctl@ceph.mon.c.smithi115.stdout: Cumulative compaction: 0.00 GB write, 0.51 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2022-05-13T23:07:56.927 INFO:journalctl@ceph.mon.c.smithi115.stdout: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2022-05-13T23:07:56.928 INFO:journalctl@ceph.mon.c.smithi115.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2022-05-13T23:07:56.928 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-13T23:07:56.928 INFO:journalctl@ceph.mon.c.smithi115.stdout: ** File Read Latency Histogram By Level [default] ** 2022-05-13T23:07:56.928 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:07:56 smithi115 ceph-mon[78593]: mon.c does not exist in monmap, will attempt to join an existing cluster 2022-05-13T23:07:56.929 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:07:56 smithi115 ceph-mon[78593]: using public_addr v2:172.21.15.115:0/0 -> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] 2022-05-13T23:07:56.929 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:07:56 smithi115 ceph-mon[78593]: starting mon.c rank -1 at public addrs [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] at bind addrs [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon_data /var/lib/ceph/mon/ceph-c fsid 16cf5f88-d311-11ec-be40-001a4aab830c 2022-05-13T23:07:56.929 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:07:56 smithi115 ceph-mon[78593]: mon.c@-1(???) e0 preinit fsid 16cf5f88-d311-11ec-be40-001a4aab830c 2022-05-13T23:07:56.929 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:07:56 smithi115 ceph-mon[78593]: mon.c@-1(synchronizing).mds e1 new map 2022-05-13T23:07:56.929 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:07:56 smithi115 ceph-mon[78593]: mon.c@-1(synchronizing).mds e1 print_map 2022-05-13T23:07:56.930 INFO:journalctl@ceph.mon.c.smithi115.stdout: e1 2022-05-13T23:07:56.930 INFO:journalctl@ceph.mon.c.smithi115.stdout: enable_multiple, ever_enabled_multiple: 1,1 2022-05-13T23:07:56.930 INFO:journalctl@ceph.mon.c.smithi115.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2022-05-13T23:07:56.930 INFO:journalctl@ceph.mon.c.smithi115.stdout: legacy client fscid: -1 2022-05-13T23:07:56.931 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-13T23:07:56.931 INFO:journalctl@ceph.mon.c.smithi115.stdout: No filesystems configured 2022-05-13T23:07:56.931 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:07:56 smithi115 ceph-mon[78593]: mon.c@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2022-05-13T23:07:56.931 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:07:56 smithi115 ceph-mon[78593]: mon.c@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2022-05-13T23:07:56.931 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:07:56 smithi115 ceph-mon[78593]: mon.c@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2022-05-13T23:07:56.932 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:07:56 smithi115 ceph-mon[78593]: mon.c@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2022-05-13T23:07:56.932 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:07:56 smithi115 ceph-mon[78593]: mon.c@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2022-05-13T23:07:56.932 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:07:56 smithi115 ceph-mon[78593]: mon.c@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2022-05-13T23:07:56.932 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:07:56 smithi115 ceph-mon[78593]: mon.c@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2022-05-13T23:07:56.933 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:07:56 smithi115 ceph-mon[78593]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2022-05-13T23:07:56.933 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:07:56 smithi115 ceph-mon[78593]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2022-05-13T23:07:56.933 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:07:56 smithi115 ceph-mon[78593]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2022-05-13T23:07:56.933 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:07:56 smithi115 ceph-mon[78593]: Deploying cephadm binary to smithi115 2022-05-13T23:07:56.933 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:07:56 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:07:56.934 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:07:56 smithi115 ceph-mon[78593]: Added host smithi115 2022-05-13T23:07:56.934 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:07:56 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-13T23:07:56.934 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:07:56 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-13T23:07:56.934 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:07:56 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-13T23:07:56.935 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:07:56 smithi115 ceph-mon[78593]: Updating smithi047:/etc/ceph/ceph.conf 2022-05-13T23:07:56.935 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:07:56 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:07:56.935 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:07:56 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:07:56.935 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:07:56 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:07:56.935 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:07:56 smithi115 ceph-mon[78593]: Updating smithi047:/etc/ceph/ceph.client.admin.keyring 2022-05-13T23:07:56.936 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:07:56 smithi115 ceph-mon[78593]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-05-13T23:07:56.936 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:07:56 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:07:56.936 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:07:56 smithi115 ceph-mon[78593]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-05-13T23:07:56.936 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:07:56 smithi115 ceph-mon[78593]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2022-05-13T23:07:56.936 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:07:56 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:07:56.937 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:07:56 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:07:56.937 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:07:56 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi115", "name": "osd_memory_target"}]: dispatch 2022-05-13T23:07:56.938 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:07:56 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:07:56.938 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:07:56 smithi115 ceph-mon[78593]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-05-13T23:07:56.938 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:07:56 smithi115 ceph-mon[78593]: from='client.? 172.21.15.2:0/3530583463' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2022-05-13T23:07:56.938 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:07:56 smithi115 ceph-mon[78593]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-05-13T23:07:56.938 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:07:56 smithi115 ceph-mon[78593]: from='client.? 172.21.15.2:0/3530583463' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2022-05-13T23:07:56.939 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:07:56 smithi115 ceph-mon[78593]: osdmap e4: 0 total, 0 up, 0 in 2022-05-13T23:07:56.939 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:07:56 smithi115 ceph-mon[78593]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-05-13T23:07:56.939 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:07:56 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:07:56.939 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:07:56 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-13T23:07:56.940 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:07:56 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-13T23:07:56.940 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:07:56 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-13T23:07:56.940 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:07:56 smithi115 ceph-mon[78593]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi002:172.21.15.2=a;smithi047:172.21.15.47=b;smithi115:172.21.15.115=c", "target": ["mon-mgr", ""]}]: dispatch 2022-05-13T23:07:56.940 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:07:56 smithi115 ceph-mon[78593]: Saving service mon spec with placement smithi002:172.21.15.2=a;smithi047:172.21.15.47=b;smithi115:172.21.15.115=c;count:3 2022-05-13T23:07:56.941 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:07:56 smithi115 ceph-mon[78593]: Updating smithi115:/etc/ceph/ceph.conf 2022-05-13T23:07:56.941 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:07:56 smithi115 ceph-mon[78593]: Updating smithi115:/etc/ceph/ceph.client.admin.keyring 2022-05-13T23:07:56.941 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:07:56 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:07:56.941 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:07:56 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:07:56.942 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:07:56 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2022-05-13T23:07:56.942 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:07:56 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-13T23:07:56.942 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:07:56 smithi115 ceph-mon[78593]: Deploying daemon mon.b on smithi047 2022-05-13T23:07:56.942 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:07:56 smithi115 ceph-mon[78593]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-05-13T23:07:56.942 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:07:56 smithi115 ceph-mon[78593]: from='client.? 172.21.15.115:0/250495708' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2022-05-13T23:07:56.943 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:07:56 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-05-13T23:07:56.943 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:07:56 smithi115 ceph-mon[78593]: mon.a calling monitor election 2022-05-13T23:07:56.943 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:07:56 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-05-13T23:07:56.943 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:07:56 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-05-13T23:07:56.943 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:07:56 smithi115 ceph-mon[78593]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-05-13T23:07:56.944 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:07:56 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-05-13T23:07:56.944 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:07:56 smithi115 ceph-mon[78593]: mon.b calling monitor election 2022-05-13T23:07:56.944 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:07:56 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-05-13T23:07:56.944 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:07:56 smithi115 ceph-mon[78593]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-05-13T23:07:56.945 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:07:56 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-05-13T23:07:56.945 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:07:56 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-05-13T23:07:56.945 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:07:56 smithi115 ceph-mon[78593]: mon.a is new leader, mons a,b in quorum (ranks 0,1) 2022-05-13T23:07:56.945 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:07:56 smithi115 ceph-mon[78593]: monmap e2: 2 mons at {a=[v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0],b=[v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0]} 2022-05-13T23:07:56.946 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:07:56 smithi115 ceph-mon[78593]: fsmap 2022-05-13T23:07:56.946 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:07:56 smithi115 ceph-mon[78593]: osdmap e4: 0 total, 0 up, 0 in 2022-05-13T23:07:56.946 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:07:56 smithi115 ceph-mon[78593]: mgrmap e13: x(active, since 47s) 2022-05-13T23:07:56.946 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:07:56 smithi115 ceph-mon[78593]: overall HEALTH_OK 2022-05-13T23:07:56.946 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:07:56 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:07:56.947 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:07:56 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2022-05-13T23:07:56.947 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:07:56 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-13T23:07:56.947 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:07:56 smithi115 ceph-mon[78593]: Deploying daemon mon.c on smithi115 2022-05-13T23:07:56.947 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:07:56 smithi115 ceph-mon[78593]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-05-13T23:07:56.948 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:07:56 smithi115 ceph-mon[78593]: from='client.? 172.21.15.115:0/718060714' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2022-05-13T23:07:56.948 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:07:56 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-05-13T23:07:56.948 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:07:56 smithi115 ceph-mon[78593]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-05-13T23:07:56.948 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:07:56 smithi115 ceph-mon[78593]: mon.c@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2022-05-13T23:07:56.948 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:07:56 smithi115 ceph-mon[78593]: mon.c@-1(synchronizing) e2 handle_conf_change mon_allow_pool_delete 2022-05-13T23:07:57.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:07:56 smithi115 bash[78281]: 099efb3934a6ad5b39a18a9fb0d9cda7fdbb06c891967cd0e106ef105fda878b 2022-05-13T23:07:57.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:07:56 smithi115 systemd[1]: Started Ceph mon.c for 16cf5f88-d311-11ec-be40-001a4aab830c. 2022-05-13T23:08:02.081 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:08:01 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-05-13T23:08:02.081 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:08:01 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-05-13T23:08:02.082 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:08:01 smithi002 ceph-mon[71816]: mon.b calling monitor election 2022-05-13T23:08:02.082 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:08:01 smithi002 ceph-mon[71816]: mon.a calling monitor election 2022-05-13T23:08:02.082 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:08:01 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-05-13T23:08:02.082 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:08:01 smithi002 ceph-mon[71816]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-05-13T23:08:02.083 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:08:01 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-05-13T23:08:02.083 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:08:01 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-05-13T23:08:02.083 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:08:01 smithi002 ceph-mon[71816]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-05-13T23:08:02.083 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:08:01 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-05-13T23:08:02.084 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:08:01 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-05-13T23:08:02.084 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:08:01 smithi002 ceph-mon[71816]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-05-13T23:08:02.084 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:08:01 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-05-13T23:08:02.084 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:08:01 smithi002 ceph-mon[71816]: mon.a is new leader, mons a,b in quorum (ranks 0,1) 2022-05-13T23:08:02.085 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:08:01 smithi002 ceph-mon[71816]: monmap e3: 3 mons at {a=[v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0],b=[v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0],c=[v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0]} 2022-05-13T23:08:02.085 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:08:01 smithi002 ceph-mon[71816]: fsmap 2022-05-13T23:08:02.085 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:08:01 smithi002 ceph-mon[71816]: osdmap e4: 0 total, 0 up, 0 in 2022-05-13T23:08:02.085 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:08:01 smithi002 ceph-mon[71816]: mgrmap e13: x(active, since 56s) 2022-05-13T23:08:02.086 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:08:01 smithi002 ceph-mon[71816]: Health check failed: 1/3 mons down, quorum a,b (MON_DOWN) 2022-05-13T23:08:02.086 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:08:01 smithi002 ceph-mon[71816]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,b 2022-05-13T23:08:02.086 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:08:01 smithi002 ceph-mon[71816]: [WRN] MON_DOWN: 1/3 mons down, quorum a,b 2022-05-13T23:08:02.086 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:08:01 smithi002 ceph-mon[71816]: mon.c (rank 2) addr [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] is down (out of quorum) 2022-05-13T23:08:02.086 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:08:01 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:08:02.087 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:08:01 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-13T23:08:02.087 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:08:01 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-13T23:08:02.087 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:08:01 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-13T23:08:02.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:08:01 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-05-13T23:08:02.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:08:01 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-05-13T23:08:02.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:08:01 smithi047 ceph-mon[77147]: mon.b calling monitor election 2022-05-13T23:08:02.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:08:01 smithi047 ceph-mon[77147]: mon.a calling monitor election 2022-05-13T23:08:02.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:08:01 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-05-13T23:08:02.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:08:01 smithi047 ceph-mon[77147]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-05-13T23:08:02.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:08:01 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-05-13T23:08:02.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:08:01 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-05-13T23:08:02.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:08:01 smithi047 ceph-mon[77147]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-05-13T23:08:02.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:08:01 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-05-13T23:08:02.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:08:01 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-05-13T23:08:02.295 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:08:01 smithi047 ceph-mon[77147]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-05-13T23:08:02.295 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:08:01 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-05-13T23:08:02.295 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:08:01 smithi047 ceph-mon[77147]: mon.a is new leader, mons a,b in quorum (ranks 0,1) 2022-05-13T23:08:02.295 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:08:01 smithi047 ceph-mon[77147]: monmap e3: 3 mons at {a=[v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0],b=[v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0],c=[v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0]} 2022-05-13T23:08:02.296 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:08:01 smithi047 ceph-mon[77147]: fsmap 2022-05-13T23:08:02.296 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:08:01 smithi047 ceph-mon[77147]: osdmap e4: 0 total, 0 up, 0 in 2022-05-13T23:08:02.296 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:08:01 smithi047 ceph-mon[77147]: mgrmap e13: x(active, since 56s) 2022-05-13T23:08:02.296 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:08:01 smithi047 ceph-mon[77147]: Health check failed: 1/3 mons down, quorum a,b (MON_DOWN) 2022-05-13T23:08:02.297 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:08:01 smithi047 ceph-mon[77147]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,b 2022-05-13T23:08:02.297 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:08:01 smithi047 ceph-mon[77147]: [WRN] MON_DOWN: 1/3 mons down, quorum a,b 2022-05-13T23:08:02.297 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:08:01 smithi047 ceph-mon[77147]: mon.c (rank 2) addr [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] is down (out of quorum) 2022-05-13T23:08:02.297 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:08:01 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:08:02.297 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:08:01 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-13T23:08:02.298 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:08:01 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-13T23:08:02.298 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:08:01 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-13T23:08:03.066 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:08:03.064+0000 7fc5bcb82700 1 -- 172.21.15.115:0/607475921 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc5a0008cb0 msgr2=0x7fc5a000b090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:08:03.066 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:08:03.064+0000 7fc5bcb82700 1 --2- 172.21.15.115:0/607475921 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc5a0008cb0 0x7fc5a000b090 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fc5b80fad90 tx=0x7fc5a8002040 comp rx=0 tx=0).stop 2022-05-13T23:08:03.067 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:08:03.064+0000 7fc5bcb82700 1 -- 172.21.15.115:0/607475921 shutdown_connections 2022-05-13T23:08:03.067 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:08:03.064+0000 7fc5bcb82700 1 --2- 172.21.15.115:0/607475921 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc5a0005e80 0x7fc5a0006270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:08:03.067 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:08:03.064+0000 7fc5bcb82700 1 --2- 172.21.15.115:0/607475921 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc5a0008cb0 0x7fc5a000b090 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:08:03.067 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:08:03.064+0000 7fc5bcb82700 1 -- 172.21.15.115:0/607475921 >> 172.21.15.115:0/607475921 conn(0x7fc5b8070300 msgr2=0x7fc5b8070700 unknown :-1 s=STATE_NONE l=0).mark_down 2022-05-13T23:08:03.068 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:08:03.064+0000 7fc5bcb82700 1 -- 172.21.15.115:0/607475921 shutdown_connections 2022-05-13T23:08:03.068 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:08:03.064+0000 7fc5bcb82700 1 -- 172.21.15.115:0/607475921 wait complete. 2022-05-13T23:08:03.071 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:08:03.065+0000 7fc5bcb82700 1 Processor -- start 2022-05-13T23:08:03.071 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:08:03.065+0000 7fc5bcb82700 1 -- start start 2022-05-13T23:08:03.071 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:08:03.065+0000 7fc5bcb82700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc5a0005e80 0x7fc5b8110950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:08:03.071 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:08:03.065+0000 7fc5bcb82700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fc5a0008cb0 0x7fc5b8110e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:08:03.072 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:08:03.065+0000 7fc5bcb82700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc5b810aa20 0x7fc5b810ae90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:08:03.072 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:08:03.065+0000 7fc5bcb82700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5b810b460 con 0x7fc5a0005e80 2022-05-13T23:08:03.072 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:08:03.065+0000 7fc5bcb82700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5b810b5a0 con 0x7fc5b810aa20 2022-05-13T23:08:03.072 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:08:03.065+0000 7fc5bcb82700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5b810b760 con 0x7fc5a0008cb0 2022-05-13T23:08:03.073 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:08:03.065+0000 7fc5b5d9b700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fc5a0008cb0 0x7fc5b8110e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:08:03.073 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:08:03.065+0000 7fc5b5d9b700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fc5a0008cb0 0x7fc5b8110e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:48860/0 (socket says 172.21.15.115:48860) 2022-05-13T23:08:03.074 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:08:03.065+0000 7fc5b5d9b700 1 -- 172.21.15.115:0/1107976753 learned_addr learned my addr 172.21.15.115:0/1107976753 (peer_addr_for_me v2:172.21.15.115:0/0) 2022-05-13T23:08:03.074 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:08:03.065+0000 7fc5b6d9d700 1 --2- 172.21.15.115:0/1107976753 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc5b810aa20 0x7fc5b810ae90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:08:03.074 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:08:03.065+0000 7fc5b5d9b700 1 -- 172.21.15.115:0/1107976753 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fc5a0008cb0 msgr2=0x7fc5b8110e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2022-05-13T23:08:03.074 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:08:03.065+0000 7fc5b5d9b700 1 -- 172.21.15.115:0/1107976753 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fc5a0008cb0 msgr2=0x7fc5b8110e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2022-05-13T23:08:03.075 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:08:03.065+0000 7fc5b5d9b700 1 --2- 172.21.15.115:0/1107976753 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fc5a0008cb0 0x7fc5b8110e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2022-05-13T23:08:03.075 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:08:03.065+0000 7fc5b5d9b700 1 --2- 172.21.15.115:0/1107976753 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fc5a0008cb0 0x7fc5b8110e90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2022-05-13T23:08:03.075 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:08:03.065+0000 7fc5b659c700 1 --2- 172.21.15.115:0/1107976753 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc5a0005e80 0x7fc5b8110950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:08:03.075 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:08:03.066+0000 7fc5b6d9d700 1 -- 172.21.15.115:0/1107976753 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fc5a0008cb0 msgr2=0x7fc5b8110e90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2022-05-13T23:08:03.076 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:08:03.066+0000 7fc5b6d9d700 1 --2- 172.21.15.115:0/1107976753 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fc5a0008cb0 0x7fc5b8110e90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:08:03.076 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:08:03.066+0000 7fc5b6d9d700 1 -- 172.21.15.115:0/1107976753 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc5a0005e80 msgr2=0x7fc5b8110950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:08:03.076 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:08:03.066+0000 7fc5b6d9d700 1 --2- 172.21.15.115:0/1107976753 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc5a0005e80 0x7fc5b8110950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:08:03.076 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:08:03.066+0000 7fc5b6d9d700 1 -- 172.21.15.115:0/1107976753 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5a8005040 con 0x7fc5b810aa20 2022-05-13T23:08:03.076 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:08:03.066+0000 7fc5b6d9d700 1 --2- 172.21.15.115:0/1107976753 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc5b810aa20 0x7fc5b810ae90 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fc5b000bad0 tx=0x7fc5b0009760 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-05-13T23:08:03.077 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:08:03.066+0000 7fc5a6ffd700 1 -- 172.21.15.115:0/1107976753 <== mon.1 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5b000c840 con 0x7fc5b810aa20 2022-05-13T23:08:03.077 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:08:03.066+0000 7fc5a6ffd700 1 -- 172.21.15.115:0/1107976753 <== mon.1 v2:172.21.15.47:3300/0 2 ==== config(26 keys) v1 ==== 1087+0+0 (secure 0 0 0) 0x7fc5b00098a0 con 0x7fc5b810aa20 2022-05-13T23:08:03.077 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:08:03.066+0000 7fc5bcb82700 1 -- 172.21.15.115:0/1107976753 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc5b81177b0 con 0x7fc5b810aa20 2022-05-13T23:08:03.077 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:08:03.066+0000 7fc5a6ffd700 1 -- 172.21.15.115:0/1107976753 <== mon.1 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5b00133c0 con 0x7fc5b810aa20 2022-05-13T23:08:03.078 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:08:03.066+0000 7fc5bcb82700 1 -- 172.21.15.115:0/1107976753 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc5b8117cf0 con 0x7fc5b810aa20 2022-05-13T23:08:03.078 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:08:03.067+0000 7fc5a6ffd700 1 -- 172.21.15.115:0/1107976753 <== mon.1 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 13) v1 ==== 41898+0+0 (secure 0 0 0) 0x7fc5b00137f0 con 0x7fc5b810aa20 2022-05-13T23:08:03.078 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:08:03.067+0000 7fc5bcb82700 1 -- 172.21.15.115:0/1107976753 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc5b804ac70 con 0x7fc5b810aa20 2022-05-13T23:08:03.078 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:08:03.067+0000 7fc5a6ffd700 1 --2- 172.21.15.115:0/1107976753 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7fc598034fc0 0x7fc598037470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:08:03.078 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:08:03.067+0000 7fc5a6ffd700 1 -- 172.21.15.115:0/1107976753 <== mon.1 v2:172.21.15.47:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7fc5b00465c0 con 0x7fc5b810aa20 2022-05-13T23:08:03.079 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:08:03.068+0000 7fc5b659c700 1 --2- 172.21.15.115:0/1107976753 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7fc598034fc0 0x7fc598037470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:08:03.079 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:08:03.068+0000 7fc5b659c700 1 --2- 172.21.15.115:0/1107976753 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7fc598034fc0 0x7fc598037470 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fc5a8000ea0 tx=0x7fc5a8002040 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-05-13T23:08:03.080 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:08:02 smithi002 ceph-mon[71816]: Updating smithi002:/etc/ceph/ceph.conf 2022-05-13T23:08:03.080 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:08:02 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:08:03.080 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:08:02 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-05-13T23:08:03.080 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:08:03.072+0000 7fc5a6ffd700 1 -- 172.21.15.115:0/1107976753 <== mon.1 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+165215 (secure 0 0 0) 0x7fc5b0001900 con 0x7fc5b810aa20 2022-05-13T23:08:03.106 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:08:02 smithi047 ceph-mon[77147]: Updating smithi002:/etc/ceph/ceph.conf 2022-05-13T23:08:03.106 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:08:02 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:08:03.107 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:08:02 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-05-13T23:08:03.329 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:08:03.327+0000 7fc5bcb82700 1 -- 172.21.15.115:0/1107976753 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fc5b8117fd0 con 0x7fc5b810aa20 2022-05-13T23:08:03.329 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:08:03.327+0000 7fc5a6ffd700 1 -- 172.21.15.115:0/1107976753 <== mon.1 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 3 v3) v1 ==== 95+0+1240 (secure 0 0 0) 0x7fc5b0001310 con 0x7fc5b810aa20 2022-05-13T23:08:03.330 INFO:teuthology.orchestra.run.smithi115.stdout: 2022-05-13T23:08:03.330 INFO:teuthology.orchestra.run.smithi115.stdout:{"epoch":3,"fsid":"16cf5f88-d311-11ec-be40-001a4aab830c","modified":"2022-05-13T23:07:56.751402Z","created":"2022-05-13T23:05:56.093353Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:3300","nonce":0},{"type":"v1","addr":"172.21.15.2:6789","nonce":0}]},"addr":"172.21.15.2:6789/0","public_addr":"172.21.15.2:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:3300","nonce":0},{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:3300","nonce":0},{"type":"v1","addr":"172.21.15.115:6789","nonce":0}]},"addr":"172.21.15.115:6789/0","public_addr":"172.21.15.115:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2022-05-13T23:08:03.331 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:08:03.329+0000 7fc5bcb82700 1 -- 172.21.15.115:0/1107976753 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7fc598034fc0 msgr2=0x7fc598037470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:08:03.331 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:08:03.329+0000 7fc5bcb82700 1 --2- 172.21.15.115:0/1107976753 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7fc598034fc0 0x7fc598037470 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fc5a8000ea0 tx=0x7fc5a8002040 comp rx=0 tx=0).stop 2022-05-13T23:08:03.332 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:08:03.329+0000 7fc5bcb82700 1 -- 172.21.15.115:0/1107976753 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc5b810aa20 msgr2=0x7fc5b810ae90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:08:03.332 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:08:03.329+0000 7fc5bcb82700 1 --2- 172.21.15.115:0/1107976753 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc5b810aa20 0x7fc5b810ae90 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fc5b000bad0 tx=0x7fc5b0009760 comp rx=0 tx=0).stop 2022-05-13T23:08:03.332 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:08:03.329+0000 7fc5bcb82700 1 -- 172.21.15.115:0/1107976753 shutdown_connections 2022-05-13T23:08:03.332 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:08:03.329+0000 7fc5bcb82700 1 --2- 172.21.15.115:0/1107976753 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc5b810aa20 0x7fc5b810ae90 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:08:03.333 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:08:03.329+0000 7fc5bcb82700 1 --2- 172.21.15.115:0/1107976753 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7fc598034fc0 0x7fc598037470 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:08:03.333 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:08:03.329+0000 7fc5bcb82700 1 --2- 172.21.15.115:0/1107976753 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc5a0005e80 0x7fc5b8110950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:08:03.333 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:08:03.329+0000 7fc5bcb82700 1 --2- 172.21.15.115:0/1107976753 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fc5a0008cb0 0x7fc5b8110e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:08:03.334 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:08:03.329+0000 7fc5bcb82700 1 -- 172.21.15.115:0/1107976753 >> 172.21.15.115:0/1107976753 conn(0x7fc5b8070300 msgr2=0x7fc5b80fe7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-05-13T23:08:03.334 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:08:03.329+0000 7fc5bcb82700 1 -- 172.21.15.115:0/1107976753 shutdown_connections 2022-05-13T23:08:03.334 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:08:03.329+0000 7fc5bcb82700 1 -- 172.21.15.115:0/1107976753 wait complete. 2022-05-13T23:08:03.335 INFO:teuthology.orchestra.run.smithi115.stderr:dumped monmap epoch 3 2022-05-13T23:08:03.868 INFO:tasks.cephadm:Generating final ceph.conf file... 2022-05-13T23:08:03.869 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:23b0658391ce94ca40799469eec00c28b298816c shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 16cf5f88-d311-11ec-be40-001a4aab830c -- ceph config generate-minimal-conf 2022-05-13T23:08:04.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:08:03 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-05-13T23:08:04.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:08:03 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-05-13T23:08:04.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:08:03 smithi115 ceph-mon[78593]: mon.b calling monitor election 2022-05-13T23:08:04.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:08:03 smithi115 ceph-mon[78593]: mon.a calling monitor election 2022-05-13T23:08:04.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:08:03 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-05-13T23:08:04.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:08:03 smithi115 ceph-mon[78593]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-05-13T23:08:04.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:08:03 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-05-13T23:08:04.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:08:03 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-05-13T23:08:04.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:08:03 smithi115 ceph-mon[78593]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-05-13T23:08:04.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:08:03 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-05-13T23:08:04.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:08:03 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-05-13T23:08:04.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:08:03 smithi115 ceph-mon[78593]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-05-13T23:08:04.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:08:03 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-05-13T23:08:04.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:08:03 smithi115 ceph-mon[78593]: mon.a is new leader, mons a,b in quorum (ranks 0,1) 2022-05-13T23:08:04.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:08:03 smithi115 ceph-mon[78593]: monmap e3: 3 mons at {a=[v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0],b=[v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0],c=[v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0]} 2022-05-13T23:08:04.112 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:08:03 smithi115 ceph-mon[78593]: fsmap 2022-05-13T23:08:04.112 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:08:03 smithi115 ceph-mon[78593]: osdmap e4: 0 total, 0 up, 0 in 2022-05-13T23:08:04.112 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:08:03 smithi115 ceph-mon[78593]: mgrmap e13: x(active, since 56s) 2022-05-13T23:08:04.112 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:08:03 smithi115 ceph-mon[78593]: Health check failed: 1/3 mons down, quorum a,b (MON_DOWN) 2022-05-13T23:08:04.112 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:08:03 smithi115 ceph-mon[78593]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,b 2022-05-13T23:08:04.113 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:08:03 smithi115 ceph-mon[78593]: [WRN] MON_DOWN: 1/3 mons down, quorum a,b 2022-05-13T23:08:04.113 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:08:03 smithi115 ceph-mon[78593]: mon.c (rank 2) addr [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] is down (out of quorum) 2022-05-13T23:08:04.113 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:08:03 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:08:04.113 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:08:03 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-13T23:08:04.114 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:08:03 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-13T23:08:04.114 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:08:03 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-13T23:08:04.114 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:08:03 smithi115 ceph-mon[78593]: Updating smithi002:/etc/ceph/ceph.conf 2022-05-13T23:08:04.114 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:08:03 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:08:04.114 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:08:03 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-05-13T23:08:04.734 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:08:04 smithi002 ceph-mon[71816]: mon.c calling monitor election 2022-05-13T23:08:04.734 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:08:04 smithi002 ceph-mon[71816]: Reconfiguring mon.a (unknown last config time)... 2022-05-13T23:08:04.734 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:08:04 smithi002 ceph-mon[71816]: Reconfiguring daemon mon.a on smithi002 2022-05-13T23:08:04.734 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:08:04 smithi002 ceph-mon[71816]: mon.c calling monitor election 2022-05-13T23:08:04.735 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:08:04 smithi002 ceph-mon[71816]: mon.b calling monitor election 2022-05-13T23:08:04.735 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:08:04 smithi002 ceph-mon[71816]: mon.a calling monitor election 2022-05-13T23:08:04.735 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:08:04 smithi002 ceph-mon[71816]: mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2022-05-13T23:08:04.735 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:08:04 smithi002 ceph-mon[71816]: monmap e3: 3 mons at {a=[v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0],b=[v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0],c=[v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0]} 2022-05-13T23:08:04.736 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:08:04 smithi002 ceph-mon[71816]: fsmap 2022-05-13T23:08:04.736 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:08:04 smithi002 ceph-mon[71816]: osdmap e4: 0 total, 0 up, 0 in 2022-05-13T23:08:04.736 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:08:04 smithi002 ceph-mon[71816]: mgrmap e13: x(active, since 58s) 2022-05-13T23:08:04.736 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:08:04 smithi002 ceph-mon[71816]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,b) 2022-05-13T23:08:04.736 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:08:04 smithi002 ceph-mon[71816]: Cluster is now healthy 2022-05-13T23:08:04.737 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:08:04 smithi002 ceph-mon[71816]: overall HEALTH_OK 2022-05-13T23:08:04.788 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/16cf5f88-d311-11ec-be40-001a4aab830c/mon.a/config 2022-05-13T23:08:04.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:08:04 smithi047 ceph-mon[77147]: mon.c calling monitor election 2022-05-13T23:08:04.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:08:04 smithi047 ceph-mon[77147]: Reconfiguring mon.a (unknown last config time)... 2022-05-13T23:08:04.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:08:04 smithi047 ceph-mon[77147]: Reconfiguring daemon mon.a on smithi002 2022-05-13T23:08:04.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:08:04 smithi047 ceph-mon[77147]: mon.c calling monitor election 2022-05-13T23:08:04.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:08:04 smithi047 ceph-mon[77147]: mon.b calling monitor election 2022-05-13T23:08:04.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:08:04 smithi047 ceph-mon[77147]: mon.a calling monitor election 2022-05-13T23:08:04.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:08:04 smithi047 ceph-mon[77147]: mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2022-05-13T23:08:04.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:08:04 smithi047 ceph-mon[77147]: monmap e3: 3 mons at {a=[v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0],b=[v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0],c=[v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0]} 2022-05-13T23:08:04.795 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:08:04 smithi047 ceph-mon[77147]: fsmap 2022-05-13T23:08:04.795 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:08:04 smithi047 ceph-mon[77147]: osdmap e4: 0 total, 0 up, 0 in 2022-05-13T23:08:04.796 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:08:04 smithi047 ceph-mon[77147]: mgrmap e13: x(active, since 58s) 2022-05-13T23:08:04.797 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:08:04 smithi047 ceph-mon[77147]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,b) 2022-05-13T23:08:04.797 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:08:04 smithi047 ceph-mon[77147]: Cluster is now healthy 2022-05-13T23:08:04.798 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:08:04 smithi047 ceph-mon[77147]: overall HEALTH_OK 2022-05-13T23:08:04.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:08:04 smithi115 ceph-mon[78593]: mon.c calling monitor election 2022-05-13T23:08:04.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:08:04 smithi115 ceph-mon[78593]: Reconfiguring mon.a (unknown last config time)... 2022-05-13T23:08:04.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:08:04 smithi115 ceph-mon[78593]: Reconfiguring daemon mon.a on smithi002 2022-05-13T23:08:04.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:08:04 smithi115 ceph-mon[78593]: mon.c calling monitor election 2022-05-13T23:08:04.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:08:04 smithi115 ceph-mon[78593]: mon.b calling monitor election 2022-05-13T23:08:04.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:08:04 smithi115 ceph-mon[78593]: mon.a calling monitor election 2022-05-13T23:08:04.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:08:04 smithi115 ceph-mon[78593]: mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2022-05-13T23:08:04.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:08:04 smithi115 ceph-mon[78593]: monmap e3: 3 mons at {a=[v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0],b=[v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0],c=[v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0]} 2022-05-13T23:08:04.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:08:04 smithi115 ceph-mon[78593]: fsmap 2022-05-13T23:08:04.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:08:04 smithi115 ceph-mon[78593]: osdmap e4: 0 total, 0 up, 0 in 2022-05-13T23:08:04.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:08:04 smithi115 ceph-mon[78593]: mgrmap e13: x(active, since 58s) 2022-05-13T23:08:04.862 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:08:04 smithi115 ceph-mon[78593]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,b) 2022-05-13T23:08:04.862 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:08:04 smithi115 ceph-mon[78593]: Cluster is now healthy 2022-05-13T23:08:04.862 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:08:04 smithi115 ceph-mon[78593]: overall HEALTH_OK 2022-05-13T23:08:05.640 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:08:05 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-05-13T23:08:05.640 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:08:05 smithi002 ceph-mon[71816]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-05-13T23:08:05.641 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:08:05 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:08:05.641 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:08:05 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-05-13T23:08:05.641 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:08:05 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-05-13T23:08:05.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:08:05 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-05-13T23:08:05.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:08:05 smithi047 ceph-mon[77147]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-05-13T23:08:05.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:08:05 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:08:05.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:08:05 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-05-13T23:08:05.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:08:05 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-05-13T23:08:05.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:08:05 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-05-13T23:08:05.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:08:05 smithi115 ceph-mon[78593]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-05-13T23:08:05.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:08:05 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:08:05.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:08:05 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-05-13T23:08:05.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:08:05 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-05-13T23:08:06.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:08:06 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:08:06.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:08:06 smithi047 ceph-mon[77147]: Reconfiguring mon.b (monmap changed)... 2022-05-13T23:08:06.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:08:06 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2022-05-13T23:08:06.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:08:06 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2022-05-13T23:08:06.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:08:06 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-13T23:08:06.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:08:06 smithi047 ceph-mon[77147]: Reconfiguring daemon mon.b on smithi047 2022-05-13T23:08:06.822 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:08:06 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:08:06.823 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:08:06 smithi002 ceph-mon[71816]: Reconfiguring mon.b (monmap changed)... 2022-05-13T23:08:06.823 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:08:06 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2022-05-13T23:08:06.823 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:08:06 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2022-05-13T23:08:06.824 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:08:06 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-13T23:08:06.824 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:08:06 smithi002 ceph-mon[71816]: Reconfiguring daemon mon.b on smithi047 2022-05-13T23:08:06.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:08:06 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:08:06.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:08:06 smithi115 ceph-mon[78593]: Reconfiguring mon.b (monmap changed)... 2022-05-13T23:08:06.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:08:06 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2022-05-13T23:08:06.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:08:06 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2022-05-13T23:08:06.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:08:06 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-13T23:08:06.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:08:06 smithi115 ceph-mon[78593]: Reconfiguring daemon mon.b on smithi047 2022-05-13T23:08:06.888 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:08:06.886+0000 7f719c9e9700 1 -- 172.21.15.2:0/2607971358 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7198101510 msgr2=0x7f71981018e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:08:06.888 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:08:06.886+0000 7f719c9e9700 1 --2- 172.21.15.2:0/2607971358 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7198101510 0x7f71981018e0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f71880047b0 tx=0x7f7188009cb0 comp rx=0 tx=0).stop 2022-05-13T23:08:06.888 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:08:06.886+0000 7f719c9e9700 1 -- 172.21.15.2:0/2607971358 shutdown_connections 2022-05-13T23:08:06.889 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:08:06.886+0000 7f719c9e9700 1 --2- 172.21.15.2:0/2607971358 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f71980fbc80 0x7f71981082d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:08:06.890 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:08:06.886+0000 7f719c9e9700 1 --2- 172.21.15.2:0/2607971358 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7198101510 0x7f71981018e0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:08:06.890 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:08:06.886+0000 7f719c9e9700 1 --2- 172.21.15.2:0/2607971358 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f7198101e20 0x7f71980fb550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:08:06.891 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:08:06.886+0000 7f719c9e9700 1 -- 172.21.15.2:0/2607971358 >> 172.21.15.2:0/2607971358 conn(0x7f71980f7090 msgr2=0x7f71980f94a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-05-13T23:08:06.891 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:08:06.886+0000 7f719c9e9700 1 -- 172.21.15.2:0/2607971358 shutdown_connections 2022-05-13T23:08:06.891 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:08:06.886+0000 7f719c9e9700 1 -- 172.21.15.2:0/2607971358 wait complete. 2022-05-13T23:08:06.894 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:08:06.887+0000 7f719c9e9700 1 Processor -- start 2022-05-13T23:08:06.895 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:08:06.887+0000 7f719c9e9700 1 -- start start 2022-05-13T23:08:06.895 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:08:06.888+0000 7f719c9e9700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f71980fbc80 0x7f719806e7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:08:06.895 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:08:06.888+0000 7f719c9e9700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7198101510 0x7f719806ed20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:08:06.896 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:08:06.888+0000 7f719c9e9700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f7198101e20 0x7f719806f260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:08:06.896 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:08:06.888+0000 7f719c9e9700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f719811a860 con 0x7f7198101510 2022-05-13T23:08:06.896 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:08:06.888+0000 7f719c9e9700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f719811a9d0 con 0x7f71980fbc80 2022-05-13T23:08:06.896 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:08:06.888+0000 7f719c9e9700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f719811ab90 con 0x7f7198101e20 2022-05-13T23:08:06.896 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:08:06.888+0000 7f7196d9d700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f71980fbc80 0x7f719806e7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:08:06.897 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:08:06.888+0000 7f719759e700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f7198101e20 0x7f719806f260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:08:06.897 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:08:06.888+0000 7f7196d9d700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f71980fbc80 0x7f719806e7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.2:44270/0 (socket says 172.21.15.2:44270) 2022-05-13T23:08:06.897 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:08:06.888+0000 7f7196d9d700 1 -- 172.21.15.2:0/1987173581 learned_addr learned my addr 172.21.15.2:0/1987173581 (peer_addr_for_me v2:172.21.15.2:0/0) 2022-05-13T23:08:06.897 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:08:06.888+0000 7f719659c700 1 --2- 172.21.15.2:0/1987173581 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7198101510 0x7f719806ed20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:08:06.898 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:08:06.889+0000 7f7196d9d700 1 -- 172.21.15.2:0/1987173581 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f7198101e20 msgr2=0x7f719806f260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:08:06.898 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:08:06.889+0000 7f7196d9d700 1 --2- 172.21.15.2:0/1987173581 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f7198101e20 0x7f719806f260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:08:06.898 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:08:06.889+0000 7f7196d9d700 1 -- 172.21.15.2:0/1987173581 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7198101510 msgr2=0x7f719806ed20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:08:06.898 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:08:06.889+0000 7f7196d9d700 1 --2- 172.21.15.2:0/1987173581 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7198101510 0x7f719806ed20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:08:06.898 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:08:06.889+0000 7f7196d9d700 1 -- 172.21.15.2:0/1987173581 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7188005040 con 0x7f71980fbc80 2022-05-13T23:08:06.899 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:08:06.889+0000 7f719759e700 1 --2- 172.21.15.2:0/1987173581 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f7198101e20 0x7f719806f260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-05-13T23:08:06.899 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:08:06.889+0000 7f7196d9d700 1 --2- 172.21.15.2:0/1987173581 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f71980fbc80 0x7f719806e7e0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f7188005010 tx=0x7f7188009cb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-05-13T23:08:06.899 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:08:06.889+0000 7f717f7fe700 1 -- 172.21.15.2:0/1987173581 <== mon.1 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7188008d80 con 0x7f71980fbc80 2022-05-13T23:08:06.899 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:08:06.889+0000 7f719c9e9700 1 -- 172.21.15.2:0/1987173581 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f719811ae10 con 0x7f71980fbc80 2022-05-13T23:08:06.900 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:08:06.889+0000 7f717f7fe700 1 -- 172.21.15.2:0/1987173581 <== mon.1 v2:172.21.15.47:3300/0 2 ==== config(26 keys) v1 ==== 1087+0+0 (secure 0 0 0) 0x7f7188008ee0 con 0x7f71980fbc80 2022-05-13T23:08:06.900 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:08:06.889+0000 7f717f7fe700 1 -- 172.21.15.2:0/1987173581 <== mon.1 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f718801cbd0 con 0x7f71980fbc80 2022-05-13T23:08:06.900 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:08:06.889+0000 7f719c9e9700 1 -- 172.21.15.2:0/1987173581 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f719811b240 con 0x7f71980fbc80 2022-05-13T23:08:06.900 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:08:06.890+0000 7f717f7fe700 1 -- 172.21.15.2:0/1987173581 <== mon.1 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 13) v1 ==== 41898+0+0 (secure 0 0 0) 0x7f7188009240 con 0x7f71980fbc80 2022-05-13T23:08:06.900 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:08:06.890+0000 7f719c9e9700 1 -- 172.21.15.2:0/1987173581 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f719804ac70 con 0x7f71980fbc80 2022-05-13T23:08:06.901 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:08:06.891+0000 7f717f7fe700 1 --2- 172.21.15.2:0/1987173581 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f7180034e90 0x7f7180037340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:08:06.901 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:08:06.891+0000 7f717f7fe700 1 -- 172.21.15.2:0/1987173581 <== mon.1 v2:172.21.15.47:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f71880476c0 con 0x7f71980fbc80 2022-05-13T23:08:06.901 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:08:06.891+0000 7f719659c700 1 --2- 172.21.15.2:0/1987173581 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f7180034e90 0x7f7180037340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:08:06.901 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:08:06.892+0000 7f719659c700 1 --2- 172.21.15.2:0/1987173581 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f7180034e90 0x7f7180037340 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f719806ffd0 tx=0x7f718c00b040 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-05-13T23:08:06.902 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:08:06.896+0000 7f717f7fe700 1 -- 172.21.15.2:0/1987173581 <== mon.1 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+165215 (secure 0 0 0) 0x7f7188011550 con 0x7f71980fbc80 2022-05-13T23:08:07.090 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:08:07.089+0000 7f719c9e9700 1 -- 172.21.15.2:0/1987173581 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) v1 -- 0x7f719811b430 con 0x7f71980fbc80 2022-05-13T23:08:07.091 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:08:07.090+0000 7f717f7fe700 1 -- 172.21.15.2:0/1987173581 <== mon.1 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v9) v1 ==== 76+0+271 (secure 0 0 0) 0x7f7188011370 con 0x7f71980fbc80 2022-05-13T23:08:07.092 INFO:teuthology.orchestra.run.smithi002.stdout:# minimal ceph.conf for 16cf5f88-d311-11ec-be40-001a4aab830c 2022-05-13T23:08:07.092 INFO:teuthology.orchestra.run.smithi002.stdout:[global] 2022-05-13T23:08:07.092 INFO:teuthology.orchestra.run.smithi002.stdout: fsid = 16cf5f88-d311-11ec-be40-001a4aab830c 2022-05-13T23:08:07.093 INFO:teuthology.orchestra.run.smithi002.stdout: mon_host = [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] 2022-05-13T23:08:07.095 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:08:07.093+0000 7f719c9e9700 1 -- 172.21.15.2:0/1987173581 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f7180034e90 msgr2=0x7f7180037340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:08:07.095 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:08:07.093+0000 7f719c9e9700 1 --2- 172.21.15.2:0/1987173581 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f7180034e90 0x7f7180037340 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f719806ffd0 tx=0x7f718c00b040 comp rx=0 tx=0).stop 2022-05-13T23:08:07.095 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:08:07.093+0000 7f719c9e9700 1 -- 172.21.15.2:0/1987173581 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f71980fbc80 msgr2=0x7f719806e7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:08:07.096 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:08:07.093+0000 7f719c9e9700 1 --2- 172.21.15.2:0/1987173581 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f71980fbc80 0x7f719806e7e0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f7188005010 tx=0x7f7188009cb0 comp rx=0 tx=0).stop 2022-05-13T23:08:07.096 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:08:07.093+0000 7f719c9e9700 1 -- 172.21.15.2:0/1987173581 shutdown_connections 2022-05-13T23:08:07.096 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:08:07.093+0000 7f719c9e9700 1 --2- 172.21.15.2:0/1987173581 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f7198101e20 0x7f719806f260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:08:07.096 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:08:07.093+0000 7f719c9e9700 1 --2- 172.21.15.2:0/1987173581 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f71980fbc80 0x7f719806e7e0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:08:07.097 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:08:07.093+0000 7f719c9e9700 1 --2- 172.21.15.2:0/1987173581 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f7180034e90 0x7f7180037340 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:08:07.097 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:08:07.093+0000 7f719c9e9700 1 --2- 172.21.15.2:0/1987173581 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7198101510 0x7f719806ed20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:08:07.097 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:08:07.093+0000 7f719c9e9700 1 -- 172.21.15.2:0/1987173581 >> 172.21.15.2:0/1987173581 conn(0x7f71980f7090 msgr2=0x7f71980ff220 unknown :-1 s=STATE_NONE l=0).mark_down 2022-05-13T23:08:07.097 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:08:07.093+0000 7f719c9e9700 1 -- 172.21.15.2:0/1987173581 shutdown_connections 2022-05-13T23:08:07.098 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:08:07.093+0000 7f719c9e9700 1 -- 172.21.15.2:0/1987173581 wait complete. 2022-05-13T23:08:07.660 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2022-05-13T23:08:07.661 DEBUG:teuthology.orchestra.run.smithi002:> set -ex 2022-05-13T23:08:07.661 DEBUG:teuthology.orchestra.run.smithi002:> sudo dd of=/etc/ceph/ceph.conf 2022-05-13T23:08:07.757 DEBUG:teuthology.orchestra.run.smithi002:> set -ex 2022-05-13T23:08:07.757 DEBUG:teuthology.orchestra.run.smithi002:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2022-05-13T23:08:07.876 DEBUG:teuthology.orchestra.run.smithi047:> set -ex 2022-05-13T23:08:07.876 DEBUG:teuthology.orchestra.run.smithi047:> sudo dd of=/etc/ceph/ceph.conf 2022-05-13T23:08:07.910 DEBUG:teuthology.orchestra.run.smithi047:> set -ex 2022-05-13T23:08:07.910 DEBUG:teuthology.orchestra.run.smithi047:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2022-05-13T23:08:07.978 DEBUG:teuthology.orchestra.run.smithi115:> set -ex 2022-05-13T23:08:07.979 DEBUG:teuthology.orchestra.run.smithi115:> sudo dd of=/etc/ceph/ceph.conf 2022-05-13T23:08:08.005 DEBUG:teuthology.orchestra.run.smithi115:> set -ex 2022-05-13T23:08:08.006 DEBUG:teuthology.orchestra.run.smithi115:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2022-05-13T23:08:08.070 INFO:tasks.cephadm:Adding mgr.x on smithi002 2022-05-13T23:08:08.070 INFO:tasks.cephadm:Adding mgr.y on smithi047 2022-05-13T23:08:08.071 INFO:tasks.cephadm:Adding mgr.z on smithi115 2022-05-13T23:08:08.071 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:23b0658391ce94ca40799469eec00c28b298816c shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 16cf5f88-d311-11ec-be40-001a4aab830c -- ceph orch apply mgr '3;smithi002=x;smithi047=y;smithi115=z' 2022-05-13T23:08:08.286 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:08:07 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:08:08.287 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:08:07 smithi115 ceph-mon[78593]: Reconfiguring mon.c (monmap changed)... 2022-05-13T23:08:08.287 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:08:07 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2022-05-13T23:08:08.287 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:08:07 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2022-05-13T23:08:08.287 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:08:07 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-13T23:08:08.288 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:08:07 smithi115 ceph-mon[78593]: Reconfiguring daemon mon.c on smithi115 2022-05-13T23:08:08.288 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:08:07 smithi115 ceph-mon[78593]: from='client.? 172.21.15.2:0/1987173581' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-13T23:08:08.288 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:08:07 smithi115 ceph-mon[78593]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-05-13T23:08:08.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:08:07 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:08:08.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:08:07 smithi047 ceph-mon[77147]: Reconfiguring mon.c (monmap changed)... 2022-05-13T23:08:08.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:08:07 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2022-05-13T23:08:08.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:08:07 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2022-05-13T23:08:08.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:08:07 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-13T23:08:08.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:08:07 smithi047 ceph-mon[77147]: Reconfiguring daemon mon.c on smithi115 2022-05-13T23:08:08.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:08:07 smithi047 ceph-mon[77147]: from='client.? 172.21.15.2:0/1987173581' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-13T23:08:08.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:08:07 smithi047 ceph-mon[77147]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-05-13T23:08:08.322 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:08:07 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:08:08.323 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:08:07 smithi002 ceph-mon[71816]: Reconfiguring mon.c (monmap changed)... 2022-05-13T23:08:08.323 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:08:07 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2022-05-13T23:08:08.323 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:08:07 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2022-05-13T23:08:08.323 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:08:07 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-13T23:08:08.324 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:08:07 smithi002 ceph-mon[71816]: Reconfiguring daemon mon.c on smithi115 2022-05-13T23:08:08.324 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:08:07 smithi002 ceph-mon[71816]: from='client.? 172.21.15.2:0/1987173581' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-13T23:08:08.324 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:08:07 smithi002 ceph-mon[71816]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-05-13T23:08:08.549 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/16cf5f88-d311-11ec-be40-001a4aab830c/mon.c/config 2022-05-13T23:08:09.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:08:09 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:08:09.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:08:09 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-13T23:08:09.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:08:09 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-13T23:08:09.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:08:09 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-13T23:08:09.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:08:09 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:08:09.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:08:09 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:08:09.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:08:09 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-13T23:08:09.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:08:09 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-13T23:08:09.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:08:09 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-13T23:08:09.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:08:09 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:08:09.822 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:08:09 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:08:09.822 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:08:09 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-13T23:08:09.823 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:08:09 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-13T23:08:09.823 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:08:09 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-13T23:08:09.823 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:08:09 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:08:09.830 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:08:09.824+0000 7fe21975f700 1 -- 172.21.15.115:0/2695040014 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe2140fb600 msgr2=0x7fe2140fba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:08:09.830 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:08:09.824+0000 7fe21975f700 1 --2- 172.21.15.115:0/2695040014 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe2140fb600 0x7fe2140fba10 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fe204004870 tx=0x7fe204009cb0 comp rx=0 tx=0).stop 2022-05-13T23:08:09.832 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:08:09.824+0000 7fe21975f700 1 -- 172.21.15.115:0/2695040014 shutdown_connections 2022-05-13T23:08:09.833 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:08:09.824+0000 7fe21975f700 1 --2- 172.21.15.115:0/2695040014 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe2140fd230 0x7fe214103af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:08:09.833 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:08:09.824+0000 7fe21975f700 1 --2- 172.21.15.115:0/2695040014 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe2140fb600 0x7fe2140fba10 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:08:09.834 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:08:09.824+0000 7fe21975f700 1 --2- 172.21.15.115:0/2695040014 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe2140fc8a0 0x7fe2140fccf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:08:09.834 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:08:09.824+0000 7fe21975f700 1 -- 172.21.15.115:0/2695040014 >> 172.21.15.115:0/2695040014 conn(0x7fe2140f6df0 msgr2=0x7fe2140f9220 unknown :-1 s=STATE_NONE l=0).mark_down 2022-05-13T23:08:09.835 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:08:09.824+0000 7fe21975f700 1 -- 172.21.15.115:0/2695040014 shutdown_connections 2022-05-13T23:08:09.835 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:08:09.824+0000 7fe21975f700 1 -- 172.21.15.115:0/2695040014 wait complete. 2022-05-13T23:08:09.835 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:08:09.825+0000 7fe21975f700 1 Processor -- start 2022-05-13T23:08:09.835 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:08:09.825+0000 7fe21975f700 1 -- start start 2022-05-13T23:08:09.836 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:08:09.825+0000 7fe21975f700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe2140fb600 0x7fe21410c3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:08:09.836 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:08:09.825+0000 7fe21975f700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe2140fc8a0 0x7fe21410c910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:08:09.836 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:08:09.825+0000 7fe21975f700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe2140fd230 0x7fe2141139a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:08:09.836 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:08:09.825+0000 7fe21975f700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe214114000 con 0x7fe2140fc8a0 2022-05-13T23:08:09.837 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:08:09.825+0000 7fe21975f700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe214114170 con 0x7fe2140fd230 2022-05-13T23:08:09.837 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:08:09.825+0000 7fe21975f700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe214114330 con 0x7fe2140fb600 2022-05-13T23:08:09.837 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:08:09.825+0000 7fe212ffd700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe2140fb600 0x7fe21410c3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:08:09.837 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:08:09.825+0000 7fe212ffd700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe2140fb600 0x7fe21410c3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:48878/0 (socket says 172.21.15.115:48878) 2022-05-13T23:08:09.837 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:08:09.825+0000 7fe212ffd700 1 -- 172.21.15.115:0/2954362304 learned_addr learned my addr 172.21.15.115:0/2954362304 (peer_addr_for_me v2:172.21.15.115:0/0) 2022-05-13T23:08:09.838 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:08:09.826+0000 7fe2127fc700 1 --2- 172.21.15.115:0/2954362304 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe2140fc8a0 0x7fe21410c910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:08:09.838 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:08:09.826+0000 7fe2137fe700 1 --2- 172.21.15.115:0/2954362304 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe2140fd230 0x7fe2141139a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:08:09.838 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:08:09.826+0000 7fe212ffd700 1 -- 172.21.15.115:0/2954362304 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe2140fd230 msgr2=0x7fe2141139a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:08:09.838 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:08:09.826+0000 7fe212ffd700 1 --2- 172.21.15.115:0/2954362304 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe2140fd230 0x7fe2141139a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:08:09.839 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:08:09.826+0000 7fe212ffd700 1 -- 172.21.15.115:0/2954362304 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe2140fc8a0 msgr2=0x7fe21410c910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:08:09.839 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:08:09.826+0000 7fe212ffd700 1 --2- 172.21.15.115:0/2954362304 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe2140fc8a0 0x7fe21410c910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:08:09.839 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:08:09.826+0000 7fe212ffd700 1 -- 172.21.15.115:0/2954362304 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe204005040 con 0x7fe2140fb600 2022-05-13T23:08:09.839 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:08:09.826+0000 7fe2137fe700 1 --2- 172.21.15.115:0/2954362304 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe2140fd230 0x7fe2141139a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2022-05-13T23:08:09.839 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:08:09.826+0000 7fe2127fc700 1 --2- 172.21.15.115:0/2954362304 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe2140fc8a0 0x7fe21410c910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2022-05-13T23:08:09.840 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:08:09.826+0000 7fe212ffd700 1 --2- 172.21.15.115:0/2954362304 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe2140fb600 0x7fe21410c3d0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fe204009d60 tx=0x7fe204009cb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-05-13T23:08:09.840 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:08:09.826+0000 7fe1fb7fe700 1 -- 172.21.15.115:0/2954362304 <== mon.2 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe20400e040 con 0x7fe2140fb600 2022-05-13T23:08:09.840 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:08:09.826+0000 7fe1fb7fe700 1 -- 172.21.15.115:0/2954362304 <== mon.2 v2:172.21.15.115:3300/0 2 ==== config(26 keys) v1 ==== 1087+0+0 (secure 0 0 0) 0x7fe2040096f0 con 0x7fe2140fb600 2022-05-13T23:08:09.840 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:08:09.826+0000 7fe1fb7fe700 1 -- 172.21.15.115:0/2954362304 <== mon.2 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe20401ce20 con 0x7fe2140fb600 2022-05-13T23:08:09.841 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:08:09.826+0000 7fe21975f700 1 -- 172.21.15.115:0/2954362304 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe2141145b0 con 0x7fe2140fb600 2022-05-13T23:08:09.841 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:08:09.826+0000 7fe21975f700 1 -- 172.21.15.115:0/2954362304 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe2141149b0 con 0x7fe2140fb600 2022-05-13T23:08:09.841 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:08:09.827+0000 7fe1fb7fe700 1 -- 172.21.15.115:0/2954362304 <== mon.2 v2:172.21.15.115:3300/0 4 ==== mgrmap(e 13) v1 ==== 41898+0+0 (secure 0 0 0) 0x7fe2040125d0 con 0x7fe2140fb600 2022-05-13T23:08:09.841 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:08:09.827+0000 7fe1fb7fe700 1 --2- 172.21.15.115:0/2954362304 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7fe1fc034e90 0x7fe1fc037340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:08:09.841 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:08:09.827+0000 7fe21975f700 1 -- 172.21.15.115:0/2954362304 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe214106ee0 con 0x7fe2140fb600 2022-05-13T23:08:09.842 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:08:09.827+0000 7fe1fb7fe700 1 -- 172.21.15.115:0/2954362304 <== mon.2 v2:172.21.15.115:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7fe2040478c0 con 0x7fe2140fb600 2022-05-13T23:08:09.842 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:08:09.828+0000 7fe2127fc700 1 --2- 172.21.15.115:0/2954362304 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7fe1fc034e90 0x7fe1fc037340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:08:09.842 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:08:09.828+0000 7fe2127fc700 1 --2- 172.21.15.115:0/2954362304 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7fe1fc034e90 0x7fe1fc037340 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fe20800a900 tx=0x7fe208008040 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-05-13T23:08:09.843 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:08:09.832+0000 7fe1fb7fe700 1 -- 172.21.15.115:0/2954362304 <== mon.2 v2:172.21.15.115:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+165215 (secure 0 0 0) 0x7fe204011960 con 0x7fe2140fb600 2022-05-13T23:08:10.012 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:08:10.011+0000 7fe21975f700 1 -- 172.21.15.115:0/2954362304 --> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi002=x;smithi047=y;smithi115=z", "target": ["mon-mgr", ""]}) v1 -- 0x7fe21405c920 con 0x7fe1fc034e90 2022-05-13T23:08:10.019 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:08:10.017+0000 7fe1fb7fe700 1 -- 172.21.15.115:0/2954362304 <== mgr.14148 v2:172.21.15.2:6800/2788339627 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7fe21405c920 con 0x7fe1fc034e90 2022-05-13T23:08:10.019 INFO:teuthology.orchestra.run.smithi115.stdout:Scheduled mgr update... 2022-05-13T23:08:10.021 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:08:10.019+0000 7fe21975f700 1 -- 172.21.15.115:0/2954362304 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7fe1fc034e90 msgr2=0x7fe1fc037340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:08:10.022 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:08:10.019+0000 7fe21975f700 1 --2- 172.21.15.115:0/2954362304 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7fe1fc034e90 0x7fe1fc037340 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fe20800a900 tx=0x7fe208008040 comp rx=0 tx=0).stop 2022-05-13T23:08:10.022 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:08:10.019+0000 7fe21975f700 1 -- 172.21.15.115:0/2954362304 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe2140fb600 msgr2=0x7fe21410c3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:08:10.022 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:08:10.019+0000 7fe21975f700 1 --2- 172.21.15.115:0/2954362304 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe2140fb600 0x7fe21410c3d0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fe204009d60 tx=0x7fe204009cb0 comp rx=0 tx=0).stop 2022-05-13T23:08:10.022 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:08:10.019+0000 7fe21975f700 1 -- 172.21.15.115:0/2954362304 shutdown_connections 2022-05-13T23:08:10.023 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:08:10.019+0000 7fe21975f700 1 --2- 172.21.15.115:0/2954362304 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe2140fd230 0x7fe2141139a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:08:10.023 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:08:10.019+0000 7fe21975f700 1 --2- 172.21.15.115:0/2954362304 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe2140fb600 0x7fe21410c3d0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:08:10.023 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:08:10.019+0000 7fe21975f700 1 --2- 172.21.15.115:0/2954362304 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7fe1fc034e90 0x7fe1fc037340 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:08:10.023 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:08:10.019+0000 7fe21975f700 1 --2- 172.21.15.115:0/2954362304 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe2140fc8a0 0x7fe21410c910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:08:10.023 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:08:10.019+0000 7fe21975f700 1 -- 172.21.15.115:0/2954362304 >> 172.21.15.115:0/2954362304 conn(0x7fe2140f6df0 msgr2=0x7fe2140f9140 unknown :-1 s=STATE_NONE l=0).mark_down 2022-05-13T23:08:10.024 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:08:10.020+0000 7fe21975f700 1 -- 172.21.15.115:0/2954362304 shutdown_connections 2022-05-13T23:08:10.024 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:08:10.020+0000 7fe21975f700 1 -- 172.21.15.115:0/2954362304 wait complete. 2022-05-13T23:08:10.449 DEBUG:teuthology.orchestra.run.smithi047:mgr.y> sudo journalctl -f -n 0 -u ceph-16cf5f88-d311-11ec-be40-001a4aab830c@mgr.y.service 2022-05-13T23:08:10.452 DEBUG:teuthology.orchestra.run.smithi115:mgr.z> sudo journalctl -f -n 0 -u ceph-16cf5f88-d311-11ec-be40-001a4aab830c@mgr.z.service 2022-05-13T23:08:10.454 INFO:tasks.cephadm:Deploying OSDs... 2022-05-13T23:08:10.455 DEBUG:teuthology.orchestra.run.smithi002:> set -ex 2022-05-13T23:08:10.455 DEBUG:teuthology.orchestra.run.smithi002:> dd if=/scratch_devs of=/dev/stdout 2022-05-13T23:08:10.481 INFO:journalctl@ceph.mgr.z.smithi115.stdout:-- Logs begin at Fri 2022-05-13 22:45:29 UTC. -- 2022-05-13T23:08:10.482 INFO:journalctl@ceph.mgr.y.smithi047.stdout:-- Logs begin at Fri 2022-05-13 22:44:47 UTC. -- 2022-05-13T23:08:10.505 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2022-05-13T23:08:10.505 DEBUG:teuthology.orchestra.run.smithi002:> stat /dev/vg_nvme/lv_1 && sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_1 2022-05-13T23:08:10.555 INFO:teuthology.orchestra.run.smithi002.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2022-05-13T23:08:10.555 INFO:teuthology.orchestra.run.smithi002.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-05-13T23:08:10.556 INFO:teuthology.orchestra.run.smithi002.stdout:Device: 5h/5d Inode: 454 Links: 1 2022-05-13T23:08:10.556 INFO:teuthology.orchestra.run.smithi002.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-05-13T23:08:10.557 INFO:teuthology.orchestra.run.smithi002.stdout:Context: system_u:object_r:device_t:s0 2022-05-13T23:08:10.557 INFO:teuthology.orchestra.run.smithi002.stdout:Access: 2022-05-13 23:07:17.011389974 +0000 2022-05-13T23:08:10.557 INFO:teuthology.orchestra.run.smithi002.stdout:Modify: 2022-05-13 23:00:08.112344619 +0000 2022-05-13T23:08:10.558 INFO:teuthology.orchestra.run.smithi002.stdout:Change: 2022-05-13 23:00:08.112344619 +0000 2022-05-13T23:08:10.558 INFO:teuthology.orchestra.run.smithi002.stdout: Birth: - 2022-05-13T23:08:10.586 INFO:teuthology.orchestra.run.smithi002.stderr:1+0 records in 2022-05-13T23:08:10.586 INFO:teuthology.orchestra.run.smithi002.stderr:1+0 records out 2022-05-13T23:08:10.587 INFO:teuthology.orchestra.run.smithi002.stderr:512 bytes copied, 0.000467087 s, 1.1 MB/s 2022-05-13T23:08:10.600 DEBUG:teuthology.orchestra.run.smithi002:> stat /dev/vg_nvme/lv_2 && sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_2 2022-05-13T23:08:10.655 INFO:teuthology.orchestra.run.smithi002.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2022-05-13T23:08:10.655 INFO:teuthology.orchestra.run.smithi002.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-05-13T23:08:10.655 INFO:teuthology.orchestra.run.smithi002.stdout:Device: 5h/5d Inode: 460 Links: 1 2022-05-13T23:08:10.656 INFO:teuthology.orchestra.run.smithi002.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-05-13T23:08:10.656 INFO:teuthology.orchestra.run.smithi002.stdout:Context: system_u:object_r:device_t:s0 2022-05-13T23:08:10.656 INFO:teuthology.orchestra.run.smithi002.stdout:Access: 2022-05-13 23:07:17.011389974 +0000 2022-05-13T23:08:10.656 INFO:teuthology.orchestra.run.smithi002.stdout:Modify: 2022-05-13 23:00:09.085323081 +0000 2022-05-13T23:08:10.657 INFO:teuthology.orchestra.run.smithi002.stdout:Change: 2022-05-13 23:00:09.085323081 +0000 2022-05-13T23:08:10.657 INFO:teuthology.orchestra.run.smithi002.stdout: Birth: - 2022-05-13T23:08:10.680 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:08:10 smithi002 ceph-mon[71816]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-05-13T23:08:10.681 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:08:10 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:08:10.681 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:08:10 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-13T23:08:10.681 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:08:10 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-13T23:08:10.682 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:08:10 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-13T23:08:10.687 INFO:teuthology.orchestra.run.smithi002.stderr:1+0 records in 2022-05-13T23:08:10.688 INFO:teuthology.orchestra.run.smithi002.stderr:1+0 records out 2022-05-13T23:08:10.688 INFO:teuthology.orchestra.run.smithi002.stderr:512 bytes copied, 0.000339016 s, 1.5 MB/s 2022-05-13T23:08:10.698 DEBUG:teuthology.orchestra.run.smithi002:> stat /dev/vg_nvme/lv_3 && sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_3 2022-05-13T23:08:10.749 INFO:teuthology.orchestra.run.smithi002.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2022-05-13T23:08:10.749 INFO:teuthology.orchestra.run.smithi002.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-05-13T23:08:10.749 INFO:teuthology.orchestra.run.smithi002.stdout:Device: 5h/5d Inode: 466 Links: 1 2022-05-13T23:08:10.750 INFO:teuthology.orchestra.run.smithi002.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-05-13T23:08:10.750 INFO:teuthology.orchestra.run.smithi002.stdout:Context: system_u:object_r:device_t:s0 2022-05-13T23:08:10.750 INFO:teuthology.orchestra.run.smithi002.stdout:Access: 2022-05-13 23:07:17.012389953 +0000 2022-05-13T23:08:10.750 INFO:teuthology.orchestra.run.smithi002.stdout:Modify: 2022-05-13 23:00:10.069301300 +0000 2022-05-13T23:08:10.750 INFO:teuthology.orchestra.run.smithi002.stdout:Change: 2022-05-13 23:00:10.069301300 +0000 2022-05-13T23:08:10.751 INFO:teuthology.orchestra.run.smithi002.stdout: Birth: - 2022-05-13T23:08:10.779 INFO:teuthology.orchestra.run.smithi002.stderr:1+0 records in 2022-05-13T23:08:10.779 INFO:teuthology.orchestra.run.smithi002.stderr:1+0 records out 2022-05-13T23:08:10.780 INFO:teuthology.orchestra.run.smithi002.stderr:512 bytes copied, 0.000348286 s, 1.5 MB/s 2022-05-13T23:08:10.789 DEBUG:teuthology.orchestra.run.smithi002:> stat /dev/vg_nvme/lv_4 && sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_4 2022-05-13T23:08:10.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:08:10 smithi047 ceph-mon[77147]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-05-13T23:08:10.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:08:10 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:08:10.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:08:10 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-13T23:08:10.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:08:10 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-13T23:08:10.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:08:10 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-13T23:08:10.839 INFO:teuthology.orchestra.run.smithi002.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2022-05-13T23:08:10.840 INFO:teuthology.orchestra.run.smithi002.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-05-13T23:08:10.840 INFO:teuthology.orchestra.run.smithi002.stdout:Device: 5h/5d Inode: 472 Links: 1 2022-05-13T23:08:10.840 INFO:teuthology.orchestra.run.smithi002.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-05-13T23:08:10.841 INFO:teuthology.orchestra.run.smithi002.stdout:Context: system_u:object_r:device_t:s0 2022-05-13T23:08:10.842 INFO:teuthology.orchestra.run.smithi002.stdout:Access: 2022-05-13 23:07:17.012389953 +0000 2022-05-13T23:08:10.842 INFO:teuthology.orchestra.run.smithi002.stdout:Modify: 2022-05-13 23:00:11.051279563 +0000 2022-05-13T23:08:10.842 INFO:teuthology.orchestra.run.smithi002.stdout:Change: 2022-05-13 23:00:11.051279563 +0000 2022-05-13T23:08:10.843 INFO:teuthology.orchestra.run.smithi002.stdout: Birth: - 2022-05-13T23:08:10.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:08:10 smithi115 ceph-mon[78593]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-05-13T23:08:10.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:08:10 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:08:10.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:08:10 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-13T23:08:10.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:08:10 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-13T23:08:10.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:08:10 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-13T23:08:10.870 INFO:teuthology.orchestra.run.smithi002.stderr:1+0 records in 2022-05-13T23:08:10.870 INFO:teuthology.orchestra.run.smithi002.stderr:1+0 records out 2022-05-13T23:08:10.871 INFO:teuthology.orchestra.run.smithi002.stderr:512 bytes copied, 0.000360732 s, 1.4 MB/s 2022-05-13T23:08:10.880 DEBUG:teuthology.orchestra.run.smithi047:> set -ex 2022-05-13T23:08:10.881 DEBUG:teuthology.orchestra.run.smithi047:> dd if=/scratch_devs of=/dev/stdout 2022-05-13T23:08:10.900 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2022-05-13T23:08:10.900 DEBUG:teuthology.orchestra.run.smithi047:> stat /dev/vg_nvme/lv_1 && sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_1 2022-05-13T23:08:10.958 INFO:teuthology.orchestra.run.smithi047.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2022-05-13T23:08:10.959 INFO:teuthology.orchestra.run.smithi047.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-05-13T23:08:10.959 INFO:teuthology.orchestra.run.smithi047.stdout:Device: 6h/6d Inode: 193144 Links: 1 2022-05-13T23:08:10.960 INFO:teuthology.orchestra.run.smithi047.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-05-13T23:08:10.960 INFO:teuthology.orchestra.run.smithi047.stdout:Context: system_u:object_r:device_t:s0 2022-05-13T23:08:10.960 INFO:teuthology.orchestra.run.smithi047.stdout:Access: 2022-05-13 23:07:47.213945368 +0000 2022-05-13T23:08:10.961 INFO:teuthology.orchestra.run.smithi047.stdout:Modify: 2022-05-13 22:57:24.899983966 +0000 2022-05-13T23:08:10.961 INFO:teuthology.orchestra.run.smithi047.stdout:Change: 2022-05-13 22:57:24.899983966 +0000 2022-05-13T23:08:10.961 INFO:teuthology.orchestra.run.smithi047.stdout: Birth: - 2022-05-13T23:08:10.970 INFO:teuthology.orchestra.run.smithi047.stderr:1+0 records in 2022-05-13T23:08:10.970 INFO:teuthology.orchestra.run.smithi047.stderr:1+0 records out 2022-05-13T23:08:10.971 INFO:teuthology.orchestra.run.smithi047.stderr:512 bytes copied, 0.000130578 s, 3.9 MB/s 2022-05-13T23:08:10.974 DEBUG:teuthology.orchestra.run.smithi047:> stat /dev/vg_nvme/lv_2 && sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_2 2022-05-13T23:08:11.032 INFO:teuthology.orchestra.run.smithi047.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2022-05-13T23:08:11.033 INFO:teuthology.orchestra.run.smithi047.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-05-13T23:08:11.033 INFO:teuthology.orchestra.run.smithi047.stdout:Device: 6h/6d Inode: 196647 Links: 1 2022-05-13T23:08:11.033 INFO:teuthology.orchestra.run.smithi047.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-05-13T23:08:11.034 INFO:teuthology.orchestra.run.smithi047.stdout:Context: system_u:object_r:device_t:s0 2022-05-13T23:08:11.034 INFO:teuthology.orchestra.run.smithi047.stdout:Access: 2022-05-13 23:07:47.213945368 +0000 2022-05-13T23:08:11.034 INFO:teuthology.orchestra.run.smithi047.stdout:Modify: 2022-05-13 22:57:25.440972652 +0000 2022-05-13T23:08:11.034 INFO:teuthology.orchestra.run.smithi047.stdout:Change: 2022-05-13 22:57:25.440972652 +0000 2022-05-13T23:08:11.035 INFO:teuthology.orchestra.run.smithi047.stdout: Birth: - 2022-05-13T23:08:11.045 INFO:teuthology.orchestra.run.smithi047.stderr:1+0 records in 2022-05-13T23:08:11.045 INFO:teuthology.orchestra.run.smithi047.stderr:1+0 records out 2022-05-13T23:08:11.045 INFO:teuthology.orchestra.run.smithi047.stderr:512 bytes copied, 0.000111335 s, 4.6 MB/s 2022-05-13T23:08:11.048 DEBUG:teuthology.orchestra.run.smithi047:> stat /dev/vg_nvme/lv_3 && sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_3 2022-05-13T23:08:11.105 INFO:teuthology.orchestra.run.smithi047.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2022-05-13T23:08:11.105 INFO:teuthology.orchestra.run.smithi047.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-05-13T23:08:11.105 INFO:teuthology.orchestra.run.smithi047.stdout:Device: 6h/6d Inode: 196911 Links: 1 2022-05-13T23:08:11.105 INFO:teuthology.orchestra.run.smithi047.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-05-13T23:08:11.106 INFO:teuthology.orchestra.run.smithi047.stdout:Context: system_u:object_r:device_t:s0 2022-05-13T23:08:11.106 INFO:teuthology.orchestra.run.smithi047.stdout:Access: 2022-05-13 23:07:47.214945347 +0000 2022-05-13T23:08:11.106 INFO:teuthology.orchestra.run.smithi047.stdout:Modify: 2022-05-13 22:57:26.049959916 +0000 2022-05-13T23:08:11.106 INFO:teuthology.orchestra.run.smithi047.stdout:Change: 2022-05-13 22:57:26.049959916 +0000 2022-05-13T23:08:11.107 INFO:teuthology.orchestra.run.smithi047.stdout: Birth: - 2022-05-13T23:08:11.116 INFO:teuthology.orchestra.run.smithi047.stderr:1+0 records in 2022-05-13T23:08:11.116 INFO:teuthology.orchestra.run.smithi047.stderr:1+0 records out 2022-05-13T23:08:11.117 INFO:teuthology.orchestra.run.smithi047.stderr:512 bytes copied, 0.000106898 s, 4.8 MB/s 2022-05-13T23:08:11.119 DEBUG:teuthology.orchestra.run.smithi047:> stat /dev/vg_nvme/lv_4 && sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_4 2022-05-13T23:08:11.176 INFO:teuthology.orchestra.run.smithi047.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2022-05-13T23:08:11.176 INFO:teuthology.orchestra.run.smithi047.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-05-13T23:08:11.177 INFO:teuthology.orchestra.run.smithi047.stdout:Device: 6h/6d Inode: 196343 Links: 1 2022-05-13T23:08:11.177 INFO:teuthology.orchestra.run.smithi047.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-05-13T23:08:11.177 INFO:teuthology.orchestra.run.smithi047.stdout:Context: system_u:object_r:device_t:s0 2022-05-13T23:08:11.177 INFO:teuthology.orchestra.run.smithi047.stdout:Access: 2022-05-13 23:07:47.214945347 +0000 2022-05-13T23:08:11.178 INFO:teuthology.orchestra.run.smithi047.stdout:Modify: 2022-05-13 22:57:26.609948205 +0000 2022-05-13T23:08:11.178 INFO:teuthology.orchestra.run.smithi047.stdout:Change: 2022-05-13 22:57:26.609948205 +0000 2022-05-13T23:08:11.178 INFO:teuthology.orchestra.run.smithi047.stdout: Birth: - 2022-05-13T23:08:11.186 INFO:teuthology.orchestra.run.smithi047.stderr:1+0 records in 2022-05-13T23:08:11.186 INFO:teuthology.orchestra.run.smithi047.stderr:1+0 records out 2022-05-13T23:08:11.187 INFO:teuthology.orchestra.run.smithi047.stderr:512 bytes copied, 0.00011301 s, 4.5 MB/s 2022-05-13T23:08:11.189 DEBUG:teuthology.orchestra.run.smithi115:> set -ex 2022-05-13T23:08:11.190 DEBUG:teuthology.orchestra.run.smithi115:> dd if=/scratch_devs of=/dev/stdout 2022-05-13T23:08:11.206 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2022-05-13T23:08:11.207 DEBUG:teuthology.orchestra.run.smithi115:> stat /dev/vg_nvme/lv_1 && sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_1 2022-05-13T23:08:11.262 INFO:teuthology.orchestra.run.smithi115.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2022-05-13T23:08:11.262 INFO:teuthology.orchestra.run.smithi115.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-05-13T23:08:11.263 INFO:teuthology.orchestra.run.smithi115.stdout:Device: 6h/6d Inode: 188090 Links: 1 2022-05-13T23:08:11.263 INFO:teuthology.orchestra.run.smithi115.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-05-13T23:08:11.263 INFO:teuthology.orchestra.run.smithi115.stdout:Context: system_u:object_r:device_t:s0 2022-05-13T23:08:11.263 INFO:teuthology.orchestra.run.smithi115.stdout:Access: 2022-05-13 23:07:55.734716071 +0000 2022-05-13T23:08:11.264 INFO:teuthology.orchestra.run.smithi115.stdout:Modify: 2022-05-13 22:56:39.723263262 +0000 2022-05-13T23:08:11.264 INFO:teuthology.orchestra.run.smithi115.stdout:Change: 2022-05-13 22:56:39.723263262 +0000 2022-05-13T23:08:11.264 INFO:teuthology.orchestra.run.smithi115.stdout: Birth: - 2022-05-13T23:08:11.271 INFO:teuthology.orchestra.run.smithi115.stderr:1+0 records in 2022-05-13T23:08:11.272 INFO:teuthology.orchestra.run.smithi115.stderr:1+0 records out 2022-05-13T23:08:11.272 INFO:teuthology.orchestra.run.smithi115.stderr:512 bytes copied, 0.000232442 s, 2.2 MB/s 2022-05-13T23:08:11.275 DEBUG:teuthology.orchestra.run.smithi115:> stat /dev/vg_nvme/lv_2 && sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_2 2022-05-13T23:08:11.329 INFO:teuthology.orchestra.run.smithi115.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2022-05-13T23:08:11.330 INFO:teuthology.orchestra.run.smithi115.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-05-13T23:08:11.331 INFO:teuthology.orchestra.run.smithi115.stdout:Device: 6h/6d Inode: 188200 Links: 1 2022-05-13T23:08:11.331 INFO:teuthology.orchestra.run.smithi115.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-05-13T23:08:11.331 INFO:teuthology.orchestra.run.smithi115.stdout:Context: system_u:object_r:device_t:s0 2022-05-13T23:08:11.331 INFO:teuthology.orchestra.run.smithi115.stdout:Access: 2022-05-13 23:07:55.734716071 +0000 2022-05-13T23:08:11.332 INFO:teuthology.orchestra.run.smithi115.stdout:Modify: 2022-05-13 22:56:40.289252879 +0000 2022-05-13T23:08:11.332 INFO:teuthology.orchestra.run.smithi115.stdout:Change: 2022-05-13 22:56:40.289252879 +0000 2022-05-13T23:08:11.332 INFO:teuthology.orchestra.run.smithi115.stdout: Birth: - 2022-05-13T23:08:11.339 INFO:teuthology.orchestra.run.smithi115.stderr:1+0 records in 2022-05-13T23:08:11.339 INFO:teuthology.orchestra.run.smithi115.stderr:1+0 records out 2022-05-13T23:08:11.339 INFO:teuthology.orchestra.run.smithi115.stderr:512 bytes copied, 0.000220915 s, 2.3 MB/s 2022-05-13T23:08:11.342 DEBUG:teuthology.orchestra.run.smithi115:> stat /dev/vg_nvme/lv_3 && sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_3 2022-05-13T23:08:11.397 INFO:teuthology.orchestra.run.smithi115.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2022-05-13T23:08:11.397 INFO:teuthology.orchestra.run.smithi115.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-05-13T23:08:11.398 INFO:teuthology.orchestra.run.smithi115.stdout:Device: 6h/6d Inode: 195015 Links: 1 2022-05-13T23:08:11.398 INFO:teuthology.orchestra.run.smithi115.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-05-13T23:08:11.398 INFO:teuthology.orchestra.run.smithi115.stdout:Context: system_u:object_r:device_t:s0 2022-05-13T23:08:11.398 INFO:teuthology.orchestra.run.smithi115.stdout:Access: 2022-05-13 23:07:55.734716071 +0000 2022-05-13T23:08:11.398 INFO:teuthology.orchestra.run.smithi115.stdout:Modify: 2022-05-13 22:56:40.892241821 +0000 2022-05-13T23:08:11.399 INFO:teuthology.orchestra.run.smithi115.stdout:Change: 2022-05-13 22:56:40.892241821 +0000 2022-05-13T23:08:11.399 INFO:teuthology.orchestra.run.smithi115.stdout: Birth: - 2022-05-13T23:08:11.407 INFO:teuthology.orchestra.run.smithi115.stderr:1+0 records in 2022-05-13T23:08:11.407 INFO:teuthology.orchestra.run.smithi115.stderr:1+0 records out 2022-05-13T23:08:11.407 INFO:teuthology.orchestra.run.smithi115.stderr:512 bytes copied, 0.000201809 s, 2.5 MB/s 2022-05-13T23:08:11.410 DEBUG:teuthology.orchestra.run.smithi115:> stat /dev/vg_nvme/lv_4 && sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_4 2022-05-13T23:08:11.466 INFO:teuthology.orchestra.run.smithi115.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2022-05-13T23:08:11.467 INFO:teuthology.orchestra.run.smithi115.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-05-13T23:08:11.467 INFO:teuthology.orchestra.run.smithi115.stdout:Device: 6h/6d Inode: 192239 Links: 1 2022-05-13T23:08:11.467 INFO:teuthology.orchestra.run.smithi115.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-05-13T23:08:11.467 INFO:teuthology.orchestra.run.smithi115.stdout:Context: system_u:object_r:device_t:s0 2022-05-13T23:08:11.468 INFO:teuthology.orchestra.run.smithi115.stdout:Access: 2022-05-13 23:07:55.734716071 +0000 2022-05-13T23:08:11.468 INFO:teuthology.orchestra.run.smithi115.stdout:Modify: 2022-05-13 22:56:41.442231733 +0000 2022-05-13T23:08:11.468 INFO:teuthology.orchestra.run.smithi115.stdout:Change: 2022-05-13 22:56:41.442231733 +0000 2022-05-13T23:08:11.468 INFO:teuthology.orchestra.run.smithi115.stdout: Birth: - 2022-05-13T23:08:11.476 INFO:teuthology.orchestra.run.smithi115.stderr:1+0 records in 2022-05-13T23:08:11.476 INFO:teuthology.orchestra.run.smithi115.stderr:1+0 records out 2022-05-13T23:08:11.477 INFO:teuthology.orchestra.run.smithi115.stderr:512 bytes copied, 0.000218022 s, 2.3 MB/s 2022-05-13T23:08:11.480 INFO:tasks.cephadm:Deploying osd.0 on smithi002 with /dev/vg_nvme/lv_4... 2022-05-13T23:08:11.481 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:23b0658391ce94ca40799469eec00c28b298816c shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 16cf5f88-d311-11ec-be40-001a4aab830c -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2022-05-13T23:08:11.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:08:11 smithi115 ceph-mon[78593]: from='client.24110 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi002=x;smithi047=y;smithi115=z", "target": ["mon-mgr", ""]}]: dispatch 2022-05-13T23:08:11.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:08:11 smithi115 ceph-mon[78593]: Saving service mgr spec with placement smithi002=x;smithi047=y;smithi115=z;count:3 2022-05-13T23:08:11.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:08:11 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:08:11.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:08:11 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2022-05-13T23:08:11.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:08:11 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2022-05-13T23:08:11.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:08:11 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2022-05-13T23:08:11.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:08:11 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-13T23:08:11.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:08:11 smithi047 ceph-mon[77147]: from='client.24110 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi002=x;smithi047=y;smithi115=z", "target": ["mon-mgr", ""]}]: dispatch 2022-05-13T23:08:11.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:08:11 smithi047 ceph-mon[77147]: Saving service mgr spec with placement smithi002=x;smithi047=y;smithi115=z;count:3 2022-05-13T23:08:11.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:08:11 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:08:11.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:08:11 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2022-05-13T23:08:11.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:08:11 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2022-05-13T23:08:11.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:08:11 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2022-05-13T23:08:11.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:08:11 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-13T23:08:11.822 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:08:11 smithi002 ceph-mon[71816]: from='client.24110 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi002=x;smithi047=y;smithi115=z", "target": ["mon-mgr", ""]}]: dispatch 2022-05-13T23:08:11.823 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:08:11 smithi002 ceph-mon[71816]: Saving service mgr spec with placement smithi002=x;smithi047=y;smithi115=z;count:3 2022-05-13T23:08:11.823 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:08:11 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:08:11.824 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:08:11 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2022-05-13T23:08:11.825 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:08:11 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2022-05-13T23:08:11.825 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:08:11 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2022-05-13T23:08:11.826 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:08:11 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-13T23:08:12.152 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/16cf5f88-d311-11ec-be40-001a4aab830c/mon.a/config 2022-05-13T23:08:12.647 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:08:12 smithi047 ceph-mon[77147]: Deploying daemon mgr.y on smithi047 2022-05-13T23:08:12.647 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:08:12 smithi047 ceph-mon[77147]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-05-13T23:08:12.647 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:08:12 smithi047 systemd[1]: Starting Ceph mgr.y for 16cf5f88-d311-11ec-be40-001a4aab830c... 2022-05-13T23:08:12.727 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:08:12 smithi002 ceph-mon[71816]: Deploying daemon mgr.y on smithi047 2022-05-13T23:08:12.727 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:08:12 smithi002 ceph-mon[71816]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-05-13T23:08:12.864 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:08:12 smithi115 ceph-mon[78593]: Deploying daemon mgr.y on smithi047 2022-05-13T23:08:12.931 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:08:12 smithi115 ceph-mon[78593]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-05-13T23:08:13.541 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:08:13 smithi047 ceph-mgr[78545]: -- 172.21.15.47:0/1166506858 <== mon.1 v2:172.21.15.47:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5622caa2c340 con 0x5622caa04800 2022-05-13T23:08:13.542 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:08:13 smithi047 bash[78335]: 6f563916e52374bcac59163444db6ae190cd4f1f15e0b06e1da42901183e7bb0 2022-05-13T23:08:13.542 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:08:13 smithi047 systemd[1]: Started Ceph mgr.y for 16cf5f88-d311-11ec-be40-001a4aab830c. 2022-05-13T23:08:13.832 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:08:13 smithi047 conmon[78521]: 2022-05-13T23:08:13.553+0000 7f53b2c8f000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2022-05-13T23:08:13.833 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:08:13 smithi047 conmon[78521]: 2022-05-13T23:08:13.651+0000 7f53b2c8f000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2022-05-13T23:08:13.833 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:08:13 smithi047 conmon[78521]: 2022-05-13T23:08:13.831+0000 7f53b2c8f000 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2022-05-13T23:08:14.291 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:08:13 smithi047 conmon[78521]: 2022-05-13T23:08:13.939+0000 7f53b2c8f000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2022-05-13T23:08:14.292 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:08:14 smithi047 conmon[78521]: 2022-05-13T23:08:14.034+0000 7f53b2c8f000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2022-05-13T23:08:14.571 INFO:teuthology.orchestra.run.smithi002.stderr:--> Zapping: /dev/vg_nvme/lv_4 2022-05-13T23:08:14.572 INFO:teuthology.orchestra.run.smithi002.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2022-05-13T23:08:14.572 INFO:teuthology.orchestra.run.smithi002.stderr: stderr: 10+0 records in 2022-05-13T23:08:14.572 INFO:teuthology.orchestra.run.smithi002.stderr:10+0 records out 2022-05-13T23:08:14.572 INFO:teuthology.orchestra.run.smithi002.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0583629 s, 180 MB/s 2022-05-13T23:08:14.572 INFO:teuthology.orchestra.run.smithi002.stderr:--> Zapping successful for: 2022-05-13T23:08:14.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:08:14 smithi115 ceph-mon[78593]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-05-13T23:08:14.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:08:14 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:08:14.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:08:14 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2022-05-13T23:08:14.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:08:14 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2022-05-13T23:08:14.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:08:14 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2022-05-13T23:08:14.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:08:14 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-13T23:08:14.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:08:14 smithi115 ceph-mon[78593]: Deploying daemon mgr.z on smithi115 2022-05-13T23:08:14.613 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:08:14 smithi002 ceph-mon[71816]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-05-13T23:08:14.613 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:08:14 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:08:14.614 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:08:14 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2022-05-13T23:08:14.614 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:08:14 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2022-05-13T23:08:14.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:08:14 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2022-05-13T23:08:14.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:08:14 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-13T23:08:14.615 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:08:14 smithi002 ceph-mon[71816]: Deploying daemon mgr.z on smithi115 2022-05-13T23:08:14.791 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:08:14 smithi047 conmon[78521]: 2022-05-13T23:08:14.402+0000 7f53b2c8f000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2022-05-13T23:08:14.792 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:08:14 smithi047 conmon[78521]: 2022-05-13T23:08:14.515+0000 7f53b2c8f000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2022-05-13T23:08:14.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:08:14 smithi047 ceph-mon[77147]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-05-13T23:08:14.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:08:14 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:08:14.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:08:14 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2022-05-13T23:08:14.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:08:14 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2022-05-13T23:08:14.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:08:14 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2022-05-13T23:08:14.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:08:14 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-13T23:08:14.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:08:14 smithi047 ceph-mon[77147]: Deploying daemon mgr.z on smithi115 2022-05-13T23:08:15.108 INFO:journalctl@ceph.mgr.z.smithi115.stdout:May 13 23:08:14 smithi115 systemd[1]: Starting Ceph mgr.z for 16cf5f88-d311-11ec-be40-001a4aab830c... 2022-05-13T23:08:15.224 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:23b0658391ce94ca40799469eec00c28b298816c shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 16cf5f88-d311-11ec-be40-001a4aab830c -- ceph orch daemon add osd smithi002:vg_nvme/lv_4 2022-05-13T23:08:15.445 INFO:journalctl@ceph.mgr.z.smithi115.stdout:May 13 23:08:15 smithi115 ceph-mgr[80424]: -- 172.21.15.115:0/3114939674 <== mon.2 v2:172.21.15.115:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55c707de8340 con 0x55c707dc1000 2022-05-13T23:08:15.446 INFO:journalctl@ceph.mgr.z.smithi115.stdout:May 13 23:08:15 smithi115 bash[80213]: 20748358073aa7076e0acac0a2509c29189e2a8d0658025eb2405b64a7b30583 2022-05-13T23:08:15.446 INFO:journalctl@ceph.mgr.z.smithi115.stdout:May 13 23:08:15 smithi115 systemd[1]: Started Ceph mgr.z for 16cf5f88-d311-11ec-be40-001a4aab830c. 2022-05-13T23:08:15.652 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:08:15 smithi047 conmon[78521]: 2022-05-13T23:08:15.371+0000 7f53b2c8f000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2022-05-13T23:08:15.652 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:08:15 smithi047 conmon[78521]: 2022-05-13T23:08:15.453+0000 7f53b2c8f000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2022-05-13T23:08:15.902 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/16cf5f88-d311-11ec-be40-001a4aab830c/mon.a/config 2022-05-13T23:08:15.924 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:08:15 smithi047 conmon[78521]: 2022-05-13T23:08:15.651+0000 7f53b2c8f000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2022-05-13T23:08:15.971 INFO:journalctl@ceph.mgr.z.smithi115.stdout:May 13 23:08:15 smithi115 conmon[80399]: 2022-05-13T23:08:15.697+0000 7f62101ab000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2022-05-13T23:08:15.971 INFO:journalctl@ceph.mgr.z.smithi115.stdout:May 13 23:08:15 smithi115 conmon[80399]: 2022-05-13T23:08:15.800+0000 7f62101ab000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2022-05-13T23:08:16.250 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:08:15 smithi047 conmon[78521]: 2022-05-13T23:08:15.923+0000 7f53b2c8f000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2022-05-13T23:08:16.250 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:08:16 smithi047 conmon[78521]: 2022-05-13T23:08:16.092+0000 7f53b2c8f000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2022-05-13T23:08:16.331 INFO:journalctl@ceph.mgr.z.smithi115.stdout:May 13 23:08:15 smithi115 conmon[80399]: 2022-05-13T23:08:15.968+0000 7f62101ab000 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2022-05-13T23:08:16.331 INFO:journalctl@ceph.mgr.z.smithi115.stdout:May 13 23:08:16 smithi115 conmon[80399]: 2022-05-13T23:08:16.082+0000 7f62101ab000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2022-05-13T23:08:16.331 INFO:journalctl@ceph.mgr.z.smithi115.stdout:May 13 23:08:16 smithi115 conmon[80399]: 2022-05-13T23:08:16.176+0000 7f62101ab000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2022-05-13T23:08:16.524 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:08:16 smithi002 ceph-mon[71816]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-05-13T23:08:16.524 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:08:16 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:08:16.525 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:08:16 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-13T23:08:16.525 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:08:16 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-13T23:08:16.525 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:08:16 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-13T23:08:16.541 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:08:16 smithi047 conmon[78521]: 2022-05-13T23:08:16.248+0000 7f53b2c8f000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2022-05-13T23:08:16.541 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:08:16 smithi047 conmon[78521]: 2022-05-13T23:08:16.459+0000 7f53b2c8f000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2022-05-13T23:08:16.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:08:16 smithi047 ceph-mon[77147]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-05-13T23:08:16.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:08:16 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:08:16.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:08:16 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-13T23:08:16.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:08:16 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-13T23:08:16.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:08:16 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-13T23:08:16.608 INFO:journalctl@ceph.mgr.z.smithi115.stdout:May 13 23:08:16 smithi115 conmon[80399]: 2022-05-13T23:08:16.555+0000 7f62101ab000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2022-05-13T23:08:16.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:08:16 smithi115 ceph-mon[78593]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-05-13T23:08:16.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:08:16 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:08:16.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:08:16 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-13T23:08:16.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:08:16 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-13T23:08:16.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:08:16 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-13T23:08:16.830 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:08:16 smithi047 conmon[78521]: 2022-05-13T23:08:16.558+0000 7f53b2c8f000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2022-05-13T23:08:16.830 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:08:16 smithi047 conmon[78521]: 2022-05-13T23:08:16.644+0000 7f53b2c8f000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2022-05-13T23:08:16.830 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:08:16 smithi047 conmon[78521]: 2022-05-13T23:08:16.740+0000 7f53b2c8f000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2022-05-13T23:08:16.830 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:08:16 smithi047 conmon[78521]: 2022-05-13T23:08:16.828+0000 7f53b2c8f000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2022-05-13T23:08:16.980 INFO:journalctl@ceph.mgr.z.smithi115.stdout:May 13 23:08:16 smithi115 conmon[80399]: 2022-05-13T23:08:16.664+0000 7f62101ab000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2022-05-13T23:08:17.167 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:08:16 smithi047 conmon[78521]: 2022-05-13T23:08:16.929+0000 7f53b2c8f000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2022-05-13T23:08:17.541 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:08:17 smithi047 conmon[78521]: 2022-05-13T23:08:17.165+0000 7f53b2c8f000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2022-05-13T23:08:17.738 INFO:journalctl@ceph.mgr.z.smithi115.stdout:May 13 23:08:17 smithi115 conmon[80399]: 2022-05-13T23:08:17.479+0000 7f62101ab000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2022-05-13T23:08:17.739 INFO:journalctl@ceph.mgr.z.smithi115.stdout:May 13 23:08:17 smithi115 conmon[80399]: 2022-05-13T23:08:17.557+0000 7f62101ab000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2022-05-13T23:08:17.837 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:08:17.835+0000 7f3400f67700 1 -- 172.21.15.2:0/2863283442 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f33fc0ff260 msgr2=0x7f33fc105b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:08:17.838 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:08:17.835+0000 7f3400f67700 1 --2- 172.21.15.2:0/2863283442 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f33fc0ff260 0x7f33fc105b20 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f33f4009be0 tx=0x7f33f4004d10 comp rx=0 tx=0).stop 2022-05-13T23:08:17.838 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:08:17.835+0000 7f3400f67700 1 -- 172.21.15.2:0/2863283442 shutdown_connections 2022-05-13T23:08:17.839 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:08:17.835+0000 7f3400f67700 1 --2- 172.21.15.2:0/2863283442 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f33fc0ff260 0x7f33fc105b20 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:08:17.839 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:08:17.835+0000 7f3400f67700 1 --2- 172.21.15.2:0/2863283442 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f33fc0fd630 0x7f33fc0fda40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:08:17.839 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:08:17.835+0000 7f3400f67700 1 --2- 172.21.15.2:0/2863283442 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f33fc0fe8d0 0x7f33fc0fed20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:08:17.840 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:08:17.835+0000 7f3400f67700 1 -- 172.21.15.2:0/2863283442 >> 172.21.15.2:0/2863283442 conn(0x7f33fc0f8e70 msgr2=0x7f33fc0fb250 unknown :-1 s=STATE_NONE l=0).mark_down 2022-05-13T23:08:17.840 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:08:17.835+0000 7f3400f67700 1 -- 172.21.15.2:0/2863283442 shutdown_connections 2022-05-13T23:08:17.840 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:08:17.836+0000 7f3400f67700 1 -- 172.21.15.2:0/2863283442 wait complete. 2022-05-13T23:08:17.844 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:08:17.837+0000 7f3400f67700 1 Processor -- start 2022-05-13T23:08:17.844 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:08:17.837+0000 7f3400f67700 1 -- start start 2022-05-13T23:08:17.844 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:08:17.838+0000 7f3400f67700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f33fc0fd630 0x7f33fc10e300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:08:17.845 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:08:17.838+0000 7f3400f67700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f33fc0fe8d0 0x7f33fc10e840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:08:17.845 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:08:17.838+0000 7f3400f67700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f33fc0ff260 0x7f33fc1158d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:08:17.845 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:08:17.838+0000 7f3400f67700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33fc115e10 con 0x7f33fc0fe8d0 2022-05-13T23:08:17.845 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:08:17.838+0000 7f3400f67700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33fc115f80 con 0x7f33fc0fd630 2022-05-13T23:08:17.846 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:08:17.838+0000 7f3400f67700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33fc116140 con 0x7f33fc0ff260 2022-05-13T23:08:17.846 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:08:17.838+0000 7f33fb7fe700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f33fc0fd630 0x7f33fc10e300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:08:17.846 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:08:17.838+0000 7f33fbfff700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f33fc0ff260 0x7f33fc1158d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:08:17.847 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:08:17.838+0000 7f33fb7fe700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f33fc0fd630 0x7f33fc10e300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.2:56458/0 (socket says 172.21.15.2:56458) 2022-05-13T23:08:17.847 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:08:17.838+0000 7f33fb7fe700 1 -- 172.21.15.2:0/990629422 learned_addr learned my addr 172.21.15.2:0/990629422 (peer_addr_for_me v2:172.21.15.2:0/0) 2022-05-13T23:08:17.847 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:08:17.838+0000 7f33faffd700 1 --2- 172.21.15.2:0/990629422 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f33fc0fe8d0 0x7f33fc10e840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:08:17.847 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:08:17.838+0000 7f33fbfff700 1 -- 172.21.15.2:0/990629422 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f33fc0fd630 msgr2=0x7f33fc10e300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:08:17.848 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:08:17.838+0000 7f33fbfff700 1 --2- 172.21.15.2:0/990629422 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f33fc0fd630 0x7f33fc10e300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:08:17.848 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:08:17.838+0000 7f33fbfff700 1 -- 172.21.15.2:0/990629422 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f33fc0fe8d0 msgr2=0x7f33fc10e840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:08:17.848 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:08:17.838+0000 7f33fbfff700 1 --2- 172.21.15.2:0/990629422 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f33fc0fe8d0 0x7f33fc10e840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:08:17.848 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:08:17.839+0000 7f33fbfff700 1 -- 172.21.15.2:0/990629422 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33f40097c0 con 0x7f33fc0ff260 2022-05-13T23:08:17.848 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:08:17.839+0000 7f33fb7fe700 1 --2- 172.21.15.2:0/990629422 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f33fc0fd630 0x7f33fc10e300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-05-13T23:08:17.849 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:08:17.839+0000 7f33fbfff700 1 --2- 172.21.15.2:0/990629422 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f33fc0ff260 0x7f33fc1158d0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f33f4015040 tx=0x7f33f4004d10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-05-13T23:08:17.849 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:08:17.839+0000 7f33e3fff700 1 -- 172.21.15.2:0/990629422 <== mon.2 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33f400e070 con 0x7f33fc0ff260 2022-05-13T23:08:17.849 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:08:17.839+0000 7f33e3fff700 1 -- 172.21.15.2:0/990629422 <== mon.2 v2:172.21.15.115:3300/0 2 ==== config(26 keys) v1 ==== 1087+0+0 (secure 0 0 0) 0x7f33f4012410 con 0x7f33fc0ff260 2022-05-13T23:08:17.849 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:08:17.839+0000 7f3400f67700 1 -- 172.21.15.2:0/990629422 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f33fc1163c0 con 0x7f33fc0ff260 2022-05-13T23:08:17.850 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:08:17.839+0000 7f33e3fff700 1 -- 172.21.15.2:0/990629422 <== mon.2 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33f401c670 con 0x7f33fc0ff260 2022-05-13T23:08:17.850 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:08:17.839+0000 7f3400f67700 1 -- 172.21.15.2:0/990629422 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f33fc1168a0 con 0x7f33fc0ff260 2022-05-13T23:08:17.850 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:08:17.840+0000 7f33e3fff700 1 -- 172.21.15.2:0/990629422 <== mon.2 v2:172.21.15.115:3300/0 4 ==== mgrmap(e 13) v1 ==== 41898+0+0 (secure 0 0 0) 0x7f33f401caa0 con 0x7f33fc0ff260 2022-05-13T23:08:17.850 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:08:17.841+0000 7f3400f67700 1 -- 172.21.15.2:0/990629422 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f33fc108e90 con 0x7f33fc0ff260 2022-05-13T23:08:17.850 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:08:17.841+0000 7f33e3fff700 1 --2- 172.21.15.2:0/990629422 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f33e4034e90 0x7f33e4037340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:08:17.851 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:08:17.841+0000 7f33e3fff700 1 -- 172.21.15.2:0/990629422 <== mon.2 v2:172.21.15.115:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f33f4047b90 con 0x7f33fc0ff260 2022-05-13T23:08:17.851 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:08:17.841+0000 7f33fb7fe700 1 --2- 172.21.15.2:0/990629422 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f33e4034e90 0x7f33e4037340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:08:17.851 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:08:17.842+0000 7f33fb7fe700 1 --2- 172.21.15.2:0/990629422 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f33e4034e90 0x7f33e4037340 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f33ec0041d0 tx=0x7f33ec00b040 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-05-13T23:08:17.852 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:08:17.846+0000 7f33e3fff700 1 -- 172.21.15.2:0/990629422 <== mon.2 v2:172.21.15.115:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+165215 (secure 0 0 0) 0x7f33f4023300 con 0x7f33fc0ff260 2022-05-13T23:08:17.988 INFO:journalctl@ceph.mgr.z.smithi115.stdout:May 13 23:08:17 smithi115 conmon[80399]: 2022-05-13T23:08:17.736+0000 7f62101ab000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2022-05-13T23:08:17.988 INFO:journalctl@ceph.mgr.z.smithi115.stdout:May 13 23:08:17 smithi115 conmon[80399]: 2022-05-13T23:08:17.984+0000 7f62101ab000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2022-05-13T23:08:17.989 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:08:17 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:08:17.989 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:08:17 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:08:17.990 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:08:17 smithi047 conmon[78521]: 2022-05-13T23:08:17.647+0000 7f53b2c8f000 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2022-05-13T23:08:17.990 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:08:17 smithi047 conmon[78521]: 2022-05-13T23:08:17.816+0000 7f53b2c8f000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2022-05-13T23:08:18.057 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:08:18.056+0000 7f3400f67700 1 -- 172.21.15.2:0/990629422 --> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi002:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) v1 -- 0x7f33fc02a530 con 0x7f33e4034e90 2022-05-13T23:08:18.239 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:08:17 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:08:18.239 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:08:17 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:08:18.239 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:08:17 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:08:18.240 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:08:17 smithi002 ceph-mon[71816]: Reconfiguring mgr.x (unknown last config time)... 2022-05-13T23:08:18.240 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:08:17 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2022-05-13T23:08:18.240 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:08:17 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2022-05-13T23:08:18.240 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:08:17 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-13T23:08:18.240 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:08:17 smithi002 ceph-mon[71816]: Reconfiguring daemon mgr.x on smithi002 2022-05-13T23:08:18.241 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:08:17 smithi002 ceph-mon[71816]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-05-13T23:08:18.266 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:08:17 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:08:18.266 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:08:17 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:08:18.267 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:08:17 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:08:18.267 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:08:17 smithi047 ceph-mon[77147]: Reconfiguring mgr.x (unknown last config time)... 2022-05-13T23:08:18.267 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:08:17 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2022-05-13T23:08:18.267 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:08:17 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2022-05-13T23:08:18.267 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:08:17 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-13T23:08:18.268 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:08:17 smithi047 ceph-mon[77147]: Reconfiguring daemon mgr.x on smithi002 2022-05-13T23:08:18.268 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:08:17 smithi047 ceph-mon[77147]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-05-13T23:08:18.281 INFO:journalctl@ceph.mgr.z.smithi115.stdout:May 13 23:08:18 smithi115 conmon[80399]: 2022-05-13T23:08:18.139+0000 7f62101ab000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2022-05-13T23:08:18.281 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:08:17 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:08:18.281 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:08:17 smithi115 ceph-mon[78593]: Reconfiguring mgr.x (unknown last config time)... 2022-05-13T23:08:18.282 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:08:17 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2022-05-13T23:08:18.282 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:08:17 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2022-05-13T23:08:18.282 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:08:17 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-13T23:08:18.282 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:08:17 smithi115 ceph-mon[78593]: Reconfiguring daemon mgr.x on smithi002 2022-05-13T23:08:18.282 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:08:17 smithi115 ceph-mon[78593]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-05-13T23:08:18.541 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:08:18 smithi047 conmon[78521]: 2022-05-13T23:08:18.264+0000 7f53b2c8f000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2022-05-13T23:08:18.542 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:08:18 smithi047 conmon[78521]: 2022-05-13T23:08:18.355+0000 7f53b2c8f000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2022-05-13T23:08:18.564 INFO:journalctl@ceph.mgr.z.smithi115.stdout:May 13 23:08:18 smithi115 conmon[80399]: 2022-05-13T23:08:18.279+0000 7f62101ab000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2022-05-13T23:08:18.564 INFO:journalctl@ceph.mgr.z.smithi115.stdout:May 13 23:08:18 smithi115 conmon[80399]: 2022-05-13T23:08:18.473+0000 7f62101ab000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2022-05-13T23:08:18.819 INFO:journalctl@ceph.mgr.z.smithi115.stdout:May 13 23:08:18 smithi115 conmon[80399]: 2022-05-13T23:08:18.562+0000 7f62101ab000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2022-05-13T23:08:18.819 INFO:journalctl@ceph.mgr.z.smithi115.stdout:May 13 23:08:18 smithi115 conmon[80399]: 2022-05-13T23:08:18.642+0000 7f62101ab000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2022-05-13T23:08:18.820 INFO:journalctl@ceph.mgr.z.smithi115.stdout:May 13 23:08:18 smithi115 conmon[80399]: 2022-05-13T23:08:18.733+0000 7f62101ab000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2022-05-13T23:08:18.820 INFO:journalctl@ceph.mgr.z.smithi115.stdout:May 13 23:08:18 smithi115 conmon[80399]: 2022-05-13T23:08:18.817+0000 7f62101ab000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2022-05-13T23:08:19.109 INFO:journalctl@ceph.mgr.z.smithi115.stdout:May 13 23:08:18 smithi115 conmon[80399]: 2022-05-13T23:08:18.907+0000 7f62101ab000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2022-05-13T23:08:19.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:08:18 smithi115 ceph-mon[78593]: from='client.24125 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi002:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2022-05-13T23:08:19.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:08:18 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2022-05-13T23:08:19.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:08:18 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2022-05-13T23:08:19.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:08:18 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-13T23:08:19.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:08:18 smithi047 ceph-mon[77147]: from='client.24125 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi002:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2022-05-13T23:08:19.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:08:18 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2022-05-13T23:08:19.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:08:18 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2022-05-13T23:08:19.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:08:18 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-13T23:08:19.323 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:08:18 smithi002 ceph-mon[71816]: from='client.24125 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi002:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2022-05-13T23:08:19.323 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:08:18 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2022-05-13T23:08:19.324 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:08:18 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2022-05-13T23:08:19.324 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:08:18 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-13T23:08:19.575 INFO:journalctl@ceph.mgr.z.smithi115.stdout:May 13 23:08:19 smithi115 conmon[80399]: 2022-05-13T23:08:19.128+0000 7f62101ab000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2022-05-13T23:08:19.858 INFO:journalctl@ceph.mgr.z.smithi115.stdout:May 13 23:08:19 smithi115 conmon[80399]: 2022-05-13T23:08:19.573+0000 7f62101ab000 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2022-05-13T23:08:19.858 INFO:journalctl@ceph.mgr.z.smithi115.stdout:May 13 23:08:19 smithi115 conmon[80399]: 2022-05-13T23:08:19.732+0000 7f62101ab000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2022-05-13T23:08:20.170 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:08:20.168+0000 7f33e3fff700 1 -- 172.21.15.2:0/990629422 <== mon.2 v2:172.21.15.115:3300/0 7 ==== mgrmap(e 14) v1 ==== 83497+0+0 (secure 0 0 0) 0x7f33f401cd30 con 0x7f33fc0ff260 2022-05-13T23:08:20.184 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:08:20 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:08:20.185 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:08:20 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-13T23:08:20.185 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:08:20 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-13T23:08:20.185 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:08:20 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-13T23:08:20.185 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:08:20 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:08:20.186 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:08:20 smithi002 ceph-mon[71816]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-05-13T23:08:20.186 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:08:20 smithi002 ceph-mon[71816]: Standby manager daemon y started 2022-05-13T23:08:20.186 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:08:20 smithi002 ceph-mon[71816]: from='mgr.? 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2022-05-13T23:08:20.186 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:08:20 smithi002 ceph-mon[71816]: from='mgr.? 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2022-05-13T23:08:20.186 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:08:20 smithi002 ceph-mon[71816]: from='mgr.? 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2022-05-13T23:08:20.187 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:08:20 smithi002 ceph-mon[71816]: from='mgr.? 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2022-05-13T23:08:20.358 INFO:journalctl@ceph.mgr.z.smithi115.stdout:May 13 23:08:20 smithi115 conmon[80399]: 2022-05-13T23:08:20.152+0000 7f62101ab000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2022-05-13T23:08:20.359 INFO:journalctl@ceph.mgr.z.smithi115.stdout:May 13 23:08:20 smithi115 conmon[80399]: 2022-05-13T23:08:20.239+0000 7f62101ab000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2022-05-13T23:08:20.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:08:20 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:08:20.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:08:20 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-13T23:08:20.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:08:20 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-13T23:08:20.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:08:20 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-13T23:08:20.362 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:08:20 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:08:20.362 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:08:20 smithi115 ceph-mon[78593]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-05-13T23:08:20.362 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:08:20 smithi115 ceph-mon[78593]: Standby manager daemon y started 2022-05-13T23:08:20.363 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:08:20 smithi115 ceph-mon[78593]: from='mgr.? 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2022-05-13T23:08:20.363 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:08:20 smithi115 ceph-mon[78593]: from='mgr.? 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2022-05-13T23:08:20.364 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:08:20 smithi115 ceph-mon[78593]: from='mgr.? 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2022-05-13T23:08:20.364 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:08:20 smithi115 ceph-mon[78593]: from='mgr.? 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2022-05-13T23:08:20.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:08:20 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:08:20.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:08:20 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-13T23:08:20.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:08:20 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-13T23:08:20.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:08:20 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-13T23:08:20.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:08:20 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:08:20.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:08:20 smithi047 ceph-mon[77147]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-05-13T23:08:20.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:08:20 smithi047 ceph-mon[77147]: Standby manager daemon y started 2022-05-13T23:08:20.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:08:20 smithi047 ceph-mon[77147]: from='mgr.? 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2022-05-13T23:08:20.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:08:20 smithi047 ceph-mon[77147]: from='mgr.? 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2022-05-13T23:08:20.545 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:08:20 smithi047 ceph-mon[77147]: from='mgr.? 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2022-05-13T23:08:20.546 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:08:20 smithi047 ceph-mon[77147]: from='mgr.? 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2022-05-13T23:08:21.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:08:21 smithi047 ceph-mon[77147]: mgrmap e14: x(active, since 75s), standbys: y 2022-05-13T23:08:21.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:08:21 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2022-05-13T23:08:21.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:08:21 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:08:21.572 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:08:21 smithi002 ceph-mon[71816]: mgrmap e14: x(active, since 75s), standbys: y 2022-05-13T23:08:21.573 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:08:21 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2022-05-13T23:08:21.573 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:08:21 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:08:21.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:08:21 smithi115 ceph-mon[78593]: mgrmap e14: x(active, since 75s), standbys: y 2022-05-13T23:08:21.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:08:21 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2022-05-13T23:08:21.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:08:21 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:08:22.240 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:08:22.237+0000 7f33e3fff700 1 -- 172.21.15.2:0/990629422 <== mon.2 v2:172.21.15.115:3300/0 8 ==== mgrmap(e 15) v1 ==== 125096+0+0 (secure 0 0 0) 0x7f33f40125f0 con 0x7f33fc0ff260 2022-05-13T23:08:22.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:08:22 smithi047 ceph-mon[77147]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-05-13T23:08:22.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:08:22 smithi047 ceph-mon[77147]: Standby manager daemon z started 2022-05-13T23:08:22.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:08:22 smithi047 ceph-mon[77147]: from='mgr.? 172.21.15.115:0/3069476012' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2022-05-13T23:08:22.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:08:22 smithi047 ceph-mon[77147]: from='mgr.? 172.21.15.115:0/3069476012' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2022-05-13T23:08:22.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:08:22 smithi047 ceph-mon[77147]: from='mgr.? 172.21.15.115:0/3069476012' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2022-05-13T23:08:22.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:08:22 smithi047 ceph-mon[77147]: from='mgr.? 172.21.15.115:0/3069476012' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2022-05-13T23:08:22.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:08:22 smithi047 ceph-mon[77147]: from='client.? 172.21.15.2:0/2726198310' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "11e700e8-f3e8-4903-b52c-db354b4395bf"}]: dispatch 2022-05-13T23:08:22.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:08:22 smithi047 ceph-mon[77147]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "11e700e8-f3e8-4903-b52c-db354b4395bf"}]: dispatch 2022-05-13T23:08:22.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:08:22 smithi047 ceph-mon[77147]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "11e700e8-f3e8-4903-b52c-db354b4395bf"}]': finished 2022-05-13T23:08:22.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:08:22 smithi047 ceph-mon[77147]: osdmap e5: 1 total, 0 up, 1 in 2022-05-13T23:08:22.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:08:22 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2022-05-13T23:08:22.572 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:08:22 smithi002 ceph-mon[71816]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-05-13T23:08:22.572 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:08:22 smithi002 ceph-mon[71816]: Standby manager daemon z started 2022-05-13T23:08:22.572 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:08:22 smithi002 ceph-mon[71816]: from='mgr.? 172.21.15.115:0/3069476012' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2022-05-13T23:08:22.573 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:08:22 smithi002 ceph-mon[71816]: from='mgr.? 172.21.15.115:0/3069476012' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2022-05-13T23:08:22.573 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:08:22 smithi002 ceph-mon[71816]: from='mgr.? 172.21.15.115:0/3069476012' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2022-05-13T23:08:22.573 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:08:22 smithi002 ceph-mon[71816]: from='mgr.? 172.21.15.115:0/3069476012' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2022-05-13T23:08:22.573 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:08:22 smithi002 ceph-mon[71816]: from='client.? 172.21.15.2:0/2726198310' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "11e700e8-f3e8-4903-b52c-db354b4395bf"}]: dispatch 2022-05-13T23:08:22.574 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:08:22 smithi002 ceph-mon[71816]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "11e700e8-f3e8-4903-b52c-db354b4395bf"}]: dispatch 2022-05-13T23:08:22.574 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:08:22 smithi002 ceph-mon[71816]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "11e700e8-f3e8-4903-b52c-db354b4395bf"}]': finished 2022-05-13T23:08:22.574 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:08:22 smithi002 ceph-mon[71816]: osdmap e5: 1 total, 0 up, 1 in 2022-05-13T23:08:22.574 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:08:22 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2022-05-13T23:08:22.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:08:22 smithi115 ceph-mon[78593]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-05-13T23:08:22.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:08:22 smithi115 ceph-mon[78593]: Standby manager daemon z started 2022-05-13T23:08:22.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:08:22 smithi115 ceph-mon[78593]: from='mgr.? 172.21.15.115:0/3069476012' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2022-05-13T23:08:22.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:08:22 smithi115 ceph-mon[78593]: from='mgr.? 172.21.15.115:0/3069476012' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2022-05-13T23:08:22.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:08:22 smithi115 ceph-mon[78593]: from='mgr.? 172.21.15.115:0/3069476012' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2022-05-13T23:08:22.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:08:22 smithi115 ceph-mon[78593]: from='mgr.? 172.21.15.115:0/3069476012' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2022-05-13T23:08:22.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:08:22 smithi115 ceph-mon[78593]: from='client.? 172.21.15.2:0/2726198310' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "11e700e8-f3e8-4903-b52c-db354b4395bf"}]: dispatch 2022-05-13T23:08:22.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:08:22 smithi115 ceph-mon[78593]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "11e700e8-f3e8-4903-b52c-db354b4395bf"}]: dispatch 2022-05-13T23:08:22.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:08:22 smithi115 ceph-mon[78593]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "11e700e8-f3e8-4903-b52c-db354b4395bf"}]': finished 2022-05-13T23:08:22.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:08:22 smithi115 ceph-mon[78593]: osdmap e5: 1 total, 0 up, 1 in 2022-05-13T23:08:22.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:08:22 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2022-05-13T23:08:23.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:08:23 smithi047 ceph-mon[77147]: mgrmap e15: x(active, since 77s), standbys: y, z 2022-05-13T23:08:23.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:08:23 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2022-05-13T23:08:23.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:08:23 smithi047 ceph-mon[77147]: from='client.? 172.21.15.2:0/2677886083' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2022-05-13T23:08:23.572 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:08:23 smithi002 ceph-mon[71816]: mgrmap e15: x(active, since 77s), standbys: y, z 2022-05-13T23:08:23.572 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:08:23 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2022-05-13T23:08:23.573 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:08:23 smithi002 ceph-mon[71816]: from='client.? 172.21.15.2:0/2677886083' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2022-05-13T23:08:23.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:08:23 smithi115 ceph-mon[78593]: mgrmap e15: x(active, since 77s), standbys: y, z 2022-05-13T23:08:23.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:08:23 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2022-05-13T23:08:23.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:08:23 smithi115 ceph-mon[78593]: from='client.? 172.21.15.2:0/2677886083' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2022-05-13T23:08:24.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:08:24 smithi047 ceph-mon[77147]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-05-13T23:08:24.572 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:08:24 smithi002 ceph-mon[71816]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-05-13T23:08:24.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:08:24 smithi115 ceph-mon[78593]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-05-13T23:08:26.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:08:26 smithi047 ceph-mon[77147]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-05-13T23:08:26.567 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:08:26 smithi002 ceph-mon[71816]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-05-13T23:08:26.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:08:26 smithi115 ceph-mon[78593]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-05-13T23:08:28.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:08:28 smithi047 ceph-mon[77147]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-05-13T23:08:28.546 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:08:28 smithi002 ceph-mon[71816]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-05-13T23:08:28.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:08:28 smithi115 ceph-mon[78593]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-05-13T23:08:30.378 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:08:30 smithi002 ceph-mon[71816]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-05-13T23:08:30.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:08:30 smithi047 ceph-mon[77147]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-05-13T23:08:30.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:08:30 smithi115 ceph-mon[78593]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-05-13T23:08:31.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:08:31 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2022-05-13T23:08:31.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:08:31 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-13T23:08:31.572 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:08:31 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2022-05-13T23:08:31.572 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:08:31 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-13T23:08:31.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:08:31 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2022-05-13T23:08:31.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:08:31 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-13T23:08:32.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:08:32 smithi047 ceph-mon[77147]: Deploying daemon osd.0 on smithi002 2022-05-13T23:08:32.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:08:32 smithi047 ceph-mon[77147]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-05-13T23:08:32.572 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:08:32 smithi002 ceph-mon[71816]: Deploying daemon osd.0 on smithi002 2022-05-13T23:08:32.573 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:08:32 smithi002 ceph-mon[71816]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-05-13T23:08:32.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:08:32 smithi115 ceph-mon[78593]: Deploying daemon osd.0 on smithi002 2022-05-13T23:08:32.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:08:32 smithi115 ceph-mon[78593]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-05-13T23:08:34.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:08:34 smithi047 ceph-mon[77147]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-05-13T23:08:34.572 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:08:34 smithi002 ceph-mon[71816]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-05-13T23:08:34.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:08:34 smithi115 ceph-mon[78593]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-05-13T23:08:36.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:08:36 smithi047 ceph-mon[77147]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-05-13T23:08:36.573 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:08:36 smithi002 ceph-mon[71816]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-05-13T23:08:36.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:08:36 smithi115 ceph-mon[78593]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-05-13T23:08:38.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:08:38 smithi047 ceph-mon[77147]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-05-13T23:08:38.572 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:08:38 smithi002 ceph-mon[71816]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-05-13T23:08:38.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:08:38 smithi115 ceph-mon[78593]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-05-13T23:08:39.974 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:08:39 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:08:39.974 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:08:39 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-13T23:08:39.975 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:08:39 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-13T23:08:39.975 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:08:39 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-13T23:08:39.975 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:08:39 smithi002 ceph-mon[71816]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-05-13T23:08:40.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:08:39 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:08:40.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:08:39 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-13T23:08:40.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:08:39 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-13T23:08:40.045 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:08:39 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-13T23:08:40.045 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:08:39 smithi047 ceph-mon[77147]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-05-13T23:08:40.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:08:39 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:08:40.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:08:39 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-13T23:08:40.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:08:39 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-13T23:08:40.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:08:39 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-13T23:08:40.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:08:39 smithi115 ceph-mon[78593]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-05-13T23:08:42.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:08:42 smithi047 ceph-mon[77147]: pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-05-13T23:08:42.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:08:42 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:08:42.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:08:42 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:08:42.548 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:08:42 smithi002 ceph-mon[71816]: pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-05-13T23:08:42.548 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:08:42 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:08:42.548 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:08:42 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:08:42.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:08:42 smithi115 ceph-mon[78593]: pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-05-13T23:08:42.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:08:42 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:08:42.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:08:42 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:08:43.120 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:08:43.118+0000 7f33e3fff700 1 -- 172.21.15.2:0/990629422 <== mgr.14148 v2:172.21.15.2:6800/2788339627 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f33fc02a530 con 0x7f33e4034e90 2022-05-13T23:08:43.120 INFO:teuthology.orchestra.run.smithi002.stdout:Created osd(s) 0 on host 'smithi002' 2022-05-13T23:08:43.123 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:08:43.121+0000 7f3400f67700 1 -- 172.21.15.2:0/990629422 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f33e4034e90 msgr2=0x7f33e4037340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:08:43.123 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:08:43.121+0000 7f3400f67700 1 --2- 172.21.15.2:0/990629422 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f33e4034e90 0x7f33e4037340 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f33ec0041d0 tx=0x7f33ec00b040 comp rx=0 tx=0).stop 2022-05-13T23:08:43.124 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:08:43.121+0000 7f3400f67700 1 -- 172.21.15.2:0/990629422 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f33fc0ff260 msgr2=0x7f33fc1158d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:08:43.124 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:08:43.121+0000 7f3400f67700 1 --2- 172.21.15.2:0/990629422 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f33fc0ff260 0x7f33fc1158d0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f33f4015040 tx=0x7f33f4004d10 comp rx=0 tx=0).stop 2022-05-13T23:08:43.125 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:08:43.121+0000 7f3400f67700 1 -- 172.21.15.2:0/990629422 shutdown_connections 2022-05-13T23:08:43.125 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:08:43.121+0000 7f3400f67700 1 --2- 172.21.15.2:0/990629422 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f33fc0ff260 0x7f33fc1158d0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:08:43.126 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:08:43.121+0000 7f3400f67700 1 --2- 172.21.15.2:0/990629422 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f33fc0fd630 0x7f33fc10e300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:08:43.126 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:08:43.121+0000 7f3400f67700 1 --2- 172.21.15.2:0/990629422 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f33e4034e90 0x7f33e4037340 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:08:43.126 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:08:43.121+0000 7f3400f67700 1 --2- 172.21.15.2:0/990629422 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f33fc0fe8d0 0x7f33fc10e840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:08:43.127 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:08:43.121+0000 7f3400f67700 1 -- 172.21.15.2:0/990629422 >> 172.21.15.2:0/990629422 conn(0x7f33fc0f8e70 msgr2=0x7f33fc101d90 unknown :-1 s=STATE_NONE l=0).mark_down 2022-05-13T23:08:43.128 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:08:43.121+0000 7f3400f67700 1 -- 172.21.15.2:0/990629422 shutdown_connections 2022-05-13T23:08:43.128 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:08:43.121+0000 7f3400f67700 1 -- 172.21.15.2:0/990629422 wait complete. 2022-05-13T23:08:43.322 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:08:43 smithi002 ceph-mon[71816]: from='osd.0 [v2:172.21.15.2:6802/1174843704,v1:172.21.15.2:6803/1174843704]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2022-05-13T23:08:43.323 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:08:43 smithi002 ceph-mon[71816]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2022-05-13T23:08:43.323 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:08:43 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:08:43.323 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:08:43 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-13T23:08:43.323 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:08:43 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-13T23:08:43.324 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:08:43 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-13T23:08:43.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:08:43 smithi047 ceph-mon[77147]: from='osd.0 [v2:172.21.15.2:6802/1174843704,v1:172.21.15.2:6803/1174843704]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2022-05-13T23:08:43.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:08:43 smithi047 ceph-mon[77147]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2022-05-13T23:08:43.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:08:43 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:08:43.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:08:43 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-13T23:08:43.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:08:43 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-13T23:08:43.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:08:43 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-13T23:08:43.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:08:43 smithi115 ceph-mon[78593]: from='osd.0 [v2:172.21.15.2:6802/1174843704,v1:172.21.15.2:6803/1174843704]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2022-05-13T23:08:43.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:08:43 smithi115 ceph-mon[78593]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2022-05-13T23:08:43.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:08:43 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:08:43.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:08:43 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-13T23:08:43.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:08:43 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-13T23:08:43.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:08:43 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-13T23:08:43.715 DEBUG:teuthology.orchestra.run.smithi002:osd.0> sudo journalctl -f -n 0 -u ceph-16cf5f88-d311-11ec-be40-001a4aab830c@osd.0.service 2022-05-13T23:08:43.721 INFO:tasks.cephadm:Deploying osd.1 on smithi047 with /dev/vg_nvme/lv_4... 2022-05-13T23:08:43.722 DEBUG:teuthology.orchestra.run.smithi047:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:23b0658391ce94ca40799469eec00c28b298816c shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 16cf5f88-d311-11ec-be40-001a4aab830c -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2022-05-13T23:08:43.802 INFO:journalctl@ceph.osd.0.smithi002.stdout:-- Logs begin at Fri 2022-05-13 22:48:11 UTC. -- 2022-05-13T23:08:44.182 INFO:teuthology.orchestra.run.smithi047.stderr:Inferring config /var/lib/ceph/16cf5f88-d311-11ec-be40-001a4aab830c/mon.b/config 2022-05-13T23:08:44.502 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:08:44 smithi047 ceph-mon[77147]: pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-05-13T23:08:44.565 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:08:44 smithi047 ceph-mon[77147]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2022-05-13T23:08:44.565 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:08:44 smithi047 ceph-mon[77147]: osdmap e6: 1 total, 0 up, 1 in 2022-05-13T23:08:44.566 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:08:44 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2022-05-13T23:08:44.566 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:08:44 smithi047 ceph-mon[77147]: from='osd.0 [v2:172.21.15.2:6802/1174843704,v1:172.21.15.2:6803/1174843704]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi002", "root=default"]}]: dispatch 2022-05-13T23:08:44.566 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:08:44 smithi047 ceph-mon[77147]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi002", "root=default"]}]: dispatch 2022-05-13T23:08:44.567 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:08:44 smithi002 ceph-mon[71816]: pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-05-13T23:08:44.567 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:08:44 smithi002 ceph-mon[71816]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2022-05-13T23:08:44.568 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:08:44 smithi002 ceph-mon[71816]: osdmap e6: 1 total, 0 up, 1 in 2022-05-13T23:08:44.568 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:08:44 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2022-05-13T23:08:44.568 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:08:44 smithi002 ceph-mon[71816]: from='osd.0 [v2:172.21.15.2:6802/1174843704,v1:172.21.15.2:6803/1174843704]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi002", "root=default"]}]: dispatch 2022-05-13T23:08:44.568 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:08:44 smithi002 ceph-mon[71816]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi002", "root=default"]}]: dispatch 2022-05-13T23:08:44.569 INFO:journalctl@ceph.osd.0.smithi002.stdout:May 13 23:08:44 smithi002 conmon[85762]: 2022-05-13T23:08:44.235+0000 7f4ffb811700 -1 osd.0 0 waiting for initial osdmap 2022-05-13T23:08:44.569 INFO:journalctl@ceph.osd.0.smithi002.stdout:May 13 23:08:44 smithi002 conmon[85762]: 2022-05-13T23:08:44.241+0000 7f4ff81ab700 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2022-05-13T23:08:44.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:08:44 smithi115 ceph-mon[78593]: pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-05-13T23:08:44.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:08:44 smithi115 ceph-mon[78593]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2022-05-13T23:08:44.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:08:44 smithi115 ceph-mon[78593]: osdmap e6: 1 total, 0 up, 1 in 2022-05-13T23:08:44.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:08:44 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2022-05-13T23:08:44.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:08:44 smithi115 ceph-mon[78593]: from='osd.0 [v2:172.21.15.2:6802/1174843704,v1:172.21.15.2:6803/1174843704]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi002", "root=default"]}]: dispatch 2022-05-13T23:08:44.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:08:44 smithi115 ceph-mon[78593]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi002", "root=default"]}]: dispatch 2022-05-13T23:08:45.530 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:08:45 smithi002 ceph-mon[71816]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi002", "root=default"]}]': finished 2022-05-13T23:08:45.531 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:08:45 smithi002 ceph-mon[71816]: osdmap e7: 1 total, 0 up, 1 in 2022-05-13T23:08:45.531 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:08:45 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2022-05-13T23:08:45.532 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:08:45 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2022-05-13T23:08:45.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:08:45 smithi047 ceph-mon[77147]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi002", "root=default"]}]': finished 2022-05-13T23:08:45.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:08:45 smithi047 ceph-mon[77147]: osdmap e7: 1 total, 0 up, 1 in 2022-05-13T23:08:45.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:08:45 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2022-05-13T23:08:45.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:08:45 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2022-05-13T23:08:45.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:08:45 smithi115 ceph-mon[78593]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi002", "root=default"]}]': finished 2022-05-13T23:08:45.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:08:45 smithi115 ceph-mon[78593]: osdmap e7: 1 total, 0 up, 1 in 2022-05-13T23:08:45.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:08:45 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2022-05-13T23:08:45.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:08:45 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2022-05-13T23:08:45.772 INFO:teuthology.orchestra.run.smithi047.stderr:--> Zapping: /dev/vg_nvme/lv_4 2022-05-13T23:08:45.772 INFO:teuthology.orchestra.run.smithi047.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2022-05-13T23:08:45.772 INFO:teuthology.orchestra.run.smithi047.stderr: stderr: 10+0 records in 2022-05-13T23:08:45.772 INFO:teuthology.orchestra.run.smithi047.stderr:10+0 records out 2022-05-13T23:08:45.773 INFO:teuthology.orchestra.run.smithi047.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.00804045 s, 1.3 GB/s 2022-05-13T23:08:45.773 INFO:teuthology.orchestra.run.smithi047.stderr:--> Zapping successful for: 2022-05-13T23:08:46.188 DEBUG:teuthology.orchestra.run.smithi047:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:23b0658391ce94ca40799469eec00c28b298816c shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 16cf5f88-d311-11ec-be40-001a4aab830c -- ceph orch daemon add osd smithi047:vg_nvme/lv_4 2022-05-13T23:08:46.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:08:46 smithi047 ceph-mon[77147]: purged_snaps scrub starts 2022-05-13T23:08:46.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:08:46 smithi047 ceph-mon[77147]: purged_snaps scrub ok 2022-05-13T23:08:46.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:08:46 smithi047 ceph-mon[77147]: pgmap v47: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-05-13T23:08:46.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:08:46 smithi047 ceph-mon[77147]: osd.0 [v2:172.21.15.2:6802/1174843704,v1:172.21.15.2:6803/1174843704] boot 2022-05-13T23:08:46.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:08:46 smithi047 ceph-mon[77147]: osdmap e8: 1 total, 1 up, 1 in 2022-05-13T23:08:46.545 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:08:46 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2022-05-13T23:08:46.572 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:08:46 smithi002 ceph-mon[71816]: purged_snaps scrub starts 2022-05-13T23:08:46.573 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:08:46 smithi002 ceph-mon[71816]: purged_snaps scrub ok 2022-05-13T23:08:46.573 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:08:46 smithi002 ceph-mon[71816]: pgmap v47: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-05-13T23:08:46.573 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:08:46 smithi002 ceph-mon[71816]: osd.0 [v2:172.21.15.2:6802/1174843704,v1:172.21.15.2:6803/1174843704] boot 2022-05-13T23:08:46.574 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:08:46 smithi002 ceph-mon[71816]: osdmap e8: 1 total, 1 up, 1 in 2022-05-13T23:08:46.574 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:08:46 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2022-05-13T23:08:46.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:08:46 smithi115 ceph-mon[78593]: purged_snaps scrub starts 2022-05-13T23:08:46.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:08:46 smithi115 ceph-mon[78593]: purged_snaps scrub ok 2022-05-13T23:08:46.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:08:46 smithi115 ceph-mon[78593]: pgmap v47: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-05-13T23:08:46.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:08:46 smithi115 ceph-mon[78593]: osd.0 [v2:172.21.15.2:6802/1174843704,v1:172.21.15.2:6803/1174843704] boot 2022-05-13T23:08:46.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:08:46 smithi115 ceph-mon[78593]: osdmap e8: 1 total, 1 up, 1 in 2022-05-13T23:08:46.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:08:46 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2022-05-13T23:08:46.639 INFO:teuthology.orchestra.run.smithi047.stderr:Inferring config /var/lib/ceph/16cf5f88-d311-11ec-be40-001a4aab830c/mon.b/config 2022-05-13T23:08:47.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:08:47 smithi047 ceph-mon[77147]: osdmap e9: 1 total, 1 up, 1 in 2022-05-13T23:08:47.572 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:08:47 smithi002 ceph-mon[71816]: osdmap e9: 1 total, 1 up, 1 in 2022-05-13T23:08:47.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:08:47 smithi115 ceph-mon[78593]: osdmap e9: 1 total, 1 up, 1 in 2022-05-13T23:08:47.921 INFO:teuthology.orchestra.run.smithi047.stderr:2022-05-13T23:08:47.917+0000 7f4fb4062700 1 -- 172.21.15.47:0/2439497038 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4fac0fbbb0 msgr2=0x7f4fac0fbfc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:08:47.922 INFO:teuthology.orchestra.run.smithi047.stderr:2022-05-13T23:08:47.917+0000 7f4fb4062700 1 --2- 172.21.15.47:0/2439497038 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4fac0fbbb0 0x7f4fac0fbfc0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f4f9c004870 tx=0x7f4f9c009cb0 comp rx=0 tx=0).stop 2022-05-13T23:08:47.922 INFO:teuthology.orchestra.run.smithi047.stderr:2022-05-13T23:08:47.917+0000 7f4fb4062700 1 -- 172.21.15.47:0/2439497038 shutdown_connections 2022-05-13T23:08:47.922 INFO:teuthology.orchestra.run.smithi047.stderr:2022-05-13T23:08:47.917+0000 7f4fb4062700 1 --2- 172.21.15.47:0/2439497038 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4fac0fac10 0x7f4fac102a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:08:47.922 INFO:teuthology.orchestra.run.smithi047.stderr:2022-05-13T23:08:47.917+0000 7f4fb4062700 1 --2- 172.21.15.47:0/2439497038 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4fac0fbbb0 0x7f4fac0fbfc0 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:08:47.923 INFO:teuthology.orchestra.run.smithi047.stderr:2022-05-13T23:08:47.917+0000 7f4fb4062700 1 --2- 172.21.15.47:0/2439497038 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f4fac0fa260 0x7f4fac0fa6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:08:47.923 INFO:teuthology.orchestra.run.smithi047.stderr:2022-05-13T23:08:47.917+0000 7f4fb4062700 1 -- 172.21.15.47:0/2439497038 >> 172.21.15.47:0/2439497038 conn(0x7f4fac0f5cf0 msgr2=0x7f4fac0f8120 unknown :-1 s=STATE_NONE l=0).mark_down 2022-05-13T23:08:47.923 INFO:teuthology.orchestra.run.smithi047.stderr:2022-05-13T23:08:47.917+0000 7f4fb4062700 1 -- 172.21.15.47:0/2439497038 shutdown_connections 2022-05-13T23:08:47.923 INFO:teuthology.orchestra.run.smithi047.stderr:2022-05-13T23:08:47.917+0000 7f4fb4062700 1 -- 172.21.15.47:0/2439497038 wait complete. 2022-05-13T23:08:47.924 INFO:teuthology.orchestra.run.smithi047.stderr:2022-05-13T23:08:47.917+0000 7f4fb4062700 1 Processor -- start 2022-05-13T23:08:47.924 INFO:teuthology.orchestra.run.smithi047.stderr:2022-05-13T23:08:47.917+0000 7f4fb4062700 1 -- start start 2022-05-13T23:08:47.924 INFO:teuthology.orchestra.run.smithi047.stderr:2022-05-13T23:08:47.917+0000 7f4fb4062700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4fac0fa260 0x7f4fac10e630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:08:47.925 INFO:teuthology.orchestra.run.smithi047.stderr:2022-05-13T23:08:47.917+0000 7f4fb4062700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4fac0fac10 0x7f4fac10eb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:08:47.925 INFO:teuthology.orchestra.run.smithi047.stderr:2022-05-13T23:08:47.917+0000 7f4fb4062700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f4fac0fbbb0 0x7f4fac115c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:08:47.925 INFO:teuthology.orchestra.run.smithi047.stderr:2022-05-13T23:08:47.917+0000 7f4fb4062700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4fac1161d0 con 0x7f4fac0fa260 2022-05-13T23:08:47.925 INFO:teuthology.orchestra.run.smithi047.stderr:2022-05-13T23:08:47.917+0000 7f4fb4062700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4fac116340 con 0x7f4fac0fac10 2022-05-13T23:08:47.926 INFO:teuthology.orchestra.run.smithi047.stderr:2022-05-13T23:08:47.917+0000 7f4fb4062700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4fac116500 con 0x7f4fac0fbbb0 2022-05-13T23:08:47.926 INFO:teuthology.orchestra.run.smithi047.stderr:2022-05-13T23:08:47.918+0000 7f4fb15fd700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4fac0fac10 0x7f4fac10eb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:08:47.926 INFO:teuthology.orchestra.run.smithi047.stderr:2022-05-13T23:08:47.918+0000 7f4fb15fd700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4fac0fac10 0x7f4fac10eb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:42010/0 (socket says 172.21.15.47:42010) 2022-05-13T23:08:47.926 INFO:teuthology.orchestra.run.smithi047.stderr:2022-05-13T23:08:47.918+0000 7f4fb15fd700 1 -- 172.21.15.47:0/1842943162 learned_addr learned my addr 172.21.15.47:0/1842943162 (peer_addr_for_me v2:172.21.15.47:0/0) 2022-05-13T23:08:47.927 INFO:teuthology.orchestra.run.smithi047.stderr:2022-05-13T23:08:47.918+0000 7f4fb25ff700 1 --2- 172.21.15.47:0/1842943162 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f4fac0fbbb0 0x7f4fac115c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:08:47.927 INFO:teuthology.orchestra.run.smithi047.stderr:2022-05-13T23:08:47.918+0000 7f4fb15fd700 1 -- 172.21.15.47:0/1842943162 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f4fac0fbbb0 msgr2=0x7f4fac115c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:08:47.927 INFO:teuthology.orchestra.run.smithi047.stderr:2022-05-13T23:08:47.918+0000 7f4fb15fd700 1 --2- 172.21.15.47:0/1842943162 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f4fac0fbbb0 0x7f4fac115c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:08:47.927 INFO:teuthology.orchestra.run.smithi047.stderr:2022-05-13T23:08:47.918+0000 7f4fb15fd700 1 -- 172.21.15.47:0/1842943162 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4fac0fa260 msgr2=0x7f4fac10e630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:08:47.928 INFO:teuthology.orchestra.run.smithi047.stderr:2022-05-13T23:08:47.918+0000 7f4fb15fd700 1 --2- 172.21.15.47:0/1842943162 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4fac0fa260 0x7f4fac10e630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:08:47.928 INFO:teuthology.orchestra.run.smithi047.stderr:2022-05-13T23:08:47.918+0000 7f4fb15fd700 1 -- 172.21.15.47:0/1842943162 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f9c005040 con 0x7f4fac0fac10 2022-05-13T23:08:47.928 INFO:teuthology.orchestra.run.smithi047.stderr:2022-05-13T23:08:47.918+0000 7f4fb15fd700 1 --2- 172.21.15.47:0/1842943162 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4fac0fac10 0x7f4fac10eb70 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f4fa80041d0 tx=0x7f4fa8009f60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-05-13T23:08:47.928 INFO:teuthology.orchestra.run.smithi047.stderr:2022-05-13T23:08:47.918+0000 7f4fa27fc700 1 -- 172.21.15.47:0/1842943162 <== mon.1 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fa8007b40 con 0x7f4fac0fac10 2022-05-13T23:08:47.929 INFO:teuthology.orchestra.run.smithi047.stderr:2022-05-13T23:08:47.918+0000 7f4fa27fc700 1 -- 172.21.15.47:0/1842943162 <== mon.1 v2:172.21.15.47:3300/0 2 ==== config(26 keys) v1 ==== 1087+0+0 (secure 0 0 0) 0x7f4fa800f040 con 0x7f4fac0fac10 2022-05-13T23:08:47.929 INFO:teuthology.orchestra.run.smithi047.stderr:2022-05-13T23:08:47.918+0000 7f4fb4062700 1 -- 172.21.15.47:0/1842943162 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4fac1167e0 con 0x7f4fac0fac10 2022-05-13T23:08:47.929 INFO:teuthology.orchestra.run.smithi047.stderr:2022-05-13T23:08:47.918+0000 7f4fa27fc700 1 -- 172.21.15.47:0/1842943162 <== mon.1 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fa8013760 con 0x7f4fac0fac10 2022-05-13T23:08:47.930 INFO:teuthology.orchestra.run.smithi047.stderr:2022-05-13T23:08:47.918+0000 7f4fb4062700 1 -- 172.21.15.47:0/1842943162 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4fac116da0 con 0x7f4fac0fac10 2022-05-13T23:08:47.930 INFO:teuthology.orchestra.run.smithi047.stderr:2022-05-13T23:08:47.919+0000 7f4fb4062700 1 -- 172.21.15.47:0/1842943162 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4fac105e80 con 0x7f4fac0fac10 2022-05-13T23:08:47.931 INFO:teuthology.orchestra.run.smithi047.stderr:2022-05-13T23:08:47.920+0000 7f4fa27fc700 1 -- 172.21.15.47:0/1842943162 <== mon.1 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 15) v1 ==== 125096+0+0 (secure 0 0 0) 0x7f4fa80090d0 con 0x7f4fac0fac10 2022-05-13T23:08:47.932 INFO:teuthology.orchestra.run.smithi047.stderr:2022-05-13T23:08:47.921+0000 7f4fa27fc700 1 --2- 172.21.15.47:0/1842943162 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f4f98096d10 0x7f4f980991c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:08:47.932 INFO:teuthology.orchestra.run.smithi047.stderr:2022-05-13T23:08:47.921+0000 7f4fa27fc700 1 -- 172.21.15.47:0/1842943162 <== mon.1 v2:172.21.15.47:3300/0 5 ==== osd_map(9..9 src has 1..9) v4 ==== 1763+0+0 (secure 0 0 0) 0x7f4fa80bdd20 con 0x7f4fac0fac10 2022-05-13T23:08:47.932 INFO:teuthology.orchestra.run.smithi047.stderr:2022-05-13T23:08:47.921+0000 7f4fb1dfe700 1 --2- 172.21.15.47:0/1842943162 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f4f98096d10 0x7f4f980991c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:08:47.932 INFO:teuthology.orchestra.run.smithi047.stderr:2022-05-13T23:08:47.922+0000 7f4fb1dfe700 1 --2- 172.21.15.47:0/1842943162 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f4f98096d10 0x7f4f980991c0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f4f9c004870 tx=0x7f4f9c009cb0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-05-13T23:08:47.933 INFO:teuthology.orchestra.run.smithi047.stderr:2022-05-13T23:08:47.925+0000 7f4fa27fc700 1 -- 172.21.15.47:0/1842943162 <== mon.1 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+165215 (secure 0 0 0) 0x7f4fa808c9e0 con 0x7f4fac0fac10 2022-05-13T23:08:48.116 INFO:teuthology.orchestra.run.smithi047.stderr:2022-05-13T23:08:48.114+0000 7f4fb4062700 1 -- 172.21.15.47:0/1842943162 --> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi047:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) v1 -- 0x7f4fac065140 con 0x7f4f98096d10 2022-05-13T23:08:48.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:08:48 smithi047 ceph-mon[77147]: pgmap v50: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2022-05-13T23:08:48.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:08:48 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:08:48.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:08:48 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2022-05-13T23:08:48.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:08:48 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:08:48.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:08:48 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:08:48.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:08:48 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2022-05-13T23:08:48.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:08:48 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2022-05-13T23:08:48.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:08:48 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-13T23:08:48.572 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:08:48 smithi002 ceph-mon[71816]: pgmap v50: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2022-05-13T23:08:48.573 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:08:48 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:08:48.573 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:08:48 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2022-05-13T23:08:48.573 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:08:48 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:08:48.574 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:08:48 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:08:48.574 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:08:48 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2022-05-13T23:08:48.574 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:08:48 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2022-05-13T23:08:48.574 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:08:48 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-13T23:08:48.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:08:48 smithi115 ceph-mon[78593]: pgmap v50: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2022-05-13T23:08:48.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:08:48 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:08:48.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:08:48 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2022-05-13T23:08:48.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:08:48 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:08:48.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:08:48 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:08:48.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:08:48 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2022-05-13T23:08:48.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:08:48 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2022-05-13T23:08:48.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:08:48 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-13T23:08:49.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:08:49 smithi047 ceph-mon[77147]: Detected new or changed devices on smithi002 2022-05-13T23:08:49.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:08:49 smithi047 ceph-mon[77147]: Adjusting osd_memory_target on smithi002 to 17277M 2022-05-13T23:08:49.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:08:49 smithi047 ceph-mon[77147]: from='client.24154 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi047:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2022-05-13T23:08:49.572 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:08:49 smithi002 ceph-mon[71816]: Detected new or changed devices on smithi002 2022-05-13T23:08:49.573 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:08:49 smithi002 ceph-mon[71816]: Adjusting osd_memory_target on smithi002 to 17277M 2022-05-13T23:08:49.573 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:08:49 smithi002 ceph-mon[71816]: from='client.24154 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi047:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2022-05-13T23:08:49.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:08:49 smithi115 ceph-mon[78593]: Detected new or changed devices on smithi002 2022-05-13T23:08:49.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:08:49 smithi115 ceph-mon[78593]: Adjusting osd_memory_target on smithi002 to 17277M 2022-05-13T23:08:49.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:08:49 smithi115 ceph-mon[78593]: from='client.24154 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi047:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2022-05-13T23:08:50.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:08:50 smithi047 ceph-mon[77147]: pgmap v51: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2022-05-13T23:08:50.572 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:08:50 smithi002 ceph-mon[71816]: pgmap v51: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2022-05-13T23:08:50.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:08:50 smithi115 ceph-mon[78593]: pgmap v51: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2022-05-13T23:08:51.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:08:51 smithi047 ceph-mon[77147]: from='client.? 172.21.15.47:0/3376261634' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "eabfa3cf-f2a3-4995-a7de-f6d8ba02ee6f"}]: dispatch 2022-05-13T23:08:51.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:08:51 smithi047 ceph-mon[77147]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "eabfa3cf-f2a3-4995-a7de-f6d8ba02ee6f"}]: dispatch 2022-05-13T23:08:51.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:08:51 smithi047 ceph-mon[77147]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "eabfa3cf-f2a3-4995-a7de-f6d8ba02ee6f"}]': finished 2022-05-13T23:08:51.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:08:51 smithi047 ceph-mon[77147]: osdmap e10: 2 total, 1 up, 2 in 2022-05-13T23:08:51.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:08:51 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2022-05-13T23:08:51.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:08:51 smithi047 ceph-mon[77147]: from='client.? 172.21.15.47:0/980315965' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2022-05-13T23:08:51.572 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:08:51 smithi002 ceph-mon[71816]: from='client.? 172.21.15.47:0/3376261634' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "eabfa3cf-f2a3-4995-a7de-f6d8ba02ee6f"}]: dispatch 2022-05-13T23:08:51.573 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:08:51 smithi002 ceph-mon[71816]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "eabfa3cf-f2a3-4995-a7de-f6d8ba02ee6f"}]: dispatch 2022-05-13T23:08:51.574 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:08:51 smithi002 ceph-mon[71816]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "eabfa3cf-f2a3-4995-a7de-f6d8ba02ee6f"}]': finished 2022-05-13T23:08:51.574 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:08:51 smithi002 ceph-mon[71816]: osdmap e10: 2 total, 1 up, 2 in 2022-05-13T23:08:51.574 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:08:51 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2022-05-13T23:08:51.575 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:08:51 smithi002 ceph-mon[71816]: from='client.? 172.21.15.47:0/980315965' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2022-05-13T23:08:51.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:08:51 smithi115 ceph-mon[78593]: from='client.? 172.21.15.47:0/3376261634' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "eabfa3cf-f2a3-4995-a7de-f6d8ba02ee6f"}]: dispatch 2022-05-13T23:08:51.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:08:51 smithi115 ceph-mon[78593]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "eabfa3cf-f2a3-4995-a7de-f6d8ba02ee6f"}]: dispatch 2022-05-13T23:08:51.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:08:51 smithi115 ceph-mon[78593]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "eabfa3cf-f2a3-4995-a7de-f6d8ba02ee6f"}]': finished 2022-05-13T23:08:51.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:08:51 smithi115 ceph-mon[78593]: osdmap e10: 2 total, 1 up, 2 in 2022-05-13T23:08:51.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:08:51 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2022-05-13T23:08:51.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:08:51 smithi115 ceph-mon[78593]: from='client.? 172.21.15.47:0/980315965' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2022-05-13T23:08:52.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:08:52 smithi115 ceph-mon[78593]: pgmap v53: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2022-05-13T23:08:52.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:08:52 smithi047 ceph-mon[77147]: pgmap v53: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2022-05-13T23:08:52.822 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:08:52 smithi002 ceph-mon[71816]: pgmap v53: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2022-05-13T23:08:54.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:08:54 smithi115 ceph-mon[78593]: pgmap v54: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2022-05-13T23:08:54.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:08:54 smithi047 ceph-mon[77147]: pgmap v54: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2022-05-13T23:08:54.822 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:08:54 smithi002 ceph-mon[71816]: pgmap v54: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2022-05-13T23:08:56.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:08:56 smithi115 ceph-mon[78593]: pgmap v55: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2022-05-13T23:08:56.725 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:08:56 smithi047 ceph-mon[77147]: pgmap v55: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2022-05-13T23:08:56.822 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:08:56 smithi002 ceph-mon[71816]: pgmap v55: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2022-05-13T23:08:57.616 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:08:57 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2022-05-13T23:08:57.616 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:08:57 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-13T23:08:57.822 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:08:57 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2022-05-13T23:08:57.823 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:08:57 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-13T23:08:57.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:08:57 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2022-05-13T23:08:57.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:08:57 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-13T23:08:58.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:08:58 smithi047 ceph-mon[77147]: Deploying daemon osd.1 on smithi047 2022-05-13T23:08:58.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:08:58 smithi047 ceph-mon[77147]: pgmap v56: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2022-05-13T23:08:58.822 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:08:58 smithi002 ceph-mon[71816]: Deploying daemon osd.1 on smithi047 2022-05-13T23:08:58.822 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:08:58 smithi002 ceph-mon[71816]: pgmap v56: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2022-05-13T23:08:58.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:08:58 smithi115 ceph-mon[78593]: Deploying daemon osd.1 on smithi047 2022-05-13T23:08:58.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:08:58 smithi115 ceph-mon[78593]: pgmap v56: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2022-05-13T23:09:00.742 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:09:00 smithi047 ceph-mon[77147]: pgmap v57: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2022-05-13T23:09:00.822 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:09:00 smithi002 ceph-mon[71816]: pgmap v57: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2022-05-13T23:09:00.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:09:00 smithi115 ceph-mon[78593]: pgmap v57: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2022-05-13T23:09:02.644 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:09:02 smithi047 ceph-mon[77147]: pgmap v58: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2022-05-13T23:09:02.644 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:09:02 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:09:02.644 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:09:02 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-13T23:09:02.644 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:09:02 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-13T23:09:02.645 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:09:02 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-13T23:09:02.822 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:09:02 smithi002 ceph-mon[71816]: pgmap v58: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2022-05-13T23:09:02.823 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:09:02 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:09:02.823 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:09:02 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-13T23:09:02.823 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:09:02 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-13T23:09:02.823 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:09:02 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-13T23:09:02.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:09:02 smithi115 ceph-mon[78593]: pgmap v58: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2022-05-13T23:09:02.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:09:02 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:09:02.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:09:02 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-13T23:09:02.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:09:02 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-13T23:09:02.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:09:02 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-13T23:09:04.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:09:04 smithi047 ceph-mon[77147]: pgmap v59: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2022-05-13T23:09:04.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:09:04 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:09:04.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:09:04 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:09:04.659 INFO:teuthology.orchestra.run.smithi047.stdout:Created osd(s) 1 on host 'smithi047' 2022-05-13T23:09:04.659 INFO:teuthology.orchestra.run.smithi047.stderr:2022-05-13T23:09:04.657+0000 7f4fa27fc700 1 -- 172.21.15.47:0/1842943162 <== mgr.14148 v2:172.21.15.2:6800/2788339627 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f4fac065140 con 0x7f4f98096d10 2022-05-13T23:09:04.661 INFO:teuthology.orchestra.run.smithi047.stderr:2022-05-13T23:09:04.660+0000 7f4fb4062700 1 -- 172.21.15.47:0/1842943162 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f4f98096d10 msgr2=0x7f4f980991c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:09:04.662 INFO:teuthology.orchestra.run.smithi047.stderr:2022-05-13T23:09:04.660+0000 7f4fb4062700 1 --2- 172.21.15.47:0/1842943162 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f4f98096d10 0x7f4f980991c0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f4f9c004870 tx=0x7f4f9c009cb0 comp rx=0 tx=0).stop 2022-05-13T23:09:04.662 INFO:teuthology.orchestra.run.smithi047.stderr:2022-05-13T23:09:04.660+0000 7f4fb4062700 1 -- 172.21.15.47:0/1842943162 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4fac0fac10 msgr2=0x7f4fac10eb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:09:04.662 INFO:teuthology.orchestra.run.smithi047.stderr:2022-05-13T23:09:04.660+0000 7f4fb4062700 1 --2- 172.21.15.47:0/1842943162 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4fac0fac10 0x7f4fac10eb70 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f4fa80041d0 tx=0x7f4fa8009f60 comp rx=0 tx=0).stop 2022-05-13T23:09:04.662 INFO:teuthology.orchestra.run.smithi047.stderr:2022-05-13T23:09:04.660+0000 7f4fb4062700 1 -- 172.21.15.47:0/1842943162 shutdown_connections 2022-05-13T23:09:04.663 INFO:teuthology.orchestra.run.smithi047.stderr:2022-05-13T23:09:04.660+0000 7f4fb4062700 1 --2- 172.21.15.47:0/1842943162 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f4fac0fbbb0 0x7f4fac115c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:09:04.663 INFO:teuthology.orchestra.run.smithi047.stderr:2022-05-13T23:09:04.660+0000 7f4fb4062700 1 --2- 172.21.15.47:0/1842943162 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f4f98096d10 0x7f4f980991c0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:09:04.663 INFO:teuthology.orchestra.run.smithi047.stderr:2022-05-13T23:09:04.660+0000 7f4fb4062700 1 --2- 172.21.15.47:0/1842943162 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4fac0fa260 0x7f4fac10e630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:09:04.663 INFO:teuthology.orchestra.run.smithi047.stderr:2022-05-13T23:09:04.660+0000 7f4fb4062700 1 --2- 172.21.15.47:0/1842943162 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4fac0fac10 0x7f4fac10eb70 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:09:04.664 INFO:teuthology.orchestra.run.smithi047.stderr:2022-05-13T23:09:04.660+0000 7f4fb4062700 1 -- 172.21.15.47:0/1842943162 >> 172.21.15.47:0/1842943162 conn(0x7f4fac0f5cf0 msgr2=0x7f4fac064a30 unknown :-1 s=STATE_NONE l=0).mark_down 2022-05-13T23:09:04.664 INFO:teuthology.orchestra.run.smithi047.stderr:2022-05-13T23:09:04.660+0000 7f4fb4062700 1 -- 172.21.15.47:0/1842943162 shutdown_connections 2022-05-13T23:09:04.664 INFO:teuthology.orchestra.run.smithi047.stderr:2022-05-13T23:09:04.660+0000 7f4fb4062700 1 -- 172.21.15.47:0/1842943162 wait complete. 2022-05-13T23:09:04.822 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:09:04 smithi002 ceph-mon[71816]: pgmap v59: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2022-05-13T23:09:04.823 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:09:04 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:09:04.823 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:09:04 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:09:04.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:09:04 smithi115 ceph-mon[78593]: pgmap v59: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2022-05-13T23:09:04.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:09:04 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:09:04.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:09:04 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:09:05.173 DEBUG:teuthology.orchestra.run.smithi047:osd.1> sudo journalctl -f -n 0 -u ceph-16cf5f88-d311-11ec-be40-001a4aab830c@osd.1.service 2022-05-13T23:09:05.176 INFO:tasks.cephadm:Deploying osd.2 on smithi115 with /dev/vg_nvme/lv_4... 2022-05-13T23:09:05.177 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:23b0658391ce94ca40799469eec00c28b298816c shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 16cf5f88-d311-11ec-be40-001a4aab830c -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2022-05-13T23:09:05.202 INFO:journalctl@ceph.osd.1.smithi047.stdout:-- Logs begin at Fri 2022-05-13 22:44:47 UTC. -- 2022-05-13T23:09:05.628 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/16cf5f88-d311-11ec-be40-001a4aab830c/mon.c/config 2022-05-13T23:09:05.791 INFO:journalctl@ceph.osd.1.smithi047.stdout:May 13 23:09:05 smithi047 conmon[82670]: 2022-05-13T23:09:05.567+0000 7f9bdbf9c3c0 -1 osd.1 0 log_to_monitors true 2022-05-13T23:09:05.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:09:05 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:09:05.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:09:05 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-13T23:09:05.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:09:05 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-13T23:09:05.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:09:05 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-13T23:09:05.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:09:05 smithi047 ceph-mon[77147]: pgmap v60: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2022-05-13T23:09:05.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:09:05 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-05-13T23:09:05.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:09:05 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-05-13T23:09:05.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:09:05 smithi047 ceph-mon[77147]: from='osd.1 [v2:172.21.15.47:6800/3853826948,v1:172.21.15.47:6801/3853826948]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2022-05-13T23:09:05.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:09:05 smithi047 ceph-mon[77147]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2022-05-13T23:09:05.914 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:09:05 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:09:05.914 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:09:05 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-13T23:09:05.915 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:09:05 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-13T23:09:05.915 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:09:05 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-13T23:09:05.915 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:09:05 smithi115 ceph-mon[78593]: pgmap v60: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2022-05-13T23:09:05.915 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:09:05 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-05-13T23:09:05.915 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:09:05 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-05-13T23:09:05.916 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:09:05 smithi115 ceph-mon[78593]: from='osd.1 [v2:172.21.15.47:6800/3853826948,v1:172.21.15.47:6801/3853826948]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2022-05-13T23:09:05.916 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:09:05 smithi115 ceph-mon[78593]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2022-05-13T23:09:06.072 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:09:05 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:09:06.073 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:09:05 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-13T23:09:06.073 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:09:05 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-13T23:09:06.073 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:09:05 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-13T23:09:06.073 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:09:05 smithi002 ceph-mon[71816]: pgmap v60: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2022-05-13T23:09:06.074 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:09:05 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-05-13T23:09:06.074 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:09:05 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-05-13T23:09:06.074 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:09:05 smithi002 ceph-mon[71816]: from='osd.1 [v2:172.21.15.47:6800/3853826948,v1:172.21.15.47:6801/3853826948]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2022-05-13T23:09:06.074 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:09:05 smithi002 ceph-mon[71816]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2022-05-13T23:09:06.739 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:09:06 smithi115 ceph-mon[78593]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2022-05-13T23:09:06.739 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:09:06 smithi115 ceph-mon[78593]: osdmap e11: 2 total, 1 up, 2 in 2022-05-13T23:09:06.739 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:09:06 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2022-05-13T23:09:06.740 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:09:06 smithi115 ceph-mon[78593]: from='osd.1 [v2:172.21.15.47:6800/3853826948,v1:172.21.15.47:6801/3853826948]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi047", "root=default"]}]: dispatch 2022-05-13T23:09:06.740 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:09:06 smithi115 ceph-mon[78593]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi047", "root=default"]}]: dispatch 2022-05-13T23:09:07.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:09:06 smithi047 ceph-mon[77147]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2022-05-13T23:09:07.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:09:06 smithi047 ceph-mon[77147]: osdmap e11: 2 total, 1 up, 2 in 2022-05-13T23:09:07.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:09:06 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2022-05-13T23:09:07.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:09:06 smithi047 ceph-mon[77147]: from='osd.1 [v2:172.21.15.47:6800/3853826948,v1:172.21.15.47:6801/3853826948]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi047", "root=default"]}]: dispatch 2022-05-13T23:09:07.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:09:06 smithi047 ceph-mon[77147]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi047", "root=default"]}]: dispatch 2022-05-13T23:09:07.043 INFO:journalctl@ceph.osd.1.smithi047.stdout:May 13 23:09:06 smithi047 conmon[82670]: 2022-05-13T23:09:06.671+0000 7f9bd29a0700 -1 osd.1 0 waiting for initial osdmap 2022-05-13T23:09:07.044 INFO:journalctl@ceph.osd.1.smithi047.stdout:May 13 23:09:06 smithi047 conmon[82670]: 2022-05-13T23:09:06.675+0000 7f9bcd336700 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2022-05-13T23:09:07.072 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:09:06 smithi002 ceph-mon[71816]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2022-05-13T23:09:07.073 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:09:06 smithi002 ceph-mon[71816]: osdmap e11: 2 total, 1 up, 2 in 2022-05-13T23:09:07.073 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:09:06 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2022-05-13T23:09:07.073 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:09:06 smithi002 ceph-mon[71816]: from='osd.1 [v2:172.21.15.47:6800/3853826948,v1:172.21.15.47:6801/3853826948]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi047", "root=default"]}]: dispatch 2022-05-13T23:09:07.073 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:09:06 smithi002 ceph-mon[71816]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi047", "root=default"]}]: dispatch 2022-05-13T23:09:07.239 INFO:teuthology.orchestra.run.smithi115.stderr:--> Zapping: /dev/vg_nvme/lv_4 2022-05-13T23:09:07.240 INFO:teuthology.orchestra.run.smithi115.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2022-05-13T23:09:07.240 INFO:teuthology.orchestra.run.smithi115.stderr: stderr: 10+0 records in 2022-05-13T23:09:07.240 INFO:teuthology.orchestra.run.smithi115.stderr:10+0 records out 2022-05-13T23:09:07.240 INFO:teuthology.orchestra.run.smithi115.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0111172 s, 943 MB/s 2022-05-13T23:09:07.241 INFO:teuthology.orchestra.run.smithi115.stderr:--> Zapping successful for: 2022-05-13T23:09:07.741 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:23b0658391ce94ca40799469eec00c28b298816c shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 16cf5f88-d311-11ec-be40-001a4aab830c -- ceph orch daemon add osd smithi115:vg_nvme/lv_4 2022-05-13T23:09:08.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:09:07 smithi047 ceph-mon[77147]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi047", "root=default"]}]': finished 2022-05-13T23:09:08.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:09:07 smithi047 ceph-mon[77147]: osdmap e12: 2 total, 1 up, 2 in 2022-05-13T23:09:08.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:09:07 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2022-05-13T23:09:08.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:09:07 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2022-05-13T23:09:08.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:09:07 smithi047 ceph-mon[77147]: pgmap v63: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2022-05-13T23:09:08.072 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:09:07 smithi002 ceph-mon[71816]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi047", "root=default"]}]': finished 2022-05-13T23:09:08.073 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:09:07 smithi002 ceph-mon[71816]: osdmap e12: 2 total, 1 up, 2 in 2022-05-13T23:09:08.073 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:09:07 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2022-05-13T23:09:08.073 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:09:07 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2022-05-13T23:09:08.073 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:09:07 smithi002 ceph-mon[71816]: pgmap v63: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2022-05-13T23:09:08.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:09:07 smithi115 ceph-mon[78593]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi047", "root=default"]}]': finished 2022-05-13T23:09:08.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:09:07 smithi115 ceph-mon[78593]: osdmap e12: 2 total, 1 up, 2 in 2022-05-13T23:09:08.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:09:07 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2022-05-13T23:09:08.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:09:07 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2022-05-13T23:09:08.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:09:07 smithi115 ceph-mon[78593]: pgmap v63: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2022-05-13T23:09:08.181 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/16cf5f88-d311-11ec-be40-001a4aab830c/mon.c/config 2022-05-13T23:09:09.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:09:08 smithi047 ceph-mon[77147]: purged_snaps scrub starts 2022-05-13T23:09:09.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:09:08 smithi047 ceph-mon[77147]: purged_snaps scrub ok 2022-05-13T23:09:09.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:09:08 smithi047 ceph-mon[77147]: osd.1 [v2:172.21.15.47:6800/3853826948,v1:172.21.15.47:6801/3853826948] boot 2022-05-13T23:09:09.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:09:08 smithi047 ceph-mon[77147]: osdmap e13: 2 total, 2 up, 2 in 2022-05-13T23:09:09.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:09:08 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2022-05-13T23:09:09.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:09:08 smithi047 ceph-mon[77147]: Detected new or changed devices on smithi047 2022-05-13T23:09:09.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:09:08 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:09:09.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:09:08 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-05-13T23:09:09.045 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:09:08 smithi047 ceph-mon[77147]: Adjusting osd_memory_target on smithi047 to 17176M 2022-05-13T23:09:09.045 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:09:08 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:09:09.045 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:09:08 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:09:09.049 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:09:08 smithi115 ceph-mon[78593]: purged_snaps scrub starts 2022-05-13T23:09:09.049 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:09:08 smithi115 ceph-mon[78593]: purged_snaps scrub ok 2022-05-13T23:09:09.049 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:09:08 smithi115 ceph-mon[78593]: osd.1 [v2:172.21.15.47:6800/3853826948,v1:172.21.15.47:6801/3853826948] boot 2022-05-13T23:09:09.050 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:09:08 smithi115 ceph-mon[78593]: osdmap e13: 2 total, 2 up, 2 in 2022-05-13T23:09:09.050 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:09:08 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2022-05-13T23:09:09.050 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:09:08 smithi115 ceph-mon[78593]: Detected new or changed devices on smithi047 2022-05-13T23:09:09.050 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:09:08 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:09:09.051 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:09:08 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-05-13T23:09:09.051 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:09:08 smithi115 ceph-mon[78593]: Adjusting osd_memory_target on smithi047 to 17176M 2022-05-13T23:09:09.051 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:09:08 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:09:09.052 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:09:08 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:09:09.072 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:09:08 smithi002 ceph-mon[71816]: purged_snaps scrub starts 2022-05-13T23:09:09.073 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:09:08 smithi002 ceph-mon[71816]: purged_snaps scrub ok 2022-05-13T23:09:09.073 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:09:08 smithi002 ceph-mon[71816]: osd.1 [v2:172.21.15.47:6800/3853826948,v1:172.21.15.47:6801/3853826948] boot 2022-05-13T23:09:09.073 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:09:08 smithi002 ceph-mon[71816]: osdmap e13: 2 total, 2 up, 2 in 2022-05-13T23:09:09.073 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:09:08 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2022-05-13T23:09:09.074 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:09:08 smithi002 ceph-mon[71816]: Detected new or changed devices on smithi047 2022-05-13T23:09:09.074 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:09:08 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:09:09.074 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:09:08 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-05-13T23:09:09.074 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:09:08 smithi002 ceph-mon[71816]: Adjusting osd_memory_target on smithi047 to 17176M 2022-05-13T23:09:09.075 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:09:08 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:09:09.075 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:09:08 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:09:09.469 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:09:09.465+0000 7f1d7f6ab700 1 -- 172.21.15.115:0/1434224220 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f1d780fb790 msgr2=0x7f1d780fbbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:09:09.469 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:09:09.465+0000 7f1d7f6ab700 1 --2- 172.21.15.115:0/1434224220 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f1d780fb790 0x7f1d780fbbe0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f1d68004870 tx=0x7f1d68009cb0 comp rx=0 tx=0).stop 2022-05-13T23:09:09.470 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:09:09.465+0000 7f1d7f6ab700 1 -- 172.21.15.115:0/1434224220 shutdown_connections 2022-05-13T23:09:09.470 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:09:09.465+0000 7f1d7f6ab700 1 --2- 172.21.15.115:0/1434224220 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1d780fc120 0x7f1d781029e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:09:09.470 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:09:09.465+0000 7f1d7f6ab700 1 --2- 172.21.15.115:0/1434224220 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1d780fa4f0 0x7f1d780fa900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:09:09.471 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:09:09.465+0000 7f1d7f6ab700 1 --2- 172.21.15.115:0/1434224220 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f1d780fb790 0x7f1d780fbbe0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:09:09.471 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:09:09.465+0000 7f1d7f6ab700 1 -- 172.21.15.115:0/1434224220 >> 172.21.15.115:0/1434224220 conn(0x7f1d780f5ce0 msgr2=0x7f1d780f8110 unknown :-1 s=STATE_NONE l=0).mark_down 2022-05-13T23:09:09.471 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:09:09.465+0000 7f1d7f6ab700 1 -- 172.21.15.115:0/1434224220 shutdown_connections 2022-05-13T23:09:09.472 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:09:09.465+0000 7f1d7f6ab700 1 -- 172.21.15.115:0/1434224220 wait complete. 2022-05-13T23:09:09.472 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:09:09.465+0000 7f1d7f6ab700 1 Processor -- start 2022-05-13T23:09:09.472 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:09:09.465+0000 7f1d7f6ab700 1 -- start start 2022-05-13T23:09:09.472 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:09:09.465+0000 7f1d7f6ab700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1d780fa4f0 0x7f1d7810b1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:09:09.472 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:09:09.465+0000 7f1d7f6ab700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f1d780fb790 0x7f1d7810b730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:09:09.473 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:09:09.465+0000 7f1d7f6ab700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1d780fc120 0x7f1d781127c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:09:09.473 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:09:09.465+0000 7f1d7f6ab700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d78112d90 con 0x7f1d780fc120 2022-05-13T23:09:09.473 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:09:09.465+0000 7f1d7f6ab700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d78112f00 con 0x7f1d780fa4f0 2022-05-13T23:09:09.473 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:09:09.465+0000 7f1d7f6ab700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d781130c0 con 0x7f1d780fb790 2022-05-13T23:09:09.474 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:09:09.465+0000 7f1d7cc46700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f1d780fb790 0x7f1d7810b730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:09:09.474 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:09:09.465+0000 7f1d7cc46700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f1d780fb790 0x7f1d7810b730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:48914/0 (socket says 172.21.15.115:48914) 2022-05-13T23:09:09.474 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:09:09.465+0000 7f1d7cc46700 1 -- 172.21.15.115:0/2881514120 learned_addr learned my addr 172.21.15.115:0/2881514120 (peer_addr_for_me v2:172.21.15.115:0/0) 2022-05-13T23:09:09.474 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:09:09.466+0000 7f1d7dc48700 1 --2- 172.21.15.115:0/2881514120 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1d780fc120 0x7f1d781127c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:09:09.475 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:09:09.466+0000 7f1d7d447700 1 --2- 172.21.15.115:0/2881514120 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1d780fa4f0 0x7f1d7810b1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:09:09.475 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:09:09.466+0000 7f1d7cc46700 1 -- 172.21.15.115:0/2881514120 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1d780fa4f0 msgr2=0x7f1d7810b1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:09:09.475 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:09:09.466+0000 7f1d7cc46700 1 --2- 172.21.15.115:0/2881514120 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1d780fa4f0 0x7f1d7810b1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:09:09.475 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:09:09.466+0000 7f1d7cc46700 1 -- 172.21.15.115:0/2881514120 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1d780fc120 msgr2=0x7f1d781127c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:09:09.475 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:09:09.466+0000 7f1d7cc46700 1 --2- 172.21.15.115:0/2881514120 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1d780fc120 0x7f1d781127c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:09:09.476 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:09:09.466+0000 7f1d7cc46700 1 -- 172.21.15.115:0/2881514120 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d68005040 con 0x7f1d780fb790 2022-05-13T23:09:09.476 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:09:09.466+0000 7f1d7dc48700 1 --2- 172.21.15.115:0/2881514120 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1d780fc120 0x7f1d781127c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2022-05-13T23:09:09.476 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:09:09.466+0000 7f1d7cc46700 1 --2- 172.21.15.115:0/2881514120 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f1d780fb790 0x7f1d7810b730 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f1d68009f20 tx=0x7f1d68009cb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-05-13T23:09:09.476 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:09:09.466+0000 7f1d6dffb700 1 -- 172.21.15.115:0/2881514120 <== mon.2 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d6800e040 con 0x7f1d780fb790 2022-05-13T23:09:09.477 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:09:09.466+0000 7f1d6dffb700 1 -- 172.21.15.115:0/2881514120 <== mon.2 v2:172.21.15.115:3300/0 2 ==== config(26 keys) v1 ==== 1087+0+0 (secure 0 0 0) 0x7f1d6801e420 con 0x7f1d780fb790 2022-05-13T23:09:09.477 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:09:09.466+0000 7f1d7f6ab700 1 -- 172.21.15.115:0/2881514120 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1d780649d0 con 0x7f1d780fb790 2022-05-13T23:09:09.477 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:09:09.466+0000 7f1d6dffb700 1 -- 172.21.15.115:0/2881514120 <== mon.2 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d68012630 con 0x7f1d780fb790 2022-05-13T23:09:09.477 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:09:09.466+0000 7f1d7f6ab700 1 -- 172.21.15.115:0/2881514120 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1d78064f40 con 0x7f1d780fb790 2022-05-13T23:09:09.478 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:09:09.467+0000 7f1d7f6ab700 1 -- 172.21.15.115:0/2881514120 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1d780658d0 con 0x7f1d780fb790 2022-05-13T23:09:09.479 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:09:09.468+0000 7f1d6dffb700 1 -- 172.21.15.115:0/2881514120 <== mon.2 v2:172.21.15.115:3300/0 4 ==== mgrmap(e 15) v1 ==== 125096+0+0 (secure 0 0 0) 0x7f1d6802e070 con 0x7f1d780fb790 2022-05-13T23:09:09.479 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:09:09.468+0000 7f1d6dffb700 1 --2- 172.21.15.115:0/2881514120 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f1d64096c90 0x7f1d64099140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:09:09.479 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:09:09.469+0000 7f1d6dffb700 1 -- 172.21.15.115:0/2881514120 <== mon.2 v2:172.21.15.115:3300/0 5 ==== osd_map(14..14 src has 1..14) v4 ==== 2305+0+0 (secure 0 0 0) 0x7f1d680c19b0 con 0x7f1d780fb790 2022-05-13T23:09:09.480 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:09:09.469+0000 7f1d7d447700 1 --2- 172.21.15.115:0/2881514120 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f1d64096c90 0x7f1d64099140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:09:09.480 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:09:09.469+0000 7f1d7d447700 1 --2- 172.21.15.115:0/2881514120 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f1d64096c90 0x7f1d64099140 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f1d780fb5f0 tx=0x7f1d74009040 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-05-13T23:09:09.480 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:09:09.473+0000 7f1d6dffb700 1 -- 172.21.15.115:0/2881514120 <== mon.2 v2:172.21.15.115:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+165215 (secure 0 0 0) 0x7f1d68090450 con 0x7f1d780fb790 2022-05-13T23:09:09.648 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:09:09.646+0000 7f1d7f6ab700 1 -- 172.21.15.115:0/2881514120 --> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi115:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) v1 -- 0x7f1d78101860 con 0x7f1d64096c90 2022-05-13T23:09:10.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:09:09 smithi047 ceph-mon[77147]: osdmap e14: 2 total, 2 up, 2 in 2022-05-13T23:09:10.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:09:09 smithi047 ceph-mon[77147]: pgmap v66: 0 pgs: ; 0 B data, 9.7 MiB used, 179 GiB / 179 GiB avail 2022-05-13T23:09:10.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:09:09 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2022-05-13T23:09:10.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:09:09 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2022-05-13T23:09:10.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:09:09 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-13T23:09:10.072 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:09:09 smithi002 ceph-mon[71816]: osdmap e14: 2 total, 2 up, 2 in 2022-05-13T23:09:10.073 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:09:09 smithi002 ceph-mon[71816]: pgmap v66: 0 pgs: ; 0 B data, 9.7 MiB used, 179 GiB / 179 GiB avail 2022-05-13T23:09:10.073 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:09:09 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2022-05-13T23:09:10.073 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:09:09 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2022-05-13T23:09:10.073 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:09:09 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-13T23:09:10.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:09:09 smithi115 ceph-mon[78593]: osdmap e14: 2 total, 2 up, 2 in 2022-05-13T23:09:10.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:09:09 smithi115 ceph-mon[78593]: pgmap v66: 0 pgs: ; 0 B data, 9.7 MiB used, 179 GiB / 179 GiB avail 2022-05-13T23:09:10.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:09:09 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2022-05-13T23:09:10.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:09:09 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2022-05-13T23:09:10.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:09:09 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-13T23:09:11.033 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:09:10 smithi115 ceph-mon[78593]: from='client.24179 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi115:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2022-05-13T23:09:11.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:09:10 smithi047 ceph-mon[77147]: from='client.24179 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi115:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2022-05-13T23:09:11.072 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:09:10 smithi002 ceph-mon[71816]: from='client.24179 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi115:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2022-05-13T23:09:12.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:09:11 smithi047 ceph-mon[77147]: pgmap v67: 0 pgs: ; 0 B data, 9.7 MiB used, 179 GiB / 179 GiB avail 2022-05-13T23:09:12.072 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:09:11 smithi002 ceph-mon[71816]: pgmap v67: 0 pgs: ; 0 B data, 9.7 MiB used, 179 GiB / 179 GiB avail 2022-05-13T23:09:12.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:09:11 smithi115 ceph-mon[78593]: pgmap v67: 0 pgs: ; 0 B data, 9.7 MiB used, 179 GiB / 179 GiB avail 2022-05-13T23:09:13.072 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:09:12 smithi002 ceph-mon[71816]: from='client.? 172.21.15.115:0/3401352297' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "153c5bd5-8200-47a7-a135-8a95b6752cd2"}]: dispatch 2022-05-13T23:09:13.073 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:09:12 smithi002 ceph-mon[71816]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "153c5bd5-8200-47a7-a135-8a95b6752cd2"}]: dispatch 2022-05-13T23:09:13.073 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:09:12 smithi002 ceph-mon[71816]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "153c5bd5-8200-47a7-a135-8a95b6752cd2"}]': finished 2022-05-13T23:09:13.073 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:09:12 smithi002 ceph-mon[71816]: osdmap e15: 3 total, 2 up, 3 in 2022-05-13T23:09:13.073 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:09:12 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2022-05-13T23:09:13.074 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:09:12 smithi002 ceph-mon[71816]: from='client.? 172.21.15.115:0/3003804891' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2022-05-13T23:09:13.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:09:12 smithi115 ceph-mon[78593]: from='client.? 172.21.15.115:0/3401352297' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "153c5bd5-8200-47a7-a135-8a95b6752cd2"}]: dispatch 2022-05-13T23:09:13.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:09:12 smithi115 ceph-mon[78593]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "153c5bd5-8200-47a7-a135-8a95b6752cd2"}]: dispatch 2022-05-13T23:09:13.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:09:12 smithi115 ceph-mon[78593]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "153c5bd5-8200-47a7-a135-8a95b6752cd2"}]': finished 2022-05-13T23:09:13.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:09:12 smithi115 ceph-mon[78593]: osdmap e15: 3 total, 2 up, 3 in 2022-05-13T23:09:13.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:09:12 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2022-05-13T23:09:13.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:09:12 smithi115 ceph-mon[78593]: from='client.? 172.21.15.115:0/3003804891' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2022-05-13T23:09:13.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:09:12 smithi047 ceph-mon[77147]: from='client.? 172.21.15.115:0/3401352297' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "153c5bd5-8200-47a7-a135-8a95b6752cd2"}]: dispatch 2022-05-13T23:09:13.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:09:12 smithi047 ceph-mon[77147]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "153c5bd5-8200-47a7-a135-8a95b6752cd2"}]: dispatch 2022-05-13T23:09:13.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:09:12 smithi047 ceph-mon[77147]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "153c5bd5-8200-47a7-a135-8a95b6752cd2"}]': finished 2022-05-13T23:09:13.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:09:12 smithi047 ceph-mon[77147]: osdmap e15: 3 total, 2 up, 3 in 2022-05-13T23:09:13.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:09:12 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2022-05-13T23:09:13.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:09:12 smithi047 ceph-mon[77147]: from='client.? 172.21.15.115:0/3003804891' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2022-05-13T23:09:14.072 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:09:13 smithi002 ceph-mon[71816]: pgmap v69: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2022-05-13T23:09:14.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:09:13 smithi115 ceph-mon[78593]: pgmap v69: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2022-05-13T23:09:14.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:09:13 smithi047 ceph-mon[77147]: pgmap v69: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2022-05-13T23:09:16.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:09:16 smithi115 ceph-mon[78593]: pgmap v70: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2022-05-13T23:09:16.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:09:16 smithi047 ceph-mon[77147]: pgmap v70: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2022-05-13T23:09:16.572 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:09:16 smithi002 ceph-mon[71816]: pgmap v70: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2022-05-13T23:09:18.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:09:18 smithi115 ceph-mon[78593]: pgmap v71: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2022-05-13T23:09:18.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:09:18 smithi047 ceph-mon[77147]: pgmap v71: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2022-05-13T23:09:18.572 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:09:18 smithi002 ceph-mon[71816]: pgmap v71: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2022-05-13T23:09:19.493 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:09:19 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2022-05-13T23:09:19.494 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:09:19 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-13T23:09:19.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:09:19 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2022-05-13T23:09:19.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:09:19 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-13T23:09:19.572 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:09:19 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2022-05-13T23:09:19.573 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:09:19 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-13T23:09:20.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:09:20 smithi047 ceph-mon[77147]: Deploying daemon osd.2 on smithi115 2022-05-13T23:09:20.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:09:20 smithi047 ceph-mon[77147]: pgmap v72: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2022-05-13T23:09:20.572 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:09:20 smithi002 ceph-mon[71816]: Deploying daemon osd.2 on smithi115 2022-05-13T23:09:20.572 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:09:20 smithi002 ceph-mon[71816]: pgmap v72: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2022-05-13T23:09:20.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:09:20 smithi115 ceph-mon[78593]: Deploying daemon osd.2 on smithi115 2022-05-13T23:09:20.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:09:20 smithi115 ceph-mon[78593]: pgmap v72: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2022-05-13T23:09:22.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:09:22 smithi047 ceph-mon[77147]: pgmap v73: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2022-05-13T23:09:22.572 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:09:22 smithi002 ceph-mon[71816]: pgmap v73: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2022-05-13T23:09:22.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:09:22 smithi115 ceph-mon[78593]: pgmap v73: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2022-05-13T23:09:24.318 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:09:24 smithi115 ceph-mon[78593]: pgmap v74: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2022-05-13T23:09:24.318 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:09:24 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:09:24.319 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:09:24 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-13T23:09:24.319 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:09:24 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-13T23:09:24.319 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:09:24 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-13T23:09:24.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:09:24 smithi047 ceph-mon[77147]: pgmap v74: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2022-05-13T23:09:24.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:09:24 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:09:24.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:09:24 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-13T23:09:24.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:09:24 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-13T23:09:24.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:09:24 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-13T23:09:24.572 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:09:24 smithi002 ceph-mon[71816]: pgmap v74: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2022-05-13T23:09:24.573 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:09:24 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:09:24.573 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:09:24 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-13T23:09:24.573 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:09:24 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-13T23:09:24.573 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:09:24 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-13T23:09:26.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:09:26 smithi115 ceph-mon[78593]: pgmap v75: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2022-05-13T23:09:26.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:09:26 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:09:26.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:09:26 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:09:26.457 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:09:26.454+0000 7f1d6dffb700 1 -- 172.21.15.115:0/2881514120 <== mgr.14148 v2:172.21.15.2:6800/2788339627 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f1d78101860 con 0x7f1d64096c90 2022-05-13T23:09:26.458 INFO:teuthology.orchestra.run.smithi115.stdout:Created osd(s) 2 on host 'smithi115' 2022-05-13T23:09:26.460 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:09:26.457+0000 7f1d7f6ab700 1 -- 172.21.15.115:0/2881514120 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f1d64096c90 msgr2=0x7f1d64099140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:09:26.460 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:09:26.457+0000 7f1d7f6ab700 1 --2- 172.21.15.115:0/2881514120 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f1d64096c90 0x7f1d64099140 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f1d780fb5f0 tx=0x7f1d74009040 comp rx=0 tx=0).stop 2022-05-13T23:09:26.460 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:09:26.457+0000 7f1d7f6ab700 1 -- 172.21.15.115:0/2881514120 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f1d780fb790 msgr2=0x7f1d7810b730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:09:26.461 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:09:26.457+0000 7f1d7f6ab700 1 --2- 172.21.15.115:0/2881514120 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f1d780fb790 0x7f1d7810b730 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f1d68009f20 tx=0x7f1d68009cb0 comp rx=0 tx=0).stop 2022-05-13T23:09:26.461 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:09:26.457+0000 7f1d7f6ab700 1 -- 172.21.15.115:0/2881514120 shutdown_connections 2022-05-13T23:09:26.462 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:09:26.457+0000 7f1d7f6ab700 1 --2- 172.21.15.115:0/2881514120 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f1d64096c90 0x7f1d64099140 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:09:26.462 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:09:26.457+0000 7f1d7f6ab700 1 --2- 172.21.15.115:0/2881514120 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1d780fc120 0x7f1d781127c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:09:26.463 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:09:26.457+0000 7f1d7f6ab700 1 --2- 172.21.15.115:0/2881514120 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1d780fa4f0 0x7f1d7810b1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:09:26.463 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:09:26.457+0000 7f1d7f6ab700 1 --2- 172.21.15.115:0/2881514120 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f1d780fb790 0x7f1d7810b730 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:09:26.463 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:09:26.457+0000 7f1d7f6ab700 1 -- 172.21.15.115:0/2881514120 >> 172.21.15.115:0/2881514120 conn(0x7f1d780f5ce0 msgr2=0x7f1d780fedd0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-05-13T23:09:26.464 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:09:26.457+0000 7f1d7f6ab700 1 -- 172.21.15.115:0/2881514120 shutdown_connections 2022-05-13T23:09:26.464 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:09:26.457+0000 7f1d7f6ab700 1 -- 172.21.15.115:0/2881514120 wait complete. 2022-05-13T23:09:26.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:09:26 smithi047 ceph-mon[77147]: pgmap v75: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2022-05-13T23:09:26.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:09:26 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:09:26.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:09:26 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:09:26.572 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:09:26 smithi002 ceph-mon[71816]: pgmap v75: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2022-05-13T23:09:26.573 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:09:26 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:09:26.573 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:09:26 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:09:27.000 DEBUG:teuthology.orchestra.run.smithi115:osd.2> sudo journalctl -f -n 0 -u ceph-16cf5f88-d311-11ec-be40-001a4aab830c@osd.2.service 2022-05-13T23:09:27.003 INFO:tasks.cephadm:Deploying osd.3 on smithi002 with /dev/vg_nvme/lv_3... 2022-05-13T23:09:27.003 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:23b0658391ce94ca40799469eec00c28b298816c shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 16cf5f88-d311-11ec-be40-001a4aab830c -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2022-05-13T23:09:27.031 INFO:journalctl@ceph.osd.2.smithi115.stdout:-- Logs begin at Fri 2022-05-13 22:45:29 UTC. -- 2022-05-13T23:09:27.677 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/16cf5f88-d311-11ec-be40-001a4aab830c/mon.a/config 2022-05-13T23:09:27.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:09:27 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:09:27.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:09:27 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-13T23:09:27.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:09:27 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-13T23:09:27.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:09:27 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-13T23:09:27.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:09:27 smithi047 ceph-mon[77147]: pgmap v76: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2022-05-13T23:09:27.818 INFO:journalctl@ceph.osd.2.smithi115.stdout:May 13 23:09:27 smithi115 conmon[84560]: 2022-05-13T23:09:27.551+0000 7ff4ac9d43c0 -1 osd.2 0 log_to_monitors true 2022-05-13T23:09:27.818 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:09:27 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:09:27.819 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:09:27 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-13T23:09:27.819 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:09:27 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-13T23:09:27.819 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:09:27 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-13T23:09:27.819 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:09:27 smithi115 ceph-mon[78593]: pgmap v76: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2022-05-13T23:09:27.822 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:09:27 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:09:27.822 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:09:27 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-13T23:09:27.823 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:09:27 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-13T23:09:27.823 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:09:27 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-13T23:09:27.823 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:09:27 smithi002 ceph-mon[71816]: pgmap v76: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2022-05-13T23:09:28.787 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:09:28 smithi002 ceph-mon[71816]: from='osd.2 [v2:172.21.15.115:6800/2562256956,v1:172.21.15.115:6801/2562256956]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2022-05-13T23:09:28.788 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:09:28 smithi002 ceph-mon[71816]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2022-05-13T23:09:28.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:09:28 smithi047 ceph-mon[77147]: from='osd.2 [v2:172.21.15.115:6800/2562256956,v1:172.21.15.115:6801/2562256956]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2022-05-13T23:09:28.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:09:28 smithi047 ceph-mon[77147]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2022-05-13T23:09:28.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:09:28 smithi115 ceph-mon[78593]: from='osd.2 [v2:172.21.15.115:6800/2562256956,v1:172.21.15.115:6801/2562256956]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2022-05-13T23:09:28.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:09:28 smithi115 ceph-mon[78593]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2022-05-13T23:09:29.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:09:29 smithi047 ceph-mon[77147]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2022-05-13T23:09:29.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:09:29 smithi047 ceph-mon[77147]: osdmap e16: 3 total, 2 up, 3 in 2022-05-13T23:09:29.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:09:29 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2022-05-13T23:09:29.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:09:29 smithi047 ceph-mon[77147]: from='osd.2 [v2:172.21.15.115:6800/2562256956,v1:172.21.15.115:6801/2562256956]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi115", "root=default"]}]: dispatch 2022-05-13T23:09:29.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:09:29 smithi047 ceph-mon[77147]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi115", "root=default"]}]: dispatch 2022-05-13T23:09:29.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:09:29 smithi047 ceph-mon[77147]: pgmap v78: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2022-05-13T23:09:29.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:09:29 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:09:29.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:09:29 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-05-13T23:09:29.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:09:29 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:09:29.795 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:09:29 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:09:29.822 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:09:29 smithi002 ceph-mon[71816]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2022-05-13T23:09:29.823 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:09:29 smithi002 ceph-mon[71816]: osdmap e16: 3 total, 2 up, 3 in 2022-05-13T23:09:29.823 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:09:29 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2022-05-13T23:09:29.823 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:09:29 smithi002 ceph-mon[71816]: from='osd.2 [v2:172.21.15.115:6800/2562256956,v1:172.21.15.115:6801/2562256956]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi115", "root=default"]}]: dispatch 2022-05-13T23:09:29.824 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:09:29 smithi002 ceph-mon[71816]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi115", "root=default"]}]: dispatch 2022-05-13T23:09:29.824 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:09:29 smithi002 ceph-mon[71816]: pgmap v78: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2022-05-13T23:09:29.824 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:09:29 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:09:29.824 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:09:29 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-05-13T23:09:29.824 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:09:29 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:09:29.825 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:09:29 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:09:29.858 INFO:journalctl@ceph.osd.2.smithi115.stdout:May 13 23:09:29 smithi115 conmon[84560]: 2022-05-13T23:09:29.477+0000 7ff4a33d8700 -1 osd.2 0 waiting for initial osdmap 2022-05-13T23:09:29.858 INFO:journalctl@ceph.osd.2.smithi115.stdout:May 13 23:09:29 smithi115 conmon[84560]: 2022-05-13T23:09:29.481+0000 7ff49dd6e700 -1 osd.2 17 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2022-05-13T23:09:29.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:09:29 smithi115 ceph-mon[78593]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2022-05-13T23:09:29.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:09:29 smithi115 ceph-mon[78593]: osdmap e16: 3 total, 2 up, 3 in 2022-05-13T23:09:29.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:09:29 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2022-05-13T23:09:29.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:09:29 smithi115 ceph-mon[78593]: from='osd.2 [v2:172.21.15.115:6800/2562256956,v1:172.21.15.115:6801/2562256956]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi115", "root=default"]}]: dispatch 2022-05-13T23:09:29.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:09:29 smithi115 ceph-mon[78593]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi115", "root=default"]}]: dispatch 2022-05-13T23:09:29.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:09:29 smithi115 ceph-mon[78593]: pgmap v78: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2022-05-13T23:09:29.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:09:29 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:09:29.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:09:29 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-05-13T23:09:29.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:09:29 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:09:29.862 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:09:29 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:09:30.090 INFO:teuthology.orchestra.run.smithi002.stderr:--> Zapping: /dev/vg_nvme/lv_3 2022-05-13T23:09:30.090 INFO:teuthology.orchestra.run.smithi002.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2022-05-13T23:09:30.090 INFO:teuthology.orchestra.run.smithi002.stderr: stderr: 10+0 records in 2022-05-13T23:09:30.090 INFO:teuthology.orchestra.run.smithi002.stderr:10+0 records out 2022-05-13T23:09:30.091 INFO:teuthology.orchestra.run.smithi002.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0475447 s, 221 MB/s 2022-05-13T23:09:30.091 INFO:teuthology.orchestra.run.smithi002.stderr:--> Zapping successful for: 2022-05-13T23:09:30.743 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:09:30 smithi002 ceph-mon[71816]: Detected new or changed devices on smithi115 2022-05-13T23:09:30.743 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:09:30 smithi002 ceph-mon[71816]: Adjusting osd_memory_target on smithi115 to 17172M 2022-05-13T23:09:30.743 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:09:30 smithi002 ceph-mon[71816]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi115", "root=default"]}]': finished 2022-05-13T23:09:30.744 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:09:30 smithi002 ceph-mon[71816]: osdmap e17: 3 total, 2 up, 3 in 2022-05-13T23:09:30.744 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:09:30 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2022-05-13T23:09:30.744 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:09:30 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2022-05-13T23:09:30.747 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:23b0658391ce94ca40799469eec00c28b298816c shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 16cf5f88-d311-11ec-be40-001a4aab830c -- ceph orch daemon add osd smithi002:vg_nvme/lv_3 2022-05-13T23:09:30.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:09:30 smithi047 ceph-mon[77147]: Detected new or changed devices on smithi115 2022-05-13T23:09:30.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:09:30 smithi047 ceph-mon[77147]: Adjusting osd_memory_target on smithi115 to 17172M 2022-05-13T23:09:30.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:09:30 smithi047 ceph-mon[77147]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi115", "root=default"]}]': finished 2022-05-13T23:09:30.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:09:30 smithi047 ceph-mon[77147]: osdmap e17: 3 total, 2 up, 3 in 2022-05-13T23:09:30.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:09:30 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2022-05-13T23:09:30.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:09:30 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2022-05-13T23:09:30.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:09:30 smithi115 ceph-mon[78593]: Detected new or changed devices on smithi115 2022-05-13T23:09:30.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:09:30 smithi115 ceph-mon[78593]: Adjusting osd_memory_target on smithi115 to 17172M 2022-05-13T23:09:30.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:09:30 smithi115 ceph-mon[78593]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi115", "root=default"]}]': finished 2022-05-13T23:09:30.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:09:30 smithi115 ceph-mon[78593]: osdmap e17: 3 total, 2 up, 3 in 2022-05-13T23:09:30.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:09:30 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2022-05-13T23:09:30.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:09:30 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2022-05-13T23:09:31.422 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/16cf5f88-d311-11ec-be40-001a4aab830c/mon.a/config 2022-05-13T23:09:32.072 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:09:32 smithi002 ceph-mon[71816]: purged_snaps scrub starts 2022-05-13T23:09:32.073 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:09:32 smithi002 ceph-mon[71816]: purged_snaps scrub ok 2022-05-13T23:09:32.073 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:09:32 smithi002 ceph-mon[71816]: osd.2 [v2:172.21.15.115:6800/2562256956,v1:172.21.15.115:6801/2562256956] boot 2022-05-13T23:09:32.073 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:09:32 smithi002 ceph-mon[71816]: osdmap e18: 3 total, 3 up, 3 in 2022-05-13T23:09:32.074 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:09:32 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2022-05-13T23:09:32.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:09:32 smithi047 ceph-mon[77147]: purged_snaps scrub starts 2022-05-13T23:09:32.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:09:32 smithi047 ceph-mon[77147]: purged_snaps scrub ok 2022-05-13T23:09:32.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:09:32 smithi047 ceph-mon[77147]: osd.2 [v2:172.21.15.115:6800/2562256956,v1:172.21.15.115:6801/2562256956] boot 2022-05-13T23:09:32.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:09:32 smithi047 ceph-mon[77147]: osdmap e18: 3 total, 3 up, 3 in 2022-05-13T23:09:32.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:09:32 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2022-05-13T23:09:32.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:09:32 smithi115 ceph-mon[78593]: purged_snaps scrub starts 2022-05-13T23:09:32.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:09:32 smithi115 ceph-mon[78593]: purged_snaps scrub ok 2022-05-13T23:09:32.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:09:32 smithi115 ceph-mon[78593]: osd.2 [v2:172.21.15.115:6800/2562256956,v1:172.21.15.115:6801/2562256956] boot 2022-05-13T23:09:32.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:09:32 smithi115 ceph-mon[78593]: osdmap e18: 3 total, 3 up, 3 in 2022-05-13T23:09:32.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:09:32 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2022-05-13T23:09:33.240 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:09:33.238+0000 7f15dd96a700 1 -- 172.21.15.2:0/199197062 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f15d80fc690 msgr2=0x7f15d80fcae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:09:33.240 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:09:33.238+0000 7f15dd96a700 1 --2- 172.21.15.2:0/199197062 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f15d80fc690 0x7f15d80fcae0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f15c4004870 tx=0x7f15c4009cb0 comp rx=0 tx=0).stop 2022-05-13T23:09:33.240 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:09:33.238+0000 7f15dd96a700 1 -- 172.21.15.2:0/199197062 shutdown_connections 2022-05-13T23:09:33.241 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:09:33.238+0000 7f15dd96a700 1 --2- 172.21.15.2:0/199197062 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f15d80fd020 0x7f15d81038e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:09:33.241 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:09:33.238+0000 7f15dd96a700 1 --2- 172.21.15.2:0/199197062 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f15d80fb3f0 0x7f15d80fb800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:09:33.241 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:09:33.238+0000 7f15dd96a700 1 --2- 172.21.15.2:0/199197062 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f15d80fc690 0x7f15d80fcae0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:09:33.241 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:09:33.238+0000 7f15dd96a700 1 -- 172.21.15.2:0/199197062 >> 172.21.15.2:0/199197062 conn(0x7f15d80f6b40 msgr2=0x7f15d80f8f70 unknown :-1 s=STATE_NONE l=0).mark_down 2022-05-13T23:09:33.242 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:09:33.238+0000 7f15dd96a700 1 -- 172.21.15.2:0/199197062 shutdown_connections 2022-05-13T23:09:33.242 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:09:33.238+0000 7f15dd96a700 1 -- 172.21.15.2:0/199197062 wait complete. 2022-05-13T23:09:33.242 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:09:33.239+0000 7f15dd96a700 1 Processor -- start 2022-05-13T23:09:33.244 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:09:33.239+0000 7f15dd96a700 1 -- start start 2022-05-13T23:09:33.244 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:09:33.239+0000 7f15dd96a700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f15d80fb3f0 0x7f15d80701d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:09:33.244 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:09:33.239+0000 7f15dd96a700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f15d80fc690 0x7f15d806e830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:09:33.245 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:09:33.239+0000 7f15dd96a700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f15d80fd020 0x7f15d806ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:09:33.245 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:09:33.239+0000 7f15dd96a700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15d806f480 con 0x7f15d80fd020 2022-05-13T23:09:33.245 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:09:33.239+0000 7f15dd96a700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15d806f5f0 con 0x7f15d80fc690 2022-05-13T23:09:33.245 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:09:33.239+0000 7f15dd96a700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15d81162d0 con 0x7f15d80fb3f0 2022-05-13T23:09:33.245 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:09:33.240+0000 7f15dc968700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f15d80fb3f0 0x7f15d80701d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:09:33.246 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:09:33.240+0000 7f15dc968700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f15d80fb3f0 0x7f15d80701d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.2:44304/0 (socket says 172.21.15.2:44304) 2022-05-13T23:09:33.246 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:09:33.240+0000 7f15dd169700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f15d80fd020 0x7f15d806ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:09:33.246 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:09:33.240+0000 7f15dc968700 1 -- 172.21.15.2:0/3213465554 learned_addr learned my addr 172.21.15.2:0/3213465554 (peer_addr_for_me v2:172.21.15.2:0/0) 2022-05-13T23:09:33.246 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:09:33.240+0000 7f15d7fff700 1 --2- 172.21.15.2:0/3213465554 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f15d80fc690 0x7f15d806e830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:09:33.246 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:09:33.240+0000 7f15dc968700 1 -- 172.21.15.2:0/3213465554 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f15d80fc690 msgr2=0x7f15d806e830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:09:33.247 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:09:33.240+0000 7f15dc968700 1 --2- 172.21.15.2:0/3213465554 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f15d80fc690 0x7f15d806e830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:09:33.247 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:09:33.240+0000 7f15dc968700 1 -- 172.21.15.2:0/3213465554 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f15d80fd020 msgr2=0x7f15d806ed70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:09:33.247 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:09:33.240+0000 7f15dc968700 1 --2- 172.21.15.2:0/3213465554 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f15d80fd020 0x7f15d806ed70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:09:33.247 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:09:33.240+0000 7f15dc968700 1 -- 172.21.15.2:0/3213465554 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15c4005040 con 0x7f15d80fb3f0 2022-05-13T23:09:33.248 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:09:33.240+0000 7f15dd169700 1 --2- 172.21.15.2:0/3213465554 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f15d80fd020 0x7f15d806ed70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-05-13T23:09:33.248 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:09:33.240+0000 7f15dc968700 1 --2- 172.21.15.2:0/3213465554 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f15d80fb3f0 0x7f15d80701d0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f15cc00e940 tx=0x7f15cc00b040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-05-13T23:09:33.248 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:09:33.241+0000 7f15d57fa700 1 -- 172.21.15.2:0/3213465554 <== mon.2 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15cc007b40 con 0x7f15d80fb3f0 2022-05-13T23:09:33.248 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:09:33.241+0000 7f15d57fa700 1 -- 172.21.15.2:0/3213465554 <== mon.2 v2:172.21.15.115:3300/0 2 ==== config(26 keys) v1 ==== 1087+0+0 (secure 0 0 0) 0x7f15cc00f040 con 0x7f15d80fb3f0 2022-05-13T23:09:33.248 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:09:33.241+0000 7f15d57fa700 1 -- 172.21.15.2:0/3213465554 <== mon.2 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15cc014890 con 0x7f15d80fb3f0 2022-05-13T23:09:33.249 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:09:33.241+0000 7f15dd96a700 1 -- 172.21.15.2:0/3213465554 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f15d8116470 con 0x7f15d80fb3f0 2022-05-13T23:09:33.249 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:09:33.241+0000 7f15dd96a700 1 -- 172.21.15.2:0/3213465554 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f15d8116930 con 0x7f15d80fb3f0 2022-05-13T23:09:33.249 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:09:33.242+0000 7f15dd96a700 1 -- 172.21.15.2:0/3213465554 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f15d8106c40 con 0x7f15d80fb3f0 2022-05-13T23:09:33.250 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:09:33.244+0000 7f15d57fa700 1 -- 172.21.15.2:0/3213465554 <== mon.2 v2:172.21.15.115:3300/0 4 ==== mgrmap(e 15) v1 ==== 125096+0+0 (secure 0 0 0) 0x7f15cc01b030 con 0x7f15d80fb3f0 2022-05-13T23:09:33.250 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:09:33.245+0000 7f15d57fa700 1 --2- 172.21.15.2:0/3213465554 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f15c8096d10 0x7f15c80991c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:09:33.251 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:09:33.245+0000 7f15d57fa700 1 -- 172.21.15.2:0/3213465554 <== mon.2 v2:172.21.15.115:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 3233+0+0 (secure 0 0 0) 0x7f15cc0bd790 con 0x7f15d80fb3f0 2022-05-13T23:09:33.251 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:09:33.245+0000 7f15d7fff700 1 --2- 172.21.15.2:0/3213465554 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f15c8096d10 0x7f15c80991c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:09:33.251 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:09:33.246+0000 7f15d7fff700 1 --2- 172.21.15.2:0/3213465554 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f15c8096d10 0x7f15c80991c0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f15c4013490 tx=0x7f15c4009cb0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-05-13T23:09:33.251 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:09:33.249+0000 7f15d57fa700 1 -- 172.21.15.2:0/3213465554 <== mon.2 v2:172.21.15.115:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+165215 (secure 0 0 0) 0x7f15cc04b570 con 0x7f15d80fb3f0 2022-05-13T23:09:33.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:09:33 smithi047 ceph-mon[77147]: pgmap v81: 0 pgs: ; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2022-05-13T23:09:33.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:09:33 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]: dispatch 2022-05-13T23:09:33.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:09:33 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]': finished 2022-05-13T23:09:33.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:09:33 smithi047 ceph-mon[77147]: osdmap e19: 3 total, 3 up, 3 in 2022-05-13T23:09:33.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:09:33 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2022-05-13T23:09:33.322 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:09:33 smithi002 ceph-mon[71816]: pgmap v81: 0 pgs: ; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2022-05-13T23:09:33.323 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:09:33 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]: dispatch 2022-05-13T23:09:33.323 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:09:33 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]': finished 2022-05-13T23:09:33.323 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:09:33 smithi002 ceph-mon[71816]: osdmap e19: 3 total, 3 up, 3 in 2022-05-13T23:09:33.323 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:09:33 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2022-05-13T23:09:33.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:09:33 smithi115 ceph-mon[78593]: pgmap v81: 0 pgs: ; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2022-05-13T23:09:33.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:09:33 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]: dispatch 2022-05-13T23:09:33.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:09:33 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]': finished 2022-05-13T23:09:33.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:09:33 smithi115 ceph-mon[78593]: osdmap e19: 3 total, 3 up, 3 in 2022-05-13T23:09:33.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:09:33 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2022-05-13T23:09:33.450 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:09:33.448+0000 7f15dd96a700 1 -- 172.21.15.2:0/3213465554 --> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi002:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) v1 -- 0x7f15d805c920 con 0x7f15c8096d10 2022-05-13T23:09:34.033 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:09:34.032+0000 7f15d57fa700 1 -- 172.21.15.2:0/3213465554 <== mon.2 v2:172.21.15.115:3300/0 7 ==== mgrmap(e 16) v1 ==== 125136+0+0 (secure 0 0 0) 0x7f15cc0bc420 con 0x7f15d80fb3f0 2022-05-13T23:09:34.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:09:34 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2022-05-13T23:09:34.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:09:34 smithi047 ceph-mon[77147]: osdmap e20: 3 total, 3 up, 3 in 2022-05-13T23:09:34.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:09:34 smithi047 ceph-mon[77147]: pgmap v84: 1 pgs: 1 unknown; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2022-05-13T23:09:34.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:09:34 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2022-05-13T23:09:34.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:09:34 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2022-05-13T23:09:34.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:09:34 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-13T23:09:34.322 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:09:34 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2022-05-13T23:09:34.323 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:09:34 smithi002 ceph-mon[71816]: osdmap e20: 3 total, 3 up, 3 in 2022-05-13T23:09:34.323 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:09:34 smithi002 ceph-mon[71816]: pgmap v84: 1 pgs: 1 unknown; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2022-05-13T23:09:34.323 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:09:34 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2022-05-13T23:09:34.323 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:09:34 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2022-05-13T23:09:34.323 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:09:34 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-13T23:09:34.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:09:34 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2022-05-13T23:09:34.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:09:34 smithi115 ceph-mon[78593]: osdmap e20: 3 total, 3 up, 3 in 2022-05-13T23:09:34.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:09:34 smithi115 ceph-mon[78593]: pgmap v84: 1 pgs: 1 unknown; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2022-05-13T23:09:34.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:09:34 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2022-05-13T23:09:34.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:09:34 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2022-05-13T23:09:34.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:09:34 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-13T23:09:34.812 INFO:journalctl@ceph.osd.0.smithi002.stdout:May 13 23:09:34 smithi002 sudo[88849]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2022-05-13T23:09:34.812 INFO:journalctl@ceph.osd.0.smithi002.stdout:May 13 23:09:34 smithi002 sudo[88849]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2022-05-13T23:09:34.813 INFO:journalctl@ceph.osd.0.smithi002.stdout:May 13 23:09:34 smithi002 sudo[88849]: pam_unix(sudo:session): session opened for user root by (uid=0) 2022-05-13T23:09:34.813 INFO:journalctl@ceph.osd.0.smithi002.stdout:May 13 23:09:34 smithi002 sudo[88849]: pam_unix(sudo:session): session closed for user root 2022-05-13T23:09:35.072 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:09:35 smithi002 ceph-mon[71816]: from='client.24206 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi002:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2022-05-13T23:09:35.072 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:09:35 smithi002 ceph-mon[71816]: osdmap e21: 3 total, 3 up, 3 in 2022-05-13T23:09:35.073 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:09:35 smithi002 ceph-mon[71816]: mgrmap e16: x(active, since 2m), standbys: y, z 2022-05-13T23:09:35.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:09:35 smithi047 ceph-mon[77147]: from='client.24206 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi002:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2022-05-13T23:09:35.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:09:35 smithi047 ceph-mon[77147]: osdmap e21: 3 total, 3 up, 3 in 2022-05-13T23:09:35.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:09:35 smithi047 ceph-mon[77147]: mgrmap e16: x(active, since 2m), standbys: y, z 2022-05-13T23:09:35.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:09:35 smithi115 ceph-mon[78593]: from='client.24206 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi002:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2022-05-13T23:09:35.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:09:35 smithi115 ceph-mon[78593]: osdmap e21: 3 total, 3 up, 3 in 2022-05-13T23:09:35.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:09:35 smithi115 ceph-mon[78593]: mgrmap e16: x(active, since 2m), standbys: y, z 2022-05-13T23:09:35.481 INFO:journalctl@ceph.osd.0.smithi002.stdout:May 13 23:09:35 smithi002 sudo[88923]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2022-05-13T23:09:35.481 INFO:journalctl@ceph.osd.0.smithi002.stdout:May 13 23:09:35 smithi002 sudo[88923]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2022-05-13T23:09:35.482 INFO:journalctl@ceph.osd.0.smithi002.stdout:May 13 23:09:35 smithi002 sudo[88923]: pam_unix(sudo:session): session opened for user root by (uid=0) 2022-05-13T23:09:35.482 INFO:journalctl@ceph.osd.0.smithi002.stdout:May 13 23:09:35 smithi002 sudo[88923]: pam_unix(sudo:session): session closed for user root 2022-05-13T23:09:36.272 INFO:journalctl@ceph.osd.1.smithi047.stdout:May 13 23:09:35 smithi047 sudo[84781]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2022-05-13T23:09:36.272 INFO:journalctl@ceph.osd.1.smithi047.stdout:May 13 23:09:35 smithi047 sudo[84781]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2022-05-13T23:09:36.273 INFO:journalctl@ceph.osd.1.smithi047.stdout:May 13 23:09:35 smithi047 sudo[84781]: pam_unix(sudo:session): session opened for user root by (uid=0) 2022-05-13T23:09:36.275 INFO:journalctl@ceph.osd.1.smithi047.stdout:May 13 23:09:35 smithi047 sudo[84781]: pam_unix(sudo:session): session closed for user root 2022-05-13T23:09:36.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:09:36 smithi047 ceph-mon[77147]: pgmap v86: 1 pgs: 1 creating+activating; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2022-05-13T23:09:36.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:09:36 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:09:36.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:09:36 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:09:36.542 INFO:journalctl@ceph.osd.1.smithi047.stdout:May 13 23:09:36 smithi047 sudo[84784]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2022-05-13T23:09:36.542 INFO:journalctl@ceph.osd.1.smithi047.stdout:May 13 23:09:36 smithi047 sudo[84784]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2022-05-13T23:09:36.543 INFO:journalctl@ceph.osd.1.smithi047.stdout:May 13 23:09:36 smithi047 sudo[84784]: pam_unix(sudo:session): session opened for user root by (uid=0) 2022-05-13T23:09:36.543 INFO:journalctl@ceph.osd.1.smithi047.stdout:May 13 23:09:36 smithi047 sudo[84784]: pam_unix(sudo:session): session closed for user root 2022-05-13T23:09:36.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:09:36 smithi115 ceph-mon[78593]: pgmap v86: 1 pgs: 1 creating+activating; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2022-05-13T23:09:36.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:09:36 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:09:36.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:09:36 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:09:36.796 INFO:journalctl@ceph.osd.2.smithi115.stdout:May 13 23:09:36 smithi115 sudo[86676]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2022-05-13T23:09:36.796 INFO:journalctl@ceph.osd.2.smithi115.stdout:May 13 23:09:36 smithi115 sudo[86676]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2022-05-13T23:09:36.796 INFO:journalctl@ceph.osd.2.smithi115.stdout:May 13 23:09:36 smithi115 sudo[86676]: pam_unix(sudo:session): session opened for user root by (uid=0) 2022-05-13T23:09:36.797 INFO:journalctl@ceph.osd.2.smithi115.stdout:May 13 23:09:36 smithi115 sudo[86676]: pam_unix(sudo:session): session closed for user root 2022-05-13T23:09:36.822 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:09:36 smithi002 ceph-mon[71816]: pgmap v86: 1 pgs: 1 creating+activating; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2022-05-13T23:09:36.823 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:09:36 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:09:36.823 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:09:36 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:09:37.108 INFO:journalctl@ceph.osd.2.smithi115.stdout:May 13 23:09:36 smithi115 sudo[86679]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2022-05-13T23:09:37.108 INFO:journalctl@ceph.osd.2.smithi115.stdout:May 13 23:09:36 smithi115 sudo[86679]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2022-05-13T23:09:37.109 INFO:journalctl@ceph.osd.2.smithi115.stdout:May 13 23:09:36 smithi115 sudo[86679]: pam_unix(sudo:session): session opened for user root by (uid=0) 2022-05-13T23:09:37.109 INFO:journalctl@ceph.osd.2.smithi115.stdout:May 13 23:09:36 smithi115 sudo[86679]: pam_unix(sudo:session): session closed for user root 2022-05-13T23:09:38.073 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:09:37 smithi002 sudo[89036]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2022-05-13T23:09:38.073 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:09:37 smithi002 sudo[89036]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2022-05-13T23:09:38.074 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:09:37 smithi002 sudo[89036]: pam_unix(sudo:session): session opened for user root by (uid=0) 2022-05-13T23:09:38.074 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:09:37 smithi002 sudo[89036]: pam_unix(sudo:session): session closed for user root 2022-05-13T23:09:38.658 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:09:38 smithi002 sudo[89045]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2022-05-13T23:09:38.659 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:09:38 smithi002 sudo[89045]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2022-05-13T23:09:38.659 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:09:38 smithi002 sudo[89045]: pam_unix(sudo:session): session opened for user root by (uid=0) 2022-05-13T23:09:38.659 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:09:38 smithi002 sudo[89045]: pam_unix(sudo:session): session closed for user root 2022-05-13T23:09:38.660 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:09:38 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:09:38.660 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:09:38 smithi002 ceph-mon[71816]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2022-05-13T23:09:38.798 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:09:38 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:09:38.799 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:09:38 smithi047 ceph-mon[77147]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2022-05-13T23:09:38.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:09:38 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:09:38.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:09:38 smithi115 ceph-mon[78593]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2022-05-13T23:09:39.278 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:09:38 smithi047 sudo[84787]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2022-05-13T23:09:39.278 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:09:38 smithi047 sudo[84787]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2022-05-13T23:09:39.279 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:09:38 smithi047 sudo[84787]: pam_unix(sudo:session): session opened for user root by (uid=0) 2022-05-13T23:09:39.279 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:09:38 smithi047 sudo[84787]: pam_unix(sudo:session): session closed for user root 2022-05-13T23:09:39.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:09:39 smithi047 sudo[84790]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2022-05-13T23:09:39.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:09:39 smithi047 sudo[84790]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2022-05-13T23:09:39.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:09:39 smithi047 sudo[84790]: pam_unix(sudo:session): session opened for user root by (uid=0) 2022-05-13T23:09:39.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:09:39 smithi047 sudo[84790]: pam_unix(sudo:session): session closed for user root 2022-05-13T23:09:39.776 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:09:39 smithi115 sudo[86682]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2022-05-13T23:09:39.777 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:09:39 smithi115 sudo[86682]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2022-05-13T23:09:39.777 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:09:39 smithi115 sudo[86682]: pam_unix(sudo:session): session opened for user root by (uid=0) 2022-05-13T23:09:39.777 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:09:39 smithi115 sudo[86682]: pam_unix(sudo:session): session closed for user root 2022-05-13T23:09:40.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:09:39 smithi047 ceph-mon[77147]: from='client.? 172.21.15.2:0/3288982711' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6ca9c602-5936-48da-a861-c437e6a33765"}]: dispatch 2022-05-13T23:09:40.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:09:39 smithi047 ceph-mon[77147]: pgmap v87: 1 pgs: 1 creating+activating; 0 B data, 17 MiB used, 268 GiB / 268 GiB avail 2022-05-13T23:09:40.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:09:39 smithi047 ceph-mon[77147]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2022-05-13T23:09:40.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:09:39 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-05-13T23:09:40.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:09:39 smithi047 ceph-mon[77147]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6ca9c602-5936-48da-a861-c437e6a33765"}]: dispatch 2022-05-13T23:09:40.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:09:39 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-05-13T23:09:40.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:09:39 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-05-13T23:09:40.045 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:09:39 smithi047 ceph-mon[77147]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "6ca9c602-5936-48da-a861-c437e6a33765"}]': finished 2022-05-13T23:09:40.045 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:09:39 smithi047 ceph-mon[77147]: osdmap e22: 4 total, 3 up, 4 in 2022-05-13T23:09:40.045 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:09:39 smithi047 ceph-mon[77147]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2022-05-13T23:09:40.045 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:09:39 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2022-05-13T23:09:40.046 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:09:39 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-05-13T23:09:40.046 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:09:39 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-05-13T23:09:40.046 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:09:39 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-05-13T23:09:40.046 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:09:39 smithi047 ceph-mon[77147]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2022-05-13T23:09:40.046 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:09:39 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-05-13T23:09:40.047 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:09:39 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-05-13T23:09:40.047 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:09:39 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-05-13T23:09:40.047 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:09:39 smithi002 ceph-mon[71816]: from='client.? 172.21.15.2:0/3288982711' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6ca9c602-5936-48da-a861-c437e6a33765"}]: dispatch 2022-05-13T23:09:40.048 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:09:39 smithi002 ceph-mon[71816]: pgmap v87: 1 pgs: 1 creating+activating; 0 B data, 17 MiB used, 268 GiB / 268 GiB avail 2022-05-13T23:09:40.048 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:09:39 smithi002 ceph-mon[71816]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2022-05-13T23:09:40.048 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:09:39 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-05-13T23:09:40.049 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:09:39 smithi002 ceph-mon[71816]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6ca9c602-5936-48da-a861-c437e6a33765"}]: dispatch 2022-05-13T23:09:40.049 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:09:39 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-05-13T23:09:40.049 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:09:39 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-05-13T23:09:40.049 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:09:39 smithi002 ceph-mon[71816]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "6ca9c602-5936-48da-a861-c437e6a33765"}]': finished 2022-05-13T23:09:40.049 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:09:39 smithi002 ceph-mon[71816]: osdmap e22: 4 total, 3 up, 4 in 2022-05-13T23:09:40.050 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:09:39 smithi002 ceph-mon[71816]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2022-05-13T23:09:40.050 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:09:39 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2022-05-13T23:09:40.050 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:09:39 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-05-13T23:09:40.050 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:09:39 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-05-13T23:09:40.051 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:09:39 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-05-13T23:09:40.051 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:09:39 smithi002 ceph-mon[71816]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2022-05-13T23:09:40.051 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:09:39 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-05-13T23:09:40.051 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:09:39 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-05-13T23:09:40.051 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:09:39 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-05-13T23:09:40.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:09:39 smithi115 sudo[86693]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2022-05-13T23:09:40.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:09:39 smithi115 sudo[86693]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2022-05-13T23:09:40.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:09:39 smithi115 sudo[86693]: pam_unix(sudo:session): session opened for user root by (uid=0) 2022-05-13T23:09:40.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:09:39 smithi115 sudo[86693]: pam_unix(sudo:session): session closed for user root 2022-05-13T23:09:40.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:09:39 smithi115 ceph-mon[78593]: from='client.? 172.21.15.2:0/3288982711' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6ca9c602-5936-48da-a861-c437e6a33765"}]: dispatch 2022-05-13T23:09:40.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:09:39 smithi115 ceph-mon[78593]: pgmap v87: 1 pgs: 1 creating+activating; 0 B data, 17 MiB used, 268 GiB / 268 GiB avail 2022-05-13T23:09:40.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:09:39 smithi115 ceph-mon[78593]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2022-05-13T23:09:40.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:09:39 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-05-13T23:09:40.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:09:39 smithi115 ceph-mon[78593]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6ca9c602-5936-48da-a861-c437e6a33765"}]: dispatch 2022-05-13T23:09:40.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:09:39 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-05-13T23:09:40.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:09:39 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-05-13T23:09:40.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:09:39 smithi115 ceph-mon[78593]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "6ca9c602-5936-48da-a861-c437e6a33765"}]': finished 2022-05-13T23:09:40.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:09:39 smithi115 ceph-mon[78593]: osdmap e22: 4 total, 3 up, 4 in 2022-05-13T23:09:40.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:09:39 smithi115 ceph-mon[78593]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2022-05-13T23:09:40.112 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:09:39 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2022-05-13T23:09:40.112 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:09:39 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-05-13T23:09:40.112 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:09:39 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-05-13T23:09:40.112 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:09:39 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-05-13T23:09:40.112 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:09:39 smithi115 ceph-mon[78593]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2022-05-13T23:09:40.113 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:09:39 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-05-13T23:09:40.113 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:09:39 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-05-13T23:09:40.113 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:09:39 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-05-13T23:09:41.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:09:40 smithi047 ceph-mon[77147]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2022-05-13T23:09:41.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:09:40 smithi047 ceph-mon[77147]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2022-05-13T23:09:41.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:09:40 smithi047 ceph-mon[77147]: from='client.? 172.21.15.2:0/1731969523' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2022-05-13T23:09:41.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:09:40 smithi047 ceph-mon[77147]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2022-05-13T23:09:41.072 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:09:40 smithi002 ceph-mon[71816]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2022-05-13T23:09:41.072 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:09:40 smithi002 ceph-mon[71816]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2022-05-13T23:09:41.073 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:09:40 smithi002 ceph-mon[71816]: from='client.? 172.21.15.2:0/1731969523' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2022-05-13T23:09:41.073 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:09:40 smithi002 ceph-mon[71816]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2022-05-13T23:09:41.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:09:40 smithi115 ceph-mon[78593]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2022-05-13T23:09:41.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:09:40 smithi115 ceph-mon[78593]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2022-05-13T23:09:41.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:09:40 smithi115 ceph-mon[78593]: from='client.? 172.21.15.2:0/1731969523' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2022-05-13T23:09:41.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:09:40 smithi115 ceph-mon[78593]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2022-05-13T23:09:41.822 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:09:41 smithi002 ceph-mon[71816]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail 2022-05-13T23:09:42.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:09:41 smithi115 ceph-mon[78593]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail 2022-05-13T23:09:42.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:09:41 smithi047 ceph-mon[77147]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail 2022-05-13T23:09:44.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:09:44 smithi047 ceph-mon[77147]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail 2022-05-13T23:09:44.545 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:09:44 smithi002 ceph-mon[71816]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail 2022-05-13T23:09:44.616 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:09:44 smithi115 ceph-mon[78593]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail 2022-05-13T23:09:46.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:09:46 smithi047 ceph-mon[77147]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail; 1.5 KiB/s rd, 44 KiB/s wr, 3 op/s 2022-05-13T23:09:46.572 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:09:46 smithi002 ceph-mon[71816]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail; 1.5 KiB/s rd, 44 KiB/s wr, 3 op/s 2022-05-13T23:09:46.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:09:46 smithi115 ceph-mon[78593]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail; 1.5 KiB/s rd, 44 KiB/s wr, 3 op/s 2022-05-13T23:09:48.422 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:09:48 smithi002 ceph-mon[71816]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 20 MiB used, 268 GiB / 268 GiB avail; 1.1 KiB/s rd, 33 KiB/s wr, 2 op/s 2022-05-13T23:09:48.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:09:48 smithi047 ceph-mon[77147]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 20 MiB used, 268 GiB / 268 GiB avail; 1.1 KiB/s rd, 33 KiB/s wr, 2 op/s 2022-05-13T23:09:48.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:09:48 smithi115 ceph-mon[78593]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 20 MiB used, 268 GiB / 268 GiB avail; 1.1 KiB/s rd, 33 KiB/s wr, 2 op/s 2022-05-13T23:09:49.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:09:49 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2022-05-13T23:09:49.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:09:49 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-13T23:09:49.572 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:09:49 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2022-05-13T23:09:49.572 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:09:49 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-13T23:09:49.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:09:49 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2022-05-13T23:09:49.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:09:49 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-13T23:09:50.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:09:50 smithi047 ceph-mon[77147]: Deploying daemon osd.3 on smithi002 2022-05-13T23:09:50.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:09:50 smithi047 ceph-mon[77147]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 20 MiB used, 268 GiB / 268 GiB avail; 921 B/s rd, 26 KiB/s wr, 2 op/s 2022-05-13T23:09:50.572 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:09:50 smithi002 ceph-mon[71816]: Deploying daemon osd.3 on smithi002 2022-05-13T23:09:50.573 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:09:50 smithi002 ceph-mon[71816]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 20 MiB used, 268 GiB / 268 GiB avail; 921 B/s rd, 26 KiB/s wr, 2 op/s 2022-05-13T23:09:50.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:09:50 smithi115 ceph-mon[78593]: Deploying daemon osd.3 on smithi002 2022-05-13T23:09:50.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:09:50 smithi115 ceph-mon[78593]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 20 MiB used, 268 GiB / 268 GiB avail; 921 B/s rd, 26 KiB/s wr, 2 op/s 2022-05-13T23:09:52.323 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:09:52 smithi002 ceph-mon[71816]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 20 MiB used, 268 GiB / 268 GiB avail; 767 B/s rd, 22 KiB/s wr, 1 op/s 2022-05-13T23:09:52.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:09:52 smithi047 ceph-mon[77147]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 20 MiB used, 268 GiB / 268 GiB avail; 767 B/s rd, 22 KiB/s wr, 1 op/s 2022-05-13T23:09:52.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:09:52 smithi115 ceph-mon[78593]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 20 MiB used, 268 GiB / 268 GiB avail; 767 B/s rd, 22 KiB/s wr, 1 op/s 2022-05-13T23:09:54.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:09:54 smithi047 ceph-mon[77147]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 20 MiB used, 268 GiB / 268 GiB avail; 767 B/s rd, 22 KiB/s wr, 1 op/s 2022-05-13T23:09:54.572 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:09:54 smithi002 ceph-mon[71816]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 20 MiB used, 268 GiB / 268 GiB avail; 767 B/s rd, 22 KiB/s wr, 1 op/s 2022-05-13T23:09:54.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:09:54 smithi115 ceph-mon[78593]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 20 MiB used, 268 GiB / 268 GiB avail; 767 B/s rd, 22 KiB/s wr, 1 op/s 2022-05-13T23:09:56.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:09:56 smithi047 ceph-mon[77147]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 20 MiB used, 268 GiB / 268 GiB avail; 767 B/s rd, 22 KiB/s wr, 1 op/s 2022-05-13T23:09:56.558 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:09:56 smithi002 ceph-mon[71816]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 20 MiB used, 268 GiB / 268 GiB avail; 767 B/s rd, 22 KiB/s wr, 1 op/s 2022-05-13T23:09:56.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:09:56 smithi115 ceph-mon[78593]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 20 MiB used, 268 GiB / 268 GiB avail; 767 B/s rd, 22 KiB/s wr, 1 op/s 2022-05-13T23:09:58.235 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:09:57 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:09:58.235 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:09:57 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-13T23:09:58.236 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:09:57 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-13T23:09:58.236 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:09:57 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-13T23:09:58.236 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:09:57 smithi002 ceph-mon[71816]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 20 MiB used, 268 GiB / 268 GiB avail 2022-05-13T23:09:58.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:09:57 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:09:58.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:09:57 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-13T23:09:58.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:09:57 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-13T23:09:58.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:09:57 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-13T23:09:58.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:09:57 smithi047 ceph-mon[77147]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 20 MiB used, 268 GiB / 268 GiB avail 2022-05-13T23:09:58.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:09:57 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:09:58.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:09:57 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-13T23:09:58.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:09:57 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-13T23:09:58.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:09:57 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-13T23:09:58.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:09:57 smithi115 ceph-mon[78593]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 20 MiB used, 268 GiB / 268 GiB avail 2022-05-13T23:10:00.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:10:00 smithi047 ceph-mon[77147]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 20 MiB used, 268 GiB / 268 GiB avail 2022-05-13T23:10:00.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:10:00 smithi047 ceph-mon[77147]: overall HEALTH_OK 2022-05-13T23:10:00.573 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:10:00 smithi002 ceph-mon[71816]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 20 MiB used, 268 GiB / 268 GiB avail 2022-05-13T23:10:00.573 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:10:00 smithi002 ceph-mon[71816]: overall HEALTH_OK 2022-05-13T23:10:00.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:10:00 smithi115 ceph-mon[78593]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 20 MiB used, 268 GiB / 268 GiB avail 2022-05-13T23:10:00.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:10:00 smithi115 ceph-mon[78593]: overall HEALTH_OK 2022-05-13T23:10:01.510 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:10:01 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:10:01.511 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:10:01 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:10:01.511 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:10:01 smithi002 ceph-mon[71816]: from='osd.3 [v2:172.21.15.2:6810/1099909411,v1:172.21.15.2:6811/1099909411]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2022-05-13T23:10:01.567 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:10:01.564+0000 7f15d57fa700 1 -- 172.21.15.2:0/3213465554 <== mgr.14148 v2:172.21.15.2:6800/2788339627 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f15d805c920 con 0x7f15c8096d10 2022-05-13T23:10:01.567 INFO:teuthology.orchestra.run.smithi002.stdout:Created osd(s) 3 on host 'smithi002' 2022-05-13T23:10:01.569 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:10:01.567+0000 7f15dd96a700 1 -- 172.21.15.2:0/3213465554 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f15c8096d10 msgr2=0x7f15c80991c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:10:01.570 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:10:01.567+0000 7f15dd96a700 1 --2- 172.21.15.2:0/3213465554 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f15c8096d10 0x7f15c80991c0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f15c4013490 tx=0x7f15c4009cb0 comp rx=0 tx=0).stop 2022-05-13T23:10:01.570 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:10:01.567+0000 7f15dd96a700 1 -- 172.21.15.2:0/3213465554 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f15d80fb3f0 msgr2=0x7f15d80701d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:10:01.570 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:10:01.567+0000 7f15dd96a700 1 --2- 172.21.15.2:0/3213465554 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f15d80fb3f0 0x7f15d80701d0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f15cc00e940 tx=0x7f15cc00b040 comp rx=0 tx=0).stop 2022-05-13T23:10:01.570 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:10:01.567+0000 7f15dd96a700 1 -- 172.21.15.2:0/3213465554 shutdown_connections 2022-05-13T23:10:01.571 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:10:01.567+0000 7f15dd96a700 1 --2- 172.21.15.2:0/3213465554 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f15c8096d10 0x7f15c80991c0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:10:01.571 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:10:01.567+0000 7f15dd96a700 1 --2- 172.21.15.2:0/3213465554 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f15d80fd020 0x7f15d806ed70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:10:01.571 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:10:01.567+0000 7f15dd96a700 1 --2- 172.21.15.2:0/3213465554 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f15d80fb3f0 0x7f15d80701d0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:10:01.571 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:10:01.567+0000 7f15dd96a700 1 --2- 172.21.15.2:0/3213465554 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f15d80fc690 0x7f15d806e830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:10:01.572 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:10:01.567+0000 7f15dd96a700 1 -- 172.21.15.2:0/3213465554 >> 172.21.15.2:0/3213465554 conn(0x7f15d80f6b40 msgr2=0x7f15d80f8eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-05-13T23:10:01.572 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:10:01.567+0000 7f15dd96a700 1 -- 172.21.15.2:0/3213465554 shutdown_connections 2022-05-13T23:10:01.572 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:10:01.567+0000 7f15dd96a700 1 -- 172.21.15.2:0/3213465554 wait complete. 2022-05-13T23:10:01.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:10:01 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:10:01.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:10:01 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:10:01.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:10:01 smithi115 ceph-mon[78593]: from='osd.3 [v2:172.21.15.2:6810/1099909411,v1:172.21.15.2:6811/1099909411]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2022-05-13T23:10:01.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:10:01 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:10:01.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:10:01 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:10:01.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:10:01 smithi047 ceph-mon[77147]: from='osd.3 [v2:172.21.15.2:6810/1099909411,v1:172.21.15.2:6811/1099909411]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2022-05-13T23:10:02.227 DEBUG:teuthology.orchestra.run.smithi002:osd.3> sudo journalctl -f -n 0 -u ceph-16cf5f88-d311-11ec-be40-001a4aab830c@osd.3.service 2022-05-13T23:10:02.232 INFO:tasks.cephadm:Deploying osd.4 on smithi047 with /dev/vg_nvme/lv_3... 2022-05-13T23:10:02.233 DEBUG:teuthology.orchestra.run.smithi047:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:23b0658391ce94ca40799469eec00c28b298816c shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 16cf5f88-d311-11ec-be40-001a4aab830c -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2022-05-13T23:10:02.319 INFO:journalctl@ceph.osd.3.smithi002.stdout:-- Logs begin at Fri 2022-05-13 22:48:11 UTC. -- 2022-05-13T23:10:02.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:10:02 smithi047 ceph-mon[77147]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 20 MiB used, 268 GiB / 268 GiB avail 2022-05-13T23:10:02.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:10:02 smithi047 ceph-mon[77147]: from='osd.3 [v2:172.21.15.2:6810/1099909411,v1:172.21.15.2:6811/1099909411]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2022-05-13T23:10:02.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:10:02 smithi047 ceph-mon[77147]: osdmap e23: 4 total, 3 up, 4 in 2022-05-13T23:10:02.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:10:02 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2022-05-13T23:10:02.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:10:02 smithi047 ceph-mon[77147]: from='osd.3 [v2:172.21.15.2:6810/1099909411,v1:172.21.15.2:6811/1099909411]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi002", "root=default"]}]: dispatch 2022-05-13T23:10:02.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:10:02 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:10:02.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:10:02 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-13T23:10:02.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:10:02 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-13T23:10:02.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:10:02 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-13T23:10:02.573 INFO:journalctl@ceph.osd.3.smithi002.stdout:May 13 23:10:02 smithi002 conmon[91792]: 2022-05-13T23:10:02.322+0000 7ff100a62700 -1 osd.3 24 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2022-05-13T23:10:02.574 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:10:02 smithi002 ceph-mon[71816]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 20 MiB used, 268 GiB / 268 GiB avail 2022-05-13T23:10:02.574 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:10:02 smithi002 ceph-mon[71816]: from='osd.3 [v2:172.21.15.2:6810/1099909411,v1:172.21.15.2:6811/1099909411]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2022-05-13T23:10:02.574 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:10:02 smithi002 ceph-mon[71816]: osdmap e23: 4 total, 3 up, 4 in 2022-05-13T23:10:02.575 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:10:02 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2022-05-13T23:10:02.575 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:10:02 smithi002 ceph-mon[71816]: from='osd.3 [v2:172.21.15.2:6810/1099909411,v1:172.21.15.2:6811/1099909411]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi002", "root=default"]}]: dispatch 2022-05-13T23:10:02.575 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:10:02 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:10:02.575 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:10:02 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-13T23:10:02.575 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:10:02 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-13T23:10:02.576 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:10:02 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-13T23:10:02.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:10:02 smithi115 ceph-mon[78593]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 20 MiB used, 268 GiB / 268 GiB avail 2022-05-13T23:10:02.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:10:02 smithi115 ceph-mon[78593]: from='osd.3 [v2:172.21.15.2:6810/1099909411,v1:172.21.15.2:6811/1099909411]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2022-05-13T23:10:02.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:10:02 smithi115 ceph-mon[78593]: osdmap e23: 4 total, 3 up, 4 in 2022-05-13T23:10:02.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:10:02 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2022-05-13T23:10:02.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:10:02 smithi115 ceph-mon[78593]: from='osd.3 [v2:172.21.15.2:6810/1099909411,v1:172.21.15.2:6811/1099909411]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi002", "root=default"]}]: dispatch 2022-05-13T23:10:02.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:10:02 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:10:02.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:10:02 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-13T23:10:02.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:10:02 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-13T23:10:02.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:10:02 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-13T23:10:02.702 INFO:teuthology.orchestra.run.smithi047.stderr:Inferring config /var/lib/ceph/16cf5f88-d311-11ec-be40-001a4aab830c/mon.b/config 2022-05-13T23:10:03.572 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:10:03 smithi002 ceph-mon[71816]: from='osd.3 [v2:172.21.15.2:6810/1099909411,v1:172.21.15.2:6811/1099909411]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi002", "root=default"]}]': finished 2022-05-13T23:10:03.573 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:10:03 smithi002 ceph-mon[71816]: osdmap e24: 4 total, 3 up, 4 in 2022-05-13T23:10:03.573 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:10:03 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2022-05-13T23:10:03.573 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:10:03 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2022-05-13T23:10:03.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:10:03 smithi115 ceph-mon[78593]: from='osd.3 [v2:172.21.15.2:6810/1099909411,v1:172.21.15.2:6811/1099909411]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi002", "root=default"]}]': finished 2022-05-13T23:10:03.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:10:03 smithi115 ceph-mon[78593]: osdmap e24: 4 total, 3 up, 4 in 2022-05-13T23:10:03.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:10:03 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2022-05-13T23:10:03.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:10:03 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2022-05-13T23:10:03.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:10:03 smithi047 ceph-mon[77147]: from='osd.3 [v2:172.21.15.2:6810/1099909411,v1:172.21.15.2:6811/1099909411]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi002", "root=default"]}]': finished 2022-05-13T23:10:03.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:10:03 smithi047 ceph-mon[77147]: osdmap e24: 4 total, 3 up, 4 in 2022-05-13T23:10:03.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:10:03 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2022-05-13T23:10:03.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:10:03 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2022-05-13T23:10:04.339 INFO:teuthology.orchestra.run.smithi047.stderr:--> Zapping: /dev/vg_nvme/lv_3 2022-05-13T23:10:04.339 INFO:teuthology.orchestra.run.smithi047.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2022-05-13T23:10:04.339 INFO:teuthology.orchestra.run.smithi047.stderr: stderr: 10+0 records in 2022-05-13T23:10:04.340 INFO:teuthology.orchestra.run.smithi047.stderr:10+0 records out 2022-05-13T23:10:04.340 INFO:teuthology.orchestra.run.smithi047.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.00832999 s, 1.3 GB/s 2022-05-13T23:10:04.341 INFO:teuthology.orchestra.run.smithi047.stderr:--> Zapping successful for: 2022-05-13T23:10:04.572 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:10:04 smithi002 ceph-mon[71816]: purged_snaps scrub starts 2022-05-13T23:10:04.573 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:10:04 smithi002 ceph-mon[71816]: purged_snaps scrub ok 2022-05-13T23:10:04.573 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:10:04 smithi002 ceph-mon[71816]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 20 MiB used, 268 GiB / 268 GiB avail 2022-05-13T23:10:04.573 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:10:04 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2022-05-13T23:10:04.573 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:10:04 smithi002 ceph-mon[71816]: osd.3 [v2:172.21.15.2:6810/1099909411,v1:172.21.15.2:6811/1099909411] boot 2022-05-13T23:10:04.574 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:10:04 smithi002 ceph-mon[71816]: osdmap e25: 4 total, 4 up, 4 in 2022-05-13T23:10:04.574 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:10:04 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2022-05-13T23:10:04.574 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:10:04 smithi002 ceph-mon[71816]: osdmap e26: 4 total, 4 up, 4 in 2022-05-13T23:10:04.583 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:10:04 smithi047 ceph-mon[77147]: purged_snaps scrub starts 2022-05-13T23:10:04.583 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:10:04 smithi047 ceph-mon[77147]: purged_snaps scrub ok 2022-05-13T23:10:04.583 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:10:04 smithi047 ceph-mon[77147]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 20 MiB used, 268 GiB / 268 GiB avail 2022-05-13T23:10:04.584 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:10:04 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2022-05-13T23:10:04.584 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:10:04 smithi047 ceph-mon[77147]: osd.3 [v2:172.21.15.2:6810/1099909411,v1:172.21.15.2:6811/1099909411] boot 2022-05-13T23:10:04.584 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:10:04 smithi047 ceph-mon[77147]: osdmap e25: 4 total, 4 up, 4 in 2022-05-13T23:10:04.584 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:10:04 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2022-05-13T23:10:04.585 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:10:04 smithi047 ceph-mon[77147]: osdmap e26: 4 total, 4 up, 4 in 2022-05-13T23:10:04.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:10:04 smithi115 ceph-mon[78593]: purged_snaps scrub starts 2022-05-13T23:10:04.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:10:04 smithi115 ceph-mon[78593]: purged_snaps scrub ok 2022-05-13T23:10:04.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:10:04 smithi115 ceph-mon[78593]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 20 MiB used, 268 GiB / 268 GiB avail 2022-05-13T23:10:04.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:10:04 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2022-05-13T23:10:04.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:10:04 smithi115 ceph-mon[78593]: osd.3 [v2:172.21.15.2:6810/1099909411,v1:172.21.15.2:6811/1099909411] boot 2022-05-13T23:10:04.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:10:04 smithi115 ceph-mon[78593]: osdmap e25: 4 total, 4 up, 4 in 2022-05-13T23:10:04.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:10:04 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2022-05-13T23:10:04.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:10:04 smithi115 ceph-mon[78593]: osdmap e26: 4 total, 4 up, 4 in 2022-05-13T23:10:04.818 DEBUG:teuthology.orchestra.run.smithi047:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:23b0658391ce94ca40799469eec00c28b298816c shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 16cf5f88-d311-11ec-be40-001a4aab830c -- ceph orch daemon add osd smithi047:vg_nvme/lv_3 2022-05-13T23:10:05.271 INFO:teuthology.orchestra.run.smithi047.stderr:Inferring config /var/lib/ceph/16cf5f88-d311-11ec-be40-001a4aab830c/mon.b/config 2022-05-13T23:10:06.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:10:06 smithi047 ceph-mon[77147]: pgmap v106: 1 pgs: 1 remapped+peering; 577 KiB data, 24 MiB used, 358 GiB / 358 GiB avail 2022-05-13T23:10:06.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:10:06 smithi047 ceph-mon[77147]: osdmap e27: 4 total, 4 up, 4 in 2022-05-13T23:10:06.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:10:06 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-05-13T23:10:06.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:10:06 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-05-13T23:10:06.573 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:10:06 smithi002 ceph-mon[71816]: pgmap v106: 1 pgs: 1 remapped+peering; 577 KiB data, 24 MiB used, 358 GiB / 358 GiB avail 2022-05-13T23:10:06.573 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:10:06 smithi002 ceph-mon[71816]: osdmap e27: 4 total, 4 up, 4 in 2022-05-13T23:10:06.574 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:10:06 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-05-13T23:10:06.574 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:10:06 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-05-13T23:10:06.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:10:06 smithi115 ceph-mon[78593]: pgmap v106: 1 pgs: 1 remapped+peering; 577 KiB data, 24 MiB used, 358 GiB / 358 GiB avail 2022-05-13T23:10:06.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:10:06 smithi115 ceph-mon[78593]: osdmap e27: 4 total, 4 up, 4 in 2022-05-13T23:10:06.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:10:06 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-05-13T23:10:06.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:10:06 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-05-13T23:10:06.815 INFO:teuthology.orchestra.run.smithi047.stderr:2022-05-13T23:10:06.814+0000 7f066d7ef700 1 -- 172.21.15.47:0/2808604881 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f06680fb7a0 msgr2=0x7f06680fbbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:10:06.824 INFO:teuthology.orchestra.run.smithi047.stderr:2022-05-13T23:10:06.814+0000 7f066d7ef700 1 --2- 172.21.15.47:0/2808604881 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f06680fb7a0 0x7f06680fbbf0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f065c004870 tx=0x7f065c009cb0 comp rx=0 tx=0).stop 2022-05-13T23:10:06.825 INFO:teuthology.orchestra.run.smithi047.stderr:2022-05-13T23:10:06.814+0000 7f066d7ef700 1 -- 172.21.15.47:0/2808604881 shutdown_connections 2022-05-13T23:10:06.825 INFO:teuthology.orchestra.run.smithi047.stderr:2022-05-13T23:10:06.814+0000 7f066d7ef700 1 --2- 172.21.15.47:0/2808604881 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f06680fc130 0x7f06681029f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:10:06.825 INFO:teuthology.orchestra.run.smithi047.stderr:2022-05-13T23:10:06.814+0000 7f066d7ef700 1 --2- 172.21.15.47:0/2808604881 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f06680fa500 0x7f06680fa910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:10:06.825 INFO:teuthology.orchestra.run.smithi047.stderr:2022-05-13T23:10:06.814+0000 7f066d7ef700 1 --2- 172.21.15.47:0/2808604881 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f06680fb7a0 0x7f06680fbbf0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:10:06.826 INFO:teuthology.orchestra.run.smithi047.stderr:2022-05-13T23:10:06.814+0000 7f066d7ef700 1 -- 172.21.15.47:0/2808604881 >> 172.21.15.47:0/2808604881 conn(0x7f06680f5cf0 msgr2=0x7f06680f8120 unknown :-1 s=STATE_NONE l=0).mark_down 2022-05-13T23:10:06.826 INFO:teuthology.orchestra.run.smithi047.stderr:2022-05-13T23:10:06.814+0000 7f066d7ef700 1 -- 172.21.15.47:0/2808604881 shutdown_connections 2022-05-13T23:10:06.826 INFO:teuthology.orchestra.run.smithi047.stderr:2022-05-13T23:10:06.814+0000 7f066d7ef700 1 -- 172.21.15.47:0/2808604881 wait complete. 2022-05-13T23:10:06.829 INFO:teuthology.orchestra.run.smithi047.stderr:2022-05-13T23:10:06.814+0000 7f066d7ef700 1 Processor -- start 2022-05-13T23:10:06.829 INFO:teuthology.orchestra.run.smithi047.stderr:2022-05-13T23:10:06.815+0000 7f066d7ef700 1 -- start start 2022-05-13T23:10:06.829 INFO:teuthology.orchestra.run.smithi047.stderr:2022-05-13T23:10:06.815+0000 7f066d7ef700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f06680fa500 0x7f0668064980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:10:06.830 INFO:teuthology.orchestra.run.smithi047.stderr:2022-05-13T23:10:06.815+0000 7f066d7ef700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f06680fb7a0 0x7f0668111140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:10:06.830 INFO:teuthology.orchestra.run.smithi047.stderr:2022-05-13T23:10:06.815+0000 7f066d7ef700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f06680fc130 0x7f0668113740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:10:06.830 INFO:teuthology.orchestra.run.smithi047.stderr:2022-05-13T23:10:06.815+0000 7f066d7ef700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0668113e50 con 0x7f06680fa500 2022-05-13T23:10:06.830 INFO:teuthology.orchestra.run.smithi047.stderr:2022-05-13T23:10:06.815+0000 7f066d7ef700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0668113fc0 con 0x7f06680fb7a0 2022-05-13T23:10:06.831 INFO:teuthology.orchestra.run.smithi047.stderr:2022-05-13T23:10:06.815+0000 7f066d7ef700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0668114180 con 0x7f06680fc130 2022-05-13T23:10:06.831 INFO:teuthology.orchestra.run.smithi047.stderr:2022-05-13T23:10:06.815+0000 7f06667fc700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f06680fb7a0 0x7f0668111140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:10:06.831 INFO:teuthology.orchestra.run.smithi047.stderr:2022-05-13T23:10:06.815+0000 7f06667fc700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f06680fb7a0 0x7f0668111140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:42086/0 (socket says 172.21.15.47:42086) 2022-05-13T23:10:06.831 INFO:teuthology.orchestra.run.smithi047.stderr:2022-05-13T23:10:06.815+0000 7f06667fc700 1 -- 172.21.15.47:0/3254162110 learned_addr learned my addr 172.21.15.47:0/3254162110 (peer_addr_for_me v2:172.21.15.47:0/0) 2022-05-13T23:10:06.831 INFO:teuthology.orchestra.run.smithi047.stderr:2022-05-13T23:10:06.815+0000 7f06667fc700 1 -- 172.21.15.47:0/3254162110 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f06680fc130 msgr2=0x7f0668113740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:10:06.832 INFO:teuthology.orchestra.run.smithi047.stderr:2022-05-13T23:10:06.815+0000 7f06667fc700 1 --2- 172.21.15.47:0/3254162110 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f06680fc130 0x7f0668113740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:10:06.832 INFO:teuthology.orchestra.run.smithi047.stderr:2022-05-13T23:10:06.815+0000 7f06667fc700 1 -- 172.21.15.47:0/3254162110 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f06680fa500 msgr2=0x7f0668064980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:10:06.832 INFO:teuthology.orchestra.run.smithi047.stderr:2022-05-13T23:10:06.815+0000 7f06667fc700 1 --2- 172.21.15.47:0/3254162110 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f06680fa500 0x7f0668064980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:10:06.833 INFO:teuthology.orchestra.run.smithi047.stderr:2022-05-13T23:10:06.815+0000 7f06667fc700 1 -- 172.21.15.47:0/3254162110 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f065c005040 con 0x7f06680fb7a0 2022-05-13T23:10:06.833 INFO:teuthology.orchestra.run.smithi047.stderr:2022-05-13T23:10:06.815+0000 7f06667fc700 1 --2- 172.21.15.47:0/3254162110 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f06680fb7a0 0x7f0668111140 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f065c008e80 tx=0x7f065c009cb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-05-13T23:10:06.833 INFO:teuthology.orchestra.run.smithi047.stderr:2022-05-13T23:10:06.815+0000 7f064f7fe700 1 -- 172.21.15.47:0/3254162110 <== mon.1 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f065c00e040 con 0x7f06680fb7a0 2022-05-13T23:10:06.833 INFO:teuthology.orchestra.run.smithi047.stderr:2022-05-13T23:10:06.815+0000 7f064f7fe700 1 -- 172.21.15.47:0/3254162110 <== mon.1 v2:172.21.15.47:3300/0 2 ==== config(26 keys) v1 ==== 1087+0+0 (secure 0 0 0) 0x7f065c0096f0 con 0x7f06680fb7a0 2022-05-13T23:10:06.835 INFO:teuthology.orchestra.run.smithi047.stderr:2022-05-13T23:10:06.815+0000 7f066d7ef700 1 -- 172.21.15.47:0/3254162110 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0668114400 con 0x7f06680fb7a0 2022-05-13T23:10:06.835 INFO:teuthology.orchestra.run.smithi047.stderr:2022-05-13T23:10:06.815+0000 7f064f7fe700 1 -- 172.21.15.47:0/3254162110 <== mon.1 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f065c01cdb0 con 0x7f06680fb7a0 2022-05-13T23:10:06.835 INFO:teuthology.orchestra.run.smithi047.stderr:2022-05-13T23:10:06.816+0000 7f066d7ef700 1 -- 172.21.15.47:0/3254162110 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f06681148e0 con 0x7f06680fb7a0 2022-05-13T23:10:06.836 INFO:teuthology.orchestra.run.smithi047.stderr:2022-05-13T23:10:06.816+0000 7f066d7ef700 1 -- 172.21.15.47:0/3254162110 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0668105cf0 con 0x7f06680fb7a0 2022-05-13T23:10:06.836 INFO:teuthology.orchestra.run.smithi047.stderr:2022-05-13T23:10:06.818+0000 7f064f7fe700 1 -- 172.21.15.47:0/3254162110 <== mon.1 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 16) v1 ==== 125136+0+0 (secure 0 0 0) 0x7f065c01c530 con 0x7f06680fb7a0 2022-05-13T23:10:06.836 INFO:teuthology.orchestra.run.smithi047.stderr:2022-05-13T23:10:06.818+0000 7f064f7fe700 1 --2- 172.21.15.47:0/3254162110 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f0650096d60 0x7f0650099210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:10:06.836 INFO:teuthology.orchestra.run.smithi047.stderr:2022-05-13T23:10:06.818+0000 7f064f7fe700 1 -- 172.21.15.47:0/3254162110 <== mon.1 v2:172.21.15.47:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 3665+0+0 (secure 0 0 0) 0x7f065c0c0080 con 0x7f06680fb7a0 2022-05-13T23:10:06.837 INFO:teuthology.orchestra.run.smithi047.stderr:2022-05-13T23:10:06.819+0000 7f0666ffd700 1 --2- 172.21.15.47:0/3254162110 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f0650096d60 0x7f0650099210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:10:06.837 INFO:teuthology.orchestra.run.smithi047.stderr:2022-05-13T23:10:06.819+0000 7f0666ffd700 1 --2- 172.21.15.47:0/3254162110 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f0650096d60 0x7f0650099210 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f06680fb600 tx=0x7f0660007be0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-05-13T23:10:06.837 INFO:teuthology.orchestra.run.smithi047.stderr:2022-05-13T23:10:06.821+0000 7f064f7fe700 1 -- 172.21.15.47:0/3254162110 <== mon.1 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+165215 (secure 0 0 0) 0x7f065c08e600 con 0x7f06680fb7a0 2022-05-13T23:10:07.010 INFO:teuthology.orchestra.run.smithi047.stderr:2022-05-13T23:10:07.009+0000 7f066d7ef700 1 -- 172.21.15.47:0/3254162110 --> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi047:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) v1 -- 0x7f066802a5e0 con 0x7f0650096d60 2022-05-13T23:10:07.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:10:07 smithi047 ceph-mon[77147]: osdmap e28: 4 total, 4 up, 4 in 2022-05-13T23:10:07.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:10:07 smithi047 ceph-mon[77147]: Detected new or changed devices on smithi002 2022-05-13T23:10:07.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:10:07 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:10:07.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:10:07 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2022-05-13T23:10:07.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:10:07 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2022-05-13T23:10:07.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:10:07 smithi047 ceph-mon[77147]: Adjusting osd_memory_target on smithi002 to 8638M 2022-05-13T23:10:07.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:10:07 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:10:07.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:10:07 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:10:07.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:10:07 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2022-05-13T23:10:07.295 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:10:07 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2022-05-13T23:10:07.295 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:10:07 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-13T23:10:07.572 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:10:07 smithi002 ceph-mon[71816]: osdmap e28: 4 total, 4 up, 4 in 2022-05-13T23:10:07.573 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:10:07 smithi002 ceph-mon[71816]: Detected new or changed devices on smithi002 2022-05-13T23:10:07.573 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:10:07 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:10:07.573 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:10:07 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2022-05-13T23:10:07.573 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:10:07 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2022-05-13T23:10:07.574 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:10:07 smithi002 ceph-mon[71816]: Adjusting osd_memory_target on smithi002 to 8638M 2022-05-13T23:10:07.574 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:10:07 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:10:07.574 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:10:07 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:10:07.574 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:10:07 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2022-05-13T23:10:07.575 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:10:07 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2022-05-13T23:10:07.575 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:10:07 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-13T23:10:07.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:10:07 smithi115 ceph-mon[78593]: osdmap e28: 4 total, 4 up, 4 in 2022-05-13T23:10:07.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:10:07 smithi115 ceph-mon[78593]: Detected new or changed devices on smithi002 2022-05-13T23:10:07.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:10:07 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:10:07.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:10:07 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2022-05-13T23:10:07.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:10:07 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2022-05-13T23:10:07.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:10:07 smithi115 ceph-mon[78593]: Adjusting osd_memory_target on smithi002 to 8638M 2022-05-13T23:10:07.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:10:07 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:10:07.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:10:07 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:10:07.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:10:07 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2022-05-13T23:10:07.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:10:07 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2022-05-13T23:10:07.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:10:07 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-13T23:10:08.508 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:10:08 smithi047 ceph-mon[77147]: from='client.24232 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi047:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2022-05-13T23:10:08.509 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:10:08 smithi047 ceph-mon[77147]: pgmap v109: 1 pgs: 1 remapped+peering; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2022-05-13T23:10:08.509 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:10:08 smithi047 ceph-mon[77147]: osdmap e29: 4 total, 4 up, 4 in 2022-05-13T23:10:08.572 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:10:08 smithi002 ceph-mon[71816]: from='client.24232 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi047:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2022-05-13T23:10:08.573 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:10:08 smithi002 ceph-mon[71816]: pgmap v109: 1 pgs: 1 remapped+peering; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2022-05-13T23:10:08.573 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:10:08 smithi002 ceph-mon[71816]: osdmap e29: 4 total, 4 up, 4 in 2022-05-13T23:10:08.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:10:08 smithi115 ceph-mon[78593]: from='client.24232 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi047:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2022-05-13T23:10:08.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:10:08 smithi115 ceph-mon[78593]: pgmap v109: 1 pgs: 1 remapped+peering; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2022-05-13T23:10:08.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:10:08 smithi115 ceph-mon[78593]: osdmap e29: 4 total, 4 up, 4 in 2022-05-13T23:10:10.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:10:10 smithi047 ceph-mon[77147]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail; 104 KiB/s, 0 objects/s recovering 2022-05-13T23:10:10.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:10:10 smithi047 ceph-mon[77147]: from='client.? 172.21.15.47:0/1285719012' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "54d943d9-2610-422a-9e59-4b8212aced31"}]: dispatch 2022-05-13T23:10:10.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:10:10 smithi047 ceph-mon[77147]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "54d943d9-2610-422a-9e59-4b8212aced31"}]: dispatch 2022-05-13T23:10:10.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:10:10 smithi047 ceph-mon[77147]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "54d943d9-2610-422a-9e59-4b8212aced31"}]': finished 2022-05-13T23:10:10.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:10:10 smithi047 ceph-mon[77147]: osdmap e30: 5 total, 4 up, 5 in 2022-05-13T23:10:10.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:10:10 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2022-05-13T23:10:10.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:10:10 smithi047 ceph-mon[77147]: from='client.? 172.21.15.47:0/3283668077' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2022-05-13T23:10:10.572 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:10:10 smithi002 ceph-mon[71816]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail; 104 KiB/s, 0 objects/s recovering 2022-05-13T23:10:10.573 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:10:10 smithi002 ceph-mon[71816]: from='client.? 172.21.15.47:0/1285719012' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "54d943d9-2610-422a-9e59-4b8212aced31"}]: dispatch 2022-05-13T23:10:10.573 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:10:10 smithi002 ceph-mon[71816]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "54d943d9-2610-422a-9e59-4b8212aced31"}]: dispatch 2022-05-13T23:10:10.573 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:10:10 smithi002 ceph-mon[71816]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "54d943d9-2610-422a-9e59-4b8212aced31"}]': finished 2022-05-13T23:10:10.574 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:10:10 smithi002 ceph-mon[71816]: osdmap e30: 5 total, 4 up, 5 in 2022-05-13T23:10:10.574 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:10:10 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2022-05-13T23:10:10.576 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:10:10 smithi002 ceph-mon[71816]: from='client.? 172.21.15.47:0/3283668077' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2022-05-13T23:10:10.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:10:10 smithi115 ceph-mon[78593]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail; 104 KiB/s, 0 objects/s recovering 2022-05-13T23:10:10.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:10:10 smithi115 ceph-mon[78593]: from='client.? 172.21.15.47:0/1285719012' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "54d943d9-2610-422a-9e59-4b8212aced31"}]: dispatch 2022-05-13T23:10:10.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:10:10 smithi115 ceph-mon[78593]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "54d943d9-2610-422a-9e59-4b8212aced31"}]: dispatch 2022-05-13T23:10:10.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:10:10 smithi115 ceph-mon[78593]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "54d943d9-2610-422a-9e59-4b8212aced31"}]': finished 2022-05-13T23:10:10.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:10:10 smithi115 ceph-mon[78593]: osdmap e30: 5 total, 4 up, 5 in 2022-05-13T23:10:10.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:10:10 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2022-05-13T23:10:10.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:10:10 smithi115 ceph-mon[78593]: from='client.? 172.21.15.47:0/3283668077' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2022-05-13T23:10:12.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:10:12 smithi047 ceph-mon[77147]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 24 MiB used, 358 GiB / 358 GiB avail; 86 KiB/s, 0 objects/s recovering 2022-05-13T23:10:12.572 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:10:12 smithi002 ceph-mon[71816]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 24 MiB used, 358 GiB / 358 GiB avail; 86 KiB/s, 0 objects/s recovering 2022-05-13T23:10:12.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:10:12 smithi115 ceph-mon[78593]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 24 MiB used, 358 GiB / 358 GiB avail; 86 KiB/s, 0 objects/s recovering 2022-05-13T23:10:14.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:10:14 smithi047 ceph-mon[77147]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 24 MiB used, 358 GiB / 358 GiB avail; 74 KiB/s, 0 objects/s recovering 2022-05-13T23:10:14.575 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:10:14 smithi002 ceph-mon[71816]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 24 MiB used, 358 GiB / 358 GiB avail; 74 KiB/s, 0 objects/s recovering 2022-05-13T23:10:14.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:10:14 smithi115 ceph-mon[78593]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 24 MiB used, 358 GiB / 358 GiB avail; 74 KiB/s, 0 objects/s recovering 2022-05-13T23:10:16.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:10:16 smithi047 ceph-mon[77147]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 24 MiB used, 358 GiB / 358 GiB avail; 65 KiB/s, 0 objects/s recovering 2022-05-13T23:10:16.572 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:10:16 smithi002 ceph-mon[71816]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 24 MiB used, 358 GiB / 358 GiB avail; 65 KiB/s, 0 objects/s recovering 2022-05-13T23:10:16.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:10:16 smithi115 ceph-mon[78593]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 24 MiB used, 358 GiB / 358 GiB avail; 65 KiB/s, 0 objects/s recovering 2022-05-13T23:10:17.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:10:17 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2022-05-13T23:10:17.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:10:17 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-13T23:10:17.572 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:10:17 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2022-05-13T23:10:17.572 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:10:17 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-13T23:10:17.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:10:17 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2022-05-13T23:10:17.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:10:17 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-13T23:10:18.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:10:18 smithi047 ceph-mon[77147]: Deploying daemon osd.4 on smithi047 2022-05-13T23:10:18.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:10:18 smithi047 ceph-mon[77147]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 24 MiB used, 358 GiB / 358 GiB avail; 52 KiB/s, 0 objects/s recovering 2022-05-13T23:10:18.572 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:10:18 smithi002 ceph-mon[71816]: Deploying daemon osd.4 on smithi047 2022-05-13T23:10:18.573 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:10:18 smithi002 ceph-mon[71816]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 24 MiB used, 358 GiB / 358 GiB avail; 52 KiB/s, 0 objects/s recovering 2022-05-13T23:10:18.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:10:18 smithi115 ceph-mon[78593]: Deploying daemon osd.4 on smithi047 2022-05-13T23:10:18.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:10:18 smithi115 ceph-mon[78593]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 24 MiB used, 358 GiB / 358 GiB avail; 52 KiB/s, 0 objects/s recovering 2022-05-13T23:10:20.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:10:20 smithi047 ceph-mon[77147]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 24 MiB used, 358 GiB / 358 GiB avail 2022-05-13T23:10:20.572 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:10:20 smithi002 ceph-mon[71816]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 24 MiB used, 358 GiB / 358 GiB avail 2022-05-13T23:10:20.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:10:20 smithi115 ceph-mon[78593]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 24 MiB used, 358 GiB / 358 GiB avail 2022-05-13T23:10:22.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:10:22 smithi115 ceph-mon[78593]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 24 MiB used, 358 GiB / 358 GiB avail 2022-05-13T23:10:22.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:10:22 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:10:22.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:10:22 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-13T23:10:22.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:10:22 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-13T23:10:22.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:10:22 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-13T23:10:22.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:10:22 smithi047 ceph-mon[77147]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 24 MiB used, 358 GiB / 358 GiB avail 2022-05-13T23:10:22.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:10:22 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:10:22.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:10:22 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-13T23:10:22.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:10:22 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-13T23:10:22.795 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:10:22 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-13T23:10:22.822 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:10:22 smithi002 ceph-mon[71816]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 24 MiB used, 358 GiB / 358 GiB avail 2022-05-13T23:10:22.823 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:10:22 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:10:22.823 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:10:22 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-13T23:10:22.823 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:10:22 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-13T23:10:22.823 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:10:22 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-13T23:10:24.216 INFO:teuthology.orchestra.run.smithi047.stdout:Created osd(s) 4 on host 'smithi047' 2022-05-13T23:10:24.217 INFO:teuthology.orchestra.run.smithi047.stderr:2022-05-13T23:10:24.213+0000 7f064f7fe700 1 -- 172.21.15.47:0/3254162110 <== mgr.14148 v2:172.21.15.2:6800/2788339627 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f066802a5e0 con 0x7f0650096d60 2022-05-13T23:10:24.219 INFO:teuthology.orchestra.run.smithi047.stderr:2022-05-13T23:10:24.216+0000 7f066d7ef700 1 -- 172.21.15.47:0/3254162110 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f0650096d60 msgr2=0x7f0650099210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:10:24.219 INFO:teuthology.orchestra.run.smithi047.stderr:2022-05-13T23:10:24.216+0000 7f066d7ef700 1 --2- 172.21.15.47:0/3254162110 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f0650096d60 0x7f0650099210 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f06680fb600 tx=0x7f0660007be0 comp rx=0 tx=0).stop 2022-05-13T23:10:24.219 INFO:teuthology.orchestra.run.smithi047.stderr:2022-05-13T23:10:24.216+0000 7f066d7ef700 1 -- 172.21.15.47:0/3254162110 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f06680fb7a0 msgr2=0x7f0668111140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:10:24.220 INFO:teuthology.orchestra.run.smithi047.stderr:2022-05-13T23:10:24.216+0000 7f066d7ef700 1 --2- 172.21.15.47:0/3254162110 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f06680fb7a0 0x7f0668111140 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f065c008e80 tx=0x7f065c009cb0 comp rx=0 tx=0).stop 2022-05-13T23:10:24.220 INFO:teuthology.orchestra.run.smithi047.stderr:2022-05-13T23:10:24.216+0000 7f066d7ef700 1 -- 172.21.15.47:0/3254162110 shutdown_connections 2022-05-13T23:10:24.220 INFO:teuthology.orchestra.run.smithi047.stderr:2022-05-13T23:10:24.216+0000 7f066d7ef700 1 --2- 172.21.15.47:0/3254162110 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f06680fc130 0x7f0668113740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:10:24.221 INFO:teuthology.orchestra.run.smithi047.stderr:2022-05-13T23:10:24.216+0000 7f066d7ef700 1 --2- 172.21.15.47:0/3254162110 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f0650096d60 0x7f0650099210 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:10:24.221 INFO:teuthology.orchestra.run.smithi047.stderr:2022-05-13T23:10:24.216+0000 7f066d7ef700 1 --2- 172.21.15.47:0/3254162110 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f06680fa500 0x7f0668064980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:10:24.221 INFO:teuthology.orchestra.run.smithi047.stderr:2022-05-13T23:10:24.216+0000 7f066d7ef700 1 --2- 172.21.15.47:0/3254162110 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f06680fb7a0 0x7f0668111140 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:10:24.221 INFO:teuthology.orchestra.run.smithi047.stderr:2022-05-13T23:10:24.216+0000 7f066d7ef700 1 -- 172.21.15.47:0/3254162110 >> 172.21.15.47:0/3254162110 conn(0x7f06680f5cf0 msgr2=0x7f06680fe9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-05-13T23:10:24.222 INFO:teuthology.orchestra.run.smithi047.stderr:2022-05-13T23:10:24.216+0000 7f066d7ef700 1 -- 172.21.15.47:0/3254162110 shutdown_connections 2022-05-13T23:10:24.222 INFO:teuthology.orchestra.run.smithi047.stderr:2022-05-13T23:10:24.216+0000 7f066d7ef700 1 -- 172.21.15.47:0/3254162110 wait complete. 2022-05-13T23:10:24.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:10:24 smithi047 ceph-mon[77147]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 24 MiB used, 358 GiB / 358 GiB avail 2022-05-13T23:10:24.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:10:24 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:10:24.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:10:24 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:10:24.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:10:24 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:10:24.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:10:24 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-13T23:10:24.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:10:24 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-13T23:10:24.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:10:24 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-13T23:10:24.707 DEBUG:teuthology.orchestra.run.smithi047:osd.4> sudo journalctl -f -n 0 -u ceph-16cf5f88-d311-11ec-be40-001a4aab830c@osd.4.service 2022-05-13T23:10:24.710 INFO:tasks.cephadm:Deploying osd.5 on smithi115 with /dev/vg_nvme/lv_3... 2022-05-13T23:10:24.711 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:23b0658391ce94ca40799469eec00c28b298816c shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 16cf5f88-d311-11ec-be40-001a4aab830c -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2022-05-13T23:10:24.736 INFO:journalctl@ceph.osd.4.smithi047.stdout:-- Logs begin at Fri 2022-05-13 22:44:47 UTC. -- 2022-05-13T23:10:24.738 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:10:24 smithi115 ceph-mon[78593]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 24 MiB used, 358 GiB / 358 GiB avail 2022-05-13T23:10:24.738 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:10:24 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:10:24.738 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:10:24 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:10:24.738 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:10:24 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:10:24.739 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:10:24 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-13T23:10:24.739 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:10:24 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-13T23:10:24.739 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:10:24 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-13T23:10:24.822 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:10:24 smithi002 ceph-mon[71816]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 24 MiB used, 358 GiB / 358 GiB avail 2022-05-13T23:10:24.823 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:10:24 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:10:24.823 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:10:24 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:10:24.824 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:10:24 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:10:24.824 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:10:24 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-13T23:10:24.824 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:10:24 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-13T23:10:24.824 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:10:24 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-13T23:10:25.163 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/16cf5f88-d311-11ec-be40-001a4aab830c/mon.c/config 2022-05-13T23:10:25.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:10:25 smithi047 ceph-mon[77147]: from='osd.4 [v2:172.21.15.47:6808/882753861,v1:172.21.15.47:6809/882753861]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2022-05-13T23:10:25.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:10:25 smithi047 ceph-mon[77147]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2022-05-13T23:10:25.542 INFO:journalctl@ceph.osd.4.smithi047.stdout:May 13 23:10:25 smithi047 conmon[88497]: 2022-05-13T23:10:25.250+0000 7f774807f3c0 -1 osd.4 0 log_to_monitors true 2022-05-13T23:10:25.614 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:10:25 smithi115 ceph-mon[78593]: from='osd.4 [v2:172.21.15.47:6808/882753861,v1:172.21.15.47:6809/882753861]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2022-05-13T23:10:25.614 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:10:25 smithi115 ceph-mon[78593]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2022-05-13T23:10:25.822 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:10:25 smithi002 ceph-mon[71816]: from='osd.4 [v2:172.21.15.47:6808/882753861,v1:172.21.15.47:6809/882753861]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2022-05-13T23:10:25.822 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:10:25 smithi002 ceph-mon[71816]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2022-05-13T23:10:26.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:10:26 smithi115 ceph-mon[78593]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 24 MiB used, 358 GiB / 358 GiB avail 2022-05-13T23:10:26.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:10:26 smithi115 ceph-mon[78593]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2022-05-13T23:10:26.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:10:26 smithi115 ceph-mon[78593]: osdmap e31: 5 total, 4 up, 5 in 2022-05-13T23:10:26.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:10:26 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2022-05-13T23:10:26.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:10:26 smithi115 ceph-mon[78593]: from='osd.4 [v2:172.21.15.47:6808/882753861,v1:172.21.15.47:6809/882753861]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi047", "root=default"]}]: dispatch 2022-05-13T23:10:26.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:10:26 smithi115 ceph-mon[78593]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi047", "root=default"]}]: dispatch 2022-05-13T23:10:26.794 INFO:journalctl@ceph.osd.4.smithi047.stdout:May 13 23:10:26 smithi047 conmon[88497]: 2022-05-13T23:10:26.374+0000 7f773ea83700 -1 osd.4 0 waiting for initial osdmap 2022-05-13T23:10:26.794 INFO:journalctl@ceph.osd.4.smithi047.stdout:May 13 23:10:26 smithi047 conmon[88497]: 2022-05-13T23:10:26.384+0000 7f7737415700 -1 osd.4 32 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2022-05-13T23:10:26.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:10:26 smithi047 ceph-mon[77147]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 24 MiB used, 358 GiB / 358 GiB avail 2022-05-13T23:10:26.795 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:10:26 smithi047 ceph-mon[77147]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2022-05-13T23:10:26.795 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:10:26 smithi047 ceph-mon[77147]: osdmap e31: 5 total, 4 up, 5 in 2022-05-13T23:10:26.795 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:10:26 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2022-05-13T23:10:26.795 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:10:26 smithi047 ceph-mon[77147]: from='osd.4 [v2:172.21.15.47:6808/882753861,v1:172.21.15.47:6809/882753861]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi047", "root=default"]}]: dispatch 2022-05-13T23:10:26.796 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:10:26 smithi047 ceph-mon[77147]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi047", "root=default"]}]: dispatch 2022-05-13T23:10:26.822 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:10:26 smithi002 ceph-mon[71816]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 24 MiB used, 358 GiB / 358 GiB avail 2022-05-13T23:10:26.823 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:10:26 smithi002 ceph-mon[71816]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2022-05-13T23:10:26.823 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:10:26 smithi002 ceph-mon[71816]: osdmap e31: 5 total, 4 up, 5 in 2022-05-13T23:10:26.823 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:10:26 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2022-05-13T23:10:26.823 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:10:26 smithi002 ceph-mon[71816]: from='osd.4 [v2:172.21.15.47:6808/882753861,v1:172.21.15.47:6809/882753861]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi047", "root=default"]}]: dispatch 2022-05-13T23:10:26.824 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:10:26 smithi002 ceph-mon[71816]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi047", "root=default"]}]: dispatch 2022-05-13T23:10:26.841 INFO:teuthology.orchestra.run.smithi115.stderr:--> Zapping: /dev/vg_nvme/lv_3 2022-05-13T23:10:26.841 INFO:teuthology.orchestra.run.smithi115.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2022-05-13T23:10:26.841 INFO:teuthology.orchestra.run.smithi115.stderr: stderr: 10+0 records in 2022-05-13T23:10:26.842 INFO:teuthology.orchestra.run.smithi115.stderr:10+0 records out 2022-05-13T23:10:26.842 INFO:teuthology.orchestra.run.smithi115.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0111992 s, 936 MB/s 2022-05-13T23:10:26.842 INFO:teuthology.orchestra.run.smithi115.stderr:--> Zapping successful for: 2022-05-13T23:10:27.266 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:23b0658391ce94ca40799469eec00c28b298816c shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 16cf5f88-d311-11ec-be40-001a4aab830c -- ceph orch daemon add osd smithi115:vg_nvme/lv_3 2022-05-13T23:10:27.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:10:27 smithi047 ceph-mon[77147]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi047", "root=default"]}]': finished 2022-05-13T23:10:27.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:10:27 smithi047 ceph-mon[77147]: osdmap e32: 5 total, 4 up, 5 in 2022-05-13T23:10:27.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:10:27 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2022-05-13T23:10:27.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:10:27 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2022-05-13T23:10:27.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:10:27 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:10:27.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:10:27 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-05-13T23:10:27.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:10:27 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-05-13T23:10:27.545 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:10:27 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:10:27.545 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:10:27 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:10:27.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:10:27 smithi115 ceph-mon[78593]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi047", "root=default"]}]': finished 2022-05-13T23:10:27.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:10:27 smithi115 ceph-mon[78593]: osdmap e32: 5 total, 4 up, 5 in 2022-05-13T23:10:27.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:10:27 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2022-05-13T23:10:27.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:10:27 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2022-05-13T23:10:27.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:10:27 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:10:27.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:10:27 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-05-13T23:10:27.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:10:27 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-05-13T23:10:27.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:10:27 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:10:27.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:10:27 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:10:27.747 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/16cf5f88-d311-11ec-be40-001a4aab830c/mon.c/config 2022-05-13T23:10:27.822 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:10:27 smithi002 ceph-mon[71816]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi047", "root=default"]}]': finished 2022-05-13T23:10:27.823 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:10:27 smithi002 ceph-mon[71816]: osdmap e32: 5 total, 4 up, 5 in 2022-05-13T23:10:27.823 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:10:27 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2022-05-13T23:10:27.824 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:10:27 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2022-05-13T23:10:27.824 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:10:27 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:10:27.824 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:10:27 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-05-13T23:10:27.824 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:10:27 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-05-13T23:10:27.825 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:10:27 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:10:27.825 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:10:27 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:10:28.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:10:28 smithi047 ceph-mon[77147]: purged_snaps scrub starts 2022-05-13T23:10:28.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:10:28 smithi047 ceph-mon[77147]: purged_snaps scrub ok 2022-05-13T23:10:28.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:10:28 smithi047 ceph-mon[77147]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 24 MiB used, 358 GiB / 358 GiB avail 2022-05-13T23:10:28.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:10:28 smithi047 ceph-mon[77147]: Detected new or changed devices on smithi047 2022-05-13T23:10:28.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:10:28 smithi047 ceph-mon[77147]: Adjusting osd_memory_target on smithi047 to 8588M 2022-05-13T23:10:28.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:10:28 smithi047 ceph-mon[77147]: osd.4 [v2:172.21.15.47:6808/882753861,v1:172.21.15.47:6809/882753861] boot 2022-05-13T23:10:28.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:10:28 smithi047 ceph-mon[77147]: osdmap e33: 5 total, 5 up, 5 in 2022-05-13T23:10:28.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:10:28 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2022-05-13T23:10:28.812 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:10:28 smithi115 ceph-mon[78593]: purged_snaps scrub starts 2022-05-13T23:10:28.813 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:10:28 smithi115 ceph-mon[78593]: purged_snaps scrub ok 2022-05-13T23:10:28.813 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:10:28 smithi115 ceph-mon[78593]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 24 MiB used, 358 GiB / 358 GiB avail 2022-05-13T23:10:28.813 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:10:28 smithi115 ceph-mon[78593]: Detected new or changed devices on smithi047 2022-05-13T23:10:28.813 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:10:28 smithi115 ceph-mon[78593]: Adjusting osd_memory_target on smithi047 to 8588M 2022-05-13T23:10:28.814 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:10:28 smithi115 ceph-mon[78593]: osd.4 [v2:172.21.15.47:6808/882753861,v1:172.21.15.47:6809/882753861] boot 2022-05-13T23:10:28.814 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:10:28 smithi115 ceph-mon[78593]: osdmap e33: 5 total, 5 up, 5 in 2022-05-13T23:10:28.814 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:10:28 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2022-05-13T23:10:28.822 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:10:28 smithi002 ceph-mon[71816]: purged_snaps scrub starts 2022-05-13T23:10:28.822 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:10:28 smithi002 ceph-mon[71816]: purged_snaps scrub ok 2022-05-13T23:10:28.823 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:10:28 smithi002 ceph-mon[71816]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 24 MiB used, 358 GiB / 358 GiB avail 2022-05-13T23:10:28.823 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:10:28 smithi002 ceph-mon[71816]: Detected new or changed devices on smithi047 2022-05-13T23:10:28.823 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:10:28 smithi002 ceph-mon[71816]: Adjusting osd_memory_target on smithi047 to 8588M 2022-05-13T23:10:28.823 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:10:28 smithi002 ceph-mon[71816]: osd.4 [v2:172.21.15.47:6808/882753861,v1:172.21.15.47:6809/882753861] boot 2022-05-13T23:10:28.824 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:10:28 smithi002 ceph-mon[71816]: osdmap e33: 5 total, 5 up, 5 in 2022-05-13T23:10:28.824 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:10:28 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2022-05-13T23:10:29.023 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:10:29.018+0000 7fbbb16f6700 1 -- 172.21.15.115:0/333925628 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fbbac0fc130 msgr2=0x7fbbac1029f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:10:29.023 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:10:29.018+0000 7fbbb16f6700 1 --2- 172.21.15.115:0/333925628 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fbbac0fc130 0x7fbbac1029f0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fbba4004870 tx=0x7fbba4009cb0 comp rx=0 tx=0).stop 2022-05-13T23:10:29.023 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:10:29.018+0000 7fbbb16f6700 1 -- 172.21.15.115:0/333925628 shutdown_connections 2022-05-13T23:10:29.024 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:10:29.018+0000 7fbbb16f6700 1 --2- 172.21.15.115:0/333925628 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fbbac0fc130 0x7fbbac1029f0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:10:29.024 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:10:29.018+0000 7fbbb16f6700 1 --2- 172.21.15.115:0/333925628 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fbbac0fa500 0x7fbbac0fa910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:10:29.024 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:10:29.018+0000 7fbbb16f6700 1 --2- 172.21.15.115:0/333925628 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fbbac0fb7a0 0x7fbbac0fbbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:10:29.024 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:10:29.018+0000 7fbbb16f6700 1 -- 172.21.15.115:0/333925628 >> 172.21.15.115:0/333925628 conn(0x7fbbac0f5cf0 msgr2=0x7fbbac0f8120 unknown :-1 s=STATE_NONE l=0).mark_down 2022-05-13T23:10:29.025 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:10:29.018+0000 7fbbb16f6700 1 -- 172.21.15.115:0/333925628 shutdown_connections 2022-05-13T23:10:29.025 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:10:29.018+0000 7fbbb16f6700 1 -- 172.21.15.115:0/333925628 wait complete. 2022-05-13T23:10:29.025 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:10:29.018+0000 7fbbb16f6700 1 Processor -- start 2022-05-13T23:10:29.025 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:10:29.018+0000 7fbbb16f6700 1 -- start start 2022-05-13T23:10:29.026 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:10:29.019+0000 7fbbb16f6700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fbbac0fa500 0x7fbbac064960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:10:29.036 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:10:29.019+0000 7fbbb16f6700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fbbac0fb7a0 0x7fbbac110f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:10:29.037 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:10:29.019+0000 7fbbb16f6700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fbbac0fc130 0x7fbbac1135f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:10:29.037 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:10:29.019+0000 7fbbb16f6700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbbac113de0 con 0x7fbbac0fb7a0 2022-05-13T23:10:29.038 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:10:29.019+0000 7fbbb16f6700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbbac113f50 con 0x7fbbac0fa500 2022-05-13T23:10:29.038 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:10:29.019+0000 7fbbb16f6700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbbac114110 con 0x7fbbac0fc130 2022-05-13T23:10:29.038 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:10:29.019+0000 7fbbab7fe700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fbbac0fc130 0x7fbbac1135f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:10:29.038 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:10:29.019+0000 7fbbab7fe700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fbbac0fc130 0x7fbbac1135f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:48986/0 (socket says 172.21.15.115:48986) 2022-05-13T23:10:29.039 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:10:29.019+0000 7fbbab7fe700 1 -- 172.21.15.115:0/2626175300 learned_addr learned my addr 172.21.15.115:0/2626175300 (peer_addr_for_me v2:172.21.15.115:0/0) 2022-05-13T23:10:29.039 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:10:29.019+0000 7fbbaaffd700 1 --2- 172.21.15.115:0/2626175300 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fbbac0fa500 0x7fbbac064960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:10:29.039 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:10:29.019+0000 7fbbab7fe700 1 -- 172.21.15.115:0/2626175300 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fbbac0fa500 msgr2=0x7fbbac064960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:10:29.039 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:10:29.019+0000 7fbbab7fe700 1 --2- 172.21.15.115:0/2626175300 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fbbac0fa500 0x7fbbac064960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:10:29.040 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:10:29.019+0000 7fbbab7fe700 1 -- 172.21.15.115:0/2626175300 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fbbac0fb7a0 msgr2=0x7fbbac110f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:10:29.040 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:10:29.019+0000 7fbbab7fe700 1 --2- 172.21.15.115:0/2626175300 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fbbac0fb7a0 0x7fbbac110f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:10:29.040 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:10:29.019+0000 7fbbab7fe700 1 -- 172.21.15.115:0/2626175300 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbba4005040 con 0x7fbbac0fc130 2022-05-13T23:10:29.040 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:10:29.019+0000 7fbbaaffd700 1 --2- 172.21.15.115:0/2626175300 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fbbac0fa500 0x7fbbac064960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2022-05-13T23:10:29.041 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:10:29.019+0000 7fbbab7fe700 1 --2- 172.21.15.115:0/2626175300 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fbbac0fc130 0x7fbbac1135f0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fbba4009d60 tx=0x7fbba4009cb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-05-13T23:10:29.041 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:10:29.019+0000 7fbba37fe700 1 -- 172.21.15.115:0/2626175300 <== mon.2 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbba400e040 con 0x7fbbac0fc130 2022-05-13T23:10:29.041 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:10:29.019+0000 7fbba37fe700 1 -- 172.21.15.115:0/2626175300 <== mon.2 v2:172.21.15.115:3300/0 2 ==== config(26 keys) v1 ==== 1087+0+0 (secure 0 0 0) 0x7fbba4009870 con 0x7fbbac0fc130 2022-05-13T23:10:29.041 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:10:29.019+0000 7fbbb16f6700 1 -- 172.21.15.115:0/2626175300 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbbac114330 con 0x7fbbac0fc130 2022-05-13T23:10:29.042 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:10:29.019+0000 7fbba37fe700 1 -- 172.21.15.115:0/2626175300 <== mon.2 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbba40123c0 con 0x7fbbac0fc130 2022-05-13T23:10:29.042 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:10:29.019+0000 7fbbb16f6700 1 -- 172.21.15.115:0/2626175300 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbbac1146e0 con 0x7fbbac0fc130 2022-05-13T23:10:29.042 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:10:29.020+0000 7fbbb16f6700 1 -- 172.21.15.115:0/2626175300 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbbac105db0 con 0x7fbbac0fc130 2022-05-13T23:10:29.044 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:10:29.021+0000 7fbba37fe700 1 -- 172.21.15.115:0/2626175300 <== mon.2 v2:172.21.15.115:3300/0 4 ==== mgrmap(e 16) v1 ==== 125136+0+0 (secure 0 0 0) 0x7fbba401c6a0 con 0x7fbbac0fc130 2022-05-13T23:10:29.044 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:10:29.022+0000 7fbba37fe700 1 --2- 172.21.15.115:0/2626175300 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7fbb98096ce0 0x7fbb98099190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:10:29.044 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:10:29.022+0000 7fbba37fe700 1 -- 172.21.15.115:0/2626175300 <== mon.2 v2:172.21.15.115:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 4097+0+0 (secure 0 0 0) 0x7fbba40bf920 con 0x7fbbac0fc130 2022-05-13T23:10:29.044 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:10:29.022+0000 7fbbaaffd700 1 --2- 172.21.15.115:0/2626175300 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7fbb98096ce0 0x7fbb98099190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:10:29.045 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:10:29.023+0000 7fbbaaffd700 1 --2- 172.21.15.115:0/2626175300 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7fbb98096ce0 0x7fbb98099190 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fbbac0fb600 tx=0x7fbb9c0074f0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-05-13T23:10:29.045 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:10:29.026+0000 7fbba37fe700 1 -- 172.21.15.115:0/2626175300 <== mon.2 v2:172.21.15.115:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+165215 (secure 0 0 0) 0x7fbba408dc70 con 0x7fbbac0fc130 2022-05-13T23:10:29.205 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:10:29.204+0000 7fbbb16f6700 1 -- 172.21.15.115:0/2626175300 --> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi115:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) v1 -- 0x7fbbac05c920 con 0x7fbb98096ce0 2022-05-13T23:10:29.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:10:29 smithi115 ceph-mon[78593]: osdmap e34: 5 total, 5 up, 5 in 2022-05-13T23:10:29.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:10:29 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2022-05-13T23:10:29.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:10:29 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2022-05-13T23:10:29.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:10:29 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-13T23:10:29.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:10:29 smithi047 ceph-mon[77147]: osdmap e34: 5 total, 5 up, 5 in 2022-05-13T23:10:29.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:10:29 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2022-05-13T23:10:29.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:10:29 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2022-05-13T23:10:29.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:10:29 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-13T23:10:29.823 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:10:29 smithi002 ceph-mon[71816]: osdmap e34: 5 total, 5 up, 5 in 2022-05-13T23:10:29.824 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:10:29 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2022-05-13T23:10:29.824 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:10:29 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2022-05-13T23:10:29.825 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:10:29 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-13T23:10:30.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:10:30 smithi047 ceph-mon[77147]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2022-05-13T23:10:30.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:10:30 smithi047 ceph-mon[77147]: from='client.24251 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi115:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2022-05-13T23:10:30.822 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:10:30 smithi002 ceph-mon[71816]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2022-05-13T23:10:30.823 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:10:30 smithi002 ceph-mon[71816]: from='client.24251 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi115:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2022-05-13T23:10:30.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:10:30 smithi115 ceph-mon[78593]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2022-05-13T23:10:30.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:10:30 smithi115 ceph-mon[78593]: from='client.24251 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi115:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2022-05-13T23:10:32.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:10:32 smithi047 ceph-mon[77147]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2022-05-13T23:10:32.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:10:32 smithi047 ceph-mon[77147]: from='client.? 172.21.15.115:0/2562660907' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c18566cf-5403-4057-b2ee-aa298e87e8e9"}]: dispatch 2022-05-13T23:10:32.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:10:32 smithi047 ceph-mon[77147]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c18566cf-5403-4057-b2ee-aa298e87e8e9"}]: dispatch 2022-05-13T23:10:32.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:10:32 smithi047 ceph-mon[77147]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c18566cf-5403-4057-b2ee-aa298e87e8e9"}]': finished 2022-05-13T23:10:32.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:10:32 smithi047 ceph-mon[77147]: osdmap e35: 6 total, 5 up, 6 in 2022-05-13T23:10:32.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:10:32 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2022-05-13T23:10:32.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:10:32 smithi047 ceph-mon[77147]: from='client.? 172.21.15.115:0/3802648976' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2022-05-13T23:10:32.822 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:10:32 smithi002 ceph-mon[71816]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2022-05-13T23:10:32.823 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:10:32 smithi002 ceph-mon[71816]: from='client.? 172.21.15.115:0/2562660907' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c18566cf-5403-4057-b2ee-aa298e87e8e9"}]: dispatch 2022-05-13T23:10:32.823 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:10:32 smithi002 ceph-mon[71816]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c18566cf-5403-4057-b2ee-aa298e87e8e9"}]: dispatch 2022-05-13T23:10:32.823 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:10:32 smithi002 ceph-mon[71816]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c18566cf-5403-4057-b2ee-aa298e87e8e9"}]': finished 2022-05-13T23:10:32.823 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:10:32 smithi002 ceph-mon[71816]: osdmap e35: 6 total, 5 up, 6 in 2022-05-13T23:10:32.824 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:10:32 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2022-05-13T23:10:32.824 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:10:32 smithi002 ceph-mon[71816]: from='client.? 172.21.15.115:0/3802648976' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2022-05-13T23:10:32.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:10:32 smithi115 ceph-mon[78593]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2022-05-13T23:10:32.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:10:32 smithi115 ceph-mon[78593]: from='client.? 172.21.15.115:0/2562660907' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c18566cf-5403-4057-b2ee-aa298e87e8e9"}]: dispatch 2022-05-13T23:10:32.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:10:32 smithi115 ceph-mon[78593]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c18566cf-5403-4057-b2ee-aa298e87e8e9"}]: dispatch 2022-05-13T23:10:32.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:10:32 smithi115 ceph-mon[78593]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c18566cf-5403-4057-b2ee-aa298e87e8e9"}]': finished 2022-05-13T23:10:32.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:10:32 smithi115 ceph-mon[78593]: osdmap e35: 6 total, 5 up, 6 in 2022-05-13T23:10:32.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:10:32 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2022-05-13T23:10:32.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:10:32 smithi115 ceph-mon[78593]: from='client.? 172.21.15.115:0/3802648976' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2022-05-13T23:10:34.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:10:34 smithi047 ceph-mon[77147]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2022-05-13T23:10:34.822 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:10:34 smithi002 ceph-mon[71816]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2022-05-13T23:10:34.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:10:34 smithi115 ceph-mon[78593]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2022-05-13T23:10:36.747 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:10:36 smithi115 ceph-mon[78593]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2022-05-13T23:10:36.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:10:36 smithi047 ceph-mon[77147]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2022-05-13T23:10:36.822 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:10:36 smithi002 ceph-mon[71816]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2022-05-13T23:10:38.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:10:38 smithi115 ceph-mon[78593]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2022-05-13T23:10:38.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:10:38 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2022-05-13T23:10:38.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:10:38 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-13T23:10:38.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:10:38 smithi047 ceph-mon[77147]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2022-05-13T23:10:38.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:10:38 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2022-05-13T23:10:38.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:10:38 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-13T23:10:38.822 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:10:38 smithi002 ceph-mon[71816]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2022-05-13T23:10:38.822 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:10:38 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2022-05-13T23:10:38.823 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:10:38 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-13T23:10:39.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:10:39 smithi047 ceph-mon[77147]: Deploying daemon osd.5 on smithi115 2022-05-13T23:10:39.823 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:10:39 smithi002 ceph-mon[71816]: Deploying daemon osd.5 on smithi115 2022-05-13T23:10:39.852 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:10:39 smithi115 ceph-mon[78593]: Deploying daemon osd.5 on smithi115 2022-05-13T23:10:40.603 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:10:40 smithi115 ceph-mon[78593]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2022-05-13T23:10:40.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:10:40 smithi047 ceph-mon[77147]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2022-05-13T23:10:40.822 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:10:40 smithi002 ceph-mon[71816]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2022-05-13T23:10:41.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:10:41 smithi115 ceph-mon[78593]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2022-05-13T23:10:41.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:10:41 smithi047 ceph-mon[77147]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2022-05-13T23:10:41.822 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:10:41 smithi002 ceph-mon[71816]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2022-05-13T23:10:44.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:10:43 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:10:44.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:10:43 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-13T23:10:44.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:10:43 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-13T23:10:44.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:10:43 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-13T23:10:44.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:10:43 smithi115 ceph-mon[78593]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2022-05-13T23:10:44.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:10:43 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:10:44.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:10:43 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-13T23:10:44.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:10:43 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-13T23:10:44.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:10:43 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-13T23:10:44.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:10:43 smithi047 ceph-mon[77147]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2022-05-13T23:10:44.323 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:10:43 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:10:44.324 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:10:43 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-13T23:10:44.324 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:10:43 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-13T23:10:44.324 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:10:43 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-13T23:10:44.325 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:10:43 smithi002 ceph-mon[71816]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2022-05-13T23:10:45.911 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:10:45.910+0000 7fbba37fe700 1 -- 172.21.15.115:0/2626175300 <== mgr.14148 v2:172.21.15.2:6800/2788339627 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7fbbac05c920 con 0x7fbb98096ce0 2022-05-13T23:10:45.912 INFO:teuthology.orchestra.run.smithi115.stdout:Created osd(s) 5 on host 'smithi115' 2022-05-13T23:10:45.913 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:10:45.911+0000 7fbbb16f6700 1 -- 172.21.15.115:0/2626175300 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7fbb98096ce0 msgr2=0x7fbb98099190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:10:45.914 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:10:45.911+0000 7fbbb16f6700 1 --2- 172.21.15.115:0/2626175300 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7fbb98096ce0 0x7fbb98099190 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fbbac0fb600 tx=0x7fbb9c0074f0 comp rx=0 tx=0).stop 2022-05-13T23:10:45.914 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:10:45.912+0000 7fbbb16f6700 1 -- 172.21.15.115:0/2626175300 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fbbac0fc130 msgr2=0x7fbbac1135f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:10:45.914 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:10:45.912+0000 7fbbb16f6700 1 --2- 172.21.15.115:0/2626175300 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fbbac0fc130 0x7fbbac1135f0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fbba4009d60 tx=0x7fbba4009cb0 comp rx=0 tx=0).stop 2022-05-13T23:10:45.914 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:10:45.912+0000 7fbbb16f6700 1 -- 172.21.15.115:0/2626175300 shutdown_connections 2022-05-13T23:10:45.915 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:10:45.912+0000 7fbbb16f6700 1 --2- 172.21.15.115:0/2626175300 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fbbac0fc130 0x7fbbac1135f0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:10:45.915 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:10:45.912+0000 7fbbb16f6700 1 --2- 172.21.15.115:0/2626175300 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fbbac0fa500 0x7fbbac064960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:10:45.915 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:10:45.912+0000 7fbbb16f6700 1 --2- 172.21.15.115:0/2626175300 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7fbb98096ce0 0x7fbb98099190 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:10:45.915 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:10:45.912+0000 7fbbb16f6700 1 --2- 172.21.15.115:0/2626175300 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fbbac0fb7a0 0x7fbbac110f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:10:45.915 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:10:45.912+0000 7fbbb16f6700 1 -- 172.21.15.115:0/2626175300 >> 172.21.15.115:0/2626175300 conn(0x7fbbac0f5cf0 msgr2=0x7fbbac0f7dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-05-13T23:10:45.916 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:10:45.912+0000 7fbbb16f6700 1 -- 172.21.15.115:0/2626175300 shutdown_connections 2022-05-13T23:10:45.916 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:10:45.912+0000 7fbbb16f6700 1 -- 172.21.15.115:0/2626175300 wait complete. 2022-05-13T23:10:46.367 DEBUG:teuthology.orchestra.run.smithi115:osd.5> sudo journalctl -f -n 0 -u ceph-16cf5f88-d311-11ec-be40-001a4aab830c@osd.5.service 2022-05-13T23:10:46.370 INFO:tasks.cephadm:Deploying osd.6 on smithi002 with /dev/vg_nvme/lv_2... 2022-05-13T23:10:46.370 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:23b0658391ce94ca40799469eec00c28b298816c shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 16cf5f88-d311-11ec-be40-001a4aab830c -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2022-05-13T23:10:46.394 INFO:journalctl@ceph.osd.5.smithi115.stdout:-- Logs begin at Fri 2022-05-13 22:45:29 UTC. -- 2022-05-13T23:10:46.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:10:46 smithi047 ceph-mon[77147]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2022-05-13T23:10:46.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:10:46 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:10:46.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:10:46 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:10:46.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:10:46 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:10:46.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:10:46 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-13T23:10:46.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:10:46 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-13T23:10:46.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:10:46 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-13T23:10:46.572 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:10:46 smithi002 ceph-mon[71816]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2022-05-13T23:10:46.573 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:10:46 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:10:46.573 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:10:46 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:10:46.573 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:10:46 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:10:46.573 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:10:46 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-13T23:10:46.574 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:10:46 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-13T23:10:46.574 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:10:46 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-13T23:10:46.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:10:46 smithi115 ceph-mon[78593]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2022-05-13T23:10:46.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:10:46 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:10:46.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:10:46 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:10:46.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:10:46 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:10:46.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:10:46 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-13T23:10:46.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:10:46 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-13T23:10:46.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:10:46 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-13T23:10:47.044 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/16cf5f88-d311-11ec-be40-001a4aab830c/mon.a/config 2022-05-13T23:10:47.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:10:46 smithi115 conmon[90393]: 2022-05-13T23:10:46.865+0000 7fa094e6f3c0 -1 osd.5 0 log_to_monitors true 2022-05-13T23:10:47.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:10:47 smithi047 ceph-mon[77147]: from='osd.5 [v2:172.21.15.115:6808/2122391993,v1:172.21.15.115:6809/2122391993]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2022-05-13T23:10:47.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:10:47 smithi047 ceph-mon[77147]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2022-05-13T23:10:47.573 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:10:47 smithi002 ceph-mon[71816]: from='osd.5 [v2:172.21.15.115:6808/2122391993,v1:172.21.15.115:6809/2122391993]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2022-05-13T23:10:47.573 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:10:47 smithi002 ceph-mon[71816]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2022-05-13T23:10:47.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:10:47 smithi115 ceph-mon[78593]: from='osd.5 [v2:172.21.15.115:6808/2122391993,v1:172.21.15.115:6809/2122391993]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2022-05-13T23:10:47.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:10:47 smithi115 ceph-mon[78593]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2022-05-13T23:10:48.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:10:48 smithi047 ceph-mon[77147]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2022-05-13T23:10:48.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:10:48 smithi047 ceph-mon[77147]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2022-05-13T23:10:48.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:10:48 smithi047 ceph-mon[77147]: osdmap e36: 6 total, 5 up, 6 in 2022-05-13T23:10:48.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:10:48 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2022-05-13T23:10:48.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:10:48 smithi047 ceph-mon[77147]: from='osd.5 [v2:172.21.15.115:6808/2122391993,v1:172.21.15.115:6809/2122391993]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi115", "root=default"]}]: dispatch 2022-05-13T23:10:48.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:10:48 smithi047 ceph-mon[77147]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi115", "root=default"]}]: dispatch 2022-05-13T23:10:48.572 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:10:48 smithi002 ceph-mon[71816]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2022-05-13T23:10:48.573 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:10:48 smithi002 ceph-mon[71816]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2022-05-13T23:10:48.573 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:10:48 smithi002 ceph-mon[71816]: osdmap e36: 6 total, 5 up, 6 in 2022-05-13T23:10:48.574 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:10:48 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2022-05-13T23:10:48.574 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:10:48 smithi002 ceph-mon[71816]: from='osd.5 [v2:172.21.15.115:6808/2122391993,v1:172.21.15.115:6809/2122391993]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi115", "root=default"]}]: dispatch 2022-05-13T23:10:48.574 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:10:48 smithi002 ceph-mon[71816]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi115", "root=default"]}]: dispatch 2022-05-13T23:10:48.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:10:48 smithi115 conmon[90393]: 2022-05-13T23:10:48.295+0000 7fa08b873700 -1 osd.5 0 waiting for initial osdmap 2022-05-13T23:10:48.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:10:48 smithi115 conmon[90393]: 2022-05-13T23:10:48.303+0000 7fa084205700 -1 osd.5 37 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2022-05-13T23:10:48.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:10:48 smithi115 ceph-mon[78593]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2022-05-13T23:10:48.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:10:48 smithi115 ceph-mon[78593]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2022-05-13T23:10:48.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:10:48 smithi115 ceph-mon[78593]: osdmap e36: 6 total, 5 up, 6 in 2022-05-13T23:10:48.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:10:48 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2022-05-13T23:10:48.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:10:48 smithi115 ceph-mon[78593]: from='osd.5 [v2:172.21.15.115:6808/2122391993,v1:172.21.15.115:6809/2122391993]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi115", "root=default"]}]: dispatch 2022-05-13T23:10:48.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:10:48 smithi115 ceph-mon[78593]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi115", "root=default"]}]: dispatch 2022-05-13T23:10:49.468 INFO:teuthology.orchestra.run.smithi002.stderr:--> Zapping: /dev/vg_nvme/lv_2 2022-05-13T23:10:49.469 INFO:teuthology.orchestra.run.smithi002.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2022-05-13T23:10:49.469 INFO:teuthology.orchestra.run.smithi002.stderr: stderr: 10+0 records in 2022-05-13T23:10:49.469 INFO:teuthology.orchestra.run.smithi002.stderr:10+0 records out 2022-05-13T23:10:49.469 INFO:teuthology.orchestra.run.smithi002.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0581849 s, 180 MB/s 2022-05-13T23:10:49.469 INFO:teuthology.orchestra.run.smithi002.stderr:--> Zapping successful for: 2022-05-13T23:10:49.572 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:10:49 smithi002 ceph-mon[71816]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi115", "root=default"]}]': finished 2022-05-13T23:10:49.572 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:10:49 smithi002 ceph-mon[71816]: osdmap e37: 6 total, 5 up, 6 in 2022-05-13T23:10:49.573 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:10:49 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2022-05-13T23:10:49.573 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:10:49 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2022-05-13T23:10:49.573 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:10:49 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:10:49.573 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:10:49 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-05-13T23:10:49.574 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:10:49 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-05-13T23:10:49.574 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:10:49 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:10:49.574 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:10:49 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:10:49.574 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:10:49 smithi002 ceph-mon[71816]: osd.5 [v2:172.21.15.115:6808/2122391993,v1:172.21.15.115:6809/2122391993] boot 2022-05-13T23:10:49.575 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:10:49 smithi002 ceph-mon[71816]: osdmap e38: 6 total, 6 up, 6 in 2022-05-13T23:10:49.575 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:10:49 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2022-05-13T23:10:49.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:10:49 smithi115 ceph-mon[78593]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi115", "root=default"]}]': finished 2022-05-13T23:10:49.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:10:49 smithi115 ceph-mon[78593]: osdmap e37: 6 total, 5 up, 6 in 2022-05-13T23:10:49.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:10:49 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2022-05-13T23:10:49.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:10:49 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2022-05-13T23:10:49.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:10:49 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:10:49.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:10:49 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-05-13T23:10:49.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:10:49 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-05-13T23:10:49.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:10:49 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:10:49.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:10:49 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:10:49.612 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:10:49 smithi115 ceph-mon[78593]: osd.5 [v2:172.21.15.115:6808/2122391993,v1:172.21.15.115:6809/2122391993] boot 2022-05-13T23:10:49.612 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:10:49 smithi115 ceph-mon[78593]: osdmap e38: 6 total, 6 up, 6 in 2022-05-13T23:10:49.612 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:10:49 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2022-05-13T23:10:49.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:10:49 smithi047 ceph-mon[77147]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi115", "root=default"]}]': finished 2022-05-13T23:10:49.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:10:49 smithi047 ceph-mon[77147]: osdmap e37: 6 total, 5 up, 6 in 2022-05-13T23:10:49.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:10:49 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2022-05-13T23:10:49.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:10:49 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2022-05-13T23:10:49.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:10:49 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:10:49.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:10:49 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-05-13T23:10:49.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:10:49 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-05-13T23:10:49.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:10:49 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:10:49.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:10:49 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:10:49.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:10:49 smithi047 ceph-mon[77147]: osd.5 [v2:172.21.15.115:6808/2122391993,v1:172.21.15.115:6809/2122391993] boot 2022-05-13T23:10:49.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:10:49 smithi047 ceph-mon[77147]: osdmap e38: 6 total, 6 up, 6 in 2022-05-13T23:10:49.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:10:49 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2022-05-13T23:10:50.104 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:23b0658391ce94ca40799469eec00c28b298816c shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 16cf5f88-d311-11ec-be40-001a4aab830c -- ceph orch daemon add osd smithi002:vg_nvme/lv_2 2022-05-13T23:10:50.572 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:10:50 smithi002 ceph-mon[71816]: purged_snaps scrub starts 2022-05-13T23:10:50.573 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:10:50 smithi002 ceph-mon[71816]: purged_snaps scrub ok 2022-05-13T23:10:50.573 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:10:50 smithi002 ceph-mon[71816]: Detected new or changed devices on smithi115 2022-05-13T23:10:50.573 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:10:50 smithi002 ceph-mon[71816]: Adjusting osd_memory_target on smithi115 to 8586M 2022-05-13T23:10:50.574 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:10:50 smithi002 ceph-mon[71816]: pgmap v139: 1 pgs: 1 peering; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2022-05-13T23:10:50.574 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:10:50 smithi002 ceph-mon[71816]: osdmap e39: 6 total, 6 up, 6 in 2022-05-13T23:10:50.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:10:50 smithi115 ceph-mon[78593]: purged_snaps scrub starts 2022-05-13T23:10:50.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:10:50 smithi115 ceph-mon[78593]: purged_snaps scrub ok 2022-05-13T23:10:50.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:10:50 smithi115 ceph-mon[78593]: Detected new or changed devices on smithi115 2022-05-13T23:10:50.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:10:50 smithi115 ceph-mon[78593]: Adjusting osd_memory_target on smithi115 to 8586M 2022-05-13T23:10:50.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:10:50 smithi115 ceph-mon[78593]: pgmap v139: 1 pgs: 1 peering; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2022-05-13T23:10:50.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:10:50 smithi115 ceph-mon[78593]: osdmap e39: 6 total, 6 up, 6 in 2022-05-13T23:10:50.783 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/16cf5f88-d311-11ec-be40-001a4aab830c/mon.a/config 2022-05-13T23:10:50.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:10:50 smithi047 ceph-mon[77147]: purged_snaps scrub starts 2022-05-13T23:10:50.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:10:50 smithi047 ceph-mon[77147]: purged_snaps scrub ok 2022-05-13T23:10:50.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:10:50 smithi047 ceph-mon[77147]: Detected new or changed devices on smithi115 2022-05-13T23:10:50.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:10:50 smithi047 ceph-mon[77147]: Adjusting osd_memory_target on smithi115 to 8586M 2022-05-13T23:10:50.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:10:50 smithi047 ceph-mon[77147]: pgmap v139: 1 pgs: 1 peering; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2022-05-13T23:10:50.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:10:50 smithi047 ceph-mon[77147]: osdmap e39: 6 total, 6 up, 6 in 2022-05-13T23:10:52.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:10:52 smithi047 ceph-mon[77147]: pgmap v142: 1 pgs: 1 peering; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2022-05-13T23:10:52.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:10:52 smithi047 ceph-mon[77147]: osdmap e40: 6 total, 6 up, 6 in 2022-05-13T23:10:52.572 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:10:52 smithi002 ceph-mon[71816]: pgmap v142: 1 pgs: 1 peering; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2022-05-13T23:10:52.573 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:10:52 smithi002 ceph-mon[71816]: osdmap e40: 6 total, 6 up, 6 in 2022-05-13T23:10:52.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:10:52 smithi115 ceph-mon[78593]: pgmap v142: 1 pgs: 1 peering; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2022-05-13T23:10:52.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:10:52 smithi115 ceph-mon[78593]: osdmap e40: 6 total, 6 up, 6 in 2022-05-13T23:10:52.706 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:10:52.704+0000 7fc529c5c700 1 -- 172.21.15.2:0/2681385335 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc5240ff2b0 msgr2=0x7fc524105b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:10:52.707 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:10:52.704+0000 7fc529c5c700 1 --2- 172.21.15.2:0/2681385335 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc5240ff2b0 0x7fc524105b70 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fc51c009c30 tx=0x7fc51c004c50 comp rx=0 tx=0).stop 2022-05-13T23:10:52.707 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:10:52.705+0000 7fc529c5c700 1 -- 172.21.15.2:0/2681385335 shutdown_connections 2022-05-13T23:10:52.707 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:10:52.705+0000 7fc529c5c700 1 --2- 172.21.15.2:0/2681385335 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc5240ff2b0 0x7fc524105b70 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:10:52.708 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:10:52.705+0000 7fc529c5c700 1 --2- 172.21.15.2:0/2681385335 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fc5240fd680 0x7fc5240fda90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:10:52.708 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:10:52.705+0000 7fc529c5c700 1 --2- 172.21.15.2:0/2681385335 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc5240fe920 0x7fc5240fed70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:10:52.708 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:10:52.705+0000 7fc529c5c700 1 -- 172.21.15.2:0/2681385335 >> 172.21.15.2:0/2681385335 conn(0x7fc5240f8e70 msgr2=0x7fc5240fb250 unknown :-1 s=STATE_NONE l=0).mark_down 2022-05-13T23:10:52.708 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:10:52.705+0000 7fc529c5c700 1 -- 172.21.15.2:0/2681385335 shutdown_connections 2022-05-13T23:10:52.709 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:10:52.705+0000 7fc529c5c700 1 -- 172.21.15.2:0/2681385335 wait complete. 2022-05-13T23:10:52.712 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:10:52.706+0000 7fc529c5c700 1 Processor -- start 2022-05-13T23:10:52.713 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:10:52.706+0000 7fc529c5c700 1 -- start start 2022-05-13T23:10:52.713 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:10:52.706+0000 7fc529c5c700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc5240fd680 0x7fc52410e270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:10:52.713 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:10:52.706+0000 7fc529c5c700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc5240fe920 0x7fc52410e7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:10:52.713 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:10:52.706+0000 7fc529c5c700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fc5240ff2b0 0x7fc524115840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:10:52.714 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:10:52.706+0000 7fc529c5c700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc524115d80 con 0x7fc5240fd680 2022-05-13T23:10:52.714 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:10:52.706+0000 7fc529c5c700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc524115ef0 con 0x7fc5240fe920 2022-05-13T23:10:52.714 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:10:52.706+0000 7fc529c5c700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5241160b0 con 0x7fc5240ff2b0 2022-05-13T23:10:52.714 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:10:52.706+0000 7fc528c5a700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc5240fd680 0x7fc52410e270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:10:52.715 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:10:52.707+0000 7fc523fff700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc5240fe920 0x7fc52410e7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:10:52.715 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:10:52.707+0000 7fc528c5a700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc5240fd680 0x7fc52410e270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:54944/0 (socket says 172.21.15.2:54944) 2022-05-13T23:10:52.715 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:10:52.707+0000 7fc528c5a700 1 -- 172.21.15.2:0/3013429790 learned_addr learned my addr 172.21.15.2:0/3013429790 (peer_addr_for_me v2:172.21.15.2:0/0) 2022-05-13T23:10:52.715 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:10:52.707+0000 7fc52945b700 1 --2- 172.21.15.2:0/3013429790 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fc5240ff2b0 0x7fc524115840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:10:52.715 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:10:52.707+0000 7fc523fff700 1 -- 172.21.15.2:0/3013429790 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fc5240ff2b0 msgr2=0x7fc524115840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:10:52.716 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:10:52.707+0000 7fc523fff700 1 --2- 172.21.15.2:0/3013429790 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fc5240ff2b0 0x7fc524115840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:10:52.716 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:10:52.707+0000 7fc523fff700 1 -- 172.21.15.2:0/3013429790 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc5240fd680 msgr2=0x7fc52410e270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:10:52.716 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:10:52.707+0000 7fc523fff700 1 --2- 172.21.15.2:0/3013429790 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc5240fd680 0x7fc52410e270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:10:52.716 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:10:52.707+0000 7fc523fff700 1 -- 172.21.15.2:0/3013429790 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc518005040 con 0x7fc5240fe920 2022-05-13T23:10:52.717 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:10:52.708+0000 7fc523fff700 1 --2- 172.21.15.2:0/3013429790 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc5240fe920 0x7fc52410e7b0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fc51000e8f0 tx=0x7fc51000b040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-05-13T23:10:52.717 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:10:52.708+0000 7fc5217fa700 1 -- 172.21.15.2:0/3013429790 <== mon.1 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc510007b40 con 0x7fc5240fe920 2022-05-13T23:10:52.717 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:10:52.708+0000 7fc5217fa700 1 -- 172.21.15.2:0/3013429790 <== mon.1 v2:172.21.15.47:3300/0 2 ==== config(26 keys) v1 ==== 1087+0+0 (secure 0 0 0) 0x7fc51000f040 con 0x7fc5240fe920 2022-05-13T23:10:52.717 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:10:52.708+0000 7fc529c5c700 1 -- 172.21.15.2:0/3013429790 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc51c0097c0 con 0x7fc5240fe920 2022-05-13T23:10:52.717 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:10:52.708+0000 7fc5217fa700 1 -- 172.21.15.2:0/3013429790 <== mon.1 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc510014950 con 0x7fc5240fe920 2022-05-13T23:10:52.718 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:10:52.708+0000 7fc529c5c700 1 -- 172.21.15.2:0/3013429790 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc524116770 con 0x7fc5240fe920 2022-05-13T23:10:52.719 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:10:52.709+0000 7fc529c5c700 1 -- 172.21.15.2:0/3013429790 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc524108e60 con 0x7fc5240fe920 2022-05-13T23:10:52.719 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:10:52.711+0000 7fc5217fa700 1 -- 172.21.15.2:0/3013429790 <== mon.1 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 16) v1 ==== 125136+0+0 (secure 0 0 0) 0x7fc51001b030 con 0x7fc5240fe920 2022-05-13T23:10:52.721 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:10:52.712+0000 7fc5217fa700 1 --2- 172.21.15.2:0/3013429790 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7fc514096ce0 0x7fc514099190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:10:52.721 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:10:52.712+0000 7fc5217fa700 1 -- 172.21.15.2:0/3013429790 <== mon.1 v2:172.21.15.47:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4529+0+0 (secure 0 0 0) 0x7fc5100bde10 con 0x7fc5240fe920 2022-05-13T23:10:52.721 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:10:52.713+0000 7fc528c5a700 1 --2- 172.21.15.2:0/3013429790 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7fc514096ce0 0x7fc514099190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:10:52.721 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:10:52.713+0000 7fc528c5a700 1 --2- 172.21.15.2:0/3013429790 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7fc514096ce0 0x7fc514099190 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fc518004910 tx=0x7fc51800b040 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-05-13T23:10:52.722 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:10:52.714+0000 7fc5217fa700 1 -- 172.21.15.2:0/3013429790 <== mon.1 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+165215 (secure 0 0 0) 0x7fc51004b570 con 0x7fc5240fe920 2022-05-13T23:10:52.915 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:10:52.913+0000 7fc529c5c700 1 -- 172.21.15.2:0/3013429790 --> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi002:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) v1 -- 0x7fc52402a530 con 0x7fc514096ce0 2022-05-13T23:10:53.322 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:10:53 smithi002 ceph-mon[71816]: osdmap e41: 6 total, 6 up, 6 in 2022-05-13T23:10:53.323 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:10:53 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2022-05-13T23:10:53.323 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:10:53 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2022-05-13T23:10:53.323 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:10:53 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-13T23:10:53.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:10:53 smithi047 ceph-mon[77147]: osdmap e41: 6 total, 6 up, 6 in 2022-05-13T23:10:53.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:10:53 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2022-05-13T23:10:53.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:10:53 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2022-05-13T23:10:53.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:10:53 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-13T23:10:53.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:10:53 smithi115 ceph-mon[78593]: osdmap e41: 6 total, 6 up, 6 in 2022-05-13T23:10:53.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:10:53 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2022-05-13T23:10:53.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:10:53 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2022-05-13T23:10:53.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:10:53 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-13T23:10:54.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:10:54 smithi047 ceph-mon[77147]: from='client.24274 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi002:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2022-05-13T23:10:54.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:10:54 smithi047 ceph-mon[77147]: pgmap v145: 1 pgs: 1 peering; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2022-05-13T23:10:54.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:10:54 smithi047 ceph-mon[77147]: osdmap e42: 6 total, 6 up, 6 in 2022-05-13T23:10:54.573 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:10:54 smithi002 ceph-mon[71816]: from='client.24274 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi002:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2022-05-13T23:10:54.574 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:10:54 smithi002 ceph-mon[71816]: pgmap v145: 1 pgs: 1 peering; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2022-05-13T23:10:54.574 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:10:54 smithi002 ceph-mon[71816]: osdmap e42: 6 total, 6 up, 6 in 2022-05-13T23:10:54.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:10:54 smithi115 ceph-mon[78593]: from='client.24274 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi002:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2022-05-13T23:10:54.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:10:54 smithi115 ceph-mon[78593]: pgmap v145: 1 pgs: 1 peering; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2022-05-13T23:10:54.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:10:54 smithi115 ceph-mon[78593]: osdmap e42: 6 total, 6 up, 6 in 2022-05-13T23:10:56.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:10:56 smithi047 ceph-mon[77147]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 104 KiB/s, 0 objects/s recovering 2022-05-13T23:10:56.572 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:10:56 smithi002 ceph-mon[71816]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 104 KiB/s, 0 objects/s recovering 2022-05-13T23:10:56.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:10:56 smithi115 ceph-mon[78593]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 104 KiB/s, 0 objects/s recovering 2022-05-13T23:10:57.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:10:57 smithi047 ceph-mon[77147]: from='client.? 172.21.15.2:0/2919162683' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "38ec7b6b-30f4-4d97-ad7e-e160ae59c843"}]: dispatch 2022-05-13T23:10:57.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:10:57 smithi047 ceph-mon[77147]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "38ec7b6b-30f4-4d97-ad7e-e160ae59c843"}]: dispatch 2022-05-13T23:10:57.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:10:57 smithi047 ceph-mon[77147]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "38ec7b6b-30f4-4d97-ad7e-e160ae59c843"}]': finished 2022-05-13T23:10:57.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:10:57 smithi047 ceph-mon[77147]: osdmap e43: 7 total, 6 up, 7 in 2022-05-13T23:10:57.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:10:57 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2022-05-13T23:10:57.573 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:10:57 smithi002 ceph-mon[71816]: from='client.? 172.21.15.2:0/2919162683' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "38ec7b6b-30f4-4d97-ad7e-e160ae59c843"}]: dispatch 2022-05-13T23:10:57.573 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:10:57 smithi002 ceph-mon[71816]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "38ec7b6b-30f4-4d97-ad7e-e160ae59c843"}]: dispatch 2022-05-13T23:10:57.573 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:10:57 smithi002 ceph-mon[71816]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "38ec7b6b-30f4-4d97-ad7e-e160ae59c843"}]': finished 2022-05-13T23:10:57.573 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:10:57 smithi002 ceph-mon[71816]: osdmap e43: 7 total, 6 up, 7 in 2022-05-13T23:10:57.574 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:10:57 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2022-05-13T23:10:57.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:10:57 smithi115 ceph-mon[78593]: from='client.? 172.21.15.2:0/2919162683' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "38ec7b6b-30f4-4d97-ad7e-e160ae59c843"}]: dispatch 2022-05-13T23:10:57.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:10:57 smithi115 ceph-mon[78593]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "38ec7b6b-30f4-4d97-ad7e-e160ae59c843"}]: dispatch 2022-05-13T23:10:57.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:10:57 smithi115 ceph-mon[78593]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "38ec7b6b-30f4-4d97-ad7e-e160ae59c843"}]': finished 2022-05-13T23:10:57.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:10:57 smithi115 ceph-mon[78593]: osdmap e43: 7 total, 6 up, 7 in 2022-05-13T23:10:57.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:10:57 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2022-05-13T23:10:58.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:10:58 smithi047 ceph-mon[77147]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 86 KiB/s, 0 objects/s recovering 2022-05-13T23:10:58.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:10:58 smithi047 ceph-mon[77147]: from='client.? 172.21.15.2:0/2210373369' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2022-05-13T23:10:58.572 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:10:58 smithi002 ceph-mon[71816]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 86 KiB/s, 0 objects/s recovering 2022-05-13T23:10:58.573 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:10:58 smithi002 ceph-mon[71816]: from='client.? 172.21.15.2:0/2210373369' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2022-05-13T23:10:58.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:10:58 smithi115 ceph-mon[78593]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 86 KiB/s, 0 objects/s recovering 2022-05-13T23:10:58.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:10:58 smithi115 ceph-mon[78593]: from='client.? 172.21.15.2:0/2210373369' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2022-05-13T23:11:00.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:11:00 smithi047 ceph-mon[77147]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 74 KiB/s, 0 objects/s recovering 2022-05-13T23:11:00.572 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:11:00 smithi002 ceph-mon[71816]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 74 KiB/s, 0 objects/s recovering 2022-05-13T23:11:00.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:11:00 smithi115 ceph-mon[78593]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 74 KiB/s, 0 objects/s recovering 2022-05-13T23:11:02.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:11:02 smithi047 ceph-mon[77147]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 65 KiB/s, 0 objects/s recovering 2022-05-13T23:11:02.573 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:11:02 smithi002 ceph-mon[71816]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 65 KiB/s, 0 objects/s recovering 2022-05-13T23:11:02.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:11:02 smithi115 ceph-mon[78593]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 65 KiB/s, 0 objects/s recovering 2022-05-13T23:11:04.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:11:04 smithi047 ceph-mon[77147]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 52 KiB/s, 0 objects/s recovering 2022-05-13T23:11:04.572 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:11:04 smithi002 ceph-mon[71816]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 52 KiB/s, 0 objects/s recovering 2022-05-13T23:11:04.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:11:04 smithi115 ceph-mon[78593]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 52 KiB/s, 0 objects/s recovering 2022-05-13T23:11:05.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:11:05 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-05-13T23:11:05.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:11:05 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-05-13T23:11:05.572 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:11:05 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-05-13T23:11:05.573 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:11:05 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-05-13T23:11:05.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:11:05 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-05-13T23:11:05.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:11:05 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-05-13T23:11:06.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:11:06 smithi047 ceph-mon[77147]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2022-05-13T23:11:06.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:11:06 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2022-05-13T23:11:06.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:11:06 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-13T23:11:06.572 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:11:06 smithi002 ceph-mon[71816]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2022-05-13T23:11:06.573 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:11:06 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2022-05-13T23:11:06.573 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:11:06 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-13T23:11:06.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:11:06 smithi115 ceph-mon[78593]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2022-05-13T23:11:06.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:11:06 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2022-05-13T23:11:06.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:11:06 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-13T23:11:07.530 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:11:07 smithi002 ceph-mon[71816]: Deploying daemon osd.6 on smithi002 2022-05-13T23:11:07.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:11:07 smithi047 ceph-mon[77147]: Deploying daemon osd.6 on smithi002 2022-05-13T23:11:07.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:11:07 smithi115 ceph-mon[78593]: Deploying daemon osd.6 on smithi002 2022-05-13T23:11:08.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:11:08 smithi047 ceph-mon[77147]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2022-05-13T23:11:08.573 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:11:08 smithi002 ceph-mon[71816]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2022-05-13T23:11:08.612 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:11:08 smithi115 ceph-mon[78593]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2022-05-13T23:11:10.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:11:10 smithi047 ceph-mon[77147]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2022-05-13T23:11:10.573 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:11:10 smithi002 ceph-mon[71816]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2022-05-13T23:11:10.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:11:10 smithi115 ceph-mon[78593]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2022-05-13T23:11:12.414 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:11:12 smithi002 ceph-mon[71816]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2022-05-13T23:11:12.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:11:12 smithi047 ceph-mon[77147]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2022-05-13T23:11:12.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:11:12 smithi115 ceph-mon[78593]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2022-05-13T23:11:14.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:11:14 smithi047 ceph-mon[77147]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2022-05-13T23:11:14.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:11:14 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:11:14.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:11:14 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-13T23:11:14.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:11:14 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-13T23:11:14.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:11:14 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-13T23:11:14.572 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:11:14 smithi002 ceph-mon[71816]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2022-05-13T23:11:14.573 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:11:14 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:11:14.573 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:11:14 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-13T23:11:14.573 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:11:14 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-13T23:11:14.573 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:11:14 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-13T23:11:14.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:11:14 smithi115 ceph-mon[78593]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2022-05-13T23:11:14.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:11:14 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:11:14.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:11:14 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-13T23:11:14.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:11:14 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-13T23:11:14.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:11:14 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-13T23:11:16.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:11:16 smithi047 ceph-mon[77147]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2022-05-13T23:11:16.572 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:11:16 smithi002 ceph-mon[71816]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2022-05-13T23:11:16.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:11:16 smithi115 ceph-mon[78593]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2022-05-13T23:11:18.323 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:11:18 smithi002 ceph-mon[71816]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2022-05-13T23:11:18.323 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:11:18 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:11:18.323 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:11:18 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:11:18.324 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:11:18 smithi002 ceph-mon[71816]: from='osd.6 [v2:172.21.15.2:6818/1892198790,v1:172.21.15.2:6819/1892198790]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2022-05-13T23:11:18.339 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:11:18.337+0000 7fc5217fa700 1 -- 172.21.15.2:0/3013429790 <== mgr.14148 v2:172.21.15.2:6800/2788339627 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7fc52402a530 con 0x7fc514096ce0 2022-05-13T23:11:18.340 INFO:teuthology.orchestra.run.smithi002.stdout:Created osd(s) 6 on host 'smithi002' 2022-05-13T23:11:18.342 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:11:18.340+0000 7fc529c5c700 1 -- 172.21.15.2:0/3013429790 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7fc514096ce0 msgr2=0x7fc514099190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:11:18.342 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:11:18.340+0000 7fc529c5c700 1 --2- 172.21.15.2:0/3013429790 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7fc514096ce0 0x7fc514099190 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fc518004910 tx=0x7fc51800b040 comp rx=0 tx=0).stop 2022-05-13T23:11:18.343 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:11:18.340+0000 7fc529c5c700 1 -- 172.21.15.2:0/3013429790 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc5240fe920 msgr2=0x7fc52410e7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:11:18.343 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:11:18.340+0000 7fc529c5c700 1 --2- 172.21.15.2:0/3013429790 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc5240fe920 0x7fc52410e7b0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fc51000e8f0 tx=0x7fc51000b040 comp rx=0 tx=0).stop 2022-05-13T23:11:18.343 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:11:18.340+0000 7fc529c5c700 1 -- 172.21.15.2:0/3013429790 shutdown_connections 2022-05-13T23:11:18.344 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:11:18.340+0000 7fc529c5c700 1 --2- 172.21.15.2:0/3013429790 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fc5240ff2b0 0x7fc524115840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:11:18.344 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:11:18.340+0000 7fc529c5c700 1 --2- 172.21.15.2:0/3013429790 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7fc514096ce0 0x7fc514099190 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:11:18.344 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:11:18.340+0000 7fc529c5c700 1 --2- 172.21.15.2:0/3013429790 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc5240fd680 0x7fc52410e270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:11:18.344 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:11:18.340+0000 7fc529c5c700 1 --2- 172.21.15.2:0/3013429790 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc5240fe920 0x7fc52410e7b0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:11:18.345 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:11:18.340+0000 7fc529c5c700 1 -- 172.21.15.2:0/3013429790 >> 172.21.15.2:0/3013429790 conn(0x7fc5240f8e70 msgr2=0x7fc524101dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-05-13T23:11:18.345 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:11:18.340+0000 7fc529c5c700 1 -- 172.21.15.2:0/3013429790 shutdown_connections 2022-05-13T23:11:18.345 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:11:18.340+0000 7fc529c5c700 1 -- 172.21.15.2:0/3013429790 wait complete. 2022-05-13T23:11:18.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:11:18 smithi047 ceph-mon[77147]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2022-05-13T23:11:18.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:11:18 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:11:18.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:11:18 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:11:18.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:11:18 smithi047 ceph-mon[77147]: from='osd.6 [v2:172.21.15.2:6818/1892198790,v1:172.21.15.2:6819/1892198790]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2022-05-13T23:11:18.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:11:18 smithi115 ceph-mon[78593]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2022-05-13T23:11:18.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:11:18 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:11:18.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:11:18 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:11:18.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:11:18 smithi115 ceph-mon[78593]: from='osd.6 [v2:172.21.15.2:6818/1892198790,v1:172.21.15.2:6819/1892198790]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2022-05-13T23:11:18.871 DEBUG:teuthology.orchestra.run.smithi002:osd.6> sudo journalctl -f -n 0 -u ceph-16cf5f88-d311-11ec-be40-001a4aab830c@osd.6.service 2022-05-13T23:11:18.878 INFO:tasks.cephadm:Deploying osd.7 on smithi047 with /dev/vg_nvme/lv_2... 2022-05-13T23:11:18.879 DEBUG:teuthology.orchestra.run.smithi047:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:23b0658391ce94ca40799469eec00c28b298816c shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 16cf5f88-d311-11ec-be40-001a4aab830c -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2022-05-13T23:11:18.957 INFO:journalctl@ceph.osd.6.smithi002.stdout:-- Logs begin at Fri 2022-05-13 22:48:11 UTC. -- 2022-05-13T23:11:19.342 INFO:teuthology.orchestra.run.smithi047.stderr:Inferring config /var/lib/ceph/16cf5f88-d311-11ec-be40-001a4aab830c/mon.b/config 2022-05-13T23:11:19.603 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:11:19 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:11:19.603 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:11:19 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-13T23:11:19.604 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:11:19 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-13T23:11:19.604 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:11:19 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-13T23:11:19.604 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:11:19 smithi047 ceph-mon[77147]: from='osd.6 [v2:172.21.15.2:6818/1892198790,v1:172.21.15.2:6819/1892198790]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2022-05-13T23:11:19.605 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:11:19 smithi047 ceph-mon[77147]: osdmap e44: 7 total, 6 up, 7 in 2022-05-13T23:11:19.608 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:11:19 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2022-05-13T23:11:19.608 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:11:19 smithi047 ceph-mon[77147]: from='osd.6 [v2:172.21.15.2:6818/1892198790,v1:172.21.15.2:6819/1892198790]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi002", "root=default"]}]: dispatch 2022-05-13T23:11:19.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:11:19 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:11:19.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:11:19 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-13T23:11:19.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:11:19 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-13T23:11:19.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:11:19 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-13T23:11:19.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:11:19 smithi115 ceph-mon[78593]: from='osd.6 [v2:172.21.15.2:6818/1892198790,v1:172.21.15.2:6819/1892198790]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2022-05-13T23:11:19.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:11:19 smithi115 ceph-mon[78593]: osdmap e44: 7 total, 6 up, 7 in 2022-05-13T23:11:19.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:11:19 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2022-05-13T23:11:19.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:11:19 smithi115 ceph-mon[78593]: from='osd.6 [v2:172.21.15.2:6818/1892198790,v1:172.21.15.2:6819/1892198790]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi002", "root=default"]}]: dispatch 2022-05-13T23:11:19.621 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:11:19 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:11:19.621 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:11:19 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-13T23:11:19.621 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:11:19 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-13T23:11:19.622 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:11:19 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-13T23:11:19.622 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:11:19 smithi002 ceph-mon[71816]: from='osd.6 [v2:172.21.15.2:6818/1892198790,v1:172.21.15.2:6819/1892198790]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2022-05-13T23:11:19.622 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:11:19 smithi002 ceph-mon[71816]: osdmap e44: 7 total, 6 up, 7 in 2022-05-13T23:11:19.622 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:11:19 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2022-05-13T23:11:19.623 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:11:19 smithi002 ceph-mon[71816]: from='osd.6 [v2:172.21.15.2:6818/1892198790,v1:172.21.15.2:6819/1892198790]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi002", "root=default"]}]: dispatch 2022-05-13T23:11:20.073 INFO:journalctl@ceph.osd.6.smithi002.stdout:May 13 23:11:19 smithi002 conmon[97720]: 2022-05-13T23:11:19.669+0000 7fb832d28700 -1 osd.6 0 waiting for initial osdmap 2022-05-13T23:11:20.073 INFO:journalctl@ceph.osd.6.smithi002.stdout:May 13 23:11:19 smithi002 conmon[97720]: 2022-05-13T23:11:19.686+0000 7fb82f6c2700 -1 osd.6 45 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2022-05-13T23:11:20.572 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:11:20 smithi002 ceph-mon[71816]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2022-05-13T23:11:20.572 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:11:20 smithi002 ceph-mon[71816]: from='osd.6 [v2:172.21.15.2:6818/1892198790,v1:172.21.15.2:6819/1892198790]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi002", "root=default"]}]': finished 2022-05-13T23:11:20.573 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:11:20 smithi002 ceph-mon[71816]: osdmap e45: 7 total, 6 up, 7 in 2022-05-13T23:11:20.573 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:11:20 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2022-05-13T23:11:20.574 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:11:20 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2022-05-13T23:11:20.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:11:20 smithi115 ceph-mon[78593]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2022-05-13T23:11:20.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:11:20 smithi115 ceph-mon[78593]: from='osd.6 [v2:172.21.15.2:6818/1892198790,v1:172.21.15.2:6819/1892198790]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi002", "root=default"]}]': finished 2022-05-13T23:11:20.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:11:20 smithi115 ceph-mon[78593]: osdmap e45: 7 total, 6 up, 7 in 2022-05-13T23:11:20.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:11:20 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2022-05-13T23:11:20.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:11:20 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2022-05-13T23:11:20.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:11:20 smithi047 ceph-mon[77147]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2022-05-13T23:11:20.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:11:20 smithi047 ceph-mon[77147]: from='osd.6 [v2:172.21.15.2:6818/1892198790,v1:172.21.15.2:6819/1892198790]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi002", "root=default"]}]': finished 2022-05-13T23:11:20.795 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:11:20 smithi047 ceph-mon[77147]: osdmap e45: 7 total, 6 up, 7 in 2022-05-13T23:11:20.796 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:11:20 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2022-05-13T23:11:20.797 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:11:20 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2022-05-13T23:11:21.049 INFO:teuthology.orchestra.run.smithi047.stderr:--> Zapping: /dev/vg_nvme/lv_2 2022-05-13T23:11:21.050 INFO:teuthology.orchestra.run.smithi047.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2022-05-13T23:11:21.050 INFO:teuthology.orchestra.run.smithi047.stderr: stderr: 10+0 records in 2022-05-13T23:11:21.050 INFO:teuthology.orchestra.run.smithi047.stderr:10+0 records out 2022-05-13T23:11:21.050 INFO:teuthology.orchestra.run.smithi047.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.00822187 s, 1.3 GB/s 2022-05-13T23:11:21.051 INFO:teuthology.orchestra.run.smithi047.stderr:--> Zapping successful for: 2022-05-13T23:11:21.449 DEBUG:teuthology.orchestra.run.smithi047:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:23b0658391ce94ca40799469eec00c28b298816c shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 16cf5f88-d311-11ec-be40-001a4aab830c -- ceph orch daemon add osd smithi047:vg_nvme/lv_2 2022-05-13T23:11:21.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:11:21 smithi047 ceph-mon[77147]: purged_snaps scrub starts 2022-05-13T23:11:21.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:11:21 smithi047 ceph-mon[77147]: purged_snaps scrub ok 2022-05-13T23:11:21.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:11:21 smithi047 ceph-mon[77147]: osd.6 [v2:172.21.15.2:6818/1892198790,v1:172.21.15.2:6819/1892198790] boot 2022-05-13T23:11:21.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:11:21 smithi047 ceph-mon[77147]: osdmap e46: 7 total, 7 up, 7 in 2022-05-13T23:11:21.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:11:21 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2022-05-13T23:11:21.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:11:21 smithi047 ceph-mon[77147]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2022-05-13T23:11:21.902 INFO:teuthology.orchestra.run.smithi047.stderr:Inferring config /var/lib/ceph/16cf5f88-d311-11ec-be40-001a4aab830c/mon.b/config 2022-05-13T23:11:22.072 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:11:21 smithi002 ceph-mon[71816]: purged_snaps scrub starts 2022-05-13T23:11:22.073 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:11:21 smithi002 ceph-mon[71816]: purged_snaps scrub ok 2022-05-13T23:11:22.073 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:11:21 smithi002 ceph-mon[71816]: osd.6 [v2:172.21.15.2:6818/1892198790,v1:172.21.15.2:6819/1892198790] boot 2022-05-13T23:11:22.074 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:11:21 smithi002 ceph-mon[71816]: osdmap e46: 7 total, 7 up, 7 in 2022-05-13T23:11:22.074 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:11:21 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2022-05-13T23:11:22.074 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:11:21 smithi002 ceph-mon[71816]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2022-05-13T23:11:22.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:11:21 smithi115 ceph-mon[78593]: purged_snaps scrub starts 2022-05-13T23:11:22.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:11:21 smithi115 ceph-mon[78593]: purged_snaps scrub ok 2022-05-13T23:11:22.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:11:21 smithi115 ceph-mon[78593]: osd.6 [v2:172.21.15.2:6818/1892198790,v1:172.21.15.2:6819/1892198790] boot 2022-05-13T23:11:22.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:11:21 smithi115 ceph-mon[78593]: osdmap e46: 7 total, 7 up, 7 in 2022-05-13T23:11:22.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:11:21 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2022-05-13T23:11:22.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:11:21 smithi115 ceph-mon[78593]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2022-05-13T23:11:22.924 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:11:22 smithi047 ceph-mon[77147]: osdmap e47: 7 total, 7 up, 7 in 2022-05-13T23:11:22.928 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:11:22 smithi002 ceph-mon[71816]: osdmap e47: 7 total, 7 up, 7 in 2022-05-13T23:11:23.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:11:22 smithi115 ceph-mon[78593]: osdmap e47: 7 total, 7 up, 7 in 2022-05-13T23:11:23.525 INFO:teuthology.orchestra.run.smithi047.stderr:2022-05-13T23:11:23.521+0000 7f04f7f6a700 1 -- 172.21.15.47:0/1768149867 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f04f00fb5f0 msgr2=0x7f04f00fba00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:11:23.525 INFO:teuthology.orchestra.run.smithi047.stderr:2022-05-13T23:11:23.521+0000 7f04f7f6a700 1 --2- 172.21.15.47:0/1768149867 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f04f00fb5f0 0x7f04f00fba00 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f04ec004870 tx=0x7f04ec009cb0 comp rx=0 tx=0).stop 2022-05-13T23:11:23.526 INFO:teuthology.orchestra.run.smithi047.stderr:2022-05-13T23:11:23.521+0000 7f04f7f6a700 1 -- 172.21.15.47:0/1768149867 shutdown_connections 2022-05-13T23:11:23.526 INFO:teuthology.orchestra.run.smithi047.stderr:2022-05-13T23:11:23.521+0000 7f04f7f6a700 1 --2- 172.21.15.47:0/1768149867 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f04f00fd220 0x7f04f0103ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:11:23.526 INFO:teuthology.orchestra.run.smithi047.stderr:2022-05-13T23:11:23.521+0000 7f04f7f6a700 1 --2- 172.21.15.47:0/1768149867 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f04f00fb5f0 0x7f04f00fba00 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:11:23.526 INFO:teuthology.orchestra.run.smithi047.stderr:2022-05-13T23:11:23.521+0000 7f04f7f6a700 1 --2- 172.21.15.47:0/1768149867 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f04f00fc890 0x7f04f00fcce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:11:23.527 INFO:teuthology.orchestra.run.smithi047.stderr:2022-05-13T23:11:23.521+0000 7f04f7f6a700 1 -- 172.21.15.47:0/1768149867 >> 172.21.15.47:0/1768149867 conn(0x7f04f00f6de0 msgr2=0x7f04f00f9210 unknown :-1 s=STATE_NONE l=0).mark_down 2022-05-13T23:11:23.527 INFO:teuthology.orchestra.run.smithi047.stderr:2022-05-13T23:11:23.521+0000 7f04f7f6a700 1 -- 172.21.15.47:0/1768149867 shutdown_connections 2022-05-13T23:11:23.527 INFO:teuthology.orchestra.run.smithi047.stderr:2022-05-13T23:11:23.521+0000 7f04f7f6a700 1 -- 172.21.15.47:0/1768149867 wait complete. 2022-05-13T23:11:23.527 INFO:teuthology.orchestra.run.smithi047.stderr:2022-05-13T23:11:23.521+0000 7f04f7f6a700 1 Processor -- start 2022-05-13T23:11:23.528 INFO:teuthology.orchestra.run.smithi047.stderr:2022-05-13T23:11:23.521+0000 7f04f7f6a700 1 -- start start 2022-05-13T23:11:23.528 INFO:teuthology.orchestra.run.smithi047.stderr:2022-05-13T23:11:23.521+0000 7f04f7f6a700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f04f00fb5f0 0x7f04f010c300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:11:23.528 INFO:teuthology.orchestra.run.smithi047.stderr:2022-05-13T23:11:23.522+0000 7f04f7f6a700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f04f00fc890 0x7f04f010c840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:11:23.528 INFO:teuthology.orchestra.run.smithi047.stderr:2022-05-13T23:11:23.522+0000 7f04f7f6a700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f04f00fd220 0x7f04f01138d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:11:23.528 INFO:teuthology.orchestra.run.smithi047.stderr:2022-05-13T23:11:23.522+0000 7f04f7f6a700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04f0113e10 con 0x7f04f00fb5f0 2022-05-13T23:11:23.529 INFO:teuthology.orchestra.run.smithi047.stderr:2022-05-13T23:11:23.522+0000 7f04f7f6a700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04f0113f80 con 0x7f04f00fd220 2022-05-13T23:11:23.529 INFO:teuthology.orchestra.run.smithi047.stderr:2022-05-13T23:11:23.522+0000 7f04f7f6a700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04f0114140 con 0x7f04f00fc890 2022-05-13T23:11:23.529 INFO:teuthology.orchestra.run.smithi047.stderr:2022-05-13T23:11:23.522+0000 7f04f6507700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f04f00fd220 0x7f04f01138d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:11:23.529 INFO:teuthology.orchestra.run.smithi047.stderr:2022-05-13T23:11:23.522+0000 7f04f6507700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f04f00fd220 0x7f04f01138d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:42204/0 (socket says 172.21.15.47:42204) 2022-05-13T23:11:23.530 INFO:teuthology.orchestra.run.smithi047.stderr:2022-05-13T23:11:23.522+0000 7f04f6507700 1 -- 172.21.15.47:0/3531777515 learned_addr learned my addr 172.21.15.47:0/3531777515 (peer_addr_for_me v2:172.21.15.47:0/0) 2022-05-13T23:11:23.530 INFO:teuthology.orchestra.run.smithi047.stderr:2022-05-13T23:11:23.522+0000 7f04f5505700 1 --2- 172.21.15.47:0/3531777515 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f04f00fc890 0x7f04f010c840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:11:23.530 INFO:teuthology.orchestra.run.smithi047.stderr:2022-05-13T23:11:23.522+0000 7f04f5d06700 1 --2- 172.21.15.47:0/3531777515 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f04f00fb5f0 0x7f04f010c300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:11:23.531 INFO:teuthology.orchestra.run.smithi047.stderr:2022-05-13T23:11:23.522+0000 7f04f6507700 1 -- 172.21.15.47:0/3531777515 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f04f00fc890 msgr2=0x7f04f010c840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:11:23.531 INFO:teuthology.orchestra.run.smithi047.stderr:2022-05-13T23:11:23.522+0000 7f04f6507700 1 --2- 172.21.15.47:0/3531777515 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f04f00fc890 0x7f04f010c840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:11:23.531 INFO:teuthology.orchestra.run.smithi047.stderr:2022-05-13T23:11:23.522+0000 7f04f6507700 1 -- 172.21.15.47:0/3531777515 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f04f00fb5f0 msgr2=0x7f04f010c300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:11:23.532 INFO:teuthology.orchestra.run.smithi047.stderr:2022-05-13T23:11:23.522+0000 7f04f6507700 1 --2- 172.21.15.47:0/3531777515 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f04f00fb5f0 0x7f04f010c300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:11:23.532 INFO:teuthology.orchestra.run.smithi047.stderr:2022-05-13T23:11:23.522+0000 7f04f6507700 1 -- 172.21.15.47:0/3531777515 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04ec005040 con 0x7f04f00fd220 2022-05-13T23:11:23.532 INFO:teuthology.orchestra.run.smithi047.stderr:2022-05-13T23:11:23.522+0000 7f04f6507700 1 --2- 172.21.15.47:0/3531777515 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f04f00fd220 0x7f04f01138d0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f04e800c870 tx=0x7f04e8007dd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-05-13T23:11:23.532 INFO:teuthology.orchestra.run.smithi047.stderr:2022-05-13T23:11:23.522+0000 7f04e67fc700 1 -- 172.21.15.47:0/3531777515 <== mon.1 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04e8004020 con 0x7f04f00fd220 2022-05-13T23:11:23.532 INFO:teuthology.orchestra.run.smithi047.stderr:2022-05-13T23:11:23.522+0000 7f04e67fc700 1 -- 172.21.15.47:0/3531777515 <== mon.1 v2:172.21.15.47:3300/0 2 ==== config(26 keys) v1 ==== 1087+0+0 (secure 0 0 0) 0x7f04e8004180 con 0x7f04f00fd220 2022-05-13T23:11:23.533 INFO:teuthology.orchestra.run.smithi047.stderr:2022-05-13T23:11:23.522+0000 7f04e67fc700 1 -- 172.21.15.47:0/3531777515 <== mon.1 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04e8003920 con 0x7f04f00fd220 2022-05-13T23:11:23.533 INFO:teuthology.orchestra.run.smithi047.stderr:2022-05-13T23:11:23.522+0000 7f04f7f6a700 1 -- 172.21.15.47:0/3531777515 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f04f0114420 con 0x7f04f00fd220 2022-05-13T23:11:23.533 INFO:teuthology.orchestra.run.smithi047.stderr:2022-05-13T23:11:23.522+0000 7f04f7f6a700 1 -- 172.21.15.47:0/3531777515 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f04f0114940 con 0x7f04f00fd220 2022-05-13T23:11:23.533 INFO:teuthology.orchestra.run.smithi047.stderr:2022-05-13T23:11:23.523+0000 7f04f7f6a700 1 -- 172.21.15.47:0/3531777515 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f04f0106de0 con 0x7f04f00fd220 2022-05-13T23:11:23.535 INFO:teuthology.orchestra.run.smithi047.stderr:2022-05-13T23:11:23.524+0000 7f04e67fc700 1 -- 172.21.15.47:0/3531777515 <== mon.1 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 16) v1 ==== 125136+0+0 (secure 0 0 0) 0x7f04e8010050 con 0x7f04f00fd220 2022-05-13T23:11:23.535 INFO:teuthology.orchestra.run.smithi047.stderr:2022-05-13T23:11:23.525+0000 7f04e67fc700 1 --2- 172.21.15.47:0/3531777515 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f04dc096ce0 0x7f04dc099190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:11:23.536 INFO:teuthology.orchestra.run.smithi047.stderr:2022-05-13T23:11:23.525+0000 7f04e67fc700 1 -- 172.21.15.47:0/3531777515 <== mon.1 v2:172.21.15.47:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 4961+0+0 (secure 0 0 0) 0x7f04e80be7a0 con 0x7f04f00fd220 2022-05-13T23:11:23.536 INFO:teuthology.orchestra.run.smithi047.stderr:2022-05-13T23:11:23.525+0000 7f04f5d06700 1 --2- 172.21.15.47:0/3531777515 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f04dc096ce0 0x7f04dc099190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:11:23.536 INFO:teuthology.orchestra.run.smithi047.stderr:2022-05-13T23:11:23.526+0000 7f04f5d06700 1 --2- 172.21.15.47:0/3531777515 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f04dc096ce0 0x7f04dc099190 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f04ec004840 tx=0x7f04ec009cb0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-05-13T23:11:23.536 INFO:teuthology.orchestra.run.smithi047.stderr:2022-05-13T23:11:23.529+0000 7f04e67fc700 1 -- 172.21.15.47:0/3531777515 <== mon.1 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+165215 (secure 0 0 0) 0x7f04e808c790 con 0x7f04f00fd220 2022-05-13T23:11:23.719 INFO:teuthology.orchestra.run.smithi047.stderr:2022-05-13T23:11:23.717+0000 7f04f7f6a700 1 -- 172.21.15.47:0/3531777515 --> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi047:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) v1 -- 0x7f04f002a560 con 0x7f04dc096ce0 2022-05-13T23:11:24.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:11:23 smithi047 ceph-mon[77147]: Detected new or changed devices on smithi002 2022-05-13T23:11:24.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:11:23 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:11:24.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:11:23 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2022-05-13T23:11:24.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:11:23 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2022-05-13T23:11:24.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:11:23 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2022-05-13T23:11:24.045 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:11:23 smithi047 ceph-mon[77147]: Adjusting osd_memory_target on smithi002 to 5759M 2022-05-13T23:11:24.045 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:11:23 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:11:24.045 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:11:23 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:11:24.046 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:11:23 smithi047 ceph-mon[77147]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2022-05-13T23:11:24.046 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:11:23 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2022-05-13T23:11:24.047 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:11:23 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2022-05-13T23:11:24.047 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:11:23 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-13T23:11:24.330 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:11:23 smithi002 ceph-mon[71816]: Detected new or changed devices on smithi002 2022-05-13T23:11:24.330 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:11:23 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:11:24.330 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:11:23 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2022-05-13T23:11:24.331 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:11:23 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2022-05-13T23:11:24.331 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:11:23 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2022-05-13T23:11:24.331 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:11:23 smithi002 ceph-mon[71816]: Adjusting osd_memory_target on smithi002 to 5759M 2022-05-13T23:11:24.331 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:11:23 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:11:24.331 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:11:23 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:11:24.332 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:11:23 smithi002 ceph-mon[71816]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2022-05-13T23:11:24.332 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:11:23 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2022-05-13T23:11:24.332 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:11:23 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2022-05-13T23:11:24.332 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:11:23 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-13T23:11:24.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:11:23 smithi115 ceph-mon[78593]: Detected new or changed devices on smithi002 2022-05-13T23:11:24.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:11:23 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:11:24.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:11:23 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2022-05-13T23:11:24.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:11:23 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2022-05-13T23:11:24.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:11:23 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2022-05-13T23:11:24.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:11:23 smithi115 ceph-mon[78593]: Adjusting osd_memory_target on smithi002 to 5759M 2022-05-13T23:11:24.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:11:23 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:11:24.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:11:23 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:11:24.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:11:23 smithi115 ceph-mon[78593]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2022-05-13T23:11:24.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:11:23 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2022-05-13T23:11:24.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:11:23 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2022-05-13T23:11:24.362 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:11:23 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-13T23:11:25.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:11:25 smithi047 ceph-mon[77147]: from='client.24301 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi047:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2022-05-13T23:11:25.572 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:11:25 smithi002 ceph-mon[71816]: from='client.24301 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi047:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2022-05-13T23:11:25.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:11:25 smithi115 ceph-mon[78593]: from='client.24301 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi047:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2022-05-13T23:11:26.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:11:26 smithi047 ceph-mon[77147]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2022-05-13T23:11:26.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:11:26 smithi047 ceph-mon[77147]: from='client.? 172.21.15.47:0/1556551800' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e799b560-c61e-4369-b08b-45e5a6af37c0"}]: dispatch 2022-05-13T23:11:26.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:11:26 smithi047 ceph-mon[77147]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e799b560-c61e-4369-b08b-45e5a6af37c0"}]: dispatch 2022-05-13T23:11:26.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:11:26 smithi047 ceph-mon[77147]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e799b560-c61e-4369-b08b-45e5a6af37c0"}]': finished 2022-05-13T23:11:26.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:11:26 smithi047 ceph-mon[77147]: osdmap e48: 8 total, 7 up, 8 in 2022-05-13T23:11:26.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:11:26 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2022-05-13T23:11:26.572 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:11:26 smithi002 ceph-mon[71816]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2022-05-13T23:11:26.573 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:11:26 smithi002 ceph-mon[71816]: from='client.? 172.21.15.47:0/1556551800' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e799b560-c61e-4369-b08b-45e5a6af37c0"}]: dispatch 2022-05-13T23:11:26.573 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:11:26 smithi002 ceph-mon[71816]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e799b560-c61e-4369-b08b-45e5a6af37c0"}]: dispatch 2022-05-13T23:11:26.573 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:11:26 smithi002 ceph-mon[71816]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e799b560-c61e-4369-b08b-45e5a6af37c0"}]': finished 2022-05-13T23:11:26.574 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:11:26 smithi002 ceph-mon[71816]: osdmap e48: 8 total, 7 up, 8 in 2022-05-13T23:11:26.574 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:11:26 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2022-05-13T23:11:26.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:11:26 smithi115 ceph-mon[78593]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2022-05-13T23:11:26.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:11:26 smithi115 ceph-mon[78593]: from='client.? 172.21.15.47:0/1556551800' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e799b560-c61e-4369-b08b-45e5a6af37c0"}]: dispatch 2022-05-13T23:11:26.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:11:26 smithi115 ceph-mon[78593]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e799b560-c61e-4369-b08b-45e5a6af37c0"}]: dispatch 2022-05-13T23:11:26.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:11:26 smithi115 ceph-mon[78593]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e799b560-c61e-4369-b08b-45e5a6af37c0"}]': finished 2022-05-13T23:11:26.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:11:26 smithi115 ceph-mon[78593]: osdmap e48: 8 total, 7 up, 8 in 2022-05-13T23:11:26.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:11:26 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2022-05-13T23:11:27.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:11:27 smithi047 ceph-mon[77147]: from='client.? 172.21.15.47:0/2098609712' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2022-05-13T23:11:27.572 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:11:27 smithi002 ceph-mon[71816]: from='client.? 172.21.15.47:0/2098609712' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2022-05-13T23:11:27.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:11:27 smithi115 ceph-mon[78593]: from='client.? 172.21.15.47:0/2098609712' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2022-05-13T23:11:28.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:11:28 smithi047 ceph-mon[77147]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2022-05-13T23:11:28.572 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:11:28 smithi002 ceph-mon[71816]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2022-05-13T23:11:28.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:11:28 smithi115 ceph-mon[78593]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2022-05-13T23:11:30.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:11:30 smithi047 ceph-mon[77147]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2022-05-13T23:11:30.572 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:11:30 smithi002 ceph-mon[71816]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2022-05-13T23:11:30.613 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:11:30 smithi115 ceph-mon[78593]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2022-05-13T23:11:32.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:11:32 smithi047 ceph-mon[77147]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2022-05-13T23:11:32.572 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:11:32 smithi002 ceph-mon[71816]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2022-05-13T23:11:32.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:11:32 smithi115 ceph-mon[78593]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2022-05-13T23:11:33.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:11:33 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2022-05-13T23:11:33.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:11:33 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-13T23:11:33.572 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:11:33 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2022-05-13T23:11:33.573 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:11:33 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-13T23:11:33.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:11:33 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2022-05-13T23:11:33.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:11:33 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-13T23:11:34.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:11:34 smithi047 ceph-mon[77147]: Deploying daemon osd.7 on smithi047 2022-05-13T23:11:34.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:11:34 smithi047 ceph-mon[77147]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2022-05-13T23:11:34.572 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:11:34 smithi002 ceph-mon[71816]: Deploying daemon osd.7 on smithi047 2022-05-13T23:11:34.573 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:11:34 smithi002 ceph-mon[71816]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2022-05-13T23:11:34.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:11:34 smithi115 ceph-mon[78593]: Deploying daemon osd.7 on smithi047 2022-05-13T23:11:34.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:11:34 smithi115 ceph-mon[78593]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2022-05-13T23:11:36.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:11:36 smithi047 ceph-mon[77147]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2022-05-13T23:11:36.572 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:11:36 smithi002 ceph-mon[71816]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2022-05-13T23:11:36.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:11:36 smithi115 ceph-mon[78593]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2022-05-13T23:11:38.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:11:38 smithi047 ceph-mon[77147]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2022-05-13T23:11:38.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:11:38 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:11:38.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:11:38 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-13T23:11:38.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:11:38 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-13T23:11:38.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:11:38 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-13T23:11:38.573 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:11:38 smithi002 ceph-mon[71816]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2022-05-13T23:11:38.573 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:11:38 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:11:38.574 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:11:38 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-13T23:11:38.574 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:11:38 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-13T23:11:38.575 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:11:38 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-13T23:11:38.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:11:38 smithi115 ceph-mon[78593]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2022-05-13T23:11:38.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:11:38 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:11:38.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:11:38 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-13T23:11:38.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:11:38 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-13T23:11:38.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:11:38 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-13T23:11:40.394 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:11:40 smithi047 ceph-mon[77147]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2022-05-13T23:11:40.394 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:11:40 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:11:40.395 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:11:40 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:11:40.404 INFO:teuthology.orchestra.run.smithi047.stderr:2022-05-13T23:11:40.403+0000 7f04e67fc700 1 -- 172.21.15.47:0/3531777515 <== mgr.14148 v2:172.21.15.2:6800/2788339627 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f04f002a560 con 0x7f04dc096ce0 2022-05-13T23:11:40.405 INFO:teuthology.orchestra.run.smithi047.stdout:Created osd(s) 7 on host 'smithi047' 2022-05-13T23:11:40.407 INFO:teuthology.orchestra.run.smithi047.stderr:2022-05-13T23:11:40.405+0000 7f04f7f6a700 1 -- 172.21.15.47:0/3531777515 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f04dc096ce0 msgr2=0x7f04dc099190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:11:40.407 INFO:teuthology.orchestra.run.smithi047.stderr:2022-05-13T23:11:40.405+0000 7f04f7f6a700 1 --2- 172.21.15.47:0/3531777515 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f04dc096ce0 0x7f04dc099190 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f04ec004840 tx=0x7f04ec009cb0 comp rx=0 tx=0).stop 2022-05-13T23:11:40.407 INFO:teuthology.orchestra.run.smithi047.stderr:2022-05-13T23:11:40.405+0000 7f04f7f6a700 1 -- 172.21.15.47:0/3531777515 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f04f00fd220 msgr2=0x7f04f01138d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:11:40.407 INFO:teuthology.orchestra.run.smithi047.stderr:2022-05-13T23:11:40.405+0000 7f04f7f6a700 1 --2- 172.21.15.47:0/3531777515 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f04f00fd220 0x7f04f01138d0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f04e800c870 tx=0x7f04e8007dd0 comp rx=0 tx=0).stop 2022-05-13T23:11:40.408 INFO:teuthology.orchestra.run.smithi047.stderr:2022-05-13T23:11:40.405+0000 7f04f7f6a700 1 -- 172.21.15.47:0/3531777515 shutdown_connections 2022-05-13T23:11:40.408 INFO:teuthology.orchestra.run.smithi047.stderr:2022-05-13T23:11:40.405+0000 7f04f7f6a700 1 --2- 172.21.15.47:0/3531777515 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f04f00fd220 0x7f04f01138d0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:11:40.408 INFO:teuthology.orchestra.run.smithi047.stderr:2022-05-13T23:11:40.405+0000 7f04f7f6a700 1 --2- 172.21.15.47:0/3531777515 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f04dc096ce0 0x7f04dc099190 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:11:40.408 INFO:teuthology.orchestra.run.smithi047.stderr:2022-05-13T23:11:40.405+0000 7f04f7f6a700 1 --2- 172.21.15.47:0/3531777515 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f04f00fb5f0 0x7f04f010c300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:11:40.408 INFO:teuthology.orchestra.run.smithi047.stderr:2022-05-13T23:11:40.405+0000 7f04f7f6a700 1 --2- 172.21.15.47:0/3531777515 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f04f00fc890 0x7f04f010c840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:11:40.409 INFO:teuthology.orchestra.run.smithi047.stderr:2022-05-13T23:11:40.406+0000 7f04f7f6a700 1 -- 172.21.15.47:0/3531777515 >> 172.21.15.47:0/3531777515 conn(0x7f04f00f6de0 msgr2=0x7f04f00f9150 unknown :-1 s=STATE_NONE l=0).mark_down 2022-05-13T23:11:40.409 INFO:teuthology.orchestra.run.smithi047.stderr:2022-05-13T23:11:40.406+0000 7f04f7f6a700 1 -- 172.21.15.47:0/3531777515 shutdown_connections 2022-05-13T23:11:40.409 INFO:teuthology.orchestra.run.smithi047.stderr:2022-05-13T23:11:40.406+0000 7f04f7f6a700 1 -- 172.21.15.47:0/3531777515 wait complete. 2022-05-13T23:11:40.572 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:11:40 smithi002 ceph-mon[71816]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2022-05-13T23:11:40.573 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:11:40 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:11:40.573 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:11:40 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:11:40.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:11:40 smithi115 ceph-mon[78593]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2022-05-13T23:11:40.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:11:40 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:11:40.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:11:40 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:11:40.929 DEBUG:teuthology.orchestra.run.smithi047:osd.7> sudo journalctl -f -n 0 -u ceph-16cf5f88-d311-11ec-be40-001a4aab830c@osd.7.service 2022-05-13T23:11:40.934 INFO:tasks.cephadm:Deploying osd.8 on smithi115 with /dev/vg_nvme/lv_2... 2022-05-13T23:11:40.935 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:23b0658391ce94ca40799469eec00c28b298816c shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 16cf5f88-d311-11ec-be40-001a4aab830c -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2022-05-13T23:11:40.959 INFO:journalctl@ceph.osd.7.smithi047.stdout:-- Logs begin at Fri 2022-05-13 22:44:47 UTC. -- 2022-05-13T23:11:41.387 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/16cf5f88-d311-11ec-be40-001a4aab830c/mon.c/config 2022-05-13T23:11:41.543 INFO:journalctl@ceph.osd.7.smithi047.stdout:May 13 23:11:41 smithi047 conmon[94366]: 2022-05-13T23:11:41.416+0000 7fb543be23c0 -1 osd.7 0 log_to_monitors true 2022-05-13T23:11:41.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:11:41 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:11:41.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:11:41 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-13T23:11:41.545 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:11:41 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-13T23:11:41.545 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:11:41 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-13T23:11:41.801 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:11:41 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:11:41.801 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:11:41 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-13T23:11:41.801 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:11:41 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-13T23:11:41.801 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:11:41 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-13T23:11:41.822 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:11:41 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:11:41.823 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:11:41 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-13T23:11:41.823 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:11:41 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-13T23:11:41.823 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:11:41 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-13T23:11:42.490 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:11:42 smithi115 ceph-mon[78593]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2022-05-13T23:11:42.491 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:11:42 smithi115 ceph-mon[78593]: from='osd.7 [v2:172.21.15.47:6816/2812542825,v1:172.21.15.47:6817/2812542825]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2022-05-13T23:11:42.491 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:11:42 smithi115 ceph-mon[78593]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2022-05-13T23:11:42.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:11:42 smithi047 ceph-mon[77147]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2022-05-13T23:11:42.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:11:42 smithi047 ceph-mon[77147]: from='osd.7 [v2:172.21.15.47:6816/2812542825,v1:172.21.15.47:6817/2812542825]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2022-05-13T23:11:42.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:11:42 smithi047 ceph-mon[77147]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2022-05-13T23:11:42.822 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:11:42 smithi002 ceph-mon[71816]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2022-05-13T23:11:42.823 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:11:42 smithi002 ceph-mon[71816]: from='osd.7 [v2:172.21.15.47:6816/2812542825,v1:172.21.15.47:6817/2812542825]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2022-05-13T23:11:42.823 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:11:42 smithi002 ceph-mon[71816]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2022-05-13T23:11:42.987 INFO:teuthology.orchestra.run.smithi115.stderr:--> Zapping: /dev/vg_nvme/lv_2 2022-05-13T23:11:42.987 INFO:teuthology.orchestra.run.smithi115.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2022-05-13T23:11:42.988 INFO:teuthology.orchestra.run.smithi115.stderr: stderr: 10+0 records in 2022-05-13T23:11:42.988 INFO:teuthology.orchestra.run.smithi115.stderr:10+0 records out 2022-05-13T23:11:42.989 INFO:teuthology.orchestra.run.smithi115.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0110604 s, 948 MB/s 2022-05-13T23:11:42.989 INFO:teuthology.orchestra.run.smithi115.stderr:--> Zapping successful for: 2022-05-13T23:11:43.492 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:23b0658391ce94ca40799469eec00c28b298816c shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 16cf5f88-d311-11ec-be40-001a4aab830c -- ceph orch daemon add osd smithi115:vg_nvme/lv_2 2022-05-13T23:11:43.791 INFO:journalctl@ceph.osd.7.smithi047.stdout:May 13 23:11:43 smithi047 conmon[94366]: 2022-05-13T23:11:43.427+0000 7fb53a5e6700 -1 osd.7 0 waiting for initial osdmap 2022-05-13T23:11:43.792 INFO:journalctl@ceph.osd.7.smithi047.stdout:May 13 23:11:43 smithi047 conmon[94366]: 2022-05-13T23:11:43.443+0000 7fb53577d700 -1 osd.7 50 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2022-05-13T23:11:43.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:11:43 smithi047 ceph-mon[77147]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2022-05-13T23:11:43.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:11:43 smithi047 ceph-mon[77147]: osdmap e49: 8 total, 7 up, 8 in 2022-05-13T23:11:43.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:11:43 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2022-05-13T23:11:43.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:11:43 smithi047 ceph-mon[77147]: from='osd.7 [v2:172.21.15.47:6816/2812542825,v1:172.21.15.47:6817/2812542825]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi047", "root=default"]}]: dispatch 2022-05-13T23:11:43.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:11:43 smithi047 ceph-mon[77147]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi047", "root=default"]}]: dispatch 2022-05-13T23:11:43.823 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:11:43 smithi002 ceph-mon[71816]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2022-05-13T23:11:43.823 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:11:43 smithi002 ceph-mon[71816]: osdmap e49: 8 total, 7 up, 8 in 2022-05-13T23:11:43.824 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:11:43 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2022-05-13T23:11:43.824 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:11:43 smithi002 ceph-mon[71816]: from='osd.7 [v2:172.21.15.47:6816/2812542825,v1:172.21.15.47:6817/2812542825]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi047", "root=default"]}]: dispatch 2022-05-13T23:11:43.824 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:11:43 smithi002 ceph-mon[71816]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi047", "root=default"]}]: dispatch 2022-05-13T23:11:43.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:11:43 smithi115 ceph-mon[78593]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2022-05-13T23:11:43.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:11:43 smithi115 ceph-mon[78593]: osdmap e49: 8 total, 7 up, 8 in 2022-05-13T23:11:43.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:11:43 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2022-05-13T23:11:43.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:11:43 smithi115 ceph-mon[78593]: from='osd.7 [v2:172.21.15.47:6816/2812542825,v1:172.21.15.47:6817/2812542825]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi047", "root=default"]}]: dispatch 2022-05-13T23:11:43.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:11:43 smithi115 ceph-mon[78593]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi047", "root=default"]}]: dispatch 2022-05-13T23:11:43.935 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/16cf5f88-d311-11ec-be40-001a4aab830c/mon.c/config 2022-05-13T23:11:44.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:11:44 smithi115 ceph-mon[78593]: purged_snaps scrub starts 2022-05-13T23:11:44.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:11:44 smithi115 ceph-mon[78593]: purged_snaps scrub ok 2022-05-13T23:11:44.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:11:44 smithi115 ceph-mon[78593]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2022-05-13T23:11:44.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:11:44 smithi115 ceph-mon[78593]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi047", "root=default"]}]': finished 2022-05-13T23:11:44.612 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:11:44 smithi115 ceph-mon[78593]: osdmap e50: 8 total, 7 up, 8 in 2022-05-13T23:11:44.612 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:11:44 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2022-05-13T23:11:44.612 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:11:44 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2022-05-13T23:11:44.612 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:11:44 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:11:44.613 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:11:44 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-05-13T23:11:44.613 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:11:44 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-05-13T23:11:44.613 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:11:44 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-05-13T23:11:44.613 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:11:44 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:11:44.614 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:11:44 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:11:44.614 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:11:44 smithi115 ceph-mon[78593]: osd.7 [v2:172.21.15.47:6816/2812542825,v1:172.21.15.47:6817/2812542825] boot 2022-05-13T23:11:44.614 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:11:44 smithi115 ceph-mon[78593]: osdmap e51: 8 total, 8 up, 8 in 2022-05-13T23:11:44.614 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:11:44 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2022-05-13T23:11:44.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:11:44 smithi047 ceph-mon[77147]: purged_snaps scrub starts 2022-05-13T23:11:44.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:11:44 smithi047 ceph-mon[77147]: purged_snaps scrub ok 2022-05-13T23:11:44.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:11:44 smithi047 ceph-mon[77147]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2022-05-13T23:11:44.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:11:44 smithi047 ceph-mon[77147]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi047", "root=default"]}]': finished 2022-05-13T23:11:44.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:11:44 smithi047 ceph-mon[77147]: osdmap e50: 8 total, 7 up, 8 in 2022-05-13T23:11:44.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:11:44 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2022-05-13T23:11:44.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:11:44 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2022-05-13T23:11:44.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:11:44 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:11:44.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:11:44 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-05-13T23:11:44.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:11:44 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-05-13T23:11:44.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:11:44 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-05-13T23:11:44.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:11:44 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:11:44.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:11:44 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:11:44.795 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:11:44 smithi047 ceph-mon[77147]: osd.7 [v2:172.21.15.47:6816/2812542825,v1:172.21.15.47:6817/2812542825] boot 2022-05-13T23:11:44.795 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:11:44 smithi047 ceph-mon[77147]: osdmap e51: 8 total, 8 up, 8 in 2022-05-13T23:11:44.795 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:11:44 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2022-05-13T23:11:44.822 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:11:44 smithi002 ceph-mon[71816]: purged_snaps scrub starts 2022-05-13T23:11:44.823 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:11:44 smithi002 ceph-mon[71816]: purged_snaps scrub ok 2022-05-13T23:11:44.823 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:11:44 smithi002 ceph-mon[71816]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2022-05-13T23:11:44.823 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:11:44 smithi002 ceph-mon[71816]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi047", "root=default"]}]': finished 2022-05-13T23:11:44.824 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:11:44 smithi002 ceph-mon[71816]: osdmap e50: 8 total, 7 up, 8 in 2022-05-13T23:11:44.824 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:11:44 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2022-05-13T23:11:44.824 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:11:44 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2022-05-13T23:11:44.824 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:11:44 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:11:44.824 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:11:44 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-05-13T23:11:44.825 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:11:44 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-05-13T23:11:44.825 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:11:44 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-05-13T23:11:44.825 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:11:44 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:11:44.825 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:11:44 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:11:44.826 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:11:44 smithi002 ceph-mon[71816]: osd.7 [v2:172.21.15.47:6816/2812542825,v1:172.21.15.47:6817/2812542825] boot 2022-05-13T23:11:44.826 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:11:44 smithi002 ceph-mon[71816]: osdmap e51: 8 total, 8 up, 8 in 2022-05-13T23:11:44.826 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:11:44 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2022-05-13T23:11:45.407 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:11:45.404+0000 7efe0c884700 1 -- 172.21.15.115:0/2100294771 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7efe080fa500 msgr2=0x7efe080fa910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:11:45.408 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:11:45.404+0000 7efe0c884700 1 --2- 172.21.15.115:0/2100294771 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7efe080fa500 0x7efe080fa910 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7efdf8004870 tx=0x7efdf8009cb0 comp rx=0 tx=0).stop 2022-05-13T23:11:45.408 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:11:45.404+0000 7efe0c884700 1 -- 172.21.15.115:0/2100294771 shutdown_connections 2022-05-13T23:11:45.408 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:11:45.404+0000 7efe0c884700 1 --2- 172.21.15.115:0/2100294771 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efe080fc130 0x7efe081029f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:11:45.408 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:11:45.404+0000 7efe0c884700 1 --2- 172.21.15.115:0/2100294771 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7efe080fa500 0x7efe080fa910 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:11:45.409 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:11:45.404+0000 7efe0c884700 1 --2- 172.21.15.115:0/2100294771 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7efe080fb7a0 0x7efe080fbbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:11:45.409 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:11:45.404+0000 7efe0c884700 1 -- 172.21.15.115:0/2100294771 >> 172.21.15.115:0/2100294771 conn(0x7efe080f5cf0 msgr2=0x7efe080f8120 unknown :-1 s=STATE_NONE l=0).mark_down 2022-05-13T23:11:45.409 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:11:45.404+0000 7efe0c884700 1 -- 172.21.15.115:0/2100294771 shutdown_connections 2022-05-13T23:11:45.409 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:11:45.404+0000 7efe0c884700 1 -- 172.21.15.115:0/2100294771 wait complete. 2022-05-13T23:11:45.410 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:11:45.405+0000 7efe0c884700 1 Processor -- start 2022-05-13T23:11:45.410 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:11:45.405+0000 7efe0c884700 1 -- start start 2022-05-13T23:11:45.410 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:11:45.405+0000 7efe0c884700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7efe080fa500 0x7efe08064960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:11:45.410 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:11:45.405+0000 7efe0c884700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7efe080fb7a0 0x7efe081111f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:11:45.411 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:11:45.405+0000 7efe0c884700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efe080fc130 0x7efe08113870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:11:45.411 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:11:45.405+0000 7efe0c884700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe08114060 con 0x7efe080fa500 2022-05-13T23:11:45.411 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:11:45.405+0000 7efe0c884700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe081141d0 con 0x7efe080fc130 2022-05-13T23:11:45.411 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:11:45.405+0000 7efe0c884700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe08114390 con 0x7efe080fb7a0 2022-05-13T23:11:45.412 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:11:45.405+0000 7efe05d9b700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7efe080fb7a0 0x7efe081111f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:11:45.412 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:11:45.405+0000 7efe05d9b700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7efe080fb7a0 0x7efe081111f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:49102/0 (socket says 172.21.15.115:49102) 2022-05-13T23:11:45.413 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:11:45.405+0000 7efe05d9b700 1 -- 172.21.15.115:0/1503863831 learned_addr learned my addr 172.21.15.115:0/1503863831 (peer_addr_for_me v2:172.21.15.115:0/0) 2022-05-13T23:11:45.413 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:11:45.405+0000 7efe06d9d700 1 --2- 172.21.15.115:0/1503863831 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efe080fc130 0x7efe08113870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:11:45.414 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:11:45.405+0000 7efe05d9b700 1 -- 172.21.15.115:0/1503863831 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efe080fc130 msgr2=0x7efe08113870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:11:45.414 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:11:45.405+0000 7efe05d9b700 1 --2- 172.21.15.115:0/1503863831 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efe080fc130 0x7efe08113870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:11:45.414 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:11:45.405+0000 7efe05d9b700 1 -- 172.21.15.115:0/1503863831 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7efe080fa500 msgr2=0x7efe08064960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:11:45.414 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:11:45.405+0000 7efe05d9b700 1 --2- 172.21.15.115:0/1503863831 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7efe080fa500 0x7efe08064960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:11:45.415 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:11:45.405+0000 7efe05d9b700 1 -- 172.21.15.115:0/1503863831 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efdf8005040 con 0x7efe080fb7a0 2022-05-13T23:11:45.415 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:11:45.405+0000 7efe05d9b700 1 --2- 172.21.15.115:0/1503863831 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7efe080fb7a0 0x7efe081111f0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7efdfc00b530 tx=0x7efdfc0090f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-05-13T23:11:45.415 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:11:45.405+0000 7efdf6ffd700 1 -- 172.21.15.115:0/1503863831 <== mon.2 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdfc00f510 con 0x7efe080fb7a0 2022-05-13T23:11:45.415 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:11:45.405+0000 7efdf6ffd700 1 -- 172.21.15.115:0/1503863831 <== mon.2 v2:172.21.15.115:3300/0 2 ==== config(26 keys) v1 ==== 1087+0+0 (secure 0 0 0) 0x7efdfc00fb20 con 0x7efe080fb7a0 2022-05-13T23:11:45.415 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:11:45.405+0000 7efe0c884700 1 -- 172.21.15.115:0/1503863831 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efe08114610 con 0x7efe080fb7a0 2022-05-13T23:11:45.416 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:11:45.405+0000 7efdf6ffd700 1 -- 172.21.15.115:0/1503863831 <== mon.2 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdfc011be0 con 0x7efe080fb7a0 2022-05-13T23:11:45.416 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:11:45.406+0000 7efe0c884700 1 -- 172.21.15.115:0/1503863831 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efe08114b50 con 0x7efe080fb7a0 2022-05-13T23:11:45.417 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:11:45.406+0000 7efe0c884700 1 -- 172.21.15.115:0/1503863831 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efe08105de0 con 0x7efe080fb7a0 2022-05-13T23:11:45.417 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:11:45.407+0000 7efdf6ffd700 1 -- 172.21.15.115:0/1503863831 <== mon.2 v2:172.21.15.115:3300/0 4 ==== mgrmap(e 16) v1 ==== 125136+0+0 (secure 0 0 0) 0x7efdfc00f070 con 0x7efe080fb7a0 2022-05-13T23:11:45.418 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:11:45.408+0000 7efdf6ffd700 1 --2- 172.21.15.115:0/1503863831 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7efdf0096d60 0x7efdf0099210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:11:45.418 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:11:45.408+0000 7efdf6ffd700 1 -- 172.21.15.115:0/1503863831 <== mon.2 v2:172.21.15.115:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 5422+0+0 (secure 0 0 0) 0x7efdfc0be130 con 0x7efe080fb7a0 2022-05-13T23:11:45.418 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:11:45.408+0000 7efe0659c700 1 --2- 172.21.15.115:0/1503863831 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7efdf0096d60 0x7efdf0099210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:11:45.418 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:11:45.409+0000 7efe0659c700 1 --2- 172.21.15.115:0/1503863831 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7efdf0096d60 0x7efdf0099210 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7efdf8009d60 tx=0x7efdf8009cb0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-05-13T23:11:45.419 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:11:45.412+0000 7efdf6ffd700 1 -- 172.21.15.115:0/1503863831 <== mon.2 v2:172.21.15.115:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+165215 (secure 0 0 0) 0x7efdfc08bf60 con 0x7efe080fb7a0 2022-05-13T23:11:45.595 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:11:45.591+0000 7efe0c884700 1 -- 172.21.15.115:0/1503863831 --> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi115:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) v1 -- 0x7efe0802a500 con 0x7efdf0096d60 2022-05-13T23:11:45.692 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:11:45 smithi115 ceph-mon[78593]: Detected new or changed devices on smithi047 2022-05-13T23:11:45.692 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:11:45 smithi115 ceph-mon[78593]: Adjusting osd_memory_target on smithi047 to 5725M 2022-05-13T23:11:45.693 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:11:45 smithi115 ceph-mon[78593]: osdmap e52: 8 total, 8 up, 8 in 2022-05-13T23:11:45.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:11:45 smithi047 ceph-mon[77147]: Detected new or changed devices on smithi047 2022-05-13T23:11:45.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:11:45 smithi047 ceph-mon[77147]: Adjusting osd_memory_target on smithi047 to 5725M 2022-05-13T23:11:45.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:11:45 smithi047 ceph-mon[77147]: osdmap e52: 8 total, 8 up, 8 in 2022-05-13T23:11:45.822 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:11:45 smithi002 ceph-mon[71816]: Detected new or changed devices on smithi047 2022-05-13T23:11:45.823 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:11:45 smithi002 ceph-mon[71816]: Adjusting osd_memory_target on smithi047 to 5725M 2022-05-13T23:11:45.823 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:11:45 smithi002 ceph-mon[71816]: osdmap e52: 8 total, 8 up, 8 in 2022-05-13T23:11:46.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:11:46 smithi115 ceph-mon[78593]: pgmap v181: 1 pgs: 1 remapped+peering; 577 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2022-05-13T23:11:46.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:11:46 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2022-05-13T23:11:46.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:11:46 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2022-05-13T23:11:46.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:11:46 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-13T23:11:46.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:11:46 smithi115 ceph-mon[78593]: osdmap e53: 8 total, 8 up, 8 in 2022-05-13T23:11:46.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:11:46 smithi047 ceph-mon[77147]: pgmap v181: 1 pgs: 1 remapped+peering; 577 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2022-05-13T23:11:46.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:11:46 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2022-05-13T23:11:46.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:11:46 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2022-05-13T23:11:46.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:11:46 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-13T23:11:46.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:11:46 smithi047 ceph-mon[77147]: osdmap e53: 8 total, 8 up, 8 in 2022-05-13T23:11:46.822 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:11:46 smithi002 ceph-mon[71816]: pgmap v181: 1 pgs: 1 remapped+peering; 577 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2022-05-13T23:11:46.823 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:11:46 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2022-05-13T23:11:46.823 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:11:46 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2022-05-13T23:11:46.823 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:11:46 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-13T23:11:46.823 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:11:46 smithi002 ceph-mon[71816]: osdmap e53: 8 total, 8 up, 8 in 2022-05-13T23:11:47.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:11:47 smithi047 ceph-mon[77147]: from='client.24326 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi115:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2022-05-13T23:11:47.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:11:47 smithi047 ceph-mon[77147]: osdmap e54: 8 total, 8 up, 8 in 2022-05-13T23:11:47.822 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:11:47 smithi002 ceph-mon[71816]: from='client.24326 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi115:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2022-05-13T23:11:47.824 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:11:47 smithi002 ceph-mon[71816]: osdmap e54: 8 total, 8 up, 8 in 2022-05-13T23:11:47.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:11:47 smithi115 ceph-mon[78593]: from='client.24326 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi115:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2022-05-13T23:11:47.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:11:47 smithi115 ceph-mon[78593]: osdmap e54: 8 total, 8 up, 8 in 2022-05-13T23:11:48.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:11:48 smithi047 ceph-mon[77147]: pgmap v184: 1 pgs: 1 remapped+peering; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2022-05-13T23:11:48.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:11:48 smithi047 ceph-mon[77147]: from='client.? 172.21.15.115:0/3046374397' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a58f2c0e-e487-41cf-8d2e-8c937cac9ebc"}]: dispatch 2022-05-13T23:11:48.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:11:48 smithi047 ceph-mon[77147]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a58f2c0e-e487-41cf-8d2e-8c937cac9ebc"}]: dispatch 2022-05-13T23:11:48.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:11:48 smithi047 ceph-mon[77147]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a58f2c0e-e487-41cf-8d2e-8c937cac9ebc"}]': finished 2022-05-13T23:11:48.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:11:48 smithi047 ceph-mon[77147]: osdmap e55: 9 total, 8 up, 9 in 2022-05-13T23:11:48.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:11:48 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2022-05-13T23:11:48.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:11:48 smithi047 ceph-mon[77147]: from='client.? 172.21.15.115:0/1903740850' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2022-05-13T23:11:48.822 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:11:48 smithi002 ceph-mon[71816]: pgmap v184: 1 pgs: 1 remapped+peering; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2022-05-13T23:11:48.823 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:11:48 smithi002 ceph-mon[71816]: from='client.? 172.21.15.115:0/3046374397' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a58f2c0e-e487-41cf-8d2e-8c937cac9ebc"}]: dispatch 2022-05-13T23:11:48.823 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:11:48 smithi002 ceph-mon[71816]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a58f2c0e-e487-41cf-8d2e-8c937cac9ebc"}]: dispatch 2022-05-13T23:11:48.824 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:11:48 smithi002 ceph-mon[71816]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a58f2c0e-e487-41cf-8d2e-8c937cac9ebc"}]': finished 2022-05-13T23:11:48.824 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:11:48 smithi002 ceph-mon[71816]: osdmap e55: 9 total, 8 up, 9 in 2022-05-13T23:11:48.824 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:11:48 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2022-05-13T23:11:48.824 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:11:48 smithi002 ceph-mon[71816]: from='client.? 172.21.15.115:0/1903740850' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2022-05-13T23:11:48.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:11:48 smithi115 ceph-mon[78593]: pgmap v184: 1 pgs: 1 remapped+peering; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2022-05-13T23:11:48.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:11:48 smithi115 ceph-mon[78593]: from='client.? 172.21.15.115:0/3046374397' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a58f2c0e-e487-41cf-8d2e-8c937cac9ebc"}]: dispatch 2022-05-13T23:11:48.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:11:48 smithi115 ceph-mon[78593]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a58f2c0e-e487-41cf-8d2e-8c937cac9ebc"}]: dispatch 2022-05-13T23:11:48.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:11:48 smithi115 ceph-mon[78593]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a58f2c0e-e487-41cf-8d2e-8c937cac9ebc"}]': finished 2022-05-13T23:11:48.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:11:48 smithi115 ceph-mon[78593]: osdmap e55: 9 total, 8 up, 9 in 2022-05-13T23:11:48.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:11:48 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2022-05-13T23:11:48.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:11:48 smithi115 ceph-mon[78593]: from='client.? 172.21.15.115:0/1903740850' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2022-05-13T23:11:50.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:11:50 smithi047 ceph-mon[77147]: pgmap v187: 1 pgs: 1 remapped+peering; 577 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2022-05-13T23:11:50.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:11:50 smithi047 ceph-mon[77147]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2022-05-13T23:11:50.572 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:11:50 smithi002 ceph-mon[71816]: pgmap v187: 1 pgs: 1 remapped+peering; 577 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2022-05-13T23:11:50.574 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:11:50 smithi002 ceph-mon[71816]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2022-05-13T23:11:50.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:11:50 smithi115 ceph-mon[78593]: pgmap v187: 1 pgs: 1 remapped+peering; 577 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2022-05-13T23:11:50.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:11:50 smithi115 ceph-mon[78593]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2022-05-13T23:11:51.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:11:51 smithi047 ceph-mon[77147]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2022-05-13T23:11:51.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:11:51 smithi047 ceph-mon[77147]: Cluster is now healthy 2022-05-13T23:11:51.572 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:11:51 smithi002 ceph-mon[71816]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2022-05-13T23:11:51.573 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:11:51 smithi002 ceph-mon[71816]: Cluster is now healthy 2022-05-13T23:11:51.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:11:51 smithi115 ceph-mon[78593]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2022-05-13T23:11:51.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:11:51 smithi115 ceph-mon[78593]: Cluster is now healthy 2022-05-13T23:11:52.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:11:52 smithi047 ceph-mon[77147]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2022-05-13T23:11:52.572 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:11:52 smithi002 ceph-mon[71816]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2022-05-13T23:11:52.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:11:52 smithi115 ceph-mon[78593]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2022-05-13T23:11:54.528 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:11:54 smithi115 ceph-mon[78593]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2022-05-13T23:11:54.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:11:54 smithi047 ceph-mon[77147]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2022-05-13T23:11:54.572 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:11:54 smithi002 ceph-mon[71816]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2022-05-13T23:11:55.574 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:11:55 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2022-05-13T23:11:55.574 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:11:55 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-13T23:11:55.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:11:55 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2022-05-13T23:11:55.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:11:55 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-13T23:11:55.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:11:55 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2022-05-13T23:11:55.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:11:55 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-13T23:11:56.521 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:11:56 smithi115 ceph-mon[78593]: Deploying daemon osd.8 on smithi115 2022-05-13T23:11:56.521 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:11:56 smithi115 ceph-mon[78593]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2022-05-13T23:11:56.572 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:11:56 smithi002 ceph-mon[71816]: Deploying daemon osd.8 on smithi115 2022-05-13T23:11:56.573 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:11:56 smithi002 ceph-mon[71816]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2022-05-13T23:11:56.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:11:56 smithi047 ceph-mon[77147]: Deploying daemon osd.8 on smithi115 2022-05-13T23:11:56.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:11:56 smithi047 ceph-mon[77147]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2022-05-13T23:11:58.433 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:11:58 smithi115 ceph-mon[78593]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2022-05-13T23:11:58.573 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:11:58 smithi002 ceph-mon[71816]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2022-05-13T23:11:58.797 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:11:58 smithi047 ceph-mon[77147]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2022-05-13T23:12:00.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:12:00 smithi047 ceph-mon[77147]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2022-05-13T23:12:00.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:12:00 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:12:00.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:12:00 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-13T23:12:00.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:12:00 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-13T23:12:00.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:12:00 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-13T23:12:00.572 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:12:00 smithi002 ceph-mon[71816]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2022-05-13T23:12:00.573 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:12:00 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:12:00.573 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:12:00 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-13T23:12:00.574 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:12:00 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-13T23:12:00.574 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:12:00 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-13T23:12:00.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:12:00 smithi115 ceph-mon[78593]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2022-05-13T23:12:00.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:12:00 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:12:00.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:12:00 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-13T23:12:00.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:12:00 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-13T23:12:00.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:12:00 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-13T23:12:01.983 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:12:01.981+0000 7efdf6ffd700 1 -- 172.21.15.115:0/1503863831 <== mgr.14148 v2:172.21.15.2:6800/2788339627 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7efe0802a500 con 0x7efdf0096d60 2022-05-13T23:12:01.984 INFO:teuthology.orchestra.run.smithi115.stdout:Created osd(s) 8 on host 'smithi115' 2022-05-13T23:12:01.985 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:12:01.983+0000 7efe0c884700 1 -- 172.21.15.115:0/1503863831 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7efdf0096d60 msgr2=0x7efdf0099210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:12:01.985 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:12:01.983+0000 7efe0c884700 1 --2- 172.21.15.115:0/1503863831 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7efdf0096d60 0x7efdf0099210 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7efdf8009d60 tx=0x7efdf8009cb0 comp rx=0 tx=0).stop 2022-05-13T23:12:01.986 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:12:01.983+0000 7efe0c884700 1 -- 172.21.15.115:0/1503863831 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7efe080fb7a0 msgr2=0x7efe081111f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:12:01.986 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:12:01.983+0000 7efe0c884700 1 --2- 172.21.15.115:0/1503863831 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7efe080fb7a0 0x7efe081111f0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7efdfc00b530 tx=0x7efdfc0090f0 comp rx=0 tx=0).stop 2022-05-13T23:12:01.986 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:12:01.983+0000 7efe0c884700 1 -- 172.21.15.115:0/1503863831 shutdown_connections 2022-05-13T23:12:01.989 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:12:01.983+0000 7efe0c884700 1 --2- 172.21.15.115:0/1503863831 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efe080fc130 0x7efe08113870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:12:01.989 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:12:01.983+0000 7efe0c884700 1 --2- 172.21.15.115:0/1503863831 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7efdf0096d60 0x7efdf0099210 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:12:01.989 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:12:01.983+0000 7efe0c884700 1 --2- 172.21.15.115:0/1503863831 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7efe080fa500 0x7efe08064960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:12:01.990 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:12:01.983+0000 7efe0c884700 1 --2- 172.21.15.115:0/1503863831 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7efe080fb7a0 0x7efe081111f0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:12:01.990 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:12:01.983+0000 7efe0c884700 1 -- 172.21.15.115:0/1503863831 >> 172.21.15.115:0/1503863831 conn(0x7efe080f5cf0 msgr2=0x7efe080f8040 unknown :-1 s=STATE_NONE l=0).mark_down 2022-05-13T23:12:01.990 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:12:01.983+0000 7efe0c884700 1 -- 172.21.15.115:0/1503863831 shutdown_connections 2022-05-13T23:12:01.990 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:12:01.983+0000 7efe0c884700 1 -- 172.21.15.115:0/1503863831 wait complete. 2022-05-13T23:12:02.461 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:12:02 smithi115 ceph-mon[78593]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2022-05-13T23:12:02.462 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:12:02 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:12:02.462 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:12:02 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:12:02.462 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:12:02 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:12:02.462 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:12:02 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-13T23:12:02.462 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:12:02 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-13T23:12:02.463 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:12:02 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-13T23:12:02.464 DEBUG:teuthology.orchestra.run.smithi115:osd.8> sudo journalctl -f -n 0 -u ceph-16cf5f88-d311-11ec-be40-001a4aab830c@osd.8.service 2022-05-13T23:12:02.467 INFO:tasks.cephadm:Deploying osd.9 on smithi002 with /dev/vg_nvme/lv_1... 2022-05-13T23:12:02.468 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:23b0658391ce94ca40799469eec00c28b298816c shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 16cf5f88-d311-11ec-be40-001a4aab830c -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2022-05-13T23:12:02.497 INFO:journalctl@ceph.osd.8.smithi115.stdout:-- Logs begin at Fri 2022-05-13 22:45:29 UTC. -- 2022-05-13T23:12:02.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:12:02 smithi047 ceph-mon[77147]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2022-05-13T23:12:02.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:12:02 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:12:02.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:12:02 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:12:02.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:12:02 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:12:02.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:12:02 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-13T23:12:02.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:12:02 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-13T23:12:02.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:12:02 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-13T23:12:02.543 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:12:02 smithi002 ceph-mon[71816]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2022-05-13T23:12:02.544 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:12:02 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:12:02.544 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:12:02 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:12:02.544 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:12:02 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:12:02.545 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:12:02 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-13T23:12:02.545 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:12:02 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-13T23:12:02.546 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:12:02 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-13T23:12:03.143 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/16cf5f88-d311-11ec-be40-001a4aab830c/mon.a/config 2022-05-13T23:12:03.536 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:12:03 smithi002 ceph-mon[71816]: from='osd.8 [v2:172.21.15.115:6816/3959632033,v1:172.21.15.115:6817/3959632033]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2022-05-13T23:12:03.536 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:12:03 smithi002 ceph-mon[71816]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2022-05-13T23:12:03.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:12:03 smithi047 ceph-mon[77147]: from='osd.8 [v2:172.21.15.115:6816/3959632033,v1:172.21.15.115:6817/3959632033]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2022-05-13T23:12:03.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:12:03 smithi047 ceph-mon[77147]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2022-05-13T23:12:03.566 INFO:journalctl@ceph.osd.8.smithi115.stdout:May 13 23:12:03 smithi115 conmon[96252]: 2022-05-13T23:12:03.177+0000 7f226ac663c0 -1 osd.8 0 log_to_monitors true 2022-05-13T23:12:03.567 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:12:03 smithi115 ceph-mon[78593]: from='osd.8 [v2:172.21.15.115:6816/3959632033,v1:172.21.15.115:6817/3959632033]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2022-05-13T23:12:03.567 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:12:03 smithi115 ceph-mon[78593]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2022-05-13T23:12:04.322 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:12:04 smithi002 ceph-mon[71816]: pgmap v194: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2022-05-13T23:12:04.323 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:12:04 smithi002 ceph-mon[71816]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2022-05-13T23:12:04.323 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:12:04 smithi002 ceph-mon[71816]: osdmap e56: 9 total, 8 up, 9 in 2022-05-13T23:12:04.323 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:12:04 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2022-05-13T23:12:04.323 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:12:04 smithi002 ceph-mon[71816]: from='osd.8 [v2:172.21.15.115:6816/3959632033,v1:172.21.15.115:6817/3959632033]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi115", "root=default"]}]: dispatch 2022-05-13T23:12:04.324 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:12:04 smithi002 ceph-mon[71816]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi115", "root=default"]}]: dispatch 2022-05-13T23:12:04.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:12:04 smithi047 ceph-mon[77147]: pgmap v194: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2022-05-13T23:12:04.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:12:04 smithi047 ceph-mon[77147]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2022-05-13T23:12:04.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:12:04 smithi047 ceph-mon[77147]: osdmap e56: 9 total, 8 up, 9 in 2022-05-13T23:12:04.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:12:04 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2022-05-13T23:12:04.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:12:04 smithi047 ceph-mon[77147]: from='osd.8 [v2:172.21.15.115:6816/3959632033,v1:172.21.15.115:6817/3959632033]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi115", "root=default"]}]: dispatch 2022-05-13T23:12:04.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:12:04 smithi047 ceph-mon[77147]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi115", "root=default"]}]: dispatch 2022-05-13T23:12:04.608 INFO:journalctl@ceph.osd.8.smithi115.stdout:May 13 23:12:04 smithi115 conmon[96252]: 2022-05-13T23:12:04.261+0000 7f226166a700 -1 osd.8 0 waiting for initial osdmap 2022-05-13T23:12:04.609 INFO:journalctl@ceph.osd.8.smithi115.stdout:May 13 23:12:04 smithi115 conmon[96252]: 2022-05-13T23:12:04.275+0000 7f225e004700 -1 osd.8 57 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2022-05-13T23:12:04.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:12:04 smithi115 ceph-mon[78593]: pgmap v194: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2022-05-13T23:12:04.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:12:04 smithi115 ceph-mon[78593]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2022-05-13T23:12:04.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:12:04 smithi115 ceph-mon[78593]: osdmap e56: 9 total, 8 up, 9 in 2022-05-13T23:12:04.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:12:04 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2022-05-13T23:12:04.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:12:04 smithi115 ceph-mon[78593]: from='osd.8 [v2:172.21.15.115:6816/3959632033,v1:172.21.15.115:6817/3959632033]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi115", "root=default"]}]: dispatch 2022-05-13T23:12:04.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:12:04 smithi115 ceph-mon[78593]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi115", "root=default"]}]: dispatch 2022-05-13T23:12:05.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:12:05 smithi047 ceph-mon[77147]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi115", "root=default"]}]': finished 2022-05-13T23:12:05.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:12:05 smithi047 ceph-mon[77147]: osdmap e57: 9 total, 8 up, 9 in 2022-05-13T23:12:05.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:12:05 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2022-05-13T23:12:05.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:12:05 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2022-05-13T23:12:05.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:12:05 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:12:05.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:12:05 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-05-13T23:12:05.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:12:05 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-05-13T23:12:05.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:12:05 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-05-13T23:12:05.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:12:05 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:12:05.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:12:05 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:12:05.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:12:05 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-05-13T23:12:05.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:12:05 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-05-13T23:12:05.573 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:12:05 smithi002 ceph-mon[71816]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi115", "root=default"]}]': finished 2022-05-13T23:12:05.573 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:12:05 smithi002 ceph-mon[71816]: osdmap e57: 9 total, 8 up, 9 in 2022-05-13T23:12:05.573 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:12:05 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2022-05-13T23:12:05.574 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:12:05 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2022-05-13T23:12:05.574 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:12:05 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:12:05.574 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:12:05 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-05-13T23:12:05.574 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:12:05 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-05-13T23:12:05.575 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:12:05 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-05-13T23:12:05.575 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:12:05 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:12:05.575 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:12:05 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:12:05.575 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:12:05 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-05-13T23:12:05.576 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:12:05 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-05-13T23:12:05.576 INFO:teuthology.orchestra.run.smithi002.stderr:--> Zapping: /dev/vg_nvme/lv_1 2022-05-13T23:12:05.577 INFO:teuthology.orchestra.run.smithi002.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2022-05-13T23:12:05.577 INFO:teuthology.orchestra.run.smithi002.stderr: stderr: 10+0 records in 2022-05-13T23:12:05.578 INFO:teuthology.orchestra.run.smithi002.stderr:10+0 records out 2022-05-13T23:12:05.578 INFO:teuthology.orchestra.run.smithi002.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0630981 s, 166 MB/s 2022-05-13T23:12:05.578 INFO:teuthology.orchestra.run.smithi002.stderr:--> Zapping successful for: 2022-05-13T23:12:05.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:12:05 smithi115 ceph-mon[78593]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi115", "root=default"]}]': finished 2022-05-13T23:12:05.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:12:05 smithi115 ceph-mon[78593]: osdmap e57: 9 total, 8 up, 9 in 2022-05-13T23:12:05.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:12:05 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2022-05-13T23:12:05.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:12:05 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2022-05-13T23:12:05.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:12:05 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:12:05.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:12:05 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-05-13T23:12:05.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:12:05 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-05-13T23:12:05.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:12:05 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-05-13T23:12:05.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:12:05 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:12:05.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:12:05 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:12:05.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:12:05 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-05-13T23:12:05.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:12:05 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-05-13T23:12:06.120 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:23b0658391ce94ca40799469eec00c28b298816c shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 16cf5f88-d311-11ec-be40-001a4aab830c -- ceph orch daemon add osd smithi002:vg_nvme/lv_1 2022-05-13T23:12:06.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:12:06 smithi047 ceph-mon[77147]: purged_snaps scrub starts 2022-05-13T23:12:06.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:12:06 smithi047 ceph-mon[77147]: purged_snaps scrub ok 2022-05-13T23:12:06.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:12:06 smithi047 ceph-mon[77147]: Detected new or changed devices on smithi115 2022-05-13T23:12:06.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:12:06 smithi047 ceph-mon[77147]: Adjusting osd_memory_target on smithi115 to 5724M 2022-05-13T23:12:06.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:12:06 smithi047 ceph-mon[77147]: pgmap v197: 1 pgs: 1 peering; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2022-05-13T23:12:06.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:12:06 smithi047 ceph-mon[77147]: osd.8 [v2:172.21.15.115:6816/3959632033,v1:172.21.15.115:6817/3959632033] boot 2022-05-13T23:12:06.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:12:06 smithi047 ceph-mon[77147]: osdmap e58: 9 total, 9 up, 9 in 2022-05-13T23:12:06.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:12:06 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2022-05-13T23:12:06.572 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:12:06 smithi002 ceph-mon[71816]: purged_snaps scrub starts 2022-05-13T23:12:06.573 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:12:06 smithi002 ceph-mon[71816]: purged_snaps scrub ok 2022-05-13T23:12:06.573 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:12:06 smithi002 ceph-mon[71816]: Detected new or changed devices on smithi115 2022-05-13T23:12:06.573 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:12:06 smithi002 ceph-mon[71816]: Adjusting osd_memory_target on smithi115 to 5724M 2022-05-13T23:12:06.573 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:12:06 smithi002 ceph-mon[71816]: pgmap v197: 1 pgs: 1 peering; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2022-05-13T23:12:06.574 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:12:06 smithi002 ceph-mon[71816]: osd.8 [v2:172.21.15.115:6816/3959632033,v1:172.21.15.115:6817/3959632033] boot 2022-05-13T23:12:06.574 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:12:06 smithi002 ceph-mon[71816]: osdmap e58: 9 total, 9 up, 9 in 2022-05-13T23:12:06.574 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:12:06 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2022-05-13T23:12:06.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:12:06 smithi115 ceph-mon[78593]: purged_snaps scrub starts 2022-05-13T23:12:06.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:12:06 smithi115 ceph-mon[78593]: purged_snaps scrub ok 2022-05-13T23:12:06.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:12:06 smithi115 ceph-mon[78593]: Detected new or changed devices on smithi115 2022-05-13T23:12:06.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:12:06 smithi115 ceph-mon[78593]: Adjusting osd_memory_target on smithi115 to 5724M 2022-05-13T23:12:06.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:12:06 smithi115 ceph-mon[78593]: pgmap v197: 1 pgs: 1 peering; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2022-05-13T23:12:06.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:12:06 smithi115 ceph-mon[78593]: osd.8 [v2:172.21.15.115:6816/3959632033,v1:172.21.15.115:6817/3959632033] boot 2022-05-13T23:12:06.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:12:06 smithi115 ceph-mon[78593]: osdmap e58: 9 total, 9 up, 9 in 2022-05-13T23:12:06.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:12:06 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2022-05-13T23:12:06.800 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/16cf5f88-d311-11ec-be40-001a4aab830c/mon.a/config 2022-05-13T23:12:07.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:12:07 smithi047 ceph-mon[77147]: osdmap e59: 9 total, 9 up, 9 in 2022-05-13T23:12:07.572 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:12:07 smithi002 ceph-mon[71816]: osdmap e59: 9 total, 9 up, 9 in 2022-05-13T23:12:07.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:12:07 smithi115 ceph-mon[78593]: osdmap e59: 9 total, 9 up, 9 in 2022-05-13T23:12:08.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:12:08 smithi047 ceph-mon[77147]: pgmap v200: 1 pgs: 1 peering; 577 KiB data, 53 MiB used, 805 GiB / 805 GiB avail 2022-05-13T23:12:08.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:12:08 smithi047 ceph-mon[77147]: osdmap e60: 9 total, 9 up, 9 in 2022-05-13T23:12:08.572 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:12:08 smithi002 ceph-mon[71816]: pgmap v200: 1 pgs: 1 peering; 577 KiB data, 53 MiB used, 805 GiB / 805 GiB avail 2022-05-13T23:12:08.573 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:12:08 smithi002 ceph-mon[71816]: osdmap e60: 9 total, 9 up, 9 in 2022-05-13T23:12:08.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:12:08 smithi115 ceph-mon[78593]: pgmap v200: 1 pgs: 1 peering; 577 KiB data, 53 MiB used, 805 GiB / 805 GiB avail 2022-05-13T23:12:08.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:12:08 smithi115 ceph-mon[78593]: osdmap e60: 9 total, 9 up, 9 in 2022-05-13T23:12:08.624 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:12:08.622+0000 7fd442750700 1 -- 172.21.15.2:0/3293080086 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd43c0ff440 msgr2=0x7fd43c105d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:12:08.625 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:12:08.622+0000 7fd442750700 1 --2- 172.21.15.2:0/3293080086 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd43c0ff440 0x7fd43c105d00 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fd434009be0 tx=0x7fd434004d10 comp rx=0 tx=0).stop 2022-05-13T23:12:08.625 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:12:08.622+0000 7fd442750700 1 -- 172.21.15.2:0/3293080086 shutdown_connections 2022-05-13T23:12:08.625 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:12:08.622+0000 7fd442750700 1 --2- 172.21.15.2:0/3293080086 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd43c0ff440 0x7fd43c105d00 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:12:08.625 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:12:08.622+0000 7fd442750700 1 --2- 172.21.15.2:0/3293080086 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd43c0fd810 0x7fd43c0fdc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:12:08.625 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:12:08.622+0000 7fd442750700 1 --2- 172.21.15.2:0/3293080086 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fd43c0feab0 0x7fd43c0fef00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:12:08.626 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:12:08.622+0000 7fd442750700 1 -- 172.21.15.2:0/3293080086 >> 172.21.15.2:0/3293080086 conn(0x7fd43c0f9030 msgr2=0x7fd43c0fb430 unknown :-1 s=STATE_NONE l=0).mark_down 2022-05-13T23:12:08.626 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:12:08.622+0000 7fd442750700 1 -- 172.21.15.2:0/3293080086 shutdown_connections 2022-05-13T23:12:08.626 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:12:08.622+0000 7fd442750700 1 -- 172.21.15.2:0/3293080086 wait complete. 2022-05-13T23:12:08.626 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:12:08.623+0000 7fd442750700 1 Processor -- start 2022-05-13T23:12:08.627 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:12:08.623+0000 7fd442750700 1 -- start start 2022-05-13T23:12:08.629 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:12:08.623+0000 7fd442750700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd43c0fd810 0x7fd43c10e470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:12:08.629 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:12:08.623+0000 7fd442750700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fd43c0feab0 0x7fd43c10e9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:12:08.629 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:12:08.623+0000 7fd442750700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd43c0ff440 0x7fd43c115a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:12:08.630 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:12:08.623+0000 7fd442750700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd43c115f80 con 0x7fd43c0fd810 2022-05-13T23:12:08.630 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:12:08.623+0000 7fd442750700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd43c1160f0 con 0x7fd43c0ff440 2022-05-13T23:12:08.630 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:12:08.623+0000 7fd442750700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd43c1162b0 con 0x7fd43c0feab0 2022-05-13T23:12:08.630 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:12:08.624+0000 7fd43bfff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd43c0fd810 0x7fd43c10e470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:12:08.630 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:12:08.624+0000 7fd43bfff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd43c0fd810 0x7fd43c10e470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:57824/0 (socket says 172.21.15.2:57824) 2022-05-13T23:12:08.631 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:12:08.624+0000 7fd43bfff700 1 -- 172.21.15.2:0/754271756 learned_addr learned my addr 172.21.15.2:0/754271756 (peer_addr_for_me v2:172.21.15.2:0/0) 2022-05-13T23:12:08.631 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:12:08.624+0000 7fd43b7fe700 1 --2- 172.21.15.2:0/754271756 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fd43c0feab0 0x7fd43c10e9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:12:08.631 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:12:08.624+0000 7fd440ced700 1 --2- 172.21.15.2:0/754271756 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd43c0ff440 0x7fd43c115a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:12:08.631 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:12:08.624+0000 7fd43bfff700 1 -- 172.21.15.2:0/754271756 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fd43c0feab0 msgr2=0x7fd43c10e9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:12:08.632 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:12:08.624+0000 7fd43bfff700 1 --2- 172.21.15.2:0/754271756 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fd43c0feab0 0x7fd43c10e9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:12:08.632 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:12:08.624+0000 7fd43bfff700 1 -- 172.21.15.2:0/754271756 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd43c0ff440 msgr2=0x7fd43c115a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:12:08.632 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:12:08.624+0000 7fd43bfff700 1 --2- 172.21.15.2:0/754271756 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd43c0ff440 0x7fd43c115a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:12:08.633 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:12:08.624+0000 7fd43bfff700 1 -- 172.21.15.2:0/754271756 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd430005040 con 0x7fd43c0fd810 2022-05-13T23:12:08.633 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:12:08.625+0000 7fd43bfff700 1 --2- 172.21.15.2:0/754271756 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd43c0fd810 0x7fd43c10e470 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fd42c00e940 tx=0x7fd42c00b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-05-13T23:12:08.633 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:12:08.625+0000 7fd438ff9700 1 -- 172.21.15.2:0/754271756 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd42c007b40 con 0x7fd43c0fd810 2022-05-13T23:12:08.633 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:12:08.625+0000 7fd438ff9700 1 -- 172.21.15.2:0/754271756 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1087+0+0 (secure 0 0 0) 0x7fd42c00f040 con 0x7fd43c0fd810 2022-05-13T23:12:08.633 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:12:08.625+0000 7fd442750700 1 -- 172.21.15.2:0/754271756 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd4340097c0 con 0x7fd43c0fd810 2022-05-13T23:12:08.634 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:12:08.625+0000 7fd438ff9700 1 -- 172.21.15.2:0/754271756 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd42c014890 con 0x7fd43c0fd810 2022-05-13T23:12:08.634 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:12:08.625+0000 7fd442750700 1 -- 172.21.15.2:0/754271756 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd43c116960 con 0x7fd43c0fd810 2022-05-13T23:12:08.634 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:12:08.626+0000 7fd442750700 1 -- 172.21.15.2:0/754271756 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd43c108ff0 con 0x7fd43c0fd810 2022-05-13T23:12:08.635 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:12:08.628+0000 7fd438ff9700 1 -- 172.21.15.2:0/754271756 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 16) v1 ==== 125136+0+0 (secure 0 0 0) 0x7fd42c01b030 con 0x7fd43c0fd810 2022-05-13T23:12:08.636 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:12:08.629+0000 7fd438ff9700 1 --2- 172.21.15.2:0/754271756 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7fd424096d60 0x7fd424099210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:12:08.636 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:12:08.629+0000 7fd438ff9700 1 -- 172.21.15.2:0/754271756 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 5825+0+0 (secure 0 0 0) 0x7fd42c0be1e0 con 0x7fd43c0fd810 2022-05-13T23:12:08.636 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:12:08.630+0000 7fd43b7fe700 1 --2- 172.21.15.2:0/754271756 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7fd424096d60 0x7fd424099210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:12:08.636 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:12:08.630+0000 7fd43b7fe700 1 --2- 172.21.15.2:0/754271756 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7fd424096d60 0x7fd424099210 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fd4300092b0 tx=0x7fd43000b040 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-05-13T23:12:08.637 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:12:08.635+0000 7fd438ff9700 1 -- 172.21.15.2:0/754271756 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+165215 (secure 0 0 0) 0x7fd42c04b570 con 0x7fd43c0fd810 2022-05-13T23:12:08.838 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:12:08.836+0000 7fd442750700 1 -- 172.21.15.2:0/754271756 --> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi002:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) v1 -- 0x7fd43c02a530 con 0x7fd424096d60 2022-05-13T23:12:09.572 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:12:09 smithi002 ceph-mon[71816]: osdmap e61: 9 total, 9 up, 9 in 2022-05-13T23:12:09.573 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:12:09 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2022-05-13T23:12:09.573 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:12:09 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2022-05-13T23:12:09.573 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:12:09 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-13T23:12:09.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:12:09 smithi115 ceph-mon[78593]: osdmap e61: 9 total, 9 up, 9 in 2022-05-13T23:12:09.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:12:09 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2022-05-13T23:12:09.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:12:09 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2022-05-13T23:12:09.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:12:09 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-13T23:12:09.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:12:09 smithi047 ceph-mon[77147]: osdmap e61: 9 total, 9 up, 9 in 2022-05-13T23:12:09.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:12:09 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2022-05-13T23:12:09.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:12:09 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2022-05-13T23:12:09.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:12:09 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-13T23:12:10.572 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:12:10 smithi002 ceph-mon[71816]: from='client.14322 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi002:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2022-05-13T23:12:10.573 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:12:10 smithi002 ceph-mon[71816]: pgmap v203: 1 pgs: 1 peering; 577 KiB data, 53 MiB used, 805 GiB / 805 GiB avail 2022-05-13T23:12:10.573 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:12:10 smithi002 ceph-mon[71816]: osdmap e62: 9 total, 9 up, 9 in 2022-05-13T23:12:10.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:12:10 smithi115 ceph-mon[78593]: from='client.14322 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi002:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2022-05-13T23:12:10.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:12:10 smithi115 ceph-mon[78593]: pgmap v203: 1 pgs: 1 peering; 577 KiB data, 53 MiB used, 805 GiB / 805 GiB avail 2022-05-13T23:12:10.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:12:10 smithi115 ceph-mon[78593]: osdmap e62: 9 total, 9 up, 9 in 2022-05-13T23:12:10.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:12:10 smithi047 ceph-mon[77147]: from='client.14322 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi002:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2022-05-13T23:12:10.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:12:10 smithi047 ceph-mon[77147]: pgmap v203: 1 pgs: 1 peering; 577 KiB data, 53 MiB used, 805 GiB / 805 GiB avail 2022-05-13T23:12:10.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:12:10 smithi047 ceph-mon[77147]: osdmap e62: 9 total, 9 up, 9 in 2022-05-13T23:12:12.572 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:12:12 smithi002 ceph-mon[71816]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 53 MiB used, 805 GiB / 805 GiB avail; 105 KiB/s, 0 objects/s recovering 2022-05-13T23:12:12.573 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:12:12 smithi002 ceph-mon[71816]: from='client.? 172.21.15.2:0/2942171717' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "12cead96-f726-4f7a-ab84-eb191b778a5c"}]: dispatch 2022-05-13T23:12:12.574 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:12:12 smithi002 ceph-mon[71816]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "12cead96-f726-4f7a-ab84-eb191b778a5c"}]: dispatch 2022-05-13T23:12:12.574 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:12:12 smithi002 ceph-mon[71816]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "12cead96-f726-4f7a-ab84-eb191b778a5c"}]': finished 2022-05-13T23:12:12.574 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:12:12 smithi002 ceph-mon[71816]: osdmap e63: 10 total, 9 up, 10 in 2022-05-13T23:12:12.574 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:12:12 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2022-05-13T23:12:12.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:12:12 smithi115 ceph-mon[78593]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 53 MiB used, 805 GiB / 805 GiB avail; 105 KiB/s, 0 objects/s recovering 2022-05-13T23:12:12.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:12:12 smithi115 ceph-mon[78593]: from='client.? 172.21.15.2:0/2942171717' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "12cead96-f726-4f7a-ab84-eb191b778a5c"}]: dispatch 2022-05-13T23:12:12.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:12:12 smithi115 ceph-mon[78593]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "12cead96-f726-4f7a-ab84-eb191b778a5c"}]: dispatch 2022-05-13T23:12:12.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:12:12 smithi115 ceph-mon[78593]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "12cead96-f726-4f7a-ab84-eb191b778a5c"}]': finished 2022-05-13T23:12:12.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:12:12 smithi115 ceph-mon[78593]: osdmap e63: 10 total, 9 up, 10 in 2022-05-13T23:12:12.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:12:12 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2022-05-13T23:12:12.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:12:12 smithi047 ceph-mon[77147]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 53 MiB used, 805 GiB / 805 GiB avail; 105 KiB/s, 0 objects/s recovering 2022-05-13T23:12:12.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:12:12 smithi047 ceph-mon[77147]: from='client.? 172.21.15.2:0/2942171717' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "12cead96-f726-4f7a-ab84-eb191b778a5c"}]: dispatch 2022-05-13T23:12:12.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:12:12 smithi047 ceph-mon[77147]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "12cead96-f726-4f7a-ab84-eb191b778a5c"}]: dispatch 2022-05-13T23:12:12.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:12:12 smithi047 ceph-mon[77147]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "12cead96-f726-4f7a-ab84-eb191b778a5c"}]': finished 2022-05-13T23:12:12.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:12:12 smithi047 ceph-mon[77147]: osdmap e63: 10 total, 9 up, 10 in 2022-05-13T23:12:12.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:12:12 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2022-05-13T23:12:13.572 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:12:13 smithi002 ceph-mon[71816]: from='client.? 172.21.15.2:0/1948822485' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2022-05-13T23:12:13.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:12:13 smithi115 ceph-mon[78593]: from='client.? 172.21.15.2:0/1948822485' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2022-05-13T23:12:13.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:12:13 smithi047 ceph-mon[77147]: from='client.? 172.21.15.2:0/1948822485' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2022-05-13T23:12:14.572 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:12:14 smithi002 ceph-mon[71816]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 54 MiB used, 805 GiB / 805 GiB avail; 87 KiB/s, 0 objects/s recovering 2022-05-13T23:12:14.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:12:14 smithi115 ceph-mon[78593]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 54 MiB used, 805 GiB / 805 GiB avail; 87 KiB/s, 0 objects/s recovering 2022-05-13T23:12:14.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:12:14 smithi047 ceph-mon[77147]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 54 MiB used, 805 GiB / 805 GiB avail; 87 KiB/s, 0 objects/s recovering 2022-05-13T23:12:16.572 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:12:16 smithi002 ceph-mon[71816]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 54 MiB used, 805 GiB / 805 GiB avail; 75 KiB/s, 0 objects/s recovering 2022-05-13T23:12:16.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:12:16 smithi115 ceph-mon[78593]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 54 MiB used, 805 GiB / 805 GiB avail; 75 KiB/s, 0 objects/s recovering 2022-05-13T23:12:16.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:12:16 smithi047 ceph-mon[77147]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 54 MiB used, 805 GiB / 805 GiB avail; 75 KiB/s, 0 objects/s recovering 2022-05-13T23:12:18.572 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:12:18 smithi002 ceph-mon[71816]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 54 MiB used, 805 GiB / 805 GiB avail; 65 KiB/s, 0 objects/s recovering 2022-05-13T23:12:18.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:12:18 smithi115 ceph-mon[78593]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 54 MiB used, 805 GiB / 805 GiB avail; 65 KiB/s, 0 objects/s recovering 2022-05-13T23:12:18.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:12:18 smithi047 ceph-mon[77147]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 54 MiB used, 805 GiB / 805 GiB avail; 65 KiB/s, 0 objects/s recovering 2022-05-13T23:12:20.572 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:12:20 smithi002 ceph-mon[71816]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 54 MiB used, 805 GiB / 805 GiB avail; 52 KiB/s, 0 objects/s recovering 2022-05-13T23:12:20.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:12:20 smithi115 ceph-mon[78593]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 54 MiB used, 805 GiB / 805 GiB avail; 52 KiB/s, 0 objects/s recovering 2022-05-13T23:12:20.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:12:20 smithi047 ceph-mon[77147]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 54 MiB used, 805 GiB / 805 GiB avail; 52 KiB/s, 0 objects/s recovering 2022-05-13T23:12:21.452 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:12:21 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2022-05-13T23:12:21.453 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:12:21 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-13T23:12:21.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:12:21 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2022-05-13T23:12:21.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:12:21 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-13T23:12:21.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:12:21 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2022-05-13T23:12:21.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:12:21 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-13T23:12:22.605 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:12:22 smithi002 ceph-mon[71816]: Deploying daemon osd.9 on smithi002 2022-05-13T23:12:22.606 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:12:22 smithi002 ceph-mon[71816]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 54 MiB used, 805 GiB / 805 GiB avail 2022-05-13T23:12:22.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:12:22 smithi115 ceph-mon[78593]: Deploying daemon osd.9 on smithi002 2022-05-13T23:12:22.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:12:22 smithi115 ceph-mon[78593]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 54 MiB used, 805 GiB / 805 GiB avail 2022-05-13T23:12:22.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:12:22 smithi047 ceph-mon[77147]: Deploying daemon osd.9 on smithi002 2022-05-13T23:12:22.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:12:22 smithi047 ceph-mon[77147]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 54 MiB used, 805 GiB / 805 GiB avail 2022-05-13T23:12:24.593 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:12:24 smithi002 ceph-mon[71816]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 54 MiB used, 805 GiB / 805 GiB avail 2022-05-13T23:12:24.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:12:24 smithi115 ceph-mon[78593]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 54 MiB used, 805 GiB / 805 GiB avail 2022-05-13T23:12:24.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:12:24 smithi047 ceph-mon[77147]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 54 MiB used, 805 GiB / 805 GiB avail 2022-05-13T23:12:26.590 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:12:26 smithi002 ceph-mon[71816]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 54 MiB used, 805 GiB / 805 GiB avail 2022-05-13T23:12:26.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:12:26 smithi115 ceph-mon[78593]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 54 MiB used, 805 GiB / 805 GiB avail 2022-05-13T23:12:26.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:12:26 smithi047 ceph-mon[77147]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 54 MiB used, 805 GiB / 805 GiB avail 2022-05-13T23:12:28.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:12:28 smithi115 ceph-mon[78593]: pgmap v214: 1 pgs: 1 active+clean; 577 KiB data, 54 MiB used, 805 GiB / 805 GiB avail 2022-05-13T23:12:28.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:12:28 smithi047 ceph-mon[77147]: pgmap v214: 1 pgs: 1 active+clean; 577 KiB data, 54 MiB used, 805 GiB / 805 GiB avail 2022-05-13T23:12:28.827 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:12:28 smithi002 ceph-mon[71816]: pgmap v214: 1 pgs: 1 active+clean; 577 KiB data, 54 MiB used, 805 GiB / 805 GiB avail 2022-05-13T23:12:30.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:12:30 smithi115 ceph-mon[78593]: pgmap v215: 1 pgs: 1 active+clean; 577 KiB data, 54 MiB used, 805 GiB / 805 GiB avail 2022-05-13T23:12:30.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:12:30 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:12:30.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:12:30 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-13T23:12:30.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:12:30 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-13T23:12:30.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:12:30 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-13T23:12:30.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:12:30 smithi047 ceph-mon[77147]: pgmap v215: 1 pgs: 1 active+clean; 577 KiB data, 54 MiB used, 805 GiB / 805 GiB avail 2022-05-13T23:12:30.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:12:30 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:12:30.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:12:30 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-13T23:12:30.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:12:30 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-13T23:12:30.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:12:30 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-13T23:12:30.813 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:12:30 smithi002 ceph-mon[71816]: pgmap v215: 1 pgs: 1 active+clean; 577 KiB data, 54 MiB used, 805 GiB / 805 GiB avail 2022-05-13T23:12:30.814 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:12:30 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:12:30.814 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:12:30 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-13T23:12:30.814 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:12:30 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-13T23:12:30.814 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:12:30 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-13T23:12:32.572 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:12:32 smithi002 ceph-mon[71816]: pgmap v216: 1 pgs: 1 active+clean; 577 KiB data, 54 MiB used, 805 GiB / 805 GiB avail 2022-05-13T23:12:32.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:12:32 smithi115 ceph-mon[78593]: pgmap v216: 1 pgs: 1 active+clean; 577 KiB data, 54 MiB used, 805 GiB / 805 GiB avail 2022-05-13T23:12:32.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:12:32 smithi047 ceph-mon[77147]: pgmap v216: 1 pgs: 1 active+clean; 577 KiB data, 54 MiB used, 805 GiB / 805 GiB avail 2022-05-13T23:12:33.837 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:12:33.835+0000 7fd438ff9700 1 -- 172.21.15.2:0/754271756 <== mgr.14148 v2:172.21.15.2:6800/2788339627 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7fd43c02a530 con 0x7fd424096d60 2022-05-13T23:12:33.838 INFO:teuthology.orchestra.run.smithi002.stdout:Created osd(s) 9 on host 'smithi002' 2022-05-13T23:12:33.840 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:12:33.838+0000 7fd442750700 1 -- 172.21.15.2:0/754271756 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7fd424096d60 msgr2=0x7fd424099210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:12:33.841 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:12:33.838+0000 7fd442750700 1 --2- 172.21.15.2:0/754271756 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7fd424096d60 0x7fd424099210 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fd4300092b0 tx=0x7fd43000b040 comp rx=0 tx=0).stop 2022-05-13T23:12:33.841 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:12:33.838+0000 7fd442750700 1 -- 172.21.15.2:0/754271756 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd43c0fd810 msgr2=0x7fd43c10e470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:12:33.842 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:12:33.838+0000 7fd442750700 1 --2- 172.21.15.2:0/754271756 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd43c0fd810 0x7fd43c10e470 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fd42c00e940 tx=0x7fd42c00b040 comp rx=0 tx=0).stop 2022-05-13T23:12:33.842 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:12:33.838+0000 7fd442750700 1 -- 172.21.15.2:0/754271756 shutdown_connections 2022-05-13T23:12:33.843 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:12:33.838+0000 7fd442750700 1 --2- 172.21.15.2:0/754271756 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd43c0ff440 0x7fd43c115a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:12:33.844 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:12:33.838+0000 7fd442750700 1 --2- 172.21.15.2:0/754271756 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7fd424096d60 0x7fd424099210 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:12:33.844 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:12:33.838+0000 7fd442750700 1 --2- 172.21.15.2:0/754271756 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd43c0fd810 0x7fd43c10e470 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:12:33.845 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:12:33.838+0000 7fd442750700 1 --2- 172.21.15.2:0/754271756 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fd43c0feab0 0x7fd43c10e9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:12:33.845 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:12:33.838+0000 7fd442750700 1 -- 172.21.15.2:0/754271756 >> 172.21.15.2:0/754271756 conn(0x7fd43c0f9030 msgr2=0x7fd43c101ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-05-13T23:12:33.845 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:12:33.838+0000 7fd442750700 1 -- 172.21.15.2:0/754271756 shutdown_connections 2022-05-13T23:12:33.845 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:12:33.838+0000 7fd442750700 1 -- 172.21.15.2:0/754271756 wait complete. 2022-05-13T23:12:34.422 DEBUG:teuthology.orchestra.run.smithi002:osd.9> sudo journalctl -f -n 0 -u ceph-16cf5f88-d311-11ec-be40-001a4aab830c@osd.9.service 2022-05-13T23:12:34.428 INFO:tasks.cephadm:Deploying osd.10 on smithi047 with /dev/vg_nvme/lv_1... 2022-05-13T23:12:34.428 DEBUG:teuthology.orchestra.run.smithi047:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:23b0658391ce94ca40799469eec00c28b298816c shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 16cf5f88-d311-11ec-be40-001a4aab830c -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2022-05-13T23:12:34.509 INFO:journalctl@ceph.osd.9.smithi002.stdout:-- Logs begin at Fri 2022-05-13 22:48:11 UTC. -- 2022-05-13T23:12:34.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:12:34 smithi047 ceph-mon[77147]: pgmap v217: 1 pgs: 1 active+clean; 577 KiB data, 54 MiB used, 805 GiB / 805 GiB avail 2022-05-13T23:12:34.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:12:34 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:12:34.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:12:34 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:12:34.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:12:34 smithi047 ceph-mon[77147]: from='osd.9 [v2:172.21.15.2:6826/3004162255,v1:172.21.15.2:6827/3004162255]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2022-05-13T23:12:34.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:12:34 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:12:34.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:12:34 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-13T23:12:34.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:12:34 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-13T23:12:34.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:12:34 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-13T23:12:34.822 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:12:34 smithi002 ceph-mon[71816]: pgmap v217: 1 pgs: 1 active+clean; 577 KiB data, 54 MiB used, 805 GiB / 805 GiB avail 2022-05-13T23:12:34.823 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:12:34 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:12:34.823 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:12:34 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:12:34.823 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:12:34 smithi002 ceph-mon[71816]: from='osd.9 [v2:172.21.15.2:6826/3004162255,v1:172.21.15.2:6827/3004162255]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2022-05-13T23:12:34.823 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:12:34 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:12:34.823 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:12:34 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-13T23:12:34.824 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:12:34 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-13T23:12:34.824 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:12:34 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-13T23:12:34.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:12:34 smithi115 ceph-mon[78593]: pgmap v217: 1 pgs: 1 active+clean; 577 KiB data, 54 MiB used, 805 GiB / 805 GiB avail 2022-05-13T23:12:34.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:12:34 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:12:34.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:12:34 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:12:34.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:12:34 smithi115 ceph-mon[78593]: from='osd.9 [v2:172.21.15.2:6826/3004162255,v1:172.21.15.2:6827/3004162255]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2022-05-13T23:12:34.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:12:34 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:12:34.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:12:34 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-13T23:12:34.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:12:34 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-13T23:12:34.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:12:34 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-13T23:12:34.891 INFO:teuthology.orchestra.run.smithi047.stderr:Inferring config /var/lib/ceph/16cf5f88-d311-11ec-be40-001a4aab830c/mon.b/config 2022-05-13T23:12:35.664 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:12:35 smithi047 ceph-mon[77147]: from='osd.9 [v2:172.21.15.2:6826/3004162255,v1:172.21.15.2:6827/3004162255]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2022-05-13T23:12:35.664 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:12:35 smithi047 ceph-mon[77147]: osdmap e64: 10 total, 9 up, 10 in 2022-05-13T23:12:35.664 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:12:35 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2022-05-13T23:12:35.665 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:12:35 smithi047 ceph-mon[77147]: from='osd.9 [v2:172.21.15.2:6826/3004162255,v1:172.21.15.2:6827/3004162255]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi002", "root=default"]}]: dispatch 2022-05-13T23:12:35.769 INFO:journalctl@ceph.osd.9.smithi002.stdout:May 13 23:12:35 smithi002 conmon[103678]: 2022-05-13T23:12:35.377+0000 7fd008073700 -1 osd.9 0 waiting for initial osdmap 2022-05-13T23:12:35.769 INFO:journalctl@ceph.osd.9.smithi002.stdout:May 13 23:12:35 smithi002 conmon[103678]: 2022-05-13T23:12:35.402+0000 7fd001a07700 -1 osd.9 65 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2022-05-13T23:12:35.770 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:12:35 smithi002 ceph-mon[71816]: from='osd.9 [v2:172.21.15.2:6826/3004162255,v1:172.21.15.2:6827/3004162255]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2022-05-13T23:12:35.770 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:12:35 smithi002 ceph-mon[71816]: osdmap e64: 10 total, 9 up, 10 in 2022-05-13T23:12:35.770 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:12:35 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2022-05-13T23:12:35.771 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:12:35 smithi002 ceph-mon[71816]: from='osd.9 [v2:172.21.15.2:6826/3004162255,v1:172.21.15.2:6827/3004162255]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi002", "root=default"]}]: dispatch 2022-05-13T23:12:35.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:12:35 smithi115 ceph-mon[78593]: from='osd.9 [v2:172.21.15.2:6826/3004162255,v1:172.21.15.2:6827/3004162255]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2022-05-13T23:12:35.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:12:35 smithi115 ceph-mon[78593]: osdmap e64: 10 total, 9 up, 10 in 2022-05-13T23:12:35.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:12:35 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2022-05-13T23:12:35.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:12:35 smithi115 ceph-mon[78593]: from='osd.9 [v2:172.21.15.2:6826/3004162255,v1:172.21.15.2:6827/3004162255]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi002", "root=default"]}]: dispatch 2022-05-13T23:12:36.534 INFO:teuthology.orchestra.run.smithi047.stderr:--> Zapping: /dev/vg_nvme/lv_1 2022-05-13T23:12:36.535 INFO:teuthology.orchestra.run.smithi047.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2022-05-13T23:12:36.535 INFO:teuthology.orchestra.run.smithi047.stderr: stderr: 10+0 records in 2022-05-13T23:12:36.535 INFO:teuthology.orchestra.run.smithi047.stderr:10+0 records out 2022-05-13T23:12:36.536 INFO:teuthology.orchestra.run.smithi047.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.013271 s, 790 MB/s 2022-05-13T23:12:36.536 INFO:teuthology.orchestra.run.smithi047.stderr:--> Zapping successful for: 2022-05-13T23:12:36.572 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:12:36 smithi002 ceph-mon[71816]: pgmap v219: 1 pgs: 1 active+clean; 577 KiB data, 54 MiB used, 805 GiB / 805 GiB avail 2022-05-13T23:12:36.573 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:12:36 smithi002 ceph-mon[71816]: from='osd.9 [v2:172.21.15.2:6826/3004162255,v1:172.21.15.2:6827/3004162255]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi002", "root=default"]}]': finished 2022-05-13T23:12:36.573 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:12:36 smithi002 ceph-mon[71816]: osdmap e65: 10 total, 9 up, 10 in 2022-05-13T23:12:36.573 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:12:36 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2022-05-13T23:12:36.573 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:12:36 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2022-05-13T23:12:36.763 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:12:36 smithi047 ceph-mon[77147]: pgmap v219: 1 pgs: 1 active+clean; 577 KiB data, 54 MiB used, 805 GiB / 805 GiB avail 2022-05-13T23:12:36.763 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:12:36 smithi047 ceph-mon[77147]: from='osd.9 [v2:172.21.15.2:6826/3004162255,v1:172.21.15.2:6827/3004162255]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi002", "root=default"]}]': finished 2022-05-13T23:12:36.763 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:12:36 smithi047 ceph-mon[77147]: osdmap e65: 10 total, 9 up, 10 in 2022-05-13T23:12:36.764 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:12:36 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2022-05-13T23:12:36.764 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:12:36 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2022-05-13T23:12:36.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:12:36 smithi115 ceph-mon[78593]: pgmap v219: 1 pgs: 1 active+clean; 577 KiB data, 54 MiB used, 805 GiB / 805 GiB avail 2022-05-13T23:12:36.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:12:36 smithi115 ceph-mon[78593]: from='osd.9 [v2:172.21.15.2:6826/3004162255,v1:172.21.15.2:6827/3004162255]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi002", "root=default"]}]': finished 2022-05-13T23:12:36.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:12:36 smithi115 ceph-mon[78593]: osdmap e65: 10 total, 9 up, 10 in 2022-05-13T23:12:36.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:12:36 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2022-05-13T23:12:36.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:12:36 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2022-05-13T23:12:36.997 DEBUG:teuthology.orchestra.run.smithi047:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:23b0658391ce94ca40799469eec00c28b298816c shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 16cf5f88-d311-11ec-be40-001a4aab830c -- ceph orch daemon add osd smithi047:vg_nvme/lv_1 2022-05-13T23:12:37.454 INFO:teuthology.orchestra.run.smithi047.stderr:Inferring config /var/lib/ceph/16cf5f88-d311-11ec-be40-001a4aab830c/mon.b/config 2022-05-13T23:12:37.721 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:12:37 smithi047 ceph-mon[77147]: purged_snaps scrub starts 2022-05-13T23:12:37.722 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:12:37 smithi047 ceph-mon[77147]: purged_snaps scrub ok 2022-05-13T23:12:37.722 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:12:37 smithi047 ceph-mon[77147]: osd.9 [v2:172.21.15.2:6826/3004162255,v1:172.21.15.2:6827/3004162255] boot 2022-05-13T23:12:37.723 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:12:37 smithi047 ceph-mon[77147]: osdmap e66: 10 total, 10 up, 10 in 2022-05-13T23:12:37.723 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:12:37 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2022-05-13T23:12:37.822 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:12:37 smithi002 ceph-mon[71816]: purged_snaps scrub starts 2022-05-13T23:12:37.823 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:12:37 smithi002 ceph-mon[71816]: purged_snaps scrub ok 2022-05-13T23:12:37.823 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:12:37 smithi002 ceph-mon[71816]: osd.9 [v2:172.21.15.2:6826/3004162255,v1:172.21.15.2:6827/3004162255] boot 2022-05-13T23:12:37.823 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:12:37 smithi002 ceph-mon[71816]: osdmap e66: 10 total, 10 up, 10 in 2022-05-13T23:12:37.824 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:12:37 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2022-05-13T23:12:37.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:12:37 smithi115 ceph-mon[78593]: purged_snaps scrub starts 2022-05-13T23:12:37.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:12:37 smithi115 ceph-mon[78593]: purged_snaps scrub ok 2022-05-13T23:12:37.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:12:37 smithi115 ceph-mon[78593]: osd.9 [v2:172.21.15.2:6826/3004162255,v1:172.21.15.2:6827/3004162255] boot 2022-05-13T23:12:37.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:12:37 smithi115 ceph-mon[78593]: osdmap e66: 10 total, 10 up, 10 in 2022-05-13T23:12:37.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:12:37 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2022-05-13T23:12:38.431 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:12:38 smithi002 ceph-mon[71816]: pgmap v222: 1 pgs: 1 active+clean; 577 KiB data, 60 MiB used, 894 GiB / 894 GiB avail 2022-05-13T23:12:38.431 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:12:38 smithi002 ceph-mon[71816]: osdmap e67: 10 total, 10 up, 10 in 2022-05-13T23:12:38.747 INFO:teuthology.orchestra.run.smithi047.stderr:2022-05-13T23:12:38.742+0000 7f57b77cc700 1 -- 172.21.15.47:0/3405598775 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f57b00fb7a0 msgr2=0x7f57b00fbbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:12:38.747 INFO:teuthology.orchestra.run.smithi047.stderr:2022-05-13T23:12:38.742+0000 7f57b77cc700 1 --2- 172.21.15.47:0/3405598775 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f57b00fb7a0 0x7f57b00fbbf0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f57a0004870 tx=0x7f57a0009cb0 comp rx=0 tx=0).stop 2022-05-13T23:12:38.747 INFO:teuthology.orchestra.run.smithi047.stderr:2022-05-13T23:12:38.742+0000 7f57b77cc700 1 -- 172.21.15.47:0/3405598775 shutdown_connections 2022-05-13T23:12:38.747 INFO:teuthology.orchestra.run.smithi047.stderr:2022-05-13T23:12:38.742+0000 7f57b77cc700 1 --2- 172.21.15.47:0/3405598775 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f57b00fc130 0x7f57b01029f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:12:38.748 INFO:teuthology.orchestra.run.smithi047.stderr:2022-05-13T23:12:38.742+0000 7f57b77cc700 1 --2- 172.21.15.47:0/3405598775 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f57b00fa500 0x7f57b00fa910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:12:38.748 INFO:teuthology.orchestra.run.smithi047.stderr:2022-05-13T23:12:38.742+0000 7f57b77cc700 1 --2- 172.21.15.47:0/3405598775 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f57b00fb7a0 0x7f57b00fbbf0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:12:38.748 INFO:teuthology.orchestra.run.smithi047.stderr:2022-05-13T23:12:38.742+0000 7f57b77cc700 1 -- 172.21.15.47:0/3405598775 >> 172.21.15.47:0/3405598775 conn(0x7f57b00f5cf0 msgr2=0x7f57b00f8120 unknown :-1 s=STATE_NONE l=0).mark_down 2022-05-13T23:12:38.748 INFO:teuthology.orchestra.run.smithi047.stderr:2022-05-13T23:12:38.742+0000 7f57b77cc700 1 -- 172.21.15.47:0/3405598775 shutdown_connections 2022-05-13T23:12:38.749 INFO:teuthology.orchestra.run.smithi047.stderr:2022-05-13T23:12:38.742+0000 7f57b77cc700 1 -- 172.21.15.47:0/3405598775 wait complete. 2022-05-13T23:12:38.749 INFO:teuthology.orchestra.run.smithi047.stderr:2022-05-13T23:12:38.743+0000 7f57b77cc700 1 Processor -- start 2022-05-13T23:12:38.749 INFO:teuthology.orchestra.run.smithi047.stderr:2022-05-13T23:12:38.743+0000 7f57b77cc700 1 -- start start 2022-05-13T23:12:38.749 INFO:teuthology.orchestra.run.smithi047.stderr:2022-05-13T23:12:38.743+0000 7f57b77cc700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f57b00fa500 0x7f57b010c330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:12:38.749 INFO:teuthology.orchestra.run.smithi047.stderr:2022-05-13T23:12:38.743+0000 7f57b77cc700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f57b00fb7a0 0x7f57b010c870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:12:38.750 INFO:teuthology.orchestra.run.smithi047.stderr:2022-05-13T23:12:38.743+0000 7f57b77cc700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f57b00fc130 0x7f57b0113900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:12:38.750 INFO:teuthology.orchestra.run.smithi047.stderr:2022-05-13T23:12:38.743+0000 7f57b77cc700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57b0113ed0 con 0x7f57b00fa500 2022-05-13T23:12:38.750 INFO:teuthology.orchestra.run.smithi047.stderr:2022-05-13T23:12:38.743+0000 7f57b77cc700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57b0114040 con 0x7f57b00fc130 2022-05-13T23:12:38.750 INFO:teuthology.orchestra.run.smithi047.stderr:2022-05-13T23:12:38.743+0000 7f57b77cc700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57b0114200 con 0x7f57b00fb7a0 2022-05-13T23:12:38.751 INFO:teuthology.orchestra.run.smithi047.stderr:2022-05-13T23:12:38.743+0000 7f57b5d69700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f57b00fc130 0x7f57b0113900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:12:38.751 INFO:teuthology.orchestra.run.smithi047.stderr:2022-05-13T23:12:38.743+0000 7f57b5d69700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f57b00fc130 0x7f57b0113900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:42350/0 (socket says 172.21.15.47:42350) 2022-05-13T23:12:38.751 INFO:teuthology.orchestra.run.smithi047.stderr:2022-05-13T23:12:38.743+0000 7f57b5d69700 1 -- 172.21.15.47:0/2972266636 learned_addr learned my addr 172.21.15.47:0/2972266636 (peer_addr_for_me v2:172.21.15.47:0/0) 2022-05-13T23:12:38.751 INFO:teuthology.orchestra.run.smithi047.stderr:2022-05-13T23:12:38.743+0000 7f57b4d67700 1 --2- 172.21.15.47:0/2972266636 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f57b00fb7a0 0x7f57b010c870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:12:38.751 INFO:teuthology.orchestra.run.smithi047.stderr:2022-05-13T23:12:38.743+0000 7f57b5568700 1 --2- 172.21.15.47:0/2972266636 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f57b00fa500 0x7f57b010c330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:12:38.752 INFO:teuthology.orchestra.run.smithi047.stderr:2022-05-13T23:12:38.743+0000 7f57b5d69700 1 -- 172.21.15.47:0/2972266636 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f57b00fb7a0 msgr2=0x7f57b010c870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:12:38.752 INFO:teuthology.orchestra.run.smithi047.stderr:2022-05-13T23:12:38.743+0000 7f57b5d69700 1 --2- 172.21.15.47:0/2972266636 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f57b00fb7a0 0x7f57b010c870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:12:38.752 INFO:teuthology.orchestra.run.smithi047.stderr:2022-05-13T23:12:38.743+0000 7f57b5d69700 1 -- 172.21.15.47:0/2972266636 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f57b00fa500 msgr2=0x7f57b010c330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:12:38.752 INFO:teuthology.orchestra.run.smithi047.stderr:2022-05-13T23:12:38.743+0000 7f57b5d69700 1 --2- 172.21.15.47:0/2972266636 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f57b00fa500 0x7f57b010c330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:12:38.753 INFO:teuthology.orchestra.run.smithi047.stderr:2022-05-13T23:12:38.743+0000 7f57b5d69700 1 -- 172.21.15.47:0/2972266636 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57a0005040 con 0x7f57b00fc130 2022-05-13T23:12:38.753 INFO:teuthology.orchestra.run.smithi047.stderr:2022-05-13T23:12:38.743+0000 7f57b5568700 1 --2- 172.21.15.47:0/2972266636 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f57b00fa500 0x7f57b010c330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2022-05-13T23:12:38.753 INFO:teuthology.orchestra.run.smithi047.stderr:2022-05-13T23:12:38.743+0000 7f57b5d69700 1 --2- 172.21.15.47:0/2972266636 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f57b00fc130 0x7f57b0113900 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f57a800bc30 tx=0x7f57a8009840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-05-13T23:12:38.753 INFO:teuthology.orchestra.run.smithi047.stderr:2022-05-13T23:12:38.744+0000 7f57a5ffb700 1 -- 172.21.15.47:0/2972266636 <== mon.1 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57a80092e0 con 0x7f57b00fc130 2022-05-13T23:12:38.753 INFO:teuthology.orchestra.run.smithi047.stderr:2022-05-13T23:12:38.744+0000 7f57a5ffb700 1 -- 172.21.15.47:0/2972266636 <== mon.1 v2:172.21.15.47:3300/0 2 ==== config(26 keys) v1 ==== 1087+0+0 (secure 0 0 0) 0x7f57a800cba0 con 0x7f57b00fc130 2022-05-13T23:12:38.754 INFO:teuthology.orchestra.run.smithi047.stderr:2022-05-13T23:12:38.744+0000 7f57b77cc700 1 -- 172.21.15.47:0/2972266636 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f57b01144e0 con 0x7f57b00fc130 2022-05-13T23:12:38.754 INFO:teuthology.orchestra.run.smithi047.stderr:2022-05-13T23:12:38.744+0000 7f57a5ffb700 1 -- 172.21.15.47:0/2972266636 <== mon.1 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57a8015cb0 con 0x7f57b00fc130 2022-05-13T23:12:38.754 INFO:teuthology.orchestra.run.smithi047.stderr:2022-05-13T23:12:38.744+0000 7f57b77cc700 1 -- 172.21.15.47:0/2972266636 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f57b0114a00 con 0x7f57b00fc130 2022-05-13T23:12:38.754 INFO:teuthology.orchestra.run.smithi047.stderr:2022-05-13T23:12:38.745+0000 7f57b77cc700 1 -- 172.21.15.47:0/2972266636 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f57b0105d50 con 0x7f57b00fc130 2022-05-13T23:12:38.755 INFO:teuthology.orchestra.run.smithi047.stderr:2022-05-13T23:12:38.746+0000 7f57a5ffb700 1 -- 172.21.15.47:0/2972266636 <== mon.1 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 16) v1 ==== 125136+0+0 (secure 0 0 0) 0x7f57a8015470 con 0x7f57b00fc130 2022-05-13T23:12:38.756 INFO:teuthology.orchestra.run.smithi047.stderr:2022-05-13T23:12:38.746+0000 7f57a5ffb700 1 --2- 172.21.15.47:0/2972266636 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f579c096ce0 0x7f579c099190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:12:38.756 INFO:teuthology.orchestra.run.smithi047.stderr:2022-05-13T23:12:38.746+0000 7f57a5ffb700 1 -- 172.21.15.47:0/2972266636 <== mon.1 v2:172.21.15.47:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 6257+0+0 (secure 0 0 0) 0x7f57a80bec20 con 0x7f57b00fc130 2022-05-13T23:12:38.756 INFO:teuthology.orchestra.run.smithi047.stderr:2022-05-13T23:12:38.746+0000 7f57b5568700 1 --2- 172.21.15.47:0/2972266636 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f579c096ce0 0x7f579c099190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:12:38.756 INFO:teuthology.orchestra.run.smithi047.stderr:2022-05-13T23:12:38.747+0000 7f57b5568700 1 --2- 172.21.15.47:0/2972266636 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f579c096ce0 0x7f579c099190 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f57b00fb600 tx=0x7f57ac009d80 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-05-13T23:12:38.757 INFO:teuthology.orchestra.run.smithi047.stderr:2022-05-13T23:12:38.750+0000 7f57a5ffb700 1 -- 172.21.15.47:0/2972266636 <== mon.1 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+165215 (secure 0 0 0) 0x7f57a808c700 con 0x7f57b00fc130 2022-05-13T23:12:38.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:12:38 smithi047 ceph-mon[77147]: pgmap v222: 1 pgs: 1 active+clean; 577 KiB data, 60 MiB used, 894 GiB / 894 GiB avail 2022-05-13T23:12:38.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:12:38 smithi047 ceph-mon[77147]: osdmap e67: 10 total, 10 up, 10 in 2022-05-13T23:12:38.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:12:38 smithi115 ceph-mon[78593]: pgmap v222: 1 pgs: 1 active+clean; 577 KiB data, 60 MiB used, 894 GiB / 894 GiB avail 2022-05-13T23:12:38.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:12:38 smithi115 ceph-mon[78593]: osdmap e67: 10 total, 10 up, 10 in 2022-05-13T23:12:38.940 INFO:teuthology.orchestra.run.smithi047.stderr:2022-05-13T23:12:38.938+0000 7f57b77cc700 1 -- 172.21.15.47:0/2972266636 --> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi047:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) v1 -- 0x7f57b0064600 con 0x7f579c096ce0 2022-05-13T23:12:39.719 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:12:39 smithi047 ceph-mon[77147]: Detected new or changed devices on smithi002 2022-05-13T23:12:39.719 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:12:39 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:12:39.720 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:12:39 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2022-05-13T23:12:39.720 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:12:39 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2022-05-13T23:12:39.720 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:12:39 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2022-05-13T23:12:39.720 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:12:39 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2022-05-13T23:12:39.721 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:12:39 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:12:39.721 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:12:39 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:12:39.721 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:12:39 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2022-05-13T23:12:39.722 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:12:39 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2022-05-13T23:12:39.722 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:12:39 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-13T23:12:39.822 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:12:39 smithi002 ceph-mon[71816]: Detected new or changed devices on smithi002 2022-05-13T23:12:39.822 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:12:39 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:12:39.823 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:12:39 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2022-05-13T23:12:39.823 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:12:39 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2022-05-13T23:12:39.823 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:12:39 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2022-05-13T23:12:39.823 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:12:39 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2022-05-13T23:12:39.824 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:12:39 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:12:39.824 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:12:39 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:12:39.824 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:12:39 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2022-05-13T23:12:39.824 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:12:39 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2022-05-13T23:12:39.824 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:12:39 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-13T23:12:39.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:12:39 smithi115 ceph-mon[78593]: Detected new or changed devices on smithi002 2022-05-13T23:12:39.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:12:39 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:12:39.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:12:39 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2022-05-13T23:12:39.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:12:39 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2022-05-13T23:12:39.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:12:39 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2022-05-13T23:12:39.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:12:39 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2022-05-13T23:12:39.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:12:39 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:12:39.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:12:39 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:12:39.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:12:39 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2022-05-13T23:12:39.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:12:39 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2022-05-13T23:12:39.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:12:39 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-13T23:12:40.719 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:12:40 smithi047 ceph-mon[77147]: Adjusting osd_memory_target on smithi002 to 4319M 2022-05-13T23:12:40.719 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:12:40 smithi047 ceph-mon[77147]: from='client.24367 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi047:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2022-05-13T23:12:40.719 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:12:40 smithi047 ceph-mon[77147]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 60 MiB used, 894 GiB / 894 GiB avail 2022-05-13T23:12:40.822 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:12:40 smithi002 ceph-mon[71816]: Adjusting osd_memory_target on smithi002 to 4319M 2022-05-13T23:12:40.822 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:12:40 smithi002 ceph-mon[71816]: from='client.24367 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi047:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2022-05-13T23:12:40.823 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:12:40 smithi002 ceph-mon[71816]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 60 MiB used, 894 GiB / 894 GiB avail 2022-05-13T23:12:40.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:12:40 smithi115 ceph-mon[78593]: Adjusting osd_memory_target on smithi002 to 4319M 2022-05-13T23:12:40.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:12:40 smithi115 ceph-mon[78593]: from='client.24367 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi047:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2022-05-13T23:12:40.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:12:40 smithi115 ceph-mon[78593]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 60 MiB used, 894 GiB / 894 GiB avail 2022-05-13T23:12:41.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:12:41 smithi047 ceph-mon[77147]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 60 MiB used, 894 GiB / 894 GiB avail 2022-05-13T23:12:41.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:12:41 smithi047 ceph-mon[77147]: from='client.? 172.21.15.47:0/1013537544' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4f648118-7855-48d6-86d5-44eed5295f0c"}]: dispatch 2022-05-13T23:12:41.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:12:41 smithi047 ceph-mon[77147]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4f648118-7855-48d6-86d5-44eed5295f0c"}]: dispatch 2022-05-13T23:12:41.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:12:41 smithi047 ceph-mon[77147]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "4f648118-7855-48d6-86d5-44eed5295f0c"}]': finished 2022-05-13T23:12:41.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:12:41 smithi047 ceph-mon[77147]: osdmap e68: 11 total, 10 up, 11 in 2022-05-13T23:12:41.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:12:41 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2022-05-13T23:12:41.823 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:12:41 smithi002 ceph-mon[71816]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 60 MiB used, 894 GiB / 894 GiB avail 2022-05-13T23:12:41.823 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:12:41 smithi002 ceph-mon[71816]: from='client.? 172.21.15.47:0/1013537544' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4f648118-7855-48d6-86d5-44eed5295f0c"}]: dispatch 2022-05-13T23:12:41.823 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:12:41 smithi002 ceph-mon[71816]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4f648118-7855-48d6-86d5-44eed5295f0c"}]: dispatch 2022-05-13T23:12:41.824 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:12:41 smithi002 ceph-mon[71816]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "4f648118-7855-48d6-86d5-44eed5295f0c"}]': finished 2022-05-13T23:12:41.824 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:12:41 smithi002 ceph-mon[71816]: osdmap e68: 11 total, 10 up, 11 in 2022-05-13T23:12:41.824 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:12:41 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2022-05-13T23:12:41.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:12:41 smithi115 ceph-mon[78593]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 60 MiB used, 894 GiB / 894 GiB avail 2022-05-13T23:12:41.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:12:41 smithi115 ceph-mon[78593]: from='client.? 172.21.15.47:0/1013537544' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4f648118-7855-48d6-86d5-44eed5295f0c"}]: dispatch 2022-05-13T23:12:41.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:12:41 smithi115 ceph-mon[78593]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4f648118-7855-48d6-86d5-44eed5295f0c"}]: dispatch 2022-05-13T23:12:41.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:12:41 smithi115 ceph-mon[78593]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "4f648118-7855-48d6-86d5-44eed5295f0c"}]': finished 2022-05-13T23:12:41.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:12:41 smithi115 ceph-mon[78593]: osdmap e68: 11 total, 10 up, 11 in 2022-05-13T23:12:41.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:12:41 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2022-05-13T23:12:42.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:12:42 smithi047 ceph-mon[77147]: from='client.? 172.21.15.47:0/1465857929' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2022-05-13T23:12:42.822 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:12:42 smithi002 ceph-mon[71816]: from='client.? 172.21.15.47:0/1465857929' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2022-05-13T23:12:42.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:12:42 smithi115 ceph-mon[78593]: from='client.? 172.21.15.47:0/1465857929' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2022-05-13T23:12:43.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:12:43 smithi047 ceph-mon[77147]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 60 MiB used, 894 GiB / 894 GiB avail 2022-05-13T23:12:43.822 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:12:43 smithi002 ceph-mon[71816]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 60 MiB used, 894 GiB / 894 GiB avail 2022-05-13T23:12:43.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:12:43 smithi115 ceph-mon[78593]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 60 MiB used, 894 GiB / 894 GiB avail 2022-05-13T23:12:46.528 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:12:46 smithi047 ceph-mon[77147]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 60 MiB used, 894 GiB / 894 GiB avail 2022-05-13T23:12:46.572 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:12:46 smithi002 ceph-mon[71816]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 60 MiB used, 894 GiB / 894 GiB avail 2022-05-13T23:12:46.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:12:46 smithi115 ceph-mon[78593]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 60 MiB used, 894 GiB / 894 GiB avail 2022-05-13T23:12:48.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:12:48 smithi047 ceph-mon[77147]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 61 MiB used, 894 GiB / 894 GiB avail 2022-05-13T23:12:48.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:12:48 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2022-05-13T23:12:48.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:12:48 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-13T23:12:48.572 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:12:48 smithi002 ceph-mon[71816]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 61 MiB used, 894 GiB / 894 GiB avail 2022-05-13T23:12:48.573 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:12:48 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2022-05-13T23:12:48.573 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:12:48 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-13T23:12:48.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:12:48 smithi115 ceph-mon[78593]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 61 MiB used, 894 GiB / 894 GiB avail 2022-05-13T23:12:48.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:12:48 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2022-05-13T23:12:48.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:12:48 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-13T23:12:49.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:12:49 smithi047 ceph-mon[77147]: Deploying daemon osd.10 on smithi047 2022-05-13T23:12:49.572 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:12:49 smithi002 ceph-mon[71816]: Deploying daemon osd.10 on smithi047 2022-05-13T23:12:49.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:12:49 smithi115 ceph-mon[78593]: Deploying daemon osd.10 on smithi047 2022-05-13T23:12:50.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:12:50 smithi047 ceph-mon[77147]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 61 MiB used, 894 GiB / 894 GiB avail 2022-05-13T23:12:50.572 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:12:50 smithi002 ceph-mon[71816]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 61 MiB used, 894 GiB / 894 GiB avail 2022-05-13T23:12:50.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:12:50 smithi115 ceph-mon[78593]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 61 MiB used, 894 GiB / 894 GiB avail 2022-05-13T23:12:52.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:12:52 smithi047 ceph-mon[77147]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 61 MiB used, 894 GiB / 894 GiB avail 2022-05-13T23:12:52.572 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:12:52 smithi002 ceph-mon[71816]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 61 MiB used, 894 GiB / 894 GiB avail 2022-05-13T23:12:52.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:12:52 smithi115 ceph-mon[78593]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 61 MiB used, 894 GiB / 894 GiB avail 2022-05-13T23:12:53.897 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:12:53 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:12:53.897 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:12:53 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-13T23:12:53.897 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:12:53 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-13T23:12:53.898 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:12:53 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-13T23:12:53.898 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:12:53 smithi047 ceph-mon[77147]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 61 MiB used, 894 GiB / 894 GiB avail 2022-05-13T23:12:54.072 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:12:53 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:12:54.073 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:12:53 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-13T23:12:54.073 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:12:53 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-13T23:12:54.073 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:12:53 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-13T23:12:54.073 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:12:53 smithi002 ceph-mon[71816]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 61 MiB used, 894 GiB / 894 GiB avail 2022-05-13T23:12:54.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:12:53 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:12:54.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:12:53 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-13T23:12:54.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:12:53 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-13T23:12:54.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:12:53 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-13T23:12:54.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:12:53 smithi115 ceph-mon[78593]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 61 MiB used, 894 GiB / 894 GiB avail 2022-05-13T23:12:55.565 INFO:teuthology.orchestra.run.smithi047.stderr:2022-05-13T23:12:55.562+0000 7f57a5ffb700 1 -- 172.21.15.47:0/2972266636 <== mgr.14148 v2:172.21.15.2:6800/2788339627 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+38 (secure 0 0 0) 0x7f57b0064600 con 0x7f579c096ce0 2022-05-13T23:12:55.566 INFO:teuthology.orchestra.run.smithi047.stdout:Created osd(s) 10 on host 'smithi047' 2022-05-13T23:12:55.568 INFO:teuthology.orchestra.run.smithi047.stderr:2022-05-13T23:12:55.565+0000 7f57b77cc700 1 -- 172.21.15.47:0/2972266636 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f579c096ce0 msgr2=0x7f579c099190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:12:55.568 INFO:teuthology.orchestra.run.smithi047.stderr:2022-05-13T23:12:55.565+0000 7f57b77cc700 1 --2- 172.21.15.47:0/2972266636 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f579c096ce0 0x7f579c099190 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f57b00fb600 tx=0x7f57ac009d80 comp rx=0 tx=0).stop 2022-05-13T23:12:55.568 INFO:teuthology.orchestra.run.smithi047.stderr:2022-05-13T23:12:55.565+0000 7f57b77cc700 1 -- 172.21.15.47:0/2972266636 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f57b00fc130 msgr2=0x7f57b0113900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:12:55.568 INFO:teuthology.orchestra.run.smithi047.stderr:2022-05-13T23:12:55.565+0000 7f57b77cc700 1 --2- 172.21.15.47:0/2972266636 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f57b00fc130 0x7f57b0113900 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f57a800bc30 tx=0x7f57a8009840 comp rx=0 tx=0).stop 2022-05-13T23:12:55.569 INFO:teuthology.orchestra.run.smithi047.stderr:2022-05-13T23:12:55.565+0000 7f57b77cc700 1 -- 172.21.15.47:0/2972266636 shutdown_connections 2022-05-13T23:12:55.569 INFO:teuthology.orchestra.run.smithi047.stderr:2022-05-13T23:12:55.565+0000 7f57b77cc700 1 --2- 172.21.15.47:0/2972266636 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f57b00fc130 0x7f57b0113900 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:12:55.569 INFO:teuthology.orchestra.run.smithi047.stderr:2022-05-13T23:12:55.565+0000 7f57b77cc700 1 --2- 172.21.15.47:0/2972266636 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f579c096ce0 0x7f579c099190 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:12:55.569 INFO:teuthology.orchestra.run.smithi047.stderr:2022-05-13T23:12:55.565+0000 7f57b77cc700 1 --2- 172.21.15.47:0/2972266636 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f57b00fa500 0x7f57b010c330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:12:55.570 INFO:teuthology.orchestra.run.smithi047.stderr:2022-05-13T23:12:55.565+0000 7f57b77cc700 1 --2- 172.21.15.47:0/2972266636 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f57b00fb7a0 0x7f57b010c870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:12:55.570 INFO:teuthology.orchestra.run.smithi047.stderr:2022-05-13T23:12:55.566+0000 7f57b77cc700 1 -- 172.21.15.47:0/2972266636 >> 172.21.15.47:0/2972266636 conn(0x7f57b00f5cf0 msgr2=0x7f57b0063ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-05-13T23:12:55.570 INFO:teuthology.orchestra.run.smithi047.stderr:2022-05-13T23:12:55.566+0000 7f57b77cc700 1 -- 172.21.15.47:0/2972266636 shutdown_connections 2022-05-13T23:12:55.570 INFO:teuthology.orchestra.run.smithi047.stderr:2022-05-13T23:12:55.566+0000 7f57b77cc700 1 -- 172.21.15.47:0/2972266636 wait complete. 2022-05-13T23:12:56.002 DEBUG:teuthology.orchestra.run.smithi047:osd.10> sudo journalctl -f -n 0 -u ceph-16cf5f88-d311-11ec-be40-001a4aab830c@osd.10.service 2022-05-13T23:12:56.006 INFO:tasks.cephadm:Deploying osd.11 on smithi115 with /dev/vg_nvme/lv_1... 2022-05-13T23:12:56.007 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:23b0658391ce94ca40799469eec00c28b298816c shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 16cf5f88-d311-11ec-be40-001a4aab830c -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2022-05-13T23:12:56.036 INFO:journalctl@ceph.osd.10.smithi047.stdout:-- Logs begin at Fri 2022-05-13 22:44:47 UTC. -- 2022-05-13T23:12:56.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:12:56 smithi115 ceph-mon[78593]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 61 MiB used, 894 GiB / 894 GiB avail 2022-05-13T23:12:56.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:12:56 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:12:56.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:12:56 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:12:56.458 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/16cf5f88-d311-11ec-be40-001a4aab830c/mon.c/config 2022-05-13T23:12:56.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:12:56 smithi047 ceph-mon[77147]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 61 MiB used, 894 GiB / 894 GiB avail 2022-05-13T23:12:56.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:12:56 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:12:56.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:12:56 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:12:56.572 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:12:56 smithi002 ceph-mon[71816]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 61 MiB used, 894 GiB / 894 GiB avail 2022-05-13T23:12:56.573 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:12:56 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:12:56.573 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:12:56 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:12:57.042 INFO:journalctl@ceph.osd.10.smithi047.stdout:May 13 23:12:56 smithi047 conmon[100262]: 2022-05-13T23:12:56.715+0000 7f5b5abf83c0 -1 osd.10 0 log_to_monitors true 2022-05-13T23:12:57.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:12:57 smithi115 ceph-mon[78593]: from='osd.10 [v2:172.21.15.47:6824/3443413998,v1:172.21.15.47:6825/3443413998]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2022-05-13T23:12:57.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:12:57 smithi115 ceph-mon[78593]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2022-05-13T23:12:57.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:12:57 smithi047 ceph-mon[77147]: from='osd.10 [v2:172.21.15.47:6824/3443413998,v1:172.21.15.47:6825/3443413998]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2022-05-13T23:12:57.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:12:57 smithi047 ceph-mon[77147]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2022-05-13T23:12:57.572 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:12:57 smithi002 ceph-mon[71816]: from='osd.10 [v2:172.21.15.47:6824/3443413998,v1:172.21.15.47:6825/3443413998]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2022-05-13T23:12:57.573 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:12:57 smithi002 ceph-mon[71816]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2022-05-13T23:12:58.060 INFO:teuthology.orchestra.run.smithi115.stderr:--> Zapping: /dev/vg_nvme/lv_1 2022-05-13T23:12:58.061 INFO:teuthology.orchestra.run.smithi115.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2022-05-13T23:12:58.061 INFO:teuthology.orchestra.run.smithi115.stderr: stderr: 10+0 records in 2022-05-13T23:12:58.061 INFO:teuthology.orchestra.run.smithi115.stderr:10+0 records out 2022-05-13T23:12:58.061 INFO:teuthology.orchestra.run.smithi115.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.013166 s, 796 MB/s 2022-05-13T23:12:58.062 INFO:teuthology.orchestra.run.smithi115.stderr:--> Zapping successful for: 2022-05-13T23:12:58.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:12:58 smithi115 ceph-mon[78593]: pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 61 MiB used, 894 GiB / 894 GiB avail 2022-05-13T23:12:58.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:12:58 smithi115 ceph-mon[78593]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2022-05-13T23:12:58.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:12:58 smithi115 ceph-mon[78593]: osdmap e69: 11 total, 10 up, 11 in 2022-05-13T23:12:58.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:12:58 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2022-05-13T23:12:58.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:12:58 smithi115 ceph-mon[78593]: from='osd.10 [v2:172.21.15.47:6824/3443413998,v1:172.21.15.47:6825/3443413998]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi047", "root=default"]}]: dispatch 2022-05-13T23:12:58.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:12:58 smithi115 ceph-mon[78593]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi047", "root=default"]}]: dispatch 2022-05-13T23:12:58.472 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:23b0658391ce94ca40799469eec00c28b298816c shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 16cf5f88-d311-11ec-be40-001a4aab830c -- ceph orch daemon add osd smithi115:vg_nvme/lv_1 2022-05-13T23:12:58.541 INFO:journalctl@ceph.osd.10.smithi047.stdout:May 13 23:12:58 smithi047 conmon[100262]: 2022-05-13T23:12:58.272+0000 7f5b515fc700 -1 osd.10 0 waiting for initial osdmap 2022-05-13T23:12:58.542 INFO:journalctl@ceph.osd.10.smithi047.stdout:May 13 23:12:58 smithi047 conmon[100262]: 2022-05-13T23:12:58.293+0000 7f5b4bf92700 -1 osd.10 70 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2022-05-13T23:12:58.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:12:58 smithi047 ceph-mon[77147]: pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 61 MiB used, 894 GiB / 894 GiB avail 2022-05-13T23:12:58.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:12:58 smithi047 ceph-mon[77147]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2022-05-13T23:12:58.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:12:58 smithi047 ceph-mon[77147]: osdmap e69: 11 total, 10 up, 11 in 2022-05-13T23:12:58.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:12:58 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2022-05-13T23:12:58.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:12:58 smithi047 ceph-mon[77147]: from='osd.10 [v2:172.21.15.47:6824/3443413998,v1:172.21.15.47:6825/3443413998]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi047", "root=default"]}]: dispatch 2022-05-13T23:12:58.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:12:58 smithi047 ceph-mon[77147]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi047", "root=default"]}]: dispatch 2022-05-13T23:12:58.572 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:12:58 smithi002 ceph-mon[71816]: pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 61 MiB used, 894 GiB / 894 GiB avail 2022-05-13T23:12:58.574 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:12:58 smithi002 ceph-mon[71816]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2022-05-13T23:12:58.574 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:12:58 smithi002 ceph-mon[71816]: osdmap e69: 11 total, 10 up, 11 in 2022-05-13T23:12:58.575 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:12:58 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2022-05-13T23:12:58.575 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:12:58 smithi002 ceph-mon[71816]: from='osd.10 [v2:172.21.15.47:6824/3443413998,v1:172.21.15.47:6825/3443413998]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi047", "root=default"]}]: dispatch 2022-05-13T23:12:58.575 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:12:58 smithi002 ceph-mon[71816]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi047", "root=default"]}]: dispatch 2022-05-13T23:12:58.955 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/16cf5f88-d311-11ec-be40-001a4aab830c/mon.c/config 2022-05-13T23:12:59.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:12:59 smithi047 ceph-mon[77147]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi047", "root=default"]}]': finished 2022-05-13T23:12:59.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:12:59 smithi047 ceph-mon[77147]: osdmap e70: 11 total, 10 up, 11 in 2022-05-13T23:12:59.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:12:59 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2022-05-13T23:12:59.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:12:59 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2022-05-13T23:12:59.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:12:59 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:12:59.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:12:59 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-05-13T23:12:59.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:12:59 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-05-13T23:12:59.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:12:59 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-05-13T23:12:59.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:12:59 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-05-13T23:12:59.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:12:59 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:12:59.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:12:59 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:12:59.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:12:59 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-13T23:12:59.545 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:12:59 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-13T23:12:59.545 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:12:59 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-13T23:12:59.545 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:12:59 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:12:59.545 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:12:59 smithi047 ceph-mon[77147]: osd.10 [v2:172.21.15.47:6824/3443413998,v1:172.21.15.47:6825/3443413998] boot 2022-05-13T23:12:59.546 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:12:59 smithi047 ceph-mon[77147]: osdmap e71: 11 total, 11 up, 11 in 2022-05-13T23:12:59.546 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:12:59 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2022-05-13T23:12:59.572 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:12:59 smithi002 ceph-mon[71816]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi047", "root=default"]}]': finished 2022-05-13T23:12:59.573 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:12:59 smithi002 ceph-mon[71816]: osdmap e70: 11 total, 10 up, 11 in 2022-05-13T23:12:59.573 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:12:59 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2022-05-13T23:12:59.573 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:12:59 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2022-05-13T23:12:59.574 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:12:59 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:12:59.574 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:12:59 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-05-13T23:12:59.574 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:12:59 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-05-13T23:12:59.574 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:12:59 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-05-13T23:12:59.575 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:12:59 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-05-13T23:12:59.575 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:12:59 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:12:59.575 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:12:59 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:12:59.576 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:12:59 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-13T23:12:59.576 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:12:59 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-13T23:12:59.576 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:12:59 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-13T23:12:59.576 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:12:59 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:12:59.577 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:12:59 smithi002 ceph-mon[71816]: osd.10 [v2:172.21.15.47:6824/3443413998,v1:172.21.15.47:6825/3443413998] boot 2022-05-13T23:12:59.577 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:12:59 smithi002 ceph-mon[71816]: osdmap e71: 11 total, 11 up, 11 in 2022-05-13T23:12:59.577 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:12:59 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2022-05-13T23:12:59.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:12:59 smithi115 ceph-mon[78593]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi047", "root=default"]}]': finished 2022-05-13T23:12:59.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:12:59 smithi115 ceph-mon[78593]: osdmap e70: 11 total, 10 up, 11 in 2022-05-13T23:12:59.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:12:59 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2022-05-13T23:12:59.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:12:59 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2022-05-13T23:12:59.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:12:59 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:12:59.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:12:59 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-05-13T23:12:59.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:12:59 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-05-13T23:12:59.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:12:59 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-05-13T23:12:59.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:12:59 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-05-13T23:12:59.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:12:59 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:12:59.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:12:59 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:12:59.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:12:59 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-13T23:12:59.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:12:59 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-13T23:12:59.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:12:59 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-13T23:12:59.612 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:12:59 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:12:59.612 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:12:59 smithi115 ceph-mon[78593]: osd.10 [v2:172.21.15.47:6824/3443413998,v1:172.21.15.47:6825/3443413998] boot 2022-05-13T23:12:59.612 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:12:59 smithi115 ceph-mon[78593]: osdmap e71: 11 total, 11 up, 11 in 2022-05-13T23:12:59.612 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:12:59 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2022-05-13T23:13:00.235 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:13:00.231+0000 7f089bb60700 1 -- 172.21.15.115:0/1855859270 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f08940fb790 msgr2=0x7f08940fbbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:13:00.235 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:13:00.231+0000 7f089bb60700 1 --2- 172.21.15.115:0/1855859270 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f08940fb790 0x7f08940fbbe0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f0884004870 tx=0x7f0884009cb0 comp rx=0 tx=0).stop 2022-05-13T23:13:00.236 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:13:00.231+0000 7f089bb60700 1 -- 172.21.15.115:0/1855859270 shutdown_connections 2022-05-13T23:13:00.236 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:13:00.231+0000 7f089bb60700 1 --2- 172.21.15.115:0/1855859270 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f08940fc120 0x7f08941029e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:13:00.236 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:13:00.231+0000 7f089bb60700 1 --2- 172.21.15.115:0/1855859270 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f08940fa4f0 0x7f08940fa900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:13:00.236 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:13:00.231+0000 7f089bb60700 1 --2- 172.21.15.115:0/1855859270 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f08940fb790 0x7f08940fbbe0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:13:00.237 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:13:00.231+0000 7f089bb60700 1 -- 172.21.15.115:0/1855859270 >> 172.21.15.115:0/1855859270 conn(0x7f08940f5ce0 msgr2=0x7f08940f8110 unknown :-1 s=STATE_NONE l=0).mark_down 2022-05-13T23:13:00.237 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:13:00.231+0000 7f089bb60700 1 -- 172.21.15.115:0/1855859270 shutdown_connections 2022-05-13T23:13:00.237 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:13:00.231+0000 7f089bb60700 1 -- 172.21.15.115:0/1855859270 wait complete. 2022-05-13T23:13:00.237 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:13:00.231+0000 7f089bb60700 1 Processor -- start 2022-05-13T23:13:00.238 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:13:00.231+0000 7f089bb60700 1 -- start start 2022-05-13T23:13:00.238 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:13:00.231+0000 7f089bb60700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f08940fa4f0 0x7f0894065480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:13:00.238 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:13:00.231+0000 7f089bb60700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f08940fb790 0x7f08941111c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:13:00.238 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:13:00.231+0000 7f089bb60700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f08940fc120 0x7f0894113840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:13:00.238 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:13:00.231+0000 7f089bb60700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0894114030 con 0x7f08940fc120 2022-05-13T23:13:00.239 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:13:00.231+0000 7f089bb60700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08941141a0 con 0x7f08940fb790 2022-05-13T23:13:00.239 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:13:00.231+0000 7f089bb60700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0894114360 con 0x7f08940fa4f0 2022-05-13T23:13:00.239 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:13:00.231+0000 7f08998fc700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f08940fa4f0 0x7f0894065480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:13:00.239 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:13:00.232+0000 7f08998fc700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f08940fa4f0 0x7f0894065480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:49252/0 (socket says 172.21.15.115:49252) 2022-05-13T23:13:00.240 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:13:00.232+0000 7f08998fc700 1 -- 172.21.15.115:0/2460540634 learned_addr learned my addr 172.21.15.115:0/2460540634 (peer_addr_for_me v2:172.21.15.115:0/0) 2022-05-13T23:13:00.240 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:13:00.232+0000 7f08990fb700 1 --2- 172.21.15.115:0/2460540634 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f08940fb790 0x7f08941111c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:13:00.240 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:13:00.232+0000 7f089a0fd700 1 --2- 172.21.15.115:0/2460540634 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f08940fc120 0x7f0894113840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:13:00.240 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:13:00.232+0000 7f08998fc700 1 -- 172.21.15.115:0/2460540634 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f08940fb790 msgr2=0x7f08941111c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:13:00.240 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:13:00.232+0000 7f08998fc700 1 --2- 172.21.15.115:0/2460540634 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f08940fb790 0x7f08941111c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:13:00.241 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:13:00.232+0000 7f08998fc700 1 -- 172.21.15.115:0/2460540634 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f08940fc120 msgr2=0x7f0894113840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:13:00.241 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:13:00.232+0000 7f08998fc700 1 --2- 172.21.15.115:0/2460540634 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f08940fc120 0x7f0894113840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:13:00.241 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:13:00.232+0000 7f08998fc700 1 -- 172.21.15.115:0/2460540634 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0884005040 con 0x7f08940fa4f0 2022-05-13T23:13:00.241 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:13:00.232+0000 7f08990fb700 1 --2- 172.21.15.115:0/2460540634 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f08940fb790 0x7f08941111c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2022-05-13T23:13:00.241 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:13:00.232+0000 7f08998fc700 1 --2- 172.21.15.115:0/2460540634 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f08940fa4f0 0x7f0894065480 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f089000b500 tx=0x7f08900090f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-05-13T23:13:00.242 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:13:00.232+0000 7f088a7fc700 1 -- 172.21.15.115:0/2460540634 <== mon.2 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f089000f510 con 0x7f08940fa4f0 2022-05-13T23:13:00.242 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:13:00.232+0000 7f088a7fc700 1 -- 172.21.15.115:0/2460540634 <== mon.2 v2:172.21.15.115:3300/0 2 ==== config(26 keys) v1 ==== 1087+0+0 (secure 0 0 0) 0x7f089000fb20 con 0x7f08940fa4f0 2022-05-13T23:13:00.242 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:13:00.232+0000 7f089bb60700 1 -- 172.21.15.115:0/2460540634 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f08941145e0 con 0x7f08940fa4f0 2022-05-13T23:13:00.242 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:13:00.232+0000 7f088a7fc700 1 -- 172.21.15.115:0/2460540634 <== mon.2 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0890011c20 con 0x7f08940fa4f0 2022-05-13T23:13:00.243 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:13:00.232+0000 7f089bb60700 1 -- 172.21.15.115:0/2460540634 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0894114ad0 con 0x7f08940fa4f0 2022-05-13T23:13:00.243 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:13:00.233+0000 7f089bb60700 1 -- 172.21.15.115:0/2460540634 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0894105da0 con 0x7f08940fa4f0 2022-05-13T23:13:00.244 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:13:00.234+0000 7f088a7fc700 1 -- 172.21.15.115:0/2460540634 <== mon.2 v2:172.21.15.115:3300/0 4 ==== mgrmap(e 16) v1 ==== 125136+0+0 (secure 0 0 0) 0x7f089000f070 con 0x7f08940fa4f0 2022-05-13T23:13:00.244 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:13:00.234+0000 7f088a7fc700 1 --2- 172.21.15.115:0/2460540634 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f0880096ce0 0x7f0880099190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:13:00.244 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:13:00.235+0000 7f088a7fc700 1 -- 172.21.15.115:0/2460540634 <== mon.2 v2:172.21.15.115:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 6723+0+0 (secure 0 0 0) 0x7f08900be6f0 con 0x7f08940fa4f0 2022-05-13T23:13:00.245 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:13:00.235+0000 7f08990fb700 1 --2- 172.21.15.115:0/2460540634 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f0880096ce0 0x7f0880099190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:13:00.245 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:13:00.235+0000 7f08990fb700 1 --2- 172.21.15.115:0/2460540634 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f0880096ce0 0x7f0880099190 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f0884009ef0 tx=0x7f0884009cb0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-05-13T23:13:00.245 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:13:00.238+0000 7f088a7fc700 1 -- 172.21.15.115:0/2460540634 <== mon.2 v2:172.21.15.115:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+165215 (secure 0 0 0) 0x7f089008c000 con 0x7f08940fa4f0 2022-05-13T23:13:00.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:13:00 smithi115 ceph-mon[78593]: purged_snaps scrub starts 2022-05-13T23:13:00.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:13:00 smithi115 ceph-mon[78593]: purged_snaps scrub ok 2022-05-13T23:13:00.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:13:00 smithi115 ceph-mon[78593]: Detected new or changed devices on smithi047 2022-05-13T23:13:00.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:13:00 smithi115 ceph-mon[78593]: Adjusting osd_memory_target on smithi047 to 4294M 2022-05-13T23:13:00.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:13:00 smithi115 ceph-mon[78593]: pgmap v237: 1 pgs: 1 active+clean; 577 KiB data, 61 MiB used, 894 GiB / 894 GiB avail 2022-05-13T23:13:00.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:13:00 smithi115 ceph-mon[78593]: osdmap e72: 11 total, 11 up, 11 in 2022-05-13T23:13:00.422 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:13:00.420+0000 7f089bb60700 1 -- 172.21.15.115:0/2460540634 --> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi115:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) v1 -- 0x7f089405c920 con 0x7f0880096ce0 2022-05-13T23:13:00.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:13:00 smithi047 ceph-mon[77147]: purged_snaps scrub starts 2022-05-13T23:13:00.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:13:00 smithi047 ceph-mon[77147]: purged_snaps scrub ok 2022-05-13T23:13:00.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:13:00 smithi047 ceph-mon[77147]: Detected new or changed devices on smithi047 2022-05-13T23:13:00.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:13:00 smithi047 ceph-mon[77147]: Adjusting osd_memory_target on smithi047 to 4294M 2022-05-13T23:13:00.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:13:00 smithi047 ceph-mon[77147]: pgmap v237: 1 pgs: 1 active+clean; 577 KiB data, 61 MiB used, 894 GiB / 894 GiB avail 2022-05-13T23:13:00.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:13:00 smithi047 ceph-mon[77147]: osdmap e72: 11 total, 11 up, 11 in 2022-05-13T23:13:00.572 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:13:00 smithi002 ceph-mon[71816]: purged_snaps scrub starts 2022-05-13T23:13:00.573 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:13:00 smithi002 ceph-mon[71816]: purged_snaps scrub ok 2022-05-13T23:13:00.573 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:13:00 smithi002 ceph-mon[71816]: Detected new or changed devices on smithi047 2022-05-13T23:13:00.573 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:13:00 smithi002 ceph-mon[71816]: Adjusting osd_memory_target on smithi047 to 4294M 2022-05-13T23:13:00.573 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:13:00 smithi002 ceph-mon[71816]: pgmap v237: 1 pgs: 1 active+clean; 577 KiB data, 61 MiB used, 894 GiB / 894 GiB avail 2022-05-13T23:13:00.574 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:13:00 smithi002 ceph-mon[71816]: osdmap e72: 11 total, 11 up, 11 in 2022-05-13T23:13:01.510 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:13:01 smithi115 ceph-mon[78593]: from='client.24401 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi115:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2022-05-13T23:13:01.510 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:13:01 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2022-05-13T23:13:01.511 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:13:01 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2022-05-13T23:13:01.511 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:13:01 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-13T23:13:01.511 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:13:01 smithi115 ceph-mon[78593]: osdmap e73: 11 total, 11 up, 11 in 2022-05-13T23:13:01.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:13:01 smithi047 ceph-mon[77147]: from='client.24401 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi115:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2022-05-13T23:13:01.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:13:01 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2022-05-13T23:13:01.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:13:01 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2022-05-13T23:13:01.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:13:01 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-13T23:13:01.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:13:01 smithi047 ceph-mon[77147]: osdmap e73: 11 total, 11 up, 11 in 2022-05-13T23:13:01.573 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:13:01 smithi002 ceph-mon[71816]: from='client.24401 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi115:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2022-05-13T23:13:01.573 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:13:01 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2022-05-13T23:13:01.573 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:13:01 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2022-05-13T23:13:01.574 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:13:01 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-13T23:13:01.574 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:13:01 smithi002 ceph-mon[71816]: osdmap e73: 11 total, 11 up, 11 in 2022-05-13T23:13:02.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:13:02 smithi047 ceph-mon[77147]: pgmap v240: 1 pgs: 1 remapped+peering; 577 KiB data, 66 MiB used, 983 GiB / 983 GiB avail 2022-05-13T23:13:02.572 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:13:02 smithi002 ceph-mon[71816]: pgmap v240: 1 pgs: 1 remapped+peering; 577 KiB data, 66 MiB used, 983 GiB / 983 GiB avail 2022-05-13T23:13:02.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:13:02 smithi115 ceph-mon[78593]: pgmap v240: 1 pgs: 1 remapped+peering; 577 KiB data, 66 MiB used, 983 GiB / 983 GiB avail 2022-05-13T23:13:03.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:13:03 smithi047 ceph-mon[77147]: osdmap e74: 11 total, 11 up, 11 in 2022-05-13T23:13:03.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:13:03 smithi047 ceph-mon[77147]: from='client.? 172.21.15.115:0/2803589855' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "916541e0-fb28-46d5-9743-801c54c5a6d1"}]: dispatch 2022-05-13T23:13:03.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:13:03 smithi047 ceph-mon[77147]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "916541e0-fb28-46d5-9743-801c54c5a6d1"}]: dispatch 2022-05-13T23:13:03.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:13:03 smithi047 ceph-mon[77147]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "916541e0-fb28-46d5-9743-801c54c5a6d1"}]': finished 2022-05-13T23:13:03.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:13:03 smithi047 ceph-mon[77147]: osdmap e75: 12 total, 11 up, 12 in 2022-05-13T23:13:03.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:13:03 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2022-05-13T23:13:03.572 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:13:03 smithi002 ceph-mon[71816]: osdmap e74: 11 total, 11 up, 11 in 2022-05-13T23:13:03.572 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:13:03 smithi002 ceph-mon[71816]: from='client.? 172.21.15.115:0/2803589855' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "916541e0-fb28-46d5-9743-801c54c5a6d1"}]: dispatch 2022-05-13T23:13:03.573 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:13:03 smithi002 ceph-mon[71816]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "916541e0-fb28-46d5-9743-801c54c5a6d1"}]: dispatch 2022-05-13T23:13:03.573 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:13:03 smithi002 ceph-mon[71816]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "916541e0-fb28-46d5-9743-801c54c5a6d1"}]': finished 2022-05-13T23:13:03.573 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:13:03 smithi002 ceph-mon[71816]: osdmap e75: 12 total, 11 up, 12 in 2022-05-13T23:13:03.573 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:13:03 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2022-05-13T23:13:03.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:13:03 smithi115 ceph-mon[78593]: osdmap e74: 11 total, 11 up, 11 in 2022-05-13T23:13:03.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:13:03 smithi115 ceph-mon[78593]: from='client.? 172.21.15.115:0/2803589855' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "916541e0-fb28-46d5-9743-801c54c5a6d1"}]: dispatch 2022-05-13T23:13:03.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:13:03 smithi115 ceph-mon[78593]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "916541e0-fb28-46d5-9743-801c54c5a6d1"}]: dispatch 2022-05-13T23:13:03.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:13:03 smithi115 ceph-mon[78593]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "916541e0-fb28-46d5-9743-801c54c5a6d1"}]': finished 2022-05-13T23:13:03.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:13:03 smithi115 ceph-mon[78593]: osdmap e75: 12 total, 11 up, 12 in 2022-05-13T23:13:03.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:13:03 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2022-05-13T23:13:04.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:13:04 smithi047 ceph-mon[77147]: pgmap v244: 1 pgs: 1 remapped+peering; 577 KiB data, 66 MiB used, 983 GiB / 983 GiB avail 2022-05-13T23:13:04.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:13:04 smithi047 ceph-mon[77147]: from='client.? 172.21.15.115:0/3169273186' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2022-05-13T23:13:04.572 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:13:04 smithi002 ceph-mon[71816]: pgmap v244: 1 pgs: 1 remapped+peering; 577 KiB data, 66 MiB used, 983 GiB / 983 GiB avail 2022-05-13T23:13:04.573 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:13:04 smithi002 ceph-mon[71816]: from='client.? 172.21.15.115:0/3169273186' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2022-05-13T23:13:04.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:13:04 smithi115 ceph-mon[78593]: pgmap v244: 1 pgs: 1 remapped+peering; 577 KiB data, 66 MiB used, 983 GiB / 983 GiB avail 2022-05-13T23:13:04.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:13:04 smithi115 ceph-mon[78593]: from='client.? 172.21.15.115:0/3169273186' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2022-05-13T23:13:05.572 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:13:05 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-05-13T23:13:05.573 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:13:05 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-05-13T23:13:05.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:13:05 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-05-13T23:13:05.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:13:05 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-05-13T23:13:05.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:13:05 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-05-13T23:13:05.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:13:05 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-05-13T23:13:06.572 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:13:06 smithi002 ceph-mon[71816]: pgmap v245: 1 pgs: 1 active+clean; 577 KiB data, 68 MiB used, 983 GiB / 983 GiB avail; 104 KiB/s, 0 objects/s recovering 2022-05-13T23:13:06.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:13:06 smithi115 ceph-mon[78593]: pgmap v245: 1 pgs: 1 active+clean; 577 KiB data, 68 MiB used, 983 GiB / 983 GiB avail; 104 KiB/s, 0 objects/s recovering 2022-05-13T23:13:06.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:13:06 smithi047 ceph-mon[77147]: pgmap v245: 1 pgs: 1 active+clean; 577 KiB data, 68 MiB used, 983 GiB / 983 GiB avail; 104 KiB/s, 0 objects/s recovering 2022-05-13T23:13:08.535 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:13:08 smithi115 ceph-mon[78593]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 67 MiB used, 983 GiB / 983 GiB avail; 86 KiB/s, 0 objects/s recovering 2022-05-13T23:13:08.572 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:13:08 smithi002 ceph-mon[71816]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 67 MiB used, 983 GiB / 983 GiB avail; 86 KiB/s, 0 objects/s recovering 2022-05-13T23:13:08.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:13:08 smithi047 ceph-mon[77147]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 67 MiB used, 983 GiB / 983 GiB avail; 86 KiB/s, 0 objects/s recovering 2022-05-13T23:13:10.550 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:13:10 smithi115 ceph-mon[78593]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 67 MiB used, 983 GiB / 983 GiB avail; 65 KiB/s, 0 objects/s recovering 2022-05-13T23:13:10.550 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:13:10 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2022-05-13T23:13:10.551 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:13:10 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-13T23:13:10.573 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:13:10 smithi002 ceph-mon[71816]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 67 MiB used, 983 GiB / 983 GiB avail; 65 KiB/s, 0 objects/s recovering 2022-05-13T23:13:10.573 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:13:10 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2022-05-13T23:13:10.573 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:13:10 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-13T23:13:10.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:13:10 smithi047 ceph-mon[77147]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 67 MiB used, 983 GiB / 983 GiB avail; 65 KiB/s, 0 objects/s recovering 2022-05-13T23:13:10.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:13:10 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2022-05-13T23:13:10.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:13:10 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-13T23:13:11.572 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:13:11 smithi002 ceph-mon[71816]: Deploying daemon osd.11 on smithi115 2022-05-13T23:13:11.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:13:11 smithi115 ceph-mon[78593]: Deploying daemon osd.11 on smithi115 2022-05-13T23:13:11.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:13:11 smithi047 ceph-mon[77147]: Deploying daemon osd.11 on smithi115 2022-05-13T23:13:12.561 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:13:12 smithi115 ceph-mon[78593]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 67 MiB used, 983 GiB / 983 GiB avail; 58 KiB/s, 0 objects/s recovering 2022-05-13T23:13:12.572 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:13:12 smithi002 ceph-mon[71816]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 67 MiB used, 983 GiB / 983 GiB avail; 58 KiB/s, 0 objects/s recovering 2022-05-13T23:13:12.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:13:12 smithi047 ceph-mon[77147]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 67 MiB used, 983 GiB / 983 GiB avail; 58 KiB/s, 0 objects/s recovering 2022-05-13T23:13:14.567 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:13:14 smithi115 ceph-mon[78593]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 67 MiB used, 983 GiB / 983 GiB avail; 50 KiB/s, 0 objects/s recovering 2022-05-13T23:13:14.568 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:13:14 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:13:14.568 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:13:14 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-13T23:13:14.568 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:13:14 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-13T23:13:14.569 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:13:14 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-13T23:13:14.572 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:13:14 smithi002 ceph-mon[71816]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 67 MiB used, 983 GiB / 983 GiB avail; 50 KiB/s, 0 objects/s recovering 2022-05-13T23:13:14.572 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:13:14 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:13:14.573 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:13:14 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-13T23:13:14.573 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:13:14 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-13T23:13:14.573 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:13:14 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-13T23:13:14.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:13:14 smithi047 ceph-mon[77147]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 67 MiB used, 983 GiB / 983 GiB avail; 50 KiB/s, 0 objects/s recovering 2022-05-13T23:13:14.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:13:14 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:13:14.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:13:14 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-13T23:13:14.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:13:14 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-13T23:13:14.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:13:14 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-13T23:13:16.572 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:13:16 smithi002 ceph-mon[71816]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 67 MiB used, 983 GiB / 983 GiB avail; 43 KiB/s, 0 objects/s recovering 2022-05-13T23:13:16.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:13:16 smithi115 ceph-mon[78593]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 67 MiB used, 983 GiB / 983 GiB avail; 43 KiB/s, 0 objects/s recovering 2022-05-13T23:13:16.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:13:16 smithi047 ceph-mon[77147]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 67 MiB used, 983 GiB / 983 GiB avail; 43 KiB/s, 0 objects/s recovering 2022-05-13T23:13:17.297 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:13:17.295+0000 7f088a7fc700 1 -- 172.21.15.115:0/2460540634 <== mgr.14148 v2:172.21.15.2:6800/2788339627 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+38 (secure 0 0 0) 0x7f089405c920 con 0x7f0880096ce0 2022-05-13T23:13:17.298 INFO:teuthology.orchestra.run.smithi115.stdout:Created osd(s) 11 on host 'smithi115' 2022-05-13T23:13:17.299 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:13:17.297+0000 7f089bb60700 1 -- 172.21.15.115:0/2460540634 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f0880096ce0 msgr2=0x7f0880099190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:13:17.300 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:13:17.297+0000 7f089bb60700 1 --2- 172.21.15.115:0/2460540634 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f0880096ce0 0x7f0880099190 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f0884009ef0 tx=0x7f0884009cb0 comp rx=0 tx=0).stop 2022-05-13T23:13:17.300 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:13:17.297+0000 7f089bb60700 1 -- 172.21.15.115:0/2460540634 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f08940fa4f0 msgr2=0x7f0894065480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:13:17.300 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:13:17.297+0000 7f089bb60700 1 --2- 172.21.15.115:0/2460540634 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f08940fa4f0 0x7f0894065480 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f089000b500 tx=0x7f08900090f0 comp rx=0 tx=0).stop 2022-05-13T23:13:17.301 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:13:17.297+0000 7f089bb60700 1 -- 172.21.15.115:0/2460540634 shutdown_connections 2022-05-13T23:13:17.301 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:13:17.297+0000 7f089bb60700 1 --2- 172.21.15.115:0/2460540634 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f0880096ce0 0x7f0880099190 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:13:17.301 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:13:17.297+0000 7f089bb60700 1 --2- 172.21.15.115:0/2460540634 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f08940fc120 0x7f0894113840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:13:17.301 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:13:17.297+0000 7f089bb60700 1 --2- 172.21.15.115:0/2460540634 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f08940fa4f0 0x7f0894065480 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:13:17.302 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:13:17.297+0000 7f089bb60700 1 --2- 172.21.15.115:0/2460540634 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f08940fb790 0x7f08941111c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:13:17.302 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:13:17.297+0000 7f089bb60700 1 -- 172.21.15.115:0/2460540634 >> 172.21.15.115:0/2460540634 conn(0x7f08940f5ce0 msgr2=0x7f08940f8030 unknown :-1 s=STATE_NONE l=0).mark_down 2022-05-13T23:13:17.302 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:13:17.297+0000 7f089bb60700 1 -- 172.21.15.115:0/2460540634 shutdown_connections 2022-05-13T23:13:17.302 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:13:17.297+0000 7f089bb60700 1 -- 172.21.15.115:0/2460540634 wait complete. 2022-05-13T23:13:17.881 DEBUG:teuthology.orchestra.run.smithi115:osd.11> sudo journalctl -f -n 0 -u ceph-16cf5f88-d311-11ec-be40-001a4aab830c@osd.11.service 2022-05-13T23:13:17.883 INFO:tasks.cephadm:Waiting for 12 OSDs to come up... 2022-05-13T23:13:17.884 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:23b0658391ce94ca40799469eec00c28b298816c shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 16cf5f88-d311-11ec-be40-001a4aab830c -- ceph osd stat -f json 2022-05-13T23:13:17.908 INFO:journalctl@ceph.osd.11.smithi115.stdout:-- Logs begin at Fri 2022-05-13 22:45:29 UTC. -- 2022-05-13T23:13:18.182 INFO:journalctl@ceph.osd.11.smithi115.stdout:May 13 23:13:18 smithi115 conmon[102145]: 2022-05-13T23:13:18.086+0000 7f98e9cb93c0 -1 osd.11 0 log_to_monitors true 2022-05-13T23:13:18.182 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:13:17 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:13:18.182 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:13:17 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:13:18.183 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:13:17 smithi115 ceph-mon[78593]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 67 MiB used, 983 GiB / 983 GiB avail 2022-05-13T23:13:18.183 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:13:17 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:13:18.183 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:13:17 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-13T23:13:18.183 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:13:17 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-13T23:13:18.184 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:13:17 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-13T23:13:18.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:13:17 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:13:18.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:13:17 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:13:18.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:13:17 smithi047 ceph-mon[77147]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 67 MiB used, 983 GiB / 983 GiB avail 2022-05-13T23:13:18.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:13:17 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:13:18.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:13:17 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-13T23:13:18.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:13:17 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-13T23:13:18.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:13:17 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-13T23:13:18.322 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:13:17 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:13:18.323 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:13:17 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:13:18.323 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:13:17 smithi002 ceph-mon[71816]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 67 MiB used, 983 GiB / 983 GiB avail 2022-05-13T23:13:18.323 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:13:17 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:13:18.323 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:13:17 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-13T23:13:18.324 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:13:17 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-13T23:13:18.324 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:13:17 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-13T23:13:18.562 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/16cf5f88-d311-11ec-be40-001a4aab830c/mon.a/config 2022-05-13T23:13:19.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:13:18 smithi047 ceph-mon[77147]: from='osd.11 [v2:172.21.15.115:6824/1758718209,v1:172.21.15.115:6825/1758718209]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2022-05-13T23:13:19.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:13:18 smithi047 ceph-mon[77147]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2022-05-13T23:13:19.322 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:13:18 smithi002 ceph-mon[71816]: from='osd.11 [v2:172.21.15.115:6824/1758718209,v1:172.21.15.115:6825/1758718209]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2022-05-13T23:13:19.323 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:13:18 smithi002 ceph-mon[71816]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2022-05-13T23:13:19.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:13:18 smithi115 ceph-mon[78593]: from='osd.11 [v2:172.21.15.115:6824/1758718209,v1:172.21.15.115:6825/1758718209]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2022-05-13T23:13:19.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:13:18 smithi115 ceph-mon[78593]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2022-05-13T23:13:20.215 INFO:journalctl@ceph.osd.11.smithi115.stdout:May 13 23:13:19 smithi115 conmon[102145]: 2022-05-13T23:13:19.938+0000 7f98e1ec0700 -1 osd.11 0 waiting for initial osdmap 2022-05-13T23:13:20.215 INFO:journalctl@ceph.osd.11.smithi115.stdout:May 13 23:13:19 smithi115 conmon[102145]: 2022-05-13T23:13:19.965+0000 7f98dc856700 -1 osd.11 77 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2022-05-13T23:13:20.216 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:13:19 smithi115 ceph-mon[78593]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2022-05-13T23:13:20.216 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:13:19 smithi115 ceph-mon[78593]: osdmap e76: 12 total, 11 up, 12 in 2022-05-13T23:13:20.216 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:13:19 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2022-05-13T23:13:20.216 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:13:19 smithi115 ceph-mon[78593]: from='osd.11 [v2:172.21.15.115:6824/1758718209,v1:172.21.15.115:6825/1758718209]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi115", "root=default"]}]: dispatch 2022-05-13T23:13:20.217 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:13:19 smithi115 ceph-mon[78593]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi115", "root=default"]}]: dispatch 2022-05-13T23:13:20.217 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:13:19 smithi115 ceph-mon[78593]: pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 67 MiB used, 983 GiB / 983 GiB avail 2022-05-13T23:13:20.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:13:19 smithi047 ceph-mon[77147]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2022-05-13T23:13:20.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:13:19 smithi047 ceph-mon[77147]: osdmap e76: 12 total, 11 up, 12 in 2022-05-13T23:13:20.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:13:19 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2022-05-13T23:13:20.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:13:19 smithi047 ceph-mon[77147]: from='osd.11 [v2:172.21.15.115:6824/1758718209,v1:172.21.15.115:6825/1758718209]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi115", "root=default"]}]: dispatch 2022-05-13T23:13:20.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:13:19 smithi047 ceph-mon[77147]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi115", "root=default"]}]: dispatch 2022-05-13T23:13:20.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:13:19 smithi047 ceph-mon[77147]: pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 67 MiB used, 983 GiB / 983 GiB avail 2022-05-13T23:13:20.322 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:13:19 smithi002 ceph-mon[71816]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2022-05-13T23:13:20.323 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:13:19 smithi002 ceph-mon[71816]: osdmap e76: 12 total, 11 up, 12 in 2022-05-13T23:13:20.323 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:13:19 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2022-05-13T23:13:20.323 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:13:19 smithi002 ceph-mon[71816]: from='osd.11 [v2:172.21.15.115:6824/1758718209,v1:172.21.15.115:6825/1758718209]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi115", "root=default"]}]: dispatch 2022-05-13T23:13:20.323 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:13:19 smithi002 ceph-mon[71816]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi115", "root=default"]}]: dispatch 2022-05-13T23:13:20.324 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:13:19 smithi002 ceph-mon[71816]: pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 67 MiB used, 983 GiB / 983 GiB avail 2022-05-13T23:13:20.362 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:20.360+0000 7f9be041e700 1 -- 172.21.15.2:0/3677003933 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9bd80fd220 msgr2=0x7f9bd8103ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:13:20.363 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:20.360+0000 7f9be041e700 1 --2- 172.21.15.2:0/3677003933 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9bd80fd220 0x7f9bd8103ae0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f9bd0009be0 tx=0x7f9bd0004d10 comp rx=0 tx=0).stop 2022-05-13T23:13:20.363 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:20.360+0000 7f9be041e700 1 -- 172.21.15.2:0/3677003933 shutdown_connections 2022-05-13T23:13:20.364 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:20.360+0000 7f9be041e700 1 --2- 172.21.15.2:0/3677003933 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9bd80fd220 0x7f9bd8103ae0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:13:20.364 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:20.360+0000 7f9be041e700 1 --2- 172.21.15.2:0/3677003933 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f9bd80fb5f0 0x7f9bd80fba00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:13:20.364 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:20.360+0000 7f9be041e700 1 --2- 172.21.15.2:0/3677003933 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f9bd80fc890 0x7f9bd80fcce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:13:20.365 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:20.360+0000 7f9be041e700 1 -- 172.21.15.2:0/3677003933 >> 172.21.15.2:0/3677003933 conn(0x7f9bd80f6e00 msgr2=0x7f9bd80f9210 unknown :-1 s=STATE_NONE l=0).mark_down 2022-05-13T23:13:20.366 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:20.360+0000 7f9be041e700 1 -- 172.21.15.2:0/3677003933 shutdown_connections 2022-05-13T23:13:20.366 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:20.360+0000 7f9be041e700 1 -- 172.21.15.2:0/3677003933 wait complete. 2022-05-13T23:13:20.370 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:20.361+0000 7f9be041e700 1 Processor -- start 2022-05-13T23:13:20.370 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:20.362+0000 7f9be041e700 1 -- start start 2022-05-13T23:13:20.371 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:20.362+0000 7f9be041e700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9bd80fb5f0 0x7f9bd80701d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:13:20.371 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:20.362+0000 7f9be041e700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f9bd80fc890 0x7f9bd806e830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:13:20.372 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:20.362+0000 7f9be041e700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f9bd80fd220 0x7f9bd806ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:13:20.372 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:20.362+0000 7f9be041e700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9bd806f480 con 0x7f9bd80fb5f0 2022-05-13T23:13:20.372 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:20.362+0000 7f9be041e700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9bd806f5f0 con 0x7f9bd80fd220 2022-05-13T23:13:20.373 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:20.362+0000 7f9be041e700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9bd8116370 con 0x7f9bd80fc890 2022-05-13T23:13:20.373 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:20.362+0000 7f9bde1ba700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9bd80fb5f0 0x7f9bd80701d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:13:20.373 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:20.362+0000 7f9bde9bb700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f9bd80fd220 0x7f9bd806ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:13:20.374 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:20.362+0000 7f9bde9bb700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f9bd80fd220 0x7f9bd806ed70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.2:54860/0 (socket says 172.21.15.2:54860) 2022-05-13T23:13:20.374 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:20.363+0000 7f9bde9bb700 1 -- 172.21.15.2:0/895896292 learned_addr learned my addr 172.21.15.2:0/895896292 (peer_addr_for_me v2:172.21.15.2:0/0) 2022-05-13T23:13:20.375 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:20.363+0000 7f9bdd9b9700 1 --2- 172.21.15.2:0/895896292 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f9bd80fc890 0x7f9bd806e830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:13:20.375 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:20.363+0000 7f9bde9bb700 1 -- 172.21.15.2:0/895896292 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f9bd80fc890 msgr2=0x7f9bd806e830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:13:20.375 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:20.363+0000 7f9bde9bb700 1 --2- 172.21.15.2:0/895896292 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f9bd80fc890 0x7f9bd806e830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:13:20.376 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:20.363+0000 7f9bde9bb700 1 -- 172.21.15.2:0/895896292 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9bd80fb5f0 msgr2=0x7f9bd80701d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:13:20.376 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:20.363+0000 7f9bde9bb700 1 --2- 172.21.15.2:0/895896292 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9bd80fb5f0 0x7f9bd80701d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:13:20.376 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:20.363+0000 7f9bde9bb700 1 -- 172.21.15.2:0/895896292 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9bd4005040 con 0x7f9bd80fd220 2022-05-13T23:13:20.377 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:20.363+0000 7f9bde9bb700 1 --2- 172.21.15.2:0/895896292 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f9bd80fd220 0x7f9bd806ed70 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f9bd0015040 tx=0x7f9bd0004d10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-05-13T23:13:20.377 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:20.364+0000 7f9bceffd700 1 -- 172.21.15.2:0/895896292 <== mon.1 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bd000e030 con 0x7f9bd80fd220 2022-05-13T23:13:20.377 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:20.364+0000 7f9be041e700 1 -- 172.21.15.2:0/895896292 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9bd00097c0 con 0x7f9bd80fd220 2022-05-13T23:13:20.378 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:20.364+0000 7f9bceffd700 1 -- 172.21.15.2:0/895896292 <== mon.1 v2:172.21.15.47:3300/0 2 ==== config(26 keys) v1 ==== 1087+0+0 (secure 0 0 0) 0x7f9bd001c440 con 0x7f9bd80fd220 2022-05-13T23:13:20.378 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:20.364+0000 7f9bceffd700 1 -- 172.21.15.2:0/895896292 <== mon.1 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bd0012680 con 0x7f9bd80fd220 2022-05-13T23:13:20.379 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:20.364+0000 7f9be041e700 1 -- 172.21.15.2:0/895896292 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9bd8116800 con 0x7f9bd80fd220 2022-05-13T23:13:20.379 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:20.365+0000 7f9be041e700 1 -- 172.21.15.2:0/895896292 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9bd8106e40 con 0x7f9bd80fd220 2022-05-13T23:13:20.379 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:20.367+0000 7f9bceffd700 1 -- 172.21.15.2:0/895896292 <== mon.1 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 16) v1 ==== 125136+0+0 (secure 0 0 0) 0x7f9bd002c070 con 0x7f9bd80fd220 2022-05-13T23:13:20.380 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:20.368+0000 7f9bceffd700 1 --2- 172.21.15.2:0/895896292 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f9bc4096ce0 0x7f9bc4099190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:13:20.380 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:20.368+0000 7f9bceffd700 1 -- 172.21.15.2:0/895896292 <== mon.1 v2:172.21.15.47:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 6843+0+0 (secure 0 0 0) 0x7f9bd00c0ff0 con 0x7f9bd80fd220 2022-05-13T23:13:20.380 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:20.369+0000 7f9bde1ba700 1 --2- 172.21.15.2:0/895896292 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f9bc4096ce0 0x7f9bc4099190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:13:20.381 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:20.369+0000 7f9bde1ba700 1 --2- 172.21.15.2:0/895896292 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f9bc4096ce0 0x7f9bc4099190 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f9bd80fc6f0 tx=0x7f9bd400b040 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-05-13T23:13:20.381 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:20.370+0000 7f9bceffd700 1 -- 172.21.15.2:0/895896292 <== mon.1 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+165215 (secure 0 0 0) 0x7f9bd005b570 con 0x7f9bd80fd220 2022-05-13T23:13:20.564 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:20.562+0000 7f9be041e700 1 -- 172.21.15.2:0/895896292 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) v1 -- 0x7f9bd81029d0 con 0x7f9bd80fd220 2022-05-13T23:13:20.565 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:20.563+0000 7f9bceffd700 1 -- 172.21.15.2:0/895896292 <== mon.1 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v77) v1 ==== 74+0+133 (secure 0 0 0) 0x7f9bd00200a0 con 0x7f9bd80fd220 2022-05-13T23:13:20.565 INFO:teuthology.orchestra.run.smithi002.stdout: 2022-05-13T23:13:20.567 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:20.566+0000 7f9be041e700 1 -- 172.21.15.2:0/895896292 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f9bc4096ce0 msgr2=0x7f9bc4099190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:13:20.568 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:20.566+0000 7f9be041e700 1 --2- 172.21.15.2:0/895896292 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f9bc4096ce0 0x7f9bc4099190 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f9bd80fc6f0 tx=0x7f9bd400b040 comp rx=0 tx=0).stop 2022-05-13T23:13:20.568 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:20.566+0000 7f9be041e700 1 -- 172.21.15.2:0/895896292 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f9bd80fd220 msgr2=0x7f9bd806ed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:13:20.568 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:20.566+0000 7f9be041e700 1 --2- 172.21.15.2:0/895896292 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f9bd80fd220 0x7f9bd806ed70 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f9bd0015040 tx=0x7f9bd0004d10 comp rx=0 tx=0).stop 2022-05-13T23:13:20.568 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:20.566+0000 7f9be041e700 1 -- 172.21.15.2:0/895896292 shutdown_connections 2022-05-13T23:13:20.568 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:20.566+0000 7f9be041e700 1 --2- 172.21.15.2:0/895896292 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f9bd80fd220 0x7f9bd806ed70 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:13:20.569 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:20.566+0000 7f9be041e700 1 --2- 172.21.15.2:0/895896292 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f9bc4096ce0 0x7f9bc4099190 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:13:20.569 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:20.566+0000 7f9be041e700 1 --2- 172.21.15.2:0/895896292 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9bd80fb5f0 0x7f9bd80701d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:13:20.569 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:20.566+0000 7f9be041e700 1 --2- 172.21.15.2:0/895896292 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f9bd80fc890 0x7f9bd806e830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:13:20.569 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:20.566+0000 7f9be041e700 1 -- 172.21.15.2:0/895896292 >> 172.21.15.2:0/895896292 conn(0x7f9bd80f6e00 msgr2=0x7f9bd80ffcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-05-13T23:13:20.570 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:20.566+0000 7f9be041e700 1 -- 172.21.15.2:0/895896292 shutdown_connections 2022-05-13T23:13:20.570 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:20.566+0000 7f9be041e700 1 -- 172.21.15.2:0/895896292 wait complete. 2022-05-13T23:13:21.132 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":77,"num_osds":12,"num_up_osds":11,"osd_up_since":1652483579,"num_in_osds":12,"osd_in_since":1652483582,"num_remapped_pgs":0} 2022-05-13T23:13:21.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:13:20 smithi047 ceph-mon[77147]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi115", "root=default"]}]': finished 2022-05-13T23:13:21.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:13:20 smithi047 ceph-mon[77147]: osdmap e77: 12 total, 11 up, 12 in 2022-05-13T23:13:21.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:13:20 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2022-05-13T23:13:21.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:13:20 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2022-05-13T23:13:21.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:13:20 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:13:21.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:13:20 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-05-13T23:13:21.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:13:20 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-05-13T23:13:21.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:13:20 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-05-13T23:13:21.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:13:20 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-05-13T23:13:21.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:13:20 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:13:21.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:13:20 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:13:21.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:13:20 smithi047 ceph-mon[77147]: from='client.? 172.21.15.2:0/895896292' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2022-05-13T23:13:21.322 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:13:20 smithi002 ceph-mon[71816]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi115", "root=default"]}]': finished 2022-05-13T23:13:21.323 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:13:20 smithi002 ceph-mon[71816]: osdmap e77: 12 total, 11 up, 12 in 2022-05-13T23:13:21.323 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:13:20 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2022-05-13T23:13:21.323 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:13:20 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2022-05-13T23:13:21.323 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:13:20 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:13:21.324 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:13:20 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-05-13T23:13:21.324 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:13:20 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-05-13T23:13:21.324 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:13:20 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-05-13T23:13:21.324 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:13:20 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-05-13T23:13:21.324 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:13:20 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:13:21.325 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:13:20 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:13:21.325 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:13:20 smithi002 ceph-mon[71816]: from='client.? 172.21.15.2:0/895896292' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2022-05-13T23:13:21.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:13:20 smithi115 ceph-mon[78593]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi115", "root=default"]}]': finished 2022-05-13T23:13:21.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:13:20 smithi115 ceph-mon[78593]: osdmap e77: 12 total, 11 up, 12 in 2022-05-13T23:13:21.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:13:20 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2022-05-13T23:13:21.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:13:20 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2022-05-13T23:13:21.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:13:20 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:13:21.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:13:20 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-05-13T23:13:21.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:13:20 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-05-13T23:13:21.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:13:20 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-05-13T23:13:21.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:13:20 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-05-13T23:13:21.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:13:20 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:13:21.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:13:20 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:13:21.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:13:20 smithi115 ceph-mon[78593]: from='client.? 172.21.15.2:0/895896292' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2022-05-13T23:13:22.134 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:23b0658391ce94ca40799469eec00c28b298816c shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 16cf5f88-d311-11ec-be40-001a4aab830c -- ceph osd stat -f json 2022-05-13T23:13:22.213 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:13:21 smithi002 ceph-mon[71816]: purged_snaps scrub starts 2022-05-13T23:13:22.213 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:13:21 smithi002 ceph-mon[71816]: purged_snaps scrub ok 2022-05-13T23:13:22.213 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:13:21 smithi002 ceph-mon[71816]: Detected new or changed devices on smithi115 2022-05-13T23:13:22.214 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:13:21 smithi002 ceph-mon[71816]: Adjusting osd_memory_target on smithi115 to 4293M 2022-05-13T23:13:22.214 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:13:21 smithi002 ceph-mon[71816]: osd.11 [v2:172.21.15.115:6824/1758718209,v1:172.21.15.115:6825/1758718209] boot 2022-05-13T23:13:22.214 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:13:21 smithi002 ceph-mon[71816]: osdmap e78: 12 total, 12 up, 12 in 2022-05-13T23:13:22.214 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:13:21 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2022-05-13T23:13:22.215 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:13:21 smithi002 ceph-mon[71816]: pgmap v256: 1 pgs: 1 active+clean; 577 KiB data, 68 MiB used, 983 GiB / 983 GiB avail 2022-05-13T23:13:22.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:13:21 smithi047 ceph-mon[77147]: purged_snaps scrub starts 2022-05-13T23:13:22.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:13:21 smithi047 ceph-mon[77147]: purged_snaps scrub ok 2022-05-13T23:13:22.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:13:21 smithi047 ceph-mon[77147]: Detected new or changed devices on smithi115 2022-05-13T23:13:22.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:13:21 smithi047 ceph-mon[77147]: Adjusting osd_memory_target on smithi115 to 4293M 2022-05-13T23:13:22.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:13:21 smithi047 ceph-mon[77147]: osd.11 [v2:172.21.15.115:6824/1758718209,v1:172.21.15.115:6825/1758718209] boot 2022-05-13T23:13:22.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:13:21 smithi047 ceph-mon[77147]: osdmap e78: 12 total, 12 up, 12 in 2022-05-13T23:13:22.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:13:21 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2022-05-13T23:13:22.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:13:21 smithi047 ceph-mon[77147]: pgmap v256: 1 pgs: 1 active+clean; 577 KiB data, 68 MiB used, 983 GiB / 983 GiB avail 2022-05-13T23:13:22.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:13:21 smithi115 ceph-mon[78593]: purged_snaps scrub starts 2022-05-13T23:13:22.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:13:21 smithi115 ceph-mon[78593]: purged_snaps scrub ok 2022-05-13T23:13:22.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:13:21 smithi115 ceph-mon[78593]: Detected new or changed devices on smithi115 2022-05-13T23:13:22.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:13:21 smithi115 ceph-mon[78593]: Adjusting osd_memory_target on smithi115 to 4293M 2022-05-13T23:13:22.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:13:21 smithi115 ceph-mon[78593]: osd.11 [v2:172.21.15.115:6824/1758718209,v1:172.21.15.115:6825/1758718209] boot 2022-05-13T23:13:22.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:13:21 smithi115 ceph-mon[78593]: osdmap e78: 12 total, 12 up, 12 in 2022-05-13T23:13:22.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:13:21 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2022-05-13T23:13:22.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:13:21 smithi115 ceph-mon[78593]: pgmap v256: 1 pgs: 1 active+clean; 577 KiB data, 68 MiB used, 983 GiB / 983 GiB avail 2022-05-13T23:13:22.824 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/16cf5f88-d311-11ec-be40-001a4aab830c/mon.a/config 2022-05-13T23:13:23.231 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:13:22 smithi002 ceph-mon[71816]: osdmap e79: 12 total, 12 up, 12 in 2022-05-13T23:13:23.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:13:22 smithi047 ceph-mon[77147]: osdmap e79: 12 total, 12 up, 12 in 2022-05-13T23:13:23.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:13:22 smithi115 ceph-mon[78593]: osdmap e79: 12 total, 12 up, 12 in 2022-05-13T23:13:24.212 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:13:23 smithi002 ceph-mon[71816]: pgmap v258: 1 pgs: 1 active+clean; 577 KiB data, 73 MiB used, 1.0 TiB / 1.0 TiB avail 2022-05-13T23:13:24.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:13:23 smithi047 ceph-mon[77147]: pgmap v258: 1 pgs: 1 active+clean; 577 KiB data, 73 MiB used, 1.0 TiB / 1.0 TiB avail 2022-05-13T23:13:24.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:13:23 smithi115 ceph-mon[78593]: pgmap v258: 1 pgs: 1 active+clean; 577 KiB data, 73 MiB used, 1.0 TiB / 1.0 TiB avail 2022-05-13T23:13:24.547 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:24.540+0000 7efcf24bc700 1 -- 172.21.15.2:0/3803748487 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7efcec103580 msgr2=0x7efcec105960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:13:24.547 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:24.540+0000 7efcf24bc700 1 --2- 172.21.15.2:0/3803748487 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7efcec103580 0x7efcec105960 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7efce4009be0 tx=0x7efce4004d10 comp rx=0 tx=0).stop 2022-05-13T23:13:24.547 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:24.540+0000 7efcf24bc700 1 -- 172.21.15.2:0/3803748487 shutdown_connections 2022-05-13T23:13:24.548 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:24.540+0000 7efcf24bc700 1 --2- 172.21.15.2:0/3803748487 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7efcec103580 0x7efcec105960 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:13:24.548 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:24.540+0000 7efcf24bc700 1 --2- 172.21.15.2:0/3803748487 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7efcec070370 0x7efcec06e870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:13:24.548 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:24.540+0000 7efcf24bc700 1 --2- 172.21.15.2:0/3803748487 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efcec06edb0 0x7efcec06f220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:13:24.549 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:24.540+0000 7efcf24bc700 1 -- 172.21.15.2:0/3803748487 >> 172.21.15.2:0/3803748487 conn(0x7efcec0f6e00 msgr2=0x7efcec0f9210 unknown :-1 s=STATE_NONE l=0).mark_down 2022-05-13T23:13:24.549 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:24.540+0000 7efcf24bc700 1 -- 172.21.15.2:0/3803748487 shutdown_connections 2022-05-13T23:13:24.549 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:24.540+0000 7efcf24bc700 1 -- 172.21.15.2:0/3803748487 wait complete. 2022-05-13T23:13:24.549 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:24.541+0000 7efcf24bc700 1 Processor -- start 2022-05-13T23:13:24.550 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:24.541+0000 7efcf24bc700 1 -- start start 2022-05-13T23:13:24.550 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:24.541+0000 7efcf24bc700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efcec06edb0 0x7efcec10e4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:13:24.550 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:24.541+0000 7efcf24bc700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7efcec070370 0x7efcec10e9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:13:24.554 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:24.541+0000 7efcf24bc700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7efcec103580 0x7efcec115a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:13:24.554 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:24.542+0000 7efcf24bc700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efcec115fb0 con 0x7efcec070370 2022-05-13T23:13:24.554 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:24.542+0000 7efcf24bc700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efcec116120 con 0x7efcec06edb0 2022-05-13T23:13:24.554 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:24.542+0000 7efcf24bc700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efcec1162e0 con 0x7efcec103580 2022-05-13T23:13:24.556 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:24.542+0000 7efceb7fe700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7efcec070370 0x7efcec10e9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:13:24.556 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:24.542+0000 7efceb7fe700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7efcec070370 0x7efcec10e9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:43470/0 (socket says 172.21.15.2:43470) 2022-05-13T23:13:24.556 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:24.542+0000 7efceb7fe700 1 -- 172.21.15.2:0/2101999950 learned_addr learned my addr 172.21.15.2:0/2101999950 (peer_addr_for_me v2:172.21.15.2:0/0) 2022-05-13T23:13:24.557 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:24.542+0000 7efcebfff700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efcec06edb0 0x7efcec10e4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:13:24.557 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:24.542+0000 7efcf0a59700 1 --2- 172.21.15.2:0/2101999950 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7efcec103580 0x7efcec115a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:13:24.557 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:24.542+0000 7efceb7fe700 1 -- 172.21.15.2:0/2101999950 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7efcec103580 msgr2=0x7efcec115a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:13:24.557 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:24.542+0000 7efceb7fe700 1 --2- 172.21.15.2:0/2101999950 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7efcec103580 0x7efcec115a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:13:24.558 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:24.542+0000 7efceb7fe700 1 -- 172.21.15.2:0/2101999950 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efcec06edb0 msgr2=0x7efcec10e4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:13:24.558 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:24.542+0000 7efceb7fe700 1 --2- 172.21.15.2:0/2101999950 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efcec06edb0 0x7efcec10e4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:13:24.558 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:24.542+0000 7efceb7fe700 1 -- 172.21.15.2:0/2101999950 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efce40097c0 con 0x7efcec070370 2022-05-13T23:13:24.558 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:24.542+0000 7efcebfff700 1 --2- 172.21.15.2:0/2101999950 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efcec06edb0 0x7efcec10e4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-05-13T23:13:24.558 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:24.543+0000 7efceb7fe700 1 --2- 172.21.15.2:0/2101999950 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7efcec070370 0x7efcec10e9e0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7efce000e970 tx=0x7efce000b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-05-13T23:13:24.559 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:24.543+0000 7efce8ff9700 1 -- 172.21.15.2:0/2101999950 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efce0004660 con 0x7efcec070370 2022-05-13T23:13:24.559 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:24.543+0000 7efce8ff9700 1 -- 172.21.15.2:0/2101999950 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1087+0+0 (secure 0 0 0) 0x7efce000f040 con 0x7efcec070370 2022-05-13T23:13:24.559 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:24.543+0000 7efcf24bc700 1 -- 172.21.15.2:0/2101999950 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efcec116560 con 0x7efcec070370 2022-05-13T23:13:24.559 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:24.543+0000 7efce8ff9700 1 -- 172.21.15.2:0/2101999950 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efce000c800 con 0x7efcec070370 2022-05-13T23:13:24.560 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:24.543+0000 7efcf24bc700 1 -- 172.21.15.2:0/2101999950 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efcec116a80 con 0x7efcec070370 2022-05-13T23:13:24.560 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:24.544+0000 7efcf24bc700 1 -- 172.21.15.2:0/2101999950 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efcec108f90 con 0x7efcec070370 2022-05-13T23:13:24.561 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:24.546+0000 7efce8ff9700 1 -- 172.21.15.2:0/2101999950 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 16) v1 ==== 125136+0+0 (secure 0 0 0) 0x7efce001b030 con 0x7efcec070370 2022-05-13T23:13:24.562 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:24.547+0000 7efce8ff9700 1 --2- 172.21.15.2:0/2101999950 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7efcd4096ce0 0x7efcd4099190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:13:24.562 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:24.548+0000 7efce8ff9700 1 -- 172.21.15.2:0/2101999950 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 7123+0+0 (secure 0 0 0) 0x7efce00bea30 con 0x7efcec070370 2022-05-13T23:13:24.562 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:24.548+0000 7efcebfff700 1 --2- 172.21.15.2:0/2101999950 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7efcd4096ce0 0x7efcd4099190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:13:24.562 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:24.549+0000 7efcebfff700 1 --2- 172.21.15.2:0/2101999950 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7efcd4096ce0 0x7efcd4099190 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7efcdc007b50 tx=0x7efcdc00b040 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-05-13T23:13:24.563 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:24.550+0000 7efce8ff9700 1 -- 172.21.15.2:0/2101999950 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+165215 (secure 0 0 0) 0x7efce008c0c0 con 0x7efcec070370 2022-05-13T23:13:24.744 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:24.742+0000 7efcf24bc700 1 -- 172.21.15.2:0/2101999950 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) v1 -- 0x7efcec102260 con 0x7efcec070370 2022-05-13T23:13:24.744 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:24.743+0000 7efce8ff9700 1 -- 172.21.15.2:0/2101999950 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v79) v1 ==== 74+0+133 (secure 0 0 0) 0x7efce008f6e0 con 0x7efcec070370 2022-05-13T23:13:24.745 INFO:teuthology.orchestra.run.smithi002.stdout: 2022-05-13T23:13:24.747 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:24.745+0000 7efcf24bc700 1 -- 172.21.15.2:0/2101999950 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7efcd4096ce0 msgr2=0x7efcd4099190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:13:24.747 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:24.745+0000 7efcf24bc700 1 --2- 172.21.15.2:0/2101999950 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7efcd4096ce0 0x7efcd4099190 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7efcdc007b50 tx=0x7efcdc00b040 comp rx=0 tx=0).stop 2022-05-13T23:13:24.747 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:24.745+0000 7efcf24bc700 1 -- 172.21.15.2:0/2101999950 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7efcec070370 msgr2=0x7efcec10e9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:13:24.747 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:24.745+0000 7efcf24bc700 1 --2- 172.21.15.2:0/2101999950 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7efcec070370 0x7efcec10e9e0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7efce000e970 tx=0x7efce000b040 comp rx=0 tx=0).stop 2022-05-13T23:13:24.748 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:24.745+0000 7efcf24bc700 1 -- 172.21.15.2:0/2101999950 shutdown_connections 2022-05-13T23:13:24.748 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:24.745+0000 7efcf24bc700 1 --2- 172.21.15.2:0/2101999950 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7efcec103580 0x7efcec115a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:13:24.748 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:24.745+0000 7efcf24bc700 1 --2- 172.21.15.2:0/2101999950 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efcec06edb0 0x7efcec10e4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:13:24.748 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:24.745+0000 7efcf24bc700 1 --2- 172.21.15.2:0/2101999950 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7efcd4096ce0 0x7efcd4099190 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:13:24.749 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:24.745+0000 7efcf24bc700 1 --2- 172.21.15.2:0/2101999950 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7efcec070370 0x7efcec10e9e0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:13:24.749 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:24.745+0000 7efcf24bc700 1 -- 172.21.15.2:0/2101999950 >> 172.21.15.2:0/2101999950 conn(0x7efcec0f6e00 msgr2=0x7efcec0ff560 unknown :-1 s=STATE_NONE l=0).mark_down 2022-05-13T23:13:24.749 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:24.745+0000 7efcf24bc700 1 -- 172.21.15.2:0/2101999950 shutdown_connections 2022-05-13T23:13:24.749 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:24.745+0000 7efcf24bc700 1 -- 172.21.15.2:0/2101999950 wait complete. 2022-05-13T23:13:25.298 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":79,"num_osds":12,"num_up_osds":12,"osd_up_since":1652483600,"num_in_osds":12,"osd_in_since":1652483582,"num_remapped_pgs":0} 2022-05-13T23:13:25.299 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:23b0658391ce94ca40799469eec00c28b298816c shell --fsid 16cf5f88-d311-11ec-be40-001a4aab830c -- ceph osd dump --format=json 2022-05-13T23:13:25.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:13:25 smithi047 ceph-mon[77147]: from='client.? 172.21.15.2:0/2101999950' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2022-05-13T23:13:25.572 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:13:25 smithi002 ceph-mon[71816]: from='client.? 172.21.15.2:0/2101999950' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2022-05-13T23:13:25.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:13:25 smithi115 ceph-mon[78593]: from='client.? 172.21.15.2:0/2101999950' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2022-05-13T23:13:25.976 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/16cf5f88-d311-11ec-be40-001a4aab830c/mon.a/config 2022-05-13T23:13:26.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:13:26 smithi047 ceph-mon[77147]: pgmap v259: 1 pgs: 1 active+clean; 577 KiB data, 74 MiB used, 1.0 TiB / 1.0 TiB avail 2022-05-13T23:13:26.555 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:13:26 smithi002 ceph-mon[71816]: pgmap v259: 1 pgs: 1 active+clean; 577 KiB data, 74 MiB used, 1.0 TiB / 1.0 TiB avail 2022-05-13T23:13:26.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:13:26 smithi115 ceph-mon[78593]: pgmap v259: 1 pgs: 1 active+clean; 577 KiB data, 74 MiB used, 1.0 TiB / 1.0 TiB avail 2022-05-13T23:13:27.775 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:27.773+0000 7f2d1fae4700 1 -- 172.21.15.2:0/4264459880 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2d180fd810 msgr2=0x7f2d180fdc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:13:27.775 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:27.773+0000 7f2d1fae4700 1 --2- 172.21.15.2:0/4264459880 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2d180fd810 0x7f2d180fdc20 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f2d14004870 tx=0x7f2d14009cb0 comp rx=0 tx=0).stop 2022-05-13T23:13:27.775 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:27.773+0000 7f2d1fae4700 1 -- 172.21.15.2:0/4264459880 shutdown_connections 2022-05-13T23:13:27.776 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:27.773+0000 7f2d1fae4700 1 --2- 172.21.15.2:0/4264459880 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f2d180ff440 0x7f2d18105d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:13:27.776 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:27.773+0000 7f2d1fae4700 1 --2- 172.21.15.2:0/4264459880 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2d180fd810 0x7f2d180fdc20 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:13:27.776 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:27.773+0000 7f2d1fae4700 1 --2- 172.21.15.2:0/4264459880 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f2d180feab0 0x7f2d180fef00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:13:27.776 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:27.773+0000 7f2d1fae4700 1 -- 172.21.15.2:0/4264459880 >> 172.21.15.2:0/4264459880 conn(0x7f2d180f9030 msgr2=0x7f2d180fb430 unknown :-1 s=STATE_NONE l=0).mark_down 2022-05-13T23:13:27.777 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:27.773+0000 7f2d1fae4700 1 -- 172.21.15.2:0/4264459880 shutdown_connections 2022-05-13T23:13:27.777 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:27.773+0000 7f2d1fae4700 1 -- 172.21.15.2:0/4264459880 wait complete. 2022-05-13T23:13:27.779 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:27.774+0000 7f2d1fae4700 1 Processor -- start 2022-05-13T23:13:27.780 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:27.774+0000 7f2d1fae4700 1 -- start start 2022-05-13T23:13:27.780 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:27.774+0000 7f2d1fae4700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f2d180fd810 0x7f2d1810e650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:13:27.780 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:27.774+0000 7f2d1fae4700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2d180feab0 0x7f2d1810eb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:13:27.780 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:27.775+0000 7f2d1fae4700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f2d180ff440 0x7f2d18115c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:13:27.780 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:27.775+0000 7f2d1fae4700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d18116160 con 0x7f2d180feab0 2022-05-13T23:13:27.781 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:27.775+0000 7f2d1fae4700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d181162d0 con 0x7f2d180fd810 2022-05-13T23:13:27.781 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:27.775+0000 7f2d1fae4700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d18116490 con 0x7f2d180ff440 2022-05-13T23:13:27.784 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:27.775+0000 7f2d1d07f700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2d180feab0 0x7f2d1810eb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:13:27.784 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:27.775+0000 7f2d1d07f700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2d180feab0 0x7f2d1810eb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:43496/0 (socket says 172.21.15.2:43496) 2022-05-13T23:13:27.784 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:27.775+0000 7f2d1d07f700 1 -- 172.21.15.2:0/2847467719 learned_addr learned my addr 172.21.15.2:0/2847467719 (peer_addr_for_me v2:172.21.15.2:0/0) 2022-05-13T23:13:27.784 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:27.775+0000 7f2d1d880700 1 --2- 172.21.15.2:0/2847467719 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f2d180fd810 0x7f2d1810e650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:13:27.785 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:27.775+0000 7f2d1e081700 1 --2- 172.21.15.2:0/2847467719 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f2d180ff440 0x7f2d18115c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:13:27.785 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:27.775+0000 7f2d1d880700 1 -- 172.21.15.2:0/2847467719 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f2d180ff440 msgr2=0x7f2d18115c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:13:27.785 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:27.775+0000 7f2d1d880700 1 --2- 172.21.15.2:0/2847467719 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f2d180ff440 0x7f2d18115c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:13:27.785 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:27.775+0000 7f2d1d880700 1 -- 172.21.15.2:0/2847467719 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2d180feab0 msgr2=0x7f2d1810eb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:13:27.786 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:27.775+0000 7f2d1d880700 1 --2- 172.21.15.2:0/2847467719 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2d180feab0 0x7f2d1810eb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:13:27.786 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:27.775+0000 7f2d1d880700 1 -- 172.21.15.2:0/2847467719 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d14005040 con 0x7f2d180fd810 2022-05-13T23:13:27.786 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:27.775+0000 7f2d1d07f700 1 --2- 172.21.15.2:0/2847467719 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2d180feab0 0x7f2d1810eb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-05-13T23:13:27.786 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:27.776+0000 7f2d1e081700 1 --2- 172.21.15.2:0/2847467719 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f2d180ff440 0x7f2d18115c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2022-05-13T23:13:27.786 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:27.776+0000 7f2d1d880700 1 --2- 172.21.15.2:0/2847467719 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f2d180fd810 0x7f2d1810e650 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f2d14004ee0 tx=0x7f2d14009cb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-05-13T23:13:27.787 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:27.776+0000 7f2d0e7fc700 1 -- 172.21.15.2:0/2847467719 <== mon.1 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d14020070 con 0x7f2d180fd810 2022-05-13T23:13:27.787 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:27.776+0000 7f2d1fae4700 1 -- 172.21.15.2:0/2847467719 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2d18116710 con 0x7f2d180fd810 2022-05-13T23:13:27.787 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:27.776+0000 7f2d0e7fc700 1 -- 172.21.15.2:0/2847467719 <== mon.1 v2:172.21.15.47:3300/0 2 ==== config(26 keys) v1 ==== 1087+0+0 (secure 0 0 0) 0x7f2d14014e40 con 0x7f2d180fd810 2022-05-13T23:13:27.787 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:27.776+0000 7f2d0e7fc700 1 -- 172.21.15.2:0/2847467719 <== mon.1 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d1400e040 con 0x7f2d180fd810 2022-05-13T23:13:27.788 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:27.776+0000 7f2d1fae4700 1 -- 172.21.15.2:0/2847467719 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2d18116c40 con 0x7f2d180fd810 2022-05-13T23:13:27.788 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:27.777+0000 7f2d1fae4700 1 -- 172.21.15.2:0/2847467719 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2d18108ff0 con 0x7f2d180fd810 2022-05-13T23:13:27.789 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:27.779+0000 7f2d0e7fc700 1 -- 172.21.15.2:0/2847467719 <== mon.1 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 16) v1 ==== 125136+0+0 (secure 0 0 0) 0x7f2d1404abc0 con 0x7f2d180fd810 2022-05-13T23:13:27.789 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:27.780+0000 7f2d0e7fc700 1 --2- 172.21.15.2:0/2847467719 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f2d04096ce0 0x7f2d04099190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:13:27.790 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:27.781+0000 7f2d0e7fc700 1 -- 172.21.15.2:0/2847467719 <== mon.1 v2:172.21.15.47:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 7123+0+0 (secure 0 0 0) 0x7f2d140bffc0 con 0x7f2d180fd810 2022-05-13T23:13:27.790 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:27.781+0000 7f2d1d07f700 1 --2- 172.21.15.2:0/2847467719 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f2d04096ce0 0x7f2d04099190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:13:27.790 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:27.781+0000 7f2d1d07f700 1 --2- 172.21.15.2:0/2847467719 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f2d04096ce0 0x7f2d04099190 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f2d08007d30 tx=0x7f2d0800b040 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-05-13T23:13:27.790 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:27.783+0000 7f2d0e7fc700 1 -- 172.21.15.2:0/2847467719 <== mon.1 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+165215 (secure 0 0 0) 0x7f2d1408d650 con 0x7f2d180fd810 2022-05-13T23:13:27.973 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:27.971+0000 7f2d1fae4700 1 -- 172.21.15.2:0/2847467719 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2d18104770 con 0x7f2d180fd810 2022-05-13T23:13:27.974 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:27.972+0000 7f2d0e7fc700 1 -- 172.21.15.2:0/2847467719 <== mon.1 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v79) v1 ==== 74+0+18042 (secure 0 0 0) 0x7f2d14004490 con 0x7f2d180fd810 2022-05-13T23:13:27.975 INFO:teuthology.orchestra.run.smithi002.stdout: 2022-05-13T23:13:27.975 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":79,"fsid":"16cf5f88-d311-11ec-be40-001a4aab830c","created":"2022-05-13T23:06:00.765453+0000","modified":"2022-05-13T23:13:21.942299+0000","last_up_change":"2022-05-13T23:13:20.935982+0000","last_in_change":"2022-05-13T23:13:02.796000+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2022-05-13T23:09:32.000644+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"11e700e8-f3e8-4903-b52c-db354b4395bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6802","nonce":1174843704},{"type":"v1","addr":"172.21.15.2:6803","nonce":1174843704}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6804","nonce":1174843704},{"type":"v1","addr":"172.21.15.2:6805","nonce":1174843704}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6808","nonce":1174843704},{"type":"v1","addr":"172.21.15.2:6809","nonce":1174843704}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6806","nonce":1174843704},{"type":"v1","addr":"172.21.15.2:6807","nonce":1174843704}]},"public_addr":"172.21.15.2:6803/1174843704","cluster_addr":"172.21.15.2:6805/1174843704","heartbeat_back_addr":"172.21.15.2:6809/1174843704","heartbeat_front_addr":"172.21.15.2:6807/1174843704","state":["exists","up"]},{"osd":1,"uuid":"eabfa3cf-f2a3-4995-a7de-f6d8ba02ee6f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6800","nonce":3853826948},{"type":"v1","addr":"172.21.15.47:6801","nonce":3853826948}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6802","nonce":3853826948},{"type":"v1","addr":"172.21.15.47:6803","nonce":3853826948}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6806","nonce":3853826948},{"type":"v1","addr":"172.21.15.47:6807","nonce":3853826948}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6804","nonce":3853826948},{"type":"v1","addr":"172.21.15.47:6805","nonce":3853826948}]},"public_addr":"172.21.15.47:6801/3853826948","cluster_addr":"172.21.15.47:6803/3853826948","heartbeat_back_addr":"172.21.15.47:6807/3853826948","heartbeat_front_addr":"172.21.15.47:6805/3853826948","state":["exists","up"]},{"osd":2,"uuid":"153c5bd5-8200-47a7-a135-8a95b6752cd2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6800","nonce":2562256956},{"type":"v1","addr":"172.21.15.115:6801","nonce":2562256956}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6802","nonce":2562256956},{"type":"v1","addr":"172.21.15.115:6803","nonce":2562256956}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6806","nonce":2562256956},{"type":"v1","addr":"172.21.15.115:6807","nonce":2562256956}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6804","nonce":2562256956},{"type":"v1","addr":"172.21.15.115:6805","nonce":2562256956}]},"public_addr":"172.21.15.115:6801/2562256956","cluster_addr":"172.21.15.115:6803/2562256956","heartbeat_back_addr":"172.21.15.115:6807/2562256956","heartbeat_front_addr":"172.21.15.115:6805/2562256956","state":["exists","up"]},{"osd":3,"uuid":"6ca9c602-5936-48da-a861-c437e6a33765","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6810","nonce":1099909411},{"type":"v1","addr":"172.21.15.2:6811","nonce":1099909411}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6812","nonce":1099909411},{"type":"v1","addr":"172.21.15.2:6813","nonce":1099909411}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6816","nonce":1099909411},{"type":"v1","addr":"172.21.15.2:6817","nonce":1099909411}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6814","nonce":1099909411},{"type":"v1","addr":"172.21.15.2:6815","nonce":1099909411}]},"public_addr":"172.21.15.2:6811/1099909411","cluster_addr":"172.21.15.2:6813/1099909411","heartbeat_back_addr":"172.21.15.2:6817/1099909411","heartbeat_front_addr":"172.21.15.2:6815/1099909411","state":["exists","up"]},{"osd":4,"uuid":"54d943d9-2610-422a-9e59-4b8212aced31","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6808","nonce":882753861},{"type":"v1","addr":"172.21.15.47:6809","nonce":882753861}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6810","nonce":882753861},{"type":"v1","addr":"172.21.15.47:6811","nonce":882753861}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6814","nonce":882753861},{"type":"v1","addr":"172.21.15.47:6815","nonce":882753861}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6812","nonce":882753861},{"type":"v1","addr":"172.21.15.47:6813","nonce":882753861}]},"public_addr":"172.21.15.47:6809/882753861","cluster_addr":"172.21.15.47:6811/882753861","heartbeat_back_addr":"172.21.15.47:6815/882753861","heartbeat_front_addr":"172.21.15.47:6813/882753861","state":["exists","up"]},{"osd":5,"uuid":"c18566cf-5403-4057-b2ee-aa298e87e8e9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6808","nonce":2122391993},{"type":"v1","addr":"172.21.15.115:6809","nonce":2122391993}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6810","nonce":2122391993},{"type":"v1","addr":"172.21.15.115:6811","nonce":2122391993}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6814","nonce":2122391993},{"type":"v1","addr":"172.21.15.115:6815","nonce":2122391993}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6812","nonce":2122391993},{"type":"v1","addr":"172.21.15.115:6813","nonce":2122391993}]},"public_addr":"172.21.15.115:6809/2122391993","cluster_addr":"172.21.15.115:6811/2122391993","heartbeat_back_addr":"172.21.15.115:6815/2122391993","heartbeat_front_addr":"172.21.15.115:6813/2122391993","state":["exists","up"]},{"osd":6,"uuid":"38ec7b6b-30f4-4d97-ad7e-e160ae59c843","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6818","nonce":1892198790},{"type":"v1","addr":"172.21.15.2:6819","nonce":1892198790}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6820","nonce":1892198790},{"type":"v1","addr":"172.21.15.2:6821","nonce":1892198790}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6824","nonce":1892198790},{"type":"v1","addr":"172.21.15.2:6825","nonce":1892198790}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6822","nonce":1892198790},{"type":"v1","addr":"172.21.15.2:6823","nonce":1892198790}]},"public_addr":"172.21.15.2:6819/1892198790","cluster_addr":"172.21.15.2:6821/1892198790","heartbeat_back_addr":"172.21.15.2:6825/1892198790","heartbeat_front_addr":"172.21.15.2:6823/1892198790","state":["exists","up"]},{"osd":7,"uuid":"e799b560-c61e-4369-b08b-45e5a6af37c0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6816","nonce":2812542825},{"type":"v1","addr":"172.21.15.47:6817","nonce":2812542825}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6818","nonce":2812542825},{"type":"v1","addr":"172.21.15.47:6819","nonce":2812542825}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6822","nonce":2812542825},{"type":"v1","addr":"172.21.15.47:6823","nonce":2812542825}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6820","nonce":2812542825},{"type":"v1","addr":"172.21.15.47:6821","nonce":2812542825}]},"public_addr":"172.21.15.47:6817/2812542825","cluster_addr":"172.21.15.47:6819/2812542825","heartbeat_back_addr":"172.21.15.47:6823/2812542825","heartbeat_front_addr":"172.21.15.47:6821/2812542825","state":["exists","up"]},{"osd":8,"uuid":"a58f2c0e-e487-41cf-8d2e-8c937cac9ebc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":58,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6816","nonce":3959632033},{"type":"v1","addr":"172.21.15.115:6817","nonce":3959632033}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6818","nonce":3959632033},{"type":"v1","addr":"172.21.15.115:6819","nonce":3959632033}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6822","nonce":3959632033},{"type":"v1","addr":"172.21.15.115:6823","nonce":3959632033}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6820","nonce":3959632033},{"type":"v1","addr":"172.21.15.115:6821","nonce":3959632033}]},"public_addr":"172.21.15.115:6817/3959632033","cluster_addr":"172.21.15.115:6819/3959632033","heartbeat_back_addr":"172.21.15.115:6823/3959632033","heartbeat_front_addr":"172.21.15.115:6821/3959632033","state":["exists","up"]},{"osd":9,"uuid":"12cead96-f726-4f7a-ab84-eb191b778a5c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":66,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6826","nonce":3004162255},{"type":"v1","addr":"172.21.15.2:6827","nonce":3004162255}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6828","nonce":3004162255},{"type":"v1","addr":"172.21.15.2:6829","nonce":3004162255}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6832","nonce":3004162255},{"type":"v1","addr":"172.21.15.2:6833","nonce":3004162255}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6830","nonce":3004162255},{"type":"v1","addr":"172.21.15.2:6831","nonce":3004162255}]},"public_addr":"172.21.15.2:6827/3004162255","cluster_addr":"172.21.15.2:6829/3004162255","heartbeat_back_addr":"172.21.15.2:6833/3004162255","heartbeat_front_addr":"172.21.15.2:6831/3004162255","state":["exists","up"]},{"osd":10,"uuid":"4f648118-7855-48d6-86d5-44eed5295f0c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":71,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6824","nonce":3443413998},{"type":"v1","addr":"172.21.15.47:6825","nonce":3443413998}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6826","nonce":3443413998},{"type":"v1","addr":"172.21.15.47:6827","nonce":3443413998}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6830","nonce":3443413998},{"type":"v1","addr":"172.21.15.47:6831","nonce":3443413998}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6828","nonce":3443413998},{"type":"v1","addr":"172.21.15.47:6829","nonce":3443413998}]},"public_addr":"172.21.15.47:6825/3443413998","cluster_addr":"172.21.15.47:6827/3443413998","heartbeat_back_addr":"172.21.15.47:6831/3443413998","heartbeat_front_addr":"172.21.15.47:6829/3443413998","state":["exists","up"]},{"osd":11,"uuid":"916541e0-fb28-46d5-9743-801c54c5a6d1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":78,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6824","nonce":1758718209},{"type":"v1","addr":"172.21.15.115:6825","nonce":1758718209}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6826","nonce":1758718209},{"type":"v1","addr":"172.21.15.115:6827","nonce":1758718209}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6830","nonce":1758718209},{"type":"v1","addr":"172.21.15.115:6831","nonce":1758718209}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6828","nonce":1758718209},{"type":"v1","addr":"172.21.15.115:6829","nonce":1758718209}]},"public_addr":"172.21.15.115:6825/1758718209","cluster_addr":"172.21.15.115:6827/1758718209","heartbeat_back_addr":"172.21.15.115:6831/1758718209","heartbeat_front_addr":"172.21.15.115:6829/1758718209","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-05-13T23:08:43.757857+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-05-13T23:09:06.537139+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-05-13T23:09:28.524837+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-05-13T23:10:02.057583+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-05-13T23:10:26.210648+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-05-13T23:10:47.885853+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-05-13T23:11:19.141865+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-05-13T23:11:42.376065+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-05-13T23:12:04.175870+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-05-13T23:12:34.695347+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-05-13T23:12:57.762613+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-05-13T23:13:19.114284+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.2:0/477249925":"2022-05-14T23:07:04.986087+0000","172.21.15.2:6800/3226099420":"2022-05-14T23:07:04.986087+0000","172.21.15.2:0/2214093915":"2022-05-14T23:07:04.986087+0000","172.21.15.2:0/1039403355":"2022-05-14T23:07:04.986087+0000","172.21.15.2:0/22091675":"2022-05-14T23:06:36.304108+0000","172.21.15.2:6800/1255134634":"2022-05-14T23:06:36.304108+0000","172.21.15.2:6801/3226099420":"2022-05-14T23:07:04.986087+0000","172.21.15.2:0/3768677276":"2022-05-14T23:06:36.304108+0000","172.21.15.2:6801/1255134634":"2022-05-14T23:06:36.304108+0000","172.21.15.2:0/407841268":"2022-05-14T23:06:36.304108+0000"},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2022-05-13T23:13:27.977 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:27.975+0000 7f2d1fae4700 1 -- 172.21.15.2:0/2847467719 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f2d04096ce0 msgr2=0x7f2d04099190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:13:27.977 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:27.975+0000 7f2d1fae4700 1 --2- 172.21.15.2:0/2847467719 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f2d04096ce0 0x7f2d04099190 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f2d08007d30 tx=0x7f2d0800b040 comp rx=0 tx=0).stop 2022-05-13T23:13:27.977 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:27.975+0000 7f2d1fae4700 1 -- 172.21.15.2:0/2847467719 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f2d180fd810 msgr2=0x7f2d1810e650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:13:27.978 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:27.975+0000 7f2d1fae4700 1 --2- 172.21.15.2:0/2847467719 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f2d180fd810 0x7f2d1810e650 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f2d14004ee0 tx=0x7f2d14009cb0 comp rx=0 tx=0).stop 2022-05-13T23:13:27.978 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:27.975+0000 7f2d1fae4700 1 -- 172.21.15.2:0/2847467719 shutdown_connections 2022-05-13T23:13:27.978 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:27.975+0000 7f2d1fae4700 1 --2- 172.21.15.2:0/2847467719 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f2d180ff440 0x7f2d18115c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:13:27.978 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:27.975+0000 7f2d1fae4700 1 --2- 172.21.15.2:0/2847467719 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f2d180fd810 0x7f2d1810e650 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:13:27.979 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:27.975+0000 7f2d1fae4700 1 --2- 172.21.15.2:0/2847467719 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f2d04096ce0 0x7f2d04099190 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:13:27.979 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:27.975+0000 7f2d1fae4700 1 --2- 172.21.15.2:0/2847467719 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2d180feab0 0x7f2d1810eb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:13:27.979 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:27.975+0000 7f2d1fae4700 1 -- 172.21.15.2:0/2847467719 >> 172.21.15.2:0/2847467719 conn(0x7f2d180f9030 msgr2=0x7f2d180fb400 unknown :-1 s=STATE_NONE l=0).mark_down 2022-05-13T23:13:27.979 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:27.975+0000 7f2d1fae4700 1 -- 172.21.15.2:0/2847467719 shutdown_connections 2022-05-13T23:13:27.980 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:27.975+0000 7f2d1fae4700 1 -- 172.21.15.2:0/2847467719 wait complete. 2022-05-13T23:13:28.532 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:13:28 smithi002 ceph-mon[71816]: pgmap v260: 1 pgs: 1 active+clean; 577 KiB data, 74 MiB used, 1.0 TiB / 1.0 TiB avail 2022-05-13T23:13:28.533 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:13:28 smithi002 ceph-mon[71816]: from='client.? 172.21.15.2:0/2847467719' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2022-05-13T23:13:28.537 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2022-05-13T23:09:32.000644+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}] 2022-05-13T23:13:28.538 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:23b0658391ce94ca40799469eec00c28b298816c shell --fsid 16cf5f88-d311-11ec-be40-001a4aab830c -- ceph osd pool get .mgr pg_num 2022-05-13T23:13:28.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:13:28 smithi047 ceph-mon[77147]: pgmap v260: 1 pgs: 1 active+clean; 577 KiB data, 74 MiB used, 1.0 TiB / 1.0 TiB avail 2022-05-13T23:13:28.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:13:28 smithi047 ceph-mon[77147]: from='client.? 172.21.15.2:0/2847467719' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2022-05-13T23:13:28.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:13:28 smithi115 ceph-mon[78593]: pgmap v260: 1 pgs: 1 active+clean; 577 KiB data, 74 MiB used, 1.0 TiB / 1.0 TiB avail 2022-05-13T23:13:28.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:13:28 smithi115 ceph-mon[78593]: from='client.? 172.21.15.2:0/2847467719' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2022-05-13T23:13:29.209 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/16cf5f88-d311-11ec-be40-001a4aab830c/mon.a/config 2022-05-13T23:13:30.322 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:13:30 smithi002 ceph-mon[71816]: pgmap v261: 1 pgs: 1 active+clean; 577 KiB data, 74 MiB used, 1.0 TiB / 1.0 TiB avail 2022-05-13T23:13:30.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:13:30 smithi047 ceph-mon[77147]: pgmap v261: 1 pgs: 1 active+clean; 577 KiB data, 74 MiB used, 1.0 TiB / 1.0 TiB avail 2022-05-13T23:13:30.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:13:30 smithi115 ceph-mon[78593]: pgmap v261: 1 pgs: 1 active+clean; 577 KiB data, 74 MiB used, 1.0 TiB / 1.0 TiB avail 2022-05-13T23:13:31.055 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:31.052+0000 7f27b0981700 1 -- 172.21.15.2:0/3557277354 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f27ac0fea90 msgr2=0x7f27ac0feee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:13:31.055 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:31.052+0000 7f27b0981700 1 --2- 172.21.15.2:0/3557277354 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f27ac0fea90 0x7f27ac0feee0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f27a0004870 tx=0x7f27a0009cb0 comp rx=0 tx=0).stop 2022-05-13T23:13:31.055 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:31.052+0000 7f27b0981700 1 -- 172.21.15.2:0/3557277354 shutdown_connections 2022-05-13T23:13:31.055 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:31.052+0000 7f27b0981700 1 --2- 172.21.15.2:0/3557277354 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f27ac0ff420 0x7f27ac105ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:13:31.056 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:31.052+0000 7f27b0981700 1 --2- 172.21.15.2:0/3557277354 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f27ac0fd7f0 0x7f27ac0fdc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:13:31.056 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:31.052+0000 7f27b0981700 1 --2- 172.21.15.2:0/3557277354 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f27ac0fea90 0x7f27ac0feee0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:13:31.056 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:31.052+0000 7f27b0981700 1 -- 172.21.15.2:0/3557277354 >> 172.21.15.2:0/3557277354 conn(0x7f27ac0f9030 msgr2=0x7f27ac0fb410 unknown :-1 s=STATE_NONE l=0).mark_down 2022-05-13T23:13:31.056 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:31.052+0000 7f27b0981700 1 -- 172.21.15.2:0/3557277354 shutdown_connections 2022-05-13T23:13:31.057 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:31.052+0000 7f27b0981700 1 -- 172.21.15.2:0/3557277354 wait complete. 2022-05-13T23:13:31.057 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:31.053+0000 7f27b0981700 1 Processor -- start 2022-05-13T23:13:31.057 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:31.053+0000 7f27b0981700 1 -- start start 2022-05-13T23:13:31.059 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:31.054+0000 7f27b0981700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f27ac0fd7f0 0x7f27ac10e630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:13:31.059 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:31.054+0000 7f27b0981700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f27ac0fea90 0x7f27ac10eb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:13:31.060 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:31.054+0000 7f27b0981700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f27ac0ff420 0x7f27ac115c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:13:31.060 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:31.054+0000 7f27b0981700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27ac116140 con 0x7f27ac0fd7f0 2022-05-13T23:13:31.061 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:31.054+0000 7f27b0981700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27ac1162b0 con 0x7f27ac0fea90 2022-05-13T23:13:31.061 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:31.054+0000 7f27b0981700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27ac116470 con 0x7f27ac0ff420 2022-05-13T23:13:31.061 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:31.054+0000 7f27aa59c700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f27ac0fd7f0 0x7f27ac10e630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:13:31.061 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:31.054+0000 7f27aa59c700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f27ac0fd7f0 0x7f27ac10e630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:35336/0 (socket says 172.21.15.2:35336) 2022-05-13T23:13:31.062 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:31.054+0000 7f27aa59c700 1 -- 172.21.15.2:0/1921153330 learned_addr learned my addr 172.21.15.2:0/1921153330 (peer_addr_for_me v2:172.21.15.2:0/0) 2022-05-13T23:13:31.062 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:31.054+0000 7f27aad9d700 1 --2- 172.21.15.2:0/1921153330 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f27ac0ff420 0x7f27ac115c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:13:31.062 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:31.054+0000 7f27a9d9b700 1 --2- 172.21.15.2:0/1921153330 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f27ac0fea90 0x7f27ac10eb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:13:31.062 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:31.055+0000 7f27aa59c700 1 -- 172.21.15.2:0/1921153330 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f27ac0ff420 msgr2=0x7f27ac115c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:13:31.063 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:31.055+0000 7f27aa59c700 1 --2- 172.21.15.2:0/1921153330 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f27ac0ff420 0x7f27ac115c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:13:31.063 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:31.055+0000 7f27aa59c700 1 -- 172.21.15.2:0/1921153330 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f27ac0fea90 msgr2=0x7f27ac10eb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:13:31.063 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:31.055+0000 7f27aa59c700 1 --2- 172.21.15.2:0/1921153330 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f27ac0fea90 0x7f27ac10eb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:13:31.063 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:31.055+0000 7f27aa59c700 1 -- 172.21.15.2:0/1921153330 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27a0005040 con 0x7f27ac0fd7f0 2022-05-13T23:13:31.063 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:31.055+0000 7f27a9d9b700 1 --2- 172.21.15.2:0/1921153330 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f27ac0fea90 0x7f27ac10eb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-05-13T23:13:31.064 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:31.055+0000 7f27aa59c700 1 --2- 172.21.15.2:0/1921153330 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f27ac0fd7f0 0x7f27ac10e630 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f279c00e940 tx=0x7f279c00b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-05-13T23:13:31.064 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:31.055+0000 7f279affd700 1 -- 172.21.15.2:0/1921153330 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f279c007b40 con 0x7f27ac0fd7f0 2022-05-13T23:13:31.064 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:31.055+0000 7f279affd700 1 -- 172.21.15.2:0/1921153330 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1087+0+0 (secure 0 0 0) 0x7f279c00f040 con 0x7f27ac0fd7f0 2022-05-13T23:13:31.065 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:31.055+0000 7f279affd700 1 -- 172.21.15.2:0/1921153330 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f279c014890 con 0x7f27ac0fd7f0 2022-05-13T23:13:31.065 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:31.055+0000 7f27b0981700 1 -- 172.21.15.2:0/1921153330 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f27ac1166f0 con 0x7f27ac0fd7f0 2022-05-13T23:13:31.065 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:31.056+0000 7f27b0981700 1 -- 172.21.15.2:0/1921153330 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f27ac116c10 con 0x7f27ac0fd7f0 2022-05-13T23:13:31.065 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:31.057+0000 7f27b0981700 1 -- 172.21.15.2:0/1921153330 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f27ac108fd0 con 0x7f27ac0fd7f0 2022-05-13T23:13:31.067 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:31.059+0000 7f279affd700 1 -- 172.21.15.2:0/1921153330 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 16) v1 ==== 125136+0+0 (secure 0 0 0) 0x7f279c01b030 con 0x7f27ac0fd7f0 2022-05-13T23:13:31.067 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:31.060+0000 7f279affd700 1 --2- 172.21.15.2:0/1921153330 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f2794096ce0 0x7f2794099190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:13:31.068 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:31.060+0000 7f279affd700 1 -- 172.21.15.2:0/1921153330 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 7123+0+0 (secure 0 0 0) 0x7f279c0beaf0 con 0x7f27ac0fd7f0 2022-05-13T23:13:31.068 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:31.060+0000 7f27a9d9b700 1 --2- 172.21.15.2:0/1921153330 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f2794096ce0 0x7f2794099190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:13:31.068 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:31.061+0000 7f27a9d9b700 1 --2- 172.21.15.2:0/1921153330 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f2794096ce0 0x7f2794099190 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f27a0004840 tx=0x7f27a0009cb0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-05-13T23:13:31.069 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:31.062+0000 7f279affd700 1 -- 172.21.15.2:0/1921153330 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+165215 (secure 0 0 0) 0x7f279c04b570 con 0x7f27ac0fd7f0 2022-05-13T23:13:31.259 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:31.257+0000 7f27b0981700 1 -- 172.21.15.2:0/1921153330 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f27ac116ef0 con 0x7f27ac0fd7f0 2022-05-13T23:13:31.260 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:31.258+0000 7f279affd700 1 -- 172.21.15.2:0/1921153330 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v79) v1 ==== 93+0+10 (secure 0 0 0) 0x7f279c08c210 con 0x7f27ac0fd7f0 2022-05-13T23:13:31.260 INFO:teuthology.orchestra.run.smithi002.stdout:pg_num: 1 2022-05-13T23:13:31.262 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:31.260+0000 7f27b0981700 1 -- 172.21.15.2:0/1921153330 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f2794096ce0 msgr2=0x7f2794099190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:13:31.263 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:31.260+0000 7f27b0981700 1 --2- 172.21.15.2:0/1921153330 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f2794096ce0 0x7f2794099190 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f27a0004840 tx=0x7f27a0009cb0 comp rx=0 tx=0).stop 2022-05-13T23:13:31.263 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:31.260+0000 7f27b0981700 1 -- 172.21.15.2:0/1921153330 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f27ac0fd7f0 msgr2=0x7f27ac10e630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:13:31.263 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:31.260+0000 7f27b0981700 1 --2- 172.21.15.2:0/1921153330 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f27ac0fd7f0 0x7f27ac10e630 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f279c00e940 tx=0x7f279c00b040 comp rx=0 tx=0).stop 2022-05-13T23:13:31.263 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:31.261+0000 7f27b0981700 1 -- 172.21.15.2:0/1921153330 shutdown_connections 2022-05-13T23:13:31.263 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:31.261+0000 7f27b0981700 1 --2- 172.21.15.2:0/1921153330 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f27ac0ff420 0x7f27ac115c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:13:31.264 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:31.261+0000 7f27b0981700 1 --2- 172.21.15.2:0/1921153330 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f2794096ce0 0x7f2794099190 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:13:31.264 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:31.261+0000 7f27b0981700 1 --2- 172.21.15.2:0/1921153330 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f27ac0fd7f0 0x7f27ac10e630 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:13:31.264 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:31.261+0000 7f27b0981700 1 --2- 172.21.15.2:0/1921153330 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f27ac0fea90 0x7f27ac10eb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:13:31.264 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:31.261+0000 7f27b0981700 1 -- 172.21.15.2:0/1921153330 >> 172.21.15.2:0/1921153330 conn(0x7f27ac0f9030 msgr2=0x7f27ac0fb3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-05-13T23:13:31.264 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:31.261+0000 7f27b0981700 1 -- 172.21.15.2:0/1921153330 shutdown_connections 2022-05-13T23:13:31.265 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:31.261+0000 7f27b0981700 1 -- 172.21.15.2:0/1921153330 wait complete. 2022-05-13T23:13:31.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:13:31 smithi047 ceph-mon[77147]: from='client.? 172.21.15.2:0/1921153330' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2022-05-13T23:13:31.573 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:13:31 smithi002 ceph-mon[71816]: from='client.? 172.21.15.2:0/1921153330' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2022-05-13T23:13:31.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:13:31 smithi115 ceph-mon[78593]: from='client.? 172.21.15.2:0/1921153330' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2022-05-13T23:13:31.784 INFO:tasks.cephadm:Adding mds.a on smithi002 2022-05-13T23:13:31.784 INFO:tasks.cephadm:Adding mds.d on smithi002 2022-05-13T23:13:31.785 INFO:tasks.cephadm:Adding mds.g on smithi002 2022-05-13T23:13:31.785 INFO:tasks.cephadm:Adding mds.j on smithi002 2022-05-13T23:13:31.785 INFO:tasks.cephadm:Adding mds.b on smithi047 2022-05-13T23:13:31.786 INFO:tasks.cephadm:Adding mds.e on smithi047 2022-05-13T23:13:31.786 INFO:tasks.cephadm:Adding mds.h on smithi047 2022-05-13T23:13:31.786 INFO:tasks.cephadm:Adding mds.k on smithi047 2022-05-13T23:13:31.787 INFO:tasks.cephadm:Adding mds.c on smithi115 2022-05-13T23:13:31.787 INFO:tasks.cephadm:Adding mds.f on smithi115 2022-05-13T23:13:31.787 INFO:tasks.cephadm:Adding mds.i on smithi115 2022-05-13T23:13:31.788 INFO:tasks.cephadm:Adding mds.l on smithi115 2022-05-13T23:13:31.788 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:23b0658391ce94ca40799469eec00c28b298816c shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 16cf5f88-d311-11ec-be40-001a4aab830c -- ceph orch apply mds all '12;smithi002=a;smithi002=d;smithi002=g;smithi002=j;smithi047=b;smithi047=e;smithi047=h;smithi047=k;smithi115=c;smithi115=f;smithi115=i;smithi115=l' 2022-05-13T23:13:32.243 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/16cf5f88-d311-11ec-be40-001a4aab830c/mon.c/config 2022-05-13T23:13:32.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:13:32 smithi047 ceph-mon[77147]: pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 74 MiB used, 1.0 TiB / 1.0 TiB avail 2022-05-13T23:13:32.551 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:13:32 smithi115 ceph-mon[78593]: pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 74 MiB used, 1.0 TiB / 1.0 TiB avail 2022-05-13T23:13:32.572 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:13:32 smithi002 ceph-mon[71816]: pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 74 MiB used, 1.0 TiB / 1.0 TiB avail 2022-05-13T23:13:33.693 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:13:33.690+0000 7f5b15dba700 1 -- 172.21.15.115:0/1675048170 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f5b100fa4f0 msgr2=0x7f5b100fa900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:13:33.694 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:13:33.690+0000 7f5b15dba700 1 --2- 172.21.15.115:0/1675048170 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f5b100fa4f0 0x7f5b100fa900 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f5b00009be0 tx=0x7f5b00004d10 comp rx=0 tx=0).stop 2022-05-13T23:13:33.694 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:13:33.690+0000 7f5b15dba700 1 -- 172.21.15.115:0/1675048170 shutdown_connections 2022-05-13T23:13:33.694 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:13:33.690+0000 7f5b15dba700 1 --2- 172.21.15.115:0/1675048170 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5b100fc120 0x7f5b101029e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:13:33.694 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:13:33.690+0000 7f5b15dba700 1 --2- 172.21.15.115:0/1675048170 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f5b100fa4f0 0x7f5b100fa900 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:13:33.695 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:13:33.690+0000 7f5b15dba700 1 --2- 172.21.15.115:0/1675048170 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5b100fb790 0x7f5b100fbbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:13:33.695 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:13:33.690+0000 7f5b15dba700 1 -- 172.21.15.115:0/1675048170 >> 172.21.15.115:0/1675048170 conn(0x7f5b100f5ce0 msgr2=0x7f5b100f8110 unknown :-1 s=STATE_NONE l=0).mark_down 2022-05-13T23:13:33.695 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:13:33.690+0000 7f5b15dba700 1 -- 172.21.15.115:0/1675048170 shutdown_connections 2022-05-13T23:13:33.695 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:13:33.690+0000 7f5b15dba700 1 -- 172.21.15.115:0/1675048170 wait complete. 2022-05-13T23:13:33.695 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:13:33.690+0000 7f5b15dba700 1 Processor -- start 2022-05-13T23:13:33.696 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:13:33.691+0000 7f5b15dba700 1 -- start start 2022-05-13T23:13:33.696 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:13:33.691+0000 7f5b15dba700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5b100fa4f0 0x7f5b1010c370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:13:33.696 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:13:33.691+0000 7f5b15dba700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5b100fb790 0x7f5b1010c8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:13:33.696 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:13:33.691+0000 7f5b15dba700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f5b100fc120 0x7f5b10113940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:13:33.696 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:13:33.691+0000 7f5b15dba700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b10113f10 con 0x7f5b100fa4f0 2022-05-13T23:13:33.697 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:13:33.691+0000 7f5b15dba700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b10114080 con 0x7f5b100fb790 2022-05-13T23:13:33.697 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:13:33.691+0000 7f5b15dba700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b10114240 con 0x7f5b100fc120 2022-05-13T23:13:33.697 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:13:33.691+0000 7f5b0ffff700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f5b100fc120 0x7f5b10113940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:13:33.697 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:13:33.691+0000 7f5b0ffff700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f5b100fc120 0x7f5b10113940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:49382/0 (socket says 172.21.15.115:49382) 2022-05-13T23:13:33.697 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:13:33.691+0000 7f5b0ffff700 1 -- 172.21.15.115:0/3546394998 learned_addr learned my addr 172.21.15.115:0/3546394998 (peer_addr_for_me v2:172.21.15.115:0/0) 2022-05-13T23:13:33.698 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:13:33.691+0000 7f5b0effd700 1 --2- 172.21.15.115:0/3546394998 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5b100fb790 0x7f5b1010c8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:13:33.698 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:13:33.691+0000 7f5b0ffff700 1 -- 172.21.15.115:0/3546394998 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5b100fb790 msgr2=0x7f5b1010c8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:13:33.698 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:13:33.691+0000 7f5b0ffff700 1 --2- 172.21.15.115:0/3546394998 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5b100fb790 0x7f5b1010c8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:13:33.698 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:13:33.691+0000 7f5b0ffff700 1 -- 172.21.15.115:0/3546394998 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5b100fa4f0 msgr2=0x7f5b1010c370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:13:33.699 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:13:33.691+0000 7f5b0f7fe700 1 --2- 172.21.15.115:0/3546394998 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5b100fa4f0 0x7f5b1010c370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:13:33.699 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:13:33.691+0000 7f5b0ffff700 1 --2- 172.21.15.115:0/3546394998 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5b100fa4f0 0x7f5b1010c370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:13:33.699 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:13:33.691+0000 7f5b0ffff700 1 -- 172.21.15.115:0/3546394998 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b000097c0 con 0x7f5b100fc120 2022-05-13T23:13:33.699 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:13:33.691+0000 7f5b0f7fe700 1 --2- 172.21.15.115:0/3546394998 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5b100fa4f0 0x7f5b1010c370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2022-05-13T23:13:33.699 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:13:33.691+0000 7f5b0ffff700 1 --2- 172.21.15.115:0/3546394998 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f5b100fc120 0x7f5b10113940 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f5b080041d0 tx=0x7f5b08009f60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-05-13T23:13:33.700 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:13:33.691+0000 7f5af7fff700 1 -- 172.21.15.115:0/3546394998 <== mon.2 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b08007b40 con 0x7f5b100fc120 2022-05-13T23:13:33.700 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:13:33.691+0000 7f5af7fff700 1 -- 172.21.15.115:0/3546394998 <== mon.2 v2:172.21.15.115:3300/0 2 ==== config(26 keys) v1 ==== 1087+0+0 (secure 0 0 0) 0x7f5b0800f040 con 0x7f5b100fc120 2022-05-13T23:13:33.700 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:13:33.691+0000 7f5b15dba700 1 -- 172.21.15.115:0/3546394998 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5b10114520 con 0x7f5b100fc120 2022-05-13T23:13:33.700 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:13:33.691+0000 7f5af7fff700 1 -- 172.21.15.115:0/3546394998 <== mon.2 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b080137e0 con 0x7f5b100fc120 2022-05-13T23:13:33.700 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:13:33.692+0000 7f5b15dba700 1 -- 172.21.15.115:0/3546394998 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5b10114a40 con 0x7f5b100fc120 2022-05-13T23:13:33.701 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:13:33.692+0000 7f5b15dba700 1 -- 172.21.15.115:0/3546394998 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5b10105d40 con 0x7f5b100fc120 2022-05-13T23:13:33.702 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:13:33.693+0000 7f5af7fff700 1 -- 172.21.15.115:0/3546394998 <== mon.2 v2:172.21.15.115:3300/0 4 ==== mgrmap(e 16) v1 ==== 125136+0+0 (secure 0 0 0) 0x7f5b08019020 con 0x7f5b100fc120 2022-05-13T23:13:33.702 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:13:33.694+0000 7f5af7fff700 1 --2- 172.21.15.115:0/3546394998 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f5af8096ce0 0x7f5af8099190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:13:33.702 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:13:33.694+0000 7f5af7fff700 1 -- 172.21.15.115:0/3546394998 <== mon.2 v2:172.21.15.115:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 7123+0+0 (secure 0 0 0) 0x7f5b080be320 con 0x7f5b100fc120 2022-05-13T23:13:33.702 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:13:33.694+0000 7f5b0f7fe700 1 --2- 172.21.15.115:0/3546394998 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f5af8096ce0 0x7f5af8099190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:13:33.702 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:13:33.695+0000 7f5b0f7fe700 1 --2- 172.21.15.115:0/3546394998 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f5af8096ce0 0x7f5af8099190 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f5b00004430 tx=0x7f5b00004d10 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-05-13T23:13:33.703 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:13:33.698+0000 7f5af7fff700 1 -- 172.21.15.115:0/3546394998 <== mon.2 v2:172.21.15.115:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+165215 (secure 0 0 0) 0x7f5b0804a570 con 0x7f5b100fc120 2022-05-13T23:13:33.879 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:13:33.877+0000 7f5b15dba700 1 -- 172.21.15.115:0/3546394998 --> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] -- mgr_command(tid 0: {"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi002=a;smithi002=d;smithi002=g;smithi002=j;smithi047=b;smithi047=e;smithi047=h;smithi047=k;smithi115=c;smithi115=f;smithi115=i;smithi115=l", "target": ["mon-mgr", ""]}) v1 -- 0x7f5b100651e0 con 0x7f5af8096ce0 2022-05-13T23:13:33.888 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:13:33.886+0000 7f5af7fff700 1 -- 172.21.15.115:0/3546394998 <== mgr.14148 v2:172.21.15.2:6800/2788339627 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+28 (secure 0 0 0) 0x7f5b100651e0 con 0x7f5af8096ce0 2022-05-13T23:13:33.888 INFO:teuthology.orchestra.run.smithi115.stdout:Scheduled mds.all update... 2022-05-13T23:13:33.890 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:13:33.888+0000 7f5b15dba700 1 -- 172.21.15.115:0/3546394998 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f5af8096ce0 msgr2=0x7f5af8099190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:13:33.890 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:13:33.888+0000 7f5b15dba700 1 --2- 172.21.15.115:0/3546394998 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f5af8096ce0 0x7f5af8099190 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f5b00004430 tx=0x7f5b00004d10 comp rx=0 tx=0).stop 2022-05-13T23:13:33.890 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:13:33.888+0000 7f5b15dba700 1 -- 172.21.15.115:0/3546394998 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f5b100fc120 msgr2=0x7f5b10113940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:13:33.891 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:13:33.888+0000 7f5b15dba700 1 --2- 172.21.15.115:0/3546394998 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f5b100fc120 0x7f5b10113940 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f5b080041d0 tx=0x7f5b08009f60 comp rx=0 tx=0).stop 2022-05-13T23:13:33.891 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:13:33.888+0000 7f5b15dba700 1 -- 172.21.15.115:0/3546394998 shutdown_connections 2022-05-13T23:13:33.891 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:13:33.888+0000 7f5b15dba700 1 --2- 172.21.15.115:0/3546394998 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f5b100fc120 0x7f5b10113940 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:13:33.891 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:13:33.888+0000 7f5b15dba700 1 --2- 172.21.15.115:0/3546394998 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f5af8096ce0 0x7f5af8099190 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:13:33.891 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:13:33.888+0000 7f5b15dba700 1 --2- 172.21.15.115:0/3546394998 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5b100fa4f0 0x7f5b1010c370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:13:33.892 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:13:33.888+0000 7f5b15dba700 1 --2- 172.21.15.115:0/3546394998 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5b100fb790 0x7f5b1010c8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:13:33.892 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:13:33.888+0000 7f5b15dba700 1 -- 172.21.15.115:0/3546394998 >> 172.21.15.115:0/3546394998 conn(0x7f5b100f5ce0 msgr2=0x7f5b100649d0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-05-13T23:13:33.892 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:13:33.888+0000 7f5b15dba700 1 -- 172.21.15.115:0/3546394998 shutdown_connections 2022-05-13T23:13:33.892 INFO:teuthology.orchestra.run.smithi115.stderr:2022-05-13T23:13:33.888+0000 7f5b15dba700 1 -- 172.21.15.115:0/3546394998 wait complete. 2022-05-13T23:13:34.342 DEBUG:teuthology.orchestra.run.smithi002:mds.a> sudo journalctl -f -n 0 -u ceph-16cf5f88-d311-11ec-be40-001a4aab830c@mds.a.service 2022-05-13T23:13:34.349 DEBUG:teuthology.orchestra.run.smithi002:mds.d> sudo journalctl -f -n 0 -u ceph-16cf5f88-d311-11ec-be40-001a4aab830c@mds.d.service 2022-05-13T23:13:34.395 DEBUG:teuthology.orchestra.run.smithi002:mds.g> sudo journalctl -f -n 0 -u ceph-16cf5f88-d311-11ec-be40-001a4aab830c@mds.g.service 2022-05-13T23:13:34.400 DEBUG:teuthology.orchestra.run.smithi002:mds.j> sudo journalctl -f -n 0 -u ceph-16cf5f88-d311-11ec-be40-001a4aab830c@mds.j.service 2022-05-13T23:13:34.439 INFO:journalctl@ceph.mds.a.smithi002.stdout:-- Logs begin at Fri 2022-05-13 22:48:11 UTC. -- 2022-05-13T23:13:34.442 DEBUG:teuthology.orchestra.run.smithi047:mds.b> sudo journalctl -f -n 0 -u ceph-16cf5f88-d311-11ec-be40-001a4aab830c@mds.b.service 2022-05-13T23:13:34.446 DEBUG:teuthology.orchestra.run.smithi047:mds.e> sudo journalctl -f -n 0 -u ceph-16cf5f88-d311-11ec-be40-001a4aab830c@mds.e.service 2022-05-13T23:13:34.471 INFO:journalctl@ceph.mds.b.smithi047.stdout:-- Logs begin at Fri 2022-05-13 22:44:47 UTC. -- 2022-05-13T23:13:34.472 DEBUG:teuthology.orchestra.run.smithi047:mds.h> sudo journalctl -f -n 0 -u ceph-16cf5f88-d311-11ec-be40-001a4aab830c@mds.h.service 2022-05-13T23:13:34.480 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:13:34 smithi115 ceph-mon[78593]: pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 74 MiB used, 1.0 TiB / 1.0 TiB avail 2022-05-13T23:13:34.481 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:13:34 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:13:34.481 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:13:34 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-13T23:13:34.481 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:13:34 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-13T23:13:34.481 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:13:34 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-13T23:13:34.482 INFO:journalctl@ceph.mds.d.smithi002.stdout:-- Logs begin at Fri 2022-05-13 22:48:11 UTC. -- 2022-05-13T23:13:34.495 INFO:journalctl@ceph.mds.g.smithi002.stdout:-- Logs begin at Fri 2022-05-13 22:48:11 UTC. -- 2022-05-13T23:13:34.497 INFO:journalctl@ceph.mds.e.smithi047.stdout:-- Logs begin at Fri 2022-05-13 22:44:47 UTC. -- 2022-05-13T23:13:34.498 DEBUG:teuthology.orchestra.run.smithi047:mds.k> sudo journalctl -f -n 0 -u ceph-16cf5f88-d311-11ec-be40-001a4aab830c@mds.k.service 2022-05-13T23:13:34.522 INFO:journalctl@ceph.mds.h.smithi047.stdout:-- Logs begin at Fri 2022-05-13 22:44:47 UTC. -- 2022-05-13T23:13:34.523 DEBUG:teuthology.orchestra.run.smithi115:mds.c> sudo journalctl -f -n 0 -u ceph-16cf5f88-d311-11ec-be40-001a4aab830c@mds.c.service 2022-05-13T23:13:34.525 DEBUG:teuthology.orchestra.run.smithi115:mds.f> sudo journalctl -f -n 0 -u ceph-16cf5f88-d311-11ec-be40-001a4aab830c@mds.f.service 2022-05-13T23:13:34.531 INFO:journalctl@ceph.mds.j.smithi002.stdout:-- Logs begin at Fri 2022-05-13 22:48:11 UTC. -- 2022-05-13T23:13:34.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:13:34 smithi047 ceph-mon[77147]: pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 74 MiB used, 1.0 TiB / 1.0 TiB avail 2022-05-13T23:13:34.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:13:34 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:13:34.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:13:34 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-13T23:13:34.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:13:34 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-13T23:13:34.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:13:34 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-13T23:13:34.547 INFO:journalctl@ceph.mds.k.smithi047.stdout:-- Logs begin at Fri 2022-05-13 22:44:47 UTC. -- 2022-05-13T23:13:34.550 INFO:journalctl@ceph.mds.c.smithi115.stdout:-- Logs begin at Fri 2022-05-13 22:45:29 UTC. -- 2022-05-13T23:13:34.551 DEBUG:teuthology.orchestra.run.smithi115:mds.i> sudo journalctl -f -n 0 -u ceph-16cf5f88-d311-11ec-be40-001a4aab830c@mds.i.service 2022-05-13T23:13:34.565 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:13:34 smithi002 ceph-mon[71816]: pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 74 MiB used, 1.0 TiB / 1.0 TiB avail 2022-05-13T23:13:34.565 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:13:34 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:13:34.565 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:13:34 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-13T23:13:34.565 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:13:34 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-13T23:13:34.566 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:13:34 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-13T23:13:34.574 INFO:journalctl@ceph.mds.f.smithi115.stdout:-- Logs begin at Fri 2022-05-13 22:45:29 UTC. -- 2022-05-13T23:13:34.575 DEBUG:teuthology.orchestra.run.smithi115:mds.l> sudo journalctl -f -n 0 -u ceph-16cf5f88-d311-11ec-be40-001a4aab830c@mds.l.service 2022-05-13T23:13:34.598 INFO:journalctl@ceph.mds.i.smithi115.stdout:-- Logs begin at Fri 2022-05-13 22:45:29 UTC. -- 2022-05-13T23:13:34.599 INFO:tasks.cephadm:Setting up CephFS filesystem(s)... 2022-05-13T23:13:34.599 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-05-13T23:13:34.622 INFO:journalctl@ceph.mds.l.smithi115.stdout:-- Logs begin at Fri 2022-05-13 22:45:29 UTC. -- 2022-05-13T23:13:34.854 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:34.852+0000 7f9ca4d9d700 1 -- 172.21.15.2:0/1788775472 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f9ca00fb490 msgr2=0x7f9ca00fb8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:13:34.854 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:34.852+0000 7f9ca4d9d700 1 --2- 172.21.15.2:0/1788775472 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f9ca00fb490 0x7f9ca00fb8a0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f9c90004870 tx=0x7f9c90009cb0 comp rx=0 tx=0).stop 2022-05-13T23:13:34.854 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:34.852+0000 7f9ca4d9d700 1 -- 172.21.15.2:0/1788775472 shutdown_connections 2022-05-13T23:13:34.855 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:34.852+0000 7f9ca4d9d700 1 --2- 172.21.15.2:0/1788775472 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f9ca00fd0c0 0x7f9ca0103980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:13:34.855 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:34.852+0000 7f9ca4d9d700 1 --2- 172.21.15.2:0/1788775472 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f9ca00fb490 0x7f9ca00fb8a0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:13:34.855 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:34.852+0000 7f9ca4d9d700 1 --2- 172.21.15.2:0/1788775472 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9ca00fc730 0x7f9ca00fcb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:13:34.855 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:34.852+0000 7f9ca4d9d700 1 -- 172.21.15.2:0/1788775472 >> 172.21.15.2:0/1788775472 conn(0x7f9ca00f6c40 msgr2=0x7f9ca00f90b0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-05-13T23:13:34.856 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:34.852+0000 7f9ca4d9d700 1 -- 172.21.15.2:0/1788775472 shutdown_connections 2022-05-13T23:13:34.856 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:34.852+0000 7f9ca4d9d700 1 -- 172.21.15.2:0/1788775472 wait complete. 2022-05-13T23:13:34.859 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:34.853+0000 7f9ca4d9d700 1 Processor -- start 2022-05-13T23:13:34.859 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:34.853+0000 7f9ca4d9d700 1 -- start start 2022-05-13T23:13:34.859 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:34.854+0000 7f9ca4d9d700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9ca00fb490 0x7f9ca00701d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:13:34.859 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:34.854+0000 7f9ca4d9d700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f9ca00fc730 0x7f9ca006e830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:13:34.860 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:34.854+0000 7f9ca4d9d700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f9ca00fd0c0 0x7f9ca006ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:13:34.860 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:34.854+0000 7f9ca4d9d700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ca006f480 con 0x7f9ca00fb490 2022-05-13T23:13:34.860 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:34.854+0000 7f9ca4d9d700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ca006f5f0 con 0x7f9ca00fd0c0 2022-05-13T23:13:34.860 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:34.854+0000 7f9ca4d9d700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ca01163e0 con 0x7f9ca00fc730 2022-05-13T23:13:34.861 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:34.854+0000 7f9c9effd700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f9ca00fc730 0x7f9ca006e830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:13:34.861 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:34.854+0000 7f9c9f7fe700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9ca00fb490 0x7f9ca00701d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:13:34.861 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:34.854+0000 7f9c9effd700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f9ca00fc730 0x7f9ca006e830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.2:59606/0 (socket says 172.21.15.2:59606) 2022-05-13T23:13:34.861 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:34.854+0000 7f9c9effd700 1 -- 172.21.15.2:0/516290896 learned_addr learned my addr 172.21.15.2:0/516290896 (peer_addr_for_me v2:172.21.15.2:0/0) 2022-05-13T23:13:34.861 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:34.854+0000 7f9c9ffff700 1 --2- 172.21.15.2:0/516290896 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f9ca00fd0c0 0x7f9ca006ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:13:34.862 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:34.854+0000 7f9c9effd700 1 -- 172.21.15.2:0/516290896 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f9ca00fd0c0 msgr2=0x7f9ca006ed70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:13:34.862 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:34.854+0000 7f9c9effd700 1 --2- 172.21.15.2:0/516290896 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f9ca00fd0c0 0x7f9ca006ed70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:13:34.862 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:34.854+0000 7f9c9effd700 1 -- 172.21.15.2:0/516290896 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9ca00fb490 msgr2=0x7f9ca00701d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:13:34.862 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:34.855+0000 7f9c9effd700 1 --2- 172.21.15.2:0/516290896 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9ca00fb490 0x7f9ca00701d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:13:34.863 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:34.855+0000 7f9c9effd700 1 -- 172.21.15.2:0/516290896 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c90005040 con 0x7f9ca00fc730 2022-05-13T23:13:34.863 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:34.855+0000 7f9c9f7fe700 1 --2- 172.21.15.2:0/516290896 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9ca00fb490 0x7f9ca00701d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-05-13T23:13:34.863 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:34.855+0000 7f9c9effd700 1 --2- 172.21.15.2:0/516290896 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f9ca00fc730 0x7f9ca006e830 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f9c9400e970 tx=0x7f9c9400b040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-05-13T23:13:34.863 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:34.855+0000 7f9c87fff700 1 -- 172.21.15.2:0/516290896 <== mon.2 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c94004660 con 0x7f9ca00fc730 2022-05-13T23:13:34.863 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:34.855+0000 7f9ca4d9d700 1 -- 172.21.15.2:0/516290896 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9ca0116580 con 0x7f9ca00fc730 2022-05-13T23:13:34.864 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:34.855+0000 7f9c87fff700 1 -- 172.21.15.2:0/516290896 <== mon.2 v2:172.21.15.115:3300/0 2 ==== config(26 keys) v1 ==== 1087+0+0 (secure 0 0 0) 0x7f9c9400f040 con 0x7f9ca00fc730 2022-05-13T23:13:34.864 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:34.855+0000 7f9c87fff700 1 -- 172.21.15.2:0/516290896 <== mon.2 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c9400c730 con 0x7f9ca00fc730 2022-05-13T23:13:34.864 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:34.855+0000 7f9ca4d9d700 1 -- 172.21.15.2:0/516290896 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ca0116ae0 con 0x7f9ca00fc730 2022-05-13T23:13:34.864 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:34.856+0000 7f9ca4d9d700 1 -- 172.21.15.2:0/516290896 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9ca0106cf0 con 0x7f9ca00fc730 2022-05-13T23:13:34.865 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:34.858+0000 7f9c87fff700 1 -- 172.21.15.2:0/516290896 <== mon.2 v2:172.21.15.115:3300/0 4 ==== mgrmap(e 16) v1 ==== 125136+0+0 (secure 0 0 0) 0x7f9c9401b030 con 0x7f9ca00fc730 2022-05-13T23:13:34.866 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:34.859+0000 7f9c87fff700 1 --2- 172.21.15.2:0/516290896 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f9c88096ce0 0x7f9c88099190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:13:34.866 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:34.860+0000 7f9c87fff700 1 -- 172.21.15.2:0/516290896 <== mon.2 v2:172.21.15.115:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 7123+0+0 (secure 0 0 0) 0x7f9c940be850 con 0x7f9ca00fc730 2022-05-13T23:13:34.866 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:34.860+0000 7f9c9f7fe700 1 --2- 172.21.15.2:0/516290896 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f9c88096ce0 0x7f9c88099190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:13:34.866 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:34.860+0000 7f9c9f7fe700 1 --2- 172.21.15.2:0/516290896 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f9c88096ce0 0x7f9c88099190 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f9c90009da0 tx=0x7f9c90009cb0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-05-13T23:13:34.867 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:34.863+0000 7f9c87fff700 1 -- 172.21.15.2:0/516290896 <== mon.2 v2:172.21.15.115:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+165215 (secure 0 0 0) 0x7f9c9404b570 con 0x7f9ca00fc730 2022-05-13T23:13:35.057 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:35.055+0000 7f9ca4d9d700 1 -- 172.21.15.2:0/516290896 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9ca0102660 con 0x7f9ca00fc730 2022-05-13T23:13:35.058 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:35.057+0000 7f9c87fff700 1 -- 172.21.15.2:0/516290896 <== mon.2 v2:172.21.15.115:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v79) v1 ==== 74+0+18042 (secure 0 0 0) 0x7f9c9408f970 con 0x7f9ca00fc730 2022-05-13T23:13:35.059 INFO:teuthology.orchestra.run.smithi002.stdout: 2022-05-13T23:13:35.059 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":79,"fsid":"16cf5f88-d311-11ec-be40-001a4aab830c","created":"2022-05-13T23:06:00.765453+0000","modified":"2022-05-13T23:13:21.942299+0000","last_up_change":"2022-05-13T23:13:20.935982+0000","last_in_change":"2022-05-13T23:13:02.796000+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2022-05-13T23:09:32.000644+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"11e700e8-f3e8-4903-b52c-db354b4395bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6802","nonce":1174843704},{"type":"v1","addr":"172.21.15.2:6803","nonce":1174843704}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6804","nonce":1174843704},{"type":"v1","addr":"172.21.15.2:6805","nonce":1174843704}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6808","nonce":1174843704},{"type":"v1","addr":"172.21.15.2:6809","nonce":1174843704}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6806","nonce":1174843704},{"type":"v1","addr":"172.21.15.2:6807","nonce":1174843704}]},"public_addr":"172.21.15.2:6803/1174843704","cluster_addr":"172.21.15.2:6805/1174843704","heartbeat_back_addr":"172.21.15.2:6809/1174843704","heartbeat_front_addr":"172.21.15.2:6807/1174843704","state":["exists","up"]},{"osd":1,"uuid":"eabfa3cf-f2a3-4995-a7de-f6d8ba02ee6f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6800","nonce":3853826948},{"type":"v1","addr":"172.21.15.47:6801","nonce":3853826948}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6802","nonce":3853826948},{"type":"v1","addr":"172.21.15.47:6803","nonce":3853826948}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6806","nonce":3853826948},{"type":"v1","addr":"172.21.15.47:6807","nonce":3853826948}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6804","nonce":3853826948},{"type":"v1","addr":"172.21.15.47:6805","nonce":3853826948}]},"public_addr":"172.21.15.47:6801/3853826948","cluster_addr":"172.21.15.47:6803/3853826948","heartbeat_back_addr":"172.21.15.47:6807/3853826948","heartbeat_front_addr":"172.21.15.47:6805/3853826948","state":["exists","up"]},{"osd":2,"uuid":"153c5bd5-8200-47a7-a135-8a95b6752cd2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6800","nonce":2562256956},{"type":"v1","addr":"172.21.15.115:6801","nonce":2562256956}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6802","nonce":2562256956},{"type":"v1","addr":"172.21.15.115:6803","nonce":2562256956}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6806","nonce":2562256956},{"type":"v1","addr":"172.21.15.115:6807","nonce":2562256956}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6804","nonce":2562256956},{"type":"v1","addr":"172.21.15.115:6805","nonce":2562256956}]},"public_addr":"172.21.15.115:6801/2562256956","cluster_addr":"172.21.15.115:6803/2562256956","heartbeat_back_addr":"172.21.15.115:6807/2562256956","heartbeat_front_addr":"172.21.15.115:6805/2562256956","state":["exists","up"]},{"osd":3,"uuid":"6ca9c602-5936-48da-a861-c437e6a33765","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6810","nonce":1099909411},{"type":"v1","addr":"172.21.15.2:6811","nonce":1099909411}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6812","nonce":1099909411},{"type":"v1","addr":"172.21.15.2:6813","nonce":1099909411}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6816","nonce":1099909411},{"type":"v1","addr":"172.21.15.2:6817","nonce":1099909411}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6814","nonce":1099909411},{"type":"v1","addr":"172.21.15.2:6815","nonce":1099909411}]},"public_addr":"172.21.15.2:6811/1099909411","cluster_addr":"172.21.15.2:6813/1099909411","heartbeat_back_addr":"172.21.15.2:6817/1099909411","heartbeat_front_addr":"172.21.15.2:6815/1099909411","state":["exists","up"]},{"osd":4,"uuid":"54d943d9-2610-422a-9e59-4b8212aced31","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6808","nonce":882753861},{"type":"v1","addr":"172.21.15.47:6809","nonce":882753861}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6810","nonce":882753861},{"type":"v1","addr":"172.21.15.47:6811","nonce":882753861}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6814","nonce":882753861},{"type":"v1","addr":"172.21.15.47:6815","nonce":882753861}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6812","nonce":882753861},{"type":"v1","addr":"172.21.15.47:6813","nonce":882753861}]},"public_addr":"172.21.15.47:6809/882753861","cluster_addr":"172.21.15.47:6811/882753861","heartbeat_back_addr":"172.21.15.47:6815/882753861","heartbeat_front_addr":"172.21.15.47:6813/882753861","state":["exists","up"]},{"osd":5,"uuid":"c18566cf-5403-4057-b2ee-aa298e87e8e9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6808","nonce":2122391993},{"type":"v1","addr":"172.21.15.115:6809","nonce":2122391993}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6810","nonce":2122391993},{"type":"v1","addr":"172.21.15.115:6811","nonce":2122391993}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6814","nonce":2122391993},{"type":"v1","addr":"172.21.15.115:6815","nonce":2122391993}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6812","nonce":2122391993},{"type":"v1","addr":"172.21.15.115:6813","nonce":2122391993}]},"public_addr":"172.21.15.115:6809/2122391993","cluster_addr":"172.21.15.115:6811/2122391993","heartbeat_back_addr":"172.21.15.115:6815/2122391993","heartbeat_front_addr":"172.21.15.115:6813/2122391993","state":["exists","up"]},{"osd":6,"uuid":"38ec7b6b-30f4-4d97-ad7e-e160ae59c843","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6818","nonce":1892198790},{"type":"v1","addr":"172.21.15.2:6819","nonce":1892198790}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6820","nonce":1892198790},{"type":"v1","addr":"172.21.15.2:6821","nonce":1892198790}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6824","nonce":1892198790},{"type":"v1","addr":"172.21.15.2:6825","nonce":1892198790}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6822","nonce":1892198790},{"type":"v1","addr":"172.21.15.2:6823","nonce":1892198790}]},"public_addr":"172.21.15.2:6819/1892198790","cluster_addr":"172.21.15.2:6821/1892198790","heartbeat_back_addr":"172.21.15.2:6825/1892198790","heartbeat_front_addr":"172.21.15.2:6823/1892198790","state":["exists","up"]},{"osd":7,"uuid":"e799b560-c61e-4369-b08b-45e5a6af37c0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6816","nonce":2812542825},{"type":"v1","addr":"172.21.15.47:6817","nonce":2812542825}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6818","nonce":2812542825},{"type":"v1","addr":"172.21.15.47:6819","nonce":2812542825}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6822","nonce":2812542825},{"type":"v1","addr":"172.21.15.47:6823","nonce":2812542825}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6820","nonce":2812542825},{"type":"v1","addr":"172.21.15.47:6821","nonce":2812542825}]},"public_addr":"172.21.15.47:6817/2812542825","cluster_addr":"172.21.15.47:6819/2812542825","heartbeat_back_addr":"172.21.15.47:6823/2812542825","heartbeat_front_addr":"172.21.15.47:6821/2812542825","state":["exists","up"]},{"osd":8,"uuid":"a58f2c0e-e487-41cf-8d2e-8c937cac9ebc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":58,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6816","nonce":3959632033},{"type":"v1","addr":"172.21.15.115:6817","nonce":3959632033}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6818","nonce":3959632033},{"type":"v1","addr":"172.21.15.115:6819","nonce":3959632033}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6822","nonce":3959632033},{"type":"v1","addr":"172.21.15.115:6823","nonce":3959632033}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6820","nonce":3959632033},{"type":"v1","addr":"172.21.15.115:6821","nonce":3959632033}]},"public_addr":"172.21.15.115:6817/3959632033","cluster_addr":"172.21.15.115:6819/3959632033","heartbeat_back_addr":"172.21.15.115:6823/3959632033","heartbeat_front_addr":"172.21.15.115:6821/3959632033","state":["exists","up"]},{"osd":9,"uuid":"12cead96-f726-4f7a-ab84-eb191b778a5c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":66,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6826","nonce":3004162255},{"type":"v1","addr":"172.21.15.2:6827","nonce":3004162255}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6828","nonce":3004162255},{"type":"v1","addr":"172.21.15.2:6829","nonce":3004162255}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6832","nonce":3004162255},{"type":"v1","addr":"172.21.15.2:6833","nonce":3004162255}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6830","nonce":3004162255},{"type":"v1","addr":"172.21.15.2:6831","nonce":3004162255}]},"public_addr":"172.21.15.2:6827/3004162255","cluster_addr":"172.21.15.2:6829/3004162255","heartbeat_back_addr":"172.21.15.2:6833/3004162255","heartbeat_front_addr":"172.21.15.2:6831/3004162255","state":["exists","up"]},{"osd":10,"uuid":"4f648118-7855-48d6-86d5-44eed5295f0c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":71,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6824","nonce":3443413998},{"type":"v1","addr":"172.21.15.47:6825","nonce":3443413998}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6826","nonce":3443413998},{"type":"v1","addr":"172.21.15.47:6827","nonce":3443413998}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6830","nonce":3443413998},{"type":"v1","addr":"172.21.15.47:6831","nonce":3443413998}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6828","nonce":3443413998},{"type":"v1","addr":"172.21.15.47:6829","nonce":3443413998}]},"public_addr":"172.21.15.47:6825/3443413998","cluster_addr":"172.21.15.47:6827/3443413998","heartbeat_back_addr":"172.21.15.47:6831/3443413998","heartbeat_front_addr":"172.21.15.47:6829/3443413998","state":["exists","up"]},{"osd":11,"uuid":"916541e0-fb28-46d5-9743-801c54c5a6d1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":78,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6824","nonce":1758718209},{"type":"v1","addr":"172.21.15.115:6825","nonce":1758718209}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6826","nonce":1758718209},{"type":"v1","addr":"172.21.15.115:6827","nonce":1758718209}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6830","nonce":1758718209},{"type":"v1","addr":"172.21.15.115:6831","nonce":1758718209}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6828","nonce":1758718209},{"type":"v1","addr":"172.21.15.115:6829","nonce":1758718209}]},"public_addr":"172.21.15.115:6825/1758718209","cluster_addr":"172.21.15.115:6827/1758718209","heartbeat_back_addr":"172.21.15.115:6831/1758718209","heartbeat_front_addr":"172.21.15.115:6829/1758718209","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-05-13T23:08:43.757857+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-05-13T23:09:06.537139+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-05-13T23:09:28.524837+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-05-13T23:10:02.057583+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-05-13T23:10:26.210648+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-05-13T23:10:47.885853+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-05-13T23:11:19.141865+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-05-13T23:11:42.376065+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-05-13T23:12:04.175870+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-05-13T23:12:34.695347+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-05-13T23:12:57.762613+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-05-13T23:13:19.114284+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.2:0/477249925":"2022-05-14T23:07:04.986087+0000","172.21.15.2:6800/3226099420":"2022-05-14T23:07:04.986087+0000","172.21.15.2:0/2214093915":"2022-05-14T23:07:04.986087+0000","172.21.15.2:0/1039403355":"2022-05-14T23:07:04.986087+0000","172.21.15.2:0/22091675":"2022-05-14T23:06:36.304108+0000","172.21.15.2:6800/1255134634":"2022-05-14T23:06:36.304108+0000","172.21.15.2:6801/3226099420":"2022-05-14T23:07:04.986087+0000","172.21.15.2:0/3768677276":"2022-05-14T23:06:36.304108+0000","172.21.15.2:6801/1255134634":"2022-05-14T23:06:36.304108+0000","172.21.15.2:0/407841268":"2022-05-14T23:06:36.304108+0000"},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2022-05-13T23:13:35.061 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:35.059+0000 7f9ca4d9d700 1 -- 172.21.15.2:0/516290896 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f9c88096ce0 msgr2=0x7f9c88099190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:13:35.061 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:35.059+0000 7f9ca4d9d700 1 --2- 172.21.15.2:0/516290896 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f9c88096ce0 0x7f9c88099190 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f9c90009da0 tx=0x7f9c90009cb0 comp rx=0 tx=0).stop 2022-05-13T23:13:35.061 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:35.059+0000 7f9ca4d9d700 1 -- 172.21.15.2:0/516290896 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f9ca00fc730 msgr2=0x7f9ca006e830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:13:35.062 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:35.059+0000 7f9ca4d9d700 1 --2- 172.21.15.2:0/516290896 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f9ca00fc730 0x7f9ca006e830 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f9c9400e970 tx=0x7f9c9400b040 comp rx=0 tx=0).stop 2022-05-13T23:13:35.062 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:35.059+0000 7f9ca4d9d700 1 -- 172.21.15.2:0/516290896 shutdown_connections 2022-05-13T23:13:35.062 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:35.059+0000 7f9ca4d9d700 1 --2- 172.21.15.2:0/516290896 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f9ca00fd0c0 0x7f9ca006ed70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:13:35.062 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:35.059+0000 7f9ca4d9d700 1 --2- 172.21.15.2:0/516290896 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f9c88096ce0 0x7f9c88099190 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:13:35.062 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:35.059+0000 7f9ca4d9d700 1 --2- 172.21.15.2:0/516290896 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9ca00fb490 0x7f9ca00701d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:13:35.063 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:35.059+0000 7f9ca4d9d700 1 --2- 172.21.15.2:0/516290896 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f9ca00fc730 0x7f9ca006e830 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:13:35.063 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:35.059+0000 7f9ca4d9d700 1 -- 172.21.15.2:0/516290896 >> 172.21.15.2:0/516290896 conn(0x7f9ca00f6c40 msgr2=0x7f9ca00ff960 unknown :-1 s=STATE_NONE l=0).mark_down 2022-05-13T23:13:35.063 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:35.060+0000 7f9ca4d9d700 1 -- 172.21.15.2:0/516290896 shutdown_connections 2022-05-13T23:13:35.063 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:35.060+0000 7f9ca4d9d700 1 -- 172.21.15.2:0/516290896 wait complete. 2022-05-13T23:13:35.161 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2022-05-13T23:09:32.000644+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}] 2022-05-13T23:13:35.162 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2022-05-13T23:13:35.399 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:35.397+0000 7fe34bfff700 1 -- 172.21.15.2:0/716044059 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe34c0fc740 msgr2=0x7fe34c0fcb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:13:35.399 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:35.397+0000 7fe34bfff700 1 --2- 172.21.15.2:0/716044059 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe34c0fc740 0x7fe34c0fcb90 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fe340004870 tx=0x7fe340009cb0 comp rx=0 tx=0).stop 2022-05-13T23:13:35.399 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:35.397+0000 7fe34bfff700 1 -- 172.21.15.2:0/716044059 shutdown_connections 2022-05-13T23:13:35.400 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:35.397+0000 7fe34bfff700 1 --2- 172.21.15.2:0/716044059 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe34c0fd0d0 0x7fe34c103990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:13:35.400 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:35.397+0000 7fe34bfff700 1 --2- 172.21.15.2:0/716044059 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe34c0fb4a0 0x7fe34c0fb8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:13:35.400 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:35.397+0000 7fe34bfff700 1 --2- 172.21.15.2:0/716044059 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe34c0fc740 0x7fe34c0fcb90 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:13:35.400 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:35.397+0000 7fe34bfff700 1 -- 172.21.15.2:0/716044059 >> 172.21.15.2:0/716044059 conn(0x7fe34c0f6c50 msgr2=0x7fe34c0f90c0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-05-13T23:13:35.400 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:35.397+0000 7fe34bfff700 1 -- 172.21.15.2:0/716044059 shutdown_connections 2022-05-13T23:13:35.401 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:35.397+0000 7fe34bfff700 1 -- 172.21.15.2:0/716044059 wait complete. 2022-05-13T23:13:35.403 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:35.398+0000 7fe34bfff700 1 Processor -- start 2022-05-13T23:13:35.403 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:35.398+0000 7fe34bfff700 1 -- start start 2022-05-13T23:13:35.403 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:35.398+0000 7fe34bfff700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe34c0fb4a0 0x7fe34c10e4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:13:35.403 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:35.398+0000 7fe34bfff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe34c0fc740 0x7fe34c10ea20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:13:35.404 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:35.398+0000 7fe34bfff700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe34c0fd0d0 0x7fe34c115ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:13:35.404 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:35.398+0000 7fe34bfff700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe34c115ff0 con 0x7fe34c0fc740 2022-05-13T23:13:35.404 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:35.398+0000 7fe34bfff700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe34c116160 con 0x7fe34c0fd0d0 2022-05-13T23:13:35.404 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:35.398+0000 7fe34bfff700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe34c116320 con 0x7fe34c0fb4a0 2022-05-13T23:13:35.405 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:35.399+0000 7fe34a7fc700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe34c0fc740 0x7fe34c10ea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:13:35.405 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:35.399+0000 7fe34affd700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe34c0fb4a0 0x7fe34c10e4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:13:35.405 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:35.399+0000 7fe34a7fc700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe34c0fc740 0x7fe34c10ea20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:35382/0 (socket says 172.21.15.2:35382) 2022-05-13T23:13:35.405 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:35.399+0000 7fe34a7fc700 1 -- 172.21.15.2:0/3693854383 learned_addr learned my addr 172.21.15.2:0/3693854383 (peer_addr_for_me v2:172.21.15.2:0/0) 2022-05-13T23:13:35.405 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:35.399+0000 7fe34b7fe700 1 --2- 172.21.15.2:0/3693854383 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe34c0fd0d0 0x7fe34c115ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:13:35.406 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:35.399+0000 7fe34a7fc700 1 -- 172.21.15.2:0/3693854383 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe34c0fb4a0 msgr2=0x7fe34c10e4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:13:35.406 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:35.399+0000 7fe34a7fc700 1 --2- 172.21.15.2:0/3693854383 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe34c0fb4a0 0x7fe34c10e4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:13:35.406 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:35.399+0000 7fe34a7fc700 1 -- 172.21.15.2:0/3693854383 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe34c0fd0d0 msgr2=0x7fe34c115ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:13:35.409 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:35.399+0000 7fe34a7fc700 1 --2- 172.21.15.2:0/3693854383 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe34c0fd0d0 0x7fe34c115ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:13:35.410 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:35.399+0000 7fe34a7fc700 1 -- 172.21.15.2:0/3693854383 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe33c005040 con 0x7fe34c0fc740 2022-05-13T23:13:35.410 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:35.399+0000 7fe34b7fe700 1 --2- 172.21.15.2:0/3693854383 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe34c0fd0d0 0x7fe34c115ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-05-13T23:13:35.410 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:35.400+0000 7fe34a7fc700 1 --2- 172.21.15.2:0/3693854383 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe34c0fc740 0x7fe34c10ea20 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fe340004ee0 tx=0x7fe340009cb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-05-13T23:13:35.410 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:35.400+0000 7fe3337fe700 1 -- 172.21.15.2:0/3693854383 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe34000e040 con 0x7fe34c0fc740 2022-05-13T23:13:35.411 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:35.400+0000 7fe34bfff700 1 -- 172.21.15.2:0/3693854383 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe340005040 con 0x7fe34c0fc740 2022-05-13T23:13:35.411 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:35.400+0000 7fe3337fe700 1 -- 172.21.15.2:0/3693854383 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1087+0+0 (secure 0 0 0) 0x7fe340004200 con 0x7fe34c0fc740 2022-05-13T23:13:35.411 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:35.400+0000 7fe3337fe700 1 -- 172.21.15.2:0/3693854383 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe34001cab0 con 0x7fe34c0fc740 2022-05-13T23:13:35.411 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:35.400+0000 7fe34bfff700 1 -- 172.21.15.2:0/3693854383 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe34c1168a0 con 0x7fe34c0fc740 2022-05-13T23:13:35.411 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:35.402+0000 7fe34bfff700 1 -- 172.21.15.2:0/3693854383 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe34c106d40 con 0x7fe34c0fc740 2022-05-13T23:13:35.412 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:35.404+0000 7fe3337fe700 1 -- 172.21.15.2:0/3693854383 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 16) v1 ==== 125136+0+0 (secure 0 0 0) 0x7fe340009240 con 0x7fe34c0fc740 2022-05-13T23:13:35.413 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:35.405+0000 7fe3337fe700 1 --2- 172.21.15.2:0/3693854383 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7fe334096ce0 0x7fe334099190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:13:35.413 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:35.405+0000 7fe3337fe700 1 -- 172.21.15.2:0/3693854383 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 7123+0+0 (secure 0 0 0) 0x7fe3400c0140 con 0x7fe34c0fc740 2022-05-13T23:13:35.413 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:35.405+0000 7fe34affd700 1 --2- 172.21.15.2:0/3693854383 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7fe334096ce0 0x7fe334099190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:13:35.414 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:35.406+0000 7fe34affd700 1 --2- 172.21.15.2:0/3693854383 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7fe334096ce0 0x7fe334099190 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fe34c0fc5a0 tx=0x7fe33c00b040 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-05-13T23:13:35.415 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:35.412+0000 7fe3337fe700 1 -- 172.21.15.2:0/3693854383 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+165215 (secure 0 0 0) 0x7fe34008d7d0 con 0x7fe34c0fc740 2022-05-13T23:13:35.572 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:13:35 smithi002 ceph-mon[71816]: from='client.24452 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi002=a;smithi002=d;smithi002=g;smithi002=j;smithi047=b;smithi047=e;smithi047=h;smithi047=k;smithi115=c;smithi115=f;smithi115=i;smithi115=l", "target": ["mon-mgr", ""]}]: dispatch 2022-05-13T23:13:35.573 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:13:35 smithi002 ceph-mon[71816]: Saving service mds.all spec with placement smithi002=a;smithi002=d;smithi002=g;smithi002=j;smithi047=b;smithi047=e;smithi047=h;smithi047=k;smithi115=c;smithi115=f;smithi115=i;smithi115=l;count:12 2022-05-13T23:13:35.573 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:13:35 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:13:35.573 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:13:35 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:13:35.573 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:13:35 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2022-05-13T23:13:35.574 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:13:35 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2022-05-13T23:13:35.574 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:13:35 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-13T23:13:35.574 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:13:35 smithi002 ceph-mon[71816]: from='client.? 172.21.15.2:0/516290896' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2022-05-13T23:13:35.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:13:35 smithi115 ceph-mon[78593]: from='client.24452 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi002=a;smithi002=d;smithi002=g;smithi002=j;smithi047=b;smithi047=e;smithi047=h;smithi047=k;smithi115=c;smithi115=f;smithi115=i;smithi115=l", "target": ["mon-mgr", ""]}]: dispatch 2022-05-13T23:13:35.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:13:35 smithi115 ceph-mon[78593]: Saving service mds.all spec with placement smithi002=a;smithi002=d;smithi002=g;smithi002=j;smithi047=b;smithi047=e;smithi047=h;smithi047=k;smithi115=c;smithi115=f;smithi115=i;smithi115=l;count:12 2022-05-13T23:13:35.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:13:35 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:13:35.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:13:35 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:13:35.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:13:35 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2022-05-13T23:13:35.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:13:35 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2022-05-13T23:13:35.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:13:35 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-13T23:13:35.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:13:35 smithi115 ceph-mon[78593]: from='client.? 172.21.15.2:0/516290896' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2022-05-13T23:13:35.620 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:35.619+0000 7fe34bfff700 1 -- 172.21.15.2:0/3693854383 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fe34c02a490 con 0x7fe34c0fc740 2022-05-13T23:13:35.620 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:35.619+0000 7fe3337fe700 1 -- 172.21.15.2:0/3693854383 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v79) v1 ==== 93+0+10 (secure 0 0 0) 0x7fe340090df0 con 0x7fe34c0fc740 2022-05-13T23:13:35.621 INFO:teuthology.orchestra.run.smithi002.stdout:pg_num: 1 2022-05-13T23:13:35.625 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:35.623+0000 7fe34bfff700 1 -- 172.21.15.2:0/3693854383 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7fe334096ce0 msgr2=0x7fe334099190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:13:35.625 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:35.623+0000 7fe34bfff700 1 --2- 172.21.15.2:0/3693854383 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7fe334096ce0 0x7fe334099190 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fe34c0fc5a0 tx=0x7fe33c00b040 comp rx=0 tx=0).stop 2022-05-13T23:13:35.625 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:35.623+0000 7fe34bfff700 1 -- 172.21.15.2:0/3693854383 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe34c0fc740 msgr2=0x7fe34c10ea20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:13:35.625 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:35.623+0000 7fe34bfff700 1 --2- 172.21.15.2:0/3693854383 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe34c0fc740 0x7fe34c10ea20 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fe340004ee0 tx=0x7fe340009cb0 comp rx=0 tx=0).stop 2022-05-13T23:13:35.626 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:35.623+0000 7fe34bfff700 1 -- 172.21.15.2:0/3693854383 shutdown_connections 2022-05-13T23:13:35.627 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:35.623+0000 7fe34bfff700 1 --2- 172.21.15.2:0/3693854383 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe34c0fd0d0 0x7fe34c115ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:13:35.627 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:35.623+0000 7fe34bfff700 1 --2- 172.21.15.2:0/3693854383 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe34c0fb4a0 0x7fe34c10e4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:13:35.627 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:35.623+0000 7fe34bfff700 1 --2- 172.21.15.2:0/3693854383 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7fe334096ce0 0x7fe334099190 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:13:35.628 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:35.623+0000 7fe34bfff700 1 --2- 172.21.15.2:0/3693854383 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe34c0fc740 0x7fe34c10ea20 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:13:35.628 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:35.623+0000 7fe34bfff700 1 -- 172.21.15.2:0/3693854383 >> 172.21.15.2:0/3693854383 conn(0x7fe34c0f6c50 msgr2=0x7fe34c102190 unknown :-1 s=STATE_NONE l=0).mark_down 2022-05-13T23:13:35.628 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:35.623+0000 7fe34bfff700 1 -- 172.21.15.2:0/3693854383 shutdown_connections 2022-05-13T23:13:35.628 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:35.623+0000 7fe34bfff700 1 -- 172.21.15.2:0/3693854383 wait complete. 2022-05-13T23:13:35.655 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-05-13T23:13:35.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:13:35 smithi047 ceph-mon[77147]: from='client.24452 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi002=a;smithi002=d;smithi002=g;smithi002=j;smithi047=b;smithi047=e;smithi047=h;smithi047=k;smithi115=c;smithi115=f;smithi115=i;smithi115=l", "target": ["mon-mgr", ""]}]: dispatch 2022-05-13T23:13:35.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:13:35 smithi047 ceph-mon[77147]: Saving service mds.all spec with placement smithi002=a;smithi002=d;smithi002=g;smithi002=j;smithi047=b;smithi047=e;smithi047=h;smithi047=k;smithi115=c;smithi115=f;smithi115=i;smithi115=l;count:12 2022-05-13T23:13:35.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:13:35 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:13:35.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:13:35 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:13:35.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:13:35 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2022-05-13T23:13:35.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:13:35 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2022-05-13T23:13:35.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:13:35 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-13T23:13:35.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:13:35 smithi047 ceph-mon[77147]: from='client.? 172.21.15.2:0/516290896' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2022-05-13T23:13:35.882 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:35.880+0000 7fd2f359e700 1 -- 172.21.15.2:0/733619121 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd2f40fc730 msgr2=0x7fd2f40fcb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:13:35.882 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:35.880+0000 7fd2f359e700 1 --2- 172.21.15.2:0/733619121 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd2f40fc730 0x7fd2f40fcb80 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fd2e8004870 tx=0x7fd2e8009cb0 comp rx=0 tx=0).stop 2022-05-13T23:13:35.882 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:35.880+0000 7fd2f359e700 1 -- 172.21.15.2:0/733619121 shutdown_connections 2022-05-13T23:13:35.883 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:35.880+0000 7fd2f359e700 1 --2- 172.21.15.2:0/733619121 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fd2f40fd0c0 0x7fd2f4103980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:13:35.883 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:35.880+0000 7fd2f359e700 1 --2- 172.21.15.2:0/733619121 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd2f40fb490 0x7fd2f40fb8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:13:35.883 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:35.880+0000 7fd2f359e700 1 --2- 172.21.15.2:0/733619121 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd2f40fc730 0x7fd2f40fcb80 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:13:35.883 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:35.880+0000 7fd2f359e700 1 -- 172.21.15.2:0/733619121 >> 172.21.15.2:0/733619121 conn(0x7fd2f40f6c40 msgr2=0x7fd2f40f90b0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-05-13T23:13:35.883 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:35.880+0000 7fd2f359e700 1 -- 172.21.15.2:0/733619121 shutdown_connections 2022-05-13T23:13:35.884 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:35.880+0000 7fd2f359e700 1 -- 172.21.15.2:0/733619121 wait complete. 2022-05-13T23:13:35.886 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:35.881+0000 7fd2f359e700 1 Processor -- start 2022-05-13T23:13:35.886 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:35.881+0000 7fd2f359e700 1 -- start start 2022-05-13T23:13:35.886 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:35.882+0000 7fd2f359e700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd2f40fb490 0x7fd2f4112860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:13:35.886 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:35.882+0000 7fd2f359e700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd2f40fc730 0x7fd2f4112da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:13:35.887 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:35.882+0000 7fd2f359e700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fd2f40fd0c0 0x7fd2f4119e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:13:35.887 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:35.882+0000 7fd2f359e700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2f411a370 con 0x7fd2f40fb490 2022-05-13T23:13:35.887 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:35.882+0000 7fd2f359e700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2f411a4e0 con 0x7fd2f40fc730 2022-05-13T23:13:35.887 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:35.882+0000 7fd2f359e700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2f411a6a0 con 0x7fd2f40fd0c0 2022-05-13T23:13:35.888 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:35.882+0000 7fd2f259c700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd2f40fb490 0x7fd2f4112860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:13:35.888 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:35.882+0000 7fd2f259c700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd2f40fb490 0x7fd2f4112860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:35402/0 (socket says 172.21.15.2:35402) 2022-05-13T23:13:35.889 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:35.882+0000 7fd2f259c700 1 -- 172.21.15.2:0/3775298027 learned_addr learned my addr 172.21.15.2:0/3775298027 (peer_addr_for_me v2:172.21.15.2:0/0) 2022-05-13T23:13:35.889 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:35.882+0000 7fd2f2d9d700 1 --2- 172.21.15.2:0/3775298027 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fd2f40fd0c0 0x7fd2f4119e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:13:35.889 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:35.882+0000 7fd2f1d9b700 1 --2- 172.21.15.2:0/3775298027 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd2f40fc730 0x7fd2f4112da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:13:35.890 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:35.883+0000 7fd2f2d9d700 1 -- 172.21.15.2:0/3775298027 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd2f40fc730 msgr2=0x7fd2f4112da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:13:35.890 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:35.883+0000 7fd2f2d9d700 1 --2- 172.21.15.2:0/3775298027 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd2f40fc730 0x7fd2f4112da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:13:35.890 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:35.883+0000 7fd2f2d9d700 1 -- 172.21.15.2:0/3775298027 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd2f40fb490 msgr2=0x7fd2f4112860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:13:35.890 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:35.883+0000 7fd2f2d9d700 1 --2- 172.21.15.2:0/3775298027 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd2f40fb490 0x7fd2f4112860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:13:35.891 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:35.883+0000 7fd2f2d9d700 1 -- 172.21.15.2:0/3775298027 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2e8005040 con 0x7fd2f40fd0c0 2022-05-13T23:13:35.891 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:35.883+0000 7fd2f259c700 1 --2- 172.21.15.2:0/3775298027 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd2f40fb490 0x7fd2f4112860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-05-13T23:13:35.891 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:35.884+0000 7fd2f2d9d700 1 --2- 172.21.15.2:0/3775298027 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fd2f40fd0c0 0x7fd2f4119e30 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fd2ec00eb70 tx=0x7fd2ec00b040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-05-13T23:13:35.891 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:35.884+0000 7fd2e2ffd700 1 -- 172.21.15.2:0/3775298027 <== mon.2 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2ec009870 con 0x7fd2f40fd0c0 2022-05-13T23:13:35.892 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:35.884+0000 7fd2e2ffd700 1 -- 172.21.15.2:0/3775298027 <== mon.2 v2:172.21.15.115:3300/0 2 ==== config(26 keys) v1 ==== 1087+0+0 (secure 0 0 0) 0x7fd2ec00f040 con 0x7fd2f40fd0c0 2022-05-13T23:13:35.892 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:35.884+0000 7fd2e2ffd700 1 -- 172.21.15.2:0/3775298027 <== mon.2 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2ec007ea0 con 0x7fd2f40fd0c0 2022-05-13T23:13:35.893 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:35.885+0000 7fd2f359e700 1 -- 172.21.15.2:0/3775298027 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd2f411a980 con 0x7fd2f40fd0c0 2022-05-13T23:13:35.894 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:35.886+0000 7fd2f359e700 1 -- 172.21.15.2:0/3775298027 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd2f411ae70 con 0x7fd2f40fd0c0 2022-05-13T23:13:35.894 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:35.887+0000 7fd2f359e700 1 -- 172.21.15.2:0/3775298027 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd2f4104a50 con 0x7fd2f40fd0c0 2022-05-13T23:13:35.894 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:35.889+0000 7fd2e2ffd700 1 -- 172.21.15.2:0/3775298027 <== mon.2 v2:172.21.15.115:3300/0 4 ==== mgrmap(e 16) v1 ==== 125136+0+0 (secure 0 0 0) 0x7fd2ec01a030 con 0x7fd2f40fd0c0 2022-05-13T23:13:35.894 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:35.890+0000 7fd2e2ffd700 1 --2- 172.21.15.2:0/3775298027 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7fd2dc096ce0 0x7fd2dc099190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:13:35.895 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:35.890+0000 7fd2e2ffd700 1 -- 172.21.15.2:0/3775298027 <== mon.2 v2:172.21.15.115:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 7123+0+0 (secure 0 0 0) 0x7fd2ec0bea80 con 0x7fd2f40fd0c0 2022-05-13T23:13:35.895 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:35.890+0000 7fd2f259c700 1 --2- 172.21.15.2:0/3775298027 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7fd2dc096ce0 0x7fd2dc099190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:13:35.895 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:35.891+0000 7fd2f259c700 1 --2- 172.21.15.2:0/3775298027 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7fd2dc096ce0 0x7fd2dc099190 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fd2e40041d0 tx=0x7fd2e400b040 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-05-13T23:13:35.896 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:35.894+0000 7fd2e2ffd700 1 -- 172.21.15.2:0/3775298027 <== mon.2 v2:172.21.15.115:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+165215 (secure 0 0 0) 0x7fd2ec08c110 con 0x7fd2f40fd0c0 2022-05-13T23:13:36.137 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:36.136+0000 7fd2f359e700 1 -- 172.21.15.2:0/3775298027 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd2f40fef30 con 0x7fd2f40fd0c0 2022-05-13T23:13:36.138 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:36.137+0000 7fd2e2ffd700 1 -- 172.21.15.2:0/3775298027 <== mon.2 v2:172.21.15.115:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 1 v1) v1 ==== 93+0+501 (secure 0 0 0) 0x7fd2ec08f730 con 0x7fd2f40fd0c0 2022-05-13T23:13:36.139 INFO:teuthology.orchestra.run.smithi002.stdout: 2022-05-13T23:13:36.139 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":1,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[]} 2022-05-13T23:13:36.141 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:36.139+0000 7fd2f359e700 1 -- 172.21.15.2:0/3775298027 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7fd2dc096ce0 msgr2=0x7fd2dc099190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:13:36.141 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:36.139+0000 7fd2f359e700 1 --2- 172.21.15.2:0/3775298027 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7fd2dc096ce0 0x7fd2dc099190 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fd2e40041d0 tx=0x7fd2e400b040 comp rx=0 tx=0).stop 2022-05-13T23:13:36.142 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:36.139+0000 7fd2f359e700 1 -- 172.21.15.2:0/3775298027 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fd2f40fd0c0 msgr2=0x7fd2f4119e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:13:36.142 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:36.139+0000 7fd2f359e700 1 --2- 172.21.15.2:0/3775298027 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fd2f40fd0c0 0x7fd2f4119e30 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fd2ec00eb70 tx=0x7fd2ec00b040 comp rx=0 tx=0).stop 2022-05-13T23:13:36.142 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:36.140+0000 7fd2f359e700 1 -- 172.21.15.2:0/3775298027 shutdown_connections 2022-05-13T23:13:36.142 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:36.140+0000 7fd2f359e700 1 --2- 172.21.15.2:0/3775298027 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fd2f40fd0c0 0x7fd2f4119e30 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:13:36.142 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:36.140+0000 7fd2f359e700 1 --2- 172.21.15.2:0/3775298027 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7fd2dc096ce0 0x7fd2dc099190 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:13:36.143 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:36.140+0000 7fd2f359e700 1 --2- 172.21.15.2:0/3775298027 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd2f40fb490 0x7fd2f4112860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:13:36.143 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:36.140+0000 7fd2f359e700 1 --2- 172.21.15.2:0/3775298027 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd2f40fc730 0x7fd2f4112da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:13:36.143 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:36.140+0000 7fd2f359e700 1 -- 172.21.15.2:0/3775298027 >> 172.21.15.2:0/3775298027 conn(0x7fd2f40f6c40 msgr2=0x7fd2f40f9060 unknown :-1 s=STATE_NONE l=0).mark_down 2022-05-13T23:13:36.143 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:36.140+0000 7fd2f359e700 1 -- 172.21.15.2:0/3775298027 shutdown_connections 2022-05-13T23:13:36.144 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:36.140+0000 7fd2f359e700 1 -- 172.21.15.2:0/3775298027 wait complete. 2022-05-13T23:13:36.144 INFO:teuthology.orchestra.run.smithi002.stderr:dumped fsmap epoch 1 2022-05-13T23:13:36.546 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:13:36 smithi002 ceph-mon[71816]: Deploying daemon mds.a on smithi002 2022-05-13T23:13:36.547 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:13:36 smithi002 ceph-mon[71816]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 74 MiB used, 1.0 TiB / 1.0 TiB avail 2022-05-13T23:13:36.547 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:13:36 smithi002 ceph-mon[71816]: from='client.? 172.21.15.2:0/3693854383' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2022-05-13T23:13:36.547 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:13:36 smithi002 ceph-mon[71816]: from='client.? 172.21.15.2:0/3775298027' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2022-05-13T23:13:36.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:13:36 smithi115 ceph-mon[78593]: Deploying daemon mds.a on smithi002 2022-05-13T23:13:36.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:13:36 smithi115 ceph-mon[78593]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 74 MiB used, 1.0 TiB / 1.0 TiB avail 2022-05-13T23:13:36.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:13:36 smithi115 ceph-mon[78593]: from='client.? 172.21.15.2:0/3693854383' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2022-05-13T23:13:36.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:13:36 smithi115 ceph-mon[78593]: from='client.? 172.21.15.2:0/3775298027' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2022-05-13T23:13:36.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:13:36 smithi047 ceph-mon[77147]: Deploying daemon mds.a on smithi002 2022-05-13T23:13:36.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:13:36 smithi047 ceph-mon[77147]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 74 MiB used, 1.0 TiB / 1.0 TiB avail 2022-05-13T23:13:36.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:13:36 smithi047 ceph-mon[77147]: from='client.? 172.21.15.2:0/3693854383' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2022-05-13T23:13:36.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:13:36 smithi047 ceph-mon[77147]: from='client.? 172.21.15.2:0/3775298027' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2022-05-13T23:13:37.823 INFO:journalctl@ceph.mds.a.smithi002.stdout:May 13 23:13:37 smithi002 systemd[1]: Starting Ceph mds.a for 16cf5f88-d311-11ec-be40-001a4aab830c... 2022-05-13T23:13:38.170 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-05-13T23:13:38.407 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:38.405+0000 7fba3cf9d700 1 -- 172.21.15.2:0/402505777 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fba381010f0 msgr2=0x7fba38103570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:13:38.407 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:38.405+0000 7fba3cf9d700 1 --2- 172.21.15.2:0/402505777 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fba381010f0 0x7fba38103570 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fba30009be0 tx=0x7fba30004d10 comp rx=0 tx=0).stop 2022-05-13T23:13:38.407 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:38.405+0000 7fba3cf9d700 1 -- 172.21.15.2:0/402505777 shutdown_connections 2022-05-13T23:13:38.408 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:38.405+0000 7fba3cf9d700 1 --2- 172.21.15.2:0/402505777 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fba381010f0 0x7fba38103570 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:13:38.408 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:38.405+0000 7fba3cf9d700 1 --2- 172.21.15.2:0/402505777 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fba38064670 0x7fba380fe100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:13:38.408 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:38.405+0000 7fba3cf9d700 1 --2- 172.21.15.2:0/402505777 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fba380fe640 0x7fba38100a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:13:38.408 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:38.405+0000 7fba3cf9d700 1 -- 172.21.15.2:0/402505777 >> 172.21.15.2:0/402505777 conn(0x7fba380f5b30 msgr2=0x7fba380f7fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-05-13T23:13:38.409 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:38.405+0000 7fba3cf9d700 1 -- 172.21.15.2:0/402505777 shutdown_connections 2022-05-13T23:13:38.409 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:38.405+0000 7fba3cf9d700 1 -- 172.21.15.2:0/402505777 wait complete. 2022-05-13T23:13:38.411 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:38.407+0000 7fba3cf9d700 1 Processor -- start 2022-05-13T23:13:38.412 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:38.407+0000 7fba3cf9d700 1 -- start start 2022-05-13T23:13:38.412 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:38.407+0000 7fba3cf9d700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fba38064670 0x7fba380fb9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:13:38.412 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:38.407+0000 7fba3cf9d700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fba380fe640 0x7fba380fa000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:13:38.412 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:38.407+0000 7fba3cf9d700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fba381010f0 0x7fba380fa540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:13:38.412 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:38.407+0000 7fba3cf9d700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba380fac50 con 0x7fba381010f0 2022-05-13T23:13:38.413 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:38.407+0000 7fba3cf9d700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba380fadc0 con 0x7fba380fe640 2022-05-13T23:13:38.413 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:38.407+0000 7fba3cf9d700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba38116220 con 0x7fba38064670 2022-05-13T23:13:38.413 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:38.408+0000 7fba36ffd700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fba380fe640 0x7fba380fa000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:13:38.413 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:38.408+0000 7fba377fe700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fba38064670 0x7fba380fb9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:13:38.414 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:38.408+0000 7fba36ffd700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fba380fe640 0x7fba380fa000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.2:35434/0 (socket says 172.21.15.2:35434) 2022-05-13T23:13:38.414 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:38.408+0000 7fba36ffd700 1 -- 172.21.15.2:0/1413313142 learned_addr learned my addr 172.21.15.2:0/1413313142 (peer_addr_for_me v2:172.21.15.2:0/0) 2022-05-13T23:13:38.414 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:38.408+0000 7fba37fff700 1 --2- 172.21.15.2:0/1413313142 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fba381010f0 0x7fba380fa540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:13:38.414 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:38.408+0000 7fba36ffd700 1 -- 172.21.15.2:0/1413313142 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fba38064670 msgr2=0x7fba380fb9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:13:38.415 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:38.408+0000 7fba36ffd700 1 --2- 172.21.15.2:0/1413313142 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fba38064670 0x7fba380fb9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:13:38.415 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:38.408+0000 7fba36ffd700 1 -- 172.21.15.2:0/1413313142 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fba381010f0 msgr2=0x7fba380fa540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:13:38.415 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:38.408+0000 7fba36ffd700 1 --2- 172.21.15.2:0/1413313142 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fba381010f0 0x7fba380fa540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:13:38.415 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:38.408+0000 7fba36ffd700 1 -- 172.21.15.2:0/1413313142 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba300097c0 con 0x7fba380fe640 2022-05-13T23:13:38.415 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:38.409+0000 7fba36ffd700 1 --2- 172.21.15.2:0/1413313142 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fba380fe640 0x7fba380fa000 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fba2c00e940 tx=0x7fba2c00b040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-05-13T23:13:38.416 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:38.409+0000 7fba1ffff700 1 -- 172.21.15.2:0/1413313142 <== mon.1 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba2c007b40 con 0x7fba380fe640 2022-05-13T23:13:38.416 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:38.409+0000 7fba1ffff700 1 -- 172.21.15.2:0/1413313142 <== mon.1 v2:172.21.15.47:3300/0 2 ==== config(26 keys) v1 ==== 1087+0+0 (secure 0 0 0) 0x7fba2c00f040 con 0x7fba380fe640 2022-05-13T23:13:38.416 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:38.409+0000 7fba3cf9d700 1 -- 172.21.15.2:0/1413313142 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fba381163c0 con 0x7fba380fe640 2022-05-13T23:13:38.416 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:38.409+0000 7fba1ffff700 1 -- 172.21.15.2:0/1413313142 <== mon.1 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba2c0148b0 con 0x7fba380fe640 2022-05-13T23:13:38.417 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:38.409+0000 7fba3cf9d700 1 -- 172.21.15.2:0/1413313142 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fba38116980 con 0x7fba380fe640 2022-05-13T23:13:38.417 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:38.410+0000 7fba3cf9d700 1 -- 172.21.15.2:0/1413313142 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fba38106ca0 con 0x7fba380fe640 2022-05-13T23:13:38.418 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:38.412+0000 7fba1ffff700 1 -- 172.21.15.2:0/1413313142 <== mon.1 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 16) v1 ==== 125136+0+0 (secure 0 0 0) 0x7fba2c01b030 con 0x7fba380fe640 2022-05-13T23:13:38.418 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:38.414+0000 7fba1ffff700 1 --2- 172.21.15.2:0/1413313142 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7fba20096d60 0x7fba20099210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:13:38.419 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:38.414+0000 7fba1ffff700 1 -- 172.21.15.2:0/1413313142 <== mon.1 v2:172.21.15.47:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 7123+0+0 (secure 0 0 0) 0x7fba2c0bed30 con 0x7fba380fe640 2022-05-13T23:13:38.419 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:38.414+0000 7fba377fe700 1 --2- 172.21.15.2:0/1413313142 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7fba20096d60 0x7fba20099210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:13:38.419 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:38.415+0000 7fba377fe700 1 --2- 172.21.15.2:0/1413313142 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7fba20096d60 0x7fba20099210 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fba280048b0 tx=0x7fba2800b040 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-05-13T23:13:38.420 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:38.416+0000 7fba1ffff700 1 -- 172.21.15.2:0/1413313142 <== mon.1 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+165215 (secure 0 0 0) 0x7fba2c08c3c0 con 0x7fba380fe640 2022-05-13T23:13:38.559 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:13:38 smithi002 ceph-mon[71816]: pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 74 MiB used, 1.0 TiB / 1.0 TiB avail 2022-05-13T23:13:38.559 INFO:journalctl@ceph.mds.a.smithi002.stdout:May 13 23:13:38 smithi002 ceph-mds[108237]: -- 172.21.15.2:0/2520905203 <== mon.2 v2:172.21.15.115:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55aa7b08c340 con 0x55aa7b064800 2022-05-13T23:13:38.559 INFO:journalctl@ceph.mds.a.smithi002.stdout:May 13 23:13:38 smithi002 conmon[108188]: starting mds.a at 2022-05-13T23:13:38.560 INFO:journalctl@ceph.mds.a.smithi002.stdout:May 13 23:13:38 smithi002 bash[107982]: 336a72a34d7ffd57366d984e5235eadb56ad6a13aa5e156ab0dc935facf32a2b 2022-05-13T23:13:38.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:13:38 smithi115 ceph-mon[78593]: pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 74 MiB used, 1.0 TiB / 1.0 TiB avail 2022-05-13T23:13:38.661 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:38.659+0000 7fba3cf9d700 1 -- 172.21.15.2:0/1413313142 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fba380f74e0 con 0x7fba380fe640 2022-05-13T23:13:38.661 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:38.660+0000 7fba1ffff700 1 -- 172.21.15.2:0/1413313142 <== mon.1 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 2 v2) v1 ==== 93+0+1254 (secure 0 0 0) 0x7fba2c019500 con 0x7fba380fe640 2022-05-13T23:13:38.662 INFO:teuthology.orchestra.run.smithi002.stdout: 2022-05-13T23:13:38.662 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":2,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24460,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.2:6835/1341618941","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6834","nonce":1341618941},{"type":"v1","addr":"172.21.15.2:6835","nonce":1341618941}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[]} 2022-05-13T23:13:38.665 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:38.663+0000 7fba3cf9d700 1 -- 172.21.15.2:0/1413313142 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7fba20096d60 msgr2=0x7fba20099210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:13:38.665 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:38.663+0000 7fba3cf9d700 1 --2- 172.21.15.2:0/1413313142 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7fba20096d60 0x7fba20099210 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fba280048b0 tx=0x7fba2800b040 comp rx=0 tx=0).stop 2022-05-13T23:13:38.665 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:38.663+0000 7fba3cf9d700 1 -- 172.21.15.2:0/1413313142 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fba380fe640 msgr2=0x7fba380fa000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:13:38.665 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:38.663+0000 7fba3cf9d700 1 --2- 172.21.15.2:0/1413313142 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fba380fe640 0x7fba380fa000 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fba2c00e940 tx=0x7fba2c00b040 comp rx=0 tx=0).stop 2022-05-13T23:13:38.666 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:38.663+0000 7fba3cf9d700 1 -- 172.21.15.2:0/1413313142 shutdown_connections 2022-05-13T23:13:38.666 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:38.663+0000 7fba3cf9d700 1 --2- 172.21.15.2:0/1413313142 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7fba20096d60 0x7fba20099210 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:13:38.666 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:38.663+0000 7fba3cf9d700 1 --2- 172.21.15.2:0/1413313142 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fba381010f0 0x7fba380fa540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:13:38.666 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:38.663+0000 7fba3cf9d700 1 --2- 172.21.15.2:0/1413313142 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fba38064670 0x7fba380fb9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:13:38.667 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:38.663+0000 7fba3cf9d700 1 --2- 172.21.15.2:0/1413313142 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fba380fe640 0x7fba380fa000 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:13:38.668 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:38.663+0000 7fba3cf9d700 1 -- 172.21.15.2:0/1413313142 >> 172.21.15.2:0/1413313142 conn(0x7fba380f5b30 msgr2=0x7fba380fca20 unknown :-1 s=STATE_NONE l=0).mark_down 2022-05-13T23:13:38.668 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:38.663+0000 7fba3cf9d700 1 -- 172.21.15.2:0/1413313142 shutdown_connections 2022-05-13T23:13:38.668 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:38.663+0000 7fba3cf9d700 1 -- 172.21.15.2:0/1413313142 wait complete. 2022-05-13T23:13:38.669 INFO:teuthology.orchestra.run.smithi002.stderr:dumped fsmap epoch 2 2022-05-13T23:13:38.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:13:38 smithi047 ceph-mon[77147]: pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 74 MiB used, 1.0 TiB / 1.0 TiB avail 2022-05-13T23:13:38.822 INFO:journalctl@ceph.mds.a.smithi002.stdout:May 13 23:13:38 smithi002 systemd[1]: Started Ceph mds.a for 16cf5f88-d311-11ec-be40-001a4aab830c. 2022-05-13T23:13:39.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:13:39 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.2:6834/1341618941,v1:172.21.15.2:6835/1341618941] up:boot 2022-05-13T23:13:39.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:13:39 smithi047 ceph-mon[77147]: fsmap 1 up:standby 2022-05-13T23:13:39.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:13:39 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2022-05-13T23:13:39.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:13:39 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:13:39.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:13:39 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2022-05-13T23:13:39.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:13:39 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2022-05-13T23:13:39.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:13:39 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-13T23:13:39.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:13:39 smithi047 ceph-mon[77147]: Deploying daemon mds.d on smithi002 2022-05-13T23:13:39.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:13:39 smithi047 ceph-mon[77147]: from='client.? 172.21.15.2:0/1413313142' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2022-05-13T23:13:39.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:13:39 smithi047 ceph-mon[77147]: pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 74 MiB used, 1.0 TiB / 1.0 TiB avail 2022-05-13T23:13:39.822 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:13:39 smithi002 ceph-mon[71816]: mds.? [v2:172.21.15.2:6834/1341618941,v1:172.21.15.2:6835/1341618941] up:boot 2022-05-13T23:13:39.823 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:13:39 smithi002 ceph-mon[71816]: fsmap 1 up:standby 2022-05-13T23:13:39.823 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:13:39 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2022-05-13T23:13:39.823 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:13:39 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:13:39.823 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:13:39 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2022-05-13T23:13:39.824 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:13:39 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2022-05-13T23:13:39.824 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:13:39 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-13T23:13:39.824 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:13:39 smithi002 ceph-mon[71816]: Deploying daemon mds.d on smithi002 2022-05-13T23:13:39.824 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:13:39 smithi002 ceph-mon[71816]: from='client.? 172.21.15.2:0/1413313142' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2022-05-13T23:13:39.825 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:13:39 smithi002 ceph-mon[71816]: pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 74 MiB used, 1.0 TiB / 1.0 TiB avail 2022-05-13T23:13:39.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:13:39 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.2:6834/1341618941,v1:172.21.15.2:6835/1341618941] up:boot 2022-05-13T23:13:39.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:13:39 smithi115 ceph-mon[78593]: fsmap 1 up:standby 2022-05-13T23:13:39.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:13:39 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2022-05-13T23:13:39.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:13:39 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:13:39.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:13:39 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2022-05-13T23:13:39.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:13:39 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2022-05-13T23:13:39.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:13:39 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-13T23:13:39.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:13:39 smithi115 ceph-mon[78593]: Deploying daemon mds.d on smithi002 2022-05-13T23:13:39.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:13:39 smithi115 ceph-mon[78593]: from='client.? 172.21.15.2:0/1413313142' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2022-05-13T23:13:39.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:13:39 smithi115 ceph-mon[78593]: pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 74 MiB used, 1.0 TiB / 1.0 TiB avail 2022-05-13T23:13:40.692 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-05-13T23:13:40.951 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:40.949+0000 7f3cab59e700 1 -- 172.21.15.2:0/7689313 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3cac0fb490 msgr2=0x7f3cac0fb8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:13:40.952 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:40.949+0000 7f3cab59e700 1 --2- 172.21.15.2:0/7689313 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3cac0fb490 0x7f3cac0fb8a0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f3c9c004870 tx=0x7f3c9c009cb0 comp rx=0 tx=0).stop 2022-05-13T23:13:40.952 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:40.950+0000 7f3cab59e700 1 -- 172.21.15.2:0/7689313 shutdown_connections 2022-05-13T23:13:40.952 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:40.950+0000 7f3cab59e700 1 --2- 172.21.15.2:0/7689313 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3cac0fd0c0 0x7f3cac103980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:13:40.953 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:40.950+0000 7f3cab59e700 1 --2- 172.21.15.2:0/7689313 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3cac0fb490 0x7f3cac0fb8a0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:13:40.953 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:40.950+0000 7f3cab59e700 1 --2- 172.21.15.2:0/7689313 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f3cac0fc730 0x7f3cac0fcb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:13:40.953 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:40.950+0000 7f3cab59e700 1 -- 172.21.15.2:0/7689313 >> 172.21.15.2:0/7689313 conn(0x7f3cac0f6c40 msgr2=0x7f3cac0f90b0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-05-13T23:13:40.953 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:40.950+0000 7f3cab59e700 1 -- 172.21.15.2:0/7689313 shutdown_connections 2022-05-13T23:13:40.954 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:40.950+0000 7f3cab59e700 1 -- 172.21.15.2:0/7689313 wait complete. 2022-05-13T23:13:40.956 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:40.951+0000 7f3cab59e700 1 Processor -- start 2022-05-13T23:13:40.957 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:40.951+0000 7f3cab59e700 1 -- start start 2022-05-13T23:13:40.957 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:40.951+0000 7f3cab59e700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3cac0fb490 0x7f3cac10e4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:13:40.970 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:40.951+0000 7f3cab59e700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3cac0fc730 0x7f3cac10ea10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:13:40.970 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:40.951+0000 7f3cab59e700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f3cac0fd0c0 0x7f3cac115aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:13:40.970 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:40.951+0000 7f3cab59e700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3cac115fe0 con 0x7f3cac0fb490 2022-05-13T23:13:40.971 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:40.951+0000 7f3cab59e700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3cac116150 con 0x7f3cac0fc730 2022-05-13T23:13:40.971 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:40.951+0000 7f3cab59e700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3cac116310 con 0x7f3cac0fd0c0 2022-05-13T23:13:40.971 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:40.952+0000 7f3caa59c700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3cac0fb490 0x7f3cac10e4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:13:40.971 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:40.952+0000 7f3caa59c700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3cac0fb490 0x7f3cac10e4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:48802/0 (socket says 172.21.15.2:48802) 2022-05-13T23:13:40.971 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:40.952+0000 7f3caa59c700 1 -- 172.21.15.2:0/2825257017 learned_addr learned my addr 172.21.15.2:0/2825257017 (peer_addr_for_me v2:172.21.15.2:0/0) 2022-05-13T23:13:40.972 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:40.952+0000 7f3ca9d9b700 1 --2- 172.21.15.2:0/2825257017 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3cac0fc730 0x7f3cac10ea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:13:40.972 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:40.952+0000 7f3caad9d700 1 --2- 172.21.15.2:0/2825257017 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f3cac0fd0c0 0x7f3cac115aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:13:40.972 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:40.952+0000 7f3caa59c700 1 -- 172.21.15.2:0/2825257017 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f3cac0fd0c0 msgr2=0x7f3cac115aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:13:40.972 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:40.952+0000 7f3caa59c700 1 --2- 172.21.15.2:0/2825257017 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f3cac0fd0c0 0x7f3cac115aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:13:40.973 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:40.952+0000 7f3caa59c700 1 -- 172.21.15.2:0/2825257017 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3cac0fc730 msgr2=0x7f3cac10ea10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:13:40.973 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:40.952+0000 7f3caa59c700 1 --2- 172.21.15.2:0/2825257017 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3cac0fc730 0x7f3cac10ea10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:13:40.973 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:40.952+0000 7f3caa59c700 1 -- 172.21.15.2:0/2825257017 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ca40097c0 con 0x7f3cac0fb490 2022-05-13T23:13:40.973 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:40.953+0000 7f3caa59c700 1 --2- 172.21.15.2:0/2825257017 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3cac0fb490 0x7f3cac10e4d0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f3c9c004710 tx=0x7f3c9c009cb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-05-13T23:13:40.973 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:40.953+0000 7f3c9affd700 1 -- 172.21.15.2:0/2825257017 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c9c00e040 con 0x7f3cac0fb490 2022-05-13T23:13:40.974 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:40.953+0000 7f3c9affd700 1 -- 172.21.15.2:0/2825257017 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1087+0+0 (secure 0 0 0) 0x7f3c9c004200 con 0x7f3cac0fb490 2022-05-13T23:13:40.974 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:40.953+0000 7f3c9affd700 1 -- 172.21.15.2:0/2825257017 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c9c01caf0 con 0x7f3cac0fb490 2022-05-13T23:13:40.974 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:40.953+0000 7f3cab59e700 1 -- 172.21.15.2:0/2825257017 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3c9c005040 con 0x7f3cac0fb490 2022-05-13T23:13:40.974 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:40.953+0000 7f3cab59e700 1 -- 172.21.15.2:0/2825257017 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3cac116880 con 0x7f3cac0fb490 2022-05-13T23:13:40.975 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:40.954+0000 7f3cab59e700 1 -- 172.21.15.2:0/2825257017 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3cac106cf0 con 0x7f3cac0fb490 2022-05-13T23:13:40.980 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:40.958+0000 7f3c9affd700 1 -- 172.21.15.2:0/2825257017 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 16) v1 ==== 125136+0+0 (secure 0 0 0) 0x7f3c9c009240 con 0x7f3cac0fb490 2022-05-13T23:13:40.980 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:40.960+0000 7f3c9affd700 1 --2- 172.21.15.2:0/2825257017 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f3c94096d60 0x7f3c94099210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:13:40.981 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:40.960+0000 7f3c9affd700 1 -- 172.21.15.2:0/2825257017 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 7123+0+0 (secure 0 0 0) 0x7f3c9c0bfef0 con 0x7f3cac0fb490 2022-05-13T23:13:40.981 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:40.960+0000 7f3ca9d9b700 1 --2- 172.21.15.2:0/2825257017 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f3c94096d60 0x7f3c94099210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:13:40.982 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:40.961+0000 7f3ca9d9b700 1 --2- 172.21.15.2:0/2825257017 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f3c94096d60 0x7f3c94099210 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f3ca00048b0 tx=0x7f3ca000b040 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-05-13T23:13:40.982 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:40.965+0000 7f3c9affd700 1 -- 172.21.15.2:0/2825257017 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+165215 (secure 0 0 0) 0x7f3c9c08d670 con 0x7f3cac0fb490 2022-05-13T23:13:41.200 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:41.198+0000 7f3cab59e700 1 -- 172.21.15.2:0/2825257017 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3cac116b60 con 0x7f3cac0fb490 2022-05-13T23:13:41.200 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:41.199+0000 7f3c9affd700 1 -- 172.21.15.2:0/2825257017 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 2 v2) v1 ==== 93+0+1254 (secure 0 0 0) 0x7f3c9c090c90 con 0x7f3cac0fb490 2022-05-13T23:13:41.201 INFO:teuthology.orchestra.run.smithi002.stdout: 2022-05-13T23:13:41.201 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":2,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24460,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.2:6835/1341618941","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6834","nonce":1341618941},{"type":"v1","addr":"172.21.15.2:6835","nonce":1341618941}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[]} 2022-05-13T23:13:41.203 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:41.202+0000 7f3cab59e700 1 -- 172.21.15.2:0/2825257017 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f3c94096d60 msgr2=0x7f3c94099210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:13:41.203 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:41.202+0000 7f3cab59e700 1 --2- 172.21.15.2:0/2825257017 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f3c94096d60 0x7f3c94099210 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f3ca00048b0 tx=0x7f3ca000b040 comp rx=0 tx=0).stop 2022-05-13T23:13:41.204 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:41.202+0000 7f3cab59e700 1 -- 172.21.15.2:0/2825257017 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3cac0fb490 msgr2=0x7f3cac10e4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:13:41.204 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:41.202+0000 7f3cab59e700 1 --2- 172.21.15.2:0/2825257017 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3cac0fb490 0x7f3cac10e4d0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f3c9c004710 tx=0x7f3c9c009cb0 comp rx=0 tx=0).stop 2022-05-13T23:13:41.204 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:41.202+0000 7f3cab59e700 1 -- 172.21.15.2:0/2825257017 shutdown_connections 2022-05-13T23:13:41.204 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:41.202+0000 7f3cab59e700 1 --2- 172.21.15.2:0/2825257017 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f3cac0fd0c0 0x7f3cac115aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:13:41.205 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:41.202+0000 7f3cab59e700 1 --2- 172.21.15.2:0/2825257017 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f3c94096d60 0x7f3c94099210 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:13:41.205 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:41.202+0000 7f3cab59e700 1 --2- 172.21.15.2:0/2825257017 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3cac0fb490 0x7f3cac10e4d0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:13:41.205 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:41.202+0000 7f3cab59e700 1 --2- 172.21.15.2:0/2825257017 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3cac0fc730 0x7f3cac10ea10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:13:41.205 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:41.202+0000 7f3cab59e700 1 -- 172.21.15.2:0/2825257017 >> 172.21.15.2:0/2825257017 conn(0x7f3cac0f6c40 msgr2=0x7f3cac0f9000 unknown :-1 s=STATE_NONE l=0).mark_down 2022-05-13T23:13:41.206 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:41.202+0000 7f3cab59e700 1 -- 172.21.15.2:0/2825257017 shutdown_connections 2022-05-13T23:13:41.206 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:41.202+0000 7f3cab59e700 1 -- 172.21.15.2:0/2825257017 wait complete. 2022-05-13T23:13:41.206 INFO:teuthology.orchestra.run.smithi002.stderr:dumped fsmap epoch 2 2022-05-13T23:13:41.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:13:41 smithi047 ceph-mon[77147]: from='client.? 172.21.15.2:0/2825257017' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2022-05-13T23:13:41.572 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:13:41 smithi002 ceph-mon[71816]: from='client.? 172.21.15.2:0/2825257017' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2022-05-13T23:13:41.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:13:41 smithi115 ceph-mon[78593]: from='client.? 172.21.15.2:0/2825257017' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2022-05-13T23:13:42.073 INFO:journalctl@ceph.mds.d.smithi002.stdout:May 13 23:13:41 smithi002 systemd[1]: Starting Ceph mds.d for 16cf5f88-d311-11ec-be40-001a4aab830c... 2022-05-13T23:13:42.533 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:13:42 smithi002 ceph-mon[71816]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 74 MiB used, 1.0 TiB / 1.0 TiB avail 2022-05-13T23:13:42.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:13:42 smithi047 ceph-mon[77147]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 74 MiB used, 1.0 TiB / 1.0 TiB avail 2022-05-13T23:13:42.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:13:42 smithi115 ceph-mon[78593]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 74 MiB used, 1.0 TiB / 1.0 TiB avail 2022-05-13T23:13:42.962 INFO:journalctl@ceph.mds.d.smithi002.stdout:May 13 23:13:42 smithi002 ceph-mds[108855]: -- 172.21.15.2:0/1295017754 <== mon.0 v2:172.21.15.2:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55e664b5c340 con 0x55e664b34800 2022-05-13T23:13:42.962 INFO:journalctl@ceph.mds.d.smithi002.stdout:May 13 23:13:42 smithi002 conmon[108831]: starting mds.d at 2022-05-13T23:13:42.962 INFO:journalctl@ceph.mds.d.smithi002.stdout:May 13 23:13:42 smithi002 bash[108641]: f91119b0055d45b80ff974e9132cb8c9aa01e04be7fc1e3347ff40052953651b 2022-05-13T23:13:42.963 INFO:journalctl@ceph.mds.d.smithi002.stdout:May 13 23:13:42 smithi002 systemd[1]: Started Ceph mds.d for 16cf5f88-d311-11ec-be40-001a4aab830c. 2022-05-13T23:13:43.232 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-05-13T23:13:43.458 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:43.456+0000 7f2c50d9d700 1 -- 172.21.15.2:0/3198305152 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f2c4c0fb490 msgr2=0x7f2c4c0fb8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:13:43.458 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:43.456+0000 7f2c50d9d700 1 --2- 172.21.15.2:0/3198305152 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f2c4c0fb490 0x7f2c4c0fb8a0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f2c3c004870 tx=0x7f2c3c009cb0 comp rx=0 tx=0).stop 2022-05-13T23:13:43.458 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:43.456+0000 7f2c50d9d700 1 -- 172.21.15.2:0/3198305152 shutdown_connections 2022-05-13T23:13:43.459 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:43.456+0000 7f2c50d9d700 1 --2- 172.21.15.2:0/3198305152 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2c4c0fd0c0 0x7f2c4c103980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:13:43.459 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:43.456+0000 7f2c50d9d700 1 --2- 172.21.15.2:0/3198305152 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f2c4c0fb490 0x7f2c4c0fb8a0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:13:43.459 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:43.456+0000 7f2c50d9d700 1 --2- 172.21.15.2:0/3198305152 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f2c4c0fc730 0x7f2c4c0fcb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:13:43.459 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:43.456+0000 7f2c50d9d700 1 -- 172.21.15.2:0/3198305152 >> 172.21.15.2:0/3198305152 conn(0x7f2c4c0f6c40 msgr2=0x7f2c4c0f90b0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-05-13T23:13:43.460 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:43.456+0000 7f2c50d9d700 1 -- 172.21.15.2:0/3198305152 shutdown_connections 2022-05-13T23:13:43.460 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:43.456+0000 7f2c50d9d700 1 -- 172.21.15.2:0/3198305152 wait complete. 2022-05-13T23:13:43.463 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:43.458+0000 7f2c50d9d700 1 Processor -- start 2022-05-13T23:13:43.463 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:43.458+0000 7f2c50d9d700 1 -- start start 2022-05-13T23:13:43.463 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:43.458+0000 7f2c50d9d700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f2c4c0fb490 0x7f2c4c10c2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:13:43.464 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:43.458+0000 7f2c50d9d700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2c4c0fc730 0x7f2c4c10c810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:13:43.464 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:43.458+0000 7f2c50d9d700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f2c4c0fd0c0 0x7f2c4c1138a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:13:43.464 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:43.458+0000 7f2c50d9d700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c4c113de0 con 0x7f2c4c0fc730 2022-05-13T23:13:43.464 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:43.458+0000 7f2c50d9d700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c4c113f50 con 0x7f2c4c0fb490 2022-05-13T23:13:43.465 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:43.458+0000 7f2c50d9d700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c4c114110 con 0x7f2c4c0fd0c0 2022-05-13T23:13:43.465 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:43.459+0000 7f2c4b7fe700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f2c4c0fb490 0x7f2c4c10c2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:13:43.465 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:43.459+0000 7f2c4affd700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2c4c0fc730 0x7f2c4c10c810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:13:43.465 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:43.459+0000 7f2c4b7fe700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f2c4c0fb490 0x7f2c4c10c2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.2:46956/0 (socket says 172.21.15.2:46956) 2022-05-13T23:13:43.466 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:43.459+0000 7f2c4b7fe700 1 -- 172.21.15.2:0/3362438555 learned_addr learned my addr 172.21.15.2:0/3362438555 (peer_addr_for_me v2:172.21.15.2:0/0) 2022-05-13T23:13:43.466 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:43.459+0000 7f2c4bfff700 1 --2- 172.21.15.2:0/3362438555 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f2c4c0fd0c0 0x7f2c4c1138a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:13:43.466 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:43.459+0000 7f2c4b7fe700 1 -- 172.21.15.2:0/3362438555 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f2c4c0fd0c0 msgr2=0x7f2c4c1138a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:13:43.466 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:43.459+0000 7f2c4b7fe700 1 --2- 172.21.15.2:0/3362438555 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f2c4c0fd0c0 0x7f2c4c1138a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:13:43.466 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:43.459+0000 7f2c4b7fe700 1 -- 172.21.15.2:0/3362438555 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2c4c0fc730 msgr2=0x7f2c4c10c810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:13:43.467 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:43.459+0000 7f2c4b7fe700 1 --2- 172.21.15.2:0/3362438555 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2c4c0fc730 0x7f2c4c10c810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:13:43.468 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:43.459+0000 7f2c4b7fe700 1 -- 172.21.15.2:0/3362438555 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c3c005040 con 0x7f2c4c0fb490 2022-05-13T23:13:43.468 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:43.459+0000 7f2c4affd700 1 --2- 172.21.15.2:0/3362438555 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2c4c0fc730 0x7f2c4c10c810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-05-13T23:13:43.468 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:43.460+0000 7f2c4b7fe700 1 --2- 172.21.15.2:0/3362438555 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f2c4c0fb490 0x7f2c4c10c2d0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f2c3c004710 tx=0x7f2c3c009cb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-05-13T23:13:43.468 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:43.460+0000 7f2c33fff700 1 -- 172.21.15.2:0/3362438555 <== mon.1 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c3c00e040 con 0x7f2c4c0fb490 2022-05-13T23:13:43.468 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:43.460+0000 7f2c33fff700 1 -- 172.21.15.2:0/3362438555 <== mon.1 v2:172.21.15.47:3300/0 2 ==== config(26 keys) v1 ==== 1087+0+0 (secure 0 0 0) 0x7f2c3c004200 con 0x7f2c4c0fb490 2022-05-13T23:13:43.469 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:43.460+0000 7f2c50d9d700 1 -- 172.21.15.2:0/3362438555 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2c4c114390 con 0x7f2c4c0fb490 2022-05-13T23:13:43.469 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:43.460+0000 7f2c33fff700 1 -- 172.21.15.2:0/3362438555 <== mon.1 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c3c01caf0 con 0x7f2c4c0fb490 2022-05-13T23:13:43.469 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:43.460+0000 7f2c50d9d700 1 -- 172.21.15.2:0/3362438555 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2c4c114870 con 0x7f2c4c0fb490 2022-05-13T23:13:43.469 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:43.461+0000 7f2c50d9d700 1 -- 172.21.15.2:0/3362438555 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2c4c106cf0 con 0x7f2c4c0fb490 2022-05-13T23:13:43.470 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:43.463+0000 7f2c33fff700 1 -- 172.21.15.2:0/3362438555 <== mon.1 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 16) v1 ==== 125136+0+0 (secure 0 0 0) 0x7f2c3c009240 con 0x7f2c4c0fb490 2022-05-13T23:13:43.471 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:43.465+0000 7f2c33fff700 1 --2- 172.21.15.2:0/3362438555 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f2c34096d60 0x7f2c34099210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:13:43.471 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:43.465+0000 7f2c33fff700 1 -- 172.21.15.2:0/3362438555 <== mon.1 v2:172.21.15.47:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 7123+0+0 (secure 0 0 0) 0x7f2c3c0bffc0 con 0x7f2c4c0fb490 2022-05-13T23:13:43.471 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:43.465+0000 7f2c4affd700 1 --2- 172.21.15.2:0/3362438555 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f2c34096d60 0x7f2c34099210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:13:43.471 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:43.466+0000 7f2c4affd700 1 --2- 172.21.15.2:0/3362438555 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f2c34096d60 0x7f2c34099210 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f2c40007bf0 tx=0x7f2c4000b040 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-05-13T23:13:43.472 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:43.469+0000 7f2c33fff700 1 -- 172.21.15.2:0/3362438555 <== mon.1 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+165215 (secure 0 0 0) 0x7f2c3c08d650 con 0x7f2c4c0fb490 2022-05-13T23:13:43.712 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:43.710+0000 7f2c50d9d700 1 -- 172.21.15.2:0/3362438555 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2c4c102660 con 0x7f2c4c0fb490 2022-05-13T23:13:43.712 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:43.711+0000 7f2c33fff700 1 -- 172.21.15.2:0/3362438555 <== mon.1 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) v1 ==== 93+0+2008 (secure 0 0 0) 0x7f2c3c090c70 con 0x7f2c4c0fb490 2022-05-13T23:13:43.713 INFO:teuthology.orchestra.run.smithi002.stdout: 2022-05-13T23:13:43.713 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":3,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24460,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.2:6835/1341618941","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6834","nonce":1341618941},{"type":"v1","addr":"172.21.15.2:6835","nonce":1341618941}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24494,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.2:6837/3754333215","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6836","nonce":3754333215},{"type":"v1","addr":"172.21.15.2:6837","nonce":3754333215}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[]} 2022-05-13T23:13:43.715 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:43.713+0000 7f2c50d9d700 1 -- 172.21.15.2:0/3362438555 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f2c34096d60 msgr2=0x7f2c34099210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:13:43.716 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:43.713+0000 7f2c50d9d700 1 --2- 172.21.15.2:0/3362438555 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f2c34096d60 0x7f2c34099210 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f2c40007bf0 tx=0x7f2c4000b040 comp rx=0 tx=0).stop 2022-05-13T23:13:43.717 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:43.713+0000 7f2c50d9d700 1 -- 172.21.15.2:0/3362438555 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f2c4c0fb490 msgr2=0x7f2c4c10c2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:13:43.717 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:43.713+0000 7f2c50d9d700 1 --2- 172.21.15.2:0/3362438555 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f2c4c0fb490 0x7f2c4c10c2d0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f2c3c004710 tx=0x7f2c3c009cb0 comp rx=0 tx=0).stop 2022-05-13T23:13:43.718 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:43.714+0000 7f2c50d9d700 1 -- 172.21.15.2:0/3362438555 shutdown_connections 2022-05-13T23:13:43.718 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:43.714+0000 7f2c50d9d700 1 --2- 172.21.15.2:0/3362438555 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f2c4c0fd0c0 0x7f2c4c1138a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:13:43.718 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:43.714+0000 7f2c50d9d700 1 --2- 172.21.15.2:0/3362438555 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f2c4c0fb490 0x7f2c4c10c2d0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:13:43.718 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:43.714+0000 7f2c50d9d700 1 --2- 172.21.15.2:0/3362438555 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f2c34096d60 0x7f2c34099210 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:13:43.719 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:43.714+0000 7f2c50d9d700 1 --2- 172.21.15.2:0/3362438555 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2c4c0fc730 0x7f2c4c10c810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:13:43.719 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:43.714+0000 7f2c50d9d700 1 -- 172.21.15.2:0/3362438555 >> 172.21.15.2:0/3362438555 conn(0x7f2c4c0f6c40 msgr2=0x7f2c4c0ff960 unknown :-1 s=STATE_NONE l=0).mark_down 2022-05-13T23:13:43.719 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:43.714+0000 7f2c50d9d700 1 -- 172.21.15.2:0/3362438555 shutdown_connections 2022-05-13T23:13:43.719 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:43.714+0000 7f2c50d9d700 1 -- 172.21.15.2:0/3362438555 wait complete. 2022-05-13T23:13:43.720 INFO:teuthology.orchestra.run.smithi002.stderr:dumped fsmap epoch 3 2022-05-13T23:13:44.072 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:13:43 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:13:44.073 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:13:43 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2022-05-13T23:13:44.073 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:13:43 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2022-05-13T23:13:44.073 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:13:43 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-13T23:13:44.073 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:13:43 smithi002 ceph-mon[71816]: Deploying daemon mds.g on smithi002 2022-05-13T23:13:44.074 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:13:43 smithi002 ceph-mon[71816]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 74 MiB used, 1.0 TiB / 1.0 TiB avail 2022-05-13T23:13:44.074 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:13:43 smithi002 ceph-mon[71816]: mds.? [v2:172.21.15.2:6836/3754333215,v1:172.21.15.2:6837/3754333215] up:boot 2022-05-13T23:13:44.074 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:13:43 smithi002 ceph-mon[71816]: fsmap 2 up:standby 2022-05-13T23:13:44.074 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:13:43 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2022-05-13T23:13:44.074 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:13:43 smithi002 ceph-mon[71816]: from='client.? 172.21.15.2:0/3362438555' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2022-05-13T23:13:44.299 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:13:43 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:13:44.300 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:13:43 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2022-05-13T23:13:44.300 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:13:43 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2022-05-13T23:13:44.300 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:13:43 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-13T23:13:44.300 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:13:43 smithi047 ceph-mon[77147]: Deploying daemon mds.g on smithi002 2022-05-13T23:13:44.300 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:13:43 smithi047 ceph-mon[77147]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 74 MiB used, 1.0 TiB / 1.0 TiB avail 2022-05-13T23:13:44.301 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:13:43 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.2:6836/3754333215,v1:172.21.15.2:6837/3754333215] up:boot 2022-05-13T23:13:44.301 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:13:43 smithi047 ceph-mon[77147]: fsmap 2 up:standby 2022-05-13T23:13:44.301 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:13:43 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2022-05-13T23:13:44.301 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:13:43 smithi047 ceph-mon[77147]: from='client.? 172.21.15.2:0/3362438555' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2022-05-13T23:13:44.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:13:43 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:13:44.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:13:43 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2022-05-13T23:13:44.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:13:43 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2022-05-13T23:13:44.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:13:43 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-13T23:13:44.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:13:43 smithi115 ceph-mon[78593]: Deploying daemon mds.g on smithi002 2022-05-13T23:13:44.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:13:43 smithi115 ceph-mon[78593]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 74 MiB used, 1.0 TiB / 1.0 TiB avail 2022-05-13T23:13:44.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:13:43 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.2:6836/3754333215,v1:172.21.15.2:6837/3754333215] up:boot 2022-05-13T23:13:44.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:13:43 smithi115 ceph-mon[78593]: fsmap 2 up:standby 2022-05-13T23:13:44.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:13:43 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2022-05-13T23:13:44.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:13:43 smithi115 ceph-mon[78593]: from='client.? 172.21.15.2:0/3362438555' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2022-05-13T23:13:45.746 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-05-13T23:13:46.145 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:46.142+0000 7f491719d700 1 -- 172.21.15.2:0/2513846163 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f49100fd0c0 msgr2=0x7f4910103980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:13:46.146 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:46.142+0000 7f491719d700 1 --2- 172.21.15.2:0/2513846163 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f49100fd0c0 0x7f4910103980 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f4908009be0 tx=0x7f4908004d10 comp rx=0 tx=0).stop 2022-05-13T23:13:46.146 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:46.142+0000 7f491719d700 1 -- 172.21.15.2:0/2513846163 shutdown_connections 2022-05-13T23:13:46.146 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:46.142+0000 7f491719d700 1 --2- 172.21.15.2:0/2513846163 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f49100fd0c0 0x7f4910103980 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:13:46.146 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:46.142+0000 7f491719d700 1 --2- 172.21.15.2:0/2513846163 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f49100fb490 0x7f49100fb8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:13:46.147 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:46.142+0000 7f491719d700 1 --2- 172.21.15.2:0/2513846163 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f49100fc730 0x7f49100fcb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:13:46.147 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:46.142+0000 7f491719d700 1 -- 172.21.15.2:0/2513846163 >> 172.21.15.2:0/2513846163 conn(0x7f49100f6c40 msgr2=0x7f49100f90b0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-05-13T23:13:46.147 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:46.143+0000 7f491719d700 1 -- 172.21.15.2:0/2513846163 shutdown_connections 2022-05-13T23:13:46.147 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:46.143+0000 7f491719d700 1 -- 172.21.15.2:0/2513846163 wait complete. 2022-05-13T23:13:46.150 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:46.144+0000 7f491719d700 1 Processor -- start 2022-05-13T23:13:46.150 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:46.145+0000 7f491719d700 1 -- start start 2022-05-13T23:13:46.150 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:46.145+0000 7f491719d700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f49100fb490 0x7f491010c170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:13:46.150 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:46.145+0000 7f491719d700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f49100fc730 0x7f491010c6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:13:46.151 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:46.145+0000 7f491719d700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f49100fd0c0 0x7f4910113740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:13:46.151 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:46.145+0000 7f491719d700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4910113c80 con 0x7f49100fd0c0 2022-05-13T23:13:46.151 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:46.145+0000 7f491719d700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4910113df0 con 0x7f49100fb490 2022-05-13T23:13:46.151 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:46.145+0000 7f491719d700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4910113fb0 con 0x7f49100fc730 2022-05-13T23:13:46.151 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:46.145+0000 7f491619b700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f49100fb490 0x7f491010c170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:13:46.152 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:46.146+0000 7f491619b700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f49100fb490 0x7f491010c170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.2:46982/0 (socket says 172.21.15.2:46982) 2022-05-13T23:13:46.152 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:46.146+0000 7f491619b700 1 -- 172.21.15.2:0/3608313770 learned_addr learned my addr 172.21.15.2:0/3608313770 (peer_addr_for_me v2:172.21.15.2:0/0) 2022-05-13T23:13:46.152 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:46.146+0000 7f491699c700 1 --2- 172.21.15.2:0/3608313770 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f49100fd0c0 0x7f4910113740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:13:46.152 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:46.146+0000 7f491599a700 1 --2- 172.21.15.2:0/3608313770 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f49100fc730 0x7f491010c6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:13:46.153 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:46.146+0000 7f491619b700 1 -- 172.21.15.2:0/3608313770 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f49100fc730 msgr2=0x7f491010c6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:13:46.153 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:46.146+0000 7f491619b700 1 --2- 172.21.15.2:0/3608313770 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f49100fc730 0x7f491010c6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:13:46.153 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:46.146+0000 7f491619b700 1 -- 172.21.15.2:0/3608313770 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f49100fd0c0 msgr2=0x7f4910113740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:13:46.153 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:46.146+0000 7f491619b700 1 --2- 172.21.15.2:0/3608313770 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f49100fd0c0 0x7f4910113740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:13:46.153 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:46.146+0000 7f491619b700 1 -- 172.21.15.2:0/3608313770 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49080097c0 con 0x7f49100fb490 2022-05-13T23:13:46.154 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:46.146+0000 7f491619b700 1 --2- 172.21.15.2:0/3608313770 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f49100fb490 0x7f491010c170 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f490c00e940 tx=0x7f490c00b040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-05-13T23:13:46.154 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:46.147+0000 7f4906ffd700 1 -- 172.21.15.2:0/3608313770 <== mon.1 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f490c007b40 con 0x7f49100fb490 2022-05-13T23:13:46.154 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:46.147+0000 7f491719d700 1 -- 172.21.15.2:0/3608313770 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4910114230 con 0x7f49100fb490 2022-05-13T23:13:46.154 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:46.147+0000 7f4906ffd700 1 -- 172.21.15.2:0/3608313770 <== mon.1 v2:172.21.15.47:3300/0 2 ==== config(26 keys) v1 ==== 1087+0+0 (secure 0 0 0) 0x7f490c00f040 con 0x7f49100fb490 2022-05-13T23:13:46.155 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:46.147+0000 7f4906ffd700 1 -- 172.21.15.2:0/3608313770 <== mon.1 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f490c014890 con 0x7f49100fb490 2022-05-13T23:13:46.155 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:46.147+0000 7f491719d700 1 -- 172.21.15.2:0/3608313770 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f49101147f0 con 0x7f49100fb490 2022-05-13T23:13:46.156 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:46.148+0000 7f491719d700 1 -- 172.21.15.2:0/3608313770 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4910106d30 con 0x7f49100fb490 2022-05-13T23:13:46.156 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:46.150+0000 7f4906ffd700 1 -- 172.21.15.2:0/3608313770 <== mon.1 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 16) v1 ==== 125136+0+0 (secure 0 0 0) 0x7f490c01b030 con 0x7f49100fb490 2022-05-13T23:13:46.156 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:46.151+0000 7f4906ffd700 1 --2- 172.21.15.2:0/3608313770 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f48fc096ce0 0x7f48fc099190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:13:46.157 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:46.151+0000 7f4906ffd700 1 -- 172.21.15.2:0/3608313770 <== mon.1 v2:172.21.15.47:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 7123+0+0 (secure 0 0 0) 0x7f490c0beb80 con 0x7f49100fb490 2022-05-13T23:13:46.157 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:46.151+0000 7f491599a700 1 --2- 172.21.15.2:0/3608313770 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f48fc096ce0 0x7f48fc099190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:13:46.157 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:46.152+0000 7f491599a700 1 --2- 172.21.15.2:0/3608313770 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f48fc096ce0 0x7f48fc099190 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f4900004790 tx=0x7f490000b040 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-05-13T23:13:46.157 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:46.153+0000 7f4906ffd700 1 -- 172.21.15.2:0/3608313770 <== mon.1 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+165215 (secure 0 0 0) 0x7f490c04b570 con 0x7f49100fb490 2022-05-13T23:13:46.274 INFO:journalctl@ceph.mds.g.smithi002.stdout:May 13 23:13:45 smithi002 systemd[1]: Starting Ceph mds.g for 16cf5f88-d311-11ec-be40-001a4aab830c... 2022-05-13T23:13:46.399 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:46.396+0000 7f491719d700 1 -- 172.21.15.2:0/3608313770 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f49101028d0 con 0x7f49100fb490 2022-05-13T23:13:46.399 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:46.397+0000 7f4906ffd700 1 -- 172.21.15.2:0/3608313770 <== mon.1 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) v1 ==== 93+0+2008 (secure 0 0 0) 0x7f490c08c210 con 0x7f49100fb490 2022-05-13T23:13:46.400 INFO:teuthology.orchestra.run.smithi002.stdout: 2022-05-13T23:13:46.400 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":3,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24460,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.2:6835/1341618941","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6834","nonce":1341618941},{"type":"v1","addr":"172.21.15.2:6835","nonce":1341618941}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24494,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.2:6837/3754333215","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6836","nonce":3754333215},{"type":"v1","addr":"172.21.15.2:6837","nonce":3754333215}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[]} 2022-05-13T23:13:46.402 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:46.399+0000 7f491719d700 1 -- 172.21.15.2:0/3608313770 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f48fc096ce0 msgr2=0x7f48fc099190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:13:46.403 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:46.400+0000 7f491719d700 1 --2- 172.21.15.2:0/3608313770 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f48fc096ce0 0x7f48fc099190 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f4900004790 tx=0x7f490000b040 comp rx=0 tx=0).stop 2022-05-13T23:13:46.403 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:46.400+0000 7f491719d700 1 -- 172.21.15.2:0/3608313770 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f49100fb490 msgr2=0x7f491010c170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:13:46.403 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:46.400+0000 7f491719d700 1 --2- 172.21.15.2:0/3608313770 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f49100fb490 0x7f491010c170 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f490c00e940 tx=0x7f490c00b040 comp rx=0 tx=0).stop 2022-05-13T23:13:46.403 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:46.400+0000 7f491719d700 1 -- 172.21.15.2:0/3608313770 shutdown_connections 2022-05-13T23:13:46.403 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:46.400+0000 7f491719d700 1 --2- 172.21.15.2:0/3608313770 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f48fc096ce0 0x7f48fc099190 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:13:46.404 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:46.400+0000 7f491719d700 1 --2- 172.21.15.2:0/3608313770 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f49100fd0c0 0x7f4910113740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:13:46.404 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:46.400+0000 7f491719d700 1 --2- 172.21.15.2:0/3608313770 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f49100fb490 0x7f491010c170 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:13:46.404 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:46.400+0000 7f491719d700 1 --2- 172.21.15.2:0/3608313770 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f49100fc730 0x7f491010c6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:13:46.404 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:46.400+0000 7f491719d700 1 -- 172.21.15.2:0/3608313770 >> 172.21.15.2:0/3608313770 conn(0x7f49100f6c40 msgr2=0x7f49100ffbd0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-05-13T23:13:46.405 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:46.400+0000 7f491719d700 1 -- 172.21.15.2:0/3608313770 shutdown_connections 2022-05-13T23:13:46.405 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:46.400+0000 7f491719d700 1 -- 172.21.15.2:0/3608313770 wait complete. 2022-05-13T23:13:46.405 INFO:teuthology.orchestra.run.smithi002.stderr:dumped fsmap epoch 3 2022-05-13T23:13:46.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:13:46 smithi047 ceph-mon[77147]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 74 MiB used, 1.0 TiB / 1.0 TiB avail 2022-05-13T23:13:46.572 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:13:46 smithi002 ceph-mon[71816]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 74 MiB used, 1.0 TiB / 1.0 TiB avail 2022-05-13T23:13:46.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:13:46 smithi115 ceph-mon[78593]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 74 MiB used, 1.0 TiB / 1.0 TiB avail 2022-05-13T23:13:47.322 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:13:47 smithi002 ceph-mon[71816]: from='client.? 172.21.15.2:0/3608313770' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2022-05-13T23:13:47.323 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:13:47 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:13:47.323 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:13:47 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2022-05-13T23:13:47.323 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:13:47 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2022-05-13T23:13:47.324 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:13:47 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-13T23:13:47.324 INFO:journalctl@ceph.mds.g.smithi002.stdout:May 13 23:13:47 smithi002 ceph-mds[109507]: -- 172.21.15.2:0/499685574 <== mon.1 v2:172.21.15.47:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x561fe2302340 con 0x561fe22dac00 2022-05-13T23:13:47.324 INFO:journalctl@ceph.mds.g.smithi002.stdout:May 13 23:13:47 smithi002 conmon[109483]: starting mds.g at 2022-05-13T23:13:47.324 INFO:journalctl@ceph.mds.g.smithi002.stdout:May 13 23:13:47 smithi002 bash[109271]: cdd0c1e5ef65b7a823a34767058749c717f5c2a63fb0dda545bb39fa7d61a79f 2022-05-13T23:13:47.325 INFO:journalctl@ceph.mds.g.smithi002.stdout:May 13 23:13:47 smithi002 systemd[1]: Started Ceph mds.g for 16cf5f88-d311-11ec-be40-001a4aab830c. 2022-05-13T23:13:47.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:13:47 smithi047 ceph-mon[77147]: from='client.? 172.21.15.2:0/3608313770' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2022-05-13T23:13:47.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:13:47 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:13:47.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:13:47 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2022-05-13T23:13:47.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:13:47 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2022-05-13T23:13:47.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:13:47 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-13T23:13:47.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:13:47 smithi115 ceph-mon[78593]: from='client.? 172.21.15.2:0/3608313770' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2022-05-13T23:13:47.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:13:47 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:13:47.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:13:47 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2022-05-13T23:13:47.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:13:47 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2022-05-13T23:13:47.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:13:47 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-13T23:13:48.431 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-05-13T23:13:48.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:13:48 smithi047 ceph-mon[77147]: Deploying daemon mds.j on smithi002 2022-05-13T23:13:48.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:13:48 smithi047 ceph-mon[77147]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 74 MiB used, 1.0 TiB / 1.0 TiB avail 2022-05-13T23:13:48.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:13:48 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.2:6838/1347050951,v1:172.21.15.2:6839/1347050951] up:boot 2022-05-13T23:13:48.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:13:48 smithi047 ceph-mon[77147]: fsmap 3 up:standby 2022-05-13T23:13:48.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:13:48 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2022-05-13T23:13:48.563 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:13:48 smithi002 ceph-mon[71816]: Deploying daemon mds.j on smithi002 2022-05-13T23:13:48.563 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:13:48 smithi002 ceph-mon[71816]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 74 MiB used, 1.0 TiB / 1.0 TiB avail 2022-05-13T23:13:48.563 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:13:48 smithi002 ceph-mon[71816]: mds.? [v2:172.21.15.2:6838/1347050951,v1:172.21.15.2:6839/1347050951] up:boot 2022-05-13T23:13:48.564 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:13:48 smithi002 ceph-mon[71816]: fsmap 3 up:standby 2022-05-13T23:13:48.564 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:13:48 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2022-05-13T23:13:48.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:13:48 smithi115 ceph-mon[78593]: Deploying daemon mds.j on smithi002 2022-05-13T23:13:48.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:13:48 smithi115 ceph-mon[78593]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 74 MiB used, 1.0 TiB / 1.0 TiB avail 2022-05-13T23:13:48.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:13:48 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.2:6838/1347050951,v1:172.21.15.2:6839/1347050951] up:boot 2022-05-13T23:13:48.612 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:13:48 smithi115 ceph-mon[78593]: fsmap 3 up:standby 2022-05-13T23:13:48.612 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:13:48 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2022-05-13T23:13:48.670 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:48.668+0000 7f53dc9fe700 1 -- 172.21.15.2:0/139268877 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f53d80ff310 msgr2=0x7f53d8105bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:13:48.671 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:48.668+0000 7f53dc9fe700 1 --2- 172.21.15.2:0/139268877 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f53d80ff310 0x7f53d8105bd0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f53d0009be0 tx=0x7f53d0004d10 comp rx=0 tx=0).stop 2022-05-13T23:13:48.671 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:48.668+0000 7f53dc9fe700 1 -- 172.21.15.2:0/139268877 shutdown_connections 2022-05-13T23:13:48.671 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:48.668+0000 7f53dc9fe700 1 --2- 172.21.15.2:0/139268877 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f53d80ff310 0x7f53d8105bd0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:13:48.672 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:48.668+0000 7f53dc9fe700 1 --2- 172.21.15.2:0/139268877 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f53d80fd6e0 0x7f53d80fdaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:13:48.672 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:48.668+0000 7f53dc9fe700 1 --2- 172.21.15.2:0/139268877 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f53d80fe980 0x7f53d80fedd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:13:48.672 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:48.668+0000 7f53dc9fe700 1 -- 172.21.15.2:0/139268877 >> 172.21.15.2:0/139268877 conn(0x7f53d80f8ed0 msgr2=0x7f53d80fb300 unknown :-1 s=STATE_NONE l=0).mark_down 2022-05-13T23:13:48.672 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:48.668+0000 7f53dc9fe700 1 -- 172.21.15.2:0/139268877 shutdown_connections 2022-05-13T23:13:48.673 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:48.668+0000 7f53dc9fe700 1 -- 172.21.15.2:0/139268877 wait complete. 2022-05-13T23:13:48.675 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:48.669+0000 7f53dc9fe700 1 Processor -- start 2022-05-13T23:13:48.676 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:48.670+0000 7f53dc9fe700 1 -- start start 2022-05-13T23:13:48.676 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:48.670+0000 7f53dc9fe700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f53d80fd6e0 0x7f53d810e330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:13:48.676 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:48.670+0000 7f53dc9fe700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f53d80fe980 0x7f53d810e870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:13:48.676 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:48.670+0000 7f53dc9fe700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f53d80ff310 0x7f53d8115900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:13:48.677 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:48.670+0000 7f53dc9fe700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53d8115e40 con 0x7f53d80fd6e0 2022-05-13T23:13:48.677 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:48.670+0000 7f53dc9fe700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53d8115fb0 con 0x7f53d80fe980 2022-05-13T23:13:48.677 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:48.670+0000 7f53dc9fe700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53d8116170 con 0x7f53d80ff310 2022-05-13T23:13:48.677 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:48.670+0000 7f53d6d9d700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f53d80fd6e0 0x7f53d810e330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:13:48.678 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:48.670+0000 7f53d6d9d700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f53d80fd6e0 0x7f53d810e330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:48868/0 (socket says 172.21.15.2:48868) 2022-05-13T23:13:48.678 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:48.670+0000 7f53d6d9d700 1 -- 172.21.15.2:0/2407913785 learned_addr learned my addr 172.21.15.2:0/2407913785 (peer_addr_for_me v2:172.21.15.2:0/0) 2022-05-13T23:13:48.678 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:48.670+0000 7f53d659c700 1 --2- 172.21.15.2:0/2407913785 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f53d80fe980 0x7f53d810e870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:13:48.678 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:48.670+0000 7f53d759e700 1 --2- 172.21.15.2:0/2407913785 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f53d80ff310 0x7f53d8115900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:13:48.678 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:48.671+0000 7f53d6d9d700 1 -- 172.21.15.2:0/2407913785 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f53d80ff310 msgr2=0x7f53d8115900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:13:48.679 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:48.671+0000 7f53d6d9d700 1 --2- 172.21.15.2:0/2407913785 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f53d80ff310 0x7f53d8115900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:13:48.679 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:48.671+0000 7f53d6d9d700 1 -- 172.21.15.2:0/2407913785 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f53d80fe980 msgr2=0x7f53d810e870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:13:48.679 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:48.671+0000 7f53d6d9d700 1 --2- 172.21.15.2:0/2407913785 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f53d80fe980 0x7f53d810e870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:13:48.679 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:48.671+0000 7f53d6d9d700 1 -- 172.21.15.2:0/2407913785 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53d00097c0 con 0x7f53d80fd6e0 2022-05-13T23:13:48.679 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:48.671+0000 7f53d759e700 1 --2- 172.21.15.2:0/2407913785 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f53d80ff310 0x7f53d8115900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-05-13T23:13:48.680 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:48.671+0000 7f53d6d9d700 1 --2- 172.21.15.2:0/2407913785 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f53d80fd6e0 0x7f53d810e330 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f53c800e940 tx=0x7f53c800b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-05-13T23:13:48.680 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:48.671+0000 7f53bf7fe700 1 -- 172.21.15.2:0/2407913785 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53c8007b40 con 0x7f53d80fd6e0 2022-05-13T23:13:48.680 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:48.672+0000 7f53bf7fe700 1 -- 172.21.15.2:0/2407913785 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1087+0+0 (secure 0 0 0) 0x7f53c800f040 con 0x7f53d80fd6e0 2022-05-13T23:13:48.680 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:48.672+0000 7f53dc9fe700 1 -- 172.21.15.2:0/2407913785 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f53d81163f0 con 0x7f53d80fd6e0 2022-05-13T23:13:48.681 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:48.672+0000 7f53bf7fe700 1 -- 172.21.15.2:0/2407913785 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53c8014890 con 0x7f53d80fd6e0 2022-05-13T23:13:48.681 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:48.672+0000 7f53dc9fe700 1 -- 172.21.15.2:0/2407913785 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f53d8116960 con 0x7f53d80fd6e0 2022-05-13T23:13:48.681 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:48.673+0000 7f53dc9fe700 1 -- 172.21.15.2:0/2407913785 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f53d8108ee0 con 0x7f53d80fd6e0 2022-05-13T23:13:48.682 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:48.675+0000 7f53bf7fe700 1 -- 172.21.15.2:0/2407913785 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 16) v1 ==== 125136+0+0 (secure 0 0 0) 0x7f53c801b030 con 0x7f53d80fd6e0 2022-05-13T23:13:48.683 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:48.676+0000 7f53bf7fe700 1 --2- 172.21.15.2:0/2407913785 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f53c0096ce0 0x7f53c0099190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:13:48.683 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:48.676+0000 7f53bf7fe700 1 -- 172.21.15.2:0/2407913785 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 7123+0+0 (secure 0 0 0) 0x7f53c80beba0 con 0x7f53d80fd6e0 2022-05-13T23:13:48.683 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:48.676+0000 7f53d659c700 1 --2- 172.21.15.2:0/2407913785 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f53c0096ce0 0x7f53c0099190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:13:48.683 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:48.677+0000 7f53d659c700 1 --2- 172.21.15.2:0/2407913785 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f53c0096ce0 0x7f53c0099190 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f53cc004790 tx=0x7f53cc00b040 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-05-13T23:13:48.685 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:48.683+0000 7f53bf7fe700 1 -- 172.21.15.2:0/2407913785 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+165215 (secure 0 0 0) 0x7f53c804b570 con 0x7f53d80fd6e0 2022-05-13T23:13:48.926 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:48.923+0000 7f53dc9fe700 1 -- 172.21.15.2:0/2407913785 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f53d802a4f0 con 0x7f53d80fd6e0 2022-05-13T23:13:48.927 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:48.924+0000 7f53bf7fe700 1 -- 172.21.15.2:0/2407913785 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 4 v4) v1 ==== 93+0+2762 (secure 0 0 0) 0x7f53c8019ae0 con 0x7f53d80fd6e0 2022-05-13T23:13:48.927 INFO:teuthology.orchestra.run.smithi002.stdout: 2022-05-13T23:13:48.927 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":4,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24460,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.2:6835/1341618941","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6834","nonce":1341618941},{"type":"v1","addr":"172.21.15.2:6835","nonce":1341618941}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24490,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.2:6839/1347050951","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6838","nonce":1347050951},{"type":"v1","addr":"172.21.15.2:6839","nonce":1347050951}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":24494,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.2:6837/3754333215","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6836","nonce":3754333215},{"type":"v1","addr":"172.21.15.2:6837","nonce":3754333215}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[]} 2022-05-13T23:13:48.930 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:48.927+0000 7f53dc9fe700 1 -- 172.21.15.2:0/2407913785 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f53c0096ce0 msgr2=0x7f53c0099190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:13:48.930 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:48.927+0000 7f53dc9fe700 1 --2- 172.21.15.2:0/2407913785 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f53c0096ce0 0x7f53c0099190 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f53cc004790 tx=0x7f53cc00b040 comp rx=0 tx=0).stop 2022-05-13T23:13:48.930 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:48.927+0000 7f53dc9fe700 1 -- 172.21.15.2:0/2407913785 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f53d80fd6e0 msgr2=0x7f53d810e330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:13:48.931 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:48.927+0000 7f53dc9fe700 1 --2- 172.21.15.2:0/2407913785 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f53d80fd6e0 0x7f53d810e330 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f53c800e940 tx=0x7f53c800b040 comp rx=0 tx=0).stop 2022-05-13T23:13:48.934 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:48.928+0000 7f53dc9fe700 1 -- 172.21.15.2:0/2407913785 shutdown_connections 2022-05-13T23:13:48.935 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:48.928+0000 7f53dc9fe700 1 --2- 172.21.15.2:0/2407913785 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f53d80ff310 0x7f53d8115900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:13:48.935 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:48.928+0000 7f53dc9fe700 1 --2- 172.21.15.2:0/2407913785 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f53c0096ce0 0x7f53c0099190 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:13:48.935 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:48.928+0000 7f53dc9fe700 1 --2- 172.21.15.2:0/2407913785 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f53d80fd6e0 0x7f53d810e330 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:13:48.936 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:48.928+0000 7f53dc9fe700 1 --2- 172.21.15.2:0/2407913785 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f53d80fe980 0x7f53d810e870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:13:48.936 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:48.928+0000 7f53dc9fe700 1 -- 172.21.15.2:0/2407913785 >> 172.21.15.2:0/2407913785 conn(0x7f53d80f8ed0 msgr2=0x7f53d8101da0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-05-13T23:13:48.937 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:48.928+0000 7f53dc9fe700 1 -- 172.21.15.2:0/2407913785 shutdown_connections 2022-05-13T23:13:48.937 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:48.928+0000 7f53dc9fe700 1 -- 172.21.15.2:0/2407913785 wait complete. 2022-05-13T23:13:48.937 INFO:teuthology.orchestra.run.smithi002.stderr:dumped fsmap epoch 4 2022-05-13T23:13:49.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:13:49 smithi047 ceph-mon[77147]: from='client.? 172.21.15.2:0/2407913785' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2022-05-13T23:13:49.573 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:13:49 smithi002 ceph-mon[71816]: from='client.? 172.21.15.2:0/2407913785' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2022-05-13T23:13:49.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:13:49 smithi115 ceph-mon[78593]: from='client.? 172.21.15.2:0/2407913785' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2022-05-13T23:13:50.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:13:50 smithi047 ceph-mon[77147]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 74 MiB used, 1.0 TiB / 1.0 TiB avail 2022-05-13T23:13:50.573 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:13:50 smithi002 ceph-mon[71816]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 74 MiB used, 1.0 TiB / 1.0 TiB avail 2022-05-13T23:13:50.574 INFO:journalctl@ceph.mds.j.smithi002.stdout:May 13 23:13:50 smithi002 systemd[1]: Starting Ceph mds.j for 16cf5f88-d311-11ec-be40-001a4aab830c... 2022-05-13T23:13:50.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:13:50 smithi115 ceph-mon[78593]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 74 MiB used, 1.0 TiB / 1.0 TiB avail 2022-05-13T23:13:50.958 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-05-13T23:13:51.197 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:51.194+0000 7fbb6a19d700 1 -- 172.21.15.2:0/2248630900 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fbb640fd0c0 msgr2=0x7fbb64103980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:13:51.198 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:51.194+0000 7fbb6a19d700 1 --2- 172.21.15.2:0/2248630900 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fbb640fd0c0 0x7fbb64103980 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fbb5c009be0 tx=0x7fbb5c004d10 comp rx=0 tx=0).stop 2022-05-13T23:13:51.199 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:51.194+0000 7fbb6a19d700 1 -- 172.21.15.2:0/2248630900 shutdown_connections 2022-05-13T23:13:51.199 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:51.194+0000 7fbb6a19d700 1 --2- 172.21.15.2:0/2248630900 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fbb640fd0c0 0x7fbb64103980 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:13:51.199 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:51.194+0000 7fbb6a19d700 1 --2- 172.21.15.2:0/2248630900 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fbb640fb490 0x7fbb640fb8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:13:51.199 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:51.194+0000 7fbb6a19d700 1 --2- 172.21.15.2:0/2248630900 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fbb640fc730 0x7fbb640fcb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:13:51.200 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:51.194+0000 7fbb6a19d700 1 -- 172.21.15.2:0/2248630900 >> 172.21.15.2:0/2248630900 conn(0x7fbb640f6c40 msgr2=0x7fbb640f90b0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-05-13T23:13:51.200 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:51.194+0000 7fbb6a19d700 1 -- 172.21.15.2:0/2248630900 shutdown_connections 2022-05-13T23:13:51.200 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:51.194+0000 7fbb6a19d700 1 -- 172.21.15.2:0/2248630900 wait complete. 2022-05-13T23:13:51.200 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:51.195+0000 7fbb6a19d700 1 Processor -- start 2022-05-13T23:13:51.200 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:51.195+0000 7fbb6a19d700 1 -- start start 2022-05-13T23:13:51.203 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:51.196+0000 7fbb6a19d700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fbb640fb490 0x7fbb6410e340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:13:51.203 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:51.196+0000 7fbb6a19d700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fbb640fc730 0x7fbb6410e880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:13:51.203 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:51.196+0000 7fbb6a19d700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fbb640fd0c0 0x7fbb64115910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:13:51.204 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:51.196+0000 7fbb6a19d700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb64115e50 con 0x7fbb640fc730 2022-05-13T23:13:51.204 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:51.196+0000 7fbb6a19d700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb64115fc0 con 0x7fbb640fd0c0 2022-05-13T23:13:51.204 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:51.196+0000 7fbb6a19d700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb64116180 con 0x7fbb640fb490 2022-05-13T23:13:51.204 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:51.196+0000 7fbb6919b700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fbb640fb490 0x7fbb6410e340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:13:51.205 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:51.196+0000 7fbb6919b700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fbb640fb490 0x7fbb6410e340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.2:53270/0 (socket says 172.21.15.2:53270) 2022-05-13T23:13:51.205 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:51.196+0000 7fbb6919b700 1 -- 172.21.15.2:0/865372100 learned_addr learned my addr 172.21.15.2:0/865372100 (peer_addr_for_me v2:172.21.15.2:0/0) 2022-05-13T23:13:51.206 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:51.196+0000 7fbb6899a700 1 --2- 172.21.15.2:0/865372100 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fbb640fc730 0x7fbb6410e880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:13:51.206 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:51.196+0000 7fbb6999c700 1 --2- 172.21.15.2:0/865372100 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fbb640fd0c0 0x7fbb64115910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:13:51.206 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:51.196+0000 7fbb6919b700 1 -- 172.21.15.2:0/865372100 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fbb640fd0c0 msgr2=0x7fbb64115910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:13:51.206 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:51.196+0000 7fbb6919b700 1 --2- 172.21.15.2:0/865372100 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fbb640fd0c0 0x7fbb64115910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:13:51.207 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:51.196+0000 7fbb6919b700 1 -- 172.21.15.2:0/865372100 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fbb640fc730 msgr2=0x7fbb6410e880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:13:51.207 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:51.197+0000 7fbb6919b700 1 --2- 172.21.15.2:0/865372100 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fbb640fc730 0x7fbb6410e880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:13:51.207 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:51.197+0000 7fbb6919b700 1 -- 172.21.15.2:0/865372100 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb5c0097c0 con 0x7fbb640fb490 2022-05-13T23:13:51.207 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:51.197+0000 7fbb6999c700 1 --2- 172.21.15.2:0/865372100 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fbb640fd0c0 0x7fbb64115910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2022-05-13T23:13:51.208 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:51.197+0000 7fbb6919b700 1 --2- 172.21.15.2:0/865372100 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fbb640fb490 0x7fbb6410e340 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fbb6000e940 tx=0x7fbb6000b040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-05-13T23:13:51.208 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:51.197+0000 7fbb59ffb700 1 -- 172.21.15.2:0/865372100 <== mon.2 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb60007b40 con 0x7fbb640fb490 2022-05-13T23:13:51.208 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:51.197+0000 7fbb59ffb700 1 -- 172.21.15.2:0/865372100 <== mon.2 v2:172.21.15.115:3300/0 2 ==== config(26 keys) v1 ==== 1087+0+0 (secure 0 0 0) 0x7fbb6000f040 con 0x7fbb640fb490 2022-05-13T23:13:51.208 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:51.197+0000 7fbb6a19d700 1 -- 172.21.15.2:0/865372100 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbb64116400 con 0x7fbb640fb490 2022-05-13T23:13:51.208 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:51.197+0000 7fbb59ffb700 1 -- 172.21.15.2:0/865372100 <== mon.2 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb60014890 con 0x7fbb640fb490 2022-05-13T23:13:51.209 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:51.197+0000 7fbb6a19d700 1 -- 172.21.15.2:0/865372100 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbb64116970 con 0x7fbb640fb490 2022-05-13T23:13:51.209 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:51.198+0000 7fbb6a19d700 1 -- 172.21.15.2:0/865372100 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbb6406f950 con 0x7fbb640fb490 2022-05-13T23:13:51.210 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:51.202+0000 7fbb59ffb700 1 -- 172.21.15.2:0/865372100 <== mon.2 v2:172.21.15.115:3300/0 4 ==== mgrmap(e 16) v1 ==== 125136+0+0 (secure 0 0 0) 0x7fbb6001b030 con 0x7fbb640fb490 2022-05-13T23:13:51.210 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:51.203+0000 7fbb59ffb700 1 --2- 172.21.15.2:0/865372100 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7fbb50096ce0 0x7fbb50099190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:13:51.211 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:51.203+0000 7fbb59ffb700 1 -- 172.21.15.2:0/865372100 <== mon.2 v2:172.21.15.115:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 7123+0+0 (secure 0 0 0) 0x7fbb600bec40 con 0x7fbb640fb490 2022-05-13T23:13:51.211 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:51.203+0000 7fbb6899a700 1 --2- 172.21.15.2:0/865372100 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7fbb50096ce0 0x7fbb50099190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:13:51.211 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:51.204+0000 7fbb59ffb700 1 -- 172.21.15.2:0/865372100 <== mon.2 v2:172.21.15.115:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+165215 (secure 0 0 0) 0x7fbb6004b570 con 0x7fbb640fb490 2022-05-13T23:13:51.211 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:51.204+0000 7fbb6899a700 1 --2- 172.21.15.2:0/865372100 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7fbb50096ce0 0x7fbb50099190 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fbb54004790 tx=0x7fbb5400b040 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-05-13T23:13:51.311 INFO:journalctl@ceph.mds.j.smithi002.stdout:May 13 23:13:51 smithi002 ceph-mds[110142]: -- 172.21.15.2:0/501381595 <== mon.2 v2:172.21.15.115:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55a77867c340 con 0x55a778654c00 2022-05-13T23:13:51.311 INFO:journalctl@ceph.mds.j.smithi002.stdout:May 13 23:13:51 smithi002 conmon[110105]: starting mds.j at 2022-05-13T23:13:51.312 INFO:journalctl@ceph.mds.j.smithi002.stdout:May 13 23:13:51 smithi002 bash[109904]: 2d1f81d9f5e6f730e1c5a97823aa97e5e533bba8ba93e51e12144b95ec78b519 2022-05-13T23:13:51.470 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:51.468+0000 7fbb6a19d700 1 -- 172.21.15.2:0/865372100 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbb64101f50 con 0x7fbb640fb490 2022-05-13T23:13:51.471 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:51.469+0000 7fbb59ffb700 1 -- 172.21.15.2:0/865372100 <== mon.2 v2:172.21.15.115:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 5 v5) v1 ==== 93+0+3513 (secure 0 0 0) 0x7fbb60019dd0 con 0x7fbb640fb490 2022-05-13T23:13:51.472 INFO:teuthology.orchestra.run.smithi002.stdout: 2022-05-13T23:13:51.472 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":5,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24460,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.2:6835/1341618941","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6834","nonce":1341618941},{"type":"v1","addr":"172.21.15.2:6835","nonce":1341618941}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24490,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.2:6839/1347050951","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6838","nonce":1347050951},{"type":"v1","addr":"172.21.15.2:6839","nonce":1347050951}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":24494,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.2:6837/3754333215","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6836","nonce":3754333215},{"type":"v1","addr":"172.21.15.2:6837","nonce":3754333215}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24530,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.2:6841/522559555","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6840","nonce":522559555},{"type":"v1","addr":"172.21.15.2:6841","nonce":522559555}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5}],"filesystems":[]} 2022-05-13T23:13:51.476 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:51.472+0000 7fbb6a19d700 1 -- 172.21.15.2:0/865372100 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7fbb50096ce0 msgr2=0x7fbb50099190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:13:51.476 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:51.472+0000 7fbb6a19d700 1 --2- 172.21.15.2:0/865372100 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7fbb50096ce0 0x7fbb50099190 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fbb54004790 tx=0x7fbb5400b040 comp rx=0 tx=0).stop 2022-05-13T23:13:51.476 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:51.472+0000 7fbb6a19d700 1 -- 172.21.15.2:0/865372100 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fbb640fb490 msgr2=0x7fbb6410e340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:13:51.476 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:51.472+0000 7fbb6a19d700 1 --2- 172.21.15.2:0/865372100 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fbb640fb490 0x7fbb6410e340 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fbb6000e940 tx=0x7fbb6000b040 comp rx=0 tx=0).stop 2022-05-13T23:13:51.477 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:51.472+0000 7fbb6a19d700 1 -- 172.21.15.2:0/865372100 shutdown_connections 2022-05-13T23:13:51.477 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:51.472+0000 7fbb6a19d700 1 --2- 172.21.15.2:0/865372100 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fbb640fd0c0 0x7fbb64115910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:13:51.477 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:51.472+0000 7fbb6a19d700 1 --2- 172.21.15.2:0/865372100 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fbb640fb490 0x7fbb6410e340 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:13:51.477 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:51.472+0000 7fbb6a19d700 1 --2- 172.21.15.2:0/865372100 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7fbb50096ce0 0x7fbb50099190 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:13:51.478 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:51.472+0000 7fbb6a19d700 1 --2- 172.21.15.2:0/865372100 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fbb640fc730 0x7fbb6410e880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:13:51.478 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:51.472+0000 7fbb6a19d700 1 -- 172.21.15.2:0/865372100 >> 172.21.15.2:0/865372100 conn(0x7fbb640f6c40 msgr2=0x7fbb640f8690 unknown :-1 s=STATE_NONE l=0).mark_down 2022-05-13T23:13:51.478 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:51.472+0000 7fbb6a19d700 1 -- 172.21.15.2:0/865372100 shutdown_connections 2022-05-13T23:13:51.478 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:51.472+0000 7fbb6a19d700 1 -- 172.21.15.2:0/865372100 wait complete. 2022-05-13T23:13:51.479 INFO:teuthology.orchestra.run.smithi002.stderr:dumped fsmap epoch 5 2022-05-13T23:13:51.572 INFO:journalctl@ceph.mds.j.smithi002.stdout:May 13 23:13:51 smithi002 systemd[1]: Started Ceph mds.j for 16cf5f88-d311-11ec-be40-001a4aab830c. 2022-05-13T23:13:52.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:13:52 smithi047 ceph-mon[77147]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 74 MiB used, 1.0 TiB / 1.0 TiB avail 2022-05-13T23:13:52.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:13:52 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.2:6840/522559555,v1:172.21.15.2:6841/522559555] up:boot 2022-05-13T23:13:52.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:13:52 smithi047 ceph-mon[77147]: fsmap 4 up:standby 2022-05-13T23:13:52.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:13:52 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2022-05-13T23:13:52.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:13:52 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:13:52.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:13:52 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2022-05-13T23:13:52.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:13:52 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2022-05-13T23:13:52.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:13:52 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-13T23:13:52.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:13:52 smithi047 ceph-mon[77147]: Deploying daemon mds.b on smithi047 2022-05-13T23:13:52.545 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:13:52 smithi047 ceph-mon[77147]: from='client.? 172.21.15.2:0/865372100' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2022-05-13T23:13:52.572 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:13:52 smithi002 ceph-mon[71816]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 74 MiB used, 1.0 TiB / 1.0 TiB avail 2022-05-13T23:13:52.573 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:13:52 smithi002 ceph-mon[71816]: mds.? [v2:172.21.15.2:6840/522559555,v1:172.21.15.2:6841/522559555] up:boot 2022-05-13T23:13:52.573 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:13:52 smithi002 ceph-mon[71816]: fsmap 4 up:standby 2022-05-13T23:13:52.573 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:13:52 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2022-05-13T23:13:52.573 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:13:52 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:13:52.574 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:13:52 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2022-05-13T23:13:52.574 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:13:52 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2022-05-13T23:13:52.574 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:13:52 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-13T23:13:52.575 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:13:52 smithi002 ceph-mon[71816]: Deploying daemon mds.b on smithi047 2022-05-13T23:13:52.575 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:13:52 smithi002 ceph-mon[71816]: from='client.? 172.21.15.2:0/865372100' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2022-05-13T23:13:52.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:13:52 smithi115 ceph-mon[78593]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 74 MiB used, 1.0 TiB / 1.0 TiB avail 2022-05-13T23:13:52.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:13:52 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.2:6840/522559555,v1:172.21.15.2:6841/522559555] up:boot 2022-05-13T23:13:52.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:13:52 smithi115 ceph-mon[78593]: fsmap 4 up:standby 2022-05-13T23:13:52.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:13:52 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2022-05-13T23:13:52.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:13:52 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:13:52.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:13:52 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2022-05-13T23:13:52.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:13:52 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2022-05-13T23:13:52.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:13:52 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-13T23:13:52.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:13:52 smithi115 ceph-mon[78593]: Deploying daemon mds.b on smithi047 2022-05-13T23:13:52.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:13:52 smithi115 ceph-mon[78593]: from='client.? 172.21.15.2:0/865372100' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2022-05-13T23:13:53.291 INFO:journalctl@ceph.mds.b.smithi047.stdout:May 13 23:13:53 smithi047 systemd[1]: Starting Ceph mds.b for 16cf5f88-d311-11ec-be40-001a4aab830c... 2022-05-13T23:13:53.501 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-05-13T23:13:53.736 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:53.727+0000 7fcc9ab9d700 1 -- 172.21.15.2:0/3275965880 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fcc940fa390 msgr2=0x7fcc940fa7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:13:53.737 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:53.727+0000 7fcc9ab9d700 1 --2- 172.21.15.2:0/3275965880 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fcc940fa390 0x7fcc940fa7a0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fcc90004870 tx=0x7fcc90009cb0 comp rx=0 tx=0).stop 2022-05-13T23:13:53.737 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:53.727+0000 7fcc9ab9d700 1 -- 172.21.15.2:0/3275965880 shutdown_connections 2022-05-13T23:13:53.737 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:53.727+0000 7fcc9ab9d700 1 --2- 172.21.15.2:0/3275965880 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcc940fbfc0 0x7fcc94102880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:13:53.738 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:53.727+0000 7fcc9ab9d700 1 --2- 172.21.15.2:0/3275965880 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fcc940fa390 0x7fcc940fa7a0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:13:53.738 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:53.727+0000 7fcc9ab9d700 1 --2- 172.21.15.2:0/3275965880 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fcc940fb630 0x7fcc940fba80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:13:53.738 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:53.727+0000 7fcc9ab9d700 1 -- 172.21.15.2:0/3275965880 >> 172.21.15.2:0/3275965880 conn(0x7fcc940f5b80 msgr2=0x7fcc940f7fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-05-13T23:13:53.739 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:53.727+0000 7fcc9ab9d700 1 -- 172.21.15.2:0/3275965880 shutdown_connections 2022-05-13T23:13:53.739 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:53.727+0000 7fcc9ab9d700 1 -- 172.21.15.2:0/3275965880 wait complete. 2022-05-13T23:13:53.739 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:53.728+0000 7fcc9ab9d700 1 Processor -- start 2022-05-13T23:13:53.740 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:53.728+0000 7fcc9ab9d700 1 -- start start 2022-05-13T23:13:53.740 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:53.729+0000 7fcc9ab9d700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcc940fa390 0x7fcc94109cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:13:53.740 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:53.729+0000 7fcc9ab9d700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fcc940fb630 0x7fcc9410a230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:13:53.740 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:53.729+0000 7fcc9ab9d700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fcc940fbfc0 0x7fcc94106830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:13:53.740 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:53.729+0000 7fcc9ab9d700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc94106f40 con 0x7fcc940fa390 2022-05-13T23:13:53.741 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:53.729+0000 7fcc9ab9d700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc941070b0 con 0x7fcc940fb630 2022-05-13T23:13:53.741 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:53.729+0000 7fcc9ab9d700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc94107270 con 0x7fcc940fbfc0 2022-05-13T23:13:53.741 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:53.729+0000 7fcc99b9b700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcc940fa390 0x7fcc94109cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:13:53.741 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:53.729+0000 7fcc9939a700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fcc940fb630 0x7fcc9410a230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:13:53.742 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:53.729+0000 7fcc99b9b700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcc940fa390 0x7fcc94109cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:35756/0 (socket says 172.21.15.2:35756) 2022-05-13T23:13:53.742 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:53.729+0000 7fcc99b9b700 1 -- 172.21.15.2:0/1982676578 learned_addr learned my addr 172.21.15.2:0/1982676578 (peer_addr_for_me v2:172.21.15.2:0/0) 2022-05-13T23:13:53.742 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:53.729+0000 7fcc9939a700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fcc940fb630 0x7fcc9410a230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.2:55646/0 (socket says 172.21.15.2:55646) 2022-05-13T23:13:53.742 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:53.729+0000 7fcc9a39c700 1 --2- 172.21.15.2:0/1982676578 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fcc940fbfc0 0x7fcc94106830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:13:53.743 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:53.729+0000 7fcc99b9b700 1 -- 172.21.15.2:0/1982676578 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fcc940fbfc0 msgr2=0x7fcc94106830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:13:53.743 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:53.729+0000 7fcc99b9b700 1 --2- 172.21.15.2:0/1982676578 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fcc940fbfc0 0x7fcc94106830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:13:53.743 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:53.729+0000 7fcc99b9b700 1 -- 172.21.15.2:0/1982676578 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fcc940fb630 msgr2=0x7fcc9410a230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:13:53.743 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:53.729+0000 7fcc99b9b700 1 --2- 172.21.15.2:0/1982676578 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fcc940fb630 0x7fcc9410a230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:13:53.744 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:53.729+0000 7fcc99b9b700 1 -- 172.21.15.2:0/1982676578 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc90005040 con 0x7fcc940fa390 2022-05-13T23:13:53.744 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:53.730+0000 7fcc99b9b700 1 --2- 172.21.15.2:0/1982676578 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcc940fa390 0x7fcc94109cf0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fcc90004710 tx=0x7fcc90009cb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-05-13T23:13:53.744 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:53.730+0000 7fcc8a7fc700 1 -- 172.21.15.2:0/1982676578 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc9000e040 con 0x7fcc940fa390 2022-05-13T23:13:53.744 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:53.730+0000 7fcc8a7fc700 1 -- 172.21.15.2:0/1982676578 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1087+0+0 (secure 0 0 0) 0x7fcc90014e40 con 0x7fcc940fa390 2022-05-13T23:13:53.744 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:53.730+0000 7fcc9ab9d700 1 -- 172.21.15.2:0/1982676578 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcc9411a990 con 0x7fcc940fa390 2022-05-13T23:13:53.745 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:53.730+0000 7fcc8a7fc700 1 -- 172.21.15.2:0/1982676578 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc9001cb20 con 0x7fcc940fa390 2022-05-13T23:13:53.745 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:53.730+0000 7fcc9ab9d700 1 -- 172.21.15.2:0/1982676578 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcc9411ae10 con 0x7fcc940fa390 2022-05-13T23:13:53.745 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:53.731+0000 7fcc9ab9d700 1 -- 172.21.15.2:0/1982676578 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcc94105ba0 con 0x7fcc940fa390 2022-05-13T23:13:53.745 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:53.733+0000 7fcc8a7fc700 1 -- 172.21.15.2:0/1982676578 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 16) v1 ==== 125136+0+0 (secure 0 0 0) 0x7fcc9004bbc0 con 0x7fcc940fa390 2022-05-13T23:13:53.746 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:53.735+0000 7fcc8a7fc700 1 --2- 172.21.15.2:0/1982676578 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7fcc80096ce0 0x7fcc80099190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:13:53.747 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:53.735+0000 7fcc8a7fc700 1 -- 172.21.15.2:0/1982676578 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 7123+0+0 (secure 0 0 0) 0x7fcc900c02b0 con 0x7fcc940fa390 2022-05-13T23:13:53.747 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:53.735+0000 7fcc9939a700 1 --2- 172.21.15.2:0/1982676578 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7fcc80096ce0 0x7fcc80099190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:13:53.747 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:53.736+0000 7fcc9939a700 1 --2- 172.21.15.2:0/1982676578 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7fcc80096ce0 0x7fcc80099190 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fcc840048b0 tx=0x7fcc8400b040 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-05-13T23:13:53.747 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:53.738+0000 7fcc8a7fc700 1 -- 172.21.15.2:0/1982676578 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+165215 (secure 0 0 0) 0x7fcc9008d9f0 con 0x7fcc940fa390 2022-05-13T23:13:53.974 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:53.972+0000 7fcc9ab9d700 1 -- 172.21.15.2:0/1982676578 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcc94105d30 con 0x7fcc940fa390 2022-05-13T23:13:53.975 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:53.972+0000 7fcc8a7fc700 1 -- 172.21.15.2:0/1982676578 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 5 v5) v1 ==== 93+0+3513 (secure 0 0 0) 0x7fcc94105d30 con 0x7fcc940fa390 2022-05-13T23:13:53.975 INFO:teuthology.orchestra.run.smithi002.stdout: 2022-05-13T23:13:53.975 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":5,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24460,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.2:6835/1341618941","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6834","nonce":1341618941},{"type":"v1","addr":"172.21.15.2:6835","nonce":1341618941}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24490,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.2:6839/1347050951","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6838","nonce":1347050951},{"type":"v1","addr":"172.21.15.2:6839","nonce":1347050951}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":24494,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.2:6837/3754333215","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6836","nonce":3754333215},{"type":"v1","addr":"172.21.15.2:6837","nonce":3754333215}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24530,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.2:6841/522559555","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6840","nonce":522559555},{"type":"v1","addr":"172.21.15.2:6841","nonce":522559555}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5}],"filesystems":[]} 2022-05-13T23:13:53.977 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:53.974+0000 7fcc9ab9d700 1 -- 172.21.15.2:0/1982676578 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7fcc80096ce0 msgr2=0x7fcc80099190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:13:53.977 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:53.974+0000 7fcc9ab9d700 1 --2- 172.21.15.2:0/1982676578 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7fcc80096ce0 0x7fcc80099190 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fcc840048b0 tx=0x7fcc8400b040 comp rx=0 tx=0).stop 2022-05-13T23:13:53.978 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:53.974+0000 7fcc9ab9d700 1 -- 172.21.15.2:0/1982676578 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcc940fa390 msgr2=0x7fcc94109cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:13:53.978 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:53.974+0000 7fcc9ab9d700 1 --2- 172.21.15.2:0/1982676578 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcc940fa390 0x7fcc94109cf0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fcc90004710 tx=0x7fcc90009cb0 comp rx=0 tx=0).stop 2022-05-13T23:13:53.978 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:53.974+0000 7fcc9ab9d700 1 -- 172.21.15.2:0/1982676578 shutdown_connections 2022-05-13T23:13:53.978 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:53.974+0000 7fcc9ab9d700 1 --2- 172.21.15.2:0/1982676578 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fcc940fbfc0 0x7fcc94106830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:13:53.978 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:53.975+0000 7fcc9ab9d700 1 --2- 172.21.15.2:0/1982676578 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7fcc80096ce0 0x7fcc80099190 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:13:53.979 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:53.975+0000 7fcc9ab9d700 1 --2- 172.21.15.2:0/1982676578 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcc940fa390 0x7fcc94109cf0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:13:53.979 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:53.975+0000 7fcc9ab9d700 1 --2- 172.21.15.2:0/1982676578 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fcc940fb630 0x7fcc9410a230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:13:53.979 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:53.975+0000 7fcc9ab9d700 1 -- 172.21.15.2:0/1982676578 >> 172.21.15.2:0/1982676578 conn(0x7fcc940f5b80 msgr2=0x7fcc940649c0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-05-13T23:13:53.979 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:53.975+0000 7fcc9ab9d700 1 -- 172.21.15.2:0/1982676578 shutdown_connections 2022-05-13T23:13:53.980 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:53.975+0000 7fcc9ab9d700 1 -- 172.21.15.2:0/1982676578 wait complete. 2022-05-13T23:13:53.980 INFO:teuthology.orchestra.run.smithi002.stderr:dumped fsmap epoch 5 2022-05-13T23:13:54.042 INFO:journalctl@ceph.mds.b.smithi047.stdout:May 13 23:13:53 smithi047 ceph-mds[103232]: -- 172.21.15.47:0/885021401 <== mon.1 v2:172.21.15.47:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x558f09ea8340 con 0x558f09e80c00 2022-05-13T23:13:54.042 INFO:journalctl@ceph.mds.b.smithi047.stdout:May 13 23:13:53 smithi047 conmon[103208]: starting mds.b at 2022-05-13T23:13:54.042 INFO:journalctl@ceph.mds.b.smithi047.stdout:May 13 23:13:53 smithi047 bash[103022]: e4a43bf12d38f638ab7734f8e9b81a58156f096e4e14cd4bd12202f593bf5a81 2022-05-13T23:13:54.042 INFO:journalctl@ceph.mds.b.smithi047.stdout:May 13 23:13:53 smithi047 systemd[1]: Started Ceph mds.b for 16cf5f88-d311-11ec-be40-001a4aab830c. 2022-05-13T23:13:54.559 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:13:54 smithi047 ceph-mon[77147]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 74 MiB used, 1.0 TiB / 1.0 TiB avail 2022-05-13T23:13:54.559 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:13:54 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:13:54.560 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:13:54 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2022-05-13T23:13:54.560 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:13:54 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2022-05-13T23:13:54.560 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:13:54 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-13T23:13:54.561 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:13:54 smithi047 ceph-mon[77147]: from='client.? 172.21.15.2:0/1982676578' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2022-05-13T23:13:54.572 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:13:54 smithi002 ceph-mon[71816]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 74 MiB used, 1.0 TiB / 1.0 TiB avail 2022-05-13T23:13:54.572 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:13:54 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:13:54.572 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:13:54 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2022-05-13T23:13:54.573 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:13:54 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2022-05-13T23:13:54.573 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:13:54 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-13T23:13:54.573 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:13:54 smithi002 ceph-mon[71816]: from='client.? 172.21.15.2:0/1982676578' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2022-05-13T23:13:54.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:13:54 smithi115 ceph-mon[78593]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 74 MiB used, 1.0 TiB / 1.0 TiB avail 2022-05-13T23:13:54.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:13:54 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:13:54.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:13:54 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2022-05-13T23:13:54.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:13:54 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2022-05-13T23:13:54.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:13:54 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-13T23:13:54.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:13:54 smithi115 ceph-mon[78593]: from='client.? 172.21.15.2:0/1982676578' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2022-05-13T23:13:55.490 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:13:55 smithi047 ceph-mon[77147]: Deploying daemon mds.e on smithi047 2022-05-13T23:13:55.490 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:13:55 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.47:6832/2783249369,v1:172.21.15.47:6833/2783249369] up:boot 2022-05-13T23:13:55.491 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:13:55 smithi047 ceph-mon[77147]: fsmap 5 up:standby 2022-05-13T23:13:55.491 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:13:55 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2022-05-13T23:13:55.574 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:13:55 smithi002 ceph-mon[71816]: Deploying daemon mds.e on smithi047 2022-05-13T23:13:55.574 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:13:55 smithi002 ceph-mon[71816]: mds.? [v2:172.21.15.47:6832/2783249369,v1:172.21.15.47:6833/2783249369] up:boot 2022-05-13T23:13:55.574 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:13:55 smithi002 ceph-mon[71816]: fsmap 5 up:standby 2022-05-13T23:13:55.574 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:13:55 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2022-05-13T23:13:55.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:13:55 smithi115 ceph-mon[78593]: Deploying daemon mds.e on smithi047 2022-05-13T23:13:55.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:13:55 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.47:6832/2783249369,v1:172.21.15.47:6833/2783249369] up:boot 2022-05-13T23:13:55.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:13:55 smithi115 ceph-mon[78593]: fsmap 5 up:standby 2022-05-13T23:13:55.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:13:55 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2022-05-13T23:13:55.791 INFO:journalctl@ceph.mds.e.smithi047.stdout:May 13 23:13:55 smithi047 systemd[1]: Starting Ceph mds.e for 16cf5f88-d311-11ec-be40-001a4aab830c... 2022-05-13T23:13:56.004 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-05-13T23:13:56.233 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:56.231+0000 7fdaad19d700 1 -- 172.21.15.2:0/2815053299 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fdaa8070370 msgr2=0x7fdaa806e870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:13:56.234 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:56.231+0000 7fdaad19d700 1 --2- 172.21.15.2:0/2815053299 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fdaa8070370 0x7fdaa806e870 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fda98004870 tx=0x7fda98009cb0 comp rx=0 tx=0).stop 2022-05-13T23:13:56.234 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:56.231+0000 7fdaad19d700 1 -- 172.21.15.2:0/2815053299 shutdown_connections 2022-05-13T23:13:56.234 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:56.231+0000 7fdaad19d700 1 --2- 172.21.15.2:0/2815053299 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fdaa8103400 0x7fdaa81057e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:13:56.234 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:56.231+0000 7fdaad19d700 1 --2- 172.21.15.2:0/2815053299 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fdaa8070370 0x7fdaa806e870 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:13:56.235 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:56.231+0000 7fdaad19d700 1 --2- 172.21.15.2:0/2815053299 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fdaa806edb0 0x7fdaa806f220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:13:56.235 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:56.231+0000 7fdaad19d700 1 -- 172.21.15.2:0/2815053299 >> 172.21.15.2:0/2815053299 conn(0x7fdaa80f6c60 msgr2=0x7fdaa80f9090 unknown :-1 s=STATE_NONE l=0).mark_down 2022-05-13T23:13:56.235 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:56.231+0000 7fdaad19d700 1 -- 172.21.15.2:0/2815053299 shutdown_connections 2022-05-13T23:13:56.235 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:56.231+0000 7fdaad19d700 1 -- 172.21.15.2:0/2815053299 wait complete. 2022-05-13T23:13:56.235 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:56.231+0000 7fdaad19d700 1 Processor -- start 2022-05-13T23:13:56.236 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:56.231+0000 7fdaad19d700 1 -- start start 2022-05-13T23:13:56.238 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:56.232+0000 7fdaad19d700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fdaa806edb0 0x7fdaa810e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:13:56.238 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:56.232+0000 7fdaad19d700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fdaa8070370 0x7fdaa810eaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:13:56.239 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:56.232+0000 7fdaad19d700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fdaa8103400 0x7fdaa8115b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:13:56.239 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:56.232+0000 7fdaad19d700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdaa81160c0 con 0x7fdaa8070370 2022-05-13T23:13:56.239 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:56.232+0000 7fdaad19d700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdaa8116230 con 0x7fdaa806edb0 2022-05-13T23:13:56.239 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:56.232+0000 7fdaad19d700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdaa81163f0 con 0x7fdaa8103400 2022-05-13T23:13:56.240 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:56.232+0000 7fdaa77fe700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fdaa8070370 0x7fdaa810eaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:13:56.240 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:56.232+0000 7fdaa7fff700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fdaa806edb0 0x7fdaa810e5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:13:56.240 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:56.232+0000 7fdaa7fff700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fdaa806edb0 0x7fdaa810e5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.2:55666/0 (socket says 172.21.15.2:55666) 2022-05-13T23:13:56.240 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:56.232+0000 7fdaa7fff700 1 -- 172.21.15.2:0/1498619223 learned_addr learned my addr 172.21.15.2:0/1498619223 (peer_addr_for_me v2:172.21.15.2:0/0) 2022-05-13T23:13:56.240 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:56.232+0000 7fdaac99c700 1 --2- 172.21.15.2:0/1498619223 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fdaa8103400 0x7fdaa8115b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:13:56.241 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:56.233+0000 7fdaa7fff700 1 -- 172.21.15.2:0/1498619223 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fdaa8103400 msgr2=0x7fdaa8115b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:13:56.241 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:56.233+0000 7fdaa7fff700 1 --2- 172.21.15.2:0/1498619223 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fdaa8103400 0x7fdaa8115b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:13:56.241 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:56.233+0000 7fdaa7fff700 1 -- 172.21.15.2:0/1498619223 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fdaa8070370 msgr2=0x7fdaa810eaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:13:56.241 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:56.233+0000 7fdaa7fff700 1 --2- 172.21.15.2:0/1498619223 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fdaa8070370 0x7fdaa810eaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:13:56.242 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:56.233+0000 7fdaa7fff700 1 -- 172.21.15.2:0/1498619223 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda98005040 con 0x7fdaa806edb0 2022-05-13T23:13:56.242 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:56.233+0000 7fdaa77fe700 1 --2- 172.21.15.2:0/1498619223 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fdaa8070370 0x7fdaa810eaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-05-13T23:13:56.242 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:56.233+0000 7fdaa7fff700 1 --2- 172.21.15.2:0/1498619223 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fdaa806edb0 0x7fdaa810e5b0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fda98004ee0 tx=0x7fda98009cb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-05-13T23:13:56.242 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:56.233+0000 7fdaa4ff9700 1 -- 172.21.15.2:0/1498619223 <== mon.1 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda9800e040 con 0x7fdaa806edb0 2022-05-13T23:13:56.243 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:56.233+0000 7fdaad19d700 1 -- 172.21.15.2:0/1498619223 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdaa8116670 con 0x7fdaa806edb0 2022-05-13T23:13:56.243 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:56.233+0000 7fdaa4ff9700 1 -- 172.21.15.2:0/1498619223 <== mon.1 v2:172.21.15.47:3300/0 2 ==== config(26 keys) v1 ==== 1087+0+0 (secure 0 0 0) 0x7fda98004200 con 0x7fdaa806edb0 2022-05-13T23:13:56.243 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:56.233+0000 7fdaa4ff9700 1 -- 172.21.15.2:0/1498619223 <== mon.1 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda9801cab0 con 0x7fdaa806edb0 2022-05-13T23:13:56.243 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:56.233+0000 7fdaad19d700 1 -- 172.21.15.2:0/1498619223 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdaa8116ad0 con 0x7fdaa806edb0 2022-05-13T23:13:56.244 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:56.234+0000 7fdaad19d700 1 -- 172.21.15.2:0/1498619223 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdaa8108e90 con 0x7fdaa806edb0 2022-05-13T23:13:56.244 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:56.236+0000 7fdaa4ff9700 1 -- 172.21.15.2:0/1498619223 <== mon.1 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 16) v1 ==== 125136+0+0 (secure 0 0 0) 0x7fda9804bbc0 con 0x7fdaa806edb0 2022-05-13T23:13:56.245 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:56.238+0000 7fdaa4ff9700 1 --2- 172.21.15.2:0/1498619223 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7fda90096ce0 0x7fda90099190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:13:56.245 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:56.238+0000 7fdaa4ff9700 1 -- 172.21.15.2:0/1498619223 <== mon.1 v2:172.21.15.47:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 7123+0+0 (secure 0 0 0) 0x7fda980c03a0 con 0x7fdaa806edb0 2022-05-13T23:13:56.246 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:56.238+0000 7fdaa77fe700 1 --2- 172.21.15.2:0/1498619223 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7fda90096ce0 0x7fda90099190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:13:56.246 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:56.238+0000 7fdaa77fe700 1 --2- 172.21.15.2:0/1498619223 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7fda90096ce0 0x7fda90099190 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fda9c007bf0 tx=0x7fda9c00b040 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-05-13T23:13:56.246 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:56.240+0000 7fdaa4ff9700 1 -- 172.21.15.2:0/1498619223 <== mon.1 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+165215 (secure 0 0 0) 0x7fda9808da30 con 0x7fdaa806edb0 2022-05-13T23:13:56.478 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:56.477+0000 7fdaad19d700 1 -- 172.21.15.2:0/1498619223 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdaa8116db0 con 0x7fdaa806edb0 2022-05-13T23:13:56.479 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:56.477+0000 7fdaa4ff9700 1 -- 172.21.15.2:0/1498619223 <== mon.1 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) v1 ==== 93+0+5027 (secure 0 0 0) 0x7fda98091050 con 0x7fdaa806edb0 2022-05-13T23:13:56.479 INFO:teuthology.orchestra.run.smithi002.stdout: 2022-05-13T23:13:56.479 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":7,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24460,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.2:6835/1341618941","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6834","nonce":1341618941},{"type":"v1","addr":"172.21.15.2:6835","nonce":1341618941}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24490,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.2:6839/1347050951","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6838","nonce":1347050951},{"type":"v1","addr":"172.21.15.2:6839","nonce":1347050951}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":24494,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.2:6837/3754333215","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6836","nonce":3754333215},{"type":"v1","addr":"172.21.15.2:6837","nonce":3754333215}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24511,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.47:6833/2783249369","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6832","nonce":2783249369},{"type":"v1","addr":"172.21.15.47:6833","nonce":2783249369}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":6},{"gid":24523,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.47:6835/4075838054","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6834","nonce":4075838054},{"type":"v1","addr":"172.21.15.47:6835","nonce":4075838054}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":7},{"gid":24530,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.2:6841/522559555","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6840","nonce":522559555},{"type":"v1","addr":"172.21.15.2:6841","nonce":522559555}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5}],"filesystems":[]} 2022-05-13T23:13:56.482 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:56.480+0000 7fdaad19d700 1 -- 172.21.15.2:0/1498619223 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7fda90096ce0 msgr2=0x7fda90099190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:13:56.482 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:56.480+0000 7fdaad19d700 1 --2- 172.21.15.2:0/1498619223 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7fda90096ce0 0x7fda90099190 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fda9c007bf0 tx=0x7fda9c00b040 comp rx=0 tx=0).stop 2022-05-13T23:13:56.482 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:56.480+0000 7fdaad19d700 1 -- 172.21.15.2:0/1498619223 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fdaa806edb0 msgr2=0x7fdaa810e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:13:56.482 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:56.480+0000 7fdaad19d700 1 --2- 172.21.15.2:0/1498619223 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fdaa806edb0 0x7fdaa810e5b0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fda98004ee0 tx=0x7fda98009cb0 comp rx=0 tx=0).stop 2022-05-13T23:13:56.483 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:56.480+0000 7fdaad19d700 1 -- 172.21.15.2:0/1498619223 shutdown_connections 2022-05-13T23:13:56.483 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:56.480+0000 7fdaad19d700 1 --2- 172.21.15.2:0/1498619223 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fdaa8103400 0x7fdaa8115b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:13:56.483 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:56.480+0000 7fdaad19d700 1 --2- 172.21.15.2:0/1498619223 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fdaa806edb0 0x7fdaa810e5b0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:13:56.483 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:56.480+0000 7fdaad19d700 1 --2- 172.21.15.2:0/1498619223 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7fda90096ce0 0x7fda90099190 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:13:56.484 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:56.480+0000 7fdaad19d700 1 --2- 172.21.15.2:0/1498619223 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fdaa8070370 0x7fdaa810eaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:13:56.484 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:56.480+0000 7fdaad19d700 1 -- 172.21.15.2:0/1498619223 >> 172.21.15.2:0/1498619223 conn(0x7fdaa80f6c60 msgr2=0x7fdaa80f9040 unknown :-1 s=STATE_NONE l=0).mark_down 2022-05-13T23:13:56.484 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:56.480+0000 7fdaad19d700 1 -- 172.21.15.2:0/1498619223 shutdown_connections 2022-05-13T23:13:56.484 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:56.480+0000 7fdaad19d700 1 -- 172.21.15.2:0/1498619223 wait complete. 2022-05-13T23:13:56.485 INFO:teuthology.orchestra.run.smithi002.stderr:dumped fsmap epoch 7 2022-05-13T23:13:56.504 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:13:56 smithi002 ceph-mon[71816]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 74 MiB used, 1.0 TiB / 1.0 TiB avail 2022-05-13T23:13:56.504 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:13:56 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:13:56.505 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:13:56 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2022-05-13T23:13:56.505 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:13:56 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2022-05-13T23:13:56.505 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:13:56 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-13T23:13:56.541 INFO:journalctl@ceph.mds.e.smithi047.stdout:May 13 23:13:56 smithi047 ceph-mds[103800]: -- 172.21.15.47:0/3828274667 <== mon.1 v2:172.21.15.47:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55668eac8340 con 0x55668eaa0c00 2022-05-13T23:13:56.542 INFO:journalctl@ceph.mds.e.smithi047.stdout:May 13 23:13:56 smithi047 conmon[103776]: starting mds.e at 2022-05-13T23:13:56.542 INFO:journalctl@ceph.mds.e.smithi047.stdout:May 13 23:13:56 smithi047 bash[103589]: 5150a7b9c7551c2e2c9c8866c4ce2b7c0a6126842b1e04fc736d16780decb358 2022-05-13T23:13:56.542 INFO:journalctl@ceph.mds.e.smithi047.stdout:May 13 23:13:56 smithi047 systemd[1]: Started Ceph mds.e for 16cf5f88-d311-11ec-be40-001a4aab830c. 2022-05-13T23:13:56.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:13:56 smithi047 ceph-mon[77147]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 74 MiB used, 1.0 TiB / 1.0 TiB avail 2022-05-13T23:13:56.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:13:56 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:13:56.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:13:56 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2022-05-13T23:13:56.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:13:56 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2022-05-13T23:13:56.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:13:56 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-13T23:13:56.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:13:56 smithi115 ceph-mon[78593]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 74 MiB used, 1.0 TiB / 1.0 TiB avail 2022-05-13T23:13:56.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:13:56 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:13:56.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:13:56 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2022-05-13T23:13:56.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:13:56 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2022-05-13T23:13:56.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:13:56 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-13T23:13:57.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:13:57 smithi047 ceph-mon[77147]: Deploying daemon mds.h on smithi047 2022-05-13T23:13:57.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:13:57 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.47:6834/4075838054,v1:172.21.15.47:6835/4075838054] up:boot 2022-05-13T23:13:57.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:13:57 smithi047 ceph-mon[77147]: fsmap 6 up:standby 2022-05-13T23:13:57.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:13:57 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2022-05-13T23:13:57.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:13:57 smithi047 ceph-mon[77147]: from='client.? 172.21.15.2:0/1498619223' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2022-05-13T23:13:57.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:13:57 smithi115 ceph-mon[78593]: Deploying daemon mds.h on smithi047 2022-05-13T23:13:57.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:13:57 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.47:6834/4075838054,v1:172.21.15.47:6835/4075838054] up:boot 2022-05-13T23:13:57.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:13:57 smithi115 ceph-mon[78593]: fsmap 6 up:standby 2022-05-13T23:13:57.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:13:57 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2022-05-13T23:13:57.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:13:57 smithi115 ceph-mon[78593]: from='client.? 172.21.15.2:0/1498619223' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2022-05-13T23:13:57.822 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:13:57 smithi002 ceph-mon[71816]: Deploying daemon mds.h on smithi047 2022-05-13T23:13:57.823 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:13:57 smithi002 ceph-mon[71816]: mds.? [v2:172.21.15.47:6834/4075838054,v1:172.21.15.47:6835/4075838054] up:boot 2022-05-13T23:13:57.823 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:13:57 smithi002 ceph-mon[71816]: fsmap 6 up:standby 2022-05-13T23:13:57.823 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:13:57 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2022-05-13T23:13:57.823 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:13:57 smithi002 ceph-mon[71816]: from='client.? 172.21.15.2:0/1498619223' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2022-05-13T23:13:58.042 INFO:journalctl@ceph.mds.h.smithi047.stdout:May 13 23:13:57 smithi047 systemd[1]: Starting Ceph mds.h for 16cf5f88-d311-11ec-be40-001a4aab830c... 2022-05-13T23:13:58.509 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-05-13T23:13:58.585 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:13:58 smithi002 ceph-mon[71816]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 74 MiB used, 1.0 TiB / 1.0 TiB avail 2022-05-13T23:13:58.586 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:13:58 smithi047 ceph-mon[77147]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 74 MiB used, 1.0 TiB / 1.0 TiB avail 2022-05-13T23:13:58.587 INFO:journalctl@ceph.mds.h.smithi047.stdout:May 13 23:13:58 smithi047 ceph-mds[104366]: -- 172.21.15.47:0/3222884266 <== mon.1 v2:172.21.15.47:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5606f761a340 con 0x5606f75f2c00 2022-05-13T23:13:58.587 INFO:journalctl@ceph.mds.h.smithi047.stdout:May 13 23:13:58 smithi047 conmon[104342]: starting mds.h at 2022-05-13T23:13:58.587 INFO:journalctl@ceph.mds.h.smithi047.stdout:May 13 23:13:58 smithi047 bash[104158]: 0223223a404666b0c69144bc6a02bda82d4756f648a4b099db40b59d1eface17 2022-05-13T23:13:58.587 INFO:journalctl@ceph.mds.h.smithi047.stdout:May 13 23:13:58 smithi047 systemd[1]: Started Ceph mds.h for 16cf5f88-d311-11ec-be40-001a4aab830c. 2022-05-13T23:13:58.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:13:58 smithi115 ceph-mon[78593]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 74 MiB used, 1.0 TiB / 1.0 TiB avail 2022-05-13T23:13:58.738 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:58.736+0000 7efcb9d9d700 1 -- 172.21.15.2:0/2886082653 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efcb40fa380 msgr2=0x7efcb40fa790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:13:58.738 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:58.736+0000 7efcb9d9d700 1 --2- 172.21.15.2:0/2886082653 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efcb40fa380 0x7efcb40fa790 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7efca8004870 tx=0x7efca8009cb0 comp rx=0 tx=0).stop 2022-05-13T23:13:58.738 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:58.736+0000 7efcb9d9d700 1 -- 172.21.15.2:0/2886082653 shutdown_connections 2022-05-13T23:13:58.739 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:58.736+0000 7efcb9d9d700 1 --2- 172.21.15.2:0/2886082653 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7efcb40fbfb0 0x7efcb4102870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:13:58.739 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:58.736+0000 7efcb9d9d700 1 --2- 172.21.15.2:0/2886082653 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efcb40fa380 0x7efcb40fa790 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:13:58.739 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:58.736+0000 7efcb9d9d700 1 --2- 172.21.15.2:0/2886082653 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7efcb40fb620 0x7efcb40fba70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:13:58.739 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:58.736+0000 7efcb9d9d700 1 -- 172.21.15.2:0/2886082653 >> 172.21.15.2:0/2886082653 conn(0x7efcb40f5b30 msgr2=0x7efcb40f7fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-05-13T23:13:58.739 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:58.736+0000 7efcb9d9d700 1 -- 172.21.15.2:0/2886082653 shutdown_connections 2022-05-13T23:13:58.740 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:58.736+0000 7efcb9d9d700 1 -- 172.21.15.2:0/2886082653 wait complete. 2022-05-13T23:13:58.740 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:58.737+0000 7efcb9d9d700 1 Processor -- start 2022-05-13T23:13:58.742 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:58.737+0000 7efcb9d9d700 1 -- start start 2022-05-13T23:13:58.742 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:58.737+0000 7efcb9d9d700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efcb40fa380 0x7efcb410c2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:13:58.743 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:58.737+0000 7efcb9d9d700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7efcb40fb620 0x7efcb410c7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:13:58.743 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:58.737+0000 7efcb9d9d700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7efcb40fbfb0 0x7efcb4113870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:13:58.743 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:58.737+0000 7efcb9d9d700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efcb4113db0 con 0x7efcb40fbfb0 2022-05-13T23:13:58.743 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:58.737+0000 7efcb9d9d700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efcb4113f20 con 0x7efcb40fa380 2022-05-13T23:13:58.744 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:58.737+0000 7efcb9d9d700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efcb41140e0 con 0x7efcb40fb620 2022-05-13T23:13:58.744 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:58.738+0000 7efcb8d9b700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efcb40fa380 0x7efcb410c2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:13:58.744 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:58.738+0000 7efcb8d9b700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efcb40fa380 0x7efcb410c2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.2:55672/0 (socket says 172.21.15.2:55672) 2022-05-13T23:13:58.744 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:58.738+0000 7efcb8d9b700 1 -- 172.21.15.2:0/741972913 learned_addr learned my addr 172.21.15.2:0/741972913 (peer_addr_for_me v2:172.21.15.2:0/0) 2022-05-13T23:13:58.744 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:58.738+0000 7efcb959c700 1 --2- 172.21.15.2:0/741972913 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7efcb40fbfb0 0x7efcb4113870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:13:58.745 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:58.738+0000 7efcb3fff700 1 --2- 172.21.15.2:0/741972913 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7efcb40fb620 0x7efcb410c7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:13:58.745 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:58.738+0000 7efcb8d9b700 1 -- 172.21.15.2:0/741972913 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7efcb40fb620 msgr2=0x7efcb410c7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:13:58.749 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:58.738+0000 7efcb8d9b700 1 --2- 172.21.15.2:0/741972913 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7efcb40fb620 0x7efcb410c7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:13:58.749 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:58.738+0000 7efcb8d9b700 1 -- 172.21.15.2:0/741972913 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7efcb40fbfb0 msgr2=0x7efcb4113870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:13:58.750 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:58.738+0000 7efcb8d9b700 1 --2- 172.21.15.2:0/741972913 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7efcb40fbfb0 0x7efcb4113870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:13:58.750 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:58.738+0000 7efcb8d9b700 1 -- 172.21.15.2:0/741972913 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efca8005040 con 0x7efcb40fa380 2022-05-13T23:13:58.750 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:58.738+0000 7efcb8d9b700 1 --2- 172.21.15.2:0/741972913 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efcb40fa380 0x7efcb410c2a0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7efca8004710 tx=0x7efca8009cb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-05-13T23:13:58.750 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:58.739+0000 7efcb17fa700 1 -- 172.21.15.2:0/741972913 <== mon.1 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efca800e040 con 0x7efcb40fa380 2022-05-13T23:13:58.750 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:58.739+0000 7efcb17fa700 1 -- 172.21.15.2:0/741972913 <== mon.1 v2:172.21.15.47:3300/0 2 ==== config(26 keys) v1 ==== 1087+0+0 (secure 0 0 0) 0x7efca8004200 con 0x7efcb40fa380 2022-05-13T23:13:58.751 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:58.739+0000 7efcb9d9d700 1 -- 172.21.15.2:0/741972913 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efcb4114360 con 0x7efcb40fa380 2022-05-13T23:13:58.751 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:58.739+0000 7efcb17fa700 1 -- 172.21.15.2:0/741972913 <== mon.1 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efca801caf0 con 0x7efcb40fa380 2022-05-13T23:13:58.751 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:58.739+0000 7efcb9d9d700 1 -- 172.21.15.2:0/741972913 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efcb41147e0 con 0x7efcb40fa380 2022-05-13T23:13:58.751 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:58.740+0000 7efcb9d9d700 1 -- 172.21.15.2:0/741972913 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efcb4105be0 con 0x7efcb40fa380 2022-05-13T23:13:58.753 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:58.742+0000 7efcb17fa700 1 -- 172.21.15.2:0/741972913 <== mon.1 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 16) v1 ==== 125136+0+0 (secure 0 0 0) 0x7efca8009240 con 0x7efcb40fa380 2022-05-13T23:13:58.753 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:58.743+0000 7efcb17fa700 1 --2- 172.21.15.2:0/741972913 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7efca4096ce0 0x7efca4099190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:13:58.753 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:58.743+0000 7efcb17fa700 1 -- 172.21.15.2:0/741972913 <== mon.1 v2:172.21.15.47:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 7123+0+0 (secure 0 0 0) 0x7efca80c0b30 con 0x7efcb40fa380 2022-05-13T23:13:58.754 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:58.743+0000 7efcb3fff700 1 --2- 172.21.15.2:0/741972913 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7efca4096ce0 0x7efca4099190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:13:58.754 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:58.744+0000 7efcb3fff700 1 --2- 172.21.15.2:0/741972913 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7efca4096ce0 0x7efca4099190 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7efca00048b0 tx=0x7efca000b040 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-05-13T23:13:58.754 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:58.749+0000 7efcb17fa700 1 -- 172.21.15.2:0/741972913 <== mon.1 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+165215 (secure 0 0 0) 0x7efca808e250 con 0x7efcb40fa380 2022-05-13T23:13:58.982 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:58.980+0000 7efcb9d9d700 1 -- 172.21.15.2:0/741972913 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7efcb4114a00 con 0x7efcb40fa380 2022-05-13T23:13:58.983 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:58.981+0000 7efcb17fa700 1 -- 172.21.15.2:0/741972913 <== mon.1 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) v1 ==== 93+0+5027 (secure 0 0 0) 0x7efca8091870 con 0x7efcb40fa380 2022-05-13T23:13:58.983 INFO:teuthology.orchestra.run.smithi002.stdout: 2022-05-13T23:13:58.984 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":7,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24460,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.2:6835/1341618941","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6834","nonce":1341618941},{"type":"v1","addr":"172.21.15.2:6835","nonce":1341618941}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24490,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.2:6839/1347050951","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6838","nonce":1347050951},{"type":"v1","addr":"172.21.15.2:6839","nonce":1347050951}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":24494,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.2:6837/3754333215","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6836","nonce":3754333215},{"type":"v1","addr":"172.21.15.2:6837","nonce":3754333215}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24511,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.47:6833/2783249369","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6832","nonce":2783249369},{"type":"v1","addr":"172.21.15.47:6833","nonce":2783249369}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":6},{"gid":24523,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.47:6835/4075838054","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6834","nonce":4075838054},{"type":"v1","addr":"172.21.15.47:6835","nonce":4075838054}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":7},{"gid":24530,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.2:6841/522559555","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6840","nonce":522559555},{"type":"v1","addr":"172.21.15.2:6841","nonce":522559555}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5}],"filesystems":[]} 2022-05-13T23:13:58.986 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:58.983+0000 7efcb9d9d700 1 -- 172.21.15.2:0/741972913 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7efca4096ce0 msgr2=0x7efca4099190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:13:58.987 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:58.983+0000 7efcb9d9d700 1 --2- 172.21.15.2:0/741972913 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7efca4096ce0 0x7efca4099190 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7efca00048b0 tx=0x7efca000b040 comp rx=0 tx=0).stop 2022-05-13T23:13:58.987 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:58.983+0000 7efcb9d9d700 1 -- 172.21.15.2:0/741972913 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efcb40fa380 msgr2=0x7efcb410c2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:13:58.988 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:58.983+0000 7efcb9d9d700 1 --2- 172.21.15.2:0/741972913 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efcb40fa380 0x7efcb410c2a0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7efca8004710 tx=0x7efca8009cb0 comp rx=0 tx=0).stop 2022-05-13T23:13:58.988 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:58.984+0000 7efcb9d9d700 1 -- 172.21.15.2:0/741972913 shutdown_connections 2022-05-13T23:13:58.989 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:58.984+0000 7efcb9d9d700 1 --2- 172.21.15.2:0/741972913 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7efca4096ce0 0x7efca4099190 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:13:58.989 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:58.984+0000 7efcb9d9d700 1 --2- 172.21.15.2:0/741972913 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7efcb40fbfb0 0x7efcb4113870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:13:58.989 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:58.984+0000 7efcb9d9d700 1 --2- 172.21.15.2:0/741972913 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efcb40fa380 0x7efcb410c2a0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:13:58.990 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:58.984+0000 7efcb9d9d700 1 --2- 172.21.15.2:0/741972913 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7efcb40fb620 0x7efcb410c7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:13:58.990 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:58.984+0000 7efcb9d9d700 1 -- 172.21.15.2:0/741972913 >> 172.21.15.2:0/741972913 conn(0x7efcb40f5b30 msgr2=0x7efcb4063a20 unknown :-1 s=STATE_NONE l=0).mark_down 2022-05-13T23:13:58.991 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:58.984+0000 7efcb9d9d700 1 -- 172.21.15.2:0/741972913 shutdown_connections 2022-05-13T23:13:58.991 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:13:58.984+0000 7efcb9d9d700 1 -- 172.21.15.2:0/741972913 wait complete. 2022-05-13T23:13:58.992 INFO:teuthology.orchestra.run.smithi002.stderr:dumped fsmap epoch 7 2022-05-13T23:13:59.702 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:13:59 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:13:59.703 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:13:59 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2022-05-13T23:13:59.703 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:13:59 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2022-05-13T23:13:59.703 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:13:59 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-13T23:13:59.704 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:13:59 smithi047 ceph-mon[77147]: Deploying daemon mds.k on smithi047 2022-05-13T23:13:59.704 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:13:59 smithi047 ceph-mon[77147]: from='client.? 172.21.15.2:0/741972913' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2022-05-13T23:13:59.704 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:13:59 smithi047 ceph-mon[77147]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 74 MiB used, 1.0 TiB / 1.0 TiB avail 2022-05-13T23:13:59.704 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:13:59 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.47:6836/4072863000,v1:172.21.15.47:6837/4072863000] up:boot 2022-05-13T23:13:59.705 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:13:59 smithi047 ceph-mon[77147]: fsmap 7 up:standby 2022-05-13T23:13:59.705 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:13:59 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2022-05-13T23:13:59.822 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:13:59 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:13:59.823 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:13:59 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2022-05-13T23:13:59.823 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:13:59 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2022-05-13T23:13:59.823 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:13:59 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-13T23:13:59.824 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:13:59 smithi002 ceph-mon[71816]: Deploying daemon mds.k on smithi047 2022-05-13T23:13:59.824 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:13:59 smithi002 ceph-mon[71816]: from='client.? 172.21.15.2:0/741972913' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2022-05-13T23:13:59.824 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:13:59 smithi002 ceph-mon[71816]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 74 MiB used, 1.0 TiB / 1.0 TiB avail 2022-05-13T23:13:59.824 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:13:59 smithi002 ceph-mon[71816]: mds.? [v2:172.21.15.47:6836/4072863000,v1:172.21.15.47:6837/4072863000] up:boot 2022-05-13T23:13:59.825 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:13:59 smithi002 ceph-mon[71816]: fsmap 7 up:standby 2022-05-13T23:13:59.825 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:13:59 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2022-05-13T23:13:59.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:13:59 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:13:59.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:13:59 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2022-05-13T23:13:59.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:13:59 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2022-05-13T23:13:59.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:13:59 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-13T23:13:59.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:13:59 smithi115 ceph-mon[78593]: Deploying daemon mds.k on smithi047 2022-05-13T23:13:59.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:13:59 smithi115 ceph-mon[78593]: from='client.? 172.21.15.2:0/741972913' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2022-05-13T23:13:59.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:13:59 smithi115 ceph-mon[78593]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 74 MiB used, 1.0 TiB / 1.0 TiB avail 2022-05-13T23:13:59.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:13:59 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.47:6836/4072863000,v1:172.21.15.47:6837/4072863000] up:boot 2022-05-13T23:13:59.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:13:59 smithi115 ceph-mon[78593]: fsmap 7 up:standby 2022-05-13T23:13:59.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:13:59 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2022-05-13T23:14:00.291 INFO:journalctl@ceph.mds.k.smithi047.stdout:May 13 23:14:00 smithi047 systemd[1]: Starting Ceph mds.k for 16cf5f88-d311-11ec-be40-001a4aab830c... 2022-05-13T23:14:01.013 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-05-13T23:14:01.041 INFO:journalctl@ceph.mds.k.smithi047.stdout:May 13 23:14:00 smithi047 ceph-mds[104947]: -- 172.21.15.47:0/3685974757 <== mon.1 v2:172.21.15.47:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5642b21b2340 con 0x5642b218a800 2022-05-13T23:14:01.042 INFO:journalctl@ceph.mds.k.smithi047.stdout:May 13 23:14:00 smithi047 conmon[104923]: starting mds.k at 2022-05-13T23:14:01.042 INFO:journalctl@ceph.mds.k.smithi047.stdout:May 13 23:14:00 smithi047 bash[104736]: 254778589c1d7c780d4be81cb4de0fcc7f52371b66c039be5e711212b7d702fd 2022-05-13T23:14:01.042 INFO:journalctl@ceph.mds.k.smithi047.stdout:May 13 23:14:00 smithi047 systemd[1]: Started Ceph mds.k for 16cf5f88-d311-11ec-be40-001a4aab830c. 2022-05-13T23:14:01.239 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:01.236+0000 7f94ddf9d700 1 -- 172.21.15.2:0/1545473160 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f94d8070370 msgr2=0x7f94d806e870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:14:01.240 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:01.236+0000 7f94ddf9d700 1 --2- 172.21.15.2:0/1545473160 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f94d8070370 0x7f94d806e870 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f94cc004870 tx=0x7f94cc009cb0 comp rx=0 tx=0).stop 2022-05-13T23:14:01.240 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:01.236+0000 7f94ddf9d700 1 -- 172.21.15.2:0/1545473160 shutdown_connections 2022-05-13T23:14:01.240 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:01.236+0000 7f94ddf9d700 1 --2- 172.21.15.2:0/1545473160 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f94d8103420 0x7f94d8105800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:14:01.240 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:01.236+0000 7f94ddf9d700 1 --2- 172.21.15.2:0/1545473160 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f94d8070370 0x7f94d806e870 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:14:01.241 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:01.236+0000 7f94ddf9d700 1 --2- 172.21.15.2:0/1545473160 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f94d806edb0 0x7f94d806f220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:14:01.241 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:01.236+0000 7f94ddf9d700 1 -- 172.21.15.2:0/1545473160 >> 172.21.15.2:0/1545473160 conn(0x7f94d80f6c80 msgr2=0x7f94d80f90b0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-05-13T23:14:01.246 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:01.236+0000 7f94ddf9d700 1 -- 172.21.15.2:0/1545473160 shutdown_connections 2022-05-13T23:14:01.246 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:01.236+0000 7f94ddf9d700 1 -- 172.21.15.2:0/1545473160 wait complete. 2022-05-13T23:14:01.246 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:01.237+0000 7f94ddf9d700 1 Processor -- start 2022-05-13T23:14:01.247 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:01.237+0000 7f94ddf9d700 1 -- start start 2022-05-13T23:14:01.247 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:01.237+0000 7f94ddf9d700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f94d806edb0 0x7f94d810e520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:14:01.247 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:01.237+0000 7f94ddf9d700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f94d8070370 0x7f94d810ea60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:14:01.247 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:01.237+0000 7f94ddf9d700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f94d8103420 0x7f94d8115af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:14:01.247 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:01.237+0000 7f94ddf9d700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94d8116030 con 0x7f94d8070370 2022-05-13T23:14:01.248 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:01.237+0000 7f94ddf9d700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94d81161a0 con 0x7f94d8103420 2022-05-13T23:14:01.248 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:01.237+0000 7f94ddf9d700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94d8116360 con 0x7f94d806edb0 2022-05-13T23:14:01.248 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:01.238+0000 7f94dcf9b700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f94d806edb0 0x7f94d810e520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:14:01.248 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:01.238+0000 7f94d7fff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f94d8070370 0x7f94d810ea60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:14:01.249 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:01.238+0000 7f94dcf9b700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f94d806edb0 0x7f94d810e520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.2:45730/0 (socket says 172.21.15.2:45730) 2022-05-13T23:14:01.249 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:01.238+0000 7f94dcf9b700 1 -- 172.21.15.2:0/2636142542 learned_addr learned my addr 172.21.15.2:0/2636142542 (peer_addr_for_me v2:172.21.15.2:0/0) 2022-05-13T23:14:01.249 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:01.238+0000 7f94dd79c700 1 --2- 172.21.15.2:0/2636142542 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f94d8103420 0x7f94d8115af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:14:01.249 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:01.238+0000 7f94dcf9b700 1 -- 172.21.15.2:0/2636142542 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f94d8103420 msgr2=0x7f94d8115af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:14:01.249 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:01.238+0000 7f94dcf9b700 1 --2- 172.21.15.2:0/2636142542 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f94d8103420 0x7f94d8115af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:14:01.250 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:01.238+0000 7f94dcf9b700 1 -- 172.21.15.2:0/2636142542 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f94d8070370 msgr2=0x7f94d810ea60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:14:01.250 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:01.238+0000 7f94dcf9b700 1 --2- 172.21.15.2:0/2636142542 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f94d8070370 0x7f94d810ea60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:14:01.250 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:01.238+0000 7f94dcf9b700 1 -- 172.21.15.2:0/2636142542 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94d00097c0 con 0x7f94d806edb0 2022-05-13T23:14:01.252 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:01.238+0000 7f94dcf9b700 1 --2- 172.21.15.2:0/2636142542 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f94d806edb0 0x7f94d810e520 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f94cc004ee0 tx=0x7f94cc009cb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-05-13T23:14:01.252 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:01.239+0000 7f94d57fa700 1 -- 172.21.15.2:0/2636142542 <== mon.2 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94cc00e040 con 0x7f94d806edb0 2022-05-13T23:14:01.253 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:01.239+0000 7f94ddf9d700 1 -- 172.21.15.2:0/2636142542 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f94cc005040 con 0x7f94d806edb0 2022-05-13T23:14:01.253 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:01.239+0000 7f94d57fa700 1 -- 172.21.15.2:0/2636142542 <== mon.2 v2:172.21.15.115:3300/0 2 ==== config(26 keys) v1 ==== 1087+0+0 (secure 0 0 0) 0x7f94cc014e40 con 0x7f94d806edb0 2022-05-13T23:14:01.253 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:01.239+0000 7f94d57fa700 1 -- 172.21.15.2:0/2636142542 <== mon.2 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94cc01cab0 con 0x7f94d806edb0 2022-05-13T23:14:01.253 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:01.239+0000 7f94ddf9d700 1 -- 172.21.15.2:0/2636142542 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f94d8116930 con 0x7f94d806edb0 2022-05-13T23:14:01.254 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:01.240+0000 7f94ddf9d700 1 -- 172.21.15.2:0/2636142542 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f94d804ac70 con 0x7f94d806edb0 2022-05-13T23:14:01.254 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:01.242+0000 7f94d57fa700 1 -- 172.21.15.2:0/2636142542 <== mon.2 v2:172.21.15.115:3300/0 4 ==== mgrmap(e 16) v1 ==== 125136+0+0 (secure 0 0 0) 0x7f94cc04bbc0 con 0x7f94d806edb0 2022-05-13T23:14:01.254 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:01.243+0000 7f94d57fa700 1 --2- 172.21.15.2:0/2636142542 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f94c8096ce0 0x7f94c8099190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:14:01.254 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:01.243+0000 7f94d57fa700 1 -- 172.21.15.2:0/2636142542 <== mon.2 v2:172.21.15.115:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 7123+0+0 (secure 0 0 0) 0x7f94cc0c0440 con 0x7f94d806edb0 2022-05-13T23:14:01.254 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:01.243+0000 7f94d7fff700 1 --2- 172.21.15.2:0/2636142542 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f94c8096ce0 0x7f94c8099190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:14:01.255 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:01.244+0000 7f94d7fff700 1 --2- 172.21.15.2:0/2636142542 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f94c8096ce0 0x7f94c8099190 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f94c40048b0 tx=0x7f94c400b040 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-05-13T23:14:01.255 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:01.245+0000 7f94d57fa700 1 -- 172.21.15.2:0/2636142542 <== mon.2 v2:172.21.15.115:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+165215 (secure 0 0 0) 0x7f94cc08dad0 con 0x7f94d806edb0 2022-05-13T23:14:01.482 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:01.479+0000 7f94ddf9d700 1 -- 172.21.15.2:0/2636142542 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f94d8108f30 con 0x7f94d806edb0 2022-05-13T23:14:01.483 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:01.480+0000 7f94d57fa700 1 -- 172.21.15.2:0/2636142542 <== mon.2 v2:172.21.15.115:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) v1 ==== 93+0+6538 (secure 0 0 0) 0x7f94cc0910f0 con 0x7f94d806edb0 2022-05-13T23:14:01.484 INFO:teuthology.orchestra.run.smithi002.stdout: 2022-05-13T23:14:01.485 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":9,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24460,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.2:6835/1341618941","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6834","nonce":1341618941},{"type":"v1","addr":"172.21.15.2:6835","nonce":1341618941}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24490,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.2:6839/1347050951","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6838","nonce":1347050951},{"type":"v1","addr":"172.21.15.2:6839","nonce":1347050951}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":24494,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.2:6837/3754333215","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6836","nonce":3754333215},{"type":"v1","addr":"172.21.15.2:6837","nonce":3754333215}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24511,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.47:6833/2783249369","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6832","nonce":2783249369},{"type":"v1","addr":"172.21.15.47:6833","nonce":2783249369}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":6},{"gid":24523,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.47:6835/4075838054","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6834","nonce":4075838054},{"type":"v1","addr":"172.21.15.47:6835","nonce":4075838054}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":7},{"gid":24530,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.2:6841/522559555","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6840","nonce":522559555},{"type":"v1","addr":"172.21.15.2:6841","nonce":522559555}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":24535,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.47:6837/4072863000","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6836","nonce":4072863000},{"type":"v1","addr":"172.21.15.47:6837","nonce":4072863000}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":8},{"gid":24547,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.47:6839/761355251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6838","nonce":761355251},{"type":"v1","addr":"172.21.15.47:6839","nonce":761355251}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":9}],"filesystems":[]} 2022-05-13T23:14:01.486 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:01.483+0000 7f94ddf9d700 1 -- 172.21.15.2:0/2636142542 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f94c8096ce0 msgr2=0x7f94c8099190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:14:01.486 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:01.483+0000 7f94ddf9d700 1 --2- 172.21.15.2:0/2636142542 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f94c8096ce0 0x7f94c8099190 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f94c40048b0 tx=0x7f94c400b040 comp rx=0 tx=0).stop 2022-05-13T23:14:01.487 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:01.483+0000 7f94ddf9d700 1 -- 172.21.15.2:0/2636142542 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f94d806edb0 msgr2=0x7f94d810e520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:14:01.487 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:01.483+0000 7f94ddf9d700 1 --2- 172.21.15.2:0/2636142542 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f94d806edb0 0x7f94d810e520 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f94cc004ee0 tx=0x7f94cc009cb0 comp rx=0 tx=0).stop 2022-05-13T23:14:01.487 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:01.483+0000 7f94ddf9d700 1 -- 172.21.15.2:0/2636142542 shutdown_connections 2022-05-13T23:14:01.487 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:01.483+0000 7f94ddf9d700 1 --2- 172.21.15.2:0/2636142542 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f94d8103420 0x7f94d8115af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:14:01.488 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:01.483+0000 7f94ddf9d700 1 --2- 172.21.15.2:0/2636142542 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f94d806edb0 0x7f94d810e520 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:14:01.488 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:01.483+0000 7f94ddf9d700 1 --2- 172.21.15.2:0/2636142542 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f94c8096ce0 0x7f94c8099190 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:14:01.488 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:01.483+0000 7f94ddf9d700 1 --2- 172.21.15.2:0/2636142542 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f94d8070370 0x7f94d810ea60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:14:01.488 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:01.483+0000 7f94ddf9d700 1 -- 172.21.15.2:0/2636142542 >> 172.21.15.2:0/2636142542 conn(0x7f94d80f6c80 msgr2=0x7f94d80f9060 unknown :-1 s=STATE_NONE l=0).mark_down 2022-05-13T23:14:01.488 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:01.483+0000 7f94ddf9d700 1 -- 172.21.15.2:0/2636142542 shutdown_connections 2022-05-13T23:14:01.489 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:01.483+0000 7f94ddf9d700 1 -- 172.21.15.2:0/2636142542 wait complete. 2022-05-13T23:14:01.489 INFO:teuthology.orchestra.run.smithi002.stderr:dumped fsmap epoch 9 2022-05-13T23:14:02.055 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:14:01 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.47:6838/761355251,v1:172.21.15.47:6839/761355251] up:boot 2022-05-13T23:14:02.055 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:14:01 smithi115 ceph-mon[78593]: fsmap 8 up:standby 2022-05-13T23:14:02.056 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:14:01 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2022-05-13T23:14:02.056 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:14:01 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:14:02.056 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:14:01 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2022-05-13T23:14:02.056 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:14:01 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2022-05-13T23:14:02.056 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:14:01 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-13T23:14:02.058 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:14:01 smithi115 ceph-mon[78593]: Deploying daemon mds.c on smithi115 2022-05-13T23:14:02.058 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:14:01 smithi115 ceph-mon[78593]: pgmap v277: 1 pgs: 1 active+clean; 577 KiB data, 74 MiB used, 1.0 TiB / 1.0 TiB avail 2022-05-13T23:14:02.058 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:14:01 smithi115 ceph-mon[78593]: from='client.? 172.21.15.2:0/2636142542' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2022-05-13T23:14:02.073 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:14:01 smithi002 ceph-mon[71816]: mds.? [v2:172.21.15.47:6838/761355251,v1:172.21.15.47:6839/761355251] up:boot 2022-05-13T23:14:02.073 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:14:01 smithi002 ceph-mon[71816]: fsmap 8 up:standby 2022-05-13T23:14:02.074 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:14:01 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2022-05-13T23:14:02.074 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:14:01 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:14:02.074 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:14:01 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2022-05-13T23:14:02.074 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:14:01 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2022-05-13T23:14:02.075 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:14:01 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-13T23:14:02.075 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:14:01 smithi002 ceph-mon[71816]: Deploying daemon mds.c on smithi115 2022-05-13T23:14:02.075 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:14:01 smithi002 ceph-mon[71816]: pgmap v277: 1 pgs: 1 active+clean; 577 KiB data, 74 MiB used, 1.0 TiB / 1.0 TiB avail 2022-05-13T23:14:02.075 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:14:01 smithi002 ceph-mon[71816]: from='client.? 172.21.15.2:0/2636142542' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2022-05-13T23:14:02.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:14:01 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.47:6838/761355251,v1:172.21.15.47:6839/761355251] up:boot 2022-05-13T23:14:02.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:14:01 smithi047 ceph-mon[77147]: fsmap 8 up:standby 2022-05-13T23:14:02.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:14:01 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2022-05-13T23:14:02.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:14:01 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:14:02.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:14:01 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2022-05-13T23:14:02.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:14:01 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2022-05-13T23:14:02.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:14:01 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-13T23:14:02.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:14:01 smithi047 ceph-mon[77147]: Deploying daemon mds.c on smithi115 2022-05-13T23:14:02.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:14:01 smithi047 ceph-mon[77147]: pgmap v277: 1 pgs: 1 active+clean; 577 KiB data, 74 MiB used, 1.0 TiB / 1.0 TiB avail 2022-05-13T23:14:02.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:14:01 smithi047 ceph-mon[77147]: from='client.? 172.21.15.2:0/2636142542' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2022-05-13T23:14:02.858 INFO:journalctl@ceph.mds.c.smithi115.stdout:May 13 23:14:02 smithi115 systemd[1]: Starting Ceph mds.c for 16cf5f88-d311-11ec-be40-001a4aab830c... 2022-05-13T23:14:03.209 INFO:journalctl@ceph.mds.c.smithi115.stdout:May 13 23:14:03 smithi115 ceph-mds[105451]: -- 172.21.15.115:0/2930711150 <== mon.2 v2:172.21.15.115:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x559a4498c340 con 0x559a44964800 2022-05-13T23:14:03.210 INFO:journalctl@ceph.mds.c.smithi115.stdout:May 13 23:14:03 smithi115 conmon[105427]: starting mds.c at 2022-05-13T23:14:03.210 INFO:journalctl@ceph.mds.c.smithi115.stdout:May 13 23:14:03 smithi115 bash[105242]: 6a91b898e3ca839b5c4f520ffc944ecbf4085bd13df5f5f3d075914962f767d7 2022-05-13T23:14:03.511 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-05-13T23:14:03.610 INFO:journalctl@ceph.mds.c.smithi115.stdout:May 13 23:14:03 smithi115 systemd[1]: Started Ceph mds.c for 16cf5f88-d311-11ec-be40-001a4aab830c. 2022-05-13T23:14:03.746 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:03.741+0000 7efd33fff700 1 -- 172.21.15.2:0/840399611 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7efd340720f0 msgr2=0x7efd34072560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:14:03.747 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:03.741+0000 7efd33fff700 1 --2- 172.21.15.2:0/840399611 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7efd340720f0 0x7efd34072560 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7efd28004870 tx=0x7efd28009cb0 comp rx=0 tx=0).stop 2022-05-13T23:14:03.747 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:03.741+0000 7efd33fff700 1 -- 172.21.15.2:0/840399611 shutdown_connections 2022-05-13T23:14:03.747 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:03.741+0000 7efd33fff700 1 --2- 172.21.15.2:0/840399611 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7efd34072aa0 0x7efd34073330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:14:03.748 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:03.741+0000 7efd33fff700 1 --2- 172.21.15.2:0/840399611 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efd34070e00 0x7efd34071210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:14:03.748 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:03.741+0000 7efd33fff700 1 --2- 172.21.15.2:0/840399611 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7efd340720f0 0x7efd34072560 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:14:03.749 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:03.741+0000 7efd33fff700 1 -- 172.21.15.2:0/840399611 >> 172.21.15.2:0/840399611 conn(0x7efd340f8e70 msgr2=0x7efd340fb2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-05-13T23:14:03.749 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:03.741+0000 7efd33fff700 1 -- 172.21.15.2:0/840399611 shutdown_connections 2022-05-13T23:14:03.751 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:03.741+0000 7efd33fff700 1 -- 172.21.15.2:0/840399611 wait complete. 2022-05-13T23:14:03.752 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:03.742+0000 7efd33fff700 1 Processor -- start 2022-05-13T23:14:03.752 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:03.742+0000 7efd33fff700 1 -- start start 2022-05-13T23:14:03.755 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:03.742+0000 7efd33fff700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7efd34070e00 0x7efd34112920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:14:03.755 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:03.743+0000 7efd33fff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7efd340720f0 0x7efd34112e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:14:03.756 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:03.743+0000 7efd33fff700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efd34072aa0 0x7efd34119ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:14:03.756 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:03.743+0000 7efd33fff700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd3411a430 con 0x7efd340720f0 2022-05-13T23:14:03.757 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:03.743+0000 7efd33fff700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd3411a5a0 con 0x7efd34072aa0 2022-05-13T23:14:03.757 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:03.743+0000 7efd33fff700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd3411a760 con 0x7efd34070e00 2022-05-13T23:14:03.764 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:03.743+0000 7efd32ffd700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7efd34070e00 0x7efd34112920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:14:03.764 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:03.743+0000 7efd327fc700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7efd340720f0 0x7efd34112e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:14:03.765 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:03.743+0000 7efd32ffd700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7efd34070e00 0x7efd34112920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.2:45748/0 (socket says 172.21.15.2:45748) 2022-05-13T23:14:03.765 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:03.743+0000 7efd327fc700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7efd340720f0 0x7efd34112e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:43276/0 (socket says 172.21.15.2:43276) 2022-05-13T23:14:03.765 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:03.743+0000 7efd32ffd700 1 -- 172.21.15.2:0/41589040 learned_addr learned my addr 172.21.15.2:0/41589040 (peer_addr_for_me v2:172.21.15.2:0/0) 2022-05-13T23:14:03.765 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:03.743+0000 7efd337fe700 1 --2- 172.21.15.2:0/41589040 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efd34072aa0 0x7efd34119ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:14:03.766 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:03.743+0000 7efd32ffd700 1 -- 172.21.15.2:0/41589040 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efd34072aa0 msgr2=0x7efd34119ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:14:03.766 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:03.743+0000 7efd32ffd700 1 --2- 172.21.15.2:0/41589040 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efd34072aa0 0x7efd34119ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:14:03.766 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:03.743+0000 7efd32ffd700 1 -- 172.21.15.2:0/41589040 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7efd340720f0 msgr2=0x7efd34112e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:14:03.766 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:03.743+0000 7efd32ffd700 1 --2- 172.21.15.2:0/41589040 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7efd340720f0 0x7efd34112e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:14:03.767 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:03.743+0000 7efd32ffd700 1 -- 172.21.15.2:0/41589040 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd28005040 con 0x7efd34070e00 2022-05-13T23:14:03.767 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:03.743+0000 7efd327fc700 1 --2- 172.21.15.2:0/41589040 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7efd340720f0 0x7efd34112e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-05-13T23:14:03.767 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:03.744+0000 7efd32ffd700 1 --2- 172.21.15.2:0/41589040 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7efd34070e00 0x7efd34112920 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7efd2400e970 tx=0x7efd2400b040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-05-13T23:14:03.767 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:03.744+0000 7efd1b7fe700 1 -- 172.21.15.2:0/41589040 <== mon.2 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd24007b40 con 0x7efd34070e00 2022-05-13T23:14:03.767 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:03.744+0000 7efd1b7fe700 1 -- 172.21.15.2:0/41589040 <== mon.2 v2:172.21.15.115:3300/0 2 ==== config(26 keys) v1 ==== 1087+0+0 (secure 0 0 0) 0x7efd2400f040 con 0x7efd34070e00 2022-05-13T23:14:03.768 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:03.744+0000 7efd33fff700 1 -- 172.21.15.2:0/41589040 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efd3411aa40 con 0x7efd34070e00 2022-05-13T23:14:03.768 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:03.744+0000 7efd1b7fe700 1 -- 172.21.15.2:0/41589040 <== mon.2 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd24014950 con 0x7efd34070e00 2022-05-13T23:14:03.768 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:03.744+0000 7efd33fff700 1 -- 172.21.15.2:0/41589040 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efd3411af80 con 0x7efd34070e00 2022-05-13T23:14:03.768 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:03.745+0000 7efd33fff700 1 -- 172.21.15.2:0/41589040 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efd3410d400 con 0x7efd34070e00 2022-05-13T23:14:03.769 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:03.747+0000 7efd1b7fe700 1 -- 172.21.15.2:0/41589040 <== mon.2 v2:172.21.15.115:3300/0 4 ==== mgrmap(e 16) v1 ==== 125136+0+0 (secure 0 0 0) 0x7efd2401b030 con 0x7efd34070e00 2022-05-13T23:14:03.769 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:03.748+0000 7efd1b7fe700 1 --2- 172.21.15.2:0/41589040 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7efd1c096d60 0x7efd1c099210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:14:03.769 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:03.748+0000 7efd1b7fe700 1 -- 172.21.15.2:0/41589040 <== mon.2 v2:172.21.15.115:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 7123+0+0 (secure 0 0 0) 0x7efd240be6a0 con 0x7efd34070e00 2022-05-13T23:14:03.769 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:03.748+0000 7efd327fc700 1 --2- 172.21.15.2:0/41589040 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7efd1c096d60 0x7efd1c099210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:14:03.770 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:03.749+0000 7efd327fc700 1 --2- 172.21.15.2:0/41589040 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7efd1c096d60 0x7efd1c099210 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7efd28004ee0 tx=0x7efd28009cb0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-05-13T23:14:03.770 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:03.752+0000 7efd1b7fe700 1 -- 172.21.15.2:0/41589040 <== mon.2 v2:172.21.15.115:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+165215 (secure 0 0 0) 0x7efd2404b570 con 0x7efd34070e00 2022-05-13T23:14:03.988 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:03.985+0000 7efd33fff700 1 -- 172.21.15.2:0/41589040 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7efd3402a4c0 con 0x7efd34070e00 2022-05-13T23:14:03.988 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:03.986+0000 7efd1b7fe700 1 -- 172.21.15.2:0/41589040 <== mon.2 v2:172.21.15.115:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+7297 (secure 0 0 0) 0x7efd2408bd30 con 0x7efd34070e00 2022-05-13T23:14:03.989 INFO:teuthology.orchestra.run.smithi002.stdout: 2022-05-13T23:14:03.989 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":10,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24460,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.2:6835/1341618941","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6834","nonce":1341618941},{"type":"v1","addr":"172.21.15.2:6835","nonce":1341618941}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24490,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.2:6839/1347050951","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6838","nonce":1347050951},{"type":"v1","addr":"172.21.15.2:6839","nonce":1347050951}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":24494,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.2:6837/3754333215","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6836","nonce":3754333215},{"type":"v1","addr":"172.21.15.2:6837","nonce":3754333215}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24511,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.47:6833/2783249369","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6832","nonce":2783249369},{"type":"v1","addr":"172.21.15.47:6833","nonce":2783249369}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":6},{"gid":24523,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.47:6835/4075838054","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6834","nonce":4075838054},{"type":"v1","addr":"172.21.15.47:6835","nonce":4075838054}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":7},{"gid":24530,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.2:6841/522559555","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6840","nonce":522559555},{"type":"v1","addr":"172.21.15.2:6841","nonce":522559555}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":24535,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.47:6837/4072863000","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6836","nonce":4072863000},{"type":"v1","addr":"172.21.15.47:6837","nonce":4072863000}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":8},{"gid":24547,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.47:6839/761355251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6838","nonce":761355251},{"type":"v1","addr":"172.21.15.47:6839","nonce":761355251}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":9},{"gid":24575,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.115:6833/381933929","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6832","nonce":381933929},{"type":"v1","addr":"172.21.15.115:6833","nonce":381933929}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[]} 2022-05-13T23:14:03.992 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:03.989+0000 7efd33fff700 1 -- 172.21.15.2:0/41589040 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7efd1c096d60 msgr2=0x7efd1c099210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:14:03.992 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:03.989+0000 7efd33fff700 1 --2- 172.21.15.2:0/41589040 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7efd1c096d60 0x7efd1c099210 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7efd28004ee0 tx=0x7efd28009cb0 comp rx=0 tx=0).stop 2022-05-13T23:14:03.992 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:03.989+0000 7efd33fff700 1 -- 172.21.15.2:0/41589040 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7efd34070e00 msgr2=0x7efd34112920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:14:03.992 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:03.989+0000 7efd33fff700 1 --2- 172.21.15.2:0/41589040 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7efd34070e00 0x7efd34112920 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7efd2400e970 tx=0x7efd2400b040 comp rx=0 tx=0).stop 2022-05-13T23:14:03.993 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:03.989+0000 7efd33fff700 1 -- 172.21.15.2:0/41589040 shutdown_connections 2022-05-13T23:14:03.993 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:03.989+0000 7efd33fff700 1 --2- 172.21.15.2:0/41589040 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efd34072aa0 0x7efd34119ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:14:03.993 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:03.989+0000 7efd33fff700 1 --2- 172.21.15.2:0/41589040 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7efd34070e00 0x7efd34112920 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:14:03.995 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:03.989+0000 7efd33fff700 1 --2- 172.21.15.2:0/41589040 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7efd1c096d60 0x7efd1c099210 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:14:03.996 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:03.989+0000 7efd33fff700 1 --2- 172.21.15.2:0/41589040 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7efd340720f0 0x7efd34112e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:14:03.996 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:03.989+0000 7efd33fff700 1 -- 172.21.15.2:0/41589040 >> 172.21.15.2:0/41589040 conn(0x7efd340f8e70 msgr2=0x7efd340fad00 unknown :-1 s=STATE_NONE l=0).mark_down 2022-05-13T23:14:03.996 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:03.989+0000 7efd33fff700 1 -- 172.21.15.2:0/41589040 shutdown_connections 2022-05-13T23:14:03.997 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:03.989+0000 7efd33fff700 1 -- 172.21.15.2:0/41589040 wait complete. 2022-05-13T23:14:03.997 INFO:teuthology.orchestra.run.smithi002.stderr:dumped fsmap epoch 10 2022-05-13T23:14:04.322 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:14:04 smithi002 ceph-mon[71816]: mds.? [v2:172.21.15.115:6832/381933929,v1:172.21.15.115:6833/381933929] up:boot 2022-05-13T23:14:04.323 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:14:04 smithi002 ceph-mon[71816]: fsmap 9 up:standby 2022-05-13T23:14:04.323 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:14:04 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2022-05-13T23:14:04.323 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:14:04 smithi002 ceph-mon[71816]: pgmap v278: 1 pgs: 1 active+clean; 577 KiB data, 74 MiB used, 1.0 TiB / 1.0 TiB avail 2022-05-13T23:14:04.323 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:14:04 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:14:04.324 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:14:04 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2022-05-13T23:14:04.324 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:14:04 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2022-05-13T23:14:04.324 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:14:04 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-13T23:14:04.324 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:14:04 smithi002 ceph-mon[71816]: Deploying daemon mds.f on smithi115 2022-05-13T23:14:04.325 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:14:04 smithi002 ceph-mon[71816]: from='client.? 172.21.15.2:0/41589040' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2022-05-13T23:14:04.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:14:04 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6832/381933929,v1:172.21.15.115:6833/381933929] up:boot 2022-05-13T23:14:04.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:14:04 smithi115 ceph-mon[78593]: fsmap 9 up:standby 2022-05-13T23:14:04.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:14:04 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2022-05-13T23:14:04.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:14:04 smithi115 ceph-mon[78593]: pgmap v278: 1 pgs: 1 active+clean; 577 KiB data, 74 MiB used, 1.0 TiB / 1.0 TiB avail 2022-05-13T23:14:04.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:14:04 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:14:04.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:14:04 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2022-05-13T23:14:04.362 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:14:04 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2022-05-13T23:14:04.362 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:14:04 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-13T23:14:04.362 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:14:04 smithi115 ceph-mon[78593]: Deploying daemon mds.f on smithi115 2022-05-13T23:14:04.363 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:14:04 smithi115 ceph-mon[78593]: from='client.? 172.21.15.2:0/41589040' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2022-05-13T23:14:04.546 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:14:04 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6832/381933929,v1:172.21.15.115:6833/381933929] up:boot 2022-05-13T23:14:04.547 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:14:04 smithi047 ceph-mon[77147]: fsmap 9 up:standby 2022-05-13T23:14:04.547 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:14:04 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2022-05-13T23:14:04.547 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:14:04 smithi047 ceph-mon[77147]: pgmap v278: 1 pgs: 1 active+clean; 577 KiB data, 74 MiB used, 1.0 TiB / 1.0 TiB avail 2022-05-13T23:14:04.547 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:14:04 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:14:04.548 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:14:04 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2022-05-13T23:14:04.548 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:14:04 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2022-05-13T23:14:04.548 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:14:04 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-13T23:14:04.548 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:14:04 smithi047 ceph-mon[77147]: Deploying daemon mds.f on smithi115 2022-05-13T23:14:04.549 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:14:04 smithi047 ceph-mon[77147]: from='client.? 172.21.15.2:0/41589040' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2022-05-13T23:14:05.108 INFO:journalctl@ceph.mds.f.smithi115.stdout:May 13 23:14:04 smithi115 systemd[1]: Starting Ceph mds.f for 16cf5f88-d311-11ec-be40-001a4aab830c... 2022-05-13T23:14:05.536 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:14:05 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-05-13T23:14:05.537 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:14:05 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-05-13T23:14:05.538 INFO:journalctl@ceph.mds.f.smithi115.stdout:May 13 23:14:05 smithi115 ceph-mds[106022]: -- 172.21.15.115:0/3270035064 <== mon.2 v2:172.21.15.115:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55dbf5e04340 con 0x55dbf5ddc800 2022-05-13T23:14:05.538 INFO:journalctl@ceph.mds.f.smithi115.stdout:May 13 23:14:05 smithi115 conmon[105998]: starting mds.f at 2022-05-13T23:14:05.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:14:05 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-05-13T23:14:05.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:14:05 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-05-13T23:14:05.573 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:14:05 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-05-13T23:14:05.573 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:14:05 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-05-13T23:14:05.786 INFO:journalctl@ceph.mds.f.smithi115.stdout:May 13 23:14:05 smithi115 bash[105814]: ee0547f882f172f984261b6edaec485e6dfc7216d6d8ef975915e57a62dd20cb 2022-05-13T23:14:05.786 INFO:journalctl@ceph.mds.f.smithi115.stdout:May 13 23:14:05 smithi115 systemd[1]: Started Ceph mds.f for 16cf5f88-d311-11ec-be40-001a4aab830c. 2022-05-13T23:14:06.018 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-05-13T23:14:06.246 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:06.244+0000 7efd29f9d700 1 -- 172.21.15.2:0/3148884719 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7efd240703b0 msgr2=0x7efd2406e8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:14:06.247 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:06.244+0000 7efd29f9d700 1 --2- 172.21.15.2:0/3148884719 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7efd240703b0 0x7efd2406e8b0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7efd180047b0 tx=0x7efd18009cb0 comp rx=0 tx=0).stop 2022-05-13T23:14:06.247 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:06.244+0000 7efd29f9d700 1 -- 172.21.15.2:0/3148884719 shutdown_connections 2022-05-13T23:14:06.247 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:06.244+0000 7efd29f9d700 1 --2- 172.21.15.2:0/3148884719 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7efd241033c0 0x7efd241057a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:14:06.247 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:06.244+0000 7efd29f9d700 1 --2- 172.21.15.2:0/3148884719 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7efd240703b0 0x7efd2406e8b0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:14:06.248 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:06.244+0000 7efd29f9d700 1 --2- 172.21.15.2:0/3148884719 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efd2406edf0 0x7efd2406f260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:14:06.248 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:06.244+0000 7efd29f9d700 1 -- 172.21.15.2:0/3148884719 >> 172.21.15.2:0/3148884719 conn(0x7efd240f6c60 msgr2=0x7efd240f9090 unknown :-1 s=STATE_NONE l=0).mark_down 2022-05-13T23:14:06.248 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:06.244+0000 7efd29f9d700 1 -- 172.21.15.2:0/3148884719 shutdown_connections 2022-05-13T23:14:06.248 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:06.244+0000 7efd29f9d700 1 -- 172.21.15.2:0/3148884719 wait complete. 2022-05-13T23:14:06.249 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:06.245+0000 7efd29f9d700 1 Processor -- start 2022-05-13T23:14:06.249 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:06.245+0000 7efd29f9d700 1 -- start start 2022-05-13T23:14:06.251 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:06.246+0000 7efd29f9d700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efd2406edf0 0x7efd240fe860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:14:06.252 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:06.246+0000 7efd29f9d700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7efd240703b0 0x7efd240feda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:14:06.252 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:06.246+0000 7efd29f9d700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7efd241033c0 0x7efd240fb3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:14:06.252 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:06.246+0000 7efd29f9d700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd240fbab0 con 0x7efd240703b0 2022-05-13T23:14:06.252 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:06.246+0000 7efd29f9d700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd240fbc20 con 0x7efd2406edf0 2022-05-13T23:14:06.253 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:06.246+0000 7efd29f9d700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd240fbde0 con 0x7efd241033c0 2022-05-13T23:14:06.253 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:06.246+0000 7efd2979c700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7efd241033c0 0x7efd240fb3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:14:06.253 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:06.246+0000 7efd2979c700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7efd241033c0 0x7efd240fb3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.2:45764/0 (socket says 172.21.15.2:45764) 2022-05-13T23:14:06.253 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:06.246+0000 7efd2979c700 1 -- 172.21.15.2:0/2884827291 learned_addr learned my addr 172.21.15.2:0/2884827291 (peer_addr_for_me v2:172.21.15.2:0/0) 2022-05-13T23:14:06.254 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:06.246+0000 7efd23fff700 1 --2- 172.21.15.2:0/2884827291 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7efd240703b0 0x7efd240feda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:14:06.254 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:06.246+0000 7efd28f9b700 1 --2- 172.21.15.2:0/2884827291 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efd2406edf0 0x7efd240fe860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:14:06.254 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:06.246+0000 7efd2979c700 1 -- 172.21.15.2:0/2884827291 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efd2406edf0 msgr2=0x7efd240fe860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:14:06.255 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:06.246+0000 7efd2979c700 1 --2- 172.21.15.2:0/2884827291 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efd2406edf0 0x7efd240fe860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:14:06.255 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:06.246+0000 7efd2979c700 1 -- 172.21.15.2:0/2884827291 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7efd240703b0 msgr2=0x7efd240feda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:14:06.255 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:06.246+0000 7efd2979c700 1 --2- 172.21.15.2:0/2884827291 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7efd240703b0 0x7efd240feda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:14:06.255 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:06.246+0000 7efd2979c700 1 -- 172.21.15.2:0/2884827291 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd18005040 con 0x7efd241033c0 2022-05-13T23:14:06.256 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:06.247+0000 7efd2979c700 1 --2- 172.21.15.2:0/2884827291 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7efd241033c0 0x7efd240fb3a0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7efd1c00eb70 tx=0x7efd1c00b040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-05-13T23:14:06.256 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:06.247+0000 7efd217fa700 1 -- 172.21.15.2:0/2884827291 <== mon.2 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd1c009860 con 0x7efd241033c0 2022-05-13T23:14:06.256 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:06.247+0000 7efd29f9d700 1 -- 172.21.15.2:0/2884827291 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efd2411a9d0 con 0x7efd241033c0 2022-05-13T23:14:06.256 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:06.247+0000 7efd217fa700 1 -- 172.21.15.2:0/2884827291 <== mon.2 v2:172.21.15.115:3300/0 2 ==== config(26 keys) v1 ==== 1087+0+0 (secure 0 0 0) 0x7efd1c00f040 con 0x7efd241033c0 2022-05-13T23:14:06.257 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:06.247+0000 7efd217fa700 1 -- 172.21.15.2:0/2884827291 <== mon.2 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd1c00c8f0 con 0x7efd241033c0 2022-05-13T23:14:06.257 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:06.247+0000 7efd29f9d700 1 -- 172.21.15.2:0/2884827291 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efd2411aef0 con 0x7efd241033c0 2022-05-13T23:14:06.258 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:06.248+0000 7efd29f9d700 1 -- 172.21.15.2:0/2884827291 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efd24108ec0 con 0x7efd241033c0 2022-05-13T23:14:06.258 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:06.250+0000 7efd217fa700 1 -- 172.21.15.2:0/2884827291 <== mon.2 v2:172.21.15.115:3300/0 4 ==== mgrmap(e 16) v1 ==== 125136+0+0 (secure 0 0 0) 0x7efd1c00cd20 con 0x7efd241033c0 2022-05-13T23:14:06.259 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:06.251+0000 7efd217fa700 1 --2- 172.21.15.2:0/2884827291 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7efd14096ce0 0x7efd14099190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:14:06.259 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:06.251+0000 7efd217fa700 1 -- 172.21.15.2:0/2884827291 <== mon.2 v2:172.21.15.115:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 7123+0+0 (secure 0 0 0) 0x7efd1c0bf320 con 0x7efd241033c0 2022-05-13T23:14:06.260 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:06.251+0000 7efd28f9b700 1 --2- 172.21.15.2:0/2884827291 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7efd14096ce0 0x7efd14099190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:14:06.260 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:06.252+0000 7efd28f9b700 1 --2- 172.21.15.2:0/2884827291 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7efd14096ce0 0x7efd14099190 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7efd18004e20 tx=0x7efd18009cb0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-05-13T23:14:06.260 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:06.253+0000 7efd217fa700 1 -- 172.21.15.2:0/2884827291 <== mon.2 v2:172.21.15.115:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+165215 (secure 0 0 0) 0x7efd1c08c9b0 con 0x7efd241033c0 2022-05-13T23:14:06.492 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:06.490+0000 7efd29f9d700 1 -- 172.21.15.2:0/2884827291 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7efd24101990 con 0x7efd241033c0 2022-05-13T23:14:06.493 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:06.491+0000 7efd217fa700 1 -- 172.21.15.2:0/2884827291 <== mon.2 v2:172.21.15.115:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 11 v11) v1 ==== 94+0+8052 (secure 0 0 0) 0x7efd1c08ffd0 con 0x7efd241033c0 2022-05-13T23:14:06.493 INFO:teuthology.orchestra.run.smithi002.stdout: 2022-05-13T23:14:06.493 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":11,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24460,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.2:6835/1341618941","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6834","nonce":1341618941},{"type":"v1","addr":"172.21.15.2:6835","nonce":1341618941}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24490,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.2:6839/1347050951","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6838","nonce":1347050951},{"type":"v1","addr":"172.21.15.2:6839","nonce":1347050951}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":24494,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.2:6837/3754333215","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6836","nonce":3754333215},{"type":"v1","addr":"172.21.15.2:6837","nonce":3754333215}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24511,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.47:6833/2783249369","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6832","nonce":2783249369},{"type":"v1","addr":"172.21.15.47:6833","nonce":2783249369}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":6},{"gid":24523,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.47:6835/4075838054","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6834","nonce":4075838054},{"type":"v1","addr":"172.21.15.47:6835","nonce":4075838054}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":7},{"gid":24530,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.2:6841/522559555","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6840","nonce":522559555},{"type":"v1","addr":"172.21.15.2:6841","nonce":522559555}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":24535,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.47:6837/4072863000","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6836","nonce":4072863000},{"type":"v1","addr":"172.21.15.47:6837","nonce":4072863000}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":8},{"gid":24547,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.47:6839/761355251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6838","nonce":761355251},{"type":"v1","addr":"172.21.15.47:6839","nonce":761355251}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":9},{"gid":24575,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.115:6833/381933929","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6832","nonce":381933929},{"type":"v1","addr":"172.21.15.115:6833","nonce":381933929}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10},{"gid":24587,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.115:6835/71062276","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6834","nonce":71062276},{"type":"v1","addr":"172.21.15.115:6835","nonce":71062276}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11}],"filesystems":[]} 2022-05-13T23:14:06.496 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:06.494+0000 7efd29f9d700 1 -- 172.21.15.2:0/2884827291 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7efd14096ce0 msgr2=0x7efd14099190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:14:06.496 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:06.494+0000 7efd29f9d700 1 --2- 172.21.15.2:0/2884827291 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7efd14096ce0 0x7efd14099190 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7efd18004e20 tx=0x7efd18009cb0 comp rx=0 tx=0).stop 2022-05-13T23:14:06.496 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:06.494+0000 7efd29f9d700 1 -- 172.21.15.2:0/2884827291 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7efd241033c0 msgr2=0x7efd240fb3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:14:06.496 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:06.494+0000 7efd29f9d700 1 --2- 172.21.15.2:0/2884827291 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7efd241033c0 0x7efd240fb3a0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7efd1c00eb70 tx=0x7efd1c00b040 comp rx=0 tx=0).stop 2022-05-13T23:14:06.497 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:06.494+0000 7efd29f9d700 1 -- 172.21.15.2:0/2884827291 shutdown_connections 2022-05-13T23:14:06.497 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:06.494+0000 7efd29f9d700 1 --2- 172.21.15.2:0/2884827291 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7efd241033c0 0x7efd240fb3a0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:14:06.497 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:06.494+0000 7efd29f9d700 1 --2- 172.21.15.2:0/2884827291 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efd2406edf0 0x7efd240fe860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:14:06.497 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:06.494+0000 7efd29f9d700 1 --2- 172.21.15.2:0/2884827291 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7efd14096ce0 0x7efd14099190 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:14:06.498 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:06.494+0000 7efd29f9d700 1 --2- 172.21.15.2:0/2884827291 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7efd240703b0 0x7efd240feda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:14:06.498 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:06.494+0000 7efd29f9d700 1 -- 172.21.15.2:0/2884827291 >> 172.21.15.2:0/2884827291 conn(0x7efd240f6c60 msgr2=0x7efd240f9040 unknown :-1 s=STATE_NONE l=0).mark_down 2022-05-13T23:14:06.499 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:06.494+0000 7efd29f9d700 1 -- 172.21.15.2:0/2884827291 shutdown_connections 2022-05-13T23:14:06.499 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:06.494+0000 7efd29f9d700 1 -- 172.21.15.2:0/2884827291 wait complete. 2022-05-13T23:14:06.500 INFO:teuthology.orchestra.run.smithi002.stderr:dumped fsmap epoch 11 2022-05-13T23:14:06.521 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:14:06 smithi002 ceph-mon[71816]: pgmap v279: 1 pgs: 1 active+clean; 577 KiB data, 74 MiB used, 1.0 TiB / 1.0 TiB avail 2022-05-13T23:14:06.521 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:14:06 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:14:06.521 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:14:06 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2022-05-13T23:14:06.522 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:14:06 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2022-05-13T23:14:06.522 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:14:06 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-13T23:14:06.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:14:06 smithi047 ceph-mon[77147]: pgmap v279: 1 pgs: 1 active+clean; 577 KiB data, 74 MiB used, 1.0 TiB / 1.0 TiB avail 2022-05-13T23:14:06.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:14:06 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:14:06.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:14:06 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2022-05-13T23:14:06.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:14:06 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2022-05-13T23:14:06.545 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:14:06 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-13T23:14:06.576 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:14:06 smithi115 ceph-mon[78593]: pgmap v279: 1 pgs: 1 active+clean; 577 KiB data, 74 MiB used, 1.0 TiB / 1.0 TiB avail 2022-05-13T23:14:06.576 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:14:06 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:14:06.577 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:14:06 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2022-05-13T23:14:06.577 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:14:06 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2022-05-13T23:14:06.578 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:14:06 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-13T23:14:07.296 INFO:journalctl@ceph.mds.i.smithi115.stdout:May 13 23:14:07 smithi115 systemd[1]: Starting Ceph mds.i for 16cf5f88-d311-11ec-be40-001a4aab830c... 2022-05-13T23:14:07.572 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:14:07 smithi002 ceph-mon[71816]: Deploying daemon mds.i on smithi115 2022-05-13T23:14:07.573 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:14:07 smithi002 ceph-mon[71816]: mds.? [v2:172.21.15.115:6834/71062276,v1:172.21.15.115:6835/71062276] up:boot 2022-05-13T23:14:07.573 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:14:07 smithi002 ceph-mon[71816]: fsmap 10 up:standby 2022-05-13T23:14:07.573 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:14:07 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2022-05-13T23:14:07.573 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:14:07 smithi002 ceph-mon[71816]: from='client.? 172.21.15.2:0/2884827291' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2022-05-13T23:14:07.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:14:07 smithi115 ceph-mon[78593]: Deploying daemon mds.i on smithi115 2022-05-13T23:14:07.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:14:07 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6834/71062276,v1:172.21.15.115:6835/71062276] up:boot 2022-05-13T23:14:07.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:14:07 smithi115 ceph-mon[78593]: fsmap 10 up:standby 2022-05-13T23:14:07.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:14:07 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2022-05-13T23:14:07.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:14:07 smithi115 ceph-mon[78593]: from='client.? 172.21.15.2:0/2884827291' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2022-05-13T23:14:07.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:14:07 smithi047 ceph-mon[77147]: Deploying daemon mds.i on smithi115 2022-05-13T23:14:07.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:14:07 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6834/71062276,v1:172.21.15.115:6835/71062276] up:boot 2022-05-13T23:14:07.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:14:07 smithi047 ceph-mon[77147]: fsmap 10 up:standby 2022-05-13T23:14:07.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:14:07 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2022-05-13T23:14:07.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:14:07 smithi047 ceph-mon[77147]: from='client.? 172.21.15.2:0/2884827291' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2022-05-13T23:14:08.108 INFO:journalctl@ceph.mds.i.smithi115.stdout:May 13 23:14:07 smithi115 ceph-mds[106591]: -- 172.21.15.115:0/69777059 <== mon.2 v2:172.21.15.115:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x556a58b44340 con 0x556a58b1d000 2022-05-13T23:14:08.108 INFO:journalctl@ceph.mds.i.smithi115.stdout:May 13 23:14:07 smithi115 conmon[106567]: starting mds.i at 2022-05-13T23:14:08.109 INFO:journalctl@ceph.mds.i.smithi115.stdout:May 13 23:14:07 smithi115 bash[106379]: a2f5aec744fe2df25d68a0fa88bd17fe8814afeabe92b3dbef0b7e9437a2b179 2022-05-13T23:14:08.109 INFO:journalctl@ceph.mds.i.smithi115.stdout:May 13 23:14:07 smithi115 systemd[1]: Started Ceph mds.i for 16cf5f88-d311-11ec-be40-001a4aab830c. 2022-05-13T23:14:08.524 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-05-13T23:14:08.572 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:14:08 smithi002 ceph-mon[71816]: pgmap v280: 1 pgs: 1 active+clean; 577 KiB data, 74 MiB used, 1.0 TiB / 1.0 TiB avail 2022-05-13T23:14:08.573 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:14:08 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:14:08.573 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:14:08 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2022-05-13T23:14:08.574 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:14:08 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2022-05-13T23:14:08.574 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:14:08 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-13T23:14:08.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:14:08 smithi115 ceph-mon[78593]: pgmap v280: 1 pgs: 1 active+clean; 577 KiB data, 74 MiB used, 1.0 TiB / 1.0 TiB avail 2022-05-13T23:14:08.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:14:08 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:14:08.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:14:08 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2022-05-13T23:14:08.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:14:08 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2022-05-13T23:14:08.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:14:08 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-13T23:14:08.750 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:08.748+0000 7f7f5b59e700 1 -- 172.21.15.2:0/3175669663 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f7f5c0fc730 msgr2=0x7f7f5c0fcb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:14:08.750 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:08.748+0000 7f7f5b59e700 1 --2- 172.21.15.2:0/3175669663 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f7f5c0fc730 0x7f7f5c0fcb80 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f7f50004870 tx=0x7f7f50009cb0 comp rx=0 tx=0).stop 2022-05-13T23:14:08.750 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:08.748+0000 7f7f5b59e700 1 -- 172.21.15.2:0/3175669663 shutdown_connections 2022-05-13T23:14:08.751 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:08.748+0000 7f7f5b59e700 1 --2- 172.21.15.2:0/3175669663 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7f5c0fd0c0 0x7f7f5c103980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:14:08.751 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:08.748+0000 7f7f5b59e700 1 --2- 172.21.15.2:0/3175669663 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7f5c0fb490 0x7f7f5c0fb8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:14:08.751 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:08.748+0000 7f7f5b59e700 1 --2- 172.21.15.2:0/3175669663 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f7f5c0fc730 0x7f7f5c0fcb80 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:14:08.751 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:08.748+0000 7f7f5b59e700 1 -- 172.21.15.2:0/3175669663 >> 172.21.15.2:0/3175669663 conn(0x7f7f5c0f6c40 msgr2=0x7f7f5c0f90b0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-05-13T23:14:08.751 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:08.748+0000 7f7f5b59e700 1 -- 172.21.15.2:0/3175669663 shutdown_connections 2022-05-13T23:14:08.752 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:08.748+0000 7f7f5b59e700 1 -- 172.21.15.2:0/3175669663 wait complete. 2022-05-13T23:14:08.752 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:08.749+0000 7f7f5b59e700 1 Processor -- start 2022-05-13T23:14:08.755 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:08.749+0000 7f7f5b59e700 1 -- start start 2022-05-13T23:14:08.755 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:08.749+0000 7f7f5b59e700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f7f5c0fb490 0x7f7f5c10e510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:14:08.756 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:08.749+0000 7f7f5b59e700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7f5c0fc730 0x7f7f5c10ea50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:14:08.756 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:08.749+0000 7f7f5b59e700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7f5c0fd0c0 0x7f7f5c115ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:14:08.756 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:08.749+0000 7f7f5b59e700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f5c116020 con 0x7f7f5c0fc730 2022-05-13T23:14:08.756 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:08.749+0000 7f7f5b59e700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f5c116190 con 0x7f7f5c0fd0c0 2022-05-13T23:14:08.756 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:08.749+0000 7f7f5b59e700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f5c116350 con 0x7f7f5c0fb490 2022-05-13T23:14:08.757 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:08.750+0000 7f7f5a59c700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f7f5c0fb490 0x7f7f5c10e510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:14:08.757 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:08.750+0000 7f7f59d9b700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7f5c0fc730 0x7f7f5c10ea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:14:08.757 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:08.750+0000 7f7f59d9b700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7f5c0fc730 0x7f7f5c10ea50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:43326/0 (socket says 172.21.15.2:43326) 2022-05-13T23:14:08.757 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:08.750+0000 7f7f59d9b700 1 -- 172.21.15.2:0/1351393479 learned_addr learned my addr 172.21.15.2:0/1351393479 (peer_addr_for_me v2:172.21.15.2:0/0) 2022-05-13T23:14:08.758 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:08.750+0000 7f7f5ad9d700 1 --2- 172.21.15.2:0/1351393479 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7f5c0fd0c0 0x7f7f5c115ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:14:08.758 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:08.750+0000 7f7f5a59c700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f7f5c0fb490 0x7f7f5c10e510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.2:45778/0 (socket says 172.21.15.2:45778) 2022-05-13T23:14:08.758 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:08.750+0000 7f7f59d9b700 1 -- 172.21.15.2:0/1351393479 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f7f5c0fb490 msgr2=0x7f7f5c10e510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:14:08.758 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:08.750+0000 7f7f59d9b700 1 --2- 172.21.15.2:0/1351393479 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f7f5c0fb490 0x7f7f5c10e510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:14:08.759 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:08.750+0000 7f7f59d9b700 1 -- 172.21.15.2:0/1351393479 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7f5c0fd0c0 msgr2=0x7f7f5c115ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:14:08.759 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:08.750+0000 7f7f59d9b700 1 --2- 172.21.15.2:0/1351393479 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7f5c0fd0c0 0x7f7f5c115ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:14:08.759 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:08.750+0000 7f7f59d9b700 1 -- 172.21.15.2:0/1351393479 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f50005040 con 0x7f7f5c0fc730 2022-05-13T23:14:08.760 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:08.750+0000 7f7f59d9b700 1 --2- 172.21.15.2:0/1351393479 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7f5c0fc730 0x7f7f5c10ea50 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f7f50004ee0 tx=0x7f7f50009cb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-05-13T23:14:08.760 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:08.751+0000 7f7f4affd700 1 -- 172.21.15.2:0/1351393479 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f5000e040 con 0x7f7f5c0fc730 2022-05-13T23:14:08.760 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:08.751+0000 7f7f4affd700 1 -- 172.21.15.2:0/1351393479 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1087+0+0 (secure 0 0 0) 0x7f7f50004200 con 0x7f7f5c0fc730 2022-05-13T23:14:08.760 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:08.751+0000 7f7f5b59e700 1 -- 172.21.15.2:0/1351393479 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7f5c1165d0 con 0x7f7f5c0fc730 2022-05-13T23:14:08.761 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:08.751+0000 7f7f4affd700 1 -- 172.21.15.2:0/1351393479 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f5001cab0 con 0x7f7f5c0fc730 2022-05-13T23:14:08.761 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:08.751+0000 7f7f5b59e700 1 -- 172.21.15.2:0/1351393479 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7f5c116a50 con 0x7f7f5c0fc730 2022-05-13T23:14:08.761 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:08.752+0000 7f7f5b59e700 1 -- 172.21.15.2:0/1351393479 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7f5c106d30 con 0x7f7f5c0fc730 2022-05-13T23:14:08.761 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:08.754+0000 7f7f4affd700 1 -- 172.21.15.2:0/1351393479 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 16) v1 ==== 125136+0+0 (secure 0 0 0) 0x7f7f50009240 con 0x7f7f5c0fc730 2022-05-13T23:14:08.762 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:08.755+0000 7f7f4affd700 1 --2- 172.21.15.2:0/1351393479 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f7f44096ce0 0x7f7f44099190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:14:08.763 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:08.755+0000 7f7f4affd700 1 -- 172.21.15.2:0/1351393479 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 7123+0+0 (secure 0 0 0) 0x7f7f500c0020 con 0x7f7f5c0fc730 2022-05-13T23:14:08.763 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:08.755+0000 7f7f5a59c700 1 --2- 172.21.15.2:0/1351393479 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f7f44096ce0 0x7f7f44099190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:14:08.763 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:08.756+0000 7f7f5a59c700 1 --2- 172.21.15.2:0/1351393479 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f7f44096ce0 0x7f7f44099190 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f7f5c0fc590 tx=0x7f7f4c00b040 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-05-13T23:14:08.763 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:08.759+0000 7f7f4affd700 1 -- 172.21.15.2:0/1351393479 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+165215 (secure 0 0 0) 0x7f7f5008d7a0 con 0x7f7f5c0fc730 2022-05-13T23:14:08.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:14:08 smithi047 ceph-mon[77147]: pgmap v280: 1 pgs: 1 active+clean; 577 KiB data, 74 MiB used, 1.0 TiB / 1.0 TiB avail 2022-05-13T23:14:08.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:14:08 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:14:08.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:14:08 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2022-05-13T23:14:08.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:14:08 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2022-05-13T23:14:08.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:14:08 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-13T23:14:08.992 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:08.990+0000 7f7f5b59e700 1 -- 172.21.15.2:0/1351393479 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7f5c117150 con 0x7f7f5c0fc730 2022-05-13T23:14:08.993 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:08.991+0000 7f7f4affd700 1 -- 172.21.15.2:0/1351393479 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 12 v12) v1 ==== 94+0+8813 (secure 0 0 0) 0x7f7f50090dc0 con 0x7f7f5c0fc730 2022-05-13T23:14:08.993 INFO:teuthology.orchestra.run.smithi002.stdout: 2022-05-13T23:14:08.993 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":12,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24460,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.2:6835/1341618941","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6834","nonce":1341618941},{"type":"v1","addr":"172.21.15.2:6835","nonce":1341618941}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24490,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.2:6839/1347050951","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6838","nonce":1347050951},{"type":"v1","addr":"172.21.15.2:6839","nonce":1347050951}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":24494,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.2:6837/3754333215","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6836","nonce":3754333215},{"type":"v1","addr":"172.21.15.2:6837","nonce":3754333215}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24511,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.47:6833/2783249369","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6832","nonce":2783249369},{"type":"v1","addr":"172.21.15.47:6833","nonce":2783249369}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":6},{"gid":24523,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.47:6835/4075838054","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6834","nonce":4075838054},{"type":"v1","addr":"172.21.15.47:6835","nonce":4075838054}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":7},{"gid":24530,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.2:6841/522559555","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6840","nonce":522559555},{"type":"v1","addr":"172.21.15.2:6841","nonce":522559555}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":24535,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.47:6837/4072863000","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6836","nonce":4072863000},{"type":"v1","addr":"172.21.15.47:6837","nonce":4072863000}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":8},{"gid":24547,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.47:6839/761355251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6838","nonce":761355251},{"type":"v1","addr":"172.21.15.47:6839","nonce":761355251}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":9},{"gid":24575,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.115:6833/381933929","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6832","nonce":381933929},{"type":"v1","addr":"172.21.15.115:6833","nonce":381933929}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10},{"gid":24587,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.115:6835/71062276","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6834","nonce":71062276},{"type":"v1","addr":"172.21.15.115:6835","nonce":71062276}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":24599,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.115:6837/2940957881","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6836","nonce":2940957881},{"type":"v1","addr":"172.21.15.115:6837","nonce":2940957881}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12}],"filesystems":[]} 2022-05-13T23:14:08.995 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:08.993+0000 7f7f5b59e700 1 -- 172.21.15.2:0/1351393479 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f7f44096ce0 msgr2=0x7f7f44099190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:14:08.995 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:08.993+0000 7f7f5b59e700 1 --2- 172.21.15.2:0/1351393479 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f7f44096ce0 0x7f7f44099190 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f7f5c0fc590 tx=0x7f7f4c00b040 comp rx=0 tx=0).stop 2022-05-13T23:14:08.995 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:08.993+0000 7f7f5b59e700 1 -- 172.21.15.2:0/1351393479 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7f5c0fc730 msgr2=0x7f7f5c10ea50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:14:08.995 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:08.993+0000 7f7f5b59e700 1 --2- 172.21.15.2:0/1351393479 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7f5c0fc730 0x7f7f5c10ea50 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f7f50004ee0 tx=0x7f7f50009cb0 comp rx=0 tx=0).stop 2022-05-13T23:14:08.996 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:08.993+0000 7f7f5b59e700 1 -- 172.21.15.2:0/1351393479 shutdown_connections 2022-05-13T23:14:08.996 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:08.993+0000 7f7f5b59e700 1 --2- 172.21.15.2:0/1351393479 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7f5c0fd0c0 0x7f7f5c115ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:14:08.996 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:08.993+0000 7f7f5b59e700 1 --2- 172.21.15.2:0/1351393479 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f7f5c0fb490 0x7f7f5c10e510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:14:08.996 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:08.993+0000 7f7f5b59e700 1 --2- 172.21.15.2:0/1351393479 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f7f44096ce0 0x7f7f44099190 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:14:08.997 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:08.993+0000 7f7f5b59e700 1 --2- 172.21.15.2:0/1351393479 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7f5c0fc730 0x7f7f5c10ea50 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:14:08.997 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:08.993+0000 7f7f5b59e700 1 -- 172.21.15.2:0/1351393479 >> 172.21.15.2:0/1351393479 conn(0x7f7f5c0f6c40 msgr2=0x7f7f5c0f8fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-05-13T23:14:08.997 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:08.993+0000 7f7f5b59e700 1 -- 172.21.15.2:0/1351393479 shutdown_connections 2022-05-13T23:14:08.997 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:08.993+0000 7f7f5b59e700 1 -- 172.21.15.2:0/1351393479 wait complete. 2022-05-13T23:14:08.998 INFO:teuthology.orchestra.run.smithi002.stderr:dumped fsmap epoch 12 2022-05-13T23:14:09.485 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:14:09 smithi115 ceph-mon[78593]: Deploying daemon mds.l on smithi115 2022-05-13T23:14:09.485 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:14:09 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6836/2940957881,v1:172.21.15.115:6837/2940957881] up:boot 2022-05-13T23:14:09.486 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:14:09 smithi115 ceph-mon[78593]: fsmap 11 up:standby 2022-05-13T23:14:09.486 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:14:09 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2022-05-13T23:14:09.486 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:14:09 smithi115 ceph-mon[78593]: from='client.? 172.21.15.2:0/1351393479' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2022-05-13T23:14:09.572 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:14:09 smithi002 ceph-mon[71816]: Deploying daemon mds.l on smithi115 2022-05-13T23:14:09.573 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:14:09 smithi002 ceph-mon[71816]: mds.? [v2:172.21.15.115:6836/2940957881,v1:172.21.15.115:6837/2940957881] up:boot 2022-05-13T23:14:09.573 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:14:09 smithi002 ceph-mon[71816]: fsmap 11 up:standby 2022-05-13T23:14:09.573 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:14:09 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2022-05-13T23:14:09.574 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:14:09 smithi002 ceph-mon[71816]: from='client.? 172.21.15.2:0/1351393479' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2022-05-13T23:14:09.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:14:09 smithi047 ceph-mon[77147]: Deploying daemon mds.l on smithi115 2022-05-13T23:14:09.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:14:09 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6836/2940957881,v1:172.21.15.115:6837/2940957881] up:boot 2022-05-13T23:14:09.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:14:09 smithi047 ceph-mon[77147]: fsmap 11 up:standby 2022-05-13T23:14:09.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:14:09 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2022-05-13T23:14:09.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:14:09 smithi047 ceph-mon[77147]: from='client.? 172.21.15.2:0/1351393479' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2022-05-13T23:14:09.858 INFO:journalctl@ceph.mds.l.smithi115.stdout:May 13 23:14:09 smithi115 systemd[1]: Starting Ceph mds.l for 16cf5f88-d311-11ec-be40-001a4aab830c... 2022-05-13T23:14:10.560 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:14:10 smithi002 ceph-mon[71816]: pgmap v281: 1 pgs: 1 active+clean; 577 KiB data, 74 MiB used, 1.0 TiB / 1.0 TiB avail 2022-05-13T23:14:10.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:14:10 smithi115 ceph-mon[78593]: pgmap v281: 1 pgs: 1 active+clean; 577 KiB data, 74 MiB used, 1.0 TiB / 1.0 TiB avail 2022-05-13T23:14:10.608 INFO:journalctl@ceph.mds.l.smithi115.stdout:May 13 23:14:10 smithi115 ceph-mds[107162]: -- 172.21.15.115:0/1423070645 <== mon.2 v2:172.21.15.115:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x556d845be340 con 0x556d84596c00 2022-05-13T23:14:10.609 INFO:journalctl@ceph.mds.l.smithi115.stdout:May 13 23:14:10 smithi115 conmon[107138]: starting mds.l at 2022-05-13T23:14:10.609 INFO:journalctl@ceph.mds.l.smithi115.stdout:May 13 23:14:10 smithi115 bash[106949]: 201973bd31ef5e921037020b5f881e00e867024bd2a9c22de2176b5cb10409a2 2022-05-13T23:14:10.609 INFO:journalctl@ceph.mds.l.smithi115.stdout:May 13 23:14:10 smithi115 systemd[1]: Started Ceph mds.l for 16cf5f88-d311-11ec-be40-001a4aab830c. 2022-05-13T23:14:10.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:14:10 smithi047 ceph-mon[77147]: pgmap v281: 1 pgs: 1 active+clean; 577 KiB data, 74 MiB used, 1.0 TiB / 1.0 TiB avail 2022-05-13T23:14:11.022 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-05-13T23:14:11.265 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:11.263+0000 7febfcb9d700 1 -- 172.21.15.2:0/2273492443 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7febf80fb490 msgr2=0x7febf80fb8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:14:11.266 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:11.263+0000 7febfcb9d700 1 --2- 172.21.15.2:0/2273492443 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7febf80fb490 0x7febf80fb8a0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7febe80047b0 tx=0x7febe8009cb0 comp rx=0 tx=0).stop 2022-05-13T23:14:11.266 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:11.263+0000 7febfcb9d700 1 -- 172.21.15.2:0/2273492443 shutdown_connections 2022-05-13T23:14:11.266 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:11.263+0000 7febfcb9d700 1 --2- 172.21.15.2:0/2273492443 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7febf80fd0c0 0x7febf8103980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:14:11.267 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:11.263+0000 7febfcb9d700 1 --2- 172.21.15.2:0/2273492443 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7febf80fb490 0x7febf80fb8a0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:14:11.267 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:11.263+0000 7febfcb9d700 1 --2- 172.21.15.2:0/2273492443 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7febf80fc730 0x7febf80fcb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:14:11.267 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:11.263+0000 7febfcb9d700 1 -- 172.21.15.2:0/2273492443 >> 172.21.15.2:0/2273492443 conn(0x7febf80f6c40 msgr2=0x7febf80f90b0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-05-13T23:14:11.267 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:11.264+0000 7febfcb9d700 1 -- 172.21.15.2:0/2273492443 shutdown_connections 2022-05-13T23:14:11.268 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:11.264+0000 7febfcb9d700 1 -- 172.21.15.2:0/2273492443 wait complete. 2022-05-13T23:14:11.268 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:11.264+0000 7febfcb9d700 1 Processor -- start 2022-05-13T23:14:11.270 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:11.265+0000 7febfcb9d700 1 -- start start 2022-05-13T23:14:11.270 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:11.265+0000 7febfcb9d700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7febf80fb490 0x7febf80701d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:14:11.271 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:11.265+0000 7febfcb9d700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7febf80fc730 0x7febf806e830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:14:11.271 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:11.265+0000 7febfcb9d700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7febf80fd0c0 0x7febf806ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:14:11.271 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:11.265+0000 7febfcb9d700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7febf8110170 con 0x7febf80fc730 2022-05-13T23:14:11.271 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:11.265+0000 7febfcb9d700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7febf806f2b0 con 0x7febf80fb490 2022-05-13T23:14:11.272 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:11.265+0000 7febfcb9d700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7febf806f470 con 0x7febf80fd0c0 2022-05-13T23:14:11.272 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:11.265+0000 7febf77fe700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7febf80fb490 0x7febf80701d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:14:11.272 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:11.265+0000 7febf6ffd700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7febf80fc730 0x7febf806e830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:14:11.272 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:11.265+0000 7febf7fff700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7febf80fd0c0 0x7febf806ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:14:11.272 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:11.265+0000 7febf77fe700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7febf80fb490 0x7febf80701d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.2:57876/0 (socket says 172.21.15.2:57876) 2022-05-13T23:14:11.273 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:11.265+0000 7febf77fe700 1 -- 172.21.15.2:0/2840076559 learned_addr learned my addr 172.21.15.2:0/2840076559 (peer_addr_for_me v2:172.21.15.2:0/0) 2022-05-13T23:14:11.273 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:11.266+0000 7febf7fff700 1 -- 172.21.15.2:0/2840076559 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7febf80fb490 msgr2=0x7febf80701d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:14:11.273 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:11.266+0000 7febf7fff700 1 --2- 172.21.15.2:0/2840076559 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7febf80fb490 0x7febf80701d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:14:11.273 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:11.266+0000 7febf7fff700 1 -- 172.21.15.2:0/2840076559 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7febf80fc730 msgr2=0x7febf806e830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:14:11.274 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:11.266+0000 7febf7fff700 1 --2- 172.21.15.2:0/2840076559 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7febf80fc730 0x7febf806e830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:14:11.274 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:11.266+0000 7febf7fff700 1 -- 172.21.15.2:0/2840076559 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7febe8005040 con 0x7febf80fd0c0 2022-05-13T23:14:11.274 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:11.266+0000 7febf6ffd700 1 --2- 172.21.15.2:0/2840076559 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7febf80fc730 0x7febf806e830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-05-13T23:14:11.274 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:11.266+0000 7febf77fe700 1 --2- 172.21.15.2:0/2840076559 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7febf80fb490 0x7febf80701d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-05-13T23:14:11.275 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:11.266+0000 7febf7fff700 1 --2- 172.21.15.2:0/2840076559 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7febf80fd0c0 0x7febf806ed70 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7febf000eb70 tx=0x7febf000b040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-05-13T23:14:11.275 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:11.266+0000 7febdffff700 1 -- 172.21.15.2:0/2840076559 <== mon.2 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febf0009870 con 0x7febf80fd0c0 2022-05-13T23:14:11.275 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:11.266+0000 7febdffff700 1 -- 172.21.15.2:0/2840076559 <== mon.2 v2:172.21.15.115:3300/0 2 ==== config(26 keys) v1 ==== 1087+0+0 (secure 0 0 0) 0x7febf000f040 con 0x7febf80fd0c0 2022-05-13T23:14:11.275 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:11.266+0000 7febdffff700 1 -- 172.21.15.2:0/2840076559 <== mon.2 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febf0007ea0 con 0x7febf80fd0c0 2022-05-13T23:14:11.275 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:11.266+0000 7febfcb9d700 1 -- 172.21.15.2:0/2840076559 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7febf811abb0 con 0x7febf80fd0c0 2022-05-13T23:14:11.276 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:11.267+0000 7febfcb9d700 1 -- 172.21.15.2:0/2840076559 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7febf811af00 con 0x7febf80fd0c0 2022-05-13T23:14:11.276 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:11.268+0000 7febfcb9d700 1 -- 172.21.15.2:0/2840076559 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7febf8106d30 con 0x7febf80fd0c0 2022-05-13T23:14:11.283 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:11.271+0000 7febdffff700 1 -- 172.21.15.2:0/2840076559 <== mon.2 v2:172.21.15.115:3300/0 4 ==== mgrmap(e 16) v1 ==== 125136+0+0 (secure 0 0 0) 0x7febf001a020 con 0x7febf80fd0c0 2022-05-13T23:14:11.283 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:11.272+0000 7febdffff700 1 --2- 172.21.15.2:0/2840076559 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7febe0096ce0 0x7febe0099190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:14:11.284 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:11.272+0000 7febdffff700 1 -- 172.21.15.2:0/2840076559 <== mon.2 v2:172.21.15.115:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 7123+0+0 (secure 0 0 0) 0x7febf00be870 con 0x7febf80fd0c0 2022-05-13T23:14:11.284 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:11.272+0000 7febf77fe700 1 --2- 172.21.15.2:0/2840076559 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7febe0096ce0 0x7febe0099190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:14:11.284 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:11.273+0000 7febdffff700 1 -- 172.21.15.2:0/2840076559 <== mon.2 v2:172.21.15.115:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+165215 (secure 0 0 0) 0x7febf008bf00 con 0x7febf80fd0c0 2022-05-13T23:14:11.284 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:11.273+0000 7febf77fe700 1 --2- 172.21.15.2:0/2840076559 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7febe0096ce0 0x7febe0099190 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7febe8004480 tx=0x7febe8009cb0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-05-13T23:14:11.526 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:11.525+0000 7febfcb9d700 1 -- 172.21.15.2:0/2840076559 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7febf81023f0 con 0x7febf80fd0c0 2022-05-13T23:14:11.527 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:11.525+0000 7febdffff700 1 -- 172.21.15.2:0/2840076559 <== mon.2 v2:172.21.15.115:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 13 v13) v1 ==== 94+0+9574 (secure 0 0 0) 0x7febf008f520 con 0x7febf80fd0c0 2022-05-13T23:14:11.527 INFO:teuthology.orchestra.run.smithi002.stdout: 2022-05-13T23:14:11.528 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":13,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24460,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.2:6835/1341618941","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6834","nonce":1341618941},{"type":"v1","addr":"172.21.15.2:6835","nonce":1341618941}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24490,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.2:6839/1347050951","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6838","nonce":1347050951},{"type":"v1","addr":"172.21.15.2:6839","nonce":1347050951}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":24494,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.2:6837/3754333215","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6836","nonce":3754333215},{"type":"v1","addr":"172.21.15.2:6837","nonce":3754333215}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24511,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.47:6833/2783249369","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6832","nonce":2783249369},{"type":"v1","addr":"172.21.15.47:6833","nonce":2783249369}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":6},{"gid":24523,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.47:6835/4075838054","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6834","nonce":4075838054},{"type":"v1","addr":"172.21.15.47:6835","nonce":4075838054}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":7},{"gid":24530,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.2:6841/522559555","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6840","nonce":522559555},{"type":"v1","addr":"172.21.15.2:6841","nonce":522559555}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":24535,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.47:6837/4072863000","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6836","nonce":4072863000},{"type":"v1","addr":"172.21.15.47:6837","nonce":4072863000}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":8},{"gid":24547,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.47:6839/761355251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6838","nonce":761355251},{"type":"v1","addr":"172.21.15.47:6839","nonce":761355251}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":9},{"gid":24575,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.115:6833/381933929","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6832","nonce":381933929},{"type":"v1","addr":"172.21.15.115:6833","nonce":381933929}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10},{"gid":24587,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.115:6835/71062276","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6834","nonce":71062276},{"type":"v1","addr":"172.21.15.115:6835","nonce":71062276}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":24599,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.115:6837/2940957881","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6836","nonce":2940957881},{"type":"v1","addr":"172.21.15.115:6837","nonce":2940957881}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":24611,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.115:6839/4149542923","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6838","nonce":4149542923},{"type":"v1","addr":"172.21.15.115:6839","nonce":4149542923}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":13}],"filesystems":[]} 2022-05-13T23:14:11.530 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:11.528+0000 7febfcb9d700 1 -- 172.21.15.2:0/2840076559 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7febe0096ce0 msgr2=0x7febe0099190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:14:11.531 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:11.528+0000 7febfcb9d700 1 --2- 172.21.15.2:0/2840076559 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7febe0096ce0 0x7febe0099190 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7febe8004480 tx=0x7febe8009cb0 comp rx=0 tx=0).stop 2022-05-13T23:14:11.531 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:11.528+0000 7febfcb9d700 1 -- 172.21.15.2:0/2840076559 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7febf80fd0c0 msgr2=0x7febf806ed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:14:11.531 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:11.528+0000 7febfcb9d700 1 --2- 172.21.15.2:0/2840076559 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7febf80fd0c0 0x7febf806ed70 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7febf000eb70 tx=0x7febf000b040 comp rx=0 tx=0).stop 2022-05-13T23:14:11.532 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:11.528+0000 7febfcb9d700 1 -- 172.21.15.2:0/2840076559 shutdown_connections 2022-05-13T23:14:11.532 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:11.528+0000 7febfcb9d700 1 --2- 172.21.15.2:0/2840076559 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7febf80fd0c0 0x7febf806ed70 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:14:11.532 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:11.528+0000 7febfcb9d700 1 --2- 172.21.15.2:0/2840076559 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7febf80fb490 0x7febf80701d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:14:11.532 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:11.528+0000 7febfcb9d700 1 --2- 172.21.15.2:0/2840076559 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7febe0096ce0 0x7febe0099190 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:14:11.533 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:11.528+0000 7febfcb9d700 1 --2- 172.21.15.2:0/2840076559 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7febf80fc730 0x7febf806e830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:14:11.533 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:11.528+0000 7febfcb9d700 1 -- 172.21.15.2:0/2840076559 >> 172.21.15.2:0/2840076559 conn(0x7febf80f6c40 msgr2=0x7febf80f9010 unknown :-1 s=STATE_NONE l=0).mark_down 2022-05-13T23:14:11.533 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:11.528+0000 7febfcb9d700 1 -- 172.21.15.2:0/2840076559 shutdown_connections 2022-05-13T23:14:11.533 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:11.529+0000 7febfcb9d700 1 -- 172.21.15.2:0/2840076559 wait complete. 2022-05-13T23:14:11.534 INFO:teuthology.orchestra.run.smithi002.stderr:dumped fsmap epoch 13 2022-05-13T23:14:11.564 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:14:11 smithi002 ceph-mon[71816]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:boot 2022-05-13T23:14:11.565 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:14:11 smithi002 ceph-mon[71816]: fsmap 12 up:standby 2022-05-13T23:14:11.565 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:14:11 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2022-05-13T23:14:11.566 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:14:11 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:14:11.566 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:14:11 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-13T23:14:11.566 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:14:11 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-13T23:14:11.567 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:14:11 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-13T23:14:11.570 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-05-13T23:14:11.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:14:11 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:boot 2022-05-13T23:14:11.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:14:11 smithi115 ceph-mon[78593]: fsmap 12 up:standby 2022-05-13T23:14:11.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:14:11 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2022-05-13T23:14:11.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:14:11 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:14:11.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:14:11 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-13T23:14:11.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:14:11 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-13T23:14:11.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:14:11 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-13T23:14:11.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:14:11 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:boot 2022-05-13T23:14:11.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:14:11 smithi047 ceph-mon[77147]: fsmap 12 up:standby 2022-05-13T23:14:11.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:14:11 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2022-05-13T23:14:11.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:14:11 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:14:11.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:14:11 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-13T23:14:11.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:14:11 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-13T23:14:11.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:14:11 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-13T23:14:11.805 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:11.799+0000 7fa22cb9d700 1 -- 172.21.15.2:0/1505381611 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa2280fd0c0 msgr2=0x7fa228103980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:14:11.805 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:11.799+0000 7fa22cb9d700 1 --2- 172.21.15.2:0/1505381611 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa2280fd0c0 0x7fa228103980 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fa220009be0 tx=0x7fa220004d10 comp rx=0 tx=0).stop 2022-05-13T23:14:11.806 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:11.799+0000 7fa22cb9d700 1 -- 172.21.15.2:0/1505381611 shutdown_connections 2022-05-13T23:14:11.806 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:11.799+0000 7fa22cb9d700 1 --2- 172.21.15.2:0/1505381611 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa2280fd0c0 0x7fa228103980 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:14:11.806 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:11.799+0000 7fa22cb9d700 1 --2- 172.21.15.2:0/1505381611 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa2280fb490 0x7fa2280fb8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:14:11.807 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:11.799+0000 7fa22cb9d700 1 --2- 172.21.15.2:0/1505381611 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fa2280fc730 0x7fa2280fcb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:14:11.807 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:11.799+0000 7fa22cb9d700 1 -- 172.21.15.2:0/1505381611 >> 172.21.15.2:0/1505381611 conn(0x7fa2280f6c40 msgr2=0x7fa2280f90b0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-05-13T23:14:11.807 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:11.800+0000 7fa22cb9d700 1 -- 172.21.15.2:0/1505381611 shutdown_connections 2022-05-13T23:14:11.807 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:11.800+0000 7fa22cb9d700 1 -- 172.21.15.2:0/1505381611 wait complete. 2022-05-13T23:14:11.808 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:11.800+0000 7fa22cb9d700 1 Processor -- start 2022-05-13T23:14:11.808 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:11.801+0000 7fa22cb9d700 1 -- start start 2022-05-13T23:14:11.808 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:11.801+0000 7fa22cb9d700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa2280fb490 0x7fa22810e340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:14:11.808 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:11.801+0000 7fa22cb9d700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fa2280fc730 0x7fa22810e880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:14:11.808 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:11.801+0000 7fa22cb9d700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa2280fd0c0 0x7fa228115910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:14:11.809 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:11.801+0000 7fa22cb9d700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa228115e50 con 0x7fa2280fd0c0 2022-05-13T23:14:11.809 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:11.801+0000 7fa22cb9d700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa228115fc0 con 0x7fa2280fb490 2022-05-13T23:14:11.809 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:11.801+0000 7fa22cb9d700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa228116180 con 0x7fa2280fc730 2022-05-13T23:14:11.809 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:11.801+0000 7fa2277fe700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa2280fb490 0x7fa22810e340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:14:11.810 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:11.801+0000 7fa226ffd700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fa2280fc730 0x7fa22810e880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:14:11.810 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:11.801+0000 7fa2277fe700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa2280fb490 0x7fa22810e340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.2:57892/0 (socket says 172.21.15.2:57892) 2022-05-13T23:14:11.810 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:11.801+0000 7fa2277fe700 1 -- 172.21.15.2:0/1347886734 learned_addr learned my addr 172.21.15.2:0/1347886734 (peer_addr_for_me v2:172.21.15.2:0/0) 2022-05-13T23:14:11.810 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:11.801+0000 7fa227fff700 1 --2- 172.21.15.2:0/1347886734 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa2280fd0c0 0x7fa228115910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:14:11.811 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:11.802+0000 7fa2277fe700 1 -- 172.21.15.2:0/1347886734 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fa2280fc730 msgr2=0x7fa22810e880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:14:11.811 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:11.802+0000 7fa2277fe700 1 --2- 172.21.15.2:0/1347886734 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fa2280fc730 0x7fa22810e880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:14:11.811 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:11.802+0000 7fa2277fe700 1 -- 172.21.15.2:0/1347886734 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa2280fd0c0 msgr2=0x7fa228115910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:14:11.812 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:11.802+0000 7fa2277fe700 1 --2- 172.21.15.2:0/1347886734 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa2280fd0c0 0x7fa228115910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:14:11.812 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:11.802+0000 7fa2277fe700 1 -- 172.21.15.2:0/1347886734 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2200097c0 con 0x7fa2280fb490 2022-05-13T23:14:11.814 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:11.802+0000 7fa227fff700 1 --2- 172.21.15.2:0/1347886734 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa2280fd0c0 0x7fa228115910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2022-05-13T23:14:11.814 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:11.802+0000 7fa2277fe700 1 --2- 172.21.15.2:0/1347886734 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa2280fb490 0x7fa22810e340 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fa21800e940 tx=0x7fa21800b040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-05-13T23:14:11.814 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:11.803+0000 7fa20ffff700 1 -- 172.21.15.2:0/1347886734 <== mon.1 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa218007b40 con 0x7fa2280fb490 2022-05-13T23:14:11.814 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:11.803+0000 7fa20ffff700 1 -- 172.21.15.2:0/1347886734 <== mon.1 v2:172.21.15.47:3300/0 2 ==== config(26 keys) v1 ==== 1087+0+0 (secure 0 0 0) 0x7fa21800f040 con 0x7fa2280fb490 2022-05-13T23:14:11.815 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:11.803+0000 7fa22cb9d700 1 -- 172.21.15.2:0/1347886734 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa228116400 con 0x7fa2280fb490 2022-05-13T23:14:11.817 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:11.803+0000 7fa20ffff700 1 -- 172.21.15.2:0/1347886734 <== mon.1 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2180148b0 con 0x7fa2280fb490 2022-05-13T23:14:11.818 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:11.803+0000 7fa22cb9d700 1 -- 172.21.15.2:0/1347886734 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa228116970 con 0x7fa2280fb490 2022-05-13T23:14:11.819 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:11.804+0000 7fa22cb9d700 1 -- 172.21.15.2:0/1347886734 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa22806f900 con 0x7fa2280fb490 2022-05-13T23:14:11.819 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:11.806+0000 7fa20ffff700 1 -- 172.21.15.2:0/1347886734 <== mon.1 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 16) v1 ==== 125136+0+0 (secure 0 0 0) 0x7fa21801b030 con 0x7fa2280fb490 2022-05-13T23:14:11.820 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:11.807+0000 7fa20ffff700 1 --2- 172.21.15.2:0/1347886734 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7fa210096e30 0x7fa2100992e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:14:11.820 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:11.807+0000 7fa20ffff700 1 -- 172.21.15.2:0/1347886734 <== mon.1 v2:172.21.15.47:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 7123+0+0 (secure 0 0 0) 0x7fa2180be5f0 con 0x7fa2280fb490 2022-05-13T23:14:11.820 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:11.807+0000 7fa226ffd700 1 --2- 172.21.15.2:0/1347886734 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7fa210096e30 0x7fa2100992e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:14:11.820 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:11.808+0000 7fa226ffd700 1 --2- 172.21.15.2:0/1347886734 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7fa210096e30 0x7fa2100992e0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fa21c004790 tx=0x7fa21c00b040 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-05-13T23:14:11.820 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:11.811+0000 7fa20ffff700 1 -- 172.21.15.2:0/1347886734 <== mon.1 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+165215 (secure 0 0 0) 0x7fa21804b570 con 0x7fa2280fb490 2022-05-13T23:14:12.010 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:12.009+0000 7fa22cb9d700 1 -- 172.21.15.2:0/1347886734 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa228101f50 con 0x7fa2280fb490 2022-05-13T23:14:12.011 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:12.010+0000 7fa20ffff700 1 -- 172.21.15.2:0/1347886734 <== mon.1 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v79) v1 ==== 74+0+18042 (secure 0 0 0) 0x7fa21808bd70 con 0x7fa2280fb490 2022-05-13T23:14:12.012 INFO:teuthology.orchestra.run.smithi002.stdout: 2022-05-13T23:14:12.012 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":79,"fsid":"16cf5f88-d311-11ec-be40-001a4aab830c","created":"2022-05-13T23:06:00.765453+0000","modified":"2022-05-13T23:13:21.942299+0000","last_up_change":"2022-05-13T23:13:20.935982+0000","last_in_change":"2022-05-13T23:13:02.796000+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2022-05-13T23:09:32.000644+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"11e700e8-f3e8-4903-b52c-db354b4395bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6802","nonce":1174843704},{"type":"v1","addr":"172.21.15.2:6803","nonce":1174843704}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6804","nonce":1174843704},{"type":"v1","addr":"172.21.15.2:6805","nonce":1174843704}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6808","nonce":1174843704},{"type":"v1","addr":"172.21.15.2:6809","nonce":1174843704}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6806","nonce":1174843704},{"type":"v1","addr":"172.21.15.2:6807","nonce":1174843704}]},"public_addr":"172.21.15.2:6803/1174843704","cluster_addr":"172.21.15.2:6805/1174843704","heartbeat_back_addr":"172.21.15.2:6809/1174843704","heartbeat_front_addr":"172.21.15.2:6807/1174843704","state":["exists","up"]},{"osd":1,"uuid":"eabfa3cf-f2a3-4995-a7de-f6d8ba02ee6f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6800","nonce":3853826948},{"type":"v1","addr":"172.21.15.47:6801","nonce":3853826948}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6802","nonce":3853826948},{"type":"v1","addr":"172.21.15.47:6803","nonce":3853826948}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6806","nonce":3853826948},{"type":"v1","addr":"172.21.15.47:6807","nonce":3853826948}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6804","nonce":3853826948},{"type":"v1","addr":"172.21.15.47:6805","nonce":3853826948}]},"public_addr":"172.21.15.47:6801/3853826948","cluster_addr":"172.21.15.47:6803/3853826948","heartbeat_back_addr":"172.21.15.47:6807/3853826948","heartbeat_front_addr":"172.21.15.47:6805/3853826948","state":["exists","up"]},{"osd":2,"uuid":"153c5bd5-8200-47a7-a135-8a95b6752cd2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6800","nonce":2562256956},{"type":"v1","addr":"172.21.15.115:6801","nonce":2562256956}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6802","nonce":2562256956},{"type":"v1","addr":"172.21.15.115:6803","nonce":2562256956}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6806","nonce":2562256956},{"type":"v1","addr":"172.21.15.115:6807","nonce":2562256956}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6804","nonce":2562256956},{"type":"v1","addr":"172.21.15.115:6805","nonce":2562256956}]},"public_addr":"172.21.15.115:6801/2562256956","cluster_addr":"172.21.15.115:6803/2562256956","heartbeat_back_addr":"172.21.15.115:6807/2562256956","heartbeat_front_addr":"172.21.15.115:6805/2562256956","state":["exists","up"]},{"osd":3,"uuid":"6ca9c602-5936-48da-a861-c437e6a33765","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6810","nonce":1099909411},{"type":"v1","addr":"172.21.15.2:6811","nonce":1099909411}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6812","nonce":1099909411},{"type":"v1","addr":"172.21.15.2:6813","nonce":1099909411}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6816","nonce":1099909411},{"type":"v1","addr":"172.21.15.2:6817","nonce":1099909411}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6814","nonce":1099909411},{"type":"v1","addr":"172.21.15.2:6815","nonce":1099909411}]},"public_addr":"172.21.15.2:6811/1099909411","cluster_addr":"172.21.15.2:6813/1099909411","heartbeat_back_addr":"172.21.15.2:6817/1099909411","heartbeat_front_addr":"172.21.15.2:6815/1099909411","state":["exists","up"]},{"osd":4,"uuid":"54d943d9-2610-422a-9e59-4b8212aced31","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6808","nonce":882753861},{"type":"v1","addr":"172.21.15.47:6809","nonce":882753861}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6810","nonce":882753861},{"type":"v1","addr":"172.21.15.47:6811","nonce":882753861}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6814","nonce":882753861},{"type":"v1","addr":"172.21.15.47:6815","nonce":882753861}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6812","nonce":882753861},{"type":"v1","addr":"172.21.15.47:6813","nonce":882753861}]},"public_addr":"172.21.15.47:6809/882753861","cluster_addr":"172.21.15.47:6811/882753861","heartbeat_back_addr":"172.21.15.47:6815/882753861","heartbeat_front_addr":"172.21.15.47:6813/882753861","state":["exists","up"]},{"osd":5,"uuid":"c18566cf-5403-4057-b2ee-aa298e87e8e9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6808","nonce":2122391993},{"type":"v1","addr":"172.21.15.115:6809","nonce":2122391993}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6810","nonce":2122391993},{"type":"v1","addr":"172.21.15.115:6811","nonce":2122391993}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6814","nonce":2122391993},{"type":"v1","addr":"172.21.15.115:6815","nonce":2122391993}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6812","nonce":2122391993},{"type":"v1","addr":"172.21.15.115:6813","nonce":2122391993}]},"public_addr":"172.21.15.115:6809/2122391993","cluster_addr":"172.21.15.115:6811/2122391993","heartbeat_back_addr":"172.21.15.115:6815/2122391993","heartbeat_front_addr":"172.21.15.115:6813/2122391993","state":["exists","up"]},{"osd":6,"uuid":"38ec7b6b-30f4-4d97-ad7e-e160ae59c843","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6818","nonce":1892198790},{"type":"v1","addr":"172.21.15.2:6819","nonce":1892198790}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6820","nonce":1892198790},{"type":"v1","addr":"172.21.15.2:6821","nonce":1892198790}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6824","nonce":1892198790},{"type":"v1","addr":"172.21.15.2:6825","nonce":1892198790}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6822","nonce":1892198790},{"type":"v1","addr":"172.21.15.2:6823","nonce":1892198790}]},"public_addr":"172.21.15.2:6819/1892198790","cluster_addr":"172.21.15.2:6821/1892198790","heartbeat_back_addr":"172.21.15.2:6825/1892198790","heartbeat_front_addr":"172.21.15.2:6823/1892198790","state":["exists","up"]},{"osd":7,"uuid":"e799b560-c61e-4369-b08b-45e5a6af37c0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6816","nonce":2812542825},{"type":"v1","addr":"172.21.15.47:6817","nonce":2812542825}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6818","nonce":2812542825},{"type":"v1","addr":"172.21.15.47:6819","nonce":2812542825}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6822","nonce":2812542825},{"type":"v1","addr":"172.21.15.47:6823","nonce":2812542825}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6820","nonce":2812542825},{"type":"v1","addr":"172.21.15.47:6821","nonce":2812542825}]},"public_addr":"172.21.15.47:6817/2812542825","cluster_addr":"172.21.15.47:6819/2812542825","heartbeat_back_addr":"172.21.15.47:6823/2812542825","heartbeat_front_addr":"172.21.15.47:6821/2812542825","state":["exists","up"]},{"osd":8,"uuid":"a58f2c0e-e487-41cf-8d2e-8c937cac9ebc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":58,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6816","nonce":3959632033},{"type":"v1","addr":"172.21.15.115:6817","nonce":3959632033}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6818","nonce":3959632033},{"type":"v1","addr":"172.21.15.115:6819","nonce":3959632033}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6822","nonce":3959632033},{"type":"v1","addr":"172.21.15.115:6823","nonce":3959632033}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6820","nonce":3959632033},{"type":"v1","addr":"172.21.15.115:6821","nonce":3959632033}]},"public_addr":"172.21.15.115:6817/3959632033","cluster_addr":"172.21.15.115:6819/3959632033","heartbeat_back_addr":"172.21.15.115:6823/3959632033","heartbeat_front_addr":"172.21.15.115:6821/3959632033","state":["exists","up"]},{"osd":9,"uuid":"12cead96-f726-4f7a-ab84-eb191b778a5c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":66,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6826","nonce":3004162255},{"type":"v1","addr":"172.21.15.2:6827","nonce":3004162255}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6828","nonce":3004162255},{"type":"v1","addr":"172.21.15.2:6829","nonce":3004162255}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6832","nonce":3004162255},{"type":"v1","addr":"172.21.15.2:6833","nonce":3004162255}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6830","nonce":3004162255},{"type":"v1","addr":"172.21.15.2:6831","nonce":3004162255}]},"public_addr":"172.21.15.2:6827/3004162255","cluster_addr":"172.21.15.2:6829/3004162255","heartbeat_back_addr":"172.21.15.2:6833/3004162255","heartbeat_front_addr":"172.21.15.2:6831/3004162255","state":["exists","up"]},{"osd":10,"uuid":"4f648118-7855-48d6-86d5-44eed5295f0c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":71,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6824","nonce":3443413998},{"type":"v1","addr":"172.21.15.47:6825","nonce":3443413998}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6826","nonce":3443413998},{"type":"v1","addr":"172.21.15.47:6827","nonce":3443413998}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6830","nonce":3443413998},{"type":"v1","addr":"172.21.15.47:6831","nonce":3443413998}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6828","nonce":3443413998},{"type":"v1","addr":"172.21.15.47:6829","nonce":3443413998}]},"public_addr":"172.21.15.47:6825/3443413998","cluster_addr":"172.21.15.47:6827/3443413998","heartbeat_back_addr":"172.21.15.47:6831/3443413998","heartbeat_front_addr":"172.21.15.47:6829/3443413998","state":["exists","up"]},{"osd":11,"uuid":"916541e0-fb28-46d5-9743-801c54c5a6d1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":78,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6824","nonce":1758718209},{"type":"v1","addr":"172.21.15.115:6825","nonce":1758718209}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6826","nonce":1758718209},{"type":"v1","addr":"172.21.15.115:6827","nonce":1758718209}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6830","nonce":1758718209},{"type":"v1","addr":"172.21.15.115:6831","nonce":1758718209}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6828","nonce":1758718209},{"type":"v1","addr":"172.21.15.115:6829","nonce":1758718209}]},"public_addr":"172.21.15.115:6825/1758718209","cluster_addr":"172.21.15.115:6827/1758718209","heartbeat_back_addr":"172.21.15.115:6831/1758718209","heartbeat_front_addr":"172.21.15.115:6829/1758718209","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-05-13T23:08:43.757857+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-05-13T23:09:06.537139+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-05-13T23:09:28.524837+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-05-13T23:10:02.057583+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-05-13T23:10:26.210648+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-05-13T23:10:47.885853+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-05-13T23:11:19.141865+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-05-13T23:11:42.376065+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-05-13T23:12:04.175870+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-05-13T23:12:34.695347+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-05-13T23:12:57.762613+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-05-13T23:13:19.114284+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.2:0/477249925":"2022-05-14T23:07:04.986087+0000","172.21.15.2:6800/3226099420":"2022-05-14T23:07:04.986087+0000","172.21.15.2:0/2214093915":"2022-05-14T23:07:04.986087+0000","172.21.15.2:0/1039403355":"2022-05-14T23:07:04.986087+0000","172.21.15.2:0/22091675":"2022-05-14T23:06:36.304108+0000","172.21.15.2:6800/1255134634":"2022-05-14T23:06:36.304108+0000","172.21.15.2:6801/3226099420":"2022-05-14T23:07:04.986087+0000","172.21.15.2:0/3768677276":"2022-05-14T23:06:36.304108+0000","172.21.15.2:6801/1255134634":"2022-05-14T23:06:36.304108+0000","172.21.15.2:0/407841268":"2022-05-14T23:06:36.304108+0000"},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2022-05-13T23:14:12.015 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:12.012+0000 7fa22cb9d700 1 -- 172.21.15.2:0/1347886734 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7fa210096e30 msgr2=0x7fa2100992e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:14:12.015 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:12.012+0000 7fa22cb9d700 1 --2- 172.21.15.2:0/1347886734 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7fa210096e30 0x7fa2100992e0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fa21c004790 tx=0x7fa21c00b040 comp rx=0 tx=0).stop 2022-05-13T23:14:12.016 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:12.012+0000 7fa22cb9d700 1 -- 172.21.15.2:0/1347886734 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa2280fb490 msgr2=0x7fa22810e340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:14:12.016 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:12.012+0000 7fa22cb9d700 1 --2- 172.21.15.2:0/1347886734 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa2280fb490 0x7fa22810e340 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fa21800e940 tx=0x7fa21800b040 comp rx=0 tx=0).stop 2022-05-13T23:14:12.016 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:12.013+0000 7fa22cb9d700 1 -- 172.21.15.2:0/1347886734 shutdown_connections 2022-05-13T23:14:12.016 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:12.013+0000 7fa22cb9d700 1 --2- 172.21.15.2:0/1347886734 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7fa210096e30 0x7fa2100992e0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:14:12.017 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:12.013+0000 7fa22cb9d700 1 --2- 172.21.15.2:0/1347886734 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa2280fd0c0 0x7fa228115910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:14:12.017 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:12.013+0000 7fa22cb9d700 1 --2- 172.21.15.2:0/1347886734 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa2280fb490 0x7fa22810e340 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:14:12.017 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:12.013+0000 7fa22cb9d700 1 --2- 172.21.15.2:0/1347886734 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fa2280fc730 0x7fa22810e880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:14:12.018 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:12.013+0000 7fa22cb9d700 1 -- 172.21.15.2:0/1347886734 >> 172.21.15.2:0/1347886734 conn(0x7fa2280f6c40 msgr2=0x7fa2280f8600 unknown :-1 s=STATE_NONE l=0).mark_down 2022-05-13T23:14:12.018 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:12.013+0000 7fa22cb9d700 1 -- 172.21.15.2:0/1347886734 shutdown_connections 2022-05-13T23:14:12.018 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:12.013+0000 7fa22cb9d700 1 -- 172.21.15.2:0/1347886734 wait complete. 2022-05-13T23:14:12.042 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2022-05-13T23:09:32.000644+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}] 2022-05-13T23:14:12.043 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2022-05-13T23:14:12.269 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:12.267+0000 7f923359e700 1 -- 172.21.15.2:0/1031100016 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f92340fd0f0 msgr2=0x7f92341039b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:14:12.269 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:12.267+0000 7f923359e700 1 --2- 172.21.15.2:0/1031100016 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f92340fd0f0 0x7f92341039b0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f922c009be0 tx=0x7f922c004d10 comp rx=0 tx=0).stop 2022-05-13T23:14:12.270 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:12.267+0000 7f923359e700 1 -- 172.21.15.2:0/1031100016 shutdown_connections 2022-05-13T23:14:12.270 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:12.267+0000 7f923359e700 1 --2- 172.21.15.2:0/1031100016 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f92340fd0f0 0x7f92341039b0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:14:12.270 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:12.267+0000 7f923359e700 1 --2- 172.21.15.2:0/1031100016 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f92340fb4c0 0x7f92340fb8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:14:12.270 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:12.267+0000 7f923359e700 1 --2- 172.21.15.2:0/1031100016 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f92340fc760 0x7f92340fcbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:14:12.271 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:12.267+0000 7f923359e700 1 -- 172.21.15.2:0/1031100016 >> 172.21.15.2:0/1031100016 conn(0x7f92340f6cb0 msgr2=0x7f92340f90e0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-05-13T23:14:12.271 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:12.267+0000 7f923359e700 1 -- 172.21.15.2:0/1031100016 shutdown_connections 2022-05-13T23:14:12.271 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:12.267+0000 7f923359e700 1 -- 172.21.15.2:0/1031100016 wait complete. 2022-05-13T23:14:12.271 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:12.268+0000 7f923359e700 1 Processor -- start 2022-05-13T23:14:12.271 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:12.268+0000 7f923359e700 1 -- start start 2022-05-13T23:14:12.274 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:12.268+0000 7f923359e700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f92340fb4c0 0x7f923410e360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:14:12.275 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:12.268+0000 7f923359e700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f92340fc760 0x7f923410e8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:14:12.275 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:12.268+0000 7f923359e700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f92340fd0f0 0x7f9234115930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:14:12.275 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:12.269+0000 7f923359e700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9234115e70 con 0x7f92340fb4c0 2022-05-13T23:14:12.275 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:12.269+0000 7f923359e700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9234115fe0 con 0x7f92340fd0f0 2022-05-13T23:14:12.275 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:12.269+0000 7f923359e700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92341161a0 con 0x7f92340fc760 2022-05-13T23:14:12.276 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:12.269+0000 7f923259c700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f92340fb4c0 0x7f923410e360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:14:12.276 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:12.269+0000 7f9231d9b700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f92340fc760 0x7f923410e8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:14:12.276 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:12.269+0000 7f9232d9d700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f92340fd0f0 0x7f9234115930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:14:12.276 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:12.269+0000 7f9232d9d700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f92340fd0f0 0x7f9234115930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.2:57914/0 (socket says 172.21.15.2:57914) 2022-05-13T23:14:12.277 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:12.269+0000 7f9232d9d700 1 -- 172.21.15.2:0/3352699056 learned_addr learned my addr 172.21.15.2:0/3352699056 (peer_addr_for_me v2:172.21.15.2:0/0) 2022-05-13T23:14:12.277 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:12.269+0000 7f9231d9b700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f92340fc760 0x7f923410e8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.2:43290/0 (socket says 172.21.15.2:43290) 2022-05-13T23:14:12.277 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:12.269+0000 7f9232d9d700 1 -- 172.21.15.2:0/3352699056 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f92340fc760 msgr2=0x7f923410e8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:14:12.277 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:12.269+0000 7f9232d9d700 1 --2- 172.21.15.2:0/3352699056 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f92340fc760 0x7f923410e8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:14:12.277 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:12.269+0000 7f9232d9d700 1 -- 172.21.15.2:0/3352699056 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f92340fb4c0 msgr2=0x7f923410e360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:14:12.278 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:12.269+0000 7f9232d9d700 1 --2- 172.21.15.2:0/3352699056 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f92340fb4c0 0x7f923410e360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:14:12.278 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:12.269+0000 7f9232d9d700 1 -- 172.21.15.2:0/3352699056 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9228005040 con 0x7f92340fd0f0 2022-05-13T23:14:12.278 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:12.269+0000 7f9232d9d700 1 --2- 172.21.15.2:0/3352699056 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f92340fd0f0 0x7f9234115930 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f922c015040 tx=0x7f922c004d10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-05-13T23:14:12.278 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:12.270+0000 7f9222ffd700 1 -- 172.21.15.2:0/3352699056 <== mon.1 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f922c00e030 con 0x7f92340fd0f0 2022-05-13T23:14:12.278 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:12.270+0000 7f923359e700 1 -- 172.21.15.2:0/3352699056 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f922c0097c0 con 0x7f92340fd0f0 2022-05-13T23:14:12.279 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:12.270+0000 7f9222ffd700 1 -- 172.21.15.2:0/3352699056 <== mon.1 v2:172.21.15.47:3300/0 2 ==== config(26 keys) v1 ==== 1087+0+0 (secure 0 0 0) 0x7f922c01c440 con 0x7f92340fd0f0 2022-05-13T23:14:12.279 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:12.270+0000 7f9222ffd700 1 -- 172.21.15.2:0/3352699056 <== mon.1 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f922c012680 con 0x7f92340fd0f0 2022-05-13T23:14:12.279 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:12.270+0000 7f923359e700 1 -- 172.21.15.2:0/3352699056 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9234116770 con 0x7f92340fd0f0 2022-05-13T23:14:12.279 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:12.271+0000 7f923359e700 1 -- 172.21.15.2:0/3352699056 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9234106d60 con 0x7f92340fd0f0 2022-05-13T23:14:12.280 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:12.273+0000 7f9222ffd700 1 -- 172.21.15.2:0/3352699056 <== mon.1 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 16) v1 ==== 125136+0+0 (secure 0 0 0) 0x7f922c02c070 con 0x7f92340fd0f0 2022-05-13T23:14:12.280 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:12.274+0000 7f9222ffd700 1 --2- 172.21.15.2:0/3352699056 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f921c096ce0 0x7f921c099190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:14:12.281 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:12.274+0000 7f9222ffd700 1 -- 172.21.15.2:0/3352699056 <== mon.1 v2:172.21.15.47:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 7123+0+0 (secure 0 0 0) 0x7f922c0c1070 con 0x7f92340fd0f0 2022-05-13T23:14:12.281 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:12.274+0000 7f923259c700 1 --2- 172.21.15.2:0/3352699056 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f921c096ce0 0x7f921c099190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:14:12.281 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:12.275+0000 7f923259c700 1 --2- 172.21.15.2:0/3352699056 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f921c096ce0 0x7f921c099190 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f92340fc5c0 tx=0x7f922400b040 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-05-13T23:14:12.282 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:12.276+0000 7f9222ffd700 1 -- 172.21.15.2:0/3352699056 <== mon.1 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+165215 (secure 0 0 0) 0x7f922c08e790 con 0x7f92340fd0f0 2022-05-13T23:14:12.472 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:12.471+0000 7f923359e700 1 -- 172.21.15.2:0/3352699056 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f9234106ef0 con 0x7f92340fd0f0 2022-05-13T23:14:12.473 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:12.471+0000 7f9222ffd700 1 -- 172.21.15.2:0/3352699056 <== mon.1 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v79) v1 ==== 93+0+10 (secure 0 0 0) 0x7f922c017020 con 0x7f92340fd0f0 2022-05-13T23:14:12.473 INFO:teuthology.orchestra.run.smithi002.stdout:pg_num: 1 2022-05-13T23:14:12.476 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:12.474+0000 7f923359e700 1 -- 172.21.15.2:0/3352699056 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f921c096ce0 msgr2=0x7f921c099190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:14:12.476 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:12.474+0000 7f923359e700 1 --2- 172.21.15.2:0/3352699056 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f921c096ce0 0x7f921c099190 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f92340fc5c0 tx=0x7f922400b040 comp rx=0 tx=0).stop 2022-05-13T23:14:12.476 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:12.474+0000 7f923359e700 1 -- 172.21.15.2:0/3352699056 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f92340fd0f0 msgr2=0x7f9234115930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:14:12.478 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:12.474+0000 7f923359e700 1 --2- 172.21.15.2:0/3352699056 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f92340fd0f0 0x7f9234115930 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f922c015040 tx=0x7f922c004d10 comp rx=0 tx=0).stop 2022-05-13T23:14:12.478 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:12.475+0000 7f923359e700 1 -- 172.21.15.2:0/3352699056 shutdown_connections 2022-05-13T23:14:12.478 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:12.475+0000 7f923359e700 1 --2- 172.21.15.2:0/3352699056 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f92340fd0f0 0x7f9234115930 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:14:12.478 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:12.475+0000 7f923359e700 1 --2- 172.21.15.2:0/3352699056 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f921c096ce0 0x7f921c099190 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:14:12.479 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:12.475+0000 7f923359e700 1 --2- 172.21.15.2:0/3352699056 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f92340fb4c0 0x7f923410e360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:14:12.479 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:12.475+0000 7f923359e700 1 --2- 172.21.15.2:0/3352699056 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f92340fc760 0x7f923410e8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:14:12.480 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:12.475+0000 7f923359e700 1 -- 172.21.15.2:0/3352699056 >> 172.21.15.2:0/3352699056 conn(0x7f92340f6cb0 msgr2=0x7f9234101f50 unknown :-1 s=STATE_NONE l=0).mark_down 2022-05-13T23:14:12.480 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:12.475+0000 7f923359e700 1 -- 172.21.15.2:0/3352699056 shutdown_connections 2022-05-13T23:14:12.480 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:12.475+0000 7f923359e700 1 -- 172.21.15.2:0/3352699056 wait complete. 2022-05-13T23:14:12.503 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2022-05-13T23:14:12.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:14:12 smithi115 ceph-mon[78593]: pgmap v282: 1 pgs: 1 active+clean; 577 KiB data, 74 MiB used, 1.0 TiB / 1.0 TiB avail 2022-05-13T23:14:12.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:14:12 smithi115 ceph-mon[78593]: from='client.? 172.21.15.2:0/2840076559' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2022-05-13T23:14:12.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:14:12 smithi115 ceph-mon[78593]: from='client.? 172.21.15.2:0/1347886734' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2022-05-13T23:14:12.726 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:12.724+0000 7f2b85d9d700 1 -- 172.21.15.2:0/362340860 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f2b800703d0 msgr2=0x7f2b8006e8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:14:12.727 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:12.724+0000 7f2b85d9d700 1 --2- 172.21.15.2:0/362340860 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f2b800703d0 0x7f2b8006e8d0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f2b7c004870 tx=0x7f2b7c009cb0 comp rx=0 tx=0).stop 2022-05-13T23:14:12.727 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:12.725+0000 7f2b85d9d700 1 -- 172.21.15.2:0/362340860 shutdown_connections 2022-05-13T23:14:12.727 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:12.725+0000 7f2b85d9d700 1 --2- 172.21.15.2:0/362340860 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2b80103410 0x7f2b801057f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:14:12.728 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:12.725+0000 7f2b85d9d700 1 --2- 172.21.15.2:0/362340860 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f2b800703d0 0x7f2b8006e8d0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:14:12.728 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:12.725+0000 7f2b85d9d700 1 --2- 172.21.15.2:0/362340860 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f2b8006ee10 0x7f2b8006f280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:14:12.728 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:12.725+0000 7f2b85d9d700 1 -- 172.21.15.2:0/362340860 >> 172.21.15.2:0/362340860 conn(0x7f2b800f6cb0 msgr2=0x7f2b800f9100 unknown :-1 s=STATE_NONE l=0).mark_down 2022-05-13T23:14:12.728 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:12.725+0000 7f2b85d9d700 1 -- 172.21.15.2:0/362340860 shutdown_connections 2022-05-13T23:14:12.728 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:12.725+0000 7f2b85d9d700 1 -- 172.21.15.2:0/362340860 wait complete. 2022-05-13T23:14:12.731 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:12.726+0000 7f2b85d9d700 1 Processor -- start 2022-05-13T23:14:12.732 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:12.726+0000 7f2b85d9d700 1 -- start start 2022-05-13T23:14:12.732 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:12.726+0000 7f2b85d9d700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f2b8006ee10 0x7f2b8010e560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:14:12.732 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:12.726+0000 7f2b85d9d700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f2b800703d0 0x7f2b8010eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:14:12.733 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:12.726+0000 7f2b85d9d700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2b80103410 0x7f2b80115b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:14:12.733 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:12.727+0000 7f2b85d9d700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b80116070 con 0x7f2b80103410 2022-05-13T23:14:12.733 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:12.727+0000 7f2b77fff700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f2b800703d0 0x7f2b8010eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:14:12.733 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:12.727+0000 7f2b85d9d700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b801161e0 con 0x7f2b8006ee10 2022-05-13T23:14:12.734 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:12.727+0000 7f2b77fff700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f2b800703d0 0x7f2b8010eaa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.2:43312/0 (socket says 172.21.15.2:43312) 2022-05-13T23:14:12.734 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:12.727+0000 7f2b77fff700 1 -- 172.21.15.2:0/1618160688 learned_addr learned my addr 172.21.15.2:0/1618160688 (peer_addr_for_me v2:172.21.15.2:0/0) 2022-05-13T23:14:12.734 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:12.727+0000 7f2b84d9b700 1 --2- 172.21.15.2:0/1618160688 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f2b8006ee10 0x7f2b8010e560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:14:12.734 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:12.727+0000 7f2b85d9d700 1 -- 172.21.15.2:0/1618160688 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b801163a0 con 0x7f2b800703d0 2022-05-13T23:14:12.735 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:12.727+0000 7f2b8559c700 1 --2- 172.21.15.2:0/1618160688 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2b80103410 0x7f2b80115b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:14:12.735 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:12.727+0000 7f2b84d9b700 1 -- 172.21.15.2:0/1618160688 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f2b800703d0 msgr2=0x7f2b8010eaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:14:12.735 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:12.728+0000 7f2b84d9b700 1 --2- 172.21.15.2:0/1618160688 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f2b800703d0 0x7f2b8010eaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:14:12.735 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:12.728+0000 7f2b84d9b700 1 -- 172.21.15.2:0/1618160688 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2b80103410 msgr2=0x7f2b80115b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:14:12.736 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:12.728+0000 7f2b84d9b700 1 --2- 172.21.15.2:0/1618160688 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2b80103410 0x7f2b80115b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:14:12.736 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:12.728+0000 7f2b84d9b700 1 -- 172.21.15.2:0/1618160688 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b7c005040 con 0x7f2b8006ee10 2022-05-13T23:14:12.736 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:12.728+0000 7f2b84d9b700 1 --2- 172.21.15.2:0/1618160688 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f2b8006ee10 0x7f2b8010e560 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f2b7c005010 tx=0x7f2b7c009cb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-05-13T23:14:12.736 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:12.728+0000 7f2b757fa700 1 -- 172.21.15.2:0/1618160688 <== mon.1 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b7c020070 con 0x7f2b8006ee10 2022-05-13T23:14:12.736 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:12.728+0000 7f2b757fa700 1 -- 172.21.15.2:0/1618160688 <== mon.1 v2:172.21.15.47:3300/0 2 ==== config(26 keys) v1 ==== 1087+0+0 (secure 0 0 0) 0x7f2b7c0096f0 con 0x7f2b8006ee10 2022-05-13T23:14:12.737 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:12.728+0000 7f2b757fa700 1 -- 172.21.15.2:0/1618160688 <== mon.1 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b7c00e040 con 0x7f2b8006ee10 2022-05-13T23:14:12.737 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:12.728+0000 7f2b85d9d700 1 -- 172.21.15.2:0/1618160688 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2b80116620 con 0x7f2b8006ee10 2022-05-13T23:14:12.737 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:12.729+0000 7f2b85d9d700 1 -- 172.21.15.2:0/1618160688 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2b80116aa0 con 0x7f2b8006ee10 2022-05-13T23:14:12.737 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:12.730+0000 7f2b85d9d700 1 -- 172.21.15.2:0/1618160688 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2b80108ee0 con 0x7f2b8006ee10 2022-05-13T23:14:12.739 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:12.731+0000 7f2b757fa700 1 -- 172.21.15.2:0/1618160688 <== mon.1 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 16) v1 ==== 125136+0+0 (secure 0 0 0) 0x7f2b7c009240 con 0x7f2b8006ee10 2022-05-13T23:14:12.739 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:12.733+0000 7f2b757fa700 1 --2- 172.21.15.2:0/1618160688 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f2b68096d60 0x7f2b68099210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:14:12.739 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:12.733+0000 7f2b757fa700 1 -- 172.21.15.2:0/1618160688 <== mon.1 v2:172.21.15.47:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 7123+0+0 (secure 0 0 0) 0x7f2b7c0bfb50 con 0x7f2b8006ee10 2022-05-13T23:14:12.739 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:12.733+0000 7f2b77fff700 1 --2- 172.21.15.2:0/1618160688 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f2b68096d60 0x7f2b68099210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:14:12.740 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:12.734+0000 7f2b77fff700 1 --2- 172.21.15.2:0/1618160688 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f2b68096d60 0x7f2b68099210 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f2b70004e80 tx=0x7f2b7000b040 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-05-13T23:14:12.740 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:12.736+0000 7f2b757fa700 1 -- 172.21.15.2:0/1618160688 <== mon.1 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+165215 (secure 0 0 0) 0x7f2b7c08d1e0 con 0x7f2b8006ee10 2022-05-13T23:14:12.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:14:12 smithi047 ceph-mon[77147]: pgmap v282: 1 pgs: 1 active+clean; 577 KiB data, 74 MiB used, 1.0 TiB / 1.0 TiB avail 2022-05-13T23:14:12.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:14:12 smithi047 ceph-mon[77147]: from='client.? 172.21.15.2:0/2840076559' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2022-05-13T23:14:12.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:14:12 smithi047 ceph-mon[77147]: from='client.? 172.21.15.2:0/1347886734' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2022-05-13T23:14:12.822 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:14:12 smithi002 ceph-mon[71816]: pgmap v282: 1 pgs: 1 active+clean; 577 KiB data, 74 MiB used, 1.0 TiB / 1.0 TiB avail 2022-05-13T23:14:12.823 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:14:12 smithi002 ceph-mon[71816]: from='client.? 172.21.15.2:0/2840076559' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2022-05-13T23:14:12.823 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:14:12 smithi002 ceph-mon[71816]: from='client.? 172.21.15.2:0/1347886734' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2022-05-13T23:14:12.938 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:12.937+0000 7f2b85d9d700 1 -- 172.21.15.2:0/1618160688 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f2b8002a6c0 con 0x7f2b8006ee10 2022-05-13T23:14:12.939 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:12.938+0000 7f2b757fa700 1 -- 172.21.15.2:0/1618160688 <== mon.1 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v79) v1 ==== 84+0+65 (secure 0 0 0) 0x7f2b7c0bde60 con 0x7f2b8006ee10 2022-05-13T23:14:12.939 INFO:teuthology.orchestra.run.smithi002.stdout: 2022-05-13T23:14:12.940 INFO:teuthology.orchestra.run.smithi002.stdout:[ 2022-05-13T23:14:12.940 INFO:teuthology.orchestra.run.smithi002.stdout: { 2022-05-13T23:14:12.940 INFO:teuthology.orchestra.run.smithi002.stdout: "poolnum": 1, 2022-05-13T23:14:12.940 INFO:teuthology.orchestra.run.smithi002.stdout: "poolname": ".mgr" 2022-05-13T23:14:12.941 INFO:teuthology.orchestra.run.smithi002.stdout: } 2022-05-13T23:14:12.941 INFO:teuthology.orchestra.run.smithi002.stdout:] 2022-05-13T23:14:12.942 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:12.940+0000 7f2b85d9d700 1 -- 172.21.15.2:0/1618160688 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f2b68096d60 msgr2=0x7f2b68099210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:14:12.942 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:12.940+0000 7f2b85d9d700 1 --2- 172.21.15.2:0/1618160688 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f2b68096d60 0x7f2b68099210 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f2b70004e80 tx=0x7f2b7000b040 comp rx=0 tx=0).stop 2022-05-13T23:14:12.942 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:12.940+0000 7f2b85d9d700 1 -- 172.21.15.2:0/1618160688 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f2b8006ee10 msgr2=0x7f2b8010e560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:14:12.943 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:12.940+0000 7f2b85d9d700 1 --2- 172.21.15.2:0/1618160688 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f2b8006ee10 0x7f2b8010e560 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f2b7c005010 tx=0x7f2b7c009cb0 comp rx=0 tx=0).stop 2022-05-13T23:14:12.943 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:12.940+0000 7f2b85d9d700 1 -- 172.21.15.2:0/1618160688 shutdown_connections 2022-05-13T23:14:12.943 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:12.940+0000 7f2b85d9d700 1 --2- 172.21.15.2:0/1618160688 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f2b68096d60 0x7f2b68099210 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:14:12.943 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:12.940+0000 7f2b85d9d700 1 --2- 172.21.15.2:0/1618160688 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2b80103410 0x7f2b80115b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:14:12.944 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:12.940+0000 7f2b85d9d700 1 --2- 172.21.15.2:0/1618160688 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f2b8006ee10 0x7f2b8010e560 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:14:12.944 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:12.940+0000 7f2b85d9d700 1 --2- 172.21.15.2:0/1618160688 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f2b800703d0 0x7f2b8010eaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:14:12.944 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:12.940+0000 7f2b85d9d700 1 -- 172.21.15.2:0/1618160688 >> 172.21.15.2:0/1618160688 conn(0x7f2b800f6cb0 msgr2=0x7f2b800f9090 unknown :-1 s=STATE_NONE l=0).mark_down 2022-05-13T23:14:12.944 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:12.940+0000 7f2b85d9d700 1 -- 172.21.15.2:0/1618160688 shutdown_connections 2022-05-13T23:14:12.945 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:12.940+0000 7f2b85d9d700 1 -- 172.21.15.2:0/1618160688 wait complete. 2022-05-13T23:14:12.969 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2022-05-13T23:14:12.970 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2022-05-13T23:14:13.211 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:13.208+0000 7f87f999d700 1 -- 172.21.15.2:0/2900416099 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f87f40703d0 msgr2=0x7f87f406e8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:14:13.211 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:13.208+0000 7f87f999d700 1 --2- 172.21.15.2:0/2900416099 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f87f40703d0 0x7f87f406e8d0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f87e8004870 tx=0x7f87e8009cb0 comp rx=0 tx=0).stop 2022-05-13T23:14:13.212 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:13.208+0000 7f87f999d700 1 -- 172.21.15.2:0/2900416099 shutdown_connections 2022-05-13T23:14:13.212 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:13.208+0000 7f87f999d700 1 --2- 172.21.15.2:0/2900416099 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f87f41033d0 0x7f87f41057b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:14:13.212 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:13.208+0000 7f87f999d700 1 --2- 172.21.15.2:0/2900416099 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f87f40703d0 0x7f87f406e8d0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:14:13.213 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:13.208+0000 7f87f999d700 1 --2- 172.21.15.2:0/2900416099 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f87f406ee10 0x7f87f406f280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:14:13.213 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:13.209+0000 7f87f999d700 1 -- 172.21.15.2:0/2900416099 >> 172.21.15.2:0/2900416099 conn(0x7f87f40f6c50 msgr2=0x7f87f40f90c0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-05-13T23:14:13.213 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:13.209+0000 7f87f999d700 1 -- 172.21.15.2:0/2900416099 shutdown_connections 2022-05-13T23:14:13.214 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:13.209+0000 7f87f999d700 1 -- 172.21.15.2:0/2900416099 wait complete. 2022-05-13T23:14:13.219 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:13.210+0000 7f87f999d700 1 Processor -- start 2022-05-13T23:14:13.219 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:13.210+0000 7f87f999d700 1 -- start start 2022-05-13T23:14:13.220 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:13.211+0000 7f87f999d700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f87f406ee10 0x7f87f410e550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:14:13.220 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:13.211+0000 7f87f999d700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f87f40703d0 0x7f87f410ea90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:14:13.220 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:13.211+0000 7f87f999d700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f87f41033d0 0x7f87f4115b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:14:13.221 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:13.211+0000 7f87f999d700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87f4116060 con 0x7f87f41033d0 2022-05-13T23:14:13.221 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:13.211+0000 7f87f999d700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87f41161d0 con 0x7f87f40703d0 2022-05-13T23:14:13.222 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:13.211+0000 7f87f999d700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87f4116390 con 0x7f87f406ee10 2022-05-13T23:14:13.222 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:13.211+0000 7f87f899b700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f87f406ee10 0x7f87f410e550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:14:13.222 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:13.211+0000 7f87f899b700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f87f406ee10 0x7f87f410e550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.2:43336/0 (socket says 172.21.15.2:43336) 2022-05-13T23:14:13.223 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:13.211+0000 7f87f899b700 1 -- 172.21.15.2:0/2746506511 learned_addr learned my addr 172.21.15.2:0/2746506511 (peer_addr_for_me v2:172.21.15.2:0/0) 2022-05-13T23:14:13.223 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:13.211+0000 7f87f3fff700 1 --2- 172.21.15.2:0/2746506511 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f87f40703d0 0x7f87f410ea90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:14:13.223 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:13.211+0000 7f87f919c700 1 --2- 172.21.15.2:0/2746506511 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f87f41033d0 0x7f87f4115b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:14:13.223 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:13.211+0000 7f87f899b700 1 -- 172.21.15.2:0/2746506511 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f87f40703d0 msgr2=0x7f87f410ea90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:14:13.223 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:13.211+0000 7f87f899b700 1 --2- 172.21.15.2:0/2746506511 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f87f40703d0 0x7f87f410ea90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:14:13.224 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:13.211+0000 7f87f899b700 1 -- 172.21.15.2:0/2746506511 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f87f41033d0 msgr2=0x7f87f4115b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:14:13.224 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:13.211+0000 7f87f899b700 1 --2- 172.21.15.2:0/2746506511 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f87f41033d0 0x7f87f4115b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:14:13.224 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:13.212+0000 7f87f899b700 1 -- 172.21.15.2:0/2746506511 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87ec0097c0 con 0x7f87f406ee10 2022-05-13T23:14:13.224 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:13.212+0000 7f87f919c700 1 --2- 172.21.15.2:0/2746506511 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f87f41033d0 0x7f87f4115b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2022-05-13T23:14:13.225 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:13.212+0000 7f87f899b700 1 --2- 172.21.15.2:0/2746506511 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f87f406ee10 0x7f87f410e550 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f87e8004ee0 tx=0x7f87e8009cb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-05-13T23:14:13.225 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:13.212+0000 7f87f17fa700 1 -- 172.21.15.2:0/2746506511 <== mon.2 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87e800e040 con 0x7f87f406ee10 2022-05-13T23:14:13.225 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:13.212+0000 7f87f17fa700 1 -- 172.21.15.2:0/2746506511 <== mon.2 v2:172.21.15.115:3300/0 2 ==== config(26 keys) v1 ==== 1087+0+0 (secure 0 0 0) 0x7f87e8014e40 con 0x7f87f406ee10 2022-05-13T23:14:13.226 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:13.212+0000 7f87f999d700 1 -- 172.21.15.2:0/2746506511 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f87e8005040 con 0x7f87f406ee10 2022-05-13T23:14:13.226 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:13.212+0000 7f87f17fa700 1 -- 172.21.15.2:0/2746506511 <== mon.2 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87e801cab0 con 0x7f87f406ee10 2022-05-13T23:14:13.226 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:13.213+0000 7f87f999d700 1 -- 172.21.15.2:0/2746506511 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f87f4116900 con 0x7f87f406ee10 2022-05-13T23:14:13.226 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:13.214+0000 7f87f999d700 1 -- 172.21.15.2:0/2746506511 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f87f4108ed0 con 0x7f87f406ee10 2022-05-13T23:14:13.227 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:13.215+0000 7f87f17fa700 1 -- 172.21.15.2:0/2746506511 <== mon.2 v2:172.21.15.115:3300/0 4 ==== mgrmap(e 16) v1 ==== 125136+0+0 (secure 0 0 0) 0x7f87e8009240 con 0x7f87f406ee10 2022-05-13T23:14:13.227 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:13.216+0000 7f87f17fa700 1 --2- 172.21.15.2:0/2746506511 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f87e4096ce0 0x7f87e4099190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:14:13.227 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:13.217+0000 7f87f17fa700 1 -- 172.21.15.2:0/2746506511 <== mon.2 v2:172.21.15.115:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 7123+0+0 (secure 0 0 0) 0x7f87e80c0060 con 0x7f87f406ee10 2022-05-13T23:14:13.228 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:13.217+0000 7f87f3fff700 1 --2- 172.21.15.2:0/2746506511 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f87e4096ce0 0x7f87e4099190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:14:13.229 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:13.218+0000 7f87f3fff700 1 --2- 172.21.15.2:0/2746506511 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f87e4096ce0 0x7f87e4099190 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f87e00048b0 tx=0x7f87e000b040 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-05-13T23:14:13.230 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:13.220+0000 7f87f17fa700 1 -- 172.21.15.2:0/2746506511 <== mon.2 v2:172.21.15.115:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+165215 (secure 0 0 0) 0x7f87e808d7e0 con 0x7f87f406ee10 2022-05-13T23:14:13.440 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:13.438+0000 7f87f999d700 1 -- 172.21.15.2:0/2746506511 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f87f4109060 con 0x7f87f406ee10 2022-05-13T23:14:13.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:14:13 smithi115 ceph-mon[78593]: from='client.? 172.21.15.2:0/3352699056' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2022-05-13T23:14:13.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:14:13 smithi115 ceph-mon[78593]: from='client.? 172.21.15.2:0/1618160688' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2022-05-13T23:14:13.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:14:13 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:14:13.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:14:13 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:14:13.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:14:13 smithi047 ceph-mon[77147]: from='client.? 172.21.15.2:0/3352699056' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2022-05-13T23:14:13.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:14:13 smithi047 ceph-mon[77147]: from='client.? 172.21.15.2:0/1618160688' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2022-05-13T23:14:13.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:14:13 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:14:13.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:14:13 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:14:13.822 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:14:13 smithi002 ceph-mon[71816]: from='client.? 172.21.15.2:0/3352699056' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2022-05-13T23:14:13.823 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:14:13 smithi002 ceph-mon[71816]: from='client.? 172.21.15.2:0/1618160688' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2022-05-13T23:14:13.823 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:14:13 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:14:13.823 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:14:13 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:14:14.362 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:14.355+0000 7f87f17fa700 1 -- 172.21.15.2:0/2746506511 <== mon.2 v2:172.21.15.115:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v80) v1 ==== 138+0+0 (secure 0 0 0) 0x7f87f4109060 con 0x7f87f406ee10 2022-05-13T23:14:14.364 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:14.361+0000 7f87f999d700 1 -- 172.21.15.2:0/2746506511 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f87e4096ce0 msgr2=0x7f87e4099190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:14:14.365 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:14.361+0000 7f87f999d700 1 --2- 172.21.15.2:0/2746506511 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f87e4096ce0 0x7f87e4099190 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f87e00048b0 tx=0x7f87e000b040 comp rx=0 tx=0).stop 2022-05-13T23:14:14.365 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:14.361+0000 7f87f999d700 1 -- 172.21.15.2:0/2746506511 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f87f406ee10 msgr2=0x7f87f410e550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:14:14.365 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:14.361+0000 7f87f999d700 1 --2- 172.21.15.2:0/2746506511 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f87f406ee10 0x7f87f410e550 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f87e8004ee0 tx=0x7f87e8009cb0 comp rx=0 tx=0).stop 2022-05-13T23:14:14.365 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:14.361+0000 7f87f999d700 1 -- 172.21.15.2:0/2746506511 shutdown_connections 2022-05-13T23:14:14.366 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:14.361+0000 7f87f999d700 1 --2- 172.21.15.2:0/2746506511 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f87e4096ce0 0x7f87e4099190 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:14:14.366 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:14.361+0000 7f87f999d700 1 --2- 172.21.15.2:0/2746506511 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f87f41033d0 0x7f87f4115b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:14:14.366 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:14.361+0000 7f87f999d700 1 --2- 172.21.15.2:0/2746506511 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f87f406ee10 0x7f87f410e550 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:14:14.366 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:14.361+0000 7f87f999d700 1 --2- 172.21.15.2:0/2746506511 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f87f40703d0 0x7f87f410ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:14:14.367 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:14.361+0000 7f87f999d700 1 -- 172.21.15.2:0/2746506511 >> 172.21.15.2:0/2746506511 conn(0x7f87f40f6c50 msgr2=0x7f87f40f9030 unknown :-1 s=STATE_NONE l=0).mark_down 2022-05-13T23:14:14.367 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:14.361+0000 7f87f999d700 1 -- 172.21.15.2:0/2746506511 shutdown_connections 2022-05-13T23:14:14.367 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:14.362+0000 7f87f999d700 1 -- 172.21.15.2:0/2746506511 wait complete. 2022-05-13T23:14:14.367 INFO:teuthology.orchestra.run.smithi002.stderr:pool 'cephfs_metadata' created 2022-05-13T23:14:14.392 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2022-05-13T23:14:14.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:14:14 smithi115 ceph-mon[78593]: pgmap v283: 1 pgs: 1 active+clean; 577 KiB data, 74 MiB used, 1.0 TiB / 1.0 TiB avail 2022-05-13T23:14:14.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:14:14 smithi115 ceph-mon[78593]: from='client.? 172.21.15.2:0/2746506511' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2022-05-13T23:14:14.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:14:14 smithi115 ceph-mon[78593]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2022-05-13T23:14:14.638 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:14.635+0000 7f1de719d700 1 -- 172.21.15.2:0/2108270617 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f1de00fb4b0 msgr2=0x7f1de00fb8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:14:14.638 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:14.635+0000 7f1de719d700 1 --2- 172.21.15.2:0/2108270617 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f1de00fb4b0 0x7f1de00fb8c0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f1ddc004870 tx=0x7f1ddc009cb0 comp rx=0 tx=0).stop 2022-05-13T23:14:14.639 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:14.636+0000 7f1de719d700 1 -- 172.21.15.2:0/2108270617 shutdown_connections 2022-05-13T23:14:14.639 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:14.636+0000 7f1de719d700 1 --2- 172.21.15.2:0/2108270617 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1de00fd0e0 0x7f1de01039a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:14:14.639 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:14.636+0000 7f1de719d700 1 --2- 172.21.15.2:0/2108270617 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f1de00fb4b0 0x7f1de00fb8c0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:14:14.639 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:14.636+0000 7f1de719d700 1 --2- 172.21.15.2:0/2108270617 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1de00fc750 0x7f1de00fcba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:14:14.639 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:14.636+0000 7f1de719d700 1 -- 172.21.15.2:0/2108270617 >> 172.21.15.2:0/2108270617 conn(0x7f1de00f6c60 msgr2=0x7f1de00f90d0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-05-13T23:14:14.640 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:14.636+0000 7f1de719d700 1 -- 172.21.15.2:0/2108270617 shutdown_connections 2022-05-13T23:14:14.640 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:14.636+0000 7f1de719d700 1 -- 172.21.15.2:0/2108270617 wait complete. 2022-05-13T23:14:14.640 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:14.637+0000 7f1de719d700 1 Processor -- start 2022-05-13T23:14:14.640 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:14.637+0000 7f1de719d700 1 -- start start 2022-05-13T23:14:14.643 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:14.637+0000 7f1de719d700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1de00fb4b0 0x7f1de010c2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:14:14.643 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:14.637+0000 7f1de719d700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1de00fc750 0x7f1de010c800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:14:14.643 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:14.637+0000 7f1de719d700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f1de00fd0e0 0x7f1de0113890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:14:14.644 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:14.637+0000 7f1de719d700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1de0113dd0 con 0x7f1de00fb4b0 2022-05-13T23:14:14.644 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:14.637+0000 7f1de719d700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1de0113f40 con 0x7f1de00fc750 2022-05-13T23:14:14.644 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:14.637+0000 7f1de719d700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1de0114100 con 0x7f1de00fd0e0 2022-05-13T23:14:14.644 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:14.638+0000 7f1de619b700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1de00fb4b0 0x7f1de010c2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:14:14.645 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:14.638+0000 7f1de599a700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1de00fc750 0x7f1de010c800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:14:14.645 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:14.638+0000 7f1de599a700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1de00fc750 0x7f1de010c800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.2:57966/0 (socket says 172.21.15.2:57966) 2022-05-13T23:14:14.645 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:14.638+0000 7f1de599a700 1 -- 172.21.15.2:0/545744284 learned_addr learned my addr 172.21.15.2:0/545744284 (peer_addr_for_me v2:172.21.15.2:0/0) 2022-05-13T23:14:14.645 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:14.638+0000 7f1de699c700 1 --2- 172.21.15.2:0/545744284 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f1de00fd0e0 0x7f1de0113890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:14:14.645 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:14.638+0000 7f1de599a700 1 -- 172.21.15.2:0/545744284 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f1de00fd0e0 msgr2=0x7f1de0113890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:14:14.646 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:14.638+0000 7f1de599a700 1 --2- 172.21.15.2:0/545744284 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f1de00fd0e0 0x7f1de0113890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:14:14.646 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:14.638+0000 7f1de599a700 1 -- 172.21.15.2:0/545744284 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1de00fb4b0 msgr2=0x7f1de010c2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:14:14.646 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:14.638+0000 7f1de599a700 1 --2- 172.21.15.2:0/545744284 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1de00fb4b0 0x7f1de010c2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:14:14.646 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:14.638+0000 7f1de599a700 1 -- 172.21.15.2:0/545744284 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ddc005040 con 0x7f1de00fc750 2022-05-13T23:14:14.647 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:14.638+0000 7f1de619b700 1 --2- 172.21.15.2:0/545744284 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1de00fb4b0 0x7f1de010c2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-05-13T23:14:14.647 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:14.638+0000 7f1de699c700 1 --2- 172.21.15.2:0/545744284 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f1de00fd0e0 0x7f1de0113890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-05-13T23:14:14.647 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:14.639+0000 7f1de599a700 1 --2- 172.21.15.2:0/545744284 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1de00fc750 0x7f1de010c800 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f1dd000e970 tx=0x7f1dd000b040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-05-13T23:14:14.647 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:14.639+0000 7f1dd6ffd700 1 -- 172.21.15.2:0/545744284 <== mon.1 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1dd0007b40 con 0x7f1de00fc750 2022-05-13T23:14:14.648 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:14.639+0000 7f1dd6ffd700 1 -- 172.21.15.2:0/545744284 <== mon.1 v2:172.21.15.47:3300/0 2 ==== config(26 keys) v1 ==== 1087+0+0 (secure 0 0 0) 0x7f1dd000f040 con 0x7f1de00fc750 2022-05-13T23:14:14.648 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:14.639+0000 7f1de719d700 1 -- 172.21.15.2:0/545744284 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1de006e770 con 0x7f1de00fc750 2022-05-13T23:14:14.648 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:14.639+0000 7f1dd6ffd700 1 -- 172.21.15.2:0/545744284 <== mon.1 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1dd0014950 con 0x7f1de00fc750 2022-05-13T23:14:14.648 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:14.639+0000 7f1de719d700 1 -- 172.21.15.2:0/545744284 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1de006ece0 con 0x7f1de00fc750 2022-05-13T23:14:14.648 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:14.640+0000 7f1de719d700 1 -- 172.21.15.2:0/545744284 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1de0106cf0 con 0x7f1de00fc750 2022-05-13T23:14:14.649 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:14:14 smithi047 ceph-mon[77147]: pgmap v283: 1 pgs: 1 active+clean; 577 KiB data, 74 MiB used, 1.0 TiB / 1.0 TiB avail 2022-05-13T23:14:14.650 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:14:14 smithi047 ceph-mon[77147]: from='client.? 172.21.15.2:0/2746506511' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2022-05-13T23:14:14.650 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:14:14 smithi047 ceph-mon[77147]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2022-05-13T23:14:14.651 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:14.642+0000 7f1dd6ffd700 1 -- 172.21.15.2:0/545744284 <== mon.1 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 16) v1 ==== 125136+0+0 (secure 0 0 0) 0x7f1dd001b030 con 0x7f1de00fc750 2022-05-13T23:14:14.651 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:14.643+0000 7f1dd6ffd700 1 --2- 172.21.15.2:0/545744284 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f1dcc096ce0 0x7f1dcc099190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:14:14.651 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:14.643+0000 7f1dd6ffd700 1 -- 172.21.15.2:0/545744284 <== mon.1 v2:172.21.15.47:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 7493+0+0 (secure 0 0 0) 0x7f1dd00be8e0 con 0x7f1de00fc750 2022-05-13T23:14:14.651 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:14.644+0000 7f1de619b700 1 --2- 172.21.15.2:0/545744284 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f1dcc096ce0 0x7f1dcc099190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:14:14.652 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:14.644+0000 7f1de619b700 1 --2- 172.21.15.2:0/545744284 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f1dcc096ce0 0x7f1dcc099190 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f1ddc009da0 tx=0x7f1ddc01d040 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-05-13T23:14:14.652 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:14.647+0000 7f1dd6ffd700 1 -- 172.21.15.2:0/545744284 <== mon.1 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+165215 (secure 0 0 0) 0x7f1dd004b570 con 0x7f1de00fc750 2022-05-13T23:14:14.679 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:14:14 smithi002 ceph-mon[71816]: pgmap v283: 1 pgs: 1 active+clean; 577 KiB data, 74 MiB used, 1.0 TiB / 1.0 TiB avail 2022-05-13T23:14:14.680 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:14:14 smithi002 ceph-mon[71816]: from='client.? 172.21.15.2:0/2746506511' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2022-05-13T23:14:14.680 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:14:14 smithi002 ceph-mon[71816]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2022-05-13T23:14:14.870 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:14.868+0000 7f1de719d700 1 -- 172.21.15.2:0/545744284 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f1de002a4c0 con 0x7f1de00fc750 2022-05-13T23:14:15.671 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:15.669+0000 7f1dd6ffd700 1 -- 172.21.15.2:0/545744284 <== mon.1 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v81) v1 ==== 170+0+0 (secure 0 0 0) 0x7f1dd008be00 con 0x7f1de00fc750 2022-05-13T23:14:15.680 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:15.679+0000 7f1de719d700 1 -- 172.21.15.2:0/545744284 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f1dcc096ce0 msgr2=0x7f1dcc099190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:14:15.681 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:15.679+0000 7f1de719d700 1 --2- 172.21.15.2:0/545744284 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f1dcc096ce0 0x7f1dcc099190 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f1ddc009da0 tx=0x7f1ddc01d040 comp rx=0 tx=0).stop 2022-05-13T23:14:15.681 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:15.679+0000 7f1de719d700 1 -- 172.21.15.2:0/545744284 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1de00fc750 msgr2=0x7f1de010c800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:14:15.681 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:15.679+0000 7f1de719d700 1 --2- 172.21.15.2:0/545744284 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1de00fc750 0x7f1de010c800 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f1dd000e970 tx=0x7f1dd000b040 comp rx=0 tx=0).stop 2022-05-13T23:14:15.683 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:15.681+0000 7f1de719d700 1 -- 172.21.15.2:0/545744284 shutdown_connections 2022-05-13T23:14:15.683 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:15.681+0000 7f1de719d700 1 --2- 172.21.15.2:0/545744284 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f1de00fd0e0 0x7f1de0113890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:14:15.684 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:15.681+0000 7f1de719d700 1 --2- 172.21.15.2:0/545744284 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f1dcc096ce0 0x7f1dcc099190 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:14:15.684 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:15.681+0000 7f1de719d700 1 --2- 172.21.15.2:0/545744284 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1de00fb4b0 0x7f1de010c2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:14:15.684 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:15.681+0000 7f1de719d700 1 --2- 172.21.15.2:0/545744284 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1de00fc750 0x7f1de010c800 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:14:15.684 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:15.681+0000 7f1de719d700 1 -- 172.21.15.2:0/545744284 >> 172.21.15.2:0/545744284 conn(0x7f1de00f6c60 msgr2=0x7f1de00ff980 unknown :-1 s=STATE_NONE l=0).mark_down 2022-05-13T23:14:15.685 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:15.682+0000 7f1de719d700 1 -- 172.21.15.2:0/545744284 shutdown_connections 2022-05-13T23:14:15.685 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:15.683+0000 7f1de719d700 1 -- 172.21.15.2:0/545744284 wait complete. 2022-05-13T23:14:15.689 INFO:teuthology.orchestra.run.smithi002.stderr:pool 'cephfs_data' created 2022-05-13T23:14:15.729 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2022-05-13T23:14:15.964 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:15.962+0000 7fe04119d700 1 -- 172.21.15.2:0/822518702 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe03c0fc730 msgr2=0x7fe03c0fcb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:14:15.965 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:15.962+0000 7fe04119d700 1 --2- 172.21.15.2:0/822518702 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe03c0fc730 0x7fe03c0fcb80 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fe030004870 tx=0x7fe030009cb0 comp rx=0 tx=0).stop 2022-05-13T23:14:15.965 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:15.962+0000 7fe04119d700 1 -- 172.21.15.2:0/822518702 shutdown_connections 2022-05-13T23:14:15.965 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:15.962+0000 7fe04119d700 1 --2- 172.21.15.2:0/822518702 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe03c0fd0c0 0x7fe03c103980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:14:15.966 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:15.962+0000 7fe04119d700 1 --2- 172.21.15.2:0/822518702 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe03c0fb490 0x7fe03c0fb8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:14:15.966 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:15.962+0000 7fe04119d700 1 --2- 172.21.15.2:0/822518702 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe03c0fc730 0x7fe03c0fcb80 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:14:15.966 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:15.962+0000 7fe04119d700 1 -- 172.21.15.2:0/822518702 >> 172.21.15.2:0/822518702 conn(0x7fe03c0f6c80 msgr2=0x7fe03c0f90b0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-05-13T23:14:15.966 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:15.962+0000 7fe04119d700 1 -- 172.21.15.2:0/822518702 shutdown_connections 2022-05-13T23:14:15.966 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:15.963+0000 7fe04119d700 1 -- 172.21.15.2:0/822518702 wait complete. 2022-05-13T23:14:15.967 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:15.963+0000 7fe04119d700 1 Processor -- start 2022-05-13T23:14:15.967 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:15.963+0000 7fe04119d700 1 -- start start 2022-05-13T23:14:15.969 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:15.964+0000 7fe04119d700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe03c0fb490 0x7fe03c0701d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:14:15.969 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:15.964+0000 7fe04119d700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe03c0fc730 0x7fe03c06e830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:14:15.970 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:15.964+0000 7fe04119d700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe03c0fd0c0 0x7fe03c06ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:14:15.970 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:15.964+0000 7fe04119d700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe03c06f480 con 0x7fe03c0fc730 2022-05-13T23:14:15.970 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:15.964+0000 7fe04119d700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe03c06f5f0 con 0x7fe03c0fd0c0 2022-05-13T23:14:15.970 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:15.964+0000 7fe04119d700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe03c1163f0 con 0x7fe03c0fb490 2022-05-13T23:14:15.970 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:15.964+0000 7fe03bfff700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe03c0fb490 0x7fe03c0701d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:14:15.971 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:15.964+0000 7fe03bfff700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe03c0fb490 0x7fe03c0701d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.2:43386/0 (socket says 172.21.15.2:43386) 2022-05-13T23:14:15.971 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:15.964+0000 7fe03bfff700 1 -- 172.21.15.2:0/649653726 learned_addr learned my addr 172.21.15.2:0/649653726 (peer_addr_for_me v2:172.21.15.2:0/0) 2022-05-13T23:14:15.971 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:15.964+0000 7fe03b7fe700 1 --2- 172.21.15.2:0/649653726 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe03c0fc730 0x7fe03c06e830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:14:15.971 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:15.964+0000 7fe04099c700 1 --2- 172.21.15.2:0/649653726 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe03c0fd0c0 0x7fe03c06ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:14:15.972 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:15.964+0000 7fe03bfff700 1 -- 172.21.15.2:0/649653726 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe03c0fd0c0 msgr2=0x7fe03c06ed70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:14:15.972 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:15.964+0000 7fe03bfff700 1 --2- 172.21.15.2:0/649653726 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe03c0fd0c0 0x7fe03c06ed70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:14:15.972 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:15.964+0000 7fe03bfff700 1 -- 172.21.15.2:0/649653726 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe03c0fc730 msgr2=0x7fe03c06e830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:14:15.972 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:15.964+0000 7fe03bfff700 1 --2- 172.21.15.2:0/649653726 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe03c0fc730 0x7fe03c06e830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:14:15.973 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:15.965+0000 7fe03bfff700 1 -- 172.21.15.2:0/649653726 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe030005040 con 0x7fe03c0fb490 2022-05-13T23:14:15.973 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:15.965+0000 7fe03b7fe700 1 --2- 172.21.15.2:0/649653726 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe03c0fc730 0x7fe03c06e830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-05-13T23:14:15.973 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:15.965+0000 7fe03bfff700 1 --2- 172.21.15.2:0/649653726 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe03c0fb490 0x7fe03c0701d0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fe02c00e940 tx=0x7fe02c00b040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-05-13T23:14:15.973 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:15.965+0000 7fe038ff9700 1 -- 172.21.15.2:0/649653726 <== mon.2 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe02c007b40 con 0x7fe03c0fb490 2022-05-13T23:14:15.973 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:15.965+0000 7fe038ff9700 1 -- 172.21.15.2:0/649653726 <== mon.2 v2:172.21.15.115:3300/0 2 ==== config(26 keys) v1 ==== 1087+0+0 (secure 0 0 0) 0x7fe02c00f040 con 0x7fe03c0fb490 2022-05-13T23:14:15.974 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:15.965+0000 7fe04119d700 1 -- 172.21.15.2:0/649653726 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe03c116590 con 0x7fe03c0fb490 2022-05-13T23:14:15.974 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:15.965+0000 7fe038ff9700 1 -- 172.21.15.2:0/649653726 <== mon.2 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe02c014890 con 0x7fe03c0fb490 2022-05-13T23:14:15.974 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:15.965+0000 7fe04119d700 1 -- 172.21.15.2:0/649653726 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe03c116aa0 con 0x7fe03c0fb490 2022-05-13T23:14:15.974 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:15.966+0000 7fe04119d700 1 -- 172.21.15.2:0/649653726 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe03c106cd0 con 0x7fe03c0fb490 2022-05-13T23:14:15.975 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:15.968+0000 7fe038ff9700 1 -- 172.21.15.2:0/649653726 <== mon.2 v2:172.21.15.115:3300/0 4 ==== mgrmap(e 16) v1 ==== 125136+0+0 (secure 0 0 0) 0x7fe02c01b030 con 0x7fe03c0fb490 2022-05-13T23:14:15.975 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:15.969+0000 7fe038ff9700 1 --2- 172.21.15.2:0/649653726 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7fe024096ce0 0x7fe024099190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:14:15.976 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:15.969+0000 7fe038ff9700 1 -- 172.21.15.2:0/649653726 <== mon.2 v2:172.21.15.115:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7875+0+0 (secure 0 0 0) 0x7fe02c0be9c0 con 0x7fe03c0fb490 2022-05-13T23:14:15.976 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:15.970+0000 7fe03b7fe700 1 --2- 172.21.15.2:0/649653726 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7fe024096ce0 0x7fe024099190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:14:15.976 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:15.970+0000 7fe03b7fe700 1 --2- 172.21.15.2:0/649653726 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7fe024096ce0 0x7fe024099190 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fe030004840 tx=0x7fe030009cb0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-05-13T23:14:15.976 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:15.974+0000 7fe038ff9700 1 -- 172.21.15.2:0/649653726 <== mon.2 v2:172.21.15.115:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+165215 (secure 0 0 0) 0x7fe02c04b570 con 0x7fe03c0fb490 2022-05-13T23:14:16.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:14:15 smithi047 ceph-mon[77147]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2022-05-13T23:14:16.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:14:15 smithi047 ceph-mon[77147]: osdmap e80: 12 total, 12 up, 12 in 2022-05-13T23:14:16.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:14:15 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:14:16.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:14:15 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:14:16.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:14:15 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-13T23:14:16.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:14:15 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-13T23:14:16.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:14:15 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-13T23:14:16.072 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:14:15 smithi002 ceph-mon[71816]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2022-05-13T23:14:16.073 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:14:15 smithi002 ceph-mon[71816]: osdmap e80: 12 total, 12 up, 12 in 2022-05-13T23:14:16.073 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:14:15 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:14:16.073 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:14:15 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:14:16.073 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:14:15 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-13T23:14:16.074 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:14:15 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-13T23:14:16.074 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:14:15 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-13T23:14:16.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:14:15 smithi115 ceph-mon[78593]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2022-05-13T23:14:16.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:14:15 smithi115 ceph-mon[78593]: osdmap e80: 12 total, 12 up, 12 in 2022-05-13T23:14:16.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:14:15 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:14:16.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:14:15 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:14:16.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:14:15 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-13T23:14:16.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:14:15 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-13T23:14:16.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:14:15 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-13T23:14:16.235 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:16.233+0000 7fe04119d700 1 -- 172.21.15.2:0/649653726 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7fe03c102660 con 0x7fe03c0fb490 2022-05-13T23:14:16.246 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:16.244+0000 7fe038ff9700 1 -- 172.21.15.2:0/649653726 <== mon.2 v2:172.21.15.115:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 2 and data pool 3 v14) v1 ==== 172+0+0 (secure 0 0 0) 0x7fe02c08f880 con 0x7fe03c0fb490 2022-05-13T23:14:16.261 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:16.260+0000 7fe04119d700 1 -- 172.21.15.2:0/649653726 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7fe024096ce0 msgr2=0x7fe024099190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:14:16.262 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:16.260+0000 7fe04119d700 1 --2- 172.21.15.2:0/649653726 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7fe024096ce0 0x7fe024099190 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fe030004840 tx=0x7fe030009cb0 comp rx=0 tx=0).stop 2022-05-13T23:14:16.262 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:16.260+0000 7fe04119d700 1 -- 172.21.15.2:0/649653726 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe03c0fb490 msgr2=0x7fe03c0701d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:14:16.262 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:16.260+0000 7fe04119d700 1 --2- 172.21.15.2:0/649653726 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe03c0fb490 0x7fe03c0701d0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fe02c00e940 tx=0x7fe02c00b040 comp rx=0 tx=0).stop 2022-05-13T23:14:16.266 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:16.264+0000 7fe04119d700 1 -- 172.21.15.2:0/649653726 shutdown_connections 2022-05-13T23:14:16.266 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:16.265+0000 7fe04119d700 1 --2- 172.21.15.2:0/649653726 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe03c0fd0c0 0x7fe03c06ed70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:14:16.266 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:16.265+0000 7fe04119d700 1 --2- 172.21.15.2:0/649653726 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe03c0fb490 0x7fe03c0701d0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:14:16.267 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:16.265+0000 7fe04119d700 1 --2- 172.21.15.2:0/649653726 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7fe024096ce0 0x7fe024099190 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:14:16.267 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:16.265+0000 7fe04119d700 1 --2- 172.21.15.2:0/649653726 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe03c0fc730 0x7fe03c06e830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:14:16.268 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:16.265+0000 7fe04119d700 1 -- 172.21.15.2:0/649653726 >> 172.21.15.2:0/649653726 conn(0x7fe03c0f6c80 msgr2=0x7fe03c0ff960 unknown :-1 s=STATE_NONE l=0).mark_down 2022-05-13T23:14:16.276 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:16.275+0000 7fe04119d700 1 -- 172.21.15.2:0/649653726 shutdown_connections 2022-05-13T23:14:16.276 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:16.275+0000 7fe04119d700 1 -- 172.21.15.2:0/649653726 wait complete. 2022-05-13T23:14:16.281 INFO:teuthology.orchestra.run.smithi002.stderr:new fs with metadata pool 2 and data pool 3 2022-05-13T23:14:16.307 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-05-13T23:14:16.543 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:16.537+0000 7f6f1db9d700 1 -- 172.21.15.2:0/443520437 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6f180fc780 msgr2=0x7f6f180fcbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:14:16.544 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:16.537+0000 7f6f1db9d700 1 --2- 172.21.15.2:0/443520437 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6f180fc780 0x7f6f180fcbd0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f6f04004870 tx=0x7f6f04009cb0 comp rx=0 tx=0).stop 2022-05-13T23:14:16.544 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:16.537+0000 7f6f1db9d700 1 -- 172.21.15.2:0/443520437 shutdown_connections 2022-05-13T23:14:16.544 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:16.537+0000 7f6f1db9d700 1 --2- 172.21.15.2:0/443520437 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6f180fd110 0x7f6f181039d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:14:16.545 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:16.537+0000 7f6f1db9d700 1 --2- 172.21.15.2:0/443520437 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f6f180fb4e0 0x7f6f180fb8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:14:16.545 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:16.538+0000 7f6f1db9d700 1 --2- 172.21.15.2:0/443520437 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6f180fc780 0x7f6f180fcbd0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:14:16.545 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:16.538+0000 7f6f1db9d700 1 -- 172.21.15.2:0/443520437 >> 172.21.15.2:0/443520437 conn(0x7f6f180f6cb0 msgr2=0x7f6f180f9100 unknown :-1 s=STATE_NONE l=0).mark_down 2022-05-13T23:14:16.546 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:16.538+0000 7f6f1db9d700 1 -- 172.21.15.2:0/443520437 shutdown_connections 2022-05-13T23:14:16.546 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:16.538+0000 7f6f1db9d700 1 -- 172.21.15.2:0/443520437 wait complete. 2022-05-13T23:14:16.546 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:16.539+0000 7f6f1db9d700 1 Processor -- start 2022-05-13T23:14:16.547 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:16.539+0000 7f6f1db9d700 1 -- start start 2022-05-13T23:14:16.547 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:16.539+0000 7f6f1db9d700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6f180fb4e0 0x7f6f1810e4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:14:16.548 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:16.539+0000 7f6f1db9d700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f6f180fc780 0x7f6f1810ea20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:14:16.548 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:16.539+0000 7f6f1db9d700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6f180fd110 0x7f6f18115ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:14:16.549 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:16.539+0000 7f6f1db9d700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f18115ff0 con 0x7f6f180fb4e0 2022-05-13T23:14:16.549 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:16.539+0000 7f6f1db9d700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f18116160 con 0x7f6f180fd110 2022-05-13T23:14:16.550 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:16.539+0000 7f6f1db9d700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f18116320 con 0x7f6f180fc780 2022-05-13T23:14:16.550 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:16.540+0000 7f6f1d39c700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6f180fd110 0x7f6f18115ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:14:16.551 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:16.540+0000 7f6f1cb9b700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6f180fb4e0 0x7f6f1810e4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:14:16.551 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:16.540+0000 7f6f1d39c700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6f180fd110 0x7f6f18115ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.2:57994/0 (socket says 172.21.15.2:57994) 2022-05-13T23:14:16.552 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:16.540+0000 7f6f1d39c700 1 -- 172.21.15.2:0/3235663203 learned_addr learned my addr 172.21.15.2:0/3235663203 (peer_addr_for_me v2:172.21.15.2:0/0) 2022-05-13T23:14:16.552 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:16.540+0000 7f6f17fff700 1 --2- 172.21.15.2:0/3235663203 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f6f180fc780 0x7f6f1810ea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:14:16.553 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:16.540+0000 7f6f1d39c700 1 -- 172.21.15.2:0/3235663203 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f6f180fc780 msgr2=0x7f6f1810ea20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:14:16.553 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:16.540+0000 7f6f1d39c700 1 --2- 172.21.15.2:0/3235663203 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f6f180fc780 0x7f6f1810ea20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:14:16.553 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:16.540+0000 7f6f1d39c700 1 -- 172.21.15.2:0/3235663203 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6f180fb4e0 msgr2=0x7f6f1810e4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:14:16.554 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:16.540+0000 7f6f1d39c700 1 --2- 172.21.15.2:0/3235663203 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6f180fb4e0 0x7f6f1810e4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:14:16.554 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:16.540+0000 7f6f1d39c700 1 -- 172.21.15.2:0/3235663203 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f04005040 con 0x7f6f180fd110 2022-05-13T23:14:16.554 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:16.540+0000 7f6f17fff700 1 --2- 172.21.15.2:0/3235663203 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f6f180fc780 0x7f6f1810ea20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-05-13T23:14:16.555 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:16.540+0000 7f6f1d39c700 1 --2- 172.21.15.2:0/3235663203 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6f180fd110 0x7f6f18115ab0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f6f1000ec20 tx=0x7f6f1000b040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-05-13T23:14:16.555 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:16.541+0000 7f6f157fa700 1 -- 172.21.15.2:0/3235663203 <== mon.1 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f100092e0 con 0x7f6f180fd110 2022-05-13T23:14:16.555 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:16.541+0000 7f6f157fa700 1 -- 172.21.15.2:0/3235663203 <== mon.1 v2:172.21.15.47:3300/0 2 ==== config(26 keys) v1 ==== 1087+0+0 (secure 0 0 0) 0x7f6f1000f040 con 0x7f6f180fd110 2022-05-13T23:14:16.555 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:16.541+0000 7f6f1db9d700 1 -- 172.21.15.2:0/3235663203 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6f18116600 con 0x7f6f180fd110 2022-05-13T23:14:16.556 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:16.541+0000 7f6f157fa700 1 -- 172.21.15.2:0/3235663203 <== mon.1 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f1000cab0 con 0x7f6f180fd110 2022-05-13T23:14:16.556 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:16.541+0000 7f6f1db9d700 1 -- 172.21.15.2:0/3235663203 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6f18116b70 con 0x7f6f180fd110 2022-05-13T23:14:16.556 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:16.542+0000 7f6f1db9d700 1 -- 172.21.15.2:0/3235663203 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6f18106d80 con 0x7f6f180fd110 2022-05-13T23:14:16.558 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:16.544+0000 7f6f157fa700 1 -- 172.21.15.2:0/3235663203 <== mon.1 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 16) v1 ==== 125136+0+0 (secure 0 0 0) 0x7f6f10042bc0 con 0x7f6f180fd110 2022-05-13T23:14:16.558 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:16.545+0000 7f6f157fa700 1 --2- 172.21.15.2:0/3235663203 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f6f08096d60 0x7f6f08099210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:14:16.558 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:16.545+0000 7f6f157fa700 1 -- 172.21.15.2:0/3235663203 <== mon.1 v2:172.21.15.47:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 7975+0+0 (secure 0 0 0) 0x7f6f100bf560 con 0x7f6f180fd110 2022-05-13T23:14:16.558 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:16.545+0000 7f6f1cb9b700 1 --2- 172.21.15.2:0/3235663203 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f6f08096d60 0x7f6f08099210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:14:16.559 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:16.546+0000 7f6f1cb9b700 1 --2- 172.21.15.2:0/3235663203 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f6f08096d60 0x7f6f08099210 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f6f180fc5e0 tx=0x7f6f0c00b040 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-05-13T23:14:16.559 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:16.547+0000 7f6f157fa700 1 -- 172.21.15.2:0/3235663203 <== mon.1 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+165215 (secure 0 0 0) 0x7f6f1008c990 con 0x7f6f180fd110 2022-05-13T23:14:16.744 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:16.743+0000 7f6f1db9d700 1 -- 172.21.15.2:0/3235663203 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6f18116d90 con 0x7f6f180fd110 2022-05-13T23:14:16.745 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:16.744+0000 7f6f157fa700 1 -- 172.21.15.2:0/3235663203 <== mon.1 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v82) v1 ==== 74+0+21280 (secure 0 0 0) 0x7f6f1000cea0 con 0x7f6f180fd110 2022-05-13T23:14:16.746 INFO:teuthology.orchestra.run.smithi002.stdout: 2022-05-13T23:14:16.746 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":82,"fsid":"16cf5f88-d311-11ec-be40-001a4aab830c","created":"2022-05-13T23:06:00.765453+0000","modified":"2022-05-13T23:14:16.235912+0000","last_up_change":"2022-05-13T23:13:20.935982+0000","last_in_change":"2022-05-13T23:13:02.796000+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2022-05-13T23:09:32.000644+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2022-05-13T23:14:14.195678+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":3,"pool_name":"cephfs_data","create_time":"2022-05-13T23:14:15.659212+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"11e700e8-f3e8-4903-b52c-db354b4395bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6802","nonce":1174843704},{"type":"v1","addr":"172.21.15.2:6803","nonce":1174843704}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6804","nonce":1174843704},{"type":"v1","addr":"172.21.15.2:6805","nonce":1174843704}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6808","nonce":1174843704},{"type":"v1","addr":"172.21.15.2:6809","nonce":1174843704}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6806","nonce":1174843704},{"type":"v1","addr":"172.21.15.2:6807","nonce":1174843704}]},"public_addr":"172.21.15.2:6803/1174843704","cluster_addr":"172.21.15.2:6805/1174843704","heartbeat_back_addr":"172.21.15.2:6809/1174843704","heartbeat_front_addr":"172.21.15.2:6807/1174843704","state":["exists","up"]},{"osd":1,"uuid":"eabfa3cf-f2a3-4995-a7de-f6d8ba02ee6f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6800","nonce":3853826948},{"type":"v1","addr":"172.21.15.47:6801","nonce":3853826948}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6802","nonce":3853826948},{"type":"v1","addr":"172.21.15.47:6803","nonce":3853826948}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6806","nonce":3853826948},{"type":"v1","addr":"172.21.15.47:6807","nonce":3853826948}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6804","nonce":3853826948},{"type":"v1","addr":"172.21.15.47:6805","nonce":3853826948}]},"public_addr":"172.21.15.47:6801/3853826948","cluster_addr":"172.21.15.47:6803/3853826948","heartbeat_back_addr":"172.21.15.47:6807/3853826948","heartbeat_front_addr":"172.21.15.47:6805/3853826948","state":["exists","up"]},{"osd":2,"uuid":"153c5bd5-8200-47a7-a135-8a95b6752cd2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6800","nonce":2562256956},{"type":"v1","addr":"172.21.15.115:6801","nonce":2562256956}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6802","nonce":2562256956},{"type":"v1","addr":"172.21.15.115:6803","nonce":2562256956}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6806","nonce":2562256956},{"type":"v1","addr":"172.21.15.115:6807","nonce":2562256956}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6804","nonce":2562256956},{"type":"v1","addr":"172.21.15.115:6805","nonce":2562256956}]},"public_addr":"172.21.15.115:6801/2562256956","cluster_addr":"172.21.15.115:6803/2562256956","heartbeat_back_addr":"172.21.15.115:6807/2562256956","heartbeat_front_addr":"172.21.15.115:6805/2562256956","state":["exists","up"]},{"osd":3,"uuid":"6ca9c602-5936-48da-a861-c437e6a33765","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6810","nonce":1099909411},{"type":"v1","addr":"172.21.15.2:6811","nonce":1099909411}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6812","nonce":1099909411},{"type":"v1","addr":"172.21.15.2:6813","nonce":1099909411}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6816","nonce":1099909411},{"type":"v1","addr":"172.21.15.2:6817","nonce":1099909411}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6814","nonce":1099909411},{"type":"v1","addr":"172.21.15.2:6815","nonce":1099909411}]},"public_addr":"172.21.15.2:6811/1099909411","cluster_addr":"172.21.15.2:6813/1099909411","heartbeat_back_addr":"172.21.15.2:6817/1099909411","heartbeat_front_addr":"172.21.15.2:6815/1099909411","state":["exists","up"]},{"osd":4,"uuid":"54d943d9-2610-422a-9e59-4b8212aced31","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6808","nonce":882753861},{"type":"v1","addr":"172.21.15.47:6809","nonce":882753861}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6810","nonce":882753861},{"type":"v1","addr":"172.21.15.47:6811","nonce":882753861}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6814","nonce":882753861},{"type":"v1","addr":"172.21.15.47:6815","nonce":882753861}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6812","nonce":882753861},{"type":"v1","addr":"172.21.15.47:6813","nonce":882753861}]},"public_addr":"172.21.15.47:6809/882753861","cluster_addr":"172.21.15.47:6811/882753861","heartbeat_back_addr":"172.21.15.47:6815/882753861","heartbeat_front_addr":"172.21.15.47:6813/882753861","state":["exists","up"]},{"osd":5,"uuid":"c18566cf-5403-4057-b2ee-aa298e87e8e9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6808","nonce":2122391993},{"type":"v1","addr":"172.21.15.115:6809","nonce":2122391993}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6810","nonce":2122391993},{"type":"v1","addr":"172.21.15.115:6811","nonce":2122391993}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6814","nonce":2122391993},{"type":"v1","addr":"172.21.15.115:6815","nonce":2122391993}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6812","nonce":2122391993},{"type":"v1","addr":"172.21.15.115:6813","nonce":2122391993}]},"public_addr":"172.21.15.115:6809/2122391993","cluster_addr":"172.21.15.115:6811/2122391993","heartbeat_back_addr":"172.21.15.115:6815/2122391993","heartbeat_front_addr":"172.21.15.115:6813/2122391993","state":["exists","up"]},{"osd":6,"uuid":"38ec7b6b-30f4-4d97-ad7e-e160ae59c843","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6818","nonce":1892198790},{"type":"v1","addr":"172.21.15.2:6819","nonce":1892198790}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6820","nonce":1892198790},{"type":"v1","addr":"172.21.15.2:6821","nonce":1892198790}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6824","nonce":1892198790},{"type":"v1","addr":"172.21.15.2:6825","nonce":1892198790}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6822","nonce":1892198790},{"type":"v1","addr":"172.21.15.2:6823","nonce":1892198790}]},"public_addr":"172.21.15.2:6819/1892198790","cluster_addr":"172.21.15.2:6821/1892198790","heartbeat_back_addr":"172.21.15.2:6825/1892198790","heartbeat_front_addr":"172.21.15.2:6823/1892198790","state":["exists","up"]},{"osd":7,"uuid":"e799b560-c61e-4369-b08b-45e5a6af37c0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6816","nonce":2812542825},{"type":"v1","addr":"172.21.15.47:6817","nonce":2812542825}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6818","nonce":2812542825},{"type":"v1","addr":"172.21.15.47:6819","nonce":2812542825}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6822","nonce":2812542825},{"type":"v1","addr":"172.21.15.47:6823","nonce":2812542825}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6820","nonce":2812542825},{"type":"v1","addr":"172.21.15.47:6821","nonce":2812542825}]},"public_addr":"172.21.15.47:6817/2812542825","cluster_addr":"172.21.15.47:6819/2812542825","heartbeat_back_addr":"172.21.15.47:6823/2812542825","heartbeat_front_addr":"172.21.15.47:6821/2812542825","state":["exists","up"]},{"osd":8,"uuid":"a58f2c0e-e487-41cf-8d2e-8c937cac9ebc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":58,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6816","nonce":3959632033},{"type":"v1","addr":"172.21.15.115:6817","nonce":3959632033}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6818","nonce":3959632033},{"type":"v1","addr":"172.21.15.115:6819","nonce":3959632033}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6822","nonce":3959632033},{"type":"v1","addr":"172.21.15.115:6823","nonce":3959632033}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6820","nonce":3959632033},{"type":"v1","addr":"172.21.15.115:6821","nonce":3959632033}]},"public_addr":"172.21.15.115:6817/3959632033","cluster_addr":"172.21.15.115:6819/3959632033","heartbeat_back_addr":"172.21.15.115:6823/3959632033","heartbeat_front_addr":"172.21.15.115:6821/3959632033","state":["exists","up"]},{"osd":9,"uuid":"12cead96-f726-4f7a-ab84-eb191b778a5c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":66,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6826","nonce":3004162255},{"type":"v1","addr":"172.21.15.2:6827","nonce":3004162255}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6828","nonce":3004162255},{"type":"v1","addr":"172.21.15.2:6829","nonce":3004162255}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6832","nonce":3004162255},{"type":"v1","addr":"172.21.15.2:6833","nonce":3004162255}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6830","nonce":3004162255},{"type":"v1","addr":"172.21.15.2:6831","nonce":3004162255}]},"public_addr":"172.21.15.2:6827/3004162255","cluster_addr":"172.21.15.2:6829/3004162255","heartbeat_back_addr":"172.21.15.2:6833/3004162255","heartbeat_front_addr":"172.21.15.2:6831/3004162255","state":["exists","up"]},{"osd":10,"uuid":"4f648118-7855-48d6-86d5-44eed5295f0c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":71,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6824","nonce":3443413998},{"type":"v1","addr":"172.21.15.47:6825","nonce":3443413998}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6826","nonce":3443413998},{"type":"v1","addr":"172.21.15.47:6827","nonce":3443413998}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6830","nonce":3443413998},{"type":"v1","addr":"172.21.15.47:6831","nonce":3443413998}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6828","nonce":3443413998},{"type":"v1","addr":"172.21.15.47:6829","nonce":3443413998}]},"public_addr":"172.21.15.47:6825/3443413998","cluster_addr":"172.21.15.47:6827/3443413998","heartbeat_back_addr":"172.21.15.47:6831/3443413998","heartbeat_front_addr":"172.21.15.47:6829/3443413998","state":["exists","up"]},{"osd":11,"uuid":"916541e0-fb28-46d5-9743-801c54c5a6d1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":78,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6824","nonce":1758718209},{"type":"v1","addr":"172.21.15.115:6825","nonce":1758718209}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6826","nonce":1758718209},{"type":"v1","addr":"172.21.15.115:6827","nonce":1758718209}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6830","nonce":1758718209},{"type":"v1","addr":"172.21.15.115:6831","nonce":1758718209}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6828","nonce":1758718209},{"type":"v1","addr":"172.21.15.115:6829","nonce":1758718209}]},"public_addr":"172.21.15.115:6825/1758718209","cluster_addr":"172.21.15.115:6827/1758718209","heartbeat_back_addr":"172.21.15.115:6831/1758718209","heartbeat_front_addr":"172.21.15.115:6829/1758718209","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-05-13T23:08:43.757857+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-05-13T23:09:06.537139+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-05-13T23:09:28.524837+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-05-13T23:10:02.057583+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-05-13T23:10:26.210648+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-05-13T23:10:47.885853+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-05-13T23:11:19.141865+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-05-13T23:11:42.376065+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-05-13T23:12:04.175870+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-05-13T23:12:34.695347+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-05-13T23:12:57.762613+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-05-13T23:13:19.114284+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.2:0/477249925":"2022-05-14T23:07:04.986087+0000","172.21.15.2:6800/3226099420":"2022-05-14T23:07:04.986087+0000","172.21.15.2:0/2214093915":"2022-05-14T23:07:04.986087+0000","172.21.15.2:0/1039403355":"2022-05-14T23:07:04.986087+0000","172.21.15.2:0/22091675":"2022-05-14T23:06:36.304108+0000","172.21.15.2:6800/1255134634":"2022-05-14T23:06:36.304108+0000","172.21.15.2:6801/3226099420":"2022-05-14T23:07:04.986087+0000","172.21.15.2:0/3768677276":"2022-05-14T23:06:36.304108+0000","172.21.15.2:6801/1255134634":"2022-05-14T23:06:36.304108+0000","172.21.15.2:0/407841268":"2022-05-14T23:06:36.304108+0000"},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2022-05-13T23:14:16.749 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:16.747+0000 7f6f1db9d700 1 -- 172.21.15.2:0/3235663203 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f6f08096d60 msgr2=0x7f6f08099210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:14:16.749 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:16.747+0000 7f6f1db9d700 1 --2- 172.21.15.2:0/3235663203 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f6f08096d60 0x7f6f08099210 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f6f180fc5e0 tx=0x7f6f0c00b040 comp rx=0 tx=0).stop 2022-05-13T23:14:16.750 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:16.747+0000 7f6f1db9d700 1 -- 172.21.15.2:0/3235663203 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6f180fd110 msgr2=0x7f6f18115ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:14:16.750 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:16.747+0000 7f6f1db9d700 1 --2- 172.21.15.2:0/3235663203 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6f180fd110 0x7f6f18115ab0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f6f1000ec20 tx=0x7f6f1000b040 comp rx=0 tx=0).stop 2022-05-13T23:14:16.750 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:16.747+0000 7f6f1db9d700 1 -- 172.21.15.2:0/3235663203 shutdown_connections 2022-05-13T23:14:16.750 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:16.747+0000 7f6f1db9d700 1 --2- 172.21.15.2:0/3235663203 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6f180fd110 0x7f6f18115ab0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:14:16.751 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:16.747+0000 7f6f1db9d700 1 --2- 172.21.15.2:0/3235663203 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f6f08096d60 0x7f6f08099210 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:14:16.751 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:16.747+0000 7f6f1db9d700 1 --2- 172.21.15.2:0/3235663203 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6f180fb4e0 0x7f6f1810e4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:14:16.751 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:16.747+0000 7f6f1db9d700 1 --2- 172.21.15.2:0/3235663203 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f6f180fc780 0x7f6f1810ea20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:14:16.751 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:16.747+0000 7f6f1db9d700 1 -- 172.21.15.2:0/3235663203 >> 172.21.15.2:0/3235663203 conn(0x7f6f180f6cb0 msgr2=0x7f6f180f8fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-05-13T23:14:16.751 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:16.747+0000 7f6f1db9d700 1 -- 172.21.15.2:0/3235663203 shutdown_connections 2022-05-13T23:14:16.752 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:16.748+0000 7f6f1db9d700 1 -- 172.21.15.2:0/3235663203 wait complete. 2022-05-13T23:14:16.776 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-05-13T23:14:17.014 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:17.012+0000 7fdefdd9d700 1 -- 172.21.15.2:0/79053056 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fdef806edb0 msgr2=0x7fdef806f220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:14:17.015 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:17.012+0000 7fdefdd9d700 1 --2- 172.21.15.2:0/79053056 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fdef806edb0 0x7fdef806f220 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fdee4004870 tx=0x7fdee4009cb0 comp rx=0 tx=0).stop 2022-05-13T23:14:17.015 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:17.012+0000 7fdefdd9d700 1 -- 172.21.15.2:0/79053056 shutdown_connections 2022-05-13T23:14:17.015 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:17.012+0000 7fdefdd9d700 1 --2- 172.21.15.2:0/79053056 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fdef8103420 0x7fdef8105800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:14:17.015 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:17.012+0000 7fdefdd9d700 1 --2- 172.21.15.2:0/79053056 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fdef8070370 0x7fdef806e870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:14:17.016 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:17.012+0000 7fdefdd9d700 1 --2- 172.21.15.2:0/79053056 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fdef806edb0 0x7fdef806f220 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:14:17.016 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:17.012+0000 7fdefdd9d700 1 -- 172.21.15.2:0/79053056 >> 172.21.15.2:0/79053056 conn(0x7fdef80f6c40 msgr2=0x7fdef80f90b0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-05-13T23:14:17.016 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:17.013+0000 7fdefdd9d700 1 -- 172.21.15.2:0/79053056 shutdown_connections 2022-05-13T23:14:17.016 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:17.013+0000 7fdefdd9d700 1 -- 172.21.15.2:0/79053056 wait complete. 2022-05-13T23:14:17.019 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:17.013+0000 7fdefdd9d700 1 Processor -- start 2022-05-13T23:14:17.019 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:17.014+0000 7fdefdd9d700 1 -- start start 2022-05-13T23:14:17.019 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:17.014+0000 7fdefdd9d700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fdef806edb0 0x7fdef80fe8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:14:17.019 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:17.014+0000 7fdefdd9d700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fdef8070370 0x7fdef80fee00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:14:17.020 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:17.014+0000 7fdefdd9d700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fdef8103420 0x7fdef80fb400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:14:17.020 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:17.014+0000 7fdefdd9d700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdef80fbb10 con 0x7fdef806edb0 2022-05-13T23:14:17.020 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:17.014+0000 7fdefdd9d700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdef80fbc80 con 0x7fdef8103420 2022-05-13T23:14:17.020 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:17.014+0000 7fdefdd9d700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdef80fbe40 con 0x7fdef8070370 2022-05-13T23:14:17.020 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:17.014+0000 7fdef7fff700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fdef8070370 0x7fdef80fee00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:14:17.021 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:17.014+0000 7fdefd59c700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fdef8103420 0x7fdef80fb400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:14:17.021 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:17.014+0000 7fdefcd9b700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fdef806edb0 0x7fdef80fe8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:14:17.021 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:17.014+0000 7fdef7fff700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fdef8070370 0x7fdef80fee00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.2:43426/0 (socket says 172.21.15.2:43426) 2022-05-13T23:14:17.021 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:17.014+0000 7fdef7fff700 1 -- 172.21.15.2:0/1106603001 learned_addr learned my addr 172.21.15.2:0/1106603001 (peer_addr_for_me v2:172.21.15.2:0/0) 2022-05-13T23:14:17.022 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:17.015+0000 7fdefd59c700 1 -- 172.21.15.2:0/1106603001 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fdef8070370 msgr2=0x7fdef80fee00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:14:17.022 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:17.015+0000 7fdefd59c700 1 --2- 172.21.15.2:0/1106603001 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fdef8070370 0x7fdef80fee00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:14:17.022 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:17.015+0000 7fdefd59c700 1 -- 172.21.15.2:0/1106603001 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fdef806edb0 msgr2=0x7fdef80fe8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:14:17.022 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:17.015+0000 7fdefd59c700 1 --2- 172.21.15.2:0/1106603001 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fdef806edb0 0x7fdef80fe8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:14:17.022 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:17.015+0000 7fdefd59c700 1 -- 172.21.15.2:0/1106603001 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdee4005040 con 0x7fdef8103420 2022-05-13T23:14:17.023 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:17.015+0000 7fdef7fff700 1 --2- 172.21.15.2:0/1106603001 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fdef8070370 0x7fdef80fee00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-05-13T23:14:17.023 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:17.015+0000 7fdefd59c700 1 --2- 172.21.15.2:0/1106603001 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fdef8103420 0x7fdef80fb400 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fdef8070140 tx=0x7fdef000b040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-05-13T23:14:17.023 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:17.015+0000 7fdef57fa700 1 -- 172.21.15.2:0/1106603001 <== mon.1 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdef0009870 con 0x7fdef8103420 2022-05-13T23:14:17.023 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:17.016+0000 7fdef57fa700 1 -- 172.21.15.2:0/1106603001 <== mon.1 v2:172.21.15.47:3300/0 2 ==== config(26 keys) v1 ==== 1087+0+0 (secure 0 0 0) 0x7fdef000f040 con 0x7fdef8103420 2022-05-13T23:14:17.024 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:17.016+0000 7fdefdd9d700 1 -- 172.21.15.2:0/1106603001 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdef811aa70 con 0x7fdef8103420 2022-05-13T23:14:17.024 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:17.016+0000 7fdef57fa700 1 -- 172.21.15.2:0/1106603001 <== mon.1 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdef0007e60 con 0x7fdef8103420 2022-05-13T23:14:17.024 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:17.016+0000 7fdefdd9d700 1 -- 172.21.15.2:0/1106603001 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdef811af90 con 0x7fdef8103420 2022-05-13T23:14:17.024 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:17.017+0000 7fdefdd9d700 1 -- 172.21.15.2:0/1106603001 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdef8108f30 con 0x7fdef8103420 2022-05-13T23:14:17.026 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:17.019+0000 7fdef57fa700 1 -- 172.21.15.2:0/1106603001 <== mon.1 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 16) v1 ==== 125136+0+0 (secure 0 0 0) 0x7fdef001a020 con 0x7fdef8103420 2022-05-13T23:14:17.027 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:17.020+0000 7fdef57fa700 1 --2- 172.21.15.2:0/1106603001 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7fdee8096ce0 0x7fdee8099190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:14:17.027 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:17.020+0000 7fdef57fa700 1 -- 172.21.15.2:0/1106603001 <== mon.1 v2:172.21.15.47:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 7975+0+0 (secure 0 0 0) 0x7fdef00bec40 con 0x7fdef8103420 2022-05-13T23:14:17.027 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:17.020+0000 7fdefcd9b700 1 --2- 172.21.15.2:0/1106603001 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7fdee8096ce0 0x7fdee8099190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:14:17.027 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:17.021+0000 7fdefcd9b700 1 --2- 172.21.15.2:0/1106603001 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7fdee8096ce0 0x7fdee8099190 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fdeec0048b0 tx=0x7fdeec00b040 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-05-13T23:14:17.028 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:17.022+0000 7fdef57fa700 1 -- 172.21.15.2:0/1106603001 <== mon.1 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+165215 (secure 0 0 0) 0x7fdef008bf80 con 0x7fdef8103420 2022-05-13T23:14:17.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:14:16 smithi047 ceph-mon[77147]: from='client.? 172.21.15.2:0/545744284' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2022-05-13T23:14:17.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:14:16 smithi047 ceph-mon[77147]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2022-05-13T23:14:17.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:14:16 smithi047 ceph-mon[77147]: pgmap v285: 33 pgs: 3 creating+peering, 29 unknown, 1 active+clean; 577 KiB data, 74 MiB used, 1.0 TiB / 1.0 TiB avail 2022-05-13T23:14:17.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:14:16 smithi047 ceph-mon[77147]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2022-05-13T23:14:17.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:14:16 smithi047 ceph-mon[77147]: osdmap e81: 12 total, 12 up, 12 in 2022-05-13T23:14:17.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:14:16 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:14:17.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:14:16 smithi047 ceph-mon[77147]: from='client.? 172.21.15.2:0/649653726' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2022-05-13T23:14:17.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:14:16 smithi047 ceph-mon[77147]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2022-05-13T23:14:17.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:14:16 smithi047 ceph-mon[77147]: daemon mds.l assigned to filesystem cephfs as rank 0 2022-05-13T23:14:17.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:14:16 smithi047 ceph-mon[77147]: osdmap e82: 12 total, 12 up, 12 in 2022-05-13T23:14:17.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:14:16 smithi047 ceph-mon[77147]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2022-05-13T23:14:17.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:14:16 smithi047 ceph-mon[77147]: fsmap cephfs:1 {0=l=up:creating} 11 up:standby 2022-05-13T23:14:17.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:14:16 smithi047 ceph-mon[77147]: daemon mds.l is now active in filesystem cephfs as rank 0 2022-05-13T23:14:17.072 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:14:16 smithi002 ceph-mon[71816]: from='client.? 172.21.15.2:0/545744284' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2022-05-13T23:14:17.073 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:14:16 smithi002 ceph-mon[71816]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2022-05-13T23:14:17.073 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:14:16 smithi002 ceph-mon[71816]: pgmap v285: 33 pgs: 3 creating+peering, 29 unknown, 1 active+clean; 577 KiB data, 74 MiB used, 1.0 TiB / 1.0 TiB avail 2022-05-13T23:14:17.073 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:14:16 smithi002 ceph-mon[71816]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2022-05-13T23:14:17.073 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:14:16 smithi002 ceph-mon[71816]: osdmap e81: 12 total, 12 up, 12 in 2022-05-13T23:14:17.074 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:14:16 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:14:17.074 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:14:16 smithi002 ceph-mon[71816]: from='client.? 172.21.15.2:0/649653726' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2022-05-13T23:14:17.074 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:14:16 smithi002 ceph-mon[71816]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2022-05-13T23:14:17.074 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:14:16 smithi002 ceph-mon[71816]: daemon mds.l assigned to filesystem cephfs as rank 0 2022-05-13T23:14:17.075 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:14:16 smithi002 ceph-mon[71816]: osdmap e82: 12 total, 12 up, 12 in 2022-05-13T23:14:17.075 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:14:16 smithi002 ceph-mon[71816]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2022-05-13T23:14:17.075 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:14:16 smithi002 ceph-mon[71816]: fsmap cephfs:1 {0=l=up:creating} 11 up:standby 2022-05-13T23:14:17.075 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:14:16 smithi002 ceph-mon[71816]: daemon mds.l is now active in filesystem cephfs as rank 0 2022-05-13T23:14:17.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:14:16 smithi115 ceph-mon[78593]: from='client.? 172.21.15.2:0/545744284' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2022-05-13T23:14:17.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:14:16 smithi115 ceph-mon[78593]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2022-05-13T23:14:17.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:14:16 smithi115 ceph-mon[78593]: pgmap v285: 33 pgs: 3 creating+peering, 29 unknown, 1 active+clean; 577 KiB data, 74 MiB used, 1.0 TiB / 1.0 TiB avail 2022-05-13T23:14:17.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:14:16 smithi115 ceph-mon[78593]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2022-05-13T23:14:17.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:14:16 smithi115 ceph-mon[78593]: osdmap e81: 12 total, 12 up, 12 in 2022-05-13T23:14:17.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:14:16 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:14:17.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:14:16 smithi115 ceph-mon[78593]: from='client.? 172.21.15.2:0/649653726' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2022-05-13T23:14:17.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:14:16 smithi115 ceph-mon[78593]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2022-05-13T23:14:17.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:14:16 smithi115 ceph-mon[78593]: daemon mds.l assigned to filesystem cephfs as rank 0 2022-05-13T23:14:17.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:14:16 smithi115 ceph-mon[78593]: osdmap e82: 12 total, 12 up, 12 in 2022-05-13T23:14:17.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:14:16 smithi115 ceph-mon[78593]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2022-05-13T23:14:17.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:14:16 smithi115 ceph-mon[78593]: fsmap cephfs:1 {0=l=up:creating} 11 up:standby 2022-05-13T23:14:17.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:14:16 smithi115 ceph-mon[78593]: daemon mds.l is now active in filesystem cephfs as rank 0 2022-05-13T23:14:17.224 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:17.223+0000 7fdefdd9d700 1 -- 172.21.15.2:0/1106603001 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fdef81019f0 con 0x7fdef8103420 2022-05-13T23:14:17.225 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:17.224+0000 7fdef57fa700 1 -- 172.21.15.2:0/1106603001 <== mon.1 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v82) v1 ==== 74+0+21280 (secure 0 0 0) 0x7fdef008f5a0 con 0x7fdef8103420 2022-05-13T23:14:17.226 INFO:teuthology.orchestra.run.smithi002.stdout: 2022-05-13T23:14:17.226 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":82,"fsid":"16cf5f88-d311-11ec-be40-001a4aab830c","created":"2022-05-13T23:06:00.765453+0000","modified":"2022-05-13T23:14:16.235912+0000","last_up_change":"2022-05-13T23:13:20.935982+0000","last_in_change":"2022-05-13T23:13:02.796000+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2022-05-13T23:09:32.000644+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2022-05-13T23:14:14.195678+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":3,"pool_name":"cephfs_data","create_time":"2022-05-13T23:14:15.659212+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"11e700e8-f3e8-4903-b52c-db354b4395bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6802","nonce":1174843704},{"type":"v1","addr":"172.21.15.2:6803","nonce":1174843704}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6804","nonce":1174843704},{"type":"v1","addr":"172.21.15.2:6805","nonce":1174843704}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6808","nonce":1174843704},{"type":"v1","addr":"172.21.15.2:6809","nonce":1174843704}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6806","nonce":1174843704},{"type":"v1","addr":"172.21.15.2:6807","nonce":1174843704}]},"public_addr":"172.21.15.2:6803/1174843704","cluster_addr":"172.21.15.2:6805/1174843704","heartbeat_back_addr":"172.21.15.2:6809/1174843704","heartbeat_front_addr":"172.21.15.2:6807/1174843704","state":["exists","up"]},{"osd":1,"uuid":"eabfa3cf-f2a3-4995-a7de-f6d8ba02ee6f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6800","nonce":3853826948},{"type":"v1","addr":"172.21.15.47:6801","nonce":3853826948}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6802","nonce":3853826948},{"type":"v1","addr":"172.21.15.47:6803","nonce":3853826948}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6806","nonce":3853826948},{"type":"v1","addr":"172.21.15.47:6807","nonce":3853826948}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6804","nonce":3853826948},{"type":"v1","addr":"172.21.15.47:6805","nonce":3853826948}]},"public_addr":"172.21.15.47:6801/3853826948","cluster_addr":"172.21.15.47:6803/3853826948","heartbeat_back_addr":"172.21.15.47:6807/3853826948","heartbeat_front_addr":"172.21.15.47:6805/3853826948","state":["exists","up"]},{"osd":2,"uuid":"153c5bd5-8200-47a7-a135-8a95b6752cd2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6800","nonce":2562256956},{"type":"v1","addr":"172.21.15.115:6801","nonce":2562256956}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6802","nonce":2562256956},{"type":"v1","addr":"172.21.15.115:6803","nonce":2562256956}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6806","nonce":2562256956},{"type":"v1","addr":"172.21.15.115:6807","nonce":2562256956}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6804","nonce":2562256956},{"type":"v1","addr":"172.21.15.115:6805","nonce":2562256956}]},"public_addr":"172.21.15.115:6801/2562256956","cluster_addr":"172.21.15.115:6803/2562256956","heartbeat_back_addr":"172.21.15.115:6807/2562256956","heartbeat_front_addr":"172.21.15.115:6805/2562256956","state":["exists","up"]},{"osd":3,"uuid":"6ca9c602-5936-48da-a861-c437e6a33765","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6810","nonce":1099909411},{"type":"v1","addr":"172.21.15.2:6811","nonce":1099909411}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6812","nonce":1099909411},{"type":"v1","addr":"172.21.15.2:6813","nonce":1099909411}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6816","nonce":1099909411},{"type":"v1","addr":"172.21.15.2:6817","nonce":1099909411}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6814","nonce":1099909411},{"type":"v1","addr":"172.21.15.2:6815","nonce":1099909411}]},"public_addr":"172.21.15.2:6811/1099909411","cluster_addr":"172.21.15.2:6813/1099909411","heartbeat_back_addr":"172.21.15.2:6817/1099909411","heartbeat_front_addr":"172.21.15.2:6815/1099909411","state":["exists","up"]},{"osd":4,"uuid":"54d943d9-2610-422a-9e59-4b8212aced31","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6808","nonce":882753861},{"type":"v1","addr":"172.21.15.47:6809","nonce":882753861}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6810","nonce":882753861},{"type":"v1","addr":"172.21.15.47:6811","nonce":882753861}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6814","nonce":882753861},{"type":"v1","addr":"172.21.15.47:6815","nonce":882753861}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6812","nonce":882753861},{"type":"v1","addr":"172.21.15.47:6813","nonce":882753861}]},"public_addr":"172.21.15.47:6809/882753861","cluster_addr":"172.21.15.47:6811/882753861","heartbeat_back_addr":"172.21.15.47:6815/882753861","heartbeat_front_addr":"172.21.15.47:6813/882753861","state":["exists","up"]},{"osd":5,"uuid":"c18566cf-5403-4057-b2ee-aa298e87e8e9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6808","nonce":2122391993},{"type":"v1","addr":"172.21.15.115:6809","nonce":2122391993}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6810","nonce":2122391993},{"type":"v1","addr":"172.21.15.115:6811","nonce":2122391993}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6814","nonce":2122391993},{"type":"v1","addr":"172.21.15.115:6815","nonce":2122391993}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6812","nonce":2122391993},{"type":"v1","addr":"172.21.15.115:6813","nonce":2122391993}]},"public_addr":"172.21.15.115:6809/2122391993","cluster_addr":"172.21.15.115:6811/2122391993","heartbeat_back_addr":"172.21.15.115:6815/2122391993","heartbeat_front_addr":"172.21.15.115:6813/2122391993","state":["exists","up"]},{"osd":6,"uuid":"38ec7b6b-30f4-4d97-ad7e-e160ae59c843","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6818","nonce":1892198790},{"type":"v1","addr":"172.21.15.2:6819","nonce":1892198790}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6820","nonce":1892198790},{"type":"v1","addr":"172.21.15.2:6821","nonce":1892198790}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6824","nonce":1892198790},{"type":"v1","addr":"172.21.15.2:6825","nonce":1892198790}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6822","nonce":1892198790},{"type":"v1","addr":"172.21.15.2:6823","nonce":1892198790}]},"public_addr":"172.21.15.2:6819/1892198790","cluster_addr":"172.21.15.2:6821/1892198790","heartbeat_back_addr":"172.21.15.2:6825/1892198790","heartbeat_front_addr":"172.21.15.2:6823/1892198790","state":["exists","up"]},{"osd":7,"uuid":"e799b560-c61e-4369-b08b-45e5a6af37c0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6816","nonce":2812542825},{"type":"v1","addr":"172.21.15.47:6817","nonce":2812542825}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6818","nonce":2812542825},{"type":"v1","addr":"172.21.15.47:6819","nonce":2812542825}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6822","nonce":2812542825},{"type":"v1","addr":"172.21.15.47:6823","nonce":2812542825}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6820","nonce":2812542825},{"type":"v1","addr":"172.21.15.47:6821","nonce":2812542825}]},"public_addr":"172.21.15.47:6817/2812542825","cluster_addr":"172.21.15.47:6819/2812542825","heartbeat_back_addr":"172.21.15.47:6823/2812542825","heartbeat_front_addr":"172.21.15.47:6821/2812542825","state":["exists","up"]},{"osd":8,"uuid":"a58f2c0e-e487-41cf-8d2e-8c937cac9ebc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":58,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6816","nonce":3959632033},{"type":"v1","addr":"172.21.15.115:6817","nonce":3959632033}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6818","nonce":3959632033},{"type":"v1","addr":"172.21.15.115:6819","nonce":3959632033}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6822","nonce":3959632033},{"type":"v1","addr":"172.21.15.115:6823","nonce":3959632033}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6820","nonce":3959632033},{"type":"v1","addr":"172.21.15.115:6821","nonce":3959632033}]},"public_addr":"172.21.15.115:6817/3959632033","cluster_addr":"172.21.15.115:6819/3959632033","heartbeat_back_addr":"172.21.15.115:6823/3959632033","heartbeat_front_addr":"172.21.15.115:6821/3959632033","state":["exists","up"]},{"osd":9,"uuid":"12cead96-f726-4f7a-ab84-eb191b778a5c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":66,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6826","nonce":3004162255},{"type":"v1","addr":"172.21.15.2:6827","nonce":3004162255}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6828","nonce":3004162255},{"type":"v1","addr":"172.21.15.2:6829","nonce":3004162255}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6832","nonce":3004162255},{"type":"v1","addr":"172.21.15.2:6833","nonce":3004162255}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6830","nonce":3004162255},{"type":"v1","addr":"172.21.15.2:6831","nonce":3004162255}]},"public_addr":"172.21.15.2:6827/3004162255","cluster_addr":"172.21.15.2:6829/3004162255","heartbeat_back_addr":"172.21.15.2:6833/3004162255","heartbeat_front_addr":"172.21.15.2:6831/3004162255","state":["exists","up"]},{"osd":10,"uuid":"4f648118-7855-48d6-86d5-44eed5295f0c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":71,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6824","nonce":3443413998},{"type":"v1","addr":"172.21.15.47:6825","nonce":3443413998}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6826","nonce":3443413998},{"type":"v1","addr":"172.21.15.47:6827","nonce":3443413998}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6830","nonce":3443413998},{"type":"v1","addr":"172.21.15.47:6831","nonce":3443413998}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6828","nonce":3443413998},{"type":"v1","addr":"172.21.15.47:6829","nonce":3443413998}]},"public_addr":"172.21.15.47:6825/3443413998","cluster_addr":"172.21.15.47:6827/3443413998","heartbeat_back_addr":"172.21.15.47:6831/3443413998","heartbeat_front_addr":"172.21.15.47:6829/3443413998","state":["exists","up"]},{"osd":11,"uuid":"916541e0-fb28-46d5-9743-801c54c5a6d1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":78,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6824","nonce":1758718209},{"type":"v1","addr":"172.21.15.115:6825","nonce":1758718209}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6826","nonce":1758718209},{"type":"v1","addr":"172.21.15.115:6827","nonce":1758718209}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6830","nonce":1758718209},{"type":"v1","addr":"172.21.15.115:6831","nonce":1758718209}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6828","nonce":1758718209},{"type":"v1","addr":"172.21.15.115:6829","nonce":1758718209}]},"public_addr":"172.21.15.115:6825/1758718209","cluster_addr":"172.21.15.115:6827/1758718209","heartbeat_back_addr":"172.21.15.115:6831/1758718209","heartbeat_front_addr":"172.21.15.115:6829/1758718209","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-05-13T23:08:43.757857+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-05-13T23:09:06.537139+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-05-13T23:09:28.524837+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-05-13T23:10:02.057583+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-05-13T23:10:26.210648+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-05-13T23:10:47.885853+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-05-13T23:11:19.141865+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-05-13T23:11:42.376065+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-05-13T23:12:04.175870+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-05-13T23:12:34.695347+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-05-13T23:12:57.762613+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-05-13T23:13:19.114284+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.2:0/477249925":"2022-05-14T23:07:04.986087+0000","172.21.15.2:6800/3226099420":"2022-05-14T23:07:04.986087+0000","172.21.15.2:0/2214093915":"2022-05-14T23:07:04.986087+0000","172.21.15.2:0/1039403355":"2022-05-14T23:07:04.986087+0000","172.21.15.2:0/22091675":"2022-05-14T23:06:36.304108+0000","172.21.15.2:6800/1255134634":"2022-05-14T23:06:36.304108+0000","172.21.15.2:6801/3226099420":"2022-05-14T23:07:04.986087+0000","172.21.15.2:0/3768677276":"2022-05-14T23:06:36.304108+0000","172.21.15.2:6801/1255134634":"2022-05-14T23:06:36.304108+0000","172.21.15.2:0/407841268":"2022-05-14T23:06:36.304108+0000"},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2022-05-13T23:14:17.228 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:17.226+0000 7fdefdd9d700 1 -- 172.21.15.2:0/1106603001 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7fdee8096ce0 msgr2=0x7fdee8099190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:14:17.229 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:17.226+0000 7fdefdd9d700 1 --2- 172.21.15.2:0/1106603001 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7fdee8096ce0 0x7fdee8099190 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fdeec0048b0 tx=0x7fdeec00b040 comp rx=0 tx=0).stop 2022-05-13T23:14:17.229 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:17.226+0000 7fdefdd9d700 1 -- 172.21.15.2:0/1106603001 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fdef8103420 msgr2=0x7fdef80fb400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:14:17.229 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:17.226+0000 7fdefdd9d700 1 --2- 172.21.15.2:0/1106603001 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fdef8103420 0x7fdef80fb400 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fdef8070140 tx=0x7fdef000b040 comp rx=0 tx=0).stop 2022-05-13T23:14:17.230 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:17.227+0000 7fdefdd9d700 1 -- 172.21.15.2:0/1106603001 shutdown_connections 2022-05-13T23:14:17.230 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:17.227+0000 7fdefdd9d700 1 --2- 172.21.15.2:0/1106603001 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fdef8103420 0x7fdef80fb400 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:14:17.230 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:17.227+0000 7fdefdd9d700 1 --2- 172.21.15.2:0/1106603001 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7fdee8096ce0 0x7fdee8099190 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:14:17.230 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:17.227+0000 7fdefdd9d700 1 --2- 172.21.15.2:0/1106603001 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fdef806edb0 0x7fdef80fe8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:14:17.230 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:17.227+0000 7fdefdd9d700 1 --2- 172.21.15.2:0/1106603001 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fdef8070370 0x7fdef80fee00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:14:17.231 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:17.227+0000 7fdefdd9d700 1 -- 172.21.15.2:0/1106603001 >> 172.21.15.2:0/1106603001 conn(0x7fdef80f6c40 msgr2=0x7fdef80f9020 unknown :-1 s=STATE_NONE l=0).mark_down 2022-05-13T23:14:17.231 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:17.227+0000 7fdefdd9d700 1 -- 172.21.15.2:0/1106603001 shutdown_connections 2022-05-13T23:14:17.231 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:17.227+0000 7fdefdd9d700 1 -- 172.21.15.2:0/1106603001 wait complete. 2022-05-13T23:14:17.261 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2022-05-13T23:14:17.500 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:17.494+0000 7f7e1e39d700 1 -- 172.21.15.2:0/2647356387 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7e180645f0 msgr2=0x7f7e180fe180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:14:17.501 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:17.494+0000 7f7e1e39d700 1 --2- 172.21.15.2:0/2647356387 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7e180645f0 0x7f7e180fe180 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f7e14004870 tx=0x7f7e14009cb0 comp rx=0 tx=0).stop 2022-05-13T23:14:17.501 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:17.495+0000 7f7e1e39d700 1 -- 172.21.15.2:0/2647356387 shutdown_connections 2022-05-13T23:14:17.502 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:17.495+0000 7f7e1e39d700 1 --2- 172.21.15.2:0/2647356387 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7e18101110 0x7f7e18103590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:14:17.502 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:17.495+0000 7f7e1e39d700 1 --2- 172.21.15.2:0/2647356387 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7e180645f0 0x7f7e180fe180 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:14:17.502 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:17.495+0000 7f7e1e39d700 1 --2- 172.21.15.2:0/2647356387 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f7e180fe6c0 0x7f7e18100aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:14:17.503 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:17.495+0000 7f7e1e39d700 1 -- 172.21.15.2:0/2647356387 >> 172.21.15.2:0/2647356387 conn(0x7f7e180f5b70 msgr2=0x7f7e180f7fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-05-13T23:14:17.503 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:17.495+0000 7f7e1e39d700 1 -- 172.21.15.2:0/2647356387 shutdown_connections 2022-05-13T23:14:17.503 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:17.495+0000 7f7e1e39d700 1 -- 172.21.15.2:0/2647356387 wait complete. 2022-05-13T23:14:17.504 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:17.495+0000 7f7e1e39d700 1 Processor -- start 2022-05-13T23:14:17.504 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:17.496+0000 7f7e1e39d700 1 -- start start 2022-05-13T23:14:17.504 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:17.496+0000 7f7e1e39d700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7e180645f0 0x7f7e180fba20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:14:17.504 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:17.496+0000 7f7e1e39d700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7e180fe6c0 0x7f7e180fa080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:14:17.504 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:17.496+0000 7f7e1e39d700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f7e18101110 0x7f7e180fa5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:14:17.505 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:17.496+0000 7f7e1e39d700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e180facd0 con 0x7f7e180fe6c0 2022-05-13T23:14:17.505 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:17.496+0000 7f7e1e39d700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e180fae40 con 0x7f7e180645f0 2022-05-13T23:14:17.505 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:17.496+0000 7f7e1e39d700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e181163f0 con 0x7f7e18101110 2022-05-13T23:14:17.505 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:17.496+0000 7f7e1cb9a700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7e180fe6c0 0x7f7e180fa080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:14:17.505 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:17.496+0000 7f7e1d39b700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7e180645f0 0x7f7e180fba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:14:17.506 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:17.496+0000 7f7e1d39b700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7e180645f0 0x7f7e180fba20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.2:58024/0 (socket says 172.21.15.2:58024) 2022-05-13T23:14:17.506 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:17.496+0000 7f7e1d39b700 1 -- 172.21.15.2:0/2690804389 learned_addr learned my addr 172.21.15.2:0/2690804389 (peer_addr_for_me v2:172.21.15.2:0/0) 2022-05-13T23:14:17.506 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:17.496+0000 7f7e1db9c700 1 --2- 172.21.15.2:0/2690804389 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f7e18101110 0x7f7e180fa5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:14:17.506 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:17.497+0000 7f7e1d39b700 1 -- 172.21.15.2:0/2690804389 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f7e18101110 msgr2=0x7f7e180fa5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:14:17.507 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:17.497+0000 7f7e1d39b700 1 --2- 172.21.15.2:0/2690804389 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f7e18101110 0x7f7e180fa5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:14:17.507 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:17.497+0000 7f7e1d39b700 1 -- 172.21.15.2:0/2690804389 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7e180fe6c0 msgr2=0x7f7e180fa080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:14:17.507 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:17.497+0000 7f7e1d39b700 1 --2- 172.21.15.2:0/2690804389 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7e180fe6c0 0x7f7e180fa080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:14:17.507 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:17.497+0000 7f7e1d39b700 1 -- 172.21.15.2:0/2690804389 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e100097c0 con 0x7f7e180645f0 2022-05-13T23:14:17.507 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:17.497+0000 7f7e1cb9a700 1 --2- 172.21.15.2:0/2690804389 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7e180fe6c0 0x7f7e180fa080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-05-13T23:14:17.508 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:17.497+0000 7f7e1d39b700 1 --2- 172.21.15.2:0/2690804389 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7e180645f0 0x7f7e180fba20 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f7e14004710 tx=0x7f7e14009cb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-05-13T23:14:17.508 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:17.497+0000 7f7e0dffb700 1 -- 172.21.15.2:0/2690804389 <== mon.1 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e1400e040 con 0x7f7e180645f0 2022-05-13T23:14:17.510 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:17.497+0000 7f7e0dffb700 1 -- 172.21.15.2:0/2690804389 <== mon.1 v2:172.21.15.47:3300/0 2 ==== config(26 keys) v1 ==== 1087+0+0 (secure 0 0 0) 0x7f7e14004200 con 0x7f7e180645f0 2022-05-13T23:14:17.511 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:17.497+0000 7f7e0dffb700 1 -- 172.21.15.2:0/2690804389 <== mon.1 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e1401caf0 con 0x7f7e180645f0 2022-05-13T23:14:17.511 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:17.497+0000 7f7e1e39d700 1 -- 172.21.15.2:0/2690804389 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7e14005040 con 0x7f7e180645f0 2022-05-13T23:14:17.511 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:17.498+0000 7f7e1e39d700 1 -- 172.21.15.2:0/2690804389 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e181167d0 con 0x7f7e180645f0 2022-05-13T23:14:17.511 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:17.499+0000 7f7e1e39d700 1 -- 172.21.15.2:0/2690804389 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7e18106cd0 con 0x7f7e180645f0 2022-05-13T23:14:17.513 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:17.500+0000 7f7e0dffb700 1 -- 172.21.15.2:0/2690804389 <== mon.1 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 16) v1 ==== 125136+0+0 (secure 0 0 0) 0x7f7e14009240 con 0x7f7e180645f0 2022-05-13T23:14:17.513 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:17.501+0000 7f7e0dffb700 1 --2- 172.21.15.2:0/2690804389 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f7e04096ce0 0x7f7e04099190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:14:17.513 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:17.501+0000 7f7e0dffb700 1 -- 172.21.15.2:0/2690804389 <== mon.1 v2:172.21.15.47:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 7975+0+0 (secure 0 0 0) 0x7f7e140c0410 con 0x7f7e180645f0 2022-05-13T23:14:17.513 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:17.502+0000 7f7e1cb9a700 1 --2- 172.21.15.2:0/2690804389 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f7e04096ce0 0x7f7e04099190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:14:17.514 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:17.502+0000 7f7e1cb9a700 1 --2- 172.21.15.2:0/2690804389 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f7e04096ce0 0x7f7e04099190 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f7e18064b00 tx=0x7f7e0800b040 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-05-13T23:14:17.514 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:17.505+0000 7f7e0dffb700 1 -- 172.21.15.2:0/2690804389 <== mon.1 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+165215 (secure 0 0 0) 0x7f7e1408d750 con 0x7f7e180645f0 2022-05-13T23:14:17.765 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:17.763+0000 7f7e1e39d700 1 -- 172.21.15.2:0/2690804389 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f7e1802a570 con 0x7f7e180645f0 2022-05-13T23:14:18.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:14:17 smithi047 ceph-mon[77147]: from='client.? 172.21.15.2:0/3235663203' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2022-05-13T23:14:18.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:14:17 smithi047 ceph-mon[77147]: from='client.? 172.21.15.2:0/1106603001' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2022-05-13T23:14:18.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:14:17 smithi047 ceph-mon[77147]: pgmap v288: 97 pgs: 10 creating+activating, 12 creating+peering, 51 unknown, 24 active+clean; 577 KiB data, 74 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 3 op/s 2022-05-13T23:14:18.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:14:17 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:14:18.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:14:17 smithi047 ceph-mon[77147]: fsmap cephfs:1 {0=l=up:active} 11 up:standby 2022-05-13T23:14:18.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:14:17 smithi047 ceph-mon[77147]: osdmap e83: 12 total, 12 up, 12 in 2022-05-13T23:14:18.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:14:17 smithi047 ceph-mon[77147]: fsmap cephfs:1 {0=l=up:active} 11 up:standby 2022-05-13T23:14:18.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:14:17 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:14:18.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:14:17 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:14:18.072 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:14:17 smithi002 ceph-mon[71816]: from='client.? 172.21.15.2:0/3235663203' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2022-05-13T23:14:18.073 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:14:17 smithi002 ceph-mon[71816]: from='client.? 172.21.15.2:0/1106603001' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2022-05-13T23:14:18.073 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:14:17 smithi002 ceph-mon[71816]: pgmap v288: 97 pgs: 10 creating+activating, 12 creating+peering, 51 unknown, 24 active+clean; 577 KiB data, 74 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 3 op/s 2022-05-13T23:14:18.073 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:14:17 smithi002 ceph-mon[71816]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:14:18.073 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:14:17 smithi002 ceph-mon[71816]: fsmap cephfs:1 {0=l=up:active} 11 up:standby 2022-05-13T23:14:18.074 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:14:17 smithi002 ceph-mon[71816]: osdmap e83: 12 total, 12 up, 12 in 2022-05-13T23:14:18.074 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:14:17 smithi002 ceph-mon[71816]: fsmap cephfs:1 {0=l=up:active} 11 up:standby 2022-05-13T23:14:18.074 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:14:17 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:14:18.074 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:14:17 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:14:18.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:14:17 smithi115 ceph-mon[78593]: from='client.? 172.21.15.2:0/3235663203' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2022-05-13T23:14:18.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:14:17 smithi115 ceph-mon[78593]: from='client.? 172.21.15.2:0/1106603001' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2022-05-13T23:14:18.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:14:17 smithi115 ceph-mon[78593]: pgmap v288: 97 pgs: 10 creating+activating, 12 creating+peering, 51 unknown, 24 active+clean; 577 KiB data, 74 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 3 op/s 2022-05-13T23:14:18.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:14:17 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:14:18.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:14:17 smithi115 ceph-mon[78593]: fsmap cephfs:1 {0=l=up:active} 11 up:standby 2022-05-13T23:14:18.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:14:17 smithi115 ceph-mon[78593]: osdmap e83: 12 total, 12 up, 12 in 2022-05-13T23:14:18.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:14:17 smithi115 ceph-mon[78593]: fsmap cephfs:1 {0=l=up:active} 11 up:standby 2022-05-13T23:14:18.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:14:17 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:14:18.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:14:17 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:14:18.692 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:18.690+0000 7f7e0dffb700 1 -- 172.21.15.2:0/2690804389 <== mon.1 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v17) v1 ==== 118+0+0 (secure 0 0 0) 0x7f7e14090d70 con 0x7f7e180645f0 2022-05-13T23:14:18.695 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:18.693+0000 7f7e1e39d700 1 -- 172.21.15.2:0/2690804389 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f7e04096ce0 msgr2=0x7f7e04099190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:14:18.696 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:18.693+0000 7f7e1e39d700 1 --2- 172.21.15.2:0/2690804389 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f7e04096ce0 0x7f7e04099190 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f7e18064b00 tx=0x7f7e0800b040 comp rx=0 tx=0).stop 2022-05-13T23:14:18.696 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:18.693+0000 7f7e1e39d700 1 -- 172.21.15.2:0/2690804389 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7e180645f0 msgr2=0x7f7e180fba20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:14:18.696 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:18.693+0000 7f7e1e39d700 1 --2- 172.21.15.2:0/2690804389 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7e180645f0 0x7f7e180fba20 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f7e14004710 tx=0x7f7e14009cb0 comp rx=0 tx=0).stop 2022-05-13T23:14:18.696 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:18.693+0000 7f7e1e39d700 1 -- 172.21.15.2:0/2690804389 shutdown_connections 2022-05-13T23:14:18.697 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:18.693+0000 7f7e1e39d700 1 --2- 172.21.15.2:0/2690804389 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f7e18101110 0x7f7e180fa5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:14:18.697 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:18.693+0000 7f7e1e39d700 1 --2- 172.21.15.2:0/2690804389 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7e180645f0 0x7f7e180fba20 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:14:18.697 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:18.693+0000 7f7e1e39d700 1 --2- 172.21.15.2:0/2690804389 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f7e04096ce0 0x7f7e04099190 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:14:18.697 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:18.693+0000 7f7e1e39d700 1 --2- 172.21.15.2:0/2690804389 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7e180fe6c0 0x7f7e180fa080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:14:18.698 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:18.693+0000 7f7e1e39d700 1 -- 172.21.15.2:0/2690804389 >> 172.21.15.2:0/2690804389 conn(0x7f7e180f5b70 msgr2=0x7f7e180fc9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-05-13T23:14:18.698 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:18.693+0000 7f7e1e39d700 1 -- 172.21.15.2:0/2690804389 shutdown_connections 2022-05-13T23:14:18.698 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:18.693+0000 7f7e1e39d700 1 -- 172.21.15.2:0/2690804389 wait complete. 2022-05-13T23:14:18.730 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs max_mds 5 2022-05-13T23:14:18.968 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:18.966+0000 7f2bcb59e700 1 -- 172.21.15.2:0/1127339458 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f2bcc0fd0d0 msgr2=0x7f2bcc103990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:14:18.968 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:18.966+0000 7f2bcb59e700 1 --2- 172.21.15.2:0/1127339458 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f2bcc0fd0d0 0x7f2bcc103990 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f2bc4009c30 tx=0x7f2bc4004c50 comp rx=0 tx=0).stop 2022-05-13T23:14:18.969 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:18.966+0000 7f2bcb59e700 1 -- 172.21.15.2:0/1127339458 shutdown_connections 2022-05-13T23:14:18.969 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:18.966+0000 7f2bcb59e700 1 --2- 172.21.15.2:0/1127339458 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f2bcc0fd0d0 0x7f2bcc103990 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:14:18.969 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:18.966+0000 7f2bcb59e700 1 --2- 172.21.15.2:0/1127339458 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f2bcc0fb4a0 0x7f2bcc0fb8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:14:18.969 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:18.966+0000 7f2bcb59e700 1 --2- 172.21.15.2:0/1127339458 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2bcc0fc740 0x7f2bcc0fcb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:14:18.970 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:18.966+0000 7f2bcb59e700 1 -- 172.21.15.2:0/1127339458 >> 172.21.15.2:0/1127339458 conn(0x7f2bcc0f6c50 msgr2=0x7f2bcc0f90c0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-05-13T23:14:18.970 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:18.966+0000 7f2bcb59e700 1 -- 172.21.15.2:0/1127339458 shutdown_connections 2022-05-13T23:14:18.970 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:18.966+0000 7f2bcb59e700 1 -- 172.21.15.2:0/1127339458 wait complete. 2022-05-13T23:14:18.973 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:18.967+0000 7f2bcb59e700 1 Processor -- start 2022-05-13T23:14:18.973 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:18.968+0000 7f2bcb59e700 1 -- start start 2022-05-13T23:14:18.973 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:18.968+0000 7f2bcb59e700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f2bcc0fb4a0 0x7f2bcc10e360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:14:18.974 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:18.968+0000 7f2bcb59e700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2bcc0fc740 0x7f2bcc10e8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:14:18.974 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:18.968+0000 7f2bcb59e700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f2bcc0fd0d0 0x7f2bcc115930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:14:18.974 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:18.968+0000 7f2bcb59e700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2bcc115e70 con 0x7f2bcc0fc740 2022-05-13T23:14:18.974 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:18.968+0000 7f2bcb59e700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2bcc115fe0 con 0x7f2bcc0fd0d0 2022-05-13T23:14:18.975 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:18.968+0000 7f2bcb59e700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2bcc1161a0 con 0x7f2bcc0fb4a0 2022-05-13T23:14:18.975 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:18.968+0000 7f2bca59c700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f2bcc0fb4a0 0x7f2bcc10e360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:14:18.975 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:18.968+0000 7f2bc9d9b700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2bcc0fc740 0x7f2bcc10e8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:14:18.976 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:18.968+0000 7f2bca59c700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f2bcc0fb4a0 0x7f2bcc10e360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.2:43448/0 (socket says 172.21.15.2:43448) 2022-05-13T23:14:18.976 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:18.968+0000 7f2bca59c700 1 -- 172.21.15.2:0/3355217592 learned_addr learned my addr 172.21.15.2:0/3355217592 (peer_addr_for_me v2:172.21.15.2:0/0) 2022-05-13T23:14:18.976 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:18.968+0000 7f2bcad9d700 1 --2- 172.21.15.2:0/3355217592 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f2bcc0fd0d0 0x7f2bcc115930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:14:18.977 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:18.969+0000 7f2bca59c700 1 -- 172.21.15.2:0/3355217592 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f2bcc0fd0d0 msgr2=0x7f2bcc115930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:14:18.977 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:18.969+0000 7f2bca59c700 1 --2- 172.21.15.2:0/3355217592 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f2bcc0fd0d0 0x7f2bcc115930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:14:18.977 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:18.969+0000 7f2bca59c700 1 -- 172.21.15.2:0/3355217592 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2bcc0fc740 msgr2=0x7f2bcc10e8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:14:18.977 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:18.969+0000 7f2bca59c700 1 --2- 172.21.15.2:0/3355217592 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2bcc0fc740 0x7f2bcc10e8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:14:18.978 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:18.969+0000 7f2bca59c700 1 -- 172.21.15.2:0/3355217592 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2bc0005040 con 0x7f2bcc0fb4a0 2022-05-13T23:14:18.978 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:18.969+0000 7f2bcad9d700 1 --2- 172.21.15.2:0/3355217592 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f2bcc0fd0d0 0x7f2bcc115930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2022-05-13T23:14:18.978 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:18.969+0000 7f2bca59c700 1 --2- 172.21.15.2:0/3355217592 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f2bcc0fb4a0 0x7f2bcc10e360 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f2bbc00e940 tx=0x7f2bbc00b040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-05-13T23:14:18.978 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:18.969+0000 7f2bbaffd700 1 -- 172.21.15.2:0/3355217592 <== mon.2 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2bbc007b40 con 0x7f2bcc0fb4a0 2022-05-13T23:14:18.979 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:18.969+0000 7f2bcb59e700 1 -- 172.21.15.2:0/3355217592 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2bc40097c0 con 0x7f2bcc0fb4a0 2022-05-13T23:14:18.979 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:18.969+0000 7f2bbaffd700 1 -- 172.21.15.2:0/3355217592 <== mon.2 v2:172.21.15.115:3300/0 2 ==== config(26 keys) v1 ==== 1087+0+0 (secure 0 0 0) 0x7f2bbc00f040 con 0x7f2bcc0fb4a0 2022-05-13T23:14:18.979 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:18.969+0000 7f2bbaffd700 1 -- 172.21.15.2:0/3355217592 <== mon.2 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2bbc014890 con 0x7f2bcc0fb4a0 2022-05-13T23:14:18.979 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:18.970+0000 7f2bcb59e700 1 -- 172.21.15.2:0/3355217592 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2bcc116850 con 0x7f2bcc0fb4a0 2022-05-13T23:14:18.980 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:18.971+0000 7f2bcb59e700 1 -- 172.21.15.2:0/3355217592 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2bcc106d40 con 0x7f2bcc0fb4a0 2022-05-13T23:14:18.980 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:14:18 smithi002 ceph-mon[71816]: from='client.? 172.21.15.2:0/2690804389' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2022-05-13T23:14:18.980 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:14:18 smithi002 ceph-mon[71816]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2022-05-13T23:14:18.981 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:18.973+0000 7f2bbaffd700 1 -- 172.21.15.2:0/3355217592 <== mon.2 v2:172.21.15.115:3300/0 4 ==== mgrmap(e 16) v1 ==== 125136+0+0 (secure 0 0 0) 0x7f2bbc01b030 con 0x7f2bcc0fb4a0 2022-05-13T23:14:18.982 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:18.974+0000 7f2bbaffd700 1 --2- 172.21.15.2:0/3355217592 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f2bb4096ce0 0x7f2bb4099190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:14:18.982 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:18.974+0000 7f2bbaffd700 1 -- 172.21.15.2:0/3355217592 <== mon.2 v2:172.21.15.115:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 7975+0+0 (secure 0 0 0) 0x7f2bbc0beb90 con 0x7f2bcc0fb4a0 2022-05-13T23:14:18.982 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:18.975+0000 7f2bc9d9b700 1 --2- 172.21.15.2:0/3355217592 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f2bb4096ce0 0x7f2bb4099190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:14:18.983 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:18.975+0000 7f2bc9d9b700 1 --2- 172.21.15.2:0/3355217592 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f2bb4096ce0 0x7f2bb4099190 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f2bc0004830 tx=0x7f2bc000b040 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-05-13T23:14:18.983 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:18.978+0000 7f2bbaffd700 1 -- 172.21.15.2:0/3355217592 <== mon.2 v2:172.21.15.115:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+165215 (secure 0 0 0) 0x7f2bbc04b570 con 0x7f2bcc0fb4a0 2022-05-13T23:14:19.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:14:18 smithi047 ceph-mon[77147]: from='client.? 172.21.15.2:0/2690804389' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2022-05-13T23:14:19.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:14:18 smithi047 ceph-mon[77147]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2022-05-13T23:14:19.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:14:18 smithi115 ceph-mon[78593]: from='client.? 172.21.15.2:0/2690804389' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2022-05-13T23:14:19.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:14:18 smithi115 ceph-mon[78593]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2022-05-13T23:14:19.219 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:19.218+0000 7f2bcb59e700 1 -- 172.21.15.2:0/3355217592 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"} v 0) v1 -- 0x7f2bcc116a70 con 0x7f2bcc0fb4a0 2022-05-13T23:14:19.285 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:19.284+0000 7f2bbaffd700 1 -- 172.21.15.2:0/3355217592 <== mon.2 v2:172.21.15.115:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]=0 v18) v1 ==== 105+0+0 (secure 0 0 0) 0x7f2bbc08bed0 con 0x7f2bcc0fb4a0 2022-05-13T23:14:19.292 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:19.289+0000 7f2bcb59e700 1 -- 172.21.15.2:0/3355217592 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f2bb4096ce0 msgr2=0x7f2bb4099190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:14:19.292 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:19.289+0000 7f2bcb59e700 1 --2- 172.21.15.2:0/3355217592 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f2bb4096ce0 0x7f2bb4099190 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f2bc0004830 tx=0x7f2bc000b040 comp rx=0 tx=0).stop 2022-05-13T23:14:19.292 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:19.289+0000 7f2bcb59e700 1 -- 172.21.15.2:0/3355217592 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f2bcc0fb4a0 msgr2=0x7f2bcc10e360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:14:19.292 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:19.289+0000 7f2bcb59e700 1 --2- 172.21.15.2:0/3355217592 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f2bcc0fb4a0 0x7f2bcc10e360 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f2bbc00e940 tx=0x7f2bbc00b040 comp rx=0 tx=0).stop 2022-05-13T23:14:19.293 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:19.290+0000 7f2bcb59e700 1 -- 172.21.15.2:0/3355217592 shutdown_connections 2022-05-13T23:14:19.293 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:19.290+0000 7f2bcb59e700 1 --2- 172.21.15.2:0/3355217592 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f2bcc0fd0d0 0x7f2bcc115930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:14:19.293 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:19.290+0000 7f2bcb59e700 1 --2- 172.21.15.2:0/3355217592 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f2bcc0fb4a0 0x7f2bcc10e360 secure :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f2bbc00e940 tx=0x7f2bbc00b040 comp rx=0 tx=0).stop 2022-05-13T23:14:19.294 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:19.290+0000 7f2bcb59e700 1 --2- 172.21.15.2:0/3355217592 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f2bb4096ce0 0x7f2bb4099190 secure :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f2bc0004830 tx=0x7f2bc000b040 comp rx=0 tx=0).stop 2022-05-13T23:14:19.306 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:19.290+0000 7f2bcb59e700 1 --2- 172.21.15.2:0/3355217592 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2bcc0fc740 0x7f2bcc10e8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:14:19.306 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:19.290+0000 7f2bcb59e700 1 -- 172.21.15.2:0/3355217592 >> 172.21.15.2:0/3355217592 conn(0x7f2bcc0f6c50 msgr2=0x7f2bcc101f40 unknown :-1 s=STATE_NONE l=0).mark_down 2022-05-13T23:14:19.308 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:19.291+0000 7f2bcb59e700 1 -- 172.21.15.2:0/3355217592 shutdown_connections 2022-05-13T23:14:19.308 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:19.291+0000 7f2bcb59e700 1 -- 172.21.15.2:0/3355217592 wait complete. 2022-05-13T23:14:19.320 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay true 2022-05-13T23:14:19.550 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:19.548+0000 7fdc6539d700 1 -- 172.21.15.2:0/3385177580 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fdc600fc740 msgr2=0x7fdc600fcb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:14:19.550 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:19.548+0000 7fdc6539d700 1 --2- 172.21.15.2:0/3385177580 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fdc600fc740 0x7fdc600fcb90 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fdc54004870 tx=0x7fdc54009cb0 comp rx=0 tx=0).stop 2022-05-13T23:14:19.551 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:19.548+0000 7fdc6539d700 1 -- 172.21.15.2:0/3385177580 shutdown_connections 2022-05-13T23:14:19.551 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:19.548+0000 7fdc6539d700 1 --2- 172.21.15.2:0/3385177580 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fdc600fd0d0 0x7fdc60103990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:14:19.551 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:19.548+0000 7fdc6539d700 1 --2- 172.21.15.2:0/3385177580 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fdc600fb4a0 0x7fdc600fb8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:14:19.551 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:19.548+0000 7fdc6539d700 1 --2- 172.21.15.2:0/3385177580 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fdc600fc740 0x7fdc600fcb90 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:14:19.552 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:19.548+0000 7fdc6539d700 1 -- 172.21.15.2:0/3385177580 >> 172.21.15.2:0/3385177580 conn(0x7fdc600f6c50 msgr2=0x7fdc600f90c0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-05-13T23:14:19.552 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:19.548+0000 7fdc6539d700 1 -- 172.21.15.2:0/3385177580 shutdown_connections 2022-05-13T23:14:19.553 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:19.548+0000 7fdc6539d700 1 -- 172.21.15.2:0/3385177580 wait complete. 2022-05-13T23:14:19.559 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:19.549+0000 7fdc6539d700 1 Processor -- start 2022-05-13T23:14:19.560 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:19.549+0000 7fdc6539d700 1 -- start start 2022-05-13T23:14:19.560 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:19.550+0000 7fdc6539d700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fdc600fb4a0 0x7fdc6010e4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:14:19.561 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:19.550+0000 7fdc6539d700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fdc600fc740 0x7fdc6010e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:14:19.561 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:19.550+0000 7fdc6539d700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fdc600fd0d0 0x7fdc60115a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:14:19.562 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:19.550+0000 7fdc6539d700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc60115fc0 con 0x7fdc600fb4a0 2022-05-13T23:14:19.562 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:19.550+0000 7fdc6539d700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc60116130 con 0x7fdc600fd0d0 2022-05-13T23:14:19.562 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:19.550+0000 7fdc6539d700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc601162f0 con 0x7fdc600fc740 2022-05-13T23:14:19.563 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:19.550+0000 7fdc5ffff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fdc600fb4a0 0x7fdc6010e4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:14:19.563 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:19.550+0000 7fdc5ffff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fdc600fb4a0 0x7fdc6010e4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:60752/0 (socket says 172.21.15.2:60752) 2022-05-13T23:14:19.564 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:19.550+0000 7fdc5ffff700 1 -- 172.21.15.2:0/955261630 learned_addr learned my addr 172.21.15.2:0/955261630 (peer_addr_for_me v2:172.21.15.2:0/0) 2022-05-13T23:14:19.564 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:19.550+0000 7fdc5f7fe700 1 --2- 172.21.15.2:0/955261630 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fdc600fc740 0x7fdc6010e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:14:19.564 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:19.550+0000 7fdc64b9c700 1 --2- 172.21.15.2:0/955261630 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fdc600fd0d0 0x7fdc60115a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:14:19.565 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:19.550+0000 7fdc5ffff700 1 -- 172.21.15.2:0/955261630 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fdc600fc740 msgr2=0x7fdc6010e9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:14:19.565 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:19.550+0000 7fdc5ffff700 1 --2- 172.21.15.2:0/955261630 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fdc600fc740 0x7fdc6010e9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:14:19.565 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:19.550+0000 7fdc5ffff700 1 -- 172.21.15.2:0/955261630 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fdc600fd0d0 msgr2=0x7fdc60115a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:14:19.566 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:19.550+0000 7fdc5ffff700 1 --2- 172.21.15.2:0/955261630 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fdc600fd0d0 0x7fdc60115a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:14:19.566 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:19.550+0000 7fdc5ffff700 1 -- 172.21.15.2:0/955261630 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc50005040 con 0x7fdc600fb4a0 2022-05-13T23:14:19.566 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:19.550+0000 7fdc5f7fe700 1 --2- 172.21.15.2:0/955261630 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fdc600fc740 0x7fdc6010e9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-05-13T23:14:19.567 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:19.551+0000 7fdc5ffff700 1 --2- 172.21.15.2:0/955261630 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fdc600fb4a0 0x7fdc6010e4b0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fdc500048f0 tx=0x7fdc50009cb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-05-13T23:14:19.567 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:19.551+0000 7fdc5cff9700 1 -- 172.21.15.2:0/955261630 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc50012570 con 0x7fdc600fb4a0 2022-05-13T23:14:19.567 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:19.551+0000 7fdc5cff9700 1 -- 172.21.15.2:0/955261630 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1087+0+0 (secure 0 0 0) 0x7fdc5000f040 con 0x7fdc600fb4a0 2022-05-13T23:14:19.568 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:19.551+0000 7fdc6539d700 1 -- 172.21.15.2:0/955261630 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdc580097c0 con 0x7fdc600fb4a0 2022-05-13T23:14:19.568 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:19.551+0000 7fdc5cff9700 1 -- 172.21.15.2:0/955261630 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc50013b50 con 0x7fdc600fb4a0 2022-05-13T23:14:19.568 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:19.551+0000 7fdc6539d700 1 -- 172.21.15.2:0/955261630 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdc54005040 con 0x7fdc600fb4a0 2022-05-13T23:14:19.569 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:19.552+0000 7fdc6539d700 1 -- 172.21.15.2:0/955261630 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdc60106cf0 con 0x7fdc600fb4a0 2022-05-13T23:14:19.569 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:19.554+0000 7fdc5cff9700 1 -- 172.21.15.2:0/955261630 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 16) v1 ==== 125136+0+0 (secure 0 0 0) 0x7fdc500126d0 con 0x7fdc600fb4a0 2022-05-13T23:14:19.569 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:19.556+0000 7fdc5cff9700 1 --2- 172.21.15.2:0/955261630 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7fdc48096e30 0x7fdc480992e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:14:19.569 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:19.556+0000 7fdc5cff9700 1 -- 172.21.15.2:0/955261630 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 7975+0+0 (secure 0 0 0) 0x7fdc500bf3a0 con 0x7fdc600fb4a0 2022-05-13T23:14:19.570 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:19.556+0000 7fdc5f7fe700 1 --2- 172.21.15.2:0/955261630 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7fdc48096e30 0x7fdc480992e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:14:19.570 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:19.557+0000 7fdc5f7fe700 1 --2- 172.21.15.2:0/955261630 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7fdc48096e30 0x7fdc480992e0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fdc54013490 tx=0x7fdc54009cb0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-05-13T23:14:19.570 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:19.558+0000 7fdc5cff9700 1 -- 172.21.15.2:0/955261630 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+165215 (secure 0 0 0) 0x7fdc5008c8f0 con 0x7fdc600fb4a0 2022-05-13T23:14:19.808 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:19.807+0000 7fdc6539d700 1 -- 172.21.15.2:0/955261630 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"} v 0) v1 -- 0x7fdc60116a40 con 0x7fdc600fb4a0 2022-05-13T23:14:20.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:14:19 smithi047 ceph-mon[77147]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2022-05-13T23:14:20.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:14:19 smithi047 ceph-mon[77147]: fsmap cephfs:1 {0=l=up:active} 11 up:standby 2022-05-13T23:14:20.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:14:19 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:14:20.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:14:19 smithi047 ceph-mon[77147]: from='client.? 172.21.15.2:0/3355217592' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]: dispatch 2022-05-13T23:14:20.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:14:19 smithi047 ceph-mon[77147]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]: dispatch 2022-05-13T23:14:20.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:14:19 smithi047 ceph-mon[77147]: pgmap v290: 97 pgs: 10 creating+activating, 12 creating+peering, 39 unknown, 36 active+clean; 577 KiB data, 75 MiB used, 1.0 TiB / 1.0 TiB avail; 419 B/s wr, 5 op/s 2022-05-13T23:14:20.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:14:19 smithi047 ceph-mon[77147]: daemon mds.i assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2022-05-13T23:14:20.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:14:19 smithi047 ceph-mon[77147]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2022-05-13T23:14:20.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:14:19 smithi047 ceph-mon[77147]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]': finished 2022-05-13T23:14:20.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:14:19 smithi047 ceph-mon[77147]: fsmap cephfs:2 {0=l=up:active,1=i=up:creating} 10 up:standby 2022-05-13T23:14:20.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:14:19 smithi047 ceph-mon[77147]: daemon mds.i is now active in filesystem cephfs as rank 1 2022-05-13T23:14:20.072 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:14:19 smithi002 ceph-mon[71816]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2022-05-13T23:14:20.073 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:14:19 smithi002 ceph-mon[71816]: fsmap cephfs:1 {0=l=up:active} 11 up:standby 2022-05-13T23:14:20.073 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:14:19 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:14:20.073 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:14:19 smithi002 ceph-mon[71816]: from='client.? 172.21.15.2:0/3355217592' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]: dispatch 2022-05-13T23:14:20.073 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:14:19 smithi002 ceph-mon[71816]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]: dispatch 2022-05-13T23:14:20.074 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:14:19 smithi002 ceph-mon[71816]: pgmap v290: 97 pgs: 10 creating+activating, 12 creating+peering, 39 unknown, 36 active+clean; 577 KiB data, 75 MiB used, 1.0 TiB / 1.0 TiB avail; 419 B/s wr, 5 op/s 2022-05-13T23:14:20.074 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:14:19 smithi002 ceph-mon[71816]: daemon mds.i assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2022-05-13T23:14:20.074 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:14:19 smithi002 ceph-mon[71816]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2022-05-13T23:14:20.074 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:14:19 smithi002 ceph-mon[71816]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]': finished 2022-05-13T23:14:20.075 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:14:19 smithi002 ceph-mon[71816]: fsmap cephfs:2 {0=l=up:active,1=i=up:creating} 10 up:standby 2022-05-13T23:14:20.075 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:14:19 smithi002 ceph-mon[71816]: daemon mds.i is now active in filesystem cephfs as rank 1 2022-05-13T23:14:20.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:14:19 smithi115 ceph-mon[78593]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2022-05-13T23:14:20.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:14:19 smithi115 ceph-mon[78593]: fsmap cephfs:1 {0=l=up:active} 11 up:standby 2022-05-13T23:14:20.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:14:19 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:14:20.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:14:19 smithi115 ceph-mon[78593]: from='client.? 172.21.15.2:0/3355217592' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]: dispatch 2022-05-13T23:14:20.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:14:19 smithi115 ceph-mon[78593]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]: dispatch 2022-05-13T23:14:20.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:14:19 smithi115 ceph-mon[78593]: pgmap v290: 97 pgs: 10 creating+activating, 12 creating+peering, 39 unknown, 36 active+clean; 577 KiB data, 75 MiB used, 1.0 TiB / 1.0 TiB avail; 419 B/s wr, 5 op/s 2022-05-13T23:14:20.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:14:19 smithi115 ceph-mon[78593]: daemon mds.i assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2022-05-13T23:14:20.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:14:19 smithi115 ceph-mon[78593]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2022-05-13T23:14:20.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:14:19 smithi115 ceph-mon[78593]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]': finished 2022-05-13T23:14:20.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:14:19 smithi115 ceph-mon[78593]: fsmap cephfs:2 {0=l=up:active,1=i=up:creating} 10 up:standby 2022-05-13T23:14:20.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:14:19 smithi115 ceph-mon[78593]: daemon mds.i is now active in filesystem cephfs as rank 1 2022-05-13T23:14:20.291 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:20.290+0000 7fdc5cff9700 1 -- 172.21.15.2:0/955261630 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]=0 v19) v1 ==== 121+0+0 (secure 0 0 0) 0x7fdc5008ff10 con 0x7fdc600fb4a0 2022-05-13T23:14:20.294 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:20.293+0000 7fdc6539d700 1 -- 172.21.15.2:0/955261630 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7fdc48096e30 msgr2=0x7fdc480992e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:14:20.295 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:20.293+0000 7fdc6539d700 1 --2- 172.21.15.2:0/955261630 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7fdc48096e30 0x7fdc480992e0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fdc54013490 tx=0x7fdc54009cb0 comp rx=0 tx=0).stop 2022-05-13T23:14:20.295 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:20.293+0000 7fdc6539d700 1 -- 172.21.15.2:0/955261630 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fdc600fb4a0 msgr2=0x7fdc6010e4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:14:20.295 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:20.293+0000 7fdc6539d700 1 --2- 172.21.15.2:0/955261630 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fdc600fb4a0 0x7fdc6010e4b0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fdc500048f0 tx=0x7fdc50009cb0 comp rx=0 tx=0).stop 2022-05-13T23:14:20.295 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:20.293+0000 7fdc6539d700 1 -- 172.21.15.2:0/955261630 shutdown_connections 2022-05-13T23:14:20.296 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:20.293+0000 7fdc6539d700 1 --2- 172.21.15.2:0/955261630 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fdc600fd0d0 0x7fdc60115a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:14:20.296 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:20.293+0000 7fdc6539d700 1 --2- 172.21.15.2:0/955261630 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7fdc48096e30 0x7fdc480992e0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:14:20.296 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:20.293+0000 7fdc6539d700 1 --2- 172.21.15.2:0/955261630 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fdc600fb4a0 0x7fdc6010e4b0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:14:20.296 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:20.293+0000 7fdc6539d700 1 --2- 172.21.15.2:0/955261630 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fdc600fc740 0x7fdc6010e9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:14:20.297 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:20.293+0000 7fdc6539d700 1 -- 172.21.15.2:0/955261630 >> 172.21.15.2:0/955261630 conn(0x7fdc600f6c50 msgr2=0x7fdc6006e710 unknown :-1 s=STATE_NONE l=0).mark_down 2022-05-13T23:14:20.297 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:20.293+0000 7fdc6539d700 1 -- 172.21.15.2:0/955261630 shutdown_connections 2022-05-13T23:14:20.297 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:20.293+0000 7fdc6539d700 1 -- 172.21.15.2:0/955261630 wait complete. 2022-05-13T23:14:20.325 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs session_timeout 300 2022-05-13T23:14:20.547 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:20.545+0000 7f2670f9d700 1 -- 172.21.15.2:0/3474909053 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f266c06ed90 msgr2=0x7f266c06f200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:14:20.547 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:20.545+0000 7f2670f9d700 1 --2- 172.21.15.2:0/3474909053 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f266c06ed90 0x7f266c06f200 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f2660004870 tx=0x7f2660009cb0 comp rx=0 tx=0).stop 2022-05-13T23:14:20.547 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:20.545+0000 7f2670f9d700 1 -- 172.21.15.2:0/3474909053 shutdown_connections 2022-05-13T23:14:20.548 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:20.545+0000 7f2670f9d700 1 --2- 172.21.15.2:0/3474909053 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f266c103430 0x7f266c105810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:14:20.548 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:20.545+0000 7f2670f9d700 1 --2- 172.21.15.2:0/3474909053 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f266c070350 0x7f266c06e850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:14:20.548 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:20.545+0000 7f2670f9d700 1 --2- 172.21.15.2:0/3474909053 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f266c06ed90 0x7f266c06f200 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:14:20.548 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:20.545+0000 7f2670f9d700 1 -- 172.21.15.2:0/3474909053 >> 172.21.15.2:0/3474909053 conn(0x7f266c0f6c50 msgr2=0x7f266c0f90a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-05-13T23:14:20.549 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:20.545+0000 7f2670f9d700 1 -- 172.21.15.2:0/3474909053 shutdown_connections 2022-05-13T23:14:20.549 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:20.545+0000 7f2670f9d700 1 -- 172.21.15.2:0/3474909053 wait complete. 2022-05-13T23:14:20.549 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:20.546+0000 7f2670f9d700 1 Processor -- start 2022-05-13T23:14:20.552 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:20.546+0000 7f2670f9d700 1 -- start start 2022-05-13T23:14:20.552 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:20.546+0000 7f2670f9d700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f266c06ed90 0x7f266c10e4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:14:20.552 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:20.546+0000 7f2670f9d700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f266c070350 0x7f266c10ea10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:14:20.552 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:20.546+0000 7f2670f9d700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f266c103430 0x7f266c115aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:14:20.553 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:20.546+0000 7f2670f9d700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f266c115fe0 con 0x7f266c103430 2022-05-13T23:14:20.553 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:20.546+0000 7f2670f9d700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f266c116150 con 0x7f266c070350 2022-05-13T23:14:20.553 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:20.546+0000 7f2670f9d700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f266c116310 con 0x7f266c06ed90 2022-05-13T23:14:20.553 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:20.547+0000 7f266b7fe700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f266c06ed90 0x7f266c10e4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:14:20.554 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:20.547+0000 7f266affd700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f266c070350 0x7f266c10ea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:14:20.554 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:20.547+0000 7f266affd700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f266c070350 0x7f266c10ea10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.2:39102/0 (socket says 172.21.15.2:39102) 2022-05-13T23:14:20.554 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:20.547+0000 7f266b7fe700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f266c06ed90 0x7f266c10e4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.2:56674/0 (socket says 172.21.15.2:56674) 2022-05-13T23:14:20.554 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:20.547+0000 7f266affd700 1 -- 172.21.15.2:0/2173314192 learned_addr learned my addr 172.21.15.2:0/2173314192 (peer_addr_for_me v2:172.21.15.2:0/0) 2022-05-13T23:14:20.554 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:20.547+0000 7f266bfff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f266c103430 0x7f266c115aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:14:20.555 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:20.547+0000 7f266b7fe700 1 -- 172.21.15.2:0/2173314192 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f266c070350 msgr2=0x7f266c10ea10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:14:20.555 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:20.547+0000 7f266b7fe700 1 --2- 172.21.15.2:0/2173314192 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f266c070350 0x7f266c10ea10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:14:20.555 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:20.547+0000 7f266b7fe700 1 -- 172.21.15.2:0/2173314192 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f266c103430 msgr2=0x7f266c115aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:14:20.555 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:20.547+0000 7f266b7fe700 1 --2- 172.21.15.2:0/2173314192 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f266c103430 0x7f266c115aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:14:20.556 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:20.547+0000 7f266b7fe700 1 -- 172.21.15.2:0/2173314192 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2660005040 con 0x7f266c06ed90 2022-05-13T23:14:20.556 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:20.547+0000 7f266bfff700 1 --2- 172.21.15.2:0/2173314192 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f266c103430 0x7f266c115aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-05-13T23:14:20.556 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:20.548+0000 7f266b7fe700 1 --2- 172.21.15.2:0/2173314192 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f266c06ed90 0x7f266c10e4d0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f265c00e8f0 tx=0x7f265c00b040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-05-13T23:14:20.556 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:20.548+0000 7f2653fff700 1 -- 172.21.15.2:0/2173314192 <== mon.2 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f265c007b40 con 0x7f266c06ed90 2022-05-13T23:14:20.556 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:20.548+0000 7f2670f9d700 1 -- 172.21.15.2:0/2173314192 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f266c1165f0 con 0x7f266c06ed90 2022-05-13T23:14:20.557 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:20.548+0000 7f2653fff700 1 -- 172.21.15.2:0/2173314192 <== mon.2 v2:172.21.15.115:3300/0 2 ==== config(26 keys) v1 ==== 1087+0+0 (secure 0 0 0) 0x7f265c00f040 con 0x7f266c06ed90 2022-05-13T23:14:20.557 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:20.548+0000 7f2653fff700 1 -- 172.21.15.2:0/2173314192 <== mon.2 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f265c014950 con 0x7f266c06ed90 2022-05-13T23:14:20.557 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:20.548+0000 7f2670f9d700 1 -- 172.21.15.2:0/2173314192 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f266c116bb0 con 0x7f266c06ed90 2022-05-13T23:14:20.557 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:20.549+0000 7f2670f9d700 1 -- 172.21.15.2:0/2173314192 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f266c108eb0 con 0x7f266c06ed90 2022-05-13T23:14:20.558 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:20.551+0000 7f2653fff700 1 -- 172.21.15.2:0/2173314192 <== mon.2 v2:172.21.15.115:3300/0 4 ==== mgrmap(e 16) v1 ==== 125136+0+0 (secure 0 0 0) 0x7f265c01b030 con 0x7f266c06ed90 2022-05-13T23:14:20.559 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:20.552+0000 7f2653fff700 1 --2- 172.21.15.2:0/2173314192 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f2654096ce0 0x7f2654099190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:14:20.559 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:20.552+0000 7f2653fff700 1 -- 172.21.15.2:0/2173314192 <== mon.2 v2:172.21.15.115:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 7975+0+0 (secure 0 0 0) 0x7f265c0bec50 con 0x7f266c06ed90 2022-05-13T23:14:20.559 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:20.553+0000 7f266affd700 1 --2- 172.21.15.2:0/2173314192 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f2654096ce0 0x7f2654099190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:14:20.559 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:20.553+0000 7f266affd700 1 --2- 172.21.15.2:0/2173314192 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f2654096ce0 0x7f2654099190 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f26600045a0 tx=0x7f2660009cb0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-05-13T23:14:20.560 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:20.554+0000 7f2653fff700 1 -- 172.21.15.2:0/2173314192 <== mon.2 v2:172.21.15.115:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+165215 (secure 0 0 0) 0x7f265c08bf90 con 0x7f266c06ed90 2022-05-13T23:14:20.608 INFO:journalctl@ceph.mds.l.smithi115.stdout:May 13 23:14:20 smithi115 conmon[107138]: 2022-05-13T23:14:20.317+0000 7f8df15c1700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2022-05-13T23:14:20.608 INFO:journalctl@ceph.mds.l.smithi115.stdout:May 13 23:14:20 smithi115 conmon[107138]: 2022-05-13T23:14:20.317+0000 7f8df15c1700 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2022-05-13T23:14:20.790 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:20.789+0000 7f2670f9d700 1 -- 172.21.15.2:0/2173314192 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"} v 0) v1 -- 0x7f266c101a00 con 0x7f266c06ed90 2022-05-13T23:14:21.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:14:20 smithi047 ceph-mon[77147]: from='client.? 172.21.15.2:0/955261630' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2022-05-13T23:14:21.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:14:20 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:14:21.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:14:20 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6836/2940957881,v1:172.21.15.115:6837/2940957881] up:active 2022-05-13T23:14:21.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:14:20 smithi047 ceph-mon[77147]: from='client.? 172.21.15.2:0/955261630' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2022-05-13T23:14:21.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:14:20 smithi047 ceph-mon[77147]: daemon mds.f assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2022-05-13T23:14:21.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:14:20 smithi047 ceph-mon[77147]: fsmap cephfs:2 {0=l=up:active,1=i=up:active} 10 up:standby 2022-05-13T23:14:21.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:14:20 smithi047 ceph-mon[77147]: fsmap cephfs:3 {0=l=up:active,1=i=up:active,2=f=up:creating} 9 up:standby 2022-05-13T23:14:21.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:14:20 smithi047 ceph-mon[77147]: daemon mds.f is now active in filesystem cephfs as rank 2 2022-05-13T23:14:21.072 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:14:20 smithi002 ceph-mon[71816]: from='client.? 172.21.15.2:0/955261630' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2022-05-13T23:14:21.073 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:14:20 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:14:21.073 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:14:20 smithi002 ceph-mon[71816]: mds.? [v2:172.21.15.115:6836/2940957881,v1:172.21.15.115:6837/2940957881] up:active 2022-05-13T23:14:21.073 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:14:20 smithi002 ceph-mon[71816]: from='client.? 172.21.15.2:0/955261630' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2022-05-13T23:14:21.073 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:14:20 smithi002 ceph-mon[71816]: daemon mds.f assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2022-05-13T23:14:21.074 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:14:20 smithi002 ceph-mon[71816]: fsmap cephfs:2 {0=l=up:active,1=i=up:active} 10 up:standby 2022-05-13T23:14:21.074 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:14:20 smithi002 ceph-mon[71816]: fsmap cephfs:3 {0=l=up:active,1=i=up:active,2=f=up:creating} 9 up:standby 2022-05-13T23:14:21.074 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:14:20 smithi002 ceph-mon[71816]: daemon mds.f is now active in filesystem cephfs as rank 2 2022-05-13T23:14:21.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:14:20 smithi115 ceph-mon[78593]: from='client.? 172.21.15.2:0/955261630' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2022-05-13T23:14:21.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:14:20 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:14:21.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:14:20 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6836/2940957881,v1:172.21.15.115:6837/2940957881] up:active 2022-05-13T23:14:21.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:14:20 smithi115 ceph-mon[78593]: from='client.? 172.21.15.2:0/955261630' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2022-05-13T23:14:21.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:14:20 smithi115 ceph-mon[78593]: daemon mds.f assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2022-05-13T23:14:21.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:14:20 smithi115 ceph-mon[78593]: fsmap cephfs:2 {0=l=up:active,1=i=up:active} 10 up:standby 2022-05-13T23:14:21.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:14:20 smithi115 ceph-mon[78593]: fsmap cephfs:3 {0=l=up:active,1=i=up:active,2=f=up:creating} 9 up:standby 2022-05-13T23:14:21.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:14:20 smithi115 ceph-mon[78593]: daemon mds.f is now active in filesystem cephfs as rank 2 2022-05-13T23:14:21.298 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:21.297+0000 7f2653fff700 1 -- 172.21.15.2:0/2173314192 <== mon.2 v2:172.21.15.115:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]=0 v21) v1 ==== 115+0+0 (secure 0 0 0) 0x7f265c0190b0 con 0x7f266c06ed90 2022-05-13T23:14:21.302 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:21.300+0000 7f2670f9d700 1 -- 172.21.15.2:0/2173314192 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f2654096ce0 msgr2=0x7f2654099190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:14:21.302 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:21.300+0000 7f2670f9d700 1 --2- 172.21.15.2:0/2173314192 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f2654096ce0 0x7f2654099190 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f26600045a0 tx=0x7f2660009cb0 comp rx=0 tx=0).stop 2022-05-13T23:14:21.302 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:21.300+0000 7f2670f9d700 1 -- 172.21.15.2:0/2173314192 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f266c06ed90 msgr2=0x7f266c10e4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:14:21.302 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:21.300+0000 7f2670f9d700 1 --2- 172.21.15.2:0/2173314192 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f266c06ed90 0x7f266c10e4d0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f265c00e8f0 tx=0x7f265c00b040 comp rx=0 tx=0).stop 2022-05-13T23:14:21.303 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:21.300+0000 7f2670f9d700 1 -- 172.21.15.2:0/2173314192 shutdown_connections 2022-05-13T23:14:21.303 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:21.300+0000 7f2670f9d700 1 --2- 172.21.15.2:0/2173314192 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f2654096ce0 0x7f2654099190 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:14:21.303 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:21.300+0000 7f2670f9d700 1 --2- 172.21.15.2:0/2173314192 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f266c103430 0x7f266c115aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:14:21.303 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:21.300+0000 7f2670f9d700 1 --2- 172.21.15.2:0/2173314192 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f266c06ed90 0x7f266c10e4d0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:14:21.304 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:21.300+0000 7f2670f9d700 1 --2- 172.21.15.2:0/2173314192 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f266c070350 0x7f266c10ea10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:14:21.304 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:21.300+0000 7f2670f9d700 1 -- 172.21.15.2:0/2173314192 >> 172.21.15.2:0/2173314192 conn(0x7f266c0f6c50 msgr2=0x7f266c0f9030 unknown :-1 s=STATE_NONE l=0).mark_down 2022-05-13T23:14:21.304 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:21.301+0000 7f2670f9d700 1 -- 172.21.15.2:0/2173314192 shutdown_connections 2022-05-13T23:14:21.304 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:21.301+0000 7f2670f9d700 1 -- 172.21.15.2:0/2173314192 wait complete. 2022-05-13T23:14:21.335 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-05-13T23:14:21.555 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:21.553+0000 7f1791f9d700 1 -- 172.21.15.2:0/3487301491 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f178c0fc730 msgr2=0x7f178c0fcb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:14:21.556 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:21.553+0000 7f1791f9d700 1 --2- 172.21.15.2:0/3487301491 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f178c0fc730 0x7f178c0fcb80 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f1778004870 tx=0x7f1778009cb0 comp rx=0 tx=0).stop 2022-05-13T23:14:21.556 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:21.553+0000 7f1791f9d700 1 -- 172.21.15.2:0/3487301491 shutdown_connections 2022-05-13T23:14:21.556 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:21.553+0000 7f1791f9d700 1 --2- 172.21.15.2:0/3487301491 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f178c0fd0c0 0x7f178c103980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:14:21.557 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:21.553+0000 7f1791f9d700 1 --2- 172.21.15.2:0/3487301491 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f178c0fb490 0x7f178c0fb8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:14:21.557 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:21.553+0000 7f1791f9d700 1 --2- 172.21.15.2:0/3487301491 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f178c0fc730 0x7f178c0fcb80 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:14:21.557 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:21.553+0000 7f1791f9d700 1 -- 172.21.15.2:0/3487301491 >> 172.21.15.2:0/3487301491 conn(0x7f178c0f6c40 msgr2=0x7f178c0f90b0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-05-13T23:14:21.557 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:21.554+0000 7f1791f9d700 1 -- 172.21.15.2:0/3487301491 shutdown_connections 2022-05-13T23:14:21.557 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:21.554+0000 7f1791f9d700 1 -- 172.21.15.2:0/3487301491 wait complete. 2022-05-13T23:14:21.558 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:21.554+0000 7f1791f9d700 1 Processor -- start 2022-05-13T23:14:21.558 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:21.555+0000 7f1791f9d700 1 -- start start 2022-05-13T23:14:21.560 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:21.555+0000 7f1791f9d700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f178c0fb490 0x7f178c10c2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:14:21.560 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:21.555+0000 7f1791f9d700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f178c0fc730 0x7f178c10c800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:14:21.561 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:21.555+0000 7f1791f9d700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f178c0fd0c0 0x7f178c113890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:14:21.561 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:21.555+0000 7f1791f9d700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f178c113dd0 con 0x7f178c0fb490 2022-05-13T23:14:21.561 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:21.555+0000 7f1791f9d700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f178c113f40 con 0x7f178c0fc730 2022-05-13T23:14:21.561 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:21.555+0000 7f1791f9d700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f178c114100 con 0x7f178c0fd0c0 2022-05-13T23:14:21.562 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:21.555+0000 7f1790f9b700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f178c0fb490 0x7f178c10c2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:14:21.562 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:21.555+0000 7f1790f9b700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f178c0fb490 0x7f178c10c2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:47192/0 (socket says 172.21.15.2:47192) 2022-05-13T23:14:21.562 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:21.555+0000 7f1790f9b700 1 -- 172.21.15.2:0/3025885018 learned_addr learned my addr 172.21.15.2:0/3025885018 (peer_addr_for_me v2:172.21.15.2:0/0) 2022-05-13T23:14:21.562 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:21.555+0000 7f178bfff700 1 --2- 172.21.15.2:0/3025885018 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f178c0fc730 0x7f178c10c800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:14:21.562 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:21.555+0000 7f179179c700 1 --2- 172.21.15.2:0/3025885018 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f178c0fd0c0 0x7f178c113890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:14:21.563 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:21.556+0000 7f178bfff700 1 -- 172.21.15.2:0/3025885018 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f178c0fd0c0 msgr2=0x7f178c113890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:14:21.563 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:21.556+0000 7f178bfff700 1 --2- 172.21.15.2:0/3025885018 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f178c0fd0c0 0x7f178c113890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:14:21.563 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:21.556+0000 7f178bfff700 1 -- 172.21.15.2:0/3025885018 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f178c0fb490 msgr2=0x7f178c10c2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:14:21.563 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:21.556+0000 7f178bfff700 1 --2- 172.21.15.2:0/3025885018 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f178c0fb490 0x7f178c10c2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:14:21.563 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:21.556+0000 7f178bfff700 1 -- 172.21.15.2:0/3025885018 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1778005040 con 0x7f178c0fc730 2022-05-13T23:14:21.564 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:21.556+0000 7f178bfff700 1 --2- 172.21.15.2:0/3025885018 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f178c0fc730 0x7f178c10c800 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f1778004710 tx=0x7f1778009cb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-05-13T23:14:21.564 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:21.556+0000 7f17897fa700 1 -- 172.21.15.2:0/3025885018 <== mon.1 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1778020070 con 0x7f178c0fc730 2022-05-13T23:14:21.564 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:21.556+0000 7f17897fa700 1 -- 172.21.15.2:0/3025885018 <== mon.1 v2:172.21.15.47:3300/0 2 ==== config(26 keys) v1 ==== 1087+0+0 (secure 0 0 0) 0x7f1778004200 con 0x7f178c0fc730 2022-05-13T23:14:21.564 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:21.556+0000 7f1791f9d700 1 -- 172.21.15.2:0/3025885018 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f178c114380 con 0x7f178c0fc730 2022-05-13T23:14:21.565 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:21.557+0000 7f17897fa700 1 -- 172.21.15.2:0/3025885018 <== mon.1 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f177800e040 con 0x7f178c0fc730 2022-05-13T23:14:21.565 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:21.557+0000 7f1791f9d700 1 -- 172.21.15.2:0/3025885018 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f178c114800 con 0x7f178c0fc730 2022-05-13T23:14:21.565 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:21.558+0000 7f1791f9d700 1 -- 172.21.15.2:0/3025885018 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f178c106d30 con 0x7f178c0fc730 2022-05-13T23:14:21.566 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:21.560+0000 7f17897fa700 1 -- 172.21.15.2:0/3025885018 <== mon.1 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 16) v1 ==== 125136+0+0 (secure 0 0 0) 0x7f1778009240 con 0x7f178c0fc730 2022-05-13T23:14:21.566 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:21.561+0000 7f17897fa700 1 --2- 172.21.15.2:0/3025885018 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f177c096ce0 0x7f177c099190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:14:21.567 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:21.561+0000 7f17897fa700 1 -- 172.21.15.2:0/3025885018 <== mon.1 v2:172.21.15.47:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 7975+0+0 (secure 0 0 0) 0x7f17780c0bc0 con 0x7f178c0fc730 2022-05-13T23:14:21.567 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:21.561+0000 7f1790f9b700 1 --2- 172.21.15.2:0/3025885018 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f177c096ce0 0x7f177c099190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:14:21.567 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:21.562+0000 7f1790f9b700 1 --2- 172.21.15.2:0/3025885018 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f177c096ce0 0x7f177c099190 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f17800041d0 tx=0x7f178000b040 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-05-13T23:14:21.567 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:21.563+0000 7f17897fa700 1 -- 172.21.15.2:0/3025885018 <== mon.1 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+165215 (secure 0 0 0) 0x7f177808df00 con 0x7f178c0fc730 2022-05-13T23:14:21.802 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:21.800+0000 7f1791f9d700 1 -- 172.21.15.2:0/3025885018 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f178c02a5b0 con 0x7f178c0fc730 2022-05-13T23:14:21.803 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:21.801+0000 7f17897fa700 1 -- 172.21.15.2:0/3025885018 <== mon.1 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) v1 ==== 94+0+10696 (secure 0 0 0) 0x7f1778091520 con 0x7f178c0fc730 2022-05-13T23:14:21.803 INFO:teuthology.orchestra.run.smithi002.stdout: 2022-05-13T23:14:21.803 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24460,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.2:6835/1341618941","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6834","nonce":1341618941},{"type":"v1","addr":"172.21.15.2:6835","nonce":1341618941}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24490,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.2:6839/1347050951","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6838","nonce":1347050951},{"type":"v1","addr":"172.21.15.2:6839","nonce":1347050951}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":24494,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.2:6837/3754333215","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6836","nonce":3754333215},{"type":"v1","addr":"172.21.15.2:6837","nonce":3754333215}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24511,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.47:6833/2783249369","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6832","nonce":2783249369},{"type":"v1","addr":"172.21.15.47:6833","nonce":2783249369}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":6},{"gid":24523,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.47:6835/4075838054","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6834","nonce":4075838054},{"type":"v1","addr":"172.21.15.47:6835","nonce":4075838054}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":7},{"gid":24530,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.2:6841/522559555","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6840","nonce":522559555},{"type":"v1","addr":"172.21.15.2:6841","nonce":522559555}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":24535,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.47:6837/4072863000","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6836","nonce":4072863000},{"type":"v1","addr":"172.21.15.47:6837","nonce":4072863000}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":8},{"gid":24547,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.47:6839/761355251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6838","nonce":761355251},{"type":"v1","addr":"172.21.15.47:6839","nonce":761355251}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":9}],"filesystems":[{"mdsmap":{"epoch":22,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2022-05-13T23:14:16.236645+0000","modified":"2022-05-13T23:14:21.297221+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3],"up":{"mds_0":24611,"mds_1":24599,"mds_2":24587,"mds_3":24575},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24575":{"gid":24575,"name":"c","rank":3,"incarnation":22,"state":"up:creating","state_seq":1,"addr":"172.21.15.115:6833/381933929","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6832","nonce":381933929},{"type":"v1","addr":"172.21.15.115:6833","nonce":381933929}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24587":{"gid":24587,"name":"f","rank":2,"incarnation":20,"state":"up:active","state_seq":5,"addr":"172.21.15.115:6835/71062276","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6834","nonce":71062276},{"type":"v1","addr":"172.21.15.115:6835","nonce":71062276}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24599":{"gid":24599,"name":"i","rank":1,"incarnation":18,"state":"up:active","state_seq":4,"addr":"172.21.15.115:6837/2940957881","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6836","nonce":2940957881},{"type":"v1","addr":"172.21.15.115:6837","nonce":2940957881}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24611":{"gid":24611,"name":"l","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.115:6839/4149542923","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6838","nonce":4149542923},{"type":"v1","addr":"172.21.15.115:6839","nonce":4149542923}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-05-13T23:14:21.805 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:21.804+0000 7f1791f9d700 1 -- 172.21.15.2:0/3025885018 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f177c096ce0 msgr2=0x7f177c099190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:14:21.806 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:21.804+0000 7f1791f9d700 1 --2- 172.21.15.2:0/3025885018 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f177c096ce0 0x7f177c099190 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f17800041d0 tx=0x7f178000b040 comp rx=0 tx=0).stop 2022-05-13T23:14:21.806 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:21.804+0000 7f1791f9d700 1 -- 172.21.15.2:0/3025885018 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f178c0fc730 msgr2=0x7f178c10c800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:14:21.806 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:21.804+0000 7f1791f9d700 1 --2- 172.21.15.2:0/3025885018 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f178c0fc730 0x7f178c10c800 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f1778004710 tx=0x7f1778009cb0 comp rx=0 tx=0).stop 2022-05-13T23:14:21.806 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:21.804+0000 7f1791f9d700 1 -- 172.21.15.2:0/3025885018 shutdown_connections 2022-05-13T23:14:21.807 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:21.804+0000 7f1791f9d700 1 --2- 172.21.15.2:0/3025885018 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f178c0fd0c0 0x7f178c113890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:14:21.807 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:21.804+0000 7f1791f9d700 1 --2- 172.21.15.2:0/3025885018 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f177c096ce0 0x7f177c099190 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:14:21.807 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:21.804+0000 7f1791f9d700 1 --2- 172.21.15.2:0/3025885018 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f178c0fb490 0x7f178c10c2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:14:21.807 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:21.804+0000 7f1791f9d700 1 --2- 172.21.15.2:0/3025885018 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f178c0fc730 0x7f178c10c800 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:14:21.807 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:21.804+0000 7f1791f9d700 1 -- 172.21.15.2:0/3025885018 >> 172.21.15.2:0/3025885018 conn(0x7f178c0f6c40 msgr2=0x7f178c0f9000 unknown :-1 s=STATE_NONE l=0).mark_down 2022-05-13T23:14:21.808 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:21.804+0000 7f1791f9d700 1 -- 172.21.15.2:0/3025885018 shutdown_connections 2022-05-13T23:14:21.808 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:21.804+0000 7f1791f9d700 1 -- 172.21.15.2:0/3025885018 wait complete. 2022-05-13T23:14:21.808 INFO:teuthology.orchestra.run.smithi002.stderr:dumped fsmap epoch 22 2022-05-13T23:14:21.832 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-05-13T23:14:22.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:14:21 smithi047 ceph-mon[77147]: from='client.? 172.21.15.2:0/2173314192' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2022-05-13T23:14:22.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:14:21 smithi047 ceph-mon[77147]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2022-05-13T23:14:22.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:14:21 smithi047 ceph-mon[77147]: pgmap v291: 97 pgs: 8 creating+activating, 89 active+clean; 581 KiB data, 76 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s wr, 12 op/s 2022-05-13T23:14:22.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:14:21 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6834/71062276,v1:172.21.15.115:6835/71062276] up:active 2022-05-13T23:14:22.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:14:21 smithi047 ceph-mon[77147]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2022-05-13T23:14:22.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:14:21 smithi047 ceph-mon[77147]: daemon mds.c assigned to filesystem cephfs as rank 3 (now has 4 ranks) 2022-05-13T23:14:22.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:14:21 smithi047 ceph-mon[77147]: fsmap cephfs:3 {0=l=up:active,1=i=up:active,2=f=up:active} 9 up:standby 2022-05-13T23:14:22.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:14:21 smithi047 ceph-mon[77147]: fsmap cephfs:4 {0=l=up:active,1=i=up:active,2=f=up:active,3=c=up:creating} 8 up:standby 2022-05-13T23:14:22.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:14:21 smithi047 ceph-mon[77147]: daemon mds.c is now active in filesystem cephfs as rank 3 2022-05-13T23:14:22.052 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:22.050+0000 7f9dfe79d700 1 -- 172.21.15.2:0/2938159221 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f9df80fd750 msgr2=0x7f9df80fdb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:14:22.052 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:22.050+0000 7f9dfe79d700 1 --2- 172.21.15.2:0/2938159221 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f9df80fd750 0x7f9df80fdb60 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f9df4004870 tx=0x7f9df4009cb0 comp rx=0 tx=0).stop 2022-05-13T23:14:22.052 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:22.050+0000 7f9dfe79d700 1 -- 172.21.15.2:0/2938159221 shutdown_connections 2022-05-13T23:14:22.052 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:22.050+0000 7f9dfe79d700 1 --2- 172.21.15.2:0/2938159221 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f9df80ff380 0x7f9df8105c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:14:22.053 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:22.050+0000 7f9dfe79d700 1 --2- 172.21.15.2:0/2938159221 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f9df80fd750 0x7f9df80fdb60 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:14:22.053 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:22.050+0000 7f9dfe79d700 1 --2- 172.21.15.2:0/2938159221 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9df80fe9f0 0x7f9df80fee40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:14:22.053 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:22.050+0000 7f9dfe79d700 1 -- 172.21.15.2:0/2938159221 >> 172.21.15.2:0/2938159221 conn(0x7f9df80f8e70 msgr2=0x7f9df80fb2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-05-13T23:14:22.053 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:22.050+0000 7f9dfe79d700 1 -- 172.21.15.2:0/2938159221 shutdown_connections 2022-05-13T23:14:22.054 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:22.050+0000 7f9dfe79d700 1 -- 172.21.15.2:0/2938159221 wait complete. 2022-05-13T23:14:22.054 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:22.051+0000 7f9dfe79d700 1 Processor -- start 2022-05-13T23:14:22.057 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:22.051+0000 7f9dfe79d700 1 -- start start 2022-05-13T23:14:22.057 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:22.051+0000 7f9dfe79d700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9df80fd750 0x7f9df810e4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:14:22.057 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:22.051+0000 7f9dfe79d700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f9df80fe9f0 0x7f9df810ea20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:14:22.057 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:22.051+0000 7f9dfe79d700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f9df80ff380 0x7f9df8115ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:14:22.058 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:22.051+0000 7f9dfe79d700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9df8115ff0 con 0x7f9df80fd750 2022-05-13T23:14:22.058 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:22.051+0000 7f9dfe79d700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9df8116160 con 0x7f9df80fe9f0 2022-05-13T23:14:22.058 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:22.051+0000 7f9dfe79d700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9df8116320 con 0x7f9df80ff380 2022-05-13T23:14:22.058 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:22.051+0000 7f9dfd79b700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9df80fd750 0x7f9df810e4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:14:22.059 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:22.052+0000 7f9dfd79b700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9df80fd750 0x7f9df810e4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:47196/0 (socket says 172.21.15.2:47196) 2022-05-13T23:14:22.059 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:22.052+0000 7f9dfd79b700 1 -- 172.21.15.2:0/2765872222 learned_addr learned my addr 172.21.15.2:0/2765872222 (peer_addr_for_me v2:172.21.15.2:0/0) 2022-05-13T23:14:22.059 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:22.052+0000 7f9dfdf9c700 1 --2- 172.21.15.2:0/2765872222 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f9df80ff380 0x7f9df8115ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:14:22.060 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:22.052+0000 7f9dfcf9a700 1 --2- 172.21.15.2:0/2765872222 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f9df80fe9f0 0x7f9df810ea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:14:22.060 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:22.052+0000 7f9dfd79b700 1 -- 172.21.15.2:0/2765872222 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f9df80ff380 msgr2=0x7f9df8115ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:14:22.060 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:22.052+0000 7f9dfd79b700 1 --2- 172.21.15.2:0/2765872222 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f9df80ff380 0x7f9df8115ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:14:22.060 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:22.052+0000 7f9dfd79b700 1 -- 172.21.15.2:0/2765872222 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f9df80fe9f0 msgr2=0x7f9df810ea20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:14:22.060 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:22.052+0000 7f9dfd79b700 1 --2- 172.21.15.2:0/2765872222 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f9df80fe9f0 0x7f9df810ea20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:14:22.061 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:22.052+0000 7f9dfd79b700 1 -- 172.21.15.2:0/2765872222 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9df4005040 con 0x7f9df80fd750 2022-05-13T23:14:22.061 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:22.052+0000 7f9dfd79b700 1 --2- 172.21.15.2:0/2765872222 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9df80fd750 0x7f9df810e4e0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f9df4004710 tx=0x7f9df4009cb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-05-13T23:14:22.061 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:22.053+0000 7f9dedffb700 1 -- 172.21.15.2:0/2765872222 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9df400e040 con 0x7f9df80fd750 2022-05-13T23:14:22.061 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:22.053+0000 7f9dfe79d700 1 -- 172.21.15.2:0/2765872222 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9df81165a0 con 0x7f9df80fd750 2022-05-13T23:14:22.062 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:22.053+0000 7f9dedffb700 1 -- 172.21.15.2:0/2765872222 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1087+0+0 (secure 0 0 0) 0x7f9df4004200 con 0x7f9df80fd750 2022-05-13T23:14:22.062 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:22.053+0000 7f9dfe79d700 1 -- 172.21.15.2:0/2765872222 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9df8116a80 con 0x7f9df80fd750 2022-05-13T23:14:22.062 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:22.053+0000 7f9dedffb700 1 -- 172.21.15.2:0/2765872222 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9df401ca90 con 0x7f9df80fd750 2022-05-13T23:14:22.062 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:22.054+0000 7f9dfe79d700 1 -- 172.21.15.2:0/2765872222 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9df8108f40 con 0x7f9df80fd750 2022-05-13T23:14:22.062 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:22.056+0000 7f9dedffb700 1 -- 172.21.15.2:0/2765872222 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 16) v1 ==== 125136+0+0 (secure 0 0 0) 0x7f9df4009240 con 0x7f9df80fd750 2022-05-13T23:14:22.063 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:22.057+0000 7f9dedffb700 1 --2- 172.21.15.2:0/2765872222 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f9de4096ce0 0x7f9de4099190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:14:22.064 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:22.057+0000 7f9dedffb700 1 -- 172.21.15.2:0/2765872222 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 7975+0+0 (secure 0 0 0) 0x7f9df40c0e90 con 0x7f9df80fd750 2022-05-13T23:14:22.064 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:22.057+0000 7f9dfcf9a700 1 --2- 172.21.15.2:0/2765872222 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f9de4096ce0 0x7f9de4099190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:14:22.064 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:22.058+0000 7f9dfcf9a700 1 --2- 172.21.15.2:0/2765872222 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f9de4096ce0 0x7f9de4099190 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f9de80048b0 tx=0x7f9de800b040 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-05-13T23:14:22.064 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:22.059+0000 7f9dedffb700 1 -- 172.21.15.2:0/2765872222 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+165215 (secure 0 0 0) 0x7f9df408e2c0 con 0x7f9df80fd750 2022-05-13T23:14:22.072 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:14:21 smithi002 ceph-mon[71816]: from='client.? 172.21.15.2:0/2173314192' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2022-05-13T23:14:22.072 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:14:21 smithi002 ceph-mon[71816]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2022-05-13T23:14:22.073 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:14:21 smithi002 ceph-mon[71816]: pgmap v291: 97 pgs: 8 creating+activating, 89 active+clean; 581 KiB data, 76 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s wr, 12 op/s 2022-05-13T23:14:22.073 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:14:21 smithi002 ceph-mon[71816]: mds.? [v2:172.21.15.115:6834/71062276,v1:172.21.15.115:6835/71062276] up:active 2022-05-13T23:14:22.073 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:14:21 smithi002 ceph-mon[71816]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2022-05-13T23:14:22.073 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:14:21 smithi002 ceph-mon[71816]: daemon mds.c assigned to filesystem cephfs as rank 3 (now has 4 ranks) 2022-05-13T23:14:22.074 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:14:21 smithi002 ceph-mon[71816]: fsmap cephfs:3 {0=l=up:active,1=i=up:active,2=f=up:active} 9 up:standby 2022-05-13T23:14:22.074 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:14:21 smithi002 ceph-mon[71816]: fsmap cephfs:4 {0=l=up:active,1=i=up:active,2=f=up:active,3=c=up:creating} 8 up:standby 2022-05-13T23:14:22.074 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:14:21 smithi002 ceph-mon[71816]: daemon mds.c is now active in filesystem cephfs as rank 3 2022-05-13T23:14:22.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:14:21 smithi115 ceph-mon[78593]: from='client.? 172.21.15.2:0/2173314192' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2022-05-13T23:14:22.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:14:21 smithi115 ceph-mon[78593]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2022-05-13T23:14:22.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:14:21 smithi115 ceph-mon[78593]: pgmap v291: 97 pgs: 8 creating+activating, 89 active+clean; 581 KiB data, 76 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s wr, 12 op/s 2022-05-13T23:14:22.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:14:21 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6834/71062276,v1:172.21.15.115:6835/71062276] up:active 2022-05-13T23:14:22.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:14:21 smithi115 ceph-mon[78593]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2022-05-13T23:14:22.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:14:21 smithi115 ceph-mon[78593]: daemon mds.c assigned to filesystem cephfs as rank 3 (now has 4 ranks) 2022-05-13T23:14:22.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:14:21 smithi115 ceph-mon[78593]: fsmap cephfs:3 {0=l=up:active,1=i=up:active,2=f=up:active} 9 up:standby 2022-05-13T23:14:22.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:14:21 smithi115 ceph-mon[78593]: fsmap cephfs:4 {0=l=up:active,1=i=up:active,2=f=up:active,3=c=up:creating} 8 up:standby 2022-05-13T23:14:22.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:14:21 smithi115 ceph-mon[78593]: daemon mds.c is now active in filesystem cephfs as rank 3 2022-05-13T23:14:22.255 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:22.253+0000 7f9dfe79d700 1 -- 172.21.15.2:0/2765872222 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9df8104920 con 0x7f9df80fd750 2022-05-13T23:14:22.256 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:22.255+0000 7f9dedffb700 1 -- 172.21.15.2:0/2765872222 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v83) v1 ==== 74+0+21267 (secure 0 0 0) 0x7f9df40918e0 con 0x7f9df80fd750 2022-05-13T23:14:22.256 INFO:teuthology.orchestra.run.smithi002.stdout: 2022-05-13T23:14:22.257 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":83,"fsid":"16cf5f88-d311-11ec-be40-001a4aab830c","created":"2022-05-13T23:06:00.765453+0000","modified":"2022-05-13T23:14:17.239435+0000","last_up_change":"2022-05-13T23:13:20.935982+0000","last_in_change":"2022-05-13T23:13:02.796000+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2022-05-13T23:09:32.000644+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2022-05-13T23:14:14.195678+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":3,"pool_name":"cephfs_data","create_time":"2022-05-13T23:14:15.659212+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"83","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"11e700e8-f3e8-4903-b52c-db354b4395bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6802","nonce":1174843704},{"type":"v1","addr":"172.21.15.2:6803","nonce":1174843704}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6804","nonce":1174843704},{"type":"v1","addr":"172.21.15.2:6805","nonce":1174843704}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6808","nonce":1174843704},{"type":"v1","addr":"172.21.15.2:6809","nonce":1174843704}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6806","nonce":1174843704},{"type":"v1","addr":"172.21.15.2:6807","nonce":1174843704}]},"public_addr":"172.21.15.2:6803/1174843704","cluster_addr":"172.21.15.2:6805/1174843704","heartbeat_back_addr":"172.21.15.2:6809/1174843704","heartbeat_front_addr":"172.21.15.2:6807/1174843704","state":["exists","up"]},{"osd":1,"uuid":"eabfa3cf-f2a3-4995-a7de-f6d8ba02ee6f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6800","nonce":3853826948},{"type":"v1","addr":"172.21.15.47:6801","nonce":3853826948}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6802","nonce":3853826948},{"type":"v1","addr":"172.21.15.47:6803","nonce":3853826948}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6806","nonce":3853826948},{"type":"v1","addr":"172.21.15.47:6807","nonce":3853826948}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6804","nonce":3853826948},{"type":"v1","addr":"172.21.15.47:6805","nonce":3853826948}]},"public_addr":"172.21.15.47:6801/3853826948","cluster_addr":"172.21.15.47:6803/3853826948","heartbeat_back_addr":"172.21.15.47:6807/3853826948","heartbeat_front_addr":"172.21.15.47:6805/3853826948","state":["exists","up"]},{"osd":2,"uuid":"153c5bd5-8200-47a7-a135-8a95b6752cd2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6800","nonce":2562256956},{"type":"v1","addr":"172.21.15.115:6801","nonce":2562256956}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6802","nonce":2562256956},{"type":"v1","addr":"172.21.15.115:6803","nonce":2562256956}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6806","nonce":2562256956},{"type":"v1","addr":"172.21.15.115:6807","nonce":2562256956}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6804","nonce":2562256956},{"type":"v1","addr":"172.21.15.115:6805","nonce":2562256956}]},"public_addr":"172.21.15.115:6801/2562256956","cluster_addr":"172.21.15.115:6803/2562256956","heartbeat_back_addr":"172.21.15.115:6807/2562256956","heartbeat_front_addr":"172.21.15.115:6805/2562256956","state":["exists","up"]},{"osd":3,"uuid":"6ca9c602-5936-48da-a861-c437e6a33765","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6810","nonce":1099909411},{"type":"v1","addr":"172.21.15.2:6811","nonce":1099909411}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6812","nonce":1099909411},{"type":"v1","addr":"172.21.15.2:6813","nonce":1099909411}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6816","nonce":1099909411},{"type":"v1","addr":"172.21.15.2:6817","nonce":1099909411}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6814","nonce":1099909411},{"type":"v1","addr":"172.21.15.2:6815","nonce":1099909411}]},"public_addr":"172.21.15.2:6811/1099909411","cluster_addr":"172.21.15.2:6813/1099909411","heartbeat_back_addr":"172.21.15.2:6817/1099909411","heartbeat_front_addr":"172.21.15.2:6815/1099909411","state":["exists","up"]},{"osd":4,"uuid":"54d943d9-2610-422a-9e59-4b8212aced31","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6808","nonce":882753861},{"type":"v1","addr":"172.21.15.47:6809","nonce":882753861}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6810","nonce":882753861},{"type":"v1","addr":"172.21.15.47:6811","nonce":882753861}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6814","nonce":882753861},{"type":"v1","addr":"172.21.15.47:6815","nonce":882753861}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6812","nonce":882753861},{"type":"v1","addr":"172.21.15.47:6813","nonce":882753861}]},"public_addr":"172.21.15.47:6809/882753861","cluster_addr":"172.21.15.47:6811/882753861","heartbeat_back_addr":"172.21.15.47:6815/882753861","heartbeat_front_addr":"172.21.15.47:6813/882753861","state":["exists","up"]},{"osd":5,"uuid":"c18566cf-5403-4057-b2ee-aa298e87e8e9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6808","nonce":2122391993},{"type":"v1","addr":"172.21.15.115:6809","nonce":2122391993}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6810","nonce":2122391993},{"type":"v1","addr":"172.21.15.115:6811","nonce":2122391993}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6814","nonce":2122391993},{"type":"v1","addr":"172.21.15.115:6815","nonce":2122391993}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6812","nonce":2122391993},{"type":"v1","addr":"172.21.15.115:6813","nonce":2122391993}]},"public_addr":"172.21.15.115:6809/2122391993","cluster_addr":"172.21.15.115:6811/2122391993","heartbeat_back_addr":"172.21.15.115:6815/2122391993","heartbeat_front_addr":"172.21.15.115:6813/2122391993","state":["exists","up"]},{"osd":6,"uuid":"38ec7b6b-30f4-4d97-ad7e-e160ae59c843","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6818","nonce":1892198790},{"type":"v1","addr":"172.21.15.2:6819","nonce":1892198790}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6820","nonce":1892198790},{"type":"v1","addr":"172.21.15.2:6821","nonce":1892198790}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6824","nonce":1892198790},{"type":"v1","addr":"172.21.15.2:6825","nonce":1892198790}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6822","nonce":1892198790},{"type":"v1","addr":"172.21.15.2:6823","nonce":1892198790}]},"public_addr":"172.21.15.2:6819/1892198790","cluster_addr":"172.21.15.2:6821/1892198790","heartbeat_back_addr":"172.21.15.2:6825/1892198790","heartbeat_front_addr":"172.21.15.2:6823/1892198790","state":["exists","up"]},{"osd":7,"uuid":"e799b560-c61e-4369-b08b-45e5a6af37c0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6816","nonce":2812542825},{"type":"v1","addr":"172.21.15.47:6817","nonce":2812542825}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6818","nonce":2812542825},{"type":"v1","addr":"172.21.15.47:6819","nonce":2812542825}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6822","nonce":2812542825},{"type":"v1","addr":"172.21.15.47:6823","nonce":2812542825}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6820","nonce":2812542825},{"type":"v1","addr":"172.21.15.47:6821","nonce":2812542825}]},"public_addr":"172.21.15.47:6817/2812542825","cluster_addr":"172.21.15.47:6819/2812542825","heartbeat_back_addr":"172.21.15.47:6823/2812542825","heartbeat_front_addr":"172.21.15.47:6821/2812542825","state":["exists","up"]},{"osd":8,"uuid":"a58f2c0e-e487-41cf-8d2e-8c937cac9ebc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":58,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6816","nonce":3959632033},{"type":"v1","addr":"172.21.15.115:6817","nonce":3959632033}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6818","nonce":3959632033},{"type":"v1","addr":"172.21.15.115:6819","nonce":3959632033}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6822","nonce":3959632033},{"type":"v1","addr":"172.21.15.115:6823","nonce":3959632033}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6820","nonce":3959632033},{"type":"v1","addr":"172.21.15.115:6821","nonce":3959632033}]},"public_addr":"172.21.15.115:6817/3959632033","cluster_addr":"172.21.15.115:6819/3959632033","heartbeat_back_addr":"172.21.15.115:6823/3959632033","heartbeat_front_addr":"172.21.15.115:6821/3959632033","state":["exists","up"]},{"osd":9,"uuid":"12cead96-f726-4f7a-ab84-eb191b778a5c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":66,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6826","nonce":3004162255},{"type":"v1","addr":"172.21.15.2:6827","nonce":3004162255}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6828","nonce":3004162255},{"type":"v1","addr":"172.21.15.2:6829","nonce":3004162255}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6832","nonce":3004162255},{"type":"v1","addr":"172.21.15.2:6833","nonce":3004162255}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6830","nonce":3004162255},{"type":"v1","addr":"172.21.15.2:6831","nonce":3004162255}]},"public_addr":"172.21.15.2:6827/3004162255","cluster_addr":"172.21.15.2:6829/3004162255","heartbeat_back_addr":"172.21.15.2:6833/3004162255","heartbeat_front_addr":"172.21.15.2:6831/3004162255","state":["exists","up"]},{"osd":10,"uuid":"4f648118-7855-48d6-86d5-44eed5295f0c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":71,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6824","nonce":3443413998},{"type":"v1","addr":"172.21.15.47:6825","nonce":3443413998}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6826","nonce":3443413998},{"type":"v1","addr":"172.21.15.47:6827","nonce":3443413998}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6830","nonce":3443413998},{"type":"v1","addr":"172.21.15.47:6831","nonce":3443413998}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6828","nonce":3443413998},{"type":"v1","addr":"172.21.15.47:6829","nonce":3443413998}]},"public_addr":"172.21.15.47:6825/3443413998","cluster_addr":"172.21.15.47:6827/3443413998","heartbeat_back_addr":"172.21.15.47:6831/3443413998","heartbeat_front_addr":"172.21.15.47:6829/3443413998","state":["exists","up"]},{"osd":11,"uuid":"916541e0-fb28-46d5-9743-801c54c5a6d1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":78,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6824","nonce":1758718209},{"type":"v1","addr":"172.21.15.115:6825","nonce":1758718209}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6826","nonce":1758718209},{"type":"v1","addr":"172.21.15.115:6827","nonce":1758718209}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6830","nonce":1758718209},{"type":"v1","addr":"172.21.15.115:6831","nonce":1758718209}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6828","nonce":1758718209},{"type":"v1","addr":"172.21.15.115:6829","nonce":1758718209}]},"public_addr":"172.21.15.115:6825/1758718209","cluster_addr":"172.21.15.115:6827/1758718209","heartbeat_back_addr":"172.21.15.115:6831/1758718209","heartbeat_front_addr":"172.21.15.115:6829/1758718209","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-05-13T23:08:43.757857+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-05-13T23:09:06.537139+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-05-13T23:09:28.524837+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-05-13T23:10:02.057583+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-05-13T23:10:26.210648+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-05-13T23:10:47.885853+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-05-13T23:11:19.141865+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-05-13T23:11:42.376065+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-05-13T23:12:04.175870+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-05-13T23:12:34.695347+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-05-13T23:12:57.762613+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-05-13T23:13:19.114284+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.2:0/477249925":"2022-05-14T23:07:04.986087+0000","172.21.15.2:6800/3226099420":"2022-05-14T23:07:04.986087+0000","172.21.15.2:0/2214093915":"2022-05-14T23:07:04.986087+0000","172.21.15.2:0/1039403355":"2022-05-14T23:07:04.986087+0000","172.21.15.2:0/22091675":"2022-05-14T23:06:36.304108+0000","172.21.15.2:6800/1255134634":"2022-05-14T23:06:36.304108+0000","172.21.15.2:6801/3226099420":"2022-05-14T23:07:04.986087+0000","172.21.15.2:0/3768677276":"2022-05-14T23:06:36.304108+0000","172.21.15.2:6801/1255134634":"2022-05-14T23:06:36.304108+0000","172.21.15.2:0/407841268":"2022-05-14T23:06:36.304108+0000"},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2022-05-13T23:14:22.260 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:22.257+0000 7f9dfe79d700 1 -- 172.21.15.2:0/2765872222 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f9de4096ce0 msgr2=0x7f9de4099190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:14:22.261 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:22.257+0000 7f9dfe79d700 1 --2- 172.21.15.2:0/2765872222 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f9de4096ce0 0x7f9de4099190 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f9de80048b0 tx=0x7f9de800b040 comp rx=0 tx=0).stop 2022-05-13T23:14:22.261 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:22.257+0000 7f9dfe79d700 1 -- 172.21.15.2:0/2765872222 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9df80fd750 msgr2=0x7f9df810e4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:14:22.261 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:22.257+0000 7f9dfe79d700 1 --2- 172.21.15.2:0/2765872222 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9df80fd750 0x7f9df810e4e0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f9df4004710 tx=0x7f9df4009cb0 comp rx=0 tx=0).stop 2022-05-13T23:14:22.261 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:22.257+0000 7f9dfe79d700 1 -- 172.21.15.2:0/2765872222 shutdown_connections 2022-05-13T23:14:22.262 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:22.257+0000 7f9dfe79d700 1 --2- 172.21.15.2:0/2765872222 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f9df80ff380 0x7f9df8115ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:14:22.262 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:22.257+0000 7f9dfe79d700 1 --2- 172.21.15.2:0/2765872222 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f9de4096ce0 0x7f9de4099190 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:14:22.262 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:22.257+0000 7f9dfe79d700 1 --2- 172.21.15.2:0/2765872222 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9df80fd750 0x7f9df810e4e0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:14:22.262 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:22.257+0000 7f9dfe79d700 1 --2- 172.21.15.2:0/2765872222 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f9df80fe9f0 0x7f9df810ea20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:14:22.262 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:22.257+0000 7f9dfe79d700 1 -- 172.21.15.2:0/2765872222 >> 172.21.15.2:0/2765872222 conn(0x7f9df80f8e70 msgr2=0x7f9df8101c20 unknown :-1 s=STATE_NONE l=0).mark_down 2022-05-13T23:14:22.263 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:22.257+0000 7f9dfe79d700 1 -- 172.21.15.2:0/2765872222 shutdown_connections 2022-05-13T23:14:22.263 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:22.258+0000 7f9dfe79d700 1 -- 172.21.15.2:0/2765872222 wait complete. 2022-05-13T23:14:22.285 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2022-05-13T23:14:22.286 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2022-05-13T23:14:22.510 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:22.508+0000 7f1b1d19d700 1 -- 172.21.15.2:0/2477563213 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f1b180fb490 msgr2=0x7f1b180fb8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:14:22.511 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:22.508+0000 7f1b1d19d700 1 --2- 172.21.15.2:0/2477563213 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f1b180fb490 0x7f1b180fb8a0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f1b08004870 tx=0x7f1b08009cb0 comp rx=0 tx=0).stop 2022-05-13T23:14:22.511 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:22.509+0000 7f1b1d19d700 1 -- 172.21.15.2:0/2477563213 shutdown_connections 2022-05-13T23:14:22.511 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:22.509+0000 7f1b1d19d700 1 --2- 172.21.15.2:0/2477563213 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1b180fd0c0 0x7f1b18103980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:14:22.511 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:22.509+0000 7f1b1d19d700 1 --2- 172.21.15.2:0/2477563213 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f1b180fb490 0x7f1b180fb8a0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:14:22.512 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:22.509+0000 7f1b1d19d700 1 --2- 172.21.15.2:0/2477563213 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1b180fc730 0x7f1b180fcb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:14:22.512 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:22.509+0000 7f1b1d19d700 1 -- 172.21.15.2:0/2477563213 >> 172.21.15.2:0/2477563213 conn(0x7f1b180f6c40 msgr2=0x7f1b180f90b0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-05-13T23:14:22.512 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:22.509+0000 7f1b1d19d700 1 -- 172.21.15.2:0/2477563213 shutdown_connections 2022-05-13T23:14:22.513 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:22.509+0000 7f1b1d19d700 1 -- 172.21.15.2:0/2477563213 wait complete. 2022-05-13T23:14:22.515 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:22.510+0000 7f1b1d19d700 1 Processor -- start 2022-05-13T23:14:22.515 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:22.510+0000 7f1b1d19d700 1 -- start start 2022-05-13T23:14:22.516 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:22.510+0000 7f1b1d19d700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1b180fb490 0x7f1b180701d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:14:22.516 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:22.510+0000 7f1b1d19d700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1b180fc730 0x7f1b1806e830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:14:22.516 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:22.510+0000 7f1b1d19d700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f1b180fd0c0 0x7f1b1806ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:14:22.516 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:22.510+0000 7f1b1d19d700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b1806f480 con 0x7f1b180fc730 2022-05-13T23:14:22.517 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:22.510+0000 7f1b1d19d700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b1806f5f0 con 0x7f1b180fb490 2022-05-13T23:14:22.517 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:22.510+0000 7f1b1d19d700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b18116400 con 0x7f1b180fd0c0 2022-05-13T23:14:22.517 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:22.510+0000 7f1b17fff700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1b180fb490 0x7f1b180701d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:14:22.518 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:22.510+0000 7f1b177fe700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1b180fc730 0x7f1b1806e830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:14:22.518 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:22.510+0000 7f1b17fff700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1b180fb490 0x7f1b180701d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.2:39168/0 (socket says 172.21.15.2:39168) 2022-05-13T23:14:22.518 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:22.510+0000 7f1b17fff700 1 -- 172.21.15.2:0/1945031932 learned_addr learned my addr 172.21.15.2:0/1945031932 (peer_addr_for_me v2:172.21.15.2:0/0) 2022-05-13T23:14:22.518 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:22.511+0000 7f1b1c99c700 1 --2- 172.21.15.2:0/1945031932 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f1b180fd0c0 0x7f1b1806ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:14:22.519 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:22.511+0000 7f1b17fff700 1 -- 172.21.15.2:0/1945031932 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f1b180fd0c0 msgr2=0x7f1b1806ed70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:14:22.519 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:22.511+0000 7f1b17fff700 1 --2- 172.21.15.2:0/1945031932 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f1b180fd0c0 0x7f1b1806ed70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:14:22.519 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:22.511+0000 7f1b17fff700 1 -- 172.21.15.2:0/1945031932 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1b180fc730 msgr2=0x7f1b1806e830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:14:22.519 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:22.511+0000 7f1b17fff700 1 --2- 172.21.15.2:0/1945031932 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1b180fc730 0x7f1b1806e830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:14:22.519 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:22.511+0000 7f1b17fff700 1 -- 172.21.15.2:0/1945031932 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b100097c0 con 0x7f1b180fb490 2022-05-13T23:14:22.520 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:22.511+0000 7f1b177fe700 1 --2- 172.21.15.2:0/1945031932 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1b180fc730 0x7f1b1806e830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-05-13T23:14:22.520 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:22.511+0000 7f1b17fff700 1 --2- 172.21.15.2:0/1945031932 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1b180fb490 0x7f1b180701d0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f1b08004710 tx=0x7f1b08009cb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-05-13T23:14:22.520 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:22.511+0000 7f1b14ff9700 1 -- 172.21.15.2:0/1945031932 <== mon.1 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b0800e040 con 0x7f1b180fb490 2022-05-13T23:14:22.520 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:22.511+0000 7f1b1d19d700 1 -- 172.21.15.2:0/1945031932 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1b08005040 con 0x7f1b180fb490 2022-05-13T23:14:22.521 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:22.512+0000 7f1b14ff9700 1 -- 172.21.15.2:0/1945031932 <== mon.1 v2:172.21.15.47:3300/0 2 ==== config(26 keys) v1 ==== 1087+0+0 (secure 0 0 0) 0x7f1b08014e40 con 0x7f1b180fb490 2022-05-13T23:14:22.521 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:22.512+0000 7f1b14ff9700 1 -- 172.21.15.2:0/1945031932 <== mon.1 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b0801ca90 con 0x7f1b180fb490 2022-05-13T23:14:22.521 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:22.512+0000 7f1b1d19d700 1 -- 172.21.15.2:0/1945031932 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1b18116910 con 0x7f1b180fb490 2022-05-13T23:14:22.521 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:22.512+0000 7f1b1d19d700 1 -- 172.21.15.2:0/1945031932 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1b18106cf0 con 0x7f1b180fb490 2022-05-13T23:14:22.522 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:22.515+0000 7f1b14ff9700 1 -- 172.21.15.2:0/1945031932 <== mon.1 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 16) v1 ==== 125136+0+0 (secure 0 0 0) 0x7f1b08009240 con 0x7f1b180fb490 2022-05-13T23:14:22.523 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:22.516+0000 7f1b14ff9700 1 --2- 172.21.15.2:0/1945031932 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f1b00096ce0 0x7f1b00099190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:14:22.523 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:22.516+0000 7f1b14ff9700 1 -- 172.21.15.2:0/1945031932 <== mon.1 v2:172.21.15.47:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 7975+0+0 (secure 0 0 0) 0x7f1b080c0fe0 con 0x7f1b180fb490 2022-05-13T23:14:22.523 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:22.516+0000 7f1b177fe700 1 --2- 172.21.15.2:0/1945031932 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f1b00096ce0 0x7f1b00099190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:14:22.523 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:22.517+0000 7f1b177fe700 1 --2- 172.21.15.2:0/1945031932 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f1b00096ce0 0x7f1b00099190 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f1b0c007bf0 tx=0x7f1b0c00b040 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-05-13T23:14:22.524 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:22.518+0000 7f1b14ff9700 1 -- 172.21.15.2:0/1945031932 <== mon.1 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+165215 (secure 0 0 0) 0x7f1b0808e410 con 0x7f1b180fb490 2022-05-13T23:14:22.708 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:22.707+0000 7f1b1d19d700 1 -- 172.21.15.2:0/1945031932 --> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f1b1805c920 con 0x7f1b00096ce0 2022-05-13T23:14:22.722 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:22.720+0000 7f1b14ff9700 1 -- 172.21.15.2:0/1945031932 <== mgr.14148 v2:172.21.15.2:6800/2788339627 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+332192 (secure 0 0 0) 0x7f1b1805c920 con 0x7f1b00096ce0 2022-05-13T23:14:22.728 INFO:teuthology.orchestra.run.smithi002.stdout: 2022-05-13T23:14:22.732 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:22.725+0000 7f1b1d19d700 1 -- 172.21.15.2:0/1945031932 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f1b00096ce0 msgr2=0x7f1b00099190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:14:22.732 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:22.725+0000 7f1b1d19d700 1 --2- 172.21.15.2:0/1945031932 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f1b00096ce0 0x7f1b00099190 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f1b0c007bf0 tx=0x7f1b0c00b040 comp rx=0 tx=0).stop 2022-05-13T23:14:22.732 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:22.725+0000 7f1b1d19d700 1 -- 172.21.15.2:0/1945031932 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1b180fb490 msgr2=0x7f1b180701d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:14:22.733 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:22.725+0000 7f1b1d19d700 1 --2- 172.21.15.2:0/1945031932 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1b180fb490 0x7f1b180701d0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f1b08004710 tx=0x7f1b08009cb0 comp rx=0 tx=0).stop 2022-05-13T23:14:22.733 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:22.726+0000 7f1b1d19d700 1 -- 172.21.15.2:0/1945031932 shutdown_connections 2022-05-13T23:14:22.733 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:22.726+0000 7f1b1d19d700 1 --2- 172.21.15.2:0/1945031932 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f1b180fd0c0 0x7f1b1806ed70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:14:22.733 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:22.726+0000 7f1b1d19d700 1 --2- 172.21.15.2:0/1945031932 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1b180fb490 0x7f1b180701d0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:14:22.734 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:22.726+0000 7f1b1d19d700 1 --2- 172.21.15.2:0/1945031932 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f1b00096ce0 0x7f1b00099190 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:14:22.734 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:22.726+0000 7f1b1d19d700 1 --2- 172.21.15.2:0/1945031932 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1b180fc730 0x7f1b1806e830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:14:22.734 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:22.726+0000 7f1b1d19d700 1 -- 172.21.15.2:0/1945031932 >> 172.21.15.2:0/1945031932 conn(0x7f1b180f6c40 msgr2=0x7f1b180ff960 unknown :-1 s=STATE_NONE l=0).mark_down 2022-05-13T23:14:22.734 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:22.726+0000 7f1b1d19d700 1 -- 172.21.15.2:0/1945031932 shutdown_connections 2022-05-13T23:14:22.735 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:22.726+0000 7f1b1d19d700 1 -- 172.21.15.2:0/1945031932 wait complete. 2022-05-13T23:14:22.735 INFO:teuthology.orchestra.run.smithi002.stderr:dumped all 2022-05-13T23:14:22.754 INFO:teuthology.orchestra.run.smithi002.stdout:{"pg_ready":true,"pg_map":{"version":291,"stamp":"2022-05-13T23:14:21.234600+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":594536,"num_objects":37,"num_object_clones":0,"num_object_copies":111,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":255,"num_write_kb":2957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1057856,"num_keys_recovered":0,"num_objects_omap":21,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":203,"ondisk_log_size":203,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":214,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":77924,"kb_used_data":11972,"kb_used_omap":0,"kb_used_meta":65856,"kb_avail":1124863900,"statfs":{"total":1151940427776,"available":1151860633600,"internally_reserved":0,"allocated":12259328,"data_stored":6628751,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":67436544},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":23,"apply_latency_ms":23,"commit_latency_ns":23000000,"apply_latency_ns":23000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":4168,"num_objects":35,"num_object_clones":0,"num_object_copies":105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":74,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":21,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001502"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":9,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-05-13T23:14:16.267128+0000","last_change":"2022-05-13T23:14:16.267128+0000","last_active":"2022-05-13T23:14:16.267128+0000","last_peered":"2022-05-13T23:14:16.267128+0000","last_clean":"2022-05-13T23:14:16.267128+0000","last_became_active":"2022-05-13T23:14:16.266687+0000","last_became_peered":"2022-05-13T23:14:16.266687+0000","last_unstale":"2022-05-13T23:14:16.267128+0000","last_undegraded":"2022-05-13T23:14:16.267128+0000","last_fullsized":"2022-05-13T23:14:16.267128+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_clean_scrub_stamp":"2022-05-13T23:14:15.659362+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T04:40:46.150557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":2,"reported_epoch":82,"state":"creating+activating","last_fresh":"2022-05-13T23:14:16.252511+0000","last_change":"2022-05-13T23:14:16.252678+0000","last_active":"2022-05-13T23:14:15.659362+0000","last_peered":"2022-05-13T23:14:15.659362+0000","last_clean":"2022-05-13T23:14:15.659362+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2022-05-13T23:14:16.252511+0000","last_undegraded":"2022-05-13T23:14:16.252511+0000","last_fullsized":"2022-05-13T23:14:16.252511+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_clean_scrub_stamp":"2022-05-13T23:14:15.659362+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T00:32:19.185037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":["0","10","11"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":3,"reported_epoch":82,"state":"creating+activating","last_fresh":"2022-05-13T23:14:16.249762+0000","last_change":"2022-05-13T23:14:16.249762+0000","last_active":"2022-05-13T23:14:15.659362+0000","last_peered":"2022-05-13T23:14:15.659362+0000","last_clean":"2022-05-13T23:14:15.659362+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2022-05-13T23:14:16.249762+0000","last_undegraded":"2022-05-13T23:14:16.249762+0000","last_fullsized":"2022-05-13T23:14:16.249762+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_clean_scrub_stamp":"2022-05-13T23:14:15.659362+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T10:03:03.774806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":["0","1","2"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":10,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-05-13T23:14:16.257175+0000","last_change":"2022-05-13T23:14:16.255885+0000","last_active":"2022-05-13T23:14:16.257175+0000","last_peered":"2022-05-13T23:14:16.257175+0000","last_clean":"2022-05-13T23:14:16.257175+0000","last_became_active":"2022-05-13T23:14:16.255551+0000","last_became_peered":"2022-05-13T23:14:16.255551+0000","last_unstale":"2022-05-13T23:14:16.257175+0000","last_undegraded":"2022-05-13T23:14:16.257175+0000","last_fullsized":"2022-05-13T23:14:16.257175+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_clean_scrub_stamp":"2022-05-13T23:14:15.659362+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T10:26:40.578066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":9,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-05-13T23:14:16.271384+0000","last_change":"2022-05-13T23:14:16.271384+0000","last_active":"2022-05-13T23:14:16.271384+0000","last_peered":"2022-05-13T23:14:16.271384+0000","last_clean":"2022-05-13T23:14:16.271384+0000","last_became_active":"2022-05-13T23:14:16.271045+0000","last_became_peered":"2022-05-13T23:14:16.271045+0000","last_unstale":"2022-05-13T23:14:16.271384+0000","last_undegraded":"2022-05-13T23:14:16.271384+0000","last_fullsized":"2022-05-13T23:14:16.271384+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_clean_scrub_stamp":"2022-05-13T23:14:15.659362+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T04:51:50.618628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":10,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-05-13T23:14:16.276293+0000","last_change":"2022-05-13T23:14:16.275906+0000","last_active":"2022-05-13T23:14:16.276293+0000","last_peered":"2022-05-13T23:14:16.276293+0000","last_clean":"2022-05-13T23:14:16.276293+0000","last_became_active":"2022-05-13T23:14:16.275651+0000","last_became_peered":"2022-05-13T23:14:16.275651+0000","last_unstale":"2022-05-13T23:14:16.276293+0000","last_undegraded":"2022-05-13T23:14:16.276293+0000","last_fullsized":"2022-05-13T23:14:16.276293+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_clean_scrub_stamp":"2022-05-13T23:14:15.659362+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T02:29:33.019268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":10,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-05-13T23:14:16.276560+0000","last_change":"2022-05-13T23:14:16.276150+0000","last_active":"2022-05-13T23:14:16.276560+0000","last_peered":"2022-05-13T23:14:16.276560+0000","last_clean":"2022-05-13T23:14:16.276560+0000","last_became_active":"2022-05-13T23:14:16.275751+0000","last_became_peered":"2022-05-13T23:14:16.275751+0000","last_unstale":"2022-05-13T23:14:16.276560+0000","last_undegraded":"2022-05-13T23:14:16.276560+0000","last_fullsized":"2022-05-13T23:14:16.276560+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_clean_scrub_stamp":"2022-05-13T23:14:15.659362+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T04:01:48.496870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":10,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-05-13T23:14:16.270951+0000","last_change":"2022-05-13T23:14:16.268816+0000","last_active":"2022-05-13T23:14:16.270951+0000","last_peered":"2022-05-13T23:14:16.270951+0000","last_clean":"2022-05-13T23:14:16.270951+0000","last_became_active":"2022-05-13T23:14:16.268429+0000","last_became_peered":"2022-05-13T23:14:16.268429+0000","last_unstale":"2022-05-13T23:14:16.270951+0000","last_undegraded":"2022-05-13T23:14:16.270951+0000","last_fullsized":"2022-05-13T23:14:16.270951+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_clean_scrub_stamp":"2022-05-13T23:14:15.659362+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T00:14:51.685287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":10,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-05-13T23:14:16.265890+0000","last_change":"2022-05-13T23:14:16.263971+0000","last_active":"2022-05-13T23:14:16.265890+0000","last_peered":"2022-05-13T23:14:16.265890+0000","last_clean":"2022-05-13T23:14:16.265890+0000","last_became_active":"2022-05-13T23:14:16.263625+0000","last_became_peered":"2022-05-13T23:14:16.263625+0000","last_unstale":"2022-05-13T23:14:16.265890+0000","last_undegraded":"2022-05-13T23:14:16.265890+0000","last_fullsized":"2022-05-13T23:14:16.265890+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_clean_scrub_stamp":"2022-05-13T23:14:15.659362+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T03:11:11.037746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":10,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-05-13T23:14:16.258455+0000","last_change":"2022-05-13T23:14:16.258111+0000","last_active":"2022-05-13T23:14:16.258455+0000","last_peered":"2022-05-13T23:14:16.258455+0000","last_clean":"2022-05-13T23:14:16.258455+0000","last_became_active":"2022-05-13T23:14:16.257768+0000","last_became_peered":"2022-05-13T23:14:16.257768+0000","last_unstale":"2022-05-13T23:14:16.258455+0000","last_undegraded":"2022-05-13T23:14:16.258455+0000","last_fullsized":"2022-05-13T23:14:16.258455+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_clean_scrub_stamp":"2022-05-13T23:14:15.659362+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T07:27:48.568156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":10,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-05-13T23:14:16.268155+0000","last_change":"2022-05-13T23:14:16.267256+0000","last_active":"2022-05-13T23:14:16.268155+0000","last_peered":"2022-05-13T23:14:16.268155+0000","last_clean":"2022-05-13T23:14:16.268155+0000","last_became_active":"2022-05-13T23:14:16.266868+0000","last_became_peered":"2022-05-13T23:14:16.266868+0000","last_unstale":"2022-05-13T23:14:16.268155+0000","last_undegraded":"2022-05-13T23:14:16.268155+0000","last_fullsized":"2022-05-13T23:14:16.268155+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_clean_scrub_stamp":"2022-05-13T23:14:15.659362+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T03:28:58.592648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":10,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-05-13T23:14:16.257196+0000","last_change":"2022-05-13T23:14:16.256051+0000","last_active":"2022-05-13T23:14:16.257196+0000","last_peered":"2022-05-13T23:14:16.257196+0000","last_clean":"2022-05-13T23:14:16.257196+0000","last_became_active":"2022-05-13T23:14:16.255729+0000","last_became_peered":"2022-05-13T23:14:16.255729+0000","last_unstale":"2022-05-13T23:14:16.257196+0000","last_undegraded":"2022-05-13T23:14:16.257196+0000","last_fullsized":"2022-05-13T23:14:16.257196+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_clean_scrub_stamp":"2022-05-13T23:14:15.659362+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T09:34:22.448938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":10,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-05-13T23:14:16.268884+0000","last_change":"2022-05-13T23:14:16.266285+0000","last_active":"2022-05-13T23:14:16.268884+0000","last_peered":"2022-05-13T23:14:16.268884+0000","last_clean":"2022-05-13T23:14:16.268884+0000","last_became_active":"2022-05-13T23:14:16.264899+0000","last_became_peered":"2022-05-13T23:14:16.264899+0000","last_unstale":"2022-05-13T23:14:16.268884+0000","last_undegraded":"2022-05-13T23:14:16.268884+0000","last_fullsized":"2022-05-13T23:14:16.268884+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_clean_scrub_stamp":"2022-05-13T23:14:15.659362+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T09:50:56.211965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":2,"reported_epoch":82,"state":"creating+activating","last_fresh":"2022-05-13T23:14:16.249497+0000","last_change":"2022-05-13T23:14:16.249760+0000","last_active":"2022-05-13T23:14:15.659362+0000","last_peered":"2022-05-13T23:14:15.659362+0000","last_clean":"2022-05-13T23:14:15.659362+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2022-05-13T23:14:16.249497+0000","last_undegraded":"2022-05-13T23:14:16.249497+0000","last_fullsized":"2022-05-13T23:14:16.249497+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_clean_scrub_stamp":"2022-05-13T23:14:15.659362+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T05:01:25.159085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":["0","8","10"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":3,"reported_epoch":82,"state":"creating+activating","last_fresh":"2022-05-13T23:14:16.249135+0000","last_change":"2022-05-13T23:14:16.249135+0000","last_active":"2022-05-13T23:14:15.659362+0000","last_peered":"2022-05-13T23:14:15.659362+0000","last_clean":"2022-05-13T23:14:15.659362+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2022-05-13T23:14:16.249135+0000","last_undegraded":"2022-05-13T23:14:16.249135+0000","last_fullsized":"2022-05-13T23:14:16.249135+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_clean_scrub_stamp":"2022-05-13T23:14:15.659362+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T02:16:47.449089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":["0","2","4"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":10,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-05-13T23:14:16.270716+0000","last_change":"2022-05-13T23:14:16.270173+0000","last_active":"2022-05-13T23:14:16.270716+0000","last_peered":"2022-05-13T23:14:16.270716+0000","last_clean":"2022-05-13T23:14:16.270716+0000","last_became_active":"2022-05-13T23:14:16.269859+0000","last_became_peered":"2022-05-13T23:14:16.269859+0000","last_unstale":"2022-05-13T23:14:16.270716+0000","last_undegraded":"2022-05-13T23:14:16.270716+0000","last_fullsized":"2022-05-13T23:14:16.270716+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_clean_scrub_stamp":"2022-05-13T23:14:15.659362+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T03:31:45.566757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":9,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-05-13T23:14:16.269290+0000","last_change":"2022-05-13T23:14:16.269290+0000","last_active":"2022-05-13T23:14:16.269290+0000","last_peered":"2022-05-13T23:14:16.269290+0000","last_clean":"2022-05-13T23:14:16.269290+0000","last_became_active":"2022-05-13T23:14:16.268985+0000","last_became_peered":"2022-05-13T23:14:16.268985+0000","last_unstale":"2022-05-13T23:14:16.269290+0000","last_undegraded":"2022-05-13T23:14:16.269290+0000","last_fullsized":"2022-05-13T23:14:16.269290+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_clean_scrub_stamp":"2022-05-13T23:14:15.659362+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T03:58:45.885549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":9,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-05-13T23:14:16.266204+0000","last_change":"2022-05-13T23:14:16.266204+0000","last_active":"2022-05-13T23:14:16.266204+0000","last_peered":"2022-05-13T23:14:16.266204+0000","last_clean":"2022-05-13T23:14:16.266204+0000","last_became_active":"2022-05-13T23:14:16.262256+0000","last_became_peered":"2022-05-13T23:14:16.262256+0000","last_unstale":"2022-05-13T23:14:16.266204+0000","last_undegraded":"2022-05-13T23:14:16.266204+0000","last_fullsized":"2022-05-13T23:14:16.266204+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_clean_scrub_stamp":"2022-05-13T23:14:15.659362+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T00:53:26.594598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":10,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-05-13T23:14:16.254638+0000","last_change":"2022-05-13T23:14:16.252421+0000","last_active":"2022-05-13T23:14:16.254638+0000","last_peered":"2022-05-13T23:14:16.254638+0000","last_clean":"2022-05-13T23:14:16.254638+0000","last_became_active":"2022-05-13T23:14:16.250585+0000","last_became_peered":"2022-05-13T23:14:16.250585+0000","last_unstale":"2022-05-13T23:14:16.254638+0000","last_undegraded":"2022-05-13T23:14:16.254638+0000","last_fullsized":"2022-05-13T23:14:16.254638+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_clean_scrub_stamp":"2022-05-13T23:14:15.659362+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T00:13:24.393163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"82'4","reported_seq":14,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-05-13T23:14:16.274879+0000","last_change":"2022-05-13T23:14:15.683837+0000","last_active":"2022-05-13T23:14:16.274879+0000","last_peered":"2022-05-13T23:14:16.274879+0000","last_clean":"2022-05-13T23:14:16.274879+0000","last_became_active":"2022-05-13T23:14:15.682976+0000","last_became_peered":"2022-05-13T23:14:15.682976+0000","last_unstale":"2022-05-13T23:14:16.274879+0000","last_undegraded":"2022-05-13T23:14:16.274879+0000","last_fullsized":"2022-05-13T23:14:16.274879+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:14.344206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:14.344206+0000","last_clean_scrub_stamp":"2022-05-13T23:14:14.344206+0000","objects_scrubbed":0,"log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T01:05:36.612390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":554,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":10,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-05-13T23:14:16.267355+0000","last_change":"2022-05-13T23:14:16.264501+0000","last_active":"2022-05-13T23:14:16.267355+0000","last_peered":"2022-05-13T23:14:16.267355+0000","last_clean":"2022-05-13T23:14:16.267355+0000","last_became_active":"2022-05-13T23:14:16.264105+0000","last_became_peered":"2022-05-13T23:14:16.264105+0000","last_unstale":"2022-05-13T23:14:16.267355+0000","last_undegraded":"2022-05-13T23:14:16.267355+0000","last_fullsized":"2022-05-13T23:14:16.267355+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_clean_scrub_stamp":"2022-05-13T23:14:15.659362+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T00:21:48.857254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"82'1","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-05-13T23:14:16.282001+0000","last_change":"2022-05-13T23:14:15.701757+0000","last_active":"2022-05-13T23:14:16.282001+0000","last_peered":"2022-05-13T23:14:16.282001+0000","last_clean":"2022-05-13T23:14:16.282001+0000","last_became_active":"2022-05-13T23:14:15.701428+0000","last_became_peered":"2022-05-13T23:14:15.701428+0000","last_unstale":"2022-05-13T23:14:16.282001+0000","last_undegraded":"2022-05-13T23:14:16.282001+0000","last_fullsized":"2022-05-13T23:14:16.282001+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:14.344206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:14.344206+0000","last_clean_scrub_stamp":"2022-05-13T23:14:14.344206+0000","objects_scrubbed":0,"log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T05:26:07.810433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":2,"reported_epoch":82,"state":"creating+activating","last_fresh":"2022-05-13T23:14:16.250105+0000","last_change":"2022-05-13T23:14:16.250286+0000","last_active":"2022-05-13T23:14:15.659362+0000","last_peered":"2022-05-13T23:14:15.659362+0000","last_clean":"2022-05-13T23:14:15.659362+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2022-05-13T23:14:16.250105+0000","last_undegraded":"2022-05-13T23:14:16.250105+0000","last_fullsized":"2022-05-13T23:14:16.250105+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_clean_scrub_stamp":"2022-05-13T23:14:15.659362+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T06:42:49.090643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":["0","1","9"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"82'2","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-05-13T23:14:16.263265+0000","last_change":"2022-05-13T23:14:15.683457+0000","last_active":"2022-05-13T23:14:16.263265+0000","last_peered":"2022-05-13T23:14:16.263265+0000","last_clean":"2022-05-13T23:14:16.263265+0000","last_became_active":"2022-05-13T23:14:15.682883+0000","last_became_peered":"2022-05-13T23:14:15.682883+0000","last_unstale":"2022-05-13T23:14:16.263265+0000","last_undegraded":"2022-05-13T23:14:16.263265+0000","last_fullsized":"2022-05-13T23:14:16.263265+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:14.344206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:14.344206+0000","last_clean_scrub_stamp":"2022-05-13T23:14:14.344206+0000","objects_scrubbed":0,"log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T07:38:58.996672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":10,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-05-13T23:14:16.261946+0000","last_change":"2022-05-13T23:14:16.261315+0000","last_active":"2022-05-13T23:14:16.261946+0000","last_peered":"2022-05-13T23:14:16.261946+0000","last_clean":"2022-05-13T23:14:16.261946+0000","last_became_active":"2022-05-13T23:14:16.260265+0000","last_became_peered":"2022-05-13T23:14:16.260265+0000","last_unstale":"2022-05-13T23:14:16.261946+0000","last_undegraded":"2022-05-13T23:14:16.261946+0000","last_fullsized":"2022-05-13T23:14:16.261946+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_clean_scrub_stamp":"2022-05-13T23:14:15.659362+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T08:02:58.080071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"82'2","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-05-13T23:14:16.262971+0000","last_change":"2022-05-13T23:14:15.682693+0000","last_active":"2022-05-13T23:14:16.262971+0000","last_peered":"2022-05-13T23:14:16.262971+0000","last_clean":"2022-05-13T23:14:16.262971+0000","last_became_active":"2022-05-13T23:14:15.682414+0000","last_became_peered":"2022-05-13T23:14:15.682414+0000","last_unstale":"2022-05-13T23:14:16.262971+0000","last_undegraded":"2022-05-13T23:14:16.262971+0000","last_fullsized":"2022-05-13T23:14:16.262971+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:14.344206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:14.344206+0000","last_clean_scrub_stamp":"2022-05-13T23:14:14.344206+0000","objects_scrubbed":0,"log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T09:15:17.699871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":10,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-05-13T23:14:16.261554+0000","last_change":"2022-05-13T23:14:16.261044+0000","last_active":"2022-05-13T23:14:16.261554+0000","last_peered":"2022-05-13T23:14:16.261554+0000","last_clean":"2022-05-13T23:14:16.261554+0000","last_became_active":"2022-05-13T23:14:16.260687+0000","last_became_peered":"2022-05-13T23:14:16.260687+0000","last_unstale":"2022-05-13T23:14:16.261554+0000","last_undegraded":"2022-05-13T23:14:16.261554+0000","last_fullsized":"2022-05-13T23:14:16.261554+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_clean_scrub_stamp":"2022-05-13T23:14:15.659362+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-14T23:38:23.413188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"83'5","reported_seq":16,"reported_epoch":83,"state":"active+clean","last_fresh":"2022-05-13T23:14:19.292636+0000","last_change":"2022-05-13T23:14:15.678240+0000","last_active":"2022-05-13T23:14:19.292636+0000","last_peered":"2022-05-13T23:14:19.292636+0000","last_clean":"2022-05-13T23:14:19.292636+0000","last_became_active":"2022-05-13T23:14:15.677918+0000","last_became_peered":"2022-05-13T23:14:15.677918+0000","last_unstale":"2022-05-13T23:14:19.292636+0000","last_undegraded":"2022-05-13T23:14:19.292636+0000","last_fullsized":"2022-05-13T23:14:19.292636+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:14.344206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:14.344206+0000","last_clean_scrub_stamp":"2022-05-13T23:14:14.344206+0000","objects_scrubbed":0,"log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T09:53:37.623728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":574,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":8,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-05-13T23:14:16.273411+0000","last_change":"2022-05-13T23:14:16.273537+0000","last_active":"2022-05-13T23:14:16.273411+0000","last_peered":"2022-05-13T23:14:16.273411+0000","last_clean":"2022-05-13T23:14:15.659362+0000","last_became_active":"2022-05-13T23:14:16.273127+0000","last_became_peered":"2022-05-13T23:14:16.273127+0000","last_unstale":"2022-05-13T23:14:16.273411+0000","last_undegraded":"2022-05-13T23:14:16.273411+0000","last_fullsized":"2022-05-13T23:14:16.273411+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_clean_scrub_stamp":"2022-05-13T23:14:15.659362+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T01:00:18.643721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"82'1","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-05-13T23:14:16.267285+0000","last_change":"2022-05-13T23:14:15.680562+0000","last_active":"2022-05-13T23:14:16.267285+0000","last_peered":"2022-05-13T23:14:16.267285+0000","last_clean":"2022-05-13T23:14:16.267285+0000","last_became_active":"2022-05-13T23:14:15.680163+0000","last_became_peered":"2022-05-13T23:14:15.680163+0000","last_unstale":"2022-05-13T23:14:16.267285+0000","last_undegraded":"2022-05-13T23:14:16.267285+0000","last_fullsized":"2022-05-13T23:14:16.267285+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:14.344206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:14.344206+0000","last_clean_scrub_stamp":"2022-05-13T23:14:14.344206+0000","objects_scrubbed":0,"log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T02:39:27.895407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":10,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-05-13T23:14:16.271643+0000","last_change":"2022-05-13T23:14:16.270878+0000","last_active":"2022-05-13T23:14:16.271643+0000","last_peered":"2022-05-13T23:14:16.271643+0000","last_clean":"2022-05-13T23:14:16.271643+0000","last_became_active":"2022-05-13T23:14:16.270416+0000","last_became_peered":"2022-05-13T23:14:16.270416+0000","last_unstale":"2022-05-13T23:14:16.271643+0000","last_undegraded":"2022-05-13T23:14:16.271643+0000","last_fullsized":"2022-05-13T23:14:16.271643+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_clean_scrub_stamp":"2022-05-13T23:14:15.659362+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T06:11:33.932416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"83'2","reported_seq":13,"reported_epoch":83,"state":"active+clean","last_fresh":"2022-05-13T23:14:20.303430+0000","last_change":"2022-05-13T23:14:15.676518+0000","last_active":"2022-05-13T23:14:20.303430+0000","last_peered":"2022-05-13T23:14:20.303430+0000","last_clean":"2022-05-13T23:14:20.303430+0000","last_became_active":"2022-05-13T23:14:15.676157+0000","last_became_peered":"2022-05-13T23:14:15.676157+0000","last_unstale":"2022-05-13T23:14:20.303430+0000","last_undegraded":"2022-05-13T23:14:20.303430+0000","last_fullsized":"2022-05-13T23:14:20.303430+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:14.344206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:14.344206+0000","last_clean_scrub_stamp":"2022-05-13T23:14:14.344206+0000","objects_scrubbed":0,"log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T10:16:49.469347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":10,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-05-13T23:14:16.262532+0000","last_change":"2022-05-13T23:14:16.257424+0000","last_active":"2022-05-13T23:14:16.262532+0000","last_peered":"2022-05-13T23:14:16.262532+0000","last_clean":"2022-05-13T23:14:16.262532+0000","last_became_active":"2022-05-13T23:14:16.254550+0000","last_became_peered":"2022-05-13T23:14:16.254550+0000","last_unstale":"2022-05-13T23:14:16.262532+0000","last_undegraded":"2022-05-13T23:14:16.262532+0000","last_fullsized":"2022-05-13T23:14:16.262532+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_clean_scrub_stamp":"2022-05-13T23:14:15.659362+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T00:04:50.150368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"83'1","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2022-05-13T23:14:17.560883+0000","last_change":"2022-05-13T23:14:15.680378+0000","last_active":"2022-05-13T23:14:17.560883+0000","last_peered":"2022-05-13T23:14:17.560883+0000","last_clean":"2022-05-13T23:14:17.560883+0000","last_became_active":"2022-05-13T23:14:15.680063+0000","last_became_peered":"2022-05-13T23:14:15.680063+0000","last_unstale":"2022-05-13T23:14:17.560883+0000","last_undegraded":"2022-05-13T23:14:17.560883+0000","last_fullsized":"2022-05-13T23:14:17.560883+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:14.344206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:14.344206+0000","last_clean_scrub_stamp":"2022-05-13T23:14:14.344206+0000","objects_scrubbed":0,"log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-14T23:52:48.817491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":9,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-05-13T23:14:16.277131+0000","last_change":"2022-05-13T23:14:16.277131+0000","last_active":"2022-05-13T23:14:16.277131+0000","last_peered":"2022-05-13T23:14:16.277131+0000","last_clean":"2022-05-13T23:14:16.277131+0000","last_became_active":"2022-05-13T23:14:16.276815+0000","last_became_peered":"2022-05-13T23:14:16.276815+0000","last_unstale":"2022-05-13T23:14:16.277131+0000","last_undegraded":"2022-05-13T23:14:16.277131+0000","last_fullsized":"2022-05-13T23:14:16.277131+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_clean_scrub_stamp":"2022-05-13T23:14:15.659362+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-14T23:22:29.938073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":11,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-05-13T23:14:16.242578+0000","last_change":"2022-05-13T23:14:15.683981+0000","last_active":"2022-05-13T23:14:16.242578+0000","last_peered":"2022-05-13T23:14:16.242578+0000","last_clean":"2022-05-13T23:14:16.242578+0000","last_became_active":"2022-05-13T23:14:15.683560+0000","last_became_peered":"2022-05-13T23:14:15.683560+0000","last_unstale":"2022-05-13T23:14:16.242578+0000","last_undegraded":"2022-05-13T23:14:16.242578+0000","last_fullsized":"2022-05-13T23:14:16.242578+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:14.344206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:14.344206+0000","last_clean_scrub_stamp":"2022-05-13T23:14:14.344206+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T00:30:01.510592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":10,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-05-13T23:14:16.269905+0000","last_change":"2022-05-13T23:14:16.269244+0000","last_active":"2022-05-13T23:14:16.269905+0000","last_peered":"2022-05-13T23:14:16.269905+0000","last_clean":"2022-05-13T23:14:16.269905+0000","last_became_active":"2022-05-13T23:14:16.268949+0000","last_became_peered":"2022-05-13T23:14:16.268949+0000","last_unstale":"2022-05-13T23:14:16.269905+0000","last_undegraded":"2022-05-13T23:14:16.269905+0000","last_fullsized":"2022-05-13T23:14:16.269905+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_clean_scrub_stamp":"2022-05-13T23:14:15.659362+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T04:34:53.311191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"83'2","reported_seq":13,"reported_epoch":83,"state":"active+clean","last_fresh":"2022-05-13T23:14:19.290837+0000","last_change":"2022-05-13T23:14:15.683758+0000","last_active":"2022-05-13T23:14:19.290837+0000","last_peered":"2022-05-13T23:14:19.290837+0000","last_clean":"2022-05-13T23:14:19.290837+0000","last_became_active":"2022-05-13T23:14:15.683356+0000","last_became_peered":"2022-05-13T23:14:15.683356+0000","last_unstale":"2022-05-13T23:14:19.290837+0000","last_undegraded":"2022-05-13T23:14:19.290837+0000","last_fullsized":"2022-05-13T23:14:19.290837+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:14.344206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:14.344206+0000","last_clean_scrub_stamp":"2022-05-13T23:14:14.344206+0000","objects_scrubbed":0,"log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T08:00:55.113935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":3,"reported_epoch":82,"state":"creating+activating","last_fresh":"2022-05-13T23:14:16.250747+0000","last_change":"2022-05-13T23:14:16.250747+0000","last_active":"2022-05-13T23:14:15.659362+0000","last_peered":"2022-05-13T23:14:15.659362+0000","last_clean":"2022-05-13T23:14:15.659362+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2022-05-13T23:14:16.250747+0000","last_undegraded":"2022-05-13T23:14:16.250747+0000","last_fullsized":"2022-05-13T23:14:16.250747+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_clean_scrub_stamp":"2022-05-13T23:14:15.659362+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T06:22:45.140343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":["0","4","5"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"83'4","reported_seq":15,"reported_epoch":83,"state":"active+clean","last_fresh":"2022-05-13T23:14:17.561666+0000","last_change":"2022-05-13T23:14:15.701120+0000","last_active":"2022-05-13T23:14:17.561666+0000","last_peered":"2022-05-13T23:14:17.561666+0000","last_clean":"2022-05-13T23:14:17.561666+0000","last_became_active":"2022-05-13T23:14:15.700812+0000","last_became_peered":"2022-05-13T23:14:15.700812+0000","last_unstale":"2022-05-13T23:14:17.561666+0000","last_undegraded":"2022-05-13T23:14:17.561666+0000","last_fullsized":"2022-05-13T23:14:17.561666+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:14.344206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:14.344206+0000","last_clean_scrub_stamp":"2022-05-13T23:14:14.344206+0000","objects_scrubbed":0,"log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T02:16:02.909062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":644,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":10,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-05-13T23:14:16.266148+0000","last_change":"2022-05-13T23:14:16.265167+0000","last_active":"2022-05-13T23:14:16.266148+0000","last_peered":"2022-05-13T23:14:16.266148+0000","last_clean":"2022-05-13T23:14:16.266148+0000","last_became_active":"2022-05-13T23:14:16.264886+0000","last_became_peered":"2022-05-13T23:14:16.264886+0000","last_unstale":"2022-05-13T23:14:16.266148+0000","last_undegraded":"2022-05-13T23:14:16.266148+0000","last_fullsized":"2022-05-13T23:14:16.266148+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_clean_scrub_stamp":"2022-05-13T23:14:15.659362+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T00:26:06.650867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"82'6","reported_seq":17,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-05-13T23:14:16.269389+0000","last_change":"2022-05-13T23:14:15.682385+0000","last_active":"2022-05-13T23:14:16.269389+0000","last_peered":"2022-05-13T23:14:16.269389+0000","last_clean":"2022-05-13T23:14:16.269389+0000","last_became_active":"2022-05-13T23:14:15.681996+0000","last_became_peered":"2022-05-13T23:14:15.681996+0000","last_unstale":"2022-05-13T23:14:16.269389+0000","last_undegraded":"2022-05-13T23:14:16.269389+0000","last_fullsized":"2022-05-13T23:14:16.269389+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:14.344206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:14.344206+0000","last_clean_scrub_stamp":"2022-05-13T23:14:14.344206+0000","objects_scrubbed":0,"log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-14T23:44:23.766036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1536,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":9,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-05-13T23:14:16.263744+0000","last_change":"2022-05-13T23:14:16.263744+0000","last_active":"2022-05-13T23:14:16.263744+0000","last_peered":"2022-05-13T23:14:16.263744+0000","last_clean":"2022-05-13T23:14:16.263744+0000","last_became_active":"2022-05-13T23:14:16.263525+0000","last_became_peered":"2022-05-13T23:14:16.263525+0000","last_unstale":"2022-05-13T23:14:16.263744+0000","last_undegraded":"2022-05-13T23:14:16.263744+0000","last_fullsized":"2022-05-13T23:14:16.263744+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_clean_scrub_stamp":"2022-05-13T23:14:15.659362+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T04:23:44.856055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"83'6","reported_seq":17,"reported_epoch":83,"state":"active+clean","last_fresh":"2022-05-13T23:14:20.301591+0000","last_change":"2022-05-13T23:14:15.683079+0000","last_active":"2022-05-13T23:14:20.301591+0000","last_peered":"2022-05-13T23:14:20.301591+0000","last_clean":"2022-05-13T23:14:20.301591+0000","last_became_active":"2022-05-13T23:14:15.682750+0000","last_became_peered":"2022-05-13T23:14:15.682750+0000","last_unstale":"2022-05-13T23:14:20.301591+0000","last_undegraded":"2022-05-13T23:14:20.301591+0000","last_fullsized":"2022-05-13T23:14:20.301591+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:14.344206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:14.344206+0000","last_clean_scrub_stamp":"2022-05-13T23:14:14.344206+0000","objects_scrubbed":0,"log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T02:37:31.530579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":10,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-05-13T23:14:16.268332+0000","last_change":"2022-05-13T23:14:16.263943+0000","last_active":"2022-05-13T23:14:16.268332+0000","last_peered":"2022-05-13T23:14:16.268332+0000","last_clean":"2022-05-13T23:14:16.268332+0000","last_became_active":"2022-05-13T23:14:16.263595+0000","last_became_peered":"2022-05-13T23:14:16.263595+0000","last_unstale":"2022-05-13T23:14:16.268332+0000","last_undegraded":"2022-05-13T23:14:16.268332+0000","last_fullsized":"2022-05-13T23:14:16.268332+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_clean_scrub_stamp":"2022-05-13T23:14:15.659362+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T04:57:04.201317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"83'3","reported_seq":14,"reported_epoch":83,"state":"active+clean","last_fresh":"2022-05-13T23:14:19.292697+0000","last_change":"2022-05-13T23:14:15.683005+0000","last_active":"2022-05-13T23:14:19.292697+0000","last_peered":"2022-05-13T23:14:19.292697+0000","last_clean":"2022-05-13T23:14:19.292697+0000","last_became_active":"2022-05-13T23:14:15.682692+0000","last_became_peered":"2022-05-13T23:14:15.682692+0000","last_unstale":"2022-05-13T23:14:19.292697+0000","last_undegraded":"2022-05-13T23:14:19.292697+0000","last_fullsized":"2022-05-13T23:14:19.292697+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:14.344206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:14.344206+0000","last_clean_scrub_stamp":"2022-05-13T23:14:14.344206+0000","objects_scrubbed":0,"log_size":3,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T00:27:25.201141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":10,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-05-13T23:14:16.265569+0000","last_change":"2022-05-13T23:14:16.264351+0000","last_active":"2022-05-13T23:14:16.265569+0000","last_peered":"2022-05-13T23:14:16.265569+0000","last_clean":"2022-05-13T23:14:16.265569+0000","last_became_active":"2022-05-13T23:14:16.264023+0000","last_became_peered":"2022-05-13T23:14:16.264023+0000","last_unstale":"2022-05-13T23:14:16.265569+0000","last_undegraded":"2022-05-13T23:14:16.265569+0000","last_fullsized":"2022-05-13T23:14:16.265569+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_clean_scrub_stamp":"2022-05-13T23:14:15.659362+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T06:13:51.169049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":11,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-05-13T23:14:16.246274+0000","last_change":"2022-05-13T23:14:15.677080+0000","last_active":"2022-05-13T23:14:16.246274+0000","last_peered":"2022-05-13T23:14:16.246274+0000","last_clean":"2022-05-13T23:14:16.246274+0000","last_became_active":"2022-05-13T23:14:15.676743+0000","last_became_peered":"2022-05-13T23:14:15.676743+0000","last_unstale":"2022-05-13T23:14:16.246274+0000","last_undegraded":"2022-05-13T23:14:16.246274+0000","last_fullsized":"2022-05-13T23:14:16.246274+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:14.344206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:14.344206+0000","last_clean_scrub_stamp":"2022-05-13T23:14:14.344206+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T09:05:30.321765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":10,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-05-13T23:14:16.263982+0000","last_change":"2022-05-13T23:14:16.262582+0000","last_active":"2022-05-13T23:14:16.263982+0000","last_peered":"2022-05-13T23:14:16.263982+0000","last_clean":"2022-05-13T23:14:16.263982+0000","last_became_active":"2022-05-13T23:14:16.262242+0000","last_became_peered":"2022-05-13T23:14:16.262242+0000","last_unstale":"2022-05-13T23:14:16.263982+0000","last_undegraded":"2022-05-13T23:14:16.263982+0000","last_fullsized":"2022-05-13T23:14:16.263982+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_clean_scrub_stamp":"2022-05-13T23:14:15.659362+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T02:27:28.155612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"82'8","reported_seq":17,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-05-13T23:14:16.269913+0000","last_change":"2022-05-13T23:14:15.698465+0000","last_active":"2022-05-13T23:14:16.269913+0000","last_peered":"2022-05-13T23:14:16.269913+0000","last_clean":"2022-05-13T23:14:16.269913+0000","last_became_active":"2022-05-13T23:14:15.698131+0000","last_became_peered":"2022-05-13T23:14:15.698131+0000","last_unstale":"2022-05-13T23:14:16.269913+0000","last_undegraded":"2022-05-13T23:14:16.269913+0000","last_fullsized":"2022-05-13T23:14:16.269913+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:14.344206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:14.344206+0000","last_clean_scrub_stamp":"2022-05-13T23:14:14.344206+0000","objects_scrubbed":0,"log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T03:03:48.104345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":10,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-05-13T23:14:16.261102+0000","last_change":"2022-05-13T23:14:16.260579+0000","last_active":"2022-05-13T23:14:16.261102+0000","last_peered":"2022-05-13T23:14:16.261102+0000","last_clean":"2022-05-13T23:14:16.261102+0000","last_became_active":"2022-05-13T23:14:16.260326+0000","last_became_peered":"2022-05-13T23:14:16.260326+0000","last_unstale":"2022-05-13T23:14:16.261102+0000","last_undegraded":"2022-05-13T23:14:16.261102+0000","last_fullsized":"2022-05-13T23:14:16.261102+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_clean_scrub_stamp":"2022-05-13T23:14:15.659362+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T06:53:23.092756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"82'1","reported_seq":11,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-05-13T23:14:16.248983+0000","last_change":"2022-05-13T23:14:15.680655+0000","last_active":"2022-05-13T23:14:16.248983+0000","last_peered":"2022-05-13T23:14:16.248983+0000","last_clean":"2022-05-13T23:14:16.248983+0000","last_became_active":"2022-05-13T23:14:15.680357+0000","last_became_peered":"2022-05-13T23:14:15.680357+0000","last_unstale":"2022-05-13T23:14:16.248983+0000","last_undegraded":"2022-05-13T23:14:16.248983+0000","last_fullsized":"2022-05-13T23:14:16.248983+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:14.344206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:14.344206+0000","last_clean_scrub_stamp":"2022-05-13T23:14:14.344206+0000","objects_scrubbed":0,"log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T11:00:22.395993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":9,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-05-13T23:14:16.260574+0000","last_change":"2022-05-13T23:14:16.260574+0000","last_active":"2022-05-13T23:14:16.260574+0000","last_peered":"2022-05-13T23:14:16.260574+0000","last_clean":"2022-05-13T23:14:16.260574+0000","last_became_active":"2022-05-13T23:14:16.260258+0000","last_became_peered":"2022-05-13T23:14:16.260258+0000","last_unstale":"2022-05-13T23:14:16.260574+0000","last_undegraded":"2022-05-13T23:14:16.260574+0000","last_fullsized":"2022-05-13T23:14:16.260574+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_clean_scrub_stamp":"2022-05-13T23:14:15.659362+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T10:49:06.210598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":10,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-05-13T23:14:15.701835+0000","last_change":"2022-05-13T23:14:15.701322+0000","last_active":"2022-05-13T23:14:15.701835+0000","last_peered":"2022-05-13T23:14:15.701835+0000","last_clean":"2022-05-13T23:14:15.701835+0000","last_became_active":"2022-05-13T23:14:15.701007+0000","last_became_peered":"2022-05-13T23:14:15.701007+0000","last_unstale":"2022-05-13T23:14:15.701835+0000","last_undegraded":"2022-05-13T23:14:15.701835+0000","last_fullsized":"2022-05-13T23:14:15.701835+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:14.344206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:14.344206+0000","last_clean_scrub_stamp":"2022-05-13T23:14:14.344206+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T00:13:11.071695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":10,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-05-13T23:14:16.259088+0000","last_change":"2022-05-13T23:14:16.249579+0000","last_active":"2022-05-13T23:14:16.259088+0000","last_peered":"2022-05-13T23:14:16.259088+0000","last_clean":"2022-05-13T23:14:16.259088+0000","last_became_active":"2022-05-13T23:14:16.249245+0000","last_became_peered":"2022-05-13T23:14:16.249245+0000","last_unstale":"2022-05-13T23:14:16.259088+0000","last_undegraded":"2022-05-13T23:14:16.259088+0000","last_fullsized":"2022-05-13T23:14:16.259088+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_clean_scrub_stamp":"2022-05-13T23:14:15.659362+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T00:26:53.404909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":11,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-05-13T23:14:16.255333+0000","last_change":"2022-05-13T23:14:15.699433+0000","last_active":"2022-05-13T23:14:16.255333+0000","last_peered":"2022-05-13T23:14:16.255333+0000","last_clean":"2022-05-13T23:14:16.255333+0000","last_became_active":"2022-05-13T23:14:15.698920+0000","last_became_peered":"2022-05-13T23:14:15.698920+0000","last_unstale":"2022-05-13T23:14:16.255333+0000","last_undegraded":"2022-05-13T23:14:16.255333+0000","last_fullsized":"2022-05-13T23:14:16.255333+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:14.344206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:14.344206+0000","last_clean_scrub_stamp":"2022-05-13T23:14:14.344206+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T07:56:34.276565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":10,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-05-13T23:14:16.260728+0000","last_change":"2022-05-13T23:14:16.259866+0000","last_active":"2022-05-13T23:14:16.260728+0000","last_peered":"2022-05-13T23:14:16.260728+0000","last_clean":"2022-05-13T23:14:16.260728+0000","last_became_active":"2022-05-13T23:14:16.259531+0000","last_became_peered":"2022-05-13T23:14:16.259531+0000","last_unstale":"2022-05-13T23:14:16.260728+0000","last_undegraded":"2022-05-13T23:14:16.260728+0000","last_fullsized":"2022-05-13T23:14:16.260728+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_clean_scrub_stamp":"2022-05-13T23:14:15.659362+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T07:41:46.019217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":10,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-05-13T23:14:16.261105+0000","last_change":"2022-05-13T23:14:16.259613+0000","last_active":"2022-05-13T23:14:16.261105+0000","last_peered":"2022-05-13T23:14:16.261105+0000","last_clean":"2022-05-13T23:14:16.261105+0000","last_became_active":"2022-05-13T23:14:16.259274+0000","last_became_peered":"2022-05-13T23:14:16.259274+0000","last_unstale":"2022-05-13T23:14:16.261105+0000","last_undegraded":"2022-05-13T23:14:16.261105+0000","last_fullsized":"2022-05-13T23:14:16.261105+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_clean_scrub_stamp":"2022-05-13T23:14:15.659362+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T05:36:21.174814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"83'4","reported_seq":15,"reported_epoch":83,"state":"active+clean","last_fresh":"2022-05-13T23:14:20.304530+0000","last_change":"2022-05-13T23:14:15.701315+0000","last_active":"2022-05-13T23:14:20.304530+0000","last_peered":"2022-05-13T23:14:20.304530+0000","last_clean":"2022-05-13T23:14:20.304530+0000","last_became_active":"2022-05-13T23:14:15.700955+0000","last_became_peered":"2022-05-13T23:14:15.700955+0000","last_unstale":"2022-05-13T23:14:20.304530+0000","last_undegraded":"2022-05-13T23:14:20.304530+0000","last_fullsized":"2022-05-13T23:14:20.304530+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:14.344206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:14.344206+0000","last_clean_scrub_stamp":"2022-05-13T23:14:14.344206+0000","objects_scrubbed":0,"log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T10:17:23.916475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":10,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-05-13T23:14:16.274093+0000","last_change":"2022-05-13T23:14:16.272364+0000","last_active":"2022-05-13T23:14:16.274093+0000","last_peered":"2022-05-13T23:14:16.274093+0000","last_clean":"2022-05-13T23:14:16.274093+0000","last_became_active":"2022-05-13T23:14:16.271612+0000","last_became_peered":"2022-05-13T23:14:16.271612+0000","last_unstale":"2022-05-13T23:14:16.274093+0000","last_undegraded":"2022-05-13T23:14:16.274093+0000","last_fullsized":"2022-05-13T23:14:16.274093+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_clean_scrub_stamp":"2022-05-13T23:14:15.659362+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T07:25:47.039264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":11,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-05-13T23:14:16.251030+0000","last_change":"2022-05-13T23:14:15.676251+0000","last_active":"2022-05-13T23:14:16.251030+0000","last_peered":"2022-05-13T23:14:16.251030+0000","last_clean":"2022-05-13T23:14:16.251030+0000","last_became_active":"2022-05-13T23:14:15.675895+0000","last_became_peered":"2022-05-13T23:14:15.675895+0000","last_unstale":"2022-05-13T23:14:16.251030+0000","last_undegraded":"2022-05-13T23:14:16.251030+0000","last_fullsized":"2022-05-13T23:14:16.251030+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:14.344206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:14.344206+0000","last_clean_scrub_stamp":"2022-05-13T23:14:14.344206+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T06:03:57.398647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":343,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-05-13T23:14:16.253992+0000","last_change":"2022-05-13T23:13:02.805009+0000","last_active":"2022-05-13T23:14:16.253992+0000","last_peered":"2022-05-13T23:14:16.253992+0000","last_clean":"2022-05-13T23:14:16.253992+0000","last_became_active":"2022-05-13T23:13:02.804350+0000","last_became_peered":"2022-05-13T23:13:02.804350+0000","last_unstale":"2022-05-13T23:14:16.253992+0000","last_undegraded":"2022-05-13T23:14:16.253992+0000","last_fullsized":"2022-05-13T23:14:16.253992+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:09:32.001256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:09:32.001256+0000","last_clean_scrub_stamp":"2022-05-13T23:09:32.001256+0000","objects_scrubbed":0,"log_size":137,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T07:28:44.497144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1057856,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":10,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-05-13T23:14:16.267352+0000","last_change":"2022-05-13T23:14:16.261286+0000","last_active":"2022-05-13T23:14:16.267352+0000","last_peered":"2022-05-13T23:14:16.267352+0000","last_clean":"2022-05-13T23:14:16.267352+0000","last_became_active":"2022-05-13T23:14:16.260870+0000","last_became_peered":"2022-05-13T23:14:16.260870+0000","last_unstale":"2022-05-13T23:14:16.267352+0000","last_undegraded":"2022-05-13T23:14:16.267352+0000","last_fullsized":"2022-05-13T23:14:16.267352+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_clean_scrub_stamp":"2022-05-13T23:14:15.659362+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T08:14:34.804797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"83'1","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2022-05-13T23:14:17.561547+0000","last_change":"2022-05-13T23:14:15.686878+0000","last_active":"2022-05-13T23:14:17.561547+0000","last_peered":"2022-05-13T23:14:17.561547+0000","last_clean":"2022-05-13T23:14:17.561547+0000","last_became_active":"2022-05-13T23:14:15.686496+0000","last_became_peered":"2022-05-13T23:14:15.686496+0000","last_unstale":"2022-05-13T23:14:17.561547+0000","last_undegraded":"2022-05-13T23:14:17.561547+0000","last_fullsized":"2022-05-13T23:14:17.561547+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:14.344206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:14.344206+0000","last_clean_scrub_stamp":"2022-05-13T23:14:14.344206+0000","objects_scrubbed":0,"log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T09:19:28.089159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":10,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-05-13T23:14:15.698991+0000","last_change":"2022-05-13T23:14:15.698503+0000","last_active":"2022-05-13T23:14:15.698991+0000","last_peered":"2022-05-13T23:14:15.698991+0000","last_clean":"2022-05-13T23:14:15.698991+0000","last_became_active":"2022-05-13T23:14:15.698156+0000","last_became_peered":"2022-05-13T23:14:15.698156+0000","last_unstale":"2022-05-13T23:14:15.698991+0000","last_undegraded":"2022-05-13T23:14:15.698991+0000","last_fullsized":"2022-05-13T23:14:15.698991+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:14.344206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:14.344206+0000","last_clean_scrub_stamp":"2022-05-13T23:14:14.344206+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T09:03:21.133631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":10,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-05-13T23:14:16.271905+0000","last_change":"2022-05-13T23:14:16.270922+0000","last_active":"2022-05-13T23:14:16.271905+0000","last_peered":"2022-05-13T23:14:16.271905+0000","last_clean":"2022-05-13T23:14:16.271905+0000","last_became_active":"2022-05-13T23:14:16.270259+0000","last_became_peered":"2022-05-13T23:14:16.270259+0000","last_unstale":"2022-05-13T23:14:16.271905+0000","last_undegraded":"2022-05-13T23:14:16.271905+0000","last_fullsized":"2022-05-13T23:14:16.271905+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_clean_scrub_stamp":"2022-05-13T23:14:15.659362+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T02:48:22.255489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":10,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-05-13T23:14:15.698665+0000","last_change":"2022-05-13T23:14:15.696304+0000","last_active":"2022-05-13T23:14:15.698665+0000","last_peered":"2022-05-13T23:14:15.698665+0000","last_clean":"2022-05-13T23:14:15.698665+0000","last_became_active":"2022-05-13T23:14:15.695476+0000","last_became_peered":"2022-05-13T23:14:15.695476+0000","last_unstale":"2022-05-13T23:14:15.698665+0000","last_undegraded":"2022-05-13T23:14:15.698665+0000","last_fullsized":"2022-05-13T23:14:15.698665+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:14.344206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:14.344206+0000","last_clean_scrub_stamp":"2022-05-13T23:14:14.344206+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T08:48:32.931789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":10,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-05-13T23:14:16.262969+0000","last_change":"2022-05-13T23:14:16.257063+0000","last_active":"2022-05-13T23:14:16.262969+0000","last_peered":"2022-05-13T23:14:16.262969+0000","last_clean":"2022-05-13T23:14:16.262969+0000","last_became_active":"2022-05-13T23:14:16.256757+0000","last_became_peered":"2022-05-13T23:14:16.256757+0000","last_unstale":"2022-05-13T23:14:16.262969+0000","last_undegraded":"2022-05-13T23:14:16.262969+0000","last_fullsized":"2022-05-13T23:14:16.262969+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_clean_scrub_stamp":"2022-05-13T23:14:15.659362+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T09:24:32.369318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"82'2","reported_seq":13,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-05-13T23:14:16.272570+0000","last_change":"2022-05-13T23:14:15.685568+0000","last_active":"2022-05-13T23:14:16.272570+0000","last_peered":"2022-05-13T23:14:16.272570+0000","last_clean":"2022-05-13T23:14:16.272570+0000","last_became_active":"2022-05-13T23:14:15.685203+0000","last_became_peered":"2022-05-13T23:14:15.685203+0000","last_unstale":"2022-05-13T23:14:16.272570+0000","last_undegraded":"2022-05-13T23:14:16.272570+0000","last_fullsized":"2022-05-13T23:14:16.272570+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:14.344206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:14.344206+0000","last_clean_scrub_stamp":"2022-05-13T23:14:14.344206+0000","objects_scrubbed":0,"log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T03:55:33.271069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":3,"reported_epoch":82,"state":"creating+activating","last_fresh":"2022-05-13T23:14:16.252040+0000","last_change":"2022-05-13T23:14:16.252040+0000","last_active":"2022-05-13T23:14:15.659362+0000","last_peered":"2022-05-13T23:14:15.659362+0000","last_clean":"2022-05-13T23:14:15.659362+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2022-05-13T23:14:16.252040+0000","last_undegraded":"2022-05-13T23:14:16.252040+0000","last_fullsized":"2022-05-13T23:14:16.252040+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_clean_scrub_stamp":"2022-05-13T23:14:15.659362+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T03:25:42.803064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":["0","3","4"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":11,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-05-13T23:14:16.248471+0000","last_change":"2022-05-13T23:14:15.698443+0000","last_active":"2022-05-13T23:14:16.248471+0000","last_peered":"2022-05-13T23:14:16.248471+0000","last_clean":"2022-05-13T23:14:16.248471+0000","last_became_active":"2022-05-13T23:14:15.698158+0000","last_became_peered":"2022-05-13T23:14:15.698158+0000","last_unstale":"2022-05-13T23:14:16.248471+0000","last_undegraded":"2022-05-13T23:14:16.248471+0000","last_fullsized":"2022-05-13T23:14:16.248471+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:14.344206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:14.344206+0000","last_clean_scrub_stamp":"2022-05-13T23:14:14.344206+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T05:06:14.946233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":9,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-05-13T23:14:16.264110+0000","last_change":"2022-05-13T23:14:16.264110+0000","last_active":"2022-05-13T23:14:16.264110+0000","last_peered":"2022-05-13T23:14:16.264110+0000","last_clean":"2022-05-13T23:14:16.264110+0000","last_became_active":"2022-05-13T23:14:16.263830+0000","last_became_peered":"2022-05-13T23:14:16.263830+0000","last_unstale":"2022-05-13T23:14:16.264110+0000","last_undegraded":"2022-05-13T23:14:16.264110+0000","last_fullsized":"2022-05-13T23:14:16.264110+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_clean_scrub_stamp":"2022-05-13T23:14:15.659362+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T00:06:26.724597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"83'3","reported_seq":14,"reported_epoch":83,"state":"active+clean","last_fresh":"2022-05-13T23:14:19.292890+0000","last_change":"2022-05-13T23:14:15.693926+0000","last_active":"2022-05-13T23:14:19.292890+0000","last_peered":"2022-05-13T23:14:19.292890+0000","last_clean":"2022-05-13T23:14:19.292890+0000","last_became_active":"2022-05-13T23:14:15.692163+0000","last_became_peered":"2022-05-13T23:14:15.692163+0000","last_unstale":"2022-05-13T23:14:19.292890+0000","last_undegraded":"2022-05-13T23:14:19.292890+0000","last_fullsized":"2022-05-13T23:14:19.292890+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:14.344206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:14.344206+0000","last_clean_scrub_stamp":"2022-05-13T23:14:14.344206+0000","objects_scrubbed":0,"log_size":3,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T03:00:12.522714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":9,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-05-13T23:14:16.265409+0000","last_change":"2022-05-13T23:14:16.265409+0000","last_active":"2022-05-13T23:14:16.265409+0000","last_peered":"2022-05-13T23:14:16.265409+0000","last_clean":"2022-05-13T23:14:16.265409+0000","last_became_active":"2022-05-13T23:14:16.265151+0000","last_became_peered":"2022-05-13T23:14:16.265151+0000","last_unstale":"2022-05-13T23:14:16.265409+0000","last_undegraded":"2022-05-13T23:14:16.265409+0000","last_fullsized":"2022-05-13T23:14:16.265409+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_clean_scrub_stamp":"2022-05-13T23:14:15.659362+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T04:05:43.762623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":11,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-05-13T23:14:16.241423+0000","last_change":"2022-05-13T23:14:15.681975+0000","last_active":"2022-05-13T23:14:16.241423+0000","last_peered":"2022-05-13T23:14:16.241423+0000","last_clean":"2022-05-13T23:14:16.241423+0000","last_became_active":"2022-05-13T23:14:15.681391+0000","last_became_peered":"2022-05-13T23:14:15.681391+0000","last_unstale":"2022-05-13T23:14:16.241423+0000","last_undegraded":"2022-05-13T23:14:16.241423+0000","last_fullsized":"2022-05-13T23:14:16.241423+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:14.344206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:14.344206+0000","last_clean_scrub_stamp":"2022-05-13T23:14:14.344206+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T06:56:25.732024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":10,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-05-13T23:14:16.274936+0000","last_change":"2022-05-13T23:14:16.272443+0000","last_active":"2022-05-13T23:14:16.274936+0000","last_peered":"2022-05-13T23:14:16.274936+0000","last_clean":"2022-05-13T23:14:16.274936+0000","last_became_active":"2022-05-13T23:14:16.272031+0000","last_became_peered":"2022-05-13T23:14:16.272031+0000","last_unstale":"2022-05-13T23:14:16.274936+0000","last_undegraded":"2022-05-13T23:14:16.274936+0000","last_fullsized":"2022-05-13T23:14:16.274936+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_clean_scrub_stamp":"2022-05-13T23:14:15.659362+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T10:23:55.694108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"82'2","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-05-13T23:14:16.264699+0000","last_change":"2022-05-13T23:14:15.681877+0000","last_active":"2022-05-13T23:14:16.264699+0000","last_peered":"2022-05-13T23:14:16.264699+0000","last_clean":"2022-05-13T23:14:16.264699+0000","last_became_active":"2022-05-13T23:14:15.681579+0000","last_became_peered":"2022-05-13T23:14:15.681579+0000","last_unstale":"2022-05-13T23:14:16.264699+0000","last_undegraded":"2022-05-13T23:14:16.264699+0000","last_fullsized":"2022-05-13T23:14:16.264699+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:14.344206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:14.344206+0000","last_clean_scrub_stamp":"2022-05-13T23:14:14.344206+0000","objects_scrubbed":0,"log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T07:00:11.132049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":10,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-05-13T23:14:16.267797+0000","last_change":"2022-05-13T23:14:16.266877+0000","last_active":"2022-05-13T23:14:16.267797+0000","last_peered":"2022-05-13T23:14:16.267797+0000","last_clean":"2022-05-13T23:14:16.267797+0000","last_became_active":"2022-05-13T23:14:16.266551+0000","last_became_peered":"2022-05-13T23:14:16.266551+0000","last_unstale":"2022-05-13T23:14:16.267797+0000","last_undegraded":"2022-05-13T23:14:16.267797+0000","last_fullsized":"2022-05-13T23:14:16.267797+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_clean_scrub_stamp":"2022-05-13T23:14:15.659362+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T05:57:29.394410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"83'3","reported_seq":14,"reported_epoch":83,"state":"active+clean","last_fresh":"2022-05-13T23:14:20.302018+0000","last_change":"2022-05-13T23:14:15.699586+0000","last_active":"2022-05-13T23:14:20.302018+0000","last_peered":"2022-05-13T23:14:20.302018+0000","last_clean":"2022-05-13T23:14:20.302018+0000","last_became_active":"2022-05-13T23:14:15.699215+0000","last_became_peered":"2022-05-13T23:14:15.699215+0000","last_unstale":"2022-05-13T23:14:20.302018+0000","last_undegraded":"2022-05-13T23:14:20.302018+0000","last_fullsized":"2022-05-13T23:14:20.302018+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:14.344206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:14.344206+0000","last_clean_scrub_stamp":"2022-05-13T23:14:14.344206+0000","objects_scrubbed":0,"log_size":3,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T04:14:54.395550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":554,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":10,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-05-13T23:14:16.267333+0000","last_change":"2022-05-13T23:14:16.265730+0000","last_active":"2022-05-13T23:14:16.267333+0000","last_peered":"2022-05-13T23:14:16.267333+0000","last_clean":"2022-05-13T23:14:16.267333+0000","last_became_active":"2022-05-13T23:14:16.265435+0000","last_became_peered":"2022-05-13T23:14:16.265435+0000","last_unstale":"2022-05-13T23:14:16.267333+0000","last_undegraded":"2022-05-13T23:14:16.267333+0000","last_fullsized":"2022-05-13T23:14:16.267333+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_clean_scrub_stamp":"2022-05-13T23:14:15.659362+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T08:55:57.574132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"82'1","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-05-13T23:14:16.270336+0000","last_change":"2022-05-13T23:14:15.682402+0000","last_active":"2022-05-13T23:14:16.270336+0000","last_peered":"2022-05-13T23:14:16.270336+0000","last_clean":"2022-05-13T23:14:16.270336+0000","last_became_active":"2022-05-13T23:14:15.682105+0000","last_became_peered":"2022-05-13T23:14:15.682105+0000","last_unstale":"2022-05-13T23:14:16.270336+0000","last_undegraded":"2022-05-13T23:14:16.270336+0000","last_fullsized":"2022-05-13T23:14:16.270336+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:14.344206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:14.344206+0000","last_clean_scrub_stamp":"2022-05-13T23:14:14.344206+0000","objects_scrubbed":0,"log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T10:37:55.092658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":10,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-05-13T23:14:16.264851+0000","last_change":"2022-05-13T23:14:16.263001+0000","last_active":"2022-05-13T23:14:16.264851+0000","last_peered":"2022-05-13T23:14:16.264851+0000","last_clean":"2022-05-13T23:14:16.264851+0000","last_became_active":"2022-05-13T23:14:16.262646+0000","last_became_peered":"2022-05-13T23:14:16.262646+0000","last_unstale":"2022-05-13T23:14:16.264851+0000","last_undegraded":"2022-05-13T23:14:16.264851+0000","last_fullsized":"2022-05-13T23:14:16.264851+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_clean_scrub_stamp":"2022-05-13T23:14:15.659362+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T05:38:53.024966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"82'2","reported_seq":13,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-05-13T23:14:16.264403+0000","last_change":"2022-05-13T23:14:15.690434+0000","last_active":"2022-05-13T23:14:16.264403+0000","last_peered":"2022-05-13T23:14:16.264403+0000","last_clean":"2022-05-13T23:14:16.264403+0000","last_became_active":"2022-05-13T23:14:15.690032+0000","last_became_peered":"2022-05-13T23:14:15.690032+0000","last_unstale":"2022-05-13T23:14:16.264403+0000","last_undegraded":"2022-05-13T23:14:16.264403+0000","last_fullsized":"2022-05-13T23:14:16.264403+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:14.344206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:14.344206+0000","last_clean_scrub_stamp":"2022-05-13T23:14:14.344206+0000","objects_scrubbed":0,"log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T06:39:08.251311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":10,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-05-13T23:14:16.265645+0000","last_change":"2022-05-13T23:14:16.264791+0000","last_active":"2022-05-13T23:14:16.265645+0000","last_peered":"2022-05-13T23:14:16.265645+0000","last_clean":"2022-05-13T23:14:16.265645+0000","last_became_active":"2022-05-13T23:14:16.264461+0000","last_became_peered":"2022-05-13T23:14:16.264461+0000","last_unstale":"2022-05-13T23:14:16.265645+0000","last_undegraded":"2022-05-13T23:14:16.265645+0000","last_fullsized":"2022-05-13T23:14:16.265645+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_clean_scrub_stamp":"2022-05-13T23:14:15.659362+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T03:02:04.280220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":9,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-05-13T23:14:16.260468+0000","last_change":"2022-05-13T23:14:16.260468+0000","last_active":"2022-05-13T23:14:16.260468+0000","last_peered":"2022-05-13T23:14:16.260468+0000","last_clean":"2022-05-13T23:14:16.260468+0000","last_became_active":"2022-05-13T23:14:16.260211+0000","last_became_peered":"2022-05-13T23:14:16.260211+0000","last_unstale":"2022-05-13T23:14:16.260468+0000","last_undegraded":"2022-05-13T23:14:16.260468+0000","last_fullsized":"2022-05-13T23:14:16.260468+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_clean_scrub_stamp":"2022-05-13T23:14:15.659362+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T05:26:57.769099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":10,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-05-13T23:14:16.254999+0000","last_change":"2022-05-13T23:14:16.253997+0000","last_active":"2022-05-13T23:14:16.254999+0000","last_peered":"2022-05-13T23:14:16.254999+0000","last_clean":"2022-05-13T23:14:16.254999+0000","last_became_active":"2022-05-13T23:14:16.253685+0000","last_became_peered":"2022-05-13T23:14:16.253685+0000","last_unstale":"2022-05-13T23:14:16.254999+0000","last_undegraded":"2022-05-13T23:14:16.254999+0000","last_fullsized":"2022-05-13T23:14:16.254999+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_clean_scrub_stamp":"2022-05-13T23:14:15.659362+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T07:42:26.765244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":10,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-05-13T23:14:16.266256+0000","last_change":"2022-05-13T23:14:16.265207+0000","last_active":"2022-05-13T23:14:16.266256+0000","last_peered":"2022-05-13T23:14:16.266256+0000","last_clean":"2022-05-13T23:14:16.266256+0000","last_became_active":"2022-05-13T23:14:16.264881+0000","last_became_peered":"2022-05-13T23:14:16.264881+0000","last_unstale":"2022-05-13T23:14:16.266256+0000","last_undegraded":"2022-05-13T23:14:16.266256+0000","last_fullsized":"2022-05-13T23:14:16.266256+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_clean_scrub_stamp":"2022-05-13T23:14:15.659362+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T04:01:48.894233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":3,"reported_epoch":82,"state":"creating+activating","last_fresh":"2022-05-13T23:14:16.249811+0000","last_change":"2022-05-13T23:14:16.249811+0000","last_active":"2022-05-13T23:14:15.659362+0000","last_peered":"2022-05-13T23:14:15.659362+0000","last_clean":"2022-05-13T23:14:15.659362+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2022-05-13T23:14:16.249811+0000","last_undegraded":"2022-05-13T23:14:16.249811+0000","last_fullsized":"2022-05-13T23:14:16.249811+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_clean_scrub_stamp":"2022-05-13T23:14:15.659362+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T03:11:10.799827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":["0","4","6"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":10,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-05-13T23:14:16.268810+0000","last_change":"2022-05-13T23:14:16.265421+0000","last_active":"2022-05-13T23:14:16.268810+0000","last_peered":"2022-05-13T23:14:16.268810+0000","last_clean":"2022-05-13T23:14:16.268810+0000","last_became_active":"2022-05-13T23:14:16.265057+0000","last_became_peered":"2022-05-13T23:14:16.265057+0000","last_unstale":"2022-05-13T23:14:16.268810+0000","last_undegraded":"2022-05-13T23:14:16.268810+0000","last_fullsized":"2022-05-13T23:14:16.268810+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_clean_scrub_stamp":"2022-05-13T23:14:15.659362+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T07:45:53.177341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":9,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-05-13T23:14:16.266317+0000","last_change":"2022-05-13T23:14:16.266317+0000","last_active":"2022-05-13T23:14:16.266317+0000","last_peered":"2022-05-13T23:14:16.266317+0000","last_clean":"2022-05-13T23:14:16.266317+0000","last_became_active":"2022-05-13T23:14:16.265959+0000","last_became_peered":"2022-05-13T23:14:16.265959+0000","last_unstale":"2022-05-13T23:14:16.266317+0000","last_undegraded":"2022-05-13T23:14:16.266317+0000","last_fullsized":"2022-05-13T23:14:16.266317+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_clean_scrub_stamp":"2022-05-13T23:14:15.659362+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T03:37:42.277033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":10,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-05-13T23:14:16.255848+0000","last_change":"2022-05-13T23:14:16.254350+0000","last_active":"2022-05-13T23:14:16.255848+0000","last_peered":"2022-05-13T23:14:16.255848+0000","last_clean":"2022-05-13T23:14:16.255848+0000","last_became_active":"2022-05-13T23:14:16.253940+0000","last_became_peered":"2022-05-13T23:14:16.253940+0000","last_unstale":"2022-05-13T23:14:16.255848+0000","last_undegraded":"2022-05-13T23:14:16.255848+0000","last_fullsized":"2022-05-13T23:14:16.255848+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_clean_scrub_stamp":"2022-05-13T23:14:15.659362+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T10:08:56.052316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":10,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-05-13T23:14:16.267243+0000","last_change":"2022-05-13T23:14:16.263153+0000","last_active":"2022-05-13T23:14:16.267243+0000","last_peered":"2022-05-13T23:14:16.267243+0000","last_clean":"2022-05-13T23:14:16.267243+0000","last_became_active":"2022-05-13T23:14:16.262652+0000","last_became_peered":"2022-05-13T23:14:16.262652+0000","last_unstale":"2022-05-13T23:14:16.267243+0000","last_undegraded":"2022-05-13T23:14:16.267243+0000","last_fullsized":"2022-05-13T23:14:16.267243+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_clean_scrub_stamp":"2022-05-13T23:14:15.659362+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T07:53:46.499880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":10,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-05-13T23:14:16.271182+0000","last_change":"2022-05-13T23:14:16.270358+0000","last_active":"2022-05-13T23:14:16.271182+0000","last_peered":"2022-05-13T23:14:16.271182+0000","last_clean":"2022-05-13T23:14:16.271182+0000","last_became_active":"2022-05-13T23:14:16.270099+0000","last_became_peered":"2022-05-13T23:14:16.270099+0000","last_unstale":"2022-05-13T23:14:16.271182+0000","last_undegraded":"2022-05-13T23:14:16.271182+0000","last_fullsized":"2022-05-13T23:14:16.271182+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_clean_scrub_stamp":"2022-05-13T23:14:15.659362+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T00:42:07.782569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":10,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-05-13T23:14:16.253696+0000","last_change":"2022-05-13T23:14:16.251188+0000","last_active":"2022-05-13T23:14:16.253696+0000","last_peered":"2022-05-13T23:14:16.253696+0000","last_clean":"2022-05-13T23:14:16.253696+0000","last_became_active":"2022-05-13T23:14:16.250816+0000","last_became_peered":"2022-05-13T23:14:16.250816+0000","last_unstale":"2022-05-13T23:14:16.253696+0000","last_undegraded":"2022-05-13T23:14:16.253696+0000","last_fullsized":"2022-05-13T23:14:16.253696+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_clean_scrub_stamp":"2022-05-13T23:14:15.659362+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T00:58:14.898104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":10,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-05-13T23:14:16.265927+0000","last_change":"2022-05-13T23:14:16.264125+0000","last_active":"2022-05-13T23:14:16.265927+0000","last_peered":"2022-05-13T23:14:16.265927+0000","last_clean":"2022-05-13T23:14:16.265927+0000","last_became_active":"2022-05-13T23:14:16.263806+0000","last_became_peered":"2022-05-13T23:14:16.263806+0000","last_unstale":"2022-05-13T23:14:16.265927+0000","last_undegraded":"2022-05-13T23:14:16.265927+0000","last_fullsized":"2022-05-13T23:14:16.265927+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_clean_scrub_stamp":"2022-05-13T23:14:15.659362+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T09:26:04.559765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":10,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-05-13T23:14:16.268902+0000","last_change":"2022-05-13T23:14:16.266991+0000","last_active":"2022-05-13T23:14:16.268902+0000","last_peered":"2022-05-13T23:14:16.268902+0000","last_clean":"2022-05-13T23:14:16.268902+0000","last_became_active":"2022-05-13T23:14:16.265551+0000","last_became_peered":"2022-05-13T23:14:16.265551+0000","last_unstale":"2022-05-13T23:14:16.268902+0000","last_undegraded":"2022-05-13T23:14:16.268902+0000","last_fullsized":"2022-05-13T23:14:16.268902+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_clean_scrub_stamp":"2022-05-13T23:14:15.659362+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T00:40:56.664387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":9,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-05-13T23:14:16.267302+0000","last_change":"2022-05-13T23:14:16.267302+0000","last_active":"2022-05-13T23:14:16.267302+0000","last_peered":"2022-05-13T23:14:16.267302+0000","last_clean":"2022-05-13T23:14:16.267302+0000","last_became_active":"2022-05-13T23:14:16.266151+0000","last_became_peered":"2022-05-13T23:14:16.266151+0000","last_unstale":"2022-05-13T23:14:16.267302+0000","last_undegraded":"2022-05-13T23:14:16.267302+0000","last_fullsized":"2022-05-13T23:14:16.267302+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_clean_scrub_stamp":"2022-05-13T23:14:15.659362+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T07:30:04.546337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1057856,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":3194880,"data_stored":3173568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":4168,"num_objects":35,"num_object_clones":0,"num_object_copies":105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":74,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":21,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":147456,"data_stored":10040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":66,"ondisk_log_size":66,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":78,"seq":335007449101,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":5856,"kb_used_data":856,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":93739296,"statfs":{"total":95995035648,"available":95989039104,"internally_reserved":0,"allocated":876544,"data_stored":416256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41199999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.437}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.30099999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42999999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47199999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38300000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51100000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.502}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.373}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57099999999999995}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54600000000000004}]}]},{"osd":10,"up_from":71,"seq":304942678034,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6600,"kb_used_data":1408,"kb_used_omap":0,"kb_used_meta":5184,"kb_avail":93738552,"statfs":{"total":95995035648,"available":95988277248,"internally_reserved":0,"allocated":1441792,"data_stored":955149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5308416},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.755}]},{"osd":1,"last update":"Fri May 13 23:14:05 2022","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.72899999999999998,"5min":0.72899999999999998,"15min":0.72899999999999998},"last":0.40600000000000003},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.57999999999999996,"15min":0.57999999999999996},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.83099999999999996,"5min":0.83099999999999996,"15min":0.83099999999999996},"last":0.42299999999999999}]},{"osd":2,"last update":"Fri May 13 23:14:05 2022","interfaces":[{"interface":"back","average":{"1min":0.69499999999999995,"5min":0.69499999999999995,"15min":0.69499999999999995},"min":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":1.3280000000000001,"5min":1.3280000000000001,"15min":1.3280000000000001},"last":0.52600000000000002},{"interface":"front","average":{"1min":0.70799999999999996,"5min":0.70799999999999996,"15min":0.70799999999999996},"min":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":0.88800000000000001,"5min":0.88800000000000001,"15min":0.88800000000000001},"last":0.64800000000000002}]},{"osd":3,"last update":"Fri May 13 23:14:05 2022","interfaces":[{"interface":"back","average":{"1min":0.73999999999999999,"5min":0.73999999999999999,"15min":0.73999999999999999},"min":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"max":{"1min":0.90100000000000002,"5min":0.90100000000000002,"15min":0.90100000000000002},"last":0.59099999999999997},{"interface":"front","average":{"1min":0.77000000000000002,"5min":0.77000000000000002,"15min":0.77000000000000002},"min":{"1min":0.60199999999999998,"5min":0.60199999999999998,"15min":0.60199999999999998},"max":{"1min":0.93500000000000005,"5min":0.93500000000000005,"15min":0.93500000000000005},"last":0.55800000000000005}]},{"osd":4,"last update":"Fri May 13 23:14:05 2022","interfaces":[{"interface":"back","average":{"1min":0.68000000000000005,"5min":0.68000000000000005,"15min":0.68000000000000005},"min":{"1min":0.432,"5min":0.432,"15min":0.432},"max":{"1min":0.97699999999999998,"5min":0.97699999999999998,"15min":0.97699999999999998},"last":0.51800000000000002},{"interface":"front","average":{"1min":0.65200000000000002,"5min":0.65200000000000002,"15min":0.65200000000000002},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":1.0720000000000001,"5min":1.0720000000000001,"15min":1.0720000000000001},"last":0.46600000000000003}]},{"osd":5,"last update":"Fri May 13 23:14:05 2022","interfaces":[{"interface":"back","average":{"1min":0.78400000000000003,"5min":0.78400000000000003,"15min":0.78400000000000003},"min":{"1min":0.496,"5min":0.496,"15min":0.496},"max":{"1min":1.4299999999999999,"5min":1.4299999999999999,"15min":1.4299999999999999},"last":0.58099999999999996},{"interface":"front","average":{"1min":0.79800000000000004,"5min":0.79800000000000004,"15min":0.79800000000000004},"min":{"1min":0.55400000000000005,"5min":0.55400000000000005,"15min":0.55400000000000005},"max":{"1min":1.0609999999999999,"5min":1.0609999999999999,"15min":1.0609999999999999},"last":0.68799999999999994}]},{"osd":6,"last update":"Fri May 13 23:14:05 2022","interfaces":[{"interface":"back","average":{"1min":0.88700000000000001,"5min":0.88700000000000001,"15min":0.88700000000000001},"min":{"1min":0.64600000000000002,"5min":0.64600000000000002,"15min":0.64600000000000002},"max":{"1min":1.157,"5min":1.157,"15min":1.157},"last":0.54200000000000004},{"interface":"front","average":{"1min":0.88500000000000001,"5min":0.88500000000000001,"15min":0.88500000000000001},"min":{"1min":0.60199999999999998,"5min":0.60199999999999998,"15min":0.60199999999999998},"max":{"1min":1.216,"5min":1.216,"15min":1.216},"last":0.66100000000000003}]},{"osd":7,"last update":"Fri May 13 23:14:05 2022","interfaces":[{"interface":"back","average":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":1.585,"5min":1.585,"15min":1.585},"last":0.496},{"interface":"front","average":{"1min":0.80300000000000005,"5min":0.80300000000000005,"15min":0.80300000000000005},"min":{"1min":0.47999999999999998,"5min":0.47999999999999998,"15min":0.47999999999999998},"max":{"1min":1.0469999999999999,"5min":1.0469999999999999,"15min":1.0469999999999999},"last":0.502}]},{"osd":8,"last update":"Fri May 13 23:14:05 2022","interfaces":[{"interface":"back","average":{"1min":0.86399999999999999,"5min":0.86399999999999999,"15min":0.86399999999999999},"min":{"1min":0.65900000000000003,"5min":0.65900000000000003,"15min":0.65900000000000003},"max":{"1min":1.163,"5min":1.163,"15min":1.163},"last":0.73299999999999998},{"interface":"front","average":{"1min":0.88200000000000001,"5min":0.88200000000000001,"15min":0.88200000000000001},"min":{"1min":0.51400000000000001,"5min":0.51400000000000001,"15min":0.51400000000000001},"max":{"1min":1.476,"5min":1.476,"15min":1.476},"last":0.71199999999999997}]},{"osd":9,"last update":"Fri May 13 23:14:05 2022","interfaces":[{"interface":"back","average":{"1min":0.94999999999999996,"5min":0.94999999999999996,"15min":0.94999999999999996},"min":{"1min":0.68999999999999995,"5min":0.68999999999999995,"15min":0.68999999999999995},"max":{"1min":1.369,"5min":1.369,"15min":1.369},"last":0.77000000000000002},{"interface":"front","average":{"1min":0.96799999999999997,"5min":0.96799999999999997,"15min":0.96799999999999997},"min":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":1.819,"5min":1.819,"15min":1.819},"last":0.67300000000000004}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72099999999999997}]}]},{"osd":9,"up_from":66,"seq":283467841557,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":5908,"kb_used_data":844,"kb_used_omap":0,"kb_used_meta":5056,"kb_avail":93739244,"statfs":{"total":95995035648,"available":95988985856,"internally_reserved":0,"allocated":864256,"data_stored":407191,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5177344},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri May 13 23:13:37 2022","interfaces":[{"interface":"back","average":{"1min":0.65200000000000002,"5min":0.65200000000000002,"15min":0.65200000000000002},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":1.024,"5min":1.024,"15min":1.024},"last":0.52100000000000002},{"interface":"front","average":{"1min":0.72299999999999998,"5min":0.72299999999999998,"15min":0.72299999999999998},"min":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":1.044,"5min":1.044,"15min":1.044},"last":0.64500000000000002}]},{"osd":1,"last update":"Fri May 13 23:13:37 2022","interfaces":[{"interface":"back","average":{"1min":0.755,"5min":0.755,"15min":0.755},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":1.3029999999999999,"5min":1.3029999999999999,"15min":1.3029999999999999},"last":0.57599999999999996},{"interface":"front","average":{"1min":0.92000000000000004,"5min":0.92000000000000004,"15min":0.92000000000000004},"min":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":1.498,"5min":1.498,"15min":1.498},"last":0.68500000000000005}]},{"osd":2,"last update":"Fri May 13 23:13:37 2022","interfaces":[{"interface":"back","average":{"1min":0.89900000000000002,"5min":0.89900000000000002,"15min":0.89900000000000002},"min":{"1min":0.58699999999999997,"5min":0.58699999999999997,"15min":0.58699999999999997},"max":{"1min":1.4419999999999999,"5min":1.4419999999999999,"15min":1.4419999999999999},"last":0.59399999999999997},{"interface":"front","average":{"1min":0.82699999999999996,"5min":0.82699999999999996,"15min":0.82699999999999996},"min":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"max":{"1min":1.5309999999999999,"5min":1.5309999999999999,"15min":1.5309999999999999},"last":0.48199999999999998}]},{"osd":3,"last update":"Fri May 13 23:13:37 2022","interfaces":[{"interface":"back","average":{"1min":0.878,"5min":0.878,"15min":0.878},"min":{"1min":0.51200000000000001,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":1.2410000000000001,"5min":1.2410000000000001,"15min":1.2410000000000001},"last":0.78900000000000003},{"interface":"front","average":{"1min":1.002,"5min":1.002,"15min":1.002},"min":{"1min":0.622,"5min":0.622,"15min":0.622},"max":{"1min":2.2069999999999999,"5min":2.2069999999999999,"15min":2.2069999999999999},"last":0.80600000000000005}]},{"osd":4,"last update":"Fri May 13 23:13:37 2022","interfaces":[{"interface":"back","average":{"1min":0.93000000000000005,"5min":0.93000000000000005,"15min":0.93000000000000005},"min":{"1min":0.47699999999999998,"5min":0.47699999999999998,"15min":0.47699999999999998},"max":{"1min":1.5600000000000001,"5min":1.5600000000000001,"15min":1.5600000000000001},"last":0.65500000000000003},{"interface":"front","average":{"1min":0.95799999999999996,"5min":0.95799999999999996,"15min":0.95799999999999996},"min":{"1min":0.58899999999999997,"5min":0.58899999999999997,"15min":0.58899999999999997},"max":{"1min":1.3320000000000001,"5min":1.3320000000000001,"15min":1.3320000000000001},"last":0.81899999999999995}]},{"osd":5,"last update":"Fri May 13 23:13:37 2022","interfaces":[{"interface":"back","average":{"1min":0.97099999999999997,"5min":0.97099999999999997,"15min":0.97099999999999997},"min":{"1min":0.56299999999999994,"5min":0.56299999999999994,"15min":0.56299999999999994},"max":{"1min":1.593,"5min":1.593,"15min":1.593},"last":0.61699999999999999},{"interface":"front","average":{"1min":1.02,"5min":1.02,"15min":1.02},"min":{"1min":0.68600000000000005,"5min":0.68600000000000005,"15min":0.68600000000000005},"max":{"1min":1.3700000000000001,"5min":1.3700000000000001,"15min":1.3700000000000001},"last":0.84499999999999997}]},{"osd":6,"last update":"Fri May 13 23:13:37 2022","interfaces":[{"interface":"back","average":{"1min":1.1399999999999999,"5min":1.1399999999999999,"15min":1.1399999999999999},"min":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"max":{"1min":2.254,"5min":2.254,"15min":2.254},"last":0.83499999999999996},{"interface":"front","average":{"1min":1.0609999999999999,"5min":1.0609999999999999,"15min":1.0609999999999999},"min":{"1min":0.70599999999999996,"5min":0.70599999999999996,"15min":0.70599999999999996},"max":{"1min":1.649,"5min":1.649,"15min":1.649},"last":0.76100000000000001}]},{"osd":7,"last update":"Fri May 13 23:13:37 2022","interfaces":[{"interface":"back","average":{"1min":1.119,"5min":1.119,"15min":1.119},"min":{"1min":0.63,"5min":0.63,"15min":0.63},"max":{"1min":2.101,"5min":2.101,"15min":2.101},"last":0.91700000000000004},{"interface":"front","average":{"1min":1.1240000000000001,"5min":1.1240000000000001,"15min":1.1240000000000001},"min":{"1min":0.69599999999999995,"5min":0.69599999999999995,"15min":0.69599999999999995},"max":{"1min":1.895,"5min":1.895,"15min":1.895},"last":0.90900000000000003}]},{"osd":8,"last update":"Fri May 13 23:13:37 2022","interfaces":[{"interface":"back","average":{"1min":1.069,"5min":1.069,"15min":1.069},"min":{"1min":0.59899999999999998,"5min":0.59899999999999998,"15min":0.59899999999999998},"max":{"1min":1.6140000000000001,"5min":1.6140000000000001,"15min":1.6140000000000001},"last":0.86599999999999999},{"interface":"front","average":{"1min":1.1419999999999999,"5min":1.1419999999999999,"15min":1.1419999999999999},"min":{"1min":0.71599999999999997,"5min":0.71599999999999997,"15min":0.71599999999999997},"max":{"1min":2.1789999999999998,"5min":2.1789999999999998,"15min":2.1789999999999998},"last":0.88500000000000001}]},{"osd":10,"last update":"Fri May 13 23:14:02 2022","interfaces":[{"interface":"back","average":{"1min":1.1040000000000001,"5min":1.1040000000000001,"15min":1.1040000000000001},"min":{"1min":0.66800000000000004,"5min":0.66800000000000004,"15min":0.66800000000000004},"max":{"1min":1.7829999999999999,"5min":1.7829999999999999,"15min":1.7829999999999999},"last":0.82999999999999996},{"interface":"front","average":{"1min":1.1319999999999999,"5min":1.1319999999999999,"15min":1.1319999999999999},"min":{"1min":0.73099999999999998,"5min":0.73099999999999998,"15min":0.73099999999999998},"max":{"1min":1.748,"5min":1.748,"15min":1.748},"last":0.94799999999999995}]}]},{"osd":8,"up_from":58,"seq":249108103197,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6036,"kb_used_data":844,"kb_used_omap":0,"kb_used_meta":5184,"kb_avail":93739116,"statfs":{"total":95995035648,"available":95988854784,"internally_reserved":0,"allocated":864256,"data_stored":407191,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5308416},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri May 13 23:14:08 2022","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.60399999999999998,"15min":0.60799999999999998},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.88900000000000001,"5min":0.88900000000000001,"15min":0.88900000000000001},"last":0.52600000000000002},{"interface":"front","average":{"1min":0.61699999999999999,"5min":0.59199999999999997,"15min":0.58799999999999997},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":1.137,"5min":1.137,"15min":1.137},"last":0.48199999999999998}]},{"osd":1,"last update":"Fri May 13 23:14:08 2022","interfaces":[{"interface":"back","average":{"1min":0.53900000000000003,"5min":0.56699999999999995,"15min":0.57099999999999995},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.67200000000000004,"5min":0.78900000000000003,"15min":0.78900000000000003},"last":0.59199999999999997},{"interface":"front","average":{"1min":0.57299999999999995,"5min":0.60199999999999998,"15min":0.60699999999999998},"min":{"1min":0.38900000000000001,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.72599999999999998,"5min":0.80700000000000005,"15min":0.80700000000000005},"last":0.69399999999999995}]},{"osd":2,"last update":"Fri May 13 23:14:08 2022","interfaces":[{"interface":"back","average":{"1min":0.44,"5min":0.45600000000000002,"15min":0.45900000000000002},"min":{"1min":0.24199999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.77400000000000002,"5min":0.77400000000000002,"15min":0.77400000000000002},"last":0.46500000000000002},{"interface":"front","average":{"1min":0.443,"5min":0.45600000000000002,"15min":0.45800000000000002},"min":{"1min":0.27800000000000002,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.79100000000000004,"5min":0.79100000000000004,"15min":0.79100000000000004},"last":0.45300000000000001}]},{"osd":3,"last update":"Fri May 13 23:14:08 2022","interfaces":[{"interface":"back","average":{"1min":0.626,"5min":0.64800000000000002,"15min":0.65200000000000002},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.88400000000000001,"5min":0.88400000000000001,"15min":0.88400000000000001},"last":0.59799999999999998},{"interface":"front","average":{"1min":0.63300000000000001,"5min":0.64300000000000002,"15min":0.64500000000000002},"min":{"1min":0.439,"5min":0.439,"15min":0.439},"max":{"1min":0.82899999999999996,"5min":0.83299999999999996,"15min":0.83299999999999996},"last":0.504}]},{"osd":4,"last update":"Fri May 13 23:14:08 2022","interfaces":[{"interface":"back","average":{"1min":0.61099999999999999,"5min":0.621,"15min":0.623},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":1.573,"5min":1.573,"15min":1.573},"last":0.58199999999999996},{"interface":"front","average":{"1min":0.57299999999999995,"5min":0.61899999999999999,"15min":0.627},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":0.746,"5min":0.90500000000000003,"15min":0.90500000000000003},"last":0.70199999999999996}]},{"osd":5,"last update":"Fri May 13 23:14:08 2022","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.57399999999999995,"15min":0.58299999999999996},"min":{"1min":0.22800000000000001,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":0.85199999999999998,"5min":0.875,"15min":0.875},"last":0.49099999999999999},{"interface":"front","average":{"1min":0.498,"5min":0.56299999999999994,"15min":0.57399999999999995},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.86199999999999999,"5min":0.88200000000000001,"15min":0.88200000000000001},"last":0.44}]},{"osd":6,"last update":"Fri May 13 23:14:08 2022","interfaces":[{"interface":"back","average":{"1min":0.70299999999999996,"5min":0.74199999999999999,"15min":0.748},"min":{"1min":0.44400000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":0.873,"5min":0.94099999999999995,"15min":0.94099999999999995},"last":0.53800000000000003},{"interface":"front","average":{"1min":0.72099999999999997,"5min":0.754,"15min":0.76000000000000001},"min":{"1min":0.51700000000000002,"5min":0.504,"15min":0.504},"max":{"1min":0.91700000000000004,"5min":0.95699999999999996,"15min":0.95699999999999996},"last":0.58599999999999997}]},{"osd":7,"last update":"Fri May 13 23:14:08 2022","interfaces":[{"interface":"back","average":{"1min":0.67100000000000004,"5min":0.69999999999999996,"15min":0.70499999999999996},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.94799999999999995,"5min":0.95799999999999996,"15min":0.95799999999999996},"last":0.63500000000000001},{"interface":"front","average":{"1min":0.70399999999999996,"5min":0.69999999999999996,"15min":0.69999999999999996},"min":{"1min":0.49099999999999999,"5min":0.437,"15min":0.437},"max":{"1min":1.609,"5min":1.609,"15min":1.609},"last":0.72899999999999998}]},{"osd":9,"last update":"Fri May 13 23:13:38 2022","interfaces":[{"interface":"back","average":{"1min":0.76500000000000001,"5min":0.76500000000000001,"15min":0.76500000000000001},"min":{"1min":0.51500000000000001,"5min":0.51500000000000001,"15min":0.51500000000000001},"max":{"1min":0.999,"5min":0.999,"15min":0.999},"last":0.68600000000000005},{"interface":"front","average":{"1min":0.80100000000000005,"5min":0.80100000000000005,"15min":0.80100000000000005},"min":{"1min":0.59899999999999998,"5min":0.59899999999999998,"15min":0.59899999999999998},"max":{"1min":0.996,"5min":0.996,"15min":0.996},"last":0.67900000000000005}]},{"osd":10,"last update":"Fri May 13 23:14:02 2022","interfaces":[{"interface":"back","average":{"1min":0.72699999999999998,"5min":0.72699999999999998,"15min":0.72699999999999998},"min":{"1min":0.45800000000000002,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":0.97799999999999998,"5min":0.97799999999999998,"15min":0.97799999999999998},"last":0.65700000000000003},{"interface":"front","average":{"1min":0.69699999999999995,"5min":0.69699999999999995,"15min":0.69699999999999995},"min":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":0.998,"5min":0.998,"15min":0.998},"last":0.70899999999999996}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55600000000000005}]}]},{"osd":7,"up_from":51,"seq":219043332128,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6776,"kb_used_data":1392,"kb_used_omap":0,"kb_used_meta":5376,"kb_avail":93738376,"statfs":{"total":95995035648,"available":95988097024,"internally_reserved":0,"allocated":1425408,"data_stored":954296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5505024},"hb_peers":[0,1,2,3,4,5,6,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri May 13 23:13:48 2022","interfaces":[{"interface":"back","average":{"1min":0.64400000000000002,"5min":0.56000000000000005,"15min":0.54600000000000004},"min":{"1min":0.48399999999999999,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":0.82599999999999996,"5min":0.82599999999999996,"15min":0.82599999999999996},"last":0.63600000000000001},{"interface":"front","average":{"1min":0.60899999999999999,"5min":0.58799999999999997,"15min":0.58399999999999996},"min":{"1min":0.42699999999999999,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":0.78800000000000003,"5min":0.86599999999999999,"15min":0.86599999999999999},"last":0.51900000000000002}]},{"osd":1,"last update":"Fri May 13 23:13:48 2022","interfaces":[{"interface":"back","average":{"1min":0.47499999999999998,"5min":0.44700000000000001,"15min":0.442},"min":{"1min":0.29999999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.79800000000000004,"5min":0.79800000000000004,"15min":0.79800000000000004},"last":0.42299999999999999},{"interface":"front","average":{"1min":0.49299999999999999,"5min":0.47299999999999998,"15min":0.46999999999999997},"min":{"1min":0.317,"5min":0.317,"15min":0.317},"max":{"1min":0.88800000000000001,"5min":0.88800000000000001,"15min":0.88800000000000001},"last":0.46500000000000002}]},{"osd":2,"last update":"Fri May 13 23:13:48 2022","interfaces":[{"interface":"back","average":{"1min":0.63100000000000001,"5min":0.63500000000000001,"15min":0.63600000000000001},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.83799999999999997,"5min":0.88800000000000001,"15min":0.88800000000000001},"last":0.55400000000000005},{"interface":"front","average":{"1min":0.60699999999999998,"5min":0.60299999999999998,"15min":0.60299999999999998},"min":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":0.78600000000000003,"5min":0.81599999999999995,"15min":0.81599999999999995},"last":0.50900000000000001}]},{"osd":3,"last update":"Fri May 13 23:13:48 2022","interfaces":[{"interface":"back","average":{"1min":0.66300000000000003,"5min":0.63500000000000001,"15min":0.63},"min":{"1min":0.50700000000000001,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":0.92200000000000004,"5min":0.92200000000000004,"15min":0.92200000000000004},"last":0.54200000000000004},{"interface":"front","average":{"1min":0.69399999999999995,"5min":0.69699999999999995,"15min":0.69699999999999995},"min":{"1min":0.49199999999999999,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":0.872,"5min":0.872,"15min":0.872},"last":0.56899999999999995}]},{"osd":4,"last update":"Fri May 13 23:13:48 2022","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.58399999999999996,"15min":0.59199999999999997},"min":{"1min":0.315,"5min":0.315,"15min":0.315},"max":{"1min":0.77800000000000002,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.40100000000000002},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.58199999999999996,"15min":0.58799999999999997},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.875,"5min":0.875,"15min":0.875},"last":0.438}]},{"osd":5,"last update":"Fri May 13 23:13:48 2022","interfaces":[{"interface":"back","average":{"1min":0.68100000000000005,"5min":0.72099999999999997,"15min":0.72699999999999998},"min":{"1min":0.47199999999999998,"5min":0.47199999999999998,"15min":0.47199999999999998},"max":{"1min":0.93300000000000005,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.59899999999999998},{"interface":"front","average":{"1min":0.68100000000000005,"5min":0.70099999999999996,"15min":0.70399999999999996},"min":{"1min":0.47999999999999998,"5min":0.47999999999999998,"15min":0.47999999999999998},"max":{"1min":0.93999999999999995,"5min":0.93999999999999995,"15min":0.93999999999999995},"last":0.58899999999999997}]},{"osd":6,"last update":"Fri May 13 23:13:48 2022","interfaces":[{"interface":"back","average":{"1min":0.76300000000000001,"5min":0.76200000000000001,"15min":0.76200000000000001},"min":{"1min":0.52700000000000002,"5min":0.52700000000000002,"15min":0.52700000000000002},"max":{"1min":1.0700000000000001,"5min":1.0700000000000001,"15min":1.0700000000000001},"last":0.61499999999999999},{"interface":"front","average":{"1min":0.81999999999999995,"5min":0.79700000000000004,"15min":0.79300000000000004},"min":{"1min":0.67600000000000005,"5min":0.63200000000000001,"15min":0.63200000000000001},"max":{"1min":1.04,"5min":1.1319999999999999,"15min":1.1319999999999999},"last":0.68999999999999995}]},{"osd":8,"last update":"Fri May 13 23:14:07 2022","interfaces":[{"interface":"back","average":{"1min":0.90600000000000003,"5min":0.77200000000000002,"15min":0.75},"min":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"max":{"1min":4.0739999999999998,"5min":4.0739999999999998,"15min":4.0739999999999998},"last":0.67100000000000004},{"interface":"front","average":{"1min":0.78100000000000003,"5min":0.76500000000000001,"15min":0.76200000000000001},"min":{"1min":0.497,"5min":0.497,"15min":0.497},"max":{"1min":1.046,"5min":1.046,"15min":1.046},"last":0.64700000000000002}]},{"osd":9,"last update":"Fri May 13 23:13:37 2022","interfaces":[{"interface":"back","average":{"1min":0.82699999999999996,"5min":0.82699999999999996,"15min":0.82699999999999996},"min":{"1min":0.57099999999999995,"5min":0.57099999999999995,"15min":0.57099999999999995},"max":{"1min":1.1339999999999999,"5min":1.1339999999999999,"15min":1.1339999999999999},"last":0.77000000000000002},{"interface":"front","average":{"1min":0.82899999999999996,"5min":0.82899999999999996,"15min":0.82899999999999996},"min":{"1min":0.64900000000000002,"5min":0.64900000000000002,"15min":0.64900000000000002},"max":{"1min":1.0409999999999999,"5min":1.0409999999999999,"15min":1.0409999999999999},"last":0.66400000000000003}]},{"osd":10,"last update":"Fri May 13 23:14:00 2022","interfaces":[{"interface":"back","average":{"1min":0.77400000000000002,"5min":0.77400000000000002,"15min":0.77400000000000002},"min":{"1min":0.47599999999999998,"5min":0.47599999999999998,"15min":0.47599999999999998},"max":{"1min":1.093,"5min":1.093,"15min":1.093},"last":0.70899999999999996},{"interface":"front","average":{"1min":0.72899999999999998,"5min":0.72899999999999998,"15min":0.72899999999999998},"min":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"max":{"1min":1.0529999999999999,"5min":1.0529999999999999,"15min":1.0529999999999999},"last":0.627}]}]},{"osd":6,"up_from":46,"seq":197568495654,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6124,"kb_used_data":868,"kb_used_omap":0,"kb_used_meta":5248,"kb_avail":93739028,"statfs":{"total":95995035648,"available":95988764672,"internally_reserved":0,"allocated":888832,"data_stored":424814,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5373952},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri May 13 23:13:27 2022","interfaces":[{"interface":"back","average":{"1min":0.58699999999999997,"5min":0.55100000000000005,"15min":0.54500000000000004},"min":{"1min":0.378,"5min":0.378,"15min":0.378},"max":{"1min":1.004,"5min":1.004,"15min":1.004},"last":0.438},{"interface":"front","average":{"1min":0.62,"5min":0.60099999999999998,"15min":0.59799999999999998},"min":{"1min":0.38800000000000001,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":1.042,"5min":1.042,"15min":1.042},"last":0.70499999999999996}]},{"osd":1,"last update":"Fri May 13 23:13:27 2022","interfaces":[{"interface":"back","average":{"1min":0.80500000000000005,"5min":0.75700000000000001,"15min":0.749},"min":{"1min":0.45700000000000002,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":1.2210000000000001,"5min":1.2210000000000001,"15min":1.2210000000000001},"last":0.61699999999999999},{"interface":"front","average":{"1min":0.72699999999999998,"5min":0.67700000000000005,"15min":0.66900000000000004},"min":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":1.28,"5min":1.28,"15min":1.28},"last":0.63}]},{"osd":2,"last update":"Fri May 13 23:13:27 2022","interfaces":[{"interface":"back","average":{"1min":0.80400000000000005,"5min":0.78800000000000003,"15min":0.78500000000000003},"min":{"1min":0.54700000000000004,"5min":0.54700000000000004,"15min":0.54700000000000004},"max":{"1min":1.1200000000000001,"5min":1.228,"15min":1.228},"last":0.79800000000000004},{"interface":"front","average":{"1min":0.83299999999999996,"5min":0.81599999999999995,"15min":0.81299999999999994},"min":{"1min":0.56599999999999995,"5min":0.53300000000000003,"15min":0.53300000000000003},"max":{"1min":1.298,"5min":1.298,"15min":1.298},"last":0.65400000000000003}]},{"osd":3,"last update":"Fri May 13 23:13:27 2022","interfaces":[{"interface":"back","average":{"1min":0.85699999999999998,"5min":0.81399999999999995,"15min":0.80700000000000005},"min":{"1min":0.60199999999999998,"5min":0.47599999999999998,"15min":0.47599999999999998},"max":{"1min":1.3839999999999999,"5min":1.3839999999999999,"15min":1.3839999999999999},"last":0.68799999999999994},{"interface":"front","average":{"1min":0.76000000000000001,"5min":0.77800000000000002,"15min":0.78100000000000003},"min":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":1.0580000000000001,"5min":1.4119999999999999,"15min":1.4119999999999999},"last":0.68100000000000005}]},{"osd":4,"last update":"Fri May 13 23:13:27 2022","interfaces":[{"interface":"back","average":{"1min":0.82299999999999995,"5min":0.83899999999999997,"15min":0.84199999999999997},"min":{"1min":0.504,"5min":0.504,"15min":0.504},"max":{"1min":1.5640000000000001,"5min":1.5640000000000001,"15min":1.5640000000000001},"last":0.88},{"interface":"front","average":{"1min":0.93200000000000005,"5min":0.90000000000000002,"15min":0.89400000000000002},"min":{"1min":0.60799999999999998,"5min":0.60799999999999998,"15min":0.60799999999999998},"max":{"1min":1.625,"5min":1.625,"15min":1.625},"last":0.78000000000000003}]},{"osd":5,"last update":"Fri May 13 23:13:27 2022","interfaces":[{"interface":"back","average":{"1min":0.84599999999999997,"5min":0.85699999999999998,"15min":0.85899999999999999},"min":{"1min":0.56799999999999995,"5min":0.50600000000000001,"15min":0.50600000000000001},"max":{"1min":1.3640000000000001,"5min":1.5029999999999999,"15min":1.5029999999999999},"last":0.76400000000000001},{"interface":"front","average":{"1min":0.93000000000000005,"5min":0.93700000000000006,"15min":0.93799999999999994},"min":{"1min":0.65900000000000003,"5min":0.65400000000000003,"15min":0.65400000000000003},"max":{"1min":1.3460000000000001,"5min":1.3460000000000001,"15min":1.3460000000000001},"last":0.92800000000000005}]},{"osd":7,"last update":"Fri May 13 23:13:47 2022","interfaces":[{"interface":"back","average":{"1min":1.0669999999999999,"5min":0.89700000000000002,"15min":0.86899999999999999},"min":{"1min":0.78500000000000003,"5min":0.66000000000000003,"15min":0.66000000000000003},"max":{"1min":1.534,"5min":1.534,"15min":1.534},"last":0.83299999999999996},{"interface":"front","average":{"1min":0.97599999999999998,"5min":0.84699999999999998,"15min":0.82499999999999996},"min":{"1min":0.66300000000000003,"5min":0.61299999999999999,"15min":0.61299999999999999},"max":{"1min":1.6180000000000001,"5min":1.6180000000000001,"15min":1.6180000000000001},"last":0.81699999999999995}]},{"osd":8,"last update":"Fri May 13 23:14:12 2022","interfaces":[{"interface":"back","average":{"1min":1.0049999999999999,"5min":0.98499999999999999,"15min":0.98099999999999998},"min":{"1min":0.69999999999999996,"5min":0.69999999999999996,"15min":0.69999999999999996},"max":{"1min":1.484,"5min":1.484,"15min":1.484},"last":0.81299999999999994},{"interface":"front","average":{"1min":0.89600000000000002,"5min":0.94499999999999995,"15min":0.95299999999999996},"min":{"1min":0.68500000000000005,"5min":0.67500000000000004,"15min":0.67500000000000004},"max":{"1min":1.548,"5min":1.548,"15min":1.548},"last":0.86499999999999999}]},{"osd":9,"last update":"Fri May 13 23:13:37 2022","interfaces":[{"interface":"back","average":{"1min":1.071,"5min":1.071,"15min":1.071},"min":{"1min":0.86299999999999999,"5min":0.86299999999999999,"15min":0.86299999999999999},"max":{"1min":1.458,"5min":1.458,"15min":1.458},"last":0.89900000000000002},{"interface":"front","average":{"1min":1.0580000000000001,"5min":1.0580000000000001,"15min":1.0580000000000001},"min":{"1min":0.78900000000000003,"5min":0.78900000000000003,"15min":0.78900000000000003},"max":{"1min":1.401,"5min":1.401,"15min":1.401},"last":0.96499999999999997}]},{"osd":10,"last update":"Fri May 13 23:14:01 2022","interfaces":[{"interface":"back","average":{"1min":1.0529999999999999,"5min":1.0529999999999999,"15min":1.0529999999999999},"min":{"1min":0.77800000000000002,"5min":0.77800000000000002,"15min":0.77800000000000002},"max":{"1min":1.645,"5min":1.645,"15min":1.645},"last":0.91900000000000004},{"interface":"front","average":{"1min":1.006,"5min":1.006,"15min":1.006},"min":{"1min":0.73999999999999999,"5min":0.73999999999999999,"15min":0.73999999999999999},"max":{"1min":1.476,"5min":1.476,"15min":1.476},"last":1.006}]}]},{"osd":1,"up_from":13,"seq":55834574913,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":7032,"kb_used_data":880,"kb_used_omap":0,"kb_used_meta":6144,"kb_avail":93738120,"statfs":{"total":95995035648,"available":95987834880,"internally_reserved":0,"allocated":901120,"data_stored":426350,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":6291456},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri May 13 23:14:14 2022","interfaces":[{"interface":"back","average":{"1min":0.60399999999999998,"5min":0.55900000000000005,"15min":0.51300000000000001},"min":{"1min":0.46700000000000003,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.755,"5min":0.91700000000000004,"15min":0.91700000000000004},"last":0.64300000000000002},{"interface":"front","average":{"1min":0.623,"5min":0.57899999999999996,"15min":0.54000000000000004},"min":{"1min":0.48399999999999999,"5min":0.309,"15min":0.309},"max":{"1min":0.82299999999999995,"5min":0.86799999999999999,"15min":0.86799999999999999},"last":0.72799999999999998}]},{"osd":2,"last update":"Fri May 13 23:13:40 2022","interfaces":[{"interface":"back","average":{"1min":0.56000000000000005,"5min":0.52300000000000002,"15min":0.51300000000000001},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.76900000000000002,"5min":0.76900000000000002,"15min":0.76900000000000002},"last":0.67400000000000004},{"interface":"front","average":{"1min":0.59899999999999998,"5min":0.54300000000000004,"15min":0.52500000000000002},"min":{"1min":0.42299999999999999,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.84599999999999997,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.61899999999999999}]},{"osd":3,"last update":"Fri May 13 23:14:12 2022","interfaces":[{"interface":"back","average":{"1min":0.64100000000000001,"5min":0.59699999999999998,"15min":0.57099999999999995},"min":{"1min":0.44600000000000001,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.84199999999999997,"5min":0.871,"15min":0.871},"last":0.95999999999999996},{"interface":"front","average":{"1min":0.60899999999999999,"5min":0.57199999999999995,"15min":0.56200000000000006},"min":{"1min":0.45400000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.77900000000000003,"5min":0.95399999999999996,"15min":0.95399999999999996},"last":0.873}]},{"osd":4,"last update":"Fri May 13 23:13:35 2022","interfaces":[{"interface":"back","average":{"1min":0.51000000000000001,"5min":0.435,"15min":0.41699999999999998},"min":{"1min":0.32100000000000001,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.77100000000000002,"5min":0.77100000000000002,"15min":0.77100000000000002},"last":0.79600000000000004},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.46899999999999997,"15min":0.44500000000000001},"min":{"1min":0.27400000000000002,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.876,"5min":0.876,"15min":0.876},"last":0.78200000000000003}]},{"osd":5,"last update":"Fri May 13 23:13:52 2022","interfaces":[{"interface":"back","average":{"1min":0.72499999999999998,"5min":0.60099999999999998,"15min":0.58499999999999996},"min":{"1min":0.497,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.88200000000000001,"5min":0.88200000000000001,"15min":0.88200000000000001},"last":0.89500000000000002},{"interface":"front","average":{"1min":0.748,"5min":0.628,"15min":0.60499999999999998},"min":{"1min":0.45500000000000002,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":0.98799999999999999,"5min":0.98799999999999999,"15min":0.98799999999999999},"last":0.81999999999999995}]},{"osd":6,"last update":"Fri May 13 23:13:23 2022","interfaces":[{"interface":"back","average":{"1min":0.76800000000000002,"5min":0.68799999999999994,"15min":0.67400000000000004},"min":{"1min":0.45000000000000001,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":1.5529999999999999,"5min":1.5529999999999999,"15min":1.5529999999999999},"last":1.0089999999999999},{"interface":"front","average":{"1min":0.75,"5min":0.71599999999999997,"15min":0.70999999999999996},"min":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":1.1419999999999999,"5min":1.1419999999999999,"15min":1.1419999999999999},"last":0.85799999999999998}]},{"osd":7,"last update":"Fri May 13 23:13:46 2022","interfaces":[{"interface":"back","average":{"1min":0.70799999999999996,"5min":0.58399999999999996,"15min":0.56299999999999994},"min":{"1min":0.38600000000000001,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.95999999999999996,"5min":0.95999999999999996,"15min":0.95999999999999996},"last":0.91400000000000003},{"interface":"front","average":{"1min":0.79500000000000004,"5min":0.65300000000000002,"15min":0.629},"min":{"1min":0.41399999999999998,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":1.042,"5min":1.042,"15min":1.042},"last":1.079}]},{"osd":8,"last update":"Fri May 13 23:14:12 2022","interfaces":[{"interface":"back","average":{"1min":0.74099999999999999,"5min":0.66900000000000004,"15min":0.65700000000000003},"min":{"1min":0.5,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":1.02,"5min":1.02,"15min":1.02},"last":1.0960000000000001},{"interface":"front","average":{"1min":0.84899999999999998,"5min":0.73299999999999998,"15min":0.71399999999999997},"min":{"1min":0.58999999999999997,"5min":0.501,"15min":0.501},"max":{"1min":1.111,"5min":1.111,"15min":1.111},"last":1.214}]},{"osd":9,"last update":"Fri May 13 23:13:39 2022","interfaces":[{"interface":"back","average":{"1min":0.88500000000000001,"5min":0.88500000000000001,"15min":0.88500000000000001},"min":{"1min":0.60999999999999999,"5min":0.60999999999999999,"15min":0.60999999999999999},"max":{"1min":1.091,"5min":1.091,"15min":1.091},"last":1.165},{"interface":"front","average":{"1min":0.84599999999999997,"5min":0.84599999999999997,"15min":0.84599999999999997},"min":{"1min":0.63600000000000001,"5min":0.63600000000000001,"15min":0.63600000000000001},"max":{"1min":1.0780000000000001,"5min":1.0780000000000001,"15min":1.0780000000000001},"last":0.92100000000000004}]},{"osd":10,"last update":"Fri May 13 23:14:01 2022","interfaces":[{"interface":"back","average":{"1min":0.78800000000000003,"5min":0.78800000000000003,"15min":0.78800000000000003},"min":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"max":{"1min":0.95399999999999996,"5min":0.95399999999999996,"15min":0.95399999999999996},"last":0.97399999999999998},{"interface":"front","average":{"1min":0.90800000000000003,"5min":0.90800000000000003,"15min":0.90800000000000003},"min":{"1min":0.623,"5min":0.623,"15min":0.623},"max":{"1min":1.105,"5min":1.105,"15min":1.105},"last":1.123}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.99299999999999999}]}]},{"osd":0,"up_from":8,"seq":34359738436,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6932,"kb_used_data":844,"kb_used_omap":0,"kb_used_meta":6080,"kb_avail":93738220,"statfs":{"total":95995035648,"available":95987937280,"internally_reserved":0,"allocated":864256,"data_stored":407191,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":6225920},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri May 13 23:14:11 2022","interfaces":[{"interface":"back","average":{"1min":0.68300000000000005,"5min":0.627,"15min":0.60999999999999999},"min":{"1min":0.40400000000000003,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.98899999999999999,"5min":1.2729999999999999,"15min":1.2729999999999999},"last":0.88600000000000001},{"interface":"front","average":{"1min":0.752,"5min":0.66200000000000003,"15min":0.627},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":0.98999999999999999,"5min":1.167,"15min":1.167},"last":0.76600000000000001}]},{"osd":2,"last update":"Fri May 13 23:13:35 2022","interfaces":[{"interface":"back","average":{"1min":0.72499999999999998,"5min":0.65400000000000003,"15min":0.64600000000000002},"min":{"1min":0.54300000000000004,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":0.96499999999999997,"5min":0.98799999999999999,"15min":0.98799999999999999},"last":0.624},{"interface":"front","average":{"1min":0.70099999999999996,"5min":0.63600000000000001,"15min":0.628},"min":{"1min":0.47799999999999998,"5min":0.45900000000000002,"15min":0.45900000000000002},"max":{"1min":1.024,"5min":1.123,"15min":1.123},"last":0.63600000000000001}]},{"osd":3,"last update":"Fri May 13 23:13:08 2022","interfaces":[{"interface":"back","average":{"1min":0.82399999999999995,"5min":0.63300000000000001,"15min":0.59999999999999998},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":1.163,"5min":1.163,"15min":1.163},"last":0.70599999999999996},{"interface":"front","average":{"1min":0.79800000000000004,"5min":0.67000000000000004,"15min":0.64600000000000002},"min":{"1min":0.46200000000000002,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":1.157,"5min":1.157,"15min":1.157},"last":0.69399999999999995}]},{"osd":4,"last update":"Fri May 13 23:13:33 2022","interfaces":[{"interface":"back","average":{"1min":0.82999999999999996,"5min":0.68200000000000005,"15min":0.65500000000000003},"min":{"1min":0.495,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":1.1990000000000001,"5min":1.1990000000000001,"15min":1.1990000000000001},"last":0.73099999999999998},{"interface":"front","average":{"1min":0.77900000000000003,"5min":0.66300000000000003,"15min":0.63300000000000001},"min":{"1min":0.50600000000000001,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":0.98999999999999999,"5min":0.98999999999999999,"15min":0.98999999999999999},"last":0.68400000000000005}]},{"osd":5,"last update":"Fri May 13 23:13:57 2022","interfaces":[{"interface":"back","average":{"1min":0.78900000000000003,"5min":0.72499999999999998,"15min":0.71099999999999997},"min":{"1min":0.46999999999999997,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":1.1220000000000001,"5min":1.1220000000000001,"15min":1.1220000000000001},"last":0.82899999999999996},{"interface":"front","average":{"1min":0.71199999999999997,"5min":0.69699999999999995,"15min":0.69599999999999995},"min":{"1min":0.49399999999999999,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":1.0409999999999999,"5min":1.056,"15min":1.056},"last":0.67500000000000004}]},{"osd":6,"last update":"Fri May 13 23:13:25 2022","interfaces":[{"interface":"back","average":{"1min":0.93600000000000005,"5min":0.77600000000000002,"15min":0.75},"min":{"1min":0.57799999999999996,"5min":0.44800000000000001,"15min":0.44800000000000001},"max":{"1min":1.288,"5min":1.288,"15min":1.288},"last":0.5},{"interface":"front","average":{"1min":0.874,"5min":0.78400000000000003,"15min":0.76900000000000002},"min":{"1min":0.51000000000000001,"5min":0.5,"15min":0.5},"max":{"1min":1.202,"5min":1.202,"15min":1.202},"last":0.90100000000000002}]},{"osd":7,"last update":"Fri May 13 23:13:48 2022","interfaces":[{"interface":"back","average":{"1min":0.77300000000000002,"5min":0.75,"15min":0.746},"min":{"1min":0.497,"5min":0.48699999999999999,"15min":0.48699999999999999},"max":{"1min":1.2050000000000001,"5min":1.2050000000000001,"15min":1.2050000000000001},"last":0.94199999999999995},{"interface":"front","average":{"1min":0.94899999999999995,"5min":0.86099999999999999,"15min":0.84699999999999998},"min":{"1min":0.69599999999999995,"5min":0.60399999999999998,"15min":0.60399999999999998},"max":{"1min":1.2350000000000001,"5min":1.2350000000000001,"15min":1.2350000000000001},"last":0.877}]},{"osd":8,"last update":"Fri May 13 23:14:09 2022","interfaces":[{"interface":"back","average":{"1min":0.82699999999999996,"5min":0.80500000000000005,"15min":0.80100000000000005},"min":{"1min":0.54700000000000004,"5min":0.52100000000000002,"15min":0.52100000000000002},"max":{"1min":1.1479999999999999,"5min":1.7070000000000001,"15min":1.7070000000000001},"last":0.71699999999999997},{"interface":"front","average":{"1min":0.91200000000000003,"5min":0.91600000000000004,"15min":0.91600000000000004},"min":{"1min":0.60599999999999998,"5min":0.60599999999999998,"15min":0.60599999999999998},"max":{"1min":1.1160000000000001,"5min":1.266,"15min":1.266},"last":0.69099999999999995}]},{"osd":9,"last update":"Fri May 13 23:13:39 2022","interfaces":[{"interface":"back","average":{"1min":1.0589999999999999,"5min":1.0589999999999999,"15min":1.0589999999999999},"min":{"1min":0.85199999999999998,"5min":0.85199999999999998,"15min":0.85199999999999998},"max":{"1min":1.397,"5min":1.397,"15min":1.397},"last":0.91100000000000003},{"interface":"front","average":{"1min":0.94099999999999995,"5min":0.94099999999999995,"15min":0.94099999999999995},"min":{"1min":0.70999999999999996,"5min":0.70999999999999996,"15min":0.70999999999999996},"max":{"1min":1.232,"5min":1.232,"15min":1.232},"last":1.0069999999999999}]},{"osd":10,"last update":"Fri May 13 23:14:02 2022","interfaces":[{"interface":"back","average":{"1min":0.97199999999999998,"5min":0.97199999999999998,"15min":0.97199999999999998},"min":{"1min":0.67700000000000005,"5min":0.67700000000000005,"15min":0.67700000000000005},"max":{"1min":1.7989999999999999,"5min":1.7989999999999999,"15min":1.7989999999999999},"last":0.997},{"interface":"front","average":{"1min":1.0629999999999999,"5min":1.0629999999999999,"15min":1.0629999999999999},"min":{"1min":0.745,"5min":0.745,"15min":0.745},"max":{"1min":1.3360000000000001,"5min":1.3360000000000001,"15min":1.3360000000000001},"last":1.018}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.96299999999999997}]}]},{"osd":2,"up_from":18,"seq":77309411387,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":7108,"kb_used_data":892,"kb_used_omap":0,"kb_used_meta":6208,"kb_avail":93738044,"statfs":{"total":95995035648,"available":95987757056,"internally_reserved":0,"allocated":913408,"data_stored":426958,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":6356992},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri May 13 23:13:37 2022","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.51700000000000002,"15min":0.52300000000000002},"min":{"1min":0.41699999999999998,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.80700000000000005,"5min":0.80700000000000005,"15min":0.80700000000000005},"last":0.439},{"interface":"front","average":{"1min":0.59499999999999997,"5min":0.52100000000000002,"15min":0.51600000000000001},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.78800000000000003,"5min":0.80700000000000005,"15min":0.80700000000000005},"last":0.64800000000000002}]},{"osd":1,"last update":"Fri May 13 23:13:37 2022","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.52500000000000002,"15min":0.52000000000000002},"min":{"1min":0.35699999999999998,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.83699999999999997,"5min":0.83699999999999997,"15min":0.83699999999999997},"last":0.69099999999999995},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.53200000000000003,"15min":0.55000000000000004},"min":{"1min":0.40100000000000002,"5min":0.313,"15min":0.313},"max":{"1min":0.77900000000000003,"5min":0.77900000000000003,"15min":0.77900000000000003},"last":0.45400000000000001}]},{"osd":3,"last update":"Fri May 13 23:14:12 2022","interfaces":[{"interface":"back","average":{"1min":0.64500000000000002,"5min":0.61599999999999999,"15min":0.61199999999999999},"min":{"1min":0.39900000000000002,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.90500000000000003,"5min":0.90500000000000003,"15min":0.90500000000000003},"last":0.51500000000000001},{"interface":"front","average":{"1min":0.66800000000000004,"5min":0.60499999999999998,"15min":0.59699999999999998},"min":{"1min":0.45000000000000001,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.89500000000000002,"5min":0.89500000000000002,"15min":0.89500000000000002},"last":0.497}]},{"osd":4,"last update":"Fri May 13 23:13:31 2022","interfaces":[{"interface":"back","average":{"1min":0.55100000000000005,"5min":0.53400000000000003,"15min":0.52600000000000002},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":0.81999999999999995,"5min":0.81999999999999995,"15min":0.81999999999999995},"last":0.59299999999999997},{"interface":"front","average":{"1min":0.53200000000000003,"5min":0.51500000000000001,"15min":0.505},"min":{"1min":0.36299999999999999,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.70099999999999996,"5min":0.752,"15min":0.752},"last":0.61299999999999999}]},{"osd":5,"last update":"Fri May 13 23:14:00 2022","interfaces":[{"interface":"back","average":{"1min":0.435,"5min":0.40300000000000002,"15min":0.38300000000000001},"min":{"1min":0.32300000000000001,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.71099999999999997,"5min":0.71999999999999997,"15min":0.71999999999999997},"last":0.39600000000000002},{"interface":"front","average":{"1min":0.47099999999999997,"5min":0.41699999999999998,"15min":0.39700000000000002},"min":{"1min":0.35499999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.748,"5min":0.748,"15min":0.748},"last":0.746}]},{"osd":6,"last update":"Fri May 13 23:13:28 2022","interfaces":[{"interface":"back","average":{"1min":0.60599999999999998,"5min":0.60599999999999998,"15min":0.60599999999999998},"min":{"1min":0.504,"5min":0.44800000000000001,"15min":0.44800000000000001},"max":{"1min":0.78500000000000003,"5min":0.78900000000000003,"15min":0.78900000000000003},"last":0.621},{"interface":"front","average":{"1min":0.63100000000000001,"5min":0.58999999999999997,"15min":0.58299999999999996},"min":{"1min":0.46500000000000002,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":0.78200000000000003,"5min":0.78200000000000003,"15min":0.78200000000000003},"last":0.63400000000000001}]},{"osd":7,"last update":"Fri May 13 23:13:45 2022","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":0.61599999999999999,"15min":0.61599999999999999},"min":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":0.80400000000000005,"5min":0.80400000000000005,"15min":0.80400000000000005},"last":0.78400000000000003},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.63800000000000001,"15min":0.64200000000000002},"min":{"1min":0.45900000000000002,"5min":0.45900000000000002,"15min":0.45900000000000002},"max":{"1min":0.83199999999999996,"5min":0.83199999999999996,"15min":0.83199999999999996},"last":0.78800000000000003}]},{"osd":8,"last update":"Fri May 13 23:14:07 2022","interfaces":[{"interface":"back","average":{"1min":0.55400000000000005,"5min":0.53600000000000003,"15min":0.53300000000000003},"min":{"1min":0.36699999999999999,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.876,"5min":0.876,"15min":0.876},"last":0.55800000000000005},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.82799999999999996,"5min":0.82799999999999996,"15min":0.82799999999999996},"last":0.54000000000000004}]},{"osd":9,"last update":"Fri May 13 23:13:37 2022","interfaces":[{"interface":"back","average":{"1min":0.70999999999999996,"5min":0.70999999999999996,"15min":0.70999999999999996},"min":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"max":{"1min":0.82799999999999996,"5min":0.82799999999999996,"15min":0.82799999999999996},"last":0.65500000000000003},{"interface":"front","average":{"1min":0.68999999999999995,"5min":0.68999999999999995,"15min":0.68999999999999995},"min":{"1min":0.55800000000000005,"5min":0.55800000000000005,"15min":0.55800000000000005},"max":{"1min":0.86799999999999999,"5min":0.86799999999999999,"15min":0.86799999999999999},"last":0.70799999999999996}]},{"osd":10,"last update":"Fri May 13 23:14:00 2022","interfaces":[{"interface":"back","average":{"1min":0.68999999999999995,"5min":0.68999999999999995,"15min":0.68999999999999995},"min":{"1min":0.47599999999999998,"5min":0.47599999999999998,"15min":0.47599999999999998},"max":{"1min":0.97099999999999997,"5min":0.97099999999999997,"15min":0.97099999999999997},"last":0.67200000000000004},{"interface":"front","average":{"1min":0.71099999999999997,"5min":0.71099999999999997,"15min":0.71099999999999997},"min":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.52100000000000002},"max":{"1min":0.89200000000000002,"5min":0.89200000000000002,"15min":0.89200000000000002},"last":0.72399999999999998}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73799999999999999}]}]},{"osd":3,"up_from":25,"seq":107374182453,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6452,"kb_used_data":876,"kb_used_omap":0,"kb_used_meta":5568,"kb_avail":93738700,"statfs":{"total":95995035648,"available":95988428800,"internally_reserved":0,"allocated":897024,"data_stored":424918,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5701632},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri May 13 23:14:15 2022","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.49399999999999999,"15min":0.46400000000000002},"min":{"1min":0.40200000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.81999999999999995,"5min":0.81999999999999995,"15min":0.81999999999999995},"last":0.45400000000000001},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.52400000000000002,"15min":0.502},"min":{"1min":0.373,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.70799999999999996,"5min":0.96299999999999997,"15min":0.96299999999999997},"last":0.47699999999999998}]},{"osd":1,"last update":"Fri May 13 23:14:15 2022","interfaces":[{"interface":"back","average":{"1min":0.749,"5min":0.67700000000000005,"15min":0.63500000000000001},"min":{"1min":0.495,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":1.014,"5min":1.014,"15min":1.014},"last":0.69399999999999995},{"interface":"front","average":{"1min":0.64100000000000001,"5min":0.64300000000000002,"15min":0.63100000000000001},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.93200000000000005,"5min":1.0229999999999999,"15min":1.0229999999999999},"last":0.55900000000000005}]},{"osd":2,"last update":"Fri May 13 23:14:15 2022","interfaces":[{"interface":"back","average":{"1min":0.79300000000000004,"5min":0.72399999999999998,"15min":0.68799999999999994},"min":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":1.0489999999999999,"5min":1.0489999999999999,"15min":1.0489999999999999},"last":0.73499999999999999},{"interface":"front","average":{"1min":0.73899999999999999,"5min":0.69699999999999995,"15min":0.67500000000000004},"min":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":1.1299999999999999,"5min":1.1299999999999999,"15min":1.1299999999999999},"last":0.64500000000000002}]},{"osd":4,"last update":"Fri May 13 23:13:33 2022","interfaces":[{"interface":"back","average":{"1min":0.90300000000000002,"5min":0.71999999999999997,"15min":0.68400000000000005},"min":{"1min":0.44400000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":1.157,"5min":1.157,"15min":1.157},"last":0.79400000000000004},{"interface":"front","average":{"1min":0.89900000000000002,"5min":0.72499999999999998,"15min":0.68799999999999994},"min":{"1min":0.56299999999999994,"5min":0.47699999999999998,"15min":0.47699999999999998},"max":{"1min":1.0660000000000001,"5min":1.0660000000000001,"15min":1.0660000000000001},"last":0.82499999999999996}]},{"osd":5,"last update":"Fri May 13 23:13:50 2022","interfaces":[{"interface":"back","average":{"1min":0.80900000000000005,"5min":0.745,"15min":0.72899999999999998},"min":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"max":{"1min":1.056,"5min":1.056,"15min":1.056},"last":0.60199999999999998},{"interface":"front","average":{"1min":0.94499999999999995,"5min":0.80000000000000004,"15min":0.76600000000000001},"min":{"1min":0.626,"5min":0.53400000000000003,"15min":0.53400000000000003},"max":{"1min":1.161,"5min":1.161,"15min":1.161},"last":0.84799999999999998}]},{"osd":6,"last update":"Fri May 13 23:13:27 2022","interfaces":[{"interface":"back","average":{"1min":0.93999999999999995,"5min":0.83299999999999996,"15min":0.81499999999999995},"min":{"1min":0.57799999999999996,"5min":0.57599999999999996,"15min":0.57599999999999996},"max":{"1min":1.2829999999999999,"5min":1.2829999999999999,"15min":1.2829999999999999},"last":0.88800000000000001},{"interface":"front","average":{"1min":0.93700000000000006,"5min":0.86399999999999999,"15min":0.85199999999999998},"min":{"1min":0.61099999999999999,"5min":0.61099999999999999,"15min":0.61099999999999999},"max":{"1min":1.256,"5min":1.256,"15min":1.256},"last":0.71999999999999997}]},{"osd":7,"last update":"Fri May 13 23:13:47 2022","interfaces":[{"interface":"back","average":{"1min":1.034,"5min":0.90700000000000003,"15min":0.88600000000000001},"min":{"1min":0.70899999999999996,"5min":0.628,"15min":0.628},"max":{"1min":1.3280000000000001,"5min":1.3280000000000001,"15min":1.3280000000000001},"last":0.93500000000000005},{"interface":"front","average":{"1min":1.028,"5min":0.89100000000000001,"15min":0.86799999999999999},"min":{"1min":0.82499999999999996,"5min":0.67800000000000005,"15min":0.67800000000000005},"max":{"1min":1.3480000000000001,"5min":1.3480000000000001,"15min":1.3480000000000001},"last":1.042}]},{"osd":8,"last update":"Fri May 13 23:14:07 2022","interfaces":[{"interface":"back","average":{"1min":0.91700000000000004,"5min":0.94699999999999995,"15min":0.95199999999999996},"min":{"1min":0.56699999999999995,"5min":0.56699999999999995,"15min":0.56699999999999995},"max":{"1min":1.2589999999999999,"5min":1.2669999999999999,"15min":1.2669999999999999},"last":1.048},{"interface":"front","average":{"1min":0.98899999999999999,"5min":1.002,"15min":1.004},"min":{"1min":0.58699999999999997,"5min":0.58699999999999997,"15min":0.58699999999999997},"max":{"1min":1.244,"5min":1.244,"15min":1.244},"last":0.91500000000000004}]},{"osd":9,"last update":"Fri May 13 23:13:39 2022","interfaces":[{"interface":"back","average":{"1min":1.131,"5min":1.131,"15min":1.131},"min":{"1min":0.79500000000000004,"5min":0.79500000000000004,"15min":0.79500000000000004},"max":{"1min":1.4319999999999999,"5min":1.4319999999999999,"15min":1.4319999999999999},"last":0.90500000000000003},{"interface":"front","average":{"1min":1.093,"5min":1.093,"15min":1.093},"min":{"1min":0.79100000000000004,"5min":0.79100000000000004,"15min":0.79100000000000004},"max":{"1min":1.4730000000000001,"5min":1.4730000000000001,"15min":1.4730000000000001},"last":0.94999999999999996}]},{"osd":10,"last update":"Fri May 13 23:14:00 2022","interfaces":[{"interface":"back","average":{"1min":1.0129999999999999,"5min":1.0129999999999999,"15min":1.0129999999999999},"min":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.52300000000000002},"max":{"1min":1.373,"5min":1.373,"15min":1.373},"last":0.68200000000000005},{"interface":"front","average":{"1min":1.127,"5min":1.127,"15min":1.127},"min":{"1min":0.78300000000000003,"5min":0.78300000000000003,"15min":0.78300000000000003},"max":{"1min":1.51,"5min":1.51,"15min":1.51},"last":0.996}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.80500000000000005}]}]},{"osd":4,"up_from":33,"seq":141733920816,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6256,"kb_used_data":872,"kb_used_omap":0,"kb_used_meta":5376,"kb_avail":93738896,"statfs":{"total":95995035648,"available":95988629504,"internally_reserved":0,"allocated":892928,"data_stored":424836,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5505024},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri May 13 23:13:35 2022","interfaces":[{"interface":"back","average":{"1min":0.627,"5min":0.56299999999999994,"15min":0.55000000000000004},"min":{"1min":0.39700000000000002,"5min":0.375,"15min":0.375},"max":{"1min":0.91800000000000004,"5min":0.91800000000000004,"15min":0.91800000000000004},"last":0.56000000000000005},{"interface":"front","average":{"1min":0.70499999999999996,"5min":0.58099999999999996,"15min":0.54500000000000004},"min":{"1min":0.47999999999999998,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.97099999999999997,"5min":0.97099999999999997,"15min":0.97099999999999997},"last":0.65200000000000002}]},{"osd":1,"last update":"Fri May 13 23:13:35 2022","interfaces":[{"interface":"back","average":{"1min":0.498,"5min":0.39200000000000002,"15min":0.372},"min":{"1min":0.309,"5min":0.217,"15min":0.217},"max":{"1min":0.84499999999999997,"5min":0.84499999999999997,"15min":0.84499999999999997},"last":0.44600000000000001},{"interface":"front","average":{"1min":0.46200000000000002,"5min":0.40300000000000002,"15min":0.39000000000000001},"min":{"1min":0.28899999999999998,"5min":0.248,"15min":0.248},"max":{"1min":0.78100000000000003,"5min":0.79900000000000004,"15min":0.79900000000000004},"last":0.496}]},{"osd":2,"last update":"Fri May 13 23:13:35 2022","interfaces":[{"interface":"back","average":{"1min":0.63800000000000001,"5min":0.54800000000000004,"15min":0.54000000000000004},"min":{"1min":0.47899999999999998,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.92600000000000005,"5min":0.92600000000000005,"15min":0.92600000000000005},"last":0.52300000000000002},{"interface":"front","average":{"1min":0.66200000000000003,"5min":0.57799999999999996,"15min":0.56499999999999995},"min":{"1min":0.5,"5min":0.372,"15min":0.372},"max":{"1min":0.92900000000000005,"5min":0.92900000000000005,"15min":0.92900000000000005},"last":0.59499999999999997}]},{"osd":3,"last update":"Fri May 13 23:13:35 2022","interfaces":[{"interface":"back","average":{"1min":0.70399999999999996,"5min":0.60799999999999998,"15min":0.58699999999999997},"min":{"1min":0.498,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.91600000000000004,"5min":0.91600000000000004,"15min":0.91600000000000004},"last":0.749},{"interface":"front","average":{"1min":0.78800000000000003,"5min":0.63700000000000001,"15min":0.61299999999999999},"min":{"1min":0.54400000000000004,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":1.0569999999999999,"5min":1.0569999999999999,"15min":1.0569999999999999},"last":0.63800000000000001}]},{"osd":5,"last update":"Fri May 13 23:13:55 2022","interfaces":[{"interface":"back","average":{"1min":0.71399999999999997,"5min":0.58699999999999997,"15min":0.55300000000000005},"min":{"1min":0.53200000000000003,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.97999999999999998,"5min":0.97999999999999998,"15min":0.97999999999999998},"last":0.67000000000000004},{"interface":"front","average":{"1min":0.70099999999999996,"5min":0.58599999999999997,"15min":0.55600000000000005},"min":{"1min":0.55600000000000005,"5min":0.371,"15min":0.371},"max":{"1min":1.1419999999999999,"5min":1.1419999999999999,"15min":1.1419999999999999},"last":0.69499999999999995}]},{"osd":6,"last update":"Fri May 13 23:13:25 2022","interfaces":[{"interface":"back","average":{"1min":0.79000000000000004,"5min":0.67800000000000005,"15min":0.66000000000000003},"min":{"1min":0.57499999999999996,"5min":0.498,"15min":0.498},"max":{"1min":1.127,"5min":1.127,"15min":1.127},"last":0.68300000000000005},{"interface":"front","average":{"1min":0.83699999999999997,"5min":0.71799999999999997,"15min":0.69799999999999995},"min":{"1min":0.60699999999999998,"5min":0.55900000000000005,"15min":0.55900000000000005},"max":{"1min":1.046,"5min":1.046,"15min":1.046},"last":0.72399999999999998}]},{"osd":7,"last update":"Fri May 13 23:13:50 2022","interfaces":[{"interface":"back","average":{"1min":0.72999999999999998,"5min":0.66500000000000004,"15min":0.65400000000000003},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.98599999999999999,"5min":0.98599999999999999,"15min":0.98599999999999999},"last":0.76600000000000001},{"interface":"front","average":{"1min":0.71699999999999997,"5min":0.64500000000000002,"15min":0.63300000000000001},"min":{"1min":0.41599999999999998,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":1.0089999999999999,"5min":1.0089999999999999,"15min":1.0089999999999999},"last":0.65200000000000002}]},{"osd":8,"last update":"Fri May 13 23:14:13 2022","interfaces":[{"interface":"back","average":{"1min":0.81499999999999995,"5min":0.75900000000000001,"15min":0.75},"min":{"1min":0.57399999999999995,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":1.099,"5min":1.099,"15min":1.099},"last":0.80100000000000005},{"interface":"front","average":{"1min":0.76000000000000001,"5min":0.754,"15min":0.753},"min":{"1min":0.44700000000000001,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":1.115,"5min":1.115,"15min":1.115},"last":0.81599999999999995}]},{"osd":9,"last update":"Fri May 13 23:13:40 2022","interfaces":[{"interface":"back","average":{"1min":0.89000000000000001,"5min":0.89000000000000001,"15min":0.89000000000000001},"min":{"1min":0.65700000000000003,"5min":0.65700000000000003,"15min":0.65700000000000003},"max":{"1min":1.1819999999999999,"5min":1.1819999999999999,"15min":1.1819999999999999},"last":0.79600000000000004},{"interface":"front","average":{"1min":0.86899999999999999,"5min":0.86899999999999999,"15min":0.86899999999999999},"min":{"1min":0.67300000000000004,"5min":0.67300000000000004,"15min":0.67300000000000004},"max":{"1min":1.171,"5min":1.171,"15min":1.171},"last":0.78700000000000003}]},{"osd":10,"last update":"Fri May 13 23:14:00 2022","interfaces":[{"interface":"back","average":{"1min":0.80700000000000005,"5min":0.80700000000000005,"15min":0.80700000000000005},"min":{"1min":0.52600000000000002,"5min":0.52600000000000002,"15min":0.52600000000000002},"max":{"1min":1.103,"5min":1.103,"15min":1.103},"last":0.77500000000000002},{"interface":"front","average":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"min":{"1min":0.48499999999999999,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":1.196,"5min":1.196,"15min":1.196},"last":0.73999999999999999}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.83399999999999996}]}]},{"osd":5,"up_from":38,"seq":163208757291,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6844,"kb_used_data":1396,"kb_used_omap":0,"kb_used_meta":5440,"kb_avail":93738308,"statfs":{"total":95995035648,"available":95988027392,"internally_reserved":0,"allocated":1429504,"data_stored":953601,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5570560},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri May 13 23:13:51 2022","interfaces":[{"interface":"back","average":{"1min":0.61099999999999999,"5min":0.53700000000000003,"15min":0.52300000000000002},"min":{"1min":0.376,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.96299999999999997,"5min":0.96299999999999997,"15min":0.96299999999999997},"last":0.53400000000000003},{"interface":"front","average":{"1min":0.57199999999999995,"5min":0.54200000000000004,"15min":0.52300000000000002},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.91300000000000003,"5min":0.96499999999999997,"15min":0.96499999999999997},"last":0.50600000000000001}]},{"osd":1,"last update":"Fri May 13 23:13:51 2022","interfaces":[{"interface":"back","average":{"1min":0.57699999999999996,"5min":0.504,"15min":0.48899999999999999},"min":{"1min":0.41499999999999998,"5min":0.317,"15min":0.317},"max":{"1min":0.76900000000000002,"5min":0.76900000000000002,"15min":0.76900000000000002},"last":0.58599999999999997},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.502,"15min":0.47999999999999998},"min":{"1min":0.42599999999999999,"5min":0.313,"15min":0.313},"max":{"1min":0.78800000000000003,"5min":0.78800000000000003,"15min":0.78800000000000003},"last":0.55200000000000005}]},{"osd":2,"last update":"Fri May 13 23:13:51 2022","interfaces":[{"interface":"back","average":{"1min":0.42399999999999999,"5min":0.38100000000000001,"15min":0.36299999999999999},"min":{"1min":0.28399999999999997,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":0.65600000000000003,"5min":0.65600000000000003,"15min":0.65600000000000003},"last":0.442},{"interface":"front","average":{"1min":0.45000000000000001,"5min":0.39500000000000002,"15min":0.375},"min":{"1min":0.28899999999999998,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.75700000000000001,"5min":0.75700000000000001,"15min":0.75700000000000001},"last":0.42499999999999999}]},{"osd":3,"last update":"Fri May 13 23:13:51 2022","interfaces":[{"interface":"back","average":{"1min":0.57499999999999996,"5min":0.56499999999999995,"15min":0.55900000000000005},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.97999999999999998,"5min":0.97999999999999998,"15min":0.97999999999999998},"last":0.48899999999999999},{"interface":"front","average":{"1min":0.625,"5min":0.57699999999999996,"15min":0.55300000000000005},"min":{"1min":0.45600000000000002,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":0.89900000000000002,"5min":1,"15min":1},"last":0.61199999999999999}]},{"osd":4,"last update":"Fri May 13 23:13:51 2022","interfaces":[{"interface":"back","average":{"1min":0.63700000000000001,"5min":0.55800000000000005,"15min":0.53700000000000003},"min":{"1min":0.38800000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.86399999999999999,"5min":0.86399999999999999,"15min":0.86399999999999999},"last":0.57599999999999996},{"interface":"front","average":{"1min":0.58299999999999996,"5min":0.54200000000000004,"15min":0.53200000000000003},"min":{"1min":0.42599999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.73399999999999999,"5min":0.89500000000000002,"15min":0.89500000000000002},"last":0.54400000000000004}]},{"osd":6,"last update":"Fri May 13 23:13:25 2022","interfaces":[{"interface":"back","average":{"1min":0.60899999999999999,"5min":0.623,"15min":0.625},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":0.872,"5min":0.95399999999999996,"15min":0.95399999999999996},"last":0.52400000000000002},{"interface":"front","average":{"1min":0.65100000000000002,"5min":0.626,"15min":0.622},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.98399999999999999,"5min":0.98399999999999999,"15min":0.98399999999999999},"last":0.63700000000000001}]},{"osd":7,"last update":"Fri May 13 23:13:46 2022","interfaces":[{"interface":"back","average":{"1min":0.629,"5min":0.64100000000000001,"15min":0.64300000000000002},"min":{"1min":0.45200000000000001,"5min":0.379,"15min":0.379},"max":{"1min":0.871,"5min":1.0489999999999999,"15min":1.0489999999999999},"last":0.621},{"interface":"front","average":{"1min":0.66700000000000004,"5min":0.64500000000000002,"15min":0.64100000000000001},"min":{"1min":0.442,"5min":0.442,"15min":0.442},"max":{"1min":0.93000000000000005,"5min":0.93000000000000005,"15min":0.93000000000000005},"last":0.67500000000000004}]},{"osd":8,"last update":"Fri May 13 23:14:06 2022","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.61599999999999999,"15min":0.622},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.89200000000000002,"5min":0.999,"15min":0.999},"last":0.59799999999999998},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.56599999999999995,"15min":0.56899999999999995},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.82699999999999996,"5min":0.82699999999999996,"15min":0.82699999999999996},"last":0.56899999999999995}]},{"osd":9,"last update":"Fri May 13 23:13:40 2022","interfaces":[{"interface":"back","average":{"1min":0.68600000000000005,"5min":0.68600000000000005,"15min":0.68600000000000005},"min":{"1min":0.51400000000000001,"5min":0.51400000000000001,"15min":0.51400000000000001},"max":{"1min":0.88500000000000001,"5min":0.88500000000000001,"15min":0.88500000000000001},"last":0.64500000000000002},{"interface":"front","average":{"1min":0.71199999999999997,"5min":0.71199999999999997,"15min":0.71199999999999997},"min":{"1min":0.505,"5min":0.505,"15min":0.505},"max":{"1min":0.99299999999999999,"5min":0.99299999999999999,"15min":0.99299999999999999},"last":0.69999999999999996}]},{"osd":10,"last update":"Fri May 13 23:13:59 2022","interfaces":[{"interface":"back","average":{"1min":0.67900000000000005,"5min":0.67900000000000005,"15min":0.67900000000000005},"min":{"1min":0.503,"5min":0.503,"15min":0.503},"max":{"1min":0.90800000000000003,"5min":0.90800000000000003,"15min":0.90800000000000003},"last":0.66800000000000004},{"interface":"front","average":{"1min":0.71399999999999997,"5min":0.71399999999999997,"15min":0.71399999999999997},"min":{"1min":0.48499999999999999,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":0.94499999999999995,"5min":0.94499999999999995,"15min":0.94499999999999995},"last":0.71699999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":2810,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":554,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1730,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2022-05-13T23:14:22.765 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2022-05-13T23:14:23.014 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:23.008+0000 7f8f0ffff700 1 -- 172.21.15.2:0/746263483 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8f100fd0e0 msgr2=0x7f8f101039a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:14:23.014 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:23.008+0000 7f8f0ffff700 1 --2- 172.21.15.2:0/746263483 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8f100fd0e0 0x7f8f101039a0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f8f08009be0 tx=0x7f8f08004d10 comp rx=0 tx=0).stop 2022-05-13T23:14:23.014 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:23.008+0000 7f8f0ffff700 1 -- 172.21.15.2:0/746263483 shutdown_connections 2022-05-13T23:14:23.015 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:23.008+0000 7f8f0ffff700 1 --2- 172.21.15.2:0/746263483 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8f100fd0e0 0x7f8f101039a0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:14:23.015 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:23.008+0000 7f8f0ffff700 1 --2- 172.21.15.2:0/746263483 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f8f100fb4b0 0x7f8f100fb8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:14:23.015 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:23.008+0000 7f8f0ffff700 1 --2- 172.21.15.2:0/746263483 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8f100fc750 0x7f8f100fcba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:14:23.016 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:23.008+0000 7f8f0ffff700 1 -- 172.21.15.2:0/746263483 >> 172.21.15.2:0/746263483 conn(0x7f8f100f6c80 msgr2=0x7f8f100f90d0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-05-13T23:14:23.016 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:23.008+0000 7f8f0ffff700 1 -- 172.21.15.2:0/746263483 shutdown_connections 2022-05-13T23:14:23.017 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:23.008+0000 7f8f0ffff700 1 -- 172.21.15.2:0/746263483 wait complete. 2022-05-13T23:14:23.017 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:23.009+0000 7f8f0ffff700 1 Processor -- start 2022-05-13T23:14:23.017 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:23.009+0000 7f8f0ffff700 1 -- start start 2022-05-13T23:14:23.017 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:23.010+0000 7f8f0ffff700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8f100fb4b0 0x7f8f100701d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:14:23.018 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:23.010+0000 7f8f0ffff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8f100fc750 0x7f8f1006e830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:14:23.018 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:23.010+0000 7f8f0ffff700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f8f100fd0e0 0x7f8f1006ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:14:23.018 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:23.010+0000 7f8f0ffff700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f1006f480 con 0x7f8f100fc750 2022-05-13T23:14:23.018 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:23.010+0000 7f8f0ffff700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f1006f5f0 con 0x7f8f100fb4b0 2022-05-13T23:14:23.019 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:23.010+0000 7f8f0ffff700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f10116280 con 0x7f8f100fd0e0 2022-05-13T23:14:23.019 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:23.010+0000 7f8f0e7fc700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8f100fc750 0x7f8f1006e830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:14:23.019 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:23.010+0000 7f8f0f7fe700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f8f100fd0e0 0x7f8f1006ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:14:23.019 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:23.010+0000 7f8f0f7fe700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f8f100fd0e0 0x7f8f1006ed70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.2:56714/0 (socket says 172.21.15.2:56714) 2022-05-13T23:14:23.020 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:23.010+0000 7f8f0f7fe700 1 -- 172.21.15.2:0/1333998809 learned_addr learned my addr 172.21.15.2:0/1333998809 (peer_addr_for_me v2:172.21.15.2:0/0) 2022-05-13T23:14:23.020 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:23.010+0000 7f8f0effd700 1 --2- 172.21.15.2:0/1333998809 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8f100fb4b0 0x7f8f100701d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:14:23.020 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:23.010+0000 7f8f0e7fc700 1 -- 172.21.15.2:0/1333998809 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f8f100fd0e0 msgr2=0x7f8f1006ed70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:14:23.020 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:23.010+0000 7f8f0e7fc700 1 --2- 172.21.15.2:0/1333998809 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f8f100fd0e0 0x7f8f1006ed70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:14:23.021 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:23.010+0000 7f8f0e7fc700 1 -- 172.21.15.2:0/1333998809 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8f100fb4b0 msgr2=0x7f8f100701d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:14:23.021 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:23.010+0000 7f8f0e7fc700 1 --2- 172.21.15.2:0/1333998809 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8f100fb4b0 0x7f8f100701d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:14:23.021 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:23.011+0000 7f8f0e7fc700 1 -- 172.21.15.2:0/1333998809 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f080097c0 con 0x7f8f100fc750 2022-05-13T23:14:23.021 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:23.011+0000 7f8f0e7fc700 1 --2- 172.21.15.2:0/1333998809 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8f100fc750 0x7f8f1006e830 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f8f0400e8f0 tx=0x7f8f0400b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-05-13T23:14:23.022 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:23.011+0000 7f8ef77fe700 1 -- 172.21.15.2:0/1333998809 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f04007b40 con 0x7f8f100fc750 2022-05-13T23:14:23.022 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:23.011+0000 7f8ef77fe700 1 -- 172.21.15.2:0/1333998809 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1087+0+0 (secure 0 0 0) 0x7f8f0400f040 con 0x7f8f100fc750 2022-05-13T23:14:23.022 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:23.011+0000 7f8f0ffff700 1 -- 172.21.15.2:0/1333998809 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8f10116480 con 0x7f8f100fc750 2022-05-13T23:14:23.022 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:23.011+0000 7f8ef77fe700 1 -- 172.21.15.2:0/1333998809 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f04014950 con 0x7f8f100fc750 2022-05-13T23:14:23.023 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:23.011+0000 7f8f0ffff700 1 -- 172.21.15.2:0/1333998809 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8f101169a0 con 0x7f8f100fc750 2022-05-13T23:14:23.023 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:23.012+0000 7f8f0ffff700 1 -- 172.21.15.2:0/1333998809 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8f10106d50 con 0x7f8f100fc750 2022-05-13T23:14:23.024 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:23.014+0000 7f8ef77fe700 1 -- 172.21.15.2:0/1333998809 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 16) v1 ==== 125136+0+0 (secure 0 0 0) 0x7f8f0401b030 con 0x7f8f100fc750 2022-05-13T23:14:23.025 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:23.016+0000 7f8ef77fe700 1 --2- 172.21.15.2:0/1333998809 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f8ef8096ce0 0x7f8ef8099190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:14:23.025 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:23.016+0000 7f8ef77fe700 1 -- 172.21.15.2:0/1333998809 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 7975+0+0 (secure 0 0 0) 0x7f8f040be790 con 0x7f8f100fc750 2022-05-13T23:14:23.025 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:23.017+0000 7f8f0effd700 1 --2- 172.21.15.2:0/1333998809 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f8ef8096ce0 0x7f8ef8099190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:14:23.026 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:23.017+0000 7f8f0effd700 1 --2- 172.21.15.2:0/1333998809 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f8ef8096ce0 0x7f8ef8099190 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f8f100fc5b0 tx=0x7f8f0000b040 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-05-13T23:14:23.026 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:23.020+0000 7f8ef77fe700 1 -- 172.21.15.2:0/1333998809 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+165215 (secure 0 0 0) 0x7f8f0404b570 con 0x7f8f100fc750 2022-05-13T23:14:23.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:14:22 smithi047 ceph-mon[77147]: from='client.? 172.21.15.2:0/3025885018' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2022-05-13T23:14:23.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:14:22 smithi047 ceph-mon[77147]: from='client.? 172.21.15.2:0/2765872222' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2022-05-13T23:14:23.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:14:22 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6832/381933929,v1:172.21.15.115:6833/381933929] up:active 2022-05-13T23:14:23.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:14:22 smithi047 ceph-mon[77147]: daemon mds.k assigned to filesystem cephfs as rank 4 (now has 5 ranks) 2022-05-13T23:14:23.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:14:22 smithi047 ceph-mon[77147]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2022-05-13T23:14:23.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:14:22 smithi047 ceph-mon[77147]: Cluster is now healthy 2022-05-13T23:14:23.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:14:22 smithi047 ceph-mon[77147]: fsmap cephfs:4 {0=l=up:active,1=i=up:active,2=f=up:active,3=c=up:active} 8 up:standby 2022-05-13T23:14:23.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:14:22 smithi047 ceph-mon[77147]: fsmap cephfs:5 1 up:creating 7 up:standby 4 up:active 2022-05-13T23:14:23.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:14:22 smithi047 ceph-mon[77147]: daemon mds.k is now active in filesystem cephfs as rank 4 2022-05-13T23:14:23.072 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:14:22 smithi002 ceph-mon[71816]: from='client.? 172.21.15.2:0/3025885018' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2022-05-13T23:14:23.073 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:14:22 smithi002 ceph-mon[71816]: from='client.? 172.21.15.2:0/2765872222' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2022-05-13T23:14:23.073 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:14:22 smithi002 ceph-mon[71816]: mds.? [v2:172.21.15.115:6832/381933929,v1:172.21.15.115:6833/381933929] up:active 2022-05-13T23:14:23.073 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:14:22 smithi002 ceph-mon[71816]: daemon mds.k assigned to filesystem cephfs as rank 4 (now has 5 ranks) 2022-05-13T23:14:23.073 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:14:22 smithi002 ceph-mon[71816]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2022-05-13T23:14:23.074 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:14:22 smithi002 ceph-mon[71816]: Cluster is now healthy 2022-05-13T23:14:23.074 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:14:22 smithi002 ceph-mon[71816]: fsmap cephfs:4 {0=l=up:active,1=i=up:active,2=f=up:active,3=c=up:active} 8 up:standby 2022-05-13T23:14:23.074 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:14:22 smithi002 ceph-mon[71816]: fsmap cephfs:5 1 up:creating 7 up:standby 4 up:active 2022-05-13T23:14:23.074 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:14:22 smithi002 ceph-mon[71816]: daemon mds.k is now active in filesystem cephfs as rank 4 2022-05-13T23:14:23.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:14:22 smithi115 ceph-mon[78593]: from='client.? 172.21.15.2:0/3025885018' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2022-05-13T23:14:23.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:14:22 smithi115 ceph-mon[78593]: from='client.? 172.21.15.2:0/2765872222' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2022-05-13T23:14:23.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:14:22 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6832/381933929,v1:172.21.15.115:6833/381933929] up:active 2022-05-13T23:14:23.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:14:22 smithi115 ceph-mon[78593]: daemon mds.k assigned to filesystem cephfs as rank 4 (now has 5 ranks) 2022-05-13T23:14:23.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:14:22 smithi115 ceph-mon[78593]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2022-05-13T23:14:23.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:14:22 smithi115 ceph-mon[78593]: Cluster is now healthy 2022-05-13T23:14:23.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:14:22 smithi115 ceph-mon[78593]: fsmap cephfs:4 {0=l=up:active,1=i=up:active,2=f=up:active,3=c=up:active} 8 up:standby 2022-05-13T23:14:23.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:14:22 smithi115 ceph-mon[78593]: fsmap cephfs:5 1 up:creating 7 up:standby 4 up:active 2022-05-13T23:14:23.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:14:22 smithi115 ceph-mon[78593]: daemon mds.k is now active in filesystem cephfs as rank 4 2022-05-13T23:14:23.206 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:23.205+0000 7f8f0ffff700 1 -- 172.21.15.2:0/1333998809 --> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f8f1002a5b0 con 0x7f8ef8096ce0 2022-05-13T23:14:23.217 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:23.216+0000 7f8ef77fe700 1 -- 172.21.15.2:0/1333998809 <== mgr.14148 v2:172.21.15.2:6800/2788339627 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+332192 (secure 0 0 0) 0x7f8f1002a5b0 con 0x7f8ef8096ce0 2022-05-13T23:14:23.222 INFO:teuthology.orchestra.run.smithi002.stdout: 2022-05-13T23:14:23.225 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:23.221+0000 7f8f0ffff700 1 -- 172.21.15.2:0/1333998809 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f8ef8096ce0 msgr2=0x7f8ef8099190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:14:23.225 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:23.221+0000 7f8f0ffff700 1 --2- 172.21.15.2:0/1333998809 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f8ef8096ce0 0x7f8ef8099190 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f8f100fc5b0 tx=0x7f8f0000b040 comp rx=0 tx=0).stop 2022-05-13T23:14:23.225 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:23.221+0000 7f8f0ffff700 1 -- 172.21.15.2:0/1333998809 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8f100fc750 msgr2=0x7f8f1006e830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:14:23.226 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:23.221+0000 7f8f0ffff700 1 --2- 172.21.15.2:0/1333998809 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8f100fc750 0x7f8f1006e830 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f8f0400e8f0 tx=0x7f8f0400b040 comp rx=0 tx=0).stop 2022-05-13T23:14:23.226 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:23.221+0000 7f8f0ffff700 1 -- 172.21.15.2:0/1333998809 shutdown_connections 2022-05-13T23:14:23.226 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:23.221+0000 7f8f0ffff700 1 --2- 172.21.15.2:0/1333998809 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f8f100fd0e0 0x7f8f1006ed70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:14:23.226 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:23.221+0000 7f8f0ffff700 1 --2- 172.21.15.2:0/1333998809 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8f100fb4b0 0x7f8f100701d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:14:23.227 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:23.221+0000 7f8f0ffff700 1 --2- 172.21.15.2:0/1333998809 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f8ef8096ce0 0x7f8ef8099190 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:14:23.227 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:23.221+0000 7f8f0ffff700 1 --2- 172.21.15.2:0/1333998809 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8f100fc750 0x7f8f1006e830 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:14:23.227 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:23.221+0000 7f8f0ffff700 1 -- 172.21.15.2:0/1333998809 >> 172.21.15.2:0/1333998809 conn(0x7f8f100f6c80 msgr2=0x7f8f100ffbf0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-05-13T23:14:23.227 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:23.221+0000 7f8f0ffff700 1 -- 172.21.15.2:0/1333998809 shutdown_connections 2022-05-13T23:14:23.227 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:23.221+0000 7f8f0ffff700 1 -- 172.21.15.2:0/1333998809 wait complete. 2022-05-13T23:14:23.228 INFO:teuthology.orchestra.run.smithi002.stderr:dumped all 2022-05-13T23:14:23.251 INFO:teuthology.orchestra.run.smithi002.stdout:{"pg_ready":true,"pg_map":{"version":291,"stamp":"2022-05-13T23:14:21.234600+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":594536,"num_objects":37,"num_object_clones":0,"num_object_copies":111,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":255,"num_write_kb":2957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1057856,"num_keys_recovered":0,"num_objects_omap":21,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":203,"ondisk_log_size":203,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":214,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":77924,"kb_used_data":11972,"kb_used_omap":0,"kb_used_meta":65856,"kb_avail":1124863900,"statfs":{"total":1151940427776,"available":1151860633600,"internally_reserved":0,"allocated":12259328,"data_stored":6628751,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":67436544},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":23,"apply_latency_ms":23,"commit_latency_ns":23000000,"apply_latency_ns":23000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":4168,"num_objects":35,"num_object_clones":0,"num_object_copies":105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":74,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":21,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001502"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":9,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-05-13T23:14:16.267128+0000","last_change":"2022-05-13T23:14:16.267128+0000","last_active":"2022-05-13T23:14:16.267128+0000","last_peered":"2022-05-13T23:14:16.267128+0000","last_clean":"2022-05-13T23:14:16.267128+0000","last_became_active":"2022-05-13T23:14:16.266687+0000","last_became_peered":"2022-05-13T23:14:16.266687+0000","last_unstale":"2022-05-13T23:14:16.267128+0000","last_undegraded":"2022-05-13T23:14:16.267128+0000","last_fullsized":"2022-05-13T23:14:16.267128+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_clean_scrub_stamp":"2022-05-13T23:14:15.659362+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T04:40:46.150557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":2,"reported_epoch":82,"state":"creating+activating","last_fresh":"2022-05-13T23:14:16.252511+0000","last_change":"2022-05-13T23:14:16.252678+0000","last_active":"2022-05-13T23:14:15.659362+0000","last_peered":"2022-05-13T23:14:15.659362+0000","last_clean":"2022-05-13T23:14:15.659362+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2022-05-13T23:14:16.252511+0000","last_undegraded":"2022-05-13T23:14:16.252511+0000","last_fullsized":"2022-05-13T23:14:16.252511+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_clean_scrub_stamp":"2022-05-13T23:14:15.659362+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T00:32:19.185037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":["0","10","11"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":3,"reported_epoch":82,"state":"creating+activating","last_fresh":"2022-05-13T23:14:16.249762+0000","last_change":"2022-05-13T23:14:16.249762+0000","last_active":"2022-05-13T23:14:15.659362+0000","last_peered":"2022-05-13T23:14:15.659362+0000","last_clean":"2022-05-13T23:14:15.659362+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2022-05-13T23:14:16.249762+0000","last_undegraded":"2022-05-13T23:14:16.249762+0000","last_fullsized":"2022-05-13T23:14:16.249762+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_clean_scrub_stamp":"2022-05-13T23:14:15.659362+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T10:03:03.774806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":["0","1","2"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":10,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-05-13T23:14:16.257175+0000","last_change":"2022-05-13T23:14:16.255885+0000","last_active":"2022-05-13T23:14:16.257175+0000","last_peered":"2022-05-13T23:14:16.257175+0000","last_clean":"2022-05-13T23:14:16.257175+0000","last_became_active":"2022-05-13T23:14:16.255551+0000","last_became_peered":"2022-05-13T23:14:16.255551+0000","last_unstale":"2022-05-13T23:14:16.257175+0000","last_undegraded":"2022-05-13T23:14:16.257175+0000","last_fullsized":"2022-05-13T23:14:16.257175+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_clean_scrub_stamp":"2022-05-13T23:14:15.659362+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T10:26:40.578066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":9,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-05-13T23:14:16.271384+0000","last_change":"2022-05-13T23:14:16.271384+0000","last_active":"2022-05-13T23:14:16.271384+0000","last_peered":"2022-05-13T23:14:16.271384+0000","last_clean":"2022-05-13T23:14:16.271384+0000","last_became_active":"2022-05-13T23:14:16.271045+0000","last_became_peered":"2022-05-13T23:14:16.271045+0000","last_unstale":"2022-05-13T23:14:16.271384+0000","last_undegraded":"2022-05-13T23:14:16.271384+0000","last_fullsized":"2022-05-13T23:14:16.271384+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_clean_scrub_stamp":"2022-05-13T23:14:15.659362+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T04:51:50.618628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":10,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-05-13T23:14:16.276293+0000","last_change":"2022-05-13T23:14:16.275906+0000","last_active":"2022-05-13T23:14:16.276293+0000","last_peered":"2022-05-13T23:14:16.276293+0000","last_clean":"2022-05-13T23:14:16.276293+0000","last_became_active":"2022-05-13T23:14:16.275651+0000","last_became_peered":"2022-05-13T23:14:16.275651+0000","last_unstale":"2022-05-13T23:14:16.276293+0000","last_undegraded":"2022-05-13T23:14:16.276293+0000","last_fullsized":"2022-05-13T23:14:16.276293+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_clean_scrub_stamp":"2022-05-13T23:14:15.659362+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T02:29:33.019268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":10,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-05-13T23:14:16.276560+0000","last_change":"2022-05-13T23:14:16.276150+0000","last_active":"2022-05-13T23:14:16.276560+0000","last_peered":"2022-05-13T23:14:16.276560+0000","last_clean":"2022-05-13T23:14:16.276560+0000","last_became_active":"2022-05-13T23:14:16.275751+0000","last_became_peered":"2022-05-13T23:14:16.275751+0000","last_unstale":"2022-05-13T23:14:16.276560+0000","last_undegraded":"2022-05-13T23:14:16.276560+0000","last_fullsized":"2022-05-13T23:14:16.276560+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_clean_scrub_stamp":"2022-05-13T23:14:15.659362+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T04:01:48.496870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":10,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-05-13T23:14:16.270951+0000","last_change":"2022-05-13T23:14:16.268816+0000","last_active":"2022-05-13T23:14:16.270951+0000","last_peered":"2022-05-13T23:14:16.270951+0000","last_clean":"2022-05-13T23:14:16.270951+0000","last_became_active":"2022-05-13T23:14:16.268429+0000","last_became_peered":"2022-05-13T23:14:16.268429+0000","last_unstale":"2022-05-13T23:14:16.270951+0000","last_undegraded":"2022-05-13T23:14:16.270951+0000","last_fullsized":"2022-05-13T23:14:16.270951+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_clean_scrub_stamp":"2022-05-13T23:14:15.659362+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T00:14:51.685287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":10,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-05-13T23:14:16.265890+0000","last_change":"2022-05-13T23:14:16.263971+0000","last_active":"2022-05-13T23:14:16.265890+0000","last_peered":"2022-05-13T23:14:16.265890+0000","last_clean":"2022-05-13T23:14:16.265890+0000","last_became_active":"2022-05-13T23:14:16.263625+0000","last_became_peered":"2022-05-13T23:14:16.263625+0000","last_unstale":"2022-05-13T23:14:16.265890+0000","last_undegraded":"2022-05-13T23:14:16.265890+0000","last_fullsized":"2022-05-13T23:14:16.265890+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_clean_scrub_stamp":"2022-05-13T23:14:15.659362+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T03:11:11.037746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":10,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-05-13T23:14:16.258455+0000","last_change":"2022-05-13T23:14:16.258111+0000","last_active":"2022-05-13T23:14:16.258455+0000","last_peered":"2022-05-13T23:14:16.258455+0000","last_clean":"2022-05-13T23:14:16.258455+0000","last_became_active":"2022-05-13T23:14:16.257768+0000","last_became_peered":"2022-05-13T23:14:16.257768+0000","last_unstale":"2022-05-13T23:14:16.258455+0000","last_undegraded":"2022-05-13T23:14:16.258455+0000","last_fullsized":"2022-05-13T23:14:16.258455+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_clean_scrub_stamp":"2022-05-13T23:14:15.659362+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T07:27:48.568156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":10,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-05-13T23:14:16.268155+0000","last_change":"2022-05-13T23:14:16.267256+0000","last_active":"2022-05-13T23:14:16.268155+0000","last_peered":"2022-05-13T23:14:16.268155+0000","last_clean":"2022-05-13T23:14:16.268155+0000","last_became_active":"2022-05-13T23:14:16.266868+0000","last_became_peered":"2022-05-13T23:14:16.266868+0000","last_unstale":"2022-05-13T23:14:16.268155+0000","last_undegraded":"2022-05-13T23:14:16.268155+0000","last_fullsized":"2022-05-13T23:14:16.268155+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_clean_scrub_stamp":"2022-05-13T23:14:15.659362+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T03:28:58.592648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":10,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-05-13T23:14:16.257196+0000","last_change":"2022-05-13T23:14:16.256051+0000","last_active":"2022-05-13T23:14:16.257196+0000","last_peered":"2022-05-13T23:14:16.257196+0000","last_clean":"2022-05-13T23:14:16.257196+0000","last_became_active":"2022-05-13T23:14:16.255729+0000","last_became_peered":"2022-05-13T23:14:16.255729+0000","last_unstale":"2022-05-13T23:14:16.257196+0000","last_undegraded":"2022-05-13T23:14:16.257196+0000","last_fullsized":"2022-05-13T23:14:16.257196+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_clean_scrub_stamp":"2022-05-13T23:14:15.659362+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T09:34:22.448938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":10,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-05-13T23:14:16.268884+0000","last_change":"2022-05-13T23:14:16.266285+0000","last_active":"2022-05-13T23:14:16.268884+0000","last_peered":"2022-05-13T23:14:16.268884+0000","last_clean":"2022-05-13T23:14:16.268884+0000","last_became_active":"2022-05-13T23:14:16.264899+0000","last_became_peered":"2022-05-13T23:14:16.264899+0000","last_unstale":"2022-05-13T23:14:16.268884+0000","last_undegraded":"2022-05-13T23:14:16.268884+0000","last_fullsized":"2022-05-13T23:14:16.268884+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_clean_scrub_stamp":"2022-05-13T23:14:15.659362+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T09:50:56.211965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":2,"reported_epoch":82,"state":"creating+activating","last_fresh":"2022-05-13T23:14:16.249497+0000","last_change":"2022-05-13T23:14:16.249760+0000","last_active":"2022-05-13T23:14:15.659362+0000","last_peered":"2022-05-13T23:14:15.659362+0000","last_clean":"2022-05-13T23:14:15.659362+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2022-05-13T23:14:16.249497+0000","last_undegraded":"2022-05-13T23:14:16.249497+0000","last_fullsized":"2022-05-13T23:14:16.249497+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_clean_scrub_stamp":"2022-05-13T23:14:15.659362+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T05:01:25.159085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":["0","8","10"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":3,"reported_epoch":82,"state":"creating+activating","last_fresh":"2022-05-13T23:14:16.249135+0000","last_change":"2022-05-13T23:14:16.249135+0000","last_active":"2022-05-13T23:14:15.659362+0000","last_peered":"2022-05-13T23:14:15.659362+0000","last_clean":"2022-05-13T23:14:15.659362+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2022-05-13T23:14:16.249135+0000","last_undegraded":"2022-05-13T23:14:16.249135+0000","last_fullsized":"2022-05-13T23:14:16.249135+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_clean_scrub_stamp":"2022-05-13T23:14:15.659362+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T02:16:47.449089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":["0","2","4"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":10,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-05-13T23:14:16.270716+0000","last_change":"2022-05-13T23:14:16.270173+0000","last_active":"2022-05-13T23:14:16.270716+0000","last_peered":"2022-05-13T23:14:16.270716+0000","last_clean":"2022-05-13T23:14:16.270716+0000","last_became_active":"2022-05-13T23:14:16.269859+0000","last_became_peered":"2022-05-13T23:14:16.269859+0000","last_unstale":"2022-05-13T23:14:16.270716+0000","last_undegraded":"2022-05-13T23:14:16.270716+0000","last_fullsized":"2022-05-13T23:14:16.270716+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_clean_scrub_stamp":"2022-05-13T23:14:15.659362+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T03:31:45.566757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":9,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-05-13T23:14:16.269290+0000","last_change":"2022-05-13T23:14:16.269290+0000","last_active":"2022-05-13T23:14:16.269290+0000","last_peered":"2022-05-13T23:14:16.269290+0000","last_clean":"2022-05-13T23:14:16.269290+0000","last_became_active":"2022-05-13T23:14:16.268985+0000","last_became_peered":"2022-05-13T23:14:16.268985+0000","last_unstale":"2022-05-13T23:14:16.269290+0000","last_undegraded":"2022-05-13T23:14:16.269290+0000","last_fullsized":"2022-05-13T23:14:16.269290+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_clean_scrub_stamp":"2022-05-13T23:14:15.659362+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T03:58:45.885549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":9,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-05-13T23:14:16.266204+0000","last_change":"2022-05-13T23:14:16.266204+0000","last_active":"2022-05-13T23:14:16.266204+0000","last_peered":"2022-05-13T23:14:16.266204+0000","last_clean":"2022-05-13T23:14:16.266204+0000","last_became_active":"2022-05-13T23:14:16.262256+0000","last_became_peered":"2022-05-13T23:14:16.262256+0000","last_unstale":"2022-05-13T23:14:16.266204+0000","last_undegraded":"2022-05-13T23:14:16.266204+0000","last_fullsized":"2022-05-13T23:14:16.266204+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_clean_scrub_stamp":"2022-05-13T23:14:15.659362+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T00:53:26.594598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":10,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-05-13T23:14:16.254638+0000","last_change":"2022-05-13T23:14:16.252421+0000","last_active":"2022-05-13T23:14:16.254638+0000","last_peered":"2022-05-13T23:14:16.254638+0000","last_clean":"2022-05-13T23:14:16.254638+0000","last_became_active":"2022-05-13T23:14:16.250585+0000","last_became_peered":"2022-05-13T23:14:16.250585+0000","last_unstale":"2022-05-13T23:14:16.254638+0000","last_undegraded":"2022-05-13T23:14:16.254638+0000","last_fullsized":"2022-05-13T23:14:16.254638+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_clean_scrub_stamp":"2022-05-13T23:14:15.659362+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T00:13:24.393163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"82'4","reported_seq":14,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-05-13T23:14:16.274879+0000","last_change":"2022-05-13T23:14:15.683837+0000","last_active":"2022-05-13T23:14:16.274879+0000","last_peered":"2022-05-13T23:14:16.274879+0000","last_clean":"2022-05-13T23:14:16.274879+0000","last_became_active":"2022-05-13T23:14:15.682976+0000","last_became_peered":"2022-05-13T23:14:15.682976+0000","last_unstale":"2022-05-13T23:14:16.274879+0000","last_undegraded":"2022-05-13T23:14:16.274879+0000","last_fullsized":"2022-05-13T23:14:16.274879+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:14.344206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:14.344206+0000","last_clean_scrub_stamp":"2022-05-13T23:14:14.344206+0000","objects_scrubbed":0,"log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T01:05:36.612390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":554,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":10,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-05-13T23:14:16.267355+0000","last_change":"2022-05-13T23:14:16.264501+0000","last_active":"2022-05-13T23:14:16.267355+0000","last_peered":"2022-05-13T23:14:16.267355+0000","last_clean":"2022-05-13T23:14:16.267355+0000","last_became_active":"2022-05-13T23:14:16.264105+0000","last_became_peered":"2022-05-13T23:14:16.264105+0000","last_unstale":"2022-05-13T23:14:16.267355+0000","last_undegraded":"2022-05-13T23:14:16.267355+0000","last_fullsized":"2022-05-13T23:14:16.267355+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_clean_scrub_stamp":"2022-05-13T23:14:15.659362+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T00:21:48.857254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"82'1","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-05-13T23:14:16.282001+0000","last_change":"2022-05-13T23:14:15.701757+0000","last_active":"2022-05-13T23:14:16.282001+0000","last_peered":"2022-05-13T23:14:16.282001+0000","last_clean":"2022-05-13T23:14:16.282001+0000","last_became_active":"2022-05-13T23:14:15.701428+0000","last_became_peered":"2022-05-13T23:14:15.701428+0000","last_unstale":"2022-05-13T23:14:16.282001+0000","last_undegraded":"2022-05-13T23:14:16.282001+0000","last_fullsized":"2022-05-13T23:14:16.282001+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:14.344206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:14.344206+0000","last_clean_scrub_stamp":"2022-05-13T23:14:14.344206+0000","objects_scrubbed":0,"log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T05:26:07.810433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":2,"reported_epoch":82,"state":"creating+activating","last_fresh":"2022-05-13T23:14:16.250105+0000","last_change":"2022-05-13T23:14:16.250286+0000","last_active":"2022-05-13T23:14:15.659362+0000","last_peered":"2022-05-13T23:14:15.659362+0000","last_clean":"2022-05-13T23:14:15.659362+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2022-05-13T23:14:16.250105+0000","last_undegraded":"2022-05-13T23:14:16.250105+0000","last_fullsized":"2022-05-13T23:14:16.250105+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_clean_scrub_stamp":"2022-05-13T23:14:15.659362+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T06:42:49.090643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":["0","1","9"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"82'2","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-05-13T23:14:16.263265+0000","last_change":"2022-05-13T23:14:15.683457+0000","last_active":"2022-05-13T23:14:16.263265+0000","last_peered":"2022-05-13T23:14:16.263265+0000","last_clean":"2022-05-13T23:14:16.263265+0000","last_became_active":"2022-05-13T23:14:15.682883+0000","last_became_peered":"2022-05-13T23:14:15.682883+0000","last_unstale":"2022-05-13T23:14:16.263265+0000","last_undegraded":"2022-05-13T23:14:16.263265+0000","last_fullsized":"2022-05-13T23:14:16.263265+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:14.344206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:14.344206+0000","last_clean_scrub_stamp":"2022-05-13T23:14:14.344206+0000","objects_scrubbed":0,"log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T07:38:58.996672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":10,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-05-13T23:14:16.261946+0000","last_change":"2022-05-13T23:14:16.261315+0000","last_active":"2022-05-13T23:14:16.261946+0000","last_peered":"2022-05-13T23:14:16.261946+0000","last_clean":"2022-05-13T23:14:16.261946+0000","last_became_active":"2022-05-13T23:14:16.260265+0000","last_became_peered":"2022-05-13T23:14:16.260265+0000","last_unstale":"2022-05-13T23:14:16.261946+0000","last_undegraded":"2022-05-13T23:14:16.261946+0000","last_fullsized":"2022-05-13T23:14:16.261946+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_clean_scrub_stamp":"2022-05-13T23:14:15.659362+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T08:02:58.080071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"82'2","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-05-13T23:14:16.262971+0000","last_change":"2022-05-13T23:14:15.682693+0000","last_active":"2022-05-13T23:14:16.262971+0000","last_peered":"2022-05-13T23:14:16.262971+0000","last_clean":"2022-05-13T23:14:16.262971+0000","last_became_active":"2022-05-13T23:14:15.682414+0000","last_became_peered":"2022-05-13T23:14:15.682414+0000","last_unstale":"2022-05-13T23:14:16.262971+0000","last_undegraded":"2022-05-13T23:14:16.262971+0000","last_fullsized":"2022-05-13T23:14:16.262971+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:14.344206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:14.344206+0000","last_clean_scrub_stamp":"2022-05-13T23:14:14.344206+0000","objects_scrubbed":0,"log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T09:15:17.699871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":10,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-05-13T23:14:16.261554+0000","last_change":"2022-05-13T23:14:16.261044+0000","last_active":"2022-05-13T23:14:16.261554+0000","last_peered":"2022-05-13T23:14:16.261554+0000","last_clean":"2022-05-13T23:14:16.261554+0000","last_became_active":"2022-05-13T23:14:16.260687+0000","last_became_peered":"2022-05-13T23:14:16.260687+0000","last_unstale":"2022-05-13T23:14:16.261554+0000","last_undegraded":"2022-05-13T23:14:16.261554+0000","last_fullsized":"2022-05-13T23:14:16.261554+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_clean_scrub_stamp":"2022-05-13T23:14:15.659362+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-14T23:38:23.413188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"83'5","reported_seq":16,"reported_epoch":83,"state":"active+clean","last_fresh":"2022-05-13T23:14:19.292636+0000","last_change":"2022-05-13T23:14:15.678240+0000","last_active":"2022-05-13T23:14:19.292636+0000","last_peered":"2022-05-13T23:14:19.292636+0000","last_clean":"2022-05-13T23:14:19.292636+0000","last_became_active":"2022-05-13T23:14:15.677918+0000","last_became_peered":"2022-05-13T23:14:15.677918+0000","last_unstale":"2022-05-13T23:14:19.292636+0000","last_undegraded":"2022-05-13T23:14:19.292636+0000","last_fullsized":"2022-05-13T23:14:19.292636+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:14.344206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:14.344206+0000","last_clean_scrub_stamp":"2022-05-13T23:14:14.344206+0000","objects_scrubbed":0,"log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T09:53:37.623728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":574,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":8,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-05-13T23:14:16.273411+0000","last_change":"2022-05-13T23:14:16.273537+0000","last_active":"2022-05-13T23:14:16.273411+0000","last_peered":"2022-05-13T23:14:16.273411+0000","last_clean":"2022-05-13T23:14:15.659362+0000","last_became_active":"2022-05-13T23:14:16.273127+0000","last_became_peered":"2022-05-13T23:14:16.273127+0000","last_unstale":"2022-05-13T23:14:16.273411+0000","last_undegraded":"2022-05-13T23:14:16.273411+0000","last_fullsized":"2022-05-13T23:14:16.273411+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_clean_scrub_stamp":"2022-05-13T23:14:15.659362+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T01:00:18.643721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"82'1","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-05-13T23:14:16.267285+0000","last_change":"2022-05-13T23:14:15.680562+0000","last_active":"2022-05-13T23:14:16.267285+0000","last_peered":"2022-05-13T23:14:16.267285+0000","last_clean":"2022-05-13T23:14:16.267285+0000","last_became_active":"2022-05-13T23:14:15.680163+0000","last_became_peered":"2022-05-13T23:14:15.680163+0000","last_unstale":"2022-05-13T23:14:16.267285+0000","last_undegraded":"2022-05-13T23:14:16.267285+0000","last_fullsized":"2022-05-13T23:14:16.267285+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:14.344206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:14.344206+0000","last_clean_scrub_stamp":"2022-05-13T23:14:14.344206+0000","objects_scrubbed":0,"log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T02:39:27.895407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":10,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-05-13T23:14:16.271643+0000","last_change":"2022-05-13T23:14:16.270878+0000","last_active":"2022-05-13T23:14:16.271643+0000","last_peered":"2022-05-13T23:14:16.271643+0000","last_clean":"2022-05-13T23:14:16.271643+0000","last_became_active":"2022-05-13T23:14:16.270416+0000","last_became_peered":"2022-05-13T23:14:16.270416+0000","last_unstale":"2022-05-13T23:14:16.271643+0000","last_undegraded":"2022-05-13T23:14:16.271643+0000","last_fullsized":"2022-05-13T23:14:16.271643+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_clean_scrub_stamp":"2022-05-13T23:14:15.659362+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T06:11:33.932416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"83'2","reported_seq":13,"reported_epoch":83,"state":"active+clean","last_fresh":"2022-05-13T23:14:20.303430+0000","last_change":"2022-05-13T23:14:15.676518+0000","last_active":"2022-05-13T23:14:20.303430+0000","last_peered":"2022-05-13T23:14:20.303430+0000","last_clean":"2022-05-13T23:14:20.303430+0000","last_became_active":"2022-05-13T23:14:15.676157+0000","last_became_peered":"2022-05-13T23:14:15.676157+0000","last_unstale":"2022-05-13T23:14:20.303430+0000","last_undegraded":"2022-05-13T23:14:20.303430+0000","last_fullsized":"2022-05-13T23:14:20.303430+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:14.344206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:14.344206+0000","last_clean_scrub_stamp":"2022-05-13T23:14:14.344206+0000","objects_scrubbed":0,"log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T10:16:49.469347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":10,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-05-13T23:14:16.262532+0000","last_change":"2022-05-13T23:14:16.257424+0000","last_active":"2022-05-13T23:14:16.262532+0000","last_peered":"2022-05-13T23:14:16.262532+0000","last_clean":"2022-05-13T23:14:16.262532+0000","last_became_active":"2022-05-13T23:14:16.254550+0000","last_became_peered":"2022-05-13T23:14:16.254550+0000","last_unstale":"2022-05-13T23:14:16.262532+0000","last_undegraded":"2022-05-13T23:14:16.262532+0000","last_fullsized":"2022-05-13T23:14:16.262532+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_clean_scrub_stamp":"2022-05-13T23:14:15.659362+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T00:04:50.150368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"83'1","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2022-05-13T23:14:17.560883+0000","last_change":"2022-05-13T23:14:15.680378+0000","last_active":"2022-05-13T23:14:17.560883+0000","last_peered":"2022-05-13T23:14:17.560883+0000","last_clean":"2022-05-13T23:14:17.560883+0000","last_became_active":"2022-05-13T23:14:15.680063+0000","last_became_peered":"2022-05-13T23:14:15.680063+0000","last_unstale":"2022-05-13T23:14:17.560883+0000","last_undegraded":"2022-05-13T23:14:17.560883+0000","last_fullsized":"2022-05-13T23:14:17.560883+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:14.344206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:14.344206+0000","last_clean_scrub_stamp":"2022-05-13T23:14:14.344206+0000","objects_scrubbed":0,"log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-14T23:52:48.817491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":9,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-05-13T23:14:16.277131+0000","last_change":"2022-05-13T23:14:16.277131+0000","last_active":"2022-05-13T23:14:16.277131+0000","last_peered":"2022-05-13T23:14:16.277131+0000","last_clean":"2022-05-13T23:14:16.277131+0000","last_became_active":"2022-05-13T23:14:16.276815+0000","last_became_peered":"2022-05-13T23:14:16.276815+0000","last_unstale":"2022-05-13T23:14:16.277131+0000","last_undegraded":"2022-05-13T23:14:16.277131+0000","last_fullsized":"2022-05-13T23:14:16.277131+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_clean_scrub_stamp":"2022-05-13T23:14:15.659362+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-14T23:22:29.938073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":11,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-05-13T23:14:16.242578+0000","last_change":"2022-05-13T23:14:15.683981+0000","last_active":"2022-05-13T23:14:16.242578+0000","last_peered":"2022-05-13T23:14:16.242578+0000","last_clean":"2022-05-13T23:14:16.242578+0000","last_became_active":"2022-05-13T23:14:15.683560+0000","last_became_peered":"2022-05-13T23:14:15.683560+0000","last_unstale":"2022-05-13T23:14:16.242578+0000","last_undegraded":"2022-05-13T23:14:16.242578+0000","last_fullsized":"2022-05-13T23:14:16.242578+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:14.344206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:14.344206+0000","last_clean_scrub_stamp":"2022-05-13T23:14:14.344206+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T00:30:01.510592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":10,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-05-13T23:14:16.269905+0000","last_change":"2022-05-13T23:14:16.269244+0000","last_active":"2022-05-13T23:14:16.269905+0000","last_peered":"2022-05-13T23:14:16.269905+0000","last_clean":"2022-05-13T23:14:16.269905+0000","last_became_active":"2022-05-13T23:14:16.268949+0000","last_became_peered":"2022-05-13T23:14:16.268949+0000","last_unstale":"2022-05-13T23:14:16.269905+0000","last_undegraded":"2022-05-13T23:14:16.269905+0000","last_fullsized":"2022-05-13T23:14:16.269905+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_clean_scrub_stamp":"2022-05-13T23:14:15.659362+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T04:34:53.311191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"83'2","reported_seq":13,"reported_epoch":83,"state":"active+clean","last_fresh":"2022-05-13T23:14:19.290837+0000","last_change":"2022-05-13T23:14:15.683758+0000","last_active":"2022-05-13T23:14:19.290837+0000","last_peered":"2022-05-13T23:14:19.290837+0000","last_clean":"2022-05-13T23:14:19.290837+0000","last_became_active":"2022-05-13T23:14:15.683356+0000","last_became_peered":"2022-05-13T23:14:15.683356+0000","last_unstale":"2022-05-13T23:14:19.290837+0000","last_undegraded":"2022-05-13T23:14:19.290837+0000","last_fullsized":"2022-05-13T23:14:19.290837+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:14.344206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:14.344206+0000","last_clean_scrub_stamp":"2022-05-13T23:14:14.344206+0000","objects_scrubbed":0,"log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T08:00:55.113935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":3,"reported_epoch":82,"state":"creating+activating","last_fresh":"2022-05-13T23:14:16.250747+0000","last_change":"2022-05-13T23:14:16.250747+0000","last_active":"2022-05-13T23:14:15.659362+0000","last_peered":"2022-05-13T23:14:15.659362+0000","last_clean":"2022-05-13T23:14:15.659362+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2022-05-13T23:14:16.250747+0000","last_undegraded":"2022-05-13T23:14:16.250747+0000","last_fullsized":"2022-05-13T23:14:16.250747+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_clean_scrub_stamp":"2022-05-13T23:14:15.659362+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T06:22:45.140343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":["0","4","5"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"83'4","reported_seq":15,"reported_epoch":83,"state":"active+clean","last_fresh":"2022-05-13T23:14:17.561666+0000","last_change":"2022-05-13T23:14:15.701120+0000","last_active":"2022-05-13T23:14:17.561666+0000","last_peered":"2022-05-13T23:14:17.561666+0000","last_clean":"2022-05-13T23:14:17.561666+0000","last_became_active":"2022-05-13T23:14:15.700812+0000","last_became_peered":"2022-05-13T23:14:15.700812+0000","last_unstale":"2022-05-13T23:14:17.561666+0000","last_undegraded":"2022-05-13T23:14:17.561666+0000","last_fullsized":"2022-05-13T23:14:17.561666+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:14.344206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:14.344206+0000","last_clean_scrub_stamp":"2022-05-13T23:14:14.344206+0000","objects_scrubbed":0,"log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T02:16:02.909062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":644,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":10,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-05-13T23:14:16.266148+0000","last_change":"2022-05-13T23:14:16.265167+0000","last_active":"2022-05-13T23:14:16.266148+0000","last_peered":"2022-05-13T23:14:16.266148+0000","last_clean":"2022-05-13T23:14:16.266148+0000","last_became_active":"2022-05-13T23:14:16.264886+0000","last_became_peered":"2022-05-13T23:14:16.264886+0000","last_unstale":"2022-05-13T23:14:16.266148+0000","last_undegraded":"2022-05-13T23:14:16.266148+0000","last_fullsized":"2022-05-13T23:14:16.266148+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_clean_scrub_stamp":"2022-05-13T23:14:15.659362+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T00:26:06.650867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"82'6","reported_seq":17,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-05-13T23:14:16.269389+0000","last_change":"2022-05-13T23:14:15.682385+0000","last_active":"2022-05-13T23:14:16.269389+0000","last_peered":"2022-05-13T23:14:16.269389+0000","last_clean":"2022-05-13T23:14:16.269389+0000","last_became_active":"2022-05-13T23:14:15.681996+0000","last_became_peered":"2022-05-13T23:14:15.681996+0000","last_unstale":"2022-05-13T23:14:16.269389+0000","last_undegraded":"2022-05-13T23:14:16.269389+0000","last_fullsized":"2022-05-13T23:14:16.269389+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:14.344206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:14.344206+0000","last_clean_scrub_stamp":"2022-05-13T23:14:14.344206+0000","objects_scrubbed":0,"log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-14T23:44:23.766036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1536,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":9,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-05-13T23:14:16.263744+0000","last_change":"2022-05-13T23:14:16.263744+0000","last_active":"2022-05-13T23:14:16.263744+0000","last_peered":"2022-05-13T23:14:16.263744+0000","last_clean":"2022-05-13T23:14:16.263744+0000","last_became_active":"2022-05-13T23:14:16.263525+0000","last_became_peered":"2022-05-13T23:14:16.263525+0000","last_unstale":"2022-05-13T23:14:16.263744+0000","last_undegraded":"2022-05-13T23:14:16.263744+0000","last_fullsized":"2022-05-13T23:14:16.263744+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_clean_scrub_stamp":"2022-05-13T23:14:15.659362+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T04:23:44.856055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"83'6","reported_seq":17,"reported_epoch":83,"state":"active+clean","last_fresh":"2022-05-13T23:14:20.301591+0000","last_change":"2022-05-13T23:14:15.683079+0000","last_active":"2022-05-13T23:14:20.301591+0000","last_peered":"2022-05-13T23:14:20.301591+0000","last_clean":"2022-05-13T23:14:20.301591+0000","last_became_active":"2022-05-13T23:14:15.682750+0000","last_became_peered":"2022-05-13T23:14:15.682750+0000","last_unstale":"2022-05-13T23:14:20.301591+0000","last_undegraded":"2022-05-13T23:14:20.301591+0000","last_fullsized":"2022-05-13T23:14:20.301591+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:14.344206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:14.344206+0000","last_clean_scrub_stamp":"2022-05-13T23:14:14.344206+0000","objects_scrubbed":0,"log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T02:37:31.530579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":10,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-05-13T23:14:16.268332+0000","last_change":"2022-05-13T23:14:16.263943+0000","last_active":"2022-05-13T23:14:16.268332+0000","last_peered":"2022-05-13T23:14:16.268332+0000","last_clean":"2022-05-13T23:14:16.268332+0000","last_became_active":"2022-05-13T23:14:16.263595+0000","last_became_peered":"2022-05-13T23:14:16.263595+0000","last_unstale":"2022-05-13T23:14:16.268332+0000","last_undegraded":"2022-05-13T23:14:16.268332+0000","last_fullsized":"2022-05-13T23:14:16.268332+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_clean_scrub_stamp":"2022-05-13T23:14:15.659362+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T04:57:04.201317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"83'3","reported_seq":14,"reported_epoch":83,"state":"active+clean","last_fresh":"2022-05-13T23:14:19.292697+0000","last_change":"2022-05-13T23:14:15.683005+0000","last_active":"2022-05-13T23:14:19.292697+0000","last_peered":"2022-05-13T23:14:19.292697+0000","last_clean":"2022-05-13T23:14:19.292697+0000","last_became_active":"2022-05-13T23:14:15.682692+0000","last_became_peered":"2022-05-13T23:14:15.682692+0000","last_unstale":"2022-05-13T23:14:19.292697+0000","last_undegraded":"2022-05-13T23:14:19.292697+0000","last_fullsized":"2022-05-13T23:14:19.292697+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:14.344206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:14.344206+0000","last_clean_scrub_stamp":"2022-05-13T23:14:14.344206+0000","objects_scrubbed":0,"log_size":3,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T00:27:25.201141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":10,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-05-13T23:14:16.265569+0000","last_change":"2022-05-13T23:14:16.264351+0000","last_active":"2022-05-13T23:14:16.265569+0000","last_peered":"2022-05-13T23:14:16.265569+0000","last_clean":"2022-05-13T23:14:16.265569+0000","last_became_active":"2022-05-13T23:14:16.264023+0000","last_became_peered":"2022-05-13T23:14:16.264023+0000","last_unstale":"2022-05-13T23:14:16.265569+0000","last_undegraded":"2022-05-13T23:14:16.265569+0000","last_fullsized":"2022-05-13T23:14:16.265569+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_clean_scrub_stamp":"2022-05-13T23:14:15.659362+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T06:13:51.169049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":11,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-05-13T23:14:16.246274+0000","last_change":"2022-05-13T23:14:15.677080+0000","last_active":"2022-05-13T23:14:16.246274+0000","last_peered":"2022-05-13T23:14:16.246274+0000","last_clean":"2022-05-13T23:14:16.246274+0000","last_became_active":"2022-05-13T23:14:15.676743+0000","last_became_peered":"2022-05-13T23:14:15.676743+0000","last_unstale":"2022-05-13T23:14:16.246274+0000","last_undegraded":"2022-05-13T23:14:16.246274+0000","last_fullsized":"2022-05-13T23:14:16.246274+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:14.344206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:14.344206+0000","last_clean_scrub_stamp":"2022-05-13T23:14:14.344206+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T09:05:30.321765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":10,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-05-13T23:14:16.263982+0000","last_change":"2022-05-13T23:14:16.262582+0000","last_active":"2022-05-13T23:14:16.263982+0000","last_peered":"2022-05-13T23:14:16.263982+0000","last_clean":"2022-05-13T23:14:16.263982+0000","last_became_active":"2022-05-13T23:14:16.262242+0000","last_became_peered":"2022-05-13T23:14:16.262242+0000","last_unstale":"2022-05-13T23:14:16.263982+0000","last_undegraded":"2022-05-13T23:14:16.263982+0000","last_fullsized":"2022-05-13T23:14:16.263982+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_clean_scrub_stamp":"2022-05-13T23:14:15.659362+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T02:27:28.155612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"82'8","reported_seq":17,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-05-13T23:14:16.269913+0000","last_change":"2022-05-13T23:14:15.698465+0000","last_active":"2022-05-13T23:14:16.269913+0000","last_peered":"2022-05-13T23:14:16.269913+0000","last_clean":"2022-05-13T23:14:16.269913+0000","last_became_active":"2022-05-13T23:14:15.698131+0000","last_became_peered":"2022-05-13T23:14:15.698131+0000","last_unstale":"2022-05-13T23:14:16.269913+0000","last_undegraded":"2022-05-13T23:14:16.269913+0000","last_fullsized":"2022-05-13T23:14:16.269913+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:14.344206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:14.344206+0000","last_clean_scrub_stamp":"2022-05-13T23:14:14.344206+0000","objects_scrubbed":0,"log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T03:03:48.104345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":10,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-05-13T23:14:16.261102+0000","last_change":"2022-05-13T23:14:16.260579+0000","last_active":"2022-05-13T23:14:16.261102+0000","last_peered":"2022-05-13T23:14:16.261102+0000","last_clean":"2022-05-13T23:14:16.261102+0000","last_became_active":"2022-05-13T23:14:16.260326+0000","last_became_peered":"2022-05-13T23:14:16.260326+0000","last_unstale":"2022-05-13T23:14:16.261102+0000","last_undegraded":"2022-05-13T23:14:16.261102+0000","last_fullsized":"2022-05-13T23:14:16.261102+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_clean_scrub_stamp":"2022-05-13T23:14:15.659362+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T06:53:23.092756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"82'1","reported_seq":11,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-05-13T23:14:16.248983+0000","last_change":"2022-05-13T23:14:15.680655+0000","last_active":"2022-05-13T23:14:16.248983+0000","last_peered":"2022-05-13T23:14:16.248983+0000","last_clean":"2022-05-13T23:14:16.248983+0000","last_became_active":"2022-05-13T23:14:15.680357+0000","last_became_peered":"2022-05-13T23:14:15.680357+0000","last_unstale":"2022-05-13T23:14:16.248983+0000","last_undegraded":"2022-05-13T23:14:16.248983+0000","last_fullsized":"2022-05-13T23:14:16.248983+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:14.344206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:14.344206+0000","last_clean_scrub_stamp":"2022-05-13T23:14:14.344206+0000","objects_scrubbed":0,"log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T11:00:22.395993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":9,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-05-13T23:14:16.260574+0000","last_change":"2022-05-13T23:14:16.260574+0000","last_active":"2022-05-13T23:14:16.260574+0000","last_peered":"2022-05-13T23:14:16.260574+0000","last_clean":"2022-05-13T23:14:16.260574+0000","last_became_active":"2022-05-13T23:14:16.260258+0000","last_became_peered":"2022-05-13T23:14:16.260258+0000","last_unstale":"2022-05-13T23:14:16.260574+0000","last_undegraded":"2022-05-13T23:14:16.260574+0000","last_fullsized":"2022-05-13T23:14:16.260574+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_clean_scrub_stamp":"2022-05-13T23:14:15.659362+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T10:49:06.210598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":10,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-05-13T23:14:15.701835+0000","last_change":"2022-05-13T23:14:15.701322+0000","last_active":"2022-05-13T23:14:15.701835+0000","last_peered":"2022-05-13T23:14:15.701835+0000","last_clean":"2022-05-13T23:14:15.701835+0000","last_became_active":"2022-05-13T23:14:15.701007+0000","last_became_peered":"2022-05-13T23:14:15.701007+0000","last_unstale":"2022-05-13T23:14:15.701835+0000","last_undegraded":"2022-05-13T23:14:15.701835+0000","last_fullsized":"2022-05-13T23:14:15.701835+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:14.344206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:14.344206+0000","last_clean_scrub_stamp":"2022-05-13T23:14:14.344206+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T00:13:11.071695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":10,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-05-13T23:14:16.259088+0000","last_change":"2022-05-13T23:14:16.249579+0000","last_active":"2022-05-13T23:14:16.259088+0000","last_peered":"2022-05-13T23:14:16.259088+0000","last_clean":"2022-05-13T23:14:16.259088+0000","last_became_active":"2022-05-13T23:14:16.249245+0000","last_became_peered":"2022-05-13T23:14:16.249245+0000","last_unstale":"2022-05-13T23:14:16.259088+0000","last_undegraded":"2022-05-13T23:14:16.259088+0000","last_fullsized":"2022-05-13T23:14:16.259088+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_clean_scrub_stamp":"2022-05-13T23:14:15.659362+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T00:26:53.404909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":11,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-05-13T23:14:16.255333+0000","last_change":"2022-05-13T23:14:15.699433+0000","last_active":"2022-05-13T23:14:16.255333+0000","last_peered":"2022-05-13T23:14:16.255333+0000","last_clean":"2022-05-13T23:14:16.255333+0000","last_became_active":"2022-05-13T23:14:15.698920+0000","last_became_peered":"2022-05-13T23:14:15.698920+0000","last_unstale":"2022-05-13T23:14:16.255333+0000","last_undegraded":"2022-05-13T23:14:16.255333+0000","last_fullsized":"2022-05-13T23:14:16.255333+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:14.344206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:14.344206+0000","last_clean_scrub_stamp":"2022-05-13T23:14:14.344206+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T07:56:34.276565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":10,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-05-13T23:14:16.260728+0000","last_change":"2022-05-13T23:14:16.259866+0000","last_active":"2022-05-13T23:14:16.260728+0000","last_peered":"2022-05-13T23:14:16.260728+0000","last_clean":"2022-05-13T23:14:16.260728+0000","last_became_active":"2022-05-13T23:14:16.259531+0000","last_became_peered":"2022-05-13T23:14:16.259531+0000","last_unstale":"2022-05-13T23:14:16.260728+0000","last_undegraded":"2022-05-13T23:14:16.260728+0000","last_fullsized":"2022-05-13T23:14:16.260728+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_clean_scrub_stamp":"2022-05-13T23:14:15.659362+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T07:41:46.019217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":10,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-05-13T23:14:16.261105+0000","last_change":"2022-05-13T23:14:16.259613+0000","last_active":"2022-05-13T23:14:16.261105+0000","last_peered":"2022-05-13T23:14:16.261105+0000","last_clean":"2022-05-13T23:14:16.261105+0000","last_became_active":"2022-05-13T23:14:16.259274+0000","last_became_peered":"2022-05-13T23:14:16.259274+0000","last_unstale":"2022-05-13T23:14:16.261105+0000","last_undegraded":"2022-05-13T23:14:16.261105+0000","last_fullsized":"2022-05-13T23:14:16.261105+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_clean_scrub_stamp":"2022-05-13T23:14:15.659362+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T05:36:21.174814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"83'4","reported_seq":15,"reported_epoch":83,"state":"active+clean","last_fresh":"2022-05-13T23:14:20.304530+0000","last_change":"2022-05-13T23:14:15.701315+0000","last_active":"2022-05-13T23:14:20.304530+0000","last_peered":"2022-05-13T23:14:20.304530+0000","last_clean":"2022-05-13T23:14:20.304530+0000","last_became_active":"2022-05-13T23:14:15.700955+0000","last_became_peered":"2022-05-13T23:14:15.700955+0000","last_unstale":"2022-05-13T23:14:20.304530+0000","last_undegraded":"2022-05-13T23:14:20.304530+0000","last_fullsized":"2022-05-13T23:14:20.304530+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:14.344206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:14.344206+0000","last_clean_scrub_stamp":"2022-05-13T23:14:14.344206+0000","objects_scrubbed":0,"log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T10:17:23.916475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":10,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-05-13T23:14:16.274093+0000","last_change":"2022-05-13T23:14:16.272364+0000","last_active":"2022-05-13T23:14:16.274093+0000","last_peered":"2022-05-13T23:14:16.274093+0000","last_clean":"2022-05-13T23:14:16.274093+0000","last_became_active":"2022-05-13T23:14:16.271612+0000","last_became_peered":"2022-05-13T23:14:16.271612+0000","last_unstale":"2022-05-13T23:14:16.274093+0000","last_undegraded":"2022-05-13T23:14:16.274093+0000","last_fullsized":"2022-05-13T23:14:16.274093+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_clean_scrub_stamp":"2022-05-13T23:14:15.659362+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T07:25:47.039264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":11,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-05-13T23:14:16.251030+0000","last_change":"2022-05-13T23:14:15.676251+0000","last_active":"2022-05-13T23:14:16.251030+0000","last_peered":"2022-05-13T23:14:16.251030+0000","last_clean":"2022-05-13T23:14:16.251030+0000","last_became_active":"2022-05-13T23:14:15.675895+0000","last_became_peered":"2022-05-13T23:14:15.675895+0000","last_unstale":"2022-05-13T23:14:16.251030+0000","last_undegraded":"2022-05-13T23:14:16.251030+0000","last_fullsized":"2022-05-13T23:14:16.251030+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:14.344206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:14.344206+0000","last_clean_scrub_stamp":"2022-05-13T23:14:14.344206+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T06:03:57.398647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":343,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-05-13T23:14:16.253992+0000","last_change":"2022-05-13T23:13:02.805009+0000","last_active":"2022-05-13T23:14:16.253992+0000","last_peered":"2022-05-13T23:14:16.253992+0000","last_clean":"2022-05-13T23:14:16.253992+0000","last_became_active":"2022-05-13T23:13:02.804350+0000","last_became_peered":"2022-05-13T23:13:02.804350+0000","last_unstale":"2022-05-13T23:14:16.253992+0000","last_undegraded":"2022-05-13T23:14:16.253992+0000","last_fullsized":"2022-05-13T23:14:16.253992+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:09:32.001256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:09:32.001256+0000","last_clean_scrub_stamp":"2022-05-13T23:09:32.001256+0000","objects_scrubbed":0,"log_size":137,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T07:28:44.497144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1057856,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":10,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-05-13T23:14:16.267352+0000","last_change":"2022-05-13T23:14:16.261286+0000","last_active":"2022-05-13T23:14:16.267352+0000","last_peered":"2022-05-13T23:14:16.267352+0000","last_clean":"2022-05-13T23:14:16.267352+0000","last_became_active":"2022-05-13T23:14:16.260870+0000","last_became_peered":"2022-05-13T23:14:16.260870+0000","last_unstale":"2022-05-13T23:14:16.267352+0000","last_undegraded":"2022-05-13T23:14:16.267352+0000","last_fullsized":"2022-05-13T23:14:16.267352+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_clean_scrub_stamp":"2022-05-13T23:14:15.659362+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T08:14:34.804797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"83'1","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2022-05-13T23:14:17.561547+0000","last_change":"2022-05-13T23:14:15.686878+0000","last_active":"2022-05-13T23:14:17.561547+0000","last_peered":"2022-05-13T23:14:17.561547+0000","last_clean":"2022-05-13T23:14:17.561547+0000","last_became_active":"2022-05-13T23:14:15.686496+0000","last_became_peered":"2022-05-13T23:14:15.686496+0000","last_unstale":"2022-05-13T23:14:17.561547+0000","last_undegraded":"2022-05-13T23:14:17.561547+0000","last_fullsized":"2022-05-13T23:14:17.561547+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:14.344206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:14.344206+0000","last_clean_scrub_stamp":"2022-05-13T23:14:14.344206+0000","objects_scrubbed":0,"log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T09:19:28.089159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":10,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-05-13T23:14:15.698991+0000","last_change":"2022-05-13T23:14:15.698503+0000","last_active":"2022-05-13T23:14:15.698991+0000","last_peered":"2022-05-13T23:14:15.698991+0000","last_clean":"2022-05-13T23:14:15.698991+0000","last_became_active":"2022-05-13T23:14:15.698156+0000","last_became_peered":"2022-05-13T23:14:15.698156+0000","last_unstale":"2022-05-13T23:14:15.698991+0000","last_undegraded":"2022-05-13T23:14:15.698991+0000","last_fullsized":"2022-05-13T23:14:15.698991+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:14.344206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:14.344206+0000","last_clean_scrub_stamp":"2022-05-13T23:14:14.344206+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T09:03:21.133631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":10,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-05-13T23:14:16.271905+0000","last_change":"2022-05-13T23:14:16.270922+0000","last_active":"2022-05-13T23:14:16.271905+0000","last_peered":"2022-05-13T23:14:16.271905+0000","last_clean":"2022-05-13T23:14:16.271905+0000","last_became_active":"2022-05-13T23:14:16.270259+0000","last_became_peered":"2022-05-13T23:14:16.270259+0000","last_unstale":"2022-05-13T23:14:16.271905+0000","last_undegraded":"2022-05-13T23:14:16.271905+0000","last_fullsized":"2022-05-13T23:14:16.271905+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_clean_scrub_stamp":"2022-05-13T23:14:15.659362+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T02:48:22.255489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":10,"reported_epoch":81,"state":"active+clean","last_fresh":"2022-05-13T23:14:15.698665+0000","last_change":"2022-05-13T23:14:15.696304+0000","last_active":"2022-05-13T23:14:15.698665+0000","last_peered":"2022-05-13T23:14:15.698665+0000","last_clean":"2022-05-13T23:14:15.698665+0000","last_became_active":"2022-05-13T23:14:15.695476+0000","last_became_peered":"2022-05-13T23:14:15.695476+0000","last_unstale":"2022-05-13T23:14:15.698665+0000","last_undegraded":"2022-05-13T23:14:15.698665+0000","last_fullsized":"2022-05-13T23:14:15.698665+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:14.344206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:14.344206+0000","last_clean_scrub_stamp":"2022-05-13T23:14:14.344206+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T08:48:32.931789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":10,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-05-13T23:14:16.262969+0000","last_change":"2022-05-13T23:14:16.257063+0000","last_active":"2022-05-13T23:14:16.262969+0000","last_peered":"2022-05-13T23:14:16.262969+0000","last_clean":"2022-05-13T23:14:16.262969+0000","last_became_active":"2022-05-13T23:14:16.256757+0000","last_became_peered":"2022-05-13T23:14:16.256757+0000","last_unstale":"2022-05-13T23:14:16.262969+0000","last_undegraded":"2022-05-13T23:14:16.262969+0000","last_fullsized":"2022-05-13T23:14:16.262969+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_clean_scrub_stamp":"2022-05-13T23:14:15.659362+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T09:24:32.369318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"82'2","reported_seq":13,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-05-13T23:14:16.272570+0000","last_change":"2022-05-13T23:14:15.685568+0000","last_active":"2022-05-13T23:14:16.272570+0000","last_peered":"2022-05-13T23:14:16.272570+0000","last_clean":"2022-05-13T23:14:16.272570+0000","last_became_active":"2022-05-13T23:14:15.685203+0000","last_became_peered":"2022-05-13T23:14:15.685203+0000","last_unstale":"2022-05-13T23:14:16.272570+0000","last_undegraded":"2022-05-13T23:14:16.272570+0000","last_fullsized":"2022-05-13T23:14:16.272570+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:14.344206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:14.344206+0000","last_clean_scrub_stamp":"2022-05-13T23:14:14.344206+0000","objects_scrubbed":0,"log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T03:55:33.271069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":3,"reported_epoch":82,"state":"creating+activating","last_fresh":"2022-05-13T23:14:16.252040+0000","last_change":"2022-05-13T23:14:16.252040+0000","last_active":"2022-05-13T23:14:15.659362+0000","last_peered":"2022-05-13T23:14:15.659362+0000","last_clean":"2022-05-13T23:14:15.659362+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2022-05-13T23:14:16.252040+0000","last_undegraded":"2022-05-13T23:14:16.252040+0000","last_fullsized":"2022-05-13T23:14:16.252040+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_clean_scrub_stamp":"2022-05-13T23:14:15.659362+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T03:25:42.803064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":["0","3","4"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":11,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-05-13T23:14:16.248471+0000","last_change":"2022-05-13T23:14:15.698443+0000","last_active":"2022-05-13T23:14:16.248471+0000","last_peered":"2022-05-13T23:14:16.248471+0000","last_clean":"2022-05-13T23:14:16.248471+0000","last_became_active":"2022-05-13T23:14:15.698158+0000","last_became_peered":"2022-05-13T23:14:15.698158+0000","last_unstale":"2022-05-13T23:14:16.248471+0000","last_undegraded":"2022-05-13T23:14:16.248471+0000","last_fullsized":"2022-05-13T23:14:16.248471+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:14.344206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:14.344206+0000","last_clean_scrub_stamp":"2022-05-13T23:14:14.344206+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T05:06:14.946233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":9,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-05-13T23:14:16.264110+0000","last_change":"2022-05-13T23:14:16.264110+0000","last_active":"2022-05-13T23:14:16.264110+0000","last_peered":"2022-05-13T23:14:16.264110+0000","last_clean":"2022-05-13T23:14:16.264110+0000","last_became_active":"2022-05-13T23:14:16.263830+0000","last_became_peered":"2022-05-13T23:14:16.263830+0000","last_unstale":"2022-05-13T23:14:16.264110+0000","last_undegraded":"2022-05-13T23:14:16.264110+0000","last_fullsized":"2022-05-13T23:14:16.264110+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_clean_scrub_stamp":"2022-05-13T23:14:15.659362+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T00:06:26.724597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"83'3","reported_seq":14,"reported_epoch":83,"state":"active+clean","last_fresh":"2022-05-13T23:14:19.292890+0000","last_change":"2022-05-13T23:14:15.693926+0000","last_active":"2022-05-13T23:14:19.292890+0000","last_peered":"2022-05-13T23:14:19.292890+0000","last_clean":"2022-05-13T23:14:19.292890+0000","last_became_active":"2022-05-13T23:14:15.692163+0000","last_became_peered":"2022-05-13T23:14:15.692163+0000","last_unstale":"2022-05-13T23:14:19.292890+0000","last_undegraded":"2022-05-13T23:14:19.292890+0000","last_fullsized":"2022-05-13T23:14:19.292890+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:14.344206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:14.344206+0000","last_clean_scrub_stamp":"2022-05-13T23:14:14.344206+0000","objects_scrubbed":0,"log_size":3,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T03:00:12.522714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":9,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-05-13T23:14:16.265409+0000","last_change":"2022-05-13T23:14:16.265409+0000","last_active":"2022-05-13T23:14:16.265409+0000","last_peered":"2022-05-13T23:14:16.265409+0000","last_clean":"2022-05-13T23:14:16.265409+0000","last_became_active":"2022-05-13T23:14:16.265151+0000","last_became_peered":"2022-05-13T23:14:16.265151+0000","last_unstale":"2022-05-13T23:14:16.265409+0000","last_undegraded":"2022-05-13T23:14:16.265409+0000","last_fullsized":"2022-05-13T23:14:16.265409+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_clean_scrub_stamp":"2022-05-13T23:14:15.659362+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T04:05:43.762623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":11,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-05-13T23:14:16.241423+0000","last_change":"2022-05-13T23:14:15.681975+0000","last_active":"2022-05-13T23:14:16.241423+0000","last_peered":"2022-05-13T23:14:16.241423+0000","last_clean":"2022-05-13T23:14:16.241423+0000","last_became_active":"2022-05-13T23:14:15.681391+0000","last_became_peered":"2022-05-13T23:14:15.681391+0000","last_unstale":"2022-05-13T23:14:16.241423+0000","last_undegraded":"2022-05-13T23:14:16.241423+0000","last_fullsized":"2022-05-13T23:14:16.241423+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:14.344206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:14.344206+0000","last_clean_scrub_stamp":"2022-05-13T23:14:14.344206+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T06:56:25.732024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":10,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-05-13T23:14:16.274936+0000","last_change":"2022-05-13T23:14:16.272443+0000","last_active":"2022-05-13T23:14:16.274936+0000","last_peered":"2022-05-13T23:14:16.274936+0000","last_clean":"2022-05-13T23:14:16.274936+0000","last_became_active":"2022-05-13T23:14:16.272031+0000","last_became_peered":"2022-05-13T23:14:16.272031+0000","last_unstale":"2022-05-13T23:14:16.274936+0000","last_undegraded":"2022-05-13T23:14:16.274936+0000","last_fullsized":"2022-05-13T23:14:16.274936+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_clean_scrub_stamp":"2022-05-13T23:14:15.659362+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T10:23:55.694108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"82'2","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-05-13T23:14:16.264699+0000","last_change":"2022-05-13T23:14:15.681877+0000","last_active":"2022-05-13T23:14:16.264699+0000","last_peered":"2022-05-13T23:14:16.264699+0000","last_clean":"2022-05-13T23:14:16.264699+0000","last_became_active":"2022-05-13T23:14:15.681579+0000","last_became_peered":"2022-05-13T23:14:15.681579+0000","last_unstale":"2022-05-13T23:14:16.264699+0000","last_undegraded":"2022-05-13T23:14:16.264699+0000","last_fullsized":"2022-05-13T23:14:16.264699+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:14.344206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:14.344206+0000","last_clean_scrub_stamp":"2022-05-13T23:14:14.344206+0000","objects_scrubbed":0,"log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T07:00:11.132049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":10,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-05-13T23:14:16.267797+0000","last_change":"2022-05-13T23:14:16.266877+0000","last_active":"2022-05-13T23:14:16.267797+0000","last_peered":"2022-05-13T23:14:16.267797+0000","last_clean":"2022-05-13T23:14:16.267797+0000","last_became_active":"2022-05-13T23:14:16.266551+0000","last_became_peered":"2022-05-13T23:14:16.266551+0000","last_unstale":"2022-05-13T23:14:16.267797+0000","last_undegraded":"2022-05-13T23:14:16.267797+0000","last_fullsized":"2022-05-13T23:14:16.267797+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_clean_scrub_stamp":"2022-05-13T23:14:15.659362+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T05:57:29.394410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"83'3","reported_seq":14,"reported_epoch":83,"state":"active+clean","last_fresh":"2022-05-13T23:14:20.302018+0000","last_change":"2022-05-13T23:14:15.699586+0000","last_active":"2022-05-13T23:14:20.302018+0000","last_peered":"2022-05-13T23:14:20.302018+0000","last_clean":"2022-05-13T23:14:20.302018+0000","last_became_active":"2022-05-13T23:14:15.699215+0000","last_became_peered":"2022-05-13T23:14:15.699215+0000","last_unstale":"2022-05-13T23:14:20.302018+0000","last_undegraded":"2022-05-13T23:14:20.302018+0000","last_fullsized":"2022-05-13T23:14:20.302018+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:14.344206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:14.344206+0000","last_clean_scrub_stamp":"2022-05-13T23:14:14.344206+0000","objects_scrubbed":0,"log_size":3,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T04:14:54.395550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":554,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":10,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-05-13T23:14:16.267333+0000","last_change":"2022-05-13T23:14:16.265730+0000","last_active":"2022-05-13T23:14:16.267333+0000","last_peered":"2022-05-13T23:14:16.267333+0000","last_clean":"2022-05-13T23:14:16.267333+0000","last_became_active":"2022-05-13T23:14:16.265435+0000","last_became_peered":"2022-05-13T23:14:16.265435+0000","last_unstale":"2022-05-13T23:14:16.267333+0000","last_undegraded":"2022-05-13T23:14:16.267333+0000","last_fullsized":"2022-05-13T23:14:16.267333+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_clean_scrub_stamp":"2022-05-13T23:14:15.659362+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T08:55:57.574132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"82'1","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-05-13T23:14:16.270336+0000","last_change":"2022-05-13T23:14:15.682402+0000","last_active":"2022-05-13T23:14:16.270336+0000","last_peered":"2022-05-13T23:14:16.270336+0000","last_clean":"2022-05-13T23:14:16.270336+0000","last_became_active":"2022-05-13T23:14:15.682105+0000","last_became_peered":"2022-05-13T23:14:15.682105+0000","last_unstale":"2022-05-13T23:14:16.270336+0000","last_undegraded":"2022-05-13T23:14:16.270336+0000","last_fullsized":"2022-05-13T23:14:16.270336+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:14.344206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:14.344206+0000","last_clean_scrub_stamp":"2022-05-13T23:14:14.344206+0000","objects_scrubbed":0,"log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T10:37:55.092658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":10,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-05-13T23:14:16.264851+0000","last_change":"2022-05-13T23:14:16.263001+0000","last_active":"2022-05-13T23:14:16.264851+0000","last_peered":"2022-05-13T23:14:16.264851+0000","last_clean":"2022-05-13T23:14:16.264851+0000","last_became_active":"2022-05-13T23:14:16.262646+0000","last_became_peered":"2022-05-13T23:14:16.262646+0000","last_unstale":"2022-05-13T23:14:16.264851+0000","last_undegraded":"2022-05-13T23:14:16.264851+0000","last_fullsized":"2022-05-13T23:14:16.264851+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_clean_scrub_stamp":"2022-05-13T23:14:15.659362+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T05:38:53.024966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"82'2","reported_seq":13,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-05-13T23:14:16.264403+0000","last_change":"2022-05-13T23:14:15.690434+0000","last_active":"2022-05-13T23:14:16.264403+0000","last_peered":"2022-05-13T23:14:16.264403+0000","last_clean":"2022-05-13T23:14:16.264403+0000","last_became_active":"2022-05-13T23:14:15.690032+0000","last_became_peered":"2022-05-13T23:14:15.690032+0000","last_unstale":"2022-05-13T23:14:16.264403+0000","last_undegraded":"2022-05-13T23:14:16.264403+0000","last_fullsized":"2022-05-13T23:14:16.264403+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:14.344206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:14.344206+0000","last_clean_scrub_stamp":"2022-05-13T23:14:14.344206+0000","objects_scrubbed":0,"log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T06:39:08.251311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":10,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-05-13T23:14:16.265645+0000","last_change":"2022-05-13T23:14:16.264791+0000","last_active":"2022-05-13T23:14:16.265645+0000","last_peered":"2022-05-13T23:14:16.265645+0000","last_clean":"2022-05-13T23:14:16.265645+0000","last_became_active":"2022-05-13T23:14:16.264461+0000","last_became_peered":"2022-05-13T23:14:16.264461+0000","last_unstale":"2022-05-13T23:14:16.265645+0000","last_undegraded":"2022-05-13T23:14:16.265645+0000","last_fullsized":"2022-05-13T23:14:16.265645+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_clean_scrub_stamp":"2022-05-13T23:14:15.659362+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T03:02:04.280220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":9,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-05-13T23:14:16.260468+0000","last_change":"2022-05-13T23:14:16.260468+0000","last_active":"2022-05-13T23:14:16.260468+0000","last_peered":"2022-05-13T23:14:16.260468+0000","last_clean":"2022-05-13T23:14:16.260468+0000","last_became_active":"2022-05-13T23:14:16.260211+0000","last_became_peered":"2022-05-13T23:14:16.260211+0000","last_unstale":"2022-05-13T23:14:16.260468+0000","last_undegraded":"2022-05-13T23:14:16.260468+0000","last_fullsized":"2022-05-13T23:14:16.260468+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_clean_scrub_stamp":"2022-05-13T23:14:15.659362+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T05:26:57.769099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":10,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-05-13T23:14:16.254999+0000","last_change":"2022-05-13T23:14:16.253997+0000","last_active":"2022-05-13T23:14:16.254999+0000","last_peered":"2022-05-13T23:14:16.254999+0000","last_clean":"2022-05-13T23:14:16.254999+0000","last_became_active":"2022-05-13T23:14:16.253685+0000","last_became_peered":"2022-05-13T23:14:16.253685+0000","last_unstale":"2022-05-13T23:14:16.254999+0000","last_undegraded":"2022-05-13T23:14:16.254999+0000","last_fullsized":"2022-05-13T23:14:16.254999+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_clean_scrub_stamp":"2022-05-13T23:14:15.659362+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T07:42:26.765244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":10,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-05-13T23:14:16.266256+0000","last_change":"2022-05-13T23:14:16.265207+0000","last_active":"2022-05-13T23:14:16.266256+0000","last_peered":"2022-05-13T23:14:16.266256+0000","last_clean":"2022-05-13T23:14:16.266256+0000","last_became_active":"2022-05-13T23:14:16.264881+0000","last_became_peered":"2022-05-13T23:14:16.264881+0000","last_unstale":"2022-05-13T23:14:16.266256+0000","last_undegraded":"2022-05-13T23:14:16.266256+0000","last_fullsized":"2022-05-13T23:14:16.266256+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_clean_scrub_stamp":"2022-05-13T23:14:15.659362+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T04:01:48.894233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":3,"reported_epoch":82,"state":"creating+activating","last_fresh":"2022-05-13T23:14:16.249811+0000","last_change":"2022-05-13T23:14:16.249811+0000","last_active":"2022-05-13T23:14:15.659362+0000","last_peered":"2022-05-13T23:14:15.659362+0000","last_clean":"2022-05-13T23:14:15.659362+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2022-05-13T23:14:16.249811+0000","last_undegraded":"2022-05-13T23:14:16.249811+0000","last_fullsized":"2022-05-13T23:14:16.249811+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_clean_scrub_stamp":"2022-05-13T23:14:15.659362+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T03:11:10.799827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":["0","4","6"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":10,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-05-13T23:14:16.268810+0000","last_change":"2022-05-13T23:14:16.265421+0000","last_active":"2022-05-13T23:14:16.268810+0000","last_peered":"2022-05-13T23:14:16.268810+0000","last_clean":"2022-05-13T23:14:16.268810+0000","last_became_active":"2022-05-13T23:14:16.265057+0000","last_became_peered":"2022-05-13T23:14:16.265057+0000","last_unstale":"2022-05-13T23:14:16.268810+0000","last_undegraded":"2022-05-13T23:14:16.268810+0000","last_fullsized":"2022-05-13T23:14:16.268810+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_clean_scrub_stamp":"2022-05-13T23:14:15.659362+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T07:45:53.177341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":9,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-05-13T23:14:16.266317+0000","last_change":"2022-05-13T23:14:16.266317+0000","last_active":"2022-05-13T23:14:16.266317+0000","last_peered":"2022-05-13T23:14:16.266317+0000","last_clean":"2022-05-13T23:14:16.266317+0000","last_became_active":"2022-05-13T23:14:16.265959+0000","last_became_peered":"2022-05-13T23:14:16.265959+0000","last_unstale":"2022-05-13T23:14:16.266317+0000","last_undegraded":"2022-05-13T23:14:16.266317+0000","last_fullsized":"2022-05-13T23:14:16.266317+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_clean_scrub_stamp":"2022-05-13T23:14:15.659362+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T03:37:42.277033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":10,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-05-13T23:14:16.255848+0000","last_change":"2022-05-13T23:14:16.254350+0000","last_active":"2022-05-13T23:14:16.255848+0000","last_peered":"2022-05-13T23:14:16.255848+0000","last_clean":"2022-05-13T23:14:16.255848+0000","last_became_active":"2022-05-13T23:14:16.253940+0000","last_became_peered":"2022-05-13T23:14:16.253940+0000","last_unstale":"2022-05-13T23:14:16.255848+0000","last_undegraded":"2022-05-13T23:14:16.255848+0000","last_fullsized":"2022-05-13T23:14:16.255848+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_clean_scrub_stamp":"2022-05-13T23:14:15.659362+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T10:08:56.052316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":10,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-05-13T23:14:16.267243+0000","last_change":"2022-05-13T23:14:16.263153+0000","last_active":"2022-05-13T23:14:16.267243+0000","last_peered":"2022-05-13T23:14:16.267243+0000","last_clean":"2022-05-13T23:14:16.267243+0000","last_became_active":"2022-05-13T23:14:16.262652+0000","last_became_peered":"2022-05-13T23:14:16.262652+0000","last_unstale":"2022-05-13T23:14:16.267243+0000","last_undegraded":"2022-05-13T23:14:16.267243+0000","last_fullsized":"2022-05-13T23:14:16.267243+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_clean_scrub_stamp":"2022-05-13T23:14:15.659362+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T07:53:46.499880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":10,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-05-13T23:14:16.271182+0000","last_change":"2022-05-13T23:14:16.270358+0000","last_active":"2022-05-13T23:14:16.271182+0000","last_peered":"2022-05-13T23:14:16.271182+0000","last_clean":"2022-05-13T23:14:16.271182+0000","last_became_active":"2022-05-13T23:14:16.270099+0000","last_became_peered":"2022-05-13T23:14:16.270099+0000","last_unstale":"2022-05-13T23:14:16.271182+0000","last_undegraded":"2022-05-13T23:14:16.271182+0000","last_fullsized":"2022-05-13T23:14:16.271182+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_clean_scrub_stamp":"2022-05-13T23:14:15.659362+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T00:42:07.782569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":10,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-05-13T23:14:16.253696+0000","last_change":"2022-05-13T23:14:16.251188+0000","last_active":"2022-05-13T23:14:16.253696+0000","last_peered":"2022-05-13T23:14:16.253696+0000","last_clean":"2022-05-13T23:14:16.253696+0000","last_became_active":"2022-05-13T23:14:16.250816+0000","last_became_peered":"2022-05-13T23:14:16.250816+0000","last_unstale":"2022-05-13T23:14:16.253696+0000","last_undegraded":"2022-05-13T23:14:16.253696+0000","last_fullsized":"2022-05-13T23:14:16.253696+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_clean_scrub_stamp":"2022-05-13T23:14:15.659362+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T00:58:14.898104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":10,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-05-13T23:14:16.265927+0000","last_change":"2022-05-13T23:14:16.264125+0000","last_active":"2022-05-13T23:14:16.265927+0000","last_peered":"2022-05-13T23:14:16.265927+0000","last_clean":"2022-05-13T23:14:16.265927+0000","last_became_active":"2022-05-13T23:14:16.263806+0000","last_became_peered":"2022-05-13T23:14:16.263806+0000","last_unstale":"2022-05-13T23:14:16.265927+0000","last_undegraded":"2022-05-13T23:14:16.265927+0000","last_fullsized":"2022-05-13T23:14:16.265927+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_clean_scrub_stamp":"2022-05-13T23:14:15.659362+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T09:26:04.559765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":10,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-05-13T23:14:16.268902+0000","last_change":"2022-05-13T23:14:16.266991+0000","last_active":"2022-05-13T23:14:16.268902+0000","last_peered":"2022-05-13T23:14:16.268902+0000","last_clean":"2022-05-13T23:14:16.268902+0000","last_became_active":"2022-05-13T23:14:16.265551+0000","last_became_peered":"2022-05-13T23:14:16.265551+0000","last_unstale":"2022-05-13T23:14:16.268902+0000","last_undegraded":"2022-05-13T23:14:16.268902+0000","last_fullsized":"2022-05-13T23:14:16.268902+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_clean_scrub_stamp":"2022-05-13T23:14:15.659362+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T00:40:56.664387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":9,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-05-13T23:14:16.267302+0000","last_change":"2022-05-13T23:14:16.267302+0000","last_active":"2022-05-13T23:14:16.267302+0000","last_peered":"2022-05-13T23:14:16.267302+0000","last_clean":"2022-05-13T23:14:16.267302+0000","last_became_active":"2022-05-13T23:14:16.266151+0000","last_became_peered":"2022-05-13T23:14:16.266151+0000","last_unstale":"2022-05-13T23:14:16.267302+0000","last_undegraded":"2022-05-13T23:14:16.267302+0000","last_fullsized":"2022-05-13T23:14:16.267302+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_clean_scrub_stamp":"2022-05-13T23:14:15.659362+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T07:30:04.546337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1057856,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":3194880,"data_stored":3173568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":4168,"num_objects":35,"num_object_clones":0,"num_object_copies":105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":74,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":21,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":147456,"data_stored":10040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":66,"ondisk_log_size":66,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":78,"seq":335007449101,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":5856,"kb_used_data":856,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":93739296,"statfs":{"total":95995035648,"available":95989039104,"internally_reserved":0,"allocated":876544,"data_stored":416256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41199999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.437}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.30099999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42999999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47199999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38300000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51100000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.502}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.373}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57099999999999995}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54600000000000004}]}]},{"osd":10,"up_from":71,"seq":304942678034,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6600,"kb_used_data":1408,"kb_used_omap":0,"kb_used_meta":5184,"kb_avail":93738552,"statfs":{"total":95995035648,"available":95988277248,"internally_reserved":0,"allocated":1441792,"data_stored":955149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5308416},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.755}]},{"osd":1,"last update":"Fri May 13 23:14:05 2022","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.72899999999999998,"5min":0.72899999999999998,"15min":0.72899999999999998},"last":0.40600000000000003},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.57999999999999996,"15min":0.57999999999999996},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.83099999999999996,"5min":0.83099999999999996,"15min":0.83099999999999996},"last":0.42299999999999999}]},{"osd":2,"last update":"Fri May 13 23:14:05 2022","interfaces":[{"interface":"back","average":{"1min":0.69499999999999995,"5min":0.69499999999999995,"15min":0.69499999999999995},"min":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":1.3280000000000001,"5min":1.3280000000000001,"15min":1.3280000000000001},"last":0.52600000000000002},{"interface":"front","average":{"1min":0.70799999999999996,"5min":0.70799999999999996,"15min":0.70799999999999996},"min":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":0.88800000000000001,"5min":0.88800000000000001,"15min":0.88800000000000001},"last":0.64800000000000002}]},{"osd":3,"last update":"Fri May 13 23:14:05 2022","interfaces":[{"interface":"back","average":{"1min":0.73999999999999999,"5min":0.73999999999999999,"15min":0.73999999999999999},"min":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"max":{"1min":0.90100000000000002,"5min":0.90100000000000002,"15min":0.90100000000000002},"last":0.59099999999999997},{"interface":"front","average":{"1min":0.77000000000000002,"5min":0.77000000000000002,"15min":0.77000000000000002},"min":{"1min":0.60199999999999998,"5min":0.60199999999999998,"15min":0.60199999999999998},"max":{"1min":0.93500000000000005,"5min":0.93500000000000005,"15min":0.93500000000000005},"last":0.55800000000000005}]},{"osd":4,"last update":"Fri May 13 23:14:05 2022","interfaces":[{"interface":"back","average":{"1min":0.68000000000000005,"5min":0.68000000000000005,"15min":0.68000000000000005},"min":{"1min":0.432,"5min":0.432,"15min":0.432},"max":{"1min":0.97699999999999998,"5min":0.97699999999999998,"15min":0.97699999999999998},"last":0.51800000000000002},{"interface":"front","average":{"1min":0.65200000000000002,"5min":0.65200000000000002,"15min":0.65200000000000002},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":1.0720000000000001,"5min":1.0720000000000001,"15min":1.0720000000000001},"last":0.46600000000000003}]},{"osd":5,"last update":"Fri May 13 23:14:05 2022","interfaces":[{"interface":"back","average":{"1min":0.78400000000000003,"5min":0.78400000000000003,"15min":0.78400000000000003},"min":{"1min":0.496,"5min":0.496,"15min":0.496},"max":{"1min":1.4299999999999999,"5min":1.4299999999999999,"15min":1.4299999999999999},"last":0.58099999999999996},{"interface":"front","average":{"1min":0.79800000000000004,"5min":0.79800000000000004,"15min":0.79800000000000004},"min":{"1min":0.55400000000000005,"5min":0.55400000000000005,"15min":0.55400000000000005},"max":{"1min":1.0609999999999999,"5min":1.0609999999999999,"15min":1.0609999999999999},"last":0.68799999999999994}]},{"osd":6,"last update":"Fri May 13 23:14:05 2022","interfaces":[{"interface":"back","average":{"1min":0.88700000000000001,"5min":0.88700000000000001,"15min":0.88700000000000001},"min":{"1min":0.64600000000000002,"5min":0.64600000000000002,"15min":0.64600000000000002},"max":{"1min":1.157,"5min":1.157,"15min":1.157},"last":0.54200000000000004},{"interface":"front","average":{"1min":0.88500000000000001,"5min":0.88500000000000001,"15min":0.88500000000000001},"min":{"1min":0.60199999999999998,"5min":0.60199999999999998,"15min":0.60199999999999998},"max":{"1min":1.216,"5min":1.216,"15min":1.216},"last":0.66100000000000003}]},{"osd":7,"last update":"Fri May 13 23:14:05 2022","interfaces":[{"interface":"back","average":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":1.585,"5min":1.585,"15min":1.585},"last":0.496},{"interface":"front","average":{"1min":0.80300000000000005,"5min":0.80300000000000005,"15min":0.80300000000000005},"min":{"1min":0.47999999999999998,"5min":0.47999999999999998,"15min":0.47999999999999998},"max":{"1min":1.0469999999999999,"5min":1.0469999999999999,"15min":1.0469999999999999},"last":0.502}]},{"osd":8,"last update":"Fri May 13 23:14:05 2022","interfaces":[{"interface":"back","average":{"1min":0.86399999999999999,"5min":0.86399999999999999,"15min":0.86399999999999999},"min":{"1min":0.65900000000000003,"5min":0.65900000000000003,"15min":0.65900000000000003},"max":{"1min":1.163,"5min":1.163,"15min":1.163},"last":0.73299999999999998},{"interface":"front","average":{"1min":0.88200000000000001,"5min":0.88200000000000001,"15min":0.88200000000000001},"min":{"1min":0.51400000000000001,"5min":0.51400000000000001,"15min":0.51400000000000001},"max":{"1min":1.476,"5min":1.476,"15min":1.476},"last":0.71199999999999997}]},{"osd":9,"last update":"Fri May 13 23:14:05 2022","interfaces":[{"interface":"back","average":{"1min":0.94999999999999996,"5min":0.94999999999999996,"15min":0.94999999999999996},"min":{"1min":0.68999999999999995,"5min":0.68999999999999995,"15min":0.68999999999999995},"max":{"1min":1.369,"5min":1.369,"15min":1.369},"last":0.77000000000000002},{"interface":"front","average":{"1min":0.96799999999999997,"5min":0.96799999999999997,"15min":0.96799999999999997},"min":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":1.819,"5min":1.819,"15min":1.819},"last":0.67300000000000004}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72099999999999997}]}]},{"osd":9,"up_from":66,"seq":283467841557,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":5908,"kb_used_data":844,"kb_used_omap":0,"kb_used_meta":5056,"kb_avail":93739244,"statfs":{"total":95995035648,"available":95988985856,"internally_reserved":0,"allocated":864256,"data_stored":407191,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5177344},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri May 13 23:13:37 2022","interfaces":[{"interface":"back","average":{"1min":0.65200000000000002,"5min":0.65200000000000002,"15min":0.65200000000000002},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":1.024,"5min":1.024,"15min":1.024},"last":0.52100000000000002},{"interface":"front","average":{"1min":0.72299999999999998,"5min":0.72299999999999998,"15min":0.72299999999999998},"min":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":1.044,"5min":1.044,"15min":1.044},"last":0.64500000000000002}]},{"osd":1,"last update":"Fri May 13 23:13:37 2022","interfaces":[{"interface":"back","average":{"1min":0.755,"5min":0.755,"15min":0.755},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":1.3029999999999999,"5min":1.3029999999999999,"15min":1.3029999999999999},"last":0.57599999999999996},{"interface":"front","average":{"1min":0.92000000000000004,"5min":0.92000000000000004,"15min":0.92000000000000004},"min":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":1.498,"5min":1.498,"15min":1.498},"last":0.68500000000000005}]},{"osd":2,"last update":"Fri May 13 23:13:37 2022","interfaces":[{"interface":"back","average":{"1min":0.89900000000000002,"5min":0.89900000000000002,"15min":0.89900000000000002},"min":{"1min":0.58699999999999997,"5min":0.58699999999999997,"15min":0.58699999999999997},"max":{"1min":1.4419999999999999,"5min":1.4419999999999999,"15min":1.4419999999999999},"last":0.59399999999999997},{"interface":"front","average":{"1min":0.82699999999999996,"5min":0.82699999999999996,"15min":0.82699999999999996},"min":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"max":{"1min":1.5309999999999999,"5min":1.5309999999999999,"15min":1.5309999999999999},"last":0.48199999999999998}]},{"osd":3,"last update":"Fri May 13 23:13:37 2022","interfaces":[{"interface":"back","average":{"1min":0.878,"5min":0.878,"15min":0.878},"min":{"1min":0.51200000000000001,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":1.2410000000000001,"5min":1.2410000000000001,"15min":1.2410000000000001},"last":0.78900000000000003},{"interface":"front","average":{"1min":1.002,"5min":1.002,"15min":1.002},"min":{"1min":0.622,"5min":0.622,"15min":0.622},"max":{"1min":2.2069999999999999,"5min":2.2069999999999999,"15min":2.2069999999999999},"last":0.80600000000000005}]},{"osd":4,"last update":"Fri May 13 23:13:37 2022","interfaces":[{"interface":"back","average":{"1min":0.93000000000000005,"5min":0.93000000000000005,"15min":0.93000000000000005},"min":{"1min":0.47699999999999998,"5min":0.47699999999999998,"15min":0.47699999999999998},"max":{"1min":1.5600000000000001,"5min":1.5600000000000001,"15min":1.5600000000000001},"last":0.65500000000000003},{"interface":"front","average":{"1min":0.95799999999999996,"5min":0.95799999999999996,"15min":0.95799999999999996},"min":{"1min":0.58899999999999997,"5min":0.58899999999999997,"15min":0.58899999999999997},"max":{"1min":1.3320000000000001,"5min":1.3320000000000001,"15min":1.3320000000000001},"last":0.81899999999999995}]},{"osd":5,"last update":"Fri May 13 23:13:37 2022","interfaces":[{"interface":"back","average":{"1min":0.97099999999999997,"5min":0.97099999999999997,"15min":0.97099999999999997},"min":{"1min":0.56299999999999994,"5min":0.56299999999999994,"15min":0.56299999999999994},"max":{"1min":1.593,"5min":1.593,"15min":1.593},"last":0.61699999999999999},{"interface":"front","average":{"1min":1.02,"5min":1.02,"15min":1.02},"min":{"1min":0.68600000000000005,"5min":0.68600000000000005,"15min":0.68600000000000005},"max":{"1min":1.3700000000000001,"5min":1.3700000000000001,"15min":1.3700000000000001},"last":0.84499999999999997}]},{"osd":6,"last update":"Fri May 13 23:13:37 2022","interfaces":[{"interface":"back","average":{"1min":1.1399999999999999,"5min":1.1399999999999999,"15min":1.1399999999999999},"min":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"max":{"1min":2.254,"5min":2.254,"15min":2.254},"last":0.83499999999999996},{"interface":"front","average":{"1min":1.0609999999999999,"5min":1.0609999999999999,"15min":1.0609999999999999},"min":{"1min":0.70599999999999996,"5min":0.70599999999999996,"15min":0.70599999999999996},"max":{"1min":1.649,"5min":1.649,"15min":1.649},"last":0.76100000000000001}]},{"osd":7,"last update":"Fri May 13 23:13:37 2022","interfaces":[{"interface":"back","average":{"1min":1.119,"5min":1.119,"15min":1.119},"min":{"1min":0.63,"5min":0.63,"15min":0.63},"max":{"1min":2.101,"5min":2.101,"15min":2.101},"last":0.91700000000000004},{"interface":"front","average":{"1min":1.1240000000000001,"5min":1.1240000000000001,"15min":1.1240000000000001},"min":{"1min":0.69599999999999995,"5min":0.69599999999999995,"15min":0.69599999999999995},"max":{"1min":1.895,"5min":1.895,"15min":1.895},"last":0.90900000000000003}]},{"osd":8,"last update":"Fri May 13 23:13:37 2022","interfaces":[{"interface":"back","average":{"1min":1.069,"5min":1.069,"15min":1.069},"min":{"1min":0.59899999999999998,"5min":0.59899999999999998,"15min":0.59899999999999998},"max":{"1min":1.6140000000000001,"5min":1.6140000000000001,"15min":1.6140000000000001},"last":0.86599999999999999},{"interface":"front","average":{"1min":1.1419999999999999,"5min":1.1419999999999999,"15min":1.1419999999999999},"min":{"1min":0.71599999999999997,"5min":0.71599999999999997,"15min":0.71599999999999997},"max":{"1min":2.1789999999999998,"5min":2.1789999999999998,"15min":2.1789999999999998},"last":0.88500000000000001}]},{"osd":10,"last update":"Fri May 13 23:14:02 2022","interfaces":[{"interface":"back","average":{"1min":1.1040000000000001,"5min":1.1040000000000001,"15min":1.1040000000000001},"min":{"1min":0.66800000000000004,"5min":0.66800000000000004,"15min":0.66800000000000004},"max":{"1min":1.7829999999999999,"5min":1.7829999999999999,"15min":1.7829999999999999},"last":0.82999999999999996},{"interface":"front","average":{"1min":1.1319999999999999,"5min":1.1319999999999999,"15min":1.1319999999999999},"min":{"1min":0.73099999999999998,"5min":0.73099999999999998,"15min":0.73099999999999998},"max":{"1min":1.748,"5min":1.748,"15min":1.748},"last":0.94799999999999995}]}]},{"osd":8,"up_from":58,"seq":249108103197,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6036,"kb_used_data":844,"kb_used_omap":0,"kb_used_meta":5184,"kb_avail":93739116,"statfs":{"total":95995035648,"available":95988854784,"internally_reserved":0,"allocated":864256,"data_stored":407191,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5308416},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri May 13 23:14:08 2022","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.60399999999999998,"15min":0.60799999999999998},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.88900000000000001,"5min":0.88900000000000001,"15min":0.88900000000000001},"last":0.52600000000000002},{"interface":"front","average":{"1min":0.61699999999999999,"5min":0.59199999999999997,"15min":0.58799999999999997},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":1.137,"5min":1.137,"15min":1.137},"last":0.48199999999999998}]},{"osd":1,"last update":"Fri May 13 23:14:08 2022","interfaces":[{"interface":"back","average":{"1min":0.53900000000000003,"5min":0.56699999999999995,"15min":0.57099999999999995},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.67200000000000004,"5min":0.78900000000000003,"15min":0.78900000000000003},"last":0.59199999999999997},{"interface":"front","average":{"1min":0.57299999999999995,"5min":0.60199999999999998,"15min":0.60699999999999998},"min":{"1min":0.38900000000000001,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.72599999999999998,"5min":0.80700000000000005,"15min":0.80700000000000005},"last":0.69399999999999995}]},{"osd":2,"last update":"Fri May 13 23:14:08 2022","interfaces":[{"interface":"back","average":{"1min":0.44,"5min":0.45600000000000002,"15min":0.45900000000000002},"min":{"1min":0.24199999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.77400000000000002,"5min":0.77400000000000002,"15min":0.77400000000000002},"last":0.46500000000000002},{"interface":"front","average":{"1min":0.443,"5min":0.45600000000000002,"15min":0.45800000000000002},"min":{"1min":0.27800000000000002,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.79100000000000004,"5min":0.79100000000000004,"15min":0.79100000000000004},"last":0.45300000000000001}]},{"osd":3,"last update":"Fri May 13 23:14:08 2022","interfaces":[{"interface":"back","average":{"1min":0.626,"5min":0.64800000000000002,"15min":0.65200000000000002},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.88400000000000001,"5min":0.88400000000000001,"15min":0.88400000000000001},"last":0.59799999999999998},{"interface":"front","average":{"1min":0.63300000000000001,"5min":0.64300000000000002,"15min":0.64500000000000002},"min":{"1min":0.439,"5min":0.439,"15min":0.439},"max":{"1min":0.82899999999999996,"5min":0.83299999999999996,"15min":0.83299999999999996},"last":0.504}]},{"osd":4,"last update":"Fri May 13 23:14:08 2022","interfaces":[{"interface":"back","average":{"1min":0.61099999999999999,"5min":0.621,"15min":0.623},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":1.573,"5min":1.573,"15min":1.573},"last":0.58199999999999996},{"interface":"front","average":{"1min":0.57299999999999995,"5min":0.61899999999999999,"15min":0.627},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":0.746,"5min":0.90500000000000003,"15min":0.90500000000000003},"last":0.70199999999999996}]},{"osd":5,"last update":"Fri May 13 23:14:08 2022","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.57399999999999995,"15min":0.58299999999999996},"min":{"1min":0.22800000000000001,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":0.85199999999999998,"5min":0.875,"15min":0.875},"last":0.49099999999999999},{"interface":"front","average":{"1min":0.498,"5min":0.56299999999999994,"15min":0.57399999999999995},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.86199999999999999,"5min":0.88200000000000001,"15min":0.88200000000000001},"last":0.44}]},{"osd":6,"last update":"Fri May 13 23:14:08 2022","interfaces":[{"interface":"back","average":{"1min":0.70299999999999996,"5min":0.74199999999999999,"15min":0.748},"min":{"1min":0.44400000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":0.873,"5min":0.94099999999999995,"15min":0.94099999999999995},"last":0.53800000000000003},{"interface":"front","average":{"1min":0.72099999999999997,"5min":0.754,"15min":0.76000000000000001},"min":{"1min":0.51700000000000002,"5min":0.504,"15min":0.504},"max":{"1min":0.91700000000000004,"5min":0.95699999999999996,"15min":0.95699999999999996},"last":0.58599999999999997}]},{"osd":7,"last update":"Fri May 13 23:14:08 2022","interfaces":[{"interface":"back","average":{"1min":0.67100000000000004,"5min":0.69999999999999996,"15min":0.70499999999999996},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.94799999999999995,"5min":0.95799999999999996,"15min":0.95799999999999996},"last":0.63500000000000001},{"interface":"front","average":{"1min":0.70399999999999996,"5min":0.69999999999999996,"15min":0.69999999999999996},"min":{"1min":0.49099999999999999,"5min":0.437,"15min":0.437},"max":{"1min":1.609,"5min":1.609,"15min":1.609},"last":0.72899999999999998}]},{"osd":9,"last update":"Fri May 13 23:13:38 2022","interfaces":[{"interface":"back","average":{"1min":0.76500000000000001,"5min":0.76500000000000001,"15min":0.76500000000000001},"min":{"1min":0.51500000000000001,"5min":0.51500000000000001,"15min":0.51500000000000001},"max":{"1min":0.999,"5min":0.999,"15min":0.999},"last":0.68600000000000005},{"interface":"front","average":{"1min":0.80100000000000005,"5min":0.80100000000000005,"15min":0.80100000000000005},"min":{"1min":0.59899999999999998,"5min":0.59899999999999998,"15min":0.59899999999999998},"max":{"1min":0.996,"5min":0.996,"15min":0.996},"last":0.67900000000000005}]},{"osd":10,"last update":"Fri May 13 23:14:02 2022","interfaces":[{"interface":"back","average":{"1min":0.72699999999999998,"5min":0.72699999999999998,"15min":0.72699999999999998},"min":{"1min":0.45800000000000002,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":0.97799999999999998,"5min":0.97799999999999998,"15min":0.97799999999999998},"last":0.65700000000000003},{"interface":"front","average":{"1min":0.69699999999999995,"5min":0.69699999999999995,"15min":0.69699999999999995},"min":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":0.998,"5min":0.998,"15min":0.998},"last":0.70899999999999996}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55600000000000005}]}]},{"osd":7,"up_from":51,"seq":219043332128,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6776,"kb_used_data":1392,"kb_used_omap":0,"kb_used_meta":5376,"kb_avail":93738376,"statfs":{"total":95995035648,"available":95988097024,"internally_reserved":0,"allocated":1425408,"data_stored":954296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5505024},"hb_peers":[0,1,2,3,4,5,6,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri May 13 23:13:48 2022","interfaces":[{"interface":"back","average":{"1min":0.64400000000000002,"5min":0.56000000000000005,"15min":0.54600000000000004},"min":{"1min":0.48399999999999999,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":0.82599999999999996,"5min":0.82599999999999996,"15min":0.82599999999999996},"last":0.63600000000000001},{"interface":"front","average":{"1min":0.60899999999999999,"5min":0.58799999999999997,"15min":0.58399999999999996},"min":{"1min":0.42699999999999999,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":0.78800000000000003,"5min":0.86599999999999999,"15min":0.86599999999999999},"last":0.51900000000000002}]},{"osd":1,"last update":"Fri May 13 23:13:48 2022","interfaces":[{"interface":"back","average":{"1min":0.47499999999999998,"5min":0.44700000000000001,"15min":0.442},"min":{"1min":0.29999999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.79800000000000004,"5min":0.79800000000000004,"15min":0.79800000000000004},"last":0.42299999999999999},{"interface":"front","average":{"1min":0.49299999999999999,"5min":0.47299999999999998,"15min":0.46999999999999997},"min":{"1min":0.317,"5min":0.317,"15min":0.317},"max":{"1min":0.88800000000000001,"5min":0.88800000000000001,"15min":0.88800000000000001},"last":0.46500000000000002}]},{"osd":2,"last update":"Fri May 13 23:13:48 2022","interfaces":[{"interface":"back","average":{"1min":0.63100000000000001,"5min":0.63500000000000001,"15min":0.63600000000000001},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.83799999999999997,"5min":0.88800000000000001,"15min":0.88800000000000001},"last":0.55400000000000005},{"interface":"front","average":{"1min":0.60699999999999998,"5min":0.60299999999999998,"15min":0.60299999999999998},"min":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":0.78600000000000003,"5min":0.81599999999999995,"15min":0.81599999999999995},"last":0.50900000000000001}]},{"osd":3,"last update":"Fri May 13 23:13:48 2022","interfaces":[{"interface":"back","average":{"1min":0.66300000000000003,"5min":0.63500000000000001,"15min":0.63},"min":{"1min":0.50700000000000001,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":0.92200000000000004,"5min":0.92200000000000004,"15min":0.92200000000000004},"last":0.54200000000000004},{"interface":"front","average":{"1min":0.69399999999999995,"5min":0.69699999999999995,"15min":0.69699999999999995},"min":{"1min":0.49199999999999999,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":0.872,"5min":0.872,"15min":0.872},"last":0.56899999999999995}]},{"osd":4,"last update":"Fri May 13 23:13:48 2022","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.58399999999999996,"15min":0.59199999999999997},"min":{"1min":0.315,"5min":0.315,"15min":0.315},"max":{"1min":0.77800000000000002,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.40100000000000002},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.58199999999999996,"15min":0.58799999999999997},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.875,"5min":0.875,"15min":0.875},"last":0.438}]},{"osd":5,"last update":"Fri May 13 23:13:48 2022","interfaces":[{"interface":"back","average":{"1min":0.68100000000000005,"5min":0.72099999999999997,"15min":0.72699999999999998},"min":{"1min":0.47199999999999998,"5min":0.47199999999999998,"15min":0.47199999999999998},"max":{"1min":0.93300000000000005,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.59899999999999998},{"interface":"front","average":{"1min":0.68100000000000005,"5min":0.70099999999999996,"15min":0.70399999999999996},"min":{"1min":0.47999999999999998,"5min":0.47999999999999998,"15min":0.47999999999999998},"max":{"1min":0.93999999999999995,"5min":0.93999999999999995,"15min":0.93999999999999995},"last":0.58899999999999997}]},{"osd":6,"last update":"Fri May 13 23:13:48 2022","interfaces":[{"interface":"back","average":{"1min":0.76300000000000001,"5min":0.76200000000000001,"15min":0.76200000000000001},"min":{"1min":0.52700000000000002,"5min":0.52700000000000002,"15min":0.52700000000000002},"max":{"1min":1.0700000000000001,"5min":1.0700000000000001,"15min":1.0700000000000001},"last":0.61499999999999999},{"interface":"front","average":{"1min":0.81999999999999995,"5min":0.79700000000000004,"15min":0.79300000000000004},"min":{"1min":0.67600000000000005,"5min":0.63200000000000001,"15min":0.63200000000000001},"max":{"1min":1.04,"5min":1.1319999999999999,"15min":1.1319999999999999},"last":0.68999999999999995}]},{"osd":8,"last update":"Fri May 13 23:14:07 2022","interfaces":[{"interface":"back","average":{"1min":0.90600000000000003,"5min":0.77200000000000002,"15min":0.75},"min":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"max":{"1min":4.0739999999999998,"5min":4.0739999999999998,"15min":4.0739999999999998},"last":0.67100000000000004},{"interface":"front","average":{"1min":0.78100000000000003,"5min":0.76500000000000001,"15min":0.76200000000000001},"min":{"1min":0.497,"5min":0.497,"15min":0.497},"max":{"1min":1.046,"5min":1.046,"15min":1.046},"last":0.64700000000000002}]},{"osd":9,"last update":"Fri May 13 23:13:37 2022","interfaces":[{"interface":"back","average":{"1min":0.82699999999999996,"5min":0.82699999999999996,"15min":0.82699999999999996},"min":{"1min":0.57099999999999995,"5min":0.57099999999999995,"15min":0.57099999999999995},"max":{"1min":1.1339999999999999,"5min":1.1339999999999999,"15min":1.1339999999999999},"last":0.77000000000000002},{"interface":"front","average":{"1min":0.82899999999999996,"5min":0.82899999999999996,"15min":0.82899999999999996},"min":{"1min":0.64900000000000002,"5min":0.64900000000000002,"15min":0.64900000000000002},"max":{"1min":1.0409999999999999,"5min":1.0409999999999999,"15min":1.0409999999999999},"last":0.66400000000000003}]},{"osd":10,"last update":"Fri May 13 23:14:00 2022","interfaces":[{"interface":"back","average":{"1min":0.77400000000000002,"5min":0.77400000000000002,"15min":0.77400000000000002},"min":{"1min":0.47599999999999998,"5min":0.47599999999999998,"15min":0.47599999999999998},"max":{"1min":1.093,"5min":1.093,"15min":1.093},"last":0.70899999999999996},{"interface":"front","average":{"1min":0.72899999999999998,"5min":0.72899999999999998,"15min":0.72899999999999998},"min":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"max":{"1min":1.0529999999999999,"5min":1.0529999999999999,"15min":1.0529999999999999},"last":0.627}]}]},{"osd":6,"up_from":46,"seq":197568495654,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6124,"kb_used_data":868,"kb_used_omap":0,"kb_used_meta":5248,"kb_avail":93739028,"statfs":{"total":95995035648,"available":95988764672,"internally_reserved":0,"allocated":888832,"data_stored":424814,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5373952},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri May 13 23:13:27 2022","interfaces":[{"interface":"back","average":{"1min":0.58699999999999997,"5min":0.55100000000000005,"15min":0.54500000000000004},"min":{"1min":0.378,"5min":0.378,"15min":0.378},"max":{"1min":1.004,"5min":1.004,"15min":1.004},"last":0.438},{"interface":"front","average":{"1min":0.62,"5min":0.60099999999999998,"15min":0.59799999999999998},"min":{"1min":0.38800000000000001,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":1.042,"5min":1.042,"15min":1.042},"last":0.70499999999999996}]},{"osd":1,"last update":"Fri May 13 23:13:27 2022","interfaces":[{"interface":"back","average":{"1min":0.80500000000000005,"5min":0.75700000000000001,"15min":0.749},"min":{"1min":0.45700000000000002,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":1.2210000000000001,"5min":1.2210000000000001,"15min":1.2210000000000001},"last":0.61699999999999999},{"interface":"front","average":{"1min":0.72699999999999998,"5min":0.67700000000000005,"15min":0.66900000000000004},"min":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":1.28,"5min":1.28,"15min":1.28},"last":0.63}]},{"osd":2,"last update":"Fri May 13 23:13:27 2022","interfaces":[{"interface":"back","average":{"1min":0.80400000000000005,"5min":0.78800000000000003,"15min":0.78500000000000003},"min":{"1min":0.54700000000000004,"5min":0.54700000000000004,"15min":0.54700000000000004},"max":{"1min":1.1200000000000001,"5min":1.228,"15min":1.228},"last":0.79800000000000004},{"interface":"front","average":{"1min":0.83299999999999996,"5min":0.81599999999999995,"15min":0.81299999999999994},"min":{"1min":0.56599999999999995,"5min":0.53300000000000003,"15min":0.53300000000000003},"max":{"1min":1.298,"5min":1.298,"15min":1.298},"last":0.65400000000000003}]},{"osd":3,"last update":"Fri May 13 23:13:27 2022","interfaces":[{"interface":"back","average":{"1min":0.85699999999999998,"5min":0.81399999999999995,"15min":0.80700000000000005},"min":{"1min":0.60199999999999998,"5min":0.47599999999999998,"15min":0.47599999999999998},"max":{"1min":1.3839999999999999,"5min":1.3839999999999999,"15min":1.3839999999999999},"last":0.68799999999999994},{"interface":"front","average":{"1min":0.76000000000000001,"5min":0.77800000000000002,"15min":0.78100000000000003},"min":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":1.0580000000000001,"5min":1.4119999999999999,"15min":1.4119999999999999},"last":0.68100000000000005}]},{"osd":4,"last update":"Fri May 13 23:13:27 2022","interfaces":[{"interface":"back","average":{"1min":0.82299999999999995,"5min":0.83899999999999997,"15min":0.84199999999999997},"min":{"1min":0.504,"5min":0.504,"15min":0.504},"max":{"1min":1.5640000000000001,"5min":1.5640000000000001,"15min":1.5640000000000001},"last":0.88},{"interface":"front","average":{"1min":0.93200000000000005,"5min":0.90000000000000002,"15min":0.89400000000000002},"min":{"1min":0.60799999999999998,"5min":0.60799999999999998,"15min":0.60799999999999998},"max":{"1min":1.625,"5min":1.625,"15min":1.625},"last":0.78000000000000003}]},{"osd":5,"last update":"Fri May 13 23:13:27 2022","interfaces":[{"interface":"back","average":{"1min":0.84599999999999997,"5min":0.85699999999999998,"15min":0.85899999999999999},"min":{"1min":0.56799999999999995,"5min":0.50600000000000001,"15min":0.50600000000000001},"max":{"1min":1.3640000000000001,"5min":1.5029999999999999,"15min":1.5029999999999999},"last":0.76400000000000001},{"interface":"front","average":{"1min":0.93000000000000005,"5min":0.93700000000000006,"15min":0.93799999999999994},"min":{"1min":0.65900000000000003,"5min":0.65400000000000003,"15min":0.65400000000000003},"max":{"1min":1.3460000000000001,"5min":1.3460000000000001,"15min":1.3460000000000001},"last":0.92800000000000005}]},{"osd":7,"last update":"Fri May 13 23:13:47 2022","interfaces":[{"interface":"back","average":{"1min":1.0669999999999999,"5min":0.89700000000000002,"15min":0.86899999999999999},"min":{"1min":0.78500000000000003,"5min":0.66000000000000003,"15min":0.66000000000000003},"max":{"1min":1.534,"5min":1.534,"15min":1.534},"last":0.83299999999999996},{"interface":"front","average":{"1min":0.97599999999999998,"5min":0.84699999999999998,"15min":0.82499999999999996},"min":{"1min":0.66300000000000003,"5min":0.61299999999999999,"15min":0.61299999999999999},"max":{"1min":1.6180000000000001,"5min":1.6180000000000001,"15min":1.6180000000000001},"last":0.81699999999999995}]},{"osd":8,"last update":"Fri May 13 23:14:12 2022","interfaces":[{"interface":"back","average":{"1min":1.0049999999999999,"5min":0.98499999999999999,"15min":0.98099999999999998},"min":{"1min":0.69999999999999996,"5min":0.69999999999999996,"15min":0.69999999999999996},"max":{"1min":1.484,"5min":1.484,"15min":1.484},"last":0.81299999999999994},{"interface":"front","average":{"1min":0.89600000000000002,"5min":0.94499999999999995,"15min":0.95299999999999996},"min":{"1min":0.68500000000000005,"5min":0.67500000000000004,"15min":0.67500000000000004},"max":{"1min":1.548,"5min":1.548,"15min":1.548},"last":0.86499999999999999}]},{"osd":9,"last update":"Fri May 13 23:13:37 2022","interfaces":[{"interface":"back","average":{"1min":1.071,"5min":1.071,"15min":1.071},"min":{"1min":0.86299999999999999,"5min":0.86299999999999999,"15min":0.86299999999999999},"max":{"1min":1.458,"5min":1.458,"15min":1.458},"last":0.89900000000000002},{"interface":"front","average":{"1min":1.0580000000000001,"5min":1.0580000000000001,"15min":1.0580000000000001},"min":{"1min":0.78900000000000003,"5min":0.78900000000000003,"15min":0.78900000000000003},"max":{"1min":1.401,"5min":1.401,"15min":1.401},"last":0.96499999999999997}]},{"osd":10,"last update":"Fri May 13 23:14:01 2022","interfaces":[{"interface":"back","average":{"1min":1.0529999999999999,"5min":1.0529999999999999,"15min":1.0529999999999999},"min":{"1min":0.77800000000000002,"5min":0.77800000000000002,"15min":0.77800000000000002},"max":{"1min":1.645,"5min":1.645,"15min":1.645},"last":0.91900000000000004},{"interface":"front","average":{"1min":1.006,"5min":1.006,"15min":1.006},"min":{"1min":0.73999999999999999,"5min":0.73999999999999999,"15min":0.73999999999999999},"max":{"1min":1.476,"5min":1.476,"15min":1.476},"last":1.006}]}]},{"osd":1,"up_from":13,"seq":55834574913,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":7032,"kb_used_data":880,"kb_used_omap":0,"kb_used_meta":6144,"kb_avail":93738120,"statfs":{"total":95995035648,"available":95987834880,"internally_reserved":0,"allocated":901120,"data_stored":426350,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":6291456},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri May 13 23:14:14 2022","interfaces":[{"interface":"back","average":{"1min":0.60399999999999998,"5min":0.55900000000000005,"15min":0.51300000000000001},"min":{"1min":0.46700000000000003,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.755,"5min":0.91700000000000004,"15min":0.91700000000000004},"last":0.64300000000000002},{"interface":"front","average":{"1min":0.623,"5min":0.57899999999999996,"15min":0.54000000000000004},"min":{"1min":0.48399999999999999,"5min":0.309,"15min":0.309},"max":{"1min":0.82299999999999995,"5min":0.86799999999999999,"15min":0.86799999999999999},"last":0.72799999999999998}]},{"osd":2,"last update":"Fri May 13 23:13:40 2022","interfaces":[{"interface":"back","average":{"1min":0.56000000000000005,"5min":0.52300000000000002,"15min":0.51300000000000001},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.76900000000000002,"5min":0.76900000000000002,"15min":0.76900000000000002},"last":0.67400000000000004},{"interface":"front","average":{"1min":0.59899999999999998,"5min":0.54300000000000004,"15min":0.52500000000000002},"min":{"1min":0.42299999999999999,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.84599999999999997,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.61899999999999999}]},{"osd":3,"last update":"Fri May 13 23:14:12 2022","interfaces":[{"interface":"back","average":{"1min":0.64100000000000001,"5min":0.59699999999999998,"15min":0.57099999999999995},"min":{"1min":0.44600000000000001,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.84199999999999997,"5min":0.871,"15min":0.871},"last":0.95999999999999996},{"interface":"front","average":{"1min":0.60899999999999999,"5min":0.57199999999999995,"15min":0.56200000000000006},"min":{"1min":0.45400000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.77900000000000003,"5min":0.95399999999999996,"15min":0.95399999999999996},"last":0.873}]},{"osd":4,"last update":"Fri May 13 23:13:35 2022","interfaces":[{"interface":"back","average":{"1min":0.51000000000000001,"5min":0.435,"15min":0.41699999999999998},"min":{"1min":0.32100000000000001,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.77100000000000002,"5min":0.77100000000000002,"15min":0.77100000000000002},"last":0.79600000000000004},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.46899999999999997,"15min":0.44500000000000001},"min":{"1min":0.27400000000000002,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.876,"5min":0.876,"15min":0.876},"last":0.78200000000000003}]},{"osd":5,"last update":"Fri May 13 23:13:52 2022","interfaces":[{"interface":"back","average":{"1min":0.72499999999999998,"5min":0.60099999999999998,"15min":0.58499999999999996},"min":{"1min":0.497,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.88200000000000001,"5min":0.88200000000000001,"15min":0.88200000000000001},"last":0.89500000000000002},{"interface":"front","average":{"1min":0.748,"5min":0.628,"15min":0.60499999999999998},"min":{"1min":0.45500000000000002,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":0.98799999999999999,"5min":0.98799999999999999,"15min":0.98799999999999999},"last":0.81999999999999995}]},{"osd":6,"last update":"Fri May 13 23:13:23 2022","interfaces":[{"interface":"back","average":{"1min":0.76800000000000002,"5min":0.68799999999999994,"15min":0.67400000000000004},"min":{"1min":0.45000000000000001,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":1.5529999999999999,"5min":1.5529999999999999,"15min":1.5529999999999999},"last":1.0089999999999999},{"interface":"front","average":{"1min":0.75,"5min":0.71599999999999997,"15min":0.70999999999999996},"min":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":1.1419999999999999,"5min":1.1419999999999999,"15min":1.1419999999999999},"last":0.85799999999999998}]},{"osd":7,"last update":"Fri May 13 23:13:46 2022","interfaces":[{"interface":"back","average":{"1min":0.70799999999999996,"5min":0.58399999999999996,"15min":0.56299999999999994},"min":{"1min":0.38600000000000001,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.95999999999999996,"5min":0.95999999999999996,"15min":0.95999999999999996},"last":0.91400000000000003},{"interface":"front","average":{"1min":0.79500000000000004,"5min":0.65300000000000002,"15min":0.629},"min":{"1min":0.41399999999999998,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":1.042,"5min":1.042,"15min":1.042},"last":1.079}]},{"osd":8,"last update":"Fri May 13 23:14:12 2022","interfaces":[{"interface":"back","average":{"1min":0.74099999999999999,"5min":0.66900000000000004,"15min":0.65700000000000003},"min":{"1min":0.5,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":1.02,"5min":1.02,"15min":1.02},"last":1.0960000000000001},{"interface":"front","average":{"1min":0.84899999999999998,"5min":0.73299999999999998,"15min":0.71399999999999997},"min":{"1min":0.58999999999999997,"5min":0.501,"15min":0.501},"max":{"1min":1.111,"5min":1.111,"15min":1.111},"last":1.214}]},{"osd":9,"last update":"Fri May 13 23:13:39 2022","interfaces":[{"interface":"back","average":{"1min":0.88500000000000001,"5min":0.88500000000000001,"15min":0.88500000000000001},"min":{"1min":0.60999999999999999,"5min":0.60999999999999999,"15min":0.60999999999999999},"max":{"1min":1.091,"5min":1.091,"15min":1.091},"last":1.165},{"interface":"front","average":{"1min":0.84599999999999997,"5min":0.84599999999999997,"15min":0.84599999999999997},"min":{"1min":0.63600000000000001,"5min":0.63600000000000001,"15min":0.63600000000000001},"max":{"1min":1.0780000000000001,"5min":1.0780000000000001,"15min":1.0780000000000001},"last":0.92100000000000004}]},{"osd":10,"last update":"Fri May 13 23:14:01 2022","interfaces":[{"interface":"back","average":{"1min":0.78800000000000003,"5min":0.78800000000000003,"15min":0.78800000000000003},"min":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"max":{"1min":0.95399999999999996,"5min":0.95399999999999996,"15min":0.95399999999999996},"last":0.97399999999999998},{"interface":"front","average":{"1min":0.90800000000000003,"5min":0.90800000000000003,"15min":0.90800000000000003},"min":{"1min":0.623,"5min":0.623,"15min":0.623},"max":{"1min":1.105,"5min":1.105,"15min":1.105},"last":1.123}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.99299999999999999}]}]},{"osd":0,"up_from":8,"seq":34359738436,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6932,"kb_used_data":844,"kb_used_omap":0,"kb_used_meta":6080,"kb_avail":93738220,"statfs":{"total":95995035648,"available":95987937280,"internally_reserved":0,"allocated":864256,"data_stored":407191,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":6225920},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri May 13 23:14:11 2022","interfaces":[{"interface":"back","average":{"1min":0.68300000000000005,"5min":0.627,"15min":0.60999999999999999},"min":{"1min":0.40400000000000003,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.98899999999999999,"5min":1.2729999999999999,"15min":1.2729999999999999},"last":0.88600000000000001},{"interface":"front","average":{"1min":0.752,"5min":0.66200000000000003,"15min":0.627},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":0.98999999999999999,"5min":1.167,"15min":1.167},"last":0.76600000000000001}]},{"osd":2,"last update":"Fri May 13 23:13:35 2022","interfaces":[{"interface":"back","average":{"1min":0.72499999999999998,"5min":0.65400000000000003,"15min":0.64600000000000002},"min":{"1min":0.54300000000000004,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":0.96499999999999997,"5min":0.98799999999999999,"15min":0.98799999999999999},"last":0.624},{"interface":"front","average":{"1min":0.70099999999999996,"5min":0.63600000000000001,"15min":0.628},"min":{"1min":0.47799999999999998,"5min":0.45900000000000002,"15min":0.45900000000000002},"max":{"1min":1.024,"5min":1.123,"15min":1.123},"last":0.63600000000000001}]},{"osd":3,"last update":"Fri May 13 23:13:08 2022","interfaces":[{"interface":"back","average":{"1min":0.82399999999999995,"5min":0.63300000000000001,"15min":0.59999999999999998},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":1.163,"5min":1.163,"15min":1.163},"last":0.70599999999999996},{"interface":"front","average":{"1min":0.79800000000000004,"5min":0.67000000000000004,"15min":0.64600000000000002},"min":{"1min":0.46200000000000002,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":1.157,"5min":1.157,"15min":1.157},"last":0.69399999999999995}]},{"osd":4,"last update":"Fri May 13 23:13:33 2022","interfaces":[{"interface":"back","average":{"1min":0.82999999999999996,"5min":0.68200000000000005,"15min":0.65500000000000003},"min":{"1min":0.495,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":1.1990000000000001,"5min":1.1990000000000001,"15min":1.1990000000000001},"last":0.73099999999999998},{"interface":"front","average":{"1min":0.77900000000000003,"5min":0.66300000000000003,"15min":0.63300000000000001},"min":{"1min":0.50600000000000001,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":0.98999999999999999,"5min":0.98999999999999999,"15min":0.98999999999999999},"last":0.68400000000000005}]},{"osd":5,"last update":"Fri May 13 23:13:57 2022","interfaces":[{"interface":"back","average":{"1min":0.78900000000000003,"5min":0.72499999999999998,"15min":0.71099999999999997},"min":{"1min":0.46999999999999997,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":1.1220000000000001,"5min":1.1220000000000001,"15min":1.1220000000000001},"last":0.82899999999999996},{"interface":"front","average":{"1min":0.71199999999999997,"5min":0.69699999999999995,"15min":0.69599999999999995},"min":{"1min":0.49399999999999999,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":1.0409999999999999,"5min":1.056,"15min":1.056},"last":0.67500000000000004}]},{"osd":6,"last update":"Fri May 13 23:13:25 2022","interfaces":[{"interface":"back","average":{"1min":0.93600000000000005,"5min":0.77600000000000002,"15min":0.75},"min":{"1min":0.57799999999999996,"5min":0.44800000000000001,"15min":0.44800000000000001},"max":{"1min":1.288,"5min":1.288,"15min":1.288},"last":0.5},{"interface":"front","average":{"1min":0.874,"5min":0.78400000000000003,"15min":0.76900000000000002},"min":{"1min":0.51000000000000001,"5min":0.5,"15min":0.5},"max":{"1min":1.202,"5min":1.202,"15min":1.202},"last":0.90100000000000002}]},{"osd":7,"last update":"Fri May 13 23:13:48 2022","interfaces":[{"interface":"back","average":{"1min":0.77300000000000002,"5min":0.75,"15min":0.746},"min":{"1min":0.497,"5min":0.48699999999999999,"15min":0.48699999999999999},"max":{"1min":1.2050000000000001,"5min":1.2050000000000001,"15min":1.2050000000000001},"last":0.94199999999999995},{"interface":"front","average":{"1min":0.94899999999999995,"5min":0.86099999999999999,"15min":0.84699999999999998},"min":{"1min":0.69599999999999995,"5min":0.60399999999999998,"15min":0.60399999999999998},"max":{"1min":1.2350000000000001,"5min":1.2350000000000001,"15min":1.2350000000000001},"last":0.877}]},{"osd":8,"last update":"Fri May 13 23:14:09 2022","interfaces":[{"interface":"back","average":{"1min":0.82699999999999996,"5min":0.80500000000000005,"15min":0.80100000000000005},"min":{"1min":0.54700000000000004,"5min":0.52100000000000002,"15min":0.52100000000000002},"max":{"1min":1.1479999999999999,"5min":1.7070000000000001,"15min":1.7070000000000001},"last":0.71699999999999997},{"interface":"front","average":{"1min":0.91200000000000003,"5min":0.91600000000000004,"15min":0.91600000000000004},"min":{"1min":0.60599999999999998,"5min":0.60599999999999998,"15min":0.60599999999999998},"max":{"1min":1.1160000000000001,"5min":1.266,"15min":1.266},"last":0.69099999999999995}]},{"osd":9,"last update":"Fri May 13 23:13:39 2022","interfaces":[{"interface":"back","average":{"1min":1.0589999999999999,"5min":1.0589999999999999,"15min":1.0589999999999999},"min":{"1min":0.85199999999999998,"5min":0.85199999999999998,"15min":0.85199999999999998},"max":{"1min":1.397,"5min":1.397,"15min":1.397},"last":0.91100000000000003},{"interface":"front","average":{"1min":0.94099999999999995,"5min":0.94099999999999995,"15min":0.94099999999999995},"min":{"1min":0.70999999999999996,"5min":0.70999999999999996,"15min":0.70999999999999996},"max":{"1min":1.232,"5min":1.232,"15min":1.232},"last":1.0069999999999999}]},{"osd":10,"last update":"Fri May 13 23:14:02 2022","interfaces":[{"interface":"back","average":{"1min":0.97199999999999998,"5min":0.97199999999999998,"15min":0.97199999999999998},"min":{"1min":0.67700000000000005,"5min":0.67700000000000005,"15min":0.67700000000000005},"max":{"1min":1.7989999999999999,"5min":1.7989999999999999,"15min":1.7989999999999999},"last":0.997},{"interface":"front","average":{"1min":1.0629999999999999,"5min":1.0629999999999999,"15min":1.0629999999999999},"min":{"1min":0.745,"5min":0.745,"15min":0.745},"max":{"1min":1.3360000000000001,"5min":1.3360000000000001,"15min":1.3360000000000001},"last":1.018}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.96299999999999997}]}]},{"osd":2,"up_from":18,"seq":77309411387,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":7108,"kb_used_data":892,"kb_used_omap":0,"kb_used_meta":6208,"kb_avail":93738044,"statfs":{"total":95995035648,"available":95987757056,"internally_reserved":0,"allocated":913408,"data_stored":426958,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":6356992},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri May 13 23:13:37 2022","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.51700000000000002,"15min":0.52300000000000002},"min":{"1min":0.41699999999999998,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.80700000000000005,"5min":0.80700000000000005,"15min":0.80700000000000005},"last":0.439},{"interface":"front","average":{"1min":0.59499999999999997,"5min":0.52100000000000002,"15min":0.51600000000000001},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.78800000000000003,"5min":0.80700000000000005,"15min":0.80700000000000005},"last":0.64800000000000002}]},{"osd":1,"last update":"Fri May 13 23:13:37 2022","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.52500000000000002,"15min":0.52000000000000002},"min":{"1min":0.35699999999999998,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.83699999999999997,"5min":0.83699999999999997,"15min":0.83699999999999997},"last":0.69099999999999995},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.53200000000000003,"15min":0.55000000000000004},"min":{"1min":0.40100000000000002,"5min":0.313,"15min":0.313},"max":{"1min":0.77900000000000003,"5min":0.77900000000000003,"15min":0.77900000000000003},"last":0.45400000000000001}]},{"osd":3,"last update":"Fri May 13 23:14:12 2022","interfaces":[{"interface":"back","average":{"1min":0.64500000000000002,"5min":0.61599999999999999,"15min":0.61199999999999999},"min":{"1min":0.39900000000000002,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.90500000000000003,"5min":0.90500000000000003,"15min":0.90500000000000003},"last":0.51500000000000001},{"interface":"front","average":{"1min":0.66800000000000004,"5min":0.60499999999999998,"15min":0.59699999999999998},"min":{"1min":0.45000000000000001,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.89500000000000002,"5min":0.89500000000000002,"15min":0.89500000000000002},"last":0.497}]},{"osd":4,"last update":"Fri May 13 23:13:31 2022","interfaces":[{"interface":"back","average":{"1min":0.55100000000000005,"5min":0.53400000000000003,"15min":0.52600000000000002},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":0.81999999999999995,"5min":0.81999999999999995,"15min":0.81999999999999995},"last":0.59299999999999997},{"interface":"front","average":{"1min":0.53200000000000003,"5min":0.51500000000000001,"15min":0.505},"min":{"1min":0.36299999999999999,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.70099999999999996,"5min":0.752,"15min":0.752},"last":0.61299999999999999}]},{"osd":5,"last update":"Fri May 13 23:14:00 2022","interfaces":[{"interface":"back","average":{"1min":0.435,"5min":0.40300000000000002,"15min":0.38300000000000001},"min":{"1min":0.32300000000000001,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.71099999999999997,"5min":0.71999999999999997,"15min":0.71999999999999997},"last":0.39600000000000002},{"interface":"front","average":{"1min":0.47099999999999997,"5min":0.41699999999999998,"15min":0.39700000000000002},"min":{"1min":0.35499999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.748,"5min":0.748,"15min":0.748},"last":0.746}]},{"osd":6,"last update":"Fri May 13 23:13:28 2022","interfaces":[{"interface":"back","average":{"1min":0.60599999999999998,"5min":0.60599999999999998,"15min":0.60599999999999998},"min":{"1min":0.504,"5min":0.44800000000000001,"15min":0.44800000000000001},"max":{"1min":0.78500000000000003,"5min":0.78900000000000003,"15min":0.78900000000000003},"last":0.621},{"interface":"front","average":{"1min":0.63100000000000001,"5min":0.58999999999999997,"15min":0.58299999999999996},"min":{"1min":0.46500000000000002,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":0.78200000000000003,"5min":0.78200000000000003,"15min":0.78200000000000003},"last":0.63400000000000001}]},{"osd":7,"last update":"Fri May 13 23:13:45 2022","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":0.61599999999999999,"15min":0.61599999999999999},"min":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":0.80400000000000005,"5min":0.80400000000000005,"15min":0.80400000000000005},"last":0.78400000000000003},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.63800000000000001,"15min":0.64200000000000002},"min":{"1min":0.45900000000000002,"5min":0.45900000000000002,"15min":0.45900000000000002},"max":{"1min":0.83199999999999996,"5min":0.83199999999999996,"15min":0.83199999999999996},"last":0.78800000000000003}]},{"osd":8,"last update":"Fri May 13 23:14:07 2022","interfaces":[{"interface":"back","average":{"1min":0.55400000000000005,"5min":0.53600000000000003,"15min":0.53300000000000003},"min":{"1min":0.36699999999999999,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.876,"5min":0.876,"15min":0.876},"last":0.55800000000000005},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.82799999999999996,"5min":0.82799999999999996,"15min":0.82799999999999996},"last":0.54000000000000004}]},{"osd":9,"last update":"Fri May 13 23:13:37 2022","interfaces":[{"interface":"back","average":{"1min":0.70999999999999996,"5min":0.70999999999999996,"15min":0.70999999999999996},"min":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"max":{"1min":0.82799999999999996,"5min":0.82799999999999996,"15min":0.82799999999999996},"last":0.65500000000000003},{"interface":"front","average":{"1min":0.68999999999999995,"5min":0.68999999999999995,"15min":0.68999999999999995},"min":{"1min":0.55800000000000005,"5min":0.55800000000000005,"15min":0.55800000000000005},"max":{"1min":0.86799999999999999,"5min":0.86799999999999999,"15min":0.86799999999999999},"last":0.70799999999999996}]},{"osd":10,"last update":"Fri May 13 23:14:00 2022","interfaces":[{"interface":"back","average":{"1min":0.68999999999999995,"5min":0.68999999999999995,"15min":0.68999999999999995},"min":{"1min":0.47599999999999998,"5min":0.47599999999999998,"15min":0.47599999999999998},"max":{"1min":0.97099999999999997,"5min":0.97099999999999997,"15min":0.97099999999999997},"last":0.67200000000000004},{"interface":"front","average":{"1min":0.71099999999999997,"5min":0.71099999999999997,"15min":0.71099999999999997},"min":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.52100000000000002},"max":{"1min":0.89200000000000002,"5min":0.89200000000000002,"15min":0.89200000000000002},"last":0.72399999999999998}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73799999999999999}]}]},{"osd":3,"up_from":25,"seq":107374182453,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6452,"kb_used_data":876,"kb_used_omap":0,"kb_used_meta":5568,"kb_avail":93738700,"statfs":{"total":95995035648,"available":95988428800,"internally_reserved":0,"allocated":897024,"data_stored":424918,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5701632},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri May 13 23:14:15 2022","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.49399999999999999,"15min":0.46400000000000002},"min":{"1min":0.40200000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.81999999999999995,"5min":0.81999999999999995,"15min":0.81999999999999995},"last":0.45400000000000001},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.52400000000000002,"15min":0.502},"min":{"1min":0.373,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.70799999999999996,"5min":0.96299999999999997,"15min":0.96299999999999997},"last":0.47699999999999998}]},{"osd":1,"last update":"Fri May 13 23:14:15 2022","interfaces":[{"interface":"back","average":{"1min":0.749,"5min":0.67700000000000005,"15min":0.63500000000000001},"min":{"1min":0.495,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":1.014,"5min":1.014,"15min":1.014},"last":0.69399999999999995},{"interface":"front","average":{"1min":0.64100000000000001,"5min":0.64300000000000002,"15min":0.63100000000000001},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.93200000000000005,"5min":1.0229999999999999,"15min":1.0229999999999999},"last":0.55900000000000005}]},{"osd":2,"last update":"Fri May 13 23:14:15 2022","interfaces":[{"interface":"back","average":{"1min":0.79300000000000004,"5min":0.72399999999999998,"15min":0.68799999999999994},"min":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":1.0489999999999999,"5min":1.0489999999999999,"15min":1.0489999999999999},"last":0.73499999999999999},{"interface":"front","average":{"1min":0.73899999999999999,"5min":0.69699999999999995,"15min":0.67500000000000004},"min":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":1.1299999999999999,"5min":1.1299999999999999,"15min":1.1299999999999999},"last":0.64500000000000002}]},{"osd":4,"last update":"Fri May 13 23:13:33 2022","interfaces":[{"interface":"back","average":{"1min":0.90300000000000002,"5min":0.71999999999999997,"15min":0.68400000000000005},"min":{"1min":0.44400000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":1.157,"5min":1.157,"15min":1.157},"last":0.79400000000000004},{"interface":"front","average":{"1min":0.89900000000000002,"5min":0.72499999999999998,"15min":0.68799999999999994},"min":{"1min":0.56299999999999994,"5min":0.47699999999999998,"15min":0.47699999999999998},"max":{"1min":1.0660000000000001,"5min":1.0660000000000001,"15min":1.0660000000000001},"last":0.82499999999999996}]},{"osd":5,"last update":"Fri May 13 23:13:50 2022","interfaces":[{"interface":"back","average":{"1min":0.80900000000000005,"5min":0.745,"15min":0.72899999999999998},"min":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"max":{"1min":1.056,"5min":1.056,"15min":1.056},"last":0.60199999999999998},{"interface":"front","average":{"1min":0.94499999999999995,"5min":0.80000000000000004,"15min":0.76600000000000001},"min":{"1min":0.626,"5min":0.53400000000000003,"15min":0.53400000000000003},"max":{"1min":1.161,"5min":1.161,"15min":1.161},"last":0.84799999999999998}]},{"osd":6,"last update":"Fri May 13 23:13:27 2022","interfaces":[{"interface":"back","average":{"1min":0.93999999999999995,"5min":0.83299999999999996,"15min":0.81499999999999995},"min":{"1min":0.57799999999999996,"5min":0.57599999999999996,"15min":0.57599999999999996},"max":{"1min":1.2829999999999999,"5min":1.2829999999999999,"15min":1.2829999999999999},"last":0.88800000000000001},{"interface":"front","average":{"1min":0.93700000000000006,"5min":0.86399999999999999,"15min":0.85199999999999998},"min":{"1min":0.61099999999999999,"5min":0.61099999999999999,"15min":0.61099999999999999},"max":{"1min":1.256,"5min":1.256,"15min":1.256},"last":0.71999999999999997}]},{"osd":7,"last update":"Fri May 13 23:13:47 2022","interfaces":[{"interface":"back","average":{"1min":1.034,"5min":0.90700000000000003,"15min":0.88600000000000001},"min":{"1min":0.70899999999999996,"5min":0.628,"15min":0.628},"max":{"1min":1.3280000000000001,"5min":1.3280000000000001,"15min":1.3280000000000001},"last":0.93500000000000005},{"interface":"front","average":{"1min":1.028,"5min":0.89100000000000001,"15min":0.86799999999999999},"min":{"1min":0.82499999999999996,"5min":0.67800000000000005,"15min":0.67800000000000005},"max":{"1min":1.3480000000000001,"5min":1.3480000000000001,"15min":1.3480000000000001},"last":1.042}]},{"osd":8,"last update":"Fri May 13 23:14:07 2022","interfaces":[{"interface":"back","average":{"1min":0.91700000000000004,"5min":0.94699999999999995,"15min":0.95199999999999996},"min":{"1min":0.56699999999999995,"5min":0.56699999999999995,"15min":0.56699999999999995},"max":{"1min":1.2589999999999999,"5min":1.2669999999999999,"15min":1.2669999999999999},"last":1.048},{"interface":"front","average":{"1min":0.98899999999999999,"5min":1.002,"15min":1.004},"min":{"1min":0.58699999999999997,"5min":0.58699999999999997,"15min":0.58699999999999997},"max":{"1min":1.244,"5min":1.244,"15min":1.244},"last":0.91500000000000004}]},{"osd":9,"last update":"Fri May 13 23:13:39 2022","interfaces":[{"interface":"back","average":{"1min":1.131,"5min":1.131,"15min":1.131},"min":{"1min":0.79500000000000004,"5min":0.79500000000000004,"15min":0.79500000000000004},"max":{"1min":1.4319999999999999,"5min":1.4319999999999999,"15min":1.4319999999999999},"last":0.90500000000000003},{"interface":"front","average":{"1min":1.093,"5min":1.093,"15min":1.093},"min":{"1min":0.79100000000000004,"5min":0.79100000000000004,"15min":0.79100000000000004},"max":{"1min":1.4730000000000001,"5min":1.4730000000000001,"15min":1.4730000000000001},"last":0.94999999999999996}]},{"osd":10,"last update":"Fri May 13 23:14:00 2022","interfaces":[{"interface":"back","average":{"1min":1.0129999999999999,"5min":1.0129999999999999,"15min":1.0129999999999999},"min":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.52300000000000002},"max":{"1min":1.373,"5min":1.373,"15min":1.373},"last":0.68200000000000005},{"interface":"front","average":{"1min":1.127,"5min":1.127,"15min":1.127},"min":{"1min":0.78300000000000003,"5min":0.78300000000000003,"15min":0.78300000000000003},"max":{"1min":1.51,"5min":1.51,"15min":1.51},"last":0.996}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.80500000000000005}]}]},{"osd":4,"up_from":33,"seq":141733920816,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6256,"kb_used_data":872,"kb_used_omap":0,"kb_used_meta":5376,"kb_avail":93738896,"statfs":{"total":95995035648,"available":95988629504,"internally_reserved":0,"allocated":892928,"data_stored":424836,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5505024},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri May 13 23:13:35 2022","interfaces":[{"interface":"back","average":{"1min":0.627,"5min":0.56299999999999994,"15min":0.55000000000000004},"min":{"1min":0.39700000000000002,"5min":0.375,"15min":0.375},"max":{"1min":0.91800000000000004,"5min":0.91800000000000004,"15min":0.91800000000000004},"last":0.56000000000000005},{"interface":"front","average":{"1min":0.70499999999999996,"5min":0.58099999999999996,"15min":0.54500000000000004},"min":{"1min":0.47999999999999998,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.97099999999999997,"5min":0.97099999999999997,"15min":0.97099999999999997},"last":0.65200000000000002}]},{"osd":1,"last update":"Fri May 13 23:13:35 2022","interfaces":[{"interface":"back","average":{"1min":0.498,"5min":0.39200000000000002,"15min":0.372},"min":{"1min":0.309,"5min":0.217,"15min":0.217},"max":{"1min":0.84499999999999997,"5min":0.84499999999999997,"15min":0.84499999999999997},"last":0.44600000000000001},{"interface":"front","average":{"1min":0.46200000000000002,"5min":0.40300000000000002,"15min":0.39000000000000001},"min":{"1min":0.28899999999999998,"5min":0.248,"15min":0.248},"max":{"1min":0.78100000000000003,"5min":0.79900000000000004,"15min":0.79900000000000004},"last":0.496}]},{"osd":2,"last update":"Fri May 13 23:13:35 2022","interfaces":[{"interface":"back","average":{"1min":0.63800000000000001,"5min":0.54800000000000004,"15min":0.54000000000000004},"min":{"1min":0.47899999999999998,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.92600000000000005,"5min":0.92600000000000005,"15min":0.92600000000000005},"last":0.52300000000000002},{"interface":"front","average":{"1min":0.66200000000000003,"5min":0.57799999999999996,"15min":0.56499999999999995},"min":{"1min":0.5,"5min":0.372,"15min":0.372},"max":{"1min":0.92900000000000005,"5min":0.92900000000000005,"15min":0.92900000000000005},"last":0.59499999999999997}]},{"osd":3,"last update":"Fri May 13 23:13:35 2022","interfaces":[{"interface":"back","average":{"1min":0.70399999999999996,"5min":0.60799999999999998,"15min":0.58699999999999997},"min":{"1min":0.498,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.91600000000000004,"5min":0.91600000000000004,"15min":0.91600000000000004},"last":0.749},{"interface":"front","average":{"1min":0.78800000000000003,"5min":0.63700000000000001,"15min":0.61299999999999999},"min":{"1min":0.54400000000000004,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":1.0569999999999999,"5min":1.0569999999999999,"15min":1.0569999999999999},"last":0.63800000000000001}]},{"osd":5,"last update":"Fri May 13 23:13:55 2022","interfaces":[{"interface":"back","average":{"1min":0.71399999999999997,"5min":0.58699999999999997,"15min":0.55300000000000005},"min":{"1min":0.53200000000000003,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.97999999999999998,"5min":0.97999999999999998,"15min":0.97999999999999998},"last":0.67000000000000004},{"interface":"front","average":{"1min":0.70099999999999996,"5min":0.58599999999999997,"15min":0.55600000000000005},"min":{"1min":0.55600000000000005,"5min":0.371,"15min":0.371},"max":{"1min":1.1419999999999999,"5min":1.1419999999999999,"15min":1.1419999999999999},"last":0.69499999999999995}]},{"osd":6,"last update":"Fri May 13 23:13:25 2022","interfaces":[{"interface":"back","average":{"1min":0.79000000000000004,"5min":0.67800000000000005,"15min":0.66000000000000003},"min":{"1min":0.57499999999999996,"5min":0.498,"15min":0.498},"max":{"1min":1.127,"5min":1.127,"15min":1.127},"last":0.68300000000000005},{"interface":"front","average":{"1min":0.83699999999999997,"5min":0.71799999999999997,"15min":0.69799999999999995},"min":{"1min":0.60699999999999998,"5min":0.55900000000000005,"15min":0.55900000000000005},"max":{"1min":1.046,"5min":1.046,"15min":1.046},"last":0.72399999999999998}]},{"osd":7,"last update":"Fri May 13 23:13:50 2022","interfaces":[{"interface":"back","average":{"1min":0.72999999999999998,"5min":0.66500000000000004,"15min":0.65400000000000003},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.98599999999999999,"5min":0.98599999999999999,"15min":0.98599999999999999},"last":0.76600000000000001},{"interface":"front","average":{"1min":0.71699999999999997,"5min":0.64500000000000002,"15min":0.63300000000000001},"min":{"1min":0.41599999999999998,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":1.0089999999999999,"5min":1.0089999999999999,"15min":1.0089999999999999},"last":0.65200000000000002}]},{"osd":8,"last update":"Fri May 13 23:14:13 2022","interfaces":[{"interface":"back","average":{"1min":0.81499999999999995,"5min":0.75900000000000001,"15min":0.75},"min":{"1min":0.57399999999999995,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":1.099,"5min":1.099,"15min":1.099},"last":0.80100000000000005},{"interface":"front","average":{"1min":0.76000000000000001,"5min":0.754,"15min":0.753},"min":{"1min":0.44700000000000001,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":1.115,"5min":1.115,"15min":1.115},"last":0.81599999999999995}]},{"osd":9,"last update":"Fri May 13 23:13:40 2022","interfaces":[{"interface":"back","average":{"1min":0.89000000000000001,"5min":0.89000000000000001,"15min":0.89000000000000001},"min":{"1min":0.65700000000000003,"5min":0.65700000000000003,"15min":0.65700000000000003},"max":{"1min":1.1819999999999999,"5min":1.1819999999999999,"15min":1.1819999999999999},"last":0.79600000000000004},{"interface":"front","average":{"1min":0.86899999999999999,"5min":0.86899999999999999,"15min":0.86899999999999999},"min":{"1min":0.67300000000000004,"5min":0.67300000000000004,"15min":0.67300000000000004},"max":{"1min":1.171,"5min":1.171,"15min":1.171},"last":0.78700000000000003}]},{"osd":10,"last update":"Fri May 13 23:14:00 2022","interfaces":[{"interface":"back","average":{"1min":0.80700000000000005,"5min":0.80700000000000005,"15min":0.80700000000000005},"min":{"1min":0.52600000000000002,"5min":0.52600000000000002,"15min":0.52600000000000002},"max":{"1min":1.103,"5min":1.103,"15min":1.103},"last":0.77500000000000002},{"interface":"front","average":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"min":{"1min":0.48499999999999999,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":1.196,"5min":1.196,"15min":1.196},"last":0.73999999999999999}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.83399999999999996}]}]},{"osd":5,"up_from":38,"seq":163208757291,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6844,"kb_used_data":1396,"kb_used_omap":0,"kb_used_meta":5440,"kb_avail":93738308,"statfs":{"total":95995035648,"available":95988027392,"internally_reserved":0,"allocated":1429504,"data_stored":953601,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5570560},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri May 13 23:13:51 2022","interfaces":[{"interface":"back","average":{"1min":0.61099999999999999,"5min":0.53700000000000003,"15min":0.52300000000000002},"min":{"1min":0.376,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.96299999999999997,"5min":0.96299999999999997,"15min":0.96299999999999997},"last":0.53400000000000003},{"interface":"front","average":{"1min":0.57199999999999995,"5min":0.54200000000000004,"15min":0.52300000000000002},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.91300000000000003,"5min":0.96499999999999997,"15min":0.96499999999999997},"last":0.50600000000000001}]},{"osd":1,"last update":"Fri May 13 23:13:51 2022","interfaces":[{"interface":"back","average":{"1min":0.57699999999999996,"5min":0.504,"15min":0.48899999999999999},"min":{"1min":0.41499999999999998,"5min":0.317,"15min":0.317},"max":{"1min":0.76900000000000002,"5min":0.76900000000000002,"15min":0.76900000000000002},"last":0.58599999999999997},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.502,"15min":0.47999999999999998},"min":{"1min":0.42599999999999999,"5min":0.313,"15min":0.313},"max":{"1min":0.78800000000000003,"5min":0.78800000000000003,"15min":0.78800000000000003},"last":0.55200000000000005}]},{"osd":2,"last update":"Fri May 13 23:13:51 2022","interfaces":[{"interface":"back","average":{"1min":0.42399999999999999,"5min":0.38100000000000001,"15min":0.36299999999999999},"min":{"1min":0.28399999999999997,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":0.65600000000000003,"5min":0.65600000000000003,"15min":0.65600000000000003},"last":0.442},{"interface":"front","average":{"1min":0.45000000000000001,"5min":0.39500000000000002,"15min":0.375},"min":{"1min":0.28899999999999998,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.75700000000000001,"5min":0.75700000000000001,"15min":0.75700000000000001},"last":0.42499999999999999}]},{"osd":3,"last update":"Fri May 13 23:13:51 2022","interfaces":[{"interface":"back","average":{"1min":0.57499999999999996,"5min":0.56499999999999995,"15min":0.55900000000000005},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.97999999999999998,"5min":0.97999999999999998,"15min":0.97999999999999998},"last":0.48899999999999999},{"interface":"front","average":{"1min":0.625,"5min":0.57699999999999996,"15min":0.55300000000000005},"min":{"1min":0.45600000000000002,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":0.89900000000000002,"5min":1,"15min":1},"last":0.61199999999999999}]},{"osd":4,"last update":"Fri May 13 23:13:51 2022","interfaces":[{"interface":"back","average":{"1min":0.63700000000000001,"5min":0.55800000000000005,"15min":0.53700000000000003},"min":{"1min":0.38800000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.86399999999999999,"5min":0.86399999999999999,"15min":0.86399999999999999},"last":0.57599999999999996},{"interface":"front","average":{"1min":0.58299999999999996,"5min":0.54200000000000004,"15min":0.53200000000000003},"min":{"1min":0.42599999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.73399999999999999,"5min":0.89500000000000002,"15min":0.89500000000000002},"last":0.54400000000000004}]},{"osd":6,"last update":"Fri May 13 23:13:25 2022","interfaces":[{"interface":"back","average":{"1min":0.60899999999999999,"5min":0.623,"15min":0.625},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":0.872,"5min":0.95399999999999996,"15min":0.95399999999999996},"last":0.52400000000000002},{"interface":"front","average":{"1min":0.65100000000000002,"5min":0.626,"15min":0.622},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.98399999999999999,"5min":0.98399999999999999,"15min":0.98399999999999999},"last":0.63700000000000001}]},{"osd":7,"last update":"Fri May 13 23:13:46 2022","interfaces":[{"interface":"back","average":{"1min":0.629,"5min":0.64100000000000001,"15min":0.64300000000000002},"min":{"1min":0.45200000000000001,"5min":0.379,"15min":0.379},"max":{"1min":0.871,"5min":1.0489999999999999,"15min":1.0489999999999999},"last":0.621},{"interface":"front","average":{"1min":0.66700000000000004,"5min":0.64500000000000002,"15min":0.64100000000000001},"min":{"1min":0.442,"5min":0.442,"15min":0.442},"max":{"1min":0.93000000000000005,"5min":0.93000000000000005,"15min":0.93000000000000005},"last":0.67500000000000004}]},{"osd":8,"last update":"Fri May 13 23:14:06 2022","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.61599999999999999,"15min":0.622},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.89200000000000002,"5min":0.999,"15min":0.999},"last":0.59799999999999998},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.56599999999999995,"15min":0.56899999999999995},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.82699999999999996,"5min":0.82699999999999996,"15min":0.82699999999999996},"last":0.56899999999999995}]},{"osd":9,"last update":"Fri May 13 23:13:40 2022","interfaces":[{"interface":"back","average":{"1min":0.68600000000000005,"5min":0.68600000000000005,"15min":0.68600000000000005},"min":{"1min":0.51400000000000001,"5min":0.51400000000000001,"15min":0.51400000000000001},"max":{"1min":0.88500000000000001,"5min":0.88500000000000001,"15min":0.88500000000000001},"last":0.64500000000000002},{"interface":"front","average":{"1min":0.71199999999999997,"5min":0.71199999999999997,"15min":0.71199999999999997},"min":{"1min":0.505,"5min":0.505,"15min":0.505},"max":{"1min":0.99299999999999999,"5min":0.99299999999999999,"15min":0.99299999999999999},"last":0.69999999999999996}]},{"osd":10,"last update":"Fri May 13 23:13:59 2022","interfaces":[{"interface":"back","average":{"1min":0.67900000000000005,"5min":0.67900000000000005,"15min":0.67900000000000005},"min":{"1min":0.503,"5min":0.503,"15min":0.503},"max":{"1min":0.90800000000000003,"5min":0.90800000000000003,"15min":0.90800000000000003},"last":0.66800000000000004},{"interface":"front","average":{"1min":0.71399999999999997,"5min":0.71399999999999997,"15min":0.71399999999999997},"min":{"1min":0.48499999999999999,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":0.94499999999999995,"5min":0.94499999999999995,"15min":0.94499999999999995},"last":0.71699999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":2810,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":554,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1730,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2022-05-13T23:14:23.259 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2022-05-13T23:14:23.480 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:23.478+0000 7f545cb9d700 1 -- 172.21.15.2:0/1296847878 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f54580ff310 msgr2=0x7f5458105bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:14:23.480 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:23.478+0000 7f545cb9d700 1 --2- 172.21.15.2:0/1296847878 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f54580ff310 0x7f5458105bd0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f5450009be0 tx=0x7f5450004d10 comp rx=0 tx=0).stop 2022-05-13T23:14:23.481 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:23.478+0000 7f545cb9d700 1 -- 172.21.15.2:0/1296847878 shutdown_connections 2022-05-13T23:14:23.481 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:23.478+0000 7f545cb9d700 1 --2- 172.21.15.2:0/1296847878 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f54580ff310 0x7f5458105bd0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:14:23.481 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:23.478+0000 7f545cb9d700 1 --2- 172.21.15.2:0/1296847878 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f54580fd6e0 0x7f54580fdaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:14:23.481 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:23.478+0000 7f545cb9d700 1 --2- 172.21.15.2:0/1296847878 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f54580fe980 0x7f54580fedd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:14:23.482 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:23.478+0000 7f545cb9d700 1 -- 172.21.15.2:0/1296847878 >> 172.21.15.2:0/1296847878 conn(0x7f54580f8ed0 msgr2=0x7f54580fb300 unknown :-1 s=STATE_NONE l=0).mark_down 2022-05-13T23:14:23.482 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:23.478+0000 7f545cb9d700 1 -- 172.21.15.2:0/1296847878 shutdown_connections 2022-05-13T23:14:23.482 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:23.478+0000 7f545cb9d700 1 -- 172.21.15.2:0/1296847878 wait complete. 2022-05-13T23:14:23.482 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:23.479+0000 7f545cb9d700 1 Processor -- start 2022-05-13T23:14:23.485 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:23.479+0000 7f545cb9d700 1 -- start start 2022-05-13T23:14:23.485 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:23.479+0000 7f545cb9d700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f54580fd6e0 0x7f545810e330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:14:23.486 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:23.479+0000 7f545cb9d700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f54580fe980 0x7f545810e870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:14:23.486 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:23.479+0000 7f545cb9d700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f54580ff310 0x7f5458115900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:14:23.486 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:23.479+0000 7f545cb9d700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5458115e40 con 0x7f54580ff310 2022-05-13T23:14:23.486 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:23.479+0000 7f545cb9d700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5458115fb0 con 0x7f54580fd6e0 2022-05-13T23:14:23.486 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:23.479+0000 7f545cb9d700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5458116170 con 0x7f54580fe980 2022-05-13T23:14:23.487 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:23.480+0000 7f54577fe700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f54580fd6e0 0x7f545810e330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:14:23.487 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:23.480+0000 7f54577fe700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f54580fd6e0 0x7f545810e330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.2:39206/0 (socket says 172.21.15.2:39206) 2022-05-13T23:14:23.487 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:23.480+0000 7f54577fe700 1 -- 172.21.15.2:0/2540311023 learned_addr learned my addr 172.21.15.2:0/2540311023 (peer_addr_for_me v2:172.21.15.2:0/0) 2022-05-13T23:14:23.487 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:23.480+0000 7f5456ffd700 1 --2- 172.21.15.2:0/2540311023 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f54580fe980 0x7f545810e870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:14:23.488 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:23.480+0000 7f5457fff700 1 --2- 172.21.15.2:0/2540311023 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f54580ff310 0x7f5458115900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:14:23.488 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:23.480+0000 7f54577fe700 1 -- 172.21.15.2:0/2540311023 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f54580fe980 msgr2=0x7f545810e870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:14:23.488 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:23.480+0000 7f54577fe700 1 --2- 172.21.15.2:0/2540311023 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f54580fe980 0x7f545810e870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:14:23.488 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:23.480+0000 7f54577fe700 1 -- 172.21.15.2:0/2540311023 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f54580ff310 msgr2=0x7f5458115900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:14:23.489 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:23.480+0000 7f54577fe700 1 --2- 172.21.15.2:0/2540311023 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f54580ff310 0x7f5458115900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:14:23.489 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:23.480+0000 7f54577fe700 1 -- 172.21.15.2:0/2540311023 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54500097c0 con 0x7f54580fd6e0 2022-05-13T23:14:23.489 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:23.480+0000 7f54577fe700 1 --2- 172.21.15.2:0/2540311023 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f54580fd6e0 0x7f545810e330 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f544800e940 tx=0x7f544800b040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-05-13T23:14:23.489 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:23.481+0000 7f543ffff700 1 -- 172.21.15.2:0/2540311023 <== mon.1 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5448007b40 con 0x7f54580fd6e0 2022-05-13T23:14:23.490 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:23.481+0000 7f545cb9d700 1 -- 172.21.15.2:0/2540311023 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f54581163f0 con 0x7f54580fd6e0 2022-05-13T23:14:23.490 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:23.481+0000 7f543ffff700 1 -- 172.21.15.2:0/2540311023 <== mon.1 v2:172.21.15.47:3300/0 2 ==== config(26 keys) v1 ==== 1087+0+0 (secure 0 0 0) 0x7f544800f040 con 0x7f54580fd6e0 2022-05-13T23:14:23.490 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:23.481+0000 7f543ffff700 1 -- 172.21.15.2:0/2540311023 <== mon.1 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5448014890 con 0x7f54580fd6e0 2022-05-13T23:14:23.490 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:23.481+0000 7f545cb9d700 1 -- 172.21.15.2:0/2540311023 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f54581169b0 con 0x7f54580fd6e0 2022-05-13T23:14:23.490 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:23.482+0000 7f545cb9d700 1 -- 172.21.15.2:0/2540311023 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5458108ee0 con 0x7f54580fd6e0 2022-05-13T23:14:23.491 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:23.484+0000 7f543ffff700 1 -- 172.21.15.2:0/2540311023 <== mon.1 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 16) v1 ==== 125136+0+0 (secure 0 0 0) 0x7f544801b030 con 0x7f54580fd6e0 2022-05-13T23:14:23.492 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:23.485+0000 7f543ffff700 1 --2- 172.21.15.2:0/2540311023 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f5440096ce0 0x7f5440099190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:14:23.492 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:23.485+0000 7f543ffff700 1 -- 172.21.15.2:0/2540311023 <== mon.1 v2:172.21.15.47:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 7975+0+0 (secure 0 0 0) 0x7f54480beef0 con 0x7f54580fd6e0 2022-05-13T23:14:23.492 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:23.485+0000 7f5456ffd700 1 --2- 172.21.15.2:0/2540311023 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f5440096ce0 0x7f5440099190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:14:23.493 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:23.486+0000 7f5456ffd700 1 --2- 172.21.15.2:0/2540311023 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f5440096ce0 0x7f5440099190 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f544c004790 tx=0x7f544c00b040 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-05-13T23:14:23.493 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:23.487+0000 7f543ffff700 1 -- 172.21.15.2:0/2540311023 <== mon.1 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+165215 (secure 0 0 0) 0x7f544804b570 con 0x7f54580fd6e0 2022-05-13T23:14:23.757 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:23.756+0000 7f545cb9d700 1 -- 172.21.15.2:0/2540311023 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "status", "format": "json"} v 0) v1 -- 0x7f5458104b20 con 0x7f54580fd6e0 2022-05-13T23:14:23.763 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:23.757+0000 7f543ffff700 1 -- 172.21.15.2:0/2540311023 <== mon.1 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) v1 ==== 72+0+3360 (secure 0 0 0) 0x7f5448019080 con 0x7f54580fd6e0 2022-05-13T23:14:23.763 INFO:teuthology.orchestra.run.smithi002.stdout: 2022-05-13T23:14:23.763 INFO:teuthology.orchestra.run.smithi002.stdout:{"fsid":"16cf5f88-d311-11ec-be40-001a4aab830c","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":14,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":379,"monmap":{"epoch":3,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":83,"num_osds":12,"num_up_osds":12,"osd_up_since":1652483600,"num_in_osds":12,"osd_in_since":1652483582,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":97}],"num_pgs":97,"num_pools":3,"num_objects":51,"data_bytes":594670,"bytes_used":80289792,"bytes_avail":1151860137984,"bytes_total":1151940427776,"write_bytes_sec":3651,"read_op_per_sec":0,"write_op_per_sec":14},"fsmap":{"epoch":26,"id":1,"up":5,"in":5,"max":5,"by_rank":[{"filesystem_id":1,"rank":4,"name":"d","status":"up:standby-replay","gid":24494},{"filesystem_id":1,"rank":3,"name":"b","status":"up:standby-replay","gid":24511},{"filesystem_id":1,"rank":2,"name":"e","status":"up:standby-replay","gid":24523},{"filesystem_id":1,"rank":1,"name":"j","status":"up:standby-replay","gid":24530},{"filesystem_id":1,"rank":0,"name":"h","status":"up:standby-replay","gid":24535},{"filesystem_id":1,"rank":4,"name":"k","status":"up:active","gid":24547},{"filesystem_id":1,"rank":3,"name":"c","status":"up:active","gid":24575},{"filesystem_id":1,"rank":2,"name":"f","status":"up:active","gid":24587},{"filesystem_id":1,"rank":1,"name":"i","status":"up:active","gid":24599},{"filesystem_id":1,"rank":0,"name":"l","status":"up:active","gid":24611}],"up:standby":2},"mgrmap":{"available":true,"num_standbys":2,"modules":["cephadm","dashboard","iostat","nfs","restful"],"services":{"dashboard":"https://172.21.15.2:8443/"}},"servicemap":{"epoch":18,"modified":"2022-05-13T23:14:23.235307+0000","services":{"mds":{"daemons":{"summary":"","a":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"b":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"c":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"d":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"e":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"f":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"g":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"h":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"i":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"j":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"k":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"l":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2022-05-13T23:14:23.765 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:23.759+0000 7f545cb9d700 1 -- 172.21.15.2:0/2540311023 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f5440096ce0 msgr2=0x7f5440099190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:14:23.765 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:23.759+0000 7f545cb9d700 1 --2- 172.21.15.2:0/2540311023 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f5440096ce0 0x7f5440099190 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f544c004790 tx=0x7f544c00b040 comp rx=0 tx=0).stop 2022-05-13T23:14:23.765 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:23.759+0000 7f545cb9d700 1 -- 172.21.15.2:0/2540311023 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f54580fd6e0 msgr2=0x7f545810e330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:14:23.765 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:23.759+0000 7f545cb9d700 1 --2- 172.21.15.2:0/2540311023 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f54580fd6e0 0x7f545810e330 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f544800e940 tx=0x7f544800b040 comp rx=0 tx=0).stop 2022-05-13T23:14:23.766 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:23.759+0000 7f545cb9d700 1 -- 172.21.15.2:0/2540311023 shutdown_connections 2022-05-13T23:14:23.766 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:23.759+0000 7f545cb9d700 1 --2- 172.21.15.2:0/2540311023 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f5440096ce0 0x7f5440099190 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:14:23.766 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:23.759+0000 7f545cb9d700 1 --2- 172.21.15.2:0/2540311023 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f54580ff310 0x7f5458115900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:14:23.766 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:23.759+0000 7f545cb9d700 1 --2- 172.21.15.2:0/2540311023 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f54580fd6e0 0x7f545810e330 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:14:23.766 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:23.759+0000 7f545cb9d700 1 --2- 172.21.15.2:0/2540311023 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f54580fe980 0x7f545810e870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:14:23.767 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:23.759+0000 7f545cb9d700 1 -- 172.21.15.2:0/2540311023 >> 172.21.15.2:0/2540311023 conn(0x7f54580f8ed0 msgr2=0x7f5458101e20 unknown :-1 s=STATE_NONE l=0).mark_down 2022-05-13T23:14:23.767 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:23.760+0000 7f545cb9d700 1 -- 172.21.15.2:0/2540311023 shutdown_connections 2022-05-13T23:14:23.767 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:23.760+0000 7f545cb9d700 1 -- 172.21.15.2:0/2540311023 wait complete. 2022-05-13T23:14:23.787 INFO:tasks.cephfs.filesystem.ceph_manager:no progress seen, keeping timeout for now 2022-05-13T23:14:23.788 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2022-05-13T23:14:24.007 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:24.004+0000 7f2e3739d700 1 -- 172.21.15.2:0/3914838610 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2e300fc6e0 msgr2=0x7f2e300fcb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:14:24.007 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:24.004+0000 7f2e3739d700 1 --2- 172.21.15.2:0/3914838610 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2e300fc6e0 0x7f2e300fcb30 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f2e20004870 tx=0x7f2e20009cb0 comp rx=0 tx=0).stop 2022-05-13T23:14:24.008 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:24.004+0000 7f2e3739d700 1 -- 172.21.15.2:0/3914838610 shutdown_connections 2022-05-13T23:14:24.008 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:24.004+0000 7f2e3739d700 1 --2- 172.21.15.2:0/3914838610 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f2e300fd070 0x7f2e30103930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:14:24.008 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:24.004+0000 7f2e3739d700 1 --2- 172.21.15.2:0/3914838610 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f2e300fb440 0x7f2e300fb850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:14:24.009 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:24.004+0000 7f2e3739d700 1 --2- 172.21.15.2:0/3914838610 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2e300fc6e0 0x7f2e300fcb30 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:14:24.009 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:24.004+0000 7f2e3739d700 1 -- 172.21.15.2:0/3914838610 >> 172.21.15.2:0/3914838610 conn(0x7f2e300f6c30 msgr2=0x7f2e300f9060 unknown :-1 s=STATE_NONE l=0).mark_down 2022-05-13T23:14:24.009 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:24.004+0000 7f2e3739d700 1 -- 172.21.15.2:0/3914838610 shutdown_connections 2022-05-13T23:14:24.009 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:24.005+0000 7f2e3739d700 1 -- 172.21.15.2:0/3914838610 wait complete. 2022-05-13T23:14:24.010 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:24.005+0000 7f2e3739d700 1 Processor -- start 2022-05-13T23:14:24.010 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:24.006+0000 7f2e3739d700 1 -- start start 2022-05-13T23:14:24.015 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:24.006+0000 7f2e3739d700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f2e300fb440 0x7f2e300701d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:14:24.016 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:24.006+0000 7f2e3739d700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f2e300fc6e0 0x7f2e3006e830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:14:24.016 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:24.006+0000 7f2e3739d700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2e300fd070 0x7f2e3006ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:14:24.017 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:24.006+0000 7f2e3739d700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e3006f480 con 0x7f2e300fd070 2022-05-13T23:14:24.017 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:24.006+0000 7f2e3739d700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e3006f5f0 con 0x7f2e300fc6e0 2022-05-13T23:14:24.017 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:24.006+0000 7f2e3739d700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e30116300 con 0x7f2e300fb440 2022-05-13T23:14:24.018 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:24.006+0000 7f2e36b9c700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2e300fd070 0x7f2e3006ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:14:24.018 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:24.006+0000 7f2e3639b700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f2e300fb440 0x7f2e300701d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:14:24.018 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:24.006+0000 7f2e35b9a700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f2e300fc6e0 0x7f2e3006e830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:14:24.019 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:24.006+0000 7f2e3639b700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f2e300fb440 0x7f2e300701d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.2:56758/0 (socket says 172.21.15.2:56758) 2022-05-13T23:14:24.019 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:24.006+0000 7f2e3639b700 1 -- 172.21.15.2:0/2064523724 learned_addr learned my addr 172.21.15.2:0/2064523724 (peer_addr_for_me v2:172.21.15.2:0/0) 2022-05-13T23:14:24.020 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:24.007+0000 7f2e3639b700 1 -- 172.21.15.2:0/2064523724 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f2e300fc6e0 msgr2=0x7f2e3006e830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:14:24.020 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:24.007+0000 7f2e3639b700 1 --2- 172.21.15.2:0/2064523724 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f2e300fc6e0 0x7f2e3006e830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:14:24.020 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:24.007+0000 7f2e3639b700 1 -- 172.21.15.2:0/2064523724 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2e300fd070 msgr2=0x7f2e3006ed70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:14:24.021 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:24.007+0000 7f2e3639b700 1 --2- 172.21.15.2:0/2064523724 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2e300fd070 0x7f2e3006ed70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:14:24.021 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:24.007+0000 7f2e3639b700 1 -- 172.21.15.2:0/2064523724 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e20005040 con 0x7f2e300fb440 2022-05-13T23:14:24.021 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:24.007+0000 7f2e3639b700 1 --2- 172.21.15.2:0/2064523724 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f2e300fb440 0x7f2e300701d0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f2e2c00e8c0 tx=0x7f2e2c00b040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-05-13T23:14:24.022 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:24.007+0000 7f2e26ffd700 1 -- 172.21.15.2:0/2064523724 <== mon.2 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e2c007ae0 con 0x7f2e300fb440 2022-05-13T23:14:24.022 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:24.007+0000 7f2e26ffd700 1 -- 172.21.15.2:0/2064523724 <== mon.2 v2:172.21.15.115:3300/0 2 ==== config(26 keys) v1 ==== 1087+0+0 (secure 0 0 0) 0x7f2e2c00f040 con 0x7f2e300fb440 2022-05-13T23:14:24.022 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:24.008+0000 7f2e3739d700 1 -- 172.21.15.2:0/2064523724 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2e301164a0 con 0x7f2e300fb440 2022-05-13T23:14:24.022 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:24.008+0000 7f2e26ffd700 1 -- 172.21.15.2:0/2064523724 <== mon.2 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e2c0148b0 con 0x7f2e300fb440 2022-05-13T23:14:24.023 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:24.008+0000 7f2e3739d700 1 -- 172.21.15.2:0/2064523724 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2e30116a10 con 0x7f2e300fb440 2022-05-13T23:14:24.023 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:24.009+0000 7f2e3739d700 1 -- 172.21.15.2:0/2064523724 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2e30106ce0 con 0x7f2e300fb440 2022-05-13T23:14:24.023 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:24.011+0000 7f2e26ffd700 1 -- 172.21.15.2:0/2064523724 <== mon.2 v2:172.21.15.115:3300/0 4 ==== mgrmap(e 16) v1 ==== 125136+0+0 (secure 0 0 0) 0x7f2e2c042bc0 con 0x7f2e300fb440 2022-05-13T23:14:24.024 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:24.012+0000 7f2e26ffd700 1 --2- 172.21.15.2:0/2064523724 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f2e1c096cd0 0x7f2e1c099180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:14:24.024 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:24.012+0000 7f2e26ffd700 1 -- 172.21.15.2:0/2064523724 <== mon.2 v2:172.21.15.115:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 7975+0+0 (secure 0 0 0) 0x7f2e2c0bf660 con 0x7f2e300fb440 2022-05-13T23:14:24.024 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:24.012+0000 7f2e35b9a700 1 --2- 172.21.15.2:0/2064523724 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f2e1c096cd0 0x7f2e1c099180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:14:24.025 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:24.013+0000 7f2e35b9a700 1 --2- 172.21.15.2:0/2064523724 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f2e1c096cd0 0x7f2e1c099180 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f2e20004840 tx=0x7f2e20009cb0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-05-13T23:14:24.025 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:24.014+0000 7f2e26ffd700 1 -- 172.21.15.2:0/2064523724 <== mon.2 v2:172.21.15.115:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+165215 (secure 0 0 0) 0x7f2e2c08c9a0 con 0x7f2e300fb440 2022-05-13T23:14:24.201 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:24.200+0000 7f2e3739d700 1 -- 172.21.15.2:0/2064523724 --> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f2e3005c920 con 0x7f2e1c096cd0 2022-05-13T23:14:24.212 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:24.211+0000 7f2e26ffd700 1 -- 172.21.15.2:0/2064523724 <== mgr.14148 v2:172.21.15.2:6800/2788339627 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+336409 (secure 0 0 0) 0x7f2e3005c920 con 0x7f2e1c096cd0 2022-05-13T23:14:24.217 INFO:teuthology.orchestra.run.smithi002.stdout: 2022-05-13T23:14:24.220 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:24.216+0000 7f2e3739d700 1 -- 172.21.15.2:0/2064523724 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f2e1c096cd0 msgr2=0x7f2e1c099180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:14:24.221 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:24.216+0000 7f2e3739d700 1 --2- 172.21.15.2:0/2064523724 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f2e1c096cd0 0x7f2e1c099180 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f2e20004840 tx=0x7f2e20009cb0 comp rx=0 tx=0).stop 2022-05-13T23:14:24.221 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:24.216+0000 7f2e3739d700 1 -- 172.21.15.2:0/2064523724 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f2e300fb440 msgr2=0x7f2e300701d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:14:24.221 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:24.216+0000 7f2e3739d700 1 --2- 172.21.15.2:0/2064523724 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f2e300fb440 0x7f2e300701d0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f2e2c00e8c0 tx=0x7f2e2c00b040 comp rx=0 tx=0).stop 2022-05-13T23:14:24.221 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:24.216+0000 7f2e3739d700 1 -- 172.21.15.2:0/2064523724 shutdown_connections 2022-05-13T23:14:24.222 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:24.216+0000 7f2e3739d700 1 --2- 172.21.15.2:0/2064523724 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f2e1c096cd0 0x7f2e1c099180 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:14:24.222 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:24.216+0000 7f2e3739d700 1 --2- 172.21.15.2:0/2064523724 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2e300fd070 0x7f2e3006ed70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:14:24.222 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:24.216+0000 7f2e3739d700 1 --2- 172.21.15.2:0/2064523724 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f2e300fb440 0x7f2e300701d0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:14:24.222 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:24.216+0000 7f2e3739d700 1 --2- 172.21.15.2:0/2064523724 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f2e300fc6e0 0x7f2e3006e830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:14:24.222 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:24.216+0000 7f2e3739d700 1 -- 172.21.15.2:0/2064523724 >> 172.21.15.2:0/2064523724 conn(0x7f2e300f6c30 msgr2=0x7f2e300f8fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-05-13T23:14:24.223 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:24.216+0000 7f2e3739d700 1 -- 172.21.15.2:0/2064523724 shutdown_connections 2022-05-13T23:14:24.223 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:24.216+0000 7f2e3739d700 1 -- 172.21.15.2:0/2064523724 wait complete. 2022-05-13T23:14:24.223 INFO:teuthology.orchestra.run.smithi002.stderr:dumped all 2022-05-13T23:14:24.244 INFO:teuthology.orchestra.run.smithi002.stdout:{"pg_ready":true,"pg_map":{"version":292,"stamp":"2022-05-13T23:14:23.235158+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":594670,"num_objects":51,"num_object_clones":0,"num_object_copies":153,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":51,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":288,"num_write_kb":2965,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1057856,"num_keys_recovered":0,"num_objects_omap":32,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":230,"ondisk_log_size":230,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":248,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":78408,"kb_used_data":12136,"kb_used_omap":0,"kb_used_meta":66176,"kb_avail":1124863416,"statfs":{"total":1151940427776,"available":1151860137984,"internally_reserved":0,"allocated":12427264,"data_stored":6694216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":67764224},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":17,"apply_latency_ms":17,"commit_latency_ns":17000000,"apply_latency_ns":17000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":4302,"num_objects":49,"num_object_clones":0,"num_object_copies":147,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":49,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":107,"num_write_kb":27,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":32,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.572025"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":10,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-05-13T23:14:16.269783+0000","last_change":"2022-05-13T23:14:16.267128+0000","last_active":"2022-05-13T23:14:16.269783+0000","last_peered":"2022-05-13T23:14:16.269783+0000","last_clean":"2022-05-13T23:14:16.269783+0000","last_became_active":"2022-05-13T23:14:16.266687+0000","last_became_peered":"2022-05-13T23:14:16.266687+0000","last_unstale":"2022-05-13T23:14:16.269783+0000","last_undegraded":"2022-05-13T23:14:16.269783+0000","last_fullsized":"2022-05-13T23:14:16.269783+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_clean_scrub_stamp":"2022-05-13T23:14:15.659362+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T04:40:46.150557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":10,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-05-13T23:14:16.280180+0000","last_change":"2022-05-13T23:14:16.272880+0000","last_active":"2022-05-13T23:14:16.280180+0000","last_peered":"2022-05-13T23:14:16.280180+0000","last_clean":"2022-05-13T23:14:16.280180+0000","last_became_active":"2022-05-13T23:14:16.272473+0000","last_became_peered":"2022-05-13T23:14:16.272473+0000","last_unstale":"2022-05-13T23:14:16.280180+0000","last_undegraded":"2022-05-13T23:14:16.280180+0000","last_fullsized":"2022-05-13T23:14:16.280180+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_clean_scrub_stamp":"2022-05-13T23:14:15.659362+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T00:32:19.185037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":10,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-05-13T23:14:16.277176+0000","last_change":"2022-05-13T23:14:16.270961+0000","last_active":"2022-05-13T23:14:16.277176+0000","last_peered":"2022-05-13T23:14:16.277176+0000","last_clean":"2022-05-13T23:14:16.277176+0000","last_became_active":"2022-05-13T23:14:16.269942+0000","last_became_peered":"2022-05-13T23:14:16.269942+0000","last_unstale":"2022-05-13T23:14:16.277176+0000","last_undegraded":"2022-05-13T23:14:16.277176+0000","last_fullsized":"2022-05-13T23:14:16.277176+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_clean_scrub_stamp":"2022-05-13T23:14:15.659362+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T10:03:03.774806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":10,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-05-13T23:14:16.257175+0000","last_change":"2022-05-13T23:14:16.255885+0000","last_active":"2022-05-13T23:14:16.257175+0000","last_peered":"2022-05-13T23:14:16.257175+0000","last_clean":"2022-05-13T23:14:16.257175+0000","last_became_active":"2022-05-13T23:14:16.255551+0000","last_became_peered":"2022-05-13T23:14:16.255551+0000","last_unstale":"2022-05-13T23:14:16.257175+0000","last_undegraded":"2022-05-13T23:14:16.257175+0000","last_fullsized":"2022-05-13T23:14:16.257175+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_clean_scrub_stamp":"2022-05-13T23:14:15.659362+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T10:26:40.578066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":10,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-05-13T23:14:16.271952+0000","last_change":"2022-05-13T23:14:16.271384+0000","last_active":"2022-05-13T23:14:16.271952+0000","last_peered":"2022-05-13T23:14:16.271952+0000","last_clean":"2022-05-13T23:14:16.271952+0000","last_became_active":"2022-05-13T23:14:16.271045+0000","last_became_peered":"2022-05-13T23:14:16.271045+0000","last_unstale":"2022-05-13T23:14:16.271952+0000","last_undegraded":"2022-05-13T23:14:16.271952+0000","last_fullsized":"2022-05-13T23:14:16.271952+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_clean_scrub_stamp":"2022-05-13T23:14:15.659362+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T04:51:50.618628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":10,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-05-13T23:14:16.276293+0000","last_change":"2022-05-13T23:14:16.275906+0000","last_active":"2022-05-13T23:14:16.276293+0000","last_peered":"2022-05-13T23:14:16.276293+0000","last_clean":"2022-05-13T23:14:16.276293+0000","last_became_active":"2022-05-13T23:14:16.275651+0000","last_became_peered":"2022-05-13T23:14:16.275651+0000","last_unstale":"2022-05-13T23:14:16.276293+0000","last_undegraded":"2022-05-13T23:14:16.276293+0000","last_fullsized":"2022-05-13T23:14:16.276293+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_clean_scrub_stamp":"2022-05-13T23:14:15.659362+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T02:29:33.019268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":10,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-05-13T23:14:16.276560+0000","last_change":"2022-05-13T23:14:16.276150+0000","last_active":"2022-05-13T23:14:16.276560+0000","last_peered":"2022-05-13T23:14:16.276560+0000","last_clean":"2022-05-13T23:14:16.276560+0000","last_became_active":"2022-05-13T23:14:16.275751+0000","last_became_peered":"2022-05-13T23:14:16.275751+0000","last_unstale":"2022-05-13T23:14:16.276560+0000","last_undegraded":"2022-05-13T23:14:16.276560+0000","last_fullsized":"2022-05-13T23:14:16.276560+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_clean_scrub_stamp":"2022-05-13T23:14:15.659362+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T04:01:48.496870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":10,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-05-13T23:14:16.270951+0000","last_change":"2022-05-13T23:14:16.268816+0000","last_active":"2022-05-13T23:14:16.270951+0000","last_peered":"2022-05-13T23:14:16.270951+0000","last_clean":"2022-05-13T23:14:16.270951+0000","last_became_active":"2022-05-13T23:14:16.268429+0000","last_became_peered":"2022-05-13T23:14:16.268429+0000","last_unstale":"2022-05-13T23:14:16.270951+0000","last_undegraded":"2022-05-13T23:14:16.270951+0000","last_fullsized":"2022-05-13T23:14:16.270951+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_clean_scrub_stamp":"2022-05-13T23:14:15.659362+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T00:14:51.685287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":10,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-05-13T23:14:16.265890+0000","last_change":"2022-05-13T23:14:16.263971+0000","last_active":"2022-05-13T23:14:16.265890+0000","last_peered":"2022-05-13T23:14:16.265890+0000","last_clean":"2022-05-13T23:14:16.265890+0000","last_became_active":"2022-05-13T23:14:16.263625+0000","last_became_peered":"2022-05-13T23:14:16.263625+0000","last_unstale":"2022-05-13T23:14:16.265890+0000","last_undegraded":"2022-05-13T23:14:16.265890+0000","last_fullsized":"2022-05-13T23:14:16.265890+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_clean_scrub_stamp":"2022-05-13T23:14:15.659362+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T03:11:11.037746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":10,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-05-13T23:14:16.258455+0000","last_change":"2022-05-13T23:14:16.258111+0000","last_active":"2022-05-13T23:14:16.258455+0000","last_peered":"2022-05-13T23:14:16.258455+0000","last_clean":"2022-05-13T23:14:16.258455+0000","last_became_active":"2022-05-13T23:14:16.257768+0000","last_became_peered":"2022-05-13T23:14:16.257768+0000","last_unstale":"2022-05-13T23:14:16.258455+0000","last_undegraded":"2022-05-13T23:14:16.258455+0000","last_fullsized":"2022-05-13T23:14:16.258455+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_clean_scrub_stamp":"2022-05-13T23:14:15.659362+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T07:27:48.568156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":10,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-05-13T23:14:16.268155+0000","last_change":"2022-05-13T23:14:16.267256+0000","last_active":"2022-05-13T23:14:16.268155+0000","last_peered":"2022-05-13T23:14:16.268155+0000","last_clean":"2022-05-13T23:14:16.268155+0000","last_became_active":"2022-05-13T23:14:16.266868+0000","last_became_peered":"2022-05-13T23:14:16.266868+0000","last_unstale":"2022-05-13T23:14:16.268155+0000","last_undegraded":"2022-05-13T23:14:16.268155+0000","last_fullsized":"2022-05-13T23:14:16.268155+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_clean_scrub_stamp":"2022-05-13T23:14:15.659362+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T03:28:58.592648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":10,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-05-13T23:14:16.257196+0000","last_change":"2022-05-13T23:14:16.256051+0000","last_active":"2022-05-13T23:14:16.257196+0000","last_peered":"2022-05-13T23:14:16.257196+0000","last_clean":"2022-05-13T23:14:16.257196+0000","last_became_active":"2022-05-13T23:14:16.255729+0000","last_became_peered":"2022-05-13T23:14:16.255729+0000","last_unstale":"2022-05-13T23:14:16.257196+0000","last_undegraded":"2022-05-13T23:14:16.257196+0000","last_fullsized":"2022-05-13T23:14:16.257196+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_clean_scrub_stamp":"2022-05-13T23:14:15.659362+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T09:34:22.448938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":10,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-05-13T23:14:16.268884+0000","last_change":"2022-05-13T23:14:16.266285+0000","last_active":"2022-05-13T23:14:16.268884+0000","last_peered":"2022-05-13T23:14:16.268884+0000","last_clean":"2022-05-13T23:14:16.268884+0000","last_became_active":"2022-05-13T23:14:16.264899+0000","last_became_peered":"2022-05-13T23:14:16.264899+0000","last_unstale":"2022-05-13T23:14:16.268884+0000","last_undegraded":"2022-05-13T23:14:16.268884+0000","last_fullsized":"2022-05-13T23:14:16.268884+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_clean_scrub_stamp":"2022-05-13T23:14:15.659362+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T09:50:56.211965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":10,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-05-13T23:14:16.279550+0000","last_change":"2022-05-13T23:14:16.271503+0000","last_active":"2022-05-13T23:14:16.279550+0000","last_peered":"2022-05-13T23:14:16.279550+0000","last_clean":"2022-05-13T23:14:16.279550+0000","last_became_active":"2022-05-13T23:14:16.271135+0000","last_became_peered":"2022-05-13T23:14:16.271135+0000","last_unstale":"2022-05-13T23:14:16.279550+0000","last_undegraded":"2022-05-13T23:14:16.279550+0000","last_fullsized":"2022-05-13T23:14:16.279550+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_clean_scrub_stamp":"2022-05-13T23:14:15.659362+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T05:01:25.159085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":10,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-05-13T23:14:16.273627+0000","last_change":"2022-05-13T23:14:16.265771+0000","last_active":"2022-05-13T23:14:16.273627+0000","last_peered":"2022-05-13T23:14:16.273627+0000","last_clean":"2022-05-13T23:14:16.273627+0000","last_became_active":"2022-05-13T23:14:16.265357+0000","last_became_peered":"2022-05-13T23:14:16.265357+0000","last_unstale":"2022-05-13T23:14:16.273627+0000","last_undegraded":"2022-05-13T23:14:16.273627+0000","last_fullsized":"2022-05-13T23:14:16.273627+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_clean_scrub_stamp":"2022-05-13T23:14:15.659362+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T02:16:47.449089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":10,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-05-13T23:14:16.270716+0000","last_change":"2022-05-13T23:14:16.270173+0000","last_active":"2022-05-13T23:14:16.270716+0000","last_peered":"2022-05-13T23:14:16.270716+0000","last_clean":"2022-05-13T23:14:16.270716+0000","last_became_active":"2022-05-13T23:14:16.269859+0000","last_became_peered":"2022-05-13T23:14:16.269859+0000","last_unstale":"2022-05-13T23:14:16.270716+0000","last_undegraded":"2022-05-13T23:14:16.270716+0000","last_fullsized":"2022-05-13T23:14:16.270716+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_clean_scrub_stamp":"2022-05-13T23:14:15.659362+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T03:31:45.566757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":10,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-05-13T23:14:16.269931+0000","last_change":"2022-05-13T23:14:16.269290+0000","last_active":"2022-05-13T23:14:16.269931+0000","last_peered":"2022-05-13T23:14:16.269931+0000","last_clean":"2022-05-13T23:14:16.269931+0000","last_became_active":"2022-05-13T23:14:16.268985+0000","last_became_peered":"2022-05-13T23:14:16.268985+0000","last_unstale":"2022-05-13T23:14:16.269931+0000","last_undegraded":"2022-05-13T23:14:16.269931+0000","last_fullsized":"2022-05-13T23:14:16.269931+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_clean_scrub_stamp":"2022-05-13T23:14:15.659362+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T03:58:45.885549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":10,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-05-13T23:14:16.267609+0000","last_change":"2022-05-13T23:14:16.266204+0000","last_active":"2022-05-13T23:14:16.267609+0000","last_peered":"2022-05-13T23:14:16.267609+0000","last_clean":"2022-05-13T23:14:16.267609+0000","last_became_active":"2022-05-13T23:14:16.262256+0000","last_became_peered":"2022-05-13T23:14:16.262256+0000","last_unstale":"2022-05-13T23:14:16.267609+0000","last_undegraded":"2022-05-13T23:14:16.267609+0000","last_fullsized":"2022-05-13T23:14:16.267609+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_clean_scrub_stamp":"2022-05-13T23:14:15.659362+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T00:53:26.594598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":10,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-05-13T23:14:16.254638+0000","last_change":"2022-05-13T23:14:16.252421+0000","last_active":"2022-05-13T23:14:16.254638+0000","last_peered":"2022-05-13T23:14:16.254638+0000","last_clean":"2022-05-13T23:14:16.254638+0000","last_became_active":"2022-05-13T23:14:16.250585+0000","last_became_peered":"2022-05-13T23:14:16.250585+0000","last_unstale":"2022-05-13T23:14:16.254638+0000","last_undegraded":"2022-05-13T23:14:16.254638+0000","last_fullsized":"2022-05-13T23:14:16.254638+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_clean_scrub_stamp":"2022-05-13T23:14:15.659362+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T00:13:24.393163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"83'11","reported_seq":22,"reported_epoch":83,"state":"active+clean","last_fresh":"2022-05-13T23:14:21.309634+0000","last_change":"2022-05-13T23:14:15.683837+0000","last_active":"2022-05-13T23:14:21.309634+0000","last_peered":"2022-05-13T23:14:21.309634+0000","last_clean":"2022-05-13T23:14:21.309634+0000","last_became_active":"2022-05-13T23:14:15.682976+0000","last_became_peered":"2022-05-13T23:14:15.682976+0000","last_unstale":"2022-05-13T23:14:21.309634+0000","last_undegraded":"2022-05-13T23:14:21.309634+0000","last_fullsized":"2022-05-13T23:14:21.309634+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:14.344206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:14.344206+0000","last_clean_scrub_stamp":"2022-05-13T23:14:14.344206+0000","objects_scrubbed":0,"log_size":11,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T01:05:36.612390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":554,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":10,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-05-13T23:14:16.267355+0000","last_change":"2022-05-13T23:14:16.264501+0000","last_active":"2022-05-13T23:14:16.267355+0000","last_peered":"2022-05-13T23:14:16.267355+0000","last_clean":"2022-05-13T23:14:16.267355+0000","last_became_active":"2022-05-13T23:14:16.264105+0000","last_became_peered":"2022-05-13T23:14:16.264105+0000","last_unstale":"2022-05-13T23:14:16.267355+0000","last_undegraded":"2022-05-13T23:14:16.267355+0000","last_fullsized":"2022-05-13T23:14:16.267355+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_clean_scrub_stamp":"2022-05-13T23:14:15.659362+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T00:21:48.857254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"82'1","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-05-13T23:14:16.282001+0000","last_change":"2022-05-13T23:14:15.701757+0000","last_active":"2022-05-13T23:14:16.282001+0000","last_peered":"2022-05-13T23:14:16.282001+0000","last_clean":"2022-05-13T23:14:16.282001+0000","last_became_active":"2022-05-13T23:14:15.701428+0000","last_became_peered":"2022-05-13T23:14:15.701428+0000","last_unstale":"2022-05-13T23:14:16.282001+0000","last_undegraded":"2022-05-13T23:14:16.282001+0000","last_fullsized":"2022-05-13T23:14:16.282001+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:14.344206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:14.344206+0000","last_clean_scrub_stamp":"2022-05-13T23:14:14.344206+0000","objects_scrubbed":0,"log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T05:26:07.810433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":10,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-05-13T23:14:16.281726+0000","last_change":"2022-05-13T23:14:16.280002+0000","last_active":"2022-05-13T23:14:16.281726+0000","last_peered":"2022-05-13T23:14:16.281726+0000","last_clean":"2022-05-13T23:14:16.281726+0000","last_became_active":"2022-05-13T23:14:16.279621+0000","last_became_peered":"2022-05-13T23:14:16.279621+0000","last_unstale":"2022-05-13T23:14:16.281726+0000","last_undegraded":"2022-05-13T23:14:16.281726+0000","last_fullsized":"2022-05-13T23:14:16.281726+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_clean_scrub_stamp":"2022-05-13T23:14:15.659362+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T06:42:49.090643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"83'4","reported_seq":15,"reported_epoch":83,"state":"active+clean","last_fresh":"2022-05-13T23:14:20.302418+0000","last_change":"2022-05-13T23:14:15.683457+0000","last_active":"2022-05-13T23:14:20.302418+0000","last_peered":"2022-05-13T23:14:20.302418+0000","last_clean":"2022-05-13T23:14:20.302418+0000","last_became_active":"2022-05-13T23:14:15.682883+0000","last_became_peered":"2022-05-13T23:14:15.682883+0000","last_unstale":"2022-05-13T23:14:20.302418+0000","last_undegraded":"2022-05-13T23:14:20.302418+0000","last_fullsized":"2022-05-13T23:14:20.302418+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:14.344206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:14.344206+0000","last_clean_scrub_stamp":"2022-05-13T23:14:14.344206+0000","objects_scrubbed":0,"log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T07:38:58.996672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":10,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-05-13T23:14:16.261946+0000","last_change":"2022-05-13T23:14:16.261315+0000","last_active":"2022-05-13T23:14:16.261946+0000","last_peered":"2022-05-13T23:14:16.261946+0000","last_clean":"2022-05-13T23:14:16.261946+0000","last_became_active":"2022-05-13T23:14:16.260265+0000","last_became_peered":"2022-05-13T23:14:16.260265+0000","last_unstale":"2022-05-13T23:14:16.261946+0000","last_undegraded":"2022-05-13T23:14:16.261946+0000","last_fullsized":"2022-05-13T23:14:16.261946+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_clean_scrub_stamp":"2022-05-13T23:14:15.659362+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T08:02:58.080071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"82'2","reported_seq":13,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-05-13T23:14:16.267221+0000","last_change":"2022-05-13T23:14:15.682693+0000","last_active":"2022-05-13T23:14:16.267221+0000","last_peered":"2022-05-13T23:14:16.267221+0000","last_clean":"2022-05-13T23:14:16.267221+0000","last_became_active":"2022-05-13T23:14:15.682414+0000","last_became_peered":"2022-05-13T23:14:15.682414+0000","last_unstale":"2022-05-13T23:14:16.267221+0000","last_undegraded":"2022-05-13T23:14:16.267221+0000","last_fullsized":"2022-05-13T23:14:16.267221+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:14.344206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:14.344206+0000","last_clean_scrub_stamp":"2022-05-13T23:14:14.344206+0000","objects_scrubbed":0,"log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T09:15:17.699871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":10,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-05-13T23:14:16.261554+0000","last_change":"2022-05-13T23:14:16.261044+0000","last_active":"2022-05-13T23:14:16.261554+0000","last_peered":"2022-05-13T23:14:16.261554+0000","last_clean":"2022-05-13T23:14:16.261554+0000","last_became_active":"2022-05-13T23:14:16.260687+0000","last_became_peered":"2022-05-13T23:14:16.260687+0000","last_unstale":"2022-05-13T23:14:16.261554+0000","last_undegraded":"2022-05-13T23:14:16.261554+0000","last_fullsized":"2022-05-13T23:14:16.261554+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_clean_scrub_stamp":"2022-05-13T23:14:15.659362+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-14T23:38:23.413188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"83'5","reported_seq":16,"reported_epoch":83,"state":"active+clean","last_fresh":"2022-05-13T23:14:19.292636+0000","last_change":"2022-05-13T23:14:15.678240+0000","last_active":"2022-05-13T23:14:19.292636+0000","last_peered":"2022-05-13T23:14:19.292636+0000","last_clean":"2022-05-13T23:14:19.292636+0000","last_became_active":"2022-05-13T23:14:15.677918+0000","last_became_peered":"2022-05-13T23:14:15.677918+0000","last_unstale":"2022-05-13T23:14:19.292636+0000","last_undegraded":"2022-05-13T23:14:19.292636+0000","last_fullsized":"2022-05-13T23:14:19.292636+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:14.344206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:14.344206+0000","last_clean_scrub_stamp":"2022-05-13T23:14:14.344206+0000","objects_scrubbed":0,"log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T09:53:37.623728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":574,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":10,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-05-13T23:14:16.274210+0000","last_change":"2022-05-13T23:14:16.273537+0000","last_active":"2022-05-13T23:14:16.274210+0000","last_peered":"2022-05-13T23:14:16.274210+0000","last_clean":"2022-05-13T23:14:16.274210+0000","last_became_active":"2022-05-13T23:14:16.273127+0000","last_became_peered":"2022-05-13T23:14:16.273127+0000","last_unstale":"2022-05-13T23:14:16.274210+0000","last_undegraded":"2022-05-13T23:14:16.274210+0000","last_fullsized":"2022-05-13T23:14:16.274210+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_clean_scrub_stamp":"2022-05-13T23:14:15.659362+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T01:00:18.643721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"82'1","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-05-13T23:14:16.267285+0000","last_change":"2022-05-13T23:14:15.680562+0000","last_active":"2022-05-13T23:14:16.267285+0000","last_peered":"2022-05-13T23:14:16.267285+0000","last_clean":"2022-05-13T23:14:16.267285+0000","last_became_active":"2022-05-13T23:14:15.680163+0000","last_became_peered":"2022-05-13T23:14:15.680163+0000","last_unstale":"2022-05-13T23:14:16.267285+0000","last_undegraded":"2022-05-13T23:14:16.267285+0000","last_fullsized":"2022-05-13T23:14:16.267285+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:14.344206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:14.344206+0000","last_clean_scrub_stamp":"2022-05-13T23:14:14.344206+0000","objects_scrubbed":0,"log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T02:39:27.895407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":10,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-05-13T23:14:16.271643+0000","last_change":"2022-05-13T23:14:16.270878+0000","last_active":"2022-05-13T23:14:16.271643+0000","last_peered":"2022-05-13T23:14:16.271643+0000","last_clean":"2022-05-13T23:14:16.271643+0000","last_became_active":"2022-05-13T23:14:16.270416+0000","last_became_peered":"2022-05-13T23:14:16.270416+0000","last_unstale":"2022-05-13T23:14:16.271643+0000","last_undegraded":"2022-05-13T23:14:16.271643+0000","last_fullsized":"2022-05-13T23:14:16.271643+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_clean_scrub_stamp":"2022-05-13T23:14:15.659362+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T06:11:33.932416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"83'2","reported_seq":13,"reported_epoch":83,"state":"active+clean","last_fresh":"2022-05-13T23:14:20.303430+0000","last_change":"2022-05-13T23:14:15.676518+0000","last_active":"2022-05-13T23:14:20.303430+0000","last_peered":"2022-05-13T23:14:20.303430+0000","last_clean":"2022-05-13T23:14:20.303430+0000","last_became_active":"2022-05-13T23:14:15.676157+0000","last_became_peered":"2022-05-13T23:14:15.676157+0000","last_unstale":"2022-05-13T23:14:20.303430+0000","last_undegraded":"2022-05-13T23:14:20.303430+0000","last_fullsized":"2022-05-13T23:14:20.303430+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:14.344206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:14.344206+0000","last_clean_scrub_stamp":"2022-05-13T23:14:14.344206+0000","objects_scrubbed":0,"log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T10:16:49.469347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":10,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-05-13T23:14:16.262532+0000","last_change":"2022-05-13T23:14:16.257424+0000","last_active":"2022-05-13T23:14:16.262532+0000","last_peered":"2022-05-13T23:14:16.262532+0000","last_clean":"2022-05-13T23:14:16.262532+0000","last_became_active":"2022-05-13T23:14:16.254550+0000","last_became_peered":"2022-05-13T23:14:16.254550+0000","last_unstale":"2022-05-13T23:14:16.262532+0000","last_undegraded":"2022-05-13T23:14:16.262532+0000","last_fullsized":"2022-05-13T23:14:16.262532+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_clean_scrub_stamp":"2022-05-13T23:14:15.659362+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T00:04:50.150368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"83'1","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2022-05-13T23:14:17.560883+0000","last_change":"2022-05-13T23:14:15.680378+0000","last_active":"2022-05-13T23:14:17.560883+0000","last_peered":"2022-05-13T23:14:17.560883+0000","last_clean":"2022-05-13T23:14:17.560883+0000","last_became_active":"2022-05-13T23:14:15.680063+0000","last_became_peered":"2022-05-13T23:14:15.680063+0000","last_unstale":"2022-05-13T23:14:17.560883+0000","last_undegraded":"2022-05-13T23:14:17.560883+0000","last_fullsized":"2022-05-13T23:14:17.560883+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:14.344206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:14.344206+0000","last_clean_scrub_stamp":"2022-05-13T23:14:14.344206+0000","objects_scrubbed":0,"log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-14T23:52:48.817491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":10,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-05-13T23:14:16.281100+0000","last_change":"2022-05-13T23:14:16.277131+0000","last_active":"2022-05-13T23:14:16.281100+0000","last_peered":"2022-05-13T23:14:16.281100+0000","last_clean":"2022-05-13T23:14:16.281100+0000","last_became_active":"2022-05-13T23:14:16.276815+0000","last_became_peered":"2022-05-13T23:14:16.276815+0000","last_unstale":"2022-05-13T23:14:16.281100+0000","last_undegraded":"2022-05-13T23:14:16.281100+0000","last_fullsized":"2022-05-13T23:14:16.281100+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_clean_scrub_stamp":"2022-05-13T23:14:15.659362+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-14T23:22:29.938073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":11,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-05-13T23:14:16.242578+0000","last_change":"2022-05-13T23:14:15.683981+0000","last_active":"2022-05-13T23:14:16.242578+0000","last_peered":"2022-05-13T23:14:16.242578+0000","last_clean":"2022-05-13T23:14:16.242578+0000","last_became_active":"2022-05-13T23:14:15.683560+0000","last_became_peered":"2022-05-13T23:14:15.683560+0000","last_unstale":"2022-05-13T23:14:16.242578+0000","last_undegraded":"2022-05-13T23:14:16.242578+0000","last_fullsized":"2022-05-13T23:14:16.242578+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:14.344206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:14.344206+0000","last_clean_scrub_stamp":"2022-05-13T23:14:14.344206+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T00:30:01.510592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":10,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-05-13T23:14:16.269905+0000","last_change":"2022-05-13T23:14:16.269244+0000","last_active":"2022-05-13T23:14:16.269905+0000","last_peered":"2022-05-13T23:14:16.269905+0000","last_clean":"2022-05-13T23:14:16.269905+0000","last_became_active":"2022-05-13T23:14:16.268949+0000","last_became_peered":"2022-05-13T23:14:16.268949+0000","last_unstale":"2022-05-13T23:14:16.269905+0000","last_undegraded":"2022-05-13T23:14:16.269905+0000","last_fullsized":"2022-05-13T23:14:16.269905+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_clean_scrub_stamp":"2022-05-13T23:14:15.659362+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T04:34:53.311191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"83'2","reported_seq":13,"reported_epoch":83,"state":"active+clean","last_fresh":"2022-05-13T23:14:19.290837+0000","last_change":"2022-05-13T23:14:15.683758+0000","last_active":"2022-05-13T23:14:19.290837+0000","last_peered":"2022-05-13T23:14:19.290837+0000","last_clean":"2022-05-13T23:14:19.290837+0000","last_became_active":"2022-05-13T23:14:15.683356+0000","last_became_peered":"2022-05-13T23:14:15.683356+0000","last_unstale":"2022-05-13T23:14:19.290837+0000","last_undegraded":"2022-05-13T23:14:19.290837+0000","last_fullsized":"2022-05-13T23:14:19.290837+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:14.344206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:14.344206+0000","last_clean_scrub_stamp":"2022-05-13T23:14:14.344206+0000","objects_scrubbed":0,"log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T08:00:55.113935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":10,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-05-13T23:14:16.274939+0000","last_change":"2022-05-13T23:14:16.268751+0000","last_active":"2022-05-13T23:14:16.274939+0000","last_peered":"2022-05-13T23:14:16.274939+0000","last_clean":"2022-05-13T23:14:16.274939+0000","last_became_active":"2022-05-13T23:14:16.268332+0000","last_became_peered":"2022-05-13T23:14:16.268332+0000","last_unstale":"2022-05-13T23:14:16.274939+0000","last_undegraded":"2022-05-13T23:14:16.274939+0000","last_fullsized":"2022-05-13T23:14:16.274939+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_clean_scrub_stamp":"2022-05-13T23:14:15.659362+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T06:22:45.140343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"83'4","reported_seq":15,"reported_epoch":83,"state":"active+clean","last_fresh":"2022-05-13T23:14:17.561666+0000","last_change":"2022-05-13T23:14:15.701120+0000","last_active":"2022-05-13T23:14:17.561666+0000","last_peered":"2022-05-13T23:14:17.561666+0000","last_clean":"2022-05-13T23:14:17.561666+0000","last_became_active":"2022-05-13T23:14:15.700812+0000","last_became_peered":"2022-05-13T23:14:15.700812+0000","last_unstale":"2022-05-13T23:14:17.561666+0000","last_undegraded":"2022-05-13T23:14:17.561666+0000","last_fullsized":"2022-05-13T23:14:17.561666+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:14.344206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:14.344206+0000","last_clean_scrub_stamp":"2022-05-13T23:14:14.344206+0000","objects_scrubbed":0,"log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T02:16:02.909062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":644,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":10,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-05-13T23:14:16.266148+0000","last_change":"2022-05-13T23:14:16.265167+0000","last_active":"2022-05-13T23:14:16.266148+0000","last_peered":"2022-05-13T23:14:16.266148+0000","last_clean":"2022-05-13T23:14:16.266148+0000","last_became_active":"2022-05-13T23:14:16.264886+0000","last_became_peered":"2022-05-13T23:14:16.264886+0000","last_unstale":"2022-05-13T23:14:16.266148+0000","last_undegraded":"2022-05-13T23:14:16.266148+0000","last_fullsized":"2022-05-13T23:14:16.266148+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_clean_scrub_stamp":"2022-05-13T23:14:15.659362+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T00:26:06.650867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"82'6","reported_seq":17,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-05-13T23:14:16.269389+0000","last_change":"2022-05-13T23:14:15.682385+0000","last_active":"2022-05-13T23:14:16.269389+0000","last_peered":"2022-05-13T23:14:16.269389+0000","last_clean":"2022-05-13T23:14:16.269389+0000","last_became_active":"2022-05-13T23:14:15.681996+0000","last_became_peered":"2022-05-13T23:14:15.681996+0000","last_unstale":"2022-05-13T23:14:16.269389+0000","last_undegraded":"2022-05-13T23:14:16.269389+0000","last_fullsized":"2022-05-13T23:14:16.269389+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:14.344206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:14.344206+0000","last_clean_scrub_stamp":"2022-05-13T23:14:14.344206+0000","objects_scrubbed":0,"log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-14T23:44:23.766036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1536,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":10,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-05-13T23:14:16.264519+0000","last_change":"2022-05-13T23:14:16.263744+0000","last_active":"2022-05-13T23:14:16.264519+0000","last_peered":"2022-05-13T23:14:16.264519+0000","last_clean":"2022-05-13T23:14:16.264519+0000","last_became_active":"2022-05-13T23:14:16.263525+0000","last_became_peered":"2022-05-13T23:14:16.263525+0000","last_unstale":"2022-05-13T23:14:16.264519+0000","last_undegraded":"2022-05-13T23:14:16.264519+0000","last_fullsized":"2022-05-13T23:14:16.264519+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_clean_scrub_stamp":"2022-05-13T23:14:15.659362+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T04:23:44.856055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"83'6","reported_seq":17,"reported_epoch":83,"state":"active+clean","last_fresh":"2022-05-13T23:14:20.301591+0000","last_change":"2022-05-13T23:14:15.683079+0000","last_active":"2022-05-13T23:14:20.301591+0000","last_peered":"2022-05-13T23:14:20.301591+0000","last_clean":"2022-05-13T23:14:20.301591+0000","last_became_active":"2022-05-13T23:14:15.682750+0000","last_became_peered":"2022-05-13T23:14:15.682750+0000","last_unstale":"2022-05-13T23:14:20.301591+0000","last_undegraded":"2022-05-13T23:14:20.301591+0000","last_fullsized":"2022-05-13T23:14:20.301591+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:14.344206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:14.344206+0000","last_clean_scrub_stamp":"2022-05-13T23:14:14.344206+0000","objects_scrubbed":0,"log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T02:37:31.530579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":10,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-05-13T23:14:16.268332+0000","last_change":"2022-05-13T23:14:16.263943+0000","last_active":"2022-05-13T23:14:16.268332+0000","last_peered":"2022-05-13T23:14:16.268332+0000","last_clean":"2022-05-13T23:14:16.268332+0000","last_became_active":"2022-05-13T23:14:16.263595+0000","last_became_peered":"2022-05-13T23:14:16.263595+0000","last_unstale":"2022-05-13T23:14:16.268332+0000","last_undegraded":"2022-05-13T23:14:16.268332+0000","last_fullsized":"2022-05-13T23:14:16.268332+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_clean_scrub_stamp":"2022-05-13T23:14:15.659362+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T04:57:04.201317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"83'3","reported_seq":14,"reported_epoch":83,"state":"active+clean","last_fresh":"2022-05-13T23:14:19.292697+0000","last_change":"2022-05-13T23:14:15.683005+0000","last_active":"2022-05-13T23:14:19.292697+0000","last_peered":"2022-05-13T23:14:19.292697+0000","last_clean":"2022-05-13T23:14:19.292697+0000","last_became_active":"2022-05-13T23:14:15.682692+0000","last_became_peered":"2022-05-13T23:14:15.682692+0000","last_unstale":"2022-05-13T23:14:19.292697+0000","last_undegraded":"2022-05-13T23:14:19.292697+0000","last_fullsized":"2022-05-13T23:14:19.292697+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:14.344206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:14.344206+0000","last_clean_scrub_stamp":"2022-05-13T23:14:14.344206+0000","objects_scrubbed":0,"log_size":3,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T00:27:25.201141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":10,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-05-13T23:14:16.265569+0000","last_change":"2022-05-13T23:14:16.264351+0000","last_active":"2022-05-13T23:14:16.265569+0000","last_peered":"2022-05-13T23:14:16.265569+0000","last_clean":"2022-05-13T23:14:16.265569+0000","last_became_active":"2022-05-13T23:14:16.264023+0000","last_became_peered":"2022-05-13T23:14:16.264023+0000","last_unstale":"2022-05-13T23:14:16.265569+0000","last_undegraded":"2022-05-13T23:14:16.265569+0000","last_fullsized":"2022-05-13T23:14:16.265569+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_clean_scrub_stamp":"2022-05-13T23:14:15.659362+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T06:13:51.169049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":11,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-05-13T23:14:16.246274+0000","last_change":"2022-05-13T23:14:15.677080+0000","last_active":"2022-05-13T23:14:16.246274+0000","last_peered":"2022-05-13T23:14:16.246274+0000","last_clean":"2022-05-13T23:14:16.246274+0000","last_became_active":"2022-05-13T23:14:15.676743+0000","last_became_peered":"2022-05-13T23:14:15.676743+0000","last_unstale":"2022-05-13T23:14:16.246274+0000","last_undegraded":"2022-05-13T23:14:16.246274+0000","last_fullsized":"2022-05-13T23:14:16.246274+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:14.344206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:14.344206+0000","last_clean_scrub_stamp":"2022-05-13T23:14:14.344206+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T09:05:30.321765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":10,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-05-13T23:14:16.263982+0000","last_change":"2022-05-13T23:14:16.262582+0000","last_active":"2022-05-13T23:14:16.263982+0000","last_peered":"2022-05-13T23:14:16.263982+0000","last_clean":"2022-05-13T23:14:16.263982+0000","last_became_active":"2022-05-13T23:14:16.262242+0000","last_became_peered":"2022-05-13T23:14:16.262242+0000","last_unstale":"2022-05-13T23:14:16.263982+0000","last_undegraded":"2022-05-13T23:14:16.263982+0000","last_fullsized":"2022-05-13T23:14:16.263982+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_clean_scrub_stamp":"2022-05-13T23:14:15.659362+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T02:27:28.155612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"83'15","reported_seq":26,"reported_epoch":83,"state":"active+clean","last_fresh":"2022-05-13T23:14:20.305188+0000","last_change":"2022-05-13T23:14:15.698465+0000","last_active":"2022-05-13T23:14:20.305188+0000","last_peered":"2022-05-13T23:14:20.305188+0000","last_clean":"2022-05-13T23:14:20.305188+0000","last_became_active":"2022-05-13T23:14:15.698131+0000","last_became_peered":"2022-05-13T23:14:15.698131+0000","last_unstale":"2022-05-13T23:14:20.305188+0000","last_undegraded":"2022-05-13T23:14:20.305188+0000","last_fullsized":"2022-05-13T23:14:20.305188+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:14.344206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:14.344206+0000","last_clean_scrub_stamp":"2022-05-13T23:14:14.344206+0000","objects_scrubbed":0,"log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T03:03:48.104345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":10,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-05-13T23:14:16.261102+0000","last_change":"2022-05-13T23:14:16.260579+0000","last_active":"2022-05-13T23:14:16.261102+0000","last_peered":"2022-05-13T23:14:16.261102+0000","last_clean":"2022-05-13T23:14:16.261102+0000","last_became_active":"2022-05-13T23:14:16.260326+0000","last_became_peered":"2022-05-13T23:14:16.260326+0000","last_unstale":"2022-05-13T23:14:16.261102+0000","last_undegraded":"2022-05-13T23:14:16.261102+0000","last_fullsized":"2022-05-13T23:14:16.261102+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_clean_scrub_stamp":"2022-05-13T23:14:15.659362+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T06:53:23.092756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"82'1","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-05-13T23:14:16.262768+0000","last_change":"2022-05-13T23:14:15.680655+0000","last_active":"2022-05-13T23:14:16.262768+0000","last_peered":"2022-05-13T23:14:16.262768+0000","last_clean":"2022-05-13T23:14:16.262768+0000","last_became_active":"2022-05-13T23:14:15.680357+0000","last_became_peered":"2022-05-13T23:14:15.680357+0000","last_unstale":"2022-05-13T23:14:16.262768+0000","last_undegraded":"2022-05-13T23:14:16.262768+0000","last_fullsized":"2022-05-13T23:14:16.262768+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:14.344206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:14.344206+0000","last_clean_scrub_stamp":"2022-05-13T23:14:14.344206+0000","objects_scrubbed":0,"log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T11:00:22.395993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":10,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-05-13T23:14:16.262415+0000","last_change":"2022-05-13T23:14:16.260574+0000","last_active":"2022-05-13T23:14:16.262415+0000","last_peered":"2022-05-13T23:14:16.262415+0000","last_clean":"2022-05-13T23:14:16.262415+0000","last_became_active":"2022-05-13T23:14:16.260258+0000","last_became_peered":"2022-05-13T23:14:16.260258+0000","last_unstale":"2022-05-13T23:14:16.262415+0000","last_undegraded":"2022-05-13T23:14:16.262415+0000","last_fullsized":"2022-05-13T23:14:16.262415+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_clean_scrub_stamp":"2022-05-13T23:14:15.659362+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T10:49:06.210598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"83'1","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2022-05-13T23:14:17.561161+0000","last_change":"2022-05-13T23:14:15.701322+0000","last_active":"2022-05-13T23:14:17.561161+0000","last_peered":"2022-05-13T23:14:17.561161+0000","last_clean":"2022-05-13T23:14:17.561161+0000","last_became_active":"2022-05-13T23:14:15.701007+0000","last_became_peered":"2022-05-13T23:14:15.701007+0000","last_unstale":"2022-05-13T23:14:17.561161+0000","last_undegraded":"2022-05-13T23:14:17.561161+0000","last_fullsized":"2022-05-13T23:14:17.561161+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:14.344206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:14.344206+0000","last_clean_scrub_stamp":"2022-05-13T23:14:14.344206+0000","objects_scrubbed":0,"log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T00:13:11.071695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":10,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-05-13T23:14:16.259088+0000","last_change":"2022-05-13T23:14:16.249579+0000","last_active":"2022-05-13T23:14:16.259088+0000","last_peered":"2022-05-13T23:14:16.259088+0000","last_clean":"2022-05-13T23:14:16.259088+0000","last_became_active":"2022-05-13T23:14:16.249245+0000","last_became_peered":"2022-05-13T23:14:16.249245+0000","last_unstale":"2022-05-13T23:14:16.259088+0000","last_undegraded":"2022-05-13T23:14:16.259088+0000","last_fullsized":"2022-05-13T23:14:16.259088+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_clean_scrub_stamp":"2022-05-13T23:14:15.659362+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T00:26:53.404909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":11,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-05-13T23:14:16.255333+0000","last_change":"2022-05-13T23:14:15.699433+0000","last_active":"2022-05-13T23:14:16.255333+0000","last_peered":"2022-05-13T23:14:16.255333+0000","last_clean":"2022-05-13T23:14:16.255333+0000","last_became_active":"2022-05-13T23:14:15.698920+0000","last_became_peered":"2022-05-13T23:14:15.698920+0000","last_unstale":"2022-05-13T23:14:16.255333+0000","last_undegraded":"2022-05-13T23:14:16.255333+0000","last_fullsized":"2022-05-13T23:14:16.255333+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:14.344206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:14.344206+0000","last_clean_scrub_stamp":"2022-05-13T23:14:14.344206+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T07:56:34.276565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":10,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-05-13T23:14:16.260728+0000","last_change":"2022-05-13T23:14:16.259866+0000","last_active":"2022-05-13T23:14:16.260728+0000","last_peered":"2022-05-13T23:14:16.260728+0000","last_clean":"2022-05-13T23:14:16.260728+0000","last_became_active":"2022-05-13T23:14:16.259531+0000","last_became_peered":"2022-05-13T23:14:16.259531+0000","last_unstale":"2022-05-13T23:14:16.260728+0000","last_undegraded":"2022-05-13T23:14:16.260728+0000","last_fullsized":"2022-05-13T23:14:16.260728+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_clean_scrub_stamp":"2022-05-13T23:14:15.659362+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T07:41:46.019217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":10,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-05-13T23:14:16.261105+0000","last_change":"2022-05-13T23:14:16.259613+0000","last_active":"2022-05-13T23:14:16.261105+0000","last_peered":"2022-05-13T23:14:16.261105+0000","last_clean":"2022-05-13T23:14:16.261105+0000","last_became_active":"2022-05-13T23:14:16.259274+0000","last_became_peered":"2022-05-13T23:14:16.259274+0000","last_unstale":"2022-05-13T23:14:16.261105+0000","last_undegraded":"2022-05-13T23:14:16.261105+0000","last_fullsized":"2022-05-13T23:14:16.261105+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_clean_scrub_stamp":"2022-05-13T23:14:15.659362+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T05:36:21.174814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"83'4","reported_seq":15,"reported_epoch":83,"state":"active+clean","last_fresh":"2022-05-13T23:14:20.304530+0000","last_change":"2022-05-13T23:14:15.701315+0000","last_active":"2022-05-13T23:14:20.304530+0000","last_peered":"2022-05-13T23:14:20.304530+0000","last_clean":"2022-05-13T23:14:20.304530+0000","last_became_active":"2022-05-13T23:14:15.700955+0000","last_became_peered":"2022-05-13T23:14:15.700955+0000","last_unstale":"2022-05-13T23:14:20.304530+0000","last_undegraded":"2022-05-13T23:14:20.304530+0000","last_fullsized":"2022-05-13T23:14:20.304530+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:14.344206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:14.344206+0000","last_clean_scrub_stamp":"2022-05-13T23:14:14.344206+0000","objects_scrubbed":0,"log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T10:17:23.916475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":10,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-05-13T23:14:16.274093+0000","last_change":"2022-05-13T23:14:16.272364+0000","last_active":"2022-05-13T23:14:16.274093+0000","last_peered":"2022-05-13T23:14:16.274093+0000","last_clean":"2022-05-13T23:14:16.274093+0000","last_became_active":"2022-05-13T23:14:16.271612+0000","last_became_peered":"2022-05-13T23:14:16.271612+0000","last_unstale":"2022-05-13T23:14:16.274093+0000","last_undegraded":"2022-05-13T23:14:16.274093+0000","last_fullsized":"2022-05-13T23:14:16.274093+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_clean_scrub_stamp":"2022-05-13T23:14:15.659362+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T07:25:47.039264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":11,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-05-13T23:14:16.251030+0000","last_change":"2022-05-13T23:14:15.676251+0000","last_active":"2022-05-13T23:14:16.251030+0000","last_peered":"2022-05-13T23:14:16.251030+0000","last_clean":"2022-05-13T23:14:16.251030+0000","last_became_active":"2022-05-13T23:14:15.675895+0000","last_became_peered":"2022-05-13T23:14:15.675895+0000","last_unstale":"2022-05-13T23:14:16.251030+0000","last_undegraded":"2022-05-13T23:14:16.251030+0000","last_fullsized":"2022-05-13T23:14:16.251030+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:14.344206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:14.344206+0000","last_clean_scrub_stamp":"2022-05-13T23:14:14.344206+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T06:03:57.398647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":343,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-05-13T23:14:16.253992+0000","last_change":"2022-05-13T23:13:02.805009+0000","last_active":"2022-05-13T23:14:16.253992+0000","last_peered":"2022-05-13T23:14:16.253992+0000","last_clean":"2022-05-13T23:14:16.253992+0000","last_became_active":"2022-05-13T23:13:02.804350+0000","last_became_peered":"2022-05-13T23:13:02.804350+0000","last_unstale":"2022-05-13T23:14:16.253992+0000","last_undegraded":"2022-05-13T23:14:16.253992+0000","last_fullsized":"2022-05-13T23:14:16.253992+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:09:32.001256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:09:32.001256+0000","last_clean_scrub_stamp":"2022-05-13T23:09:32.001256+0000","objects_scrubbed":0,"log_size":137,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T07:28:44.497144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1057856,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":10,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-05-13T23:14:16.267352+0000","last_change":"2022-05-13T23:14:16.261286+0000","last_active":"2022-05-13T23:14:16.267352+0000","last_peered":"2022-05-13T23:14:16.267352+0000","last_clean":"2022-05-13T23:14:16.267352+0000","last_became_active":"2022-05-13T23:14:16.260870+0000","last_became_peered":"2022-05-13T23:14:16.260870+0000","last_unstale":"2022-05-13T23:14:16.267352+0000","last_undegraded":"2022-05-13T23:14:16.267352+0000","last_fullsized":"2022-05-13T23:14:16.267352+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_clean_scrub_stamp":"2022-05-13T23:14:15.659362+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T08:14:34.804797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"83'1","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2022-05-13T23:14:17.561547+0000","last_change":"2022-05-13T23:14:15.686878+0000","last_active":"2022-05-13T23:14:17.561547+0000","last_peered":"2022-05-13T23:14:17.561547+0000","last_clean":"2022-05-13T23:14:17.561547+0000","last_became_active":"2022-05-13T23:14:15.686496+0000","last_became_peered":"2022-05-13T23:14:15.686496+0000","last_unstale":"2022-05-13T23:14:17.561547+0000","last_undegraded":"2022-05-13T23:14:17.561547+0000","last_fullsized":"2022-05-13T23:14:17.561547+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:14.344206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:14.344206+0000","last_clean_scrub_stamp":"2022-05-13T23:14:14.344206+0000","objects_scrubbed":0,"log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T09:19:28.089159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"83'1","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2022-05-13T23:14:17.250484+0000","last_change":"2022-05-13T23:14:15.698503+0000","last_active":"2022-05-13T23:14:17.250484+0000","last_peered":"2022-05-13T23:14:17.250484+0000","last_clean":"2022-05-13T23:14:17.250484+0000","last_became_active":"2022-05-13T23:14:15.698156+0000","last_became_peered":"2022-05-13T23:14:15.698156+0000","last_unstale":"2022-05-13T23:14:17.250484+0000","last_undegraded":"2022-05-13T23:14:17.250484+0000","last_fullsized":"2022-05-13T23:14:17.250484+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:14.344206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:14.344206+0000","last_clean_scrub_stamp":"2022-05-13T23:14:14.344206+0000","objects_scrubbed":0,"log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T09:03:21.133631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":10,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-05-13T23:14:16.271905+0000","last_change":"2022-05-13T23:14:16.270922+0000","last_active":"2022-05-13T23:14:16.271905+0000","last_peered":"2022-05-13T23:14:16.271905+0000","last_clean":"2022-05-13T23:14:16.271905+0000","last_became_active":"2022-05-13T23:14:16.270259+0000","last_became_peered":"2022-05-13T23:14:16.270259+0000","last_unstale":"2022-05-13T23:14:16.271905+0000","last_undegraded":"2022-05-13T23:14:16.271905+0000","last_fullsized":"2022-05-13T23:14:16.271905+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_clean_scrub_stamp":"2022-05-13T23:14:15.659362+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T02:48:22.255489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"83'7","reported_seq":18,"reported_epoch":83,"state":"active+clean","last_fresh":"2022-05-13T23:14:21.311871+0000","last_change":"2022-05-13T23:14:15.696304+0000","last_active":"2022-05-13T23:14:21.311871+0000","last_peered":"2022-05-13T23:14:21.311871+0000","last_clean":"2022-05-13T23:14:21.311871+0000","last_became_active":"2022-05-13T23:14:15.695476+0000","last_became_peered":"2022-05-13T23:14:15.695476+0000","last_unstale":"2022-05-13T23:14:21.311871+0000","last_undegraded":"2022-05-13T23:14:21.311871+0000","last_fullsized":"2022-05-13T23:14:21.311871+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:14.344206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:14.344206+0000","last_clean_scrub_stamp":"2022-05-13T23:14:14.344206+0000","objects_scrubbed":0,"log_size":7,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T08:48:32.931789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":10,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-05-13T23:14:16.262969+0000","last_change":"2022-05-13T23:14:16.257063+0000","last_active":"2022-05-13T23:14:16.262969+0000","last_peered":"2022-05-13T23:14:16.262969+0000","last_clean":"2022-05-13T23:14:16.262969+0000","last_became_active":"2022-05-13T23:14:16.256757+0000","last_became_peered":"2022-05-13T23:14:16.256757+0000","last_unstale":"2022-05-13T23:14:16.262969+0000","last_undegraded":"2022-05-13T23:14:16.262969+0000","last_fullsized":"2022-05-13T23:14:16.262969+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_clean_scrub_stamp":"2022-05-13T23:14:15.659362+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T09:24:32.369318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"82'2","reported_seq":13,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-05-13T23:14:16.272570+0000","last_change":"2022-05-13T23:14:15.685568+0000","last_active":"2022-05-13T23:14:16.272570+0000","last_peered":"2022-05-13T23:14:16.272570+0000","last_clean":"2022-05-13T23:14:16.272570+0000","last_became_active":"2022-05-13T23:14:15.685203+0000","last_became_peered":"2022-05-13T23:14:15.685203+0000","last_unstale":"2022-05-13T23:14:16.272570+0000","last_undegraded":"2022-05-13T23:14:16.272570+0000","last_fullsized":"2022-05-13T23:14:16.272570+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:14.344206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:14.344206+0000","last_clean_scrub_stamp":"2022-05-13T23:14:14.344206+0000","objects_scrubbed":0,"log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T03:55:33.271069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":10,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-05-13T23:14:16.281757+0000","last_change":"2022-05-13T23:14:16.280330+0000","last_active":"2022-05-13T23:14:16.281757+0000","last_peered":"2022-05-13T23:14:16.281757+0000","last_clean":"2022-05-13T23:14:16.281757+0000","last_became_active":"2022-05-13T23:14:16.279908+0000","last_became_peered":"2022-05-13T23:14:16.279908+0000","last_unstale":"2022-05-13T23:14:16.281757+0000","last_undegraded":"2022-05-13T23:14:16.281757+0000","last_fullsized":"2022-05-13T23:14:16.281757+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_clean_scrub_stamp":"2022-05-13T23:14:15.659362+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T03:25:42.803064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":11,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-05-13T23:14:16.248471+0000","last_change":"2022-05-13T23:14:15.698443+0000","last_active":"2022-05-13T23:14:16.248471+0000","last_peered":"2022-05-13T23:14:16.248471+0000","last_clean":"2022-05-13T23:14:16.248471+0000","last_became_active":"2022-05-13T23:14:15.698158+0000","last_became_peered":"2022-05-13T23:14:15.698158+0000","last_unstale":"2022-05-13T23:14:16.248471+0000","last_undegraded":"2022-05-13T23:14:16.248471+0000","last_fullsized":"2022-05-13T23:14:16.248471+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:14.344206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:14.344206+0000","last_clean_scrub_stamp":"2022-05-13T23:14:14.344206+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T05:06:14.946233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":10,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-05-13T23:14:16.264874+0000","last_change":"2022-05-13T23:14:16.264110+0000","last_active":"2022-05-13T23:14:16.264874+0000","last_peered":"2022-05-13T23:14:16.264874+0000","last_clean":"2022-05-13T23:14:16.264874+0000","last_became_active":"2022-05-13T23:14:16.263830+0000","last_became_peered":"2022-05-13T23:14:16.263830+0000","last_unstale":"2022-05-13T23:14:16.264874+0000","last_undegraded":"2022-05-13T23:14:16.264874+0000","last_fullsized":"2022-05-13T23:14:16.264874+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_clean_scrub_stamp":"2022-05-13T23:14:15.659362+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T00:06:26.724597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"83'3","reported_seq":14,"reported_epoch":83,"state":"active+clean","last_fresh":"2022-05-13T23:14:19.292890+0000","last_change":"2022-05-13T23:14:15.693926+0000","last_active":"2022-05-13T23:14:19.292890+0000","last_peered":"2022-05-13T23:14:19.292890+0000","last_clean":"2022-05-13T23:14:19.292890+0000","last_became_active":"2022-05-13T23:14:15.692163+0000","last_became_peered":"2022-05-13T23:14:15.692163+0000","last_unstale":"2022-05-13T23:14:19.292890+0000","last_undegraded":"2022-05-13T23:14:19.292890+0000","last_fullsized":"2022-05-13T23:14:19.292890+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:14.344206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:14.344206+0000","last_clean_scrub_stamp":"2022-05-13T23:14:14.344206+0000","objects_scrubbed":0,"log_size":3,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T03:00:12.522714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":10,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-05-13T23:14:16.266559+0000","last_change":"2022-05-13T23:14:16.265409+0000","last_active":"2022-05-13T23:14:16.266559+0000","last_peered":"2022-05-13T23:14:16.266559+0000","last_clean":"2022-05-13T23:14:16.266559+0000","last_became_active":"2022-05-13T23:14:16.265151+0000","last_became_peered":"2022-05-13T23:14:16.265151+0000","last_unstale":"2022-05-13T23:14:16.266559+0000","last_undegraded":"2022-05-13T23:14:16.266559+0000","last_fullsized":"2022-05-13T23:14:16.266559+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_clean_scrub_stamp":"2022-05-13T23:14:15.659362+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T04:05:43.762623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":11,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-05-13T23:14:16.241423+0000","last_change":"2022-05-13T23:14:15.681975+0000","last_active":"2022-05-13T23:14:16.241423+0000","last_peered":"2022-05-13T23:14:16.241423+0000","last_clean":"2022-05-13T23:14:16.241423+0000","last_became_active":"2022-05-13T23:14:15.681391+0000","last_became_peered":"2022-05-13T23:14:15.681391+0000","last_unstale":"2022-05-13T23:14:16.241423+0000","last_undegraded":"2022-05-13T23:14:16.241423+0000","last_fullsized":"2022-05-13T23:14:16.241423+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:14.344206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:14.344206+0000","last_clean_scrub_stamp":"2022-05-13T23:14:14.344206+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T06:56:25.732024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":10,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-05-13T23:14:16.274936+0000","last_change":"2022-05-13T23:14:16.272443+0000","last_active":"2022-05-13T23:14:16.274936+0000","last_peered":"2022-05-13T23:14:16.274936+0000","last_clean":"2022-05-13T23:14:16.274936+0000","last_became_active":"2022-05-13T23:14:16.272031+0000","last_became_peered":"2022-05-13T23:14:16.272031+0000","last_unstale":"2022-05-13T23:14:16.274936+0000","last_undegraded":"2022-05-13T23:14:16.274936+0000","last_fullsized":"2022-05-13T23:14:16.274936+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_clean_scrub_stamp":"2022-05-13T23:14:15.659362+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T10:23:55.694108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"83'4","reported_seq":15,"reported_epoch":83,"state":"active+clean","last_fresh":"2022-05-13T23:14:19.291967+0000","last_change":"2022-05-13T23:14:15.681877+0000","last_active":"2022-05-13T23:14:19.291967+0000","last_peered":"2022-05-13T23:14:19.291967+0000","last_clean":"2022-05-13T23:14:19.291967+0000","last_became_active":"2022-05-13T23:14:15.681579+0000","last_became_peered":"2022-05-13T23:14:15.681579+0000","last_unstale":"2022-05-13T23:14:19.291967+0000","last_undegraded":"2022-05-13T23:14:19.291967+0000","last_fullsized":"2022-05-13T23:14:19.291967+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:14.344206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:14.344206+0000","last_clean_scrub_stamp":"2022-05-13T23:14:14.344206+0000","objects_scrubbed":0,"log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T07:00:11.132049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":10,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-05-13T23:14:16.267797+0000","last_change":"2022-05-13T23:14:16.266877+0000","last_active":"2022-05-13T23:14:16.267797+0000","last_peered":"2022-05-13T23:14:16.267797+0000","last_clean":"2022-05-13T23:14:16.267797+0000","last_became_active":"2022-05-13T23:14:16.266551+0000","last_became_peered":"2022-05-13T23:14:16.266551+0000","last_unstale":"2022-05-13T23:14:16.267797+0000","last_undegraded":"2022-05-13T23:14:16.267797+0000","last_fullsized":"2022-05-13T23:14:16.267797+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_clean_scrub_stamp":"2022-05-13T23:14:15.659362+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T05:57:29.394410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"83'3","reported_seq":14,"reported_epoch":83,"state":"active+clean","last_fresh":"2022-05-13T23:14:20.302018+0000","last_change":"2022-05-13T23:14:15.699586+0000","last_active":"2022-05-13T23:14:20.302018+0000","last_peered":"2022-05-13T23:14:20.302018+0000","last_clean":"2022-05-13T23:14:20.302018+0000","last_became_active":"2022-05-13T23:14:15.699215+0000","last_became_peered":"2022-05-13T23:14:15.699215+0000","last_unstale":"2022-05-13T23:14:20.302018+0000","last_undegraded":"2022-05-13T23:14:20.302018+0000","last_fullsized":"2022-05-13T23:14:20.302018+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:14.344206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:14.344206+0000","last_clean_scrub_stamp":"2022-05-13T23:14:14.344206+0000","objects_scrubbed":0,"log_size":3,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T04:14:54.395550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":554,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":10,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-05-13T23:14:16.267333+0000","last_change":"2022-05-13T23:14:16.265730+0000","last_active":"2022-05-13T23:14:16.267333+0000","last_peered":"2022-05-13T23:14:16.267333+0000","last_clean":"2022-05-13T23:14:16.267333+0000","last_became_active":"2022-05-13T23:14:16.265435+0000","last_became_peered":"2022-05-13T23:14:16.265435+0000","last_unstale":"2022-05-13T23:14:16.267333+0000","last_undegraded":"2022-05-13T23:14:16.267333+0000","last_fullsized":"2022-05-13T23:14:16.267333+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_clean_scrub_stamp":"2022-05-13T23:14:15.659362+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T08:55:57.574132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"82'1","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-05-13T23:14:16.270336+0000","last_change":"2022-05-13T23:14:15.682402+0000","last_active":"2022-05-13T23:14:16.270336+0000","last_peered":"2022-05-13T23:14:16.270336+0000","last_clean":"2022-05-13T23:14:16.270336+0000","last_became_active":"2022-05-13T23:14:15.682105+0000","last_became_peered":"2022-05-13T23:14:15.682105+0000","last_unstale":"2022-05-13T23:14:16.270336+0000","last_undegraded":"2022-05-13T23:14:16.270336+0000","last_fullsized":"2022-05-13T23:14:16.270336+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:14.344206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:14.344206+0000","last_clean_scrub_stamp":"2022-05-13T23:14:14.344206+0000","objects_scrubbed":0,"log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T10:37:55.092658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":10,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-05-13T23:14:16.264851+0000","last_change":"2022-05-13T23:14:16.263001+0000","last_active":"2022-05-13T23:14:16.264851+0000","last_peered":"2022-05-13T23:14:16.264851+0000","last_clean":"2022-05-13T23:14:16.264851+0000","last_became_active":"2022-05-13T23:14:16.262646+0000","last_became_peered":"2022-05-13T23:14:16.262646+0000","last_unstale":"2022-05-13T23:14:16.264851+0000","last_undegraded":"2022-05-13T23:14:16.264851+0000","last_fullsized":"2022-05-13T23:14:16.264851+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_clean_scrub_stamp":"2022-05-13T23:14:15.659362+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T05:38:53.024966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"82'2","reported_seq":13,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-05-13T23:14:16.264403+0000","last_change":"2022-05-13T23:14:15.690434+0000","last_active":"2022-05-13T23:14:16.264403+0000","last_peered":"2022-05-13T23:14:16.264403+0000","last_clean":"2022-05-13T23:14:16.264403+0000","last_became_active":"2022-05-13T23:14:15.690032+0000","last_became_peered":"2022-05-13T23:14:15.690032+0000","last_unstale":"2022-05-13T23:14:16.264403+0000","last_undegraded":"2022-05-13T23:14:16.264403+0000","last_fullsized":"2022-05-13T23:14:16.264403+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:14.344206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:14.344206+0000","last_clean_scrub_stamp":"2022-05-13T23:14:14.344206+0000","objects_scrubbed":0,"log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T06:39:08.251311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":10,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-05-13T23:14:16.265645+0000","last_change":"2022-05-13T23:14:16.264791+0000","last_active":"2022-05-13T23:14:16.265645+0000","last_peered":"2022-05-13T23:14:16.265645+0000","last_clean":"2022-05-13T23:14:16.265645+0000","last_became_active":"2022-05-13T23:14:16.264461+0000","last_became_peered":"2022-05-13T23:14:16.264461+0000","last_unstale":"2022-05-13T23:14:16.265645+0000","last_undegraded":"2022-05-13T23:14:16.265645+0000","last_fullsized":"2022-05-13T23:14:16.265645+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_clean_scrub_stamp":"2022-05-13T23:14:15.659362+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T03:02:04.280220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":10,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-05-13T23:14:16.261071+0000","last_change":"2022-05-13T23:14:16.260468+0000","last_active":"2022-05-13T23:14:16.261071+0000","last_peered":"2022-05-13T23:14:16.261071+0000","last_clean":"2022-05-13T23:14:16.261071+0000","last_became_active":"2022-05-13T23:14:16.260211+0000","last_became_peered":"2022-05-13T23:14:16.260211+0000","last_unstale":"2022-05-13T23:14:16.261071+0000","last_undegraded":"2022-05-13T23:14:16.261071+0000","last_fullsized":"2022-05-13T23:14:16.261071+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_clean_scrub_stamp":"2022-05-13T23:14:15.659362+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T05:26:57.769099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":10,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-05-13T23:14:16.254999+0000","last_change":"2022-05-13T23:14:16.253997+0000","last_active":"2022-05-13T23:14:16.254999+0000","last_peered":"2022-05-13T23:14:16.254999+0000","last_clean":"2022-05-13T23:14:16.254999+0000","last_became_active":"2022-05-13T23:14:16.253685+0000","last_became_peered":"2022-05-13T23:14:16.253685+0000","last_unstale":"2022-05-13T23:14:16.254999+0000","last_undegraded":"2022-05-13T23:14:16.254999+0000","last_fullsized":"2022-05-13T23:14:16.254999+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_clean_scrub_stamp":"2022-05-13T23:14:15.659362+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T07:42:26.765244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":10,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-05-13T23:14:16.266256+0000","last_change":"2022-05-13T23:14:16.265207+0000","last_active":"2022-05-13T23:14:16.266256+0000","last_peered":"2022-05-13T23:14:16.266256+0000","last_clean":"2022-05-13T23:14:16.266256+0000","last_became_active":"2022-05-13T23:14:16.264881+0000","last_became_peered":"2022-05-13T23:14:16.264881+0000","last_unstale":"2022-05-13T23:14:16.266256+0000","last_undegraded":"2022-05-13T23:14:16.266256+0000","last_fullsized":"2022-05-13T23:14:16.266256+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_clean_scrub_stamp":"2022-05-13T23:14:15.659362+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T04:01:48.894233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":10,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-05-13T23:14:16.273903+0000","last_change":"2022-05-13T23:14:16.264286+0000","last_active":"2022-05-13T23:14:16.273903+0000","last_peered":"2022-05-13T23:14:16.273903+0000","last_clean":"2022-05-13T23:14:16.273903+0000","last_became_active":"2022-05-13T23:14:16.263978+0000","last_became_peered":"2022-05-13T23:14:16.263978+0000","last_unstale":"2022-05-13T23:14:16.273903+0000","last_undegraded":"2022-05-13T23:14:16.273903+0000","last_fullsized":"2022-05-13T23:14:16.273903+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_clean_scrub_stamp":"2022-05-13T23:14:15.659362+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T03:11:10.799827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":10,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-05-13T23:14:16.268810+0000","last_change":"2022-05-13T23:14:16.265421+0000","last_active":"2022-05-13T23:14:16.268810+0000","last_peered":"2022-05-13T23:14:16.268810+0000","last_clean":"2022-05-13T23:14:16.268810+0000","last_became_active":"2022-05-13T23:14:16.265057+0000","last_became_peered":"2022-05-13T23:14:16.265057+0000","last_unstale":"2022-05-13T23:14:16.268810+0000","last_undegraded":"2022-05-13T23:14:16.268810+0000","last_fullsized":"2022-05-13T23:14:16.268810+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_clean_scrub_stamp":"2022-05-13T23:14:15.659362+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T07:45:53.177341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":10,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-05-13T23:14:16.269637+0000","last_change":"2022-05-13T23:14:16.266317+0000","last_active":"2022-05-13T23:14:16.269637+0000","last_peered":"2022-05-13T23:14:16.269637+0000","last_clean":"2022-05-13T23:14:16.269637+0000","last_became_active":"2022-05-13T23:14:16.265959+0000","last_became_peered":"2022-05-13T23:14:16.265959+0000","last_unstale":"2022-05-13T23:14:16.269637+0000","last_undegraded":"2022-05-13T23:14:16.269637+0000","last_fullsized":"2022-05-13T23:14:16.269637+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_clean_scrub_stamp":"2022-05-13T23:14:15.659362+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T03:37:42.277033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":10,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-05-13T23:14:16.255848+0000","last_change":"2022-05-13T23:14:16.254350+0000","last_active":"2022-05-13T23:14:16.255848+0000","last_peered":"2022-05-13T23:14:16.255848+0000","last_clean":"2022-05-13T23:14:16.255848+0000","last_became_active":"2022-05-13T23:14:16.253940+0000","last_became_peered":"2022-05-13T23:14:16.253940+0000","last_unstale":"2022-05-13T23:14:16.255848+0000","last_undegraded":"2022-05-13T23:14:16.255848+0000","last_fullsized":"2022-05-13T23:14:16.255848+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_clean_scrub_stamp":"2022-05-13T23:14:15.659362+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T10:08:56.052316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":10,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-05-13T23:14:16.267243+0000","last_change":"2022-05-13T23:14:16.263153+0000","last_active":"2022-05-13T23:14:16.267243+0000","last_peered":"2022-05-13T23:14:16.267243+0000","last_clean":"2022-05-13T23:14:16.267243+0000","last_became_active":"2022-05-13T23:14:16.262652+0000","last_became_peered":"2022-05-13T23:14:16.262652+0000","last_unstale":"2022-05-13T23:14:16.267243+0000","last_undegraded":"2022-05-13T23:14:16.267243+0000","last_fullsized":"2022-05-13T23:14:16.267243+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_clean_scrub_stamp":"2022-05-13T23:14:15.659362+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T07:53:46.499880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":10,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-05-13T23:14:16.271182+0000","last_change":"2022-05-13T23:14:16.270358+0000","last_active":"2022-05-13T23:14:16.271182+0000","last_peered":"2022-05-13T23:14:16.271182+0000","last_clean":"2022-05-13T23:14:16.271182+0000","last_became_active":"2022-05-13T23:14:16.270099+0000","last_became_peered":"2022-05-13T23:14:16.270099+0000","last_unstale":"2022-05-13T23:14:16.271182+0000","last_undegraded":"2022-05-13T23:14:16.271182+0000","last_fullsized":"2022-05-13T23:14:16.271182+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_clean_scrub_stamp":"2022-05-13T23:14:15.659362+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T00:42:07.782569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":10,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-05-13T23:14:16.253696+0000","last_change":"2022-05-13T23:14:16.251188+0000","last_active":"2022-05-13T23:14:16.253696+0000","last_peered":"2022-05-13T23:14:16.253696+0000","last_clean":"2022-05-13T23:14:16.253696+0000","last_became_active":"2022-05-13T23:14:16.250816+0000","last_became_peered":"2022-05-13T23:14:16.250816+0000","last_unstale":"2022-05-13T23:14:16.253696+0000","last_undegraded":"2022-05-13T23:14:16.253696+0000","last_fullsized":"2022-05-13T23:14:16.253696+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_clean_scrub_stamp":"2022-05-13T23:14:15.659362+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T00:58:14.898104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":10,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-05-13T23:14:16.265927+0000","last_change":"2022-05-13T23:14:16.264125+0000","last_active":"2022-05-13T23:14:16.265927+0000","last_peered":"2022-05-13T23:14:16.265927+0000","last_clean":"2022-05-13T23:14:16.265927+0000","last_became_active":"2022-05-13T23:14:16.263806+0000","last_became_peered":"2022-05-13T23:14:16.263806+0000","last_unstale":"2022-05-13T23:14:16.265927+0000","last_undegraded":"2022-05-13T23:14:16.265927+0000","last_fullsized":"2022-05-13T23:14:16.265927+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_clean_scrub_stamp":"2022-05-13T23:14:15.659362+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T09:26:04.559765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":10,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-05-13T23:14:16.268902+0000","last_change":"2022-05-13T23:14:16.266991+0000","last_active":"2022-05-13T23:14:16.268902+0000","last_peered":"2022-05-13T23:14:16.268902+0000","last_clean":"2022-05-13T23:14:16.268902+0000","last_became_active":"2022-05-13T23:14:16.265551+0000","last_became_peered":"2022-05-13T23:14:16.265551+0000","last_unstale":"2022-05-13T23:14:16.268902+0000","last_undegraded":"2022-05-13T23:14:16.268902+0000","last_fullsized":"2022-05-13T23:14:16.268902+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_clean_scrub_stamp":"2022-05-13T23:14:15.659362+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T00:40:56.664387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":10,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-05-13T23:14:16.269805+0000","last_change":"2022-05-13T23:14:16.267302+0000","last_active":"2022-05-13T23:14:16.269805+0000","last_peered":"2022-05-13T23:14:16.269805+0000","last_clean":"2022-05-13T23:14:16.269805+0000","last_became_active":"2022-05-13T23:14:16.266151+0000","last_became_peered":"2022-05-13T23:14:16.266151+0000","last_unstale":"2022-05-13T23:14:16.269805+0000","last_undegraded":"2022-05-13T23:14:16.269805+0000","last_fullsized":"2022-05-13T23:14:16.269805+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_clean_scrub_stamp":"2022-05-13T23:14:15.659362+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T07:30:04.546337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1057856,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":3194880,"data_stored":3173568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":4302,"num_objects":49,"num_object_clones":0,"num_object_copies":147,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":49,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":107,"num_write_kb":27,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":32,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":221184,"data_stored":13038,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":93,"ondisk_log_size":93,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":78,"seq":335007449102,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":5880,"kb_used_data":880,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":93739272,"statfs":{"total":95995035648,"available":95989014528,"internally_reserved":0,"allocated":901120,"data_stored":425930,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri May 13 23:14:21 2022","interfaces":[{"interface":"back","average":{"1min":0.60299999999999998,"5min":0.60299999999999998,"15min":0.60299999999999998},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.84699999999999998,"5min":0.84699999999999998,"15min":0.84699999999999998},"last":0.749},{"interface":"front","average":{"1min":0.64200000000000002,"5min":0.64200000000000002,"15min":0.64200000000000002},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":0.81599999999999995,"5min":0.81599999999999995,"15min":0.81599999999999995},"last":0.745}]},{"osd":1,"last update":"Fri May 13 23:14:21 2022","interfaces":[{"interface":"back","average":{"1min":0.59699999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":0.81499999999999995,"5min":0.81499999999999995,"15min":0.81499999999999995},"last":0.71499999999999997},{"interface":"front","average":{"1min":0.58899999999999997,"5min":0.58899999999999997,"15min":0.58899999999999997},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.89500000000000002,"5min":0.89500000000000002,"15min":0.89500000000000002},"last":0.56399999999999995}]},{"osd":2,"last update":"Fri May 13 23:14:21 2022","interfaces":[{"interface":"back","average":{"1min":0.46500000000000002,"5min":0.46500000000000002,"15min":0.46500000000000002},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.76900000000000002,"5min":0.76900000000000002,"15min":0.76900000000000002},"last":0.52800000000000002},{"interface":"front","average":{"1min":0.46800000000000003,"5min":0.46800000000000003,"15min":0.46800000000000003},"min":{"1min":0.317,"5min":0.317,"15min":0.317},"max":{"1min":0.71299999999999997,"5min":0.71299999999999997,"15min":0.71299999999999997},"last":0.503}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42999999999999999}]},{"osd":4,"last update":"Fri May 13 23:14:21 2022","interfaces":[{"interface":"back","average":{"1min":0.622,"5min":0.622,"15min":0.622},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":1.0289999999999999,"5min":1.0289999999999999,"15min":1.0289999999999999},"last":0.64100000000000001},{"interface":"front","average":{"1min":0.63400000000000001,"5min":0.63400000000000001,"15min":0.63400000000000001},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":0.93700000000000006,"5min":0.93700000000000006,"15min":0.93700000000000006},"last":0.65600000000000003}]},{"osd":5,"last update":"Fri May 13 23:14:21 2022","interfaces":[{"interface":"back","average":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.59099999999999997},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":1.3799999999999999,"5min":1.3799999999999999,"15min":1.3799999999999999},"last":0.69599999999999995},{"interface":"front","average":{"1min":0.60299999999999998,"5min":0.60299999999999998,"15min":0.60299999999999998},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.88,"5min":0.88,"15min":0.88},"last":0.78800000000000003}]},{"osd":6,"last update":"Fri May 13 23:14:21 2022","interfaces":[{"interface":"back","average":{"1min":0.73999999999999999,"5min":0.73999999999999999,"15min":0.73999999999999999},"min":{"1min":0.48499999999999999,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":1.0669999999999999,"5min":1.0669999999999999,"15min":1.0669999999999999},"last":0.92200000000000004},{"interface":"front","average":{"1min":0.79200000000000004,"5min":0.79200000000000004,"15min":0.79200000000000004},"min":{"1min":0.49099999999999999,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":1.655,"5min":1.655,"15min":1.655},"last":0.86599999999999999}]},{"osd":7,"last update":"Fri May 13 23:14:21 2022","interfaces":[{"interface":"back","average":{"1min":0.72299999999999998,"5min":0.72299999999999998,"15min":0.72299999999999998},"min":{"1min":0.502,"5min":0.502,"15min":0.502},"max":{"1min":1.107,"5min":1.107,"15min":1.107},"last":0.84999999999999998},{"interface":"front","average":{"1min":0.745,"5min":0.745,"15min":0.745},"min":{"1min":0.46999999999999997,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":1.3620000000000001,"5min":1.3620000000000001,"15min":1.3620000000000001},"last":0.88700000000000001}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82299999999999995}]},{"osd":9,"last update":"Fri May 13 23:14:21 2022","interfaces":[{"interface":"back","average":{"1min":0.83699999999999997,"5min":0.83699999999999997,"15min":0.83699999999999997},"min":{"1min":0.57099999999999995,"5min":0.57099999999999995,"15min":0.57099999999999995},"max":{"1min":1.7729999999999999,"5min":1.7729999999999999,"15min":1.7729999999999999},"last":0.90600000000000003},{"interface":"front","average":{"1min":0.79500000000000004,"5min":0.79500000000000004,"15min":0.79500000000000004},"min":{"1min":0.58099999999999996,"5min":0.58099999999999996,"15min":0.58099999999999996},"max":{"1min":1.3220000000000001,"5min":1.3220000000000001,"15min":1.3220000000000001},"last":0.876}]},{"osd":10,"last update":"Fri May 13 23:14:21 2022","interfaces":[{"interface":"back","average":{"1min":0.75700000000000001,"5min":0.75700000000000001,"15min":0.75700000000000001},"min":{"1min":0.54100000000000004,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":1.26,"5min":1.26,"15min":1.26},"last":0.94999999999999996},{"interface":"front","average":{"1min":0.77600000000000002,"5min":0.77600000000000002,"15min":0.77600000000000002},"min":{"1min":0.51900000000000002,"5min":0.51900000000000002,"15min":0.51900000000000002},"max":{"1min":1.7350000000000001,"5min":1.7350000000000001,"15min":1.7350000000000001},"last":0.84099999999999997}]}]},{"osd":10,"up_from":71,"seq":304942678034,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6600,"kb_used_data":1408,"kb_used_omap":0,"kb_used_meta":5184,"kb_avail":93738552,"statfs":{"total":95995035648,"available":95988277248,"internally_reserved":0,"allocated":1441792,"data_stored":955149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5308416},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.755}]},{"osd":1,"last update":"Fri May 13 23:14:05 2022","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.72899999999999998,"5min":0.72899999999999998,"15min":0.72899999999999998},"last":0.40600000000000003},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.57999999999999996,"15min":0.57999999999999996},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.83099999999999996,"5min":0.83099999999999996,"15min":0.83099999999999996},"last":0.42299999999999999}]},{"osd":2,"last update":"Fri May 13 23:14:05 2022","interfaces":[{"interface":"back","average":{"1min":0.69499999999999995,"5min":0.69499999999999995,"15min":0.69499999999999995},"min":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":1.3280000000000001,"5min":1.3280000000000001,"15min":1.3280000000000001},"last":0.52600000000000002},{"interface":"front","average":{"1min":0.70799999999999996,"5min":0.70799999999999996,"15min":0.70799999999999996},"min":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":0.88800000000000001,"5min":0.88800000000000001,"15min":0.88800000000000001},"last":0.64800000000000002}]},{"osd":3,"last update":"Fri May 13 23:14:05 2022","interfaces":[{"interface":"back","average":{"1min":0.73999999999999999,"5min":0.73999999999999999,"15min":0.73999999999999999},"min":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"max":{"1min":0.90100000000000002,"5min":0.90100000000000002,"15min":0.90100000000000002},"last":0.59099999999999997},{"interface":"front","average":{"1min":0.77000000000000002,"5min":0.77000000000000002,"15min":0.77000000000000002},"min":{"1min":0.60199999999999998,"5min":0.60199999999999998,"15min":0.60199999999999998},"max":{"1min":0.93500000000000005,"5min":0.93500000000000005,"15min":0.93500000000000005},"last":0.55800000000000005}]},{"osd":4,"last update":"Fri May 13 23:14:05 2022","interfaces":[{"interface":"back","average":{"1min":0.68000000000000005,"5min":0.68000000000000005,"15min":0.68000000000000005},"min":{"1min":0.432,"5min":0.432,"15min":0.432},"max":{"1min":0.97699999999999998,"5min":0.97699999999999998,"15min":0.97699999999999998},"last":0.51800000000000002},{"interface":"front","average":{"1min":0.65200000000000002,"5min":0.65200000000000002,"15min":0.65200000000000002},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":1.0720000000000001,"5min":1.0720000000000001,"15min":1.0720000000000001},"last":0.46600000000000003}]},{"osd":5,"last update":"Fri May 13 23:14:05 2022","interfaces":[{"interface":"back","average":{"1min":0.78400000000000003,"5min":0.78400000000000003,"15min":0.78400000000000003},"min":{"1min":0.496,"5min":0.496,"15min":0.496},"max":{"1min":1.4299999999999999,"5min":1.4299999999999999,"15min":1.4299999999999999},"last":0.58099999999999996},{"interface":"front","average":{"1min":0.79800000000000004,"5min":0.79800000000000004,"15min":0.79800000000000004},"min":{"1min":0.55400000000000005,"5min":0.55400000000000005,"15min":0.55400000000000005},"max":{"1min":1.0609999999999999,"5min":1.0609999999999999,"15min":1.0609999999999999},"last":0.68799999999999994}]},{"osd":6,"last update":"Fri May 13 23:14:05 2022","interfaces":[{"interface":"back","average":{"1min":0.88700000000000001,"5min":0.88700000000000001,"15min":0.88700000000000001},"min":{"1min":0.64600000000000002,"5min":0.64600000000000002,"15min":0.64600000000000002},"max":{"1min":1.157,"5min":1.157,"15min":1.157},"last":0.54200000000000004},{"interface":"front","average":{"1min":0.88500000000000001,"5min":0.88500000000000001,"15min":0.88500000000000001},"min":{"1min":0.60199999999999998,"5min":0.60199999999999998,"15min":0.60199999999999998},"max":{"1min":1.216,"5min":1.216,"15min":1.216},"last":0.66100000000000003}]},{"osd":7,"last update":"Fri May 13 23:14:05 2022","interfaces":[{"interface":"back","average":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":1.585,"5min":1.585,"15min":1.585},"last":0.496},{"interface":"front","average":{"1min":0.80300000000000005,"5min":0.80300000000000005,"15min":0.80300000000000005},"min":{"1min":0.47999999999999998,"5min":0.47999999999999998,"15min":0.47999999999999998},"max":{"1min":1.0469999999999999,"5min":1.0469999999999999,"15min":1.0469999999999999},"last":0.502}]},{"osd":8,"last update":"Fri May 13 23:14:05 2022","interfaces":[{"interface":"back","average":{"1min":0.86399999999999999,"5min":0.86399999999999999,"15min":0.86399999999999999},"min":{"1min":0.65900000000000003,"5min":0.65900000000000003,"15min":0.65900000000000003},"max":{"1min":1.163,"5min":1.163,"15min":1.163},"last":0.73299999999999998},{"interface":"front","average":{"1min":0.88200000000000001,"5min":0.88200000000000001,"15min":0.88200000000000001},"min":{"1min":0.51400000000000001,"5min":0.51400000000000001,"15min":0.51400000000000001},"max":{"1min":1.476,"5min":1.476,"15min":1.476},"last":0.71199999999999997}]},{"osd":9,"last update":"Fri May 13 23:14:05 2022","interfaces":[{"interface":"back","average":{"1min":0.94999999999999996,"5min":0.94999999999999996,"15min":0.94999999999999996},"min":{"1min":0.68999999999999995,"5min":0.68999999999999995,"15min":0.68999999999999995},"max":{"1min":1.369,"5min":1.369,"15min":1.369},"last":0.77000000000000002},{"interface":"front","average":{"1min":0.96799999999999997,"5min":0.96799999999999997,"15min":0.96799999999999997},"min":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":1.819,"5min":1.819,"15min":1.819},"last":0.67300000000000004}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72099999999999997}]}]},{"osd":9,"up_from":66,"seq":283467841558,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6016,"kb_used_data":888,"kb_used_omap":0,"kb_used_meta":5120,"kb_avail":93739136,"statfs":{"total":95995035648,"available":95988875264,"internally_reserved":0,"allocated":909312,"data_stored":425604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5242880},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri May 13 23:13:37 2022","interfaces":[{"interface":"back","average":{"1min":0.65200000000000002,"5min":0.65200000000000002,"15min":0.65200000000000002},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":1.024,"5min":1.024,"15min":1.024},"last":0.41199999999999998},{"interface":"front","average":{"1min":0.72299999999999998,"5min":0.72299999999999998,"15min":0.72299999999999998},"min":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":1.044,"5min":1.044,"15min":1.044},"last":0.45800000000000002}]},{"osd":1,"last update":"Fri May 13 23:13:37 2022","interfaces":[{"interface":"back","average":{"1min":0.755,"5min":0.755,"15min":0.755},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":1.3029999999999999,"5min":1.3029999999999999,"15min":1.3029999999999999},"last":0.432},{"interface":"front","average":{"1min":0.92000000000000004,"5min":0.92000000000000004,"15min":0.92000000000000004},"min":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":1.498,"5min":1.498,"15min":1.498},"last":0.72899999999999998}]},{"osd":2,"last update":"Fri May 13 23:13:37 2022","interfaces":[{"interface":"back","average":{"1min":0.89900000000000002,"5min":0.89900000000000002,"15min":0.89900000000000002},"min":{"1min":0.58699999999999997,"5min":0.58699999999999997,"15min":0.58699999999999997},"max":{"1min":1.4419999999999999,"5min":1.4419999999999999,"15min":1.4419999999999999},"last":0.83899999999999997},{"interface":"front","average":{"1min":0.82699999999999996,"5min":0.82699999999999996,"15min":0.82699999999999996},"min":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"max":{"1min":1.5309999999999999,"5min":1.5309999999999999,"15min":1.5309999999999999},"last":0.75}]},{"osd":3,"last update":"Fri May 13 23:13:37 2022","interfaces":[{"interface":"back","average":{"1min":0.878,"5min":0.878,"15min":0.878},"min":{"1min":0.51200000000000001,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":1.2410000000000001,"5min":1.2410000000000001,"15min":1.2410000000000001},"last":0.61199999999999999},{"interface":"front","average":{"1min":1.002,"5min":1.002,"15min":1.002},"min":{"1min":0.622,"5min":0.622,"15min":0.622},"max":{"1min":2.2069999999999999,"5min":2.2069999999999999,"15min":2.2069999999999999},"last":0.78200000000000003}]},{"osd":4,"last update":"Fri May 13 23:13:37 2022","interfaces":[{"interface":"back","average":{"1min":0.93000000000000005,"5min":0.93000000000000005,"15min":0.93000000000000005},"min":{"1min":0.47699999999999998,"5min":0.47699999999999998,"15min":0.47699999999999998},"max":{"1min":1.5600000000000001,"5min":1.5600000000000001,"15min":1.5600000000000001},"last":0.67800000000000005},{"interface":"front","average":{"1min":0.95799999999999996,"5min":0.95799999999999996,"15min":0.95799999999999996},"min":{"1min":0.58899999999999997,"5min":0.58899999999999997,"15min":0.58899999999999997},"max":{"1min":1.3320000000000001,"5min":1.3320000000000001,"15min":1.3320000000000001},"last":0.68700000000000006}]},{"osd":5,"last update":"Fri May 13 23:13:37 2022","interfaces":[{"interface":"back","average":{"1min":0.97099999999999997,"5min":0.97099999999999997,"15min":0.97099999999999997},"min":{"1min":0.56299999999999994,"5min":0.56299999999999994,"15min":0.56299999999999994},"max":{"1min":1.593,"5min":1.593,"15min":1.593},"last":0.85699999999999998},{"interface":"front","average":{"1min":1.02,"5min":1.02,"15min":1.02},"min":{"1min":0.68600000000000005,"5min":0.68600000000000005,"15min":0.68600000000000005},"max":{"1min":1.3700000000000001,"5min":1.3700000000000001,"15min":1.3700000000000001},"last":0.84799999999999998}]},{"osd":6,"last update":"Fri May 13 23:13:37 2022","interfaces":[{"interface":"back","average":{"1min":1.1399999999999999,"5min":1.1399999999999999,"15min":1.1399999999999999},"min":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"max":{"1min":2.254,"5min":2.254,"15min":2.254},"last":0.88100000000000001},{"interface":"front","average":{"1min":1.0609999999999999,"5min":1.0609999999999999,"15min":1.0609999999999999},"min":{"1min":0.70599999999999996,"5min":0.70599999999999996,"15min":0.70599999999999996},"max":{"1min":1.649,"5min":1.649,"15min":1.649},"last":0.82199999999999995}]},{"osd":7,"last update":"Fri May 13 23:13:37 2022","interfaces":[{"interface":"back","average":{"1min":1.119,"5min":1.119,"15min":1.119},"min":{"1min":0.63,"5min":0.63,"15min":0.63},"max":{"1min":2.101,"5min":2.101,"15min":2.101},"last":0.77300000000000002},{"interface":"front","average":{"1min":1.1240000000000001,"5min":1.1240000000000001,"15min":1.1240000000000001},"min":{"1min":0.69599999999999995,"5min":0.69599999999999995,"15min":0.69599999999999995},"max":{"1min":1.895,"5min":1.895,"15min":1.895},"last":0.92200000000000004}]},{"osd":8,"last update":"Fri May 13 23:13:37 2022","interfaces":[{"interface":"back","average":{"1min":1.069,"5min":1.069,"15min":1.069},"min":{"1min":0.59899999999999998,"5min":0.59899999999999998,"15min":0.59899999999999998},"max":{"1min":1.6140000000000001,"5min":1.6140000000000001,"15min":1.6140000000000001},"last":0.88800000000000001},{"interface":"front","average":{"1min":1.1419999999999999,"5min":1.1419999999999999,"15min":1.1419999999999999},"min":{"1min":0.71599999999999997,"5min":0.71599999999999997,"15min":0.71599999999999997},"max":{"1min":2.1789999999999998,"5min":2.1789999999999998,"15min":2.1789999999999998},"last":0.81200000000000006}]},{"osd":10,"last update":"Fri May 13 23:14:02 2022","interfaces":[{"interface":"back","average":{"1min":1.1040000000000001,"5min":1.1040000000000001,"15min":1.1040000000000001},"min":{"1min":0.66800000000000004,"5min":0.66800000000000004,"15min":0.66800000000000004},"max":{"1min":1.7829999999999999,"5min":1.7829999999999999,"15min":1.7829999999999999},"last":0.91200000000000003},{"interface":"front","average":{"1min":1.1319999999999999,"5min":1.1319999999999999,"15min":1.1319999999999999},"min":{"1min":0.73099999999999998,"5min":0.73099999999999998,"15min":0.73099999999999998},"max":{"1min":1.748,"5min":1.748,"15min":1.748},"last":0.95599999999999996}]}]},{"osd":8,"up_from":58,"seq":249108103198,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6208,"kb_used_data":888,"kb_used_omap":0,"kb_used_meta":5312,"kb_avail":93738944,"statfs":{"total":95995035648,"available":95988678656,"internally_reserved":0,"allocated":909312,"data_stored":425524,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5439488},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri May 13 23:14:08 2022","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.60399999999999998,"15min":0.60799999999999998},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.88900000000000001,"5min":0.88900000000000001,"15min":0.88900000000000001},"last":0.60799999999999998},{"interface":"front","average":{"1min":0.61699999999999999,"5min":0.59199999999999997,"15min":0.58799999999999997},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":1.137,"5min":1.137,"15min":1.137},"last":0.52100000000000002}]},{"osd":1,"last update":"Fri May 13 23:14:08 2022","interfaces":[{"interface":"back","average":{"1min":0.53900000000000003,"5min":0.56699999999999995,"15min":0.57099999999999995},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.67200000000000004,"5min":0.78900000000000003,"15min":0.78900000000000003},"last":0.64400000000000002},{"interface":"front","average":{"1min":0.57299999999999995,"5min":0.60199999999999998,"15min":0.60699999999999998},"min":{"1min":0.38900000000000001,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.72599999999999998,"5min":0.80700000000000005,"15min":0.80700000000000005},"last":0.71099999999999997}]},{"osd":2,"last update":"Fri May 13 23:14:08 2022","interfaces":[{"interface":"back","average":{"1min":0.44,"5min":0.45600000000000002,"15min":0.45900000000000002},"min":{"1min":0.24199999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.77400000000000002,"5min":0.77400000000000002,"15min":0.77400000000000002},"last":0.46800000000000003},{"interface":"front","average":{"1min":0.443,"5min":0.45600000000000002,"15min":0.45800000000000002},"min":{"1min":0.27800000000000002,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.79100000000000004,"5min":0.79100000000000004,"15min":0.79100000000000004},"last":0.48699999999999999}]},{"osd":3,"last update":"Fri May 13 23:14:08 2022","interfaces":[{"interface":"back","average":{"1min":0.626,"5min":0.64800000000000002,"15min":0.65200000000000002},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.88400000000000001,"5min":0.88400000000000001,"15min":0.88400000000000001},"last":0.63300000000000001},{"interface":"front","average":{"1min":0.63300000000000001,"5min":0.64300000000000002,"15min":0.64500000000000002},"min":{"1min":0.439,"5min":0.439,"15min":0.439},"max":{"1min":0.82899999999999996,"5min":0.83299999999999996,"15min":0.83299999999999996},"last":0.55200000000000005}]},{"osd":4,"last update":"Fri May 13 23:14:08 2022","interfaces":[{"interface":"back","average":{"1min":0.61099999999999999,"5min":0.621,"15min":0.623},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":1.573,"5min":1.573,"15min":1.573},"last":0.70499999999999996},{"interface":"front","average":{"1min":0.57299999999999995,"5min":0.61899999999999999,"15min":0.627},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":0.746,"5min":0.90500000000000003,"15min":0.90500000000000003},"last":0.69699999999999995}]},{"osd":5,"last update":"Fri May 13 23:14:08 2022","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.57399999999999995,"15min":0.58299999999999996},"min":{"1min":0.22800000000000001,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":0.85199999999999998,"5min":0.875,"15min":0.875},"last":0.66600000000000004},{"interface":"front","average":{"1min":0.498,"5min":0.56299999999999994,"15min":0.57399999999999995},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.86199999999999999,"5min":0.88200000000000001,"15min":0.88200000000000001},"last":0.67700000000000005}]},{"osd":6,"last update":"Fri May 13 23:14:08 2022","interfaces":[{"interface":"back","average":{"1min":0.70299999999999996,"5min":0.74199999999999999,"15min":0.748},"min":{"1min":0.44400000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":0.873,"5min":0.94099999999999995,"15min":0.94099999999999995},"last":0.75700000000000001},{"interface":"front","average":{"1min":0.72099999999999997,"5min":0.754,"15min":0.76000000000000001},"min":{"1min":0.51700000000000002,"5min":0.504,"15min":0.504},"max":{"1min":0.91700000000000004,"5min":0.95699999999999996,"15min":0.95699999999999996},"last":0.88600000000000001}]},{"osd":7,"last update":"Fri May 13 23:14:08 2022","interfaces":[{"interface":"back","average":{"1min":0.67100000000000004,"5min":0.69999999999999996,"15min":0.70499999999999996},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.94799999999999995,"5min":0.95799999999999996,"15min":0.95799999999999996},"last":0.85999999999999999},{"interface":"front","average":{"1min":0.70399999999999996,"5min":0.69999999999999996,"15min":0.69999999999999996},"min":{"1min":0.49099999999999999,"5min":0.437,"15min":0.437},"max":{"1min":1.609,"5min":1.609,"15min":1.609},"last":0.76700000000000002}]},{"osd":9,"last update":"Fri May 13 23:13:38 2022","interfaces":[{"interface":"back","average":{"1min":0.76500000000000001,"5min":0.76500000000000001,"15min":0.76500000000000001},"min":{"1min":0.51500000000000001,"5min":0.51500000000000001,"15min":0.51500000000000001},"max":{"1min":0.999,"5min":0.999,"15min":0.999},"last":0.84699999999999998},{"interface":"front","average":{"1min":0.80100000000000005,"5min":0.80100000000000005,"15min":0.80100000000000005},"min":{"1min":0.59899999999999998,"5min":0.59899999999999998,"15min":0.59899999999999998},"max":{"1min":0.996,"5min":0.996,"15min":0.996},"last":0.80100000000000005}]},{"osd":10,"last update":"Fri May 13 23:14:02 2022","interfaces":[{"interface":"back","average":{"1min":0.72699999999999998,"5min":0.72699999999999998,"15min":0.72699999999999998},"min":{"1min":0.45800000000000002,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":0.97799999999999998,"5min":0.97799999999999998,"15min":0.97799999999999998},"last":0.82599999999999996},{"interface":"front","average":{"1min":0.69699999999999995,"5min":0.69699999999999995,"15min":0.69699999999999995},"min":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":0.998,"5min":0.998,"15min":0.998},"last":0.79200000000000004}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77800000000000002}]}]},{"osd":7,"up_from":51,"seq":219043332128,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6776,"kb_used_data":1392,"kb_used_omap":0,"kb_used_meta":5376,"kb_avail":93738376,"statfs":{"total":95995035648,"available":95988097024,"internally_reserved":0,"allocated":1425408,"data_stored":954296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5505024},"hb_peers":[0,1,2,3,4,5,6,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri May 13 23:13:48 2022","interfaces":[{"interface":"back","average":{"1min":0.64400000000000002,"5min":0.56000000000000005,"15min":0.54600000000000004},"min":{"1min":0.48399999999999999,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":0.82599999999999996,"5min":0.82599999999999996,"15min":0.82599999999999996},"last":0.63600000000000001},{"interface":"front","average":{"1min":0.60899999999999999,"5min":0.58799999999999997,"15min":0.58399999999999996},"min":{"1min":0.42699999999999999,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":0.78800000000000003,"5min":0.86599999999999999,"15min":0.86599999999999999},"last":0.51900000000000002}]},{"osd":1,"last update":"Fri May 13 23:13:48 2022","interfaces":[{"interface":"back","average":{"1min":0.47499999999999998,"5min":0.44700000000000001,"15min":0.442},"min":{"1min":0.29999999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.79800000000000004,"5min":0.79800000000000004,"15min":0.79800000000000004},"last":0.42299999999999999},{"interface":"front","average":{"1min":0.49299999999999999,"5min":0.47299999999999998,"15min":0.46999999999999997},"min":{"1min":0.317,"5min":0.317,"15min":0.317},"max":{"1min":0.88800000000000001,"5min":0.88800000000000001,"15min":0.88800000000000001},"last":0.46500000000000002}]},{"osd":2,"last update":"Fri May 13 23:13:48 2022","interfaces":[{"interface":"back","average":{"1min":0.63100000000000001,"5min":0.63500000000000001,"15min":0.63600000000000001},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.83799999999999997,"5min":0.88800000000000001,"15min":0.88800000000000001},"last":0.55400000000000005},{"interface":"front","average":{"1min":0.60699999999999998,"5min":0.60299999999999998,"15min":0.60299999999999998},"min":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":0.78600000000000003,"5min":0.81599999999999995,"15min":0.81599999999999995},"last":0.50900000000000001}]},{"osd":3,"last update":"Fri May 13 23:13:48 2022","interfaces":[{"interface":"back","average":{"1min":0.66300000000000003,"5min":0.63500000000000001,"15min":0.63},"min":{"1min":0.50700000000000001,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":0.92200000000000004,"5min":0.92200000000000004,"15min":0.92200000000000004},"last":0.54200000000000004},{"interface":"front","average":{"1min":0.69399999999999995,"5min":0.69699999999999995,"15min":0.69699999999999995},"min":{"1min":0.49199999999999999,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":0.872,"5min":0.872,"15min":0.872},"last":0.56899999999999995}]},{"osd":4,"last update":"Fri May 13 23:13:48 2022","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.58399999999999996,"15min":0.59199999999999997},"min":{"1min":0.315,"5min":0.315,"15min":0.315},"max":{"1min":0.77800000000000002,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.40100000000000002},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.58199999999999996,"15min":0.58799999999999997},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.875,"5min":0.875,"15min":0.875},"last":0.438}]},{"osd":5,"last update":"Fri May 13 23:13:48 2022","interfaces":[{"interface":"back","average":{"1min":0.68100000000000005,"5min":0.72099999999999997,"15min":0.72699999999999998},"min":{"1min":0.47199999999999998,"5min":0.47199999999999998,"15min":0.47199999999999998},"max":{"1min":0.93300000000000005,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.59899999999999998},{"interface":"front","average":{"1min":0.68100000000000005,"5min":0.70099999999999996,"15min":0.70399999999999996},"min":{"1min":0.47999999999999998,"5min":0.47999999999999998,"15min":0.47999999999999998},"max":{"1min":0.93999999999999995,"5min":0.93999999999999995,"15min":0.93999999999999995},"last":0.58899999999999997}]},{"osd":6,"last update":"Fri May 13 23:13:48 2022","interfaces":[{"interface":"back","average":{"1min":0.76300000000000001,"5min":0.76200000000000001,"15min":0.76200000000000001},"min":{"1min":0.52700000000000002,"5min":0.52700000000000002,"15min":0.52700000000000002},"max":{"1min":1.0700000000000001,"5min":1.0700000000000001,"15min":1.0700000000000001},"last":0.61499999999999999},{"interface":"front","average":{"1min":0.81999999999999995,"5min":0.79700000000000004,"15min":0.79300000000000004},"min":{"1min":0.67600000000000005,"5min":0.63200000000000001,"15min":0.63200000000000001},"max":{"1min":1.04,"5min":1.1319999999999999,"15min":1.1319999999999999},"last":0.68999999999999995}]},{"osd":8,"last update":"Fri May 13 23:14:07 2022","interfaces":[{"interface":"back","average":{"1min":0.90600000000000003,"5min":0.77200000000000002,"15min":0.75},"min":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"max":{"1min":4.0739999999999998,"5min":4.0739999999999998,"15min":4.0739999999999998},"last":0.67100000000000004},{"interface":"front","average":{"1min":0.78100000000000003,"5min":0.76500000000000001,"15min":0.76200000000000001},"min":{"1min":0.497,"5min":0.497,"15min":0.497},"max":{"1min":1.046,"5min":1.046,"15min":1.046},"last":0.64700000000000002}]},{"osd":9,"last update":"Fri May 13 23:13:37 2022","interfaces":[{"interface":"back","average":{"1min":0.82699999999999996,"5min":0.82699999999999996,"15min":0.82699999999999996},"min":{"1min":0.57099999999999995,"5min":0.57099999999999995,"15min":0.57099999999999995},"max":{"1min":1.1339999999999999,"5min":1.1339999999999999,"15min":1.1339999999999999},"last":0.77000000000000002},{"interface":"front","average":{"1min":0.82899999999999996,"5min":0.82899999999999996,"15min":0.82899999999999996},"min":{"1min":0.64900000000000002,"5min":0.64900000000000002,"15min":0.64900000000000002},"max":{"1min":1.0409999999999999,"5min":1.0409999999999999,"15min":1.0409999999999999},"last":0.66400000000000003}]},{"osd":10,"last update":"Fri May 13 23:14:00 2022","interfaces":[{"interface":"back","average":{"1min":0.77400000000000002,"5min":0.77400000000000002,"15min":0.77400000000000002},"min":{"1min":0.47599999999999998,"5min":0.47599999999999998,"15min":0.47599999999999998},"max":{"1min":1.093,"5min":1.093,"15min":1.093},"last":0.70899999999999996},{"interface":"front","average":{"1min":0.72899999999999998,"5min":0.72899999999999998,"15min":0.72899999999999998},"min":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"max":{"1min":1.0529999999999999,"5min":1.0529999999999999,"15min":1.0529999999999999},"last":0.627}]}]},{"osd":6,"up_from":46,"seq":197568495654,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6124,"kb_used_data":868,"kb_used_omap":0,"kb_used_meta":5248,"kb_avail":93739028,"statfs":{"total":95995035648,"available":95988764672,"internally_reserved":0,"allocated":888832,"data_stored":424814,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5373952},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri May 13 23:13:27 2022","interfaces":[{"interface":"back","average":{"1min":0.58699999999999997,"5min":0.55100000000000005,"15min":0.54500000000000004},"min":{"1min":0.378,"5min":0.378,"15min":0.378},"max":{"1min":1.004,"5min":1.004,"15min":1.004},"last":0.438},{"interface":"front","average":{"1min":0.62,"5min":0.60099999999999998,"15min":0.59799999999999998},"min":{"1min":0.38800000000000001,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":1.042,"5min":1.042,"15min":1.042},"last":0.70499999999999996}]},{"osd":1,"last update":"Fri May 13 23:13:27 2022","interfaces":[{"interface":"back","average":{"1min":0.80500000000000005,"5min":0.75700000000000001,"15min":0.749},"min":{"1min":0.45700000000000002,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":1.2210000000000001,"5min":1.2210000000000001,"15min":1.2210000000000001},"last":0.61699999999999999},{"interface":"front","average":{"1min":0.72699999999999998,"5min":0.67700000000000005,"15min":0.66900000000000004},"min":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":1.28,"5min":1.28,"15min":1.28},"last":0.63}]},{"osd":2,"last update":"Fri May 13 23:13:27 2022","interfaces":[{"interface":"back","average":{"1min":0.80400000000000005,"5min":0.78800000000000003,"15min":0.78500000000000003},"min":{"1min":0.54700000000000004,"5min":0.54700000000000004,"15min":0.54700000000000004},"max":{"1min":1.1200000000000001,"5min":1.228,"15min":1.228},"last":0.79800000000000004},{"interface":"front","average":{"1min":0.83299999999999996,"5min":0.81599999999999995,"15min":0.81299999999999994},"min":{"1min":0.56599999999999995,"5min":0.53300000000000003,"15min":0.53300000000000003},"max":{"1min":1.298,"5min":1.298,"15min":1.298},"last":0.65400000000000003}]},{"osd":3,"last update":"Fri May 13 23:13:27 2022","interfaces":[{"interface":"back","average":{"1min":0.85699999999999998,"5min":0.81399999999999995,"15min":0.80700000000000005},"min":{"1min":0.60199999999999998,"5min":0.47599999999999998,"15min":0.47599999999999998},"max":{"1min":1.3839999999999999,"5min":1.3839999999999999,"15min":1.3839999999999999},"last":0.68799999999999994},{"interface":"front","average":{"1min":0.76000000000000001,"5min":0.77800000000000002,"15min":0.78100000000000003},"min":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":1.0580000000000001,"5min":1.4119999999999999,"15min":1.4119999999999999},"last":0.68100000000000005}]},{"osd":4,"last update":"Fri May 13 23:13:27 2022","interfaces":[{"interface":"back","average":{"1min":0.82299999999999995,"5min":0.83899999999999997,"15min":0.84199999999999997},"min":{"1min":0.504,"5min":0.504,"15min":0.504},"max":{"1min":1.5640000000000001,"5min":1.5640000000000001,"15min":1.5640000000000001},"last":0.88},{"interface":"front","average":{"1min":0.93200000000000005,"5min":0.90000000000000002,"15min":0.89400000000000002},"min":{"1min":0.60799999999999998,"5min":0.60799999999999998,"15min":0.60799999999999998},"max":{"1min":1.625,"5min":1.625,"15min":1.625},"last":0.78000000000000003}]},{"osd":5,"last update":"Fri May 13 23:13:27 2022","interfaces":[{"interface":"back","average":{"1min":0.84599999999999997,"5min":0.85699999999999998,"15min":0.85899999999999999},"min":{"1min":0.56799999999999995,"5min":0.50600000000000001,"15min":0.50600000000000001},"max":{"1min":1.3640000000000001,"5min":1.5029999999999999,"15min":1.5029999999999999},"last":0.76400000000000001},{"interface":"front","average":{"1min":0.93000000000000005,"5min":0.93700000000000006,"15min":0.93799999999999994},"min":{"1min":0.65900000000000003,"5min":0.65400000000000003,"15min":0.65400000000000003},"max":{"1min":1.3460000000000001,"5min":1.3460000000000001,"15min":1.3460000000000001},"last":0.92800000000000005}]},{"osd":7,"last update":"Fri May 13 23:13:47 2022","interfaces":[{"interface":"back","average":{"1min":1.0669999999999999,"5min":0.89700000000000002,"15min":0.86899999999999999},"min":{"1min":0.78500000000000003,"5min":0.66000000000000003,"15min":0.66000000000000003},"max":{"1min":1.534,"5min":1.534,"15min":1.534},"last":0.83299999999999996},{"interface":"front","average":{"1min":0.97599999999999998,"5min":0.84699999999999998,"15min":0.82499999999999996},"min":{"1min":0.66300000000000003,"5min":0.61299999999999999,"15min":0.61299999999999999},"max":{"1min":1.6180000000000001,"5min":1.6180000000000001,"15min":1.6180000000000001},"last":0.81699999999999995}]},{"osd":8,"last update":"Fri May 13 23:14:12 2022","interfaces":[{"interface":"back","average":{"1min":1.0049999999999999,"5min":0.98499999999999999,"15min":0.98099999999999998},"min":{"1min":0.69999999999999996,"5min":0.69999999999999996,"15min":0.69999999999999996},"max":{"1min":1.484,"5min":1.484,"15min":1.484},"last":0.81299999999999994},{"interface":"front","average":{"1min":0.89600000000000002,"5min":0.94499999999999995,"15min":0.95299999999999996},"min":{"1min":0.68500000000000005,"5min":0.67500000000000004,"15min":0.67500000000000004},"max":{"1min":1.548,"5min":1.548,"15min":1.548},"last":0.86499999999999999}]},{"osd":9,"last update":"Fri May 13 23:13:37 2022","interfaces":[{"interface":"back","average":{"1min":1.071,"5min":1.071,"15min":1.071},"min":{"1min":0.86299999999999999,"5min":0.86299999999999999,"15min":0.86299999999999999},"max":{"1min":1.458,"5min":1.458,"15min":1.458},"last":0.89900000000000002},{"interface":"front","average":{"1min":1.0580000000000001,"5min":1.0580000000000001,"15min":1.0580000000000001},"min":{"1min":0.78900000000000003,"5min":0.78900000000000003,"15min":0.78900000000000003},"max":{"1min":1.401,"5min":1.401,"15min":1.401},"last":0.96499999999999997}]},{"osd":10,"last update":"Fri May 13 23:14:01 2022","interfaces":[{"interface":"back","average":{"1min":1.0529999999999999,"5min":1.0529999999999999,"15min":1.0529999999999999},"min":{"1min":0.77800000000000002,"5min":0.77800000000000002,"15min":0.77800000000000002},"max":{"1min":1.645,"5min":1.645,"15min":1.645},"last":0.91900000000000004},{"interface":"front","average":{"1min":1.006,"5min":1.006,"15min":1.006},"min":{"1min":0.73999999999999999,"5min":0.73999999999999999,"15min":0.73999999999999999},"max":{"1min":1.476,"5min":1.476,"15min":1.476},"last":1.006}]}]},{"osd":1,"up_from":13,"seq":55834574913,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":7032,"kb_used_data":880,"kb_used_omap":0,"kb_used_meta":6144,"kb_avail":93738120,"statfs":{"total":95995035648,"available":95987834880,"internally_reserved":0,"allocated":901120,"data_stored":426350,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":6291456},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri May 13 23:14:14 2022","interfaces":[{"interface":"back","average":{"1min":0.60399999999999998,"5min":0.55900000000000005,"15min":0.51300000000000001},"min":{"1min":0.46700000000000003,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.755,"5min":0.91700000000000004,"15min":0.91700000000000004},"last":0.64300000000000002},{"interface":"front","average":{"1min":0.623,"5min":0.57899999999999996,"15min":0.54000000000000004},"min":{"1min":0.48399999999999999,"5min":0.309,"15min":0.309},"max":{"1min":0.82299999999999995,"5min":0.86799999999999999,"15min":0.86799999999999999},"last":0.72799999999999998}]},{"osd":2,"last update":"Fri May 13 23:13:40 2022","interfaces":[{"interface":"back","average":{"1min":0.56000000000000005,"5min":0.52300000000000002,"15min":0.51300000000000001},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.76900000000000002,"5min":0.76900000000000002,"15min":0.76900000000000002},"last":0.67400000000000004},{"interface":"front","average":{"1min":0.59899999999999998,"5min":0.54300000000000004,"15min":0.52500000000000002},"min":{"1min":0.42299999999999999,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.84599999999999997,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.61899999999999999}]},{"osd":3,"last update":"Fri May 13 23:14:12 2022","interfaces":[{"interface":"back","average":{"1min":0.64100000000000001,"5min":0.59699999999999998,"15min":0.57099999999999995},"min":{"1min":0.44600000000000001,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.84199999999999997,"5min":0.871,"15min":0.871},"last":0.95999999999999996},{"interface":"front","average":{"1min":0.60899999999999999,"5min":0.57199999999999995,"15min":0.56200000000000006},"min":{"1min":0.45400000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.77900000000000003,"5min":0.95399999999999996,"15min":0.95399999999999996},"last":0.873}]},{"osd":4,"last update":"Fri May 13 23:13:35 2022","interfaces":[{"interface":"back","average":{"1min":0.51000000000000001,"5min":0.435,"15min":0.41699999999999998},"min":{"1min":0.32100000000000001,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.77100000000000002,"5min":0.77100000000000002,"15min":0.77100000000000002},"last":0.79600000000000004},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.46899999999999997,"15min":0.44500000000000001},"min":{"1min":0.27400000000000002,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.876,"5min":0.876,"15min":0.876},"last":0.78200000000000003}]},{"osd":5,"last update":"Fri May 13 23:13:52 2022","interfaces":[{"interface":"back","average":{"1min":0.72499999999999998,"5min":0.60099999999999998,"15min":0.58499999999999996},"min":{"1min":0.497,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.88200000000000001,"5min":0.88200000000000001,"15min":0.88200000000000001},"last":0.89500000000000002},{"interface":"front","average":{"1min":0.748,"5min":0.628,"15min":0.60499999999999998},"min":{"1min":0.45500000000000002,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":0.98799999999999999,"5min":0.98799999999999999,"15min":0.98799999999999999},"last":0.81999999999999995}]},{"osd":6,"last update":"Fri May 13 23:13:23 2022","interfaces":[{"interface":"back","average":{"1min":0.76800000000000002,"5min":0.68799999999999994,"15min":0.67400000000000004},"min":{"1min":0.45000000000000001,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":1.5529999999999999,"5min":1.5529999999999999,"15min":1.5529999999999999},"last":1.0089999999999999},{"interface":"front","average":{"1min":0.75,"5min":0.71599999999999997,"15min":0.70999999999999996},"min":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":1.1419999999999999,"5min":1.1419999999999999,"15min":1.1419999999999999},"last":0.85799999999999998}]},{"osd":7,"last update":"Fri May 13 23:13:46 2022","interfaces":[{"interface":"back","average":{"1min":0.70799999999999996,"5min":0.58399999999999996,"15min":0.56299999999999994},"min":{"1min":0.38600000000000001,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.95999999999999996,"5min":0.95999999999999996,"15min":0.95999999999999996},"last":0.91400000000000003},{"interface":"front","average":{"1min":0.79500000000000004,"5min":0.65300000000000002,"15min":0.629},"min":{"1min":0.41399999999999998,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":1.042,"5min":1.042,"15min":1.042},"last":1.079}]},{"osd":8,"last update":"Fri May 13 23:14:12 2022","interfaces":[{"interface":"back","average":{"1min":0.74099999999999999,"5min":0.66900000000000004,"15min":0.65700000000000003},"min":{"1min":0.5,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":1.02,"5min":1.02,"15min":1.02},"last":1.0960000000000001},{"interface":"front","average":{"1min":0.84899999999999998,"5min":0.73299999999999998,"15min":0.71399999999999997},"min":{"1min":0.58999999999999997,"5min":0.501,"15min":0.501},"max":{"1min":1.111,"5min":1.111,"15min":1.111},"last":1.214}]},{"osd":9,"last update":"Fri May 13 23:13:39 2022","interfaces":[{"interface":"back","average":{"1min":0.88500000000000001,"5min":0.88500000000000001,"15min":0.88500000000000001},"min":{"1min":0.60999999999999999,"5min":0.60999999999999999,"15min":0.60999999999999999},"max":{"1min":1.091,"5min":1.091,"15min":1.091},"last":1.165},{"interface":"front","average":{"1min":0.84599999999999997,"5min":0.84599999999999997,"15min":0.84599999999999997},"min":{"1min":0.63600000000000001,"5min":0.63600000000000001,"15min":0.63600000000000001},"max":{"1min":1.0780000000000001,"5min":1.0780000000000001,"15min":1.0780000000000001},"last":0.92100000000000004}]},{"osd":10,"last update":"Fri May 13 23:14:01 2022","interfaces":[{"interface":"back","average":{"1min":0.78800000000000003,"5min":0.78800000000000003,"15min":0.78800000000000003},"min":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"max":{"1min":0.95399999999999996,"5min":0.95399999999999996,"15min":0.95399999999999996},"last":0.97399999999999998},{"interface":"front","average":{"1min":0.90800000000000003,"5min":0.90800000000000003,"15min":0.90800000000000003},"min":{"1min":0.623,"5min":0.623,"15min":0.623},"max":{"1min":1.105,"5min":1.105,"15min":1.105},"last":1.123}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.99299999999999999}]}]},{"osd":0,"up_from":8,"seq":34359738437,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":7112,"kb_used_data":896,"kb_used_omap":0,"kb_used_meta":6208,"kb_avail":93738040,"statfs":{"total":95995035648,"available":95987752960,"internally_reserved":0,"allocated":917504,"data_stored":426236,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":6356992},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri May 13 23:14:11 2022","interfaces":[{"interface":"back","average":{"1min":0.68300000000000005,"5min":0.627,"15min":0.60999999999999999},"min":{"1min":0.40400000000000003,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.98899999999999999,"5min":1.2729999999999999,"15min":1.2729999999999999},"last":0.65300000000000002},{"interface":"front","average":{"1min":0.752,"5min":0.66200000000000003,"15min":0.627},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":0.98999999999999999,"5min":1.167,"15min":1.167},"last":0.80100000000000005}]},{"osd":2,"last update":"Fri May 13 23:13:35 2022","interfaces":[{"interface":"back","average":{"1min":0.72499999999999998,"5min":0.65400000000000003,"15min":0.64600000000000002},"min":{"1min":0.54300000000000004,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":0.96499999999999997,"5min":0.98799999999999999,"15min":0.98799999999999999},"last":0.85399999999999998},{"interface":"front","average":{"1min":0.70099999999999996,"5min":0.63600000000000001,"15min":0.628},"min":{"1min":0.47799999999999998,"5min":0.45900000000000002,"15min":0.45900000000000002},"max":{"1min":1.024,"5min":1.123,"15min":1.123},"last":0.81100000000000005}]},{"osd":3,"last update":"Fri May 13 23:13:08 2022","interfaces":[{"interface":"back","average":{"1min":0.82399999999999995,"5min":0.63300000000000001,"15min":0.59999999999999998},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":1.163,"5min":1.163,"15min":1.163},"last":0.69999999999999996},{"interface":"front","average":{"1min":0.79800000000000004,"5min":0.67000000000000004,"15min":0.64600000000000002},"min":{"1min":0.46200000000000002,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":1.157,"5min":1.157,"15min":1.157},"last":0.57499999999999996}]},{"osd":4,"last update":"Fri May 13 23:13:33 2022","interfaces":[{"interface":"back","average":{"1min":0.82999999999999996,"5min":0.68200000000000005,"15min":0.65500000000000003},"min":{"1min":0.495,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":1.1990000000000001,"5min":1.1990000000000001,"15min":1.1990000000000001},"last":0.59699999999999998},{"interface":"front","average":{"1min":0.77900000000000003,"5min":0.66300000000000003,"15min":0.63300000000000001},"min":{"1min":0.50600000000000001,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":0.98999999999999999,"5min":0.98999999999999999,"15min":0.98999999999999999},"last":0.55600000000000005}]},{"osd":5,"last update":"Fri May 13 23:13:57 2022","interfaces":[{"interface":"back","average":{"1min":0.78900000000000003,"5min":0.72499999999999998,"15min":0.71099999999999997},"min":{"1min":0.46999999999999997,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":1.1220000000000001,"5min":1.1220000000000001,"15min":1.1220000000000001},"last":0.75},{"interface":"front","average":{"1min":0.71199999999999997,"5min":0.69699999999999995,"15min":0.69599999999999995},"min":{"1min":0.49399999999999999,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":1.0409999999999999,"5min":1.056,"15min":1.056},"last":0.69199999999999995}]},{"osd":6,"last update":"Fri May 13 23:13:25 2022","interfaces":[{"interface":"back","average":{"1min":0.93600000000000005,"5min":0.77600000000000002,"15min":0.75},"min":{"1min":0.57799999999999996,"5min":0.44800000000000001,"15min":0.44800000000000001},"max":{"1min":1.288,"5min":1.288,"15min":1.288},"last":0.89700000000000002},{"interface":"front","average":{"1min":0.874,"5min":0.78400000000000003,"15min":0.76900000000000002},"min":{"1min":0.51000000000000001,"5min":0.5,"15min":0.5},"max":{"1min":1.202,"5min":1.202,"15min":1.202},"last":0.84799999999999998}]},{"osd":7,"last update":"Fri May 13 23:13:48 2022","interfaces":[{"interface":"back","average":{"1min":0.77300000000000002,"5min":0.75,"15min":0.746},"min":{"1min":0.497,"5min":0.48699999999999999,"15min":0.48699999999999999},"max":{"1min":1.2050000000000001,"5min":1.2050000000000001,"15min":1.2050000000000001},"last":0.94199999999999995},{"interface":"front","average":{"1min":0.94899999999999995,"5min":0.86099999999999999,"15min":0.84699999999999998},"min":{"1min":0.69599999999999995,"5min":0.60399999999999998,"15min":0.60399999999999998},"max":{"1min":1.2350000000000001,"5min":1.2350000000000001,"15min":1.2350000000000001},"last":0.877}]},{"osd":8,"last update":"Fri May 13 23:14:09 2022","interfaces":[{"interface":"back","average":{"1min":0.82699999999999996,"5min":0.80500000000000005,"15min":0.80100000000000005},"min":{"1min":0.54700000000000004,"5min":0.52100000000000002,"15min":0.52100000000000002},"max":{"1min":1.1479999999999999,"5min":1.7070000000000001,"15min":1.7070000000000001},"last":0.72499999999999998},{"interface":"front","average":{"1min":0.91200000000000003,"5min":0.91600000000000004,"15min":0.91600000000000004},"min":{"1min":0.60599999999999998,"5min":0.60599999999999998,"15min":0.60599999999999998},"max":{"1min":1.1160000000000001,"5min":1.266,"15min":1.266},"last":0.93999999999999995}]},{"osd":9,"last update":"Fri May 13 23:13:39 2022","interfaces":[{"interface":"back","average":{"1min":1.0589999999999999,"5min":1.0589999999999999,"15min":1.0589999999999999},"min":{"1min":0.85199999999999998,"5min":0.85199999999999998,"15min":0.85199999999999998},"max":{"1min":1.397,"5min":1.397,"15min":1.397},"last":0.99199999999999999},{"interface":"front","average":{"1min":0.94099999999999995,"5min":0.94099999999999995,"15min":0.94099999999999995},"min":{"1min":0.70999999999999996,"5min":0.70999999999999996,"15min":0.70999999999999996},"max":{"1min":1.232,"5min":1.232,"15min":1.232},"last":0.93100000000000005}]},{"osd":10,"last update":"Fri May 13 23:14:02 2022","interfaces":[{"interface":"back","average":{"1min":0.97199999999999998,"5min":0.97199999999999998,"15min":0.97199999999999998},"min":{"1min":0.67700000000000005,"5min":0.67700000000000005,"15min":0.67700000000000005},"max":{"1min":1.7989999999999999,"5min":1.7989999999999999,"15min":1.7989999999999999},"last":0.81999999999999995},{"interface":"front","average":{"1min":1.0629999999999999,"5min":1.0629999999999999,"15min":1.0629999999999999},"min":{"1min":0.745,"5min":0.745,"15min":0.745},"max":{"1min":1.3360000000000001,"5min":1.3360000000000001,"15min":1.3360000000000001},"last":1.0329999999999999}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.97599999999999998}]}]},{"osd":2,"up_from":18,"seq":77309411387,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":7108,"kb_used_data":892,"kb_used_omap":0,"kb_used_meta":6208,"kb_avail":93738044,"statfs":{"total":95995035648,"available":95987757056,"internally_reserved":0,"allocated":913408,"data_stored":426958,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":6356992},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri May 13 23:13:37 2022","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.51700000000000002,"15min":0.52300000000000002},"min":{"1min":0.41699999999999998,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.80700000000000005,"5min":0.80700000000000005,"15min":0.80700000000000005},"last":0.439},{"interface":"front","average":{"1min":0.59499999999999997,"5min":0.52100000000000002,"15min":0.51600000000000001},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.78800000000000003,"5min":0.80700000000000005,"15min":0.80700000000000005},"last":0.64800000000000002}]},{"osd":1,"last update":"Fri May 13 23:13:37 2022","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.52500000000000002,"15min":0.52000000000000002},"min":{"1min":0.35699999999999998,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.83699999999999997,"5min":0.83699999999999997,"15min":0.83699999999999997},"last":0.69099999999999995},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.53200000000000003,"15min":0.55000000000000004},"min":{"1min":0.40100000000000002,"5min":0.313,"15min":0.313},"max":{"1min":0.77900000000000003,"5min":0.77900000000000003,"15min":0.77900000000000003},"last":0.45400000000000001}]},{"osd":3,"last update":"Fri May 13 23:14:12 2022","interfaces":[{"interface":"back","average":{"1min":0.64500000000000002,"5min":0.61599999999999999,"15min":0.61199999999999999},"min":{"1min":0.39900000000000002,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.90500000000000003,"5min":0.90500000000000003,"15min":0.90500000000000003},"last":0.51500000000000001},{"interface":"front","average":{"1min":0.66800000000000004,"5min":0.60499999999999998,"15min":0.59699999999999998},"min":{"1min":0.45000000000000001,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.89500000000000002,"5min":0.89500000000000002,"15min":0.89500000000000002},"last":0.497}]},{"osd":4,"last update":"Fri May 13 23:13:31 2022","interfaces":[{"interface":"back","average":{"1min":0.55100000000000005,"5min":0.53400000000000003,"15min":0.52600000000000002},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":0.81999999999999995,"5min":0.81999999999999995,"15min":0.81999999999999995},"last":0.59299999999999997},{"interface":"front","average":{"1min":0.53200000000000003,"5min":0.51500000000000001,"15min":0.505},"min":{"1min":0.36299999999999999,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.70099999999999996,"5min":0.752,"15min":0.752},"last":0.61299999999999999}]},{"osd":5,"last update":"Fri May 13 23:14:00 2022","interfaces":[{"interface":"back","average":{"1min":0.435,"5min":0.40300000000000002,"15min":0.38300000000000001},"min":{"1min":0.32300000000000001,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.71099999999999997,"5min":0.71999999999999997,"15min":0.71999999999999997},"last":0.39600000000000002},{"interface":"front","average":{"1min":0.47099999999999997,"5min":0.41699999999999998,"15min":0.39700000000000002},"min":{"1min":0.35499999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.748,"5min":0.748,"15min":0.748},"last":0.746}]},{"osd":6,"last update":"Fri May 13 23:13:28 2022","interfaces":[{"interface":"back","average":{"1min":0.60599999999999998,"5min":0.60599999999999998,"15min":0.60599999999999998},"min":{"1min":0.504,"5min":0.44800000000000001,"15min":0.44800000000000001},"max":{"1min":0.78500000000000003,"5min":0.78900000000000003,"15min":0.78900000000000003},"last":0.621},{"interface":"front","average":{"1min":0.63100000000000001,"5min":0.58999999999999997,"15min":0.58299999999999996},"min":{"1min":0.46500000000000002,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":0.78200000000000003,"5min":0.78200000000000003,"15min":0.78200000000000003},"last":0.63400000000000001}]},{"osd":7,"last update":"Fri May 13 23:13:45 2022","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":0.61599999999999999,"15min":0.61599999999999999},"min":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":0.80400000000000005,"5min":0.80400000000000005,"15min":0.80400000000000005},"last":0.78400000000000003},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.63800000000000001,"15min":0.64200000000000002},"min":{"1min":0.45900000000000002,"5min":0.45900000000000002,"15min":0.45900000000000002},"max":{"1min":0.83199999999999996,"5min":0.83199999999999996,"15min":0.83199999999999996},"last":0.78800000000000003}]},{"osd":8,"last update":"Fri May 13 23:14:07 2022","interfaces":[{"interface":"back","average":{"1min":0.55400000000000005,"5min":0.53600000000000003,"15min":0.53300000000000003},"min":{"1min":0.36699999999999999,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.876,"5min":0.876,"15min":0.876},"last":0.55800000000000005},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.82799999999999996,"5min":0.82799999999999996,"15min":0.82799999999999996},"last":0.54000000000000004}]},{"osd":9,"last update":"Fri May 13 23:13:37 2022","interfaces":[{"interface":"back","average":{"1min":0.70999999999999996,"5min":0.70999999999999996,"15min":0.70999999999999996},"min":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"max":{"1min":0.82799999999999996,"5min":0.82799999999999996,"15min":0.82799999999999996},"last":0.65500000000000003},{"interface":"front","average":{"1min":0.68999999999999995,"5min":0.68999999999999995,"15min":0.68999999999999995},"min":{"1min":0.55800000000000005,"5min":0.55800000000000005,"15min":0.55800000000000005},"max":{"1min":0.86799999999999999,"5min":0.86799999999999999,"15min":0.86799999999999999},"last":0.70799999999999996}]},{"osd":10,"last update":"Fri May 13 23:14:00 2022","interfaces":[{"interface":"back","average":{"1min":0.68999999999999995,"5min":0.68999999999999995,"15min":0.68999999999999995},"min":{"1min":0.47599999999999998,"5min":0.47599999999999998,"15min":0.47599999999999998},"max":{"1min":0.97099999999999997,"5min":0.97099999999999997,"15min":0.97099999999999997},"last":0.67200000000000004},{"interface":"front","average":{"1min":0.71099999999999997,"5min":0.71099999999999997,"15min":0.71099999999999997},"min":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.52100000000000002},"max":{"1min":0.89200000000000002,"5min":0.89200000000000002,"15min":0.89200000000000002},"last":0.72399999999999998}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73799999999999999}]}]},{"osd":3,"up_from":25,"seq":107374182453,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6452,"kb_used_data":876,"kb_used_omap":0,"kb_used_meta":5568,"kb_avail":93738700,"statfs":{"total":95995035648,"available":95988428800,"internally_reserved":0,"allocated":897024,"data_stored":424918,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5701632},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri May 13 23:14:15 2022","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.49399999999999999,"15min":0.46400000000000002},"min":{"1min":0.40200000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.81999999999999995,"5min":0.81999999999999995,"15min":0.81999999999999995},"last":0.45400000000000001},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.52400000000000002,"15min":0.502},"min":{"1min":0.373,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.70799999999999996,"5min":0.96299999999999997,"15min":0.96299999999999997},"last":0.47699999999999998}]},{"osd":1,"last update":"Fri May 13 23:14:15 2022","interfaces":[{"interface":"back","average":{"1min":0.749,"5min":0.67700000000000005,"15min":0.63500000000000001},"min":{"1min":0.495,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":1.014,"5min":1.014,"15min":1.014},"last":0.69399999999999995},{"interface":"front","average":{"1min":0.64100000000000001,"5min":0.64300000000000002,"15min":0.63100000000000001},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.93200000000000005,"5min":1.0229999999999999,"15min":1.0229999999999999},"last":0.55900000000000005}]},{"osd":2,"last update":"Fri May 13 23:14:15 2022","interfaces":[{"interface":"back","average":{"1min":0.79300000000000004,"5min":0.72399999999999998,"15min":0.68799999999999994},"min":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":1.0489999999999999,"5min":1.0489999999999999,"15min":1.0489999999999999},"last":0.73499999999999999},{"interface":"front","average":{"1min":0.73899999999999999,"5min":0.69699999999999995,"15min":0.67500000000000004},"min":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":1.1299999999999999,"5min":1.1299999999999999,"15min":1.1299999999999999},"last":0.64500000000000002}]},{"osd":4,"last update":"Fri May 13 23:13:33 2022","interfaces":[{"interface":"back","average":{"1min":0.90300000000000002,"5min":0.71999999999999997,"15min":0.68400000000000005},"min":{"1min":0.44400000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":1.157,"5min":1.157,"15min":1.157},"last":0.79400000000000004},{"interface":"front","average":{"1min":0.89900000000000002,"5min":0.72499999999999998,"15min":0.68799999999999994},"min":{"1min":0.56299999999999994,"5min":0.47699999999999998,"15min":0.47699999999999998},"max":{"1min":1.0660000000000001,"5min":1.0660000000000001,"15min":1.0660000000000001},"last":0.82499999999999996}]},{"osd":5,"last update":"Fri May 13 23:13:50 2022","interfaces":[{"interface":"back","average":{"1min":0.80900000000000005,"5min":0.745,"15min":0.72899999999999998},"min":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"max":{"1min":1.056,"5min":1.056,"15min":1.056},"last":0.60199999999999998},{"interface":"front","average":{"1min":0.94499999999999995,"5min":0.80000000000000004,"15min":0.76600000000000001},"min":{"1min":0.626,"5min":0.53400000000000003,"15min":0.53400000000000003},"max":{"1min":1.161,"5min":1.161,"15min":1.161},"last":0.84799999999999998}]},{"osd":6,"last update":"Fri May 13 23:13:27 2022","interfaces":[{"interface":"back","average":{"1min":0.93999999999999995,"5min":0.83299999999999996,"15min":0.81499999999999995},"min":{"1min":0.57799999999999996,"5min":0.57599999999999996,"15min":0.57599999999999996},"max":{"1min":1.2829999999999999,"5min":1.2829999999999999,"15min":1.2829999999999999},"last":0.88800000000000001},{"interface":"front","average":{"1min":0.93700000000000006,"5min":0.86399999999999999,"15min":0.85199999999999998},"min":{"1min":0.61099999999999999,"5min":0.61099999999999999,"15min":0.61099999999999999},"max":{"1min":1.256,"5min":1.256,"15min":1.256},"last":0.71999999999999997}]},{"osd":7,"last update":"Fri May 13 23:13:47 2022","interfaces":[{"interface":"back","average":{"1min":1.034,"5min":0.90700000000000003,"15min":0.88600000000000001},"min":{"1min":0.70899999999999996,"5min":0.628,"15min":0.628},"max":{"1min":1.3280000000000001,"5min":1.3280000000000001,"15min":1.3280000000000001},"last":0.93500000000000005},{"interface":"front","average":{"1min":1.028,"5min":0.89100000000000001,"15min":0.86799999999999999},"min":{"1min":0.82499999999999996,"5min":0.67800000000000005,"15min":0.67800000000000005},"max":{"1min":1.3480000000000001,"5min":1.3480000000000001,"15min":1.3480000000000001},"last":1.042}]},{"osd":8,"last update":"Fri May 13 23:14:07 2022","interfaces":[{"interface":"back","average":{"1min":0.91700000000000004,"5min":0.94699999999999995,"15min":0.95199999999999996},"min":{"1min":0.56699999999999995,"5min":0.56699999999999995,"15min":0.56699999999999995},"max":{"1min":1.2589999999999999,"5min":1.2669999999999999,"15min":1.2669999999999999},"last":1.048},{"interface":"front","average":{"1min":0.98899999999999999,"5min":1.002,"15min":1.004},"min":{"1min":0.58699999999999997,"5min":0.58699999999999997,"15min":0.58699999999999997},"max":{"1min":1.244,"5min":1.244,"15min":1.244},"last":0.91500000000000004}]},{"osd":9,"last update":"Fri May 13 23:13:39 2022","interfaces":[{"interface":"back","average":{"1min":1.131,"5min":1.131,"15min":1.131},"min":{"1min":0.79500000000000004,"5min":0.79500000000000004,"15min":0.79500000000000004},"max":{"1min":1.4319999999999999,"5min":1.4319999999999999,"15min":1.4319999999999999},"last":0.90500000000000003},{"interface":"front","average":{"1min":1.093,"5min":1.093,"15min":1.093},"min":{"1min":0.79100000000000004,"5min":0.79100000000000004,"15min":0.79100000000000004},"max":{"1min":1.4730000000000001,"5min":1.4730000000000001,"15min":1.4730000000000001},"last":0.94999999999999996}]},{"osd":10,"last update":"Fri May 13 23:14:00 2022","interfaces":[{"interface":"back","average":{"1min":1.0129999999999999,"5min":1.0129999999999999,"15min":1.0129999999999999},"min":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.52300000000000002},"max":{"1min":1.373,"5min":1.373,"15min":1.373},"last":0.68200000000000005},{"interface":"front","average":{"1min":1.127,"5min":1.127,"15min":1.127},"min":{"1min":0.78300000000000003,"5min":0.78300000000000003,"15min":0.78300000000000003},"max":{"1min":1.51,"5min":1.51,"15min":1.51},"last":0.996}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.80500000000000005}]}]},{"osd":4,"up_from":33,"seq":141733920816,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6256,"kb_used_data":872,"kb_used_omap":0,"kb_used_meta":5376,"kb_avail":93738896,"statfs":{"total":95995035648,"available":95988629504,"internally_reserved":0,"allocated":892928,"data_stored":424836,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5505024},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri May 13 23:13:35 2022","interfaces":[{"interface":"back","average":{"1min":0.627,"5min":0.56299999999999994,"15min":0.55000000000000004},"min":{"1min":0.39700000000000002,"5min":0.375,"15min":0.375},"max":{"1min":0.91800000000000004,"5min":0.91800000000000004,"15min":0.91800000000000004},"last":0.56000000000000005},{"interface":"front","average":{"1min":0.70499999999999996,"5min":0.58099999999999996,"15min":0.54500000000000004},"min":{"1min":0.47999999999999998,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.97099999999999997,"5min":0.97099999999999997,"15min":0.97099999999999997},"last":0.65200000000000002}]},{"osd":1,"last update":"Fri May 13 23:13:35 2022","interfaces":[{"interface":"back","average":{"1min":0.498,"5min":0.39200000000000002,"15min":0.372},"min":{"1min":0.309,"5min":0.217,"15min":0.217},"max":{"1min":0.84499999999999997,"5min":0.84499999999999997,"15min":0.84499999999999997},"last":0.44600000000000001},{"interface":"front","average":{"1min":0.46200000000000002,"5min":0.40300000000000002,"15min":0.39000000000000001},"min":{"1min":0.28899999999999998,"5min":0.248,"15min":0.248},"max":{"1min":0.78100000000000003,"5min":0.79900000000000004,"15min":0.79900000000000004},"last":0.496}]},{"osd":2,"last update":"Fri May 13 23:13:35 2022","interfaces":[{"interface":"back","average":{"1min":0.63800000000000001,"5min":0.54800000000000004,"15min":0.54000000000000004},"min":{"1min":0.47899999999999998,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.92600000000000005,"5min":0.92600000000000005,"15min":0.92600000000000005},"last":0.52300000000000002},{"interface":"front","average":{"1min":0.66200000000000003,"5min":0.57799999999999996,"15min":0.56499999999999995},"min":{"1min":0.5,"5min":0.372,"15min":0.372},"max":{"1min":0.92900000000000005,"5min":0.92900000000000005,"15min":0.92900000000000005},"last":0.59499999999999997}]},{"osd":3,"last update":"Fri May 13 23:13:35 2022","interfaces":[{"interface":"back","average":{"1min":0.70399999999999996,"5min":0.60799999999999998,"15min":0.58699999999999997},"min":{"1min":0.498,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.91600000000000004,"5min":0.91600000000000004,"15min":0.91600000000000004},"last":0.749},{"interface":"front","average":{"1min":0.78800000000000003,"5min":0.63700000000000001,"15min":0.61299999999999999},"min":{"1min":0.54400000000000004,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":1.0569999999999999,"5min":1.0569999999999999,"15min":1.0569999999999999},"last":0.63800000000000001}]},{"osd":5,"last update":"Fri May 13 23:13:55 2022","interfaces":[{"interface":"back","average":{"1min":0.71399999999999997,"5min":0.58699999999999997,"15min":0.55300000000000005},"min":{"1min":0.53200000000000003,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.97999999999999998,"5min":0.97999999999999998,"15min":0.97999999999999998},"last":0.67000000000000004},{"interface":"front","average":{"1min":0.70099999999999996,"5min":0.58599999999999997,"15min":0.55600000000000005},"min":{"1min":0.55600000000000005,"5min":0.371,"15min":0.371},"max":{"1min":1.1419999999999999,"5min":1.1419999999999999,"15min":1.1419999999999999},"last":0.69499999999999995}]},{"osd":6,"last update":"Fri May 13 23:13:25 2022","interfaces":[{"interface":"back","average":{"1min":0.79000000000000004,"5min":0.67800000000000005,"15min":0.66000000000000003},"min":{"1min":0.57499999999999996,"5min":0.498,"15min":0.498},"max":{"1min":1.127,"5min":1.127,"15min":1.127},"last":0.68300000000000005},{"interface":"front","average":{"1min":0.83699999999999997,"5min":0.71799999999999997,"15min":0.69799999999999995},"min":{"1min":0.60699999999999998,"5min":0.55900000000000005,"15min":0.55900000000000005},"max":{"1min":1.046,"5min":1.046,"15min":1.046},"last":0.72399999999999998}]},{"osd":7,"last update":"Fri May 13 23:13:50 2022","interfaces":[{"interface":"back","average":{"1min":0.72999999999999998,"5min":0.66500000000000004,"15min":0.65400000000000003},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.98599999999999999,"5min":0.98599999999999999,"15min":0.98599999999999999},"last":0.76600000000000001},{"interface":"front","average":{"1min":0.71699999999999997,"5min":0.64500000000000002,"15min":0.63300000000000001},"min":{"1min":0.41599999999999998,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":1.0089999999999999,"5min":1.0089999999999999,"15min":1.0089999999999999},"last":0.65200000000000002}]},{"osd":8,"last update":"Fri May 13 23:14:13 2022","interfaces":[{"interface":"back","average":{"1min":0.81499999999999995,"5min":0.75900000000000001,"15min":0.75},"min":{"1min":0.57399999999999995,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":1.099,"5min":1.099,"15min":1.099},"last":0.80100000000000005},{"interface":"front","average":{"1min":0.76000000000000001,"5min":0.754,"15min":0.753},"min":{"1min":0.44700000000000001,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":1.115,"5min":1.115,"15min":1.115},"last":0.81599999999999995}]},{"osd":9,"last update":"Fri May 13 23:13:40 2022","interfaces":[{"interface":"back","average":{"1min":0.89000000000000001,"5min":0.89000000000000001,"15min":0.89000000000000001},"min":{"1min":0.65700000000000003,"5min":0.65700000000000003,"15min":0.65700000000000003},"max":{"1min":1.1819999999999999,"5min":1.1819999999999999,"15min":1.1819999999999999},"last":0.79600000000000004},{"interface":"front","average":{"1min":0.86899999999999999,"5min":0.86899999999999999,"15min":0.86899999999999999},"min":{"1min":0.67300000000000004,"5min":0.67300000000000004,"15min":0.67300000000000004},"max":{"1min":1.171,"5min":1.171,"15min":1.171},"last":0.78700000000000003}]},{"osd":10,"last update":"Fri May 13 23:14:00 2022","interfaces":[{"interface":"back","average":{"1min":0.80700000000000005,"5min":0.80700000000000005,"15min":0.80700000000000005},"min":{"1min":0.52600000000000002,"5min":0.52600000000000002,"15min":0.52600000000000002},"max":{"1min":1.103,"5min":1.103,"15min":1.103},"last":0.77500000000000002},{"interface":"front","average":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"min":{"1min":0.48499999999999999,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":1.196,"5min":1.196,"15min":1.196},"last":0.73999999999999999}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.83399999999999996}]}]},{"osd":5,"up_from":38,"seq":163208757291,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6844,"kb_used_data":1396,"kb_used_omap":0,"kb_used_meta":5440,"kb_avail":93738308,"statfs":{"total":95995035648,"available":95988027392,"internally_reserved":0,"allocated":1429504,"data_stored":953601,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5570560},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri May 13 23:13:51 2022","interfaces":[{"interface":"back","average":{"1min":0.61099999999999999,"5min":0.53700000000000003,"15min":0.52300000000000002},"min":{"1min":0.376,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.96299999999999997,"5min":0.96299999999999997,"15min":0.96299999999999997},"last":0.53400000000000003},{"interface":"front","average":{"1min":0.57199999999999995,"5min":0.54200000000000004,"15min":0.52300000000000002},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.91300000000000003,"5min":0.96499999999999997,"15min":0.96499999999999997},"last":0.50600000000000001}]},{"osd":1,"last update":"Fri May 13 23:13:51 2022","interfaces":[{"interface":"back","average":{"1min":0.57699999999999996,"5min":0.504,"15min":0.48899999999999999},"min":{"1min":0.41499999999999998,"5min":0.317,"15min":0.317},"max":{"1min":0.76900000000000002,"5min":0.76900000000000002,"15min":0.76900000000000002},"last":0.58599999999999997},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.502,"15min":0.47999999999999998},"min":{"1min":0.42599999999999999,"5min":0.313,"15min":0.313},"max":{"1min":0.78800000000000003,"5min":0.78800000000000003,"15min":0.78800000000000003},"last":0.55200000000000005}]},{"osd":2,"last update":"Fri May 13 23:13:51 2022","interfaces":[{"interface":"back","average":{"1min":0.42399999999999999,"5min":0.38100000000000001,"15min":0.36299999999999999},"min":{"1min":0.28399999999999997,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":0.65600000000000003,"5min":0.65600000000000003,"15min":0.65600000000000003},"last":0.442},{"interface":"front","average":{"1min":0.45000000000000001,"5min":0.39500000000000002,"15min":0.375},"min":{"1min":0.28899999999999998,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.75700000000000001,"5min":0.75700000000000001,"15min":0.75700000000000001},"last":0.42499999999999999}]},{"osd":3,"last update":"Fri May 13 23:13:51 2022","interfaces":[{"interface":"back","average":{"1min":0.57499999999999996,"5min":0.56499999999999995,"15min":0.55900000000000005},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.97999999999999998,"5min":0.97999999999999998,"15min":0.97999999999999998},"last":0.48899999999999999},{"interface":"front","average":{"1min":0.625,"5min":0.57699999999999996,"15min":0.55300000000000005},"min":{"1min":0.45600000000000002,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":0.89900000000000002,"5min":1,"15min":1},"last":0.61199999999999999}]},{"osd":4,"last update":"Fri May 13 23:13:51 2022","interfaces":[{"interface":"back","average":{"1min":0.63700000000000001,"5min":0.55800000000000005,"15min":0.53700000000000003},"min":{"1min":0.38800000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.86399999999999999,"5min":0.86399999999999999,"15min":0.86399999999999999},"last":0.57599999999999996},{"interface":"front","average":{"1min":0.58299999999999996,"5min":0.54200000000000004,"15min":0.53200000000000003},"min":{"1min":0.42599999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.73399999999999999,"5min":0.89500000000000002,"15min":0.89500000000000002},"last":0.54400000000000004}]},{"osd":6,"last update":"Fri May 13 23:13:25 2022","interfaces":[{"interface":"back","average":{"1min":0.60899999999999999,"5min":0.623,"15min":0.625},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":0.872,"5min":0.95399999999999996,"15min":0.95399999999999996},"last":0.52400000000000002},{"interface":"front","average":{"1min":0.65100000000000002,"5min":0.626,"15min":0.622},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.98399999999999999,"5min":0.98399999999999999,"15min":0.98399999999999999},"last":0.63700000000000001}]},{"osd":7,"last update":"Fri May 13 23:13:46 2022","interfaces":[{"interface":"back","average":{"1min":0.629,"5min":0.64100000000000001,"15min":0.64300000000000002},"min":{"1min":0.45200000000000001,"5min":0.379,"15min":0.379},"max":{"1min":0.871,"5min":1.0489999999999999,"15min":1.0489999999999999},"last":0.621},{"interface":"front","average":{"1min":0.66700000000000004,"5min":0.64500000000000002,"15min":0.64100000000000001},"min":{"1min":0.442,"5min":0.442,"15min":0.442},"max":{"1min":0.93000000000000005,"5min":0.93000000000000005,"15min":0.93000000000000005},"last":0.67500000000000004}]},{"osd":8,"last update":"Fri May 13 23:14:06 2022","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.61599999999999999,"15min":0.622},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.89200000000000002,"5min":0.999,"15min":0.999},"last":0.59799999999999998},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.56599999999999995,"15min":0.56899999999999995},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.82699999999999996,"5min":0.82699999999999996,"15min":0.82699999999999996},"last":0.56899999999999995}]},{"osd":9,"last update":"Fri May 13 23:13:40 2022","interfaces":[{"interface":"back","average":{"1min":0.68600000000000005,"5min":0.68600000000000005,"15min":0.68600000000000005},"min":{"1min":0.51400000000000001,"5min":0.51400000000000001,"15min":0.51400000000000001},"max":{"1min":0.88500000000000001,"5min":0.88500000000000001,"15min":0.88500000000000001},"last":0.64500000000000002},{"interface":"front","average":{"1min":0.71199999999999997,"5min":0.71199999999999997,"15min":0.71199999999999997},"min":{"1min":0.505,"5min":0.505,"15min":0.505},"max":{"1min":0.99299999999999999,"5min":0.99299999999999999,"15min":0.99299999999999999},"last":0.69999999999999996}]},{"osd":10,"last update":"Fri May 13 23:13:59 2022","interfaces":[{"interface":"back","average":{"1min":0.67900000000000005,"5min":0.67900000000000005,"15min":0.67900000000000005},"min":{"1min":0.503,"5min":0.503,"15min":0.503},"max":{"1min":0.90800000000000003,"5min":0.90800000000000003,"15min":0.90800000000000003},"last":0.66800000000000004},{"interface":"front","average":{"1min":0.71399999999999997,"5min":0.71399999999999997,"15min":0.71399999999999997},"min":{"1min":0.48499999999999999,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":0.94499999999999995,"5min":0.94499999999999995,"15min":0.94499999999999995},"last":0.71699999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1422,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":2810,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":710,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1730,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1116,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2022-05-13T23:14:24.572 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:14:24 smithi002 ceph-mon[71816]: from='client.24682 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2022-05-13T23:14:24.573 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:14:24 smithi002 ceph-mon[71816]: from='client.14595 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2022-05-13T23:14:24.573 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:14:24 smithi002 ceph-mon[71816]: pgmap v292: 97 pgs: 97 active+clean; 581 KiB data, 77 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s wr, 14 op/s 2022-05-13T23:14:24.573 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:14:24 smithi002 ceph-mon[71816]: mds.? [v2:172.21.15.47:6838/761355251,v1:172.21.15.47:6839/761355251] up:active 2022-05-13T23:14:24.573 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:14:24 smithi002 ceph-mon[71816]: fsmap cephfs:5 7 up:standby 5 up:active 2022-05-13T23:14:24.574 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:14:24 smithi002 ceph-mon[71816]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2022-05-13T23:14:24.574 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:14:24 smithi002 ceph-mon[71816]: from='client.? 172.21.15.2:0/2540311023' entity='client.admin' cmd=[{"prefix": "status", "format": "json"}]: dispatch 2022-05-13T23:14:24.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:14:24 smithi115 ceph-mon[78593]: from='client.24682 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2022-05-13T23:14:24.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:14:24 smithi115 ceph-mon[78593]: from='client.14595 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2022-05-13T23:14:24.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:14:24 smithi115 ceph-mon[78593]: pgmap v292: 97 pgs: 97 active+clean; 581 KiB data, 77 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s wr, 14 op/s 2022-05-13T23:14:24.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:14:24 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.47:6838/761355251,v1:172.21.15.47:6839/761355251] up:active 2022-05-13T23:14:24.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:14:24 smithi115 ceph-mon[78593]: fsmap cephfs:5 7 up:standby 5 up:active 2022-05-13T23:14:24.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:14:24 smithi115 ceph-mon[78593]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2022-05-13T23:14:24.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:14:24 smithi115 ceph-mon[78593]: from='client.? 172.21.15.2:0/2540311023' entity='client.admin' cmd=[{"prefix": "status", "format": "json"}]: dispatch 2022-05-13T23:14:24.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:14:24 smithi047 ceph-mon[77147]: from='client.24682 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2022-05-13T23:14:24.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:14:24 smithi047 ceph-mon[77147]: from='client.14595 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2022-05-13T23:14:24.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:14:24 smithi047 ceph-mon[77147]: pgmap v292: 97 pgs: 97 active+clean; 581 KiB data, 77 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s wr, 14 op/s 2022-05-13T23:14:24.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:14:24 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.47:6838/761355251,v1:172.21.15.47:6839/761355251] up:active 2022-05-13T23:14:24.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:14:24 smithi047 ceph-mon[77147]: fsmap cephfs:5 7 up:standby 5 up:active 2022-05-13T23:14:24.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:14:24 smithi047 ceph-mon[77147]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2022-05-13T23:14:24.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:14:24 smithi047 ceph-mon[77147]: from='client.? 172.21.15.2:0/2540311023' entity='client.admin' cmd=[{"prefix": "status", "format": "json"}]: dispatch 2022-05-13T23:14:25.572 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:14:25 smithi002 ceph-mon[71816]: from='client.24725 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2022-05-13T23:14:25.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:14:25 smithi115 ceph-mon[78593]: from='client.24725 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2022-05-13T23:14:25.609 INFO:journalctl@ceph.mds.l.smithi115.stdout:May 13 23:14:25 smithi115 conmon[107138]: 2022-05-13T23:14:25.317+0000 7f8df15c1700 -1 mds.pinger is_rank_lagging: rank=3 was never sent ping request. 2022-05-13T23:14:25.609 INFO:journalctl@ceph.mds.l.smithi115.stdout:May 13 23:14:25 smithi115 conmon[107138]: 2022-05-13T23:14:25.317+0000 7f8df15c1700 -1 mds.pinger is_rank_lagging: rank=4 was never sent ping request. 2022-05-13T23:14:25.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:14:25 smithi047 ceph-mon[77147]: from='client.24725 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2022-05-13T23:14:26.572 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:14:26 smithi002 ceph-mon[71816]: pgmap v293: 97 pgs: 97 active+clean; 583 KiB data, 77 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 5.1 KiB/s wr, 21 op/s 2022-05-13T23:14:26.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:14:26 smithi115 ceph-mon[78593]: pgmap v293: 97 pgs: 97 active+clean; 583 KiB data, 77 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 5.1 KiB/s wr, 21 op/s 2022-05-13T23:14:26.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:14:26 smithi047 ceph-mon[77147]: pgmap v293: 97 pgs: 97 active+clean; 583 KiB data, 77 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 5.1 KiB/s wr, 21 op/s 2022-05-13T23:14:27.255 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2022-05-13T23:14:27.483 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:27.481+0000 7f533539d700 1 -- 172.21.15.2:0/2303862157 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f53300fb490 msgr2=0x7f53300fb8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:14:27.483 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:27.481+0000 7f533539d700 1 --2- 172.21.15.2:0/2303862157 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f53300fb490 0x7f53300fb8a0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f5320004870 tx=0x7f5320009cb0 comp rx=0 tx=0).stop 2022-05-13T23:14:27.484 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:27.481+0000 7f533539d700 1 -- 172.21.15.2:0/2303862157 shutdown_connections 2022-05-13T23:14:27.484 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:27.481+0000 7f533539d700 1 --2- 172.21.15.2:0/2303862157 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f53300fd0c0 0x7f5330103980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:14:27.484 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:27.481+0000 7f533539d700 1 --2- 172.21.15.2:0/2303862157 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f53300fb490 0x7f53300fb8a0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:14:27.484 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:27.481+0000 7f533539d700 1 --2- 172.21.15.2:0/2303862157 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f53300fc730 0x7f53300fcb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:14:27.485 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:27.481+0000 7f533539d700 1 -- 172.21.15.2:0/2303862157 >> 172.21.15.2:0/2303862157 conn(0x7f53300f6c80 msgr2=0x7f53300f90b0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-05-13T23:14:27.485 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:27.481+0000 7f533539d700 1 -- 172.21.15.2:0/2303862157 shutdown_connections 2022-05-13T23:14:27.485 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:27.481+0000 7f533539d700 1 -- 172.21.15.2:0/2303862157 wait complete. 2022-05-13T23:14:27.485 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:27.482+0000 7f533539d700 1 Processor -- start 2022-05-13T23:14:27.489 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:27.482+0000 7f533539d700 1 -- start start 2022-05-13T23:14:27.489 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:27.482+0000 7f533539d700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f53300fb490 0x7f533010e4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:14:27.489 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:27.483+0000 7f533539d700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f53300fc730 0x7f533010ea30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:14:27.490 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:27.483+0000 7f533539d700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f53300fd0c0 0x7f5330115ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:14:27.491 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:27.483+0000 7f533539d700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5330116000 con 0x7f53300fc730 2022-05-13T23:14:27.491 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:27.483+0000 7f533539d700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5330116170 con 0x7f53300fb490 2022-05-13T23:14:27.491 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:27.483+0000 7f533539d700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5330116330 con 0x7f53300fd0c0 2022-05-13T23:14:27.492 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:27.483+0000 7f532ffff700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f53300fb490 0x7f533010e4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:14:27.492 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:27.483+0000 7f532f7fe700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f53300fc730 0x7f533010ea30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:14:27.493 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:27.483+0000 7f532ffff700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f53300fb490 0x7f533010e4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.2:39226/0 (socket says 172.21.15.2:39226) 2022-05-13T23:14:27.493 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:27.483+0000 7f532ffff700 1 -- 172.21.15.2:0/1834927540 learned_addr learned my addr 172.21.15.2:0/1834927540 (peer_addr_for_me v2:172.21.15.2:0/0) 2022-05-13T23:14:27.493 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:27.483+0000 7f5334b9c700 1 --2- 172.21.15.2:0/1834927540 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f53300fd0c0 0x7f5330115ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:14:27.493 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:27.483+0000 7f532ffff700 1 -- 172.21.15.2:0/1834927540 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f53300fd0c0 msgr2=0x7f5330115ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:14:27.494 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:27.483+0000 7f532ffff700 1 --2- 172.21.15.2:0/1834927540 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f53300fd0c0 0x7f5330115ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:14:27.494 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:27.483+0000 7f532ffff700 1 -- 172.21.15.2:0/1834927540 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f53300fc730 msgr2=0x7f533010ea30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:14:27.494 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:27.483+0000 7f532ffff700 1 --2- 172.21.15.2:0/1834927540 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f53300fc730 0x7f533010ea30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:14:27.494 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:27.483+0000 7f532ffff700 1 -- 172.21.15.2:0/1834927540 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5320005040 con 0x7f53300fb490 2022-05-13T23:14:27.495 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:27.483+0000 7f532f7fe700 1 --2- 172.21.15.2:0/1834927540 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f53300fc730 0x7f533010ea30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-05-13T23:14:27.495 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:27.484+0000 7f532ffff700 1 --2- 172.21.15.2:0/1834927540 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f53300fb490 0x7f533010e4f0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f5320004710 tx=0x7f5320009cb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-05-13T23:14:27.496 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:27.484+0000 7f532cff9700 1 -- 172.21.15.2:0/1834927540 <== mon.1 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f532000e040 con 0x7f53300fb490 2022-05-13T23:14:27.496 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:27.484+0000 7f533539d700 1 -- 172.21.15.2:0/1834927540 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f53301165b0 con 0x7f53300fb490 2022-05-13T23:14:27.496 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:27.484+0000 7f533539d700 1 -- 172.21.15.2:0/1834927540 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5330116a90 con 0x7f53300fb490 2022-05-13T23:14:27.496 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:27.484+0000 7f532cff9700 1 -- 172.21.15.2:0/1834927540 <== mon.1 v2:172.21.15.47:3300/0 2 ==== config(26 keys) v1 ==== 1087+0+0 (secure 0 0 0) 0x7f5320004200 con 0x7f53300fb490 2022-05-13T23:14:27.496 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:27.484+0000 7f532cff9700 1 -- 172.21.15.2:0/1834927540 <== mon.1 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5320026aa0 con 0x7f53300fb490 2022-05-13T23:14:27.497 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:27.485+0000 7f533539d700 1 -- 172.21.15.2:0/1834927540 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5330106cd0 con 0x7f53300fb490 2022-05-13T23:14:27.497 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:27.487+0000 7f532cff9700 1 -- 172.21.15.2:0/1834927540 <== mon.1 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 16) v1 ==== 125136+0+0 (secure 0 0 0) 0x7f5320026460 con 0x7f53300fb490 2022-05-13T23:14:27.498 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:27.488+0000 7f532cff9700 1 --2- 172.21.15.2:0/1834927540 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f5318096ce0 0x7f5318099190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:14:27.499 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:27.488+0000 7f532cff9700 1 -- 172.21.15.2:0/1834927540 <== mon.1 v2:172.21.15.47:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 7975+0+0 (secure 0 0 0) 0x7f53200bfd60 con 0x7f53300fb490 2022-05-13T23:14:27.499 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:27.489+0000 7f532f7fe700 1 --2- 172.21.15.2:0/1834927540 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f5318096ce0 0x7f5318099190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:14:27.499 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:27.489+0000 7f532f7fe700 1 --2- 172.21.15.2:0/1834927540 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f5318096ce0 0x7f5318099190 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f5324007bf0 tx=0x7f532400b040 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-05-13T23:14:27.499 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:27.491+0000 7f532cff9700 1 -- 172.21.15.2:0/1834927540 <== mon.1 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+165215 (secure 0 0 0) 0x7f532008d0a0 con 0x7f53300fb490 2022-05-13T23:14:27.678 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:27.677+0000 7f533539d700 1 -- 172.21.15.2:0/1834927540 --> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f533006fe30 con 0x7f5318096ce0 2022-05-13T23:14:27.689 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:27.688+0000 7f532cff9700 1 -- 172.21.15.2:0/1834927540 <== mgr.14148 v2:172.21.15.2:6800/2788339627 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+337417 (secure 0 0 0) 0x7f533006fe30 con 0x7f5318096ce0 2022-05-13T23:14:27.696 INFO:teuthology.orchestra.run.smithi002.stdout: 2022-05-13T23:14:27.699 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:27.694+0000 7f533539d700 1 -- 172.21.15.2:0/1834927540 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f5318096ce0 msgr2=0x7f5318099190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:14:27.699 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:27.694+0000 7f533539d700 1 --2- 172.21.15.2:0/1834927540 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f5318096ce0 0x7f5318099190 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f5324007bf0 tx=0x7f532400b040 comp rx=0 tx=0).stop 2022-05-13T23:14:27.700 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:27.694+0000 7f533539d700 1 -- 172.21.15.2:0/1834927540 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f53300fb490 msgr2=0x7f533010e4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:14:27.700 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:27.694+0000 7f533539d700 1 --2- 172.21.15.2:0/1834927540 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f53300fb490 0x7f533010e4f0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f5320004710 tx=0x7f5320009cb0 comp rx=0 tx=0).stop 2022-05-13T23:14:27.701 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:27.694+0000 7f533539d700 1 -- 172.21.15.2:0/1834927540 shutdown_connections 2022-05-13T23:14:27.701 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:27.694+0000 7f533539d700 1 --2- 172.21.15.2:0/1834927540 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f53300fd0c0 0x7f5330115ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:14:27.701 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:27.694+0000 7f533539d700 1 --2- 172.21.15.2:0/1834927540 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f53300fb490 0x7f533010e4f0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:14:27.701 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:27.694+0000 7f533539d700 1 --2- 172.21.15.2:0/1834927540 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f5318096ce0 0x7f5318099190 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:14:27.702 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:27.694+0000 7f533539d700 1 --2- 172.21.15.2:0/1834927540 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f53300fc730 0x7f533010ea30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:14:27.702 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:27.694+0000 7f533539d700 1 -- 172.21.15.2:0/1834927540 >> 172.21.15.2:0/1834927540 conn(0x7f53300f6c80 msgr2=0x7f533006e710 unknown :-1 s=STATE_NONE l=0).mark_down 2022-05-13T23:14:27.702 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:27.694+0000 7f533539d700 1 -- 172.21.15.2:0/1834927540 shutdown_connections 2022-05-13T23:14:27.703 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:27.694+0000 7f533539d700 1 -- 172.21.15.2:0/1834927540 wait complete. 2022-05-13T23:14:27.703 INFO:teuthology.orchestra.run.smithi002.stderr:dumped all 2022-05-13T23:14:27.722 INFO:teuthology.orchestra.run.smithi002.stdout:{"pg_ready":true,"pg_map":{"version":294,"stamp":"2022-05-13T23:14:27.236349+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":597994,"num_objects":100,"num_object_clones":0,"num_object_copies":300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":100,"num_whiteouts":0,"num_read":176,"num_read_kb":151,"num_write":386,"num_write_kb":2995,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1057856,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":316,"ondisk_log_size":316,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":79264,"kb_used_data":12352,"kb_used_omap":0,"kb_used_meta":66816,"kb_avail":1124862560,"statfs":{"total":1151940427776,"available":1151859261440,"internally_reserved":0,"allocated":12648448,"data_stored":6706784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":68419584},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":7050,"num_objects":88,"num_object_clones":0,"num_object_copies":264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":88,"num_whiteouts":0,"num_read":50,"num_read_kb":42,"num_write":181,"num_write_kb":55,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":58,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.002877"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":10,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-05-13T23:14:16.269783+0000","last_change":"2022-05-13T23:14:16.267128+0000","last_active":"2022-05-13T23:14:16.269783+0000","last_peered":"2022-05-13T23:14:16.269783+0000","last_clean":"2022-05-13T23:14:16.269783+0000","last_became_active":"2022-05-13T23:14:16.266687+0000","last_became_peered":"2022-05-13T23:14:16.266687+0000","last_unstale":"2022-05-13T23:14:16.269783+0000","last_undegraded":"2022-05-13T23:14:16.269783+0000","last_fullsized":"2022-05-13T23:14:16.269783+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_clean_scrub_stamp":"2022-05-13T23:14:15.659362+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T04:40:46.150557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":10,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-05-13T23:14:16.280180+0000","last_change":"2022-05-13T23:14:16.272880+0000","last_active":"2022-05-13T23:14:16.280180+0000","last_peered":"2022-05-13T23:14:16.280180+0000","last_clean":"2022-05-13T23:14:16.280180+0000","last_became_active":"2022-05-13T23:14:16.272473+0000","last_became_peered":"2022-05-13T23:14:16.272473+0000","last_unstale":"2022-05-13T23:14:16.280180+0000","last_undegraded":"2022-05-13T23:14:16.280180+0000","last_fullsized":"2022-05-13T23:14:16.280180+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_clean_scrub_stamp":"2022-05-13T23:14:15.659362+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T00:32:19.185037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":10,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-05-13T23:14:16.277176+0000","last_change":"2022-05-13T23:14:16.270961+0000","last_active":"2022-05-13T23:14:16.277176+0000","last_peered":"2022-05-13T23:14:16.277176+0000","last_clean":"2022-05-13T23:14:16.277176+0000","last_became_active":"2022-05-13T23:14:16.269942+0000","last_became_peered":"2022-05-13T23:14:16.269942+0000","last_unstale":"2022-05-13T23:14:16.277176+0000","last_undegraded":"2022-05-13T23:14:16.277176+0000","last_fullsized":"2022-05-13T23:14:16.277176+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_clean_scrub_stamp":"2022-05-13T23:14:15.659362+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T10:03:03.774806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":10,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-05-13T23:14:16.257175+0000","last_change":"2022-05-13T23:14:16.255885+0000","last_active":"2022-05-13T23:14:16.257175+0000","last_peered":"2022-05-13T23:14:16.257175+0000","last_clean":"2022-05-13T23:14:16.257175+0000","last_became_active":"2022-05-13T23:14:16.255551+0000","last_became_peered":"2022-05-13T23:14:16.255551+0000","last_unstale":"2022-05-13T23:14:16.257175+0000","last_undegraded":"2022-05-13T23:14:16.257175+0000","last_fullsized":"2022-05-13T23:14:16.257175+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_clean_scrub_stamp":"2022-05-13T23:14:15.659362+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T10:26:40.578066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":10,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-05-13T23:14:16.271952+0000","last_change":"2022-05-13T23:14:16.271384+0000","last_active":"2022-05-13T23:14:16.271952+0000","last_peered":"2022-05-13T23:14:16.271952+0000","last_clean":"2022-05-13T23:14:16.271952+0000","last_became_active":"2022-05-13T23:14:16.271045+0000","last_became_peered":"2022-05-13T23:14:16.271045+0000","last_unstale":"2022-05-13T23:14:16.271952+0000","last_undegraded":"2022-05-13T23:14:16.271952+0000","last_fullsized":"2022-05-13T23:14:16.271952+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_clean_scrub_stamp":"2022-05-13T23:14:15.659362+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T04:51:50.618628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":10,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-05-13T23:14:16.276293+0000","last_change":"2022-05-13T23:14:16.275906+0000","last_active":"2022-05-13T23:14:16.276293+0000","last_peered":"2022-05-13T23:14:16.276293+0000","last_clean":"2022-05-13T23:14:16.276293+0000","last_became_active":"2022-05-13T23:14:16.275651+0000","last_became_peered":"2022-05-13T23:14:16.275651+0000","last_unstale":"2022-05-13T23:14:16.276293+0000","last_undegraded":"2022-05-13T23:14:16.276293+0000","last_fullsized":"2022-05-13T23:14:16.276293+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_clean_scrub_stamp":"2022-05-13T23:14:15.659362+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T02:29:33.019268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":10,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-05-13T23:14:16.276560+0000","last_change":"2022-05-13T23:14:16.276150+0000","last_active":"2022-05-13T23:14:16.276560+0000","last_peered":"2022-05-13T23:14:16.276560+0000","last_clean":"2022-05-13T23:14:16.276560+0000","last_became_active":"2022-05-13T23:14:16.275751+0000","last_became_peered":"2022-05-13T23:14:16.275751+0000","last_unstale":"2022-05-13T23:14:16.276560+0000","last_undegraded":"2022-05-13T23:14:16.276560+0000","last_fullsized":"2022-05-13T23:14:16.276560+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_clean_scrub_stamp":"2022-05-13T23:14:15.659362+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T04:01:48.496870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":10,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-05-13T23:14:16.270951+0000","last_change":"2022-05-13T23:14:16.268816+0000","last_active":"2022-05-13T23:14:16.270951+0000","last_peered":"2022-05-13T23:14:16.270951+0000","last_clean":"2022-05-13T23:14:16.270951+0000","last_became_active":"2022-05-13T23:14:16.268429+0000","last_became_peered":"2022-05-13T23:14:16.268429+0000","last_unstale":"2022-05-13T23:14:16.270951+0000","last_undegraded":"2022-05-13T23:14:16.270951+0000","last_fullsized":"2022-05-13T23:14:16.270951+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_clean_scrub_stamp":"2022-05-13T23:14:15.659362+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T00:14:51.685287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":10,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-05-13T23:14:16.265890+0000","last_change":"2022-05-13T23:14:16.263971+0000","last_active":"2022-05-13T23:14:16.265890+0000","last_peered":"2022-05-13T23:14:16.265890+0000","last_clean":"2022-05-13T23:14:16.265890+0000","last_became_active":"2022-05-13T23:14:16.263625+0000","last_became_peered":"2022-05-13T23:14:16.263625+0000","last_unstale":"2022-05-13T23:14:16.265890+0000","last_undegraded":"2022-05-13T23:14:16.265890+0000","last_fullsized":"2022-05-13T23:14:16.265890+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_clean_scrub_stamp":"2022-05-13T23:14:15.659362+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T03:11:11.037746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":10,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-05-13T23:14:16.258455+0000","last_change":"2022-05-13T23:14:16.258111+0000","last_active":"2022-05-13T23:14:16.258455+0000","last_peered":"2022-05-13T23:14:16.258455+0000","last_clean":"2022-05-13T23:14:16.258455+0000","last_became_active":"2022-05-13T23:14:16.257768+0000","last_became_peered":"2022-05-13T23:14:16.257768+0000","last_unstale":"2022-05-13T23:14:16.258455+0000","last_undegraded":"2022-05-13T23:14:16.258455+0000","last_fullsized":"2022-05-13T23:14:16.258455+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_clean_scrub_stamp":"2022-05-13T23:14:15.659362+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T07:27:48.568156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":10,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-05-13T23:14:16.268155+0000","last_change":"2022-05-13T23:14:16.267256+0000","last_active":"2022-05-13T23:14:16.268155+0000","last_peered":"2022-05-13T23:14:16.268155+0000","last_clean":"2022-05-13T23:14:16.268155+0000","last_became_active":"2022-05-13T23:14:16.266868+0000","last_became_peered":"2022-05-13T23:14:16.266868+0000","last_unstale":"2022-05-13T23:14:16.268155+0000","last_undegraded":"2022-05-13T23:14:16.268155+0000","last_fullsized":"2022-05-13T23:14:16.268155+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_clean_scrub_stamp":"2022-05-13T23:14:15.659362+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T03:28:58.592648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":10,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-05-13T23:14:16.257196+0000","last_change":"2022-05-13T23:14:16.256051+0000","last_active":"2022-05-13T23:14:16.257196+0000","last_peered":"2022-05-13T23:14:16.257196+0000","last_clean":"2022-05-13T23:14:16.257196+0000","last_became_active":"2022-05-13T23:14:16.255729+0000","last_became_peered":"2022-05-13T23:14:16.255729+0000","last_unstale":"2022-05-13T23:14:16.257196+0000","last_undegraded":"2022-05-13T23:14:16.257196+0000","last_fullsized":"2022-05-13T23:14:16.257196+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_clean_scrub_stamp":"2022-05-13T23:14:15.659362+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T09:34:22.448938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":10,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-05-13T23:14:16.268884+0000","last_change":"2022-05-13T23:14:16.266285+0000","last_active":"2022-05-13T23:14:16.268884+0000","last_peered":"2022-05-13T23:14:16.268884+0000","last_clean":"2022-05-13T23:14:16.268884+0000","last_became_active":"2022-05-13T23:14:16.264899+0000","last_became_peered":"2022-05-13T23:14:16.264899+0000","last_unstale":"2022-05-13T23:14:16.268884+0000","last_undegraded":"2022-05-13T23:14:16.268884+0000","last_fullsized":"2022-05-13T23:14:16.268884+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_clean_scrub_stamp":"2022-05-13T23:14:15.659362+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T09:50:56.211965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":10,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-05-13T23:14:16.279550+0000","last_change":"2022-05-13T23:14:16.271503+0000","last_active":"2022-05-13T23:14:16.279550+0000","last_peered":"2022-05-13T23:14:16.279550+0000","last_clean":"2022-05-13T23:14:16.279550+0000","last_became_active":"2022-05-13T23:14:16.271135+0000","last_became_peered":"2022-05-13T23:14:16.271135+0000","last_unstale":"2022-05-13T23:14:16.279550+0000","last_undegraded":"2022-05-13T23:14:16.279550+0000","last_fullsized":"2022-05-13T23:14:16.279550+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_clean_scrub_stamp":"2022-05-13T23:14:15.659362+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T05:01:25.159085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":10,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-05-13T23:14:16.273627+0000","last_change":"2022-05-13T23:14:16.265771+0000","last_active":"2022-05-13T23:14:16.273627+0000","last_peered":"2022-05-13T23:14:16.273627+0000","last_clean":"2022-05-13T23:14:16.273627+0000","last_became_active":"2022-05-13T23:14:16.265357+0000","last_became_peered":"2022-05-13T23:14:16.265357+0000","last_unstale":"2022-05-13T23:14:16.273627+0000","last_undegraded":"2022-05-13T23:14:16.273627+0000","last_fullsized":"2022-05-13T23:14:16.273627+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_clean_scrub_stamp":"2022-05-13T23:14:15.659362+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T02:16:47.449089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":10,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-05-13T23:14:16.270716+0000","last_change":"2022-05-13T23:14:16.270173+0000","last_active":"2022-05-13T23:14:16.270716+0000","last_peered":"2022-05-13T23:14:16.270716+0000","last_clean":"2022-05-13T23:14:16.270716+0000","last_became_active":"2022-05-13T23:14:16.269859+0000","last_became_peered":"2022-05-13T23:14:16.269859+0000","last_unstale":"2022-05-13T23:14:16.270716+0000","last_undegraded":"2022-05-13T23:14:16.270716+0000","last_fullsized":"2022-05-13T23:14:16.270716+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_clean_scrub_stamp":"2022-05-13T23:14:15.659362+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T03:31:45.566757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":10,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-05-13T23:14:16.269931+0000","last_change":"2022-05-13T23:14:16.269290+0000","last_active":"2022-05-13T23:14:16.269931+0000","last_peered":"2022-05-13T23:14:16.269931+0000","last_clean":"2022-05-13T23:14:16.269931+0000","last_became_active":"2022-05-13T23:14:16.268985+0000","last_became_peered":"2022-05-13T23:14:16.268985+0000","last_unstale":"2022-05-13T23:14:16.269931+0000","last_undegraded":"2022-05-13T23:14:16.269931+0000","last_fullsized":"2022-05-13T23:14:16.269931+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_clean_scrub_stamp":"2022-05-13T23:14:15.659362+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T03:58:45.885549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":10,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-05-13T23:14:16.267609+0000","last_change":"2022-05-13T23:14:16.266204+0000","last_active":"2022-05-13T23:14:16.267609+0000","last_peered":"2022-05-13T23:14:16.267609+0000","last_clean":"2022-05-13T23:14:16.267609+0000","last_became_active":"2022-05-13T23:14:16.262256+0000","last_became_peered":"2022-05-13T23:14:16.262256+0000","last_unstale":"2022-05-13T23:14:16.267609+0000","last_undegraded":"2022-05-13T23:14:16.267609+0000","last_fullsized":"2022-05-13T23:14:16.267609+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_clean_scrub_stamp":"2022-05-13T23:14:15.659362+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T00:53:26.594598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":10,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-05-13T23:14:16.254638+0000","last_change":"2022-05-13T23:14:16.252421+0000","last_active":"2022-05-13T23:14:16.254638+0000","last_peered":"2022-05-13T23:14:16.254638+0000","last_clean":"2022-05-13T23:14:16.254638+0000","last_became_active":"2022-05-13T23:14:16.250585+0000","last_became_peered":"2022-05-13T23:14:16.250585+0000","last_unstale":"2022-05-13T23:14:16.254638+0000","last_undegraded":"2022-05-13T23:14:16.254638+0000","last_fullsized":"2022-05-13T23:14:16.254638+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_clean_scrub_stamp":"2022-05-13T23:14:15.659362+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T00:13:24.393163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"83'13","reported_seq":29,"reported_epoch":83,"state":"active+clean","last_fresh":"2022-05-13T23:14:25.325289+0000","last_change":"2022-05-13T23:14:15.683837+0000","last_active":"2022-05-13T23:14:25.325289+0000","last_peered":"2022-05-13T23:14:25.325289+0000","last_clean":"2022-05-13T23:14:25.325289+0000","last_became_active":"2022-05-13T23:14:15.682976+0000","last_became_peered":"2022-05-13T23:14:15.682976+0000","last_unstale":"2022-05-13T23:14:25.325289+0000","last_undegraded":"2022-05-13T23:14:25.325289+0000","last_fullsized":"2022-05-13T23:14:25.325289+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:14.344206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:14.344206+0000","last_clean_scrub_stamp":"2022-05-13T23:14:14.344206+0000","objects_scrubbed":0,"log_size":13,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T01:05:36.612390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":644,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":10,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-05-13T23:14:16.267355+0000","last_change":"2022-05-13T23:14:16.264501+0000","last_active":"2022-05-13T23:14:16.267355+0000","last_peered":"2022-05-13T23:14:16.267355+0000","last_clean":"2022-05-13T23:14:16.267355+0000","last_became_active":"2022-05-13T23:14:16.264105+0000","last_became_peered":"2022-05-13T23:14:16.264105+0000","last_unstale":"2022-05-13T23:14:16.267355+0000","last_undegraded":"2022-05-13T23:14:16.267355+0000","last_fullsized":"2022-05-13T23:14:16.267355+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_clean_scrub_stamp":"2022-05-13T23:14:15.659362+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T00:21:48.857254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"83'2","reported_seq":13,"reported_epoch":83,"state":"active+clean","last_fresh":"2022-05-13T23:14:17.553136+0000","last_change":"2022-05-13T23:14:15.701757+0000","last_active":"2022-05-13T23:14:17.553136+0000","last_peered":"2022-05-13T23:14:17.553136+0000","last_clean":"2022-05-13T23:14:17.553136+0000","last_became_active":"2022-05-13T23:14:15.701428+0000","last_became_peered":"2022-05-13T23:14:15.701428+0000","last_unstale":"2022-05-13T23:14:17.553136+0000","last_undegraded":"2022-05-13T23:14:17.553136+0000","last_fullsized":"2022-05-13T23:14:17.553136+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:14.344206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:14.344206+0000","last_clean_scrub_stamp":"2022-05-13T23:14:14.344206+0000","objects_scrubbed":0,"log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T05:26:07.810433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":10,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-05-13T23:14:16.281726+0000","last_change":"2022-05-13T23:14:16.280002+0000","last_active":"2022-05-13T23:14:16.281726+0000","last_peered":"2022-05-13T23:14:16.281726+0000","last_clean":"2022-05-13T23:14:16.281726+0000","last_became_active":"2022-05-13T23:14:16.279621+0000","last_became_peered":"2022-05-13T23:14:16.279621+0000","last_unstale":"2022-05-13T23:14:16.281726+0000","last_undegraded":"2022-05-13T23:14:16.281726+0000","last_fullsized":"2022-05-13T23:14:16.281726+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_clean_scrub_stamp":"2022-05-13T23:14:15.659362+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T06:42:49.090643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"83'12","reported_seq":24,"reported_epoch":83,"state":"active+clean","last_fresh":"2022-05-13T23:14:23.316963+0000","last_change":"2022-05-13T23:14:15.683457+0000","last_active":"2022-05-13T23:14:23.316963+0000","last_peered":"2022-05-13T23:14:23.316963+0000","last_clean":"2022-05-13T23:14:23.316963+0000","last_became_active":"2022-05-13T23:14:15.682883+0000","last_became_peered":"2022-05-13T23:14:15.682883+0000","last_unstale":"2022-05-13T23:14:23.316963+0000","last_undegraded":"2022-05-13T23:14:23.316963+0000","last_fullsized":"2022-05-13T23:14:23.316963+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:14.344206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:14.344206+0000","last_clean_scrub_stamp":"2022-05-13T23:14:14.344206+0000","objects_scrubbed":0,"log_size":12,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T07:38:58.996672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":16,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":10,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-05-13T23:14:16.261946+0000","last_change":"2022-05-13T23:14:16.261315+0000","last_active":"2022-05-13T23:14:16.261946+0000","last_peered":"2022-05-13T23:14:16.261946+0000","last_clean":"2022-05-13T23:14:16.261946+0000","last_became_active":"2022-05-13T23:14:16.260265+0000","last_became_peered":"2022-05-13T23:14:16.260265+0000","last_unstale":"2022-05-13T23:14:16.261946+0000","last_undegraded":"2022-05-13T23:14:16.261946+0000","last_fullsized":"2022-05-13T23:14:16.261946+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_clean_scrub_stamp":"2022-05-13T23:14:15.659362+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T08:02:58.080071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"83'5","reported_seq":17,"reported_epoch":83,"state":"active+clean","last_fresh":"2022-05-13T23:14:22.317079+0000","last_change":"2022-05-13T23:14:15.682693+0000","last_active":"2022-05-13T23:14:22.317079+0000","last_peered":"2022-05-13T23:14:22.317079+0000","last_clean":"2022-05-13T23:14:22.317079+0000","last_became_active":"2022-05-13T23:14:15.682414+0000","last_became_peered":"2022-05-13T23:14:15.682414+0000","last_unstale":"2022-05-13T23:14:22.317079+0000","last_undegraded":"2022-05-13T23:14:22.317079+0000","last_fullsized":"2022-05-13T23:14:22.317079+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:14.344206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:14.344206+0000","last_clean_scrub_stamp":"2022-05-13T23:14:14.344206+0000","objects_scrubbed":0,"log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T09:15:17.699871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":554,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":10,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-05-13T23:14:16.261554+0000","last_change":"2022-05-13T23:14:16.261044+0000","last_active":"2022-05-13T23:14:16.261554+0000","last_peered":"2022-05-13T23:14:16.261554+0000","last_clean":"2022-05-13T23:14:16.261554+0000","last_became_active":"2022-05-13T23:14:16.260687+0000","last_became_peered":"2022-05-13T23:14:16.260687+0000","last_unstale":"2022-05-13T23:14:16.261554+0000","last_undegraded":"2022-05-13T23:14:16.261554+0000","last_fullsized":"2022-05-13T23:14:16.261554+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_clean_scrub_stamp":"2022-05-13T23:14:15.659362+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-14T23:38:23.413188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"83'5","reported_seq":19,"reported_epoch":83,"state":"active+clean","last_fresh":"2022-05-13T23:14:23.319605+0000","last_change":"2022-05-13T23:14:15.678240+0000","last_active":"2022-05-13T23:14:23.319605+0000","last_peered":"2022-05-13T23:14:23.319605+0000","last_clean":"2022-05-13T23:14:23.319605+0000","last_became_active":"2022-05-13T23:14:15.677918+0000","last_became_peered":"2022-05-13T23:14:15.677918+0000","last_unstale":"2022-05-13T23:14:23.319605+0000","last_undegraded":"2022-05-13T23:14:23.319605+0000","last_fullsized":"2022-05-13T23:14:23.319605+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:14.344206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:14.344206+0000","last_clean_scrub_stamp":"2022-05-13T23:14:14.344206+0000","objects_scrubbed":0,"log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T09:53:37.623728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":574,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":2,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":10,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-05-13T23:14:16.274210+0000","last_change":"2022-05-13T23:14:16.273537+0000","last_active":"2022-05-13T23:14:16.274210+0000","last_peered":"2022-05-13T23:14:16.274210+0000","last_clean":"2022-05-13T23:14:16.274210+0000","last_became_active":"2022-05-13T23:14:16.273127+0000","last_became_peered":"2022-05-13T23:14:16.273127+0000","last_unstale":"2022-05-13T23:14:16.274210+0000","last_undegraded":"2022-05-13T23:14:16.274210+0000","last_fullsized":"2022-05-13T23:14:16.274210+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_clean_scrub_stamp":"2022-05-13T23:14:15.659362+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T01:00:18.643721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"83'7","reported_seq":18,"reported_epoch":83,"state":"active+clean","last_fresh":"2022-05-13T23:14:21.311151+0000","last_change":"2022-05-13T23:14:15.680562+0000","last_active":"2022-05-13T23:14:21.311151+0000","last_peered":"2022-05-13T23:14:21.311151+0000","last_clean":"2022-05-13T23:14:21.311151+0000","last_became_active":"2022-05-13T23:14:15.680163+0000","last_became_peered":"2022-05-13T23:14:15.680163+0000","last_unstale":"2022-05-13T23:14:21.311151+0000","last_undegraded":"2022-05-13T23:14:21.311151+0000","last_fullsized":"2022-05-13T23:14:21.311151+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:14.344206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:14.344206+0000","last_clean_scrub_stamp":"2022-05-13T23:14:14.344206+0000","objects_scrubbed":0,"log_size":7,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T02:39:27.895407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":10,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-05-13T23:14:16.271643+0000","last_change":"2022-05-13T23:14:16.270878+0000","last_active":"2022-05-13T23:14:16.271643+0000","last_peered":"2022-05-13T23:14:16.271643+0000","last_clean":"2022-05-13T23:14:16.271643+0000","last_became_active":"2022-05-13T23:14:16.270416+0000","last_became_peered":"2022-05-13T23:14:16.270416+0000","last_unstale":"2022-05-13T23:14:16.271643+0000","last_undegraded":"2022-05-13T23:14:16.271643+0000","last_fullsized":"2022-05-13T23:14:16.271643+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_clean_scrub_stamp":"2022-05-13T23:14:15.659362+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T06:11:33.932416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"83'3","reported_seq":14,"reported_epoch":83,"state":"active+clean","last_fresh":"2022-05-13T23:14:20.304539+0000","last_change":"2022-05-13T23:14:15.676518+0000","last_active":"2022-05-13T23:14:20.304539+0000","last_peered":"2022-05-13T23:14:20.304539+0000","last_clean":"2022-05-13T23:14:20.304539+0000","last_became_active":"2022-05-13T23:14:15.676157+0000","last_became_peered":"2022-05-13T23:14:15.676157+0000","last_unstale":"2022-05-13T23:14:20.304539+0000","last_undegraded":"2022-05-13T23:14:20.304539+0000","last_fullsized":"2022-05-13T23:14:20.304539+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:14.344206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:14.344206+0000","last_clean_scrub_stamp":"2022-05-13T23:14:14.344206+0000","objects_scrubbed":0,"log_size":3,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T10:16:49.469347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":10,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-05-13T23:14:16.262532+0000","last_change":"2022-05-13T23:14:16.257424+0000","last_active":"2022-05-13T23:14:16.262532+0000","last_peered":"2022-05-13T23:14:16.262532+0000","last_clean":"2022-05-13T23:14:16.262532+0000","last_became_active":"2022-05-13T23:14:16.254550+0000","last_became_peered":"2022-05-13T23:14:16.254550+0000","last_unstale":"2022-05-13T23:14:16.262532+0000","last_undegraded":"2022-05-13T23:14:16.262532+0000","last_fullsized":"2022-05-13T23:14:16.262532+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_clean_scrub_stamp":"2022-05-13T23:14:15.659362+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T00:04:50.150368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"83'3","reported_seq":15,"reported_epoch":83,"state":"active+clean","last_fresh":"2022-05-13T23:14:22.314791+0000","last_change":"2022-05-13T23:14:15.680378+0000","last_active":"2022-05-13T23:14:22.314791+0000","last_peered":"2022-05-13T23:14:22.314791+0000","last_clean":"2022-05-13T23:14:22.314791+0000","last_became_active":"2022-05-13T23:14:15.680063+0000","last_became_peered":"2022-05-13T23:14:15.680063+0000","last_unstale":"2022-05-13T23:14:22.314791+0000","last_undegraded":"2022-05-13T23:14:22.314791+0000","last_fullsized":"2022-05-13T23:14:22.314791+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:14.344206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:14.344206+0000","last_clean_scrub_stamp":"2022-05-13T23:14:14.344206+0000","objects_scrubbed":0,"log_size":3,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-14T23:52:48.817491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":554,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":10,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-05-13T23:14:16.281100+0000","last_change":"2022-05-13T23:14:16.277131+0000","last_active":"2022-05-13T23:14:16.281100+0000","last_peered":"2022-05-13T23:14:16.281100+0000","last_clean":"2022-05-13T23:14:16.281100+0000","last_became_active":"2022-05-13T23:14:16.276815+0000","last_became_peered":"2022-05-13T23:14:16.276815+0000","last_unstale":"2022-05-13T23:14:16.281100+0000","last_undegraded":"2022-05-13T23:14:16.281100+0000","last_fullsized":"2022-05-13T23:14:16.281100+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_clean_scrub_stamp":"2022-05-13T23:14:15.659362+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-14T23:22:29.938073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"83'4","reported_seq":17,"reported_epoch":83,"state":"active+clean","last_fresh":"2022-05-13T23:14:23.318749+0000","last_change":"2022-05-13T23:14:15.683981+0000","last_active":"2022-05-13T23:14:23.318749+0000","last_peered":"2022-05-13T23:14:23.318749+0000","last_clean":"2022-05-13T23:14:23.318749+0000","last_became_active":"2022-05-13T23:14:15.683560+0000","last_became_peered":"2022-05-13T23:14:15.683560+0000","last_unstale":"2022-05-13T23:14:23.318749+0000","last_undegraded":"2022-05-13T23:14:23.318749+0000","last_fullsized":"2022-05-13T23:14:23.318749+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:14.344206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:14.344206+0000","last_clean_scrub_stamp":"2022-05-13T23:14:14.344206+0000","objects_scrubbed":0,"log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T00:30:01.510592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":10,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-05-13T23:14:16.269905+0000","last_change":"2022-05-13T23:14:16.269244+0000","last_active":"2022-05-13T23:14:16.269905+0000","last_peered":"2022-05-13T23:14:16.269905+0000","last_clean":"2022-05-13T23:14:16.269905+0000","last_became_active":"2022-05-13T23:14:16.268949+0000","last_became_peered":"2022-05-13T23:14:16.268949+0000","last_unstale":"2022-05-13T23:14:16.269905+0000","last_undegraded":"2022-05-13T23:14:16.269905+0000","last_fullsized":"2022-05-13T23:14:16.269905+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_clean_scrub_stamp":"2022-05-13T23:14:15.659362+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T04:34:53.311191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"83'4","reported_seq":15,"reported_epoch":83,"state":"active+clean","last_fresh":"2022-05-13T23:14:22.317425+0000","last_change":"2022-05-13T23:14:15.683758+0000","last_active":"2022-05-13T23:14:22.317425+0000","last_peered":"2022-05-13T23:14:22.317425+0000","last_clean":"2022-05-13T23:14:22.317425+0000","last_became_active":"2022-05-13T23:14:15.683356+0000","last_became_peered":"2022-05-13T23:14:15.683356+0000","last_unstale":"2022-05-13T23:14:22.317425+0000","last_undegraded":"2022-05-13T23:14:22.317425+0000","last_fullsized":"2022-05-13T23:14:22.317425+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:14.344206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:14.344206+0000","last_clean_scrub_stamp":"2022-05-13T23:14:14.344206+0000","objects_scrubbed":0,"log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T08:00:55.113935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":10,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-05-13T23:14:16.274939+0000","last_change":"2022-05-13T23:14:16.268751+0000","last_active":"2022-05-13T23:14:16.274939+0000","last_peered":"2022-05-13T23:14:16.274939+0000","last_clean":"2022-05-13T23:14:16.274939+0000","last_became_active":"2022-05-13T23:14:16.268332+0000","last_became_peered":"2022-05-13T23:14:16.268332+0000","last_unstale":"2022-05-13T23:14:16.274939+0000","last_undegraded":"2022-05-13T23:14:16.274939+0000","last_fullsized":"2022-05-13T23:14:16.274939+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_clean_scrub_stamp":"2022-05-13T23:14:15.659362+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T06:22:45.140343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"83'5","reported_seq":17,"reported_epoch":83,"state":"active+clean","last_fresh":"2022-05-13T23:14:21.310103+0000","last_change":"2022-05-13T23:14:15.701120+0000","last_active":"2022-05-13T23:14:21.310103+0000","last_peered":"2022-05-13T23:14:21.310103+0000","last_clean":"2022-05-13T23:14:21.310103+0000","last_became_active":"2022-05-13T23:14:15.700812+0000","last_became_peered":"2022-05-13T23:14:15.700812+0000","last_unstale":"2022-05-13T23:14:21.310103+0000","last_undegraded":"2022-05-13T23:14:21.310103+0000","last_fullsized":"2022-05-13T23:14:21.310103+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:14.344206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:14.344206+0000","last_clean_scrub_stamp":"2022-05-13T23:14:14.344206+0000","objects_scrubbed":0,"log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T02:16:02.909062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":644,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":10,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-05-13T23:14:16.266148+0000","last_change":"2022-05-13T23:14:16.265167+0000","last_active":"2022-05-13T23:14:16.266148+0000","last_peered":"2022-05-13T23:14:16.266148+0000","last_clean":"2022-05-13T23:14:16.266148+0000","last_became_active":"2022-05-13T23:14:16.264886+0000","last_became_peered":"2022-05-13T23:14:16.264886+0000","last_unstale":"2022-05-13T23:14:16.266148+0000","last_undegraded":"2022-05-13T23:14:16.266148+0000","last_fullsized":"2022-05-13T23:14:16.266148+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_clean_scrub_stamp":"2022-05-13T23:14:15.659362+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T00:26:06.650867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"83'10","reported_seq":25,"reported_epoch":83,"state":"active+clean","last_fresh":"2022-05-13T23:14:23.320414+0000","last_change":"2022-05-13T23:14:15.682385+0000","last_active":"2022-05-13T23:14:23.320414+0000","last_peered":"2022-05-13T23:14:23.320414+0000","last_clean":"2022-05-13T23:14:23.320414+0000","last_became_active":"2022-05-13T23:14:15.681996+0000","last_became_peered":"2022-05-13T23:14:15.681996+0000","last_unstale":"2022-05-13T23:14:23.320414+0000","last_undegraded":"2022-05-13T23:14:23.320414+0000","last_fullsized":"2022-05-13T23:14:23.320414+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:14.344206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:14.344206+0000","last_clean_scrub_stamp":"2022-05-13T23:14:14.344206+0000","objects_scrubbed":0,"log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-14T23:44:23.766036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1536,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":4,"num_read_kb":3,"num_write":12,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":10,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-05-13T23:14:16.264519+0000","last_change":"2022-05-13T23:14:16.263744+0000","last_active":"2022-05-13T23:14:16.264519+0000","last_peered":"2022-05-13T23:14:16.264519+0000","last_clean":"2022-05-13T23:14:16.264519+0000","last_became_active":"2022-05-13T23:14:16.263525+0000","last_became_peered":"2022-05-13T23:14:16.263525+0000","last_unstale":"2022-05-13T23:14:16.264519+0000","last_undegraded":"2022-05-13T23:14:16.264519+0000","last_fullsized":"2022-05-13T23:14:16.264519+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_clean_scrub_stamp":"2022-05-13T23:14:15.659362+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T04:23:44.856055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"83'9","reported_seq":22,"reported_epoch":83,"state":"active+clean","last_fresh":"2022-05-13T23:14:23.317189+0000","last_change":"2022-05-13T23:14:15.683079+0000","last_active":"2022-05-13T23:14:23.317189+0000","last_peered":"2022-05-13T23:14:23.317189+0000","last_clean":"2022-05-13T23:14:23.317189+0000","last_became_active":"2022-05-13T23:14:15.682750+0000","last_became_peered":"2022-05-13T23:14:15.682750+0000","last_unstale":"2022-05-13T23:14:23.317189+0000","last_undegraded":"2022-05-13T23:14:23.317189+0000","last_fullsized":"2022-05-13T23:14:23.317189+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:14.344206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:14.344206+0000","last_clean_scrub_stamp":"2022-05-13T23:14:14.344206+0000","objects_scrubbed":0,"log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T02:37:31.530579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":10,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-05-13T23:14:16.268332+0000","last_change":"2022-05-13T23:14:16.263943+0000","last_active":"2022-05-13T23:14:16.268332+0000","last_peered":"2022-05-13T23:14:16.268332+0000","last_clean":"2022-05-13T23:14:16.268332+0000","last_became_active":"2022-05-13T23:14:16.263595+0000","last_became_peered":"2022-05-13T23:14:16.263595+0000","last_unstale":"2022-05-13T23:14:16.268332+0000","last_undegraded":"2022-05-13T23:14:16.268332+0000","last_fullsized":"2022-05-13T23:14:16.268332+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_clean_scrub_stamp":"2022-05-13T23:14:15.659362+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T04:57:04.201317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"83'5","reported_seq":17,"reported_epoch":83,"state":"active+clean","last_fresh":"2022-05-13T23:14:22.315301+0000","last_change":"2022-05-13T23:14:15.683005+0000","last_active":"2022-05-13T23:14:22.315301+0000","last_peered":"2022-05-13T23:14:22.315301+0000","last_clean":"2022-05-13T23:14:22.315301+0000","last_became_active":"2022-05-13T23:14:15.682692+0000","last_became_peered":"2022-05-13T23:14:15.682692+0000","last_unstale":"2022-05-13T23:14:22.315301+0000","last_undegraded":"2022-05-13T23:14:22.315301+0000","last_fullsized":"2022-05-13T23:14:22.315301+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:14.344206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:14.344206+0000","last_clean_scrub_stamp":"2022-05-13T23:14:14.344206+0000","objects_scrubbed":0,"log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T00:27:25.201141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":10,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-05-13T23:14:16.265569+0000","last_change":"2022-05-13T23:14:16.264351+0000","last_active":"2022-05-13T23:14:16.265569+0000","last_peered":"2022-05-13T23:14:16.265569+0000","last_clean":"2022-05-13T23:14:16.265569+0000","last_became_active":"2022-05-13T23:14:16.264023+0000","last_became_peered":"2022-05-13T23:14:16.264023+0000","last_unstale":"2022-05-13T23:14:16.265569+0000","last_undegraded":"2022-05-13T23:14:16.265569+0000","last_fullsized":"2022-05-13T23:14:16.265569+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_clean_scrub_stamp":"2022-05-13T23:14:15.659362+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T06:13:51.169049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"83'5","reported_seq":19,"reported_epoch":83,"state":"active+clean","last_fresh":"2022-05-13T23:14:23.319990+0000","last_change":"2022-05-13T23:14:15.677080+0000","last_active":"2022-05-13T23:14:23.319990+0000","last_peered":"2022-05-13T23:14:23.319990+0000","last_clean":"2022-05-13T23:14:23.319990+0000","last_became_active":"2022-05-13T23:14:15.676743+0000","last_became_peered":"2022-05-13T23:14:15.676743+0000","last_unstale":"2022-05-13T23:14:23.319990+0000","last_undegraded":"2022-05-13T23:14:23.319990+0000","last_fullsized":"2022-05-13T23:14:23.319990+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:14.344206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:14.344206+0000","last_clean_scrub_stamp":"2022-05-13T23:14:14.344206+0000","objects_scrubbed":0,"log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T09:05:30.321765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":540,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":4,"num_read_kb":3,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":10,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-05-13T23:14:16.263982+0000","last_change":"2022-05-13T23:14:16.262582+0000","last_active":"2022-05-13T23:14:16.263982+0000","last_peered":"2022-05-13T23:14:16.263982+0000","last_clean":"2022-05-13T23:14:16.263982+0000","last_became_active":"2022-05-13T23:14:16.262242+0000","last_became_peered":"2022-05-13T23:14:16.262242+0000","last_unstale":"2022-05-13T23:14:16.263982+0000","last_undegraded":"2022-05-13T23:14:16.263982+0000","last_fullsized":"2022-05-13T23:14:16.263982+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_clean_scrub_stamp":"2022-05-13T23:14:15.659362+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T02:27:28.155612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"83'17","reported_seq":29,"reported_epoch":83,"state":"active+clean","last_fresh":"2022-05-13T23:14:22.314798+0000","last_change":"2022-05-13T23:14:15.698465+0000","last_active":"2022-05-13T23:14:22.314798+0000","last_peered":"2022-05-13T23:14:22.314798+0000","last_clean":"2022-05-13T23:14:22.314798+0000","last_became_active":"2022-05-13T23:14:15.698131+0000","last_became_peered":"2022-05-13T23:14:15.698131+0000","last_unstale":"2022-05-13T23:14:22.314798+0000","last_undegraded":"2022-05-13T23:14:22.314798+0000","last_fullsized":"2022-05-13T23:14:22.314798+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:14.344206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:14.344206+0000","last_clean_scrub_stamp":"2022-05-13T23:14:14.344206+0000","objects_scrubbed":0,"log_size":17,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T03:03:48.104345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":10,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-05-13T23:14:16.261102+0000","last_change":"2022-05-13T23:14:16.260579+0000","last_active":"2022-05-13T23:14:16.261102+0000","last_peered":"2022-05-13T23:14:16.261102+0000","last_clean":"2022-05-13T23:14:16.261102+0000","last_became_active":"2022-05-13T23:14:16.260326+0000","last_became_peered":"2022-05-13T23:14:16.260326+0000","last_unstale":"2022-05-13T23:14:16.261102+0000","last_undegraded":"2022-05-13T23:14:16.261102+0000","last_fullsized":"2022-05-13T23:14:16.261102+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_clean_scrub_stamp":"2022-05-13T23:14:15.659362+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T06:53:23.092756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"83'2","reported_seq":14,"reported_epoch":83,"state":"active+clean","last_fresh":"2022-05-13T23:14:23.317375+0000","last_change":"2022-05-13T23:14:15.680655+0000","last_active":"2022-05-13T23:14:23.317375+0000","last_peered":"2022-05-13T23:14:23.317375+0000","last_clean":"2022-05-13T23:14:23.317375+0000","last_became_active":"2022-05-13T23:14:15.680357+0000","last_became_peered":"2022-05-13T23:14:15.680357+0000","last_unstale":"2022-05-13T23:14:23.317375+0000","last_undegraded":"2022-05-13T23:14:23.317375+0000","last_fullsized":"2022-05-13T23:14:23.317375+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:14.344206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:14.344206+0000","last_clean_scrub_stamp":"2022-05-13T23:14:14.344206+0000","objects_scrubbed":0,"log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T11:00:22.395993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":10,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-05-13T23:14:16.262415+0000","last_change":"2022-05-13T23:14:16.260574+0000","last_active":"2022-05-13T23:14:16.262415+0000","last_peered":"2022-05-13T23:14:16.262415+0000","last_clean":"2022-05-13T23:14:16.262415+0000","last_became_active":"2022-05-13T23:14:16.260258+0000","last_became_peered":"2022-05-13T23:14:16.260258+0000","last_unstale":"2022-05-13T23:14:16.262415+0000","last_undegraded":"2022-05-13T23:14:16.262415+0000","last_fullsized":"2022-05-13T23:14:16.262415+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_clean_scrub_stamp":"2022-05-13T23:14:15.659362+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T10:49:06.210598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"83'1","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2022-05-13T23:14:17.561161+0000","last_change":"2022-05-13T23:14:15.701322+0000","last_active":"2022-05-13T23:14:17.561161+0000","last_peered":"2022-05-13T23:14:17.561161+0000","last_clean":"2022-05-13T23:14:17.561161+0000","last_became_active":"2022-05-13T23:14:15.701007+0000","last_became_peered":"2022-05-13T23:14:15.701007+0000","last_unstale":"2022-05-13T23:14:17.561161+0000","last_undegraded":"2022-05-13T23:14:17.561161+0000","last_fullsized":"2022-05-13T23:14:17.561161+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:14.344206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:14.344206+0000","last_clean_scrub_stamp":"2022-05-13T23:14:14.344206+0000","objects_scrubbed":0,"log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T00:13:11.071695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":10,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-05-13T23:14:16.259088+0000","last_change":"2022-05-13T23:14:16.249579+0000","last_active":"2022-05-13T23:14:16.259088+0000","last_peered":"2022-05-13T23:14:16.259088+0000","last_clean":"2022-05-13T23:14:16.259088+0000","last_became_active":"2022-05-13T23:14:16.249245+0000","last_became_peered":"2022-05-13T23:14:16.249245+0000","last_unstale":"2022-05-13T23:14:16.259088+0000","last_undegraded":"2022-05-13T23:14:16.259088+0000","last_fullsized":"2022-05-13T23:14:16.259088+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_clean_scrub_stamp":"2022-05-13T23:14:15.659362+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T00:26:53.404909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"83'5","reported_seq":19,"reported_epoch":83,"state":"active+clean","last_fresh":"2022-05-13T23:14:23.323009+0000","last_change":"2022-05-13T23:14:15.699433+0000","last_active":"2022-05-13T23:14:23.323009+0000","last_peered":"2022-05-13T23:14:23.323009+0000","last_clean":"2022-05-13T23:14:23.323009+0000","last_became_active":"2022-05-13T23:14:15.698920+0000","last_became_peered":"2022-05-13T23:14:15.698920+0000","last_unstale":"2022-05-13T23:14:23.323009+0000","last_undegraded":"2022-05-13T23:14:23.323009+0000","last_fullsized":"2022-05-13T23:14:23.323009+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:14.344206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:14.344206+0000","last_clean_scrub_stamp":"2022-05-13T23:14:14.344206+0000","objects_scrubbed":0,"log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T07:56:34.276565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":540,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":10,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-05-13T23:14:16.260728+0000","last_change":"2022-05-13T23:14:16.259866+0000","last_active":"2022-05-13T23:14:16.260728+0000","last_peered":"2022-05-13T23:14:16.260728+0000","last_clean":"2022-05-13T23:14:16.260728+0000","last_became_active":"2022-05-13T23:14:16.259531+0000","last_became_peered":"2022-05-13T23:14:16.259531+0000","last_unstale":"2022-05-13T23:14:16.260728+0000","last_undegraded":"2022-05-13T23:14:16.260728+0000","last_fullsized":"2022-05-13T23:14:16.260728+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_clean_scrub_stamp":"2022-05-13T23:14:15.659362+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T07:41:46.019217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":10,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-05-13T23:14:16.261105+0000","last_change":"2022-05-13T23:14:16.259613+0000","last_active":"2022-05-13T23:14:16.261105+0000","last_peered":"2022-05-13T23:14:16.261105+0000","last_clean":"2022-05-13T23:14:16.261105+0000","last_became_active":"2022-05-13T23:14:16.259274+0000","last_became_peered":"2022-05-13T23:14:16.259274+0000","last_unstale":"2022-05-13T23:14:16.261105+0000","last_undegraded":"2022-05-13T23:14:16.261105+0000","last_fullsized":"2022-05-13T23:14:16.261105+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_clean_scrub_stamp":"2022-05-13T23:14:15.659362+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T05:36:21.174814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"83'5","reported_seq":20,"reported_epoch":83,"state":"active+clean","last_fresh":"2022-05-13T23:14:24.323218+0000","last_change":"2022-05-13T23:14:15.701315+0000","last_active":"2022-05-13T23:14:24.323218+0000","last_peered":"2022-05-13T23:14:24.323218+0000","last_clean":"2022-05-13T23:14:24.323218+0000","last_became_active":"2022-05-13T23:14:15.700955+0000","last_became_peered":"2022-05-13T23:14:15.700955+0000","last_unstale":"2022-05-13T23:14:24.323218+0000","last_undegraded":"2022-05-13T23:14:24.323218+0000","last_fullsized":"2022-05-13T23:14:24.323218+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:14.344206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:14.344206+0000","last_clean_scrub_stamp":"2022-05-13T23:14:14.344206+0000","objects_scrubbed":0,"log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T10:17:23.916475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":10,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-05-13T23:14:16.274093+0000","last_change":"2022-05-13T23:14:16.272364+0000","last_active":"2022-05-13T23:14:16.274093+0000","last_peered":"2022-05-13T23:14:16.274093+0000","last_clean":"2022-05-13T23:14:16.274093+0000","last_became_active":"2022-05-13T23:14:16.271612+0000","last_became_peered":"2022-05-13T23:14:16.271612+0000","last_unstale":"2022-05-13T23:14:16.274093+0000","last_undegraded":"2022-05-13T23:14:16.274093+0000","last_fullsized":"2022-05-13T23:14:16.274093+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_clean_scrub_stamp":"2022-05-13T23:14:15.659362+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T07:25:47.039264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"83'6","reported_seq":20,"reported_epoch":83,"state":"active+clean","last_fresh":"2022-05-13T23:14:23.317524+0000","last_change":"2022-05-13T23:14:15.676251+0000","last_active":"2022-05-13T23:14:23.317524+0000","last_peered":"2022-05-13T23:14:23.317524+0000","last_clean":"2022-05-13T23:14:23.317524+0000","last_became_active":"2022-05-13T23:14:15.675895+0000","last_became_peered":"2022-05-13T23:14:15.675895+0000","last_unstale":"2022-05-13T23:14:23.317524+0000","last_undegraded":"2022-05-13T23:14:23.317524+0000","last_fullsized":"2022-05-13T23:14:23.317524+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:14.344206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:14.344206+0000","last_clean_scrub_stamp":"2022-05-13T23:14:14.344206+0000","objects_scrubbed":0,"log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T06:03:57.398647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":343,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-05-13T23:14:16.253992+0000","last_change":"2022-05-13T23:13:02.805009+0000","last_active":"2022-05-13T23:14:16.253992+0000","last_peered":"2022-05-13T23:14:16.253992+0000","last_clean":"2022-05-13T23:14:16.253992+0000","last_became_active":"2022-05-13T23:13:02.804350+0000","last_became_peered":"2022-05-13T23:13:02.804350+0000","last_unstale":"2022-05-13T23:14:16.253992+0000","last_undegraded":"2022-05-13T23:14:16.253992+0000","last_fullsized":"2022-05-13T23:14:16.253992+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:09:32.001256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:09:32.001256+0000","last_clean_scrub_stamp":"2022-05-13T23:09:32.001256+0000","objects_scrubbed":0,"log_size":137,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T07:28:44.497144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1057856,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":10,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-05-13T23:14:16.267352+0000","last_change":"2022-05-13T23:14:16.261286+0000","last_active":"2022-05-13T23:14:16.267352+0000","last_peered":"2022-05-13T23:14:16.267352+0000","last_clean":"2022-05-13T23:14:16.267352+0000","last_became_active":"2022-05-13T23:14:16.260870+0000","last_became_peered":"2022-05-13T23:14:16.260870+0000","last_unstale":"2022-05-13T23:14:16.267352+0000","last_undegraded":"2022-05-13T23:14:16.267352+0000","last_fullsized":"2022-05-13T23:14:16.267352+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_clean_scrub_stamp":"2022-05-13T23:14:15.659362+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T08:14:34.804797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"83'4","reported_seq":15,"reported_epoch":83,"state":"active+clean","last_fresh":"2022-05-13T23:14:20.302197+0000","last_change":"2022-05-13T23:14:15.686878+0000","last_active":"2022-05-13T23:14:20.302197+0000","last_peered":"2022-05-13T23:14:20.302197+0000","last_clean":"2022-05-13T23:14:20.302197+0000","last_became_active":"2022-05-13T23:14:15.686496+0000","last_became_peered":"2022-05-13T23:14:15.686496+0000","last_unstale":"2022-05-13T23:14:20.302197+0000","last_undegraded":"2022-05-13T23:14:20.302197+0000","last_fullsized":"2022-05-13T23:14:20.302197+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:14.344206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:14.344206+0000","last_clean_scrub_stamp":"2022-05-13T23:14:14.344206+0000","objects_scrubbed":0,"log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T09:19:28.089159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"83'4","reported_seq":16,"reported_epoch":83,"state":"active+clean","last_fresh":"2022-05-13T23:14:22.316395+0000","last_change":"2022-05-13T23:14:15.698503+0000","last_active":"2022-05-13T23:14:22.316395+0000","last_peered":"2022-05-13T23:14:22.316395+0000","last_clean":"2022-05-13T23:14:22.316395+0000","last_became_active":"2022-05-13T23:14:15.698156+0000","last_became_peered":"2022-05-13T23:14:15.698156+0000","last_unstale":"2022-05-13T23:14:22.316395+0000","last_undegraded":"2022-05-13T23:14:22.316395+0000","last_fullsized":"2022-05-13T23:14:22.316395+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:14.344206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:14.344206+0000","last_clean_scrub_stamp":"2022-05-13T23:14:14.344206+0000","objects_scrubbed":0,"log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T09:03:21.133631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":10,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-05-13T23:14:16.271905+0000","last_change":"2022-05-13T23:14:16.270922+0000","last_active":"2022-05-13T23:14:16.271905+0000","last_peered":"2022-05-13T23:14:16.271905+0000","last_clean":"2022-05-13T23:14:16.271905+0000","last_became_active":"2022-05-13T23:14:16.270259+0000","last_became_peered":"2022-05-13T23:14:16.270259+0000","last_unstale":"2022-05-13T23:14:16.271905+0000","last_undegraded":"2022-05-13T23:14:16.271905+0000","last_fullsized":"2022-05-13T23:14:16.271905+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_clean_scrub_stamp":"2022-05-13T23:14:15.659362+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T02:48:22.255489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"83'7","reported_seq":18,"reported_epoch":83,"state":"active+clean","last_fresh":"2022-05-13T23:14:21.311871+0000","last_change":"2022-05-13T23:14:15.696304+0000","last_active":"2022-05-13T23:14:21.311871+0000","last_peered":"2022-05-13T23:14:21.311871+0000","last_clean":"2022-05-13T23:14:21.311871+0000","last_became_active":"2022-05-13T23:14:15.695476+0000","last_became_peered":"2022-05-13T23:14:15.695476+0000","last_unstale":"2022-05-13T23:14:21.311871+0000","last_undegraded":"2022-05-13T23:14:21.311871+0000","last_fullsized":"2022-05-13T23:14:21.311871+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:14.344206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:14.344206+0000","last_clean_scrub_stamp":"2022-05-13T23:14:14.344206+0000","objects_scrubbed":0,"log_size":7,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T08:48:32.931789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":10,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-05-13T23:14:16.262969+0000","last_change":"2022-05-13T23:14:16.257063+0000","last_active":"2022-05-13T23:14:16.262969+0000","last_peered":"2022-05-13T23:14:16.262969+0000","last_clean":"2022-05-13T23:14:16.262969+0000","last_became_active":"2022-05-13T23:14:16.256757+0000","last_became_peered":"2022-05-13T23:14:16.256757+0000","last_unstale":"2022-05-13T23:14:16.262969+0000","last_undegraded":"2022-05-13T23:14:16.262969+0000","last_fullsized":"2022-05-13T23:14:16.262969+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_clean_scrub_stamp":"2022-05-13T23:14:15.659362+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T09:24:32.369318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"83'7","reported_seq":19,"reported_epoch":83,"state":"active+clean","last_fresh":"2022-05-13T23:14:22.315844+0000","last_change":"2022-05-13T23:14:15.685568+0000","last_active":"2022-05-13T23:14:22.315844+0000","last_peered":"2022-05-13T23:14:22.315844+0000","last_clean":"2022-05-13T23:14:22.315844+0000","last_became_active":"2022-05-13T23:14:15.685203+0000","last_became_peered":"2022-05-13T23:14:15.685203+0000","last_unstale":"2022-05-13T23:14:22.315844+0000","last_undegraded":"2022-05-13T23:14:22.315844+0000","last_fullsized":"2022-05-13T23:14:22.315844+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:14.344206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:14.344206+0000","last_clean_scrub_stamp":"2022-05-13T23:14:14.344206+0000","objects_scrubbed":0,"log_size":7,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T03:55:33.271069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":10,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-05-13T23:14:16.281757+0000","last_change":"2022-05-13T23:14:16.280330+0000","last_active":"2022-05-13T23:14:16.281757+0000","last_peered":"2022-05-13T23:14:16.281757+0000","last_clean":"2022-05-13T23:14:16.281757+0000","last_became_active":"2022-05-13T23:14:16.279908+0000","last_became_peered":"2022-05-13T23:14:16.279908+0000","last_unstale":"2022-05-13T23:14:16.281757+0000","last_undegraded":"2022-05-13T23:14:16.281757+0000","last_fullsized":"2022-05-13T23:14:16.281757+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_clean_scrub_stamp":"2022-05-13T23:14:15.659362+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T03:25:42.803064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"83'2","reported_seq":13,"reported_epoch":83,"state":"active+clean","last_fresh":"2022-05-13T23:14:19.292624+0000","last_change":"2022-05-13T23:14:15.698443+0000","last_active":"2022-05-13T23:14:19.292624+0000","last_peered":"2022-05-13T23:14:19.292624+0000","last_clean":"2022-05-13T23:14:19.292624+0000","last_became_active":"2022-05-13T23:14:15.698158+0000","last_became_peered":"2022-05-13T23:14:15.698158+0000","last_unstale":"2022-05-13T23:14:19.292624+0000","last_undegraded":"2022-05-13T23:14:19.292624+0000","last_fullsized":"2022-05-13T23:14:19.292624+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:14.344206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:14.344206+0000","last_clean_scrub_stamp":"2022-05-13T23:14:14.344206+0000","objects_scrubbed":0,"log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T05:06:14.946233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":10,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-05-13T23:14:16.264874+0000","last_change":"2022-05-13T23:14:16.264110+0000","last_active":"2022-05-13T23:14:16.264874+0000","last_peered":"2022-05-13T23:14:16.264874+0000","last_clean":"2022-05-13T23:14:16.264874+0000","last_became_active":"2022-05-13T23:14:16.263830+0000","last_became_peered":"2022-05-13T23:14:16.263830+0000","last_unstale":"2022-05-13T23:14:16.264874+0000","last_undegraded":"2022-05-13T23:14:16.264874+0000","last_fullsized":"2022-05-13T23:14:16.264874+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_clean_scrub_stamp":"2022-05-13T23:14:15.659362+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T00:06:26.724597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"83'7","reported_seq":18,"reported_epoch":83,"state":"active+clean","last_fresh":"2022-05-13T23:14:22.318011+0000","last_change":"2022-05-13T23:14:15.693926+0000","last_active":"2022-05-13T23:14:22.318011+0000","last_peered":"2022-05-13T23:14:22.318011+0000","last_clean":"2022-05-13T23:14:22.318011+0000","last_became_active":"2022-05-13T23:14:15.692163+0000","last_became_peered":"2022-05-13T23:14:15.692163+0000","last_unstale":"2022-05-13T23:14:22.318011+0000","last_undegraded":"2022-05-13T23:14:22.318011+0000","last_fullsized":"2022-05-13T23:14:22.318011+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:14.344206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:14.344206+0000","last_clean_scrub_stamp":"2022-05-13T23:14:14.344206+0000","objects_scrubbed":0,"log_size":7,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T03:00:12.522714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":10,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-05-13T23:14:16.266559+0000","last_change":"2022-05-13T23:14:16.265409+0000","last_active":"2022-05-13T23:14:16.266559+0000","last_peered":"2022-05-13T23:14:16.266559+0000","last_clean":"2022-05-13T23:14:16.266559+0000","last_became_active":"2022-05-13T23:14:16.265151+0000","last_became_peered":"2022-05-13T23:14:16.265151+0000","last_unstale":"2022-05-13T23:14:16.266559+0000","last_undegraded":"2022-05-13T23:14:16.266559+0000","last_fullsized":"2022-05-13T23:14:16.266559+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_clean_scrub_stamp":"2022-05-13T23:14:15.659362+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T04:05:43.762623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"83'3","reported_seq":14,"reported_epoch":83,"state":"active+clean","last_fresh":"2022-05-13T23:14:22.855083+0000","last_change":"2022-05-13T23:14:15.681975+0000","last_active":"2022-05-13T23:14:22.855083+0000","last_peered":"2022-05-13T23:14:22.855083+0000","last_clean":"2022-05-13T23:14:22.855083+0000","last_became_active":"2022-05-13T23:14:15.681391+0000","last_became_peered":"2022-05-13T23:14:15.681391+0000","last_unstale":"2022-05-13T23:14:22.855083+0000","last_undegraded":"2022-05-13T23:14:22.855083+0000","last_fullsized":"2022-05-13T23:14:22.855083+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:14.344206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:14.344206+0000","last_clean_scrub_stamp":"2022-05-13T23:14:14.344206+0000","objects_scrubbed":0,"log_size":3,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T06:56:25.732024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":10,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-05-13T23:14:16.274936+0000","last_change":"2022-05-13T23:14:16.272443+0000","last_active":"2022-05-13T23:14:16.274936+0000","last_peered":"2022-05-13T23:14:16.274936+0000","last_clean":"2022-05-13T23:14:16.274936+0000","last_became_active":"2022-05-13T23:14:16.272031+0000","last_became_peered":"2022-05-13T23:14:16.272031+0000","last_unstale":"2022-05-13T23:14:16.274936+0000","last_undegraded":"2022-05-13T23:14:16.274936+0000","last_fullsized":"2022-05-13T23:14:16.274936+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_clean_scrub_stamp":"2022-05-13T23:14:15.659362+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T10:23:55.694108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"83'4","reported_seq":16,"reported_epoch":83,"state":"active+clean","last_fresh":"2022-05-13T23:14:19.292144+0000","last_change":"2022-05-13T23:14:15.681877+0000","last_active":"2022-05-13T23:14:19.292144+0000","last_peered":"2022-05-13T23:14:19.292144+0000","last_clean":"2022-05-13T23:14:19.292144+0000","last_became_active":"2022-05-13T23:14:15.681579+0000","last_became_peered":"2022-05-13T23:14:15.681579+0000","last_unstale":"2022-05-13T23:14:19.292144+0000","last_undegraded":"2022-05-13T23:14:19.292144+0000","last_fullsized":"2022-05-13T23:14:19.292144+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:14.344206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:14.344206+0000","last_clean_scrub_stamp":"2022-05-13T23:14:14.344206+0000","objects_scrubbed":0,"log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T07:00:11.132049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":10,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-05-13T23:14:16.267797+0000","last_change":"2022-05-13T23:14:16.266877+0000","last_active":"2022-05-13T23:14:16.267797+0000","last_peered":"2022-05-13T23:14:16.267797+0000","last_clean":"2022-05-13T23:14:16.267797+0000","last_became_active":"2022-05-13T23:14:16.266551+0000","last_became_peered":"2022-05-13T23:14:16.266551+0000","last_unstale":"2022-05-13T23:14:16.267797+0000","last_undegraded":"2022-05-13T23:14:16.267797+0000","last_fullsized":"2022-05-13T23:14:16.267797+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_clean_scrub_stamp":"2022-05-13T23:14:15.659362+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T05:57:29.394410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"83'5","reported_seq":21,"reported_epoch":83,"state":"active+clean","last_fresh":"2022-05-13T23:14:24.323280+0000","last_change":"2022-05-13T23:14:15.699586+0000","last_active":"2022-05-13T23:14:24.323280+0000","last_peered":"2022-05-13T23:14:24.323280+0000","last_clean":"2022-05-13T23:14:24.323280+0000","last_became_active":"2022-05-13T23:14:15.699215+0000","last_became_peered":"2022-05-13T23:14:15.699215+0000","last_unstale":"2022-05-13T23:14:24.323280+0000","last_undegraded":"2022-05-13T23:14:24.323280+0000","last_fullsized":"2022-05-13T23:14:24.323280+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:14.344206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:14.344206+0000","last_clean_scrub_stamp":"2022-05-13T23:14:14.344206+0000","objects_scrubbed":0,"log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T04:14:54.395550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1094,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":10,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-05-13T23:14:16.267333+0000","last_change":"2022-05-13T23:14:16.265730+0000","last_active":"2022-05-13T23:14:16.267333+0000","last_peered":"2022-05-13T23:14:16.267333+0000","last_clean":"2022-05-13T23:14:16.267333+0000","last_became_active":"2022-05-13T23:14:16.265435+0000","last_became_peered":"2022-05-13T23:14:16.265435+0000","last_unstale":"2022-05-13T23:14:16.267333+0000","last_undegraded":"2022-05-13T23:14:16.267333+0000","last_fullsized":"2022-05-13T23:14:16.267333+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_clean_scrub_stamp":"2022-05-13T23:14:15.659362+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T08:55:57.574132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"83'3","reported_seq":15,"reported_epoch":83,"state":"active+clean","last_fresh":"2022-05-13T23:14:22.316840+0000","last_change":"2022-05-13T23:14:15.682402+0000","last_active":"2022-05-13T23:14:22.316840+0000","last_peered":"2022-05-13T23:14:22.316840+0000","last_clean":"2022-05-13T23:14:22.316840+0000","last_became_active":"2022-05-13T23:14:15.682105+0000","last_became_peered":"2022-05-13T23:14:15.682105+0000","last_unstale":"2022-05-13T23:14:22.316840+0000","last_undegraded":"2022-05-13T23:14:22.316840+0000","last_fullsized":"2022-05-13T23:14:22.316840+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:14.344206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:14.344206+0000","last_clean_scrub_stamp":"2022-05-13T23:14:14.344206+0000","objects_scrubbed":0,"log_size":3,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T10:37:55.092658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":10,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-05-13T23:14:16.264851+0000","last_change":"2022-05-13T23:14:16.263001+0000","last_active":"2022-05-13T23:14:16.264851+0000","last_peered":"2022-05-13T23:14:16.264851+0000","last_clean":"2022-05-13T23:14:16.264851+0000","last_became_active":"2022-05-13T23:14:16.262646+0000","last_became_peered":"2022-05-13T23:14:16.262646+0000","last_unstale":"2022-05-13T23:14:16.264851+0000","last_undegraded":"2022-05-13T23:14:16.264851+0000","last_fullsized":"2022-05-13T23:14:16.264851+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_clean_scrub_stamp":"2022-05-13T23:14:15.659362+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T05:38:53.024966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"83'5","reported_seq":16,"reported_epoch":83,"state":"active+clean","last_fresh":"2022-05-13T23:14:22.315583+0000","last_change":"2022-05-13T23:14:15.690434+0000","last_active":"2022-05-13T23:14:22.315583+0000","last_peered":"2022-05-13T23:14:22.315583+0000","last_clean":"2022-05-13T23:14:22.315583+0000","last_became_active":"2022-05-13T23:14:15.690032+0000","last_became_peered":"2022-05-13T23:14:15.690032+0000","last_unstale":"2022-05-13T23:14:22.315583+0000","last_undegraded":"2022-05-13T23:14:22.315583+0000","last_fullsized":"2022-05-13T23:14:22.315583+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:14.344206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:14.344206+0000","last_clean_scrub_stamp":"2022-05-13T23:14:14.344206+0000","objects_scrubbed":0,"log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T06:39:08.251311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":10,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-05-13T23:14:16.265645+0000","last_change":"2022-05-13T23:14:16.264791+0000","last_active":"2022-05-13T23:14:16.265645+0000","last_peered":"2022-05-13T23:14:16.265645+0000","last_clean":"2022-05-13T23:14:16.265645+0000","last_became_active":"2022-05-13T23:14:16.264461+0000","last_became_peered":"2022-05-13T23:14:16.264461+0000","last_unstale":"2022-05-13T23:14:16.265645+0000","last_undegraded":"2022-05-13T23:14:16.265645+0000","last_fullsized":"2022-05-13T23:14:16.265645+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_clean_scrub_stamp":"2022-05-13T23:14:15.659362+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T03:02:04.280220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":10,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-05-13T23:14:16.261071+0000","last_change":"2022-05-13T23:14:16.260468+0000","last_active":"2022-05-13T23:14:16.261071+0000","last_peered":"2022-05-13T23:14:16.261071+0000","last_clean":"2022-05-13T23:14:16.261071+0000","last_became_active":"2022-05-13T23:14:16.260211+0000","last_became_peered":"2022-05-13T23:14:16.260211+0000","last_unstale":"2022-05-13T23:14:16.261071+0000","last_undegraded":"2022-05-13T23:14:16.261071+0000","last_fullsized":"2022-05-13T23:14:16.261071+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_clean_scrub_stamp":"2022-05-13T23:14:15.659362+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T05:26:57.769099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":10,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-05-13T23:14:16.254999+0000","last_change":"2022-05-13T23:14:16.253997+0000","last_active":"2022-05-13T23:14:16.254999+0000","last_peered":"2022-05-13T23:14:16.254999+0000","last_clean":"2022-05-13T23:14:16.254999+0000","last_became_active":"2022-05-13T23:14:16.253685+0000","last_became_peered":"2022-05-13T23:14:16.253685+0000","last_unstale":"2022-05-13T23:14:16.254999+0000","last_undegraded":"2022-05-13T23:14:16.254999+0000","last_fullsized":"2022-05-13T23:14:16.254999+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_clean_scrub_stamp":"2022-05-13T23:14:15.659362+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T07:42:26.765244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":10,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-05-13T23:14:16.266256+0000","last_change":"2022-05-13T23:14:16.265207+0000","last_active":"2022-05-13T23:14:16.266256+0000","last_peered":"2022-05-13T23:14:16.266256+0000","last_clean":"2022-05-13T23:14:16.266256+0000","last_became_active":"2022-05-13T23:14:16.264881+0000","last_became_peered":"2022-05-13T23:14:16.264881+0000","last_unstale":"2022-05-13T23:14:16.266256+0000","last_undegraded":"2022-05-13T23:14:16.266256+0000","last_fullsized":"2022-05-13T23:14:16.266256+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_clean_scrub_stamp":"2022-05-13T23:14:15.659362+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T04:01:48.894233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":10,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-05-13T23:14:16.273903+0000","last_change":"2022-05-13T23:14:16.264286+0000","last_active":"2022-05-13T23:14:16.273903+0000","last_peered":"2022-05-13T23:14:16.273903+0000","last_clean":"2022-05-13T23:14:16.273903+0000","last_became_active":"2022-05-13T23:14:16.263978+0000","last_became_peered":"2022-05-13T23:14:16.263978+0000","last_unstale":"2022-05-13T23:14:16.273903+0000","last_undegraded":"2022-05-13T23:14:16.273903+0000","last_fullsized":"2022-05-13T23:14:16.273903+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_clean_scrub_stamp":"2022-05-13T23:14:15.659362+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T03:11:10.799827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":10,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-05-13T23:14:16.268810+0000","last_change":"2022-05-13T23:14:16.265421+0000","last_active":"2022-05-13T23:14:16.268810+0000","last_peered":"2022-05-13T23:14:16.268810+0000","last_clean":"2022-05-13T23:14:16.268810+0000","last_became_active":"2022-05-13T23:14:16.265057+0000","last_became_peered":"2022-05-13T23:14:16.265057+0000","last_unstale":"2022-05-13T23:14:16.268810+0000","last_undegraded":"2022-05-13T23:14:16.268810+0000","last_fullsized":"2022-05-13T23:14:16.268810+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_clean_scrub_stamp":"2022-05-13T23:14:15.659362+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T07:45:53.177341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":10,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-05-13T23:14:16.269637+0000","last_change":"2022-05-13T23:14:16.266317+0000","last_active":"2022-05-13T23:14:16.269637+0000","last_peered":"2022-05-13T23:14:16.269637+0000","last_clean":"2022-05-13T23:14:16.269637+0000","last_became_active":"2022-05-13T23:14:16.265959+0000","last_became_peered":"2022-05-13T23:14:16.265959+0000","last_unstale":"2022-05-13T23:14:16.269637+0000","last_undegraded":"2022-05-13T23:14:16.269637+0000","last_fullsized":"2022-05-13T23:14:16.269637+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_clean_scrub_stamp":"2022-05-13T23:14:15.659362+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T03:37:42.277033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":10,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-05-13T23:14:16.255848+0000","last_change":"2022-05-13T23:14:16.254350+0000","last_active":"2022-05-13T23:14:16.255848+0000","last_peered":"2022-05-13T23:14:16.255848+0000","last_clean":"2022-05-13T23:14:16.255848+0000","last_became_active":"2022-05-13T23:14:16.253940+0000","last_became_peered":"2022-05-13T23:14:16.253940+0000","last_unstale":"2022-05-13T23:14:16.255848+0000","last_undegraded":"2022-05-13T23:14:16.255848+0000","last_fullsized":"2022-05-13T23:14:16.255848+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_clean_scrub_stamp":"2022-05-13T23:14:15.659362+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T10:08:56.052316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":10,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-05-13T23:14:16.267243+0000","last_change":"2022-05-13T23:14:16.263153+0000","last_active":"2022-05-13T23:14:16.267243+0000","last_peered":"2022-05-13T23:14:16.267243+0000","last_clean":"2022-05-13T23:14:16.267243+0000","last_became_active":"2022-05-13T23:14:16.262652+0000","last_became_peered":"2022-05-13T23:14:16.262652+0000","last_unstale":"2022-05-13T23:14:16.267243+0000","last_undegraded":"2022-05-13T23:14:16.267243+0000","last_fullsized":"2022-05-13T23:14:16.267243+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_clean_scrub_stamp":"2022-05-13T23:14:15.659362+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T07:53:46.499880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":10,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-05-13T23:14:16.271182+0000","last_change":"2022-05-13T23:14:16.270358+0000","last_active":"2022-05-13T23:14:16.271182+0000","last_peered":"2022-05-13T23:14:16.271182+0000","last_clean":"2022-05-13T23:14:16.271182+0000","last_became_active":"2022-05-13T23:14:16.270099+0000","last_became_peered":"2022-05-13T23:14:16.270099+0000","last_unstale":"2022-05-13T23:14:16.271182+0000","last_undegraded":"2022-05-13T23:14:16.271182+0000","last_fullsized":"2022-05-13T23:14:16.271182+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_clean_scrub_stamp":"2022-05-13T23:14:15.659362+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T00:42:07.782569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":10,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-05-13T23:14:16.253696+0000","last_change":"2022-05-13T23:14:16.251188+0000","last_active":"2022-05-13T23:14:16.253696+0000","last_peered":"2022-05-13T23:14:16.253696+0000","last_clean":"2022-05-13T23:14:16.253696+0000","last_became_active":"2022-05-13T23:14:16.250816+0000","last_became_peered":"2022-05-13T23:14:16.250816+0000","last_unstale":"2022-05-13T23:14:16.253696+0000","last_undegraded":"2022-05-13T23:14:16.253696+0000","last_fullsized":"2022-05-13T23:14:16.253696+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_clean_scrub_stamp":"2022-05-13T23:14:15.659362+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T00:58:14.898104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":10,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-05-13T23:14:16.265927+0000","last_change":"2022-05-13T23:14:16.264125+0000","last_active":"2022-05-13T23:14:16.265927+0000","last_peered":"2022-05-13T23:14:16.265927+0000","last_clean":"2022-05-13T23:14:16.265927+0000","last_became_active":"2022-05-13T23:14:16.263806+0000","last_became_peered":"2022-05-13T23:14:16.263806+0000","last_unstale":"2022-05-13T23:14:16.265927+0000","last_undegraded":"2022-05-13T23:14:16.265927+0000","last_fullsized":"2022-05-13T23:14:16.265927+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_clean_scrub_stamp":"2022-05-13T23:14:15.659362+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T09:26:04.559765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":10,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-05-13T23:14:16.268902+0000","last_change":"2022-05-13T23:14:16.266991+0000","last_active":"2022-05-13T23:14:16.268902+0000","last_peered":"2022-05-13T23:14:16.268902+0000","last_clean":"2022-05-13T23:14:16.268902+0000","last_became_active":"2022-05-13T23:14:16.265551+0000","last_became_peered":"2022-05-13T23:14:16.265551+0000","last_unstale":"2022-05-13T23:14:16.268902+0000","last_undegraded":"2022-05-13T23:14:16.268902+0000","last_fullsized":"2022-05-13T23:14:16.268902+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_clean_scrub_stamp":"2022-05-13T23:14:15.659362+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T00:40:56.664387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":10,"reported_epoch":82,"state":"active+clean","last_fresh":"2022-05-13T23:14:16.269805+0000","last_change":"2022-05-13T23:14:16.267302+0000","last_active":"2022-05-13T23:14:16.269805+0000","last_peered":"2022-05-13T23:14:16.269805+0000","last_clean":"2022-05-13T23:14:16.269805+0000","last_became_active":"2022-05-13T23:14:16.266151+0000","last_became_peered":"2022-05-13T23:14:16.266151+0000","last_unstale":"2022-05-13T23:14:16.269805+0000","last_undegraded":"2022-05-13T23:14:16.269805+0000","last_fullsized":"2022-05-13T23:14:16.269805+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_clean_scrub_stamp":"2022-05-13T23:14:15.659362+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T07:30:04.546337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1057856,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":3194880,"data_stored":3173568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":7626,"num_objects":98,"num_object_clones":0,"num_object_copies":294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98,"num_whiteouts":0,"num_read":50,"num_read_kb":42,"num_write":205,"num_write_kb":57,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":393216,"data_stored":22878,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":179,"ondisk_log_size":179,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":78,"seq":335007449103,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":5880,"kb_used_data":880,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":93739272,"statfs":{"total":95995035648,"available":95989014528,"internally_reserved":0,"allocated":901120,"data_stored":425930,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri May 13 23:14:21 2022","interfaces":[{"interface":"back","average":{"1min":0.60299999999999998,"5min":0.60299999999999998,"15min":0.60299999999999998},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.84699999999999998,"5min":0.84699999999999998,"15min":0.84699999999999998},"last":0.64600000000000002},{"interface":"front","average":{"1min":0.64200000000000002,"5min":0.64200000000000002,"15min":0.64200000000000002},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":0.81599999999999995,"5min":0.81599999999999995,"15min":0.81599999999999995},"last":0.58999999999999997}]},{"osd":1,"last update":"Fri May 13 23:14:21 2022","interfaces":[{"interface":"back","average":{"1min":0.59699999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":0.81499999999999995,"5min":0.81499999999999995,"15min":0.81499999999999995},"last":0.61199999999999999},{"interface":"front","average":{"1min":0.58899999999999997,"5min":0.58899999999999997,"15min":0.58899999999999997},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.89500000000000002,"5min":0.89500000000000002,"15min":0.89500000000000002},"last":0.47899999999999998}]},{"osd":2,"last update":"Fri May 13 23:14:21 2022","interfaces":[{"interface":"back","average":{"1min":0.46500000000000002,"5min":0.46500000000000002,"15min":0.46500000000000002},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.76900000000000002,"5min":0.76900000000000002,"15min":0.76900000000000002},"last":0.503},{"interface":"front","average":{"1min":0.46800000000000003,"5min":0.46800000000000003,"15min":0.46800000000000003},"min":{"1min":0.317,"5min":0.317,"15min":0.317},"max":{"1min":0.71299999999999997,"5min":0.71299999999999997,"15min":0.71299999999999997},"last":0.42299999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42999999999999999}]},{"osd":4,"last update":"Fri May 13 23:14:21 2022","interfaces":[{"interface":"back","average":{"1min":0.622,"5min":0.622,"15min":0.622},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":1.0289999999999999,"5min":1.0289999999999999,"15min":1.0289999999999999},"last":0.66100000000000003},{"interface":"front","average":{"1min":0.63400000000000001,"5min":0.63400000000000001,"15min":0.63400000000000001},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":0.93700000000000006,"5min":0.93700000000000006,"15min":0.93700000000000006},"last":0.55300000000000005}]},{"osd":5,"last update":"Fri May 13 23:14:21 2022","interfaces":[{"interface":"back","average":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.59099999999999997},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":1.3799999999999999,"5min":1.3799999999999999,"15min":1.3799999999999999},"last":0.45500000000000002},{"interface":"front","average":{"1min":0.60299999999999998,"5min":0.60299999999999998,"15min":0.60299999999999998},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.88,"5min":0.88,"15min":0.88},"last":0.57199999999999995}]},{"osd":6,"last update":"Fri May 13 23:14:21 2022","interfaces":[{"interface":"back","average":{"1min":0.73999999999999999,"5min":0.73999999999999999,"15min":0.73999999999999999},"min":{"1min":0.48499999999999999,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":1.0669999999999999,"5min":1.0669999999999999,"15min":1.0669999999999999},"last":0.69999999999999996},{"interface":"front","average":{"1min":0.79200000000000004,"5min":0.79200000000000004,"15min":0.79200000000000004},"min":{"1min":0.49099999999999999,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":1.655,"5min":1.655,"15min":1.655},"last":0.67500000000000004}]},{"osd":7,"last update":"Fri May 13 23:14:21 2022","interfaces":[{"interface":"back","average":{"1min":0.72299999999999998,"5min":0.72299999999999998,"15min":0.72299999999999998},"min":{"1min":0.502,"5min":0.502,"15min":0.502},"max":{"1min":1.107,"5min":1.107,"15min":1.107},"last":0.68999999999999995},{"interface":"front","average":{"1min":0.745,"5min":0.745,"15min":0.745},"min":{"1min":0.46999999999999997,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":1.3620000000000001,"5min":1.3620000000000001,"15min":1.3620000000000001},"last":0.745}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.78000000000000003}]},{"osd":9,"last update":"Fri May 13 23:14:21 2022","interfaces":[{"interface":"back","average":{"1min":0.83699999999999997,"5min":0.83699999999999997,"15min":0.83699999999999997},"min":{"1min":0.57099999999999995,"5min":0.57099999999999995,"15min":0.57099999999999995},"max":{"1min":1.7729999999999999,"5min":1.7729999999999999,"15min":1.7729999999999999},"last":0.79500000000000004},{"interface":"front","average":{"1min":0.79500000000000004,"5min":0.79500000000000004,"15min":0.79500000000000004},"min":{"1min":0.58099999999999996,"5min":0.58099999999999996,"15min":0.58099999999999996},"max":{"1min":1.3220000000000001,"5min":1.3220000000000001,"15min":1.3220000000000001},"last":0.73199999999999998}]},{"osd":10,"last update":"Fri May 13 23:14:21 2022","interfaces":[{"interface":"back","average":{"1min":0.75700000000000001,"5min":0.75700000000000001,"15min":0.75700000000000001},"min":{"1min":0.54100000000000004,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":1.26,"5min":1.26,"15min":1.26},"last":0.81200000000000006},{"interface":"front","average":{"1min":0.77600000000000002,"5min":0.77600000000000002,"15min":0.77600000000000002},"min":{"1min":0.51900000000000002,"5min":0.51900000000000002,"15min":0.51900000000000002},"max":{"1min":1.7350000000000001,"5min":1.7350000000000001,"15min":1.7350000000000001},"last":0.71299999999999997}]}]},{"osd":10,"up_from":71,"seq":304942678035,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6696,"kb_used_data":1440,"kb_used_omap":0,"kb_used_meta":5248,"kb_avail":93738456,"statfs":{"total":95995035648,"available":95988178944,"internally_reserved":0,"allocated":1474560,"data_stored":956615,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5373952},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55600000000000005}]},{"osd":1,"last update":"Fri May 13 23:14:05 2022","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.72899999999999998,"5min":0.72899999999999998,"15min":0.72899999999999998},"last":0.53300000000000003},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.57999999999999996,"15min":0.57999999999999996},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.83099999999999996,"5min":0.83099999999999996,"15min":0.83099999999999996},"last":0.57899999999999996}]},{"osd":2,"last update":"Fri May 13 23:14:05 2022","interfaces":[{"interface":"back","average":{"1min":0.69499999999999995,"5min":0.69499999999999995,"15min":0.69499999999999995},"min":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":1.3280000000000001,"5min":1.3280000000000001,"15min":1.3280000000000001},"last":0.69699999999999995},{"interface":"front","average":{"1min":0.70799999999999996,"5min":0.70799999999999996,"15min":0.70799999999999996},"min":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":0.88800000000000001,"5min":0.88800000000000001,"15min":0.88800000000000001},"last":0.72599999999999998}]},{"osd":3,"last update":"Fri May 13 23:14:05 2022","interfaces":[{"interface":"back","average":{"1min":0.73999999999999999,"5min":0.73999999999999999,"15min":0.73999999999999999},"min":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"max":{"1min":0.90100000000000002,"5min":0.90100000000000002,"15min":0.90100000000000002},"last":0.63400000000000001},{"interface":"front","average":{"1min":0.77000000000000002,"5min":0.77000000000000002,"15min":0.77000000000000002},"min":{"1min":0.60199999999999998,"5min":0.60199999999999998,"15min":0.60199999999999998},"max":{"1min":0.93500000000000005,"5min":0.93500000000000005,"15min":0.93500000000000005},"last":0.753}]},{"osd":4,"last update":"Fri May 13 23:14:05 2022","interfaces":[{"interface":"back","average":{"1min":0.68000000000000005,"5min":0.68000000000000005,"15min":0.68000000000000005},"min":{"1min":0.432,"5min":0.432,"15min":0.432},"max":{"1min":0.97699999999999998,"5min":0.97699999999999998,"15min":0.97699999999999998},"last":0.70799999999999996},{"interface":"front","average":{"1min":0.65200000000000002,"5min":0.65200000000000002,"15min":0.65200000000000002},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":1.0720000000000001,"5min":1.0720000000000001,"15min":1.0720000000000001},"last":0.82099999999999995}]},{"osd":5,"last update":"Fri May 13 23:14:05 2022","interfaces":[{"interface":"back","average":{"1min":0.78400000000000003,"5min":0.78400000000000003,"15min":0.78400000000000003},"min":{"1min":0.496,"5min":0.496,"15min":0.496},"max":{"1min":1.4299999999999999,"5min":1.4299999999999999,"15min":1.4299999999999999},"last":0.77800000000000002},{"interface":"front","average":{"1min":0.79800000000000004,"5min":0.79800000000000004,"15min":0.79800000000000004},"min":{"1min":0.55400000000000005,"5min":0.55400000000000005,"15min":0.55400000000000005},"max":{"1min":1.0609999999999999,"5min":1.0609999999999999,"15min":1.0609999999999999},"last":0.79600000000000004}]},{"osd":6,"last update":"Fri May 13 23:14:05 2022","interfaces":[{"interface":"back","average":{"1min":0.88700000000000001,"5min":0.88700000000000001,"15min":0.88700000000000001},"min":{"1min":0.64600000000000002,"5min":0.64600000000000002,"15min":0.64600000000000002},"max":{"1min":1.157,"5min":1.157,"15min":1.157},"last":0.67500000000000004},{"interface":"front","average":{"1min":0.88500000000000001,"5min":0.88500000000000001,"15min":0.88500000000000001},"min":{"1min":0.60199999999999998,"5min":0.60199999999999998,"15min":0.60199999999999998},"max":{"1min":1.216,"5min":1.216,"15min":1.216},"last":0.874}]},{"osd":7,"last update":"Fri May 13 23:14:05 2022","interfaces":[{"interface":"back","average":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":1.585,"5min":1.585,"15min":1.585},"last":0.496},{"interface":"front","average":{"1min":0.80300000000000005,"5min":0.80300000000000005,"15min":0.80300000000000005},"min":{"1min":0.47999999999999998,"5min":0.47999999999999998,"15min":0.47999999999999998},"max":{"1min":1.0469999999999999,"5min":1.0469999999999999,"15min":1.0469999999999999},"last":0.502}]},{"osd":8,"last update":"Fri May 13 23:14:05 2022","interfaces":[{"interface":"back","average":{"1min":0.86399999999999999,"5min":0.86399999999999999,"15min":0.86399999999999999},"min":{"1min":0.65900000000000003,"5min":0.65900000000000003,"15min":0.65900000000000003},"max":{"1min":1.163,"5min":1.163,"15min":1.163},"last":0.84099999999999997},{"interface":"front","average":{"1min":0.88200000000000001,"5min":0.88200000000000001,"15min":0.88200000000000001},"min":{"1min":0.51400000000000001,"5min":0.51400000000000001,"15min":0.51400000000000001},"max":{"1min":1.476,"5min":1.476,"15min":1.476},"last":0.89100000000000001}]},{"osd":9,"last update":"Fri May 13 23:14:05 2022","interfaces":[{"interface":"back","average":{"1min":0.94999999999999996,"5min":0.94999999999999996,"15min":0.94999999999999996},"min":{"1min":0.68999999999999995,"5min":0.68999999999999995,"15min":0.68999999999999995},"max":{"1min":1.369,"5min":1.369,"15min":1.369},"last":0.92600000000000005},{"interface":"front","average":{"1min":0.96799999999999997,"5min":0.96799999999999997,"15min":0.96799999999999997},"min":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":1.819,"5min":1.819,"15min":1.819},"last":0.93899999999999995}]},{"osd":11,"last update":"Fri May 13 23:14:23 2022","interfaces":[{"interface":"back","average":{"1min":0.84299999999999997,"5min":0.84299999999999997,"15min":0.84299999999999997},"min":{"1min":0.63300000000000001,"5min":0.63300000000000001,"15min":0.63300000000000001},"max":{"1min":1.1060000000000001,"5min":1.1060000000000001,"15min":1.1060000000000001},"last":0.85399999999999998},{"interface":"front","average":{"1min":0.84999999999999998,"5min":0.84999999999999998,"15min":0.84999999999999998},"min":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"max":{"1min":1.139,"5min":1.139,"15min":1.139},"last":0.96599999999999997}]}]},{"osd":9,"up_from":66,"seq":283467841559,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6096,"kb_used_data":904,"kb_used_omap":0,"kb_used_meta":5184,"kb_avail":93739056,"statfs":{"total":95995035648,"available":95988793344,"internally_reserved":0,"allocated":925696,"data_stored":425908,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5308416},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri May 13 23:13:37 2022","interfaces":[{"interface":"back","average":{"1min":0.65200000000000002,"5min":0.65200000000000002,"15min":0.65200000000000002},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":1.024,"5min":1.024,"15min":1.024},"last":0.64600000000000002},{"interface":"front","average":{"1min":0.72299999999999998,"5min":0.72299999999999998,"15min":0.72299999999999998},"min":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":1.044,"5min":1.044,"15min":1.044},"last":0.57999999999999996}]},{"osd":1,"last update":"Fri May 13 23:13:37 2022","interfaces":[{"interface":"back","average":{"1min":0.755,"5min":0.755,"15min":0.755},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":1.3029999999999999,"5min":1.3029999999999999,"15min":1.3029999999999999},"last":0.68799999999999994},{"interface":"front","average":{"1min":0.92000000000000004,"5min":0.92000000000000004,"15min":0.92000000000000004},"min":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":1.498,"5min":1.498,"15min":1.498},"last":0.76900000000000002}]},{"osd":2,"last update":"Fri May 13 23:13:37 2022","interfaces":[{"interface":"back","average":{"1min":0.89900000000000002,"5min":0.89900000000000002,"15min":0.89900000000000002},"min":{"1min":0.58699999999999997,"5min":0.58699999999999997,"15min":0.58699999999999997},"max":{"1min":1.4419999999999999,"5min":1.4419999999999999,"15min":1.4419999999999999},"last":0.89200000000000002},{"interface":"front","average":{"1min":0.82699999999999996,"5min":0.82699999999999996,"15min":0.82699999999999996},"min":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"max":{"1min":1.5309999999999999,"5min":1.5309999999999999,"15min":1.5309999999999999},"last":0.85299999999999998}]},{"osd":3,"last update":"Fri May 13 23:13:37 2022","interfaces":[{"interface":"back","average":{"1min":0.878,"5min":0.878,"15min":0.878},"min":{"1min":0.51200000000000001,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":1.2410000000000001,"5min":1.2410000000000001,"15min":1.2410000000000001},"last":0.86399999999999999},{"interface":"front","average":{"1min":1.002,"5min":1.002,"15min":1.002},"min":{"1min":0.622,"5min":0.622,"15min":0.622},"max":{"1min":2.2069999999999999,"5min":2.2069999999999999,"15min":2.2069999999999999},"last":0.81499999999999995}]},{"osd":4,"last update":"Fri May 13 23:13:37 2022","interfaces":[{"interface":"back","average":{"1min":0.93000000000000005,"5min":0.93000000000000005,"15min":0.93000000000000005},"min":{"1min":0.47699999999999998,"5min":0.47699999999999998,"15min":0.47699999999999998},"max":{"1min":1.5600000000000001,"5min":1.5600000000000001,"15min":1.5600000000000001},"last":0.745},{"interface":"front","average":{"1min":0.95799999999999996,"5min":0.95799999999999996,"15min":0.95799999999999996},"min":{"1min":0.58899999999999997,"5min":0.58899999999999997,"15min":0.58899999999999997},"max":{"1min":1.3320000000000001,"5min":1.3320000000000001,"15min":1.3320000000000001},"last":0.96099999999999997}]},{"osd":5,"last update":"Fri May 13 23:13:37 2022","interfaces":[{"interface":"back","average":{"1min":0.97099999999999997,"5min":0.97099999999999997,"15min":0.97099999999999997},"min":{"1min":0.56299999999999994,"5min":0.56299999999999994,"15min":0.56299999999999994},"max":{"1min":1.593,"5min":1.593,"15min":1.593},"last":0.80600000000000005},{"interface":"front","average":{"1min":1.02,"5min":1.02,"15min":1.02},"min":{"1min":0.68600000000000005,"5min":0.68600000000000005,"15min":0.68600000000000005},"max":{"1min":1.3700000000000001,"5min":1.3700000000000001,"15min":1.3700000000000001},"last":1}]},{"osd":6,"last update":"Fri May 13 23:13:37 2022","interfaces":[{"interface":"back","average":{"1min":1.1399999999999999,"5min":1.1399999999999999,"15min":1.1399999999999999},"min":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"max":{"1min":2.254,"5min":2.254,"15min":2.254},"last":1.052},{"interface":"front","average":{"1min":1.0609999999999999,"5min":1.0609999999999999,"15min":1.0609999999999999},"min":{"1min":0.70599999999999996,"5min":0.70599999999999996,"15min":0.70599999999999996},"max":{"1min":1.649,"5min":1.649,"15min":1.649},"last":1.0289999999999999}]},{"osd":7,"last update":"Fri May 13 23:13:37 2022","interfaces":[{"interface":"back","average":{"1min":1.119,"5min":1.119,"15min":1.119},"min":{"1min":0.63,"5min":0.63,"15min":0.63},"max":{"1min":2.101,"5min":2.101,"15min":2.101},"last":1.0640000000000001},{"interface":"front","average":{"1min":1.1240000000000001,"5min":1.1240000000000001,"15min":1.1240000000000001},"min":{"1min":0.69599999999999995,"5min":0.69599999999999995,"15min":0.69599999999999995},"max":{"1min":1.895,"5min":1.895,"15min":1.895},"last":0.96999999999999997}]},{"osd":8,"last update":"Fri May 13 23:13:37 2022","interfaces":[{"interface":"back","average":{"1min":1.069,"5min":1.069,"15min":1.069},"min":{"1min":0.59899999999999998,"5min":0.59899999999999998,"15min":0.59899999999999998},"max":{"1min":1.6140000000000001,"5min":1.6140000000000001,"15min":1.6140000000000001},"last":0.92000000000000004},{"interface":"front","average":{"1min":1.1419999999999999,"5min":1.1419999999999999,"15min":1.1419999999999999},"min":{"1min":0.71599999999999997,"5min":0.71599999999999997,"15min":0.71599999999999997},"max":{"1min":2.1789999999999998,"5min":2.1789999999999998,"15min":2.1789999999999998},"last":1.0369999999999999}]},{"osd":10,"last update":"Fri May 13 23:14:02 2022","interfaces":[{"interface":"back","average":{"1min":1.1040000000000001,"5min":1.1040000000000001,"15min":1.1040000000000001},"min":{"1min":0.66800000000000004,"5min":0.66800000000000004,"15min":0.66800000000000004},"max":{"1min":1.7829999999999999,"5min":1.7829999999999999,"15min":1.7829999999999999},"last":0.98299999999999998},{"interface":"front","average":{"1min":1.1319999999999999,"5min":1.1319999999999999,"15min":1.1319999999999999},"min":{"1min":0.73099999999999998,"5min":0.73099999999999998,"15min":0.73099999999999998},"max":{"1min":1.748,"5min":1.748,"15min":1.748},"last":1.0129999999999999}]}]},{"osd":8,"up_from":58,"seq":249108103199,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6284,"kb_used_data":900,"kb_used_omap":0,"kb_used_meta":5376,"kb_avail":93738868,"statfs":{"total":95995035648,"available":95988600832,"internally_reserved":0,"allocated":921600,"data_stored":426258,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5505024},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri May 13 23:14:08 2022","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.60399999999999998,"15min":0.60799999999999998},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.88900000000000001,"5min":0.88900000000000001,"15min":0.88900000000000001},"last":0.753},{"interface":"front","average":{"1min":0.61699999999999999,"5min":0.59199999999999997,"15min":0.58799999999999997},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":1.137,"5min":1.137,"15min":1.137},"last":0.72499999999999998}]},{"osd":1,"last update":"Fri May 13 23:14:08 2022","interfaces":[{"interface":"back","average":{"1min":0.53900000000000003,"5min":0.56699999999999995,"15min":0.57099999999999995},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.67200000000000004,"5min":0.78900000000000003,"15min":0.78900000000000003},"last":0.64000000000000001},{"interface":"front","average":{"1min":0.57299999999999995,"5min":0.60199999999999998,"15min":0.60699999999999998},"min":{"1min":0.38900000000000001,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.72599999999999998,"5min":0.80700000000000005,"15min":0.80700000000000005},"last":0.79900000000000004}]},{"osd":2,"last update":"Fri May 13 23:14:08 2022","interfaces":[{"interface":"back","average":{"1min":0.44,"5min":0.45600000000000002,"15min":0.45900000000000002},"min":{"1min":0.24199999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.77400000000000002,"5min":0.77400000000000002,"15min":0.77400000000000002},"last":0.56100000000000005},{"interface":"front","average":{"1min":0.443,"5min":0.45600000000000002,"15min":0.45800000000000002},"min":{"1min":0.27800000000000002,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.79100000000000004,"5min":0.79100000000000004,"15min":0.79100000000000004},"last":0.58199999999999996}]},{"osd":3,"last update":"Fri May 13 23:14:08 2022","interfaces":[{"interface":"back","average":{"1min":0.626,"5min":0.64800000000000002,"15min":0.65200000000000002},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.88400000000000001,"5min":0.88400000000000001,"15min":0.88400000000000001},"last":0.76500000000000001},{"interface":"front","average":{"1min":0.63300000000000001,"5min":0.64300000000000002,"15min":0.64500000000000002},"min":{"1min":0.439,"5min":0.439,"15min":0.439},"max":{"1min":0.82899999999999996,"5min":0.83299999999999996,"15min":0.83299999999999996},"last":0.67000000000000004}]},{"osd":4,"last update":"Fri May 13 23:14:08 2022","interfaces":[{"interface":"back","average":{"1min":0.61099999999999999,"5min":0.621,"15min":0.623},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":1.573,"5min":1.573,"15min":1.573},"last":0.82999999999999996},{"interface":"front","average":{"1min":0.57299999999999995,"5min":0.61899999999999999,"15min":0.627},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":0.746,"5min":0.90500000000000003,"15min":0.90500000000000003},"last":0.74099999999999999}]},{"osd":5,"last update":"Fri May 13 23:14:08 2022","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.57399999999999995,"15min":0.58299999999999996},"min":{"1min":0.22800000000000001,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":0.85199999999999998,"5min":0.875,"15min":0.875},"last":0.71599999999999997},{"interface":"front","average":{"1min":0.498,"5min":0.56299999999999994,"15min":0.57399999999999995},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.86199999999999999,"5min":0.88200000000000001,"15min":0.88200000000000001},"last":0.60899999999999999}]},{"osd":6,"last update":"Fri May 13 23:14:08 2022","interfaces":[{"interface":"back","average":{"1min":0.70299999999999996,"5min":0.74199999999999999,"15min":0.748},"min":{"1min":0.44400000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":0.873,"5min":0.94099999999999995,"15min":0.94099999999999995},"last":0.89600000000000002},{"interface":"front","average":{"1min":0.72099999999999997,"5min":0.754,"15min":0.76000000000000001},"min":{"1min":0.51700000000000002,"5min":0.504,"15min":0.504},"max":{"1min":0.91700000000000004,"5min":0.95699999999999996,"15min":0.95699999999999996},"last":0.95399999999999996}]},{"osd":7,"last update":"Fri May 13 23:14:08 2022","interfaces":[{"interface":"back","average":{"1min":0.67100000000000004,"5min":0.69999999999999996,"15min":0.70499999999999996},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.94799999999999995,"5min":0.95799999999999996,"15min":0.95799999999999996},"last":0.90600000000000003},{"interface":"front","average":{"1min":0.70399999999999996,"5min":0.69999999999999996,"15min":0.69999999999999996},"min":{"1min":0.49099999999999999,"5min":0.437,"15min":0.437},"max":{"1min":1.609,"5min":1.609,"15min":1.609},"last":0.88}]},{"osd":9,"last update":"Fri May 13 23:13:38 2022","interfaces":[{"interface":"back","average":{"1min":0.76500000000000001,"5min":0.76500000000000001,"15min":0.76500000000000001},"min":{"1min":0.51500000000000001,"5min":0.51500000000000001,"15min":0.51500000000000001},"max":{"1min":0.999,"5min":0.999,"15min":0.999},"last":1},{"interface":"front","average":{"1min":0.80100000000000005,"5min":0.80100000000000005,"15min":0.80100000000000005},"min":{"1min":0.59899999999999998,"5min":0.59899999999999998,"15min":0.59899999999999998},"max":{"1min":0.996,"5min":0.996,"15min":0.996},"last":0.85199999999999998}]},{"osd":10,"last update":"Fri May 13 23:14:02 2022","interfaces":[{"interface":"back","average":{"1min":0.72699999999999998,"5min":0.72699999999999998,"15min":0.72699999999999998},"min":{"1min":0.45800000000000002,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":0.97799999999999998,"5min":0.97799999999999998,"15min":0.97799999999999998},"last":0.92900000000000005},{"interface":"front","average":{"1min":0.69699999999999995,"5min":0.69699999999999995,"15min":0.69699999999999995},"min":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":0.998,"5min":0.998,"15min":0.998},"last":0.96399999999999997}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.86699999999999999}]}]},{"osd":7,"up_from":51,"seq":219043332129,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6868,"kb_used_data":1420,"kb_used_omap":0,"kb_used_meta":5440,"kb_avail":93738284,"statfs":{"total":95995035648,"available":95988002816,"internally_reserved":0,"allocated":1454080,"data_stored":956684,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5570560},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri May 13 23:13:48 2022","interfaces":[{"interface":"back","average":{"1min":0.64400000000000002,"5min":0.56000000000000005,"15min":0.54600000000000004},"min":{"1min":0.48399999999999999,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":0.82599999999999996,"5min":0.82599999999999996,"15min":0.82599999999999996},"last":0.63600000000000001},{"interface":"front","average":{"1min":0.60899999999999999,"5min":0.58799999999999997,"15min":0.58399999999999996},"min":{"1min":0.42699999999999999,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":0.78800000000000003,"5min":0.86599999999999999,"15min":0.86599999999999999},"last":0.51900000000000002}]},{"osd":1,"last update":"Fri May 13 23:13:48 2022","interfaces":[{"interface":"back","average":{"1min":0.47499999999999998,"5min":0.44700000000000001,"15min":0.442},"min":{"1min":0.29999999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.79800000000000004,"5min":0.79800000000000004,"15min":0.79800000000000004},"last":0.52100000000000002},{"interface":"front","average":{"1min":0.49299999999999999,"5min":0.47299999999999998,"15min":0.46999999999999997},"min":{"1min":0.317,"5min":0.317,"15min":0.317},"max":{"1min":0.88800000000000001,"5min":0.88800000000000001,"15min":0.88800000000000001},"last":0.63900000000000001}]},{"osd":2,"last update":"Fri May 13 23:13:48 2022","interfaces":[{"interface":"back","average":{"1min":0.63100000000000001,"5min":0.63500000000000001,"15min":0.63600000000000001},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.83799999999999997,"5min":0.88800000000000001,"15min":0.88800000000000001},"last":0.751},{"interface":"front","average":{"1min":0.60699999999999998,"5min":0.60299999999999998,"15min":0.60299999999999998},"min":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":0.78600000000000003,"5min":0.81599999999999995,"15min":0.81599999999999995},"last":0.65500000000000003}]},{"osd":3,"last update":"Fri May 13 23:13:48 2022","interfaces":[{"interface":"back","average":{"1min":0.66300000000000003,"5min":0.63500000000000001,"15min":0.63},"min":{"1min":0.50700000000000001,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":0.92200000000000004,"5min":0.92200000000000004,"15min":0.92200000000000004},"last":0.75900000000000001},{"interface":"front","average":{"1min":0.69399999999999995,"5min":0.69699999999999995,"15min":0.69699999999999995},"min":{"1min":0.49199999999999999,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":0.872,"5min":0.872,"15min":0.872},"last":0.67500000000000004}]},{"osd":4,"last update":"Fri May 13 23:13:48 2022","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.58399999999999996,"15min":0.59199999999999997},"min":{"1min":0.315,"5min":0.315,"15min":0.315},"max":{"1min":0.77800000000000002,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.55200000000000005},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.58199999999999996,"15min":0.58799999999999997},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.875,"5min":0.875,"15min":0.875},"last":0.61799999999999999}]},{"osd":5,"last update":"Fri May 13 23:13:48 2022","interfaces":[{"interface":"back","average":{"1min":0.68100000000000005,"5min":0.72099999999999997,"15min":0.72699999999999998},"min":{"1min":0.47199999999999998,"5min":0.47199999999999998,"15min":0.47199999999999998},"max":{"1min":0.93300000000000005,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.69399999999999995},{"interface":"front","average":{"1min":0.68100000000000005,"5min":0.70099999999999996,"15min":0.70399999999999996},"min":{"1min":0.47999999999999998,"5min":0.47999999999999998,"15min":0.47999999999999998},"max":{"1min":0.93999999999999995,"5min":0.93999999999999995,"15min":0.93999999999999995},"last":0.70199999999999996}]},{"osd":6,"last update":"Fri May 13 23:13:48 2022","interfaces":[{"interface":"back","average":{"1min":0.76300000000000001,"5min":0.76200000000000001,"15min":0.76200000000000001},"min":{"1min":0.52700000000000002,"5min":0.52700000000000002,"15min":0.52700000000000002},"max":{"1min":1.0700000000000001,"5min":1.0700000000000001,"15min":1.0700000000000001},"last":0.73599999999999999},{"interface":"front","average":{"1min":0.81999999999999995,"5min":0.79700000000000004,"15min":0.79300000000000004},"min":{"1min":0.67600000000000005,"5min":0.63200000000000001,"15min":0.63200000000000001},"max":{"1min":1.04,"5min":1.1319999999999999,"15min":1.1319999999999999},"last":0.57599999999999996}]},{"osd":8,"last update":"Fri May 13 23:14:07 2022","interfaces":[{"interface":"back","average":{"1min":0.90600000000000003,"5min":0.77200000000000002,"15min":0.75},"min":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"max":{"1min":4.0739999999999998,"5min":4.0739999999999998,"15min":4.0739999999999998},"last":0.66700000000000004},{"interface":"front","average":{"1min":0.78100000000000003,"5min":0.76500000000000001,"15min":0.76200000000000001},"min":{"1min":0.497,"5min":0.497,"15min":0.497},"max":{"1min":1.046,"5min":1.046,"15min":1.046},"last":0.72299999999999998}]},{"osd":9,"last update":"Fri May 13 23:13:37 2022","interfaces":[{"interface":"back","average":{"1min":0.82699999999999996,"5min":0.82699999999999996,"15min":0.82699999999999996},"min":{"1min":0.57099999999999995,"5min":0.57099999999999995,"15min":0.57099999999999995},"max":{"1min":1.1339999999999999,"5min":1.1339999999999999,"15min":1.1339999999999999},"last":0.86799999999999999},{"interface":"front","average":{"1min":0.82899999999999996,"5min":0.82899999999999996,"15min":0.82899999999999996},"min":{"1min":0.64900000000000002,"5min":0.64900000000000002,"15min":0.64900000000000002},"max":{"1min":1.0409999999999999,"5min":1.0409999999999999,"15min":1.0409999999999999},"last":0.77600000000000002}]},{"osd":10,"last update":"Fri May 13 23:14:00 2022","interfaces":[{"interface":"back","average":{"1min":0.77400000000000002,"5min":0.77400000000000002,"15min":0.77400000000000002},"min":{"1min":0.47599999999999998,"5min":0.47599999999999998,"15min":0.47599999999999998},"max":{"1min":1.093,"5min":1.093,"15min":1.093},"last":0.79100000000000004},{"interface":"front","average":{"1min":0.72899999999999998,"5min":0.72899999999999998,"15min":0.72899999999999998},"min":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"max":{"1min":1.0529999999999999,"5min":1.0529999999999999,"15min":1.0529999999999999},"last":0.82199999999999995}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.90500000000000003}]}]},{"osd":6,"up_from":46,"seq":197568495655,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6204,"kb_used_data":884,"kb_used_omap":0,"kb_used_meta":5312,"kb_avail":93738948,"statfs":{"total":95995035648,"available":95988682752,"internally_reserved":0,"allocated":905216,"data_stored":425488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5439488},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri May 13 23:13:27 2022","interfaces":[{"interface":"back","average":{"1min":0.58699999999999997,"5min":0.55100000000000005,"15min":0.54500000000000004},"min":{"1min":0.378,"5min":0.378,"15min":0.378},"max":{"1min":1.004,"5min":1.004,"15min":1.004},"last":0.61499999999999999},{"interface":"front","average":{"1min":0.62,"5min":0.60099999999999998,"15min":0.59799999999999998},"min":{"1min":0.38800000000000001,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":1.042,"5min":1.042,"15min":1.042},"last":0.66900000000000004}]},{"osd":1,"last update":"Fri May 13 23:13:27 2022","interfaces":[{"interface":"back","average":{"1min":0.80500000000000005,"5min":0.75700000000000001,"15min":0.749},"min":{"1min":0.45700000000000002,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":1.2210000000000001,"5min":1.2210000000000001,"15min":1.2210000000000001},"last":0.77100000000000002},{"interface":"front","average":{"1min":0.72699999999999998,"5min":0.67700000000000005,"15min":0.66900000000000004},"min":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":1.28,"5min":1.28,"15min":1.28},"last":0.89700000000000002}]},{"osd":2,"last update":"Fri May 13 23:13:27 2022","interfaces":[{"interface":"back","average":{"1min":0.80400000000000005,"5min":0.78800000000000003,"15min":0.78500000000000003},"min":{"1min":0.54700000000000004,"5min":0.54700000000000004,"15min":0.54700000000000004},"max":{"1min":1.1200000000000001,"5min":1.228,"15min":1.228},"last":0.94599999999999995},{"interface":"front","average":{"1min":0.83299999999999996,"5min":0.81599999999999995,"15min":0.81299999999999994},"min":{"1min":0.56599999999999995,"5min":0.53300000000000003,"15min":0.53300000000000003},"max":{"1min":1.298,"5min":1.298,"15min":1.298},"last":0.81299999999999994}]},{"osd":3,"last update":"Fri May 13 23:13:27 2022","interfaces":[{"interface":"back","average":{"1min":0.85699999999999998,"5min":0.81399999999999995,"15min":0.80700000000000005},"min":{"1min":0.60199999999999998,"5min":0.47599999999999998,"15min":0.47599999999999998},"max":{"1min":1.3839999999999999,"5min":1.3839999999999999,"15min":1.3839999999999999},"last":0.85599999999999998},{"interface":"front","average":{"1min":0.76000000000000001,"5min":0.77800000000000002,"15min":0.78100000000000003},"min":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":1.0580000000000001,"5min":1.4119999999999999,"15min":1.4119999999999999},"last":0.70999999999999996}]},{"osd":4,"last update":"Fri May 13 23:13:27 2022","interfaces":[{"interface":"back","average":{"1min":0.82299999999999995,"5min":0.83899999999999997,"15min":0.84199999999999997},"min":{"1min":0.504,"5min":0.504,"15min":0.504},"max":{"1min":1.5640000000000001,"5min":1.5640000000000001,"15min":1.5640000000000001},"last":0.88500000000000001},{"interface":"front","average":{"1min":0.93200000000000005,"5min":0.90000000000000002,"15min":0.89400000000000002},"min":{"1min":0.60799999999999998,"5min":0.60799999999999998,"15min":0.60799999999999998},"max":{"1min":1.625,"5min":1.625,"15min":1.625},"last":0.91900000000000004}]},{"osd":5,"last update":"Fri May 13 23:13:27 2022","interfaces":[{"interface":"back","average":{"1min":0.84599999999999997,"5min":0.85699999999999998,"15min":0.85899999999999999},"min":{"1min":0.56799999999999995,"5min":0.50600000000000001,"15min":0.50600000000000001},"max":{"1min":1.3640000000000001,"5min":1.5029999999999999,"15min":1.5029999999999999},"last":1.123},{"interface":"front","average":{"1min":0.93000000000000005,"5min":0.93700000000000006,"15min":0.93799999999999994},"min":{"1min":0.65900000000000003,"5min":0.65400000000000003,"15min":0.65400000000000003},"max":{"1min":1.3460000000000001,"5min":1.3460000000000001,"15min":1.3460000000000001},"last":0.98999999999999999}]},{"osd":7,"last update":"Fri May 13 23:13:47 2022","interfaces":[{"interface":"back","average":{"1min":1.0669999999999999,"5min":0.89700000000000002,"15min":0.86899999999999999},"min":{"1min":0.78500000000000003,"5min":0.66000000000000003,"15min":0.66000000000000003},"max":{"1min":1.534,"5min":1.534,"15min":1.534},"last":0.999},{"interface":"front","average":{"1min":0.97599999999999998,"5min":0.84699999999999998,"15min":0.82499999999999996},"min":{"1min":0.66300000000000003,"5min":0.61299999999999999,"15min":0.61299999999999999},"max":{"1min":1.6180000000000001,"5min":1.6180000000000001,"15min":1.6180000000000001},"last":0.96099999999999997}]},{"osd":8,"last update":"Fri May 13 23:14:12 2022","interfaces":[{"interface":"back","average":{"1min":1.0049999999999999,"5min":0.98499999999999999,"15min":0.98099999999999998},"min":{"1min":0.69999999999999996,"5min":0.69999999999999996,"15min":0.69999999999999996},"max":{"1min":1.484,"5min":1.484,"15min":1.484},"last":0.97099999999999997},{"interface":"front","average":{"1min":0.89600000000000002,"5min":0.94499999999999995,"15min":0.95299999999999996},"min":{"1min":0.68500000000000005,"5min":0.67500000000000004,"15min":0.67500000000000004},"max":{"1min":1.548,"5min":1.548,"15min":1.548},"last":1.0449999999999999}]},{"osd":9,"last update":"Fri May 13 23:13:37 2022","interfaces":[{"interface":"back","average":{"1min":1.071,"5min":1.071,"15min":1.071},"min":{"1min":0.86299999999999999,"5min":0.86299999999999999,"15min":0.86299999999999999},"max":{"1min":1.458,"5min":1.458,"15min":1.458},"last":1.165},{"interface":"front","average":{"1min":1.0580000000000001,"5min":1.0580000000000001,"15min":1.0580000000000001},"min":{"1min":0.78900000000000003,"5min":0.78900000000000003,"15min":0.78900000000000003},"max":{"1min":1.401,"5min":1.401,"15min":1.401},"last":1.0229999999999999}]},{"osd":10,"last update":"Fri May 13 23:14:01 2022","interfaces":[{"interface":"back","average":{"1min":1.0529999999999999,"5min":1.0529999999999999,"15min":1.0529999999999999},"min":{"1min":0.77800000000000002,"5min":0.77800000000000002,"15min":0.77800000000000002},"max":{"1min":1.645,"5min":1.645,"15min":1.645},"last":1.1990000000000001},{"interface":"front","average":{"1min":1.006,"5min":1.006,"15min":1.006},"min":{"1min":0.73999999999999999,"5min":0.73999999999999999,"15min":0.73999999999999999},"max":{"1min":1.476,"5min":1.476,"15min":1.476},"last":1.097}]}]},{"osd":1,"up_from":13,"seq":55834574914,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":7108,"kb_used_data":892,"kb_used_omap":0,"kb_used_meta":6208,"kb_avail":93738044,"statfs":{"total":95995035648,"available":95987757056,"internally_reserved":0,"allocated":913408,"data_stored":427998,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":6356992},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri May 13 23:14:14 2022","interfaces":[{"interface":"back","average":{"1min":0.60399999999999998,"5min":0.55900000000000005,"15min":0.51300000000000001},"min":{"1min":0.46700000000000003,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.755,"5min":0.91700000000000004,"15min":0.91700000000000004},"last":0.64200000000000002},{"interface":"front","average":{"1min":0.623,"5min":0.57899999999999996,"15min":0.54000000000000004},"min":{"1min":0.48399999999999999,"5min":0.309,"15min":0.309},"max":{"1min":0.82299999999999995,"5min":0.86799999999999999,"15min":0.86799999999999999},"last":0.66100000000000003}]},{"osd":2,"last update":"Fri May 13 23:13:40 2022","interfaces":[{"interface":"back","average":{"1min":0.56000000000000005,"5min":0.52300000000000002,"15min":0.51300000000000001},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.76900000000000002,"5min":0.76900000000000002,"15min":0.76900000000000002},"last":0.69899999999999995},{"interface":"front","average":{"1min":0.59899999999999998,"5min":0.54300000000000004,"15min":0.52500000000000002},"min":{"1min":0.42299999999999999,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.84599999999999997,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.90400000000000003}]},{"osd":3,"last update":"Fri May 13 23:14:12 2022","interfaces":[{"interface":"back","average":{"1min":0.64100000000000001,"5min":0.59699999999999998,"15min":0.57099999999999995},"min":{"1min":0.44600000000000001,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.84199999999999997,"5min":0.871,"15min":0.871},"last":0.72899999999999998},{"interface":"front","average":{"1min":0.60899999999999999,"5min":0.57199999999999995,"15min":0.56200000000000006},"min":{"1min":0.45400000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.77900000000000003,"5min":0.95399999999999996,"15min":0.95399999999999996},"last":0.629}]},{"osd":4,"last update":"Fri May 13 23:13:35 2022","interfaces":[{"interface":"back","average":{"1min":0.51000000000000001,"5min":0.435,"15min":0.41699999999999998},"min":{"1min":0.32100000000000001,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.77100000000000002,"5min":0.77100000000000002,"15min":0.77100000000000002},"last":0.55000000000000004},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.46899999999999997,"15min":0.44500000000000001},"min":{"1min":0.27400000000000002,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.876,"5min":0.876,"15min":0.876},"last":0.52500000000000002}]},{"osd":5,"last update":"Fri May 13 23:13:52 2022","interfaces":[{"interface":"back","average":{"1min":0.72499999999999998,"5min":0.60099999999999998,"15min":0.58499999999999996},"min":{"1min":0.497,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.88200000000000001,"5min":0.88200000000000001,"15min":0.88200000000000001},"last":0.93300000000000005},{"interface":"front","average":{"1min":0.748,"5min":0.628,"15min":0.60499999999999998},"min":{"1min":0.45500000000000002,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":0.98799999999999999,"5min":0.98799999999999999,"15min":0.98799999999999999},"last":0.95599999999999996}]},{"osd":6,"last update":"Fri May 13 23:13:23 2022","interfaces":[{"interface":"back","average":{"1min":0.76800000000000002,"5min":0.68799999999999994,"15min":0.67400000000000004},"min":{"1min":0.45000000000000001,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":1.5529999999999999,"5min":1.5529999999999999,"15min":1.5529999999999999},"last":0.748},{"interface":"front","average":{"1min":0.75,"5min":0.71599999999999997,"15min":0.70999999999999996},"min":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":1.1419999999999999,"5min":1.1419999999999999,"15min":1.1419999999999999},"last":0.85499999999999998}]},{"osd":7,"last update":"Fri May 13 23:13:46 2022","interfaces":[{"interface":"back","average":{"1min":0.70799999999999996,"5min":0.58399999999999996,"15min":0.56299999999999994},"min":{"1min":0.38600000000000001,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.95999999999999996,"5min":0.95999999999999996,"15min":0.95999999999999996},"last":0.57699999999999996},{"interface":"front","average":{"1min":0.79500000000000004,"5min":0.65300000000000002,"15min":0.629},"min":{"1min":0.41399999999999998,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":1.042,"5min":1.042,"15min":1.042},"last":0.83399999999999996}]},{"osd":8,"last update":"Fri May 13 23:14:12 2022","interfaces":[{"interface":"back","average":{"1min":0.74099999999999999,"5min":0.66900000000000004,"15min":0.65700000000000003},"min":{"1min":0.5,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":1.02,"5min":1.02,"15min":1.02},"last":0.82099999999999995},{"interface":"front","average":{"1min":0.84899999999999998,"5min":0.73299999999999998,"15min":0.71399999999999997},"min":{"1min":0.58999999999999997,"5min":0.501,"15min":0.501},"max":{"1min":1.111,"5min":1.111,"15min":1.111},"last":0.96799999999999997}]},{"osd":9,"last update":"Fri May 13 23:13:39 2022","interfaces":[{"interface":"back","average":{"1min":0.88500000000000001,"5min":0.88500000000000001,"15min":0.88500000000000001},"min":{"1min":0.60999999999999999,"5min":0.60999999999999999,"15min":0.60999999999999999},"max":{"1min":1.091,"5min":1.091,"15min":1.091},"last":0.878},{"interface":"front","average":{"1min":0.84599999999999997,"5min":0.84599999999999997,"15min":0.84599999999999997},"min":{"1min":0.63600000000000001,"5min":0.63600000000000001,"15min":0.63600000000000001},"max":{"1min":1.0780000000000001,"5min":1.0780000000000001,"15min":1.0780000000000001},"last":1.01}]},{"osd":10,"last update":"Fri May 13 23:14:01 2022","interfaces":[{"interface":"back","average":{"1min":0.78800000000000003,"5min":0.78800000000000003,"15min":0.78800000000000003},"min":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"max":{"1min":0.95399999999999996,"5min":0.95399999999999996,"15min":0.95399999999999996},"last":0.77400000000000002},{"interface":"front","average":{"1min":0.90800000000000003,"5min":0.90800000000000003,"15min":0.90800000000000003},"min":{"1min":0.623,"5min":0.623,"15min":0.623},"max":{"1min":1.105,"5min":1.105,"15min":1.105},"last":0.997}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.98799999999999999}]}]},{"osd":0,"up_from":8,"seq":34359738438,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":7124,"kb_used_data":908,"kb_used_omap":0,"kb_used_meta":6208,"kb_avail":93738028,"statfs":{"total":95995035648,"available":95987740672,"internally_reserved":0,"allocated":929792,"data_stored":426888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":6356992},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri May 13 23:14:11 2022","interfaces":[{"interface":"back","average":{"1min":0.68300000000000005,"5min":0.627,"15min":0.60999999999999999},"min":{"1min":0.40400000000000003,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.98899999999999999,"5min":1.2729999999999999,"15min":1.2729999999999999},"last":0.67700000000000005},{"interface":"front","average":{"1min":0.752,"5min":0.66200000000000003,"15min":0.627},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":0.98999999999999999,"5min":1.167,"15min":1.167},"last":0.90300000000000002}]},{"osd":2,"last update":"Fri May 13 23:13:35 2022","interfaces":[{"interface":"back","average":{"1min":0.72499999999999998,"5min":0.65400000000000003,"15min":0.64600000000000002},"min":{"1min":0.54300000000000004,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":0.96499999999999997,"5min":0.98799999999999999,"15min":0.98799999999999999},"last":0.94999999999999996},{"interface":"front","average":{"1min":0.70099999999999996,"5min":0.63600000000000001,"15min":0.628},"min":{"1min":0.47799999999999998,"5min":0.45900000000000002,"15min":0.45900000000000002},"max":{"1min":1.024,"5min":1.123,"15min":1.123},"last":0.73899999999999999}]},{"osd":3,"last update":"Fri May 13 23:13:08 2022","interfaces":[{"interface":"back","average":{"1min":0.82399999999999995,"5min":0.63300000000000001,"15min":0.59999999999999998},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":1.163,"5min":1.163,"15min":1.163},"last":0.59199999999999997},{"interface":"front","average":{"1min":0.79800000000000004,"5min":0.67000000000000004,"15min":0.64600000000000002},"min":{"1min":0.46200000000000002,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":1.157,"5min":1.157,"15min":1.157},"last":0.63200000000000001}]},{"osd":4,"last update":"Fri May 13 23:13:33 2022","interfaces":[{"interface":"back","average":{"1min":0.82999999999999996,"5min":0.68200000000000005,"15min":0.65500000000000003},"min":{"1min":0.495,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":1.1990000000000001,"5min":1.1990000000000001,"15min":1.1990000000000001},"last":0.99299999999999999},{"interface":"front","average":{"1min":0.77900000000000003,"5min":0.66300000000000003,"15min":0.63300000000000001},"min":{"1min":0.50600000000000001,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":0.98999999999999999,"5min":0.98999999999999999,"15min":0.98999999999999999},"last":0.88400000000000001}]},{"osd":5,"last update":"Fri May 13 23:13:57 2022","interfaces":[{"interface":"back","average":{"1min":0.78900000000000003,"5min":0.72499999999999998,"15min":0.71099999999999997},"min":{"1min":0.46999999999999997,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":1.1220000000000001,"5min":1.1220000000000001,"15min":1.1220000000000001},"last":0.84399999999999997},{"interface":"front","average":{"1min":0.71199999999999997,"5min":0.69699999999999995,"15min":0.69599999999999995},"min":{"1min":0.49399999999999999,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":1.0409999999999999,"5min":1.056,"15min":1.056},"last":0.752}]},{"osd":6,"last update":"Fri May 13 23:13:25 2022","interfaces":[{"interface":"back","average":{"1min":0.93600000000000005,"5min":0.77600000000000002,"15min":0.75},"min":{"1min":0.57799999999999996,"5min":0.44800000000000001,"15min":0.44800000000000001},"max":{"1min":1.288,"5min":1.288,"15min":1.288},"last":0.85699999999999998},{"interface":"front","average":{"1min":0.874,"5min":0.78400000000000003,"15min":0.76900000000000002},"min":{"1min":0.51000000000000001,"5min":0.5,"15min":0.5},"max":{"1min":1.202,"5min":1.202,"15min":1.202},"last":0.80100000000000005}]},{"osd":7,"last update":"Fri May 13 23:13:48 2022","interfaces":[{"interface":"back","average":{"1min":0.77300000000000002,"5min":0.75,"15min":0.746},"min":{"1min":0.497,"5min":0.48699999999999999,"15min":0.48699999999999999},"max":{"1min":1.2050000000000001,"5min":1.2050000000000001,"15min":1.2050000000000001},"last":0.94199999999999995},{"interface":"front","average":{"1min":0.94899999999999995,"5min":0.86099999999999999,"15min":0.84699999999999998},"min":{"1min":0.69599999999999995,"5min":0.60399999999999998,"15min":0.60399999999999998},"max":{"1min":1.2350000000000001,"5min":1.2350000000000001,"15min":1.2350000000000001},"last":0.877}]},{"osd":8,"last update":"Fri May 13 23:14:09 2022","interfaces":[{"interface":"back","average":{"1min":0.82699999999999996,"5min":0.80500000000000005,"15min":0.80100000000000005},"min":{"1min":0.54700000000000004,"5min":0.52100000000000002,"15min":0.52100000000000002},"max":{"1min":1.1479999999999999,"5min":1.7070000000000001,"15min":1.7070000000000001},"last":0.83299999999999996},{"interface":"front","average":{"1min":0.91200000000000003,"5min":0.91600000000000004,"15min":0.91600000000000004},"min":{"1min":0.60599999999999998,"5min":0.60599999999999998,"15min":0.60599999999999998},"max":{"1min":1.1160000000000001,"5min":1.266,"15min":1.266},"last":1.0629999999999999}]},{"osd":9,"last update":"Fri May 13 23:13:39 2022","interfaces":[{"interface":"back","average":{"1min":1.0589999999999999,"5min":1.0589999999999999,"15min":1.0589999999999999},"min":{"1min":0.85199999999999998,"5min":0.85199999999999998,"15min":0.85199999999999998},"max":{"1min":1.397,"5min":1.397,"15min":1.397},"last":1.093},{"interface":"front","average":{"1min":0.94099999999999995,"5min":0.94099999999999995,"15min":0.94099999999999995},"min":{"1min":0.70999999999999996,"5min":0.70999999999999996,"15min":0.70999999999999996},"max":{"1min":1.232,"5min":1.232,"15min":1.232},"last":0.92400000000000004}]},{"osd":10,"last update":"Fri May 13 23:14:02 2022","interfaces":[{"interface":"back","average":{"1min":0.97199999999999998,"5min":0.97199999999999998,"15min":0.97199999999999998},"min":{"1min":0.67700000000000005,"5min":0.67700000000000005,"15min":0.67700000000000005},"max":{"1min":1.7989999999999999,"5min":1.7989999999999999,"15min":1.7989999999999999},"last":1.044},{"interface":"front","average":{"1min":1.0629999999999999,"5min":1.0629999999999999,"15min":1.0629999999999999},"min":{"1min":0.745,"5min":0.745,"15min":0.745},"max":{"1min":1.3360000000000001,"5min":1.3360000000000001,"15min":1.3360000000000001},"last":1.2869999999999999}]},{"osd":11,"last update":"Fri May 13 23:14:22 2022","interfaces":[{"interface":"back","average":{"1min":0.97899999999999998,"5min":0.97899999999999998,"15min":0.97899999999999998},"min":{"1min":0.63400000000000001,"5min":0.63400000000000001,"15min":0.63400000000000001},"max":{"1min":1.617,"5min":1.617,"15min":1.617},"last":0.95799999999999996},{"interface":"front","average":{"1min":1.0229999999999999,"5min":1.0229999999999999,"15min":1.0229999999999999},"min":{"1min":0.68400000000000005,"5min":0.68400000000000005,"15min":0.68400000000000005},"max":{"1min":1.266,"5min":1.266,"15min":1.266},"last":1.234}]}]},{"osd":2,"up_from":18,"seq":77309411388,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":7192,"kb_used_data":912,"kb_used_omap":0,"kb_used_meta":6272,"kb_avail":93737960,"statfs":{"total":95995035648,"available":95987671040,"internally_reserved":0,"allocated":933888,"data_stored":428198,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":6422528},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri May 13 23:13:37 2022","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.51700000000000002,"15min":0.52300000000000002},"min":{"1min":0.41699999999999998,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.80700000000000005,"5min":0.80700000000000005,"15min":0.80700000000000005},"last":0.55300000000000005},{"interface":"front","average":{"1min":0.59499999999999997,"5min":0.52100000000000002,"15min":0.51600000000000001},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.78800000000000003,"5min":0.80700000000000005,"15min":0.80700000000000005},"last":0.57299999999999995}]},{"osd":1,"last update":"Fri May 13 23:13:37 2022","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.52500000000000002,"15min":0.52000000000000002},"min":{"1min":0.35699999999999998,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.83699999999999997,"5min":0.83699999999999997,"15min":0.83699999999999997},"last":0.71399999999999997},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.53200000000000003,"15min":0.55000000000000004},"min":{"1min":0.40100000000000002,"5min":0.313,"15min":0.313},"max":{"1min":0.77900000000000003,"5min":0.77900000000000003,"15min":0.77900000000000003},"last":0.60199999999999998}]},{"osd":3,"last update":"Fri May 13 23:14:12 2022","interfaces":[{"interface":"back","average":{"1min":0.64500000000000002,"5min":0.61599999999999999,"15min":0.61199999999999999},"min":{"1min":0.39900000000000002,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.90500000000000003,"5min":0.90500000000000003,"15min":0.90500000000000003},"last":0.70299999999999996},{"interface":"front","average":{"1min":0.66800000000000004,"5min":0.60499999999999998,"15min":0.59699999999999998},"min":{"1min":0.45000000000000001,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.89500000000000002,"5min":0.89500000000000002,"15min":0.89500000000000002},"last":0.73199999999999998}]},{"osd":4,"last update":"Fri May 13 23:13:31 2022","interfaces":[{"interface":"back","average":{"1min":0.55100000000000005,"5min":0.53400000000000003,"15min":0.52600000000000002},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":0.81999999999999995,"5min":0.81999999999999995,"15min":0.81999999999999995},"last":0.76200000000000001},{"interface":"front","average":{"1min":0.53200000000000003,"5min":0.51500000000000001,"15min":0.505},"min":{"1min":0.36299999999999999,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.70099999999999996,"5min":0.752,"15min":0.752},"last":0.77200000000000002}]},{"osd":5,"last update":"Fri May 13 23:14:00 2022","interfaces":[{"interface":"back","average":{"1min":0.435,"5min":0.40300000000000002,"15min":0.38300000000000001},"min":{"1min":0.32300000000000001,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.71099999999999997,"5min":0.71999999999999997,"15min":0.71999999999999997},"last":0.67800000000000005},{"interface":"front","average":{"1min":0.47099999999999997,"5min":0.41699999999999998,"15min":0.39700000000000002},"min":{"1min":0.35499999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.748,"5min":0.748,"15min":0.748},"last":0.749}]},{"osd":6,"last update":"Fri May 13 23:13:28 2022","interfaces":[{"interface":"back","average":{"1min":0.60599999999999998,"5min":0.60599999999999998,"15min":0.60599999999999998},"min":{"1min":0.504,"5min":0.44800000000000001,"15min":0.44800000000000001},"max":{"1min":0.78500000000000003,"5min":0.78900000000000003,"15min":0.78900000000000003},"last":0.90000000000000002},{"interface":"front","average":{"1min":0.63100000000000001,"5min":0.58999999999999997,"15min":0.58299999999999996},"min":{"1min":0.46500000000000002,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":0.78200000000000003,"5min":0.78200000000000003,"15min":0.78200000000000003},"last":0.93100000000000005}]},{"osd":7,"last update":"Fri May 13 23:13:45 2022","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":0.61599999999999999,"15min":0.61599999999999999},"min":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":0.80400000000000005,"5min":0.80400000000000005,"15min":0.80400000000000005},"last":0.78400000000000003},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.63800000000000001,"15min":0.64200000000000002},"min":{"1min":0.45900000000000002,"5min":0.45900000000000002,"15min":0.45900000000000002},"max":{"1min":0.83199999999999996,"5min":0.83199999999999996,"15min":0.83199999999999996},"last":0.78800000000000003}]},{"osd":8,"last update":"Fri May 13 23:14:07 2022","interfaces":[{"interface":"back","average":{"1min":0.55400000000000005,"5min":0.53600000000000003,"15min":0.53300000000000003},"min":{"1min":0.36699999999999999,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.876,"5min":0.876,"15min":0.876},"last":0.82999999999999996},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.82799999999999996,"5min":0.82799999999999996,"15min":0.82799999999999996},"last":0.97199999999999998}]},{"osd":9,"last update":"Fri May 13 23:13:37 2022","interfaces":[{"interface":"back","average":{"1min":0.70999999999999996,"5min":0.70999999999999996,"15min":0.70999999999999996},"min":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"max":{"1min":0.82799999999999996,"5min":0.82799999999999996,"15min":0.82799999999999996},"last":0.94099999999999995},{"interface":"front","average":{"1min":0.68999999999999995,"5min":0.68999999999999995,"15min":0.68999999999999995},"min":{"1min":0.55800000000000005,"5min":0.55800000000000005,"15min":0.55800000000000005},"max":{"1min":0.86799999999999999,"5min":0.86799999999999999,"15min":0.86799999999999999},"last":0.89000000000000001}]},{"osd":10,"last update":"Fri May 13 23:14:00 2022","interfaces":[{"interface":"back","average":{"1min":0.68999999999999995,"5min":0.68999999999999995,"15min":0.68999999999999995},"min":{"1min":0.47599999999999998,"5min":0.47599999999999998,"15min":0.47599999999999998},"max":{"1min":0.97099999999999997,"5min":0.97099999999999997,"15min":0.97099999999999997},"last":0.86299999999999999},{"interface":"front","average":{"1min":0.71099999999999997,"5min":0.71099999999999997,"15min":0.71099999999999997},"min":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.52100000000000002},"max":{"1min":0.89200000000000002,"5min":0.89200000000000002,"15min":0.89200000000000002},"last":1.014}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.81599999999999995}]}]},{"osd":3,"up_from":25,"seq":107374182454,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6536,"kb_used_data":896,"kb_used_omap":0,"kb_used_meta":5632,"kb_avail":93738616,"statfs":{"total":95995035648,"available":95988342784,"internally_reserved":0,"allocated":917504,"data_stored":425244,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5767168},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri May 13 23:14:15 2022","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.49399999999999999,"15min":0.46400000000000002},"min":{"1min":0.40200000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.81999999999999995,"5min":0.81999999999999995,"15min":0.81999999999999995},"last":0.47299999999999998},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.52400000000000002,"15min":0.502},"min":{"1min":0.373,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.70799999999999996,"5min":0.96299999999999997,"15min":0.96299999999999997},"last":0.495}]},{"osd":1,"last update":"Fri May 13 23:14:15 2022","interfaces":[{"interface":"back","average":{"1min":0.749,"5min":0.67700000000000005,"15min":0.63500000000000001},"min":{"1min":0.495,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":1.014,"5min":1.014,"15min":1.014},"last":0.64400000000000002},{"interface":"front","average":{"1min":0.64100000000000001,"5min":0.64300000000000002,"15min":0.63100000000000001},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.93200000000000005,"5min":1.0229999999999999,"15min":1.0229999999999999},"last":0.52500000000000002}]},{"osd":2,"last update":"Fri May 13 23:14:15 2022","interfaces":[{"interface":"back","average":{"1min":0.79300000000000004,"5min":0.72399999999999998,"15min":0.68799999999999994},"min":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":1.0489999999999999,"5min":1.0489999999999999,"15min":1.0489999999999999},"last":0.81999999999999995},{"interface":"front","average":{"1min":0.73899999999999999,"5min":0.69699999999999995,"15min":0.67500000000000004},"min":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":1.1299999999999999,"5min":1.1299999999999999,"15min":1.1299999999999999},"last":0.61899999999999999}]},{"osd":4,"last update":"Fri May 13 23:13:33 2022","interfaces":[{"interface":"back","average":{"1min":0.90300000000000002,"5min":0.71999999999999997,"15min":0.68400000000000005},"min":{"1min":0.44400000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":1.157,"5min":1.157,"15min":1.157},"last":0.68400000000000005},{"interface":"front","average":{"1min":0.89900000000000002,"5min":0.72499999999999998,"15min":0.68799999999999994},"min":{"1min":0.56299999999999994,"5min":0.47699999999999998,"15min":0.47699999999999998},"max":{"1min":1.0660000000000001,"5min":1.0660000000000001,"15min":1.0660000000000001},"last":0.70299999999999996}]},{"osd":5,"last update":"Fri May 13 23:13:50 2022","interfaces":[{"interface":"back","average":{"1min":0.80900000000000005,"5min":0.745,"15min":0.72899999999999998},"min":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"max":{"1min":1.056,"5min":1.056,"15min":1.056},"last":0.72299999999999998},{"interface":"front","average":{"1min":0.94499999999999995,"5min":0.80000000000000004,"15min":0.76600000000000001},"min":{"1min":0.626,"5min":0.53400000000000003,"15min":0.53400000000000003},"max":{"1min":1.161,"5min":1.161,"15min":1.161},"last":0.74199999999999999}]},{"osd":6,"last update":"Fri May 13 23:13:27 2022","interfaces":[{"interface":"back","average":{"1min":0.93999999999999995,"5min":0.83299999999999996,"15min":0.81499999999999995},"min":{"1min":0.57799999999999996,"5min":0.57599999999999996,"15min":0.57599999999999996},"max":{"1min":1.2829999999999999,"5min":1.2829999999999999,"15min":1.2829999999999999},"last":0.755},{"interface":"front","average":{"1min":0.93700000000000006,"5min":0.86399999999999999,"15min":0.85199999999999998},"min":{"1min":0.61099999999999999,"5min":0.61099999999999999,"15min":0.61099999999999999},"max":{"1min":1.256,"5min":1.256,"15min":1.256},"last":0.76300000000000001}]},{"osd":7,"last update":"Fri May 13 23:13:47 2022","interfaces":[{"interface":"back","average":{"1min":1.034,"5min":0.90700000000000003,"15min":0.88600000000000001},"min":{"1min":0.70899999999999996,"5min":0.628,"15min":0.628},"max":{"1min":1.3280000000000001,"5min":1.3280000000000001,"15min":1.3280000000000001},"last":0.71599999999999997},{"interface":"front","average":{"1min":1.028,"5min":0.89100000000000001,"15min":0.86799999999999999},"min":{"1min":0.82499999999999996,"5min":0.67800000000000005,"15min":0.67800000000000005},"max":{"1min":1.3480000000000001,"5min":1.3480000000000001,"15min":1.3480000000000001},"last":0.83999999999999997}]},{"osd":8,"last update":"Fri May 13 23:14:07 2022","interfaces":[{"interface":"back","average":{"1min":0.91700000000000004,"5min":0.94699999999999995,"15min":0.95199999999999996},"min":{"1min":0.56699999999999995,"5min":0.56699999999999995,"15min":0.56699999999999995},"max":{"1min":1.2589999999999999,"5min":1.2669999999999999,"15min":1.2669999999999999},"last":1.048},{"interface":"front","average":{"1min":0.98899999999999999,"5min":1.002,"15min":1.004},"min":{"1min":0.58699999999999997,"5min":0.58699999999999997,"15min":0.58699999999999997},"max":{"1min":1.244,"5min":1.244,"15min":1.244},"last":0.91500000000000004}]},{"osd":9,"last update":"Fri May 13 23:13:39 2022","interfaces":[{"interface":"back","average":{"1min":1.131,"5min":1.131,"15min":1.131},"min":{"1min":0.79500000000000004,"5min":0.79500000000000004,"15min":0.79500000000000004},"max":{"1min":1.4319999999999999,"5min":1.4319999999999999,"15min":1.4319999999999999},"last":0.91500000000000004},{"interface":"front","average":{"1min":1.093,"5min":1.093,"15min":1.093},"min":{"1min":0.79100000000000004,"5min":0.79100000000000004,"15min":0.79100000000000004},"max":{"1min":1.4730000000000001,"5min":1.4730000000000001,"15min":1.4730000000000001},"last":0.876}]},{"osd":10,"last update":"Fri May 13 23:14:00 2022","interfaces":[{"interface":"back","average":{"1min":1.0129999999999999,"5min":1.0129999999999999,"15min":1.0129999999999999},"min":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.52300000000000002},"max":{"1min":1.373,"5min":1.373,"15min":1.373},"last":0.66200000000000003},{"interface":"front","average":{"1min":1.127,"5min":1.127,"15min":1.127},"min":{"1min":0.78300000000000003,"5min":0.78300000000000003,"15min":0.78300000000000003},"max":{"1min":1.51,"5min":1.51,"15min":1.51},"last":0.86799999999999999}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.85399999999999998}]}]},{"osd":4,"up_from":33,"seq":141733920817,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6336,"kb_used_data":888,"kb_used_omap":0,"kb_used_meta":5440,"kb_avail":93738816,"statfs":{"total":95995035648,"available":95988547584,"internally_reserved":0,"allocated":909312,"data_stored":425466,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5570560},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri May 13 23:13:35 2022","interfaces":[{"interface":"back","average":{"1min":0.627,"5min":0.56299999999999994,"15min":0.55000000000000004},"min":{"1min":0.39700000000000002,"5min":0.375,"15min":0.375},"max":{"1min":0.91800000000000004,"5min":0.91800000000000004,"15min":0.91800000000000004},"last":0.56000000000000005},{"interface":"front","average":{"1min":0.70499999999999996,"5min":0.58099999999999996,"15min":0.54500000000000004},"min":{"1min":0.47999999999999998,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.97099999999999997,"5min":0.97099999999999997,"15min":0.97099999999999997},"last":0.65200000000000002}]},{"osd":1,"last update":"Fri May 13 23:13:35 2022","interfaces":[{"interface":"back","average":{"1min":0.498,"5min":0.39200000000000002,"15min":0.372},"min":{"1min":0.309,"5min":0.217,"15min":0.217},"max":{"1min":0.84499999999999997,"5min":0.84499999999999997,"15min":0.84499999999999997},"last":0.59999999999999998},{"interface":"front","average":{"1min":0.46200000000000002,"5min":0.40300000000000002,"15min":0.39000000000000001},"min":{"1min":0.28899999999999998,"5min":0.248,"15min":0.248},"max":{"1min":0.78100000000000003,"5min":0.79900000000000004,"15min":0.79900000000000004},"last":0.622}]},{"osd":2,"last update":"Fri May 13 23:13:35 2022","interfaces":[{"interface":"back","average":{"1min":0.63800000000000001,"5min":0.54800000000000004,"15min":0.54000000000000004},"min":{"1min":0.47899999999999998,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.92600000000000005,"5min":0.92600000000000005,"15min":0.92600000000000005},"last":0.65300000000000002},{"interface":"front","average":{"1min":0.66200000000000003,"5min":0.57799999999999996,"15min":0.56499999999999995},"min":{"1min":0.5,"5min":0.372,"15min":0.372},"max":{"1min":0.92900000000000005,"5min":0.92900000000000005,"15min":0.92900000000000005},"last":0.71699999999999997}]},{"osd":3,"last update":"Fri May 13 23:13:35 2022","interfaces":[{"interface":"back","average":{"1min":0.70399999999999996,"5min":0.60799999999999998,"15min":0.58699999999999997},"min":{"1min":0.498,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.91600000000000004,"5min":0.91600000000000004,"15min":0.91600000000000004},"last":0.80500000000000005},{"interface":"front","average":{"1min":0.78800000000000003,"5min":0.63700000000000001,"15min":0.61299999999999999},"min":{"1min":0.54400000000000004,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":1.0569999999999999,"5min":1.0569999999999999,"15min":1.0569999999999999},"last":0.78800000000000003}]},{"osd":5,"last update":"Fri May 13 23:13:55 2022","interfaces":[{"interface":"back","average":{"1min":0.71399999999999997,"5min":0.58699999999999997,"15min":0.55300000000000005},"min":{"1min":0.53200000000000003,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.97999999999999998,"5min":0.97999999999999998,"15min":0.97999999999999998},"last":0.77700000000000002},{"interface":"front","average":{"1min":0.70099999999999996,"5min":0.58599999999999997,"15min":0.55600000000000005},"min":{"1min":0.55600000000000005,"5min":0.371,"15min":0.371},"max":{"1min":1.1419999999999999,"5min":1.1419999999999999,"15min":1.1419999999999999},"last":0.87}]},{"osd":6,"last update":"Fri May 13 23:13:25 2022","interfaces":[{"interface":"back","average":{"1min":0.79000000000000004,"5min":0.67800000000000005,"15min":0.66000000000000003},"min":{"1min":0.57499999999999996,"5min":0.498,"15min":0.498},"max":{"1min":1.127,"5min":1.127,"15min":1.127},"last":0.89200000000000002},{"interface":"front","average":{"1min":0.83699999999999997,"5min":0.71799999999999997,"15min":0.69799999999999995},"min":{"1min":0.60699999999999998,"5min":0.55900000000000005,"15min":0.55900000000000005},"max":{"1min":1.046,"5min":1.046,"15min":1.046},"last":0.91000000000000003}]},{"osd":7,"last update":"Fri May 13 23:13:50 2022","interfaces":[{"interface":"back","average":{"1min":0.72999999999999998,"5min":0.66500000000000004,"15min":0.65400000000000003},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.98599999999999999,"5min":0.98599999999999999,"15min":0.98599999999999999},"last":0.85299999999999998},{"interface":"front","average":{"1min":0.71699999999999997,"5min":0.64500000000000002,"15min":0.63300000000000001},"min":{"1min":0.41599999999999998,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":1.0089999999999999,"5min":1.0089999999999999,"15min":1.0089999999999999},"last":0.96699999999999997}]},{"osd":8,"last update":"Fri May 13 23:14:13 2022","interfaces":[{"interface":"back","average":{"1min":0.81499999999999995,"5min":0.75900000000000001,"15min":0.75},"min":{"1min":0.57399999999999995,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":1.099,"5min":1.099,"15min":1.099},"last":0.95599999999999996},{"interface":"front","average":{"1min":0.76000000000000001,"5min":0.754,"15min":0.753},"min":{"1min":0.44700000000000001,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":1.115,"5min":1.115,"15min":1.115},"last":0.93000000000000005}]},{"osd":9,"last update":"Fri May 13 23:13:40 2022","interfaces":[{"interface":"back","average":{"1min":0.89000000000000001,"5min":0.89000000000000001,"15min":0.89000000000000001},"min":{"1min":0.65700000000000003,"5min":0.65700000000000003,"15min":0.65700000000000003},"max":{"1min":1.1819999999999999,"5min":1.1819999999999999,"15min":1.1819999999999999},"last":1.0660000000000001},{"interface":"front","average":{"1min":0.86899999999999999,"5min":0.86899999999999999,"15min":0.86899999999999999},"min":{"1min":0.67300000000000004,"5min":0.67300000000000004,"15min":0.67300000000000004},"max":{"1min":1.171,"5min":1.171,"15min":1.171},"last":0.98399999999999999}]},{"osd":10,"last update":"Fri May 13 23:14:00 2022","interfaces":[{"interface":"back","average":{"1min":0.80700000000000005,"5min":0.80700000000000005,"15min":0.80700000000000005},"min":{"1min":0.52600000000000002,"5min":0.52600000000000002,"15min":0.52600000000000002},"max":{"1min":1.103,"5min":1.103,"15min":1.103},"last":1.016},{"interface":"front","average":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"min":{"1min":0.48499999999999999,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":1.196,"5min":1.196,"15min":1.196},"last":0.94099999999999995}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0329999999999999}]}]},{"osd":5,"up_from":38,"seq":163208757292,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6940,"kb_used_data":1428,"kb_used_omap":0,"kb_used_meta":5504,"kb_avail":93738212,"statfs":{"total":95995035648,"available":95987929088,"internally_reserved":0,"allocated":1462272,"data_stored":956107,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5636096},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri May 13 23:13:51 2022","interfaces":[{"interface":"back","average":{"1min":0.61099999999999999,"5min":0.53700000000000003,"15min":0.52300000000000002},"min":{"1min":0.376,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.96299999999999997,"5min":0.96299999999999997,"15min":0.96299999999999997},"last":0.59199999999999997},{"interface":"front","average":{"1min":0.57199999999999995,"5min":0.54200000000000004,"15min":0.52300000000000002},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.91300000000000003,"5min":0.96499999999999997,"15min":0.96499999999999997},"last":0.69599999999999995}]},{"osd":1,"last update":"Fri May 13 23:13:51 2022","interfaces":[{"interface":"back","average":{"1min":0.57699999999999996,"5min":0.504,"15min":0.48899999999999999},"min":{"1min":0.41499999999999998,"5min":0.317,"15min":0.317},"max":{"1min":0.76900000000000002,"5min":0.76900000000000002,"15min":0.76900000000000002},"last":0.52200000000000002},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.502,"15min":0.47999999999999998},"min":{"1min":0.42599999999999999,"5min":0.313,"15min":0.313},"max":{"1min":0.78800000000000003,"5min":0.78800000000000003,"15min":0.78800000000000003},"last":0.443}]},{"osd":2,"last update":"Fri May 13 23:13:51 2022","interfaces":[{"interface":"back","average":{"1min":0.42399999999999999,"5min":0.38100000000000001,"15min":0.36299999999999999},"min":{"1min":0.28399999999999997,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":0.65600000000000003,"5min":0.65600000000000003,"15min":0.65600000000000003},"last":0.40300000000000002},{"interface":"front","average":{"1min":0.45000000000000001,"5min":0.39500000000000002,"15min":0.375},"min":{"1min":0.28899999999999998,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.75700000000000001,"5min":0.75700000000000001,"15min":0.75700000000000001},"last":0.36599999999999999}]},{"osd":3,"last update":"Fri May 13 23:13:51 2022","interfaces":[{"interface":"back","average":{"1min":0.57499999999999996,"5min":0.56499999999999995,"15min":0.55900000000000005},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.97999999999999998,"5min":0.97999999999999998,"15min":0.97999999999999998},"last":0.71699999999999997},{"interface":"front","average":{"1min":0.625,"5min":0.57699999999999996,"15min":0.55300000000000005},"min":{"1min":0.45600000000000002,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":0.89900000000000002,"5min":1,"15min":1},"last":0.56599999999999995}]},{"osd":4,"last update":"Fri May 13 23:13:51 2022","interfaces":[{"interface":"back","average":{"1min":0.63700000000000001,"5min":0.55800000000000005,"15min":0.53700000000000003},"min":{"1min":0.38800000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.86399999999999999,"5min":0.86399999999999999,"15min":0.86399999999999999},"last":0.53800000000000003},{"interface":"front","average":{"1min":0.58299999999999996,"5min":0.54200000000000004,"15min":0.53200000000000003},"min":{"1min":0.42599999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.73399999999999999,"5min":0.89500000000000002,"15min":0.89500000000000002},"last":0.50600000000000001}]},{"osd":6,"last update":"Fri May 13 23:13:25 2022","interfaces":[{"interface":"back","average":{"1min":0.60899999999999999,"5min":0.623,"15min":0.625},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":0.872,"5min":0.95399999999999996,"15min":0.95399999999999996},"last":0.57899999999999996},{"interface":"front","average":{"1min":0.65100000000000002,"5min":0.626,"15min":0.622},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.98399999999999999,"5min":0.98399999999999999,"15min":0.98399999999999999},"last":0.60999999999999999}]},{"osd":7,"last update":"Fri May 13 23:13:46 2022","interfaces":[{"interface":"back","average":{"1min":0.629,"5min":0.64100000000000001,"15min":0.64300000000000002},"min":{"1min":0.45200000000000001,"5min":0.379,"15min":0.379},"max":{"1min":0.871,"5min":1.0489999999999999,"15min":1.0489999999999999},"last":0.65800000000000003},{"interface":"front","average":{"1min":0.66700000000000004,"5min":0.64500000000000002,"15min":0.64100000000000001},"min":{"1min":0.442,"5min":0.442,"15min":0.442},"max":{"1min":0.93000000000000005,"5min":0.93000000000000005,"15min":0.93000000000000005},"last":0.67100000000000004}]},{"osd":8,"last update":"Fri May 13 23:14:06 2022","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.61599999999999999,"15min":0.622},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.89200000000000002,"5min":0.999,"15min":0.999},"last":0.59599999999999997},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.56599999999999995,"15min":0.56899999999999995},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.82699999999999996,"5min":0.82699999999999996,"15min":0.82699999999999996},"last":0.64100000000000001}]},{"osd":9,"last update":"Fri May 13 23:13:40 2022","interfaces":[{"interface":"back","average":{"1min":0.68600000000000005,"5min":0.68600000000000005,"15min":0.68600000000000005},"min":{"1min":0.51400000000000001,"5min":0.51400000000000001,"15min":0.51400000000000001},"max":{"1min":0.88500000000000001,"5min":0.88500000000000001,"15min":0.88500000000000001},"last":0.753},{"interface":"front","average":{"1min":0.71199999999999997,"5min":0.71199999999999997,"15min":0.71199999999999997},"min":{"1min":0.505,"5min":0.505,"15min":0.505},"max":{"1min":0.99299999999999999,"5min":0.99299999999999999,"15min":0.99299999999999999},"last":0.73899999999999999}]},{"osd":10,"last update":"Fri May 13 23:13:59 2022","interfaces":[{"interface":"back","average":{"1min":0.67900000000000005,"5min":0.67900000000000005,"15min":0.67900000000000005},"min":{"1min":0.503,"5min":0.503,"15min":0.503},"max":{"1min":0.90800000000000003,"5min":0.90800000000000003,"15min":0.90800000000000003},"last":0.68100000000000005},{"interface":"front","average":{"1min":0.71399999999999997,"5min":0.71399999999999997,"15min":0.71399999999999997},"min":{"1min":0.48499999999999999,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":0.94499999999999995,"5min":0.94499999999999995,"15min":0.94499999999999995},"last":0.70499999999999996}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2074,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":3184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":3384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":2688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":674,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":2942,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":1444,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1094,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":3196,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1116,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2022-05-13T23:14:27.730 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2022-05-13T23:14:27.732 INFO:tasks.cephadm:Setting up client nodes... 2022-05-13T23:14:27.732 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:23b0658391ce94ca40799469eec00c28b298816c shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 16cf5f88-d311-11ec-be40-001a4aab830c -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2022-05-13T23:14:28.433 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/16cf5f88-d311-11ec-be40-001a4aab830c/mon.a/config 2022-05-13T23:14:28.572 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:14:28 smithi002 ceph-mon[71816]: pgmap v294: 97 pgs: 97 active+clean; 584 KiB data, 77 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 5.5 KiB/s wr, 23 op/s 2022-05-13T23:14:28.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:14:28 smithi115 ceph-mon[78593]: pgmap v294: 97 pgs: 97 active+clean; 584 KiB data, 77 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 5.5 KiB/s wr, 23 op/s 2022-05-13T23:14:28.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:14:28 smithi047 ceph-mon[77147]: pgmap v294: 97 pgs: 97 active+clean; 584 KiB data, 77 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 5.5 KiB/s wr, 23 op/s 2022-05-13T23:14:29.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:14:29 smithi115 ceph-mon[78593]: from='client.24706 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2022-05-13T23:14:29.660 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:14:29 smithi002 ceph-mon[71816]: from='client.24706 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2022-05-13T23:14:29.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:14:29 smithi047 ceph-mon[77147]: from='client.24706 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2022-05-13T23:14:30.572 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:14:30 smithi002 ceph-mon[71816]: pgmap v295: 97 pgs: 97 active+clean; 584 KiB data, 77 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 4.6 KiB/s wr, 19 op/s 2022-05-13T23:14:30.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:14:30 smithi115 ceph-mon[78593]: pgmap v295: 97 pgs: 97 active+clean; 584 KiB data, 77 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 4.6 KiB/s wr, 19 op/s 2022-05-13T23:14:30.609 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:30.606+0000 7fc2cf05a700 1 -- 172.21.15.2:0/3262827257 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc2c8070d50 msgr2=0x7fc2c8071160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:14:30.609 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:30.606+0000 7fc2cf05a700 1 --2- 172.21.15.2:0/3262827257 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc2c8070d50 0x7fc2c8071160 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fc2c4004870 tx=0x7fc2c4009cb0 comp rx=0 tx=0).stop 2022-05-13T23:14:30.609 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:30.606+0000 7fc2cf05a700 1 -- 172.21.15.2:0/3262827257 shutdown_connections 2022-05-13T23:14:30.609 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:30.606+0000 7fc2cf05a700 1 --2- 172.21.15.2:0/3262827257 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc2c80729f0 0x7fc2c8073280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:14:30.610 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:30.606+0000 7fc2cf05a700 1 --2- 172.21.15.2:0/3262827257 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc2c8070d50 0x7fc2c8071160 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:14:30.610 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:30.606+0000 7fc2cf05a700 1 --2- 172.21.15.2:0/3262827257 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fc2c8072040 0x7fc2c80724b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:14:30.610 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:30.606+0000 7fc2cf05a700 1 -- 172.21.15.2:0/3262827257 >> 172.21.15.2:0/3262827257 conn(0x7fc2c80f8e70 msgr2=0x7fc2c80fb250 unknown :-1 s=STATE_NONE l=0).mark_down 2022-05-13T23:14:30.610 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:30.606+0000 7fc2cf05a700 1 -- 172.21.15.2:0/3262827257 shutdown_connections 2022-05-13T23:14:30.610 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:30.606+0000 7fc2cf05a700 1 -- 172.21.15.2:0/3262827257 wait complete. 2022-05-13T23:14:30.612 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:30.607+0000 7fc2cf05a700 1 Processor -- start 2022-05-13T23:14:30.613 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:30.607+0000 7fc2cf05a700 1 -- start start 2022-05-13T23:14:30.613 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:30.608+0000 7fc2cf05a700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc2c8070d50 0x7fc2c81126d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:14:30.613 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:30.608+0000 7fc2cf05a700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc2c8072040 0x7fc2c8112c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:14:30.613 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:30.608+0000 7fc2cf05a700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fc2c80729f0 0x7fc2c8119ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:14:30.614 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:30.608+0000 7fc2cf05a700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2c811a1e0 con 0x7fc2c8072040 2022-05-13T23:14:30.614 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:30.608+0000 7fc2cf05a700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2c811a350 con 0x7fc2c8070d50 2022-05-13T23:14:30.614 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:30.608+0000 7fc2cf05a700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2c811a510 con 0x7fc2c80729f0 2022-05-13T23:14:30.614 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:30.608+0000 7fc2cd857700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc2c8072040 0x7fc2c8112c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:14:30.614 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:30.608+0000 7fc2cd857700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc2c8072040 0x7fc2c8112c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:50780/0 (socket says 172.21.15.2:50780) 2022-05-13T23:14:30.615 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:30.608+0000 7fc2ce058700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc2c8070d50 0x7fc2c81126d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:14:30.615 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:30.608+0000 7fc2cd857700 1 -- 172.21.15.2:0/3179484711 learned_addr learned my addr 172.21.15.2:0/3179484711 (peer_addr_for_me v2:172.21.15.2:0/0) 2022-05-13T23:14:30.615 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:30.608+0000 7fc2ce859700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fc2c80729f0 0x7fc2c8119ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:14:30.615 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:30.608+0000 7fc2cd857700 1 -- 172.21.15.2:0/3179484711 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fc2c80729f0 msgr2=0x7fc2c8119ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:14:30.615 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:30.608+0000 7fc2cd857700 1 --2- 172.21.15.2:0/3179484711 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fc2c80729f0 0x7fc2c8119ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:14:30.616 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:30.608+0000 7fc2cd857700 1 -- 172.21.15.2:0/3179484711 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc2c8070d50 msgr2=0x7fc2c81126d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:14:30.616 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:30.608+0000 7fc2cd857700 1 --2- 172.21.15.2:0/3179484711 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc2c8070d50 0x7fc2c81126d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:14:30.616 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:30.609+0000 7fc2cd857700 1 -- 172.21.15.2:0/3179484711 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2c00097c0 con 0x7fc2c8072040 2022-05-13T23:14:30.617 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:30.609+0000 7fc2ce058700 1 --2- 172.21.15.2:0/3179484711 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc2c8070d50 0x7fc2c81126d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-05-13T23:14:30.617 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:30.609+0000 7fc2cd857700 1 --2- 172.21.15.2:0/3179484711 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc2c8072040 0x7fc2c8112c10 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fc2b80027f0 tx=0x7fc2b8002b00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-05-13T23:14:30.617 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:30.609+0000 7fc2beffd700 1 -- 172.21.15.2:0/3179484711 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2b8007b40 con 0x7fc2c8072040 2022-05-13T23:14:30.617 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:30.609+0000 7fc2cf05a700 1 -- 172.21.15.2:0/3179484711 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc2c4005040 con 0x7fc2c8072040 2022-05-13T23:14:30.617 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:30.609+0000 7fc2beffd700 1 -- 172.21.15.2:0/3179484711 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1087+0+0 (secure 0 0 0) 0x7fc2b800f040 con 0x7fc2c8072040 2022-05-13T23:14:30.618 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:30.609+0000 7fc2beffd700 1 -- 172.21.15.2:0/3179484711 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2b8013730 con 0x7fc2c8072040 2022-05-13T23:14:30.618 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:30.609+0000 7fc2cf05a700 1 -- 172.21.15.2:0/3179484711 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc2c811abd0 con 0x7fc2c8072040 2022-05-13T23:14:30.619 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:30.611+0000 7fc2cf05a700 1 -- 172.21.15.2:0/3179484711 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc2c810d200 con 0x7fc2c8072040 2022-05-13T23:14:30.619 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:30.612+0000 7fc2beffd700 1 -- 172.21.15.2:0/3179484711 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 16) v1 ==== 125136+0+0 (secure 0 0 0) 0x7fc2b8013b60 con 0x7fc2c8072040 2022-05-13T23:14:30.620 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:30.614+0000 7fc2beffd700 1 --2- 172.21.15.2:0/3179484711 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7fc2b4096ce0 0x7fc2b4099190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:14:30.620 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:30.614+0000 7fc2beffd700 1 -- 172.21.15.2:0/3179484711 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 7975+0+0 (secure 0 0 0) 0x7fc2b80be9a0 con 0x7fc2c8072040 2022-05-13T23:14:30.620 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:30.614+0000 7fc2ce058700 1 --2- 172.21.15.2:0/3179484711 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7fc2b4096ce0 0x7fc2b4099190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:14:30.621 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:30.615+0000 7fc2ce058700 1 --2- 172.21.15.2:0/3179484711 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7fc2b4096ce0 0x7fc2b4099190 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fc2c4013490 tx=0x7fc2c4009cb0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-05-13T23:14:30.621 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:30.616+0000 7fc2beffd700 1 -- 172.21.15.2:0/3179484711 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+165215 (secure 0 0 0) 0x7fc2b808bce0 con 0x7fc2c8072040 2022-05-13T23:14:30.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:14:30 smithi047 ceph-mon[77147]: pgmap v295: 97 pgs: 97 active+clean; 584 KiB data, 77 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 4.6 KiB/s wr, 19 op/s 2022-05-13T23:14:30.886 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:30.883+0000 7fc2cf05a700 1 -- 172.21.15.2:0/3179484711 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} v 0) v1 -- 0x7fc2c810a010 con 0x7fc2c8072040 2022-05-13T23:14:30.890 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:30.888+0000 7fc2beffd700 1 -- 172.21.15.2:0/3179484711 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]=0 v32) v1 ==== 170+0+59 (secure 0 0 0) 0x7fc2b808f300 con 0x7fc2c8072040 2022-05-13T23:14:30.891 INFO:teuthology.orchestra.run.smithi002.stdout:[client.0] 2022-05-13T23:14:30.892 INFO:teuthology.orchestra.run.smithi002.stdout: key = AQBW5n5iFhfMNBAAVZUkGgQci1JcBZvP5zcswg== 2022-05-13T23:14:30.894 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:30.891+0000 7fc2cf05a700 1 -- 172.21.15.2:0/3179484711 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7fc2b4096ce0 msgr2=0x7fc2b4099190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:14:30.894 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:30.891+0000 7fc2cf05a700 1 --2- 172.21.15.2:0/3179484711 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7fc2b4096ce0 0x7fc2b4099190 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fc2c4013490 tx=0x7fc2c4009cb0 comp rx=0 tx=0).stop 2022-05-13T23:14:30.894 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:30.891+0000 7fc2cf05a700 1 -- 172.21.15.2:0/3179484711 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc2c8072040 msgr2=0x7fc2c8112c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:14:30.895 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:30.891+0000 7fc2cf05a700 1 --2- 172.21.15.2:0/3179484711 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc2c8072040 0x7fc2c8112c10 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fc2b80027f0 tx=0x7fc2b8002b00 comp rx=0 tx=0).stop 2022-05-13T23:14:30.895 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:30.891+0000 7fc2cf05a700 1 -- 172.21.15.2:0/3179484711 shutdown_connections 2022-05-13T23:14:30.895 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:30.891+0000 7fc2cf05a700 1 --2- 172.21.15.2:0/3179484711 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fc2c80729f0 0x7fc2c8119ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:14:30.896 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:30.891+0000 7fc2cf05a700 1 --2- 172.21.15.2:0/3179484711 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc2c8070d50 0x7fc2c81126d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:14:30.896 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:30.891+0000 7fc2cf05a700 1 --2- 172.21.15.2:0/3179484711 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7fc2b4096ce0 0x7fc2b4099190 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:14:30.896 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:30.891+0000 7fc2cf05a700 1 --2- 172.21.15.2:0/3179484711 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc2c8072040 0x7fc2c8112c10 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:14:30.896 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:30.891+0000 7fc2cf05a700 1 -- 172.21.15.2:0/3179484711 >> 172.21.15.2:0/3179484711 conn(0x7fc2c80f8e70 msgr2=0x7fc2c80fabd0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-05-13T23:14:30.896 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:30.891+0000 7fc2cf05a700 1 -- 172.21.15.2:0/3179484711 shutdown_connections 2022-05-13T23:14:30.897 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:30.891+0000 7fc2cf05a700 1 -- 172.21.15.2:0/3179484711 wait complete. 2022-05-13T23:14:31.393 DEBUG:teuthology.orchestra.run.smithi002:> set -ex 2022-05-13T23:14:31.394 DEBUG:teuthology.orchestra.run.smithi002:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2022-05-13T23:14:31.394 DEBUG:teuthology.orchestra.run.smithi002:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2022-05-13T23:14:31.466 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:14:31 smithi002 ceph-mon[71816]: from='client.? 172.21.15.2:0/3179484711' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2022-05-13T23:14:31.467 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:14:31 smithi002 ceph-mon[71816]: from='client.? 172.21.15.2:0/3179484711' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2022-05-13T23:14:31.512 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2022-05-13T23:14:31.514 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2022-05-13T23:14:31.515 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:23b0658391ce94ca40799469eec00c28b298816c shell --fsid 16cf5f88-d311-11ec-be40-001a4aab830c -- ceph mgr dump --format=json 2022-05-13T23:14:31.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:14:31 smithi115 ceph-mon[78593]: from='client.? 172.21.15.2:0/3179484711' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2022-05-13T23:14:31.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:14:31 smithi115 ceph-mon[78593]: from='client.? 172.21.15.2:0/3179484711' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2022-05-13T23:14:31.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:14:31 smithi047 ceph-mon[77147]: from='client.? 172.21.15.2:0/3179484711' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2022-05-13T23:14:31.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:14:31 smithi047 ceph-mon[77147]: from='client.? 172.21.15.2:0/3179484711' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2022-05-13T23:14:32.187 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/16cf5f88-d311-11ec-be40-001a4aab830c/mon.a/config 2022-05-13T23:14:32.597 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:14:32 smithi002 ceph-mon[71816]: pgmap v296: 97 pgs: 97 active+clean; 584 KiB data, 77 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 4.6 KiB/s wr, 22 op/s 2022-05-13T23:14:32.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:14:32 smithi115 ceph-mon[78593]: pgmap v296: 97 pgs: 97 active+clean; 584 KiB data, 77 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 4.6 KiB/s wr, 22 op/s 2022-05-13T23:14:32.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:14:32 smithi047 ceph-mon[77147]: pgmap v296: 97 pgs: 97 active+clean; 584 KiB data, 77 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 4.6 KiB/s wr, 22 op/s 2022-05-13T23:14:34.016 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:34.013+0000 7fbc92fe7700 1 -- 172.21.15.2:0/1640572664 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fbc8c0ff440 msgr2=0x7fbc8c105d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:14:34.017 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:34.013+0000 7fbc92fe7700 1 --2- 172.21.15.2:0/1640572664 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fbc8c0ff440 0x7fbc8c105d00 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fbc84009be0 tx=0x7fbc84004d10 comp rx=0 tx=0).stop 2022-05-13T23:14:34.017 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:34.014+0000 7fbc92fe7700 1 -- 172.21.15.2:0/1640572664 shutdown_connections 2022-05-13T23:14:34.017 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:34.014+0000 7fbc92fe7700 1 --2- 172.21.15.2:0/1640572664 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fbc8c0ff440 0x7fbc8c105d00 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:14:34.017 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:34.014+0000 7fbc92fe7700 1 --2- 172.21.15.2:0/1640572664 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fbc8c0fd810 0x7fbc8c0fdc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:14:34.018 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:34.014+0000 7fbc92fe7700 1 --2- 172.21.15.2:0/1640572664 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fbc8c0feab0 0x7fbc8c0fef00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:14:34.018 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:34.014+0000 7fbc92fe7700 1 -- 172.21.15.2:0/1640572664 >> 172.21.15.2:0/1640572664 conn(0x7fbc8c0f9030 msgr2=0x7fbc8c0fb430 unknown :-1 s=STATE_NONE l=0).mark_down 2022-05-13T23:14:34.018 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:34.014+0000 7fbc92fe7700 1 -- 172.21.15.2:0/1640572664 shutdown_connections 2022-05-13T23:14:34.018 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:34.014+0000 7fbc92fe7700 1 -- 172.21.15.2:0/1640572664 wait complete. 2022-05-13T23:14:34.020 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:34.015+0000 7fbc92fe7700 1 Processor -- start 2022-05-13T23:14:34.021 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:34.015+0000 7fbc92fe7700 1 -- start start 2022-05-13T23:14:34.021 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:34.015+0000 7fbc92fe7700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fbc8c0fd810 0x7fbc8c10e4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:14:34.021 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:34.015+0000 7fbc92fe7700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fbc8c0feab0 0x7fbc8c10e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:14:34.021 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:34.015+0000 7fbc92fe7700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fbc8c0ff440 0x7fbc8c115a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:14:34.022 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:34.015+0000 7fbc92fe7700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc8c115fc0 con 0x7fbc8c0fd810 2022-05-13T23:14:34.022 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:34.015+0000 7fbc92fe7700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc8c116130 con 0x7fbc8c0feab0 2022-05-13T23:14:34.022 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:34.015+0000 7fbc92fe7700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc8c1162f0 con 0x7fbc8c0ff440 2022-05-13T23:14:34.023 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:34.015+0000 7fbc90d83700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fbc8c0fd810 0x7fbc8c10e4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:14:34.023 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:34.016+0000 7fbc90d83700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fbc8c0fd810 0x7fbc8c10e4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:50808/0 (socket says 172.21.15.2:50808) 2022-05-13T23:14:34.023 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:34.016+0000 7fbc8bfff700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fbc8c0feab0 0x7fbc8c10e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:14:34.023 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:34.016+0000 7fbc90d83700 1 -- 172.21.15.2:0/3887011484 learned_addr learned my addr 172.21.15.2:0/3887011484 (peer_addr_for_me v2:172.21.15.2:0/0) 2022-05-13T23:14:34.024 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:34.016+0000 7fbc91584700 1 --2- 172.21.15.2:0/3887011484 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fbc8c0ff440 0x7fbc8c115a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:14:34.024 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:34.016+0000 7fbc8bfff700 1 -- 172.21.15.2:0/3887011484 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fbc8c0ff440 msgr2=0x7fbc8c115a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:14:34.024 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:34.016+0000 7fbc8bfff700 1 --2- 172.21.15.2:0/3887011484 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fbc8c0ff440 0x7fbc8c115a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:14:34.024 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:34.016+0000 7fbc8bfff700 1 -- 172.21.15.2:0/3887011484 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fbc8c0fd810 msgr2=0x7fbc8c10e4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:14:34.024 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:34.016+0000 7fbc8bfff700 1 --2- 172.21.15.2:0/3887011484 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fbc8c0fd810 0x7fbc8c10e4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:14:34.025 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:34.016+0000 7fbc8bfff700 1 -- 172.21.15.2:0/3887011484 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc78005040 con 0x7fbc8c0feab0 2022-05-13T23:14:34.025 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:34.016+0000 7fbc8bfff700 1 --2- 172.21.15.2:0/3887011484 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fbc8c0feab0 0x7fbc8c10e9f0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fbc78004920 tx=0x7fbc78009cb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-05-13T23:14:34.025 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:34.017+0000 7fbc897fa700 1 -- 172.21.15.2:0/3887011484 <== mon.1 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc7800e050 con 0x7fbc8c0feab0 2022-05-13T23:14:34.025 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:34.017+0000 7fbc92fe7700 1 -- 172.21.15.2:0/3887011484 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbc840097c0 con 0x7fbc8c0feab0 2022-05-13T23:14:34.026 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:34.017+0000 7fbc92fe7700 1 -- 172.21.15.2:0/3887011484 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbc8c116a00 con 0x7fbc8c0feab0 2022-05-13T23:14:34.026 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:34.017+0000 7fbc897fa700 1 -- 172.21.15.2:0/3887011484 <== mon.1 v2:172.21.15.47:3300/0 2 ==== config(26 keys) v1 ==== 1087+0+0 (secure 0 0 0) 0x7fbc7800f040 con 0x7fbc8c0feab0 2022-05-13T23:14:34.026 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:34.017+0000 7fbc897fa700 1 -- 172.21.15.2:0/3887011484 <== mon.1 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc78014970 con 0x7fbc8c0feab0 2022-05-13T23:14:34.026 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:34.018+0000 7fbc92fe7700 1 -- 172.21.15.2:0/3887011484 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbc8c108fc0 con 0x7fbc8c0feab0 2022-05-13T23:14:34.027 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:34.020+0000 7fbc897fa700 1 -- 172.21.15.2:0/3887011484 <== mon.1 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 16) v1 ==== 125136+0+0 (secure 0 0 0) 0x7fbc7801b030 con 0x7fbc8c0feab0 2022-05-13T23:14:34.028 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:34.021+0000 7fbc897fa700 1 --2- 172.21.15.2:0/3887011484 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7fbc7c096ce0 0x7fbc7c099190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:14:34.028 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:34.021+0000 7fbc897fa700 1 -- 172.21.15.2:0/3887011484 <== mon.1 v2:172.21.15.47:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 7975+0+0 (secure 0 0 0) 0x7fbc780be970 con 0x7fbc8c0feab0 2022-05-13T23:14:34.028 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:34.022+0000 7fbc90d83700 1 --2- 172.21.15.2:0/3887011484 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7fbc7c096ce0 0x7fbc7c099190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:14:34.028 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:34.022+0000 7fbc90d83700 1 --2- 172.21.15.2:0/3887011484 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7fbc7c096ce0 0x7fbc7c099190 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fbc800041d0 tx=0x7fbc8000b040 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-05-13T23:14:34.029 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:34.024+0000 7fbc897fa700 1 -- 172.21.15.2:0/3887011484 <== mon.1 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+165215 (secure 0 0 0) 0x7fbc7808bcb0 con 0x7fbc8c0feab0 2022-05-13T23:14:34.268 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:34.265+0000 7fbc92fe7700 1 -- 172.21.15.2:0/3887011484 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) v1 -- 0x7fbc8c104cc0 con 0x7fbc8c0feab0 2022-05-13T23:14:34.273 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:34.271+0000 7fbc897fa700 1 -- 172.21.15.2:0/3887011484 <== mon.1 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v16) v1 ==== 74+0+241810 (secure 0 0 0) 0x7fbc7808f2d0 con 0x7fbc8c0feab0 2022-05-13T23:14:34.281 INFO:teuthology.orchestra.run.smithi002.stdout: 2022-05-13T23:14:34.285 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:34.276+0000 7fbc92fe7700 1 -- 172.21.15.2:0/3887011484 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7fbc7c096ce0 msgr2=0x7fbc7c099190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:14:34.285 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:34.276+0000 7fbc92fe7700 1 --2- 172.21.15.2:0/3887011484 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7fbc7c096ce0 0x7fbc7c099190 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fbc800041d0 tx=0x7fbc8000b040 comp rx=0 tx=0).stop 2022-05-13T23:14:34.286 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:34.276+0000 7fbc92fe7700 1 -- 172.21.15.2:0/3887011484 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fbc8c0feab0 msgr2=0x7fbc8c10e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:14:34.286 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:34.276+0000 7fbc92fe7700 1 --2- 172.21.15.2:0/3887011484 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fbc8c0feab0 0x7fbc8c10e9f0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fbc78004920 tx=0x7fbc78009cb0 comp rx=0 tx=0).stop 2022-05-13T23:14:34.286 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:34.276+0000 7fbc92fe7700 1 -- 172.21.15.2:0/3887011484 shutdown_connections 2022-05-13T23:14:34.287 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:34.276+0000 7fbc92fe7700 1 --2- 172.21.15.2:0/3887011484 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fbc8c0ff440 0x7fbc8c115a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:14:34.287 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:34.276+0000 7fbc92fe7700 1 --2- 172.21.15.2:0/3887011484 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7fbc7c096ce0 0x7fbc7c099190 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:14:34.288 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:34.276+0000 7fbc92fe7700 1 --2- 172.21.15.2:0/3887011484 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fbc8c0fd810 0x7fbc8c10e4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:14:34.288 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:34.276+0000 7fbc92fe7700 1 --2- 172.21.15.2:0/3887011484 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fbc8c0feab0 0x7fbc8c10e9f0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:14:34.289 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:34.276+0000 7fbc92fe7700 1 -- 172.21.15.2:0/3887011484 >> 172.21.15.2:0/3887011484 conn(0x7fbc8c0f9030 msgr2=0x7fbc8c101fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-05-13T23:14:34.289 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:34.276+0000 7fbc92fe7700 1 -- 172.21.15.2:0/3887011484 shutdown_connections 2022-05-13T23:14:34.289 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:34.276+0000 7fbc92fe7700 1 -- 172.21.15.2:0/3887011484 wait complete. 2022-05-13T23:14:34.572 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:14:34 smithi002 ceph-mon[71816]: pgmap v297: 97 pgs: 97 active+clean; 584 KiB data, 77 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 3.2 KiB/s wr, 19 op/s 2022-05-13T23:14:34.573 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:14:34 smithi002 ceph-mon[71816]: from='client.? 172.21.15.2:0/3887011484' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2022-05-13T23:14:34.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:14:34 smithi115 ceph-mon[78593]: pgmap v297: 97 pgs: 97 active+clean; 584 KiB data, 77 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 3.2 KiB/s wr, 19 op/s 2022-05-13T23:14:34.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:14:34 smithi115 ceph-mon[78593]: from='client.? 172.21.15.2:0/3887011484' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2022-05-13T23:14:34.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:14:34 smithi047 ceph-mon[77147]: pgmap v297: 97 pgs: 97 active+clean; 584 KiB data, 77 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 3.2 KiB/s wr, 19 op/s 2022-05-13T23:14:34.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:14:34 smithi047 ceph-mon[77147]: from='client.? 172.21.15.2:0/3887011484' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2022-05-13T23:14:34.850 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":16,"active_gid":14148,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6800","nonce":2788339627},{"type":"v1","addr":"172.21.15.2:6801","nonce":2788339627}]},"active_addr":"172.21.15.2:6801/2788339627","active_change":"2022-05-13T23:07:04.986544+0000","active_mgr_features":4540138303579357183,"available":true,"standbys":[{"gid":24115,"name":"y","mgr_features":4540138303579357183,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.23.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:8.3.5","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"docker.io/library/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"docker.io/arcts/keepalived","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.3.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.33.4","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"noautoscale":{"name":"noautoscale","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"global autoscale flag","long_desc":"Option to turn on/off the autoscaler for all pools","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]},{"gid":24119,"name":"z","mgr_features":4540138303579357183,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.23.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:8.3.5","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"docker.io/library/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"docker.io/arcts/keepalived","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.3.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.33.4","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"noautoscale":{"name":"noautoscale","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"global autoscale flag","long_desc":"Option to turn on/off the autoscaler for all pools","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.23.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:8.3.5","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"docker.io/library/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"docker.io/arcts/keepalived","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.3.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.33.4","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"noautoscale":{"name":"noautoscale","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"global autoscale flag","long_desc":"Option to turn on/off the autoscaler for all pools","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.2:8443/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"last_failure_osd_epoch":3,"active_clients":[{"addrvec":[{"type":"v2","addr":"172.21.15.2:0","nonce":482592463}]},{"addrvec":[{"type":"v2","addr":"172.21.15.2:0","nonce":1504548965}]},{"addrvec":[{"type":"v2","addr":"172.21.15.2:0","nonce":3509967930}]},{"addrvec":[{"type":"v2","addr":"172.21.15.2:0","nonce":2926980153}]}]}} 2022-05-13T23:14:34.857 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2022-05-13T23:14:34.857 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2022-05-13T23:14:34.858 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:23b0658391ce94ca40799469eec00c28b298816c shell --fsid 16cf5f88-d311-11ec-be40-001a4aab830c -- ceph osd dump --format=json 2022-05-13T23:14:35.532 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/16cf5f88-d311-11ec-be40-001a4aab830c/mon.a/config 2022-05-13T23:14:36.572 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:14:36 smithi002 ceph-mon[71816]: pgmap v298: 97 pgs: 97 active+clean; 584 KiB data, 77 MiB used, 1.0 TiB / 1.0 TiB avail; 6.8 KiB/s rd, 2.5 KiB/s wr, 19 op/s 2022-05-13T23:14:36.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:14:36 smithi115 ceph-mon[78593]: pgmap v298: 97 pgs: 97 active+clean; 584 KiB data, 77 MiB used, 1.0 TiB / 1.0 TiB avail; 6.8 KiB/s rd, 2.5 KiB/s wr, 19 op/s 2022-05-13T23:14:36.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:14:36 smithi047 ceph-mon[77147]: pgmap v298: 97 pgs: 97 active+clean; 584 KiB data, 77 MiB used, 1.0 TiB / 1.0 TiB avail; 6.8 KiB/s rd, 2.5 KiB/s wr, 19 op/s 2022-05-13T23:14:37.244 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:37.241+0000 7f240dad8700 1 -- 172.21.15.2:0/1864979135 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2408103400 msgr2=0x7f24081057e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:14:37.245 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:37.241+0000 7f240dad8700 1 --2- 172.21.15.2:0/1864979135 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2408103400 0x7f24081057e0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f2400009be0 tx=0x7f2400004d10 comp rx=0 tx=0).stop 2022-05-13T23:14:37.246 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:37.241+0000 7f240dad8700 1 -- 172.21.15.2:0/1864979135 shutdown_connections 2022-05-13T23:14:37.246 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:37.241+0000 7f240dad8700 1 --2- 172.21.15.2:0/1864979135 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2408103400 0x7f24081057e0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:14:37.246 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:37.241+0000 7f240dad8700 1 --2- 172.21.15.2:0/1864979135 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f2408070370 0x7f240806e870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:14:37.247 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:37.241+0000 7f240dad8700 1 --2- 172.21.15.2:0/1864979135 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f240806edb0 0x7f240806f220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:14:37.247 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:37.241+0000 7f240dad8700 1 -- 172.21.15.2:0/1864979135 >> 172.21.15.2:0/1864979135 conn(0x7f24080f6ba0 msgr2=0x7f24080f8ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-05-13T23:14:37.248 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:37.241+0000 7f240dad8700 1 -- 172.21.15.2:0/1864979135 shutdown_connections 2022-05-13T23:14:37.248 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:37.241+0000 7f240dad8700 1 -- 172.21.15.2:0/1864979135 wait complete. 2022-05-13T23:14:37.249 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:37.242+0000 7f240dad8700 1 Processor -- start 2022-05-13T23:14:37.250 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:37.242+0000 7f240dad8700 1 -- start start 2022-05-13T23:14:37.254 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:37.242+0000 7f240dad8700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f240806edb0 0x7f240810e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:14:37.254 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:37.242+0000 7f240dad8700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f2408070370 0x7f240810e830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:14:37.255 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:37.242+0000 7f240dad8700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f2408103400 0x7f2408115870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:14:37.255 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:37.242+0000 7f240dad8700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2408115db0 con 0x7f240806edb0 2022-05-13T23:14:37.255 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:37.242+0000 7f240dad8700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2408115f20 con 0x7f2408070370 2022-05-13T23:14:37.256 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:37.242+0000 7f240dad8700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24081160e0 con 0x7f2408103400 2022-05-13T23:14:37.256 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:37.243+0000 7f240cad6700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f240806edb0 0x7f240810e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:14:37.257 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:37.243+0000 7f2407fff700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f2408070370 0x7f240810e830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:14:37.257 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:37.243+0000 7f2407fff700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f2408070370 0x7f240810e830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.2:40108/0 (socket says 172.21.15.2:40108) 2022-05-13T23:14:37.257 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:37.243+0000 7f2407fff700 1 -- 172.21.15.2:0/1454874490 learned_addr learned my addr 172.21.15.2:0/1454874490 (peer_addr_for_me v2:172.21.15.2:0/0) 2022-05-13T23:14:37.258 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:37.243+0000 7f240d2d7700 1 --2- 172.21.15.2:0/1454874490 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f2408103400 0x7f2408115870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:14:37.258 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:37.243+0000 7f2407fff700 1 -- 172.21.15.2:0/1454874490 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f2408103400 msgr2=0x7f2408115870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:14:37.258 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:37.243+0000 7f2407fff700 1 --2- 172.21.15.2:0/1454874490 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f2408103400 0x7f2408115870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:14:37.259 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:37.243+0000 7f2407fff700 1 -- 172.21.15.2:0/1454874490 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f240806edb0 msgr2=0x7f240810e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:14:37.259 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:37.243+0000 7f2407fff700 1 --2- 172.21.15.2:0/1454874490 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f240806edb0 0x7f240810e2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:14:37.259 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:37.243+0000 7f2407fff700 1 -- 172.21.15.2:0/1454874490 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24000097c0 con 0x7f2408070370 2022-05-13T23:14:37.259 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:37.243+0000 7f2407fff700 1 --2- 172.21.15.2:0/1454874490 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f2408070370 0x7f240810e830 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f23f400e970 tx=0x7f23f400b040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-05-13T23:14:37.260 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:37.244+0000 7f24057fa700 1 -- 172.21.15.2:0/1454874490 <== mon.1 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23f4007b40 con 0x7f2408070370 2022-05-13T23:14:37.260 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:37.244+0000 7f24057fa700 1 -- 172.21.15.2:0/1454874490 <== mon.1 v2:172.21.15.47:3300/0 2 ==== config(26 keys) v1 ==== 1087+0+0 (secure 0 0 0) 0x7f23f400f040 con 0x7f2408070370 2022-05-13T23:14:37.260 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:37.244+0000 7f240dad8700 1 -- 172.21.15.2:0/1454874490 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f24081163c0 con 0x7f2408070370 2022-05-13T23:14:37.260 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:37.244+0000 7f24057fa700 1 -- 172.21.15.2:0/1454874490 <== mon.1 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23f4014950 con 0x7f2408070370 2022-05-13T23:14:37.261 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:37.244+0000 7f240dad8700 1 -- 172.21.15.2:0/1454874490 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2408116930 con 0x7f2408070370 2022-05-13T23:14:37.261 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:37.245+0000 7f240dad8700 1 -- 172.21.15.2:0/1454874490 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2408108ea0 con 0x7f2408070370 2022-05-13T23:14:37.261 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:37.247+0000 7f24057fa700 1 -- 172.21.15.2:0/1454874490 <== mon.1 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 16) v1 ==== 125136+0+0 (secure 0 0 0) 0x7f23f401b030 con 0x7f2408070370 2022-05-13T23:14:37.261 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:37.248+0000 7f24057fa700 1 --2- 172.21.15.2:0/1454874490 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f23f8096ce0 0x7f23f8099190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:14:37.261 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:37.248+0000 7f24057fa700 1 -- 172.21.15.2:0/1454874490 <== mon.1 v2:172.21.15.47:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 7975+0+0 (secure 0 0 0) 0x7f23f40bedd0 con 0x7f2408070370 2022-05-13T23:14:37.262 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:37.249+0000 7f240cad6700 1 --2- 172.21.15.2:0/1454874490 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f23f8096ce0 0x7f23f8099190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:14:37.262 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:37.249+0000 7f240cad6700 1 --2- 172.21.15.2:0/1454874490 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f23f8096ce0 0x7f23f8099190 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f23fc004790 tx=0x7f23fc00b040 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-05-13T23:14:37.262 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:37.250+0000 7f24057fa700 1 -- 172.21.15.2:0/1454874490 <== mon.1 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+165215 (secure 0 0 0) 0x7f23f408c110 con 0x7f2408070370 2022-05-13T23:14:37.446 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:37.443+0000 7f240dad8700 1 -- 172.21.15.2:0/1454874490 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f24081020e0 con 0x7f2408070370 2022-05-13T23:14:37.447 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:37.444+0000 7f24057fa700 1 -- 172.21.15.2:0/1454874490 <== mon.1 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v83) v1 ==== 74+0+21267 (secure 0 0 0) 0x7f23f4019090 con 0x7f2408070370 2022-05-13T23:14:37.448 INFO:teuthology.orchestra.run.smithi002.stdout: 2022-05-13T23:14:37.448 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":83,"fsid":"16cf5f88-d311-11ec-be40-001a4aab830c","created":"2022-05-13T23:06:00.765453+0000","modified":"2022-05-13T23:14:17.239435+0000","last_up_change":"2022-05-13T23:13:20.935982+0000","last_in_change":"2022-05-13T23:13:02.796000+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2022-05-13T23:09:32.000644+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2022-05-13T23:14:14.195678+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":3,"pool_name":"cephfs_data","create_time":"2022-05-13T23:14:15.659212+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"83","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"11e700e8-f3e8-4903-b52c-db354b4395bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6802","nonce":1174843704},{"type":"v1","addr":"172.21.15.2:6803","nonce":1174843704}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6804","nonce":1174843704},{"type":"v1","addr":"172.21.15.2:6805","nonce":1174843704}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6808","nonce":1174843704},{"type":"v1","addr":"172.21.15.2:6809","nonce":1174843704}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6806","nonce":1174843704},{"type":"v1","addr":"172.21.15.2:6807","nonce":1174843704}]},"public_addr":"172.21.15.2:6803/1174843704","cluster_addr":"172.21.15.2:6805/1174843704","heartbeat_back_addr":"172.21.15.2:6809/1174843704","heartbeat_front_addr":"172.21.15.2:6807/1174843704","state":["exists","up"]},{"osd":1,"uuid":"eabfa3cf-f2a3-4995-a7de-f6d8ba02ee6f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6800","nonce":3853826948},{"type":"v1","addr":"172.21.15.47:6801","nonce":3853826948}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6802","nonce":3853826948},{"type":"v1","addr":"172.21.15.47:6803","nonce":3853826948}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6806","nonce":3853826948},{"type":"v1","addr":"172.21.15.47:6807","nonce":3853826948}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6804","nonce":3853826948},{"type":"v1","addr":"172.21.15.47:6805","nonce":3853826948}]},"public_addr":"172.21.15.47:6801/3853826948","cluster_addr":"172.21.15.47:6803/3853826948","heartbeat_back_addr":"172.21.15.47:6807/3853826948","heartbeat_front_addr":"172.21.15.47:6805/3853826948","state":["exists","up"]},{"osd":2,"uuid":"153c5bd5-8200-47a7-a135-8a95b6752cd2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6800","nonce":2562256956},{"type":"v1","addr":"172.21.15.115:6801","nonce":2562256956}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6802","nonce":2562256956},{"type":"v1","addr":"172.21.15.115:6803","nonce":2562256956}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6806","nonce":2562256956},{"type":"v1","addr":"172.21.15.115:6807","nonce":2562256956}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6804","nonce":2562256956},{"type":"v1","addr":"172.21.15.115:6805","nonce":2562256956}]},"public_addr":"172.21.15.115:6801/2562256956","cluster_addr":"172.21.15.115:6803/2562256956","heartbeat_back_addr":"172.21.15.115:6807/2562256956","heartbeat_front_addr":"172.21.15.115:6805/2562256956","state":["exists","up"]},{"osd":3,"uuid":"6ca9c602-5936-48da-a861-c437e6a33765","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6810","nonce":1099909411},{"type":"v1","addr":"172.21.15.2:6811","nonce":1099909411}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6812","nonce":1099909411},{"type":"v1","addr":"172.21.15.2:6813","nonce":1099909411}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6816","nonce":1099909411},{"type":"v1","addr":"172.21.15.2:6817","nonce":1099909411}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6814","nonce":1099909411},{"type":"v1","addr":"172.21.15.2:6815","nonce":1099909411}]},"public_addr":"172.21.15.2:6811/1099909411","cluster_addr":"172.21.15.2:6813/1099909411","heartbeat_back_addr":"172.21.15.2:6817/1099909411","heartbeat_front_addr":"172.21.15.2:6815/1099909411","state":["exists","up"]},{"osd":4,"uuid":"54d943d9-2610-422a-9e59-4b8212aced31","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6808","nonce":882753861},{"type":"v1","addr":"172.21.15.47:6809","nonce":882753861}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6810","nonce":882753861},{"type":"v1","addr":"172.21.15.47:6811","nonce":882753861}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6814","nonce":882753861},{"type":"v1","addr":"172.21.15.47:6815","nonce":882753861}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6812","nonce":882753861},{"type":"v1","addr":"172.21.15.47:6813","nonce":882753861}]},"public_addr":"172.21.15.47:6809/882753861","cluster_addr":"172.21.15.47:6811/882753861","heartbeat_back_addr":"172.21.15.47:6815/882753861","heartbeat_front_addr":"172.21.15.47:6813/882753861","state":["exists","up"]},{"osd":5,"uuid":"c18566cf-5403-4057-b2ee-aa298e87e8e9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6808","nonce":2122391993},{"type":"v1","addr":"172.21.15.115:6809","nonce":2122391993}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6810","nonce":2122391993},{"type":"v1","addr":"172.21.15.115:6811","nonce":2122391993}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6814","nonce":2122391993},{"type":"v1","addr":"172.21.15.115:6815","nonce":2122391993}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6812","nonce":2122391993},{"type":"v1","addr":"172.21.15.115:6813","nonce":2122391993}]},"public_addr":"172.21.15.115:6809/2122391993","cluster_addr":"172.21.15.115:6811/2122391993","heartbeat_back_addr":"172.21.15.115:6815/2122391993","heartbeat_front_addr":"172.21.15.115:6813/2122391993","state":["exists","up"]},{"osd":6,"uuid":"38ec7b6b-30f4-4d97-ad7e-e160ae59c843","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6818","nonce":1892198790},{"type":"v1","addr":"172.21.15.2:6819","nonce":1892198790}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6820","nonce":1892198790},{"type":"v1","addr":"172.21.15.2:6821","nonce":1892198790}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6824","nonce":1892198790},{"type":"v1","addr":"172.21.15.2:6825","nonce":1892198790}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6822","nonce":1892198790},{"type":"v1","addr":"172.21.15.2:6823","nonce":1892198790}]},"public_addr":"172.21.15.2:6819/1892198790","cluster_addr":"172.21.15.2:6821/1892198790","heartbeat_back_addr":"172.21.15.2:6825/1892198790","heartbeat_front_addr":"172.21.15.2:6823/1892198790","state":["exists","up"]},{"osd":7,"uuid":"e799b560-c61e-4369-b08b-45e5a6af37c0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6816","nonce":2812542825},{"type":"v1","addr":"172.21.15.47:6817","nonce":2812542825}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6818","nonce":2812542825},{"type":"v1","addr":"172.21.15.47:6819","nonce":2812542825}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6822","nonce":2812542825},{"type":"v1","addr":"172.21.15.47:6823","nonce":2812542825}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6820","nonce":2812542825},{"type":"v1","addr":"172.21.15.47:6821","nonce":2812542825}]},"public_addr":"172.21.15.47:6817/2812542825","cluster_addr":"172.21.15.47:6819/2812542825","heartbeat_back_addr":"172.21.15.47:6823/2812542825","heartbeat_front_addr":"172.21.15.47:6821/2812542825","state":["exists","up"]},{"osd":8,"uuid":"a58f2c0e-e487-41cf-8d2e-8c937cac9ebc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":58,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6816","nonce":3959632033},{"type":"v1","addr":"172.21.15.115:6817","nonce":3959632033}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6818","nonce":3959632033},{"type":"v1","addr":"172.21.15.115:6819","nonce":3959632033}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6822","nonce":3959632033},{"type":"v1","addr":"172.21.15.115:6823","nonce":3959632033}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6820","nonce":3959632033},{"type":"v1","addr":"172.21.15.115:6821","nonce":3959632033}]},"public_addr":"172.21.15.115:6817/3959632033","cluster_addr":"172.21.15.115:6819/3959632033","heartbeat_back_addr":"172.21.15.115:6823/3959632033","heartbeat_front_addr":"172.21.15.115:6821/3959632033","state":["exists","up"]},{"osd":9,"uuid":"12cead96-f726-4f7a-ab84-eb191b778a5c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":66,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6826","nonce":3004162255},{"type":"v1","addr":"172.21.15.2:6827","nonce":3004162255}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6828","nonce":3004162255},{"type":"v1","addr":"172.21.15.2:6829","nonce":3004162255}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6832","nonce":3004162255},{"type":"v1","addr":"172.21.15.2:6833","nonce":3004162255}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6830","nonce":3004162255},{"type":"v1","addr":"172.21.15.2:6831","nonce":3004162255}]},"public_addr":"172.21.15.2:6827/3004162255","cluster_addr":"172.21.15.2:6829/3004162255","heartbeat_back_addr":"172.21.15.2:6833/3004162255","heartbeat_front_addr":"172.21.15.2:6831/3004162255","state":["exists","up"]},{"osd":10,"uuid":"4f648118-7855-48d6-86d5-44eed5295f0c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":71,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6824","nonce":3443413998},{"type":"v1","addr":"172.21.15.47:6825","nonce":3443413998}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6826","nonce":3443413998},{"type":"v1","addr":"172.21.15.47:6827","nonce":3443413998}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6830","nonce":3443413998},{"type":"v1","addr":"172.21.15.47:6831","nonce":3443413998}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6828","nonce":3443413998},{"type":"v1","addr":"172.21.15.47:6829","nonce":3443413998}]},"public_addr":"172.21.15.47:6825/3443413998","cluster_addr":"172.21.15.47:6827/3443413998","heartbeat_back_addr":"172.21.15.47:6831/3443413998","heartbeat_front_addr":"172.21.15.47:6829/3443413998","state":["exists","up"]},{"osd":11,"uuid":"916541e0-fb28-46d5-9743-801c54c5a6d1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":78,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6824","nonce":1758718209},{"type":"v1","addr":"172.21.15.115:6825","nonce":1758718209}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6826","nonce":1758718209},{"type":"v1","addr":"172.21.15.115:6827","nonce":1758718209}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6830","nonce":1758718209},{"type":"v1","addr":"172.21.15.115:6831","nonce":1758718209}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6828","nonce":1758718209},{"type":"v1","addr":"172.21.15.115:6829","nonce":1758718209}]},"public_addr":"172.21.15.115:6825/1758718209","cluster_addr":"172.21.15.115:6827/1758718209","heartbeat_back_addr":"172.21.15.115:6831/1758718209","heartbeat_front_addr":"172.21.15.115:6829/1758718209","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-05-13T23:08:43.757857+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-05-13T23:09:06.537139+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-05-13T23:09:28.524837+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-05-13T23:10:02.057583+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-05-13T23:10:26.210648+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-05-13T23:10:47.885853+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-05-13T23:11:19.141865+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-05-13T23:11:42.376065+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-05-13T23:12:04.175870+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-05-13T23:12:34.695347+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-05-13T23:12:57.762613+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-05-13T23:13:19.114284+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.2:0/477249925":"2022-05-14T23:07:04.986087+0000","172.21.15.2:6800/3226099420":"2022-05-14T23:07:04.986087+0000","172.21.15.2:0/2214093915":"2022-05-14T23:07:04.986087+0000","172.21.15.2:0/1039403355":"2022-05-14T23:07:04.986087+0000","172.21.15.2:0/22091675":"2022-05-14T23:06:36.304108+0000","172.21.15.2:6800/1255134634":"2022-05-14T23:06:36.304108+0000","172.21.15.2:6801/3226099420":"2022-05-14T23:07:04.986087+0000","172.21.15.2:0/3768677276":"2022-05-14T23:06:36.304108+0000","172.21.15.2:6801/1255134634":"2022-05-14T23:06:36.304108+0000","172.21.15.2:0/407841268":"2022-05-14T23:06:36.304108+0000"},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2022-05-13T23:14:37.450 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:37.447+0000 7f240dad8700 1 -- 172.21.15.2:0/1454874490 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f23f8096ce0 msgr2=0x7f23f8099190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:14:37.450 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:37.447+0000 7f240dad8700 1 --2- 172.21.15.2:0/1454874490 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f23f8096ce0 0x7f23f8099190 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f23fc004790 tx=0x7f23fc00b040 comp rx=0 tx=0).stop 2022-05-13T23:14:37.450 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:37.447+0000 7f240dad8700 1 -- 172.21.15.2:0/1454874490 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f2408070370 msgr2=0x7f240810e830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:14:37.451 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:37.447+0000 7f240dad8700 1 --2- 172.21.15.2:0/1454874490 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f2408070370 0x7f240810e830 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f23f400e970 tx=0x7f23f400b040 comp rx=0 tx=0).stop 2022-05-13T23:14:37.451 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:37.447+0000 7f240dad8700 1 -- 172.21.15.2:0/1454874490 shutdown_connections 2022-05-13T23:14:37.451 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:37.447+0000 7f240dad8700 1 --2- 172.21.15.2:0/1454874490 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f2408103400 0x7f2408115870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:14:37.451 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:37.447+0000 7f240dad8700 1 --2- 172.21.15.2:0/1454874490 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f23f8096ce0 0x7f23f8099190 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:14:37.452 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:37.447+0000 7f240dad8700 1 --2- 172.21.15.2:0/1454874490 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f240806edb0 0x7f240810e2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:14:37.452 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:37.447+0000 7f240dad8700 1 --2- 172.21.15.2:0/1454874490 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f2408070370 0x7f240810e830 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:14:37.452 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:37.447+0000 7f240dad8700 1 -- 172.21.15.2:0/1454874490 >> 172.21.15.2:0/1454874490 conn(0x7f24080f6ba0 msgr2=0x7f24080ff3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-05-13T23:14:37.452 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:37.447+0000 7f240dad8700 1 -- 172.21.15.2:0/1454874490 shutdown_connections 2022-05-13T23:14:37.453 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:37.448+0000 7f240dad8700 1 -- 172.21.15.2:0/1454874490 wait complete. 2022-05-13T23:14:38.116 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2022-05-13T23:14:38.117 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:23b0658391ce94ca40799469eec00c28b298816c shell --fsid 16cf5f88-d311-11ec-be40-001a4aab830c -- ceph osd dump --format=json 2022-05-13T23:14:38.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:14:38 smithi115 ceph-mon[78593]: pgmap v299: 97 pgs: 97 active+clean; 584 KiB data, 77 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 938 B/s wr, 13 op/s 2022-05-13T23:14:38.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:14:38 smithi115 ceph-mon[78593]: from='client.? 172.21.15.2:0/1454874490' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2022-05-13T23:14:38.788 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/16cf5f88-d311-11ec-be40-001a4aab830c/mon.a/config 2022-05-13T23:14:38.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:14:38 smithi047 ceph-mon[77147]: pgmap v299: 97 pgs: 97 active+clean; 584 KiB data, 77 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 938 B/s wr, 13 op/s 2022-05-13T23:14:38.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:14:38 smithi047 ceph-mon[77147]: from='client.? 172.21.15.2:0/1454874490' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2022-05-13T23:14:38.822 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:14:38 smithi002 ceph-mon[71816]: pgmap v299: 97 pgs: 97 active+clean; 584 KiB data, 77 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 938 B/s wr, 13 op/s 2022-05-13T23:14:38.822 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:14:38 smithi002 ceph-mon[71816]: from='client.? 172.21.15.2:0/1454874490' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2022-05-13T23:14:40.572 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:14:40 smithi002 ceph-mon[71816]: pgmap v300: 97 pgs: 97 active+clean; 584 KiB data, 77 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-05-13T23:14:40.602 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:40.599+0000 7fe7c6de0700 1 -- 172.21.15.2:0/416143508 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe7c006ed90 msgr2=0x7fe7c006f200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:14:40.603 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:40.599+0000 7fe7c6de0700 1 --2- 172.21.15.2:0/416143508 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe7c006ed90 0x7fe7c006f200 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fe7b0004870 tx=0x7fe7b0009cb0 comp rx=0 tx=0).stop 2022-05-13T23:14:40.603 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:40.599+0000 7fe7c6de0700 1 -- 172.21.15.2:0/416143508 shutdown_connections 2022-05-13T23:14:40.603 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:40.599+0000 7fe7c6de0700 1 --2- 172.21.15.2:0/416143508 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe7c0103420 0x7fe7c0105800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:14:40.604 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:40.599+0000 7fe7c6de0700 1 --2- 172.21.15.2:0/416143508 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe7c0070350 0x7fe7c006e850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:14:40.604 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:40.599+0000 7fe7c6de0700 1 --2- 172.21.15.2:0/416143508 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe7c006ed90 0x7fe7c006f200 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:14:40.604 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:40.599+0000 7fe7c6de0700 1 -- 172.21.15.2:0/416143508 >> 172.21.15.2:0/416143508 conn(0x7fe7c00f6ba0 msgr2=0x7fe7c00f8ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-05-13T23:14:40.604 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:40.599+0000 7fe7c6de0700 1 -- 172.21.15.2:0/416143508 shutdown_connections 2022-05-13T23:14:40.605 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:40.599+0000 7fe7c6de0700 1 -- 172.21.15.2:0/416143508 wait complete. 2022-05-13T23:14:40.605 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:40.600+0000 7fe7c6de0700 1 Processor -- start 2022-05-13T23:14:40.605 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:40.600+0000 7fe7c6de0700 1 -- start start 2022-05-13T23:14:40.607 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:40.601+0000 7fe7c6de0700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe7c006ed90 0x7fe7c010e440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:14:40.607 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:40.601+0000 7fe7c6de0700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe7c0070350 0x7fe7c010e980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:14:40.608 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:40.601+0000 7fe7c6de0700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe7c0103420 0x7fe7c0115a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:14:40.608 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:40.601+0000 7fe7c6de0700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7c0115f50 con 0x7fe7c006ed90 2022-05-13T23:14:40.608 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:40.601+0000 7fe7c6de0700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7c01160c0 con 0x7fe7c0070350 2022-05-13T23:14:40.608 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:40.601+0000 7fe7c6de0700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7c0116280 con 0x7fe7c0103420 2022-05-13T23:14:40.609 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:40.601+0000 7fe7c5dde700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe7c006ed90 0x7fe7c010e440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:14:40.609 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:40.601+0000 7fe7c55dd700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe7c0070350 0x7fe7c010e980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:14:40.609 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:40.601+0000 7fe7c5dde700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe7c006ed90 0x7fe7c010e440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:48532/0 (socket says 172.21.15.2:48532) 2022-05-13T23:14:40.609 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:40.601+0000 7fe7c5dde700 1 -- 172.21.15.2:0/3476437161 learned_addr learned my addr 172.21.15.2:0/3476437161 (peer_addr_for_me v2:172.21.15.2:0/0) 2022-05-13T23:14:40.610 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:40.601+0000 7fe7c65df700 1 --2- 172.21.15.2:0/3476437161 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe7c0103420 0x7fe7c0115a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:14:40.610 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:40.602+0000 7fe7c55dd700 1 -- 172.21.15.2:0/3476437161 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe7c0103420 msgr2=0x7fe7c0115a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:14:40.610 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:40.602+0000 7fe7c55dd700 1 --2- 172.21.15.2:0/3476437161 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe7c0103420 0x7fe7c0115a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:14:40.611 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:40.602+0000 7fe7c55dd700 1 -- 172.21.15.2:0/3476437161 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe7c006ed90 msgr2=0x7fe7c010e440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:14:40.611 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:40.602+0000 7fe7c55dd700 1 --2- 172.21.15.2:0/3476437161 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe7c006ed90 0x7fe7c010e440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:14:40.611 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:40.602+0000 7fe7c55dd700 1 -- 172.21.15.2:0/3476437161 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7b80097c0 con 0x7fe7c0070350 2022-05-13T23:14:40.614 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:40.602+0000 7fe7c5dde700 1 --2- 172.21.15.2:0/3476437161 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe7c006ed90 0x7fe7c010e440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-05-13T23:14:40.614 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:40.602+0000 7fe7c65df700 1 --2- 172.21.15.2:0/3476437161 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe7c0103420 0x7fe7c0115a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-05-13T23:14:40.614 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:40.602+0000 7fe7c55dd700 1 --2- 172.21.15.2:0/3476437161 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe7c0070350 0x7fe7c010e980 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fe7b0004710 tx=0x7fe7b0009cb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-05-13T23:14:40.615 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:40.602+0000 7fe7b67fc700 1 -- 172.21.15.2:0/3476437161 <== mon.1 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7b0020070 con 0x7fe7c0070350 2022-05-13T23:14:40.615 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:40.602+0000 7fe7b67fc700 1 -- 172.21.15.2:0/3476437161 <== mon.1 v2:172.21.15.47:3300/0 2 ==== config(26 keys) v1 ==== 1087+0+0 (secure 0 0 0) 0x7fe7b0004200 con 0x7fe7c0070350 2022-05-13T23:14:40.615 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:40.602+0000 7fe7b67fc700 1 -- 172.21.15.2:0/3476437161 <== mon.1 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7b000e040 con 0x7fe7c0070350 2022-05-13T23:14:40.615 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:40.602+0000 7fe7c6de0700 1 -- 172.21.15.2:0/3476437161 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe7b0005040 con 0x7fe7c0070350 2022-05-13T23:14:40.616 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:40.603+0000 7fe7c6de0700 1 -- 172.21.15.2:0/3476437161 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe7c01167a0 con 0x7fe7c0070350 2022-05-13T23:14:40.616 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:40.604+0000 7fe7c6de0700 1 -- 172.21.15.2:0/3476437161 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe7c0108e70 con 0x7fe7c0070350 2022-05-13T23:14:40.617 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:14:40 smithi115 ceph-mon[78593]: pgmap v300: 97 pgs: 97 active+clean; 584 KiB data, 77 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-05-13T23:14:40.618 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:40.605+0000 7fe7b67fc700 1 -- 172.21.15.2:0/3476437161 <== mon.1 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 16) v1 ==== 125136+0+0 (secure 0 0 0) 0x7fe7b0049b70 con 0x7fe7c0070350 2022-05-13T23:14:40.619 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:40.607+0000 7fe7b67fc700 1 --2- 172.21.15.2:0/3476437161 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7fe7ac096ce0 0x7fe7ac099190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:14:40.620 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:40.607+0000 7fe7b67fc700 1 -- 172.21.15.2:0/3476437161 <== mon.1 v2:172.21.15.47:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 7975+0+0 (secure 0 0 0) 0x7fe7b00bfff0 con 0x7fe7c0070350 2022-05-13T23:14:40.620 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:40.607+0000 7fe7c5dde700 1 --2- 172.21.15.2:0/3476437161 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7fe7ac096ce0 0x7fe7ac099190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:14:40.620 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:40.607+0000 7fe7c5dde700 1 --2- 172.21.15.2:0/3476437161 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7fe7ac096ce0 0x7fe7ac099190 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fe7bc007d30 tx=0x7fe7bc00b040 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-05-13T23:14:40.621 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:40.609+0000 7fe7b67fc700 1 -- 172.21.15.2:0/3476437161 <== mon.1 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+165215 (secure 0 0 0) 0x7fe7b008d370 con 0x7fe7c0070350 2022-05-13T23:14:40.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:14:40 smithi047 ceph-mon[77147]: pgmap v300: 97 pgs: 97 active+clean; 584 KiB data, 77 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-05-13T23:14:40.802 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:40.800+0000 7fe7c6de0700 1 -- 172.21.15.2:0/3476437161 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe7c01019f0 con 0x7fe7c0070350 2022-05-13T23:14:40.803 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:40.801+0000 7fe7b67fc700 1 -- 172.21.15.2:0/3476437161 <== mon.1 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v83) v1 ==== 74+0+21267 (secure 0 0 0) 0x7fe7b0090990 con 0x7fe7c0070350 2022-05-13T23:14:40.804 INFO:teuthology.orchestra.run.smithi002.stdout: 2022-05-13T23:14:40.805 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":83,"fsid":"16cf5f88-d311-11ec-be40-001a4aab830c","created":"2022-05-13T23:06:00.765453+0000","modified":"2022-05-13T23:14:17.239435+0000","last_up_change":"2022-05-13T23:13:20.935982+0000","last_in_change":"2022-05-13T23:13:02.796000+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2022-05-13T23:09:32.000644+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2022-05-13T23:14:14.195678+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":3,"pool_name":"cephfs_data","create_time":"2022-05-13T23:14:15.659212+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"83","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"11e700e8-f3e8-4903-b52c-db354b4395bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6802","nonce":1174843704},{"type":"v1","addr":"172.21.15.2:6803","nonce":1174843704}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6804","nonce":1174843704},{"type":"v1","addr":"172.21.15.2:6805","nonce":1174843704}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6808","nonce":1174843704},{"type":"v1","addr":"172.21.15.2:6809","nonce":1174843704}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6806","nonce":1174843704},{"type":"v1","addr":"172.21.15.2:6807","nonce":1174843704}]},"public_addr":"172.21.15.2:6803/1174843704","cluster_addr":"172.21.15.2:6805/1174843704","heartbeat_back_addr":"172.21.15.2:6809/1174843704","heartbeat_front_addr":"172.21.15.2:6807/1174843704","state":["exists","up"]},{"osd":1,"uuid":"eabfa3cf-f2a3-4995-a7de-f6d8ba02ee6f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6800","nonce":3853826948},{"type":"v1","addr":"172.21.15.47:6801","nonce":3853826948}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6802","nonce":3853826948},{"type":"v1","addr":"172.21.15.47:6803","nonce":3853826948}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6806","nonce":3853826948},{"type":"v1","addr":"172.21.15.47:6807","nonce":3853826948}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6804","nonce":3853826948},{"type":"v1","addr":"172.21.15.47:6805","nonce":3853826948}]},"public_addr":"172.21.15.47:6801/3853826948","cluster_addr":"172.21.15.47:6803/3853826948","heartbeat_back_addr":"172.21.15.47:6807/3853826948","heartbeat_front_addr":"172.21.15.47:6805/3853826948","state":["exists","up"]},{"osd":2,"uuid":"153c5bd5-8200-47a7-a135-8a95b6752cd2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6800","nonce":2562256956},{"type":"v1","addr":"172.21.15.115:6801","nonce":2562256956}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6802","nonce":2562256956},{"type":"v1","addr":"172.21.15.115:6803","nonce":2562256956}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6806","nonce":2562256956},{"type":"v1","addr":"172.21.15.115:6807","nonce":2562256956}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6804","nonce":2562256956},{"type":"v1","addr":"172.21.15.115:6805","nonce":2562256956}]},"public_addr":"172.21.15.115:6801/2562256956","cluster_addr":"172.21.15.115:6803/2562256956","heartbeat_back_addr":"172.21.15.115:6807/2562256956","heartbeat_front_addr":"172.21.15.115:6805/2562256956","state":["exists","up"]},{"osd":3,"uuid":"6ca9c602-5936-48da-a861-c437e6a33765","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6810","nonce":1099909411},{"type":"v1","addr":"172.21.15.2:6811","nonce":1099909411}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6812","nonce":1099909411},{"type":"v1","addr":"172.21.15.2:6813","nonce":1099909411}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6816","nonce":1099909411},{"type":"v1","addr":"172.21.15.2:6817","nonce":1099909411}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6814","nonce":1099909411},{"type":"v1","addr":"172.21.15.2:6815","nonce":1099909411}]},"public_addr":"172.21.15.2:6811/1099909411","cluster_addr":"172.21.15.2:6813/1099909411","heartbeat_back_addr":"172.21.15.2:6817/1099909411","heartbeat_front_addr":"172.21.15.2:6815/1099909411","state":["exists","up"]},{"osd":4,"uuid":"54d943d9-2610-422a-9e59-4b8212aced31","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6808","nonce":882753861},{"type":"v1","addr":"172.21.15.47:6809","nonce":882753861}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6810","nonce":882753861},{"type":"v1","addr":"172.21.15.47:6811","nonce":882753861}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6814","nonce":882753861},{"type":"v1","addr":"172.21.15.47:6815","nonce":882753861}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6812","nonce":882753861},{"type":"v1","addr":"172.21.15.47:6813","nonce":882753861}]},"public_addr":"172.21.15.47:6809/882753861","cluster_addr":"172.21.15.47:6811/882753861","heartbeat_back_addr":"172.21.15.47:6815/882753861","heartbeat_front_addr":"172.21.15.47:6813/882753861","state":["exists","up"]},{"osd":5,"uuid":"c18566cf-5403-4057-b2ee-aa298e87e8e9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6808","nonce":2122391993},{"type":"v1","addr":"172.21.15.115:6809","nonce":2122391993}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6810","nonce":2122391993},{"type":"v1","addr":"172.21.15.115:6811","nonce":2122391993}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6814","nonce":2122391993},{"type":"v1","addr":"172.21.15.115:6815","nonce":2122391993}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6812","nonce":2122391993},{"type":"v1","addr":"172.21.15.115:6813","nonce":2122391993}]},"public_addr":"172.21.15.115:6809/2122391993","cluster_addr":"172.21.15.115:6811/2122391993","heartbeat_back_addr":"172.21.15.115:6815/2122391993","heartbeat_front_addr":"172.21.15.115:6813/2122391993","state":["exists","up"]},{"osd":6,"uuid":"38ec7b6b-30f4-4d97-ad7e-e160ae59c843","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6818","nonce":1892198790},{"type":"v1","addr":"172.21.15.2:6819","nonce":1892198790}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6820","nonce":1892198790},{"type":"v1","addr":"172.21.15.2:6821","nonce":1892198790}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6824","nonce":1892198790},{"type":"v1","addr":"172.21.15.2:6825","nonce":1892198790}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6822","nonce":1892198790},{"type":"v1","addr":"172.21.15.2:6823","nonce":1892198790}]},"public_addr":"172.21.15.2:6819/1892198790","cluster_addr":"172.21.15.2:6821/1892198790","heartbeat_back_addr":"172.21.15.2:6825/1892198790","heartbeat_front_addr":"172.21.15.2:6823/1892198790","state":["exists","up"]},{"osd":7,"uuid":"e799b560-c61e-4369-b08b-45e5a6af37c0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6816","nonce":2812542825},{"type":"v1","addr":"172.21.15.47:6817","nonce":2812542825}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6818","nonce":2812542825},{"type":"v1","addr":"172.21.15.47:6819","nonce":2812542825}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6822","nonce":2812542825},{"type":"v1","addr":"172.21.15.47:6823","nonce":2812542825}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6820","nonce":2812542825},{"type":"v1","addr":"172.21.15.47:6821","nonce":2812542825}]},"public_addr":"172.21.15.47:6817/2812542825","cluster_addr":"172.21.15.47:6819/2812542825","heartbeat_back_addr":"172.21.15.47:6823/2812542825","heartbeat_front_addr":"172.21.15.47:6821/2812542825","state":["exists","up"]},{"osd":8,"uuid":"a58f2c0e-e487-41cf-8d2e-8c937cac9ebc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":58,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6816","nonce":3959632033},{"type":"v1","addr":"172.21.15.115:6817","nonce":3959632033}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6818","nonce":3959632033},{"type":"v1","addr":"172.21.15.115:6819","nonce":3959632033}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6822","nonce":3959632033},{"type":"v1","addr":"172.21.15.115:6823","nonce":3959632033}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6820","nonce":3959632033},{"type":"v1","addr":"172.21.15.115:6821","nonce":3959632033}]},"public_addr":"172.21.15.115:6817/3959632033","cluster_addr":"172.21.15.115:6819/3959632033","heartbeat_back_addr":"172.21.15.115:6823/3959632033","heartbeat_front_addr":"172.21.15.115:6821/3959632033","state":["exists","up"]},{"osd":9,"uuid":"12cead96-f726-4f7a-ab84-eb191b778a5c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":66,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6826","nonce":3004162255},{"type":"v1","addr":"172.21.15.2:6827","nonce":3004162255}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6828","nonce":3004162255},{"type":"v1","addr":"172.21.15.2:6829","nonce":3004162255}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6832","nonce":3004162255},{"type":"v1","addr":"172.21.15.2:6833","nonce":3004162255}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6830","nonce":3004162255},{"type":"v1","addr":"172.21.15.2:6831","nonce":3004162255}]},"public_addr":"172.21.15.2:6827/3004162255","cluster_addr":"172.21.15.2:6829/3004162255","heartbeat_back_addr":"172.21.15.2:6833/3004162255","heartbeat_front_addr":"172.21.15.2:6831/3004162255","state":["exists","up"]},{"osd":10,"uuid":"4f648118-7855-48d6-86d5-44eed5295f0c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":71,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6824","nonce":3443413998},{"type":"v1","addr":"172.21.15.47:6825","nonce":3443413998}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6826","nonce":3443413998},{"type":"v1","addr":"172.21.15.47:6827","nonce":3443413998}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6830","nonce":3443413998},{"type":"v1","addr":"172.21.15.47:6831","nonce":3443413998}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6828","nonce":3443413998},{"type":"v1","addr":"172.21.15.47:6829","nonce":3443413998}]},"public_addr":"172.21.15.47:6825/3443413998","cluster_addr":"172.21.15.47:6827/3443413998","heartbeat_back_addr":"172.21.15.47:6831/3443413998","heartbeat_front_addr":"172.21.15.47:6829/3443413998","state":["exists","up"]},{"osd":11,"uuid":"916541e0-fb28-46d5-9743-801c54c5a6d1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":78,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6824","nonce":1758718209},{"type":"v1","addr":"172.21.15.115:6825","nonce":1758718209}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6826","nonce":1758718209},{"type":"v1","addr":"172.21.15.115:6827","nonce":1758718209}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6830","nonce":1758718209},{"type":"v1","addr":"172.21.15.115:6831","nonce":1758718209}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6828","nonce":1758718209},{"type":"v1","addr":"172.21.15.115:6829","nonce":1758718209}]},"public_addr":"172.21.15.115:6825/1758718209","cluster_addr":"172.21.15.115:6827/1758718209","heartbeat_back_addr":"172.21.15.115:6831/1758718209","heartbeat_front_addr":"172.21.15.115:6829/1758718209","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-05-13T23:08:43.757857+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-05-13T23:09:06.537139+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-05-13T23:09:28.524837+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-05-13T23:10:02.057583+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-05-13T23:10:26.210648+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-05-13T23:10:47.885853+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-05-13T23:11:19.141865+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-05-13T23:11:42.376065+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-05-13T23:12:04.175870+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-05-13T23:12:34.695347+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-05-13T23:12:57.762613+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-05-13T23:13:19.114284+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.2:0/477249925":"2022-05-14T23:07:04.986087+0000","172.21.15.2:6800/3226099420":"2022-05-14T23:07:04.986087+0000","172.21.15.2:0/2214093915":"2022-05-14T23:07:04.986087+0000","172.21.15.2:0/1039403355":"2022-05-14T23:07:04.986087+0000","172.21.15.2:0/22091675":"2022-05-14T23:06:36.304108+0000","172.21.15.2:6800/1255134634":"2022-05-14T23:06:36.304108+0000","172.21.15.2:6801/3226099420":"2022-05-14T23:07:04.986087+0000","172.21.15.2:0/3768677276":"2022-05-14T23:06:36.304108+0000","172.21.15.2:6801/1255134634":"2022-05-14T23:06:36.304108+0000","172.21.15.2:0/407841268":"2022-05-14T23:06:36.304108+0000"},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2022-05-13T23:14:40.807 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:40.804+0000 7fe7c6de0700 1 -- 172.21.15.2:0/3476437161 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7fe7ac096ce0 msgr2=0x7fe7ac099190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:14:40.807 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:40.804+0000 7fe7c6de0700 1 --2- 172.21.15.2:0/3476437161 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7fe7ac096ce0 0x7fe7ac099190 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fe7bc007d30 tx=0x7fe7bc00b040 comp rx=0 tx=0).stop 2022-05-13T23:14:40.808 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:40.804+0000 7fe7c6de0700 1 -- 172.21.15.2:0/3476437161 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe7c0070350 msgr2=0x7fe7c010e980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:14:40.808 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:40.804+0000 7fe7c6de0700 1 --2- 172.21.15.2:0/3476437161 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe7c0070350 0x7fe7c010e980 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fe7b0004710 tx=0x7fe7b0009cb0 comp rx=0 tx=0).stop 2022-05-13T23:14:40.808 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:40.804+0000 7fe7c6de0700 1 -- 172.21.15.2:0/3476437161 shutdown_connections 2022-05-13T23:14:40.808 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:40.804+0000 7fe7c6de0700 1 --2- 172.21.15.2:0/3476437161 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe7c0103420 0x7fe7c0115a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:14:40.809 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:40.804+0000 7fe7c6de0700 1 --2- 172.21.15.2:0/3476437161 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7fe7ac096ce0 0x7fe7ac099190 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:14:40.809 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:40.804+0000 7fe7c6de0700 1 --2- 172.21.15.2:0/3476437161 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe7c006ed90 0x7fe7c010e440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:14:40.809 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:40.804+0000 7fe7c6de0700 1 --2- 172.21.15.2:0/3476437161 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe7c0070350 0x7fe7c010e980 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:14:40.809 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:40.804+0000 7fe7c6de0700 1 -- 172.21.15.2:0/3476437161 >> 172.21.15.2:0/3476437161 conn(0x7fe7c00f6ba0 msgr2=0x7fe7c00f8f80 unknown :-1 s=STATE_NONE l=0).mark_down 2022-05-13T23:14:40.810 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:40.804+0000 7fe7c6de0700 1 -- 172.21.15.2:0/3476437161 shutdown_connections 2022-05-13T23:14:40.810 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:40.804+0000 7fe7c6de0700 1 -- 172.21.15.2:0/3476437161 wait complete. 2022-05-13T23:14:41.347 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:23b0658391ce94ca40799469eec00c28b298816c shell --fsid 16cf5f88-d311-11ec-be40-001a4aab830c -- ceph tell osd.0 flush_pg_stats 2022-05-13T23:14:41.348 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:23b0658391ce94ca40799469eec00c28b298816c shell --fsid 16cf5f88-d311-11ec-be40-001a4aab830c -- ceph tell osd.1 flush_pg_stats 2022-05-13T23:14:41.349 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:23b0658391ce94ca40799469eec00c28b298816c shell --fsid 16cf5f88-d311-11ec-be40-001a4aab830c -- ceph tell osd.2 flush_pg_stats 2022-05-13T23:14:41.350 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:23b0658391ce94ca40799469eec00c28b298816c shell --fsid 16cf5f88-d311-11ec-be40-001a4aab830c -- ceph tell osd.3 flush_pg_stats 2022-05-13T23:14:41.350 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:23b0658391ce94ca40799469eec00c28b298816c shell --fsid 16cf5f88-d311-11ec-be40-001a4aab830c -- ceph tell osd.4 flush_pg_stats 2022-05-13T23:14:41.351 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:23b0658391ce94ca40799469eec00c28b298816c shell --fsid 16cf5f88-d311-11ec-be40-001a4aab830c -- ceph tell osd.5 flush_pg_stats 2022-05-13T23:14:41.352 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:23b0658391ce94ca40799469eec00c28b298816c shell --fsid 16cf5f88-d311-11ec-be40-001a4aab830c -- ceph tell osd.6 flush_pg_stats 2022-05-13T23:14:41.352 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:23b0658391ce94ca40799469eec00c28b298816c shell --fsid 16cf5f88-d311-11ec-be40-001a4aab830c -- ceph tell osd.7 flush_pg_stats 2022-05-13T23:14:41.354 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:23b0658391ce94ca40799469eec00c28b298816c shell --fsid 16cf5f88-d311-11ec-be40-001a4aab830c -- ceph tell osd.8 flush_pg_stats 2022-05-13T23:14:41.355 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:23b0658391ce94ca40799469eec00c28b298816c shell --fsid 16cf5f88-d311-11ec-be40-001a4aab830c -- ceph tell osd.9 flush_pg_stats 2022-05-13T23:14:41.355 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:23b0658391ce94ca40799469eec00c28b298816c shell --fsid 16cf5f88-d311-11ec-be40-001a4aab830c -- ceph tell osd.10 flush_pg_stats 2022-05-13T23:14:41.356 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:23b0658391ce94ca40799469eec00c28b298816c shell --fsid 16cf5f88-d311-11ec-be40-001a4aab830c -- ceph tell osd.11 flush_pg_stats 2022-05-13T23:14:41.505 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:14:41 smithi002 ceph-mon[71816]: from='client.? 172.21.15.2:0/3476437161' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2022-05-13T23:14:41.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:14:41 smithi115 ceph-mon[78593]: from='client.? 172.21.15.2:0/3476437161' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2022-05-13T23:14:41.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:14:41 smithi047 ceph-mon[77147]: from='client.? 172.21.15.2:0/3476437161' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2022-05-13T23:14:42.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:14:42 smithi115 ceph-mon[78593]: pgmap v301: 97 pgs: 97 active+clean; 584 KiB data, 77 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2022-05-13T23:14:42.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:14:42 smithi047 ceph-mon[77147]: pgmap v301: 97 pgs: 97 active+clean; 584 KiB data, 77 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2022-05-13T23:14:42.823 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:14:42 smithi002 ceph-mon[71816]: pgmap v301: 97 pgs: 97 active+clean; 584 KiB data, 77 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2022-05-13T23:14:42.979 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/16cf5f88-d311-11ec-be40-001a4aab830c/mon.a/config 2022-05-13T23:14:43.071 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/16cf5f88-d311-11ec-be40-001a4aab830c/mon.a/config 2022-05-13T23:14:43.127 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/16cf5f88-d311-11ec-be40-001a4aab830c/mon.a/config 2022-05-13T23:14:43.230 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/16cf5f88-d311-11ec-be40-001a4aab830c/mon.a/config 2022-05-13T23:14:43.262 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/16cf5f88-d311-11ec-be40-001a4aab830c/mon.a/config 2022-05-13T23:14:43.323 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/16cf5f88-d311-11ec-be40-001a4aab830c/mon.a/config 2022-05-13T23:14:43.530 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/16cf5f88-d311-11ec-be40-001a4aab830c/mon.a/config 2022-05-13T23:14:43.806 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/16cf5f88-d311-11ec-be40-001a4aab830c/mon.a/config 2022-05-13T23:14:44.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:14:44 smithi115 ceph-mon[78593]: pgmap v302: 97 pgs: 97 active+clean; 584 KiB data, 77 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2022-05-13T23:14:44.666 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:14:44 smithi002 ceph-mon[71816]: pgmap v302: 97 pgs: 97 active+clean; 584 KiB data, 77 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2022-05-13T23:14:44.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:14:44 smithi047 ceph-mon[77147]: pgmap v302: 97 pgs: 97 active+clean; 584 KiB data, 77 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2022-05-13T23:14:46.575 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:14:46 smithi002 ceph-mon[71816]: pgmap v303: 97 pgs: 97 active+clean; 584 KiB data, 77 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 11 op/s 2022-05-13T23:14:46.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:14:46 smithi047 ceph-mon[77147]: pgmap v303: 97 pgs: 97 active+clean; 584 KiB data, 77 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 11 op/s 2022-05-13T23:14:46.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:14:46 smithi115 ceph-mon[78593]: pgmap v303: 97 pgs: 97 active+clean; 584 KiB data, 77 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 11 op/s 2022-05-13T23:14:47.275 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/16cf5f88-d311-11ec-be40-001a4aab830c/mon.a/config 2022-05-13T23:14:47.970 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:47.872+0000 7f7f64951700 1 -- 172.21.15.2:0/1443644119 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7f60103580 msgr2=0x7f7f60105960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:14:47.970 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:47.872+0000 7f7f64951700 1 --2- 172.21.15.2:0/1443644119 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7f60103580 0x7f7f60105960 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f7f58009be0 tx=0x7f7f58004d10 comp rx=0 tx=0).stop 2022-05-13T23:14:47.971 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:47.872+0000 7f7f64951700 1 -- 172.21.15.2:0/1443644119 shutdown_connections 2022-05-13T23:14:47.971 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:47.872+0000 7f7f64951700 1 --2- 172.21.15.2:0/1443644119 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7f60103580 0x7f7f60105960 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:14:47.971 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:47.872+0000 7f7f64951700 1 --2- 172.21.15.2:0/1443644119 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7f60070370 0x7f7f6006e870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:14:47.972 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:47.872+0000 7f7f64951700 1 --2- 172.21.15.2:0/1443644119 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f7f6006edb0 0x7f7f6006f220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:14:47.972 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:47.872+0000 7f7f64951700 1 -- 172.21.15.2:0/1443644119 >> 172.21.15.2:0/1443644119 conn(0x7f7f600f6e00 msgr2=0x7f7f600f9210 unknown :-1 s=STATE_NONE l=0).mark_down 2022-05-13T23:14:47.973 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:47.872+0000 7f7f64951700 1 -- 172.21.15.2:0/1443644119 shutdown_connections 2022-05-13T23:14:47.973 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:47.872+0000 7f7f64951700 1 -- 172.21.15.2:0/1443644119 wait complete. 2022-05-13T23:14:47.973 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:47.873+0000 7f7f64951700 1 Processor -- start 2022-05-13T23:14:47.974 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:47.873+0000 7f7f64951700 1 -- start start 2022-05-13T23:14:47.974 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:47.874+0000 7f7f64951700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7f6006edb0 0x7f7f6010e4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:14:47.975 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:47.874+0000 7f7f64951700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7f60070370 0x7f7f6010ea00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:14:47.975 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:47.874+0000 7f7f64951700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f7f60103580 0x7f7f60115a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:14:47.975 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:47.874+0000 7f7f64951700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f60115fd0 con 0x7f7f60070370 2022-05-13T23:14:47.976 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:47.874+0000 7f7f64951700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f60116140 con 0x7f7f6006edb0 2022-05-13T23:14:47.976 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:47.874+0000 7f7f64951700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f60116300 con 0x7f7f60103580 2022-05-13T23:14:47.977 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:47.874+0000 7f7f5e59c700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7f6006edb0 0x7f7f6010e4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:14:47.977 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:47.874+0000 7f7f55d9b700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7f60070370 0x7f7f6010ea00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:14:47.977 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:47.874+0000 7f7f5e59c700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7f6006edb0 0x7f7f6010e4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.2:55040/0 (socket says 172.21.15.2:55040) 2022-05-13T23:14:47.978 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:47.874+0000 7f7f5e59c700 1 -- 172.21.15.2:0/821890608 learned_addr learned my addr 172.21.15.2:0/821890608 (peer_addr_for_me v2:172.21.15.2:0/0) 2022-05-13T23:14:47.978 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:47.874+0000 7f7f5ed9d700 1 --2- 172.21.15.2:0/821890608 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f7f60103580 0x7f7f60115a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:14:47.979 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:47.874+0000 7f7f5e59c700 1 -- 172.21.15.2:0/821890608 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f7f60103580 msgr2=0x7f7f60115a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:14:47.979 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:47.874+0000 7f7f5e59c700 1 --2- 172.21.15.2:0/821890608 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f7f60103580 0x7f7f60115a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:14:47.979 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:47.874+0000 7f7f5e59c700 1 -- 172.21.15.2:0/821890608 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7f60070370 msgr2=0x7f7f6010ea00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:14:47.979 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:47.874+0000 7f7f5e59c700 1 --2- 172.21.15.2:0/821890608 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7f60070370 0x7f7f6010ea00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:14:47.980 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:47.875+0000 7f7f5e59c700 1 -- 172.21.15.2:0/821890608 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f580097c0 con 0x7f7f6006edb0 2022-05-13T23:14:47.980 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:47.875+0000 7f7f5e59c700 1 --2- 172.21.15.2:0/821890608 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7f6006edb0 0x7f7f6010e4c0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f7f5000e970 tx=0x7f7f5000b040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-05-13T23:14:47.980 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:47.875+0000 7f7f577fe700 1 -- 172.21.15.2:0/821890608 <== mon.1 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f50007b40 con 0x7f7f6006edb0 2022-05-13T23:14:47.981 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:47.875+0000 7f7f577fe700 1 -- 172.21.15.2:0/821890608 <== mon.1 v2:172.21.15.47:3300/0 2 ==== config(26 keys) v1 ==== 1087+0+0 (secure 0 0 0) 0x7f7f5000f040 con 0x7f7f6006edb0 2022-05-13T23:14:47.981 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:47.875+0000 7f7f64951700 1 -- 172.21.15.2:0/821890608 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7f601165e0 con 0x7f7f6006edb0 2022-05-13T23:14:47.982 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:47.875+0000 7f7f577fe700 1 -- 172.21.15.2:0/821890608 <== mon.1 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f50014950 con 0x7f7f6006edb0 2022-05-13T23:14:47.982 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:47.875+0000 7f7f64951700 1 -- 172.21.15.2:0/821890608 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7f60116b50 con 0x7f7f6006edb0 2022-05-13T23:14:47.982 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:47.876+0000 7f7f64951700 1 -- 172.21.15.2:0/821890608 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f7f600627a0 con 0x7f7f6006edb0 2022-05-13T23:14:47.982 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:47.879+0000 7f7f577fe700 1 -- 172.21.15.2:0/821890608 <== mon.1 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 16) v1 ==== 125136+0+0 (secure 0 0 0) 0x7f7f5001b030 con 0x7f7f6006edb0 2022-05-13T23:14:47.983 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:47.880+0000 7f7f577fe700 1 --2- 172.21.15.2:0/821890608 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f7f4c096ce0 0x7f7f4c099190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:14:47.983 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:47.880+0000 7f7f577fe700 1 -- 172.21.15.2:0/821890608 <== mon.1 v2:172.21.15.47:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 7975+0+0 (secure 0 0 0) 0x7f7f500bece0 con 0x7f7f6006edb0 2022-05-13T23:14:47.983 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:47.881+0000 7f7f55d9b700 1 --2- 172.21.15.2:0/821890608 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f7f4c096ce0 0x7f7f4c099190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:14:47.983 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:47.881+0000 7f7f577fe700 1 --2- 172.21.15.2:0/821890608 >> [v2:172.21.15.115:6800/2562256956,v1:172.21.15.115:6801/2562256956] conn(0x7f7f4c0a04b0 0x7f7f4c0a28c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:14:47.983 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:47.881+0000 7f7f577fe700 1 -- 172.21.15.2:0/821890608 --> [v2:172.21.15.115:6800/2562256956,v1:172.21.15.115:6801/2562256956] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f7f4c0a2f70 con 0x7f7f4c0a04b0 2022-05-13T23:14:47.984 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:47.881+0000 7f7f577fe700 1 -- 172.21.15.2:0/821890608 <== mon.1 v2:172.21.15.47:3300/0 6 ==== mon_get_version_reply(handle=1 version=83) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7f500befd0 con 0x7f7f6006edb0 2022-05-13T23:14:47.984 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:47.881+0000 7f7f5ed9d700 1 --2- 172.21.15.2:0/821890608 >> [v2:172.21.15.115:6800/2562256956,v1:172.21.15.115:6801/2562256956] conn(0x7f7f4c0a04b0 0x7f7f4c0a28c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:14:47.984 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:47.881+0000 7f7f55d9b700 1 --2- 172.21.15.2:0/821890608 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f7f4c096ce0 0x7f7f4c099190 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f7f480048b0 tx=0x7f7f4800b040 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-05-13T23:14:47.984 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:47.882+0000 7f7f5ed9d700 1 --2- 172.21.15.2:0/821890608 >> [v2:172.21.15.115:6800/2562256956,v1:172.21.15.115:6801/2562256956] conn(0x7f7f4c0a04b0 0x7f7f4c0a28c0 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-05-13T23:14:47.985 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:47.883+0000 7f7f577fe700 1 -- 172.21.15.2:0/821890608 <== osd.2 v2:172.21.15.115:6800/2562256956 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23747 (crc 0 0 0) 0x7f7f4c0a2f70 con 0x7f7f4c0a04b0 2022-05-13T23:14:47.985 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:47.909+0000 7f7f64951700 1 -- 172.21.15.2:0/821890608 --> [v2:172.21.15.115:6800/2562256956,v1:172.21.15.115:6801/2562256956] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f7f6005cba0 con 0x7f7f4c0a04b0 2022-05-13T23:14:47.985 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:47.909+0000 7f7f577fe700 1 -- 172.21.15.2:0/821890608 <== osd.2 v2:172.21.15.115:6800/2562256956 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f7f6005cba0 con 0x7f7f4c0a04b0 2022-05-13T23:14:47.986 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:47.910+0000 7f7f64951700 1 -- 172.21.15.2:0/821890608 >> [v2:172.21.15.115:6800/2562256956,v1:172.21.15.115:6801/2562256956] conn(0x7f7f4c0a04b0 msgr2=0x7f7f4c0a28c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:14:47.986 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:47.910+0000 7f7f64951700 1 --2- 172.21.15.2:0/821890608 >> [v2:172.21.15.115:6800/2562256956,v1:172.21.15.115:6801/2562256956] conn(0x7f7f4c0a04b0 0x7f7f4c0a28c0 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:14:47.986 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:47.910+0000 7f7f64951700 1 -- 172.21.15.2:0/821890608 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f7f4c096ce0 msgr2=0x7f7f4c099190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:14:47.987 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:47.910+0000 7f7f64951700 1 --2- 172.21.15.2:0/821890608 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f7f4c096ce0 0x7f7f4c099190 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f7f480048b0 tx=0x7f7f4800b040 comp rx=0 tx=0).stop 2022-05-13T23:14:47.987 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:47.910+0000 7f7f64951700 1 -- 172.21.15.2:0/821890608 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7f6006edb0 msgr2=0x7f7f6010e4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:14:47.987 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:47.910+0000 7f7f64951700 1 --2- 172.21.15.2:0/821890608 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7f6006edb0 0x7f7f6010e4c0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f7f5000e970 tx=0x7f7f5000b040 comp rx=0 tx=0).stop 2022-05-13T23:14:47.987 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:47.910+0000 7f7f5ed9d700 1 -- 172.21.15.2:0/821890608 reap_dead start 2022-05-13T23:14:47.988 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:47.910+0000 7f7f64951700 1 -- 172.21.15.2:0/821890608 shutdown_connections 2022-05-13T23:14:47.988 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:47.910+0000 7f7f64951700 1 -- 172.21.15.2:0/821890608 >> 172.21.15.2:0/821890608 conn(0x7f7f600f6e00 msgr2=0x7f7f601056e0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-05-13T23:14:47.988 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:47.910+0000 7f7f64951700 1 -- 172.21.15.2:0/821890608 shutdown_connections 2022-05-13T23:14:47.988 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:47.910+0000 7f7f64951700 1 -- 172.21.15.2:0/821890608 wait complete. 2022-05-13T23:14:48.418 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/16cf5f88-d311-11ec-be40-001a4aab830c/mon.a/config 2022-05-13T23:14:48.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:14:48 smithi047 ceph-mon[77147]: pgmap v304: 97 pgs: 97 active+clean; 584 KiB data, 77 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2022-05-13T23:14:48.795 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:14:48 smithi002 ceph-mon[71816]: pgmap v304: 97 pgs: 97 active+clean; 584 KiB data, 77 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2022-05-13T23:14:48.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:14:48 smithi115 ceph-mon[78593]: pgmap v304: 97 pgs: 97 active+clean; 584 KiB data, 77 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2022-05-13T23:14:49.322 INFO:teuthology.orchestra.run.smithi002.stdout:77309411393 2022-05-13T23:14:49.323 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:23b0658391ce94ca40799469eec00c28b298816c shell --fsid 16cf5f88-d311-11ec-be40-001a4aab830c -- ceph osd last-stat-seq osd.2 2022-05-13T23:14:49.427 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:49.048+0000 7fd0a62f2700 1 -- 172.21.15.2:0/2420609321 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fd0a00fb5f0 msgr2=0x7fd0a00fba00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:14:49.427 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:49.048+0000 7fd0a62f2700 1 --2- 172.21.15.2:0/2420609321 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fd0a00fb5f0 0x7fd0a00fba00 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fd0900047b0 tx=0x7fd090009cb0 comp rx=0 tx=0).stop 2022-05-13T23:14:49.427 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:49.048+0000 7fd0a62f2700 1 -- 172.21.15.2:0/2420609321 shutdown_connections 2022-05-13T23:14:49.427 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:49.048+0000 7fd0a62f2700 1 --2- 172.21.15.2:0/2420609321 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd0a00fd220 0x7fd0a0103ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:14:49.428 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:49.048+0000 7fd0a62f2700 1 --2- 172.21.15.2:0/2420609321 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fd0a00fb5f0 0x7fd0a00fba00 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:14:49.428 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:49.048+0000 7fd0a62f2700 1 --2- 172.21.15.2:0/2420609321 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd0a00fc890 0x7fd0a00fcce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:14:49.428 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:49.048+0000 7fd0a62f2700 1 -- 172.21.15.2:0/2420609321 >> 172.21.15.2:0/2420609321 conn(0x7fd0a00f6e00 msgr2=0x7fd0a00f9210 unknown :-1 s=STATE_NONE l=0).mark_down 2022-05-13T23:14:49.428 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:49.048+0000 7fd0a62f2700 1 -- 172.21.15.2:0/2420609321 shutdown_connections 2022-05-13T23:14:49.429 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:49.048+0000 7fd0a62f2700 1 -- 172.21.15.2:0/2420609321 wait complete. 2022-05-13T23:14:49.429 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:49.049+0000 7fd0a62f2700 1 Processor -- start 2022-05-13T23:14:49.429 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:49.049+0000 7fd0a62f2700 1 -- start start 2022-05-13T23:14:49.429 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:49.049+0000 7fd0a62f2700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fd0a00fb5f0 0x7fd0a0112a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:14:49.429 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:49.049+0000 7fd0a62f2700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd0a00fc890 0x7fd0a0112f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:14:49.430 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:49.049+0000 7fd0a62f2700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd0a00fd220 0x7fd0a011a000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:14:49.430 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:49.050+0000 7fd0a62f2700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0a011a540 con 0x7fd0a00fc890 2022-05-13T23:14:49.430 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:49.050+0000 7fd0a62f2700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0a011a6b0 con 0x7fd0a00fd220 2022-05-13T23:14:49.430 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:49.050+0000 7fd0a62f2700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0a011a870 con 0x7fd0a00fb5f0 2022-05-13T23:14:49.430 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:49.050+0000 7fd09f7fe700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd0a00fc890 0x7fd0a0112f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:14:49.431 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:49.050+0000 7fd09ffff700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fd0a00fb5f0 0x7fd0a0112a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:14:49.431 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:49.050+0000 7fd09f7fe700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd0a00fc890 0x7fd0a0112f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:48566/0 (socket says 172.21.15.2:48566) 2022-05-13T23:14:49.431 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:49.050+0000 7fd09f7fe700 1 -- 172.21.15.2:0/2843335143 learned_addr learned my addr 172.21.15.2:0/2843335143 (peer_addr_for_me v2:172.21.15.2:0/0) 2022-05-13T23:14:49.431 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:49.050+0000 7fd0a488f700 1 --2- 172.21.15.2:0/2843335143 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd0a00fd220 0x7fd0a011a000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:14:49.432 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:49.050+0000 7fd09ffff700 1 -- 172.21.15.2:0/2843335143 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd0a00fd220 msgr2=0x7fd0a011a000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:14:49.432 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:49.050+0000 7fd09ffff700 1 --2- 172.21.15.2:0/2843335143 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd0a00fd220 0x7fd0a011a000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:14:49.433 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:49.050+0000 7fd09ffff700 1 -- 172.21.15.2:0/2843335143 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd0a00fc890 msgr2=0x7fd0a0112f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:14:49.433 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:49.050+0000 7fd09ffff700 1 --2- 172.21.15.2:0/2843335143 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd0a00fc890 0x7fd0a0112f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:14:49.434 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:49.050+0000 7fd09ffff700 1 -- 172.21.15.2:0/2843335143 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd090005040 con 0x7fd0a00fb5f0 2022-05-13T23:14:49.434 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:49.050+0000 7fd09f7fe700 1 --2- 172.21.15.2:0/2843335143 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd0a00fc890 0x7fd0a0112f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-05-13T23:14:49.435 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:49.051+0000 7fd09ffff700 1 --2- 172.21.15.2:0/2843335143 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fd0a00fb5f0 0x7fd0a0112a30 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fd090005010 tx=0x7fd090009cb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-05-13T23:14:49.435 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:49.051+0000 7fd09cff9700 1 -- 172.21.15.2:0/2843335143 <== mon.2 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd090020070 con 0x7fd0a00fb5f0 2022-05-13T23:14:49.435 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:49.051+0000 7fd0a62f2700 1 -- 172.21.15.2:0/2843335143 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd0a011aaf0 con 0x7fd0a00fb5f0 2022-05-13T23:14:49.436 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:49.051+0000 7fd09cff9700 1 -- 172.21.15.2:0/2843335143 <== mon.2 v2:172.21.15.115:3300/0 2 ==== config(26 keys) v1 ==== 1087+0+0 (secure 0 0 0) 0x7fd090000ce0 con 0x7fd0a00fb5f0 2022-05-13T23:14:49.436 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:49.051+0000 7fd09cff9700 1 -- 172.21.15.2:0/2843335143 <== mon.2 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd09001cbf0 con 0x7fd0a00fb5f0 2022-05-13T23:14:49.436 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:49.051+0000 7fd0a62f2700 1 -- 172.21.15.2:0/2843335143 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd0a011af80 con 0x7fd0a00fb5f0 2022-05-13T23:14:49.437 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:49.052+0000 7fd0a62f2700 1 -- 172.21.15.2:0/2843335143 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fd0a00627a0 con 0x7fd0a00fb5f0 2022-05-13T23:14:49.437 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:49.055+0000 7fd09cff9700 1 -- 172.21.15.2:0/2843335143 <== mon.2 v2:172.21.15.115:3300/0 4 ==== mgrmap(e 16) v1 ==== 125136+0+0 (secure 0 0 0) 0x7fd090009240 con 0x7fd0a00fb5f0 2022-05-13T23:14:49.438 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:49.056+0000 7fd09cff9700 1 --2- 172.21.15.2:0/2843335143 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7fd088096ce0 0x7fd088099190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:14:49.438 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:49.056+0000 7fd09cff9700 1 -- 172.21.15.2:0/2843335143 <== mon.2 v2:172.21.15.115:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 7975+0+0 (secure 0 0 0) 0x7fd0900c0cf0 con 0x7fd0a00fb5f0 2022-05-13T23:14:49.438 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:49.056+0000 7fd09cff9700 1 --2- 172.21.15.2:0/2843335143 >> [v2:172.21.15.47:6800/3853826948,v1:172.21.15.47:6801/3853826948] conn(0x7fd0880a04b0 0x7fd0880a28c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:14:49.439 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:49.057+0000 7fd09f7fe700 1 --2- 172.21.15.2:0/2843335143 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7fd088096ce0 0x7fd088099190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:14:49.440 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:49.057+0000 7fd09cff9700 1 -- 172.21.15.2:0/2843335143 --> [v2:172.21.15.47:6800/3853826948,v1:172.21.15.47:6801/3853826948] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd0880a2f70 con 0x7fd0880a04b0 2022-05-13T23:14:49.440 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:49.057+0000 7fd09cff9700 1 -- 172.21.15.2:0/2843335143 <== mon.2 v2:172.21.15.115:3300/0 6 ==== mon_get_version_reply(handle=1 version=83) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd0900c0fe0 con 0x7fd0a00fb5f0 2022-05-13T23:14:49.440 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:49.057+0000 7fd0a488f700 1 --2- 172.21.15.2:0/2843335143 >> [v2:172.21.15.47:6800/3853826948,v1:172.21.15.47:6801/3853826948] conn(0x7fd0880a04b0 0x7fd0880a28c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:14:49.440 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:49.057+0000 7fd09f7fe700 1 --2- 172.21.15.2:0/2843335143 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7fd088096ce0 0x7fd088099190 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fd094007cb0 tx=0x7fd09400b040 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-05-13T23:14:49.441 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:49.058+0000 7fd0a488f700 1 --2- 172.21.15.2:0/2843335143 >> [v2:172.21.15.47:6800/3853826948,v1:172.21.15.47:6801/3853826948] conn(0x7fd0880a04b0 0x7fd0880a28c0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-05-13T23:14:49.441 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:49.058+0000 7fd09cff9700 1 -- 172.21.15.2:0/2843335143 <== osd.1 v2:172.21.15.47:6800/3853826948 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23747 (crc 0 0 0) 0x7fd0880a2f70 con 0x7fd0880a04b0 2022-05-13T23:14:49.441 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:49.080+0000 7fd0a62f2700 1 -- 172.21.15.2:0/2843335143 --> [v2:172.21.15.47:6800/3853826948,v1:172.21.15.47:6801/3853826948] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fd0a005cb60 con 0x7fd0880a04b0 2022-05-13T23:14:49.441 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:49.081+0000 7fd09cff9700 1 -- 172.21.15.2:0/2843335143 <== osd.1 v2:172.21.15.47:6800/3853826948 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fd0a005cb60 con 0x7fd0880a04b0 2022-05-13T23:14:49.442 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:49.081+0000 7fd0a62f2700 1 -- 172.21.15.2:0/2843335143 >> [v2:172.21.15.47:6800/3853826948,v1:172.21.15.47:6801/3853826948] conn(0x7fd0880a04b0 msgr2=0x7fd0880a28c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:14:49.442 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:49.081+0000 7fd0a62f2700 1 --2- 172.21.15.2:0/2843335143 >> [v2:172.21.15.47:6800/3853826948,v1:172.21.15.47:6801/3853826948] conn(0x7fd0880a04b0 0x7fd0880a28c0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:14:49.442 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:49.082+0000 7fd0a62f2700 1 -- 172.21.15.2:0/2843335143 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7fd088096ce0 msgr2=0x7fd088099190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:14:49.443 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:49.082+0000 7fd0a62f2700 1 --2- 172.21.15.2:0/2843335143 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7fd088096ce0 0x7fd088099190 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fd094007cb0 tx=0x7fd09400b040 comp rx=0 tx=0).stop 2022-05-13T23:14:49.443 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:49.082+0000 7fd0a62f2700 1 -- 172.21.15.2:0/2843335143 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fd0a00fb5f0 msgr2=0x7fd0a0112a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:14:49.443 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:49.082+0000 7fd0a62f2700 1 --2- 172.21.15.2:0/2843335143 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fd0a00fb5f0 0x7fd0a0112a30 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fd090005010 tx=0x7fd090009cb0 comp rx=0 tx=0).stop 2022-05-13T23:14:49.444 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:49.082+0000 7fd0a488f700 1 -- 172.21.15.2:0/2843335143 reap_dead start 2022-05-13T23:14:49.444 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:49.082+0000 7fd0a62f2700 1 -- 172.21.15.2:0/2843335143 shutdown_connections 2022-05-13T23:14:49.444 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:49.082+0000 7fd0a62f2700 1 -- 172.21.15.2:0/2843335143 >> 172.21.15.2:0/2843335143 conn(0x7fd0a00f6e00 msgr2=0x7fd0a00fdab0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-05-13T23:14:49.444 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:49.082+0000 7fd0a62f2700 1 -- 172.21.15.2:0/2843335143 shutdown_connections 2022-05-13T23:14:49.445 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:49.082+0000 7fd0a62f2700 1 -- 172.21.15.2:0/2843335143 wait complete. 2022-05-13T23:14:49.542 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/16cf5f88-d311-11ec-be40-001a4aab830c/mon.a/config 2022-05-13T23:14:49.906 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/16cf5f88-d311-11ec-be40-001a4aab830c/mon.a/config 2022-05-13T23:14:50.747 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:14:50 smithi002 ceph-mon[71816]: pgmap v305: 97 pgs: 97 active+clean; 584 KiB data, 77 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-05-13T23:14:50.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:14:50 smithi047 ceph-mon[77147]: pgmap v305: 97 pgs: 97 active+clean; 584 KiB data, 77 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-05-13T23:14:50.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:14:50 smithi115 ceph-mon[78593]: pgmap v305: 97 pgs: 97 active+clean; 584 KiB data, 77 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-05-13T23:14:51.417 INFO:teuthology.orchestra.run.smithi002.stdout:55834574920 2022-05-13T23:14:51.417 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:23b0658391ce94ca40799469eec00c28b298816c shell --fsid 16cf5f88-d311-11ec-be40-001a4aab830c -- ceph osd last-stat-seq osd.1 2022-05-13T23:14:51.710 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:50.869+0000 7f7cc023f700 1 -- 172.21.15.2:0/2242960 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f7cb8103580 msgr2=0x7f7cb8105960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:14:51.710 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:50.869+0000 7f7cc023f700 1 --2- 172.21.15.2:0/2242960 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f7cb8103580 0x7f7cb8105960 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f7cb0009c30 tx=0x7f7cb0004c50 comp rx=0 tx=0).stop 2022-05-13T23:14:51.710 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:50.869+0000 7f7cc023f700 1 -- 172.21.15.2:0/2242960 shutdown_connections 2022-05-13T23:14:51.711 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:50.869+0000 7f7cc023f700 1 --2- 172.21.15.2:0/2242960 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f7cb8103580 0x7f7cb8105960 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:14:51.711 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:50.869+0000 7f7cc023f700 1 --2- 172.21.15.2:0/2242960 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7cb8070370 0x7f7cb806e870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:14:51.711 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:50.869+0000 7f7cc023f700 1 --2- 172.21.15.2:0/2242960 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7cb806edb0 0x7f7cb806f220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:14:51.711 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:50.869+0000 7f7cc023f700 1 -- 172.21.15.2:0/2242960 >> 172.21.15.2:0/2242960 conn(0x7f7cb80f6e00 msgr2=0x7f7cb80f9210 unknown :-1 s=STATE_NONE l=0).mark_down 2022-05-13T23:14:51.712 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:50.869+0000 7f7cc023f700 1 -- 172.21.15.2:0/2242960 shutdown_connections 2022-05-13T23:14:51.712 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:50.869+0000 7f7cc023f700 1 -- 172.21.15.2:0/2242960 wait complete. 2022-05-13T23:14:51.712 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:50.870+0000 7f7cc023f700 1 Processor -- start 2022-05-13T23:14:51.712 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:50.871+0000 7f7cc023f700 1 -- start start 2022-05-13T23:14:51.712 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:50.871+0000 7f7cc023f700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7cb806edb0 0x7f7cb810e4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:14:51.713 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:50.871+0000 7f7cc023f700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f7cb8070370 0x7f7cb810ea20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:14:51.713 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:50.871+0000 7f7cc023f700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7cb8103580 0x7f7cb8115ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:14:51.713 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:50.871+0000 7f7cc023f700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7cb8115ff0 con 0x7f7cb806edb0 2022-05-13T23:14:51.713 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:50.871+0000 7f7cc023f700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7cb8116160 con 0x7f7cb8103580 2022-05-13T23:14:51.714 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:50.871+0000 7f7cc023f700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7cb8116320 con 0x7f7cb8070370 2022-05-13T23:14:51.714 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:50.871+0000 7f7cbdfdb700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7cb806edb0 0x7f7cb810e4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:14:51.714 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:50.871+0000 7f7cbdfdb700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7cb806edb0 0x7f7cb810e4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:54314/0 (socket says 172.21.15.2:54314) 2022-05-13T23:14:51.714 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:50.871+0000 7f7cbdfdb700 1 -- 172.21.15.2:0/3726837537 learned_addr learned my addr 172.21.15.2:0/3726837537 (peer_addr_for_me v2:172.21.15.2:0/0) 2022-05-13T23:14:51.714 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:50.871+0000 7f7cbe7dc700 1 --2- 172.21.15.2:0/3726837537 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7cb8103580 0x7f7cb8115ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:14:51.715 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:50.871+0000 7f7cbd7da700 1 --2- 172.21.15.2:0/3726837537 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f7cb8070370 0x7f7cb810ea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:14:51.715 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:50.872+0000 7f7cbdfdb700 1 -- 172.21.15.2:0/3726837537 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f7cb8070370 msgr2=0x7f7cb810ea20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:14:51.715 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:50.872+0000 7f7cbdfdb700 1 --2- 172.21.15.2:0/3726837537 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f7cb8070370 0x7f7cb810ea20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:14:51.715 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:50.872+0000 7f7cbdfdb700 1 -- 172.21.15.2:0/3726837537 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7cb8103580 msgr2=0x7f7cb8115ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:14:51.715 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:50.872+0000 7f7cbdfdb700 1 --2- 172.21.15.2:0/3726837537 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7cb8103580 0x7f7cb8115ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:14:51.716 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:50.872+0000 7f7cbdfdb700 1 -- 172.21.15.2:0/3726837537 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7cb4005040 con 0x7f7cb806edb0 2022-05-13T23:14:51.716 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:50.872+0000 7f7cbe7dc700 1 --2- 172.21.15.2:0/3726837537 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7cb8103580 0x7f7cb8115ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-05-13T23:14:51.716 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:50.872+0000 7f7cbdfdb700 1 --2- 172.21.15.2:0/3726837537 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7cb806edb0 0x7f7cb810e4e0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f7cb4009400 tx=0x7f7cb4009cb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-05-13T23:14:51.716 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:50.872+0000 7f7cae7fc700 1 -- 172.21.15.2:0/3726837537 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7cb4012570 con 0x7f7cb806edb0 2022-05-13T23:14:51.717 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:50.872+0000 7f7cae7fc700 1 -- 172.21.15.2:0/3726837537 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1087+0+0 (secure 0 0 0) 0x7f7cb400f040 con 0x7f7cb806edb0 2022-05-13T23:14:51.717 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:50.872+0000 7f7cc023f700 1 -- 172.21.15.2:0/3726837537 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7cb00097c0 con 0x7f7cb806edb0 2022-05-13T23:14:51.718 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:50.872+0000 7f7cae7fc700 1 -- 172.21.15.2:0/3726837537 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7cb4013b80 con 0x7f7cb806edb0 2022-05-13T23:14:51.718 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:50.873+0000 7f7cc023f700 1 -- 172.21.15.2:0/3726837537 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7cb8116980 con 0x7f7cb806edb0 2022-05-13T23:14:51.718 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:50.873+0000 7f7cc023f700 1 -- 172.21.15.2:0/3726837537 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f7cb80627a0 con 0x7f7cb806edb0 2022-05-13T23:14:51.718 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:50.875+0000 7f7cae7fc700 1 -- 172.21.15.2:0/3726837537 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 16) v1 ==== 125136+0+0 (secure 0 0 0) 0x7f7cb40126d0 con 0x7f7cb806edb0 2022-05-13T23:14:51.719 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:50.877+0000 7f7cae7fc700 1 --2- 172.21.15.2:0/3726837537 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f7ca4096ce0 0x7f7ca4099190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:14:51.719 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:50.877+0000 7f7cae7fc700 1 -- 172.21.15.2:0/3726837537 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 7975+0+0 (secure 0 0 0) 0x7f7cb40bf8d0 con 0x7f7cb806edb0 2022-05-13T23:14:51.719 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:50.877+0000 7f7cbd7da700 1 --2- 172.21.15.2:0/3726837537 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f7ca4096ce0 0x7f7ca4099190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:14:51.719 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:50.877+0000 7f7cae7fc700 1 --2- 172.21.15.2:0/3726837537 >> [v2:172.21.15.47:6816/2812542825,v1:172.21.15.47:6817/2812542825] conn(0x7f7ca40a04b0 0x7f7ca40a28c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:14:51.719 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:50.877+0000 7f7cae7fc700 1 -- 172.21.15.2:0/3726837537 --> [v2:172.21.15.47:6816/2812542825,v1:172.21.15.47:6817/2812542825] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f7ca40a2f70 con 0x7f7ca40a04b0 2022-05-13T23:14:51.720 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:50.877+0000 7f7cae7fc700 1 -- 172.21.15.2:0/3726837537 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_get_version_reply(handle=1 version=83) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7cb40bfcb0 con 0x7f7cb806edb0 2022-05-13T23:14:51.720 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:50.877+0000 7f7cbe7dc700 1 --2- 172.21.15.2:0/3726837537 >> [v2:172.21.15.47:6816/2812542825,v1:172.21.15.47:6817/2812542825] conn(0x7f7ca40a04b0 0x7f7ca40a28c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:14:51.720 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:50.878+0000 7f7cbd7da700 1 --2- 172.21.15.2:0/3726837537 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f7ca4096ce0 0x7f7ca4099190 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f7ca80048b0 tx=0x7f7ca800b040 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-05-13T23:14:51.720 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:50.878+0000 7f7cbe7dc700 1 --2- 172.21.15.2:0/3726837537 >> [v2:172.21.15.47:6816/2812542825,v1:172.21.15.47:6817/2812542825] conn(0x7f7ca40a04b0 0x7f7ca40a28c0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-05-13T23:14:51.720 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:50.879+0000 7f7cae7fc700 1 -- 172.21.15.2:0/3726837537 <== osd.7 v2:172.21.15.47:6816/2812542825 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23747 (crc 0 0 0) 0x7f7ca40a2f70 con 0x7f7ca40a04b0 2022-05-13T23:14:51.721 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:50.895+0000 7f7cc023f700 1 -- 172.21.15.2:0/3726837537 --> [v2:172.21.15.47:6816/2812542825,v1:172.21.15.47:6817/2812542825] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f7cb805cc20 con 0x7f7ca40a04b0 2022-05-13T23:14:51.721 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:50.895+0000 7f7cae7fc700 1 -- 172.21.15.2:0/3726837537 <== osd.7 v2:172.21.15.47:6816/2812542825 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f7cb805cc20 con 0x7f7ca40a04b0 2022-05-13T23:14:51.721 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:50.896+0000 7f7cc023f700 1 -- 172.21.15.2:0/3726837537 >> [v2:172.21.15.47:6816/2812542825,v1:172.21.15.47:6817/2812542825] conn(0x7f7ca40a04b0 msgr2=0x7f7ca40a28c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:14:51.721 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:50.896+0000 7f7cc023f700 1 --2- 172.21.15.2:0/3726837537 >> [v2:172.21.15.47:6816/2812542825,v1:172.21.15.47:6817/2812542825] conn(0x7f7ca40a04b0 0x7f7ca40a28c0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:14:51.721 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:50.896+0000 7f7cc023f700 1 -- 172.21.15.2:0/3726837537 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f7ca4096ce0 msgr2=0x7f7ca4099190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:14:51.722 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:50.896+0000 7f7cc023f700 1 --2- 172.21.15.2:0/3726837537 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f7ca4096ce0 0x7f7ca4099190 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f7ca80048b0 tx=0x7f7ca800b040 comp rx=0 tx=0).stop 2022-05-13T23:14:51.722 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:50.896+0000 7f7cc023f700 1 -- 172.21.15.2:0/3726837537 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7cb806edb0 msgr2=0x7f7cb810e4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:14:51.722 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:50.896+0000 7f7cc023f700 1 --2- 172.21.15.2:0/3726837537 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7cb806edb0 0x7f7cb810e4e0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f7cb4009400 tx=0x7f7cb4009cb0 comp rx=0 tx=0).stop 2022-05-13T23:14:51.722 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:50.896+0000 7f7cbe7dc700 1 -- 172.21.15.2:0/3726837537 reap_dead start 2022-05-13T23:14:51.723 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:50.896+0000 7f7cc023f700 1 -- 172.21.15.2:0/3726837537 shutdown_connections 2022-05-13T23:14:51.723 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:50.896+0000 7f7cc023f700 1 -- 172.21.15.2:0/3726837537 >> 172.21.15.2:0/3726837537 conn(0x7f7cb80f6e00 msgr2=0x7f7cb8105740 unknown :-1 s=STATE_NONE l=0).mark_down 2022-05-13T23:14:51.723 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:50.896+0000 7f7cc023f700 1 -- 172.21.15.2:0/3726837537 shutdown_connections 2022-05-13T23:14:51.723 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:50.896+0000 7f7cc023f700 1 -- 172.21.15.2:0/3726837537 wait complete. 2022-05-13T23:14:52.632 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:14:52 smithi002 ceph-mon[71816]: pgmap v306: 97 pgs: 97 active+clean; 584 KiB data, 77 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2022-05-13T23:14:52.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:14:52 smithi047 ceph-mon[77147]: pgmap v306: 97 pgs: 97 active+clean; 584 KiB data, 77 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2022-05-13T23:14:52.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:14:52 smithi115 ceph-mon[78593]: pgmap v306: 97 pgs: 97 active+clean; 584 KiB data, 77 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2022-05-13T23:14:52.996 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:52.903+0000 7fc240d01700 1 -- 172.21.15.2:0/3845489022 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc23c0fe5b0 msgr2=0x7fc23c1009f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:14:52.997 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:52.903+0000 7fc240d01700 1 --2- 172.21.15.2:0/3845489022 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc23c0fe5b0 0x7fc23c1009f0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fc2300047b0 tx=0x7fc230009cb0 comp rx=0 tx=0).stop 2022-05-13T23:14:52.997 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:52.903+0000 7fc240d01700 1 -- 172.21.15.2:0/3845489022 shutdown_connections 2022-05-13T23:14:52.997 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:52.903+0000 7fc240d01700 1 --2- 172.21.15.2:0/3845489022 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc23c101080 0x7fc23c103500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:14:52.998 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:52.903+0000 7fc240d01700 1 --2- 172.21.15.2:0/3845489022 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fc23c064ba0 0x7fc23c0fdfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:14:52.998 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:52.903+0000 7fc240d01700 1 --2- 172.21.15.2:0/3845489022 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc23c0fe5b0 0x7fc23c1009f0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:14:52.998 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:52.903+0000 7fc240d01700 1 -- 172.21.15.2:0/3845489022 >> 172.21.15.2:0/3845489022 conn(0x7fc23c0f5a30 msgr2=0x7fc23c0f7e80 unknown :-1 s=STATE_NONE l=0).mark_down 2022-05-13T23:14:52.998 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:52.903+0000 7fc240d01700 1 -- 172.21.15.2:0/3845489022 shutdown_connections 2022-05-13T23:14:52.999 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:52.904+0000 7fc240d01700 1 -- 172.21.15.2:0/3845489022 wait complete. 2022-05-13T23:14:52.999 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:52.904+0000 7fc240d01700 1 Processor -- start 2022-05-13T23:14:52.999 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:52.905+0000 7fc240d01700 1 -- start start 2022-05-13T23:14:52.999 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:52.905+0000 7fc240d01700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc23c064ba0 0x7fc23c0fb880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:14:53.000 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:52.905+0000 7fc240d01700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc23c0fe5b0 0x7fc23c0f9ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:14:53.000 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:52.905+0000 7fc240d01700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fc23c101080 0x7fc23c0fa420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:14:53.001 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:52.905+0000 7fc240d01700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc23c0fab30 con 0x7fc23c0fe5b0 2022-05-13T23:14:53.001 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:52.905+0000 7fc240d01700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc23c0faca0 con 0x7fc23c064ba0 2022-05-13T23:14:53.001 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:52.905+0000 7fc240d01700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc23c116330 con 0x7fc23c101080 2022-05-13T23:14:53.002 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:52.905+0000 7fc23affd700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc23c0fe5b0 0x7fc23c0f9ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:14:53.002 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:52.905+0000 7fc23b7fe700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc23c064ba0 0x7fc23c0fb880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:14:53.002 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:52.905+0000 7fc23b7fe700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc23c064ba0 0x7fc23c0fb880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.2:46154/0 (socket says 172.21.15.2:46154) 2022-05-13T23:14:53.003 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:52.905+0000 7fc23b7fe700 1 -- 172.21.15.2:0/2574741105 learned_addr learned my addr 172.21.15.2:0/2574741105 (peer_addr_for_me v2:172.21.15.2:0/0) 2022-05-13T23:14:53.003 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:52.905+0000 7fc23bfff700 1 --2- 172.21.15.2:0/2574741105 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fc23c101080 0x7fc23c0fa420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:14:53.003 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:52.906+0000 7fc23b7fe700 1 -- 172.21.15.2:0/2574741105 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fc23c101080 msgr2=0x7fc23c0fa420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:14:53.004 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:52.906+0000 7fc23b7fe700 1 --2- 172.21.15.2:0/2574741105 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fc23c101080 0x7fc23c0fa420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:14:53.004 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:52.906+0000 7fc23b7fe700 1 -- 172.21.15.2:0/2574741105 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc23c0fe5b0 msgr2=0x7fc23c0f9ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:14:53.005 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:52.906+0000 7fc23b7fe700 1 --2- 172.21.15.2:0/2574741105 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc23c0fe5b0 0x7fc23c0f9ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:14:53.005 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:52.906+0000 7fc23b7fe700 1 -- 172.21.15.2:0/2574741105 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc230005040 con 0x7fc23c064ba0 2022-05-13T23:14:53.005 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:52.906+0000 7fc23affd700 1 --2- 172.21.15.2:0/2574741105 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc23c0fe5b0 0x7fc23c0f9ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-05-13T23:14:53.006 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:52.906+0000 7fc23b7fe700 1 --2- 172.21.15.2:0/2574741105 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc23c064ba0 0x7fc23c0fb880 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fc22c00e940 tx=0x7fc22c00b040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-05-13T23:14:53.006 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:52.907+0000 7fc223fff700 1 -- 172.21.15.2:0/2574741105 <== mon.1 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc22c007b40 con 0x7fc23c064ba0 2022-05-13T23:14:53.006 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:52.907+0000 7fc223fff700 1 -- 172.21.15.2:0/2574741105 <== mon.1 v2:172.21.15.47:3300/0 2 ==== config(26 keys) v1 ==== 1087+0+0 (secure 0 0 0) 0x7fc22c00f040 con 0x7fc23c064ba0 2022-05-13T23:14:53.007 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:52.907+0000 7fc223fff700 1 -- 172.21.15.2:0/2574741105 <== mon.1 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc22c0148c0 con 0x7fc23c064ba0 2022-05-13T23:14:53.007 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:52.907+0000 7fc240d01700 1 -- 172.21.15.2:0/2574741105 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc23c1164d0 con 0x7fc23c064ba0 2022-05-13T23:14:53.008 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:52.907+0000 7fc240d01700 1 -- 172.21.15.2:0/2574741105 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc23c1169f0 con 0x7fc23c064ba0 2022-05-13T23:14:53.008 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:52.908+0000 7fc240d01700 1 -- 172.21.15.2:0/2574741105 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fc23c04ac70 con 0x7fc23c064ba0 2022-05-13T23:14:53.008 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:52.910+0000 7fc223fff700 1 -- 172.21.15.2:0/2574741105 <== mon.1 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 16) v1 ==== 125136+0+0 (secure 0 0 0) 0x7fc22c01b030 con 0x7fc23c064ba0 2022-05-13T23:14:53.009 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:52.911+0000 7fc223fff700 1 --2- 172.21.15.2:0/2574741105 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7fc224096ce0 0x7fc224099190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:14:53.009 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:52.911+0000 7fc223fff700 1 -- 172.21.15.2:0/2574741105 <== mon.1 v2:172.21.15.47:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 7975+0+0 (secure 0 0 0) 0x7fc22c0bee50 con 0x7fc23c064ba0 2022-05-13T23:14:53.010 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:52.912+0000 7fc23affd700 1 --2- 172.21.15.2:0/2574741105 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7fc224096ce0 0x7fc224099190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:14:53.010 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:52.912+0000 7fc223fff700 1 --2- 172.21.15.2:0/2574741105 >> [v2:172.21.15.115:6816/3959632033,v1:172.21.15.115:6817/3959632033] conn(0x7fc2240a04b0 0x7fc2240a28c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:14:53.010 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:52.912+0000 7fc223fff700 1 -- 172.21.15.2:0/2574741105 --> [v2:172.21.15.115:6816/3959632033,v1:172.21.15.115:6817/3959632033] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fc2240a2f70 con 0x7fc2240a04b0 2022-05-13T23:14:53.011 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:52.912+0000 7fc223fff700 1 -- 172.21.15.2:0/2574741105 <== mon.1 v2:172.21.15.47:3300/0 6 ==== mon_get_version_reply(handle=1 version=83) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc22c0bf180 con 0x7fc23c064ba0 2022-05-13T23:14:53.011 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:52.912+0000 7fc23bfff700 1 --2- 172.21.15.2:0/2574741105 >> [v2:172.21.15.115:6816/3959632033,v1:172.21.15.115:6817/3959632033] conn(0x7fc2240a04b0 0x7fc2240a28c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:14:53.012 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:52.913+0000 7fc23affd700 1 --2- 172.21.15.2:0/2574741105 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7fc224096ce0 0x7fc224099190 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fc230009e00 tx=0x7fc230009cb0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-05-13T23:14:53.012 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:52.913+0000 7fc23bfff700 1 --2- 172.21.15.2:0/2574741105 >> [v2:172.21.15.115:6816/3959632033,v1:172.21.15.115:6817/3959632033] conn(0x7fc2240a04b0 0x7fc2240a28c0 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-05-13T23:14:53.012 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:52.914+0000 7fc223fff700 1 -- 172.21.15.2:0/2574741105 <== osd.8 v2:172.21.15.115:6816/3959632033 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23747 (crc 0 0 0) 0x7fc2240a2f70 con 0x7fc2240a04b0 2022-05-13T23:14:53.013 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:52.931+0000 7fc240d01700 1 -- 172.21.15.2:0/2574741105 --> [v2:172.21.15.115:6816/3959632033,v1:172.21.15.115:6817/3959632033] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fc23c0fda70 con 0x7fc2240a04b0 2022-05-13T23:14:53.015 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:52.932+0000 7fc223fff700 1 -- 172.21.15.2:0/2574741105 <== osd.8 v2:172.21.15.115:6816/3959632033 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7fc23c0fda70 con 0x7fc2240a04b0 2022-05-13T23:14:53.015 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:52.932+0000 7fc240d01700 1 -- 172.21.15.2:0/2574741105 >> [v2:172.21.15.115:6816/3959632033,v1:172.21.15.115:6817/3959632033] conn(0x7fc2240a04b0 msgr2=0x7fc2240a28c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:14:53.016 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:52.932+0000 7fc240d01700 1 --2- 172.21.15.2:0/2574741105 >> [v2:172.21.15.115:6816/3959632033,v1:172.21.15.115:6817/3959632033] conn(0x7fc2240a04b0 0x7fc2240a28c0 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:14:53.016 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:52.933+0000 7fc240d01700 1 -- 172.21.15.2:0/2574741105 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7fc224096ce0 msgr2=0x7fc224099190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:14:53.017 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:52.933+0000 7fc240d01700 1 --2- 172.21.15.2:0/2574741105 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7fc224096ce0 0x7fc224099190 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fc230009e00 tx=0x7fc230009cb0 comp rx=0 tx=0).stop 2022-05-13T23:14:53.017 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:52.933+0000 7fc240d01700 1 -- 172.21.15.2:0/2574741105 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc23c064ba0 msgr2=0x7fc23c0fb880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:14:53.017 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:52.933+0000 7fc240d01700 1 --2- 172.21.15.2:0/2574741105 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc23c064ba0 0x7fc23c0fb880 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fc22c00e940 tx=0x7fc22c00b040 comp rx=0 tx=0).stop 2022-05-13T23:14:53.018 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:52.933+0000 7fc23bfff700 1 -- 172.21.15.2:0/2574741105 reap_dead start 2022-05-13T23:14:53.018 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:52.933+0000 7fc240d01700 1 -- 172.21.15.2:0/2574741105 shutdown_connections 2022-05-13T23:14:53.019 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:52.933+0000 7fc240d01700 1 -- 172.21.15.2:0/2574741105 >> 172.21.15.2:0/2574741105 conn(0x7fc23c0f5a30 msgr2=0x7fc23c0fd060 unknown :-1 s=STATE_NONE l=0).mark_down 2022-05-13T23:14:53.019 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:52.933+0000 7fc240d01700 1 -- 172.21.15.2:0/2574741105 shutdown_connections 2022-05-13T23:14:53.020 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:52.933+0000 7fc240d01700 1 -- 172.21.15.2:0/2574741105 wait complete. 2022-05-13T23:14:53.109 INFO:teuthology.orchestra.run.smithi002.stdout:219043332135 2022-05-13T23:14:53.109 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:23b0658391ce94ca40799469eec00c28b298816c shell --fsid 16cf5f88-d311-11ec-be40-001a4aab830c -- ceph osd last-stat-seq osd.7 2022-05-13T23:14:53.112 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:53.076+0000 7f879bb72700 1 -- 172.21.15.2:0/184993433 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f8794103580 msgr2=0x7f8794105960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:14:53.112 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:53.076+0000 7f879bb72700 1 --2- 172.21.15.2:0/184993433 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f8794103580 0x7f8794105960 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f878c009c30 tx=0x7f878c004c50 comp rx=0 tx=0).stop 2022-05-13T23:14:53.113 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:53.076+0000 7f879bb72700 1 -- 172.21.15.2:0/184993433 shutdown_connections 2022-05-13T23:14:53.113 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:53.076+0000 7f879bb72700 1 --2- 172.21.15.2:0/184993433 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f8794103580 0x7f8794105960 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:14:53.113 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:53.076+0000 7f879bb72700 1 --2- 172.21.15.2:0/184993433 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8794070370 0x7f879406e870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:14:53.113 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:53.076+0000 7f879bb72700 1 --2- 172.21.15.2:0/184993433 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f879406edb0 0x7f879406f220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:14:53.114 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:53.076+0000 7f879bb72700 1 -- 172.21.15.2:0/184993433 >> 172.21.15.2:0/184993433 conn(0x7f87940f6e00 msgr2=0x7f87940f9210 unknown :-1 s=STATE_NONE l=0).mark_down 2022-05-13T23:14:53.114 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:53.076+0000 7f879bb72700 1 -- 172.21.15.2:0/184993433 shutdown_connections 2022-05-13T23:14:53.114 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:53.076+0000 7f879bb72700 1 -- 172.21.15.2:0/184993433 wait complete. 2022-05-13T23:14:53.114 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:53.078+0000 7f879bb72700 1 Processor -- start 2022-05-13T23:14:53.115 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:53.078+0000 7f879bb72700 1 -- start start 2022-05-13T23:14:53.115 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:53.079+0000 7f879bb72700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f879406edb0 0x7f879410e4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:14:53.115 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:53.079+0000 7f879bb72700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f8794070370 0x7f879410ea20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:14:53.115 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:53.079+0000 7f879bb72700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8794103580 0x7f8794115ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:14:53.116 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:53.079+0000 7f879910d700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f8794070370 0x7f879410ea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:14:53.116 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:53.079+0000 7f879910d700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f8794070370 0x7f879410ea20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.2:46190/0 (socket says 172.21.15.2:46190) 2022-05-13T23:14:53.116 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:53.079+0000 7f879a10f700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8794103580 0x7f8794115ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:14:53.116 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:53.079+0000 7f879910d700 1 -- 172.21.15.2:0/2222043529 learned_addr learned my addr 172.21.15.2:0/2222043529 (peer_addr_for_me v2:172.21.15.2:0/0) 2022-05-13T23:14:53.116 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:53.079+0000 7f879a10f700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8794103580 0x7f8794115ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:54354/0 (socket says 172.21.15.2:54354) 2022-05-13T23:14:53.117 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:53.080+0000 7f879bb72700 1 -- 172.21.15.2:0/2222043529 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8794115ff0 con 0x7f8794103580 2022-05-13T23:14:53.117 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:53.080+0000 7f879bb72700 1 -- 172.21.15.2:0/2222043529 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8794116160 con 0x7f879406edb0 2022-05-13T23:14:53.117 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:53.080+0000 7f879bb72700 1 -- 172.21.15.2:0/2222043529 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8794116320 con 0x7f8794070370 2022-05-13T23:14:53.117 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:53.080+0000 7f879990e700 1 --2- 172.21.15.2:0/2222043529 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f879406edb0 0x7f879410e4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:14:53.118 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:53.080+0000 7f879910d700 1 -- 172.21.15.2:0/2222043529 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f879406edb0 msgr2=0x7f879410e4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:14:53.118 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:53.080+0000 7f879910d700 1 --2- 172.21.15.2:0/2222043529 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f879406edb0 0x7f879410e4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:14:53.118 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:53.080+0000 7f879910d700 1 -- 172.21.15.2:0/2222043529 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8794103580 msgr2=0x7f8794115ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:14:53.118 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:53.080+0000 7f879910d700 1 --2- 172.21.15.2:0/2222043529 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8794103580 0x7f8794115ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:14:53.118 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:53.080+0000 7f879910d700 1 -- 172.21.15.2:0/2222043529 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8784005040 con 0x7f8794070370 2022-05-13T23:14:53.119 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:53.080+0000 7f879a10f700 1 --2- 172.21.15.2:0/2222043529 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8794103580 0x7f8794115ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-05-13T23:14:53.119 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:53.080+0000 7f879990e700 1 --2- 172.21.15.2:0/2222043529 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f879406edb0 0x7f879410e4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-05-13T23:14:53.119 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:53.081+0000 7f879910d700 1 --2- 172.21.15.2:0/2222043529 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f8794070370 0x7f879410ea20 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f87840048f0 tx=0x7f8784009cb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-05-13T23:14:53.119 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:53.081+0000 7f878a7fc700 1 -- 172.21.15.2:0/2222043529 <== mon.2 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8784012570 con 0x7f8794070370 2022-05-13T23:14:53.120 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:53.081+0000 7f878a7fc700 1 -- 172.21.15.2:0/2222043529 <== mon.2 v2:172.21.15.115:3300/0 2 ==== config(26 keys) v1 ==== 1087+0+0 (secure 0 0 0) 0x7f878400f040 con 0x7f8794070370 2022-05-13T23:14:53.120 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:53.081+0000 7f878a7fc700 1 -- 172.21.15.2:0/2222043529 <== mon.2 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8784013b90 con 0x7f8794070370 2022-05-13T23:14:53.120 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:53.081+0000 7f879bb72700 1 -- 172.21.15.2:0/2222043529 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f878c0097c0 con 0x7f8794070370 2022-05-13T23:14:53.120 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:53.081+0000 7f879bb72700 1 -- 172.21.15.2:0/2222043529 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8794116990 con 0x7f8794070370 2022-05-13T23:14:53.120 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:53.082+0000 7f879bb72700 1 -- 172.21.15.2:0/2222043529 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f87940627a0 con 0x7f8794070370 2022-05-13T23:14:53.121 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:53.085+0000 7f878a7fc700 1 -- 172.21.15.2:0/2222043529 <== mon.2 v2:172.21.15.115:3300/0 4 ==== mgrmap(e 16) v1 ==== 125136+0+0 (secure 0 0 0) 0x7f8784012d80 con 0x7f8794070370 2022-05-13T23:14:53.121 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:53.087+0000 7f878a7fc700 1 --2- 172.21.15.2:0/2222043529 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f8780096d60 0x7f8780099210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:14:53.121 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:53.088+0000 7f878a7fc700 1 -- 172.21.15.2:0/2222043529 <== mon.2 v2:172.21.15.115:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 7975+0+0 (secure 0 0 0) 0x7f87840bf560 con 0x7f8794070370 2022-05-13T23:14:53.121 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:53.088+0000 7f879990e700 1 --2- 172.21.15.2:0/2222043529 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f8780096d60 0x7f8780099210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:14:53.122 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:53.088+0000 7f878a7fc700 1 --2- 172.21.15.2:0/2222043529 >> [v2:172.21.15.47:6824/3443413998,v1:172.21.15.47:6825/3443413998] conn(0x7f87800a0530 0x7f87800a2940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:14:53.122 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:53.088+0000 7f878a7fc700 1 -- 172.21.15.2:0/2222043529 --> [v2:172.21.15.47:6824/3443413998,v1:172.21.15.47:6825/3443413998] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f87800a2ff0 con 0x7f87800a0530 2022-05-13T23:14:53.122 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:53.088+0000 7f878a7fc700 1 -- 172.21.15.2:0/2222043529 <== mon.2 v2:172.21.15.115:3300/0 6 ==== mon_get_version_reply(handle=1 version=83) v2 ==== 24+0+0 (secure 0 0 0) 0x7f87840bf850 con 0x7f8794070370 2022-05-13T23:14:53.122 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:53.088+0000 7f879a10f700 1 --2- 172.21.15.2:0/2222043529 >> [v2:172.21.15.47:6824/3443413998,v1:172.21.15.47:6825/3443413998] conn(0x7f87800a0530 0x7f87800a2940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:14:53.122 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:53.089+0000 7f879990e700 1 --2- 172.21.15.2:0/2222043529 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f8780096d60 0x7f8780099210 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f8790009a00 tx=0x7f879000b040 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-05-13T23:14:53.123 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:53.089+0000 7f879a10f700 1 --2- 172.21.15.2:0/2222043529 >> [v2:172.21.15.47:6824/3443413998,v1:172.21.15.47:6825/3443413998] conn(0x7f87800a0530 0x7f87800a2940 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.10 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-05-13T23:14:53.123 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:53.090+0000 7f878a7fc700 1 -- 172.21.15.2:0/2222043529 <== osd.10 v2:172.21.15.47:6824/3443413998 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23747 (crc 0 0 0) 0x7f87800a2ff0 con 0x7f87800a0530 2022-05-13T23:14:53.123 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:53.110+0000 7f879bb72700 1 -- 172.21.15.2:0/2222043529 --> [v2:172.21.15.47:6824/3443413998,v1:172.21.15.47:6825/3443413998] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f879405cbe0 con 0x7f87800a0530 2022-05-13T23:14:53.123 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:53.111+0000 7f878a7fc700 1 -- 172.21.15.2:0/2222043529 <== osd.10 v2:172.21.15.47:6824/3443413998 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f879405cbe0 con 0x7f87800a0530 2022-05-13T23:14:53.125 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:53.112+0000 7f879bb72700 1 -- 172.21.15.2:0/2222043529 >> [v2:172.21.15.47:6824/3443413998,v1:172.21.15.47:6825/3443413998] conn(0x7f87800a0530 msgr2=0x7f87800a2940 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:14:53.125 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:53.112+0000 7f879bb72700 1 --2- 172.21.15.2:0/2222043529 >> [v2:172.21.15.47:6824/3443413998,v1:172.21.15.47:6825/3443413998] conn(0x7f87800a0530 0x7f87800a2940 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:14:53.125 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:53.112+0000 7f879bb72700 1 -- 172.21.15.2:0/2222043529 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f8780096d60 msgr2=0x7f8780099210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:14:53.125 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:53.112+0000 7f879bb72700 1 --2- 172.21.15.2:0/2222043529 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f8780096d60 0x7f8780099210 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f8790009a00 tx=0x7f879000b040 comp rx=0 tx=0).stop 2022-05-13T23:14:53.126 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:53.112+0000 7f879bb72700 1 -- 172.21.15.2:0/2222043529 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f8794070370 msgr2=0x7f879410ea20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:14:53.126 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:53.112+0000 7f879bb72700 1 --2- 172.21.15.2:0/2222043529 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f8794070370 0x7f879410ea20 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f87840048f0 tx=0x7f8784009cb0 comp rx=0 tx=0).stop 2022-05-13T23:14:53.126 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:53.112+0000 7f879a10f700 1 -- 172.21.15.2:0/2222043529 reap_dead start 2022-05-13T23:14:53.126 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:53.112+0000 7f879bb72700 1 -- 172.21.15.2:0/2222043529 shutdown_connections 2022-05-13T23:14:53.127 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:53.112+0000 7f879bb72700 1 -- 172.21.15.2:0/2222043529 >> 172.21.15.2:0/2222043529 conn(0x7f87940f6e00 msgr2=0x7f87940f91e0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-05-13T23:14:53.127 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:53.112+0000 7f879bb72700 1 -- 172.21.15.2:0/2222043529 shutdown_connections 2022-05-13T23:14:53.127 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:53.112+0000 7f879bb72700 1 -- 172.21.15.2:0/2222043529 wait complete. 2022-05-13T23:14:53.865 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:53.504+0000 7f096b216700 1 -- 172.21.15.2:0/611938822 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f09640fb3f0 msgr2=0x7f09640fb800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:14:53.867 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:53.504+0000 7f096b216700 1 --2- 172.21.15.2:0/611938822 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f09640fb3f0 0x7f09640fb800 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f0960004870 tx=0x7f0960009cb0 comp rx=0 tx=0).stop 2022-05-13T23:14:53.867 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:53.504+0000 7f096b216700 1 -- 172.21.15.2:0/611938822 shutdown_connections 2022-05-13T23:14:53.867 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:53.504+0000 7f096b216700 1 --2- 172.21.15.2:0/611938822 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f09640fd020 0x7f09641038e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:14:53.868 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:53.504+0000 7f096b216700 1 --2- 172.21.15.2:0/611938822 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f09640fb3f0 0x7f09640fb800 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:14:53.868 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:53.505+0000 7f096b216700 1 --2- 172.21.15.2:0/611938822 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f09640fc690 0x7f09640fcae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:14:53.869 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:53.505+0000 7f096b216700 1 -- 172.21.15.2:0/611938822 >> 172.21.15.2:0/611938822 conn(0x7f09640f6b40 msgr2=0x7f09640f8f70 unknown :-1 s=STATE_NONE l=0).mark_down 2022-05-13T23:14:53.869 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:53.505+0000 7f096b216700 1 -- 172.21.15.2:0/611938822 shutdown_connections 2022-05-13T23:14:53.870 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:53.505+0000 7f096b216700 1 -- 172.21.15.2:0/611938822 wait complete. 2022-05-13T23:14:53.870 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:53.506+0000 7f096b216700 1 Processor -- start 2022-05-13T23:14:53.870 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:53.506+0000 7f096b216700 1 -- start start 2022-05-13T23:14:53.871 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:53.506+0000 7f096b216700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f09640fb3f0 0x7f09640701d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:14:53.871 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:53.506+0000 7f096b216700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f09640fc690 0x7f096406e830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:14:53.872 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:53.506+0000 7f096b216700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f09640fd020 0x7f096406ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:14:53.873 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:53.506+0000 7f096b216700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f096406f480 con 0x7f09640fd020 2022-05-13T23:14:53.873 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:53.506+0000 7f096b216700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f096406f5f0 con 0x7f09640fb3f0 2022-05-13T23:14:53.873 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:53.506+0000 7f096b216700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0964116360 con 0x7f09640fc690 2022-05-13T23:14:53.874 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:53.506+0000 7f096a214700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f09640fb3f0 0x7f09640701d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:14:53.874 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:53.506+0000 7f096a214700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f09640fb3f0 0x7f09640701d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.2:46200/0 (socket says 172.21.15.2:46200) 2022-05-13T23:14:53.875 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:53.506+0000 7f096a214700 1 -- 172.21.15.2:0/2584401738 learned_addr learned my addr 172.21.15.2:0/2584401738 (peer_addr_for_me v2:172.21.15.2:0/0) 2022-05-13T23:14:53.875 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:53.506+0000 7f0969a13700 1 --2- 172.21.15.2:0/2584401738 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f09640fc690 0x7f096406e830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:14:53.875 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:53.506+0000 7f096aa15700 1 --2- 172.21.15.2:0/2584401738 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f09640fd020 0x7f096406ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:14:53.876 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:53.507+0000 7f0969a13700 1 -- 172.21.15.2:0/2584401738 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f09640fb3f0 msgr2=0x7f09640701d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:14:53.876 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:53.507+0000 7f0969a13700 1 --2- 172.21.15.2:0/2584401738 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f09640fb3f0 0x7f09640701d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:14:53.877 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:53.507+0000 7f0969a13700 1 -- 172.21.15.2:0/2584401738 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f09640fd020 msgr2=0x7f096406ed70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:14:53.878 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:53.507+0000 7f0969a13700 1 --2- 172.21.15.2:0/2584401738 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f09640fd020 0x7f096406ed70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:14:53.878 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:53.507+0000 7f0969a13700 1 -- 172.21.15.2:0/2584401738 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0960005040 con 0x7f09640fc690 2022-05-13T23:14:53.878 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:53.507+0000 7f096aa15700 1 --2- 172.21.15.2:0/2584401738 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f09640fd020 0x7f096406ed70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-05-13T23:14:53.879 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:53.507+0000 7f0969a13700 1 --2- 172.21.15.2:0/2584401738 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f09640fc690 0x7f096406e830 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f095400e8f0 tx=0x7f095400b040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-05-13T23:14:53.879 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:53.508+0000 7f095affd700 1 -- 172.21.15.2:0/2584401738 <== mon.2 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0954007b40 con 0x7f09640fc690 2022-05-13T23:14:53.880 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:53.508+0000 7f095affd700 1 -- 172.21.15.2:0/2584401738 <== mon.2 v2:172.21.15.115:3300/0 2 ==== config(26 keys) v1 ==== 1087+0+0 (secure 0 0 0) 0x7f095400f040 con 0x7f09640fc690 2022-05-13T23:14:53.880 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:53.508+0000 7f095affd700 1 -- 172.21.15.2:0/2584401738 <== mon.2 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0954014950 con 0x7f09640fc690 2022-05-13T23:14:53.881 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:53.508+0000 7f096b216700 1 -- 172.21.15.2:0/2584401738 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0964116500 con 0x7f09640fc690 2022-05-13T23:14:53.882 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:53.508+0000 7f096b216700 1 -- 172.21.15.2:0/2584401738 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0964116a10 con 0x7f09640fc690 2022-05-13T23:14:53.882 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:53.509+0000 7f096b216700 1 -- 172.21.15.2:0/2584401738 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f096404ac70 con 0x7f09640fc690 2022-05-13T23:14:53.882 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:53.511+0000 7f095affd700 1 -- 172.21.15.2:0/2584401738 <== mon.2 v2:172.21.15.115:3300/0 4 ==== mgrmap(e 16) v1 ==== 125136+0+0 (secure 0 0 0) 0x7f095401b030 con 0x7f09640fc690 2022-05-13T23:14:53.883 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:53.512+0000 7f095affd700 1 --2- 172.21.15.2:0/2584401738 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f0950096d60 0x7f0950099210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:14:53.883 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:53.512+0000 7f095affd700 1 -- 172.21.15.2:0/2584401738 <== mon.2 v2:172.21.15.115:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 7975+0+0 (secure 0 0 0) 0x7f09540beb80 con 0x7f09640fc690 2022-05-13T23:14:53.884 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:53.513+0000 7f096a214700 1 --2- 172.21.15.2:0/2584401738 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f0950096d60 0x7f0950099210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:14:53.884 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:53.513+0000 7f095affd700 1 --2- 172.21.15.2:0/2584401738 >> [v2:172.21.15.115:6824/1758718209,v1:172.21.15.115:6825/1758718209] conn(0x7f09500a0530 0x7f09500a2940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:14:53.884 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:53.513+0000 7f095affd700 1 -- 172.21.15.2:0/2584401738 --> [v2:172.21.15.115:6824/1758718209,v1:172.21.15.115:6825/1758718209] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f09500a2ff0 con 0x7f09500a0530 2022-05-13T23:14:53.884 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:53.513+0000 7f095affd700 1 -- 172.21.15.2:0/2584401738 <== mon.2 v2:172.21.15.115:3300/0 6 ==== mon_get_version_reply(handle=1 version=83) v2 ==== 24+0+0 (secure 0 0 0) 0x7f09540bef60 con 0x7f09640fc690 2022-05-13T23:14:53.884 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:53.513+0000 7f096aa15700 1 --2- 172.21.15.2:0/2584401738 >> [v2:172.21.15.115:6824/1758718209,v1:172.21.15.115:6825/1758718209] conn(0x7f09500a0530 0x7f09500a2940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:14:53.885 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:53.513+0000 7f096a214700 1 --2- 172.21.15.2:0/2584401738 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f0950096d60 0x7f0950099210 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f0960005010 tx=0x7f0960009cb0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-05-13T23:14:53.885 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:53.514+0000 7f096aa15700 1 --2- 172.21.15.2:0/2584401738 >> [v2:172.21.15.115:6824/1758718209,v1:172.21.15.115:6825/1758718209] conn(0x7f09500a0530 0x7f09500a2940 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.11 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-05-13T23:14:53.885 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:53.515+0000 7f095affd700 1 -- 172.21.15.2:0/2584401738 <== osd.11 v2:172.21.15.115:6824/1758718209 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23747 (crc 0 0 0) 0x7f09500a2ff0 con 0x7f09500a0530 2022-05-13T23:14:53.886 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:53.532+0000 7f096b216700 1 -- 172.21.15.2:0/2584401738 --> [v2:172.21.15.115:6824/1758718209,v1:172.21.15.115:6825/1758718209] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f096405cbe0 con 0x7f09500a0530 2022-05-13T23:14:53.886 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:53.533+0000 7f095affd700 1 -- 172.21.15.2:0/2584401738 <== osd.11 v2:172.21.15.115:6824/1758718209 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f096405cbe0 con 0x7f09500a0530 2022-05-13T23:14:53.886 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:53.533+0000 7f096b216700 1 -- 172.21.15.2:0/2584401738 >> [v2:172.21.15.115:6824/1758718209,v1:172.21.15.115:6825/1758718209] conn(0x7f09500a0530 msgr2=0x7f09500a2940 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:14:53.886 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:53.533+0000 7f096b216700 1 --2- 172.21.15.2:0/2584401738 >> [v2:172.21.15.115:6824/1758718209,v1:172.21.15.115:6825/1758718209] conn(0x7f09500a0530 0x7f09500a2940 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:14:53.886 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:53.534+0000 7f096b216700 1 -- 172.21.15.2:0/2584401738 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f0950096d60 msgr2=0x7f0950099210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:14:53.887 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:53.534+0000 7f096b216700 1 --2- 172.21.15.2:0/2584401738 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f0950096d60 0x7f0950099210 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f0960005010 tx=0x7f0960009cb0 comp rx=0 tx=0).stop 2022-05-13T23:14:53.887 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:53.534+0000 7f096b216700 1 -- 172.21.15.2:0/2584401738 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f09640fc690 msgr2=0x7f096406e830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:14:53.887 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:53.534+0000 7f096b216700 1 --2- 172.21.15.2:0/2584401738 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f09640fc690 0x7f096406e830 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f095400e8f0 tx=0x7f095400b040 comp rx=0 tx=0).stop 2022-05-13T23:14:53.887 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:53.534+0000 7f096aa15700 1 -- 172.21.15.2:0/2584401738 reap_dead start 2022-05-13T23:14:53.888 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:53.534+0000 7f096b216700 1 -- 172.21.15.2:0/2584401738 shutdown_connections 2022-05-13T23:14:53.888 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:53.534+0000 7f096b216700 1 -- 172.21.15.2:0/2584401738 >> 172.21.15.2:0/2584401738 conn(0x7f09640f6b40 msgr2=0x7f09640ff8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-05-13T23:14:53.888 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:53.534+0000 7f096b216700 1 -- 172.21.15.2:0/2584401738 shutdown_connections 2022-05-13T23:14:53.888 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:53.534+0000 7f096b216700 1 -- 172.21.15.2:0/2584401738 wait complete. 2022-05-13T23:14:54.368 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:54.330+0000 7f7581028700 1 -- 172.21.15.2:0/3111927869 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f757c0fd220 msgr2=0x7f757c103ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:14:54.369 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:54.330+0000 7f7581028700 1 --2- 172.21.15.2:0/3111927869 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f757c0fd220 0x7f757c103ae0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f7574009c30 tx=0x7f7574004c50 comp rx=0 tx=0).stop 2022-05-13T23:14:54.369 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:54.330+0000 7f7581028700 1 -- 172.21.15.2:0/3111927869 shutdown_connections 2022-05-13T23:14:54.369 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:54.330+0000 7f7581028700 1 --2- 172.21.15.2:0/3111927869 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f757c0fd220 0x7f757c103ae0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:14:54.370 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:54.330+0000 7f7581028700 1 --2- 172.21.15.2:0/3111927869 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f757c0fb5f0 0x7f757c0fba00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:14:54.370 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:54.330+0000 7f7581028700 1 --2- 172.21.15.2:0/3111927869 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f757c0fc890 0x7f757c0fcce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:14:54.370 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:54.330+0000 7f7581028700 1 -- 172.21.15.2:0/3111927869 >> 172.21.15.2:0/3111927869 conn(0x7f757c0f6e00 msgr2=0x7f757c0f9210 unknown :-1 s=STATE_NONE l=0).mark_down 2022-05-13T23:14:54.370 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:54.330+0000 7f7581028700 1 -- 172.21.15.2:0/3111927869 shutdown_connections 2022-05-13T23:14:54.371 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:54.330+0000 7f7581028700 1 -- 172.21.15.2:0/3111927869 wait complete. 2022-05-13T23:14:54.371 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:54.331+0000 7f7581028700 1 Processor -- start 2022-05-13T23:14:54.371 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:54.331+0000 7f7581028700 1 -- start start 2022-05-13T23:14:54.371 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:54.331+0000 7f7581028700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f757c0fb5f0 0x7f757c0701d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:14:54.371 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:54.331+0000 7f7581028700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f757c0fc890 0x7f757c06e830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:14:54.372 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:54.332+0000 7f7581028700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f757c0fd220 0x7f757c06ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:14:54.372 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:54.332+0000 7f7581028700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f757c06f480 con 0x7f757c0fc890 2022-05-13T23:14:54.372 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:54.332+0000 7f7581028700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f757c06f5f0 con 0x7f757c0fb5f0 2022-05-13T23:14:54.372 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:54.332+0000 7f7581028700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f757c1163d0 con 0x7f757c0fd220 2022-05-13T23:14:54.373 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:54.332+0000 7f757a59c700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f757c0fc890 0x7f757c06e830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:14:54.373 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:54.332+0000 7f757a59c700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f757c0fc890 0x7f757c06e830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:54382/0 (socket says 172.21.15.2:54382) 2022-05-13T23:14:54.373 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:54.332+0000 7f757a59c700 1 -- 172.21.15.2:0/136790792 learned_addr learned my addr 172.21.15.2:0/136790792 (peer_addr_for_me v2:172.21.15.2:0/0) 2022-05-13T23:14:54.373 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:54.332+0000 7f757ad9d700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f757c0fb5f0 0x7f757c0701d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:14:54.373 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:54.332+0000 7f757b59e700 1 --2- 172.21.15.2:0/136790792 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f757c0fd220 0x7f757c06ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:14:54.374 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:54.332+0000 7f757a59c700 1 -- 172.21.15.2:0/136790792 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f757c0fd220 msgr2=0x7f757c06ed70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:14:54.374 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:54.332+0000 7f757a59c700 1 --2- 172.21.15.2:0/136790792 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f757c0fd220 0x7f757c06ed70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:14:54.374 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:54.332+0000 7f757a59c700 1 -- 172.21.15.2:0/136790792 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f757c0fb5f0 msgr2=0x7f757c0701d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:14:54.374 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:54.332+0000 7f757a59c700 1 --2- 172.21.15.2:0/136790792 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f757c0fb5f0 0x7f757c0701d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:14:54.375 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:54.332+0000 7f757a59c700 1 -- 172.21.15.2:0/136790792 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f756c005040 con 0x7f757c0fc890 2022-05-13T23:14:54.375 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:54.332+0000 7f757ad9d700 1 --2- 172.21.15.2:0/136790792 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f757c0fb5f0 0x7f757c0701d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-05-13T23:14:54.375 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:54.333+0000 7f757a59c700 1 --2- 172.21.15.2:0/136790792 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f757c0fc890 0x7f757c06e830 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f757000e970 tx=0x7f757000b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-05-13T23:14:54.375 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:54.333+0000 7f75637fe700 1 -- 172.21.15.2:0/136790792 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7570007b40 con 0x7f757c0fc890 2022-05-13T23:14:54.375 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:54.333+0000 7f75637fe700 1 -- 172.21.15.2:0/136790792 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1087+0+0 (secure 0 0 0) 0x7f757000f040 con 0x7f757c0fc890 2022-05-13T23:14:54.376 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:54.333+0000 7f7581028700 1 -- 172.21.15.2:0/136790792 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f75740097c0 con 0x7f757c0fc890 2022-05-13T23:14:54.376 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:54.333+0000 7f75637fe700 1 -- 172.21.15.2:0/136790792 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7570014950 con 0x7f757c0fc890 2022-05-13T23:14:54.376 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:54.333+0000 7f7581028700 1 -- 172.21.15.2:0/136790792 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f757c1169b0 con 0x7f757c0fc890 2022-05-13T23:14:54.377 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:54.334+0000 7f7581028700 1 -- 172.21.15.2:0/136790792 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f757c0627a0 con 0x7f757c0fc890 2022-05-13T23:14:54.377 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:54.336+0000 7f75637fe700 1 -- 172.21.15.2:0/136790792 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 16) v1 ==== 125136+0+0 (secure 0 0 0) 0x7f757001b030 con 0x7f757c0fc890 2022-05-13T23:14:54.377 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:54.338+0000 7f75637fe700 1 --2- 172.21.15.2:0/136790792 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f7564096ce0 0x7f7564099190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:14:54.378 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:54.338+0000 7f75637fe700 1 -- 172.21.15.2:0/136790792 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 7975+0+0 (secure 0 0 0) 0x7f75700bec70 con 0x7f757c0fc890 2022-05-13T23:14:54.378 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:54.339+0000 7f757ad9d700 1 --2- 172.21.15.2:0/136790792 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f7564096ce0 0x7f7564099190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:14:54.378 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:54.339+0000 7f75637fe700 1 --2- 172.21.15.2:0/136790792 >> [v2:172.21.15.2:6802/1174843704,v1:172.21.15.2:6803/1174843704] conn(0x7f75640a04b0 0x7f75640a28c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:14:54.378 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:54.339+0000 7f75637fe700 1 -- 172.21.15.2:0/136790792 --> [v2:172.21.15.2:6802/1174843704,v1:172.21.15.2:6803/1174843704] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f75640a2f70 con 0x7f75640a04b0 2022-05-13T23:14:54.379 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:54.339+0000 7f75637fe700 1 -- 172.21.15.2:0/136790792 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_get_version_reply(handle=1 version=83) v2 ==== 24+0+0 (secure 0 0 0) 0x7f75700bef60 con 0x7f757c0fc890 2022-05-13T23:14:54.379 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:54.339+0000 7f757b59e700 1 --2- 172.21.15.2:0/136790792 >> [v2:172.21.15.2:6802/1174843704,v1:172.21.15.2:6803/1174843704] conn(0x7f75640a04b0 0x7f75640a28c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:14:54.379 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:54.339+0000 7f757ad9d700 1 --2- 172.21.15.2:0/136790792 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f7564096ce0 0x7f7564099190 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f757c0fc6f0 tx=0x7f756c00b040 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-05-13T23:14:54.379 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:54.340+0000 7f757b59e700 1 --2- 172.21.15.2:0/136790792 >> [v2:172.21.15.2:6802/1174843704,v1:172.21.15.2:6803/1174843704] conn(0x7f75640a04b0 0x7f75640a28c0 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-05-13T23:14:54.379 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:54.341+0000 7f75637fe700 1 -- 172.21.15.2:0/136790792 <== osd.0 v2:172.21.15.2:6802/1174843704 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23747 (crc 0 0 0) 0x7f75640a2f70 con 0x7f75640a04b0 2022-05-13T23:14:54.380 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:54.360+0000 7f7581028700 1 -- 172.21.15.2:0/136790792 --> [v2:172.21.15.2:6802/1174843704,v1:172.21.15.2:6803/1174843704] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f757c05cbe0 con 0x7f75640a04b0 2022-05-13T23:14:54.380 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:54.360+0000 7f75637fe700 1 -- 172.21.15.2:0/136790792 <== osd.0 v2:172.21.15.2:6802/1174843704 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f757c05cbe0 con 0x7f75640a04b0 2022-05-13T23:14:54.381 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:54.361+0000 7f7581028700 1 -- 172.21.15.2:0/136790792 >> [v2:172.21.15.2:6802/1174843704,v1:172.21.15.2:6803/1174843704] conn(0x7f75640a04b0 msgr2=0x7f75640a28c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:14:54.381 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:54.361+0000 7f7581028700 1 --2- 172.21.15.2:0/136790792 >> [v2:172.21.15.2:6802/1174843704,v1:172.21.15.2:6803/1174843704] conn(0x7f75640a04b0 0x7f75640a28c0 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:14:54.382 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:54.361+0000 7f7581028700 1 -- 172.21.15.2:0/136790792 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f7564096ce0 msgr2=0x7f7564099190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:14:54.382 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:54.361+0000 7f7581028700 1 --2- 172.21.15.2:0/136790792 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f7564096ce0 0x7f7564099190 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f757c0fc6f0 tx=0x7f756c00b040 comp rx=0 tx=0).stop 2022-05-13T23:14:54.382 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:54.361+0000 7f7581028700 1 -- 172.21.15.2:0/136790792 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f757c0fc890 msgr2=0x7f757c06e830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:14:54.383 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:54.361+0000 7f7581028700 1 --2- 172.21.15.2:0/136790792 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f757c0fc890 0x7f757c06e830 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f757000e970 tx=0x7f757000b040 comp rx=0 tx=0).stop 2022-05-13T23:14:54.383 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:54.361+0000 7f757b59e700 1 -- 172.21.15.2:0/136790792 reap_dead start 2022-05-13T23:14:54.384 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:54.361+0000 7f7581028700 1 -- 172.21.15.2:0/136790792 shutdown_connections 2022-05-13T23:14:54.384 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:54.361+0000 7f7581028700 1 -- 172.21.15.2:0/136790792 >> 172.21.15.2:0/136790792 conn(0x7f757c0f6e00 msgr2=0x7f757c0ffd30 unknown :-1 s=STATE_NONE l=0).mark_down 2022-05-13T23:14:54.384 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:54.361+0000 7f7581028700 1 -- 172.21.15.2:0/136790792 shutdown_connections 2022-05-13T23:14:54.385 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:54.362+0000 7f7581028700 1 -- 172.21.15.2:0/136790792 wait complete. 2022-05-13T23:14:54.543 INFO:teuthology.orchestra.run.smithi002.stdout:249108103205 2022-05-13T23:14:54.543 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:23b0658391ce94ca40799469eec00c28b298816c shell --fsid 16cf5f88-d311-11ec-be40-001a4aab830c -- ceph osd last-stat-seq osd.8 2022-05-13T23:14:54.618 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:14:54 smithi002 ceph-mon[71816]: pgmap v307: 97 pgs: 97 active+clean; 584 KiB data, 77 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 10 op/s 2022-05-13T23:14:54.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:14:54 smithi047 ceph-mon[77147]: pgmap v307: 97 pgs: 97 active+clean; 584 KiB data, 77 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 10 op/s 2022-05-13T23:14:54.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:14:54 smithi115 ceph-mon[78593]: pgmap v307: 97 pgs: 97 active+clean; 584 KiB data, 77 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 10 op/s 2022-05-13T23:14:55.017 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:55.013+0000 7f29bd4b4700 1 -- 172.21.15.2:0/3739288325 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29a80138f0 con 0x7f29b80fc690 2022-05-13T23:14:55.017 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:55.014+0000 7f29bf4b8700 1 -- 172.21.15.2:0/3739288325 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f29b80fc690 msgr2=0x7f29b80fcae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:14:55.017 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:55.014+0000 7f29bf4b8700 1 --2- 172.21.15.2:0/3739288325 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f29b80fc690 0x7f29b80fcae0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f29a8004870 tx=0x7f29a8009cb0 comp rx=0 tx=0).stop 2022-05-13T23:14:55.018 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:55.014+0000 7f29bf4b8700 1 -- 172.21.15.2:0/3739288325 shutdown_connections 2022-05-13T23:14:55.018 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:55.014+0000 7f29bf4b8700 1 --2- 172.21.15.2:0/3739288325 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f29b80fd020 0x7f29b81038e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:14:55.018 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:55.014+0000 7f29bf4b8700 1 --2- 172.21.15.2:0/3739288325 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f29b80fb3f0 0x7f29b80fb800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:14:55.018 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:55.014+0000 7f29bf4b8700 1 --2- 172.21.15.2:0/3739288325 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f29b80fc690 0x7f29b80fcae0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:14:55.019 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:55.014+0000 7f29bf4b8700 1 -- 172.21.15.2:0/3739288325 >> 172.21.15.2:0/3739288325 conn(0x7f29b80f6b40 msgr2=0x7f29b80f8f70 unknown :-1 s=STATE_NONE l=0).mark_down 2022-05-13T23:14:55.019 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:55.014+0000 7f29bf4b8700 1 -- 172.21.15.2:0/3739288325 shutdown_connections 2022-05-13T23:14:55.019 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:55.014+0000 7f29bf4b8700 1 -- 172.21.15.2:0/3739288325 wait complete. 2022-05-13T23:14:55.019 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:55.015+0000 7f29bf4b8700 1 Processor -- start 2022-05-13T23:14:55.019 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:55.016+0000 7f29bf4b8700 1 -- start start 2022-05-13T23:14:55.021 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:55.016+0000 7f29bf4b8700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f29b80fb3f0 0x7f29b810e3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:14:55.022 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:55.016+0000 7f29bf4b8700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f29b80fc690 0x7f29b810e8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:14:55.022 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:55.016+0000 7f29bf4b8700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f29b80fd020 0x7f29b8115980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:14:55.022 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:55.016+0000 7f29bf4b8700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29b8115ec0 con 0x7f29b80fb3f0 2022-05-13T23:14:55.022 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:55.016+0000 7f29bf4b8700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29b8116030 con 0x7f29b80fc690 2022-05-13T23:14:55.023 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:55.016+0000 7f29bf4b8700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29b81161f0 con 0x7f29b80fd020 2022-05-13T23:14:55.023 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:55.016+0000 7f29be4b6700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f29b80fb3f0 0x7f29b810e3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:14:55.023 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:55.016+0000 7f29be4b6700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f29b80fb3f0 0x7f29b810e3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:54402/0 (socket says 172.21.15.2:54402) 2022-05-13T23:14:55.023 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:55.016+0000 7f29be4b6700 1 -- 172.21.15.2:0/3273811456 learned_addr learned my addr 172.21.15.2:0/3273811456 (peer_addr_for_me v2:172.21.15.2:0/0) 2022-05-13T23:14:55.023 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:55.016+0000 7f29bdcb5700 1 --2- 172.21.15.2:0/3273811456 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f29b80fc690 0x7f29b810e8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:14:55.024 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:55.017+0000 7f29becb7700 1 --2- 172.21.15.2:0/3273811456 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f29b80fd020 0x7f29b8115980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:14:55.024 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:55.017+0000 7f29be4b6700 1 -- 172.21.15.2:0/3273811456 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f29b80fd020 msgr2=0x7f29b8115980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:14:55.024 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:55.017+0000 7f29be4b6700 1 --2- 172.21.15.2:0/3273811456 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f29b80fd020 0x7f29b8115980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:14:55.024 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:55.017+0000 7f29be4b6700 1 -- 172.21.15.2:0/3273811456 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f29b80fc690 msgr2=0x7f29b810e8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:14:55.025 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:55.017+0000 7f29be4b6700 1 --2- 172.21.15.2:0/3273811456 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f29b80fc690 0x7f29b810e8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:14:55.025 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:55.017+0000 7f29be4b6700 1 -- 172.21.15.2:0/3273811456 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29a8005040 con 0x7f29b80fb3f0 2022-05-13T23:14:55.025 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:55.017+0000 7f29be4b6700 1 --2- 172.21.15.2:0/3273811456 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f29b80fb3f0 0x7f29b810e3b0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f29b400e940 tx=0x7f29b400b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-05-13T23:14:55.025 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:55.017+0000 7f29aeffd700 1 -- 172.21.15.2:0/3273811456 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29b4007b40 con 0x7f29b80fb3f0 2022-05-13T23:14:55.025 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:55.018+0000 7f29bf4b8700 1 -- 172.21.15.2:0/3273811456 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f29b8116470 con 0x7f29b80fb3f0 2022-05-13T23:14:55.026 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:55.018+0000 7f29aeffd700 1 -- 172.21.15.2:0/3273811456 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1087+0+0 (secure 0 0 0) 0x7f29b400f040 con 0x7f29b80fb3f0 2022-05-13T23:14:55.026 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:55.018+0000 7f29aeffd700 1 -- 172.21.15.2:0/3273811456 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29b4014890 con 0x7f29b80fb3f0 2022-05-13T23:14:55.026 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:55.018+0000 7f29bf4b8700 1 -- 172.21.15.2:0/3273811456 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f29b8116a30 con 0x7f29b80fb3f0 2022-05-13T23:14:55.026 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:55.019+0000 7f29bf4b8700 1 -- 172.21.15.2:0/3273811456 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f29b804ac70 con 0x7f29b80fb3f0 2022-05-13T23:14:55.027 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:55.021+0000 7f29aeffd700 1 -- 172.21.15.2:0/3273811456 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 16) v1 ==== 125136+0+0 (secure 0 0 0) 0x7f29b401b030 con 0x7f29b80fb3f0 2022-05-13T23:14:55.027 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:55.022+0000 7f29aeffd700 1 --2- 172.21.15.2:0/3273811456 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f29a4096ce0 0x7f29a4099190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:14:55.028 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:55.022+0000 7f29aeffd700 1 -- 172.21.15.2:0/3273811456 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 7975+0+0 (secure 0 0 0) 0x7f29b40befe0 con 0x7f29b80fb3f0 2022-05-13T23:14:55.034 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:55.022+0000 7f29bdcb5700 1 --2- 172.21.15.2:0/3273811456 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f29a4096ce0 0x7f29a4099190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:14:55.034 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:55.022+0000 7f29aeffd700 1 --2- 172.21.15.2:0/3273811456 >> [v2:172.21.15.2:6818/1892198790,v1:172.21.15.2:6819/1892198790] conn(0x7f29a40a04b0 0x7f29a40a28c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:14:55.034 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:55.022+0000 7f29aeffd700 1 -- 172.21.15.2:0/3273811456 --> [v2:172.21.15.2:6818/1892198790,v1:172.21.15.2:6819/1892198790] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f29a40a2f70 con 0x7f29a40a04b0 2022-05-13T23:14:55.035 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:55.022+0000 7f29aeffd700 1 -- 172.21.15.2:0/3273811456 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_get_version_reply(handle=1 version=83) v2 ==== 24+0+0 (secure 0 0 0) 0x7f29b40bf310 con 0x7f29b80fb3f0 2022-05-13T23:14:55.035 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:55.023+0000 7f29becb7700 1 --2- 172.21.15.2:0/3273811456 >> [v2:172.21.15.2:6818/1892198790,v1:172.21.15.2:6819/1892198790] conn(0x7f29a40a04b0 0x7f29a40a28c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:14:55.035 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:55.023+0000 7f29bdcb5700 1 --2- 172.21.15.2:0/3273811456 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f29a4096ce0 0x7f29a4099190 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f29a8013490 tx=0x7f29a8009cb0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-05-13T23:14:55.035 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:55.024+0000 7f29becb7700 1 --2- 172.21.15.2:0/3273811456 >> [v2:172.21.15.2:6818/1892198790,v1:172.21.15.2:6819/1892198790] conn(0x7f29a40a04b0 0x7f29a40a28c0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-05-13T23:14:55.036 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:55.025+0000 7f29aeffd700 1 -- 172.21.15.2:0/3273811456 <== osd.6 v2:172.21.15.2:6818/1892198790 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23747 (crc 0 0 0) 0x7f29a40a2f70 con 0x7f29a40a04b0 2022-05-13T23:14:55.043 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:55.042+0000 7f29bf4b8700 1 -- 172.21.15.2:0/3273811456 --> [v2:172.21.15.2:6818/1892198790,v1:172.21.15.2:6819/1892198790] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f29b805cc20 con 0x7f29a40a04b0 2022-05-13T23:14:55.043 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:55.042+0000 7f29aeffd700 1 -- 172.21.15.2:0/3273811456 <== osd.6 v2:172.21.15.2:6818/1892198790 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f29b805cc20 con 0x7f29a40a04b0 2022-05-13T23:14:55.045 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:55.043+0000 7f29bf4b8700 1 -- 172.21.15.2:0/3273811456 >> [v2:172.21.15.2:6818/1892198790,v1:172.21.15.2:6819/1892198790] conn(0x7f29a40a04b0 msgr2=0x7f29a40a28c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:14:55.045 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:55.043+0000 7f29bf4b8700 1 --2- 172.21.15.2:0/3273811456 >> [v2:172.21.15.2:6818/1892198790,v1:172.21.15.2:6819/1892198790] conn(0x7f29a40a04b0 0x7f29a40a28c0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:14:55.045 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:55.043+0000 7f29bf4b8700 1 -- 172.21.15.2:0/3273811456 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f29a4096ce0 msgr2=0x7f29a4099190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:14:55.045 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:55.043+0000 7f29bf4b8700 1 --2- 172.21.15.2:0/3273811456 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f29a4096ce0 0x7f29a4099190 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f29a8013490 tx=0x7f29a8009cb0 comp rx=0 tx=0).stop 2022-05-13T23:14:55.048 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:55.043+0000 7f29bf4b8700 1 -- 172.21.15.2:0/3273811456 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f29b80fb3f0 msgr2=0x7f29b810e3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:14:55.049 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:55.043+0000 7f29bf4b8700 1 --2- 172.21.15.2:0/3273811456 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f29b80fb3f0 0x7f29b810e3b0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f29b400e940 tx=0x7f29b400b040 comp rx=0 tx=0).stop 2022-05-13T23:14:55.049 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:55.043+0000 7f29becb7700 1 -- 172.21.15.2:0/3273811456 reap_dead start 2022-05-13T23:14:55.049 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:55.043+0000 7f29bf4b8700 1 -- 172.21.15.2:0/3273811456 shutdown_connections 2022-05-13T23:14:55.049 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:55.043+0000 7f29bf4b8700 1 -- 172.21.15.2:0/3273811456 >> 172.21.15.2:0/3273811456 conn(0x7f29b80f6b40 msgr2=0x7f29b80ff8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-05-13T23:14:55.049 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:55.043+0000 7f29bf4b8700 1 -- 172.21.15.2:0/3273811456 shutdown_connections 2022-05-13T23:14:55.050 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:55.044+0000 7f29bf4b8700 1 -- 172.21.15.2:0/3273811456 wait complete. 2022-05-13T23:14:55.393 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:55.109+0000 7fd088d70700 1 -- 172.21.15.2:0/3785353470 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd084103360 msgr2=0x7fd084105740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:14:55.394 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:55.109+0000 7fd088d70700 1 --2- 172.21.15.2:0/3785353470 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd084103360 0x7fd084105740 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fd07c009be0 tx=0x7fd07c004d10 comp rx=0 tx=0).stop 2022-05-13T23:14:55.394 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:55.109+0000 7fd088d70700 1 -- 172.21.15.2:0/3785353470 shutdown_connections 2022-05-13T23:14:55.395 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:55.109+0000 7fd088d70700 1 --2- 172.21.15.2:0/3785353470 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd084103360 0x7fd084105740 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:14:55.395 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:55.109+0000 7fd088d70700 1 --2- 172.21.15.2:0/3785353470 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd084070390 0x7fd08406e890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:14:55.395 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:55.109+0000 7fd088d70700 1 --2- 172.21.15.2:0/3785353470 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fd08406edd0 0x7fd08406f240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:14:55.395 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:55.109+0000 7fd088d70700 1 -- 172.21.15.2:0/3785353470 >> 172.21.15.2:0/3785353470 conn(0x7fd0840f6b40 msgr2=0x7fd0840f8f70 unknown :-1 s=STATE_NONE l=0).mark_down 2022-05-13T23:14:55.396 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:55.110+0000 7fd088d70700 1 -- 172.21.15.2:0/3785353470 shutdown_connections 2022-05-13T23:14:55.396 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:55.110+0000 7fd088d70700 1 -- 172.21.15.2:0/3785353470 wait complete. 2022-05-13T23:14:55.397 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:55.110+0000 7fd088d70700 1 Processor -- start 2022-05-13T23:14:55.397 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:55.111+0000 7fd088d70700 1 -- start start 2022-05-13T23:14:55.397 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:55.111+0000 7fd088d70700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fd08406edd0 0x7fd08410e2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:14:55.397 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:55.111+0000 7fd088d70700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd084070390 0x7fd08410e7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:14:55.398 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:55.111+0000 7fd088d70700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd084103360 0x7fd084115880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:14:55.398 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:55.111+0000 7fd088d70700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd084115dc0 con 0x7fd084070390 2022-05-13T23:14:55.398 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:55.111+0000 7fd088d70700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd084115f30 con 0x7fd084103360 2022-05-13T23:14:55.399 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:55.111+0000 7fd088d70700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0841160f0 con 0x7fd08406edd0 2022-05-13T23:14:55.399 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:55.111+0000 7fd0837fe700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fd08406edd0 0x7fd08410e2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:14:55.400 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:55.111+0000 7fd083fff700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd084103360 0x7fd084115880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:14:55.401 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:55.111+0000 7fd082ffd700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd084070390 0x7fd08410e7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:14:55.401 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:55.111+0000 7fd0837fe700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fd08406edd0 0x7fd08410e2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.2:46262/0 (socket says 172.21.15.2:46262) 2022-05-13T23:14:55.402 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:55.112+0000 7fd0837fe700 1 -- 172.21.15.2:0/69148519 learned_addr learned my addr 172.21.15.2:0/69148519 (peer_addr_for_me v2:172.21.15.2:0/0) 2022-05-13T23:14:55.402 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:55.112+0000 7fd0837fe700 1 -- 172.21.15.2:0/69148519 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd084103360 msgr2=0x7fd084115880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:14:55.402 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:55.112+0000 7fd0837fe700 1 --2- 172.21.15.2:0/69148519 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd084103360 0x7fd084115880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:14:55.403 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:55.112+0000 7fd0837fe700 1 -- 172.21.15.2:0/69148519 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd084070390 msgr2=0x7fd08410e7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:14:55.403 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:55.112+0000 7fd0837fe700 1 --2- 172.21.15.2:0/69148519 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd084070390 0x7fd08410e7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:14:55.403 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:55.112+0000 7fd0837fe700 1 -- 172.21.15.2:0/69148519 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd07c0097c0 con 0x7fd08406edd0 2022-05-13T23:14:55.404 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:55.113+0000 7fd0837fe700 1 --2- 172.21.15.2:0/69148519 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fd08406edd0 0x7fd08410e2b0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fd07400e970 tx=0x7fd07400b040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-05-13T23:14:55.404 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:55.113+0000 7fd06bfff700 1 -- 172.21.15.2:0/69148519 <== mon.2 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd074007b40 con 0x7fd08406edd0 2022-05-13T23:14:55.404 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:55.113+0000 7fd06bfff700 1 -- 172.21.15.2:0/69148519 <== mon.2 v2:172.21.15.115:3300/0 2 ==== config(26 keys) v1 ==== 1087+0+0 (secure 0 0 0) 0x7fd07400f040 con 0x7fd08406edd0 2022-05-13T23:14:55.405 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:55.113+0000 7fd088d70700 1 -- 172.21.15.2:0/69148519 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd0841163d0 con 0x7fd08406edd0 2022-05-13T23:14:55.405 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:55.113+0000 7fd06bfff700 1 -- 172.21.15.2:0/69148519 <== mon.2 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd074014950 con 0x7fd08406edd0 2022-05-13T23:14:55.406 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:55.113+0000 7fd088d70700 1 -- 172.21.15.2:0/69148519 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd084116940 con 0x7fd08406edd0 2022-05-13T23:14:55.406 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:55.114+0000 7fd088d70700 1 -- 172.21.15.2:0/69148519 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fd08404ac70 con 0x7fd08406edd0 2022-05-13T23:14:55.406 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:55.116+0000 7fd06bfff700 1 -- 172.21.15.2:0/69148519 <== mon.2 v2:172.21.15.115:3300/0 4 ==== mgrmap(e 16) v1 ==== 125136+0+0 (secure 0 0 0) 0x7fd07401b030 con 0x7fd08406edd0 2022-05-13T23:14:55.406 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:55.117+0000 7fd06bfff700 1 --2- 172.21.15.2:0/69148519 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7fd06c096ce0 0x7fd06c099190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:14:55.407 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:55.118+0000 7fd06bfff700 1 -- 172.21.15.2:0/69148519 <== mon.2 v2:172.21.15.115:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 7975+0+0 (secure 0 0 0) 0x7fd0740bece0 con 0x7fd08406edd0 2022-05-13T23:14:55.407 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:55.118+0000 7fd082ffd700 1 --2- 172.21.15.2:0/69148519 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7fd06c096ce0 0x7fd06c099190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:14:55.408 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:55.118+0000 7fd06bfff700 1 --2- 172.21.15.2:0/69148519 >> [v2:172.21.15.2:6810/1099909411,v1:172.21.15.2:6811/1099909411] conn(0x7fd06c0a04b0 0x7fd06c0a28c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:14:55.408 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:55.118+0000 7fd06bfff700 1 -- 172.21.15.2:0/69148519 --> [v2:172.21.15.2:6810/1099909411,v1:172.21.15.2:6811/1099909411] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd06c0a2f70 con 0x7fd06c0a04b0 2022-05-13T23:14:55.409 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:55.118+0000 7fd06bfff700 1 -- 172.21.15.2:0/69148519 <== mon.2 v2:172.21.15.115:3300/0 6 ==== mon_get_version_reply(handle=1 version=83) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd0740befd0 con 0x7fd08406edd0 2022-05-13T23:14:55.409 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:55.118+0000 7fd083fff700 1 --2- 172.21.15.2:0/69148519 >> [v2:172.21.15.2:6810/1099909411,v1:172.21.15.2:6811/1099909411] conn(0x7fd06c0a04b0 0x7fd06c0a28c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:14:55.409 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:55.118+0000 7fd082ffd700 1 --2- 172.21.15.2:0/69148519 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7fd06c096ce0 0x7fd06c099190 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fd0780048b0 tx=0x7fd07800b040 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-05-13T23:14:55.410 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:55.119+0000 7fd083fff700 1 --2- 172.21.15.2:0/69148519 >> [v2:172.21.15.2:6810/1099909411,v1:172.21.15.2:6811/1099909411] conn(0x7fd06c0a04b0 0x7fd06c0a28c0 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-05-13T23:14:55.411 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:55.120+0000 7fd06bfff700 1 -- 172.21.15.2:0/69148519 <== osd.3 v2:172.21.15.2:6810/1099909411 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23747 (crc 0 0 0) 0x7fd06c0a2f70 con 0x7fd06c0a04b0 2022-05-13T23:14:55.411 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:55.138+0000 7fd088d70700 1 -- 172.21.15.2:0/69148519 --> [v2:172.21.15.2:6810/1099909411,v1:172.21.15.2:6811/1099909411] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fd08405cba0 con 0x7fd06c0a04b0 2022-05-13T23:14:55.412 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:55.138+0000 7fd06bfff700 1 -- 172.21.15.2:0/69148519 <== osd.3 v2:172.21.15.2:6810/1099909411 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7fd08405cba0 con 0x7fd06c0a04b0 2022-05-13T23:14:55.414 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:55.139+0000 7fd088d70700 1 -- 172.21.15.2:0/69148519 >> [v2:172.21.15.2:6810/1099909411,v1:172.21.15.2:6811/1099909411] conn(0x7fd06c0a04b0 msgr2=0x7fd06c0a28c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:14:55.414 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:55.139+0000 7fd088d70700 1 --2- 172.21.15.2:0/69148519 >> [v2:172.21.15.2:6810/1099909411,v1:172.21.15.2:6811/1099909411] conn(0x7fd06c0a04b0 0x7fd06c0a28c0 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:14:55.415 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:55.139+0000 7fd088d70700 1 -- 172.21.15.2:0/69148519 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7fd06c096ce0 msgr2=0x7fd06c099190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:14:55.415 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:55.139+0000 7fd088d70700 1 --2- 172.21.15.2:0/69148519 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7fd06c096ce0 0x7fd06c099190 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fd0780048b0 tx=0x7fd07800b040 comp rx=0 tx=0).stop 2022-05-13T23:14:55.415 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:55.139+0000 7fd088d70700 1 -- 172.21.15.2:0/69148519 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fd08406edd0 msgr2=0x7fd08410e2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:14:55.416 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:55.139+0000 7fd088d70700 1 --2- 172.21.15.2:0/69148519 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fd08406edd0 0x7fd08410e2b0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fd07400e970 tx=0x7fd07400b040 comp rx=0 tx=0).stop 2022-05-13T23:14:55.417 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:55.139+0000 7fd083fff700 1 -- 172.21.15.2:0/69148519 reap_dead start 2022-05-13T23:14:55.417 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:55.139+0000 7fd088d70700 1 -- 172.21.15.2:0/69148519 shutdown_connections 2022-05-13T23:14:55.417 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:55.139+0000 7fd088d70700 1 -- 172.21.15.2:0/69148519 >> 172.21.15.2:0/69148519 conn(0x7fd0840f6b40 msgr2=0x7fd0840f8f20 unknown :-1 s=STATE_NONE l=0).mark_down 2022-05-13T23:14:55.417 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:55.139+0000 7fd088d70700 1 -- 172.21.15.2:0/69148519 shutdown_connections 2022-05-13T23:14:55.418 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:55.139+0000 7fd088d70700 1 -- 172.21.15.2:0/69148519 wait complete. 2022-05-13T23:14:55.476 INFO:teuthology.orchestra.run.smithi002.stdout:304942678041 2022-05-13T23:14:55.477 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:23b0658391ce94ca40799469eec00c28b298816c shell --fsid 16cf5f88-d311-11ec-be40-001a4aab830c -- ceph osd last-stat-seq osd.10 2022-05-13T23:14:55.959 INFO:teuthology.orchestra.run.smithi002.stdout:34359738444 2022-05-13T23:14:55.960 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:23b0658391ce94ca40799469eec00c28b298816c shell --fsid 16cf5f88-d311-11ec-be40-001a4aab830c -- ceph osd last-stat-seq osd.0 2022-05-13T23:14:56.167 INFO:teuthology.orchestra.run.smithi002.stdout:107374182461 2022-05-13T23:14:56.167 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:23b0658391ce94ca40799469eec00c28b298816c shell --fsid 16cf5f88-d311-11ec-be40-001a4aab830c -- ceph osd last-stat-seq osd.3 2022-05-13T23:14:56.441 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:14:56 smithi002 ceph-mon[71816]: pgmap v308: 97 pgs: 97 active+clean; 584 KiB data, 77 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 11 op/s 2022-05-13T23:14:56.453 INFO:teuthology.orchestra.run.smithi002.stdout:335007449109 2022-05-13T23:14:56.454 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:23b0658391ce94ca40799469eec00c28b298816c shell --fsid 16cf5f88-d311-11ec-be40-001a4aab830c -- ceph osd last-stat-seq osd.11 2022-05-13T23:14:56.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:14:56 smithi047 ceph-mon[77147]: pgmap v308: 97 pgs: 97 active+clean; 584 KiB data, 77 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 11 op/s 2022-05-13T23:14:56.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:14:56 smithi115 ceph-mon[78593]: pgmap v308: 97 pgs: 97 active+clean; 584 KiB data, 77 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 11 op/s 2022-05-13T23:14:57.286 INFO:teuthology.orchestra.run.smithi002.stdout:197568495661 2022-05-13T23:14:57.287 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:23b0658391ce94ca40799469eec00c28b298816c shell --fsid 16cf5f88-d311-11ec-be40-001a4aab830c -- ceph osd last-stat-seq osd.6 2022-05-13T23:14:57.376 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:56.712+0000 7fc3344f4700 1 -- 172.21.15.2:0/1368108366 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fc32c070370 msgr2=0x7fc32c06e870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:14:57.376 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:56.712+0000 7fc3344f4700 1 --2- 172.21.15.2:0/1368108366 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fc32c070370 0x7fc32c06e870 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fc3280047b0 tx=0x7fc328009cb0 comp rx=0 tx=0).stop 2022-05-13T23:14:57.377 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:56.712+0000 7fc3344f4700 1 -- 172.21.15.2:0/1368108366 shutdown_connections 2022-05-13T23:14:57.377 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:56.712+0000 7fc3344f4700 1 --2- 172.21.15.2:0/1368108366 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc32c103580 0x7fc32c105960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:14:57.377 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:56.712+0000 7fc3344f4700 1 --2- 172.21.15.2:0/1368108366 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fc32c070370 0x7fc32c06e870 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:14:57.377 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:56.712+0000 7fc3344f4700 1 --2- 172.21.15.2:0/1368108366 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc32c06edb0 0x7fc32c06f220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:14:57.378 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:56.712+0000 7fc3344f4700 1 -- 172.21.15.2:0/1368108366 >> 172.21.15.2:0/1368108366 conn(0x7fc32c0f6e00 msgr2=0x7fc32c0f9210 unknown :-1 s=STATE_NONE l=0).mark_down 2022-05-13T23:14:57.378 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:56.712+0000 7fc3344f4700 1 -- 172.21.15.2:0/1368108366 shutdown_connections 2022-05-13T23:14:57.378 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:56.712+0000 7fc3344f4700 1 -- 172.21.15.2:0/1368108366 wait complete. 2022-05-13T23:14:57.378 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:56.713+0000 7fc3344f4700 1 Processor -- start 2022-05-13T23:14:57.379 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:56.713+0000 7fc3344f4700 1 -- start start 2022-05-13T23:14:57.379 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:56.714+0000 7fc3344f4700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc32c06edb0 0x7fc32c10e620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:14:57.380 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:56.714+0000 7fc3344f4700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fc32c070370 0x7fc32c10eb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:14:57.380 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:56.714+0000 7fc3344f4700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc32c103580 0x7fc32c115bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:14:57.380 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:56.714+0000 7fc3344f4700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc32c116130 con 0x7fc32c06edb0 2022-05-13T23:14:57.380 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:56.714+0000 7fc3344f4700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc32c1162a0 con 0x7fc32c103580 2022-05-13T23:14:57.381 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:56.714+0000 7fc3344f4700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc32c116460 con 0x7fc32c070370 2022-05-13T23:14:57.381 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:56.714+0000 7fc332290700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc32c06edb0 0x7fc32c10e620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:14:57.381 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:56.714+0000 7fc331a8f700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fc32c070370 0x7fc32c10eb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:14:57.381 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:56.714+0000 7fc332a91700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc32c103580 0x7fc32c115bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:14:57.382 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:56.714+0000 7fc332290700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc32c06edb0 0x7fc32c10e620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:54430/0 (socket says 172.21.15.2:54430) 2022-05-13T23:14:57.382 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:56.714+0000 7fc332290700 1 -- 172.21.15.2:0/858807030 learned_addr learned my addr 172.21.15.2:0/858807030 (peer_addr_for_me v2:172.21.15.2:0/0) 2022-05-13T23:14:57.382 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:56.715+0000 7fc332a91700 1 -- 172.21.15.2:0/858807030 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fc32c070370 msgr2=0x7fc32c10eb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:14:57.382 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:56.715+0000 7fc332a91700 1 --2- 172.21.15.2:0/858807030 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fc32c070370 0x7fc32c10eb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:14:57.383 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:56.715+0000 7fc332a91700 1 -- 172.21.15.2:0/858807030 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc32c06edb0 msgr2=0x7fc32c10e620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:14:57.383 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:56.715+0000 7fc332a91700 1 --2- 172.21.15.2:0/858807030 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc32c06edb0 0x7fc32c10e620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:14:57.383 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:56.715+0000 7fc332a91700 1 -- 172.21.15.2:0/858807030 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3240097c0 con 0x7fc32c103580 2022-05-13T23:14:57.383 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:56.715+0000 7fc332290700 1 --2- 172.21.15.2:0/858807030 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc32c06edb0 0x7fc32c10e620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-05-13T23:14:57.383 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:56.715+0000 7fc332a91700 1 --2- 172.21.15.2:0/858807030 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc32c103580 0x7fc32c115bf0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fc32c070140 tx=0x7fc324004d10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-05-13T23:14:57.384 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:56.715+0000 7fc322ffd700 1 -- 172.21.15.2:0/858807030 <== mon.1 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3240099f0 con 0x7fc32c103580 2022-05-13T23:14:57.384 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:56.715+0000 7fc3344f4700 1 -- 172.21.15.2:0/858807030 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc328005040 con 0x7fc32c103580 2022-05-13T23:14:57.384 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:56.715+0000 7fc322ffd700 1 -- 172.21.15.2:0/858807030 <== mon.1 v2:172.21.15.47:3300/0 2 ==== config(26 keys) v1 ==== 1087+0+0 (secure 0 0 0) 0x7fc32400f8c0 con 0x7fc32c103580 2022-05-13T23:14:57.384 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:56.715+0000 7fc322ffd700 1 -- 172.21.15.2:0/858807030 <== mon.1 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc324007d30 con 0x7fc32c103580 2022-05-13T23:14:57.385 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:56.715+0000 7fc3344f4700 1 -- 172.21.15.2:0/858807030 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc32c116b20 con 0x7fc32c103580 2022-05-13T23:14:57.385 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:56.716+0000 7fc3344f4700 1 -- 172.21.15.2:0/858807030 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fc32c0627a0 con 0x7fc32c103580 2022-05-13T23:14:57.386 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:56.718+0000 7fc322ffd700 1 -- 172.21.15.2:0/858807030 <== mon.1 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 16) v1 ==== 125136+0+0 (secure 0 0 0) 0x7fc32401c030 con 0x7fc32c103580 2022-05-13T23:14:57.386 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:56.720+0000 7fc322ffd700 1 --2- 172.21.15.2:0/858807030 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7fc318096ce0 0x7fc318099190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:14:57.386 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:56.720+0000 7fc322ffd700 1 -- 172.21.15.2:0/858807030 <== mon.1 v2:172.21.15.47:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 7975+0+0 (secure 0 0 0) 0x7fc3240bf280 con 0x7fc32c103580 2022-05-13T23:14:57.386 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:56.720+0000 7fc332290700 1 --2- 172.21.15.2:0/858807030 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7fc318096ce0 0x7fc318099190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:14:57.387 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:56.720+0000 7fc322ffd700 1 --2- 172.21.15.2:0/858807030 >> [v2:172.21.15.47:6808/882753861,v1:172.21.15.47:6809/882753861] conn(0x7fc3180a04b0 0x7fc3180a28c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:14:57.388 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:56.720+0000 7fc322ffd700 1 -- 172.21.15.2:0/858807030 --> [v2:172.21.15.47:6808/882753861,v1:172.21.15.47:6809/882753861] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fc3180a2f70 con 0x7fc3180a04b0 2022-05-13T23:14:57.388 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:56.720+0000 7fc322ffd700 1 -- 172.21.15.2:0/858807030 <== mon.1 v2:172.21.15.47:3300/0 6 ==== mon_get_version_reply(handle=1 version=83) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc3240bf570 con 0x7fc32c103580 2022-05-13T23:14:57.388 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:56.721+0000 7fc331a8f700 1 --2- 172.21.15.2:0/858807030 >> [v2:172.21.15.47:6808/882753861,v1:172.21.15.47:6809/882753861] conn(0x7fc3180a04b0 0x7fc3180a28c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:14:57.389 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:56.721+0000 7fc332290700 1 --2- 172.21.15.2:0/858807030 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7fc318096ce0 0x7fc318099190 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fc328000e50 tx=0x7fc328009cb0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-05-13T23:14:57.389 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:56.721+0000 7fc331a8f700 1 --2- 172.21.15.2:0/858807030 >> [v2:172.21.15.47:6808/882753861,v1:172.21.15.47:6809/882753861] conn(0x7fc3180a04b0 0x7fc3180a28c0 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-05-13T23:14:57.389 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:56.722+0000 7fc322ffd700 1 -- 172.21.15.2:0/858807030 <== osd.4 v2:172.21.15.47:6808/882753861 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23747 (crc 0 0 0) 0x7fc3180a2f70 con 0x7fc3180a04b0 2022-05-13T23:14:57.389 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:56.738+0000 7fc3344f4700 1 -- 172.21.15.2:0/858807030 --> [v2:172.21.15.47:6808/882753861,v1:172.21.15.47:6809/882753861] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fc32c05cbe0 con 0x7fc3180a04b0 2022-05-13T23:14:57.390 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:56.739+0000 7fc322ffd700 1 -- 172.21.15.2:0/858807030 <== osd.4 v2:172.21.15.47:6808/882753861 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7fc32c05cbe0 con 0x7fc3180a04b0 2022-05-13T23:14:57.391 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:56.739+0000 7fc3344f4700 1 -- 172.21.15.2:0/858807030 >> [v2:172.21.15.47:6808/882753861,v1:172.21.15.47:6809/882753861] conn(0x7fc3180a04b0 msgr2=0x7fc3180a28c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:14:57.391 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:56.739+0000 7fc3344f4700 1 --2- 172.21.15.2:0/858807030 >> [v2:172.21.15.47:6808/882753861,v1:172.21.15.47:6809/882753861] conn(0x7fc3180a04b0 0x7fc3180a28c0 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:14:57.391 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:56.739+0000 7fc3344f4700 1 -- 172.21.15.2:0/858807030 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7fc318096ce0 msgr2=0x7fc318099190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:14:57.391 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:56.739+0000 7fc3344f4700 1 --2- 172.21.15.2:0/858807030 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7fc318096ce0 0x7fc318099190 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fc328000e50 tx=0x7fc328009cb0 comp rx=0 tx=0).stop 2022-05-13T23:14:57.392 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:56.739+0000 7fc3344f4700 1 -- 172.21.15.2:0/858807030 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc32c103580 msgr2=0x7fc32c115bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:14:57.392 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:56.739+0000 7fc3344f4700 1 --2- 172.21.15.2:0/858807030 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc32c103580 0x7fc32c115bf0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fc32c070140 tx=0x7fc324004d10 comp rx=0 tx=0).stop 2022-05-13T23:14:57.392 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:56.739+0000 7fc332a91700 1 -- 172.21.15.2:0/858807030 reap_dead start 2022-05-13T23:14:57.392 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:56.739+0000 7fc3344f4700 1 -- 172.21.15.2:0/858807030 shutdown_connections 2022-05-13T23:14:57.393 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:56.739+0000 7fc3344f4700 1 -- 172.21.15.2:0/858807030 >> 172.21.15.2:0/858807030 conn(0x7fc32c0f6e00 msgr2=0x7fc32c0f91e0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-05-13T23:14:57.393 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:56.740+0000 7fc3344f4700 1 -- 172.21.15.2:0/858807030 shutdown_connections 2022-05-13T23:14:57.393 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:56.740+0000 7fc3344f4700 1 -- 172.21.15.2:0/858807030 wait complete. 2022-05-13T23:14:57.639 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/16cf5f88-d311-11ec-be40-001a4aab830c/mon.a/config 2022-05-13T23:14:57.918 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/16cf5f88-d311-11ec-be40-001a4aab830c/mon.a/config 2022-05-13T23:14:57.941 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/16cf5f88-d311-11ec-be40-001a4aab830c/mon.a/config 2022-05-13T23:14:58.391 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/16cf5f88-d311-11ec-be40-001a4aab830c/mon.a/config 2022-05-13T23:14:58.563 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/16cf5f88-d311-11ec-be40-001a4aab830c/mon.a/config 2022-05-13T23:14:58.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:14:58 smithi047 ceph-mon[77147]: pgmap v309: 97 pgs: 97 active+clean; 584 KiB data, 77 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2022-05-13T23:14:58.822 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:14:58 smithi002 ceph-mon[71816]: pgmap v309: 97 pgs: 97 active+clean; 584 KiB data, 77 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2022-05-13T23:14:58.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:14:58 smithi115 ceph-mon[78593]: pgmap v309: 97 pgs: 97 active+clean; 584 KiB data, 77 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2022-05-13T23:14:59.217 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:59.035+0000 7f3a98753700 1 -- 172.21.15.2:0/1709461962 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f3a900fb7a0 msgr2=0x7f3a900fbbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:14:59.218 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:59.035+0000 7f3a98753700 1 --2- 172.21.15.2:0/1709461962 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f3a900fb7a0 0x7f3a900fbbf0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f3a80004870 tx=0x7f3a80009cb0 comp rx=0 tx=0).stop 2022-05-13T23:14:59.218 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:59.035+0000 7f3a98753700 1 -- 172.21.15.2:0/1709461962 shutdown_connections 2022-05-13T23:14:59.218 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:59.035+0000 7f3a98753700 1 --2- 172.21.15.2:0/1709461962 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3a900fc130 0x7f3a901029f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:14:59.218 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:59.035+0000 7f3a98753700 1 --2- 172.21.15.2:0/1709461962 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3a900fa500 0x7f3a900fa910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:14:59.219 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:59.035+0000 7f3a98753700 1 --2- 172.21.15.2:0/1709461962 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f3a900fb7a0 0x7f3a900fbbf0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:14:59.219 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:59.035+0000 7f3a98753700 1 -- 172.21.15.2:0/1709461962 >> 172.21.15.2:0/1709461962 conn(0x7f3a900f5cf0 msgr2=0x7f3a900f8120 unknown :-1 s=STATE_NONE l=0).mark_down 2022-05-13T23:14:59.219 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:59.035+0000 7f3a98753700 1 -- 172.21.15.2:0/1709461962 shutdown_connections 2022-05-13T23:14:59.219 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:59.035+0000 7f3a98753700 1 -- 172.21.15.2:0/1709461962 wait complete. 2022-05-13T23:14:59.219 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:59.036+0000 7f3a98753700 1 Processor -- start 2022-05-13T23:14:59.220 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:59.036+0000 7f3a98753700 1 -- start start 2022-05-13T23:14:59.220 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:59.036+0000 7f3a98753700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3a900fa500 0x7f3a90064980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:14:59.220 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:59.036+0000 7f3a98753700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f3a900fb7a0 0x7f3a90111230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:14:59.220 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:59.036+0000 7f3a98753700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3a900fc130 0x7f3a90113830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:14:59.221 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:59.036+0000 7f3a98753700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a90113f40 con 0x7f3a900fa500 2022-05-13T23:14:59.221 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:59.036+0000 7f3a98753700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a901140b0 con 0x7f3a900fc130 2022-05-13T23:14:59.222 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:59.036+0000 7f3a98753700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a90114270 con 0x7f3a900fb7a0 2022-05-13T23:14:59.222 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:59.037+0000 7f3a964ef700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3a900fa500 0x7f3a90064980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:14:59.222 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:59.037+0000 7f3a95cee700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f3a900fb7a0 0x7f3a90111230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:14:59.223 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:59.037+0000 7f3a96cf0700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3a900fc130 0x7f3a90113830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:14:59.223 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:59.037+0000 7f3a964ef700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3a900fa500 0x7f3a90064980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:54448/0 (socket says 172.21.15.2:54448) 2022-05-13T23:14:59.223 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:59.037+0000 7f3a964ef700 1 -- 172.21.15.2:0/2144668592 learned_addr learned my addr 172.21.15.2:0/2144668592 (peer_addr_for_me v2:172.21.15.2:0/0) 2022-05-13T23:14:59.223 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:59.037+0000 7f3a964ef700 1 -- 172.21.15.2:0/2144668592 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f3a900fb7a0 msgr2=0x7f3a90111230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:14:59.224 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:59.037+0000 7f3a964ef700 1 --2- 172.21.15.2:0/2144668592 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f3a900fb7a0 0x7f3a90111230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:14:59.224 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:59.037+0000 7f3a964ef700 1 -- 172.21.15.2:0/2144668592 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3a900fc130 msgr2=0x7f3a90113830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:14:59.225 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:59.037+0000 7f3a964ef700 1 --2- 172.21.15.2:0/2144668592 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3a900fc130 0x7f3a90113830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:14:59.225 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:59.037+0000 7f3a964ef700 1 -- 172.21.15.2:0/2144668592 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a80005040 con 0x7f3a900fa500 2022-05-13T23:14:59.225 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:59.038+0000 7f3a964ef700 1 --2- 172.21.15.2:0/2144668592 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3a900fa500 0x7f3a90064980 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f3a8c00e940 tx=0x7f3a8c00b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-05-13T23:14:59.226 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:59.038+0000 7f3a86ffd700 1 -- 172.21.15.2:0/2144668592 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a8c007b40 con 0x7f3a900fa500 2022-05-13T23:14:59.226 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:59.038+0000 7f3a98753700 1 -- 172.21.15.2:0/2144668592 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3a9010b000 con 0x7f3a900fa500 2022-05-13T23:14:59.226 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:59.038+0000 7f3a86ffd700 1 -- 172.21.15.2:0/2144668592 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1087+0+0 (secure 0 0 0) 0x7f3a8c00f040 con 0x7f3a900fa500 2022-05-13T23:14:59.227 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:59.038+0000 7f3a86ffd700 1 -- 172.21.15.2:0/2144668592 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a8c014890 con 0x7f3a900fa500 2022-05-13T23:14:59.230 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:59.038+0000 7f3a98753700 1 -- 172.21.15.2:0/2144668592 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3a9010b5c0 con 0x7f3a900fa500 2022-05-13T23:14:59.230 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:59.039+0000 7f3a98753700 1 -- 172.21.15.2:0/2144668592 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f3a900627a0 con 0x7f3a900fa500 2022-05-13T23:14:59.231 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:59.041+0000 7f3a86ffd700 1 -- 172.21.15.2:0/2144668592 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 16) v1 ==== 125136+0+0 (secure 0 0 0) 0x7f3a8c01b030 con 0x7f3a900fa500 2022-05-13T23:14:59.231 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:59.042+0000 7f3a86ffd700 1 --2- 172.21.15.2:0/2144668592 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f3a7c096ce0 0x7f3a7c099190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:14:59.232 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:59.042+0000 7f3a86ffd700 1 -- 172.21.15.2:0/2144668592 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 7975+0+0 (secure 0 0 0) 0x7f3a8c0bee90 con 0x7f3a900fa500 2022-05-13T23:14:59.232 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:59.043+0000 7f3a95cee700 1 --2- 172.21.15.2:0/2144668592 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f3a7c096ce0 0x7f3a7c099190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:14:59.232 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:59.043+0000 7f3a86ffd700 1 --2- 172.21.15.2:0/2144668592 >> [v2:172.21.15.115:6808/2122391993,v1:172.21.15.115:6809/2122391993] conn(0x7f3a7c0a04b0 0x7f3a7c0a28c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:14:59.233 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:59.043+0000 7f3a86ffd700 1 -- 172.21.15.2:0/2144668592 --> [v2:172.21.15.115:6808/2122391993,v1:172.21.15.115:6809/2122391993] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f3a7c0a2f70 con 0x7f3a7c0a04b0 2022-05-13T23:14:59.233 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:59.043+0000 7f3a86ffd700 1 -- 172.21.15.2:0/2144668592 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_get_version_reply(handle=1 version=83) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3a8c0bf1c0 con 0x7f3a900fa500 2022-05-13T23:14:59.234 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:59.043+0000 7f3a96cf0700 1 --2- 172.21.15.2:0/2144668592 >> [v2:172.21.15.115:6808/2122391993,v1:172.21.15.115:6809/2122391993] conn(0x7f3a7c0a04b0 0x7f3a7c0a28c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:14:59.234 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:59.043+0000 7f3a95cee700 1 --2- 172.21.15.2:0/2144668592 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f3a7c096ce0 0x7f3a7c099190 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f3a80004840 tx=0x7f3a80009cb0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-05-13T23:14:59.235 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:59.044+0000 7f3a96cf0700 1 --2- 172.21.15.2:0/2144668592 >> [v2:172.21.15.115:6808/2122391993,v1:172.21.15.115:6809/2122391993] conn(0x7f3a7c0a04b0 0x7f3a7c0a28c0 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-05-13T23:14:59.235 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:59.045+0000 7f3a86ffd700 1 -- 172.21.15.2:0/2144668592 <== osd.5 v2:172.21.15.115:6808/2122391993 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23747 (crc 0 0 0) 0x7f3a7c0a2f70 con 0x7f3a7c0a04b0 2022-05-13T23:14:59.235 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:59.061+0000 7f3a98753700 1 -- 172.21.15.2:0/2144668592 --> [v2:172.21.15.115:6808/2122391993,v1:172.21.15.115:6809/2122391993] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f3a9005cc20 con 0x7f3a7c0a04b0 2022-05-13T23:14:59.236 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:59.061+0000 7f3a86ffd700 1 -- 172.21.15.2:0/2144668592 <== osd.5 v2:172.21.15.115:6808/2122391993 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f3a9005cc20 con 0x7f3a7c0a04b0 2022-05-13T23:14:59.238 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:59.062+0000 7f3a98753700 1 -- 172.21.15.2:0/2144668592 >> [v2:172.21.15.115:6808/2122391993,v1:172.21.15.115:6809/2122391993] conn(0x7f3a7c0a04b0 msgr2=0x7f3a7c0a28c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:14:59.238 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:59.062+0000 7f3a98753700 1 --2- 172.21.15.2:0/2144668592 >> [v2:172.21.15.115:6808/2122391993,v1:172.21.15.115:6809/2122391993] conn(0x7f3a7c0a04b0 0x7f3a7c0a28c0 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:14:59.239 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:59.062+0000 7f3a98753700 1 -- 172.21.15.2:0/2144668592 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f3a7c096ce0 msgr2=0x7f3a7c099190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:14:59.239 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:59.062+0000 7f3a98753700 1 --2- 172.21.15.2:0/2144668592 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f3a7c096ce0 0x7f3a7c099190 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f3a80004840 tx=0x7f3a80009cb0 comp rx=0 tx=0).stop 2022-05-13T23:14:59.240 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:59.062+0000 7f3a98753700 1 -- 172.21.15.2:0/2144668592 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3a900fa500 msgr2=0x7f3a90064980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:14:59.240 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:59.062+0000 7f3a98753700 1 --2- 172.21.15.2:0/2144668592 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3a900fa500 0x7f3a90064980 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f3a8c00e940 tx=0x7f3a8c00b040 comp rx=0 tx=0).stop 2022-05-13T23:14:59.241 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:59.062+0000 7f3a96cf0700 1 -- 172.21.15.2:0/2144668592 reap_dead start 2022-05-13T23:14:59.241 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:59.062+0000 7f3a98753700 1 -- 172.21.15.2:0/2144668592 shutdown_connections 2022-05-13T23:14:59.242 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:59.062+0000 7f3a98753700 1 -- 172.21.15.2:0/2144668592 >> 172.21.15.2:0/2144668592 conn(0x7f3a900f5cf0 msgr2=0x7f3a900f8070 unknown :-1 s=STATE_NONE l=0).mark_down 2022-05-13T23:14:59.242 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:59.062+0000 7f3a98753700 1 -- 172.21.15.2:0/2144668592 shutdown_connections 2022-05-13T23:14:59.242 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:14:59.062+0000 7f3a98753700 1 -- 172.21.15.2:0/2144668592 wait complete. 2022-05-13T23:14:59.243 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/16cf5f88-d311-11ec-be40-001a4aab830c/mon.a/config 2022-05-13T23:14:59.501 INFO:teuthology.orchestra.run.smithi002.stdout:141733920824 2022-05-13T23:14:59.501 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:23b0658391ce94ca40799469eec00c28b298816c shell --fsid 16cf5f88-d311-11ec-be40-001a4aab830c -- ceph osd last-stat-seq osd.4 2022-05-13T23:14:59.979 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/16cf5f88-d311-11ec-be40-001a4aab830c/mon.a/config 2022-05-13T23:15:00.641 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/16cf5f88-d311-11ec-be40-001a4aab830c/mon.a/config 2022-05-13T23:15:00.682 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:15:00 smithi002 ceph-mon[71816]: pgmap v310: 97 pgs: 97 active+clean; 584 KiB data, 77 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2022-05-13T23:15:00.683 INFO:teuthology.orchestra.run.smithi002.stdout:163208757299 2022-05-13T23:15:00.684 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:23b0658391ce94ca40799469eec00c28b298816c shell --fsid 16cf5f88-d311-11ec-be40-001a4aab830c -- ceph osd last-stat-seq osd.5 2022-05-13T23:15:00.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:15:00 smithi047 ceph-mon[77147]: pgmap v310: 97 pgs: 97 active+clean; 584 KiB data, 77 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2022-05-13T23:15:00.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:15:00 smithi115 ceph-mon[78593]: pgmap v310: 97 pgs: 97 active+clean; 584 KiB data, 77 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2022-05-13T23:15:01.296 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/16cf5f88-d311-11ec-be40-001a4aab830c/mon.a/config 2022-05-13T23:15:02.703 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:15:02 smithi002 ceph-mon[71816]: pgmap v311: 97 pgs: 97 active+clean; 584 KiB data, 77 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2022-05-13T23:15:02.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:15:02 smithi047 ceph-mon[77147]: pgmap v311: 97 pgs: 97 active+clean; 584 KiB data, 77 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2022-05-13T23:15:02.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:15:02 smithi115 ceph-mon[78593]: pgmap v311: 97 pgs: 97 active+clean; 584 KiB data, 77 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2022-05-13T23:15:02.976 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:02.428+0000 7f8ab65c9700 1 -- 172.21.15.2:0/2318991566 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f8ab0064b80 msgr2=0x7f8ab00fe000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:15:02.976 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:02.428+0000 7f8ab65c9700 1 --2- 172.21.15.2:0/2318991566 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f8ab0064b80 0x7f8ab00fe000 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f8aac004870 tx=0x7f8aac009cb0 comp rx=0 tx=0).stop 2022-05-13T23:15:02.977 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:02.428+0000 7f8ab65c9700 1 -- 172.21.15.2:0/2318991566 shutdown_connections 2022-05-13T23:15:02.977 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:02.428+0000 7f8ab65c9700 1 --2- 172.21.15.2:0/2318991566 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8ab0101080 0x7f8ab0103500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:02.977 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:02.428+0000 7f8ab65c9700 1 --2- 172.21.15.2:0/2318991566 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f8ab0064b80 0x7f8ab00fe000 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:02.978 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:02.428+0000 7f8ab65c9700 1 --2- 172.21.15.2:0/2318991566 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8ab00fe5d0 0x7f8ab01009f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:02.978 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:02.428+0000 7f8ab65c9700 1 -- 172.21.15.2:0/2318991566 >> 172.21.15.2:0/2318991566 conn(0x7f8ab00f5a30 msgr2=0x7f8ab00f7e80 unknown :-1 s=STATE_NONE l=0).mark_down 2022-05-13T23:15:02.978 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:02.429+0000 7f8ab65c9700 1 -- 172.21.15.2:0/2318991566 shutdown_connections 2022-05-13T23:15:02.978 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:02.429+0000 7f8ab65c9700 1 -- 172.21.15.2:0/2318991566 wait complete. 2022-05-13T23:15:02.979 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:02.430+0000 7f8ab65c9700 1 Processor -- start 2022-05-13T23:15:02.979 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:02.430+0000 7f8ab65c9700 1 -- start start 2022-05-13T23:15:02.979 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:02.430+0000 7f8ab65c9700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8ab0064b80 0x7f8ab010c250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:15:02.979 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:02.430+0000 7f8ab65c9700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f8ab00fe5d0 0x7f8ab010c790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:15:02.979 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:02.430+0000 7f8ab65c9700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8ab0101080 0x7f8ab0113820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:15:02.980 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:02.430+0000 7f8ab65c9700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ab0113d60 con 0x7f8ab0101080 2022-05-13T23:15:02.980 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:02.430+0000 7f8ab65c9700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ab0113ed0 con 0x7f8ab0064b80 2022-05-13T23:15:02.980 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:02.430+0000 7f8ab65c9700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ab0114090 con 0x7f8ab00fe5d0 2022-05-13T23:15:02.980 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:02.430+0000 7f8ab55c7700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8ab0064b80 0x7f8ab010c250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:15:02.981 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:02.431+0000 7f8ab55c7700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8ab0064b80 0x7f8ab010c250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.2:53806/0 (socket says 172.21.15.2:53806) 2022-05-13T23:15:02.981 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:02.431+0000 7f8ab55c7700 1 -- 172.21.15.2:0/178962950 learned_addr learned my addr 172.21.15.2:0/178962950 (peer_addr_for_me v2:172.21.15.2:0/0) 2022-05-13T23:15:02.981 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:02.431+0000 7f8ab5dc8700 1 --2- 172.21.15.2:0/178962950 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8ab0101080 0x7f8ab0113820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:15:02.981 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:02.431+0000 7f8ab4dc6700 1 --2- 172.21.15.2:0/178962950 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f8ab00fe5d0 0x7f8ab010c790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:15:02.981 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:02.431+0000 7f8ab55c7700 1 -- 172.21.15.2:0/178962950 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f8ab00fe5d0 msgr2=0x7f8ab010c790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:15:02.983 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:02.431+0000 7f8ab55c7700 1 --2- 172.21.15.2:0/178962950 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f8ab00fe5d0 0x7f8ab010c790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:02.984 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:02.431+0000 7f8ab55c7700 1 -- 172.21.15.2:0/178962950 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8ab0101080 msgr2=0x7f8ab0113820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:15:02.984 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:02.431+0000 7f8ab55c7700 1 --2- 172.21.15.2:0/178962950 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8ab0101080 0x7f8ab0113820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:02.984 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:02.431+0000 7f8ab55c7700 1 -- 172.21.15.2:0/178962950 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8aac005040 con 0x7f8ab0064b80 2022-05-13T23:15:02.984 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:02.431+0000 7f8ab5dc8700 1 --2- 172.21.15.2:0/178962950 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8ab0101080 0x7f8ab0113820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2022-05-13T23:15:02.984 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:02.431+0000 7f8ab55c7700 1 --2- 172.21.15.2:0/178962950 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8ab0064b80 0x7f8ab010c250 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f8aac004710 tx=0x7f8aac009cb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-05-13T23:15:02.985 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:02.432+0000 7f8aa5ffb700 1 -- 172.21.15.2:0/178962950 <== mon.1 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8aac00e040 con 0x7f8ab0064b80 2022-05-13T23:15:02.985 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:02.432+0000 7f8ab65c9700 1 -- 172.21.15.2:0/178962950 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8ab0114310 con 0x7f8ab0064b80 2022-05-13T23:15:02.985 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:02.432+0000 7f8aa5ffb700 1 -- 172.21.15.2:0/178962950 <== mon.1 v2:172.21.15.47:3300/0 2 ==== config(26 keys) v1 ==== 1087+0+0 (secure 0 0 0) 0x7f8aac014e40 con 0x7f8ab0064b80 2022-05-13T23:15:02.985 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:02.432+0000 7f8aa5ffb700 1 -- 172.21.15.2:0/178962950 <== mon.1 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8aac01cb20 con 0x7f8ab0064b80 2022-05-13T23:15:02.986 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:02.432+0000 7f8ab65c9700 1 -- 172.21.15.2:0/178962950 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ab0114790 con 0x7f8ab0064b80 2022-05-13T23:15:02.986 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:02.433+0000 7f8ab65c9700 1 -- 172.21.15.2:0/178962950 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8ab0106c30 con 0x7f8ab0064b80 2022-05-13T23:15:02.986 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:02.435+0000 7f8aa5ffb700 1 -- 172.21.15.2:0/178962950 <== mon.1 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 16) v1 ==== 125136+0+0 (secure 0 0 0) 0x7f8aac04bbc0 con 0x7f8ab0064b80 2022-05-13T23:15:02.987 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:02.436+0000 7f8aa5ffb700 1 --2- 172.21.15.2:0/178962950 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f8a9c096ce0 0x7f8a9c099190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:15:02.987 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:02.436+0000 7f8aa5ffb700 1 -- 172.21.15.2:0/178962950 <== mon.1 v2:172.21.15.47:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 7975+0+0 (secure 0 0 0) 0x7f8aac0c06a0 con 0x7f8ab0064b80 2022-05-13T23:15:02.987 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:02.437+0000 7f8ab4dc6700 1 --2- 172.21.15.2:0/178962950 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f8a9c096ce0 0x7f8a9c099190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:15:02.987 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:02.437+0000 7f8ab4dc6700 1 --2- 172.21.15.2:0/178962950 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f8a9c096ce0 0x7f8a9c099190 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f8ab0064950 tx=0x7f8aa000b040 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-05-13T23:15:02.988 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:02.439+0000 7f8aa5ffb700 1 -- 172.21.15.2:0/178962950 <== mon.1 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+165215 (secure 0 0 0) 0x7f8aac08d9e0 con 0x7f8ab0064b80 2022-05-13T23:15:02.988 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:02.663+0000 7f8ab65c9700 1 -- 172.21.15.2:0/178962950 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7f8ab00f73e0 con 0x7f8ab0064b80 2022-05-13T23:15:02.988 INFO:teuthology.orchestra.run.smithi002.stdout:77309411396 2022-05-13T23:15:02.989 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:02.664+0000 7f8aa5ffb700 1 -- 172.21.15.2:0/178962950 <== mon.1 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f8aac091000 con 0x7f8ab0064b80 2022-05-13T23:15:02.989 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:02.667+0000 7f8ab65c9700 1 -- 172.21.15.2:0/178962950 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f8a9c096ce0 msgr2=0x7f8a9c099190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:15:02.990 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:02.667+0000 7f8ab65c9700 1 --2- 172.21.15.2:0/178962950 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f8a9c096ce0 0x7f8a9c099190 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f8ab0064950 tx=0x7f8aa000b040 comp rx=0 tx=0).stop 2022-05-13T23:15:02.990 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:02.667+0000 7f8ab65c9700 1 -- 172.21.15.2:0/178962950 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8ab0064b80 msgr2=0x7f8ab010c250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:15:02.990 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:02.667+0000 7f8ab65c9700 1 --2- 172.21.15.2:0/178962950 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8ab0064b80 0x7f8ab010c250 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f8aac004710 tx=0x7f8aac009cb0 comp rx=0 tx=0).stop 2022-05-13T23:15:02.991 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:02.667+0000 7f8ab65c9700 1 -- 172.21.15.2:0/178962950 shutdown_connections 2022-05-13T23:15:02.991 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:02.667+0000 7f8ab65c9700 1 --2- 172.21.15.2:0/178962950 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f8a9c096ce0 0x7f8a9c099190 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:02.991 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:02.667+0000 7f8ab65c9700 1 --2- 172.21.15.2:0/178962950 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8ab0101080 0x7f8ab0113820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:02.991 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:02.667+0000 7f8ab65c9700 1 --2- 172.21.15.2:0/178962950 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8ab0064b80 0x7f8ab010c250 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:02.992 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:02.667+0000 7f8ab65c9700 1 --2- 172.21.15.2:0/178962950 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f8ab00fe5d0 0x7f8ab010c790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:02.992 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:02.667+0000 7f8ab65c9700 1 -- 172.21.15.2:0/178962950 >> 172.21.15.2:0/178962950 conn(0x7f8ab00f5a30 msgr2=0x7f8ab00fc860 unknown :-1 s=STATE_NONE l=0).mark_down 2022-05-13T23:15:02.992 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:02.667+0000 7f8ab65c9700 1 -- 172.21.15.2:0/178962950 shutdown_connections 2022-05-13T23:15:02.992 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:02.667+0000 7f8ab65c9700 1 -- 172.21.15.2:0/178962950 wait complete. 2022-05-13T23:15:03.517 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:01.451+0000 7f577e2ba700 1 -- 172.21.15.2:0/889228530 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f57780fc890 msgr2=0x7f57780fcce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:15:03.517 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:01.451+0000 7f577e2ba700 1 --2- 172.21.15.2:0/889228530 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f57780fc890 0x7f57780fcce0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f576c004870 tx=0x7f576c009cb0 comp rx=0 tx=0).stop 2022-05-13T23:15:03.517 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:01.451+0000 7f577e2ba700 1 -- 172.21.15.2:0/889228530 shutdown_connections 2022-05-13T23:15:03.518 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:01.451+0000 7f577e2ba700 1 --2- 172.21.15.2:0/889228530 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f57780fd220 0x7f5778103ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:03.518 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:01.451+0000 7f577e2ba700 1 --2- 172.21.15.2:0/889228530 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f57780fb5f0 0x7f57780fba00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:03.518 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:01.451+0000 7f577e2ba700 1 --2- 172.21.15.2:0/889228530 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f57780fc890 0x7f57780fcce0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:03.518 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:01.451+0000 7f577e2ba700 1 -- 172.21.15.2:0/889228530 >> 172.21.15.2:0/889228530 conn(0x7f57780f6e00 msgr2=0x7f57780f9210 unknown :-1 s=STATE_NONE l=0).mark_down 2022-05-13T23:15:03.519 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:01.451+0000 7f577e2ba700 1 -- 172.21.15.2:0/889228530 shutdown_connections 2022-05-13T23:15:03.519 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:01.451+0000 7f577e2ba700 1 -- 172.21.15.2:0/889228530 wait complete. 2022-05-13T23:15:03.519 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:01.452+0000 7f577e2ba700 1 Processor -- start 2022-05-13T23:15:03.520 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:01.453+0000 7f577e2ba700 1 -- start start 2022-05-13T23:15:03.520 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:01.453+0000 7f577e2ba700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f57780fb5f0 0x7f577810c3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:15:03.520 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:01.453+0000 7f577e2ba700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f57780fc890 0x7f577810c920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:15:03.520 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:01.453+0000 7f577e2ba700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f57780fd220 0x7f57781139b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:15:03.521 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:01.453+0000 7f577e2ba700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5778113ef0 con 0x7f57780fb5f0 2022-05-13T23:15:03.521 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:01.453+0000 7f577e2ba700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5778114060 con 0x7f57780fc890 2022-05-13T23:15:03.521 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:01.453+0000 7f577e2ba700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5778114220 con 0x7f57780fd220 2022-05-13T23:15:03.521 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:01.453+0000 7f5777fff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f57780fb5f0 0x7f577810c3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:15:03.522 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:01.454+0000 7f5777fff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f57780fb5f0 0x7f577810c3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:48262/0 (socket says 172.21.15.2:48262) 2022-05-13T23:15:03.522 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:01.454+0000 7f5777fff700 1 -- 172.21.15.2:0/61228779 learned_addr learned my addr 172.21.15.2:0/61228779 (peer_addr_for_me v2:172.21.15.2:0/0) 2022-05-13T23:15:03.522 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:01.454+0000 7f57777fe700 1 --2- 172.21.15.2:0/61228779 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f57780fc890 0x7f577810c920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:15:03.523 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:01.454+0000 7f577c857700 1 --2- 172.21.15.2:0/61228779 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f57780fd220 0x7f57781139b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:15:03.523 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:01.454+0000 7f5777fff700 1 -- 172.21.15.2:0/61228779 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f57780fd220 msgr2=0x7f57781139b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:15:03.523 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:01.454+0000 7f5777fff700 1 --2- 172.21.15.2:0/61228779 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f57780fd220 0x7f57781139b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:03.524 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:01.454+0000 7f5777fff700 1 -- 172.21.15.2:0/61228779 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f57780fc890 msgr2=0x7f577810c920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:15:03.524 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:01.454+0000 7f5777fff700 1 --2- 172.21.15.2:0/61228779 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f57780fc890 0x7f577810c920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:03.524 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:01.454+0000 7f5777fff700 1 -- 172.21.15.2:0/61228779 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f576c005040 con 0x7f57780fb5f0 2022-05-13T23:15:03.524 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:01.454+0000 7f57777fe700 1 --2- 172.21.15.2:0/61228779 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f57780fc890 0x7f577810c920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-05-13T23:15:03.524 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:01.455+0000 7f5777fff700 1 --2- 172.21.15.2:0/61228779 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f57780fb5f0 0x7f577810c3e0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f576800e940 tx=0x7f576800b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-05-13T23:15:03.525 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:01.455+0000 7f5774ff9700 1 -- 172.21.15.2:0/61228779 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5768007b40 con 0x7f57780fb5f0 2022-05-13T23:15:03.525 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:01.455+0000 7f577e2ba700 1 -- 172.21.15.2:0/61228779 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f57781144a0 con 0x7f57780fb5f0 2022-05-13T23:15:03.525 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:01.455+0000 7f5774ff9700 1 -- 172.21.15.2:0/61228779 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1087+0+0 (secure 0 0 0) 0x7f576800f040 con 0x7f57780fb5f0 2022-05-13T23:15:03.525 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:01.455+0000 7f5774ff9700 1 -- 172.21.15.2:0/61228779 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5768014890 con 0x7f57780fb5f0 2022-05-13T23:15:03.526 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:01.455+0000 7f577e2ba700 1 -- 172.21.15.2:0/61228779 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5778114a10 con 0x7f57780fb5f0 2022-05-13T23:15:03.526 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:01.456+0000 7f577e2ba700 1 -- 172.21.15.2:0/61228779 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f57780627a0 con 0x7f57780fb5f0 2022-05-13T23:15:03.526 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:01.458+0000 7f5774ff9700 1 -- 172.21.15.2:0/61228779 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 16) v1 ==== 125136+0+0 (secure 0 0 0) 0x7f576801b030 con 0x7f57780fb5f0 2022-05-13T23:15:03.526 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:01.460+0000 7f5774ff9700 1 --2- 172.21.15.2:0/61228779 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f5760096ce0 0x7f5760099190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:15:03.526 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:01.460+0000 7f5774ff9700 1 -- 172.21.15.2:0/61228779 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 7975+0+0 (secure 0 0 0) 0x7f57680bea20 con 0x7f57780fb5f0 2022-05-13T23:15:03.527 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:01.460+0000 7f57777fe700 1 --2- 172.21.15.2:0/61228779 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f5760096ce0 0x7f5760099190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:15:03.527 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:01.460+0000 7f5774ff9700 1 --2- 172.21.15.2:0/61228779 >> [v2:172.21.15.2:6826/3004162255,v1:172.21.15.2:6827/3004162255] conn(0x7f57600a04b0 0x7f57600a28c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:15:03.527 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:01.460+0000 7f5774ff9700 1 -- 172.21.15.2:0/61228779 --> [v2:172.21.15.2:6826/3004162255,v1:172.21.15.2:6827/3004162255] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f57600a2f70 con 0x7f57600a04b0 2022-05-13T23:15:03.527 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:01.460+0000 7f5774ff9700 1 -- 172.21.15.2:0/61228779 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_get_version_reply(handle=1 version=83) v2 ==== 24+0+0 (secure 0 0 0) 0x7f57680c3220 con 0x7f57780fb5f0 2022-05-13T23:15:03.528 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:01.461+0000 7f577c857700 1 --2- 172.21.15.2:0/61228779 >> [v2:172.21.15.2:6826/3004162255,v1:172.21.15.2:6827/3004162255] conn(0x7f57600a04b0 0x7f57600a28c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:15:03.528 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:01.461+0000 7f57777fe700 1 --2- 172.21.15.2:0/61228779 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f5760096ce0 0x7f5760099190 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f576c013490 tx=0x7f576c009cb0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-05-13T23:15:03.528 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:01.462+0000 7f577c857700 1 --2- 172.21.15.2:0/61228779 >> [v2:172.21.15.2:6826/3004162255,v1:172.21.15.2:6827/3004162255] conn(0x7f57600a04b0 0x7f57600a28c0 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.9 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-05-13T23:15:03.528 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:01.463+0000 7f5774ff9700 1 -- 172.21.15.2:0/61228779 <== osd.9 v2:172.21.15.2:6826/3004162255 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23747 (crc 0 0 0) 0x7f57600a2f70 con 0x7f57600a04b0 2022-05-13T23:15:03.528 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:01.486+0000 7f577e2ba700 1 -- 172.21.15.2:0/61228779 --> [v2:172.21.15.2:6826/3004162255,v1:172.21.15.2:6827/3004162255] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f5778101220 con 0x7f57600a04b0 2022-05-13T23:15:03.529 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:01.487+0000 7f5774ff9700 1 -- 172.21.15.2:0/61228779 <== osd.9 v2:172.21.15.2:6826/3004162255 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f5778101220 con 0x7f57600a04b0 2022-05-13T23:15:03.529 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:01.487+0000 7f577e2ba700 1 -- 172.21.15.2:0/61228779 >> [v2:172.21.15.2:6826/3004162255,v1:172.21.15.2:6827/3004162255] conn(0x7f57600a04b0 msgr2=0x7f57600a28c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:15:03.529 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:01.487+0000 7f577e2ba700 1 --2- 172.21.15.2:0/61228779 >> [v2:172.21.15.2:6826/3004162255,v1:172.21.15.2:6827/3004162255] conn(0x7f57600a04b0 0x7f57600a28c0 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:03.530 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:01.488+0000 7f577e2ba700 1 -- 172.21.15.2:0/61228779 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f5760096ce0 msgr2=0x7f5760099190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:15:03.530 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:01.488+0000 7f577e2ba700 1 --2- 172.21.15.2:0/61228779 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f5760096ce0 0x7f5760099190 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f576c013490 tx=0x7f576c009cb0 comp rx=0 tx=0).stop 2022-05-13T23:15:03.530 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:01.488+0000 7f577e2ba700 1 -- 172.21.15.2:0/61228779 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f57780fb5f0 msgr2=0x7f577810c3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:15:03.530 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:01.488+0000 7f577e2ba700 1 --2- 172.21.15.2:0/61228779 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f57780fb5f0 0x7f577810c3e0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f576800e940 tx=0x7f576800b040 comp rx=0 tx=0).stop 2022-05-13T23:15:03.531 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:01.488+0000 7f577c857700 1 -- 172.21.15.2:0/61228779 reap_dead start 2022-05-13T23:15:03.531 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:01.488+0000 7f577e2ba700 1 -- 172.21.15.2:0/61228779 shutdown_connections 2022-05-13T23:15:03.531 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:01.488+0000 7f577e2ba700 1 -- 172.21.15.2:0/61228779 >> 172.21.15.2:0/61228779 conn(0x7f57780f6e00 msgr2=0x7f57780f9190 unknown :-1 s=STATE_NONE l=0).mark_down 2022-05-13T23:15:03.532 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:01.488+0000 7f577e2ba700 1 -- 172.21.15.2:0/61228779 shutdown_connections 2022-05-13T23:15:03.532 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:01.488+0000 7f577e2ba700 1 -- 172.21.15.2:0/61228779 wait complete. 2022-05-13T23:15:03.707 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:15:03 smithi002 ceph-mon[71816]: from='client.? 172.21.15.2:0/178962950' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2022-05-13T23:15:03.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:15:03 smithi047 ceph-mon[77147]: from='client.? 172.21.15.2:0/178962950' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2022-05-13T23:15:03.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:15:03 smithi115 ceph-mon[78593]: from='client.? 172.21.15.2:0/178962950' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2022-05-13T23:15:04.501 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:15:04 smithi002 ceph-mon[71816]: pgmap v312: 97 pgs: 97 active+clean; 584 KiB data, 77 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2022-05-13T23:15:04.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:15:04 smithi047 ceph-mon[77147]: pgmap v312: 97 pgs: 97 active+clean; 584 KiB data, 77 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2022-05-13T23:15:04.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:15:04 smithi115 ceph-mon[78593]: pgmap v312: 97 pgs: 97 active+clean; 584 KiB data, 77 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2022-05-13T23:15:05.264 INFO:tasks.cephadm.ceph_manager.ceph:need seq 77309411393 got 77309411396 for osd.2 2022-05-13T23:15:05.265 DEBUG:teuthology.parallel:result is None 2022-05-13T23:15:05.406 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:15:05 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.f", "id": [8, 11]}]: dispatch 2022-05-13T23:15:05.407 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:15:05 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.13", "id": [2, 11]}]: dispatch 2022-05-13T23:15:05.407 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:15:05 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.10", "id": [5, 9]}]: dispatch 2022-05-13T23:15:05.407 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:15:05 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2c", "id": [4, 9]}]: dispatch 2022-05-13T23:15:05.408 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:15:05 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.36", "id": [4, 9]}]: dispatch 2022-05-13T23:15:05.408 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:15:05 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-05-13T23:15:05.408 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:15:05 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-05-13T23:15:05.437 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/16cf5f88-d311-11ec-be40-001a4aab830c/mon.a/config 2022-05-13T23:15:05.707 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:05.600+0000 7f2a4eeb1700 1 -- 172.21.15.2:0/552125998 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f2a48070370 msgr2=0x7f2a4806e870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:15:05.708 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:05.600+0000 7f2a4eeb1700 1 --2- 172.21.15.2:0/552125998 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f2a48070370 0x7f2a4806e870 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f2a3c004870 tx=0x7f2a3c009cb0 comp rx=0 tx=0).stop 2022-05-13T23:15:05.708 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:05.600+0000 7f2a4eeb1700 1 -- 172.21.15.2:0/552125998 shutdown_connections 2022-05-13T23:15:05.708 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:05.600+0000 7f2a4eeb1700 1 --2- 172.21.15.2:0/552125998 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f2a48103580 0x7f2a48105960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:05.709 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:05.600+0000 7f2a4eeb1700 1 --2- 172.21.15.2:0/552125998 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f2a48070370 0x7f2a4806e870 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:05.709 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:05.600+0000 7f2a4eeb1700 1 --2- 172.21.15.2:0/552125998 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2a4806edb0 0x7f2a4806f220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:05.709 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:05.600+0000 7f2a4eeb1700 1 -- 172.21.15.2:0/552125998 >> 172.21.15.2:0/552125998 conn(0x7f2a480f6e00 msgr2=0x7f2a480f9210 unknown :-1 s=STATE_NONE l=0).mark_down 2022-05-13T23:15:05.709 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:05.600+0000 7f2a4eeb1700 1 -- 172.21.15.2:0/552125998 shutdown_connections 2022-05-13T23:15:05.710 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:05.600+0000 7f2a4eeb1700 1 -- 172.21.15.2:0/552125998 wait complete. 2022-05-13T23:15:05.711 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:05.601+0000 7f2a4eeb1700 1 Processor -- start 2022-05-13T23:15:05.711 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:05.602+0000 7f2a4eeb1700 1 -- start start 2022-05-13T23:15:05.711 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:05.602+0000 7f2a4eeb1700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f2a4806edb0 0x7f2a4810e630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:15:05.712 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:05.602+0000 7f2a4eeb1700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2a48070370 0x7f2a4810eb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:15:05.712 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:05.602+0000 7f2a4eeb1700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f2a48103580 0x7f2a48115c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:15:05.712 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:05.602+0000 7f2a4eeb1700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a48116140 con 0x7f2a48070370 2022-05-13T23:15:05.712 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:05.602+0000 7f2a4eeb1700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a481162b0 con 0x7f2a4806edb0 2022-05-13T23:15:05.713 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:05.602+0000 7f2a4eeb1700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a48116470 con 0x7f2a48103580 2022-05-13T23:15:05.713 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:05.602+0000 7f2a47fff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2a48070370 0x7f2a4810eb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:15:05.713 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:05.602+0000 7f2a4cc4d700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f2a4806edb0 0x7f2a4810e630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:15:05.713 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:05.602+0000 7f2a4d44e700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f2a48103580 0x7f2a48115c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:15:05.714 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:05.602+0000 7f2a47fff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2a48070370 0x7f2a4810eb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:48296/0 (socket says 172.21.15.2:48296) 2022-05-13T23:15:05.714 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:05.602+0000 7f2a47fff700 1 -- 172.21.15.2:0/3419572207 learned_addr learned my addr 172.21.15.2:0/3419572207 (peer_addr_for_me v2:172.21.15.2:0/0) 2022-05-13T23:15:05.714 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:05.603+0000 7f2a4cc4d700 1 -- 172.21.15.2:0/3419572207 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f2a48103580 msgr2=0x7f2a48115c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:15:05.714 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:05.603+0000 7f2a4cc4d700 1 --2- 172.21.15.2:0/3419572207 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f2a48103580 0x7f2a48115c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:05.714 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:05.603+0000 7f2a4cc4d700 1 -- 172.21.15.2:0/3419572207 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2a48070370 msgr2=0x7f2a4810eb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:15:05.715 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:05.603+0000 7f2a4cc4d700 1 --2- 172.21.15.2:0/3419572207 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2a48070370 0x7f2a4810eb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:05.715 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:05.603+0000 7f2a4cc4d700 1 -- 172.21.15.2:0/3419572207 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a3c005040 con 0x7f2a4806edb0 2022-05-13T23:15:05.715 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:05.603+0000 7f2a47fff700 1 --2- 172.21.15.2:0/3419572207 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2a48070370 0x7f2a4810eb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-05-13T23:15:05.715 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:05.603+0000 7f2a4d44e700 1 --2- 172.21.15.2:0/3419572207 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f2a48103580 0x7f2a48115c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-05-13T23:15:05.715 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:05.603+0000 7f2a4cc4d700 1 --2- 172.21.15.2:0/3419572207 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f2a4806edb0 0x7f2a4810e630 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f2a3c004710 tx=0x7f2a3c009cb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-05-13T23:15:05.716 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:05.603+0000 7f2a457fa700 1 -- 172.21.15.2:0/3419572207 <== mon.1 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a3c020070 con 0x7f2a4806edb0 2022-05-13T23:15:05.716 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:05.604+0000 7f2a4eeb1700 1 -- 172.21.15.2:0/3419572207 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2a481166f0 con 0x7f2a4806edb0 2022-05-13T23:15:05.716 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:05.604+0000 7f2a457fa700 1 -- 172.21.15.2:0/3419572207 <== mon.1 v2:172.21.15.47:3300/0 2 ==== config(26 keys) v1 ==== 1087+0+0 (secure 0 0 0) 0x7f2a3c004200 con 0x7f2a4806edb0 2022-05-13T23:15:05.716 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:05.604+0000 7f2a457fa700 1 -- 172.21.15.2:0/3419572207 <== mon.1 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a3c00e040 con 0x7f2a4806edb0 2022-05-13T23:15:05.717 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:05.604+0000 7f2a4eeb1700 1 -- 172.21.15.2:0/3419572207 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2a48116bd0 con 0x7f2a4806edb0 2022-05-13T23:15:05.717 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:05.605+0000 7f2a4eeb1700 1 -- 172.21.15.2:0/3419572207 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2a48109020 con 0x7f2a4806edb0 2022-05-13T23:15:05.717 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:05.607+0000 7f2a457fa700 1 -- 172.21.15.2:0/3419572207 <== mon.1 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 16) v1 ==== 125136+0+0 (secure 0 0 0) 0x7f2a3c009240 con 0x7f2a4806edb0 2022-05-13T23:15:05.717 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:05.608+0000 7f2a457fa700 1 --2- 172.21.15.2:0/3419572207 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f2a38096ce0 0x7f2a38099190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:15:05.717 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:05.608+0000 7f2a457fa700 1 -- 172.21.15.2:0/3419572207 <== mon.1 v2:172.21.15.47:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 8120+0+0 (secure 0 0 0) 0x7f2a3c0c0900 con 0x7f2a4806edb0 2022-05-13T23:15:05.718 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:05.608+0000 7f2a47fff700 1 --2- 172.21.15.2:0/3419572207 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f2a38096ce0 0x7f2a38099190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:15:05.718 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:05.609+0000 7f2a47fff700 1 --2- 172.21.15.2:0/3419572207 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f2a38096ce0 0x7f2a38099190 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f2a34007d30 tx=0x7f2a3400b040 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-05-13T23:15:05.718 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:05.610+0000 7f2a457fa700 1 -- 172.21.15.2:0/3419572207 <== mon.1 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+165215 (secure 0 0 0) 0x7f2a3c08dbb0 con 0x7f2a4806edb0 2022-05-13T23:15:05.799 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:15:05 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.f", "id": [8, 11]}]: dispatch 2022-05-13T23:15:05.799 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:15:05 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.13", "id": [2, 11]}]: dispatch 2022-05-13T23:15:05.799 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:15:05 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.10", "id": [5, 9]}]: dispatch 2022-05-13T23:15:05.799 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:15:05 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2c", "id": [4, 9]}]: dispatch 2022-05-13T23:15:05.800 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:15:05 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.36", "id": [4, 9]}]: dispatch 2022-05-13T23:15:05.800 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:15:05 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-05-13T23:15:05.800 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:15:05 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-05-13T23:15:05.815 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:05.813+0000 7f2a4eeb1700 1 -- 172.21.15.2:0/3419572207 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 8} v 0) v1 -- 0x7f2a48101b50 con 0x7f2a4806edb0 2022-05-13T23:15:05.816 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:05.814+0000 7f2a457fa700 1 -- 172.21.15.2:0/3419572207 <== mon.1 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 8}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f2a3c0911d0 con 0x7f2a4806edb0 2022-05-13T23:15:05.816 INFO:teuthology.orchestra.run.smithi002.stdout:249108103207 2022-05-13T23:15:05.818 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:05.816+0000 7f2a4eeb1700 1 -- 172.21.15.2:0/3419572207 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f2a38096ce0 msgr2=0x7f2a38099190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:15:05.818 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:05.816+0000 7f2a4eeb1700 1 --2- 172.21.15.2:0/3419572207 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f2a38096ce0 0x7f2a38099190 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f2a34007d30 tx=0x7f2a3400b040 comp rx=0 tx=0).stop 2022-05-13T23:15:05.818 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:05.816+0000 7f2a4eeb1700 1 -- 172.21.15.2:0/3419572207 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f2a4806edb0 msgr2=0x7f2a4810e630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:15:05.819 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:05.816+0000 7f2a4eeb1700 1 --2- 172.21.15.2:0/3419572207 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f2a4806edb0 0x7f2a4810e630 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f2a3c004710 tx=0x7f2a3c009cb0 comp rx=0 tx=0).stop 2022-05-13T23:15:05.819 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:05.816+0000 7f2a4eeb1700 1 -- 172.21.15.2:0/3419572207 shutdown_connections 2022-05-13T23:15:05.820 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:05.816+0000 7f2a4eeb1700 1 --2- 172.21.15.2:0/3419572207 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f2a48103580 0x7f2a48115c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:05.820 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:05.816+0000 7f2a4eeb1700 1 --2- 172.21.15.2:0/3419572207 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f2a4806edb0 0x7f2a4810e630 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:05.820 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:05.816+0000 7f2a4eeb1700 1 --2- 172.21.15.2:0/3419572207 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f2a38096ce0 0x7f2a38099190 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:05.821 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:05.816+0000 7f2a4eeb1700 1 --2- 172.21.15.2:0/3419572207 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2a48070370 0x7f2a4810eb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:05.821 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:05.816+0000 7f2a4eeb1700 1 -- 172.21.15.2:0/3419572207 >> 172.21.15.2:0/3419572207 conn(0x7f2a480f6e00 msgr2=0x7f2a480f91e0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-05-13T23:15:05.822 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:05.816+0000 7f2a4eeb1700 1 -- 172.21.15.2:0/3419572207 shutdown_connections 2022-05-13T23:15:05.822 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:05.816+0000 7f2a4eeb1700 1 -- 172.21.15.2:0/3419572207 wait complete. 2022-05-13T23:15:05.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:15:05 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.f", "id": [8, 11]}]: dispatch 2022-05-13T23:15:05.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:15:05 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.13", "id": [2, 11]}]: dispatch 2022-05-13T23:15:05.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:15:05 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.10", "id": [5, 9]}]: dispatch 2022-05-13T23:15:05.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:15:05 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2c", "id": [4, 9]}]: dispatch 2022-05-13T23:15:05.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:15:05 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.36", "id": [4, 9]}]: dispatch 2022-05-13T23:15:05.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:15:05 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-05-13T23:15:05.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:15:05 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-05-13T23:15:06.414 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:15:06 smithi002 ceph-mon[71816]: pgmap v313: 97 pgs: 97 active+clean; 584 KiB data, 77 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 9 op/s 2022-05-13T23:15:06.414 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:15:06 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.f", "id": [8, 11]}]': finished 2022-05-13T23:15:06.415 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:15:06 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.13", "id": [2, 11]}]': finished 2022-05-13T23:15:06.415 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:15:06 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.10", "id": [5, 9]}]': finished 2022-05-13T23:15:06.416 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:15:06 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2c", "id": [4, 9]}]': finished 2022-05-13T23:15:06.416 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:15:06 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.36", "id": [4, 9]}]': finished 2022-05-13T23:15:06.417 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:15:06 smithi002 ceph-mon[71816]: osdmap e84: 12 total, 12 up, 12 in 2022-05-13T23:15:06.417 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:15:06 smithi002 ceph-mon[71816]: from='client.? 172.21.15.2:0/3419572207' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2022-05-13T23:15:06.524 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:06.435+0000 7feef91f6700 1 -- 172.21.15.2:0/3719153686 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7feef4103520 msgr2=0x7feef4105900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:15:06.525 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:06.435+0000 7feef91f6700 1 --2- 172.21.15.2:0/3719153686 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7feef4103520 0x7feef4105900 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7feeec009be0 tx=0x7feeec004d10 comp rx=0 tx=0).stop 2022-05-13T23:15:06.525 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:06.436+0000 7feef91f6700 1 -- 172.21.15.2:0/3719153686 shutdown_connections 2022-05-13T23:15:06.525 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:06.436+0000 7feef91f6700 1 --2- 172.21.15.2:0/3719153686 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7feef4103520 0x7feef4105900 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:06.525 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:06.436+0000 7feef91f6700 1 --2- 172.21.15.2:0/3719153686 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7feef40703d0 0x7feef406e8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:06.526 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:06.436+0000 7feef91f6700 1 --2- 172.21.15.2:0/3719153686 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7feef406ee10 0x7feef406f280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:06.526 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:06.436+0000 7feef91f6700 1 -- 172.21.15.2:0/3719153686 >> 172.21.15.2:0/3719153686 conn(0x7feef40f6e00 msgr2=0x7feef40f9210 unknown :-1 s=STATE_NONE l=0).mark_down 2022-05-13T23:15:06.526 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:06.436+0000 7feef91f6700 1 -- 172.21.15.2:0/3719153686 shutdown_connections 2022-05-13T23:15:06.527 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:06.436+0000 7feef91f6700 1 -- 172.21.15.2:0/3719153686 wait complete. 2022-05-13T23:15:06.527 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:06.437+0000 7feef91f6700 1 Processor -- start 2022-05-13T23:15:06.527 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:06.437+0000 7feef91f6700 1 -- start start 2022-05-13T23:15:06.528 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:06.437+0000 7feef91f6700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7feef406ee10 0x7feef410e510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:15:06.528 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:06.437+0000 7feef91f6700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7feef40703d0 0x7feef410ea50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:15:06.528 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:06.437+0000 7feef91f6700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7feef4103520 0x7feef4115ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:15:06.529 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:06.437+0000 7feef91f6700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feef4116020 con 0x7feef40703d0 2022-05-13T23:15:06.529 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:06.437+0000 7feef91f6700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feef4116190 con 0x7feef406ee10 2022-05-13T23:15:06.529 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:06.437+0000 7feef91f6700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feef4116350 con 0x7feef4103520 2022-05-13T23:15:06.530 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:06.437+0000 7feef2d9d700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7feef406ee10 0x7feef410e510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:15:06.530 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:06.437+0000 7feef2d9d700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7feef406ee10 0x7feef410e510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.2:53836/0 (socket says 172.21.15.2:53836) 2022-05-13T23:15:06.530 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:06.437+0000 7feef259c700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7feef40703d0 0x7feef410ea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:15:06.530 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:06.437+0000 7feef2d9d700 1 -- 172.21.15.2:0/243074291 learned_addr learned my addr 172.21.15.2:0/243074291 (peer_addr_for_me v2:172.21.15.2:0/0) 2022-05-13T23:15:06.531 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:06.437+0000 7feef359e700 1 --2- 172.21.15.2:0/243074291 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7feef4103520 0x7feef4115ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:15:06.531 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:06.438+0000 7feef2d9d700 1 -- 172.21.15.2:0/243074291 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7feef4103520 msgr2=0x7feef4115ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:15:06.531 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:06.438+0000 7feef2d9d700 1 --2- 172.21.15.2:0/243074291 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7feef4103520 0x7feef4115ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:06.531 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:06.438+0000 7feef2d9d700 1 -- 172.21.15.2:0/243074291 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7feef40703d0 msgr2=0x7feef410ea50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:15:06.532 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:06.438+0000 7feef2d9d700 1 --2- 172.21.15.2:0/243074291 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7feef40703d0 0x7feef410ea50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:06.532 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:06.438+0000 7feef2d9d700 1 -- 172.21.15.2:0/243074291 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feeec0097c0 con 0x7feef406ee10 2022-05-13T23:15:06.533 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:06.438+0000 7feef359e700 1 --2- 172.21.15.2:0/243074291 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7feef4103520 0x7feef4115ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-05-13T23:15:06.533 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:06.438+0000 7feef2d9d700 1 --2- 172.21.15.2:0/243074291 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7feef406ee10 0x7feef410e510 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7feee400e970 tx=0x7feee400b040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-05-13T23:15:06.533 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:06.438+0000 7feedb7fe700 1 -- 172.21.15.2:0/243074291 <== mon.1 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feee4007b40 con 0x7feef406ee10 2022-05-13T23:15:06.533 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:06.438+0000 7feedb7fe700 1 -- 172.21.15.2:0/243074291 <== mon.1 v2:172.21.15.47:3300/0 2 ==== config(26 keys) v1 ==== 1087+0+0 (secure 0 0 0) 0x7feee400f040 con 0x7feef406ee10 2022-05-13T23:15:06.534 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:06.438+0000 7feedb7fe700 1 -- 172.21.15.2:0/243074291 <== mon.1 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feee4014950 con 0x7feef406ee10 2022-05-13T23:15:06.534 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:06.438+0000 7feef91f6700 1 -- 172.21.15.2:0/243074291 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feef4116630 con 0x7feef406ee10 2022-05-13T23:15:06.534 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:06.439+0000 7feef91f6700 1 -- 172.21.15.2:0/243074291 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feef4116ad0 con 0x7feef406ee10 2022-05-13T23:15:06.534 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:06.440+0000 7feef91f6700 1 -- 172.21.15.2:0/243074291 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feef4108fd0 con 0x7feef406ee10 2022-05-13T23:15:06.535 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:06.442+0000 7feedb7fe700 1 -- 172.21.15.2:0/243074291 <== mon.1 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 16) v1 ==== 125136+0+0 (secure 0 0 0) 0x7feee401b030 con 0x7feef406ee10 2022-05-13T23:15:06.535 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:06.443+0000 7feedb7fe700 1 --2- 172.21.15.2:0/243074291 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7feedc096ce0 0x7feedc099190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:15:06.535 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:06.444+0000 7feedb7fe700 1 -- 172.21.15.2:0/243074291 <== mon.1 v2:172.21.15.47:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8120+0+0 (secure 0 0 0) 0x7feee40bece0 con 0x7feef406ee10 2022-05-13T23:15:06.535 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:06.444+0000 7feef259c700 1 --2- 172.21.15.2:0/243074291 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7feedc096ce0 0x7feedc099190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:15:06.536 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:06.445+0000 7feef259c700 1 --2- 172.21.15.2:0/243074291 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7feedc096ce0 0x7feedc099190 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7feee80048b0 tx=0x7feee800b040 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-05-13T23:15:06.536 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:06.448+0000 7feedb7fe700 1 -- 172.21.15.2:0/243074291 <== mon.1 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+165215 (secure 0 0 0) 0x7feee408bf90 con 0x7feef406ee10 2022-05-13T23:15:06.647 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:06.646+0000 7feef91f6700 1 -- 172.21.15.2:0/243074291 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7feef4102200 con 0x7feef406ee10 2022-05-13T23:15:06.648 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:06.647+0000 7feedb7fe700 1 -- 172.21.15.2:0/243074291 <== mon.1 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7feee408f5b0 con 0x7feef406ee10 2022-05-13T23:15:06.649 INFO:teuthology.orchestra.run.smithi002.stdout:34359738446 2022-05-13T23:15:06.652 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:06.649+0000 7feef91f6700 1 -- 172.21.15.2:0/243074291 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7feedc096ce0 msgr2=0x7feedc099190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:15:06.652 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:06.649+0000 7feef91f6700 1 --2- 172.21.15.2:0/243074291 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7feedc096ce0 0x7feedc099190 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7feee80048b0 tx=0x7feee800b040 comp rx=0 tx=0).stop 2022-05-13T23:15:06.652 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:06.650+0000 7feef91f6700 1 -- 172.21.15.2:0/243074291 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7feef406ee10 msgr2=0x7feef410e510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:15:06.652 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:06.650+0000 7feef91f6700 1 --2- 172.21.15.2:0/243074291 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7feef406ee10 0x7feef410e510 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7feee400e970 tx=0x7feee400b040 comp rx=0 tx=0).stop 2022-05-13T23:15:06.653 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:06.650+0000 7feef91f6700 1 -- 172.21.15.2:0/243074291 shutdown_connections 2022-05-13T23:15:06.653 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:06.650+0000 7feef91f6700 1 --2- 172.21.15.2:0/243074291 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7feef4103520 0x7feef4115ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:06.653 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:06.650+0000 7feef91f6700 1 --2- 172.21.15.2:0/243074291 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7feef406ee10 0x7feef410e510 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:06.653 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:06.650+0000 7feef91f6700 1 --2- 172.21.15.2:0/243074291 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7feedc096ce0 0x7feedc099190 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:06.655 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:06.650+0000 7feef91f6700 1 --2- 172.21.15.2:0/243074291 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7feef40703d0 0x7feef410ea50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:06.655 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:06.650+0000 7feef91f6700 1 -- 172.21.15.2:0/243074291 >> 172.21.15.2:0/243074291 conn(0x7feef40f6e00 msgr2=0x7feef40ff500 unknown :-1 s=STATE_NONE l=0).mark_down 2022-05-13T23:15:06.655 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:06.650+0000 7feef91f6700 1 -- 172.21.15.2:0/243074291 shutdown_connections 2022-05-13T23:15:06.655 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:06.650+0000 7feef91f6700 1 -- 172.21.15.2:0/243074291 wait complete. 2022-05-13T23:15:06.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:15:06 smithi047 ceph-mon[77147]: pgmap v313: 97 pgs: 97 active+clean; 584 KiB data, 77 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 9 op/s 2022-05-13T23:15:06.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:15:06 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.f", "id": [8, 11]}]': finished 2022-05-13T23:15:06.795 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:15:06 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.13", "id": [2, 11]}]': finished 2022-05-13T23:15:06.795 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:15:06 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.10", "id": [5, 9]}]': finished 2022-05-13T23:15:06.795 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:15:06 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2c", "id": [4, 9]}]': finished 2022-05-13T23:15:06.795 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:15:06 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.36", "id": [4, 9]}]': finished 2022-05-13T23:15:06.796 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:15:06 smithi047 ceph-mon[77147]: osdmap e84: 12 total, 12 up, 12 in 2022-05-13T23:15:06.796 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:15:06 smithi047 ceph-mon[77147]: from='client.? 172.21.15.2:0/3419572207' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2022-05-13T23:15:06.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:15:06 smithi115 ceph-mon[78593]: pgmap v313: 97 pgs: 97 active+clean; 584 KiB data, 77 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 9 op/s 2022-05-13T23:15:06.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:15:06 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.f", "id": [8, 11]}]': finished 2022-05-13T23:15:06.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:15:06 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.13", "id": [2, 11]}]': finished 2022-05-13T23:15:06.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:15:06 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.10", "id": [5, 9]}]': finished 2022-05-13T23:15:06.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:15:06 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2c", "id": [4, 9]}]': finished 2022-05-13T23:15:06.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:15:06 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.36", "id": [4, 9]}]': finished 2022-05-13T23:15:06.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:15:06 smithi115 ceph-mon[78593]: osdmap e84: 12 total, 12 up, 12 in 2022-05-13T23:15:06.862 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:15:06 smithi115 ceph-mon[78593]: from='client.? 172.21.15.2:0/3419572207' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2022-05-13T23:15:06.928 INFO:tasks.cephadm.ceph_manager.ceph:need seq 249108103205 got 249108103207 for osd.8 2022-05-13T23:15:06.928 DEBUG:teuthology.parallel:result is None 2022-05-13T23:15:07.416 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:15:07 smithi002 ceph-mon[71816]: osdmap e85: 12 total, 12 up, 12 in 2022-05-13T23:15:07.417 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:15:07 smithi002 ceph-mon[71816]: from='client.? 172.21.15.2:0/243074291' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2022-05-13T23:15:07.441 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:07.372+0000 7f35ce040700 1 -- 172.21.15.2:0/1067078196 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f35c80703d0 msgr2=0x7f35c806e8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:15:07.441 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:07.372+0000 7f35ce040700 1 --2- 172.21.15.2:0/1067078196 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f35c80703d0 0x7f35c806e8d0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f35c4004870 tx=0x7f35c4009cb0 comp rx=0 tx=0).stop 2022-05-13T23:15:07.441 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:07.372+0000 7f35ce040700 1 -- 172.21.15.2:0/1067078196 shutdown_connections 2022-05-13T23:15:07.442 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:07.372+0000 7f35ce040700 1 --2- 172.21.15.2:0/1067078196 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f35c8103340 0x7f35c8105720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:07.442 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:07.372+0000 7f35ce040700 1 --2- 172.21.15.2:0/1067078196 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f35c80703d0 0x7f35c806e8d0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:07.442 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:07.372+0000 7f35ce040700 1 --2- 172.21.15.2:0/1067078196 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f35c806ee10 0x7f35c806f280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:07.443 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:07.372+0000 7f35ce040700 1 -- 172.21.15.2:0/1067078196 >> 172.21.15.2:0/1067078196 conn(0x7f35c80f6b40 msgr2=0x7f35c80f8f90 unknown :-1 s=STATE_NONE l=0).mark_down 2022-05-13T23:15:07.443 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:07.372+0000 7f35ce040700 1 -- 172.21.15.2:0/1067078196 shutdown_connections 2022-05-13T23:15:07.443 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:07.372+0000 7f35ce040700 1 -- 172.21.15.2:0/1067078196 wait complete. 2022-05-13T23:15:07.443 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:07.373+0000 7f35ce040700 1 Processor -- start 2022-05-13T23:15:07.444 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:07.373+0000 7f35ce040700 1 -- start start 2022-05-13T23:15:07.444 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:07.374+0000 7f35ce040700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f35c806ee10 0x7f35c80fe7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:15:07.444 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:07.374+0000 7f35ce040700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f35c80703d0 0x7f35c80fed20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:15:07.444 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:07.374+0000 7f35ce040700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f35c8103340 0x7f35c80fb320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:15:07.444 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:07.374+0000 7f35ce040700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35c80fba30 con 0x7f35c8103340 2022-05-13T23:15:07.445 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:07.374+0000 7f35ce040700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35c80fbba0 con 0x7f35c80703d0 2022-05-13T23:15:07.445 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:07.374+0000 7f35ce040700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35c80fbd60 con 0x7f35c806ee10 2022-05-13T23:15:07.446 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:07.374+0000 7f35cc83d700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f35c80703d0 0x7f35c80fed20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:15:07.446 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:07.374+0000 7f35cd83f700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f35c8103340 0x7f35c80fb320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:15:07.446 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:07.374+0000 7f35cc83d700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f35c80703d0 0x7f35c80fed20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.2:53840/0 (socket says 172.21.15.2:53840) 2022-05-13T23:15:07.446 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:07.374+0000 7f35cc83d700 1 -- 172.21.15.2:0/682573836 learned_addr learned my addr 172.21.15.2:0/682573836 (peer_addr_for_me v2:172.21.15.2:0/0) 2022-05-13T23:15:07.447 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:07.374+0000 7f35cd03e700 1 --2- 172.21.15.2:0/682573836 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f35c806ee10 0x7f35c80fe7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:15:07.447 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:07.374+0000 7f35cc83d700 1 -- 172.21.15.2:0/682573836 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f35c806ee10 msgr2=0x7f35c80fe7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:15:07.447 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:07.374+0000 7f35cc83d700 1 --2- 172.21.15.2:0/682573836 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f35c806ee10 0x7f35c80fe7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:07.447 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:07.374+0000 7f35cc83d700 1 -- 172.21.15.2:0/682573836 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f35c8103340 msgr2=0x7f35c80fb320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:15:07.447 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:07.374+0000 7f35cc83d700 1 --2- 172.21.15.2:0/682573836 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f35c8103340 0x7f35c80fb320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:07.448 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:07.374+0000 7f35cc83d700 1 -- 172.21.15.2:0/682573836 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35c4005040 con 0x7f35c80703d0 2022-05-13T23:15:07.448 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:07.375+0000 7f35cc83d700 1 --2- 172.21.15.2:0/682573836 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f35c80703d0 0x7f35c80fed20 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f35b800e970 tx=0x7f35b800b040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-05-13T23:15:07.448 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:07.375+0000 7f35bdffb700 1 -- 172.21.15.2:0/682573836 <== mon.1 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35b8007b40 con 0x7f35c80703d0 2022-05-13T23:15:07.448 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:07.375+0000 7f35ce040700 1 -- 172.21.15.2:0/682573836 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f35c811a9b0 con 0x7f35c80703d0 2022-05-13T23:15:07.449 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:07.375+0000 7f35bdffb700 1 -- 172.21.15.2:0/682573836 <== mon.1 v2:172.21.15.47:3300/0 2 ==== config(26 keys) v1 ==== 1087+0+0 (secure 0 0 0) 0x7f35b800f040 con 0x7f35c80703d0 2022-05-13T23:15:07.449 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:07.375+0000 7f35bdffb700 1 -- 172.21.15.2:0/682573836 <== mon.1 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35b8014950 con 0x7f35c80703d0 2022-05-13T23:15:07.449 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:07.375+0000 7f35ce040700 1 -- 172.21.15.2:0/682573836 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f35c811af70 con 0x7f35c80703d0 2022-05-13T23:15:07.449 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:07.376+0000 7f35ce040700 1 -- 172.21.15.2:0/682573836 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f35c8108e40 con 0x7f35c80703d0 2022-05-13T23:15:07.449 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:07.379+0000 7f35bdffb700 1 -- 172.21.15.2:0/682573836 <== mon.1 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 16) v1 ==== 125136+0+0 (secure 0 0 0) 0x7f35b801b030 con 0x7f35c80703d0 2022-05-13T23:15:07.450 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:07.380+0000 7f35bdffb700 1 --2- 172.21.15.2:0/682573836 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f35b4096ce0 0x7f35b4099190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:15:07.450 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:07.380+0000 7f35bdffb700 1 -- 172.21.15.2:0/682573836 <== mon.1 v2:172.21.15.47:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8120+0+0 (secure 0 0 0) 0x7f35b80beda0 con 0x7f35c80703d0 2022-05-13T23:15:07.450 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:07.380+0000 7f35cd03e700 1 --2- 172.21.15.2:0/682573836 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f35b4096ce0 0x7f35b4099190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:15:07.450 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:07.381+0000 7f35cd03e700 1 --2- 172.21.15.2:0/682573836 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f35b4096ce0 0x7f35b4099190 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f35c4004ee0 tx=0x7f35c4009cb0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-05-13T23:15:07.451 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:07.382+0000 7f35bdffb700 1 -- 172.21.15.2:0/682573836 <== mon.1 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+165215 (secure 0 0 0) 0x7f35b808c0e0 con 0x7f35c80703d0 2022-05-13T23:15:07.639 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:07.636+0000 7f35ce040700 1 -- 172.21.15.2:0/682573836 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) v1 -- 0x7f35c8101910 con 0x7f35c80703d0 2022-05-13T23:15:07.639 INFO:teuthology.orchestra.run.smithi002.stdout:219043332138 2022-05-13T23:15:07.640 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:07.637+0000 7f35bdffb700 1 -- 172.21.15.2:0/682573836 <== mon.1 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f35b8019090 con 0x7f35c80703d0 2022-05-13T23:15:07.641 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:07.640+0000 7f35ce040700 1 -- 172.21.15.2:0/682573836 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f35b4096ce0 msgr2=0x7f35b4099190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:15:07.641 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:07.640+0000 7f35ce040700 1 --2- 172.21.15.2:0/682573836 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f35b4096ce0 0x7f35b4099190 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f35c4004ee0 tx=0x7f35c4009cb0 comp rx=0 tx=0).stop 2022-05-13T23:15:07.642 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:07.640+0000 7f35ce040700 1 -- 172.21.15.2:0/682573836 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f35c80703d0 msgr2=0x7f35c80fed20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:15:07.642 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:07.640+0000 7f35ce040700 1 --2- 172.21.15.2:0/682573836 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f35c80703d0 0x7f35c80fed20 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f35b800e970 tx=0x7f35b800b040 comp rx=0 tx=0).stop 2022-05-13T23:15:07.642 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:07.640+0000 7f35ce040700 1 -- 172.21.15.2:0/682573836 shutdown_connections 2022-05-13T23:15:07.642 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:07.640+0000 7f35ce040700 1 --2- 172.21.15.2:0/682573836 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f35b4096ce0 0x7f35b4099190 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:07.643 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:07.640+0000 7f35ce040700 1 --2- 172.21.15.2:0/682573836 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f35c8103340 0x7f35c80fb320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:07.643 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:07.640+0000 7f35ce040700 1 --2- 172.21.15.2:0/682573836 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f35c806ee10 0x7f35c80fe7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:07.643 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:07.640+0000 7f35ce040700 1 --2- 172.21.15.2:0/682573836 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f35c80703d0 0x7f35c80fed20 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:07.643 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:07.640+0000 7f35ce040700 1 -- 172.21.15.2:0/682573836 >> 172.21.15.2:0/682573836 conn(0x7f35c80f6b40 msgr2=0x7f35c80f8f20 unknown :-1 s=STATE_NONE l=0).mark_down 2022-05-13T23:15:07.644 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:07.640+0000 7f35ce040700 1 -- 172.21.15.2:0/682573836 shutdown_connections 2022-05-13T23:15:07.644 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:07.640+0000 7f35ce040700 1 -- 172.21.15.2:0/682573836 wait complete. 2022-05-13T23:15:07.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:15:07 smithi047 ceph-mon[77147]: osdmap e85: 12 total, 12 up, 12 in 2022-05-13T23:15:07.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:15:07 smithi047 ceph-mon[77147]: from='client.? 172.21.15.2:0/243074291' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2022-05-13T23:15:07.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:15:07 smithi115 ceph-mon[78593]: osdmap e85: 12 total, 12 up, 12 in 2022-05-13T23:15:07.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:15:07 smithi115 ceph-mon[78593]: from='client.? 172.21.15.2:0/243074291' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2022-05-13T23:15:08.487 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:15:08 smithi002 ceph-mon[71816]: pgmap v316: 97 pgs: 97 active+clean; 584 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s; 11 B/s, 1 keys/s, 0 objects/s recovering 2022-05-13T23:15:08.487 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:15:08 smithi002 ceph-mon[71816]: from='client.? 172.21.15.2:0/682573836' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2022-05-13T23:15:08.708 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738444 got 34359738446 for osd.0 2022-05-13T23:15:08.708 DEBUG:teuthology.parallel:result is None 2022-05-13T23:15:08.760 INFO:tasks.cephadm.ceph_manager.ceph:need seq 219043332135 got 219043332138 for osd.7 2022-05-13T23:15:08.760 DEBUG:teuthology.parallel:result is None 2022-05-13T23:15:08.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:15:08 smithi047 ceph-mon[77147]: pgmap v316: 97 pgs: 97 active+clean; 584 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s; 11 B/s, 1 keys/s, 0 objects/s recovering 2022-05-13T23:15:08.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:15:08 smithi047 ceph-mon[77147]: from='client.? 172.21.15.2:0/682573836' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2022-05-13T23:15:08.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:15:08 smithi115 ceph-mon[78593]: pgmap v316: 97 pgs: 97 active+clean; 584 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s; 11 B/s, 1 keys/s, 0 objects/s recovering 2022-05-13T23:15:08.862 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:15:08 smithi115 ceph-mon[78593]: from='client.? 172.21.15.2:0/682573836' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2022-05-13T23:15:08.971 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:08.585+0000 7fbc9d82d700 1 -- 172.21.15.2:0/1257814714 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fbc980fd220 msgr2=0x7fbc98103ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:15:08.972 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:08.585+0000 7fbc9d82d700 1 --2- 172.21.15.2:0/1257814714 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fbc980fd220 0x7fbc98103ae0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fbc90009c30 tx=0x7fbc90004c50 comp rx=0 tx=0).stop 2022-05-13T23:15:08.972 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:08.585+0000 7fbc9d82d700 1 -- 172.21.15.2:0/1257814714 shutdown_connections 2022-05-13T23:15:08.972 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:08.585+0000 7fbc9d82d700 1 --2- 172.21.15.2:0/1257814714 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fbc980fd220 0x7fbc98103ae0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:08.973 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:08.585+0000 7fbc9d82d700 1 --2- 172.21.15.2:0/1257814714 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fbc980fb5f0 0x7fbc980fba00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:08.973 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:08.585+0000 7fbc9d82d700 1 --2- 172.21.15.2:0/1257814714 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fbc980fc890 0x7fbc980fcce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:08.973 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:08.585+0000 7fbc9d82d700 1 -- 172.21.15.2:0/1257814714 >> 172.21.15.2:0/1257814714 conn(0x7fbc980f6e00 msgr2=0x7fbc980f9210 unknown :-1 s=STATE_NONE l=0).mark_down 2022-05-13T23:15:08.973 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:08.586+0000 7fbc9d82d700 1 -- 172.21.15.2:0/1257814714 shutdown_connections 2022-05-13T23:15:08.974 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:08.586+0000 7fbc9d82d700 1 -- 172.21.15.2:0/1257814714 wait complete. 2022-05-13T23:15:08.974 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:08.586+0000 7fbc9d82d700 1 Processor -- start 2022-05-13T23:15:08.974 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:08.587+0000 7fbc9d82d700 1 -- start start 2022-05-13T23:15:08.974 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:08.587+0000 7fbc9d82d700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fbc980fb5f0 0x7fbc9810c220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:15:08.974 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:08.587+0000 7fbc9d82d700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fbc980fc890 0x7fbc9810c760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:15:08.975 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:08.587+0000 7fbc9d82d700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fbc980fd220 0x7fbc981137f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:15:08.975 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:08.587+0000 7fbc9d82d700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc98113d30 con 0x7fbc980fb5f0 2022-05-13T23:15:08.975 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:08.587+0000 7fbc9d82d700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc98113ea0 con 0x7fbc980fd220 2022-05-13T23:15:08.975 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:08.587+0000 7fbc9d82d700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc98114060 con 0x7fbc980fc890 2022-05-13T23:15:08.975 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:08.587+0000 7fbc96ffd700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fbc980fb5f0 0x7fbc9810c220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:15:08.976 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:08.587+0000 7fbc96ffd700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fbc980fb5f0 0x7fbc9810c220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:48352/0 (socket says 172.21.15.2:48352) 2022-05-13T23:15:08.976 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:08.587+0000 7fbc96ffd700 1 -- 172.21.15.2:0/3650704256 learned_addr learned my addr 172.21.15.2:0/3650704256 (peer_addr_for_me v2:172.21.15.2:0/0) 2022-05-13T23:15:08.976 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:08.587+0000 7fbc967fc700 1 --2- 172.21.15.2:0/3650704256 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fbc980fc890 0x7fbc9810c760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:15:08.977 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:08.587+0000 7fbc977fe700 1 --2- 172.21.15.2:0/3650704256 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fbc980fd220 0x7fbc981137f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:15:08.977 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:08.588+0000 7fbc96ffd700 1 -- 172.21.15.2:0/3650704256 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fbc980fc890 msgr2=0x7fbc9810c760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:15:08.977 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:08.588+0000 7fbc96ffd700 1 --2- 172.21.15.2:0/3650704256 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fbc980fc890 0x7fbc9810c760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:08.977 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:08.588+0000 7fbc96ffd700 1 -- 172.21.15.2:0/3650704256 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fbc980fd220 msgr2=0x7fbc981137f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:15:08.979 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:08.588+0000 7fbc96ffd700 1 --2- 172.21.15.2:0/3650704256 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fbc980fd220 0x7fbc981137f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:08.980 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:08.588+0000 7fbc96ffd700 1 -- 172.21.15.2:0/3650704256 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc8c005040 con 0x7fbc980fb5f0 2022-05-13T23:15:08.980 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:08.588+0000 7fbc96ffd700 1 --2- 172.21.15.2:0/3650704256 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fbc980fb5f0 0x7fbc9810c220 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fbc8800e940 tx=0x7fbc8800b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-05-13T23:15:08.980 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:08.588+0000 7fbc7f7fe700 1 -- 172.21.15.2:0/3650704256 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc88007b40 con 0x7fbc980fb5f0 2022-05-13T23:15:08.981 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:08.588+0000 7fbc9d82d700 1 -- 172.21.15.2:0/3650704256 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbc900097c0 con 0x7fbc980fb5f0 2022-05-13T23:15:08.981 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:08.588+0000 7fbc7f7fe700 1 -- 172.21.15.2:0/3650704256 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1087+0+0 (secure 0 0 0) 0x7fbc8800f040 con 0x7fbc980fb5f0 2022-05-13T23:15:08.981 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:08.588+0000 7fbc7f7fe700 1 -- 172.21.15.2:0/3650704256 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc88014890 con 0x7fbc980fb5f0 2022-05-13T23:15:08.981 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:08.589+0000 7fbc9d82d700 1 -- 172.21.15.2:0/3650704256 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbc98114710 con 0x7fbc980fb5f0 2022-05-13T23:15:08.982 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:08.590+0000 7fbc9d82d700 1 -- 172.21.15.2:0/3650704256 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbc98106e40 con 0x7fbc980fb5f0 2022-05-13T23:15:08.982 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:08.591+0000 7fbc7f7fe700 1 -- 172.21.15.2:0/3650704256 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 16) v1 ==== 125136+0+0 (secure 0 0 0) 0x7fbc8801b030 con 0x7fbc980fb5f0 2022-05-13T23:15:08.982 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:08.593+0000 7fbc7f7fe700 1 --2- 172.21.15.2:0/3650704256 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7fbc80096d60 0x7fbc80099210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:15:08.982 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:08.593+0000 7fbc7f7fe700 1 -- 172.21.15.2:0/3650704256 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8120+0+0 (secure 0 0 0) 0x7fbc880bead0 con 0x7fbc980fb5f0 2022-05-13T23:15:08.983 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:08.593+0000 7fbc967fc700 1 --2- 172.21.15.2:0/3650704256 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7fbc80096d60 0x7fbc80099210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:15:08.983 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:08.594+0000 7fbc967fc700 1 --2- 172.21.15.2:0/3650704256 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7fbc80096d60 0x7fbc80099210 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fbc8c004830 tx=0x7fbc8c00b040 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-05-13T23:15:08.983 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:08.597+0000 7fbc7f7fe700 1 -- 172.21.15.2:0/3650704256 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+165215 (secure 0 0 0) 0x7fbc8804b570 con 0x7fbc980fb5f0 2022-05-13T23:15:08.984 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:08.822+0000 7fbc9d82d700 1 -- 172.21.15.2:0/3650704256 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7fbc98114930 con 0x7fbc980fb5f0 2022-05-13T23:15:08.984 INFO:teuthology.orchestra.run.smithi002.stdout:55834574923 2022-05-13T23:15:08.985 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:08.822+0000 7fbc7f7fe700 1 -- 172.21.15.2:0/3650704256 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fbc8808bd80 con 0x7fbc980fb5f0 2022-05-13T23:15:08.985 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:08.825+0000 7fbc9d82d700 1 -- 172.21.15.2:0/3650704256 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7fbc80096d60 msgr2=0x7fbc80099210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:15:08.986 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:08.825+0000 7fbc9d82d700 1 --2- 172.21.15.2:0/3650704256 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7fbc80096d60 0x7fbc80099210 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fbc8c004830 tx=0x7fbc8c00b040 comp rx=0 tx=0).stop 2022-05-13T23:15:08.986 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:08.825+0000 7fbc9d82d700 1 -- 172.21.15.2:0/3650704256 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fbc980fb5f0 msgr2=0x7fbc9810c220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:15:08.989 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:08.825+0000 7fbc9d82d700 1 --2- 172.21.15.2:0/3650704256 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fbc980fb5f0 0x7fbc9810c220 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fbc8800e940 tx=0x7fbc8800b040 comp rx=0 tx=0).stop 2022-05-13T23:15:08.989 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:08.825+0000 7fbc9d82d700 1 -- 172.21.15.2:0/3650704256 shutdown_connections 2022-05-13T23:15:08.990 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:08.825+0000 7fbc9d82d700 1 --2- 172.21.15.2:0/3650704256 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fbc980fd220 0x7fbc981137f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:08.990 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:08.825+0000 7fbc9d82d700 1 --2- 172.21.15.2:0/3650704256 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7fbc80096d60 0x7fbc80099210 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:08.990 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:08.825+0000 7fbc9d82d700 1 --2- 172.21.15.2:0/3650704256 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fbc980fb5f0 0x7fbc9810c220 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:08.990 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:08.825+0000 7fbc9d82d700 1 --2- 172.21.15.2:0/3650704256 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fbc980fc890 0x7fbc9810c760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:08.991 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:08.825+0000 7fbc9d82d700 1 -- 172.21.15.2:0/3650704256 >> 172.21.15.2:0/3650704256 conn(0x7fbc980f6e00 msgr2=0x7fbc980f8fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-05-13T23:15:08.992 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:08.825+0000 7fbc9d82d700 1 -- 172.21.15.2:0/3650704256 shutdown_connections 2022-05-13T23:15:08.993 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:08.825+0000 7fbc9d82d700 1 -- 172.21.15.2:0/3650704256 wait complete. 2022-05-13T23:15:09.609 INFO:teuthology.orchestra.run.smithi002.stdout:283467841567 2022-05-13T23:15:09.610 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:23b0658391ce94ca40799469eec00c28b298816c shell --fsid 16cf5f88-d311-11ec-be40-001a4aab830c -- ceph osd last-stat-seq osd.9 2022-05-13T23:15:09.689 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:15:09 smithi002 ceph-mon[71816]: from='client.? 172.21.15.2:0/3650704256' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2022-05-13T23:15:09.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:15:09 smithi047 ceph-mon[77147]: from='client.? 172.21.15.2:0/3650704256' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2022-05-13T23:15:09.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:15:09 smithi115 ceph-mon[78593]: from='client.? 172.21.15.2:0/3650704256' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2022-05-13T23:15:10.248 INFO:tasks.cephadm.ceph_manager.ceph:need seq 55834574920 got 55834574923 for osd.1 2022-05-13T23:15:10.249 DEBUG:teuthology.parallel:result is None 2022-05-13T23:15:10.317 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:10.308+0000 7fb711282700 1 -- 172.21.15.2:0/3699220006 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb70c0fd190 msgr2=0x7fb70c103a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:15:10.317 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:10.308+0000 7fb711282700 1 --2- 172.21.15.2:0/3699220006 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb70c0fd190 0x7fb70c103a50 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fb7000047b0 tx=0x7fb700009cb0 comp rx=0 tx=0).stop 2022-05-13T23:15:10.317 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:10.309+0000 7fb711282700 1 -- 172.21.15.2:0/3699220006 shutdown_connections 2022-05-13T23:15:10.318 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:10.309+0000 7fb711282700 1 --2- 172.21.15.2:0/3699220006 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb70c0fd190 0x7fb70c103a50 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:10.318 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:10.309+0000 7fb711282700 1 --2- 172.21.15.2:0/3699220006 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb70c0fb560 0x7fb70c0fb970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:10.318 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:10.309+0000 7fb711282700 1 --2- 172.21.15.2:0/3699220006 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fb70c0fc800 0x7fb70c0fcc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:10.318 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:10.309+0000 7fb711282700 1 -- 172.21.15.2:0/3699220006 >> 172.21.15.2:0/3699220006 conn(0x7fb70c0f6d30 msgr2=0x7fb70c0f9180 unknown :-1 s=STATE_NONE l=0).mark_down 2022-05-13T23:15:10.318 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:10.309+0000 7fb711282700 1 -- 172.21.15.2:0/3699220006 shutdown_connections 2022-05-13T23:15:10.319 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:10.309+0000 7fb711282700 1 -- 172.21.15.2:0/3699220006 wait complete. 2022-05-13T23:15:10.319 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:10.310+0000 7fb711282700 1 Processor -- start 2022-05-13T23:15:10.319 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:10.310+0000 7fb711282700 1 -- start start 2022-05-13T23:15:10.319 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:10.311+0000 7fb711282700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fb70c0fb560 0x7fb70c10e310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:15:10.320 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:10.311+0000 7fb711282700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb70c0fc800 0x7fb70c10e850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:15:10.320 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:10.311+0000 7fb711282700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb70c0fd190 0x7fb70c1158e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:15:10.320 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:10.311+0000 7fb711282700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb70c115e20 con 0x7fb70c0fd190 2022-05-13T23:15:10.320 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:10.311+0000 7fb711282700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb70c115f90 con 0x7fb70c0fc800 2022-05-13T23:15:10.320 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:10.311+0000 7fb711282700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb70c116150 con 0x7fb70c0fb560 2022-05-13T23:15:10.321 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:10.311+0000 7fb70affd700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fb70c0fb560 0x7fb70c10e310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:15:10.321 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:10.311+0000 7fb70affd700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fb70c0fb560 0x7fb70c10e310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.2:51470/0 (socket says 172.21.15.2:51470) 2022-05-13T23:15:10.321 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:10.311+0000 7fb70affd700 1 -- 172.21.15.2:0/1034059460 learned_addr learned my addr 172.21.15.2:0/1034059460 (peer_addr_for_me v2:172.21.15.2:0/0) 2022-05-13T23:15:10.321 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:10.311+0000 7fb70b7fe700 1 --2- 172.21.15.2:0/1034059460 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb70c0fd190 0x7fb70c1158e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:15:10.322 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:10.311+0000 7fb70a7fc700 1 --2- 172.21.15.2:0/1034059460 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb70c0fc800 0x7fb70c10e850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:15:10.324 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:10.312+0000 7fb70affd700 1 -- 172.21.15.2:0/1034059460 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb70c0fc800 msgr2=0x7fb70c10e850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:15:10.324 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:10.312+0000 7fb70affd700 1 --2- 172.21.15.2:0/1034059460 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb70c0fc800 0x7fb70c10e850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:10.325 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:10.312+0000 7fb70affd700 1 -- 172.21.15.2:0/1034059460 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb70c0fd190 msgr2=0x7fb70c1158e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:15:10.325 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:10.312+0000 7fb70affd700 1 --2- 172.21.15.2:0/1034059460 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb70c0fd190 0x7fb70c1158e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:10.325 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:10.312+0000 7fb70affd700 1 -- 172.21.15.2:0/1034059460 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb700005040 con 0x7fb70c0fb560 2022-05-13T23:15:10.325 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:10.312+0000 7fb70a7fc700 1 --2- 172.21.15.2:0/1034059460 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb70c0fc800 0x7fb70c10e850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2022-05-13T23:15:10.326 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:10.312+0000 7fb70affd700 1 --2- 172.21.15.2:0/1034059460 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fb70c0fb560 0x7fb70c10e310 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fb70400eca0 tx=0x7fb70400b040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-05-13T23:15:10.326 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:10.312+0000 7fb6f37fe700 1 -- 172.21.15.2:0/1034059460 <== mon.2 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7040092e0 con 0x7fb70c0fb560 2022-05-13T23:15:10.326 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:10.312+0000 7fb6f37fe700 1 -- 172.21.15.2:0/1034059460 <== mon.2 v2:172.21.15.115:3300/0 2 ==== config(26 keys) v1 ==== 1087+0+0 (secure 0 0 0) 0x7fb70400f040 con 0x7fb70c0fb560 2022-05-13T23:15:10.326 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:10.313+0000 7fb6f37fe700 1 -- 172.21.15.2:0/1034059460 <== mon.2 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb70400ca80 con 0x7fb70c0fb560 2022-05-13T23:15:10.326 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:10.313+0000 7fb711282700 1 -- 172.21.15.2:0/1034059460 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb70c116430 con 0x7fb70c0fb560 2022-05-13T23:15:10.327 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:10.313+0000 7fb711282700 1 -- 172.21.15.2:0/1034059460 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb70c116950 con 0x7fb70c0fb560 2022-05-13T23:15:10.327 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:10.314+0000 7fb711282700 1 -- 172.21.15.2:0/1034059460 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb70c106dc0 con 0x7fb70c0fb560 2022-05-13T23:15:10.328 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:10.316+0000 7fb6f37fe700 1 -- 172.21.15.2:0/1034059460 <== mon.2 v2:172.21.15.115:3300/0 4 ==== mgrmap(e 16) v1 ==== 125136+0+0 (secure 0 0 0) 0x7fb7040077a0 con 0x7fb70c0fb560 2022-05-13T23:15:10.328 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:10.317+0000 7fb6f37fe700 1 --2- 172.21.15.2:0/1034059460 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7fb6f4096d60 0x7fb6f4099210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:15:10.329 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:10.318+0000 7fb6f37fe700 1 -- 172.21.15.2:0/1034059460 <== mon.2 v2:172.21.15.115:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8120+0+0 (secure 0 0 0) 0x7fb7040bf0d0 con 0x7fb70c0fb560 2022-05-13T23:15:10.329 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:10.318+0000 7fb70a7fc700 1 --2- 172.21.15.2:0/1034059460 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7fb6f4096d60 0x7fb6f4099210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:15:10.329 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:10.319+0000 7fb70a7fc700 1 --2- 172.21.15.2:0/1034059460 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7fb6f4096d60 0x7fb6f4099210 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fb6fc009f20 tx=0x7fb6fc00b040 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-05-13T23:15:10.329 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:10.322+0000 7fb6f37fe700 1 -- 172.21.15.2:0/1034059460 <== mon.2 v2:172.21.15.115:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+165215 (secure 0 0 0) 0x7fb70404b570 con 0x7fb70c0fb560 2022-05-13T23:15:10.365 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:09.907+0000 7f1837b7e700 1 -- 172.21.15.2:0/721671039 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f183006ed90 msgr2=0x7f183006f200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:15:10.366 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:09.907+0000 7f1837b7e700 1 --2- 172.21.15.2:0/721671039 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f183006ed90 0x7f183006f200 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f182c004870 tx=0x7f182c009cb0 comp rx=0 tx=0).stop 2022-05-13T23:15:10.366 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:09.907+0000 7f1837b7e700 1 -- 172.21.15.2:0/721671039 shutdown_connections 2022-05-13T23:15:10.366 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:09.907+0000 7f1837b7e700 1 --2- 172.21.15.2:0/721671039 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f18301035a0 0x7f1830105980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:10.366 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:09.907+0000 7f1837b7e700 1 --2- 172.21.15.2:0/721671039 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1830070350 0x7f183006e850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:10.367 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:09.907+0000 7f1837b7e700 1 --2- 172.21.15.2:0/721671039 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f183006ed90 0x7f183006f200 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:10.367 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:09.907+0000 7f1837b7e700 1 -- 172.21.15.2:0/721671039 >> 172.21.15.2:0/721671039 conn(0x7f18300f6e00 msgr2=0x7f18300f9210 unknown :-1 s=STATE_NONE l=0).mark_down 2022-05-13T23:15:10.368 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:09.907+0000 7f1837b7e700 1 -- 172.21.15.2:0/721671039 shutdown_connections 2022-05-13T23:15:10.368 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:09.907+0000 7f1837b7e700 1 -- 172.21.15.2:0/721671039 wait complete. 2022-05-13T23:15:10.368 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:09.908+0000 7f1837b7e700 1 Processor -- start 2022-05-13T23:15:10.368 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:09.909+0000 7f1837b7e700 1 -- start start 2022-05-13T23:15:10.368 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:09.909+0000 7f1837b7e700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f183006ed90 0x7f183010e610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:15:10.369 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:09.909+0000 7f1837b7e700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1830070350 0x7f183010eb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:15:10.369 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:09.909+0000 7f1837b7e700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f18301035a0 0x7f1830115be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:15:10.372 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:09.909+0000 7f1837b7e700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1830116120 con 0x7f1830070350 2022-05-13T23:15:10.372 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:09.909+0000 7f1837b7e700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1830116290 con 0x7f183006ed90 2022-05-13T23:15:10.372 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:09.909+0000 7f1837b7e700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1830116450 con 0x7f18301035a0 2022-05-13T23:15:10.372 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:09.910+0000 7f183591a700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f183006ed90 0x7f183010e610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:15:10.373 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:09.910+0000 7f1835119700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1830070350 0x7f183010eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:15:10.373 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:09.910+0000 7f183611b700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f18301035a0 0x7f1830115be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:15:10.373 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:09.910+0000 7f183591a700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f183006ed90 0x7f183010e610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.2:42292/0 (socket says 172.21.15.2:42292) 2022-05-13T23:15:10.373 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:09.910+0000 7f183591a700 1 -- 172.21.15.2:0/224548720 learned_addr learned my addr 172.21.15.2:0/224548720 (peer_addr_for_me v2:172.21.15.2:0/0) 2022-05-13T23:15:10.374 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:09.910+0000 7f183611b700 1 -- 172.21.15.2:0/224548720 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f183006ed90 msgr2=0x7f183010e610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:15:10.374 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:09.910+0000 7f183611b700 1 --2- 172.21.15.2:0/224548720 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f183006ed90 0x7f183010e610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:10.374 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:09.910+0000 7f183611b700 1 -- 172.21.15.2:0/224548720 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1830070350 msgr2=0x7f183010eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:15:10.374 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:09.910+0000 7f183611b700 1 --2- 172.21.15.2:0/224548720 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1830070350 0x7f183010eb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:10.375 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:09.910+0000 7f183611b700 1 -- 172.21.15.2:0/224548720 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1820005040 con 0x7f18301035a0 2022-05-13T23:15:10.375 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:09.910+0000 7f1835119700 1 --2- 172.21.15.2:0/224548720 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1830070350 0x7f183010eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-05-13T23:15:10.375 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:09.910+0000 7f183591a700 1 --2- 172.21.15.2:0/224548720 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f183006ed90 0x7f183010e610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-05-13T23:15:10.375 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:09.911+0000 7f183611b700 1 --2- 172.21.15.2:0/224548720 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f18301035a0 0x7f1830115be0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f1830070120 tx=0x7f1828004c50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-05-13T23:15:10.376 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:09.911+0000 7f18267fc700 1 -- 172.21.15.2:0/224548720 <== mon.2 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18280092e0 con 0x7f18301035a0 2022-05-13T23:15:10.376 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:09.911+0000 7f18267fc700 1 -- 172.21.15.2:0/224548720 <== mon.2 v2:172.21.15.115:3300/0 2 ==== config(26 keys) v1 ==== 1087+0+0 (secure 0 0 0) 0x7f1828009440 con 0x7f18301035a0 2022-05-13T23:15:10.376 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:09.911+0000 7f18267fc700 1 -- 172.21.15.2:0/224548720 <== mon.2 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f182800fd30 con 0x7f18301035a0 2022-05-13T23:15:10.376 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:09.911+0000 7f1837b7e700 1 -- 172.21.15.2:0/224548720 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f18280097c0 con 0x7f18301035a0 2022-05-13T23:15:10.377 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:09.911+0000 7f1837b7e700 1 -- 172.21.15.2:0/224548720 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f182c005040 con 0x7f18301035a0 2022-05-13T23:15:10.377 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:09.913+0000 7f1837b7e700 1 -- 172.21.15.2:0/224548720 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1830108ff0 con 0x7f18301035a0 2022-05-13T23:15:10.377 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:09.917+0000 7f18267fc700 1 -- 172.21.15.2:0/224548720 <== mon.2 v2:172.21.15.115:3300/0 4 ==== mgrmap(e 16) v1 ==== 125136+0+0 (secure 0 0 0) 0x7f182801c030 con 0x7f18301035a0 2022-05-13T23:15:10.377 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:09.919+0000 7f18267fc700 1 --2- 172.21.15.2:0/224548720 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f181c096d60 0x7f181c099210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:15:10.377 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:09.920+0000 7f18267fc700 1 -- 172.21.15.2:0/224548720 <== mon.2 v2:172.21.15.115:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8120+0+0 (secure 0 0 0) 0x7f18280bf2f0 con 0x7f18301035a0 2022-05-13T23:15:10.378 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:09.920+0000 7f18267fc700 1 -- 172.21.15.2:0/224548720 <== mon.2 v2:172.21.15.115:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+165215 (secure 0 0 0) 0x7f18280bf830 con 0x7f18301035a0 2022-05-13T23:15:10.378 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:09.920+0000 7f183591a700 1 --2- 172.21.15.2:0/224548720 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f181c096d60 0x7f181c099210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:15:10.378 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:09.921+0000 7f183591a700 1 --2- 172.21.15.2:0/224548720 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f181c096d60 0x7f181c099210 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f1820009810 tx=0x7f182000b040 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-05-13T23:15:10.378 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:10.136+0000 7f1837b7e700 1 -- 172.21.15.2:0/224548720 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7f1830101b70 con 0x7f18301035a0 2022-05-13T23:15:10.379 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:10.137+0000 7f18267fc700 1 -- 172.21.15.2:0/224548720 <== mon.2 v2:172.21.15.115:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f182808c690 con 0x7f18301035a0 2022-05-13T23:15:10.379 INFO:teuthology.orchestra.run.smithi002.stdout:107374182463 2022-05-13T23:15:10.380 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:10.139+0000 7f1837b7e700 1 -- 172.21.15.2:0/224548720 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f181c096d60 msgr2=0x7f181c099210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:15:10.380 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:10.139+0000 7f1837b7e700 1 --2- 172.21.15.2:0/224548720 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f181c096d60 0x7f181c099210 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f1820009810 tx=0x7f182000b040 comp rx=0 tx=0).stop 2022-05-13T23:15:10.380 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:10.139+0000 7f1837b7e700 1 -- 172.21.15.2:0/224548720 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f18301035a0 msgr2=0x7f1830115be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:15:10.381 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:10.139+0000 7f1837b7e700 1 --2- 172.21.15.2:0/224548720 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f18301035a0 0x7f1830115be0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f1830070120 tx=0x7f1828004c50 comp rx=0 tx=0).stop 2022-05-13T23:15:10.381 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:10.140+0000 7f1837b7e700 1 -- 172.21.15.2:0/224548720 shutdown_connections 2022-05-13T23:15:10.381 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:10.140+0000 7f1837b7e700 1 --2- 172.21.15.2:0/224548720 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f18301035a0 0x7f1830115be0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:10.381 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:10.140+0000 7f1837b7e700 1 --2- 172.21.15.2:0/224548720 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f183006ed90 0x7f183010e610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:10.382 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:10.140+0000 7f1837b7e700 1 --2- 172.21.15.2:0/224548720 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f181c096d60 0x7f181c099210 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:10.382 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:10.140+0000 7f1837b7e700 1 --2- 172.21.15.2:0/224548720 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1830070350 0x7f183010eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:10.382 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:10.140+0000 7f1837b7e700 1 -- 172.21.15.2:0/224548720 >> 172.21.15.2:0/224548720 conn(0x7f18300f6e00 msgr2=0x7f18300f91e0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-05-13T23:15:10.382 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:10.140+0000 7f1837b7e700 1 -- 172.21.15.2:0/224548720 shutdown_connections 2022-05-13T23:15:10.383 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:10.140+0000 7f1837b7e700 1 -- 172.21.15.2:0/224548720 wait complete. 2022-05-13T23:15:10.404 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/16cf5f88-d311-11ec-be40-001a4aab830c/mon.a/config 2022-05-13T23:15:10.564 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:10.563+0000 7fb711282700 1 -- 172.21.15.2:0/1034059460 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 10} v 0) v1 -- 0x7fb70c116b70 con 0x7fb70c0fb560 2022-05-13T23:15:10.565 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:10.563+0000 7fb6f37fe700 1 -- 172.21.15.2:0/1034059460 <== mon.2 v2:172.21.15.115:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 10}]=0 v0) v1 ==== 75+0+13 (secure 0 0 0) 0x7fb70408c380 con 0x7fb70c0fb560 2022-05-13T23:15:10.566 INFO:teuthology.orchestra.run.smithi002.stdout:304942678044 2022-05-13T23:15:10.568 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:10.566+0000 7fb711282700 1 -- 172.21.15.2:0/1034059460 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7fb6f4096d60 msgr2=0x7fb6f4099210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:15:10.568 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:10.566+0000 7fb711282700 1 --2- 172.21.15.2:0/1034059460 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7fb6f4096d60 0x7fb6f4099210 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fb6fc009f20 tx=0x7fb6fc00b040 comp rx=0 tx=0).stop 2022-05-13T23:15:10.569 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:10.566+0000 7fb711282700 1 -- 172.21.15.2:0/1034059460 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fb70c0fb560 msgr2=0x7fb70c10e310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:15:10.569 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:10.566+0000 7fb711282700 1 --2- 172.21.15.2:0/1034059460 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fb70c0fb560 0x7fb70c10e310 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fb70400eca0 tx=0x7fb70400b040 comp rx=0 tx=0).stop 2022-05-13T23:15:10.569 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:10.567+0000 7fb711282700 1 -- 172.21.15.2:0/1034059460 shutdown_connections 2022-05-13T23:15:10.570 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:10.567+0000 7fb711282700 1 --2- 172.21.15.2:0/1034059460 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7fb6f4096d60 0x7fb6f4099210 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:10.570 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:10.567+0000 7fb711282700 1 --2- 172.21.15.2:0/1034059460 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb70c0fd190 0x7fb70c1158e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:10.570 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:10.567+0000 7fb711282700 1 --2- 172.21.15.2:0/1034059460 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fb70c0fb560 0x7fb70c10e310 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:10.570 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:10.567+0000 7fb711282700 1 --2- 172.21.15.2:0/1034059460 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb70c0fc800 0x7fb70c10e850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:10.571 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:10.567+0000 7fb711282700 1 -- 172.21.15.2:0/1034059460 >> 172.21.15.2:0/1034059460 conn(0x7fb70c0f6d30 msgr2=0x7fb70c0f8e50 unknown :-1 s=STATE_NONE l=0).mark_down 2022-05-13T23:15:10.571 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:10.567+0000 7fb711282700 1 -- 172.21.15.2:0/1034059460 shutdown_connections 2022-05-13T23:15:10.571 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:10.567+0000 7fb711282700 1 -- 172.21.15.2:0/1034059460 wait complete. 2022-05-13T23:15:10.572 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:15:10 smithi002 ceph-mon[71816]: pgmap v317: 97 pgs: 97 active+clean; 584 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s; 11 B/s, 1 keys/s, 0 objects/s recovering 2022-05-13T23:15:10.572 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:15:10 smithi002 ceph-mon[71816]: from='client.? 172.21.15.2:0/224548720' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2022-05-13T23:15:10.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:15:10 smithi047 ceph-mon[77147]: pgmap v317: 97 pgs: 97 active+clean; 584 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s; 11 B/s, 1 keys/s, 0 objects/s recovering 2022-05-13T23:15:10.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:15:10 smithi047 ceph-mon[77147]: from='client.? 172.21.15.2:0/224548720' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2022-05-13T23:15:10.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:15:10 smithi115 ceph-mon[78593]: pgmap v317: 97 pgs: 97 active+clean; 584 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s; 11 B/s, 1 keys/s, 0 objects/s recovering 2022-05-13T23:15:10.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:15:10 smithi115 ceph-mon[78593]: from='client.? 172.21.15.2:0/224548720' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2022-05-13T23:15:10.916 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:10.618+0000 7fbbff59e700 1 -- 172.21.15.2:0/1774513410 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fbc000fd020 msgr2=0x7fbc001038e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:15:10.917 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:10.619+0000 7fbbff59e700 1 --2- 172.21.15.2:0/1774513410 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fbc000fd020 0x7fbc001038e0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fbbf8009c40 tx=0x7fbbf8004d10 comp rx=0 tx=0).stop 2022-05-13T23:15:10.917 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:10.619+0000 7fbbff59e700 1 -- 172.21.15.2:0/1774513410 shutdown_connections 2022-05-13T23:15:10.917 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:10.619+0000 7fbbff59e700 1 --2- 172.21.15.2:0/1774513410 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fbc000fd020 0x7fbc001038e0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:10.918 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:10.619+0000 7fbbff59e700 1 --2- 172.21.15.2:0/1774513410 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fbc000fb3f0 0x7fbc000fb800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:10.918 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:10.619+0000 7fbbff59e700 1 --2- 172.21.15.2:0/1774513410 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fbc000fc690 0x7fbc000fcae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:10.918 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:10.619+0000 7fbbff59e700 1 -- 172.21.15.2:0/1774513410 >> 172.21.15.2:0/1774513410 conn(0x7fbc000f6b40 msgr2=0x7fbc000f8f70 unknown :-1 s=STATE_NONE l=0).mark_down 2022-05-13T23:15:10.918 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:10.619+0000 7fbbff59e700 1 -- 172.21.15.2:0/1774513410 shutdown_connections 2022-05-13T23:15:10.919 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:10.619+0000 7fbbff59e700 1 -- 172.21.15.2:0/1774513410 wait complete. 2022-05-13T23:15:10.919 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:10.620+0000 7fbbff59e700 1 Processor -- start 2022-05-13T23:15:10.919 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:10.620+0000 7fbbff59e700 1 -- start start 2022-05-13T23:15:10.919 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:10.620+0000 7fbbff59e700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fbc000fb3f0 0x7fbc00109e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:15:10.920 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:10.620+0000 7fbbff59e700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fbc000fc690 0x7fbc0010a360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:15:10.920 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:10.620+0000 7fbbff59e700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fbc000fd020 0x7fbc001113f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:15:10.920 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:10.620+0000 7fbbff59e700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc00111930 con 0x7fbc000fb3f0 2022-05-13T23:15:10.920 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:10.620+0000 7fbbff59e700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc00111aa0 con 0x7fbc000fd020 2022-05-13T23:15:10.921 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:10.620+0000 7fbbff59e700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc00111c60 con 0x7fbc000fc690 2022-05-13T23:15:10.921 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:10.621+0000 7fbbfe59c700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fbc000fb3f0 0x7fbc00109e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:15:10.921 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:10.621+0000 7fbbfed9d700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fbc000fd020 0x7fbc001113f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:15:10.924 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:10.621+0000 7fbbfed9d700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fbc000fd020 0x7fbc001113f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.2:42318/0 (socket says 172.21.15.2:42318) 2022-05-13T23:15:10.924 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:10.621+0000 7fbbfed9d700 1 -- 172.21.15.2:0/3731911377 learned_addr learned my addr 172.21.15.2:0/3731911377 (peer_addr_for_me v2:172.21.15.2:0/0) 2022-05-13T23:15:10.924 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:10.621+0000 7fbbfdd9b700 1 --2- 172.21.15.2:0/3731911377 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fbc000fc690 0x7fbc0010a360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:15:10.924 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:10.621+0000 7fbbfe59c700 1 -- 172.21.15.2:0/3731911377 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fbc000fc690 msgr2=0x7fbc0010a360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:15:10.925 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:10.621+0000 7fbbfe59c700 1 --2- 172.21.15.2:0/3731911377 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fbc000fc690 0x7fbc0010a360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:10.925 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:10.621+0000 7fbbfe59c700 1 -- 172.21.15.2:0/3731911377 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fbc000fd020 msgr2=0x7fbc001113f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:15:10.925 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:10.621+0000 7fbbfe59c700 1 --2- 172.21.15.2:0/3731911377 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fbc000fd020 0x7fbc001113f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:10.925 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:10.621+0000 7fbbfe59c700 1 -- 172.21.15.2:0/3731911377 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbbf80097c0 con 0x7fbc000fb3f0 2022-05-13T23:15:10.926 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:10.622+0000 7fbbfe59c700 1 --2- 172.21.15.2:0/3731911377 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fbc000fb3f0 0x7fbc00109e20 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fbbf0002730 tx=0x7fbbf0009e60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-05-13T23:15:10.926 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:10.622+0000 7fbbeeffd700 1 -- 172.21.15.2:0/3731911377 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbf0007b40 con 0x7fbc000fb3f0 2022-05-13T23:15:10.926 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:10.622+0000 7fbbeeffd700 1 -- 172.21.15.2:0/3731911377 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1087+0+0 (secure 0 0 0) 0x7fbbf000f040 con 0x7fbc000fb3f0 2022-05-13T23:15:10.927 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:10.622+0000 7fbbeeffd700 1 -- 172.21.15.2:0/3731911377 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbf00137c0 con 0x7fbc000fb3f0 2022-05-13T23:15:10.927 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:10.622+0000 7fbbff59e700 1 -- 172.21.15.2:0/3731911377 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbc00111f40 con 0x7fbc000fb3f0 2022-05-13T23:15:10.929 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:10.622+0000 7fbbff59e700 1 -- 172.21.15.2:0/3731911377 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbc00112370 con 0x7fbc000fb3f0 2022-05-13T23:15:10.930 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:10.624+0000 7fbbff59e700 1 -- 172.21.15.2:0/3731911377 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbc00104af0 con 0x7fbc000fb3f0 2022-05-13T23:15:10.930 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:10.626+0000 7fbbeeffd700 1 -- 172.21.15.2:0/3731911377 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 16) v1 ==== 125136+0+0 (secure 0 0 0) 0x7fbbf001a030 con 0x7fbc000fb3f0 2022-05-13T23:15:10.931 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:10.628+0000 7fbbeeffd700 1 --2- 172.21.15.2:0/3731911377 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7fbbe8096ce0 0x7fbbe8099190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:15:10.931 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:10.628+0000 7fbbeeffd700 1 -- 172.21.15.2:0/3731911377 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8120+0+0 (secure 0 0 0) 0x7fbbf00be9e0 con 0x7fbc000fb3f0 2022-05-13T23:15:10.931 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:10.628+0000 7fbbfdd9b700 1 --2- 172.21.15.2:0/3731911377 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7fbbe8096ce0 0x7fbbe8099190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:15:10.932 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:10.629+0000 7fbbfdd9b700 1 --2- 172.21.15.2:0/3731911377 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7fbbe8096ce0 0x7fbbe8099190 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fbbf40048b0 tx=0x7fbbf400b040 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-05-13T23:15:10.932 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:10.630+0000 7fbbeeffd700 1 -- 172.21.15.2:0/3731911377 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+165215 (secure 0 0 0) 0x7fbbf008bc90 con 0x7fbc000fb3f0 2022-05-13T23:15:10.932 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:10.844+0000 7fbbff59e700 1 -- 172.21.15.2:0/3731911377 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) v1 -- 0x7fbc0002a6c0 con 0x7fbc000fb3f0 2022-05-13T23:15:10.933 INFO:teuthology.orchestra.run.smithi002.stdout:197568495664 2022-05-13T23:15:10.934 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:10.845+0000 7fbbeeffd700 1 -- 172.21.15.2:0/3731911377 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7fbbf0018090 con 0x7fbc000fb3f0 2022-05-13T23:15:10.934 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:10.847+0000 7fbbff59e700 1 -- 172.21.15.2:0/3731911377 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7fbbe8096ce0 msgr2=0x7fbbe8099190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:15:10.934 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:10.847+0000 7fbbff59e700 1 --2- 172.21.15.2:0/3731911377 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7fbbe8096ce0 0x7fbbe8099190 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fbbf40048b0 tx=0x7fbbf400b040 comp rx=0 tx=0).stop 2022-05-13T23:15:10.935 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:10.847+0000 7fbbff59e700 1 -- 172.21.15.2:0/3731911377 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fbc000fb3f0 msgr2=0x7fbc00109e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:15:10.935 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:10.847+0000 7fbbff59e700 1 --2- 172.21.15.2:0/3731911377 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fbc000fb3f0 0x7fbc00109e20 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fbbf0002730 tx=0x7fbbf0009e60 comp rx=0 tx=0).stop 2022-05-13T23:15:10.935 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:10.847+0000 7fbbff59e700 1 -- 172.21.15.2:0/3731911377 shutdown_connections 2022-05-13T23:15:10.935 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:10.847+0000 7fbbff59e700 1 --2- 172.21.15.2:0/3731911377 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fbc000fd020 0x7fbc001113f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:10.936 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:10.847+0000 7fbbff59e700 1 --2- 172.21.15.2:0/3731911377 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7fbbe8096ce0 0x7fbbe8099190 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:10.936 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:10.847+0000 7fbbff59e700 1 --2- 172.21.15.2:0/3731911377 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fbc000fb3f0 0x7fbc00109e20 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:10.936 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:10.847+0000 7fbbff59e700 1 --2- 172.21.15.2:0/3731911377 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fbc000fc690 0x7fbc0010a360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:10.936 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:10.847+0000 7fbbff59e700 1 -- 172.21.15.2:0/3731911377 >> 172.21.15.2:0/3731911377 conn(0x7fbc000f6b40 msgr2=0x7fbc000ffad0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-05-13T23:15:10.937 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:10.848+0000 7fbbff59e700 1 -- 172.21.15.2:0/3731911377 shutdown_connections 2022-05-13T23:15:10.937 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:10.848+0000 7fbbff59e700 1 -- 172.21.15.2:0/3731911377 wait complete. 2022-05-13T23:15:11.126 INFO:tasks.cephadm.ceph_manager.ceph:need seq 107374182461 got 107374182463 for osd.3 2022-05-13T23:15:11.126 DEBUG:teuthology.parallel:result is None 2022-05-13T23:15:11.149 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:10.850+0000 7f8ca90fa700 1 -- 172.21.15.2:0/2150047077 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8ca4101270 msgr2=0x7f8ca41036f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:15:11.150 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:10.850+0000 7f8ca90fa700 1 --2- 172.21.15.2:0/2150047077 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8ca4101270 0x7f8ca41036f0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f8c9c009c30 tx=0x7f8c9c004c50 comp rx=0 tx=0).stop 2022-05-13T23:15:11.150 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:10.850+0000 7f8ca90fa700 1 -- 172.21.15.2:0/2150047077 shutdown_connections 2022-05-13T23:15:11.150 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:10.850+0000 7f8ca90fa700 1 --2- 172.21.15.2:0/2150047077 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8ca4101270 0x7f8ca41036f0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:11.150 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:10.850+0000 7f8ca90fa700 1 --2- 172.21.15.2:0/2150047077 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8ca4065680 0x7f8ca40fe1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:11.151 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:10.850+0000 7f8ca90fa700 1 --2- 172.21.15.2:0/2150047077 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f8ca40fe7a0 0x7f8ca4100be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:11.151 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:10.850+0000 7f8ca90fa700 1 -- 172.21.15.2:0/2150047077 >> 172.21.15.2:0/2150047077 conn(0x7f8ca40f5ce0 msgr2=0x7f8ca40f8110 unknown :-1 s=STATE_NONE l=0).mark_down 2022-05-13T23:15:11.151 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:10.851+0000 7f8ca90fa700 1 -- 172.21.15.2:0/2150047077 shutdown_connections 2022-05-13T23:15:11.151 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:10.851+0000 7f8ca90fa700 1 -- 172.21.15.2:0/2150047077 wait complete. 2022-05-13T23:15:11.152 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:10.852+0000 7f8ca90fa700 1 Processor -- start 2022-05-13T23:15:11.152 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:10.852+0000 7f8ca90fa700 1 -- start start 2022-05-13T23:15:11.152 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:10.853+0000 7f8ca90fa700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8ca4065680 0x7f8ca40fba70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:15:11.152 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:10.853+0000 7f8ca90fa700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f8ca40fe7a0 0x7f8ca40fa0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:15:11.153 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:10.853+0000 7f8ca90fa700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8ca4101270 0x7f8ca40fa610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:15:11.153 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:10.853+0000 7f8ca90fa700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ca40fad20 con 0x7f8ca4101270 2022-05-13T23:15:11.153 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:10.853+0000 7f8ca90fa700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ca40fae90 con 0x7f8ca4065680 2022-05-13T23:15:11.153 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:10.853+0000 7f8ca90fa700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ca41163e0 con 0x7f8ca40fe7a0 2022-05-13T23:15:11.153 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:10.853+0000 7f8ca2d9d700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8ca4065680 0x7f8ca40fba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:15:11.154 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:10.853+0000 7f8ca359e700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8ca4101270 0x7f8ca40fa610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:15:11.154 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:10.853+0000 7f8ca259c700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f8ca40fe7a0 0x7f8ca40fa0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:15:11.154 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:10.853+0000 7f8ca2d9d700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8ca4065680 0x7f8ca40fba70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.2:42326/0 (socket says 172.21.15.2:42326) 2022-05-13T23:15:11.154 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:10.853+0000 7f8ca2d9d700 1 -- 172.21.15.2:0/2232128586 learned_addr learned my addr 172.21.15.2:0/2232128586 (peer_addr_for_me v2:172.21.15.2:0/0) 2022-05-13T23:15:11.154 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:10.854+0000 7f8ca2d9d700 1 -- 172.21.15.2:0/2232128586 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f8ca40fe7a0 msgr2=0x7f8ca40fa0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:15:11.155 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:10.854+0000 7f8ca2d9d700 1 --2- 172.21.15.2:0/2232128586 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f8ca40fe7a0 0x7f8ca40fa0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:11.155 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:10.854+0000 7f8ca2d9d700 1 -- 172.21.15.2:0/2232128586 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8ca4101270 msgr2=0x7f8ca40fa610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:15:11.155 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:10.854+0000 7f8ca2d9d700 1 --2- 172.21.15.2:0/2232128586 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8ca4101270 0x7f8ca40fa610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:11.155 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:10.854+0000 7f8ca2d9d700 1 -- 172.21.15.2:0/2232128586 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c94005040 con 0x7f8ca4065680 2022-05-13T23:15:11.155 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:10.854+0000 7f8ca2d9d700 1 --2- 172.21.15.2:0/2232128586 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8ca4065680 0x7f8ca40fba70 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f8c940093d0 tx=0x7f8c94009cb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-05-13T23:15:11.156 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:10.854+0000 7f8c8b7fe700 1 -- 172.21.15.2:0/2232128586 <== mon.1 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c94012570 con 0x7f8ca4065680 2022-05-13T23:15:11.156 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:10.854+0000 7f8ca90fa700 1 -- 172.21.15.2:0/2232128586 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8c9c0097c0 con 0x7f8ca4065680 2022-05-13T23:15:11.156 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:10.854+0000 7f8c8b7fe700 1 -- 172.21.15.2:0/2232128586 <== mon.1 v2:172.21.15.47:3300/0 2 ==== config(26 keys) v1 ==== 1087+0+0 (secure 0 0 0) 0x7f8c9400f040 con 0x7f8ca4065680 2022-05-13T23:15:11.156 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:10.854+0000 7f8c8b7fe700 1 -- 172.21.15.2:0/2232128586 <== mon.1 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c94013b80 con 0x7f8ca4065680 2022-05-13T23:15:11.157 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:10.855+0000 7f8ca90fa700 1 -- 172.21.15.2:0/2232128586 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ca4116960 con 0x7f8ca4065680 2022-05-13T23:15:11.157 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:10.856+0000 7f8ca90fa700 1 -- 172.21.15.2:0/2232128586 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8ca4106e20 con 0x7f8ca4065680 2022-05-13T23:15:11.157 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:10.858+0000 7f8c8b7fe700 1 -- 172.21.15.2:0/2232128586 <== mon.1 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 16) v1 ==== 125136+0+0 (secure 0 0 0) 0x7f8c94041bc0 con 0x7f8ca4065680 2022-05-13T23:15:11.157 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:10.859+0000 7f8c8b7fe700 1 --2- 172.21.15.2:0/2232128586 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f8c8c096ce0 0x7f8c8c099190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:15:11.157 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:10.859+0000 7f8c8b7fe700 1 -- 172.21.15.2:0/2232128586 <== mon.1 v2:172.21.15.47:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8120+0+0 (secure 0 0 0) 0x7f8c940bf1b0 con 0x7f8ca4065680 2022-05-13T23:15:11.158 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:10.859+0000 7f8ca259c700 1 --2- 172.21.15.2:0/2232128586 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f8c8c096ce0 0x7f8c8c099190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:15:11.158 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:10.860+0000 7f8ca259c700 1 --2- 172.21.15.2:0/2232128586 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f8c8c096ce0 0x7f8c8c099190 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f8ca4065450 tx=0x7f8c9800b040 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-05-13T23:15:11.158 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:10.861+0000 7f8c8b7fe700 1 -- 172.21.15.2:0/2232128586 <== mon.1 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+165215 (secure 0 0 0) 0x7f8c9408c460 con 0x7f8ca4065680 2022-05-13T23:15:11.158 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:11.077+0000 7f8ca90fa700 1 -- 172.21.15.2:0/2232128586 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 11} v 0) v1 -- 0x7f8ca40f7690 con 0x7f8ca4065680 2022-05-13T23:15:11.159 INFO:teuthology.orchestra.run.smithi002.stdout:335007449112 2022-05-13T23:15:11.159 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:11.077+0000 7f8c8b7fe700 1 -- 172.21.15.2:0/2232128586 <== mon.1 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 11}]=0 v0) v1 ==== 75+0+13 (secure 0 0 0) 0x7f8c94094030 con 0x7f8ca4065680 2022-05-13T23:15:11.160 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:11.080+0000 7f8ca90fa700 1 -- 172.21.15.2:0/2232128586 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f8c8c096ce0 msgr2=0x7f8c8c099190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:15:11.160 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:11.080+0000 7f8ca90fa700 1 --2- 172.21.15.2:0/2232128586 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f8c8c096ce0 0x7f8c8c099190 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f8ca4065450 tx=0x7f8c9800b040 comp rx=0 tx=0).stop 2022-05-13T23:15:11.160 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:11.080+0000 7f8ca90fa700 1 -- 172.21.15.2:0/2232128586 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8ca4065680 msgr2=0x7f8ca40fba70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:15:11.160 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:11.080+0000 7f8ca90fa700 1 --2- 172.21.15.2:0/2232128586 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8ca4065680 0x7f8ca40fba70 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f8c940093d0 tx=0x7f8c94009cb0 comp rx=0 tx=0).stop 2022-05-13T23:15:11.161 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:11.080+0000 7f8ca90fa700 1 -- 172.21.15.2:0/2232128586 shutdown_connections 2022-05-13T23:15:11.161 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:11.080+0000 7f8ca90fa700 1 --2- 172.21.15.2:0/2232128586 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f8c8c096ce0 0x7f8c8c099190 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:11.161 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:11.080+0000 7f8ca90fa700 1 --2- 172.21.15.2:0/2232128586 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8ca4101270 0x7f8ca40fa610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:11.161 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:11.080+0000 7f8ca90fa700 1 --2- 172.21.15.2:0/2232128586 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8ca4065680 0x7f8ca40fba70 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:11.162 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:11.080+0000 7f8ca90fa700 1 --2- 172.21.15.2:0/2232128586 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f8ca40fe7a0 0x7f8ca40fa0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:11.162 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:11.080+0000 7f8ca90fa700 1 -- 172.21.15.2:0/2232128586 >> 172.21.15.2:0/2232128586 conn(0x7f8ca40f5ce0 msgr2=0x7f8ca40fcaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-05-13T23:15:11.162 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:11.081+0000 7f8ca90fa700 1 -- 172.21.15.2:0/2232128586 shutdown_connections 2022-05-13T23:15:11.162 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:11.081+0000 7f8ca90fa700 1 -- 172.21.15.2:0/2232128586 wait complete. 2022-05-13T23:15:11.565 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:15:11 smithi002 ceph-mon[71816]: from='client.? 172.21.15.2:0/1034059460' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2022-05-13T23:15:11.618 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:15:11 smithi002 ceph-mon[71816]: from='client.? 172.21.15.2:0/3731911377' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2022-05-13T23:15:11.619 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:15:11 smithi002 ceph-mon[71816]: from='client.? 172.21.15.2:0/2232128586' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2022-05-13T23:15:11.706 INFO:tasks.cephadm.ceph_manager.ceph:need seq 197568495661 got 197568495664 for osd.6 2022-05-13T23:15:11.707 DEBUG:teuthology.parallel:result is None 2022-05-13T23:15:11.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:15:11 smithi047 ceph-mon[77147]: from='client.? 172.21.15.2:0/1034059460' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2022-05-13T23:15:11.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:15:11 smithi047 ceph-mon[77147]: from='client.? 172.21.15.2:0/3731911377' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2022-05-13T23:15:11.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:15:11 smithi047 ceph-mon[77147]: from='client.? 172.21.15.2:0/2232128586' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2022-05-13T23:15:11.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:15:11 smithi115 ceph-mon[78593]: from='client.? 172.21.15.2:0/1034059460' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2022-05-13T23:15:11.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:15:11 smithi115 ceph-mon[78593]: from='client.? 172.21.15.2:0/3731911377' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2022-05-13T23:15:11.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:15:11 smithi115 ceph-mon[78593]: from='client.? 172.21.15.2:0/2232128586' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2022-05-13T23:15:12.343 INFO:tasks.cephadm.ceph_manager.ceph:need seq 304942678041 got 304942678044 for osd.10 2022-05-13T23:15:12.344 DEBUG:teuthology.parallel:result is None 2022-05-13T23:15:12.572 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:15:12 smithi002 ceph-mon[71816]: pgmap v318: 97 pgs: 97 active+clean; 584 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 11 op/s; 11 B/s, 1 keys/s, 0 objects/s recovering 2022-05-13T23:15:12.586 INFO:tasks.cephadm.ceph_manager.ceph:need seq 335007449109 got 335007449112 for osd.11 2022-05-13T23:15:12.587 DEBUG:teuthology.parallel:result is None 2022-05-13T23:15:12.665 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/16cf5f88-d311-11ec-be40-001a4aab830c/mon.a/config 2022-05-13T23:15:12.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:15:12 smithi047 ceph-mon[77147]: pgmap v318: 97 pgs: 97 active+clean; 584 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 11 op/s; 11 B/s, 1 keys/s, 0 objects/s recovering 2022-05-13T23:15:12.862 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:15:12 smithi115 ceph-mon[78593]: pgmap v318: 97 pgs: 97 active+clean; 584 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 11 op/s; 11 B/s, 1 keys/s, 0 objects/s recovering 2022-05-13T23:15:14.132 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:14.037+0000 7fc563fff700 1 -- 172.21.15.2:0/99697125 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc5640fc690 msgr2=0x7fc5640fcae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:15:14.132 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:14.037+0000 7fc563fff700 1 --2- 172.21.15.2:0/99697125 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc5640fc690 0x7fc5640fcae0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fc558004870 tx=0x7fc558009cb0 comp rx=0 tx=0).stop 2022-05-13T23:15:14.132 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:14.037+0000 7fc563fff700 1 -- 172.21.15.2:0/99697125 shutdown_connections 2022-05-13T23:15:14.133 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:14.037+0000 7fc563fff700 1 --2- 172.21.15.2:0/99697125 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc5640fd020 0x7fc5641038e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:14.133 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:14.037+0000 7fc563fff700 1 --2- 172.21.15.2:0/99697125 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fc5640fb3f0 0x7fc5640fb800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:14.134 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:14.037+0000 7fc563fff700 1 --2- 172.21.15.2:0/99697125 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc5640fc690 0x7fc5640fcae0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:14.134 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:14.037+0000 7fc563fff700 1 -- 172.21.15.2:0/99697125 >> 172.21.15.2:0/99697125 conn(0x7fc5640f6b40 msgr2=0x7fc5640f8f70 unknown :-1 s=STATE_NONE l=0).mark_down 2022-05-13T23:15:14.134 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:14.037+0000 7fc563fff700 1 -- 172.21.15.2:0/99697125 shutdown_connections 2022-05-13T23:15:14.135 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:14.037+0000 7fc563fff700 1 -- 172.21.15.2:0/99697125 wait complete. 2022-05-13T23:15:14.135 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:14.038+0000 7fc563fff700 1 Processor -- start 2022-05-13T23:15:14.135 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:14.038+0000 7fc563fff700 1 -- start start 2022-05-13T23:15:14.135 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:14.038+0000 7fc563fff700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fc5640fb3f0 0x7fc5640701d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:15:14.135 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:14.038+0000 7fc563fff700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc5640fc690 0x7fc56406e830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:15:14.136 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:14.038+0000 7fc563fff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc5640fd020 0x7fc56406ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:15:14.136 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:14.038+0000 7fc563fff700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc56406f480 con 0x7fc5640fd020 2022-05-13T23:15:14.136 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:14.038+0000 7fc563fff700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc56406f5f0 con 0x7fc5640fc690 2022-05-13T23:15:14.136 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:14.038+0000 7fc563fff700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc564116370 con 0x7fc5640fb3f0 2022-05-13T23:15:14.137 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:14.039+0000 7fc562ffd700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fc5640fb3f0 0x7fc5640701d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:15:14.137 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:14.039+0000 7fc5637fe700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc5640fd020 0x7fc56406ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:15:14.137 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:14.039+0000 7fc562ffd700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fc5640fb3f0 0x7fc5640701d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.2:51530/0 (socket says 172.21.15.2:51530) 2022-05-13T23:15:14.137 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:14.039+0000 7fc5627fc700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc5640fc690 0x7fc56406e830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:15:14.138 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:14.039+0000 7fc562ffd700 1 -- 172.21.15.2:0/3177238343 learned_addr learned my addr 172.21.15.2:0/3177238343 (peer_addr_for_me v2:172.21.15.2:0/0) 2022-05-13T23:15:14.138 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:14.039+0000 7fc562ffd700 1 -- 172.21.15.2:0/3177238343 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc5640fc690 msgr2=0x7fc56406e830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:15:14.138 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:14.039+0000 7fc562ffd700 1 --2- 172.21.15.2:0/3177238343 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc5640fc690 0x7fc56406e830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:14.138 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:14.039+0000 7fc562ffd700 1 -- 172.21.15.2:0/3177238343 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc5640fd020 msgr2=0x7fc56406ed70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:15:14.139 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:14.039+0000 7fc562ffd700 1 --2- 172.21.15.2:0/3177238343 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc5640fd020 0x7fc56406ed70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:14.139 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:14.039+0000 7fc562ffd700 1 -- 172.21.15.2:0/3177238343 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc55c0097c0 con 0x7fc5640fb3f0 2022-05-13T23:15:14.139 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:14.040+0000 7fc562ffd700 1 --2- 172.21.15.2:0/3177238343 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fc5640fb3f0 0x7fc5640701d0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fc55400e940 tx=0x7fc55400b040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-05-13T23:15:14.139 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:14.040+0000 7fc54b7fe700 1 -- 172.21.15.2:0/3177238343 <== mon.2 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc554007b40 con 0x7fc5640fb3f0 2022-05-13T23:15:14.139 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:14.040+0000 7fc563fff700 1 -- 172.21.15.2:0/3177238343 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc558005040 con 0x7fc5640fb3f0 2022-05-13T23:15:14.140 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:14.040+0000 7fc54b7fe700 1 -- 172.21.15.2:0/3177238343 <== mon.2 v2:172.21.15.115:3300/0 2 ==== config(26 keys) v1 ==== 1087+0+0 (secure 0 0 0) 0x7fc55400f040 con 0x7fc5640fb3f0 2022-05-13T23:15:14.140 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:14.040+0000 7fc54b7fe700 1 -- 172.21.15.2:0/3177238343 <== mon.2 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc554014890 con 0x7fc5640fb3f0 2022-05-13T23:15:14.140 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:14.040+0000 7fc563fff700 1 -- 172.21.15.2:0/3177238343 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc5641168e0 con 0x7fc5640fb3f0 2022-05-13T23:15:14.141 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:14.041+0000 7fc563fff700 1 -- 172.21.15.2:0/3177238343 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc564106c40 con 0x7fc5640fb3f0 2022-05-13T23:15:14.142 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:14.043+0000 7fc54b7fe700 1 -- 172.21.15.2:0/3177238343 <== mon.2 v2:172.21.15.115:3300/0 4 ==== mgrmap(e 16) v1 ==== 125136+0+0 (secure 0 0 0) 0x7fc55401b030 con 0x7fc5640fb3f0 2022-05-13T23:15:14.142 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:14.045+0000 7fc54b7fe700 1 --2- 172.21.15.2:0/3177238343 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7fc54c096d60 0x7fc54c099210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:15:14.142 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:14.045+0000 7fc54b7fe700 1 -- 172.21.15.2:0/3177238343 <== mon.2 v2:172.21.15.115:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8120+0+0 (secure 0 0 0) 0x7fc5540bedf0 con 0x7fc5640fb3f0 2022-05-13T23:15:14.143 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:14.045+0000 7fc5627fc700 1 --2- 172.21.15.2:0/3177238343 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7fc54c096d60 0x7fc54c099210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:15:14.143 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:14.046+0000 7fc5627fc700 1 --2- 172.21.15.2:0/3177238343 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7fc54c096d60 0x7fc54c099210 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fc558004840 tx=0x7fc558009cb0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-05-13T23:15:14.143 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:14.046+0000 7fc54b7fe700 1 -- 172.21.15.2:0/3177238343 <== mon.2 v2:172.21.15.115:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+165215 (secure 0 0 0) 0x7fc55404b570 con 0x7fc5640fb3f0 2022-05-13T23:15:14.263 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:14.262+0000 7fc563fff700 1 -- 172.21.15.2:0/3177238343 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7fc56402a4f0 con 0x7fc5640fb3f0 2022-05-13T23:15:14.264 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:14.262+0000 7fc54b7fe700 1 -- 172.21.15.2:0/3177238343 <== mon.2 v2:172.21.15.115:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7fc554019090 con 0x7fc5640fb3f0 2022-05-13T23:15:14.264 INFO:teuthology.orchestra.run.smithi002.stdout:141733920827 2022-05-13T23:15:14.266 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:14.265+0000 7fc563fff700 1 -- 172.21.15.2:0/3177238343 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7fc54c096d60 msgr2=0x7fc54c099210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:15:14.267 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:14.265+0000 7fc563fff700 1 --2- 172.21.15.2:0/3177238343 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7fc54c096d60 0x7fc54c099210 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fc558004840 tx=0x7fc558009cb0 comp rx=0 tx=0).stop 2022-05-13T23:15:14.267 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:14.265+0000 7fc563fff700 1 -- 172.21.15.2:0/3177238343 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fc5640fb3f0 msgr2=0x7fc5640701d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:15:14.267 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:14.265+0000 7fc563fff700 1 --2- 172.21.15.2:0/3177238343 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fc5640fb3f0 0x7fc5640701d0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fc55400e940 tx=0x7fc55400b040 comp rx=0 tx=0).stop 2022-05-13T23:15:14.268 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:14.265+0000 7fc563fff700 1 -- 172.21.15.2:0/3177238343 shutdown_connections 2022-05-13T23:15:14.268 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:14.265+0000 7fc563fff700 1 --2- 172.21.15.2:0/3177238343 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7fc54c096d60 0x7fc54c099210 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:14.268 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:14.265+0000 7fc563fff700 1 --2- 172.21.15.2:0/3177238343 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc5640fd020 0x7fc56406ed70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:14.268 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:14.265+0000 7fc563fff700 1 --2- 172.21.15.2:0/3177238343 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fc5640fb3f0 0x7fc5640701d0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:14.268 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:14.265+0000 7fc563fff700 1 --2- 172.21.15.2:0/3177238343 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc5640fc690 0x7fc56406e830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:14.269 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:14.265+0000 7fc563fff700 1 -- 172.21.15.2:0/3177238343 >> 172.21.15.2:0/3177238343 conn(0x7fc5640f6b40 msgr2=0x7fc5640ff8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-05-13T23:15:14.269 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:14.265+0000 7fc563fff700 1 -- 172.21.15.2:0/3177238343 shutdown_connections 2022-05-13T23:15:14.269 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:14.265+0000 7fc563fff700 1 -- 172.21.15.2:0/3177238343 wait complete. 2022-05-13T23:15:14.501 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:15:14 smithi002 ceph-mon[71816]: pgmap v319: 97 pgs: 97 active+clean; 584 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s; 11 B/s, 1 keys/s, 0 objects/s recovering 2022-05-13T23:15:14.501 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:15:14 smithi002 ceph-mon[71816]: from='client.? 172.21.15.2:0/3177238343' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2022-05-13T23:15:14.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:15:14 smithi047 ceph-mon[77147]: pgmap v319: 97 pgs: 97 active+clean; 584 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s; 11 B/s, 1 keys/s, 0 objects/s recovering 2022-05-13T23:15:14.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:15:14 smithi047 ceph-mon[77147]: from='client.? 172.21.15.2:0/3177238343' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2022-05-13T23:15:14.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:15:14 smithi115 ceph-mon[78593]: pgmap v319: 97 pgs: 97 active+clean; 584 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s; 11 B/s, 1 keys/s, 0 objects/s recovering 2022-05-13T23:15:14.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:15:14 smithi115 ceph-mon[78593]: from='client.? 172.21.15.2:0/3177238343' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2022-05-13T23:15:15.134 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:15.132+0000 7fb2f4cd0700 1 -- 172.21.15.2:0/3038684111 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb2f0070370 msgr2=0x7fb2f006e870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:15:15.135 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:15.132+0000 7fb2f4cd0700 1 --2- 172.21.15.2:0/3038684111 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb2f0070370 0x7fb2f006e870 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fb2e0004870 tx=0x7fb2e0009cb0 comp rx=0 tx=0).stop 2022-05-13T23:15:15.135 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:15.132+0000 7fb2f4cd0700 1 -- 172.21.15.2:0/3038684111 shutdown_connections 2022-05-13T23:15:15.135 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:15.132+0000 7fb2f4cd0700 1 --2- 172.21.15.2:0/3038684111 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fb2f0103580 0x7fb2f0105960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:15.135 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:15.132+0000 7fb2f4cd0700 1 --2- 172.21.15.2:0/3038684111 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb2f0070370 0x7fb2f006e870 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:15.135 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:15.133+0000 7fb2f4cd0700 1 --2- 172.21.15.2:0/3038684111 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb2f006edb0 0x7fb2f006f220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:15.136 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:15.133+0000 7fb2f4cd0700 1 -- 172.21.15.2:0/3038684111 >> 172.21.15.2:0/3038684111 conn(0x7fb2f00f6e00 msgr2=0x7fb2f00f9210 unknown :-1 s=STATE_NONE l=0).mark_down 2022-05-13T23:15:15.136 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:15.133+0000 7fb2f4cd0700 1 -- 172.21.15.2:0/3038684111 shutdown_connections 2022-05-13T23:15:15.136 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:15.133+0000 7fb2f4cd0700 1 -- 172.21.15.2:0/3038684111 wait complete. 2022-05-13T23:15:15.138 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:15.133+0000 7fb2f4cd0700 1 Processor -- start 2022-05-13T23:15:15.138 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:15.134+0000 7fb2f4cd0700 1 -- start start 2022-05-13T23:15:15.139 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:15.134+0000 7fb2f4cd0700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb2f006edb0 0x7fb2f010e630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:15:15.139 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:15.134+0000 7fb2f4cd0700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb2f0070370 0x7fb2f010eb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:15:15.139 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:15.134+0000 7fb2f4cd0700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fb2f0103580 0x7fb2f0115c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:15:15.139 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:15.134+0000 7fb2f4cd0700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2f0116140 con 0x7fb2f0070370 2022-05-13T23:15:15.140 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:15.134+0000 7fb2f4cd0700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2f01162b0 con 0x7fb2f006edb0 2022-05-13T23:15:15.140 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:15.134+0000 7fb2f4cd0700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2f0116470 con 0x7fb2f0103580 2022-05-13T23:15:15.141 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:15.134+0000 7fb2ee59c700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb2f006edb0 0x7fb2f010e630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:15:15.141 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:15.134+0000 7fb2ee59c700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb2f006edb0 0x7fb2f010e630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.2:42346/0 (socket says 172.21.15.2:42346) 2022-05-13T23:15:15.141 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:15.134+0000 7fb2ee59c700 1 -- 172.21.15.2:0/874032884 learned_addr learned my addr 172.21.15.2:0/874032884 (peer_addr_for_me v2:172.21.15.2:0/0) 2022-05-13T23:15:15.141 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:15.135+0000 7fb2edd9b700 1 --2- 172.21.15.2:0/874032884 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb2f0070370 0x7fb2f010eb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:15:15.142 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:15.135+0000 7fb2eed9d700 1 --2- 172.21.15.2:0/874032884 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fb2f0103580 0x7fb2f0115c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:15:15.142 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:15.135+0000 7fb2eed9d700 1 -- 172.21.15.2:0/874032884 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb2f006edb0 msgr2=0x7fb2f010e630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:15:15.142 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:15.135+0000 7fb2eed9d700 1 --2- 172.21.15.2:0/874032884 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb2f006edb0 0x7fb2f010e630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:15.142 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:15.135+0000 7fb2eed9d700 1 -- 172.21.15.2:0/874032884 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb2f0070370 msgr2=0x7fb2f010eb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:15:15.143 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:15.135+0000 7fb2eed9d700 1 --2- 172.21.15.2:0/874032884 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb2f0070370 0x7fb2f010eb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:15.143 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:15.135+0000 7fb2eed9d700 1 -- 172.21.15.2:0/874032884 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2e0005040 con 0x7fb2f0103580 2022-05-13T23:15:15.143 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:15.135+0000 7fb2edd9b700 1 --2- 172.21.15.2:0/874032884 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb2f0070370 0x7fb2f010eb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-05-13T23:15:15.143 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:15.135+0000 7fb2eed9d700 1 --2- 172.21.15.2:0/874032884 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fb2f0103580 0x7fb2f0115c00 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fb2f0070140 tx=0x7fb2e800b040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-05-13T23:15:15.143 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:15.136+0000 7fb2deffd700 1 -- 172.21.15.2:0/874032884 <== mon.2 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2e8009870 con 0x7fb2f0103580 2022-05-13T23:15:15.144 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:15.136+0000 7fb2f4cd0700 1 -- 172.21.15.2:0/874032884 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb2f0116750 con 0x7fb2f0103580 2022-05-13T23:15:15.144 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:15.136+0000 7fb2deffd700 1 -- 172.21.15.2:0/874032884 <== mon.2 v2:172.21.15.115:3300/0 2 ==== config(26 keys) v1 ==== 1087+0+0 (secure 0 0 0) 0x7fb2e800f040 con 0x7fb2f0103580 2022-05-13T23:15:15.144 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:15.136+0000 7fb2deffd700 1 -- 172.21.15.2:0/874032884 <== mon.2 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2e8007e60 con 0x7fb2f0103580 2022-05-13T23:15:15.144 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:15.136+0000 7fb2f4cd0700 1 -- 172.21.15.2:0/874032884 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb2f0116cc0 con 0x7fb2f0103580 2022-05-13T23:15:15.145 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:15.137+0000 7fb2f4cd0700 1 -- 172.21.15.2:0/874032884 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb2f0109020 con 0x7fb2f0103580 2022-05-13T23:15:15.147 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:15.139+0000 7fb2deffd700 1 -- 172.21.15.2:0/874032884 <== mon.2 v2:172.21.15.115:3300/0 4 ==== mgrmap(e 16) v1 ==== 125136+0+0 (secure 0 0 0) 0x7fb2e801a020 con 0x7fb2f0103580 2022-05-13T23:15:15.147 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:15.140+0000 7fb2deffd700 1 --2- 172.21.15.2:0/874032884 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7fb2d8096ce0 0x7fb2d8099190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:15:15.148 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:15.140+0000 7fb2deffd700 1 -- 172.21.15.2:0/874032884 <== mon.2 v2:172.21.15.115:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8120+0+0 (secure 0 0 0) 0x7fb2e80bee10 con 0x7fb2f0103580 2022-05-13T23:15:15.148 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:15.141+0000 7fb2ee59c700 1 --2- 172.21.15.2:0/874032884 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7fb2d8096ce0 0x7fb2d8099190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:15:15.148 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:15.141+0000 7fb2ee59c700 1 --2- 172.21.15.2:0/874032884 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7fb2d8096ce0 0x7fb2d8099190 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fb2e00045a0 tx=0x7fb2e0009cb0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-05-13T23:15:15.148 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:15.142+0000 7fb2deffd700 1 -- 172.21.15.2:0/874032884 <== mon.2 v2:172.21.15.115:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+165215 (secure 0 0 0) 0x7fb2e808c1b0 con 0x7fb2f0103580 2022-05-13T23:15:15.208 INFO:tasks.cephadm.ceph_manager.ceph:need seq 141733920824 got 141733920827 for osd.4 2022-05-13T23:15:15.209 DEBUG:teuthology.parallel:result is None 2022-05-13T23:15:15.357 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:15.355+0000 7fb2f4cd0700 1 -- 172.21.15.2:0/874032884 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7fb2f0101b50 con 0x7fb2f0103580 2022-05-13T23:15:15.358 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:15.356+0000 7fb2deffd700 1 -- 172.21.15.2:0/874032884 <== mon.2 v2:172.21.15.115:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7fb2e808f7d0 con 0x7fb2f0103580 2022-05-13T23:15:15.358 INFO:teuthology.orchestra.run.smithi002.stdout:163208757302 2022-05-13T23:15:15.360 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:15.358+0000 7fb2f4cd0700 1 -- 172.21.15.2:0/874032884 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7fb2d8096ce0 msgr2=0x7fb2d8099190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:15:15.361 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:15.358+0000 7fb2f4cd0700 1 --2- 172.21.15.2:0/874032884 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7fb2d8096ce0 0x7fb2d8099190 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fb2e00045a0 tx=0x7fb2e0009cb0 comp rx=0 tx=0).stop 2022-05-13T23:15:15.361 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:15.359+0000 7fb2f4cd0700 1 -- 172.21.15.2:0/874032884 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fb2f0103580 msgr2=0x7fb2f0115c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:15:15.361 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:15.359+0000 7fb2f4cd0700 1 --2- 172.21.15.2:0/874032884 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fb2f0103580 0x7fb2f0115c00 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fb2f0070140 tx=0x7fb2e800b040 comp rx=0 tx=0).stop 2022-05-13T23:15:15.362 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:15.359+0000 7fb2f4cd0700 1 -- 172.21.15.2:0/874032884 shutdown_connections 2022-05-13T23:15:15.362 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:15.359+0000 7fb2f4cd0700 1 --2- 172.21.15.2:0/874032884 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fb2f0103580 0x7fb2f0115c00 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:15.362 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:15.359+0000 7fb2f4cd0700 1 --2- 172.21.15.2:0/874032884 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb2f006edb0 0x7fb2f010e630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:15.362 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:15.359+0000 7fb2f4cd0700 1 --2- 172.21.15.2:0/874032884 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7fb2d8096ce0 0x7fb2d8099190 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:15.362 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:15.359+0000 7fb2f4cd0700 1 --2- 172.21.15.2:0/874032884 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb2f0070370 0x7fb2f010eb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:15.363 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:15.359+0000 7fb2f4cd0700 1 -- 172.21.15.2:0/874032884 >> 172.21.15.2:0/874032884 conn(0x7fb2f00f6e00 msgr2=0x7fb2f00f91e0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-05-13T23:15:15.363 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:15.359+0000 7fb2f4cd0700 1 -- 172.21.15.2:0/874032884 shutdown_connections 2022-05-13T23:15:15.363 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:15.359+0000 7fb2f4cd0700 1 -- 172.21.15.2:0/874032884 wait complete. 2022-05-13T23:15:15.548 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:15.547+0000 7f7d21f25700 1 -- 172.21.15.2:0/4052718396 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f7d1c06edb0 msgr2=0x7f7d1c06f220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:15:15.549 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:15.547+0000 7f7d21f25700 1 --2- 172.21.15.2:0/4052718396 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f7d1c06edb0 0x7f7d1c06f220 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f7d100047b0 tx=0x7f7d10009cb0 comp rx=0 tx=0).stop 2022-05-13T23:15:15.549 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:15.547+0000 7f7d21f25700 1 -- 172.21.15.2:0/4052718396 shutdown_connections 2022-05-13T23:15:15.549 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:15.547+0000 7f7d21f25700 1 --2- 172.21.15.2:0/4052718396 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7d1c103580 0x7f7d1c105960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:15.549 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:15.547+0000 7f7d21f25700 1 --2- 172.21.15.2:0/4052718396 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7d1c070370 0x7f7d1c06e870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:15.550 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:15.547+0000 7f7d21f25700 1 --2- 172.21.15.2:0/4052718396 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f7d1c06edb0 0x7f7d1c06f220 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:15.550 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:15.547+0000 7f7d21f25700 1 -- 172.21.15.2:0/4052718396 >> 172.21.15.2:0/4052718396 conn(0x7f7d1c0f6e00 msgr2=0x7f7d1c0f9210 unknown :-1 s=STATE_NONE l=0).mark_down 2022-05-13T23:15:15.550 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:15.547+0000 7f7d21f25700 1 -- 172.21.15.2:0/4052718396 shutdown_connections 2022-05-13T23:15:15.550 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:15.547+0000 7f7d21f25700 1 -- 172.21.15.2:0/4052718396 wait complete. 2022-05-13T23:15:15.553 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:15.548+0000 7f7d21f25700 1 Processor -- start 2022-05-13T23:15:15.553 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:15.548+0000 7f7d21f25700 1 -- start start 2022-05-13T23:15:15.553 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:15.548+0000 7f7d21f25700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7d1c06edb0 0x7f7d1c10e680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:15:15.554 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:15.548+0000 7f7d21f25700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f7d1c070370 0x7f7d1c10ebc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:15:15.554 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:15.548+0000 7f7d21f25700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7d1c103580 0x7f7d1c115c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:15:15.554 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:15.548+0000 7f7d21f25700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d1c116190 con 0x7f7d1c103580 2022-05-13T23:15:15.554 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:15.548+0000 7f7d21f25700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d1c116300 con 0x7f7d1c06edb0 2022-05-13T23:15:15.555 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:15.548+0000 7f7d21f25700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d1c1164c0 con 0x7f7d1c070370 2022-05-13T23:15:15.555 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:15.549+0000 7f7d1bfff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7d1c103580 0x7f7d1c115c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:15:15.555 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:15.549+0000 7f7d1b7fe700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7d1c06edb0 0x7f7d1c10e680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:15:15.555 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:15.549+0000 7f7d1affd700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f7d1c070370 0x7f7d1c10ebc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:15:15.556 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:15.549+0000 7f7d1bfff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7d1c103580 0x7f7d1c115c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:57586/0 (socket says 172.21.15.2:57586) 2022-05-13T23:15:15.556 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:15.549+0000 7f7d1bfff700 1 -- 172.21.15.2:0/2332341199 learned_addr learned my addr 172.21.15.2:0/2332341199 (peer_addr_for_me v2:172.21.15.2:0/0) 2022-05-13T23:15:15.556 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:15.549+0000 7f7d1affd700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f7d1c070370 0x7f7d1c10ebc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.2:51570/0 (socket says 172.21.15.2:51570) 2022-05-13T23:15:15.556 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:15.549+0000 7f7d1affd700 1 -- 172.21.15.2:0/2332341199 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7d1c06edb0 msgr2=0x7f7d1c10e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:15:15.556 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:15.549+0000 7f7d1affd700 1 --2- 172.21.15.2:0/2332341199 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7d1c06edb0 0x7f7d1c10e680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:15.557 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:15.549+0000 7f7d1affd700 1 -- 172.21.15.2:0/2332341199 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7d1c103580 msgr2=0x7f7d1c115c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:15:15.557 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:15.549+0000 7f7d1affd700 1 --2- 172.21.15.2:0/2332341199 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7d1c103580 0x7f7d1c115c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:15.557 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:15.549+0000 7f7d1affd700 1 -- 172.21.15.2:0/2332341199 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d0c005040 con 0x7f7d1c070370 2022-05-13T23:15:15.558 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:15.549+0000 7f7d1b7fe700 1 --2- 172.21.15.2:0/2332341199 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7d1c06edb0 0x7f7d1c10e680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-05-13T23:15:15.558 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:15.549+0000 7f7d1affd700 1 --2- 172.21.15.2:0/2332341199 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f7d1c070370 0x7f7d1c10ebc0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f7d10009f20 tx=0x7f7d10009cb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-05-13T23:15:15.558 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:15.550+0000 7f7d03fff700 1 -- 172.21.15.2:0/2332341199 <== mon.2 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d10020070 con 0x7f7d1c070370 2022-05-13T23:15:15.558 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:15.550+0000 7f7d21f25700 1 -- 172.21.15.2:0/2332341199 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7d10005040 con 0x7f7d1c070370 2022-05-13T23:15:15.560 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:15.550+0000 7f7d03fff700 1 -- 172.21.15.2:0/2332341199 <== mon.2 v2:172.21.15.115:3300/0 2 ==== config(26 keys) v1 ==== 1087+0+0 (secure 0 0 0) 0x7f7d10000ce0 con 0x7f7d1c070370 2022-05-13T23:15:15.560 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:15.550+0000 7f7d03fff700 1 -- 172.21.15.2:0/2332341199 <== mon.2 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d1001cc30 con 0x7f7d1c070370 2022-05-13T23:15:15.561 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:15.550+0000 7f7d21f25700 1 -- 172.21.15.2:0/2332341199 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7d1c116a30 con 0x7f7d1c070370 2022-05-13T23:15:15.561 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:15.551+0000 7f7d21f25700 1 -- 172.21.15.2:0/2332341199 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7d1c109020 con 0x7f7d1c070370 2022-05-13T23:15:15.563 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:15.553+0000 7f7d03fff700 1 -- 172.21.15.2:0/2332341199 <== mon.2 v2:172.21.15.115:3300/0 4 ==== mgrmap(e 16) v1 ==== 125136+0+0 (secure 0 0 0) 0x7f7d1001c430 con 0x7f7d1c070370 2022-05-13T23:15:15.563 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:15.554+0000 7f7d03fff700 1 --2- 172.21.15.2:0/2332341199 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f7d04096ce0 0x7f7d04099190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:15:15.564 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:15.554+0000 7f7d03fff700 1 -- 172.21.15.2:0/2332341199 <== mon.2 v2:172.21.15.115:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8120+0+0 (secure 0 0 0) 0x7f7d100c0170 con 0x7f7d1c070370 2022-05-13T23:15:15.564 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:15.555+0000 7f7d1b7fe700 1 --2- 172.21.15.2:0/2332341199 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f7d04096ce0 0x7f7d04099190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:15:15.565 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:15.555+0000 7f7d1b7fe700 1 --2- 172.21.15.2:0/2332341199 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f7d04096ce0 0x7f7d04099190 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f7d0c004c80 tx=0x7f7d0c00b040 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-05-13T23:15:15.565 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:15.557+0000 7f7d03fff700 1 -- 172.21.15.2:0/2332341199 <== mon.2 v2:172.21.15.115:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+165215 (secure 0 0 0) 0x7f7d1008d460 con 0x7f7d1c070370 2022-05-13T23:15:15.572 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:15:15 smithi002 ceph-mon[71816]: from='client.? 172.21.15.2:0/874032884' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2022-05-13T23:15:15.758 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:15.755+0000 7f7d21f25700 1 -- 172.21.15.2:0/2332341199 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 9} v 0) v1 -- 0x7f7d1c101b50 con 0x7f7d1c070370 2022-05-13T23:15:15.759 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:15.756+0000 7f7d03fff700 1 -- 172.21.15.2:0/2332341199 <== mon.2 v2:172.21.15.115:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 9}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f7d10095030 con 0x7f7d1c070370 2022-05-13T23:15:15.760 INFO:teuthology.orchestra.run.smithi002.stdout:283467841569 2022-05-13T23:15:15.761 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:15.759+0000 7f7d21f25700 1 -- 172.21.15.2:0/2332341199 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f7d04096ce0 msgr2=0x7f7d04099190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:15:15.761 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:15.759+0000 7f7d21f25700 1 --2- 172.21.15.2:0/2332341199 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f7d04096ce0 0x7f7d04099190 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f7d0c004c80 tx=0x7f7d0c00b040 comp rx=0 tx=0).stop 2022-05-13T23:15:15.763 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:15.759+0000 7f7d21f25700 1 -- 172.21.15.2:0/2332341199 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f7d1c070370 msgr2=0x7f7d1c10ebc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:15:15.764 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:15.759+0000 7f7d21f25700 1 --2- 172.21.15.2:0/2332341199 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f7d1c070370 0x7f7d1c10ebc0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f7d10009f20 tx=0x7f7d10009cb0 comp rx=0 tx=0).stop 2022-05-13T23:15:15.764 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:15.759+0000 7f7d21f25700 1 -- 172.21.15.2:0/2332341199 shutdown_connections 2022-05-13T23:15:15.765 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:15.759+0000 7f7d21f25700 1 --2- 172.21.15.2:0/2332341199 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f7d04096ce0 0x7f7d04099190 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:15.765 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:15.759+0000 7f7d21f25700 1 --2- 172.21.15.2:0/2332341199 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7d1c103580 0x7f7d1c115c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:15.765 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:15.759+0000 7f7d21f25700 1 --2- 172.21.15.2:0/2332341199 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7d1c06edb0 0x7f7d1c10e680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:15.766 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:15.759+0000 7f7d21f25700 1 --2- 172.21.15.2:0/2332341199 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f7d1c070370 0x7f7d1c10ebc0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:15.766 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:15.759+0000 7f7d21f25700 1 -- 172.21.15.2:0/2332341199 >> 172.21.15.2:0/2332341199 conn(0x7f7d1c0f6e00 msgr2=0x7f7d1c0f91e0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-05-13T23:15:15.767 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:15.759+0000 7f7d21f25700 1 -- 172.21.15.2:0/2332341199 shutdown_connections 2022-05-13T23:15:15.767 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:15.759+0000 7f7d21f25700 1 -- 172.21.15.2:0/2332341199 wait complete. 2022-05-13T23:15:15.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:15:15 smithi047 ceph-mon[77147]: from='client.? 172.21.15.2:0/874032884' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2022-05-13T23:15:15.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:15:15 smithi115 ceph-mon[78593]: from='client.? 172.21.15.2:0/874032884' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2022-05-13T23:15:15.936 INFO:tasks.cephadm.ceph_manager.ceph:need seq 163208757299 got 163208757302 for osd.5 2022-05-13T23:15:15.936 DEBUG:teuthology.parallel:result is None 2022-05-13T23:15:16.365 INFO:tasks.cephadm.ceph_manager.ceph:need seq 283467841567 got 283467841569 for osd.9 2022-05-13T23:15:16.366 DEBUG:teuthology.parallel:result is None 2022-05-13T23:15:16.366 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2022-05-13T23:15:16.367 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:23b0658391ce94ca40799469eec00c28b298816c shell --fsid 16cf5f88-d311-11ec-be40-001a4aab830c -- ceph pg dump --format=json 2022-05-13T23:15:16.572 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:15:16 smithi002 ceph-mon[71816]: pgmap v320: 97 pgs: 97 active+clean; 584 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 10 op/s; 9 B/s, 1 keys/s, 0 objects/s recovering 2022-05-13T23:15:16.572 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:15:16 smithi002 ceph-mon[71816]: from='client.? 172.21.15.2:0/2332341199' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2022-05-13T23:15:16.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:15:16 smithi047 ceph-mon[77147]: pgmap v320: 97 pgs: 97 active+clean; 584 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 10 op/s; 9 B/s, 1 keys/s, 0 objects/s recovering 2022-05-13T23:15:16.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:15:16 smithi047 ceph-mon[77147]: from='client.? 172.21.15.2:0/2332341199' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2022-05-13T23:15:16.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:15:16 smithi115 ceph-mon[78593]: pgmap v320: 97 pgs: 97 active+clean; 584 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 10 op/s; 9 B/s, 1 keys/s, 0 objects/s recovering 2022-05-13T23:15:16.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:15:16 smithi115 ceph-mon[78593]: from='client.? 172.21.15.2:0/2332341199' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2022-05-13T23:15:17.039 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/16cf5f88-d311-11ec-be40-001a4aab830c/mon.a/config 2022-05-13T23:15:18.718 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:18.711+0000 7fa1b9977700 1 -- 172.21.15.2:0/341306445 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa1b40fb600 msgr2=0x7fa1b40fba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:15:18.719 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:18.711+0000 7fa1b9977700 1 --2- 172.21.15.2:0/341306445 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa1b40fb600 0x7fa1b40fba10 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fa1a4004870 tx=0x7fa1a4009cb0 comp rx=0 tx=0).stop 2022-05-13T23:15:18.720 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:18.711+0000 7fa1b9977700 1 -- 172.21.15.2:0/341306445 shutdown_connections 2022-05-13T23:15:18.721 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:18.711+0000 7fa1b9977700 1 --2- 172.21.15.2:0/341306445 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa1b40fd230 0x7fa1b4103af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:18.721 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:18.711+0000 7fa1b9977700 1 --2- 172.21.15.2:0/341306445 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa1b40fb600 0x7fa1b40fba10 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:18.722 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:18.711+0000 7fa1b9977700 1 --2- 172.21.15.2:0/341306445 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fa1b40fc8a0 0x7fa1b40fccf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:18.722 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:18.711+0000 7fa1b9977700 1 -- 172.21.15.2:0/341306445 >> 172.21.15.2:0/341306445 conn(0x7fa1b40f6df0 msgr2=0x7fa1b40f9220 unknown :-1 s=STATE_NONE l=0).mark_down 2022-05-13T23:15:18.722 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:18.711+0000 7fa1b9977700 1 -- 172.21.15.2:0/341306445 shutdown_connections 2022-05-13T23:15:18.723 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:18.712+0000 7fa1b9977700 1 -- 172.21.15.2:0/341306445 wait complete. 2022-05-13T23:15:18.723 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:18.712+0000 7fa1b9977700 1 Processor -- start 2022-05-13T23:15:18.724 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:18.713+0000 7fa1b9977700 1 -- start start 2022-05-13T23:15:18.724 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:18.713+0000 7fa1b9977700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fa1b40fb600 0x7fa1b410e690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:15:18.725 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:18.713+0000 7fa1b9977700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa1b40fc8a0 0x7fa1b410ebd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:15:18.725 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:18.713+0000 7fa1b9977700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa1b40fd230 0x7fa1b4115c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:15:18.725 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:18.713+0000 7fa1b9977700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1b41161a0 con 0x7fa1b40fc8a0 2022-05-13T23:15:18.726 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:18.713+0000 7fa1b9977700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1b4116310 con 0x7fa1b40fd230 2022-05-13T23:15:18.726 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:18.713+0000 7fa1b9977700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1b41164d0 con 0x7fa1b40fb600 2022-05-13T23:15:18.727 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:18.713+0000 7fa1b27fc700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa1b40fc8a0 0x7fa1b410ebd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:15:18.727 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:18.713+0000 7fa1b27fc700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa1b40fc8a0 0x7fa1b410ebd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:57596/0 (socket says 172.21.15.2:57596) 2022-05-13T23:15:18.728 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:18.713+0000 7fa1b27fc700 1 -- 172.21.15.2:0/1043561318 learned_addr learned my addr 172.21.15.2:0/1043561318 (peer_addr_for_me v2:172.21.15.2:0/0) 2022-05-13T23:15:18.728 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:18.713+0000 7fa1b37fe700 1 --2- 172.21.15.2:0/1043561318 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa1b40fd230 0x7fa1b4115c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:15:18.729 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:18.713+0000 7fa1b2ffd700 1 --2- 172.21.15.2:0/1043561318 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fa1b40fb600 0x7fa1b410e690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:15:18.729 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:18.714+0000 7fa1b37fe700 1 -- 172.21.15.2:0/1043561318 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fa1b40fb600 msgr2=0x7fa1b410e690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:15:18.730 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:18.714+0000 7fa1b37fe700 1 --2- 172.21.15.2:0/1043561318 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fa1b40fb600 0x7fa1b410e690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:18.730 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:18.714+0000 7fa1b37fe700 1 -- 172.21.15.2:0/1043561318 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa1b40fc8a0 msgr2=0x7fa1b410ebd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:15:18.731 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:18.714+0000 7fa1b37fe700 1 --2- 172.21.15.2:0/1043561318 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa1b40fc8a0 0x7fa1b410ebd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:18.731 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:18.714+0000 7fa1b37fe700 1 -- 172.21.15.2:0/1043561318 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1ac0097c0 con 0x7fa1b40fd230 2022-05-13T23:15:18.732 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:18.714+0000 7fa1b27fc700 1 --2- 172.21.15.2:0/1043561318 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa1b40fc8a0 0x7fa1b410ebd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-05-13T23:15:18.732 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:18.714+0000 7fa1b37fe700 1 --2- 172.21.15.2:0/1043561318 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa1b40fd230 0x7fa1b4115c60 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fa1ac001cb0 tx=0x7fa1ac004c50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-05-13T23:15:18.732 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:18.714+0000 7fa19b7fe700 1 -- 172.21.15.2:0/1043561318 <== mon.1 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1ac0092e0 con 0x7fa1b40fd230 2022-05-13T23:15:18.733 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:18.714+0000 7fa19b7fe700 1 -- 172.21.15.2:0/1043561318 <== mon.1 v2:172.21.15.47:3300/0 2 ==== config(26 keys) v1 ==== 1087+0+0 (secure 0 0 0) 0x7fa1ac009440 con 0x7fa1b40fd230 2022-05-13T23:15:18.733 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:18.714+0000 7fa1b9977700 1 -- 172.21.15.2:0/1043561318 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa1a4005040 con 0x7fa1b40fd230 2022-05-13T23:15:18.734 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:18.714+0000 7fa19b7fe700 1 -- 172.21.15.2:0/1043561318 <== mon.1 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1ac00fd50 con 0x7fa1b40fd230 2022-05-13T23:15:18.734 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:18.715+0000 7fa1b9977700 1 -- 172.21.15.2:0/1043561318 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa1b4116b90 con 0x7fa1b40fd230 2022-05-13T23:15:18.734 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:18.716+0000 7fa1b9977700 1 -- 172.21.15.2:0/1043561318 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa1b4106e60 con 0x7fa1b40fd230 2022-05-13T23:15:18.736 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:18.717+0000 7fa19b7fe700 1 -- 172.21.15.2:0/1043561318 <== mon.1 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 16) v1 ==== 125136+0+0 (secure 0 0 0) 0x7fa1ac004070 con 0x7fa1b40fd230 2022-05-13T23:15:18.737 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:18.719+0000 7fa19b7fe700 1 --2- 172.21.15.2:0/1043561318 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7fa19c096ce0 0x7fa19c099190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:15:18.737 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:18.719+0000 7fa19b7fe700 1 -- 172.21.15.2:0/1043561318 <== mon.1 v2:172.21.15.47:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8120+0+0 (secure 0 0 0) 0x7fa1ac0bf060 con 0x7fa1b40fd230 2022-05-13T23:15:18.737 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:18.719+0000 7fa1b2ffd700 1 --2- 172.21.15.2:0/1043561318 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7fa19c096ce0 0x7fa19c099190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:15:18.738 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:18.720+0000 7fa1b2ffd700 1 --2- 172.21.15.2:0/1043561318 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7fa19c096ce0 0x7fa19c099190 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fa1a4009da0 tx=0x7fa1a4009cb0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-05-13T23:15:18.738 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:18.721+0000 7fa19b7fe700 1 -- 172.21.15.2:0/1043561318 <== mon.1 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+165215 (secure 0 0 0) 0x7fa1ac08c310 con 0x7fa1b40fd230 2022-05-13T23:15:18.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:15:18 smithi047 ceph-mon[77147]: pgmap v321: 97 pgs: 97 active+clean; 584 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s; 8 B/s, 0 keys/s, 0 objects/s recovering 2022-05-13T23:15:18.822 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:15:18 smithi002 ceph-mon[71816]: pgmap v321: 97 pgs: 97 active+clean; 584 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s; 8 B/s, 0 keys/s, 0 objects/s recovering 2022-05-13T23:15:18.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:15:18 smithi115 ceph-mon[78593]: pgmap v321: 97 pgs: 97 active+clean; 584 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s; 8 B/s, 0 keys/s, 0 objects/s recovering 2022-05-13T23:15:18.913 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:18.910+0000 7fa1b9977700 1 -- 172.21.15.2:0/1043561318 --> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fa1b4103370 con 0x7fa19c096ce0 2022-05-13T23:15:18.924 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:18.922+0000 7fa19b7fe700 1 -- 172.21.15.2:0/1043561318 <== mgr.14148 v2:172.21.15.2:6800/2788339627 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+337838 (secure 0 0 0) 0x7fa1b4103370 con 0x7fa19c096ce0 2022-05-13T23:15:18.931 INFO:teuthology.orchestra.run.smithi002.stdout: 2022-05-13T23:15:18.936 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:18.929+0000 7fa1b9977700 1 -- 172.21.15.2:0/1043561318 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7fa19c096ce0 msgr2=0x7fa19c099190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:15:18.937 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:18.929+0000 7fa1b9977700 1 --2- 172.21.15.2:0/1043561318 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7fa19c096ce0 0x7fa19c099190 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fa1a4009da0 tx=0x7fa1a4009cb0 comp rx=0 tx=0).stop 2022-05-13T23:15:18.937 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:18.929+0000 7fa1b9977700 1 -- 172.21.15.2:0/1043561318 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa1b40fd230 msgr2=0x7fa1b4115c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:15:18.937 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:18.929+0000 7fa1b9977700 1 --2- 172.21.15.2:0/1043561318 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa1b40fd230 0x7fa1b4115c60 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fa1ac001cb0 tx=0x7fa1ac004c50 comp rx=0 tx=0).stop 2022-05-13T23:15:18.937 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:18.930+0000 7fa1b9977700 1 -- 172.21.15.2:0/1043561318 shutdown_connections 2022-05-13T23:15:18.938 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:18.930+0000 7fa1b9977700 1 --2- 172.21.15.2:0/1043561318 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa1b40fd230 0x7fa1b4115c60 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:18.938 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:18.930+0000 7fa1b9977700 1 --2- 172.21.15.2:0/1043561318 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fa1b40fb600 0x7fa1b410e690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:18.939 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:18.930+0000 7fa1b9977700 1 --2- 172.21.15.2:0/1043561318 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7fa19c096ce0 0x7fa19c099190 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:18.939 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:18.930+0000 7fa1b9977700 1 --2- 172.21.15.2:0/1043561318 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa1b40fc8a0 0x7fa1b410ebd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:18.939 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:18.930+0000 7fa1b9977700 1 -- 172.21.15.2:0/1043561318 >> 172.21.15.2:0/1043561318 conn(0x7fa1b40f6df0 msgr2=0x7fa1b40f91c0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-05-13T23:15:18.939 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:18.930+0000 7fa1b9977700 1 -- 172.21.15.2:0/1043561318 shutdown_connections 2022-05-13T23:15:18.940 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:18.930+0000 7fa1b9977700 1 -- 172.21.15.2:0/1043561318 wait complete. 2022-05-13T23:15:18.940 INFO:teuthology.orchestra.run.smithi002.stderr:dumped all 2022-05-13T23:15:19.500 INFO:teuthology.orchestra.run.smithi002.stdout:{"pg_ready":true,"pg_map":{"version":321,"stamp":"2022-05-13T23:15:17.249764+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":597994,"num_objects":100,"num_object_clones":0,"num_object_copies":300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":100,"num_whiteouts":0,"num_read":676,"num_read_kb":401,"num_write":386,"num_write_kb":2995,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":1057946,"num_keys_recovered":10,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":316,"ondisk_log_size":316,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":79744,"kb_used_data":12640,"kb_used_omap":0,"kb_used_meta":67008,"kb_avail":1124862080,"statfs":{"total":1151940427776,"available":1151858769920,"internally_reserved":0,"allocated":12943360,"data_stored":6908300,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":68616192},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":100,"num_read_kb":50,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":90,"num_keys_recovered":10,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.844597"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2022-05-13T23:15:05.411066+0000","last_change":"2022-05-13T23:14:16.267128+0000","last_active":"2022-05-13T23:15:05.411066+0000","last_peered":"2022-05-13T23:15:05.411066+0000","last_clean":"2022-05-13T23:15:05.411066+0000","last_became_active":"2022-05-13T23:14:16.266687+0000","last_became_peered":"2022-05-13T23:14:16.266687+0000","last_unstale":"2022-05-13T23:15:05.411066+0000","last_undegraded":"2022-05-13T23:15:05.411066+0000","last_fullsized":"2022-05-13T23:15:05.411066+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_clean_scrub_stamp":"2022-05-13T23:14:15.659362+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T04:40:46.150557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2022-05-13T23:15:05.514842+0000","last_change":"2022-05-13T23:14:16.272880+0000","last_active":"2022-05-13T23:15:05.514842+0000","last_peered":"2022-05-13T23:15:05.514842+0000","last_clean":"2022-05-13T23:15:05.514842+0000","last_became_active":"2022-05-13T23:14:16.272473+0000","last_became_peered":"2022-05-13T23:14:16.272473+0000","last_unstale":"2022-05-13T23:15:05.514842+0000","last_undegraded":"2022-05-13T23:15:05.514842+0000","last_fullsized":"2022-05-13T23:15:05.514842+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_clean_scrub_stamp":"2022-05-13T23:14:15.659362+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T00:32:19.185037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2022-05-13T23:15:05.515282+0000","last_change":"2022-05-13T23:14:16.270961+0000","last_active":"2022-05-13T23:15:05.515282+0000","last_peered":"2022-05-13T23:15:05.515282+0000","last_clean":"2022-05-13T23:15:05.515282+0000","last_became_active":"2022-05-13T23:14:16.269942+0000","last_became_peered":"2022-05-13T23:14:16.269942+0000","last_unstale":"2022-05-13T23:15:05.515282+0000","last_undegraded":"2022-05-13T23:15:05.515282+0000","last_fullsized":"2022-05-13T23:15:05.515282+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_clean_scrub_stamp":"2022-05-13T23:14:15.659362+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T10:03:03.774806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2022-05-13T23:15:05.412122+0000","last_change":"2022-05-13T23:14:16.255885+0000","last_active":"2022-05-13T23:15:05.412122+0000","last_peered":"2022-05-13T23:15:05.412122+0000","last_clean":"2022-05-13T23:15:05.412122+0000","last_became_active":"2022-05-13T23:14:16.255551+0000","last_became_peered":"2022-05-13T23:14:16.255551+0000","last_unstale":"2022-05-13T23:15:05.412122+0000","last_undegraded":"2022-05-13T23:15:05.412122+0000","last_fullsized":"2022-05-13T23:15:05.412122+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_clean_scrub_stamp":"2022-05-13T23:14:15.659362+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T10:26:40.578066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2022-05-13T23:15:05.411360+0000","last_change":"2022-05-13T23:14:16.271384+0000","last_active":"2022-05-13T23:15:05.411360+0000","last_peered":"2022-05-13T23:15:05.411360+0000","last_clean":"2022-05-13T23:15:05.411360+0000","last_became_active":"2022-05-13T23:14:16.271045+0000","last_became_peered":"2022-05-13T23:14:16.271045+0000","last_unstale":"2022-05-13T23:15:05.411360+0000","last_undegraded":"2022-05-13T23:15:05.411360+0000","last_fullsized":"2022-05-13T23:15:05.411360+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_clean_scrub_stamp":"2022-05-13T23:14:15.659362+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T04:51:50.618628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2022-05-13T23:15:05.411722+0000","last_change":"2022-05-13T23:14:16.275906+0000","last_active":"2022-05-13T23:15:05.411722+0000","last_peered":"2022-05-13T23:15:05.411722+0000","last_clean":"2022-05-13T23:15:05.411722+0000","last_became_active":"2022-05-13T23:14:16.275651+0000","last_became_peered":"2022-05-13T23:14:16.275651+0000","last_unstale":"2022-05-13T23:15:05.411722+0000","last_undegraded":"2022-05-13T23:15:05.411722+0000","last_fullsized":"2022-05-13T23:15:05.411722+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_clean_scrub_stamp":"2022-05-13T23:14:15.659362+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T02:29:33.019268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2022-05-13T23:15:05.408529+0000","last_change":"2022-05-13T23:14:16.276150+0000","last_active":"2022-05-13T23:15:05.408529+0000","last_peered":"2022-05-13T23:15:05.408529+0000","last_clean":"2022-05-13T23:15:05.408529+0000","last_became_active":"2022-05-13T23:14:16.275751+0000","last_became_peered":"2022-05-13T23:14:16.275751+0000","last_unstale":"2022-05-13T23:15:05.408529+0000","last_undegraded":"2022-05-13T23:15:05.408529+0000","last_fullsized":"2022-05-13T23:15:05.408529+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_clean_scrub_stamp":"2022-05-13T23:14:15.659362+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T04:01:48.496870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2022-05-13T23:15:05.416379+0000","last_change":"2022-05-13T23:14:16.268816+0000","last_active":"2022-05-13T23:15:05.416379+0000","last_peered":"2022-05-13T23:15:05.416379+0000","last_clean":"2022-05-13T23:15:05.416379+0000","last_became_active":"2022-05-13T23:14:16.268429+0000","last_became_peered":"2022-05-13T23:14:16.268429+0000","last_unstale":"2022-05-13T23:15:05.416379+0000","last_undegraded":"2022-05-13T23:15:05.416379+0000","last_fullsized":"2022-05-13T23:15:05.416379+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_clean_scrub_stamp":"2022-05-13T23:14:15.659362+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T00:14:51.685287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2022-05-13T23:15:05.405791+0000","last_change":"2022-05-13T23:14:16.263971+0000","last_active":"2022-05-13T23:15:05.405791+0000","last_peered":"2022-05-13T23:15:05.405791+0000","last_clean":"2022-05-13T23:15:05.405791+0000","last_became_active":"2022-05-13T23:14:16.263625+0000","last_became_peered":"2022-05-13T23:14:16.263625+0000","last_unstale":"2022-05-13T23:15:05.405791+0000","last_undegraded":"2022-05-13T23:15:05.405791+0000","last_fullsized":"2022-05-13T23:15:05.405791+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_clean_scrub_stamp":"2022-05-13T23:14:15.659362+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T03:11:11.037746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":22,"reported_epoch":85,"state":"active+clean","last_fresh":"2022-05-13T23:15:06.541777+0000","last_change":"2022-05-13T23:15:06.541777+0000","last_active":"2022-05-13T23:15:06.541777+0000","last_peered":"2022-05-13T23:15:06.541777+0000","last_clean":"2022-05-13T23:15:06.541777+0000","last_became_active":"2022-05-13T23:15:06.541253+0000","last_became_peered":"2022-05-13T23:15:06.541253+0000","last_unstale":"2022-05-13T23:15:06.541777+0000","last_undegraded":"2022-05-13T23:15:06.541777+0000","last_fullsized":"2022-05-13T23:15:06.541777+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_clean_scrub_stamp":"2022-05-13T23:14:15.659362+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T02:54:36.682507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,9],"acting":[10,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2022-05-13T23:15:05.408958+0000","last_change":"2022-05-13T23:14:16.267256+0000","last_active":"2022-05-13T23:15:05.408958+0000","last_peered":"2022-05-13T23:15:05.408958+0000","last_clean":"2022-05-13T23:15:05.408958+0000","last_became_active":"2022-05-13T23:14:16.266868+0000","last_became_peered":"2022-05-13T23:14:16.266868+0000","last_unstale":"2022-05-13T23:15:05.408958+0000","last_undegraded":"2022-05-13T23:15:05.408958+0000","last_fullsized":"2022-05-13T23:15:05.408958+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_clean_scrub_stamp":"2022-05-13T23:14:15.659362+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T03:28:58.592648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2022-05-13T23:15:05.412058+0000","last_change":"2022-05-13T23:14:16.256051+0000","last_active":"2022-05-13T23:15:05.412058+0000","last_peered":"2022-05-13T23:15:05.412058+0000","last_clean":"2022-05-13T23:15:05.412058+0000","last_became_active":"2022-05-13T23:14:16.255729+0000","last_became_peered":"2022-05-13T23:14:16.255729+0000","last_unstale":"2022-05-13T23:15:05.412058+0000","last_undegraded":"2022-05-13T23:15:05.412058+0000","last_fullsized":"2022-05-13T23:15:05.412058+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_clean_scrub_stamp":"2022-05-13T23:14:15.659362+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T09:34:22.448938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2022-05-13T23:15:05.406411+0000","last_change":"2022-05-13T23:14:16.266285+0000","last_active":"2022-05-13T23:15:05.406411+0000","last_peered":"2022-05-13T23:15:05.406411+0000","last_clean":"2022-05-13T23:15:05.406411+0000","last_became_active":"2022-05-13T23:14:16.264899+0000","last_became_peered":"2022-05-13T23:14:16.264899+0000","last_unstale":"2022-05-13T23:15:05.406411+0000","last_undegraded":"2022-05-13T23:15:05.406411+0000","last_fullsized":"2022-05-13T23:15:05.406411+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_clean_scrub_stamp":"2022-05-13T23:14:15.659362+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T09:50:56.211965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2022-05-13T23:15:05.514102+0000","last_change":"2022-05-13T23:14:16.271503+0000","last_active":"2022-05-13T23:15:05.514102+0000","last_peered":"2022-05-13T23:15:05.514102+0000","last_clean":"2022-05-13T23:15:05.514102+0000","last_became_active":"2022-05-13T23:14:16.271135+0000","last_became_peered":"2022-05-13T23:14:16.271135+0000","last_unstale":"2022-05-13T23:15:05.514102+0000","last_undegraded":"2022-05-13T23:15:05.514102+0000","last_fullsized":"2022-05-13T23:15:05.514102+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_clean_scrub_stamp":"2022-05-13T23:14:15.659362+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T05:01:25.159085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2022-05-13T23:15:05.513931+0000","last_change":"2022-05-13T23:14:16.265771+0000","last_active":"2022-05-13T23:15:05.513931+0000","last_peered":"2022-05-13T23:15:05.513931+0000","last_clean":"2022-05-13T23:15:05.513931+0000","last_became_active":"2022-05-13T23:14:16.265357+0000","last_became_peered":"2022-05-13T23:14:16.265357+0000","last_unstale":"2022-05-13T23:15:05.513931+0000","last_undegraded":"2022-05-13T23:15:05.513931+0000","last_fullsized":"2022-05-13T23:15:05.513931+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_clean_scrub_stamp":"2022-05-13T23:14:15.659362+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T02:16:47.449089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2022-05-13T23:15:05.405257+0000","last_change":"2022-05-13T23:14:16.270173+0000","last_active":"2022-05-13T23:15:05.405257+0000","last_peered":"2022-05-13T23:15:05.405257+0000","last_clean":"2022-05-13T23:15:05.405257+0000","last_became_active":"2022-05-13T23:14:16.269859+0000","last_became_peered":"2022-05-13T23:14:16.269859+0000","last_unstale":"2022-05-13T23:15:05.405257+0000","last_undegraded":"2022-05-13T23:15:05.405257+0000","last_fullsized":"2022-05-13T23:15:05.405257+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_clean_scrub_stamp":"2022-05-13T23:14:15.659362+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T03:31:45.566757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2022-05-13T23:15:05.410783+0000","last_change":"2022-05-13T23:14:16.269290+0000","last_active":"2022-05-13T23:15:05.410783+0000","last_peered":"2022-05-13T23:15:05.410783+0000","last_clean":"2022-05-13T23:15:05.410783+0000","last_became_active":"2022-05-13T23:14:16.268985+0000","last_became_peered":"2022-05-13T23:14:16.268985+0000","last_unstale":"2022-05-13T23:15:05.410783+0000","last_undegraded":"2022-05-13T23:15:05.410783+0000","last_fullsized":"2022-05-13T23:15:05.410783+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_clean_scrub_stamp":"2022-05-13T23:14:15.659362+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T03:58:45.885549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2022-05-13T23:15:05.411828+0000","last_change":"2022-05-13T23:14:16.266204+0000","last_active":"2022-05-13T23:15:05.411828+0000","last_peered":"2022-05-13T23:15:05.411828+0000","last_clean":"2022-05-13T23:15:05.411828+0000","last_became_active":"2022-05-13T23:14:16.262256+0000","last_became_peered":"2022-05-13T23:14:16.262256+0000","last_unstale":"2022-05-13T23:15:05.411828+0000","last_undegraded":"2022-05-13T23:15:05.411828+0000","last_fullsized":"2022-05-13T23:15:05.411828+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_clean_scrub_stamp":"2022-05-13T23:14:15.659362+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T00:53:26.594598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2022-05-13T23:15:05.417356+0000","last_change":"2022-05-13T23:14:16.252421+0000","last_active":"2022-05-13T23:15:05.417356+0000","last_peered":"2022-05-13T23:15:05.417356+0000","last_clean":"2022-05-13T23:15:05.417356+0000","last_became_active":"2022-05-13T23:14:16.250585+0000","last_became_peered":"2022-05-13T23:14:16.250585+0000","last_unstale":"2022-05-13T23:15:05.417356+0000","last_undegraded":"2022-05-13T23:15:05.417356+0000","last_fullsized":"2022-05-13T23:15:05.417356+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_clean_scrub_stamp":"2022-05-13T23:14:15.659362+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T00:13:24.393163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"83'13","reported_seq":81,"reported_epoch":85,"state":"active+clean","last_fresh":"2022-05-13T23:15:15.396211+0000","last_change":"2022-05-13T23:14:15.683837+0000","last_active":"2022-05-13T23:15:15.396211+0000","last_peered":"2022-05-13T23:15:15.396211+0000","last_clean":"2022-05-13T23:15:15.396211+0000","last_became_active":"2022-05-13T23:14:15.682976+0000","last_became_peered":"2022-05-13T23:14:15.682976+0000","last_unstale":"2022-05-13T23:15:15.396211+0000","last_undegraded":"2022-05-13T23:15:15.396211+0000","last_fullsized":"2022-05-13T23:15:15.396211+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:14.344206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:14.344206+0000","last_clean_scrub_stamp":"2022-05-13T23:14:14.344206+0000","objects_scrubbed":0,"log_size":13,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T01:05:36.612390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":644,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":55,"num_read_kb":55,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2022-05-13T23:15:05.417111+0000","last_change":"2022-05-13T23:14:16.264501+0000","last_active":"2022-05-13T23:15:05.417111+0000","last_peered":"2022-05-13T23:15:05.417111+0000","last_clean":"2022-05-13T23:15:05.417111+0000","last_became_active":"2022-05-13T23:14:16.264105+0000","last_became_peered":"2022-05-13T23:14:16.264105+0000","last_unstale":"2022-05-13T23:15:05.417111+0000","last_undegraded":"2022-05-13T23:15:05.417111+0000","last_fullsized":"2022-05-13T23:15:05.417111+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_clean_scrub_stamp":"2022-05-13T23:14:15.659362+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T00:21:48.857254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"83'2","reported_seq":15,"reported_epoch":84,"state":"active+clean","last_fresh":"2022-05-13T23:15:05.417387+0000","last_change":"2022-05-13T23:14:15.701757+0000","last_active":"2022-05-13T23:15:05.417387+0000","last_peered":"2022-05-13T23:15:05.417387+0000","last_clean":"2022-05-13T23:15:05.417387+0000","last_became_active":"2022-05-13T23:14:15.701428+0000","last_became_peered":"2022-05-13T23:14:15.701428+0000","last_unstale":"2022-05-13T23:15:05.417387+0000","last_undegraded":"2022-05-13T23:15:05.417387+0000","last_fullsized":"2022-05-13T23:15:05.417387+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:14.344206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:14.344206+0000","last_clean_scrub_stamp":"2022-05-13T23:14:14.344206+0000","objects_scrubbed":0,"log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T05:26:07.810433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2022-05-13T23:15:05.515198+0000","last_change":"2022-05-13T23:14:16.280002+0000","last_active":"2022-05-13T23:15:05.515198+0000","last_peered":"2022-05-13T23:15:05.515198+0000","last_clean":"2022-05-13T23:15:05.515198+0000","last_became_active":"2022-05-13T23:14:16.279621+0000","last_became_peered":"2022-05-13T23:14:16.279621+0000","last_unstale":"2022-05-13T23:15:05.515198+0000","last_undegraded":"2022-05-13T23:15:05.515198+0000","last_fullsized":"2022-05-13T23:15:05.515198+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_clean_scrub_stamp":"2022-05-13T23:14:15.659362+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T06:42:49.090643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"83'12","reported_seq":26,"reported_epoch":84,"state":"active+clean","last_fresh":"2022-05-13T23:15:05.411853+0000","last_change":"2022-05-13T23:14:15.683457+0000","last_active":"2022-05-13T23:15:05.411853+0000","last_peered":"2022-05-13T23:15:05.411853+0000","last_clean":"2022-05-13T23:15:05.411853+0000","last_became_active":"2022-05-13T23:14:15.682883+0000","last_became_peered":"2022-05-13T23:14:15.682883+0000","last_unstale":"2022-05-13T23:15:05.411853+0000","last_undegraded":"2022-05-13T23:15:05.411853+0000","last_fullsized":"2022-05-13T23:15:05.411853+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:14.344206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:14.344206+0000","last_clean_scrub_stamp":"2022-05-13T23:14:14.344206+0000","objects_scrubbed":0,"log_size":12,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T07:38:58.996672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":16,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2022-05-13T23:15:05.405436+0000","last_change":"2022-05-13T23:14:16.261315+0000","last_active":"2022-05-13T23:15:05.405436+0000","last_peered":"2022-05-13T23:15:05.405436+0000","last_clean":"2022-05-13T23:15:05.405436+0000","last_became_active":"2022-05-13T23:14:16.260265+0000","last_became_peered":"2022-05-13T23:14:16.260265+0000","last_unstale":"2022-05-13T23:15:05.405436+0000","last_undegraded":"2022-05-13T23:15:05.405436+0000","last_fullsized":"2022-05-13T23:15:05.405436+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_clean_scrub_stamp":"2022-05-13T23:14:15.659362+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T08:02:58.080071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"83'5","reported_seq":19,"reported_epoch":84,"state":"active+clean","last_fresh":"2022-05-13T23:15:05.412036+0000","last_change":"2022-05-13T23:14:15.682693+0000","last_active":"2022-05-13T23:15:05.412036+0000","last_peered":"2022-05-13T23:15:05.412036+0000","last_clean":"2022-05-13T23:15:05.412036+0000","last_became_active":"2022-05-13T23:14:15.682414+0000","last_became_peered":"2022-05-13T23:14:15.682414+0000","last_unstale":"2022-05-13T23:15:05.412036+0000","last_undegraded":"2022-05-13T23:15:05.412036+0000","last_fullsized":"2022-05-13T23:15:05.412036+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:14.344206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:14.344206+0000","last_clean_scrub_stamp":"2022-05-13T23:14:14.344206+0000","objects_scrubbed":0,"log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T09:15:17.699871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":554,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2022-05-13T23:15:05.406324+0000","last_change":"2022-05-13T23:14:16.261044+0000","last_active":"2022-05-13T23:15:05.406324+0000","last_peered":"2022-05-13T23:15:05.406324+0000","last_clean":"2022-05-13T23:15:05.406324+0000","last_became_active":"2022-05-13T23:14:16.260687+0000","last_became_peered":"2022-05-13T23:14:16.260687+0000","last_unstale":"2022-05-13T23:15:05.406324+0000","last_undegraded":"2022-05-13T23:15:05.406324+0000","last_fullsized":"2022-05-13T23:15:05.406324+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_clean_scrub_stamp":"2022-05-13T23:14:15.659362+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-14T23:38:23.413188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"83'5","reported_seq":71,"reported_epoch":85,"state":"active+clean","last_fresh":"2022-05-13T23:15:12.393296+0000","last_change":"2022-05-13T23:14:15.678240+0000","last_active":"2022-05-13T23:15:12.393296+0000","last_peered":"2022-05-13T23:15:12.393296+0000","last_clean":"2022-05-13T23:15:12.393296+0000","last_became_active":"2022-05-13T23:14:15.677918+0000","last_became_peered":"2022-05-13T23:14:15.677918+0000","last_unstale":"2022-05-13T23:15:12.393296+0000","last_undegraded":"2022-05-13T23:15:12.393296+0000","last_fullsized":"2022-05-13T23:15:12.393296+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:14.344206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:14.344206+0000","last_clean_scrub_stamp":"2022-05-13T23:14:14.344206+0000","objects_scrubbed":0,"log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T09:53:37.623728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":574,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":53,"num_read_kb":2,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2022-05-13T23:15:05.411315+0000","last_change":"2022-05-13T23:14:16.273537+0000","last_active":"2022-05-13T23:15:05.411315+0000","last_peered":"2022-05-13T23:15:05.411315+0000","last_clean":"2022-05-13T23:15:05.411315+0000","last_became_active":"2022-05-13T23:14:16.273127+0000","last_became_peered":"2022-05-13T23:14:16.273127+0000","last_unstale":"2022-05-13T23:15:05.411315+0000","last_undegraded":"2022-05-13T23:15:05.411315+0000","last_fullsized":"2022-05-13T23:15:05.411315+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_clean_scrub_stamp":"2022-05-13T23:14:15.659362+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T01:00:18.643721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"83'7","reported_seq":20,"reported_epoch":84,"state":"active+clean","last_fresh":"2022-05-13T23:15:05.411760+0000","last_change":"2022-05-13T23:14:15.680562+0000","last_active":"2022-05-13T23:15:05.411760+0000","last_peered":"2022-05-13T23:15:05.411760+0000","last_clean":"2022-05-13T23:15:05.411760+0000","last_became_active":"2022-05-13T23:14:15.680163+0000","last_became_peered":"2022-05-13T23:14:15.680163+0000","last_unstale":"2022-05-13T23:15:05.411760+0000","last_undegraded":"2022-05-13T23:15:05.411760+0000","last_fullsized":"2022-05-13T23:15:05.411760+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:14.344206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:14.344206+0000","last_clean_scrub_stamp":"2022-05-13T23:14:14.344206+0000","objects_scrubbed":0,"log_size":7,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T02:39:27.895407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2022-05-13T23:15:05.406200+0000","last_change":"2022-05-13T23:14:16.270878+0000","last_active":"2022-05-13T23:15:05.406200+0000","last_peered":"2022-05-13T23:15:05.406200+0000","last_clean":"2022-05-13T23:15:05.406200+0000","last_became_active":"2022-05-13T23:14:16.270416+0000","last_became_peered":"2022-05-13T23:14:16.270416+0000","last_unstale":"2022-05-13T23:15:05.406200+0000","last_undegraded":"2022-05-13T23:15:05.406200+0000","last_fullsized":"2022-05-13T23:15:05.406200+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_clean_scrub_stamp":"2022-05-13T23:14:15.659362+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T06:11:33.932416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"83'3","reported_seq":16,"reported_epoch":84,"state":"active+clean","last_fresh":"2022-05-13T23:15:05.406038+0000","last_change":"2022-05-13T23:14:15.676518+0000","last_active":"2022-05-13T23:15:05.406038+0000","last_peered":"2022-05-13T23:15:05.406038+0000","last_clean":"2022-05-13T23:15:05.406038+0000","last_became_active":"2022-05-13T23:14:15.676157+0000","last_became_peered":"2022-05-13T23:14:15.676157+0000","last_unstale":"2022-05-13T23:15:05.406038+0000","last_undegraded":"2022-05-13T23:15:05.406038+0000","last_fullsized":"2022-05-13T23:15:05.406038+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:14.344206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:14.344206+0000","last_clean_scrub_stamp":"2022-05-13T23:14:14.344206+0000","objects_scrubbed":0,"log_size":3,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T10:16:49.469347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2022-05-13T23:15:05.416449+0000","last_change":"2022-05-13T23:14:16.257424+0000","last_active":"2022-05-13T23:15:05.416449+0000","last_peered":"2022-05-13T23:15:05.416449+0000","last_clean":"2022-05-13T23:15:05.416449+0000","last_became_active":"2022-05-13T23:14:16.254550+0000","last_became_peered":"2022-05-13T23:14:16.254550+0000","last_unstale":"2022-05-13T23:15:05.416449+0000","last_undegraded":"2022-05-13T23:15:05.416449+0000","last_fullsized":"2022-05-13T23:15:05.416449+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_clean_scrub_stamp":"2022-05-13T23:14:15.659362+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T00:04:50.150368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"83'3","reported_seq":17,"reported_epoch":84,"state":"active+clean","last_fresh":"2022-05-13T23:15:05.405880+0000","last_change":"2022-05-13T23:14:15.680378+0000","last_active":"2022-05-13T23:15:05.405880+0000","last_peered":"2022-05-13T23:15:05.405880+0000","last_clean":"2022-05-13T23:15:05.405880+0000","last_became_active":"2022-05-13T23:14:15.680063+0000","last_became_peered":"2022-05-13T23:14:15.680063+0000","last_unstale":"2022-05-13T23:15:05.405880+0000","last_undegraded":"2022-05-13T23:15:05.405880+0000","last_fullsized":"2022-05-13T23:15:05.405880+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:14.344206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:14.344206+0000","last_clean_scrub_stamp":"2022-05-13T23:14:14.344206+0000","objects_scrubbed":0,"log_size":3,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-14T23:52:48.817491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":554,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2022-05-13T23:15:05.411502+0000","last_change":"2022-05-13T23:14:16.277131+0000","last_active":"2022-05-13T23:15:05.411502+0000","last_peered":"2022-05-13T23:15:05.411502+0000","last_clean":"2022-05-13T23:15:05.411502+0000","last_became_active":"2022-05-13T23:14:16.276815+0000","last_became_peered":"2022-05-13T23:14:16.276815+0000","last_unstale":"2022-05-13T23:15:05.411502+0000","last_undegraded":"2022-05-13T23:15:05.411502+0000","last_fullsized":"2022-05-13T23:15:05.411502+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_clean_scrub_stamp":"2022-05-13T23:14:15.659362+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-14T23:22:29.938073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"83'4","reported_seq":119,"reported_epoch":85,"state":"active+clean","last_fresh":"2022-05-13T23:15:13.389853+0000","last_change":"2022-05-13T23:14:15.683981+0000","last_active":"2022-05-13T23:15:13.389853+0000","last_peered":"2022-05-13T23:15:13.389853+0000","last_clean":"2022-05-13T23:15:13.389853+0000","last_became_active":"2022-05-13T23:14:15.683560+0000","last_became_peered":"2022-05-13T23:14:15.683560+0000","last_unstale":"2022-05-13T23:15:13.389853+0000","last_undegraded":"2022-05-13T23:15:13.389853+0000","last_fullsized":"2022-05-13T23:15:13.389853+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:14.344206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:14.344206+0000","last_clean_scrub_stamp":"2022-05-13T23:14:14.344206+0000","objects_scrubbed":0,"log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T00:30:01.510592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":102,"num_read_kb":102,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2022-05-13T23:15:05.405763+0000","last_change":"2022-05-13T23:14:16.269244+0000","last_active":"2022-05-13T23:15:05.405763+0000","last_peered":"2022-05-13T23:15:05.405763+0000","last_clean":"2022-05-13T23:15:05.405763+0000","last_became_active":"2022-05-13T23:14:16.268949+0000","last_became_peered":"2022-05-13T23:14:16.268949+0000","last_unstale":"2022-05-13T23:15:05.405763+0000","last_undegraded":"2022-05-13T23:15:05.405763+0000","last_fullsized":"2022-05-13T23:15:05.405763+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_clean_scrub_stamp":"2022-05-13T23:14:15.659362+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T04:34:53.311191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"83'4","reported_seq":17,"reported_epoch":84,"state":"active+clean","last_fresh":"2022-05-13T23:15:05.408697+0000","last_change":"2022-05-13T23:14:15.683758+0000","last_active":"2022-05-13T23:15:05.408697+0000","last_peered":"2022-05-13T23:15:05.408697+0000","last_clean":"2022-05-13T23:15:05.408697+0000","last_became_active":"2022-05-13T23:14:15.683356+0000","last_became_peered":"2022-05-13T23:14:15.683356+0000","last_unstale":"2022-05-13T23:15:05.408697+0000","last_undegraded":"2022-05-13T23:15:05.408697+0000","last_fullsized":"2022-05-13T23:15:05.408697+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:14.344206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:14.344206+0000","last_clean_scrub_stamp":"2022-05-13T23:14:14.344206+0000","objects_scrubbed":0,"log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T08:00:55.113935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2022-05-13T23:15:05.515128+0000","last_change":"2022-05-13T23:14:16.268751+0000","last_active":"2022-05-13T23:15:05.515128+0000","last_peered":"2022-05-13T23:15:05.515128+0000","last_clean":"2022-05-13T23:15:05.515128+0000","last_became_active":"2022-05-13T23:14:16.268332+0000","last_became_peered":"2022-05-13T23:14:16.268332+0000","last_unstale":"2022-05-13T23:15:05.515128+0000","last_undegraded":"2022-05-13T23:15:05.515128+0000","last_fullsized":"2022-05-13T23:15:05.515128+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_clean_scrub_stamp":"2022-05-13T23:14:15.659362+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T06:22:45.140343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"83'5","reported_seq":19,"reported_epoch":84,"state":"active+clean","last_fresh":"2022-05-13T23:15:05.406410+0000","last_change":"2022-05-13T23:14:15.701120+0000","last_active":"2022-05-13T23:15:05.406410+0000","last_peered":"2022-05-13T23:15:05.406410+0000","last_clean":"2022-05-13T23:15:05.406410+0000","last_became_active":"2022-05-13T23:14:15.700812+0000","last_became_peered":"2022-05-13T23:14:15.700812+0000","last_unstale":"2022-05-13T23:15:05.406410+0000","last_undegraded":"2022-05-13T23:15:05.406410+0000","last_fullsized":"2022-05-13T23:15:05.406410+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:14.344206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:14.344206+0000","last_clean_scrub_stamp":"2022-05-13T23:14:14.344206+0000","objects_scrubbed":0,"log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T02:16:02.909062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":644,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2022-05-13T23:15:05.405572+0000","last_change":"2022-05-13T23:14:16.265167+0000","last_active":"2022-05-13T23:15:05.405572+0000","last_peered":"2022-05-13T23:15:05.405572+0000","last_clean":"2022-05-13T23:15:05.405572+0000","last_became_active":"2022-05-13T23:14:16.264886+0000","last_became_peered":"2022-05-13T23:14:16.264886+0000","last_unstale":"2022-05-13T23:15:05.405572+0000","last_undegraded":"2022-05-13T23:15:05.405572+0000","last_fullsized":"2022-05-13T23:15:05.405572+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_clean_scrub_stamp":"2022-05-13T23:14:15.659362+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T00:26:06.650867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"83'10","reported_seq":127,"reported_epoch":85,"state":"active+clean","last_fresh":"2022-05-13T23:15:13.384338+0000","last_change":"2022-05-13T23:14:15.682385+0000","last_active":"2022-05-13T23:15:13.384338+0000","last_peered":"2022-05-13T23:15:13.384338+0000","last_clean":"2022-05-13T23:15:13.384338+0000","last_became_active":"2022-05-13T23:14:15.681996+0000","last_became_peered":"2022-05-13T23:14:15.681996+0000","last_unstale":"2022-05-13T23:15:13.384338+0000","last_undegraded":"2022-05-13T23:15:13.384338+0000","last_fullsized":"2022-05-13T23:15:13.384338+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:14.344206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:14.344206+0000","last_clean_scrub_stamp":"2022-05-13T23:14:14.344206+0000","objects_scrubbed":0,"log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-14T23:44:23.766036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1536,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":104,"num_read_kb":53,"num_write":12,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2022-05-13T23:15:05.411635+0000","last_change":"2022-05-13T23:14:16.263744+0000","last_active":"2022-05-13T23:15:05.411635+0000","last_peered":"2022-05-13T23:15:05.411635+0000","last_clean":"2022-05-13T23:15:05.411635+0000","last_became_active":"2022-05-13T23:14:16.263525+0000","last_became_peered":"2022-05-13T23:14:16.263525+0000","last_unstale":"2022-05-13T23:15:05.411635+0000","last_undegraded":"2022-05-13T23:15:05.411635+0000","last_fullsized":"2022-05-13T23:15:05.411635+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_clean_scrub_stamp":"2022-05-13T23:14:15.659362+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T04:23:44.856055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"83'9","reported_seq":24,"reported_epoch":85,"state":"active+clean","last_fresh":"2022-05-13T23:15:06.425091+0000","last_change":"2022-05-13T23:15:06.425091+0000","last_active":"2022-05-13T23:15:06.425091+0000","last_peered":"2022-05-13T23:15:06.425091+0000","last_clean":"2022-05-13T23:15:06.425091+0000","last_became_active":"2022-05-13T23:15:06.414405+0000","last_became_peered":"2022-05-13T23:15:06.414405+0000","last_unstale":"2022-05-13T23:15:06.425091+0000","last_undegraded":"2022-05-13T23:15:06.425091+0000","last_fullsized":"2022-05-13T23:15:06.425091+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:14.344206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:14.344206+0000","last_clean_scrub_stamp":"2022-05-13T23:14:14.344206+0000","objects_scrubbed":0,"log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T06:02:59.163940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":68,"num_keys_recovered":10,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,10,3],"acting":[11,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2022-05-13T23:15:05.406327+0000","last_change":"2022-05-13T23:14:16.263943+0000","last_active":"2022-05-13T23:15:05.406327+0000","last_peered":"2022-05-13T23:15:05.406327+0000","last_clean":"2022-05-13T23:15:05.406327+0000","last_became_active":"2022-05-13T23:14:16.263595+0000","last_became_peered":"2022-05-13T23:14:16.263595+0000","last_unstale":"2022-05-13T23:15:05.406327+0000","last_undegraded":"2022-05-13T23:15:05.406327+0000","last_fullsized":"2022-05-13T23:15:05.406327+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_clean_scrub_stamp":"2022-05-13T23:14:15.659362+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T04:57:04.201317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"83'5","reported_seq":19,"reported_epoch":84,"state":"active+clean","last_fresh":"2022-05-13T23:15:05.405936+0000","last_change":"2022-05-13T23:14:15.683005+0000","last_active":"2022-05-13T23:15:05.405936+0000","last_peered":"2022-05-13T23:15:05.405936+0000","last_clean":"2022-05-13T23:15:05.405936+0000","last_became_active":"2022-05-13T23:14:15.682692+0000","last_became_peered":"2022-05-13T23:14:15.682692+0000","last_unstale":"2022-05-13T23:15:05.405936+0000","last_undegraded":"2022-05-13T23:15:05.405936+0000","last_fullsized":"2022-05-13T23:15:05.405936+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:14.344206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:14.344206+0000","last_clean_scrub_stamp":"2022-05-13T23:14:14.344206+0000","objects_scrubbed":0,"log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T00:27:25.201141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2022-05-13T23:15:05.408693+0000","last_change":"2022-05-13T23:14:16.264351+0000","last_active":"2022-05-13T23:15:05.408693+0000","last_peered":"2022-05-13T23:15:05.408693+0000","last_clean":"2022-05-13T23:15:05.408693+0000","last_became_active":"2022-05-13T23:14:16.264023+0000","last_became_peered":"2022-05-13T23:14:16.264023+0000","last_unstale":"2022-05-13T23:15:05.408693+0000","last_undegraded":"2022-05-13T23:15:05.408693+0000","last_fullsized":"2022-05-13T23:15:05.408693+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_clean_scrub_stamp":"2022-05-13T23:14:15.659362+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T06:13:51.169049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"83'5","reported_seq":71,"reported_epoch":85,"state":"active+clean","last_fresh":"2022-05-13T23:15:12.388359+0000","last_change":"2022-05-13T23:14:15.677080+0000","last_active":"2022-05-13T23:15:12.388359+0000","last_peered":"2022-05-13T23:15:12.388359+0000","last_clean":"2022-05-13T23:15:12.388359+0000","last_became_active":"2022-05-13T23:14:15.676743+0000","last_became_peered":"2022-05-13T23:14:15.676743+0000","last_unstale":"2022-05-13T23:15:12.388359+0000","last_undegraded":"2022-05-13T23:15:12.388359+0000","last_fullsized":"2022-05-13T23:15:12.388359+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:14.344206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:14.344206+0000","last_clean_scrub_stamp":"2022-05-13T23:14:14.344206+0000","objects_scrubbed":0,"log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T09:05:30.321765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":540,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":54,"num_read_kb":3,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":22,"reported_epoch":85,"state":"active+clean","last_fresh":"2022-05-13T23:15:06.541517+0000","last_change":"2022-05-13T23:15:06.541517+0000","last_active":"2022-05-13T23:15:06.541517+0000","last_peered":"2022-05-13T23:15:06.541517+0000","last_clean":"2022-05-13T23:15:06.541517+0000","last_became_active":"2022-05-13T23:15:06.541056+0000","last_became_peered":"2022-05-13T23:15:06.541056+0000","last_unstale":"2022-05-13T23:15:06.541517+0000","last_undegraded":"2022-05-13T23:15:06.541517+0000","last_fullsized":"2022-05-13T23:15:06.541517+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_clean_scrub_stamp":"2022-05-13T23:14:15.659362+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T00:36:00.479693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,10],"acting":[7,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"83'17","reported_seq":31,"reported_epoch":84,"state":"active+clean","last_fresh":"2022-05-13T23:15:05.411453+0000","last_change":"2022-05-13T23:14:15.698465+0000","last_active":"2022-05-13T23:15:05.411453+0000","last_peered":"2022-05-13T23:15:05.411453+0000","last_clean":"2022-05-13T23:15:05.411453+0000","last_became_active":"2022-05-13T23:14:15.698131+0000","last_became_peered":"2022-05-13T23:14:15.698131+0000","last_unstale":"2022-05-13T23:15:05.411453+0000","last_undegraded":"2022-05-13T23:15:05.411453+0000","last_fullsized":"2022-05-13T23:15:05.411453+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:14.344206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:14.344206+0000","last_clean_scrub_stamp":"2022-05-13T23:14:14.344206+0000","objects_scrubbed":0,"log_size":17,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T03:03:48.104345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2022-05-13T23:15:05.405840+0000","last_change":"2022-05-13T23:14:16.260579+0000","last_active":"2022-05-13T23:15:05.405840+0000","last_peered":"2022-05-13T23:15:05.405840+0000","last_clean":"2022-05-13T23:15:05.405840+0000","last_became_active":"2022-05-13T23:14:16.260326+0000","last_became_peered":"2022-05-13T23:14:16.260326+0000","last_unstale":"2022-05-13T23:15:05.405840+0000","last_undegraded":"2022-05-13T23:15:05.405840+0000","last_fullsized":"2022-05-13T23:15:05.405840+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_clean_scrub_stamp":"2022-05-13T23:14:15.659362+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T06:53:23.092756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"83'2","reported_seq":16,"reported_epoch":85,"state":"active+clean","last_fresh":"2022-05-13T23:15:06.770067+0000","last_change":"2022-05-13T23:15:06.770067+0000","last_active":"2022-05-13T23:15:06.770067+0000","last_peered":"2022-05-13T23:15:06.770067+0000","last_clean":"2022-05-13T23:15:06.770067+0000","last_became_active":"2022-05-13T23:15:06.414616+0000","last_became_peered":"2022-05-13T23:15:06.414616+0000","last_unstale":"2022-05-13T23:15:06.770067+0000","last_undegraded":"2022-05-13T23:15:06.770067+0000","last_fullsized":"2022-05-13T23:15:06.770067+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:14.344206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:14.344206+0000","last_clean_scrub_stamp":"2022-05-13T23:14:14.344206+0000","objects_scrubbed":0,"log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T10:19:50.958083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":22,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2022-05-13T23:15:05.410109+0000","last_change":"2022-05-13T23:14:16.260574+0000","last_active":"2022-05-13T23:15:05.410109+0000","last_peered":"2022-05-13T23:15:05.410109+0000","last_clean":"2022-05-13T23:15:05.410109+0000","last_became_active":"2022-05-13T23:14:16.260258+0000","last_became_peered":"2022-05-13T23:14:16.260258+0000","last_unstale":"2022-05-13T23:15:05.410109+0000","last_undegraded":"2022-05-13T23:15:05.410109+0000","last_fullsized":"2022-05-13T23:15:05.410109+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_clean_scrub_stamp":"2022-05-13T23:14:15.659362+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T10:49:06.210598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"83'1","reported_seq":14,"reported_epoch":84,"state":"active+clean","last_fresh":"2022-05-13T23:15:05.411700+0000","last_change":"2022-05-13T23:14:15.701322+0000","last_active":"2022-05-13T23:15:05.411700+0000","last_peered":"2022-05-13T23:15:05.411700+0000","last_clean":"2022-05-13T23:15:05.411700+0000","last_became_active":"2022-05-13T23:14:15.701007+0000","last_became_peered":"2022-05-13T23:14:15.701007+0000","last_unstale":"2022-05-13T23:15:05.411700+0000","last_undegraded":"2022-05-13T23:15:05.411700+0000","last_fullsized":"2022-05-13T23:15:05.411700+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:14.344206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:14.344206+0000","last_clean_scrub_stamp":"2022-05-13T23:14:14.344206+0000","objects_scrubbed":0,"log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T00:13:11.071695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2022-05-13T23:15:05.412544+0000","last_change":"2022-05-13T23:14:16.249579+0000","last_active":"2022-05-13T23:15:05.412544+0000","last_peered":"2022-05-13T23:15:05.412544+0000","last_clean":"2022-05-13T23:15:05.412544+0000","last_became_active":"2022-05-13T23:14:16.249245+0000","last_became_peered":"2022-05-13T23:14:16.249245+0000","last_unstale":"2022-05-13T23:15:05.412544+0000","last_undegraded":"2022-05-13T23:15:05.412544+0000","last_fullsized":"2022-05-13T23:15:05.412544+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_clean_scrub_stamp":"2022-05-13T23:14:15.659362+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T00:26:53.404909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"83'5","reported_seq":71,"reported_epoch":85,"state":"active+clean","last_fresh":"2022-05-13T23:15:13.394081+0000","last_change":"2022-05-13T23:14:15.699433+0000","last_active":"2022-05-13T23:15:13.394081+0000","last_peered":"2022-05-13T23:15:13.394081+0000","last_clean":"2022-05-13T23:15:13.394081+0000","last_became_active":"2022-05-13T23:14:15.698920+0000","last_became_peered":"2022-05-13T23:14:15.698920+0000","last_unstale":"2022-05-13T23:15:13.394081+0000","last_undegraded":"2022-05-13T23:15:13.394081+0000","last_fullsized":"2022-05-13T23:15:13.394081+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:14.344206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:14.344206+0000","last_clean_scrub_stamp":"2022-05-13T23:14:14.344206+0000","objects_scrubbed":0,"log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T07:56:34.276565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":540,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":53,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2022-05-13T23:15:05.412760+0000","last_change":"2022-05-13T23:14:16.259866+0000","last_active":"2022-05-13T23:15:05.412760+0000","last_peered":"2022-05-13T23:15:05.412760+0000","last_clean":"2022-05-13T23:15:05.412760+0000","last_became_active":"2022-05-13T23:14:16.259531+0000","last_became_peered":"2022-05-13T23:14:16.259531+0000","last_unstale":"2022-05-13T23:15:05.412760+0000","last_undegraded":"2022-05-13T23:15:05.412760+0000","last_fullsized":"2022-05-13T23:15:05.412760+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_clean_scrub_stamp":"2022-05-13T23:14:15.659362+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T07:41:46.019217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2022-05-13T23:15:05.405937+0000","last_change":"2022-05-13T23:14:16.259613+0000","last_active":"2022-05-13T23:15:05.405937+0000","last_peered":"2022-05-13T23:15:05.405937+0000","last_clean":"2022-05-13T23:15:05.405937+0000","last_became_active":"2022-05-13T23:14:16.259274+0000","last_became_peered":"2022-05-13T23:14:16.259274+0000","last_unstale":"2022-05-13T23:15:05.405937+0000","last_undegraded":"2022-05-13T23:15:05.405937+0000","last_fullsized":"2022-05-13T23:15:05.405937+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_clean_scrub_stamp":"2022-05-13T23:14:15.659362+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T05:36:21.174814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"83'5","reported_seq":72,"reported_epoch":85,"state":"active+clean","last_fresh":"2022-05-13T23:15:14.390474+0000","last_change":"2022-05-13T23:14:15.701315+0000","last_active":"2022-05-13T23:15:14.390474+0000","last_peered":"2022-05-13T23:15:14.390474+0000","last_clean":"2022-05-13T23:15:14.390474+0000","last_became_active":"2022-05-13T23:14:15.700955+0000","last_became_peered":"2022-05-13T23:14:15.700955+0000","last_unstale":"2022-05-13T23:15:14.390474+0000","last_undegraded":"2022-05-13T23:15:14.390474+0000","last_fullsized":"2022-05-13T23:15:14.390474+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:14.344206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:14.344206+0000","last_clean_scrub_stamp":"2022-05-13T23:14:14.344206+0000","objects_scrubbed":0,"log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T10:17:23.916475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":54,"num_read_kb":54,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2022-05-13T23:15:05.416788+0000","last_change":"2022-05-13T23:14:16.272364+0000","last_active":"2022-05-13T23:15:05.416788+0000","last_peered":"2022-05-13T23:15:05.416788+0000","last_clean":"2022-05-13T23:15:05.416788+0000","last_became_active":"2022-05-13T23:14:16.271612+0000","last_became_peered":"2022-05-13T23:14:16.271612+0000","last_unstale":"2022-05-13T23:15:05.416788+0000","last_undegraded":"2022-05-13T23:15:05.416788+0000","last_fullsized":"2022-05-13T23:15:05.416788+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_clean_scrub_stamp":"2022-05-13T23:14:15.659362+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T07:25:47.039264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"83'6","reported_seq":22,"reported_epoch":84,"state":"active+clean","last_fresh":"2022-05-13T23:15:05.411756+0000","last_change":"2022-05-13T23:14:15.676251+0000","last_active":"2022-05-13T23:15:05.411756+0000","last_peered":"2022-05-13T23:15:05.411756+0000","last_clean":"2022-05-13T23:15:05.411756+0000","last_became_active":"2022-05-13T23:14:15.675895+0000","last_became_peered":"2022-05-13T23:14:15.675895+0000","last_unstale":"2022-05-13T23:15:05.411756+0000","last_undegraded":"2022-05-13T23:15:05.411756+0000","last_fullsized":"2022-05-13T23:15:05.411756+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:14.344206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:14.344206+0000","last_clean_scrub_stamp":"2022-05-13T23:14:14.344206+0000","objects_scrubbed":0,"log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T06:03:57.398647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":345,"reported_epoch":84,"state":"active+clean","last_fresh":"2022-05-13T23:15:05.409382+0000","last_change":"2022-05-13T23:13:02.805009+0000","last_active":"2022-05-13T23:15:05.409382+0000","last_peered":"2022-05-13T23:15:05.409382+0000","last_clean":"2022-05-13T23:15:05.409382+0000","last_became_active":"2022-05-13T23:13:02.804350+0000","last_became_peered":"2022-05-13T23:13:02.804350+0000","last_unstale":"2022-05-13T23:15:05.409382+0000","last_undegraded":"2022-05-13T23:15:05.409382+0000","last_fullsized":"2022-05-13T23:15:05.409382+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:09:32.001256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:09:32.001256+0000","last_clean_scrub_stamp":"2022-05-13T23:09:32.001256+0000","objects_scrubbed":0,"log_size":137,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T07:28:44.497144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1057856,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2022-05-13T23:15:05.405887+0000","last_change":"2022-05-13T23:14:16.261286+0000","last_active":"2022-05-13T23:15:05.405887+0000","last_peered":"2022-05-13T23:15:05.405887+0000","last_clean":"2022-05-13T23:15:05.405887+0000","last_became_active":"2022-05-13T23:14:16.260870+0000","last_became_peered":"2022-05-13T23:14:16.260870+0000","last_unstale":"2022-05-13T23:15:05.405887+0000","last_undegraded":"2022-05-13T23:15:05.405887+0000","last_fullsized":"2022-05-13T23:15:05.405887+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_clean_scrub_stamp":"2022-05-13T23:14:15.659362+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T08:14:34.804797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"83'4","reported_seq":17,"reported_epoch":84,"state":"active+clean","last_fresh":"2022-05-13T23:15:05.409557+0000","last_change":"2022-05-13T23:14:15.686878+0000","last_active":"2022-05-13T23:15:05.409557+0000","last_peered":"2022-05-13T23:15:05.409557+0000","last_clean":"2022-05-13T23:15:05.409557+0000","last_became_active":"2022-05-13T23:14:15.686496+0000","last_became_peered":"2022-05-13T23:14:15.686496+0000","last_unstale":"2022-05-13T23:15:05.409557+0000","last_undegraded":"2022-05-13T23:15:05.409557+0000","last_fullsized":"2022-05-13T23:15:05.409557+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:14.344206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:14.344206+0000","last_clean_scrub_stamp":"2022-05-13T23:14:14.344206+0000","objects_scrubbed":0,"log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T09:19:28.089159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"83'4","reported_seq":18,"reported_epoch":84,"state":"active+clean","last_fresh":"2022-05-13T23:15:05.412102+0000","last_change":"2022-05-13T23:14:15.698503+0000","last_active":"2022-05-13T23:15:05.412102+0000","last_peered":"2022-05-13T23:15:05.412102+0000","last_clean":"2022-05-13T23:15:05.412102+0000","last_became_active":"2022-05-13T23:14:15.698156+0000","last_became_peered":"2022-05-13T23:14:15.698156+0000","last_unstale":"2022-05-13T23:15:05.412102+0000","last_undegraded":"2022-05-13T23:15:05.412102+0000","last_fullsized":"2022-05-13T23:15:05.412102+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:14.344206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:14.344206+0000","last_clean_scrub_stamp":"2022-05-13T23:14:14.344206+0000","objects_scrubbed":0,"log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T09:03:21.133631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2022-05-13T23:15:05.406566+0000","last_change":"2022-05-13T23:14:16.270922+0000","last_active":"2022-05-13T23:15:05.406566+0000","last_peered":"2022-05-13T23:15:05.406566+0000","last_clean":"2022-05-13T23:15:05.406566+0000","last_became_active":"2022-05-13T23:14:16.270259+0000","last_became_peered":"2022-05-13T23:14:16.270259+0000","last_unstale":"2022-05-13T23:15:05.406566+0000","last_undegraded":"2022-05-13T23:15:05.406566+0000","last_fullsized":"2022-05-13T23:15:05.406566+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_clean_scrub_stamp":"2022-05-13T23:14:15.659362+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T02:48:22.255489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"83'7","reported_seq":20,"reported_epoch":84,"state":"active+clean","last_fresh":"2022-05-13T23:15:05.411879+0000","last_change":"2022-05-13T23:14:15.696304+0000","last_active":"2022-05-13T23:15:05.411879+0000","last_peered":"2022-05-13T23:15:05.411879+0000","last_clean":"2022-05-13T23:15:05.411879+0000","last_became_active":"2022-05-13T23:14:15.695476+0000","last_became_peered":"2022-05-13T23:14:15.695476+0000","last_unstale":"2022-05-13T23:15:05.411879+0000","last_undegraded":"2022-05-13T23:15:05.411879+0000","last_fullsized":"2022-05-13T23:15:05.411879+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:14.344206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:14.344206+0000","last_clean_scrub_stamp":"2022-05-13T23:14:14.344206+0000","objects_scrubbed":0,"log_size":7,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T08:48:32.931789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2022-05-13T23:15:05.417489+0000","last_change":"2022-05-13T23:14:16.257063+0000","last_active":"2022-05-13T23:15:05.417489+0000","last_peered":"2022-05-13T23:15:05.417489+0000","last_clean":"2022-05-13T23:15:05.417489+0000","last_became_active":"2022-05-13T23:14:16.256757+0000","last_became_peered":"2022-05-13T23:14:16.256757+0000","last_unstale":"2022-05-13T23:15:05.417489+0000","last_undegraded":"2022-05-13T23:15:05.417489+0000","last_fullsized":"2022-05-13T23:15:05.417489+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_clean_scrub_stamp":"2022-05-13T23:14:15.659362+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T09:24:32.369318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"83'7","reported_seq":21,"reported_epoch":84,"state":"active+clean","last_fresh":"2022-05-13T23:15:05.411692+0000","last_change":"2022-05-13T23:14:15.685568+0000","last_active":"2022-05-13T23:15:05.411692+0000","last_peered":"2022-05-13T23:15:05.411692+0000","last_clean":"2022-05-13T23:15:05.411692+0000","last_became_active":"2022-05-13T23:14:15.685203+0000","last_became_peered":"2022-05-13T23:14:15.685203+0000","last_unstale":"2022-05-13T23:15:05.411692+0000","last_undegraded":"2022-05-13T23:15:05.411692+0000","last_fullsized":"2022-05-13T23:15:05.411692+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:14.344206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:14.344206+0000","last_clean_scrub_stamp":"2022-05-13T23:14:14.344206+0000","objects_scrubbed":0,"log_size":7,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T03:55:33.271069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2022-05-13T23:15:05.514919+0000","last_change":"2022-05-13T23:14:16.280330+0000","last_active":"2022-05-13T23:15:05.514919+0000","last_peered":"2022-05-13T23:15:05.514919+0000","last_clean":"2022-05-13T23:15:05.514919+0000","last_became_active":"2022-05-13T23:14:16.279908+0000","last_became_peered":"2022-05-13T23:14:16.279908+0000","last_unstale":"2022-05-13T23:15:05.514919+0000","last_undegraded":"2022-05-13T23:15:05.514919+0000","last_fullsized":"2022-05-13T23:15:05.514919+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_clean_scrub_stamp":"2022-05-13T23:14:15.659362+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T03:25:42.803064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"83'2","reported_seq":15,"reported_epoch":84,"state":"active+clean","last_fresh":"2022-05-13T23:15:05.405262+0000","last_change":"2022-05-13T23:14:15.698443+0000","last_active":"2022-05-13T23:15:05.405262+0000","last_peered":"2022-05-13T23:15:05.405262+0000","last_clean":"2022-05-13T23:15:05.405262+0000","last_became_active":"2022-05-13T23:14:15.698158+0000","last_became_peered":"2022-05-13T23:14:15.698158+0000","last_unstale":"2022-05-13T23:15:05.405262+0000","last_undegraded":"2022-05-13T23:15:05.405262+0000","last_fullsized":"2022-05-13T23:15:05.405262+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:14.344206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:14.344206+0000","last_clean_scrub_stamp":"2022-05-13T23:14:14.344206+0000","objects_scrubbed":0,"log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T05:06:14.946233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2022-05-13T23:15:05.411226+0000","last_change":"2022-05-13T23:14:16.264110+0000","last_active":"2022-05-13T23:15:05.411226+0000","last_peered":"2022-05-13T23:15:05.411226+0000","last_clean":"2022-05-13T23:15:05.411226+0000","last_became_active":"2022-05-13T23:14:16.263830+0000","last_became_peered":"2022-05-13T23:14:16.263830+0000","last_unstale":"2022-05-13T23:15:05.411226+0000","last_undegraded":"2022-05-13T23:15:05.411226+0000","last_fullsized":"2022-05-13T23:15:05.411226+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_clean_scrub_stamp":"2022-05-13T23:14:15.659362+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T00:06:26.724597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"83'7","reported_seq":20,"reported_epoch":84,"state":"active+clean","last_fresh":"2022-05-13T23:15:05.416532+0000","last_change":"2022-05-13T23:14:15.693926+0000","last_active":"2022-05-13T23:15:05.416532+0000","last_peered":"2022-05-13T23:15:05.416532+0000","last_clean":"2022-05-13T23:15:05.416532+0000","last_became_active":"2022-05-13T23:14:15.692163+0000","last_became_peered":"2022-05-13T23:14:15.692163+0000","last_unstale":"2022-05-13T23:15:05.416532+0000","last_undegraded":"2022-05-13T23:15:05.416532+0000","last_fullsized":"2022-05-13T23:15:05.416532+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:14.344206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:14.344206+0000","last_clean_scrub_stamp":"2022-05-13T23:14:14.344206+0000","objects_scrubbed":0,"log_size":7,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T03:00:12.522714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2022-05-13T23:15:05.409774+0000","last_change":"2022-05-13T23:14:16.265409+0000","last_active":"2022-05-13T23:15:05.409774+0000","last_peered":"2022-05-13T23:15:05.409774+0000","last_clean":"2022-05-13T23:15:05.409774+0000","last_became_active":"2022-05-13T23:14:16.265151+0000","last_became_peered":"2022-05-13T23:14:16.265151+0000","last_unstale":"2022-05-13T23:15:05.409774+0000","last_undegraded":"2022-05-13T23:15:05.409774+0000","last_fullsized":"2022-05-13T23:15:05.409774+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_clean_scrub_stamp":"2022-05-13T23:14:15.659362+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T04:05:43.762623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"83'3","reported_seq":16,"reported_epoch":84,"state":"active+clean","last_fresh":"2022-05-13T23:15:05.417167+0000","last_change":"2022-05-13T23:14:15.681975+0000","last_active":"2022-05-13T23:15:05.417167+0000","last_peered":"2022-05-13T23:15:05.417167+0000","last_clean":"2022-05-13T23:15:05.417167+0000","last_became_active":"2022-05-13T23:14:15.681391+0000","last_became_peered":"2022-05-13T23:14:15.681391+0000","last_unstale":"2022-05-13T23:15:05.417167+0000","last_undegraded":"2022-05-13T23:15:05.417167+0000","last_fullsized":"2022-05-13T23:15:05.417167+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:14.344206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:14.344206+0000","last_clean_scrub_stamp":"2022-05-13T23:14:14.344206+0000","objects_scrubbed":0,"log_size":3,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T06:56:25.732024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2022-05-13T23:15:05.416655+0000","last_change":"2022-05-13T23:14:16.272443+0000","last_active":"2022-05-13T23:15:05.416655+0000","last_peered":"2022-05-13T23:15:05.416655+0000","last_clean":"2022-05-13T23:15:05.416655+0000","last_became_active":"2022-05-13T23:14:16.272031+0000","last_became_peered":"2022-05-13T23:14:16.272031+0000","last_unstale":"2022-05-13T23:15:05.416655+0000","last_undegraded":"2022-05-13T23:15:05.416655+0000","last_fullsized":"2022-05-13T23:15:05.416655+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_clean_scrub_stamp":"2022-05-13T23:14:15.659362+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T10:23:55.694108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"83'4","reported_seq":18,"reported_epoch":84,"state":"active+clean","last_fresh":"2022-05-13T23:15:05.411301+0000","last_change":"2022-05-13T23:14:15.681877+0000","last_active":"2022-05-13T23:15:05.411301+0000","last_peered":"2022-05-13T23:15:05.411301+0000","last_clean":"2022-05-13T23:15:05.411301+0000","last_became_active":"2022-05-13T23:14:15.681579+0000","last_became_peered":"2022-05-13T23:14:15.681579+0000","last_unstale":"2022-05-13T23:15:05.411301+0000","last_undegraded":"2022-05-13T23:15:05.411301+0000","last_fullsized":"2022-05-13T23:15:05.411301+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:14.344206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:14.344206+0000","last_clean_scrub_stamp":"2022-05-13T23:14:14.344206+0000","objects_scrubbed":0,"log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T07:00:11.132049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2022-05-13T23:15:05.408596+0000","last_change":"2022-05-13T23:14:16.266877+0000","last_active":"2022-05-13T23:15:05.408596+0000","last_peered":"2022-05-13T23:15:05.408596+0000","last_clean":"2022-05-13T23:15:05.408596+0000","last_became_active":"2022-05-13T23:14:16.266551+0000","last_became_peered":"2022-05-13T23:14:16.266551+0000","last_unstale":"2022-05-13T23:15:05.408596+0000","last_undegraded":"2022-05-13T23:15:05.408596+0000","last_fullsized":"2022-05-13T23:15:05.408596+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_clean_scrub_stamp":"2022-05-13T23:14:15.659362+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T05:57:29.394410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"83'5","reported_seq":73,"reported_epoch":85,"state":"active+clean","last_fresh":"2022-05-13T23:15:14.391995+0000","last_change":"2022-05-13T23:14:15.699586+0000","last_active":"2022-05-13T23:15:14.391995+0000","last_peered":"2022-05-13T23:15:14.391995+0000","last_clean":"2022-05-13T23:15:14.391995+0000","last_became_active":"2022-05-13T23:14:15.699215+0000","last_became_peered":"2022-05-13T23:14:15.699215+0000","last_unstale":"2022-05-13T23:15:14.391995+0000","last_undegraded":"2022-05-13T23:15:14.391995+0000","last_fullsized":"2022-05-13T23:15:14.391995+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:14.344206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:14.344206+0000","last_clean_scrub_stamp":"2022-05-13T23:14:14.344206+0000","objects_scrubbed":0,"log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T04:14:54.395550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1094,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":55,"num_read_kb":2,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2022-05-13T23:15:05.408938+0000","last_change":"2022-05-13T23:14:16.265730+0000","last_active":"2022-05-13T23:15:05.408938+0000","last_peered":"2022-05-13T23:15:05.408938+0000","last_clean":"2022-05-13T23:15:05.408938+0000","last_became_active":"2022-05-13T23:14:16.265435+0000","last_became_peered":"2022-05-13T23:14:16.265435+0000","last_unstale":"2022-05-13T23:15:05.408938+0000","last_undegraded":"2022-05-13T23:15:05.408938+0000","last_fullsized":"2022-05-13T23:15:05.408938+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_clean_scrub_stamp":"2022-05-13T23:14:15.659362+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T08:55:57.574132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"83'3","reported_seq":17,"reported_epoch":84,"state":"active+clean","last_fresh":"2022-05-13T23:15:05.406349+0000","last_change":"2022-05-13T23:14:15.682402+0000","last_active":"2022-05-13T23:15:05.406349+0000","last_peered":"2022-05-13T23:15:05.406349+0000","last_clean":"2022-05-13T23:15:05.406349+0000","last_became_active":"2022-05-13T23:14:15.682105+0000","last_became_peered":"2022-05-13T23:14:15.682105+0000","last_unstale":"2022-05-13T23:15:05.406349+0000","last_undegraded":"2022-05-13T23:15:05.406349+0000","last_fullsized":"2022-05-13T23:15:05.406349+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:14.344206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:14.344206+0000","last_clean_scrub_stamp":"2022-05-13T23:14:14.344206+0000","objects_scrubbed":0,"log_size":3,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T10:37:55.092658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2022-05-13T23:15:05.408714+0000","last_change":"2022-05-13T23:14:16.263001+0000","last_active":"2022-05-13T23:15:05.408714+0000","last_peered":"2022-05-13T23:15:05.408714+0000","last_clean":"2022-05-13T23:15:05.408714+0000","last_became_active":"2022-05-13T23:14:16.262646+0000","last_became_peered":"2022-05-13T23:14:16.262646+0000","last_unstale":"2022-05-13T23:15:05.408714+0000","last_undegraded":"2022-05-13T23:15:05.408714+0000","last_fullsized":"2022-05-13T23:15:05.408714+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_clean_scrub_stamp":"2022-05-13T23:14:15.659362+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T05:38:53.024966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"83'5","reported_seq":18,"reported_epoch":84,"state":"active+clean","last_fresh":"2022-05-13T23:15:05.406697+0000","last_change":"2022-05-13T23:14:15.690434+0000","last_active":"2022-05-13T23:15:05.406697+0000","last_peered":"2022-05-13T23:15:05.406697+0000","last_clean":"2022-05-13T23:15:05.406697+0000","last_became_active":"2022-05-13T23:14:15.690032+0000","last_became_peered":"2022-05-13T23:14:15.690032+0000","last_unstale":"2022-05-13T23:15:05.406697+0000","last_undegraded":"2022-05-13T23:15:05.406697+0000","last_fullsized":"2022-05-13T23:15:05.406697+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:14.344206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:14.344206+0000","last_clean_scrub_stamp":"2022-05-13T23:14:14.344206+0000","objects_scrubbed":0,"log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T06:39:08.251311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2022-05-13T23:15:05.409028+0000","last_change":"2022-05-13T23:14:16.264791+0000","last_active":"2022-05-13T23:15:05.409028+0000","last_peered":"2022-05-13T23:15:05.409028+0000","last_clean":"2022-05-13T23:15:05.409028+0000","last_became_active":"2022-05-13T23:14:16.264461+0000","last_became_peered":"2022-05-13T23:14:16.264461+0000","last_unstale":"2022-05-13T23:15:05.409028+0000","last_undegraded":"2022-05-13T23:15:05.409028+0000","last_fullsized":"2022-05-13T23:15:05.409028+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_clean_scrub_stamp":"2022-05-13T23:14:15.659362+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T03:02:04.280220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2022-05-13T23:15:05.409787+0000","last_change":"2022-05-13T23:14:16.260468+0000","last_active":"2022-05-13T23:15:05.409787+0000","last_peered":"2022-05-13T23:15:05.409787+0000","last_clean":"2022-05-13T23:15:05.409787+0000","last_became_active":"2022-05-13T23:14:16.260211+0000","last_became_peered":"2022-05-13T23:14:16.260211+0000","last_unstale":"2022-05-13T23:15:05.409787+0000","last_undegraded":"2022-05-13T23:15:05.409787+0000","last_fullsized":"2022-05-13T23:15:05.409787+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_clean_scrub_stamp":"2022-05-13T23:14:15.659362+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T05:26:57.769099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2022-05-13T23:15:05.417082+0000","last_change":"2022-05-13T23:14:16.253997+0000","last_active":"2022-05-13T23:15:05.417082+0000","last_peered":"2022-05-13T23:15:05.417082+0000","last_clean":"2022-05-13T23:15:05.417082+0000","last_became_active":"2022-05-13T23:14:16.253685+0000","last_became_peered":"2022-05-13T23:14:16.253685+0000","last_unstale":"2022-05-13T23:15:05.417082+0000","last_undegraded":"2022-05-13T23:15:05.417082+0000","last_fullsized":"2022-05-13T23:15:05.417082+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_clean_scrub_stamp":"2022-05-13T23:14:15.659362+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T07:42:26.765244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2022-05-13T23:15:05.417031+0000","last_change":"2022-05-13T23:14:16.265207+0000","last_active":"2022-05-13T23:15:05.417031+0000","last_peered":"2022-05-13T23:15:05.417031+0000","last_clean":"2022-05-13T23:15:05.417031+0000","last_became_active":"2022-05-13T23:14:16.264881+0000","last_became_peered":"2022-05-13T23:14:16.264881+0000","last_unstale":"2022-05-13T23:15:05.417031+0000","last_undegraded":"2022-05-13T23:15:05.417031+0000","last_fullsized":"2022-05-13T23:15:05.417031+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_clean_scrub_stamp":"2022-05-13T23:14:15.659362+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T04:01:48.894233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2022-05-13T23:15:05.514329+0000","last_change":"2022-05-13T23:14:16.264286+0000","last_active":"2022-05-13T23:15:05.514329+0000","last_peered":"2022-05-13T23:15:05.514329+0000","last_clean":"2022-05-13T23:15:05.514329+0000","last_became_active":"2022-05-13T23:14:16.263978+0000","last_became_peered":"2022-05-13T23:14:16.263978+0000","last_unstale":"2022-05-13T23:15:05.514329+0000","last_undegraded":"2022-05-13T23:15:05.514329+0000","last_fullsized":"2022-05-13T23:15:05.514329+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_clean_scrub_stamp":"2022-05-13T23:14:15.659362+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T03:11:10.799827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2022-05-13T23:15:05.406569+0000","last_change":"2022-05-13T23:14:16.265421+0000","last_active":"2022-05-13T23:15:05.406569+0000","last_peered":"2022-05-13T23:15:05.406569+0000","last_clean":"2022-05-13T23:15:05.406569+0000","last_became_active":"2022-05-13T23:14:16.265057+0000","last_became_peered":"2022-05-13T23:14:16.265057+0000","last_unstale":"2022-05-13T23:15:05.406569+0000","last_undegraded":"2022-05-13T23:15:05.406569+0000","last_fullsized":"2022-05-13T23:15:05.406569+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_clean_scrub_stamp":"2022-05-13T23:14:15.659362+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T07:45:53.177341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2022-05-13T23:15:05.410182+0000","last_change":"2022-05-13T23:14:16.266317+0000","last_active":"2022-05-13T23:15:05.410182+0000","last_peered":"2022-05-13T23:15:05.410182+0000","last_clean":"2022-05-13T23:15:05.410182+0000","last_became_active":"2022-05-13T23:14:16.265959+0000","last_became_peered":"2022-05-13T23:14:16.265959+0000","last_unstale":"2022-05-13T23:15:05.410182+0000","last_undegraded":"2022-05-13T23:15:05.410182+0000","last_fullsized":"2022-05-13T23:15:05.410182+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_clean_scrub_stamp":"2022-05-13T23:14:15.659362+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T03:37:42.277033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2022-05-13T23:15:05.417785+0000","last_change":"2022-05-13T23:14:16.254350+0000","last_active":"2022-05-13T23:15:05.417785+0000","last_peered":"2022-05-13T23:15:05.417785+0000","last_clean":"2022-05-13T23:15:05.417785+0000","last_became_active":"2022-05-13T23:14:16.253940+0000","last_became_peered":"2022-05-13T23:14:16.253940+0000","last_unstale":"2022-05-13T23:15:05.417785+0000","last_undegraded":"2022-05-13T23:15:05.417785+0000","last_fullsized":"2022-05-13T23:15:05.417785+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_clean_scrub_stamp":"2022-05-13T23:14:15.659362+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T10:08:56.052316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2022-05-13T23:15:05.417437+0000","last_change":"2022-05-13T23:14:16.263153+0000","last_active":"2022-05-13T23:15:05.417437+0000","last_peered":"2022-05-13T23:15:05.417437+0000","last_clean":"2022-05-13T23:15:05.417437+0000","last_became_active":"2022-05-13T23:14:16.262652+0000","last_became_peered":"2022-05-13T23:14:16.262652+0000","last_unstale":"2022-05-13T23:15:05.417437+0000","last_undegraded":"2022-05-13T23:15:05.417437+0000","last_fullsized":"2022-05-13T23:15:05.417437+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_clean_scrub_stamp":"2022-05-13T23:14:15.659362+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T07:53:46.499880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2022-05-13T23:15:05.405856+0000","last_change":"2022-05-13T23:14:16.270358+0000","last_active":"2022-05-13T23:15:05.405856+0000","last_peered":"2022-05-13T23:15:05.405856+0000","last_clean":"2022-05-13T23:15:05.405856+0000","last_became_active":"2022-05-13T23:14:16.270099+0000","last_became_peered":"2022-05-13T23:14:16.270099+0000","last_unstale":"2022-05-13T23:15:05.405856+0000","last_undegraded":"2022-05-13T23:15:05.405856+0000","last_fullsized":"2022-05-13T23:15:05.405856+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_clean_scrub_stamp":"2022-05-13T23:14:15.659362+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T00:42:07.782569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2022-05-13T23:15:05.416427+0000","last_change":"2022-05-13T23:14:16.251188+0000","last_active":"2022-05-13T23:15:05.416427+0000","last_peered":"2022-05-13T23:15:05.416427+0000","last_clean":"2022-05-13T23:15:05.416427+0000","last_became_active":"2022-05-13T23:14:16.250816+0000","last_became_peered":"2022-05-13T23:14:16.250816+0000","last_unstale":"2022-05-13T23:15:05.416427+0000","last_undegraded":"2022-05-13T23:15:05.416427+0000","last_fullsized":"2022-05-13T23:15:05.416427+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_clean_scrub_stamp":"2022-05-13T23:14:15.659362+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T00:58:14.898104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2022-05-13T23:15:05.405457+0000","last_change":"2022-05-13T23:14:16.264125+0000","last_active":"2022-05-13T23:15:05.405457+0000","last_peered":"2022-05-13T23:15:05.405457+0000","last_clean":"2022-05-13T23:15:05.405457+0000","last_became_active":"2022-05-13T23:14:16.263806+0000","last_became_peered":"2022-05-13T23:14:16.263806+0000","last_unstale":"2022-05-13T23:15:05.405457+0000","last_undegraded":"2022-05-13T23:15:05.405457+0000","last_fullsized":"2022-05-13T23:15:05.405457+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_clean_scrub_stamp":"2022-05-13T23:14:15.659362+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T09:26:04.559765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2022-05-13T23:15:05.406402+0000","last_change":"2022-05-13T23:14:16.266991+0000","last_active":"2022-05-13T23:15:05.406402+0000","last_peered":"2022-05-13T23:15:05.406402+0000","last_clean":"2022-05-13T23:15:05.406402+0000","last_became_active":"2022-05-13T23:14:16.265551+0000","last_became_peered":"2022-05-13T23:14:16.265551+0000","last_unstale":"2022-05-13T23:15:05.406402+0000","last_undegraded":"2022-05-13T23:15:05.406402+0000","last_fullsized":"2022-05-13T23:15:05.406402+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_clean_scrub_stamp":"2022-05-13T23:14:15.659362+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T00:40:56.664387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":22,"reported_epoch":85,"state":"active+clean","last_fresh":"2022-05-13T23:15:06.541292+0000","last_change":"2022-05-13T23:15:06.541292+0000","last_active":"2022-05-13T23:15:06.541292+0000","last_peered":"2022-05-13T23:15:06.541292+0000","last_clean":"2022-05-13T23:15:06.541292+0000","last_became_active":"2022-05-13T23:15:06.540909+0000","last_became_peered":"2022-05-13T23:15:06.540909+0000","last_unstale":"2022-05-13T23:15:06.541292+0000","last_undegraded":"2022-05-13T23:15:06.541292+0000","last_fullsized":"2022-05-13T23:15:06.541292+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_clean_scrub_stamp":"2022-05-13T23:14:15.659362+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T09:55:15.142184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,8],"acting":[11,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1057856,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":3194880,"data_stored":3173568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":7626,"num_objects":98,"num_object_clones":0,"num_object_copies":294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98,"num_whiteouts":0,"num_read":550,"num_read_kb":292,"num_write":205,"num_write_kb":57,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":90,"num_keys_recovered":10,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":393216,"data_stored":22878,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":179,"ondisk_log_size":179,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":78,"seq":335007449114,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":5980,"kb_used_data":916,"kb_used_omap":0,"kb_used_meta":5056,"kb_avail":93739172,"statfs":{"total":95995035648,"available":95988912128,"internally_reserved":0,"allocated":937984,"data_stored":442813,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5177344},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri May 13 23:14:21 2022","interfaces":[{"interface":"back","average":{"1min":0.60299999999999998,"5min":0.60299999999999998,"15min":0.60299999999999998},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.84699999999999998,"5min":0.84699999999999998,"15min":0.84699999999999998},"last":0.41599999999999998},{"interface":"front","average":{"1min":0.64200000000000002,"5min":0.64200000000000002,"15min":0.64200000000000002},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":0.81599999999999995,"5min":0.81599999999999995,"15min":0.81599999999999995},"last":0.47999999999999998}]},{"osd":1,"last update":"Fri May 13 23:14:21 2022","interfaces":[{"interface":"back","average":{"1min":0.59699999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":0.81499999999999995,"5min":0.81499999999999995,"15min":0.81499999999999995},"last":0.61399999999999999},{"interface":"front","average":{"1min":0.58899999999999997,"5min":0.58899999999999997,"15min":0.58899999999999997},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.89500000000000002,"5min":0.89500000000000002,"15min":0.89500000000000002},"last":0.59199999999999997}]},{"osd":2,"last update":"Fri May 13 23:14:21 2022","interfaces":[{"interface":"back","average":{"1min":0.46500000000000002,"5min":0.46500000000000002,"15min":0.46500000000000002},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.76900000000000002,"5min":0.76900000000000002,"15min":0.76900000000000002},"last":0.36199999999999999},{"interface":"front","average":{"1min":0.46800000000000003,"5min":0.46800000000000003,"15min":0.46800000000000003},"min":{"1min":0.317,"5min":0.317,"15min":0.317},"max":{"1min":0.71299999999999997,"5min":0.71299999999999997,"15min":0.71299999999999997},"last":0.38800000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53000000000000003}]},{"osd":4,"last update":"Fri May 13 23:14:21 2022","interfaces":[{"interface":"back","average":{"1min":0.622,"5min":0.622,"15min":0.622},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":1.0289999999999999,"5min":1.0289999999999999,"15min":1.0289999999999999},"last":0.499},{"interface":"front","average":{"1min":0.63400000000000001,"5min":0.63400000000000001,"15min":0.63400000000000001},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":0.93700000000000006,"5min":0.93700000000000006,"15min":0.93700000000000006},"last":0.629}]},{"osd":5,"last update":"Fri May 13 23:14:21 2022","interfaces":[{"interface":"back","average":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.59099999999999997},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":1.3799999999999999,"5min":1.3799999999999999,"15min":1.3799999999999999},"last":0.52000000000000002},{"interface":"front","average":{"1min":0.60299999999999998,"5min":0.60299999999999998,"15min":0.60299999999999998},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.88,"5min":0.88,"15min":0.88},"last":0.53900000000000003}]},{"osd":6,"last update":"Fri May 13 23:14:21 2022","interfaces":[{"interface":"back","average":{"1min":0.73999999999999999,"5min":0.73999999999999999,"15min":0.73999999999999999},"min":{"1min":0.48499999999999999,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":1.0669999999999999,"5min":1.0669999999999999,"15min":1.0669999999999999},"last":0.72299999999999998},{"interface":"front","average":{"1min":0.79200000000000004,"5min":0.79200000000000004,"15min":0.79200000000000004},"min":{"1min":0.49099999999999999,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":1.655,"5min":1.655,"15min":1.655},"last":0.70499999999999996}]},{"osd":7,"last update":"Fri May 13 23:14:21 2022","interfaces":[{"interface":"back","average":{"1min":0.72299999999999998,"5min":0.72299999999999998,"15min":0.72299999999999998},"min":{"1min":0.502,"5min":0.502,"15min":0.502},"max":{"1min":1.107,"5min":1.107,"15min":1.107},"last":0.57899999999999996},{"interface":"front","average":{"1min":0.745,"5min":0.745,"15min":0.745},"min":{"1min":0.46999999999999997,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":1.3620000000000001,"5min":1.3620000000000001,"15min":1.3620000000000001},"last":0.66100000000000003}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68200000000000005}]},{"osd":9,"last update":"Fri May 13 23:14:21 2022","interfaces":[{"interface":"back","average":{"1min":0.83699999999999997,"5min":0.83699999999999997,"15min":0.83699999999999997},"min":{"1min":0.57099999999999995,"5min":0.57099999999999995,"15min":0.57099999999999995},"max":{"1min":1.7729999999999999,"5min":1.7729999999999999,"15min":1.7729999999999999},"last":0.67700000000000005},{"interface":"front","average":{"1min":0.79500000000000004,"5min":0.79500000000000004,"15min":0.79500000000000004},"min":{"1min":0.58099999999999996,"5min":0.58099999999999996,"15min":0.58099999999999996},"max":{"1min":1.3220000000000001,"5min":1.3220000000000001,"15min":1.3220000000000001},"last":0.68700000000000006}]},{"osd":10,"last update":"Fri May 13 23:14:21 2022","interfaces":[{"interface":"back","average":{"1min":0.75700000000000001,"5min":0.75700000000000001,"15min":0.75700000000000001},"min":{"1min":0.54100000000000004,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":1.26,"5min":1.26,"15min":1.26},"last":0.69999999999999996},{"interface":"front","average":{"1min":0.77600000000000002,"5min":0.77600000000000002,"15min":0.77600000000000002},"min":{"1min":0.51900000000000002,"5min":0.51900000000000002,"15min":0.51900000000000002},"max":{"1min":1.7350000000000001,"5min":1.7350000000000001,"15min":1.7350000000000001},"last":0.64400000000000002}]}]},{"osd":10,"up_from":71,"seq":304942678046,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6720,"kb_used_data":1464,"kb_used_omap":0,"kb_used_meta":5248,"kb_avail":93738432,"statfs":{"total":95995035648,"available":95988154368,"internally_reserved":0,"allocated":1499136,"data_stored":973408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5373952},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35499999999999998}]},{"osd":1,"last update":"Fri May 13 23:15:07 2022","interfaces":[{"interface":"back","average":{"1min":0.45300000000000001,"5min":0.51700000000000002,"15min":0.52800000000000002},"min":{"1min":0.33100000000000002,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.75900000000000001,"5min":0.75900000000000001,"15min":0.75900000000000001},"last":0.33700000000000002},{"interface":"front","average":{"1min":0.45800000000000002,"5min":0.55500000000000005,"15min":0.57099999999999995},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":0.59399999999999997,"5min":0.83099999999999996,"15min":0.83099999999999996},"last":0.53000000000000003}]},{"osd":2,"last update":"Fri May 13 23:15:07 2022","interfaces":[{"interface":"back","average":{"1min":0.61399999999999999,"5min":0.67800000000000005,"15min":0.68899999999999995},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.85399999999999998,"5min":1.3280000000000001,"15min":1.3280000000000001},"last":0.51700000000000002},{"interface":"front","average":{"1min":0.629,"5min":0.69199999999999995,"15min":0.70199999999999996},"min":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":0.80700000000000005,"5min":0.88800000000000001,"15min":0.88800000000000001},"last":0.498}]},{"osd":3,"last update":"Fri May 13 23:15:07 2022","interfaces":[{"interface":"back","average":{"1min":0.61399999999999999,"5min":0.71399999999999997,"15min":0.73099999999999998},"min":{"1min":0.443,"5min":0.443,"15min":0.443},"max":{"1min":0.89200000000000002,"5min":0.90100000000000002,"15min":0.90100000000000002},"last":0.436},{"interface":"front","average":{"1min":0.624,"5min":0.73999999999999999,"15min":0.76000000000000001},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.76100000000000001,"5min":0.93500000000000005,"15min":0.93500000000000005},"last":0.58499999999999996}]},{"osd":4,"last update":"Fri May 13 23:15:07 2022","interfaces":[{"interface":"back","average":{"1min":0.54500000000000004,"5min":0.65300000000000002,"15min":0.67100000000000004},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":0.71899999999999997,"5min":0.97699999999999998,"15min":0.97699999999999998},"last":0.38600000000000001},{"interface":"front","average":{"1min":0.59899999999999998,"5min":0.64100000000000001,"15min":0.64800000000000002},"min":{"1min":0.33900000000000002,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.82099999999999995,"5min":1.0720000000000001,"15min":1.0720000000000001},"last":0.40699999999999997}]},{"osd":5,"last update":"Fri May 13 23:15:07 2022","interfaces":[{"interface":"back","average":{"1min":0.69199999999999995,"5min":0.76500000000000001,"15min":0.77700000000000002},"min":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":0.90500000000000003,"5min":1.4299999999999999,"15min":1.4299999999999999},"last":0.60599999999999998},{"interface":"front","average":{"1min":0.67500000000000004,"5min":0.77300000000000002,"15min":0.78900000000000003},"min":{"1min":0.497,"5min":0.497,"15min":0.497},"max":{"1min":0.877,"5min":1.0609999999999999,"15min":1.0609999999999999},"last":0.55100000000000005}]},{"osd":6,"last update":"Fri May 13 23:15:07 2022","interfaces":[{"interface":"back","average":{"1min":0.69999999999999996,"5min":0.84899999999999998,"15min":0.874},"min":{"1min":0.44700000000000001,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":0.86399999999999999,"5min":1.157,"15min":1.157},"last":0.46899999999999997},{"interface":"front","average":{"1min":0.71299999999999997,"5min":0.84999999999999998,"15min":0.873},"min":{"1min":0.55600000000000005,"5min":0.55600000000000005,"15min":0.55600000000000005},"max":{"1min":0.879,"5min":1.216,"15min":1.216},"last":0.70699999999999996}]},{"osd":7,"last update":"Fri May 13 23:14:05 2022","interfaces":[{"interface":"back","average":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":1.585,"5min":1.585,"15min":1.585},"last":0.496},{"interface":"front","average":{"1min":0.80300000000000005,"5min":0.80300000000000005,"15min":0.80300000000000005},"min":{"1min":0.47999999999999998,"5min":0.47999999999999998,"15min":0.47999999999999998},"max":{"1min":1.0469999999999999,"5min":1.0469999999999999,"15min":1.0469999999999999},"last":0.502}]},{"osd":8,"last update":"Fri May 13 23:15:07 2022","interfaces":[{"interface":"back","average":{"1min":0.72399999999999998,"5min":0.83599999999999997,"15min":0.85399999999999998},"min":{"1min":0.47699999999999998,"5min":0.47699999999999998,"15min":0.47699999999999998},"max":{"1min":0.92300000000000004,"5min":1.163,"15min":1.163},"last":0.64000000000000001},{"interface":"front","average":{"1min":0.75900000000000001,"5min":0.85699999999999998,"15min":0.873},"min":{"1min":0.52500000000000002,"5min":0.51400000000000001,"15min":0.51400000000000001},"max":{"1min":0.89700000000000002,"5min":1.476,"15min":1.476},"last":0.56399999999999995}]},{"osd":9,"last update":"Fri May 13 23:15:07 2022","interfaces":[{"interface":"back","average":{"1min":0.81699999999999995,"5min":0.92300000000000004,"15min":0.94099999999999995},"min":{"1min":0.58099999999999996,"5min":0.58099999999999996,"15min":0.58099999999999996},"max":{"1min":1.0029999999999999,"5min":1.369,"15min":1.369},"last":0.80200000000000005},{"interface":"front","average":{"1min":0.82099999999999995,"5min":0.93799999999999994,"15min":0.95799999999999996},"min":{"1min":0.624,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":0.95799999999999996,"5min":1.819,"15min":1.819},"last":0.68600000000000005}]},{"osd":11,"last update":"Fri May 13 23:14:23 2022","interfaces":[{"interface":"back","average":{"1min":0.84299999999999997,"5min":0.84299999999999997,"15min":0.84299999999999997},"min":{"1min":0.63300000000000001,"5min":0.63300000000000001,"15min":0.63300000000000001},"max":{"1min":1.1060000000000001,"5min":1.1060000000000001,"15min":1.1060000000000001},"last":0.621},{"interface":"front","average":{"1min":0.84999999999999998,"5min":0.84999999999999998,"15min":0.84999999999999998},"min":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"max":{"1min":1.139,"5min":1.139,"15min":1.139},"last":0.76000000000000001}]}]},{"osd":9,"up_from":66,"seq":283467841570,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6120,"kb_used_data":928,"kb_used_omap":0,"kb_used_meta":5184,"kb_avail":93739032,"statfs":{"total":95995035648,"available":95988768768,"internally_reserved":0,"allocated":950272,"data_stored":442701,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5308416},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri May 13 23:14:37 2022","interfaces":[{"interface":"back","average":{"1min":0.58899999999999997,"5min":0.63900000000000001,"15min":0.64700000000000002},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":1.133,"5min":1.133,"15min":1.133},"last":0.56999999999999995},{"interface":"front","average":{"1min":0.59699999999999998,"5min":0.69699999999999995,"15min":0.71399999999999997},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.86399999999999999,"5min":1.044,"15min":1.044},"last":0.40300000000000002}]},{"osd":1,"last update":"Fri May 13 23:14:37 2022","interfaces":[{"interface":"back","average":{"1min":0.73999999999999999,"5min":0.752,"15min":0.754},"min":{"1min":0.432,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":1.6220000000000001,"5min":1.6220000000000001,"15min":1.6220000000000001},"last":0.72899999999999998},{"interface":"front","average":{"1min":0.82699999999999996,"5min":0.90100000000000002,"15min":0.91300000000000003},"min":{"1min":0.50900000000000001,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":1.3480000000000001,"5min":1.498,"15min":1.498},"last":0.65900000000000003}]},{"osd":2,"last update":"Fri May 13 23:14:37 2022","interfaces":[{"interface":"back","average":{"1min":0.83799999999999997,"5min":0.88600000000000001,"15min":0.89400000000000002},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":1.282,"5min":1.4419999999999999,"15min":1.4419999999999999},"last":0.54800000000000004},{"interface":"front","average":{"1min":0.76100000000000001,"5min":0.81299999999999994,"15min":0.82199999999999995},"min":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":1.573,"5min":1.573,"15min":1.573},"last":0.50900000000000001}]},{"osd":3,"last update":"Fri May 13 23:14:37 2022","interfaces":[{"interface":"back","average":{"1min":0.77400000000000002,"5min":0.85699999999999998,"15min":0.871},"min":{"1min":0.45200000000000001,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":1.552,"5min":1.552,"15min":1.552},"last":0.53000000000000003},{"interface":"front","average":{"1min":0.81799999999999995,"5min":0.96499999999999997,"15min":0.98899999999999999},"min":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":1.3169999999999999,"5min":2.2069999999999999,"15min":2.2069999999999999},"last":0.621}]},{"osd":4,"last update":"Fri May 13 23:14:37 2022","interfaces":[{"interface":"back","average":{"1min":0.82799999999999996,"5min":0.90900000000000003,"15min":0.92300000000000004},"min":{"1min":0.54400000000000004,"5min":0.47699999999999998,"15min":0.47699999999999998},"max":{"1min":1.7430000000000001,"5min":1.7430000000000001,"15min":1.7430000000000001},"last":0.78700000000000003},{"interface":"front","average":{"1min":0.86299999999999999,"5min":0.93899999999999995,"15min":0.95099999999999996},"min":{"1min":0.55200000000000005,"5min":0.55200000000000005,"15min":0.55200000000000005},"max":{"1min":1.2130000000000001,"5min":1.3320000000000001,"15min":1.3320000000000001},"last":0.71499999999999997}]},{"osd":5,"last update":"Fri May 13 23:14:37 2022","interfaces":[{"interface":"back","average":{"1min":0.84999999999999998,"5min":0.94599999999999995,"15min":0.96199999999999997},"min":{"1min":0.61399999999999999,"5min":0.56299999999999994,"15min":0.56299999999999994},"max":{"1min":1.661,"5min":1.661,"15min":1.661},"last":0.56000000000000005},{"interface":"front","average":{"1min":0.88500000000000001,"5min":0.99299999999999999,"15min":1.0109999999999999},"min":{"1min":0.52600000000000002,"5min":0.52600000000000002,"15min":0.52600000000000002},"max":{"1min":1.2,"5min":1.3700000000000001,"15min":1.3700000000000001},"last":0.64500000000000002}]},{"osd":6,"last update":"Fri May 13 23:14:37 2022","interfaces":[{"interface":"back","average":{"1min":0.94999999999999996,"5min":1.1020000000000001,"15min":1.127},"min":{"1min":0.54700000000000004,"5min":0.54700000000000004,"15min":0.54700000000000004},"max":{"1min":1.377,"5min":2.254,"15min":2.254},"last":0.501},{"interface":"front","average":{"1min":0.93799999999999994,"5min":1.036,"15min":1.052},"min":{"1min":0.68300000000000005,"5min":0.68300000000000005,"15min":0.68300000000000005},"max":{"1min":1.8200000000000001,"5min":1.8200000000000001,"15min":1.8200000000000001},"last":0.59399999999999997}]},{"osd":7,"last update":"Fri May 13 23:14:37 2022","interfaces":[{"interface":"back","average":{"1min":0.95699999999999996,"5min":1.0860000000000001,"15min":1.1080000000000001},"min":{"1min":0.68500000000000005,"5min":0.63,"15min":0.63},"max":{"1min":1.3080000000000001,"5min":2.101,"15min":2.101},"last":0.752},{"interface":"front","average":{"1min":1.0109999999999999,"5min":1.101,"15min":1.1160000000000001},"min":{"1min":0.73399999999999999,"5min":0.69599999999999995,"15min":0.69599999999999995},"max":{"1min":1.407,"5min":1.895,"15min":1.895},"last":0.68200000000000005}]},{"osd":8,"last update":"Fri May 13 23:14:37 2022","interfaces":[{"interface":"back","average":{"1min":0.996,"5min":1.054,"15min":1.0640000000000001},"min":{"1min":0.63500000000000001,"5min":0.59899999999999998,"15min":0.59899999999999998},"max":{"1min":1.7,"5min":1.7,"15min":1.7},"last":0.67500000000000004},{"interface":"front","average":{"1min":0.97699999999999998,"5min":1.109,"15min":1.131},"min":{"1min":0.59299999999999997,"5min":0.59299999999999997,"15min":0.59299999999999997},"max":{"1min":1.2949999999999999,"5min":2.1789999999999998,"15min":2.1789999999999998},"last":0.84399999999999997}]},{"osd":10,"last update":"Fri May 13 23:15:03 2022","interfaces":[{"interface":"back","average":{"1min":0.93300000000000005,"5min":1.069,"15min":1.0920000000000001},"min":{"1min":0.73199999999999998,"5min":0.66800000000000004,"15min":0.66800000000000004},"max":{"1min":1.252,"5min":1.7829999999999999,"15min":1.7829999999999999},"last":0.82699999999999996},{"interface":"front","average":{"1min":0.96799999999999997,"5min":1.099,"15min":1.121},"min":{"1min":0.75600000000000001,"5min":0.73099999999999998,"15min":0.73099999999999998},"max":{"1min":1.3759999999999999,"5min":1.748,"15min":1.748},"last":0.70599999999999996}]}]},{"osd":8,"up_from":58,"seq":249108103210,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6304,"kb_used_data":920,"kb_used_omap":0,"kb_used_meta":5376,"kb_avail":93738848,"statfs":{"total":95995035648,"available":95988580352,"internally_reserved":0,"allocated":942080,"data_stored":443029,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5505024},"hb_peers":[0,1,2,3,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri May 13 23:15:13 2022","interfaces":[{"interface":"back","average":{"1min":0.59099999999999997,"5min":0.59999999999999998,"15min":0.60699999999999998},"min":{"1min":0.44500000000000001,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":1.026,"5min":1.026,"15min":1.026},"last":0.55000000000000004},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.58799999999999997,"15min":0.58599999999999997},"min":{"1min":0.41499999999999998,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.91200000000000003,"5min":1.137,"15min":1.137},"last":0.55900000000000005}]},{"osd":1,"last update":"Fri May 13 23:15:13 2022","interfaces":[{"interface":"back","average":{"1min":0.627,"5min":0.57699999999999996,"15min":0.57499999999999996},"min":{"1min":0.40300000000000002,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.86699999999999999,"5min":0.86699999999999999,"15min":0.86699999999999999},"last":0.41299999999999998},{"interface":"front","average":{"1min":0.68899999999999995,"5min":0.61799999999999999,"15min":0.61199999999999999},"min":{"1min":0.42199999999999999,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.82199999999999995,"5min":0.82199999999999995,"15min":0.82199999999999995},"last":0.60799999999999998}]},{"osd":2,"last update":"Fri May 13 23:15:13 2022","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.46400000000000002,"15min":0.46200000000000002},"min":{"1min":0.32300000000000001,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.63300000000000001,"5min":0.77400000000000002,"15min":0.77400000000000002},"last":0.57599999999999996},{"interface":"front","average":{"1min":0.54200000000000004,"5min":0.47299999999999998,"15min":0.46400000000000002},"min":{"1min":0.34000000000000002,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.73199999999999998,"5min":0.79100000000000004,"15min":0.79100000000000004},"last":0.39200000000000002}]},{"osd":3,"last update":"Fri May 13 23:15:13 2022","interfaces":[{"interface":"back","average":{"1min":0.60899999999999999,"5min":0.63900000000000001,"15min":0.64900000000000002},"min":{"1min":0.40899999999999997,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.76500000000000001,"5min":0.88400000000000001,"15min":0.88400000000000001},"last":0.61899999999999999},{"interface":"front","average":{"1min":0.58599999999999997,"5min":0.63100000000000001,"15min":0.64100000000000001},"min":{"1min":0.47899999999999998,"5min":0.439,"15min":0.439},"max":{"1min":0.67900000000000005,"5min":0.83299999999999996,"15min":0.83299999999999996},"last":0.628}]},{"osd":4,"last update":"Fri May 13 23:14:08 2022","interfaces":[{"interface":"back","average":{"1min":0.61099999999999999,"5min":0.621,"15min":0.623},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":1.573,"5min":1.573,"15min":1.573},"last":0.60299999999999998},{"interface":"front","average":{"1min":0.57299999999999995,"5min":0.61899999999999999,"15min":0.627},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":0.746,"5min":0.90500000000000003,"15min":0.90500000000000003},"last":0.71299999999999997}]},{"osd":5,"last update":"Fri May 13 23:15:13 2022","interfaces":[{"interface":"back","average":{"1min":0.63300000000000001,"5min":0.58299999999999996,"15min":0.58599999999999997},"min":{"1min":0.40000000000000002,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":0.94199999999999995,"5min":0.94199999999999995,"15min":0.94199999999999995},"last":0.5},{"interface":"front","average":{"1min":0.60499999999999998,"5min":0.56799999999999995,"15min":0.57599999999999996},"min":{"1min":0.371,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.83099999999999996,"5min":0.88200000000000001,"15min":0.88200000000000001},"last":0.43099999999999999}]},{"osd":6,"last update":"Fri May 13 23:15:13 2022","interfaces":[{"interface":"back","average":{"1min":0.68200000000000005,"5min":0.72799999999999998,"15min":0.74399999999999999},"min":{"1min":0.50700000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":0.89600000000000002,"5min":0.94099999999999995,"15min":0.94099999999999995},"last":0.46800000000000003},{"interface":"front","average":{"1min":0.77900000000000003,"5min":0.75700000000000001,"15min":0.76100000000000001},"min":{"1min":0.57199999999999995,"5min":0.504,"15min":0.504},"max":{"1min":0.95399999999999996,"5min":0.95699999999999996,"15min":0.95699999999999996},"last":0.68899999999999995}]},{"osd":7,"last update":"Fri May 13 23:15:13 2022","interfaces":[{"interface":"back","average":{"1min":0.78600000000000003,"5min":0.71599999999999997,"15min":0.70999999999999996},"min":{"1min":0.48299999999999998,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":1.0069999999999999,"5min":1.0069999999999999,"15min":1.0069999999999999},"last":0.64700000000000002},{"interface":"front","average":{"1min":0.75600000000000001,"5min":0.71199999999999997,"15min":0.70399999999999996},"min":{"1min":0.47499999999999998,"5min":0.437,"15min":0.437},"max":{"1min":1.0189999999999999,"5min":1.609,"15min":1.609},"last":0.59599999999999997}]},{"osd":9,"last update":"Fri May 13 23:14:41 2022","interfaces":[{"interface":"back","average":{"1min":0.77900000000000003,"5min":0.76700000000000002,"15min":0.76500000000000001},"min":{"1min":0.59099999999999997,"5min":0.51500000000000001,"15min":0.51500000000000001},"max":{"1min":1,"5min":1,"15min":1},"last":0.63900000000000001},{"interface":"front","average":{"1min":0.77900000000000003,"5min":0.79600000000000004,"15min":0.79900000000000004},"min":{"1min":0.60599999999999998,"5min":0.59899999999999998,"15min":0.59899999999999998},"max":{"1min":0.93799999999999994,"5min":0.996,"15min":0.996},"last":0.66300000000000003}]},{"osd":10,"last update":"Fri May 13 23:15:07 2022","interfaces":[{"interface":"back","average":{"1min":0.79600000000000004,"5min":0.73999999999999999,"15min":0.73099999999999998},"min":{"1min":0.56399999999999995,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":0.99299999999999999,"5min":0.99299999999999999,"15min":0.99299999999999999},"last":0.75},{"interface":"front","average":{"1min":0.80300000000000005,"5min":0.71799999999999997,"15min":0.70399999999999996},"min":{"1min":0.53200000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":1.042,"5min":1.042,"15min":1.042},"last":0.67600000000000005}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72099999999999997}]}]},{"osd":7,"up_from":51,"seq":219043332140,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6892,"kb_used_data":1444,"kb_used_omap":0,"kb_used_meta":5440,"kb_avail":93738260,"statfs":{"total":95995035648,"available":95987978240,"internally_reserved":0,"allocated":1478656,"data_stored":973477,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5570560},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri May 13 23:13:48 2022","interfaces":[{"interface":"back","average":{"1min":0.64400000000000002,"5min":0.56000000000000005,"15min":0.54600000000000004},"min":{"1min":0.48399999999999999,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":0.82599999999999996,"5min":0.82599999999999996,"15min":0.82599999999999996},"last":0.63600000000000001},{"interface":"front","average":{"1min":0.60899999999999999,"5min":0.58799999999999997,"15min":0.58399999999999996},"min":{"1min":0.42699999999999999,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":0.78800000000000003,"5min":0.86599999999999999,"15min":0.86599999999999999},"last":0.51900000000000002}]},{"osd":1,"last update":"Fri May 13 23:14:52 2022","interfaces":[{"interface":"back","average":{"1min":0.46600000000000003,"5min":0.45200000000000001,"15min":0.44400000000000001},"min":{"1min":0.30499999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.60199999999999998,"5min":0.79800000000000004,"15min":0.79800000000000004},"last":0.436},{"interface":"front","average":{"1min":0.5,"5min":0.47999999999999998,"15min":0.47199999999999998},"min":{"1min":0.33600000000000002,"5min":0.317,"15min":0.317},"max":{"1min":0.63900000000000001,"5min":0.88800000000000001,"15min":0.88800000000000001},"last":0.46200000000000002}]},{"osd":2,"last update":"Fri May 13 23:14:52 2022","interfaces":[{"interface":"back","average":{"1min":0.80900000000000005,"5min":0.67000000000000004,"15min":0.64800000000000002},"min":{"1min":0.46600000000000003,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":3.9769999999999999,"5min":3.9769999999999999,"15min":3.9769999999999999},"last":0.48599999999999999},{"interface":"front","average":{"1min":0.64600000000000002,"5min":0.61199999999999999,"15min":0.60599999999999998},"min":{"1min":0.48799999999999999,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":0.83399999999999996,"5min":0.83399999999999996,"15min":0.83399999999999996},"last":0.57699999999999996}]},{"osd":3,"last update":"Fri May 13 23:14:52 2022","interfaces":[{"interface":"back","average":{"1min":0.65700000000000003,"5min":0.64000000000000001,"15min":0.63200000000000001},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.81299999999999994,"5min":0.92200000000000004,"15min":0.92200000000000004},"last":0.623},{"interface":"front","average":{"1min":0.68300000000000005,"5min":0.69399999999999995,"15min":0.69599999999999995},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":0.874,"5min":0.874,"15min":0.874},"last":0.60999999999999999}]},{"osd":4,"last update":"Fri May 13 23:14:52 2022","interfaces":[{"interface":"back","average":{"1min":0.55700000000000005,"5min":0.57599999999999996,"15min":0.58899999999999997},"min":{"1min":0.29999999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.78000000000000003,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.51700000000000002},{"interface":"front","average":{"1min":0.53700000000000003,"5min":0.57099999999999995,"15min":0.58499999999999996},"min":{"1min":0.378,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.80900000000000005,"5min":0.875,"15min":0.875},"last":0.56299999999999994}]},{"osd":5,"last update":"Fri May 13 23:14:52 2022","interfaces":[{"interface":"back","average":{"1min":0.70299999999999996,"5min":0.71499999999999997,"15min":0.72499999999999998},"min":{"1min":0.57599999999999996,"5min":0.47199999999999998,"15min":0.47199999999999998},"max":{"1min":0.91800000000000004,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.59899999999999998},{"interface":"front","average":{"1min":0.70599999999999996,"5min":0.70099999999999996,"15min":0.70399999999999996},"min":{"1min":0.438,"5min":0.438,"15min":0.438},"max":{"1min":0.89700000000000002,"5min":0.93999999999999995,"15min":0.93999999999999995},"last":0.72099999999999997}]},{"osd":6,"last update":"Fri May 13 23:14:52 2022","interfaces":[{"interface":"back","average":{"1min":0.73699999999999999,"5min":0.75700000000000001,"15min":0.76000000000000001},"min":{"1min":0.501,"5min":0.501,"15min":0.501},"max":{"1min":0.99199999999999999,"5min":1.0700000000000001,"15min":1.0700000000000001},"last":0.65400000000000003},{"interface":"front","average":{"1min":0.73699999999999999,"5min":0.78600000000000003,"15min":0.79000000000000004},"min":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"max":{"1min":1.054,"5min":1.1319999999999999,"15min":1.1319999999999999},"last":0.68799999999999994}]},{"osd":8,"last update":"Fri May 13 23:15:07 2022","interfaces":[{"interface":"back","average":{"1min":0.80000000000000004,"5min":0.78400000000000003,"15min":0.754},"min":{"1min":0.627,"5min":0.51300000000000001,"15min":0.51300000000000001},"max":{"1min":0.95099999999999996,"5min":4.0739999999999998,"15min":4.0739999999999998},"last":0.63700000000000001},{"interface":"front","average":{"1min":0.84299999999999997,"5min":0.78100000000000003,"15min":0.76700000000000002},"min":{"1min":0.64400000000000002,"5min":0.497,"15min":0.497},"max":{"1min":1.006,"5min":1.046,"15min":1.046},"last":0.73899999999999999}]},{"osd":9,"last update":"Fri May 13 23:14:38 2022","interfaces":[{"interface":"back","average":{"1min":0.84099999999999997,"5min":0.82899999999999996,"15min":0.82699999999999996},"min":{"1min":0.55200000000000005,"5min":0.55200000000000005,"15min":0.55200000000000005},"max":{"1min":0.98299999999999998,"5min":1.1339999999999999,"15min":1.1339999999999999},"last":0.78700000000000003},{"interface":"front","average":{"1min":0.85199999999999998,"5min":0.83299999999999996,"15min":0.82999999999999996},"min":{"1min":0.66400000000000003,"5min":0.64900000000000002,"15min":0.64900000000000002},"max":{"1min":1.0389999999999999,"5min":1.0409999999999999,"15min":1.0409999999999999},"last":0.81999999999999995}]},{"osd":10,"last update":"Fri May 13 23:15:00 2022","interfaces":[{"interface":"back","average":{"1min":0.80700000000000005,"5min":0.78000000000000003,"15min":0.77600000000000002},"min":{"1min":0.54000000000000004,"5min":0.47599999999999998,"15min":0.47599999999999998},"max":{"1min":1.0069999999999999,"5min":1.093,"15min":1.093},"last":0.80300000000000005},{"interface":"front","average":{"1min":0.81699999999999995,"5min":0.746,"15min":0.73399999999999999},"min":{"1min":0.627,"5min":0.51300000000000001,"15min":0.51300000000000001},"max":{"1min":1.026,"5min":1.0529999999999999,"15min":1.0529999999999999},"last":0.67400000000000004}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.83299999999999996}]}]},{"osd":6,"up_from":46,"seq":197568495666,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6228,"kb_used_data":908,"kb_used_omap":0,"kb_used_meta":5312,"kb_avail":93738924,"statfs":{"total":95995035648,"available":95988658176,"internally_reserved":0,"allocated":929792,"data_stored":442281,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5439488},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri May 13 23:14:30 2022","interfaces":[{"interface":"back","average":{"1min":0.59999999999999998,"5min":0.56299999999999994,"15min":0.54900000000000004},"min":{"1min":0.39200000000000002,"5min":0.378,"15min":0.378},"max":{"1min":1.071,"5min":1.071,"15min":1.071},"last":0.40699999999999997},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.60499999999999998,"15min":0.59899999999999998},"min":{"1min":0.36699999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":1.262,"5min":1.262,"15min":1.262},"last":0.53100000000000003}]},{"osd":1,"last update":"Fri May 13 23:14:30 2022","interfaces":[{"interface":"back","average":{"1min":0.76300000000000001,"5min":0.76100000000000001,"15min":0.751},"min":{"1min":0.52600000000000002,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":1.1830000000000001,"5min":1.2210000000000001,"15min":1.2210000000000001},"last":0.73799999999999999},{"interface":"front","average":{"1min":0.70799999999999996,"5min":0.68600000000000005,"15min":0.67200000000000004},"min":{"1min":0.52200000000000002,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":0.91700000000000004,"5min":1.28,"15min":1.28},"last":0.92500000000000004}]},{"osd":2,"last update":"Fri May 13 23:14:30 2022","interfaces":[{"interface":"back","average":{"1min":0.78300000000000003,"5min":0.78700000000000003,"15min":0.78500000000000003},"min":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.51600000000000001},"max":{"1min":1.3540000000000001,"5min":1.3540000000000001,"15min":1.3540000000000001},"last":0.76700000000000002},{"interface":"front","average":{"1min":0.79600000000000004,"5min":0.81299999999999994,"15min":0.81200000000000006},"min":{"1min":0.60799999999999998,"5min":0.53300000000000003,"15min":0.53300000000000003},"max":{"1min":1.0620000000000001,"5min":1.298,"15min":1.298},"last":0.64100000000000001}]},{"osd":3,"last update":"Fri May 13 23:14:30 2022","interfaces":[{"interface":"back","average":{"1min":0.77500000000000002,"5min":0.80800000000000005,"15min":0.80500000000000005},"min":{"1min":0.52500000000000002,"5min":0.47599999999999998,"15min":0.47599999999999998},"max":{"1min":1.105,"5min":1.3839999999999999,"15min":1.3839999999999999},"last":0.51600000000000001},{"interface":"front","average":{"1min":0.68600000000000005,"5min":0.75900000000000001,"15min":0.77500000000000002},"min":{"1min":0.48499999999999999,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":0.91400000000000003,"5min":1.4119999999999999,"15min":1.4119999999999999},"last":0.56399999999999995}]},{"osd":4,"last update":"Fri May 13 23:14:30 2022","interfaces":[{"interface":"back","average":{"1min":0.80800000000000005,"5min":0.83199999999999996,"15min":0.83999999999999997},"min":{"1min":0.505,"5min":0.504,"15min":0.504},"max":{"1min":1.3120000000000001,"5min":1.5640000000000001,"15min":1.5640000000000001},"last":0.70799999999999996},{"interface":"front","average":{"1min":0.88700000000000001,"5min":0.89900000000000002,"15min":0.89400000000000002},"min":{"1min":0.61199999999999999,"5min":0.60799999999999998,"15min":0.60799999999999998},"max":{"1min":1.238,"5min":1.625,"15min":1.625},"last":0.83799999999999997}]},{"osd":5,"last update":"Fri May 13 23:14:30 2022","interfaces":[{"interface":"back","average":{"1min":0.79400000000000004,"5min":0.84399999999999997,"15min":0.85399999999999998},"min":{"1min":0.57199999999999995,"5min":0.50600000000000001,"15min":0.50600000000000001},"max":{"1min":1.123,"5min":1.5029999999999999,"15min":1.5029999999999999},"last":0.81299999999999994},{"interface":"front","average":{"1min":0.88300000000000001,"5min":0.92600000000000005,"15min":0.93400000000000005},"min":{"1min":0.623,"5min":0.623,"15min":0.623},"max":{"1min":1.3999999999999999,"5min":1.3999999999999999,"15min":1.3999999999999999},"last":0.88700000000000001}]},{"osd":7,"last update":"Fri May 13 23:14:53 2022","interfaces":[{"interface":"back","average":{"1min":0.93000000000000005,"5min":0.91200000000000003,"15min":0.874},"min":{"1min":0.67700000000000005,"5min":0.66000000000000003,"15min":0.66000000000000003},"max":{"1min":1.1990000000000001,"5min":1.534,"15min":1.534},"last":0.96399999999999997},{"interface":"front","average":{"1min":0.83999999999999997,"5min":0.85199999999999998,"15min":0.82699999999999996},"min":{"1min":0.627,"5min":0.61299999999999999,"15min":0.61299999999999999},"max":{"1min":1.1279999999999999,"5min":1.6180000000000001,"15min":1.6180000000000001},"last":0.89600000000000002}]},{"osd":8,"last update":"Fri May 13 23:15:13 2022","interfaces":[{"interface":"back","average":{"1min":1.03,"5min":0.995,"15min":0.98499999999999999},"min":{"1min":0.81299999999999994,"5min":0.69999999999999996,"15min":0.69999999999999996},"max":{"1min":1.411,"5min":1.484,"15min":1.484},"last":0.90200000000000002},{"interface":"front","average":{"1min":0.95999999999999996,"5min":0.94599999999999995,"15min":0.95399999999999996},"min":{"1min":0.68300000000000005,"5min":0.67500000000000004,"15min":0.67500000000000004},"max":{"1min":1.4219999999999999,"5min":1.548,"15min":1.548},"last":0.86699999999999999}]},{"osd":9,"last update":"Fri May 13 23:14:39 2022","interfaces":[{"interface":"back","average":{"1min":0.93300000000000005,"5min":1.0429999999999999,"15min":1.0609999999999999},"min":{"1min":0.68500000000000005,"5min":0.68500000000000005,"15min":0.68500000000000005},"max":{"1min":1.2629999999999999,"5min":1.458,"15min":1.458},"last":0.68799999999999994},{"interface":"front","average":{"1min":0.94099999999999995,"5min":1.034,"15min":1.05},"min":{"1min":0.58099999999999996,"5min":0.58099999999999996,"15min":0.58099999999999996},"max":{"1min":1.4390000000000001,"5min":1.4390000000000001,"15min":1.4390000000000001},"last":0.85899999999999999}]},{"osd":10,"last update":"Fri May 13 23:15:02 2022","interfaces":[{"interface":"back","average":{"1min":1.0549999999999999,"5min":1.0529999999999999,"15min":1.0529999999999999},"min":{"1min":0.82799999999999996,"5min":0.77800000000000002,"15min":0.77800000000000002},"max":{"1min":1.4710000000000001,"5min":1.645,"15min":1.645},"last":0.94999999999999996},{"interface":"front","average":{"1min":1.0609999999999999,"5min":1.0169999999999999,"15min":1.0089999999999999},"min":{"1min":0.81399999999999995,"5min":0.73999999999999999,"15min":0.73999999999999999},"max":{"1min":1.4990000000000001,"5min":1.4990000000000001,"15min":1.4990000000000001},"last":0.91400000000000003}]}]},{"osd":1,"up_from":13,"seq":55834574925,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":7132,"kb_used_data":916,"kb_used_omap":0,"kb_used_meta":6208,"kb_avail":93738020,"statfs":{"total":95995035648,"available":95987732480,"internally_reserved":0,"allocated":937984,"data_stored":444791,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":6356992},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri May 13 23:14:14 2022","interfaces":[{"interface":"back","average":{"1min":0.60399999999999998,"5min":0.55900000000000005,"15min":0.51300000000000001},"min":{"1min":0.46700000000000003,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.755,"5min":0.91700000000000004,"15min":0.91700000000000004},"last":0.86899999999999999},{"interface":"front","average":{"1min":0.623,"5min":0.57899999999999996,"15min":0.54000000000000004},"min":{"1min":0.48399999999999999,"5min":0.309,"15min":0.309},"max":{"1min":0.82299999999999995,"5min":0.86799999999999999,"15min":0.86799999999999999},"last":0.70399999999999996}]},{"osd":2,"last update":"Fri May 13 23:14:44 2022","interfaces":[{"interface":"back","average":{"1min":0.59999999999999998,"5min":0.54100000000000004,"15min":0.51900000000000002},"min":{"1min":0.39900000000000002,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.77600000000000002,"5min":0.77600000000000002,"15min":0.77600000000000002},"last":0.82299999999999995},{"interface":"front","average":{"1min":0.67100000000000004,"5min":0.57399999999999995,"15min":0.53500000000000003},"min":{"1min":0.41599999999999998,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.96399999999999997,"5min":0.96399999999999997,"15min":0.96399999999999997},"last":0.89400000000000002}]},{"osd":3,"last update":"Fri May 13 23:15:13 2022","interfaces":[{"interface":"back","average":{"1min":0.71599999999999997,"5min":0.628,"15min":0.58199999999999996},"min":{"1min":0.503,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.96499999999999997,"5min":0.96499999999999997,"15min":0.96499999999999997},"last":0.84099999999999997},{"interface":"front","average":{"1min":0.66700000000000004,"5min":0.59399999999999997,"15min":0.56899999999999995},"min":{"1min":0.48099999999999998,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.873,"5min":0.95399999999999996,"15min":0.95399999999999996},"last":0.72499999999999998}]},{"osd":4,"last update":"Fri May 13 23:14:35 2022","interfaces":[{"interface":"back","average":{"1min":0.59099999999999997,"5min":0.47099999999999997,"15min":0.42899999999999999},"min":{"1min":0.35499999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.91400000000000003,"5min":0.91400000000000003,"15min":0.91400000000000003},"last":0.67500000000000004},{"interface":"front","average":{"1min":0.60899999999999999,"5min":0.504,"15min":0.45700000000000002},"min":{"1min":0.34100000000000003,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.90200000000000002,"5min":0.90200000000000002,"15min":0.90200000000000002},"last":0.749}]},{"osd":5,"last update":"Fri May 13 23:14:55 2022","interfaces":[{"interface":"back","average":{"1min":0.73299999999999998,"5min":0.63200000000000001,"15min":0.59599999999999997},"min":{"1min":0.42999999999999999,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.98699999999999999,"5min":0.98699999999999999,"15min":0.98699999999999999},"last":0.93799999999999994},{"interface":"front","average":{"1min":0.75700000000000001,"5min":0.66000000000000003,"15min":0.61599999999999999},"min":{"1min":0.49199999999999999,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":0.95599999999999996,"5min":0.98799999999999999,"15min":0.98799999999999999},"last":0.96699999999999997}]},{"osd":6,"last update":"Fri May 13 23:14:26 2022","interfaces":[{"interface":"back","average":{"1min":0.78100000000000003,"5min":0.70999999999999996,"15min":0.68200000000000005},"min":{"1min":0.497,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":1.0840000000000001,"5min":1.5529999999999999,"15min":1.5529999999999999},"last":1.002},{"interface":"front","average":{"1min":0.80800000000000005,"5min":0.73599999999999999,"15min":0.71699999999999997},"min":{"1min":0.48499999999999999,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":1.0209999999999999,"5min":1.1419999999999999,"15min":1.1419999999999999},"last":1.032}]},{"osd":7,"last update":"Fri May 13 23:14:49 2022","interfaces":[{"interface":"back","average":{"1min":0.628,"5min":0.59899999999999998,"15min":0.56799999999999995},"min":{"1min":0.39800000000000002,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.91400000000000003,"5min":0.95999999999999996,"15min":0.95999999999999996},"last":0.91600000000000004},{"interface":"front","average":{"1min":0.75600000000000001,"5min":0.68100000000000005,"15min":0.63900000000000001},"min":{"1min":0.45800000000000002,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":1.079,"5min":1.079,"15min":1.079},"last":1.0649999999999999}]},{"osd":8,"last update":"Fri May 13 23:15:13 2022","interfaces":[{"interface":"back","average":{"1min":0.83299999999999996,"5min":0.70499999999999996,"15min":0.66900000000000004},"min":{"1min":0.50700000000000001,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":1.0960000000000001,"5min":1.0960000000000001,"15min":1.0960000000000001},"last":1.0549999999999999},{"interface":"front","average":{"1min":0.93799999999999994,"5min":0.78000000000000003,"15min":0.72999999999999998},"min":{"1min":0.501,"5min":0.501,"15min":0.501},"max":{"1min":1.214,"5min":1.214,"15min":1.214},"last":1.1379999999999999}]},{"osd":9,"last update":"Fri May 13 23:14:44 2022","interfaces":[{"interface":"back","average":{"1min":0.89400000000000002,"5min":0.88600000000000001,"15min":0.88500000000000001},"min":{"1min":0.55000000000000004,"5min":0.55000000000000004,"15min":0.55000000000000004},"max":{"1min":1.4550000000000001,"5min":1.4550000000000001,"15min":1.4550000000000001},"last":1.1639999999999999},{"interface":"front","average":{"1min":0.86799999999999999,"5min":0.84999999999999998,"15min":0.84699999999999998},"min":{"1min":0.59499999999999997,"5min":0.59499999999999997,"15min":0.59499999999999997},"max":{"1min":1.099,"5min":1.099,"15min":1.099},"last":1.085}]},{"osd":10,"last update":"Fri May 13 23:15:04 2022","interfaces":[{"interface":"back","average":{"1min":0.79100000000000004,"5min":0.78800000000000003,"15min":0.78800000000000003},"min":{"1min":0.53000000000000003,"5min":0.51300000000000001,"15min":0.51300000000000001},"max":{"1min":0.97399999999999998,"5min":0.97399999999999998,"15min":0.97399999999999998},"last":0.98899999999999999},{"interface":"front","average":{"1min":0.91700000000000004,"5min":0.90900000000000003,"15min":0.90800000000000003},"min":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":1.123,"5min":1.123,"15min":1.123},"last":1.109}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.121}]}]},{"osd":0,"up_from":8,"seq":34359738449,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":7148,"kb_used_data":932,"kb_used_omap":0,"kb_used_meta":6208,"kb_avail":93738004,"statfs":{"total":95995035648,"available":95987716096,"internally_reserved":0,"allocated":954368,"data_stored":443681,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":6356992},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri May 13 23:15:14 2022","interfaces":[{"interface":"back","average":{"1min":0.70199999999999996,"5min":0.64700000000000002,"15min":0.61699999999999999},"min":{"1min":0.42799999999999999,"5min":0.38500000000000001,"15min":0.38300000000000001},"max":{"1min":1.131,"5min":1.2729999999999999,"15min":1.2729999999999999},"last":0.94199999999999995},{"interface":"front","average":{"1min":0.755,"5min":0.69099999999999995,"15min":0.63700000000000001},"min":{"1min":0.48899999999999999,"5min":0.376,"15min":0.376},"max":{"1min":1.0329999999999999,"5min":1.167,"15min":1.167},"last":1.0329999999999999}]},{"osd":2,"last update":"Fri May 13 23:14:36 2022","interfaces":[{"interface":"back","average":{"1min":0.748,"5min":0.67500000000000004,"15min":0.65300000000000002},"min":{"1min":0.48899999999999999,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":1.024,"5min":1.024,"15min":1.024},"last":1.0980000000000001},{"interface":"front","average":{"1min":0.69399999999999995,"5min":0.65000000000000002,"15min":0.63300000000000001},"min":{"1min":0.46500000000000002,"5min":0.45900000000000002,"15min":0.45900000000000002},"max":{"1min":1.071,"5min":1.123,"15min":1.123},"last":0.68000000000000005}]},{"osd":3,"last update":"Fri May 13 23:13:08 2022","interfaces":[{"interface":"back","average":{"1min":0.82399999999999995,"5min":0.63300000000000001,"15min":0.59999999999999998},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":1.163,"5min":1.163,"15min":1.163},"last":0.74399999999999999},{"interface":"front","average":{"1min":0.79800000000000004,"5min":0.67000000000000004,"15min":0.64600000000000002},"min":{"1min":0.46200000000000002,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":1.157,"5min":1.157,"15min":1.157},"last":1.161}]},{"osd":4,"last update":"Fri May 13 23:14:35 2022","interfaces":[{"interface":"back","average":{"1min":0.754,"5min":0.70399999999999996,"15min":0.66200000000000003},"min":{"1min":0.57399999999999995,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":1.0169999999999999,"5min":1.1990000000000001,"15min":1.1990000000000001},"last":0.996},{"interface":"front","average":{"1min":0.71599999999999997,"5min":0.68200000000000005,"15min":0.63900000000000001},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":1.1830000000000001,"5min":1.1830000000000001,"15min":1.1830000000000001},"last":0.63100000000000001}]},{"osd":5,"last update":"Fri May 13 23:15:01 2022","interfaces":[{"interface":"back","average":{"1min":0.78000000000000003,"5min":0.73999999999999999,"15min":0.71599999999999997},"min":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":1.837,"5min":1.837,"15min":1.837},"last":1.1319999999999999},{"interface":"front","average":{"1min":0.67400000000000004,"5min":0.69199999999999995,"15min":0.69399999999999995},"min":{"1min":0.46899999999999997,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":0.83599999999999997,"5min":1.056,"15min":1.056},"last":1.117}]},{"osd":6,"last update":"Fri May 13 23:14:27 2022","interfaces":[{"interface":"back","average":{"1min":0.752,"5min":0.77900000000000003,"15min":0.751},"min":{"1min":0.45100000000000001,"5min":0.44800000000000001,"15min":0.44800000000000001},"max":{"1min":0.94999999999999996,"5min":1.288,"15min":1.288},"last":0.84699999999999998},{"interface":"front","average":{"1min":0.79000000000000004,"5min":0.79000000000000004,"15min":0.77100000000000002},"min":{"1min":0.502,"5min":0.5,"15min":0.5},"max":{"1min":1.24,"5min":1.24,"15min":1.24},"last":1.179}]},{"osd":7,"last update":"Fri May 13 23:13:48 2022","interfaces":[{"interface":"back","average":{"1min":0.77300000000000002,"5min":0.75,"15min":0.746},"min":{"1min":0.497,"5min":0.48699999999999999,"15min":0.48699999999999999},"max":{"1min":1.2050000000000001,"5min":1.2050000000000001,"15min":1.2050000000000001},"last":0.94199999999999995},{"interface":"front","average":{"1min":0.94899999999999995,"5min":0.86099999999999999,"15min":0.84699999999999998},"min":{"1min":0.69599999999999995,"5min":0.60399999999999998,"15min":0.60399999999999998},"max":{"1min":1.2350000000000001,"5min":1.2350000000000001,"15min":1.2350000000000001},"last":0.877}]},{"osd":8,"last update":"Fri May 13 23:15:09 2022","interfaces":[{"interface":"back","average":{"1min":0.76900000000000002,"5min":0.79900000000000004,"15min":0.79900000000000004},"min":{"1min":0.58499999999999996,"5min":0.52100000000000002,"15min":0.52100000000000002},"max":{"1min":1.105,"5min":1.7070000000000001,"15min":1.7070000000000001},"last":1.0149999999999999},{"interface":"front","average":{"1min":0.88,"5min":0.90800000000000003,"15min":0.91400000000000003},"min":{"1min":0.622,"5min":0.60599999999999998,"15min":0.60599999999999998},"max":{"1min":1.5680000000000001,"5min":1.5680000000000001,"15min":1.5680000000000001},"last":0.91800000000000004}]},{"osd":9,"last update":"Fri May 13 23:14:39 2022","interfaces":[{"interface":"back","average":{"1min":0.92600000000000005,"5min":1.032,"15min":1.05},"min":{"1min":0.60499999999999998,"5min":0.60499999999999998,"15min":0.60499999999999998},"max":{"1min":1.1759999999999999,"5min":1.397,"15min":1.397},"last":0.80300000000000005},{"interface":"front","average":{"1min":0.88600000000000001,"5min":0.93000000000000005,"15min":0.93700000000000006},"min":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.59099999999999997},"max":{"1min":1.28,"5min":1.28,"15min":1.28},"last":0.71399999999999997}]},{"osd":10,"last update":"Fri May 13 23:15:03 2022","interfaces":[{"interface":"back","average":{"1min":0.84399999999999997,"5min":0.94599999999999995,"15min":0.96299999999999997},"min":{"1min":0.61899999999999999,"5min":0.61899999999999999,"15min":0.61899999999999999},"max":{"1min":1.26,"5min":1.7989999999999999,"15min":1.7989999999999999},"last":0.97899999999999998},{"interface":"front","average":{"1min":1.0109999999999999,"5min":1.052,"15min":1.0589999999999999},"min":{"1min":0.753,"5min":0.745,"15min":0.745},"max":{"1min":1.9279999999999999,"5min":1.9279999999999999,"15min":1.9279999999999999},"last":0.754}]},{"osd":11,"last update":"Fri May 13 23:14:22 2022","interfaces":[{"interface":"back","average":{"1min":0.97899999999999998,"5min":0.97899999999999998,"15min":0.97899999999999998},"min":{"1min":0.63400000000000001,"5min":0.63400000000000001,"15min":0.63400000000000001},"max":{"1min":1.617,"5min":1.617,"15min":1.617},"last":1.0449999999999999},{"interface":"front","average":{"1min":1.0229999999999999,"5min":1.0229999999999999,"15min":1.0229999999999999},"min":{"1min":0.68400000000000005,"5min":0.68400000000000005,"15min":0.68400000000000005},"max":{"1min":1.266,"5min":1.266,"15min":1.266},"last":0.95599999999999996}]}]},{"osd":2,"up_from":18,"seq":77309411399,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":7272,"kb_used_data":928,"kb_used_omap":0,"kb_used_meta":6336,"kb_avail":93737880,"statfs":{"total":95995035648,"available":95987589120,"internally_reserved":0,"allocated":950272,"data_stored":444923,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":6488064},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri May 13 23:14:38 2022","interfaces":[{"interface":"back","average":{"1min":0.56000000000000005,"5min":0.52300000000000002,"15min":0.52500000000000002},"min":{"1min":0.374,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.93600000000000005,"5min":0.93600000000000005,"15min":0.93600000000000005},"last":0.50600000000000001},{"interface":"front","average":{"1min":0.58599999999999997,"5min":0.53500000000000003,"15min":0.52100000000000002},"min":{"1min":0.442,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.91200000000000003,"5min":0.91200000000000003,"15min":0.91200000000000003},"last":0.53300000000000003}]},{"osd":1,"last update":"Fri May 13 23:14:38 2022","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.54200000000000004,"15min":0.52600000000000002},"min":{"1min":0.439,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.75600000000000001,"5min":0.83699999999999997,"15min":0.83699999999999997},"last":0.72299999999999998},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.53100000000000003,"15min":0.54900000000000004},"min":{"1min":0.35199999999999998,"5min":0.313,"15min":0.313},"max":{"1min":0.82699999999999996,"5min":0.82699999999999996,"15min":0.82699999999999996},"last":0.74399999999999999}]},{"osd":3,"last update":"Fri May 13 23:15:15 2022","interfaces":[{"interface":"back","average":{"1min":0.64300000000000002,"5min":0.623,"15min":0.61399999999999999},"min":{"1min":0.51300000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.80200000000000005,"5min":0.90500000000000003,"15min":0.90500000000000003},"last":0.78300000000000003},{"interface":"front","average":{"1min":0.61699999999999999,"5min":0.60999999999999999,"15min":0.59899999999999998},"min":{"1min":0.42899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.82699999999999996,"5min":0.89500000000000002,"15min":0.89500000000000002},"last":0.82699999999999996}]},{"osd":4,"last update":"Fri May 13 23:13:31 2022","interfaces":[{"interface":"back","average":{"1min":0.55100000000000005,"5min":0.53400000000000003,"15min":0.52600000000000002},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":0.81999999999999995,"5min":0.81999999999999995,"15min":0.81999999999999995},"last":0.61599999999999999},{"interface":"front","average":{"1min":0.53200000000000003,"5min":0.51500000000000001,"15min":0.505},"min":{"1min":0.36299999999999999,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.70099999999999996,"5min":0.752,"15min":0.752},"last":0.754}]},{"osd":5,"last update":"Fri May 13 23:15:01 2022","interfaces":[{"interface":"back","average":{"1min":0.48099999999999998,"5min":0.42399999999999999,"15min":0.39000000000000001},"min":{"1min":0.33700000000000002,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.77900000000000003,"5min":0.77900000000000003,"15min":0.77900000000000003},"last":0.65600000000000003},{"interface":"front","average":{"1min":0.56399999999999995,"5min":0.45300000000000001,"15min":0.40899999999999997},"min":{"1min":0.33100000000000002,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.80600000000000005,"5min":0.80600000000000005,"15min":0.80600000000000005},"last":0.63300000000000001}]},{"osd":6,"last update":"Fri May 13 23:14:32 2022","interfaces":[{"interface":"back","average":{"1min":0.73599999999999999,"5min":0.63200000000000001,"15min":0.61399999999999999},"min":{"1min":0.61599999999999999,"5min":0.44800000000000001,"15min":0.44800000000000001},"max":{"1min":0.96599999999999997,"5min":0.96599999999999997,"15min":0.96599999999999997},"last":0.92200000000000004},{"interface":"front","average":{"1min":0.66200000000000003,"5min":0.60599999999999998,"15min":0.58799999999999997},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.93100000000000005,"5min":0.93100000000000005,"15min":0.93100000000000005},"last":0.70699999999999996}]},{"osd":7,"last update":"Fri May 13 23:13:45 2022","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":0.61599999999999999,"15min":0.61599999999999999},"min":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":0.80400000000000005,"5min":0.80400000000000005,"15min":0.80400000000000005},"last":0.78400000000000003},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.63800000000000001,"15min":0.64200000000000002},"min":{"1min":0.45900000000000002,"5min":0.45900000000000002,"15min":0.45900000000000002},"max":{"1min":0.83199999999999996,"5min":0.83199999999999996,"15min":0.83199999999999996},"last":0.78800000000000003}]},{"osd":8,"last update":"Fri May 13 23:15:09 2022","interfaces":[{"interface":"back","average":{"1min":0.56899999999999995,"5min":0.54300000000000004,"15min":0.53500000000000003},"min":{"1min":0.42199999999999999,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.83399999999999996,"5min":0.876,"15min":0.876},"last":0.79700000000000004},{"interface":"front","average":{"1min":0.629,"5min":0.57899999999999996,"15min":0.57099999999999995},"min":{"1min":0.49399999999999999,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.97199999999999998,"5min":0.97199999999999998,"15min":0.97199999999999998},"last":0.76900000000000002}]},{"osd":9,"last update":"Fri May 13 23:14:38 2022","interfaces":[{"interface":"back","average":{"1min":0.77500000000000002,"5min":0.72299999999999998,"15min":0.71399999999999997},"min":{"1min":0.57099999999999995,"5min":0.55100000000000005,"15min":0.55100000000000005},"max":{"1min":0.94099999999999995,"5min":0.94099999999999995,"15min":0.94099999999999995},"last":0.81000000000000005},{"interface":"front","average":{"1min":0.73999999999999999,"5min":0.69999999999999996,"15min":0.69299999999999995},"min":{"1min":0.498,"5min":0.498,"15min":0.498},"max":{"1min":0.89000000000000001,"5min":0.89000000000000001,"15min":0.89000000000000001},"last":0.80400000000000005}]},{"osd":10,"last update":"Fri May 13 23:15:01 2022","interfaces":[{"interface":"back","average":{"1min":0.70899999999999996,"5min":0.69299999999999995,"15min":0.69099999999999995},"min":{"1min":0.58099999999999996,"5min":0.47599999999999998,"15min":0.47599999999999998},"max":{"1min":0.92700000000000005,"5min":0.97099999999999997,"15min":0.97099999999999997},"last":0.87},{"interface":"front","average":{"1min":0.76500000000000001,"5min":0.72099999999999997,"15min":0.71399999999999997},"min":{"1min":0.60299999999999998,"5min":0.52100000000000002,"15min":0.52100000000000002},"max":{"1min":1.014,"5min":1.014,"15min":1.014},"last":0.89000000000000001}]},{"osd":11,"last update":"Fri May 13 23:15:15 2022","interfaces":[{"interface":"back","average":{"1min":0.68799999999999994,"5min":0.68799999999999994,"15min":0.68799999999999994},"min":{"1min":0.47899999999999998,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":0.91100000000000003,"5min":0.91100000000000003,"15min":0.91100000000000003},"last":0.83799999999999997},{"interface":"front","average":{"1min":0.74299999999999999,"5min":0.74299999999999999,"15min":0.74299999999999999},"min":{"1min":0.57999999999999996,"5min":0.57999999999999996,"15min":0.57999999999999996},"max":{"1min":0.99399999999999999,"5min":0.99399999999999999,"15min":0.99399999999999999},"last":0.84999999999999998}]}]},{"osd":3,"up_from":25,"seq":107374182465,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6560,"kb_used_data":920,"kb_used_omap":0,"kb_used_meta":5632,"kb_avail":93738592,"statfs":{"total":95995035648,"available":95988318208,"internally_reserved":0,"allocated":942080,"data_stored":442037,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5767168},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri May 13 23:14:15 2022","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.49399999999999999,"15min":0.46400000000000002},"min":{"1min":0.40200000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.81999999999999995,"5min":0.81999999999999995,"15min":0.81999999999999995},"last":0.42699999999999999},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.52400000000000002,"15min":0.502},"min":{"1min":0.373,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.70799999999999996,"5min":0.96299999999999997,"15min":0.96299999999999997},"last":0.437}]},{"osd":1,"last update":"Fri May 13 23:14:15 2022","interfaces":[{"interface":"back","average":{"1min":0.749,"5min":0.67700000000000005,"15min":0.63500000000000001},"min":{"1min":0.495,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":1.014,"5min":1.014,"15min":1.014},"last":0.67900000000000005},{"interface":"front","average":{"1min":0.64100000000000001,"5min":0.64300000000000002,"15min":0.63100000000000001},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.93200000000000005,"5min":1.0229999999999999,"15min":1.0229999999999999},"last":0.746}]},{"osd":2,"last update":"Fri May 13 23:14:15 2022","interfaces":[{"interface":"back","average":{"1min":0.79300000000000004,"5min":0.72399999999999998,"15min":0.68799999999999994},"min":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":1.0489999999999999,"5min":1.0489999999999999,"15min":1.0489999999999999},"last":0.68500000000000005},{"interface":"front","average":{"1min":0.73899999999999999,"5min":0.69699999999999995,"15min":0.67500000000000004},"min":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":1.1299999999999999,"5min":1.1299999999999999,"15min":1.1299999999999999},"last":0.66000000000000003}]},{"osd":4,"last update":"Fri May 13 23:14:34 2022","interfaces":[{"interface":"back","average":{"1min":0.79100000000000004,"5min":0.745,"15min":0.69199999999999995},"min":{"1min":0.51300000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":1.1499999999999999,"5min":1.157,"15min":1.157},"last":0.77200000000000002},{"interface":"front","average":{"1min":0.75700000000000001,"5min":0.74199999999999999,"15min":0.69399999999999995},"min":{"1min":0.55300000000000005,"5min":0.47699999999999998,"15min":0.47699999999999998},"max":{"1min":1.075,"5min":1.075,"15min":1.075},"last":0.629}]},{"osd":5,"last update":"Fri May 13 23:14:55 2022","interfaces":[{"interface":"back","average":{"1min":0.72999999999999998,"5min":0.746,"15min":0.72999999999999998},"min":{"1min":0.51100000000000001,"5min":0.51100000000000001,"15min":0.51100000000000001},"max":{"1min":1.0409999999999999,"5min":1.056,"15min":1.056},"last":0.503},{"interface":"front","average":{"1min":0.79700000000000004,"5min":0.80900000000000005,"15min":0.76900000000000002},"min":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"max":{"1min":1.1060000000000001,"5min":1.161,"15min":1.161},"last":0.60699999999999998}]},{"osd":6,"last update":"Fri May 13 23:14:28 2022","interfaces":[{"interface":"back","average":{"1min":0.85599999999999998,"5min":0.84299999999999997,"15min":0.81899999999999995},"min":{"1min":0.497,"5min":0.497,"15min":0.497},"max":{"1min":1.1579999999999999,"5min":1.2829999999999999,"15min":1.2829999999999999},"last":0.75900000000000001},{"interface":"front","average":{"1min":0.84699999999999998,"5min":0.86399999999999999,"15min":0.85199999999999998},"min":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":1.2549999999999999,"5min":1.256,"15min":1.256},"last":0.81399999999999995}]},{"osd":7,"last update":"Fri May 13 23:14:49 2022","interfaces":[{"interface":"back","average":{"1min":0.87,"5min":0.90600000000000003,"15min":0.88600000000000001},"min":{"1min":0.58699999999999997,"5min":0.58699999999999997,"15min":0.58699999999999997},"max":{"1min":1.204,"5min":1.3280000000000001,"15min":1.3280000000000001},"last":0.72799999999999998},{"interface":"front","average":{"1min":0.90400000000000003,"5min":0.90000000000000002,"15min":0.871},"min":{"1min":0.623,"5min":0.623,"15min":0.623},"max":{"1min":1.2689999999999999,"5min":1.3480000000000001,"15min":1.3480000000000001},"last":0.72099999999999997}]},{"osd":8,"last update":"Fri May 13 23:14:07 2022","interfaces":[{"interface":"back","average":{"1min":0.91700000000000004,"5min":0.94699999999999995,"15min":0.95199999999999996},"min":{"1min":0.56699999999999995,"5min":0.56699999999999995,"15min":0.56699999999999995},"max":{"1min":1.2589999999999999,"5min":1.2669999999999999,"15min":1.2669999999999999},"last":1.048},{"interface":"front","average":{"1min":0.98899999999999999,"5min":1.002,"15min":1.004},"min":{"1min":0.58699999999999997,"5min":0.58699999999999997,"15min":0.58699999999999997},"max":{"1min":1.244,"5min":1.244,"15min":1.244},"last":0.91500000000000004}]},{"osd":9,"last update":"Fri May 13 23:14:40 2022","interfaces":[{"interface":"back","average":{"1min":0.97999999999999998,"5min":1.1000000000000001,"15min":1.1200000000000001},"min":{"1min":0.72099999999999997,"5min":0.72099999999999997,"15min":0.72099999999999997},"max":{"1min":1.2470000000000001,"5min":1.4319999999999999,"15min":1.4319999999999999},"last":0.80700000000000005},{"interface":"front","average":{"1min":0.92500000000000004,"5min":1.0589999999999999,"15min":1.081},"min":{"1min":0.71399999999999997,"5min":0.71399999999999997,"15min":0.71399999999999997},"max":{"1min":1.2210000000000001,"5min":1.4730000000000001,"15min":1.4730000000000001},"last":0.90100000000000002}]},{"osd":10,"last update":"Fri May 13 23:15:01 2022","interfaces":[{"interface":"back","average":{"1min":0.87,"5min":0.98399999999999999,"15min":1.0029999999999999},"min":{"1min":0.65100000000000002,"5min":0.52300000000000002,"15min":0.52300000000000002},"max":{"1min":1.2290000000000001,"5min":1.373,"15min":1.373},"last":0.85499999999999998},{"interface":"front","average":{"1min":0.97099999999999997,"5min":1.095,"15min":1.1160000000000001},"min":{"1min":0.66000000000000003,"5min":0.66000000000000003,"15min":0.66000000000000003},"max":{"1min":1.405,"5min":1.51,"15min":1.51},"last":0.84099999999999997}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.78700000000000003}]}]},{"osd":4,"up_from":33,"seq":141733920828,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6360,"kb_used_data":912,"kb_used_omap":0,"kb_used_meta":5440,"kb_avail":93738792,"statfs":{"total":95995035648,"available":95988523008,"internally_reserved":0,"allocated":933888,"data_stored":442259,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5570560},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri May 13 23:13:35 2022","interfaces":[{"interface":"back","average":{"1min":0.627,"5min":0.56299999999999994,"15min":0.55000000000000004},"min":{"1min":0.39700000000000002,"5min":0.375,"15min":0.375},"max":{"1min":0.91800000000000004,"5min":0.91800000000000004,"15min":0.91800000000000004},"last":0.56000000000000005},{"interface":"front","average":{"1min":0.70499999999999996,"5min":0.58099999999999996,"15min":0.54500000000000004},"min":{"1min":0.47999999999999998,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.97099999999999997,"5min":0.97099999999999997,"15min":0.97099999999999997},"last":0.65200000000000002}]},{"osd":1,"last update":"Fri May 13 23:14:36 2022","interfaces":[{"interface":"back","average":{"1min":0.45300000000000001,"5min":0.40999999999999998,"15min":0.378},"min":{"1min":0.28999999999999998,"5min":0.217,"15min":0.217},"max":{"1min":0.81100000000000005,"5min":0.84499999999999997,"15min":0.84499999999999997},"last":0.42299999999999999},{"interface":"front","average":{"1min":0.48399999999999999,"5min":0.42299999999999999,"15min":0.39700000000000002},"min":{"1min":0.29599999999999999,"5min":0.248,"15min":0.248},"max":{"1min":0.622,"5min":0.79900000000000004,"15min":0.79900000000000004},"last":0.442}]},{"osd":2,"last update":"Fri May 13 23:14:36 2022","interfaces":[{"interface":"back","average":{"1min":0.64500000000000002,"5min":0.56999999999999995,"15min":0.54700000000000004},"min":{"1min":0.45000000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":1.0640000000000001,"5min":1.0640000000000001,"15min":1.0640000000000001},"last":0.51800000000000002},{"interface":"front","average":{"1min":0.63800000000000001,"5min":0.59399999999999997,"15min":0.56999999999999995},"min":{"1min":0.49099999999999999,"5min":0.372,"15min":0.372},"max":{"1min":0.89800000000000002,"5min":0.92900000000000005,"15min":0.92900000000000005},"last":0.62}]},{"osd":3,"last update":"Fri May 13 23:14:36 2022","interfaces":[{"interface":"back","average":{"1min":0.68200000000000005,"5min":0.629,"15min":0.59399999999999997},"min":{"1min":0.49399999999999999,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.84999999999999998,"5min":0.91600000000000004,"15min":0.91600000000000004},"last":0.70799999999999996},{"interface":"front","average":{"1min":0.69899999999999995,"5min":0.65600000000000003,"15min":0.61899999999999999},"min":{"1min":0.441,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":1.1579999999999999,"5min":1.1579999999999999,"15min":1.1579999999999999},"last":0.78400000000000003}]},{"osd":5,"last update":"Fri May 13 23:14:59 2022","interfaces":[{"interface":"back","average":{"1min":0.66300000000000003,"5min":0.61299999999999999,"15min":0.56100000000000005},"min":{"1min":0.42799999999999999,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.96999999999999997,"5min":0.97999999999999998,"15min":0.97999999999999998},"last":0.68700000000000006},{"interface":"front","average":{"1min":0.68899999999999995,"5min":0.61599999999999999,"15min":0.56599999999999995},"min":{"1min":0.40600000000000003,"5min":0.371,"15min":0.371},"max":{"1min":0.94299999999999995,"5min":1.1419999999999999,"15min":1.1419999999999999},"last":0.65100000000000002}]},{"osd":6,"last update":"Fri May 13 23:14:25 2022","interfaces":[{"interface":"back","average":{"1min":0.72899999999999998,"5min":0.69399999999999995,"15min":0.66500000000000004},"min":{"1min":0.502,"5min":0.498,"15min":0.498},"max":{"1min":1.1839999999999999,"5min":1.1839999999999999,"15min":1.1839999999999999},"last":0.92800000000000005},{"interface":"front","average":{"1min":0.74199999999999999,"5min":0.72899999999999998,"15min":0.70199999999999996},"min":{"1min":0.47499999999999998,"5min":0.47499999999999998,"15min":0.47499999999999998},"max":{"1min":0.97199999999999998,"5min":1.046,"15min":1.046},"last":0.86499999999999999}]},{"osd":7,"last update":"Fri May 13 23:14:53 2022","interfaces":[{"interface":"back","average":{"1min":0.63900000000000001,"5min":0.66300000000000003,"15min":0.65300000000000002},"min":{"1min":0.39700000000000002,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.85299999999999998,"5min":0.98599999999999999,"15min":0.98599999999999999},"last":0.75700000000000001},{"interface":"front","average":{"1min":0.66500000000000004,"5min":0.65300000000000002,"15min":0.63600000000000001},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":1.0409999999999999,"5min":1.0409999999999999,"15min":1.0409999999999999},"last":0.80300000000000005}]},{"osd":8,"last update":"Fri May 13 23:14:13 2022","interfaces":[{"interface":"back","average":{"1min":0.81499999999999995,"5min":0.75900000000000001,"15min":0.75},"min":{"1min":0.57399999999999995,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":1.099,"5min":1.099,"15min":1.099},"last":0.82599999999999996},{"interface":"front","average":{"1min":0.76000000000000001,"5min":0.754,"15min":0.753},"min":{"1min":0.44700000000000001,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":1.115,"5min":1.115,"15min":1.115},"last":0.94499999999999995}]},{"osd":9,"last update":"Fri May 13 23:14:45 2022","interfaces":[{"interface":"back","average":{"1min":0.81000000000000005,"5min":0.874,"15min":0.88400000000000001},"min":{"1min":0.51900000000000002,"5min":0.51900000000000002,"15min":0.51900000000000002},"max":{"1min":1.069,"5min":1.1819999999999999,"15min":1.1819999999999999},"last":0.98699999999999999},{"interface":"front","average":{"1min":0.78900000000000003,"5min":0.85299999999999998,"15min":0.86299999999999999},"min":{"1min":0.61499999999999999,"5min":0.61499999999999999,"15min":0.61499999999999999},"max":{"1min":1.216,"5min":1.216,"15min":1.216},"last":0.89400000000000002}]},{"osd":10,"last update":"Fri May 13 23:15:03 2022","interfaces":[{"interface":"back","average":{"1min":0.80200000000000005,"5min":0.80600000000000005,"15min":0.80600000000000005},"min":{"1min":0.5,"5min":0.5,"15min":0.5},"max":{"1min":1.1120000000000001,"5min":1.1120000000000001,"15min":1.1120000000000001},"last":0.875},{"interface":"front","average":{"1min":0.71599999999999997,"5min":0.78400000000000003,"15min":0.79600000000000004},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":1.073,"5min":1.196,"15min":1.196},"last":0.60299999999999998}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73999999999999999}]}]},{"osd":5,"up_from":38,"seq":163208757303,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":7028,"kb_used_data":1452,"kb_used_omap":0,"kb_used_meta":5568,"kb_avail":93738124,"statfs":{"total":95995035648,"available":95987838976,"internally_reserved":0,"allocated":1486848,"data_stored":972900,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5701632},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri May 13 23:14:51 2022","interfaces":[{"interface":"back","average":{"1min":0.59199999999999997,"5min":0.55200000000000005,"15min":0.52800000000000002},"min":{"1min":0.42399999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.83399999999999996,"5min":0.96299999999999997,"15min":0.96299999999999997},"last":0.78000000000000003},{"interface":"front","average":{"1min":0.59099999999999997,"5min":0.55700000000000005,"15min":0.52800000000000002},"min":{"1min":0.40300000000000002,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.85099999999999998,"5min":0.96499999999999997,"15min":0.96499999999999997},"last":0.70899999999999996}]},{"osd":1,"last update":"Fri May 13 23:14:51 2022","interfaces":[{"interface":"back","average":{"1min":0.58599999999999997,"5min":0.52500000000000002,"15min":0.496},"min":{"1min":0.36699999999999999,"5min":0.317,"15min":0.317},"max":{"1min":0.77500000000000002,"5min":0.77500000000000002,"15min":0.77500000000000002},"last":0.61799999999999999},{"interface":"front","average":{"1min":0.58499999999999996,"5min":0.52600000000000002,"15min":0.48799999999999999},"min":{"1min":0.36499999999999999,"5min":0.313,"15min":0.313},"max":{"1min":0.78800000000000003,"5min":0.78800000000000003,"15min":0.78800000000000003},"last":0.64300000000000002}]},{"osd":2,"last update":"Fri May 13 23:14:51 2022","interfaces":[{"interface":"back","average":{"1min":0.432,"5min":0.39600000000000002,"15min":0.36799999999999999},"min":{"1min":0.28599999999999998,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":0.54500000000000004,"5min":0.65600000000000003,"15min":0.65600000000000003},"last":0.51100000000000001},{"interface":"front","average":{"1min":0.439,"5min":0.40899999999999997,"15min":0.38},"min":{"1min":0.28100000000000003,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.69699999999999995,"5min":0.75700000000000001,"15min":0.75700000000000001},"last":0.53900000000000003}]},{"osd":3,"last update":"Fri May 13 23:14:51 2022","interfaces":[{"interface":"back","average":{"1min":0.60699999999999998,"5min":0.57499999999999996,"15min":0.56200000000000006},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":0.95899999999999996,"5min":0.97999999999999998,"15min":0.97999999999999998},"last":0.96399999999999997},{"interface":"front","average":{"1min":0.65500000000000003,"5min":0.59999999999999998,"15min":0.56000000000000005},"min":{"1min":0.47099999999999997,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":0.89100000000000001,"5min":1,"15min":1},"last":0.753}]},{"osd":4,"last update":"Fri May 13 23:14:51 2022","interfaces":[{"interface":"back","average":{"1min":0.624,"5min":0.57699999999999996,"15min":0.54300000000000004},"min":{"1min":0.45800000000000002,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.82599999999999996,"5min":0.86399999999999999,"15min":0.86399999999999999},"last":0.76500000000000001},{"interface":"front","average":{"1min":0.58499999999999996,"5min":0.55300000000000005,"15min":0.53500000000000003},"min":{"1min":0.41299999999999998,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.80000000000000004,"5min":0.89500000000000002,"15min":0.89500000000000002},"last":0.91800000000000004}]},{"osd":6,"last update":"Fri May 13 23:14:25 2022","interfaces":[{"interface":"back","average":{"1min":0.66200000000000003,"5min":0.63,"15min":0.628},"min":{"1min":0.48399999999999999,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":0.89800000000000002,"5min":0.95399999999999996,"15min":0.95399999999999996},"last":0.84099999999999997},{"interface":"front","average":{"1min":0.71599999999999997,"5min":0.64500000000000002,"15min":0.628},"min":{"1min":0.503,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":1.026,"5min":1.026,"15min":1.026},"last":0.86799999999999999}]},{"osd":7,"last update":"Fri May 13 23:14:46 2022","interfaces":[{"interface":"back","average":{"1min":0.68700000000000006,"5min":0.64900000000000002,"15min":0.64500000000000002},"min":{"1min":0.51200000000000001,"5min":0.379,"15min":0.379},"max":{"1min":0.92700000000000005,"5min":1.0489999999999999,"15min":1.0489999999999999},"last":0.90000000000000002},{"interface":"front","average":{"1min":0.70699999999999996,"5min":0.65800000000000003,"15min":0.64600000000000002},"min":{"1min":0.53200000000000003,"5min":0.442,"15min":0.442},"max":{"1min":0.96699999999999997,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.89200000000000002}]},{"osd":8,"last update":"Fri May 13 23:15:07 2022","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.60999999999999999,"15min":0.62},"min":{"1min":0.36899999999999999,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.90300000000000002,"5min":0.999,"15min":0.999},"last":0.83199999999999996},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.56399999999999995,"15min":0.56799999999999995},"min":{"1min":0.38400000000000001,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.72399999999999998,"5min":0.82699999999999996,"15min":0.82699999999999996},"last":0.60299999999999998}]},{"osd":9,"last update":"Fri May 13 23:14:44 2022","interfaces":[{"interface":"back","average":{"1min":0.73399999999999999,"5min":0.69499999999999995,"15min":0.68899999999999995},"min":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":0.90200000000000002,"5min":0.90200000000000002,"15min":0.90200000000000002},"last":0.79600000000000004},{"interface":"front","average":{"1min":0.77400000000000002,"5min":0.72399999999999998,"15min":0.71599999999999997},"min":{"1min":0.55000000000000004,"5min":0.505,"15min":0.505},"max":{"1min":0.92900000000000005,"5min":0.99299999999999999,"15min":0.99299999999999999},"last":0.81000000000000005}]},{"osd":10,"last update":"Fri May 13 23:15:02 2022","interfaces":[{"interface":"back","average":{"1min":0.67800000000000005,"5min":0.67800000000000005,"15min":0.67800000000000005},"min":{"1min":0.496,"5min":0.496,"15min":0.496},"max":{"1min":0.84799999999999998,"5min":0.90800000000000003,"15min":0.90800000000000003},"last":0.875},{"interface":"front","average":{"1min":0.72499999999999998,"5min":0.71599999999999997,"15min":0.71399999999999997},"min":{"1min":0.54900000000000004,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":0.89000000000000001,"5min":0.94499999999999995,"15min":0.94499999999999995},"last":0.85099999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2074,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":3184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":3316,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":2688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":674,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":2942,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1422,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1094,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":3196,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1206,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2022-05-13T23:15:19.515 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:23b0658391ce94ca40799469eec00c28b298816c shell --fsid 16cf5f88-d311-11ec-be40-001a4aab830c -- ceph pg dump --format=json 2022-05-13T23:15:20.200 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/16cf5f88-d311-11ec-be40-001a4aab830c/mon.a/config 2022-05-13T23:15:20.734 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:15:20 smithi002 ceph-mon[71816]: from='client.24877 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2022-05-13T23:15:20.734 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:15:20 smithi002 ceph-mon[71816]: pgmap v322: 97 pgs: 97 active+clean; 584 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-05-13T23:15:20.734 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:15:20 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-13T23:15:20.735 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:15:20 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-13T23:15:20.735 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:15:20 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-13T23:15:20.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:15:20 smithi047 ceph-mon[77147]: from='client.24877 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2022-05-13T23:15:20.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:15:20 smithi047 ceph-mon[77147]: pgmap v322: 97 pgs: 97 active+clean; 584 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-05-13T23:15:20.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:15:20 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-13T23:15:20.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:15:20 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-13T23:15:20.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:15:20 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-13T23:15:20.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:15:20 smithi115 ceph-mon[78593]: from='client.24877 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2022-05-13T23:15:20.863 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:15:20 smithi115 ceph-mon[78593]: pgmap v322: 97 pgs: 97 active+clean; 584 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-05-13T23:15:20.863 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:15:20 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-13T23:15:20.864 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:15:20 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-13T23:15:20.864 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:15:20 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-13T23:15:22.029 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:22.026+0000 7f8d68055700 1 -- 172.21.15.2:0/2679956619 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8d6006edb0 msgr2=0x7f8d6006f220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:15:22.029 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:22.026+0000 7f8d68055700 1 --2- 172.21.15.2:0/2679956619 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8d6006edb0 0x7f8d6006f220 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f8d50004870 tx=0x7f8d50009cb0 comp rx=0 tx=0).stop 2022-05-13T23:15:22.030 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:22.026+0000 7f8d68055700 1 -- 172.21.15.2:0/2679956619 shutdown_connections 2022-05-13T23:15:22.030 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:22.026+0000 7f8d68055700 1 --2- 172.21.15.2:0/2679956619 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f8d60103580 0x7f8d60105960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:22.030 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:22.026+0000 7f8d68055700 1 --2- 172.21.15.2:0/2679956619 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8d60070370 0x7f8d6006e870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:22.030 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:22.026+0000 7f8d68055700 1 --2- 172.21.15.2:0/2679956619 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8d6006edb0 0x7f8d6006f220 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:22.031 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:22.026+0000 7f8d68055700 1 -- 172.21.15.2:0/2679956619 >> 172.21.15.2:0/2679956619 conn(0x7f8d600f6e00 msgr2=0x7f8d600f9210 unknown :-1 s=STATE_NONE l=0).mark_down 2022-05-13T23:15:22.031 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:22.026+0000 7f8d68055700 1 -- 172.21.15.2:0/2679956619 shutdown_connections 2022-05-13T23:15:22.031 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:22.026+0000 7f8d68055700 1 -- 172.21.15.2:0/2679956619 wait complete. 2022-05-13T23:15:22.033 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:22.027+0000 7f8d68055700 1 Processor -- start 2022-05-13T23:15:22.033 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:22.027+0000 7f8d68055700 1 -- start start 2022-05-13T23:15:22.034 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:22.028+0000 7f8d68055700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8d6006edb0 0x7f8d6010e5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:15:22.034 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:22.028+0000 7f8d68055700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8d60070370 0x7f8d6010eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:15:22.034 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:22.028+0000 7f8d68055700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f8d60103580 0x7f8d60115bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:15:22.034 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:22.028+0000 7f8d68055700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d60116100 con 0x7f8d6006edb0 2022-05-13T23:15:22.035 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:22.028+0000 7f8d68055700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d60116270 con 0x7f8d60070370 2022-05-13T23:15:22.035 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:22.028+0000 7f8d68055700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d60116430 con 0x7f8d60103580 2022-05-13T23:15:22.036 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:22.028+0000 7f8d65df1700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8d6006edb0 0x7f8d6010e5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:15:22.036 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:22.028+0000 7f8d665f2700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f8d60103580 0x7f8d60115bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:15:22.036 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:22.028+0000 7f8d65df1700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8d6006edb0 0x7f8d6010e5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:46008/0 (socket says 172.21.15.2:46008) 2022-05-13T23:15:22.037 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:22.028+0000 7f8d65df1700 1 -- 172.21.15.2:0/2452170044 learned_addr learned my addr 172.21.15.2:0/2452170044 (peer_addr_for_me v2:172.21.15.2:0/0) 2022-05-13T23:15:22.038 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:22.028+0000 7f8d655f0700 1 --2- 172.21.15.2:0/2452170044 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8d60070370 0x7f8d6010eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:15:22.038 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:22.029+0000 7f8d65df1700 1 -- 172.21.15.2:0/2452170044 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f8d60103580 msgr2=0x7f8d60115bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:15:22.038 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:22.029+0000 7f8d65df1700 1 --2- 172.21.15.2:0/2452170044 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f8d60103580 0x7f8d60115bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:22.038 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:22.029+0000 7f8d65df1700 1 -- 172.21.15.2:0/2452170044 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8d60070370 msgr2=0x7f8d6010eb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:15:22.039 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:22.029+0000 7f8d65df1700 1 --2- 172.21.15.2:0/2452170044 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8d60070370 0x7f8d6010eb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:22.039 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:22.029+0000 7f8d65df1700 1 -- 172.21.15.2:0/2452170044 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d5c005040 con 0x7f8d6006edb0 2022-05-13T23:15:22.039 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:22.029+0000 7f8d665f2700 1 --2- 172.21.15.2:0/2452170044 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f8d60103580 0x7f8d60115bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-05-13T23:15:22.039 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:22.029+0000 7f8d65df1700 1 --2- 172.21.15.2:0/2452170044 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8d6006edb0 0x7f8d6010e5f0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f8d5c0048f0 tx=0x7f8d5c009cb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-05-13T23:15:22.039 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:22.029+0000 7f8d567fc700 1 -- 172.21.15.2:0/2452170044 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d5c012570 con 0x7f8d6006edb0 2022-05-13T23:15:22.040 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:22.029+0000 7f8d567fc700 1 -- 172.21.15.2:0/2452170044 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1087+0+0 (secure 0 0 0) 0x7f8d5c00f040 con 0x7f8d6006edb0 2022-05-13T23:15:22.040 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:22.029+0000 7f8d567fc700 1 -- 172.21.15.2:0/2452170044 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d5c013b90 con 0x7f8d6006edb0 2022-05-13T23:15:22.040 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:22.029+0000 7f8d68055700 1 -- 172.21.15.2:0/2452170044 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8d50005040 con 0x7f8d6006edb0 2022-05-13T23:15:22.040 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:22.029+0000 7f8d68055700 1 -- 172.21.15.2:0/2452170044 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8d60116aa0 con 0x7f8d6006edb0 2022-05-13T23:15:22.041 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:22.031+0000 7f8d68055700 1 -- 172.21.15.2:0/2452170044 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8d60108fc0 con 0x7f8d6006edb0 2022-05-13T23:15:22.043 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:22.033+0000 7f8d567fc700 1 -- 172.21.15.2:0/2452170044 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 16) v1 ==== 125136+0+0 (secure 0 0 0) 0x7f8d5c0126d0 con 0x7f8d6006edb0 2022-05-13T23:15:22.043 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:22.034+0000 7f8d567fc700 1 --2- 172.21.15.2:0/2452170044 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f8d4c096d60 0x7f8d4c099210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:15:22.044 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:22.034+0000 7f8d567fc700 1 -- 172.21.15.2:0/2452170044 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8120+0+0 (secure 0 0 0) 0x7f8d5c0bef40 con 0x7f8d6006edb0 2022-05-13T23:15:22.044 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:22.034+0000 7f8d655f0700 1 --2- 172.21.15.2:0/2452170044 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f8d4c096d60 0x7f8d4c099210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:15:22.044 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:22.035+0000 7f8d655f0700 1 --2- 172.21.15.2:0/2452170044 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f8d4c096d60 0x7f8d4c099210 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f8d50013490 tx=0x7f8d50009cb0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-05-13T23:15:22.045 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:22.038+0000 7f8d567fc700 1 -- 172.21.15.2:0/2452170044 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+165215 (secure 0 0 0) 0x7f8d5c08c1f0 con 0x7f8d6006edb0 2022-05-13T23:15:22.244 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:22.241+0000 7f8d68055700 1 -- 172.21.15.2:0/2452170044 --> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f8d6002a5b0 con 0x7f8d4c096d60 2022-05-13T23:15:22.255 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:22.253+0000 7f8d567fc700 1 -- 172.21.15.2:0/2452170044 <== mgr.14148 v2:172.21.15.2:6800/2788339627 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+340119 (secure 0 0 0) 0x7f8d6002a5b0 con 0x7f8d4c096d60 2022-05-13T23:15:22.262 INFO:teuthology.orchestra.run.smithi002.stdout: 2022-05-13T23:15:22.265 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:22.260+0000 7f8d68055700 1 -- 172.21.15.2:0/2452170044 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f8d4c096d60 msgr2=0x7f8d4c099210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:15:22.265 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:22.260+0000 7f8d68055700 1 --2- 172.21.15.2:0/2452170044 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f8d4c096d60 0x7f8d4c099210 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f8d50013490 tx=0x7f8d50009cb0 comp rx=0 tx=0).stop 2022-05-13T23:15:22.266 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:22.260+0000 7f8d68055700 1 -- 172.21.15.2:0/2452170044 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8d6006edb0 msgr2=0x7f8d6010e5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:15:22.266 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:22.260+0000 7f8d68055700 1 --2- 172.21.15.2:0/2452170044 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8d6006edb0 0x7f8d6010e5f0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f8d5c0048f0 tx=0x7f8d5c009cb0 comp rx=0 tx=0).stop 2022-05-13T23:15:22.266 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:22.260+0000 7f8d68055700 1 -- 172.21.15.2:0/2452170044 shutdown_connections 2022-05-13T23:15:22.266 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:22.260+0000 7f8d68055700 1 --2- 172.21.15.2:0/2452170044 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f8d60103580 0x7f8d60115bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:22.266 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:22.260+0000 7f8d68055700 1 --2- 172.21.15.2:0/2452170044 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f8d4c096d60 0x7f8d4c099210 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:22.267 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:22.260+0000 7f8d68055700 1 --2- 172.21.15.2:0/2452170044 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8d6006edb0 0x7f8d6010e5f0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:22.267 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:22.260+0000 7f8d68055700 1 --2- 172.21.15.2:0/2452170044 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8d60070370 0x7f8d6010eb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:22.267 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:22.260+0000 7f8d68055700 1 -- 172.21.15.2:0/2452170044 >> 172.21.15.2:0/2452170044 conn(0x7f8d600f6e00 msgr2=0x7f8d600f91e0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-05-13T23:15:22.267 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:22.260+0000 7f8d68055700 1 -- 172.21.15.2:0/2452170044 shutdown_connections 2022-05-13T23:15:22.268 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:22.260+0000 7f8d68055700 1 -- 172.21.15.2:0/2452170044 wait complete. 2022-05-13T23:15:22.268 INFO:teuthology.orchestra.run.smithi002.stderr:dumped all 2022-05-13T23:15:22.572 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:15:22 smithi002 ceph-mon[71816]: pgmap v323: 97 pgs: 97 active+clean; 584 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2022-05-13T23:15:22.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:15:22 smithi047 ceph-mon[77147]: pgmap v323: 97 pgs: 97 active+clean; 584 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2022-05-13T23:15:22.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:15:22 smithi115 ceph-mon[78593]: pgmap v323: 97 pgs: 97 active+clean; 584 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2022-05-13T23:15:23.001 INFO:teuthology.orchestra.run.smithi002.stdout:{"pg_ready":true,"pg_map":{"version":323,"stamp":"2022-05-13T23:15:21.250862+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":597994,"num_objects":100,"num_object_clones":0,"num_object_copies":300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":100,"num_whiteouts":0,"num_read":721,"num_read_kb":421,"num_write":386,"num_write_kb":2995,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":1057946,"num_keys_recovered":10,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":316,"ondisk_log_size":316,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":79744,"kb_used_data":12640,"kb_used_omap":0,"kb_used_meta":67008,"kb_avail":1124862080,"statfs":{"total":1151940427776,"available":1151858769920,"internally_reserved":0,"allocated":12943360,"data_stored":6908300,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":68616192},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":140,"num_read_kb":70,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003611"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2022-05-13T23:15:05.411066+0000","last_change":"2022-05-13T23:14:16.267128+0000","last_active":"2022-05-13T23:15:05.411066+0000","last_peered":"2022-05-13T23:15:05.411066+0000","last_clean":"2022-05-13T23:15:05.411066+0000","last_became_active":"2022-05-13T23:14:16.266687+0000","last_became_peered":"2022-05-13T23:14:16.266687+0000","last_unstale":"2022-05-13T23:15:05.411066+0000","last_undegraded":"2022-05-13T23:15:05.411066+0000","last_fullsized":"2022-05-13T23:15:05.411066+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_clean_scrub_stamp":"2022-05-13T23:14:15.659362+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T04:40:46.150557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2022-05-13T23:15:05.514842+0000","last_change":"2022-05-13T23:14:16.272880+0000","last_active":"2022-05-13T23:15:05.514842+0000","last_peered":"2022-05-13T23:15:05.514842+0000","last_clean":"2022-05-13T23:15:05.514842+0000","last_became_active":"2022-05-13T23:14:16.272473+0000","last_became_peered":"2022-05-13T23:14:16.272473+0000","last_unstale":"2022-05-13T23:15:05.514842+0000","last_undegraded":"2022-05-13T23:15:05.514842+0000","last_fullsized":"2022-05-13T23:15:05.514842+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_clean_scrub_stamp":"2022-05-13T23:14:15.659362+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T00:32:19.185037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2022-05-13T23:15:05.515282+0000","last_change":"2022-05-13T23:14:16.270961+0000","last_active":"2022-05-13T23:15:05.515282+0000","last_peered":"2022-05-13T23:15:05.515282+0000","last_clean":"2022-05-13T23:15:05.515282+0000","last_became_active":"2022-05-13T23:14:16.269942+0000","last_became_peered":"2022-05-13T23:14:16.269942+0000","last_unstale":"2022-05-13T23:15:05.515282+0000","last_undegraded":"2022-05-13T23:15:05.515282+0000","last_fullsized":"2022-05-13T23:15:05.515282+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_clean_scrub_stamp":"2022-05-13T23:14:15.659362+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T10:03:03.774806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2022-05-13T23:15:05.412122+0000","last_change":"2022-05-13T23:14:16.255885+0000","last_active":"2022-05-13T23:15:05.412122+0000","last_peered":"2022-05-13T23:15:05.412122+0000","last_clean":"2022-05-13T23:15:05.412122+0000","last_became_active":"2022-05-13T23:14:16.255551+0000","last_became_peered":"2022-05-13T23:14:16.255551+0000","last_unstale":"2022-05-13T23:15:05.412122+0000","last_undegraded":"2022-05-13T23:15:05.412122+0000","last_fullsized":"2022-05-13T23:15:05.412122+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_clean_scrub_stamp":"2022-05-13T23:14:15.659362+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T10:26:40.578066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2022-05-13T23:15:05.411360+0000","last_change":"2022-05-13T23:14:16.271384+0000","last_active":"2022-05-13T23:15:05.411360+0000","last_peered":"2022-05-13T23:15:05.411360+0000","last_clean":"2022-05-13T23:15:05.411360+0000","last_became_active":"2022-05-13T23:14:16.271045+0000","last_became_peered":"2022-05-13T23:14:16.271045+0000","last_unstale":"2022-05-13T23:15:05.411360+0000","last_undegraded":"2022-05-13T23:15:05.411360+0000","last_fullsized":"2022-05-13T23:15:05.411360+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_clean_scrub_stamp":"2022-05-13T23:14:15.659362+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T04:51:50.618628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2022-05-13T23:15:05.411722+0000","last_change":"2022-05-13T23:14:16.275906+0000","last_active":"2022-05-13T23:15:05.411722+0000","last_peered":"2022-05-13T23:15:05.411722+0000","last_clean":"2022-05-13T23:15:05.411722+0000","last_became_active":"2022-05-13T23:14:16.275651+0000","last_became_peered":"2022-05-13T23:14:16.275651+0000","last_unstale":"2022-05-13T23:15:05.411722+0000","last_undegraded":"2022-05-13T23:15:05.411722+0000","last_fullsized":"2022-05-13T23:15:05.411722+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_clean_scrub_stamp":"2022-05-13T23:14:15.659362+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T02:29:33.019268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2022-05-13T23:15:05.408529+0000","last_change":"2022-05-13T23:14:16.276150+0000","last_active":"2022-05-13T23:15:05.408529+0000","last_peered":"2022-05-13T23:15:05.408529+0000","last_clean":"2022-05-13T23:15:05.408529+0000","last_became_active":"2022-05-13T23:14:16.275751+0000","last_became_peered":"2022-05-13T23:14:16.275751+0000","last_unstale":"2022-05-13T23:15:05.408529+0000","last_undegraded":"2022-05-13T23:15:05.408529+0000","last_fullsized":"2022-05-13T23:15:05.408529+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_clean_scrub_stamp":"2022-05-13T23:14:15.659362+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T04:01:48.496870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2022-05-13T23:15:05.416379+0000","last_change":"2022-05-13T23:14:16.268816+0000","last_active":"2022-05-13T23:15:05.416379+0000","last_peered":"2022-05-13T23:15:05.416379+0000","last_clean":"2022-05-13T23:15:05.416379+0000","last_became_active":"2022-05-13T23:14:16.268429+0000","last_became_peered":"2022-05-13T23:14:16.268429+0000","last_unstale":"2022-05-13T23:15:05.416379+0000","last_undegraded":"2022-05-13T23:15:05.416379+0000","last_fullsized":"2022-05-13T23:15:05.416379+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_clean_scrub_stamp":"2022-05-13T23:14:15.659362+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T00:14:51.685287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2022-05-13T23:15:05.405791+0000","last_change":"2022-05-13T23:14:16.263971+0000","last_active":"2022-05-13T23:15:05.405791+0000","last_peered":"2022-05-13T23:15:05.405791+0000","last_clean":"2022-05-13T23:15:05.405791+0000","last_became_active":"2022-05-13T23:14:16.263625+0000","last_became_peered":"2022-05-13T23:14:16.263625+0000","last_unstale":"2022-05-13T23:15:05.405791+0000","last_undegraded":"2022-05-13T23:15:05.405791+0000","last_fullsized":"2022-05-13T23:15:05.405791+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_clean_scrub_stamp":"2022-05-13T23:14:15.659362+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T03:11:11.037746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":22,"reported_epoch":85,"state":"active+clean","last_fresh":"2022-05-13T23:15:06.541777+0000","last_change":"2022-05-13T23:15:06.541777+0000","last_active":"2022-05-13T23:15:06.541777+0000","last_peered":"2022-05-13T23:15:06.541777+0000","last_clean":"2022-05-13T23:15:06.541777+0000","last_became_active":"2022-05-13T23:15:06.541253+0000","last_became_peered":"2022-05-13T23:15:06.541253+0000","last_unstale":"2022-05-13T23:15:06.541777+0000","last_undegraded":"2022-05-13T23:15:06.541777+0000","last_fullsized":"2022-05-13T23:15:06.541777+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_clean_scrub_stamp":"2022-05-13T23:14:15.659362+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T02:54:36.682507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,9],"acting":[10,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2022-05-13T23:15:05.408958+0000","last_change":"2022-05-13T23:14:16.267256+0000","last_active":"2022-05-13T23:15:05.408958+0000","last_peered":"2022-05-13T23:15:05.408958+0000","last_clean":"2022-05-13T23:15:05.408958+0000","last_became_active":"2022-05-13T23:14:16.266868+0000","last_became_peered":"2022-05-13T23:14:16.266868+0000","last_unstale":"2022-05-13T23:15:05.408958+0000","last_undegraded":"2022-05-13T23:15:05.408958+0000","last_fullsized":"2022-05-13T23:15:05.408958+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_clean_scrub_stamp":"2022-05-13T23:14:15.659362+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T03:28:58.592648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2022-05-13T23:15:05.412058+0000","last_change":"2022-05-13T23:14:16.256051+0000","last_active":"2022-05-13T23:15:05.412058+0000","last_peered":"2022-05-13T23:15:05.412058+0000","last_clean":"2022-05-13T23:15:05.412058+0000","last_became_active":"2022-05-13T23:14:16.255729+0000","last_became_peered":"2022-05-13T23:14:16.255729+0000","last_unstale":"2022-05-13T23:15:05.412058+0000","last_undegraded":"2022-05-13T23:15:05.412058+0000","last_fullsized":"2022-05-13T23:15:05.412058+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_clean_scrub_stamp":"2022-05-13T23:14:15.659362+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T09:34:22.448938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2022-05-13T23:15:05.406411+0000","last_change":"2022-05-13T23:14:16.266285+0000","last_active":"2022-05-13T23:15:05.406411+0000","last_peered":"2022-05-13T23:15:05.406411+0000","last_clean":"2022-05-13T23:15:05.406411+0000","last_became_active":"2022-05-13T23:14:16.264899+0000","last_became_peered":"2022-05-13T23:14:16.264899+0000","last_unstale":"2022-05-13T23:15:05.406411+0000","last_undegraded":"2022-05-13T23:15:05.406411+0000","last_fullsized":"2022-05-13T23:15:05.406411+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_clean_scrub_stamp":"2022-05-13T23:14:15.659362+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T09:50:56.211965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2022-05-13T23:15:05.514102+0000","last_change":"2022-05-13T23:14:16.271503+0000","last_active":"2022-05-13T23:15:05.514102+0000","last_peered":"2022-05-13T23:15:05.514102+0000","last_clean":"2022-05-13T23:15:05.514102+0000","last_became_active":"2022-05-13T23:14:16.271135+0000","last_became_peered":"2022-05-13T23:14:16.271135+0000","last_unstale":"2022-05-13T23:15:05.514102+0000","last_undegraded":"2022-05-13T23:15:05.514102+0000","last_fullsized":"2022-05-13T23:15:05.514102+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_clean_scrub_stamp":"2022-05-13T23:14:15.659362+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T05:01:25.159085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2022-05-13T23:15:05.513931+0000","last_change":"2022-05-13T23:14:16.265771+0000","last_active":"2022-05-13T23:15:05.513931+0000","last_peered":"2022-05-13T23:15:05.513931+0000","last_clean":"2022-05-13T23:15:05.513931+0000","last_became_active":"2022-05-13T23:14:16.265357+0000","last_became_peered":"2022-05-13T23:14:16.265357+0000","last_unstale":"2022-05-13T23:15:05.513931+0000","last_undegraded":"2022-05-13T23:15:05.513931+0000","last_fullsized":"2022-05-13T23:15:05.513931+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_clean_scrub_stamp":"2022-05-13T23:14:15.659362+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T02:16:47.449089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2022-05-13T23:15:05.405257+0000","last_change":"2022-05-13T23:14:16.270173+0000","last_active":"2022-05-13T23:15:05.405257+0000","last_peered":"2022-05-13T23:15:05.405257+0000","last_clean":"2022-05-13T23:15:05.405257+0000","last_became_active":"2022-05-13T23:14:16.269859+0000","last_became_peered":"2022-05-13T23:14:16.269859+0000","last_unstale":"2022-05-13T23:15:05.405257+0000","last_undegraded":"2022-05-13T23:15:05.405257+0000","last_fullsized":"2022-05-13T23:15:05.405257+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_clean_scrub_stamp":"2022-05-13T23:14:15.659362+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T03:31:45.566757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2022-05-13T23:15:05.410783+0000","last_change":"2022-05-13T23:14:16.269290+0000","last_active":"2022-05-13T23:15:05.410783+0000","last_peered":"2022-05-13T23:15:05.410783+0000","last_clean":"2022-05-13T23:15:05.410783+0000","last_became_active":"2022-05-13T23:14:16.268985+0000","last_became_peered":"2022-05-13T23:14:16.268985+0000","last_unstale":"2022-05-13T23:15:05.410783+0000","last_undegraded":"2022-05-13T23:15:05.410783+0000","last_fullsized":"2022-05-13T23:15:05.410783+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_clean_scrub_stamp":"2022-05-13T23:14:15.659362+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T03:58:45.885549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2022-05-13T23:15:05.411828+0000","last_change":"2022-05-13T23:14:16.266204+0000","last_active":"2022-05-13T23:15:05.411828+0000","last_peered":"2022-05-13T23:15:05.411828+0000","last_clean":"2022-05-13T23:15:05.411828+0000","last_became_active":"2022-05-13T23:14:16.262256+0000","last_became_peered":"2022-05-13T23:14:16.262256+0000","last_unstale":"2022-05-13T23:15:05.411828+0000","last_undegraded":"2022-05-13T23:15:05.411828+0000","last_fullsized":"2022-05-13T23:15:05.411828+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_clean_scrub_stamp":"2022-05-13T23:14:15.659362+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T00:53:26.594598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2022-05-13T23:15:05.417356+0000","last_change":"2022-05-13T23:14:16.252421+0000","last_active":"2022-05-13T23:15:05.417356+0000","last_peered":"2022-05-13T23:15:05.417356+0000","last_clean":"2022-05-13T23:15:05.417356+0000","last_became_active":"2022-05-13T23:14:16.250585+0000","last_became_peered":"2022-05-13T23:14:16.250585+0000","last_unstale":"2022-05-13T23:15:05.417356+0000","last_undegraded":"2022-05-13T23:15:05.417356+0000","last_fullsized":"2022-05-13T23:15:05.417356+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_clean_scrub_stamp":"2022-05-13T23:14:15.659362+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T00:13:24.393163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"83'13","reported_seq":81,"reported_epoch":85,"state":"active+clean","last_fresh":"2022-05-13T23:15:15.396211+0000","last_change":"2022-05-13T23:14:15.683837+0000","last_active":"2022-05-13T23:15:15.396211+0000","last_peered":"2022-05-13T23:15:15.396211+0000","last_clean":"2022-05-13T23:15:15.396211+0000","last_became_active":"2022-05-13T23:14:15.682976+0000","last_became_peered":"2022-05-13T23:14:15.682976+0000","last_unstale":"2022-05-13T23:15:15.396211+0000","last_undegraded":"2022-05-13T23:15:15.396211+0000","last_fullsized":"2022-05-13T23:15:15.396211+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:14.344206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:14.344206+0000","last_clean_scrub_stamp":"2022-05-13T23:14:14.344206+0000","objects_scrubbed":0,"log_size":13,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T01:05:36.612390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":644,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":55,"num_read_kb":55,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2022-05-13T23:15:05.417111+0000","last_change":"2022-05-13T23:14:16.264501+0000","last_active":"2022-05-13T23:15:05.417111+0000","last_peered":"2022-05-13T23:15:05.417111+0000","last_clean":"2022-05-13T23:15:05.417111+0000","last_became_active":"2022-05-13T23:14:16.264105+0000","last_became_peered":"2022-05-13T23:14:16.264105+0000","last_unstale":"2022-05-13T23:15:05.417111+0000","last_undegraded":"2022-05-13T23:15:05.417111+0000","last_fullsized":"2022-05-13T23:15:05.417111+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_clean_scrub_stamp":"2022-05-13T23:14:15.659362+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T00:21:48.857254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"83'2","reported_seq":15,"reported_epoch":84,"state":"active+clean","last_fresh":"2022-05-13T23:15:05.417387+0000","last_change":"2022-05-13T23:14:15.701757+0000","last_active":"2022-05-13T23:15:05.417387+0000","last_peered":"2022-05-13T23:15:05.417387+0000","last_clean":"2022-05-13T23:15:05.417387+0000","last_became_active":"2022-05-13T23:14:15.701428+0000","last_became_peered":"2022-05-13T23:14:15.701428+0000","last_unstale":"2022-05-13T23:15:05.417387+0000","last_undegraded":"2022-05-13T23:15:05.417387+0000","last_fullsized":"2022-05-13T23:15:05.417387+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:14.344206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:14.344206+0000","last_clean_scrub_stamp":"2022-05-13T23:14:14.344206+0000","objects_scrubbed":0,"log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T05:26:07.810433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2022-05-13T23:15:05.515198+0000","last_change":"2022-05-13T23:14:16.280002+0000","last_active":"2022-05-13T23:15:05.515198+0000","last_peered":"2022-05-13T23:15:05.515198+0000","last_clean":"2022-05-13T23:15:05.515198+0000","last_became_active":"2022-05-13T23:14:16.279621+0000","last_became_peered":"2022-05-13T23:14:16.279621+0000","last_unstale":"2022-05-13T23:15:05.515198+0000","last_undegraded":"2022-05-13T23:15:05.515198+0000","last_fullsized":"2022-05-13T23:15:05.515198+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_clean_scrub_stamp":"2022-05-13T23:14:15.659362+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T06:42:49.090643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"83'12","reported_seq":26,"reported_epoch":84,"state":"active+clean","last_fresh":"2022-05-13T23:15:05.411853+0000","last_change":"2022-05-13T23:14:15.683457+0000","last_active":"2022-05-13T23:15:05.411853+0000","last_peered":"2022-05-13T23:15:05.411853+0000","last_clean":"2022-05-13T23:15:05.411853+0000","last_became_active":"2022-05-13T23:14:15.682883+0000","last_became_peered":"2022-05-13T23:14:15.682883+0000","last_unstale":"2022-05-13T23:15:05.411853+0000","last_undegraded":"2022-05-13T23:15:05.411853+0000","last_fullsized":"2022-05-13T23:15:05.411853+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:14.344206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:14.344206+0000","last_clean_scrub_stamp":"2022-05-13T23:14:14.344206+0000","objects_scrubbed":0,"log_size":12,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T07:38:58.996672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":16,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2022-05-13T23:15:05.405436+0000","last_change":"2022-05-13T23:14:16.261315+0000","last_active":"2022-05-13T23:15:05.405436+0000","last_peered":"2022-05-13T23:15:05.405436+0000","last_clean":"2022-05-13T23:15:05.405436+0000","last_became_active":"2022-05-13T23:14:16.260265+0000","last_became_peered":"2022-05-13T23:14:16.260265+0000","last_unstale":"2022-05-13T23:15:05.405436+0000","last_undegraded":"2022-05-13T23:15:05.405436+0000","last_fullsized":"2022-05-13T23:15:05.405436+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_clean_scrub_stamp":"2022-05-13T23:14:15.659362+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T08:02:58.080071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"83'5","reported_seq":19,"reported_epoch":84,"state":"active+clean","last_fresh":"2022-05-13T23:15:05.412036+0000","last_change":"2022-05-13T23:14:15.682693+0000","last_active":"2022-05-13T23:15:05.412036+0000","last_peered":"2022-05-13T23:15:05.412036+0000","last_clean":"2022-05-13T23:15:05.412036+0000","last_became_active":"2022-05-13T23:14:15.682414+0000","last_became_peered":"2022-05-13T23:14:15.682414+0000","last_unstale":"2022-05-13T23:15:05.412036+0000","last_undegraded":"2022-05-13T23:15:05.412036+0000","last_fullsized":"2022-05-13T23:15:05.412036+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:14.344206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:14.344206+0000","last_clean_scrub_stamp":"2022-05-13T23:14:14.344206+0000","objects_scrubbed":0,"log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T09:15:17.699871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":554,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2022-05-13T23:15:05.406324+0000","last_change":"2022-05-13T23:14:16.261044+0000","last_active":"2022-05-13T23:15:05.406324+0000","last_peered":"2022-05-13T23:15:05.406324+0000","last_clean":"2022-05-13T23:15:05.406324+0000","last_became_active":"2022-05-13T23:14:16.260687+0000","last_became_peered":"2022-05-13T23:14:16.260687+0000","last_unstale":"2022-05-13T23:15:05.406324+0000","last_undegraded":"2022-05-13T23:15:05.406324+0000","last_fullsized":"2022-05-13T23:15:05.406324+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_clean_scrub_stamp":"2022-05-13T23:14:15.659362+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-14T23:38:23.413188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"83'5","reported_seq":76,"reported_epoch":85,"state":"active+clean","last_fresh":"2022-05-13T23:15:17.400976+0000","last_change":"2022-05-13T23:14:15.678240+0000","last_active":"2022-05-13T23:15:17.400976+0000","last_peered":"2022-05-13T23:15:17.400976+0000","last_clean":"2022-05-13T23:15:17.400976+0000","last_became_active":"2022-05-13T23:14:15.677918+0000","last_became_peered":"2022-05-13T23:14:15.677918+0000","last_unstale":"2022-05-13T23:15:17.400976+0000","last_undegraded":"2022-05-13T23:15:17.400976+0000","last_fullsized":"2022-05-13T23:15:17.400976+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:14.344206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:14.344206+0000","last_clean_scrub_stamp":"2022-05-13T23:14:14.344206+0000","objects_scrubbed":0,"log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T09:53:37.623728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":574,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":58,"num_read_kb":2,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2022-05-13T23:15:05.411315+0000","last_change":"2022-05-13T23:14:16.273537+0000","last_active":"2022-05-13T23:15:05.411315+0000","last_peered":"2022-05-13T23:15:05.411315+0000","last_clean":"2022-05-13T23:15:05.411315+0000","last_became_active":"2022-05-13T23:14:16.273127+0000","last_became_peered":"2022-05-13T23:14:16.273127+0000","last_unstale":"2022-05-13T23:15:05.411315+0000","last_undegraded":"2022-05-13T23:15:05.411315+0000","last_fullsized":"2022-05-13T23:15:05.411315+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_clean_scrub_stamp":"2022-05-13T23:14:15.659362+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T01:00:18.643721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"83'7","reported_seq":20,"reported_epoch":84,"state":"active+clean","last_fresh":"2022-05-13T23:15:05.411760+0000","last_change":"2022-05-13T23:14:15.680562+0000","last_active":"2022-05-13T23:15:05.411760+0000","last_peered":"2022-05-13T23:15:05.411760+0000","last_clean":"2022-05-13T23:15:05.411760+0000","last_became_active":"2022-05-13T23:14:15.680163+0000","last_became_peered":"2022-05-13T23:14:15.680163+0000","last_unstale":"2022-05-13T23:15:05.411760+0000","last_undegraded":"2022-05-13T23:15:05.411760+0000","last_fullsized":"2022-05-13T23:15:05.411760+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:14.344206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:14.344206+0000","last_clean_scrub_stamp":"2022-05-13T23:14:14.344206+0000","objects_scrubbed":0,"log_size":7,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T02:39:27.895407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2022-05-13T23:15:05.406200+0000","last_change":"2022-05-13T23:14:16.270878+0000","last_active":"2022-05-13T23:15:05.406200+0000","last_peered":"2022-05-13T23:15:05.406200+0000","last_clean":"2022-05-13T23:15:05.406200+0000","last_became_active":"2022-05-13T23:14:16.270416+0000","last_became_peered":"2022-05-13T23:14:16.270416+0000","last_unstale":"2022-05-13T23:15:05.406200+0000","last_undegraded":"2022-05-13T23:15:05.406200+0000","last_fullsized":"2022-05-13T23:15:05.406200+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_clean_scrub_stamp":"2022-05-13T23:14:15.659362+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T06:11:33.932416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"83'3","reported_seq":16,"reported_epoch":84,"state":"active+clean","last_fresh":"2022-05-13T23:15:05.406038+0000","last_change":"2022-05-13T23:14:15.676518+0000","last_active":"2022-05-13T23:15:05.406038+0000","last_peered":"2022-05-13T23:15:05.406038+0000","last_clean":"2022-05-13T23:15:05.406038+0000","last_became_active":"2022-05-13T23:14:15.676157+0000","last_became_peered":"2022-05-13T23:14:15.676157+0000","last_unstale":"2022-05-13T23:15:05.406038+0000","last_undegraded":"2022-05-13T23:15:05.406038+0000","last_fullsized":"2022-05-13T23:15:05.406038+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:14.344206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:14.344206+0000","last_clean_scrub_stamp":"2022-05-13T23:14:14.344206+0000","objects_scrubbed":0,"log_size":3,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T10:16:49.469347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2022-05-13T23:15:05.416449+0000","last_change":"2022-05-13T23:14:16.257424+0000","last_active":"2022-05-13T23:15:05.416449+0000","last_peered":"2022-05-13T23:15:05.416449+0000","last_clean":"2022-05-13T23:15:05.416449+0000","last_became_active":"2022-05-13T23:14:16.254550+0000","last_became_peered":"2022-05-13T23:14:16.254550+0000","last_unstale":"2022-05-13T23:15:05.416449+0000","last_undegraded":"2022-05-13T23:15:05.416449+0000","last_fullsized":"2022-05-13T23:15:05.416449+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_clean_scrub_stamp":"2022-05-13T23:14:15.659362+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T00:04:50.150368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"83'3","reported_seq":17,"reported_epoch":84,"state":"active+clean","last_fresh":"2022-05-13T23:15:05.405880+0000","last_change":"2022-05-13T23:14:15.680378+0000","last_active":"2022-05-13T23:15:05.405880+0000","last_peered":"2022-05-13T23:15:05.405880+0000","last_clean":"2022-05-13T23:15:05.405880+0000","last_became_active":"2022-05-13T23:14:15.680063+0000","last_became_peered":"2022-05-13T23:14:15.680063+0000","last_unstale":"2022-05-13T23:15:05.405880+0000","last_undegraded":"2022-05-13T23:15:05.405880+0000","last_fullsized":"2022-05-13T23:15:05.405880+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:14.344206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:14.344206+0000","last_clean_scrub_stamp":"2022-05-13T23:14:14.344206+0000","objects_scrubbed":0,"log_size":3,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-14T23:52:48.817491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":554,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2022-05-13T23:15:05.411502+0000","last_change":"2022-05-13T23:14:16.277131+0000","last_active":"2022-05-13T23:15:05.411502+0000","last_peered":"2022-05-13T23:15:05.411502+0000","last_clean":"2022-05-13T23:15:05.411502+0000","last_became_active":"2022-05-13T23:14:16.276815+0000","last_became_peered":"2022-05-13T23:14:16.276815+0000","last_unstale":"2022-05-13T23:15:05.411502+0000","last_undegraded":"2022-05-13T23:15:05.411502+0000","last_fullsized":"2022-05-13T23:15:05.411502+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_clean_scrub_stamp":"2022-05-13T23:14:15.659362+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-14T23:22:29.938073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"83'4","reported_seq":129,"reported_epoch":85,"state":"active+clean","last_fresh":"2022-05-13T23:15:18.397748+0000","last_change":"2022-05-13T23:14:15.683981+0000","last_active":"2022-05-13T23:15:18.397748+0000","last_peered":"2022-05-13T23:15:18.397748+0000","last_clean":"2022-05-13T23:15:18.397748+0000","last_became_active":"2022-05-13T23:14:15.683560+0000","last_became_peered":"2022-05-13T23:14:15.683560+0000","last_unstale":"2022-05-13T23:15:18.397748+0000","last_undegraded":"2022-05-13T23:15:18.397748+0000","last_fullsized":"2022-05-13T23:15:18.397748+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:14.344206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:14.344206+0000","last_clean_scrub_stamp":"2022-05-13T23:14:14.344206+0000","objects_scrubbed":0,"log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T00:30:01.510592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":112,"num_read_kb":112,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2022-05-13T23:15:05.405763+0000","last_change":"2022-05-13T23:14:16.269244+0000","last_active":"2022-05-13T23:15:05.405763+0000","last_peered":"2022-05-13T23:15:05.405763+0000","last_clean":"2022-05-13T23:15:05.405763+0000","last_became_active":"2022-05-13T23:14:16.268949+0000","last_became_peered":"2022-05-13T23:14:16.268949+0000","last_unstale":"2022-05-13T23:15:05.405763+0000","last_undegraded":"2022-05-13T23:15:05.405763+0000","last_fullsized":"2022-05-13T23:15:05.405763+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_clean_scrub_stamp":"2022-05-13T23:14:15.659362+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T04:34:53.311191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"83'4","reported_seq":17,"reported_epoch":84,"state":"active+clean","last_fresh":"2022-05-13T23:15:05.408697+0000","last_change":"2022-05-13T23:14:15.683758+0000","last_active":"2022-05-13T23:15:05.408697+0000","last_peered":"2022-05-13T23:15:05.408697+0000","last_clean":"2022-05-13T23:15:05.408697+0000","last_became_active":"2022-05-13T23:14:15.683356+0000","last_became_peered":"2022-05-13T23:14:15.683356+0000","last_unstale":"2022-05-13T23:15:05.408697+0000","last_undegraded":"2022-05-13T23:15:05.408697+0000","last_fullsized":"2022-05-13T23:15:05.408697+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:14.344206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:14.344206+0000","last_clean_scrub_stamp":"2022-05-13T23:14:14.344206+0000","objects_scrubbed":0,"log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T08:00:55.113935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2022-05-13T23:15:05.515128+0000","last_change":"2022-05-13T23:14:16.268751+0000","last_active":"2022-05-13T23:15:05.515128+0000","last_peered":"2022-05-13T23:15:05.515128+0000","last_clean":"2022-05-13T23:15:05.515128+0000","last_became_active":"2022-05-13T23:14:16.268332+0000","last_became_peered":"2022-05-13T23:14:16.268332+0000","last_unstale":"2022-05-13T23:15:05.515128+0000","last_undegraded":"2022-05-13T23:15:05.515128+0000","last_fullsized":"2022-05-13T23:15:05.515128+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_clean_scrub_stamp":"2022-05-13T23:14:15.659362+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T06:22:45.140343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"83'5","reported_seq":19,"reported_epoch":84,"state":"active+clean","last_fresh":"2022-05-13T23:15:05.406410+0000","last_change":"2022-05-13T23:14:15.701120+0000","last_active":"2022-05-13T23:15:05.406410+0000","last_peered":"2022-05-13T23:15:05.406410+0000","last_clean":"2022-05-13T23:15:05.406410+0000","last_became_active":"2022-05-13T23:14:15.700812+0000","last_became_peered":"2022-05-13T23:14:15.700812+0000","last_unstale":"2022-05-13T23:15:05.406410+0000","last_undegraded":"2022-05-13T23:15:05.406410+0000","last_fullsized":"2022-05-13T23:15:05.406410+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:14.344206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:14.344206+0000","last_clean_scrub_stamp":"2022-05-13T23:14:14.344206+0000","objects_scrubbed":0,"log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T02:16:02.909062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":644,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2022-05-13T23:15:05.405572+0000","last_change":"2022-05-13T23:14:16.265167+0000","last_active":"2022-05-13T23:15:05.405572+0000","last_peered":"2022-05-13T23:15:05.405572+0000","last_clean":"2022-05-13T23:15:05.405572+0000","last_became_active":"2022-05-13T23:14:16.264886+0000","last_became_peered":"2022-05-13T23:14:16.264886+0000","last_unstale":"2022-05-13T23:15:05.405572+0000","last_undegraded":"2022-05-13T23:15:05.405572+0000","last_fullsized":"2022-05-13T23:15:05.405572+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_clean_scrub_stamp":"2022-05-13T23:14:15.659362+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T00:26:06.650867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"83'10","reported_seq":137,"reported_epoch":85,"state":"active+clean","last_fresh":"2022-05-13T23:15:18.390987+0000","last_change":"2022-05-13T23:14:15.682385+0000","last_active":"2022-05-13T23:15:18.390987+0000","last_peered":"2022-05-13T23:15:18.390987+0000","last_clean":"2022-05-13T23:15:18.390987+0000","last_became_active":"2022-05-13T23:14:15.681996+0000","last_became_peered":"2022-05-13T23:14:15.681996+0000","last_unstale":"2022-05-13T23:15:18.390987+0000","last_undegraded":"2022-05-13T23:15:18.390987+0000","last_fullsized":"2022-05-13T23:15:18.390987+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:14.344206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:14.344206+0000","last_clean_scrub_stamp":"2022-05-13T23:14:14.344206+0000","objects_scrubbed":0,"log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-14T23:44:23.766036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1536,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":114,"num_read_kb":58,"num_write":12,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2022-05-13T23:15:05.411635+0000","last_change":"2022-05-13T23:14:16.263744+0000","last_active":"2022-05-13T23:15:05.411635+0000","last_peered":"2022-05-13T23:15:05.411635+0000","last_clean":"2022-05-13T23:15:05.411635+0000","last_became_active":"2022-05-13T23:14:16.263525+0000","last_became_peered":"2022-05-13T23:14:16.263525+0000","last_unstale":"2022-05-13T23:15:05.411635+0000","last_undegraded":"2022-05-13T23:15:05.411635+0000","last_fullsized":"2022-05-13T23:15:05.411635+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_clean_scrub_stamp":"2022-05-13T23:14:15.659362+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T04:23:44.856055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"83'9","reported_seq":24,"reported_epoch":85,"state":"active+clean","last_fresh":"2022-05-13T23:15:06.425091+0000","last_change":"2022-05-13T23:15:06.425091+0000","last_active":"2022-05-13T23:15:06.425091+0000","last_peered":"2022-05-13T23:15:06.425091+0000","last_clean":"2022-05-13T23:15:06.425091+0000","last_became_active":"2022-05-13T23:15:06.414405+0000","last_became_peered":"2022-05-13T23:15:06.414405+0000","last_unstale":"2022-05-13T23:15:06.425091+0000","last_undegraded":"2022-05-13T23:15:06.425091+0000","last_fullsized":"2022-05-13T23:15:06.425091+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:14.344206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:14.344206+0000","last_clean_scrub_stamp":"2022-05-13T23:14:14.344206+0000","objects_scrubbed":0,"log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T06:02:59.163940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":68,"num_keys_recovered":10,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,10,3],"acting":[11,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2022-05-13T23:15:05.406327+0000","last_change":"2022-05-13T23:14:16.263943+0000","last_active":"2022-05-13T23:15:05.406327+0000","last_peered":"2022-05-13T23:15:05.406327+0000","last_clean":"2022-05-13T23:15:05.406327+0000","last_became_active":"2022-05-13T23:14:16.263595+0000","last_became_peered":"2022-05-13T23:14:16.263595+0000","last_unstale":"2022-05-13T23:15:05.406327+0000","last_undegraded":"2022-05-13T23:15:05.406327+0000","last_fullsized":"2022-05-13T23:15:05.406327+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_clean_scrub_stamp":"2022-05-13T23:14:15.659362+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T04:57:04.201317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"83'5","reported_seq":19,"reported_epoch":84,"state":"active+clean","last_fresh":"2022-05-13T23:15:05.405936+0000","last_change":"2022-05-13T23:14:15.683005+0000","last_active":"2022-05-13T23:15:05.405936+0000","last_peered":"2022-05-13T23:15:05.405936+0000","last_clean":"2022-05-13T23:15:05.405936+0000","last_became_active":"2022-05-13T23:14:15.682692+0000","last_became_peered":"2022-05-13T23:14:15.682692+0000","last_unstale":"2022-05-13T23:15:05.405936+0000","last_undegraded":"2022-05-13T23:15:05.405936+0000","last_fullsized":"2022-05-13T23:15:05.405936+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:14.344206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:14.344206+0000","last_clean_scrub_stamp":"2022-05-13T23:14:14.344206+0000","objects_scrubbed":0,"log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T00:27:25.201141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2022-05-13T23:15:05.408693+0000","last_change":"2022-05-13T23:14:16.264351+0000","last_active":"2022-05-13T23:15:05.408693+0000","last_peered":"2022-05-13T23:15:05.408693+0000","last_clean":"2022-05-13T23:15:05.408693+0000","last_became_active":"2022-05-13T23:14:16.264023+0000","last_became_peered":"2022-05-13T23:14:16.264023+0000","last_unstale":"2022-05-13T23:15:05.408693+0000","last_undegraded":"2022-05-13T23:15:05.408693+0000","last_fullsized":"2022-05-13T23:15:05.408693+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_clean_scrub_stamp":"2022-05-13T23:14:15.659362+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T06:13:51.169049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"83'5","reported_seq":76,"reported_epoch":85,"state":"active+clean","last_fresh":"2022-05-13T23:15:17.396279+0000","last_change":"2022-05-13T23:14:15.677080+0000","last_active":"2022-05-13T23:15:17.396279+0000","last_peered":"2022-05-13T23:15:17.396279+0000","last_clean":"2022-05-13T23:15:17.396279+0000","last_became_active":"2022-05-13T23:14:15.676743+0000","last_became_peered":"2022-05-13T23:14:15.676743+0000","last_unstale":"2022-05-13T23:15:17.396279+0000","last_undegraded":"2022-05-13T23:15:17.396279+0000","last_fullsized":"2022-05-13T23:15:17.396279+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:14.344206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:14.344206+0000","last_clean_scrub_stamp":"2022-05-13T23:14:14.344206+0000","objects_scrubbed":0,"log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T09:05:30.321765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":540,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":59,"num_read_kb":3,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":22,"reported_epoch":85,"state":"active+clean","last_fresh":"2022-05-13T23:15:06.541517+0000","last_change":"2022-05-13T23:15:06.541517+0000","last_active":"2022-05-13T23:15:06.541517+0000","last_peered":"2022-05-13T23:15:06.541517+0000","last_clean":"2022-05-13T23:15:06.541517+0000","last_became_active":"2022-05-13T23:15:06.541056+0000","last_became_peered":"2022-05-13T23:15:06.541056+0000","last_unstale":"2022-05-13T23:15:06.541517+0000","last_undegraded":"2022-05-13T23:15:06.541517+0000","last_fullsized":"2022-05-13T23:15:06.541517+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_clean_scrub_stamp":"2022-05-13T23:14:15.659362+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T00:36:00.479693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,10],"acting":[7,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"83'17","reported_seq":31,"reported_epoch":84,"state":"active+clean","last_fresh":"2022-05-13T23:15:05.411453+0000","last_change":"2022-05-13T23:14:15.698465+0000","last_active":"2022-05-13T23:15:05.411453+0000","last_peered":"2022-05-13T23:15:05.411453+0000","last_clean":"2022-05-13T23:15:05.411453+0000","last_became_active":"2022-05-13T23:14:15.698131+0000","last_became_peered":"2022-05-13T23:14:15.698131+0000","last_unstale":"2022-05-13T23:15:05.411453+0000","last_undegraded":"2022-05-13T23:15:05.411453+0000","last_fullsized":"2022-05-13T23:15:05.411453+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:14.344206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:14.344206+0000","last_clean_scrub_stamp":"2022-05-13T23:14:14.344206+0000","objects_scrubbed":0,"log_size":17,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T03:03:48.104345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2022-05-13T23:15:05.405840+0000","last_change":"2022-05-13T23:14:16.260579+0000","last_active":"2022-05-13T23:15:05.405840+0000","last_peered":"2022-05-13T23:15:05.405840+0000","last_clean":"2022-05-13T23:15:05.405840+0000","last_became_active":"2022-05-13T23:14:16.260326+0000","last_became_peered":"2022-05-13T23:14:16.260326+0000","last_unstale":"2022-05-13T23:15:05.405840+0000","last_undegraded":"2022-05-13T23:15:05.405840+0000","last_fullsized":"2022-05-13T23:15:05.405840+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_clean_scrub_stamp":"2022-05-13T23:14:15.659362+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T06:53:23.092756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"83'2","reported_seq":16,"reported_epoch":85,"state":"active+clean","last_fresh":"2022-05-13T23:15:06.770067+0000","last_change":"2022-05-13T23:15:06.770067+0000","last_active":"2022-05-13T23:15:06.770067+0000","last_peered":"2022-05-13T23:15:06.770067+0000","last_clean":"2022-05-13T23:15:06.770067+0000","last_became_active":"2022-05-13T23:15:06.414616+0000","last_became_peered":"2022-05-13T23:15:06.414616+0000","last_unstale":"2022-05-13T23:15:06.770067+0000","last_undegraded":"2022-05-13T23:15:06.770067+0000","last_fullsized":"2022-05-13T23:15:06.770067+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:14.344206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:14.344206+0000","last_clean_scrub_stamp":"2022-05-13T23:14:14.344206+0000","objects_scrubbed":0,"log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T10:19:50.958083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":22,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2022-05-13T23:15:05.410109+0000","last_change":"2022-05-13T23:14:16.260574+0000","last_active":"2022-05-13T23:15:05.410109+0000","last_peered":"2022-05-13T23:15:05.410109+0000","last_clean":"2022-05-13T23:15:05.410109+0000","last_became_active":"2022-05-13T23:14:16.260258+0000","last_became_peered":"2022-05-13T23:14:16.260258+0000","last_unstale":"2022-05-13T23:15:05.410109+0000","last_undegraded":"2022-05-13T23:15:05.410109+0000","last_fullsized":"2022-05-13T23:15:05.410109+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_clean_scrub_stamp":"2022-05-13T23:14:15.659362+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T10:49:06.210598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"83'1","reported_seq":14,"reported_epoch":84,"state":"active+clean","last_fresh":"2022-05-13T23:15:05.411700+0000","last_change":"2022-05-13T23:14:15.701322+0000","last_active":"2022-05-13T23:15:05.411700+0000","last_peered":"2022-05-13T23:15:05.411700+0000","last_clean":"2022-05-13T23:15:05.411700+0000","last_became_active":"2022-05-13T23:14:15.701007+0000","last_became_peered":"2022-05-13T23:14:15.701007+0000","last_unstale":"2022-05-13T23:15:05.411700+0000","last_undegraded":"2022-05-13T23:15:05.411700+0000","last_fullsized":"2022-05-13T23:15:05.411700+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:14.344206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:14.344206+0000","last_clean_scrub_stamp":"2022-05-13T23:14:14.344206+0000","objects_scrubbed":0,"log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T00:13:11.071695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2022-05-13T23:15:05.412544+0000","last_change":"2022-05-13T23:14:16.249579+0000","last_active":"2022-05-13T23:15:05.412544+0000","last_peered":"2022-05-13T23:15:05.412544+0000","last_clean":"2022-05-13T23:15:05.412544+0000","last_became_active":"2022-05-13T23:14:16.249245+0000","last_became_peered":"2022-05-13T23:14:16.249245+0000","last_unstale":"2022-05-13T23:15:05.412544+0000","last_undegraded":"2022-05-13T23:15:05.412544+0000","last_fullsized":"2022-05-13T23:15:05.412544+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_clean_scrub_stamp":"2022-05-13T23:14:15.659362+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T00:26:53.404909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"83'5","reported_seq":76,"reported_epoch":85,"state":"active+clean","last_fresh":"2022-05-13T23:15:18.401537+0000","last_change":"2022-05-13T23:14:15.699433+0000","last_active":"2022-05-13T23:15:18.401537+0000","last_peered":"2022-05-13T23:15:18.401537+0000","last_clean":"2022-05-13T23:15:18.401537+0000","last_became_active":"2022-05-13T23:14:15.698920+0000","last_became_peered":"2022-05-13T23:14:15.698920+0000","last_unstale":"2022-05-13T23:15:18.401537+0000","last_undegraded":"2022-05-13T23:15:18.401537+0000","last_fullsized":"2022-05-13T23:15:18.401537+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:14.344206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:14.344206+0000","last_clean_scrub_stamp":"2022-05-13T23:14:14.344206+0000","objects_scrubbed":0,"log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T07:56:34.276565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":540,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":58,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2022-05-13T23:15:05.412760+0000","last_change":"2022-05-13T23:14:16.259866+0000","last_active":"2022-05-13T23:15:05.412760+0000","last_peered":"2022-05-13T23:15:05.412760+0000","last_clean":"2022-05-13T23:15:05.412760+0000","last_became_active":"2022-05-13T23:14:16.259531+0000","last_became_peered":"2022-05-13T23:14:16.259531+0000","last_unstale":"2022-05-13T23:15:05.412760+0000","last_undegraded":"2022-05-13T23:15:05.412760+0000","last_fullsized":"2022-05-13T23:15:05.412760+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_clean_scrub_stamp":"2022-05-13T23:14:15.659362+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T07:41:46.019217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2022-05-13T23:15:05.405937+0000","last_change":"2022-05-13T23:14:16.259613+0000","last_active":"2022-05-13T23:15:05.405937+0000","last_peered":"2022-05-13T23:15:05.405937+0000","last_clean":"2022-05-13T23:15:05.405937+0000","last_became_active":"2022-05-13T23:14:16.259274+0000","last_became_peered":"2022-05-13T23:14:16.259274+0000","last_unstale":"2022-05-13T23:15:05.405937+0000","last_undegraded":"2022-05-13T23:15:05.405937+0000","last_fullsized":"2022-05-13T23:15:05.405937+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_clean_scrub_stamp":"2022-05-13T23:14:15.659362+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T05:36:21.174814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"83'5","reported_seq":77,"reported_epoch":85,"state":"active+clean","last_fresh":"2022-05-13T23:15:19.398854+0000","last_change":"2022-05-13T23:14:15.701315+0000","last_active":"2022-05-13T23:15:19.398854+0000","last_peered":"2022-05-13T23:15:19.398854+0000","last_clean":"2022-05-13T23:15:19.398854+0000","last_became_active":"2022-05-13T23:14:15.700955+0000","last_became_peered":"2022-05-13T23:14:15.700955+0000","last_unstale":"2022-05-13T23:15:19.398854+0000","last_undegraded":"2022-05-13T23:15:19.398854+0000","last_fullsized":"2022-05-13T23:15:19.398854+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:14.344206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:14.344206+0000","last_clean_scrub_stamp":"2022-05-13T23:14:14.344206+0000","objects_scrubbed":0,"log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T10:17:23.916475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":59,"num_read_kb":59,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2022-05-13T23:15:05.416788+0000","last_change":"2022-05-13T23:14:16.272364+0000","last_active":"2022-05-13T23:15:05.416788+0000","last_peered":"2022-05-13T23:15:05.416788+0000","last_clean":"2022-05-13T23:15:05.416788+0000","last_became_active":"2022-05-13T23:14:16.271612+0000","last_became_peered":"2022-05-13T23:14:16.271612+0000","last_unstale":"2022-05-13T23:15:05.416788+0000","last_undegraded":"2022-05-13T23:15:05.416788+0000","last_fullsized":"2022-05-13T23:15:05.416788+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_clean_scrub_stamp":"2022-05-13T23:14:15.659362+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T07:25:47.039264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"83'6","reported_seq":22,"reported_epoch":84,"state":"active+clean","last_fresh":"2022-05-13T23:15:05.411756+0000","last_change":"2022-05-13T23:14:15.676251+0000","last_active":"2022-05-13T23:15:05.411756+0000","last_peered":"2022-05-13T23:15:05.411756+0000","last_clean":"2022-05-13T23:15:05.411756+0000","last_became_active":"2022-05-13T23:14:15.675895+0000","last_became_peered":"2022-05-13T23:14:15.675895+0000","last_unstale":"2022-05-13T23:15:05.411756+0000","last_undegraded":"2022-05-13T23:15:05.411756+0000","last_fullsized":"2022-05-13T23:15:05.411756+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:14.344206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:14.344206+0000","last_clean_scrub_stamp":"2022-05-13T23:14:14.344206+0000","objects_scrubbed":0,"log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T06:03:57.398647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":345,"reported_epoch":84,"state":"active+clean","last_fresh":"2022-05-13T23:15:05.409382+0000","last_change":"2022-05-13T23:13:02.805009+0000","last_active":"2022-05-13T23:15:05.409382+0000","last_peered":"2022-05-13T23:15:05.409382+0000","last_clean":"2022-05-13T23:15:05.409382+0000","last_became_active":"2022-05-13T23:13:02.804350+0000","last_became_peered":"2022-05-13T23:13:02.804350+0000","last_unstale":"2022-05-13T23:15:05.409382+0000","last_undegraded":"2022-05-13T23:15:05.409382+0000","last_fullsized":"2022-05-13T23:15:05.409382+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:09:32.001256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:09:32.001256+0000","last_clean_scrub_stamp":"2022-05-13T23:09:32.001256+0000","objects_scrubbed":0,"log_size":137,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T07:28:44.497144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1057856,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2022-05-13T23:15:05.405887+0000","last_change":"2022-05-13T23:14:16.261286+0000","last_active":"2022-05-13T23:15:05.405887+0000","last_peered":"2022-05-13T23:15:05.405887+0000","last_clean":"2022-05-13T23:15:05.405887+0000","last_became_active":"2022-05-13T23:14:16.260870+0000","last_became_peered":"2022-05-13T23:14:16.260870+0000","last_unstale":"2022-05-13T23:15:05.405887+0000","last_undegraded":"2022-05-13T23:15:05.405887+0000","last_fullsized":"2022-05-13T23:15:05.405887+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_clean_scrub_stamp":"2022-05-13T23:14:15.659362+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T08:14:34.804797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"83'4","reported_seq":17,"reported_epoch":84,"state":"active+clean","last_fresh":"2022-05-13T23:15:05.409557+0000","last_change":"2022-05-13T23:14:15.686878+0000","last_active":"2022-05-13T23:15:05.409557+0000","last_peered":"2022-05-13T23:15:05.409557+0000","last_clean":"2022-05-13T23:15:05.409557+0000","last_became_active":"2022-05-13T23:14:15.686496+0000","last_became_peered":"2022-05-13T23:14:15.686496+0000","last_unstale":"2022-05-13T23:15:05.409557+0000","last_undegraded":"2022-05-13T23:15:05.409557+0000","last_fullsized":"2022-05-13T23:15:05.409557+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:14.344206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:14.344206+0000","last_clean_scrub_stamp":"2022-05-13T23:14:14.344206+0000","objects_scrubbed":0,"log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T09:19:28.089159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"83'4","reported_seq":18,"reported_epoch":84,"state":"active+clean","last_fresh":"2022-05-13T23:15:05.412102+0000","last_change":"2022-05-13T23:14:15.698503+0000","last_active":"2022-05-13T23:15:05.412102+0000","last_peered":"2022-05-13T23:15:05.412102+0000","last_clean":"2022-05-13T23:15:05.412102+0000","last_became_active":"2022-05-13T23:14:15.698156+0000","last_became_peered":"2022-05-13T23:14:15.698156+0000","last_unstale":"2022-05-13T23:15:05.412102+0000","last_undegraded":"2022-05-13T23:15:05.412102+0000","last_fullsized":"2022-05-13T23:15:05.412102+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:14.344206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:14.344206+0000","last_clean_scrub_stamp":"2022-05-13T23:14:14.344206+0000","objects_scrubbed":0,"log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T09:03:21.133631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2022-05-13T23:15:05.406566+0000","last_change":"2022-05-13T23:14:16.270922+0000","last_active":"2022-05-13T23:15:05.406566+0000","last_peered":"2022-05-13T23:15:05.406566+0000","last_clean":"2022-05-13T23:15:05.406566+0000","last_became_active":"2022-05-13T23:14:16.270259+0000","last_became_peered":"2022-05-13T23:14:16.270259+0000","last_unstale":"2022-05-13T23:15:05.406566+0000","last_undegraded":"2022-05-13T23:15:05.406566+0000","last_fullsized":"2022-05-13T23:15:05.406566+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_clean_scrub_stamp":"2022-05-13T23:14:15.659362+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T02:48:22.255489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"83'7","reported_seq":20,"reported_epoch":84,"state":"active+clean","last_fresh":"2022-05-13T23:15:05.411879+0000","last_change":"2022-05-13T23:14:15.696304+0000","last_active":"2022-05-13T23:15:05.411879+0000","last_peered":"2022-05-13T23:15:05.411879+0000","last_clean":"2022-05-13T23:15:05.411879+0000","last_became_active":"2022-05-13T23:14:15.695476+0000","last_became_peered":"2022-05-13T23:14:15.695476+0000","last_unstale":"2022-05-13T23:15:05.411879+0000","last_undegraded":"2022-05-13T23:15:05.411879+0000","last_fullsized":"2022-05-13T23:15:05.411879+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:14.344206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:14.344206+0000","last_clean_scrub_stamp":"2022-05-13T23:14:14.344206+0000","objects_scrubbed":0,"log_size":7,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T08:48:32.931789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2022-05-13T23:15:05.417489+0000","last_change":"2022-05-13T23:14:16.257063+0000","last_active":"2022-05-13T23:15:05.417489+0000","last_peered":"2022-05-13T23:15:05.417489+0000","last_clean":"2022-05-13T23:15:05.417489+0000","last_became_active":"2022-05-13T23:14:16.256757+0000","last_became_peered":"2022-05-13T23:14:16.256757+0000","last_unstale":"2022-05-13T23:15:05.417489+0000","last_undegraded":"2022-05-13T23:15:05.417489+0000","last_fullsized":"2022-05-13T23:15:05.417489+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_clean_scrub_stamp":"2022-05-13T23:14:15.659362+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T09:24:32.369318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"83'7","reported_seq":21,"reported_epoch":84,"state":"active+clean","last_fresh":"2022-05-13T23:15:05.411692+0000","last_change":"2022-05-13T23:14:15.685568+0000","last_active":"2022-05-13T23:15:05.411692+0000","last_peered":"2022-05-13T23:15:05.411692+0000","last_clean":"2022-05-13T23:15:05.411692+0000","last_became_active":"2022-05-13T23:14:15.685203+0000","last_became_peered":"2022-05-13T23:14:15.685203+0000","last_unstale":"2022-05-13T23:15:05.411692+0000","last_undegraded":"2022-05-13T23:15:05.411692+0000","last_fullsized":"2022-05-13T23:15:05.411692+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:14.344206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:14.344206+0000","last_clean_scrub_stamp":"2022-05-13T23:14:14.344206+0000","objects_scrubbed":0,"log_size":7,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T03:55:33.271069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2022-05-13T23:15:05.514919+0000","last_change":"2022-05-13T23:14:16.280330+0000","last_active":"2022-05-13T23:15:05.514919+0000","last_peered":"2022-05-13T23:15:05.514919+0000","last_clean":"2022-05-13T23:15:05.514919+0000","last_became_active":"2022-05-13T23:14:16.279908+0000","last_became_peered":"2022-05-13T23:14:16.279908+0000","last_unstale":"2022-05-13T23:15:05.514919+0000","last_undegraded":"2022-05-13T23:15:05.514919+0000","last_fullsized":"2022-05-13T23:15:05.514919+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_clean_scrub_stamp":"2022-05-13T23:14:15.659362+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T03:25:42.803064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"83'2","reported_seq":15,"reported_epoch":84,"state":"active+clean","last_fresh":"2022-05-13T23:15:05.405262+0000","last_change":"2022-05-13T23:14:15.698443+0000","last_active":"2022-05-13T23:15:05.405262+0000","last_peered":"2022-05-13T23:15:05.405262+0000","last_clean":"2022-05-13T23:15:05.405262+0000","last_became_active":"2022-05-13T23:14:15.698158+0000","last_became_peered":"2022-05-13T23:14:15.698158+0000","last_unstale":"2022-05-13T23:15:05.405262+0000","last_undegraded":"2022-05-13T23:15:05.405262+0000","last_fullsized":"2022-05-13T23:15:05.405262+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:14.344206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:14.344206+0000","last_clean_scrub_stamp":"2022-05-13T23:14:14.344206+0000","objects_scrubbed":0,"log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T05:06:14.946233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2022-05-13T23:15:05.411226+0000","last_change":"2022-05-13T23:14:16.264110+0000","last_active":"2022-05-13T23:15:05.411226+0000","last_peered":"2022-05-13T23:15:05.411226+0000","last_clean":"2022-05-13T23:15:05.411226+0000","last_became_active":"2022-05-13T23:14:16.263830+0000","last_became_peered":"2022-05-13T23:14:16.263830+0000","last_unstale":"2022-05-13T23:15:05.411226+0000","last_undegraded":"2022-05-13T23:15:05.411226+0000","last_fullsized":"2022-05-13T23:15:05.411226+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_clean_scrub_stamp":"2022-05-13T23:14:15.659362+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T00:06:26.724597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"83'7","reported_seq":20,"reported_epoch":84,"state":"active+clean","last_fresh":"2022-05-13T23:15:05.416532+0000","last_change":"2022-05-13T23:14:15.693926+0000","last_active":"2022-05-13T23:15:05.416532+0000","last_peered":"2022-05-13T23:15:05.416532+0000","last_clean":"2022-05-13T23:15:05.416532+0000","last_became_active":"2022-05-13T23:14:15.692163+0000","last_became_peered":"2022-05-13T23:14:15.692163+0000","last_unstale":"2022-05-13T23:15:05.416532+0000","last_undegraded":"2022-05-13T23:15:05.416532+0000","last_fullsized":"2022-05-13T23:15:05.416532+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:14.344206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:14.344206+0000","last_clean_scrub_stamp":"2022-05-13T23:14:14.344206+0000","objects_scrubbed":0,"log_size":7,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T03:00:12.522714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2022-05-13T23:15:05.409774+0000","last_change":"2022-05-13T23:14:16.265409+0000","last_active":"2022-05-13T23:15:05.409774+0000","last_peered":"2022-05-13T23:15:05.409774+0000","last_clean":"2022-05-13T23:15:05.409774+0000","last_became_active":"2022-05-13T23:14:16.265151+0000","last_became_peered":"2022-05-13T23:14:16.265151+0000","last_unstale":"2022-05-13T23:15:05.409774+0000","last_undegraded":"2022-05-13T23:15:05.409774+0000","last_fullsized":"2022-05-13T23:15:05.409774+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_clean_scrub_stamp":"2022-05-13T23:14:15.659362+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T04:05:43.762623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"83'3","reported_seq":16,"reported_epoch":84,"state":"active+clean","last_fresh":"2022-05-13T23:15:05.417167+0000","last_change":"2022-05-13T23:14:15.681975+0000","last_active":"2022-05-13T23:15:05.417167+0000","last_peered":"2022-05-13T23:15:05.417167+0000","last_clean":"2022-05-13T23:15:05.417167+0000","last_became_active":"2022-05-13T23:14:15.681391+0000","last_became_peered":"2022-05-13T23:14:15.681391+0000","last_unstale":"2022-05-13T23:15:05.417167+0000","last_undegraded":"2022-05-13T23:15:05.417167+0000","last_fullsized":"2022-05-13T23:15:05.417167+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:14.344206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:14.344206+0000","last_clean_scrub_stamp":"2022-05-13T23:14:14.344206+0000","objects_scrubbed":0,"log_size":3,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T06:56:25.732024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2022-05-13T23:15:05.416655+0000","last_change":"2022-05-13T23:14:16.272443+0000","last_active":"2022-05-13T23:15:05.416655+0000","last_peered":"2022-05-13T23:15:05.416655+0000","last_clean":"2022-05-13T23:15:05.416655+0000","last_became_active":"2022-05-13T23:14:16.272031+0000","last_became_peered":"2022-05-13T23:14:16.272031+0000","last_unstale":"2022-05-13T23:15:05.416655+0000","last_undegraded":"2022-05-13T23:15:05.416655+0000","last_fullsized":"2022-05-13T23:15:05.416655+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_clean_scrub_stamp":"2022-05-13T23:14:15.659362+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T10:23:55.694108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"83'4","reported_seq":18,"reported_epoch":84,"state":"active+clean","last_fresh":"2022-05-13T23:15:05.411301+0000","last_change":"2022-05-13T23:14:15.681877+0000","last_active":"2022-05-13T23:15:05.411301+0000","last_peered":"2022-05-13T23:15:05.411301+0000","last_clean":"2022-05-13T23:15:05.411301+0000","last_became_active":"2022-05-13T23:14:15.681579+0000","last_became_peered":"2022-05-13T23:14:15.681579+0000","last_unstale":"2022-05-13T23:15:05.411301+0000","last_undegraded":"2022-05-13T23:15:05.411301+0000","last_fullsized":"2022-05-13T23:15:05.411301+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:14.344206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:14.344206+0000","last_clean_scrub_stamp":"2022-05-13T23:14:14.344206+0000","objects_scrubbed":0,"log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T07:00:11.132049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2022-05-13T23:15:05.408596+0000","last_change":"2022-05-13T23:14:16.266877+0000","last_active":"2022-05-13T23:15:05.408596+0000","last_peered":"2022-05-13T23:15:05.408596+0000","last_clean":"2022-05-13T23:15:05.408596+0000","last_became_active":"2022-05-13T23:14:16.266551+0000","last_became_peered":"2022-05-13T23:14:16.266551+0000","last_unstale":"2022-05-13T23:15:05.408596+0000","last_undegraded":"2022-05-13T23:15:05.408596+0000","last_fullsized":"2022-05-13T23:15:05.408596+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_clean_scrub_stamp":"2022-05-13T23:14:15.659362+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T05:57:29.394410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"83'5","reported_seq":78,"reported_epoch":85,"state":"active+clean","last_fresh":"2022-05-13T23:15:19.399934+0000","last_change":"2022-05-13T23:14:15.699586+0000","last_active":"2022-05-13T23:15:19.399934+0000","last_peered":"2022-05-13T23:15:19.399934+0000","last_clean":"2022-05-13T23:15:19.399934+0000","last_became_active":"2022-05-13T23:14:15.699215+0000","last_became_peered":"2022-05-13T23:14:15.699215+0000","last_unstale":"2022-05-13T23:15:19.399934+0000","last_undegraded":"2022-05-13T23:15:19.399934+0000","last_fullsized":"2022-05-13T23:15:19.399934+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:14.344206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:14.344206+0000","last_clean_scrub_stamp":"2022-05-13T23:14:14.344206+0000","objects_scrubbed":0,"log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T04:14:54.395550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1094,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":60,"num_read_kb":2,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2022-05-13T23:15:05.408938+0000","last_change":"2022-05-13T23:14:16.265730+0000","last_active":"2022-05-13T23:15:05.408938+0000","last_peered":"2022-05-13T23:15:05.408938+0000","last_clean":"2022-05-13T23:15:05.408938+0000","last_became_active":"2022-05-13T23:14:16.265435+0000","last_became_peered":"2022-05-13T23:14:16.265435+0000","last_unstale":"2022-05-13T23:15:05.408938+0000","last_undegraded":"2022-05-13T23:15:05.408938+0000","last_fullsized":"2022-05-13T23:15:05.408938+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_clean_scrub_stamp":"2022-05-13T23:14:15.659362+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T08:55:57.574132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"83'3","reported_seq":17,"reported_epoch":84,"state":"active+clean","last_fresh":"2022-05-13T23:15:05.406349+0000","last_change":"2022-05-13T23:14:15.682402+0000","last_active":"2022-05-13T23:15:05.406349+0000","last_peered":"2022-05-13T23:15:05.406349+0000","last_clean":"2022-05-13T23:15:05.406349+0000","last_became_active":"2022-05-13T23:14:15.682105+0000","last_became_peered":"2022-05-13T23:14:15.682105+0000","last_unstale":"2022-05-13T23:15:05.406349+0000","last_undegraded":"2022-05-13T23:15:05.406349+0000","last_fullsized":"2022-05-13T23:15:05.406349+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:14.344206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:14.344206+0000","last_clean_scrub_stamp":"2022-05-13T23:14:14.344206+0000","objects_scrubbed":0,"log_size":3,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T10:37:55.092658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2022-05-13T23:15:05.408714+0000","last_change":"2022-05-13T23:14:16.263001+0000","last_active":"2022-05-13T23:15:05.408714+0000","last_peered":"2022-05-13T23:15:05.408714+0000","last_clean":"2022-05-13T23:15:05.408714+0000","last_became_active":"2022-05-13T23:14:16.262646+0000","last_became_peered":"2022-05-13T23:14:16.262646+0000","last_unstale":"2022-05-13T23:15:05.408714+0000","last_undegraded":"2022-05-13T23:15:05.408714+0000","last_fullsized":"2022-05-13T23:15:05.408714+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_clean_scrub_stamp":"2022-05-13T23:14:15.659362+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T05:38:53.024966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"83'5","reported_seq":18,"reported_epoch":84,"state":"active+clean","last_fresh":"2022-05-13T23:15:05.406697+0000","last_change":"2022-05-13T23:14:15.690434+0000","last_active":"2022-05-13T23:15:05.406697+0000","last_peered":"2022-05-13T23:15:05.406697+0000","last_clean":"2022-05-13T23:15:05.406697+0000","last_became_active":"2022-05-13T23:14:15.690032+0000","last_became_peered":"2022-05-13T23:14:15.690032+0000","last_unstale":"2022-05-13T23:15:05.406697+0000","last_undegraded":"2022-05-13T23:15:05.406697+0000","last_fullsized":"2022-05-13T23:15:05.406697+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:14.344206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:14.344206+0000","last_clean_scrub_stamp":"2022-05-13T23:14:14.344206+0000","objects_scrubbed":0,"log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T06:39:08.251311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2022-05-13T23:15:05.409028+0000","last_change":"2022-05-13T23:14:16.264791+0000","last_active":"2022-05-13T23:15:05.409028+0000","last_peered":"2022-05-13T23:15:05.409028+0000","last_clean":"2022-05-13T23:15:05.409028+0000","last_became_active":"2022-05-13T23:14:16.264461+0000","last_became_peered":"2022-05-13T23:14:16.264461+0000","last_unstale":"2022-05-13T23:15:05.409028+0000","last_undegraded":"2022-05-13T23:15:05.409028+0000","last_fullsized":"2022-05-13T23:15:05.409028+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_clean_scrub_stamp":"2022-05-13T23:14:15.659362+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T03:02:04.280220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2022-05-13T23:15:05.409787+0000","last_change":"2022-05-13T23:14:16.260468+0000","last_active":"2022-05-13T23:15:05.409787+0000","last_peered":"2022-05-13T23:15:05.409787+0000","last_clean":"2022-05-13T23:15:05.409787+0000","last_became_active":"2022-05-13T23:14:16.260211+0000","last_became_peered":"2022-05-13T23:14:16.260211+0000","last_unstale":"2022-05-13T23:15:05.409787+0000","last_undegraded":"2022-05-13T23:15:05.409787+0000","last_fullsized":"2022-05-13T23:15:05.409787+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_clean_scrub_stamp":"2022-05-13T23:14:15.659362+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T05:26:57.769099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2022-05-13T23:15:05.417082+0000","last_change":"2022-05-13T23:14:16.253997+0000","last_active":"2022-05-13T23:15:05.417082+0000","last_peered":"2022-05-13T23:15:05.417082+0000","last_clean":"2022-05-13T23:15:05.417082+0000","last_became_active":"2022-05-13T23:14:16.253685+0000","last_became_peered":"2022-05-13T23:14:16.253685+0000","last_unstale":"2022-05-13T23:15:05.417082+0000","last_undegraded":"2022-05-13T23:15:05.417082+0000","last_fullsized":"2022-05-13T23:15:05.417082+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_clean_scrub_stamp":"2022-05-13T23:14:15.659362+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T07:42:26.765244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2022-05-13T23:15:05.417031+0000","last_change":"2022-05-13T23:14:16.265207+0000","last_active":"2022-05-13T23:15:05.417031+0000","last_peered":"2022-05-13T23:15:05.417031+0000","last_clean":"2022-05-13T23:15:05.417031+0000","last_became_active":"2022-05-13T23:14:16.264881+0000","last_became_peered":"2022-05-13T23:14:16.264881+0000","last_unstale":"2022-05-13T23:15:05.417031+0000","last_undegraded":"2022-05-13T23:15:05.417031+0000","last_fullsized":"2022-05-13T23:15:05.417031+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_clean_scrub_stamp":"2022-05-13T23:14:15.659362+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T04:01:48.894233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2022-05-13T23:15:05.514329+0000","last_change":"2022-05-13T23:14:16.264286+0000","last_active":"2022-05-13T23:15:05.514329+0000","last_peered":"2022-05-13T23:15:05.514329+0000","last_clean":"2022-05-13T23:15:05.514329+0000","last_became_active":"2022-05-13T23:14:16.263978+0000","last_became_peered":"2022-05-13T23:14:16.263978+0000","last_unstale":"2022-05-13T23:15:05.514329+0000","last_undegraded":"2022-05-13T23:15:05.514329+0000","last_fullsized":"2022-05-13T23:15:05.514329+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_clean_scrub_stamp":"2022-05-13T23:14:15.659362+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T03:11:10.799827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2022-05-13T23:15:05.406569+0000","last_change":"2022-05-13T23:14:16.265421+0000","last_active":"2022-05-13T23:15:05.406569+0000","last_peered":"2022-05-13T23:15:05.406569+0000","last_clean":"2022-05-13T23:15:05.406569+0000","last_became_active":"2022-05-13T23:14:16.265057+0000","last_became_peered":"2022-05-13T23:14:16.265057+0000","last_unstale":"2022-05-13T23:15:05.406569+0000","last_undegraded":"2022-05-13T23:15:05.406569+0000","last_fullsized":"2022-05-13T23:15:05.406569+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_clean_scrub_stamp":"2022-05-13T23:14:15.659362+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T07:45:53.177341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2022-05-13T23:15:05.410182+0000","last_change":"2022-05-13T23:14:16.266317+0000","last_active":"2022-05-13T23:15:05.410182+0000","last_peered":"2022-05-13T23:15:05.410182+0000","last_clean":"2022-05-13T23:15:05.410182+0000","last_became_active":"2022-05-13T23:14:16.265959+0000","last_became_peered":"2022-05-13T23:14:16.265959+0000","last_unstale":"2022-05-13T23:15:05.410182+0000","last_undegraded":"2022-05-13T23:15:05.410182+0000","last_fullsized":"2022-05-13T23:15:05.410182+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_clean_scrub_stamp":"2022-05-13T23:14:15.659362+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T03:37:42.277033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2022-05-13T23:15:05.417785+0000","last_change":"2022-05-13T23:14:16.254350+0000","last_active":"2022-05-13T23:15:05.417785+0000","last_peered":"2022-05-13T23:15:05.417785+0000","last_clean":"2022-05-13T23:15:05.417785+0000","last_became_active":"2022-05-13T23:14:16.253940+0000","last_became_peered":"2022-05-13T23:14:16.253940+0000","last_unstale":"2022-05-13T23:15:05.417785+0000","last_undegraded":"2022-05-13T23:15:05.417785+0000","last_fullsized":"2022-05-13T23:15:05.417785+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_clean_scrub_stamp":"2022-05-13T23:14:15.659362+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T10:08:56.052316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2022-05-13T23:15:05.417437+0000","last_change":"2022-05-13T23:14:16.263153+0000","last_active":"2022-05-13T23:15:05.417437+0000","last_peered":"2022-05-13T23:15:05.417437+0000","last_clean":"2022-05-13T23:15:05.417437+0000","last_became_active":"2022-05-13T23:14:16.262652+0000","last_became_peered":"2022-05-13T23:14:16.262652+0000","last_unstale":"2022-05-13T23:15:05.417437+0000","last_undegraded":"2022-05-13T23:15:05.417437+0000","last_fullsized":"2022-05-13T23:15:05.417437+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_clean_scrub_stamp":"2022-05-13T23:14:15.659362+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T07:53:46.499880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2022-05-13T23:15:05.405856+0000","last_change":"2022-05-13T23:14:16.270358+0000","last_active":"2022-05-13T23:15:05.405856+0000","last_peered":"2022-05-13T23:15:05.405856+0000","last_clean":"2022-05-13T23:15:05.405856+0000","last_became_active":"2022-05-13T23:14:16.270099+0000","last_became_peered":"2022-05-13T23:14:16.270099+0000","last_unstale":"2022-05-13T23:15:05.405856+0000","last_undegraded":"2022-05-13T23:15:05.405856+0000","last_fullsized":"2022-05-13T23:15:05.405856+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_clean_scrub_stamp":"2022-05-13T23:14:15.659362+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T00:42:07.782569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2022-05-13T23:15:05.416427+0000","last_change":"2022-05-13T23:14:16.251188+0000","last_active":"2022-05-13T23:15:05.416427+0000","last_peered":"2022-05-13T23:15:05.416427+0000","last_clean":"2022-05-13T23:15:05.416427+0000","last_became_active":"2022-05-13T23:14:16.250816+0000","last_became_peered":"2022-05-13T23:14:16.250816+0000","last_unstale":"2022-05-13T23:15:05.416427+0000","last_undegraded":"2022-05-13T23:15:05.416427+0000","last_fullsized":"2022-05-13T23:15:05.416427+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_clean_scrub_stamp":"2022-05-13T23:14:15.659362+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T00:58:14.898104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2022-05-13T23:15:05.405457+0000","last_change":"2022-05-13T23:14:16.264125+0000","last_active":"2022-05-13T23:15:05.405457+0000","last_peered":"2022-05-13T23:15:05.405457+0000","last_clean":"2022-05-13T23:15:05.405457+0000","last_became_active":"2022-05-13T23:14:16.263806+0000","last_became_peered":"2022-05-13T23:14:16.263806+0000","last_unstale":"2022-05-13T23:15:05.405457+0000","last_undegraded":"2022-05-13T23:15:05.405457+0000","last_fullsized":"2022-05-13T23:15:05.405457+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_clean_scrub_stamp":"2022-05-13T23:14:15.659362+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T09:26:04.559765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2022-05-13T23:15:05.406402+0000","last_change":"2022-05-13T23:14:16.266991+0000","last_active":"2022-05-13T23:15:05.406402+0000","last_peered":"2022-05-13T23:15:05.406402+0000","last_clean":"2022-05-13T23:15:05.406402+0000","last_became_active":"2022-05-13T23:14:16.265551+0000","last_became_peered":"2022-05-13T23:14:16.265551+0000","last_unstale":"2022-05-13T23:15:05.406402+0000","last_undegraded":"2022-05-13T23:15:05.406402+0000","last_fullsized":"2022-05-13T23:15:05.406402+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_clean_scrub_stamp":"2022-05-13T23:14:15.659362+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T00:40:56.664387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":22,"reported_epoch":85,"state":"active+clean","last_fresh":"2022-05-13T23:15:06.541292+0000","last_change":"2022-05-13T23:15:06.541292+0000","last_active":"2022-05-13T23:15:06.541292+0000","last_peered":"2022-05-13T23:15:06.541292+0000","last_clean":"2022-05-13T23:15:06.541292+0000","last_became_active":"2022-05-13T23:15:06.540909+0000","last_became_peered":"2022-05-13T23:15:06.540909+0000","last_unstale":"2022-05-13T23:15:06.541292+0000","last_undegraded":"2022-05-13T23:15:06.541292+0000","last_fullsized":"2022-05-13T23:15:06.541292+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-05-13T23:14:15.659362+0000","last_clean_scrub_stamp":"2022-05-13T23:14:15.659362+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-05-15T09:55:15.142184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,8],"acting":[11,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1057856,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":3194880,"data_stored":3173568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":7626,"num_objects":98,"num_object_clones":0,"num_object_copies":294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98,"num_whiteouts":0,"num_read":595,"num_read_kb":312,"num_write":205,"num_write_kb":57,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":90,"num_keys_recovered":10,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":393216,"data_stored":22878,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":179,"ondisk_log_size":179,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":78,"seq":335007449114,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":5980,"kb_used_data":916,"kb_used_omap":0,"kb_used_meta":5056,"kb_avail":93739172,"statfs":{"total":95995035648,"available":95988912128,"internally_reserved":0,"allocated":937984,"data_stored":442813,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5177344},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri May 13 23:14:21 2022","interfaces":[{"interface":"back","average":{"1min":0.60299999999999998,"5min":0.60299999999999998,"15min":0.60299999999999998},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.84699999999999998,"5min":0.84699999999999998,"15min":0.84699999999999998},"last":0.41599999999999998},{"interface":"front","average":{"1min":0.64200000000000002,"5min":0.64200000000000002,"15min":0.64200000000000002},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":0.81599999999999995,"5min":0.81599999999999995,"15min":0.81599999999999995},"last":0.47999999999999998}]},{"osd":1,"last update":"Fri May 13 23:14:21 2022","interfaces":[{"interface":"back","average":{"1min":0.59699999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":0.81499999999999995,"5min":0.81499999999999995,"15min":0.81499999999999995},"last":0.61399999999999999},{"interface":"front","average":{"1min":0.58899999999999997,"5min":0.58899999999999997,"15min":0.58899999999999997},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.89500000000000002,"5min":0.89500000000000002,"15min":0.89500000000000002},"last":0.59199999999999997}]},{"osd":2,"last update":"Fri May 13 23:14:21 2022","interfaces":[{"interface":"back","average":{"1min":0.46500000000000002,"5min":0.46500000000000002,"15min":0.46500000000000002},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.76900000000000002,"5min":0.76900000000000002,"15min":0.76900000000000002},"last":0.36199999999999999},{"interface":"front","average":{"1min":0.46800000000000003,"5min":0.46800000000000003,"15min":0.46800000000000003},"min":{"1min":0.317,"5min":0.317,"15min":0.317},"max":{"1min":0.71299999999999997,"5min":0.71299999999999997,"15min":0.71299999999999997},"last":0.38800000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53000000000000003}]},{"osd":4,"last update":"Fri May 13 23:14:21 2022","interfaces":[{"interface":"back","average":{"1min":0.622,"5min":0.622,"15min":0.622},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":1.0289999999999999,"5min":1.0289999999999999,"15min":1.0289999999999999},"last":0.499},{"interface":"front","average":{"1min":0.63400000000000001,"5min":0.63400000000000001,"15min":0.63400000000000001},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":0.93700000000000006,"5min":0.93700000000000006,"15min":0.93700000000000006},"last":0.629}]},{"osd":5,"last update":"Fri May 13 23:14:21 2022","interfaces":[{"interface":"back","average":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.59099999999999997},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":1.3799999999999999,"5min":1.3799999999999999,"15min":1.3799999999999999},"last":0.52000000000000002},{"interface":"front","average":{"1min":0.60299999999999998,"5min":0.60299999999999998,"15min":0.60299999999999998},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.88,"5min":0.88,"15min":0.88},"last":0.53900000000000003}]},{"osd":6,"last update":"Fri May 13 23:14:21 2022","interfaces":[{"interface":"back","average":{"1min":0.73999999999999999,"5min":0.73999999999999999,"15min":0.73999999999999999},"min":{"1min":0.48499999999999999,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":1.0669999999999999,"5min":1.0669999999999999,"15min":1.0669999999999999},"last":0.72299999999999998},{"interface":"front","average":{"1min":0.79200000000000004,"5min":0.79200000000000004,"15min":0.79200000000000004},"min":{"1min":0.49099999999999999,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":1.655,"5min":1.655,"15min":1.655},"last":0.70499999999999996}]},{"osd":7,"last update":"Fri May 13 23:14:21 2022","interfaces":[{"interface":"back","average":{"1min":0.72299999999999998,"5min":0.72299999999999998,"15min":0.72299999999999998},"min":{"1min":0.502,"5min":0.502,"15min":0.502},"max":{"1min":1.107,"5min":1.107,"15min":1.107},"last":0.57899999999999996},{"interface":"front","average":{"1min":0.745,"5min":0.745,"15min":0.745},"min":{"1min":0.46999999999999997,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":1.3620000000000001,"5min":1.3620000000000001,"15min":1.3620000000000001},"last":0.66100000000000003}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68200000000000005}]},{"osd":9,"last update":"Fri May 13 23:14:21 2022","interfaces":[{"interface":"back","average":{"1min":0.83699999999999997,"5min":0.83699999999999997,"15min":0.83699999999999997},"min":{"1min":0.57099999999999995,"5min":0.57099999999999995,"15min":0.57099999999999995},"max":{"1min":1.7729999999999999,"5min":1.7729999999999999,"15min":1.7729999999999999},"last":0.67700000000000005},{"interface":"front","average":{"1min":0.79500000000000004,"5min":0.79500000000000004,"15min":0.79500000000000004},"min":{"1min":0.58099999999999996,"5min":0.58099999999999996,"15min":0.58099999999999996},"max":{"1min":1.3220000000000001,"5min":1.3220000000000001,"15min":1.3220000000000001},"last":0.68700000000000006}]},{"osd":10,"last update":"Fri May 13 23:14:21 2022","interfaces":[{"interface":"back","average":{"1min":0.75700000000000001,"5min":0.75700000000000001,"15min":0.75700000000000001},"min":{"1min":0.54100000000000004,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":1.26,"5min":1.26,"15min":1.26},"last":0.69999999999999996},{"interface":"front","average":{"1min":0.77600000000000002,"5min":0.77600000000000002,"15min":0.77600000000000002},"min":{"1min":0.51900000000000002,"5min":0.51900000000000002,"15min":0.51900000000000002},"max":{"1min":1.7350000000000001,"5min":1.7350000000000001,"15min":1.7350000000000001},"last":0.64400000000000002}]}]},{"osd":10,"up_from":71,"seq":304942678047,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6720,"kb_used_data":1464,"kb_used_omap":0,"kb_used_meta":5248,"kb_avail":93738432,"statfs":{"total":95995035648,"available":95988154368,"internally_reserved":0,"allocated":1499136,"data_stored":973408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5373952},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri May 13 23:15:17 2022","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.86899999999999999,"5min":0.86899999999999999,"15min":0.86899999999999999},"last":0.54900000000000004},{"interface":"front","average":{"1min":0.64300000000000002,"5min":0.64300000000000002,"15min":0.64300000000000002},"min":{"1min":0.43099999999999999,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":0.90400000000000003,"5min":0.90400000000000003,"15min":0.90400000000000003},"last":0.66800000000000004}]},{"osd":1,"last update":"Fri May 13 23:15:07 2022","interfaces":[{"interface":"back","average":{"1min":0.45300000000000001,"5min":0.51700000000000002,"15min":0.52800000000000002},"min":{"1min":0.33100000000000002,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.75900000000000001,"5min":0.75900000000000001,"15min":0.75900000000000001},"last":0.59999999999999998},{"interface":"front","average":{"1min":0.45800000000000002,"5min":0.55500000000000005,"15min":0.57099999999999995},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":0.59399999999999997,"5min":0.83099999999999996,"15min":0.83099999999999996},"last":0.57199999999999995}]},{"osd":2,"last update":"Fri May 13 23:15:07 2022","interfaces":[{"interface":"back","average":{"1min":0.61399999999999999,"5min":0.67800000000000005,"15min":0.68899999999999995},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.85399999999999998,"5min":1.3280000000000001,"15min":1.3280000000000001},"last":0.76000000000000001},{"interface":"front","average":{"1min":0.629,"5min":0.69199999999999995,"15min":0.70199999999999996},"min":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":0.80700000000000005,"5min":0.88800000000000001,"15min":0.88800000000000001},"last":0.84799999999999998}]},{"osd":3,"last update":"Fri May 13 23:15:07 2022","interfaces":[{"interface":"back","average":{"1min":0.61399999999999999,"5min":0.71399999999999997,"15min":0.73099999999999998},"min":{"1min":0.443,"5min":0.443,"15min":0.443},"max":{"1min":0.89200000000000002,"5min":0.90100000000000002,"15min":0.90100000000000002},"last":0.74299999999999999},{"interface":"front","average":{"1min":0.624,"5min":0.73999999999999999,"15min":0.76000000000000001},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.76100000000000001,"5min":0.93500000000000005,"15min":0.93500000000000005},"last":0.77500000000000002}]},{"osd":4,"last update":"Fri May 13 23:15:07 2022","interfaces":[{"interface":"back","average":{"1min":0.54500000000000004,"5min":0.65300000000000002,"15min":0.67100000000000004},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":0.71899999999999997,"5min":0.97699999999999998,"15min":0.97699999999999998},"last":0.70699999999999996},{"interface":"front","average":{"1min":0.59899999999999998,"5min":0.64100000000000001,"15min":0.64800000000000002},"min":{"1min":0.33900000000000002,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.82099999999999995,"5min":1.0720000000000001,"15min":1.0720000000000001},"last":0.81999999999999995}]},{"osd":5,"last update":"Fri May 13 23:15:07 2022","interfaces":[{"interface":"back","average":{"1min":0.69199999999999995,"5min":0.76500000000000001,"15min":0.77700000000000002},"min":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":0.90500000000000003,"5min":1.4299999999999999,"15min":1.4299999999999999},"last":0.90700000000000003},{"interface":"front","average":{"1min":0.67500000000000004,"5min":0.77300000000000002,"15min":0.78900000000000003},"min":{"1min":0.497,"5min":0.497,"15min":0.497},"max":{"1min":0.877,"5min":1.0609999999999999,"15min":1.0609999999999999},"last":0.91800000000000004}]},{"osd":6,"last update":"Fri May 13 23:15:07 2022","interfaces":[{"interface":"back","average":{"1min":0.69999999999999996,"5min":0.84899999999999998,"15min":0.874},"min":{"1min":0.44700000000000001,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":0.86399999999999999,"5min":1.157,"15min":1.157},"last":0.95199999999999996},{"interface":"front","average":{"1min":0.71299999999999997,"5min":0.84999999999999998,"15min":0.873},"min":{"1min":0.55600000000000005,"5min":0.55600000000000005,"15min":0.55600000000000005},"max":{"1min":0.879,"5min":1.216,"15min":1.216},"last":0.83599999999999997}]},{"osd":7,"last update":"Fri May 13 23:14:05 2022","interfaces":[{"interface":"back","average":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":1.585,"5min":1.585,"15min":1.585},"last":0.496},{"interface":"front","average":{"1min":0.80300000000000005,"5min":0.80300000000000005,"15min":0.80300000000000005},"min":{"1min":0.47999999999999998,"5min":0.47999999999999998,"15min":0.47999999999999998},"max":{"1min":1.0469999999999999,"5min":1.0469999999999999,"15min":1.0469999999999999},"last":0.502}]},{"osd":8,"last update":"Fri May 13 23:15:07 2022","interfaces":[{"interface":"back","average":{"1min":0.72399999999999998,"5min":0.83599999999999997,"15min":0.85399999999999998},"min":{"1min":0.47699999999999998,"5min":0.47699999999999998,"15min":0.47699999999999998},"max":{"1min":0.92300000000000004,"5min":1.163,"15min":1.163},"last":0.88900000000000001},{"interface":"front","average":{"1min":0.75900000000000001,"5min":0.85699999999999998,"15min":0.873},"min":{"1min":0.52500000000000002,"5min":0.51400000000000001,"15min":0.51400000000000001},"max":{"1min":0.89700000000000002,"5min":1.476,"15min":1.476},"last":0.876}]},{"osd":9,"last update":"Fri May 13 23:15:07 2022","interfaces":[{"interface":"back","average":{"1min":0.81699999999999995,"5min":0.92300000000000004,"15min":0.94099999999999995},"min":{"1min":0.58099999999999996,"5min":0.58099999999999996,"15min":0.58099999999999996},"max":{"1min":1.0029999999999999,"5min":1.369,"15min":1.369},"last":0.97799999999999998},{"interface":"front","average":{"1min":0.82099999999999995,"5min":0.93799999999999994,"15min":0.95799999999999996},"min":{"1min":0.624,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":0.95799999999999996,"5min":1.819,"15min":1.819},"last":0.93200000000000005}]},{"osd":11,"last update":"Fri May 13 23:14:23 2022","interfaces":[{"interface":"back","average":{"1min":0.84299999999999997,"5min":0.84299999999999997,"15min":0.84299999999999997},"min":{"1min":0.63300000000000001,"5min":0.63300000000000001,"15min":0.63300000000000001},"max":{"1min":1.1060000000000001,"5min":1.1060000000000001,"15min":1.1060000000000001},"last":0.98899999999999999},{"interface":"front","average":{"1min":0.84999999999999998,"5min":0.84999999999999998,"15min":0.84999999999999998},"min":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"max":{"1min":1.139,"5min":1.139,"15min":1.139},"last":0.94099999999999995}]}]},{"osd":9,"up_from":66,"seq":283467841570,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6120,"kb_used_data":928,"kb_used_omap":0,"kb_used_meta":5184,"kb_avail":93739032,"statfs":{"total":95995035648,"available":95988768768,"internally_reserved":0,"allocated":950272,"data_stored":442701,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5308416},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri May 13 23:14:37 2022","interfaces":[{"interface":"back","average":{"1min":0.58899999999999997,"5min":0.63900000000000001,"15min":0.64700000000000002},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":1.133,"5min":1.133,"15min":1.133},"last":0.56999999999999995},{"interface":"front","average":{"1min":0.59699999999999998,"5min":0.69699999999999995,"15min":0.71399999999999997},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.86399999999999999,"5min":1.044,"15min":1.044},"last":0.40300000000000002}]},{"osd":1,"last update":"Fri May 13 23:14:37 2022","interfaces":[{"interface":"back","average":{"1min":0.73999999999999999,"5min":0.752,"15min":0.754},"min":{"1min":0.432,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":1.6220000000000001,"5min":1.6220000000000001,"15min":1.6220000000000001},"last":0.72899999999999998},{"interface":"front","average":{"1min":0.82699999999999996,"5min":0.90100000000000002,"15min":0.91300000000000003},"min":{"1min":0.50900000000000001,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":1.3480000000000001,"5min":1.498,"15min":1.498},"last":0.65900000000000003}]},{"osd":2,"last update":"Fri May 13 23:14:37 2022","interfaces":[{"interface":"back","average":{"1min":0.83799999999999997,"5min":0.88600000000000001,"15min":0.89400000000000002},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":1.282,"5min":1.4419999999999999,"15min":1.4419999999999999},"last":0.54800000000000004},{"interface":"front","average":{"1min":0.76100000000000001,"5min":0.81299999999999994,"15min":0.82199999999999995},"min":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":1.573,"5min":1.573,"15min":1.573},"last":0.50900000000000001}]},{"osd":3,"last update":"Fri May 13 23:14:37 2022","interfaces":[{"interface":"back","average":{"1min":0.77400000000000002,"5min":0.85699999999999998,"15min":0.871},"min":{"1min":0.45200000000000001,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":1.552,"5min":1.552,"15min":1.552},"last":0.53000000000000003},{"interface":"front","average":{"1min":0.81799999999999995,"5min":0.96499999999999997,"15min":0.98899999999999999},"min":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":1.3169999999999999,"5min":2.2069999999999999,"15min":2.2069999999999999},"last":0.621}]},{"osd":4,"last update":"Fri May 13 23:14:37 2022","interfaces":[{"interface":"back","average":{"1min":0.82799999999999996,"5min":0.90900000000000003,"15min":0.92300000000000004},"min":{"1min":0.54400000000000004,"5min":0.47699999999999998,"15min":0.47699999999999998},"max":{"1min":1.7430000000000001,"5min":1.7430000000000001,"15min":1.7430000000000001},"last":0.78700000000000003},{"interface":"front","average":{"1min":0.86299999999999999,"5min":0.93899999999999995,"15min":0.95099999999999996},"min":{"1min":0.55200000000000005,"5min":0.55200000000000005,"15min":0.55200000000000005},"max":{"1min":1.2130000000000001,"5min":1.3320000000000001,"15min":1.3320000000000001},"last":0.71499999999999997}]},{"osd":5,"last update":"Fri May 13 23:14:37 2022","interfaces":[{"interface":"back","average":{"1min":0.84999999999999998,"5min":0.94599999999999995,"15min":0.96199999999999997},"min":{"1min":0.61399999999999999,"5min":0.56299999999999994,"15min":0.56299999999999994},"max":{"1min":1.661,"5min":1.661,"15min":1.661},"last":0.56000000000000005},{"interface":"front","average":{"1min":0.88500000000000001,"5min":0.99299999999999999,"15min":1.0109999999999999},"min":{"1min":0.52600000000000002,"5min":0.52600000000000002,"15min":0.52600000000000002},"max":{"1min":1.2,"5min":1.3700000000000001,"15min":1.3700000000000001},"last":0.64500000000000002}]},{"osd":6,"last update":"Fri May 13 23:14:37 2022","interfaces":[{"interface":"back","average":{"1min":0.94999999999999996,"5min":1.1020000000000001,"15min":1.127},"min":{"1min":0.54700000000000004,"5min":0.54700000000000004,"15min":0.54700000000000004},"max":{"1min":1.377,"5min":2.254,"15min":2.254},"last":0.501},{"interface":"front","average":{"1min":0.93799999999999994,"5min":1.036,"15min":1.052},"min":{"1min":0.68300000000000005,"5min":0.68300000000000005,"15min":0.68300000000000005},"max":{"1min":1.8200000000000001,"5min":1.8200000000000001,"15min":1.8200000000000001},"last":0.59399999999999997}]},{"osd":7,"last update":"Fri May 13 23:14:37 2022","interfaces":[{"interface":"back","average":{"1min":0.95699999999999996,"5min":1.0860000000000001,"15min":1.1080000000000001},"min":{"1min":0.68500000000000005,"5min":0.63,"15min":0.63},"max":{"1min":1.3080000000000001,"5min":2.101,"15min":2.101},"last":0.752},{"interface":"front","average":{"1min":1.0109999999999999,"5min":1.101,"15min":1.1160000000000001},"min":{"1min":0.73399999999999999,"5min":0.69599999999999995,"15min":0.69599999999999995},"max":{"1min":1.407,"5min":1.895,"15min":1.895},"last":0.68200000000000005}]},{"osd":8,"last update":"Fri May 13 23:14:37 2022","interfaces":[{"interface":"back","average":{"1min":0.996,"5min":1.054,"15min":1.0640000000000001},"min":{"1min":0.63500000000000001,"5min":0.59899999999999998,"15min":0.59899999999999998},"max":{"1min":1.7,"5min":1.7,"15min":1.7},"last":0.67500000000000004},{"interface":"front","average":{"1min":0.97699999999999998,"5min":1.109,"15min":1.131},"min":{"1min":0.59299999999999997,"5min":0.59299999999999997,"15min":0.59299999999999997},"max":{"1min":1.2949999999999999,"5min":2.1789999999999998,"15min":2.1789999999999998},"last":0.84399999999999997}]},{"osd":10,"last update":"Fri May 13 23:15:03 2022","interfaces":[{"interface":"back","average":{"1min":0.93300000000000005,"5min":1.069,"15min":1.0920000000000001},"min":{"1min":0.73199999999999998,"5min":0.66800000000000004,"15min":0.66800000000000004},"max":{"1min":1.252,"5min":1.7829999999999999,"15min":1.7829999999999999},"last":0.82699999999999996},{"interface":"front","average":{"1min":0.96799999999999997,"5min":1.099,"15min":1.121},"min":{"1min":0.75600000000000001,"5min":0.73099999999999998,"15min":0.73099999999999998},"max":{"1min":1.3759999999999999,"5min":1.748,"15min":1.748},"last":0.70599999999999996}]}]},{"osd":8,"up_from":58,"seq":249108103210,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6304,"kb_used_data":920,"kb_used_omap":0,"kb_used_meta":5376,"kb_avail":93738848,"statfs":{"total":95995035648,"available":95988580352,"internally_reserved":0,"allocated":942080,"data_stored":443029,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5505024},"hb_peers":[0,1,2,3,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri May 13 23:15:13 2022","interfaces":[{"interface":"back","average":{"1min":0.59099999999999997,"5min":0.59999999999999998,"15min":0.60699999999999998},"min":{"1min":0.44500000000000001,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":1.026,"5min":1.026,"15min":1.026},"last":0.55000000000000004},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.58799999999999997,"15min":0.58599999999999997},"min":{"1min":0.41499999999999998,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.91200000000000003,"5min":1.137,"15min":1.137},"last":0.55900000000000005}]},{"osd":1,"last update":"Fri May 13 23:15:13 2022","interfaces":[{"interface":"back","average":{"1min":0.627,"5min":0.57699999999999996,"15min":0.57499999999999996},"min":{"1min":0.40300000000000002,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.86699999999999999,"5min":0.86699999999999999,"15min":0.86699999999999999},"last":0.41299999999999998},{"interface":"front","average":{"1min":0.68899999999999995,"5min":0.61799999999999999,"15min":0.61199999999999999},"min":{"1min":0.42199999999999999,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.82199999999999995,"5min":0.82199999999999995,"15min":0.82199999999999995},"last":0.60799999999999998}]},{"osd":2,"last update":"Fri May 13 23:15:13 2022","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.46400000000000002,"15min":0.46200000000000002},"min":{"1min":0.32300000000000001,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.63300000000000001,"5min":0.77400000000000002,"15min":0.77400000000000002},"last":0.57599999999999996},{"interface":"front","average":{"1min":0.54200000000000004,"5min":0.47299999999999998,"15min":0.46400000000000002},"min":{"1min":0.34000000000000002,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.73199999999999998,"5min":0.79100000000000004,"15min":0.79100000000000004},"last":0.39200000000000002}]},{"osd":3,"last update":"Fri May 13 23:15:13 2022","interfaces":[{"interface":"back","average":{"1min":0.60899999999999999,"5min":0.63900000000000001,"15min":0.64900000000000002},"min":{"1min":0.40899999999999997,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.76500000000000001,"5min":0.88400000000000001,"15min":0.88400000000000001},"last":0.61899999999999999},{"interface":"front","average":{"1min":0.58599999999999997,"5min":0.63100000000000001,"15min":0.64100000000000001},"min":{"1min":0.47899999999999998,"5min":0.439,"15min":0.439},"max":{"1min":0.67900000000000005,"5min":0.83299999999999996,"15min":0.83299999999999996},"last":0.628}]},{"osd":4,"last update":"Fri May 13 23:14:08 2022","interfaces":[{"interface":"back","average":{"1min":0.61099999999999999,"5min":0.621,"15min":0.623},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":1.573,"5min":1.573,"15min":1.573},"last":0.60299999999999998},{"interface":"front","average":{"1min":0.57299999999999995,"5min":0.61899999999999999,"15min":0.627},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":0.746,"5min":0.90500000000000003,"15min":0.90500000000000003},"last":0.71299999999999997}]},{"osd":5,"last update":"Fri May 13 23:15:13 2022","interfaces":[{"interface":"back","average":{"1min":0.63300000000000001,"5min":0.58299999999999996,"15min":0.58599999999999997},"min":{"1min":0.40000000000000002,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":0.94199999999999995,"5min":0.94199999999999995,"15min":0.94199999999999995},"last":0.5},{"interface":"front","average":{"1min":0.60499999999999998,"5min":0.56799999999999995,"15min":0.57599999999999996},"min":{"1min":0.371,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.83099999999999996,"5min":0.88200000000000001,"15min":0.88200000000000001},"last":0.43099999999999999}]},{"osd":6,"last update":"Fri May 13 23:15:13 2022","interfaces":[{"interface":"back","average":{"1min":0.68200000000000005,"5min":0.72799999999999998,"15min":0.74399999999999999},"min":{"1min":0.50700000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":0.89600000000000002,"5min":0.94099999999999995,"15min":0.94099999999999995},"last":0.46800000000000003},{"interface":"front","average":{"1min":0.77900000000000003,"5min":0.75700000000000001,"15min":0.76100000000000001},"min":{"1min":0.57199999999999995,"5min":0.504,"15min":0.504},"max":{"1min":0.95399999999999996,"5min":0.95699999999999996,"15min":0.95699999999999996},"last":0.68899999999999995}]},{"osd":7,"last update":"Fri May 13 23:15:13 2022","interfaces":[{"interface":"back","average":{"1min":0.78600000000000003,"5min":0.71599999999999997,"15min":0.70999999999999996},"min":{"1min":0.48299999999999998,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":1.0069999999999999,"5min":1.0069999999999999,"15min":1.0069999999999999},"last":0.64700000000000002},{"interface":"front","average":{"1min":0.75600000000000001,"5min":0.71199999999999997,"15min":0.70399999999999996},"min":{"1min":0.47499999999999998,"5min":0.437,"15min":0.437},"max":{"1min":1.0189999999999999,"5min":1.609,"15min":1.609},"last":0.59599999999999997}]},{"osd":9,"last update":"Fri May 13 23:14:41 2022","interfaces":[{"interface":"back","average":{"1min":0.77900000000000003,"5min":0.76700000000000002,"15min":0.76500000000000001},"min":{"1min":0.59099999999999997,"5min":0.51500000000000001,"15min":0.51500000000000001},"max":{"1min":1,"5min":1,"15min":1},"last":0.63900000000000001},{"interface":"front","average":{"1min":0.77900000000000003,"5min":0.79600000000000004,"15min":0.79900000000000004},"min":{"1min":0.60599999999999998,"5min":0.59899999999999998,"15min":0.59899999999999998},"max":{"1min":0.93799999999999994,"5min":0.996,"15min":0.996},"last":0.66300000000000003}]},{"osd":10,"last update":"Fri May 13 23:15:07 2022","interfaces":[{"interface":"back","average":{"1min":0.79600000000000004,"5min":0.73999999999999999,"15min":0.73099999999999998},"min":{"1min":0.56399999999999995,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":0.99299999999999999,"5min":0.99299999999999999,"15min":0.99299999999999999},"last":0.75},{"interface":"front","average":{"1min":0.80300000000000005,"5min":0.71799999999999997,"15min":0.70399999999999996},"min":{"1min":0.53200000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":1.042,"5min":1.042,"15min":1.042},"last":0.67600000000000005}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72099999999999997}]}]},{"osd":7,"up_from":51,"seq":219043332141,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6892,"kb_used_data":1444,"kb_used_omap":0,"kb_used_meta":5440,"kb_avail":93738260,"statfs":{"total":95995035648,"available":95987978240,"internally_reserved":0,"allocated":1478656,"data_stored":973477,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5570560},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri May 13 23:13:48 2022","interfaces":[{"interface":"back","average":{"1min":0.64400000000000002,"5min":0.56000000000000005,"15min":0.54600000000000004},"min":{"1min":0.48399999999999999,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":0.82599999999999996,"5min":0.82599999999999996,"15min":0.82599999999999996},"last":0.63600000000000001},{"interface":"front","average":{"1min":0.60899999999999999,"5min":0.58799999999999997,"15min":0.58399999999999996},"min":{"1min":0.42699999999999999,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":0.78800000000000003,"5min":0.86599999999999999,"15min":0.86599999999999999},"last":0.51900000000000002}]},{"osd":1,"last update":"Fri May 13 23:14:52 2022","interfaces":[{"interface":"back","average":{"1min":0.46600000000000003,"5min":0.45200000000000001,"15min":0.44400000000000001},"min":{"1min":0.30499999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.60199999999999998,"5min":0.79800000000000004,"15min":0.79800000000000004},"last":0.60399999999999998},{"interface":"front","average":{"1min":0.5,"5min":0.47999999999999998,"15min":0.47199999999999998},"min":{"1min":0.33600000000000002,"5min":0.317,"15min":0.317},"max":{"1min":0.63900000000000001,"5min":0.88800000000000001,"15min":0.88800000000000001},"last":0.625}]},{"osd":2,"last update":"Fri May 13 23:14:52 2022","interfaces":[{"interface":"back","average":{"1min":0.80900000000000005,"5min":0.67000000000000004,"15min":0.64800000000000002},"min":{"1min":0.46600000000000003,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":3.9769999999999999,"5min":3.9769999999999999,"15min":3.9769999999999999},"last":0.65200000000000002},{"interface":"front","average":{"1min":0.64600000000000002,"5min":0.61199999999999999,"15min":0.60599999999999998},"min":{"1min":0.48799999999999999,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":0.83399999999999996,"5min":0.83399999999999996,"15min":0.83399999999999996},"last":0.79500000000000004}]},{"osd":3,"last update":"Fri May 13 23:14:52 2022","interfaces":[{"interface":"back","average":{"1min":0.65700000000000003,"5min":0.64000000000000001,"15min":0.63200000000000001},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.81299999999999994,"5min":0.92200000000000004,"15min":0.92200000000000004},"last":0.73199999999999998},{"interface":"front","average":{"1min":0.68300000000000005,"5min":0.69399999999999995,"15min":0.69599999999999995},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":0.874,"5min":0.874,"15min":0.874},"last":0.70899999999999996}]},{"osd":4,"last update":"Fri May 13 23:14:52 2022","interfaces":[{"interface":"back","average":{"1min":0.55700000000000005,"5min":0.57599999999999996,"15min":0.58899999999999997},"min":{"1min":0.29999999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.78000000000000003,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.752},{"interface":"front","average":{"1min":0.53700000000000003,"5min":0.57099999999999995,"15min":0.58499999999999996},"min":{"1min":0.378,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.80900000000000005,"5min":0.875,"15min":0.875},"last":0.72099999999999997}]},{"osd":5,"last update":"Fri May 13 23:14:52 2022","interfaces":[{"interface":"back","average":{"1min":0.70299999999999996,"5min":0.71499999999999997,"15min":0.72499999999999998},"min":{"1min":0.57599999999999996,"5min":0.47199999999999998,"15min":0.47199999999999998},"max":{"1min":0.91800000000000004,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.82199999999999995},{"interface":"front","average":{"1min":0.70599999999999996,"5min":0.70099999999999996,"15min":0.70399999999999996},"min":{"1min":0.438,"5min":0.438,"15min":0.438},"max":{"1min":0.89700000000000002,"5min":0.93999999999999995,"15min":0.93999999999999995},"last":0.90000000000000002}]},{"osd":6,"last update":"Fri May 13 23:14:52 2022","interfaces":[{"interface":"back","average":{"1min":0.73699999999999999,"5min":0.75700000000000001,"15min":0.76000000000000001},"min":{"1min":0.501,"5min":0.501,"15min":0.501},"max":{"1min":0.99199999999999999,"5min":1.0700000000000001,"15min":1.0700000000000001},"last":0.78100000000000003},{"interface":"front","average":{"1min":0.73699999999999999,"5min":0.78600000000000003,"15min":0.79000000000000004},"min":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"max":{"1min":1.054,"5min":1.1319999999999999,"15min":1.1319999999999999},"last":0.86899999999999999}]},{"osd":8,"last update":"Fri May 13 23:15:07 2022","interfaces":[{"interface":"back","average":{"1min":0.80000000000000004,"5min":0.78400000000000003,"15min":0.754},"min":{"1min":0.627,"5min":0.51300000000000001,"15min":0.51300000000000001},"max":{"1min":0.95099999999999996,"5min":4.0739999999999998,"15min":4.0739999999999998},"last":1.0449999999999999},{"interface":"front","average":{"1min":0.84299999999999997,"5min":0.78100000000000003,"15min":0.76700000000000002},"min":{"1min":0.64400000000000002,"5min":0.497,"15min":0.497},"max":{"1min":1.006,"5min":1.046,"15min":1.046},"last":0.97999999999999998}]},{"osd":9,"last update":"Fri May 13 23:14:38 2022","interfaces":[{"interface":"back","average":{"1min":0.84099999999999997,"5min":0.82899999999999996,"15min":0.82699999999999996},"min":{"1min":0.55200000000000005,"5min":0.55200000000000005,"15min":0.55200000000000005},"max":{"1min":0.98299999999999998,"5min":1.1339999999999999,"15min":1.1339999999999999},"last":1.0109999999999999},{"interface":"front","average":{"1min":0.85199999999999998,"5min":0.83299999999999996,"15min":0.82999999999999996},"min":{"1min":0.66400000000000003,"5min":0.64900000000000002,"15min":0.64900000000000002},"max":{"1min":1.0389999999999999,"5min":1.0409999999999999,"15min":1.0409999999999999},"last":0.91800000000000004}]},{"osd":10,"last update":"Fri May 13 23:15:00 2022","interfaces":[{"interface":"back","average":{"1min":0.80700000000000005,"5min":0.78000000000000003,"15min":0.77600000000000002},"min":{"1min":0.54000000000000004,"5min":0.47599999999999998,"15min":0.47599999999999998},"max":{"1min":1.0069999999999999,"5min":1.093,"15min":1.093},"last":0.93500000000000005},{"interface":"front","average":{"1min":0.81699999999999995,"5min":0.746,"15min":0.73399999999999999},"min":{"1min":0.627,"5min":0.51300000000000001,"15min":0.51300000000000001},"max":{"1min":1.026,"5min":1.0529999999999999,"15min":1.0529999999999999},"last":0.95399999999999996}]},{"osd":11,"last update":"Fri May 13 23:15:17 2022","interfaces":[{"interface":"back","average":{"1min":0.88200000000000001,"5min":0.88200000000000001,"15min":0.88200000000000001},"min":{"1min":0.69899999999999995,"5min":0.69899999999999995,"15min":0.69899999999999995},"max":{"1min":1.079,"5min":1.079,"15min":1.079},"last":1.0700000000000001},{"interface":"front","average":{"1min":0.82899999999999996,"5min":0.82899999999999996,"15min":0.82899999999999996},"min":{"1min":0.746,"5min":0.746,"15min":0.746},"max":{"1min":0.95799999999999996,"5min":0.95799999999999996,"15min":0.95799999999999996},"last":0.88900000000000001}]}]},{"osd":6,"up_from":46,"seq":197568495667,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6228,"kb_used_data":908,"kb_used_omap":0,"kb_used_meta":5312,"kb_avail":93738924,"statfs":{"total":95995035648,"available":95988658176,"internally_reserved":0,"allocated":929792,"data_stored":442281,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5439488},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri May 13 23:14:30 2022","interfaces":[{"interface":"back","average":{"1min":0.59999999999999998,"5min":0.56299999999999994,"15min":0.54900000000000004},"min":{"1min":0.39200000000000002,"5min":0.378,"15min":0.378},"max":{"1min":1.071,"5min":1.071,"15min":1.071},"last":0.441},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.60499999999999998,"15min":0.59899999999999998},"min":{"1min":0.36699999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":1.262,"5min":1.262,"15min":1.262},"last":0.33100000000000002}]},{"osd":1,"last update":"Fri May 13 23:14:30 2022","interfaces":[{"interface":"back","average":{"1min":0.76300000000000001,"5min":0.76100000000000001,"15min":0.751},"min":{"1min":0.52600000000000002,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":1.1830000000000001,"5min":1.2210000000000001,"15min":1.2210000000000001},"last":0.66600000000000004},{"interface":"front","average":{"1min":0.70799999999999996,"5min":0.68600000000000005,"15min":0.67200000000000004},"min":{"1min":0.52200000000000002,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":0.91700000000000004,"5min":1.28,"15min":1.28},"last":0.69399999999999995}]},{"osd":2,"last update":"Fri May 13 23:14:30 2022","interfaces":[{"interface":"back","average":{"1min":0.78300000000000003,"5min":0.78700000000000003,"15min":0.78500000000000003},"min":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.51600000000000001},"max":{"1min":1.3540000000000001,"5min":1.3540000000000001,"15min":1.3540000000000001},"last":0.61199999999999999},{"interface":"front","average":{"1min":0.79600000000000004,"5min":0.81299999999999994,"15min":0.81200000000000006},"min":{"1min":0.60799999999999998,"5min":0.53300000000000003,"15min":0.53300000000000003},"max":{"1min":1.0620000000000001,"5min":1.298,"15min":1.298},"last":0.71699999999999997}]},{"osd":3,"last update":"Fri May 13 23:14:30 2022","interfaces":[{"interface":"back","average":{"1min":0.77500000000000002,"5min":0.80800000000000005,"15min":0.80500000000000005},"min":{"1min":0.52500000000000002,"5min":0.47599999999999998,"15min":0.47599999999999998},"max":{"1min":1.105,"5min":1.3839999999999999,"15min":1.3839999999999999},"last":0.58699999999999997},{"interface":"front","average":{"1min":0.68600000000000005,"5min":0.75900000000000001,"15min":0.77500000000000002},"min":{"1min":0.48499999999999999,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":0.91400000000000003,"5min":1.4119999999999999,"15min":1.4119999999999999},"last":0.46400000000000002}]},{"osd":4,"last update":"Fri May 13 23:14:30 2022","interfaces":[{"interface":"back","average":{"1min":0.80800000000000005,"5min":0.83199999999999996,"15min":0.83999999999999997},"min":{"1min":0.505,"5min":0.504,"15min":0.504},"max":{"1min":1.3120000000000001,"5min":1.5640000000000001,"15min":1.5640000000000001},"last":0.67700000000000005},{"interface":"front","average":{"1min":0.88700000000000001,"5min":0.89900000000000002,"15min":0.89400000000000002},"min":{"1min":0.61199999999999999,"5min":0.60799999999999998,"15min":0.60799999999999998},"max":{"1min":1.238,"5min":1.625,"15min":1.625},"last":0.747}]},{"osd":5,"last update":"Fri May 13 23:14:30 2022","interfaces":[{"interface":"back","average":{"1min":0.79400000000000004,"5min":0.84399999999999997,"15min":0.85399999999999998},"min":{"1min":0.57199999999999995,"5min":0.50600000000000001,"15min":0.50600000000000001},"max":{"1min":1.123,"5min":1.5029999999999999,"15min":1.5029999999999999},"last":0.75700000000000001},{"interface":"front","average":{"1min":0.88300000000000001,"5min":0.92600000000000005,"15min":0.93400000000000005},"min":{"1min":0.623,"5min":0.623,"15min":0.623},"max":{"1min":1.3999999999999999,"5min":1.3999999999999999,"15min":1.3999999999999999},"last":0.57399999999999995}]},{"osd":7,"last update":"Fri May 13 23:14:53 2022","interfaces":[{"interface":"back","average":{"1min":0.93000000000000005,"5min":0.91200000000000003,"15min":0.874},"min":{"1min":0.67700000000000005,"5min":0.66000000000000003,"15min":0.66000000000000003},"max":{"1min":1.1990000000000001,"5min":1.534,"15min":1.534},"last":0.79000000000000004},{"interface":"front","average":{"1min":0.83999999999999997,"5min":0.85199999999999998,"15min":0.82699999999999996},"min":{"1min":0.627,"5min":0.61299999999999999,"15min":0.61299999999999999},"max":{"1min":1.1279999999999999,"5min":1.6180000000000001,"15min":1.6180000000000001},"last":0.80000000000000004}]},{"osd":8,"last update":"Fri May 13 23:15:13 2022","interfaces":[{"interface":"back","average":{"1min":1.03,"5min":0.995,"15min":0.98499999999999999},"min":{"1min":0.81299999999999994,"5min":0.69999999999999996,"15min":0.69999999999999996},"max":{"1min":1.411,"5min":1.484,"15min":1.484},"last":0.76900000000000002},{"interface":"front","average":{"1min":0.95999999999999996,"5min":0.94599999999999995,"15min":0.95399999999999996},"min":{"1min":0.68300000000000005,"5min":0.67500000000000004,"15min":0.67500000000000004},"max":{"1min":1.4219999999999999,"5min":1.548,"15min":1.548},"last":0.82499999999999996}]},{"osd":9,"last update":"Fri May 13 23:14:39 2022","interfaces":[{"interface":"back","average":{"1min":0.93300000000000005,"5min":1.0429999999999999,"15min":1.0609999999999999},"min":{"1min":0.68500000000000005,"5min":0.68500000000000005,"15min":0.68500000000000005},"max":{"1min":1.2629999999999999,"5min":1.458,"15min":1.458},"last":0.85399999999999998},{"interface":"front","average":{"1min":0.94099999999999995,"5min":1.034,"15min":1.05},"min":{"1min":0.58099999999999996,"5min":0.58099999999999996,"15min":0.58099999999999996},"max":{"1min":1.4390000000000001,"5min":1.4390000000000001,"15min":1.4390000000000001},"last":0.71199999999999997}]},{"osd":10,"last update":"Fri May 13 23:15:02 2022","interfaces":[{"interface":"back","average":{"1min":1.0549999999999999,"5min":1.0529999999999999,"15min":1.0529999999999999},"min":{"1min":0.82799999999999996,"5min":0.77800000000000002,"15min":0.77800000000000002},"max":{"1min":1.4710000000000001,"5min":1.645,"15min":1.645},"last":0.88},{"interface":"front","average":{"1min":1.0609999999999999,"5min":1.0169999999999999,"15min":1.0089999999999999},"min":{"1min":0.81399999999999995,"5min":0.73999999999999999,"15min":0.73999999999999999},"max":{"1min":1.4990000000000001,"5min":1.4990000000000001,"15min":1.4990000000000001},"last":0.73999999999999999}]}]},{"osd":1,"up_from":13,"seq":55834574926,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":7132,"kb_used_data":916,"kb_used_omap":0,"kb_used_meta":6208,"kb_avail":93738020,"statfs":{"total":95995035648,"available":95987732480,"internally_reserved":0,"allocated":937984,"data_stored":444791,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":6356992},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri May 13 23:15:14 2022","interfaces":[{"interface":"back","average":{"1min":0.64700000000000002,"5min":0.58999999999999997,"15min":0.52300000000000002},"min":{"1min":0.45400000000000001,"5min":0.34799999999999998,"15min":0.32600000000000001},"max":{"1min":0.86899999999999999,"5min":0.91700000000000004,"15min":0.91700000000000004},"last":0.64600000000000002},{"interface":"front","average":{"1min":0.627,"5min":0.59999999999999998,"15min":0.54700000000000004},"min":{"1min":0.45700000000000002,"5min":0.373,"15min":0.309},"max":{"1min":0.81299999999999994,"5min":0.86799999999999999,"15min":0.86799999999999999},"last":0.749}]},{"osd":2,"last update":"Fri May 13 23:14:44 2022","interfaces":[{"interface":"back","average":{"1min":0.59999999999999998,"5min":0.54100000000000004,"15min":0.51900000000000002},"min":{"1min":0.39900000000000002,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.77600000000000002,"5min":0.77600000000000002,"15min":0.77600000000000002},"last":0.68899999999999995},{"interface":"front","average":{"1min":0.67100000000000004,"5min":0.57399999999999995,"15min":0.53500000000000003},"min":{"1min":0.41599999999999998,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.96399999999999997,"5min":0.96399999999999997,"15min":0.96399999999999997},"last":0.90800000000000003}]},{"osd":3,"last update":"Fri May 13 23:15:13 2022","interfaces":[{"interface":"back","average":{"1min":0.71599999999999997,"5min":0.628,"15min":0.58199999999999996},"min":{"1min":0.503,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.96499999999999997,"5min":0.96499999999999997,"15min":0.96499999999999997},"last":0.67000000000000004},{"interface":"front","average":{"1min":0.66700000000000004,"5min":0.59399999999999997,"15min":0.56899999999999995},"min":{"1min":0.48099999999999998,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.873,"5min":0.95399999999999996,"15min":0.95399999999999996},"last":0.80900000000000005}]},{"osd":4,"last update":"Fri May 13 23:14:35 2022","interfaces":[{"interface":"back","average":{"1min":0.59099999999999997,"5min":0.47099999999999997,"15min":0.42899999999999999},"min":{"1min":0.35499999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.91400000000000003,"5min":0.91400000000000003,"15min":0.91400000000000003},"last":0.80000000000000004},{"interface":"front","average":{"1min":0.60899999999999999,"5min":0.504,"15min":0.45700000000000002},"min":{"1min":0.34100000000000003,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.90200000000000002,"5min":0.90200000000000002,"15min":0.90200000000000002},"last":0.83199999999999996}]},{"osd":5,"last update":"Fri May 13 23:14:55 2022","interfaces":[{"interface":"back","average":{"1min":0.73299999999999998,"5min":0.63200000000000001,"15min":0.59599999999999997},"min":{"1min":0.42999999999999999,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.98699999999999999,"5min":0.98699999999999999,"15min":0.98699999999999999},"last":0.84999999999999998},{"interface":"front","average":{"1min":0.75700000000000001,"5min":0.66000000000000003,"15min":0.61599999999999999},"min":{"1min":0.49199999999999999,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":0.95599999999999996,"5min":0.98799999999999999,"15min":0.98799999999999999},"last":0.96399999999999997}]},{"osd":6,"last update":"Fri May 13 23:14:26 2022","interfaces":[{"interface":"back","average":{"1min":0.78100000000000003,"5min":0.70999999999999996,"15min":0.68200000000000005},"min":{"1min":0.497,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":1.0840000000000001,"5min":1.5529999999999999,"15min":1.5529999999999999},"last":0.88800000000000001},{"interface":"front","average":{"1min":0.80800000000000005,"5min":0.73599999999999999,"15min":0.71699999999999997},"min":{"1min":0.48499999999999999,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":1.0209999999999999,"5min":1.1419999999999999,"15min":1.1419999999999999},"last":1.0149999999999999}]},{"osd":7,"last update":"Fri May 13 23:14:49 2022","interfaces":[{"interface":"back","average":{"1min":0.628,"5min":0.59899999999999998,"15min":0.56799999999999995},"min":{"1min":0.39800000000000002,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.91400000000000003,"5min":0.95999999999999996,"15min":0.95999999999999996},"last":0.871},{"interface":"front","average":{"1min":0.75600000000000001,"5min":0.68100000000000005,"15min":0.63900000000000001},"min":{"1min":0.45800000000000002,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":1.079,"5min":1.079,"15min":1.079},"last":0.94999999999999996}]},{"osd":8,"last update":"Fri May 13 23:15:13 2022","interfaces":[{"interface":"back","average":{"1min":0.83299999999999996,"5min":0.70499999999999996,"15min":0.66900000000000004},"min":{"1min":0.50700000000000001,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":1.0960000000000001,"5min":1.0960000000000001,"15min":1.0960000000000001},"last":0.99299999999999999},{"interface":"front","average":{"1min":0.93799999999999994,"5min":0.78000000000000003,"15min":0.72999999999999998},"min":{"1min":0.501,"5min":0.501,"15min":0.501},"max":{"1min":1.214,"5min":1.214,"15min":1.214},"last":1.03}]},{"osd":9,"last update":"Fri May 13 23:14:44 2022","interfaces":[{"interface":"back","average":{"1min":0.89400000000000002,"5min":0.88600000000000001,"15min":0.88500000000000001},"min":{"1min":0.55000000000000004,"5min":0.55000000000000004,"15min":0.55000000000000004},"max":{"1min":1.4550000000000001,"5min":1.4550000000000001,"15min":1.4550000000000001},"last":1.0649999999999999},{"interface":"front","average":{"1min":0.86799999999999999,"5min":0.84999999999999998,"15min":0.84699999999999998},"min":{"1min":0.59499999999999997,"5min":0.59499999999999997,"15min":0.59499999999999997},"max":{"1min":1.099,"5min":1.099,"15min":1.099},"last":1.075}]},{"osd":10,"last update":"Fri May 13 23:15:04 2022","interfaces":[{"interface":"back","average":{"1min":0.79100000000000004,"5min":0.78800000000000003,"15min":0.78800000000000003},"min":{"1min":0.53000000000000003,"5min":0.51300000000000001,"15min":0.51300000000000001},"max":{"1min":0.97399999999999998,"5min":0.97399999999999998,"15min":0.97399999999999998},"last":1.0429999999999999},{"interface":"front","average":{"1min":0.91700000000000004,"5min":0.90900000000000003,"15min":0.90800000000000003},"min":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":1.123,"5min":1.123,"15min":1.123},"last":0.98199999999999998}]},{"osd":11,"last update":"Fri May 13 23:15:17 2022","interfaces":[{"interface":"back","average":{"1min":0.96799999999999997,"5min":0.96799999999999997,"15min":0.96799999999999997},"min":{"1min":0.80700000000000005,"5min":0.80700000000000005,"15min":0.80700000000000005},"max":{"1min":1.121,"5min":1.121,"15min":1.121},"last":1.0529999999999999},{"interface":"front","average":{"1min":1.0449999999999999,"5min":1.0449999999999999,"15min":1.0449999999999999},"min":{"1min":0.80300000000000005,"5min":0.80300000000000005,"15min":0.80300000000000005},"max":{"1min":1.254,"5min":1.254,"15min":1.254},"last":1.0840000000000001}]}]},{"osd":0,"up_from":8,"seq":34359738449,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":7148,"kb_used_data":932,"kb_used_omap":0,"kb_used_meta":6208,"kb_avail":93738004,"statfs":{"total":95995035648,"available":95987716096,"internally_reserved":0,"allocated":954368,"data_stored":443681,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":6356992},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri May 13 23:15:14 2022","interfaces":[{"interface":"back","average":{"1min":0.70199999999999996,"5min":0.64700000000000002,"15min":0.61699999999999999},"min":{"1min":0.42799999999999999,"5min":0.38500000000000001,"15min":0.38300000000000001},"max":{"1min":1.131,"5min":1.2729999999999999,"15min":1.2729999999999999},"last":0.94199999999999995},{"interface":"front","average":{"1min":0.755,"5min":0.69099999999999995,"15min":0.63700000000000001},"min":{"1min":0.48899999999999999,"5min":0.376,"15min":0.376},"max":{"1min":1.0329999999999999,"5min":1.167,"15min":1.167},"last":1.0329999999999999}]},{"osd":2,"last update":"Fri May 13 23:14:36 2022","interfaces":[{"interface":"back","average":{"1min":0.748,"5min":0.67500000000000004,"15min":0.65300000000000002},"min":{"1min":0.48899999999999999,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":1.024,"5min":1.024,"15min":1.024},"last":1.0980000000000001},{"interface":"front","average":{"1min":0.69399999999999995,"5min":0.65000000000000002,"15min":0.63300000000000001},"min":{"1min":0.46500000000000002,"5min":0.45900000000000002,"15min":0.45900000000000002},"max":{"1min":1.071,"5min":1.123,"15min":1.123},"last":0.68000000000000005}]},{"osd":3,"last update":"Fri May 13 23:13:08 2022","interfaces":[{"interface":"back","average":{"1min":0.82399999999999995,"5min":0.63300000000000001,"15min":0.59999999999999998},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":1.163,"5min":1.163,"15min":1.163},"last":0.74399999999999999},{"interface":"front","average":{"1min":0.79800000000000004,"5min":0.67000000000000004,"15min":0.64600000000000002},"min":{"1min":0.46200000000000002,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":1.157,"5min":1.157,"15min":1.157},"last":1.161}]},{"osd":4,"last update":"Fri May 13 23:14:35 2022","interfaces":[{"interface":"back","average":{"1min":0.754,"5min":0.70399999999999996,"15min":0.66200000000000003},"min":{"1min":0.57399999999999995,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":1.0169999999999999,"5min":1.1990000000000001,"15min":1.1990000000000001},"last":0.996},{"interface":"front","average":{"1min":0.71599999999999997,"5min":0.68200000000000005,"15min":0.63900000000000001},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":1.1830000000000001,"5min":1.1830000000000001,"15min":1.1830000000000001},"last":0.63100000000000001}]},{"osd":5,"last update":"Fri May 13 23:15:01 2022","interfaces":[{"interface":"back","average":{"1min":0.78000000000000003,"5min":0.73999999999999999,"15min":0.71599999999999997},"min":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":1.837,"5min":1.837,"15min":1.837},"last":1.1319999999999999},{"interface":"front","average":{"1min":0.67400000000000004,"5min":0.69199999999999995,"15min":0.69399999999999995},"min":{"1min":0.46899999999999997,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":0.83599999999999997,"5min":1.056,"15min":1.056},"last":1.117}]},{"osd":6,"last update":"Fri May 13 23:14:27 2022","interfaces":[{"interface":"back","average":{"1min":0.752,"5min":0.77900000000000003,"15min":0.751},"min":{"1min":0.45100000000000001,"5min":0.44800000000000001,"15min":0.44800000000000001},"max":{"1min":0.94999999999999996,"5min":1.288,"15min":1.288},"last":0.84699999999999998},{"interface":"front","average":{"1min":0.79000000000000004,"5min":0.79000000000000004,"15min":0.77100000000000002},"min":{"1min":0.502,"5min":0.5,"15min":0.5},"max":{"1min":1.24,"5min":1.24,"15min":1.24},"last":1.179}]},{"osd":7,"last update":"Fri May 13 23:13:48 2022","interfaces":[{"interface":"back","average":{"1min":0.77300000000000002,"5min":0.75,"15min":0.746},"min":{"1min":0.497,"5min":0.48699999999999999,"15min":0.48699999999999999},"max":{"1min":1.2050000000000001,"5min":1.2050000000000001,"15min":1.2050000000000001},"last":0.94199999999999995},{"interface":"front","average":{"1min":0.94899999999999995,"5min":0.86099999999999999,"15min":0.84699999999999998},"min":{"1min":0.69599999999999995,"5min":0.60399999999999998,"15min":0.60399999999999998},"max":{"1min":1.2350000000000001,"5min":1.2350000000000001,"15min":1.2350000000000001},"last":0.877}]},{"osd":8,"last update":"Fri May 13 23:15:09 2022","interfaces":[{"interface":"back","average":{"1min":0.76900000000000002,"5min":0.79900000000000004,"15min":0.79900000000000004},"min":{"1min":0.58499999999999996,"5min":0.52100000000000002,"15min":0.52100000000000002},"max":{"1min":1.105,"5min":1.7070000000000001,"15min":1.7070000000000001},"last":1.0149999999999999},{"interface":"front","average":{"1min":0.88,"5min":0.90800000000000003,"15min":0.91400000000000003},"min":{"1min":0.622,"5min":0.60599999999999998,"15min":0.60599999999999998},"max":{"1min":1.5680000000000001,"5min":1.5680000000000001,"15min":1.5680000000000001},"last":0.91800000000000004}]},{"osd":9,"last update":"Fri May 13 23:14:39 2022","interfaces":[{"interface":"back","average":{"1min":0.92600000000000005,"5min":1.032,"15min":1.05},"min":{"1min":0.60499999999999998,"5min":0.60499999999999998,"15min":0.60499999999999998},"max":{"1min":1.1759999999999999,"5min":1.397,"15min":1.397},"last":0.80300000000000005},{"interface":"front","average":{"1min":0.88600000000000001,"5min":0.93000000000000005,"15min":0.93700000000000006},"min":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.59099999999999997},"max":{"1min":1.28,"5min":1.28,"15min":1.28},"last":0.71399999999999997}]},{"osd":10,"last update":"Fri May 13 23:15:03 2022","interfaces":[{"interface":"back","average":{"1min":0.84399999999999997,"5min":0.94599999999999995,"15min":0.96299999999999997},"min":{"1min":0.61899999999999999,"5min":0.61899999999999999,"15min":0.61899999999999999},"max":{"1min":1.26,"5min":1.7989999999999999,"15min":1.7989999999999999},"last":0.97899999999999998},{"interface":"front","average":{"1min":1.0109999999999999,"5min":1.052,"15min":1.0589999999999999},"min":{"1min":0.753,"5min":0.745,"15min":0.745},"max":{"1min":1.9279999999999999,"5min":1.9279999999999999,"15min":1.9279999999999999},"last":0.754}]},{"osd":11,"last update":"Fri May 13 23:14:22 2022","interfaces":[{"interface":"back","average":{"1min":0.97899999999999998,"5min":0.97899999999999998,"15min":0.97899999999999998},"min":{"1min":0.63400000000000001,"5min":0.63400000000000001,"15min":0.63400000000000001},"max":{"1min":1.617,"5min":1.617,"15min":1.617},"last":1.0449999999999999},{"interface":"front","average":{"1min":1.0229999999999999,"5min":1.0229999999999999,"15min":1.0229999999999999},"min":{"1min":0.68400000000000005,"5min":0.68400000000000005,"15min":0.68400000000000005},"max":{"1min":1.266,"5min":1.266,"15min":1.266},"last":0.95599999999999996}]}]},{"osd":2,"up_from":18,"seq":77309411400,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":7272,"kb_used_data":928,"kb_used_omap":0,"kb_used_meta":6336,"kb_avail":93737880,"statfs":{"total":95995035648,"available":95987589120,"internally_reserved":0,"allocated":950272,"data_stored":444923,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":6488064},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri May 13 23:14:38 2022","interfaces":[{"interface":"back","average":{"1min":0.56000000000000005,"5min":0.52300000000000002,"15min":0.52500000000000002},"min":{"1min":0.374,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.93600000000000005,"5min":0.93600000000000005,"15min":0.93600000000000005},"last":0.45300000000000001},{"interface":"front","average":{"1min":0.58599999999999997,"5min":0.53500000000000003,"15min":0.52100000000000002},"min":{"1min":0.442,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.91200000000000003,"5min":0.91200000000000003,"15min":0.91200000000000003},"last":0.55900000000000005}]},{"osd":1,"last update":"Fri May 13 23:14:38 2022","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.54200000000000004,"15min":0.52600000000000002},"min":{"1min":0.439,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.75600000000000001,"5min":0.83699999999999997,"15min":0.83699999999999997},"last":0.52400000000000002},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.53100000000000003,"15min":0.54900000000000004},"min":{"1min":0.35199999999999998,"5min":0.313,"15min":0.313},"max":{"1min":0.82699999999999996,"5min":0.82699999999999996,"15min":0.82699999999999996},"last":0.47899999999999998}]},{"osd":3,"last update":"Fri May 13 23:15:15 2022","interfaces":[{"interface":"back","average":{"1min":0.64300000000000002,"5min":0.623,"15min":0.61399999999999999},"min":{"1min":0.51300000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.80200000000000005,"5min":0.90500000000000003,"15min":0.90500000000000003},"last":0.71599999999999997},{"interface":"front","average":{"1min":0.61699999999999999,"5min":0.60999999999999999,"15min":0.59899999999999998},"min":{"1min":0.42899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.82699999999999996,"5min":0.89500000000000002,"15min":0.89500000000000002},"last":0.502}]},{"osd":4,"last update":"Fri May 13 23:15:16 2022","interfaces":[{"interface":"back","average":{"1min":0.61099999999999999,"5min":0.61099999999999999,"15min":0.61099999999999999},"min":{"1min":0.438,"5min":0.438,"15min":0.438},"max":{"1min":0.79400000000000004,"5min":0.79400000000000004,"15min":0.79400000000000004},"last":0.77900000000000003},{"interface":"front","average":{"1min":0.629,"5min":0.629,"15min":0.629},"min":{"1min":0.47699999999999998,"5min":0.47699999999999998,"15min":0.47699999999999998},"max":{"1min":0.88200000000000001,"5min":0.88200000000000001,"15min":0.88200000000000001},"last":0.64600000000000002}]},{"osd":5,"last update":"Fri May 13 23:15:01 2022","interfaces":[{"interface":"back","average":{"1min":0.48099999999999998,"5min":0.42399999999999999,"15min":0.39000000000000001},"min":{"1min":0.33700000000000002,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.77900000000000003,"5min":0.77900000000000003,"15min":0.77900000000000003},"last":0.59799999999999998},{"interface":"front","average":{"1min":0.56399999999999995,"5min":0.45300000000000001,"15min":0.40899999999999997},"min":{"1min":0.33100000000000002,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.80600000000000005,"5min":0.80600000000000005,"15min":0.80600000000000005},"last":0.623}]},{"osd":6,"last update":"Fri May 13 23:14:32 2022","interfaces":[{"interface":"back","average":{"1min":0.73599999999999999,"5min":0.63200000000000001,"15min":0.61399999999999999},"min":{"1min":0.61599999999999999,"5min":0.44800000000000001,"15min":0.44800000000000001},"max":{"1min":0.96599999999999997,"5min":0.96599999999999997,"15min":0.96599999999999997},"last":0.77000000000000002},{"interface":"front","average":{"1min":0.66200000000000003,"5min":0.60599999999999998,"15min":0.58799999999999997},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.93100000000000005,"5min":0.93100000000000005,"15min":0.93100000000000005},"last":0.82599999999999996}]},{"osd":7,"last update":"Fri May 13 23:13:45 2022","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":0.61599999999999999,"15min":0.61599999999999999},"min":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":0.80400000000000005,"5min":0.80400000000000005,"15min":0.80400000000000005},"last":0.78400000000000003},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.63800000000000001,"15min":0.64200000000000002},"min":{"1min":0.45900000000000002,"5min":0.45900000000000002,"15min":0.45900000000000002},"max":{"1min":0.83199999999999996,"5min":0.83199999999999996,"15min":0.83199999999999996},"last":0.78800000000000003}]},{"osd":8,"last update":"Fri May 13 23:15:09 2022","interfaces":[{"interface":"back","average":{"1min":0.56899999999999995,"5min":0.54300000000000004,"15min":0.53500000000000003},"min":{"1min":0.42199999999999999,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.83399999999999996,"5min":0.876,"15min":0.876},"last":0.69399999999999995},{"interface":"front","average":{"1min":0.629,"5min":0.57899999999999996,"15min":0.57099999999999995},"min":{"1min":0.49399999999999999,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.97199999999999998,"5min":0.97199999999999998,"15min":0.97199999999999998},"last":0.67900000000000005}]},{"osd":9,"last update":"Fri May 13 23:14:38 2022","interfaces":[{"interface":"back","average":{"1min":0.77500000000000002,"5min":0.72299999999999998,"15min":0.71399999999999997},"min":{"1min":0.57099999999999995,"5min":0.55100000000000005,"15min":0.55100000000000005},"max":{"1min":0.94099999999999995,"5min":0.94099999999999995,"15min":0.94099999999999995},"last":0.73599999999999999},{"interface":"front","average":{"1min":0.73999999999999999,"5min":0.69999999999999996,"15min":0.69299999999999995},"min":{"1min":0.498,"5min":0.498,"15min":0.498},"max":{"1min":0.89000000000000001,"5min":0.89000000000000001,"15min":0.89000000000000001},"last":0.86799999999999999}]},{"osd":10,"last update":"Fri May 13 23:15:01 2022","interfaces":[{"interface":"back","average":{"1min":0.70899999999999996,"5min":0.69299999999999995,"15min":0.69099999999999995},"min":{"1min":0.58099999999999996,"5min":0.47599999999999998,"15min":0.47599999999999998},"max":{"1min":0.92700000000000005,"5min":0.97099999999999997,"15min":0.97099999999999997},"last":0.80400000000000005},{"interface":"front","average":{"1min":0.76500000000000001,"5min":0.72099999999999997,"15min":0.71399999999999997},"min":{"1min":0.60299999999999998,"5min":0.52100000000000002,"15min":0.52100000000000002},"max":{"1min":1.014,"5min":1.014,"15min":1.014},"last":0.88200000000000001}]},{"osd":11,"last update":"Fri May 13 23:15:15 2022","interfaces":[{"interface":"back","average":{"1min":0.68799999999999994,"5min":0.68799999999999994,"15min":0.68799999999999994},"min":{"1min":0.47899999999999998,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":0.91100000000000003,"5min":0.91100000000000003,"15min":0.91100000000000003},"last":0.90600000000000003},{"interface":"front","average":{"1min":0.74299999999999999,"5min":0.74299999999999999,"15min":0.74299999999999999},"min":{"1min":0.57999999999999996,"5min":0.57999999999999996,"15min":0.57999999999999996},"max":{"1min":0.99399999999999999,"5min":0.99399999999999999,"15min":0.99399999999999999},"last":0.83899999999999997}]}]},{"osd":3,"up_from":25,"seq":107374182466,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6560,"kb_used_data":920,"kb_used_omap":0,"kb_used_meta":5632,"kb_avail":93738592,"statfs":{"total":95995035648,"available":95988318208,"internally_reserved":0,"allocated":942080,"data_stored":442037,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5767168},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri May 13 23:15:16 2022","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":0.51600000000000001,"15min":0.47199999999999998},"min":{"1min":0.41499999999999998,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":1.2689999999999999,"5min":1.2689999999999999,"15min":1.2689999999999999},"last":0.41899999999999998},{"interface":"front","average":{"1min":0.61899999999999999,"5min":0.55000000000000004,"15min":0.51000000000000001},"min":{"1min":0.36799999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":1.262,"5min":1.262,"15min":1.262},"last":0.51300000000000001}]},{"osd":1,"last update":"Fri May 13 23:15:16 2022","interfaces":[{"interface":"back","average":{"1min":0.72699999999999998,"5min":0.69999999999999996,"15min":0.64200000000000002},"min":{"1min":0.45100000000000001,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":1.1950000000000001,"5min":1.1950000000000001,"15min":1.1950000000000001},"last":0.76200000000000001},{"interface":"front","average":{"1min":0.72199999999999998,"5min":0.66200000000000003,"15min":0.63800000000000001},"min":{"1min":0.51200000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":1.0089999999999999,"5min":1.0229999999999999,"15min":1.0229999999999999},"last":0.64100000000000001}]},{"osd":2,"last update":"Fri May 13 23:15:16 2022","interfaces":[{"interface":"back","average":{"1min":0.76400000000000001,"5min":0.74299999999999999,"15min":0.69399999999999995},"min":{"1min":0.47899999999999998,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":1.349,"5min":1.349,"15min":1.349},"last":0.66600000000000004},{"interface":"front","average":{"1min":0.71499999999999997,"5min":0.70699999999999996,"15min":0.67900000000000005},"min":{"1min":0.55500000000000005,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":1.1619999999999999,"5min":1.1619999999999999,"15min":1.1619999999999999},"last":0.83099999999999996}]},{"osd":4,"last update":"Fri May 13 23:14:34 2022","interfaces":[{"interface":"back","average":{"1min":0.79100000000000004,"5min":0.745,"15min":0.69199999999999995},"min":{"1min":0.51300000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":1.1499999999999999,"5min":1.157,"15min":1.157},"last":0.73099999999999998},{"interface":"front","average":{"1min":0.75700000000000001,"5min":0.74199999999999999,"15min":0.69399999999999995},"min":{"1min":0.55300000000000005,"5min":0.47699999999999998,"15min":0.47699999999999998},"max":{"1min":1.075,"5min":1.075,"15min":1.075},"last":0.91500000000000004}]},{"osd":5,"last update":"Fri May 13 23:14:55 2022","interfaces":[{"interface":"back","average":{"1min":0.72999999999999998,"5min":0.746,"15min":0.72999999999999998},"min":{"1min":0.51100000000000001,"5min":0.51100000000000001,"15min":0.51100000000000001},"max":{"1min":1.0409999999999999,"5min":1.056,"15min":1.056},"last":0.72199999999999998},{"interface":"front","average":{"1min":0.79700000000000004,"5min":0.80900000000000005,"15min":0.76900000000000002},"min":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"max":{"1min":1.1060000000000001,"5min":1.161,"15min":1.161},"last":0.89800000000000002}]},{"osd":6,"last update":"Fri May 13 23:14:28 2022","interfaces":[{"interface":"back","average":{"1min":0.85599999999999998,"5min":0.84299999999999997,"15min":0.81899999999999995},"min":{"1min":0.497,"5min":0.497,"15min":0.497},"max":{"1min":1.1579999999999999,"5min":1.2829999999999999,"15min":1.2829999999999999},"last":0.97099999999999997},{"interface":"front","average":{"1min":0.84699999999999998,"5min":0.86399999999999999,"15min":0.85199999999999998},"min":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":1.2549999999999999,"5min":1.256,"15min":1.256},"last":0.59199999999999997}]},{"osd":7,"last update":"Fri May 13 23:14:49 2022","interfaces":[{"interface":"back","average":{"1min":0.87,"5min":0.90600000000000003,"15min":0.88600000000000001},"min":{"1min":0.58699999999999997,"5min":0.58699999999999997,"15min":0.58699999999999997},"max":{"1min":1.204,"5min":1.3280000000000001,"15min":1.3280000000000001},"last":0.94299999999999995},{"interface":"front","average":{"1min":0.90400000000000003,"5min":0.90000000000000002,"15min":0.871},"min":{"1min":0.623,"5min":0.623,"15min":0.623},"max":{"1min":1.2689999999999999,"5min":1.3480000000000001,"15min":1.3480000000000001},"last":0.94799999999999995}]},{"osd":8,"last update":"Fri May 13 23:14:07 2022","interfaces":[{"interface":"back","average":{"1min":0.91700000000000004,"5min":0.94699999999999995,"15min":0.95199999999999996},"min":{"1min":0.56699999999999995,"5min":0.56699999999999995,"15min":0.56699999999999995},"max":{"1min":1.2589999999999999,"5min":1.2669999999999999,"15min":1.2669999999999999},"last":1.048},{"interface":"front","average":{"1min":0.98899999999999999,"5min":1.002,"15min":1.004},"min":{"1min":0.58699999999999997,"5min":0.58699999999999997,"15min":0.58699999999999997},"max":{"1min":1.244,"5min":1.244,"15min":1.244},"last":0.91500000000000004}]},{"osd":9,"last update":"Fri May 13 23:14:40 2022","interfaces":[{"interface":"back","average":{"1min":0.97999999999999998,"5min":1.1000000000000001,"15min":1.1200000000000001},"min":{"1min":0.72099999999999997,"5min":0.72099999999999997,"15min":0.72099999999999997},"max":{"1min":1.2470000000000001,"5min":1.4319999999999999,"15min":1.4319999999999999},"last":0.80300000000000005},{"interface":"front","average":{"1min":0.92500000000000004,"5min":1.0589999999999999,"15min":1.081},"min":{"1min":0.71399999999999997,"5min":0.71399999999999997,"15min":0.71399999999999997},"max":{"1min":1.2210000000000001,"5min":1.4730000000000001,"15min":1.4730000000000001},"last":0.85899999999999999}]},{"osd":10,"last update":"Fri May 13 23:15:01 2022","interfaces":[{"interface":"back","average":{"1min":0.87,"5min":0.98399999999999999,"15min":1.0029999999999999},"min":{"1min":0.65100000000000002,"5min":0.52300000000000002,"15min":0.52300000000000002},"max":{"1min":1.2290000000000001,"5min":1.373,"15min":1.373},"last":0.92700000000000005},{"interface":"front","average":{"1min":0.97099999999999997,"5min":1.095,"15min":1.1160000000000001},"min":{"1min":0.66000000000000003,"5min":0.66000000000000003,"15min":0.66000000000000003},"max":{"1min":1.405,"5min":1.51,"15min":1.51},"last":0.96599999999999997}]},{"osd":11,"last update":"Fri May 13 23:15:16 2022","interfaces":[{"interface":"back","average":{"1min":0.91600000000000004,"5min":0.91600000000000004,"15min":0.91600000000000004},"min":{"1min":0.749,"5min":0.749,"15min":0.749},"max":{"1min":1.252,"5min":1.252,"15min":1.252},"last":0.88300000000000001},{"interface":"front","average":{"1min":0.99399999999999999,"5min":0.99399999999999999,"15min":0.99399999999999999},"min":{"1min":0.68899999999999995,"5min":0.68899999999999995,"15min":0.68899999999999995},"max":{"1min":1.4470000000000001,"5min":1.4470000000000001,"15min":1.4470000000000001},"last":0.98699999999999999}]}]},{"osd":4,"up_from":33,"seq":141733920829,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6360,"kb_used_data":912,"kb_used_omap":0,"kb_used_meta":5440,"kb_avail":93738792,"statfs":{"total":95995035648,"available":95988523008,"internally_reserved":0,"allocated":933888,"data_stored":442259,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5570560},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri May 13 23:13:35 2022","interfaces":[{"interface":"back","average":{"1min":0.627,"5min":0.56299999999999994,"15min":0.55000000000000004},"min":{"1min":0.39700000000000002,"5min":0.375,"15min":0.375},"max":{"1min":0.91800000000000004,"5min":0.91800000000000004,"15min":0.91800000000000004},"last":0.56000000000000005},{"interface":"front","average":{"1min":0.70499999999999996,"5min":0.58099999999999996,"15min":0.54500000000000004},"min":{"1min":0.47999999999999998,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.97099999999999997,"5min":0.97099999999999997,"15min":0.97099999999999997},"last":0.65200000000000002}]},{"osd":1,"last update":"Fri May 13 23:14:36 2022","interfaces":[{"interface":"back","average":{"1min":0.45300000000000001,"5min":0.40999999999999998,"15min":0.378},"min":{"1min":0.28999999999999998,"5min":0.217,"15min":0.217},"max":{"1min":0.81100000000000005,"5min":0.84499999999999997,"15min":0.84499999999999997},"last":0.39300000000000002},{"interface":"front","average":{"1min":0.48399999999999999,"5min":0.42299999999999999,"15min":0.39700000000000002},"min":{"1min":0.29599999999999999,"5min":0.248,"15min":0.248},"max":{"1min":0.622,"5min":0.79900000000000004,"15min":0.79900000000000004},"last":0.42099999999999999}]},{"osd":2,"last update":"Fri May 13 23:14:36 2022","interfaces":[{"interface":"back","average":{"1min":0.64500000000000002,"5min":0.56999999999999995,"15min":0.54700000000000004},"min":{"1min":0.45000000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":1.0640000000000001,"5min":1.0640000000000001,"15min":1.0640000000000001},"last":0.498},{"interface":"front","average":{"1min":0.63800000000000001,"5min":0.59399999999999997,"15min":0.56999999999999995},"min":{"1min":0.49099999999999999,"5min":0.372,"15min":0.372},"max":{"1min":0.89800000000000002,"5min":0.92900000000000005,"15min":0.92900000000000005},"last":0.51100000000000001}]},{"osd":3,"last update":"Fri May 13 23:14:36 2022","interfaces":[{"interface":"back","average":{"1min":0.68200000000000005,"5min":0.629,"15min":0.59399999999999997},"min":{"1min":0.49399999999999999,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.84999999999999998,"5min":0.91600000000000004,"15min":0.91600000000000004},"last":0.66900000000000004},{"interface":"front","average":{"1min":0.69899999999999995,"5min":0.65600000000000003,"15min":0.61899999999999999},"min":{"1min":0.441,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":1.1579999999999999,"5min":1.1579999999999999,"15min":1.1579999999999999},"last":0.44900000000000001}]},{"osd":5,"last update":"Fri May 13 23:14:59 2022","interfaces":[{"interface":"back","average":{"1min":0.66300000000000003,"5min":0.61299999999999999,"15min":0.56100000000000005},"min":{"1min":0.42799999999999999,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.96999999999999997,"5min":0.97999999999999998,"15min":0.97999999999999998},"last":0.63200000000000001},{"interface":"front","average":{"1min":0.68899999999999995,"5min":0.61599999999999999,"15min":0.56599999999999995},"min":{"1min":0.40600000000000003,"5min":0.371,"15min":0.371},"max":{"1min":0.94299999999999995,"5min":1.1419999999999999,"15min":1.1419999999999999},"last":0.57899999999999996}]},{"osd":6,"last update":"Fri May 13 23:14:25 2022","interfaces":[{"interface":"back","average":{"1min":0.72899999999999998,"5min":0.69399999999999995,"15min":0.66500000000000004},"min":{"1min":0.502,"5min":0.498,"15min":0.498},"max":{"1min":1.1839999999999999,"5min":1.1839999999999999,"15min":1.1839999999999999},"last":0.61499999999999999},{"interface":"front","average":{"1min":0.74199999999999999,"5min":0.72899999999999998,"15min":0.70199999999999996},"min":{"1min":0.47499999999999998,"5min":0.47499999999999998,"15min":0.47499999999999998},"max":{"1min":0.97199999999999998,"5min":1.046,"15min":1.046},"last":0.77600000000000002}]},{"osd":7,"last update":"Fri May 13 23:14:53 2022","interfaces":[{"interface":"back","average":{"1min":0.63900000000000001,"5min":0.66300000000000003,"15min":0.65300000000000002},"min":{"1min":0.39700000000000002,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.85299999999999998,"5min":0.98599999999999999,"15min":0.98599999999999999},"last":0.72999999999999998},{"interface":"front","average":{"1min":0.66500000000000004,"5min":0.65300000000000002,"15min":0.63600000000000001},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":1.0409999999999999,"5min":1.0409999999999999,"15min":1.0409999999999999},"last":0.72099999999999997}]},{"osd":8,"last update":"Fri May 13 23:15:15 2022","interfaces":[{"interface":"back","average":{"1min":0.79500000000000004,"5min":0.76900000000000002,"15min":0.753},"min":{"1min":0.59699999999999998,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":0.95599999999999996,"5min":1.099,"15min":1.099},"last":0.82499999999999996},{"interface":"front","average":{"1min":0.82799999999999996,"5min":0.76900000000000002,"15min":0.75800000000000001},"min":{"1min":0.59299999999999997,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":0.96599999999999997,"5min":1.115,"15min":1.115},"last":0.83599999999999997}]},{"osd":9,"last update":"Fri May 13 23:14:45 2022","interfaces":[{"interface":"back","average":{"1min":0.81000000000000005,"5min":0.874,"15min":0.88400000000000001},"min":{"1min":0.51900000000000002,"5min":0.51900000000000002,"15min":0.51900000000000002},"max":{"1min":1.069,"5min":1.1819999999999999,"15min":1.1819999999999999},"last":0.80300000000000005},{"interface":"front","average":{"1min":0.78900000000000003,"5min":0.85299999999999998,"15min":0.86299999999999999},"min":{"1min":0.61499999999999999,"5min":0.61499999999999999,"15min":0.61499999999999999},"max":{"1min":1.216,"5min":1.216,"15min":1.216},"last":0.69099999999999995}]},{"osd":10,"last update":"Fri May 13 23:15:03 2022","interfaces":[{"interface":"back","average":{"1min":0.80200000000000005,"5min":0.80600000000000005,"15min":0.80600000000000005},"min":{"1min":0.5,"5min":0.5,"15min":0.5},"max":{"1min":1.1120000000000001,"5min":1.1120000000000001,"15min":1.1120000000000001},"last":0.75900000000000001},{"interface":"front","average":{"1min":0.71599999999999997,"5min":0.78400000000000003,"15min":0.79600000000000004},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":1.073,"5min":1.196,"15min":1.196},"last":0.749}]},{"osd":11,"last update":"Fri May 13 23:15:15 2022","interfaces":[{"interface":"back","average":{"1min":0.83299999999999996,"5min":0.83299999999999996,"15min":0.83299999999999996},"min":{"1min":0.63200000000000001,"5min":0.63200000000000001,"15min":0.63200000000000001},"max":{"1min":1.0329999999999999,"5min":1.0329999999999999,"15min":1.0329999999999999},"last":0.81000000000000005},{"interface":"front","average":{"1min":0.86099999999999999,"5min":0.86099999999999999,"15min":0.86099999999999999},"min":{"1min":0.71299999999999997,"5min":0.71299999999999997,"15min":0.71299999999999997},"max":{"1min":1.081,"5min":1.081,"15min":1.081},"last":0.88300000000000001}]}]},{"osd":5,"up_from":38,"seq":163208757304,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":7028,"kb_used_data":1452,"kb_used_omap":0,"kb_used_meta":5568,"kb_avail":93738124,"statfs":{"total":95995035648,"available":95987838976,"internally_reserved":0,"allocated":1486848,"data_stored":972900,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5701632},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri May 13 23:14:51 2022","interfaces":[{"interface":"back","average":{"1min":0.59199999999999997,"5min":0.55200000000000005,"15min":0.52800000000000002},"min":{"1min":0.42399999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.83399999999999996,"5min":0.96299999999999997,"15min":0.96299999999999997},"last":0.42199999999999999},{"interface":"front","average":{"1min":0.59099999999999997,"5min":0.55700000000000005,"15min":0.52800000000000002},"min":{"1min":0.40300000000000002,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.85099999999999998,"5min":0.96499999999999997,"15min":0.96499999999999997},"last":0.434}]},{"osd":1,"last update":"Fri May 13 23:14:51 2022","interfaces":[{"interface":"back","average":{"1min":0.58599999999999997,"5min":0.52500000000000002,"15min":0.496},"min":{"1min":0.36699999999999999,"5min":0.317,"15min":0.317},"max":{"1min":0.77500000000000002,"5min":0.77500000000000002,"15min":0.77500000000000002},"last":0.64200000000000002},{"interface":"front","average":{"1min":0.58499999999999996,"5min":0.52600000000000002,"15min":0.48799999999999999},"min":{"1min":0.36499999999999999,"5min":0.313,"15min":0.313},"max":{"1min":0.78800000000000003,"5min":0.78800000000000003,"15min":0.78800000000000003},"last":0.63200000000000001}]},{"osd":2,"last update":"Fri May 13 23:14:51 2022","interfaces":[{"interface":"back","average":{"1min":0.432,"5min":0.39600000000000002,"15min":0.36799999999999999},"min":{"1min":0.28599999999999998,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":0.54500000000000004,"5min":0.65600000000000003,"15min":0.65600000000000003},"last":0.50800000000000001},{"interface":"front","average":{"1min":0.439,"5min":0.40899999999999997,"15min":0.38},"min":{"1min":0.28100000000000003,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.69699999999999995,"5min":0.75700000000000001,"15min":0.75700000000000001},"last":0.55600000000000005}]},{"osd":3,"last update":"Fri May 13 23:14:51 2022","interfaces":[{"interface":"back","average":{"1min":0.60699999999999998,"5min":0.57499999999999996,"15min":0.56200000000000006},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":0.95899999999999996,"5min":0.97999999999999998,"15min":0.97999999999999998},"last":0.71399999999999997},{"interface":"front","average":{"1min":0.65500000000000003,"5min":0.59999999999999998,"15min":0.56000000000000005},"min":{"1min":0.47099999999999997,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":0.89100000000000001,"5min":1,"15min":1},"last":0.69299999999999995}]},{"osd":4,"last update":"Fri May 13 23:14:51 2022","interfaces":[{"interface":"back","average":{"1min":0.624,"5min":0.57699999999999996,"15min":0.54300000000000004},"min":{"1min":0.45800000000000002,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.82599999999999996,"5min":0.86399999999999999,"15min":0.86399999999999999},"last":0.755},{"interface":"front","average":{"1min":0.58499999999999996,"5min":0.55300000000000005,"15min":0.53500000000000003},"min":{"1min":0.41299999999999998,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.80000000000000004,"5min":0.89500000000000002,"15min":0.89500000000000002},"last":0.77000000000000002}]},{"osd":6,"last update":"Fri May 13 23:14:25 2022","interfaces":[{"interface":"back","average":{"1min":0.66200000000000003,"5min":0.63,"15min":0.628},"min":{"1min":0.48399999999999999,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":0.89800000000000002,"5min":0.95399999999999996,"15min":0.95399999999999996},"last":0.68100000000000005},{"interface":"front","average":{"1min":0.71599999999999997,"5min":0.64500000000000002,"15min":0.628},"min":{"1min":0.503,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":1.026,"5min":1.026,"15min":1.026},"last":0.73599999999999999}]},{"osd":7,"last update":"Fri May 13 23:14:46 2022","interfaces":[{"interface":"back","average":{"1min":0.68700000000000006,"5min":0.64900000000000002,"15min":0.64500000000000002},"min":{"1min":0.51200000000000001,"5min":0.379,"15min":0.379},"max":{"1min":0.92700000000000005,"5min":1.0489999999999999,"15min":1.0489999999999999},"last":0.78400000000000003},{"interface":"front","average":{"1min":0.70699999999999996,"5min":0.65800000000000003,"15min":0.64600000000000002},"min":{"1min":0.53200000000000003,"5min":0.442,"15min":0.442},"max":{"1min":0.96699999999999997,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.79300000000000004}]},{"osd":8,"last update":"Fri May 13 23:15:07 2022","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.60999999999999999,"15min":0.62},"min":{"1min":0.36899999999999999,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.90300000000000002,"5min":0.999,"15min":0.999},"last":0.66700000000000004},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.56399999999999995,"15min":0.56799999999999995},"min":{"1min":0.38400000000000001,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.72399999999999998,"5min":0.82699999999999996,"15min":0.82699999999999996},"last":0.59599999999999997}]},{"osd":9,"last update":"Fri May 13 23:14:44 2022","interfaces":[{"interface":"back","average":{"1min":0.73399999999999999,"5min":0.69499999999999995,"15min":0.68899999999999995},"min":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":0.90200000000000002,"5min":0.90200000000000002,"15min":0.90200000000000002},"last":0.80700000000000005},{"interface":"front","average":{"1min":0.77400000000000002,"5min":0.72399999999999998,"15min":0.71599999999999997},"min":{"1min":0.55000000000000004,"5min":0.505,"15min":0.505},"max":{"1min":0.92900000000000005,"5min":0.99299999999999999,"15min":0.99299999999999999},"last":0.82599999999999996}]},{"osd":10,"last update":"Fri May 13 23:15:02 2022","interfaces":[{"interface":"back","average":{"1min":0.67800000000000005,"5min":0.67800000000000005,"15min":0.67800000000000005},"min":{"1min":0.496,"5min":0.496,"15min":0.496},"max":{"1min":0.84799999999999998,"5min":0.90800000000000003,"15min":0.90800000000000003},"last":0.83699999999999997},{"interface":"front","average":{"1min":0.72499999999999998,"5min":0.71599999999999997,"15min":0.71399999999999997},"min":{"1min":0.54900000000000004,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":0.89000000000000001,"5min":0.94499999999999995,"15min":0.94499999999999995},"last":0.84899999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2074,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":3184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":3316,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":2688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":674,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":2942,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1422,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1094,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":3196,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1206,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2022-05-13T23:15:23.009 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2022-05-13T23:15:23.009 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2022-05-13T23:15:23.009 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2022-05-13T23:15:23.010 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:23b0658391ce94ca40799469eec00c28b298816c shell --fsid 16cf5f88-d311-11ec-be40-001a4aab830c -- ceph health --format=json 2022-05-13T23:15:23.723 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/16cf5f88-d311-11ec-be40-001a4aab830c/mon.a/config 2022-05-13T23:15:23.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:15:23 smithi047 ceph-mon[77147]: from='client.14793 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2022-05-13T23:15:23.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:15:23 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:15:23.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:15:23 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:15:23.823 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:15:23 smithi002 ceph-mon[71816]: from='client.14793 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2022-05-13T23:15:23.823 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:15:23 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:15:23.824 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:15:23 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:15:23.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:15:23 smithi115 ceph-mon[78593]: from='client.14793 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2022-05-13T23:15:23.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:15:23 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:15:23.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:15:23 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:15:24.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:15:24 smithi047 ceph-mon[77147]: pgmap v324: 97 pgs: 97 active+clean; 584 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2022-05-13T23:15:24.822 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:15:24 smithi002 ceph-mon[71816]: pgmap v324: 97 pgs: 97 active+clean; 584 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2022-05-13T23:15:24.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:15:24 smithi115 ceph-mon[78593]: pgmap v324: 97 pgs: 97 active+clean; 584 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2022-05-13T23:15:25.539 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:25.536+0000 7fe96966c700 1 -- 172.21.15.2:0/2614603833 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe96406edb0 msgr2=0x7fe96406f220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:15:25.540 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:25.536+0000 7fe96966c700 1 --2- 172.21.15.2:0/2614603833 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe96406edb0 0x7fe96406f220 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fe958004870 tx=0x7fe958009cb0 comp rx=0 tx=0).stop 2022-05-13T23:15:25.540 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:25.536+0000 7fe96966c700 1 -- 172.21.15.2:0/2614603833 shutdown_connections 2022-05-13T23:15:25.540 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:25.536+0000 7fe96966c700 1 --2- 172.21.15.2:0/2614603833 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe964103580 0x7fe964105960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:25.540 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:25.536+0000 7fe96966c700 1 --2- 172.21.15.2:0/2614603833 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe964070370 0x7fe96406e870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:25.541 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:25.536+0000 7fe96966c700 1 --2- 172.21.15.2:0/2614603833 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe96406edb0 0x7fe96406f220 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:25.541 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:25.536+0000 7fe96966c700 1 -- 172.21.15.2:0/2614603833 >> 172.21.15.2:0/2614603833 conn(0x7fe9640f6e00 msgr2=0x7fe9640f9210 unknown :-1 s=STATE_NONE l=0).mark_down 2022-05-13T23:15:25.543 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:25.537+0000 7fe96966c700 1 -- 172.21.15.2:0/2614603833 shutdown_connections 2022-05-13T23:15:25.544 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:25.537+0000 7fe96966c700 1 -- 172.21.15.2:0/2614603833 wait complete. 2022-05-13T23:15:25.546 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:25.538+0000 7fe96966c700 1 Processor -- start 2022-05-13T23:15:25.546 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:25.538+0000 7fe96966c700 1 -- start start 2022-05-13T23:15:25.546 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:25.539+0000 7fe96966c700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe96406edb0 0x7fe9640fea20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:15:25.547 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:25.539+0000 7fe96966c700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe964070370 0x7fe9640fef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:15:25.547 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:25.539+0000 7fe96966c700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe964103580 0x7fe9640fb560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:15:25.547 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:25.539+0000 7fe96966c700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9640fbc70 con 0x7fe96406edb0 2022-05-13T23:15:25.547 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:25.539+0000 7fe96966c700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9640fbde0 con 0x7fe964070370 2022-05-13T23:15:25.548 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:25.539+0000 7fe96966c700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9640fbfa0 con 0x7fe964103580 2022-05-13T23:15:25.548 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:25.539+0000 7fe962ffd700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe96406edb0 0x7fe9640fea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:15:25.548 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:25.539+0000 7fe962ffd700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe96406edb0 0x7fe9640fea20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:46036/0 (socket says 172.21.15.2:46036) 2022-05-13T23:15:25.548 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:25.539+0000 7fe9637fe700 1 --2- 172.21.15.2:0/3439176320 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe964103580 0x7fe9640fb560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:15:25.548 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:25.539+0000 7fe962ffd700 1 -- 172.21.15.2:0/3439176320 learned_addr learned my addr 172.21.15.2:0/3439176320 (peer_addr_for_me v2:172.21.15.2:0/0) 2022-05-13T23:15:25.549 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:25.539+0000 7fe9627fc700 1 --2- 172.21.15.2:0/3439176320 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe964070370 0x7fe9640fef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:15:25.549 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:25.540+0000 7fe9637fe700 1 -- 172.21.15.2:0/3439176320 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe964070370 msgr2=0x7fe9640fef60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:15:25.549 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:25.540+0000 7fe9637fe700 1 --2- 172.21.15.2:0/3439176320 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe964070370 0x7fe9640fef60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:25.549 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:25.540+0000 7fe9637fe700 1 -- 172.21.15.2:0/3439176320 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe96406edb0 msgr2=0x7fe9640fea20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:15:25.550 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:25.540+0000 7fe9637fe700 1 --2- 172.21.15.2:0/3439176320 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe96406edb0 0x7fe9640fea20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:25.550 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:25.540+0000 7fe9637fe700 1 -- 172.21.15.2:0/3439176320 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe954005040 con 0x7fe964103580 2022-05-13T23:15:25.550 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:25.540+0000 7fe9637fe700 1 --2- 172.21.15.2:0/3439176320 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe964103580 0x7fe9640fb560 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fe964070140 tx=0x7fe95c00b040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-05-13T23:15:25.550 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:25.540+0000 7fe94b7fe700 1 -- 172.21.15.2:0/3439176320 <== mon.2 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe95c009870 con 0x7fe964103580 2022-05-13T23:15:25.550 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:25.540+0000 7fe96966c700 1 -- 172.21.15.2:0/3439176320 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe958005040 con 0x7fe964103580 2022-05-13T23:15:25.551 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:25.540+0000 7fe94b7fe700 1 -- 172.21.15.2:0/3439176320 <== mon.2 v2:172.21.15.115:3300/0 2 ==== config(26 keys) v1 ==== 1087+0+0 (secure 0 0 0) 0x7fe95c00f040 con 0x7fe964103580 2022-05-13T23:15:25.551 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:25.540+0000 7fe94b7fe700 1 -- 172.21.15.2:0/3439176320 <== mon.2 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe95c007e60 con 0x7fe964103580 2022-05-13T23:15:25.551 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:25.541+0000 7fe96966c700 1 -- 172.21.15.2:0/3439176320 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe96411ae90 con 0x7fe964103580 2022-05-13T23:15:25.552 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:25.542+0000 7fe96966c700 1 -- 172.21.15.2:0/3439176320 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe964108fc0 con 0x7fe964103580 2022-05-13T23:15:25.553 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:25.545+0000 7fe94b7fe700 1 -- 172.21.15.2:0/3439176320 <== mon.2 v2:172.21.15.115:3300/0 4 ==== mgrmap(e 16) v1 ==== 125136+0+0 (secure 0 0 0) 0x7fe95c01a020 con 0x7fe964103580 2022-05-13T23:15:25.553 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:25.547+0000 7fe94b7fe700 1 --2- 172.21.15.2:0/3439176320 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7fe94c096ce0 0x7fe94c099190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:15:25.553 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:25.547+0000 7fe94b7fe700 1 -- 172.21.15.2:0/3439176320 <== mon.2 v2:172.21.15.115:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8120+0+0 (secure 0 0 0) 0x7fe95c0bebf0 con 0x7fe964103580 2022-05-13T23:15:25.554 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:25.547+0000 7fe962ffd700 1 --2- 172.21.15.2:0/3439176320 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7fe94c096ce0 0x7fe94c099190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:15:25.554 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:25.547+0000 7fe94b7fe700 1 -- 172.21.15.2:0/3439176320 <== mon.2 v2:172.21.15.115:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+165215 (secure 0 0 0) 0x7fe95c08bf90 con 0x7fe964103580 2022-05-13T23:15:25.554 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:25.548+0000 7fe962ffd700 1 --2- 172.21.15.2:0/3439176320 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7fe94c096ce0 0x7fe94c099190 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fe954009410 tx=0x7fe95400b040 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-05-13T23:15:25.792 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:25.790+0000 7fe96966c700 1 -- 172.21.15.2:0/3439176320 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fe964109150 con 0x7fe964103580 2022-05-13T23:15:25.793 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:25.790+0000 7fe94b7fe700 1 -- 172.21.15.2:0/3439176320 <== mon.2 v2:172.21.15.115:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+46 (secure 0 0 0) 0x7fe964109150 con 0x7fe964103580 2022-05-13T23:15:25.793 INFO:teuthology.orchestra.run.smithi002.stdout: 2022-05-13T23:15:25.794 INFO:teuthology.orchestra.run.smithi002.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2022-05-13T23:15:25.796 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:25.793+0000 7fe96966c700 1 -- 172.21.15.2:0/3439176320 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7fe94c096ce0 msgr2=0x7fe94c099190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:15:25.796 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:25.793+0000 7fe96966c700 1 --2- 172.21.15.2:0/3439176320 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7fe94c096ce0 0x7fe94c099190 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fe954009410 tx=0x7fe95400b040 comp rx=0 tx=0).stop 2022-05-13T23:15:25.797 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:25.793+0000 7fe96966c700 1 -- 172.21.15.2:0/3439176320 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe964103580 msgr2=0x7fe9640fb560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:15:25.797 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:25.793+0000 7fe96966c700 1 --2- 172.21.15.2:0/3439176320 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe964103580 0x7fe9640fb560 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fe964070140 tx=0x7fe95c00b040 comp rx=0 tx=0).stop 2022-05-13T23:15:25.797 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:25.793+0000 7fe96966c700 1 -- 172.21.15.2:0/3439176320 shutdown_connections 2022-05-13T23:15:25.797 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:25.793+0000 7fe96966c700 1 --2- 172.21.15.2:0/3439176320 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe964103580 0x7fe9640fb560 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:25.797 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:25.793+0000 7fe96966c700 1 --2- 172.21.15.2:0/3439176320 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7fe94c096ce0 0x7fe94c099190 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:25.798 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:25.793+0000 7fe96966c700 1 --2- 172.21.15.2:0/3439176320 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe96406edb0 0x7fe9640fea20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:25.798 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:25.793+0000 7fe96966c700 1 --2- 172.21.15.2:0/3439176320 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe964070370 0x7fe9640fef60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:25.798 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:25.793+0000 7fe96966c700 1 -- 172.21.15.2:0/3439176320 >> 172.21.15.2:0/3439176320 conn(0x7fe9640f6e00 msgr2=0x7fe9640f91e0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-05-13T23:15:25.799 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:25.793+0000 7fe96966c700 1 -- 172.21.15.2:0/3439176320 shutdown_connections 2022-05-13T23:15:25.799 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:25.793+0000 7fe96966c700 1 -- 172.21.15.2:0/3439176320 wait complete. 2022-05-13T23:15:26.295 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2022-05-13T23:15:26.296 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-05-13T23:15:26.529 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:26.515+0000 7fbd6e39d700 1 -- 172.21.15.2:0/1533450887 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fbd680fb630 msgr2=0x7fbd680fba80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:15:26.529 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:26.515+0000 7fbd6e39d700 1 --2- 172.21.15.2:0/1533450887 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fbd680fb630 0x7fbd680fba80 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fbd58004870 tx=0x7fbd58009cb0 comp rx=0 tx=0).stop 2022-05-13T23:15:26.529 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:26.515+0000 7fbd6e39d700 1 -- 172.21.15.2:0/1533450887 shutdown_connections 2022-05-13T23:15:26.530 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:26.515+0000 7fbd6e39d700 1 --2- 172.21.15.2:0/1533450887 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fbd680fbfc0 0x7fbd68102880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:26.530 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:26.515+0000 7fbd6e39d700 1 --2- 172.21.15.2:0/1533450887 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fbd680fa390 0x7fbd680fa7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:26.530 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:26.515+0000 7fbd6e39d700 1 --2- 172.21.15.2:0/1533450887 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fbd680fb630 0x7fbd680fba80 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:26.530 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:26.515+0000 7fbd6e39d700 1 -- 172.21.15.2:0/1533450887 >> 172.21.15.2:0/1533450887 conn(0x7fbd680f5b80 msgr2=0x7fbd680f7fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-05-13T23:15:26.531 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:26.515+0000 7fbd6e39d700 1 -- 172.21.15.2:0/1533450887 shutdown_connections 2022-05-13T23:15:26.531 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:26.515+0000 7fbd6e39d700 1 -- 172.21.15.2:0/1533450887 wait complete. 2022-05-13T23:15:26.531 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:26.516+0000 7fbd6e39d700 1 Processor -- start 2022-05-13T23:15:26.531 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:26.516+0000 7fbd6e39d700 1 -- start start 2022-05-13T23:15:26.531 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:26.517+0000 7fbd6e39d700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fbd680fa390 0x7fbd6810e4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:15:26.532 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:26.517+0000 7fbd6e39d700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fbd680fb630 0x7fbd6810e9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:15:26.532 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:26.517+0000 7fbd6e39d700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fbd680fbfc0 0x7fbd68115a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:15:26.532 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:26.517+0000 7fbd6e39d700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd68115fb0 con 0x7fbd680fa390 2022-05-13T23:15:26.532 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:26.517+0000 7fbd6e39d700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd68116120 con 0x7fbd680fbfc0 2022-05-13T23:15:26.533 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:26.517+0000 7fbd6e39d700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd681162e0 con 0x7fbd680fb630 2022-05-13T23:15:26.533 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:26.517+0000 7fbd6d39b700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fbd680fa390 0x7fbd6810e4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:15:26.533 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:26.517+0000 7fbd6d39b700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fbd680fa390 0x7fbd6810e4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:46056/0 (socket says 172.21.15.2:46056) 2022-05-13T23:15:26.533 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:26.517+0000 7fbd6d39b700 1 -- 172.21.15.2:0/17743713 learned_addr learned my addr 172.21.15.2:0/17743713 (peer_addr_for_me v2:172.21.15.2:0/0) 2022-05-13T23:15:26.533 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:26.517+0000 7fbd6cb9a700 1 --2- 172.21.15.2:0/17743713 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fbd680fb630 0x7fbd6810e9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:15:26.534 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:26.517+0000 7fbd6db9c700 1 --2- 172.21.15.2:0/17743713 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fbd680fbfc0 0x7fbd68115a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:15:26.534 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:26.518+0000 7fbd6d39b700 1 -- 172.21.15.2:0/17743713 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fbd680fb630 msgr2=0x7fbd6810e9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:15:26.534 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:26.518+0000 7fbd6d39b700 1 --2- 172.21.15.2:0/17743713 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fbd680fb630 0x7fbd6810e9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:26.534 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:26.518+0000 7fbd6d39b700 1 -- 172.21.15.2:0/17743713 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fbd680fbfc0 msgr2=0x7fbd68115a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:15:26.535 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:26.518+0000 7fbd6d39b700 1 --2- 172.21.15.2:0/17743713 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fbd680fbfc0 0x7fbd68115a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:26.535 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:26.518+0000 7fbd6d39b700 1 -- 172.21.15.2:0/17743713 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd58005040 con 0x7fbd680fa390 2022-05-13T23:15:26.535 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:26.518+0000 7fbd6db9c700 1 --2- 172.21.15.2:0/17743713 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fbd680fbfc0 0x7fbd68115a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-05-13T23:15:26.536 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:26.518+0000 7fbd6d39b700 1 --2- 172.21.15.2:0/17743713 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fbd680fa390 0x7fbd6810e4a0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fbd6400e940 tx=0x7fbd6400b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-05-13T23:15:26.536 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:26.518+0000 7fbd5dffb700 1 -- 172.21.15.2:0/17743713 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd64007b40 con 0x7fbd680fa390 2022-05-13T23:15:26.536 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:26.518+0000 7fbd5dffb700 1 -- 172.21.15.2:0/17743713 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1087+0+0 (secure 0 0 0) 0x7fbd6400f040 con 0x7fbd680fa390 2022-05-13T23:15:26.536 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:26.518+0000 7fbd6e39d700 1 -- 172.21.15.2:0/17743713 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbd68116560 con 0x7fbd680fa390 2022-05-13T23:15:26.537 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:26.518+0000 7fbd5dffb700 1 -- 172.21.15.2:0/17743713 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd64014890 con 0x7fbd680fa390 2022-05-13T23:15:26.537 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:26.518+0000 7fbd6e39d700 1 -- 172.21.15.2:0/17743713 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbd68116b20 con 0x7fbd680fa390 2022-05-13T23:15:26.537 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:26.519+0000 7fbd6e39d700 1 -- 172.21.15.2:0/17743713 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbd68105bd0 con 0x7fbd680fa390 2022-05-13T23:15:26.537 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:26.521+0000 7fbd5dffb700 1 -- 172.21.15.2:0/17743713 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 16) v1 ==== 125136+0+0 (secure 0 0 0) 0x7fbd6401b030 con 0x7fbd680fa390 2022-05-13T23:15:26.539 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:26.522+0000 7fbd5dffb700 1 --2- 172.21.15.2:0/17743713 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7fbd54096d60 0x7fbd54099210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:15:26.540 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:26.523+0000 7fbd5dffb700 1 -- 172.21.15.2:0/17743713 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8120+0+0 (secure 0 0 0) 0x7fbd640beb00 con 0x7fbd680fa390 2022-05-13T23:15:26.540 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:26.523+0000 7fbd6cb9a700 1 --2- 172.21.15.2:0/17743713 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7fbd54096d60 0x7fbd54099210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:15:26.540 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:26.523+0000 7fbd6cb9a700 1 --2- 172.21.15.2:0/17743713 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7fbd54096d60 0x7fbd54099210 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fbd58004840 tx=0x7fbd58009cb0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-05-13T23:15:26.540 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:26.527+0000 7fbd5dffb700 1 -- 172.21.15.2:0/17743713 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+165215 (secure 0 0 0) 0x7fbd6404b570 con 0x7fbd680fa390 2022-05-13T23:15:26.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:15:26 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:15:26.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:15:26 smithi047 ceph-mon[77147]: pgmap v325: 97 pgs: 97 active+clean; 584 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 11 op/s 2022-05-13T23:15:26.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:15:26 smithi047 ceph-mon[77147]: from='client.? 172.21.15.2:0/3439176320' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2022-05-13T23:15:26.572 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:15:26 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:15:26.573 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:15:26 smithi002 ceph-mon[71816]: pgmap v325: 97 pgs: 97 active+clean; 584 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 11 op/s 2022-05-13T23:15:26.573 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:15:26 smithi002 ceph-mon[71816]: from='client.? 172.21.15.2:0/3439176320' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2022-05-13T23:15:26.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:15:26 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:15:26.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:15:26 smithi115 ceph-mon[78593]: pgmap v325: 97 pgs: 97 active+clean; 584 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 11 op/s 2022-05-13T23:15:26.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:15:26 smithi115 ceph-mon[78593]: from='client.? 172.21.15.2:0/3439176320' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2022-05-13T23:15:26.722 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:26.718+0000 7fbd6e39d700 1 -- 172.21.15.2:0/17743713 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbd681171a0 con 0x7fbd680fa390 2022-05-13T23:15:26.722 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:26.719+0000 7fbd5dffb700 1 -- 172.21.15.2:0/17743713 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v85) v1 ==== 74+0+21505 (secure 0 0 0) 0x7fbd640c0e60 con 0x7fbd680fa390 2022-05-13T23:15:26.723 INFO:teuthology.orchestra.run.smithi002.stdout: 2022-05-13T23:15:26.723 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":85,"fsid":"16cf5f88-d311-11ec-be40-001a4aab830c","created":"2022-05-13T23:06:00.765453+0000","modified":"2022-05-13T23:15:06.402793+0000","last_up_change":"2022-05-13T23:13:20.935982+0000","last_in_change":"2022-05-13T23:13:02.796000+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2022-05-13T23:09:32.000644+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2022-05-13T23:14:14.195678+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":3,"pool_name":"cephfs_data","create_time":"2022-05-13T23:14:15.659212+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"83","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"11e700e8-f3e8-4903-b52c-db354b4395bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6802","nonce":1174843704},{"type":"v1","addr":"172.21.15.2:6803","nonce":1174843704}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6804","nonce":1174843704},{"type":"v1","addr":"172.21.15.2:6805","nonce":1174843704}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6808","nonce":1174843704},{"type":"v1","addr":"172.21.15.2:6809","nonce":1174843704}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6806","nonce":1174843704},{"type":"v1","addr":"172.21.15.2:6807","nonce":1174843704}]},"public_addr":"172.21.15.2:6803/1174843704","cluster_addr":"172.21.15.2:6805/1174843704","heartbeat_back_addr":"172.21.15.2:6809/1174843704","heartbeat_front_addr":"172.21.15.2:6807/1174843704","state":["exists","up"]},{"osd":1,"uuid":"eabfa3cf-f2a3-4995-a7de-f6d8ba02ee6f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6800","nonce":3853826948},{"type":"v1","addr":"172.21.15.47:6801","nonce":3853826948}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6802","nonce":3853826948},{"type":"v1","addr":"172.21.15.47:6803","nonce":3853826948}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6806","nonce":3853826948},{"type":"v1","addr":"172.21.15.47:6807","nonce":3853826948}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6804","nonce":3853826948},{"type":"v1","addr":"172.21.15.47:6805","nonce":3853826948}]},"public_addr":"172.21.15.47:6801/3853826948","cluster_addr":"172.21.15.47:6803/3853826948","heartbeat_back_addr":"172.21.15.47:6807/3853826948","heartbeat_front_addr":"172.21.15.47:6805/3853826948","state":["exists","up"]},{"osd":2,"uuid":"153c5bd5-8200-47a7-a135-8a95b6752cd2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6800","nonce":2562256956},{"type":"v1","addr":"172.21.15.115:6801","nonce":2562256956}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6802","nonce":2562256956},{"type":"v1","addr":"172.21.15.115:6803","nonce":2562256956}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6806","nonce":2562256956},{"type":"v1","addr":"172.21.15.115:6807","nonce":2562256956}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6804","nonce":2562256956},{"type":"v1","addr":"172.21.15.115:6805","nonce":2562256956}]},"public_addr":"172.21.15.115:6801/2562256956","cluster_addr":"172.21.15.115:6803/2562256956","heartbeat_back_addr":"172.21.15.115:6807/2562256956","heartbeat_front_addr":"172.21.15.115:6805/2562256956","state":["exists","up"]},{"osd":3,"uuid":"6ca9c602-5936-48da-a861-c437e6a33765","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6810","nonce":1099909411},{"type":"v1","addr":"172.21.15.2:6811","nonce":1099909411}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6812","nonce":1099909411},{"type":"v1","addr":"172.21.15.2:6813","nonce":1099909411}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6816","nonce":1099909411},{"type":"v1","addr":"172.21.15.2:6817","nonce":1099909411}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6814","nonce":1099909411},{"type":"v1","addr":"172.21.15.2:6815","nonce":1099909411}]},"public_addr":"172.21.15.2:6811/1099909411","cluster_addr":"172.21.15.2:6813/1099909411","heartbeat_back_addr":"172.21.15.2:6817/1099909411","heartbeat_front_addr":"172.21.15.2:6815/1099909411","state":["exists","up"]},{"osd":4,"uuid":"54d943d9-2610-422a-9e59-4b8212aced31","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6808","nonce":882753861},{"type":"v1","addr":"172.21.15.47:6809","nonce":882753861}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6810","nonce":882753861},{"type":"v1","addr":"172.21.15.47:6811","nonce":882753861}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6814","nonce":882753861},{"type":"v1","addr":"172.21.15.47:6815","nonce":882753861}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6812","nonce":882753861},{"type":"v1","addr":"172.21.15.47:6813","nonce":882753861}]},"public_addr":"172.21.15.47:6809/882753861","cluster_addr":"172.21.15.47:6811/882753861","heartbeat_back_addr":"172.21.15.47:6815/882753861","heartbeat_front_addr":"172.21.15.47:6813/882753861","state":["exists","up"]},{"osd":5,"uuid":"c18566cf-5403-4057-b2ee-aa298e87e8e9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6808","nonce":2122391993},{"type":"v1","addr":"172.21.15.115:6809","nonce":2122391993}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6810","nonce":2122391993},{"type":"v1","addr":"172.21.15.115:6811","nonce":2122391993}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6814","nonce":2122391993},{"type":"v1","addr":"172.21.15.115:6815","nonce":2122391993}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6812","nonce":2122391993},{"type":"v1","addr":"172.21.15.115:6813","nonce":2122391993}]},"public_addr":"172.21.15.115:6809/2122391993","cluster_addr":"172.21.15.115:6811/2122391993","heartbeat_back_addr":"172.21.15.115:6815/2122391993","heartbeat_front_addr":"172.21.15.115:6813/2122391993","state":["exists","up"]},{"osd":6,"uuid":"38ec7b6b-30f4-4d97-ad7e-e160ae59c843","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6818","nonce":1892198790},{"type":"v1","addr":"172.21.15.2:6819","nonce":1892198790}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6820","nonce":1892198790},{"type":"v1","addr":"172.21.15.2:6821","nonce":1892198790}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6824","nonce":1892198790},{"type":"v1","addr":"172.21.15.2:6825","nonce":1892198790}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6822","nonce":1892198790},{"type":"v1","addr":"172.21.15.2:6823","nonce":1892198790}]},"public_addr":"172.21.15.2:6819/1892198790","cluster_addr":"172.21.15.2:6821/1892198790","heartbeat_back_addr":"172.21.15.2:6825/1892198790","heartbeat_front_addr":"172.21.15.2:6823/1892198790","state":["exists","up"]},{"osd":7,"uuid":"e799b560-c61e-4369-b08b-45e5a6af37c0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6816","nonce":2812542825},{"type":"v1","addr":"172.21.15.47:6817","nonce":2812542825}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6818","nonce":2812542825},{"type":"v1","addr":"172.21.15.47:6819","nonce":2812542825}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6822","nonce":2812542825},{"type":"v1","addr":"172.21.15.47:6823","nonce":2812542825}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6820","nonce":2812542825},{"type":"v1","addr":"172.21.15.47:6821","nonce":2812542825}]},"public_addr":"172.21.15.47:6817/2812542825","cluster_addr":"172.21.15.47:6819/2812542825","heartbeat_back_addr":"172.21.15.47:6823/2812542825","heartbeat_front_addr":"172.21.15.47:6821/2812542825","state":["exists","up"]},{"osd":8,"uuid":"a58f2c0e-e487-41cf-8d2e-8c937cac9ebc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":58,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6816","nonce":3959632033},{"type":"v1","addr":"172.21.15.115:6817","nonce":3959632033}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6818","nonce":3959632033},{"type":"v1","addr":"172.21.15.115:6819","nonce":3959632033}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6822","nonce":3959632033},{"type":"v1","addr":"172.21.15.115:6823","nonce":3959632033}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6820","nonce":3959632033},{"type":"v1","addr":"172.21.15.115:6821","nonce":3959632033}]},"public_addr":"172.21.15.115:6817/3959632033","cluster_addr":"172.21.15.115:6819/3959632033","heartbeat_back_addr":"172.21.15.115:6823/3959632033","heartbeat_front_addr":"172.21.15.115:6821/3959632033","state":["exists","up"]},{"osd":9,"uuid":"12cead96-f726-4f7a-ab84-eb191b778a5c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":66,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6826","nonce":3004162255},{"type":"v1","addr":"172.21.15.2:6827","nonce":3004162255}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6828","nonce":3004162255},{"type":"v1","addr":"172.21.15.2:6829","nonce":3004162255}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6832","nonce":3004162255},{"type":"v1","addr":"172.21.15.2:6833","nonce":3004162255}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6830","nonce":3004162255},{"type":"v1","addr":"172.21.15.2:6831","nonce":3004162255}]},"public_addr":"172.21.15.2:6827/3004162255","cluster_addr":"172.21.15.2:6829/3004162255","heartbeat_back_addr":"172.21.15.2:6833/3004162255","heartbeat_front_addr":"172.21.15.2:6831/3004162255","state":["exists","up"]},{"osd":10,"uuid":"4f648118-7855-48d6-86d5-44eed5295f0c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":71,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6824","nonce":3443413998},{"type":"v1","addr":"172.21.15.47:6825","nonce":3443413998}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6826","nonce":3443413998},{"type":"v1","addr":"172.21.15.47:6827","nonce":3443413998}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6830","nonce":3443413998},{"type":"v1","addr":"172.21.15.47:6831","nonce":3443413998}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6828","nonce":3443413998},{"type":"v1","addr":"172.21.15.47:6829","nonce":3443413998}]},"public_addr":"172.21.15.47:6825/3443413998","cluster_addr":"172.21.15.47:6827/3443413998","heartbeat_back_addr":"172.21.15.47:6831/3443413998","heartbeat_front_addr":"172.21.15.47:6829/3443413998","state":["exists","up"]},{"osd":11,"uuid":"916541e0-fb28-46d5-9743-801c54c5a6d1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":78,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6824","nonce":1758718209},{"type":"v1","addr":"172.21.15.115:6825","nonce":1758718209}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6826","nonce":1758718209},{"type":"v1","addr":"172.21.15.115:6827","nonce":1758718209}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6830","nonce":1758718209},{"type":"v1","addr":"172.21.15.115:6831","nonce":1758718209}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6828","nonce":1758718209},{"type":"v1","addr":"172.21.15.115:6829","nonce":1758718209}]},"public_addr":"172.21.15.115:6825/1758718209","cluster_addr":"172.21.15.115:6827/1758718209","heartbeat_back_addr":"172.21.15.115:6831/1758718209","heartbeat_front_addr":"172.21.15.115:6829/1758718209","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-05-13T23:08:43.757857+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-05-13T23:09:06.537139+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-05-13T23:09:28.524837+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-05-13T23:10:02.057583+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-05-13T23:10:26.210648+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-05-13T23:10:47.885853+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-05-13T23:11:19.141865+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-05-13T23:11:42.376065+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-05-13T23:12:04.175870+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-05-13T23:12:34.695347+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-05-13T23:12:57.762613+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-05-13T23:13:19.114284+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.f","mappings":[{"from":8,"to":11}]},{"pgid":"2.13","mappings":[{"from":2,"to":11}]},{"pgid":"3.10","mappings":[{"from":5,"to":9}]},{"pgid":"3.2c","mappings":[{"from":4,"to":9}]},{"pgid":"3.36","mappings":[{"from":4,"to":9}]}],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.2:0/477249925":"2022-05-14T23:07:04.986087+0000","172.21.15.2:6800/3226099420":"2022-05-14T23:07:04.986087+0000","172.21.15.2:0/2214093915":"2022-05-14T23:07:04.986087+0000","172.21.15.2:0/1039403355":"2022-05-14T23:07:04.986087+0000","172.21.15.2:0/22091675":"2022-05-14T23:06:36.304108+0000","172.21.15.2:6800/1255134634":"2022-05-14T23:06:36.304108+0000","172.21.15.2:6801/3226099420":"2022-05-14T23:07:04.986087+0000","172.21.15.2:0/3768677276":"2022-05-14T23:06:36.304108+0000","172.21.15.2:6801/1255134634":"2022-05-14T23:06:36.304108+0000","172.21.15.2:0/407841268":"2022-05-14T23:06:36.304108+0000"},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2022-05-13T23:15:26.725 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:26.722+0000 7fbd6e39d700 1 -- 172.21.15.2:0/17743713 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7fbd54096d60 msgr2=0x7fbd54099210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:15:26.725 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:26.722+0000 7fbd6e39d700 1 --2- 172.21.15.2:0/17743713 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7fbd54096d60 0x7fbd54099210 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fbd58004840 tx=0x7fbd58009cb0 comp rx=0 tx=0).stop 2022-05-13T23:15:26.725 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:26.722+0000 7fbd6e39d700 1 -- 172.21.15.2:0/17743713 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fbd680fa390 msgr2=0x7fbd6810e4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:15:26.726 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:26.722+0000 7fbd6e39d700 1 --2- 172.21.15.2:0/17743713 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fbd680fa390 0x7fbd6810e4a0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fbd6400e940 tx=0x7fbd6400b040 comp rx=0 tx=0).stop 2022-05-13T23:15:26.726 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:26.722+0000 7fbd6e39d700 1 -- 172.21.15.2:0/17743713 shutdown_connections 2022-05-13T23:15:26.726 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:26.722+0000 7fbd6e39d700 1 --2- 172.21.15.2:0/17743713 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fbd680fbfc0 0x7fbd68115a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:26.726 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:26.722+0000 7fbd6e39d700 1 --2- 172.21.15.2:0/17743713 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7fbd54096d60 0x7fbd54099210 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:26.726 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:26.722+0000 7fbd6e39d700 1 --2- 172.21.15.2:0/17743713 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fbd680fa390 0x7fbd6810e4a0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:26.727 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:26.722+0000 7fbd6e39d700 1 --2- 172.21.15.2:0/17743713 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fbd680fb630 0x7fbd6810e9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:26.727 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:26.722+0000 7fbd6e39d700 1 -- 172.21.15.2:0/17743713 >> 172.21.15.2:0/17743713 conn(0x7fbd680f5b80 msgr2=0x7fbd680f7610 unknown :-1 s=STATE_NONE l=0).mark_down 2022-05-13T23:15:26.727 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:26.722+0000 7fbd6e39d700 1 -- 172.21.15.2:0/17743713 shutdown_connections 2022-05-13T23:15:26.728 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:26.722+0000 7fbd6e39d700 1 -- 172.21.15.2:0/17743713 wait complete. 2022-05-13T23:15:26.753 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2022-05-13T23:09:32.000644+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2022-05-13T23:14:14.195678+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '82', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2022-05-13T23:14:15.659212+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '83', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2022-05-13T23:15:26.754 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2022-05-13T23:15:26.979 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:26.972+0000 7fc376b9d700 1 -- 172.21.15.2:0/1525144034 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fc3700fe940 msgr2=0x7fc3700fed90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:15:26.980 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:26.972+0000 7fc376b9d700 1 --2- 172.21.15.2:0/1525144034 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fc3700fe940 0x7fc3700fed90 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fc360004870 tx=0x7fc360009cb0 comp rx=0 tx=0).stop 2022-05-13T23:15:26.980 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:26.972+0000 7fc376b9d700 1 -- 172.21.15.2:0/1525144034 shutdown_connections 2022-05-13T23:15:26.981 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:26.972+0000 7fc376b9d700 1 --2- 172.21.15.2:0/1525144034 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc3700ff2d0 0x7fc370105b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:26.981 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:26.972+0000 7fc376b9d700 1 --2- 172.21.15.2:0/1525144034 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc3700fd6a0 0x7fc3700fdab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:26.981 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:26.972+0000 7fc376b9d700 1 --2- 172.21.15.2:0/1525144034 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fc3700fe940 0x7fc3700fed90 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:26.981 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:26.972+0000 7fc376b9d700 1 -- 172.21.15.2:0/1525144034 >> 172.21.15.2:0/1525144034 conn(0x7fc3700f8e70 msgr2=0x7fc3700fb2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-05-13T23:15:26.982 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:26.973+0000 7fc376b9d700 1 -- 172.21.15.2:0/1525144034 shutdown_connections 2022-05-13T23:15:26.982 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:26.973+0000 7fc376b9d700 1 -- 172.21.15.2:0/1525144034 wait complete. 2022-05-13T23:15:26.982 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:26.973+0000 7fc376b9d700 1 Processor -- start 2022-05-13T23:15:26.982 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:26.974+0000 7fc376b9d700 1 -- start start 2022-05-13T23:15:26.983 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:26.974+0000 7fc376b9d700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc3700fd6a0 0x7fc37010e4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:15:26.983 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:26.974+0000 7fc376b9d700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc3700fe940 0x7fc37010ea00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:15:26.983 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:26.974+0000 7fc376b9d700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fc3700ff2d0 0x7fc370115a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:15:26.983 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:26.974+0000 7fc376b9d700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc370115fd0 con 0x7fc3700fd6a0 2022-05-13T23:15:26.983 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:26.974+0000 7fc376b9d700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc370116140 con 0x7fc3700fe940 2022-05-13T23:15:26.984 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:26.974+0000 7fc376b9d700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc370116300 con 0x7fc3700ff2d0 2022-05-13T23:15:26.984 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:26.974+0000 7fc375b9b700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc3700fd6a0 0x7fc37010e4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:15:26.984 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:26.974+0000 7fc375b9b700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc3700fd6a0 0x7fc37010e4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:46064/0 (socket says 172.21.15.2:46064) 2022-05-13T23:15:26.984 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:26.974+0000 7fc375b9b700 1 -- 172.21.15.2:0/1344170588 learned_addr learned my addr 172.21.15.2:0/1344170588 (peer_addr_for_me v2:172.21.15.2:0/0) 2022-05-13T23:15:26.984 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:26.974+0000 7fc37539a700 1 --2- 172.21.15.2:0/1344170588 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc3700fe940 0x7fc37010ea00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:15:26.985 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:26.975+0000 7fc37639c700 1 --2- 172.21.15.2:0/1344170588 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fc3700ff2d0 0x7fc370115a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:15:26.985 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:26.975+0000 7fc37539a700 1 -- 172.21.15.2:0/1344170588 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fc3700ff2d0 msgr2=0x7fc370115a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:15:26.985 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:26.975+0000 7fc37539a700 1 --2- 172.21.15.2:0/1344170588 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fc3700ff2d0 0x7fc370115a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:26.985 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:26.975+0000 7fc37539a700 1 -- 172.21.15.2:0/1344170588 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc3700fd6a0 msgr2=0x7fc37010e4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:15:26.986 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:26.975+0000 7fc37539a700 1 --2- 172.21.15.2:0/1344170588 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc3700fd6a0 0x7fc37010e4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:26.986 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:26.975+0000 7fc37539a700 1 -- 172.21.15.2:0/1344170588 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3680097c0 con 0x7fc3700fe940 2022-05-13T23:15:26.986 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:26.975+0000 7fc375b9b700 1 --2- 172.21.15.2:0/1344170588 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc3700fd6a0 0x7fc37010e4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-05-13T23:15:26.986 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:26.975+0000 7fc37639c700 1 --2- 172.21.15.2:0/1344170588 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fc3700ff2d0 0x7fc370115a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2022-05-13T23:15:26.987 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:26.975+0000 7fc37539a700 1 --2- 172.21.15.2:0/1344170588 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc3700fe940 0x7fc37010ea00 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fc360004710 tx=0x7fc360009cb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-05-13T23:15:26.987 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:26.975+0000 7fc3667fc700 1 -- 172.21.15.2:0/1344170588 <== mon.1 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc360020070 con 0x7fc3700fe940 2022-05-13T23:15:26.987 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:26.976+0000 7fc3667fc700 1 -- 172.21.15.2:0/1344170588 <== mon.1 v2:172.21.15.47:3300/0 2 ==== config(26 keys) v1 ==== 1087+0+0 (secure 0 0 0) 0x7fc360014e40 con 0x7fc3700fe940 2022-05-13T23:15:26.988 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:26.976+0000 7fc376b9d700 1 -- 172.21.15.2:0/1344170588 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc360005040 con 0x7fc3700fe940 2022-05-13T23:15:26.988 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:26.976+0000 7fc3667fc700 1 -- 172.21.15.2:0/1344170588 <== mon.1 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc36000e040 con 0x7fc3700fe940 2022-05-13T23:15:26.989 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:26.976+0000 7fc376b9d700 1 -- 172.21.15.2:0/1344170588 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc370116900 con 0x7fc3700fe940 2022-05-13T23:15:26.989 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:26.977+0000 7fc376b9d700 1 -- 172.21.15.2:0/1344170588 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc370108e90 con 0x7fc3700fe940 2022-05-13T23:15:26.991 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:26.979+0000 7fc3667fc700 1 -- 172.21.15.2:0/1344170588 <== mon.1 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 16) v1 ==== 125136+0+0 (secure 0 0 0) 0x7fc360009240 con 0x7fc3700fe940 2022-05-13T23:15:26.991 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:26.980+0000 7fc3667fc700 1 --2- 172.21.15.2:0/1344170588 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7fc35c096ce0 0x7fc35c099190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:15:26.991 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:26.980+0000 7fc3667fc700 1 -- 172.21.15.2:0/1344170588 <== mon.1 v2:172.21.15.47:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8120+0+0 (secure 0 0 0) 0x7fc3600c0950 con 0x7fc3700fe940 2022-05-13T23:15:26.991 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:26.980+0000 7fc375b9b700 1 --2- 172.21.15.2:0/1344170588 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7fc35c096ce0 0x7fc35c099190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:15:26.992 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:26.981+0000 7fc375b9b700 1 --2- 172.21.15.2:0/1344170588 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7fc35c096ce0 0x7fc35c099190 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fc36c0041d0 tx=0x7fc36c00b040 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-05-13T23:15:26.992 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:26.982+0000 7fc3667fc700 1 -- 172.21.15.2:0/1344170588 <== mon.1 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+165215 (secure 0 0 0) 0x7fc36008dcf0 con 0x7fc3700fe940 2022-05-13T23:15:27.176 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:27.174+0000 7fc376b9d700 1 -- 172.21.15.2:0/1344170588 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fc370104a00 con 0x7fc3700fe940 2022-05-13T23:15:27.177 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:27.174+0000 7fc3667fc700 1 -- 172.21.15.2:0/1344170588 <== mon.1 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v85) v1 ==== 93+0+10 (secure 0 0 0) 0x7fc360091310 con 0x7fc3700fe940 2022-05-13T23:15:27.177 INFO:teuthology.orchestra.run.smithi002.stdout:pg_num: 1 2022-05-13T23:15:27.180 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:27.177+0000 7fc376b9d700 1 -- 172.21.15.2:0/1344170588 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7fc35c096ce0 msgr2=0x7fc35c099190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:15:27.183 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:27.177+0000 7fc376b9d700 1 --2- 172.21.15.2:0/1344170588 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7fc35c096ce0 0x7fc35c099190 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fc36c0041d0 tx=0x7fc36c00b040 comp rx=0 tx=0).stop 2022-05-13T23:15:27.183 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:27.177+0000 7fc376b9d700 1 -- 172.21.15.2:0/1344170588 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc3700fe940 msgr2=0x7fc37010ea00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:15:27.183 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:27.177+0000 7fc376b9d700 1 --2- 172.21.15.2:0/1344170588 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc3700fe940 0x7fc37010ea00 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fc360004710 tx=0x7fc360009cb0 comp rx=0 tx=0).stop 2022-05-13T23:15:27.183 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:27.177+0000 7fc376b9d700 1 -- 172.21.15.2:0/1344170588 shutdown_connections 2022-05-13T23:15:27.183 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:27.177+0000 7fc376b9d700 1 --2- 172.21.15.2:0/1344170588 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fc3700ff2d0 0x7fc370115a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:27.184 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:27.177+0000 7fc376b9d700 1 --2- 172.21.15.2:0/1344170588 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7fc35c096ce0 0x7fc35c099190 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:27.184 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:27.177+0000 7fc376b9d700 1 --2- 172.21.15.2:0/1344170588 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc3700fd6a0 0x7fc37010e4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:27.184 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:27.177+0000 7fc376b9d700 1 --2- 172.21.15.2:0/1344170588 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc3700fe940 0x7fc37010ea00 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:27.184 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:27.177+0000 7fc376b9d700 1 -- 172.21.15.2:0/1344170588 >> 172.21.15.2:0/1344170588 conn(0x7fc3700f8e70 msgr2=0x7fc370101b70 unknown :-1 s=STATE_NONE l=0).mark_down 2022-05-13T23:15:27.185 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:27.177+0000 7fc376b9d700 1 -- 172.21.15.2:0/1344170588 shutdown_connections 2022-05-13T23:15:27.185 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:27.177+0000 7fc376b9d700 1 -- 172.21.15.2:0/1344170588 wait complete. 2022-05-13T23:15:27.207 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2022-05-13T23:15:27.428 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:27.425+0000 7f217a79d700 1 -- 172.21.15.2:0/3610372002 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2174064610 msgr2=0x7f21740fe160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:15:27.428 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:27.425+0000 7f217a79d700 1 --2- 172.21.15.2:0/3610372002 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2174064610 0x7f21740fe160 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f2170004870 tx=0x7f2170009cb0 comp rx=0 tx=0).stop 2022-05-13T23:15:27.429 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:27.426+0000 7f217a79d700 1 -- 172.21.15.2:0/3610372002 shutdown_connections 2022-05-13T23:15:27.429 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:27.426+0000 7f217a79d700 1 --2- 172.21.15.2:0/3610372002 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f2174101110 0x7f2174103590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:27.429 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:27.426+0000 7f217a79d700 1 --2- 172.21.15.2:0/3610372002 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2174064610 0x7f21740fe160 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:27.429 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:27.426+0000 7f217a79d700 1 --2- 172.21.15.2:0/3610372002 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f21740fe6a0 0x7f2174100a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:27.430 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:27.426+0000 7f217a79d700 1 -- 172.21.15.2:0/3610372002 >> 172.21.15.2:0/3610372002 conn(0x7f21740f5b70 msgr2=0x7f21740f7fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-05-13T23:15:27.430 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:27.426+0000 7f217a79d700 1 -- 172.21.15.2:0/3610372002 shutdown_connections 2022-05-13T23:15:27.430 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:27.426+0000 7f217a79d700 1 -- 172.21.15.2:0/3610372002 wait complete. 2022-05-13T23:15:27.433 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:27.427+0000 7f217a79d700 1 Processor -- start 2022-05-13T23:15:27.433 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:27.427+0000 7f217a79d700 1 -- start start 2022-05-13T23:15:27.434 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:27.427+0000 7f217a79d700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f2174064610 0x7f217410c2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:15:27.435 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:27.427+0000 7f217a79d700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f21740fe6a0 0x7f217410c7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:15:27.435 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:27.427+0000 7f217a79d700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2174101110 0x7f2174113880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:15:27.435 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:27.427+0000 7f217a79d700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2174113dc0 con 0x7f2174101110 2022-05-13T23:15:27.435 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:27.427+0000 7f217a79d700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2174113f30 con 0x7f21740fe6a0 2022-05-13T23:15:27.436 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:27.427+0000 7f217a79d700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21741140f0 con 0x7f2174064610 2022-05-13T23:15:27.436 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:27.427+0000 7f217979b700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f2174064610 0x7f217410c2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:15:27.436 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:27.427+0000 7f217979b700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f2174064610 0x7f217410c2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.2:53472/0 (socket says 172.21.15.2:53472) 2022-05-13T23:15:27.436 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:27.427+0000 7f217979b700 1 -- 172.21.15.2:0/1129361427 learned_addr learned my addr 172.21.15.2:0/1129361427 (peer_addr_for_me v2:172.21.15.2:0/0) 2022-05-13T23:15:27.437 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:27.428+0000 7f2179f9c700 1 --2- 172.21.15.2:0/1129361427 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2174101110 0x7f2174113880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:15:27.437 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:27.428+0000 7f2178f9a700 1 --2- 172.21.15.2:0/1129361427 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f21740fe6a0 0x7f217410c7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:15:27.437 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:27.428+0000 7f217979b700 1 -- 172.21.15.2:0/1129361427 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f21740fe6a0 msgr2=0x7f217410c7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:15:27.437 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:27.428+0000 7f217979b700 1 --2- 172.21.15.2:0/1129361427 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f21740fe6a0 0x7f217410c7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:27.438 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:27.428+0000 7f217979b700 1 -- 172.21.15.2:0/1129361427 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2174101110 msgr2=0x7f2174113880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:15:27.438 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:27.428+0000 7f217979b700 1 --2- 172.21.15.2:0/1129361427 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2174101110 0x7f2174113880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:27.438 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:27.428+0000 7f217979b700 1 -- 172.21.15.2:0/1129361427 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2170005040 con 0x7f2174064610 2022-05-13T23:15:27.438 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:27.428+0000 7f2179f9c700 1 --2- 172.21.15.2:0/1129361427 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2174101110 0x7f2174113880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-05-13T23:15:27.438 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:27.428+0000 7f217979b700 1 --2- 172.21.15.2:0/1129361427 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f2174064610 0x7f217410c2b0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f2170004710 tx=0x7f2170009cb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-05-13T23:15:27.439 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:27.429+0000 7f2169ffb700 1 -- 172.21.15.2:0/1129361427 <== mon.2 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f217000e040 con 0x7f2174064610 2022-05-13T23:15:27.439 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:27.429+0000 7f2169ffb700 1 -- 172.21.15.2:0/1129361427 <== mon.2 v2:172.21.15.115:3300/0 2 ==== config(26 keys) v1 ==== 1087+0+0 (secure 0 0 0) 0x7f2170014e40 con 0x7f2174064610 2022-05-13T23:15:27.439 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:27.429+0000 7f217a79d700 1 -- 172.21.15.2:0/1129361427 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f21740f9f40 con 0x7f2174064610 2022-05-13T23:15:27.440 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:27.429+0000 7f2169ffb700 1 -- 172.21.15.2:0/1129361427 <== mon.2 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2170024440 con 0x7f2174064610 2022-05-13T23:15:27.440 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:27.429+0000 7f217a79d700 1 -- 172.21.15.2:0/1129361427 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f21740fa4a0 con 0x7f2174064610 2022-05-13T23:15:27.440 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:27.430+0000 7f217a79d700 1 -- 172.21.15.2:0/1129361427 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2174106c80 con 0x7f2174064610 2022-05-13T23:15:27.442 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:27.432+0000 7f2169ffb700 1 -- 172.21.15.2:0/1129361427 <== mon.2 v2:172.21.15.115:3300/0 4 ==== mgrmap(e 16) v1 ==== 125136+0+0 (secure 0 0 0) 0x7f217001a570 con 0x7f2174064610 2022-05-13T23:15:27.442 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:27.433+0000 7f2169ffb700 1 --2- 172.21.15.2:0/1129361427 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f2160096d60 0x7f2160099210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:15:27.442 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:27.433+0000 7f2169ffb700 1 -- 172.21.15.2:0/1129361427 <== mon.2 v2:172.21.15.115:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8120+0+0 (secure 0 0 0) 0x7f21700c4b50 con 0x7f2174064610 2022-05-13T23:15:27.442 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:27.433+0000 7f2178f9a700 1 --2- 172.21.15.2:0/1129361427 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f2160096d60 0x7f2160099210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:15:27.443 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:27.434+0000 7f2178f9a700 1 --2- 172.21.15.2:0/1129361427 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f2160096d60 0x7f2160099210 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f21640048a0 tx=0x7f216400b040 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-05-13T23:15:27.443 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:27.435+0000 7f2169ffb700 1 -- 172.21.15.2:0/1129361427 <== mon.2 v2:172.21.15.115:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+165215 (secure 0 0 0) 0x7f2170091e80 con 0x7f2174064610 2022-05-13T23:15:27.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:15:27 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:15:27.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:15:27 smithi047 ceph-mon[77147]: from='client.? 172.21.15.2:0/17743713' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2022-05-13T23:15:27.572 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:15:27 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:15:27.572 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:15:27 smithi002 ceph-mon[71816]: from='client.? 172.21.15.2:0/17743713' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2022-05-13T23:15:27.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:15:27 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:15:27.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:15:27 smithi115 ceph-mon[78593]: from='client.? 172.21.15.2:0/17743713' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2022-05-13T23:15:27.629 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:27.627+0000 7f217a79d700 1 -- 172.21.15.2:0/1129361427 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f21740f7520 con 0x7f2174064610 2022-05-13T23:15:27.630 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:27.629+0000 7f2169ffb700 1 -- 172.21.15.2:0/1129361427 <== mon.2 v2:172.21.15.115:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v85) v1 ==== 104+0+11 (secure 0 0 0) 0x7f21700954a0 con 0x7f2174064610 2022-05-13T23:15:27.631 INFO:teuthology.orchestra.run.smithi002.stdout:pg_num: 32 2022-05-13T23:15:27.633 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:27.630+0000 7f217a79d700 1 -- 172.21.15.2:0/1129361427 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f2160096d60 msgr2=0x7f2160099210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:15:27.633 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:27.630+0000 7f217a79d700 1 --2- 172.21.15.2:0/1129361427 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f2160096d60 0x7f2160099210 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f21640048a0 tx=0x7f216400b040 comp rx=0 tx=0).stop 2022-05-13T23:15:27.634 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:27.630+0000 7f217a79d700 1 -- 172.21.15.2:0/1129361427 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f2174064610 msgr2=0x7f217410c2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:15:27.634 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:27.630+0000 7f217a79d700 1 --2- 172.21.15.2:0/1129361427 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f2174064610 0x7f217410c2b0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f2170004710 tx=0x7f2170009cb0 comp rx=0 tx=0).stop 2022-05-13T23:15:27.634 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:27.630+0000 7f217a79d700 1 -- 172.21.15.2:0/1129361427 shutdown_connections 2022-05-13T23:15:27.634 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:27.630+0000 7f217a79d700 1 --2- 172.21.15.2:0/1129361427 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f2160096d60 0x7f2160099210 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:27.635 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:27.630+0000 7f217a79d700 1 --2- 172.21.15.2:0/1129361427 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2174101110 0x7f2174113880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:27.635 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:27.630+0000 7f217a79d700 1 --2- 172.21.15.2:0/1129361427 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f2174064610 0x7f217410c2b0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:27.636 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:27.630+0000 7f217a79d700 1 --2- 172.21.15.2:0/1129361427 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f21740fe6a0 0x7f217410c7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:27.636 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:27.630+0000 7f217a79d700 1 -- 172.21.15.2:0/1129361427 >> 172.21.15.2:0/1129361427 conn(0x7f21740f5b70 msgr2=0x7f21740fc9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-05-13T23:15:27.636 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:27.631+0000 7f217a79d700 1 -- 172.21.15.2:0/1129361427 shutdown_connections 2022-05-13T23:15:27.636 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:27.631+0000 7f217a79d700 1 -- 172.21.15.2:0/1129361427 wait complete. 2022-05-13T23:15:27.660 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2022-05-13T23:15:27.890 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:27.881+0000 7f8bdf59d700 1 -- 172.21.15.2:0/2497972074 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8bd8072040 msgr2=0x7f8bd80724b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:15:27.890 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:27.881+0000 7f8bdf59d700 1 --2- 172.21.15.2:0/2497972074 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8bd8072040 0x7f8bd80724b0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f8bc80047b0 tx=0x7f8bc8009cb0 comp rx=0 tx=0).stop 2022-05-13T23:15:27.891 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:27.882+0000 7f8bdf59d700 1 -- 172.21.15.2:0/2497972074 shutdown_connections 2022-05-13T23:15:27.891 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:27.882+0000 7f8bdf59d700 1 --2- 172.21.15.2:0/2497972074 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f8bd80729f0 0x7f8bd8073280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:27.891 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:27.882+0000 7f8bdf59d700 1 --2- 172.21.15.2:0/2497972074 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8bd8070d50 0x7f8bd8071160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:27.891 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:27.882+0000 7f8bdf59d700 1 --2- 172.21.15.2:0/2497972074 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8bd8072040 0x7f8bd80724b0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:27.892 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:27.882+0000 7f8bdf59d700 1 -- 172.21.15.2:0/2497972074 >> 172.21.15.2:0/2497972074 conn(0x7f8bd80f8ea0 msgr2=0x7f8bd80fb2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-05-13T23:15:27.892 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:27.882+0000 7f8bdf59d700 1 -- 172.21.15.2:0/2497972074 shutdown_connections 2022-05-13T23:15:27.892 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:27.882+0000 7f8bdf59d700 1 -- 172.21.15.2:0/2497972074 wait complete. 2022-05-13T23:15:27.892 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:27.882+0000 7f8bdf59d700 1 Processor -- start 2022-05-13T23:15:27.892 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:27.883+0000 7f8bdf59d700 1 -- start start 2022-05-13T23:15:27.893 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:27.883+0000 7f8bdf59d700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8bd8070d50 0x7f8bd8112910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:15:27.893 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:27.883+0000 7f8bdf59d700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f8bd8072040 0x7f8bd8112e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:15:27.893 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:27.883+0000 7f8bdf59d700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8bd80729f0 0x7f8bd8119ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:15:27.893 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:27.883+0000 7f8bdf59d700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8bd811a420 con 0x7f8bd80729f0 2022-05-13T23:15:27.894 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:27.883+0000 7f8bdf59d700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8bd811a590 con 0x7f8bd8070d50 2022-05-13T23:15:27.894 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:27.883+0000 7f8bdf59d700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8bd811a750 con 0x7f8bd8072040 2022-05-13T23:15:27.894 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:27.883+0000 7f8bde59b700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8bd8070d50 0x7f8bd8112910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:15:27.894 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:27.883+0000 7f8bddd9a700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f8bd8072040 0x7f8bd8112e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:15:27.894 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:27.883+0000 7f8bde59b700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8bd8070d50 0x7f8bd8112910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.2:41374/0 (socket says 172.21.15.2:41374) 2022-05-13T23:15:27.895 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:27.883+0000 7f8bded9c700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8bd80729f0 0x7f8bd8119ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:15:27.895 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:27.883+0000 7f8bde59b700 1 -- 172.21.15.2:0/2723773940 learned_addr learned my addr 172.21.15.2:0/2723773940 (peer_addr_for_me v2:172.21.15.2:0/0) 2022-05-13T23:15:27.895 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:27.884+0000 7f8bde59b700 1 -- 172.21.15.2:0/2723773940 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f8bd8072040 msgr2=0x7f8bd8112e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:15:27.895 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:27.884+0000 7f8bde59b700 1 --2- 172.21.15.2:0/2723773940 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f8bd8072040 0x7f8bd8112e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:27.896 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:27.884+0000 7f8bde59b700 1 -- 172.21.15.2:0/2723773940 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8bd80729f0 msgr2=0x7f8bd8119ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:15:27.896 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:27.884+0000 7f8bde59b700 1 --2- 172.21.15.2:0/2723773940 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8bd80729f0 0x7f8bd8119ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:27.896 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:27.884+0000 7f8bde59b700 1 -- 172.21.15.2:0/2723773940 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8bd4005040 con 0x7f8bd8070d50 2022-05-13T23:15:27.896 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:27.884+0000 7f8bde59b700 1 --2- 172.21.15.2:0/2723773940 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8bd8070d50 0x7f8bd8112910 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f8bd4009400 tx=0x7f8bd4009cb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-05-13T23:15:27.896 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:27.884+0000 7f8bceffd700 1 -- 172.21.15.2:0/2723773940 <== mon.1 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8bd4012570 con 0x7f8bd8070d50 2022-05-13T23:15:27.897 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:27.884+0000 7f8bceffd700 1 -- 172.21.15.2:0/2723773940 <== mon.1 v2:172.21.15.47:3300/0 2 ==== config(26 keys) v1 ==== 1087+0+0 (secure 0 0 0) 0x7f8bd400f040 con 0x7f8bd8070d50 2022-05-13T23:15:27.897 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:27.884+0000 7f8bdf59d700 1 -- 172.21.15.2:0/2723773940 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8bd00097c0 con 0x7f8bd8070d50 2022-05-13T23:15:27.897 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:27.884+0000 7f8bceffd700 1 -- 172.21.15.2:0/2723773940 <== mon.1 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8bd4013b80 con 0x7f8bd8070d50 2022-05-13T23:15:27.897 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:27.885+0000 7f8bdf59d700 1 -- 172.21.15.2:0/2723773940 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8bc8005040 con 0x7f8bd8070d50 2022-05-13T23:15:27.898 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:27.886+0000 7f8bdf59d700 1 -- 172.21.15.2:0/2723773940 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8bd810d2e0 con 0x7f8bd8070d50 2022-05-13T23:15:27.898 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:27.888+0000 7f8bceffd700 1 -- 172.21.15.2:0/2723773940 <== mon.1 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 16) v1 ==== 125136+0+0 (secure 0 0 0) 0x7f8bd40126d0 con 0x7f8bd8070d50 2022-05-13T23:15:27.899 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:27.889+0000 7f8bceffd700 1 --2- 172.21.15.2:0/2723773940 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f8bc4096d60 0x7f8bc4099210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:15:27.899 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:27.889+0000 7f8bceffd700 1 -- 172.21.15.2:0/2723773940 <== mon.1 v2:172.21.15.47:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8120+0+0 (secure 0 0 0) 0x7f8bd40bfa90 con 0x7f8bd8070d50 2022-05-13T23:15:27.900 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:27.889+0000 7f8bddd9a700 1 --2- 172.21.15.2:0/2723773940 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f8bc4096d60 0x7f8bc4099210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:15:27.900 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:27.890+0000 7f8bddd9a700 1 --2- 172.21.15.2:0/2723773940 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f8bc4096d60 0x7f8bc4099210 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f8bc8004e20 tx=0x7f8bc8009cb0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-05-13T23:15:27.900 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:27.891+0000 7f8bceffd700 1 -- 172.21.15.2:0/2723773940 <== mon.1 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+165215 (secure 0 0 0) 0x7f8bd408cd40 con 0x7f8bd8070d50 2022-05-13T23:15:28.086 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:28.083+0000 7f8bdf59d700 1 -- 172.21.15.2:0/2723773940 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f8bd810d470 con 0x7f8bd8070d50 2022-05-13T23:15:28.087 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:28.084+0000 7f8bceffd700 1 -- 172.21.15.2:0/2723773940 <== mon.1 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v85) v1 ==== 100+0+11 (secure 0 0 0) 0x7f8bd810d470 con 0x7f8bd8070d50 2022-05-13T23:15:28.087 INFO:teuthology.orchestra.run.smithi002.stdout:pg_num: 64 2022-05-13T23:15:28.092 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:28.087+0000 7f8bdf59d700 1 -- 172.21.15.2:0/2723773940 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f8bc4096d60 msgr2=0x7f8bc4099210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:15:28.092 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:28.087+0000 7f8bdf59d700 1 --2- 172.21.15.2:0/2723773940 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f8bc4096d60 0x7f8bc4099210 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f8bc8004e20 tx=0x7f8bc8009cb0 comp rx=0 tx=0).stop 2022-05-13T23:15:28.092 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:28.087+0000 7f8bdf59d700 1 -- 172.21.15.2:0/2723773940 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8bd8070d50 msgr2=0x7f8bd8112910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:15:28.093 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:28.087+0000 7f8bdf59d700 1 --2- 172.21.15.2:0/2723773940 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8bd8070d50 0x7f8bd8112910 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f8bd4009400 tx=0x7f8bd4009cb0 comp rx=0 tx=0).stop 2022-05-13T23:15:28.094 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:28.087+0000 7f8bdf59d700 1 -- 172.21.15.2:0/2723773940 shutdown_connections 2022-05-13T23:15:28.095 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:28.087+0000 7f8bdf59d700 1 --2- 172.21.15.2:0/2723773940 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f8bc4096d60 0x7f8bc4099210 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:28.095 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:28.087+0000 7f8bdf59d700 1 --2- 172.21.15.2:0/2723773940 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8bd80729f0 0x7f8bd8119ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:28.095 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:28.087+0000 7f8bdf59d700 1 --2- 172.21.15.2:0/2723773940 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8bd8070d50 0x7f8bd8112910 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:28.096 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:28.087+0000 7f8bdf59d700 1 --2- 172.21.15.2:0/2723773940 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f8bd8072040 0x7f8bd8112e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:28.096 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:28.087+0000 7f8bdf59d700 1 -- 172.21.15.2:0/2723773940 >> 172.21.15.2:0/2723773940 conn(0x7f8bd80f8ea0 msgr2=0x7f8bd80fae10 unknown :-1 s=STATE_NONE l=0).mark_down 2022-05-13T23:15:28.096 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:28.087+0000 7f8bdf59d700 1 -- 172.21.15.2:0/2723773940 shutdown_connections 2022-05-13T23:15:28.096 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:28.087+0000 7f8bdf59d700 1 -- 172.21.15.2:0/2723773940 wait complete. 2022-05-13T23:15:28.117 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-05-13T23:15:28.342 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:28.338+0000 7f3d33fff700 1 -- 172.21.15.2:0/1218708625 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3d340fb4e0 msgr2=0x7f3d340fb8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:15:28.342 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:28.338+0000 7f3d33fff700 1 --2- 172.21.15.2:0/1218708625 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3d340fb4e0 0x7f3d340fb8f0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f3d24004870 tx=0x7f3d24009cb0 comp rx=0 tx=0).stop 2022-05-13T23:15:28.342 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:28.339+0000 7f3d33fff700 1 -- 172.21.15.2:0/1218708625 shutdown_connections 2022-05-13T23:15:28.343 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:28.339+0000 7f3d33fff700 1 --2- 172.21.15.2:0/1218708625 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3d340fd110 0x7f3d341039d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:28.343 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:28.339+0000 7f3d33fff700 1 --2- 172.21.15.2:0/1218708625 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3d340fb4e0 0x7f3d340fb8f0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:28.343 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:28.339+0000 7f3d33fff700 1 --2- 172.21.15.2:0/1218708625 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f3d340fc780 0x7f3d340fcbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:28.343 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:28.339+0000 7f3d33fff700 1 -- 172.21.15.2:0/1218708625 >> 172.21.15.2:0/1218708625 conn(0x7f3d340f6cb0 msgr2=0x7f3d340f9100 unknown :-1 s=STATE_NONE l=0).mark_down 2022-05-13T23:15:28.344 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:28.339+0000 7f3d33fff700 1 -- 172.21.15.2:0/1218708625 shutdown_connections 2022-05-13T23:15:28.344 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:28.339+0000 7f3d33fff700 1 -- 172.21.15.2:0/1218708625 wait complete. 2022-05-13T23:15:28.344 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:28.340+0000 7f3d33fff700 1 Processor -- start 2022-05-13T23:15:28.347 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:28.340+0000 7f3d33fff700 1 -- start start 2022-05-13T23:15:28.347 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:28.340+0000 7f3d33fff700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f3d340fb4e0 0x7f3d3410c210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:15:28.347 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:28.340+0000 7f3d33fff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3d340fc780 0x7f3d3410c750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:15:28.348 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:28.340+0000 7f3d33fff700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3d340fd110 0x7f3d341137e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:15:28.348 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:28.340+0000 7f3d33fff700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d34113d20 con 0x7f3d340fc780 2022-05-13T23:15:28.348 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:28.340+0000 7f3d33fff700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d34113e90 con 0x7f3d340fd110 2022-05-13T23:15:28.349 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:28.340+0000 7f3d33fff700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d34114050 con 0x7f3d340fb4e0 2022-05-13T23:15:28.349 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:28.341+0000 7f3d337fe700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3d340fd110 0x7f3d341137e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:15:28.349 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:28.341+0000 7f3d327fc700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3d340fc780 0x7f3d3410c750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:15:28.349 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:28.341+0000 7f3d32ffd700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f3d340fb4e0 0x7f3d3410c210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:15:28.350 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:28.341+0000 7f3d337fe700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3d340fd110 0x7f3d341137e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.2:41398/0 (socket says 172.21.15.2:41398) 2022-05-13T23:15:28.350 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:28.341+0000 7f3d337fe700 1 -- 172.21.15.2:0/1549106648 learned_addr learned my addr 172.21.15.2:0/1549106648 (peer_addr_for_me v2:172.21.15.2:0/0) 2022-05-13T23:15:28.350 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:28.341+0000 7f3d337fe700 1 -- 172.21.15.2:0/1549106648 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f3d340fb4e0 msgr2=0x7f3d3410c210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:15:28.350 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:28.341+0000 7f3d337fe700 1 --2- 172.21.15.2:0/1549106648 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f3d340fb4e0 0x7f3d3410c210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:28.351 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:28.341+0000 7f3d337fe700 1 -- 172.21.15.2:0/1549106648 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3d340fc780 msgr2=0x7f3d3410c750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:15:28.351 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:28.341+0000 7f3d337fe700 1 --2- 172.21.15.2:0/1549106648 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3d340fc780 0x7f3d3410c750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:28.351 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:28.341+0000 7f3d337fe700 1 -- 172.21.15.2:0/1549106648 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d24005040 con 0x7f3d340fd110 2022-05-13T23:15:28.351 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:28.341+0000 7f3d337fe700 1 --2- 172.21.15.2:0/1549106648 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3d340fd110 0x7f3d341137e0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f3d2c00ec20 tx=0x7f3d2c00b040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-05-13T23:15:28.351 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:28.342+0000 7f3d1b7fe700 1 -- 172.21.15.2:0/1549106648 <== mon.1 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d2c0092e0 con 0x7f3d340fd110 2022-05-13T23:15:28.352 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:28.342+0000 7f3d33fff700 1 -- 172.21.15.2:0/1549106648 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3d34114330 con 0x7f3d340fd110 2022-05-13T23:15:28.352 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:28.342+0000 7f3d1b7fe700 1 -- 172.21.15.2:0/1549106648 <== mon.1 v2:172.21.15.47:3300/0 2 ==== config(26 keys) v1 ==== 1087+0+0 (secure 0 0 0) 0x7f3d2c00f040 con 0x7f3d340fd110 2022-05-13T23:15:28.352 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:28.342+0000 7f3d1b7fe700 1 -- 172.21.15.2:0/1549106648 <== mon.1 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d2c00cab0 con 0x7f3d340fd110 2022-05-13T23:15:28.352 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:28.342+0000 7f3d33fff700 1 -- 172.21.15.2:0/1549106648 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3d34114850 con 0x7f3d340fd110 2022-05-13T23:15:28.353 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:28.343+0000 7f3d33fff700 1 -- 172.21.15.2:0/1549106648 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3d34106d40 con 0x7f3d340fd110 2022-05-13T23:15:28.353 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:28.346+0000 7f3d1b7fe700 1 -- 172.21.15.2:0/1549106648 <== mon.1 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 16) v1 ==== 125136+0+0 (secure 0 0 0) 0x7f3d2c0077a0 con 0x7f3d340fd110 2022-05-13T23:15:28.354 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:28.347+0000 7f3d1b7fe700 1 --2- 172.21.15.2:0/1549106648 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f3d1c096ce0 0x7f3d1c099190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:15:28.355 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:28.347+0000 7f3d1b7fe700 1 -- 172.21.15.2:0/1549106648 <== mon.1 v2:172.21.15.47:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8120+0+0 (secure 0 0 0) 0x7f3d2c0bfdb0 con 0x7f3d340fd110 2022-05-13T23:15:28.355 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:28.347+0000 7f3d32ffd700 1 --2- 172.21.15.2:0/1549106648 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f3d1c096ce0 0x7f3d1c099190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:15:28.355 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:28.347+0000 7f3d32ffd700 1 --2- 172.21.15.2:0/1549106648 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f3d1c096ce0 0x7f3d1c099190 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f3d24009da0 tx=0x7f3d24009cb0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-05-13T23:15:28.355 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:28.348+0000 7f3d1b7fe700 1 -- 172.21.15.2:0/1549106648 <== mon.1 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+165215 (secure 0 0 0) 0x7f3d2c08d0a0 con 0x7f3d340fd110 2022-05-13T23:15:28.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:15:28 smithi047 ceph-mon[77147]: from='client.? 172.21.15.2:0/1344170588' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2022-05-13T23:15:28.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:15:28 smithi047 ceph-mon[77147]: pgmap v326: 97 pgs: 97 active+clean; 584 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2022-05-13T23:15:28.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:15:28 smithi047 ceph-mon[77147]: from='client.? 172.21.15.2:0/1129361427' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2022-05-13T23:15:28.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:15:28 smithi047 ceph-mon[77147]: from='client.? 172.21.15.2:0/2723773940' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2022-05-13T23:15:28.572 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:15:28 smithi002 ceph-mon[71816]: from='client.? 172.21.15.2:0/1344170588' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2022-05-13T23:15:28.573 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:15:28 smithi002 ceph-mon[71816]: pgmap v326: 97 pgs: 97 active+clean; 584 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2022-05-13T23:15:28.573 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:15:28 smithi002 ceph-mon[71816]: from='client.? 172.21.15.2:0/1129361427' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2022-05-13T23:15:28.574 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:15:28 smithi002 ceph-mon[71816]: from='client.? 172.21.15.2:0/2723773940' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2022-05-13T23:15:28.586 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:28.584+0000 7f3d33fff700 1 -- 172.21.15.2:0/1549106648 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3d341026b0 con 0x7f3d340fd110 2022-05-13T23:15:28.586 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:28.585+0000 7f3d1b7fe700 1 -- 172.21.15.2:0/1549106648 <== mon.1 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) v1 ==== 94+0+10751 (secure 0 0 0) 0x7f3d2c00cea0 con 0x7f3d340fd110 2022-05-13T23:15:28.587 INFO:teuthology.orchestra.run.smithi002.stdout: 2022-05-13T23:15:28.587 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24460,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.2:6835/1341618941","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6834","nonce":1341618941},{"type":"v1","addr":"172.21.15.2:6835","nonce":1341618941}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24490,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.2:6839/1347050951","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6838","nonce":1347050951},{"type":"v1","addr":"172.21.15.2:6839","nonce":1347050951}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4}],"filesystems":[{"mdsmap":{"epoch":26,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2022-05-13T23:14:16.236645+0000","modified":"2022-05-13T23:14:23.309447+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24611,"mds_1":24599,"mds_2":24587,"mds_3":24575,"mds_4":24547},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24494":{"gid":24494,"name":"d","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.2:6837/3754333215","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6836","nonce":3754333215},{"type":"v1","addr":"172.21.15.2:6837","nonce":3754333215}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24511":{"gid":24511,"name":"b","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.47:6833/2783249369","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6832","nonce":2783249369},{"type":"v1","addr":"172.21.15.47:6833","nonce":2783249369}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24523":{"gid":24523,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.47:6835/4075838054","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6834","nonce":4075838054},{"type":"v1","addr":"172.21.15.47:6835","nonce":4075838054}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24530":{"gid":24530,"name":"j","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.2:6841/522559555","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6840","nonce":522559555},{"type":"v1","addr":"172.21.15.2:6841","nonce":522559555}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24535":{"gid":24535,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.47:6837/4072863000","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6836","nonce":4072863000},{"type":"v1","addr":"172.21.15.47:6837","nonce":4072863000}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24547":{"gid":24547,"name":"k","rank":4,"incarnation":24,"state":"up:active","state_seq":7,"addr":"172.21.15.47:6839/761355251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6838","nonce":761355251},{"type":"v1","addr":"172.21.15.47:6839","nonce":761355251}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24575":{"gid":24575,"name":"c","rank":3,"incarnation":22,"state":"up:active","state_seq":6,"addr":"172.21.15.115:6833/381933929","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6832","nonce":381933929},{"type":"v1","addr":"172.21.15.115:6833","nonce":381933929}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24587":{"gid":24587,"name":"f","rank":2,"incarnation":20,"state":"up:active","state_seq":5,"addr":"172.21.15.115:6835/71062276","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6834","nonce":71062276},{"type":"v1","addr":"172.21.15.115:6835","nonce":71062276}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24599":{"gid":24599,"name":"i","rank":1,"incarnation":18,"state":"up:active","state_seq":4,"addr":"172.21.15.115:6837/2940957881","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6836","nonce":2940957881},{"type":"v1","addr":"172.21.15.115:6837","nonce":2940957881}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24611":{"gid":24611,"name":"l","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.115:6839/4149542923","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6838","nonce":4149542923},{"type":"v1","addr":"172.21.15.115:6839","nonce":4149542923}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-05-13T23:15:28.589 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:28.587+0000 7f3d33fff700 1 -- 172.21.15.2:0/1549106648 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f3d1c096ce0 msgr2=0x7f3d1c099190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:15:28.590 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:28.587+0000 7f3d33fff700 1 --2- 172.21.15.2:0/1549106648 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f3d1c096ce0 0x7f3d1c099190 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f3d24009da0 tx=0x7f3d24009cb0 comp rx=0 tx=0).stop 2022-05-13T23:15:28.592 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:28.587+0000 7f3d33fff700 1 -- 172.21.15.2:0/1549106648 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3d340fd110 msgr2=0x7f3d341137e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:15:28.592 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:28.587+0000 7f3d33fff700 1 --2- 172.21.15.2:0/1549106648 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3d340fd110 0x7f3d341137e0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f3d2c00ec20 tx=0x7f3d2c00b040 comp rx=0 tx=0).stop 2022-05-13T23:15:28.592 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:28.588+0000 7f3d33fff700 1 -- 172.21.15.2:0/1549106648 shutdown_connections 2022-05-13T23:15:28.592 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:28.588+0000 7f3d33fff700 1 --2- 172.21.15.2:0/1549106648 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3d340fd110 0x7f3d341137e0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:28.593 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:28.588+0000 7f3d33fff700 1 --2- 172.21.15.2:0/1549106648 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f3d340fb4e0 0x7f3d3410c210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:28.593 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:28.588+0000 7f3d33fff700 1 --2- 172.21.15.2:0/1549106648 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f3d1c096ce0 0x7f3d1c099190 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:28.594 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:28.588+0000 7f3d33fff700 1 --2- 172.21.15.2:0/1549106648 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3d340fc780 0x7f3d3410c750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:28.594 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:28.588+0000 7f3d33fff700 1 -- 172.21.15.2:0/1549106648 >> 172.21.15.2:0/1549106648 conn(0x7f3d340f6cb0 msgr2=0x7f3d340ff9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-05-13T23:15:28.594 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:28.588+0000 7f3d33fff700 1 -- 172.21.15.2:0/1549106648 shutdown_connections 2022-05-13T23:15:28.594 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:28.588+0000 7f3d33fff700 1 -- 172.21.15.2:0/1549106648 wait complete. 2022-05-13T23:15:28.595 INFO:teuthology.orchestra.run.smithi002.stderr:dumped fsmap epoch 26 2022-05-13T23:15:28.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:15:28 smithi115 ceph-mon[78593]: from='client.? 172.21.15.2:0/1344170588' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2022-05-13T23:15:28.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:15:28 smithi115 ceph-mon[78593]: pgmap v326: 97 pgs: 97 active+clean; 584 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2022-05-13T23:15:28.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:15:28 smithi115 ceph-mon[78593]: from='client.? 172.21.15.2:0/1129361427' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2022-05-13T23:15:28.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:15:28 smithi115 ceph-mon[78593]: from='client.? 172.21.15.2:0/2723773940' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2022-05-13T23:15:28.617 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 26, 'flags': 50, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2022-05-13T23:14:16.236645+0000', 'modified': '2022-05-13T23:14:23.309447+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 24611, 'mds_1': 24599, 'mds_2': 24587, 'mds_3': 24575, 'mds_4': 24547}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24494': {'gid': 24494, 'name': 'd', 'rank': 4, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.2:6837/3754333215', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.2:6836', 'nonce': 3754333215}, {'type': 'v1', 'addr': '172.21.15.2:6837', 'nonce': 3754333215}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138303579357183, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}, 'gid_24511': {'gid': 24511, 'name': 'b', 'rank': 3, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.47:6833/2783249369', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.47:6832', 'nonce': 2783249369}, {'type': 'v1', 'addr': '172.21.15.47:6833', 'nonce': 2783249369}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138303579357183, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}, 'gid_24523': {'gid': 24523, 'name': 'e', 'rank': 2, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.47:6835/4075838054', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.47:6834', 'nonce': 4075838054}, {'type': 'v1', 'addr': '172.21.15.47:6835', 'nonce': 4075838054}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138303579357183, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}, 'gid_24530': {'gid': 24530, 'name': 'j', 'rank': 1, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.2:6841/522559555', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.2:6840', 'nonce': 522559555}, {'type': 'v1', 'addr': '172.21.15.2:6841', 'nonce': 522559555}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138303579357183, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}, 'gid_24535': {'gid': 24535, 'name': 'h', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.47:6837/4072863000', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.47:6836', 'nonce': 4072863000}, {'type': 'v1', 'addr': '172.21.15.47:6837', 'nonce': 4072863000}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138303579357183, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}, 'gid_24547': {'gid': 24547, 'name': 'k', 'rank': 4, 'incarnation': 24, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.47:6839/761355251', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.47:6838', 'nonce': 761355251}, {'type': 'v1', 'addr': '172.21.15.47:6839', 'nonce': 761355251}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138303579357183, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}, 'gid_24575': {'gid': 24575, 'name': 'c', 'rank': 3, 'incarnation': 22, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.115:6833/381933929', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.115:6832', 'nonce': 381933929}, {'type': 'v1', 'addr': '172.21.15.115:6833', 'nonce': 381933929}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138303579357183, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}, 'gid_24587': {'gid': 24587, 'name': 'f', 'rank': 2, 'incarnation': 20, 'state': 'up:active', 'state_seq': 5, 'addr': '172.21.15.115:6835/71062276', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.115:6834', 'nonce': 71062276}, {'type': 'v1', 'addr': '172.21.15.115:6835', 'nonce': 71062276}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138303579357183, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}, 'gid_24599': {'gid': 24599, 'name': 'i', 'rank': 1, 'incarnation': 18, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.115:6837/2940957881', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.115:6836', 'nonce': 2940957881}, {'type': 'v1', 'addr': '172.21.15.115:6837', 'nonce': 2940957881}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138303579357183, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}, 'gid_24611': {'gid': 24611, 'name': 'l', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.115:6839/4149542923', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.115:6838', 'nonce': 4149542923}, {'type': 'v1', 'addr': '172.21.15.115:6839', 'nonce': 4149542923}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138303579357183, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2022-05-13T23:15:28.618 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 5/5 2022-05-13T23:15:28.618 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.k status 2022-05-13T23:15:28.836 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:28.835+0000 7f415c9fe700 1 -- 172.21.15.2:0/1788775047 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f415806ee10 msgr2=0x7f415806f280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:15:28.837 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:28.835+0000 7f415c9fe700 1 --2- 172.21.15.2:0/1788775047 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f415806ee10 0x7f415806f280 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f414c0047b0 tx=0x7f414c009cb0 comp rx=0 tx=0).stop 2022-05-13T23:15:28.837 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:28.835+0000 7f415c9fe700 1 -- 172.21.15.2:0/1788775047 shutdown_connections 2022-05-13T23:15:28.838 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:28.835+0000 7f415c9fe700 1 --2- 172.21.15.2:0/1788775047 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f41580fefb0 0x7f4158109bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:28.838 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:28.835+0000 7f415c9fe700 1 --2- 172.21.15.2:0/1788775047 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f41580703d0 0x7f415806e8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:28.838 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:28.835+0000 7f415c9fe700 1 --2- 172.21.15.2:0/1788775047 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f415806ee10 0x7f415806f280 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:28.838 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:28.835+0000 7f415c9fe700 1 -- 172.21.15.2:0/1788775047 >> 172.21.15.2:0/1788775047 conn(0x7f41580f6c40 msgr2=0x7f41580f90b0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-05-13T23:15:28.839 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:28.835+0000 7f415c9fe700 1 -- 172.21.15.2:0/1788775047 shutdown_connections 2022-05-13T23:15:28.839 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:28.835+0000 7f415c9fe700 1 -- 172.21.15.2:0/1788775047 wait complete. 2022-05-13T23:15:28.854 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:28.836+0000 7f415c9fe700 1 Processor -- start 2022-05-13T23:15:28.855 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:28.836+0000 7f415c9fe700 1 -- start start 2022-05-13T23:15:28.855 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:28.836+0000 7f415c9fe700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f415806ee10 0x7f41581128f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:15:28.856 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:28.836+0000 7f415c9fe700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f41580703d0 0x7f4158112e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:15:28.856 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:28.836+0000 7f415c9fe700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f41580fefb0 0x7f4158119ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:15:28.857 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:28.836+0000 7f415c9fe700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f415811a400 con 0x7f41580703d0 2022-05-13T23:15:28.857 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:28.836+0000 7f415c9fe700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f415811a570 con 0x7f415806ee10 2022-05-13T23:15:28.857 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:28.836+0000 7f415c9fe700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f415811a730 con 0x7f41580fefb0 2022-05-13T23:15:28.857 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:28.837+0000 7f4156d9d700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f415806ee10 0x7f41581128f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:15:28.858 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:28.837+0000 7f4156d9d700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f415806ee10 0x7f41581128f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.2:41410/0 (socket says 172.21.15.2:41410) 2022-05-13T23:15:28.858 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:28.837+0000 7f4156d9d700 1 -- 172.21.15.2:0/2602020166 learned_addr learned my addr 172.21.15.2:0/2602020166 (peer_addr_for_me v2:172.21.15.2:0/0) 2022-05-13T23:15:28.858 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:28.837+0000 7f415659c700 1 --2- 172.21.15.2:0/2602020166 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f41580703d0 0x7f4158112e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:15:28.858 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:28.837+0000 7f415759e700 1 --2- 172.21.15.2:0/2602020166 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f41580fefb0 0x7f4158119ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:15:28.859 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:28.837+0000 7f4156d9d700 1 -- 172.21.15.2:0/2602020166 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f41580fefb0 msgr2=0x7f4158119ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:15:28.859 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:28.837+0000 7f4156d9d700 1 --2- 172.21.15.2:0/2602020166 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f41580fefb0 0x7f4158119ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:28.859 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:28.837+0000 7f4156d9d700 1 -- 172.21.15.2:0/2602020166 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f41580703d0 msgr2=0x7f4158112e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:15:28.859 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:28.837+0000 7f4156d9d700 1 --2- 172.21.15.2:0/2602020166 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f41580703d0 0x7f4158112e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:28.859 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:28.837+0000 7f4156d9d700 1 -- 172.21.15.2:0/2602020166 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f414c005040 con 0x7f415806ee10 2022-05-13T23:15:28.860 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:28.837+0000 7f4156d9d700 1 --2- 172.21.15.2:0/2602020166 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f415806ee10 0x7f41581128f0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f414800e970 tx=0x7f414800b040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-05-13T23:15:28.860 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:28.838+0000 7f413f7fe700 1 -- 172.21.15.2:0/2602020166 <== mon.1 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4148007b40 con 0x7f415806ee10 2022-05-13T23:15:28.860 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:28.838+0000 7f415c9fe700 1 -- 172.21.15.2:0/2602020166 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f415811aa10 con 0x7f415806ee10 2022-05-13T23:15:28.860 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:28.838+0000 7f413f7fe700 1 -- 172.21.15.2:0/2602020166 <== mon.1 v2:172.21.15.47:3300/0 2 ==== config(26 keys) v1 ==== 1087+0+0 (secure 0 0 0) 0x7f414800f040 con 0x7f415806ee10 2022-05-13T23:15:28.861 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:28.838+0000 7f413f7fe700 1 -- 172.21.15.2:0/2602020166 <== mon.1 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4148014950 con 0x7f415806ee10 2022-05-13T23:15:28.861 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:28.838+0000 7f415c9fe700 1 -- 172.21.15.2:0/2602020166 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f415811af80 con 0x7f415806ee10 2022-05-13T23:15:28.862 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:28.841+0000 7f413f7fe700 1 -- 172.21.15.2:0/2602020166 <== mon.1 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 16) v1 ==== 125136+0+0 (secure 0 0 0) 0x7f414801b030 con 0x7f415806ee10 2022-05-13T23:15:28.862 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:28.842+0000 7f416c436b80 1 Processor -- start 2022-05-13T23:15:28.862 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:28.843+0000 7f416c436b80 1 -- start start 2022-05-13T23:15:28.862 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:28.843+0000 7f413f7fe700 1 --2- 172.21.15.2:0/2602020166 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f4140096ce0 0x7f4140099190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:15:28.863 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:28.843+0000 7f416c436b80 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55ae01f1acf0 0x55ae01f1b100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:15:28.863 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:28.843+0000 7f413f7fe700 1 -- 172.21.15.2:0/2602020166 <== mon.1 v2:172.21.15.47:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8120+0+0 (secure 0 0 0) 0x7f41480bea00 con 0x7f415806ee10 2022-05-13T23:15:28.863 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:28.843+0000 7f416c436b80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55ae01f1b640 0x55ae01f1bf80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:15:28.864 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:28.843+0000 7f416c436b80 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55ae01f1c5a0 0x55ae01f1c9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:15:28.864 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:28.843+0000 7f416c436b80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ae01e153a0 con 0x55ae01f1b640 2022-05-13T23:15:28.864 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:28.843+0000 7f416c436b80 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ae01dea400 con 0x55ae01f1c5a0 2022-05-13T23:15:28.864 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:28.843+0000 7f416c436b80 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ae01e6e6c0 con 0x55ae01f1acf0 2022-05-13T23:15:28.865 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:28.843+0000 7f415659c700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55ae01f1b640 0x55ae01f1bf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:15:28.865 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:28.843+0000 7f4156d9d700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55ae01f1acf0 0x55ae01f1b100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:15:28.865 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:28.843+0000 7f415759e700 1 --2- 172.21.15.2:0/2602020166 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f4140096ce0 0x7f4140099190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:15:28.865 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:28.843+0000 7f415659c700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55ae01f1b640 0x55ae01f1bf80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:46132/0 (socket says 172.21.15.2:46132) 2022-05-13T23:15:28.866 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:28.843+0000 7f415659c700 1 -- 172.21.15.2:0/2213052860 learned_addr learned my addr 172.21.15.2:0/2213052860 (peer_addr_for_me v2:172.21.15.2:0/0) 2022-05-13T23:15:28.866 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:28.844+0000 7f415759e700 1 --2- 172.21.15.2:0/2213052860 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55ae01f1c5a0 0x55ae01f1c9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:15:28.866 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:28.844+0000 7f4156d9d700 1 -- 172.21.15.2:0/2213052860 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55ae01f1c5a0 msgr2=0x55ae01f1c9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-05-13T23:15:28.866 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:28.844+0000 7f4156d9d700 1 --2- 172.21.15.2:0/2213052860 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55ae01f1c5a0 0x55ae01f1c9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:28.867 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:28.844+0000 7f4156d9d700 1 -- 172.21.15.2:0/2213052860 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55ae01f1b640 msgr2=0x55ae01f1bf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-05-13T23:15:28.867 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:28.844+0000 7f4156d9d700 1 --2- 172.21.15.2:0/2213052860 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55ae01f1b640 0x55ae01f1bf80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:28.867 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:28.844+0000 7f4156d9d700 1 -- 172.21.15.2:0/2213052860 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ae01d00360 con 0x55ae01f1acf0 2022-05-13T23:15:28.867 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:28.844+0000 7f4156d9d700 1 --2- 172.21.15.2:0/2213052860 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55ae01f1acf0 0x55ae01f1b100 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f414808f670 tx=0x7f414808b610 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7b09ee2ba5c1232c server_cookie=0 in_seq=0 out_seq=0 2022-05-13T23:15:28.868 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:28.844+0000 7f415759e700 1 --2- 172.21.15.2:0/2602020166 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f4140096ce0 0x7f4140099190 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f4150004fd0 tx=0x7f415000b040 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-05-13T23:15:28.868 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:28.844+0000 7f413c9fe700 1 -- 172.21.15.2:0/2213052860 <== mon.2 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f414808f2d0 con 0x55ae01f1acf0 2022-05-13T23:15:28.868 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:28.844+0000 7f413c9fe700 1 -- 172.21.15.2:0/2213052860 <== mon.2 v2:172.21.15.115:3300/0 2 ==== config(26 keys) v1 ==== 1087+0+0 (secure 0 0 0) 0x7f4148090070 con 0x55ae01f1acf0 2022-05-13T23:15:28.868 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:28.844+0000 7f413c9fe700 1 -- 172.21.15.2:0/2213052860 <== mon.2 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f414808fe40 con 0x55ae01f1acf0 2022-05-13T23:15:28.868 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:28.845+0000 7f416c436b80 1 -- 172.21.15.2:0/2213052860 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55ae01f1acf0 msgr2=0x55ae01f1b100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:15:28.869 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:28.845+0000 7f416c436b80 1 --2- 172.21.15.2:0/2213052860 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55ae01f1acf0 0x55ae01f1b100 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f414808f670 tx=0x7f414808b610 comp rx=0 tx=0).stop 2022-05-13T23:15:28.869 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:28.845+0000 7f416c436b80 1 -- 172.21.15.2:0/2213052860 shutdown_connections 2022-05-13T23:15:28.869 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:28.845+0000 7f416c436b80 1 --2- 172.21.15.2:0/2213052860 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55ae01f1c5a0 0x55ae01f1c9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:28.869 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:28.845+0000 7f416c436b80 1 --2- 172.21.15.2:0/2213052860 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55ae01f1acf0 0x55ae01f1b100 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:28.870 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:28.845+0000 7f416c436b80 1 --2- 172.21.15.2:0/2213052860 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55ae01f1b640 0x55ae01f1bf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:28.870 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:28.845+0000 7f416c436b80 1 -- 172.21.15.2:0/2213052860 >> 172.21.15.2:0/2213052860 conn(0x55ae01f15b10 msgr2=0x55ae01f15f00 unknown :-1 s=STATE_NONE l=0).mark_down 2022-05-13T23:15:28.870 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:28.845+0000 7f416c436b80 1 -- 172.21.15.2:0/2213052860 shutdown_connections 2022-05-13T23:15:28.870 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:28.845+0000 7f416c436b80 1 -- 172.21.15.2:0/2213052860 wait complete. 2022-05-13T23:15:28.870 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:28.846+0000 7f416c436b80 1 -- start start 2022-05-13T23:15:28.871 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:28.846+0000 7f416c436b80 1 Processor -- start 2022-05-13T23:15:28.871 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:28.847+0000 7f416c436b80 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55ae01f1acf0 0x55ae01f24540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:15:28.871 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:28.847+0000 7f416c436b80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55ae01f1b640 0x55ae01f24a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:15:28.871 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:28.847+0000 7f416c436b80 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55ae01f1c5a0 0x55ae01f2bae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:15:28.872 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:28.847+0000 7f416c436b80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ae01e687a0 con 0x55ae01f1b640 2022-05-13T23:15:28.872 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:28.847+0000 7f416c436b80 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ae01d02650 con 0x55ae01f1acf0 2022-05-13T23:15:28.872 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:28.847+0000 7f416c436b80 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ae01f2c110 con 0x55ae01f1c5a0 2022-05-13T23:15:28.872 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:28.847+0000 7f416c436b80 10 client.0 _ll_register_callbacks cb 0x55ae01edcde0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2022-05-13T23:15:28.873 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:28.847+0000 7f415659c700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55ae01f1b640 0x55ae01f24a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:15:28.873 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:28.847+0000 7f4156d9d700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55ae01f1acf0 0x55ae01f24540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:15:28.873 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:28.847+0000 7f4156d9d700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55ae01f1acf0 0x55ae01f24540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.2:41426/0 (socket says 172.21.15.2:41426) 2022-05-13T23:15:28.873 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:28.847+0000 7f4156d9d700 1 -- 172.21.15.2:0/1629696444 learned_addr learned my addr 172.21.15.2:0/1629696444 (peer_addr_for_me v2:172.21.15.2:0/0) 2022-05-13T23:15:28.874 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:28.847+0000 7f415659c700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55ae01f1b640 0x55ae01f24a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:46148/0 (socket says 172.21.15.2:46148) 2022-05-13T23:15:28.874 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:28.847+0000 7f415759e700 1 --2- 172.21.15.2:0/1629696444 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55ae01f1c5a0 0x55ae01f2bae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:15:28.874 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:28.848+0000 7f415659c700 1 -- 172.21.15.2:0/1629696444 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55ae01f1c5a0 msgr2=0x55ae01f2bae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-05-13T23:15:28.875 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:28.848+0000 7f415659c700 1 --2- 172.21.15.2:0/1629696444 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55ae01f1c5a0 0x55ae01f2bae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:28.875 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:28.848+0000 7f415659c700 1 -- 172.21.15.2:0/1629696444 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55ae01f1acf0 msgr2=0x55ae01f24540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-05-13T23:15:28.875 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:28.848+0000 7f415659c700 1 --2- 172.21.15.2:0/1629696444 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55ae01f1acf0 0x55ae01f24540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:28.875 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:28.848+0000 7f415659c700 1 -- 172.21.15.2:0/1629696444 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f415811af80 con 0x55ae01f1b640 2022-05-13T23:15:28.876 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:28.848+0000 7f415759e700 1 --2- 172.21.15.2:0/1629696444 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55ae01f1c5a0 0x55ae01f2bae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-05-13T23:15:28.876 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:28.848+0000 7f4156d9d700 1 --2- 172.21.15.2:0/1629696444 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55ae01f1acf0 0x55ae01f24540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-05-13T23:15:28.876 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:28.848+0000 7f415659c700 1 --2- 172.21.15.2:0/1629696444 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55ae01f1b640 0x55ae01f24a80 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f414c009d60 tx=0x7f414c009cb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c8564cb49235e7a9 server_cookie=0 in_seq=0 out_seq=0 2022-05-13T23:15:28.877 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:28.848+0000 7f4135ffb700 10 client.0 ms_handle_connect on v2:172.21.15.2:3300/0 2022-05-13T23:15:28.877 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:28.848+0000 7f4135ffb700 1 -- 172.21.15.2:0/1629696444 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f414c007b40 con 0x55ae01f1b640 2022-05-13T23:15:28.877 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:28.848+0000 7f416c436b80 1 -- 172.21.15.2:0/1629696444 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55ae01e1c9a0 con 0x55ae01f1b640 2022-05-13T23:15:28.877 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:28.848+0000 7f4135ffb700 1 -- 172.21.15.2:0/1629696444 <== mon.0 v2:172.21.15.2:3300/0 2 ==== osd_map(85..85 src has 1..85) v4 ==== 8120+0+0 (secure 0 0 0) 0x7f414c007ca0 con 0x55ae01f1b640 2022-05-13T23:15:28.878 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:28.848+0000 7f4135ffb700 1 -- 172.21.15.2:0/1629696444 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7f414c007ea0 con 0x55ae01f1b640 2022-05-13T23:15:28.878 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:28.848+0000 7f416c436b80 10 client.14841 fetch_fsmap learned FSMap version 26 2022-05-13T23:15:28.878 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:28.849+0000 7f416c436b80 1 -- 172.21.15.2:0/1629696444 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) v3 -- 0x55ae01caafe0 con 0x55ae01f1b640 2022-05-13T23:15:28.878 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:28.849+0000 7f4135ffb700 1 -- 172.21.15.2:0/1629696444 <== mon.0 v2:172.21.15.2:3300/0 4 ==== config(26 keys) v1 ==== 1087+0+0 (secure 0 0 0) 0x55ae01e1c9a0 con 0x55ae01f1b640 2022-05-13T23:15:28.878 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:28.849+0000 7f4135ffb700 1 -- 172.21.15.2:0/1629696444 <== mon.0 v2:172.21.15.2:3300/0 5 ==== fsmap(e 26) v1 ==== 7448+0+0 (secure 0 0 0) 0x7f414c009450 con 0x55ae01f1b640 2022-05-13T23:15:28.879 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:28.849+0000 7f416c436b80 10 client.14841 fetch_fsmap finished waiting for FSMap version 26 2022-05-13T23:15:28.879 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:28.850+0000 7f416c436b80 10 client.14841 resolve_mds: resolved name 'k' to daemon mds.k 2022-05-13T23:15:28.879 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:28.850+0000 7f416c436b80 20 client.14841 populate_metadata read hostname 'smithi002' 2022-05-13T23:15:28.879 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:28.850+0000 7f416c436b80 1 --2- 172.21.15.2:0/1629696444 >> [v2:172.21.15.47:6838/761355251,v1:172.21.15.47:6839/761355251] conn(0x55ae01f2f1d0 0x55ae01f344d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:15:28.880 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:28.850+0000 7f416c436b80 4 client.14841 mds_command: new command op to 24547 tid=0[{"prefix": "get_command_descriptions"}] 2022-05-13T23:15:28.880 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:28.850+0000 7f416c436b80 1 -- 172.21.15.2:0/1629696444 --> [v2:172.21.15.47:6838/761355251,v1:172.21.15.47:6839/761355251] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55ae01e3d330 con 0x55ae01f2f1d0 2022-05-13T23:15:28.880 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:28.850+0000 7f4156d9d700 1 --2- 172.21.15.2:0/1629696444 >> [v2:172.21.15.47:6838/761355251,v1:172.21.15.47:6839/761355251] conn(0x55ae01f2f1d0 0x55ae01f344d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:15:28.880 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:28.850+0000 7f4156d9d700 1 --2- 172.21.15.2:0/1629696444 >> [v2:172.21.15.47:6838/761355251,v1:172.21.15.47:6839/761355251] conn(0x55ae01f2f1d0 0x55ae01f344d0 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=1a9d7031c1ec6168 server_cookie=280f9b971121eb0a in_seq=0 out_seq=0 2022-05-13T23:15:28.880 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:28.850+0000 7f4135ffb700 10 client.14841 ms_handle_connect on v2:172.21.15.47:6838/761355251 2022-05-13T23:15:28.881 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:28.851+0000 7f4135ffb700 1 -- 172.21.15.2:0/1629696444 <== mds.4 v2:172.21.15.47:6838/761355251 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17090 (crc 0 0 0) 0x7f41480b6e90 con 0x55ae01f2f1d0 2022-05-13T23:15:28.881 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:28.851+0000 7f4135ffb700 10 client.14841 handle_command_reply: tid=0 2022-05-13T23:15:28.881 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:28.851+0000 7f416c436b80 1 client.14841 shutdown 2022-05-13T23:15:28.881 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:28.852+0000 7f416c436b80 1 -- 172.21.15.2:0/1629696444 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55ae01f1b640 msgr2=0x55ae01f24a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:15:28.882 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:28.852+0000 7f416c436b80 1 --2- 172.21.15.2:0/1629696444 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55ae01f1b640 0x55ae01f24a80 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f414c009d60 tx=0x7f414c009cb0 comp rx=0 tx=0).stop 2022-05-13T23:15:28.882 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:28.852+0000 7f416c436b80 1 -- 172.21.15.2:0/1629696444 shutdown_connections 2022-05-13T23:15:28.882 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:28.852+0000 7f416c436b80 1 --2- 172.21.15.2:0/1629696444 >> [v2:172.21.15.47:6838/761355251,v1:172.21.15.47:6839/761355251] conn(0x55ae01f2f1d0 0x55ae01f344d0 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:28.883 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:28.852+0000 7f416c436b80 1 --2- 172.21.15.2:0/1629696444 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55ae01f1c5a0 0x55ae01f2bae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:28.883 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:28.852+0000 7f416c436b80 1 --2- 172.21.15.2:0/1629696444 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55ae01f1acf0 0x55ae01f24540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:28.883 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:28.852+0000 7f416c436b80 1 --2- 172.21.15.2:0/1629696444 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55ae01f1b640 0x55ae01f24a80 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:28.883 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:28.852+0000 7f4135ffb700 0 client.14841 ms_handle_reset on v2:172.21.15.47:6838/761355251 2022-05-13T23:15:28.884 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:28.852+0000 7f416c436b80 1 -- 172.21.15.2:0/1629696444 >> 172.21.15.2:0/1629696444 conn(0x55ae01f15b10 msgr2=0x55ae01f14d80 unknown :-1 s=STATE_NONE l=0).mark_down 2022-05-13T23:15:28.884 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:28.852+0000 7f416c436b80 1 -- 172.21.15.2:0/1629696444 shutdown_connections 2022-05-13T23:15:28.884 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:28.852+0000 7f416c436b80 1 -- 172.21.15.2:0/1629696444 wait complete. 2022-05-13T23:15:28.884 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:28.852+0000 7f416c436b80 20 client.14841 trim_cache size 0 max 16384 2022-05-13T23:15:28.894 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:28.870+0000 7f416c436b80 1 Processor -- start 2022-05-13T23:15:28.895 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:28.870+0000 7f416c436b80 1 -- start start 2022-05-13T23:15:28.895 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:28.870+0000 7f416c436b80 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55ae01f1acf0 0x55ae01f384f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:15:28.895 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:28.870+0000 7f416c436b80 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55ae01f1b640 0x55ae01f23cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:15:28.895 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:28.870+0000 7f416c436b80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55ae01f1c5a0 0x55ae01f14300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:15:28.896 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:28.871+0000 7f416c436b80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ae01e31be0 con 0x55ae01f1c5a0 2022-05-13T23:15:28.896 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:28.871+0000 7f416c436b80 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ae01e1cde0 con 0x55ae01f1acf0 2022-05-13T23:15:28.896 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:28.871+0000 7f416c436b80 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ae01e20a00 con 0x55ae01f1b640 2022-05-13T23:15:28.896 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:28.871+0000 7f4156d9d700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55ae01f1acf0 0x55ae01f384f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:15:28.896 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:28.871+0000 7f4156d9d700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55ae01f1acf0 0x55ae01f384f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.2:41432/0 (socket says 172.21.15.2:41432) 2022-05-13T23:15:28.897 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:28.871+0000 7f4156d9d700 1 -- 172.21.15.2:0/723833817 learned_addr learned my addr 172.21.15.2:0/723833817 (peer_addr_for_me v2:172.21.15.2:0/0) 2022-05-13T23:15:28.897 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:28.871+0000 7f415759e700 1 --2- 172.21.15.2:0/723833817 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55ae01f1c5a0 0x55ae01f14300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:15:28.897 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:28.871+0000 7f415659c700 1 --2- 172.21.15.2:0/723833817 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55ae01f1b640 0x55ae01f23cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:15:28.897 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:28.871+0000 7f4156d9d700 1 -- 172.21.15.2:0/723833817 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55ae01f1b640 msgr2=0x55ae01f23cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-05-13T23:15:28.898 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:28.871+0000 7f4156d9d700 1 --2- 172.21.15.2:0/723833817 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55ae01f1b640 0x55ae01f23cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:28.898 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:28.871+0000 7f4156d9d700 1 -- 172.21.15.2:0/723833817 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55ae01f1c5a0 msgr2=0x55ae01f14300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-05-13T23:15:28.898 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:28.871+0000 7f4156d9d700 1 --2- 172.21.15.2:0/723833817 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55ae01f1c5a0 0x55ae01f14300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:28.898 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:28.871+0000 7f4156d9d700 1 -- 172.21.15.2:0/723833817 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ae01cd2f80 con 0x55ae01f1acf0 2022-05-13T23:15:28.898 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:28.872+0000 7f4156d9d700 1 --2- 172.21.15.2:0/723833817 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55ae01f1acf0 0x55ae01f384f0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f414808f900 tx=0x7f4148002040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=98d85ddae6955fa server_cookie=0 in_seq=0 out_seq=0 2022-05-13T23:15:28.899 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:28.872+0000 7f413d1ff700 1 -- 172.21.15.2:0/723833817 <== mon.1 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f414808c080 con 0x55ae01f1acf0 2022-05-13T23:15:28.899 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:28.872+0000 7f413d1ff700 1 -- 172.21.15.2:0/723833817 <== mon.1 v2:172.21.15.47:3300/0 2 ==== config(26 keys) v1 ==== 1087+0+0 (secure 0 0 0) 0x7f41480b6e90 con 0x55ae01f1acf0 2022-05-13T23:15:28.899 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:28.872+0000 7f413d1ff700 1 -- 172.21.15.2:0/723833817 <== mon.1 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f414808f070 con 0x55ae01f1acf0 2022-05-13T23:15:28.899 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:28.872+0000 7f416c436b80 1 -- 172.21.15.2:0/723833817 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55ae01f1acf0 msgr2=0x55ae01f384f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:15:28.900 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:28.872+0000 7f416c436b80 1 --2- 172.21.15.2:0/723833817 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55ae01f1acf0 0x55ae01f384f0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f414808f900 tx=0x7f4148002040 comp rx=0 tx=0).stop 2022-05-13T23:15:28.900 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:28.872+0000 7f416c436b80 1 -- 172.21.15.2:0/723833817 shutdown_connections 2022-05-13T23:15:28.900 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:28.872+0000 7f416c436b80 1 --2- 172.21.15.2:0/723833817 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55ae01f1c5a0 0x55ae01f14300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:28.900 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:28.872+0000 7f416c436b80 1 --2- 172.21.15.2:0/723833817 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55ae01f1acf0 0x55ae01f384f0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:28.900 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:28.872+0000 7f416c436b80 1 --2- 172.21.15.2:0/723833817 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55ae01f1b640 0x55ae01f23cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:28.901 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:28.872+0000 7f416c436b80 1 -- 172.21.15.2:0/723833817 >> 172.21.15.2:0/723833817 conn(0x55ae01f15b10 msgr2=0x55ae01f14d80 unknown :-1 s=STATE_NONE l=0).mark_down 2022-05-13T23:15:28.901 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:28.873+0000 7f416c436b80 1 -- 172.21.15.2:0/723833817 shutdown_connections 2022-05-13T23:15:28.901 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:28.873+0000 7f416c436b80 1 -- 172.21.15.2:0/723833817 wait complete. 2022-05-13T23:15:28.901 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:28.873+0000 7f416c436b80 1 -- start start 2022-05-13T23:15:28.901 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:28.873+0000 7f416c436b80 1 Processor -- start 2022-05-13T23:15:28.902 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:28.874+0000 7f416c436b80 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55ae01f1acf0 0x55ae01f28be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:15:28.902 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:28.874+0000 7f416c436b80 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55ae01f1b640 0x55ae01f29120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:15:28.902 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:28.874+0000 7f416c436b80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55ae01f1c5a0 0x55ae01f814b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:15:28.902 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:28.874+0000 7f416c436b80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ae01f81ac0 con 0x55ae01f1c5a0 2022-05-13T23:15:28.903 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:28.874+0000 7f416c436b80 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ae01f81c30 con 0x55ae01f1b640 2022-05-13T23:15:28.903 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:28.874+0000 7f416c436b80 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ae01f81df0 con 0x55ae01f1acf0 2022-05-13T23:15:28.903 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:28.874+0000 7f416c436b80 10 client.0 _ll_register_callbacks cb 0x55ae01edcde0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2022-05-13T23:15:28.903 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:28.874+0000 7f415759e700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55ae01f1c5a0 0x55ae01f814b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:15:28.903 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:28.874+0000 7f4156d9d700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55ae01f1acf0 0x55ae01f28be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:15:28.904 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:28.874+0000 7f4156d9d700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55ae01f1acf0 0x55ae01f28be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.2:53564/0 (socket says 172.21.15.2:53564) 2022-05-13T23:15:28.904 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:28.874+0000 7f4156d9d700 1 -- 172.21.15.2:0/4147802839 learned_addr learned my addr 172.21.15.2:0/4147802839 (peer_addr_for_me v2:172.21.15.2:0/0) 2022-05-13T23:15:28.905 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:28.874+0000 7f415659c700 1 --2- 172.21.15.2:0/4147802839 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55ae01f1b640 0x55ae01f29120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:15:28.905 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:28.874+0000 7f4156d9d700 1 -- 172.21.15.2:0/4147802839 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55ae01f1b640 msgr2=0x55ae01f29120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-05-13T23:15:28.905 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:28.874+0000 7f4156d9d700 1 --2- 172.21.15.2:0/4147802839 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55ae01f1b640 0x55ae01f29120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:28.905 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:28.874+0000 7f4156d9d700 1 -- 172.21.15.2:0/4147802839 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55ae01f1c5a0 msgr2=0x55ae01f814b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-05-13T23:15:28.906 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:28.874+0000 7f4156d9d700 1 --2- 172.21.15.2:0/4147802839 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55ae01f1c5a0 0x55ae01f814b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:28.906 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:28.874+0000 7f4156d9d700 1 -- 172.21.15.2:0/4147802839 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55ae01d00360 con 0x55ae01f1acf0 2022-05-13T23:15:28.906 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:28.875+0000 7f4156d9d700 1 --2- 172.21.15.2:0/4147802839 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55ae01f1acf0 0x55ae01f28be0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f4148009900 tx=0x7f4148002040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a7db19a9a379e0dd server_cookie=0 in_seq=0 out_seq=0 2022-05-13T23:15:28.906 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:28.875+0000 7f4135ffb700 10 client.0 ms_handle_connect on v2:172.21.15.115:3300/0 2022-05-13T23:15:28.906 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:28.875+0000 7f4135ffb700 1 -- 172.21.15.2:0/4147802839 <== mon.2 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4148090070 con 0x55ae01f1acf0 2022-05-13T23:15:28.907 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:28.875+0000 7f4135ffb700 1 -- 172.21.15.2:0/4147802839 <== mon.2 v2:172.21.15.115:3300/0 2 ==== osd_map(85..85 src has 1..85) v4 ==== 8120+0+0 (secure 0 0 0) 0x7f414808f070 con 0x55ae01f1acf0 2022-05-13T23:15:28.907 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:28.875+0000 7f416c436b80 1 -- 172.21.15.2:0/4147802839 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55ae01eeb5f0 con 0x55ae01f1acf0 2022-05-13T23:15:28.907 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:28.875+0000 7f4135ffb700 1 -- 172.21.15.2:0/4147802839 <== mon.2 v2:172.21.15.115:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4148095bd0 con 0x55ae01f1acf0 2022-05-13T23:15:28.907 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:28.875+0000 7f416c436b80 10 client.24965 fetch_fsmap learned FSMap version 26 2022-05-13T23:15:28.908 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:28.875+0000 7f416c436b80 1 -- 172.21.15.2:0/4147802839 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) v3 -- 0x55ae01cacb10 con 0x55ae01f1acf0 2022-05-13T23:15:28.908 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:28.876+0000 7f4135ffb700 1 -- 172.21.15.2:0/4147802839 <== mon.2 v2:172.21.15.115:3300/0 4 ==== config(26 keys) v1 ==== 1087+0+0 (secure 0 0 0) 0x55ae01eeb5f0 con 0x55ae01f1acf0 2022-05-13T23:15:28.908 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:28.876+0000 7f4135ffb700 1 -- 172.21.15.2:0/4147802839 <== mon.2 v2:172.21.15.115:3300/0 5 ==== fsmap(e 26) v1 ==== 7448+0+0 (secure 0 0 0) 0x7f4148097070 con 0x55ae01f1acf0 2022-05-13T23:15:28.908 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:28.876+0000 7f416c436b80 10 client.24965 fetch_fsmap finished waiting for FSMap version 26 2022-05-13T23:15:28.908 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:28.876+0000 7f416c436b80 10 client.24965 resolve_mds: resolved name 'k' to daemon mds.k 2022-05-13T23:15:28.909 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:28.876+0000 7f416c436b80 20 client.24965 populate_metadata read hostname 'smithi002' 2022-05-13T23:15:28.909 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:28.876+0000 7f416c436b80 1 --2- 172.21.15.2:0/4147802839 >> [v2:172.21.15.47:6838/761355251,v1:172.21.15.47:6839/761355251] conn(0x55ae01f86010 0x55ae01f82b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:15:28.909 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:28.876+0000 7f416c436b80 4 client.24965 mds_command: new command op to 24547 tid=0[{"prefix": "status"}] 2022-05-13T23:15:28.909 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:28.876+0000 7f416c436b80 1 -- 172.21.15.2:0/4147802839 --> [v2:172.21.15.47:6838/761355251,v1:172.21.15.47:6839/761355251] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55ae01e0e790 con 0x55ae01f86010 2022-05-13T23:15:28.910 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:28.876+0000 7f415659c700 1 --2- 172.21.15.2:0/4147802839 >> [v2:172.21.15.47:6838/761355251,v1:172.21.15.47:6839/761355251] conn(0x55ae01f86010 0x55ae01f82b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:15:28.910 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:28.877+0000 7f415659c700 1 --2- 172.21.15.2:0/4147802839 >> [v2:172.21.15.47:6838/761355251,v1:172.21.15.47:6839/761355251] conn(0x55ae01f86010 0x55ae01f82b60 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=b573a3b2a0347069 server_cookie=49b234f0fe9bf2db in_seq=0 out_seq=0 2022-05-13T23:15:28.910 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:28.877+0000 7f4135ffb700 10 client.24965 ms_handle_connect on v2:172.21.15.47:6838/761355251 2022-05-13T23:15:28.910 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:28.877+0000 7f4135ffb700 1 -- 172.21.15.2:0/4147802839 <== mds.4 v2:172.21.15.47:6838/761355251 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+332 (crc 0 0 0) 0x55ae01e0e790 con 0x55ae01f86010 2022-05-13T23:15:28.910 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:28.877+0000 7f4135ffb700 10 client.24965 handle_command_reply: tid=0 2022-05-13T23:15:28.911 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:28.877+0000 7f416c436b80 1 client.24965 shutdown 2022-05-13T23:15:28.911 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:28.877+0000 7f416c436b80 1 -- 172.21.15.2:0/4147802839 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55ae01f1acf0 msgr2=0x55ae01f28be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:15:28.911 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:28.877+0000 7f416c436b80 1 --2- 172.21.15.2:0/4147802839 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55ae01f1acf0 0x55ae01f28be0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f4148009900 tx=0x7f4148002040 comp rx=0 tx=0).stop 2022-05-13T23:15:28.911 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:28.877+0000 7f416c436b80 1 -- 172.21.15.2:0/4147802839 shutdown_connections 2022-05-13T23:15:28.911 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:28.877+0000 7f416c436b80 1 --2- 172.21.15.2:0/4147802839 >> [v2:172.21.15.47:6838/761355251,v1:172.21.15.47:6839/761355251] conn(0x55ae01f86010 0x55ae01f82b60 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:28.912 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:28.877+0000 7f416c436b80 1 --2- 172.21.15.2:0/4147802839 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55ae01f1c5a0 0x55ae01f814b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:28.912 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:28.877+0000 7f416c436b80 1 --2- 172.21.15.2:0/4147802839 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55ae01f1acf0 0x55ae01f28be0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:28.912 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:28.877+0000 7f416c436b80 1 --2- 172.21.15.2:0/4147802839 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55ae01f1b640 0x55ae01f29120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:28.912 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:28.877+0000 7f4135ffb700 0 client.24965 ms_handle_reset on v2:172.21.15.47:6838/761355251 2022-05-13T23:15:28.913 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:28.877+0000 7f416c436b80 1 -- 172.21.15.2:0/4147802839 >> 172.21.15.2:0/4147802839 conn(0x55ae01f15b10 msgr2=0x55ae01f384f0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-05-13T23:15:28.913 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:28.877+0000 7f416c436b80 1 -- 172.21.15.2:0/4147802839 shutdown_connections 2022-05-13T23:15:28.913 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:28.877+0000 7f416c436b80 1 -- 172.21.15.2:0/4147802839 wait complete. 2022-05-13T23:15:28.913 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:28.878+0000 7f416c436b80 20 client.24965 trim_cache size 0 max 16384 2022-05-13T23:15:28.914 INFO:teuthology.orchestra.run.smithi002.stdout:{ 2022-05-13T23:15:28.914 INFO:teuthology.orchestra.run.smithi002.stdout: "cluster_fsid": "16cf5f88-d311-11ec-be40-001a4aab830c", 2022-05-13T23:15:28.914 INFO:teuthology.orchestra.run.smithi002.stdout: "whoami": 4, 2022-05-13T23:15:28.915 INFO:teuthology.orchestra.run.smithi002.stdout: "id": 24547, 2022-05-13T23:15:28.915 INFO:teuthology.orchestra.run.smithi002.stdout: "want_state": "up:active", 2022-05-13T23:15:28.916 INFO:teuthology.orchestra.run.smithi002.stdout: "state": "up:active", 2022-05-13T23:15:28.916 INFO:teuthology.orchestra.run.smithi002.stdout: "fs_name": "cephfs", 2022-05-13T23:15:28.916 INFO:teuthology.orchestra.run.smithi002.stdout: "rank_uptime": 66.571310334000003, 2022-05-13T23:15:28.916 INFO:teuthology.orchestra.run.smithi002.stdout: "mdsmap_epoch": 26, 2022-05-13T23:15:28.916 INFO:teuthology.orchestra.run.smithi002.stdout: "osdmap_epoch": 85, 2022-05-13T23:15:28.917 INFO:teuthology.orchestra.run.smithi002.stdout: "osdmap_epoch_barrier": 83, 2022-05-13T23:15:28.917 INFO:teuthology.orchestra.run.smithi002.stdout: "uptime": 88.127762649999994 2022-05-13T23:15:28.917 INFO:teuthology.orchestra.run.smithi002.stdout:} 2022-05-13T23:15:28.919 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:28.878+0000 7f415c9fe700 1 -- 172.21.15.2:0/2602020166 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f4140096ce0 msgr2=0x7f4140099190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:15:28.919 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:28.878+0000 7f415c9fe700 1 --2- 172.21.15.2:0/2602020166 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f4140096ce0 0x7f4140099190 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f4150004fd0 tx=0x7f415000b040 comp rx=0 tx=0).stop 2022-05-13T23:15:28.919 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:28.878+0000 7f415c9fe700 1 -- 172.21.15.2:0/2602020166 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f415806ee10 msgr2=0x7f41581128f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:15:28.920 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:28.878+0000 7f415c9fe700 1 --2- 172.21.15.2:0/2602020166 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f415806ee10 0x7f41581128f0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f414800e970 tx=0x7f414800b040 comp rx=0 tx=0).stop 2022-05-13T23:15:28.920 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:28.878+0000 7f415c9fe700 1 -- 172.21.15.2:0/2602020166 shutdown_connections 2022-05-13T23:15:28.920 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:28.878+0000 7f415c9fe700 1 --2- 172.21.15.2:0/2602020166 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f41580fefb0 0x7f4158119ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:28.920 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:28.878+0000 7f415c9fe700 1 --2- 172.21.15.2:0/2602020166 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f415806ee10 0x7f41581128f0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:28.921 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:28.878+0000 7f415c9fe700 1 --2- 172.21.15.2:0/2602020166 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f4140096ce0 0x7f4140099190 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:28.921 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:28.878+0000 7f415c9fe700 1 --2- 172.21.15.2:0/2602020166 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f41580703d0 0x7f4158112e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:28.921 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:28.878+0000 7f415c9fe700 1 -- 172.21.15.2:0/2602020166 >> 172.21.15.2:0/2602020166 conn(0x7f41580f6c40 msgr2=0x7f41580f9020 unknown :-1 s=STATE_NONE l=0).mark_down 2022-05-13T23:15:28.921 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:28.878+0000 7f415c9fe700 1 -- 172.21.15.2:0/2602020166 shutdown_connections 2022-05-13T23:15:28.921 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:28.878+0000 7f415c9fe700 1 -- 172.21.15.2:0/2602020166 wait complete. 2022-05-13T23:15:28.922 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2022-05-13T23:15:29.140 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.138+0000 7f32de19d700 1 -- 172.21.15.2:0/2288840943 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f32d80fb4e0 msgr2=0x7f32d80fb8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:15:29.140 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.138+0000 7f32de19d700 1 --2- 172.21.15.2:0/2288840943 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f32d80fb4e0 0x7f32d80fb8f0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f32d4004870 tx=0x7f32d4009cb0 comp rx=0 tx=0).stop 2022-05-13T23:15:29.140 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.138+0000 7f32de19d700 1 -- 172.21.15.2:0/2288840943 shutdown_connections 2022-05-13T23:15:29.141 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.138+0000 7f32de19d700 1 --2- 172.21.15.2:0/2288840943 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f32d80fd110 0x7f32d81039d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:29.141 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.138+0000 7f32de19d700 1 --2- 172.21.15.2:0/2288840943 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f32d80fb4e0 0x7f32d80fb8f0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:29.141 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.138+0000 7f32de19d700 1 --2- 172.21.15.2:0/2288840943 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f32d80fc780 0x7f32d80fcbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:29.141 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.138+0000 7f32de19d700 1 -- 172.21.15.2:0/2288840943 >> 172.21.15.2:0/2288840943 conn(0x7f32d80f6cb0 msgr2=0x7f32d80f9100 unknown :-1 s=STATE_NONE l=0).mark_down 2022-05-13T23:15:29.142 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.138+0000 7f32de19d700 1 -- 172.21.15.2:0/2288840943 shutdown_connections 2022-05-13T23:15:29.143 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.138+0000 7f32de19d700 1 -- 172.21.15.2:0/2288840943 wait complete. 2022-05-13T23:15:29.143 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.139+0000 7f32de19d700 1 Processor -- start 2022-05-13T23:15:29.147 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.139+0000 7f32de19d700 1 -- start start 2022-05-13T23:15:29.147 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.139+0000 7f32de19d700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f32d80fb4e0 0x7f32d810c2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:15:29.147 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.139+0000 7f32de19d700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f32d80fc780 0x7f32d810c7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:15:29.147 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.139+0000 7f32de19d700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f32d80fd110 0x7f32d8113880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:15:29.148 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.139+0000 7f32de19d700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32d8113dc0 con 0x7f32d80fc780 2022-05-13T23:15:29.148 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.139+0000 7f32de19d700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32d8113f30 con 0x7f32d80fb4e0 2022-05-13T23:15:29.148 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.139+0000 7f32de19d700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32d81140f0 con 0x7f32d80fd110 2022-05-13T23:15:29.148 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.140+0000 7f32dd19b700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f32d80fb4e0 0x7f32d810c2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:15:29.149 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.140+0000 7f32dd19b700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f32d80fb4e0 0x7f32d810c2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.2:41462/0 (socket says 172.21.15.2:41462) 2022-05-13T23:15:29.150 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.140+0000 7f32dd19b700 1 -- 172.21.15.2:0/467152150 learned_addr learned my addr 172.21.15.2:0/467152150 (peer_addr_for_me v2:172.21.15.2:0/0) 2022-05-13T23:15:29.151 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.140+0000 7f32dc99a700 1 --2- 172.21.15.2:0/467152150 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f32d80fc780 0x7f32d810c7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:15:29.151 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.140+0000 7f32dd99c700 1 --2- 172.21.15.2:0/467152150 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f32d80fd110 0x7f32d8113880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:15:29.151 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.140+0000 7f32dd19b700 1 -- 172.21.15.2:0/467152150 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f32d80fd110 msgr2=0x7f32d8113880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:15:29.152 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.140+0000 7f32dd19b700 1 --2- 172.21.15.2:0/467152150 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f32d80fd110 0x7f32d8113880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:29.152 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.140+0000 7f32dd19b700 1 -- 172.21.15.2:0/467152150 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f32d80fc780 msgr2=0x7f32d810c7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:15:29.152 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.140+0000 7f32dd19b700 1 --2- 172.21.15.2:0/467152150 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f32d80fc780 0x7f32d810c7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:29.152 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.140+0000 7f32dd19b700 1 -- 172.21.15.2:0/467152150 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32d4005040 con 0x7f32d80fb4e0 2022-05-13T23:15:29.153 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.140+0000 7f32dd19b700 1 --2- 172.21.15.2:0/467152150 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f32d80fb4e0 0x7f32d810c2b0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f32d4004710 tx=0x7f32d4009cb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-05-13T23:15:29.153 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.141+0000 7f32cdffb700 1 -- 172.21.15.2:0/467152150 <== mon.1 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32d400e040 con 0x7f32d80fb4e0 2022-05-13T23:15:29.153 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.141+0000 7f32cdffb700 1 -- 172.21.15.2:0/467152150 <== mon.1 v2:172.21.15.47:3300/0 2 ==== config(26 keys) v1 ==== 1087+0+0 (secure 0 0 0) 0x7f32d4004200 con 0x7f32d80fb4e0 2022-05-13T23:15:29.153 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.141+0000 7f32cdffb700 1 -- 172.21.15.2:0/467152150 <== mon.1 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32d401caf0 con 0x7f32d80fb4e0 2022-05-13T23:15:29.154 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.141+0000 7f32de19d700 1 -- 172.21.15.2:0/467152150 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f32d8114370 con 0x7f32d80fb4e0 2022-05-13T23:15:29.154 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.141+0000 7f32de19d700 1 -- 172.21.15.2:0/467152150 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f32d81147a0 con 0x7f32d80fb4e0 2022-05-13T23:15:29.154 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.144+0000 7f32cdffb700 1 -- 172.21.15.2:0/467152150 <== mon.1 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 16) v1 ==== 125136+0+0 (secure 0 0 0) 0x7f32d4009240 con 0x7f32d80fb4e0 2022-05-13T23:15:29.165 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.146+0000 7f32ee635b80 1 Processor -- start 2022-05-13T23:15:29.166 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.146+0000 7f32ee635b80 1 -- start start 2022-05-13T23:15:29.166 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.147+0000 7f32cdffb700 1 --2- 172.21.15.2:0/467152150 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f32c4096ce0 0x7f32c4099190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:15:29.166 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.147+0000 7f32ee635b80 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55cb3d6b9cf0 0x55cb3d6ba100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:15:29.166 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.147+0000 7f32ee635b80 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55cb3d6ba640 0x55cb3d6baf80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:15:29.167 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.147+0000 7f32ee635b80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55cb3d6bb5a0 0x55cb3d6bb9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:15:29.167 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.147+0000 7f32ee635b80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cb3d5b43a0 con 0x55cb3d6bb5a0 2022-05-13T23:15:29.167 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.147+0000 7f32ee635b80 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cb3d589400 con 0x55cb3d6ba640 2022-05-13T23:15:29.168 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.147+0000 7f32ee635b80 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cb3d60d6c0 con 0x55cb3d6b9cf0 2022-05-13T23:15:29.169 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.147+0000 7f32cdffb700 1 -- 172.21.15.2:0/467152150 <== mon.1 v2:172.21.15.47:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8120+0+0 (secure 0 0 0) 0x7f32d40c0440 con 0x7f32d80fb4e0 2022-05-13T23:15:29.170 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.147+0000 7f32dd99c700 1 --2- 172.21.15.2:0/467152150 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f32c4096ce0 0x7f32c4099190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:15:29.170 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.147+0000 7f32dc99a700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55cb3d6ba640 0x55cb3d6baf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:15:29.170 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.147+0000 7f32dd19b700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55cb3d6b9cf0 0x55cb3d6ba100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:15:29.171 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.147+0000 7f32dc99a700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55cb3d6ba640 0x55cb3d6baf80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.2:41466/0 (socket says 172.21.15.2:41466) 2022-05-13T23:15:29.171 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.147+0000 7f32dc99a700 1 -- 172.21.15.2:0/4226022230 learned_addr learned my addr 172.21.15.2:0/4226022230 (peer_addr_for_me v2:172.21.15.2:0/0) 2022-05-13T23:15:29.171 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.147+0000 7f32dd99c700 1 --2- 172.21.15.2:0/4226022230 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55cb3d6bb5a0 0x55cb3d6bb9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:15:29.172 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.147+0000 7f32dd19b700 1 -- 172.21.15.2:0/4226022230 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55cb3d6ba640 msgr2=0x55cb3d6baf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-05-13T23:15:29.172 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.147+0000 7f32dd19b700 1 --2- 172.21.15.2:0/4226022230 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55cb3d6ba640 0x55cb3d6baf80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:29.172 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.147+0000 7f32dd19b700 1 -- 172.21.15.2:0/4226022230 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55cb3d6bb5a0 msgr2=0x55cb3d6bb9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-05-13T23:15:29.172 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.147+0000 7f32dd19b700 1 --2- 172.21.15.2:0/4226022230 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55cb3d6bb5a0 0x55cb3d6bb9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:29.173 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.147+0000 7f32dd19b700 1 -- 172.21.15.2:0/4226022230 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55cb3d49f360 con 0x55cb3d6b9cf0 2022-05-13T23:15:29.173 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.148+0000 7f32dd99c700 1 --2- 172.21.15.2:0/4226022230 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55cb3d6bb5a0 0x55cb3d6bb9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2022-05-13T23:15:29.174 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.148+0000 7f32dd19b700 1 --2- 172.21.15.2:0/4226022230 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55cb3d6b9cf0 0x55cb3d6ba100 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f32d408de20 tx=0x7f32d408d040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5ec7fd103c500017 server_cookie=0 in_seq=0 out_seq=0 2022-05-13T23:15:29.175 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.148+0000 7f32ba9fe700 1 -- 172.21.15.2:0/4226022230 <== mon.2 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32d4090ef0 con 0x55cb3d6b9cf0 2022-05-13T23:15:29.175 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.148+0000 7f32dd99c700 1 --2- 172.21.15.2:0/467152150 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f32c4096ce0 0x7f32c4099190 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f32d0004fd0 tx=0x7f32d000b040 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-05-13T23:15:29.176 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.148+0000 7f32ba9fe700 1 -- 172.21.15.2:0/4226022230 <== mon.2 v2:172.21.15.115:3300/0 2 ==== config(26 keys) v1 ==== 1087+0+0 (secure 0 0 0) 0x7f32d4090d10 con 0x55cb3d6b9cf0 2022-05-13T23:15:29.176 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.148+0000 7f32ba9fe700 1 -- 172.21.15.2:0/4226022230 <== mon.2 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32d408f4f0 con 0x55cb3d6b9cf0 2022-05-13T23:15:29.177 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.149+0000 7f32ee635b80 1 -- 172.21.15.2:0/4226022230 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55cb3d6b9cf0 msgr2=0x55cb3d6ba100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:15:29.177 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.149+0000 7f32ee635b80 1 --2- 172.21.15.2:0/4226022230 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55cb3d6b9cf0 0x55cb3d6ba100 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f32d408de20 tx=0x7f32d408d040 comp rx=0 tx=0).stop 2022-05-13T23:15:29.178 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.149+0000 7f32ee635b80 1 -- 172.21.15.2:0/4226022230 shutdown_connections 2022-05-13T23:15:29.178 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.149+0000 7f32ee635b80 1 --2- 172.21.15.2:0/4226022230 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55cb3d6bb5a0 0x55cb3d6bb9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:29.179 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.149+0000 7f32ee635b80 1 --2- 172.21.15.2:0/4226022230 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55cb3d6b9cf0 0x55cb3d6ba100 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:29.179 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.149+0000 7f32ee635b80 1 --2- 172.21.15.2:0/4226022230 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55cb3d6ba640 0x55cb3d6baf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:29.180 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.149+0000 7f32ee635b80 1 -- 172.21.15.2:0/4226022230 >> 172.21.15.2:0/4226022230 conn(0x55cb3d6b4b10 msgr2=0x55cb3d6b4f00 unknown :-1 s=STATE_NONE l=0).mark_down 2022-05-13T23:15:29.180 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.149+0000 7f32ee635b80 1 -- 172.21.15.2:0/4226022230 shutdown_connections 2022-05-13T23:15:29.181 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.149+0000 7f32ee635b80 1 -- 172.21.15.2:0/4226022230 wait complete. 2022-05-13T23:15:29.181 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.149+0000 7f32ee635b80 1 -- start start 2022-05-13T23:15:29.181 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.150+0000 7f32ee635b80 1 Processor -- start 2022-05-13T23:15:29.182 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.150+0000 7f32ee635b80 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55cb3d6b9cf0 0x55cb3d6c7660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:15:29.182 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.150+0000 7f32ee635b80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55cb3d6ba640 0x55cb3d6c7ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:15:29.183 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.150+0000 7f32ee635b80 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55cb3d6bb5a0 0x55cb3d6cec00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:15:29.183 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.150+0000 7f32ee635b80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cb3d6077a0 con 0x55cb3d6ba640 2022-05-13T23:15:29.184 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.150+0000 7f32ee635b80 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cb3d4a1650 con 0x55cb3d6b9cf0 2022-05-13T23:15:29.184 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.150+0000 7f32ee635b80 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cb3d6cf230 con 0x55cb3d6bb5a0 2022-05-13T23:15:29.185 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.150+0000 7f32ee635b80 10 client.0 _ll_register_callbacks cb 0x55cb3d67bde0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2022-05-13T23:15:29.185 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.151+0000 7f32dd19b700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55cb3d6b9cf0 0x55cb3d6c7660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:15:29.186 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.151+0000 7f32dd19b700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55cb3d6b9cf0 0x55cb3d6c7660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.2:41478/0 (socket says 172.21.15.2:41478) 2022-05-13T23:15:29.186 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.151+0000 7f32dd19b700 1 -- 172.21.15.2:0/1460506905 learned_addr learned my addr 172.21.15.2:0/1460506905 (peer_addr_for_me v2:172.21.15.2:0/0) 2022-05-13T23:15:29.187 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.151+0000 7f32dc99a700 1 --2- 172.21.15.2:0/1460506905 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55cb3d6ba640 0x55cb3d6c7ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:15:29.187 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.151+0000 7f32dd99c700 1 --2- 172.21.15.2:0/1460506905 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55cb3d6bb5a0 0x55cb3d6cec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:15:29.188 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.151+0000 7f32dd19b700 1 -- 172.21.15.2:0/1460506905 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55cb3d6bb5a0 msgr2=0x55cb3d6cec00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-05-13T23:15:29.188 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.151+0000 7f32dd19b700 1 --2- 172.21.15.2:0/1460506905 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55cb3d6bb5a0 0x55cb3d6cec00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:29.189 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.151+0000 7f32dd19b700 1 -- 172.21.15.2:0/1460506905 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55cb3d6ba640 msgr2=0x55cb3d6c7ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-05-13T23:15:29.189 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.151+0000 7f32dd19b700 1 --2- 172.21.15.2:0/1460506905 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55cb3d6ba640 0x55cb3d6c7ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:29.190 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.151+0000 7f32dd19b700 1 -- 172.21.15.2:0/1460506905 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f32d81147a0 con 0x55cb3d6b9cf0 2022-05-13T23:15:29.191 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.151+0000 7f32dd19b700 1 --2- 172.21.15.2:0/1460506905 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55cb3d6b9cf0 0x55cb3d6c7660 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f32d401c570 tx=0x7f32d408d040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a2ef4a2b8c226765 server_cookie=0 in_seq=0 out_seq=0 2022-05-13T23:15:29.191 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.151+0000 7f32b3fff700 10 client.0 ms_handle_connect on v2:172.21.15.47:3300/0 2022-05-13T23:15:29.192 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.151+0000 7f32b3fff700 1 -- 172.21.15.2:0/1460506905 <== mon.1 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32d40be0f0 con 0x55cb3d6b9cf0 2022-05-13T23:15:29.192 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.151+0000 7f32ee635b80 1 -- 172.21.15.2:0/1460506905 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55cb3d5bb9a0 con 0x55cb3d6b9cf0 2022-05-13T23:15:29.193 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.151+0000 7f32b3fff700 1 -- 172.21.15.2:0/1460506905 <== mon.1 v2:172.21.15.47:3300/0 2 ==== osd_map(85..85 src has 1..85) v4 ==== 8120+0+0 (secure 0 0 0) 0x7f32d4091ab0 con 0x55cb3d6b9cf0 2022-05-13T23:15:29.193 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.152+0000 7f32b3fff700 1 -- 172.21.15.2:0/1460506905 <== mon.1 v2:172.21.15.47:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7f32d40b8ae0 con 0x55cb3d6b9cf0 2022-05-13T23:15:29.194 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.152+0000 7f32ee635b80 10 client.24949 fetch_fsmap learned FSMap version 26 2022-05-13T23:15:29.194 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.152+0000 7f32ee635b80 1 -- 172.21.15.2:0/1460506905 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) v3 -- 0x55cb3d449fe0 con 0x55cb3d6b9cf0 2022-05-13T23:15:29.195 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.152+0000 7f32b3fff700 1 -- 172.21.15.2:0/1460506905 <== mon.1 v2:172.21.15.47:3300/0 4 ==== config(26 keys) v1 ==== 1087+0+0 (secure 0 0 0) 0x55cb3d5bb9a0 con 0x55cb3d6b9cf0 2022-05-13T23:15:29.195 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.152+0000 7f32b3fff700 1 -- 172.21.15.2:0/1460506905 <== mon.1 v2:172.21.15.47:3300/0 5 ==== fsmap(e 26) v1 ==== 7448+0+0 (secure 0 0 0) 0x7f32d40908a0 con 0x55cb3d6b9cf0 2022-05-13T23:15:29.196 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.153+0000 7f32ee635b80 10 client.24949 fetch_fsmap finished waiting for FSMap version 26 2022-05-13T23:15:29.196 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.153+0000 7f32ee635b80 10 client.24949 resolve_mds: resolved name 'c' to daemon mds.c 2022-05-13T23:15:29.196 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.153+0000 7f32ee635b80 20 client.24949 populate_metadata read hostname 'smithi002' 2022-05-13T23:15:29.197 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.153+0000 7f32ee635b80 1 --2- 172.21.15.2:0/1460506905 >> [v2:172.21.15.115:6832/381933929,v1:172.21.15.115:6833/381933929] conn(0x55cb3d6d1d20 0x55cb3d6d06b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:15:29.197 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.153+0000 7f32ee635b80 4 client.24949 mds_command: new command op to 24575 tid=0[{"prefix": "get_command_descriptions"}] 2022-05-13T23:15:29.198 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.153+0000 7f32ee635b80 1 -- 172.21.15.2:0/1460506905 --> [v2:172.21.15.115:6832/381933929,v1:172.21.15.115:6833/381933929] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55cb3d5dc330 con 0x55cb3d6d1d20 2022-05-13T23:15:29.198 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.153+0000 7f32dc99a700 1 --2- 172.21.15.2:0/1460506905 >> [v2:172.21.15.115:6832/381933929,v1:172.21.15.115:6833/381933929] conn(0x55cb3d6d1d20 0x55cb3d6d06b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:15:29.199 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.154+0000 7f32b3fff700 10 client.24949 ms_handle_connect on v2:172.21.15.115:6832/381933929 2022-05-13T23:15:29.199 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.154+0000 7f32dc99a700 1 --2- 172.21.15.2:0/1460506905 >> [v2:172.21.15.115:6832/381933929,v1:172.21.15.115:6833/381933929] conn(0x55cb3d6d1d20 0x55cb3d6d06b0 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=b14e1a28b7f90c44 server_cookie=de3f74e7b72df786 in_seq=0 out_seq=0 2022-05-13T23:15:29.200 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.155+0000 7f32b3fff700 1 -- 172.21.15.2:0/1460506905 <== mds.3 v2:172.21.15.115:6832/381933929 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17090 (crc 0 0 0) 0x55cb3d5dc330 con 0x55cb3d6d1d20 2022-05-13T23:15:29.200 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.155+0000 7f32b3fff700 10 client.24949 handle_command_reply: tid=0 2022-05-13T23:15:29.201 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.155+0000 7f32ee635b80 1 client.24949 shutdown 2022-05-13T23:15:29.202 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.155+0000 7f32ee635b80 1 -- 172.21.15.2:0/1460506905 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55cb3d6b9cf0 msgr2=0x55cb3d6c7660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:15:29.202 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.155+0000 7f32ee635b80 1 --2- 172.21.15.2:0/1460506905 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55cb3d6b9cf0 0x55cb3d6c7660 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f32d401c570 tx=0x7f32d408d040 comp rx=0 tx=0).stop 2022-05-13T23:15:29.203 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.155+0000 7f32ee635b80 1 -- 172.21.15.2:0/1460506905 shutdown_connections 2022-05-13T23:15:29.204 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.155+0000 7f32ee635b80 1 --2- 172.21.15.2:0/1460506905 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55cb3d6bb5a0 0x55cb3d6cec00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:29.204 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.155+0000 7f32ee635b80 1 --2- 172.21.15.2:0/1460506905 >> [v2:172.21.15.115:6832/381933929,v1:172.21.15.115:6833/381933929] conn(0x55cb3d6d1d20 0x55cb3d6d06b0 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:29.205 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.155+0000 7f32ee635b80 1 --2- 172.21.15.2:0/1460506905 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55cb3d6b9cf0 0x55cb3d6c7660 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:29.205 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.155+0000 7f32ee635b80 1 --2- 172.21.15.2:0/1460506905 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55cb3d6ba640 0x55cb3d6c7ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:29.206 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.155+0000 7f32ee635b80 1 -- 172.21.15.2:0/1460506905 >> 172.21.15.2:0/1460506905 conn(0x55cb3d6b4b10 msgr2=0x55cb3d6b3d80 unknown :-1 s=STATE_NONE l=0).mark_down 2022-05-13T23:15:29.206 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.155+0000 7f32b3fff700 0 client.24949 ms_handle_reset on v2:172.21.15.115:6832/381933929 2022-05-13T23:15:29.206 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.155+0000 7f32ee635b80 1 -- 172.21.15.2:0/1460506905 shutdown_connections 2022-05-13T23:15:29.207 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.155+0000 7f32ee635b80 1 -- 172.21.15.2:0/1460506905 wait complete. 2022-05-13T23:15:29.207 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.155+0000 7f32ee635b80 20 client.24949 trim_cache size 0 max 16384 2022-05-13T23:15:29.221 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.173+0000 7f32ee635b80 1 Processor -- start 2022-05-13T23:15:29.221 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.173+0000 7f32ee635b80 1 -- start start 2022-05-13T23:15:29.222 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.174+0000 7f32ee635b80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55cb3d6b9cf0 0x55cb3d6b3d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:15:29.222 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.174+0000 7f32ee635b80 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55cb3d6ba640 0x55cb3d6d05f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:15:29.223 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.174+0000 7f32ee635b80 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55cb3d6bb5a0 0x55cb3d6d0b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:15:29.223 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.174+0000 7f32ee635b80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cb3d6aff50 con 0x55cb3d6b9cf0 2022-05-13T23:15:29.223 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.174+0000 7f32ee635b80 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cb3d5dc330 con 0x55cb3d6bb5a0 2022-05-13T23:15:29.224 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.174+0000 7f32ee635b80 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cb3d5bb9a0 con 0x55cb3d6ba640 2022-05-13T23:15:29.224 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.174+0000 7f32dc99a700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55cb3d6ba640 0x55cb3d6d05f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:15:29.225 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.174+0000 7f32dd99c700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55cb3d6bb5a0 0x55cb3d6d0b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:15:29.225 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.174+0000 7f32dd19b700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55cb3d6b9cf0 0x55cb3d6b3d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:15:29.226 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.174+0000 7f32dc99a700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55cb3d6ba640 0x55cb3d6d05f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.2:53610/0 (socket says 172.21.15.2:53610) 2022-05-13T23:15:29.226 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.174+0000 7f32dc99a700 1 -- 172.21.15.2:0/20154441 learned_addr learned my addr 172.21.15.2:0/20154441 (peer_addr_for_me v2:172.21.15.2:0/0) 2022-05-13T23:15:29.227 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.174+0000 7f32dc99a700 1 -- 172.21.15.2:0/20154441 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55cb3d6bb5a0 msgr2=0x55cb3d6d0b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-05-13T23:15:29.227 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.174+0000 7f32dc99a700 1 --2- 172.21.15.2:0/20154441 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55cb3d6bb5a0 0x55cb3d6d0b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:29.228 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.174+0000 7f32dc99a700 1 -- 172.21.15.2:0/20154441 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55cb3d6b9cf0 msgr2=0x55cb3d6b3d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-05-13T23:15:29.228 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.174+0000 7f32dc99a700 1 --2- 172.21.15.2:0/20154441 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55cb3d6b9cf0 0x55cb3d6b3d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:29.228 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.175+0000 7f32dc99a700 1 -- 172.21.15.2:0/20154441 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55cb3d471f80 con 0x55cb3d6ba640 2022-05-13T23:15:29.229 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.175+0000 7f32dc99a700 1 --2- 172.21.15.2:0/20154441 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55cb3d6ba640 0x55cb3d6d05f0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x55cb3d4a16e0 tx=0x7f32c800b040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=692bec31e0919094 server_cookie=0 in_seq=0 out_seq=0 2022-05-13T23:15:29.229 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.175+0000 7f32bb1ff700 1 -- 172.21.15.2:0/20154441 <== mon.2 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32c800e040 con 0x55cb3d6ba640 2022-05-13T23:15:29.230 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.175+0000 7f32bb1ff700 1 -- 172.21.15.2:0/20154441 <== mon.2 v2:172.21.15.115:3300/0 2 ==== config(26 keys) v1 ==== 1087+0+0 (secure 0 0 0) 0x7f32c800f040 con 0x55cb3d6ba640 2022-05-13T23:15:29.230 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.175+0000 7f32bb1ff700 1 -- 172.21.15.2:0/20154441 <== mon.2 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32c801f9c0 con 0x55cb3d6ba640 2022-05-13T23:15:29.230 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.176+0000 7f32ee635b80 1 -- 172.21.15.2:0/20154441 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55cb3d6ba640 msgr2=0x55cb3d6d05f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:15:29.231 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.176+0000 7f32ee635b80 1 --2- 172.21.15.2:0/20154441 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55cb3d6ba640 0x55cb3d6d05f0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x55cb3d4a16e0 tx=0x7f32c800b040 comp rx=0 tx=0).stop 2022-05-13T23:15:29.232 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.176+0000 7f32ee635b80 1 -- 172.21.15.2:0/20154441 shutdown_connections 2022-05-13T23:15:29.233 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.176+0000 7f32ee635b80 1 --2- 172.21.15.2:0/20154441 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55cb3d6bb5a0 0x55cb3d6d0b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:29.233 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.176+0000 7f32ee635b80 1 --2- 172.21.15.2:0/20154441 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55cb3d6b9cf0 0x55cb3d6b3d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:29.234 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.176+0000 7f32ee635b80 1 --2- 172.21.15.2:0/20154441 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55cb3d6ba640 0x55cb3d6d05f0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:29.234 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.176+0000 7f32ee635b80 1 -- 172.21.15.2:0/20154441 >> 172.21.15.2:0/20154441 conn(0x55cb3d6b4b10 msgr2=0x55cb3d6b4ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-05-13T23:15:29.235 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.176+0000 7f32ee635b80 1 -- 172.21.15.2:0/20154441 shutdown_connections 2022-05-13T23:15:29.235 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.176+0000 7f32ee635b80 1 -- 172.21.15.2:0/20154441 wait complete. 2022-05-13T23:15:29.235 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.176+0000 7f32ee635b80 1 -- start start 2022-05-13T23:15:29.236 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.177+0000 7f32ee635b80 1 Processor -- start 2022-05-13T23:15:29.236 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.177+0000 7f32ee635b80 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55cb3d6b9cf0 0x55cb3d6dbd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:15:29.236 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.177+0000 7f32ee635b80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55cb3d6ba640 0x55cb3d6d8970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:15:29.237 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.177+0000 7f32ee635b80 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55cb3d6bb5a0 0x55cb3d6d8ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:15:29.237 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.177+0000 7f32ee635b80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cb3d724960 con 0x55cb3d6ba640 2022-05-13T23:15:29.238 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.177+0000 7f32ee635b80 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cb3d724aa0 con 0x55cb3d6b9cf0 2022-05-13T23:15:29.238 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.177+0000 7f32ee635b80 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cb3d724be0 con 0x55cb3d6bb5a0 2022-05-13T23:15:29.239 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.177+0000 7f32ee635b80 10 client.0 _ll_register_callbacks cb 0x55cb3d67bde0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2022-05-13T23:15:29.239 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.177+0000 7f32dd19b700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55cb3d6b9cf0 0x55cb3d6dbd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:15:29.239 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.177+0000 7f32dd19b700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55cb3d6b9cf0 0x55cb3d6dbd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.2:41498/0 (socket says 172.21.15.2:41498) 2022-05-13T23:15:29.240 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.177+0000 7f32dc99a700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55cb3d6ba640 0x55cb3d6d8970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:15:29.240 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.177+0000 7f32dd19b700 1 -- 172.21.15.2:0/406562400 learned_addr learned my addr 172.21.15.2:0/406562400 (peer_addr_for_me v2:172.21.15.2:0/0) 2022-05-13T23:15:29.240 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.177+0000 7f32dd99c700 1 --2- 172.21.15.2:0/406562400 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55cb3d6bb5a0 0x55cb3d6d8ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:15:29.240 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.178+0000 7f32dd19b700 1 -- 172.21.15.2:0/406562400 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55cb3d6bb5a0 msgr2=0x55cb3d6d8ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-05-13T23:15:29.241 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.178+0000 7f32dd19b700 1 --2- 172.21.15.2:0/406562400 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55cb3d6bb5a0 0x55cb3d6d8ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:29.241 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.178+0000 7f32dd19b700 1 -- 172.21.15.2:0/406562400 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55cb3d6ba640 msgr2=0x55cb3d6d8970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-05-13T23:15:29.242 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.178+0000 7f32dd19b700 1 --2- 172.21.15.2:0/406562400 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55cb3d6ba640 0x55cb3d6d8970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:29.242 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.178+0000 7f32dd19b700 1 -- 172.21.15.2:0/406562400 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f32c8005040 con 0x55cb3d6b9cf0 2022-05-13T23:15:29.242 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.178+0000 7f32dd99c700 1 --2- 172.21.15.2:0/406562400 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55cb3d6bb5a0 0x55cb3d6d8ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2022-05-13T23:15:29.243 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.178+0000 7f32dd19b700 1 --2- 172.21.15.2:0/406562400 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55cb3d6b9cf0 0x55cb3d6dbd90 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f32d4005010 tx=0x7f32d408d040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c8319df2c22e1d6f server_cookie=0 in_seq=0 out_seq=0 2022-05-13T23:15:29.243 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.178+0000 7f32b3fff700 10 client.0 ms_handle_connect on v2:172.21.15.47:3300/0 2022-05-13T23:15:29.243 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.178+0000 7f32b3fff700 1 -- 172.21.15.2:0/406562400 <== mon.1 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32d40be0f0 con 0x55cb3d6b9cf0 2022-05-13T23:15:29.243 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.178+0000 7f32ee635b80 1 -- 172.21.15.2:0/406562400 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55cb3d68a5f0 con 0x55cb3d6b9cf0 2022-05-13T23:15:29.244 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.178+0000 7f32b3fff700 1 -- 172.21.15.2:0/406562400 <== mon.1 v2:172.21.15.47:3300/0 2 ==== osd_map(85..85 src has 1..85) v4 ==== 8120+0+0 (secure 0 0 0) 0x7f32d40b8ae0 con 0x55cb3d6b9cf0 2022-05-13T23:15:29.244 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.179+0000 7f32b3fff700 1 -- 172.21.15.2:0/406562400 <== mon.1 v2:172.21.15.47:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7f32d4091430 con 0x55cb3d6b9cf0 2022-05-13T23:15:29.244 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.179+0000 7f32ee635b80 10 client.24955 fetch_fsmap learned FSMap version 26 2022-05-13T23:15:29.244 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.179+0000 7f32ee635b80 1 -- 172.21.15.2:0/406562400 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) v3 -- 0x55cb3d44bb10 con 0x55cb3d6b9cf0 2022-05-13T23:15:29.245 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.179+0000 7f32b3fff700 1 -- 172.21.15.2:0/406562400 <== mon.1 v2:172.21.15.47:3300/0 4 ==== config(26 keys) v1 ==== 1087+0+0 (secure 0 0 0) 0x55cb3d68a5f0 con 0x55cb3d6b9cf0 2022-05-13T23:15:29.245 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.179+0000 7f32b3fff700 1 -- 172.21.15.2:0/406562400 <== mon.1 v2:172.21.15.47:3300/0 5 ==== fsmap(e 26) v1 ==== 7448+0+0 (secure 0 0 0) 0x7f32d40b81e0 con 0x55cb3d6b9cf0 2022-05-13T23:15:29.245 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.179+0000 7f32ee635b80 10 client.24955 fetch_fsmap finished waiting for FSMap version 26 2022-05-13T23:15:29.246 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.179+0000 7f32ee635b80 10 client.24955 resolve_mds: resolved name 'c' to daemon mds.c 2022-05-13T23:15:29.246 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.179+0000 7f32ee635b80 20 client.24955 populate_metadata read hostname 'smithi002' 2022-05-13T23:15:29.246 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.179+0000 7f32ee635b80 1 --2- 172.21.15.2:0/406562400 >> [v2:172.21.15.115:6832/381933929,v1:172.21.15.115:6833/381933929] conn(0x55cb3d728d30 0x55cb3d725870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:15:29.246 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.179+0000 7f32ee635b80 4 client.24955 mds_command: new command op to 24575 tid=0[{"prefix": "status"}] 2022-05-13T23:15:29.246 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.179+0000 7f32ee635b80 1 -- 172.21.15.2:0/406562400 --> [v2:172.21.15.115:6832/381933929,v1:172.21.15.115:6833/381933929] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55cb3d5ad790 con 0x55cb3d728d30 2022-05-13T23:15:29.247 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.180+0000 7f32dc99a700 1 --2- 172.21.15.2:0/406562400 >> [v2:172.21.15.115:6832/381933929,v1:172.21.15.115:6833/381933929] conn(0x55cb3d728d30 0x55cb3d725870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:15:29.247 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.180+0000 7f32dc99a700 1 --2- 172.21.15.2:0/406562400 >> [v2:172.21.15.115:6832/381933929,v1:172.21.15.115:6833/381933929] conn(0x55cb3d728d30 0x55cb3d725870 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=200075587c2e978f server_cookie=23b2db6e57fd5fb6 in_seq=0 out_seq=0 2022-05-13T23:15:29.247 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.180+0000 7f32b3fff700 10 client.24955 ms_handle_connect on v2:172.21.15.115:6832/381933929 2022-05-13T23:15:29.247 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.181+0000 7f32b3fff700 1 -- 172.21.15.2:0/406562400 <== mds.3 v2:172.21.15.115:6832/381933929 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+326 (crc 0 0 0) 0x55cb3d5ad790 con 0x55cb3d728d30 2022-05-13T23:15:29.248 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.181+0000 7f32b3fff700 10 client.24955 handle_command_reply: tid=0 2022-05-13T23:15:29.248 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.181+0000 7f32ee635b80 1 client.24955 shutdown 2022-05-13T23:15:29.248 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.181+0000 7f32ee635b80 1 -- 172.21.15.2:0/406562400 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55cb3d6b9cf0 msgr2=0x55cb3d6dbd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:15:29.248 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.181+0000 7f32ee635b80 1 --2- 172.21.15.2:0/406562400 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55cb3d6b9cf0 0x55cb3d6dbd90 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f32d4005010 tx=0x7f32d408d040 comp rx=0 tx=0).stop 2022-05-13T23:15:29.248 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.181+0000 7f32ee635b80 1 -- 172.21.15.2:0/406562400 shutdown_connections 2022-05-13T23:15:29.249 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.181+0000 7f32ee635b80 1 --2- 172.21.15.2:0/406562400 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55cb3d6bb5a0 0x55cb3d6d8ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:29.249 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.181+0000 7f32ee635b80 1 --2- 172.21.15.2:0/406562400 >> [v2:172.21.15.115:6832/381933929,v1:172.21.15.115:6833/381933929] conn(0x55cb3d728d30 0x55cb3d725870 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:29.249 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.181+0000 7f32ee635b80 1 --2- 172.21.15.2:0/406562400 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55cb3d6b9cf0 0x55cb3d6dbd90 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:29.250 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.181+0000 7f32b3fff700 0 client.24955 ms_handle_reset on v2:172.21.15.115:6832/381933929 2022-05-13T23:15:29.250 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.181+0000 7f32ee635b80 1 --2- 172.21.15.2:0/406562400 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55cb3d6ba640 0x55cb3d6d8970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:29.251 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.181+0000 7f32ee635b80 1 -- 172.21.15.2:0/406562400 >> 172.21.15.2:0/406562400 conn(0x55cb3d6b4b10 msgr2=0x55cb3d6b3d80 unknown :-1 s=STATE_NONE l=0).mark_down 2022-05-13T23:15:29.251 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.181+0000 7f32ee635b80 1 -- 172.21.15.2:0/406562400 shutdown_connections 2022-05-13T23:15:29.251 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.181+0000 7f32ee635b80 1 -- 172.21.15.2:0/406562400 wait complete. 2022-05-13T23:15:29.252 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.181+0000 7f32ee635b80 20 client.24955 trim_cache size 0 max 16384 2022-05-13T23:15:29.252 INFO:teuthology.orchestra.run.smithi002.stdout:{ 2022-05-13T23:15:29.253 INFO:teuthology.orchestra.run.smithi002.stdout: "cluster_fsid": "16cf5f88-d311-11ec-be40-001a4aab830c", 2022-05-13T23:15:29.253 INFO:teuthology.orchestra.run.smithi002.stdout: "whoami": 3, 2022-05-13T23:15:29.253 INFO:teuthology.orchestra.run.smithi002.stdout: "id": 24575, 2022-05-13T23:15:29.254 INFO:teuthology.orchestra.run.smithi002.stdout: "want_state": "up:active", 2022-05-13T23:15:29.254 INFO:teuthology.orchestra.run.smithi002.stdout: "state": "up:active", 2022-05-13T23:15:29.254 INFO:teuthology.orchestra.run.smithi002.stdout: "fs_name": "cephfs", 2022-05-13T23:15:29.255 INFO:teuthology.orchestra.run.smithi002.stdout: "rank_uptime": 67.881836128000003, 2022-05-13T23:15:29.255 INFO:teuthology.orchestra.run.smithi002.stdout: "mdsmap_epoch": 26, 2022-05-13T23:15:29.255 INFO:teuthology.orchestra.run.smithi002.stdout: "osdmap_epoch": 85, 2022-05-13T23:15:29.255 INFO:teuthology.orchestra.run.smithi002.stdout: "osdmap_epoch_barrier": 83, 2022-05-13T23:15:29.256 INFO:teuthology.orchestra.run.smithi002.stdout: "uptime": 86.071577159 2022-05-13T23:15:29.256 INFO:teuthology.orchestra.run.smithi002.stdout:} 2022-05-13T23:15:29.258 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.182+0000 7f32de19d700 1 -- 172.21.15.2:0/467152150 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f32c4096ce0 msgr2=0x7f32c4099190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:15:29.258 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.182+0000 7f32de19d700 1 --2- 172.21.15.2:0/467152150 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f32c4096ce0 0x7f32c4099190 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f32d0004fd0 tx=0x7f32d000b040 comp rx=0 tx=0).stop 2022-05-13T23:15:29.258 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.182+0000 7f32de19d700 1 -- 172.21.15.2:0/467152150 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f32d80fb4e0 msgr2=0x7f32d810c2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:15:29.259 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.182+0000 7f32de19d700 1 --2- 172.21.15.2:0/467152150 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f32d80fb4e0 0x7f32d810c2b0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f32d4004710 tx=0x7f32d4009cb0 comp rx=0 tx=0).stop 2022-05-13T23:15:29.259 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.182+0000 7f32de19d700 1 -- 172.21.15.2:0/467152150 shutdown_connections 2022-05-13T23:15:29.259 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.182+0000 7f32de19d700 1 --2- 172.21.15.2:0/467152150 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f32d80fd110 0x7f32d8113880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:29.259 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.182+0000 7f32de19d700 1 --2- 172.21.15.2:0/467152150 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f32d80fb4e0 0x7f32d810c2b0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:29.260 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.182+0000 7f32de19d700 1 --2- 172.21.15.2:0/467152150 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f32c4096ce0 0x7f32c4099190 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:29.260 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.182+0000 7f32de19d700 1 --2- 172.21.15.2:0/467152150 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f32d80fc780 0x7f32d810c7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:29.260 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.182+0000 7f32de19d700 1 -- 172.21.15.2:0/467152150 >> 172.21.15.2:0/467152150 conn(0x7f32d80f6cb0 msgr2=0x7f32d80ff9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-05-13T23:15:29.260 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.182+0000 7f32de19d700 1 -- 172.21.15.2:0/467152150 shutdown_connections 2022-05-13T23:15:29.260 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.182+0000 7f32de19d700 1 -- 172.21.15.2:0/467152150 wait complete. 2022-05-13T23:15:29.261 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.f status 2022-05-13T23:15:29.489 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.479+0000 7fe5effff700 1 -- 172.21.15.2:0/1828945612 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe5f00fd0a0 msgr2=0x7fe5f0103960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:15:29.490 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.479+0000 7fe5effff700 1 --2- 172.21.15.2:0/1828945612 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe5f00fd0a0 0x7fe5f0103960 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fe5e8009be0 tx=0x7fe5e8004d10 comp rx=0 tx=0).stop 2022-05-13T23:15:29.490 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.479+0000 7fe5effff700 1 -- 172.21.15.2:0/1828945612 shutdown_connections 2022-05-13T23:15:29.490 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.479+0000 7fe5effff700 1 --2- 172.21.15.2:0/1828945612 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe5f00fd0a0 0x7fe5f0103960 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:29.490 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.479+0000 7fe5effff700 1 --2- 172.21.15.2:0/1828945612 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe5f00fb470 0x7fe5f00fb880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:29.491 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.479+0000 7fe5effff700 1 --2- 172.21.15.2:0/1828945612 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe5f00fc710 0x7fe5f00fcb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:29.491 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.479+0000 7fe5effff700 1 -- 172.21.15.2:0/1828945612 >> 172.21.15.2:0/1828945612 conn(0x7fe5f00f6c60 msgr2=0x7fe5f00f9090 unknown :-1 s=STATE_NONE l=0).mark_down 2022-05-13T23:15:29.503 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.479+0000 7fe5effff700 1 -- 172.21.15.2:0/1828945612 shutdown_connections 2022-05-13T23:15:29.504 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.479+0000 7fe5effff700 1 -- 172.21.15.2:0/1828945612 wait complete. 2022-05-13T23:15:29.504 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.480+0000 7fe5effff700 1 Processor -- start 2022-05-13T23:15:29.504 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.480+0000 7fe5effff700 1 -- start start 2022-05-13T23:15:29.504 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.480+0000 7fe5effff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe5f00fb470 0x7fe5f010e380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:15:29.505 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.480+0000 7fe5effff700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe5f00fc710 0x7fe5f010e8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:15:29.505 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.480+0000 7fe5effff700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe5f00fd0a0 0x7fe5f0115950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:15:29.505 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.480+0000 7fe5effff700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5f0115e90 con 0x7fe5f00fb470 2022-05-13T23:15:29.505 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.480+0000 7fe5effff700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5f0116000 con 0x7fe5f00fc710 2022-05-13T23:15:29.505 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.480+0000 7fe5effff700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5f01161c0 con 0x7fe5f00fd0a0 2022-05-13T23:15:29.506 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.481+0000 7fe5eeffd700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe5f00fb470 0x7fe5f010e380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:15:29.506 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.481+0000 7fe5eeffd700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe5f00fb470 0x7fe5f010e380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:46228/0 (socket says 172.21.15.2:46228) 2022-05-13T23:15:29.506 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.481+0000 7fe5eeffd700 1 -- 172.21.15.2:0/2764764927 learned_addr learned my addr 172.21.15.2:0/2764764927 (peer_addr_for_me v2:172.21.15.2:0/0) 2022-05-13T23:15:29.506 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.481+0000 7fe5ee7fc700 1 --2- 172.21.15.2:0/2764764927 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe5f00fc710 0x7fe5f010e8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:15:29.506 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.481+0000 7fe5ef7fe700 1 --2- 172.21.15.2:0/2764764927 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe5f00fd0a0 0x7fe5f0115950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:15:29.507 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.481+0000 7fe5eeffd700 1 -- 172.21.15.2:0/2764764927 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe5f00fd0a0 msgr2=0x7fe5f0115950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:15:29.507 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.481+0000 7fe5eeffd700 1 --2- 172.21.15.2:0/2764764927 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe5f00fd0a0 0x7fe5f0115950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:29.507 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.481+0000 7fe5eeffd700 1 -- 172.21.15.2:0/2764764927 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe5f00fc710 msgr2=0x7fe5f010e8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:15:29.507 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.481+0000 7fe5eeffd700 1 --2- 172.21.15.2:0/2764764927 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe5f00fc710 0x7fe5f010e8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:29.508 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.481+0000 7fe5eeffd700 1 -- 172.21.15.2:0/2764764927 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5e80097c0 con 0x7fe5f00fb470 2022-05-13T23:15:29.508 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.481+0000 7fe5eeffd700 1 --2- 172.21.15.2:0/2764764927 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe5f00fb470 0x7fe5f010e380 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fe5e000e940 tx=0x7fe5e000b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-05-13T23:15:29.508 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.482+0000 7fe5d77fe700 1 -- 172.21.15.2:0/2764764927 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5e0007b40 con 0x7fe5f00fb470 2022-05-13T23:15:29.508 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.482+0000 7fe5d77fe700 1 -- 172.21.15.2:0/2764764927 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1087+0+0 (secure 0 0 0) 0x7fe5e000f040 con 0x7fe5f00fb470 2022-05-13T23:15:29.508 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.482+0000 7fe5effff700 1 -- 172.21.15.2:0/2764764927 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe5f0116440 con 0x7fe5f00fb470 2022-05-13T23:15:29.509 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.482+0000 7fe5d77fe700 1 -- 172.21.15.2:0/2764764927 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5e0014890 con 0x7fe5f00fb470 2022-05-13T23:15:29.509 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.482+0000 7fe5effff700 1 -- 172.21.15.2:0/2764764927 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe5f0116a00 con 0x7fe5f00fb470 2022-05-13T23:15:29.509 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.485+0000 7fe5d77fe700 1 -- 172.21.15.2:0/2764764927 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 16) v1 ==== 125136+0+0 (secure 0 0 0) 0x7fe5e001b030 con 0x7fe5f00fb470 2022-05-13T23:15:29.509 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.486+0000 7fe604964b80 1 Processor -- start 2022-05-13T23:15:29.509 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.486+0000 7fe5d77fe700 1 --2- 172.21.15.2:0/2764764927 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7fe5d8096d60 0x7fe5d8099210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:15:29.510 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.487+0000 7fe604964b80 1 -- start start 2022-05-13T23:15:29.510 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.487+0000 7fe5d77fe700 1 -- 172.21.15.2:0/2764764927 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8120+0+0 (secure 0 0 0) 0x7fe5e00bead0 con 0x7fe5f00fb470 2022-05-13T23:15:29.510 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.487+0000 7fe5ef7fe700 1 --2- 172.21.15.2:0/2764764927 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7fe5d8096d60 0x7fe5d8099210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:15:29.510 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.487+0000 7fe604964b80 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x5652e2a60cf0 0x5652e2a61100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:15:29.511 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.487+0000 7fe604964b80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5652e2a61640 0x5652e2a61f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:15:29.511 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.487+0000 7fe604964b80 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5652e2a625a0 0x5652e2a629f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:15:29.511 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.487+0000 7fe604964b80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x5652e295b3a0 con 0x5652e2a61640 2022-05-13T23:15:29.511 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.487+0000 7fe604964b80 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x5652e2930400 con 0x5652e2a60cf0 2022-05-13T23:15:29.511 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.487+0000 7fe604964b80 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x5652e29b46c0 con 0x5652e2a625a0 2022-05-13T23:15:29.512 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.487+0000 7fe5ee7fc700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5652e2a61640 0x5652e2a61f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:15:29.512 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.488+0000 7fe5eeffd700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x5652e2a60cf0 0x5652e2a61100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:15:29.512 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.488+0000 7fe5eeffd700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x5652e2a60cf0 0x5652e2a61100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.2:41528/0 (socket says 172.21.15.2:41528) 2022-05-13T23:15:29.513 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.488+0000 7fe5eeffd700 1 -- 172.21.15.2:0/163929843 learned_addr learned my addr 172.21.15.2:0/163929843 (peer_addr_for_me v2:172.21.15.2:0/0) 2022-05-13T23:15:29.513 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.488+0000 7fe5ef7fe700 1 --2- 172.21.15.2:0/163929843 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5652e2a625a0 0x5652e2a629f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:15:29.513 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.488+0000 7fe5ef7fe700 1 --2- 172.21.15.2:0/2764764927 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7fe5d8096d60 0x7fe5d8099210 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fe5e800b3b0 tx=0x7fe5e8004d10 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-05-13T23:15:29.513 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.488+0000 7fe5ee7fc700 1 -- 172.21.15.2:0/163929843 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5652e2a625a0 msgr2=0x5652e2a629f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-05-13T23:15:29.514 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.488+0000 7fe5ee7fc700 1 --2- 172.21.15.2:0/163929843 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5652e2a625a0 0x5652e2a629f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:29.514 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.488+0000 7fe5ee7fc700 1 -- 172.21.15.2:0/163929843 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x5652e2a60cf0 msgr2=0x5652e2a61100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-05-13T23:15:29.514 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.488+0000 7fe5ee7fc700 1 --2- 172.21.15.2:0/163929843 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x5652e2a60cf0 0x5652e2a61100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:29.514 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.488+0000 7fe5ee7fc700 1 -- 172.21.15.2:0/163929843 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5652e2846360 con 0x5652e2a61640 2022-05-13T23:15:29.515 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.488+0000 7fe5eeffd700 1 --2- 172.21.15.2:0/163929843 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x5652e2a60cf0 0x5652e2a61100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-05-13T23:15:29.515 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.488+0000 7fe5ee7fc700 1 --2- 172.21.15.2:0/163929843 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5652e2a61640 0x5652e2a61f80 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fe5e4008cd0 tx=0x7fe5e400b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ba8e3ea1b72dcd61 server_cookie=0 in_seq=0 out_seq=0 2022-05-13T23:15:29.515 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.489+0000 7fe5d49fe700 1 -- 172.21.15.2:0/163929843 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5e400e070 con 0x5652e2a61640 2022-05-13T23:15:29.515 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.489+0000 7fe5d49fe700 1 -- 172.21.15.2:0/163929843 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1087+0+0 (secure 0 0 0) 0x7fe5e400f040 con 0x5652e2a61640 2022-05-13T23:15:29.515 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.489+0000 7fe5d49fe700 1 -- 172.21.15.2:0/163929843 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5e4015b40 con 0x5652e2a61640 2022-05-13T23:15:29.516 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.489+0000 7fe604964b80 1 -- 172.21.15.2:0/163929843 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5652e2a61640 msgr2=0x5652e2a61f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:15:29.516 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.489+0000 7fe604964b80 1 --2- 172.21.15.2:0/163929843 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5652e2a61640 0x5652e2a61f80 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fe5e4008cd0 tx=0x7fe5e400b040 comp rx=0 tx=0).stop 2022-05-13T23:15:29.516 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.490+0000 7fe604964b80 1 -- 172.21.15.2:0/163929843 shutdown_connections 2022-05-13T23:15:29.516 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.490+0000 7fe604964b80 1 --2- 172.21.15.2:0/163929843 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5652e2a625a0 0x5652e2a629f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:29.516 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.490+0000 7fe604964b80 1 --2- 172.21.15.2:0/163929843 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x5652e2a60cf0 0x5652e2a61100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:29.517 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.490+0000 7fe604964b80 1 --2- 172.21.15.2:0/163929843 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5652e2a61640 0x5652e2a61f80 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:29.517 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.490+0000 7fe604964b80 1 -- 172.21.15.2:0/163929843 >> 172.21.15.2:0/163929843 conn(0x5652e2a5bb10 msgr2=0x5652e2a5bf00 unknown :-1 s=STATE_NONE l=0).mark_down 2022-05-13T23:15:29.517 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.490+0000 7fe604964b80 1 -- 172.21.15.2:0/163929843 shutdown_connections 2022-05-13T23:15:29.518 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.490+0000 7fe604964b80 1 -- 172.21.15.2:0/163929843 wait complete. 2022-05-13T23:15:29.518 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.490+0000 7fe604964b80 1 -- start start 2022-05-13T23:15:29.518 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.490+0000 7fe604964b80 1 Processor -- start 2022-05-13T23:15:29.518 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.491+0000 7fe604964b80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5652e2a60cf0 0x5652e2a6a4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:15:29.518 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.491+0000 7fe604964b80 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x5652e2a61640 0x5652e2a6aa30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:15:29.519 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.491+0000 7fe604964b80 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5652e2a625a0 0x5652e2a71a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:15:29.519 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.491+0000 7fe604964b80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x5652e29ae7a0 con 0x5652e2a60cf0 2022-05-13T23:15:29.519 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.491+0000 7fe604964b80 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x5652e2848650 con 0x5652e2a61640 2022-05-13T23:15:29.519 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.491+0000 7fe604964b80 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x5652e2a720c0 con 0x5652e2a625a0 2022-05-13T23:15:29.519 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.491+0000 7fe604964b80 10 client.0 _ll_register_callbacks cb 0x5652e2a22de0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2022-05-13T23:15:29.520 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.491+0000 7fe5eeffd700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5652e2a60cf0 0x5652e2a6a4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:15:29.520 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.491+0000 7fe5eeffd700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5652e2a60cf0 0x5652e2a6a4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:46250/0 (socket says 172.21.15.2:46250) 2022-05-13T23:15:29.520 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.491+0000 7fe5eeffd700 1 -- 172.21.15.2:0/2751663132 learned_addr learned my addr 172.21.15.2:0/2751663132 (peer_addr_for_me v2:172.21.15.2:0/0) 2022-05-13T23:15:29.520 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.491+0000 7fe5ef7fe700 1 --2- 172.21.15.2:0/2751663132 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5652e2a625a0 0x5652e2a71a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:15:29.520 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.491+0000 7fe5ee7fc700 1 --2- 172.21.15.2:0/2751663132 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x5652e2a61640 0x5652e2a6aa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:15:29.521 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.492+0000 7fe5eeffd700 1 -- 172.21.15.2:0/2751663132 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5652e2a625a0 msgr2=0x5652e2a71a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-05-13T23:15:29.521 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.492+0000 7fe5eeffd700 1 --2- 172.21.15.2:0/2751663132 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5652e2a625a0 0x5652e2a71a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:29.521 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.492+0000 7fe5eeffd700 1 -- 172.21.15.2:0/2751663132 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x5652e2a61640 msgr2=0x5652e2a6aa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-05-13T23:15:29.521 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.492+0000 7fe5eeffd700 1 --2- 172.21.15.2:0/2751663132 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x5652e2a61640 0x5652e2a6aa30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:29.522 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.492+0000 7fe5eeffd700 1 -- 172.21.15.2:0/2751663132 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe5f0116a00 con 0x5652e2a60cf0 2022-05-13T23:15:29.522 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.492+0000 7fe5eeffd700 1 --2- 172.21.15.2:0/2751663132 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5652e2a60cf0 0x5652e2a6a4f0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fe5e008bf40 tx=0x7fe5e0004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f461bbfca7796d2 server_cookie=0 in_seq=0 out_seq=0 2022-05-13T23:15:29.522 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.492+0000 7fe5cdffb700 10 client.0 ms_handle_connect on v2:172.21.15.2:3300/0 2022-05-13T23:15:29.522 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.492+0000 7fe5cdffb700 1 -- 172.21.15.2:0/2751663132 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5e0007ca0 con 0x5652e2a60cf0 2022-05-13T23:15:29.522 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.492+0000 7fe5cdffb700 1 -- 172.21.15.2:0/2751663132 <== mon.0 v2:172.21.15.2:3300/0 2 ==== osd_map(85..85 src has 1..85) v4 ==== 8120+0+0 (secure 0 0 0) 0x7fe5e0090140 con 0x5652e2a60cf0 2022-05-13T23:15:29.523 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.492+0000 7fe604964b80 1 -- 172.21.15.2:0/2751663132 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5652e29629a0 con 0x5652e2a60cf0 2022-05-13T23:15:29.523 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.492+0000 7fe5cdffb700 1 -- 172.21.15.2:0/2751663132 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe5e00a8070 con 0x5652e2a60cf0 2022-05-13T23:15:29.523 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.492+0000 7fe604964b80 10 client.14874 fetch_fsmap learned FSMap version 26 2022-05-13T23:15:29.523 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.492+0000 7fe604964b80 1 -- 172.21.15.2:0/2751663132 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) v3 -- 0x7fe5e4005040 con 0x5652e2a60cf0 2022-05-13T23:15:29.524 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.493+0000 7fe5cdffb700 1 -- 172.21.15.2:0/2751663132 <== mon.0 v2:172.21.15.2:3300/0 4 ==== config(26 keys) v1 ==== 1087+0+0 (secure 0 0 0) 0x5652e29629a0 con 0x5652e2a60cf0 2022-05-13T23:15:29.524 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.493+0000 7fe5cdffb700 1 -- 172.21.15.2:0/2751663132 <== mon.0 v2:172.21.15.2:3300/0 5 ==== fsmap(e 26) v1 ==== 7448+0+0 (secure 0 0 0) 0x7fe5e006fb90 con 0x5652e2a60cf0 2022-05-13T23:15:29.524 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.493+0000 7fe604964b80 10 client.14874 fetch_fsmap finished waiting for FSMap version 26 2022-05-13T23:15:29.524 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.493+0000 7fe604964b80 10 client.14874 resolve_mds: resolved name 'f' to daemon mds.f 2022-05-13T23:15:29.524 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.493+0000 7fe604964b80 20 client.14874 populate_metadata read hostname 'smithi002' 2022-05-13T23:15:29.525 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.493+0000 7fe604964b80 1 --2- 172.21.15.2:0/2751663132 >> [v2:172.21.15.115:6834/71062276,v1:172.21.15.115:6835/71062276] conn(0x5652e2a751e0 0x5652e2a7a4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:15:29.525 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.493+0000 7fe604964b80 4 client.14874 mds_command: new command op to 24587 tid=0[{"prefix": "get_command_descriptions"}] 2022-05-13T23:15:29.525 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.493+0000 7fe604964b80 1 -- 172.21.15.2:0/2751663132 --> [v2:172.21.15.115:6834/71062276,v1:172.21.15.115:6835/71062276] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5652e2983330 con 0x5652e2a751e0 2022-05-13T23:15:29.525 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.493+0000 7fe5ee7fc700 1 --2- 172.21.15.2:0/2751663132 >> [v2:172.21.15.115:6834/71062276,v1:172.21.15.115:6835/71062276] conn(0x5652e2a751e0 0x5652e2a7a4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:15:29.525 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.494+0000 7fe5ee7fc700 1 --2- 172.21.15.2:0/2751663132 >> [v2:172.21.15.115:6834/71062276,v1:172.21.15.115:6835/71062276] conn(0x5652e2a751e0 0x5652e2a7a4e0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=a93768e844b5c5d3 server_cookie=e52086abe0546d3c in_seq=0 out_seq=0 2022-05-13T23:15:29.526 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.494+0000 7fe5cdffb700 10 client.14874 ms_handle_connect on v2:172.21.15.115:6834/71062276 2022-05-13T23:15:29.526 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.495+0000 7fe5cdffb700 1 -- 172.21.15.2:0/2751663132 <== mds.2 v2:172.21.15.115:6834/71062276 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17090 (crc 0 0 0) 0x5652e2983330 con 0x5652e2a751e0 2022-05-13T23:15:29.526 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.495+0000 7fe5cdffb700 10 client.14874 handle_command_reply: tid=0 2022-05-13T23:15:29.526 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.495+0000 7fe604964b80 1 client.14874 shutdown 2022-05-13T23:15:29.526 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.495+0000 7fe604964b80 1 -- 172.21.15.2:0/2751663132 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5652e2a60cf0 msgr2=0x5652e2a6a4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:15:29.527 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.495+0000 7fe604964b80 1 --2- 172.21.15.2:0/2751663132 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5652e2a60cf0 0x5652e2a6a4f0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fe5e008bf40 tx=0x7fe5e0004730 comp rx=0 tx=0).stop 2022-05-13T23:15:29.527 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.495+0000 7fe604964b80 1 -- 172.21.15.2:0/2751663132 shutdown_connections 2022-05-13T23:15:29.527 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.495+0000 7fe604964b80 1 --2- 172.21.15.2:0/2751663132 >> [v2:172.21.15.115:6834/71062276,v1:172.21.15.115:6835/71062276] conn(0x5652e2a751e0 0x5652e2a7a4e0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:29.527 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.495+0000 7fe604964b80 1 --2- 172.21.15.2:0/2751663132 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5652e2a625a0 0x5652e2a71a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:29.528 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.495+0000 7fe604964b80 1 --2- 172.21.15.2:0/2751663132 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5652e2a60cf0 0x5652e2a6a4f0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:29.528 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.495+0000 7fe5cdffb700 0 client.14874 ms_handle_reset on v2:172.21.15.115:6834/71062276 2022-05-13T23:15:29.528 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.495+0000 7fe604964b80 1 --2- 172.21.15.2:0/2751663132 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x5652e2a61640 0x5652e2a6aa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:29.528 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.495+0000 7fe604964b80 1 -- 172.21.15.2:0/2751663132 >> 172.21.15.2:0/2751663132 conn(0x5652e2a5bb10 msgr2=0x5652e2a5ad80 unknown :-1 s=STATE_NONE l=0).mark_down 2022-05-13T23:15:29.529 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.495+0000 7fe604964b80 1 -- 172.21.15.2:0/2751663132 shutdown_connections 2022-05-13T23:15:29.529 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.495+0000 7fe604964b80 1 -- 172.21.15.2:0/2751663132 wait complete. 2022-05-13T23:15:29.529 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.496+0000 7fe604964b80 20 client.14874 trim_cache size 0 max 16384 2022-05-13T23:15:29.536 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.513+0000 7fe604964b80 1 Processor -- start 2022-05-13T23:15:29.536 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.513+0000 7fe604964b80 1 -- start start 2022-05-13T23:15:29.536 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.513+0000 7fe604964b80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5652e2a60cf0 0x5652e2a610c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:15:29.537 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.513+0000 7fe604964b80 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5652e2a61640 0x5652e2a7e500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:15:29.537 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.513+0000 7fe604964b80 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x5652e2a625a0 0x5652e2a5a300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:15:29.537 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.513+0000 7fe604964b80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x5652e2983360 con 0x5652e2a60cf0 2022-05-13T23:15:29.537 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.513+0000 7fe604964b80 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x5652e29629d0 con 0x5652e2a625a0 2022-05-13T23:15:29.538 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.513+0000 7fe604964b80 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x5652e2962db0 con 0x5652e2a61640 2022-05-13T23:15:29.538 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.514+0000 7fe5eeffd700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5652e2a60cf0 0x5652e2a610c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:15:29.538 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.514+0000 7fe5eeffd700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5652e2a60cf0 0x5652e2a610c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:46262/0 (socket says 172.21.15.2:46262) 2022-05-13T23:15:29.538 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.514+0000 7fe5eeffd700 1 -- 172.21.15.2:0/3969794757 learned_addr learned my addr 172.21.15.2:0/3969794757 (peer_addr_for_me v2:172.21.15.2:0/0) 2022-05-13T23:15:29.539 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.514+0000 7fe5ef7fe700 1 --2- 172.21.15.2:0/3969794757 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x5652e2a625a0 0x5652e2a5a300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:15:29.539 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.514+0000 7fe5ee7fc700 1 --2- 172.21.15.2:0/3969794757 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5652e2a61640 0x5652e2a7e500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:15:29.539 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.514+0000 7fe5eeffd700 1 -- 172.21.15.2:0/3969794757 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5652e2a61640 msgr2=0x5652e2a7e500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-05-13T23:15:29.539 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.514+0000 7fe5eeffd700 1 --2- 172.21.15.2:0/3969794757 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5652e2a61640 0x5652e2a7e500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:29.539 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.514+0000 7fe5eeffd700 1 -- 172.21.15.2:0/3969794757 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x5652e2a625a0 msgr2=0x5652e2a5a300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-05-13T23:15:29.540 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.514+0000 7fe5eeffd700 1 --2- 172.21.15.2:0/3969794757 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x5652e2a625a0 0x5652e2a5a300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:29.540 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.514+0000 7fe5eeffd700 1 -- 172.21.15.2:0/3969794757 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5652e27f0fe0 con 0x5652e2a60cf0 2022-05-13T23:15:29.540 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.514+0000 7fe5ee7fc700 1 --2- 172.21.15.2:0/3969794757 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5652e2a61640 0x5652e2a7e500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2022-05-13T23:15:29.540 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.514+0000 7fe5eeffd700 1 --2- 172.21.15.2:0/3969794757 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5652e2a60cf0 0x5652e2a610c0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fe5e008f740 tx=0x7fe5e0004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5a42c2d7c1023d9f server_cookie=0 in_seq=0 out_seq=0 2022-05-13T23:15:29.541 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.515+0000 7fe5d51ff700 1 -- 172.21.15.2:0/3969794757 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5e00a8070 con 0x5652e2a60cf0 2022-05-13T23:15:29.541 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.515+0000 7fe5d51ff700 1 -- 172.21.15.2:0/3969794757 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1087+0+0 (secure 0 0 0) 0x7fe5e00bc1d0 con 0x5652e2a60cf0 2022-05-13T23:15:29.541 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.515+0000 7fe5d51ff700 1 -- 172.21.15.2:0/3969794757 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5e0096130 con 0x5652e2a60cf0 2022-05-13T23:15:29.541 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.515+0000 7fe604964b80 1 -- 172.21.15.2:0/3969794757 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5652e2a60cf0 msgr2=0x5652e2a610c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:15:29.541 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.515+0000 7fe604964b80 1 --2- 172.21.15.2:0/3969794757 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5652e2a60cf0 0x5652e2a610c0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fe5e008f740 tx=0x7fe5e0004730 comp rx=0 tx=0).stop 2022-05-13T23:15:29.542 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.515+0000 7fe604964b80 1 -- 172.21.15.2:0/3969794757 shutdown_connections 2022-05-13T23:15:29.542 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.515+0000 7fe604964b80 1 --2- 172.21.15.2:0/3969794757 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x5652e2a625a0 0x5652e2a5a300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:29.542 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.515+0000 7fe604964b80 1 --2- 172.21.15.2:0/3969794757 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5652e2a60cf0 0x5652e2a610c0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:29.542 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.515+0000 7fe604964b80 1 --2- 172.21.15.2:0/3969794757 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5652e2a61640 0x5652e2a7e500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:29.542 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.515+0000 7fe604964b80 1 -- 172.21.15.2:0/3969794757 >> 172.21.15.2:0/3969794757 conn(0x5652e2a5bb10 msgr2=0x5652e2a5ad80 unknown :-1 s=STATE_NONE l=0).mark_down 2022-05-13T23:15:29.543 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.516+0000 7fe604964b80 1 -- 172.21.15.2:0/3969794757 shutdown_connections 2022-05-13T23:15:29.543 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.516+0000 7fe604964b80 1 -- 172.21.15.2:0/3969794757 wait complete. 2022-05-13T23:15:29.543 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.516+0000 7fe604964b80 1 -- start start 2022-05-13T23:15:29.543 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.516+0000 7fe604964b80 1 Processor -- start 2022-05-13T23:15:29.543 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.517+0000 7fe604964b80 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x5652e2a60cf0 0x5652e2a6ec30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:15:29.544 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.517+0000 7fe604964b80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5652e2a61640 0x5652e2a6f170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:15:29.544 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.517+0000 7fe604964b80 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5652e2a625a0 0x5652e2ac7200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:15:29.544 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.517+0000 7fe604964b80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x5652e2a6f8a0 con 0x5652e2a61640 2022-05-13T23:15:29.545 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.517+0000 7fe604964b80 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x5652e2ac7740 con 0x5652e2a60cf0 2022-05-13T23:15:29.545 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.517+0000 7fe604964b80 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x5652e2ac7900 con 0x5652e2a625a0 2022-05-13T23:15:29.545 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.517+0000 7fe604964b80 10 client.0 _ll_register_callbacks cb 0x5652e2a22de0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2022-05-13T23:15:29.546 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.517+0000 7fe5ee7fc700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5652e2a61640 0x5652e2a6f170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:15:29.546 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.517+0000 7fe5eeffd700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x5652e2a60cf0 0x5652e2a6ec30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:15:29.546 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.517+0000 7fe5ef7fe700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5652e2a625a0 0x5652e2ac7200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:15:29.547 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.517+0000 7fe5ee7fc700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5652e2a61640 0x5652e2a6f170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:46266/0 (socket says 172.21.15.2:46266) 2022-05-13T23:15:29.547 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.517+0000 7fe5ee7fc700 1 -- 172.21.15.2:0/2934098446 learned_addr learned my addr 172.21.15.2:0/2934098446 (peer_addr_for_me v2:172.21.15.2:0/0) 2022-05-13T23:15:29.547 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.518+0000 7fe5ef7fe700 1 -- 172.21.15.2:0/2934098446 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x5652e2a60cf0 msgr2=0x5652e2a6ec30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-05-13T23:15:29.547 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.518+0000 7fe5ef7fe700 1 --2- 172.21.15.2:0/2934098446 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x5652e2a60cf0 0x5652e2a6ec30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:29.548 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.518+0000 7fe5ef7fe700 1 -- 172.21.15.2:0/2934098446 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5652e2a61640 msgr2=0x5652e2a6f170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-05-13T23:15:29.548 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.518+0000 7fe5ef7fe700 1 --2- 172.21.15.2:0/2934098446 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5652e2a61640 0x5652e2a6f170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:29.548 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.518+0000 7fe5ef7fe700 1 -- 172.21.15.2:0/2934098446 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe5e4005040 con 0x5652e2a625a0 2022-05-13T23:15:29.548 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.518+0000 7fe5eeffd700 1 --2- 172.21.15.2:0/2934098446 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x5652e2a60cf0 0x5652e2a6ec30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-05-13T23:15:29.548 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.518+0000 7fe5ee7fc700 1 --2- 172.21.15.2:0/2934098446 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5652e2a61640 0x5652e2a6f170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-05-13T23:15:29.549 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.518+0000 7fe5ef7fe700 1 --2- 172.21.15.2:0/2934098446 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5652e2a625a0 0x5652e2ac7200 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fe5e8009bb0 tx=0x7fe5e8021000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f7ca7c5e3e301f20 server_cookie=0 in_seq=0 out_seq=0 2022-05-13T23:15:29.549 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.518+0000 7fe5cdffb700 10 client.0 ms_handle_connect on v2:172.21.15.115:3300/0 2022-05-13T23:15:29.549 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.518+0000 7fe5cdffb700 1 -- 172.21.15.2:0/2934098446 <== mon.2 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5e802e070 con 0x5652e2a625a0 2022-05-13T23:15:29.549 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.518+0000 7fe604964b80 1 -- 172.21.15.2:0/2934098446 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5652e2a315f0 con 0x5652e2a625a0 2022-05-13T23:15:29.550 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.518+0000 7fe5cdffb700 1 -- 172.21.15.2:0/2934098446 <== mon.2 v2:172.21.15.115:3300/0 2 ==== osd_map(85..85 src has 1..85) v4 ==== 8120+0+0 (secure 0 0 0) 0x7fe5e8029620 con 0x5652e2a625a0 2022-05-13T23:15:29.550 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.519+0000 7fe5cdffb700 1 -- 172.21.15.2:0/2934098446 <== mon.2 v2:172.21.15.115:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe5e800e070 con 0x5652e2a625a0 2022-05-13T23:15:29.550 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.519+0000 7fe604964b80 10 client.25001 fetch_fsmap learned FSMap version 26 2022-05-13T23:15:29.550 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.519+0000 7fe604964b80 1 -- 172.21.15.2:0/2934098446 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) v3 -- 0x5652e2818f80 con 0x5652e2a625a0 2022-05-13T23:15:29.550 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.519+0000 7fe5cdffb700 1 -- 172.21.15.2:0/2934098446 <== mon.2 v2:172.21.15.115:3300/0 4 ==== config(26 keys) v1 ==== 1087+0+0 (secure 0 0 0) 0x5652e2a315f0 con 0x5652e2a625a0 2022-05-13T23:15:29.551 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.519+0000 7fe5cdffb700 1 -- 172.21.15.2:0/2934098446 <== mon.2 v2:172.21.15.115:3300/0 5 ==== fsmap(e 26) v1 ==== 7448+0+0 (secure 0 0 0) 0x7fe5e8033b40 con 0x5652e2a625a0 2022-05-13T23:15:29.551 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.519+0000 7fe604964b80 10 client.25001 fetch_fsmap finished waiting for FSMap version 26 2022-05-13T23:15:29.551 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.519+0000 7fe604964b80 10 client.25001 resolve_mds: resolved name 'f' to daemon mds.f 2022-05-13T23:15:29.551 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.519+0000 7fe604964b80 20 client.25001 populate_metadata read hostname 'smithi002' 2022-05-13T23:15:29.551 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.519+0000 7fe604964b80 1 --2- 172.21.15.2:0/2934098446 >> [v2:172.21.15.115:6834/71062276,v1:172.21.15.115:6835/71062276] conn(0x5652e2acc1c0 0x5652e2ac8b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:15:29.552 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.519+0000 7fe604964b80 4 client.25001 mds_command: new command op to 24587 tid=0[{"prefix": "status"}] 2022-05-13T23:15:29.552 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.519+0000 7fe604964b80 1 -- 172.21.15.2:0/2934098446 --> [v2:172.21.15.115:6834/71062276,v1:172.21.15.115:6835/71062276] -- command(tid 0: {"prefix": "status"}) v1 -- 0x5652e2954790 con 0x5652e2acc1c0 2022-05-13T23:15:29.552 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.519+0000 7fe5eeffd700 1 --2- 172.21.15.2:0/2934098446 >> [v2:172.21.15.115:6834/71062276,v1:172.21.15.115:6835/71062276] conn(0x5652e2acc1c0 0x5652e2ac8b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:15:29.552 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.520+0000 7fe5eeffd700 1 --2- 172.21.15.2:0/2934098446 >> [v2:172.21.15.115:6834/71062276,v1:172.21.15.115:6835/71062276] conn(0x5652e2acc1c0 0x5652e2ac8b10 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=daf54e50d1ff8780 server_cookie=68b946f494df53d7 in_seq=0 out_seq=0 2022-05-13T23:15:29.552 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.520+0000 7fe5cdffb700 10 client.25001 ms_handle_connect on v2:172.21.15.115:6834/71062276 2022-05-13T23:15:29.553 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.520+0000 7fe5cdffb700 1 -- 172.21.15.2:0/2934098446 <== mds.2 v2:172.21.15.115:6834/71062276 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+332 (crc 0 0 0) 0x5652e2954790 con 0x5652e2acc1c0 2022-05-13T23:15:29.553 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.520+0000 7fe5cdffb700 10 client.25001 handle_command_reply: tid=0 2022-05-13T23:15:29.553 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.520+0000 7fe604964b80 1 client.25001 shutdown 2022-05-13T23:15:29.553 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.520+0000 7fe604964b80 1 -- 172.21.15.2:0/2934098446 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5652e2a625a0 msgr2=0x5652e2ac7200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:15:29.554 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.520+0000 7fe604964b80 1 --2- 172.21.15.2:0/2934098446 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5652e2a625a0 0x5652e2ac7200 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fe5e8009bb0 tx=0x7fe5e8021000 comp rx=0 tx=0).stop 2022-05-13T23:15:29.554 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.520+0000 7fe604964b80 1 -- 172.21.15.2:0/2934098446 shutdown_connections 2022-05-13T23:15:29.554 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.520+0000 7fe604964b80 1 --2- 172.21.15.2:0/2934098446 >> [v2:172.21.15.115:6834/71062276,v1:172.21.15.115:6835/71062276] conn(0x5652e2acc1c0 0x5652e2ac8b10 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:29.554 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.520+0000 7fe604964b80 1 --2- 172.21.15.2:0/2934098446 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5652e2a625a0 0x5652e2ac7200 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:29.554 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.520+0000 7fe604964b80 1 --2- 172.21.15.2:0/2934098446 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x5652e2a60cf0 0x5652e2a6ec30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:29.555 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.520+0000 7fe604964b80 1 --2- 172.21.15.2:0/2934098446 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5652e2a61640 0x5652e2a6f170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:29.555 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.520+0000 7fe5cdffb700 0 client.25001 ms_handle_reset on v2:172.21.15.115:6834/71062276 2022-05-13T23:15:29.555 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.521+0000 7fe604964b80 1 -- 172.21.15.2:0/2934098446 >> 172.21.15.2:0/2934098446 conn(0x5652e2a5bb10 msgr2=0x5652e2a7e500 unknown :-1 s=STATE_NONE l=0).mark_down 2022-05-13T23:15:29.555 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.521+0000 7fe604964b80 1 -- 172.21.15.2:0/2934098446 shutdown_connections 2022-05-13T23:15:29.556 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.521+0000 7fe604964b80 1 -- 172.21.15.2:0/2934098446 wait complete. 2022-05-13T23:15:29.556 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.521+0000 7fe604964b80 20 client.25001 trim_cache size 0 max 16384 2022-05-13T23:15:29.556 INFO:teuthology.orchestra.run.smithi002.stdout:{ 2022-05-13T23:15:29.557 INFO:teuthology.orchestra.run.smithi002.stdout: "cluster_fsid": "16cf5f88-d311-11ec-be40-001a4aab830c", 2022-05-13T23:15:29.557 INFO:teuthology.orchestra.run.smithi002.stdout: "whoami": 2, 2022-05-13T23:15:29.557 INFO:teuthology.orchestra.run.smithi002.stdout: "id": 24587, 2022-05-13T23:15:29.557 INFO:teuthology.orchestra.run.smithi002.stdout: "want_state": "up:active", 2022-05-13T23:15:29.558 INFO:teuthology.orchestra.run.smithi002.stdout: "state": "up:active", 2022-05-13T23:15:29.558 INFO:teuthology.orchestra.run.smithi002.stdout: "fs_name": "cephfs", 2022-05-13T23:15:29.558 INFO:teuthology.orchestra.run.smithi002.stdout: "rank_uptime": 69.228107338000001, 2022-05-13T23:15:29.558 INFO:teuthology.orchestra.run.smithi002.stdout: "mdsmap_epoch": 26, 2022-05-13T23:15:29.558 INFO:teuthology.orchestra.run.smithi002.stdout: "osdmap_epoch": 85, 2022-05-13T23:15:29.559 INFO:teuthology.orchestra.run.smithi002.stdout: "osdmap_epoch_barrier": 83, 2022-05-13T23:15:29.559 INFO:teuthology.orchestra.run.smithi002.stdout: "uptime": 83.994305855999997 2022-05-13T23:15:29.559 INFO:teuthology.orchestra.run.smithi002.stdout:} 2022-05-13T23:15:29.560 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.521+0000 7fe5effff700 1 -- 172.21.15.2:0/2764764927 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7fe5d8096d60 msgr2=0x7fe5d8099210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:15:29.560 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.521+0000 7fe5effff700 1 --2- 172.21.15.2:0/2764764927 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7fe5d8096d60 0x7fe5d8099210 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fe5e800b3b0 tx=0x7fe5e8004d10 comp rx=0 tx=0).stop 2022-05-13T23:15:29.561 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.521+0000 7fe5effff700 1 -- 172.21.15.2:0/2764764927 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe5f00fb470 msgr2=0x7fe5f010e380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:15:29.561 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.521+0000 7fe5effff700 1 --2- 172.21.15.2:0/2764764927 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe5f00fb470 0x7fe5f010e380 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fe5e000e940 tx=0x7fe5e000b040 comp rx=0 tx=0).stop 2022-05-13T23:15:29.561 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.521+0000 7fe5effff700 1 -- 172.21.15.2:0/2764764927 shutdown_connections 2022-05-13T23:15:29.561 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.521+0000 7fe5effff700 1 --2- 172.21.15.2:0/2764764927 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe5f00fd0a0 0x7fe5f0115950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:29.562 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.521+0000 7fe5effff700 1 --2- 172.21.15.2:0/2764764927 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7fe5d8096d60 0x7fe5d8099210 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:29.562 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.521+0000 7fe5effff700 1 --2- 172.21.15.2:0/2764764927 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe5f00fb470 0x7fe5f010e380 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:29.562 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.521+0000 7fe5effff700 1 --2- 172.21.15.2:0/2764764927 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe5f00fc710 0x7fe5f010e8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:29.562 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.521+0000 7fe5effff700 1 -- 172.21.15.2:0/2764764927 >> 172.21.15.2:0/2764764927 conn(0x7fe5f00f6c60 msgr2=0x7fe5f0101fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-05-13T23:15:29.562 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.522+0000 7fe5effff700 1 -- 172.21.15.2:0/2764764927 shutdown_connections 2022-05-13T23:15:29.563 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.522+0000 7fe5effff700 1 -- 172.21.15.2:0/2764764927 wait complete. 2022-05-13T23:15:29.563 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:15:29 smithi002 ceph-mon[71816]: from='client.? 172.21.15.2:0/1549106648' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2022-05-13T23:15:29.564 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:15:29 smithi047 ceph-mon[77147]: from='client.? 172.21.15.2:0/1549106648' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2022-05-13T23:15:29.565 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.i status 2022-05-13T23:15:29.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:15:29 smithi115 ceph-mon[78593]: from='client.? 172.21.15.2:0/1549106648' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2022-05-13T23:15:29.788 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.781+0000 7f6c74b9d700 1 -- 172.21.15.2:0/475937651 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6c70103400 msgr2=0x7f6c701057e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:15:29.789 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.781+0000 7f6c74b9d700 1 --2- 172.21.15.2:0/475937651 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6c70103400 0x7f6c701057e0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f6c68009c30 tx=0x7f6c68004c50 comp rx=0 tx=0).stop 2022-05-13T23:15:29.789 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.781+0000 7f6c74b9d700 1 -- 172.21.15.2:0/475937651 shutdown_connections 2022-05-13T23:15:29.789 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.781+0000 7f6c74b9d700 1 --2- 172.21.15.2:0/475937651 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6c70103400 0x7f6c701057e0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:29.789 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.781+0000 7f6c74b9d700 1 --2- 172.21.15.2:0/475937651 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6c70070370 0x7f6c7006e870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:29.790 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.781+0000 7f6c74b9d700 1 --2- 172.21.15.2:0/475937651 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f6c7006edb0 0x7f6c7006f220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:29.790 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.781+0000 7f6c74b9d700 1 -- 172.21.15.2:0/475937651 >> 172.21.15.2:0/475937651 conn(0x7f6c700f6c60 msgr2=0x7f6c700f9090 unknown :-1 s=STATE_NONE l=0).mark_down 2022-05-13T23:15:29.791 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.781+0000 7f6c74b9d700 1 -- 172.21.15.2:0/475937651 shutdown_connections 2022-05-13T23:15:29.791 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.781+0000 7f6c74b9d700 1 -- 172.21.15.2:0/475937651 wait complete. 2022-05-13T23:15:29.791 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.782+0000 7f6c74b9d700 1 Processor -- start 2022-05-13T23:15:29.792 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.782+0000 7f6c74b9d700 1 -- start start 2022-05-13T23:15:29.794 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.783+0000 7f6c74b9d700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6c7006edb0 0x7f6c7010e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:15:29.794 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.783+0000 7f6c74b9d700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6c70070370 0x7f6c7010e830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:15:29.795 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.783+0000 7f6c74b9d700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f6c70103400 0x7f6c701158c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:15:29.795 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.783+0000 7f6c74b9d700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c70115e00 con 0x7f6c70070370 2022-05-13T23:15:29.795 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.783+0000 7f6c74b9d700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c70115f70 con 0x7f6c7006edb0 2022-05-13T23:15:29.795 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.783+0000 7f6c74b9d700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c70116130 con 0x7f6c70103400 2022-05-13T23:15:29.795 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.783+0000 7f6c6effd700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6c70070370 0x7f6c7010e830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:15:29.796 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.783+0000 7f6c6effd700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6c70070370 0x7f6c7010e830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:51872/0 (socket says 172.21.15.2:51872) 2022-05-13T23:15:29.796 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.783+0000 7f6c6effd700 1 -- 172.21.15.2:0/3425172745 learned_addr learned my addr 172.21.15.2:0/3425172745 (peer_addr_for_me v2:172.21.15.2:0/0) 2022-05-13T23:15:29.796 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.783+0000 7f6c6f7fe700 1 --2- 172.21.15.2:0/3425172745 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6c7006edb0 0x7f6c7010e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:15:29.796 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.783+0000 7f6c6ffff700 1 --2- 172.21.15.2:0/3425172745 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f6c70103400 0x7f6c701158c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:15:29.796 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.783+0000 7f6c6effd700 1 -- 172.21.15.2:0/3425172745 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f6c70103400 msgr2=0x7f6c701158c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:15:29.797 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.783+0000 7f6c6effd700 1 --2- 172.21.15.2:0/3425172745 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f6c70103400 0x7f6c701158c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:29.797 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.783+0000 7f6c6effd700 1 -- 172.21.15.2:0/3425172745 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6c7006edb0 msgr2=0x7f6c7010e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:15:29.797 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.783+0000 7f6c6effd700 1 --2- 172.21.15.2:0/3425172745 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6c7006edb0 0x7f6c7010e2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:29.797 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.783+0000 7f6c6effd700 1 -- 172.21.15.2:0/3425172745 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c60005040 con 0x7f6c70070370 2022-05-13T23:15:29.798 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.784+0000 7f6c6effd700 1 --2- 172.21.15.2:0/3425172745 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6c70070370 0x7f6c7010e830 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f6c6400e970 tx=0x7f6c6400b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-05-13T23:15:29.798 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.784+0000 7f6c57fff700 1 -- 172.21.15.2:0/3425172745 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c64007b40 con 0x7f6c70070370 2022-05-13T23:15:29.798 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.784+0000 7f6c57fff700 1 -- 172.21.15.2:0/3425172745 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1087+0+0 (secure 0 0 0) 0x7f6c6400f040 con 0x7f6c70070370 2022-05-13T23:15:29.798 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.784+0000 7f6c74b9d700 1 -- 172.21.15.2:0/3425172745 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6c680097c0 con 0x7f6c70070370 2022-05-13T23:15:29.798 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.784+0000 7f6c57fff700 1 -- 172.21.15.2:0/3425172745 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c64014950 con 0x7f6c70070370 2022-05-13T23:15:29.799 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.784+0000 7f6c74b9d700 1 -- 172.21.15.2:0/3425172745 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6c70116840 con 0x7f6c70070370 2022-05-13T23:15:29.812 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.788+0000 7f6c57fff700 1 -- 172.21.15.2:0/3425172745 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 16) v1 ==== 125136+0+0 (secure 0 0 0) 0x7f6c6401b030 con 0x7f6c70070370 2022-05-13T23:15:29.812 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.789+0000 7f6c850d3b80 1 Processor -- start 2022-05-13T23:15:29.813 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.789+0000 7f6c850d3b80 1 -- start start 2022-05-13T23:15:29.813 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.789+0000 7f6c57fff700 1 --2- 172.21.15.2:0/3425172745 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f6c58096d60 0x7f6c58099210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:15:29.814 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.789+0000 7f6c850d3b80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55b025f70cf0 0x55b025f71100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:15:29.814 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.789+0000 7f6c850d3b80 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55b025f71640 0x55b025f71f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:15:29.815 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.789+0000 7f6c850d3b80 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55b025f725a0 0x55b025f729f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:15:29.815 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.789+0000 7f6c850d3b80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b025e6b3a0 con 0x55b025f70cf0 2022-05-13T23:15:29.815 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.789+0000 7f6c850d3b80 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b025e40400 con 0x55b025f725a0 2022-05-13T23:15:29.816 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.789+0000 7f6c850d3b80 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b025ec46c0 con 0x55b025f71640 2022-05-13T23:15:29.816 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.789+0000 7f6c57fff700 1 -- 172.21.15.2:0/3425172745 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8120+0+0 (secure 0 0 0) 0x7f6c640bea80 con 0x7f6c70070370 2022-05-13T23:15:29.817 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.790+0000 7f6c6ffff700 1 --2- 172.21.15.2:0/3425172745 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f6c58096d60 0x7f6c58099210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:15:29.817 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.790+0000 7f6c6effd700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55b025f71640 0x55b025f71f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:15:29.818 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.790+0000 7f6c6effd700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55b025f71640 0x55b025f71f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.2:56492/0 (socket says 172.21.15.2:56492) 2022-05-13T23:15:29.819 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.790+0000 7f6c6effd700 1 -- 172.21.15.2:0/3205851005 learned_addr learned my addr 172.21.15.2:0/3205851005 (peer_addr_for_me v2:172.21.15.2:0/0) 2022-05-13T23:15:29.820 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.790+0000 7f6c6ffff700 1 --2- 172.21.15.2:0/3205851005 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55b025f725a0 0x55b025f729f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:15:29.820 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.790+0000 7f6c6f7fe700 1 --2- 172.21.15.2:0/3205851005 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55b025f70cf0 0x55b025f71100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:15:29.820 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.790+0000 7f6c6effd700 1 -- 172.21.15.2:0/3205851005 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55b025f725a0 msgr2=0x55b025f729f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-05-13T23:15:29.821 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.790+0000 7f6c6effd700 1 --2- 172.21.15.2:0/3205851005 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55b025f725a0 0x55b025f729f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:29.822 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.790+0000 7f6c6effd700 1 -- 172.21.15.2:0/3205851005 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55b025f70cf0 msgr2=0x55b025f71100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-05-13T23:15:29.822 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.790+0000 7f6c6effd700 1 --2- 172.21.15.2:0/3205851005 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55b025f70cf0 0x55b025f71100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:29.822 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.790+0000 7f6c6effd700 1 -- 172.21.15.2:0/3205851005 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b025d56360 con 0x55b025f71640 2022-05-13T23:15:29.823 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.791+0000 7f6c6effd700 1 --2- 172.21.15.2:0/3205851005 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55b025f71640 0x55b025f71f80 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f6c6408f980 tx=0x7f6c6408b690 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c74bf9c87ccb2a60 server_cookie=0 in_seq=0 out_seq=0 2022-05-13T23:15:29.823 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.791+0000 7f6c6ffff700 1 --2- 172.21.15.2:0/3425172745 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f6c58096d60 0x7f6c58099210 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f6c68009ad0 tx=0x7f6c68004c50 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-05-13T23:15:29.824 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.791+0000 7f6c551fe700 1 -- 172.21.15.2:0/3205851005 <== mon.2 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c6408f530 con 0x55b025f71640 2022-05-13T23:15:29.824 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.791+0000 7f6c551fe700 1 -- 172.21.15.2:0/3205851005 <== mon.2 v2:172.21.15.115:3300/0 2 ==== config(26 keys) v1 ==== 1087+0+0 (secure 0 0 0) 0x7f6c6408f350 con 0x55b025f71640 2022-05-13T23:15:29.825 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.791+0000 7f6c551fe700 1 -- 172.21.15.2:0/3205851005 <== mon.2 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c6408fc50 con 0x55b025f71640 2022-05-13T23:15:29.825 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.792+0000 7f6c850d3b80 1 -- 172.21.15.2:0/3205851005 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55b025f71640 msgr2=0x55b025f71f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:15:29.825 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.792+0000 7f6c850d3b80 1 --2- 172.21.15.2:0/3205851005 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55b025f71640 0x55b025f71f80 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f6c6408f980 tx=0x7f6c6408b690 comp rx=0 tx=0).stop 2022-05-13T23:15:29.826 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.792+0000 7f6c850d3b80 1 -- 172.21.15.2:0/3205851005 shutdown_connections 2022-05-13T23:15:29.826 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.792+0000 7f6c850d3b80 1 --2- 172.21.15.2:0/3205851005 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55b025f725a0 0x55b025f729f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:29.827 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.792+0000 7f6c850d3b80 1 --2- 172.21.15.2:0/3205851005 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55b025f70cf0 0x55b025f71100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:29.827 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.792+0000 7f6c850d3b80 1 --2- 172.21.15.2:0/3205851005 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55b025f71640 0x55b025f71f80 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:29.828 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.792+0000 7f6c850d3b80 1 -- 172.21.15.2:0/3205851005 >> 172.21.15.2:0/3205851005 conn(0x55b025f6bb10 msgr2=0x55b025f6bf00 unknown :-1 s=STATE_NONE l=0).mark_down 2022-05-13T23:15:29.828 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.792+0000 7f6c850d3b80 1 -- 172.21.15.2:0/3205851005 shutdown_connections 2022-05-13T23:15:29.829 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.792+0000 7f6c850d3b80 1 -- 172.21.15.2:0/3205851005 wait complete. 2022-05-13T23:15:29.830 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.792+0000 7f6c850d3b80 1 -- start start 2022-05-13T23:15:29.830 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.793+0000 7f6c850d3b80 1 Processor -- start 2022-05-13T23:15:29.831 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.793+0000 7f6c850d3b80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55b025f70cf0 0x55b025f7a470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:15:29.831 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.793+0000 7f6c850d3b80 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55b025f71640 0x55b025f7a9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:15:29.831 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.793+0000 7f6c850d3b80 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55b025f725a0 0x55b025f81a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:15:29.832 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.793+0000 7f6c850d3b80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b025ebe7a0 con 0x55b025f70cf0 2022-05-13T23:15:29.832 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.793+0000 7f6c850d3b80 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b025d58650 con 0x55b025f725a0 2022-05-13T23:15:29.832 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.793+0000 7f6c850d3b80 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b025f82040 con 0x55b025f71640 2022-05-13T23:15:29.833 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.793+0000 7f6c850d3b80 10 client.0 _ll_register_callbacks cb 0x55b025f32de0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2022-05-13T23:15:29.833 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.793+0000 7f6c6f7fe700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55b025f70cf0 0x55b025f7a470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:15:29.833 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.793+0000 7f6c6f7fe700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55b025f70cf0 0x55b025f7a470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:51892/0 (socket says 172.21.15.2:51892) 2022-05-13T23:15:29.834 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.793+0000 7f6c6f7fe700 1 -- 172.21.15.2:0/1912494364 learned_addr learned my addr 172.21.15.2:0/1912494364 (peer_addr_for_me v2:172.21.15.2:0/0) 2022-05-13T23:15:29.834 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.794+0000 7f6c6ffff700 1 --2- 172.21.15.2:0/1912494364 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55b025f725a0 0x55b025f81a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:15:29.834 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.794+0000 7f6c6effd700 1 --2- 172.21.15.2:0/1912494364 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55b025f71640 0x55b025f7a9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:15:29.835 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.794+0000 7f6c6f7fe700 1 -- 172.21.15.2:0/1912494364 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55b025f71640 msgr2=0x55b025f7a9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-05-13T23:15:29.835 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.794+0000 7f6c6f7fe700 1 --2- 172.21.15.2:0/1912494364 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55b025f71640 0x55b025f7a9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:29.835 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.794+0000 7f6c6f7fe700 1 -- 172.21.15.2:0/1912494364 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55b025f725a0 msgr2=0x55b025f81a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-05-13T23:15:29.836 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.794+0000 7f6c6f7fe700 1 --2- 172.21.15.2:0/1912494364 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55b025f725a0 0x55b025f81a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:29.836 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.794+0000 7f6c6f7fe700 1 -- 172.21.15.2:0/1912494364 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6c70116840 con 0x55b025f70cf0 2022-05-13T23:15:29.836 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.794+0000 7f6c6effd700 1 --2- 172.21.15.2:0/1912494364 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55b025f71640 0x55b025f7a9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2022-05-13T23:15:29.837 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.794+0000 7f6c6f7fe700 1 --2- 172.21.15.2:0/1912494364 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55b025f70cf0 0x55b025f7a470 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f6c60009080 tx=0x7f6c6000b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=344c14db8cab12e4 server_cookie=0 in_seq=0 out_seq=0 2022-05-13T23:15:29.837 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.794+0000 7f6c4e7fc700 10 client.0 ms_handle_connect on v2:172.21.15.2:3300/0 2022-05-13T23:15:29.838 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.794+0000 7f6c4e7fc700 1 -- 172.21.15.2:0/1912494364 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c6000e040 con 0x55b025f70cf0 2022-05-13T23:15:29.838 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.795+0000 7f6c850d3b80 1 -- 172.21.15.2:0/1912494364 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55b025e729a0 con 0x55b025f70cf0 2022-05-13T23:15:29.838 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.795+0000 7f6c4e7fc700 1 -- 172.21.15.2:0/1912494364 <== mon.0 v2:172.21.15.2:3300/0 2 ==== osd_map(85..85 src has 1..85) v4 ==== 8120+0+0 (secure 0 0 0) 0x7f6c600166e0 con 0x55b025f70cf0 2022-05-13T23:15:29.838 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.795+0000 7f6c4e7fc700 1 -- 172.21.15.2:0/1912494364 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6c60016ab0 con 0x55b025f70cf0 2022-05-13T23:15:29.839 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.795+0000 7f6c850d3b80 10 client.14892 fetch_fsmap learned FSMap version 26 2022-05-13T23:15:29.839 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.795+0000 7f6c850d3b80 1 -- 172.21.15.2:0/1912494364 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) v3 -- 0x55b025d00fe0 con 0x55b025f70cf0 2022-05-13T23:15:29.839 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.795+0000 7f6c4e7fc700 1 -- 172.21.15.2:0/1912494364 <== mon.0 v2:172.21.15.2:3300/0 4 ==== config(26 keys) v1 ==== 1087+0+0 (secure 0 0 0) 0x55b025e729a0 con 0x55b025f70cf0 2022-05-13T23:15:29.839 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.795+0000 7f6c4e7fc700 1 -- 172.21.15.2:0/1912494364 <== mon.0 v2:172.21.15.2:3300/0 5 ==== fsmap(e 26) v1 ==== 7448+0+0 (secure 0 0 0) 0x7f6c6001ecd0 con 0x55b025f70cf0 2022-05-13T23:15:29.840 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.795+0000 7f6c850d3b80 10 client.14892 fetch_fsmap finished waiting for FSMap version 26 2022-05-13T23:15:29.841 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.795+0000 7f6c850d3b80 10 client.14892 resolve_mds: resolved name 'i' to daemon mds.i 2022-05-13T23:15:29.841 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.795+0000 7f6c850d3b80 20 client.14892 populate_metadata read hostname 'smithi002' 2022-05-13T23:15:29.841 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.795+0000 7f6c850d3b80 1 --2- 172.21.15.2:0/1912494364 >> [v2:172.21.15.115:6836/2940957881,v1:172.21.15.115:6837/2940957881] conn(0x55b025f851e0 0x55b025f8a4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:15:29.841 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.795+0000 7f6c850d3b80 4 client.14892 mds_command: new command op to 24599 tid=0[{"prefix": "get_command_descriptions"}] 2022-05-13T23:15:29.842 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.795+0000 7f6c850d3b80 1 -- 172.21.15.2:0/1912494364 --> [v2:172.21.15.115:6836/2940957881,v1:172.21.15.115:6837/2940957881] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55b025e93330 con 0x55b025f851e0 2022-05-13T23:15:29.842 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.796+0000 7f6c6effd700 1 --2- 172.21.15.2:0/1912494364 >> [v2:172.21.15.115:6836/2940957881,v1:172.21.15.115:6837/2940957881] conn(0x55b025f851e0 0x55b025f8a4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:15:29.842 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.797+0000 7f6c4e7fc700 10 client.14892 ms_handle_connect on v2:172.21.15.115:6836/2940957881 2022-05-13T23:15:29.842 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.797+0000 7f6c6effd700 1 --2- 172.21.15.2:0/1912494364 >> [v2:172.21.15.115:6836/2940957881,v1:172.21.15.115:6837/2940957881] conn(0x55b025f851e0 0x55b025f8a4e0 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=73ba484846d7bd69 server_cookie=2503ff32bebf455d in_seq=0 out_seq=0 2022-05-13T23:15:29.843 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.797+0000 7f6c4e7fc700 1 -- 172.21.15.2:0/1912494364 <== mds.1 v2:172.21.15.115:6836/2940957881 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17090 (crc 0 0 0) 0x55b025e93330 con 0x55b025f851e0 2022-05-13T23:15:29.845 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.797+0000 7f6c4e7fc700 10 client.14892 handle_command_reply: tid=0 2022-05-13T23:15:29.845 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.798+0000 7f6c850d3b80 1 client.14892 shutdown 2022-05-13T23:15:29.845 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.798+0000 7f6c850d3b80 1 -- 172.21.15.2:0/1912494364 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55b025f70cf0 msgr2=0x55b025f7a470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:15:29.845 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.798+0000 7f6c850d3b80 1 --2- 172.21.15.2:0/1912494364 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55b025f70cf0 0x55b025f7a470 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f6c60009080 tx=0x7f6c6000b040 comp rx=0 tx=0).stop 2022-05-13T23:15:29.846 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.798+0000 7f6c850d3b80 1 -- 172.21.15.2:0/1912494364 shutdown_connections 2022-05-13T23:15:29.846 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.798+0000 7f6c850d3b80 1 --2- 172.21.15.2:0/1912494364 >> [v2:172.21.15.115:6836/2940957881,v1:172.21.15.115:6837/2940957881] conn(0x55b025f851e0 0x55b025f8a4e0 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:29.846 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.798+0000 7f6c850d3b80 1 --2- 172.21.15.2:0/1912494364 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55b025f725a0 0x55b025f81a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:29.846 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.798+0000 7f6c850d3b80 1 --2- 172.21.15.2:0/1912494364 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55b025f70cf0 0x55b025f7a470 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:29.846 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.798+0000 7f6c4e7fc700 0 client.14892 ms_handle_reset on v2:172.21.15.115:6836/2940957881 2022-05-13T23:15:29.847 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.798+0000 7f6c850d3b80 1 --2- 172.21.15.2:0/1912494364 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55b025f71640 0x55b025f7a9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:29.847 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.798+0000 7f6c850d3b80 1 -- 172.21.15.2:0/1912494364 >> 172.21.15.2:0/1912494364 conn(0x55b025f6bb10 msgr2=0x55b025f6ad80 unknown :-1 s=STATE_NONE l=0).mark_down 2022-05-13T23:15:29.847 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.798+0000 7f6c850d3b80 1 -- 172.21.15.2:0/1912494364 shutdown_connections 2022-05-13T23:15:29.847 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.798+0000 7f6c850d3b80 1 -- 172.21.15.2:0/1912494364 wait complete. 2022-05-13T23:15:29.848 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.798+0000 7f6c850d3b80 20 client.14892 trim_cache size 0 max 16384 2022-05-13T23:15:29.855 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.816+0000 7f6c850d3b80 1 Processor -- start 2022-05-13T23:15:29.855 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.816+0000 7f6c850d3b80 1 -- start start 2022-05-13T23:15:29.855 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.816+0000 7f6c850d3b80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55b025f70cf0 0x55b025f72970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:15:29.856 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.816+0000 7f6c850d3b80 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55b025f71640 0x55b025f8e500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:15:29.856 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.816+0000 7f6c850d3b80 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55b025f725a0 0x55b025f6a300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:15:29.856 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.816+0000 7f6c850d3b80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b025f83430 con 0x55b025f70cf0 2022-05-13T23:15:29.856 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.816+0000 7f6c850d3b80 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b025e87be0 con 0x55b025f71640 2022-05-13T23:15:29.856 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.816+0000 7f6c850d3b80 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b025d64b60 con 0x55b025f725a0 2022-05-13T23:15:29.857 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.817+0000 7f6c6f7fe700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55b025f70cf0 0x55b025f72970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:15:29.857 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.817+0000 7f6c6f7fe700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55b025f70cf0 0x55b025f72970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:51900/0 (socket says 172.21.15.2:51900) 2022-05-13T23:15:29.857 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.817+0000 7f6c6f7fe700 1 -- 172.21.15.2:0/1706705361 learned_addr learned my addr 172.21.15.2:0/1706705361 (peer_addr_for_me v2:172.21.15.2:0/0) 2022-05-13T23:15:29.857 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.817+0000 7f6c6ffff700 1 --2- 172.21.15.2:0/1706705361 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55b025f725a0 0x55b025f6a300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:15:29.858 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.817+0000 7f6c6effd700 1 --2- 172.21.15.2:0/1706705361 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55b025f71640 0x55b025f8e500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:15:29.858 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.817+0000 7f6c6f7fe700 1 -- 172.21.15.2:0/1706705361 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55b025f725a0 msgr2=0x55b025f6a300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-05-13T23:15:29.858 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.817+0000 7f6c6f7fe700 1 --2- 172.21.15.2:0/1706705361 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55b025f725a0 0x55b025f6a300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:29.858 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.817+0000 7f6c6f7fe700 1 -- 172.21.15.2:0/1706705361 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55b025f71640 msgr2=0x55b025f8e500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-05-13T23:15:29.858 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.817+0000 7f6c6f7fe700 1 --2- 172.21.15.2:0/1706705361 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55b025f71640 0x55b025f8e500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:29.859 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.817+0000 7f6c6f7fe700 1 -- 172.21.15.2:0/1706705361 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b025d28f80 con 0x55b025f70cf0 2022-05-13T23:15:29.859 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.817+0000 7f6c6ffff700 1 --2- 172.21.15.2:0/1706705361 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55b025f725a0 0x55b025f6a300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2022-05-13T23:15:29.859 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.817+0000 7f6c6f7fe700 1 --2- 172.21.15.2:0/1706705361 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55b025f70cf0 0x55b025f72970 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f6c60009720 tx=0x7f6c6000b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eff7a57697454009 server_cookie=0 in_seq=0 out_seq=0 2022-05-13T23:15:29.859 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.818+0000 7f6c559ff700 1 -- 172.21.15.2:0/1706705361 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c6000e070 con 0x55b025f70cf0 2022-05-13T23:15:29.860 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.818+0000 7f6c559ff700 1 -- 172.21.15.2:0/1706705361 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1087+0+0 (secure 0 0 0) 0x7f6c6000f040 con 0x55b025f70cf0 2022-05-13T23:15:29.860 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.818+0000 7f6c559ff700 1 -- 172.21.15.2:0/1706705361 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c60015920 con 0x55b025f70cf0 2022-05-13T23:15:29.860 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.818+0000 7f6c850d3b80 1 -- 172.21.15.2:0/1706705361 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55b025f70cf0 msgr2=0x55b025f72970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:15:29.861 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.818+0000 7f6c850d3b80 1 --2- 172.21.15.2:0/1706705361 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55b025f70cf0 0x55b025f72970 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f6c60009720 tx=0x7f6c6000b040 comp rx=0 tx=0).stop 2022-05-13T23:15:29.861 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.818+0000 7f6c850d3b80 1 -- 172.21.15.2:0/1706705361 shutdown_connections 2022-05-13T23:15:29.861 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.818+0000 7f6c850d3b80 1 --2- 172.21.15.2:0/1706705361 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55b025f725a0 0x55b025f6a300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:29.861 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.818+0000 7f6c850d3b80 1 --2- 172.21.15.2:0/1706705361 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55b025f70cf0 0x55b025f72970 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:29.862 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.818+0000 7f6c850d3b80 1 --2- 172.21.15.2:0/1706705361 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55b025f71640 0x55b025f8e500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:29.862 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.818+0000 7f6c850d3b80 1 -- 172.21.15.2:0/1706705361 >> 172.21.15.2:0/1706705361 conn(0x55b025f6bb10 msgr2=0x55b025f6ad80 unknown :-1 s=STATE_NONE l=0).mark_down 2022-05-13T23:15:29.862 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.818+0000 7f6c850d3b80 1 -- 172.21.15.2:0/1706705361 shutdown_connections 2022-05-13T23:15:29.862 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.818+0000 7f6c850d3b80 1 -- 172.21.15.2:0/1706705361 wait complete. 2022-05-13T23:15:29.862 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.819+0000 7f6c850d3b80 1 -- start start 2022-05-13T23:15:29.863 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.819+0000 7f6c850d3b80 1 Processor -- start 2022-05-13T23:15:29.863 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.819+0000 7f6c850d3b80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55b025f70cf0 0x55b025f7e980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:15:29.863 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.819+0000 7f6c850d3b80 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55b025f71640 0x55b025f7eec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:15:29.863 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.819+0000 7f6c850d3b80 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55b025f725a0 0x55b025fd7200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:15:29.864 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.820+0000 7f6c850d3b80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b025f7f6d0 con 0x55b025f70cf0 2022-05-13T23:15:29.864 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.820+0000 7f6c850d3b80 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b025f7f840 con 0x55b025f725a0 2022-05-13T23:15:29.864 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.820+0000 7f6c850d3b80 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b025fd77c0 con 0x55b025f71640 2022-05-13T23:15:29.864 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.820+0000 7f6c850d3b80 10 client.0 _ll_register_callbacks cb 0x55b025f32de0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2022-05-13T23:15:29.865 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.820+0000 7f6c6f7fe700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55b025f70cf0 0x55b025f7e980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:15:29.865 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.820+0000 7f6c6f7fe700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55b025f70cf0 0x55b025f7e980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:51906/0 (socket says 172.21.15.2:51906) 2022-05-13T23:15:29.865 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.820+0000 7f6c6f7fe700 1 -- 172.21.15.2:0/837406088 learned_addr learned my addr 172.21.15.2:0/837406088 (peer_addr_for_me v2:172.21.15.2:0/0) 2022-05-13T23:15:29.866 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.820+0000 7f6c6ffff700 1 --2- 172.21.15.2:0/837406088 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55b025f725a0 0x55b025fd7200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:15:29.866 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.820+0000 7f6c6effd700 1 --2- 172.21.15.2:0/837406088 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55b025f71640 0x55b025f7eec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:15:29.866 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.820+0000 7f6c6f7fe700 1 -- 172.21.15.2:0/837406088 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55b025f71640 msgr2=0x55b025f7eec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-05-13T23:15:29.866 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.820+0000 7f6c6f7fe700 1 --2- 172.21.15.2:0/837406088 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55b025f71640 0x55b025f7eec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:29.867 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.820+0000 7f6c6f7fe700 1 -- 172.21.15.2:0/837406088 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55b025f725a0 msgr2=0x55b025fd7200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-05-13T23:15:29.867 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.820+0000 7f6c6f7fe700 1 --2- 172.21.15.2:0/837406088 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55b025f725a0 0x55b025fd7200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:29.867 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.820+0000 7f6c6f7fe700 1 -- 172.21.15.2:0/837406088 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55b025d00fe0 con 0x55b025f70cf0 2022-05-13T23:15:29.867 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.821+0000 7f6c6f7fe700 1 --2- 172.21.15.2:0/837406088 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55b025f70cf0 0x55b025f7e980 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f6c6000b010 tx=0x7f6c6000b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1f5b54014c636ec0 server_cookie=0 in_seq=0 out_seq=0 2022-05-13T23:15:29.867 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.821+0000 7f6c4e7fc700 10 client.0 ms_handle_connect on v2:172.21.15.2:3300/0 2022-05-13T23:15:29.868 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.821+0000 7f6c4e7fc700 1 -- 172.21.15.2:0/837406088 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c6000e040 con 0x55b025f70cf0 2022-05-13T23:15:29.868 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.821+0000 7f6c4e7fc700 1 -- 172.21.15.2:0/837406088 <== mon.0 v2:172.21.15.2:3300/0 2 ==== osd_map(85..85 src has 1..85) v4 ==== 8120+0+0 (secure 0 0 0) 0x7f6c60015d50 con 0x55b025f70cf0 2022-05-13T23:15:29.868 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.821+0000 7f6c850d3b80 1 -- 172.21.15.2:0/837406088 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55b025f415f0 con 0x55b025f70cf0 2022-05-13T23:15:29.868 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.821+0000 7f6c4e7fc700 1 -- 172.21.15.2:0/837406088 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6c600168f0 con 0x55b025f70cf0 2022-05-13T23:15:29.868 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.821+0000 7f6c850d3b80 10 client.14898 fetch_fsmap learned FSMap version 26 2022-05-13T23:15:29.869 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.821+0000 7f6c850d3b80 1 -- 172.21.15.2:0/837406088 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) v3 -- 0x55b025d02b10 con 0x55b025f70cf0 2022-05-13T23:15:29.869 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.822+0000 7f6c4e7fc700 1 -- 172.21.15.2:0/837406088 <== mon.0 v2:172.21.15.2:3300/0 4 ==== config(26 keys) v1 ==== 1087+0+0 (secure 0 0 0) 0x55b025f415f0 con 0x55b025f70cf0 2022-05-13T23:15:29.869 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.822+0000 7f6c4e7fc700 1 -- 172.21.15.2:0/837406088 <== mon.0 v2:172.21.15.2:3300/0 5 ==== fsmap(e 26) v1 ==== 7448+0+0 (secure 0 0 0) 0x7f6c600339d0 con 0x55b025f70cf0 2022-05-13T23:15:29.869 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.822+0000 7f6c850d3b80 10 client.14898 fetch_fsmap finished waiting for FSMap version 26 2022-05-13T23:15:29.869 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.822+0000 7f6c850d3b80 10 client.14898 resolve_mds: resolved name 'i' to daemon mds.i 2022-05-13T23:15:29.870 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.822+0000 7f6c850d3b80 20 client.14898 populate_metadata read hostname 'smithi002' 2022-05-13T23:15:29.870 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.822+0000 7f6c850d3b80 1 --2- 172.21.15.2:0/837406088 >> [v2:172.21.15.115:6836/2940957881,v1:172.21.15.115:6837/2940957881] conn(0x55b025fdbe70 0x55b025fd8820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:15:29.870 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.822+0000 7f6c850d3b80 4 client.14898 mds_command: new command op to 24599 tid=0[{"prefix": "status"}] 2022-05-13T23:15:29.870 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.822+0000 7f6c850d3b80 1 -- 172.21.15.2:0/837406088 --> [v2:172.21.15.115:6836/2940957881,v1:172.21.15.115:6837/2940957881] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55b025e64790 con 0x55b025fdbe70 2022-05-13T23:15:29.871 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.822+0000 7f6c6effd700 1 --2- 172.21.15.2:0/837406088 >> [v2:172.21.15.115:6836/2940957881,v1:172.21.15.115:6837/2940957881] conn(0x55b025fdbe70 0x55b025fd8820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:15:29.871 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.823+0000 7f6c4e7fc700 10 client.14898 ms_handle_connect on v2:172.21.15.115:6836/2940957881 2022-05-13T23:15:29.871 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.823+0000 7f6c6effd700 1 --2- 172.21.15.2:0/837406088 >> [v2:172.21.15.115:6836/2940957881,v1:172.21.15.115:6837/2940957881] conn(0x55b025fdbe70 0x55b025fd8820 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=ca314aebab998a76 server_cookie=69a289be3579bf85 in_seq=0 out_seq=0 2022-05-13T23:15:29.871 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.823+0000 7f6c4e7fc700 1 -- 172.21.15.2:0/837406088 <== mds.1 v2:172.21.15.115:6836/2940957881 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+332 (crc 0 0 0) 0x55b025e64790 con 0x55b025fdbe70 2022-05-13T23:15:29.871 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.823+0000 7f6c4e7fc700 10 client.14898 handle_command_reply: tid=0 2022-05-13T23:15:29.872 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.823+0000 7f6c850d3b80 1 client.14898 shutdown 2022-05-13T23:15:29.872 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.823+0000 7f6c850d3b80 1 -- 172.21.15.2:0/837406088 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55b025f70cf0 msgr2=0x55b025f7e980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:15:29.872 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.823+0000 7f6c850d3b80 1 --2- 172.21.15.2:0/837406088 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55b025f70cf0 0x55b025f7e980 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f6c6000b010 tx=0x7f6c6000b040 comp rx=0 tx=0).stop 2022-05-13T23:15:29.872 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.823+0000 7f6c850d3b80 1 -- 172.21.15.2:0/837406088 shutdown_connections 2022-05-13T23:15:29.872 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.823+0000 7f6c850d3b80 1 --2- 172.21.15.2:0/837406088 >> [v2:172.21.15.115:6836/2940957881,v1:172.21.15.115:6837/2940957881] conn(0x55b025fdbe70 0x55b025fd8820 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:29.873 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.823+0000 7f6c850d3b80 1 --2- 172.21.15.2:0/837406088 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55b025f725a0 0x55b025fd7200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:29.873 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.823+0000 7f6c850d3b80 1 --2- 172.21.15.2:0/837406088 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55b025f70cf0 0x55b025f7e980 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:29.873 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.823+0000 7f6c4e7fc700 0 client.14898 ms_handle_reset on v2:172.21.15.115:6836/2940957881 2022-05-13T23:15:29.874 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.823+0000 7f6c850d3b80 1 --2- 172.21.15.2:0/837406088 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55b025f71640 0x55b025f7eec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:29.874 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.823+0000 7f6c850d3b80 1 -- 172.21.15.2:0/837406088 >> 172.21.15.2:0/837406088 conn(0x55b025f6bb10 msgr2=0x55b025f6a300 unknown :-1 s=STATE_NONE l=0).mark_down 2022-05-13T23:15:29.874 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.824+0000 7f6c850d3b80 1 -- 172.21.15.2:0/837406088 shutdown_connections 2022-05-13T23:15:29.874 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.824+0000 7f6c850d3b80 1 -- 172.21.15.2:0/837406088 wait complete. 2022-05-13T23:15:29.875 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.824+0000 7f6c850d3b80 20 client.14898 trim_cache size 0 max 16384 2022-05-13T23:15:29.876 INFO:teuthology.orchestra.run.smithi002.stdout:{ 2022-05-13T23:15:29.876 INFO:teuthology.orchestra.run.smithi002.stdout: "cluster_fsid": "16cf5f88-d311-11ec-be40-001a4aab830c", 2022-05-13T23:15:29.876 INFO:teuthology.orchestra.run.smithi002.stdout: "whoami": 1, 2022-05-13T23:15:29.876 INFO:teuthology.orchestra.run.smithi002.stdout: "id": 24599, 2022-05-13T23:15:29.877 INFO:teuthology.orchestra.run.smithi002.stdout: "want_state": "up:active", 2022-05-13T23:15:29.877 INFO:teuthology.orchestra.run.smithi002.stdout: "state": "up:active", 2022-05-13T23:15:29.877 INFO:teuthology.orchestra.run.smithi002.stdout: "fs_name": "cephfs", 2022-05-13T23:15:29.877 INFO:teuthology.orchestra.run.smithi002.stdout: "rank_uptime": 70.542355323999999, 2022-05-13T23:15:29.878 INFO:teuthology.orchestra.run.smithi002.stdout: "mdsmap_epoch": 26, 2022-05-13T23:15:29.878 INFO:teuthology.orchestra.run.smithi002.stdout: "osdmap_epoch": 85, 2022-05-13T23:15:29.878 INFO:teuthology.orchestra.run.smithi002.stdout: "osdmap_epoch_barrier": 83, 2022-05-13T23:15:29.878 INFO:teuthology.orchestra.run.smithi002.stdout: "uptime": 81.972510982000003 2022-05-13T23:15:29.879 INFO:teuthology.orchestra.run.smithi002.stdout:} 2022-05-13T23:15:29.880 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.824+0000 7f6c74b9d700 1 -- 172.21.15.2:0/3425172745 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f6c58096d60 msgr2=0x7f6c58099210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:15:29.881 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.824+0000 7f6c74b9d700 1 --2- 172.21.15.2:0/3425172745 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f6c58096d60 0x7f6c58099210 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f6c68009ad0 tx=0x7f6c68004c50 comp rx=0 tx=0).stop 2022-05-13T23:15:29.881 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.824+0000 7f6c74b9d700 1 -- 172.21.15.2:0/3425172745 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6c70070370 msgr2=0x7f6c7010e830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:15:29.881 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.824+0000 7f6c74b9d700 1 --2- 172.21.15.2:0/3425172745 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6c70070370 0x7f6c7010e830 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f6c6400e970 tx=0x7f6c6400b040 comp rx=0 tx=0).stop 2022-05-13T23:15:29.882 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.824+0000 7f6c74b9d700 1 -- 172.21.15.2:0/3425172745 shutdown_connections 2022-05-13T23:15:29.882 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.824+0000 7f6c74b9d700 1 --2- 172.21.15.2:0/3425172745 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f6c70103400 0x7f6c701158c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:29.883 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.824+0000 7f6c74b9d700 1 --2- 172.21.15.2:0/3425172745 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6c7006edb0 0x7f6c7010e2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:29.883 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.824+0000 7f6c74b9d700 1 --2- 172.21.15.2:0/3425172745 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f6c58096d60 0x7f6c58099210 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:29.884 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.824+0000 7f6c74b9d700 1 --2- 172.21.15.2:0/3425172745 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6c70070370 0x7f6c7010e830 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:29.884 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.824+0000 7f6c74b9d700 1 -- 172.21.15.2:0/3425172745 >> 172.21.15.2:0/3425172745 conn(0x7f6c700f6c60 msgr2=0x7f6c700f9040 unknown :-1 s=STATE_NONE l=0).mark_down 2022-05-13T23:15:29.885 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.825+0000 7f6c74b9d700 1 -- 172.21.15.2:0/3425172745 shutdown_connections 2022-05-13T23:15:29.885 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:29.825+0000 7f6c74b9d700 1 -- 172.21.15.2:0/3425172745 wait complete. 2022-05-13T23:15:29.886 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.l status 2022-05-13T23:15:30.107 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:30.105+0000 7f53a499d700 1 -- 172.21.15.2:0/1287687604 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f53a01009a0 msgr2=0x7f53a0102d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:15:30.107 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:30.105+0000 7f53a499d700 1 --2- 172.21.15.2:0/1287687604 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f53a01009a0 0x7f53a0102d80 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f5394004870 tx=0x7f5394009cb0 comp rx=0 tx=0).stop 2022-05-13T23:15:30.107 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:30.105+0000 7f53a499d700 1 -- 172.21.15.2:0/1287687604 shutdown_connections 2022-05-13T23:15:30.108 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:30.105+0000 7f53a499d700 1 --2- 172.21.15.2:0/1287687604 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f53a0103310 0x7f53a0105790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:30.108 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:30.105+0000 7f53a499d700 1 --2- 172.21.15.2:0/1287687604 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f53a0064710 0x7f53a0100460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:30.108 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:30.105+0000 7f53a499d700 1 --2- 172.21.15.2:0/1287687604 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f53a01009a0 0x7f53a0102d80 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:30.108 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:30.105+0000 7f53a499d700 1 -- 172.21.15.2:0/1287687604 >> 172.21.15.2:0/1287687604 conn(0x7f53a00f5b30 msgr2=0x7f53a00f7fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-05-13T23:15:30.108 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:30.105+0000 7f53a499d700 1 -- 172.21.15.2:0/1287687604 shutdown_connections 2022-05-13T23:15:30.109 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:30.105+0000 7f53a499d700 1 -- 172.21.15.2:0/1287687604 wait complete. 2022-05-13T23:15:30.111 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:30.106+0000 7f53a499d700 1 Processor -- start 2022-05-13T23:15:30.111 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:30.106+0000 7f53a499d700 1 -- start start 2022-05-13T23:15:30.111 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:30.106+0000 7f53a499d700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f53a0064710 0x7f53a010e540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:15:30.112 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:30.107+0000 7f53a499d700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f53a01009a0 0x7f53a010ea80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:15:30.112 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:30.107+0000 7f53a499d700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f53a0103310 0x7f53a0115b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:15:30.112 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:30.107+0000 7f53a499d700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53a0116050 con 0x7f53a0103310 2022-05-13T23:15:30.112 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:30.107+0000 7f53a499d700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53a01161c0 con 0x7f53a0064710 2022-05-13T23:15:30.113 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:30.107+0000 7f53a499d700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53a0116380 con 0x7f53a01009a0 2022-05-13T23:15:30.113 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:30.107+0000 7f539ffff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f53a0103310 0x7f53a0115b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:15:30.113 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:30.107+0000 7f539f7fe700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f53a0064710 0x7f53a010e540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:15:30.113 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:30.107+0000 7f539ffff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f53a0103310 0x7f53a0115b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:51912/0 (socket says 172.21.15.2:51912) 2022-05-13T23:15:30.113 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:30.107+0000 7f539ffff700 1 -- 172.21.15.2:0/1374721098 learned_addr learned my addr 172.21.15.2:0/1374721098 (peer_addr_for_me v2:172.21.15.2:0/0) 2022-05-13T23:15:30.114 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:30.107+0000 7f539f7fe700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f53a0064710 0x7f53a010e540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.2:52246/0 (socket says 172.21.15.2:52246) 2022-05-13T23:15:30.114 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:30.107+0000 7f539effd700 1 --2- 172.21.15.2:0/1374721098 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f53a01009a0 0x7f53a010ea80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:15:30.114 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:30.107+0000 7f539f7fe700 1 -- 172.21.15.2:0/1374721098 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f53a01009a0 msgr2=0x7f53a010ea80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:15:30.114 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:30.107+0000 7f539f7fe700 1 --2- 172.21.15.2:0/1374721098 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f53a01009a0 0x7f53a010ea80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:30.114 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:30.107+0000 7f539f7fe700 1 -- 172.21.15.2:0/1374721098 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f53a0103310 msgr2=0x7f53a0115b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:15:30.115 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:30.107+0000 7f539f7fe700 1 --2- 172.21.15.2:0/1374721098 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f53a0103310 0x7f53a0115b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:30.115 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:30.107+0000 7f539f7fe700 1 -- 172.21.15.2:0/1374721098 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5394005040 con 0x7f53a0064710 2022-05-13T23:15:30.115 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:30.107+0000 7f539ffff700 1 --2- 172.21.15.2:0/1374721098 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f53a0103310 0x7f53a0115b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-05-13T23:15:30.115 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:30.108+0000 7f539f7fe700 1 --2- 172.21.15.2:0/1374721098 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f53a0064710 0x7f53a010e540 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f539000e8c0 tx=0x7f539000b040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-05-13T23:15:30.116 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:30.108+0000 7f5387fff700 1 -- 172.21.15.2:0/1374721098 <== mon.1 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5390007b40 con 0x7f53a0064710 2022-05-13T23:15:30.116 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:30.108+0000 7f53a499d700 1 -- 172.21.15.2:0/1374721098 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f53a0116600 con 0x7f53a0064710 2022-05-13T23:15:30.116 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:30.108+0000 7f5387fff700 1 -- 172.21.15.2:0/1374721098 <== mon.1 v2:172.21.15.47:3300/0 2 ==== config(26 keys) v1 ==== 1087+0+0 (secure 0 0 0) 0x7f539000f040 con 0x7f53a0064710 2022-05-13T23:15:30.116 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:30.108+0000 7f5387fff700 1 -- 172.21.15.2:0/1374721098 <== mon.1 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5390014950 con 0x7f53a0064710 2022-05-13T23:15:30.116 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:30.108+0000 7f53a499d700 1 -- 172.21.15.2:0/1374721098 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f53a0116b70 con 0x7f53a0064710 2022-05-13T23:15:30.122 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:30.111+0000 7f5387fff700 1 -- 172.21.15.2:0/1374721098 <== mon.1 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 16) v1 ==== 125136+0+0 (secure 0 0 0) 0x7f539001b030 con 0x7f53a0064710 2022-05-13T23:15:30.122 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:30.112+0000 7f53b4ed4b80 1 Processor -- start 2022-05-13T23:15:30.122 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:30.113+0000 7f53b4ed4b80 1 -- start start 2022-05-13T23:15:30.123 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:30.113+0000 7f5387fff700 1 --2- 172.21.15.2:0/1374721098 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f5388096ce0 0x7f5388099190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:15:30.123 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:30.113+0000 7f5387fff700 1 -- 172.21.15.2:0/1374721098 <== mon.1 v2:172.21.15.47:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8120+0+0 (secure 0 0 0) 0x7f53900bea00 con 0x7f53a0064710 2022-05-13T23:15:30.123 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:30.113+0000 7f53b4ed4b80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5566ced05cf0 0x5566ced06100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:15:30.123 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:30.113+0000 7f53b4ed4b80 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5566ced06640 0x5566ced06f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:15:30.124 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:30.113+0000 7f53b4ed4b80 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x5566ced075a0 0x5566ced079f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:15:30.124 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:30.113+0000 7f539ffff700 1 --2- 172.21.15.2:0/1374721098 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f5388096ce0 0x7f5388099190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:15:30.124 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:30.113+0000 7f53b4ed4b80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x5566cec003a0 con 0x5566ced05cf0 2022-05-13T23:15:30.124 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:30.113+0000 7f53b4ed4b80 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x5566cebd5400 con 0x5566ced075a0 2022-05-13T23:15:30.124 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:30.113+0000 7f53b4ed4b80 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x5566cec596c0 con 0x5566ced06640 2022-05-13T23:15:30.125 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:30.114+0000 7f539effd700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5566ced06640 0x5566ced06f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:15:30.125 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:30.114+0000 7f539effd700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5566ced06640 0x5566ced06f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.2:56548/0 (socket says 172.21.15.2:56548) 2022-05-13T23:15:30.125 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:30.114+0000 7f539effd700 1 -- 172.21.15.2:0/2240629307 learned_addr learned my addr 172.21.15.2:0/2240629307 (peer_addr_for_me v2:172.21.15.2:0/0) 2022-05-13T23:15:30.125 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:30.114+0000 7f539f7fe700 1 --2- 172.21.15.2:0/2240629307 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5566ced05cf0 0x5566ced06100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:15:30.126 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:30.114+0000 7f539ffff700 1 --2- 172.21.15.2:0/2240629307 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x5566ced075a0 0x5566ced079f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:15:30.126 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:30.114+0000 7f539effd700 1 -- 172.21.15.2:0/2240629307 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x5566ced075a0 msgr2=0x5566ced079f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-05-13T23:15:30.126 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:30.114+0000 7f539effd700 1 --2- 172.21.15.2:0/2240629307 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x5566ced075a0 0x5566ced079f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:30.126 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:30.114+0000 7f539effd700 1 -- 172.21.15.2:0/2240629307 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5566ced05cf0 msgr2=0x5566ced06100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-05-13T23:15:30.126 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:30.114+0000 7f539effd700 1 --2- 172.21.15.2:0/2240629307 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5566ced05cf0 0x5566ced06100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:30.127 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:30.114+0000 7f539effd700 1 -- 172.21.15.2:0/2240629307 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5566ceaeb360 con 0x5566ced06640 2022-05-13T23:15:30.127 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:30.114+0000 7f539ffff700 1 --2- 172.21.15.2:0/1374721098 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f5388096ce0 0x7f5388099190 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f53980079e0 tx=0x7f539800b040 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-05-13T23:15:30.127 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:30.115+0000 7f539effd700 1 --2- 172.21.15.2:0/2240629307 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5566ced06640 0x5566ced06f80 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f5394004840 tx=0x7f5394009cb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=75cfca3ce28f6db9 server_cookie=0 in_seq=0 out_seq=0 2022-05-13T23:15:30.127 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:30.115+0000 7f53851fe700 1 -- 172.21.15.2:0/2240629307 <== mon.2 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5394023070 con 0x5566ced06640 2022-05-13T23:15:30.127 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:30.115+0000 7f53851fe700 1 -- 172.21.15.2:0/2240629307 <== mon.2 v2:172.21.15.115:3300/0 2 ==== config(26 keys) v1 ==== 1087+0+0 (secure 0 0 0) 0x7f5394000bc0 con 0x5566ced06640 2022-05-13T23:15:30.128 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:30.115+0000 7f53851fe700 1 -- 172.21.15.2:0/2240629307 <== mon.2 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f539400e040 con 0x5566ced06640 2022-05-13T23:15:30.128 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:30.115+0000 7f53b4ed4b80 1 -- 172.21.15.2:0/2240629307 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5566ced06640 msgr2=0x5566ced06f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:15:30.128 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:30.115+0000 7f53b4ed4b80 1 --2- 172.21.15.2:0/2240629307 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5566ced06640 0x5566ced06f80 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f5394004840 tx=0x7f5394009cb0 comp rx=0 tx=0).stop 2022-05-13T23:15:30.128 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:30.116+0000 7f53b4ed4b80 1 -- 172.21.15.2:0/2240629307 shutdown_connections 2022-05-13T23:15:30.129 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:30.116+0000 7f53b4ed4b80 1 --2- 172.21.15.2:0/2240629307 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x5566ced075a0 0x5566ced079f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:30.129 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:30.116+0000 7f53b4ed4b80 1 --2- 172.21.15.2:0/2240629307 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5566ced05cf0 0x5566ced06100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:30.129 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:30.116+0000 7f53b4ed4b80 1 --2- 172.21.15.2:0/2240629307 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5566ced06640 0x5566ced06f80 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:30.129 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:30.116+0000 7f53b4ed4b80 1 -- 172.21.15.2:0/2240629307 >> 172.21.15.2:0/2240629307 conn(0x5566ced00b10 msgr2=0x5566ced00f00 unknown :-1 s=STATE_NONE l=0).mark_down 2022-05-13T23:15:30.129 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:30.116+0000 7f53b4ed4b80 1 -- 172.21.15.2:0/2240629307 shutdown_connections 2022-05-13T23:15:30.130 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:30.116+0000 7f53b4ed4b80 1 -- 172.21.15.2:0/2240629307 wait complete. 2022-05-13T23:15:30.130 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:30.116+0000 7f53b4ed4b80 1 -- start start 2022-05-13T23:15:30.130 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:30.117+0000 7f53b4ed4b80 1 Processor -- start 2022-05-13T23:15:30.130 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:30.117+0000 7f53b4ed4b80 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5566ced05cf0 0x5566ced0f530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:15:30.131 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:30.117+0000 7f53b4ed4b80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5566ced06640 0x5566ced0fa70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:15:30.131 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:30.117+0000 7f53b4ed4b80 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x5566ced075a0 0x5566ced16ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:15:30.131 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:30.117+0000 7f53b4ed4b80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x5566cec537a0 con 0x5566ced06640 2022-05-13T23:15:30.131 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:30.117+0000 7f53b4ed4b80 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x5566ceaed650 con 0x5566ced075a0 2022-05-13T23:15:30.131 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:30.117+0000 7f53b4ed4b80 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x5566ced17100 con 0x5566ced05cf0 2022-05-13T23:15:30.132 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:30.117+0000 7f53b4ed4b80 10 client.0 _ll_register_callbacks cb 0x5566cecc7de0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2022-05-13T23:15:30.132 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:30.117+0000 7f539effd700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5566ced06640 0x5566ced0fa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:15:30.132 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:30.118+0000 7f539f7fe700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5566ced05cf0 0x5566ced0f530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:15:30.132 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:30.118+0000 7f539effd700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5566ced06640 0x5566ced0fa70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:51932/0 (socket says 172.21.15.2:51932) 2022-05-13T23:15:30.133 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:30.118+0000 7f539effd700 1 -- 172.21.15.2:0/4035799948 learned_addr learned my addr 172.21.15.2:0/4035799948 (peer_addr_for_me v2:172.21.15.2:0/0) 2022-05-13T23:15:30.133 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:30.118+0000 7f539ffff700 1 --2- 172.21.15.2:0/4035799948 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x5566ced075a0 0x5566ced16ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:15:30.133 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:30.118+0000 7f539f7fe700 1 -- 172.21.15.2:0/4035799948 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x5566ced075a0 msgr2=0x5566ced16ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-05-13T23:15:30.133 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:30.118+0000 7f539f7fe700 1 --2- 172.21.15.2:0/4035799948 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x5566ced075a0 0x5566ced16ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:30.134 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:30.118+0000 7f539f7fe700 1 -- 172.21.15.2:0/4035799948 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5566ced06640 msgr2=0x5566ced0fa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-05-13T23:15:30.134 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:30.118+0000 7f539f7fe700 1 --2- 172.21.15.2:0/4035799948 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5566ced06640 0x5566ced0fa70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:30.134 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:30.118+0000 7f539f7fe700 1 -- 172.21.15.2:0/4035799948 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5394017040 con 0x5566ced05cf0 2022-05-13T23:15:30.134 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:30.118+0000 7f539effd700 1 --2- 172.21.15.2:0/4035799948 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5566ced06640 0x5566ced0fa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-05-13T23:15:30.134 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:30.118+0000 7f539f7fe700 1 --2- 172.21.15.2:0/4035799948 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5566ced05cf0 0x5566ced0f530 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f539008f900 tx=0x7f5390002040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7781c40772d4437f server_cookie=0 in_seq=0 out_seq=0 2022-05-13T23:15:30.135 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:30.118+0000 7f537e7fc700 10 client.0 ms_handle_connect on v2:172.21.15.115:3300/0 2022-05-13T23:15:30.135 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:30.118+0000 7f537e7fc700 1 -- 172.21.15.2:0/4035799948 <== mon.2 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f539008f9f0 con 0x5566ced05cf0 2022-05-13T23:15:30.135 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:30.118+0000 7f53b4ed4b80 1 -- 172.21.15.2:0/4035799948 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5566cec079a0 con 0x5566ced05cf0 2022-05-13T23:15:30.135 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:30.118+0000 7f537e7fc700 1 -- 172.21.15.2:0/4035799948 <== mon.2 v2:172.21.15.115:3300/0 2 ==== osd_map(85..85 src has 1..85) v4 ==== 8120+0+0 (secure 0 0 0) 0x7f5390096060 con 0x5566ced05cf0 2022-05-13T23:15:30.135 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:30.119+0000 7f537e7fc700 1 -- 172.21.15.2:0/4035799948 <== mon.2 v2:172.21.15.115:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7f539008fde0 con 0x5566ced05cf0 2022-05-13T23:15:30.136 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:30.119+0000 7f53b4ed4b80 10 client.25031 fetch_fsmap learned FSMap version 26 2022-05-13T23:15:30.136 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:30.119+0000 7f53b4ed4b80 1 -- 172.21.15.2:0/4035799948 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) v3 -- 0x5566cea95fe0 con 0x5566ced05cf0 2022-05-13T23:15:30.136 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:30.119+0000 7f537e7fc700 1 -- 172.21.15.2:0/4035799948 <== mon.2 v2:172.21.15.115:3300/0 4 ==== config(26 keys) v1 ==== 1087+0+0 (secure 0 0 0) 0x5566cec079a0 con 0x5566ced05cf0 2022-05-13T23:15:30.136 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:30.119+0000 7f537e7fc700 1 -- 172.21.15.2:0/4035799948 <== mon.2 v2:172.21.15.115:3300/0 5 ==== fsmap(e 26) v1 ==== 7448+0+0 (secure 0 0 0) 0x7f539008b760 con 0x5566ced05cf0 2022-05-13T23:15:30.137 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:30.119+0000 7f53b4ed4b80 10 client.25031 fetch_fsmap finished waiting for FSMap version 26 2022-05-13T23:15:30.137 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:30.119+0000 7f53b4ed4b80 10 client.25031 resolve_mds: resolved name 'l' to daemon mds.l 2022-05-13T23:15:30.137 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:30.119+0000 7f53b4ed4b80 20 client.25031 populate_metadata read hostname 'smithi002' 2022-05-13T23:15:30.137 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:30.119+0000 7f53b4ed4b80 1 --2- 172.21.15.2:0/4035799948 >> [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] conn(0x5566ced1a170 0x5566ced1f470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:15:30.137 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:30.119+0000 7f53b4ed4b80 4 client.25031 mds_command: new command op to 24611 tid=0[{"prefix": "get_command_descriptions"}] 2022-05-13T23:15:30.138 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:30.119+0000 7f53b4ed4b80 1 -- 172.21.15.2:0/4035799948 --> [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5566cec28330 con 0x5566ced1a170 2022-05-13T23:15:30.138 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:30.120+0000 7f539effd700 1 --2- 172.21.15.2:0/4035799948 >> [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] conn(0x5566ced1a170 0x5566ced1f470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:15:30.138 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:30.120+0000 7f539effd700 1 --2- 172.21.15.2:0/4035799948 >> [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] conn(0x5566ced1a170 0x5566ced1f470 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c5d8d7ef27311384 server_cookie=10cc99ce8dd97413 in_seq=0 out_seq=0 2022-05-13T23:15:30.138 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:30.120+0000 7f537e7fc700 10 client.25031 ms_handle_connect on v2:172.21.15.115:6838/4149542923 2022-05-13T23:15:30.140 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:30.121+0000 7f537e7fc700 1 -- 172.21.15.2:0/4035799948 <== mds.0 v2:172.21.15.115:6838/4149542923 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17090 (crc 0 0 0) 0x5566cec28330 con 0x5566ced1a170 2022-05-13T23:15:30.140 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:30.121+0000 7f537e7fc700 10 client.25031 handle_command_reply: tid=0 2022-05-13T23:15:30.140 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:30.121+0000 7f53b4ed4b80 1 client.25031 shutdown 2022-05-13T23:15:30.140 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:30.122+0000 7f53b4ed4b80 1 -- 172.21.15.2:0/4035799948 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5566ced05cf0 msgr2=0x5566ced0f530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:15:30.141 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:30.122+0000 7f53b4ed4b80 1 --2- 172.21.15.2:0/4035799948 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5566ced05cf0 0x5566ced0f530 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f539008f900 tx=0x7f5390002040 comp rx=0 tx=0).stop 2022-05-13T23:15:30.141 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:30.122+0000 7f53b4ed4b80 1 -- 172.21.15.2:0/4035799948 shutdown_connections 2022-05-13T23:15:30.141 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:30.122+0000 7f53b4ed4b80 1 --2- 172.21.15.2:0/4035799948 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x5566ced075a0 0x5566ced16ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:30.141 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:30.122+0000 7f53b4ed4b80 1 --2- 172.21.15.2:0/4035799948 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5566ced05cf0 0x5566ced0f530 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:30.142 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:30.122+0000 7f53b4ed4b80 1 --2- 172.21.15.2:0/4035799948 >> [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] conn(0x5566ced1a170 0x5566ced1f470 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:30.142 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:30.122+0000 7f53b4ed4b80 1 --2- 172.21.15.2:0/4035799948 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5566ced06640 0x5566ced0fa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:30.142 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:30.122+0000 7f53b4ed4b80 1 -- 172.21.15.2:0/4035799948 >> 172.21.15.2:0/4035799948 conn(0x5566ced00b10 msgr2=0x5566cecffd80 unknown :-1 s=STATE_NONE l=0).mark_down 2022-05-13T23:15:30.142 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:30.122+0000 7f537e7fc700 0 client.25031 ms_handle_reset on v2:172.21.15.115:6838/4149542923 2022-05-13T23:15:30.142 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:30.122+0000 7f53b4ed4b80 1 -- 172.21.15.2:0/4035799948 shutdown_connections 2022-05-13T23:15:30.143 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:30.122+0000 7f53b4ed4b80 1 -- 172.21.15.2:0/4035799948 wait complete. 2022-05-13T23:15:30.143 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:30.122+0000 7f53b4ed4b80 20 client.25031 trim_cache size 0 max 16384 2022-05-13T23:15:30.149 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:30.139+0000 7f53b4ed4b80 1 Processor -- start 2022-05-13T23:15:30.149 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:30.140+0000 7f53b4ed4b80 1 -- start start 2022-05-13T23:15:30.150 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:30.140+0000 7f53b4ed4b80 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x5566ced05cf0 0x5566ced02790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:15:30.150 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:30.140+0000 7f53b4ed4b80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5566ced06640 0x5566ced0ece0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:15:30.150 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:30.140+0000 7f53b4ed4b80 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5566ced075a0 0x5566ced0a1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:15:30.150 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:30.140+0000 7f53b4ed4b80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x5566cec079a0 con 0x5566ced06640 2022-05-13T23:15:30.151 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:30.140+0000 7f53b4ed4b80 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x5566ced0fe40 con 0x5566ced05cf0 2022-05-13T23:15:30.151 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:30.140+0000 7f53b4ed4b80 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x5566ced23b10 con 0x5566ced075a0 2022-05-13T23:15:30.151 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:30.140+0000 7f539effd700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5566ced06640 0x5566ced0ece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:15:30.151 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:30.140+0000 7f539f7fe700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x5566ced05cf0 0x5566ced02790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:15:30.151 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:30.140+0000 7f539ffff700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5566ced075a0 0x5566ced0a1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:15:30.152 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:30.140+0000 7f539f7fe700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x5566ced05cf0 0x5566ced02790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.2:52282/0 (socket says 172.21.15.2:52282) 2022-05-13T23:15:30.152 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:30.140+0000 7f539ffff700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5566ced075a0 0x5566ced0a1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.2:56572/0 (socket says 172.21.15.2:56572) 2022-05-13T23:15:30.152 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:30.140+0000 7f539f7fe700 1 -- 172.21.15.2:0/1576255350 learned_addr learned my addr 172.21.15.2:0/1576255350 (peer_addr_for_me v2:172.21.15.2:0/0) 2022-05-13T23:15:30.152 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:30.140+0000 7f539effd700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5566ced06640 0x5566ced0ece0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:51944/0 (socket says 172.21.15.2:51944) 2022-05-13T23:15:30.152 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:30.141+0000 7f539ffff700 1 -- 172.21.15.2:0/1576255350 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x5566ced05cf0 msgr2=0x5566ced02790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-05-13T23:15:30.153 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:30.141+0000 7f539ffff700 1 --2- 172.21.15.2:0/1576255350 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x5566ced05cf0 0x5566ced02790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:30.153 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:30.141+0000 7f539ffff700 1 -- 172.21.15.2:0/1576255350 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5566ced06640 msgr2=0x5566ced0ece0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-05-13T23:15:30.153 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:30.141+0000 7f539ffff700 1 --2- 172.21.15.2:0/1576255350 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5566ced06640 0x5566ced0ece0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:30.153 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:30.141+0000 7f539ffff700 1 -- 172.21.15.2:0/1576255350 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5566ceabdf80 con 0x5566ced075a0 2022-05-13T23:15:30.154 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:30.141+0000 7f539ffff700 1 --2- 172.21.15.2:0/1576255350 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5566ced075a0 0x5566ced0a1b0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f5398018640 tx=0x7f5398014000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c76fd93ae281591e server_cookie=0 in_seq=0 out_seq=0 2022-05-13T23:15:30.154 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:30.141+0000 7f53859ff700 1 -- 172.21.15.2:0/1576255350 <== mon.2 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f539802a070 con 0x5566ced075a0 2022-05-13T23:15:30.154 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:30.141+0000 7f53859ff700 1 -- 172.21.15.2:0/1576255350 <== mon.2 v2:172.21.15.115:3300/0 2 ==== config(26 keys) v1 ==== 1087+0+0 (secure 0 0 0) 0x7f539801e070 con 0x5566ced075a0 2022-05-13T23:15:30.154 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:30.141+0000 7f53859ff700 1 -- 172.21.15.2:0/1576255350 <== mon.2 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f539800e070 con 0x5566ced075a0 2022-05-13T23:15:30.154 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:30.142+0000 7f53b4ed4b80 1 -- 172.21.15.2:0/1576255350 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5566ced075a0 msgr2=0x5566ced0a1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:15:30.155 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:30.142+0000 7f53b4ed4b80 1 --2- 172.21.15.2:0/1576255350 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5566ced075a0 0x5566ced0a1b0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f5398018640 tx=0x7f5398014000 comp rx=0 tx=0).stop 2022-05-13T23:15:30.155 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:30.142+0000 7f53b4ed4b80 1 -- 172.21.15.2:0/1576255350 shutdown_connections 2022-05-13T23:15:30.155 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:30.142+0000 7f53b4ed4b80 1 --2- 172.21.15.2:0/1576255350 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5566ced075a0 0x5566ced0a1b0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:30.155 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:30.142+0000 7f53b4ed4b80 1 --2- 172.21.15.2:0/1576255350 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x5566ced05cf0 0x5566ced02790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:30.156 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:30.142+0000 7f53b4ed4b80 1 --2- 172.21.15.2:0/1576255350 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5566ced06640 0x5566ced0ece0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:30.156 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:30.142+0000 7f53b4ed4b80 1 -- 172.21.15.2:0/1576255350 >> 172.21.15.2:0/1576255350 conn(0x5566ced00b10 msgr2=0x5566cecff300 unknown :-1 s=STATE_NONE l=0).mark_down 2022-05-13T23:15:30.156 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:30.142+0000 7f53b4ed4b80 1 -- 172.21.15.2:0/1576255350 shutdown_connections 2022-05-13T23:15:30.156 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:30.142+0000 7f53b4ed4b80 1 -- 172.21.15.2:0/1576255350 wait complete. 2022-05-13T23:15:30.156 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:30.142+0000 7f53b4ed4b80 1 -- start start 2022-05-13T23:15:30.157 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:30.143+0000 7f53b4ed4b80 1 Processor -- start 2022-05-13T23:15:30.157 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:30.143+0000 7f53b4ed4b80 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x5566ced05cf0 0x5566ced13c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:15:30.157 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:30.143+0000 7f53b4ed4b80 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5566ced06640 0x5566ced14180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:15:30.158 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:30.143+0000 7f53b4ed4b80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5566ced075a0 0x5566ced6c450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:15:30.158 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:30.143+0000 7f53b4ed4b80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x5566ced6ca00 con 0x5566ced075a0 2022-05-13T23:15:30.158 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:30.143+0000 7f53b4ed4b80 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x5566ced6cb70 con 0x5566ced05cf0 2022-05-13T23:15:30.158 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:30.143+0000 7f53b4ed4b80 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x5566ced6cd30 con 0x5566ced06640 2022-05-13T23:15:30.159 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:30.143+0000 7f53b4ed4b80 10 client.0 _ll_register_callbacks cb 0x5566cecc7de0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2022-05-13T23:15:30.159 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:30.143+0000 7f539ffff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5566ced075a0 0x5566ced6c450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:15:30.159 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:30.143+0000 7f539effd700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5566ced06640 0x5566ced14180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:15:30.159 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:30.143+0000 7f539effd700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5566ced06640 0x5566ced14180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.2:56584/0 (socket says 172.21.15.2:56584) 2022-05-13T23:15:30.159 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:30.143+0000 7f539effd700 1 -- 172.21.15.2:0/567612222 learned_addr learned my addr 172.21.15.2:0/567612222 (peer_addr_for_me v2:172.21.15.2:0/0) 2022-05-13T23:15:30.160 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:30.143+0000 7f539f7fe700 1 --2- 172.21.15.2:0/567612222 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x5566ced05cf0 0x5566ced13c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:15:30.160 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:30.144+0000 7f539effd700 1 -- 172.21.15.2:0/567612222 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x5566ced05cf0 msgr2=0x5566ced13c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-05-13T23:15:30.160 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:30.144+0000 7f539effd700 1 --2- 172.21.15.2:0/567612222 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x5566ced05cf0 0x5566ced13c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:30.160 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:30.144+0000 7f539effd700 1 -- 172.21.15.2:0/567612222 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5566ced075a0 msgr2=0x5566ced6c450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-05-13T23:15:30.161 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:30.144+0000 7f539effd700 1 --2- 172.21.15.2:0/567612222 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5566ced075a0 0x5566ced6c450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:30.161 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:30.144+0000 7f539effd700 1 -- 172.21.15.2:0/567612222 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5398009680 con 0x5566ced06640 2022-05-13T23:15:30.161 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:30.144+0000 7f539effd700 1 --2- 172.21.15.2:0/567612222 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5566ced06640 0x5566ced14180 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f5394013490 tx=0x7f539402b040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5267e4a97c0c309f server_cookie=0 in_seq=0 out_seq=0 2022-05-13T23:15:30.161 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:30.144+0000 7f537e7fc700 10 client.0 ms_handle_connect on v2:172.21.15.115:3300/0 2022-05-13T23:15:30.162 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:30.144+0000 7f537e7fc700 1 -- 172.21.15.2:0/567612222 <== mon.2 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f539402e070 con 0x5566ced06640 2022-05-13T23:15:30.162 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:30.144+0000 7f53b4ed4b80 1 -- 172.21.15.2:0/567612222 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5566cecd65f0 con 0x5566ced06640 2022-05-13T23:15:30.162 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:30.144+0000 7f537e7fc700 1 -- 172.21.15.2:0/567612222 <== mon.2 v2:172.21.15.115:3300/0 2 ==== osd_map(85..85 src has 1..85) v4 ==== 8120+0+0 (secure 0 0 0) 0x7f5394007930 con 0x5566ced06640 2022-05-13T23:15:30.162 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:30.145+0000 7f537e7fc700 1 -- 172.21.15.2:0/567612222 <== mon.2 v2:172.21.15.115:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5394023040 con 0x5566ced06640 2022-05-13T23:15:30.162 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:30.145+0000 7f53b4ed4b80 10 client.25037 fetch_fsmap learned FSMap version 26 2022-05-13T23:15:30.163 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:30.145+0000 7f53b4ed4b80 1 -- 172.21.15.2:0/567612222 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) v3 -- 0x5566cea97b10 con 0x5566ced06640 2022-05-13T23:15:30.163 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:30.145+0000 7f537e7fc700 1 -- 172.21.15.2:0/567612222 <== mon.2 v2:172.21.15.115:3300/0 4 ==== config(26 keys) v1 ==== 1087+0+0 (secure 0 0 0) 0x5566cecd65f0 con 0x5566ced06640 2022-05-13T23:15:30.163 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:30.145+0000 7f537e7fc700 1 -- 172.21.15.2:0/567612222 <== mon.2 v2:172.21.15.115:3300/0 5 ==== fsmap(e 26) v1 ==== 7448+0+0 (secure 0 0 0) 0x7f539401e440 con 0x5566ced06640 2022-05-13T23:15:30.163 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:30.145+0000 7f53b4ed4b80 10 client.25037 fetch_fsmap finished waiting for FSMap version 26 2022-05-13T23:15:30.164 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:30.145+0000 7f53b4ed4b80 10 client.25037 resolve_mds: resolved name 'l' to daemon mds.l 2022-05-13T23:15:30.164 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:30.145+0000 7f53b4ed4b80 20 client.25037 populate_metadata read hostname 'smithi002' 2022-05-13T23:15:30.164 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:30.145+0000 7f53b4ed4b80 1 --2- 172.21.15.2:0/567612222 >> [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] conn(0x5566ced71320 0x5566ced6dc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:15:30.164 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:30.145+0000 7f53b4ed4b80 4 client.25037 mds_command: new command op to 24611 tid=0[{"prefix": "status"}] 2022-05-13T23:15:30.164 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:30.145+0000 7f53b4ed4b80 1 -- 172.21.15.2:0/567612222 --> [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] -- command(tid 0: {"prefix": "status"}) v1 -- 0x5566cebf9790 con 0x5566ced71320 2022-05-13T23:15:30.165 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:30.146+0000 7f539f7fe700 1 --2- 172.21.15.2:0/567612222 >> [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] conn(0x5566ced71320 0x5566ced6dc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:15:30.165 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:30.146+0000 7f539f7fe700 1 --2- 172.21.15.2:0/567612222 >> [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] conn(0x5566ced71320 0x5566ced6dc90 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b2231bbc755a3de7 server_cookie=d47002feb9ecc51c in_seq=0 out_seq=0 2022-05-13T23:15:30.165 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:30.146+0000 7f537e7fc700 10 client.25037 ms_handle_connect on v2:172.21.15.115:6838/4149542923 2022-05-13T23:15:30.165 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:30.146+0000 7f537e7fc700 1 -- 172.21.15.2:0/567612222 <== mds.0 v2:172.21.15.115:6838/4149542923 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+332 (crc 0 0 0) 0x5566cebf9790 con 0x5566ced71320 2022-05-13T23:15:30.166 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:30.146+0000 7f537e7fc700 10 client.25037 handle_command_reply: tid=0 2022-05-13T23:15:30.166 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:30.146+0000 7f53b4ed4b80 1 client.25037 shutdown 2022-05-13T23:15:30.166 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:30.146+0000 7f53b4ed4b80 1 -- 172.21.15.2:0/567612222 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5566ced06640 msgr2=0x5566ced14180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:15:30.166 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:30.146+0000 7f53b4ed4b80 1 --2- 172.21.15.2:0/567612222 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5566ced06640 0x5566ced14180 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f5394013490 tx=0x7f539402b040 comp rx=0 tx=0).stop 2022-05-13T23:15:30.166 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:30.147+0000 7f53b4ed4b80 1 -- 172.21.15.2:0/567612222 shutdown_connections 2022-05-13T23:15:30.167 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:30.147+0000 7f53b4ed4b80 1 --2- 172.21.15.2:0/567612222 >> [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] conn(0x5566ced71320 0x5566ced6dc90 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:30.167 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:30.147+0000 7f53b4ed4b80 1 --2- 172.21.15.2:0/567612222 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5566ced075a0 0x5566ced6c450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:30.167 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:30.147+0000 7f53b4ed4b80 1 --2- 172.21.15.2:0/567612222 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x5566ced05cf0 0x5566ced13c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:30.167 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:30.147+0000 7f537e7fc700 0 client.25037 ms_handle_reset on v2:172.21.15.115:6838/4149542923 2022-05-13T23:15:30.168 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:30.147+0000 7f53b4ed4b80 1 --2- 172.21.15.2:0/567612222 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5566ced06640 0x5566ced14180 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:30.168 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:30.147+0000 7f53b4ed4b80 1 -- 172.21.15.2:0/567612222 >> 172.21.15.2:0/567612222 conn(0x5566ced00b10 msgr2=0x5566ced0a1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-05-13T23:15:30.168 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:30.147+0000 7f53b4ed4b80 1 -- 172.21.15.2:0/567612222 shutdown_connections 2022-05-13T23:15:30.168 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:30.147+0000 7f53b4ed4b80 1 -- 172.21.15.2:0/567612222 wait complete. 2022-05-13T23:15:30.168 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:30.147+0000 7f53b4ed4b80 20 client.25037 trim_cache size 0 max 16384 2022-05-13T23:15:30.169 INFO:teuthology.orchestra.run.smithi002.stdout:{ 2022-05-13T23:15:30.169 INFO:teuthology.orchestra.run.smithi002.stdout: "cluster_fsid": "16cf5f88-d311-11ec-be40-001a4aab830c", 2022-05-13T23:15:30.170 INFO:teuthology.orchestra.run.smithi002.stdout: "whoami": 0, 2022-05-13T23:15:30.170 INFO:teuthology.orchestra.run.smithi002.stdout: "id": 24611, 2022-05-13T23:15:30.170 INFO:teuthology.orchestra.run.smithi002.stdout: "want_state": "up:active", 2022-05-13T23:15:30.170 INFO:teuthology.orchestra.run.smithi002.stdout: "state": "up:active", 2022-05-13T23:15:30.170 INFO:teuthology.orchestra.run.smithi002.stdout: "fs_name": "cephfs", 2022-05-13T23:15:30.171 INFO:teuthology.orchestra.run.smithi002.stdout: "rank_uptime": 73.907044526999996, 2022-05-13T23:15:30.171 INFO:teuthology.orchestra.run.smithi002.stdout: "mdsmap_epoch": 26, 2022-05-13T23:15:30.171 INFO:teuthology.orchestra.run.smithi002.stdout: "osdmap_epoch": 85, 2022-05-13T23:15:30.171 INFO:teuthology.orchestra.run.smithi002.stdout: "osdmap_epoch_barrier": 83, 2022-05-13T23:15:30.171 INFO:teuthology.orchestra.run.smithi002.stdout: "uptime": 79.916444483999996 2022-05-13T23:15:30.172 INFO:teuthology.orchestra.run.smithi002.stdout:} 2022-05-13T23:15:30.173 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:30.147+0000 7f53a499d700 1 -- 172.21.15.2:0/1374721098 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f5388096ce0 msgr2=0x7f5388099190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:15:30.173 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:30.147+0000 7f53a499d700 1 --2- 172.21.15.2:0/1374721098 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f5388096ce0 0x7f5388099190 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f53980079e0 tx=0x7f539800b040 comp rx=0 tx=0).stop 2022-05-13T23:15:30.173 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:30.147+0000 7f53a499d700 1 -- 172.21.15.2:0/1374721098 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f53a0064710 msgr2=0x7f53a010e540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:15:30.173 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:30.148+0000 7f53a499d700 1 --2- 172.21.15.2:0/1374721098 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f53a0064710 0x7f53a010e540 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f539000e8c0 tx=0x7f539000b040 comp rx=0 tx=0).stop 2022-05-13T23:15:30.174 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:30.148+0000 7f53a499d700 1 -- 172.21.15.2:0/1374721098 shutdown_connections 2022-05-13T23:15:30.174 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:30.148+0000 7f53a499d700 1 --2- 172.21.15.2:0/1374721098 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f5388096ce0 0x7f5388099190 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:30.174 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:30.148+0000 7f53a499d700 1 --2- 172.21.15.2:0/1374721098 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f53a0103310 0x7f53a0115b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:30.174 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:30.148+0000 7f53a499d700 1 --2- 172.21.15.2:0/1374721098 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f53a0064710 0x7f53a010e540 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:30.175 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:30.148+0000 7f53a499d700 1 --2- 172.21.15.2:0/1374721098 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f53a01009a0 0x7f53a010ea80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:30.175 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:30.148+0000 7f53a499d700 1 -- 172.21.15.2:0/1374721098 >> 172.21.15.2:0/1374721098 conn(0x7f53a00f5b30 msgr2=0x7f53a0103ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-05-13T23:15:30.175 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:30.148+0000 7f53a499d700 1 -- 172.21.15.2:0/1374721098 shutdown_connections 2022-05-13T23:15:30.175 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:30.148+0000 7f53a499d700 1 -- 172.21.15.2:0/1374721098 wait complete. 2022-05-13T23:15:30.176 INFO:tasks.cephadm:Setup complete, yielding 2022-05-13T23:15:30.177 INFO:teuthology.run_tasks:Running task cephadm.shell... 2022-05-13T23:15:30.184 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi002.front.sepia.ceph.com 2022-05-13T23:15:30.185 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:23b0658391ce94ca40799469eec00c28b298816c shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 16cf5f88-d311-11ec-be40-001a4aab830c -- bash -c 'ceph orch status' 2022-05-13T23:15:30.574 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:15:30 smithi002 ceph-mon[71816]: pgmap v327: 97 pgs: 97 active+clean; 584 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-05-13T23:15:30.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:15:30 smithi115 ceph-mon[78593]: pgmap v327: 97 pgs: 97 active+clean; 584 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-05-13T23:15:30.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:15:30 smithi047 ceph-mon[77147]: pgmap v327: 97 pgs: 97 active+clean; 584 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-05-13T23:15:30.858 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/16cf5f88-d311-11ec-be40-001a4aab830c/mon.a/config 2022-05-13T23:15:32.573 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:15:32 smithi002 ceph-mon[71816]: pgmap v328: 97 pgs: 97 active+clean; 584 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 11 op/s 2022-05-13T23:15:32.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:15:32 smithi115 ceph-mon[78593]: pgmap v328: 97 pgs: 97 active+clean; 584 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 11 op/s 2022-05-13T23:15:32.713 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:32.688+0000 7f3a04ac6700 1 -- 172.21.15.2:0/2809467697 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f3a000fc890 msgr2=0x7f3a000fcce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:15:32.713 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:32.688+0000 7f3a04ac6700 1 --2- 172.21.15.2:0/2809467697 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f3a000fc890 0x7f3a000fcce0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f39f4004870 tx=0x7f39f4009cb0 comp rx=0 tx=0).stop 2022-05-13T23:15:32.714 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:32.688+0000 7f3a04ac6700 1 -- 172.21.15.2:0/2809467697 shutdown_connections 2022-05-13T23:15:32.714 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:32.688+0000 7f3a04ac6700 1 --2- 172.21.15.2:0/2809467697 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3a000fd220 0x7f3a00103ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:32.714 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:32.688+0000 7f3a04ac6700 1 --2- 172.21.15.2:0/2809467697 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3a000fb5f0 0x7f3a000fba00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:32.714 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:32.688+0000 7f3a04ac6700 1 --2- 172.21.15.2:0/2809467697 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f3a000fc890 0x7f3a000fcce0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:32.715 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:32.688+0000 7f3a04ac6700 1 -- 172.21.15.2:0/2809467697 >> 172.21.15.2:0/2809467697 conn(0x7f3a000f6de0 msgr2=0x7f3a000f9210 unknown :-1 s=STATE_NONE l=0).mark_down 2022-05-13T23:15:32.715 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:32.688+0000 7f3a04ac6700 1 -- 172.21.15.2:0/2809467697 shutdown_connections 2022-05-13T23:15:32.715 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:32.688+0000 7f3a04ac6700 1 -- 172.21.15.2:0/2809467697 wait complete. 2022-05-13T23:15:32.715 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:32.689+0000 7f3a04ac6700 1 Processor -- start 2022-05-13T23:15:32.715 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:32.689+0000 7f3a04ac6700 1 -- start start 2022-05-13T23:15:32.716 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:32.689+0000 7f3a04ac6700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3a000fb5f0 0x7f3a0010c3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:15:32.716 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:32.689+0000 7f3a04ac6700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f3a000fc890 0x7f3a0010c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:15:32.716 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:32.689+0000 7f3a04ac6700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3a000fd220 0x7f3a00113980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:15:32.716 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:32.689+0000 7f3a04ac6700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a00113ec0 con 0x7f3a000fb5f0 2022-05-13T23:15:32.717 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:32.689+0000 7f3a04ac6700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a00114030 con 0x7f3a000fd220 2022-05-13T23:15:32.717 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:32.689+0000 7f3a04ac6700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a001141f0 con 0x7f3a000fc890 2022-05-13T23:15:32.717 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:32.690+0000 7f39fdd9b700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f3a000fc890 0x7f3a0010c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:15:32.717 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:32.690+0000 7f39fe59c700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3a000fb5f0 0x7f3a0010c3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:15:32.717 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:32.690+0000 7f39fdd9b700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f3a000fc890 0x7f3a0010c8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.2:56602/0 (socket says 172.21.15.2:56602) 2022-05-13T23:15:32.718 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:32.690+0000 7f39fdd9b700 1 -- 172.21.15.2:0/116271540 learned_addr learned my addr 172.21.15.2:0/116271540 (peer_addr_for_me v2:172.21.15.2:0/0) 2022-05-13T23:15:32.718 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:32.690+0000 7f39fed9d700 1 --2- 172.21.15.2:0/116271540 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3a000fd220 0x7f3a00113980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:15:32.718 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:32.690+0000 7f39fdd9b700 1 -- 172.21.15.2:0/116271540 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3a000fd220 msgr2=0x7f3a00113980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:15:32.718 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:32.690+0000 7f39fdd9b700 1 --2- 172.21.15.2:0/116271540 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3a000fd220 0x7f3a00113980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:32.719 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:32.690+0000 7f39fdd9b700 1 -- 172.21.15.2:0/116271540 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3a000fb5f0 msgr2=0x7f3a0010c3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:15:32.719 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:32.690+0000 7f39fdd9b700 1 --2- 172.21.15.2:0/116271540 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3a000fb5f0 0x7f3a0010c3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:32.719 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:32.690+0000 7f39fdd9b700 1 -- 172.21.15.2:0/116271540 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39f80097c0 con 0x7f3a000fc890 2022-05-13T23:15:32.719 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:32.690+0000 7f39fe59c700 1 --2- 172.21.15.2:0/116271540 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3a000fb5f0 0x7f3a0010c3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-05-13T23:15:32.719 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:32.690+0000 7f39fdd9b700 1 --2- 172.21.15.2:0/116271540 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f3a000fc890 0x7f3a0010c8f0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f39f4004ee0 tx=0x7f39f4009cb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-05-13T23:15:32.720 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:32.691+0000 7f39eeffd700 1 -- 172.21.15.2:0/116271540 <== mon.2 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39f400e040 con 0x7f3a000fc890 2022-05-13T23:15:32.720 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:32.691+0000 7f3a04ac6700 1 -- 172.21.15.2:0/116271540 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f39f4005040 con 0x7f3a000fc890 2022-05-13T23:15:32.720 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:32.691+0000 7f39eeffd700 1 -- 172.21.15.2:0/116271540 <== mon.2 v2:172.21.15.115:3300/0 2 ==== config(26 keys) v1 ==== 1087+0+0 (secure 0 0 0) 0x7f39f4004200 con 0x7f3a000fc890 2022-05-13T23:15:32.720 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:32.691+0000 7f39eeffd700 1 -- 172.21.15.2:0/116271540 <== mon.2 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39f401ca80 con 0x7f3a000fc890 2022-05-13T23:15:32.721 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:32.691+0000 7f3a04ac6700 1 -- 172.21.15.2:0/116271540 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3a0006ea60 con 0x7f3a000fc890 2022-05-13T23:15:32.721 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:32.692+0000 7f3a04ac6700 1 -- 172.21.15.2:0/116271540 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3a00106e90 con 0x7f3a000fc890 2022-05-13T23:15:32.721 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:32.694+0000 7f39eeffd700 1 -- 172.21.15.2:0/116271540 <== mon.2 v2:172.21.15.115:3300/0 4 ==== mgrmap(e 16) v1 ==== 125136+0+0 (secure 0 0 0) 0x7f39f4009240 con 0x7f3a000fc890 2022-05-13T23:15:32.721 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:32.695+0000 7f39eeffd700 1 --2- 172.21.15.2:0/116271540 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f39e8096ce0 0x7f39e8099190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:15:32.722 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:32.696+0000 7f39eeffd700 1 -- 172.21.15.2:0/116271540 <== mon.2 v2:172.21.15.115:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8120+0+0 (secure 0 0 0) 0x7f39f40c0fc0 con 0x7f3a000fc890 2022-05-13T23:15:32.722 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:32.696+0000 7f39fe59c700 1 --2- 172.21.15.2:0/116271540 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f39e8096ce0 0x7f39e8099190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:15:32.722 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:32.696+0000 7f39fe59c700 1 --2- 172.21.15.2:0/116271540 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f39e8096ce0 0x7f39e8099190 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f3a000fc6f0 tx=0x7f39f000b040 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-05-13T23:15:32.722 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:32.698+0000 7f39eeffd700 1 -- 172.21.15.2:0/116271540 <== mon.2 v2:172.21.15.115:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+165215 (secure 0 0 0) 0x7f39f408e270 con 0x7f3a000fc890 2022-05-13T23:15:32.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:15:32 smithi047 ceph-mon[77147]: pgmap v328: 97 pgs: 97 active+clean; 584 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 11 op/s 2022-05-13T23:15:32.905 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:32.903+0000 7f3a04ac6700 1 -- 172.21.15.2:0/116271540 --> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] -- mgr_command(tid 0: {"prefix": "orch status", "target": ["mon-mgr", ""]}) v1 -- 0x7f3a0002a500 con 0x7f39e8096ce0 2022-05-13T23:15:32.909 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:32.904+0000 7f39eeffd700 1 -- 172.21.15.2:0/116271540 <== mgr.14148 v2:172.21.15.2:6800/2788339627 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+43 (secure 0 0 0) 0x7f3a0002a500 con 0x7f39e8096ce0 2022-05-13T23:15:32.909 INFO:teuthology.orchestra.run.smithi002.stdout:Backend: cephadm 2022-05-13T23:15:32.910 INFO:teuthology.orchestra.run.smithi002.stdout:Available: Yes 2022-05-13T23:15:32.910 INFO:teuthology.orchestra.run.smithi002.stdout:Paused: No 2022-05-13T23:15:32.911 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:32.907+0000 7f3a04ac6700 1 -- 172.21.15.2:0/116271540 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f39e8096ce0 msgr2=0x7f39e8099190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:15:32.911 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:32.907+0000 7f3a04ac6700 1 --2- 172.21.15.2:0/116271540 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f39e8096ce0 0x7f39e8099190 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f3a000fc6f0 tx=0x7f39f000b040 comp rx=0 tx=0).stop 2022-05-13T23:15:32.911 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:32.907+0000 7f3a04ac6700 1 -- 172.21.15.2:0/116271540 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f3a000fc890 msgr2=0x7f3a0010c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:15:32.913 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:32.907+0000 7f3a04ac6700 1 --2- 172.21.15.2:0/116271540 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f3a000fc890 0x7f3a0010c8f0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f39f4004ee0 tx=0x7f39f4009cb0 comp rx=0 tx=0).stop 2022-05-13T23:15:32.913 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:32.907+0000 7f3a04ac6700 1 -- 172.21.15.2:0/116271540 shutdown_connections 2022-05-13T23:15:32.914 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:32.907+0000 7f3a04ac6700 1 --2- 172.21.15.2:0/116271540 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3a000fd220 0x7f3a00113980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:32.914 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:32.907+0000 7f3a04ac6700 1 --2- 172.21.15.2:0/116271540 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f39e8096ce0 0x7f39e8099190 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:32.914 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:32.907+0000 7f3a04ac6700 1 --2- 172.21.15.2:0/116271540 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3a000fb5f0 0x7f3a0010c3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:32.915 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:32.907+0000 7f3a04ac6700 1 --2- 172.21.15.2:0/116271540 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f3a000fc890 0x7f3a0010c8f0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:32.915 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:32.907+0000 7f3a04ac6700 1 -- 172.21.15.2:0/116271540 >> 172.21.15.2:0/116271540 conn(0x7f3a000f6de0 msgr2=0x7f3a000ffac0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-05-13T23:15:32.915 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:32.907+0000 7f3a04ac6700 1 -- 172.21.15.2:0/116271540 shutdown_connections 2022-05-13T23:15:32.916 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:32.907+0000 7f3a04ac6700 1 -- 172.21.15.2:0/116271540 wait complete. 2022-05-13T23:15:33.427 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:23b0658391ce94ca40799469eec00c28b298816c shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 16cf5f88-d311-11ec-be40-001a4aab830c -- bash -c 'ceph orch ps' 2022-05-13T23:15:34.103 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/16cf5f88-d311-11ec-be40-001a4aab830c/mon.a/config 2022-05-13T23:15:34.572 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:15:34 smithi002 ceph-mon[71816]: from='client.25043 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2022-05-13T23:15:34.573 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:15:34 smithi002 ceph-mon[71816]: pgmap v329: 97 pgs: 97 active+clean; 584 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2022-05-13T23:15:34.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:15:34 smithi115 ceph-mon[78593]: from='client.25043 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2022-05-13T23:15:34.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:15:34 smithi115 ceph-mon[78593]: pgmap v329: 97 pgs: 97 active+clean; 584 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2022-05-13T23:15:34.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:15:34 smithi047 ceph-mon[77147]: from='client.25043 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2022-05-13T23:15:34.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:15:34 smithi047 ceph-mon[77147]: pgmap v329: 97 pgs: 97 active+clean; 584 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2022-05-13T23:15:35.905 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:35.902+0000 7f0ada7ad700 1 -- 172.21.15.2:0/3791472871 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0ad40ff420 msgr2=0x7f0ad4105ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:15:35.905 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:35.902+0000 7f0ada7ad700 1 --2- 172.21.15.2:0/3791472871 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0ad40ff420 0x7f0ad4105ce0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f0acc009be0 tx=0x7f0acc004d10 comp rx=0 tx=0).stop 2022-05-13T23:15:35.905 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:35.902+0000 7f0ada7ad700 1 -- 172.21.15.2:0/3791472871 shutdown_connections 2022-05-13T23:15:35.906 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:35.902+0000 7f0ada7ad700 1 --2- 172.21.15.2:0/3791472871 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0ad40ff420 0x7f0ad4105ce0 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:35.906 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:35.902+0000 7f0ada7ad700 1 --2- 172.21.15.2:0/3791472871 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0ad40fd7f0 0x7f0ad40fdc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:35.906 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:35.902+0000 7f0ada7ad700 1 --2- 172.21.15.2:0/3791472871 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f0ad40fea90 0x7f0ad40feee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:35.907 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:35.902+0000 7f0ada7ad700 1 -- 172.21.15.2:0/3791472871 >> 172.21.15.2:0/3791472871 conn(0x7f0ad40f9030 msgr2=0x7f0ad40fb410 unknown :-1 s=STATE_NONE l=0).mark_down 2022-05-13T23:15:35.907 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:35.902+0000 7f0ada7ad700 1 -- 172.21.15.2:0/3791472871 shutdown_connections 2022-05-13T23:15:35.907 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:35.903+0000 7f0ada7ad700 1 -- 172.21.15.2:0/3791472871 wait complete. 2022-05-13T23:15:35.908 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:35.903+0000 7f0ada7ad700 1 Processor -- start 2022-05-13T23:15:35.908 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:35.904+0000 7f0ada7ad700 1 -- start start 2022-05-13T23:15:35.911 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:35.904+0000 7f0ada7ad700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f0ad40fd7f0 0x7f0ad410e440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:15:35.911 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:35.904+0000 7f0ada7ad700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0ad40fea90 0x7f0ad410e980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:15:35.911 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:35.904+0000 7f0ada7ad700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0ad40ff420 0x7f0ad4115a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:15:35.912 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:35.904+0000 7f0ada7ad700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ad4115f50 con 0x7f0ad40ff420 2022-05-13T23:15:35.912 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:35.904+0000 7f0ada7ad700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ad41160c0 con 0x7f0ad40fea90 2022-05-13T23:15:35.913 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:35.904+0000 7f0ada7ad700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ad4116280 con 0x7f0ad40fd7f0 2022-05-13T23:15:35.913 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:35.904+0000 7f0ad3fff700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f0ad40fd7f0 0x7f0ad410e440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:15:35.913 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:35.904+0000 7f0ad3fff700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f0ad40fd7f0 0x7f0ad410e440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.2:56612/0 (socket says 172.21.15.2:56612) 2022-05-13T23:15:35.914 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:35.904+0000 7f0ad3fff700 1 -- 172.21.15.2:0/2187999559 learned_addr learned my addr 172.21.15.2:0/2187999559 (peer_addr_for_me v2:172.21.15.2:0/0) 2022-05-13T23:15:35.914 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:35.904+0000 7f0ad8d4a700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0ad40ff420 0x7f0ad4115a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:15:35.914 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:35.904+0000 7f0ad37fe700 1 --2- 172.21.15.2:0/2187999559 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0ad40fea90 0x7f0ad410e980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:15:35.915 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:35.905+0000 7f0ad3fff700 1 -- 172.21.15.2:0/2187999559 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0ad40fea90 msgr2=0x7f0ad410e980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:15:35.915 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:35.905+0000 7f0ad3fff700 1 --2- 172.21.15.2:0/2187999559 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0ad40fea90 0x7f0ad410e980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:35.915 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:35.905+0000 7f0ad3fff700 1 -- 172.21.15.2:0/2187999559 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0ad40ff420 msgr2=0x7f0ad4115a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:15:35.916 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:35.905+0000 7f0ad3fff700 1 --2- 172.21.15.2:0/2187999559 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0ad40ff420 0x7f0ad4115a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:35.916 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:35.905+0000 7f0ad3fff700 1 -- 172.21.15.2:0/2187999559 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0acc0097c0 con 0x7f0ad40fd7f0 2022-05-13T23:15:35.916 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:35.905+0000 7f0ad3fff700 1 --2- 172.21.15.2:0/2187999559 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f0ad40fd7f0 0x7f0ad410e440 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f0ac400e940 tx=0x7f0ac400b040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-05-13T23:15:35.916 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:35.905+0000 7f0ad0ff9700 1 -- 172.21.15.2:0/2187999559 <== mon.2 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ac4007b40 con 0x7f0ad40fd7f0 2022-05-13T23:15:35.917 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:35.906+0000 7f0ad0ff9700 1 -- 172.21.15.2:0/2187999559 <== mon.2 v2:172.21.15.115:3300/0 2 ==== config(26 keys) v1 ==== 1087+0+0 (secure 0 0 0) 0x7f0ac400f040 con 0x7f0ad40fd7f0 2022-05-13T23:15:35.917 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:35.906+0000 7f0ada7ad700 1 -- 172.21.15.2:0/2187999559 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0ad4116500 con 0x7f0ad40fd7f0 2022-05-13T23:15:35.917 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:35.906+0000 7f0ad0ff9700 1 -- 172.21.15.2:0/2187999559 <== mon.2 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ac4014890 con 0x7f0ad40fd7f0 2022-05-13T23:15:35.917 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:35.906+0000 7f0ada7ad700 1 -- 172.21.15.2:0/2187999559 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ad4116a70 con 0x7f0ad40fd7f0 2022-05-13T23:15:35.918 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:35.907+0000 7f0ada7ad700 1 -- 172.21.15.2:0/2187999559 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0ad4108fd0 con 0x7f0ad40fd7f0 2022-05-13T23:15:35.918 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:35.909+0000 7f0ad0ff9700 1 -- 172.21.15.2:0/2187999559 <== mon.2 v2:172.21.15.115:3300/0 4 ==== mgrmap(e 16) v1 ==== 125136+0+0 (secure 0 0 0) 0x7f0ac401b030 con 0x7f0ad40fd7f0 2022-05-13T23:15:35.919 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:35.910+0000 7f0ad0ff9700 1 --2- 172.21.15.2:0/2187999559 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f0abc096ce0 0x7f0abc099190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:15:35.919 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:35.910+0000 7f0ad0ff9700 1 -- 172.21.15.2:0/2187999559 <== mon.2 v2:172.21.15.115:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8120+0+0 (secure 0 0 0) 0x7f0ac40bee90 con 0x7f0ad40fd7f0 2022-05-13T23:15:35.920 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:35.910+0000 7f0ad37fe700 1 --2- 172.21.15.2:0/2187999559 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f0abc096ce0 0x7f0abc099190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:15:35.920 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:35.911+0000 7f0ad37fe700 1 --2- 172.21.15.2:0/2187999559 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f0abc096ce0 0x7f0abc099190 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f0ac8004790 tx=0x7f0ac800b040 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-05-13T23:15:35.920 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:35.912+0000 7f0ad0ff9700 1 -- 172.21.15.2:0/2187999559 <== mon.2 v2:172.21.15.115:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+165215 (secure 0 0 0) 0x7f0ac404b570 con 0x7f0ad40fd7f0 2022-05-13T23:15:36.114 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:36.112+0000 7f0ada7ad700 1 -- 172.21.15.2:0/2187999559 --> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f0ad402a530 con 0x7f0abc096ce0 2022-05-13T23:15:36.131 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:36.126+0000 7f0ad0ff9700 1 -- 172.21.15.2:0/2187999559 <== mgr.14148 v2:172.21.15.2:6800/2788339627 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3999 (secure 0 0 0) 0x7f0ad402a530 con 0x7f0abc096ce0 2022-05-13T23:15:36.131 INFO:teuthology.orchestra.run.smithi002.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-05-13T23:15:36.132 INFO:teuthology.orchestra.run.smithi002.stdout:mds.a smithi002 running (118s) 11s ago 117s 12.8M - 17.2.0-188-g23b06583 efd555832784 336a72a34d7f 2022-05-13T23:15:36.132 INFO:teuthology.orchestra.run.smithi002.stdout:mds.b smithi047 running (102s) 13s ago 102s 18.4M - 17.2.0-188-g23b06583 efd555832784 e4a43bf12d38 2022-05-13T23:15:36.132 INFO:teuthology.orchestra.run.smithi002.stdout:mds.c smithi115 running (93s) 13s ago 92s 17.9M - 17.2.0-188-g23b06583 efd555832784 6a91b898e3ca 2022-05-13T23:15:36.135 INFO:teuthology.orchestra.run.smithi002.stdout:mds.d smithi002 running (113s) 11s ago 113s 16.0M - 17.2.0-188-g23b06583 efd555832784 f91119b0055d 2022-05-13T23:15:36.135 INFO:teuthology.orchestra.run.smithi002.stdout:mds.e smithi047 running (100s) 13s ago 100s 18.3M - 17.2.0-188-g23b06583 efd555832784 5150a7b9c755 2022-05-13T23:15:36.135 INFO:teuthology.orchestra.run.smithi002.stdout:mds.f smithi115 running (90s) 13s ago 90s 20.2M - 17.2.0-188-g23b06583 efd555832784 ee0547f882f1 2022-05-13T23:15:36.135 INFO:teuthology.orchestra.run.smithi002.stdout:mds.g smithi002 running (109s) 11s ago 109s 12.7M - 17.2.0-188-g23b06583 efd555832784 cdd0c1e5ef65 2022-05-13T23:15:36.136 INFO:teuthology.orchestra.run.smithi002.stdout:mds.h smithi047 running (98s) 13s ago 97s 19.6M - 17.2.0-188-g23b06583 efd555832784 0223223a4046 2022-05-13T23:15:36.136 INFO:teuthology.orchestra.run.smithi002.stdout:mds.i smithi115 running (88s) 13s ago 88s 18.2M - 17.2.0-188-g23b06583 efd555832784 a2f5aec744fe 2022-05-13T23:15:36.136 INFO:teuthology.orchestra.run.smithi002.stdout:mds.j smithi002 running (105s) 11s ago 104s 16.0M - 17.2.0-188-g23b06583 efd555832784 2d1f81d9f5e6 2022-05-13T23:15:36.136 INFO:teuthology.orchestra.run.smithi002.stdout:mds.k smithi047 running (95s) 13s ago 95s 20.0M - 17.2.0-188-g23b06583 efd555832784 254778589c1d 2022-05-13T23:15:36.137 INFO:teuthology.orchestra.run.smithi002.stdout:mds.l smithi115 running (86s) 13s ago 85s 22.2M - 17.2.0-188-g23b06583 efd555832784 201973bd31ef 2022-05-13T23:15:36.137 INFO:teuthology.orchestra.run.smithi002.stdout:mgr.x smithi002 *:9283 running (9m) 11s ago 9m 487M - 17.2.0-188-g23b06583 efd555832784 6418c2a5057e 2022-05-13T23:15:36.137 INFO:teuthology.orchestra.run.smithi002.stdout:mgr.y smithi047 *:8443 running (7m) 13s ago 7m 426M - 17.2.0-188-g23b06583 efd555832784 6f563916e523 2022-05-13T23:15:36.137 INFO:teuthology.orchestra.run.smithi002.stdout:mgr.z smithi115 *:8443 running (7m) 13s ago 7m 428M - 17.2.0-188-g23b06583 efd555832784 20748358073a 2022-05-13T23:15:36.138 INFO:teuthology.orchestra.run.smithi002.stdout:mon.a smithi002 running (9m) 11s ago 9m 88.9M 2048M 17.2.0-188-g23b06583 efd555832784 36eab9452fd0 2022-05-13T23:15:36.138 INFO:teuthology.orchestra.run.smithi002.stdout:mon.b smithi047 running (7m) 13s ago 7m 62.3M 2048M 17.2.0-188-g23b06583 efd555832784 01602766cbfc 2022-05-13T23:15:36.138 INFO:teuthology.orchestra.run.smithi002.stdout:mon.c smithi115 running (7m) 13s ago 7m 65.0M 2048M 17.2.0-188-g23b06583 efd555832784 099efb3934a6 2022-05-13T23:15:36.138 INFO:teuthology.orchestra.run.smithi002.stdout:osd.0 smithi002 running (6m) 11s ago 6m 61.0M 4319M 17.2.0-188-g23b06583 efd555832784 52e8e0e25670 2022-05-13T23:15:36.139 INFO:teuthology.orchestra.run.smithi002.stdout:osd.1 smithi047 running (6m) 13s ago 6m 65.4M 4294M 17.2.0-188-g23b06583 efd555832784 e48ce0311c03 2022-05-13T23:15:36.139 INFO:teuthology.orchestra.run.smithi002.stdout:osd.2 smithi115 running (6m) 13s ago 6m 64.4M 4293M 17.2.0-188-g23b06583 efd555832784 a43b6689593a 2022-05-13T23:15:36.139 INFO:teuthology.orchestra.run.smithi002.stdout:osd.3 smithi002 running (5m) 11s ago 5m 59.0M 4319M 17.2.0-188-g23b06583 efd555832784 d72ffd81eba0 2022-05-13T23:15:36.139 INFO:teuthology.orchestra.run.smithi002.stdout:osd.4 smithi047 running (5m) 13s ago 5m 59.7M 4294M 17.2.0-188-g23b06583 efd555832784 6494fa555c06 2022-05-13T23:15:36.139 INFO:teuthology.orchestra.run.smithi002.stdout:osd.5 smithi115 running (4m) 13s ago 4m 63.9M 4293M 17.2.0-188-g23b06583 efd555832784 29ea814195c4 2022-05-13T23:15:36.140 INFO:teuthology.orchestra.run.smithi002.stdout:osd.6 smithi002 running (4m) 11s ago 4m 55.9M 4319M 17.2.0-188-g23b06583 efd555832784 98340e81f364 2022-05-13T23:15:36.140 INFO:teuthology.orchestra.run.smithi002.stdout:osd.7 smithi047 running (3m) 13s ago 3m 61.9M 4294M 17.2.0-188-g23b06583 efd555832784 9807cb37b73b 2022-05-13T23:15:36.140 INFO:teuthology.orchestra.run.smithi002.stdout:osd.8 smithi115 running (3m) 13s ago 3m 60.9M 4293M 17.2.0-188-g23b06583 efd555832784 2c53fca77ebc 2022-05-13T23:15:36.140 INFO:teuthology.orchestra.run.smithi002.stdout:osd.9 smithi002 running (3m) 11s ago 3m 54.7M 4319M 17.2.0-188-g23b06583 efd555832784 9d85841440ad 2022-05-13T23:15:36.141 INFO:teuthology.orchestra.run.smithi002.stdout:osd.10 smithi047 running (2m) 13s ago 2m 61.6M 4294M 17.2.0-188-g23b06583 efd555832784 3a14db4ed870 2022-05-13T23:15:36.141 INFO:teuthology.orchestra.run.smithi002.stdout:osd.11 smithi115 running (2m) 13s ago 2m 54.6M 4293M 17.2.0-188-g23b06583 efd555832784 571170a588fd 2022-05-13T23:15:36.142 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:36.128+0000 7f0ada7ad700 1 -- 172.21.15.2:0/2187999559 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f0abc096ce0 msgr2=0x7f0abc099190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:15:36.142 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:36.128+0000 7f0ada7ad700 1 --2- 172.21.15.2:0/2187999559 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f0abc096ce0 0x7f0abc099190 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f0ac8004790 tx=0x7f0ac800b040 comp rx=0 tx=0).stop 2022-05-13T23:15:36.143 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:36.128+0000 7f0ada7ad700 1 -- 172.21.15.2:0/2187999559 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f0ad40fd7f0 msgr2=0x7f0ad410e440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:15:36.143 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:36.128+0000 7f0ada7ad700 1 --2- 172.21.15.2:0/2187999559 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f0ad40fd7f0 0x7f0ad410e440 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f0ac400e940 tx=0x7f0ac400b040 comp rx=0 tx=0).stop 2022-05-13T23:15:36.143 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:36.128+0000 7f0ada7ad700 1 -- 172.21.15.2:0/2187999559 shutdown_connections 2022-05-13T23:15:36.143 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:36.128+0000 7f0ada7ad700 1 --2- 172.21.15.2:0/2187999559 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f0abc096ce0 0x7f0abc099190 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:36.143 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:36.128+0000 7f0ada7ad700 1 --2- 172.21.15.2:0/2187999559 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0ad40ff420 0x7f0ad4115a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:36.144 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:36.128+0000 7f0ada7ad700 1 --2- 172.21.15.2:0/2187999559 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f0ad40fd7f0 0x7f0ad410e440 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:36.144 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:36.128+0000 7f0ada7ad700 1 --2- 172.21.15.2:0/2187999559 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0ad40fea90 0x7f0ad410e980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:36.144 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:36.128+0000 7f0ada7ad700 1 -- 172.21.15.2:0/2187999559 >> 172.21.15.2:0/2187999559 conn(0x7f0ad40f9030 msgr2=0x7f0ad4101f20 unknown :-1 s=STATE_NONE l=0).mark_down 2022-05-13T23:15:36.144 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:36.128+0000 7f0ada7ad700 1 -- 172.21.15.2:0/2187999559 shutdown_connections 2022-05-13T23:15:36.145 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:36.128+0000 7f0ada7ad700 1 -- 172.21.15.2:0/2187999559 wait complete. 2022-05-13T23:15:36.572 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:15:36 smithi002 ceph-mon[71816]: pgmap v330: 97 pgs: 97 active+clean; 584 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 11 op/s 2022-05-13T23:15:36.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:15:36 smithi115 ceph-mon[78593]: pgmap v330: 97 pgs: 97 active+clean; 584 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 11 op/s 2022-05-13T23:15:36.676 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:23b0658391ce94ca40799469eec00c28b298816c shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 16cf5f88-d311-11ec-be40-001a4aab830c -- bash -c 'ceph orch ls' 2022-05-13T23:15:36.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:15:36 smithi047 ceph-mon[77147]: pgmap v330: 97 pgs: 97 active+clean; 584 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 11 op/s 2022-05-13T23:15:37.351 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/16cf5f88-d311-11ec-be40-001a4aab830c/mon.a/config 2022-05-13T23:15:37.572 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:15:37 smithi002 ceph-mon[71816]: from='client.25049 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-05-13T23:15:37.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:15:37 smithi115 ceph-mon[78593]: from='client.25049 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-05-13T23:15:37.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:15:37 smithi047 ceph-mon[77147]: from='client.25049 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-05-13T23:15:38.489 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:15:38 smithi002 ceph-mon[71816]: pgmap v331: 97 pgs: 97 active+clean; 584 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2022-05-13T23:15:38.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:15:38 smithi115 ceph-mon[78593]: pgmap v331: 97 pgs: 97 active+clean; 584 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2022-05-13T23:15:38.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:15:38 smithi047 ceph-mon[77147]: pgmap v331: 97 pgs: 97 active+clean; 584 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2022-05-13T23:15:39.121 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:39.120+0000 7f575fb28700 1 -- 172.21.15.2:0/3924490420 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5758070370 msgr2=0x7f575806e870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:15:39.122 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:39.120+0000 7f575fb28700 1 --2- 172.21.15.2:0/3924490420 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5758070370 0x7f575806e870 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f5754004870 tx=0x7f5754009cb0 comp rx=0 tx=0).stop 2022-05-13T23:15:39.122 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:39.120+0000 7f575fb28700 1 -- 172.21.15.2:0/3924490420 shutdown_connections 2022-05-13T23:15:39.122 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:39.120+0000 7f575fb28700 1 --2- 172.21.15.2:0/3924490420 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5758103580 0x7f5758105960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:39.123 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:39.120+0000 7f575fb28700 1 --2- 172.21.15.2:0/3924490420 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5758070370 0x7f575806e870 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:39.123 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:39.120+0000 7f575fb28700 1 --2- 172.21.15.2:0/3924490420 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f575806edb0 0x7f575806f220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:39.123 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:39.120+0000 7f575fb28700 1 -- 172.21.15.2:0/3924490420 >> 172.21.15.2:0/3924490420 conn(0x7f57580f6e00 msgr2=0x7f57580f9210 unknown :-1 s=STATE_NONE l=0).mark_down 2022-05-13T23:15:39.123 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:39.120+0000 7f575fb28700 1 -- 172.21.15.2:0/3924490420 shutdown_connections 2022-05-13T23:15:39.124 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:39.120+0000 7f575fb28700 1 -- 172.21.15.2:0/3924490420 wait complete. 2022-05-13T23:15:39.126 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:39.121+0000 7f575fb28700 1 Processor -- start 2022-05-13T23:15:39.126 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:39.121+0000 7f575fb28700 1 -- start start 2022-05-13T23:15:39.126 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:39.121+0000 7f575fb28700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f575806edb0 0x7f575810e590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:15:39.127 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:39.121+0000 7f575fb28700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f5758070370 0x7f575810ead0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:15:39.127 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:39.121+0000 7f575fb28700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5758103580 0x7f5758115b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:15:39.127 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:39.121+0000 7f575fb28700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57581160a0 con 0x7f5758103580 2022-05-13T23:15:39.127 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:39.121+0000 7f575fb28700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5758116210 con 0x7f575806edb0 2022-05-13T23:15:39.127 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:39.121+0000 7f575fb28700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57581163d0 con 0x7f5758070370 2022-05-13T23:15:39.128 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:39.122+0000 7f575d8c4700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f575806edb0 0x7f575810e590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:15:39.128 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:39.122+0000 7f575d8c4700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f575806edb0 0x7f575810e590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.2:52342/0 (socket says 172.21.15.2:52342) 2022-05-13T23:15:39.128 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:39.122+0000 7f575d8c4700 1 -- 172.21.15.2:0/452214307 learned_addr learned my addr 172.21.15.2:0/452214307 (peer_addr_for_me v2:172.21.15.2:0/0) 2022-05-13T23:15:39.128 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:39.122+0000 7f575e0c5700 1 --2- 172.21.15.2:0/452214307 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5758103580 0x7f5758115b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:15:39.129 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:39.122+0000 7f575d0c3700 1 --2- 172.21.15.2:0/452214307 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f5758070370 0x7f575810ead0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:15:39.129 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:39.122+0000 7f575d8c4700 1 -- 172.21.15.2:0/452214307 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f5758070370 msgr2=0x7f575810ead0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:15:39.129 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:39.122+0000 7f575d8c4700 1 --2- 172.21.15.2:0/452214307 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f5758070370 0x7f575810ead0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:39.129 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:39.122+0000 7f575d8c4700 1 -- 172.21.15.2:0/452214307 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5758103580 msgr2=0x7f5758115b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:15:39.130 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:39.122+0000 7f575d8c4700 1 --2- 172.21.15.2:0/452214307 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5758103580 0x7f5758115b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:39.130 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:39.122+0000 7f575d8c4700 1 -- 172.21.15.2:0/452214307 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5754005040 con 0x7f575806edb0 2022-05-13T23:15:39.130 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:39.122+0000 7f575d8c4700 1 --2- 172.21.15.2:0/452214307 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f575806edb0 0x7f575810e590 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f5754004ee0 tx=0x7f5754009cb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-05-13T23:15:39.130 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:39.123+0000 7f574e7fc700 1 -- 172.21.15.2:0/452214307 <== mon.1 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f575400e040 con 0x7f575806edb0 2022-05-13T23:15:39.130 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:39.123+0000 7f575fb28700 1 -- 172.21.15.2:0/452214307 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5758116650 con 0x7f575806edb0 2022-05-13T23:15:39.131 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:39.123+0000 7f574e7fc700 1 -- 172.21.15.2:0/452214307 <== mon.1 v2:172.21.15.47:3300/0 2 ==== config(26 keys) v1 ==== 1087+0+0 (secure 0 0 0) 0x7f5754000d10 con 0x7f575806edb0 2022-05-13T23:15:39.131 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:39.123+0000 7f574e7fc700 1 -- 172.21.15.2:0/452214307 <== mon.1 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f575401cc70 con 0x7f575806edb0 2022-05-13T23:15:39.131 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:39.123+0000 7f575fb28700 1 -- 172.21.15.2:0/452214307 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5758116b30 con 0x7f575806edb0 2022-05-13T23:15:39.131 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:39.124+0000 7f575fb28700 1 -- 172.21.15.2:0/452214307 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5758109020 con 0x7f575806edb0 2022-05-13T23:15:39.132 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:39.126+0000 7f574e7fc700 1 -- 172.21.15.2:0/452214307 <== mon.1 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 16) v1 ==== 125136+0+0 (secure 0 0 0) 0x7f5754009240 con 0x7f575806edb0 2022-05-13T23:15:39.133 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:39.127+0000 7f574e7fc700 1 --2- 172.21.15.2:0/452214307 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f5744096ce0 0x7f5744099190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:15:39.133 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:39.127+0000 7f574e7fc700 1 -- 172.21.15.2:0/452214307 <== mon.1 v2:172.21.15.47:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8120+0+0 (secure 0 0 0) 0x7f57540c1230 con 0x7f575806edb0 2022-05-13T23:15:39.133 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:39.128+0000 7f575d0c3700 1 --2- 172.21.15.2:0/452214307 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f5744096ce0 0x7f5744099190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:15:39.133 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:39.128+0000 7f575d0c3700 1 --2- 172.21.15.2:0/452214307 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f5744096ce0 0x7f5744099190 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f5748009170 tx=0x7f5748009d60 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-05-13T23:15:39.134 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:39.129+0000 7f574e7fc700 1 -- 172.21.15.2:0/452214307 <== mon.1 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+165215 (secure 0 0 0) 0x7f575408e570 con 0x7f575806edb0 2022-05-13T23:15:39.333 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:39.332+0000 7f575fb28700 1 -- 172.21.15.2:0/452214307 --> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] -- mgr_command(tid 0: {"prefix": "orch ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f575805c920 con 0x7f5744096ce0 2022-05-13T23:15:39.339 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:39.337+0000 7f574e7fc700 1 -- 172.21.15.2:0/452214307 <== mgr.14148 v2:172.21.15.2:6800/2788339627 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+980 (secure 0 0 0) 0x7f575805c920 con 0x7f5744096ce0 2022-05-13T23:15:39.339 INFO:teuthology.orchestra.run.smithi002.stdout:NAME PORTS RUNNING REFRESHED AGE PLACEMENT 2022-05-13T23:15:39.340 INFO:teuthology.orchestra.run.smithi002.stdout:mds.all 12/12 16s ago 2m smithi002=a;smithi002=d;smithi002=g;smithi002=j;smithi047=b;smithi047=e;smithi047=h;smithi047=k;smithi115=c;smithi115=f;smithi115=i;smithi115=l;count:12 2022-05-13T23:15:39.340 INFO:teuthology.orchestra.run.smithi002.stdout:mgr 3/3 16s ago 7m smithi002=x;smithi047=y;smithi115=z;count:3 2022-05-13T23:15:39.340 INFO:teuthology.orchestra.run.smithi002.stdout:mon 3/3 16s ago 7m smithi002:172.21.15.2=a;smithi047:172.21.15.47=b;smithi115:172.21.15.115=c;count:3 2022-05-13T23:15:39.340 INFO:teuthology.orchestra.run.smithi002.stdout:osd 12 16s ago - 2022-05-13T23:15:39.342 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:39.340+0000 7f575fb28700 1 -- 172.21.15.2:0/452214307 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f5744096ce0 msgr2=0x7f5744099190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:15:39.342 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:39.340+0000 7f575fb28700 1 --2- 172.21.15.2:0/452214307 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f5744096ce0 0x7f5744099190 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f5748009170 tx=0x7f5748009d60 comp rx=0 tx=0).stop 2022-05-13T23:15:39.343 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:39.340+0000 7f575fb28700 1 -- 172.21.15.2:0/452214307 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f575806edb0 msgr2=0x7f575810e590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:15:39.343 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:39.340+0000 7f575fb28700 1 --2- 172.21.15.2:0/452214307 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f575806edb0 0x7f575810e590 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f5754004ee0 tx=0x7f5754009cb0 comp rx=0 tx=0).stop 2022-05-13T23:15:39.343 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:39.340+0000 7f575fb28700 1 -- 172.21.15.2:0/452214307 shutdown_connections 2022-05-13T23:15:39.343 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:39.340+0000 7f575fb28700 1 --2- 172.21.15.2:0/452214307 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f5744096ce0 0x7f5744099190 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:39.344 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:39.340+0000 7f575fb28700 1 --2- 172.21.15.2:0/452214307 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5758103580 0x7f5758115b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:39.344 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:39.340+0000 7f575fb28700 1 --2- 172.21.15.2:0/452214307 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f575806edb0 0x7f575810e590 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:39.344 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:39.340+0000 7f575fb28700 1 --2- 172.21.15.2:0/452214307 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f5758070370 0x7f575810ead0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:39.345 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:39.340+0000 7f575fb28700 1 -- 172.21.15.2:0/452214307 >> 172.21.15.2:0/452214307 conn(0x7f57580f6e00 msgr2=0x7f57580f91e0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-05-13T23:15:39.346 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:39.340+0000 7f575fb28700 1 -- 172.21.15.2:0/452214307 shutdown_connections 2022-05-13T23:15:39.346 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:39.340+0000 7f575fb28700 1 -- 172.21.15.2:0/452214307 wait complete. 2022-05-13T23:15:39.916 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:23b0658391ce94ca40799469eec00c28b298816c shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 16cf5f88-d311-11ec-be40-001a4aab830c -- bash -c 'ceph orch host ls' 2022-05-13T23:15:40.594 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/16cf5f88-d311-11ec-be40-001a4aab830c/mon.a/config 2022-05-13T23:15:40.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:15:40 smithi115 ceph-mon[78593]: pgmap v332: 97 pgs: 97 active+clean; 584 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-05-13T23:15:40.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:15:40 smithi115 ceph-mon[78593]: from='client.25027 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2022-05-13T23:15:40.796 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:15:40 smithi047 ceph-mon[77147]: pgmap v332: 97 pgs: 97 active+clean; 584 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-05-13T23:15:40.797 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:15:40 smithi047 ceph-mon[77147]: from='client.25027 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2022-05-13T23:15:40.822 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:15:40 smithi002 ceph-mon[71816]: pgmap v332: 97 pgs: 97 active+clean; 584 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-05-13T23:15:40.823 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:15:40 smithi002 ceph-mon[71816]: from='client.25027 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2022-05-13T23:15:42.499 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:42.497+0000 7fa88d368700 1 -- 172.21.15.2:0/2216677909 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa888070370 msgr2=0x7fa88806e870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:15:42.500 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:42.497+0000 7fa88d368700 1 --2- 172.21.15.2:0/2216677909 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa888070370 0x7fa88806e870 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fa878004870 tx=0x7fa878009cb0 comp rx=0 tx=0).stop 2022-05-13T23:15:42.500 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:42.497+0000 7fa88d368700 1 -- 172.21.15.2:0/2216677909 shutdown_connections 2022-05-13T23:15:42.500 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:42.497+0000 7fa88d368700 1 --2- 172.21.15.2:0/2216677909 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa888103580 0x7fa888105960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:42.500 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:42.497+0000 7fa88d368700 1 --2- 172.21.15.2:0/2216677909 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa888070370 0x7fa88806e870 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:42.501 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:42.497+0000 7fa88d368700 1 --2- 172.21.15.2:0/2216677909 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fa88806edb0 0x7fa88806f220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:42.501 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:42.497+0000 7fa88d368700 1 -- 172.21.15.2:0/2216677909 >> 172.21.15.2:0/2216677909 conn(0x7fa8880f6e00 msgr2=0x7fa8880f9210 unknown :-1 s=STATE_NONE l=0).mark_down 2022-05-13T23:15:42.501 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:42.497+0000 7fa88d368700 1 -- 172.21.15.2:0/2216677909 shutdown_connections 2022-05-13T23:15:42.501 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:42.497+0000 7fa88d368700 1 -- 172.21.15.2:0/2216677909 wait complete. 2022-05-13T23:15:42.502 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:42.498+0000 7fa88d368700 1 Processor -- start 2022-05-13T23:15:42.509 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:42.498+0000 7fa88d368700 1 -- start start 2022-05-13T23:15:42.509 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:42.499+0000 7fa88d368700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa88806edb0 0x7fa88810e650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:15:42.510 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:42.499+0000 7fa88d368700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa888070370 0x7fa88810eb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:15:42.510 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:42.499+0000 7fa88d368700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fa888103580 0x7fa888115c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:15:42.510 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:42.499+0000 7fa88d368700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa888116160 con 0x7fa88806edb0 2022-05-13T23:15:42.510 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:42.499+0000 7fa88d368700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8881162d0 con 0x7fa888070370 2022-05-13T23:15:42.511 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:42.499+0000 7fa88d368700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa888116490 con 0x7fa888103580 2022-05-13T23:15:42.511 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:42.499+0000 7fa886ffd700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa88806edb0 0x7fa88810e650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:15:42.511 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:42.499+0000 7fa886ffd700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa88806edb0 0x7fa88810e650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:36408/0 (socket says 172.21.15.2:36408) 2022-05-13T23:15:42.511 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:42.499+0000 7fa886ffd700 1 -- 172.21.15.2:0/689125365 learned_addr learned my addr 172.21.15.2:0/689125365 (peer_addr_for_me v2:172.21.15.2:0/0) 2022-05-13T23:15:42.512 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:42.499+0000 7fa8877fe700 1 --2- 172.21.15.2:0/689125365 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fa888103580 0x7fa888115c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:15:42.512 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:42.499+0000 7fa8867fc700 1 --2- 172.21.15.2:0/689125365 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa888070370 0x7fa88810eb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:15:42.512 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:42.500+0000 7fa886ffd700 1 -- 172.21.15.2:0/689125365 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fa888103580 msgr2=0x7fa888115c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:15:42.512 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:42.500+0000 7fa886ffd700 1 --2- 172.21.15.2:0/689125365 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fa888103580 0x7fa888115c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:42.512 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:42.500+0000 7fa886ffd700 1 -- 172.21.15.2:0/689125365 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa888070370 msgr2=0x7fa88810eb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:15:42.513 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:42.500+0000 7fa886ffd700 1 --2- 172.21.15.2:0/689125365 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa888070370 0x7fa88810eb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:42.513 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:42.500+0000 7fa886ffd700 1 -- 172.21.15.2:0/689125365 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa878005040 con 0x7fa88806edb0 2022-05-13T23:15:42.513 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:42.500+0000 7fa8867fc700 1 --2- 172.21.15.2:0/689125365 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa888070370 0x7fa88810eb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-05-13T23:15:42.520 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:42.500+0000 7fa886ffd700 1 --2- 172.21.15.2:0/689125365 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa88806edb0 0x7fa88810e650 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fa878008e80 tx=0x7fa878009cb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-05-13T23:15:42.520 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:42.500+0000 7fa86f7fe700 1 -- 172.21.15.2:0/689125365 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa87800e040 con 0x7fa88806edb0 2022-05-13T23:15:42.520 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:42.500+0000 7fa86f7fe700 1 -- 172.21.15.2:0/689125365 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1087+0+0 (secure 0 0 0) 0x7fa878014e40 con 0x7fa88806edb0 2022-05-13T23:15:42.520 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:42.500+0000 7fa88d368700 1 -- 172.21.15.2:0/689125365 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa888116710 con 0x7fa88806edb0 2022-05-13T23:15:42.523 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:42.500+0000 7fa86f7fe700 1 -- 172.21.15.2:0/689125365 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa87801caf0 con 0x7fa88806edb0 2022-05-13T23:15:42.523 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:42.500+0000 7fa88d368700 1 -- 172.21.15.2:0/689125365 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa888116c70 con 0x7fa88806edb0 2022-05-13T23:15:42.523 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:42.501+0000 7fa88d368700 1 -- 172.21.15.2:0/689125365 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa888108fc0 con 0x7fa88806edb0 2022-05-13T23:15:42.523 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:42.503+0000 7fa86f7fe700 1 -- 172.21.15.2:0/689125365 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 16) v1 ==== 125136+0+0 (secure 0 0 0) 0x7fa8780097f0 con 0x7fa88806edb0 2022-05-13T23:15:42.524 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:42.505+0000 7fa86f7fe700 1 --2- 172.21.15.2:0/689125365 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7fa870096d60 0x7fa870099210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:15:42.524 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:42.505+0000 7fa86f7fe700 1 -- 172.21.15.2:0/689125365 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8120+0+0 (secure 0 0 0) 0x7fa8780c0d20 con 0x7fa88806edb0 2022-05-13T23:15:42.524 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:42.505+0000 7fa8867fc700 1 --2- 172.21.15.2:0/689125365 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7fa870096d60 0x7fa870099210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:15:42.524 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:42.506+0000 7fa8867fc700 1 --2- 172.21.15.2:0/689125365 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7fa870096d60 0x7fa870099210 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fa87c007bf0 tx=0x7fa87c00b040 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-05-13T23:15:42.525 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:42.507+0000 7fa86f7fe700 1 -- 172.21.15.2:0/689125365 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+165215 (secure 0 0 0) 0x7fa87808e060 con 0x7fa88806edb0 2022-05-13T23:15:42.572 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:15:42 smithi002 ceph-mon[71816]: pgmap v333: 97 pgs: 97 active+clean; 584 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2022-05-13T23:15:42.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:15:42 smithi115 ceph-mon[78593]: pgmap v333: 97 pgs: 97 active+clean; 584 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2022-05-13T23:15:42.708 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:42.705+0000 7fa88d368700 1 -- 172.21.15.2:0/689125365 --> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""]}) v1 -- 0x7fa88802a500 con 0x7fa870096d60 2022-05-13T23:15:42.708 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:42.706+0000 7fa86f7fe700 1 -- 172.21.15.2:0/689125365 <== mgr.14148 v2:172.21.15.2:6800/2788339627 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+191 (secure 0 0 0) 0x7fa88802a500 con 0x7fa870096d60 2022-05-13T23:15:42.709 INFO:teuthology.orchestra.run.smithi002.stdout:HOST ADDR LABELS STATUS 2022-05-13T23:15:42.709 INFO:teuthology.orchestra.run.smithi002.stdout:smithi002 172.21.15.2 2022-05-13T23:15:42.710 INFO:teuthology.orchestra.run.smithi002.stdout:smithi047 172.21.15.47 2022-05-13T23:15:42.710 INFO:teuthology.orchestra.run.smithi002.stdout:smithi115 172.21.15.115 2022-05-13T23:15:42.710 INFO:teuthology.orchestra.run.smithi002.stdout:3 hosts in cluster 2022-05-13T23:15:42.712 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:42.709+0000 7fa88d368700 1 -- 172.21.15.2:0/689125365 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7fa870096d60 msgr2=0x7fa870099210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:15:42.713 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:42.709+0000 7fa88d368700 1 --2- 172.21.15.2:0/689125365 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7fa870096d60 0x7fa870099210 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fa87c007bf0 tx=0x7fa87c00b040 comp rx=0 tx=0).stop 2022-05-13T23:15:42.713 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:42.709+0000 7fa88d368700 1 -- 172.21.15.2:0/689125365 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa88806edb0 msgr2=0x7fa88810e650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:15:42.713 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:42.709+0000 7fa88d368700 1 --2- 172.21.15.2:0/689125365 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa88806edb0 0x7fa88810e650 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fa878008e80 tx=0x7fa878009cb0 comp rx=0 tx=0).stop 2022-05-13T23:15:42.713 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:42.710+0000 7fa88d368700 1 -- 172.21.15.2:0/689125365 shutdown_connections 2022-05-13T23:15:42.714 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:42.710+0000 7fa88d368700 1 --2- 172.21.15.2:0/689125365 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fa888103580 0x7fa888115c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:42.714 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:42.710+0000 7fa88d368700 1 --2- 172.21.15.2:0/689125365 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7fa870096d60 0x7fa870099210 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:42.714 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:42.710+0000 7fa88d368700 1 --2- 172.21.15.2:0/689125365 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa88806edb0 0x7fa88810e650 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:42.714 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:42.710+0000 7fa88d368700 1 --2- 172.21.15.2:0/689125365 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa888070370 0x7fa88810eb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:42.715 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:42.710+0000 7fa88d368700 1 -- 172.21.15.2:0/689125365 >> 172.21.15.2:0/689125365 conn(0x7fa8880f6e00 msgr2=0x7fa8880f91e0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-05-13T23:15:42.715 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:42.710+0000 7fa88d368700 1 -- 172.21.15.2:0/689125365 shutdown_connections 2022-05-13T23:15:42.715 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:42.710+0000 7fa88d368700 1 -- 172.21.15.2:0/689125365 wait complete. 2022-05-13T23:15:42.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:15:42 smithi047 ceph-mon[77147]: pgmap v333: 97 pgs: 97 active+clean; 584 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2022-05-13T23:15:43.348 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:23b0658391ce94ca40799469eec00c28b298816c shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 16cf5f88-d311-11ec-be40-001a4aab830c -- bash -c 'ceph orch device ls' 2022-05-13T23:15:44.021 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/16cf5f88-d311-11ec-be40-001a4aab830c/mon.a/config 2022-05-13T23:15:44.584 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:15:44 smithi002 ceph-mon[71816]: from='client.14940 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2022-05-13T23:15:44.584 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:15:44 smithi002 ceph-mon[71816]: pgmap v334: 97 pgs: 97 active+clean; 584 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2022-05-13T23:15:44.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:15:44 smithi115 ceph-mon[78593]: from='client.14940 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2022-05-13T23:15:44.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:15:44 smithi115 ceph-mon[78593]: pgmap v334: 97 pgs: 97 active+clean; 584 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2022-05-13T23:15:44.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:15:44 smithi047 ceph-mon[77147]: from='client.14940 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2022-05-13T23:15:44.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:15:44 smithi047 ceph-mon[77147]: pgmap v334: 97 pgs: 97 active+clean; 584 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2022-05-13T23:15:45.839 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:45.836+0000 7fd09b80f700 1 -- 172.21.15.2:0/2624067670 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd0940fc8a0 msgr2=0x7fd0940fccf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:15:45.839 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:45.836+0000 7fd09b80f700 1 --2- 172.21.15.2:0/2624067670 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd0940fc8a0 0x7fd0940fccf0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fd084004870 tx=0x7fd084009cb0 comp rx=0 tx=0).stop 2022-05-13T23:15:45.839 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:45.836+0000 7fd09b80f700 1 -- 172.21.15.2:0/2624067670 shutdown_connections 2022-05-13T23:15:45.840 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:45.836+0000 7fd09b80f700 1 --2- 172.21.15.2:0/2624067670 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fd0940fd230 0x7fd094103af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:45.840 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:45.836+0000 7fd09b80f700 1 --2- 172.21.15.2:0/2624067670 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd0940fb600 0x7fd0940fba10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:45.840 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:45.836+0000 7fd09b80f700 1 --2- 172.21.15.2:0/2624067670 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd0940fc8a0 0x7fd0940fccf0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:45.841 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:45.836+0000 7fd09b80f700 1 -- 172.21.15.2:0/2624067670 >> 172.21.15.2:0/2624067670 conn(0x7fd0940f6df0 msgr2=0x7fd0940f9220 unknown :-1 s=STATE_NONE l=0).mark_down 2022-05-13T23:15:45.841 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:45.836+0000 7fd09b80f700 1 -- 172.21.15.2:0/2624067670 shutdown_connections 2022-05-13T23:15:45.841 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:45.836+0000 7fd09b80f700 1 -- 172.21.15.2:0/2624067670 wait complete. 2022-05-13T23:15:45.841 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:45.837+0000 7fd09b80f700 1 Processor -- start 2022-05-13T23:15:45.842 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:45.838+0000 7fd09b80f700 1 -- start start 2022-05-13T23:15:45.843 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:45.838+0000 7fd09b80f700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fd0940fb600 0x7fd09410e600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:15:45.844 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:45.838+0000 7fd09b80f700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd0940fc8a0 0x7fd09410eb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:15:45.844 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:45.838+0000 7fd09b80f700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd0940fd230 0x7fd094115bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:15:45.844 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:45.838+0000 7fd09b80f700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd094116110 con 0x7fd0940fd230 2022-05-13T23:15:45.844 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:45.838+0000 7fd09b80f700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd094116280 con 0x7fd0940fc8a0 2022-05-13T23:15:45.844 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:45.838+0000 7fd09b80f700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd094116440 con 0x7fd0940fb600 2022-05-13T23:15:45.845 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:45.838+0000 7fd0995ab700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fd0940fb600 0x7fd09410e600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:15:45.845 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:45.838+0000 7fd098daa700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd0940fc8a0 0x7fd09410eb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:15:45.845 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:45.838+0000 7fd0995ab700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fd0940fb600 0x7fd09410e600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.2:43312/0 (socket says 172.21.15.2:43312) 2022-05-13T23:15:45.845 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:45.838+0000 7fd0995ab700 1 -- 172.21.15.2:0/3738145149 learned_addr learned my addr 172.21.15.2:0/3738145149 (peer_addr_for_me v2:172.21.15.2:0/0) 2022-05-13T23:15:45.846 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:45.838+0000 7fd099dac700 1 --2- 172.21.15.2:0/3738145149 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd0940fd230 0x7fd094115bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:15:45.846 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:45.839+0000 7fd098daa700 1 -- 172.21.15.2:0/3738145149 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fd0940fb600 msgr2=0x7fd09410e600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:15:45.846 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:45.839+0000 7fd098daa700 1 --2- 172.21.15.2:0/3738145149 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fd0940fb600 0x7fd09410e600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:45.846 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:45.839+0000 7fd098daa700 1 -- 172.21.15.2:0/3738145149 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd0940fd230 msgr2=0x7fd094115bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:15:45.846 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:45.839+0000 7fd098daa700 1 --2- 172.21.15.2:0/3738145149 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd0940fd230 0x7fd094115bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:45.847 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:45.839+0000 7fd098daa700 1 -- 172.21.15.2:0/3738145149 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd08c0097c0 con 0x7fd0940fc8a0 2022-05-13T23:15:45.847 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:45.839+0000 7fd098daa700 1 --2- 172.21.15.2:0/3738145149 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd0940fc8a0 0x7fd09410eb40 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fd084004710 tx=0x7fd084009cb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-05-13T23:15:45.847 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:45.839+0000 7fd089ffb700 1 -- 172.21.15.2:0/3738145149 <== mon.1 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd084020070 con 0x7fd0940fc8a0 2022-05-13T23:15:45.847 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:45.839+0000 7fd09b80f700 1 -- 172.21.15.2:0/3738145149 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd084005040 con 0x7fd0940fc8a0 2022-05-13T23:15:45.847 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:45.840+0000 7fd089ffb700 1 -- 172.21.15.2:0/3738145149 <== mon.1 v2:172.21.15.47:3300/0 2 ==== config(26 keys) v1 ==== 1087+0+0 (secure 0 0 0) 0x7fd084014e40 con 0x7fd0940fc8a0 2022-05-13T23:15:45.848 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:45.840+0000 7fd089ffb700 1 -- 172.21.15.2:0/3738145149 <== mon.1 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd08400e040 con 0x7fd0940fc8a0 2022-05-13T23:15:45.848 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:45.840+0000 7fd09b80f700 1 -- 172.21.15.2:0/3738145149 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd094116a10 con 0x7fd0940fc8a0 2022-05-13T23:15:45.849 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:45.841+0000 7fd09b80f700 1 -- 172.21.15.2:0/3738145149 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd094106e50 con 0x7fd0940fc8a0 2022-05-13T23:15:45.851 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:45.843+0000 7fd089ffb700 1 -- 172.21.15.2:0/3738145149 <== mon.1 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 16) v1 ==== 125136+0+0 (secure 0 0 0) 0x7fd084009240 con 0x7fd0940fc8a0 2022-05-13T23:15:45.851 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:45.844+0000 7fd089ffb700 1 --2- 172.21.15.2:0/3738145149 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7fd080096ce0 0x7fd080099190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:15:45.851 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:45.844+0000 7fd089ffb700 1 -- 172.21.15.2:0/3738145149 <== mon.1 v2:172.21.15.47:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8120+0+0 (secure 0 0 0) 0x7fd0840c0880 con 0x7fd0940fc8a0 2022-05-13T23:15:45.852 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:45.844+0000 7fd0995ab700 1 --2- 172.21.15.2:0/3738145149 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7fd080096ce0 0x7fd080099190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:15:45.852 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:45.845+0000 7fd0995ab700 1 --2- 172.21.15.2:0/3738145149 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7fd080096ce0 0x7fd080099190 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fd0900041d0 tx=0x7fd09000b040 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-05-13T23:15:45.852 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:45.846+0000 7fd089ffb700 1 -- 172.21.15.2:0/3738145149 <== mon.1 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+165215 (secure 0 0 0) 0x7fd08408db30 con 0x7fd0940fc8a0 2022-05-13T23:15:46.046 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:46.045+0000 7fd09b80f700 1 -- 172.21.15.2:0/3738145149 --> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] -- mgr_command(tid 0: {"prefix": "orch device ls", "target": ["mon-mgr", ""]}) v1 -- 0x7fd09402a460 con 0x7fd080096ce0 2022-05-13T23:15:46.047 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:46.046+0000 7fd089ffb700 1 -- 172.21.15.2:0/3738145149 <== mgr.14148 v2:172.21.15.2:6800/2788339627 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+492 (secure 0 0 0) 0x7fd09402a460 con 0x7fd080096ce0 2022-05-13T23:15:46.048 INFO:teuthology.orchestra.run.smithi002.stdout:HOST PATH TYPE DEVICE ID SIZE AVAILABLE REFRESHED REJECT REASONS 2022-05-13T23:15:46.048 INFO:teuthology.orchestra.run.smithi002.stdout:smithi002 /dev/nvme0n1 ssd INTEL_SSDPED1D480GA_PHMB751300EV480DGN 480G 3m ago LVM detected, locked 2022-05-13T23:15:46.048 INFO:teuthology.orchestra.run.smithi002.stdout:smithi047 /dev/nvme0n1 ssd INTEL_SSDPED1D480GA_PHMB75130022480DGN 480G 2m ago LVM detected, locked 2022-05-13T23:15:46.049 INFO:teuthology.orchestra.run.smithi002.stdout:smithi115 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_PHFT620400UU400BGN 400G 2m ago LVM detected, locked 2022-05-13T23:15:46.050 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:46.049+0000 7fd09b80f700 1 -- 172.21.15.2:0/3738145149 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7fd080096ce0 msgr2=0x7fd080099190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:15:46.051 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:46.049+0000 7fd09b80f700 1 --2- 172.21.15.2:0/3738145149 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7fd080096ce0 0x7fd080099190 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fd0900041d0 tx=0x7fd09000b040 comp rx=0 tx=0).stop 2022-05-13T23:15:46.051 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:46.049+0000 7fd09b80f700 1 -- 172.21.15.2:0/3738145149 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd0940fc8a0 msgr2=0x7fd09410eb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:15:46.051 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:46.049+0000 7fd09b80f700 1 --2- 172.21.15.2:0/3738145149 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd0940fc8a0 0x7fd09410eb40 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fd084004710 tx=0x7fd084009cb0 comp rx=0 tx=0).stop 2022-05-13T23:15:46.051 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:46.049+0000 7fd09b80f700 1 -- 172.21.15.2:0/3738145149 shutdown_connections 2022-05-13T23:15:46.052 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:46.049+0000 7fd09b80f700 1 --2- 172.21.15.2:0/3738145149 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7fd080096ce0 0x7fd080099190 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:46.052 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:46.049+0000 7fd09b80f700 1 --2- 172.21.15.2:0/3738145149 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd0940fd230 0x7fd094115bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:46.052 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:46.049+0000 7fd09b80f700 1 --2- 172.21.15.2:0/3738145149 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fd0940fb600 0x7fd09410e600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:46.052 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:46.049+0000 7fd09b80f700 1 --2- 172.21.15.2:0/3738145149 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd0940fc8a0 0x7fd09410eb40 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:46.053 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:46.049+0000 7fd09b80f700 1 -- 172.21.15.2:0/3738145149 >> 172.21.15.2:0/3738145149 conn(0x7fd0940f6df0 msgr2=0x7fd094102310 unknown :-1 s=STATE_NONE l=0).mark_down 2022-05-13T23:15:46.053 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:46.049+0000 7fd09b80f700 1 -- 172.21.15.2:0/3738145149 shutdown_connections 2022-05-13T23:15:46.053 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:46.049+0000 7fd09b80f700 1 -- 172.21.15.2:0/3738145149 wait complete. 2022-05-13T23:15:46.602 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:15:46 smithi002 ceph-mon[71816]: pgmap v335: 97 pgs: 97 active+clean; 584 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 11 op/s 2022-05-13T23:15:46.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:15:46 smithi115 ceph-mon[78593]: pgmap v335: 97 pgs: 97 active+clean; 584 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 11 op/s 2022-05-13T23:15:46.693 INFO:teuthology.run_tasks:Running task cephadm.shell... 2022-05-13T23:15:46.708 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi002.front.sepia.ceph.com 2022-05-13T23:15:46.708 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:23b0658391ce94ca40799469eec00c28b298816c shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 16cf5f88-d311-11ec-be40-001a4aab830c -- bash -c 'ceph fs dump' 2022-05-13T23:15:46.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:15:46 smithi047 ceph-mon[77147]: pgmap v335: 97 pgs: 97 active+clean; 584 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 11 op/s 2022-05-13T23:15:47.384 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/16cf5f88-d311-11ec-be40-001a4aab830c/mon.a/config 2022-05-13T23:15:47.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:15:47 smithi115 ceph-mon[78593]: from='client.25039 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2022-05-13T23:15:47.712 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:15:47 smithi002 ceph-mon[71816]: from='client.25039 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2022-05-13T23:15:47.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:15:47 smithi047 ceph-mon[77147]: from='client.25039 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2022-05-13T23:15:48.531 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:15:48 smithi002 ceph-mon[71816]: pgmap v336: 97 pgs: 97 active+clean; 584 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2022-05-13T23:15:48.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:15:48 smithi115 ceph-mon[78593]: pgmap v336: 97 pgs: 97 active+clean; 584 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2022-05-13T23:15:48.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:15:48 smithi047 ceph-mon[77147]: pgmap v336: 97 pgs: 97 active+clean; 584 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2022-05-13T23:15:49.206 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:49.204+0000 7f557b631700 1 -- 172.21.15.2:0/2488876468 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5574103580 msgr2=0x7f5574105960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:15:49.206 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:49.204+0000 7f557b631700 1 --2- 172.21.15.2:0/2488876468 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5574103580 0x7f5574105960 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f556c009be0 tx=0x7f556c004d10 comp rx=0 tx=0).stop 2022-05-13T23:15:49.206 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:49.204+0000 7f557b631700 1 -- 172.21.15.2:0/2488876468 shutdown_connections 2022-05-13T23:15:49.207 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:49.204+0000 7f557b631700 1 --2- 172.21.15.2:0/2488876468 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5574103580 0x7f5574105960 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:49.207 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:49.204+0000 7f557b631700 1 --2- 172.21.15.2:0/2488876468 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5574070370 0x7f557406e870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:49.207 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:49.204+0000 7f557b631700 1 --2- 172.21.15.2:0/2488876468 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f557406edb0 0x7f557406f220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:49.207 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:49.204+0000 7f557b631700 1 -- 172.21.15.2:0/2488876468 >> 172.21.15.2:0/2488876468 conn(0x7f55740f6e00 msgr2=0x7f55740f9210 unknown :-1 s=STATE_NONE l=0).mark_down 2022-05-13T23:15:49.207 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:49.204+0000 7f557b631700 1 -- 172.21.15.2:0/2488876468 shutdown_connections 2022-05-13T23:15:49.208 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:49.204+0000 7f557b631700 1 -- 172.21.15.2:0/2488876468 wait complete. 2022-05-13T23:15:49.210 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:49.205+0000 7f557b631700 1 Processor -- start 2022-05-13T23:15:49.211 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:49.205+0000 7f557b631700 1 -- start start 2022-05-13T23:15:49.211 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:49.206+0000 7f557b631700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f557406edb0 0x7f557410e3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:15:49.211 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:49.206+0000 7f557b631700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5574070370 0x7f557410e8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:15:49.211 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:49.206+0000 7f557b631700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f5574103580 0x7f5574115970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:15:49.212 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:49.206+0000 7f557b631700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5574115eb0 con 0x7f557406edb0 2022-05-13T23:15:49.214 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:49.206+0000 7f557b631700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5574116020 con 0x7f5574070370 2022-05-13T23:15:49.214 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:49.206+0000 7f557b631700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55741161e0 con 0x7f5574103580 2022-05-13T23:15:49.214 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:49.206+0000 7f55793cd700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f557406edb0 0x7f557410e3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:15:49.215 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:49.206+0000 7f5578bcc700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5574070370 0x7f557410e8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:15:49.215 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:49.206+0000 7f5578bcc700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5574070370 0x7f557410e8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.2:56074/0 (socket says 172.21.15.2:56074) 2022-05-13T23:15:49.215 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:49.206+0000 7f5578bcc700 1 -- 172.21.15.2:0/1253488647 learned_addr learned my addr 172.21.15.2:0/1253488647 (peer_addr_for_me v2:172.21.15.2:0/0) 2022-05-13T23:15:49.215 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:49.206+0000 7f5579bce700 1 --2- 172.21.15.2:0/1253488647 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f5574103580 0x7f5574115970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:15:49.216 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:49.206+0000 7f5578bcc700 1 -- 172.21.15.2:0/1253488647 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f5574103580 msgr2=0x7f5574115970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:15:49.216 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:49.207+0000 7f5578bcc700 1 --2- 172.21.15.2:0/1253488647 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f5574103580 0x7f5574115970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:49.216 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:49.207+0000 7f5578bcc700 1 -- 172.21.15.2:0/1253488647 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f557406edb0 msgr2=0x7f557410e3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:15:49.216 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:49.207+0000 7f5578bcc700 1 --2- 172.21.15.2:0/1253488647 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f557406edb0 0x7f557410e3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:49.216 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:49.207+0000 7f5578bcc700 1 -- 172.21.15.2:0/1253488647 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f556c0097c0 con 0x7f5574070370 2022-05-13T23:15:49.217 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:49.207+0000 7f55793cd700 1 --2- 172.21.15.2:0/1253488647 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f557406edb0 0x7f557410e3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-05-13T23:15:49.217 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:49.207+0000 7f5578bcc700 1 --2- 172.21.15.2:0/1253488647 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5574070370 0x7f557410e8e0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f556400e970 tx=0x7f556400b040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-05-13T23:15:49.217 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:49.207+0000 7f5569ffb700 1 -- 172.21.15.2:0/1253488647 <== mon.1 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5564007b40 con 0x7f5574070370 2022-05-13T23:15:49.217 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:49.207+0000 7f557b631700 1 -- 172.21.15.2:0/1253488647 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f55741164c0 con 0x7f5574070370 2022-05-13T23:15:49.217 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:49.207+0000 7f5569ffb700 1 -- 172.21.15.2:0/1253488647 <== mon.1 v2:172.21.15.47:3300/0 2 ==== config(26 keys) v1 ==== 1087+0+0 (secure 0 0 0) 0x7f556400f040 con 0x7f5574070370 2022-05-13T23:15:49.218 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:49.207+0000 7f5569ffb700 1 -- 172.21.15.2:0/1253488647 <== mon.1 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5564014950 con 0x7f5574070370 2022-05-13T23:15:49.218 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:49.207+0000 7f557b631700 1 -- 172.21.15.2:0/1253488647 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5574116a80 con 0x7f5574070370 2022-05-13T23:15:49.218 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:49.208+0000 7f557b631700 1 -- 172.21.15.2:0/1253488647 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5574108fd0 con 0x7f5574070370 2022-05-13T23:15:49.220 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:49.211+0000 7f5569ffb700 1 -- 172.21.15.2:0/1253488647 <== mon.1 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 16) v1 ==== 125136+0+0 (secure 0 0 0) 0x7f556401b030 con 0x7f5574070370 2022-05-13T23:15:49.220 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:49.212+0000 7f5569ffb700 1 --2- 172.21.15.2:0/1253488647 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f5560096ce0 0x7f5560099190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:15:49.220 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:49.212+0000 7f5569ffb700 1 -- 172.21.15.2:0/1253488647 <== mon.1 v2:172.21.15.47:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8120+0+0 (secure 0 0 0) 0x7f55640bee60 con 0x7f5574070370 2022-05-13T23:15:49.220 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:49.212+0000 7f55793cd700 1 --2- 172.21.15.2:0/1253488647 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f5560096ce0 0x7f5560099190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:15:49.221 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:49.213+0000 7f55793cd700 1 --2- 172.21.15.2:0/1253488647 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f5560096ce0 0x7f5560099190 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f5570010420 tx=0x7f557000b040 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-05-13T23:15:49.221 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:49.214+0000 7f5569ffb700 1 -- 172.21.15.2:0/1253488647 <== mon.1 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+165215 (secure 0 0 0) 0x7f556408c110 con 0x7f5574070370 2022-05-13T23:15:49.451 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:49.450+0000 7f557b631700 1 -- 172.21.15.2:0/1253488647 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f5574101b50 con 0x7f5574070370 2022-05-13T23:15:49.452 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:49.451+0000 7f5569ffb700 1 -- 172.21.15.2:0/1253488647 <== mon.1 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 26 v26) v1 ==== 76+0+2869 (secure 0 0 0) 0x7f5564019070 con 0x7f5574070370 2022-05-13T23:15:49.453 INFO:teuthology.orchestra.run.smithi002.stdout:e26 2022-05-13T23:15:49.453 INFO:teuthology.orchestra.run.smithi002.stdout:enable_multiple, ever_enabled_multiple: 1,1 2022-05-13T23:15:49.453 INFO:teuthology.orchestra.run.smithi002.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2022-05-13T23:15:49.453 INFO:teuthology.orchestra.run.smithi002.stdout:legacy client fscid: 1 2022-05-13T23:15:49.454 INFO:teuthology.orchestra.run.smithi002.stdout: 2022-05-13T23:15:49.454 INFO:teuthology.orchestra.run.smithi002.stdout:Filesystem 'cephfs' (1) 2022-05-13T23:15:49.454 INFO:teuthology.orchestra.run.smithi002.stdout:fs_name cephfs 2022-05-13T23:15:49.454 INFO:teuthology.orchestra.run.smithi002.stdout:epoch 26 2022-05-13T23:15:49.455 INFO:teuthology.orchestra.run.smithi002.stdout:flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay 2022-05-13T23:15:49.455 INFO:teuthology.orchestra.run.smithi002.stdout:created 2022-05-13T23:14:16.236645+0000 2022-05-13T23:15:49.455 INFO:teuthology.orchestra.run.smithi002.stdout:modified 2022-05-13T23:14:23.309447+0000 2022-05-13T23:15:49.455 INFO:teuthology.orchestra.run.smithi002.stdout:tableserver 0 2022-05-13T23:15:49.455 INFO:teuthology.orchestra.run.smithi002.stdout:root 0 2022-05-13T23:15:49.456 INFO:teuthology.orchestra.run.smithi002.stdout:session_timeout 300 2022-05-13T23:15:49.456 INFO:teuthology.orchestra.run.smithi002.stdout:session_autoclose 300 2022-05-13T23:15:49.456 INFO:teuthology.orchestra.run.smithi002.stdout:max_file_size 1099511627776 2022-05-13T23:15:49.456 INFO:teuthology.orchestra.run.smithi002.stdout:required_client_features {} 2022-05-13T23:15:49.457 INFO:teuthology.orchestra.run.smithi002.stdout:last_failure 0 2022-05-13T23:15:49.457 INFO:teuthology.orchestra.run.smithi002.stdout:last_failure_osd_epoch 0 2022-05-13T23:15:49.457 INFO:teuthology.orchestra.run.smithi002.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2022-05-13T23:15:49.457 INFO:teuthology.orchestra.run.smithi002.stdout:max_mds 5 2022-05-13T23:15:49.457 INFO:teuthology.orchestra.run.smithi002.stdout:in 0,1,2,3,4 2022-05-13T23:15:49.458 INFO:teuthology.orchestra.run.smithi002.stdout:up {0=24611,1=24599,2=24587,3=24575,4=24547} 2022-05-13T23:15:49.458 INFO:teuthology.orchestra.run.smithi002.stdout:failed 2022-05-13T23:15:49.458 INFO:teuthology.orchestra.run.smithi002.stdout:damaged 2022-05-13T23:15:49.458 INFO:teuthology.orchestra.run.smithi002.stdout:stopped 2022-05-13T23:15:49.458 INFO:teuthology.orchestra.run.smithi002.stdout:data_pools [3] 2022-05-13T23:15:49.459 INFO:teuthology.orchestra.run.smithi002.stdout:metadata_pool 2 2022-05-13T23:15:49.459 INFO:teuthology.orchestra.run.smithi002.stdout:inline_data disabled 2022-05-13T23:15:49.459 INFO:teuthology.orchestra.run.smithi002.stdout:balancer 2022-05-13T23:15:49.459 INFO:teuthology.orchestra.run.smithi002.stdout:standby_count_wanted 0 2022-05-13T23:15:49.459 INFO:teuthology.orchestra.run.smithi002.stdout:[mds.l{0:24611} state up:active seq 3 addr [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] compat {c=[1],r=[1],i=[7ff]}] 2022-05-13T23:15:49.460 INFO:teuthology.orchestra.run.smithi002.stdout:[mds.h{0:24535} state up:standby-replay seq 1 addr [v2:172.21.15.47:6836/4072863000,v1:172.21.15.47:6837/4072863000] compat {c=[1],r=[1],i=[7ff]}] 2022-05-13T23:15:49.460 INFO:teuthology.orchestra.run.smithi002.stdout:[mds.i{1:24599} state up:active seq 4 addr [v2:172.21.15.115:6836/2940957881,v1:172.21.15.115:6837/2940957881] compat {c=[1],r=[1],i=[7ff]}] 2022-05-13T23:15:49.460 INFO:teuthology.orchestra.run.smithi002.stdout:[mds.j{1:24530} state up:standby-replay seq 1 addr [v2:172.21.15.2:6840/522559555,v1:172.21.15.2:6841/522559555] compat {c=[1],r=[1],i=[7ff]}] 2022-05-13T23:15:49.460 INFO:teuthology.orchestra.run.smithi002.stdout:[mds.f{2:24587} state up:active seq 5 addr [v2:172.21.15.115:6834/71062276,v1:172.21.15.115:6835/71062276] compat {c=[1],r=[1],i=[7ff]}] 2022-05-13T23:15:49.460 INFO:teuthology.orchestra.run.smithi002.stdout:[mds.e{2:24523} state up:standby-replay seq 1 addr [v2:172.21.15.47:6834/4075838054,v1:172.21.15.47:6835/4075838054] compat {c=[1],r=[1],i=[7ff]}] 2022-05-13T23:15:49.461 INFO:teuthology.orchestra.run.smithi002.stdout:[mds.c{3:24575} state up:active seq 6 addr [v2:172.21.15.115:6832/381933929,v1:172.21.15.115:6833/381933929] compat {c=[1],r=[1],i=[7ff]}] 2022-05-13T23:15:49.461 INFO:teuthology.orchestra.run.smithi002.stdout:[mds.b{3:24511} state up:standby-replay seq 1 addr [v2:172.21.15.47:6832/2783249369,v1:172.21.15.47:6833/2783249369] compat {c=[1],r=[1],i=[7ff]}] 2022-05-13T23:15:49.461 INFO:teuthology.orchestra.run.smithi002.stdout:[mds.k{4:24547} state up:active seq 7 addr [v2:172.21.15.47:6838/761355251,v1:172.21.15.47:6839/761355251] compat {c=[1],r=[1],i=[7ff]}] 2022-05-13T23:15:49.461 INFO:teuthology.orchestra.run.smithi002.stdout:[mds.d{4:24494} state up:standby-replay seq 1 addr [v2:172.21.15.2:6836/3754333215,v1:172.21.15.2:6837/3754333215] compat {c=[1],r=[1],i=[7ff]}] 2022-05-13T23:15:49.462 INFO:teuthology.orchestra.run.smithi002.stdout: 2022-05-13T23:15:49.462 INFO:teuthology.orchestra.run.smithi002.stdout: 2022-05-13T23:15:49.462 INFO:teuthology.orchestra.run.smithi002.stdout:Standby daemons: 2022-05-13T23:15:49.462 INFO:teuthology.orchestra.run.smithi002.stdout: 2022-05-13T23:15:49.462 INFO:teuthology.orchestra.run.smithi002.stdout:[mds.a{-1:24460} state up:standby seq 1 addr [v2:172.21.15.2:6834/1341618941,v1:172.21.15.2:6835/1341618941] compat {c=[1],r=[1],i=[7ff]}] 2022-05-13T23:15:49.463 INFO:teuthology.orchestra.run.smithi002.stdout:[mds.g{-1:24490} state up:standby seq 1 addr [v2:172.21.15.2:6838/1347050951,v1:172.21.15.2:6839/1347050951] compat {c=[1],r=[1],i=[7ff]}] 2022-05-13T23:15:49.464 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:49.454+0000 7f557b631700 1 -- 172.21.15.2:0/1253488647 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f5560096ce0 msgr2=0x7f5560099190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:15:49.464 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:49.454+0000 7f557b631700 1 --2- 172.21.15.2:0/1253488647 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f5560096ce0 0x7f5560099190 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f5570010420 tx=0x7f557000b040 comp rx=0 tx=0).stop 2022-05-13T23:15:49.464 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:49.454+0000 7f557b631700 1 -- 172.21.15.2:0/1253488647 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5574070370 msgr2=0x7f557410e8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:15:49.464 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:49.454+0000 7f557b631700 1 --2- 172.21.15.2:0/1253488647 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5574070370 0x7f557410e8e0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f556400e970 tx=0x7f556400b040 comp rx=0 tx=0).stop 2022-05-13T23:15:49.465 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:49.454+0000 7f557b631700 1 -- 172.21.15.2:0/1253488647 shutdown_connections 2022-05-13T23:15:49.465 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:49.454+0000 7f557b631700 1 --2- 172.21.15.2:0/1253488647 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f5574103580 0x7f5574115970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:49.465 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:49.454+0000 7f557b631700 1 --2- 172.21.15.2:0/1253488647 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f5560096ce0 0x7f5560099190 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:49.465 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:49.454+0000 7f557b631700 1 --2- 172.21.15.2:0/1253488647 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f557406edb0 0x7f557410e3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:49.466 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:49.454+0000 7f557b631700 1 --2- 172.21.15.2:0/1253488647 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5574070370 0x7f557410e8e0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:49.466 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:49.454+0000 7f557b631700 1 -- 172.21.15.2:0/1253488647 >> 172.21.15.2:0/1253488647 conn(0x7f55740f6e00 msgr2=0x7f55741056b0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-05-13T23:15:49.466 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:49.454+0000 7f557b631700 1 -- 172.21.15.2:0/1253488647 shutdown_connections 2022-05-13T23:15:49.466 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:49.454+0000 7f557b631700 1 -- 172.21.15.2:0/1253488647 wait complete. 2022-05-13T23:15:49.466 INFO:teuthology.orchestra.run.smithi002.stderr:dumped fsmap epoch 26 2022-05-13T23:15:50.047 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:23b0658391ce94ca40799469eec00c28b298816c shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 16cf5f88-d311-11ec-be40-001a4aab830c -- bash -c 'ceph osd dump' 2022-05-13T23:15:50.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:15:50 smithi115 ceph-mon[78593]: pgmap v337: 97 pgs: 97 active+clean; 584 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-05-13T23:15:50.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:15:50 smithi115 ceph-mon[78593]: from='client.? 172.21.15.2:0/1253488647' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2022-05-13T23:15:50.722 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/16cf5f88-d311-11ec-be40-001a4aab830c/mon.a/config 2022-05-13T23:15:50.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:15:50 smithi047 ceph-mon[77147]: pgmap v337: 97 pgs: 97 active+clean; 584 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-05-13T23:15:50.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:15:50 smithi047 ceph-mon[77147]: from='client.? 172.21.15.2:0/1253488647' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2022-05-13T23:15:50.822 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:15:50 smithi002 ceph-mon[71816]: pgmap v337: 97 pgs: 97 active+clean; 584 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-05-13T23:15:50.822 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:15:50 smithi002 ceph-mon[71816]: from='client.? 172.21.15.2:0/1253488647' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2022-05-13T23:15:52.545 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:52.537+0000 7fec06e12700 1 -- 172.21.15.2:0/334011627 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fec000fea90 msgr2=0x7fec000feee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:15:52.546 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:52.537+0000 7fec06e12700 1 --2- 172.21.15.2:0/334011627 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fec000fea90 0x7fec000feee0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7febec004870 tx=0x7febec009cb0 comp rx=0 tx=0).stop 2022-05-13T23:15:52.547 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:52.537+0000 7fec06e12700 1 -- 172.21.15.2:0/334011627 shutdown_connections 2022-05-13T23:15:52.547 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:52.537+0000 7fec06e12700 1 --2- 172.21.15.2:0/334011627 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fec000ff420 0x7fec00105ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:52.548 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:52.537+0000 7fec06e12700 1 --2- 172.21.15.2:0/334011627 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fec000fd7f0 0x7fec000fdc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:52.548 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:52.537+0000 7fec06e12700 1 --2- 172.21.15.2:0/334011627 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fec000fea90 0x7fec000feee0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:52.548 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:52.537+0000 7fec06e12700 1 -- 172.21.15.2:0/334011627 >> 172.21.15.2:0/334011627 conn(0x7fec000f9030 msgr2=0x7fec000fb410 unknown :-1 s=STATE_NONE l=0).mark_down 2022-05-13T23:15:52.549 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:52.537+0000 7fec06e12700 1 -- 172.21.15.2:0/334011627 shutdown_connections 2022-05-13T23:15:52.549 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:52.537+0000 7fec06e12700 1 -- 172.21.15.2:0/334011627 wait complete. 2022-05-13T23:15:52.549 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:52.538+0000 7fec06e12700 1 Processor -- start 2022-05-13T23:15:52.551 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:52.538+0000 7fec06e12700 1 -- start start 2022-05-13T23:15:52.551 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:52.538+0000 7fec06e12700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fec000fd7f0 0x7fec0010e630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:15:52.551 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:52.538+0000 7fec06e12700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fec000fea90 0x7fec0010eb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:15:52.552 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:52.538+0000 7fec06e12700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fec000ff420 0x7fec00115c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:15:52.552 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:52.538+0000 7fec06e12700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec00116140 con 0x7fec000fea90 2022-05-13T23:15:52.552 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:52.538+0000 7fec06e12700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec001162b0 con 0x7fec000ff420 2022-05-13T23:15:52.552 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:52.538+0000 7fec06e12700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec00116470 con 0x7fec000fd7f0 2022-05-13T23:15:52.552 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:52.539+0000 7febfffff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fec000fea90 0x7fec0010eb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:15:52.553 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:52.539+0000 7fec04bae700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fec000fd7f0 0x7fec0010e630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:15:52.553 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:52.539+0000 7febfffff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fec000fea90 0x7fec0010eb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:53872/0 (socket says 172.21.15.2:53872) 2022-05-13T23:15:52.553 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:52.539+0000 7fec04bae700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fec000fd7f0 0x7fec0010e630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.2:45152/0 (socket says 172.21.15.2:45152) 2022-05-13T23:15:52.553 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:52.539+0000 7febfffff700 1 -- 172.21.15.2:0/3226788168 learned_addr learned my addr 172.21.15.2:0/3226788168 (peer_addr_for_me v2:172.21.15.2:0/0) 2022-05-13T23:15:52.554 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:52.539+0000 7fec053af700 1 --2- 172.21.15.2:0/3226788168 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fec000ff420 0x7fec00115c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:15:52.554 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:52.539+0000 7febfffff700 1 -- 172.21.15.2:0/3226788168 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fec000fd7f0 msgr2=0x7fec0010e630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:15:52.554 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:52.539+0000 7febfffff700 1 --2- 172.21.15.2:0/3226788168 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fec000fd7f0 0x7fec0010e630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:52.554 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:52.539+0000 7febfffff700 1 -- 172.21.15.2:0/3226788168 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fec000ff420 msgr2=0x7fec00115c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:15:52.554 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:52.539+0000 7febfffff700 1 --2- 172.21.15.2:0/3226788168 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fec000ff420 0x7fec00115c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:52.555 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:52.539+0000 7febfffff700 1 -- 172.21.15.2:0/3226788168 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7febec005040 con 0x7fec000fea90 2022-05-13T23:15:52.555 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:52.540+0000 7febfffff700 1 --2- 172.21.15.2:0/3226788168 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fec000fea90 0x7fec0010eb70 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7febec004ee0 tx=0x7febec009cb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-05-13T23:15:52.555 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:52.540+0000 7febfd7fa700 1 -- 172.21.15.2:0/3226788168 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febec00e040 con 0x7fec000fea90 2022-05-13T23:15:52.556 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:52.540+0000 7febfd7fa700 1 -- 172.21.15.2:0/3226788168 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1087+0+0 (secure 0 0 0) 0x7febec014e40 con 0x7fec000fea90 2022-05-13T23:15:52.556 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:52.540+0000 7fec06e12700 1 -- 172.21.15.2:0/3226788168 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fec001166f0 con 0x7fec000fea90 2022-05-13T23:15:52.556 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:52.540+0000 7febfd7fa700 1 -- 172.21.15.2:0/3226788168 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febec01cab0 con 0x7fec000fea90 2022-05-13T23:15:52.556 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:52.540+0000 7fec06e12700 1 -- 172.21.15.2:0/3226788168 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fec00116b70 con 0x7fec000fea90 2022-05-13T23:15:52.556 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:52.541+0000 7fec06e12700 1 -- 172.21.15.2:0/3226788168 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fec00108fd0 con 0x7fec000fea90 2022-05-13T23:15:52.557 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:52.543+0000 7febfd7fa700 1 -- 172.21.15.2:0/3226788168 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 16) v1 ==== 125136+0+0 (secure 0 0 0) 0x7febec009240 con 0x7fec000fea90 2022-05-13T23:15:52.558 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:52.544+0000 7febfd7fa700 1 --2- 172.21.15.2:0/3226788168 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7febf0096ce0 0x7febf0099190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:15:52.558 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:52.544+0000 7febfd7fa700 1 -- 172.21.15.2:0/3226788168 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8120+0+0 (secure 0 0 0) 0x7febec0c0560 con 0x7fec000fea90 2022-05-13T23:15:52.558 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:52.545+0000 7fec04bae700 1 --2- 172.21.15.2:0/3226788168 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7febf0096ce0 0x7febf0099190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:15:52.559 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:52.545+0000 7fec04bae700 1 --2- 172.21.15.2:0/3226788168 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7febf0096ce0 0x7febf0099190 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fec000fe8f0 tx=0x7febf400b040 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-05-13T23:15:52.559 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:52.549+0000 7febfd7fa700 1 -- 172.21.15.2:0/3226788168 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+165215 (secure 0 0 0) 0x7febec08d810 con 0x7fec000fea90 2022-05-13T23:15:52.576 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:15:52 smithi002 ceph-mon[71816]: pgmap v338: 97 pgs: 97 active+clean; 584 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2022-05-13T23:15:52.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:15:52 smithi115 ceph-mon[78593]: pgmap v338: 97 pgs: 97 active+clean; 584 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2022-05-13T23:15:52.742 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:52.740+0000 7fec06e12700 1 -- 172.21.15.2:0/3226788168 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd dump"} v 0) v1 -- 0x7fec00109160 con 0x7fec000fea90 2022-05-13T23:15:52.743 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:52.740+0000 7febfd7fa700 1 -- 172.21.15.2:0/3226788168 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump"}]=0 v85) v1 ==== 56+0+5099 (secure 0 0 0) 0x7febec091e60 con 0x7fec000fea90 2022-05-13T23:15:52.746 INFO:teuthology.orchestra.run.smithi002.stdout:epoch 85 2022-05-13T23:15:52.746 INFO:teuthology.orchestra.run.smithi002.stdout:fsid 16cf5f88-d311-11ec-be40-001a4aab830c 2022-05-13T23:15:52.746 INFO:teuthology.orchestra.run.smithi002.stdout:created 2022-05-13T23:06:00.765453+0000 2022-05-13T23:15:52.747 INFO:teuthology.orchestra.run.smithi002.stdout:modified 2022-05-13T23:15:06.402793+0000 2022-05-13T23:15:52.747 INFO:teuthology.orchestra.run.smithi002.stdout:flags sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit 2022-05-13T23:15:52.747 INFO:teuthology.orchestra.run.smithi002.stdout:crush_version 26 2022-05-13T23:15:52.747 INFO:teuthology.orchestra.run.smithi002.stdout:full_ratio 0.95 2022-05-13T23:15:52.748 INFO:teuthology.orchestra.run.smithi002.stdout:backfillfull_ratio 0.9 2022-05-13T23:15:52.748 INFO:teuthology.orchestra.run.smithi002.stdout:nearfull_ratio 0.85 2022-05-13T23:15:52.749 INFO:teuthology.orchestra.run.smithi002.stdout:require_min_compat_client luminous 2022-05-13T23:15:52.749 INFO:teuthology.orchestra.run.smithi002.stdout:min_compat_client luminous 2022-05-13T23:15:52.749 INFO:teuthology.orchestra.run.smithi002.stdout:require_osd_release quincy 2022-05-13T23:15:52.749 INFO:teuthology.orchestra.run.smithi002.stdout:stretch_mode_enabled false 2022-05-13T23:15:52.750 INFO:teuthology.orchestra.run.smithi002.stdout:pool 1 '.mgr' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode off last_change 21 flags hashpspool stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr 2022-05-13T23:15:52.750 INFO:teuthology.orchestra.run.smithi002.stdout:pool 2 'cephfs_metadata' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 82 flags hashpspool stripe_width 0 pg_autoscale_bias 4 pg_num_min 16 recovery_priority 5 application cephfs 2022-05-13T23:15:52.750 INFO:teuthology.orchestra.run.smithi002.stdout:pool 3 'cephfs_data' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 64 pgp_num 64 autoscale_mode off last_change 83 flags hashpspool stripe_width 0 pg_num_min 64 target_size_ratio 0.9 application cephfs 2022-05-13T23:15:52.750 INFO:teuthology.orchestra.run.smithi002.stdout:max_osd 12 2022-05-13T23:15:52.751 INFO:teuthology.orchestra.run.smithi002.stdout:osd.0 up in weight 1 up_from 8 up_thru 81 down_at 0 last_clean_interval [0,0) [v2:172.21.15.2:6802/1174843704,v1:172.21.15.2:6803/1174843704] [v2:172.21.15.2:6804/1174843704,v1:172.21.15.2:6805/1174843704] exists,up 11e700e8-f3e8-4903-b52c-db354b4395bf 2022-05-13T23:15:52.751 INFO:teuthology.orchestra.run.smithi002.stdout:osd.1 up in weight 1 up_from 13 up_thru 81 down_at 0 last_clean_interval [0,0) [v2:172.21.15.47:6800/3853826948,v1:172.21.15.47:6801/3853826948] [v2:172.21.15.47:6802/3853826948,v1:172.21.15.47:6803/3853826948] exists,up eabfa3cf-f2a3-4995-a7de-f6d8ba02ee6f 2022-05-13T23:15:52.752 INFO:teuthology.orchestra.run.smithi002.stdout:osd.2 up in weight 1 up_from 18 up_thru 81 down_at 0 last_clean_interval [0,0) [v2:172.21.15.115:6800/2562256956,v1:172.21.15.115:6801/2562256956] [v2:172.21.15.115:6802/2562256956,v1:172.21.15.115:6803/2562256956] exists,up 153c5bd5-8200-47a7-a135-8a95b6752cd2 2022-05-13T23:15:52.752 INFO:teuthology.orchestra.run.smithi002.stdout:osd.3 up in weight 1 up_from 25 up_thru 81 down_at 0 last_clean_interval [0,0) [v2:172.21.15.2:6810/1099909411,v1:172.21.15.2:6811/1099909411] [v2:172.21.15.2:6812/1099909411,v1:172.21.15.2:6813/1099909411] exists,up 6ca9c602-5936-48da-a861-c437e6a33765 2022-05-13T23:15:52.752 INFO:teuthology.orchestra.run.smithi002.stdout:osd.4 up in weight 1 up_from 33 up_thru 81 down_at 0 last_clean_interval [0,0) [v2:172.21.15.47:6808/882753861,v1:172.21.15.47:6809/882753861] [v2:172.21.15.47:6810/882753861,v1:172.21.15.47:6811/882753861] exists,up 54d943d9-2610-422a-9e59-4b8212aced31 2022-05-13T23:15:52.753 INFO:teuthology.orchestra.run.smithi002.stdout:osd.5 up in weight 1 up_from 38 up_thru 81 down_at 0 last_clean_interval [0,0) [v2:172.21.15.115:6808/2122391993,v1:172.21.15.115:6809/2122391993] [v2:172.21.15.115:6810/2122391993,v1:172.21.15.115:6811/2122391993] exists,up c18566cf-5403-4057-b2ee-aa298e87e8e9 2022-05-13T23:15:52.753 INFO:teuthology.orchestra.run.smithi002.stdout:osd.6 up in weight 1 up_from 46 up_thru 81 down_at 0 last_clean_interval [0,0) [v2:172.21.15.2:6818/1892198790,v1:172.21.15.2:6819/1892198790] [v2:172.21.15.2:6820/1892198790,v1:172.21.15.2:6821/1892198790] exists,up 38ec7b6b-30f4-4d97-ad7e-e160ae59c843 2022-05-13T23:15:52.753 INFO:teuthology.orchestra.run.smithi002.stdout:osd.7 up in weight 1 up_from 51 up_thru 84 down_at 0 last_clean_interval [0,0) [v2:172.21.15.47:6816/2812542825,v1:172.21.15.47:6817/2812542825] [v2:172.21.15.47:6818/2812542825,v1:172.21.15.47:6819/2812542825] exists,up e799b560-c61e-4369-b08b-45e5a6af37c0 2022-05-13T23:15:52.754 INFO:teuthology.orchestra.run.smithi002.stdout:osd.8 up in weight 1 up_from 58 up_thru 81 down_at 0 last_clean_interval [0,0) [v2:172.21.15.115:6816/3959632033,v1:172.21.15.115:6817/3959632033] [v2:172.21.15.115:6818/3959632033,v1:172.21.15.115:6819/3959632033] exists,up a58f2c0e-e487-41cf-8d2e-8c937cac9ebc 2022-05-13T23:15:52.754 INFO:teuthology.orchestra.run.smithi002.stdout:osd.9 up in weight 1 up_from 66 up_thru 81 down_at 0 last_clean_interval [0,0) [v2:172.21.15.2:6826/3004162255,v1:172.21.15.2:6827/3004162255] [v2:172.21.15.2:6828/3004162255,v1:172.21.15.2:6829/3004162255] exists,up 12cead96-f726-4f7a-ab84-eb191b778a5c 2022-05-13T23:15:52.754 INFO:teuthology.orchestra.run.smithi002.stdout:osd.10 up in weight 1 up_from 71 up_thru 84 down_at 0 last_clean_interval [0,0) [v2:172.21.15.47:6824/3443413998,v1:172.21.15.47:6825/3443413998] [v2:172.21.15.47:6826/3443413998,v1:172.21.15.47:6827/3443413998] exists,up 4f648118-7855-48d6-86d5-44eed5295f0c 2022-05-13T23:15:52.754 INFO:teuthology.orchestra.run.smithi002.stdout:osd.11 up in weight 1 up_from 78 up_thru 84 down_at 0 last_clean_interval [0,0) [v2:172.21.15.115:6824/1758718209,v1:172.21.15.115:6825/1758718209] [v2:172.21.15.115:6826/1758718209,v1:172.21.15.115:6827/1758718209] exists,up 916541e0-fb28-46d5-9743-801c54c5a6d1 2022-05-13T23:15:52.754 INFO:teuthology.orchestra.run.smithi002.stdout:pg_upmap_items 2.f [8,11] 2022-05-13T23:15:52.755 INFO:teuthology.orchestra.run.smithi002.stdout:pg_upmap_items 2.13 [2,11] 2022-05-13T23:15:52.755 INFO:teuthology.orchestra.run.smithi002.stdout:pg_upmap_items 3.10 [5,9] 2022-05-13T23:15:52.755 INFO:teuthology.orchestra.run.smithi002.stdout:pg_upmap_items 3.2c [4,9] 2022-05-13T23:15:52.755 INFO:teuthology.orchestra.run.smithi002.stdout:pg_upmap_items 3.36 [4,9] 2022-05-13T23:15:52.756 INFO:teuthology.orchestra.run.smithi002.stdout:blocklist 172.21.15.2:0/477249925 expires 2022-05-14T23:07:04.986087+0000 2022-05-13T23:15:52.756 INFO:teuthology.orchestra.run.smithi002.stdout:blocklist 172.21.15.2:6800/3226099420 expires 2022-05-14T23:07:04.986087+0000 2022-05-13T23:15:52.756 INFO:teuthology.orchestra.run.smithi002.stdout:blocklist 172.21.15.2:0/2214093915 expires 2022-05-14T23:07:04.986087+0000 2022-05-13T23:15:52.756 INFO:teuthology.orchestra.run.smithi002.stdout:blocklist 172.21.15.2:0/1039403355 expires 2022-05-14T23:07:04.986087+0000 2022-05-13T23:15:52.757 INFO:teuthology.orchestra.run.smithi002.stdout:blocklist 172.21.15.2:0/22091675 expires 2022-05-14T23:06:36.304108+0000 2022-05-13T23:15:52.757 INFO:teuthology.orchestra.run.smithi002.stdout:blocklist 172.21.15.2:6800/1255134634 expires 2022-05-14T23:06:36.304108+0000 2022-05-13T23:15:52.757 INFO:teuthology.orchestra.run.smithi002.stdout:blocklist 172.21.15.2:6801/3226099420 expires 2022-05-14T23:07:04.986087+0000 2022-05-13T23:15:52.757 INFO:teuthology.orchestra.run.smithi002.stdout:blocklist 172.21.15.2:0/3768677276 expires 2022-05-14T23:06:36.304108+0000 2022-05-13T23:15:52.758 INFO:teuthology.orchestra.run.smithi002.stdout:blocklist 172.21.15.2:6801/1255134634 expires 2022-05-14T23:06:36.304108+0000 2022-05-13T23:15:52.758 INFO:teuthology.orchestra.run.smithi002.stdout:blocklist 172.21.15.2:0/407841268 expires 2022-05-14T23:06:36.304108+0000 2022-05-13T23:15:52.759 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:52.743+0000 7fec06e12700 1 -- 172.21.15.2:0/3226788168 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7febf0096ce0 msgr2=0x7febf0099190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:15:52.759 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:52.743+0000 7fec06e12700 1 --2- 172.21.15.2:0/3226788168 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7febf0096ce0 0x7febf0099190 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fec000fe8f0 tx=0x7febf400b040 comp rx=0 tx=0).stop 2022-05-13T23:15:52.759 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:52.743+0000 7fec06e12700 1 -- 172.21.15.2:0/3226788168 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fec000fea90 msgr2=0x7fec0010eb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:15:52.760 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:52.743+0000 7fec06e12700 1 --2- 172.21.15.2:0/3226788168 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fec000fea90 0x7fec0010eb70 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7febec004ee0 tx=0x7febec009cb0 comp rx=0 tx=0).stop 2022-05-13T23:15:52.760 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:52.743+0000 7fec06e12700 1 -- 172.21.15.2:0/3226788168 shutdown_connections 2022-05-13T23:15:52.760 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:52.743+0000 7fec06e12700 1 --2- 172.21.15.2:0/3226788168 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fec000ff420 0x7fec00115c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:52.761 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:52.743+0000 7fec06e12700 1 --2- 172.21.15.2:0/3226788168 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fec000fd7f0 0x7fec0010e630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:52.761 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:52.743+0000 7fec06e12700 1 --2- 172.21.15.2:0/3226788168 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7febf0096ce0 0x7febf0099190 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:52.761 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:52.743+0000 7fec06e12700 1 --2- 172.21.15.2:0/3226788168 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fec000fea90 0x7fec0010eb70 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:52.762 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:52.743+0000 7fec06e12700 1 -- 172.21.15.2:0/3226788168 >> 172.21.15.2:0/3226788168 conn(0x7fec000f9030 msgr2=0x7fec000fb3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-05-13T23:15:52.762 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:52.743+0000 7fec06e12700 1 -- 172.21.15.2:0/3226788168 shutdown_connections 2022-05-13T23:15:52.762 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:52.743+0000 7fec06e12700 1 -- 172.21.15.2:0/3226788168 wait complete. 2022-05-13T23:15:52.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:15:52 smithi047 ceph-mon[77147]: pgmap v338: 97 pgs: 97 active+clean; 584 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2022-05-13T23:15:53.292 INFO:teuthology.run_tasks:Running task fs.ready... 2022-05-13T23:15:53.320 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-05-13T23:15:53.544 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:53.542+0000 7fe82a59d700 1 -- 172.21.15.2:0/748875758 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe824070370 msgr2=0x7fe82406e870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:15:53.544 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:53.542+0000 7fe82a59d700 1 --2- 172.21.15.2:0/748875758 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe824070370 0x7fe82406e870 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fe820004870 tx=0x7fe820009cb0 comp rx=0 tx=0).stop 2022-05-13T23:15:53.544 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:53.542+0000 7fe82a59d700 1 -- 172.21.15.2:0/748875758 shutdown_connections 2022-05-13T23:15:53.545 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:53.542+0000 7fe82a59d700 1 --2- 172.21.15.2:0/748875758 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe824103420 0x7fe824105800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:53.545 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:53.542+0000 7fe82a59d700 1 --2- 172.21.15.2:0/748875758 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe824070370 0x7fe82406e870 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:53.545 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:53.542+0000 7fe82a59d700 1 --2- 172.21.15.2:0/748875758 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe82406edb0 0x7fe82406f220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:53.545 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:53.542+0000 7fe82a59d700 1 -- 172.21.15.2:0/748875758 >> 172.21.15.2:0/748875758 conn(0x7fe8240f6c80 msgr2=0x7fe8240f90b0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-05-13T23:15:53.546 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:53.542+0000 7fe82a59d700 1 -- 172.21.15.2:0/748875758 shutdown_connections 2022-05-13T23:15:53.546 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:53.542+0000 7fe82a59d700 1 -- 172.21.15.2:0/748875758 wait complete. 2022-05-13T23:15:53.551 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:53.543+0000 7fe82a59d700 1 Processor -- start 2022-05-13T23:15:53.551 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:53.543+0000 7fe82a59d700 1 -- start start 2022-05-13T23:15:53.551 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:53.543+0000 7fe82a59d700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe82406edb0 0x7fe82410e510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:15:53.552 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:53.543+0000 7fe82a59d700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe824070370 0x7fe82410ea50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:15:53.552 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:53.544+0000 7fe82a59d700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe824103420 0x7fe824115ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:15:53.553 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:53.544+0000 7fe82a59d700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe824116020 con 0x7fe824103420 2022-05-13T23:15:53.553 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:53.544+0000 7fe82a59d700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe824116190 con 0x7fe824070370 2022-05-13T23:15:53.553 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:53.544+0000 7fe82a59d700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe824116350 con 0x7fe82406edb0 2022-05-13T23:15:53.553 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:53.544+0000 7fe829d9c700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe824103420 0x7fe824115ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:15:53.554 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:53.544+0000 7fe828d9a700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe824070370 0x7fe82410ea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:15:53.554 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:53.544+0000 7fe828d9a700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe824070370 0x7fe82410ea50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.2:37214/0 (socket says 172.21.15.2:37214) 2022-05-13T23:15:53.554 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:53.544+0000 7fe829d9c700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe824103420 0x7fe824115ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:53888/0 (socket says 172.21.15.2:53888) 2022-05-13T23:15:53.556 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:53.544+0000 7fe828d9a700 1 -- 172.21.15.2:0/4024401114 learned_addr learned my addr 172.21.15.2:0/4024401114 (peer_addr_for_me v2:172.21.15.2:0/0) 2022-05-13T23:15:53.557 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:53.544+0000 7fe82959b700 1 --2- 172.21.15.2:0/4024401114 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe82406edb0 0x7fe82410e510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:15:53.557 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:53.544+0000 7fe828d9a700 1 -- 172.21.15.2:0/4024401114 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe82406edb0 msgr2=0x7fe82410e510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:15:53.557 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:53.544+0000 7fe828d9a700 1 --2- 172.21.15.2:0/4024401114 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe82406edb0 0x7fe82410e510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:53.558 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:53.544+0000 7fe828d9a700 1 -- 172.21.15.2:0/4024401114 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe824103420 msgr2=0x7fe824115ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:15:53.558 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:53.544+0000 7fe828d9a700 1 --2- 172.21.15.2:0/4024401114 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe824103420 0x7fe824115ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:53.558 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:53.544+0000 7fe828d9a700 1 -- 172.21.15.2:0/4024401114 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe820005040 con 0x7fe824070370 2022-05-13T23:15:53.559 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:53.544+0000 7fe829d9c700 1 --2- 172.21.15.2:0/4024401114 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe824103420 0x7fe824115ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-05-13T23:15:53.559 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:53.545+0000 7fe828d9a700 1 --2- 172.21.15.2:0/4024401114 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe824070370 0x7fe82410ea50 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fe81400e970 tx=0x7fe81400b040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-05-13T23:15:53.560 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:53.545+0000 7fe819ffb700 1 -- 172.21.15.2:0/4024401114 <== mon.1 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe814007b40 con 0x7fe824070370 2022-05-13T23:15:53.560 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:53.545+0000 7fe819ffb700 1 -- 172.21.15.2:0/4024401114 <== mon.1 v2:172.21.15.47:3300/0 2 ==== config(26 keys) v1 ==== 1087+0+0 (secure 0 0 0) 0x7fe81400f040 con 0x7fe824070370 2022-05-13T23:15:53.560 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:53.545+0000 7fe82a59d700 1 -- 172.21.15.2:0/4024401114 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe824116630 con 0x7fe824070370 2022-05-13T23:15:53.561 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:53.545+0000 7fe819ffb700 1 -- 172.21.15.2:0/4024401114 <== mon.1 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe814014950 con 0x7fe824070370 2022-05-13T23:15:53.561 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:53.545+0000 7fe82a59d700 1 -- 172.21.15.2:0/4024401114 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe824116ba0 con 0x7fe824070370 2022-05-13T23:15:53.561 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:53.546+0000 7fe82a59d700 1 -- 172.21.15.2:0/4024401114 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe824108eb0 con 0x7fe824070370 2022-05-13T23:15:53.561 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:53.548+0000 7fe819ffb700 1 -- 172.21.15.2:0/4024401114 <== mon.1 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 16) v1 ==== 125136+0+0 (secure 0 0 0) 0x7fe81401b030 con 0x7fe824070370 2022-05-13T23:15:53.562 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:53.549+0000 7fe819ffb700 1 --2- 172.21.15.2:0/4024401114 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7fe810096ce0 0x7fe810099190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:15:53.562 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:53.550+0000 7fe819ffb700 1 -- 172.21.15.2:0/4024401114 <== mon.1 v2:172.21.15.47:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8120+0+0 (secure 0 0 0) 0x7fe8140bee60 con 0x7fe824070370 2022-05-13T23:15:53.562 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:53.550+0000 7fe82959b700 1 --2- 172.21.15.2:0/4024401114 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7fe810096ce0 0x7fe810099190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:15:53.563 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:53.550+0000 7fe82959b700 1 --2- 172.21.15.2:0/4024401114 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7fe810096ce0 0x7fe810099190 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fe820004840 tx=0x7fe820009cb0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-05-13T23:15:53.564 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:53.552+0000 7fe819ffb700 1 -- 172.21.15.2:0/4024401114 <== mon.1 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+165215 (secure 0 0 0) 0x7fe81408c110 con 0x7fe824070370 2022-05-13T23:15:53.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:15:53 smithi115 ceph-mon[78593]: from='client.? 172.21.15.2:0/3226788168' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2022-05-13T23:15:53.745 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:53.744+0000 7fe82a59d700 1 -- 172.21.15.2:0/4024401114 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe8241019f0 con 0x7fe824070370 2022-05-13T23:15:53.746 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:53.745+0000 7fe819ffb700 1 -- 172.21.15.2:0/4024401114 <== mon.1 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v85) v1 ==== 74+0+21505 (secure 0 0 0) 0x7fe814019090 con 0x7fe824070370 2022-05-13T23:15:53.747 INFO:teuthology.orchestra.run.smithi002.stdout: 2022-05-13T23:15:53.748 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":85,"fsid":"16cf5f88-d311-11ec-be40-001a4aab830c","created":"2022-05-13T23:06:00.765453+0000","modified":"2022-05-13T23:15:06.402793+0000","last_up_change":"2022-05-13T23:13:20.935982+0000","last_in_change":"2022-05-13T23:13:02.796000+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2022-05-13T23:09:32.000644+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2022-05-13T23:14:14.195678+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":3,"pool_name":"cephfs_data","create_time":"2022-05-13T23:14:15.659212+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"83","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"11e700e8-f3e8-4903-b52c-db354b4395bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6802","nonce":1174843704},{"type":"v1","addr":"172.21.15.2:6803","nonce":1174843704}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6804","nonce":1174843704},{"type":"v1","addr":"172.21.15.2:6805","nonce":1174843704}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6808","nonce":1174843704},{"type":"v1","addr":"172.21.15.2:6809","nonce":1174843704}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6806","nonce":1174843704},{"type":"v1","addr":"172.21.15.2:6807","nonce":1174843704}]},"public_addr":"172.21.15.2:6803/1174843704","cluster_addr":"172.21.15.2:6805/1174843704","heartbeat_back_addr":"172.21.15.2:6809/1174843704","heartbeat_front_addr":"172.21.15.2:6807/1174843704","state":["exists","up"]},{"osd":1,"uuid":"eabfa3cf-f2a3-4995-a7de-f6d8ba02ee6f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6800","nonce":3853826948},{"type":"v1","addr":"172.21.15.47:6801","nonce":3853826948}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6802","nonce":3853826948},{"type":"v1","addr":"172.21.15.47:6803","nonce":3853826948}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6806","nonce":3853826948},{"type":"v1","addr":"172.21.15.47:6807","nonce":3853826948}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6804","nonce":3853826948},{"type":"v1","addr":"172.21.15.47:6805","nonce":3853826948}]},"public_addr":"172.21.15.47:6801/3853826948","cluster_addr":"172.21.15.47:6803/3853826948","heartbeat_back_addr":"172.21.15.47:6807/3853826948","heartbeat_front_addr":"172.21.15.47:6805/3853826948","state":["exists","up"]},{"osd":2,"uuid":"153c5bd5-8200-47a7-a135-8a95b6752cd2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6800","nonce":2562256956},{"type":"v1","addr":"172.21.15.115:6801","nonce":2562256956}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6802","nonce":2562256956},{"type":"v1","addr":"172.21.15.115:6803","nonce":2562256956}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6806","nonce":2562256956},{"type":"v1","addr":"172.21.15.115:6807","nonce":2562256956}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6804","nonce":2562256956},{"type":"v1","addr":"172.21.15.115:6805","nonce":2562256956}]},"public_addr":"172.21.15.115:6801/2562256956","cluster_addr":"172.21.15.115:6803/2562256956","heartbeat_back_addr":"172.21.15.115:6807/2562256956","heartbeat_front_addr":"172.21.15.115:6805/2562256956","state":["exists","up"]},{"osd":3,"uuid":"6ca9c602-5936-48da-a861-c437e6a33765","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6810","nonce":1099909411},{"type":"v1","addr":"172.21.15.2:6811","nonce":1099909411}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6812","nonce":1099909411},{"type":"v1","addr":"172.21.15.2:6813","nonce":1099909411}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6816","nonce":1099909411},{"type":"v1","addr":"172.21.15.2:6817","nonce":1099909411}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6814","nonce":1099909411},{"type":"v1","addr":"172.21.15.2:6815","nonce":1099909411}]},"public_addr":"172.21.15.2:6811/1099909411","cluster_addr":"172.21.15.2:6813/1099909411","heartbeat_back_addr":"172.21.15.2:6817/1099909411","heartbeat_front_addr":"172.21.15.2:6815/1099909411","state":["exists","up"]},{"osd":4,"uuid":"54d943d9-2610-422a-9e59-4b8212aced31","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6808","nonce":882753861},{"type":"v1","addr":"172.21.15.47:6809","nonce":882753861}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6810","nonce":882753861},{"type":"v1","addr":"172.21.15.47:6811","nonce":882753861}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6814","nonce":882753861},{"type":"v1","addr":"172.21.15.47:6815","nonce":882753861}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6812","nonce":882753861},{"type":"v1","addr":"172.21.15.47:6813","nonce":882753861}]},"public_addr":"172.21.15.47:6809/882753861","cluster_addr":"172.21.15.47:6811/882753861","heartbeat_back_addr":"172.21.15.47:6815/882753861","heartbeat_front_addr":"172.21.15.47:6813/882753861","state":["exists","up"]},{"osd":5,"uuid":"c18566cf-5403-4057-b2ee-aa298e87e8e9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6808","nonce":2122391993},{"type":"v1","addr":"172.21.15.115:6809","nonce":2122391993}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6810","nonce":2122391993},{"type":"v1","addr":"172.21.15.115:6811","nonce":2122391993}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6814","nonce":2122391993},{"type":"v1","addr":"172.21.15.115:6815","nonce":2122391993}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6812","nonce":2122391993},{"type":"v1","addr":"172.21.15.115:6813","nonce":2122391993}]},"public_addr":"172.21.15.115:6809/2122391993","cluster_addr":"172.21.15.115:6811/2122391993","heartbeat_back_addr":"172.21.15.115:6815/2122391993","heartbeat_front_addr":"172.21.15.115:6813/2122391993","state":["exists","up"]},{"osd":6,"uuid":"38ec7b6b-30f4-4d97-ad7e-e160ae59c843","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6818","nonce":1892198790},{"type":"v1","addr":"172.21.15.2:6819","nonce":1892198790}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6820","nonce":1892198790},{"type":"v1","addr":"172.21.15.2:6821","nonce":1892198790}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6824","nonce":1892198790},{"type":"v1","addr":"172.21.15.2:6825","nonce":1892198790}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6822","nonce":1892198790},{"type":"v1","addr":"172.21.15.2:6823","nonce":1892198790}]},"public_addr":"172.21.15.2:6819/1892198790","cluster_addr":"172.21.15.2:6821/1892198790","heartbeat_back_addr":"172.21.15.2:6825/1892198790","heartbeat_front_addr":"172.21.15.2:6823/1892198790","state":["exists","up"]},{"osd":7,"uuid":"e799b560-c61e-4369-b08b-45e5a6af37c0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6816","nonce":2812542825},{"type":"v1","addr":"172.21.15.47:6817","nonce":2812542825}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6818","nonce":2812542825},{"type":"v1","addr":"172.21.15.47:6819","nonce":2812542825}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6822","nonce":2812542825},{"type":"v1","addr":"172.21.15.47:6823","nonce":2812542825}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6820","nonce":2812542825},{"type":"v1","addr":"172.21.15.47:6821","nonce":2812542825}]},"public_addr":"172.21.15.47:6817/2812542825","cluster_addr":"172.21.15.47:6819/2812542825","heartbeat_back_addr":"172.21.15.47:6823/2812542825","heartbeat_front_addr":"172.21.15.47:6821/2812542825","state":["exists","up"]},{"osd":8,"uuid":"a58f2c0e-e487-41cf-8d2e-8c937cac9ebc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":58,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6816","nonce":3959632033},{"type":"v1","addr":"172.21.15.115:6817","nonce":3959632033}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6818","nonce":3959632033},{"type":"v1","addr":"172.21.15.115:6819","nonce":3959632033}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6822","nonce":3959632033},{"type":"v1","addr":"172.21.15.115:6823","nonce":3959632033}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6820","nonce":3959632033},{"type":"v1","addr":"172.21.15.115:6821","nonce":3959632033}]},"public_addr":"172.21.15.115:6817/3959632033","cluster_addr":"172.21.15.115:6819/3959632033","heartbeat_back_addr":"172.21.15.115:6823/3959632033","heartbeat_front_addr":"172.21.15.115:6821/3959632033","state":["exists","up"]},{"osd":9,"uuid":"12cead96-f726-4f7a-ab84-eb191b778a5c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":66,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6826","nonce":3004162255},{"type":"v1","addr":"172.21.15.2:6827","nonce":3004162255}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6828","nonce":3004162255},{"type":"v1","addr":"172.21.15.2:6829","nonce":3004162255}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6832","nonce":3004162255},{"type":"v1","addr":"172.21.15.2:6833","nonce":3004162255}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6830","nonce":3004162255},{"type":"v1","addr":"172.21.15.2:6831","nonce":3004162255}]},"public_addr":"172.21.15.2:6827/3004162255","cluster_addr":"172.21.15.2:6829/3004162255","heartbeat_back_addr":"172.21.15.2:6833/3004162255","heartbeat_front_addr":"172.21.15.2:6831/3004162255","state":["exists","up"]},{"osd":10,"uuid":"4f648118-7855-48d6-86d5-44eed5295f0c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":71,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6824","nonce":3443413998},{"type":"v1","addr":"172.21.15.47:6825","nonce":3443413998}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6826","nonce":3443413998},{"type":"v1","addr":"172.21.15.47:6827","nonce":3443413998}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6830","nonce":3443413998},{"type":"v1","addr":"172.21.15.47:6831","nonce":3443413998}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6828","nonce":3443413998},{"type":"v1","addr":"172.21.15.47:6829","nonce":3443413998}]},"public_addr":"172.21.15.47:6825/3443413998","cluster_addr":"172.21.15.47:6827/3443413998","heartbeat_back_addr":"172.21.15.47:6831/3443413998","heartbeat_front_addr":"172.21.15.47:6829/3443413998","state":["exists","up"]},{"osd":11,"uuid":"916541e0-fb28-46d5-9743-801c54c5a6d1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":78,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6824","nonce":1758718209},{"type":"v1","addr":"172.21.15.115:6825","nonce":1758718209}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6826","nonce":1758718209},{"type":"v1","addr":"172.21.15.115:6827","nonce":1758718209}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6830","nonce":1758718209},{"type":"v1","addr":"172.21.15.115:6831","nonce":1758718209}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6828","nonce":1758718209},{"type":"v1","addr":"172.21.15.115:6829","nonce":1758718209}]},"public_addr":"172.21.15.115:6825/1758718209","cluster_addr":"172.21.15.115:6827/1758718209","heartbeat_back_addr":"172.21.15.115:6831/1758718209","heartbeat_front_addr":"172.21.15.115:6829/1758718209","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-05-13T23:08:43.757857+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-05-13T23:09:06.537139+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-05-13T23:09:28.524837+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-05-13T23:10:02.057583+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-05-13T23:10:26.210648+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-05-13T23:10:47.885853+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-05-13T23:11:19.141865+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-05-13T23:11:42.376065+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-05-13T23:12:04.175870+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-05-13T23:12:34.695347+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-05-13T23:12:57.762613+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-05-13T23:13:19.114284+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.f","mappings":[{"from":8,"to":11}]},{"pgid":"2.13","mappings":[{"from":2,"to":11}]},{"pgid":"3.10","mappings":[{"from":5,"to":9}]},{"pgid":"3.2c","mappings":[{"from":4,"to":9}]},{"pgid":"3.36","mappings":[{"from":4,"to":9}]}],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.2:0/477249925":"2022-05-14T23:07:04.986087+0000","172.21.15.2:6800/3226099420":"2022-05-14T23:07:04.986087+0000","172.21.15.2:0/2214093915":"2022-05-14T23:07:04.986087+0000","172.21.15.2:0/1039403355":"2022-05-14T23:07:04.986087+0000","172.21.15.2:0/22091675":"2022-05-14T23:06:36.304108+0000","172.21.15.2:6800/1255134634":"2022-05-14T23:06:36.304108+0000","172.21.15.2:6801/3226099420":"2022-05-14T23:07:04.986087+0000","172.21.15.2:0/3768677276":"2022-05-14T23:06:36.304108+0000","172.21.15.2:6801/1255134634":"2022-05-14T23:06:36.304108+0000","172.21.15.2:0/407841268":"2022-05-14T23:06:36.304108+0000"},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2022-05-13T23:15:53.751 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:53.748+0000 7fe82a59d700 1 -- 172.21.15.2:0/4024401114 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7fe810096ce0 msgr2=0x7fe810099190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:15:53.751 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:53.748+0000 7fe82a59d700 1 --2- 172.21.15.2:0/4024401114 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7fe810096ce0 0x7fe810099190 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fe820004840 tx=0x7fe820009cb0 comp rx=0 tx=0).stop 2022-05-13T23:15:53.752 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:53.748+0000 7fe82a59d700 1 -- 172.21.15.2:0/4024401114 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe824070370 msgr2=0x7fe82410ea50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:15:53.752 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:53.748+0000 7fe82a59d700 1 --2- 172.21.15.2:0/4024401114 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe824070370 0x7fe82410ea50 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fe81400e970 tx=0x7fe81400b040 comp rx=0 tx=0).stop 2022-05-13T23:15:53.752 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:53.748+0000 7fe82a59d700 1 -- 172.21.15.2:0/4024401114 shutdown_connections 2022-05-13T23:15:53.752 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:53.748+0000 7fe82a59d700 1 --2- 172.21.15.2:0/4024401114 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7fe810096ce0 0x7fe810099190 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:53.753 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:53.748+0000 7fe82a59d700 1 --2- 172.21.15.2:0/4024401114 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe824103420 0x7fe824115ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:53.753 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:53.748+0000 7fe82a59d700 1 --2- 172.21.15.2:0/4024401114 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe82406edb0 0x7fe82410e510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:53.758 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:53.748+0000 7fe82a59d700 1 --2- 172.21.15.2:0/4024401114 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe824070370 0x7fe82410ea50 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:53.758 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:53.748+0000 7fe82a59d700 1 -- 172.21.15.2:0/4024401114 >> 172.21.15.2:0/4024401114 conn(0x7fe8240f6c80 msgr2=0x7fe8240f9060 unknown :-1 s=STATE_NONE l=0).mark_down 2022-05-13T23:15:53.758 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:53.749+0000 7fe82a59d700 1 -- 172.21.15.2:0/4024401114 shutdown_connections 2022-05-13T23:15:53.758 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:53.749+0000 7fe82a59d700 1 -- 172.21.15.2:0/4024401114 wait complete. 2022-05-13T23:15:53.772 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:15:53 smithi002 ceph-mon[71816]: from='client.? 172.21.15.2:0/3226788168' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2022-05-13T23:15:53.776 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2022-05-13T23:09:32.000644+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2022-05-13T23:14:14.195678+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '82', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2022-05-13T23:14:15.659212+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '83', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2022-05-13T23:15:53.776 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2022-05-13T23:15:53.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:15:53 smithi047 ceph-mon[77147]: from='client.? 172.21.15.2:0/3226788168' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2022-05-13T23:15:54.008 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:54.001+0000 7f941dd9d700 1 -- 172.21.15.2:0/3657388374 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f941806edb0 msgr2=0x7f941806f220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:15:54.008 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:54.001+0000 7f941dd9d700 1 --2- 172.21.15.2:0/3657388374 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f941806edb0 0x7f941806f220 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f9404004870 tx=0x7f9404009cb0 comp rx=0 tx=0).stop 2022-05-13T23:15:54.008 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:54.001+0000 7f941dd9d700 1 -- 172.21.15.2:0/3657388374 shutdown_connections 2022-05-13T23:15:54.009 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:54.001+0000 7f941dd9d700 1 --2- 172.21.15.2:0/3657388374 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f9418103410 0x7f94181057f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:54.009 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:54.001+0000 7f941dd9d700 1 --2- 172.21.15.2:0/3657388374 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f9418070370 0x7f941806e870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:54.009 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:54.001+0000 7f941dd9d700 1 --2- 172.21.15.2:0/3657388374 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f941806edb0 0x7f941806f220 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:54.009 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:54.001+0000 7f941dd9d700 1 -- 172.21.15.2:0/3657388374 >> 172.21.15.2:0/3657388374 conn(0x7f94180f6c50 msgr2=0x7f94180f90a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-05-13T23:15:54.010 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:54.001+0000 7f941dd9d700 1 -- 172.21.15.2:0/3657388374 shutdown_connections 2022-05-13T23:15:54.010 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:54.001+0000 7f941dd9d700 1 -- 172.21.15.2:0/3657388374 wait complete. 2022-05-13T23:15:54.010 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:54.002+0000 7f941dd9d700 1 Processor -- start 2022-05-13T23:15:54.010 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:54.002+0000 7f941dd9d700 1 -- start start 2022-05-13T23:15:54.011 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:54.003+0000 7f941dd9d700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f941806edb0 0x7f941810e5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:15:54.012 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:54.003+0000 7f941dd9d700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9418070370 0x7f941810eae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:15:54.012 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:54.003+0000 7f941dd9d700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f9418103410 0x7f9418115b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:15:54.012 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:54.003+0000 7f941dd9d700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94181160b0 con 0x7f9418070370 2022-05-13T23:15:54.012 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:54.003+0000 7f941dd9d700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9418116220 con 0x7f941806edb0 2022-05-13T23:15:54.013 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:54.003+0000 7f941dd9d700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94181163e0 con 0x7f9418103410 2022-05-13T23:15:54.013 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:54.003+0000 7f9417fff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9418070370 0x7f941810eae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:15:54.013 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:54.003+0000 7f9417fff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9418070370 0x7f941810eae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:53902/0 (socket says 172.21.15.2:53902) 2022-05-13T23:15:54.013 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:54.003+0000 7f9417fff700 1 -- 172.21.15.2:0/3615924619 learned_addr learned my addr 172.21.15.2:0/3615924619 (peer_addr_for_me v2:172.21.15.2:0/0) 2022-05-13T23:15:54.014 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:54.003+0000 7f941d59c700 1 --2- 172.21.15.2:0/3615924619 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f9418103410 0x7f9418115b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:15:54.014 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:54.003+0000 7f941cd9b700 1 --2- 172.21.15.2:0/3615924619 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f941806edb0 0x7f941810e5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:15:54.017 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:54.004+0000 7f9417fff700 1 -- 172.21.15.2:0/3615924619 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f9418103410 msgr2=0x7f9418115b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:15:54.017 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:54.004+0000 7f9417fff700 1 --2- 172.21.15.2:0/3615924619 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f9418103410 0x7f9418115b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:54.018 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:54.004+0000 7f9417fff700 1 -- 172.21.15.2:0/3615924619 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f941806edb0 msgr2=0x7f941810e5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:15:54.018 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:54.004+0000 7f9417fff700 1 --2- 172.21.15.2:0/3615924619 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f941806edb0 0x7f941810e5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:54.018 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:54.004+0000 7f9417fff700 1 -- 172.21.15.2:0/3615924619 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9404005040 con 0x7f9418070370 2022-05-13T23:15:54.018 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:54.004+0000 7f9417fff700 1 --2- 172.21.15.2:0/3615924619 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9418070370 0x7f941810eae0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f9404004ee0 tx=0x7f9404009cb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-05-13T23:15:54.019 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:54.004+0000 7f94157fa700 1 -- 172.21.15.2:0/3615924619 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f940400e040 con 0x7f9418070370 2022-05-13T23:15:54.019 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:54.004+0000 7f94157fa700 1 -- 172.21.15.2:0/3615924619 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1087+0+0 (secure 0 0 0) 0x7f9404000c90 con 0x7f9418070370 2022-05-13T23:15:54.019 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:54.004+0000 7f941dd9d700 1 -- 172.21.15.2:0/3615924619 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9418116660 con 0x7f9418070370 2022-05-13T23:15:54.019 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:54.004+0000 7f94157fa700 1 -- 172.21.15.2:0/3615924619 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f940401cb70 con 0x7f9418070370 2022-05-13T23:15:54.019 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:54.004+0000 7f941dd9d700 1 -- 172.21.15.2:0/3615924619 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9418116a60 con 0x7f9418070370 2022-05-13T23:15:54.020 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:54.005+0000 7f941dd9d700 1 -- 172.21.15.2:0/3615924619 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9418108f20 con 0x7f9418070370 2022-05-13T23:15:54.021 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:54.007+0000 7f94157fa700 1 -- 172.21.15.2:0/3615924619 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 16) v1 ==== 125136+0+0 (secure 0 0 0) 0x7f9404009240 con 0x7f9418070370 2022-05-13T23:15:54.022 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:54.008+0000 7f94157fa700 1 --2- 172.21.15.2:0/3615924619 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f9408096ce0 0x7f9408099190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:15:54.022 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:54.009+0000 7f94157fa700 1 -- 172.21.15.2:0/3615924619 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8120+0+0 (secure 0 0 0) 0x7f94040c0560 con 0x7f9418070370 2022-05-13T23:15:54.022 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:54.009+0000 7f941cd9b700 1 --2- 172.21.15.2:0/3615924619 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f9408096ce0 0x7f9408099190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:15:54.022 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:54.009+0000 7f941cd9b700 1 --2- 172.21.15.2:0/3615924619 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f9408096ce0 0x7f9408099190 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f940c0048b0 tx=0x7f940c00b040 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-05-13T23:15:54.023 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:54.012+0000 7f94157fa700 1 -- 172.21.15.2:0/3615924619 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+165215 (secure 0 0 0) 0x7f940408d810 con 0x7f9418070370 2022-05-13T23:15:54.206 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:54.205+0000 7f941dd9d700 1 -- 172.21.15.2:0/3615924619 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f941802a540 con 0x7f9418070370 2022-05-13T23:15:54.207 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:54.206+0000 7f94157fa700 1 -- 172.21.15.2:0/3615924619 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v85) v1 ==== 93+0+10 (secure 0 0 0) 0x7f9404090e30 con 0x7f9418070370 2022-05-13T23:15:54.207 INFO:teuthology.orchestra.run.smithi002.stdout:pg_num: 1 2022-05-13T23:15:54.211 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:54.208+0000 7f941dd9d700 1 -- 172.21.15.2:0/3615924619 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f9408096ce0 msgr2=0x7f9408099190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:15:54.213 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:54.208+0000 7f941dd9d700 1 --2- 172.21.15.2:0/3615924619 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f9408096ce0 0x7f9408099190 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f940c0048b0 tx=0x7f940c00b040 comp rx=0 tx=0).stop 2022-05-13T23:15:54.214 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:54.208+0000 7f941dd9d700 1 -- 172.21.15.2:0/3615924619 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9418070370 msgr2=0x7f941810eae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:15:54.214 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:54.208+0000 7f941dd9d700 1 --2- 172.21.15.2:0/3615924619 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9418070370 0x7f941810eae0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f9404004ee0 tx=0x7f9404009cb0 comp rx=0 tx=0).stop 2022-05-13T23:15:54.214 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:54.209+0000 7f941dd9d700 1 -- 172.21.15.2:0/3615924619 shutdown_connections 2022-05-13T23:15:54.214 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:54.209+0000 7f941dd9d700 1 --2- 172.21.15.2:0/3615924619 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f9418103410 0x7f9418115b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:54.215 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:54.209+0000 7f941dd9d700 1 --2- 172.21.15.2:0/3615924619 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f941806edb0 0x7f941810e5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:54.215 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:54.209+0000 7f941dd9d700 1 --2- 172.21.15.2:0/3615924619 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f9408096ce0 0x7f9408099190 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:54.215 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:54.209+0000 7f941dd9d700 1 --2- 172.21.15.2:0/3615924619 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9418070370 0x7f941810eae0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:54.215 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:54.209+0000 7f941dd9d700 1 -- 172.21.15.2:0/3615924619 >> 172.21.15.2:0/3615924619 conn(0x7f94180f6c50 msgr2=0x7f94180f9030 unknown :-1 s=STATE_NONE l=0).mark_down 2022-05-13T23:15:54.216 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:54.209+0000 7f941dd9d700 1 -- 172.21.15.2:0/3615924619 shutdown_connections 2022-05-13T23:15:54.216 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:54.209+0000 7f941dd9d700 1 -- 172.21.15.2:0/3615924619 wait complete. 2022-05-13T23:15:54.236 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2022-05-13T23:15:54.460 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:54.458+0000 7f27fb39d700 1 -- 172.21.15.2:0/4194163551 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f27f40ff2e0 msgr2=0x7f27f4105ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:15:54.461 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:54.458+0000 7f27fb39d700 1 --2- 172.21.15.2:0/4194163551 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f27f40ff2e0 0x7f27f4105ba0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f27ec009c30 tx=0x7f27ec004c50 comp rx=0 tx=0).stop 2022-05-13T23:15:54.462 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:54.459+0000 7f27fb39d700 1 -- 172.21.15.2:0/4194163551 shutdown_connections 2022-05-13T23:15:54.466 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:54.459+0000 7f27fb39d700 1 --2- 172.21.15.2:0/4194163551 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f27f40ff2e0 0x7f27f4105ba0 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:54.466 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:54.459+0000 7f27fb39d700 1 --2- 172.21.15.2:0/4194163551 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f27f40fd6b0 0x7f27f40fdac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:54.466 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:54.459+0000 7f27fb39d700 1 --2- 172.21.15.2:0/4194163551 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f27f40fe950 0x7f27f40feda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:54.467 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:54.459+0000 7f27fb39d700 1 -- 172.21.15.2:0/4194163551 >> 172.21.15.2:0/4194163551 conn(0x7f27f40f8ea0 msgr2=0x7f27f40fb2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-05-13T23:15:54.467 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:54.459+0000 7f27fb39d700 1 -- 172.21.15.2:0/4194163551 shutdown_connections 2022-05-13T23:15:54.467 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:54.459+0000 7f27fb39d700 1 -- 172.21.15.2:0/4194163551 wait complete. 2022-05-13T23:15:54.471 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:54.460+0000 7f27fb39d700 1 Processor -- start 2022-05-13T23:15:54.472 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:54.460+0000 7f27fb39d700 1 -- start start 2022-05-13T23:15:54.472 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:54.460+0000 7f27fb39d700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f27f40fd6b0 0x7f27f410e3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:15:54.472 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:54.460+0000 7f27fb39d700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f27f40fe950 0x7f27f410e8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:15:54.472 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:54.460+0000 7f27fb39d700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f27f40ff2e0 0x7f27f4115970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:15:54.473 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:54.460+0000 7f27fb39d700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27f4115eb0 con 0x7f27f40fe950 2022-05-13T23:15:54.473 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:54.460+0000 7f27fb39d700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27f4116020 con 0x7f27f40fd6b0 2022-05-13T23:15:54.473 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:54.460+0000 7f27fb39d700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27f41161e0 con 0x7f27f40ff2e0 2022-05-13T23:15:54.473 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:54.460+0000 7f27fa39b700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f27f40fd6b0 0x7f27f410e3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:15:54.474 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:54.460+0000 7f27f9b9a700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f27f40fe950 0x7f27f410e8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:15:54.474 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:54.460+0000 7f27fa39b700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f27f40fd6b0 0x7f27f410e3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.2:37244/0 (socket says 172.21.15.2:37244) 2022-05-13T23:15:54.474 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:54.460+0000 7f27f9b9a700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f27f40fe950 0x7f27f410e8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:53924/0 (socket says 172.21.15.2:53924) 2022-05-13T23:15:54.475 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:54.460+0000 7f27fa39b700 1 -- 172.21.15.2:0/408053590 learned_addr learned my addr 172.21.15.2:0/408053590 (peer_addr_for_me v2:172.21.15.2:0/0) 2022-05-13T23:15:54.475 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:54.461+0000 7f27fab9c700 1 --2- 172.21.15.2:0/408053590 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f27f40ff2e0 0x7f27f4115970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:15:54.475 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:54.461+0000 7f27fa39b700 1 -- 172.21.15.2:0/408053590 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f27f40ff2e0 msgr2=0x7f27f4115970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:15:54.475 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:54.461+0000 7f27fa39b700 1 --2- 172.21.15.2:0/408053590 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f27f40ff2e0 0x7f27f4115970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:54.476 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:54.461+0000 7f27fa39b700 1 -- 172.21.15.2:0/408053590 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f27f40fe950 msgr2=0x7f27f410e8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:15:54.476 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:54.461+0000 7f27fa39b700 1 --2- 172.21.15.2:0/408053590 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f27f40fe950 0x7f27f410e8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:54.476 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:54.461+0000 7f27fa39b700 1 -- 172.21.15.2:0/408053590 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27f0005040 con 0x7f27f40fd6b0 2022-05-13T23:15:54.476 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:54.461+0000 7f27fab9c700 1 --2- 172.21.15.2:0/408053590 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f27f40ff2e0 0x7f27f4115970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2022-05-13T23:15:54.477 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:54.461+0000 7f27fa39b700 1 --2- 172.21.15.2:0/408053590 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f27f40fd6b0 0x7f27f410e3a0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f27f00093d0 tx=0x7f27f0009cb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-05-13T23:15:54.477 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:54.462+0000 7f27eaffd700 1 -- 172.21.15.2:0/408053590 <== mon.1 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27f000e630 con 0x7f27f40fd6b0 2022-05-13T23:15:54.477 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:54.462+0000 7f27eaffd700 1 -- 172.21.15.2:0/408053590 <== mon.1 v2:172.21.15.47:3300/0 2 ==== config(26 keys) v1 ==== 1087+0+0 (secure 0 0 0) 0x7f27f000f040 con 0x7f27f40fd6b0 2022-05-13T23:15:54.477 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:54.462+0000 7f27eaffd700 1 -- 172.21.15.2:0/408053590 <== mon.1 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27f0013b50 con 0x7f27f40fd6b0 2022-05-13T23:15:54.478 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:54.462+0000 7f27fb39d700 1 -- 172.21.15.2:0/408053590 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f27ec0097c0 con 0x7f27f40fd6b0 2022-05-13T23:15:54.478 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:54.462+0000 7f27fb39d700 1 -- 172.21.15.2:0/408053590 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f27f4116770 con 0x7f27f40fd6b0 2022-05-13T23:15:54.478 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:54.463+0000 7f27fb39d700 1 -- 172.21.15.2:0/408053590 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f27f4108ea0 con 0x7f27f40fd6b0 2022-05-13T23:15:54.478 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:54.465+0000 7f27eaffd700 1 -- 172.21.15.2:0/408053590 <== mon.1 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 16) v1 ==== 125136+0+0 (secure 0 0 0) 0x7f27f0041bc0 con 0x7f27f40fd6b0 2022-05-13T23:15:54.478 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:54.466+0000 7f27eaffd700 1 --2- 172.21.15.2:0/408053590 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f27e0096ce0 0x7f27e0099190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:15:54.479 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:54.466+0000 7f27eaffd700 1 -- 172.21.15.2:0/408053590 <== mon.1 v2:172.21.15.47:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8120+0+0 (secure 0 0 0) 0x7f27f00bf1f0 con 0x7f27f40fd6b0 2022-05-13T23:15:54.479 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:54.466+0000 7f27f9b9a700 1 --2- 172.21.15.2:0/408053590 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f27e0096ce0 0x7f27e0099190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:15:54.479 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:54.467+0000 7f27f9b9a700 1 --2- 172.21.15.2:0/408053590 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f27e0096ce0 0x7f27e0099190 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f27e4004790 tx=0x7f27e400b040 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-05-13T23:15:54.479 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:54.468+0000 7f27eaffd700 1 -- 172.21.15.2:0/408053590 <== mon.1 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+165215 (secure 0 0 0) 0x7f27f008c590 con 0x7f27f40fd6b0 2022-05-13T23:15:54.574 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:15:54 smithi002 ceph-mon[71816]: pgmap v339: 97 pgs: 97 active+clean; 584 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2022-05-13T23:15:54.575 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:15:54 smithi002 ceph-mon[71816]: from='client.? 172.21.15.2:0/4024401114' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2022-05-13T23:15:54.575 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:15:54 smithi002 ceph-mon[71816]: from='client.? 172.21.15.2:0/3615924619' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2022-05-13T23:15:54.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:15:54 smithi115 ceph-mon[78593]: pgmap v339: 97 pgs: 97 active+clean; 584 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2022-05-13T23:15:54.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:15:54 smithi115 ceph-mon[78593]: from='client.? 172.21.15.2:0/4024401114' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2022-05-13T23:15:54.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:15:54 smithi115 ceph-mon[78593]: from='client.? 172.21.15.2:0/3615924619' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2022-05-13T23:15:54.659 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:54.657+0000 7f27fb39d700 1 -- 172.21.15.2:0/408053590 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f27f4104ca0 con 0x7f27f40fd6b0 2022-05-13T23:15:54.659 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:54.658+0000 7f27eaffd700 1 -- 172.21.15.2:0/408053590 <== mon.1 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v85) v1 ==== 104+0+11 (secure 0 0 0) 0x7f27f008fbb0 con 0x7f27f40fd6b0 2022-05-13T23:15:54.660 INFO:teuthology.orchestra.run.smithi002.stdout:pg_num: 32 2022-05-13T23:15:54.662 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:54.660+0000 7f27fb39d700 1 -- 172.21.15.2:0/408053590 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f27e0096ce0 msgr2=0x7f27e0099190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:15:54.663 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:54.660+0000 7f27fb39d700 1 --2- 172.21.15.2:0/408053590 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f27e0096ce0 0x7f27e0099190 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f27e4004790 tx=0x7f27e400b040 comp rx=0 tx=0).stop 2022-05-13T23:15:54.663 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:54.660+0000 7f27fb39d700 1 -- 172.21.15.2:0/408053590 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f27f40fd6b0 msgr2=0x7f27f410e3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:15:54.663 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:54.660+0000 7f27fb39d700 1 --2- 172.21.15.2:0/408053590 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f27f40fd6b0 0x7f27f410e3a0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f27f00093d0 tx=0x7f27f0009cb0 comp rx=0 tx=0).stop 2022-05-13T23:15:54.663 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:54.661+0000 7f27fb39d700 1 -- 172.21.15.2:0/408053590 shutdown_connections 2022-05-13T23:15:54.664 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:54.661+0000 7f27fb39d700 1 --2- 172.21.15.2:0/408053590 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f27f40ff2e0 0x7f27f4115970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:54.664 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:54.661+0000 7f27fb39d700 1 --2- 172.21.15.2:0/408053590 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f27f40fd6b0 0x7f27f410e3a0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:54.664 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:54.661+0000 7f27fb39d700 1 --2- 172.21.15.2:0/408053590 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f27e0096ce0 0x7f27e0099190 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:54.664 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:54.661+0000 7f27fb39d700 1 --2- 172.21.15.2:0/408053590 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f27f40fe950 0x7f27f410e8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:54.665 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:54.661+0000 7f27fb39d700 1 -- 172.21.15.2:0/408053590 >> 172.21.15.2:0/408053590 conn(0x7f27f40f8ea0 msgr2=0x7f27f4101df0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-05-13T23:15:54.665 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:54.661+0000 7f27fb39d700 1 -- 172.21.15.2:0/408053590 shutdown_connections 2022-05-13T23:15:54.666 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:54.661+0000 7f27fb39d700 1 -- 172.21.15.2:0/408053590 wait complete. 2022-05-13T23:15:54.688 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2022-05-13T23:15:54.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:15:54 smithi047 ceph-mon[77147]: pgmap v339: 97 pgs: 97 active+clean; 584 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2022-05-13T23:15:54.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:15:54 smithi047 ceph-mon[77147]: from='client.? 172.21.15.2:0/4024401114' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2022-05-13T23:15:54.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:15:54 smithi047 ceph-mon[77147]: from='client.? 172.21.15.2:0/3615924619' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2022-05-13T23:15:54.911 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:54.907+0000 7fc52319d700 1 -- 172.21.15.2:0/2359922573 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc51c065540 msgr2=0x7fc51c1004d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:15:54.911 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:54.907+0000 7fc52319d700 1 --2- 172.21.15.2:0/2359922573 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc51c065540 0x7fc51c1004d0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fc518004870 tx=0x7fc518009cb0 comp rx=0 tx=0).stop 2022-05-13T23:15:54.912 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:54.907+0000 7fc52319d700 1 -- 172.21.15.2:0/2359922573 shutdown_connections 2022-05-13T23:15:54.912 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:54.907+0000 7fc52319d700 1 --2- 172.21.15.2:0/2359922573 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc51c103330 0x7fc51c105790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:54.912 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:54.907+0000 7fc52319d700 1 --2- 172.21.15.2:0/2359922573 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc51c065540 0x7fc51c1004d0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:54.912 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:54.907+0000 7fc52319d700 1 --2- 172.21.15.2:0/2359922573 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fc51c100a10 0x7fc51c102df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:54.912 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:54.907+0000 7fc52319d700 1 -- 172.21.15.2:0/2359922573 >> 172.21.15.2:0/2359922573 conn(0x7fc51c0f5b80 msgr2=0x7fc51c0f7fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-05-13T23:15:54.913 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:54.907+0000 7fc52319d700 1 -- 172.21.15.2:0/2359922573 shutdown_connections 2022-05-13T23:15:54.913 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:54.907+0000 7fc52319d700 1 -- 172.21.15.2:0/2359922573 wait complete. 2022-05-13T23:15:54.915 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:54.908+0000 7fc52319d700 1 Processor -- start 2022-05-13T23:15:54.916 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:54.908+0000 7fc52319d700 1 -- start start 2022-05-13T23:15:54.916 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:54.909+0000 7fc52319d700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc51c065540 0x7fc51c10e540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:15:54.916 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:54.909+0000 7fc52319d700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc51c100a10 0x7fc51c10ea80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:15:54.917 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:54.909+0000 7fc52319d700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fc51c103330 0x7fc51c115b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:15:54.917 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:54.909+0000 7fc52319d700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc51c116050 con 0x7fc51c100a10 2022-05-13T23:15:54.917 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:54.909+0000 7fc52319d700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc51c1161c0 con 0x7fc51c065540 2022-05-13T23:15:54.917 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:54.909+0000 7fc52319d700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc51c116380 con 0x7fc51c103330 2022-05-13T23:15:54.918 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:54.909+0000 7fc52199a700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc51c100a10 0x7fc51c10ea80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:15:54.918 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:54.909+0000 7fc52199a700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc51c100a10 0x7fc51c10ea80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:53948/0 (socket says 172.21.15.2:53948) 2022-05-13T23:15:54.918 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:54.909+0000 7fc52199a700 1 -- 172.21.15.2:0/4283065060 learned_addr learned my addr 172.21.15.2:0/4283065060 (peer_addr_for_me v2:172.21.15.2:0/0) 2022-05-13T23:15:54.919 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:54.909+0000 7fc52219b700 1 --2- 172.21.15.2:0/4283065060 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc51c065540 0x7fc51c10e540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:15:54.919 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:54.909+0000 7fc52299c700 1 --2- 172.21.15.2:0/4283065060 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fc51c103330 0x7fc51c115b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:15:54.919 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:54.909+0000 7fc52199a700 1 -- 172.21.15.2:0/4283065060 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fc51c103330 msgr2=0x7fc51c115b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:15:54.920 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:54.909+0000 7fc52199a700 1 --2- 172.21.15.2:0/4283065060 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fc51c103330 0x7fc51c115b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:54.920 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:54.909+0000 7fc52199a700 1 -- 172.21.15.2:0/4283065060 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc51c065540 msgr2=0x7fc51c10e540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:15:54.920 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:54.909+0000 7fc52199a700 1 --2- 172.21.15.2:0/4283065060 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc51c065540 0x7fc51c10e540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:54.921 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:54.909+0000 7fc52199a700 1 -- 172.21.15.2:0/4283065060 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc518005040 con 0x7fc51c100a10 2022-05-13T23:15:54.921 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:54.910+0000 7fc52199a700 1 --2- 172.21.15.2:0/4283065060 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc51c100a10 0x7fc51c10ea80 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fc50c00e910 tx=0x7fc50c00b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-05-13T23:15:54.923 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:54.910+0000 7fc512ffd700 1 -- 172.21.15.2:0/4283065060 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc50c007b40 con 0x7fc51c100a10 2022-05-13T23:15:54.924 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:54.910+0000 7fc52319d700 1 -- 172.21.15.2:0/4283065060 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc51c116600 con 0x7fc51c100a10 2022-05-13T23:15:54.924 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:54.910+0000 7fc512ffd700 1 -- 172.21.15.2:0/4283065060 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1087+0+0 (secure 0 0 0) 0x7fc50c00f040 con 0x7fc51c100a10 2022-05-13T23:15:54.924 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:54.910+0000 7fc512ffd700 1 -- 172.21.15.2:0/4283065060 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc50c0148b0 con 0x7fc51c100a10 2022-05-13T23:15:54.925 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:54.910+0000 7fc52319d700 1 -- 172.21.15.2:0/4283065060 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc51c116bc0 con 0x7fc51c100a10 2022-05-13T23:15:54.925 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:54.911+0000 7fc52319d700 1 -- 172.21.15.2:0/4283065060 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc51c108f10 con 0x7fc51c100a10 2022-05-13T23:15:54.925 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:54.913+0000 7fc512ffd700 1 -- 172.21.15.2:0/4283065060 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 16) v1 ==== 125136+0+0 (secure 0 0 0) 0x7fc50c01b030 con 0x7fc51c100a10 2022-05-13T23:15:54.926 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:54.914+0000 7fc512ffd700 1 --2- 172.21.15.2:0/4283065060 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7fc508096ce0 0x7fc508099190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:15:54.926 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:54.915+0000 7fc512ffd700 1 -- 172.21.15.2:0/4283065060 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8120+0+0 (secure 0 0 0) 0x7fc50c0bea80 con 0x7fc51c100a10 2022-05-13T23:15:54.926 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:54.915+0000 7fc52219b700 1 --2- 172.21.15.2:0/4283065060 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7fc508096ce0 0x7fc508099190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:15:54.926 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:54.915+0000 7fc52219b700 1 --2- 172.21.15.2:0/4283065060 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7fc508096ce0 0x7fc508099190 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fc518004840 tx=0x7fc518009cb0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-05-13T23:15:54.927 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:54.918+0000 7fc512ffd700 1 -- 172.21.15.2:0/4283065060 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+165215 (secure 0 0 0) 0x7fc50c04b570 con 0x7fc51c100a10 2022-05-13T23:15:55.116 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:55.112+0000 7fc52319d700 1 -- 172.21.15.2:0/4283065060 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fc51c1090a0 con 0x7fc51c100a10 2022-05-13T23:15:55.117 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:55.112+0000 7fc512ffd700 1 -- 172.21.15.2:0/4283065060 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v85) v1 ==== 100+0+11 (secure 0 0 0) 0x7fc50c019020 con 0x7fc51c100a10 2022-05-13T23:15:55.117 INFO:teuthology.orchestra.run.smithi002.stdout:pg_num: 64 2022-05-13T23:15:55.118 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:55.115+0000 7fc52319d700 1 -- 172.21.15.2:0/4283065060 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7fc508096ce0 msgr2=0x7fc508099190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:15:55.119 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:55.115+0000 7fc52319d700 1 --2- 172.21.15.2:0/4283065060 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7fc508096ce0 0x7fc508099190 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fc518004840 tx=0x7fc518009cb0 comp rx=0 tx=0).stop 2022-05-13T23:15:55.119 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:55.115+0000 7fc52319d700 1 -- 172.21.15.2:0/4283065060 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc51c100a10 msgr2=0x7fc51c10ea80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:15:55.119 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:55.115+0000 7fc52319d700 1 --2- 172.21.15.2:0/4283065060 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc51c100a10 0x7fc51c10ea80 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fc50c00e910 tx=0x7fc50c00b040 comp rx=0 tx=0).stop 2022-05-13T23:15:55.119 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:55.115+0000 7fc52319d700 1 -- 172.21.15.2:0/4283065060 shutdown_connections 2022-05-13T23:15:55.120 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:55.115+0000 7fc52319d700 1 --2- 172.21.15.2:0/4283065060 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fc51c103330 0x7fc51c115b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:55.120 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:55.115+0000 7fc52319d700 1 --2- 172.21.15.2:0/4283065060 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc51c065540 0x7fc51c10e540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:55.120 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:55.115+0000 7fc52319d700 1 --2- 172.21.15.2:0/4283065060 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7fc508096ce0 0x7fc508099190 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:55.120 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:55.115+0000 7fc52319d700 1 --2- 172.21.15.2:0/4283065060 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc51c100a10 0x7fc51c10ea80 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:55.120 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:55.115+0000 7fc52319d700 1 -- 172.21.15.2:0/4283065060 >> 172.21.15.2:0/4283065060 conn(0x7fc51c0f5b80 msgr2=0x7fc51c101650 unknown :-1 s=STATE_NONE l=0).mark_down 2022-05-13T23:15:55.121 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:55.115+0000 7fc52319d700 1 -- 172.21.15.2:0/4283065060 shutdown_connections 2022-05-13T23:15:55.121 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:55.115+0000 7fc52319d700 1 -- 172.21.15.2:0/4283065060 wait complete. 2022-05-13T23:15:55.142 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-05-13T23:15:55.361 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:55.360+0000 7f5c5d99d700 1 -- 172.21.15.2:0/618720209 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5c5806edb0 msgr2=0x7f5c5806f220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:15:55.362 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:55.360+0000 7f5c5d99d700 1 --2- 172.21.15.2:0/618720209 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5c5806edb0 0x7f5c5806f220 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f5c44004870 tx=0x7f5c44009cb0 comp rx=0 tx=0).stop 2022-05-13T23:15:55.363 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:55.360+0000 7f5c5d99d700 1 -- 172.21.15.2:0/618720209 shutdown_connections 2022-05-13T23:15:55.363 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:55.360+0000 7f5c5d99d700 1 --2- 172.21.15.2:0/618720209 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5c58103420 0x7f5c58105800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:55.363 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:55.360+0000 7f5c5d99d700 1 --2- 172.21.15.2:0/618720209 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f5c58070370 0x7f5c5806e870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:55.364 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:55.360+0000 7f5c5d99d700 1 --2- 172.21.15.2:0/618720209 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5c5806edb0 0x7f5c5806f220 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:55.365 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:55.360+0000 7f5c5d99d700 1 -- 172.21.15.2:0/618720209 >> 172.21.15.2:0/618720209 conn(0x7f5c580f6c40 msgr2=0x7f5c580f90b0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-05-13T23:15:55.365 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:55.360+0000 7f5c5d99d700 1 -- 172.21.15.2:0/618720209 shutdown_connections 2022-05-13T23:15:55.366 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:55.360+0000 7f5c5d99d700 1 -- 172.21.15.2:0/618720209 wait complete. 2022-05-13T23:15:55.369 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:55.361+0000 7f5c5d99d700 1 Processor -- start 2022-05-13T23:15:55.369 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:55.361+0000 7f5c5d99d700 1 -- start start 2022-05-13T23:15:55.370 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:55.361+0000 7f5c5d99d700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5c5806edb0 0x7f5c5810e4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:15:55.370 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:55.361+0000 7f5c5d99d700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f5c58070370 0x7f5c5810e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:15:55.370 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:55.361+0000 7f5c5d99d700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5c58103420 0x7f5c58115a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:15:55.371 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:55.361+0000 7f5c5d99d700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c58115fc0 con 0x7f5c5806edb0 2022-05-13T23:15:55.371 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:55.361+0000 7f5c5d99d700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c58116130 con 0x7f5c58103420 2022-05-13T23:15:55.371 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:55.361+0000 7f5c5d99d700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c581162f0 con 0x7f5c58070370 2022-05-13T23:15:55.372 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:55.361+0000 7f5c5c99b700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5c5806edb0 0x7f5c5810e4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:15:55.372 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:55.362+0000 7f5c5d19c700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5c58103420 0x7f5c58115a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:15:55.373 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:55.362+0000 7f5c57fff700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f5c58070370 0x7f5c5810e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:15:55.373 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:55.362+0000 7f5c5c99b700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5c5806edb0 0x7f5c5810e4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:53968/0 (socket says 172.21.15.2:53968) 2022-05-13T23:15:55.373 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:55.362+0000 7f5c57fff700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f5c58070370 0x7f5c5810e9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.2:45244/0 (socket says 172.21.15.2:45244) 2022-05-13T23:15:55.374 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:55.362+0000 7f5c57fff700 1 -- 172.21.15.2:0/2193157704 learned_addr learned my addr 172.21.15.2:0/2193157704 (peer_addr_for_me v2:172.21.15.2:0/0) 2022-05-13T23:15:55.374 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:55.362+0000 7f5c5d19c700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5c58103420 0x7f5c58115a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.2:37286/0 (socket says 172.21.15.2:37286) 2022-05-13T23:15:55.374 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:55.362+0000 7f5c5d19c700 1 -- 172.21.15.2:0/2193157704 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f5c58070370 msgr2=0x7f5c5810e9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:15:55.375 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:55.362+0000 7f5c5d19c700 1 --2- 172.21.15.2:0/2193157704 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f5c58070370 0x7f5c5810e9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:55.375 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:55.362+0000 7f5c5d19c700 1 -- 172.21.15.2:0/2193157704 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5c5806edb0 msgr2=0x7f5c5810e4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:15:55.375 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:55.362+0000 7f5c5d19c700 1 --2- 172.21.15.2:0/2193157704 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5c5806edb0 0x7f5c5810e4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:55.376 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:55.362+0000 7f5c5d19c700 1 -- 172.21.15.2:0/2193157704 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c44005040 con 0x7f5c58103420 2022-05-13T23:15:55.376 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:55.362+0000 7f5c57fff700 1 --2- 172.21.15.2:0/2193157704 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f5c58070370 0x7f5c5810e9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-05-13T23:15:55.376 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:55.362+0000 7f5c5d19c700 1 --2- 172.21.15.2:0/2193157704 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5c58103420 0x7f5c58115a80 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f5c58070140 tx=0x7f5c5000b040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-05-13T23:15:55.377 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:55.363+0000 7f5c557fa700 1 -- 172.21.15.2:0/2193157704 <== mon.1 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c50009870 con 0x7f5c58103420 2022-05-13T23:15:55.377 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:55.363+0000 7f5c557fa700 1 -- 172.21.15.2:0/2193157704 <== mon.1 v2:172.21.15.47:3300/0 2 ==== config(26 keys) v1 ==== 1087+0+0 (secure 0 0 0) 0x7f5c5000f040 con 0x7f5c58103420 2022-05-13T23:15:55.377 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:55.363+0000 7f5c5d99d700 1 -- 172.21.15.2:0/2193157704 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5c581165d0 con 0x7f5c58103420 2022-05-13T23:15:55.378 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:55.363+0000 7f5c557fa700 1 -- 172.21.15.2:0/2193157704 <== mon.1 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c50007e60 con 0x7f5c58103420 2022-05-13T23:15:55.378 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:55.363+0000 7f5c5d99d700 1 -- 172.21.15.2:0/2193157704 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5c58116af0 con 0x7f5c58103420 2022-05-13T23:15:55.378 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:55.364+0000 7f5c5d99d700 1 -- 172.21.15.2:0/2193157704 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5c58108f30 con 0x7f5c58103420 2022-05-13T23:15:55.379 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:55.366+0000 7f5c557fa700 1 -- 172.21.15.2:0/2193157704 <== mon.1 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 16) v1 ==== 125136+0+0 (secure 0 0 0) 0x7f5c5001a020 con 0x7f5c58103420 2022-05-13T23:15:55.380 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:55.367+0000 7f5c557fa700 1 --2- 172.21.15.2:0/2193157704 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f5c48096ce0 0x7f5c48099190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:15:55.380 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:55.367+0000 7f5c557fa700 1 -- 172.21.15.2:0/2193157704 <== mon.1 v2:172.21.15.47:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8120+0+0 (secure 0 0 0) 0x7f5c500bebf0 con 0x7f5c58103420 2022-05-13T23:15:55.380 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:55.367+0000 7f5c5c99b700 1 --2- 172.21.15.2:0/2193157704 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f5c48096ce0 0x7f5c48099190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:15:55.381 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:55.368+0000 7f5c5c99b700 1 --2- 172.21.15.2:0/2193157704 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f5c48096ce0 0x7f5c48099190 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f5c4c0048b0 tx=0x7f5c4c00b040 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-05-13T23:15:55.382 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:55.369+0000 7f5c557fa700 1 -- 172.21.15.2:0/2193157704 <== mon.1 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+165215 (secure 0 0 0) 0x7f5c5008bf90 con 0x7f5c58103420 2022-05-13T23:15:55.572 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:15:55 smithi002 ceph-mon[71816]: from='client.? 172.21.15.2:0/408053590' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2022-05-13T23:15:55.573 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:15:55 smithi002 ceph-mon[71816]: from='client.? 172.21.15.2:0/4283065060' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2022-05-13T23:15:55.623 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:55.602+0000 7f5c5d99d700 1 -- 172.21.15.2:0/2193157704 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5c581019f0 con 0x7f5c58103420 2022-05-13T23:15:55.624 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:55.603+0000 7f5c557fa700 1 -- 172.21.15.2:0/2193157704 <== mon.1 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) v1 ==== 94+0+10751 (secure 0 0 0) 0x7f5c5008f5b0 con 0x7f5c58103420 2022-05-13T23:15:55.625 INFO:teuthology.orchestra.run.smithi002.stdout: 2022-05-13T23:15:55.625 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24460,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.2:6835/1341618941","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6834","nonce":1341618941},{"type":"v1","addr":"172.21.15.2:6835","nonce":1341618941}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24490,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.2:6839/1347050951","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6838","nonce":1347050951},{"type":"v1","addr":"172.21.15.2:6839","nonce":1347050951}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4}],"filesystems":[{"mdsmap":{"epoch":26,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2022-05-13T23:14:16.236645+0000","modified":"2022-05-13T23:14:23.309447+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24611,"mds_1":24599,"mds_2":24587,"mds_3":24575,"mds_4":24547},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24494":{"gid":24494,"name":"d","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.2:6837/3754333215","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6836","nonce":3754333215},{"type":"v1","addr":"172.21.15.2:6837","nonce":3754333215}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24511":{"gid":24511,"name":"b","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.47:6833/2783249369","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6832","nonce":2783249369},{"type":"v1","addr":"172.21.15.47:6833","nonce":2783249369}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24523":{"gid":24523,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.47:6835/4075838054","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6834","nonce":4075838054},{"type":"v1","addr":"172.21.15.47:6835","nonce":4075838054}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24530":{"gid":24530,"name":"j","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.2:6841/522559555","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6840","nonce":522559555},{"type":"v1","addr":"172.21.15.2:6841","nonce":522559555}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24535":{"gid":24535,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.47:6837/4072863000","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6836","nonce":4072863000},{"type":"v1","addr":"172.21.15.47:6837","nonce":4072863000}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24547":{"gid":24547,"name":"k","rank":4,"incarnation":24,"state":"up:active","state_seq":7,"addr":"172.21.15.47:6839/761355251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6838","nonce":761355251},{"type":"v1","addr":"172.21.15.47:6839","nonce":761355251}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24575":{"gid":24575,"name":"c","rank":3,"incarnation":22,"state":"up:active","state_seq":6,"addr":"172.21.15.115:6833/381933929","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6832","nonce":381933929},{"type":"v1","addr":"172.21.15.115:6833","nonce":381933929}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24587":{"gid":24587,"name":"f","rank":2,"incarnation":20,"state":"up:active","state_seq":5,"addr":"172.21.15.115:6835/71062276","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6834","nonce":71062276},{"type":"v1","addr":"172.21.15.115:6835","nonce":71062276}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24599":{"gid":24599,"name":"i","rank":1,"incarnation":18,"state":"up:active","state_seq":4,"addr":"172.21.15.115:6837/2940957881","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6836","nonce":2940957881},{"type":"v1","addr":"172.21.15.115:6837","nonce":2940957881}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24611":{"gid":24611,"name":"l","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.115:6839/4149542923","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6838","nonce":4149542923},{"type":"v1","addr":"172.21.15.115:6839","nonce":4149542923}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-05-13T23:15:55.626 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:55.606+0000 7f5c5d99d700 1 -- 172.21.15.2:0/2193157704 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f5c48096ce0 msgr2=0x7f5c48099190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:15:55.627 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:55.606+0000 7f5c5d99d700 1 --2- 172.21.15.2:0/2193157704 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f5c48096ce0 0x7f5c48099190 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f5c4c0048b0 tx=0x7f5c4c00b040 comp rx=0 tx=0).stop 2022-05-13T23:15:55.627 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:55.606+0000 7f5c5d99d700 1 -- 172.21.15.2:0/2193157704 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5c58103420 msgr2=0x7f5c58115a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:15:55.627 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:55.606+0000 7f5c5d99d700 1 --2- 172.21.15.2:0/2193157704 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5c58103420 0x7f5c58115a80 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f5c58070140 tx=0x7f5c5000b040 comp rx=0 tx=0).stop 2022-05-13T23:15:55.627 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:55.606+0000 7f5c5d99d700 1 -- 172.21.15.2:0/2193157704 shutdown_connections 2022-05-13T23:15:55.627 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:55.606+0000 7f5c5d99d700 1 --2- 172.21.15.2:0/2193157704 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5c58103420 0x7f5c58115a80 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:55.628 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:55.606+0000 7f5c5d99d700 1 --2- 172.21.15.2:0/2193157704 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f5c48096ce0 0x7f5c48099190 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:55.628 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:55.606+0000 7f5c5d99d700 1 --2- 172.21.15.2:0/2193157704 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5c5806edb0 0x7f5c5810e4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:55.628 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:55.606+0000 7f5c5d99d700 1 --2- 172.21.15.2:0/2193157704 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f5c58070370 0x7f5c5810e9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:55.628 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:55.606+0000 7f5c5d99d700 1 -- 172.21.15.2:0/2193157704 >> 172.21.15.2:0/2193157704 conn(0x7f5c580f6c40 msgr2=0x7f5c580f9020 unknown :-1 s=STATE_NONE l=0).mark_down 2022-05-13T23:15:55.629 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:55.606+0000 7f5c5d99d700 1 -- 172.21.15.2:0/2193157704 shutdown_connections 2022-05-13T23:15:55.629 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:55.606+0000 7f5c5d99d700 1 -- 172.21.15.2:0/2193157704 wait complete. 2022-05-13T23:15:55.629 INFO:teuthology.orchestra.run.smithi002.stderr:dumped fsmap epoch 26 2022-05-13T23:15:55.637 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-05-13T23:15:55.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:15:55 smithi047 ceph-mon[77147]: from='client.? 172.21.15.2:0/408053590' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2022-05-13T23:15:55.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:15:55 smithi047 ceph-mon[77147]: from='client.? 172.21.15.2:0/4283065060' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2022-05-13T23:15:55.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:15:55 smithi115 ceph-mon[78593]: from='client.? 172.21.15.2:0/408053590' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2022-05-13T23:15:55.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:15:55 smithi115 ceph-mon[78593]: from='client.? 172.21.15.2:0/4283065060' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2022-05-13T23:15:55.868 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:55.854+0000 7f800bfff700 1 -- 172.21.15.2:0/118308758 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f800c0fe640 msgr2=0x7f800c100a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:15:55.868 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:55.854+0000 7f800bfff700 1 --2- 172.21.15.2:0/118308758 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f800c0fe640 0x7f800c100a80 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f8000004870 tx=0x7f8000009cb0 comp rx=0 tx=0).stop 2022-05-13T23:15:55.869 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:55.854+0000 7f800bfff700 1 -- 172.21.15.2:0/118308758 shutdown_connections 2022-05-13T23:15:55.874 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:55.854+0000 7f800bfff700 1 --2- 172.21.15.2:0/118308758 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f800c101110 0x7f800c103590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:55.874 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:55.854+0000 7f800bfff700 1 --2- 172.21.15.2:0/118308758 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f800c065630 0x7f800c0fe070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:55.875 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:55.854+0000 7f800bfff700 1 --2- 172.21.15.2:0/118308758 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f800c0fe640 0x7f800c100a80 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:55.875 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:55.854+0000 7f800bfff700 1 -- 172.21.15.2:0/118308758 >> 172.21.15.2:0/118308758 conn(0x7f800c0f5b80 msgr2=0x7f800c0f7fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-05-13T23:15:55.876 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:55.855+0000 7f800bfff700 1 -- 172.21.15.2:0/118308758 shutdown_connections 2022-05-13T23:15:55.876 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:55.855+0000 7f800bfff700 1 -- 172.21.15.2:0/118308758 wait complete. 2022-05-13T23:15:55.877 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:55.856+0000 7f800bfff700 1 Processor -- start 2022-05-13T23:15:55.877 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:55.856+0000 7f800bfff700 1 -- start start 2022-05-13T23:15:55.877 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:55.856+0000 7f800bfff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f800c065630 0x7f800c10e4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:15:55.878 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:55.856+0000 7f800bfff700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f800c0fe640 0x7f800c10ea00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:15:55.878 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:55.856+0000 7f800bfff700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f800c101110 0x7f800c115a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:15:55.879 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:55.856+0000 7f800bfff700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f800c115fd0 con 0x7f800c065630 2022-05-13T23:15:55.879 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:55.856+0000 7f800bfff700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f800c116140 con 0x7f800c0fe640 2022-05-13T23:15:55.879 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:55.856+0000 7f800bfff700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f800c116300 con 0x7f800c101110 2022-05-13T23:15:55.880 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:55.856+0000 7f800affd700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f800c065630 0x7f800c10e4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:15:55.880 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:55.856+0000 7f800affd700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f800c065630 0x7f800c10e4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:53984/0 (socket says 172.21.15.2:53984) 2022-05-13T23:15:55.880 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:55.856+0000 7f800affd700 1 -- 172.21.15.2:0/1446227772 learned_addr learned my addr 172.21.15.2:0/1446227772 (peer_addr_for_me v2:172.21.15.2:0/0) 2022-05-13T23:15:55.881 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:55.856+0000 7f800a7fc700 1 --2- 172.21.15.2:0/1446227772 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f800c0fe640 0x7f800c10ea00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:15:55.881 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:55.857+0000 7f800b7fe700 1 --2- 172.21.15.2:0/1446227772 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f800c101110 0x7f800c115a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:15:55.882 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:55.857+0000 7f800affd700 1 -- 172.21.15.2:0/1446227772 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f800c101110 msgr2=0x7f800c115a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:15:55.882 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:55.857+0000 7f800affd700 1 --2- 172.21.15.2:0/1446227772 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f800c101110 0x7f800c115a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:55.883 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:55.857+0000 7f800affd700 1 -- 172.21.15.2:0/1446227772 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f800c0fe640 msgr2=0x7f800c10ea00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:15:55.883 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:55.857+0000 7f800affd700 1 --2- 172.21.15.2:0/1446227772 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f800c0fe640 0x7f800c10ea00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:55.883 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:55.857+0000 7f800affd700 1 -- 172.21.15.2:0/1446227772 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ffc005040 con 0x7f800c065630 2022-05-13T23:15:55.884 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:55.857+0000 7f800a7fc700 1 --2- 172.21.15.2:0/1446227772 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f800c0fe640 0x7f800c10ea00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-05-13T23:15:55.884 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:55.857+0000 7f800affd700 1 --2- 172.21.15.2:0/1446227772 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f800c065630 0x7f800c10e4c0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f7ffc007fd0 tx=0x7f7ffc009cb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-05-13T23:15:55.884 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:55.857+0000 7f7ff37fe700 1 -- 172.21.15.2:0/1446227772 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ffc012570 con 0x7f800c065630 2022-05-13T23:15:55.884 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:55.857+0000 7f7ff37fe700 1 -- 172.21.15.2:0/1446227772 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1087+0+0 (secure 0 0 0) 0x7f7ffc00f040 con 0x7f800c065630 2022-05-13T23:15:55.885 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:55.857+0000 7f800bfff700 1 -- 172.21.15.2:0/1446227772 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8000005040 con 0x7f800c065630 2022-05-13T23:15:55.885 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:55.858+0000 7f7ff37fe700 1 -- 172.21.15.2:0/1446227772 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ffc013a40 con 0x7f800c065630 2022-05-13T23:15:55.886 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:55.858+0000 7f800bfff700 1 -- 172.21.15.2:0/1446227772 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f800c1169b0 con 0x7f800c065630 2022-05-13T23:15:55.886 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:55.859+0000 7f800bfff700 1 -- 172.21.15.2:0/1446227772 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f800c106c80 con 0x7f800c065630 2022-05-13T23:15:55.886 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:55.860+0000 7f7ff37fe700 1 -- 172.21.15.2:0/1446227772 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 16) v1 ==== 125136+0+0 (secure 0 0 0) 0x7f7ffc0126d0 con 0x7f800c065630 2022-05-13T23:15:55.887 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:55.862+0000 7f7ff37fe700 1 --2- 172.21.15.2:0/1446227772 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f7ff4096d60 0x7f7ff4099210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:15:55.887 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:55.862+0000 7f7ff37fe700 1 -- 172.21.15.2:0/1446227772 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8120+0+0 (secure 0 0 0) 0x7f7ffc0bef80 con 0x7f800c065630 2022-05-13T23:15:55.888 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:55.862+0000 7f800a7fc700 1 --2- 172.21.15.2:0/1446227772 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f7ff4096d60 0x7f7ff4099210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:15:55.888 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:55.863+0000 7f800a7fc700 1 --2- 172.21.15.2:0/1446227772 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f7ff4096d60 0x7f7ff4099210 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f8000004840 tx=0x7f8000009cb0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-05-13T23:15:55.888 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:55.865+0000 7f7ff37fe700 1 -- 172.21.15.2:0/1446227772 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+165215 (secure 0 0 0) 0x7f7ffc08c230 con 0x7f800c065630 2022-05-13T23:15:56.061 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:56.058+0000 7f800bfff700 1 -- 172.21.15.2:0/1446227772 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f800c0fb430 con 0x7f800c065630 2022-05-13T23:15:56.061 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:56.059+0000 7f7ff37fe700 1 -- 172.21.15.2:0/1446227772 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v85) v1 ==== 74+0+21505 (secure 0 0 0) 0x7f7ffc08f850 con 0x7f800c065630 2022-05-13T23:15:56.062 INFO:teuthology.orchestra.run.smithi002.stdout: 2022-05-13T23:15:56.062 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":85,"fsid":"16cf5f88-d311-11ec-be40-001a4aab830c","created":"2022-05-13T23:06:00.765453+0000","modified":"2022-05-13T23:15:06.402793+0000","last_up_change":"2022-05-13T23:13:20.935982+0000","last_in_change":"2022-05-13T23:13:02.796000+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2022-05-13T23:09:32.000644+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2022-05-13T23:14:14.195678+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":3,"pool_name":"cephfs_data","create_time":"2022-05-13T23:14:15.659212+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"83","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"11e700e8-f3e8-4903-b52c-db354b4395bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6802","nonce":1174843704},{"type":"v1","addr":"172.21.15.2:6803","nonce":1174843704}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6804","nonce":1174843704},{"type":"v1","addr":"172.21.15.2:6805","nonce":1174843704}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6808","nonce":1174843704},{"type":"v1","addr":"172.21.15.2:6809","nonce":1174843704}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6806","nonce":1174843704},{"type":"v1","addr":"172.21.15.2:6807","nonce":1174843704}]},"public_addr":"172.21.15.2:6803/1174843704","cluster_addr":"172.21.15.2:6805/1174843704","heartbeat_back_addr":"172.21.15.2:6809/1174843704","heartbeat_front_addr":"172.21.15.2:6807/1174843704","state":["exists","up"]},{"osd":1,"uuid":"eabfa3cf-f2a3-4995-a7de-f6d8ba02ee6f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6800","nonce":3853826948},{"type":"v1","addr":"172.21.15.47:6801","nonce":3853826948}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6802","nonce":3853826948},{"type":"v1","addr":"172.21.15.47:6803","nonce":3853826948}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6806","nonce":3853826948},{"type":"v1","addr":"172.21.15.47:6807","nonce":3853826948}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6804","nonce":3853826948},{"type":"v1","addr":"172.21.15.47:6805","nonce":3853826948}]},"public_addr":"172.21.15.47:6801/3853826948","cluster_addr":"172.21.15.47:6803/3853826948","heartbeat_back_addr":"172.21.15.47:6807/3853826948","heartbeat_front_addr":"172.21.15.47:6805/3853826948","state":["exists","up"]},{"osd":2,"uuid":"153c5bd5-8200-47a7-a135-8a95b6752cd2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6800","nonce":2562256956},{"type":"v1","addr":"172.21.15.115:6801","nonce":2562256956}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6802","nonce":2562256956},{"type":"v1","addr":"172.21.15.115:6803","nonce":2562256956}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6806","nonce":2562256956},{"type":"v1","addr":"172.21.15.115:6807","nonce":2562256956}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6804","nonce":2562256956},{"type":"v1","addr":"172.21.15.115:6805","nonce":2562256956}]},"public_addr":"172.21.15.115:6801/2562256956","cluster_addr":"172.21.15.115:6803/2562256956","heartbeat_back_addr":"172.21.15.115:6807/2562256956","heartbeat_front_addr":"172.21.15.115:6805/2562256956","state":["exists","up"]},{"osd":3,"uuid":"6ca9c602-5936-48da-a861-c437e6a33765","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6810","nonce":1099909411},{"type":"v1","addr":"172.21.15.2:6811","nonce":1099909411}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6812","nonce":1099909411},{"type":"v1","addr":"172.21.15.2:6813","nonce":1099909411}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6816","nonce":1099909411},{"type":"v1","addr":"172.21.15.2:6817","nonce":1099909411}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6814","nonce":1099909411},{"type":"v1","addr":"172.21.15.2:6815","nonce":1099909411}]},"public_addr":"172.21.15.2:6811/1099909411","cluster_addr":"172.21.15.2:6813/1099909411","heartbeat_back_addr":"172.21.15.2:6817/1099909411","heartbeat_front_addr":"172.21.15.2:6815/1099909411","state":["exists","up"]},{"osd":4,"uuid":"54d943d9-2610-422a-9e59-4b8212aced31","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6808","nonce":882753861},{"type":"v1","addr":"172.21.15.47:6809","nonce":882753861}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6810","nonce":882753861},{"type":"v1","addr":"172.21.15.47:6811","nonce":882753861}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6814","nonce":882753861},{"type":"v1","addr":"172.21.15.47:6815","nonce":882753861}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6812","nonce":882753861},{"type":"v1","addr":"172.21.15.47:6813","nonce":882753861}]},"public_addr":"172.21.15.47:6809/882753861","cluster_addr":"172.21.15.47:6811/882753861","heartbeat_back_addr":"172.21.15.47:6815/882753861","heartbeat_front_addr":"172.21.15.47:6813/882753861","state":["exists","up"]},{"osd":5,"uuid":"c18566cf-5403-4057-b2ee-aa298e87e8e9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6808","nonce":2122391993},{"type":"v1","addr":"172.21.15.115:6809","nonce":2122391993}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6810","nonce":2122391993},{"type":"v1","addr":"172.21.15.115:6811","nonce":2122391993}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6814","nonce":2122391993},{"type":"v1","addr":"172.21.15.115:6815","nonce":2122391993}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6812","nonce":2122391993},{"type":"v1","addr":"172.21.15.115:6813","nonce":2122391993}]},"public_addr":"172.21.15.115:6809/2122391993","cluster_addr":"172.21.15.115:6811/2122391993","heartbeat_back_addr":"172.21.15.115:6815/2122391993","heartbeat_front_addr":"172.21.15.115:6813/2122391993","state":["exists","up"]},{"osd":6,"uuid":"38ec7b6b-30f4-4d97-ad7e-e160ae59c843","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6818","nonce":1892198790},{"type":"v1","addr":"172.21.15.2:6819","nonce":1892198790}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6820","nonce":1892198790},{"type":"v1","addr":"172.21.15.2:6821","nonce":1892198790}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6824","nonce":1892198790},{"type":"v1","addr":"172.21.15.2:6825","nonce":1892198790}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6822","nonce":1892198790},{"type":"v1","addr":"172.21.15.2:6823","nonce":1892198790}]},"public_addr":"172.21.15.2:6819/1892198790","cluster_addr":"172.21.15.2:6821/1892198790","heartbeat_back_addr":"172.21.15.2:6825/1892198790","heartbeat_front_addr":"172.21.15.2:6823/1892198790","state":["exists","up"]},{"osd":7,"uuid":"e799b560-c61e-4369-b08b-45e5a6af37c0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6816","nonce":2812542825},{"type":"v1","addr":"172.21.15.47:6817","nonce":2812542825}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6818","nonce":2812542825},{"type":"v1","addr":"172.21.15.47:6819","nonce":2812542825}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6822","nonce":2812542825},{"type":"v1","addr":"172.21.15.47:6823","nonce":2812542825}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6820","nonce":2812542825},{"type":"v1","addr":"172.21.15.47:6821","nonce":2812542825}]},"public_addr":"172.21.15.47:6817/2812542825","cluster_addr":"172.21.15.47:6819/2812542825","heartbeat_back_addr":"172.21.15.47:6823/2812542825","heartbeat_front_addr":"172.21.15.47:6821/2812542825","state":["exists","up"]},{"osd":8,"uuid":"a58f2c0e-e487-41cf-8d2e-8c937cac9ebc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":58,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6816","nonce":3959632033},{"type":"v1","addr":"172.21.15.115:6817","nonce":3959632033}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6818","nonce":3959632033},{"type":"v1","addr":"172.21.15.115:6819","nonce":3959632033}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6822","nonce":3959632033},{"type":"v1","addr":"172.21.15.115:6823","nonce":3959632033}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6820","nonce":3959632033},{"type":"v1","addr":"172.21.15.115:6821","nonce":3959632033}]},"public_addr":"172.21.15.115:6817/3959632033","cluster_addr":"172.21.15.115:6819/3959632033","heartbeat_back_addr":"172.21.15.115:6823/3959632033","heartbeat_front_addr":"172.21.15.115:6821/3959632033","state":["exists","up"]},{"osd":9,"uuid":"12cead96-f726-4f7a-ab84-eb191b778a5c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":66,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6826","nonce":3004162255},{"type":"v1","addr":"172.21.15.2:6827","nonce":3004162255}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6828","nonce":3004162255},{"type":"v1","addr":"172.21.15.2:6829","nonce":3004162255}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6832","nonce":3004162255},{"type":"v1","addr":"172.21.15.2:6833","nonce":3004162255}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6830","nonce":3004162255},{"type":"v1","addr":"172.21.15.2:6831","nonce":3004162255}]},"public_addr":"172.21.15.2:6827/3004162255","cluster_addr":"172.21.15.2:6829/3004162255","heartbeat_back_addr":"172.21.15.2:6833/3004162255","heartbeat_front_addr":"172.21.15.2:6831/3004162255","state":["exists","up"]},{"osd":10,"uuid":"4f648118-7855-48d6-86d5-44eed5295f0c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":71,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6824","nonce":3443413998},{"type":"v1","addr":"172.21.15.47:6825","nonce":3443413998}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6826","nonce":3443413998},{"type":"v1","addr":"172.21.15.47:6827","nonce":3443413998}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6830","nonce":3443413998},{"type":"v1","addr":"172.21.15.47:6831","nonce":3443413998}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6828","nonce":3443413998},{"type":"v1","addr":"172.21.15.47:6829","nonce":3443413998}]},"public_addr":"172.21.15.47:6825/3443413998","cluster_addr":"172.21.15.47:6827/3443413998","heartbeat_back_addr":"172.21.15.47:6831/3443413998","heartbeat_front_addr":"172.21.15.47:6829/3443413998","state":["exists","up"]},{"osd":11,"uuid":"916541e0-fb28-46d5-9743-801c54c5a6d1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":78,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6824","nonce":1758718209},{"type":"v1","addr":"172.21.15.115:6825","nonce":1758718209}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6826","nonce":1758718209},{"type":"v1","addr":"172.21.15.115:6827","nonce":1758718209}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6830","nonce":1758718209},{"type":"v1","addr":"172.21.15.115:6831","nonce":1758718209}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6828","nonce":1758718209},{"type":"v1","addr":"172.21.15.115:6829","nonce":1758718209}]},"public_addr":"172.21.15.115:6825/1758718209","cluster_addr":"172.21.15.115:6827/1758718209","heartbeat_back_addr":"172.21.15.115:6831/1758718209","heartbeat_front_addr":"172.21.15.115:6829/1758718209","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-05-13T23:08:43.757857+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-05-13T23:09:06.537139+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-05-13T23:09:28.524837+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-05-13T23:10:02.057583+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-05-13T23:10:26.210648+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-05-13T23:10:47.885853+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-05-13T23:11:19.141865+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-05-13T23:11:42.376065+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-05-13T23:12:04.175870+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-05-13T23:12:34.695347+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-05-13T23:12:57.762613+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-05-13T23:13:19.114284+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.f","mappings":[{"from":8,"to":11}]},{"pgid":"2.13","mappings":[{"from":2,"to":11}]},{"pgid":"3.10","mappings":[{"from":5,"to":9}]},{"pgid":"3.2c","mappings":[{"from":4,"to":9}]},{"pgid":"3.36","mappings":[{"from":4,"to":9}]}],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.2:0/477249925":"2022-05-14T23:07:04.986087+0000","172.21.15.2:6800/3226099420":"2022-05-14T23:07:04.986087+0000","172.21.15.2:0/2214093915":"2022-05-14T23:07:04.986087+0000","172.21.15.2:0/1039403355":"2022-05-14T23:07:04.986087+0000","172.21.15.2:0/22091675":"2022-05-14T23:06:36.304108+0000","172.21.15.2:6800/1255134634":"2022-05-14T23:06:36.304108+0000","172.21.15.2:6801/3226099420":"2022-05-14T23:07:04.986087+0000","172.21.15.2:0/3768677276":"2022-05-14T23:06:36.304108+0000","172.21.15.2:6801/1255134634":"2022-05-14T23:06:36.304108+0000","172.21.15.2:0/407841268":"2022-05-14T23:06:36.304108+0000"},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2022-05-13T23:15:56.064 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:56.062+0000 7f800bfff700 1 -- 172.21.15.2:0/1446227772 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f7ff4096d60 msgr2=0x7f7ff4099210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:15:56.064 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:56.062+0000 7f800bfff700 1 --2- 172.21.15.2:0/1446227772 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f7ff4096d60 0x7f7ff4099210 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f8000004840 tx=0x7f8000009cb0 comp rx=0 tx=0).stop 2022-05-13T23:15:56.065 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:56.062+0000 7f800bfff700 1 -- 172.21.15.2:0/1446227772 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f800c065630 msgr2=0x7f800c10e4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:15:56.065 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:56.062+0000 7f800bfff700 1 --2- 172.21.15.2:0/1446227772 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f800c065630 0x7f800c10e4c0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f7ffc007fd0 tx=0x7f7ffc009cb0 comp rx=0 tx=0).stop 2022-05-13T23:15:56.065 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:56.062+0000 7f800bfff700 1 -- 172.21.15.2:0/1446227772 shutdown_connections 2022-05-13T23:15:56.065 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:56.062+0000 7f800bfff700 1 --2- 172.21.15.2:0/1446227772 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f800c101110 0x7f800c115a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:56.065 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:56.062+0000 7f800bfff700 1 --2- 172.21.15.2:0/1446227772 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f7ff4096d60 0x7f7ff4099210 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:56.066 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:56.062+0000 7f800bfff700 1 --2- 172.21.15.2:0/1446227772 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f800c065630 0x7f800c10e4c0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:56.066 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:56.062+0000 7f800bfff700 1 --2- 172.21.15.2:0/1446227772 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f800c0fe640 0x7f800c10ea00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:56.066 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:56.062+0000 7f800bfff700 1 -- 172.21.15.2:0/1446227772 >> 172.21.15.2:0/1446227772 conn(0x7f800c0f5b80 msgr2=0x7f800c101e70 unknown :-1 s=STATE_NONE l=0).mark_down 2022-05-13T23:15:56.066 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:56.062+0000 7f800bfff700 1 -- 172.21.15.2:0/1446227772 shutdown_connections 2022-05-13T23:15:56.067 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:56.063+0000 7f800bfff700 1 -- 172.21.15.2:0/1446227772 wait complete. 2022-05-13T23:15:56.094 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2022-05-13T23:09:32.000644+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2022-05-13T23:14:14.195678+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '82', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2022-05-13T23:14:15.659212+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '83', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2022-05-13T23:15:56.097 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2022-05-13T23:15:56.323 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:56.318+0000 7f2c9a99d700 1 -- 172.21.15.2:0/3954542855 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2c940ff2d0 msgr2=0x7f2c94105b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:15:56.323 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:56.318+0000 7f2c9a99d700 1 --2- 172.21.15.2:0/3954542855 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2c940ff2d0 0x7f2c94105b90 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f2c8c009be0 tx=0x7f2c8c004d10 comp rx=0 tx=0).stop 2022-05-13T23:15:56.324 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:56.318+0000 7f2c9a99d700 1 -- 172.21.15.2:0/3954542855 shutdown_connections 2022-05-13T23:15:56.324 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:56.318+0000 7f2c9a99d700 1 --2- 172.21.15.2:0/3954542855 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2c940ff2d0 0x7f2c94105b90 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:56.324 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:56.318+0000 7f2c9a99d700 1 --2- 172.21.15.2:0/3954542855 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f2c940fd6a0 0x7f2c940fdab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:56.324 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:56.318+0000 7f2c9a99d700 1 --2- 172.21.15.2:0/3954542855 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f2c940fe940 0x7f2c940fed90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:56.324 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:56.318+0000 7f2c9a99d700 1 -- 172.21.15.2:0/3954542855 >> 172.21.15.2:0/3954542855 conn(0x7f2c940f8e70 msgr2=0x7f2c940fb2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-05-13T23:15:56.325 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:56.318+0000 7f2c9a99d700 1 -- 172.21.15.2:0/3954542855 shutdown_connections 2022-05-13T23:15:56.325 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:56.318+0000 7f2c9a99d700 1 -- 172.21.15.2:0/3954542855 wait complete. 2022-05-13T23:15:56.328 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:56.319+0000 7f2c9a99d700 1 Processor -- start 2022-05-13T23:15:56.328 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:56.319+0000 7f2c9a99d700 1 -- start start 2022-05-13T23:15:56.328 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:56.320+0000 7f2c9a99d700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2c940fd6a0 0x7f2c940741f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:15:56.329 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:56.320+0000 7f2c9a99d700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f2c940fe940 0x7f2c94074730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:15:56.329 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:56.320+0000 7f2c9a99d700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f2c940ff2d0 0x7f2c94070d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:15:56.330 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:56.320+0000 7f2c9a99d700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c94071440 con 0x7f2c940fd6a0 2022-05-13T23:15:56.330 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:56.320+0000 7f2c9a99d700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c940715b0 con 0x7f2c940ff2d0 2022-05-13T23:15:56.331 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:56.320+0000 7f2c9a99d700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c94071770 con 0x7f2c940fe940 2022-05-13T23:15:56.331 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:56.320+0000 7f2c9999b700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2c940fd6a0 0x7f2c940741f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:15:56.332 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:56.320+0000 7f2c9999b700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2c940fd6a0 0x7f2c940741f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:53990/0 (socket says 172.21.15.2:53990) 2022-05-13T23:15:56.332 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:56.320+0000 7f2c9999b700 1 -- 172.21.15.2:0/1711768477 learned_addr learned my addr 172.21.15.2:0/1711768477 (peer_addr_for_me v2:172.21.15.2:0/0) 2022-05-13T23:15:56.332 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:56.320+0000 7f2c9919a700 1 --2- 172.21.15.2:0/1711768477 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f2c940fe940 0x7f2c94074730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:15:56.333 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:56.320+0000 7f2c9a19c700 1 --2- 172.21.15.2:0/1711768477 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f2c940ff2d0 0x7f2c94070d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:15:56.334 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:56.320+0000 7f2c9999b700 1 -- 172.21.15.2:0/1711768477 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f2c940fe940 msgr2=0x7f2c94074730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:15:56.334 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:56.320+0000 7f2c9999b700 1 --2- 172.21.15.2:0/1711768477 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f2c940fe940 0x7f2c94074730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:56.335 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:56.320+0000 7f2c9999b700 1 -- 172.21.15.2:0/1711768477 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f2c940ff2d0 msgr2=0x7f2c94070d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:15:56.335 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:56.320+0000 7f2c9999b700 1 --2- 172.21.15.2:0/1711768477 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f2c940ff2d0 0x7f2c94070d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:56.335 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:56.320+0000 7f2c9999b700 1 -- 172.21.15.2:0/1711768477 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c90005040 con 0x7f2c940fd6a0 2022-05-13T23:15:56.336 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:56.321+0000 7f2c9999b700 1 --2- 172.21.15.2:0/1711768477 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2c940fd6a0 0x7f2c940741f0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f2c900048f0 tx=0x7f2c90009cb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-05-13T23:15:56.336 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:56.321+0000 7f2c8a7fc700 1 -- 172.21.15.2:0/1711768477 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c90012570 con 0x7f2c940fd6a0 2022-05-13T23:15:56.336 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:56.321+0000 7f2c8a7fc700 1 -- 172.21.15.2:0/1711768477 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1087+0+0 (secure 0 0 0) 0x7f2c9000f040 con 0x7f2c940fd6a0 2022-05-13T23:15:56.337 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:56.321+0000 7f2c9a99d700 1 -- 172.21.15.2:0/1711768477 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2c8c0097c0 con 0x7f2c940fd6a0 2022-05-13T23:15:56.339 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:56.321+0000 7f2c8a7fc700 1 -- 172.21.15.2:0/1711768477 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c90013b50 con 0x7f2c940fd6a0 2022-05-13T23:15:56.340 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:56.321+0000 7f2c9a99d700 1 -- 172.21.15.2:0/1711768477 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2c9411abf0 con 0x7f2c940fd6a0 2022-05-13T23:15:56.345 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:56.322+0000 7f2c9a99d700 1 -- 172.21.15.2:0/1711768477 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2c94108eb0 con 0x7f2c940fd6a0 2022-05-13T23:15:56.346 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:56.324+0000 7f2c8a7fc700 1 -- 172.21.15.2:0/1711768477 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 16) v1 ==== 125136+0+0 (secure 0 0 0) 0x7f2c900126d0 con 0x7f2c940fd6a0 2022-05-13T23:15:56.346 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:56.325+0000 7f2c8a7fc700 1 --2- 172.21.15.2:0/1711768477 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f2c80096ce0 0x7f2c80099190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:15:56.347 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:56.325+0000 7f2c8a7fc700 1 -- 172.21.15.2:0/1711768477 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8120+0+0 (secure 0 0 0) 0x7f2c900befa0 con 0x7f2c940fd6a0 2022-05-13T23:15:56.347 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:56.326+0000 7f2c9919a700 1 --2- 172.21.15.2:0/1711768477 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f2c80096ce0 0x7f2c80099190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:15:56.348 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:56.326+0000 7f2c9919a700 1 --2- 172.21.15.2:0/1711768477 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f2c80096ce0 0x7f2c80099190 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f2c84004790 tx=0x7f2c8400b040 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-05-13T23:15:56.348 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:56.330+0000 7f2c8a7fc700 1 -- 172.21.15.2:0/1711768477 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+165215 (secure 0 0 0) 0x7f2c9008c250 con 0x7f2c940fd6a0 2022-05-13T23:15:56.521 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:56.519+0000 7f2c9a99d700 1 -- 172.21.15.2:0/1711768477 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f2c9402a490 con 0x7f2c940fd6a0 2022-05-13T23:15:56.521 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:56.520+0000 7f2c8a7fc700 1 -- 172.21.15.2:0/1711768477 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v85) v1 ==== 93+0+10 (secure 0 0 0) 0x7f2c9008f870 con 0x7f2c940fd6a0 2022-05-13T23:15:56.522 INFO:teuthology.orchestra.run.smithi002.stdout:pg_num: 1 2022-05-13T23:15:56.524 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:56.522+0000 7f2c9a99d700 1 -- 172.21.15.2:0/1711768477 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f2c80096ce0 msgr2=0x7f2c80099190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:15:56.525 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:56.522+0000 7f2c9a99d700 1 --2- 172.21.15.2:0/1711768477 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f2c80096ce0 0x7f2c80099190 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f2c84004790 tx=0x7f2c8400b040 comp rx=0 tx=0).stop 2022-05-13T23:15:56.525 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:56.522+0000 7f2c9a99d700 1 -- 172.21.15.2:0/1711768477 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2c940fd6a0 msgr2=0x7f2c940741f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:15:56.525 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:56.522+0000 7f2c9a99d700 1 --2- 172.21.15.2:0/1711768477 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2c940fd6a0 0x7f2c940741f0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f2c900048f0 tx=0x7f2c90009cb0 comp rx=0 tx=0).stop 2022-05-13T23:15:56.525 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:56.522+0000 7f2c9a99d700 1 -- 172.21.15.2:0/1711768477 shutdown_connections 2022-05-13T23:15:56.526 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:56.522+0000 7f2c9a99d700 1 --2- 172.21.15.2:0/1711768477 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f2c940ff2d0 0x7f2c94070d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:56.526 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:56.522+0000 7f2c9a99d700 1 --2- 172.21.15.2:0/1711768477 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f2c80096ce0 0x7f2c80099190 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:56.526 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:56.522+0000 7f2c9a99d700 1 --2- 172.21.15.2:0/1711768477 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2c940fd6a0 0x7f2c940741f0 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:56.526 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:56.522+0000 7f2c9a99d700 1 --2- 172.21.15.2:0/1711768477 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f2c940fe940 0x7f2c94074730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:56.527 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:56.522+0000 7f2c9a99d700 1 -- 172.21.15.2:0/1711768477 >> 172.21.15.2:0/1711768477 conn(0x7f2c940f8e70 msgr2=0x7f2c94101de0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-05-13T23:15:56.527 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:56.522+0000 7f2c9a99d700 1 -- 172.21.15.2:0/1711768477 shutdown_connections 2022-05-13T23:15:56.527 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:56.522+0000 7f2c9a99d700 1 -- 172.21.15.2:0/1711768477 wait complete. 2022-05-13T23:15:56.551 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2022-05-13T23:15:56.620 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:15:56 smithi002 ceph-mon[71816]: pgmap v340: 97 pgs: 97 active+clean; 584 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 11 op/s 2022-05-13T23:15:56.621 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:15:56 smithi002 ceph-mon[71816]: from='client.? 172.21.15.2:0/2193157704' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2022-05-13T23:15:56.621 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:15:56 smithi002 ceph-mon[71816]: from='client.? 172.21.15.2:0/1446227772' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2022-05-13T23:15:56.782 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:56.770+0000 7f7cc759e700 1 -- 172.21.15.2:0/2996208793 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7cc8065540 msgr2=0x7f7cc81004d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:15:56.784 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:56.770+0000 7f7cc759e700 1 --2- 172.21.15.2:0/2996208793 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7cc8065540 0x7f7cc81004d0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f7cb8004870 tx=0x7f7cb8009cb0 comp rx=0 tx=0).stop 2022-05-13T23:15:56.784 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:56.770+0000 7f7cc759e700 1 -- 172.21.15.2:0/2996208793 shutdown_connections 2022-05-13T23:15:56.785 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:56.770+0000 7f7cc759e700 1 --2- 172.21.15.2:0/2996208793 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7cc8103330 0x7f7cc8105790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:56.785 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:56.770+0000 7f7cc759e700 1 --2- 172.21.15.2:0/2996208793 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7cc8065540 0x7f7cc81004d0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:56.786 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:56.770+0000 7f7cc759e700 1 --2- 172.21.15.2:0/2996208793 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f7cc8100a10 0x7f7cc8102df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:56.786 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:56.770+0000 7f7cc759e700 1 -- 172.21.15.2:0/2996208793 >> 172.21.15.2:0/2996208793 conn(0x7f7cc80f5b80 msgr2=0x7f7cc80f7fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-05-13T23:15:56.786 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:56.770+0000 7f7cc759e700 1 -- 172.21.15.2:0/2996208793 shutdown_connections 2022-05-13T23:15:56.787 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:56.770+0000 7f7cc759e700 1 -- 172.21.15.2:0/2996208793 wait complete. 2022-05-13T23:15:56.787 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:56.771+0000 7f7cc759e700 1 Processor -- start 2022-05-13T23:15:56.788 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:56.771+0000 7f7cc759e700 1 -- start start 2022-05-13T23:15:56.789 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:56.772+0000 7f7cc759e700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7cc8065540 0x7f7cc810e560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:15:56.789 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:56.772+0000 7f7cc759e700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f7cc8100a10 0x7f7cc810eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:15:56.790 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:56.772+0000 7f7cc759e700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7cc8103330 0x7f7cc8115b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:15:56.790 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:56.772+0000 7f7cc759e700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7cc8116070 con 0x7f7cc8065540 2022-05-13T23:15:56.790 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:56.772+0000 7f7cc759e700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7cc81161e0 con 0x7f7cc8103330 2022-05-13T23:15:56.791 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:56.772+0000 7f7cc759e700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7cc81163a0 con 0x7f7cc8100a10 2022-05-13T23:15:56.791 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:56.772+0000 7f7cc659c700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7cc8065540 0x7f7cc810e560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:15:56.791 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:56.772+0000 7f7cc6d9d700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7cc8103330 0x7f7cc8115b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:15:56.791 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:56.772+0000 7f7cc5d9b700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f7cc8100a10 0x7f7cc810eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:15:56.792 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:56.772+0000 7f7cc6d9d700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7cc8103330 0x7f7cc8115b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.2:37338/0 (socket says 172.21.15.2:37338) 2022-05-13T23:15:56.792 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:56.772+0000 7f7cc6d9d700 1 -- 172.21.15.2:0/662866924 learned_addr learned my addr 172.21.15.2:0/662866924 (peer_addr_for_me v2:172.21.15.2:0/0) 2022-05-13T23:15:56.792 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:56.772+0000 7f7cc659c700 1 -- 172.21.15.2:0/662866924 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f7cc8100a10 msgr2=0x7f7cc810eaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:15:56.792 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:56.772+0000 7f7cc659c700 1 --2- 172.21.15.2:0/662866924 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f7cc8100a10 0x7f7cc810eaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:56.792 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:56.772+0000 7f7cc659c700 1 -- 172.21.15.2:0/662866924 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7cc8103330 msgr2=0x7f7cc8115b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:15:56.793 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:56.772+0000 7f7cc659c700 1 --2- 172.21.15.2:0/662866924 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7cc8103330 0x7f7cc8115b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:56.793 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:56.772+0000 7f7cc659c700 1 -- 172.21.15.2:0/662866924 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7cc00097c0 con 0x7f7cc8065540 2022-05-13T23:15:56.793 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:56.773+0000 7f7cc659c700 1 --2- 172.21.15.2:0/662866924 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7cc8065540 0x7f7cc810e560 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f7cb8004710 tx=0x7f7cb8009cb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-05-13T23:15:56.793 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:56.773+0000 7f7cb6ffd700 1 -- 172.21.15.2:0/662866924 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7cb8020070 con 0x7f7cc8065540 2022-05-13T23:15:56.794 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:56.773+0000 7f7cb6ffd700 1 -- 172.21.15.2:0/662866924 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1087+0+0 (secure 0 0 0) 0x7f7cb8004200 con 0x7f7cc8065540 2022-05-13T23:15:56.794 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:56.773+0000 7f7cc759e700 1 -- 172.21.15.2:0/662866924 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7cb8005040 con 0x7f7cc8065540 2022-05-13T23:15:56.794 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:56.773+0000 7f7cb6ffd700 1 -- 172.21.15.2:0/662866924 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7cb800e040 con 0x7f7cc8065540 2022-05-13T23:15:56.794 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:56.773+0000 7f7cc759e700 1 -- 172.21.15.2:0/662866924 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7cc8116910 con 0x7f7cc8065540 2022-05-13T23:15:56.794 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:56.774+0000 7f7cc759e700 1 -- 172.21.15.2:0/662866924 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7cc804ac70 con 0x7f7cc8065540 2022-05-13T23:15:56.795 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:56.776+0000 7f7cb6ffd700 1 -- 172.21.15.2:0/662866924 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 16) v1 ==== 125136+0+0 (secure 0 0 0) 0x7f7cb8009240 con 0x7f7cc8065540 2022-05-13T23:15:56.795 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:56.777+0000 7f7cb6ffd700 1 --2- 172.21.15.2:0/662866924 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f7cb0096ce0 0x7f7cb0099190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:15:56.795 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:56.777+0000 7f7cb6ffd700 1 -- 172.21.15.2:0/662866924 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8120+0+0 (secure 0 0 0) 0x7f7cb80bff70 con 0x7f7cc8065540 2022-05-13T23:15:56.795 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:56.777+0000 7f7cc5d9b700 1 --2- 172.21.15.2:0/662866924 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f7cb0096ce0 0x7f7cb0099190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:15:56.796 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:56.778+0000 7f7cc5d9b700 1 --2- 172.21.15.2:0/662866924 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f7cb0096ce0 0x7f7cb0099190 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f7cc8065990 tx=0x7f7cbc00b040 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-05-13T23:15:56.797 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:15:56 smithi047 ceph-mon[77147]: pgmap v340: 97 pgs: 97 active+clean; 584 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 11 op/s 2022-05-13T23:15:56.797 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:15:56 smithi047 ceph-mon[77147]: from='client.? 172.21.15.2:0/2193157704' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2022-05-13T23:15:56.797 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:15:56 smithi047 ceph-mon[77147]: from='client.? 172.21.15.2:0/1446227772' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2022-05-13T23:15:56.798 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:56.782+0000 7f7cb6ffd700 1 -- 172.21.15.2:0/662866924 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+165215 (secure 0 0 0) 0x7f7cb808d220 con 0x7f7cc8065540 2022-05-13T23:15:56.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:15:56 smithi115 ceph-mon[78593]: pgmap v340: 97 pgs: 97 active+clean; 584 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 11 op/s 2022-05-13T23:15:56.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:15:56 smithi115 ceph-mon[78593]: from='client.? 172.21.15.2:0/2193157704' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2022-05-13T23:15:56.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:15:56 smithi115 ceph-mon[78593]: from='client.? 172.21.15.2:0/1446227772' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2022-05-13T23:15:56.974 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:56.972+0000 7f7cc759e700 1 -- 172.21.15.2:0/662866924 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f7cc802a420 con 0x7f7cc8065540 2022-05-13T23:15:56.974 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:56.972+0000 7f7cb6ffd700 1 -- 172.21.15.2:0/662866924 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v85) v1 ==== 104+0+11 (secure 0 0 0) 0x7f7cb8090840 con 0x7f7cc8065540 2022-05-13T23:15:56.975 INFO:teuthology.orchestra.run.smithi002.stdout:pg_num: 32 2022-05-13T23:15:56.978 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:56.975+0000 7f7cc759e700 1 -- 172.21.15.2:0/662866924 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f7cb0096ce0 msgr2=0x7f7cb0099190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:15:56.978 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:56.975+0000 7f7cc759e700 1 --2- 172.21.15.2:0/662866924 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f7cb0096ce0 0x7f7cb0099190 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f7cc8065990 tx=0x7f7cbc00b040 comp rx=0 tx=0).stop 2022-05-13T23:15:56.979 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:56.975+0000 7f7cc759e700 1 -- 172.21.15.2:0/662866924 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7cc8065540 msgr2=0x7f7cc810e560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:15:56.979 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:56.975+0000 7f7cc759e700 1 --2- 172.21.15.2:0/662866924 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7cc8065540 0x7f7cc810e560 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f7cb8004710 tx=0x7f7cb8009cb0 comp rx=0 tx=0).stop 2022-05-13T23:15:56.980 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:56.975+0000 7f7cc759e700 1 -- 172.21.15.2:0/662866924 shutdown_connections 2022-05-13T23:15:56.980 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:56.975+0000 7f7cc759e700 1 --2- 172.21.15.2:0/662866924 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7cc8103330 0x7f7cc8115b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:56.981 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:56.975+0000 7f7cc759e700 1 --2- 172.21.15.2:0/662866924 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f7cb0096ce0 0x7f7cb0099190 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:56.981 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:56.975+0000 7f7cc759e700 1 --2- 172.21.15.2:0/662866924 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7cc8065540 0x7f7cc810e560 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:56.982 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:56.975+0000 7f7cc759e700 1 --2- 172.21.15.2:0/662866924 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f7cc8100a10 0x7f7cc810eaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:56.982 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:56.975+0000 7f7cc759e700 1 -- 172.21.15.2:0/662866924 >> 172.21.15.2:0/662866924 conn(0x7f7cc80f5b80 msgr2=0x7f7cc8101650 unknown :-1 s=STATE_NONE l=0).mark_down 2022-05-13T23:15:56.983 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:56.975+0000 7f7cc759e700 1 -- 172.21.15.2:0/662866924 shutdown_connections 2022-05-13T23:15:56.983 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:56.975+0000 7f7cc759e700 1 -- 172.21.15.2:0/662866924 wait complete. 2022-05-13T23:15:57.011 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2022-05-13T23:15:57.233 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:57.231+0000 7f8a8539d700 1 -- 172.21.15.2:0/1648888698 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8a80101110 msgr2=0x7f8a80103590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:15:57.234 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:57.231+0000 7f8a8539d700 1 --2- 172.21.15.2:0/1648888698 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8a80101110 0x7f8a80103590 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f8a78009c30 tx=0x7f8a78004c50 comp rx=0 tx=0).stop 2022-05-13T23:15:57.234 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:57.231+0000 7f8a8539d700 1 -- 172.21.15.2:0/1648888698 shutdown_connections 2022-05-13T23:15:57.234 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:57.231+0000 7f8a8539d700 1 --2- 172.21.15.2:0/1648888698 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8a80101110 0x7f8a80103590 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:57.235 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:57.231+0000 7f8a8539d700 1 --2- 172.21.15.2:0/1648888698 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8a80065560 0x7f8a800fe0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:57.235 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:57.231+0000 7f8a8539d700 1 --2- 172.21.15.2:0/1648888698 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f8a800fe680 0x7f8a80100a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:57.235 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:57.231+0000 7f8a8539d700 1 -- 172.21.15.2:0/1648888698 >> 172.21.15.2:0/1648888698 conn(0x7f8a800f5b80 msgr2=0x7f8a800f7fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-05-13T23:15:57.235 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:57.231+0000 7f8a8539d700 1 -- 172.21.15.2:0/1648888698 shutdown_connections 2022-05-13T23:15:57.235 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:57.231+0000 7f8a8539d700 1 -- 172.21.15.2:0/1648888698 wait complete. 2022-05-13T23:15:57.236 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:57.232+0000 7f8a8539d700 1 Processor -- start 2022-05-13T23:15:57.236 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:57.232+0000 7f8a8539d700 1 -- start start 2022-05-13T23:15:57.238 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:57.233+0000 7f8a8539d700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8a80065560 0x7f8a8010e3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:15:57.239 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:57.233+0000 7f8a8539d700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f8a800fe680 0x7f8a8010e8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:15:57.239 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:57.233+0000 7f8a8539d700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8a80101110 0x7f8a80115970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:15:57.239 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:57.233+0000 7f8a8539d700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a80115eb0 con 0x7f8a80065560 2022-05-13T23:15:57.239 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:57.233+0000 7f8a8539d700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a80116020 con 0x7f8a80101110 2022-05-13T23:15:57.240 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:57.233+0000 7f8a8539d700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a801161e0 con 0x7f8a800fe680 2022-05-13T23:15:57.240 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:57.233+0000 7f8a84b9c700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8a80101110 0x7f8a80115970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:15:57.240 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:57.233+0000 7f8a7f7fe700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f8a800fe680 0x7f8a8010e8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:15:57.240 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:57.233+0000 7f8a7ffff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8a80065560 0x7f8a8010e3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:15:57.240 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:57.233+0000 7f8a84b9c700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8a80101110 0x7f8a80115970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.2:37350/0 (socket says 172.21.15.2:37350) 2022-05-13T23:15:57.241 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:57.233+0000 7f8a7f7fe700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f8a800fe680 0x7f8a8010e8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.2:45348/0 (socket says 172.21.15.2:45348) 2022-05-13T23:15:57.241 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:57.233+0000 7f8a84b9c700 1 -- 172.21.15.2:0/2968696232 learned_addr learned my addr 172.21.15.2:0/2968696232 (peer_addr_for_me v2:172.21.15.2:0/0) 2022-05-13T23:15:57.241 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:57.234+0000 7f8a7f7fe700 1 -- 172.21.15.2:0/2968696232 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8a80101110 msgr2=0x7f8a80115970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:15:57.241 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:57.234+0000 7f8a7f7fe700 1 --2- 172.21.15.2:0/2968696232 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8a80101110 0x7f8a80115970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:57.242 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:57.234+0000 7f8a7f7fe700 1 -- 172.21.15.2:0/2968696232 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8a80065560 msgr2=0x7f8a8010e3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:15:57.242 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:57.234+0000 7f8a7f7fe700 1 --2- 172.21.15.2:0/2968696232 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8a80065560 0x7f8a8010e3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:57.242 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:57.234+0000 7f8a7f7fe700 1 -- 172.21.15.2:0/2968696232 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a780097c0 con 0x7f8a800fe680 2022-05-13T23:15:57.242 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:57.234+0000 7f8a7ffff700 1 --2- 172.21.15.2:0/2968696232 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8a80065560 0x7f8a8010e3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-05-13T23:15:57.242 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:57.234+0000 7f8a7f7fe700 1 --2- 172.21.15.2:0/2968696232 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f8a800fe680 0x7f8a8010e8e0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f8a80065330 tx=0x7f8a7400b040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-05-13T23:15:57.243 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:57.234+0000 7f8a7cff9700 1 -- 172.21.15.2:0/2968696232 <== mon.2 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a74007b40 con 0x7f8a800fe680 2022-05-13T23:15:57.243 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:57.234+0000 7f8a8539d700 1 -- 172.21.15.2:0/2968696232 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8a80116460 con 0x7f8a800fe680 2022-05-13T23:15:57.243 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:57.234+0000 7f8a7cff9700 1 -- 172.21.15.2:0/2968696232 <== mon.2 v2:172.21.15.115:3300/0 2 ==== config(26 keys) v1 ==== 1087+0+0 (secure 0 0 0) 0x7f8a7400f040 con 0x7f8a800fe680 2022-05-13T23:15:57.243 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:57.234+0000 7f8a7cff9700 1 -- 172.21.15.2:0/2968696232 <== mon.2 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a740147f0 con 0x7f8a800fe680 2022-05-13T23:15:57.244 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:57.234+0000 7f8a8539d700 1 -- 172.21.15.2:0/2968696232 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8a80116a20 con 0x7f8a800fe680 2022-05-13T23:15:57.244 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:57.235+0000 7f8a8539d700 1 -- 172.21.15.2:0/2968696232 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8a80106d10 con 0x7f8a800fe680 2022-05-13T23:15:57.244 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:57.237+0000 7f8a7cff9700 1 -- 172.21.15.2:0/2968696232 <== mon.2 v2:172.21.15.115:3300/0 4 ==== mgrmap(e 16) v1 ==== 125136+0+0 (secure 0 0 0) 0x7f8a7401b030 con 0x7f8a800fe680 2022-05-13T23:15:57.245 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:57.238+0000 7f8a7cff9700 1 --2- 172.21.15.2:0/2968696232 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f8a68096ce0 0x7f8a68099190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:15:57.246 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:57.239+0000 7f8a7cff9700 1 -- 172.21.15.2:0/2968696232 <== mon.2 v2:172.21.15.115:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8120+0+0 (secure 0 0 0) 0x7f8a740bea30 con 0x7f8a800fe680 2022-05-13T23:15:57.246 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:57.239+0000 7f8a7ffff700 1 --2- 172.21.15.2:0/2968696232 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f8a68096ce0 0x7f8a68099190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:15:57.246 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:57.239+0000 7f8a7ffff700 1 --2- 172.21.15.2:0/2968696232 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f8a68096ce0 0x7f8a68099190 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f8a70004fd0 tx=0x7f8a7000b040 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-05-13T23:15:57.246 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:57.241+0000 7f8a7cff9700 1 -- 172.21.15.2:0/2968696232 <== mon.2 v2:172.21.15.115:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+165215 (secure 0 0 0) 0x7f8a7408bce0 con 0x7f8a800fe680 2022-05-13T23:15:57.436 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:57.433+0000 7f8a8539d700 1 -- 172.21.15.2:0/2968696232 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f8a80116c40 con 0x7f8a800fe680 2022-05-13T23:15:57.436 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:57.434+0000 7f8a7cff9700 1 -- 172.21.15.2:0/2968696232 <== mon.2 v2:172.21.15.115:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v85) v1 ==== 100+0+11 (secure 0 0 0) 0x7f8a7408f300 con 0x7f8a800fe680 2022-05-13T23:15:57.438 INFO:teuthology.orchestra.run.smithi002.stdout:pg_num: 64 2022-05-13T23:15:57.440 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:57.437+0000 7f8a8539d700 1 -- 172.21.15.2:0/2968696232 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f8a68096ce0 msgr2=0x7f8a68099190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:15:57.440 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:57.437+0000 7f8a8539d700 1 --2- 172.21.15.2:0/2968696232 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f8a68096ce0 0x7f8a68099190 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f8a70004fd0 tx=0x7f8a7000b040 comp rx=0 tx=0).stop 2022-05-13T23:15:57.441 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:57.437+0000 7f8a8539d700 1 -- 172.21.15.2:0/2968696232 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f8a800fe680 msgr2=0x7f8a8010e8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:15:57.446 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:57.437+0000 7f8a8539d700 1 --2- 172.21.15.2:0/2968696232 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f8a800fe680 0x7f8a8010e8e0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f8a80065330 tx=0x7f8a7400b040 comp rx=0 tx=0).stop 2022-05-13T23:15:57.447 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:57.437+0000 7f8a8539d700 1 -- 172.21.15.2:0/2968696232 shutdown_connections 2022-05-13T23:15:57.447 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:57.437+0000 7f8a8539d700 1 --2- 172.21.15.2:0/2968696232 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8a80101110 0x7f8a80115970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:57.448 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:57.437+0000 7f8a8539d700 1 --2- 172.21.15.2:0/2968696232 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f8a68096ce0 0x7f8a68099190 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:57.448 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:57.437+0000 7f8a8539d700 1 --2- 172.21.15.2:0/2968696232 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8a80065560 0x7f8a8010e3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:57.449 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:57.437+0000 7f8a8539d700 1 --2- 172.21.15.2:0/2968696232 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f8a800fe680 0x7f8a8010e8e0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:57.450 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:57.437+0000 7f8a8539d700 1 -- 172.21.15.2:0/2968696232 >> 172.21.15.2:0/2968696232 conn(0x7f8a800f5b80 msgr2=0x7f8a800f7f60 unknown :-1 s=STATE_NONE l=0).mark_down 2022-05-13T23:15:57.450 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:57.437+0000 7f8a8539d700 1 -- 172.21.15.2:0/2968696232 shutdown_connections 2022-05-13T23:15:57.450 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:57.437+0000 7f8a8539d700 1 -- 172.21.15.2:0/2968696232 wait complete. 2022-05-13T23:15:57.466 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-05-13T23:15:57.684 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:57.683+0000 7fe95d79d700 1 -- 172.21.15.2:0/2725735229 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe9580fc780 msgr2=0x7fe9580fcbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:15:57.685 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:57.683+0000 7fe95d79d700 1 --2- 172.21.15.2:0/2725735229 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe9580fc780 0x7fe9580fcbd0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fe94c004870 tx=0x7fe94c009cb0 comp rx=0 tx=0).stop 2022-05-13T23:15:57.686 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:57.683+0000 7fe95d79d700 1 -- 172.21.15.2:0/2725735229 shutdown_connections 2022-05-13T23:15:57.687 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:57.683+0000 7fe95d79d700 1 --2- 172.21.15.2:0/2725735229 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe9580fd110 0x7fe9581039d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:57.687 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:57.683+0000 7fe95d79d700 1 --2- 172.21.15.2:0/2725735229 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe9580fb4e0 0x7fe9580fb8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:57.687 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:57.683+0000 7fe95d79d700 1 --2- 172.21.15.2:0/2725735229 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe9580fc780 0x7fe9580fcbd0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:57.687 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:57.683+0000 7fe95d79d700 1 -- 172.21.15.2:0/2725735229 >> 172.21.15.2:0/2725735229 conn(0x7fe9580f6cb0 msgr2=0x7fe9580f9100 unknown :-1 s=STATE_NONE l=0).mark_down 2022-05-13T23:15:57.688 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:57.683+0000 7fe95d79d700 1 -- 172.21.15.2:0/2725735229 shutdown_connections 2022-05-13T23:15:57.688 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:57.683+0000 7fe95d79d700 1 -- 172.21.15.2:0/2725735229 wait complete. 2022-05-13T23:15:57.690 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:57.684+0000 7fe95d79d700 1 Processor -- start 2022-05-13T23:15:57.691 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:57.684+0000 7fe95d79d700 1 -- start start 2022-05-13T23:15:57.691 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:57.684+0000 7fe95d79d700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe9580fb4e0 0x7fe9580701d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:15:57.691 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:57.684+0000 7fe95d79d700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe9580fc780 0x7fe95806e830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:15:57.691 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:57.684+0000 7fe95d79d700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe9580fd110 0x7fe95806ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:15:57.692 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:57.684+0000 7fe95d79d700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe95806f480 con 0x7fe9580fc780 2022-05-13T23:15:57.692 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:57.684+0000 7fe95d79d700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe95806f5f0 con 0x7fe9580fd110 2022-05-13T23:15:57.692 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:57.684+0000 7fe95d79d700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9581163e0 con 0x7fe9580fb4e0 2022-05-13T23:15:57.692 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:57.685+0000 7fe957fff700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe9580fb4e0 0x7fe9580701d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:15:57.693 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:57.685+0000 7fe9577fe700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe9580fc780 0x7fe95806e830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:15:57.693 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:57.685+0000 7fe957fff700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe9580fb4e0 0x7fe9580701d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.2:45362/0 (socket says 172.21.15.2:45362) 2022-05-13T23:15:57.693 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:57.685+0000 7fe957fff700 1 -- 172.21.15.2:0/1321215985 learned_addr learned my addr 172.21.15.2:0/1321215985 (peer_addr_for_me v2:172.21.15.2:0/0) 2022-05-13T23:15:57.693 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:57.685+0000 7fe95cf9c700 1 --2- 172.21.15.2:0/1321215985 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe9580fd110 0x7fe95806ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:15:57.694 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:57.685+0000 7fe957fff700 1 -- 172.21.15.2:0/1321215985 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe9580fd110 msgr2=0x7fe95806ed70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:15:57.694 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:57.685+0000 7fe957fff700 1 --2- 172.21.15.2:0/1321215985 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe9580fd110 0x7fe95806ed70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:57.694 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:57.685+0000 7fe957fff700 1 -- 172.21.15.2:0/1321215985 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe9580fc780 msgr2=0x7fe95806e830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:15:57.694 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:57.685+0000 7fe957fff700 1 --2- 172.21.15.2:0/1321215985 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe9580fc780 0x7fe95806e830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:57.695 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:57.685+0000 7fe957fff700 1 -- 172.21.15.2:0/1321215985 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe94c005040 con 0x7fe9580fb4e0 2022-05-13T23:15:57.695 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:57.685+0000 7fe957fff700 1 --2- 172.21.15.2:0/1321215985 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe9580fb4e0 0x7fe9580701d0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fe94800e940 tx=0x7fe94800b040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-05-13T23:15:57.695 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:57.686+0000 7fe954ff9700 1 -- 172.21.15.2:0/1321215985 <== mon.2 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe948007b40 con 0x7fe9580fb4e0 2022-05-13T23:15:57.695 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:57.686+0000 7fe95d79d700 1 -- 172.21.15.2:0/1321215985 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe958116580 con 0x7fe9580fb4e0 2022-05-13T23:15:57.696 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:57.686+0000 7fe954ff9700 1 -- 172.21.15.2:0/1321215985 <== mon.2 v2:172.21.15.115:3300/0 2 ==== config(26 keys) v1 ==== 1087+0+0 (secure 0 0 0) 0x7fe94800f040 con 0x7fe9580fb4e0 2022-05-13T23:15:57.696 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:57.686+0000 7fe954ff9700 1 -- 172.21.15.2:0/1321215985 <== mon.2 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe948014890 con 0x7fe9580fb4e0 2022-05-13T23:15:57.696 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:57.686+0000 7fe95d79d700 1 -- 172.21.15.2:0/1321215985 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe958116ae0 con 0x7fe9580fb4e0 2022-05-13T23:15:57.696 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:57.687+0000 7fe95d79d700 1 -- 172.21.15.2:0/1321215985 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe958106d20 con 0x7fe9580fb4e0 2022-05-13T23:15:57.697 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:57.689+0000 7fe954ff9700 1 -- 172.21.15.2:0/1321215985 <== mon.2 v2:172.21.15.115:3300/0 4 ==== mgrmap(e 16) v1 ==== 125136+0+0 (secure 0 0 0) 0x7fe94801b030 con 0x7fe9580fb4e0 2022-05-13T23:15:57.698 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:57.690+0000 7fe954ff9700 1 --2- 172.21.15.2:0/1321215985 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7fe940096ce0 0x7fe940099190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:15:57.699 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:57.690+0000 7fe954ff9700 1 -- 172.21.15.2:0/1321215985 <== mon.2 v2:172.21.15.115:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8120+0+0 (secure 0 0 0) 0x7fe9480bef20 con 0x7fe9580fb4e0 2022-05-13T23:15:57.699 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:57.690+0000 7fe9577fe700 1 --2- 172.21.15.2:0/1321215985 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7fe940096ce0 0x7fe940099190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:15:57.699 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:57.691+0000 7fe9577fe700 1 --2- 172.21.15.2:0/1321215985 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7fe940096ce0 0x7fe940099190 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fe94c004840 tx=0x7fe94c009cb0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-05-13T23:15:57.699 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:57.692+0000 7fe954ff9700 1 -- 172.21.15.2:0/1321215985 <== mon.2 v2:172.21.15.115:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+165215 (secure 0 0 0) 0x7fe94804b570 con 0x7fe9580fb4e0 2022-05-13T23:15:57.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:15:57 smithi047 ceph-mon[77147]: from='client.? 172.21.15.2:0/1711768477' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2022-05-13T23:15:57.795 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:15:57 smithi047 ceph-mon[77147]: from='client.? 172.21.15.2:0/662866924' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2022-05-13T23:15:57.823 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:15:57 smithi002 ceph-mon[71816]: from='client.? 172.21.15.2:0/1711768477' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2022-05-13T23:15:57.823 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:15:57 smithi002 ceph-mon[71816]: from='client.? 172.21.15.2:0/662866924' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2022-05-13T23:15:57.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:15:57 smithi115 ceph-mon[78593]: from='client.? 172.21.15.2:0/1711768477' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2022-05-13T23:15:57.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:15:57 smithi115 ceph-mon[78593]: from='client.? 172.21.15.2:0/662866924' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2022-05-13T23:15:57.930 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:57.927+0000 7fe95d79d700 1 -- 172.21.15.2:0/1321215985 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe958102440 con 0x7fe9580fb4e0 2022-05-13T23:15:57.931 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:57.928+0000 7fe954ff9700 1 -- 172.21.15.2:0/1321215985 <== mon.2 v2:172.21.15.115:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) v1 ==== 94+0+10751 (secure 0 0 0) 0x7fe948019090 con 0x7fe9580fb4e0 2022-05-13T23:15:57.932 INFO:teuthology.orchestra.run.smithi002.stdout: 2022-05-13T23:15:57.932 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24460,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.2:6835/1341618941","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6834","nonce":1341618941},{"type":"v1","addr":"172.21.15.2:6835","nonce":1341618941}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24490,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.2:6839/1347050951","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6838","nonce":1347050951},{"type":"v1","addr":"172.21.15.2:6839","nonce":1347050951}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4}],"filesystems":[{"mdsmap":{"epoch":26,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2022-05-13T23:14:16.236645+0000","modified":"2022-05-13T23:14:23.309447+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24611,"mds_1":24599,"mds_2":24587,"mds_3":24575,"mds_4":24547},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24494":{"gid":24494,"name":"d","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.2:6837/3754333215","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6836","nonce":3754333215},{"type":"v1","addr":"172.21.15.2:6837","nonce":3754333215}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24511":{"gid":24511,"name":"b","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.47:6833/2783249369","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6832","nonce":2783249369},{"type":"v1","addr":"172.21.15.47:6833","nonce":2783249369}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24523":{"gid":24523,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.47:6835/4075838054","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6834","nonce":4075838054},{"type":"v1","addr":"172.21.15.47:6835","nonce":4075838054}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24530":{"gid":24530,"name":"j","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.2:6841/522559555","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6840","nonce":522559555},{"type":"v1","addr":"172.21.15.2:6841","nonce":522559555}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24535":{"gid":24535,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.47:6837/4072863000","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6836","nonce":4072863000},{"type":"v1","addr":"172.21.15.47:6837","nonce":4072863000}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24547":{"gid":24547,"name":"k","rank":4,"incarnation":24,"state":"up:active","state_seq":7,"addr":"172.21.15.47:6839/761355251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6838","nonce":761355251},{"type":"v1","addr":"172.21.15.47:6839","nonce":761355251}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24575":{"gid":24575,"name":"c","rank":3,"incarnation":22,"state":"up:active","state_seq":6,"addr":"172.21.15.115:6833/381933929","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6832","nonce":381933929},{"type":"v1","addr":"172.21.15.115:6833","nonce":381933929}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24587":{"gid":24587,"name":"f","rank":2,"incarnation":20,"state":"up:active","state_seq":5,"addr":"172.21.15.115:6835/71062276","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6834","nonce":71062276},{"type":"v1","addr":"172.21.15.115:6835","nonce":71062276}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24599":{"gid":24599,"name":"i","rank":1,"incarnation":18,"state":"up:active","state_seq":4,"addr":"172.21.15.115:6837/2940957881","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6836","nonce":2940957881},{"type":"v1","addr":"172.21.15.115:6837","nonce":2940957881}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24611":{"gid":24611,"name":"l","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.115:6839/4149542923","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6838","nonce":4149542923},{"type":"v1","addr":"172.21.15.115:6839","nonce":4149542923}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-05-13T23:15:57.934 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:57.931+0000 7fe95d79d700 1 -- 172.21.15.2:0/1321215985 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7fe940096ce0 msgr2=0x7fe940099190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:15:57.935 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:57.931+0000 7fe95d79d700 1 --2- 172.21.15.2:0/1321215985 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7fe940096ce0 0x7fe940099190 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fe94c004840 tx=0x7fe94c009cb0 comp rx=0 tx=0).stop 2022-05-13T23:15:57.935 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:57.931+0000 7fe95d79d700 1 -- 172.21.15.2:0/1321215985 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe9580fb4e0 msgr2=0x7fe9580701d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:15:57.935 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:57.931+0000 7fe95d79d700 1 --2- 172.21.15.2:0/1321215985 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe9580fb4e0 0x7fe9580701d0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fe94800e940 tx=0x7fe94800b040 comp rx=0 tx=0).stop 2022-05-13T23:15:57.936 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:57.931+0000 7fe95d79d700 1 -- 172.21.15.2:0/1321215985 shutdown_connections 2022-05-13T23:15:57.936 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:57.931+0000 7fe95d79d700 1 --2- 172.21.15.2:0/1321215985 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe9580fd110 0x7fe95806ed70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:57.936 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:57.931+0000 7fe95d79d700 1 --2- 172.21.15.2:0/1321215985 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fe9580fb4e0 0x7fe9580701d0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:57.936 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:57.931+0000 7fe95d79d700 1 --2- 172.21.15.2:0/1321215985 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7fe940096ce0 0x7fe940099190 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:57.936 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:57.931+0000 7fe95d79d700 1 --2- 172.21.15.2:0/1321215985 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe9580fc780 0x7fe95806e830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:57.937 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:57.931+0000 7fe95d79d700 1 -- 172.21.15.2:0/1321215985 >> 172.21.15.2:0/1321215985 conn(0x7fe9580f6cb0 msgr2=0x7fe9580f9030 unknown :-1 s=STATE_NONE l=0).mark_down 2022-05-13T23:15:57.940 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:57.931+0000 7fe95d79d700 1 -- 172.21.15.2:0/1321215985 shutdown_connections 2022-05-13T23:15:57.940 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:57.931+0000 7fe95d79d700 1 -- 172.21.15.2:0/1321215985 wait complete. 2022-05-13T23:15:57.941 INFO:teuthology.orchestra.run.smithi002.stderr:dumped fsmap epoch 26 2022-05-13T23:15:57.959 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-05-13T23:15:58.179 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.176+0000 7f9d2ffff700 1 -- 172.21.15.2:0/17254020 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9d300655d0 msgr2=0x7f9d300fe0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:15:58.179 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.176+0000 7f9d2ffff700 1 --2- 172.21.15.2:0/17254020 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9d300655d0 0x7f9d300fe0d0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f9d20004870 tx=0x7f9d20009cb0 comp rx=0 tx=0).stop 2022-05-13T23:15:58.180 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.177+0000 7f9d2ffff700 1 -- 172.21.15.2:0/17254020 shutdown_connections 2022-05-13T23:15:58.180 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.177+0000 7f9d2ffff700 1 --2- 172.21.15.2:0/17254020 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f9d30101110 0x7f9d30103590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:58.180 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.177+0000 7f9d2ffff700 1 --2- 172.21.15.2:0/17254020 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9d300655d0 0x7f9d300fe0d0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:58.180 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.177+0000 7f9d2ffff700 1 --2- 172.21.15.2:0/17254020 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f9d300fe6a0 0x7f9d30100a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:58.181 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.177+0000 7f9d2ffff700 1 -- 172.21.15.2:0/17254020 >> 172.21.15.2:0/17254020 conn(0x7f9d300f5b80 msgr2=0x7f9d300f7fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-05-13T23:15:58.181 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.177+0000 7f9d2ffff700 1 -- 172.21.15.2:0/17254020 shutdown_connections 2022-05-13T23:15:58.181 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.177+0000 7f9d2ffff700 1 -- 172.21.15.2:0/17254020 wait complete. 2022-05-13T23:15:58.182 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.178+0000 7f9d2ffff700 1 Processor -- start 2022-05-13T23:15:58.185 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.178+0000 7f9d2ffff700 1 -- start start 2022-05-13T23:15:58.186 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.178+0000 7f9d2ffff700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f9d300655d0 0x7f9d3010e4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:15:58.186 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.178+0000 7f9d2ffff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9d300fe6a0 0x7f9d3010ea10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:15:58.187 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.178+0000 7f9d2ffff700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f9d30101110 0x7f9d30115aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:15:58.187 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.178+0000 7f9d2ffff700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d30115fe0 con 0x7f9d300fe6a0 2022-05-13T23:15:58.187 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.178+0000 7f9d2ffff700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d30116150 con 0x7f9d30101110 2022-05-13T23:15:58.188 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.178+0000 7f9d2ffff700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d30116310 con 0x7f9d300655d0 2022-05-13T23:15:58.188 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.178+0000 7f9d2e7fc700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9d300fe6a0 0x7f9d3010ea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:15:58.188 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.178+0000 7f9d2effd700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f9d300655d0 0x7f9d3010e4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:15:58.188 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.178+0000 7f9d2f7fe700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f9d30101110 0x7f9d30115aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:15:58.189 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.179+0000 7f9d2effd700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f9d300655d0 0x7f9d3010e4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.2:45386/0 (socket says 172.21.15.2:45386) 2022-05-13T23:15:58.189 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.179+0000 7f9d2effd700 1 -- 172.21.15.2:0/1257682873 learned_addr learned my addr 172.21.15.2:0/1257682873 (peer_addr_for_me v2:172.21.15.2:0/0) 2022-05-13T23:15:58.189 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.179+0000 7f9d2f7fe700 1 -- 172.21.15.2:0/1257682873 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f9d300655d0 msgr2=0x7f9d3010e4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:15:58.189 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.179+0000 7f9d2f7fe700 1 --2- 172.21.15.2:0/1257682873 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f9d300655d0 0x7f9d3010e4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:58.190 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.179+0000 7f9d2f7fe700 1 -- 172.21.15.2:0/1257682873 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9d300fe6a0 msgr2=0x7f9d3010ea10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:15:58.190 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.179+0000 7f9d2f7fe700 1 --2- 172.21.15.2:0/1257682873 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9d300fe6a0 0x7f9d3010ea10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:58.190 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.179+0000 7f9d2f7fe700 1 -- 172.21.15.2:0/1257682873 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d20005040 con 0x7f9d30101110 2022-05-13T23:15:58.190 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.179+0000 7f9d2f7fe700 1 --2- 172.21.15.2:0/1257682873 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f9d30101110 0x7f9d30115aa0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f9d2800eba0 tx=0x7f9d2800b040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-05-13T23:15:58.191 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.179+0000 7f9d177fe700 1 -- 172.21.15.2:0/1257682873 <== mon.1 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d280092e0 con 0x7f9d30101110 2022-05-13T23:15:58.191 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.180+0000 7f9d2ffff700 1 -- 172.21.15.2:0/1257682873 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9d30116590 con 0x7f9d30101110 2022-05-13T23:15:58.191 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.180+0000 7f9d177fe700 1 -- 172.21.15.2:0/1257682873 <== mon.1 v2:172.21.15.47:3300/0 2 ==== config(26 keys) v1 ==== 1087+0+0 (secure 0 0 0) 0x7f9d2800f040 con 0x7f9d30101110 2022-05-13T23:15:58.192 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.180+0000 7f9d177fe700 1 -- 172.21.15.2:0/1257682873 <== mon.1 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d2800cac0 con 0x7f9d30101110 2022-05-13T23:15:58.192 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.180+0000 7f9d2ffff700 1 -- 172.21.15.2:0/1257682873 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9d30116b00 con 0x7f9d30101110 2022-05-13T23:15:58.192 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.181+0000 7f9d2ffff700 1 -- 172.21.15.2:0/1257682873 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9d30106cc0 con 0x7f9d30101110 2022-05-13T23:15:58.193 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.183+0000 7f9d177fe700 1 -- 172.21.15.2:0/1257682873 <== mon.1 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 16) v1 ==== 125136+0+0 (secure 0 0 0) 0x7f9d280077a0 con 0x7f9d30101110 2022-05-13T23:15:58.193 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.184+0000 7f9d177fe700 1 --2- 172.21.15.2:0/1257682873 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f9d18096ce0 0x7f9d18099190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:15:58.193 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.184+0000 7f9d177fe700 1 -- 172.21.15.2:0/1257682873 <== mon.1 v2:172.21.15.47:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8120+0+0 (secure 0 0 0) 0x7f9d280bfe20 con 0x7f9d30101110 2022-05-13T23:15:58.193 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.184+0000 7f9d2effd700 1 --2- 172.21.15.2:0/1257682873 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f9d18096ce0 0x7f9d18099190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:15:58.194 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.185+0000 7f9d2effd700 1 --2- 172.21.15.2:0/1257682873 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f9d18096ce0 0x7f9d18099190 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f9d20004560 tx=0x7f9d20009cb0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-05-13T23:15:58.195 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.186+0000 7f9d177fe700 1 -- 172.21.15.2:0/1257682873 <== mon.1 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+165215 (secure 0 0 0) 0x7f9d2808d110 con 0x7f9d30101110 2022-05-13T23:15:58.382 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.379+0000 7f9d2ffff700 1 -- 172.21.15.2:0/1257682873 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9d300fb490 con 0x7f9d30101110 2022-05-13T23:15:58.383 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.381+0000 7f9d177fe700 1 -- 172.21.15.2:0/1257682873 <== mon.1 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v85) v1 ==== 74+0+21505 (secure 0 0 0) 0x7f9d28090730 con 0x7f9d30101110 2022-05-13T23:15:58.385 INFO:teuthology.orchestra.run.smithi002.stdout: 2022-05-13T23:15:58.386 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":85,"fsid":"16cf5f88-d311-11ec-be40-001a4aab830c","created":"2022-05-13T23:06:00.765453+0000","modified":"2022-05-13T23:15:06.402793+0000","last_up_change":"2022-05-13T23:13:20.935982+0000","last_in_change":"2022-05-13T23:13:02.796000+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2022-05-13T23:09:32.000644+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2022-05-13T23:14:14.195678+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":3,"pool_name":"cephfs_data","create_time":"2022-05-13T23:14:15.659212+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"83","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"11e700e8-f3e8-4903-b52c-db354b4395bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6802","nonce":1174843704},{"type":"v1","addr":"172.21.15.2:6803","nonce":1174843704}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6804","nonce":1174843704},{"type":"v1","addr":"172.21.15.2:6805","nonce":1174843704}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6808","nonce":1174843704},{"type":"v1","addr":"172.21.15.2:6809","nonce":1174843704}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6806","nonce":1174843704},{"type":"v1","addr":"172.21.15.2:6807","nonce":1174843704}]},"public_addr":"172.21.15.2:6803/1174843704","cluster_addr":"172.21.15.2:6805/1174843704","heartbeat_back_addr":"172.21.15.2:6809/1174843704","heartbeat_front_addr":"172.21.15.2:6807/1174843704","state":["exists","up"]},{"osd":1,"uuid":"eabfa3cf-f2a3-4995-a7de-f6d8ba02ee6f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6800","nonce":3853826948},{"type":"v1","addr":"172.21.15.47:6801","nonce":3853826948}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6802","nonce":3853826948},{"type":"v1","addr":"172.21.15.47:6803","nonce":3853826948}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6806","nonce":3853826948},{"type":"v1","addr":"172.21.15.47:6807","nonce":3853826948}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6804","nonce":3853826948},{"type":"v1","addr":"172.21.15.47:6805","nonce":3853826948}]},"public_addr":"172.21.15.47:6801/3853826948","cluster_addr":"172.21.15.47:6803/3853826948","heartbeat_back_addr":"172.21.15.47:6807/3853826948","heartbeat_front_addr":"172.21.15.47:6805/3853826948","state":["exists","up"]},{"osd":2,"uuid":"153c5bd5-8200-47a7-a135-8a95b6752cd2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6800","nonce":2562256956},{"type":"v1","addr":"172.21.15.115:6801","nonce":2562256956}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6802","nonce":2562256956},{"type":"v1","addr":"172.21.15.115:6803","nonce":2562256956}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6806","nonce":2562256956},{"type":"v1","addr":"172.21.15.115:6807","nonce":2562256956}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6804","nonce":2562256956},{"type":"v1","addr":"172.21.15.115:6805","nonce":2562256956}]},"public_addr":"172.21.15.115:6801/2562256956","cluster_addr":"172.21.15.115:6803/2562256956","heartbeat_back_addr":"172.21.15.115:6807/2562256956","heartbeat_front_addr":"172.21.15.115:6805/2562256956","state":["exists","up"]},{"osd":3,"uuid":"6ca9c602-5936-48da-a861-c437e6a33765","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6810","nonce":1099909411},{"type":"v1","addr":"172.21.15.2:6811","nonce":1099909411}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6812","nonce":1099909411},{"type":"v1","addr":"172.21.15.2:6813","nonce":1099909411}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6816","nonce":1099909411},{"type":"v1","addr":"172.21.15.2:6817","nonce":1099909411}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6814","nonce":1099909411},{"type":"v1","addr":"172.21.15.2:6815","nonce":1099909411}]},"public_addr":"172.21.15.2:6811/1099909411","cluster_addr":"172.21.15.2:6813/1099909411","heartbeat_back_addr":"172.21.15.2:6817/1099909411","heartbeat_front_addr":"172.21.15.2:6815/1099909411","state":["exists","up"]},{"osd":4,"uuid":"54d943d9-2610-422a-9e59-4b8212aced31","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6808","nonce":882753861},{"type":"v1","addr":"172.21.15.47:6809","nonce":882753861}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6810","nonce":882753861},{"type":"v1","addr":"172.21.15.47:6811","nonce":882753861}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6814","nonce":882753861},{"type":"v1","addr":"172.21.15.47:6815","nonce":882753861}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6812","nonce":882753861},{"type":"v1","addr":"172.21.15.47:6813","nonce":882753861}]},"public_addr":"172.21.15.47:6809/882753861","cluster_addr":"172.21.15.47:6811/882753861","heartbeat_back_addr":"172.21.15.47:6815/882753861","heartbeat_front_addr":"172.21.15.47:6813/882753861","state":["exists","up"]},{"osd":5,"uuid":"c18566cf-5403-4057-b2ee-aa298e87e8e9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6808","nonce":2122391993},{"type":"v1","addr":"172.21.15.115:6809","nonce":2122391993}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6810","nonce":2122391993},{"type":"v1","addr":"172.21.15.115:6811","nonce":2122391993}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6814","nonce":2122391993},{"type":"v1","addr":"172.21.15.115:6815","nonce":2122391993}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6812","nonce":2122391993},{"type":"v1","addr":"172.21.15.115:6813","nonce":2122391993}]},"public_addr":"172.21.15.115:6809/2122391993","cluster_addr":"172.21.15.115:6811/2122391993","heartbeat_back_addr":"172.21.15.115:6815/2122391993","heartbeat_front_addr":"172.21.15.115:6813/2122391993","state":["exists","up"]},{"osd":6,"uuid":"38ec7b6b-30f4-4d97-ad7e-e160ae59c843","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6818","nonce":1892198790},{"type":"v1","addr":"172.21.15.2:6819","nonce":1892198790}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6820","nonce":1892198790},{"type":"v1","addr":"172.21.15.2:6821","nonce":1892198790}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6824","nonce":1892198790},{"type":"v1","addr":"172.21.15.2:6825","nonce":1892198790}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6822","nonce":1892198790},{"type":"v1","addr":"172.21.15.2:6823","nonce":1892198790}]},"public_addr":"172.21.15.2:6819/1892198790","cluster_addr":"172.21.15.2:6821/1892198790","heartbeat_back_addr":"172.21.15.2:6825/1892198790","heartbeat_front_addr":"172.21.15.2:6823/1892198790","state":["exists","up"]},{"osd":7,"uuid":"e799b560-c61e-4369-b08b-45e5a6af37c0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6816","nonce":2812542825},{"type":"v1","addr":"172.21.15.47:6817","nonce":2812542825}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6818","nonce":2812542825},{"type":"v1","addr":"172.21.15.47:6819","nonce":2812542825}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6822","nonce":2812542825},{"type":"v1","addr":"172.21.15.47:6823","nonce":2812542825}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6820","nonce":2812542825},{"type":"v1","addr":"172.21.15.47:6821","nonce":2812542825}]},"public_addr":"172.21.15.47:6817/2812542825","cluster_addr":"172.21.15.47:6819/2812542825","heartbeat_back_addr":"172.21.15.47:6823/2812542825","heartbeat_front_addr":"172.21.15.47:6821/2812542825","state":["exists","up"]},{"osd":8,"uuid":"a58f2c0e-e487-41cf-8d2e-8c937cac9ebc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":58,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6816","nonce":3959632033},{"type":"v1","addr":"172.21.15.115:6817","nonce":3959632033}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6818","nonce":3959632033},{"type":"v1","addr":"172.21.15.115:6819","nonce":3959632033}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6822","nonce":3959632033},{"type":"v1","addr":"172.21.15.115:6823","nonce":3959632033}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6820","nonce":3959632033},{"type":"v1","addr":"172.21.15.115:6821","nonce":3959632033}]},"public_addr":"172.21.15.115:6817/3959632033","cluster_addr":"172.21.15.115:6819/3959632033","heartbeat_back_addr":"172.21.15.115:6823/3959632033","heartbeat_front_addr":"172.21.15.115:6821/3959632033","state":["exists","up"]},{"osd":9,"uuid":"12cead96-f726-4f7a-ab84-eb191b778a5c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":66,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6826","nonce":3004162255},{"type":"v1","addr":"172.21.15.2:6827","nonce":3004162255}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6828","nonce":3004162255},{"type":"v1","addr":"172.21.15.2:6829","nonce":3004162255}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6832","nonce":3004162255},{"type":"v1","addr":"172.21.15.2:6833","nonce":3004162255}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6830","nonce":3004162255},{"type":"v1","addr":"172.21.15.2:6831","nonce":3004162255}]},"public_addr":"172.21.15.2:6827/3004162255","cluster_addr":"172.21.15.2:6829/3004162255","heartbeat_back_addr":"172.21.15.2:6833/3004162255","heartbeat_front_addr":"172.21.15.2:6831/3004162255","state":["exists","up"]},{"osd":10,"uuid":"4f648118-7855-48d6-86d5-44eed5295f0c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":71,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6824","nonce":3443413998},{"type":"v1","addr":"172.21.15.47:6825","nonce":3443413998}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6826","nonce":3443413998},{"type":"v1","addr":"172.21.15.47:6827","nonce":3443413998}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6830","nonce":3443413998},{"type":"v1","addr":"172.21.15.47:6831","nonce":3443413998}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6828","nonce":3443413998},{"type":"v1","addr":"172.21.15.47:6829","nonce":3443413998}]},"public_addr":"172.21.15.47:6825/3443413998","cluster_addr":"172.21.15.47:6827/3443413998","heartbeat_back_addr":"172.21.15.47:6831/3443413998","heartbeat_front_addr":"172.21.15.47:6829/3443413998","state":["exists","up"]},{"osd":11,"uuid":"916541e0-fb28-46d5-9743-801c54c5a6d1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":78,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6824","nonce":1758718209},{"type":"v1","addr":"172.21.15.115:6825","nonce":1758718209}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6826","nonce":1758718209},{"type":"v1","addr":"172.21.15.115:6827","nonce":1758718209}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6830","nonce":1758718209},{"type":"v1","addr":"172.21.15.115:6831","nonce":1758718209}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6828","nonce":1758718209},{"type":"v1","addr":"172.21.15.115:6829","nonce":1758718209}]},"public_addr":"172.21.15.115:6825/1758718209","cluster_addr":"172.21.15.115:6827/1758718209","heartbeat_back_addr":"172.21.15.115:6831/1758718209","heartbeat_front_addr":"172.21.15.115:6829/1758718209","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-05-13T23:08:43.757857+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-05-13T23:09:06.537139+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-05-13T23:09:28.524837+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-05-13T23:10:02.057583+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-05-13T23:10:26.210648+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-05-13T23:10:47.885853+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-05-13T23:11:19.141865+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-05-13T23:11:42.376065+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-05-13T23:12:04.175870+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-05-13T23:12:34.695347+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-05-13T23:12:57.762613+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-05-13T23:13:19.114284+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.f","mappings":[{"from":8,"to":11}]},{"pgid":"2.13","mappings":[{"from":2,"to":11}]},{"pgid":"3.10","mappings":[{"from":5,"to":9}]},{"pgid":"3.2c","mappings":[{"from":4,"to":9}]},{"pgid":"3.36","mappings":[{"from":4,"to":9}]}],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.2:0/477249925":"2022-05-14T23:07:04.986087+0000","172.21.15.2:6800/3226099420":"2022-05-14T23:07:04.986087+0000","172.21.15.2:0/2214093915":"2022-05-14T23:07:04.986087+0000","172.21.15.2:0/1039403355":"2022-05-14T23:07:04.986087+0000","172.21.15.2:0/22091675":"2022-05-14T23:06:36.304108+0000","172.21.15.2:6800/1255134634":"2022-05-14T23:06:36.304108+0000","172.21.15.2:6801/3226099420":"2022-05-14T23:07:04.986087+0000","172.21.15.2:0/3768677276":"2022-05-14T23:06:36.304108+0000","172.21.15.2:6801/1255134634":"2022-05-14T23:06:36.304108+0000","172.21.15.2:0/407841268":"2022-05-14T23:06:36.304108+0000"},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2022-05-13T23:15:58.388 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.384+0000 7f9d2ffff700 1 -- 172.21.15.2:0/1257682873 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f9d18096ce0 msgr2=0x7f9d18099190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:15:58.388 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.384+0000 7f9d2ffff700 1 --2- 172.21.15.2:0/1257682873 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f9d18096ce0 0x7f9d18099190 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f9d20004560 tx=0x7f9d20009cb0 comp rx=0 tx=0).stop 2022-05-13T23:15:58.388 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.384+0000 7f9d2ffff700 1 -- 172.21.15.2:0/1257682873 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f9d30101110 msgr2=0x7f9d30115aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:15:58.389 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.384+0000 7f9d2ffff700 1 --2- 172.21.15.2:0/1257682873 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f9d30101110 0x7f9d30115aa0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f9d2800eba0 tx=0x7f9d2800b040 comp rx=0 tx=0).stop 2022-05-13T23:15:58.389 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.384+0000 7f9d2ffff700 1 -- 172.21.15.2:0/1257682873 shutdown_connections 2022-05-13T23:15:58.389 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.384+0000 7f9d2ffff700 1 --2- 172.21.15.2:0/1257682873 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f9d30101110 0x7f9d30115aa0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:58.390 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.384+0000 7f9d2ffff700 1 --2- 172.21.15.2:0/1257682873 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f9d300655d0 0x7f9d3010e4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:58.390 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.384+0000 7f9d2ffff700 1 --2- 172.21.15.2:0/1257682873 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f9d18096ce0 0x7f9d18099190 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:58.390 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.384+0000 7f9d2ffff700 1 --2- 172.21.15.2:0/1257682873 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9d300fe6a0 0x7f9d3010ea10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:58.391 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.384+0000 7f9d2ffff700 1 -- 172.21.15.2:0/1257682873 >> 172.21.15.2:0/1257682873 conn(0x7f9d300f5b80 msgr2=0x7f9d300f7f60 unknown :-1 s=STATE_NONE l=0).mark_down 2022-05-13T23:15:58.391 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.384+0000 7f9d2ffff700 1 -- 172.21.15.2:0/1257682873 shutdown_connections 2022-05-13T23:15:58.391 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.384+0000 7f9d2ffff700 1 -- 172.21.15.2:0/1257682873 wait complete. 2022-05-13T23:15:58.412 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 26, 'flags': 50, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2022-05-13T23:14:16.236645+0000', 'modified': '2022-05-13T23:14:23.309447+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 24611, 'mds_1': 24599, 'mds_2': 24587, 'mds_3': 24575, 'mds_4': 24547}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24494': {'gid': 24494, 'name': 'd', 'rank': 4, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.2:6837/3754333215', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.2:6836', 'nonce': 3754333215}, {'type': 'v1', 'addr': '172.21.15.2:6837', 'nonce': 3754333215}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138303579357183, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}, 'gid_24511': {'gid': 24511, 'name': 'b', 'rank': 3, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.47:6833/2783249369', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.47:6832', 'nonce': 2783249369}, {'type': 'v1', 'addr': '172.21.15.47:6833', 'nonce': 2783249369}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138303579357183, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}, 'gid_24523': {'gid': 24523, 'name': 'e', 'rank': 2, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.47:6835/4075838054', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.47:6834', 'nonce': 4075838054}, {'type': 'v1', 'addr': '172.21.15.47:6835', 'nonce': 4075838054}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138303579357183, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}, 'gid_24530': {'gid': 24530, 'name': 'j', 'rank': 1, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.2:6841/522559555', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.2:6840', 'nonce': 522559555}, {'type': 'v1', 'addr': '172.21.15.2:6841', 'nonce': 522559555}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138303579357183, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}, 'gid_24535': {'gid': 24535, 'name': 'h', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.47:6837/4072863000', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.47:6836', 'nonce': 4072863000}, {'type': 'v1', 'addr': '172.21.15.47:6837', 'nonce': 4072863000}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138303579357183, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}, 'gid_24547': {'gid': 24547, 'name': 'k', 'rank': 4, 'incarnation': 24, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.47:6839/761355251', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.47:6838', 'nonce': 761355251}, {'type': 'v1', 'addr': '172.21.15.47:6839', 'nonce': 761355251}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138303579357183, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}, 'gid_24575': {'gid': 24575, 'name': 'c', 'rank': 3, 'incarnation': 22, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.115:6833/381933929', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.115:6832', 'nonce': 381933929}, {'type': 'v1', 'addr': '172.21.15.115:6833', 'nonce': 381933929}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138303579357183, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}, 'gid_24587': {'gid': 24587, 'name': 'f', 'rank': 2, 'incarnation': 20, 'state': 'up:active', 'state_seq': 5, 'addr': '172.21.15.115:6835/71062276', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.115:6834', 'nonce': 71062276}, {'type': 'v1', 'addr': '172.21.15.115:6835', 'nonce': 71062276}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138303579357183, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}, 'gid_24599': {'gid': 24599, 'name': 'i', 'rank': 1, 'incarnation': 18, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.115:6837/2940957881', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.115:6836', 'nonce': 2940957881}, {'type': 'v1', 'addr': '172.21.15.115:6837', 'nonce': 2940957881}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138303579357183, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}, 'gid_24611': {'gid': 24611, 'name': 'l', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.115:6839/4149542923', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.115:6838', 'nonce': 4149542923}, {'type': 'v1', 'addr': '172.21.15.115:6839', 'nonce': 4149542923}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138303579357183, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2022-05-13T23:15:58.413 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 5/5 2022-05-13T23:15:58.414 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.k status 2022-05-13T23:15:58.637 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.631+0000 7fb47939d700 1 -- 172.21.15.2:0/3356314436 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb4740fb470 msgr2=0x7fb4740fb880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:15:58.637 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.631+0000 7fb47939d700 1 --2- 172.21.15.2:0/3356314436 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb4740fb470 0x7fb4740fb880 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fb464004870 tx=0x7fb464009cb0 comp rx=0 tx=0).stop 2022-05-13T23:15:58.638 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.631+0000 7fb47939d700 1 -- 172.21.15.2:0/3356314436 shutdown_connections 2022-05-13T23:15:58.638 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.631+0000 7fb47939d700 1 --2- 172.21.15.2:0/3356314436 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fb4740fd0a0 0x7fb474103960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:58.638 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.631+0000 7fb47939d700 1 --2- 172.21.15.2:0/3356314436 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb4740fb470 0x7fb4740fb880 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:58.638 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.631+0000 7fb47939d700 1 --2- 172.21.15.2:0/3356314436 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb4740fc710 0x7fb4740fcb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:58.639 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.631+0000 7fb47939d700 1 -- 172.21.15.2:0/3356314436 >> 172.21.15.2:0/3356314436 conn(0x7fb4740f6c60 msgr2=0x7fb4740f9090 unknown :-1 s=STATE_NONE l=0).mark_down 2022-05-13T23:15:58.639 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.631+0000 7fb47939d700 1 -- 172.21.15.2:0/3356314436 shutdown_connections 2022-05-13T23:15:58.639 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.631+0000 7fb47939d700 1 -- 172.21.15.2:0/3356314436 wait complete. 2022-05-13T23:15:58.639 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.632+0000 7fb47939d700 1 Processor -- start 2022-05-13T23:15:58.640 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.632+0000 7fb47939d700 1 -- start start 2022-05-13T23:15:58.640 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.633+0000 7fb47939d700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb4740fb470 0x7fb47410e4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:15:58.640 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.633+0000 7fb47939d700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb4740fc710 0x7fb47410ea20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:15:58.640 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.633+0000 7fb47939d700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fb4740fd0a0 0x7fb474115ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:15:58.641 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.633+0000 7fb47939d700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb474115ff0 con 0x7fb4740fb470 2022-05-13T23:15:58.641 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.633+0000 7fb47939d700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb474116160 con 0x7fb4740fc710 2022-05-13T23:15:58.641 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.633+0000 7fb47939d700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb474116320 con 0x7fb4740fd0a0 2022-05-13T23:15:58.641 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.633+0000 7fb473fff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb4740fb470 0x7fb47410e4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:15:58.641 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.633+0000 7fb473fff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb4740fb470 0x7fb47410e4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:54046/0 (socket says 172.21.15.2:54046) 2022-05-13T23:15:58.642 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.633+0000 7fb473fff700 1 -- 172.21.15.2:0/1100336209 learned_addr learned my addr 172.21.15.2:0/1100336209 (peer_addr_for_me v2:172.21.15.2:0/0) 2022-05-13T23:15:58.642 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.633+0000 7fb478b9c700 1 --2- 172.21.15.2:0/1100336209 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fb4740fd0a0 0x7fb474115ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:15:58.642 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.633+0000 7fb4737fe700 1 --2- 172.21.15.2:0/1100336209 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb4740fc710 0x7fb47410ea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:15:58.645 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.633+0000 7fb473fff700 1 -- 172.21.15.2:0/1100336209 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fb4740fd0a0 msgr2=0x7fb474115ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:15:58.645 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.633+0000 7fb473fff700 1 --2- 172.21.15.2:0/1100336209 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fb4740fd0a0 0x7fb474115ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:58.645 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.633+0000 7fb473fff700 1 -- 172.21.15.2:0/1100336209 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb4740fc710 msgr2=0x7fb47410ea20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:15:58.646 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.633+0000 7fb473fff700 1 --2- 172.21.15.2:0/1100336209 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb4740fc710 0x7fb47410ea20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:58.646 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.633+0000 7fb473fff700 1 -- 172.21.15.2:0/1100336209 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb464005040 con 0x7fb4740fb470 2022-05-13T23:15:58.646 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.634+0000 7fb473fff700 1 --2- 172.21.15.2:0/1100336209 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb4740fb470 0x7fb47410e4e0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fb464004710 tx=0x7fb464009cb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-05-13T23:15:58.646 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.634+0000 7fb470ff9700 1 -- 172.21.15.2:0/1100336209 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb46400e040 con 0x7fb4740fb470 2022-05-13T23:15:58.646 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.634+0000 7fb470ff9700 1 -- 172.21.15.2:0/1100336209 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1087+0+0 (secure 0 0 0) 0x7fb464004200 con 0x7fb4740fb470 2022-05-13T23:15:58.647 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.634+0000 7fb47939d700 1 -- 172.21.15.2:0/1100336209 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb4741165a0 con 0x7fb4740fb470 2022-05-13T23:15:58.647 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.634+0000 7fb470ff9700 1 -- 172.21.15.2:0/1100336209 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb46401caf0 con 0x7fb4740fb470 2022-05-13T23:15:58.647 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.634+0000 7fb47939d700 1 -- 172.21.15.2:0/1100336209 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb474116a20 con 0x7fb4740fb470 2022-05-13T23:15:58.656 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.637+0000 7fb470ff9700 1 -- 172.21.15.2:0/1100336209 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 16) v1 ==== 125136+0+0 (secure 0 0 0) 0x7fb464009240 con 0x7fb4740fb470 2022-05-13T23:15:58.657 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.638+0000 7fb48985bb80 1 Processor -- start 2022-05-13T23:15:58.657 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.639+0000 7fb470ff9700 1 --2- 172.21.15.2:0/1100336209 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7fb45c096ce0 0x7fb45c099190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:15:58.657 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.639+0000 7fb48985bb80 1 -- start start 2022-05-13T23:15:58.657 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.639+0000 7fb470ff9700 1 -- 172.21.15.2:0/1100336209 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8120+0+0 (secure 0 0 0) 0x7fb4640c0440 con 0x7fb4740fb470 2022-05-13T23:15:58.658 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.639+0000 7fb478b9c700 1 --2- 172.21.15.2:0/1100336209 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7fb45c096ce0 0x7fb45c099190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:15:58.658 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.639+0000 7fb48985bb80 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5626fd149cf0 0x5626fd14a100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:15:58.658 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.639+0000 7fb48985bb80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5626fd14a640 0x5626fd14af80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:15:58.658 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.639+0000 7fb48985bb80 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x5626fd14b5a0 0x5626fd14b9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:15:58.658 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.639+0000 7fb48985bb80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x5626fd0443a0 con 0x5626fd14a640 2022-05-13T23:15:58.659 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.639+0000 7fb48985bb80 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x5626fd019400 con 0x5626fd14b5a0 2022-05-13T23:15:58.659 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.639+0000 7fb48985bb80 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x5626fd09d6c0 con 0x5626fd149cf0 2022-05-13T23:15:58.659 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.639+0000 7fb473fff700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5626fd149cf0 0x5626fd14a100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:15:58.659 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.640+0000 7fb473fff700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5626fd149cf0 0x5626fd14a100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.2:45408/0 (socket says 172.21.15.2:45408) 2022-05-13T23:15:58.660 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.640+0000 7fb473fff700 1 -- 172.21.15.2:0/2947789262 learned_addr learned my addr 172.21.15.2:0/2947789262 (peer_addr_for_me v2:172.21.15.2:0/0) 2022-05-13T23:15:58.660 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.640+0000 7fb4737fe700 1 --2- 172.21.15.2:0/2947789262 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5626fd14a640 0x5626fd14af80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:15:58.660 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.640+0000 7fb478b9c700 1 --2- 172.21.15.2:0/2947789262 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x5626fd14b5a0 0x5626fd14b9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:15:58.660 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.640+0000 7fb473fff700 1 -- 172.21.15.2:0/2947789262 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x5626fd14b5a0 msgr2=0x5626fd14b9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-05-13T23:15:58.660 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.640+0000 7fb473fff700 1 --2- 172.21.15.2:0/2947789262 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x5626fd14b5a0 0x5626fd14b9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:58.661 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.640+0000 7fb473fff700 1 -- 172.21.15.2:0/2947789262 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5626fd14a640 msgr2=0x5626fd14af80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-05-13T23:15:58.661 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.640+0000 7fb473fff700 1 --2- 172.21.15.2:0/2947789262 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5626fd14a640 0x5626fd14af80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:58.661 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.640+0000 7fb473fff700 1 -- 172.21.15.2:0/2947789262 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5626fcf2f360 con 0x5626fd149cf0 2022-05-13T23:15:58.661 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.640+0000 7fb478b9c700 1 --2- 172.21.15.2:0/2947789262 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x5626fd14b5a0 0x5626fd14b9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2022-05-13T23:15:58.662 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.640+0000 7fb478b9c700 1 --2- 172.21.15.2:0/1100336209 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7fb45c096ce0 0x7fb45c099190 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fb46c004fd0 tx=0x7fb46c00b040 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-05-13T23:15:58.662 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.640+0000 7fb473fff700 1 --2- 172.21.15.2:0/2947789262 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5626fd149cf0 0x5626fd14a100 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fb4640043b0 tx=0x7fb46401e040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=83dbd32b9a9c9fdf server_cookie=0 in_seq=0 out_seq=0 2022-05-13T23:15:58.662 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.640+0000 7fb4599fe700 1 -- 172.21.15.2:0/2947789262 <== mon.2 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb464090ef0 con 0x5626fd149cf0 2022-05-13T23:15:58.662 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.640+0000 7fb4599fe700 1 -- 172.21.15.2:0/2947789262 <== mon.2 v2:172.21.15.115:3300/0 2 ==== config(26 keys) v1 ==== 1087+0+0 (secure 0 0 0) 0x7fb464090d10 con 0x5626fd149cf0 2022-05-13T23:15:58.662 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.640+0000 7fb4599fe700 1 -- 172.21.15.2:0/2947789262 <== mon.2 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb46401b390 con 0x5626fd149cf0 2022-05-13T23:15:58.663 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.641+0000 7fb48985bb80 1 -- 172.21.15.2:0/2947789262 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5626fd149cf0 msgr2=0x5626fd14a100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:15:58.663 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.641+0000 7fb48985bb80 1 --2- 172.21.15.2:0/2947789262 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5626fd149cf0 0x5626fd14a100 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fb4640043b0 tx=0x7fb46401e040 comp rx=0 tx=0).stop 2022-05-13T23:15:58.663 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.641+0000 7fb48985bb80 1 -- 172.21.15.2:0/2947789262 shutdown_connections 2022-05-13T23:15:58.663 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.641+0000 7fb48985bb80 1 --2- 172.21.15.2:0/2947789262 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x5626fd14b5a0 0x5626fd14b9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:58.664 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.641+0000 7fb48985bb80 1 --2- 172.21.15.2:0/2947789262 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5626fd149cf0 0x5626fd14a100 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:58.664 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.641+0000 7fb48985bb80 1 --2- 172.21.15.2:0/2947789262 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5626fd14a640 0x5626fd14af80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:58.664 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.641+0000 7fb48985bb80 1 -- 172.21.15.2:0/2947789262 >> 172.21.15.2:0/2947789262 conn(0x5626fd144b10 msgr2=0x5626fd144f00 unknown :-1 s=STATE_NONE l=0).mark_down 2022-05-13T23:15:58.664 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.641+0000 7fb48985bb80 1 -- 172.21.15.2:0/2947789262 shutdown_connections 2022-05-13T23:15:58.664 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.641+0000 7fb48985bb80 1 -- 172.21.15.2:0/2947789262 wait complete. 2022-05-13T23:15:58.665 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.642+0000 7fb48985bb80 1 -- start start 2022-05-13T23:15:58.665 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.642+0000 7fb48985bb80 1 Processor -- start 2022-05-13T23:15:58.665 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.643+0000 7fb48985bb80 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5626fd149cf0 0x5626fd153330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:15:58.665 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.643+0000 7fb48985bb80 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x5626fd14a640 0x5626fd153870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:15:58.666 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.643+0000 7fb48985bb80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5626fd14b5a0 0x5626fd15a8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:15:58.666 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.643+0000 7fb48985bb80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x5626fd0977a0 con 0x5626fd14b5a0 2022-05-13T23:15:58.666 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.643+0000 7fb48985bb80 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x5626fcf31650 con 0x5626fd14a640 2022-05-13T23:15:58.666 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.643+0000 7fb48985bb80 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x5626fd15af00 con 0x5626fd149cf0 2022-05-13T23:15:58.666 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.643+0000 7fb48985bb80 10 client.0 _ll_register_callbacks cb 0x5626fd10bde0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2022-05-13T23:15:58.667 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.643+0000 7fb473fff700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5626fd149cf0 0x5626fd153330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:15:58.668 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.643+0000 7fb473fff700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5626fd149cf0 0x5626fd153330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.2:45418/0 (socket says 172.21.15.2:45418) 2022-05-13T23:15:58.668 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.643+0000 7fb473fff700 1 -- 172.21.15.2:0/1731524764 learned_addr learned my addr 172.21.15.2:0/1731524764 (peer_addr_for_me v2:172.21.15.2:0/0) 2022-05-13T23:15:58.668 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.643+0000 7fb478b9c700 1 --2- 172.21.15.2:0/1731524764 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5626fd14b5a0 0x5626fd15a8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:15:58.668 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.643+0000 7fb4737fe700 1 --2- 172.21.15.2:0/1731524764 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x5626fd14a640 0x5626fd153870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:15:58.669 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.643+0000 7fb473fff700 1 -- 172.21.15.2:0/1731524764 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x5626fd14a640 msgr2=0x5626fd153870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-05-13T23:15:58.669 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.643+0000 7fb473fff700 1 --2- 172.21.15.2:0/1731524764 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x5626fd14a640 0x5626fd153870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:58.669 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.643+0000 7fb473fff700 1 -- 172.21.15.2:0/1731524764 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5626fd14b5a0 msgr2=0x5626fd15a8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-05-13T23:15:58.669 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.643+0000 7fb473fff700 1 --2- 172.21.15.2:0/1731524764 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5626fd14b5a0 0x5626fd15a8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:58.669 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.644+0000 7fb473fff700 1 -- 172.21.15.2:0/1731524764 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb474116a20 con 0x5626fd149cf0 2022-05-13T23:15:58.670 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.644+0000 7fb473fff700 1 --2- 172.21.15.2:0/1731524764 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5626fd149cf0 0x5626fd153330 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fb46401e010 tx=0x7fb46401e040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=db16d312829eec11 server_cookie=0 in_seq=0 out_seq=0 2022-05-13T23:15:58.670 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.644+0000 7fb44effd700 10 client.0 ms_handle_connect on v2:172.21.15.115:3300/0 2022-05-13T23:15:58.670 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.644+0000 7fb44effd700 1 -- 172.21.15.2:0/1731524764 <== mon.2 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4640ad2a0 con 0x5626fd149cf0 2022-05-13T23:15:58.670 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.644+0000 7fb44effd700 1 -- 172.21.15.2:0/1731524764 <== mon.2 v2:172.21.15.115:3300/0 2 ==== osd_map(85..85 src has 1..85) v4 ==== 8120+0+0 (secure 0 0 0) 0x7fb4640913b0 con 0x5626fd149cf0 2022-05-13T23:15:58.671 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.644+0000 7fb48985bb80 1 -- 172.21.15.2:0/1731524764 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5626fd04b9a0 con 0x5626fd149cf0 2022-05-13T23:15:58.671 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.644+0000 7fb44effd700 1 -- 172.21.15.2:0/1731524764 <== mon.2 v2:172.21.15.115:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb464090ef0 con 0x5626fd149cf0 2022-05-13T23:15:58.671 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.644+0000 7fb48985bb80 10 client.25157 fetch_fsmap learned FSMap version 26 2022-05-13T23:15:58.671 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.644+0000 7fb48985bb80 1 -- 172.21.15.2:0/1731524764 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) v3 -- 0x5626fced9fe0 con 0x5626fd149cf0 2022-05-13T23:15:58.671 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.645+0000 7fb44effd700 1 -- 172.21.15.2:0/1731524764 <== mon.2 v2:172.21.15.115:3300/0 4 ==== config(26 keys) v1 ==== 1087+0+0 (secure 0 0 0) 0x5626fd04b9a0 con 0x5626fd149cf0 2022-05-13T23:15:58.672 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.645+0000 7fb44effd700 1 -- 172.21.15.2:0/1731524764 <== mon.2 v2:172.21.15.115:3300/0 5 ==== fsmap(e 26) v1 ==== 7448+0+0 (secure 0 0 0) 0x7fb464097610 con 0x5626fd149cf0 2022-05-13T23:15:58.672 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.645+0000 7fb48985bb80 10 client.25157 fetch_fsmap finished waiting for FSMap version 26 2022-05-13T23:15:58.672 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.645+0000 7fb48985bb80 10 client.25157 resolve_mds: resolved name 'k' to daemon mds.k 2022-05-13T23:15:58.672 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.645+0000 7fb48985bb80 20 client.25157 populate_metadata read hostname 'smithi002' 2022-05-13T23:15:58.672 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.645+0000 7fb48985bb80 1 --2- 172.21.15.2:0/1731524764 >> [v2:172.21.15.47:6838/761355251,v1:172.21.15.47:6839/761355251] conn(0x5626fd15d970 0x5626fd15c380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:15:58.673 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.645+0000 7fb48985bb80 4 client.25157 mds_command: new command op to 24547 tid=0[{"prefix": "get_command_descriptions"}] 2022-05-13T23:15:58.673 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.645+0000 7fb48985bb80 1 -- 172.21.15.2:0/1731524764 --> [v2:172.21.15.47:6838/761355251,v1:172.21.15.47:6839/761355251] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5626fd06c330 con 0x5626fd15d970 2022-05-13T23:15:58.673 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.645+0000 7fb4737fe700 1 --2- 172.21.15.2:0/1731524764 >> [v2:172.21.15.47:6838/761355251,v1:172.21.15.47:6839/761355251] conn(0x5626fd15d970 0x5626fd15c380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:15:58.673 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.646+0000 7fb44effd700 10 client.25157 ms_handle_connect on v2:172.21.15.47:6838/761355251 2022-05-13T23:15:58.674 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.646+0000 7fb4737fe700 1 --2- 172.21.15.2:0/1731524764 >> [v2:172.21.15.47:6838/761355251,v1:172.21.15.47:6839/761355251] conn(0x5626fd15d970 0x5626fd15c380 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=3e1f91741c8f6c33 server_cookie=661ee46453cee1e6 in_seq=0 out_seq=0 2022-05-13T23:15:58.674 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.647+0000 7fb44effd700 1 -- 172.21.15.2:0/1731524764 <== mds.4 v2:172.21.15.47:6838/761355251 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17090 (crc 0 0 0) 0x5626fd06c330 con 0x5626fd15d970 2022-05-13T23:15:58.674 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.647+0000 7fb44effd700 10 client.25157 handle_command_reply: tid=0 2022-05-13T23:15:58.674 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.647+0000 7fb48985bb80 1 client.25157 shutdown 2022-05-13T23:15:58.674 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.647+0000 7fb48985bb80 1 -- 172.21.15.2:0/1731524764 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5626fd149cf0 msgr2=0x5626fd153330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:15:58.675 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.647+0000 7fb48985bb80 1 --2- 172.21.15.2:0/1731524764 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5626fd149cf0 0x5626fd153330 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fb46401e010 tx=0x7fb46401e040 comp rx=0 tx=0).stop 2022-05-13T23:15:58.675 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.647+0000 7fb48985bb80 1 -- 172.21.15.2:0/1731524764 shutdown_connections 2022-05-13T23:15:58.675 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.647+0000 7fb48985bb80 1 --2- 172.21.15.2:0/1731524764 >> [v2:172.21.15.47:6838/761355251,v1:172.21.15.47:6839/761355251] conn(0x5626fd15d970 0x5626fd15c380 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:58.675 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.647+0000 7fb48985bb80 1 --2- 172.21.15.2:0/1731524764 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5626fd14b5a0 0x5626fd15a8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:58.676 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.647+0000 7fb48985bb80 1 --2- 172.21.15.2:0/1731524764 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5626fd149cf0 0x5626fd153330 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:58.676 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.647+0000 7fb44effd700 0 client.25157 ms_handle_reset on v2:172.21.15.47:6838/761355251 2022-05-13T23:15:58.676 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.647+0000 7fb48985bb80 1 --2- 172.21.15.2:0/1731524764 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x5626fd14a640 0x5626fd153870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:58.676 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.647+0000 7fb48985bb80 1 -- 172.21.15.2:0/1731524764 >> 172.21.15.2:0/1731524764 conn(0x5626fd144b10 msgr2=0x5626fd143d80 unknown :-1 s=STATE_NONE l=0).mark_down 2022-05-13T23:15:58.676 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.647+0000 7fb48985bb80 1 -- 172.21.15.2:0/1731524764 shutdown_connections 2022-05-13T23:15:58.677 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.648+0000 7fb48985bb80 1 -- 172.21.15.2:0/1731524764 wait complete. 2022-05-13T23:15:58.677 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.648+0000 7fb48985bb80 20 client.25157 trim_cache size 0 max 16384 2022-05-13T23:15:58.684 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.665+0000 7fb48985bb80 1 Processor -- start 2022-05-13T23:15:58.685 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.665+0000 7fb48985bb80 1 -- start start 2022-05-13T23:15:58.685 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.665+0000 7fb48985bb80 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5626fd149cf0 0x5626fd15c2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:15:58.685 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.665+0000 7fb48985bb80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5626fd14a640 0x5626fd15c800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:15:58.686 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.665+0000 7fb48985bb80 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x5626fd14b5a0 0x5626fd145b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:15:58.686 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.666+0000 7fb48985bb80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x5626fd06c330 con 0x5626fd14a640 2022-05-13T23:15:58.686 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.666+0000 7fb48985bb80 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x5626fd04b9a0 con 0x5626fd14b5a0 2022-05-13T23:15:58.686 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.666+0000 7fb48985bb80 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x5626fcf337b0 con 0x5626fd149cf0 2022-05-13T23:15:58.686 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.666+0000 7fb4737fe700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5626fd14a640 0x5626fd15c800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:15:58.687 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.666+0000 7fb4737fe700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5626fd14a640 0x5626fd15c800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:54094/0 (socket says 172.21.15.2:54094) 2022-05-13T23:15:58.687 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.666+0000 7fb4737fe700 1 -- 172.21.15.2:0/1936303577 learned_addr learned my addr 172.21.15.2:0/1936303577 (peer_addr_for_me v2:172.21.15.2:0/0) 2022-05-13T23:15:58.687 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.666+0000 7fb473fff700 1 --2- 172.21.15.2:0/1936303577 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5626fd149cf0 0x5626fd15c2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:15:58.687 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.666+0000 7fb478b9c700 1 --2- 172.21.15.2:0/1936303577 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x5626fd14b5a0 0x5626fd145b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:15:58.688 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.666+0000 7fb4737fe700 1 -- 172.21.15.2:0/1936303577 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5626fd149cf0 msgr2=0x5626fd15c2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-05-13T23:15:58.688 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.666+0000 7fb4737fe700 1 --2- 172.21.15.2:0/1936303577 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5626fd149cf0 0x5626fd15c2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:58.688 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.666+0000 7fb4737fe700 1 -- 172.21.15.2:0/1936303577 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x5626fd14b5a0 msgr2=0x5626fd145b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-05-13T23:15:58.688 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.666+0000 7fb4737fe700 1 --2- 172.21.15.2:0/1936303577 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x5626fd14b5a0 0x5626fd145b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:58.688 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.666+0000 7fb4737fe700 1 -- 172.21.15.2:0/1936303577 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5626fcf01f80 con 0x5626fd14a640 2022-05-13T23:15:58.689 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.667+0000 7fb4737fe700 1 --2- 172.21.15.2:0/1936303577 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5626fd14a640 0x5626fd15c800 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x5626fd1543b0 tx=0x7fb46800b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3ae2ee45a58771f0 server_cookie=0 in_seq=0 out_seq=0 2022-05-13T23:15:58.689 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.667+0000 7fb45a1ff700 1 -- 172.21.15.2:0/1936303577 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb46800e040 con 0x5626fd14a640 2022-05-13T23:15:58.689 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.667+0000 7fb45a1ff700 1 -- 172.21.15.2:0/1936303577 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1087+0+0 (secure 0 0 0) 0x7fb46800f040 con 0x5626fd14a640 2022-05-13T23:15:58.689 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.667+0000 7fb45a1ff700 1 -- 172.21.15.2:0/1936303577 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb468008030 con 0x5626fd14a640 2022-05-13T23:15:58.690 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.667+0000 7fb48985bb80 1 -- 172.21.15.2:0/1936303577 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5626fd14a640 msgr2=0x5626fd15c800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:15:58.690 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.667+0000 7fb48985bb80 1 --2- 172.21.15.2:0/1936303577 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5626fd14a640 0x5626fd15c800 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x5626fd1543b0 tx=0x7fb46800b040 comp rx=0 tx=0).stop 2022-05-13T23:15:58.691 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.667+0000 7fb48985bb80 1 -- 172.21.15.2:0/1936303577 shutdown_connections 2022-05-13T23:15:58.691 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.667+0000 7fb48985bb80 1 --2- 172.21.15.2:0/1936303577 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x5626fd14b5a0 0x5626fd145b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:58.691 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.667+0000 7fb48985bb80 1 --2- 172.21.15.2:0/1936303577 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5626fd149cf0 0x5626fd15c2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:58.692 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.667+0000 7fb48985bb80 1 --2- 172.21.15.2:0/1936303577 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5626fd14a640 0x5626fd15c800 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:58.692 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.667+0000 7fb48985bb80 1 -- 172.21.15.2:0/1936303577 >> 172.21.15.2:0/1936303577 conn(0x5626fd144b10 msgr2=0x5626fd143300 unknown :-1 s=STATE_NONE l=0).mark_down 2022-05-13T23:15:58.692 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.668+0000 7fb48985bb80 1 -- 172.21.15.2:0/1936303577 shutdown_connections 2022-05-13T23:15:58.693 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.668+0000 7fb48985bb80 1 -- 172.21.15.2:0/1936303577 wait complete. 2022-05-13T23:15:58.693 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.668+0000 7fb48985bb80 1 -- start start 2022-05-13T23:15:58.693 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.668+0000 7fb48985bb80 1 Processor -- start 2022-05-13T23:15:58.694 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.669+0000 7fb48985bb80 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x5626fd149cf0 0x5626fd1646b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:15:58.694 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.669+0000 7fb48985bb80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5626fd14a640 0x5626fd164bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:15:58.694 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.669+0000 7fb48985bb80 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5626fd14b5a0 0x5626fd1b05b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:15:58.695 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.669+0000 7fb48985bb80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x5626fd167bd0 con 0x5626fd14a640 2022-05-13T23:15:58.695 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.669+0000 7fb48985bb80 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x5626fd167d40 con 0x5626fd149cf0 2022-05-13T23:15:58.696 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.669+0000 7fb48985bb80 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x5626fd167f00 con 0x5626fd14b5a0 2022-05-13T23:15:58.696 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.669+0000 7fb48985bb80 10 client.0 _ll_register_callbacks cb 0x5626fd10bde0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2022-05-13T23:15:58.696 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.669+0000 7fb4737fe700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5626fd14a640 0x5626fd164bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:15:58.697 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.669+0000 7fb478b9c700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5626fd14b5a0 0x5626fd1b05b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:15:58.697 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.669+0000 7fb473fff700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x5626fd149cf0 0x5626fd1646b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:15:58.697 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.669+0000 7fb4737fe700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5626fd14a640 0x5626fd164bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:54104/0 (socket says 172.21.15.2:54104) 2022-05-13T23:15:58.697 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.669+0000 7fb478b9c700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5626fd14b5a0 0x5626fd1b05b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.2:45444/0 (socket says 172.21.15.2:45444) 2022-05-13T23:15:58.698 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.669+0000 7fb473fff700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x5626fd149cf0 0x5626fd1646b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.2:37430/0 (socket says 172.21.15.2:37430) 2022-05-13T23:15:58.698 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.669+0000 7fb4737fe700 1 -- 172.21.15.2:0/1932014510 learned_addr learned my addr 172.21.15.2:0/1932014510 (peer_addr_for_me v2:172.21.15.2:0/0) 2022-05-13T23:15:58.698 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.669+0000 7fb4737fe700 1 -- 172.21.15.2:0/1932014510 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5626fd14b5a0 msgr2=0x5626fd1b05b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-05-13T23:15:58.699 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.669+0000 7fb4737fe700 1 --2- 172.21.15.2:0/1932014510 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5626fd14b5a0 0x5626fd1b05b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:58.699 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.670+0000 7fb4737fe700 1 -- 172.21.15.2:0/1932014510 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x5626fd149cf0 msgr2=0x5626fd1646b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-05-13T23:15:58.699 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.670+0000 7fb4737fe700 1 --2- 172.21.15.2:0/1932014510 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x5626fd149cf0 0x5626fd1646b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:58.699 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.670+0000 7fb4737fe700 1 -- 172.21.15.2:0/1932014510 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb468005040 con 0x5626fd14a640 2022-05-13T23:15:58.700 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.670+0000 7fb4737fe700 1 --2- 172.21.15.2:0/1932014510 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5626fd14a640 0x5626fd164bf0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fb468009cf0 tx=0x7fb468018040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8f05c01c48fa7bfd server_cookie=0 in_seq=0 out_seq=0 2022-05-13T23:15:58.700 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.670+0000 7fb44effd700 10 client.0 ms_handle_connect on v2:172.21.15.2:3300/0 2022-05-13T23:15:58.700 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.670+0000 7fb44effd700 1 -- 172.21.15.2:0/1932014510 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb46800e030 con 0x5626fd14a640 2022-05-13T23:15:58.700 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.670+0000 7fb48985bb80 1 -- 172.21.15.2:0/1932014510 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5626fd11a5f0 con 0x5626fd14a640 2022-05-13T23:15:58.701 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.670+0000 7fb44effd700 1 -- 172.21.15.2:0/1932014510 <== mon.0 v2:172.21.15.2:3300/0 2 ==== osd_map(85..85 src has 1..85) v4 ==== 8120+0+0 (secure 0 0 0) 0x7fb468008670 con 0x5626fd14a640 2022-05-13T23:15:58.701 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.670+0000 7fb44effd700 1 -- 172.21.15.2:0/1932014510 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb468037030 con 0x5626fd14a640 2022-05-13T23:15:58.701 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.670+0000 7fb48985bb80 10 client.15042 fetch_fsmap learned FSMap version 26 2022-05-13T23:15:58.702 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.670+0000 7fb48985bb80 1 -- 172.21.15.2:0/1932014510 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) v3 -- 0x5626fcedbb10 con 0x5626fd14a640 2022-05-13T23:15:58.702 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.671+0000 7fb44effd700 1 -- 172.21.15.2:0/1932014510 <== mon.0 v2:172.21.15.2:3300/0 4 ==== config(26 keys) v1 ==== 1087+0+0 (secure 0 0 0) 0x5626fd11a5f0 con 0x5626fd14a640 2022-05-13T23:15:58.702 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.671+0000 7fb44effd700 1 -- 172.21.15.2:0/1932014510 <== mon.0 v2:172.21.15.2:3300/0 5 ==== fsmap(e 26) v1 ==== 7448+0+0 (secure 0 0 0) 0x7fb46802f020 con 0x5626fd14a640 2022-05-13T23:15:58.703 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.671+0000 7fb48985bb80 10 client.15042 fetch_fsmap finished waiting for FSMap version 26 2022-05-13T23:15:58.703 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.671+0000 7fb48985bb80 10 client.15042 resolve_mds: resolved name 'k' to daemon mds.k 2022-05-13T23:15:58.704 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.671+0000 7fb48985bb80 20 client.15042 populate_metadata read hostname 'smithi002' 2022-05-13T23:15:58.704 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.671+0000 7fb48985bb80 1 --2- 172.21.15.2:0/1932014510 >> [v2:172.21.15.47:6838/761355251,v1:172.21.15.47:6839/761355251] conn(0x5626fd1b5140 0x5626fd1b1a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:15:58.704 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.671+0000 7fb48985bb80 4 client.15042 mds_command: new command op to 24547 tid=0[{"prefix": "status"}] 2022-05-13T23:15:58.705 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.671+0000 7fb48985bb80 1 -- 172.21.15.2:0/1932014510 --> [v2:172.21.15.47:6838/761355251,v1:172.21.15.47:6839/761355251] -- command(tid 0: {"prefix": "status"}) v1 -- 0x5626fd03d790 con 0x5626fd1b5140 2022-05-13T23:15:58.705 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.671+0000 7fb473fff700 1 --2- 172.21.15.2:0/1932014510 >> [v2:172.21.15.47:6838/761355251,v1:172.21.15.47:6839/761355251] conn(0x5626fd1b5140 0x5626fd1b1a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:15:58.706 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.672+0000 7fb44effd700 10 client.15042 ms_handle_connect on v2:172.21.15.47:6838/761355251 2022-05-13T23:15:58.706 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.672+0000 7fb473fff700 1 --2- 172.21.15.2:0/1932014510 >> [v2:172.21.15.47:6838/761355251,v1:172.21.15.47:6839/761355251] conn(0x5626fd1b5140 0x5626fd1b1a90 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=4beaccfc98c5caf5 server_cookie=1413ae405fdbb664 in_seq=0 out_seq=0 2022-05-13T23:15:58.706 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.672+0000 7fb44effd700 1 -- 172.21.15.2:0/1932014510 <== mds.4 v2:172.21.15.47:6838/761355251 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+332 (crc 0 0 0) 0x5626fd03d790 con 0x5626fd1b5140 2022-05-13T23:15:58.706 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.672+0000 7fb44effd700 10 client.15042 handle_command_reply: tid=0 2022-05-13T23:15:58.707 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.672+0000 7fb48985bb80 1 client.15042 shutdown 2022-05-13T23:15:58.707 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.673+0000 7fb48985bb80 1 -- 172.21.15.2:0/1932014510 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5626fd14a640 msgr2=0x5626fd164bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:15:58.707 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.673+0000 7fb48985bb80 1 --2- 172.21.15.2:0/1932014510 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5626fd14a640 0x5626fd164bf0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fb468009cf0 tx=0x7fb468018040 comp rx=0 tx=0).stop 2022-05-13T23:15:58.707 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.673+0000 7fb48985bb80 1 -- 172.21.15.2:0/1932014510 shutdown_connections 2022-05-13T23:15:58.708 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.673+0000 7fb48985bb80 1 --2- 172.21.15.2:0/1932014510 >> [v2:172.21.15.47:6838/761355251,v1:172.21.15.47:6839/761355251] conn(0x5626fd1b5140 0x5626fd1b1a90 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:58.708 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.673+0000 7fb48985bb80 1 --2- 172.21.15.2:0/1932014510 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5626fd14b5a0 0x5626fd1b05b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:58.708 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.673+0000 7fb44effd700 0 client.15042 ms_handle_reset on v2:172.21.15.47:6838/761355251 2022-05-13T23:15:58.708 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.673+0000 7fb48985bb80 1 --2- 172.21.15.2:0/1932014510 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x5626fd149cf0 0x5626fd1646b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:58.709 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.673+0000 7fb48985bb80 1 --2- 172.21.15.2:0/1932014510 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5626fd14a640 0x5626fd164bf0 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:58.709 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.673+0000 7fb48985bb80 1 -- 172.21.15.2:0/1932014510 >> 172.21.15.2:0/1932014510 conn(0x5626fd144b10 msgr2=0x5626fd15c240 unknown :-1 s=STATE_NONE l=0).mark_down 2022-05-13T23:15:58.709 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.673+0000 7fb48985bb80 1 -- 172.21.15.2:0/1932014510 shutdown_connections 2022-05-13T23:15:58.709 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.673+0000 7fb48985bb80 1 -- 172.21.15.2:0/1932014510 wait complete. 2022-05-13T23:15:58.709 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.673+0000 7fb48985bb80 20 client.15042 trim_cache size 0 max 16384 2022-05-13T23:15:58.710 INFO:teuthology.orchestra.run.smithi002.stdout:{ 2022-05-13T23:15:58.710 INFO:teuthology.orchestra.run.smithi002.stdout: "cluster_fsid": "16cf5f88-d311-11ec-be40-001a4aab830c", 2022-05-13T23:15:58.711 INFO:teuthology.orchestra.run.smithi002.stdout: "whoami": 4, 2022-05-13T23:15:58.711 INFO:teuthology.orchestra.run.smithi002.stdout: "id": 24547, 2022-05-13T23:15:58.711 INFO:teuthology.orchestra.run.smithi002.stdout: "want_state": "up:active", 2022-05-13T23:15:58.712 INFO:teuthology.orchestra.run.smithi002.stdout: "state": "up:active", 2022-05-13T23:15:58.712 INFO:teuthology.orchestra.run.smithi002.stdout: "fs_name": "cephfs", 2022-05-13T23:15:58.712 INFO:teuthology.orchestra.run.smithi002.stdout: "rank_uptime": 96.366718981999995, 2022-05-13T23:15:58.712 INFO:teuthology.orchestra.run.smithi002.stdout: "mdsmap_epoch": 26, 2022-05-13T23:15:58.713 INFO:teuthology.orchestra.run.smithi002.stdout: "osdmap_epoch": 85, 2022-05-13T23:15:58.713 INFO:teuthology.orchestra.run.smithi002.stdout: "osdmap_epoch_barrier": 83, 2022-05-13T23:15:58.713 INFO:teuthology.orchestra.run.smithi002.stdout: "uptime": 117.92317125700001 2022-05-13T23:15:58.713 INFO:teuthology.orchestra.run.smithi002.stdout:} 2022-05-13T23:15:58.715 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.674+0000 7fb47939d700 1 -- 172.21.15.2:0/1100336209 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7fb45c096ce0 msgr2=0x7fb45c099190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:15:58.715 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.674+0000 7fb47939d700 1 --2- 172.21.15.2:0/1100336209 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7fb45c096ce0 0x7fb45c099190 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fb46c004fd0 tx=0x7fb46c00b040 comp rx=0 tx=0).stop 2022-05-13T23:15:58.716 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.674+0000 7fb47939d700 1 -- 172.21.15.2:0/1100336209 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb4740fb470 msgr2=0x7fb47410e4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:15:58.716 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.674+0000 7fb47939d700 1 --2- 172.21.15.2:0/1100336209 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb4740fb470 0x7fb47410e4e0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fb464004710 tx=0x7fb464009cb0 comp rx=0 tx=0).stop 2022-05-13T23:15:58.716 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.674+0000 7fb47939d700 1 -- 172.21.15.2:0/1100336209 shutdown_connections 2022-05-13T23:15:58.717 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.674+0000 7fb47939d700 1 --2- 172.21.15.2:0/1100336209 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fb4740fd0a0 0x7fb474115ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:58.717 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.674+0000 7fb47939d700 1 --2- 172.21.15.2:0/1100336209 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7fb45c096ce0 0x7fb45c099190 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:58.717 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.674+0000 7fb47939d700 1 --2- 172.21.15.2:0/1100336209 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb4740fb470 0x7fb47410e4e0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:58.717 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.674+0000 7fb47939d700 1 --2- 172.21.15.2:0/1100336209 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb4740fc710 0x7fb47410ea20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:58.718 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.674+0000 7fb47939d700 1 -- 172.21.15.2:0/1100336209 >> 172.21.15.2:0/1100336209 conn(0x7fb4740f6c60 msgr2=0x7fb47406e710 unknown :-1 s=STATE_NONE l=0).mark_down 2022-05-13T23:15:58.718 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.674+0000 7fb47939d700 1 -- 172.21.15.2:0/1100336209 shutdown_connections 2022-05-13T23:15:58.718 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.674+0000 7fb47939d700 1 -- 172.21.15.2:0/1100336209 wait complete. 2022-05-13T23:15:58.719 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:15:58 smithi002 ceph-mon[71816]: pgmap v341: 97 pgs: 97 active+clean; 584 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2022-05-13T23:15:58.719 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:15:58 smithi002 ceph-mon[71816]: from='client.? 172.21.15.2:0/2968696232' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2022-05-13T23:15:58.719 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:15:58 smithi002 ceph-mon[71816]: from='client.? 172.21.15.2:0/1321215985' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2022-05-13T23:15:58.721 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2022-05-13T23:15:58.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:15:58 smithi047 ceph-mon[77147]: pgmap v341: 97 pgs: 97 active+clean; 584 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2022-05-13T23:15:58.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:15:58 smithi047 ceph-mon[77147]: from='client.? 172.21.15.2:0/2968696232' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2022-05-13T23:15:58.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:15:58 smithi047 ceph-mon[77147]: from='client.? 172.21.15.2:0/1321215985' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2022-05-13T23:15:58.864 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:15:58 smithi115 ceph-mon[78593]: pgmap v341: 97 pgs: 97 active+clean; 584 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2022-05-13T23:15:58.865 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:15:58 smithi115 ceph-mon[78593]: from='client.? 172.21.15.2:0/2968696232' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2022-05-13T23:15:58.865 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:15:58 smithi115 ceph-mon[78593]: from='client.? 172.21.15.2:0/1321215985' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2022-05-13T23:15:58.945 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.939+0000 7f2c5519d700 1 -- 172.21.15.2:0/2293946304 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f2c500fb4e0 msgr2=0x7f2c500fb8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:15:58.945 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.939+0000 7f2c5519d700 1 --2- 172.21.15.2:0/2293946304 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f2c500fb4e0 0x7f2c500fb8f0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f2c400047b0 tx=0x7f2c40009cb0 comp rx=0 tx=0).stop 2022-05-13T23:15:58.946 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.939+0000 7f2c5519d700 1 -- 172.21.15.2:0/2293946304 shutdown_connections 2022-05-13T23:15:58.946 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.939+0000 7f2c5519d700 1 --2- 172.21.15.2:0/2293946304 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f2c500fd110 0x7f2c501039d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:58.947 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.939+0000 7f2c5519d700 1 --2- 172.21.15.2:0/2293946304 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f2c500fb4e0 0x7f2c500fb8f0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:58.948 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.939+0000 7f2c5519d700 1 --2- 172.21.15.2:0/2293946304 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2c500fc780 0x7f2c500fcbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:58.948 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.939+0000 7f2c5519d700 1 -- 172.21.15.2:0/2293946304 >> 172.21.15.2:0/2293946304 conn(0x7f2c500f6cb0 msgr2=0x7f2c500f9100 unknown :-1 s=STATE_NONE l=0).mark_down 2022-05-13T23:15:58.949 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.939+0000 7f2c5519d700 1 -- 172.21.15.2:0/2293946304 shutdown_connections 2022-05-13T23:15:58.949 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.939+0000 7f2c5519d700 1 -- 172.21.15.2:0/2293946304 wait complete. 2022-05-13T23:15:58.950 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.940+0000 7f2c5519d700 1 Processor -- start 2022-05-13T23:15:58.950 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.940+0000 7f2c5519d700 1 -- start start 2022-05-13T23:15:58.950 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.940+0000 7f2c5519d700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f2c500fb4e0 0x7f2c500701d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:15:58.951 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.940+0000 7f2c5519d700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f2c500fc780 0x7f2c5006e830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:15:58.951 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.940+0000 7f2c5519d700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2c500fd110 0x7f2c5006ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:15:58.951 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.940+0000 7f2c5519d700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c5006f480 con 0x7f2c500fd110 2022-05-13T23:15:58.952 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.940+0000 7f2c5519d700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c5006f5f0 con 0x7f2c500fb4e0 2022-05-13T23:15:58.952 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.940+0000 7f2c5519d700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c50116430 con 0x7f2c500fc780 2022-05-13T23:15:58.953 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.941+0000 7f2c4ffff700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f2c500fb4e0 0x7f2c500701d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:15:58.954 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.941+0000 7f2c5499c700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2c500fd110 0x7f2c5006ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:15:58.954 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.941+0000 7f2c4ffff700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f2c500fb4e0 0x7f2c500701d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.2:37436/0 (socket says 172.21.15.2:37436) 2022-05-13T23:15:58.955 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.941+0000 7f2c4ffff700 1 -- 172.21.15.2:0/4203339945 learned_addr learned my addr 172.21.15.2:0/4203339945 (peer_addr_for_me v2:172.21.15.2:0/0) 2022-05-13T23:15:58.956 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.941+0000 7f2c4f7fe700 1 --2- 172.21.15.2:0/4203339945 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f2c500fc780 0x7f2c5006e830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:15:58.956 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.941+0000 7f2c4ffff700 1 -- 172.21.15.2:0/4203339945 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f2c500fc780 msgr2=0x7f2c5006e830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:15:58.957 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.941+0000 7f2c4ffff700 1 --2- 172.21.15.2:0/4203339945 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f2c500fc780 0x7f2c5006e830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:58.957 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.941+0000 7f2c4ffff700 1 -- 172.21.15.2:0/4203339945 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2c500fd110 msgr2=0x7f2c5006ed70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:15:58.957 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.941+0000 7f2c4ffff700 1 --2- 172.21.15.2:0/4203339945 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2c500fd110 0x7f2c5006ed70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:58.958 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.941+0000 7f2c4ffff700 1 -- 172.21.15.2:0/4203339945 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c480097c0 con 0x7f2c500fb4e0 2022-05-13T23:15:58.958 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.941+0000 7f2c4ffff700 1 --2- 172.21.15.2:0/4203339945 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f2c500fb4e0 0x7f2c500701d0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f2c40009e00 tx=0x7f2c40009cb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-05-13T23:15:58.958 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.942+0000 7f2c4cff9700 1 -- 172.21.15.2:0/4203339945 <== mon.1 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c40000ce0 con 0x7f2c500fb4e0 2022-05-13T23:15:58.959 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.942+0000 7f2c5519d700 1 -- 172.21.15.2:0/4203339945 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2c40005040 con 0x7f2c500fb4e0 2022-05-13T23:15:58.959 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.942+0000 7f2c4cff9700 1 -- 172.21.15.2:0/4203339945 <== mon.1 v2:172.21.15.47:3300/0 2 ==== config(26 keys) v1 ==== 1087+0+0 (secure 0 0 0) 0x7f2c400096f0 con 0x7f2c500fb4e0 2022-05-13T23:15:58.959 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.942+0000 7f2c4cff9700 1 -- 172.21.15.2:0/4203339945 <== mon.1 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c4001cd20 con 0x7f2c500fb4e0 2022-05-13T23:15:58.960 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.942+0000 7f2c5519d700 1 -- 172.21.15.2:0/4203339945 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2c501168d0 con 0x7f2c500fb4e0 2022-05-13T23:15:58.972 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.945+0000 7f2c4cff9700 1 -- 172.21.15.2:0/4203339945 <== mon.1 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 16) v1 ==== 125136+0+0 (secure 0 0 0) 0x7f2c40009240 con 0x7f2c500fb4e0 2022-05-13T23:15:58.972 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.946+0000 7f2c65764b80 1 Processor -- start 2022-05-13T23:15:58.973 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.946+0000 7f2c65764b80 1 -- start start 2022-05-13T23:15:58.973 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.946+0000 7f2c65764b80 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x561f70d0bcf0 0x561f70d0c100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:15:58.973 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.946+0000 7f2c65764b80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x561f70d0c640 0x561f70d0cf80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:15:58.974 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.946+0000 7f2c65764b80 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x561f70d0d5a0 0x561f70d0d9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:15:58.974 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.946+0000 7f2c65764b80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x561f70c063a0 con 0x561f70d0c640 2022-05-13T23:15:58.974 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.946+0000 7f2c65764b80 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x561f70bdb400 con 0x561f70d0d5a0 2022-05-13T23:15:58.974 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.946+0000 7f2c65764b80 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x561f70c5f6c0 con 0x561f70d0bcf0 2022-05-13T23:15:58.975 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.947+0000 7f2c4cff9700 1 --2- 172.21.15.2:0/4203339945 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f2c38096ce0 0x7f2c38099190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:15:58.975 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.947+0000 7f2c4ffff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x561f70d0c640 0x561f70d0cf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:15:58.975 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.947+0000 7f2c5499c700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x561f70d0bcf0 0x561f70d0c100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:15:58.976 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.947+0000 7f2c5499c700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x561f70d0bcf0 0x561f70d0c100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.2:45474/0 (socket says 172.21.15.2:45474) 2022-05-13T23:15:58.976 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.947+0000 7f2c5499c700 1 -- 172.21.15.2:0/2118907798 learned_addr learned my addr 172.21.15.2:0/2118907798 (peer_addr_for_me v2:172.21.15.2:0/0) 2022-05-13T23:15:58.977 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.947+0000 7f2c4f7fe700 1 --2- 172.21.15.2:0/2118907798 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x561f70d0d5a0 0x561f70d0d9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:15:58.977 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.947+0000 7f2c4cff9700 1 -- 172.21.15.2:0/4203339945 <== mon.1 v2:172.21.15.47:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8120+0+0 (secure 0 0 0) 0x7f2c400c04c0 con 0x7f2c500fb4e0 2022-05-13T23:15:58.978 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.947+0000 7f2c5499c700 1 --2- 172.21.15.2:0/4203339945 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f2c38096ce0 0x7f2c38099190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:15:58.978 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.947+0000 7f2c4ffff700 1 -- 172.21.15.2:0/2118907798 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x561f70d0bcf0 msgr2=0x561f70d0c100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-05-13T23:15:58.979 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.947+0000 7f2c4ffff700 1 --2- 172.21.15.2:0/2118907798 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x561f70d0bcf0 0x561f70d0c100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:58.979 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.947+0000 7f2c4ffff700 1 -- 172.21.15.2:0/2118907798 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x561f70d0d5a0 msgr2=0x561f70d0d9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-05-13T23:15:58.980 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.947+0000 7f2c4ffff700 1 --2- 172.21.15.2:0/2118907798 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x561f70d0d5a0 0x561f70d0d9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:58.980 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.947+0000 7f2c4ffff700 1 -- 172.21.15.2:0/2118907798 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x561f70af1360 con 0x561f70d0c640 2022-05-13T23:15:58.980 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.947+0000 7f2c4f7fe700 1 --2- 172.21.15.2:0/2118907798 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x561f70d0d5a0 0x561f70d0d9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-05-13T23:15:58.981 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.948+0000 7f2c4ffff700 1 --2- 172.21.15.2:0/2118907798 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x561f70d0c640 0x561f70d0cf80 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f2c40009d60 tx=0x7f2c40000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c2c6ee00994e65ed server_cookie=0 in_seq=0 out_seq=0 2022-05-13T23:15:58.981 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.948+0000 7f2c5499c700 1 --2- 172.21.15.2:0/4203339945 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f2c38096ce0 0x7f2c38099190 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f2c48007c50 tx=0x7f2c48004d10 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-05-13T23:15:58.982 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.948+0000 7f2c359fe700 1 -- 172.21.15.2:0/2118907798 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c400912f0 con 0x561f70d0c640 2022-05-13T23:15:58.982 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.948+0000 7f2c359fe700 1 -- 172.21.15.2:0/2118907798 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1087+0+0 (secure 0 0 0) 0x7f2c4001de90 con 0x561f70d0c640 2022-05-13T23:15:58.982 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.948+0000 7f2c359fe700 1 -- 172.21.15.2:0/2118907798 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c40091520 con 0x561f70d0c640 2022-05-13T23:15:58.983 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.949+0000 7f2c65764b80 1 -- 172.21.15.2:0/2118907798 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x561f70d0c640 msgr2=0x561f70d0cf80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:15:58.983 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.949+0000 7f2c65764b80 1 --2- 172.21.15.2:0/2118907798 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x561f70d0c640 0x561f70d0cf80 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f2c40009d60 tx=0x7f2c40000bc0 comp rx=0 tx=0).stop 2022-05-13T23:15:58.983 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.949+0000 7f2c65764b80 1 -- 172.21.15.2:0/2118907798 shutdown_connections 2022-05-13T23:15:58.983 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.949+0000 7f2c65764b80 1 --2- 172.21.15.2:0/2118907798 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x561f70d0d5a0 0x561f70d0d9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:58.984 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.949+0000 7f2c65764b80 1 --2- 172.21.15.2:0/2118907798 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x561f70d0bcf0 0x561f70d0c100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:58.984 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.949+0000 7f2c65764b80 1 --2- 172.21.15.2:0/2118907798 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x561f70d0c640 0x561f70d0cf80 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:58.984 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.949+0000 7f2c65764b80 1 -- 172.21.15.2:0/2118907798 >> 172.21.15.2:0/2118907798 conn(0x561f70d06b10 msgr2=0x561f70d06f00 unknown :-1 s=STATE_NONE l=0).mark_down 2022-05-13T23:15:58.984 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.949+0000 7f2c65764b80 1 -- 172.21.15.2:0/2118907798 shutdown_connections 2022-05-13T23:15:58.985 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.949+0000 7f2c65764b80 1 -- 172.21.15.2:0/2118907798 wait complete. 2022-05-13T23:15:58.985 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.950+0000 7f2c65764b80 1 -- start start 2022-05-13T23:15:58.985 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.950+0000 7f2c65764b80 1 Processor -- start 2022-05-13T23:15:58.986 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.950+0000 7f2c65764b80 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x561f70d0bcf0 0x561f70d15490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:15:58.986 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.950+0000 7f2c65764b80 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x561f70d0c640 0x561f70d159d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:15:58.987 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.950+0000 7f2c65764b80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x561f70d0d5a0 0x561f70d1ca30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:15:58.987 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.950+0000 7f2c65764b80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x561f70c597a0 con 0x561f70d0d5a0 2022-05-13T23:15:58.988 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.950+0000 7f2c65764b80 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x561f70af3650 con 0x561f70d0c640 2022-05-13T23:15:58.988 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.950+0000 7f2c65764b80 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x561f70d1d060 con 0x561f70d0bcf0 2022-05-13T23:15:58.989 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.951+0000 7f2c65764b80 10 client.0 _ll_register_callbacks cb 0x561f70ccdde0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2022-05-13T23:15:58.989 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.951+0000 7f2c4ffff700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x561f70d0bcf0 0x561f70d15490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:15:58.990 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.951+0000 7f2c5499c700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x561f70d0d5a0 0x561f70d1ca30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:15:58.990 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.951+0000 7f2c4ffff700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x561f70d0bcf0 0x561f70d15490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.2:45488/0 (socket says 172.21.15.2:45488) 2022-05-13T23:15:58.990 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.951+0000 7f2c4ffff700 1 -- 172.21.15.2:0/1928757593 learned_addr learned my addr 172.21.15.2:0/1928757593 (peer_addr_for_me v2:172.21.15.2:0/0) 2022-05-13T23:15:58.990 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.951+0000 7f2c4f7fe700 1 --2- 172.21.15.2:0/1928757593 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x561f70d0c640 0x561f70d159d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:15:58.991 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.951+0000 7f2c4ffff700 1 -- 172.21.15.2:0/1928757593 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x561f70d0c640 msgr2=0x561f70d159d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-05-13T23:15:58.992 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.951+0000 7f2c4ffff700 1 --2- 172.21.15.2:0/1928757593 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x561f70d0c640 0x561f70d159d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:58.992 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.951+0000 7f2c4ffff700 1 -- 172.21.15.2:0/1928757593 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x561f70d0d5a0 msgr2=0x561f70d1ca30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-05-13T23:15:58.993 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.951+0000 7f2c4ffff700 1 --2- 172.21.15.2:0/1928757593 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x561f70d0d5a0 0x561f70d1ca30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:58.993 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.951+0000 7f2c4ffff700 1 -- 172.21.15.2:0/1928757593 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2c44005040 con 0x561f70d0bcf0 2022-05-13T23:15:58.994 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.951+0000 7f2c4ffff700 1 --2- 172.21.15.2:0/1928757593 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x561f70d0bcf0 0x561f70d15490 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f2c40091220 tx=0x7f2c4000b040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2ae1959a54c3fd72 server_cookie=0 in_seq=0 out_seq=0 2022-05-13T23:15:58.994 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.951+0000 7f2c2affd700 10 client.0 ms_handle_connect on v2:172.21.15.115:3300/0 2022-05-13T23:15:58.994 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.952+0000 7f2c2affd700 1 -- 172.21.15.2:0/1928757593 <== mon.2 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c400912f0 con 0x561f70d0bcf0 2022-05-13T23:15:58.994 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.952+0000 7f2c65764b80 1 -- 172.21.15.2:0/1928757593 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x561f70c0d9a0 con 0x561f70d0bcf0 2022-05-13T23:15:58.995 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.952+0000 7f2c2affd700 1 -- 172.21.15.2:0/1928757593 <== mon.2 v2:172.21.15.115:3300/0 2 ==== osd_map(85..85 src has 1..85) v4 ==== 8120+0+0 (secure 0 0 0) 0x7f2c40097c10 con 0x561f70d0bcf0 2022-05-13T23:15:58.995 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.952+0000 7f2c2affd700 1 -- 172.21.15.2:0/1928757593 <== mon.2 v2:172.21.15.115:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2c40091060 con 0x561f70d0bcf0 2022-05-13T23:15:58.995 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.952+0000 7f2c65764b80 10 client.25175 fetch_fsmap learned FSMap version 26 2022-05-13T23:15:58.996 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.952+0000 7f2c65764b80 1 -- 172.21.15.2:0/1928757593 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) v3 -- 0x7f2c501168d0 con 0x561f70d0bcf0 2022-05-13T23:15:58.996 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.952+0000 7f2c2affd700 1 -- 172.21.15.2:0/1928757593 <== mon.2 v2:172.21.15.115:3300/0 4 ==== config(26 keys) v1 ==== 1087+0+0 (secure 0 0 0) 0x561f70c0d9a0 con 0x561f70d0bcf0 2022-05-13T23:15:58.996 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.952+0000 7f2c2affd700 1 -- 172.21.15.2:0/1928757593 <== mon.2 v2:172.21.15.115:3300/0 5 ==== fsmap(e 26) v1 ==== 7448+0+0 (secure 0 0 0) 0x7f2c40097780 con 0x561f70d0bcf0 2022-05-13T23:15:58.996 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.952+0000 7f2c65764b80 10 client.25175 fetch_fsmap finished waiting for FSMap version 26 2022-05-13T23:15:58.997 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.952+0000 7f2c65764b80 10 client.25175 resolve_mds: resolved name 'c' to daemon mds.c 2022-05-13T23:15:58.999 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.952+0000 7f2c65764b80 20 client.25175 populate_metadata read hostname 'smithi002' 2022-05-13T23:15:58.999 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.952+0000 7f2c65764b80 1 --2- 172.21.15.2:0/1928757593 >> [v2:172.21.15.115:6832/381933929,v1:172.21.15.115:6833/381933929] conn(0x561f70d1fcc0 0x561f70d1e4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:15:59.000 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.952+0000 7f2c65764b80 4 client.25175 mds_command: new command op to 24575 tid=0[{"prefix": "get_command_descriptions"}] 2022-05-13T23:15:59.000 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.952+0000 7f2c65764b80 1 -- 172.21.15.2:0/1928757593 --> [v2:172.21.15.115:6832/381933929,v1:172.21.15.115:6833/381933929] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x561f70c2e330 con 0x561f70d1fcc0 2022-05-13T23:15:59.000 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.953+0000 7f2c4f7fe700 1 --2- 172.21.15.2:0/1928757593 >> [v2:172.21.15.115:6832/381933929,v1:172.21.15.115:6833/381933929] conn(0x561f70d1fcc0 0x561f70d1e4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:15:59.001 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.954+0000 7f2c2affd700 10 client.25175 ms_handle_connect on v2:172.21.15.115:6832/381933929 2022-05-13T23:15:59.001 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.954+0000 7f2c4f7fe700 1 --2- 172.21.15.2:0/1928757593 >> [v2:172.21.15.115:6832/381933929,v1:172.21.15.115:6833/381933929] conn(0x561f70d1fcc0 0x561f70d1e4e0 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=d8d299d2f19adee3 server_cookie=55cf3949d83ad94 in_seq=0 out_seq=0 2022-05-13T23:15:59.002 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.954+0000 7f2c2affd700 1 -- 172.21.15.2:0/1928757593 <== mds.3 v2:172.21.15.115:6832/381933929 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17090 (crc 0 0 0) 0x561f70c2e330 con 0x561f70d1fcc0 2022-05-13T23:15:59.002 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.954+0000 7f2c2affd700 10 client.25175 handle_command_reply: tid=0 2022-05-13T23:15:59.003 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.954+0000 7f2c65764b80 1 client.25175 shutdown 2022-05-13T23:15:59.003 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.955+0000 7f2c65764b80 1 -- 172.21.15.2:0/1928757593 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x561f70d0bcf0 msgr2=0x561f70d15490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:15:59.003 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.955+0000 7f2c65764b80 1 --2- 172.21.15.2:0/1928757593 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x561f70d0bcf0 0x561f70d15490 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f2c40091220 tx=0x7f2c4000b040 comp rx=0 tx=0).stop 2022-05-13T23:15:59.004 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.955+0000 7f2c65764b80 1 -- 172.21.15.2:0/1928757593 shutdown_connections 2022-05-13T23:15:59.004 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.955+0000 7f2c65764b80 1 --2- 172.21.15.2:0/1928757593 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x561f70d0d5a0 0x561f70d1ca30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:59.005 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.955+0000 7f2c65764b80 1 --2- 172.21.15.2:0/1928757593 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x561f70d0bcf0 0x561f70d15490 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:59.005 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.955+0000 7f2c65764b80 1 --2- 172.21.15.2:0/1928757593 >> [v2:172.21.15.115:6832/381933929,v1:172.21.15.115:6833/381933929] conn(0x561f70d1fcc0 0x561f70d1e4e0 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:59.006 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.955+0000 7f2c65764b80 1 --2- 172.21.15.2:0/1928757593 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x561f70d0c640 0x561f70d159d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:59.006 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.955+0000 7f2c2affd700 0 client.25175 ms_handle_reset on v2:172.21.15.115:6832/381933929 2022-05-13T23:15:59.006 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.955+0000 7f2c65764b80 1 -- 172.21.15.2:0/1928757593 >> 172.21.15.2:0/1928757593 conn(0x561f70d06b10 msgr2=0x561f70d05d80 unknown :-1 s=STATE_NONE l=0).mark_down 2022-05-13T23:15:59.007 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.955+0000 7f2c65764b80 1 -- 172.21.15.2:0/1928757593 shutdown_connections 2022-05-13T23:15:59.007 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.955+0000 7f2c65764b80 1 -- 172.21.15.2:0/1928757593 wait complete. 2022-05-13T23:15:59.007 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.955+0000 7f2c65764b80 20 client.25175 trim_cache size 0 max 16384 2022-05-13T23:15:59.015 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.973+0000 7f2c65764b80 1 Processor -- start 2022-05-13T23:15:59.015 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.973+0000 7f2c65764b80 1 -- start start 2022-05-13T23:15:59.015 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.973+0000 7f2c65764b80 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x561f70d0bcf0 0x561f70d1e420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:15:59.015 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.973+0000 7f2c65764b80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x561f70d0c640 0x561f70d101b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:15:59.016 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.973+0000 7f2c65764b80 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x561f70d0d5a0 0x561f70d106f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:15:59.016 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.973+0000 7f2c65764b80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x561f70d05a10 con 0x561f70d0c640 2022-05-13T23:15:59.016 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.973+0000 7f2c65764b80 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x561f70c2e330 con 0x561f70d0bcf0 2022-05-13T23:15:59.016 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.973+0000 7f2c65764b80 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x561f70c0d9a0 con 0x561f70d0d5a0 2022-05-13T23:15:59.016 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.973+0000 7f2c4f7fe700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x561f70d0c640 0x561f70d101b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:15:59.017 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.973+0000 7f2c4f7fe700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x561f70d0c640 0x561f70d101b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:54164/0 (socket says 172.21.15.2:54164) 2022-05-13T23:15:59.017 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.973+0000 7f2c4f7fe700 1 -- 172.21.15.2:0/3465471 learned_addr learned my addr 172.21.15.2:0/3465471 (peer_addr_for_me v2:172.21.15.2:0/0) 2022-05-13T23:15:59.017 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.973+0000 7f2c4ffff700 1 --2- 172.21.15.2:0/3465471 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x561f70d0bcf0 0x561f70d1e420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:15:59.017 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.973+0000 7f2c5499c700 1 --2- 172.21.15.2:0/3465471 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x561f70d0d5a0 0x561f70d106f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:15:59.018 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.974+0000 7f2c4f7fe700 1 -- 172.21.15.2:0/3465471 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x561f70d0d5a0 msgr2=0x561f70d106f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-05-13T23:15:59.018 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.974+0000 7f2c4f7fe700 1 --2- 172.21.15.2:0/3465471 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x561f70d0d5a0 0x561f70d106f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:59.018 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.974+0000 7f2c4f7fe700 1 -- 172.21.15.2:0/3465471 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x561f70d0bcf0 msgr2=0x561f70d1e420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-05-13T23:15:59.024 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.974+0000 7f2c4f7fe700 1 --2- 172.21.15.2:0/3465471 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x561f70d0bcf0 0x561f70d1e420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:59.025 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.974+0000 7f2c4f7fe700 1 -- 172.21.15.2:0/3465471 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x561f70a9bfe0 con 0x561f70d0c640 2022-05-13T23:15:59.025 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.974+0000 7f2c4f7fe700 1 --2- 172.21.15.2:0/3465471 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x561f70d0c640 0x561f70d101b0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x561f70d14630 tx=0x7f2c4400b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e613ced9249693dd server_cookie=0 in_seq=0 out_seq=0 2022-05-13T23:15:59.025 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.974+0000 7f2c361ff700 1 -- 172.21.15.2:0/3465471 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c4400e040 con 0x561f70d0c640 2022-05-13T23:15:59.025 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.974+0000 7f2c361ff700 1 -- 172.21.15.2:0/3465471 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1087+0+0 (secure 0 0 0) 0x7f2c4400f040 con 0x561f70d0c640 2022-05-13T23:15:59.026 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.974+0000 7f2c361ff700 1 -- 172.21.15.2:0/3465471 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c44008350 con 0x561f70d0c640 2022-05-13T23:15:59.026 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.975+0000 7f2c65764b80 1 -- 172.21.15.2:0/3465471 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x561f70d0c640 msgr2=0x561f70d101b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:15:59.026 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.975+0000 7f2c65764b80 1 --2- 172.21.15.2:0/3465471 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x561f70d0c640 0x561f70d101b0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x561f70d14630 tx=0x7f2c4400b040 comp rx=0 tx=0).stop 2022-05-13T23:15:59.026 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.975+0000 7f2c65764b80 1 -- 172.21.15.2:0/3465471 shutdown_connections 2022-05-13T23:15:59.027 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.975+0000 7f2c65764b80 1 --2- 172.21.15.2:0/3465471 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x561f70d0d5a0 0x561f70d106f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:59.027 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.975+0000 7f2c65764b80 1 --2- 172.21.15.2:0/3465471 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x561f70d0bcf0 0x561f70d1e420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:59.028 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.975+0000 7f2c65764b80 1 --2- 172.21.15.2:0/3465471 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x561f70d0c640 0x561f70d101b0 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:59.028 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.975+0000 7f2c65764b80 1 -- 172.21.15.2:0/3465471 >> 172.21.15.2:0/3465471 conn(0x561f70d06b10 msgr2=0x561f70d05300 unknown :-1 s=STATE_NONE l=0).mark_down 2022-05-13T23:15:59.028 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.975+0000 7f2c65764b80 1 -- 172.21.15.2:0/3465471 shutdown_connections 2022-05-13T23:15:59.028 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.975+0000 7f2c65764b80 1 -- 172.21.15.2:0/3465471 wait complete. 2022-05-13T23:15:59.028 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.976+0000 7f2c65764b80 1 -- start start 2022-05-13T23:15:59.029 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.976+0000 7f2c65764b80 1 Processor -- start 2022-05-13T23:15:59.029 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.976+0000 7f2c65764b80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x561f70d0bcf0 0x561f70d26ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:15:59.029 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.976+0000 7f2c65764b80 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x561f70d0c640 0x561f70d27020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:15:59.029 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.976+0000 7f2c65764b80 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x561f70d0d5a0 0x561f70d728c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:15:59.030 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.976+0000 7f2c65764b80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x561f70d29fa0 con 0x561f70d0bcf0 2022-05-13T23:15:59.030 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.976+0000 7f2c65764b80 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x561f70d2a110 con 0x561f70d0d5a0 2022-05-13T23:15:59.030 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.976+0000 7f2c65764b80 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x561f70d72e00 con 0x561f70d0c640 2022-05-13T23:15:59.030 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.976+0000 7f2c65764b80 10 client.0 _ll_register_callbacks cb 0x561f70ccdde0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2022-05-13T23:15:59.031 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.977+0000 7f2c4ffff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x561f70d0bcf0 0x561f70d26ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:15:59.031 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.977+0000 7f2c4f7fe700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x561f70d0c640 0x561f70d27020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:15:59.031 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.977+0000 7f2c4ffff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x561f70d0bcf0 0x561f70d26ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:54168/0 (socket says 172.21.15.2:54168) 2022-05-13T23:15:59.032 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.977+0000 7f2c4ffff700 1 -- 172.21.15.2:0/3812769871 learned_addr learned my addr 172.21.15.2:0/3812769871 (peer_addr_for_me v2:172.21.15.2:0/0) 2022-05-13T23:15:59.032 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.977+0000 7f2c5499c700 1 --2- 172.21.15.2:0/3812769871 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x561f70d0d5a0 0x561f70d728c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:15:59.032 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.977+0000 7f2c4ffff700 1 -- 172.21.15.2:0/3812769871 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x561f70d0c640 msgr2=0x561f70d27020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-05-13T23:15:59.032 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.977+0000 7f2c4ffff700 1 --2- 172.21.15.2:0/3812769871 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x561f70d0c640 0x561f70d27020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:59.032 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.977+0000 7f2c4ffff700 1 -- 172.21.15.2:0/3812769871 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x561f70d0d5a0 msgr2=0x561f70d728c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-05-13T23:15:59.033 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.977+0000 7f2c4ffff700 1 --2- 172.21.15.2:0/3812769871 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x561f70d0d5a0 0x561f70d728c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:59.033 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.977+0000 7f2c4ffff700 1 -- 172.21.15.2:0/3812769871 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2c44004e50 con 0x561f70d0bcf0 2022-05-13T23:15:59.033 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.977+0000 7f2c4f7fe700 1 --2- 172.21.15.2:0/3812769871 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x561f70d0c640 0x561f70d27020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-05-13T23:15:59.033 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.977+0000 7f2c4ffff700 1 --2- 172.21.15.2:0/3812769871 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x561f70d0bcf0 0x561f70d26ae0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f2c40000c00 tx=0x7f2c4000b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c03d3cc867d13f7d server_cookie=0 in_seq=0 out_seq=0 2022-05-13T23:15:59.034 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.977+0000 7f2c2affd700 10 client.0 ms_handle_connect on v2:172.21.15.2:3300/0 2022-05-13T23:15:59.034 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.977+0000 7f2c2affd700 1 -- 172.21.15.2:0/3812769871 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c40097990 con 0x561f70d0bcf0 2022-05-13T23:15:59.034 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.978+0000 7f2c2affd700 1 -- 172.21.15.2:0/3812769871 <== mon.0 v2:172.21.15.2:3300/0 2 ==== osd_map(85..85 src has 1..85) v4 ==== 8120+0+0 (secure 0 0 0) 0x7f2c40097c10 con 0x561f70d0bcf0 2022-05-13T23:15:59.034 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.978+0000 7f2c65764b80 1 -- 172.21.15.2:0/3812769871 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x561f70cdc5f0 con 0x561f70d0bcf0 2022-05-13T23:15:59.035 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.978+0000 7f2c2affd700 1 -- 172.21.15.2:0/3812769871 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2c400ad440 con 0x561f70d0bcf0 2022-05-13T23:15:59.035 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.978+0000 7f2c65764b80 10 client.15060 fetch_fsmap learned FSMap version 26 2022-05-13T23:15:59.035 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.978+0000 7f2c65764b80 1 -- 172.21.15.2:0/3812769871 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) v3 -- 0x561f70ac3f80 con 0x561f70d0bcf0 2022-05-13T23:15:59.035 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.978+0000 7f2c2affd700 1 -- 172.21.15.2:0/3812769871 <== mon.0 v2:172.21.15.2:3300/0 4 ==== config(26 keys) v1 ==== 1087+0+0 (secure 0 0 0) 0x561f70cdc5f0 con 0x561f70d0bcf0 2022-05-13T23:15:59.035 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.978+0000 7f2c2affd700 1 -- 172.21.15.2:0/3812769871 <== mon.0 v2:172.21.15.2:3300/0 5 ==== fsmap(e 26) v1 ==== 7448+0+0 (secure 0 0 0) 0x7f2c400c00d0 con 0x561f70d0bcf0 2022-05-13T23:15:59.036 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.978+0000 7f2c65764b80 10 client.15060 fetch_fsmap finished waiting for FSMap version 26 2022-05-13T23:15:59.036 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.978+0000 7f2c65764b80 10 client.15060 resolve_mds: resolved name 'c' to daemon mds.c 2022-05-13T23:15:59.036 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.978+0000 7f2c65764b80 20 client.15060 populate_metadata read hostname 'smithi002' 2022-05-13T23:15:59.036 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.978+0000 7f2c65764b80 1 --2- 172.21.15.2:0/3812769871 >> [v2:172.21.15.115:6832/381933929,v1:172.21.15.115:6833/381933929] conn(0x561f70d76f50 0x561f70d73a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:15:59.037 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.978+0000 7f2c65764b80 4 client.15060 mds_command: new command op to 24575 tid=0[{"prefix": "status"}] 2022-05-13T23:15:59.037 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.978+0000 7f2c65764b80 1 -- 172.21.15.2:0/3812769871 --> [v2:172.21.15.115:6832/381933929,v1:172.21.15.115:6833/381933929] -- command(tid 0: {"prefix": "status"}) v1 -- 0x561f70bff790 con 0x561f70d76f50 2022-05-13T23:15:59.037 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.979+0000 7f2c4f7fe700 1 --2- 172.21.15.2:0/3812769871 >> [v2:172.21.15.115:6832/381933929,v1:172.21.15.115:6833/381933929] conn(0x561f70d76f50 0x561f70d73a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:15:59.037 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.979+0000 7f2c4f7fe700 1 --2- 172.21.15.2:0/3812769871 >> [v2:172.21.15.115:6832/381933929,v1:172.21.15.115:6833/381933929] conn(0x561f70d76f50 0x561f70d73a90 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=ff3abe3c5657caed server_cookie=302fe1ee6e7550e in_seq=0 out_seq=0 2022-05-13T23:15:59.038 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.979+0000 7f2c2affd700 10 client.15060 ms_handle_connect on v2:172.21.15.115:6832/381933929 2022-05-13T23:15:59.038 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.979+0000 7f2c2affd700 1 -- 172.21.15.2:0/3812769871 <== mds.3 v2:172.21.15.115:6832/381933929 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+332 (crc 0 0 0) 0x561f70bff790 con 0x561f70d76f50 2022-05-13T23:15:59.038 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.979+0000 7f2c2affd700 10 client.15060 handle_command_reply: tid=0 2022-05-13T23:15:59.038 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.980+0000 7f2c65764b80 1 client.15060 shutdown 2022-05-13T23:15:59.038 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.980+0000 7f2c65764b80 1 -- 172.21.15.2:0/3812769871 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x561f70d0bcf0 msgr2=0x561f70d26ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:15:59.039 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.980+0000 7f2c65764b80 1 --2- 172.21.15.2:0/3812769871 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x561f70d0bcf0 0x561f70d26ae0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f2c40000c00 tx=0x7f2c4000b040 comp rx=0 tx=0).stop 2022-05-13T23:15:59.039 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.980+0000 7f2c65764b80 1 -- 172.21.15.2:0/3812769871 shutdown_connections 2022-05-13T23:15:59.039 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.980+0000 7f2c65764b80 1 --2- 172.21.15.2:0/3812769871 >> [v2:172.21.15.115:6832/381933929,v1:172.21.15.115:6833/381933929] conn(0x561f70d76f50 0x561f70d73a90 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:59.039 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.980+0000 7f2c65764b80 1 --2- 172.21.15.2:0/3812769871 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x561f70d0d5a0 0x561f70d728c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:59.040 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.980+0000 7f2c65764b80 1 --2- 172.21.15.2:0/3812769871 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x561f70d0bcf0 0x561f70d26ae0 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:59.040 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.980+0000 7f2c65764b80 1 --2- 172.21.15.2:0/3812769871 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x561f70d0c640 0x561f70d27020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:59.040 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.980+0000 7f2c65764b80 1 -- 172.21.15.2:0/3812769871 >> 172.21.15.2:0/3812769871 conn(0x561f70d06b10 msgr2=0x561f70d1e3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-05-13T23:15:59.040 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.980+0000 7f2c2affd700 0 client.15060 ms_handle_reset on v2:172.21.15.115:6832/381933929 2022-05-13T23:15:59.041 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.980+0000 7f2c65764b80 1 -- 172.21.15.2:0/3812769871 shutdown_connections 2022-05-13T23:15:59.041 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.980+0000 7f2c65764b80 1 -- 172.21.15.2:0/3812769871 wait complete. 2022-05-13T23:15:59.041 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.980+0000 7f2c65764b80 20 client.15060 trim_cache size 0 max 16384 2022-05-13T23:15:59.042 INFO:teuthology.orchestra.run.smithi002.stdout:{ 2022-05-13T23:15:59.042 INFO:teuthology.orchestra.run.smithi002.stdout: "cluster_fsid": "16cf5f88-d311-11ec-be40-001a4aab830c", 2022-05-13T23:15:59.042 INFO:teuthology.orchestra.run.smithi002.stdout: "whoami": 3, 2022-05-13T23:15:59.042 INFO:teuthology.orchestra.run.smithi002.stdout: "id": 24575, 2022-05-13T23:15:59.043 INFO:teuthology.orchestra.run.smithi002.stdout: "want_state": "up:active", 2022-05-13T23:15:59.043 INFO:teuthology.orchestra.run.smithi002.stdout: "state": "up:active", 2022-05-13T23:15:59.043 INFO:teuthology.orchestra.run.smithi002.stdout: "fs_name": "cephfs", 2022-05-13T23:15:59.043 INFO:teuthology.orchestra.run.smithi002.stdout: "rank_uptime": 97.680648085000001, 2022-05-13T23:15:59.043 INFO:teuthology.orchestra.run.smithi002.stdout: "mdsmap_epoch": 26, 2022-05-13T23:15:59.044 INFO:teuthology.orchestra.run.smithi002.stdout: "osdmap_epoch": 85, 2022-05-13T23:15:59.044 INFO:teuthology.orchestra.run.smithi002.stdout: "osdmap_epoch_barrier": 83, 2022-05-13T23:15:59.044 INFO:teuthology.orchestra.run.smithi002.stdout: "uptime": 115.87039005299999 2022-05-13T23:15:59.044 INFO:teuthology.orchestra.run.smithi002.stdout:} 2022-05-13T23:15:59.046 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.981+0000 7f2c5519d700 1 -- 172.21.15.2:0/4203339945 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f2c38096ce0 msgr2=0x7f2c38099190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:15:59.046 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.981+0000 7f2c5519d700 1 --2- 172.21.15.2:0/4203339945 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f2c38096ce0 0x7f2c38099190 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f2c48007c50 tx=0x7f2c48004d10 comp rx=0 tx=0).stop 2022-05-13T23:15:59.047 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.981+0000 7f2c5519d700 1 -- 172.21.15.2:0/4203339945 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f2c500fb4e0 msgr2=0x7f2c500701d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:15:59.047 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.981+0000 7f2c5519d700 1 --2- 172.21.15.2:0/4203339945 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f2c500fb4e0 0x7f2c500701d0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f2c40009e00 tx=0x7f2c40009cb0 comp rx=0 tx=0).stop 2022-05-13T23:15:59.047 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.981+0000 7f2c5519d700 1 -- 172.21.15.2:0/4203339945 shutdown_connections 2022-05-13T23:15:59.047 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.981+0000 7f2c5519d700 1 --2- 172.21.15.2:0/4203339945 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f2c38096ce0 0x7f2c38099190 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:59.048 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.981+0000 7f2c5519d700 1 --2- 172.21.15.2:0/4203339945 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2c500fd110 0x7f2c5006ed70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:59.048 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.981+0000 7f2c5519d700 1 --2- 172.21.15.2:0/4203339945 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f2c500fb4e0 0x7f2c500701d0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:59.048 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.981+0000 7f2c5519d700 1 --2- 172.21.15.2:0/4203339945 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f2c500fc780 0x7f2c5006e830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:59.048 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.981+0000 7f2c5519d700 1 -- 172.21.15.2:0/4203339945 >> 172.21.15.2:0/4203339945 conn(0x7f2c500f6cb0 msgr2=0x7f2c500f90a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-05-13T23:15:59.049 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.981+0000 7f2c5519d700 1 -- 172.21.15.2:0/4203339945 shutdown_connections 2022-05-13T23:15:59.049 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:58.981+0000 7f2c5519d700 1 -- 172.21.15.2:0/4203339945 wait complete. 2022-05-13T23:15:59.049 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.f status 2022-05-13T23:15:59.271 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.267+0000 7f23d659d700 1 -- 172.21.15.2:0/129729787 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f23d0065630 msgr2=0x7f23d00fe070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:15:59.272 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.267+0000 7f23d659d700 1 --2- 172.21.15.2:0/129729787 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f23d0065630 0x7f23d00fe070 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f23cc004870 tx=0x7f23cc009cb0 comp rx=0 tx=0).stop 2022-05-13T23:15:59.272 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.267+0000 7f23d659d700 1 -- 172.21.15.2:0/129729787 shutdown_connections 2022-05-13T23:15:59.272 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.267+0000 7f23d659d700 1 --2- 172.21.15.2:0/129729787 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f23d0101110 0x7f23d0103590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:59.273 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.267+0000 7f23d659d700 1 --2- 172.21.15.2:0/129729787 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f23d0065630 0x7f23d00fe070 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:59.273 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.267+0000 7f23d659d700 1 --2- 172.21.15.2:0/129729787 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f23d00fe640 0x7f23d0100a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:59.273 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.267+0000 7f23d659d700 1 -- 172.21.15.2:0/129729787 >> 172.21.15.2:0/129729787 conn(0x7f23d00f5b80 msgr2=0x7f23d00f7fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-05-13T23:15:59.273 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.267+0000 7f23d659d700 1 -- 172.21.15.2:0/129729787 shutdown_connections 2022-05-13T23:15:59.273 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.267+0000 7f23d659d700 1 -- 172.21.15.2:0/129729787 wait complete. 2022-05-13T23:15:59.274 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.268+0000 7f23d659d700 1 Processor -- start 2022-05-13T23:15:59.274 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.268+0000 7f23d659d700 1 -- start start 2022-05-13T23:15:59.274 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.268+0000 7f23d659d700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f23d0065630 0x7f23d010e520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:15:59.274 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.268+0000 7f23d659d700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f23d00fe640 0x7f23d010ea60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:15:59.275 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.268+0000 7f23d659d700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f23d0101110 0x7f23d0115af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:15:59.275 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.268+0000 7f23d659d700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23d0116030 con 0x7f23d0101110 2022-05-13T23:15:59.275 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.268+0000 7f23d659d700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23d01161a0 con 0x7f23d00fe640 2022-05-13T23:15:59.275 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.268+0000 7f23d659d700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23d0116360 con 0x7f23d0065630 2022-05-13T23:15:59.281 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.269+0000 7f23d4d9a700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f23d00fe640 0x7f23d010ea60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:15:59.282 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.269+0000 7f23d559b700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f23d0065630 0x7f23d010e520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:15:59.282 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.269+0000 7f23d5d9c700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f23d0101110 0x7f23d0115af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:15:59.282 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.269+0000 7f23d4d9a700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f23d00fe640 0x7f23d010ea60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.2:37476/0 (socket says 172.21.15.2:37476) 2022-05-13T23:15:59.282 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.269+0000 7f23d4d9a700 1 -- 172.21.15.2:0/4280401755 learned_addr learned my addr 172.21.15.2:0/4280401755 (peer_addr_for_me v2:172.21.15.2:0/0) 2022-05-13T23:15:59.283 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.269+0000 7f23d4d9a700 1 -- 172.21.15.2:0/4280401755 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f23d0065630 msgr2=0x7f23d010e520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:15:59.283 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.269+0000 7f23d4d9a700 1 --2- 172.21.15.2:0/4280401755 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f23d0065630 0x7f23d010e520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:59.283 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.269+0000 7f23d4d9a700 1 -- 172.21.15.2:0/4280401755 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f23d0101110 msgr2=0x7f23d0115af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:15:59.283 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.269+0000 7f23d4d9a700 1 --2- 172.21.15.2:0/4280401755 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f23d0101110 0x7f23d0115af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:59.283 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.269+0000 7f23d4d9a700 1 -- 172.21.15.2:0/4280401755 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23cc005040 con 0x7f23d00fe640 2022-05-13T23:15:59.284 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.270+0000 7f23d4d9a700 1 --2- 172.21.15.2:0/4280401755 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f23d00fe640 0x7f23d010ea60 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f23c000e940 tx=0x7f23c000b040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-05-13T23:15:59.284 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.270+0000 7f23c5ffb700 1 -- 172.21.15.2:0/4280401755 <== mon.1 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23c0007b40 con 0x7f23d00fe640 2022-05-13T23:15:59.284 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.270+0000 7f23d659d700 1 -- 172.21.15.2:0/4280401755 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f23d01165e0 con 0x7f23d00fe640 2022-05-13T23:15:59.284 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.270+0000 7f23c5ffb700 1 -- 172.21.15.2:0/4280401755 <== mon.1 v2:172.21.15.47:3300/0 2 ==== config(26 keys) v1 ==== 1087+0+0 (secure 0 0 0) 0x7f23c000f040 con 0x7f23d00fe640 2022-05-13T23:15:59.285 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.270+0000 7f23c5ffb700 1 -- 172.21.15.2:0/4280401755 <== mon.1 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23c00148b0 con 0x7f23d00fe640 2022-05-13T23:15:59.285 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.270+0000 7f23d659d700 1 -- 172.21.15.2:0/4280401755 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f23d0116ba0 con 0x7f23d00fe640 2022-05-13T23:15:59.295 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.273+0000 7f23c5ffb700 1 -- 172.21.15.2:0/4280401755 <== mon.1 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 16) v1 ==== 125136+0+0 (secure 0 0 0) 0x7f23c001b030 con 0x7f23d00fe640 2022-05-13T23:15:59.296 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.274+0000 7f23e6a70b80 1 Processor -- start 2022-05-13T23:15:59.296 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.274+0000 7f23c5ffb700 1 --2- 172.21.15.2:0/4280401755 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f23bc096d60 0x7f23bc099210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:15:59.296 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.274+0000 7f23e6a70b80 1 -- start start 2022-05-13T23:15:59.297 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.275+0000 7f23c5ffb700 1 -- 172.21.15.2:0/4280401755 <== mon.1 v2:172.21.15.47:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8120+0+0 (secure 0 0 0) 0x7f23c00beaa0 con 0x7f23d00fe640 2022-05-13T23:15:59.297 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.275+0000 7f23d5d9c700 1 --2- 172.21.15.2:0/4280401755 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f23bc096d60 0x7f23bc099210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:15:59.297 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.275+0000 7f23e6a70b80 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x5564fe515cf0 0x5564fe516100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:15:59.298 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.275+0000 7f23e6a70b80 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5564fe516640 0x5564fe516f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:15:59.298 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.275+0000 7f23e6a70b80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5564fe5175a0 0x5564fe5179f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:15:59.298 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.275+0000 7f23e6a70b80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x5564fe4103a0 con 0x5564fe5175a0 2022-05-13T23:15:59.299 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.275+0000 7f23e6a70b80 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x5564fe3e5400 con 0x5564fe515cf0 2022-05-13T23:15:59.299 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.275+0000 7f23e6a70b80 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x5564fe4696c0 con 0x5564fe516640 2022-05-13T23:15:59.299 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.275+0000 7f23d559b700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x5564fe515cf0 0x5564fe516100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:15:59.299 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.275+0000 7f23d4d9a700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5564fe516640 0x5564fe516f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:15:59.300 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.275+0000 7f23d5d9c700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5564fe5175a0 0x5564fe5179f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:15:59.300 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.275+0000 7f23d559b700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x5564fe515cf0 0x5564fe516100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.2:37488/0 (socket says 172.21.15.2:37488) 2022-05-13T23:15:59.300 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.275+0000 7f23d559b700 1 -- 172.21.15.2:0/138529122 learned_addr learned my addr 172.21.15.2:0/138529122 (peer_addr_for_me v2:172.21.15.2:0/0) 2022-05-13T23:15:59.300 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.276+0000 7f23d4d9a700 1 -- 172.21.15.2:0/138529122 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x5564fe515cf0 msgr2=0x5564fe516100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-05-13T23:15:59.301 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.276+0000 7f23d4d9a700 1 --2- 172.21.15.2:0/138529122 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x5564fe515cf0 0x5564fe516100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:59.301 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.276+0000 7f23d4d9a700 1 -- 172.21.15.2:0/138529122 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5564fe5175a0 msgr2=0x5564fe5179f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-05-13T23:15:59.301 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.276+0000 7f23d4d9a700 1 --2- 172.21.15.2:0/138529122 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5564fe5175a0 0x5564fe5179f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:59.302 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.276+0000 7f23d4d9a700 1 -- 172.21.15.2:0/138529122 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5564fe2fb360 con 0x5564fe516640 2022-05-13T23:15:59.302 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.276+0000 7f23d5d9c700 1 --2- 172.21.15.2:0/4280401755 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f23bc096d60 0x7f23bc099210 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f23c8004fd0 tx=0x7f23c8009040 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-05-13T23:15:59.302 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.276+0000 7f23d4d9a700 1 --2- 172.21.15.2:0/138529122 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5564fe516640 0x5564fe516f80 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f23c00097b0 tx=0x7f23c0004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=53f31af7ab13cec0 server_cookie=0 in_seq=0 out_seq=0 2022-05-13T23:15:59.302 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.276+0000 7f23b2ffd700 1 -- 172.21.15.2:0/138529122 <== mon.2 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23c008f370 con 0x5564fe516640 2022-05-13T23:15:59.303 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.276+0000 7f23b2ffd700 1 -- 172.21.15.2:0/138529122 <== mon.2 v2:172.21.15.115:3300/0 2 ==== config(26 keys) v1 ==== 1087+0+0 (secure 0 0 0) 0x7f23c0007ca0 con 0x5564fe516640 2022-05-13T23:15:59.303 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.276+0000 7f23b2ffd700 1 -- 172.21.15.2:0/138529122 <== mon.2 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23c008fee0 con 0x5564fe516640 2022-05-13T23:15:59.303 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.277+0000 7f23e6a70b80 1 -- 172.21.15.2:0/138529122 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5564fe516640 msgr2=0x5564fe516f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:15:59.303 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.277+0000 7f23e6a70b80 1 --2- 172.21.15.2:0/138529122 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5564fe516640 0x5564fe516f80 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f23c00097b0 tx=0x7f23c0004730 comp rx=0 tx=0).stop 2022-05-13T23:15:59.304 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.277+0000 7f23e6a70b80 1 -- 172.21.15.2:0/138529122 shutdown_connections 2022-05-13T23:15:59.304 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.277+0000 7f23e6a70b80 1 --2- 172.21.15.2:0/138529122 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5564fe5175a0 0x5564fe5179f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:59.304 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.277+0000 7f23e6a70b80 1 --2- 172.21.15.2:0/138529122 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x5564fe515cf0 0x5564fe516100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:59.305 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.277+0000 7f23e6a70b80 1 --2- 172.21.15.2:0/138529122 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5564fe516640 0x5564fe516f80 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:59.305 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.277+0000 7f23e6a70b80 1 -- 172.21.15.2:0/138529122 >> 172.21.15.2:0/138529122 conn(0x5564fe510b10 msgr2=0x5564fe510f00 unknown :-1 s=STATE_NONE l=0).mark_down 2022-05-13T23:15:59.305 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.277+0000 7f23e6a70b80 1 -- 172.21.15.2:0/138529122 shutdown_connections 2022-05-13T23:15:59.305 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.277+0000 7f23e6a70b80 1 -- 172.21.15.2:0/138529122 wait complete. 2022-05-13T23:15:59.306 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.278+0000 7f23e6a70b80 1 -- start start 2022-05-13T23:15:59.306 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.278+0000 7f23e6a70b80 1 Processor -- start 2022-05-13T23:15:59.306 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.278+0000 7f23e6a70b80 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5564fe515cf0 0x5564fe51f590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:15:59.306 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.278+0000 7f23e6a70b80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5564fe516640 0x5564fe51fad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:15:59.307 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.278+0000 7f23e6a70b80 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x5564fe5175a0 0x5564fe526b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:15:59.307 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.279+0000 7f23e6a70b80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x5564fe4637a0 con 0x5564fe516640 2022-05-13T23:15:59.307 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.279+0000 7f23e6a70b80 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x5564fe2fd650 con 0x5564fe5175a0 2022-05-13T23:15:59.308 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.279+0000 7f23e6a70b80 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x5564fe527160 con 0x5564fe515cf0 2022-05-13T23:15:59.308 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.279+0000 7f23e6a70b80 10 client.0 _ll_register_callbacks cb 0x5564fe4d7de0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2022-05-13T23:15:59.308 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.279+0000 7f23d4d9a700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5564fe516640 0x5564fe51fad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:15:59.309 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.279+0000 7f23d4d9a700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5564fe516640 0x5564fe51fad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:54208/0 (socket says 172.21.15.2:54208) 2022-05-13T23:15:59.309 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.279+0000 7f23d4d9a700 1 -- 172.21.15.2:0/180771857 learned_addr learned my addr 172.21.15.2:0/180771857 (peer_addr_for_me v2:172.21.15.2:0/0) 2022-05-13T23:15:59.309 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.279+0000 7f23d5d9c700 1 --2- 172.21.15.2:0/180771857 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x5564fe5175a0 0x5564fe526b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:15:59.309 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.279+0000 7f23d559b700 1 --2- 172.21.15.2:0/180771857 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5564fe515cf0 0x5564fe51f590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:15:59.310 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.279+0000 7f23d4d9a700 1 -- 172.21.15.2:0/180771857 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5564fe515cf0 msgr2=0x5564fe51f590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-05-13T23:15:59.310 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.279+0000 7f23d4d9a700 1 --2- 172.21.15.2:0/180771857 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5564fe515cf0 0x5564fe51f590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:59.310 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.279+0000 7f23d4d9a700 1 -- 172.21.15.2:0/180771857 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x5564fe5175a0 msgr2=0x5564fe526b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-05-13T23:15:59.310 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.279+0000 7f23d4d9a700 1 --2- 172.21.15.2:0/180771857 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x5564fe5175a0 0x5564fe526b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:59.311 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.279+0000 7f23d4d9a700 1 -- 172.21.15.2:0/180771857 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f23d0116ba0 con 0x5564fe516640 2022-05-13T23:15:59.311 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.279+0000 7f23d559b700 1 --2- 172.21.15.2:0/180771857 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5564fe515cf0 0x5564fe51f590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2022-05-13T23:15:59.311 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.280+0000 7f23d4d9a700 1 --2- 172.21.15.2:0/180771857 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5564fe516640 0x5564fe51fad0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f23c0014a50 tx=0x7f23c0004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=192036b43f4e891a server_cookie=0 in_seq=0 out_seq=0 2022-05-13T23:15:59.312 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.280+0000 7f23abfff700 10 client.0 ms_handle_connect on v2:172.21.15.2:3300/0 2022-05-13T23:15:59.312 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.280+0000 7f23abfff700 1 -- 172.21.15.2:0/180771857 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23c008f370 con 0x5564fe516640 2022-05-13T23:15:59.312 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.280+0000 7f23e6a70b80 1 -- 172.21.15.2:0/180771857 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5564fe4179a0 con 0x5564fe516640 2022-05-13T23:15:59.313 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.280+0000 7f23abfff700 1 -- 172.21.15.2:0/180771857 <== mon.0 v2:172.21.15.2:3300/0 2 ==== osd_map(85..85 src has 1..85) v4 ==== 8120+0+0 (secure 0 0 0) 0x7f23c008fa10 con 0x5564fe516640 2022-05-13T23:15:59.313 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.280+0000 7f23abfff700 1 -- 172.21.15.2:0/180771857 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7f23c00aa070 con 0x5564fe516640 2022-05-13T23:15:59.313 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.280+0000 7f23e6a70b80 10 client.15072 fetch_fsmap learned FSMap version 26 2022-05-13T23:15:59.314 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.280+0000 7f23e6a70b80 1 -- 172.21.15.2:0/180771857 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) v3 -- 0x5564fe2a5fe0 con 0x5564fe516640 2022-05-13T23:15:59.314 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.280+0000 7f23abfff700 1 -- 172.21.15.2:0/180771857 <== mon.0 v2:172.21.15.2:3300/0 4 ==== config(26 keys) v1 ==== 1087+0+0 (secure 0 0 0) 0x7f23c0073080 con 0x5564fe516640 2022-05-13T23:15:59.314 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.280+0000 7f23abfff700 1 -- 172.21.15.2:0/180771857 <== mon.0 v2:172.21.15.2:3300/0 5 ==== fsmap(e 26) v1 ==== 7448+0+0 (secure 0 0 0) 0x7f23c006fc80 con 0x5564fe516640 2022-05-13T23:15:59.314 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.281+0000 7f23e6a70b80 10 client.15072 fetch_fsmap finished waiting for FSMap version 26 2022-05-13T23:15:59.315 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.281+0000 7f23e6a70b80 10 client.15072 resolve_mds: resolved name 'f' to daemon mds.f 2022-05-13T23:15:59.315 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.281+0000 7f23e6a70b80 20 client.15072 populate_metadata read hostname 'smithi002' 2022-05-13T23:15:59.315 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.281+0000 7f23e6a70b80 1 --2- 172.21.15.2:0/180771857 >> [v2:172.21.15.115:6834/71062276,v1:172.21.15.115:6835/71062276] conn(0x5564fe529f90 0x5564fe52f270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:15:59.316 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.281+0000 7f23e6a70b80 4 client.15072 mds_command: new command op to 24587 tid=0[{"prefix": "get_command_descriptions"}] 2022-05-13T23:15:59.316 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.281+0000 7f23e6a70b80 1 -- 172.21.15.2:0/180771857 --> [v2:172.21.15.115:6834/71062276,v1:172.21.15.115:6835/71062276] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5564fe438330 con 0x5564fe529f90 2022-05-13T23:15:59.316 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.281+0000 7f23d559b700 1 --2- 172.21.15.2:0/180771857 >> [v2:172.21.15.115:6834/71062276,v1:172.21.15.115:6835/71062276] conn(0x5564fe529f90 0x5564fe52f270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:15:59.317 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.282+0000 7f23abfff700 10 client.15072 ms_handle_connect on v2:172.21.15.115:6834/71062276 2022-05-13T23:15:59.317 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.282+0000 7f23d559b700 1 --2- 172.21.15.2:0/180771857 >> [v2:172.21.15.115:6834/71062276,v1:172.21.15.115:6835/71062276] conn(0x5564fe529f90 0x5564fe52f270 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=5555587ad6f3e2af server_cookie=aa6d56e1b7133db0 in_seq=0 out_seq=0 2022-05-13T23:15:59.317 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.282+0000 7f23abfff700 1 -- 172.21.15.2:0/180771857 <== mds.2 v2:172.21.15.115:6834/71062276 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17090 (crc 0 0 0) 0x5564fe438330 con 0x5564fe529f90 2022-05-13T23:15:59.318 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.282+0000 7f23abfff700 10 client.15072 handle_command_reply: tid=0 2022-05-13T23:15:59.318 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.282+0000 7f23e6a70b80 1 client.15072 shutdown 2022-05-13T23:15:59.318 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.283+0000 7f23e6a70b80 1 -- 172.21.15.2:0/180771857 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5564fe516640 msgr2=0x5564fe51fad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:15:59.319 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.283+0000 7f23e6a70b80 1 --2- 172.21.15.2:0/180771857 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5564fe516640 0x5564fe51fad0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f23c0014a50 tx=0x7f23c0004730 comp rx=0 tx=0).stop 2022-05-13T23:15:59.319 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.283+0000 7f23e6a70b80 1 -- 172.21.15.2:0/180771857 shutdown_connections 2022-05-13T23:15:59.319 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.283+0000 7f23e6a70b80 1 --2- 172.21.15.2:0/180771857 >> [v2:172.21.15.115:6834/71062276,v1:172.21.15.115:6835/71062276] conn(0x5564fe529f90 0x5564fe52f270 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:59.320 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.283+0000 7f23e6a70b80 1 --2- 172.21.15.2:0/180771857 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x5564fe5175a0 0x5564fe526b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:59.320 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.283+0000 7f23e6a70b80 1 --2- 172.21.15.2:0/180771857 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5564fe515cf0 0x5564fe51f590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:59.321 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.283+0000 7f23e6a70b80 1 --2- 172.21.15.2:0/180771857 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5564fe516640 0x5564fe51fad0 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:59.322 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.283+0000 7f23abfff700 0 client.15072 ms_handle_reset on v2:172.21.15.115:6834/71062276 2022-05-13T23:15:59.322 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.283+0000 7f23e6a70b80 1 -- 172.21.15.2:0/180771857 >> 172.21.15.2:0/180771857 conn(0x5564fe510b10 msgr2=0x5564fe50fd80 unknown :-1 s=STATE_NONE l=0).mark_down 2022-05-13T23:15:59.323 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.283+0000 7f23e6a70b80 1 -- 172.21.15.2:0/180771857 shutdown_connections 2022-05-13T23:15:59.323 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.283+0000 7f23e6a70b80 1 -- 172.21.15.2:0/180771857 wait complete. 2022-05-13T23:15:59.323 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.283+0000 7f23e6a70b80 20 client.15072 trim_cache size 0 max 16384 2022-05-13T23:15:59.338 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.301+0000 7f23e6a70b80 1 Processor -- start 2022-05-13T23:15:59.338 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.301+0000 7f23e6a70b80 1 -- start start 2022-05-13T23:15:59.339 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.301+0000 7f23e6a70b80 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x5564fe515cf0 0x5564fe51ed40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:15:59.340 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.301+0000 7f23e6a70b80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5564fe516640 0x5564fe50f300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:15:59.341 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.301+0000 7f23e6a70b80 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5564fe5175a0 0x5564fe51f560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:15:59.342 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.301+0000 7f23e6a70b80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x5564fe2ff7f0 con 0x5564fe516640 2022-05-13T23:15:59.342 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.301+0000 7f23e6a70b80 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x5564fe42cbe0 con 0x5564fe515cf0 2022-05-13T23:15:59.343 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.301+0000 7f23e6a70b80 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x5564fe309b60 con 0x5564fe5175a0 2022-05-13T23:15:59.343 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.302+0000 7f23d559b700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x5564fe515cf0 0x5564fe51ed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:15:59.343 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.302+0000 7f23d5d9c700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5564fe5175a0 0x5564fe51f560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:15:59.344 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.302+0000 7f23d559b700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x5564fe515cf0 0x5564fe51ed40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.2:37504/0 (socket says 172.21.15.2:37504) 2022-05-13T23:15:59.344 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.302+0000 7f23d559b700 1 -- 172.21.15.2:0/3738916847 learned_addr learned my addr 172.21.15.2:0/3738916847 (peer_addr_for_me v2:172.21.15.2:0/0) 2022-05-13T23:15:59.344 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.302+0000 7f23d4d9a700 1 --2- 172.21.15.2:0/3738916847 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5564fe516640 0x5564fe50f300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:15:59.344 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.302+0000 7f23d5d9c700 1 -- 172.21.15.2:0/3738916847 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x5564fe515cf0 msgr2=0x5564fe51ed40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-05-13T23:15:59.345 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.302+0000 7f23d5d9c700 1 --2- 172.21.15.2:0/3738916847 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x5564fe515cf0 0x5564fe51ed40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:59.345 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.302+0000 7f23d5d9c700 1 -- 172.21.15.2:0/3738916847 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5564fe516640 msgr2=0x5564fe50f300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-05-13T23:15:59.345 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.302+0000 7f23d5d9c700 1 --2- 172.21.15.2:0/3738916847 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5564fe516640 0x5564fe50f300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:59.345 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.302+0000 7f23d5d9c700 1 -- 172.21.15.2:0/3738916847 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5564fe2cdf80 con 0x5564fe5175a0 2022-05-13T23:15:59.346 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.302+0000 7f23d559b700 1 --2- 172.21.15.2:0/3738916847 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x5564fe515cf0 0x5564fe51ed40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-05-13T23:15:59.346 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.302+0000 7f23d4d9a700 1 --2- 172.21.15.2:0/3738916847 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5564fe516640 0x5564fe50f300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-05-13T23:15:59.346 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.303+0000 7f23d5d9c700 1 --2- 172.21.15.2:0/3738916847 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5564fe5175a0 0x5564fe51f560 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f23c8014d30 tx=0x7f23c801e040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=30b6a5408b5c7b69 server_cookie=0 in_seq=0 out_seq=0 2022-05-13T23:15:59.346 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.303+0000 7f23b37fe700 1 -- 172.21.15.2:0/3738916847 <== mon.2 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23c8021050 con 0x5564fe5175a0 2022-05-13T23:15:59.347 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.303+0000 7f23b37fe700 1 -- 172.21.15.2:0/3738916847 <== mon.2 v2:172.21.15.115:3300/0 2 ==== config(26 keys) v1 ==== 1087+0+0 (secure 0 0 0) 0x7f23c8022040 con 0x5564fe5175a0 2022-05-13T23:15:59.347 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.303+0000 7f23b37fe700 1 -- 172.21.15.2:0/3738916847 <== mon.2 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23c801ac20 con 0x5564fe5175a0 2022-05-13T23:15:59.347 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.303+0000 7f23e6a70b80 1 -- 172.21.15.2:0/3738916847 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5564fe5175a0 msgr2=0x5564fe51f560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:15:59.347 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.303+0000 7f23e6a70b80 1 --2- 172.21.15.2:0/3738916847 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5564fe5175a0 0x5564fe51f560 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f23c8014d30 tx=0x7f23c801e040 comp rx=0 tx=0).stop 2022-05-13T23:15:59.348 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.303+0000 7f23e6a70b80 1 -- 172.21.15.2:0/3738916847 shutdown_connections 2022-05-13T23:15:59.348 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.303+0000 7f23e6a70b80 1 --2- 172.21.15.2:0/3738916847 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5564fe5175a0 0x5564fe51f560 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:59.348 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.303+0000 7f23e6a70b80 1 --2- 172.21.15.2:0/3738916847 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x5564fe515cf0 0x5564fe51ed40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:59.348 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.303+0000 7f23e6a70b80 1 --2- 172.21.15.2:0/3738916847 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5564fe516640 0x5564fe50f300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:59.349 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.303+0000 7f23e6a70b80 1 -- 172.21.15.2:0/3738916847 >> 172.21.15.2:0/3738916847 conn(0x5564fe510b10 msgr2=0x5564fe533290 unknown :-1 s=STATE_NONE l=0).mark_down 2022-05-13T23:15:59.349 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.304+0000 7f23e6a70b80 1 -- 172.21.15.2:0/3738916847 shutdown_connections 2022-05-13T23:15:59.349 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.304+0000 7f23e6a70b80 1 -- 172.21.15.2:0/3738916847 wait complete. 2022-05-13T23:15:59.350 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.304+0000 7f23e6a70b80 1 -- start start 2022-05-13T23:15:59.350 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.304+0000 7f23e6a70b80 1 Processor -- start 2022-05-13T23:15:59.350 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.305+0000 7f23e6a70b80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5564fe515cf0 0x5564fe523cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:15:59.350 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.305+0000 7f23e6a70b80 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5564fe516640 0x5564fe5241f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:15:59.351 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.305+0000 7f23e6a70b80 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x5564fe5175a0 0x5564fe57c350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:15:59.351 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.305+0000 7f23e6a70b80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x5564fe57c8d0 con 0x5564fe515cf0 2022-05-13T23:15:59.351 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.305+0000 7f23e6a70b80 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x5564fe57ca40 con 0x5564fe5175a0 2022-05-13T23:15:59.351 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.305+0000 7f23e6a70b80 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x5564fe57cc00 con 0x5564fe516640 2022-05-13T23:15:59.352 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.305+0000 7f23e6a70b80 10 client.0 _ll_register_callbacks cb 0x5564fe4d7de0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2022-05-13T23:15:59.352 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.305+0000 7f23d559b700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5564fe515cf0 0x5564fe523cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:15:59.352 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.305+0000 7f23d559b700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5564fe515cf0 0x5564fe523cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:54224/0 (socket says 172.21.15.2:54224) 2022-05-13T23:15:59.352 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.305+0000 7f23d559b700 1 -- 172.21.15.2:0/2752686484 learned_addr learned my addr 172.21.15.2:0/2752686484 (peer_addr_for_me v2:172.21.15.2:0/0) 2022-05-13T23:15:59.353 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.305+0000 7f23d4d9a700 1 --2- 172.21.15.2:0/2752686484 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5564fe516640 0x5564fe5241f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:15:59.353 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.305+0000 7f23d5d9c700 1 --2- 172.21.15.2:0/2752686484 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x5564fe5175a0 0x5564fe57c350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:15:59.354 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.305+0000 7f23d559b700 1 -- 172.21.15.2:0/2752686484 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5564fe516640 msgr2=0x5564fe5241f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-05-13T23:15:59.354 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.305+0000 7f23d559b700 1 --2- 172.21.15.2:0/2752686484 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5564fe516640 0x5564fe5241f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:59.355 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.305+0000 7f23d559b700 1 -- 172.21.15.2:0/2752686484 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x5564fe5175a0 msgr2=0x5564fe57c350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-05-13T23:15:59.355 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.305+0000 7f23d559b700 1 --2- 172.21.15.2:0/2752686484 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x5564fe5175a0 0x5564fe57c350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:59.355 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.305+0000 7f23d559b700 1 -- 172.21.15.2:0/2752686484 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f23cc017040 con 0x5564fe515cf0 2022-05-13T23:15:59.355 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.306+0000 7f23d559b700 1 --2- 172.21.15.2:0/2752686484 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5564fe515cf0 0x5564fe523cb0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f23cc004870 tx=0x7f23cc023040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bfae7e78332b0a24 server_cookie=0 in_seq=0 out_seq=0 2022-05-13T23:15:59.356 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.306+0000 7f23abfff700 10 client.0 ms_handle_connect on v2:172.21.15.2:3300/0 2022-05-13T23:15:59.356 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.306+0000 7f23abfff700 1 -- 172.21.15.2:0/2752686484 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23cc026040 con 0x5564fe515cf0 2022-05-13T23:15:59.356 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.306+0000 7f23e6a70b80 1 -- 172.21.15.2:0/2752686484 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5564fe4e65f0 con 0x5564fe515cf0 2022-05-13T23:15:59.356 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.306+0000 7f23abfff700 1 -- 172.21.15.2:0/2752686484 <== mon.0 v2:172.21.15.2:3300/0 2 ==== osd_map(85..85 src has 1..85) v4 ==== 8120+0+0 (secure 0 0 0) 0x7f23cc013760 con 0x5564fe515cf0 2022-05-13T23:15:59.356 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.306+0000 7f23abfff700 1 -- 172.21.15.2:0/2752686484 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7f23cc00e040 con 0x5564fe515cf0 2022-05-13T23:15:59.357 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.306+0000 7f23e6a70b80 10 client.15078 fetch_fsmap learned FSMap version 26 2022-05-13T23:15:59.357 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.306+0000 7f23e6a70b80 1 -- 172.21.15.2:0/2752686484 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) v3 -- 0x5564fe2a5fe0 con 0x5564fe515cf0 2022-05-13T23:15:59.357 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.307+0000 7f23abfff700 1 -- 172.21.15.2:0/2752686484 <== mon.0 v2:172.21.15.2:3300/0 4 ==== config(26 keys) v1 ==== 1087+0+0 (secure 0 0 0) 0x5564fe4e65f0 con 0x5564fe515cf0 2022-05-13T23:15:59.357 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.307+0000 7f23abfff700 1 -- 172.21.15.2:0/2752686484 <== mon.0 v2:172.21.15.2:3300/0 5 ==== fsmap(e 26) v1 ==== 7448+0+0 (secure 0 0 0) 0x7f23cc028600 con 0x5564fe515cf0 2022-05-13T23:15:59.358 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.307+0000 7f23e6a70b80 10 client.15078 fetch_fsmap finished waiting for FSMap version 26 2022-05-13T23:15:59.358 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.307+0000 7f23e6a70b80 10 client.15078 resolve_mds: resolved name 'f' to daemon mds.f 2022-05-13T23:15:59.358 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.307+0000 7f23e6a70b80 20 client.15078 populate_metadata read hostname 'smithi002' 2022-05-13T23:15:59.358 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.307+0000 7f23e6a70b80 1 --2- 172.21.15.2:0/2752686484 >> [v2:172.21.15.115:6834/71062276,v1:172.21.15.115:6835/71062276] conn(0x5564fe5814a0 0x5564fe57ddf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:15:59.358 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.307+0000 7f23e6a70b80 4 client.15078 mds_command: new command op to 24587 tid=0[{"prefix": "status"}] 2022-05-13T23:15:59.359 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.307+0000 7f23e6a70b80 1 -- 172.21.15.2:0/2752686484 --> [v2:172.21.15.115:6834/71062276,v1:172.21.15.115:6835/71062276] -- command(tid 0: {"prefix": "status"}) v1 -- 0x5564fe409790 con 0x5564fe5814a0 2022-05-13T23:15:59.359 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.307+0000 7f23d4d9a700 1 --2- 172.21.15.2:0/2752686484 >> [v2:172.21.15.115:6834/71062276,v1:172.21.15.115:6835/71062276] conn(0x5564fe5814a0 0x5564fe57ddf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:15:59.360 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.308+0000 7f23abfff700 10 client.15078 ms_handle_connect on v2:172.21.15.115:6834/71062276 2022-05-13T23:15:59.360 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.308+0000 7f23d4d9a700 1 --2- 172.21.15.2:0/2752686484 >> [v2:172.21.15.115:6834/71062276,v1:172.21.15.115:6835/71062276] conn(0x5564fe5814a0 0x5564fe57ddf0 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=8a59ed8b028f24a3 server_cookie=1be5ea2bb74a92fd in_seq=0 out_seq=0 2022-05-13T23:15:59.360 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.308+0000 7f23abfff700 1 -- 172.21.15.2:0/2752686484 <== mds.2 v2:172.21.15.115:6834/71062276 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+327 (crc 0 0 0) 0x5564fe409790 con 0x5564fe5814a0 2022-05-13T23:15:59.361 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.308+0000 7f23abfff700 10 client.15078 handle_command_reply: tid=0 2022-05-13T23:15:59.361 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.308+0000 7f23e6a70b80 1 client.15078 shutdown 2022-05-13T23:15:59.361 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.308+0000 7f23e6a70b80 1 -- 172.21.15.2:0/2752686484 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5564fe515cf0 msgr2=0x5564fe523cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:15:59.361 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.308+0000 7f23e6a70b80 1 --2- 172.21.15.2:0/2752686484 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5564fe515cf0 0x5564fe523cb0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f23cc004870 tx=0x7f23cc023040 comp rx=0 tx=0).stop 2022-05-13T23:15:59.361 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.308+0000 7f23e6a70b80 1 -- 172.21.15.2:0/2752686484 shutdown_connections 2022-05-13T23:15:59.362 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.308+0000 7f23e6a70b80 1 --2- 172.21.15.2:0/2752686484 >> [v2:172.21.15.115:6834/71062276,v1:172.21.15.115:6835/71062276] conn(0x5564fe5814a0 0x5564fe57ddf0 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:59.363 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.308+0000 7f23e6a70b80 1 --2- 172.21.15.2:0/2752686484 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x5564fe5175a0 0x5564fe57c350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:59.363 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.308+0000 7f23abfff700 0 client.15078 ms_handle_reset on v2:172.21.15.115:6834/71062276 2022-05-13T23:15:59.363 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.308+0000 7f23e6a70b80 1 --2- 172.21.15.2:0/2752686484 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5564fe515cf0 0x5564fe523cb0 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:59.363 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.308+0000 7f23e6a70b80 1 --2- 172.21.15.2:0/2752686484 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x5564fe516640 0x5564fe5241f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:59.364 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.308+0000 7f23e6a70b80 1 -- 172.21.15.2:0/2752686484 >> 172.21.15.2:0/2752686484 conn(0x5564fe510b10 msgr2=0x5564fe50f300 unknown :-1 s=STATE_NONE l=0).mark_down 2022-05-13T23:15:59.364 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.308+0000 7f23e6a70b80 1 -- 172.21.15.2:0/2752686484 shutdown_connections 2022-05-13T23:15:59.364 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.309+0000 7f23e6a70b80 1 -- 172.21.15.2:0/2752686484 wait complete. 2022-05-13T23:15:59.364 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.309+0000 7f23e6a70b80 20 client.15078 trim_cache size 0 max 16384 2022-05-13T23:15:59.365 INFO:teuthology.orchestra.run.smithi002.stdout:{ 2022-05-13T23:15:59.365 INFO:teuthology.orchestra.run.smithi002.stdout: "cluster_fsid": "16cf5f88-d311-11ec-be40-001a4aab830c", 2022-05-13T23:15:59.366 INFO:teuthology.orchestra.run.smithi002.stdout: "whoami": 2, 2022-05-13T23:15:59.366 INFO:teuthology.orchestra.run.smithi002.stdout: "id": 24587, 2022-05-13T23:15:59.366 INFO:teuthology.orchestra.run.smithi002.stdout: "want_state": "up:active", 2022-05-13T23:15:59.367 INFO:teuthology.orchestra.run.smithi002.stdout: "state": "up:active", 2022-05-13T23:15:59.367 INFO:teuthology.orchestra.run.smithi002.stdout: "fs_name": "cephfs", 2022-05-13T23:15:59.367 INFO:teuthology.orchestra.run.smithi002.stdout: "rank_uptime": 99.015891573999994, 2022-05-13T23:15:59.368 INFO:teuthology.orchestra.run.smithi002.stdout: "mdsmap_epoch": 26, 2022-05-13T23:15:59.368 INFO:teuthology.orchestra.run.smithi002.stdout: "osdmap_epoch": 85, 2022-05-13T23:15:59.368 INFO:teuthology.orchestra.run.smithi002.stdout: "osdmap_epoch_barrier": 83, 2022-05-13T23:15:59.368 INFO:teuthology.orchestra.run.smithi002.stdout: "uptime": 113.782089353 2022-05-13T23:15:59.369 INFO:teuthology.orchestra.run.smithi002.stdout:} 2022-05-13T23:15:59.370 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.309+0000 7f23d659d700 1 -- 172.21.15.2:0/4280401755 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f23bc096d60 msgr2=0x7f23bc099210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:15:59.370 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.309+0000 7f23d659d700 1 --2- 172.21.15.2:0/4280401755 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f23bc096d60 0x7f23bc099210 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f23c8004fd0 tx=0x7f23c8009040 comp rx=0 tx=0).stop 2022-05-13T23:15:59.371 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.309+0000 7f23d659d700 1 -- 172.21.15.2:0/4280401755 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f23d00fe640 msgr2=0x7f23d010ea60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:15:59.371 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.309+0000 7f23d659d700 1 --2- 172.21.15.2:0/4280401755 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f23d00fe640 0x7f23d010ea60 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f23c000e940 tx=0x7f23c000b040 comp rx=0 tx=0).stop 2022-05-13T23:15:59.371 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.310+0000 7f23d659d700 1 -- 172.21.15.2:0/4280401755 shutdown_connections 2022-05-13T23:15:59.371 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.310+0000 7f23d659d700 1 --2- 172.21.15.2:0/4280401755 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f23bc096d60 0x7f23bc099210 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:59.372 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.310+0000 7f23d659d700 1 --2- 172.21.15.2:0/4280401755 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f23d0101110 0x7f23d0115af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:59.372 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.310+0000 7f23d659d700 1 --2- 172.21.15.2:0/4280401755 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f23d0065630 0x7f23d010e520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:59.372 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.310+0000 7f23d659d700 1 --2- 172.21.15.2:0/4280401755 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f23d00fe640 0x7f23d010ea60 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:59.372 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.310+0000 7f23d659d700 1 -- 172.21.15.2:0/4280401755 >> 172.21.15.2:0/4280401755 conn(0x7f23d00f5b80 msgr2=0x7f23d0101e50 unknown :-1 s=STATE_NONE l=0).mark_down 2022-05-13T23:15:59.373 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.310+0000 7f23d659d700 1 -- 172.21.15.2:0/4280401755 shutdown_connections 2022-05-13T23:15:59.373 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.310+0000 7f23d659d700 1 -- 172.21.15.2:0/4280401755 wait complete. 2022-05-13T23:15:59.374 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.i status 2022-05-13T23:15:59.595 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.591+0000 7fdde7fff700 1 -- 172.21.15.2:0/2220181521 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fdde8070390 msgr2=0x7fdde806e890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:15:59.596 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.591+0000 7fdde7fff700 1 --2- 172.21.15.2:0/2220181521 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fdde8070390 0x7fdde806e890 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fddd8004870 tx=0x7fddd8009cb0 comp rx=0 tx=0).stop 2022-05-13T23:15:59.596 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.591+0000 7fdde7fff700 1 -- 172.21.15.2:0/2220181521 shutdown_connections 2022-05-13T23:15:59.596 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.591+0000 7fdde7fff700 1 --2- 172.21.15.2:0/2220181521 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fdde81033e0 0x7fdde81057c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:59.597 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.591+0000 7fdde7fff700 1 --2- 172.21.15.2:0/2220181521 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fdde8070390 0x7fdde806e890 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:59.597 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.591+0000 7fdde7fff700 1 --2- 172.21.15.2:0/2220181521 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fdde806edd0 0x7fdde806f240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:59.597 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.591+0000 7fdde7fff700 1 -- 172.21.15.2:0/2220181521 >> 172.21.15.2:0/2220181521 conn(0x7fdde80f6c60 msgr2=0x7fdde80f9090 unknown :-1 s=STATE_NONE l=0).mark_down 2022-05-13T23:15:59.597 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.591+0000 7fdde7fff700 1 -- 172.21.15.2:0/2220181521 shutdown_connections 2022-05-13T23:15:59.598 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.591+0000 7fdde7fff700 1 -- 172.21.15.2:0/2220181521 wait complete. 2022-05-13T23:15:59.598 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.592+0000 7fdde7fff700 1 Processor -- start 2022-05-13T23:15:59.598 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.592+0000 7fdde7fff700 1 -- start start 2022-05-13T23:15:59.598 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.593+0000 7fdde7fff700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fdde806edd0 0x7fdde810e4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:15:59.599 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.593+0000 7fdde7fff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fdde8070390 0x7fdde810e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:15:59.599 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.593+0000 7fdde7fff700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fdde81033e0 0x7fdde8115a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:15:59.599 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.593+0000 7fdde7fff700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdde8115fc0 con 0x7fdde8070390 2022-05-13T23:15:59.599 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.593+0000 7fdde7fff700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdde8116130 con 0x7fdde806edd0 2022-05-13T23:15:59.600 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.593+0000 7fdde7fff700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdde81162f0 con 0x7fdde81033e0 2022-05-13T23:15:59.600 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.593+0000 7fdde67fc700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fdde8070390 0x7fdde810e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:15:59.600 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.593+0000 7fdde67fc700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fdde8070390 0x7fdde810e9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:54248/0 (socket says 172.21.15.2:54248) 2022-05-13T23:15:59.600 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.593+0000 7fdde67fc700 1 -- 172.21.15.2:0/418770889 learned_addr learned my addr 172.21.15.2:0/418770889 (peer_addr_for_me v2:172.21.15.2:0/0) 2022-05-13T23:15:59.601 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.593+0000 7fdde6ffd700 1 --2- 172.21.15.2:0/418770889 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fdde806edd0 0x7fdde810e4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:15:59.601 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.593+0000 7fdde77fe700 1 --2- 172.21.15.2:0/418770889 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fdde81033e0 0x7fdde8115a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:15:59.601 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.593+0000 7fdde67fc700 1 -- 172.21.15.2:0/418770889 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fdde81033e0 msgr2=0x7fdde8115a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:15:59.602 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.593+0000 7fdde67fc700 1 --2- 172.21.15.2:0/418770889 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fdde81033e0 0x7fdde8115a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:59.602 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.593+0000 7fdde67fc700 1 -- 172.21.15.2:0/418770889 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fdde806edd0 msgr2=0x7fdde810e4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:15:59.602 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.593+0000 7fdde67fc700 1 --2- 172.21.15.2:0/418770889 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fdde806edd0 0x7fdde810e4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:59.603 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.593+0000 7fdde67fc700 1 -- 172.21.15.2:0/418770889 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fddd8005040 con 0x7fdde8070390 2022-05-13T23:15:59.603 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.593+0000 7fdde6ffd700 1 --2- 172.21.15.2:0/418770889 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fdde806edd0 0x7fdde810e4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-05-13T23:15:59.603 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.594+0000 7fdde67fc700 1 --2- 172.21.15.2:0/418770889 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fdde8070390 0x7fdde810e9f0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fdddc00e970 tx=0x7fdddc00b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-05-13T23:15:59.603 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.594+0000 7fddcf7fe700 1 -- 172.21.15.2:0/418770889 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdddc007b40 con 0x7fdde8070390 2022-05-13T23:15:59.604 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.594+0000 7fdde7fff700 1 -- 172.21.15.2:0/418770889 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdde81165d0 con 0x7fdde8070390 2022-05-13T23:15:59.604 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.594+0000 7fddcf7fe700 1 -- 172.21.15.2:0/418770889 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1087+0+0 (secure 0 0 0) 0x7fdddc00f040 con 0x7fdde8070390 2022-05-13T23:15:59.604 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.594+0000 7fddcf7fe700 1 -- 172.21.15.2:0/418770889 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdddc014950 con 0x7fdde8070390 2022-05-13T23:15:59.604 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.594+0000 7fdde7fff700 1 -- 172.21.15.2:0/418770889 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdde8116b90 con 0x7fdde8070390 2022-05-13T23:15:59.605 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.597+0000 7fddcf7fe700 1 -- 172.21.15.2:0/418770889 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 16) v1 ==== 125136+0+0 (secure 0 0 0) 0x7fdddc01b030 con 0x7fdde8070390 2022-05-13T23:15:59.606 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.599+0000 7fddfc685b80 1 Processor -- start 2022-05-13T23:15:59.617 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.599+0000 7fddfc685b80 1 -- start start 2022-05-13T23:15:59.617 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.599+0000 7fddcf7fe700 1 --2- 172.21.15.2:0/418770889 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7fddd0096d60 0x7fddd0099210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:15:59.618 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.599+0000 7fddfc685b80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55a12f677500 0x55a12f677910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:15:59.618 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.599+0000 7fddcf7fe700 1 -- 172.21.15.2:0/418770889 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8120+0+0 (secure 0 0 0) 0x7fdddc0bead0 con 0x7fdde8070390 2022-05-13T23:15:59.618 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.599+0000 7fddfc685b80 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55a12f677e50 0x55a12f678790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:15:59.618 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.599+0000 7fddfc685b80 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a12f678db0 0x55a12f679200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:15:59.619 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.599+0000 7fddfc685b80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a12f5693a0 con 0x55a12f677500 2022-05-13T23:15:59.619 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.599+0000 7fddfc685b80 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a12f53e400 con 0x55a12f677e50 2022-05-13T23:15:59.619 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.599+0000 7fddfc685b80 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a12f5c26c0 con 0x55a12f678db0 2022-05-13T23:15:59.619 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.599+0000 7fdde77fe700 1 --2- 172.21.15.2:0/418770889 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7fddd0096d60 0x7fddd0099210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:15:59.619 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.599+0000 7fdde6ffd700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55a12f677500 0x55a12f677910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:15:59.620 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.599+0000 7fdde67fc700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55a12f677e50 0x55a12f678790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:15:59.620 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.600+0000 7fdde6ffd700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55a12f677500 0x55a12f677910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:54250/0 (socket says 172.21.15.2:54250) 2022-05-13T23:15:59.620 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.600+0000 7fdde6ffd700 1 -- 172.21.15.2:0/2473804246 learned_addr learned my addr 172.21.15.2:0/2473804246 (peer_addr_for_me v2:172.21.15.2:0/0) 2022-05-13T23:15:59.621 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.600+0000 7fdde77fe700 1 --2- 172.21.15.2:0/2473804246 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a12f678db0 0x55a12f679200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:15:59.621 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.600+0000 7fdde67fc700 1 -- 172.21.15.2:0/2473804246 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a12f678db0 msgr2=0x55a12f679200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-05-13T23:15:59.621 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.600+0000 7fdde67fc700 1 --2- 172.21.15.2:0/2473804246 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a12f678db0 0x55a12f679200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:59.621 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.600+0000 7fdde67fc700 1 -- 172.21.15.2:0/2473804246 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55a12f677500 msgr2=0x55a12f677910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-05-13T23:15:59.622 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.600+0000 7fdde67fc700 1 --2- 172.21.15.2:0/2473804246 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55a12f677500 0x55a12f677910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:59.622 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.600+0000 7fdde67fc700 1 -- 172.21.15.2:0/2473804246 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a12f454360 con 0x55a12f677e50 2022-05-13T23:15:59.622 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.600+0000 7fdde6ffd700 1 --2- 172.21.15.2:0/2473804246 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55a12f677500 0x55a12f677910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-05-13T23:15:59.622 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.600+0000 7fdde77fe700 1 --2- 172.21.15.2:0/418770889 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7fddd0096d60 0x7fddd0099210 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fdde0009fd0 tx=0x7fdde000b040 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-05-13T23:15:59.622 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.600+0000 7fdde67fc700 1 --2- 172.21.15.2:0/2473804246 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55a12f677e50 0x55a12f678790 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fdddc009cf0 tx=0x7fdddc004730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=98bcdb75cae73a53 server_cookie=0 in_seq=0 out_seq=0 2022-05-13T23:15:59.623 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.600+0000 7fddcc9fe700 1 -- 172.21.15.2:0/2473804246 <== mon.1 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdddc08f3a0 con 0x55a12f677e50 2022-05-13T23:15:59.624 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.601+0000 7fddcc9fe700 1 -- 172.21.15.2:0/2473804246 <== mon.1 v2:172.21.15.47:3300/0 2 ==== config(26 keys) v1 ==== 1087+0+0 (secure 0 0 0) 0x7fdddc090140 con 0x55a12f677e50 2022-05-13T23:15:59.624 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.601+0000 7fddcc9fe700 1 -- 172.21.15.2:0/2473804246 <== mon.1 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdddc08ff10 con 0x55a12f677e50 2022-05-13T23:15:59.624 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.601+0000 7fddfc685b80 1 -- 172.21.15.2:0/2473804246 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55a12f677e50 msgr2=0x55a12f678790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:15:59.624 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.601+0000 7fddfc685b80 1 --2- 172.21.15.2:0/2473804246 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55a12f677e50 0x55a12f678790 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fdddc009cf0 tx=0x7fdddc004730 comp rx=0 tx=0).stop 2022-05-13T23:15:59.624 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.601+0000 7fddfc685b80 1 -- 172.21.15.2:0/2473804246 shutdown_connections 2022-05-13T23:15:59.625 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.601+0000 7fddfc685b80 1 --2- 172.21.15.2:0/2473804246 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a12f678db0 0x55a12f679200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:59.625 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.601+0000 7fddfc685b80 1 --2- 172.21.15.2:0/2473804246 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55a12f677500 0x55a12f677910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:59.625 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.601+0000 7fddfc685b80 1 --2- 172.21.15.2:0/2473804246 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55a12f677e50 0x55a12f678790 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:59.625 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.601+0000 7fddfc685b80 1 -- 172.21.15.2:0/2473804246 >> 172.21.15.2:0/2473804246 conn(0x55a12f669b10 msgr2=0x55a12f669f00 unknown :-1 s=STATE_NONE l=0).mark_down 2022-05-13T23:15:59.626 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.601+0000 7fddfc685b80 1 -- 172.21.15.2:0/2473804246 shutdown_connections 2022-05-13T23:15:59.626 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.601+0000 7fddfc685b80 1 -- 172.21.15.2:0/2473804246 wait complete. 2022-05-13T23:15:59.626 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.602+0000 7fddfc685b80 1 -- start start 2022-05-13T23:15:59.626 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.602+0000 7fddfc685b80 1 Processor -- start 2022-05-13T23:15:59.626 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.603+0000 7fddfc685b80 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55a12f677500 0x55a12f680cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:15:59.627 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.603+0000 7fddfc685b80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55a12f677e50 0x55a12f681200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:15:59.628 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.603+0000 7fddfc685b80 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a12f678db0 0x55a12f688260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:15:59.628 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.603+0000 7fddfc685b80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a12f5bc7a0 con 0x55a12f677e50 2022-05-13T23:15:59.628 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.603+0000 7fddfc685b80 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a12f456650 con 0x55a12f677500 2022-05-13T23:15:59.629 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.603+0000 7fddfc685b80 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a12f688890 con 0x55a12f678db0 2022-05-13T23:15:59.629 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.603+0000 7fddfc685b80 10 client.0 _ll_register_callbacks cb 0x55a12f630de0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2022-05-13T23:15:59.630 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.603+0000 7fdde67fc700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55a12f677e50 0x55a12f681200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:15:59.630 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.603+0000 7fdde67fc700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55a12f677e50 0x55a12f681200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:54262/0 (socket says 172.21.15.2:54262) 2022-05-13T23:15:59.631 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.603+0000 7fdde77fe700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a12f678db0 0x55a12f688260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:15:59.632 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.603+0000 7fdde6ffd700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55a12f677500 0x55a12f680cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:15:59.632 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.603+0000 7fdde67fc700 1 -- 172.21.15.2:0/3796314569 learned_addr learned my addr 172.21.15.2:0/3796314569 (peer_addr_for_me v2:172.21.15.2:0/0) 2022-05-13T23:15:59.632 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.603+0000 7fdde67fc700 1 -- 172.21.15.2:0/3796314569 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a12f678db0 msgr2=0x55a12f688260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-05-13T23:15:59.633 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.603+0000 7fdde67fc700 1 --2- 172.21.15.2:0/3796314569 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a12f678db0 0x55a12f688260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:59.633 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.603+0000 7fdde67fc700 1 -- 172.21.15.2:0/3796314569 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55a12f677500 msgr2=0x55a12f680cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-05-13T23:15:59.634 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.603+0000 7fdde67fc700 1 --2- 172.21.15.2:0/3796314569 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55a12f677500 0x55a12f680cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:59.635 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.603+0000 7fdde67fc700 1 -- 172.21.15.2:0/3796314569 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fddd8017040 con 0x55a12f677e50 2022-05-13T23:15:59.635 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.604+0000 7fdde67fc700 1 --2- 172.21.15.2:0/3796314569 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55a12f677e50 0x55a12f681200 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fdddc08f9d0 tx=0x7fdddc002040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3b78b94ef7b9be23 server_cookie=0 in_seq=0 out_seq=0 2022-05-13T23:15:59.635 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.604+0000 7fddc5ffb700 10 client.0 ms_handle_connect on v2:172.21.15.2:3300/0 2022-05-13T23:15:59.636 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.604+0000 7fddc5ffb700 1 -- 172.21.15.2:0/3796314569 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdddc08fa40 con 0x55a12f677e50 2022-05-13T23:15:59.636 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.604+0000 7fddfc685b80 1 -- 172.21.15.2:0/3796314569 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55a12f5709a0 con 0x55a12f677e50 2022-05-13T23:15:59.636 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.604+0000 7fddc5ffb700 1 -- 172.21.15.2:0/3796314569 <== mon.0 v2:172.21.15.2:3300/0 2 ==== osd_map(85..85 src has 1..85) v4 ==== 8120+0+0 (secure 0 0 0) 0x7fdddc096130 con 0x55a12f677e50 2022-05-13T23:15:59.637 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.604+0000 7fddc5ffb700 1 -- 172.21.15.2:0/3796314569 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7fdddc090140 con 0x55a12f677e50 2022-05-13T23:15:59.637 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.604+0000 7fddfc685b80 10 client.15090 fetch_fsmap learned FSMap version 26 2022-05-13T23:15:59.637 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.604+0000 7fddfc685b80 1 -- 172.21.15.2:0/3796314569 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) v3 -- 0x7fdde8116b90 con 0x55a12f677e50 2022-05-13T23:15:59.638 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.605+0000 7fddc5ffb700 1 -- 172.21.15.2:0/3796314569 <== mon.0 v2:172.21.15.2:3300/0 4 ==== config(26 keys) v1 ==== 1087+0+0 (secure 0 0 0) 0x55a12f5709a0 con 0x55a12f677e50 2022-05-13T23:15:59.638 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.605+0000 7fddc5ffb700 1 -- 172.21.15.2:0/3796314569 <== mon.0 v2:172.21.15.2:3300/0 5 ==== fsmap(e 26) v1 ==== 7448+0+0 (secure 0 0 0) 0x7fdddc08e9a0 con 0x55a12f677e50 2022-05-13T23:15:59.639 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.605+0000 7fddfc685b80 10 client.15090 fetch_fsmap finished waiting for FSMap version 26 2022-05-13T23:15:59.639 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.605+0000 7fddfc685b80 10 client.15090 resolve_mds: resolved name 'i' to daemon mds.i 2022-05-13T23:15:59.639 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.605+0000 7fddfc685b80 20 client.15090 populate_metadata read hostname 'smithi002' 2022-05-13T23:15:59.639 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.605+0000 7fddfc685b80 1 --2- 172.21.15.2:0/3796314569 >> [v2:172.21.15.115:6836/2940957881,v1:172.21.15.115:6837/2940957881] conn(0x55a12f68b710 0x55a12f6909d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:15:59.640 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.605+0000 7fddfc685b80 4 client.15090 mds_command: new command op to 24599 tid=0[{"prefix": "get_command_descriptions"}] 2022-05-13T23:15:59.640 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.605+0000 7fddfc685b80 1 -- 172.21.15.2:0/3796314569 --> [v2:172.21.15.115:6836/2940957881,v1:172.21.15.115:6837/2940957881] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55a12f591330 con 0x55a12f68b710 2022-05-13T23:15:59.640 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.605+0000 7fdde6ffd700 1 --2- 172.21.15.2:0/3796314569 >> [v2:172.21.15.115:6836/2940957881,v1:172.21.15.115:6837/2940957881] conn(0x55a12f68b710 0x55a12f6909d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:15:59.640 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.606+0000 7fdde6ffd700 1 --2- 172.21.15.2:0/3796314569 >> [v2:172.21.15.115:6836/2940957881,v1:172.21.15.115:6837/2940957881] conn(0x55a12f68b710 0x55a12f6909d0 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=9fd9125a492713d server_cookie=37c116826c5605b6 in_seq=0 out_seq=0 2022-05-13T23:15:59.641 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.606+0000 7fddc5ffb700 10 client.15090 ms_handle_connect on v2:172.21.15.115:6836/2940957881 2022-05-13T23:15:59.641 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.607+0000 7fddc5ffb700 1 -- 172.21.15.2:0/3796314569 <== mds.1 v2:172.21.15.115:6836/2940957881 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17090 (crc 0 0 0) 0x55a12f591330 con 0x55a12f68b710 2022-05-13T23:15:59.641 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.607+0000 7fddc5ffb700 10 client.15090 handle_command_reply: tid=0 2022-05-13T23:15:59.641 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.607+0000 7fddfc685b80 1 client.15090 shutdown 2022-05-13T23:15:59.642 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.607+0000 7fddfc685b80 1 -- 172.21.15.2:0/3796314569 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55a12f677e50 msgr2=0x55a12f681200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:15:59.642 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.607+0000 7fddfc685b80 1 --2- 172.21.15.2:0/3796314569 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55a12f677e50 0x55a12f681200 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fdddc08f9d0 tx=0x7fdddc002040 comp rx=0 tx=0).stop 2022-05-13T23:15:59.642 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.607+0000 7fddfc685b80 1 -- 172.21.15.2:0/3796314569 shutdown_connections 2022-05-13T23:15:59.642 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.607+0000 7fddfc685b80 1 --2- 172.21.15.2:0/3796314569 >> [v2:172.21.15.115:6836/2940957881,v1:172.21.15.115:6837/2940957881] conn(0x55a12f68b710 0x55a12f6909d0 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:59.643 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.607+0000 7fddfc685b80 1 --2- 172.21.15.2:0/3796314569 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a12f678db0 0x55a12f688260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:59.643 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.607+0000 7fddfc685b80 1 --2- 172.21.15.2:0/3796314569 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55a12f677500 0x55a12f680cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:59.643 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.607+0000 7fddc5ffb700 0 client.15090 ms_handle_reset on v2:172.21.15.115:6836/2940957881 2022-05-13T23:15:59.653 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.607+0000 7fddfc685b80 1 --2- 172.21.15.2:0/3796314569 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55a12f677e50 0x55a12f681200 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:59.653 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.607+0000 7fddfc685b80 1 -- 172.21.15.2:0/3796314569 >> 172.21.15.2:0/3796314569 conn(0x55a12f669b10 msgr2=0x55a12f668d80 unknown :-1 s=STATE_NONE l=0).mark_down 2022-05-13T23:15:59.654 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.607+0000 7fddfc685b80 1 -- 172.21.15.2:0/3796314569 shutdown_connections 2022-05-13T23:15:59.654 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.607+0000 7fddfc685b80 1 -- 172.21.15.2:0/3796314569 wait complete. 2022-05-13T23:15:59.654 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.607+0000 7fddfc685b80 20 client.15090 trim_cache size 0 max 16384 2022-05-13T23:15:59.663 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.625+0000 7fddfc685b80 1 Processor -- start 2022-05-13T23:15:59.664 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.625+0000 7fddfc685b80 1 -- start start 2022-05-13T23:15:59.664 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.626+0000 7fddfc685b80 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a12f677500 0x55a12f669ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:15:59.664 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.626+0000 7fddfc685b80 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55a12f677e50 0x55a12f6949f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:15:59.665 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.626+0000 7fddfc685b80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55a12f678db0 0x55a12f680470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:15:59.665 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.626+0000 7fddfc685b80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a12f68ae90 con 0x55a12f678db0 2022-05-13T23:15:59.665 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.626+0000 7fddfc685b80 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a12f66a780 con 0x55a12f677e50 2022-05-13T23:15:59.666 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.626+0000 7fddfc685b80 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a12f6802a0 con 0x55a12f677500 2022-05-13T23:15:59.666 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.626+0000 7fdde77fe700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55a12f678db0 0x55a12f680470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:15:59.667 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.626+0000 7fdde6ffd700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a12f677500 0x55a12f669ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:15:59.667 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.626+0000 7fdde6ffd700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a12f677500 0x55a12f669ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.2:45622/0 (socket says 172.21.15.2:45622) 2022-05-13T23:15:59.667 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.626+0000 7fdde6ffd700 1 -- 172.21.15.2:0/1661521952 learned_addr learned my addr 172.21.15.2:0/1661521952 (peer_addr_for_me v2:172.21.15.2:0/0) 2022-05-13T23:15:59.668 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.626+0000 7fdde67fc700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55a12f677e50 0x55a12f6949f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:15:59.668 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.626+0000 7fdde77fe700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55a12f678db0 0x55a12f680470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:54274/0 (socket says 172.21.15.2:54274) 2022-05-13T23:15:59.668 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.626+0000 7fdde6ffd700 1 -- 172.21.15.2:0/1661521952 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55a12f677e50 msgr2=0x55a12f6949f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-05-13T23:15:59.668 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.626+0000 7fdde6ffd700 1 --2- 172.21.15.2:0/1661521952 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55a12f677e50 0x55a12f6949f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:59.669 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.627+0000 7fdde6ffd700 1 -- 172.21.15.2:0/1661521952 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55a12f678db0 msgr2=0x55a12f680470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-05-13T23:15:59.669 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.627+0000 7fdde6ffd700 1 --2- 172.21.15.2:0/1661521952 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55a12f678db0 0x55a12f680470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:59.669 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.627+0000 7fdde6ffd700 1 -- 172.21.15.2:0/1661521952 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a12f3fefe0 con 0x55a12f677500 2022-05-13T23:15:59.670 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.627+0000 7fdde6ffd700 1 --2- 172.21.15.2:0/1661521952 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a12f677500 0x55a12f669ee0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fddd8013490 tx=0x7fddd8009cb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=757697a3a3e2407e server_cookie=0 in_seq=0 out_seq=0 2022-05-13T23:15:59.670 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.627+0000 7fddcd1ff700 1 -- 172.21.15.2:0/1661521952 <== mon.2 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddd8024070 con 0x55a12f677500 2022-05-13T23:15:59.670 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.627+0000 7fddcd1ff700 1 -- 172.21.15.2:0/1661521952 <== mon.2 v2:172.21.15.115:3300/0 2 ==== config(26 keys) v1 ==== 1087+0+0 (secure 0 0 0) 0x7fddd8000c90 con 0x55a12f677500 2022-05-13T23:15:59.670 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.627+0000 7fddcd1ff700 1 -- 172.21.15.2:0/1661521952 <== mon.2 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddd800e040 con 0x55a12f677500 2022-05-13T23:15:59.671 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.628+0000 7fddfc685b80 1 -- 172.21.15.2:0/1661521952 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a12f677500 msgr2=0x55a12f669ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:15:59.671 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.628+0000 7fddfc685b80 1 --2- 172.21.15.2:0/1661521952 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a12f677500 0x55a12f669ee0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fddd8013490 tx=0x7fddd8009cb0 comp rx=0 tx=0).stop 2022-05-13T23:15:59.671 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.628+0000 7fddfc685b80 1 -- 172.21.15.2:0/1661521952 shutdown_connections 2022-05-13T23:15:59.671 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.628+0000 7fddfc685b80 1 --2- 172.21.15.2:0/1661521952 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55a12f678db0 0x55a12f680470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:59.671 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.628+0000 7fddfc685b80 1 --2- 172.21.15.2:0/1661521952 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a12f677500 0x55a12f669ee0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:59.672 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.628+0000 7fddfc685b80 1 --2- 172.21.15.2:0/1661521952 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55a12f677e50 0x55a12f6949f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:59.672 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.628+0000 7fddfc685b80 1 -- 172.21.15.2:0/1661521952 >> 172.21.15.2:0/1661521952 conn(0x55a12f669b10 msgr2=0x55a12f668d80 unknown :-1 s=STATE_NONE l=0).mark_down 2022-05-13T23:15:59.672 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.628+0000 7fddfc685b80 1 -- 172.21.15.2:0/1661521952 shutdown_connections 2022-05-13T23:15:59.672 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.628+0000 7fddfc685b80 1 -- 172.21.15.2:0/1661521952 wait complete. 2022-05-13T23:15:59.673 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.628+0000 7fddfc685b80 1 -- start start 2022-05-13T23:15:59.673 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.628+0000 7fddfc685b80 1 Processor -- start 2022-05-13T23:15:59.673 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.629+0000 7fddfc685b80 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a12f677500 0x55a12f685470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:15:59.673 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.629+0000 7fddfc685b80 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55a12f677e50 0x55a12f6859b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:15:59.673 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.629+0000 7fddfc685b80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55a12f678db0 0x55a12f6ddb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:15:59.674 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.629+0000 7fddfc685b80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a12f6de0d0 con 0x55a12f678db0 2022-05-13T23:15:59.674 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.629+0000 7fddfc685b80 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a12f6de240 con 0x55a12f677e50 2022-05-13T23:15:59.674 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.629+0000 7fddfc685b80 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a12f6de400 con 0x55a12f677500 2022-05-13T23:15:59.674 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.629+0000 7fddfc685b80 10 client.0 _ll_register_callbacks cb 0x55a12f630de0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2022-05-13T23:15:59.674 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.629+0000 7fdde6ffd700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a12f677500 0x55a12f685470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:15:59.675 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.629+0000 7fdde77fe700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55a12f678db0 0x55a12f6ddb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:15:59.675 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.629+0000 7fdde6ffd700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a12f677500 0x55a12f685470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.2:45626/0 (socket says 172.21.15.2:45626) 2022-05-13T23:15:59.675 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.629+0000 7fdde6ffd700 1 -- 172.21.15.2:0/1161790168 learned_addr learned my addr 172.21.15.2:0/1161790168 (peer_addr_for_me v2:172.21.15.2:0/0) 2022-05-13T23:15:59.675 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.629+0000 7fdde67fc700 1 --2- 172.21.15.2:0/1161790168 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55a12f677e50 0x55a12f6859b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:15:59.676 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.630+0000 7fdde6ffd700 1 -- 172.21.15.2:0/1161790168 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55a12f677e50 msgr2=0x55a12f6859b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-05-13T23:15:59.676 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.630+0000 7fdde6ffd700 1 --2- 172.21.15.2:0/1161790168 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55a12f677e50 0x55a12f6859b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:59.677 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.630+0000 7fdde6ffd700 1 -- 172.21.15.2:0/1161790168 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55a12f678db0 msgr2=0x55a12f6ddb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-05-13T23:15:59.677 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.630+0000 7fdde6ffd700 1 --2- 172.21.15.2:0/1161790168 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55a12f678db0 0x55a12f6ddb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:59.677 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.630+0000 7fdde6ffd700 1 -- 172.21.15.2:0/1161790168 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdde8106430 con 0x55a12f677500 2022-05-13T23:15:59.677 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.630+0000 7fdde6ffd700 1 --2- 172.21.15.2:0/1161790168 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a12f677500 0x55a12f685470 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fddd8004ee0 tx=0x7fddd8009cb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=11661130424d1d05 server_cookie=0 in_seq=0 out_seq=0 2022-05-13T23:15:59.678 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.630+0000 7fddc5ffb700 10 client.0 ms_handle_connect on v2:172.21.15.115:3300/0 2022-05-13T23:15:59.678 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.630+0000 7fddc5ffb700 1 -- 172.21.15.2:0/1161790168 <== mon.2 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddd800e040 con 0x55a12f677500 2022-05-13T23:15:59.678 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.630+0000 7fddfc685b80 1 -- 172.21.15.2:0/1161790168 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55a12f63f5f0 con 0x55a12f677500 2022-05-13T23:15:59.678 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.630+0000 7fddc5ffb700 1 -- 172.21.15.2:0/1161790168 <== mon.2 v2:172.21.15.115:3300/0 2 ==== osd_map(85..85 src has 1..85) v4 ==== 8120+0+0 (secure 0 0 0) 0x7fddd8013dd0 con 0x55a12f677500 2022-05-13T23:15:59.678 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.631+0000 7fddc5ffb700 1 -- 172.21.15.2:0/1161790168 <== mon.2 v2:172.21.15.115:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7fddd8024040 con 0x55a12f677500 2022-05-13T23:15:59.679 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.631+0000 7fddfc685b80 10 client.25217 fetch_fsmap learned FSMap version 26 2022-05-13T23:15:59.679 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.631+0000 7fddfc685b80 1 -- 172.21.15.2:0/1161790168 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) v3 -- 0x55a12f426f80 con 0x55a12f677500 2022-05-13T23:15:59.679 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.631+0000 7fddc5ffb700 1 -- 172.21.15.2:0/1161790168 <== mon.2 v2:172.21.15.115:3300/0 4 ==== config(26 keys) v1 ==== 1087+0+0 (secure 0 0 0) 0x55a12f63f5f0 con 0x55a12f677500 2022-05-13T23:15:59.679 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.631+0000 7fddc5ffb700 1 -- 172.21.15.2:0/1161790168 <== mon.2 v2:172.21.15.115:3300/0 5 ==== fsmap(e 26) v1 ==== 7448+0+0 (secure 0 0 0) 0x7fddd8037070 con 0x55a12f677500 2022-05-13T23:15:59.679 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.631+0000 7fddfc685b80 10 client.25217 fetch_fsmap finished waiting for FSMap version 26 2022-05-13T23:15:59.680 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.631+0000 7fddfc685b80 10 client.25217 resolve_mds: resolved name 'i' to daemon mds.i 2022-05-13T23:15:59.680 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.631+0000 7fddfc685b80 20 client.25217 populate_metadata read hostname 'smithi002' 2022-05-13T23:15:59.680 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.631+0000 7fddfc685b80 1 --2- 172.21.15.2:0/1161790168 >> [v2:172.21.15.115:6836/2940957881,v1:172.21.15.115:6837/2940957881] conn(0x55a12f6e2a60 0x55a12f6df3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:15:59.680 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.631+0000 7fddfc685b80 4 client.25217 mds_command: new command op to 24599 tid=0[{"prefix": "status"}] 2022-05-13T23:15:59.681 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.631+0000 7fddfc685b80 1 -- 172.21.15.2:0/1161790168 --> [v2:172.21.15.115:6836/2940957881,v1:172.21.15.115:6837/2940957881] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55a12f562790 con 0x55a12f6e2a60 2022-05-13T23:15:59.681 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.632+0000 7fdde67fc700 1 --2- 172.21.15.2:0/1161790168 >> [v2:172.21.15.115:6836/2940957881,v1:172.21.15.115:6837/2940957881] conn(0x55a12f6e2a60 0x55a12f6df3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:15:59.681 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.632+0000 7fdde67fc700 1 --2- 172.21.15.2:0/1161790168 >> [v2:172.21.15.115:6836/2940957881,v1:172.21.15.115:6837/2940957881] conn(0x55a12f6e2a60 0x55a12f6df3b0 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=69f449942245b5b5 server_cookie=4fedc1afc4d86576 in_seq=0 out_seq=0 2022-05-13T23:15:59.681 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.632+0000 7fddc5ffb700 10 client.25217 ms_handle_connect on v2:172.21.15.115:6836/2940957881 2022-05-13T23:15:59.681 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.632+0000 7fddc5ffb700 1 -- 172.21.15.2:0/1161790168 <== mds.1 v2:172.21.15.115:6836/2940957881 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+327 (crc 0 0 0) 0x55a12f562790 con 0x55a12f6e2a60 2022-05-13T23:15:59.682 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.632+0000 7fddc5ffb700 10 client.25217 handle_command_reply: tid=0 2022-05-13T23:15:59.682 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.632+0000 7fddfc685b80 1 client.25217 shutdown 2022-05-13T23:15:59.682 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.632+0000 7fddfc685b80 1 -- 172.21.15.2:0/1161790168 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a12f677500 msgr2=0x55a12f685470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:15:59.682 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.632+0000 7fddfc685b80 1 --2- 172.21.15.2:0/1161790168 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a12f677500 0x55a12f685470 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fddd8004ee0 tx=0x7fddd8009cb0 comp rx=0 tx=0).stop 2022-05-13T23:15:59.683 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.633+0000 7fddfc685b80 1 -- 172.21.15.2:0/1161790168 shutdown_connections 2022-05-13T23:15:59.683 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.633+0000 7fddfc685b80 1 --2- 172.21.15.2:0/1161790168 >> [v2:172.21.15.115:6836/2940957881,v1:172.21.15.115:6837/2940957881] conn(0x55a12f6e2a60 0x55a12f6df3b0 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:59.683 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.633+0000 7fddfc685b80 1 --2- 172.21.15.2:0/1161790168 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55a12f678db0 0x55a12f6ddb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:59.683 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.633+0000 7fddfc685b80 1 --2- 172.21.15.2:0/1161790168 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x55a12f677500 0x55a12f685470 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:59.683 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.633+0000 7fddfc685b80 1 --2- 172.21.15.2:0/1161790168 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55a12f677e50 0x55a12f6859b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:59.684 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.633+0000 7fddc5ffb700 0 client.25217 ms_handle_reset on v2:172.21.15.115:6836/2940957881 2022-05-13T23:15:59.684 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.633+0000 7fddfc685b80 1 -- 172.21.15.2:0/1161790168 >> 172.21.15.2:0/1161790168 conn(0x55a12f669b10 msgr2=0x55a12f6949f0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-05-13T23:15:59.684 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.633+0000 7fddfc685b80 1 -- 172.21.15.2:0/1161790168 shutdown_connections 2022-05-13T23:15:59.684 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.633+0000 7fddfc685b80 1 -- 172.21.15.2:0/1161790168 wait complete. 2022-05-13T23:15:59.685 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.633+0000 7fddfc685b80 20 client.25217 trim_cache size 0 max 16384 2022-05-13T23:15:59.685 INFO:teuthology.orchestra.run.smithi002.stdout:{ 2022-05-13T23:15:59.685 INFO:teuthology.orchestra.run.smithi002.stdout: "cluster_fsid": "16cf5f88-d311-11ec-be40-001a4aab830c", 2022-05-13T23:15:59.686 INFO:teuthology.orchestra.run.smithi002.stdout: "whoami": 1, 2022-05-13T23:15:59.686 INFO:teuthology.orchestra.run.smithi002.stdout: "id": 24599, 2022-05-13T23:15:59.686 INFO:teuthology.orchestra.run.smithi002.stdout: "want_state": "up:active", 2022-05-13T23:15:59.686 INFO:teuthology.orchestra.run.smithi002.stdout: "state": "up:active", 2022-05-13T23:15:59.687 INFO:teuthology.orchestra.run.smithi002.stdout: "fs_name": "cephfs", 2022-05-13T23:15:59.687 INFO:teuthology.orchestra.run.smithi002.stdout: "rank_uptime": 100.35159171399999, 2022-05-13T23:15:59.688 INFO:teuthology.orchestra.run.smithi002.stdout: "mdsmap_epoch": 26, 2022-05-13T23:15:59.688 INFO:teuthology.orchestra.run.smithi002.stdout: "osdmap_epoch": 85, 2022-05-13T23:15:59.688 INFO:teuthology.orchestra.run.smithi002.stdout: "osdmap_epoch_barrier": 83, 2022-05-13T23:15:59.688 INFO:teuthology.orchestra.run.smithi002.stdout: "uptime": 111.781748507 2022-05-13T23:15:59.689 INFO:teuthology.orchestra.run.smithi002.stdout:} 2022-05-13T23:15:59.691 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.633+0000 7fdde7fff700 1 -- 172.21.15.2:0/418770889 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7fddd0096d60 msgr2=0x7fddd0099210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:15:59.691 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.633+0000 7fdde7fff700 1 --2- 172.21.15.2:0/418770889 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7fddd0096d60 0x7fddd0099210 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fdde0009fd0 tx=0x7fdde000b040 comp rx=0 tx=0).stop 2022-05-13T23:15:59.691 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.633+0000 7fdde7fff700 1 -- 172.21.15.2:0/418770889 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fdde8070390 msgr2=0x7fdde810e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:15:59.691 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.633+0000 7fdde7fff700 1 --2- 172.21.15.2:0/418770889 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fdde8070390 0x7fdde810e9f0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fdddc00e970 tx=0x7fdddc00b040 comp rx=0 tx=0).stop 2022-05-13T23:15:59.692 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.634+0000 7fdde7fff700 1 -- 172.21.15.2:0/418770889 shutdown_connections 2022-05-13T23:15:59.692 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.634+0000 7fdde7fff700 1 --2- 172.21.15.2:0/418770889 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fdde81033e0 0x7fdde8115a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:59.692 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.634+0000 7fdde7fff700 1 --2- 172.21.15.2:0/418770889 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fdde806edd0 0x7fdde810e4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:59.692 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.634+0000 7fdde7fff700 1 --2- 172.21.15.2:0/418770889 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7fddd0096d60 0x7fddd0099210 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:59.693 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.634+0000 7fdde7fff700 1 --2- 172.21.15.2:0/418770889 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fdde8070390 0x7fdde810e9f0 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:59.693 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.634+0000 7fdde7fff700 1 -- 172.21.15.2:0/418770889 >> 172.21.15.2:0/418770889 conn(0x7fdde80f6c60 msgr2=0x7fdde80f9040 unknown :-1 s=STATE_NONE l=0).mark_down 2022-05-13T23:15:59.693 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.634+0000 7fdde7fff700 1 -- 172.21.15.2:0/418770889 shutdown_connections 2022-05-13T23:15:59.693 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.634+0000 7fdde7fff700 1 -- 172.21.15.2:0/418770889 wait complete. 2022-05-13T23:15:59.694 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:15:59 smithi002 ceph-mon[71816]: from='client.? 172.21.15.2:0/1257682873' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2022-05-13T23:15:59.695 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.l status 2022-05-13T23:15:59.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:15:59 smithi047 ceph-mon[77147]: from='client.? 172.21.15.2:0/1257682873' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2022-05-13T23:15:59.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:15:59 smithi115 ceph-mon[78593]: from='client.? 172.21.15.2:0/1257682873' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2022-05-13T23:15:59.934 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.913+0000 7f63f259d700 1 -- 172.21.15.2:0/3945436952 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f63ec0fc710 msgr2=0x7f63ec0fcb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:15:59.935 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.913+0000 7f63f259d700 1 --2- 172.21.15.2:0/3945436952 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f63ec0fc710 0x7f63ec0fcb60 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f63dc004870 tx=0x7f63dc009cb0 comp rx=0 tx=0).stop 2022-05-13T23:15:59.935 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.913+0000 7f63f259d700 1 -- 172.21.15.2:0/3945436952 shutdown_connections 2022-05-13T23:15:59.935 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.913+0000 7f63f259d700 1 --2- 172.21.15.2:0/3945436952 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f63ec0fd0a0 0x7f63ec103960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:59.936 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.913+0000 7f63f259d700 1 --2- 172.21.15.2:0/3945436952 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f63ec0fb470 0x7f63ec0fb880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:59.936 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.913+0000 7f63f259d700 1 --2- 172.21.15.2:0/3945436952 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f63ec0fc710 0x7f63ec0fcb60 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:59.936 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.913+0000 7f63f259d700 1 -- 172.21.15.2:0/3945436952 >> 172.21.15.2:0/3945436952 conn(0x7f63ec0f6c60 msgr2=0x7f63ec0f9090 unknown :-1 s=STATE_NONE l=0).mark_down 2022-05-13T23:15:59.936 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.913+0000 7f63f259d700 1 -- 172.21.15.2:0/3945436952 shutdown_connections 2022-05-13T23:15:59.936 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.913+0000 7f63f259d700 1 -- 172.21.15.2:0/3945436952 wait complete. 2022-05-13T23:15:59.937 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.914+0000 7f63f259d700 1 Processor -- start 2022-05-13T23:15:59.937 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.914+0000 7f63f259d700 1 -- start start 2022-05-13T23:15:59.937 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.915+0000 7f63f259d700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f63ec0fb470 0x7f63ec112850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:15:59.937 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.915+0000 7f63f259d700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f63ec0fc710 0x7f63ec112d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:15:59.938 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.915+0000 7f63f259d700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f63ec0fd0a0 0x7f63ec119e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:15:59.938 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.915+0000 7f63f259d700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63ec11a360 con 0x7f63ec0fc710 2022-05-13T23:15:59.938 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.915+0000 7f63f259d700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63ec11a4d0 con 0x7f63ec0fb470 2022-05-13T23:15:59.938 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.915+0000 7f63f259d700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63ec11a690 con 0x7f63ec0fd0a0 2022-05-13T23:15:59.939 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.915+0000 7f63f0d9a700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f63ec0fc710 0x7f63ec112d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:15:59.939 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.915+0000 7f63f1d9c700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f63ec0fd0a0 0x7f63ec119e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:15:59.939 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.915+0000 7f63f0d9a700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f63ec0fc710 0x7f63ec112d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:54488/0 (socket says 172.21.15.2:54488) 2022-05-13T23:15:59.939 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.915+0000 7f63f0d9a700 1 -- 172.21.15.2:0/2157128646 learned_addr learned my addr 172.21.15.2:0/2157128646 (peer_addr_for_me v2:172.21.15.2:0/0) 2022-05-13T23:15:59.939 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.915+0000 7f63f159b700 1 --2- 172.21.15.2:0/2157128646 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f63ec0fb470 0x7f63ec112850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:15:59.940 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.916+0000 7f63f0d9a700 1 -- 172.21.15.2:0/2157128646 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f63ec0fd0a0 msgr2=0x7f63ec119e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:15:59.940 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.916+0000 7f63f0d9a700 1 --2- 172.21.15.2:0/2157128646 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f63ec0fd0a0 0x7f63ec119e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:59.940 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.916+0000 7f63f0d9a700 1 -- 172.21.15.2:0/2157128646 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f63ec0fb470 msgr2=0x7f63ec112850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:15:59.940 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.916+0000 7f63f0d9a700 1 --2- 172.21.15.2:0/2157128646 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f63ec0fb470 0x7f63ec112850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:59.941 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.916+0000 7f63f0d9a700 1 -- 172.21.15.2:0/2157128646 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63dc005040 con 0x7f63ec0fc710 2022-05-13T23:15:59.941 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.916+0000 7f63f0d9a700 1 --2- 172.21.15.2:0/2157128646 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f63ec0fc710 0x7f63ec112d90 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f63dc004ee0 tx=0x7f63dc009cb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-05-13T23:15:59.941 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.916+0000 7f63e1ffb700 1 -- 172.21.15.2:0/2157128646 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63dc00e040 con 0x7f63ec0fc710 2022-05-13T23:15:59.941 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.916+0000 7f63f259d700 1 -- 172.21.15.2:0/2157128646 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f63ec11a910 con 0x7f63ec0fc710 2022-05-13T23:15:59.942 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.916+0000 7f63e1ffb700 1 -- 172.21.15.2:0/2157128646 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1087+0+0 (secure 0 0 0) 0x7f63dc004200 con 0x7f63ec0fc710 2022-05-13T23:15:59.942 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.916+0000 7f63e1ffb700 1 -- 172.21.15.2:0/2157128646 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63dc01cab0 con 0x7f63ec0fc710 2022-05-13T23:15:59.942 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.916+0000 7f63f259d700 1 -- 172.21.15.2:0/2157128646 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f63ec11adf0 con 0x7f63ec0fc710 2022-05-13T23:15:59.942 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.921+0000 7f63e1ffb700 1 -- 172.21.15.2:0/2157128646 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 16) v1 ==== 125136+0+0 (secure 0 0 0) 0x7f63dc009240 con 0x7f63ec0fc710 2022-05-13T23:15:59.942 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.922+0000 7f6402b82b80 1 Processor -- start 2022-05-13T23:15:59.943 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.922+0000 7f6402b82b80 1 -- start start 2022-05-13T23:15:59.943 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.922+0000 7f6402b82b80 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x561501b47cf0 0x561501b48100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:15:59.943 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.922+0000 7f6402b82b80 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x561501b48640 0x561501b48f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:15:59.943 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.922+0000 7f6402b82b80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x561501b495a0 0x561501b499f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:15:59.944 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.922+0000 7f63e1ffb700 1 --2- 172.21.15.2:0/2157128646 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f63d8096d60 0x7f63d8099210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:15:59.944 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.922+0000 7f6402b82b80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x561501a423a0 con 0x561501b495a0 2022-05-13T23:15:59.944 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.922+0000 7f6402b82b80 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x561501a17400 con 0x561501b47cf0 2022-05-13T23:15:59.944 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.922+0000 7f6402b82b80 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x561501a9b6c0 con 0x561501b48640 2022-05-13T23:15:59.945 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.923+0000 7f63e1ffb700 1 -- 172.21.15.2:0/2157128646 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8120+0+0 (secure 0 0 0) 0x7f63dc0c0440 con 0x7f63ec0fc710 2022-05-13T23:15:59.945 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.923+0000 7f63f0d9a700 1 --2- 172.21.15.2:0/2157128646 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f63d8096d60 0x7f63d8099210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:15:59.949 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.923+0000 7f63f1d9c700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x561501b47cf0 0x561501b48100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:15:59.949 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.923+0000 7f63f1d9c700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x561501b495a0 0x561501b499f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:15:59.949 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.923+0000 7f63f159b700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x561501b48640 0x561501b48f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:15:59.950 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.923+0000 7f63f159b700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x561501b48640 0x561501b48f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.2:47012/0 (socket says 172.21.15.2:47012) 2022-05-13T23:15:59.950 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.923+0000 7f63f159b700 1 -- 172.21.15.2:0/2892671588 learned_addr learned my addr 172.21.15.2:0/2892671588 (peer_addr_for_me v2:172.21.15.2:0/0) 2022-05-13T23:15:59.950 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.923+0000 7f63f159b700 1 -- 172.21.15.2:0/2892671588 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x561501b47cf0 msgr2=0x561501b48100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-05-13T23:15:59.954 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.923+0000 7f63f159b700 1 --2- 172.21.15.2:0/2892671588 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x561501b47cf0 0x561501b48100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:59.954 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.923+0000 7f63f159b700 1 -- 172.21.15.2:0/2892671588 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x561501b495a0 msgr2=0x561501b499f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-05-13T23:15:59.955 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.923+0000 7f63f159b700 1 --2- 172.21.15.2:0/2892671588 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x561501b495a0 0x561501b499f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:59.955 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.923+0000 7f63f159b700 1 -- 172.21.15.2:0/2892671588 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56150192d360 con 0x561501b48640 2022-05-13T23:15:59.955 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.923+0000 7f63f1d9c700 1 --2- 172.21.15.2:0/2892671588 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x561501b47cf0 0x561501b48100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2022-05-13T23:15:59.955 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.924+0000 7f63f0d9a700 1 --2- 172.21.15.2:0/2157128646 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f63d8096d60 0x7f63d8099210 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f63dc0043d0 tx=0x7f63dc08d040 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-05-13T23:15:59.956 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.924+0000 7f63f159b700 1 --2- 172.21.15.2:0/2892671588 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x561501b48640 0x561501b48f80 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f63e8007d90 tx=0x7f63e800b040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=78727ccb66f5f91e server_cookie=0 in_seq=0 out_seq=0 2022-05-13T23:15:59.956 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.924+0000 7f63ceffd700 1 -- 172.21.15.2:0/2892671588 <== mon.2 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63e801a070 con 0x561501b48640 2022-05-13T23:15:59.956 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.924+0000 7f63ceffd700 1 -- 172.21.15.2:0/2892671588 <== mon.2 v2:172.21.15.115:3300/0 2 ==== config(26 keys) v1 ==== 1087+0+0 (secure 0 0 0) 0x7f63e800f040 con 0x561501b48640 2022-05-13T23:15:59.956 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.924+0000 7f63ceffd700 1 -- 172.21.15.2:0/2892671588 <== mon.2 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63e800e030 con 0x561501b48640 2022-05-13T23:15:59.956 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.925+0000 7f6402b82b80 1 -- 172.21.15.2:0/2892671588 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x561501b48640 msgr2=0x561501b48f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:15:59.957 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.925+0000 7f6402b82b80 1 --2- 172.21.15.2:0/2892671588 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x561501b48640 0x561501b48f80 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f63e8007d90 tx=0x7f63e800b040 comp rx=0 tx=0).stop 2022-05-13T23:15:59.958 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.925+0000 7f6402b82b80 1 -- 172.21.15.2:0/2892671588 shutdown_connections 2022-05-13T23:15:59.958 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.925+0000 7f6402b82b80 1 --2- 172.21.15.2:0/2892671588 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x561501b495a0 0x561501b499f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:59.958 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.925+0000 7f6402b82b80 1 --2- 172.21.15.2:0/2892671588 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x561501b47cf0 0x561501b48100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:59.959 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.925+0000 7f6402b82b80 1 --2- 172.21.15.2:0/2892671588 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x561501b48640 0x561501b48f80 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:59.959 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.925+0000 7f6402b82b80 1 -- 172.21.15.2:0/2892671588 >> 172.21.15.2:0/2892671588 conn(0x561501b42b10 msgr2=0x561501b42f00 unknown :-1 s=STATE_NONE l=0).mark_down 2022-05-13T23:15:59.960 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.925+0000 7f6402b82b80 1 -- 172.21.15.2:0/2892671588 shutdown_connections 2022-05-13T23:15:59.960 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.925+0000 7f6402b82b80 1 -- 172.21.15.2:0/2892671588 wait complete. 2022-05-13T23:15:59.960 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.926+0000 7f6402b82b80 1 -- start start 2022-05-13T23:15:59.961 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.926+0000 7f6402b82b80 1 Processor -- start 2022-05-13T23:15:59.961 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.926+0000 7f6402b82b80 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x561501b47cf0 0x561501b51580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:15:59.961 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.926+0000 7f6402b82b80 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x561501b48640 0x561501b51ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:15:59.962 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.926+0000 7f6402b82b80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x561501b495a0 0x561501b58b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:15:59.962 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.926+0000 7f6402b82b80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x561501a957a0 con 0x561501b495a0 2022-05-13T23:15:59.963 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.926+0000 7f6402b82b80 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x56150192f650 con 0x561501b47cf0 2022-05-13T23:15:59.963 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.926+0000 7f6402b82b80 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x561501b59150 con 0x561501b48640 2022-05-13T23:15:59.963 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.926+0000 7f6402b82b80 10 client.0 _ll_register_callbacks cb 0x561501b09de0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2022-05-13T23:15:59.963 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.927+0000 7f63f1d9c700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x561501b47cf0 0x561501b51580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:15:59.964 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.927+0000 7f63f159b700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x561501b48640 0x561501b51ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:15:59.964 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.927+0000 7f63f1d9c700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x561501b495a0 0x561501b58b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:15:59.964 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.927+0000 7f63f159b700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x561501b48640 0x561501b51ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.2:47028/0 (socket says 172.21.15.2:47028) 2022-05-13T23:15:59.964 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.927+0000 7f63f159b700 1 -- 172.21.15.2:0/4269932887 learned_addr learned my addr 172.21.15.2:0/4269932887 (peer_addr_for_me v2:172.21.15.2:0/0) 2022-05-13T23:15:59.965 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.927+0000 7f63f159b700 1 -- 172.21.15.2:0/4269932887 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x561501b47cf0 msgr2=0x561501b51580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-05-13T23:15:59.965 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.927+0000 7f63f159b700 1 --2- 172.21.15.2:0/4269932887 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x561501b47cf0 0x561501b51580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:59.965 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.927+0000 7f63f159b700 1 -- 172.21.15.2:0/4269932887 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x561501b495a0 msgr2=0x561501b58b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-05-13T23:15:59.965 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.927+0000 7f63f159b700 1 --2- 172.21.15.2:0/4269932887 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x561501b495a0 0x561501b58b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:59.966 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.927+0000 7f63f159b700 1 -- 172.21.15.2:0/4269932887 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f63e8005040 con 0x561501b48640 2022-05-13T23:15:59.966 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.927+0000 7f63f159b700 1 --2- 172.21.15.2:0/4269932887 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x561501b48640 0x561501b51ac0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f63e80046f0 tx=0x7f63e800b040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a6ec6b753b56a73b server_cookie=0 in_seq=0 out_seq=0 2022-05-13T23:15:59.966 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.927+0000 7f63c7fff700 10 client.0 ms_handle_connect on v2:172.21.15.115:3300/0 2022-05-13T23:15:59.966 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.928+0000 7f63c7fff700 1 -- 172.21.15.2:0/4269932887 <== mon.2 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63e801a070 con 0x561501b48640 2022-05-13T23:15:59.966 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.928+0000 7f6402b82b80 1 -- 172.21.15.2:0/4269932887 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x561501a499a0 con 0x561501b48640 2022-05-13T23:15:59.967 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.928+0000 7f63c7fff700 1 -- 172.21.15.2:0/4269932887 <== mon.2 v2:172.21.15.115:3300/0 2 ==== osd_map(85..85 src has 1..85) v4 ==== 8120+0+0 (secure 0 0 0) 0x7f63e8015ba0 con 0x561501b48640 2022-05-13T23:15:59.967 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.928+0000 7f63c7fff700 1 -- 172.21.15.2:0/4269932887 <== mon.2 v2:172.21.15.115:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7f63e800e040 con 0x561501b48640 2022-05-13T23:15:59.967 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.928+0000 7f6402b82b80 10 client.25229 fetch_fsmap learned FSMap version 26 2022-05-13T23:15:59.967 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.928+0000 7f6402b82b80 1 -- 172.21.15.2:0/4269932887 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) v3 -- 0x5615018d7fe0 con 0x561501b48640 2022-05-13T23:15:59.968 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.928+0000 7f63c7fff700 1 -- 172.21.15.2:0/4269932887 <== mon.2 v2:172.21.15.115:3300/0 4 ==== config(26 keys) v1 ==== 1087+0+0 (secure 0 0 0) 0x561501a499a0 con 0x561501b48640 2022-05-13T23:15:59.968 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.929+0000 7f63c7fff700 1 -- 172.21.15.2:0/4269932887 <== mon.2 v2:172.21.15.115:3300/0 5 ==== fsmap(e 26) v1 ==== 7448+0+0 (secure 0 0 0) 0x7f63e8020ac0 con 0x561501b48640 2022-05-13T23:15:59.968 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.929+0000 7f6402b82b80 10 client.25229 fetch_fsmap finished waiting for FSMap version 26 2022-05-13T23:15:59.969 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.929+0000 7f6402b82b80 10 client.25229 resolve_mds: resolved name 'l' to daemon mds.l 2022-05-13T23:15:59.970 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.929+0000 7f6402b82b80 20 client.25229 populate_metadata read hostname 'smithi002' 2022-05-13T23:15:59.970 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.929+0000 7f6402b82b80 1 --2- 172.21.15.2:0/4269932887 >> [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] conn(0x561501b5c1d0 0x561501b614b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:15:59.970 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.929+0000 7f6402b82b80 4 client.25229 mds_command: new command op to 24611 tid=0[{"prefix": "get_command_descriptions"}] 2022-05-13T23:15:59.970 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.929+0000 7f6402b82b80 1 -- 172.21.15.2:0/4269932887 --> [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x561501a6a330 con 0x561501b5c1d0 2022-05-13T23:15:59.971 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.929+0000 7f63f1d9c700 1 --2- 172.21.15.2:0/4269932887 >> [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] conn(0x561501b5c1d0 0x561501b614b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:15:59.971 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.930+0000 7f63f1d9c700 1 --2- 172.21.15.2:0/4269932887 >> [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] conn(0x561501b5c1d0 0x561501b614b0 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=94607d1043e8094 server_cookie=7fb69137571a3fa in_seq=0 out_seq=0 2022-05-13T23:15:59.971 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.930+0000 7f63c7fff700 10 client.25229 ms_handle_connect on v2:172.21.15.115:6838/4149542923 2022-05-13T23:15:59.971 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.930+0000 7f63c7fff700 1 -- 172.21.15.2:0/4269932887 <== mds.0 v2:172.21.15.115:6838/4149542923 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17090 (crc 0 0 0) 0x7f63e40262d0 con 0x561501b5c1d0 2022-05-13T23:15:59.972 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.930+0000 7f63c7fff700 10 client.25229 handle_command_reply: tid=0 2022-05-13T23:15:59.972 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.931+0000 7f6402b82b80 1 client.25229 shutdown 2022-05-13T23:15:59.972 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.931+0000 7f6402b82b80 1 -- 172.21.15.2:0/4269932887 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x561501b48640 msgr2=0x561501b51ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:15:59.972 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.931+0000 7f6402b82b80 1 --2- 172.21.15.2:0/4269932887 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x561501b48640 0x561501b51ac0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f63e80046f0 tx=0x7f63e800b040 comp rx=0 tx=0).stop 2022-05-13T23:15:59.973 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.931+0000 7f6402b82b80 1 -- 172.21.15.2:0/4269932887 shutdown_connections 2022-05-13T23:15:59.973 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.931+0000 7f6402b82b80 1 --2- 172.21.15.2:0/4269932887 >> [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] conn(0x561501b5c1d0 0x561501b614b0 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:59.973 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.931+0000 7f6402b82b80 1 --2- 172.21.15.2:0/4269932887 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x561501b495a0 0x561501b58b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:59.973 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.931+0000 7f6402b82b80 1 --2- 172.21.15.2:0/4269932887 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x561501b47cf0 0x561501b51580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:59.974 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.931+0000 7f6402b82b80 1 --2- 172.21.15.2:0/4269932887 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x561501b48640 0x561501b51ac0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:59.974 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.931+0000 7f6402b82b80 1 -- 172.21.15.2:0/4269932887 >> 172.21.15.2:0/4269932887 conn(0x561501b42b10 msgr2=0x561501b41d80 unknown :-1 s=STATE_NONE l=0).mark_down 2022-05-13T23:15:59.974 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.931+0000 7f63c7fff700 0 client.25229 ms_handle_reset on v2:172.21.15.115:6838/4149542923 2022-05-13T23:15:59.975 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.931+0000 7f6402b82b80 1 -- 172.21.15.2:0/4269932887 shutdown_connections 2022-05-13T23:15:59.975 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.931+0000 7f6402b82b80 1 -- 172.21.15.2:0/4269932887 wait complete. 2022-05-13T23:15:59.976 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.931+0000 7f6402b82b80 20 client.25229 trim_cache size 0 max 16384 2022-05-13T23:15:59.986 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.949+0000 7f6402b82b80 1 Processor -- start 2022-05-13T23:15:59.987 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.949+0000 7f6402b82b80 1 -- start start 2022-05-13T23:15:59.987 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.949+0000 7f6402b82b80 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x561501b47cf0 0x561501b654d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:15:59.987 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.949+0000 7f6402b82b80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x561501b48640 0x561501b41300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:15:59.988 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.949+0000 7f6402b82b80 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x561501b495a0 0x561501b511b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:15:59.988 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.949+0000 7f6402b82b80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x561501a5ebb0 con 0x561501b48640 2022-05-13T23:15:59.988 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.949+0000 7f6402b82b80 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x561501a49db0 con 0x561501b47cf0 2022-05-13T23:15:59.988 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.949+0000 7f6402b82b80 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x561501a4da00 con 0x561501b495a0 2022-05-13T23:15:59.989 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.950+0000 7f63f159b700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x561501b48640 0x561501b41300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:15:59.989 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.950+0000 7f63f1d9c700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x561501b47cf0 0x561501b654d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:15:59.989 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.950+0000 7f63f1d9c700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x561501b47cf0 0x561501b654d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.2:46492/0 (socket says 172.21.15.2:46492) 2022-05-13T23:15:59.989 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.950+0000 7f63f1d9c700 1 -- 172.21.15.2:0/3362932418 learned_addr learned my addr 172.21.15.2:0/3362932418 (peer_addr_for_me v2:172.21.15.2:0/0) 2022-05-13T23:15:59.990 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.950+0000 7f63f1d9c700 1 --2- 172.21.15.2:0/3362932418 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x561501b495a0 0x561501b511b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:15:59.990 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.950+0000 7f63f159b700 1 -- 172.21.15.2:0/3362932418 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x561501b495a0 msgr2=0x561501b511b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-05-13T23:15:59.991 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.950+0000 7f63f159b700 1 --2- 172.21.15.2:0/3362932418 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x561501b495a0 0x561501b511b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:59.991 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.950+0000 7f63f159b700 1 -- 172.21.15.2:0/3362932418 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x561501b47cf0 msgr2=0x561501b654d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-05-13T23:15:59.991 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.950+0000 7f63f159b700 1 --2- 172.21.15.2:0/3362932418 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x561501b47cf0 0x561501b654d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:59.991 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.950+0000 7f63f159b700 1 -- 172.21.15.2:0/3362932418 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x561501b51000 con 0x561501b48640 2022-05-13T23:15:59.992 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.950+0000 7f63f159b700 1 --2- 172.21.15.2:0/3362932418 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x561501b48640 0x561501b41300 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f63e8000ea0 tx=0x7f63e800b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=619ba86c5e2f9f95 server_cookie=0 in_seq=0 out_seq=0 2022-05-13T23:15:59.992 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.950+0000 7f63cf7fe700 1 -- 172.21.15.2:0/3362932418 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63e800e070 con 0x561501b48640 2022-05-13T23:15:59.992 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.951+0000 7f63cf7fe700 1 -- 172.21.15.2:0/3362932418 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1087+0+0 (secure 0 0 0) 0x7f63e8007ac0 con 0x561501b48640 2022-05-13T23:15:59.993 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.951+0000 7f63cf7fe700 1 -- 172.21.15.2:0/3362932418 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63e801a030 con 0x561501b48640 2022-05-13T23:15:59.993 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.951+0000 7f6402b82b80 1 -- 172.21.15.2:0/3362932418 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x561501b48640 msgr2=0x561501b41300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:15:59.993 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.951+0000 7f6402b82b80 1 --2- 172.21.15.2:0/3362932418 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x561501b48640 0x561501b41300 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f63e8000ea0 tx=0x7f63e800b040 comp rx=0 tx=0).stop 2022-05-13T23:15:59.993 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.951+0000 7f6402b82b80 1 -- 172.21.15.2:0/3362932418 shutdown_connections 2022-05-13T23:15:59.994 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.951+0000 7f6402b82b80 1 --2- 172.21.15.2:0/3362932418 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x561501b495a0 0x561501b511b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:59.994 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.951+0000 7f6402b82b80 1 --2- 172.21.15.2:0/3362932418 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x561501b47cf0 0x561501b654d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:59.994 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.951+0000 7f6402b82b80 1 --2- 172.21.15.2:0/3362932418 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x561501b48640 0x561501b41300 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:15:59.994 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.951+0000 7f6402b82b80 1 -- 172.21.15.2:0/3362932418 >> 172.21.15.2:0/3362932418 conn(0x561501b42b10 msgr2=0x561501b41d80 unknown :-1 s=STATE_NONE l=0).mark_down 2022-05-13T23:15:59.995 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.951+0000 7f6402b82b80 1 -- 172.21.15.2:0/3362932418 shutdown_connections 2022-05-13T23:15:59.995 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.951+0000 7f6402b82b80 1 -- 172.21.15.2:0/3362932418 wait complete. 2022-05-13T23:15:59.995 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.952+0000 7f6402b82b80 1 -- start start 2022-05-13T23:15:59.995 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.952+0000 7f6402b82b80 1 Processor -- start 2022-05-13T23:15:59.996 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.952+0000 7f6402b82b80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x561501b47cf0 0x561501b55910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:15:59.996 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.952+0000 7f6402b82b80 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x561501b48640 0x561501b55e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:15:59.996 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.952+0000 7f6402b82b80 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x561501b495a0 0x561501bae320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:15:59.997 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.952+0000 7f6402b82b80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x561501bae8a0 con 0x561501b47cf0 2022-05-13T23:15:59.997 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.952+0000 7f6402b82b80 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x561501baea10 con 0x561501b495a0 2022-05-13T23:15:59.997 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.952+0000 7f6402b82b80 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x561501baebd0 con 0x561501b48640 2022-05-13T23:15:59.997 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.952+0000 7f6402b82b80 10 client.0 _ll_register_callbacks cb 0x561501b09de0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2022-05-13T23:15:59.998 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.953+0000 7f63f1d9c700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x561501b47cf0 0x561501b55910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:15:59.998 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.953+0000 7f63f1d9c700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x561501b47cf0 0x561501b55910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:54526/0 (socket says 172.21.15.2:54526) 2022-05-13T23:15:59.998 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.953+0000 7f63f1d9c700 1 -- 172.21.15.2:0/2053415700 learned_addr learned my addr 172.21.15.2:0/2053415700 (peer_addr_for_me v2:172.21.15.2:0/0) 2022-05-13T23:15:59.999 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.953+0000 7f63f159b700 1 --2- 172.21.15.2:0/2053415700 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x561501b48640 0x561501b55e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:15:59.999 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.953+0000 7f63f1d9c700 1 --2- 172.21.15.2:0/2053415700 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x561501b495a0 0x561501bae320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:15:59.999 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.953+0000 7f63f1d9c700 1 -- 172.21.15.2:0/2053415700 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x561501b48640 msgr2=0x561501b55e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-05-13T23:15:59.999 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.953+0000 7f63f1d9c700 1 --2- 172.21.15.2:0/2053415700 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x561501b48640 0x561501b55e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:16:00.000 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.953+0000 7f63f1d9c700 1 -- 172.21.15.2:0/2053415700 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x561501b495a0 msgr2=0x561501bae320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-05-13T23:16:00.000 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.953+0000 7f63f1d9c700 1 --2- 172.21.15.2:0/2053415700 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x561501b495a0 0x561501bae320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:16:00.000 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.953+0000 7f63f1d9c700 1 -- 172.21.15.2:0/2053415700 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5615018d7fe0 con 0x561501b47cf0 2022-05-13T23:16:00.000 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.953+0000 7f63f1d9c700 1 --2- 172.21.15.2:0/2053415700 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x561501b47cf0 0x561501b55910 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f63e4019d30 tx=0x7f63e400b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=57eefcfcb68bf8ee server_cookie=0 in_seq=0 out_seq=0 2022-05-13T23:16:00.001 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.953+0000 7f63c7fff700 10 client.0 ms_handle_connect on v2:172.21.15.2:3300/0 2022-05-13T23:16:00.001 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.954+0000 7f63c7fff700 1 -- 172.21.15.2:0/2053415700 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63e400e040 con 0x561501b47cf0 2022-05-13T23:16:00.002 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.954+0000 7f6402b82b80 1 -- 172.21.15.2:0/2053415700 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x561501b185f0 con 0x561501b47cf0 2022-05-13T23:16:00.002 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.954+0000 7f63c7fff700 1 -- 172.21.15.2:0/2053415700 <== mon.0 v2:172.21.15.2:3300/0 2 ==== osd_map(85..85 src has 1..85) v4 ==== 8120+0+0 (secure 0 0 0) 0x7f63e40093d0 con 0x561501b47cf0 2022-05-13T23:16:00.002 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.954+0000 7f63c7fff700 1 -- 172.21.15.2:0/2053415700 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7f63e4004020 con 0x561501b47cf0 2022-05-13T23:16:00.003 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.954+0000 7f6402b82b80 10 client.15114 fetch_fsmap learned FSMap version 26 2022-05-13T23:16:00.003 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.954+0000 7f6402b82b80 1 -- 172.21.15.2:0/2053415700 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) v3 -- 0x5615018fff80 con 0x561501b47cf0 2022-05-13T23:16:00.003 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.954+0000 7f63c7fff700 1 -- 172.21.15.2:0/2053415700 <== mon.0 v2:172.21.15.2:3300/0 4 ==== config(26 keys) v1 ==== 1087+0+0 (secure 0 0 0) 0x561501b185f0 con 0x561501b47cf0 2022-05-13T23:16:00.004 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.954+0000 7f63c7fff700 1 -- 172.21.15.2:0/2053415700 <== mon.0 v2:172.21.15.2:3300/0 5 ==== fsmap(e 26) v1 ==== 7448+0+0 (secure 0 0 0) 0x7f63e402c5e0 con 0x561501b47cf0 2022-05-13T23:16:00.004 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.954+0000 7f6402b82b80 10 client.15114 fetch_fsmap finished waiting for FSMap version 26 2022-05-13T23:16:00.004 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.954+0000 7f6402b82b80 10 client.15114 resolve_mds: resolved name 'l' to daemon mds.l 2022-05-13T23:16:00.005 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.954+0000 7f6402b82b80 20 client.15114 populate_metadata read hostname 'smithi002' 2022-05-13T23:16:00.005 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.954+0000 7f6402b82b80 1 --2- 172.21.15.2:0/2053415700 >> [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] conn(0x561501bb3490 0x561501bafe00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:16:00.005 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.954+0000 7f6402b82b80 4 client.15114 mds_command: new command op to 24611 tid=0[{"prefix": "status"}] 2022-05-13T23:16:00.006 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.955+0000 7f6402b82b80 1 -- 172.21.15.2:0/2053415700 --> [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] -- command(tid 0: {"prefix": "status"}) v1 -- 0x561501a3b790 con 0x561501bb3490 2022-05-13T23:16:00.006 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.955+0000 7f63f159b700 1 --2- 172.21.15.2:0/2053415700 >> [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] conn(0x561501bb3490 0x561501bafe00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:16:00.006 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.955+0000 7f63f159b700 1 --2- 172.21.15.2:0/2053415700 >> [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] conn(0x561501bb3490 0x561501bafe00 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=acfff6487d07091a server_cookie=520984f815ccb26f in_seq=0 out_seq=0 2022-05-13T23:16:00.007 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.955+0000 7f63c7fff700 10 client.15114 ms_handle_connect on v2:172.21.15.115:6838/4149542923 2022-05-13T23:16:00.007 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.955+0000 7f63c7fff700 1 -- 172.21.15.2:0/2053415700 <== mds.0 v2:172.21.15.115:6838/4149542923 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+322 (crc 0 0 0) 0x561501a3b790 con 0x561501bb3490 2022-05-13T23:16:00.007 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.955+0000 7f63c7fff700 10 client.15114 handle_command_reply: tid=0 2022-05-13T23:16:00.008 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.956+0000 7f6402b82b80 1 client.15114 shutdown 2022-05-13T23:16:00.008 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.956+0000 7f6402b82b80 1 -- 172.21.15.2:0/2053415700 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x561501b47cf0 msgr2=0x561501b55910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:16:00.011 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.956+0000 7f6402b82b80 1 --2- 172.21.15.2:0/2053415700 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x561501b47cf0 0x561501b55910 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f63e4019d30 tx=0x7f63e400b040 comp rx=0 tx=0).stop 2022-05-13T23:16:00.011 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.956+0000 7f6402b82b80 1 -- 172.21.15.2:0/2053415700 shutdown_connections 2022-05-13T23:16:00.011 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.956+0000 7f6402b82b80 1 --2- 172.21.15.2:0/2053415700 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x561501b495a0 0x561501bae320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:16:00.012 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.956+0000 7f6402b82b80 1 --2- 172.21.15.2:0/2053415700 >> [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] conn(0x561501bb3490 0x561501bafe00 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:16:00.012 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.956+0000 7f6402b82b80 1 --2- 172.21.15.2:0/2053415700 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x561501b47cf0 0x561501b55910 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:16:00.012 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.956+0000 7f6402b82b80 1 --2- 172.21.15.2:0/2053415700 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x561501b48640 0x561501b55e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:16:00.012 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.956+0000 7f63c7fff700 0 client.15114 ms_handle_reset on v2:172.21.15.115:6838/4149542923 2022-05-13T23:16:00.013 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.956+0000 7f6402b82b80 1 -- 172.21.15.2:0/2053415700 >> 172.21.15.2:0/2053415700 conn(0x561501b42b10 msgr2=0x561501b654d0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-05-13T23:16:00.013 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.956+0000 7f6402b82b80 1 -- 172.21.15.2:0/2053415700 shutdown_connections 2022-05-13T23:16:00.013 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.956+0000 7f6402b82b80 1 -- 172.21.15.2:0/2053415700 wait complete. 2022-05-13T23:16:00.014 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.956+0000 7f6402b82b80 20 client.15114 trim_cache size 0 max 16384 2022-05-13T23:16:00.015 INFO:teuthology.orchestra.run.smithi002.stdout:{ 2022-05-13T23:16:00.016 INFO:teuthology.orchestra.run.smithi002.stdout: "cluster_fsid": "16cf5f88-d311-11ec-be40-001a4aab830c", 2022-05-13T23:16:00.016 INFO:teuthology.orchestra.run.smithi002.stdout: "whoami": 0, 2022-05-13T23:16:00.016 INFO:teuthology.orchestra.run.smithi002.stdout: "id": 24611, 2022-05-13T23:16:00.016 INFO:teuthology.orchestra.run.smithi002.stdout: "want_state": "up:active", 2022-05-13T23:16:00.017 INFO:teuthology.orchestra.run.smithi002.stdout: "state": "up:active", 2022-05-13T23:16:00.017 INFO:teuthology.orchestra.run.smithi002.stdout: "fs_name": "cephfs", 2022-05-13T23:16:00.017 INFO:teuthology.orchestra.run.smithi002.stdout: "rank_uptime": 103.716307818, 2022-05-13T23:16:00.017 INFO:teuthology.orchestra.run.smithi002.stdout: "mdsmap_epoch": 26, 2022-05-13T23:16:00.018 INFO:teuthology.orchestra.run.smithi002.stdout: "osdmap_epoch": 85, 2022-05-13T23:16:00.018 INFO:teuthology.orchestra.run.smithi002.stdout: "osdmap_epoch_barrier": 83, 2022-05-13T23:16:00.018 INFO:teuthology.orchestra.run.smithi002.stdout: "uptime": 109.725705926 2022-05-13T23:16:00.018 INFO:teuthology.orchestra.run.smithi002.stdout:} 2022-05-13T23:16:00.021 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.957+0000 7f63f259d700 1 -- 172.21.15.2:0/2157128646 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f63d8096d60 msgr2=0x7f63d8099210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:16:00.021 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.957+0000 7f63f259d700 1 --2- 172.21.15.2:0/2157128646 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f63d8096d60 0x7f63d8099210 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f63dc0043d0 tx=0x7f63dc08d040 comp rx=0 tx=0).stop 2022-05-13T23:16:00.022 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.957+0000 7f63f259d700 1 -- 172.21.15.2:0/2157128646 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f63ec0fc710 msgr2=0x7f63ec112d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:16:00.022 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.957+0000 7f63f259d700 1 --2- 172.21.15.2:0/2157128646 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f63ec0fc710 0x7f63ec112d90 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f63dc004ee0 tx=0x7f63dc009cb0 comp rx=0 tx=0).stop 2022-05-13T23:16:00.022 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.957+0000 7f63f259d700 1 -- 172.21.15.2:0/2157128646 shutdown_connections 2022-05-13T23:16:00.022 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.957+0000 7f63f259d700 1 --2- 172.21.15.2:0/2157128646 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f63ec0fd0a0 0x7f63ec119e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:16:00.022 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.957+0000 7f63f259d700 1 --2- 172.21.15.2:0/2157128646 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f63ec0fb470 0x7f63ec112850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:16:00.023 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.957+0000 7f63f259d700 1 --2- 172.21.15.2:0/2157128646 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f63d8096d60 0x7f63d8099210 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:16:00.023 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.957+0000 7f63f259d700 1 --2- 172.21.15.2:0/2157128646 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f63ec0fc710 0x7f63ec112d90 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:16:00.023 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.957+0000 7f63f259d700 1 -- 172.21.15.2:0/2157128646 >> 172.21.15.2:0/2157128646 conn(0x7f63ec0f6c60 msgr2=0x7f63ec0f9030 unknown :-1 s=STATE_NONE l=0).mark_down 2022-05-13T23:16:00.023 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.957+0000 7f63f259d700 1 -- 172.21.15.2:0/2157128646 shutdown_connections 2022-05-13T23:16:00.024 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:15:59.957+0000 7f63f259d700 1 -- 172.21.15.2:0/2157128646 wait complete. 2022-05-13T23:16:00.024 INFO:teuthology.run_tasks:Running task kclient... 2022-05-13T23:16:00.048 INFO:tasks.kclient:Mounting kernel clients... 2022-05-13T23:16:00.049 INFO:tasks.kclient:config is {'client.0': {}} 2022-05-13T23:16:00.049 INFO:teuthology.orchestra.run:Running command with timeout 300 2022-05-13T23:16:00.050 DEBUG:teuthology.orchestra.run.smithi002:> ip netns list 2022-05-13T23:16:00.101 INFO:teuthology.orchestra.run:Running command with timeout 300 2022-05-13T23:16:00.102 DEBUG:teuthology.orchestra.run.smithi002:> sudo ip link delete ceph-brx 2022-05-13T23:16:00.223 INFO:teuthology.orchestra.run.smithi002.stderr:Cannot find device "ceph-brx" 2022-05-13T23:16:00.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-05-13T23:16:00.229 INFO:tasks.kclient:client.0 config is {'mntopts': ['ms_mode=crc', 'nowsync'], 'syntax': 'v2'} 2022-05-13T23:16:00.230 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:23b0658391ce94ca40799469eec00c28b298816c shell --fsid 16cf5f88-d311-11ec-be40-001a4aab830c -- ceph fs ls 2022-05-13T23:16:00.572 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:16:00 smithi002 ceph-mon[71816]: pgmap v342: 97 pgs: 97 active+clean; 584 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-05-13T23:16:00.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:16:00 smithi047 ceph-mon[77147]: pgmap v342: 97 pgs: 97 active+clean; 584 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-05-13T23:16:00.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:16:00 smithi115 ceph-mon[78593]: pgmap v342: 97 pgs: 97 active+clean; 584 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-05-13T23:16:00.914 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/16cf5f88-d311-11ec-be40-001a4aab830c/mon.a/config 2022-05-13T23:16:02.756 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:16:02.740+0000 7f644f4b5700 1 -- 172.21.15.2:0/3181160971 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f64480fb560 msgr2=0x7f64480fb970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:16:02.756 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:16:02.740+0000 7f644f4b5700 1 --2- 172.21.15.2:0/3181160971 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f64480fb560 0x7f64480fb970 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f6444004870 tx=0x7f6444009cb0 comp rx=0 tx=0).stop 2022-05-13T23:16:02.757 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:16:02.741+0000 7f644f4b5700 1 -- 172.21.15.2:0/3181160971 shutdown_connections 2022-05-13T23:16:02.757 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:16:02.741+0000 7f644f4b5700 1 --2- 172.21.15.2:0/3181160971 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f64480fd190 0x7f6448103a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:16:02.758 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:16:02.741+0000 7f644f4b5700 1 --2- 172.21.15.2:0/3181160971 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f64480fb560 0x7f64480fb970 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:16:02.758 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:16:02.741+0000 7f644f4b5700 1 --2- 172.21.15.2:0/3181160971 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f64480fc800 0x7f64480fcc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:16:02.759 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:16:02.741+0000 7f644f4b5700 1 -- 172.21.15.2:0/3181160971 >> 172.21.15.2:0/3181160971 conn(0x7f64480f6d30 msgr2=0x7f64480f9180 unknown :-1 s=STATE_NONE l=0).mark_down 2022-05-13T23:16:02.759 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:16:02.741+0000 7f644f4b5700 1 -- 172.21.15.2:0/3181160971 shutdown_connections 2022-05-13T23:16:02.759 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:16:02.741+0000 7f644f4b5700 1 -- 172.21.15.2:0/3181160971 wait complete. 2022-05-13T23:16:02.760 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:16:02.742+0000 7f644f4b5700 1 Processor -- start 2022-05-13T23:16:02.760 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:16:02.742+0000 7f644f4b5700 1 -- start start 2022-05-13T23:16:02.761 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:16:02.742+0000 7f644f4b5700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f64480fb560 0x7f644810c350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:16:02.761 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:16:02.742+0000 7f644f4b5700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f64480fc800 0x7f644810c890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:16:02.761 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:16:02.742+0000 7f644f4b5700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f64480fd190 0x7f6448113920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:16:02.762 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:16:02.742+0000 7f644f4b5700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6448113e60 con 0x7f64480fd190 2022-05-13T23:16:02.762 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:16:02.742+0000 7f644f4b5700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6448113fd0 con 0x7f64480fc800 2022-05-13T23:16:02.762 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:16:02.742+0000 7f644f4b5700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6448114190 con 0x7f64480fb560 2022-05-13T23:16:02.763 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:16:02.743+0000 7f644da52700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f64480fd190 0x7f6448113920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:16:02.763 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:16:02.743+0000 7f644ca50700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f64480fc800 0x7f644810c890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:16:02.764 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:16:02.743+0000 7f644da52700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f64480fd190 0x7f6448113920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:54558/0 (socket says 172.21.15.2:54558) 2022-05-13T23:16:02.764 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:16:02.743+0000 7f644da52700 1 -- 172.21.15.2:0/3058657670 learned_addr learned my addr 172.21.15.2:0/3058657670 (peer_addr_for_me v2:172.21.15.2:0/0) 2022-05-13T23:16:02.765 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:16:02.743+0000 7f644d251700 1 --2- 172.21.15.2:0/3058657670 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f64480fb560 0x7f644810c350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:16:02.765 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:16:02.743+0000 7f644da52700 1 -- 172.21.15.2:0/3058657670 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f64480fb560 msgr2=0x7f644810c350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:16:02.765 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:16:02.743+0000 7f644da52700 1 --2- 172.21.15.2:0/3058657670 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f64480fb560 0x7f644810c350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:16:02.766 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:16:02.743+0000 7f644da52700 1 -- 172.21.15.2:0/3058657670 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f64480fc800 msgr2=0x7f644810c890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:16:02.766 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:16:02.743+0000 7f644da52700 1 --2- 172.21.15.2:0/3058657670 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f64480fc800 0x7f644810c890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:16:02.766 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:16:02.743+0000 7f644da52700 1 -- 172.21.15.2:0/3058657670 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6438005040 con 0x7f64480fd190 2022-05-13T23:16:02.767 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:16:02.743+0000 7f644ca50700 1 --2- 172.21.15.2:0/3058657670 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f64480fc800 0x7f644810c890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-05-13T23:16:02.767 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:16:02.743+0000 7f644da52700 1 --2- 172.21.15.2:0/3058657670 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f64480fd190 0x7f6448113920 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f644000ec20 tx=0x7f644000b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-05-13T23:16:02.767 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:16:02.744+0000 7f643dffb700 1 -- 172.21.15.2:0/3058657670 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64400092e0 con 0x7f64480fd190 2022-05-13T23:16:02.767 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:16:02.744+0000 7f643dffb700 1 -- 172.21.15.2:0/3058657670 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1087+0+0 (secure 0 0 0) 0x7f644000f040 con 0x7f64480fd190 2022-05-13T23:16:02.768 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:16:02.744+0000 7f643dffb700 1 -- 172.21.15.2:0/3058657670 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f644000cab0 con 0x7f64480fd190 2022-05-13T23:16:02.768 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:16:02.744+0000 7f644f4b5700 1 -- 172.21.15.2:0/3058657670 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6444005040 con 0x7f64480fd190 2022-05-13T23:16:02.769 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:16:02.744+0000 7f644f4b5700 1 -- 172.21.15.2:0/3058657670 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6448114800 con 0x7f64480fd190 2022-05-13T23:16:02.769 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:16:02.745+0000 7f644f4b5700 1 -- 172.21.15.2:0/3058657670 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6448106d50 con 0x7f64480fd190 2022-05-13T23:16:02.771 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:16:02.747+0000 7f643dffb700 1 -- 172.21.15.2:0/3058657670 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 16) v1 ==== 125136+0+0 (secure 0 0 0) 0x7f64400077a0 con 0x7f64480fd190 2022-05-13T23:16:02.772 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:16:02.748+0000 7f643dffb700 1 --2- 172.21.15.2:0/3058657670 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f6434096ce0 0x7f6434099190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-05-13T23:16:02.772 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:16:02.748+0000 7f643dffb700 1 -- 172.21.15.2:0/3058657670 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8120+0+0 (secure 0 0 0) 0x7f64400bf740 con 0x7f64480fd190 2022-05-13T23:16:02.773 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:16:02.748+0000 7f644d251700 1 --2- 172.21.15.2:0/3058657670 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f6434096ce0 0x7f6434099190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-05-13T23:16:02.773 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:16:02.749+0000 7f644d251700 1 --2- 172.21.15.2:0/3058657670 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f6434096ce0 0x7f6434099190 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f6444004840 tx=0x7f6444009cb0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-05-13T23:16:02.782 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:16:02.752+0000 7f643dffb700 1 -- 172.21.15.2:0/3058657670 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+165215 (secure 0 0 0) 0x7f644008c9f0 con 0x7f64480fd190 2022-05-13T23:16:02.799 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:16:02 smithi047 ceph-mon[77147]: pgmap v343: 97 pgs: 97 active+clean; 584 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2022-05-13T23:16:02.823 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:16:02 smithi002 ceph-mon[71816]: pgmap v343: 97 pgs: 97 active+clean; 584 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2022-05-13T23:16:02.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:16:02 smithi115 ceph-mon[78593]: pgmap v343: 97 pgs: 97 active+clean; 584 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2022-05-13T23:16:02.991 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:16:02.987+0000 7f644f4b5700 1 -- 172.21.15.2:0/3058657670 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f644802a4f0 con 0x7f64480fd190 2022-05-13T23:16:02.999 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:16:02.987+0000 7f643dffb700 1 -- 172.21.15.2:0/3058657670 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v26) v1 ==== 53+0+73 (secure 0 0 0) 0x7f644000cea0 con 0x7f64480fd190 2022-05-13T23:16:02.999 INFO:teuthology.orchestra.run.smithi002.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2022-05-13T23:16:03.000 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:16:02.990+0000 7f644f4b5700 1 -- 172.21.15.2:0/3058657670 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f6434096ce0 msgr2=0x7f6434099190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:16:03.001 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:16:02.990+0000 7f644f4b5700 1 --2- 172.21.15.2:0/3058657670 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f6434096ce0 0x7f6434099190 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f6444004840 tx=0x7f6444009cb0 comp rx=0 tx=0).stop 2022-05-13T23:16:03.001 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:16:02.990+0000 7f644f4b5700 1 -- 172.21.15.2:0/3058657670 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f64480fd190 msgr2=0x7f6448113920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-05-13T23:16:03.001 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:16:02.990+0000 7f644f4b5700 1 --2- 172.21.15.2:0/3058657670 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f64480fd190 0x7f6448113920 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f644000ec20 tx=0x7f644000b040 comp rx=0 tx=0).stop 2022-05-13T23:16:03.002 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:16:02.990+0000 7f644f4b5700 1 -- 172.21.15.2:0/3058657670 shutdown_connections 2022-05-13T23:16:03.003 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:16:02.990+0000 7f644f4b5700 1 --2- 172.21.15.2:0/3058657670 >> [v2:172.21.15.2:6800/2788339627,v1:172.21.15.2:6801/2788339627] conn(0x7f6434096ce0 0x7f6434099190 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:16:03.003 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:16:02.990+0000 7f644f4b5700 1 --2- 172.21.15.2:0/3058657670 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f64480fd190 0x7f6448113920 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:16:03.003 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:16:02.990+0000 7f644f4b5700 1 --2- 172.21.15.2:0/3058657670 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f64480fb560 0x7f644810c350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:16:03.003 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:16:02.990+0000 7f644f4b5700 1 --2- 172.21.15.2:0/3058657670 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f64480fc800 0x7f644810c890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-05-13T23:16:03.003 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:16:02.990+0000 7f644f4b5700 1 -- 172.21.15.2:0/3058657670 >> 172.21.15.2:0/3058657670 conn(0x7f64480f6d30 msgr2=0x7f64480ffa30 unknown :-1 s=STATE_NONE l=0).mark_down 2022-05-13T23:16:03.004 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:16:02.990+0000 7f644f4b5700 1 -- 172.21.15.2:0/3058657670 shutdown_connections 2022-05-13T23:16:03.004 INFO:teuthology.orchestra.run.smithi002.stderr:2022-05-13T23:16:02.990+0000 7f644f4b5700 1 -- 172.21.15.2:0/3058657670 wait complete. 2022-05-13T23:16:03.654 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:16:03 smithi002 ceph-mon[71816]: from='client.? 172.21.15.2:0/3058657670' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2022-05-13T23:16:03.688 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2022-05-13T23:16:03.689 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2022-05-13T23:16:03.689 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi002.front.sepia.ceph.com 2022-05-13T23:16:03.690 INFO:tasks.cephfs.mount:self.client.name = client.0 2022-05-13T23:16:03.690 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2022-05-13T23:16:03.691 INFO:tasks.cephfs.mount:self.cephfs_name = None 2022-05-13T23:16:03.691 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2022-05-13T23:16:03.692 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2022-05-13T23:16:03.692 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.0' 2022-05-13T23:16:03.693 INFO:teuthology.orchestra.run:Running command with timeout 300 2022-05-13T23:16:03.694 DEBUG:teuthology.orchestra.run.smithi002:> ip addr 2022-05-13T23:16:03.750 INFO:teuthology.orchestra.run.smithi002.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2022-05-13T23:16:03.751 INFO:teuthology.orchestra.run.smithi002.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2022-05-13T23:16:03.752 INFO:teuthology.orchestra.run.smithi002.stdout: inet 127.0.0.1/8 scope host lo 2022-05-13T23:16:03.752 INFO:teuthology.orchestra.run.smithi002.stdout: valid_lft forever preferred_lft forever 2022-05-13T23:16:03.753 INFO:teuthology.orchestra.run.smithi002.stdout: inet6 ::1/128 scope host 2022-05-13T23:16:03.753 INFO:teuthology.orchestra.run.smithi002.stdout: valid_lft forever preferred_lft forever 2022-05-13T23:16:03.754 INFO:teuthology.orchestra.run.smithi002.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2022-05-13T23:16:03.754 INFO:teuthology.orchestra.run.smithi002.stdout: link/ether 0c:c4:7a:6b:fe:02 brd ff:ff:ff:ff:ff:ff 2022-05-13T23:16:03.756 INFO:teuthology.orchestra.run.smithi002.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2022-05-13T23:16:03.757 INFO:teuthology.orchestra.run.smithi002.stdout: link/ether 0c:c4:7a:6b:fe:03 brd ff:ff:ff:ff:ff:ff 2022-05-13T23:16:03.757 INFO:teuthology.orchestra.run.smithi002.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2022-05-13T23:16:03.758 INFO:teuthology.orchestra.run.smithi002.stdout: link/ether 0c:c4:7a:bd:19:20 brd ff:ff:ff:ff:ff:ff 2022-05-13T23:16:03.758 INFO:teuthology.orchestra.run.smithi002.stdout: inet 172.21.15.2/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2022-05-13T23:16:03.759 INFO:teuthology.orchestra.run.smithi002.stdout: valid_lft 41545sec preferred_lft 41545sec 2022-05-13T23:16:03.759 INFO:teuthology.orchestra.run.smithi002.stdout: inet6 fe80::ec4:7aff:febd:1920/64 scope link noprefixroute 2022-05-13T23:16:03.759 INFO:teuthology.orchestra.run.smithi002.stdout: valid_lft forever preferred_lft forever 2022-05-13T23:16:03.760 INFO:teuthology.orchestra.run.smithi002.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2022-05-13T23:16:03.760 INFO:teuthology.orchestra.run.smithi002.stdout: link/ether 0c:c4:7a:bd:19:21 brd ff:ff:ff:ff:ff:ff 2022-05-13T23:16:03.772 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2022-05-13T23:16:03.774 INFO:teuthology.orchestra.run:Running command with timeout 300 2022-05-13T23:16:03.774 DEBUG:teuthology.orchestra.run.smithi002:> (cd / && exec bash -c ' 2022-05-13T23:16:03.775 DEBUG:teuthology.orchestra.run.smithi002:> set -e 2022-05-13T23:16:03.775 DEBUG:teuthology.orchestra.run.smithi002:> sudo ip link add name ceph-brx type bridge 2022-05-13T23:16:03.775 DEBUG:teuthology.orchestra.run.smithi002:> sudo ip addr flush dev ceph-brx 2022-05-13T23:16:03.776 DEBUG:teuthology.orchestra.run.smithi002:> sudo ip link set ceph-brx up 2022-05-13T23:16:03.777 DEBUG:teuthology.orchestra.run.smithi002:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2022-05-13T23:16:03.777 DEBUG:teuthology.orchestra.run.smithi002:> ') 2022-05-13T23:16:03.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:16:03 smithi047 ceph-mon[77147]: from='client.? 172.21.15.2:0/3058657670' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2022-05-13T23:16:03.862 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:16:03 smithi115 ceph-mon[78593]: from='client.? 172.21.15.2:0/3058657670' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2022-05-13T23:16:04.099 INFO:teuthology.orchestra.run:Running command with timeout 300 2022-05-13T23:16:04.099 DEBUG:teuthology.orchestra.run.smithi002:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2022-05-13T23:16:04.182 INFO:teuthology.orchestra.run.smithi002.stdout:1 2022-05-13T23:16:04.188 INFO:teuthology.orchestra.run:Running command with timeout 300 2022-05-13T23:16:04.188 DEBUG:teuthology.orchestra.run.smithi002:> route 2022-05-13T23:16:04.278 INFO:teuthology.orchestra.run.smithi002.stdout:Kernel IP routing table 2022-05-13T23:16:04.279 INFO:teuthology.orchestra.run.smithi002.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2022-05-13T23:16:04.279 INFO:teuthology.orchestra.run.smithi002.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2022-05-13T23:16:04.280 INFO:teuthology.orchestra.run.smithi002.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2022-05-13T23:16:04.280 INFO:teuthology.orchestra.run.smithi002.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2022-05-13T23:16:04.282 INFO:teuthology.orchestra.run:Running command with timeout 300 2022-05-13T23:16:04.282 DEBUG:teuthology.orchestra.run.smithi002:> (cd / && exec bash -c ' 2022-05-13T23:16:04.283 DEBUG:teuthology.orchestra.run.smithi002:> set -e 2022-05-13T23:16:04.284 DEBUG:teuthology.orchestra.run.smithi002:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2022-05-13T23:16:04.285 DEBUG:teuthology.orchestra.run.smithi002:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2022-05-13T23:16:04.286 DEBUG:teuthology.orchestra.run.smithi002:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2022-05-13T23:16:04.287 DEBUG:teuthology.orchestra.run.smithi002:> ') 2022-05-13T23:16:04.637 INFO:teuthology.orchestra.run:Running command with timeout 300 2022-05-13T23:16:04.638 DEBUG:teuthology.orchestra.run.smithi002:> ip netns list 2022-05-13T23:16:04.691 INFO:teuthology.orchestra.run:Running command with timeout 300 2022-05-13T23:16:04.692 DEBUG:teuthology.orchestra.run.smithi002:> ip netns list-id 2022-05-13T23:16:04.742 INFO:teuthology.orchestra.run:Running command with timeout 300 2022-05-13T23:16:04.743 DEBUG:teuthology.orchestra.run.smithi002:> (cd / && exec bash -c ' 2022-05-13T23:16:04.744 DEBUG:teuthology.orchestra.run.smithi002:> set -e 2022-05-13T23:16:04.744 DEBUG:teuthology.orchestra.run.smithi002:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.0 2022-05-13T23:16:04.745 DEBUG:teuthology.orchestra.run.smithi002:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.0 0 2022-05-13T23:16:04.745 DEBUG:teuthology.orchestra.run.smithi002:> ') 2022-05-13T23:16:04.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:16:04 smithi047 ceph-mon[77147]: pgmap v344: 97 pgs: 97 active+clean; 584 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2022-05-13T23:16:04.824 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:16:04 smithi002 ceph-mon[71816]: pgmap v344: 97 pgs: 97 active+clean; 584 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2022-05-13T23:16:04.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:16:04 smithi115 ceph-mon[78593]: pgmap v344: 97 pgs: 97 active+clean; 584 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2022-05-13T23:16:04.914 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.0' with 192.168.0.1/16 2022-05-13T23:16:04.915 INFO:teuthology.orchestra.run:Running command with timeout 300 2022-05-13T23:16:04.916 DEBUG:teuthology.orchestra.run.smithi002:> (cd / && exec bash -c ' 2022-05-13T23:16:04.916 DEBUG:teuthology.orchestra.run.smithi002:> set -e 2022-05-13T23:16:04.916 DEBUG:teuthology.orchestra.run.smithi002:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.0 type veth peer name brx.0 2022-05-13T23:16:04.917 DEBUG:teuthology.orchestra.run.smithi002:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2022-05-13T23:16:04.917 DEBUG:teuthology.orchestra.run.smithi002:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set veth0 up 2022-05-13T23:16:04.917 DEBUG:teuthology.orchestra.run.smithi002:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set lo up 2022-05-13T23:16:04.918 DEBUG:teuthology.orchestra.run.smithi002:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip route add default via 192.168.255.254 2022-05-13T23:16:04.918 DEBUG:teuthology.orchestra.run.smithi002:> ') 2022-05-13T23:16:05.216 INFO:teuthology.orchestra.run:Running command with timeout 300 2022-05-13T23:16:05.217 DEBUG:teuthology.orchestra.run.smithi002:> (cd / && exec bash -c ' 2022-05-13T23:16:05.217 DEBUG:teuthology.orchestra.run.smithi002:> set -e 2022-05-13T23:16:05.218 DEBUG:teuthology.orchestra.run.smithi002:> sudo ip link set brx.0 up 2022-05-13T23:16:05.218 DEBUG:teuthology.orchestra.run.smithi002:> sudo ip link set dev brx.0 master ceph-brx 2022-05-13T23:16:05.218 DEBUG:teuthology.orchestra.run.smithi002:> ') 2022-05-13T23:16:05.419 INFO:teuthology.orchestra.run:Running command with timeout 60 2022-05-13T23:16:05.420 DEBUG:teuthology.orchestra.run.smithi002:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2022-05-13T23:16:05.472 INFO:teuthology.orchestra.run.smithi002.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2022-05-13T23:16:05.473 INFO:teuthology.orchestra.run:Running command with timeout 60 2022-05-13T23:16:05.473 DEBUG:teuthology.orchestra.run.smithi002:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2022-05-13T23:16:05.563 INFO:tasks.cephfs.kernel_mount:mounting using device: 0@.cephfs=/ 2022-05-13T23:16:05.564 INFO:teuthology.orchestra.run:Running command with timeout 1800 2022-05-13T23:16:05.564 DEBUG:teuthology.orchestra.run.smithi002:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 0@.cephfs=/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,ms_mode=crc,nowsync,nofallback 2022-05-13T23:16:05.671 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:16:05 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-05-13T23:16:05.672 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:16:05 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-05-13T23:16:05.734 INFO:teuthology.orchestra.run.smithi002.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,ms_mode=crc,nowsync,nofallback 2022-05-13T23:16:05.735 INFO:teuthology.orchestra.run.smithi002.stdout:mount.ceph: options "norequire_active_mds,norbytes,ms_mode=crc,nowsync". 2022-05-13T23:16:05.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:16:05 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-05-13T23:16:05.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:16:05 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-05-13T23:16:05.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:16:05 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-05-13T23:16:05.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:16:05 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-05-13T23:16:05.884 INFO:teuthology.orchestra.run.smithi002.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,ms_mode=crc,nowsync,nofallback 2022-05-13T23:16:05.884 INFO:teuthology.orchestra.run.smithi002.stdout:mount.ceph: options "norequire_active_mds,norbytes,ms_mode=crc,nowsync". 2022-05-13T23:16:05.884 INFO:teuthology.orchestra.run.smithi002.stdout:mount.ceph: resolved to: "172.21.15.2:3300,172.21.15.47:3300,172.21.15.115:3300" 2022-05-13T23:16:05.885 INFO:teuthology.orchestra.run.smithi002.stdout:mount.ceph: trying mount with new device syntax: 0@16cf5f88-d311-11ec-be40-001a4aab830c.cephfs=/ 2022-05-13T23:16:05.885 INFO:teuthology.orchestra.run.smithi002.stdout:mount.ceph: options "norequire_active_mds,norbytes,ms_mode=crc,nowsync,name=0,key=0,mon_addr=172.21.15.2:3300/172.21.15.47:3300/172.21.15.115:3300" will pass to kernel 2022-05-13T23:16:05.894 INFO:tasks.cephfs.kernel_mount:mount command passed 2022-05-13T23:16:05.895 INFO:teuthology.orchestra.run:Running command with timeout 300 2022-05-13T23:16:05.896 DEBUG:teuthology.orchestra.run.smithi002:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2022-05-13T23:16:05.981 INFO:teuthology.run_tasks:Running task check-counter... 2022-05-13T23:16:05.996 INFO:teuthology.run_tasks:Running task workunit... 2022-05-13T23:16:06.191 INFO:tasks.workunit:Pulling workunits from ref 23b0658391ce94ca40799469eec00c28b298816c 2022-05-13T23:16:06.194 INFO:tasks.workunit:Making a separate scratch dir for every client... 2022-05-13T23:16:06.195 INFO:tasks.workunit:timeout=6h 2022-05-13T23:16:06.197 INFO:tasks.workunit:cleanup=True 2022-05-13T23:16:06.198 DEBUG:teuthology.orchestra.run.smithi002:> stat -- /home/ubuntu/cephtest/mnt.0 2022-05-13T23:16:06.251 INFO:teuthology.orchestra.run.smithi002.stdout: File: /home/ubuntu/cephtest/mnt.0 2022-05-13T23:16:06.252 INFO:teuthology.orchestra.run.smithi002.stdout: Size: 0 Blocks: 0 IO Block: 65536 directory 2022-05-13T23:16:06.252 INFO:teuthology.orchestra.run.smithi002.stdout:Device: 79h/121d Inode: 1 Links: 2 2022-05-13T23:16:06.252 INFO:teuthology.orchestra.run.smithi002.stdout:Access: (1777/drwxrwxrwt) Uid: ( 0/ root) Gid: ( 0/ root) 2022-05-13T23:16:06.252 INFO:teuthology.orchestra.run.smithi002.stdout:Context: system_u:object_r:cephfs_t:s0 2022-05-13T23:16:06.253 INFO:teuthology.orchestra.run.smithi002.stdout:Access: 1970-01-01 00:00:00.000000000 +0000 2022-05-13T23:16:06.253 INFO:teuthology.orchestra.run.smithi002.stdout:Modify: 2022-05-13 23:14:16.241160860 +0000 2022-05-13T23:16:06.253 INFO:teuthology.orchestra.run.smithi002.stdout:Change: 2022-05-13 23:16:05.971917127 +0000 2022-05-13T23:16:06.253 INFO:teuthology.orchestra.run.smithi002.stdout: Birth: - 2022-05-13T23:16:06.254 INFO:tasks.workunit:Did not need to create dir /home/ubuntu/cephtest/mnt.0 2022-05-13T23:16:06.254 DEBUG:teuthology.orchestra.run.smithi002:> cd -- /home/ubuntu/cephtest/mnt.0 && sudo install -d -m 0755 --owner=ubuntu -- client.0 2022-05-13T23:16:06.351 DEBUG:teuthology.orchestra.run.smithi002:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://github.com/ceph/ceph-ci.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout 23b0658391ce94ca40799469eec00c28b298816c 2022-05-13T23:16:06.405 INFO:tasks.workunit.client.0.smithi002.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2022-05-13T23:16:06.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:16:06 smithi047 ceph-mon[77147]: pgmap v345: 97 pgs: 97 active+clean; 584 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 11 op/s 2022-05-13T23:16:06.823 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:16:06 smithi002 ceph-mon[71816]: pgmap v345: 97 pgs: 97 active+clean; 584 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 11 op/s 2022-05-13T23:16:06.862 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:16:06 smithi115 ceph-mon[78593]: pgmap v345: 97 pgs: 97 active+clean; 584 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 11 op/s 2022-05-13T23:16:08.797 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:16:08 smithi047 ceph-mon[77147]: pgmap v346: 97 pgs: 97 active+clean; 584 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2022-05-13T23:16:08.823 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:16:08 smithi002 ceph-mon[71816]: pgmap v346: 97 pgs: 97 active+clean; 584 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2022-05-13T23:16:08.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:16:08 smithi115 ceph-mon[78593]: pgmap v346: 97 pgs: 97 active+clean; 584 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2022-05-13T23:16:10.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:16:10 smithi047 ceph-mon[77147]: pgmap v347: 97 pgs: 97 active+clean; 584 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-05-13T23:16:10.825 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:16:10 smithi002 ceph-mon[71816]: pgmap v347: 97 pgs: 97 active+clean; 584 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-05-13T23:16:10.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:16:10 smithi115 ceph-mon[78593]: pgmap v347: 97 pgs: 97 active+clean; 584 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-05-13T23:16:12.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:16:12 smithi047 ceph-mon[77147]: pgmap v348: 97 pgs: 97 active+clean; 587 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 426 B/s wr, 11 op/s 2022-05-13T23:16:12.822 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:16:12 smithi002 ceph-mon[71816]: pgmap v348: 97 pgs: 97 active+clean; 587 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 426 B/s wr, 11 op/s 2022-05-13T23:16:12.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:16:12 smithi115 ceph-mon[78593]: pgmap v348: 97 pgs: 97 active+clean; 587 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 426 B/s wr, 11 op/s 2022-05-13T23:16:14.778 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:16:14 smithi002 ceph-mon[71816]: pgmap v349: 97 pgs: 97 active+clean; 587 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 511 B/s wr, 9 op/s 2022-05-13T23:16:14.802 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:16:14 smithi047 ceph-mon[77147]: pgmap v349: 97 pgs: 97 active+clean; 587 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 511 B/s wr, 9 op/s 2022-05-13T23:16:14.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:16:14 smithi115 ceph-mon[78593]: pgmap v349: 97 pgs: 97 active+clean; 587 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 511 B/s wr, 9 op/s 2022-05-13T23:16:16.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:16:16 smithi047 ceph-mon[77147]: pgmap v350: 97 pgs: 97 active+clean; 587 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 511 B/s wr, 11 op/s 2022-05-13T23:16:16.823 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:16:16 smithi002 ceph-mon[71816]: pgmap v350: 97 pgs: 97 active+clean; 587 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 511 B/s wr, 11 op/s 2022-05-13T23:16:16.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:16:16 smithi115 ceph-mon[78593]: pgmap v350: 97 pgs: 97 active+clean; 587 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 511 B/s wr, 11 op/s 2022-05-13T23:16:18.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:16:18 smithi047 ceph-mon[77147]: pgmap v351: 97 pgs: 97 active+clean; 587 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 511 B/s wr, 10 op/s 2022-05-13T23:16:18.822 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:16:18 smithi002 ceph-mon[71816]: pgmap v351: 97 pgs: 97 active+clean; 587 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 511 B/s wr, 10 op/s 2022-05-13T23:16:18.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:16:18 smithi115 ceph-mon[78593]: pgmap v351: 97 pgs: 97 active+clean; 587 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 511 B/s wr, 10 op/s 2022-05-13T23:16:20.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:16:20 smithi047 ceph-mon[77147]: pgmap v352: 97 pgs: 97 active+clean; 587 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 511 B/s wr, 9 op/s 2022-05-13T23:16:20.822 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:16:20 smithi002 ceph-mon[71816]: pgmap v352: 97 pgs: 97 active+clean; 587 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 511 B/s wr, 9 op/s 2022-05-13T23:16:20.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:16:20 smithi115 ceph-mon[78593]: pgmap v352: 97 pgs: 97 active+clean; 587 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 511 B/s wr, 9 op/s 2022-05-13T23:16:22.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:16:22 smithi047 ceph-mon[77147]: pgmap v353: 97 pgs: 97 active+clean; 587 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 511 B/s wr, 12 op/s 2022-05-13T23:16:22.824 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:16:22 smithi002 ceph-mon[71816]: pgmap v353: 97 pgs: 97 active+clean; 587 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 511 B/s wr, 12 op/s 2022-05-13T23:16:22.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:16:22 smithi115 ceph-mon[78593]: pgmap v353: 97 pgs: 97 active+clean; 587 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 511 B/s wr, 12 op/s 2022-05-13T23:16:24.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:16:24 smithi047 ceph-mon[77147]: pgmap v354: 97 pgs: 97 active+clean; 587 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 85 B/s wr, 8 op/s 2022-05-13T23:16:24.822 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:16:24 smithi002 ceph-mon[71816]: pgmap v354: 97 pgs: 97 active+clean; 587 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 85 B/s wr, 8 op/s 2022-05-13T23:16:24.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:16:24 smithi115 ceph-mon[78593]: pgmap v354: 97 pgs: 97 active+clean; 587 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 85 B/s wr, 8 op/s 2022-05-13T23:16:26.534 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:16:26 smithi002 ceph-mon[71816]: pgmap v355: 97 pgs: 97 active+clean; 587 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 11 op/s 2022-05-13T23:16:26.535 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:16:26 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-13T23:16:26.535 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:16:26 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-13T23:16:26.535 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:16:26 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-13T23:16:26.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:16:26 smithi047 ceph-mon[77147]: pgmap v355: 97 pgs: 97 active+clean; 587 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 11 op/s 2022-05-13T23:16:26.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:16:26 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-13T23:16:26.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:16:26 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-13T23:16:26.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:16:26 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-13T23:16:26.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:16:26 smithi115 ceph-mon[78593]: pgmap v355: 97 pgs: 97 active+clean; 587 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 11 op/s 2022-05-13T23:16:26.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:16:26 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-13T23:16:26.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:16:26 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-13T23:16:26.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:16:26 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-13T23:16:28.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:16:28 smithi047 ceph-mon[77147]: pgmap v356: 97 pgs: 97 active+clean; 587 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2022-05-13T23:16:28.822 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:16:28 smithi002 ceph-mon[71816]: pgmap v356: 97 pgs: 97 active+clean; 587 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2022-05-13T23:16:28.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:16:28 smithi115 ceph-mon[78593]: pgmap v356: 97 pgs: 97 active+clean; 587 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2022-05-13T23:16:30.073 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:16:29 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:16:30.073 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:16:29 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:16:30.074 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:16:29 smithi002 ceph-mon[71816]: pgmap v357: 97 pgs: 97 active+clean; 587 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-05-13T23:16:30.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:16:29 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:16:30.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:16:29 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:16:30.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:16:29 smithi115 ceph-mon[78593]: pgmap v357: 97 pgs: 97 active+clean; 587 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-05-13T23:16:30.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:16:29 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:16:30.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:16:29 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:16:30.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:16:29 smithi047 ceph-mon[77147]: pgmap v357: 97 pgs: 97 active+clean; 587 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-05-13T23:16:32.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:16:31 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:16:32.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:16:31 smithi047 ceph-mon[77147]: pgmap v358: 97 pgs: 97 active+clean; 587 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2022-05-13T23:16:32.322 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:16:31 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:16:32.323 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:16:31 smithi002 ceph-mon[71816]: pgmap v358: 97 pgs: 97 active+clean; 587 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2022-05-13T23:16:32.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:16:31 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:16:32.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:16:31 smithi115 ceph-mon[78593]: pgmap v358: 97 pgs: 97 active+clean; 587 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2022-05-13T23:16:33.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:16:33 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:16:33.572 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:16:33 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:16:33.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:16:33 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' 2022-05-13T23:16:34.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:16:34 smithi047 ceph-mon[77147]: pgmap v359: 97 pgs: 97 active+clean; 587 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2022-05-13T23:16:34.572 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:16:34 smithi002 ceph-mon[71816]: pgmap v359: 97 pgs: 97 active+clean; 587 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2022-05-13T23:16:34.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:16:34 smithi115 ceph-mon[78593]: pgmap v359: 97 pgs: 97 active+clean; 587 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2022-05-13T23:16:36.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:16:36 smithi115 ceph-mon[78593]: pgmap v360: 97 pgs: 97 active+clean; 587 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 11 op/s 2022-05-13T23:16:36.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:16:36 smithi047 ceph-mon[77147]: pgmap v360: 97 pgs: 97 active+clean; 587 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 11 op/s 2022-05-13T23:16:36.822 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:16:36 smithi002 ceph-mon[71816]: pgmap v360: 97 pgs: 97 active+clean; 587 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 11 op/s 2022-05-13T23:16:38.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:16:38 smithi115 ceph-mon[78593]: pgmap v361: 97 pgs: 97 active+clean; 587 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2022-05-13T23:16:38.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:16:38 smithi047 ceph-mon[77147]: pgmap v361: 97 pgs: 97 active+clean; 587 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2022-05-13T23:16:38.822 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:16:38 smithi002 ceph-mon[71816]: pgmap v361: 97 pgs: 97 active+clean; 587 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2022-05-13T23:16:40.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:16:40 smithi115 ceph-mon[78593]: pgmap v362: 97 pgs: 97 active+clean; 587 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-05-13T23:16:40.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:16:40 smithi047 ceph-mon[77147]: pgmap v362: 97 pgs: 97 active+clean; 587 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-05-13T23:16:40.822 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:16:40 smithi002 ceph-mon[71816]: pgmap v362: 97 pgs: 97 active+clean; 587 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-05-13T23:16:42.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:16:42 smithi115 ceph-mon[78593]: pgmap v363: 97 pgs: 97 active+clean; 589 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 255 B/s wr, 11 op/s 2022-05-13T23:16:42.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:16:42 smithi047 ceph-mon[77147]: pgmap v363: 97 pgs: 97 active+clean; 589 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 255 B/s wr, 11 op/s 2022-05-13T23:16:42.822 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:16:42 smithi002 ceph-mon[71816]: pgmap v363: 97 pgs: 97 active+clean; 589 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 255 B/s wr, 11 op/s 2022-05-13T23:16:44.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:16:44 smithi115 ceph-mon[78593]: pgmap v364: 97 pgs: 97 active+clean; 589 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 341 B/s wr, 9 op/s 2022-05-13T23:16:44.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:16:44 smithi047 ceph-mon[77147]: pgmap v364: 97 pgs: 97 active+clean; 589 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 341 B/s wr, 9 op/s 2022-05-13T23:16:44.822 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:16:44 smithi002 ceph-mon[71816]: pgmap v364: 97 pgs: 97 active+clean; 589 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 341 B/s wr, 9 op/s 2022-05-13T23:16:46.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:16:46 smithi115 ceph-mon[78593]: pgmap v365: 97 pgs: 97 active+clean; 589 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 341 B/s wr, 11 op/s 2022-05-13T23:16:46.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:16:46 smithi047 ceph-mon[77147]: pgmap v365: 97 pgs: 97 active+clean; 589 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 341 B/s wr, 11 op/s 2022-05-13T23:16:46.822 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:16:46 smithi002 ceph-mon[71816]: pgmap v365: 97 pgs: 97 active+clean; 589 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 341 B/s wr, 11 op/s 2022-05-13T23:16:48.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:16:48 smithi115 ceph-mon[78593]: pgmap v366: 97 pgs: 97 active+clean; 589 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 426 B/s wr, 10 op/s 2022-05-13T23:16:48.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:16:48 smithi047 ceph-mon[77147]: pgmap v366: 97 pgs: 97 active+clean; 589 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 426 B/s wr, 10 op/s 2022-05-13T23:16:48.823 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:16:48 smithi002 ceph-mon[71816]: pgmap v366: 97 pgs: 97 active+clean; 589 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 426 B/s wr, 10 op/s 2022-05-13T23:16:49.385 INFO:tasks.workunit.client.0.smithi002.stderr:Updating files: 73% (5018/6793) Updating files: 74% (5027/6793) Updating files: 75% (5095/6793) Updating files: 76% (5163/6793) Updating files: 77% (5231/6793) Updating files: 78% (5299/6793) Updating files: 79% (5367/6793) Updating files: 80% (5435/6793) Updating files: 81% (5503/6793) Updating files: 82% (5571/6793) Updating files: 83% (5639/6793) Updating files: 84% (5707/6793) Updating files: 85% (5775/6793) Updating files: 86% (5842/6793) Updating files: 87% (5910/6793) Updating files: 88% (5978/6793) Updating files: 89% (6046/6793) Updating files: 90% (6114/6793) Updating files: 91% (6182/6793) Updating files: 92% (6250/6793) Updating files: 93% (6318/6793) Updating files: 94% (6386/6793) Updating files: 95% (6454/6793) Updating files: 96% (6522/6793) Updating files: 97% (6590/6793) Updating files: 98% (6658/6793) Updating files: 99% (6726/6793) Updating files: 100% (6793/6793) Updating files: 100% (6793/6793), done. 2022-05-13T23:16:50.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:16:50 smithi115 ceph-mon[78593]: pgmap v367: 97 pgs: 97 active+clean; 589 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 426 B/s wr, 9 op/s 2022-05-13T23:16:50.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:16:50 smithi047 ceph-mon[77147]: pgmap v367: 97 pgs: 97 active+clean; 589 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 426 B/s wr, 9 op/s 2022-05-13T23:16:50.822 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:16:50 smithi002 ceph-mon[71816]: pgmap v367: 97 pgs: 97 active+clean; 589 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 426 B/s wr, 9 op/s 2022-05-13T23:16:52.259 INFO:tasks.workunit.client.0.smithi002.stderr:Note: switching to '23b0658391ce94ca40799469eec00c28b298816c'. 2022-05-13T23:16:52.259 INFO:tasks.workunit.client.0.smithi002.stderr: 2022-05-13T23:16:52.260 INFO:tasks.workunit.client.0.smithi002.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2022-05-13T23:16:52.260 INFO:tasks.workunit.client.0.smithi002.stderr:changes and commit them, and you can discard any commits you make in this 2022-05-13T23:16:52.260 INFO:tasks.workunit.client.0.smithi002.stderr:state without impacting any branches by switching back to a branch. 2022-05-13T23:16:52.260 INFO:tasks.workunit.client.0.smithi002.stderr: 2022-05-13T23:16:52.261 INFO:tasks.workunit.client.0.smithi002.stderr:If you want to create a new branch to retain commits you create, you may 2022-05-13T23:16:52.261 INFO:tasks.workunit.client.0.smithi002.stderr:do so (now or later) by using -c with the switch command. Example: 2022-05-13T23:16:52.261 INFO:tasks.workunit.client.0.smithi002.stderr: 2022-05-13T23:16:52.261 INFO:tasks.workunit.client.0.smithi002.stderr: git switch -c 2022-05-13T23:16:52.261 INFO:tasks.workunit.client.0.smithi002.stderr: 2022-05-13T23:16:52.262 INFO:tasks.workunit.client.0.smithi002.stderr:Or undo this operation with: 2022-05-13T23:16:52.262 INFO:tasks.workunit.client.0.smithi002.stderr: 2022-05-13T23:16:52.264 INFO:tasks.workunit.client.0.smithi002.stderr: git switch - 2022-05-13T23:16:52.265 INFO:tasks.workunit.client.0.smithi002.stderr: 2022-05-13T23:16:52.265 INFO:tasks.workunit.client.0.smithi002.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2022-05-13T23:16:52.265 INFO:tasks.workunit.client.0.smithi002.stderr: 2022-05-13T23:16:52.265 INFO:tasks.workunit.client.0.smithi002.stderr:HEAD is now at 23b0658391c Merge branch 'wip-use-system-libpmem-on-c9s-quincy' of https://github.com/idryomov/ceph into wip-yuri-testing-2022-05-10-1027-quincy 2022-05-13T23:16:52.311 DEBUG:teuthology.orchestra.run.smithi002:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2022-05-13T23:16:52.376 INFO:tasks.workunit.client.0.smithi002.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2022-05-13T23:16:52.382 INFO:tasks.workunit.client.0.smithi002.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2022-05-13T23:16:52.382 INFO:tasks.workunit.client.0.smithi002.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2022-05-13T23:16:52.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:16:52 smithi115 ceph-mon[78593]: pgmap v368: 97 pgs: 97 active+clean; 589 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 426 B/s wr, 12 op/s 2022-05-13T23:16:52.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:16:52 smithi047 ceph-mon[77147]: pgmap v368: 97 pgs: 97 active+clean; 589 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 426 B/s wr, 12 op/s 2022-05-13T23:16:52.838 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:16:52 smithi002 ceph-mon[71816]: pgmap v368: 97 pgs: 97 active+clean; 589 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 426 B/s wr, 12 op/s 2022-05-13T23:16:53.298 INFO:tasks.workunit.client.0.smithi002.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2022-05-13T23:16:53.399 INFO:tasks.workunit.client.0.smithi002.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2022-05-13T23:16:53.481 INFO:tasks.workunit.client.0.smithi002.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2022-05-13T23:16:53.485 INFO:tasks.workunit.client.0.smithi002.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2022-05-13T23:16:53.486 INFO:tasks.workunit.client.0.smithi002.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2022-05-13T23:16:53.566 INFO:tasks.workunit.client.0.smithi002.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2022-05-13T23:16:53.578 DEBUG:teuthology.orchestra.run.smithi002:> set -ex 2022-05-13T23:16:53.579 DEBUG:teuthology.orchestra.run.smithi002:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2022-05-13T23:16:53.628 INFO:tasks.workunit:Running workunits matching suites/fsstress.sh on client.0... 2022-05-13T23:16:53.631 INFO:tasks.workunit:Running workunit suites/fsstress.sh... 2022-05-13T23:16:53.632 DEBUG:teuthology.orchestra.run.smithi002:workunit test suites/fsstress.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=23b0658391ce94ca40799469eec00c28b298816c TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/fsstress.sh 2022-05-13T23:16:53.737 INFO:tasks.workunit.client.0.smithi002.stderr:+ mkdir -p fsstress 2022-05-13T23:16:53.754 INFO:tasks.workunit.client.0.smithi002.stderr:+ pushd fsstress 2022-05-13T23:16:53.754 INFO:tasks.workunit.client.0.smithi002.stdout:~/cephtest/mnt.0/client.0/tmp/fsstress ~/cephtest/mnt.0/client.0/tmp 2022-05-13T23:16:53.755 INFO:tasks.workunit.client.0.smithi002.stderr:+ wget -q -O ltp-full.tgz http://download.ceph.com/qa/ltp-full-20091231.tgz 2022-05-13T23:16:54.382 INFO:tasks.workunit.client.0.smithi002.stderr:+ tar xzf ltp-full.tgz 2022-05-13T23:16:54.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:16:54 smithi047 ceph-mon[77147]: pgmap v369: 97 pgs: 97 active+clean; 589 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 170 B/s wr, 9 op/s 2022-05-13T23:16:54.825 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:16:54 smithi002 ceph-mon[71816]: pgmap v369: 97 pgs: 97 active+clean; 589 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 170 B/s wr, 9 op/s 2022-05-13T23:16:54.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:16:54 smithi115 ceph-mon[78593]: pgmap v369: 97 pgs: 97 active+clean; 589 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 170 B/s wr, 9 op/s 2022-05-13T23:16:56.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:16:56 smithi047 ceph-mon[77147]: pgmap v370: 97 pgs: 97 active+clean; 600 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 1.2 KiB/s wr, 11 op/s 2022-05-13T23:16:56.822 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:16:56 smithi002 ceph-mon[71816]: pgmap v370: 97 pgs: 97 active+clean; 600 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 1.2 KiB/s wr, 11 op/s 2022-05-13T23:16:56.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:16:56 smithi115 ceph-mon[78593]: pgmap v370: 97 pgs: 97 active+clean; 600 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 1.2 KiB/s wr, 11 op/s 2022-05-13T23:16:58.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:16:58 smithi047 ceph-mon[77147]: pgmap v371: 97 pgs: 97 active+clean; 600 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 10 KiB/s wr, 10 op/s 2022-05-13T23:16:58.822 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:16:58 smithi002 ceph-mon[71816]: pgmap v371: 97 pgs: 97 active+clean; 600 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 10 KiB/s wr, 10 op/s 2022-05-13T23:16:58.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:16:58 smithi115 ceph-mon[78593]: pgmap v371: 97 pgs: 97 active+clean; 600 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 10 KiB/s wr, 10 op/s 2022-05-13T23:17:00.796 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:17:00 smithi047 ceph-mon[77147]: pgmap v372: 97 pgs: 97 active+clean; 600 KiB data, 83 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 14 KiB/s wr, 10 op/s 2022-05-13T23:17:00.822 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:17:00 smithi002 ceph-mon[71816]: pgmap v372: 97 pgs: 97 active+clean; 600 KiB data, 83 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 14 KiB/s wr, 10 op/s 2022-05-13T23:17:00.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:17:00 smithi115 ceph-mon[78593]: pgmap v372: 97 pgs: 97 active+clean; 600 KiB data, 83 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 14 KiB/s wr, 10 op/s 2022-05-13T23:17:01.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:17:01 smithi047 ceph-mon[77147]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2022-05-13T23:17:01.822 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:17:01 smithi002 ceph-mon[71816]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2022-05-13T23:17:01.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:17:01 smithi115 ceph-mon[78593]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2022-05-13T23:17:02.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:17:02 smithi047 ceph-mon[77147]: pgmap v373: 97 pgs: 97 active+clean; 21 MiB data, 128 MiB used, 1.0 TiB / 1.0 TiB avail; 688 KiB/s rd, 1.8 MiB/s wr, 65 op/s 2022-05-13T23:17:02.823 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:17:02 smithi002 ceph-mon[71816]: pgmap v373: 97 pgs: 97 active+clean; 21 MiB data, 128 MiB used, 1.0 TiB / 1.0 TiB avail; 688 KiB/s rd, 1.8 MiB/s wr, 65 op/s 2022-05-13T23:17:02.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:17:02 smithi115 ceph-mon[78593]: pgmap v373: 97 pgs: 97 active+clean; 21 MiB data, 128 MiB used, 1.0 TiB / 1.0 TiB avail; 688 KiB/s rd, 1.8 MiB/s wr, 65 op/s 2022-05-13T23:17:04.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:17:04 smithi047 ceph-mon[77147]: pgmap v374: 97 pgs: 97 active+clean; 29 MiB data, 149 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.5 MiB/s wr, 84 op/s 2022-05-13T23:17:04.822 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:17:04 smithi002 ceph-mon[71816]: pgmap v374: 97 pgs: 97 active+clean; 29 MiB data, 149 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.5 MiB/s wr, 84 op/s 2022-05-13T23:17:04.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:17:04 smithi115 ceph-mon[78593]: pgmap v374: 97 pgs: 97 active+clean; 29 MiB data, 149 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.5 MiB/s wr, 84 op/s 2022-05-13T23:17:05.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:17:05 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-05-13T23:17:05.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:17:05 smithi047 ceph-mon[77147]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-05-13T23:17:05.822 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:17:05 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-05-13T23:17:05.823 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:17:05 smithi002 ceph-mon[71816]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-05-13T23:17:05.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:17:05 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-05-13T23:17:05.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:17:05 smithi115 ceph-mon[78593]: from='mgr.14148 172.21.15.2:0/3919260873' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-05-13T23:17:06.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:17:06 smithi047 ceph-mon[77147]: pgmap v375: 97 pgs: 97 active+clean; 31 MiB data, 178 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.7 MiB/s wr, 136 op/s 2022-05-13T23:17:06.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:17:06 smithi047 ceph-mon[77147]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2022-05-13T23:17:06.823 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:17:06 smithi002 ceph-mon[71816]: pgmap v375: 97 pgs: 97 active+clean; 31 MiB data, 178 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.7 MiB/s wr, 136 op/s 2022-05-13T23:17:06.823 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:17:06 smithi002 ceph-mon[71816]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2022-05-13T23:17:06.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:17:06 smithi115 ceph-mon[78593]: pgmap v375: 97 pgs: 97 active+clean; 31 MiB data, 178 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.7 MiB/s wr, 136 op/s 2022-05-13T23:17:06.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:17:06 smithi115 ceph-mon[78593]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2022-05-13T23:17:08.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:17:08 smithi047 ceph-mon[77147]: pgmap v376: 97 pgs: 97 active+clean; 41 MiB data, 233 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 3.6 MiB/s wr, 288 op/s 2022-05-13T23:17:08.822 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:17:08 smithi002 ceph-mon[71816]: pgmap v376: 97 pgs: 97 active+clean; 41 MiB data, 233 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 3.6 MiB/s wr, 288 op/s 2022-05-13T23:17:08.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:17:08 smithi115 ceph-mon[78593]: pgmap v376: 97 pgs: 97 active+clean; 41 MiB data, 233 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 3.6 MiB/s wr, 288 op/s 2022-05-13T23:17:10.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:17:10 smithi047 ceph-mon[77147]: pgmap v377: 97 pgs: 97 active+clean; 44 MiB data, 250 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 3.9 MiB/s wr, 325 op/s 2022-05-13T23:17:10.822 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:17:10 smithi002 ceph-mon[71816]: pgmap v377: 97 pgs: 97 active+clean; 44 MiB data, 250 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 3.9 MiB/s wr, 325 op/s 2022-05-13T23:17:10.862 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:17:10 smithi115 ceph-mon[78593]: pgmap v377: 97 pgs: 97 active+clean; 44 MiB data, 250 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 3.9 MiB/s wr, 325 op/s 2022-05-13T23:17:11.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:17:11 smithi047 ceph-mon[77147]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2022-05-13T23:17:11.823 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:17:11 smithi002 ceph-mon[71816]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2022-05-13T23:17:11.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:17:11 smithi115 ceph-mon[78593]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2022-05-13T23:17:12.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:17:12 smithi047 ceph-mon[77147]: pgmap v378: 97 pgs: 97 active+clean; 59 MiB data, 315 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 5.2 MiB/s wr, 421 op/s 2022-05-13T23:17:12.823 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:17:12 smithi002 ceph-mon[71816]: pgmap v378: 97 pgs: 97 active+clean; 59 MiB data, 315 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 5.2 MiB/s wr, 421 op/s 2022-05-13T23:17:12.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:17:12 smithi115 ceph-mon[78593]: pgmap v378: 97 pgs: 97 active+clean; 59 MiB data, 315 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 5.2 MiB/s wr, 421 op/s 2022-05-13T23:17:13.546 INFO:tasks.workunit.client.0.smithi002.stderr:+ pushd ltp-full-20091231/testcases/kernel/fs/fsstress 2022-05-13T23:17:13.547 INFO:tasks.workunit.client.0.smithi002.stdout:~/cephtest/mnt.0/client.0/tmp/fsstress/ltp-full-20091231/testcases/kernel/fs/fsstress ~/cephtest/mnt.0/client.0/tmp/fsstress ~/cephtest/mnt.0/client.0/tmp 2022-05-13T23:17:13.547 INFO:tasks.workunit.client.0.smithi002.stderr:+ make 2022-05-13T23:17:13.604 INFO:tasks.workunit.client.0.smithi002.stdout:cc -DNO_XFS -I/home/ubuntu/cephtest/mnt.0/client.0/tmp/fsstress/ltp-full-20091231/testcases/kernel/fs/fsstress -D_LARGEFILE64_SOURCE -D_GNU_SOURCE -I../../../../include -I../../../../include -L../../../../lib fsstress.c -o fsstress 2022-05-13T23:17:14.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:17:14 smithi047 ceph-mon[77147]: pgmap v379: 97 pgs: 97 active+clean; 66 MiB data, 323 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 4.0 MiB/s wr, 375 op/s 2022-05-13T23:17:14.822 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:17:14 smithi002 ceph-mon[71816]: pgmap v379: 97 pgs: 97 active+clean; 66 MiB data, 323 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 4.0 MiB/s wr, 375 op/s 2022-05-13T23:17:14.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:17:14 smithi115 ceph-mon[78593]: pgmap v379: 97 pgs: 97 active+clean; 66 MiB data, 323 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 4.0 MiB/s wr, 375 op/s 2022-05-13T23:17:15.393 INFO:tasks.workunit.client.0.smithi002.stderr:++ readlink -f fsstress 2022-05-13T23:17:15.406 INFO:tasks.workunit.client.0.smithi002.stderr:+ BIN=/home/ubuntu/cephtest/mnt.0/client.0/tmp/fsstress/ltp-full-20091231/testcases/kernel/fs/fsstress/fsstress 2022-05-13T23:17:15.406 INFO:tasks.workunit.client.0.smithi002.stderr:+ popd 2022-05-13T23:17:15.407 INFO:tasks.workunit.client.0.smithi002.stdout:~/cephtest/mnt.0/client.0/tmp/fsstress ~/cephtest/mnt.0/client.0/tmp 2022-05-13T23:17:15.407 INFO:tasks.workunit.client.0.smithi002.stderr:+ popd 2022-05-13T23:17:15.408 INFO:tasks.workunit.client.0.smithi002.stdout:~/cephtest/mnt.0/client.0/tmp 2022-05-13T23:17:15.408 INFO:tasks.workunit.client.0.smithi002.stderr:++ mktemp -d -p . 2022-05-13T23:17:15.409 INFO:tasks.workunit.client.0.smithi002.stderr:+ T=./tmp.A70ob8HHEM 2022-05-13T23:17:15.409 INFO:tasks.workunit.client.0.smithi002.stderr:+ /home/ubuntu/cephtest/mnt.0/client.0/tmp/fsstress/ltp-full-20091231/testcases/kernel/fs/fsstress/fsstress -d ./tmp.A70ob8HHEM -l 1 -n 1000 -p 10 -v 2022-05-13T23:17:15.412 INFO:tasks.workunit.client.0.smithi002.stdout:seed = 1652076100 2022-05-13T23:17:15.416 INFO:tasks.workunit.client.0.smithi002.stdout:0/0: dread - no filename 2022-05-13T23:17:15.416 INFO:tasks.workunit.client.0.smithi002.stdout:0/1: dread - no filename 2022-05-13T23:17:15.419 INFO:tasks.workunit.client.0.smithi002.stdout:5/0: dread - no filename 2022-05-13T23:17:15.419 INFO:tasks.workunit.client.0.smithi002.stdout:5/1: dread - no filename 2022-05-13T23:17:15.420 INFO:tasks.workunit.client.0.smithi002.stdout:5/2: getdents . 0 2022-05-13T23:17:15.421 INFO:tasks.workunit.client.0.smithi002.stdout:5/3: dread - no filename 2022-05-13T23:17:15.421 INFO:tasks.workunit.client.0.smithi002.stdout:5/4: chown . 1057652 1 2022-05-13T23:17:15.421 INFO:tasks.workunit.client.0.smithi002.stdout:5/5: write - no filename 2022-05-13T23:17:15.421 INFO:tasks.workunit.client.0.smithi002.stdout:0/2: mkdir d0 0 2022-05-13T23:17:15.422 INFO:tasks.workunit.client.0.smithi002.stdout:0/3: chown d0 11203379 1 2022-05-13T23:17:15.422 INFO:tasks.workunit.client.0.smithi002.stdout:5/6: mkdir d0 0 2022-05-13T23:17:15.422 INFO:tasks.workunit.client.0.smithi002.stdout:5/7: dread - no filename 2022-05-13T23:17:15.422 INFO:tasks.workunit.client.0.smithi002.stdout:5/8: dread - no filename 2022-05-13T23:17:15.423 INFO:tasks.workunit.client.0.smithi002.stdout:5/9: write - no filename 2022-05-13T23:17:15.423 INFO:tasks.workunit.client.0.smithi002.stdout:5/10: truncate - no filename 2022-05-13T23:17:15.423 INFO:tasks.workunit.client.0.smithi002.stdout:5/11: write - no filename 2022-05-13T23:17:15.423 INFO:tasks.workunit.client.0.smithi002.stdout:5/12: dread - no filename 2022-05-13T23:17:15.423 INFO:tasks.workunit.client.0.smithi002.stdout:0/4: creat d0/f1 x:0 0 0 2022-05-13T23:17:15.424 INFO:tasks.workunit.client.0.smithi002.stdout:0/5: chown d0/f1 35 1 2022-05-13T23:17:15.425 INFO:tasks.workunit.client.0.smithi002.stdout:2/0: unlink - no file 2022-05-13T23:17:15.428 INFO:tasks.workunit.client.0.smithi002.stdout:1/0: mknod c0 0 2022-05-13T23:17:15.435 INFO:tasks.workunit.client.0.smithi002.stdout:5/13: creat d0/f1 x:0 0 0 2022-05-13T23:17:15.435 INFO:tasks.workunit.client.0.smithi002.stdout:5/14: readlink - no filename 2022-05-13T23:17:15.435 INFO:tasks.workunit.client.0.smithi002.stdout:0/6: symlink d0/l2 0 2022-05-13T23:17:15.435 INFO:tasks.workunit.client.0.smithi002.stdout:3/0: dwrite - no filename 2022-05-13T23:17:15.436 INFO:tasks.workunit.client.0.smithi002.stdout:3/1: dread - no filename 2022-05-13T23:17:15.439 INFO:tasks.workunit.client.0.smithi002.stdout:3/2: chown . 43 1 2022-05-13T23:17:15.439 INFO:tasks.workunit.client.0.smithi002.stdout:2/1: symlink l0 0 2022-05-13T23:17:15.439 INFO:tasks.workunit.client.0.smithi002.stdout:2/2: dwrite - no filename 2022-05-13T23:17:15.439 INFO:tasks.workunit.client.0.smithi002.stdout:2/3: dread - no filename 2022-05-13T23:17:15.441 INFO:tasks.workunit.client.0.smithi002.stdout:1/1: unlink c0 0 2022-05-13T23:17:15.441 INFO:tasks.workunit.client.0.smithi002.stdout:1/2: unlink - no file 2022-05-13T23:17:15.441 INFO:tasks.workunit.client.0.smithi002.stdout:1/3: rmdir - no directory 2022-05-13T23:17:15.441 INFO:tasks.workunit.client.0.smithi002.stdout:1/4: readlink - no filename 2022-05-13T23:17:15.442 INFO:tasks.workunit.client.0.smithi002.stdout:1/5: chown . 1 1 2022-05-13T23:17:15.442 INFO:tasks.workunit.client.0.smithi002.stdout:1/6: dwrite - no filename 2022-05-13T23:17:15.442 INFO:tasks.workunit.client.0.smithi002.stdout:1/7: dread - no filename 2022-05-13T23:17:15.442 INFO:tasks.workunit.client.0.smithi002.stdout:1/8: dwrite - no filename 2022-05-13T23:17:15.443 INFO:tasks.workunit.client.0.smithi002.stdout:1/9: rmdir - no directory 2022-05-13T23:17:15.443 INFO:tasks.workunit.client.0.smithi002.stdout:1/10: read - no filename 2022-05-13T23:17:15.443 INFO:tasks.workunit.client.0.smithi002.stdout:4/0: chown . 0 1 2022-05-13T23:17:15.443 INFO:tasks.workunit.client.0.smithi002.stdout:4/1: dread - no filename 2022-05-13T23:17:15.443 INFO:tasks.workunit.client.0.smithi002.stdout:3/3: symlink l0 0 2022-05-13T23:17:15.444 INFO:tasks.workunit.client.0.smithi002.stdout:2/4: mkdir d1 0 2022-05-13T23:17:15.444 INFO:tasks.workunit.client.0.smithi002.stdout:2/5: write - no filename 2022-05-13T23:17:15.444 INFO:tasks.workunit.client.0.smithi002.stdout:2/6: readlink l0 0 2022-05-13T23:17:15.444 INFO:tasks.workunit.client.0.smithi002.stdout:2/7: dread - no filename 2022-05-13T23:17:15.444 INFO:tasks.workunit.client.0.smithi002.stdout:2/8: write - no filename 2022-05-13T23:17:15.445 INFO:tasks.workunit.client.0.smithi002.stdout:5/15: getdents d0 0 2022-05-13T23:17:15.445 INFO:tasks.workunit.client.0.smithi002.stdout:0/7: rmdir d0 39 2022-05-13T23:17:15.446 INFO:tasks.workunit.client.0.smithi002.stdout:4/2: mknod c0 0 2022-05-13T23:17:15.447 INFO:tasks.workunit.client.0.smithi002.stdout:4/3: dread - no filename 2022-05-13T23:17:15.447 INFO:tasks.workunit.client.0.smithi002.stdout:4/4: chown c0 6111773 1 2022-05-13T23:17:15.447 INFO:tasks.workunit.client.0.smithi002.stdout:4/5: write - no filename 2022-05-13T23:17:15.448 INFO:tasks.workunit.client.0.smithi002.stdout:3/4: creat f1 x:0 0 0 2022-05-13T23:17:15.448 INFO:tasks.workunit.client.0.smithi002.stdout:2/9: creat d1/f2 x:0 0 0 2022-05-13T23:17:15.448 INFO:tasks.workunit.client.0.smithi002.stdout:1/11: getdents . 0 2022-05-13T23:17:15.448 INFO:tasks.workunit.client.0.smithi002.stdout:1/12: read - no filename 2022-05-13T23:17:15.449 INFO:tasks.workunit.client.0.smithi002.stdout:5/16: symlink d0/l2 0 2022-05-13T23:17:15.449 INFO:tasks.workunit.client.0.smithi002.stdout:0/8: link d0/f1 d0/f3 0 2022-05-13T23:17:15.452 INFO:tasks.workunit.client.0.smithi002.stdout:6/0: creat f0 x:0 0 0 2022-05-13T23:17:15.452 INFO:tasks.workunit.client.0.smithi002.stdout:6/1: chown f0 15556 1 2022-05-13T23:17:15.453 INFO:tasks.workunit.client.0.smithi002.stdout:4/6: mkdir d1 0 2022-05-13T23:17:15.454 INFO:tasks.workunit.client.0.smithi002.stdout:3/5: rename l0 to l2 0 2022-05-13T23:17:15.456 INFO:tasks.workunit.client.0.smithi002.stdout:2/10: mkdir d1/d3 0 2022-05-13T23:17:15.456 INFO:tasks.workunit.client.0.smithi002.stdout:2/11: getdents d1/d3 0 2022-05-13T23:17:15.456 INFO:tasks.workunit.client.0.smithi002.stdout:1/13: symlink l1 0 2022-05-13T23:17:15.457 INFO:tasks.workunit.client.0.smithi002.stdout:1/14: dwrite - no filename 2022-05-13T23:17:15.458 INFO:tasks.workunit.client.0.smithi002.stdout:5/17: write d0/f1 [824352,110434] 0 2022-05-13T23:17:15.458 INFO:tasks.workunit.client.0.smithi002.stdout:5/18: chown d0/l2 4389408 1 2022-05-13T23:17:15.459 INFO:tasks.workunit.client.0.smithi002.stdout:0/9: creat d0/f4 x:0 0 0 2022-05-13T23:17:15.460 INFO:tasks.workunit.client.0.smithi002.stdout:0/10: chown d0 5574424 1 2022-05-13T23:17:15.461 INFO:tasks.workunit.client.0.smithi002.stdout:8/0: unlink - no file 2022-05-13T23:17:15.462 INFO:tasks.workunit.client.0.smithi002.stdout:8/1: write - no filename 2022-05-13T23:17:15.463 INFO:tasks.workunit.client.0.smithi002.stdout:7/0: creat f0 x:0 0 0 2022-05-13T23:17:15.463 INFO:tasks.workunit.client.0.smithi002.stdout:7/1: creat f1 x:0 0 0 2022-05-13T23:17:15.466 INFO:tasks.workunit.client.0.smithi002.stdout:4/7: unlink c0 0 2022-05-13T23:17:15.470 INFO:tasks.workunit.client.0.smithi002.stdout:3/6: symlink l3 0 2022-05-13T23:17:15.477 INFO:tasks.workunit.client.0.smithi002.stdout:2/12: mknod d1/c4 0 2022-05-13T23:17:15.478 INFO:tasks.workunit.client.0.smithi002.stdout:2/13: creat d1/f5 x:0 0 0 2022-05-13T23:17:15.478 INFO:tasks.workunit.client.0.smithi002.stdout:2/14: truncate d1/f5 61196 0 2022-05-13T23:17:15.488 INFO:tasks.workunit.client.0.smithi002.stdout:5/19: creat d0/f3 x:0 0 0 2022-05-13T23:17:15.489 INFO:tasks.workunit.client.0.smithi002.stdout:8/2: symlink l0 0 2022-05-13T23:17:15.489 INFO:tasks.workunit.client.0.smithi002.stdout:9/0: chown . 7246 1 2022-05-13T23:17:15.490 INFO:tasks.workunit.client.0.smithi002.stdout:7/2: rename f0 to f2 0 2022-05-13T23:17:15.496 INFO:tasks.workunit.client.0.smithi002.stdout:4/8: mkdir d1/d2 0 2022-05-13T23:17:15.496 INFO:tasks.workunit.client.0.smithi002.stdout:4/9: rename d1 to d1/d3 22 2022-05-13T23:17:15.522 INFO:tasks.workunit.client.0.smithi002.stdout:2/15: creat d1/d3/f6 x:0 0 0 2022-05-13T23:17:15.523 INFO:tasks.workunit.client.0.smithi002.stdout:1/15: mkdir d2 0 2022-05-13T23:17:15.535 INFO:tasks.workunit.client.0.smithi002.stdout:8/3: mknod c1 0 2022-05-13T23:17:15.536 INFO:tasks.workunit.client.0.smithi002.stdout:9/1: creat f0 x:0 0 0 2022-05-13T23:17:15.543 INFO:tasks.workunit.client.0.smithi002.stdout:4/10: creat d1/d2/f4 x:0 0 0 2022-05-13T23:17:15.745 INFO:tasks.workunit.client.0.smithi002.stdout:1/16: creat d2/f3 x:0 0 0 2022-05-13T23:17:15.745 INFO:tasks.workunit.client.0.smithi002.stdout:1/17: creat d2/f4 x:0 0 0 2022-05-13T23:17:15.745 INFO:tasks.workunit.client.0.smithi002.stdout:1/18: truncate d2/f4 436910 0 2022-05-13T23:17:15.751 INFO:tasks.workunit.client.0.smithi002.stdout:5/20: rename d0/f1 to d0/f4 0 2022-05-13T23:17:15.767 INFO:tasks.workunit.client.0.smithi002.stdout:4/11: fsync d1/d2/f4 0 2022-05-13T23:17:15.767 INFO:tasks.workunit.client.0.smithi002.stdout:4/12: creat d1/d2/f5 x:0 0 0 2022-05-13T23:17:15.777 INFO:tasks.workunit.client.0.smithi002.stdout:2/16: dread d1/f5 [0,4194304] 0 2022-05-13T23:17:15.809 INFO:tasks.workunit.client.0.smithi002.stdout:3/7: dwrite f1 [0,4194304] 0 2022-05-13T23:17:15.860 INFO:tasks.workunit.client.0.smithi002.stdout:8/4: symlink l2 0 2022-05-13T23:17:15.860 INFO:tasks.workunit.client.0.smithi002.stdout:8/5: write - no filename 2022-05-13T23:17:15.895 INFO:tasks.workunit.client.0.smithi002.stdout:7/3: getdents . 0 2022-05-13T23:17:15.929 INFO:tasks.workunit.client.0.smithi002.stdout:1/19: dwrite d2/f4 [0,4194304] 0 2022-05-13T23:17:15.929 INFO:tasks.workunit.client.0.smithi002.stdout:1/20: creat d2/f5 x:0 0 0 2022-05-13T23:17:15.960 INFO:tasks.workunit.client.0.smithi002.stdout:5/21: dwrite d0/f3 [0,4194304] 0 2022-05-13T23:17:15.961 INFO:tasks.workunit.client.0.smithi002.stdout:5/22: stat d0/f4 0 2022-05-13T23:17:16.022 INFO:tasks.workunit.client.0.smithi002.stdout:1/21: dread d2/f4 [0,4194304] 0 2022-05-13T23:17:16.022 INFO:tasks.workunit.client.0.smithi002.stdout:1/22: fdatasync d2/f4 0 2022-05-13T23:17:16.022 INFO:tasks.workunit.client.0.smithi002.stdout:1/23: creat d2/f6 x:0 0 0 2022-05-13T23:17:16.023 INFO:tasks.workunit.client.0.smithi002.stdout:1/24: readlink l1 0 2022-05-13T23:17:16.053 INFO:tasks.workunit.client.0.smithi002.stdout:1/25: write d2/f4 [2127137,47719] 0 2022-05-13T23:17:16.164 INFO:tasks.workunit.client.0.smithi002.stdout:0/11: dwrite d0/f3 [0,4194304] 0 2022-05-13T23:17:16.459 INFO:tasks.workunit.client.0.smithi002.stdout:2/17: dwrite d1/d3/f6 [0,4194304] 0 2022-05-13T23:17:16.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:17:16 smithi047 ceph-mon[77147]: pgmap v380: 97 pgs: 97 active+clean; 69 MiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 3.5 MiB/s wr, 358 op/s 2022-05-13T23:17:16.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:17:16 smithi047 ceph-mon[77147]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2022-05-13T23:17:16.822 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:17:16 smithi002 ceph-mon[71816]: pgmap v380: 97 pgs: 97 active+clean; 69 MiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 3.5 MiB/s wr, 358 op/s 2022-05-13T23:17:16.823 INFO:journalctl@ceph.mon.a.smithi002.stdout:May 13 23:17:16 smithi002 ceph-mon[71816]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2022-05-13T23:17:16.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:17:16 smithi115 ceph-mon[78593]: pgmap v380: 97 pgs: 97 active+clean; 69 MiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 3.5 MiB/s wr, 358 op/s 2022-05-13T23:17:16.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:17:16 smithi115 ceph-mon[78593]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2022-05-13T23:17:37.542 INFO:journalctl@ceph.osd.10.smithi047.stdout:May 13 23:17:37 smithi047 conmon[100262]: 2022-05-13T23:17:37.133+0000 7f5b4f799700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2022-05-13T23:17:28.666723+0000 front 2022-05-13T23:17:10.965064+0000 (oldest deadline 2022-05-13T23:17:36.864957+0000) 2022-05-13T23:17:37.543 INFO:journalctl@ceph.osd.10.smithi047.stdout:May 13 23:17:37 smithi047 conmon[100262]: 2022-05-13T23:17:37.133+0000 7f5b4f799700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.2:6830 osd.9 since back 2022-05-13T23:17:28.667102+0000 front 2022-05-13T23:17:10.965263+0000 (oldest deadline 2022-05-13T23:17:36.864957+0000) 2022-05-13T23:17:37.858 INFO:journalctl@ceph.osd.8.smithi115.stdout:May 13 23:17:37 smithi115 conmon[96252]: 2022-05-13T23:17:37.532+0000 7f225f807700 -1 osd.8 85 heartbeat_check: no reply from 172.21.15.2:6830 osd.9 since back 2022-05-13T23:17:15.136602+0000 front 2022-05-13T23:17:29.739512+0000 (oldest deadline 2022-05-13T23:17:36.836498+0000) 2022-05-13T23:17:38.042 INFO:journalctl@ceph.osd.1.smithi047.stdout:May 13 23:17:37 smithi047 conmon[82670]: 2022-05-13T23:17:37.681+0000 7f9bceb39700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.2:6822 osd.6 since back 2022-05-13T23:17:19.858121+0000 front 2022-05-13T23:17:16.457356+0000 (oldest deadline 2022-05-13T23:17:36.956651+0000) 2022-05-13T23:17:38.456 INFO:journalctl@ceph.osd.10.smithi047.stdout:May 13 23:17:38 smithi047 conmon[100262]: 2022-05-13T23:17:38.101+0000 7f5b4f799700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2022-05-13T23:17:28.666723+0000 front 2022-05-13T23:17:10.965064+0000 (oldest deadline 2022-05-13T23:17:36.864957+0000) 2022-05-13T23:17:38.457 INFO:journalctl@ceph.osd.10.smithi047.stdout:May 13 23:17:38 smithi047 conmon[100262]: 2022-05-13T23:17:38.101+0000 7f5b4f799700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.2:6830 osd.9 since back 2022-05-13T23:17:28.667102+0000 front 2022-05-13T23:17:10.965263+0000 (oldest deadline 2022-05-13T23:17:36.864957+0000) 2022-05-13T23:17:38.457 INFO:journalctl@ceph.osd.11.smithi115.stdout:May 13 23:17:38 smithi115 conmon[102145]: 2022-05-13T23:17:38.162+0000 7f98de059700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2022-05-13T23:17:16.234161+0000 front 2022-05-13T23:17:16.357197+0000 (oldest deadline 2022-05-13T23:17:37.333596+0000) 2022-05-13T23:17:38.458 INFO:journalctl@ceph.osd.11.smithi115.stdout:May 13 23:17:38 smithi115 conmon[102145]: 2022-05-13T23:17:38.162+0000 7f98de059700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.2:6830 osd.9 since back 2022-05-13T23:17:20.735459+0000 front 2022-05-13T23:17:16.233824+0000 (oldest deadline 2022-05-13T23:17:37.333596+0000) 2022-05-13T23:17:38.710 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:17:38 smithi047 ceph-mon[77147]: pgmap v381: 97 pgs: 97 active+clean; 86 MiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 4.8 MiB/s wr, 424 op/s 2022-05-13T23:17:38.858 INFO:journalctl@ceph.osd.8.smithi115.stdout:May 13 23:17:38 smithi115 conmon[96252]: 2022-05-13T23:17:38.538+0000 7f225f807700 -1 osd.8 85 heartbeat_check: no reply from 172.21.15.2:6830 osd.9 since back 2022-05-13T23:17:15.136602+0000 front 2022-05-13T23:17:29.739512+0000 (oldest deadline 2022-05-13T23:17:36.836498+0000) 2022-05-13T23:17:38.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:17:38 smithi115 ceph-mon[78593]: pgmap v381: 97 pgs: 97 active+clean; 86 MiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 4.8 MiB/s wr, 424 op/s 2022-05-13T23:17:39.041 INFO:journalctl@ceph.osd.1.smithi047.stdout:May 13 23:17:38 smithi047 conmon[82670]: 2022-05-13T23:17:38.709+0000 7f9bceb39700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.2:6822 osd.6 since back 2022-05-13T23:17:19.858121+0000 front 2022-05-13T23:17:16.457356+0000 (oldest deadline 2022-05-13T23:17:36.956651+0000) 2022-05-13T23:17:39.458 INFO:journalctl@ceph.osd.10.smithi047.stdout:May 13 23:17:39 smithi047 conmon[100262]: 2022-05-13T23:17:39.093+0000 7f5b4f799700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2022-05-13T23:17:28.666723+0000 front 2022-05-13T23:17:10.965064+0000 (oldest deadline 2022-05-13T23:17:36.864957+0000) 2022-05-13T23:17:39.458 INFO:journalctl@ceph.osd.10.smithi047.stdout:May 13 23:17:39 smithi047 conmon[100262]: 2022-05-13T23:17:39.093+0000 7f5b4f799700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.2:6830 osd.9 since back 2022-05-13T23:17:28.667102+0000 front 2022-05-13T23:17:10.965263+0000 (oldest deadline 2022-05-13T23:17:36.864957+0000) 2022-05-13T23:17:39.459 INFO:journalctl@ceph.osd.11.smithi115.stdout:May 13 23:17:39 smithi115 conmon[102145]: 2022-05-13T23:17:39.189+0000 7f98de059700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2022-05-13T23:17:16.234161+0000 front 2022-05-13T23:17:16.357197+0000 (oldest deadline 2022-05-13T23:17:37.333596+0000) 2022-05-13T23:17:39.459 INFO:journalctl@ceph.osd.11.smithi115.stdout:May 13 23:17:39 smithi115 conmon[102145]: 2022-05-13T23:17:39.189+0000 7f98de059700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.2:6830 osd.9 since back 2022-05-13T23:17:20.735459+0000 front 2022-05-13T23:17:16.233824+0000 (oldest deadline 2022-05-13T23:17:37.333596+0000) 2022-05-13T23:17:39.791 INFO:journalctl@ceph.osd.1.smithi047.stdout:May 13 23:17:39 smithi047 conmon[82670]: 2022-05-13T23:17:39.664+0000 7f9bceb39700 -1 osd.1 85 heartbeat_check: no reply from 172.21.15.2:6822 osd.6 since back 2022-05-13T23:17:19.858121+0000 front 2022-05-13T23:17:16.457356+0000 (oldest deadline 2022-05-13T23:17:36.956651+0000) 2022-05-13T23:17:39.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:17:39 smithi047 ceph-mon[77147]: mon.c calling monitor election 2022-05-13T23:17:39.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:17:39 smithi047 ceph-mon[77147]: mon.b calling monitor election 2022-05-13T23:17:39.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:17:39 smithi047 ceph-mon[77147]: mon.b calling monitor election 2022-05-13T23:17:39.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:17:39 smithi047 ceph-mon[77147]: mon.b is new leader, mons b,c in quorum (ranks 1,2) 2022-05-13T23:17:39.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:17:39 smithi047 ceph-mon[77147]: monmap e3: 3 mons at {a=[v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0],b=[v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0],c=[v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0]} 2022-05-13T23:17:39.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:17:39 smithi047 ceph-mon[77147]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2022-05-13T23:17:39.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:17:39 smithi047 ceph-mon[77147]: osdmap e85: 12 total, 12 up, 12 in 2022-05-13T23:17:39.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:17:39 smithi047 ceph-mon[77147]: mgrmap e16: x(active, since 10m), standbys: y, z 2022-05-13T23:17:39.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:17:39 smithi047 ceph-mon[77147]: Health check failed: 1/3 mons down, quorum b,c (MON_DOWN) 2022-05-13T23:17:39.795 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:17:39 smithi047 ceph-mon[77147]: osd.6 reported failed by osd.1 2022-05-13T23:17:39.795 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:17:39 smithi047 ceph-mon[77147]: osd.0 reported failed by osd.10 2022-05-13T23:17:39.795 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:17:39 smithi047 ceph-mon[77147]: osd.9 reported failed by osd.10 2022-05-13T23:17:39.795 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:17:39 smithi047 ceph-mon[77147]: Health detail: HEALTH_WARN 1/3 mons down, quorum b,c 2022-05-13T23:17:39.796 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:17:39 smithi047 ceph-mon[77147]: [WRN] MON_DOWN: 1/3 mons down, quorum b,c 2022-05-13T23:17:39.796 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:17:39 smithi047 ceph-mon[77147]: mon.a (rank 0) addr [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] is down (out of quorum) 2022-05-13T23:17:39.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:17:39 smithi115 conmon[90393]: 2022-05-13T23:17:39.569+0000 7fa085a08700 -1 osd.5 85 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2022-05-13T23:17:16.437333+0000 front 2022-05-13T23:17:21.637869+0000 (oldest deadline 2022-05-13T23:17:39.336666+0000) 2022-05-13T23:17:39.859 INFO:journalctl@ceph.osd.8.smithi115.stdout:May 13 23:17:39 smithi115 conmon[96252]: 2022-05-13T23:17:39.578+0000 7f225f807700 -1 osd.8 85 heartbeat_check: no reply from 172.21.15.2:6830 osd.9 since back 2022-05-13T23:17:15.136602+0000 front 2022-05-13T23:17:29.739512+0000 (oldest deadline 2022-05-13T23:17:36.836498+0000) 2022-05-13T23:17:39.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:17:39 smithi115 ceph-mon[78593]: mon.c calling monitor election 2022-05-13T23:17:39.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:17:39 smithi115 ceph-mon[78593]: mon.b calling monitor election 2022-05-13T23:17:39.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:17:39 smithi115 ceph-mon[78593]: mon.b calling monitor election 2022-05-13T23:17:39.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:17:39 smithi115 ceph-mon[78593]: mon.b is new leader, mons b,c in quorum (ranks 1,2) 2022-05-13T23:17:39.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:17:39 smithi115 ceph-mon[78593]: monmap e3: 3 mons at {a=[v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0],b=[v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0],c=[v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0]} 2022-05-13T23:17:39.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:17:39 smithi115 ceph-mon[78593]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2022-05-13T23:17:39.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:17:39 smithi115 ceph-mon[78593]: osdmap e85: 12 total, 12 up, 12 in 2022-05-13T23:17:39.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:17:39 smithi115 ceph-mon[78593]: mgrmap e16: x(active, since 10m), standbys: y, z 2022-05-13T23:17:39.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:17:39 smithi115 ceph-mon[78593]: Health check failed: 1/3 mons down, quorum b,c (MON_DOWN) 2022-05-13T23:17:39.862 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:17:39 smithi115 ceph-mon[78593]: osd.6 reported failed by osd.1 2022-05-13T23:17:39.862 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:17:39 smithi115 ceph-mon[78593]: osd.0 reported failed by osd.10 2022-05-13T23:17:39.862 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:17:39 smithi115 ceph-mon[78593]: osd.9 reported failed by osd.10 2022-05-13T23:17:39.862 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:17:39 smithi115 ceph-mon[78593]: Health detail: HEALTH_WARN 1/3 mons down, quorum b,c 2022-05-13T23:17:39.863 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:17:39 smithi115 ceph-mon[78593]: [WRN] MON_DOWN: 1/3 mons down, quorum b,c 2022-05-13T23:17:39.863 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:17:39 smithi115 ceph-mon[78593]: mon.a (rank 0) addr [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] is down (out of quorum) 2022-05-13T23:17:40.460 INFO:journalctl@ceph.osd.10.smithi047.stdout:May 13 23:17:40 smithi047 conmon[100262]: 2022-05-13T23:17:40.054+0000 7f5b4f799700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2022-05-13T23:17:28.666723+0000 front 2022-05-13T23:17:10.965064+0000 (oldest deadline 2022-05-13T23:17:36.864957+0000) 2022-05-13T23:17:40.460 INFO:journalctl@ceph.osd.10.smithi047.stdout:May 13 23:17:40 smithi047 conmon[100262]: 2022-05-13T23:17:40.054+0000 7f5b4f799700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.2:6830 osd.9 since back 2022-05-13T23:17:28.667102+0000 front 2022-05-13T23:17:10.965263+0000 (oldest deadline 2022-05-13T23:17:36.864957+0000) 2022-05-13T23:17:40.608 INFO:journalctl@ceph.osd.11.smithi115.stdout:May 13 23:17:40 smithi115 conmon[102145]: 2022-05-13T23:17:40.222+0000 7f98de059700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2022-05-13T23:17:16.234161+0000 front 2022-05-13T23:17:16.357197+0000 (oldest deadline 2022-05-13T23:17:37.333596+0000) 2022-05-13T23:17:40.609 INFO:journalctl@ceph.osd.11.smithi115.stdout:May 13 23:17:40 smithi115 conmon[102145]: 2022-05-13T23:17:40.222+0000 7f98de059700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.2:6830 osd.9 since back 2022-05-13T23:17:20.735459+0000 front 2022-05-13T23:17:16.233824+0000 (oldest deadline 2022-05-13T23:17:37.333596+0000) 2022-05-13T23:17:40.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:17:40 smithi115 ceph-mon[78593]: osd.9 reported failed by osd.8 2022-05-13T23:17:40.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:17:40 smithi115 ceph-mon[78593]: osd.9 failed (root=default,host=smithi002) (2 reporters from different osd after 24.000177 >= grace 20.000000) 2022-05-13T23:17:40.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:17:40 smithi115 ceph-mon[78593]: osd.0 reported failed by osd.11 2022-05-13T23:17:40.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:17:40 smithi115 ceph-mon[78593]: osd.0 failed (root=default,host=smithi002) (2 reporters from different osd after 23.000260 >= grace 20.000000) 2022-05-13T23:17:40.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:17:40 smithi115 ceph-mon[78593]: osd.9 reported failed by osd.11 2022-05-13T23:17:40.791 INFO:journalctl@ceph.osd.1.smithi047.stdout:May 13 23:17:40 smithi047 conmon[82670]: 2022-05-13T23:17:40.665+0000 7f9bceb39700 -1 osd.1 86 heartbeat_check: no reply from 172.21.15.2:6822 osd.6 since back 2022-05-13T23:17:19.858121+0000 front 2022-05-13T23:17:16.457356+0000 (oldest deadline 2022-05-13T23:17:36.956651+0000) 2022-05-13T23:17:40.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:17:40 smithi047 ceph-mon[77147]: osd.9 reported failed by osd.8 2022-05-13T23:17:40.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:17:40 smithi047 ceph-mon[77147]: osd.9 failed (root=default,host=smithi002) (2 reporters from different osd after 24.000177 >= grace 20.000000) 2022-05-13T23:17:40.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:17:40 smithi047 ceph-mon[77147]: osd.0 reported failed by osd.11 2022-05-13T23:17:40.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:17:40 smithi047 ceph-mon[77147]: osd.0 failed (root=default,host=smithi002) (2 reporters from different osd after 23.000260 >= grace 20.000000) 2022-05-13T23:17:40.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:17:40 smithi047 ceph-mon[77147]: osd.9 reported failed by osd.11 2022-05-13T23:17:41.291 INFO:journalctl@ceph.osd.4.smithi047.stdout:May 13 23:17:40 smithi047 conmon[88497]: 2022-05-13T23:17:40.832+0000 7f7738c18700 -1 osd.4 86 heartbeat_check: no reply from 172.21.15.2:6814 osd.3 since back 2022-05-13T23:17:16.428633+0000 front 2022-05-13T23:17:28.629743+0000 (oldest deadline 2022-05-13T23:17:39.927084+0000) 2022-05-13T23:17:41.681 INFO:journalctl@ceph.osd.7.smithi047.stdout:May 13 23:17:41 smithi047 conmon[94366]: 2022-05-13T23:17:41.405+0000 7fb536f80700 -1 osd.7 86 heartbeat_check: no reply from 172.21.15.2:6814 osd.3 since back 2022-05-13T23:17:16.285738+0000 front 2022-05-13T23:17:24.484675+0000 (oldest deadline 2022-05-13T23:17:40.983254+0000) 2022-05-13T23:17:41.681 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:17:41 smithi047 ceph-mon[77147]: Health check failed: 2 osds down (OSD_DOWN) 2022-05-13T23:17:41.682 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:17:41 smithi047 ceph-mon[77147]: osdmap e86: 12 total, 10 up, 12 in 2022-05-13T23:17:41.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:17:41 smithi115 ceph-mon[78593]: Health check failed: 2 osds down (OSD_DOWN) 2022-05-13T23:17:41.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:17:41 smithi115 ceph-mon[78593]: osdmap e86: 12 total, 10 up, 12 in 2022-05-13T23:17:41.859 INFO:journalctl@ceph.osd.2.smithi115.stdout:May 13 23:17:41 smithi115 conmon[84560]: 2022-05-13T23:17:41.686+0000 7ff49f571700 -1 osd.2 87 heartbeat_check: no reply from 172.21.15.2:6814 osd.3 since back 2022-05-13T23:17:28.232065+0000 front 2022-05-13T23:17:16.233471+0000 (oldest deadline 2022-05-13T23:17:40.927780+0000) 2022-05-13T23:17:41.859 INFO:journalctl@ceph.osd.2.smithi115.stdout:May 13 23:17:41 smithi115 conmon[84560]: 2022-05-13T23:17:41.686+0000 7ff49f571700 -1 osd.2 87 heartbeat_check: no reply from 172.21.15.2:6822 osd.6 since back 2022-05-13T23:17:22.029162+0000 front 2022-05-13T23:17:16.228795+0000 (oldest deadline 2022-05-13T23:17:40.927780+0000) 2022-05-13T23:17:42.041 INFO:journalctl@ceph.osd.4.smithi047.stdout:May 13 23:17:41 smithi047 conmon[88497]: 2022-05-13T23:17:41.811+0000 7f7738c18700 -1 osd.4 87 heartbeat_check: no reply from 172.21.15.2:6814 osd.3 since back 2022-05-13T23:17:16.428633+0000 front 2022-05-13T23:17:28.629743+0000 (oldest deadline 2022-05-13T23:17:39.927084+0000) 2022-05-13T23:17:42.042 INFO:journalctl@ceph.osd.1.smithi047.stdout:May 13 23:17:41 smithi047 conmon[82670]: 2022-05-13T23:17:41.679+0000 7f9bceb39700 -1 osd.1 87 heartbeat_check: no reply from 172.21.15.2:6822 osd.6 since back 2022-05-13T23:17:19.858121+0000 front 2022-05-13T23:17:16.457356+0000 (oldest deadline 2022-05-13T23:17:36.956651+0000) 2022-05-13T23:17:42.704 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:17:42 smithi047 ceph-mon[77147]: osdmap e87: 12 total, 10 up, 12 in 2022-05-13T23:17:42.704 INFO:journalctl@ceph.osd.7.smithi047.stdout:May 13 23:17:42 smithi047 conmon[94366]: 2022-05-13T23:17:42.432+0000 7fb536f80700 -1 osd.7 87 heartbeat_check: no reply from 172.21.15.2:6814 osd.3 since back 2022-05-13T23:17:16.285738+0000 front 2022-05-13T23:17:24.484675+0000 (oldest deadline 2022-05-13T23:17:40.983254+0000) 2022-05-13T23:17:42.860 INFO:journalctl@ceph.osd.2.smithi115.stdout:May 13 23:17:42 smithi115 conmon[84560]: 2022-05-13T23:17:42.636+0000 7ff49f571700 -1 osd.2 87 heartbeat_check: no reply from 172.21.15.2:6814 osd.3 since back 2022-05-13T23:17:28.232065+0000 front 2022-05-13T23:17:16.233471+0000 (oldest deadline 2022-05-13T23:17:40.927780+0000) 2022-05-13T23:17:42.860 INFO:journalctl@ceph.osd.2.smithi115.stdout:May 13 23:17:42 smithi115 conmon[84560]: 2022-05-13T23:17:42.636+0000 7ff49f571700 -1 osd.2 87 heartbeat_check: no reply from 172.21.15.2:6822 osd.6 since back 2022-05-13T23:17:22.029162+0000 front 2022-05-13T23:17:16.228795+0000 (oldest deadline 2022-05-13T23:17:40.927780+0000) 2022-05-13T23:17:42.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:17:42 smithi115 ceph-mon[78593]: osdmap e87: 12 total, 10 up, 12 in 2022-05-13T23:17:43.031 INFO:journalctl@ceph.osd.1.smithi047.stdout:May 13 23:17:42 smithi047 conmon[82670]: 2022-05-13T23:17:42.701+0000 7f9bceb39700 -1 osd.1 87 heartbeat_check: no reply from 172.21.15.2:6822 osd.6 since back 2022-05-13T23:17:19.858121+0000 front 2022-05-13T23:17:16.457356+0000 (oldest deadline 2022-05-13T23:17:36.956651+0000) 2022-05-13T23:17:43.032 INFO:journalctl@ceph.osd.4.smithi047.stdout:May 13 23:17:42 smithi047 conmon[88497]: 2022-05-13T23:17:42.762+0000 7f7738c18700 -1 osd.4 87 heartbeat_check: no reply from 172.21.15.2:6814 osd.3 since back 2022-05-13T23:17:16.428633+0000 front 2022-05-13T23:17:28.629743+0000 (oldest deadline 2022-05-13T23:17:39.927084+0000) 2022-05-13T23:17:43.291 INFO:journalctl@ceph.osd.10.smithi047.stdout:May 13 23:17:43 smithi047 conmon[100262]: 2022-05-13T23:17:43.029+0000 7f5b4f799700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.2:6814 osd.3 since back 2022-05-13T23:17:16.865434+0000 front 2022-05-13T23:17:28.666944+0000 (oldest deadline 2022-05-13T23:17:42.765555+0000) 2022-05-13T23:17:43.292 INFO:journalctl@ceph.osd.10.smithi047.stdout:May 13 23:17:43 smithi047 conmon[100262]: 2022-05-13T23:17:43.029+0000 7f5b4f799700 -1 osd.10 87 heartbeat_check: no reply from 172.21.15.2:6822 osd.6 since back 2022-05-13T23:17:16.865577+0000 front 2022-05-13T23:17:16.865710+0000 (oldest deadline 2022-05-13T23:17:42.765555+0000) 2022-05-13T23:17:43.767 INFO:journalctl@ceph.osd.7.smithi047.stdout:May 13 23:17:43 smithi047 conmon[94366]: 2022-05-13T23:17:43.473+0000 7fb536f80700 -1 osd.7 87 heartbeat_check: no reply from 172.21.15.2:6814 osd.3 since back 2022-05-13T23:17:16.285738+0000 front 2022-05-13T23:17:24.484675+0000 (oldest deadline 2022-05-13T23:17:40.983254+0000) 2022-05-13T23:17:43.768 INFO:journalctl@ceph.osd.1.smithi047.stdout:May 13 23:17:43 smithi047 conmon[82670]: 2022-05-13T23:17:43.684+0000 7f9bceb39700 -1 osd.1 87 heartbeat_check: no reply from 172.21.15.2:6822 osd.6 since back 2022-05-13T23:17:19.858121+0000 front 2022-05-13T23:17:16.457356+0000 (oldest deadline 2022-05-13T23:17:36.956651+0000) 2022-05-13T23:17:43.858 INFO:journalctl@ceph.osd.2.smithi115.stdout:May 13 23:17:43 smithi115 conmon[84560]: 2022-05-13T23:17:43.601+0000 7ff49f571700 -1 osd.2 87 heartbeat_check: no reply from 172.21.15.2:6814 osd.3 since back 2022-05-13T23:17:28.232065+0000 front 2022-05-13T23:17:16.233471+0000 (oldest deadline 2022-05-13T23:17:40.927780+0000) 2022-05-13T23:17:43.859 INFO:journalctl@ceph.osd.2.smithi115.stdout:May 13 23:17:43 smithi115 conmon[84560]: 2022-05-13T23:17:43.601+0000 7ff49f571700 -1 osd.2 87 heartbeat_check: no reply from 172.21.15.2:6822 osd.6 since back 2022-05-13T23:17:22.029162+0000 front 2022-05-13T23:17:16.228795+0000 (oldest deadline 2022-05-13T23:17:40.927780+0000) 2022-05-13T23:17:44.041 INFO:journalctl@ceph.osd.4.smithi047.stdout:May 13 23:17:43 smithi047 conmon[88497]: 2022-05-13T23:17:43.764+0000 7f7738c18700 -1 osd.4 87 heartbeat_check: no reply from 172.21.15.2:6814 osd.3 since back 2022-05-13T23:17:16.428633+0000 front 2022-05-13T23:17:28.629743+0000 (oldest deadline 2022-05-13T23:17:39.927084+0000) 2022-05-13T23:17:44.042 INFO:journalctl@ceph.osd.4.smithi047.stdout:May 13 23:17:43 smithi047 conmon[88497]: 2022-05-13T23:17:43.764+0000 7f7738c18700 -1 osd.4 87 heartbeat_check: no reply from 172.21.15.2:6822 osd.6 since back 2022-05-13T23:17:19.927951+0000 front 2022-05-13T23:17:19.928015+0000 (oldest deadline 2022-05-13T23:17:42.827888+0000) 2022-05-13T23:17:44.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:17:43 smithi047 ceph-mon[77147]: osd.3 reported failed by osd.10 2022-05-13T23:17:44.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:17:43 smithi047 ceph-mon[77147]: osd.6 reported failed by osd.10 2022-05-13T23:17:44.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:17:43 smithi047 ceph-mon[77147]: osd.6 failed (root=default,host=smithi002) (2 reporters from different osd after 25.566639 >= grace 20.000000) 2022-05-13T23:17:44.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:17:43 smithi047 ceph-mon[77147]: osd.3 reported failed by osd.7 2022-05-13T23:17:44.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:17:43 smithi047 ceph-mon[77147]: osd.3 failed (root=default,host=smithi002) (2 reporters from different osd after 26.443307 >= grace 20.000000) 2022-05-13T23:17:44.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:17:43 smithi047 ceph-mon[77147]: osd.3 reported failed by osd.4 2022-05-13T23:17:44.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:17:43 smithi047 ceph-mon[77147]: osd.6 reported failed by osd.4 2022-05-13T23:17:44.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:17:43 smithi115 ceph-mon[78593]: osd.3 reported failed by osd.10 2022-05-13T23:17:44.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:17:43 smithi115 ceph-mon[78593]: osd.6 reported failed by osd.10 2022-05-13T23:17:44.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:17:43 smithi115 ceph-mon[78593]: osd.6 failed (root=default,host=smithi002) (2 reporters from different osd after 25.566639 >= grace 20.000000) 2022-05-13T23:17:44.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:17:43 smithi115 ceph-mon[78593]: osd.3 reported failed by osd.7 2022-05-13T23:17:44.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:17:43 smithi115 ceph-mon[78593]: osd.3 failed (root=default,host=smithi002) (2 reporters from different osd after 26.443307 >= grace 20.000000) 2022-05-13T23:17:44.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:17:43 smithi115 ceph-mon[78593]: osd.3 reported failed by osd.4 2022-05-13T23:17:44.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:17:43 smithi115 ceph-mon[78593]: osd.6 reported failed by osd.4 2022-05-13T23:17:45.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:17:44 smithi047 ceph-mon[77147]: Health check failed: 1 host (4 osds) down (OSD_HOST_DOWN) 2022-05-13T23:17:45.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:17:44 smithi047 ceph-mon[77147]: osdmap e88: 12 total, 8 up, 12 in 2022-05-13T23:17:45.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:17:44 smithi115 ceph-mon[78593]: Health check failed: 1 host (4 osds) down (OSD_HOST_DOWN) 2022-05-13T23:17:45.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:17:44 smithi115 ceph-mon[78593]: osdmap e88: 12 total, 8 up, 12 in 2022-05-13T23:17:46.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:17:45 smithi047 ceph-mon[77147]: osdmap e89: 12 total, 8 up, 12 in 2022-05-13T23:17:46.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:17:45 smithi115 ceph-mon[78593]: osdmap e89: 12 total, 8 up, 12 in 2022-05-13T23:17:48.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:17:47 smithi047 ceph-mon[77147]: Health check update: 4 osds down (OSD_DOWN) 2022-05-13T23:17:48.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:17:47 smithi115 ceph-mon[78593]: Health check update: 4 osds down (OSD_DOWN) 2022-05-13T23:18:13.292 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:18:13 smithi047 conmon[78521]: 2022-05-13T23:18:13.163+0000 7f531d533700 -1 client.0 error registering admin socket command: (17) File exists 2022-05-13T23:18:13.292 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:18:13 smithi047 conmon[78521]: 2022-05-13T23:18:13.163+0000 7f531d533700 -1 client.0 error registering admin socket command: (17) File exists 2022-05-13T23:18:13.293 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:18:13 smithi047 conmon[78521]: 2022-05-13T23:18:13.163+0000 7f531d533700 -1 client.0 error registering admin socket command: (17) File exists 2022-05-13T23:18:13.293 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:18:13 smithi047 conmon[78521]: 2022-05-13T23:18:13.163+0000 7f531d533700 -1 client.0 error registering admin socket command: (17) File exists 2022-05-13T23:18:13.293 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:18:13 smithi047 conmon[78521]: 2022-05-13T23:18:13.163+0000 7f531d533700 -1 client.0 error registering admin socket command: (17) File exists 2022-05-13T23:18:13.293 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:18:13 smithi047 conmon[78521]: 2022-05-13T23:18:13.163+0000 7f5320539700 -1 client.0 error registering admin socket command: (17) File exists 2022-05-13T23:18:13.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:18:13 smithi047 conmon[78521]: 2022-05-13T23:18:13.163+0000 7f5320539700 -1 client.0 error registering admin socket command: (17) File exists 2022-05-13T23:18:13.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:18:13 smithi047 conmon[78521]: 2022-05-13T23:18:13.163+0000 7f5320539700 -1 client.0 error registering admin socket command: (17) File exists 2022-05-13T23:18:13.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:18:13 smithi047 conmon[78521]: 2022-05-13T23:18:13.163+0000 7f5320539700 -1 client.0 error registering admin socket command: (17) File exists 2022-05-13T23:18:13.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:18:13 smithi047 conmon[78521]: 2022-05-13T23:18:13.163+0000 7f5320539700 -1 client.0 error registering admin socket command: (17) File exists 2022-05-13T23:18:13.295 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:18:13 smithi047 ceph-mon[77147]: Manager daemon x is unresponsive, replacing it with standby daemon y 2022-05-13T23:18:13.295 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:18:13 smithi047 ceph-mon[77147]: osdmap e90: 12 total, 8 up, 12 in 2022-05-13T23:18:13.295 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:18:13 smithi047 ceph-mon[77147]: mgrmap e17: y(active, starting, since 0.00853748s), standbys: z 2022-05-13T23:18:13.296 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:18:13 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-05-13T23:18:13.296 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:18:13 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-05-13T23:18:13.296 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:18:13 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-05-13T23:18:13.296 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:18:13 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2022-05-13T23:18:13.297 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:18:13 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2022-05-13T23:18:13.297 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:18:13 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2022-05-13T23:18:13.297 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:18:13 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2022-05-13T23:18:13.297 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:18:13 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2022-05-13T23:18:13.298 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:18:13 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2022-05-13T23:18:13.298 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:18:13 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2022-05-13T23:18:13.298 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:18:13 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2022-05-13T23:18:13.298 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:18:13 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2022-05-13T23:18:13.299 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:18:13 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2022-05-13T23:18:13.299 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:18:13 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2022-05-13T23:18:13.299 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:18:13 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2022-05-13T23:18:13.300 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:18:13 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2022-05-13T23:18:13.300 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:18:13 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2022-05-13T23:18:13.300 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:18:13 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2022-05-13T23:18:13.300 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:18:13 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2022-05-13T23:18:13.301 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:18:13 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2022-05-13T23:18:13.301 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:18:13 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2022-05-13T23:18:13.301 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:18:13 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2022-05-13T23:18:13.301 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:18:13 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2022-05-13T23:18:13.301 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:18:13 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2022-05-13T23:18:13.302 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:18:13 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2022-05-13T23:18:13.302 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:18:13 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2022-05-13T23:18:13.302 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:18:13 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2022-05-13T23:18:13.302 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:18:13 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2022-05-13T23:18:13.303 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:18:13 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2022-05-13T23:18:13.303 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:18:13 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2022-05-13T23:18:13.303 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:18:13 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2022-05-13T23:18:13.303 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:18:13 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2022-05-13T23:18:13.354 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:18:13 smithi115 ceph-mon[78593]: Manager daemon x is unresponsive, replacing it with standby daemon y 2022-05-13T23:18:13.355 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:18:13 smithi115 ceph-mon[78593]: osdmap e90: 12 total, 8 up, 12 in 2022-05-13T23:18:13.355 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:18:13 smithi115 ceph-mon[78593]: mgrmap e17: y(active, starting, since 0.00853748s), standbys: z 2022-05-13T23:18:13.355 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:18:13 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-05-13T23:18:13.355 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:18:13 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-05-13T23:18:13.356 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:18:13 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-05-13T23:18:13.356 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:18:13 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2022-05-13T23:18:13.356 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:18:13 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2022-05-13T23:18:13.356 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:18:13 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2022-05-13T23:18:13.356 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:18:13 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2022-05-13T23:18:13.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:18:13 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2022-05-13T23:18:13.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:18:13 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2022-05-13T23:18:13.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:18:13 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2022-05-13T23:18:13.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:18:13 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2022-05-13T23:18:13.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:18:13 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2022-05-13T23:18:13.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:18:13 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2022-05-13T23:18:13.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:18:13 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2022-05-13T23:18:13.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:18:13 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2022-05-13T23:18:13.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:18:13 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2022-05-13T23:18:13.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:18:13 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2022-05-13T23:18:13.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:18:13 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2022-05-13T23:18:13.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:18:13 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2022-05-13T23:18:13.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:18:13 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2022-05-13T23:18:13.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:18:13 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2022-05-13T23:18:13.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:18:13 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2022-05-13T23:18:13.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:18:13 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2022-05-13T23:18:13.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:18:13 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2022-05-13T23:18:13.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:18:13 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2022-05-13T23:18:13.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:18:13 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2022-05-13T23:18:13.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:18:13 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2022-05-13T23:18:13.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:18:13 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2022-05-13T23:18:13.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:18:13 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2022-05-13T23:18:13.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:18:13 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2022-05-13T23:18:13.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:18:13 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2022-05-13T23:18:13.362 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:18:13 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2022-05-13T23:18:14.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:18:14 smithi115 ceph-mon[78593]: Manager daemon y is now available 2022-05-13T23:18:14.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:18:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-13T23:18:14.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:18:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-13T23:18:14.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:18:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-13T23:18:14.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:18:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-13T23:18:14.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:18:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-13T23:18:14.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:18:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-13T23:18:14.112 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:18:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' 2022-05-13T23:18:14.112 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:18:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' 2022-05-13T23:18:14.247 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:18:14 smithi047 ceph-mon[77147]: Manager daemon y is now available 2022-05-13T23:18:14.247 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:18:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-13T23:18:14.248 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:18:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-13T23:18:14.248 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:18:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-13T23:18:14.248 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:18:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-13T23:18:14.248 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:18:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-13T23:18:14.248 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:18:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-13T23:18:14.249 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:18:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' 2022-05-13T23:18:14.249 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:18:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' 2022-05-13T23:18:14.541 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:18:14 smithi047 conmon[78521]: [13/May/2022:23:18:14] ENGINE Bus STARTING 2022-05-13T23:18:14.542 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:18:14 smithi047 conmon[78521]: [13/May/2022:23:18:14] ENGINE Serving on https://172.21.15.47:7150 2022-05-13T23:18:14.542 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:18:14 smithi047 conmon[78521]: [13/May/2022:23:18:14] ENGINE Bus STARTED 2022-05-13T23:18:15.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:18:15 smithi047 ceph-mon[77147]: mgrmap e18: y(active, since 1.06339s), standbys: z 2022-05-13T23:18:15.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:18:15 smithi047 ceph-mon[77147]: [13/May/2022:23:18:14] ENGINE Bus STARTING 2022-05-13T23:18:15.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:18:15 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' 2022-05-13T23:18:15.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:18:15 smithi115 ceph-mon[78593]: mgrmap e18: y(active, since 1.06339s), standbys: z 2022-05-13T23:18:15.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:18:15 smithi115 ceph-mon[78593]: [13/May/2022:23:18:14] ENGINE Bus STARTING 2022-05-13T23:18:15.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:18:15 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' 2022-05-13T23:18:16.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:18:16 smithi047 ceph-mon[77147]: [13/May/2022:23:18:14] ENGINE Serving on https://172.21.15.47:7150 2022-05-13T23:18:16.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:18:16 smithi047 ceph-mon[77147]: [13/May/2022:23:18:14] ENGINE Bus STARTED 2022-05-13T23:18:16.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:18:16 smithi047 ceph-mon[77147]: pgmap v6: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 138 MiB data, 464 MiB used, 715 GiB / 715 GiB avail; 10439/32427 objects degraded (32.192%) 2022-05-13T23:18:16.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:18:16 smithi047 ceph-mon[77147]: Health check failed: Reduced data availability: 2 pgs inactive (PG_AVAILABILITY) 2022-05-13T23:18:16.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:18:16 smithi047 ceph-mon[77147]: Health check failed: Degraded data redundancy: 10439/32427 objects degraded (32.192%), 67 pgs degraded (PG_DEGRADED) 2022-05-13T23:18:16.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:18:16 smithi115 ceph-mon[78593]: [13/May/2022:23:18:14] ENGINE Serving on https://172.21.15.47:7150 2022-05-13T23:18:16.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:18:16 smithi115 ceph-mon[78593]: [13/May/2022:23:18:14] ENGINE Bus STARTED 2022-05-13T23:18:16.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:18:16 smithi115 ceph-mon[78593]: pgmap v6: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 138 MiB data, 464 MiB used, 715 GiB / 715 GiB avail; 10439/32427 objects degraded (32.192%) 2022-05-13T23:18:16.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:18:16 smithi115 ceph-mon[78593]: Health check failed: Reduced data availability: 2 pgs inactive (PG_AVAILABILITY) 2022-05-13T23:18:16.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:18:16 smithi115 ceph-mon[78593]: Health check failed: Degraded data redundancy: 10439/32427 objects degraded (32.192%), 67 pgs degraded (PG_DEGRADED) 2022-05-13T23:18:17.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:18:17 smithi047 ceph-mon[77147]: mgrmap e19: y(active, since 3s), standbys: z 2022-05-13T23:18:17.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:18:17 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' 2022-05-13T23:18:17.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:18:17 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' 2022-05-13T23:18:17.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:18:17 smithi115 ceph-mon[78593]: mgrmap e19: y(active, since 3s), standbys: z 2022-05-13T23:18:17.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:18:17 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' 2022-05-13T23:18:17.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:18:17 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' 2022-05-13T23:18:18.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:18:18 smithi047 ceph-mon[77147]: pgmap v7: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 138 MiB data, 464 MiB used, 715 GiB / 715 GiB avail; 10439/32427 objects degraded (32.192%) 2022-05-13T23:18:18.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:18:18 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' 2022-05-13T23:18:18.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:18:18 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-05-13T23:18:18.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:18:18 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' 2022-05-13T23:18:18.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:18:18 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-05-13T23:18:18.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:18:18 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-05-13T23:18:18.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:18:18 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-05-13T23:18:18.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:18:18 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-05-13T23:18:18.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:18:18 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-05-13T23:18:18.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:18:18 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-05-13T23:18:18.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:18:18 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-05-13T23:18:18.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:18:18 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' 2022-05-13T23:18:18.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:18:18 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' 2022-05-13T23:18:18.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:18:18 smithi115 ceph-mon[78593]: pgmap v7: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 138 MiB data, 464 MiB used, 715 GiB / 715 GiB avail; 10439/32427 objects degraded (32.192%) 2022-05-13T23:18:18.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:18:18 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' 2022-05-13T23:18:18.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:18:18 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-05-13T23:18:18.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:18:18 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' 2022-05-13T23:18:18.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:18:18 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-05-13T23:18:18.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:18:18 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-05-13T23:18:18.362 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:18:18 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-05-13T23:18:18.362 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:18:18 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-05-13T23:18:18.362 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:18:18 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-05-13T23:18:18.362 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:18:18 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-05-13T23:18:18.363 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:18:18 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-05-13T23:18:18.363 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:18:18 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' 2022-05-13T23:18:18.363 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:18:18 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' 2022-05-13T23:18:19.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:18:19 smithi047 ceph-mon[77147]: Adjusting osd_memory_target on smithi047 to 198.1M 2022-05-13T23:18:19.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:18:19 smithi047 ceph-mon[77147]: Unable to set osd_memory_target on smithi047 to 207755878: error parsing value: Value '207755878' is below minimum 939524096 2022-05-13T23:18:19.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:18:19 smithi047 ceph-mon[77147]: Updating smithi047:/etc/ceph/ceph.conf 2022-05-13T23:18:19.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:18:19 smithi047 ceph-mon[77147]: Adjusting osd_memory_target on smithi115 to 197.0M 2022-05-13T23:18:19.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:18:19 smithi047 ceph-mon[77147]: Unable to set osd_memory_target on smithi115 to 206629785: error parsing value: Value '206629785' is below minimum 939524096 2022-05-13T23:18:19.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:18:19 smithi047 ceph-mon[77147]: Updating smithi115:/etc/ceph/ceph.conf 2022-05-13T23:18:19.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:18:19 smithi047 ceph-mon[77147]: Updating smithi115:/etc/ceph/ceph.client.admin.keyring 2022-05-13T23:18:19.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:18:19 smithi047 ceph-mon[77147]: Updating smithi047:/etc/ceph/ceph.client.admin.keyring 2022-05-13T23:18:19.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:18:19 smithi115 ceph-mon[78593]: Adjusting osd_memory_target on smithi047 to 198.1M 2022-05-13T23:18:19.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:18:19 smithi115 ceph-mon[78593]: Unable to set osd_memory_target on smithi047 to 207755878: error parsing value: Value '207755878' is below minimum 939524096 2022-05-13T23:18:19.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:18:19 smithi115 ceph-mon[78593]: Updating smithi047:/etc/ceph/ceph.conf 2022-05-13T23:18:19.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:18:19 smithi115 ceph-mon[78593]: Adjusting osd_memory_target on smithi115 to 197.0M 2022-05-13T23:18:19.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:18:19 smithi115 ceph-mon[78593]: Unable to set osd_memory_target on smithi115 to 206629785: error parsing value: Value '206629785' is below minimum 939524096 2022-05-13T23:18:19.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:18:19 smithi115 ceph-mon[78593]: Updating smithi115:/etc/ceph/ceph.conf 2022-05-13T23:18:19.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:18:19 smithi115 ceph-mon[78593]: Updating smithi115:/etc/ceph/ceph.client.admin.keyring 2022-05-13T23:18:19.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:18:19 smithi115 ceph-mon[78593]: Updating smithi047:/etc/ceph/ceph.client.admin.keyring 2022-05-13T23:18:20.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:18:20 smithi047 ceph-mon[77147]: pgmap v8: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 138 MiB data, 464 MiB used, 715 GiB / 715 GiB avail; 10439/32427 objects degraded (32.192%) 2022-05-13T23:18:20.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:18:20 smithi115 ceph-mon[78593]: pgmap v8: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 138 MiB data, 464 MiB used, 715 GiB / 715 GiB avail; 10439/32427 objects degraded (32.192%) 2022-05-13T23:18:22.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:18:22 smithi047 ceph-mon[77147]: pgmap v9: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 138 MiB data, 464 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 2 op/s; 10439/32427 objects degraded (32.192%) 2022-05-13T23:18:22.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:18:22 smithi115 ceph-mon[78593]: pgmap v9: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 138 MiB data, 464 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 2 op/s; 10439/32427 objects degraded (32.192%) 2022-05-13T23:18:24.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:18:23 smithi047 ceph-mon[77147]: pgmap v10: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 138 MiB data, 464 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 2 op/s; 10439/32427 objects degraded (32.192%) 2022-05-13T23:18:24.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:18:23 smithi115 ceph-mon[78593]: pgmap v10: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 138 MiB data, 464 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 2 op/s; 10439/32427 objects degraded (32.192%) 2022-05-13T23:18:26.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:18:26 smithi047 ceph-mon[77147]: pgmap v11: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 138 MiB data, 464 MiB used, 715 GiB / 715 GiB avail; 1.8 KiB/s rd, 3 op/s; 10439/32427 objects degraded (32.192%) 2022-05-13T23:18:26.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:18:26 smithi115 ceph-mon[78593]: pgmap v11: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 138 MiB data, 464 MiB used, 715 GiB / 715 GiB avail; 1.8 KiB/s rd, 3 op/s; 10439/32427 objects degraded (32.192%) 2022-05-13T23:18:28.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:18:28 smithi047 ceph-mon[77147]: pgmap v12: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 138 MiB data, 464 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 3 op/s; 10439/32427 objects degraded (32.192%) 2022-05-13T23:18:28.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:18:28 smithi115 ceph-mon[78593]: pgmap v12: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 138 MiB data, 464 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 3 op/s; 10439/32427 objects degraded (32.192%) 2022-05-13T23:18:30.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:18:30 smithi047 ceph-mon[77147]: pgmap v13: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 138 MiB data, 464 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 3 op/s; 10439/32427 objects degraded (32.192%) 2022-05-13T23:18:30.293 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:18:29 smithi047 conmon[78521]: 2022-05-13T23:18:29.937+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:18:30.293 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:18:29 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-13T23:18:30.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:18:29 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-13T23:18:30.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:18:29 smithi047 conmon[78521]: yield 2022-05-13T23:18:30.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:18:29 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-13T23:18:30.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:18:29 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-13T23:18:30.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:18:29 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-13T23:18:30.297 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:18:29 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-13T23:18:30.297 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:18:29 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-13T23:18:30.301 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:18:29 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-13T23:18:30.301 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:18:29 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-13T23:18:30.302 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:18:29 smithi047 conmon[78521]: raise exceptions[0] 2022-05-13T23:18:30.302 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:18:29 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-13T23:18:30.302 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:18:29 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-13T23:18:30.302 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:18:29 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-13T23:18:30.303 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:18:29 smithi047 conmon[78521]: return (yield from fut) 2022-05-13T23:18:30.303 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:18:29 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-13T23:18:30.303 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:18:29 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-13T23:18:30.304 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:18:29 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:18:30.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:18:30 smithi115 ceph-mon[78593]: pgmap v13: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 138 MiB data, 464 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 3 op/s; 10439/32427 objects degraded (32.192%) 2022-05-13T23:18:31.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:18:31 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:18:31.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-13T23:18:31.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-13T23:18:31.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-13T23:18:31.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-13T23:18:31.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-13T23:18:31.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-13T23:18:31.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-13T23:18:31.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-13T23:18:31.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-13T23:18:31.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-13T23:18:31.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-13T23:18:31.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-13T23:18:31.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-13T23:18:31.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-13T23:18:31.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-13T23:18:31.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-13T23:18:31.365 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-13T23:18:31.365 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:18:31.365 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:18:31 smithi115 ceph-mon[78593]: Updating smithi002:/etc/ceph/ceph.conf 2022-05-13T23:18:31.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:18:31 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:18:31.542 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-13T23:18:31.542 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-13T23:18:31.542 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-13T23:18:31.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-13T23:18:31.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-13T23:18:31.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-13T23:18:31.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-13T23:18:31.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-13T23:18:31.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-13T23:18:31.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-13T23:18:31.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-13T23:18:31.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-13T23:18:31.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-13T23:18:31.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-13T23:18:31.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-13T23:18:31.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-13T23:18:31.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-13T23:18:31.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:18:31.546 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:18:31 smithi047 ceph-mon[77147]: Updating smithi002:/etc/ceph/ceph.conf 2022-05-13T23:18:32.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:18:32 smithi115 ceph-mon[78593]: pgmap v14: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 138 MiB data, 464 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s; 10439/32427 objects degraded (32.192%) 2022-05-13T23:18:32.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:18:32 smithi047 ceph-mon[77147]: pgmap v14: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 138 MiB data, 464 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s; 10439/32427 objects degraded (32.192%) 2022-05-13T23:18:33.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:18:33 smithi047 conmon[78521]: 2022-05-13T23:18:33.008+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:18:33.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:18:33 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-13T23:18:33.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:18:33 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-13T23:18:33.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:18:33 smithi047 conmon[78521]: yield 2022-05-13T23:18:33.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:18:33 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-13T23:18:33.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:18:33 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-13T23:18:33.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:18:33 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-13T23:18:33.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:18:33 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-13T23:18:33.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:18:33 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-13T23:18:33.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:18:33 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-13T23:18:33.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:18:33 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-13T23:18:33.297 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:18:33 smithi047 conmon[78521]: raise exceptions[0] 2022-05-13T23:18:33.297 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:18:33 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-13T23:18:33.297 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:18:33 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-13T23:18:33.297 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:18:33 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-13T23:18:33.298 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:18:33 smithi047 conmon[78521]: return (yield from fut) 2022-05-13T23:18:33.298 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:18:33 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-13T23:18:33.298 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:18:33 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-13T23:18:33.298 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:18:33 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:18:33.298 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:18:33 smithi047 conmon[78521]: 2022-05-13T23:18:33.008+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Unable to write smithi002:/etc/ceph/ceph.conf: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:18:33.299 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:18:33 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-13T23:18:33.299 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:18:33 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-13T23:18:33.299 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:18:33 smithi047 conmon[78521]: yield 2022-05-13T23:18:33.299 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:18:33 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-13T23:18:33.300 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:18:33 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-13T23:18:33.300 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:18:33 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-13T23:18:33.300 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:18:33 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-13T23:18:33.300 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:18:33 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-13T23:18:33.300 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:18:33 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-13T23:18:33.301 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:18:33 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-13T23:18:33.301 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:18:33 smithi047 conmon[78521]: raise exceptions[0] 2022-05-13T23:18:33.301 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:18:33 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-13T23:18:33.301 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:18:33 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-13T23:18:33.302 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:18:33 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-13T23:18:33.302 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:18:33 smithi047 conmon[78521]: return (yield from fut) 2022-05-13T23:18:33.302 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:18:33 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-13T23:18:33.302 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:18:33 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-13T23:18:33.302 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:18:33 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:18:33.303 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:18:33 smithi047 conmon[78521]: 2022-05-13T23:18:33.304 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:18:33 smithi047 conmon[78521]: During handling of the above exception, another exception occurred: 2022-05-13T23:18:33.304 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:18:33 smithi047 conmon[78521]: 2022-05-13T23:18:33.304 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:18:33 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-13T23:18:33.304 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:18:33 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 208, in _write_remote_file 2022-05-13T23:18:33.304 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:18:33 smithi047 conmon[78521]: await self._check_execute_command(host, ['mkdir', '-p', dirname], addr=addr) 2022-05-13T23:18:33.305 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:18:33 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-13T23:18:33.305 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:18:33 smithi047 conmon[78521]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-13T23:18:33.305 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:18:33 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-13T23:18:33.305 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:18:33 smithi047 conmon[78521]: conn = await self._remote_connection(host, addr) 2022-05-13T23:18:33.306 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:18:33 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-13T23:18:33.306 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:18:33 smithi047 conmon[78521]: raise 2022-05-13T23:18:33.306 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:18:33 smithi047 conmon[78521]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-13T23:18:33.306 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:18:33 smithi047 conmon[78521]: self.gen.throw(type, value, traceback) 2022-05-13T23:18:33.307 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:18:33 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-13T23:18:33.312 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:18:33 smithi047 conmon[78521]: raise OrchestratorError(msg) 2022-05-13T23:18:33.312 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:18:33 smithi047 conmon[78521]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:18:33.312 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:18:33 smithi047 conmon[78521]: 2022-05-13T23:18:33.010+0000 7f5345f84700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-13T23:18:33.313 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:18:33 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-13T23:18:33.313 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:18:33 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-13T23:18:33.313 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:18:33 smithi047 conmon[78521]: yield 2022-05-13T23:18:33.313 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:18:33 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-13T23:18:33.313 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:18:33 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-13T23:18:33.314 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:18:33 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-13T23:18:33.314 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:18:33 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-13T23:18:33.314 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:18:33 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-13T23:18:33.314 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:18:33 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-13T23:18:33.315 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:18:33 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-13T23:18:33.315 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:18:33 smithi047 conmon[78521]: raise exceptions[0] 2022-05-13T23:18:33.315 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:18:33 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-13T23:18:33.315 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:18:33 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-13T23:18:33.316 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:18:33 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-13T23:18:33.316 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:18:33 smithi047 conmon[78521]: return (yield from fut) 2022-05-13T23:18:33.316 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:18:33 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-13T23:18:33.316 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:18:33 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-13T23:18:33.316 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:18:33 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:18:33.317 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:18:33 smithi047 conmon[78521]: 2022-05-13T23:18:33.317 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:18:33 smithi047 conmon[78521]: During handling of the above exception, another exception occurred: 2022-05-13T23:18:33.317 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:18:33 smithi047 conmon[78521]: 2022-05-13T23:18:33.317 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:18:33 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-13T23:18:33.317 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:18:33 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 208, in _write_remote_file 2022-05-13T23:18:33.318 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:18:33 smithi047 conmon[78521]: await self._check_execute_command(host, ['mkdir', '-p', dirname], addr=addr) 2022-05-13T23:18:33.318 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:18:33 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-13T23:18:33.318 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:18:33 smithi047 conmon[78521]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-13T23:18:33.318 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:18:33 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-13T23:18:33.319 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:18:33 smithi047 conmon[78521]: conn = await self._remote_connection(host, addr) 2022-05-13T23:18:33.319 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:18:33 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-13T23:18:33.319 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:18:33 smithi047 conmon[78521]: raise 2022-05-13T23:18:33.319 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:18:33 smithi047 conmon[78521]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-13T23:18:33.319 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:18:33 smithi047 conmon[78521]: self.gen.throw(type, value, traceback) 2022-05-13T23:18:33.320 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:18:33 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-13T23:18:33.320 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:18:33 smithi047 conmon[78521]: raise OrchestratorError(msg) 2022-05-13T23:18:33.320 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:18:33 smithi047 conmon[78521]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:18:33.320 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:18:33 smithi047 conmon[78521]: 2022-05-13T23:18:33.320 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:18:33 smithi047 conmon[78521]: During handling of the above exception, another exception occurred: 2022-05-13T23:18:33.321 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:18:33 smithi047 conmon[78521]: 2022-05-13T23:18:33.321 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:18:33 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-13T23:18:33.321 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:18:33 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-13T23:18:33.321 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:18:33 smithi047 conmon[78521]: return f(*arg) 2022-05-13T23:18:33.322 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:18:33 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-13T23:18:33.322 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:18:33 smithi047 conmon[78521]: self._write_client_files(client_files, host) 2022-05-13T23:18:33.322 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:18:33 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1054, in _write_client_files 2022-05-13T23:18:33.322 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:18:33 smithi047 conmon[78521]: self.mgr.ssh.write_remote_file(host, path, content, mode, uid, gid) 2022-05-13T23:18:33.322 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:18:33 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 242, in write_remote_file 2022-05-13T23:18:33.323 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:18:33 smithi047 conmon[78521]: host, path, content, mode, uid, gid, addr)) 2022-05-13T23:18:33.323 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:18:33 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-13T23:18:33.323 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:18:33 smithi047 conmon[78521]: return self.event_loop.get_result(coro) 2022-05-13T23:18:33.323 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:18:33 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-13T23:18:33.323 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:18:33 smithi047 conmon[78521]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-13T23:18:33.324 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:18:33 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-13T23:18:33.324 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:18:33 smithi047 conmon[78521]: return self.__get_result() 2022-05-13T23:18:33.324 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:18:33 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-13T23:18:33.324 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:18:33 smithi047 conmon[78521]: raise self._exception 2022-05-13T23:18:33.324 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:18:33 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 230, in _write_remote_file 2022-05-13T23:18:33.325 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:18:33 smithi047 conmon[78521]: raise OrchestratorError(msg) 2022-05-13T23:18:33.325 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:18:33 smithi047 conmon[78521]: orchestrator._interface.OrchestratorError: Unable to write smithi002:/etc/ceph/ceph.conf: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:18:34.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:18:33 smithi047 ceph-mon[77147]: pgmap v15: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 138 MiB data, 464 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 3 op/s; 10439/32427 objects degraded (32.192%) 2022-05-13T23:18:34.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:18:33 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:18:34.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-13T23:18:34.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-13T23:18:34.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-13T23:18:34.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-13T23:18:34.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-13T23:18:34.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-13T23:18:34.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-13T23:18:34.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-13T23:18:34.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-13T23:18:34.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-13T23:18:34.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-13T23:18:34.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-13T23:18:34.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-13T23:18:34.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-13T23:18:34.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-13T23:18:34.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-13T23:18:34.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-13T23:18:34.298 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:18:34.298 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:18:33 smithi047 ceph-mon[77147]: Unable to write smithi002:/etc/ceph/ceph.conf: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:18:34.298 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-13T23:18:34.298 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-13T23:18:34.298 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-13T23:18:34.300 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-13T23:18:34.300 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-13T23:18:34.300 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-13T23:18:34.301 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-13T23:18:34.301 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-13T23:18:34.301 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-13T23:18:34.302 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-13T23:18:34.302 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-13T23:18:34.302 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-13T23:18:34.303 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-13T23:18:34.304 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-13T23:18:34.304 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-13T23:18:34.305 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-13T23:18:34.305 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-13T23:18:34.305 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:18:34.306 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-13T23:18:34.306 INFO:journalctl@ceph.mon.b.smithi047.stdout: During handling of the above exception, another exception occurred: 2022-05-13T23:18:34.306 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-13T23:18:34.306 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-13T23:18:34.306 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 208, in _write_remote_file 2022-05-13T23:18:34.307 INFO:journalctl@ceph.mon.b.smithi047.stdout: await self._check_execute_command(host, ['mkdir', '-p', dirname], addr=addr) 2022-05-13T23:18:34.307 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-13T23:18:34.308 INFO:journalctl@ceph.mon.b.smithi047.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-13T23:18:34.308 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-13T23:18:34.309 INFO:journalctl@ceph.mon.b.smithi047.stdout: conn = await self._remote_connection(host, addr) 2022-05-13T23:18:34.309 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-13T23:18:34.309 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise 2022-05-13T23:18:34.309 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-13T23:18:34.310 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.gen.throw(type, value, traceback) 2022-05-13T23:18:34.310 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-13T23:18:34.310 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OrchestratorError(msg) 2022-05-13T23:18:34.310 INFO:journalctl@ceph.mon.b.smithi047.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:18:34.310 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:18:33 smithi047 ceph-mon[77147]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-13T23:18:34.311 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-13T23:18:34.311 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-13T23:18:34.311 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-13T23:18:34.312 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-13T23:18:34.312 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-13T23:18:34.312 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-13T23:18:34.312 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-13T23:18:34.312 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-13T23:18:34.313 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-13T23:18:34.313 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-13T23:18:34.313 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-13T23:18:34.313 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-13T23:18:34.314 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-13T23:18:34.314 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-13T23:18:34.314 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-13T23:18:34.314 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-13T23:18:34.314 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-13T23:18:34.315 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:18:34.315 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-13T23:18:34.315 INFO:journalctl@ceph.mon.b.smithi047.stdout: During handling of the above exception, another exception occurred: 2022-05-13T23:18:34.315 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-13T23:18:34.316 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-13T23:18:34.316 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 208, in _write_remote_file 2022-05-13T23:18:34.316 INFO:journalctl@ceph.mon.b.smithi047.stdout: await self._check_execute_command(host, ['mkdir', '-p', dirname], addr=addr) 2022-05-13T23:18:34.316 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-13T23:18:34.316 INFO:journalctl@ceph.mon.b.smithi047.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-13T23:18:34.317 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-13T23:18:34.317 INFO:journalctl@ceph.mon.b.smithi047.stdout: conn = await self._remote_connection(host, addr) 2022-05-13T23:18:34.317 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-13T23:18:34.317 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise 2022-05-13T23:18:34.318 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-13T23:18:34.318 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.gen.throw(type, value, traceback) 2022-05-13T23:18:34.318 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-13T23:18:34.318 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OrchestratorError(msg) 2022-05-13T23:18:34.319 INFO:journalctl@ceph.mon.b.smithi047.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:18:34.319 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-13T23:18:34.319 INFO:journalctl@ceph.mon.b.smithi047.stdout: During handling of the above exception, another exception occurred: 2022-05-13T23:18:34.320 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-13T23:18:34.320 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-13T23:18:34.320 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-13T23:18:34.320 INFO:journalctl@ceph.mon.b.smithi047.stdout: return f(*arg) 2022-05-13T23:18:34.321 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-13T23:18:34.321 INFO:journalctl@ceph.mon.b.smithi047.stdout: self._write_client_files(client_files, host) 2022-05-13T23:18:34.321 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1054, in _write_client_files 2022-05-13T23:18:34.321 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.mgr.ssh.write_remote_file(host, path, content, mode, uid, gid) 2022-05-13T23:18:34.322 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 242, in write_remote_file 2022-05-13T23:18:34.322 INFO:journalctl@ceph.mon.b.smithi047.stdout: host, path, content, mode, uid, gid, addr)) 2022-05-13T23:18:34.323 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-13T23:18:34.323 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.event_loop.get_result(coro) 2022-05-13T23:18:34.323 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-13T23:18:34.324 INFO:journalctl@ceph.mon.b.smithi047.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-13T23:18:34.325 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-13T23:18:34.325 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.__get_result() 2022-05-13T23:18:34.326 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-13T23:18:34.326 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise self._exception 2022-05-13T23:18:34.326 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 230, in _write_remote_file 2022-05-13T23:18:34.327 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OrchestratorError(msg) 2022-05-13T23:18:34.327 INFO:journalctl@ceph.mon.b.smithi047.stdout: orchestrator._interface.OrchestratorError: Unable to write smithi002:/etc/ceph/ceph.conf: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:18:34.328 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:18:33 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-13T23:18:34.328 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:18:33 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-13T23:18:34.329 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:18:33 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-13T23:18:34.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:18:33 smithi115 ceph-mon[78593]: pgmap v15: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 138 MiB data, 464 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 3 op/s; 10439/32427 objects degraded (32.192%) 2022-05-13T23:18:34.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:18:33 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:18:34.359 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-13T23:18:34.359 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-13T23:18:34.359 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-13T23:18:34.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-13T23:18:34.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-13T23:18:34.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-13T23:18:34.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-13T23:18:34.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-13T23:18:34.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-13T23:18:34.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-13T23:18:34.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-13T23:18:34.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-13T23:18:34.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-13T23:18:34.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-13T23:18:34.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-13T23:18:34.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-13T23:18:34.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-13T23:18:34.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:18:34.363 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:18:33 smithi115 ceph-mon[78593]: Unable to write smithi002:/etc/ceph/ceph.conf: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:18:34.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-13T23:18:34.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-13T23:18:34.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-13T23:18:34.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-13T23:18:34.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-13T23:18:34.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-13T23:18:34.365 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-13T23:18:34.365 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-13T23:18:34.365 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-13T23:18:34.365 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-13T23:18:34.365 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-13T23:18:34.366 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-13T23:18:34.366 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-13T23:18:34.366 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-13T23:18:34.366 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-13T23:18:34.367 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-13T23:18:34.367 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-13T23:18:34.367 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:18:34.367 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-13T23:18:34.367 INFO:journalctl@ceph.mon.c.smithi115.stdout: During handling of the above exception, another exception occurred: 2022-05-13T23:18:34.368 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-13T23:18:34.368 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-13T23:18:34.368 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 208, in _write_remote_file 2022-05-13T23:18:34.368 INFO:journalctl@ceph.mon.c.smithi115.stdout: await self._check_execute_command(host, ['mkdir', '-p', dirname], addr=addr) 2022-05-13T23:18:34.368 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-13T23:18:34.369 INFO:journalctl@ceph.mon.c.smithi115.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-13T23:18:34.369 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-13T23:18:34.369 INFO:journalctl@ceph.mon.c.smithi115.stdout: conn = await self._remote_connection(host, addr) 2022-05-13T23:18:34.370 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-13T23:18:34.370 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise 2022-05-13T23:18:34.370 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-13T23:18:34.370 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.gen.throw(type, value, traceback) 2022-05-13T23:18:34.370 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-13T23:18:34.371 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OrchestratorError(msg) 2022-05-13T23:18:34.371 INFO:journalctl@ceph.mon.c.smithi115.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:18:34.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:18:33 smithi115 ceph-mon[78593]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-13T23:18:34.371 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-13T23:18:34.371 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-13T23:18:34.372 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-13T23:18:34.372 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-13T23:18:34.372 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-13T23:18:34.372 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-13T23:18:34.373 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-13T23:18:34.373 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-13T23:18:34.373 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-13T23:18:34.374 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-13T23:18:34.374 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-13T23:18:34.374 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-13T23:18:34.374 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-13T23:18:34.375 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-13T23:18:34.375 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-13T23:18:34.375 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-13T23:18:34.375 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-13T23:18:34.375 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:18:34.376 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-13T23:18:34.376 INFO:journalctl@ceph.mon.c.smithi115.stdout: During handling of the above exception, another exception occurred: 2022-05-13T23:18:34.376 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-13T23:18:34.376 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-13T23:18:34.376 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 208, in _write_remote_file 2022-05-13T23:18:34.377 INFO:journalctl@ceph.mon.c.smithi115.stdout: await self._check_execute_command(host, ['mkdir', '-p', dirname], addr=addr) 2022-05-13T23:18:34.377 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-13T23:18:34.377 INFO:journalctl@ceph.mon.c.smithi115.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-13T23:18:34.377 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-13T23:18:34.378 INFO:journalctl@ceph.mon.c.smithi115.stdout: conn = await self._remote_connection(host, addr) 2022-05-13T23:18:34.378 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-13T23:18:34.378 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise 2022-05-13T23:18:34.378 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-13T23:18:34.378 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.gen.throw(type, value, traceback) 2022-05-13T23:18:34.379 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-13T23:18:34.379 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OrchestratorError(msg) 2022-05-13T23:18:34.379 INFO:journalctl@ceph.mon.c.smithi115.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:18:34.379 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-13T23:18:34.379 INFO:journalctl@ceph.mon.c.smithi115.stdout: During handling of the above exception, another exception occurred: 2022-05-13T23:18:34.380 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-13T23:18:34.380 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-13T23:18:34.380 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-13T23:18:34.380 INFO:journalctl@ceph.mon.c.smithi115.stdout: return f(*arg) 2022-05-13T23:18:34.381 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-13T23:18:34.381 INFO:journalctl@ceph.mon.c.smithi115.stdout: self._write_client_files(client_files, host) 2022-05-13T23:18:34.381 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1054, in _write_client_files 2022-05-13T23:18:34.381 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.mgr.ssh.write_remote_file(host, path, content, mode, uid, gid) 2022-05-13T23:18:34.382 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 242, in write_remote_file 2022-05-13T23:18:34.382 INFO:journalctl@ceph.mon.c.smithi115.stdout: host, path, content, mode, uid, gid, addr)) 2022-05-13T23:18:34.382 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-13T23:18:34.382 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.event_loop.get_result(coro) 2022-05-13T23:18:34.382 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-13T23:18:34.383 INFO:journalctl@ceph.mon.c.smithi115.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-13T23:18:34.383 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-13T23:18:34.383 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.__get_result() 2022-05-13T23:18:34.384 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-13T23:18:34.384 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise self._exception 2022-05-13T23:18:34.384 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 230, in _write_remote_file 2022-05-13T23:18:34.384 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OrchestratorError(msg) 2022-05-13T23:18:34.385 INFO:journalctl@ceph.mon.c.smithi115.stdout: orchestrator._interface.OrchestratorError: Unable to write smithi002:/etc/ceph/ceph.conf: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:18:34.385 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:18:33 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-13T23:18:34.385 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:18:33 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-13T23:18:34.385 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:18:33 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-13T23:18:36.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:18:36 smithi115 ceph-mon[78593]: pgmap v16: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 138 MiB data, 464 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s; 10439/32427 objects degraded (32.192%) 2022-05-13T23:18:36.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:18:36 smithi047 ceph-mon[77147]: pgmap v16: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 138 MiB data, 464 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s; 10439/32427 objects degraded (32.192%) 2022-05-13T23:18:36.542 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:18:36 smithi047 conmon[78521]: 2022-05-13T23:18:36.080+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:18:36.542 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:18:36 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-13T23:18:36.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:18:36 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-13T23:18:36.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:18:36 smithi047 conmon[78521]: yield 2022-05-13T23:18:36.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:18:36 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-13T23:18:36.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:18:36 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-13T23:18:36.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:18:36 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-13T23:18:36.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:18:36 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-13T23:18:36.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:18:36 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-13T23:18:36.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:18:36 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-13T23:18:36.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:18:36 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-13T23:18:36.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:18:36 smithi047 conmon[78521]: raise exceptions[0] 2022-05-13T23:18:36.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:18:36 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-13T23:18:36.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:18:36 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-13T23:18:36.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:18:36 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-13T23:18:36.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:18:36 smithi047 conmon[78521]: return (yield from fut) 2022-05-13T23:18:36.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:18:36 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-13T23:18:36.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:18:36 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-13T23:18:36.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:18:36 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:18:37.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:18:37 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:18:37.359 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-13T23:18:37.359 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-13T23:18:37.359 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-13T23:18:37.359 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-13T23:18:37.359 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-13T23:18:37.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-13T23:18:37.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-13T23:18:37.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-13T23:18:37.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-13T23:18:37.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-13T23:18:37.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-13T23:18:37.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-13T23:18:37.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-13T23:18:37.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-13T23:18:37.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-13T23:18:37.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-13T23:18:37.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-13T23:18:37.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:18:37.363 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:18:37 smithi115 ceph-mon[78593]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-13T23:18:37.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:18:37 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:18:37.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-13T23:18:37.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-13T23:18:37.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-13T23:18:37.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-13T23:18:37.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-13T23:18:37.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-13T23:18:37.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-13T23:18:37.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-13T23:18:37.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-13T23:18:37.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-13T23:18:37.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-13T23:18:37.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-13T23:18:37.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-13T23:18:37.548 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-13T23:18:37.548 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-13T23:18:37.548 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-13T23:18:37.548 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-13T23:18:37.548 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:18:37.549 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:18:37 smithi047 ceph-mon[77147]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-13T23:18:38.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:18:38 smithi115 ceph-mon[78593]: pgmap v17: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 138 MiB data, 464 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 3 op/s; 10439/32427 objects degraded (32.192%) 2022-05-13T23:18:38.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:18:38 smithi047 ceph-mon[77147]: pgmap v17: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 138 MiB data, 464 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 3 op/s; 10439/32427 objects degraded (32.192%) 2022-05-13T23:18:39.542 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:18:39 smithi047 conmon[78521]: 2022-05-13T23:18:39.152+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:18:39.542 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:18:39 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-13T23:18:39.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:18:39 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-13T23:18:39.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:18:39 smithi047 conmon[78521]: yield 2022-05-13T23:18:39.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:18:39 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-13T23:18:39.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:18:39 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-13T23:18:39.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:18:39 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-13T23:18:39.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:18:39 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-13T23:18:39.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:18:39 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-13T23:18:39.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:18:39 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-13T23:18:39.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:18:39 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-13T23:18:39.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:18:39 smithi047 conmon[78521]: raise exceptions[0] 2022-05-13T23:18:39.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:18:39 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-13T23:18:39.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:18:39 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-13T23:18:39.547 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:18:39 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-13T23:18:39.547 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:18:39 smithi047 conmon[78521]: return (yield from fut) 2022-05-13T23:18:39.547 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:18:39 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-13T23:18:39.547 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:18:39 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-13T23:18:39.547 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:18:39 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:18:39.548 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:18:39 smithi047 conmon[78521]: 2022-05-13T23:18:39.154+0000 7f5340f7a700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-13T23:18:39.548 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:18:39 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-13T23:18:39.548 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:18:39 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-13T23:18:39.548 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:18:39 smithi047 conmon[78521]: yield 2022-05-13T23:18:39.549 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:18:39 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-13T23:18:39.549 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:18:39 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-13T23:18:39.549 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:18:39 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-13T23:18:39.549 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:18:39 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-13T23:18:39.549 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:18:39 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-13T23:18:39.550 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:18:39 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-13T23:18:39.550 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:18:39 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-13T23:18:39.550 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:18:39 smithi047 conmon[78521]: raise exceptions[0] 2022-05-13T23:18:39.550 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:18:39 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-13T23:18:39.551 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:18:39 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-13T23:18:39.551 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:18:39 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-13T23:18:39.551 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:18:39 smithi047 conmon[78521]: return (yield from fut) 2022-05-13T23:18:39.551 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:18:39 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-13T23:18:39.551 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:18:39 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-13T23:18:39.552 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:18:39 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:18:39.552 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:18:39 smithi047 conmon[78521]: 2022-05-13T23:18:39.552 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:18:39 smithi047 conmon[78521]: During handling of the above exception, another exception occurred: 2022-05-13T23:18:39.552 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:18:39 smithi047 conmon[78521]: 2022-05-13T23:18:39.552 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:18:39 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-13T23:18:39.553 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:18:39 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-13T23:18:39.553 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:18:39 smithi047 conmon[78521]: return f(*arg) 2022-05-13T23:18:39.553 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:18:39 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-13T23:18:39.553 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:18:39 smithi047 conmon[78521]: self._write_client_files(client_files, host) 2022-05-13T23:18:39.554 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:18:39 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-13T23:18:39.554 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:18:39 smithi047 conmon[78521]: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-13T23:18:39.554 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:18:39 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-13T23:18:39.554 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:18:39 smithi047 conmon[78521]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-13T23:18:39.554 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:18:39 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-13T23:18:39.555 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:18:39 smithi047 conmon[78521]: return self.event_loop.get_result(coro) 2022-05-13T23:18:39.555 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:18:39 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-13T23:18:39.555 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:18:39 smithi047 conmon[78521]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-13T23:18:39.555 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:18:39 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-13T23:18:39.556 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:18:39 smithi047 conmon[78521]: return self.__get_result() 2022-05-13T23:18:39.556 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:18:39 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-13T23:18:39.556 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:18:39 smithi047 conmon[78521]: raise self._exception 2022-05-13T23:18:39.556 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:18:39 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-13T23:18:39.556 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:18:39 smithi047 conmon[78521]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-13T23:18:39.557 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:18:39 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-13T23:18:39.557 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:18:39 smithi047 conmon[78521]: conn = await self._remote_connection(host, addr) 2022-05-13T23:18:39.557 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:18:39 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-13T23:18:39.557 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:18:39 smithi047 conmon[78521]: raise 2022-05-13T23:18:39.557 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:18:39 smithi047 conmon[78521]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-13T23:18:39.558 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:18:39 smithi047 conmon[78521]: self.gen.throw(type, value, traceback) 2022-05-13T23:18:39.558 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:18:39 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-13T23:18:39.558 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:18:39 smithi047 conmon[78521]: raise OrchestratorError(msg) 2022-05-13T23:18:39.558 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:18:39 smithi047 conmon[78521]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:18:40.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:18:40 smithi115 ceph-mon[78593]: pgmap v18: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 138 MiB data, 464 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 3 op/s; 10439/32427 objects degraded (32.192%) 2022-05-13T23:18:40.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:18:40 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:18:40.359 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-13T23:18:40.359 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-13T23:18:40.359 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-13T23:18:40.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-13T23:18:40.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-13T23:18:40.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-13T23:18:40.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-13T23:18:40.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-13T23:18:40.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-13T23:18:40.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-13T23:18:40.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-13T23:18:40.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-13T23:18:40.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-13T23:18:40.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-13T23:18:40.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-13T23:18:40.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-13T23:18:40.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-13T23:18:40.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:18:40.364 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:18:40 smithi115 ceph-mon[78593]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-13T23:18:40.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-13T23:18:40.365 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-13T23:18:40.365 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-13T23:18:40.365 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-13T23:18:40.365 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-13T23:18:40.366 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-13T23:18:40.366 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-13T23:18:40.366 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-13T23:18:40.366 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-13T23:18:40.367 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-13T23:18:40.367 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-13T23:18:40.367 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-13T23:18:40.367 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-13T23:18:40.367 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-13T23:18:40.368 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-13T23:18:40.368 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-13T23:18:40.368 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-13T23:18:40.368 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:18:40.368 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-13T23:18:40.369 INFO:journalctl@ceph.mon.c.smithi115.stdout: During handling of the above exception, another exception occurred: 2022-05-13T23:18:40.369 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-13T23:18:40.369 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-13T23:18:40.369 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-13T23:18:40.370 INFO:journalctl@ceph.mon.c.smithi115.stdout: return f(*arg) 2022-05-13T23:18:40.370 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-13T23:18:40.370 INFO:journalctl@ceph.mon.c.smithi115.stdout: self._write_client_files(client_files, host) 2022-05-13T23:18:40.370 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-13T23:18:40.370 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-13T23:18:40.371 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-13T23:18:40.371 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-13T23:18:40.371 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-13T23:18:40.371 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.event_loop.get_result(coro) 2022-05-13T23:18:40.372 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-13T23:18:40.372 INFO:journalctl@ceph.mon.c.smithi115.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-13T23:18:40.372 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-13T23:18:40.372 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.__get_result() 2022-05-13T23:18:40.372 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-13T23:18:40.373 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise self._exception 2022-05-13T23:18:40.373 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-13T23:18:40.374 INFO:journalctl@ceph.mon.c.smithi115.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-13T23:18:40.374 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-13T23:18:40.374 INFO:journalctl@ceph.mon.c.smithi115.stdout: conn = await self._remote_connection(host, addr) 2022-05-13T23:18:40.374 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-13T23:18:40.375 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise 2022-05-13T23:18:40.375 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-13T23:18:40.375 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.gen.throw(type, value, traceback) 2022-05-13T23:18:40.375 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-13T23:18:40.375 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OrchestratorError(msg) 2022-05-13T23:18:40.376 INFO:journalctl@ceph.mon.c.smithi115.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:18:40.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:18:40 smithi047 ceph-mon[77147]: pgmap v18: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 138 MiB data, 464 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 3 op/s; 10439/32427 objects degraded (32.192%) 2022-05-13T23:18:40.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:18:40 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:18:40.542 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-13T23:18:40.542 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-13T23:18:40.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-13T23:18:40.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-13T23:18:40.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-13T23:18:40.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-13T23:18:40.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-13T23:18:40.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-13T23:18:40.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-13T23:18:40.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-13T23:18:40.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-13T23:18:40.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-13T23:18:40.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-13T23:18:40.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-13T23:18:40.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-13T23:18:40.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-13T23:18:40.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-13T23:18:40.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:18:40.547 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:18:40 smithi047 ceph-mon[77147]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-13T23:18:40.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-13T23:18:40.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-13T23:18:40.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-13T23:18:40.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-13T23:18:40.548 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-13T23:18:40.548 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-13T23:18:40.548 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-13T23:18:40.548 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-13T23:18:40.549 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-13T23:18:40.549 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-13T23:18:40.550 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-13T23:18:40.550 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-13T23:18:40.550 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-13T23:18:40.550 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-13T23:18:40.550 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-13T23:18:40.551 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-13T23:18:40.551 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-13T23:18:40.551 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:18:40.551 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-13T23:18:40.551 INFO:journalctl@ceph.mon.b.smithi047.stdout: During handling of the above exception, another exception occurred: 2022-05-13T23:18:40.552 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-13T23:18:40.552 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-13T23:18:40.552 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-13T23:18:40.552 INFO:journalctl@ceph.mon.b.smithi047.stdout: return f(*arg) 2022-05-13T23:18:40.552 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-13T23:18:40.553 INFO:journalctl@ceph.mon.b.smithi047.stdout: self._write_client_files(client_files, host) 2022-05-13T23:18:40.553 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-13T23:18:40.553 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-13T23:18:40.553 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-13T23:18:40.554 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-13T23:18:40.554 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-13T23:18:40.555 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.event_loop.get_result(coro) 2022-05-13T23:18:40.555 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-13T23:18:40.555 INFO:journalctl@ceph.mon.b.smithi047.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-13T23:18:40.555 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-13T23:18:40.555 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.__get_result() 2022-05-13T23:18:40.556 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-13T23:18:40.556 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise self._exception 2022-05-13T23:18:40.556 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-13T23:18:40.556 INFO:journalctl@ceph.mon.b.smithi047.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-13T23:18:40.556 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-13T23:18:40.557 INFO:journalctl@ceph.mon.b.smithi047.stdout: conn = await self._remote_connection(host, addr) 2022-05-13T23:18:40.557 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-13T23:18:40.557 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise 2022-05-13T23:18:40.557 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-13T23:18:40.558 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.gen.throw(type, value, traceback) 2022-05-13T23:18:40.558 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-13T23:18:40.558 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OrchestratorError(msg) 2022-05-13T23:18:40.558 INFO:journalctl@ceph.mon.b.smithi047.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:18:41.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:18:41 smithi115 ceph-mon[78593]: Health check update: Reduced data availability: 7 pgs inactive (PG_AVAILABILITY) 2022-05-13T23:18:41.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:18:41 smithi047 ceph-mon[77147]: Health check update: Reduced data availability: 7 pgs inactive (PG_AVAILABILITY) 2022-05-13T23:18:42.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:18:42 smithi115 ceph-mon[78593]: pgmap v19: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 138 MiB data, 464 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s; 10439/32427 objects degraded (32.192%) 2022-05-13T23:18:42.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:18:42 smithi047 ceph-mon[77147]: pgmap v19: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 138 MiB data, 464 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s; 10439/32427 objects degraded (32.192%) 2022-05-13T23:18:43.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:18:43 smithi047 ceph-mon[77147]: MDS daemon mds.a is removed because it is dead or otherwise unavailable. 2022-05-13T23:18:43.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:18:43 smithi047 ceph-mon[77147]: MDS daemon mds.g is removed because it is dead or otherwise unavailable. 2022-05-13T23:18:43.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:18:43 smithi047 ceph-mon[77147]: MDS daemon mds.d is removed because it is dead or otherwise unavailable. 2022-05-13T23:18:43.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:18:43 smithi047 ceph-mon[77147]: MDS daemon mds.j is removed because it is dead or otherwise unavailable. 2022-05-13T23:18:43.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:18:43 smithi047 ceph-mon[77147]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2022-05-13T23:18:43.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:18:43 smithi115 ceph-mon[78593]: MDS daemon mds.a is removed because it is dead or otherwise unavailable. 2022-05-13T23:18:43.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:18:43 smithi115 ceph-mon[78593]: MDS daemon mds.g is removed because it is dead or otherwise unavailable. 2022-05-13T23:18:43.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:18:43 smithi115 ceph-mon[78593]: MDS daemon mds.d is removed because it is dead or otherwise unavailable. 2022-05-13T23:18:43.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:18:43 smithi115 ceph-mon[78593]: MDS daemon mds.j is removed because it is dead or otherwise unavailable. 2022-05-13T23:18:43.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:18:43 smithi115 ceph-mon[78593]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2022-05-13T23:18:44.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:18:44 smithi047 ceph-mon[77147]: pgmap v20: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 138 MiB data, 464 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 3 op/s; 10439/32427 objects degraded (32.192%) 2022-05-13T23:18:44.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:18:44 smithi047 ceph-mon[77147]: Health check update: Degraded data redundancy: 10439/32427 objects degraded (32.192%), 67 pgs degraded, 28 pgs undersized (PG_DEGRADED) 2022-05-13T23:18:44.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:18:44 smithi115 ceph-mon[78593]: pgmap v20: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 138 MiB data, 464 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 3 op/s; 10439/32427 objects degraded (32.192%) 2022-05-13T23:18:44.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:18:44 smithi115 ceph-mon[78593]: Health check update: Degraded data redundancy: 10439/32427 objects degraded (32.192%), 67 pgs degraded, 28 pgs undersized (PG_DEGRADED) 2022-05-13T23:18:46.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:18:46 smithi047 ceph-mon[77147]: pgmap v21: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 138 MiB data, 464 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s; 10439/32427 objects degraded (32.192%) 2022-05-13T23:18:46.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:18:46 smithi115 ceph-mon[78593]: pgmap v21: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 138 MiB data, 464 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s; 10439/32427 objects degraded (32.192%) 2022-05-13T23:18:48.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:18:48 smithi047 ceph-mon[77147]: pgmap v22: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 138 MiB data, 464 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 3 op/s; 10439/32427 objects degraded (32.192%) 2022-05-13T23:18:48.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:18:48 smithi047 ceph-mon[77147]: Health check update: Reduced data availability: 23 pgs inactive (PG_AVAILABILITY) 2022-05-13T23:18:48.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:18:48 smithi115 ceph-mon[78593]: pgmap v22: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 138 MiB data, 464 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 3 op/s; 10439/32427 objects degraded (32.192%) 2022-05-13T23:18:48.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:18:48 smithi115 ceph-mon[78593]: Health check update: Reduced data availability: 23 pgs inactive (PG_AVAILABILITY) 2022-05-13T23:18:50.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:18:50 smithi047 ceph-mon[77147]: pgmap v23: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 138 MiB data, 464 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 3 op/s; 10439/32427 objects degraded (32.192%) 2022-05-13T23:18:50.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:18:50 smithi115 ceph-mon[78593]: pgmap v23: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 138 MiB data, 464 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 3 op/s; 10439/32427 objects degraded (32.192%) 2022-05-13T23:18:52.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:18:52 smithi047 ceph-mon[77147]: pgmap v24: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 138 MiB data, 464 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s; 10439/32427 objects degraded (32.192%) 2022-05-13T23:18:52.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:18:52 smithi115 ceph-mon[78593]: pgmap v24: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 138 MiB data, 464 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s; 10439/32427 objects degraded (32.192%) 2022-05-13T23:18:53.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:18:53 smithi047 ceph-mon[77147]: Health check update: Degraded data redundancy: 10439/32427 objects degraded (32.192%), 67 pgs degraded, 68 pgs undersized (PG_DEGRADED) 2022-05-13T23:18:53.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:18:53 smithi115 ceph-mon[78593]: Health check update: Degraded data redundancy: 10439/32427 objects degraded (32.192%), 67 pgs degraded, 68 pgs undersized (PG_DEGRADED) 2022-05-13T23:18:54.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:18:54 smithi047 ceph-mon[77147]: pgmap v25: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 138 MiB data, 464 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 3 op/s; 10439/32427 objects degraded (32.192%) 2022-05-13T23:18:54.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:18:54 smithi115 ceph-mon[78593]: pgmap v25: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 138 MiB data, 464 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 3 op/s; 10439/32427 objects degraded (32.192%) 2022-05-13T23:18:56.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:18:56 smithi047 ceph-mon[77147]: pgmap v26: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 138 MiB data, 464 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s; 10439/32427 objects degraded (32.192%) 2022-05-13T23:18:56.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:18:56 smithi115 ceph-mon[78593]: pgmap v26: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 138 MiB data, 464 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s; 10439/32427 objects degraded (32.192%) 2022-05-13T23:18:58.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:18:58 smithi047 ceph-mon[77147]: pgmap v27: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 138 MiB data, 464 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 3 op/s; 10439/32427 objects degraded (32.192%) 2022-05-13T23:18:58.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:18:58 smithi115 ceph-mon[78593]: pgmap v27: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 138 MiB data, 464 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 3 op/s; 10439/32427 objects degraded (32.192%) 2022-05-13T23:19:00.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:19:00 smithi047 ceph-mon[77147]: pgmap v28: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 138 MiB data, 464 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 3 op/s; 10439/32427 objects degraded (32.192%) 2022-05-13T23:19:00.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:19:00 smithi115 ceph-mon[78593]: pgmap v28: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 138 MiB data, 464 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 3 op/s; 10439/32427 objects degraded (32.192%) 2022-05-13T23:19:02.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:19:02 smithi047 ceph-mon[77147]: pgmap v29: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 138 MiB data, 464 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s; 10439/32427 objects degraded (32.192%) 2022-05-13T23:19:02.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:19:02 smithi115 ceph-mon[78593]: pgmap v29: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 138 MiB data, 464 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s; 10439/32427 objects degraded (32.192%) 2022-05-13T23:19:04.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:19:03 smithi047 ceph-mon[77147]: pgmap v30: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 138 MiB data, 464 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 3 op/s; 10439/32427 objects degraded (32.192%) 2022-05-13T23:19:04.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:19:03 smithi115 ceph-mon[78593]: pgmap v30: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 138 MiB data, 464 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 3 op/s; 10439/32427 objects degraded (32.192%) 2022-05-13T23:19:06.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:19:06 smithi115 ceph-mon[78593]: pgmap v31: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 138 MiB data, 464 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s; 10439/32427 objects degraded (32.192%) 2022-05-13T23:19:06.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:19:06 smithi047 ceph-mon[77147]: pgmap v31: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 138 MiB data, 464 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s; 10439/32427 objects degraded (32.192%) 2022-05-13T23:19:08.362 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:19:08 smithi115 ceph-mon[78593]: pgmap v32: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 138 MiB data, 464 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 3 op/s; 10439/32427 objects degraded (32.192%) 2022-05-13T23:19:08.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:19:08 smithi047 ceph-mon[77147]: pgmap v32: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 138 MiB data, 464 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 3 op/s; 10439/32427 objects degraded (32.192%) 2022-05-13T23:19:10.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:19:10 smithi115 ceph-mon[78593]: pgmap v33: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 138 MiB data, 464 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 3 op/s; 10439/32427 objects degraded (32.192%) 2022-05-13T23:19:10.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:19:10 smithi047 ceph-mon[77147]: pgmap v33: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 138 MiB data, 464 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 3 op/s; 10439/32427 objects degraded (32.192%) 2022-05-13T23:19:12.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:19:12 smithi115 ceph-mon[78593]: pgmap v34: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 138 MiB data, 464 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s; 10439/32427 objects degraded (32.192%) 2022-05-13T23:19:12.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:19:12 smithi047 ceph-mon[77147]: pgmap v34: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 138 MiB data, 464 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s; 10439/32427 objects degraded (32.192%) 2022-05-13T23:19:14.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:19:13 smithi047 ceph-mon[77147]: pgmap v35: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 138 MiB data, 464 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 3 op/s; 10439/32427 objects degraded (32.192%) 2022-05-13T23:19:14.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:19:13 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-13T23:19:14.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:19:13 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-13T23:19:14.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:19:13 smithi115 ceph-mon[78593]: pgmap v35: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 138 MiB data, 464 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 3 op/s; 10439/32427 objects degraded (32.192%) 2022-05-13T23:19:14.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:19:13 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-13T23:19:14.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:19:13 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-13T23:19:16.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:19:16 smithi115 ceph-mon[78593]: pgmap v36: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 138 MiB data, 464 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s; 10439/32427 objects degraded (32.192%) 2022-05-13T23:19:16.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:19:16 smithi047 ceph-mon[77147]: pgmap v36: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 138 MiB data, 464 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s; 10439/32427 objects degraded (32.192%) 2022-05-13T23:19:18.363 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:19:18 smithi115 ceph-mon[78593]: pgmap v37: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 138 MiB data, 464 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 3 op/s; 10439/32427 objects degraded (32.192%) 2022-05-13T23:19:18.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:19:18 smithi047 ceph-mon[77147]: pgmap v37: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 138 MiB data, 464 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 3 op/s; 10439/32427 objects degraded (32.192%) 2022-05-13T23:19:20.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:19:20 smithi115 ceph-mon[78593]: pgmap v38: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 138 MiB data, 464 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 3 op/s; 10439/32427 objects degraded (32.192%) 2022-05-13T23:19:20.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:19:20 smithi047 ceph-mon[77147]: pgmap v38: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 138 MiB data, 464 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 3 op/s; 10439/32427 objects degraded (32.192%) 2022-05-13T23:19:22.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:19:22 smithi115 ceph-mon[78593]: pgmap v39: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 138 MiB data, 464 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s; 10439/32427 objects degraded (32.192%) 2022-05-13T23:19:22.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:19:22 smithi047 ceph-mon[77147]: pgmap v39: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 138 MiB data, 464 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s; 10439/32427 objects degraded (32.192%) 2022-05-13T23:19:24.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:19:23 smithi047 ceph-mon[77147]: pgmap v40: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 138 MiB data, 464 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 3 op/s; 10439/32427 objects degraded (32.192%) 2022-05-13T23:19:24.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:19:23 smithi115 ceph-mon[78593]: pgmap v40: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 138 MiB data, 464 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 3 op/s; 10439/32427 objects degraded (32.192%) 2022-05-13T23:19:26.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:19:26 smithi115 ceph-mon[78593]: pgmap v41: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 138 MiB data, 464 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s; 10439/32427 objects degraded (32.192%) 2022-05-13T23:19:26.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:19:26 smithi047 ceph-mon[77147]: pgmap v41: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 138 MiB data, 464 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s; 10439/32427 objects degraded (32.192%) 2022-05-13T23:19:28.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:19:28 smithi115 ceph-mon[78593]: pgmap v42: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 138 MiB data, 464 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 3 op/s; 10439/32427 objects degraded (32.192%) 2022-05-13T23:19:28.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:19:28 smithi047 ceph-mon[77147]: pgmap v42: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 138 MiB data, 464 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 3 op/s; 10439/32427 objects degraded (32.192%) 2022-05-13T23:19:30.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:19:30 smithi115 ceph-mon[78593]: pgmap v43: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 138 MiB data, 464 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 3 op/s; 10439/32427 objects degraded (32.192%) 2022-05-13T23:19:30.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:19:30 smithi047 ceph-mon[77147]: pgmap v43: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 138 MiB data, 464 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 3 op/s; 10439/32427 objects degraded (32.192%) 2022-05-13T23:19:32.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:19:32 smithi115 ceph-mon[78593]: pgmap v44: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 138 MiB data, 464 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s; 10439/32427 objects degraded (32.192%) 2022-05-13T23:19:32.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:19:32 smithi047 ceph-mon[77147]: pgmap v44: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 138 MiB data, 464 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s; 10439/32427 objects degraded (32.192%) 2022-05-13T23:19:34.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:19:33 smithi047 ceph-mon[77147]: pgmap v45: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 138 MiB data, 464 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 3 op/s; 10439/32427 objects degraded (32.192%) 2022-05-13T23:19:34.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:19:33 smithi115 ceph-mon[78593]: pgmap v45: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 138 MiB data, 464 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 3 op/s; 10439/32427 objects degraded (32.192%) 2022-05-13T23:19:36.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:19:36 smithi115 ceph-mon[78593]: pgmap v46: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 138 MiB data, 464 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s; 10439/32427 objects degraded (32.192%) 2022-05-13T23:19:36.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:19:36 smithi047 ceph-mon[77147]: pgmap v46: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 138 MiB data, 464 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s; 10439/32427 objects degraded (32.192%) 2022-05-13T23:19:37.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:19:37 smithi115 ceph-mon[78593]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2022-05-13T23:19:37.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:19:37 smithi047 ceph-mon[77147]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2022-05-13T23:19:38.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:19:38 smithi115 ceph-mon[78593]: pgmap v47: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 138 MiB data, 464 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 3 op/s; 10439/32427 objects degraded (32.192%) 2022-05-13T23:19:38.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:19:38 smithi047 ceph-mon[77147]: pgmap v47: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 138 MiB data, 464 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 3 op/s; 10439/32427 objects degraded (32.192%) 2022-05-13T23:19:40.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:19:40 smithi115 ceph-mon[78593]: pgmap v48: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 138 MiB data, 464 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 3 op/s; 10439/32427 objects degraded (32.192%) 2022-05-13T23:19:40.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:19:40 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-13T23:19:40.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:19:40 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-13T23:19:40.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:19:40 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-13T23:19:40.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:19:40 smithi047 ceph-mon[77147]: pgmap v48: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 138 MiB data, 464 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 3 op/s; 10439/32427 objects degraded (32.192%) 2022-05-13T23:19:40.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:19:40 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-13T23:19:40.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:19:40 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-13T23:19:40.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:19:40 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-13T23:19:42.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:19:42 smithi115 ceph-mon[78593]: pgmap v49: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 138 MiB data, 464 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s; 10439/32427 objects degraded (32.192%) 2022-05-13T23:19:42.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:19:42 smithi047 ceph-mon[77147]: pgmap v49: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 138 MiB data, 464 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s; 10439/32427 objects degraded (32.192%) 2022-05-13T23:19:42.542 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:19:42 smithi047 conmon[78521]: 2022-05-13T23:19:42.255+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:19:42.542 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:19:42 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-13T23:19:42.542 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:19:42 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-13T23:19:42.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:19:42 smithi047 conmon[78521]: yield 2022-05-13T23:19:42.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:19:42 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-13T23:19:42.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:19:42 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-13T23:19:42.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:19:42 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-13T23:19:42.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:19:42 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-13T23:19:42.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:19:42 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-13T23:19:42.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:19:42 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-13T23:19:42.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:19:42 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-13T23:19:42.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:19:42 smithi047 conmon[78521]: raise exceptions[0] 2022-05-13T23:19:42.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:19:42 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-13T23:19:42.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:19:42 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-13T23:19:42.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:19:42 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-13T23:19:42.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:19:42 smithi047 conmon[78521]: return (yield from fut) 2022-05-13T23:19:42.547 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:19:42 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-13T23:19:42.547 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:19:42 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-13T23:19:42.547 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:19:42 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:19:43.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:19:43 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:19:43.358 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-13T23:19:43.359 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-13T23:19:43.359 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-13T23:19:43.359 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-13T23:19:43.359 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-13T23:19:43.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-13T23:19:43.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-13T23:19:43.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-13T23:19:43.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-13T23:19:43.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-13T23:19:43.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-13T23:19:43.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-13T23:19:43.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-13T23:19:43.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-13T23:19:43.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-13T23:19:43.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-13T23:19:43.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-13T23:19:43.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:19:43.363 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:19:43 smithi115 ceph-mon[78593]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-13T23:19:43.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:19:43 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:19:43.542 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-13T23:19:43.542 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-13T23:19:43.542 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-13T23:19:43.542 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-13T23:19:43.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-13T23:19:43.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-13T23:19:43.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-13T23:19:43.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-13T23:19:43.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-13T23:19:43.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-13T23:19:43.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-13T23:19:43.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-13T23:19:43.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-13T23:19:43.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-13T23:19:43.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-13T23:19:43.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-13T23:19:43.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-13T23:19:43.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:19:43.546 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:19:43 smithi047 ceph-mon[77147]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-13T23:19:44.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:19:44 smithi115 ceph-mon[78593]: pgmap v50: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 138 MiB data, 464 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 3 op/s; 10439/32427 objects degraded (32.192%) 2022-05-13T23:19:44.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:19:44 smithi047 ceph-mon[77147]: pgmap v50: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 138 MiB data, 464 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 3 op/s; 10439/32427 objects degraded (32.192%) 2022-05-13T23:19:45.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:19:45 smithi115 ceph-mon[78593]: mgrmap e20: y(active, since 91s), standbys: z 2022-05-13T23:19:45.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:19:45 smithi047 ceph-mon[77147]: mgrmap e20: y(active, since 91s), standbys: z 2022-05-13T23:19:45.542 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:19:45 smithi047 conmon[78521]: 2022-05-13T23:19:45.327+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:19:45.542 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:19:45 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-13T23:19:45.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:19:45 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-13T23:19:45.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:19:45 smithi047 conmon[78521]: yield 2022-05-13T23:19:45.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:19:45 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-13T23:19:45.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:19:45 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-13T23:19:45.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:19:45 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-13T23:19:45.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:19:45 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-13T23:19:45.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:19:45 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-13T23:19:45.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:19:45 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-13T23:19:45.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:19:45 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-13T23:19:45.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:19:45 smithi047 conmon[78521]: raise exceptions[0] 2022-05-13T23:19:45.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:19:45 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-13T23:19:45.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:19:45 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-13T23:19:45.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:19:45 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-13T23:19:45.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:19:45 smithi047 conmon[78521]: return (yield from fut) 2022-05-13T23:19:45.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:19:45 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-13T23:19:45.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:19:45 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-13T23:19:45.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:19:45 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:19:45.547 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:19:45 smithi047 conmon[78521]: 2022-05-13T23:19:45.328+0000 7f5340779700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-13T23:19:45.547 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:19:45 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-13T23:19:45.547 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:19:45 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-13T23:19:45.547 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:19:45 smithi047 conmon[78521]: yield 2022-05-13T23:19:45.548 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:19:45 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-13T23:19:45.548 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:19:45 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-13T23:19:45.548 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:19:45 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-13T23:19:45.548 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:19:45 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-13T23:19:45.548 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:19:45 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-13T23:19:45.549 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:19:45 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-13T23:19:45.549 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:19:45 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-13T23:19:45.549 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:19:45 smithi047 conmon[78521]: raise exceptions[0] 2022-05-13T23:19:45.550 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:19:45 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-13T23:19:45.550 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:19:45 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-13T23:19:45.551 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:19:45 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-13T23:19:45.551 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:19:45 smithi047 conmon[78521]: return (yield from fut) 2022-05-13T23:19:45.551 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:19:45 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-13T23:19:45.551 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:19:45 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-13T23:19:45.551 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:19:45 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:19:45.552 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:19:45 smithi047 conmon[78521]: 2022-05-13T23:19:45.552 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:19:45 smithi047 conmon[78521]: During handling of the above exception, another exception occurred: 2022-05-13T23:19:45.552 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:19:45 smithi047 conmon[78521]: 2022-05-13T23:19:45.553 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:19:45 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-13T23:19:45.553 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:19:45 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-13T23:19:45.553 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:19:45 smithi047 conmon[78521]: return f(*arg) 2022-05-13T23:19:45.554 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:19:45 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-13T23:19:45.554 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:19:45 smithi047 conmon[78521]: self._write_client_files(client_files, host) 2022-05-13T23:19:45.554 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:19:45 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-13T23:19:45.555 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:19:45 smithi047 conmon[78521]: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-13T23:19:45.555 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:19:45 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-13T23:19:45.555 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:19:45 smithi047 conmon[78521]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-13T23:19:45.556 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:19:45 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-13T23:19:45.556 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:19:45 smithi047 conmon[78521]: return self.event_loop.get_result(coro) 2022-05-13T23:19:45.557 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:19:45 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-13T23:19:45.557 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:19:45 smithi047 conmon[78521]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-13T23:19:45.557 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:19:45 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-13T23:19:45.557 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:19:45 smithi047 conmon[78521]: return self.__get_result() 2022-05-13T23:19:45.558 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:19:45 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-13T23:19:45.558 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:19:45 smithi047 conmon[78521]: raise self._exception 2022-05-13T23:19:45.558 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:19:45 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-13T23:19:45.558 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:19:45 smithi047 conmon[78521]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-13T23:19:45.559 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:19:45 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-13T23:19:45.559 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:19:45 smithi047 conmon[78521]: conn = await self._remote_connection(host, addr) 2022-05-13T23:19:45.559 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:19:45 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-13T23:19:45.560 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:19:45 smithi047 conmon[78521]: raise 2022-05-13T23:19:45.560 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:19:45 smithi047 conmon[78521]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-13T23:19:45.560 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:19:45 smithi047 conmon[78521]: self.gen.throw(type, value, traceback) 2022-05-13T23:19:45.560 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:19:45 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-13T23:19:45.560 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:19:45 smithi047 conmon[78521]: raise OrchestratorError(msg) 2022-05-13T23:19:45.561 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:19:45 smithi047 conmon[78521]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:19:46.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:19:46 smithi115 ceph-mon[78593]: pgmap v51: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 138 MiB data, 465 MiB used, 715 GiB / 715 GiB avail; 2.7 KiB/s rd, 1.2 KiB/s wr, 6 op/s; 10439/32427 objects degraded (32.192%) 2022-05-13T23:19:46.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:19:46 smithi047 ceph-mon[77147]: pgmap v51: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 138 MiB data, 465 MiB used, 715 GiB / 715 GiB avail; 2.7 KiB/s rd, 1.2 KiB/s wr, 6 op/s; 10439/32427 objects degraded (32.192%) 2022-05-13T23:19:47.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:19:47 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:19:47.542 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-13T23:19:47.542 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-13T23:19:47.542 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-13T23:19:47.542 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-13T23:19:47.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-13T23:19:47.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-13T23:19:47.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-13T23:19:47.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-13T23:19:47.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-13T23:19:47.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-13T23:19:47.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-13T23:19:47.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-13T23:19:47.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-13T23:19:47.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-13T23:19:47.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-13T23:19:47.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-13T23:19:47.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-13T23:19:47.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:19:47.546 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:19:47 smithi047 ceph-mon[77147]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-13T23:19:47.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-13T23:19:47.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-13T23:19:47.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-13T23:19:47.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-13T23:19:47.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-13T23:19:47.548 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-13T23:19:47.548 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-13T23:19:47.548 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-13T23:19:47.548 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-13T23:19:47.549 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-13T23:19:47.549 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-13T23:19:47.549 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-13T23:19:47.549 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-13T23:19:47.549 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-13T23:19:47.550 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-13T23:19:47.550 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-13T23:19:47.550 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-13T23:19:47.550 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:19:47.550 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-13T23:19:47.551 INFO:journalctl@ceph.mon.b.smithi047.stdout: During handling of the above exception, another exception occurred: 2022-05-13T23:19:47.551 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-13T23:19:47.551 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-13T23:19:47.551 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-13T23:19:47.552 INFO:journalctl@ceph.mon.b.smithi047.stdout: return f(*arg) 2022-05-13T23:19:47.552 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-13T23:19:47.552 INFO:journalctl@ceph.mon.b.smithi047.stdout: self._write_client_files(client_files, host) 2022-05-13T23:19:47.552 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-13T23:19:47.552 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-13T23:19:47.553 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-13T23:19:47.553 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-13T23:19:47.553 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-13T23:19:47.553 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.event_loop.get_result(coro) 2022-05-13T23:19:47.553 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-13T23:19:47.554 INFO:journalctl@ceph.mon.b.smithi047.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-13T23:19:47.554 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-13T23:19:47.554 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.__get_result() 2022-05-13T23:19:47.554 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-13T23:19:47.555 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise self._exception 2022-05-13T23:19:47.555 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-13T23:19:47.555 INFO:journalctl@ceph.mon.b.smithi047.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-13T23:19:47.555 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-13T23:19:47.555 INFO:journalctl@ceph.mon.b.smithi047.stdout: conn = await self._remote_connection(host, addr) 2022-05-13T23:19:47.556 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-13T23:19:47.556 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise 2022-05-13T23:19:47.556 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-13T23:19:47.557 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.gen.throw(type, value, traceback) 2022-05-13T23:19:47.557 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-13T23:19:47.558 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OrchestratorError(msg) 2022-05-13T23:19:47.558 INFO:journalctl@ceph.mon.b.smithi047.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:19:47.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:19:47 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:19:47.608 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-13T23:19:47.609 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-13T23:19:47.609 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-13T23:19:47.609 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-13T23:19:47.609 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-13T23:19:47.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-13T23:19:47.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-13T23:19:47.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-13T23:19:47.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-13T23:19:47.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-13T23:19:47.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-13T23:19:47.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-13T23:19:47.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-13T23:19:47.613 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-13T23:19:47.613 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-13T23:19:47.613 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-13T23:19:47.614 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-13T23:19:47.614 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:19:47.614 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:19:47 smithi115 ceph-mon[78593]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-13T23:19:47.614 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-13T23:19:47.614 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-13T23:19:47.615 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-13T23:19:47.615 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-13T23:19:47.615 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-13T23:19:47.615 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-13T23:19:47.615 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-13T23:19:47.616 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-13T23:19:47.616 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-13T23:19:47.616 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-13T23:19:47.616 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-13T23:19:47.616 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-13T23:19:47.617 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-13T23:19:47.617 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-13T23:19:47.617 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-13T23:19:47.617 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-13T23:19:47.618 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-13T23:19:47.618 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:19:47.618 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-13T23:19:47.618 INFO:journalctl@ceph.mon.c.smithi115.stdout: During handling of the above exception, another exception occurred: 2022-05-13T23:19:47.618 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-13T23:19:47.619 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-13T23:19:47.619 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-13T23:19:47.619 INFO:journalctl@ceph.mon.c.smithi115.stdout: return f(*arg) 2022-05-13T23:19:47.619 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-13T23:19:47.619 INFO:journalctl@ceph.mon.c.smithi115.stdout: self._write_client_files(client_files, host) 2022-05-13T23:19:47.620 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-13T23:19:47.620 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-13T23:19:47.620 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-13T23:19:47.620 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-13T23:19:47.620 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-13T23:19:47.621 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.event_loop.get_result(coro) 2022-05-13T23:19:47.621 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-13T23:19:47.621 INFO:journalctl@ceph.mon.c.smithi115.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-13T23:19:47.621 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-13T23:19:47.622 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.__get_result() 2022-05-13T23:19:47.622 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-13T23:19:47.622 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise self._exception 2022-05-13T23:19:47.622 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-13T23:19:47.622 INFO:journalctl@ceph.mon.c.smithi115.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-13T23:19:47.623 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-13T23:19:47.623 INFO:journalctl@ceph.mon.c.smithi115.stdout: conn = await self._remote_connection(host, addr) 2022-05-13T23:19:47.624 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-13T23:19:47.624 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise 2022-05-13T23:19:47.624 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-13T23:19:47.624 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.gen.throw(type, value, traceback) 2022-05-13T23:19:47.624 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-13T23:19:47.625 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OrchestratorError(msg) 2022-05-13T23:19:47.625 INFO:journalctl@ceph.mon.c.smithi115.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:19:48.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:19:48 smithi047 ceph-mon[77147]: pgmap v52: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 138 MiB data, 465 MiB used, 715 GiB / 715 GiB avail; 1.8 KiB/s rd, 1.2 KiB/s wr, 4 op/s; 10439/32427 objects degraded (32.192%) 2022-05-13T23:19:48.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:19:48 smithi115 ceph-mon[78593]: pgmap v52: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 138 MiB data, 465 MiB used, 715 GiB / 715 GiB avail; 1.8 KiB/s rd, 1.2 KiB/s wr, 4 op/s; 10439/32427 objects degraded (32.192%) 2022-05-13T23:19:50.546 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:19:50 smithi047 ceph-mon[77147]: pgmap v53: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 138 MiB data, 465 MiB used, 715 GiB / 715 GiB avail; 1.8 KiB/s rd, 1.2 KiB/s wr, 4 op/s; 10439/32427 objects degraded (32.192%) 2022-05-13T23:19:50.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:19:50 smithi115 ceph-mon[78593]: pgmap v53: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 138 MiB data, 465 MiB used, 715 GiB / 715 GiB avail; 1.8 KiB/s rd, 1.2 KiB/s wr, 4 op/s; 10439/32427 objects degraded (32.192%) 2022-05-13T23:19:52.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:19:52 smithi047 ceph-mon[77147]: pgmap v54: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 138 MiB data, 465 MiB used, 715 GiB / 715 GiB avail; 2.7 KiB/s rd, 1.2 KiB/s wr, 6 op/s; 10439/32427 objects degraded (32.192%) 2022-05-13T23:19:52.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:19:52 smithi115 ceph-mon[78593]: pgmap v54: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 138 MiB data, 465 MiB used, 715 GiB / 715 GiB avail; 2.7 KiB/s rd, 1.2 KiB/s wr, 6 op/s; 10439/32427 objects degraded (32.192%) 2022-05-13T23:19:54.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:19:53 smithi047 ceph-mon[77147]: pgmap v55: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 138 MiB data, 465 MiB used, 715 GiB / 715 GiB avail; 1.8 KiB/s rd, 1.2 KiB/s wr, 4 op/s; 10439/32427 objects degraded (32.192%) 2022-05-13T23:19:54.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:19:53 smithi115 ceph-mon[78593]: pgmap v55: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 138 MiB data, 465 MiB used, 715 GiB / 715 GiB avail; 1.8 KiB/s rd, 1.2 KiB/s wr, 4 op/s; 10439/32427 objects degraded (32.192%) 2022-05-13T23:19:56.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:19:56 smithi115 ceph-mon[78593]: pgmap v56: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 138 MiB data, 465 MiB used, 715 GiB / 715 GiB avail; 2.7 KiB/s rd, 1.2 KiB/s wr, 6 op/s; 10439/32427 objects degraded (32.192%) 2022-05-13T23:19:56.548 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:19:56 smithi047 ceph-mon[77147]: pgmap v56: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 138 MiB data, 465 MiB used, 715 GiB / 715 GiB avail; 2.7 KiB/s rd, 1.2 KiB/s wr, 6 op/s; 10439/32427 objects degraded (32.192%) 2022-05-13T23:19:58.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:19:58 smithi115 ceph-mon[78593]: pgmap v57: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 138 MiB data, 465 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 3 op/s; 10439/32427 objects degraded (32.192%) 2022-05-13T23:19:58.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:19:58 smithi047 ceph-mon[77147]: pgmap v57: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 138 MiB data, 465 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 3 op/s; 10439/32427 objects degraded (32.192%) 2022-05-13T23:20:00.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:20:00 smithi115 ceph-mon[78593]: pgmap v58: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 138 MiB data, 465 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 3 op/s; 10439/32427 objects degraded (32.192%) 2022-05-13T23:20:00.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:20:00 smithi115 ceph-mon[78593]: Health detail: HEALTH_WARN 1/3 mons down, quorum b,c; 4 osds down; 1 host (4 osds) down; Reduced data availability: 23 pgs inactive; Degraded data redundancy: 10439/32427 objects degraded (32.192%), 67 pgs degraded, 68 pgs undersized 2022-05-13T23:20:00.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:20:00 smithi115 ceph-mon[78593]: [WRN] MON_DOWN: 1/3 mons down, quorum b,c 2022-05-13T23:20:00.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:20:00 smithi115 ceph-mon[78593]: mon.a (rank 0) addr [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] is down (out of quorum) 2022-05-13T23:20:00.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:20:00 smithi115 ceph-mon[78593]: [WRN] OSD_DOWN: 4 osds down 2022-05-13T23:20:00.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:20:00 smithi115 ceph-mon[78593]: osd.0 (root=default,host=smithi002) is down 2022-05-13T23:20:00.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:20:00 smithi115 ceph-mon[78593]: osd.3 (root=default,host=smithi002) is down 2022-05-13T23:20:00.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:20:00 smithi115 ceph-mon[78593]: osd.6 (root=default,host=smithi002) is down 2022-05-13T23:20:00.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:20:00 smithi115 ceph-mon[78593]: osd.9 (root=default,host=smithi002) is down 2022-05-13T23:20:00.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:20:00 smithi115 ceph-mon[78593]: [WRN] OSD_HOST_DOWN: 1 host (4 osds) down 2022-05-13T23:20:00.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:20:00 smithi115 ceph-mon[78593]: host smithi002 (root=default) (4 osds) is down 2022-05-13T23:20:00.362 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:20:00 smithi115 ceph-mon[78593]: [WRN] PG_AVAILABILITY: Reduced data availability: 23 pgs inactive 2022-05-13T23:20:00.362 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:20:00 smithi115 ceph-mon[78593]: pg 2.1 is stuck inactive for 4m, current state unknown, last acting [] 2022-05-13T23:20:00.363 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:20:00 smithi115 ceph-mon[78593]: pg 2.4 is stuck inactive for 5m, current state undersized+peered, last acting [2] 2022-05-13T23:20:00.364 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:20:00 smithi115 ceph-mon[78593]: pg 2.5 is stuck inactive for 2m, current state undersized+degraded+peered, last acting [8] 2022-05-13T23:20:00.364 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:20:00 smithi115 ceph-mon[78593]: pg 2.6 is stuck inactive for 2m, current state undersized+degraded+peered, last acting [8] 2022-05-13T23:20:00.364 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:20:00 smithi115 ceph-mon[78593]: pg 2.9 is stuck inactive for 2m, current state undersized+degraded+peered, last acting [10] 2022-05-13T23:20:00.364 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:20:00 smithi115 ceph-mon[78593]: pg 2.e is stuck inactive for 2m, current state undersized+degraded+peered, last acting [2] 2022-05-13T23:20:00.365 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:20:00 smithi115 ceph-mon[78593]: pg 2.19 is stuck inactive for 2m, current state undersized+degraded+peered, last acting [4] 2022-05-13T23:20:00.365 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:20:00 smithi115 ceph-mon[78593]: pg 2.1e is stuck inactive for 2m, current state undersized+degraded+peered, last acting [5] 2022-05-13T23:20:00.365 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:20:00 smithi115 ceph-mon[78593]: pg 3.4 is stuck inactive for 2m, current state undersized+degraded+peered, last acting [2] 2022-05-13T23:20:00.365 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:20:00 smithi115 ceph-mon[78593]: pg 3.5 is stuck inactive for 2m, current state undersized+degraded+peered, last acting [10] 2022-05-13T23:20:00.366 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:20:00 smithi115 ceph-mon[78593]: pg 3.6 is stuck inactive for 2m, current state undersized+degraded+peered, last acting [4] 2022-05-13T23:20:00.366 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:20:00 smithi115 ceph-mon[78593]: pg 3.b is stuck inactive for 2m, current state undersized+degraded+peered, last acting [10] 2022-05-13T23:20:00.366 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:20:00 smithi115 ceph-mon[78593]: pg 3.12 is stuck inactive for 2m, current state undersized+degraded+peered, last acting [7] 2022-05-13T23:20:00.366 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:20:00 smithi115 ceph-mon[78593]: pg 3.18 is stuck inactive for 2m, current state undersized+degraded+peered, last acting [11] 2022-05-13T23:20:00.366 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:20:00 smithi115 ceph-mon[78593]: pg 3.19 is stuck inactive for 2m, current state undersized+degraded+peered, last acting [10] 2022-05-13T23:20:00.367 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:20:00 smithi115 ceph-mon[78593]: pg 3.1f is stuck inactive for 2m, current state undersized+degraded+peered, last acting [1] 2022-05-13T23:20:00.368 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:20:00 smithi115 ceph-mon[78593]: pg 3.22 is stuck inactive for 2m, current state undersized+degraded+peered, last acting [2] 2022-05-13T23:20:00.368 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:20:00 smithi115 ceph-mon[78593]: pg 3.23 is stuck inactive for 2m, current state undersized+degraded+peered, last acting [4] 2022-05-13T23:20:00.368 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:20:00 smithi115 ceph-mon[78593]: pg 3.24 is stuck inactive for 2m, current state undersized+degraded+peered, last acting [4] 2022-05-13T23:20:00.368 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:20:00 smithi115 ceph-mon[78593]: pg 3.28 is stuck inactive for 2m, current state undersized+degraded+peered, last acting [5] 2022-05-13T23:20:00.368 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:20:00 smithi115 ceph-mon[78593]: pg 3.2b is stuck inactive for 2m, current state undersized+degraded+peered, last acting [7] 2022-05-13T23:20:00.369 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:20:00 smithi115 ceph-mon[78593]: pg 3.2d is stuck inactive for 2m, current state undersized+degraded+peered, last acting [2] 2022-05-13T23:20:00.369 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:20:00 smithi115 ceph-mon[78593]: pg 3.3b is stuck inactive for 2m, current state undersized+degraded+peered, last acting [8] 2022-05-13T23:20:00.369 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:20:00 smithi115 ceph-mon[78593]: [WRN] PG_DEGRADED: Degraded data redundancy: 10439/32427 objects degraded (32.192%), 67 pgs degraded, 68 pgs undersized 2022-05-13T23:20:00.370 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:20:00 smithi115 ceph-mon[78593]: pg 2.2 is stuck undersized for 2m, current state active+undersized+degraded, last acting [7,1] 2022-05-13T23:20:00.370 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:20:00 smithi115 ceph-mon[78593]: pg 2.3 is stuck undersized for 2m, current state active+undersized+degraded, last acting [4,10] 2022-05-13T23:20:00.370 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:20:00 smithi115 ceph-mon[78593]: pg 2.4 is stuck undersized for 2m, current state undersized+peered, last acting [2] 2022-05-13T23:20:00.370 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:20:00 smithi115 ceph-mon[78593]: pg 2.5 is stuck undersized for 2m, current state undersized+degraded+peered, last acting [8] 2022-05-13T23:20:00.370 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:20:00 smithi115 ceph-mon[78593]: pg 2.6 is stuck undersized for 2m, current state undersized+degraded+peered, last acting [8] 2022-05-13T23:20:00.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:20:00 smithi115 ceph-mon[78593]: pg 2.7 is stuck undersized for 2m, current state active+undersized+degraded, last acting [4,11] 2022-05-13T23:20:00.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:20:00 smithi115 ceph-mon[78593]: pg 2.8 is stuck undersized for 2m, current state active+undersized+degraded, last acting [1,2] 2022-05-13T23:20:00.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:20:00 smithi115 ceph-mon[78593]: pg 2.9 is undersized+degraded+peered, acting [10] 2022-05-13T23:20:00.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:20:00 smithi115 ceph-mon[78593]: pg 2.10 is stuck undersized for 2m, current state active+undersized+degraded, last acting [8,2] 2022-05-13T23:20:00.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:20:00 smithi115 ceph-mon[78593]: pg 2.12 is stuck undersized for 2m, current state active+undersized+degraded, last acting [5,7] 2022-05-13T23:20:00.372 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:20:00 smithi115 ceph-mon[78593]: pg 2.13 is stuck undersized for 2m, current state active+undersized+degraded, last acting [11,10] 2022-05-13T23:20:00.372 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:20:00 smithi115 ceph-mon[78593]: pg 2.15 is stuck undersized for 2m, current state active+undersized+degraded, last acting [5,10] 2022-05-13T23:20:00.373 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:20:00 smithi115 ceph-mon[78593]: pg 2.17 is stuck undersized for 2m, current state active+undersized+degraded, last acting [10,5] 2022-05-13T23:20:00.373 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:20:00 smithi115 ceph-mon[78593]: pg 2.19 is stuck undersized for 2m, current state undersized+degraded+peered, last acting [4] 2022-05-13T23:20:00.373 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:20:00 smithi115 ceph-mon[78593]: pg 2.1a is stuck undersized for 2m, current state active+undersized+degraded, last acting [10,8] 2022-05-13T23:20:00.374 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:20:00 smithi115 ceph-mon[78593]: pg 2.1e is stuck undersized for 2m, current state undersized+degraded+peered, last acting [5] 2022-05-13T23:20:00.374 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:20:00 smithi115 ceph-mon[78593]: pg 2.1f is stuck undersized for 2m, current state active+undersized+degraded, last acting [7,8] 2022-05-13T23:20:00.374 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:20:00 smithi115 ceph-mon[78593]: pg 3.1 is stuck undersized for 2m, current state active+undersized+degraded, last acting [4,2] 2022-05-13T23:20:00.374 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:20:00 smithi115 ceph-mon[78593]: pg 3.2 is stuck undersized for 2m, current state active+undersized+degraded, last acting [5,1] 2022-05-13T23:20:00.375 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:20:00 smithi115 ceph-mon[78593]: pg 3.3 is stuck undersized for 2m, current state active+undersized+degraded, last acting [10,4] 2022-05-13T23:20:00.375 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:20:00 smithi115 ceph-mon[78593]: pg 3.4 is stuck undersized for 2m, current state undersized+degraded+peered, last acting [2] 2022-05-13T23:20:00.375 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:20:00 smithi115 ceph-mon[78593]: pg 3.5 is stuck undersized for 2m, current state undersized+degraded+peered, last acting [10] 2022-05-13T23:20:00.375 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:20:00 smithi115 ceph-mon[78593]: pg 3.6 is stuck undersized for 2m, current state undersized+degraded+peered, last acting [4] 2022-05-13T23:20:00.376 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:20:00 smithi115 ceph-mon[78593]: pg 3.7 is stuck undersized for 2m, current state active+undersized+degraded, last acting [11,7] 2022-05-13T23:20:00.376 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:20:00 smithi115 ceph-mon[78593]: pg 3.10 is stuck undersized for 2m, current state active+undersized+degraded, last acting [7,10] 2022-05-13T23:20:00.376 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:20:00 smithi115 ceph-mon[78593]: pg 3.12 is stuck undersized for 2m, current state undersized+degraded+peered, last acting [7] 2022-05-13T23:20:00.376 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:20:00 smithi115 ceph-mon[78593]: pg 3.13 is stuck undersized for 2m, current state active+undersized+degraded, last acting [7,4] 2022-05-13T23:20:00.377 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:20:00 smithi115 ceph-mon[78593]: pg 3.16 is stuck undersized for 2m, current state active+undersized+degraded, last acting [1,10] 2022-05-13T23:20:00.377 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:20:00 smithi115 ceph-mon[78593]: pg 3.17 is stuck undersized for 2m, current state active+undersized+degraded, last acting [4,5] 2022-05-13T23:20:00.377 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:20:00 smithi115 ceph-mon[78593]: pg 3.18 is stuck undersized for 2m, current state undersized+degraded+peered, last acting [11] 2022-05-13T23:20:00.377 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:20:00 smithi115 ceph-mon[78593]: pg 3.19 is stuck undersized for 2m, current state undersized+degraded+peered, last acting [10] 2022-05-13T23:20:00.378 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:20:00 smithi115 ceph-mon[78593]: pg 3.1a is stuck undersized for 2m, current state active+undersized+degraded, last acting [8,5] 2022-05-13T23:20:00.378 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:20:00 smithi115 ceph-mon[78593]: pg 3.1b is stuck undersized for 2m, current state active+undersized+degraded, last acting [5,4] 2022-05-13T23:20:00.379 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:20:00 smithi115 ceph-mon[78593]: pg 3.1c is stuck undersized for 2m, current state active+undersized+degraded, last acting [1,10] 2022-05-13T23:20:00.379 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:20:00 smithi115 ceph-mon[78593]: pg 3.1e is stuck undersized for 2m, current state active+undersized+degraded, last acting [2,5] 2022-05-13T23:20:00.379 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:20:00 smithi115 ceph-mon[78593]: pg 3.1f is stuck undersized for 2m, current state undersized+degraded+peered, last acting [1] 2022-05-13T23:20:00.379 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:20:00 smithi115 ceph-mon[78593]: pg 3.2d is stuck undersized for 2m, current state undersized+degraded+peered, last acting [2] 2022-05-13T23:20:00.380 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:20:00 smithi115 ceph-mon[78593]: pg 3.2e is stuck undersized for 2m, current state active+undersized+degraded, last acting [1,10] 2022-05-13T23:20:00.380 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:20:00 smithi115 ceph-mon[78593]: pg 3.2f is stuck undersized for 2m, current state active+undersized+degraded, last acting [11,1] 2022-05-13T23:20:00.380 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:20:00 smithi115 ceph-mon[78593]: pg 3.30 is stuck undersized for 2m, current state active+undersized+degraded, last acting [1,7] 2022-05-13T23:20:00.380 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:20:00 smithi115 ceph-mon[78593]: pg 3.31 is stuck undersized for 2m, current state active+undersized+degraded, last acting [4,2] 2022-05-13T23:20:00.380 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:20:00 smithi115 ceph-mon[78593]: pg 3.32 is stuck undersized for 2m, current state active+undersized+degraded, last acting [8,10] 2022-05-13T23:20:00.381 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:20:00 smithi115 ceph-mon[78593]: pg 3.33 is stuck undersized for 2m, current state active+undersized+degraded, last acting [10,5] 2022-05-13T23:20:00.381 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:20:00 smithi115 ceph-mon[78593]: pg 3.36 is stuck undersized for 2m, current state active+undersized+degraded, last acting [10,2] 2022-05-13T23:20:00.381 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:20:00 smithi115 ceph-mon[78593]: pg 3.38 is stuck undersized for 2m, current state active+undersized+degraded, last acting [5,10] 2022-05-13T23:20:00.381 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:20:00 smithi115 ceph-mon[78593]: pg 3.39 is stuck undersized for 2m, current state active+undersized+degraded, last acting [7,5] 2022-05-13T23:20:00.381 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:20:00 smithi115 ceph-mon[78593]: pg 3.3a is stuck undersized for 2m, current state active+undersized+degraded, last acting [10,4] 2022-05-13T23:20:00.382 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:20:00 smithi115 ceph-mon[78593]: pg 3.3b is stuck undersized for 2m, current state undersized+degraded+peered, last acting [8] 2022-05-13T23:20:00.382 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:20:00 smithi115 ceph-mon[78593]: pg 3.3d is stuck undersized for 2m, current state active+undersized+degraded, last acting [2,1] 2022-05-13T23:20:00.382 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:20:00 smithi115 ceph-mon[78593]: pg 3.3e is stuck undersized for 2m, current state active+undersized+degraded, last acting [11,10] 2022-05-13T23:20:00.382 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:20:00 smithi115 ceph-mon[78593]: pg 3.3f is stuck undersized for 2m, current state active+undersized+degraded, last acting [11,1] 2022-05-13T23:20:00.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:20:00 smithi047 ceph-mon[77147]: pgmap v58: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 138 MiB data, 465 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 3 op/s; 10439/32427 objects degraded (32.192%) 2022-05-13T23:20:00.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:20:00 smithi047 ceph-mon[77147]: Health detail: HEALTH_WARN 1/3 mons down, quorum b,c; 4 osds down; 1 host (4 osds) down; Reduced data availability: 23 pgs inactive; Degraded data redundancy: 10439/32427 objects degraded (32.192%), 67 pgs degraded, 68 pgs undersized 2022-05-13T23:20:00.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:20:00 smithi047 ceph-mon[77147]: [WRN] MON_DOWN: 1/3 mons down, quorum b,c 2022-05-13T23:20:00.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:20:00 smithi047 ceph-mon[77147]: mon.a (rank 0) addr [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] is down (out of quorum) 2022-05-13T23:20:00.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:20:00 smithi047 ceph-mon[77147]: [WRN] OSD_DOWN: 4 osds down 2022-05-13T23:20:00.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:20:00 smithi047 ceph-mon[77147]: osd.0 (root=default,host=smithi002) is down 2022-05-13T23:20:00.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:20:00 smithi047 ceph-mon[77147]: osd.3 (root=default,host=smithi002) is down 2022-05-13T23:20:00.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:20:00 smithi047 ceph-mon[77147]: osd.6 (root=default,host=smithi002) is down 2022-05-13T23:20:00.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:20:00 smithi047 ceph-mon[77147]: osd.9 (root=default,host=smithi002) is down 2022-05-13T23:20:00.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:20:00 smithi047 ceph-mon[77147]: [WRN] OSD_HOST_DOWN: 1 host (4 osds) down 2022-05-13T23:20:00.545 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:20:00 smithi047 ceph-mon[77147]: host smithi002 (root=default) (4 osds) is down 2022-05-13T23:20:00.547 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:20:00 smithi047 ceph-mon[77147]: [WRN] PG_AVAILABILITY: Reduced data availability: 23 pgs inactive 2022-05-13T23:20:00.547 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:20:00 smithi047 ceph-mon[77147]: pg 2.1 is stuck inactive for 4m, current state unknown, last acting [] 2022-05-13T23:20:00.548 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:20:00 smithi047 ceph-mon[77147]: pg 2.4 is stuck inactive for 5m, current state undersized+peered, last acting [2] 2022-05-13T23:20:00.548 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:20:00 smithi047 ceph-mon[77147]: pg 2.5 is stuck inactive for 2m, current state undersized+degraded+peered, last acting [8] 2022-05-13T23:20:00.548 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:20:00 smithi047 ceph-mon[77147]: pg 2.6 is stuck inactive for 2m, current state undersized+degraded+peered, last acting [8] 2022-05-13T23:20:00.548 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:20:00 smithi047 ceph-mon[77147]: pg 2.9 is stuck inactive for 2m, current state undersized+degraded+peered, last acting [10] 2022-05-13T23:20:00.549 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:20:00 smithi047 ceph-mon[77147]: pg 2.e is stuck inactive for 2m, current state undersized+degraded+peered, last acting [2] 2022-05-13T23:20:00.549 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:20:00 smithi047 ceph-mon[77147]: pg 2.19 is stuck inactive for 2m, current state undersized+degraded+peered, last acting [4] 2022-05-13T23:20:00.549 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:20:00 smithi047 ceph-mon[77147]: pg 2.1e is stuck inactive for 2m, current state undersized+degraded+peered, last acting [5] 2022-05-13T23:20:00.549 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:20:00 smithi047 ceph-mon[77147]: pg 3.4 is stuck inactive for 2m, current state undersized+degraded+peered, last acting [2] 2022-05-13T23:20:00.550 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:20:00 smithi047 ceph-mon[77147]: pg 3.5 is stuck inactive for 2m, current state undersized+degraded+peered, last acting [10] 2022-05-13T23:20:00.550 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:20:00 smithi047 ceph-mon[77147]: pg 3.6 is stuck inactive for 2m, current state undersized+degraded+peered, last acting [4] 2022-05-13T23:20:00.550 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:20:00 smithi047 ceph-mon[77147]: pg 3.b is stuck inactive for 2m, current state undersized+degraded+peered, last acting [10] 2022-05-13T23:20:00.550 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:20:00 smithi047 ceph-mon[77147]: pg 3.12 is stuck inactive for 2m, current state undersized+degraded+peered, last acting [7] 2022-05-13T23:20:00.551 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:20:00 smithi047 ceph-mon[77147]: pg 3.18 is stuck inactive for 2m, current state undersized+degraded+peered, last acting [11] 2022-05-13T23:20:00.551 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:20:00 smithi047 ceph-mon[77147]: pg 3.19 is stuck inactive for 2m, current state undersized+degraded+peered, last acting [10] 2022-05-13T23:20:00.551 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:20:00 smithi047 ceph-mon[77147]: pg 3.1f is stuck inactive for 2m, current state undersized+degraded+peered, last acting [1] 2022-05-13T23:20:00.551 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:20:00 smithi047 ceph-mon[77147]: pg 3.22 is stuck inactive for 2m, current state undersized+degraded+peered, last acting [2] 2022-05-13T23:20:00.551 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:20:00 smithi047 ceph-mon[77147]: pg 3.23 is stuck inactive for 2m, current state undersized+degraded+peered, last acting [4] 2022-05-13T23:20:00.552 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:20:00 smithi047 ceph-mon[77147]: pg 3.24 is stuck inactive for 2m, current state undersized+degraded+peered, last acting [4] 2022-05-13T23:20:00.552 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:20:00 smithi047 ceph-mon[77147]: pg 3.28 is stuck inactive for 2m, current state undersized+degraded+peered, last acting [5] 2022-05-13T23:20:00.552 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:20:00 smithi047 ceph-mon[77147]: pg 3.2b is stuck inactive for 2m, current state undersized+degraded+peered, last acting [7] 2022-05-13T23:20:00.552 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:20:00 smithi047 ceph-mon[77147]: pg 3.2d is stuck inactive for 2m, current state undersized+degraded+peered, last acting [2] 2022-05-13T23:20:00.553 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:20:00 smithi047 ceph-mon[77147]: pg 3.3b is stuck inactive for 2m, current state undersized+degraded+peered, last acting [8] 2022-05-13T23:20:00.553 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:20:00 smithi047 ceph-mon[77147]: [WRN] PG_DEGRADED: Degraded data redundancy: 10439/32427 objects degraded (32.192%), 67 pgs degraded, 68 pgs undersized 2022-05-13T23:20:00.553 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:20:00 smithi047 ceph-mon[77147]: pg 2.2 is stuck undersized for 2m, current state active+undersized+degraded, last acting [7,1] 2022-05-13T23:20:00.553 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:20:00 smithi047 ceph-mon[77147]: pg 2.3 is stuck undersized for 2m, current state active+undersized+degraded, last acting [4,10] 2022-05-13T23:20:00.554 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:20:00 smithi047 ceph-mon[77147]: pg 2.4 is stuck undersized for 2m, current state undersized+peered, last acting [2] 2022-05-13T23:20:00.554 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:20:00 smithi047 ceph-mon[77147]: pg 2.5 is stuck undersized for 2m, current state undersized+degraded+peered, last acting [8] 2022-05-13T23:20:00.554 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:20:00 smithi047 ceph-mon[77147]: pg 2.6 is stuck undersized for 2m, current state undersized+degraded+peered, last acting [8] 2022-05-13T23:20:00.555 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:20:00 smithi047 ceph-mon[77147]: pg 2.7 is stuck undersized for 2m, current state active+undersized+degraded, last acting [4,11] 2022-05-13T23:20:00.555 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:20:00 smithi047 ceph-mon[77147]: pg 2.8 is stuck undersized for 2m, current state active+undersized+degraded, last acting [1,2] 2022-05-13T23:20:00.555 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:20:00 smithi047 ceph-mon[77147]: pg 2.9 is undersized+degraded+peered, acting [10] 2022-05-13T23:20:00.555 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:20:00 smithi047 ceph-mon[77147]: pg 2.10 is stuck undersized for 2m, current state active+undersized+degraded, last acting [8,2] 2022-05-13T23:20:00.555 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:20:00 smithi047 ceph-mon[77147]: pg 2.12 is stuck undersized for 2m, current state active+undersized+degraded, last acting [5,7] 2022-05-13T23:20:00.556 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:20:00 smithi047 ceph-mon[77147]: pg 2.13 is stuck undersized for 2m, current state active+undersized+degraded, last acting [11,10] 2022-05-13T23:20:00.556 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:20:00 smithi047 ceph-mon[77147]: pg 2.15 is stuck undersized for 2m, current state active+undersized+degraded, last acting [5,10] 2022-05-13T23:20:00.556 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:20:00 smithi047 ceph-mon[77147]: pg 2.17 is stuck undersized for 2m, current state active+undersized+degraded, last acting [10,5] 2022-05-13T23:20:00.557 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:20:00 smithi047 ceph-mon[77147]: pg 2.19 is stuck undersized for 2m, current state undersized+degraded+peered, last acting [4] 2022-05-13T23:20:00.557 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:20:00 smithi047 ceph-mon[77147]: pg 2.1a is stuck undersized for 2m, current state active+undersized+degraded, last acting [10,8] 2022-05-13T23:20:00.557 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:20:00 smithi047 ceph-mon[77147]: pg 2.1e is stuck undersized for 2m, current state undersized+degraded+peered, last acting [5] 2022-05-13T23:20:00.557 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:20:00 smithi047 ceph-mon[77147]: pg 2.1f is stuck undersized for 2m, current state active+undersized+degraded, last acting [7,8] 2022-05-13T23:20:00.558 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:20:00 smithi047 ceph-mon[77147]: pg 3.1 is stuck undersized for 2m, current state active+undersized+degraded, last acting [4,2] 2022-05-13T23:20:00.558 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:20:00 smithi047 ceph-mon[77147]: pg 3.2 is stuck undersized for 2m, current state active+undersized+degraded, last acting [5,1] 2022-05-13T23:20:00.558 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:20:00 smithi047 ceph-mon[77147]: pg 3.3 is stuck undersized for 2m, current state active+undersized+degraded, last acting [10,4] 2022-05-13T23:20:00.558 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:20:00 smithi047 ceph-mon[77147]: pg 3.4 is stuck undersized for 2m, current state undersized+degraded+peered, last acting [2] 2022-05-13T23:20:00.559 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:20:00 smithi047 ceph-mon[77147]: pg 3.5 is stuck undersized for 2m, current state undersized+degraded+peered, last acting [10] 2022-05-13T23:20:00.559 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:20:00 smithi047 ceph-mon[77147]: pg 3.6 is stuck undersized for 2m, current state undersized+degraded+peered, last acting [4] 2022-05-13T23:20:00.559 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:20:00 smithi047 ceph-mon[77147]: pg 3.7 is stuck undersized for 2m, current state active+undersized+degraded, last acting [11,7] 2022-05-13T23:20:00.560 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:20:00 smithi047 ceph-mon[77147]: pg 3.10 is stuck undersized for 2m, current state active+undersized+degraded, last acting [7,10] 2022-05-13T23:20:00.560 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:20:00 smithi047 ceph-mon[77147]: pg 3.12 is stuck undersized for 2m, current state undersized+degraded+peered, last acting [7] 2022-05-13T23:20:00.561 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:20:00 smithi047 ceph-mon[77147]: pg 3.13 is stuck undersized for 2m, current state active+undersized+degraded, last acting [7,4] 2022-05-13T23:20:00.561 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:20:00 smithi047 ceph-mon[77147]: pg 3.16 is stuck undersized for 2m, current state active+undersized+degraded, last acting [1,10] 2022-05-13T23:20:00.561 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:20:00 smithi047 ceph-mon[77147]: pg 3.17 is stuck undersized for 2m, current state active+undersized+degraded, last acting [4,5] 2022-05-13T23:20:00.561 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:20:00 smithi047 ceph-mon[77147]: pg 3.18 is stuck undersized for 2m, current state undersized+degraded+peered, last acting [11] 2022-05-13T23:20:00.562 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:20:00 smithi047 ceph-mon[77147]: pg 3.19 is stuck undersized for 2m, current state undersized+degraded+peered, last acting [10] 2022-05-13T23:20:00.562 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:20:00 smithi047 ceph-mon[77147]: pg 3.1a is stuck undersized for 2m, current state active+undersized+degraded, last acting [8,5] 2022-05-13T23:20:00.562 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:20:00 smithi047 ceph-mon[77147]: pg 3.1b is stuck undersized for 2m, current state active+undersized+degraded, last acting [5,4] 2022-05-13T23:20:00.562 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:20:00 smithi047 ceph-mon[77147]: pg 3.1c is stuck undersized for 2m, current state active+undersized+degraded, last acting [1,10] 2022-05-13T23:20:00.562 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:20:00 smithi047 ceph-mon[77147]: pg 3.1e is stuck undersized for 2m, current state active+undersized+degraded, last acting [2,5] 2022-05-13T23:20:00.563 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:20:00 smithi047 ceph-mon[77147]: pg 3.1f is stuck undersized for 2m, current state undersized+degraded+peered, last acting [1] 2022-05-13T23:20:00.563 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:20:00 smithi047 ceph-mon[77147]: pg 3.2d is stuck undersized for 2m, current state undersized+degraded+peered, last acting [2] 2022-05-13T23:20:00.563 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:20:00 smithi047 ceph-mon[77147]: pg 3.2e is stuck undersized for 2m, current state active+undersized+degraded, last acting [1,10] 2022-05-13T23:20:00.563 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:20:00 smithi047 ceph-mon[77147]: pg 3.2f is stuck undersized for 2m, current state active+undersized+degraded, last acting [11,1] 2022-05-13T23:20:00.564 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:20:00 smithi047 ceph-mon[77147]: pg 3.30 is stuck undersized for 2m, current state active+undersized+degraded, last acting [1,7] 2022-05-13T23:20:00.564 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:20:00 smithi047 ceph-mon[77147]: pg 3.31 is stuck undersized for 2m, current state active+undersized+degraded, last acting [4,2] 2022-05-13T23:20:00.564 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:20:00 smithi047 ceph-mon[77147]: pg 3.32 is stuck undersized for 2m, current state active+undersized+degraded, last acting [8,10] 2022-05-13T23:20:00.564 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:20:00 smithi047 ceph-mon[77147]: pg 3.33 is stuck undersized for 2m, current state active+undersized+degraded, last acting [10,5] 2022-05-13T23:20:00.565 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:20:00 smithi047 ceph-mon[77147]: pg 3.36 is stuck undersized for 2m, current state active+undersized+degraded, last acting [10,2] 2022-05-13T23:20:00.565 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:20:00 smithi047 ceph-mon[77147]: pg 3.38 is stuck undersized for 2m, current state active+undersized+degraded, last acting [5,10] 2022-05-13T23:20:00.565 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:20:00 smithi047 ceph-mon[77147]: pg 3.39 is stuck undersized for 2m, current state active+undersized+degraded, last acting [7,5] 2022-05-13T23:20:00.565 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:20:00 smithi047 ceph-mon[77147]: pg 3.3a is stuck undersized for 2m, current state active+undersized+degraded, last acting [10,4] 2022-05-13T23:20:00.566 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:20:00 smithi047 ceph-mon[77147]: pg 3.3b is stuck undersized for 2m, current state undersized+degraded+peered, last acting [8] 2022-05-13T23:20:00.566 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:20:00 smithi047 ceph-mon[77147]: pg 3.3d is stuck undersized for 2m, current state active+undersized+degraded, last acting [2,1] 2022-05-13T23:20:00.566 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:20:00 smithi047 ceph-mon[77147]: pg 3.3e is stuck undersized for 2m, current state active+undersized+degraded, last acting [11,10] 2022-05-13T23:20:00.569 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:20:00 smithi047 ceph-mon[77147]: pg 3.3f is stuck undersized for 2m, current state active+undersized+degraded, last acting [11,1] 2022-05-13T23:20:02.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:20:02 smithi115 ceph-mon[78593]: pgmap v59: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 138 MiB data, 465 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s; 10439/32427 objects degraded (32.192%) 2022-05-13T23:20:02.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:20:02 smithi115 ceph-mon[78593]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2022-05-13T23:20:02.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:20:02 smithi047 ceph-mon[77147]: pgmap v59: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 138 MiB data, 465 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s; 10439/32427 objects degraded (32.192%) 2022-05-13T23:20:02.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:20:02 smithi047 ceph-mon[77147]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2022-05-13T23:20:04.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:20:03 smithi047 ceph-mon[77147]: pgmap v60: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 138 MiB data, 465 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 3 op/s; 10439/32427 objects degraded (32.192%) 2022-05-13T23:20:04.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:20:03 smithi115 ceph-mon[78593]: pgmap v60: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 138 MiB data, 465 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 3 op/s; 10439/32427 objects degraded (32.192%) 2022-05-13T23:20:06.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:20:06 smithi115 ceph-mon[78593]: pgmap v61: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 138 MiB data, 465 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s; 10439/32427 objects degraded (32.192%) 2022-05-13T23:20:06.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:20:06 smithi047 ceph-mon[77147]: pgmap v61: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 138 MiB data, 465 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s; 10439/32427 objects degraded (32.192%) 2022-05-13T23:20:08.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:20:08 smithi115 ceph-mon[78593]: pgmap v62: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 138 MiB data, 465 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 3 op/s; 10439/32427 objects degraded (32.192%) 2022-05-13T23:20:08.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:20:08 smithi047 ceph-mon[77147]: pgmap v62: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 138 MiB data, 465 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 3 op/s; 10439/32427 objects degraded (32.192%) 2022-05-13T23:20:10.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:20:10 smithi115 ceph-mon[78593]: pgmap v63: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 138 MiB data, 465 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 3 op/s; 10439/32427 objects degraded (32.192%) 2022-05-13T23:20:10.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:20:10 smithi047 ceph-mon[77147]: pgmap v63: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 138 MiB data, 465 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 3 op/s; 10439/32427 objects degraded (32.192%) 2022-05-13T23:20:12.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:20:12 smithi115 ceph-mon[78593]: pgmap v64: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 138 MiB data, 465 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s; 10439/32427 objects degraded (32.192%) 2022-05-13T23:20:12.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:20:12 smithi047 ceph-mon[77147]: pgmap v64: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 138 MiB data, 465 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s; 10439/32427 objects degraded (32.192%) 2022-05-13T23:20:14.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:20:13 smithi047 ceph-mon[77147]: pgmap v65: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 138 MiB data, 465 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 3 op/s; 10439/32427 objects degraded (32.192%) 2022-05-13T23:20:14.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:20:13 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-13T23:20:14.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:20:13 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-13T23:20:14.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:20:13 smithi115 ceph-mon[78593]: pgmap v65: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 138 MiB data, 465 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 3 op/s; 10439/32427 objects degraded (32.192%) 2022-05-13T23:20:14.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:20:13 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-13T23:20:14.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:20:13 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-13T23:20:16.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:20:16 smithi115 ceph-mon[78593]: pgmap v66: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 138 MiB data, 465 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s; 10439/32427 objects degraded (32.192%) 2022-05-13T23:20:16.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:20:16 smithi047 ceph-mon[77147]: pgmap v66: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 138 MiB data, 465 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s; 10439/32427 objects degraded (32.192%) 2022-05-13T23:20:18.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:20:18 smithi115 ceph-mon[78593]: pgmap v67: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 138 MiB data, 465 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 3 op/s; 10439/32427 objects degraded (32.192%) 2022-05-13T23:20:18.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:20:18 smithi047 ceph-mon[77147]: pgmap v67: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 138 MiB data, 465 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 3 op/s; 10439/32427 objects degraded (32.192%) 2022-05-13T23:20:20.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:20:20 smithi115 ceph-mon[78593]: pgmap v68: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 138 MiB data, 465 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s; 10439/32427 objects degraded (32.192%) 2022-05-13T23:20:20.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:20:20 smithi047 ceph-mon[77147]: pgmap v68: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 138 MiB data, 465 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s; 10439/32427 objects degraded (32.192%) 2022-05-13T23:20:21.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:20:21 smithi115 ceph-mon[78593]: 1 slow requests, 1 included below; oldest blocked for > 184.074699 secs 2022-05-13T23:20:21.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:20:21 smithi115 ceph-mon[78593]: slow request 184.074699 seconds old, received at 2022-05-13T23:17:16.166863+0000: client_request(client.25244:12077 setattr size=2217921 #0x10000002fec 2022-05-13T23:17:16.163392+0000 caller_uid=1000, caller_gid=1262{6,36,1000,1262,}) currently failed to xlock, waiting 2022-05-13T23:20:21.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:20:21 smithi047 ceph-mon[77147]: 1 slow requests, 1 included below; oldest blocked for > 184.074699 secs 2022-05-13T23:20:21.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:20:21 smithi047 ceph-mon[77147]: slow request 184.074699 seconds old, received at 2022-05-13T23:17:16.166863+0000: client_request(client.25244:12077 setattr size=2217921 #0x10000002fec 2022-05-13T23:17:16.163392+0000 caller_uid=1000, caller_gid=1262{6,36,1000,1262,}) currently failed to xlock, waiting 2022-05-13T23:20:22.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:20:22 smithi115 ceph-mon[78593]: pgmap v69: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 138 MiB data, 465 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s; 10439/32427 objects degraded (32.192%) 2022-05-13T23:20:22.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:20:22 smithi115 ceph-mon[78593]: Health check failed: 1 MDSs report slow metadata IOs (MDS_SLOW_METADATA_IO) 2022-05-13T23:20:22.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:20:22 smithi115 ceph-mon[78593]: Health check failed: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2022-05-13T23:20:22.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:20:22 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:20:22.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:20:22 smithi115 ceph-mon[78593]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2022-05-13T23:20:22.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:20:22 smithi047 ceph-mon[77147]: pgmap v69: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 138 MiB data, 465 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s; 10439/32427 objects degraded (32.192%) 2022-05-13T23:20:22.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:20:22 smithi047 ceph-mon[77147]: Health check failed: 1 MDSs report slow metadata IOs (MDS_SLOW_METADATA_IO) 2022-05-13T23:20:22.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:20:22 smithi047 ceph-mon[77147]: Health check failed: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2022-05-13T23:20:22.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:20:22 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:20:22.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:20:22 smithi047 ceph-mon[77147]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2022-05-13T23:20:24.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:20:23 smithi047 ceph-mon[77147]: pgmap v70: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 138 MiB data, 465 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 3 op/s; 10439/32427 objects degraded (32.192%) 2022-05-13T23:20:24.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:20:23 smithi115 ceph-mon[78593]: pgmap v70: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 138 MiB data, 465 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 3 op/s; 10439/32427 objects degraded (32.192%) 2022-05-13T23:20:25.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:20:25 smithi047 ceph-mon[77147]: osdmap e91: 12 total, 8 up, 12 in 2022-05-13T23:20:25.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:20:25 smithi047 ceph-mon[77147]: fsmap cephfs:5 2 up:standby-replay 5 up:active 2022-05-13T23:20:25.293 INFO:journalctl@ceph.mds.e.smithi047.stdout:May 13 23:20:25 smithi047 conmon[103776]: ignoring --setuser ceph since I am not root 2022-05-13T23:20:25.294 INFO:journalctl@ceph.mds.e.smithi047.stdout:May 13 23:20:25 smithi047 conmon[103776]: ignoring --setgroup ceph since I am not root 2022-05-13T23:20:25.294 INFO:journalctl@ceph.mds.e.smithi047.stdout:May 13 23:20:25 smithi047 ceph-mds[103800]: -- 172.21.15.47:0/3105495638 <== mon.1 v2:172.21.15.47:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5654ca0fe340 con 0x5654ca0d6800 2022-05-13T23:20:25.295 INFO:journalctl@ceph.mds.e.smithi047.stdout:May 13 23:20:25 smithi047 conmon[103776]: starting mds.e at 2022-05-13T23:20:25.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:20:25 smithi115 ceph-mon[78593]: osdmap e91: 12 total, 8 up, 12 in 2022-05-13T23:20:25.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:20:25 smithi115 ceph-mon[78593]: fsmap cephfs:5 2 up:standby-replay 5 up:active 2022-05-13T23:20:26.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:20:26 smithi115 ceph-mon[78593]: pgmap v72: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 138 MiB data, 465 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 3 op/s; 10439/32427 objects degraded (32.192%) 2022-05-13T23:20:26.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:20:26 smithi115 ceph-mon[78593]: 1 slow requests, 0 included below; oldest blocked for > 189.074801 secs 2022-05-13T23:20:26.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:20:26 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.47:6834/3993126251,v1:172.21.15.47:6835/3993126251] up:boot 2022-05-13T23:20:26.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:20:26 smithi115 ceph-mon[78593]: fsmap cephfs:5 2 up:standby-replay 1 up:standby 5 up:active 2022-05-13T23:20:26.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:20:26 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2022-05-13T23:20:26.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:20:26 smithi115 ceph-mon[78593]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2022-05-13T23:20:26.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:20:26 smithi047 ceph-mon[77147]: pgmap v72: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 138 MiB data, 465 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 3 op/s; 10439/32427 objects degraded (32.192%) 2022-05-13T23:20:26.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:20:26 smithi047 ceph-mon[77147]: 1 slow requests, 0 included below; oldest blocked for > 189.074801 secs 2022-05-13T23:20:26.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:20:26 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.47:6834/3993126251,v1:172.21.15.47:6835/3993126251] up:boot 2022-05-13T23:20:26.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:20:26 smithi047 ceph-mon[77147]: fsmap cephfs:5 2 up:standby-replay 1 up:standby 5 up:active 2022-05-13T23:20:26.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:20:26 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2022-05-13T23:20:26.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:20:26 smithi047 ceph-mon[77147]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2022-05-13T23:20:28.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:20:28 smithi115 ceph-mon[78593]: pgmap v73: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 138 MiB data, 465 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 3 op/s; 10439/32427 objects degraded (32.192%) 2022-05-13T23:20:28.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:20:28 smithi047 ceph-mon[77147]: pgmap v73: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 138 MiB data, 465 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 3 op/s; 10439/32427 objects degraded (32.192%) 2022-05-13T23:20:30.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:20:30 smithi115 ceph-mon[78593]: pgmap v74: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 138 MiB data, 465 MiB used, 715 GiB / 715 GiB avail; 2.3 KiB/s rd, 4 op/s; 10439/32427 objects degraded (32.192%) 2022-05-13T23:20:30.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:20:30 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:20:30.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:20:30 smithi115 ceph-mon[78593]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2022-05-13T23:20:30.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:20:30 smithi047 ceph-mon[77147]: pgmap v74: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 138 MiB data, 465 MiB used, 715 GiB / 715 GiB avail; 2.3 KiB/s rd, 4 op/s; 10439/32427 objects degraded (32.192%) 2022-05-13T23:20:30.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:20:30 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:20:30.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:20:30 smithi047 ceph-mon[77147]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2022-05-13T23:20:31.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:20:31 smithi047 ceph-mon[77147]: 1 slow requests, 0 included below; oldest blocked for > 194.074956 secs 2022-05-13T23:20:31.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:20:31 smithi115 ceph-mon[78593]: 1 slow requests, 0 included below; oldest blocked for > 194.074956 secs 2022-05-13T23:20:32.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:20:32 smithi047 ceph-mon[77147]: pgmap v75: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 138 MiB data, 465 MiB used, 715 GiB / 715 GiB avail; 2.3 KiB/s rd, 4 op/s; 10439/32427 objects degraded (32.192%) 2022-05-13T23:20:32.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:20:32 smithi115 ceph-mon[78593]: pgmap v75: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 138 MiB data, 465 MiB used, 715 GiB / 715 GiB avail; 2.3 KiB/s rd, 4 op/s; 10439/32427 objects degraded (32.192%) 2022-05-13T23:20:34.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:20:34 smithi047 ceph-mon[77147]: pgmap v76: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 138 MiB data, 465 MiB used, 715 GiB / 715 GiB avail; 2.3 KiB/s rd, 4 op/s; 10439/32427 objects degraded (32.192%) 2022-05-13T23:20:34.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:20:34 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:20:34.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:20:34 smithi047 ceph-mon[77147]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2022-05-13T23:20:34.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:20:34 smithi115 ceph-mon[78593]: pgmap v76: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 138 MiB data, 465 MiB used, 715 GiB / 715 GiB avail; 2.3 KiB/s rd, 4 op/s; 10439/32427 objects degraded (32.192%) 2022-05-13T23:20:34.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:20:34 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:20:34.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:20:34 smithi115 ceph-mon[78593]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2022-05-13T23:20:36.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:20:36 smithi047 ceph-mon[77147]: pgmap v77: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 138 MiB data, 465 MiB used, 715 GiB / 715 GiB avail; 2.1 KiB/s rd, 3 op/s; 10439/32427 objects degraded (32.192%) 2022-05-13T23:20:36.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:20:36 smithi047 ceph-mon[77147]: 1 slow requests, 0 included below; oldest blocked for > 199.075097 secs 2022-05-13T23:20:36.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:20:36 smithi115 ceph-mon[78593]: pgmap v77: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 138 MiB data, 465 MiB used, 715 GiB / 715 GiB avail; 2.1 KiB/s rd, 3 op/s; 10439/32427 objects degraded (32.192%) 2022-05-13T23:20:36.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:20:36 smithi115 ceph-mon[78593]: 1 slow requests, 0 included below; oldest blocked for > 199.075097 secs 2022-05-13T23:20:38.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:20:38 smithi047 ceph-mon[77147]: pgmap v78: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 138 MiB data, 465 MiB used, 715 GiB / 715 GiB avail; 1.9 KiB/s rd, 3 op/s; 10439/32427 objects degraded (32.192%) 2022-05-13T23:20:38.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:20:38 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:20:38.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:20:38 smithi047 ceph-mon[77147]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2022-05-13T23:20:38.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:20:38 smithi115 ceph-mon[78593]: pgmap v78: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 138 MiB data, 465 MiB used, 715 GiB / 715 GiB avail; 1.9 KiB/s rd, 3 op/s; 10439/32427 objects degraded (32.192%) 2022-05-13T23:20:38.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:20:38 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:20:38.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:20:38 smithi115 ceph-mon[78593]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2022-05-13T23:20:40.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:20:40 smithi047 ceph-mon[77147]: pgmap v79: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 138 MiB data, 465 MiB used, 715 GiB / 715 GiB avail; 2.7 KiB/s rd, 5 op/s; 10439/32427 objects degraded (32.192%) 2022-05-13T23:20:40.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:20:40 smithi115 ceph-mon[78593]: pgmap v79: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 138 MiB data, 465 MiB used, 715 GiB / 715 GiB avail; 2.7 KiB/s rd, 5 op/s; 10439/32427 objects degraded (32.192%) 2022-05-13T23:20:41.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:20:41 smithi047 ceph-mon[77147]: 2 slow requests, 1 included below; oldest blocked for > 204.075217 secs 2022-05-13T23:20:41.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:20:41 smithi047 ceph-mon[77147]: slow request 180.001247 seconds old, received at 2022-05-13T23:17:40.240831+0000: internal op exportdir:mds.0:25 currently failed to wrlock, waiting 2022-05-13T23:20:41.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:20:41 smithi115 ceph-mon[78593]: 2 slow requests, 1 included below; oldest blocked for > 204.075217 secs 2022-05-13T23:20:41.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:20:41 smithi115 ceph-mon[78593]: slow request 180.001247 seconds old, received at 2022-05-13T23:17:40.240831+0000: internal op exportdir:mds.0:25 currently failed to wrlock, waiting 2022-05-13T23:20:42.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:20:42 smithi047 ceph-mon[77147]: pgmap v80: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 138 MiB data, 465 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 3 op/s; 10439/32427 objects degraded (32.192%) 2022-05-13T23:20:42.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:20:42 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:20:42.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:20:42 smithi047 ceph-mon[77147]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2022-05-13T23:20:42.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:20:42 smithi115 ceph-mon[78593]: pgmap v80: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 138 MiB data, 465 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 3 op/s; 10439/32427 objects degraded (32.192%) 2022-05-13T23:20:42.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:20:42 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:20:42.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:20:42 smithi115 ceph-mon[78593]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2022-05-13T23:20:44.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:20:43 smithi047 ceph-mon[77147]: pgmap v81: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 138 MiB data, 465 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 3 op/s; 10439/32427 objects degraded (32.192%) 2022-05-13T23:20:44.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:20:43 smithi115 ceph-mon[78593]: pgmap v81: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 138 MiB data, 465 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 3 op/s; 10439/32427 objects degraded (32.192%) 2022-05-13T23:20:46.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:20:46 smithi047 ceph-mon[77147]: pgmap v82: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 138 MiB data, 465 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s; 10439/32427 objects degraded (32.192%) 2022-05-13T23:20:46.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:20:46 smithi047 ceph-mon[77147]: 2 slow requests, 0 included below; oldest blocked for > 209.075331 secs 2022-05-13T23:20:46.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:20:46 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-13T23:20:46.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:20:46 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-13T23:20:46.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:20:46 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-13T23:20:46.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:20:46 smithi115 ceph-mon[78593]: pgmap v82: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 138 MiB data, 465 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s; 10439/32427 objects degraded (32.192%) 2022-05-13T23:20:46.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:20:46 smithi115 ceph-mon[78593]: 2 slow requests, 0 included below; oldest blocked for > 209.075331 secs 2022-05-13T23:20:46.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:20:46 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-13T23:20:46.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:20:46 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-13T23:20:46.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:20:46 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-13T23:20:48.434 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:20:48 smithi047 ceph-mon[77147]: pgmap v83: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 138 MiB data, 465 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 3 op/s; 10439/32427 objects degraded (32.192%) 2022-05-13T23:20:48.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:20:48 smithi115 ceph-mon[78593]: pgmap v83: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 138 MiB data, 465 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 3 op/s; 10439/32427 objects degraded (32.192%) 2022-05-13T23:20:48.791 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:20:48 smithi047 conmon[78521]: 2022-05-13T23:20:48.431+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:20:48.792 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:20:48 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-13T23:20:48.792 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:20:48 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-13T23:20:48.792 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:20:48 smithi047 conmon[78521]: yield 2022-05-13T23:20:48.792 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:20:48 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-13T23:20:48.793 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:20:48 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-13T23:20:48.793 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:20:48 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-13T23:20:48.793 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:20:48 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-13T23:20:48.793 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:20:48 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-13T23:20:48.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:20:48 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-13T23:20:48.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:20:48 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-13T23:20:48.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:20:48 smithi047 conmon[78521]: raise exceptions[0] 2022-05-13T23:20:48.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:20:48 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-13T23:20:48.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:20:48 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-13T23:20:48.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:20:48 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-13T23:20:48.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:20:48 smithi047 conmon[78521]: return (yield from fut) 2022-05-13T23:20:48.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:20:48 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-13T23:20:48.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:20:48 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-13T23:20:48.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:20:48 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:20:50.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:20:50 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:20:50.542 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-13T23:20:50.542 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-13T23:20:50.542 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-13T23:20:50.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-13T23:20:50.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-13T23:20:50.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-13T23:20:50.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-13T23:20:50.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-13T23:20:50.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-13T23:20:50.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-13T23:20:50.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-13T23:20:50.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-13T23:20:50.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-13T23:20:50.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-13T23:20:50.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-13T23:20:50.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-13T23:20:50.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-13T23:20:50.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:20:50.546 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:20:50 smithi047 ceph-mon[77147]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-13T23:20:50.546 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:20:50 smithi047 ceph-mon[77147]: pgmap v84: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 138 MiB data, 465 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s; 10439/32427 objects degraded (32.192%) 2022-05-13T23:20:50.546 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:20:50 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:20:50.546 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:20:50 smithi047 ceph-mon[77147]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2022-05-13T23:20:50.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:20:50 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:20:50.608 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-13T23:20:50.609 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-13T23:20:50.609 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-13T23:20:50.609 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-13T23:20:50.609 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-13T23:20:50.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-13T23:20:50.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-13T23:20:50.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-13T23:20:50.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-13T23:20:50.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-13T23:20:50.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-13T23:20:50.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-13T23:20:50.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-13T23:20:50.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-13T23:20:50.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-13T23:20:50.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-13T23:20:50.613 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-13T23:20:50.613 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:20:50.613 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:20:50 smithi115 ceph-mon[78593]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-13T23:20:50.613 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:20:50 smithi115 ceph-mon[78593]: pgmap v84: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 138 MiB data, 465 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s; 10439/32427 objects degraded (32.192%) 2022-05-13T23:20:50.614 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:20:50 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:20:50.614 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:20:50 smithi115 ceph-mon[78593]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2022-05-13T23:20:51.505 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:20:51 smithi047 ceph-mon[77147]: 2 slow requests, 0 included below; oldest blocked for > 214.075437 secs 2022-05-13T23:20:51.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:20:51 smithi115 ceph-mon[78593]: 2 slow requests, 0 included below; oldest blocked for > 214.075437 secs 2022-05-13T23:20:51.793 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:20:51 smithi047 conmon[78521]: 2022-05-13T23:20:51.503+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:20:51.793 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:20:51 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-13T23:20:51.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:20:51 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-13T23:20:51.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:20:51 smithi047 conmon[78521]: yield 2022-05-13T23:20:51.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:20:51 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-13T23:20:51.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:20:51 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-13T23:20:51.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:20:51 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-13T23:20:51.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:20:51 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-13T23:20:51.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:20:51 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-13T23:20:51.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:20:51 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-13T23:20:51.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:20:51 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-13T23:20:51.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:20:51 smithi047 conmon[78521]: raise exceptions[0] 2022-05-13T23:20:51.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:20:51 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-13T23:20:51.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:20:51 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-13T23:20:51.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:20:51 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-13T23:20:51.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:20:51 smithi047 conmon[78521]: return (yield from fut) 2022-05-13T23:20:51.797 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:20:51 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-13T23:20:51.797 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:20:51 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-13T23:20:51.797 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:20:51 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:20:51.797 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:20:51 smithi047 conmon[78521]: 2022-05-13T23:20:51.504+0000 7f533ef76700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-13T23:20:51.798 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:20:51 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-13T23:20:51.798 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:20:51 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-13T23:20:51.798 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:20:51 smithi047 conmon[78521]: yield 2022-05-13T23:20:51.798 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:20:51 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-13T23:20:51.799 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:20:51 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-13T23:20:51.799 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:20:51 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-13T23:20:51.799 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:20:51 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-13T23:20:51.799 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:20:51 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-13T23:20:51.800 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:20:51 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-13T23:20:51.800 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:20:51 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-13T23:20:51.800 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:20:51 smithi047 conmon[78521]: raise exceptions[0] 2022-05-13T23:20:51.800 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:20:51 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-13T23:20:51.801 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:20:51 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-13T23:20:51.801 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:20:51 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-13T23:20:51.801 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:20:51 smithi047 conmon[78521]: return (yield from fut) 2022-05-13T23:20:51.802 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:20:51 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-13T23:20:51.802 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:20:51 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-13T23:20:51.802 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:20:51 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:20:51.802 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:20:51 smithi047 conmon[78521]: 2022-05-13T23:20:51.802 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:20:51 smithi047 conmon[78521]: During handling of the above exception, another exception occurred: 2022-05-13T23:20:51.803 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:20:51 smithi047 conmon[78521]: 2022-05-13T23:20:51.803 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:20:51 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-13T23:20:51.803 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:20:51 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-13T23:20:51.803 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:20:51 smithi047 conmon[78521]: return f(*arg) 2022-05-13T23:20:51.803 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:20:51 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-13T23:20:51.804 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:20:51 smithi047 conmon[78521]: self._write_client_files(client_files, host) 2022-05-13T23:20:51.804 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:20:51 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-13T23:20:51.804 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:20:51 smithi047 conmon[78521]: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-13T23:20:51.804 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:20:51 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-13T23:20:51.805 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:20:51 smithi047 conmon[78521]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-13T23:20:51.805 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:20:51 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-13T23:20:51.805 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:20:51 smithi047 conmon[78521]: return self.event_loop.get_result(coro) 2022-05-13T23:20:51.805 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:20:51 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-13T23:20:51.806 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:20:51 smithi047 conmon[78521]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-13T23:20:51.806 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:20:51 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-13T23:20:51.806 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:20:51 smithi047 conmon[78521]: return self.__get_result() 2022-05-13T23:20:51.806 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:20:51 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-13T23:20:51.807 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:20:51 smithi047 conmon[78521]: raise self._exception 2022-05-13T23:20:51.807 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:20:51 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-13T23:20:51.807 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:20:51 smithi047 conmon[78521]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-13T23:20:51.807 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:20:51 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-13T23:20:51.807 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:20:51 smithi047 conmon[78521]: conn = await self._remote_connection(host, addr) 2022-05-13T23:20:51.808 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:20:51 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-13T23:20:51.808 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:20:51 smithi047 conmon[78521]: raise 2022-05-13T23:20:51.808 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:20:51 smithi047 conmon[78521]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-13T23:20:51.808 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:20:51 smithi047 conmon[78521]: self.gen.throw(type, value, traceback) 2022-05-13T23:20:51.809 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:20:51 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-13T23:20:51.809 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:20:51 smithi047 conmon[78521]: raise OrchestratorError(msg) 2022-05-13T23:20:51.809 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:20:51 smithi047 conmon[78521]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:20:52.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:20:52 smithi047 ceph-mon[77147]: pgmap v85: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 138 MiB data, 465 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 3 op/s; 10439/32427 objects degraded (32.192%) 2022-05-13T23:20:52.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:20:52 smithi115 ceph-mon[78593]: pgmap v85: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 138 MiB data, 465 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 3 op/s; 10439/32427 objects degraded (32.192%) 2022-05-13T23:20:53.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:20:53 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:20:53.542 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-13T23:20:53.542 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-13T23:20:53.542 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-13T23:20:53.542 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-13T23:20:53.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-13T23:20:53.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-13T23:20:53.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-13T23:20:53.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-13T23:20:53.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-13T23:20:53.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-13T23:20:53.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-13T23:20:53.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-13T23:20:53.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-13T23:20:53.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-13T23:20:53.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-13T23:20:53.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-13T23:20:53.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-13T23:20:53.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:20:53.546 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:20:53 smithi047 ceph-mon[77147]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-13T23:20:53.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-13T23:20:53.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-13T23:20:53.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-13T23:20:53.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-13T23:20:53.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-13T23:20:53.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-13T23:20:53.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-13T23:20:53.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-13T23:20:53.548 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-13T23:20:53.548 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-13T23:20:53.548 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-13T23:20:53.548 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-13T23:20:53.550 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-13T23:20:53.550 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-13T23:20:53.550 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-13T23:20:53.550 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-13T23:20:53.551 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-13T23:20:53.551 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:20:53.552 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-13T23:20:53.552 INFO:journalctl@ceph.mon.b.smithi047.stdout: During handling of the above exception, another exception occurred: 2022-05-13T23:20:53.552 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-13T23:20:53.552 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-13T23:20:53.552 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-13T23:20:53.553 INFO:journalctl@ceph.mon.b.smithi047.stdout: return f(*arg) 2022-05-13T23:20:53.554 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-13T23:20:53.554 INFO:journalctl@ceph.mon.b.smithi047.stdout: self._write_client_files(client_files, host) 2022-05-13T23:20:53.554 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-13T23:20:53.555 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-13T23:20:53.557 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-13T23:20:53.557 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-13T23:20:53.557 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-13T23:20:53.558 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.event_loop.get_result(coro) 2022-05-13T23:20:53.558 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-13T23:20:53.558 INFO:journalctl@ceph.mon.b.smithi047.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-13T23:20:53.558 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-13T23:20:53.559 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.__get_result() 2022-05-13T23:20:53.559 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-13T23:20:53.559 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise self._exception 2022-05-13T23:20:53.559 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-13T23:20:53.559 INFO:journalctl@ceph.mon.b.smithi047.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-13T23:20:53.560 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-13T23:20:53.560 INFO:journalctl@ceph.mon.b.smithi047.stdout: conn = await self._remote_connection(host, addr) 2022-05-13T23:20:53.560 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-13T23:20:53.560 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise 2022-05-13T23:20:53.561 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-13T23:20:53.561 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.gen.throw(type, value, traceback) 2022-05-13T23:20:53.561 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-13T23:20:53.562 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OrchestratorError(msg) 2022-05-13T23:20:53.562 INFO:journalctl@ceph.mon.b.smithi047.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:20:53.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:20:53 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:20:53.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-13T23:20:53.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-13T23:20:53.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-13T23:20:53.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-13T23:20:53.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-13T23:20:53.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-13T23:20:53.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-13T23:20:53.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-13T23:20:53.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-13T23:20:53.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-13T23:20:53.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-13T23:20:53.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-13T23:20:53.613 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-13T23:20:53.613 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-13T23:20:53.613 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-13T23:20:53.613 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-13T23:20:53.613 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-13T23:20:53.614 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:20:53.614 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:20:53 smithi115 ceph-mon[78593]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-13T23:20:53.614 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-13T23:20:53.614 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-13T23:20:53.614 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-13T23:20:53.615 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-13T23:20:53.615 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-13T23:20:53.615 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-13T23:20:53.615 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-13T23:20:53.616 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-13T23:20:53.616 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-13T23:20:53.616 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-13T23:20:53.616 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-13T23:20:53.616 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-13T23:20:53.617 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-13T23:20:53.617 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-13T23:20:53.617 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-13T23:20:53.617 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-13T23:20:53.618 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-13T23:20:53.618 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:20:53.618 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-13T23:20:53.618 INFO:journalctl@ceph.mon.c.smithi115.stdout: During handling of the above exception, another exception occurred: 2022-05-13T23:20:53.618 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-13T23:20:53.619 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-13T23:20:53.619 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-13T23:20:53.619 INFO:journalctl@ceph.mon.c.smithi115.stdout: return f(*arg) 2022-05-13T23:20:53.619 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-13T23:20:53.619 INFO:journalctl@ceph.mon.c.smithi115.stdout: self._write_client_files(client_files, host) 2022-05-13T23:20:53.620 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-13T23:20:53.620 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-13T23:20:53.620 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-13T23:20:53.620 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-13T23:20:53.621 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-13T23:20:53.621 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.event_loop.get_result(coro) 2022-05-13T23:20:53.621 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-13T23:20:53.621 INFO:journalctl@ceph.mon.c.smithi115.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-13T23:20:53.621 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-13T23:20:53.622 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.__get_result() 2022-05-13T23:20:53.622 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-13T23:20:53.622 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise self._exception 2022-05-13T23:20:53.622 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-13T23:20:53.622 INFO:journalctl@ceph.mon.c.smithi115.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-13T23:20:53.623 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-13T23:20:53.623 INFO:journalctl@ceph.mon.c.smithi115.stdout: conn = await self._remote_connection(host, addr) 2022-05-13T23:20:53.623 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-13T23:20:53.623 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise 2022-05-13T23:20:53.624 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-13T23:20:53.624 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.gen.throw(type, value, traceback) 2022-05-13T23:20:53.624 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-13T23:20:53.624 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OrchestratorError(msg) 2022-05-13T23:20:53.624 INFO:journalctl@ceph.mon.c.smithi115.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:20:54.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:20:54 smithi047 ceph-mon[77147]: pgmap v86: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 138 MiB data, 465 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 3 op/s; 10439/32427 objects degraded (32.192%) 2022-05-13T23:20:54.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:20:54 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:20:54.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:20:54 smithi047 ceph-mon[77147]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2022-05-13T23:20:54.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:20:54 smithi115 ceph-mon[78593]: pgmap v86: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 138 MiB data, 465 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 3 op/s; 10439/32427 objects degraded (32.192%) 2022-05-13T23:20:54.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:20:54 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:20:54.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:20:54 smithi115 ceph-mon[78593]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2022-05-13T23:20:56.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:20:56 smithi047 ceph-mon[77147]: pgmap v87: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 138 MiB data, 465 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s; 10439/32427 objects degraded (32.192%) 2022-05-13T23:20:56.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:20:56 smithi047 ceph-mon[77147]: 2 slow requests, 0 included below; oldest blocked for > 219.075576 secs 2022-05-13T23:20:56.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:20:56 smithi115 ceph-mon[78593]: pgmap v87: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 138 MiB data, 465 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s; 10439/32427 objects degraded (32.192%) 2022-05-13T23:20:56.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:20:56 smithi115 ceph-mon[78593]: 2 slow requests, 0 included below; oldest blocked for > 219.075576 secs 2022-05-13T23:20:58.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:20:58 smithi047 ceph-mon[77147]: pgmap v88: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 138 MiB data, 465 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 3 op/s; 10439/32427 objects degraded (32.192%) 2022-05-13T23:20:58.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:20:58 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:20:58.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:20:58 smithi047 ceph-mon[77147]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2022-05-13T23:20:58.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:20:58 smithi115 ceph-mon[78593]: pgmap v88: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 138 MiB data, 465 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 3 op/s; 10439/32427 objects degraded (32.192%) 2022-05-13T23:20:58.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:20:58 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:20:58.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:20:58 smithi115 ceph-mon[78593]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2022-05-13T23:21:00.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:21:00 smithi047 ceph-mon[77147]: pgmap v89: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 138 MiB data, 465 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s; 10439/32427 objects degraded (32.192%) 2022-05-13T23:21:00.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:21:00 smithi115 ceph-mon[78593]: pgmap v89: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 138 MiB data, 465 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s; 10439/32427 objects degraded (32.192%) 2022-05-13T23:21:01.546 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:21:01 smithi047 ceph-mon[77147]: 2 slow requests, 0 included below; oldest blocked for > 224.075691 secs 2022-05-13T23:21:01.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:21:01 smithi115 ceph-mon[78593]: 2 slow requests, 0 included below; oldest blocked for > 224.075691 secs 2022-05-13T23:21:02.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:21:02 smithi047 ceph-mon[77147]: pgmap v90: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 138 MiB data, 465 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 3 op/s; 10439/32427 objects degraded (32.192%) 2022-05-13T23:21:02.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:21:02 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:21:02.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:21:02 smithi047 ceph-mon[77147]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2022-05-13T23:21:02.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:21:02 smithi115 ceph-mon[78593]: pgmap v90: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 138 MiB data, 465 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 3 op/s; 10439/32427 objects degraded (32.192%) 2022-05-13T23:21:02.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:21:02 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:21:02.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:21:02 smithi115 ceph-mon[78593]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2022-05-13T23:21:04.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:21:03 smithi047 ceph-mon[77147]: pgmap v91: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 138 MiB data, 465 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 3 op/s; 10439/32427 objects degraded (32.192%) 2022-05-13T23:21:04.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:21:04 smithi115 ceph-mon[78593]: pgmap v91: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 138 MiB data, 465 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 3 op/s; 10439/32427 objects degraded (32.192%) 2022-05-13T23:21:06.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:21:06 smithi047 ceph-mon[77147]: pgmap v92: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 138 MiB data, 465 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s; 10439/32427 objects degraded (32.192%) 2022-05-13T23:21:06.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:21:06 smithi047 ceph-mon[77147]: 2 slow requests, 0 included below; oldest blocked for > 229.075816 secs 2022-05-13T23:21:06.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:21:06 smithi115 ceph-mon[78593]: pgmap v92: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 138 MiB data, 465 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s; 10439/32427 objects degraded (32.192%) 2022-05-13T23:21:06.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:21:06 smithi115 ceph-mon[78593]: 2 slow requests, 0 included below; oldest blocked for > 229.075816 secs 2022-05-13T23:21:08.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:21:08 smithi047 ceph-mon[77147]: pgmap v93: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 138 MiB data, 465 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 3 op/s; 10439/32427 objects degraded (32.192%) 2022-05-13T23:21:08.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:21:08 smithi115 ceph-mon[78593]: pgmap v93: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 138 MiB data, 465 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 3 op/s; 10439/32427 objects degraded (32.192%) 2022-05-13T23:21:10.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:21:10 smithi047 ceph-mon[77147]: pgmap v94: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 138 MiB data, 465 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s; 10439/32427 objects degraded (32.192%) 2022-05-13T23:21:10.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:21:10 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:21:10.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:21:10 smithi047 ceph-mon[77147]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2022-05-13T23:21:10.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:21:10 smithi115 ceph-mon[78593]: pgmap v94: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 138 MiB data, 465 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s; 10439/32427 objects degraded (32.192%) 2022-05-13T23:21:10.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:21:10 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:21:10.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:21:10 smithi115 ceph-mon[78593]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2022-05-13T23:21:11.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:21:11 smithi047 ceph-mon[77147]: 2 slow requests, 0 included below; oldest blocked for > 234.075931 secs 2022-05-13T23:21:11.752 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:21:11 smithi115 ceph-mon[78593]: 2 slow requests, 0 included below; oldest blocked for > 234.075931 secs 2022-05-13T23:21:12.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:21:12 smithi047 ceph-mon[77147]: pgmap v95: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 138 MiB data, 465 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 3 op/s; 10439/32427 objects degraded (32.192%) 2022-05-13T23:21:12.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:21:12 smithi115 ceph-mon[78593]: pgmap v95: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 138 MiB data, 465 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 3 op/s; 10439/32427 objects degraded (32.192%) 2022-05-13T23:21:14.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:21:13 smithi047 ceph-mon[77147]: pgmap v96: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 138 MiB data, 465 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 3 op/s; 10439/32427 objects degraded (32.192%) 2022-05-13T23:21:14.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:21:13 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-13T23:21:14.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:21:13 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-13T23:21:14.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:21:13 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:21:14.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:21:13 smithi047 ceph-mon[77147]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2022-05-13T23:21:14.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:21:14 smithi115 ceph-mon[78593]: pgmap v96: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 138 MiB data, 465 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 3 op/s; 10439/32427 objects degraded (32.192%) 2022-05-13T23:21:14.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:21:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-13T23:21:14.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:21:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-13T23:21:14.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:21:14 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:21:14.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:21:14 smithi115 ceph-mon[78593]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2022-05-13T23:21:16.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:21:16 smithi047 ceph-mon[77147]: pgmap v97: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 138 MiB data, 465 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s; 10439/32427 objects degraded (32.192%) 2022-05-13T23:21:16.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:21:16 smithi047 ceph-mon[77147]: 2 slow requests, 0 included below; oldest blocked for > 239.076053 secs 2022-05-13T23:21:16.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:21:16 smithi115 ceph-mon[78593]: pgmap v97: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 138 MiB data, 465 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s; 10439/32427 objects degraded (32.192%) 2022-05-13T23:21:16.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:21:16 smithi115 ceph-mon[78593]: 2 slow requests, 0 included below; oldest blocked for > 239.076053 secs 2022-05-13T23:21:18.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:21:18 smithi047 ceph-mon[77147]: pgmap v98: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 138 MiB data, 465 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 3 op/s; 10439/32427 objects degraded (32.192%) 2022-05-13T23:21:18.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:21:18 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:21:18.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:21:18 smithi047 ceph-mon[77147]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2022-05-13T23:21:18.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:21:18 smithi115 ceph-mon[78593]: pgmap v98: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 138 MiB data, 465 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 3 op/s; 10439/32427 objects degraded (32.192%) 2022-05-13T23:21:18.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:21:18 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:21:18.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:21:18 smithi115 ceph-mon[78593]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2022-05-13T23:21:20.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:21:20 smithi047 ceph-mon[77147]: pgmap v99: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 138 MiB data, 465 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s; 10439/32427 objects degraded (32.192%) 2022-05-13T23:21:20.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:21:20 smithi115 ceph-mon[78593]: pgmap v99: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 138 MiB data, 465 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s; 10439/32427 objects degraded (32.192%) 2022-05-13T23:21:21.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:21:21 smithi047 ceph-mon[77147]: 2 slow requests, 0 included below; oldest blocked for > 244.076157 secs 2022-05-13T23:21:21.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:21:21 smithi115 ceph-mon[78593]: 2 slow requests, 0 included below; oldest blocked for > 244.076157 secs 2022-05-13T23:21:22.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:21:22 smithi047 ceph-mon[77147]: pgmap v100: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 138 MiB data, 465 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 3 op/s; 10439/32427 objects degraded (32.192%) 2022-05-13T23:21:22.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:21:22 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:21:22.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:21:22 smithi047 ceph-mon[77147]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2022-05-13T23:21:22.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:21:22 smithi115 ceph-mon[78593]: pgmap v100: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 138 MiB data, 465 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 3 op/s; 10439/32427 objects degraded (32.192%) 2022-05-13T23:21:22.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:21:22 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:21:22.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:21:22 smithi115 ceph-mon[78593]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2022-05-13T23:21:24.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:21:24 smithi047 ceph-mon[77147]: pgmap v101: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 138 MiB data, 465 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 3 op/s; 10439/32427 objects degraded (32.192%) 2022-05-13T23:21:24.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:21:24 smithi115 ceph-mon[78593]: pgmap v101: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 138 MiB data, 465 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 3 op/s; 10439/32427 objects degraded (32.192%) 2022-05-13T23:21:26.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:21:26 smithi047 ceph-mon[77147]: pgmap v102: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 138 MiB data, 465 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s; 10439/32427 objects degraded (32.192%) 2022-05-13T23:21:26.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:21:26 smithi047 ceph-mon[77147]: 2 slow requests, 0 included below; oldest blocked for > 249.076254 secs 2022-05-13T23:21:26.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:21:26 smithi115 ceph-mon[78593]: pgmap v102: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 138 MiB data, 465 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s; 10439/32427 objects degraded (32.192%) 2022-05-13T23:21:26.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:21:26 smithi115 ceph-mon[78593]: 2 slow requests, 0 included below; oldest blocked for > 249.076254 secs 2022-05-13T23:21:28.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:21:28 smithi047 ceph-mon[77147]: pgmap v103: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 138 MiB data, 465 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 3 op/s; 10439/32427 objects degraded (32.192%) 2022-05-13T23:21:28.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:21:28 smithi115 ceph-mon[78593]: pgmap v103: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 138 MiB data, 465 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 3 op/s; 10439/32427 objects degraded (32.192%) 2022-05-13T23:21:30.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:21:30 smithi047 ceph-mon[77147]: pgmap v104: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 138 MiB data, 465 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s; 10439/32427 objects degraded (32.192%) 2022-05-13T23:21:30.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:21:30 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:21:30.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:21:30 smithi047 ceph-mon[77147]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2022-05-13T23:21:30.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:21:30 smithi115 ceph-mon[78593]: pgmap v104: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 138 MiB data, 465 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s; 10439/32427 objects degraded (32.192%) 2022-05-13T23:21:30.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:21:30 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:21:30.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:21:30 smithi115 ceph-mon[78593]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2022-05-13T23:21:31.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:21:31 smithi047 ceph-mon[77147]: 2 slow requests, 0 included below; oldest blocked for > 254.076382 secs 2022-05-13T23:21:31.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:21:31 smithi115 ceph-mon[78593]: 2 slow requests, 0 included below; oldest blocked for > 254.076382 secs 2022-05-13T23:21:32.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:21:32 smithi047 ceph-mon[77147]: pgmap v105: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 138 MiB data, 465 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 3 op/s; 10439/32427 objects degraded (32.192%) 2022-05-13T23:21:32.614 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:21:32 smithi115 ceph-mon[78593]: pgmap v105: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 138 MiB data, 465 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 3 op/s; 10439/32427 objects degraded (32.192%) 2022-05-13T23:21:34.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:21:34 smithi047 ceph-mon[77147]: pgmap v106: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 138 MiB data, 465 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 3 op/s; 10439/32427 objects degraded (32.192%) 2022-05-13T23:21:34.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:21:34 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:21:34.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:21:34 smithi047 ceph-mon[77147]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2022-05-13T23:21:34.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:21:34 smithi115 ceph-mon[78593]: pgmap v106: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 138 MiB data, 465 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 3 op/s; 10439/32427 objects degraded (32.192%) 2022-05-13T23:21:34.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:21:34 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:21:34.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:21:34 smithi115 ceph-mon[78593]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2022-05-13T23:21:36.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:21:36 smithi047 ceph-mon[77147]: pgmap v107: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 138 MiB data, 465 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s; 10439/32427 objects degraded (32.192%) 2022-05-13T23:21:36.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:21:36 smithi047 ceph-mon[77147]: 2 slow requests, 0 included below; oldest blocked for > 259.076504 secs 2022-05-13T23:21:36.616 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:21:36 smithi115 ceph-mon[78593]: pgmap v107: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 138 MiB data, 465 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s; 10439/32427 objects degraded (32.192%) 2022-05-13T23:21:36.616 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:21:36 smithi115 ceph-mon[78593]: 2 slow requests, 0 included below; oldest blocked for > 259.076504 secs 2022-05-13T23:21:38.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:21:38 smithi047 ceph-mon[77147]: pgmap v108: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 138 MiB data, 465 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 3 op/s; 10439/32427 objects degraded (32.192%) 2022-05-13T23:21:38.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:21:38 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:21:38.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:21:38 smithi047 ceph-mon[77147]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2022-05-13T23:21:38.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:21:38 smithi115 ceph-mon[78593]: pgmap v108: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 138 MiB data, 465 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 3 op/s; 10439/32427 objects degraded (32.192%) 2022-05-13T23:21:38.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:21:38 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:21:38.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:21:38 smithi115 ceph-mon[78593]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2022-05-13T23:21:40.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:21:40 smithi047 ceph-mon[77147]: pgmap v109: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 138 MiB data, 465 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s; 10439/32427 objects degraded (32.192%) 2022-05-13T23:21:40.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:21:40 smithi115 ceph-mon[78593]: pgmap v109: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 138 MiB data, 465 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s; 10439/32427 objects degraded (32.192%) 2022-05-13T23:21:41.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:21:41 smithi047 ceph-mon[77147]: 2 slow requests, 0 included below; oldest blocked for > 264.076614 secs 2022-05-13T23:21:41.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:21:41 smithi115 ceph-mon[78593]: 2 slow requests, 0 included below; oldest blocked for > 264.076614 secs 2022-05-13T23:21:42.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:21:42 smithi047 ceph-mon[77147]: pgmap v110: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 138 MiB data, 465 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 3 op/s; 10439/32427 objects degraded (32.192%) 2022-05-13T23:21:42.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:21:42 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:21:42.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:21:42 smithi047 ceph-mon[77147]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2022-05-13T23:21:42.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:21:42 smithi115 ceph-mon[78593]: pgmap v110: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 138 MiB data, 465 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 3 op/s; 10439/32427 objects degraded (32.192%) 2022-05-13T23:21:42.612 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:21:42 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:21:42.612 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:21:42 smithi115 ceph-mon[78593]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2022-05-13T23:21:44.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:21:44 smithi047 ceph-mon[77147]: pgmap v111: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 138 MiB data, 465 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 3 op/s; 10439/32427 objects degraded (32.192%) 2022-05-13T23:21:44.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:21:44 smithi115 ceph-mon[78593]: pgmap v111: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 138 MiB data, 465 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 3 op/s; 10439/32427 objects degraded (32.192%) 2022-05-13T23:21:46.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:21:46 smithi047 ceph-mon[77147]: pgmap v112: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 138 MiB data, 465 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s; 10439/32427 objects degraded (32.192%) 2022-05-13T23:21:46.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:21:46 smithi047 ceph-mon[77147]: 2 slow requests, 0 included below; oldest blocked for > 269.076757 secs 2022-05-13T23:21:46.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:21:46 smithi115 ceph-mon[78593]: pgmap v112: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 138 MiB data, 465 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s; 10439/32427 objects degraded (32.192%) 2022-05-13T23:21:46.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:21:46 smithi115 ceph-mon[78593]: 2 slow requests, 0 included below; oldest blocked for > 269.076757 secs 2022-05-13T23:21:48.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:21:48 smithi047 ceph-mon[77147]: pgmap v113: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 138 MiB data, 465 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 3 op/s; 10439/32427 objects degraded (32.192%) 2022-05-13T23:21:48.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:21:48 smithi115 ceph-mon[78593]: pgmap v113: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 138 MiB data, 465 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 3 op/s; 10439/32427 objects degraded (32.192%) 2022-05-13T23:21:50.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:21:50 smithi047 ceph-mon[77147]: pgmap v114: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 138 MiB data, 465 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s; 10439/32427 objects degraded (32.192%) 2022-05-13T23:21:50.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:21:50 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:21:50.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:21:50 smithi047 ceph-mon[77147]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2022-05-13T23:21:50.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:21:50 smithi115 ceph-mon[78593]: pgmap v114: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 138 MiB data, 465 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s; 10439/32427 objects degraded (32.192%) 2022-05-13T23:21:50.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:21:50 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:21:50.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:21:50 smithi115 ceph-mon[78593]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2022-05-13T23:21:51.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:21:51 smithi047 ceph-mon[77147]: 2 slow requests, 0 included below; oldest blocked for > 274.076877 secs 2022-05-13T23:21:51.554 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:21:51 smithi115 ceph-mon[78593]: 2 slow requests, 0 included below; oldest blocked for > 274.076877 secs 2022-05-13T23:21:52.029 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:21:51 smithi047 conmon[78521]: 2022-05-13T23:21:51.665+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:21:52.029 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:21:51 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-13T23:21:52.030 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:21:51 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-13T23:21:52.030 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:21:51 smithi047 conmon[78521]: yield 2022-05-13T23:21:52.030 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:21:51 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-13T23:21:52.030 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:21:51 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-13T23:21:52.031 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:21:51 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-13T23:21:52.031 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:21:51 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-13T23:21:52.031 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:21:51 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-13T23:21:52.031 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:21:51 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-13T23:21:52.031 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:21:51 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-13T23:21:52.032 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:21:51 smithi047 conmon[78521]: raise exceptions[0] 2022-05-13T23:21:52.032 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:21:51 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-13T23:21:52.032 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:21:51 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-13T23:21:52.032 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:21:51 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-13T23:21:52.032 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:21:51 smithi047 conmon[78521]: return (yield from fut) 2022-05-13T23:21:52.033 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:21:51 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-13T23:21:52.033 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:21:51 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-13T23:21:52.033 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:21:51 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:21:52.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:21:52 smithi047 ceph-mon[77147]: pgmap v115: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 138 MiB data, 465 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 3 op/s; 10439/32427 objects degraded (32.192%) 2022-05-13T23:21:52.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:21:52 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-13T23:21:52.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:21:52 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-13T23:21:52.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:21:52 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-13T23:21:52.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:21:52 smithi115 ceph-mon[78593]: pgmap v115: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 138 MiB data, 465 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 3 op/s; 10439/32427 objects degraded (32.192%) 2022-05-13T23:21:52.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:21:52 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-13T23:21:52.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:21:52 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-13T23:21:52.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:21:52 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-13T23:21:53.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:21:53 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:21:53.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-13T23:21:53.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-13T23:21:53.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-13T23:21:53.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-13T23:21:53.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-13T23:21:53.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-13T23:21:53.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-13T23:21:53.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-13T23:21:53.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-13T23:21:53.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-13T23:21:53.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-13T23:21:53.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-13T23:21:53.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-13T23:21:53.548 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-13T23:21:53.548 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-13T23:21:53.548 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-13T23:21:53.548 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-13T23:21:53.548 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:21:53.549 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:21:53 smithi047 ceph-mon[77147]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-13T23:21:53.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:21:53 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:21:53.608 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-13T23:21:53.609 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-13T23:21:53.609 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-13T23:21:53.609 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-13T23:21:53.609 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-13T23:21:53.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-13T23:21:53.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-13T23:21:53.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-13T23:21:53.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-13T23:21:53.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-13T23:21:53.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-13T23:21:53.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-13T23:21:53.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-13T23:21:53.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-13T23:21:53.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-13T23:21:53.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-13T23:21:53.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-13T23:21:53.613 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:21:53.613 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:21:53 smithi115 ceph-mon[78593]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-13T23:21:54.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:21:54 smithi047 ceph-mon[77147]: pgmap v116: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 138 MiB data, 465 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 3 op/s; 10439/32427 objects degraded (32.192%) 2022-05-13T23:21:54.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:21:54 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:21:54.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:21:54 smithi047 ceph-mon[77147]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2022-05-13T23:21:54.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:21:54 smithi115 ceph-mon[78593]: pgmap v116: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 138 MiB data, 465 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 3 op/s; 10439/32427 objects degraded (32.192%) 2022-05-13T23:21:54.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:21:54 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:21:54.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:21:54 smithi115 ceph-mon[78593]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2022-05-13T23:21:55.042 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:21:54 smithi047 conmon[78521]: 2022-05-13T23:21:54.736+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:21:55.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:21:54 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-13T23:21:55.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:21:54 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-13T23:21:55.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:21:54 smithi047 conmon[78521]: yield 2022-05-13T23:21:55.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:21:54 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-13T23:21:55.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:21:54 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-13T23:21:55.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:21:54 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-13T23:21:55.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:21:54 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-13T23:21:55.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:21:54 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-13T23:21:55.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:21:54 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-13T23:21:55.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:21:54 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-13T23:21:55.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:21:54 smithi047 conmon[78521]: raise exceptions[0] 2022-05-13T23:21:55.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:21:54 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-13T23:21:55.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:21:54 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-13T23:21:55.048 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:21:54 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-13T23:21:55.048 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:21:54 smithi047 conmon[78521]: return (yield from fut) 2022-05-13T23:21:55.048 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:21:54 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-13T23:21:55.052 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:21:54 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-13T23:21:55.052 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:21:54 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:21:55.052 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:21:54 smithi047 conmon[78521]: 2022-05-13T23:21:54.737+0000 7f5345f84700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-13T23:21:55.053 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:21:54 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-13T23:21:55.053 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:21:54 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-13T23:21:55.053 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:21:54 smithi047 conmon[78521]: yield 2022-05-13T23:21:55.053 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:21:54 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-13T23:21:55.054 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:21:54 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-13T23:21:55.054 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:21:54 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-13T23:21:55.054 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:21:54 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-13T23:21:55.054 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:21:54 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-13T23:21:55.054 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:21:54 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-13T23:21:55.055 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:21:54 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-13T23:21:55.055 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:21:54 smithi047 conmon[78521]: raise exceptions[0] 2022-05-13T23:21:55.055 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:21:54 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-13T23:21:55.055 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:21:54 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-13T23:21:55.055 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:21:54 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-13T23:21:55.056 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:21:54 smithi047 conmon[78521]: return (yield from fut) 2022-05-13T23:21:55.056 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:21:54 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-13T23:21:55.056 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:21:54 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-13T23:21:55.056 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:21:54 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:21:55.056 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:21:54 smithi047 conmon[78521]: 2022-05-13T23:21:55.057 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:21:54 smithi047 conmon[78521]: During handling of the above exception, another exception occurred: 2022-05-13T23:21:55.057 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:21:54 smithi047 conmon[78521]: 2022-05-13T23:21:55.057 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:21:54 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-13T23:21:55.057 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:21:54 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-13T23:21:55.058 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:21:54 smithi047 conmon[78521]: return f(*arg) 2022-05-13T23:21:55.058 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:21:54 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-13T23:21:55.058 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:21:54 smithi047 conmon[78521]: self._write_client_files(client_files, host) 2022-05-13T23:21:55.058 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:21:54 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-13T23:21:55.058 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:21:54 smithi047 conmon[78521]: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-13T23:21:55.059 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:21:54 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-13T23:21:55.065 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:21:54 smithi047 conmon[78521]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-13T23:21:55.066 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:21:54 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-13T23:21:55.066 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:21:54 smithi047 conmon[78521]: return self.event_loop.get_result(coro) 2022-05-13T23:21:55.066 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:21:54 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-13T23:21:55.067 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:21:54 smithi047 conmon[78521]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-13T23:21:55.067 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:21:54 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-13T23:21:55.068 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:21:54 smithi047 conmon[78521]: return self.__get_result() 2022-05-13T23:21:55.068 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:21:54 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-13T23:21:55.069 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:21:54 smithi047 conmon[78521]: raise self._exception 2022-05-13T23:21:55.069 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:21:54 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-13T23:21:55.069 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:21:54 smithi047 conmon[78521]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-13T23:21:55.070 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:21:54 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-13T23:21:55.070 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:21:54 smithi047 conmon[78521]: conn = await self._remote_connection(host, addr) 2022-05-13T23:21:55.070 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:21:54 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-13T23:21:55.071 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:21:54 smithi047 conmon[78521]: raise 2022-05-13T23:21:55.071 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:21:54 smithi047 conmon[78521]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-13T23:21:55.071 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:21:54 smithi047 conmon[78521]: self.gen.throw(type, value, traceback) 2022-05-13T23:21:55.072 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:21:54 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-13T23:21:55.072 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:21:54 smithi047 conmon[78521]: raise OrchestratorError(msg) 2022-05-13T23:21:55.072 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:21:54 smithi047 conmon[78521]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:21:56.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:21:56 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:21:56.542 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-13T23:21:56.542 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-13T23:21:56.542 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-13T23:21:56.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-13T23:21:56.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-13T23:21:56.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-13T23:21:56.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-13T23:21:56.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-13T23:21:56.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-13T23:21:56.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-13T23:21:56.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-13T23:21:56.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-13T23:21:56.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-13T23:21:56.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-13T23:21:56.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-13T23:21:56.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-13T23:21:56.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-13T23:21:56.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:21:56.546 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:21:56 smithi047 ceph-mon[77147]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-13T23:21:56.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-13T23:21:56.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-13T23:21:56.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-13T23:21:56.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-13T23:21:56.548 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-13T23:21:56.548 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-13T23:21:56.548 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-13T23:21:56.548 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-13T23:21:56.557 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-13T23:21:56.557 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-13T23:21:56.557 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-13T23:21:56.557 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-13T23:21:56.558 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-13T23:21:56.558 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-13T23:21:56.558 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-13T23:21:56.558 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-13T23:21:56.558 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-13T23:21:56.559 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:21:56.559 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-13T23:21:56.559 INFO:journalctl@ceph.mon.b.smithi047.stdout: During handling of the above exception, another exception occurred: 2022-05-13T23:21:56.559 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-13T23:21:56.559 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-13T23:21:56.560 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-13T23:21:56.560 INFO:journalctl@ceph.mon.b.smithi047.stdout: return f(*arg) 2022-05-13T23:21:56.560 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-13T23:21:56.560 INFO:journalctl@ceph.mon.b.smithi047.stdout: self._write_client_files(client_files, host) 2022-05-13T23:21:56.561 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-13T23:21:56.561 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-13T23:21:56.561 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-13T23:21:56.561 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-13T23:21:56.561 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-13T23:21:56.562 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.event_loop.get_result(coro) 2022-05-13T23:21:56.562 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-13T23:21:56.562 INFO:journalctl@ceph.mon.b.smithi047.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-13T23:21:56.562 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-13T23:21:56.563 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.__get_result() 2022-05-13T23:21:56.563 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-13T23:21:56.563 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise self._exception 2022-05-13T23:21:56.563 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-13T23:21:56.563 INFO:journalctl@ceph.mon.b.smithi047.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-13T23:21:56.564 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-13T23:21:56.565 INFO:journalctl@ceph.mon.b.smithi047.stdout: conn = await self._remote_connection(host, addr) 2022-05-13T23:21:56.566 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-13T23:21:56.566 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise 2022-05-13T23:21:56.566 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-13T23:21:56.567 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.gen.throw(type, value, traceback) 2022-05-13T23:21:56.567 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-13T23:21:56.568 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OrchestratorError(msg) 2022-05-13T23:21:56.568 INFO:journalctl@ceph.mon.b.smithi047.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:21:56.569 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:21:56 smithi047 ceph-mon[77147]: pgmap v117: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 138 MiB data, 465 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s; 10439/32427 objects degraded (32.192%) 2022-05-13T23:21:56.569 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:21:56 smithi047 ceph-mon[77147]: 2 slow requests, 0 included below; oldest blocked for > 279.077019 secs 2022-05-13T23:21:56.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:21:56 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:21:56.609 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-13T23:21:56.609 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-13T23:21:56.609 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-13T23:21:56.609 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-13T23:21:56.609 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-13T23:21:56.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-13T23:21:56.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-13T23:21:56.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-13T23:21:56.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-13T23:21:56.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-13T23:21:56.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-13T23:21:56.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-13T23:21:56.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-13T23:21:56.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-13T23:21:56.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-13T23:21:56.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-13T23:21:56.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-13T23:21:56.613 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:21:56.613 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:21:56 smithi115 ceph-mon[78593]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-13T23:21:56.613 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-13T23:21:56.613 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-13T23:21:56.613 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-13T23:21:56.614 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-13T23:21:56.614 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-13T23:21:56.614 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-13T23:21:56.614 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-13T23:21:56.615 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-13T23:21:56.615 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-13T23:21:56.615 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-13T23:21:56.615 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-13T23:21:56.615 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-13T23:21:56.616 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-13T23:21:56.617 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-13T23:21:56.617 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-13T23:21:56.617 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-13T23:21:56.618 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-13T23:21:56.618 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:21:56.618 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-13T23:21:56.618 INFO:journalctl@ceph.mon.c.smithi115.stdout: During handling of the above exception, another exception occurred: 2022-05-13T23:21:56.618 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-13T23:21:56.619 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-13T23:21:56.619 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-13T23:21:56.619 INFO:journalctl@ceph.mon.c.smithi115.stdout: return f(*arg) 2022-05-13T23:21:56.619 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-13T23:21:56.620 INFO:journalctl@ceph.mon.c.smithi115.stdout: self._write_client_files(client_files, host) 2022-05-13T23:21:56.620 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-13T23:21:56.620 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-13T23:21:56.620 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-13T23:21:56.620 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-13T23:21:56.621 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-13T23:21:56.621 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.event_loop.get_result(coro) 2022-05-13T23:21:56.621 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-13T23:21:56.621 INFO:journalctl@ceph.mon.c.smithi115.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-13T23:21:56.622 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-13T23:21:56.622 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.__get_result() 2022-05-13T23:21:56.622 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-13T23:21:56.622 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise self._exception 2022-05-13T23:21:56.622 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-13T23:21:56.623 INFO:journalctl@ceph.mon.c.smithi115.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-13T23:21:56.623 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-13T23:21:56.623 INFO:journalctl@ceph.mon.c.smithi115.stdout: conn = await self._remote_connection(host, addr) 2022-05-13T23:21:56.623 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-13T23:21:56.623 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise 2022-05-13T23:21:56.624 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-13T23:21:56.624 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.gen.throw(type, value, traceback) 2022-05-13T23:21:56.624 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-13T23:21:56.624 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OrchestratorError(msg) 2022-05-13T23:21:56.625 INFO:journalctl@ceph.mon.c.smithi115.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:21:56.625 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:21:56 smithi115 ceph-mon[78593]: pgmap v117: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 138 MiB data, 465 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s; 10439/32427 objects degraded (32.192%) 2022-05-13T23:21:56.625 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:21:56 smithi115 ceph-mon[78593]: 2 slow requests, 0 included below; oldest blocked for > 279.077019 secs 2022-05-13T23:21:58.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:21:58 smithi047 ceph-mon[77147]: pgmap v118: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 138 MiB data, 465 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 3 op/s; 10439/32427 objects degraded (32.192%) 2022-05-13T23:21:58.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:21:58 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:21:58.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:21:58 smithi047 ceph-mon[77147]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2022-05-13T23:21:58.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:21:58 smithi115 ceph-mon[78593]: pgmap v118: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 138 MiB data, 465 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 3 op/s; 10439/32427 objects degraded (32.192%) 2022-05-13T23:21:58.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:21:58 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:21:58.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:21:58 smithi115 ceph-mon[78593]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2022-05-13T23:22:00.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:22:00 smithi047 ceph-mon[77147]: pgmap v119: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 138 MiB data, 465 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s; 10439/32427 objects degraded (32.192%) 2022-05-13T23:22:00.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:22:00 smithi115 ceph-mon[78593]: pgmap v119: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 138 MiB data, 465 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s; 10439/32427 objects degraded (32.192%) 2022-05-13T23:22:01.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:22:01 smithi047 ceph-mon[77147]: 2 slow requests, 0 included below; oldest blocked for > 284.077139 secs 2022-05-13T23:22:01.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:22:01 smithi115 ceph-mon[78593]: 2 slow requests, 0 included below; oldest blocked for > 284.077139 secs 2022-05-13T23:22:02.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:22:02 smithi047 ceph-mon[77147]: pgmap v120: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 138 MiB data, 465 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 3 op/s; 10439/32427 objects degraded (32.192%) 2022-05-13T23:22:02.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:22:02 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:22:02.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:22:02 smithi047 ceph-mon[77147]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2022-05-13T23:22:02.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:22:02 smithi115 ceph-mon[78593]: pgmap v120: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 138 MiB data, 465 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 3 op/s; 10439/32427 objects degraded (32.192%) 2022-05-13T23:22:02.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:22:02 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:22:02.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:22:02 smithi115 ceph-mon[78593]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2022-05-13T23:22:04.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:22:04 smithi047 ceph-mon[77147]: pgmap v121: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 138 MiB data, 465 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 3 op/s; 10439/32427 objects degraded (32.192%) 2022-05-13T23:22:04.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:22:04 smithi115 ceph-mon[78593]: pgmap v121: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 138 MiB data, 465 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 3 op/s; 10439/32427 objects degraded (32.192%) 2022-05-13T23:22:06.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:22:06 smithi047 ceph-mon[77147]: pgmap v122: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 138 MiB data, 465 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s; 10439/32427 objects degraded (32.192%) 2022-05-13T23:22:06.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:22:06 smithi047 ceph-mon[77147]: 2 slow requests, 0 included below; oldest blocked for > 289.077271 secs 2022-05-13T23:22:06.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:22:06 smithi115 ceph-mon[78593]: pgmap v122: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 138 MiB data, 465 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s; 10439/32427 objects degraded (32.192%) 2022-05-13T23:22:06.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:22:06 smithi115 ceph-mon[78593]: 2 slow requests, 0 included below; oldest blocked for > 289.077271 secs 2022-05-13T23:22:08.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:22:08 smithi047 ceph-mon[77147]: pgmap v123: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 138 MiB data, 465 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 3 op/s; 10439/32427 objects degraded (32.192%) 2022-05-13T23:22:08.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:22:08 smithi115 ceph-mon[78593]: pgmap v123: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 138 MiB data, 465 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 3 op/s; 10439/32427 objects degraded (32.192%) 2022-05-13T23:22:10.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:22:10 smithi047 ceph-mon[77147]: pgmap v124: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 138 MiB data, 465 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s; 10439/32427 objects degraded (32.192%) 2022-05-13T23:22:10.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:22:10 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:22:10.545 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:22:10 smithi047 ceph-mon[77147]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2022-05-13T23:22:10.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:22:10 smithi115 ceph-mon[78593]: pgmap v124: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 138 MiB data, 465 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s; 10439/32427 objects degraded (32.192%) 2022-05-13T23:22:10.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:22:10 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:22:10.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:22:10 smithi115 ceph-mon[78593]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2022-05-13T23:22:11.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:22:11 smithi047 ceph-mon[77147]: 2 slow requests, 0 included below; oldest blocked for > 294.077388 secs 2022-05-13T23:22:11.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:22:11 smithi115 ceph-mon[78593]: 2 slow requests, 0 included below; oldest blocked for > 294.077388 secs 2022-05-13T23:22:12.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:22:12 smithi047 ceph-mon[77147]: pgmap v125: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 138 MiB data, 465 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 3 op/s; 10439/32427 objects degraded (32.192%) 2022-05-13T23:22:12.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:22:12 smithi115 ceph-mon[78593]: pgmap v125: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 138 MiB data, 465 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 3 op/s; 10439/32427 objects degraded (32.192%) 2022-05-13T23:22:14.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:22:14 smithi047 ceph-mon[77147]: pgmap v126: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 138 MiB data, 465 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 3 op/s; 10439/32427 objects degraded (32.192%) 2022-05-13T23:22:14.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:22:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-13T23:22:14.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:22:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-13T23:22:14.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:22:14 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:22:14.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:22:14 smithi047 ceph-mon[77147]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2022-05-13T23:22:14.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:22:14 smithi115 ceph-mon[78593]: pgmap v126: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 138 MiB data, 465 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 3 op/s; 10439/32427 objects degraded (32.192%) 2022-05-13T23:22:14.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:22:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-13T23:22:14.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:22:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-13T23:22:14.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:22:14 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:22:14.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:22:14 smithi115 ceph-mon[78593]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2022-05-13T23:22:16.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:22:16 smithi047 ceph-mon[77147]: pgmap v127: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 138 MiB data, 465 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s; 10439/32427 objects degraded (32.192%) 2022-05-13T23:22:16.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:22:16 smithi047 ceph-mon[77147]: 2 slow requests, 0 included below; oldest blocked for > 299.077511 secs 2022-05-13T23:22:16.616 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:22:16 smithi115 ceph-mon[78593]: pgmap v127: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 138 MiB data, 465 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s; 10439/32427 objects degraded (32.192%) 2022-05-13T23:22:16.616 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:22:16 smithi115 ceph-mon[78593]: 2 slow requests, 0 included below; oldest blocked for > 299.077511 secs 2022-05-13T23:22:18.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:22:18 smithi047 ceph-mon[77147]: pgmap v128: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 138 MiB data, 465 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 3 op/s; 10439/32427 objects degraded (32.192%) 2022-05-13T23:22:18.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:22:18 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:22:18.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:22:18 smithi047 ceph-mon[77147]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2022-05-13T23:22:18.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:22:18 smithi115 ceph-mon[78593]: pgmap v128: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 138 MiB data, 465 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 3 op/s; 10439/32427 objects degraded (32.192%) 2022-05-13T23:22:18.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:22:18 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:22:18.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:22:18 smithi115 ceph-mon[78593]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2022-05-13T23:22:20.546 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:22:20 smithi047 ceph-mon[77147]: pgmap v129: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 138 MiB data, 465 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s; 10439/32427 objects degraded (32.192%) 2022-05-13T23:22:20.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:22:20 smithi115 ceph-mon[78593]: pgmap v129: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 138 MiB data, 465 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s; 10439/32427 objects degraded (32.192%) 2022-05-13T23:22:21.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:22:21 smithi047 ceph-mon[77147]: 2 slow requests, 0 included below; oldest blocked for > 304.077629 secs 2022-05-13T23:22:21.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:22:21 smithi047 ceph-mon[77147]: client.25244 isn't responding to mclientcaps(revoke), ino 0x10000002d5a pending pAsLsXsFsx issued pAsLsXsFsxc, sent 303.388609 seconds ago 2022-05-13T23:22:21.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:22:21 smithi047 ceph-mon[77147]: client.25244 isn't responding to mclientcaps(revoke), ino 0x10000002d5c pending pAsLsXsFsx issued pAsLsXsFsxc, sent 303.388579 seconds ago 2022-05-13T23:22:21.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:22:21 smithi047 ceph-mon[77147]: client.25244 isn't responding to mclientcaps(revoke), ino 0x10000002d5e pending pAsLsXsFsx issued pAsLsXsFsxc, sent 303.388560 seconds ago 2022-05-13T23:22:21.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:22:21 smithi047 ceph-mon[77147]: client.25244 isn't responding to mclientcaps(revoke), ino 0x10000002d60 pending pAsLsXsFsx issued pAsLsXsFsxc, sent 303.388539 seconds ago 2022-05-13T23:22:21.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:22:21 smithi047 ceph-mon[77147]: client.25244 isn't responding to mclientcaps(revoke), ino 0x10000002d62 pending pAsLsXsFsx issued pAsLsXsFsxc, sent 303.388511 seconds ago 2022-05-13T23:22:21.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:22:21 smithi047 ceph-mon[77147]: client.25244 isn't responding to mclientcaps(revoke), ino 0x10000002d64 pending pAsLsXsFsx issued pAsLsXsFsxc, sent 303.388489 seconds ago 2022-05-13T23:22:21.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:22:21 smithi047 ceph-mon[77147]: client.25244 isn't responding to mclientcaps(revoke), ino 0x10000002d66 pending pAsLsXsFsx issued pAsLsXsFsxc, sent 303.388470 seconds ago 2022-05-13T23:22:21.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:22:21 smithi047 ceph-mon[77147]: client.25244 isn't responding to mclientcaps(revoke), ino 0x10000002d68 pending pAsLsXsFsx issued pAsLsXsFsxc, sent 303.388445 seconds ago 2022-05-13T23:22:21.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:22:21 smithi047 ceph-mon[77147]: client.25244 isn't responding to mclientcaps(revoke), ino 0x10000002d6d pending pAsLsXsFsx issued pAsLsXsFsxc, sent 303.388402 seconds ago 2022-05-13T23:22:21.545 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:22:21 smithi047 ceph-mon[77147]: client.25244 isn't responding to mclientcaps(revoke), ino 0x10000002d8f pending pAsLsXsFsx issued pAsLsXsFsxc, sent 303.388373 seconds ago 2022-05-13T23:22:21.545 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:22:21 smithi047 ceph-mon[77147]: client.25244 isn't responding to mclientcaps(revoke), ino 0x10000002e8e pending pAsLsXsFsx issued pAsLsXsFsxc, sent 303.388351 seconds ago 2022-05-13T23:22:21.545 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:22:21 smithi047 ceph-mon[77147]: client.25244 isn't responding to mclientcaps(revoke), ino 0x10000002d6b pending pAsLsXsFsx issued pAsLsXsFsxc, sent 303.388321 seconds ago 2022-05-13T23:22:21.545 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:22:21 smithi047 ceph-mon[77147]: client.25244 isn't responding to mclientcaps(revoke), ino 0x10000002d90 pending pAsLsXsFsx issued pAsLsXsFsxc, sent 303.388293 seconds ago 2022-05-13T23:22:21.546 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:22:21 smithi047 ceph-mon[77147]: client.25244 isn't responding to mclientcaps(revoke), ino 0x10000002d6e pending pAsLsXsFsx issued pAsLsXsFsxc, sent 303.388261 seconds ago 2022-05-13T23:22:21.546 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:22:21 smithi047 ceph-mon[77147]: client.25244 isn't responding to mclientcaps(revoke), ino 0x10000002d70 pending pAsLsXsFsx issued pAsLsXsFsxc, sent 303.388231 seconds ago 2022-05-13T23:22:21.546 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:22:21 smithi047 ceph-mon[77147]: client.25244 isn't responding to mclientcaps(revoke), ino 0x10000002d72 pending pAsLsXsFsx issued pAsLsXsFsxc, sent 303.388209 seconds ago 2022-05-13T23:22:21.546 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:22:21 smithi047 ceph-mon[77147]: client.25244 isn't responding to mclientcaps(revoke), ino 0x10000002d74 pending pAsLsXsFsx issued pAsLsXsFsxc, sent 303.388187 seconds ago 2022-05-13T23:22:21.547 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:22:21 smithi047 ceph-mon[77147]: client.25244 isn't responding to mclientcaps(revoke), ino 0x10000002d76 pending pAsLsXsFsx issued pAsLsXsFsxc, sent 303.388167 seconds ago 2022-05-13T23:22:21.547 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:22:21 smithi047 ceph-mon[77147]: client.25244 isn't responding to mclientcaps(revoke), ino 0x10000002d79 pending pAsLsXsFsx issued pAsLsXsFsxc, sent 303.388146 seconds ago 2022-05-13T23:22:21.547 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:22:21 smithi047 ceph-mon[77147]: client.25244 isn't responding to mclientcaps(revoke), ino 0x10000002d7b pending pAsLsXsFsx issued pAsLsXsFsxc, sent 303.388125 seconds ago 2022-05-13T23:22:21.547 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:22:21 smithi047 ceph-mon[77147]: client.25244 isn't responding to mclientcaps(revoke), ino 0x10000002d7d pending pAsLsXsFsx issued pAsLsXsFsxc, sent 303.388095 seconds ago 2022-05-13T23:22:21.547 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:22:21 smithi047 ceph-mon[77147]: client.25244 isn't responding to mclientcaps(revoke), ino 0x10000002d7f pending pAsLsXsFsx issued pAsLsXsFsxc, sent 303.388065 seconds ago 2022-05-13T23:22:21.548 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:22:21 smithi047 ceph-mon[77147]: client.25244 isn't responding to mclientcaps(revoke), ino 0x10000002d81 pending pAsLsXsFsx issued pAsLsXsFsxc, sent 303.388044 seconds ago 2022-05-13T23:22:21.548 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:22:21 smithi047 ceph-mon[77147]: client.25244 isn't responding to mclientcaps(revoke), ino 0x10000002d83 pending pAsLsXsFsx issued pAsLsXsFsxc, sent 303.388022 seconds ago 2022-05-13T23:22:21.548 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:22:21 smithi047 ceph-mon[77147]: client.25244 isn't responding to mclientcaps(revoke), ino 0x10000002d85 pending pAsLsXsFsx issued pAsLsXsFsxc, sent 303.387995 seconds ago 2022-05-13T23:22:21.548 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:22:21 smithi047 ceph-mon[77147]: client.25244 isn't responding to mclientcaps(revoke), ino 0x10000002d87 pending pAsLsXsFsx issued pAsLsXsFsxc, sent 303.387974 seconds ago 2022-05-13T23:22:21.549 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:22:21 smithi047 ceph-mon[77147]: client.25244 isn't responding to mclientcaps(revoke), ino 0x10000002d89 pending pAsLsXsFsx issued pAsLsXsFsxc, sent 303.387952 seconds ago 2022-05-13T23:22:21.549 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:22:21 smithi047 ceph-mon[77147]: client.25244 isn't responding to mclientcaps(revoke), ino 0x10000002d8b pending pAsLsXsFsx issued pAsLsXsFsxc, sent 303.387930 seconds ago 2022-05-13T23:22:21.550 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:22:21 smithi047 ceph-mon[77147]: client.25244 isn't responding to mclientcaps(revoke), ino 0x10000002d8d pending pAsLsXsFsx issued pAsLsXsFsxc, sent 303.387909 seconds ago 2022-05-13T23:22:21.550 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:22:21 smithi047 ceph-mon[77147]: client.25244 isn't responding to mclientcaps(revoke), ino 0x10000002d58 pending pAsLsXsFsx issued pAsLsXsFsxc, sent 303.387878 seconds ago 2022-05-13T23:22:21.550 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:22:21 smithi047 ceph-mon[77147]: client.25244 isn't responding to mclientcaps(revoke), ino 0x10000001cfb pending pAsLsXsFsxcrwb issued pAsxLsXsxFsxcrwb, sent 303.318563 seconds ago 2022-05-13T23:22:21.550 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:22:21 smithi047 ceph-mon[77147]: client.25244 isn't responding to mclientcaps(revoke), ino 0x10000001cfa pending pAsLsXsFsxcrwb issued pAsxLsXsxFsxcrwb, sent 303.252752 seconds ago 2022-05-13T23:22:21.551 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:22:21 smithi047 ceph-mon[77147]: client.25244 isn't responding to mclientcaps(revoke), ino 0x10000002fec pending pAsxLsXsxFcb issued pAsxLsXsxFcwb, sent 302.410197 seconds ago 2022-05-13T23:22:21.551 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:22:21 smithi047 ceph-mon[77147]: client.25244 isn't responding to mclientcaps(revoke), ino 0x10000001eb4 pending pAsLsXsFsxcrwb issued pAsxLsXsxFsxcrwb, sent 301.816832 seconds ago 2022-05-13T23:22:21.551 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:22:21 smithi047 ceph-mon[77147]: client.25244 isn't responding to mclientcaps(revoke), ino 0x10000001cf9 pending pAsLsXsFsxcrwb issued pAsxLsXsxFsxcrwb, sent 301.718125 seconds ago 2022-05-13T23:22:21.551 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:22:21 smithi047 ceph-mon[77147]: client.25244 isn't responding to mclientcaps(revoke), ino 0x100000029e8 pending pAsLsXsFsx issued pAsLsXsFsxc, sent 300.002997 seconds ago 2022-05-13T23:22:21.551 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:22:21 smithi047 ceph-mon[77147]: client.25244 isn't responding to mclientcaps(revoke), ino 0x10000002a06 pending pAsLsXsFsx issued pAsLsXsFsxc, sent 300.002877 seconds ago 2022-05-13T23:22:21.552 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:22:21 smithi047 ceph-mon[77147]: client.25244 isn't responding to mclientcaps(revoke), ino 0x10000002a08 pending pAsLsXsFsx issued pAsLsXsFsxc, sent 300.002718 seconds ago 2022-05-13T23:22:21.552 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:22:21 smithi047 ceph-mon[77147]: client.25244 isn't responding to mclientcaps(revoke), ino 0x10000002ab0 pending pAsLsXsFsx issued pAsLsXsFsxc, sent 300.002584 seconds ago 2022-05-13T23:22:21.552 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:22:21 smithi047 ceph-mon[77147]: client.25244 isn't responding to mclientcaps(revoke), ino 0x10000002c0d pending pAsLsXsFsx issued pAsLsXsFsxc, sent 300.002439 seconds ago 2022-05-13T23:22:21.553 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:22:21 smithi047 ceph-mon[77147]: client.25244 isn't responding to mclientcaps(revoke), ino 0x10000002c15 pending pAsLsXsFsx issued pAsLsXsFsxc, sent 300.002289 seconds ago 2022-05-13T23:22:21.553 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:22:21 smithi047 ceph-mon[77147]: client.25244 isn't responding to mclientcaps(revoke), ino 0x10000002c27 pending pAsLsXsFsx issued pAsLsXsFsxc, sent 300.002145 seconds ago 2022-05-13T23:22:21.553 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:22:21 smithi047 ceph-mon[77147]: client.25244 isn't responding to mclientcaps(revoke), ino 0x10000002e90 pending pAsLsXsFsx issued pAsLsXsFsxc, sent 300.001516 seconds ago 2022-05-13T23:22:21.553 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:22:21 smithi047 ceph-mon[77147]: client.25244 isn't responding to mclientcaps(revoke), ino 0x10000002e94 pending pAsLsXsFsx issued pAsLsXsFsxc, sent 300.001399 seconds ago 2022-05-13T23:22:21.554 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:22:21 smithi047 ceph-mon[77147]: client.25244 isn't responding to mclientcaps(revoke), ino 0x10000002e9e pending pAsLsXsFsx issued pAsLsXsFsxc, sent 300.001298 seconds ago 2022-05-13T23:22:21.554 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:22:21 smithi047 ceph-mon[77147]: client.25244 isn't responding to mclientcaps(revoke), ino 0x10000002ea6 pending pAsLsXsFsx issued pAsLsXsFsxc, sent 300.001199 seconds ago 2022-05-13T23:22:21.554 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:22:21 smithi047 ceph-mon[77147]: client.25244 isn't responding to mclientcaps(revoke), ino 0x10000002eb4 pending pAsLsXsFsx issued pAsLsXsFsxc, sent 300.001018 seconds ago 2022-05-13T23:22:21.554 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:22:21 smithi047 ceph-mon[77147]: client.25244 isn't responding to mclientcaps(revoke), ino 0x10000002ec4 pending pAsLsXsFsx issued pAsLsXsFsxc, sent 300.000919 seconds ago 2022-05-13T23:22:21.554 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:22:21 smithi047 ceph-mon[77147]: client.25244 isn't responding to mclientcaps(revoke), ino 0x10000002fc7 pending pAsLsXsFsx issued pAsLsXsFsxc, sent 300.000821 seconds ago 2022-05-13T23:22:21.555 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:22:21 smithi047 ceph-mon[77147]: client.25244 isn't responding to mclientcaps(revoke), ino 0x10000002fd2 pending pAsLsXsFsx issued pAsLsXsFsxc, sent 300.000712 seconds ago 2022-05-13T23:22:21.555 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:22:21 smithi047 ceph-mon[77147]: client.25244 isn't responding to mclientcaps(revoke), ino 0x10000002fd4 pending pAsLsXsFsx issued pAsLsXsFsxc, sent 300.000609 seconds ago 2022-05-13T23:22:21.555 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:22:21 smithi047 ceph-mon[77147]: client.25244 isn't responding to mclientcaps(revoke), ino 0x10000002fda pending pAsLsXsFsx issued pAsLsXsFsxc, sent 300.000497 seconds ago 2022-05-13T23:22:21.555 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:22:21 smithi047 ceph-mon[77147]: client.25244 isn't responding to mclientcaps(revoke), ino 0x10000002fe0 pending pAsLsXsFsx issued pAsLsXsFsxc, sent 300.000399 seconds ago 2022-05-13T23:22:21.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:22:21 smithi115 ceph-mon[78593]: 2 slow requests, 0 included below; oldest blocked for > 304.077629 secs 2022-05-13T23:22:21.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:22:21 smithi115 ceph-mon[78593]: client.25244 isn't responding to mclientcaps(revoke), ino 0x10000002d5a pending pAsLsXsFsx issued pAsLsXsFsxc, sent 303.388609 seconds ago 2022-05-13T23:22:21.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:22:21 smithi115 ceph-mon[78593]: client.25244 isn't responding to mclientcaps(revoke), ino 0x10000002d5c pending pAsLsXsFsx issued pAsLsXsFsxc, sent 303.388579 seconds ago 2022-05-13T23:22:21.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:22:21 smithi115 ceph-mon[78593]: client.25244 isn't responding to mclientcaps(revoke), ino 0x10000002d5e pending pAsLsXsFsx issued pAsLsXsFsxc, sent 303.388560 seconds ago 2022-05-13T23:22:21.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:22:21 smithi115 ceph-mon[78593]: client.25244 isn't responding to mclientcaps(revoke), ino 0x10000002d60 pending pAsLsXsFsx issued pAsLsXsFsxc, sent 303.388539 seconds ago 2022-05-13T23:22:21.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:22:21 smithi115 ceph-mon[78593]: client.25244 isn't responding to mclientcaps(revoke), ino 0x10000002d62 pending pAsLsXsFsx issued pAsLsXsFsxc, sent 303.388511 seconds ago 2022-05-13T23:22:21.612 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:22:21 smithi115 ceph-mon[78593]: client.25244 isn't responding to mclientcaps(revoke), ino 0x10000002d64 pending pAsLsXsFsx issued pAsLsXsFsxc, sent 303.388489 seconds ago 2022-05-13T23:22:21.612 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:22:21 smithi115 ceph-mon[78593]: client.25244 isn't responding to mclientcaps(revoke), ino 0x10000002d66 pending pAsLsXsFsx issued pAsLsXsFsxc, sent 303.388470 seconds ago 2022-05-13T23:22:21.612 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:22:21 smithi115 ceph-mon[78593]: client.25244 isn't responding to mclientcaps(revoke), ino 0x10000002d68 pending pAsLsXsFsx issued pAsLsXsFsxc, sent 303.388445 seconds ago 2022-05-13T23:22:21.612 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:22:21 smithi115 ceph-mon[78593]: client.25244 isn't responding to mclientcaps(revoke), ino 0x10000002d6d pending pAsLsXsFsx issued pAsLsXsFsxc, sent 303.388402 seconds ago 2022-05-13T23:22:21.612 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:22:21 smithi115 ceph-mon[78593]: client.25244 isn't responding to mclientcaps(revoke), ino 0x10000002d8f pending pAsLsXsFsx issued pAsLsXsFsxc, sent 303.388373 seconds ago 2022-05-13T23:22:21.613 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:22:21 smithi115 ceph-mon[78593]: client.25244 isn't responding to mclientcaps(revoke), ino 0x10000002e8e pending pAsLsXsFsx issued pAsLsXsFsxc, sent 303.388351 seconds ago 2022-05-13T23:22:21.613 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:22:21 smithi115 ceph-mon[78593]: client.25244 isn't responding to mclientcaps(revoke), ino 0x10000002d6b pending pAsLsXsFsx issued pAsLsXsFsxc, sent 303.388321 seconds ago 2022-05-13T23:22:21.613 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:22:21 smithi115 ceph-mon[78593]: client.25244 isn't responding to mclientcaps(revoke), ino 0x10000002d90 pending pAsLsXsFsx issued pAsLsXsFsxc, sent 303.388293 seconds ago 2022-05-13T23:22:21.613 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:22:21 smithi115 ceph-mon[78593]: client.25244 isn't responding to mclientcaps(revoke), ino 0x10000002d6e pending pAsLsXsFsx issued pAsLsXsFsxc, sent 303.388261 seconds ago 2022-05-13T23:22:21.614 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:22:21 smithi115 ceph-mon[78593]: client.25244 isn't responding to mclientcaps(revoke), ino 0x10000002d70 pending pAsLsXsFsx issued pAsLsXsFsxc, sent 303.388231 seconds ago 2022-05-13T23:22:21.614 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:22:21 smithi115 ceph-mon[78593]: client.25244 isn't responding to mclientcaps(revoke), ino 0x10000002d72 pending pAsLsXsFsx issued pAsLsXsFsxc, sent 303.388209 seconds ago 2022-05-13T23:22:21.614 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:22:21 smithi115 ceph-mon[78593]: client.25244 isn't responding to mclientcaps(revoke), ino 0x10000002d74 pending pAsLsXsFsx issued pAsLsXsFsxc, sent 303.388187 seconds ago 2022-05-13T23:22:21.614 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:22:21 smithi115 ceph-mon[78593]: client.25244 isn't responding to mclientcaps(revoke), ino 0x10000002d76 pending pAsLsXsFsx issued pAsLsXsFsxc, sent 303.388167 seconds ago 2022-05-13T23:22:21.615 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:22:21 smithi115 ceph-mon[78593]: client.25244 isn't responding to mclientcaps(revoke), ino 0x10000002d79 pending pAsLsXsFsx issued pAsLsXsFsxc, sent 303.388146 seconds ago 2022-05-13T23:22:21.615 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:22:21 smithi115 ceph-mon[78593]: client.25244 isn't responding to mclientcaps(revoke), ino 0x10000002d7b pending pAsLsXsFsx issued pAsLsXsFsxc, sent 303.388125 seconds ago 2022-05-13T23:22:21.615 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:22:21 smithi115 ceph-mon[78593]: client.25244 isn't responding to mclientcaps(revoke), ino 0x10000002d7d pending pAsLsXsFsx issued pAsLsXsFsxc, sent 303.388095 seconds ago 2022-05-13T23:22:21.616 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:22:21 smithi115 ceph-mon[78593]: client.25244 isn't responding to mclientcaps(revoke), ino 0x10000002d7f pending pAsLsXsFsx issued pAsLsXsFsxc, sent 303.388065 seconds ago 2022-05-13T23:22:21.616 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:22:21 smithi115 ceph-mon[78593]: client.25244 isn't responding to mclientcaps(revoke), ino 0x10000002d81 pending pAsLsXsFsx issued pAsLsXsFsxc, sent 303.388044 seconds ago 2022-05-13T23:22:21.616 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:22:21 smithi115 ceph-mon[78593]: client.25244 isn't responding to mclientcaps(revoke), ino 0x10000002d83 pending pAsLsXsFsx issued pAsLsXsFsxc, sent 303.388022 seconds ago 2022-05-13T23:22:21.617 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:22:21 smithi115 ceph-mon[78593]: client.25244 isn't responding to mclientcaps(revoke), ino 0x10000002d85 pending pAsLsXsFsx issued pAsLsXsFsxc, sent 303.387995 seconds ago 2022-05-13T23:22:21.617 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:22:21 smithi115 ceph-mon[78593]: client.25244 isn't responding to mclientcaps(revoke), ino 0x10000002d87 pending pAsLsXsFsx issued pAsLsXsFsxc, sent 303.387974 seconds ago 2022-05-13T23:22:21.617 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:22:21 smithi115 ceph-mon[78593]: client.25244 isn't responding to mclientcaps(revoke), ino 0x10000002d89 pending pAsLsXsFsx issued pAsLsXsFsxc, sent 303.387952 seconds ago 2022-05-13T23:22:21.617 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:22:21 smithi115 ceph-mon[78593]: client.25244 isn't responding to mclientcaps(revoke), ino 0x10000002d8b pending pAsLsXsFsx issued pAsLsXsFsxc, sent 303.387930 seconds ago 2022-05-13T23:22:21.618 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:22:21 smithi115 ceph-mon[78593]: client.25244 isn't responding to mclientcaps(revoke), ino 0x10000002d8d pending pAsLsXsFsx issued pAsLsXsFsxc, sent 303.387909 seconds ago 2022-05-13T23:22:21.618 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:22:21 smithi115 ceph-mon[78593]: client.25244 isn't responding to mclientcaps(revoke), ino 0x10000002d58 pending pAsLsXsFsx issued pAsLsXsFsxc, sent 303.387878 seconds ago 2022-05-13T23:22:21.618 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:22:21 smithi115 ceph-mon[78593]: client.25244 isn't responding to mclientcaps(revoke), ino 0x10000001cfb pending pAsLsXsFsxcrwb issued pAsxLsXsxFsxcrwb, sent 303.318563 seconds ago 2022-05-13T23:22:21.618 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:22:21 smithi115 ceph-mon[78593]: client.25244 isn't responding to mclientcaps(revoke), ino 0x10000001cfa pending pAsLsXsFsxcrwb issued pAsxLsXsxFsxcrwb, sent 303.252752 seconds ago 2022-05-13T23:22:21.619 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:22:21 smithi115 ceph-mon[78593]: client.25244 isn't responding to mclientcaps(revoke), ino 0x10000002fec pending pAsxLsXsxFcb issued pAsxLsXsxFcwb, sent 302.410197 seconds ago 2022-05-13T23:22:21.619 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:22:21 smithi115 ceph-mon[78593]: client.25244 isn't responding to mclientcaps(revoke), ino 0x10000001eb4 pending pAsLsXsFsxcrwb issued pAsxLsXsxFsxcrwb, sent 301.816832 seconds ago 2022-05-13T23:22:21.620 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:22:21 smithi115 ceph-mon[78593]: client.25244 isn't responding to mclientcaps(revoke), ino 0x10000001cf9 pending pAsLsXsFsxcrwb issued pAsxLsXsxFsxcrwb, sent 301.718125 seconds ago 2022-05-13T23:22:21.620 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:22:21 smithi115 ceph-mon[78593]: client.25244 isn't responding to mclientcaps(revoke), ino 0x100000029e8 pending pAsLsXsFsx issued pAsLsXsFsxc, sent 300.002997 seconds ago 2022-05-13T23:22:21.620 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:22:21 smithi115 ceph-mon[78593]: client.25244 isn't responding to mclientcaps(revoke), ino 0x10000002a06 pending pAsLsXsFsx issued pAsLsXsFsxc, sent 300.002877 seconds ago 2022-05-13T23:22:21.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:22:21 smithi115 ceph-mon[78593]: client.25244 isn't responding to mclientcaps(revoke), ino 0x10000002a08 pending pAsLsXsFsx issued pAsLsXsFsxc, sent 300.002718 seconds ago 2022-05-13T23:22:21.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:22:21 smithi115 ceph-mon[78593]: client.25244 isn't responding to mclientcaps(revoke), ino 0x10000002ab0 pending pAsLsXsFsx issued pAsLsXsFsxc, sent 300.002584 seconds ago 2022-05-13T23:22:21.622 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:22:21 smithi115 ceph-mon[78593]: client.25244 isn't responding to mclientcaps(revoke), ino 0x10000002c0d pending pAsLsXsFsx issued pAsLsXsFsxc, sent 300.002439 seconds ago 2022-05-13T23:22:21.622 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:22:21 smithi115 ceph-mon[78593]: client.25244 isn't responding to mclientcaps(revoke), ino 0x10000002c15 pending pAsLsXsFsx issued pAsLsXsFsxc, sent 300.002289 seconds ago 2022-05-13T23:22:21.622 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:22:21 smithi115 ceph-mon[78593]: client.25244 isn't responding to mclientcaps(revoke), ino 0x10000002c27 pending pAsLsXsFsx issued pAsLsXsFsxc, sent 300.002145 seconds ago 2022-05-13T23:22:21.622 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:22:21 smithi115 ceph-mon[78593]: client.25244 isn't responding to mclientcaps(revoke), ino 0x10000002e90 pending pAsLsXsFsx issued pAsLsXsFsxc, sent 300.001516 seconds ago 2022-05-13T23:22:21.623 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:22:21 smithi115 ceph-mon[78593]: client.25244 isn't responding to mclientcaps(revoke), ino 0x10000002e94 pending pAsLsXsFsx issued pAsLsXsFsxc, sent 300.001399 seconds ago 2022-05-13T23:22:21.623 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:22:21 smithi115 ceph-mon[78593]: client.25244 isn't responding to mclientcaps(revoke), ino 0x10000002e9e pending pAsLsXsFsx issued pAsLsXsFsxc, sent 300.001298 seconds ago 2022-05-13T23:22:21.624 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:22:21 smithi115 ceph-mon[78593]: client.25244 isn't responding to mclientcaps(revoke), ino 0x10000002ea6 pending pAsLsXsFsx issued pAsLsXsFsxc, sent 300.001199 seconds ago 2022-05-13T23:22:21.624 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:22:21 smithi115 ceph-mon[78593]: client.25244 isn't responding to mclientcaps(revoke), ino 0x10000002eb4 pending pAsLsXsFsx issued pAsLsXsFsxc, sent 300.001018 seconds ago 2022-05-13T23:22:21.624 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:22:21 smithi115 ceph-mon[78593]: client.25244 isn't responding to mclientcaps(revoke), ino 0x10000002ec4 pending pAsLsXsFsx issued pAsLsXsFsxc, sent 300.000919 seconds ago 2022-05-13T23:22:21.625 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:22:21 smithi115 ceph-mon[78593]: client.25244 isn't responding to mclientcaps(revoke), ino 0x10000002fc7 pending pAsLsXsFsx issued pAsLsXsFsxc, sent 300.000821 seconds ago 2022-05-13T23:22:21.625 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:22:21 smithi115 ceph-mon[78593]: client.25244 isn't responding to mclientcaps(revoke), ino 0x10000002fd2 pending pAsLsXsFsx issued pAsLsXsFsxc, sent 300.000712 seconds ago 2022-05-13T23:22:21.626 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:22:21 smithi115 ceph-mon[78593]: client.25244 isn't responding to mclientcaps(revoke), ino 0x10000002fd4 pending pAsLsXsFsx issued pAsLsXsFsxc, sent 300.000609 seconds ago 2022-05-13T23:22:21.626 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:22:21 smithi115 ceph-mon[78593]: client.25244 isn't responding to mclientcaps(revoke), ino 0x10000002fda pending pAsLsXsFsx issued pAsLsXsFsxc, sent 300.000497 seconds ago 2022-05-13T23:22:21.627 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:22:21 smithi115 ceph-mon[78593]: client.25244 isn't responding to mclientcaps(revoke), ino 0x10000002fe0 pending pAsLsXsFsx issued pAsLsXsFsxc, sent 300.000399 seconds ago 2022-05-13T23:22:22.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:22:22 smithi047 ceph-mon[77147]: pgmap v130: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 138 MiB data, 465 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 3 op/s; 10439/32427 objects degraded (32.192%) 2022-05-13T23:22:22.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:22:22 smithi047 ceph-mon[77147]: Health check failed: 1 clients failing to respond to capability release (MDS_CLIENT_LATE_RELEASE) 2022-05-13T23:22:22.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:22:22 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:22:22.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:22:22 smithi047 ceph-mon[77147]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2022-05-13T23:22:22.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:22:22 smithi115 ceph-mon[78593]: pgmap v130: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 138 MiB data, 465 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 3 op/s; 10439/32427 objects degraded (32.192%) 2022-05-13T23:22:22.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:22:22 smithi115 ceph-mon[78593]: Health check failed: 1 clients failing to respond to capability release (MDS_CLIENT_LATE_RELEASE) 2022-05-13T23:22:22.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:22:22 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:22:22.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:22:22 smithi115 ceph-mon[78593]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2022-05-13T23:22:24.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:22:24 smithi047 ceph-mon[77147]: pgmap v131: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 138 MiB data, 465 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 3 op/s; 10439/32427 objects degraded (32.192%) 2022-05-13T23:22:24.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:22:24 smithi115 ceph-mon[78593]: pgmap v131: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 138 MiB data, 465 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 3 op/s; 10439/32427 objects degraded (32.192%) 2022-05-13T23:22:26.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:22:26 smithi047 ceph-mon[77147]: pgmap v132: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 138 MiB data, 465 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s; 10439/32427 objects degraded (32.192%) 2022-05-13T23:22:26.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:22:26 smithi047 ceph-mon[77147]: 2 slow requests, 0 included below; oldest blocked for > 309.077748 secs 2022-05-13T23:22:26.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:22:26 smithi047 ceph-mon[77147]: client.25244 isn't responding to mclientcaps(revoke), ino 0x10000002fe5 pending pAsLsXsFsx issued pAsLsXsFsxcr, sent 300.006453 seconds ago 2022-05-13T23:22:26.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:22:26 smithi047 ceph-mon[77147]: client.25244 isn't responding to mclientcaps(revoke), ino 0x10000002ff3 pending pAsLsXsFsx issued pAsLsXsFsxr, sent 300.006184 seconds ago 2022-05-13T23:22:26.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:22:26 smithi047 ceph-mon[77147]: client.25244 isn't responding to mclientcaps(revoke), ino 0x10000002ff5 pending pAsLsXsFsx issued pAsLsXsFsxc, sent 300.005979 seconds ago 2022-05-13T23:22:26.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:22:26 smithi115 ceph-mon[78593]: pgmap v132: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 138 MiB data, 465 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s; 10439/32427 objects degraded (32.192%) 2022-05-13T23:22:26.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:22:26 smithi115 ceph-mon[78593]: 2 slow requests, 0 included below; oldest blocked for > 309.077748 secs 2022-05-13T23:22:26.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:22:26 smithi115 ceph-mon[78593]: client.25244 isn't responding to mclientcaps(revoke), ino 0x10000002fe5 pending pAsLsXsFsx issued pAsLsXsFsxcr, sent 300.006453 seconds ago 2022-05-13T23:22:26.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:22:26 smithi115 ceph-mon[78593]: client.25244 isn't responding to mclientcaps(revoke), ino 0x10000002ff3 pending pAsLsXsFsx issued pAsLsXsFsxr, sent 300.006184 seconds ago 2022-05-13T23:22:26.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:22:26 smithi115 ceph-mon[78593]: client.25244 isn't responding to mclientcaps(revoke), ino 0x10000002ff5 pending pAsLsXsFsx issued pAsLsXsFsxc, sent 300.005979 seconds ago 2022-05-13T23:22:28.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:22:28 smithi047 ceph-mon[77147]: pgmap v133: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 138 MiB data, 465 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 3 op/s; 10439/32427 objects degraded (32.192%) 2022-05-13T23:22:28.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:22:28 smithi115 ceph-mon[78593]: pgmap v133: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 138 MiB data, 465 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 3 op/s; 10439/32427 objects degraded (32.192%) 2022-05-13T23:22:30.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:22:30 smithi047 ceph-mon[77147]: pgmap v134: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 138 MiB data, 465 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s; 10439/32427 objects degraded (32.192%) 2022-05-13T23:22:30.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:22:30 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:22:30.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:22:30 smithi047 ceph-mon[77147]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2022-05-13T23:22:30.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:22:30 smithi115 ceph-mon[78593]: pgmap v134: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 138 MiB data, 465 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s; 10439/32427 objects degraded (32.192%) 2022-05-13T23:22:30.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:22:30 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:22:30.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:22:30 smithi115 ceph-mon[78593]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2022-05-13T23:22:31.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:22:31 smithi047 ceph-mon[77147]: 2 slow requests, 0 included below; oldest blocked for > 314.077876 secs 2022-05-13T23:22:31.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:22:31 smithi115 ceph-mon[78593]: 2 slow requests, 0 included below; oldest blocked for > 314.077876 secs 2022-05-13T23:22:32.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:22:32 smithi047 ceph-mon[77147]: pgmap v135: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 138 MiB data, 465 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 3 op/s; 10439/32427 objects degraded (32.192%) 2022-05-13T23:22:32.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:22:32 smithi115 ceph-mon[78593]: pgmap v135: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 138 MiB data, 465 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 3 op/s; 10439/32427 objects degraded (32.192%) 2022-05-13T23:22:34.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:22:34 smithi047 ceph-mon[77147]: pgmap v136: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 138 MiB data, 465 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 3 op/s; 10439/32427 objects degraded (32.192%) 2022-05-13T23:22:34.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:22:34 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:22:34.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:22:34 smithi047 ceph-mon[77147]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2022-05-13T23:22:34.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:22:34 smithi115 ceph-mon[78593]: pgmap v136: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 138 MiB data, 465 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 3 op/s; 10439/32427 objects degraded (32.192%) 2022-05-13T23:22:34.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:22:34 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:22:34.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:22:34 smithi115 ceph-mon[78593]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2022-05-13T23:22:36.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:22:36 smithi047 ceph-mon[77147]: pgmap v137: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 138 MiB data, 465 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s; 10439/32427 objects degraded (32.192%) 2022-05-13T23:22:36.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:22:36 smithi047 ceph-mon[77147]: 2 slow requests, 0 included below; oldest blocked for > 319.078003 secs 2022-05-13T23:22:36.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:22:36 smithi115 ceph-mon[78593]: pgmap v137: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 138 MiB data, 465 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s; 10439/32427 objects degraded (32.192%) 2022-05-13T23:22:36.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:22:36 smithi115 ceph-mon[78593]: 2 slow requests, 0 included below; oldest blocked for > 319.078003 secs 2022-05-13T23:22:38.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:22:38 smithi047 ceph-mon[77147]: pgmap v138: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 138 MiB data, 465 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 3 op/s; 10439/32427 objects degraded (32.192%) 2022-05-13T23:22:38.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:22:38 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:22:38.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:22:38 smithi047 ceph-mon[77147]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2022-05-13T23:22:38.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:22:38 smithi115 ceph-mon[78593]: pgmap v138: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 138 MiB data, 465 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 3 op/s; 10439/32427 objects degraded (32.192%) 2022-05-13T23:22:38.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:22:38 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:22:38.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:22:38 smithi115 ceph-mon[78593]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2022-05-13T23:22:40.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:22:40 smithi047 ceph-mon[77147]: pgmap v139: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 138 MiB data, 465 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s; 10439/32427 objects degraded (32.192%) 2022-05-13T23:22:40.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:22:40 smithi115 ceph-mon[78593]: pgmap v139: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 138 MiB data, 465 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s; 10439/32427 objects degraded (32.192%) 2022-05-13T23:22:41.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:22:41 smithi047 ceph-mon[77147]: 2 slow requests, 0 included below; oldest blocked for > 324.078121 secs 2022-05-13T23:22:41.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:22:41 smithi047 ceph-mon[77147]: evicting unresponsive client smithi002:0 (25244), after 394.366 seconds 2022-05-13T23:22:41.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:22:41 smithi047 ceph-mon[77147]: Evicting (and blocklisting) client session 25244 (172.21.15.2:0/2590900211) 2022-05-13T23:22:41.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:22:41 smithi047 ceph-mon[77147]: from='mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923]' entity='mds.l' cmd=[{"prefix":"osd blocklist", "blocklistop":"add","addr":"172.21.15.2:0/2590900211"}]: dispatch 2022-05-13T23:22:41.545 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:22:41 smithi047 ceph-mon[77147]: client.25244 isn't responding to mclientcaps(revoke), ino 0x1000000136e pending pAsLsXs issued pAsLsXsFsx, sent 300.004687 seconds ago 2022-05-13T23:22:41.545 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:22:41 smithi047 ceph-mon[77147]: from='mds.? ' entity='mds.l' cmd=[{"prefix":"osd blocklist", "blocklistop":"add","addr":"172.21.15.2:0/2590900211"}]: dispatch 2022-05-13T23:22:41.545 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:22:41 smithi047 ceph-mon[77147]: evicting unresponsive client (25244), after 332.807 seconds 2022-05-13T23:22:41.545 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:22:41 smithi047 ceph-mon[77147]: Evicting (and blocklisting) client session 25244 (172.21.15.2:0/2590900211) 2022-05-13T23:22:41.546 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:22:41 smithi047 ceph-mon[77147]: from='mds.? [v2:172.21.15.115:6834/71062276,v1:172.21.15.115:6835/71062276]' entity='mds.f' cmd=[{"prefix":"osd blocklist", "blocklistop":"add","addr":"172.21.15.2:0/2590900211"}]: dispatch 2022-05-13T23:22:41.546 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:22:41 smithi047 ceph-mon[77147]: from='mds.? ' entity='mds.f' cmd=[{"prefix":"osd blocklist", "blocklistop":"add","addr":"172.21.15.2:0/2590900211"}]: dispatch 2022-05-13T23:22:41.546 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:22:41 smithi047 ceph-mon[77147]: evicting unresponsive client (25244), after 325.313 seconds 2022-05-13T23:22:41.546 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:22:41 smithi047 ceph-mon[77147]: Evicting (and blocklisting) client session 25244 (172.21.15.2:0/2590900211) 2022-05-13T23:22:41.547 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:22:41 smithi047 ceph-mon[77147]: from='mds.4 [v2:172.21.15.47:6838/761355251,v1:172.21.15.47:6839/761355251]' entity='mds.k' cmd=[{"prefix":"osd blocklist", "blocklistop":"add","addr":"172.21.15.2:0/2590900211"}]: dispatch 2022-05-13T23:22:41.547 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:22:41 smithi047 ceph-mon[77147]: from='mds.4 ' entity='mds.k' cmd=[{"prefix":"osd blocklist", "blocklistop":"add","addr":"172.21.15.2:0/2590900211"}]: dispatch 2022-05-13T23:22:41.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:22:41 smithi115 ceph-mon[78593]: 2 slow requests, 0 included below; oldest blocked for > 324.078121 secs 2022-05-13T23:22:41.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:22:41 smithi115 ceph-mon[78593]: evicting unresponsive client smithi002:0 (25244), after 394.366 seconds 2022-05-13T23:22:41.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:22:41 smithi115 ceph-mon[78593]: Evicting (and blocklisting) client session 25244 (172.21.15.2:0/2590900211) 2022-05-13T23:22:41.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:22:41 smithi115 ceph-mon[78593]: from='mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923]' entity='mds.l' cmd=[{"prefix":"osd blocklist", "blocklistop":"add","addr":"172.21.15.2:0/2590900211"}]: dispatch 2022-05-13T23:22:41.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:22:41 smithi115 ceph-mon[78593]: client.25244 isn't responding to mclientcaps(revoke), ino 0x1000000136e pending pAsLsXs issued pAsLsXsFsx, sent 300.004687 seconds ago 2022-05-13T23:22:41.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:22:41 smithi115 ceph-mon[78593]: from='mds.? ' entity='mds.l' cmd=[{"prefix":"osd blocklist", "blocklistop":"add","addr":"172.21.15.2:0/2590900211"}]: dispatch 2022-05-13T23:22:41.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:22:41 smithi115 ceph-mon[78593]: evicting unresponsive client (25244), after 332.807 seconds 2022-05-13T23:22:41.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:22:41 smithi115 ceph-mon[78593]: Evicting (and blocklisting) client session 25244 (172.21.15.2:0/2590900211) 2022-05-13T23:22:41.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:22:41 smithi115 ceph-mon[78593]: from='mds.? [v2:172.21.15.115:6834/71062276,v1:172.21.15.115:6835/71062276]' entity='mds.f' cmd=[{"prefix":"osd blocklist", "blocklistop":"add","addr":"172.21.15.2:0/2590900211"}]: dispatch 2022-05-13T23:22:41.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:22:41 smithi115 ceph-mon[78593]: from='mds.? ' entity='mds.f' cmd=[{"prefix":"osd blocklist", "blocklistop":"add","addr":"172.21.15.2:0/2590900211"}]: dispatch 2022-05-13T23:22:41.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:22:41 smithi115 ceph-mon[78593]: evicting unresponsive client (25244), after 325.313 seconds 2022-05-13T23:22:41.612 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:22:41 smithi115 ceph-mon[78593]: Evicting (and blocklisting) client session 25244 (172.21.15.2:0/2590900211) 2022-05-13T23:22:41.612 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:22:41 smithi115 ceph-mon[78593]: from='mds.4 [v2:172.21.15.47:6838/761355251,v1:172.21.15.47:6839/761355251]' entity='mds.k' cmd=[{"prefix":"osd blocklist", "blocklistop":"add","addr":"172.21.15.2:0/2590900211"}]: dispatch 2022-05-13T23:22:41.612 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:22:41 smithi115 ceph-mon[78593]: from='mds.4 ' entity='mds.k' cmd=[{"prefix":"osd blocklist", "blocklistop":"add","addr":"172.21.15.2:0/2590900211"}]: dispatch 2022-05-13T23:22:42.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:22:42 smithi047 ceph-mon[77147]: pgmap v140: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 138 MiB data, 465 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 3 op/s; 10439/32427 objects degraded (32.192%) 2022-05-13T23:22:42.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:22:42 smithi047 ceph-mon[77147]: from='mds.? ' entity='mds.l' cmd='[{"prefix":"osd blocklist", "blocklistop":"add","addr":"172.21.15.2:0/2590900211"}]': finished 2022-05-13T23:22:42.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:22:42 smithi047 ceph-mon[77147]: from='mds.? ' entity='mds.f' cmd='[{"prefix":"osd blocklist", "blocklistop":"add","addr":"172.21.15.2:0/2590900211"}]': finished 2022-05-13T23:22:42.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:22:42 smithi047 ceph-mon[77147]: from='mds.4 ' entity='mds.k' cmd='[{"prefix":"osd blocklist", "blocklistop":"add","addr":"172.21.15.2:0/2590900211"}]': finished 2022-05-13T23:22:42.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:22:42 smithi047 ceph-mon[77147]: osdmap e92: 12 total, 8 up, 12 in 2022-05-13T23:22:42.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:22:42 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:22:42.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:22:42 smithi047 ceph-mon[77147]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2022-05-13T23:22:42.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:22:42 smithi115 ceph-mon[78593]: pgmap v140: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 138 MiB data, 465 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 3 op/s; 10439/32427 objects degraded (32.192%) 2022-05-13T23:22:42.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:22:42 smithi115 ceph-mon[78593]: from='mds.? ' entity='mds.l' cmd='[{"prefix":"osd blocklist", "blocklistop":"add","addr":"172.21.15.2:0/2590900211"}]': finished 2022-05-13T23:22:42.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:22:42 smithi115 ceph-mon[78593]: from='mds.? ' entity='mds.f' cmd='[{"prefix":"osd blocklist", "blocklistop":"add","addr":"172.21.15.2:0/2590900211"}]': finished 2022-05-13T23:22:42.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:22:42 smithi115 ceph-mon[78593]: from='mds.4 ' entity='mds.k' cmd='[{"prefix":"osd blocklist", "blocklistop":"add","addr":"172.21.15.2:0/2590900211"}]': finished 2022-05-13T23:22:42.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:22:42 smithi115 ceph-mon[78593]: osdmap e92: 12 total, 8 up, 12 in 2022-05-13T23:22:42.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:22:42 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:22:42.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:22:42 smithi115 ceph-mon[78593]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2022-05-13T23:22:44.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:22:44 smithi047 ceph-mon[77147]: pgmap v142: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 138 MiB data, 465 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 3 op/s; 10439/32427 objects degraded (32.192%) 2022-05-13T23:22:44.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:22:44 smithi115 ceph-mon[78593]: pgmap v142: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 138 MiB data, 465 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 3 op/s; 10439/32427 objects degraded (32.192%) 2022-05-13T23:22:46.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:22:46 smithi047 ceph-mon[77147]: pgmap v143: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 139 MiB data, 468 MiB used, 715 GiB / 715 GiB avail; 54 KiB/s rd, 95 KiB/s wr, 45 op/s; 10439/32427 objects degraded (32.192%) 2022-05-13T23:22:46.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:22:46 smithi047 ceph-mon[77147]: 2 slow requests, 0 included below; oldest blocked for > 329.078238 secs 2022-05-13T23:22:46.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:22:46 smithi115 ceph-mon[78593]: pgmap v143: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 139 MiB data, 468 MiB used, 715 GiB / 715 GiB avail; 54 KiB/s rd, 95 KiB/s wr, 45 op/s; 10439/32427 objects degraded (32.192%) 2022-05-13T23:22:46.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:22:46 smithi115 ceph-mon[78593]: 2 slow requests, 0 included below; oldest blocked for > 329.078238 secs 2022-05-13T23:22:47.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:22:47 smithi047 ceph-mon[77147]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2022-05-13T23:22:47.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:22:47 smithi115 ceph-mon[78593]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2022-05-13T23:22:48.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:22:48 smithi047 ceph-mon[77147]: pgmap v144: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 139 MiB data, 468 MiB used, 715 GiB / 715 GiB avail; 54 KiB/s rd, 95 KiB/s wr, 45 op/s; 10439/32427 objects degraded (32.192%) 2022-05-13T23:22:48.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:22:48 smithi115 ceph-mon[78593]: pgmap v144: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 139 MiB data, 468 MiB used, 715 GiB / 715 GiB avail; 54 KiB/s rd, 95 KiB/s wr, 45 op/s; 10439/32427 objects degraded (32.192%) 2022-05-13T23:22:49.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:22:49 smithi047 ceph-mon[77147]: MDS health message cleared (mds.?): Client smithi002:0 failing to respond to capability release 2022-05-13T23:22:49.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:22:49 smithi115 ceph-mon[78593]: MDS health message cleared (mds.?): Client smithi002:0 failing to respond to capability release 2022-05-13T23:22:50.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:22:50 smithi047 ceph-mon[77147]: pgmap v145: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 139 MiB data, 472 MiB used, 715 GiB / 715 GiB avail; 56 KiB/s rd, 187 KiB/s wr, 51 op/s; 10440/32430 objects degraded (32.192%) 2022-05-13T23:22:50.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:22:50 smithi047 ceph-mon[77147]: Health check cleared: MDS_CLIENT_LATE_RELEASE (was: 1 clients failing to respond to capability release) 2022-05-13T23:22:50.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:22:50 smithi047 ceph-mon[77147]: Health check update: Degraded data redundancy: 10440/32430 objects degraded (32.192%), 67 pgs degraded, 68 pgs undersized (PG_DEGRADED) 2022-05-13T23:22:50.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:22:50 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:22:50.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:22:50 smithi047 ceph-mon[77147]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2022-05-13T23:22:50.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:22:50 smithi115 ceph-mon[78593]: pgmap v145: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 139 MiB data, 472 MiB used, 715 GiB / 715 GiB avail; 56 KiB/s rd, 187 KiB/s wr, 51 op/s; 10440/32430 objects degraded (32.192%) 2022-05-13T23:22:50.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:22:50 smithi115 ceph-mon[78593]: Health check cleared: MDS_CLIENT_LATE_RELEASE (was: 1 clients failing to respond to capability release) 2022-05-13T23:22:50.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:22:50 smithi115 ceph-mon[78593]: Health check update: Degraded data redundancy: 10440/32430 objects degraded (32.192%), 67 pgs degraded, 68 pgs undersized (PG_DEGRADED) 2022-05-13T23:22:50.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:22:50 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:22:50.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:22:50 smithi115 ceph-mon[78593]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2022-05-13T23:22:51.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:22:51 smithi047 ceph-mon[77147]: 1 slow requests, 0 included below; oldest blocked for > 334.078359 secs 2022-05-13T23:22:51.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:22:51 smithi115 ceph-mon[78593]: 1 slow requests, 0 included below; oldest blocked for > 334.078359 secs 2022-05-13T23:22:52.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:22:52 smithi047 ceph-mon[77147]: pgmap v146: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 139 MiB data, 472 MiB used, 715 GiB / 715 GiB avail; 56 KiB/s rd, 187 KiB/s wr, 51 op/s; 10440/32430 objects degraded (32.192%) 2022-05-13T23:22:52.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:22:52 smithi115 ceph-mon[78593]: pgmap v146: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 139 MiB data, 472 MiB used, 715 GiB / 715 GiB avail; 56 KiB/s rd, 187 KiB/s wr, 51 op/s; 10440/32430 objects degraded (32.192%) 2022-05-13T23:22:54.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:22:54 smithi047 ceph-mon[77147]: pgmap v147: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 139 MiB data, 472 MiB used, 715 GiB / 715 GiB avail; 47 KiB/s rd, 157 KiB/s wr, 43 op/s; 10440/32430 objects degraded (32.192%) 2022-05-13T23:22:54.545 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:22:54 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:22:54.545 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:22:54 smithi047 ceph-mon[77147]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2022-05-13T23:22:54.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:22:54 smithi115 ceph-mon[78593]: pgmap v147: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 139 MiB data, 472 MiB used, 715 GiB / 715 GiB avail; 47 KiB/s rd, 157 KiB/s wr, 43 op/s; 10440/32430 objects degraded (32.192%) 2022-05-13T23:22:54.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:22:54 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:22:54.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:22:54 smithi115 ceph-mon[78593]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2022-05-13T23:22:55.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:22:55 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-13T23:22:55.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:22:55 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-13T23:22:55.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:22:55 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-13T23:22:55.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:22:55 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-13T23:22:55.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:22:55 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-13T23:22:55.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:22:55 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-13T23:22:56.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:22:56 smithi047 ceph-mon[77147]: pgmap v148: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 139 MiB data, 472 MiB used, 715 GiB / 715 GiB avail; 50 KiB/s rd, 159 KiB/s wr, 46 op/s; 10440/32430 objects degraded (32.192%) 2022-05-13T23:22:56.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:22:56 smithi047 ceph-mon[77147]: 1 slow requests, 0 included below; oldest blocked for > 339.078429 secs 2022-05-13T23:22:56.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:22:56 smithi115 ceph-mon[78593]: pgmap v148: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 139 MiB data, 472 MiB used, 715 GiB / 715 GiB avail; 50 KiB/s rd, 159 KiB/s wr, 46 op/s; 10440/32430 objects degraded (32.192%) 2022-05-13T23:22:56.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:22:56 smithi115 ceph-mon[78593]: 1 slow requests, 0 included below; oldest blocked for > 339.078429 secs 2022-05-13T23:22:57.042 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:22:56 smithi047 conmon[78521]: 2022-05-13T23:22:56.560+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:22:57.042 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:22:56 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-13T23:22:57.042 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:22:56 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-13T23:22:57.042 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:22:56 smithi047 conmon[78521]: yield 2022-05-13T23:22:57.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:22:56 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-13T23:22:57.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:22:56 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-13T23:22:57.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:22:56 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-13T23:22:57.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:22:56 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-13T23:22:57.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:22:56 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-13T23:22:57.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:22:56 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-13T23:22:57.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:22:56 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-13T23:22:57.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:22:56 smithi047 conmon[78521]: raise exceptions[0] 2022-05-13T23:22:57.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:22:56 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-13T23:22:57.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:22:56 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-13T23:22:57.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:22:56 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-13T23:22:57.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:22:56 smithi047 conmon[78521]: return (yield from fut) 2022-05-13T23:22:57.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:22:56 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-13T23:22:57.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:22:56 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-13T23:22:57.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:22:56 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:22:57.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:22:57 smithi047 ceph-mon[77147]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2022-05-13T23:22:57.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:22:57 smithi115 ceph-mon[78593]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2022-05-13T23:22:58.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:22:58 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:22:58.542 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-13T23:22:58.542 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-13T23:22:58.542 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-13T23:22:58.542 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-13T23:22:58.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-13T23:22:58.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-13T23:22:58.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-13T23:22:58.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-13T23:22:58.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-13T23:22:58.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-13T23:22:58.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-13T23:22:58.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-13T23:22:58.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-13T23:22:58.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-13T23:22:58.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-13T23:22:58.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-13T23:22:58.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-13T23:22:58.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:22:58.546 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:22:58 smithi047 ceph-mon[77147]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-13T23:22:58.546 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:22:58 smithi047 ceph-mon[77147]: pgmap v149: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 139 MiB data, 472 MiB used, 715 GiB / 715 GiB avail; 6.1 KiB/s rd, 80 KiB/s wr, 9 op/s; 10440/32430 objects degraded (32.192%) 2022-05-13T23:22:58.546 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:22:58 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:22:58.546 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:22:58 smithi047 ceph-mon[77147]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2022-05-13T23:22:58.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:22:58 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:22:58.608 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-13T23:22:58.609 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-13T23:22:58.609 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-13T23:22:58.609 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-13T23:22:58.609 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-13T23:22:58.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-13T23:22:58.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-13T23:22:58.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-13T23:22:58.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-13T23:22:58.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-13T23:22:58.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-13T23:22:58.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-13T23:22:58.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-13T23:22:58.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-13T23:22:58.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-13T23:22:58.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-13T23:22:58.613 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-13T23:22:58.613 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:22:58.613 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:22:58 smithi115 ceph-mon[78593]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-13T23:22:58.613 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:22:58 smithi115 ceph-mon[78593]: pgmap v149: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 139 MiB data, 472 MiB used, 715 GiB / 715 GiB avail; 6.1 KiB/s rd, 80 KiB/s wr, 9 op/s; 10440/32430 objects degraded (32.192%) 2022-05-13T23:22:58.614 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:22:58 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:22:58.614 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:22:58 smithi115 ceph-mon[78593]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2022-05-13T23:23:00.042 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:22:59 smithi047 conmon[78521]: 2022-05-13T23:22:59.631+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:23:00.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:22:59 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-13T23:23:00.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:22:59 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-13T23:23:00.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:22:59 smithi047 conmon[78521]: yield 2022-05-13T23:23:00.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:22:59 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-13T23:23:00.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:22:59 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-13T23:23:00.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:22:59 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-13T23:23:00.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:22:59 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-13T23:23:00.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:22:59 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-13T23:23:00.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:22:59 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-13T23:23:00.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:22:59 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-13T23:23:00.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:22:59 smithi047 conmon[78521]: raise exceptions[0] 2022-05-13T23:23:00.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:22:59 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-13T23:23:00.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:22:59 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-13T23:23:00.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:22:59 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-13T23:23:00.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:22:59 smithi047 conmon[78521]: return (yield from fut) 2022-05-13T23:23:00.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:22:59 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-13T23:23:00.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:22:59 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-13T23:23:00.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:22:59 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:23:00.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:22:59 smithi047 conmon[78521]: 2022-05-13T23:22:59.632+0000 7f5340f7a700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-13T23:23:00.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:22:59 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-13T23:23:00.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:22:59 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-13T23:23:00.048 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:22:59 smithi047 conmon[78521]: yield 2022-05-13T23:23:00.048 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:22:59 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-13T23:23:00.048 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:22:59 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-13T23:23:00.048 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:22:59 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-13T23:23:00.048 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:22:59 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-13T23:23:00.049 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:22:59 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-13T23:23:00.049 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:22:59 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-13T23:23:00.049 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:22:59 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-13T23:23:00.049 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:22:59 smithi047 conmon[78521]: raise exceptions[0] 2022-05-13T23:23:00.050 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:22:59 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-13T23:23:00.050 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:22:59 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-13T23:23:00.051 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:22:59 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-13T23:23:00.051 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:22:59 smithi047 conmon[78521]: return (yield from fut) 2022-05-13T23:23:00.052 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:22:59 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-13T23:23:00.052 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:22:59 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-13T23:23:00.052 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:22:59 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:23:00.052 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:22:59 smithi047 conmon[78521]: 2022-05-13T23:23:00.053 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:22:59 smithi047 conmon[78521]: During handling of the above exception, another exception occurred: 2022-05-13T23:23:00.053 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:22:59 smithi047 conmon[78521]: 2022-05-13T23:23:00.053 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:22:59 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-13T23:23:00.054 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:22:59 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-13T23:23:00.054 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:22:59 smithi047 conmon[78521]: return f(*arg) 2022-05-13T23:23:00.054 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:22:59 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-13T23:23:00.054 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:22:59 smithi047 conmon[78521]: self._write_client_files(client_files, host) 2022-05-13T23:23:00.055 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:22:59 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-13T23:23:00.055 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:22:59 smithi047 conmon[78521]: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-13T23:23:00.055 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:22:59 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-13T23:23:00.055 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:22:59 smithi047 conmon[78521]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-13T23:23:00.056 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:22:59 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-13T23:23:00.056 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:22:59 smithi047 conmon[78521]: return self.event_loop.get_result(coro) 2022-05-13T23:23:00.056 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:22:59 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-13T23:23:00.056 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:22:59 smithi047 conmon[78521]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-13T23:23:00.057 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:22:59 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-13T23:23:00.057 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:22:59 smithi047 conmon[78521]: return self.__get_result() 2022-05-13T23:23:00.057 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:22:59 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-13T23:23:00.057 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:22:59 smithi047 conmon[78521]: raise self._exception 2022-05-13T23:23:00.058 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:22:59 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-13T23:23:00.058 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:22:59 smithi047 conmon[78521]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-13T23:23:00.058 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:22:59 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-13T23:23:00.058 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:22:59 smithi047 conmon[78521]: conn = await self._remote_connection(host, addr) 2022-05-13T23:23:00.059 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:22:59 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-13T23:23:00.059 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:22:59 smithi047 conmon[78521]: raise 2022-05-13T23:23:00.059 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:22:59 smithi047 conmon[78521]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-13T23:23:00.059 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:22:59 smithi047 conmon[78521]: self.gen.throw(type, value, traceback) 2022-05-13T23:23:00.060 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:22:59 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-13T23:23:00.060 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:22:59 smithi047 conmon[78521]: raise OrchestratorError(msg) 2022-05-13T23:23:00.060 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:22:59 smithi047 conmon[78521]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:23:00.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:23:00 smithi047 ceph-mon[77147]: pgmap v150: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 139 MiB data, 472 MiB used, 715 GiB / 715 GiB avail; 7.5 KiB/s rd, 82 KiB/s wr, 12 op/s; 10440/32430 objects degraded (32.192%) 2022-05-13T23:23:00.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:23:00 smithi115 ceph-mon[78593]: pgmap v150: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 139 MiB data, 472 MiB used, 715 GiB / 715 GiB avail; 7.5 KiB/s rd, 82 KiB/s wr, 12 op/s; 10440/32430 objects degraded (32.192%) 2022-05-13T23:23:01.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:23:01 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:23:01.542 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-13T23:23:01.542 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-13T23:23:01.542 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-13T23:23:01.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-13T23:23:01.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-13T23:23:01.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-13T23:23:01.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-13T23:23:01.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-13T23:23:01.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-13T23:23:01.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-13T23:23:01.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-13T23:23:01.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-13T23:23:01.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-13T23:23:01.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-13T23:23:01.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-13T23:23:01.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-13T23:23:01.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-13T23:23:01.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:23:01.546 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:23:01 smithi047 ceph-mon[77147]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-13T23:23:01.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-13T23:23:01.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-13T23:23:01.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-13T23:23:01.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-13T23:23:01.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-13T23:23:01.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-13T23:23:01.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-13T23:23:01.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-13T23:23:01.548 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-13T23:23:01.548 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-13T23:23:01.549 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-13T23:23:01.549 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-13T23:23:01.549 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-13T23:23:01.549 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-13T23:23:01.550 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-13T23:23:01.550 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-13T23:23:01.550 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-13T23:23:01.550 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:23:01.550 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-13T23:23:01.551 INFO:journalctl@ceph.mon.b.smithi047.stdout: During handling of the above exception, another exception occurred: 2022-05-13T23:23:01.551 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-13T23:23:01.551 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-13T23:23:01.551 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-13T23:23:01.551 INFO:journalctl@ceph.mon.b.smithi047.stdout: return f(*arg) 2022-05-13T23:23:01.552 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-13T23:23:01.552 INFO:journalctl@ceph.mon.b.smithi047.stdout: self._write_client_files(client_files, host) 2022-05-13T23:23:01.552 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-13T23:23:01.552 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-13T23:23:01.552 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-13T23:23:01.553 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-13T23:23:01.553 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-13T23:23:01.553 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.event_loop.get_result(coro) 2022-05-13T23:23:01.553 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-13T23:23:01.554 INFO:journalctl@ceph.mon.b.smithi047.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-13T23:23:01.554 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-13T23:23:01.554 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.__get_result() 2022-05-13T23:23:01.554 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-13T23:23:01.554 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise self._exception 2022-05-13T23:23:01.555 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-13T23:23:01.555 INFO:journalctl@ceph.mon.b.smithi047.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-13T23:23:01.555 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-13T23:23:01.555 INFO:journalctl@ceph.mon.b.smithi047.stdout: conn = await self._remote_connection(host, addr) 2022-05-13T23:23:01.555 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-13T23:23:01.556 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise 2022-05-13T23:23:01.556 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-13T23:23:01.556 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.gen.throw(type, value, traceback) 2022-05-13T23:23:01.556 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-13T23:23:01.556 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OrchestratorError(msg) 2022-05-13T23:23:01.557 INFO:journalctl@ceph.mon.b.smithi047.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:23:01.557 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:23:01 smithi047 ceph-mon[77147]: 1 slow requests, 0 included below; oldest blocked for > 344.078547 secs 2022-05-13T23:23:01.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:23:01 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:23:01.608 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-13T23:23:01.609 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-13T23:23:01.609 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-13T23:23:01.609 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-13T23:23:01.609 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-13T23:23:01.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-13T23:23:01.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-13T23:23:01.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-13T23:23:01.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-13T23:23:01.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-13T23:23:01.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-13T23:23:01.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-13T23:23:01.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-13T23:23:01.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-13T23:23:01.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-13T23:23:01.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-13T23:23:01.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-13T23:23:01.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:23:01.612 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:23:01 smithi115 ceph-mon[78593]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-13T23:23:01.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-13T23:23:01.613 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-13T23:23:01.613 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-13T23:23:01.613 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-13T23:23:01.613 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-13T23:23:01.614 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-13T23:23:01.614 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-13T23:23:01.614 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-13T23:23:01.614 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-13T23:23:01.614 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-13T23:23:01.615 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-13T23:23:01.615 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-13T23:23:01.615 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-13T23:23:01.615 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-13T23:23:01.615 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-13T23:23:01.616 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-13T23:23:01.616 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-13T23:23:01.616 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:23:01.616 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-13T23:23:01.616 INFO:journalctl@ceph.mon.c.smithi115.stdout: During handling of the above exception, another exception occurred: 2022-05-13T23:23:01.617 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-13T23:23:01.617 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-13T23:23:01.617 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-13T23:23:01.617 INFO:journalctl@ceph.mon.c.smithi115.stdout: return f(*arg) 2022-05-13T23:23:01.618 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-13T23:23:01.618 INFO:journalctl@ceph.mon.c.smithi115.stdout: self._write_client_files(client_files, host) 2022-05-13T23:23:01.618 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-13T23:23:01.618 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-13T23:23:01.618 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-13T23:23:01.619 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-13T23:23:01.619 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-13T23:23:01.619 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.event_loop.get_result(coro) 2022-05-13T23:23:01.619 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-13T23:23:01.619 INFO:journalctl@ceph.mon.c.smithi115.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-13T23:23:01.620 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-13T23:23:01.620 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.__get_result() 2022-05-13T23:23:01.620 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-13T23:23:01.620 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise self._exception 2022-05-13T23:23:01.620 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-13T23:23:01.621 INFO:journalctl@ceph.mon.c.smithi115.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-13T23:23:01.621 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-13T23:23:01.621 INFO:journalctl@ceph.mon.c.smithi115.stdout: conn = await self._remote_connection(host, addr) 2022-05-13T23:23:01.621 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-13T23:23:01.621 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise 2022-05-13T23:23:01.622 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-13T23:23:01.622 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.gen.throw(type, value, traceback) 2022-05-13T23:23:01.622 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-13T23:23:01.622 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OrchestratorError(msg) 2022-05-13T23:23:01.622 INFO:journalctl@ceph.mon.c.smithi115.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:23:01.623 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:23:01 smithi115 ceph-mon[78593]: 1 slow requests, 0 included below; oldest blocked for > 344.078547 secs 2022-05-13T23:23:02.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:23:02 smithi047 ceph-mon[77147]: pgmap v151: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 139 MiB data, 472 MiB used, 715 GiB / 715 GiB avail; 4.9 KiB/s rd, 5.3 KiB/s wr, 5 op/s; 10440/32430 objects degraded (32.192%) 2022-05-13T23:23:02.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:23:02 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:23:02.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:23:02 smithi047 ceph-mon[77147]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2022-05-13T23:23:02.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:23:02 smithi115 ceph-mon[78593]: pgmap v151: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 139 MiB data, 472 MiB used, 715 GiB / 715 GiB avail; 4.9 KiB/s rd, 5.3 KiB/s wr, 5 op/s; 10440/32430 objects degraded (32.192%) 2022-05-13T23:23:02.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:23:02 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:23:02.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:23:02 smithi115 ceph-mon[78593]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2022-05-13T23:23:04.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:23:04 smithi047 ceph-mon[77147]: pgmap v152: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 139 MiB data, 472 MiB used, 715 GiB / 715 GiB avail; 4.9 KiB/s rd, 5.3 KiB/s wr, 5 op/s; 10440/32430 objects degraded (32.192%) 2022-05-13T23:23:04.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:23:04 smithi115 ceph-mon[78593]: pgmap v152: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 139 MiB data, 472 MiB used, 715 GiB / 715 GiB avail; 4.9 KiB/s rd, 5.3 KiB/s wr, 5 op/s; 10440/32430 objects degraded (32.192%) 2022-05-13T23:23:06.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:23:06 smithi047 ceph-mon[77147]: pgmap v153: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 139 MiB data, 473 MiB used, 715 GiB / 715 GiB avail; 7.1 KiB/s rd, 7.1 KiB/s wr, 8 op/s; 10440/32430 objects degraded (32.192%) 2022-05-13T23:23:06.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:23:06 smithi047 ceph-mon[77147]: 1 slow requests, 0 included below; oldest blocked for > 349.078667 secs 2022-05-13T23:23:06.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:23:06 smithi115 ceph-mon[78593]: pgmap v153: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 139 MiB data, 473 MiB used, 715 GiB / 715 GiB avail; 7.1 KiB/s rd, 7.1 KiB/s wr, 8 op/s; 10440/32430 objects degraded (32.192%) 2022-05-13T23:23:06.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:23:06 smithi115 ceph-mon[78593]: 1 slow requests, 0 included below; oldest blocked for > 349.078667 secs 2022-05-13T23:23:08.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:23:08 smithi047 ceph-mon[77147]: pgmap v154: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 139 MiB data, 473 MiB used, 715 GiB / 715 GiB avail; 3.6 KiB/s rd, 3.8 KiB/s wr, 4 op/s; 10440/32430 objects degraded (32.192%) 2022-05-13T23:23:08.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:23:08 smithi115 ceph-mon[78593]: pgmap v154: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 139 MiB data, 473 MiB used, 715 GiB / 715 GiB avail; 3.6 KiB/s rd, 3.8 KiB/s wr, 4 op/s; 10440/32430 objects degraded (32.192%) 2022-05-13T23:23:10.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:23:10 smithi047 ceph-mon[77147]: pgmap v155: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 139 MiB data, 473 MiB used, 715 GiB / 715 GiB avail; 5.1 KiB/s rd, 4.7 KiB/s wr, 7 op/s; 10440/32430 objects degraded (32.192%) 2022-05-13T23:23:10.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:23:10 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:23:10.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:23:10 smithi047 ceph-mon[77147]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2022-05-13T23:23:10.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:23:10 smithi115 ceph-mon[78593]: pgmap v155: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 139 MiB data, 473 MiB used, 715 GiB / 715 GiB avail; 5.1 KiB/s rd, 4.7 KiB/s wr, 7 op/s; 10440/32430 objects degraded (32.192%) 2022-05-13T23:23:10.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:23:10 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:23:10.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:23:10 smithi115 ceph-mon[78593]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2022-05-13T23:23:11.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:23:11 smithi047 ceph-mon[77147]: 1 slow requests, 0 included below; oldest blocked for > 354.078735 secs 2022-05-13T23:23:11.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:23:11 smithi115 ceph-mon[78593]: 1 slow requests, 0 included below; oldest blocked for > 354.078735 secs 2022-05-13T23:23:12.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:23:12 smithi047 ceph-mon[77147]: pgmap v156: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 139 MiB data, 473 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 2.7 KiB/s wr, 4 op/s; 10440/32430 objects degraded (32.192%) 2022-05-13T23:23:12.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:23:12 smithi115 ceph-mon[78593]: pgmap v156: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 139 MiB data, 473 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 2.7 KiB/s wr, 4 op/s; 10440/32430 objects degraded (32.192%) 2022-05-13T23:23:14.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:23:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-13T23:23:14.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:23:14 smithi047 ceph-mon[77147]: pgmap v157: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 139 MiB data, 473 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 2.7 KiB/s wr, 4 op/s; 10440/32430 objects degraded (32.192%) 2022-05-13T23:23:14.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:23:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-13T23:23:14.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:23:14 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:23:14.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:23:14 smithi047 ceph-mon[77147]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2022-05-13T23:23:14.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:23:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-13T23:23:14.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:23:14 smithi115 ceph-mon[78593]: pgmap v157: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 139 MiB data, 473 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 2.7 KiB/s wr, 4 op/s; 10440/32430 objects degraded (32.192%) 2022-05-13T23:23:14.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:23:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-13T23:23:14.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:23:14 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:23:14.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:23:14 smithi115 ceph-mon[78593]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2022-05-13T23:23:16.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:23:16 smithi047 ceph-mon[77147]: pgmap v158: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 139 MiB data, 473 MiB used, 715 GiB / 715 GiB avail; 5.0 KiB/s rd, 3.4 KiB/s wr, 6 op/s; 10440/32430 objects degraded (32.192%) 2022-05-13T23:23:16.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:23:16 smithi047 ceph-mon[77147]: 1 slow requests, 0 included below; oldest blocked for > 359.078883 secs 2022-05-13T23:23:16.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:23:16 smithi115 ceph-mon[78593]: pgmap v158: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 139 MiB data, 473 MiB used, 715 GiB / 715 GiB avail; 5.0 KiB/s rd, 3.4 KiB/s wr, 6 op/s; 10440/32430 objects degraded (32.192%) 2022-05-13T23:23:16.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:23:16 smithi115 ceph-mon[78593]: 1 slow requests, 0 included below; oldest blocked for > 359.078883 secs 2022-05-13T23:23:18.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:23:18 smithi047 ceph-mon[77147]: pgmap v159: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 139 MiB data, 473 MiB used, 715 GiB / 715 GiB avail; 2.8 KiB/s rd, 1.7 KiB/s wr, 4 op/s; 10440/32430 objects degraded (32.192%) 2022-05-13T23:23:18.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:23:18 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:23:18.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:23:18 smithi047 ceph-mon[77147]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2022-05-13T23:23:18.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:23:18 smithi115 ceph-mon[78593]: pgmap v159: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 139 MiB data, 473 MiB used, 715 GiB / 715 GiB avail; 2.8 KiB/s rd, 1.7 KiB/s wr, 4 op/s; 10440/32430 objects degraded (32.192%) 2022-05-13T23:23:18.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:23:18 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:23:18.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:23:18 smithi115 ceph-mon[78593]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2022-05-13T23:23:20.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:23:20 smithi047 ceph-mon[77147]: pgmap v160: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 139 MiB data, 473 MiB used, 715 GiB / 715 GiB avail; 4.2 KiB/s rd, 2.3 KiB/s wr, 6 op/s; 10440/32430 objects degraded (32.192%) 2022-05-13T23:23:20.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:23:20 smithi115 ceph-mon[78593]: pgmap v160: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 139 MiB data, 473 MiB used, 715 GiB / 715 GiB avail; 4.2 KiB/s rd, 2.3 KiB/s wr, 6 op/s; 10440/32430 objects degraded (32.192%) 2022-05-13T23:23:21.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:23:21 smithi047 ceph-mon[77147]: 1 slow requests, 1 included below; oldest blocked for > 364.079036 secs 2022-05-13T23:23:21.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:23:21 smithi047 ceph-mon[77147]: slow request 364.079035 seconds old, received at 2022-05-13T23:17:16.166863+0000: client_request(client.25244:12077 setattr size=2217921 #0x10000002fec 2022-05-13T23:17:16.163392+0000 caller_uid=1000, caller_gid=1262{6,36,1000,1262,}) currently cleaned up request 2022-05-13T23:23:21.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:23:21 smithi115 ceph-mon[78593]: 1 slow requests, 1 included below; oldest blocked for > 364.079036 secs 2022-05-13T23:23:21.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:23:21 smithi115 ceph-mon[78593]: slow request 364.079035 seconds old, received at 2022-05-13T23:17:16.166863+0000: client_request(client.25244:12077 setattr size=2217921 #0x10000002fec 2022-05-13T23:17:16.163392+0000 caller_uid=1000, caller_gid=1262{6,36,1000,1262,}) currently cleaned up request 2022-05-13T23:23:22.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:23:22 smithi047 ceph-mon[77147]: pgmap v161: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 139 MiB data, 473 MiB used, 715 GiB / 715 GiB avail; 2.7 KiB/s rd, 1.4 KiB/s wr, 3 op/s; 10440/32430 objects degraded (32.192%) 2022-05-13T23:23:22.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:23:22 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:23:22.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:23:22 smithi047 ceph-mon[77147]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2022-05-13T23:23:22.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:23:22 smithi115 ceph-mon[78593]: pgmap v161: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 139 MiB data, 473 MiB used, 715 GiB / 715 GiB avail; 2.7 KiB/s rd, 1.4 KiB/s wr, 3 op/s; 10440/32430 objects degraded (32.192%) 2022-05-13T23:23:22.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:23:22 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:23:22.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:23:22 smithi115 ceph-mon[78593]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2022-05-13T23:23:24.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:23:24 smithi047 ceph-mon[77147]: pgmap v162: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 139 MiB data, 473 MiB used, 715 GiB / 715 GiB avail; 2.7 KiB/s rd, 1.4 KiB/s wr, 3 op/s; 10440/32430 objects degraded (32.192%) 2022-05-13T23:23:24.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:23:24 smithi115 ceph-mon[78593]: pgmap v162: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 139 MiB data, 473 MiB used, 715 GiB / 715 GiB avail; 2.7 KiB/s rd, 1.4 KiB/s wr, 3 op/s; 10440/32430 objects degraded (32.192%) 2022-05-13T23:23:26.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:23:26 smithi047 ceph-mon[77147]: pgmap v163: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 139 MiB data, 473 MiB used, 715 GiB / 715 GiB avail; 4.0 KiB/s rd, 2.0 KiB/s wr, 6 op/s; 10440/32430 objects degraded (32.192%) 2022-05-13T23:23:26.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:23:26 smithi047 ceph-mon[77147]: 1 slow requests, 0 included below; oldest blocked for > 369.079161 secs 2022-05-13T23:23:26.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:23:26 smithi115 ceph-mon[78593]: pgmap v163: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 139 MiB data, 473 MiB used, 715 GiB / 715 GiB avail; 4.0 KiB/s rd, 2.0 KiB/s wr, 6 op/s; 10440/32430 objects degraded (32.192%) 2022-05-13T23:23:26.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:23:26 smithi115 ceph-mon[78593]: 1 slow requests, 0 included below; oldest blocked for > 369.079161 secs 2022-05-13T23:23:28.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:23:28 smithi047 ceph-mon[77147]: pgmap v164: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 139 MiB data, 473 MiB used, 715 GiB / 715 GiB avail; 2.7 KiB/s rd, 1.2 KiB/s wr, 3 op/s; 10440/32430 objects degraded (32.192%) 2022-05-13T23:23:28.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:23:28 smithi115 ceph-mon[78593]: pgmap v164: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 139 MiB data, 473 MiB used, 715 GiB / 715 GiB avail; 2.7 KiB/s rd, 1.2 KiB/s wr, 3 op/s; 10440/32430 objects degraded (32.192%) 2022-05-13T23:23:30.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:23:30 smithi047 ceph-mon[77147]: pgmap v165: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 139 MiB data, 473 MiB used, 715 GiB / 715 GiB avail; 3.3 KiB/s rd, 1.2 KiB/s wr, 5 op/s; 10440/32430 objects degraded (32.192%) 2022-05-13T23:23:30.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:23:30 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:23:30.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:23:30 smithi047 ceph-mon[77147]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2022-05-13T23:23:30.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:23:30 smithi115 ceph-mon[78593]: pgmap v165: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 139 MiB data, 473 MiB used, 715 GiB / 715 GiB avail; 3.3 KiB/s rd, 1.2 KiB/s wr, 5 op/s; 10440/32430 objects degraded (32.192%) 2022-05-13T23:23:30.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:23:30 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:23:30.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:23:30 smithi115 ceph-mon[78593]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2022-05-13T23:23:31.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:23:31 smithi047 ceph-mon[77147]: 1 slow requests, 0 included below; oldest blocked for > 374.079245 secs 2022-05-13T23:23:31.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:23:31 smithi115 ceph-mon[78593]: 1 slow requests, 0 included below; oldest blocked for > 374.079245 secs 2022-05-13T23:23:32.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:23:32 smithi047 ceph-mon[77147]: pgmap v166: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 139 MiB data, 473 MiB used, 715 GiB / 715 GiB avail; 1.9 KiB/s rd, 597 B/s wr, 3 op/s; 10440/32430 objects degraded (32.192%) 2022-05-13T23:23:32.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:23:32 smithi115 ceph-mon[78593]: pgmap v166: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 139 MiB data, 473 MiB used, 715 GiB / 715 GiB avail; 1.9 KiB/s rd, 597 B/s wr, 3 op/s; 10440/32430 objects degraded (32.192%) 2022-05-13T23:23:34.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:23:34 smithi047 ceph-mon[77147]: pgmap v167: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 139 MiB data, 473 MiB used, 715 GiB / 715 GiB avail; 1.9 KiB/s rd, 597 B/s wr, 3 op/s; 10440/32430 objects degraded (32.192%) 2022-05-13T23:23:34.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:23:34 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:23:34.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:23:34 smithi047 ceph-mon[77147]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2022-05-13T23:23:34.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:23:34 smithi115 ceph-mon[78593]: pgmap v167: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 139 MiB data, 473 MiB used, 715 GiB / 715 GiB avail; 1.9 KiB/s rd, 597 B/s wr, 3 op/s; 10440/32430 objects degraded (32.192%) 2022-05-13T23:23:34.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:23:34 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:23:34.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:23:34 smithi115 ceph-mon[78593]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2022-05-13T23:23:36.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:23:36 smithi047 ceph-mon[77147]: pgmap v168: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 139 MiB data, 473 MiB used, 715 GiB / 715 GiB avail; 3.1 KiB/s rd, 938 B/s wr, 5 op/s; 10440/32430 objects degraded (32.192%) 2022-05-13T23:23:36.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:23:36 smithi047 ceph-mon[77147]: 1 slow requests, 0 included below; oldest blocked for > 379.079358 secs 2022-05-13T23:23:36.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:23:36 smithi115 ceph-mon[78593]: pgmap v168: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 139 MiB data, 473 MiB used, 715 GiB / 715 GiB avail; 3.1 KiB/s rd, 938 B/s wr, 5 op/s; 10440/32430 objects degraded (32.192%) 2022-05-13T23:23:36.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:23:36 smithi115 ceph-mon[78593]: 1 slow requests, 0 included below; oldest blocked for > 379.079358 secs 2022-05-13T23:23:38.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:23:38 smithi047 ceph-mon[77147]: pgmap v169: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 139 MiB data, 473 MiB used, 715 GiB / 715 GiB avail; 1.8 KiB/s rd, 341 B/s wr, 3 op/s; 10440/32430 objects degraded (32.192%) 2022-05-13T23:23:38.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:23:38 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:23:38.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:23:38 smithi047 ceph-mon[77147]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2022-05-13T23:23:38.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:23:38 smithi115 ceph-mon[78593]: pgmap v169: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 139 MiB data, 473 MiB used, 715 GiB / 715 GiB avail; 1.8 KiB/s rd, 341 B/s wr, 3 op/s; 10440/32430 objects degraded (32.192%) 2022-05-13T23:23:38.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:23:38 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:23:38.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:23:38 smithi115 ceph-mon[78593]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2022-05-13T23:23:40.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:23:40 smithi047 ceph-mon[77147]: pgmap v170: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 139 MiB data, 473 MiB used, 715 GiB / 715 GiB avail; 2.7 KiB/s rd, 341 B/s wr, 4 op/s; 10440/32430 objects degraded (32.192%) 2022-05-13T23:23:40.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:23:40 smithi115 ceph-mon[78593]: pgmap v170: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 139 MiB data, 473 MiB used, 715 GiB / 715 GiB avail; 2.7 KiB/s rd, 341 B/s wr, 4 op/s; 10440/32430 objects degraded (32.192%) 2022-05-13T23:23:41.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:23:41 smithi047 ceph-mon[77147]: 1 slow requests, 0 included below; oldest blocked for > 384.079455 secs 2022-05-13T23:23:41.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:23:41 smithi115 ceph-mon[78593]: 1 slow requests, 0 included below; oldest blocked for > 384.079455 secs 2022-05-13T23:23:42.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:23:42 smithi047 ceph-mon[77147]: pgmap v171: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 139 MiB data, 473 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 341 B/s wr, 3 op/s; 10440/32430 objects degraded (32.192%) 2022-05-13T23:23:42.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:23:42 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:23:42.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:23:42 smithi047 ceph-mon[77147]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2022-05-13T23:23:42.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:23:42 smithi115 ceph-mon[78593]: pgmap v171: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 139 MiB data, 473 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 341 B/s wr, 3 op/s; 10440/32430 objects degraded (32.192%) 2022-05-13T23:23:42.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:23:42 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:23:42.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:23:42 smithi115 ceph-mon[78593]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2022-05-13T23:23:44.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:23:44 smithi047 ceph-mon[77147]: pgmap v172: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 139 MiB data, 473 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 341 B/s wr, 3 op/s; 10440/32430 objects degraded (32.192%) 2022-05-13T23:23:44.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:23:44 smithi115 ceph-mon[78593]: pgmap v172: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 139 MiB data, 473 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 341 B/s wr, 3 op/s; 10440/32430 objects degraded (32.192%) 2022-05-13T23:23:46.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:23:46 smithi047 ceph-mon[77147]: pgmap v173: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 139 MiB data, 473 MiB used, 715 GiB / 715 GiB avail; 2.8 KiB/s rd, 341 B/s wr, 5 op/s; 10440/32430 objects degraded (32.192%) 2022-05-13T23:23:46.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:23:46 smithi047 ceph-mon[77147]: 1 slow requests, 0 included below; oldest blocked for > 389.079549 secs 2022-05-13T23:23:46.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:23:46 smithi115 ceph-mon[78593]: pgmap v173: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 139 MiB data, 473 MiB used, 715 GiB / 715 GiB avail; 2.8 KiB/s rd, 341 B/s wr, 5 op/s; 10440/32430 objects degraded (32.192%) 2022-05-13T23:23:46.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:23:46 smithi115 ceph-mon[78593]: 1 slow requests, 0 included below; oldest blocked for > 389.079549 secs 2022-05-13T23:23:48.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:23:48 smithi047 ceph-mon[77147]: pgmap v174: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 139 MiB data, 473 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 3 op/s; 10440/32430 objects degraded (32.192%) 2022-05-13T23:23:48.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:23:48 smithi115 ceph-mon[78593]: pgmap v174: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 139 MiB data, 473 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 3 op/s; 10440/32430 objects degraded (32.192%) 2022-05-13T23:23:50.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:23:50 smithi047 ceph-mon[77147]: pgmap v175: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 139 MiB data, 473 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s; 10440/32430 objects degraded (32.192%) 2022-05-13T23:23:50.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:23:50 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:23:50.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:23:50 smithi047 ceph-mon[77147]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2022-05-13T23:23:50.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:23:50 smithi115 ceph-mon[78593]: pgmap v175: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 139 MiB data, 473 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s; 10440/32430 objects degraded (32.192%) 2022-05-13T23:23:50.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:23:50 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:23:50.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:23:50 smithi115 ceph-mon[78593]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2022-05-13T23:23:51.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:23:51 smithi047 ceph-mon[77147]: 1 slow requests, 0 included below; oldest blocked for > 394.079689 secs 2022-05-13T23:23:51.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:23:51 smithi115 ceph-mon[78593]: 1 slow requests, 0 included below; oldest blocked for > 394.079689 secs 2022-05-13T23:23:52.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:23:52 smithi047 ceph-mon[77147]: pgmap v176: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 139 MiB data, 473 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 3 op/s; 10440/32430 objects degraded (32.192%) 2022-05-13T23:23:52.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:23:52 smithi115 ceph-mon[78593]: pgmap v176: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 139 MiB data, 473 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 3 op/s; 10440/32430 objects degraded (32.192%) 2022-05-13T23:23:54.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:23:54 smithi047 ceph-mon[77147]: pgmap v177: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 139 MiB data, 473 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 3 op/s; 10440/32430 objects degraded (32.192%) 2022-05-13T23:23:54.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:23:54 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:23:54.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:23:54 smithi047 ceph-mon[77147]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2022-05-13T23:23:54.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:23:54 smithi115 ceph-mon[78593]: pgmap v177: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 139 MiB data, 473 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 3 op/s; 10440/32430 objects degraded (32.192%) 2022-05-13T23:23:54.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:23:54 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:23:54.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:23:54 smithi115 ceph-mon[78593]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2022-05-13T23:23:56.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:23:56 smithi047 ceph-mon[77147]: pgmap v178: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 139 MiB data, 473 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s; 10440/32430 objects degraded (32.192%) 2022-05-13T23:23:56.570 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:23:56 smithi047 ceph-mon[77147]: 1 slow requests, 0 included below; oldest blocked for > 399.079826 secs 2022-05-13T23:23:56.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:23:56 smithi115 ceph-mon[78593]: pgmap v178: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 139 MiB data, 473 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s; 10440/32430 objects degraded (32.192%) 2022-05-13T23:23:56.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:23:56 smithi115 ceph-mon[78593]: 1 slow requests, 0 included below; oldest blocked for > 399.079826 secs 2022-05-13T23:23:58.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:23:58 smithi047 ceph-mon[77147]: pgmap v179: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 139 MiB data, 473 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 3 op/s; 10440/32430 objects degraded (32.192%) 2022-05-13T23:23:58.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:23:58 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:23:58.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:23:58 smithi047 ceph-mon[77147]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2022-05-13T23:23:58.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:23:58 smithi115 ceph-mon[78593]: pgmap v179: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 139 MiB data, 473 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 3 op/s; 10440/32430 objects degraded (32.192%) 2022-05-13T23:23:58.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:23:58 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:23:58.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:23:58 smithi115 ceph-mon[78593]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2022-05-13T23:24:00.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:24:00 smithi047 ceph-mon[77147]: pgmap v180: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 139 MiB data, 473 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s; 10440/32430 objects degraded (32.192%) 2022-05-13T23:24:00.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:24:00 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-13T23:24:00.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:24:00 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-13T23:24:00.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:24:00 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-13T23:24:00.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:24:00 smithi115 ceph-mon[78593]: pgmap v180: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 139 MiB data, 473 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s; 10440/32430 objects degraded (32.192%) 2022-05-13T23:24:00.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:24:00 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-13T23:24:00.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:24:00 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-13T23:24:00.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:24:00 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-13T23:24:01.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:24:01 smithi047 ceph-mon[77147]: 1 slow requests, 0 included below; oldest blocked for > 404.079945 secs 2022-05-13T23:24:01.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:24:01 smithi047 conmon[78521]: 2022-05-13T23:24:01.071+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:24:01.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:24:01 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-13T23:24:01.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:24:01 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-13T23:24:01.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:24:01 smithi047 conmon[78521]: yield 2022-05-13T23:24:01.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:24:01 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-13T23:24:01.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:24:01 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-13T23:24:01.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:24:01 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-13T23:24:01.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:24:01 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-13T23:24:01.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:24:01 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-13T23:24:01.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:24:01 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-13T23:24:01.547 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:24:01 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-13T23:24:01.547 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:24:01 smithi047 conmon[78521]: raise exceptions[0] 2022-05-13T23:24:01.548 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:24:01 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-13T23:24:01.548 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:24:01 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-13T23:24:01.549 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:24:01 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-13T23:24:01.549 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:24:01 smithi047 conmon[78521]: return (yield from fut) 2022-05-13T23:24:01.550 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:24:01 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-13T23:24:01.551 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:24:01 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-13T23:24:01.551 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:24:01 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:24:01.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:24:01 smithi115 ceph-mon[78593]: 1 slow requests, 0 included below; oldest blocked for > 404.079945 secs 2022-05-13T23:24:02.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:24:02 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:24:02.542 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-13T23:24:02.542 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-13T23:24:02.542 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-13T23:24:02.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-13T23:24:02.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-13T23:24:02.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-13T23:24:02.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-13T23:24:02.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-13T23:24:02.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-13T23:24:02.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-13T23:24:02.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-13T23:24:02.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-13T23:24:02.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-13T23:24:02.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-13T23:24:02.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-13T23:24:02.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-13T23:24:02.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-13T23:24:02.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:24:02.546 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:24:02 smithi047 ceph-mon[77147]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-13T23:24:02.546 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:24:02 smithi047 ceph-mon[77147]: pgmap v181: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 139 MiB data, 473 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 3 op/s; 10440/32430 objects degraded (32.192%) 2022-05-13T23:24:02.546 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:24:02 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:24:02.547 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:24:02 smithi047 ceph-mon[77147]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2022-05-13T23:24:02.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:24:02 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:24:02.608 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-13T23:24:02.609 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-13T23:24:02.609 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-13T23:24:02.609 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-13T23:24:02.609 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-13T23:24:02.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-13T23:24:02.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-13T23:24:02.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-13T23:24:02.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-13T23:24:02.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-13T23:24:02.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-13T23:24:02.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-13T23:24:02.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-13T23:24:02.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-13T23:24:02.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-13T23:24:02.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-13T23:24:02.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-13T23:24:02.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:24:02.612 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:24:02 smithi115 ceph-mon[78593]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-13T23:24:02.612 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:24:02 smithi115 ceph-mon[78593]: pgmap v181: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 139 MiB data, 473 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 3 op/s; 10440/32430 objects degraded (32.192%) 2022-05-13T23:24:02.613 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:24:02 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:24:02.613 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:24:02 smithi115 ceph-mon[78593]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2022-05-13T23:24:04.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:24:04 smithi115 ceph-mon[78593]: pgmap v182: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 139 MiB data, 473 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 3 op/s; 10440/32430 objects degraded (32.192%) 2022-05-13T23:24:04.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:24:04 smithi047 ceph-mon[77147]: pgmap v182: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 139 MiB data, 473 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 3 op/s; 10440/32430 objects degraded (32.192%) 2022-05-13T23:24:04.542 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:24:04 smithi047 conmon[78521]: 2022-05-13T23:24:04.144+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:24:04.542 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:24:04 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-13T23:24:04.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:24:04 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-13T23:24:04.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:24:04 smithi047 conmon[78521]: yield 2022-05-13T23:24:04.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:24:04 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-13T23:24:04.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:24:04 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-13T23:24:04.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:24:04 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-13T23:24:04.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:24:04 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-13T23:24:04.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:24:04 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-13T23:24:04.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:24:04 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-13T23:24:04.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:24:04 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-13T23:24:04.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:24:04 smithi047 conmon[78521]: raise exceptions[0] 2022-05-13T23:24:04.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:24:04 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-13T23:24:04.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:24:04 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-13T23:24:04.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:24:04 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-13T23:24:04.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:24:04 smithi047 conmon[78521]: return (yield from fut) 2022-05-13T23:24:04.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:24:04 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-13T23:24:04.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:24:04 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-13T23:24:04.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:24:04 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:24:04.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:24:04 smithi047 conmon[78521]: 2022-05-13T23:24:04.144+0000 7f5340779700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-13T23:24:04.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:24:04 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-13T23:24:04.547 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:24:04 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-13T23:24:04.547 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:24:04 smithi047 conmon[78521]: yield 2022-05-13T23:24:04.547 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:24:04 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-13T23:24:04.548 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:24:04 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-13T23:24:04.548 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:24:04 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-13T23:24:04.548 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:24:04 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-13T23:24:04.548 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:24:04 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-13T23:24:04.549 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:24:04 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-13T23:24:04.549 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:24:04 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-13T23:24:04.549 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:24:04 smithi047 conmon[78521]: raise exceptions[0] 2022-05-13T23:24:04.549 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:24:04 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-13T23:24:04.550 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:24:04 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-13T23:24:04.550 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:24:04 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-13T23:24:04.550 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:24:04 smithi047 conmon[78521]: return (yield from fut) 2022-05-13T23:24:04.551 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:24:04 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-13T23:24:04.551 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:24:04 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-13T23:24:04.551 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:24:04 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:24:04.551 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:24:04 smithi047 conmon[78521]: 2022-05-13T23:24:04.552 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:24:04 smithi047 conmon[78521]: During handling of the above exception, another exception occurred: 2022-05-13T23:24:04.552 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:24:04 smithi047 conmon[78521]: 2022-05-13T23:24:04.552 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:24:04 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-13T23:24:04.552 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:24:04 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-13T23:24:04.552 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:24:04 smithi047 conmon[78521]: return f(*arg) 2022-05-13T23:24:04.553 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:24:04 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-13T23:24:04.553 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:24:04 smithi047 conmon[78521]: self._write_client_files(client_files, host) 2022-05-13T23:24:04.553 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:24:04 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-13T23:24:04.553 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:24:04 smithi047 conmon[78521]: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-13T23:24:04.554 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:24:04 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-13T23:24:04.554 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:24:04 smithi047 conmon[78521]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-13T23:24:04.554 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:24:04 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-13T23:24:04.554 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:24:04 smithi047 conmon[78521]: return self.event_loop.get_result(coro) 2022-05-13T23:24:04.555 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:24:04 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-13T23:24:04.555 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:24:04 smithi047 conmon[78521]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-13T23:24:04.555 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:24:04 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-13T23:24:04.556 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:24:04 smithi047 conmon[78521]: return self.__get_result() 2022-05-13T23:24:04.556 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:24:04 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-13T23:24:04.556 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:24:04 smithi047 conmon[78521]: raise self._exception 2022-05-13T23:24:04.556 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:24:04 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-13T23:24:04.556 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:24:04 smithi047 conmon[78521]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-13T23:24:04.557 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:24:04 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-13T23:24:04.557 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:24:04 smithi047 conmon[78521]: conn = await self._remote_connection(host, addr) 2022-05-13T23:24:04.557 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:24:04 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-13T23:24:04.557 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:24:04 smithi047 conmon[78521]: raise 2022-05-13T23:24:04.558 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:24:04 smithi047 conmon[78521]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-13T23:24:04.558 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:24:04 smithi047 conmon[78521]: self.gen.throw(type, value, traceback) 2022-05-13T23:24:04.558 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:24:04 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-13T23:24:04.558 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:24:04 smithi047 conmon[78521]: raise OrchestratorError(msg) 2022-05-13T23:24:04.558 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:24:04 smithi047 conmon[78521]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:24:05.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:24:05 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:24:05.359 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-13T23:24:05.359 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-13T23:24:05.359 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-13T23:24:05.359 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-13T23:24:05.359 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-13T23:24:05.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-13T23:24:05.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-13T23:24:05.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-13T23:24:05.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-13T23:24:05.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-13T23:24:05.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-13T23:24:05.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-13T23:24:05.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-13T23:24:05.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-13T23:24:05.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-13T23:24:05.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-13T23:24:05.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-13T23:24:05.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:24:05.363 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:24:05 smithi115 ceph-mon[78593]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-13T23:24:05.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-13T23:24:05.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-13T23:24:05.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-13T23:24:05.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-13T23:24:05.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-13T23:24:05.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-13T23:24:05.365 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-13T23:24:05.365 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-13T23:24:05.365 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-13T23:24:05.365 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-13T23:24:05.366 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-13T23:24:05.366 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-13T23:24:05.366 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-13T23:24:05.366 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-13T23:24:05.367 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-13T23:24:05.367 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-13T23:24:05.367 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-13T23:24:05.367 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:24:05.367 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-13T23:24:05.368 INFO:journalctl@ceph.mon.c.smithi115.stdout: During handling of the above exception, another exception occurred: 2022-05-13T23:24:05.368 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-13T23:24:05.368 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-13T23:24:05.368 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-13T23:24:05.368 INFO:journalctl@ceph.mon.c.smithi115.stdout: return f(*arg) 2022-05-13T23:24:05.369 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-13T23:24:05.369 INFO:journalctl@ceph.mon.c.smithi115.stdout: self._write_client_files(client_files, host) 2022-05-13T23:24:05.369 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-13T23:24:05.370 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-13T23:24:05.370 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-13T23:24:05.370 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-13T23:24:05.370 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-13T23:24:05.370 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.event_loop.get_result(coro) 2022-05-13T23:24:05.371 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-13T23:24:05.371 INFO:journalctl@ceph.mon.c.smithi115.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-13T23:24:05.371 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-13T23:24:05.371 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.__get_result() 2022-05-13T23:24:05.372 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-13T23:24:05.373 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise self._exception 2022-05-13T23:24:05.373 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-13T23:24:05.373 INFO:journalctl@ceph.mon.c.smithi115.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-13T23:24:05.373 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-13T23:24:05.374 INFO:journalctl@ceph.mon.c.smithi115.stdout: conn = await self._remote_connection(host, addr) 2022-05-13T23:24:05.374 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-13T23:24:05.374 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise 2022-05-13T23:24:05.374 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-13T23:24:05.374 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.gen.throw(type, value, traceback) 2022-05-13T23:24:05.375 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-13T23:24:05.375 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OrchestratorError(msg) 2022-05-13T23:24:05.375 INFO:journalctl@ceph.mon.c.smithi115.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:24:05.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:24:05 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:24:05.542 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-13T23:24:05.542 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-13T23:24:05.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-13T23:24:05.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-13T23:24:05.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-13T23:24:05.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-13T23:24:05.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-13T23:24:05.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-13T23:24:05.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-13T23:24:05.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-13T23:24:05.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-13T23:24:05.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-13T23:24:05.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-13T23:24:05.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-13T23:24:05.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-13T23:24:05.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-13T23:24:05.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-13T23:24:05.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:24:05.546 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:24:05 smithi047 ceph-mon[77147]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-13T23:24:05.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-13T23:24:05.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-13T23:24:05.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-13T23:24:05.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-13T23:24:05.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-13T23:24:05.548 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-13T23:24:05.548 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-13T23:24:05.548 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-13T23:24:05.548 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-13T23:24:05.548 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-13T23:24:05.549 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-13T23:24:05.549 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-13T23:24:05.549 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-13T23:24:05.550 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-13T23:24:05.550 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-13T23:24:05.550 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-13T23:24:05.550 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-13T23:24:05.550 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:24:05.551 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-13T23:24:05.552 INFO:journalctl@ceph.mon.b.smithi047.stdout: During handling of the above exception, another exception occurred: 2022-05-13T23:24:05.552 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-13T23:24:05.552 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-13T23:24:05.552 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-13T23:24:05.552 INFO:journalctl@ceph.mon.b.smithi047.stdout: return f(*arg) 2022-05-13T23:24:05.553 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-13T23:24:05.553 INFO:journalctl@ceph.mon.b.smithi047.stdout: self._write_client_files(client_files, host) 2022-05-13T23:24:05.553 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-13T23:24:05.553 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-13T23:24:05.553 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-13T23:24:05.554 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-13T23:24:05.554 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-13T23:24:05.554 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.event_loop.get_result(coro) 2022-05-13T23:24:05.554 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-13T23:24:05.555 INFO:journalctl@ceph.mon.b.smithi047.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-13T23:24:05.555 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-13T23:24:05.555 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.__get_result() 2022-05-13T23:24:05.555 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-13T23:24:05.555 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise self._exception 2022-05-13T23:24:05.556 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-13T23:24:05.556 INFO:journalctl@ceph.mon.b.smithi047.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-13T23:24:05.556 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-13T23:24:05.556 INFO:journalctl@ceph.mon.b.smithi047.stdout: conn = await self._remote_connection(host, addr) 2022-05-13T23:24:05.556 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-13T23:24:05.557 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise 2022-05-13T23:24:05.557 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-13T23:24:05.557 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.gen.throw(type, value, traceback) 2022-05-13T23:24:05.557 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-13T23:24:05.558 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OrchestratorError(msg) 2022-05-13T23:24:05.558 INFO:journalctl@ceph.mon.b.smithi047.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:24:06.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:24:06 smithi115 ceph-mon[78593]: pgmap v183: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 139 MiB data, 473 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s; 10440/32430 objects degraded (32.192%) 2022-05-13T23:24:06.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:24:06 smithi115 ceph-mon[78593]: 1 slow requests, 0 included below; oldest blocked for > 409.080064 secs 2022-05-13T23:24:06.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:24:06 smithi047 ceph-mon[77147]: pgmap v183: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 139 MiB data, 473 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s; 10440/32430 objects degraded (32.192%) 2022-05-13T23:24:06.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:24:06 smithi047 ceph-mon[77147]: 1 slow requests, 0 included below; oldest blocked for > 409.080064 secs 2022-05-13T23:24:08.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:24:08 smithi047 ceph-mon[77147]: pgmap v184: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 139 MiB data, 473 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 3 op/s; 10440/32430 objects degraded (32.192%) 2022-05-13T23:24:08.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:24:08 smithi115 ceph-mon[78593]: pgmap v184: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 139 MiB data, 473 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 3 op/s; 10440/32430 objects degraded (32.192%) 2022-05-13T23:24:10.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:24:10 smithi047 ceph-mon[77147]: pgmap v185: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 139 MiB data, 473 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s; 10440/32430 objects degraded (32.192%) 2022-05-13T23:24:10.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:24:10 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:24:10.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:24:10 smithi047 ceph-mon[77147]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2022-05-13T23:24:10.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:24:10 smithi115 ceph-mon[78593]: pgmap v185: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 139 MiB data, 473 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s; 10440/32430 objects degraded (32.192%) 2022-05-13T23:24:10.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:24:10 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:24:10.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:24:10 smithi115 ceph-mon[78593]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2022-05-13T23:24:11.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:24:11 smithi047 ceph-mon[77147]: 1 slow requests, 0 included below; oldest blocked for > 414.080164 secs 2022-05-13T23:24:11.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:24:11 smithi115 ceph-mon[78593]: 1 slow requests, 0 included below; oldest blocked for > 414.080164 secs 2022-05-13T23:24:12.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:24:12 smithi047 ceph-mon[77147]: pgmap v186: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 139 MiB data, 473 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 3 op/s; 10440/32430 objects degraded (32.192%) 2022-05-13T23:24:12.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:24:12 smithi115 ceph-mon[78593]: pgmap v186: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 139 MiB data, 473 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 3 op/s; 10440/32430 objects degraded (32.192%) 2022-05-13T23:24:14.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:24:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-13T23:24:14.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:24:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-13T23:24:14.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:24:14 smithi115 ceph-mon[78593]: pgmap v187: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 139 MiB data, 473 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 3 op/s; 10440/32430 objects degraded (32.192%) 2022-05-13T23:24:14.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:24:14 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:24:14.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:24:14 smithi115 ceph-mon[78593]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2022-05-13T23:24:14.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:24:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-13T23:24:14.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:24:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-13T23:24:14.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:24:14 smithi047 ceph-mon[77147]: pgmap v187: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 139 MiB data, 473 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 3 op/s; 10440/32430 objects degraded (32.192%) 2022-05-13T23:24:14.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:24:14 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:24:14.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:24:14 smithi047 ceph-mon[77147]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2022-05-13T23:24:16.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:24:16 smithi047 ceph-mon[77147]: pgmap v188: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 139 MiB data, 473 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s; 10440/32430 objects degraded (32.192%) 2022-05-13T23:24:16.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:24:16 smithi047 ceph-mon[77147]: 1 slow requests, 0 included below; oldest blocked for > 419.080281 secs 2022-05-13T23:24:16.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:24:16 smithi115 ceph-mon[78593]: pgmap v188: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 139 MiB data, 473 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s; 10440/32430 objects degraded (32.192%) 2022-05-13T23:24:16.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:24:16 smithi115 ceph-mon[78593]: 1 slow requests, 0 included below; oldest blocked for > 419.080281 secs 2022-05-13T23:24:18.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:24:18 smithi047 ceph-mon[77147]: pgmap v189: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 139 MiB data, 473 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 3 op/s; 10440/32430 objects degraded (32.192%) 2022-05-13T23:24:18.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:24:18 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:24:18.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:24:18 smithi047 ceph-mon[77147]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2022-05-13T23:24:18.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:24:18 smithi115 ceph-mon[78593]: pgmap v189: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 139 MiB data, 473 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 3 op/s; 10440/32430 objects degraded (32.192%) 2022-05-13T23:24:18.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:24:18 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:24:18.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:24:18 smithi115 ceph-mon[78593]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2022-05-13T23:24:20.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:24:20 smithi047 ceph-mon[77147]: pgmap v190: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 139 MiB data, 473 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s; 10440/32430 objects degraded (32.192%) 2022-05-13T23:24:20.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:24:20 smithi115 ceph-mon[78593]: pgmap v190: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 139 MiB data, 473 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s; 10440/32430 objects degraded (32.192%) 2022-05-13T23:24:21.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:24:21 smithi047 ceph-mon[77147]: 1 slow requests, 0 included below; oldest blocked for > 424.080409 secs 2022-05-13T23:24:21.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:24:21 smithi115 ceph-mon[78593]: 1 slow requests, 0 included below; oldest blocked for > 424.080409 secs 2022-05-13T23:24:22.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:24:22 smithi047 ceph-mon[77147]: pgmap v191: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 139 MiB data, 473 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 3 op/s; 10440/32430 objects degraded (32.192%) 2022-05-13T23:24:22.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:24:22 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:24:22.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:24:22 smithi047 ceph-mon[77147]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2022-05-13T23:24:22.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:24:22 smithi115 ceph-mon[78593]: pgmap v191: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 139 MiB data, 473 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 3 op/s; 10440/32430 objects degraded (32.192%) 2022-05-13T23:24:22.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:24:22 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:24:22.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:24:22 smithi115 ceph-mon[78593]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2022-05-13T23:24:24.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:24:24 smithi115 ceph-mon[78593]: pgmap v192: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 139 MiB data, 473 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 3 op/s; 10440/32430 objects degraded (32.192%) 2022-05-13T23:24:24.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:24:24 smithi047 ceph-mon[77147]: pgmap v192: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 139 MiB data, 473 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 3 op/s; 10440/32430 objects degraded (32.192%) 2022-05-13T23:24:26.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:24:26 smithi047 ceph-mon[77147]: pgmap v193: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 139 MiB data, 473 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s; 10440/32430 objects degraded (32.192%) 2022-05-13T23:24:26.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:24:26 smithi047 ceph-mon[77147]: 1 slow requests, 0 included below; oldest blocked for > 429.080530 secs 2022-05-13T23:24:26.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:24:26 smithi115 ceph-mon[78593]: pgmap v193: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 139 MiB data, 473 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s; 10440/32430 objects degraded (32.192%) 2022-05-13T23:24:26.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:24:26 smithi115 ceph-mon[78593]: 1 slow requests, 0 included below; oldest blocked for > 429.080530 secs 2022-05-13T23:24:28.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:24:28 smithi047 ceph-mon[77147]: pgmap v194: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 139 MiB data, 473 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 3 op/s; 10440/32430 objects degraded (32.192%) 2022-05-13T23:24:28.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:24:28 smithi115 ceph-mon[78593]: pgmap v194: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 139 MiB data, 473 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 3 op/s; 10440/32430 objects degraded (32.192%) 2022-05-13T23:24:30.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:24:30 smithi047 ceph-mon[77147]: pgmap v195: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 139 MiB data, 473 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s; 10440/32430 objects degraded (32.192%) 2022-05-13T23:24:30.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:24:30 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:24:30.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:24:30 smithi047 ceph-mon[77147]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2022-05-13T23:24:30.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:24:30 smithi115 ceph-mon[78593]: pgmap v195: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 139 MiB data, 473 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s; 10440/32430 objects degraded (32.192%) 2022-05-13T23:24:30.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:24:30 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:24:30.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:24:30 smithi115 ceph-mon[78593]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2022-05-13T23:24:31.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:24:31 smithi047 ceph-mon[77147]: 1 slow requests, 0 included below; oldest blocked for > 434.080630 secs 2022-05-13T23:24:31.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:24:31 smithi115 ceph-mon[78593]: 1 slow requests, 0 included below; oldest blocked for > 434.080630 secs 2022-05-13T23:24:32.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:24:32 smithi047 ceph-mon[77147]: pgmap v196: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 139 MiB data, 473 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 3 op/s; 10440/32430 objects degraded (32.192%) 2022-05-13T23:24:32.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:24:32 smithi115 ceph-mon[78593]: pgmap v196: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 139 MiB data, 473 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 3 op/s; 10440/32430 objects degraded (32.192%) 2022-05-13T23:24:34.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:24:34 smithi047 ceph-mon[77147]: pgmap v197: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 139 MiB data, 473 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 3 op/s; 10440/32430 objects degraded (32.192%) 2022-05-13T23:24:34.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:24:34 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:24:34.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:24:34 smithi047 ceph-mon[77147]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2022-05-13T23:24:34.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:24:34 smithi115 ceph-mon[78593]: pgmap v197: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 139 MiB data, 473 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 3 op/s; 10440/32430 objects degraded (32.192%) 2022-05-13T23:24:34.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:24:34 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:24:34.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:24:34 smithi115 ceph-mon[78593]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2022-05-13T23:24:36.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:24:36 smithi047 ceph-mon[77147]: pgmap v198: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 139 MiB data, 473 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s; 10440/32430 objects degraded (32.192%) 2022-05-13T23:24:36.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:24:36 smithi047 ceph-mon[77147]: 1 slow requests, 0 included below; oldest blocked for > 439.080767 secs 2022-05-13T23:24:36.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:24:36 smithi115 ceph-mon[78593]: pgmap v198: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 139 MiB data, 473 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s; 10440/32430 objects degraded (32.192%) 2022-05-13T23:24:36.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:24:36 smithi115 ceph-mon[78593]: 1 slow requests, 0 included below; oldest blocked for > 439.080767 secs 2022-05-13T23:24:38.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:24:38 smithi047 ceph-mon[77147]: pgmap v199: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 139 MiB data, 473 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 3 op/s; 10440/32430 objects degraded (32.192%) 2022-05-13T23:24:38.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:24:38 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:24:38.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:24:38 smithi047 ceph-mon[77147]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2022-05-13T23:24:38.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:24:38 smithi115 ceph-mon[78593]: pgmap v199: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 139 MiB data, 473 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 3 op/s; 10440/32430 objects degraded (32.192%) 2022-05-13T23:24:38.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:24:38 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:24:38.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:24:38 smithi115 ceph-mon[78593]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2022-05-13T23:24:40.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:24:40 smithi047 ceph-mon[77147]: pgmap v200: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 139 MiB data, 473 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s; 10440/32430 objects degraded (32.192%) 2022-05-13T23:24:40.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:24:40 smithi115 ceph-mon[78593]: pgmap v200: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 139 MiB data, 473 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s; 10440/32430 objects degraded (32.192%) 2022-05-13T23:24:41.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:24:41 smithi047 ceph-mon[77147]: 1 slow requests, 0 included below; oldest blocked for > 444.080897 secs 2022-05-13T23:24:41.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:24:41 smithi115 ceph-mon[78593]: 1 slow requests, 0 included below; oldest blocked for > 444.080897 secs 2022-05-13T23:24:42.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:24:42 smithi047 ceph-mon[77147]: pgmap v201: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 139 MiB data, 473 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 3 op/s; 10440/32430 objects degraded (32.192%) 2022-05-13T23:24:42.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:24:42 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:24:42.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:24:42 smithi047 ceph-mon[77147]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2022-05-13T23:24:42.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:24:42 smithi115 ceph-mon[78593]: pgmap v201: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 139 MiB data, 473 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 3 op/s; 10440/32430 objects degraded (32.192%) 2022-05-13T23:24:42.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:24:42 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:24:42.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:24:42 smithi115 ceph-mon[78593]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2022-05-13T23:24:44.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:24:44 smithi115 ceph-mon[78593]: pgmap v202: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 139 MiB data, 473 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 3 op/s; 10440/32430 objects degraded (32.192%) 2022-05-13T23:24:44.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:24:44 smithi047 ceph-mon[77147]: pgmap v202: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 139 MiB data, 473 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 3 op/s; 10440/32430 objects degraded (32.192%) 2022-05-13T23:24:46.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:24:46 smithi047 ceph-mon[77147]: pgmap v203: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 139 MiB data, 473 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s; 10440/32430 objects degraded (32.192%) 2022-05-13T23:24:46.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:24:46 smithi047 ceph-mon[77147]: 1 slow requests, 0 included below; oldest blocked for > 449.081031 secs 2022-05-13T23:24:46.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:24:46 smithi115 ceph-mon[78593]: pgmap v203: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 139 MiB data, 473 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s; 10440/32430 objects degraded (32.192%) 2022-05-13T23:24:46.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:24:46 smithi115 ceph-mon[78593]: 1 slow requests, 0 included below; oldest blocked for > 449.081031 secs 2022-05-13T23:24:47.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:24:47 smithi047 ceph-mon[77147]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2022-05-13T23:24:47.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:24:47 smithi115 ceph-mon[78593]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2022-05-13T23:24:48.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:24:48 smithi047 ceph-mon[77147]: pgmap v204: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 139 MiB data, 473 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 3 op/s; 10440/32430 objects degraded (32.192%) 2022-05-13T23:24:48.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:24:48 smithi115 ceph-mon[78593]: pgmap v204: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 139 MiB data, 473 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 3 op/s; 10440/32430 objects degraded (32.192%) 2022-05-13T23:24:50.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:24:50 smithi047 ceph-mon[77147]: pgmap v205: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 139 MiB data, 473 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s; 10440/32430 objects degraded (32.192%) 2022-05-13T23:24:50.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:24:50 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:24:50.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:24:50 smithi047 ceph-mon[77147]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2022-05-13T23:24:50.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:24:50 smithi115 ceph-mon[78593]: pgmap v205: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 139 MiB data, 473 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s; 10440/32430 objects degraded (32.192%) 2022-05-13T23:24:50.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:24:50 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:24:50.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:24:50 smithi115 ceph-mon[78593]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2022-05-13T23:24:51.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:24:51 smithi047 ceph-mon[77147]: 1 slow requests, 0 included below; oldest blocked for > 454.081113 secs 2022-05-13T23:24:51.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:24:51 smithi115 ceph-mon[78593]: 1 slow requests, 0 included below; oldest blocked for > 454.081113 secs 2022-05-13T23:24:52.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:24:52 smithi047 ceph-mon[77147]: pgmap v206: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 139 MiB data, 473 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 3 op/s; 10440/32430 objects degraded (32.192%) 2022-05-13T23:24:52.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:24:52 smithi115 ceph-mon[78593]: pgmap v206: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 139 MiB data, 473 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 3 op/s; 10440/32430 objects degraded (32.192%) 2022-05-13T23:24:54.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:24:54 smithi047 ceph-mon[77147]: pgmap v207: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 139 MiB data, 473 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 3 op/s; 10440/32430 objects degraded (32.192%) 2022-05-13T23:24:54.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:24:54 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:24:54.545 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:24:54 smithi047 ceph-mon[77147]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2022-05-13T23:24:54.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:24:54 smithi115 ceph-mon[78593]: pgmap v207: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 139 MiB data, 473 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 3 op/s; 10440/32430 objects degraded (32.192%) 2022-05-13T23:24:54.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:24:54 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:24:54.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:24:54 smithi115 ceph-mon[78593]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2022-05-13T23:24:56.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:24:56 smithi047 ceph-mon[77147]: pgmap v208: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 139 MiB data, 473 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s; 10440/32430 objects degraded (32.192%) 2022-05-13T23:24:56.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:24:56 smithi047 ceph-mon[77147]: 1 slow requests, 0 included below; oldest blocked for > 459.081209 secs 2022-05-13T23:24:56.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:24:56 smithi115 ceph-mon[78593]: pgmap v208: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 139 MiB data, 473 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s; 10440/32430 objects degraded (32.192%) 2022-05-13T23:24:56.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:24:56 smithi115 ceph-mon[78593]: 1 slow requests, 0 included below; oldest blocked for > 459.081209 secs 2022-05-13T23:24:58.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:24:58 smithi047 ceph-mon[77147]: pgmap v209: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 139 MiB data, 473 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 3 op/s; 10440/32430 objects degraded (32.192%) 2022-05-13T23:24:58.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:24:58 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:24:58.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:24:58 smithi047 ceph-mon[77147]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2022-05-13T23:24:58.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:24:58 smithi115 ceph-mon[78593]: pgmap v209: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 139 MiB data, 473 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 3 op/s; 10440/32430 objects degraded (32.192%) 2022-05-13T23:24:58.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:24:58 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:24:58.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:24:58 smithi115 ceph-mon[78593]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2022-05-13T23:25:00.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:25:00 smithi047 ceph-mon[77147]: pgmap v210: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 139 MiB data, 473 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s; 10440/32430 objects degraded (32.192%) 2022-05-13T23:25:00.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:25:00 smithi115 ceph-mon[78593]: pgmap v210: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 139 MiB data, 473 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s; 10440/32430 objects degraded (32.192%) 2022-05-13T23:25:01.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:25:01 smithi047 ceph-mon[77147]: 1 slow requests, 0 included below; oldest blocked for > 464.081321 secs 2022-05-13T23:25:01.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:25:01 smithi115 ceph-mon[78593]: 1 slow requests, 0 included below; oldest blocked for > 464.081321 secs 2022-05-13T23:25:02.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:25:02 smithi047 ceph-mon[77147]: pgmap v211: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 139 MiB data, 473 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 3 op/s; 10440/32430 objects degraded (32.192%) 2022-05-13T23:25:02.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:25:02 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:25:02.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:25:02 smithi047 ceph-mon[77147]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2022-05-13T23:25:02.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:25:02 smithi115 ceph-mon[78593]: pgmap v211: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 139 MiB data, 473 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 3 op/s; 10440/32430 objects degraded (32.192%) 2022-05-13T23:25:02.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:25:02 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:25:02.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:25:02 smithi115 ceph-mon[78593]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2022-05-13T23:25:04.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:25:04 smithi115 ceph-mon[78593]: pgmap v212: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 139 MiB data, 473 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 3 op/s; 10440/32430 objects degraded (32.192%) 2022-05-13T23:25:04.498 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:25:04 smithi047 ceph-mon[77147]: pgmap v212: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 139 MiB data, 473 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 3 op/s; 10440/32430 objects degraded (32.192%) 2022-05-13T23:25:04.791 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:25:04 smithi047 conmon[78521]: 2022-05-13T23:25:04.496+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:25:04.792 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:25:04 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-13T23:25:04.792 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:25:04 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-13T23:25:04.792 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:25:04 smithi047 conmon[78521]: yield 2022-05-13T23:25:04.792 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:25:04 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-13T23:25:04.793 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:25:04 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-13T23:25:04.793 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:25:04 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-13T23:25:04.793 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:25:04 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-13T23:25:04.793 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:25:04 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-13T23:25:04.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:25:04 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-13T23:25:04.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:25:04 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-13T23:25:04.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:25:04 smithi047 conmon[78521]: raise exceptions[0] 2022-05-13T23:25:04.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:25:04 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-13T23:25:04.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:25:04 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-13T23:25:04.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:25:04 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-13T23:25:04.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:25:04 smithi047 conmon[78521]: return (yield from fut) 2022-05-13T23:25:04.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:25:04 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-13T23:25:04.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:25:04 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-13T23:25:04.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:25:04 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:25:05.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:25:05 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-13T23:25:05.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:25:05 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-13T23:25:05.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:25:05 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-13T23:25:05.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:25:05 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-13T23:25:05.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:25:05 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-13T23:25:05.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:25:05 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-13T23:25:06.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:25:06 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:25:06.359 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-13T23:25:06.359 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-13T23:25:06.359 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-13T23:25:06.359 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-13T23:25:06.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-13T23:25:06.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-13T23:25:06.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-13T23:25:06.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-13T23:25:06.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-13T23:25:06.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-13T23:25:06.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-13T23:25:06.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-13T23:25:06.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-13T23:25:06.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-13T23:25:06.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-13T23:25:06.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-13T23:25:06.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-13T23:25:06.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:25:06.363 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:25:06 smithi115 ceph-mon[78593]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-13T23:25:06.363 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:25:06 smithi115 ceph-mon[78593]: pgmap v213: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 139 MiB data, 473 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s; 10440/32430 objects degraded (32.192%) 2022-05-13T23:25:06.363 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:25:06 smithi115 ceph-mon[78593]: 1 slow requests, 0 included below; oldest blocked for > 469.081409 secs 2022-05-13T23:25:06.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:25:06 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:25:06.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-13T23:25:06.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-13T23:25:06.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-13T23:25:06.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-13T23:25:06.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-13T23:25:06.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-13T23:25:06.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-13T23:25:06.548 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-13T23:25:06.548 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-13T23:25:06.549 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-13T23:25:06.549 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-13T23:25:06.549 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-13T23:25:06.550 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-13T23:25:06.550 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-13T23:25:06.551 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-13T23:25:06.551 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-13T23:25:06.552 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-13T23:25:06.552 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:25:06.552 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:25:06 smithi047 ceph-mon[77147]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-13T23:25:06.553 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:25:06 smithi047 ceph-mon[77147]: pgmap v213: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 139 MiB data, 473 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s; 10440/32430 objects degraded (32.192%) 2022-05-13T23:25:06.553 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:25:06 smithi047 ceph-mon[77147]: 1 slow requests, 0 included below; oldest blocked for > 469.081409 secs 2022-05-13T23:25:08.042 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:25:07 smithi047 conmon[78521]: 2022-05-13T23:25:07.568+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:25:08.042 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:25:07 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-13T23:25:08.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:25:07 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-13T23:25:08.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:25:07 smithi047 conmon[78521]: yield 2022-05-13T23:25:08.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:25:07 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-13T23:25:08.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:25:07 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-13T23:25:08.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:25:07 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-13T23:25:08.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:25:07 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-13T23:25:08.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:25:07 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-13T23:25:08.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:25:07 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-13T23:25:08.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:25:07 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-13T23:25:08.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:25:07 smithi047 conmon[78521]: raise exceptions[0] 2022-05-13T23:25:08.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:25:07 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-13T23:25:08.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:25:07 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-13T23:25:08.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:25:07 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-13T23:25:08.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:25:07 smithi047 conmon[78521]: return (yield from fut) 2022-05-13T23:25:08.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:25:07 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-13T23:25:08.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:25:07 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-13T23:25:08.048 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:25:07 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:25:08.048 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:25:07 smithi047 conmon[78521]: 2022-05-13T23:25:07.569+0000 7f533ef76700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-13T23:25:08.049 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:25:07 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-13T23:25:08.050 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:25:07 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-13T23:25:08.052 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:25:07 smithi047 conmon[78521]: yield 2022-05-13T23:25:08.052 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:25:07 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-13T23:25:08.053 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:25:07 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-13T23:25:08.053 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:25:07 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-13T23:25:08.053 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:25:07 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-13T23:25:08.054 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:25:07 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-13T23:25:08.054 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:25:07 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-13T23:25:08.054 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:25:07 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-13T23:25:08.055 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:25:07 smithi047 conmon[78521]: raise exceptions[0] 2022-05-13T23:25:08.055 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:25:07 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-13T23:25:08.056 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:25:07 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-13T23:25:08.056 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:25:07 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-13T23:25:08.057 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:25:07 smithi047 conmon[78521]: return (yield from fut) 2022-05-13T23:25:08.057 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:25:07 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-13T23:25:08.057 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:25:07 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-13T23:25:08.058 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:25:07 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:25:08.059 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:25:07 smithi047 conmon[78521]: 2022-05-13T23:25:08.060 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:25:07 smithi047 conmon[78521]: During handling of the above exception, another exception occurred: 2022-05-13T23:25:08.060 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:25:07 smithi047 conmon[78521]: 2022-05-13T23:25:08.060 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:25:07 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-13T23:25:08.061 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:25:07 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-13T23:25:08.061 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:25:07 smithi047 conmon[78521]: return f(*arg) 2022-05-13T23:25:08.062 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:25:07 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-13T23:25:08.062 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:25:07 smithi047 conmon[78521]: self._write_client_files(client_files, host) 2022-05-13T23:25:08.062 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:25:07 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-13T23:25:08.176 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:25:07 smithi047 conmon[78521]: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-13T23:25:08.177 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:25:07 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-13T23:25:08.177 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:25:07 smithi047 conmon[78521]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-13T23:25:08.178 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:25:07 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-13T23:25:08.178 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:25:07 smithi047 conmon[78521]: return self.event_loop.get_result(coro) 2022-05-13T23:25:08.178 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:25:07 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-13T23:25:08.178 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:25:07 smithi047 conmon[78521]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-13T23:25:08.179 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:25:07 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-13T23:25:08.179 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:25:07 smithi047 conmon[78521]: return self.__get_result() 2022-05-13T23:25:08.180 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:25:07 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-13T23:25:08.180 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:25:07 smithi047 conmon[78521]: raise self._exception 2022-05-13T23:25:08.181 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:25:07 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-13T23:25:08.183 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:25:07 smithi047 conmon[78521]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-13T23:25:08.183 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:25:07 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-13T23:25:08.183 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:25:07 smithi047 conmon[78521]: conn = await self._remote_connection(host, addr) 2022-05-13T23:25:08.184 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:25:07 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-13T23:25:08.184 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:25:07 smithi047 conmon[78521]: raise 2022-05-13T23:25:08.184 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:25:07 smithi047 conmon[78521]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-13T23:25:08.184 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:25:07 smithi047 conmon[78521]: self.gen.throw(type, value, traceback) 2022-05-13T23:25:08.185 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:25:07 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-13T23:25:08.185 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:25:07 smithi047 conmon[78521]: raise OrchestratorError(msg) 2022-05-13T23:25:08.185 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:25:07 smithi047 conmon[78521]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:25:08.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:25:08 smithi047 ceph-mon[77147]: pgmap v214: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 139 MiB data, 473 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 3 op/s; 10440/32430 objects degraded (32.192%) 2022-05-13T23:25:08.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:25:08 smithi115 ceph-mon[78593]: pgmap v214: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 139 MiB data, 473 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 3 op/s; 10440/32430 objects degraded (32.192%) 2022-05-13T23:25:09.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:25:09 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:25:09.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-13T23:25:09.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-13T23:25:09.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-13T23:25:09.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-13T23:25:09.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-13T23:25:09.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-13T23:25:09.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-13T23:25:09.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-13T23:25:09.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-13T23:25:09.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-13T23:25:09.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-13T23:25:09.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-13T23:25:09.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-13T23:25:09.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-13T23:25:09.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-13T23:25:09.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-13T23:25:09.548 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-13T23:25:09.548 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:25:09.548 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:25:09 smithi047 ceph-mon[77147]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-13T23:25:09.548 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-13T23:25:09.549 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-13T23:25:09.549 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-13T23:25:09.549 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-13T23:25:09.549 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-13T23:25:09.550 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-13T23:25:09.550 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-13T23:25:09.550 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-13T23:25:09.550 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-13T23:25:09.551 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-13T23:25:09.551 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-13T23:25:09.551 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-13T23:25:09.551 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-13T23:25:09.552 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-13T23:25:09.552 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-13T23:25:09.552 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-13T23:25:09.553 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-13T23:25:09.553 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:25:09.556 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-13T23:25:09.557 INFO:journalctl@ceph.mon.b.smithi047.stdout: During handling of the above exception, another exception occurred: 2022-05-13T23:25:09.557 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-13T23:25:09.557 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-13T23:25:09.557 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-13T23:25:09.558 INFO:journalctl@ceph.mon.b.smithi047.stdout: return f(*arg) 2022-05-13T23:25:09.558 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-13T23:25:09.558 INFO:journalctl@ceph.mon.b.smithi047.stdout: self._write_client_files(client_files, host) 2022-05-13T23:25:09.558 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-13T23:25:09.558 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-13T23:25:09.559 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-13T23:25:09.559 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-13T23:25:09.559 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-13T23:25:09.559 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.event_loop.get_result(coro) 2022-05-13T23:25:09.559 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-13T23:25:09.560 INFO:journalctl@ceph.mon.b.smithi047.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-13T23:25:09.560 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-13T23:25:09.560 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.__get_result() 2022-05-13T23:25:09.560 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-13T23:25:09.560 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise self._exception 2022-05-13T23:25:09.561 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-13T23:25:09.561 INFO:journalctl@ceph.mon.b.smithi047.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-13T23:25:09.561 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-13T23:25:09.561 INFO:journalctl@ceph.mon.b.smithi047.stdout: conn = await self._remote_connection(host, addr) 2022-05-13T23:25:09.562 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-13T23:25:09.562 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise 2022-05-13T23:25:09.562 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-13T23:25:09.562 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.gen.throw(type, value, traceback) 2022-05-13T23:25:09.562 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-13T23:25:09.563 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OrchestratorError(msg) 2022-05-13T23:25:09.563 INFO:journalctl@ceph.mon.b.smithi047.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:25:09.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:25:09 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:25:09.609 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-13T23:25:09.609 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-13T23:25:09.609 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-13T23:25:09.609 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-13T23:25:09.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-13T23:25:09.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-13T23:25:09.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-13T23:25:09.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-13T23:25:09.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-13T23:25:09.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-13T23:25:09.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-13T23:25:09.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-13T23:25:09.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-13T23:25:09.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-13T23:25:09.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-13T23:25:09.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-13T23:25:09.613 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-13T23:25:09.613 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:25:09.613 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:25:09 smithi115 ceph-mon[78593]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-13T23:25:09.613 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-13T23:25:09.614 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-13T23:25:09.614 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-13T23:25:09.614 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-13T23:25:09.614 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-13T23:25:09.615 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-13T23:25:09.615 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-13T23:25:09.615 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-13T23:25:09.615 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-13T23:25:09.616 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-13T23:25:09.616 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-13T23:25:09.616 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-13T23:25:09.616 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-13T23:25:09.617 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-13T23:25:09.617 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-13T23:25:09.617 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-13T23:25:09.618 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-13T23:25:09.618 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:25:09.619 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-13T23:25:09.619 INFO:journalctl@ceph.mon.c.smithi115.stdout: During handling of the above exception, another exception occurred: 2022-05-13T23:25:09.619 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-13T23:25:09.620 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-13T23:25:09.620 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-13T23:25:09.620 INFO:journalctl@ceph.mon.c.smithi115.stdout: return f(*arg) 2022-05-13T23:25:09.620 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-13T23:25:09.620 INFO:journalctl@ceph.mon.c.smithi115.stdout: self._write_client_files(client_files, host) 2022-05-13T23:25:09.621 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-13T23:25:09.621 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-13T23:25:09.622 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-13T23:25:09.622 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-13T23:25:09.622 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-13T23:25:09.623 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.event_loop.get_result(coro) 2022-05-13T23:25:09.623 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-13T23:25:09.623 INFO:journalctl@ceph.mon.c.smithi115.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-13T23:25:09.623 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-13T23:25:09.624 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.__get_result() 2022-05-13T23:25:09.624 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-13T23:25:09.624 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise self._exception 2022-05-13T23:25:09.624 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-13T23:25:09.625 INFO:journalctl@ceph.mon.c.smithi115.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-13T23:25:09.625 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-13T23:25:09.625 INFO:journalctl@ceph.mon.c.smithi115.stdout: conn = await self._remote_connection(host, addr) 2022-05-13T23:25:09.625 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-13T23:25:09.625 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise 2022-05-13T23:25:09.626 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-13T23:25:09.626 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.gen.throw(type, value, traceback) 2022-05-13T23:25:09.626 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-13T23:25:09.626 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OrchestratorError(msg) 2022-05-13T23:25:09.627 INFO:journalctl@ceph.mon.c.smithi115.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:25:10.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:25:10 smithi047 ceph-mon[77147]: pgmap v215: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 139 MiB data, 473 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s; 10440/32430 objects degraded (32.192%) 2022-05-13T23:25:10.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:25:10 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:25:10.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:25:10 smithi047 ceph-mon[77147]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2022-05-13T23:25:10.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:25:10 smithi115 ceph-mon[78593]: pgmap v215: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 139 MiB data, 473 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s; 10440/32430 objects degraded (32.192%) 2022-05-13T23:25:10.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:25:10 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:25:10.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:25:10 smithi115 ceph-mon[78593]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2022-05-13T23:25:11.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:25:11 smithi047 ceph-mon[77147]: 1 slow requests, 0 included below; oldest blocked for > 474.081497 secs 2022-05-13T23:25:11.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:25:11 smithi115 ceph-mon[78593]: 1 slow requests, 0 included below; oldest blocked for > 474.081497 secs 2022-05-13T23:25:12.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:25:12 smithi047 ceph-mon[77147]: pgmap v216: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 139 MiB data, 473 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 3 op/s; 10440/32430 objects degraded (32.192%) 2022-05-13T23:25:12.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:25:12 smithi115 ceph-mon[78593]: pgmap v216: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 139 MiB data, 473 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 3 op/s; 10440/32430 objects degraded (32.192%) 2022-05-13T23:25:14.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:25:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-13T23:25:14.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:25:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-13T23:25:14.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:25:14 smithi115 ceph-mon[78593]: pgmap v217: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 139 MiB data, 473 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 3 op/s; 10440/32430 objects degraded (32.192%) 2022-05-13T23:25:14.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:25:14 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:25:14.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:25:14 smithi115 ceph-mon[78593]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2022-05-13T23:25:14.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:25:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-13T23:25:14.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:25:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-13T23:25:14.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:25:14 smithi047 ceph-mon[77147]: pgmap v217: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 139 MiB data, 473 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 3 op/s; 10440/32430 objects degraded (32.192%) 2022-05-13T23:25:14.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:25:14 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:25:14.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:25:14 smithi047 ceph-mon[77147]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2022-05-13T23:25:16.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:25:16 smithi047 ceph-mon[77147]: pgmap v218: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 139 MiB data, 473 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s; 10440/32430 objects degraded (32.192%) 2022-05-13T23:25:16.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:25:16 smithi047 ceph-mon[77147]: 1 slow requests, 0 included below; oldest blocked for > 479.081620 secs 2022-05-13T23:25:16.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:25:16 smithi115 ceph-mon[78593]: pgmap v218: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 139 MiB data, 473 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s; 10440/32430 objects degraded (32.192%) 2022-05-13T23:25:16.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:25:16 smithi115 ceph-mon[78593]: 1 slow requests, 0 included below; oldest blocked for > 479.081620 secs 2022-05-13T23:25:18.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:25:18 smithi047 ceph-mon[77147]: pgmap v219: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 139 MiB data, 473 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 3 op/s; 10440/32430 objects degraded (32.192%) 2022-05-13T23:25:18.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:25:18 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:25:18.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:25:18 smithi047 ceph-mon[77147]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2022-05-13T23:25:18.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:25:18 smithi115 ceph-mon[78593]: pgmap v219: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 139 MiB data, 473 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 3 op/s; 10440/32430 objects degraded (32.192%) 2022-05-13T23:25:18.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:25:18 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:25:18.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:25:18 smithi115 ceph-mon[78593]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2022-05-13T23:25:20.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:25:20 smithi047 ceph-mon[77147]: pgmap v220: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 139 MiB data, 473 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s; 10440/32430 objects degraded (32.192%) 2022-05-13T23:25:20.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:25:20 smithi115 ceph-mon[78593]: pgmap v220: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 139 MiB data, 473 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s; 10440/32430 objects degraded (32.192%) 2022-05-13T23:25:21.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:25:21 smithi047 ceph-mon[77147]: 1 slow requests, 0 included below; oldest blocked for > 484.081750 secs 2022-05-13T23:25:21.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:25:21 smithi115 ceph-mon[78593]: 1 slow requests, 0 included below; oldest blocked for > 484.081750 secs 2022-05-13T23:25:22.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:25:22 smithi047 ceph-mon[77147]: pgmap v221: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 139 MiB data, 473 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 3 op/s; 10440/32430 objects degraded (32.192%) 2022-05-13T23:25:22.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:25:22 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:25:22.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:25:22 smithi047 ceph-mon[77147]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2022-05-13T23:25:22.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:25:22 smithi115 ceph-mon[78593]: pgmap v221: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 139 MiB data, 473 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 3 op/s; 10440/32430 objects degraded (32.192%) 2022-05-13T23:25:22.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:25:22 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:25:22.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:25:22 smithi115 ceph-mon[78593]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2022-05-13T23:25:24.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:25:24 smithi115 ceph-mon[78593]: pgmap v222: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 139 MiB data, 473 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 3 op/s; 10440/32430 objects degraded (32.192%) 2022-05-13T23:25:24.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:25:24 smithi047 ceph-mon[77147]: pgmap v222: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 139 MiB data, 473 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 3 op/s; 10440/32430 objects degraded (32.192%) 2022-05-13T23:25:26.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:25:26 smithi047 ceph-mon[77147]: pgmap v223: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 139 MiB data, 473 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s; 10440/32430 objects degraded (32.192%) 2022-05-13T23:25:26.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:25:26 smithi047 ceph-mon[77147]: 1 slow requests, 0 included below; oldest blocked for > 489.081856 secs 2022-05-13T23:25:26.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:25:26 smithi115 ceph-mon[78593]: pgmap v223: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 139 MiB data, 473 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s; 10440/32430 objects degraded (32.192%) 2022-05-13T23:25:26.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:25:26 smithi115 ceph-mon[78593]: 1 slow requests, 0 included below; oldest blocked for > 489.081856 secs 2022-05-13T23:25:28.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:25:28 smithi047 ceph-mon[77147]: pgmap v224: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 139 MiB data, 473 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 3 op/s; 10440/32430 objects degraded (32.192%) 2022-05-13T23:25:28.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:25:28 smithi115 ceph-mon[78593]: pgmap v224: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 139 MiB data, 473 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 3 op/s; 10440/32430 objects degraded (32.192%) 2022-05-13T23:25:30.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:25:30 smithi047 ceph-mon[77147]: pgmap v225: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 139 MiB data, 473 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s; 10440/32430 objects degraded (32.192%) 2022-05-13T23:25:30.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:25:30 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:25:30.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:25:30 smithi047 ceph-mon[77147]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2022-05-13T23:25:30.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:25:30 smithi115 ceph-mon[78593]: pgmap v225: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 139 MiB data, 473 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s; 10440/32430 objects degraded (32.192%) 2022-05-13T23:25:30.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:25:30 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:25:30.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:25:30 smithi115 ceph-mon[78593]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2022-05-13T23:25:31.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:25:31 smithi047 ceph-mon[77147]: 1 slow requests, 0 included below; oldest blocked for > 494.081945 secs 2022-05-13T23:25:31.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:25:31 smithi115 ceph-mon[78593]: 1 slow requests, 0 included below; oldest blocked for > 494.081945 secs 2022-05-13T23:25:32.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:25:32 smithi047 ceph-mon[77147]: pgmap v226: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 139 MiB data, 473 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 3 op/s; 10440/32430 objects degraded (32.192%) 2022-05-13T23:25:32.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:25:32 smithi115 ceph-mon[78593]: pgmap v226: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 139 MiB data, 473 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 3 op/s; 10440/32430 objects degraded (32.192%) 2022-05-13T23:25:34.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:25:34 smithi047 ceph-mon[77147]: pgmap v227: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 139 MiB data, 473 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 3 op/s; 10440/32430 objects degraded (32.192%) 2022-05-13T23:25:34.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:25:34 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:25:34.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:25:34 smithi047 ceph-mon[77147]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2022-05-13T23:25:34.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:25:34 smithi115 ceph-mon[78593]: pgmap v227: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 139 MiB data, 473 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 3 op/s; 10440/32430 objects degraded (32.192%) 2022-05-13T23:25:34.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:25:34 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:25:34.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:25:34 smithi115 ceph-mon[78593]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2022-05-13T23:25:36.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:25:36 smithi047 ceph-mon[77147]: pgmap v228: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 139 MiB data, 473 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s; 10440/32430 objects degraded (32.192%) 2022-05-13T23:25:36.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:25:36 smithi047 ceph-mon[77147]: 1 slow requests, 0 included below; oldest blocked for > 499.082043 secs 2022-05-13T23:25:36.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:25:36 smithi047 ceph-mon[77147]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2022-05-13T23:25:36.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:25:36 smithi115 ceph-mon[78593]: pgmap v228: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 139 MiB data, 473 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s; 10440/32430 objects degraded (32.192%) 2022-05-13T23:25:36.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:25:36 smithi115 ceph-mon[78593]: 1 slow requests, 0 included below; oldest blocked for > 499.082043 secs 2022-05-13T23:25:36.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:25:36 smithi115 ceph-mon[78593]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2022-05-13T23:25:38.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:25:38 smithi047 ceph-mon[77147]: pgmap v229: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 139 MiB data, 473 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 3 op/s; 10440/32430 objects degraded (32.192%) 2022-05-13T23:25:38.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:25:38 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:25:38.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:25:38 smithi047 ceph-mon[77147]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2022-05-13T23:25:38.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:25:38 smithi115 ceph-mon[78593]: pgmap v229: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 139 MiB data, 473 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 3 op/s; 10440/32430 objects degraded (32.192%) 2022-05-13T23:25:38.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:25:38 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:25:38.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:25:38 smithi115 ceph-mon[78593]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2022-05-13T23:25:40.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:25:40 smithi047 ceph-mon[77147]: pgmap v230: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 139 MiB data, 473 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s; 10440/32430 objects degraded (32.192%) 2022-05-13T23:25:40.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:25:40 smithi115 ceph-mon[78593]: pgmap v230: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 139 MiB data, 473 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s; 10440/32430 objects degraded (32.192%) 2022-05-13T23:25:41.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:25:41 smithi047 ceph-mon[77147]: 1 slow requests, 0 included below; oldest blocked for > 504.082163 secs 2022-05-13T23:25:41.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:25:41 smithi115 ceph-mon[78593]: 1 slow requests, 0 included below; oldest blocked for > 504.082163 secs 2022-05-13T23:25:42.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:25:42 smithi047 ceph-mon[77147]: pgmap v231: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 139 MiB data, 473 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 3 op/s; 10440/32430 objects degraded (32.192%) 2022-05-13T23:25:42.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:25:42 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:25:42.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:25:42 smithi047 ceph-mon[77147]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2022-05-13T23:25:42.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:25:42 smithi115 ceph-mon[78593]: pgmap v231: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 139 MiB data, 473 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 3 op/s; 10440/32430 objects degraded (32.192%) 2022-05-13T23:25:42.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:25:42 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:25:42.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:25:42 smithi115 ceph-mon[78593]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2022-05-13T23:25:44.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:25:44 smithi115 ceph-mon[78593]: pgmap v232: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 139 MiB data, 473 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 3 op/s; 10440/32430 objects degraded (32.192%) 2022-05-13T23:25:44.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:25:44 smithi047 ceph-mon[77147]: pgmap v232: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 139 MiB data, 473 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 3 op/s; 10440/32430 objects degraded (32.192%) 2022-05-13T23:25:46.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:25:46 smithi047 ceph-mon[77147]: pgmap v233: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 139 MiB data, 473 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s; 10440/32430 objects degraded (32.192%) 2022-05-13T23:25:46.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:25:46 smithi047 ceph-mon[77147]: 1 slow requests, 0 included below; oldest blocked for > 509.082259 secs 2022-05-13T23:25:46.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:25:46 smithi115 ceph-mon[78593]: pgmap v233: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 139 MiB data, 473 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s; 10440/32430 objects degraded (32.192%) 2022-05-13T23:25:46.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:25:46 smithi115 ceph-mon[78593]: 1 slow requests, 0 included below; oldest blocked for > 509.082259 secs 2022-05-13T23:25:48.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:25:48 smithi047 ceph-mon[77147]: pgmap v234: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 139 MiB data, 473 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 3 op/s; 10440/32430 objects degraded (32.192%) 2022-05-13T23:25:48.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:25:48 smithi047 ceph-mon[77147]: 1 slow requests, 1 included below; oldest blocked for > 181.103267 secs 2022-05-13T23:25:48.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:25:48 smithi047 ceph-mon[77147]: slow request 181.103266 seconds old, received at 2022-05-13T23:22:46.765072+0000: internal op fragmentdir:mds.1:1 currently dispatched 2022-05-13T23:25:48.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:25:48 smithi115 ceph-mon[78593]: pgmap v234: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 139 MiB data, 473 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 3 op/s; 10440/32430 objects degraded (32.192%) 2022-05-13T23:25:48.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:25:48 smithi115 ceph-mon[78593]: 1 slow requests, 1 included below; oldest blocked for > 181.103267 secs 2022-05-13T23:25:48.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:25:48 smithi115 ceph-mon[78593]: slow request 181.103266 seconds old, received at 2022-05-13T23:22:46.765072+0000: internal op fragmentdir:mds.1:1 currently dispatched 2022-05-13T23:25:50.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:25:50 smithi047 ceph-mon[77147]: pgmap v235: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 139 MiB data, 473 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s; 10440/32430 objects degraded (32.192%) 2022-05-13T23:25:50.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:25:50 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:25:50.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:25:50 smithi047 ceph-mon[77147]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2022-05-13T23:25:50.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:25:50 smithi115 ceph-mon[78593]: pgmap v235: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 139 MiB data, 473 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s; 10440/32430 objects degraded (32.192%) 2022-05-13T23:25:50.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:25:50 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:25:50.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:25:50 smithi115 ceph-mon[78593]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2022-05-13T23:25:51.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:25:51 smithi047 ceph-mon[77147]: 1 slow requests, 0 included below; oldest blocked for > 514.082335 secs 2022-05-13T23:25:51.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:25:51 smithi115 ceph-mon[78593]: 1 slow requests, 0 included below; oldest blocked for > 514.082335 secs 2022-05-13T23:25:52.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:25:52 smithi047 ceph-mon[77147]: pgmap v236: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 139 MiB data, 473 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 3 op/s; 10440/32430 objects degraded (32.192%) 2022-05-13T23:25:52.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:25:52 smithi115 ceph-mon[78593]: pgmap v236: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 139 MiB data, 473 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 3 op/s; 10440/32430 objects degraded (32.192%) 2022-05-13T23:25:53.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:25:53 smithi047 ceph-mon[77147]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2022-05-13T23:25:53.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:25:53 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6836/2940957881,v1:172.21.15.115:6837/2940957881] up:active 2022-05-13T23:25:53.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:25:53 smithi047 ceph-mon[77147]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2022-05-13T23:25:53.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:25:53 smithi047 ceph-mon[77147]: 1 slow requests, 0 included below; oldest blocked for > 186.103388 secs 2022-05-13T23:25:53.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:25:53 smithi115 ceph-mon[78593]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2022-05-13T23:25:53.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:25:53 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6836/2940957881,v1:172.21.15.115:6837/2940957881] up:active 2022-05-13T23:25:53.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:25:53 smithi115 ceph-mon[78593]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2022-05-13T23:25:53.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:25:53 smithi115 ceph-mon[78593]: 1 slow requests, 0 included below; oldest blocked for > 186.103388 secs 2022-05-13T23:25:54.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:25:54 smithi047 ceph-mon[77147]: pgmap v237: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 139 MiB data, 473 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 3 op/s; 10440/32430 objects degraded (32.192%) 2022-05-13T23:25:54.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:25:54 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:25:54.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:25:54 smithi047 ceph-mon[77147]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2022-05-13T23:25:54.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:25:54 smithi115 ceph-mon[78593]: pgmap v237: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 139 MiB data, 473 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 3 op/s; 10440/32430 objects degraded (32.192%) 2022-05-13T23:25:54.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:25:54 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:25:54.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:25:54 smithi115 ceph-mon[78593]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2022-05-13T23:25:56.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:25:56 smithi047 ceph-mon[77147]: pgmap v238: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 139 MiB data, 473 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s; 10440/32430 objects degraded (32.192%) 2022-05-13T23:25:56.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:25:56 smithi047 ceph-mon[77147]: 1 slow requests, 0 included below; oldest blocked for > 519.082456 secs 2022-05-13T23:25:56.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:25:56 smithi115 ceph-mon[78593]: pgmap v238: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 139 MiB data, 473 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s; 10440/32430 objects degraded (32.192%) 2022-05-13T23:25:56.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:25:56 smithi115 ceph-mon[78593]: 1 slow requests, 0 included below; oldest blocked for > 519.082456 secs 2022-05-13T23:25:58.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:25:58 smithi047 ceph-mon[77147]: pgmap v239: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 139 MiB data, 473 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 3 op/s; 10440/32430 objects degraded (32.192%) 2022-05-13T23:25:58.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:25:58 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:25:58.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:25:58 smithi047 ceph-mon[77147]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2022-05-13T23:25:58.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:25:58 smithi047 ceph-mon[77147]: 1 slow requests, 0 included below; oldest blocked for > 191.103522 secs 2022-05-13T23:25:58.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:25:58 smithi115 ceph-mon[78593]: pgmap v239: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 139 MiB data, 473 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 3 op/s; 10440/32430 objects degraded (32.192%) 2022-05-13T23:25:58.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:25:58 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:25:58.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:25:58 smithi115 ceph-mon[78593]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2022-05-13T23:25:58.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:25:58 smithi115 ceph-mon[78593]: 1 slow requests, 0 included below; oldest blocked for > 191.103522 secs 2022-05-13T23:26:00.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:26:00 smithi047 ceph-mon[77147]: pgmap v240: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 139 MiB data, 473 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s; 10440/32430 objects degraded (32.192%) 2022-05-13T23:26:00.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:26:00 smithi115 ceph-mon[78593]: pgmap v240: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 139 MiB data, 473 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s; 10440/32430 objects degraded (32.192%) 2022-05-13T23:26:01.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:26:01 smithi047 ceph-mon[77147]: 1 slow requests, 0 included below; oldest blocked for > 524.082546 secs 2022-05-13T23:26:01.558 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:26:01 smithi115 ceph-mon[78593]: 1 slow requests, 0 included below; oldest blocked for > 524.082546 secs 2022-05-13T23:26:02.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:26:02 smithi047 ceph-mon[77147]: pgmap v241: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 139 MiB data, 473 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 3 op/s; 10440/32430 objects degraded (32.192%) 2022-05-13T23:26:02.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:26:02 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:26:02.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:26:02 smithi047 ceph-mon[77147]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2022-05-13T23:26:02.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:26:02 smithi115 ceph-mon[78593]: pgmap v241: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 139 MiB data, 473 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 3 op/s; 10440/32430 objects degraded (32.192%) 2022-05-13T23:26:02.612 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:26:02 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:26:02.612 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:26:02 smithi115 ceph-mon[78593]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2022-05-13T23:26:03.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:26:03 smithi047 ceph-mon[77147]: 1 slow requests, 0 included below; oldest blocked for > 196.103684 secs 2022-05-13T23:26:03.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:26:03 smithi115 ceph-mon[78593]: 1 slow requests, 0 included below; oldest blocked for > 196.103684 secs 2022-05-13T23:26:04.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:26:04 smithi047 ceph-mon[77147]: pgmap v242: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 139 MiB data, 473 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 3 op/s; 10440/32430 objects degraded (32.192%) 2022-05-13T23:26:04.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:26:04 smithi115 ceph-mon[78593]: pgmap v242: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 139 MiB data, 473 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 3 op/s; 10440/32430 objects degraded (32.192%) 2022-05-13T23:26:06.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:26:06 smithi115 ceph-mon[78593]: pgmap v243: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 139 MiB data, 473 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s; 10440/32430 objects degraded (32.192%) 2022-05-13T23:26:06.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:26:06 smithi115 ceph-mon[78593]: 1 slow requests, 0 included below; oldest blocked for > 529.082664 secs 2022-05-13T23:26:06.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:26:06 smithi047 ceph-mon[77147]: pgmap v243: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 139 MiB data, 473 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s; 10440/32430 objects degraded (32.192%) 2022-05-13T23:26:06.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:26:06 smithi047 ceph-mon[77147]: 1 slow requests, 0 included below; oldest blocked for > 529.082664 secs 2022-05-13T23:26:08.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:26:08 smithi115 ceph-mon[78593]: pgmap v244: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 139 MiB data, 473 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 3 op/s; 10440/32430 objects degraded (32.192%) 2022-05-13T23:26:08.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:26:08 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-13T23:26:08.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:26:08 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-13T23:26:08.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:26:08 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-13T23:26:08.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:26:08 smithi115 ceph-mon[78593]: 1 slow requests, 0 included below; oldest blocked for > 201.103782 secs 2022-05-13T23:26:08.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:26:08 smithi047 ceph-mon[77147]: pgmap v244: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 139 MiB data, 473 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 3 op/s; 10440/32430 objects degraded (32.192%) 2022-05-13T23:26:08.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:26:08 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-13T23:26:08.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:26:08 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-13T23:26:08.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:26:08 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-13T23:26:08.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:26:08 smithi047 ceph-mon[77147]: 1 slow requests, 0 included below; oldest blocked for > 201.103782 secs 2022-05-13T23:26:09.291 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:26:09 smithi047 conmon[78521]: 2022-05-13T23:26:09.007+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:26:09.292 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:26:09 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-13T23:26:09.292 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:26:09 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-13T23:26:09.292 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:26:09 smithi047 conmon[78521]: yield 2022-05-13T23:26:09.293 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:26:09 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-13T23:26:09.293 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:26:09 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-13T23:26:09.293 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:26:09 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-13T23:26:09.293 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:26:09 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-13T23:26:09.293 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:26:09 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-13T23:26:09.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:26:09 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-13T23:26:09.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:26:09 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-13T23:26:09.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:26:09 smithi047 conmon[78521]: raise exceptions[0] 2022-05-13T23:26:09.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:26:09 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-13T23:26:09.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:26:09 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-13T23:26:09.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:26:09 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-13T23:26:09.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:26:09 smithi047 conmon[78521]: return (yield from fut) 2022-05-13T23:26:09.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:26:09 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-13T23:26:09.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:26:09 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-13T23:26:09.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:26:09 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:26:10.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:26:10 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:26:10.608 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-13T23:26:10.609 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-13T23:26:10.609 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-13T23:26:10.609 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-13T23:26:10.609 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-13T23:26:10.609 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-13T23:26:10.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-13T23:26:10.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-13T23:26:10.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-13T23:26:10.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-13T23:26:10.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-13T23:26:10.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-13T23:26:10.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-13T23:26:10.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-13T23:26:10.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-13T23:26:10.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-13T23:26:10.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-13T23:26:10.614 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:26:10.614 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:26:10 smithi115 ceph-mon[78593]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-13T23:26:10.614 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:26:10 smithi115 ceph-mon[78593]: pgmap v245: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 139 MiB data, 473 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s; 10440/32430 objects degraded (32.192%) 2022-05-13T23:26:10.614 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:26:10 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:26:10.615 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:26:10 smithi115 ceph-mon[78593]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2022-05-13T23:26:10.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:26:10 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:26:10.792 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-13T23:26:10.792 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-13T23:26:10.792 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-13T23:26:10.792 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-13T23:26:10.793 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-13T23:26:10.793 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-13T23:26:10.793 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-13T23:26:10.793 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-13T23:26:10.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-13T23:26:10.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-13T23:26:10.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-13T23:26:10.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-13T23:26:10.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-13T23:26:10.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-13T23:26:10.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-13T23:26:10.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-13T23:26:10.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-13T23:26:10.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:26:10.796 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:26:10 smithi047 ceph-mon[77147]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-13T23:26:10.796 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:26:10 smithi047 ceph-mon[77147]: pgmap v245: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 139 MiB data, 473 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s; 10440/32430 objects degraded (32.192%) 2022-05-13T23:26:10.796 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:26:10 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:26:10.797 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:26:10 smithi047 ceph-mon[77147]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2022-05-13T23:26:11.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:26:11 smithi115 ceph-mon[78593]: 1 slow requests, 0 included below; oldest blocked for > 534.082764 secs 2022-05-13T23:26:11.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:26:11 smithi047 ceph-mon[77147]: 1 slow requests, 0 included below; oldest blocked for > 534.082764 secs 2022-05-13T23:26:12.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:26:12 smithi047 ceph-mon[77147]: pgmap v246: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 139 MiB data, 473 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 3 op/s; 10440/32430 objects degraded (32.192%) 2022-05-13T23:26:12.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:26:12 smithi047 conmon[78521]: 2022-05-13T23:26:12.079+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:26:12.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:26:12 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-13T23:26:12.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:26:12 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-13T23:26:12.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:26:12 smithi047 conmon[78521]: yield 2022-05-13T23:26:12.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:26:12 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-13T23:26:12.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:26:12 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-13T23:26:12.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:26:12 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-13T23:26:12.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:26:12 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-13T23:26:12.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:26:12 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-13T23:26:12.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:26:12 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-13T23:26:12.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:26:12 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-13T23:26:12.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:26:12 smithi047 conmon[78521]: raise exceptions[0] 2022-05-13T23:26:12.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:26:12 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-13T23:26:12.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:26:12 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-13T23:26:12.547 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:26:12 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-13T23:26:12.547 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:26:12 smithi047 conmon[78521]: return (yield from fut) 2022-05-13T23:26:12.547 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:26:12 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-13T23:26:12.547 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:26:12 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-13T23:26:12.547 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:26:12 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:26:12.548 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:26:12 smithi047 conmon[78521]: 2022-05-13T23:26:12.080+0000 7f5345f84700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-13T23:26:12.548 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:26:12 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-13T23:26:12.548 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:26:12 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-13T23:26:12.548 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:26:12 smithi047 conmon[78521]: yield 2022-05-13T23:26:12.549 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:26:12 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-13T23:26:12.549 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:26:12 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-13T23:26:12.549 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:26:12 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-13T23:26:12.549 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:26:12 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-13T23:26:12.549 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:26:12 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-13T23:26:12.550 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:26:12 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-13T23:26:12.550 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:26:12 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-13T23:26:12.550 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:26:12 smithi047 conmon[78521]: raise exceptions[0] 2022-05-13T23:26:12.550 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:26:12 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-13T23:26:12.551 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:26:12 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-13T23:26:12.551 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:26:12 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-13T23:26:12.551 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:26:12 smithi047 conmon[78521]: return (yield from fut) 2022-05-13T23:26:12.551 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:26:12 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-13T23:26:12.551 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:26:12 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-13T23:26:12.552 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:26:12 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:26:12.552 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:26:12 smithi047 conmon[78521]: 2022-05-13T23:26:12.552 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:26:12 smithi047 conmon[78521]: During handling of the above exception, another exception occurred: 2022-05-13T23:26:12.552 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:26:12 smithi047 conmon[78521]: 2022-05-13T23:26:12.552 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:26:12 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-13T23:26:12.553 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:26:12 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-13T23:26:12.553 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:26:12 smithi047 conmon[78521]: return f(*arg) 2022-05-13T23:26:12.553 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:26:12 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-13T23:26:12.553 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:26:12 smithi047 conmon[78521]: self._write_client_files(client_files, host) 2022-05-13T23:26:12.553 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:26:12 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-13T23:26:12.554 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:26:12 smithi047 conmon[78521]: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-13T23:26:12.554 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:26:12 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-13T23:26:12.554 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:26:12 smithi047 conmon[78521]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-13T23:26:12.554 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:26:12 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-13T23:26:12.555 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:26:12 smithi047 conmon[78521]: return self.event_loop.get_result(coro) 2022-05-13T23:26:12.555 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:26:12 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-13T23:26:12.555 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:26:12 smithi047 conmon[78521]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-13T23:26:12.555 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:26:12 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-13T23:26:12.556 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:26:12 smithi047 conmon[78521]: return self.__get_result() 2022-05-13T23:26:12.556 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:26:12 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-13T23:26:12.556 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:26:12 smithi047 conmon[78521]: raise self._exception 2022-05-13T23:26:12.556 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:26:12 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-13T23:26:12.556 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:26:12 smithi047 conmon[78521]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-13T23:26:12.557 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:26:12 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-13T23:26:12.557 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:26:12 smithi047 conmon[78521]: conn = await self._remote_connection(host, addr) 2022-05-13T23:26:12.557 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:26:12 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-13T23:26:12.557 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:26:12 smithi047 conmon[78521]: raise 2022-05-13T23:26:12.557 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:26:12 smithi047 conmon[78521]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-13T23:26:12.558 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:26:12 smithi047 conmon[78521]: self.gen.throw(type, value, traceback) 2022-05-13T23:26:12.558 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:26:12 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-13T23:26:12.558 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:26:12 smithi047 conmon[78521]: raise OrchestratorError(msg) 2022-05-13T23:26:12.558 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:26:12 smithi047 conmon[78521]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:26:12.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:26:12 smithi115 ceph-mon[78593]: pgmap v246: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 139 MiB data, 473 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 3 op/s; 10440/32430 objects degraded (32.192%) 2022-05-13T23:26:13.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:26:13 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:26:13.608 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-13T23:26:13.609 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-13T23:26:13.609 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-13T23:26:13.609 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-13T23:26:13.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-13T23:26:13.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-13T23:26:13.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-13T23:26:13.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-13T23:26:13.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-13T23:26:13.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-13T23:26:13.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-13T23:26:13.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-13T23:26:13.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-13T23:26:13.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-13T23:26:13.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-13T23:26:13.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-13T23:26:13.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-13T23:26:13.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:26:13.613 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:26:13 smithi115 ceph-mon[78593]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-13T23:26:13.613 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-13T23:26:13.613 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-13T23:26:13.613 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-13T23:26:13.614 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-13T23:26:13.614 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-13T23:26:13.614 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-13T23:26:13.614 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-13T23:26:13.614 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-13T23:26:13.615 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-13T23:26:13.615 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-13T23:26:13.615 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-13T23:26:13.615 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-13T23:26:13.615 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-13T23:26:13.616 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-13T23:26:13.616 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-13T23:26:13.616 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-13T23:26:13.616 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-13T23:26:13.617 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:26:13.617 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-13T23:26:13.617 INFO:journalctl@ceph.mon.c.smithi115.stdout: During handling of the above exception, another exception occurred: 2022-05-13T23:26:13.617 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-13T23:26:13.617 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-13T23:26:13.618 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-13T23:26:13.618 INFO:journalctl@ceph.mon.c.smithi115.stdout: return f(*arg) 2022-05-13T23:26:13.618 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-13T23:26:13.618 INFO:journalctl@ceph.mon.c.smithi115.stdout: self._write_client_files(client_files, host) 2022-05-13T23:26:13.619 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-13T23:26:13.619 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-13T23:26:13.619 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-13T23:26:13.619 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-13T23:26:13.619 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-13T23:26:13.620 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.event_loop.get_result(coro) 2022-05-13T23:26:13.620 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-13T23:26:13.620 INFO:journalctl@ceph.mon.c.smithi115.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-13T23:26:13.620 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-13T23:26:13.621 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.__get_result() 2022-05-13T23:26:13.621 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-13T23:26:13.621 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise self._exception 2022-05-13T23:26:13.621 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-13T23:26:13.622 INFO:journalctl@ceph.mon.c.smithi115.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-13T23:26:13.622 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-13T23:26:13.622 INFO:journalctl@ceph.mon.c.smithi115.stdout: conn = await self._remote_connection(host, addr) 2022-05-13T23:26:13.622 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-13T23:26:13.623 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise 2022-05-13T23:26:13.623 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-13T23:26:13.623 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.gen.throw(type, value, traceback) 2022-05-13T23:26:13.623 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-13T23:26:13.623 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OrchestratorError(msg) 2022-05-13T23:26:13.624 INFO:journalctl@ceph.mon.c.smithi115.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:26:13.624 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:26:13 smithi115 ceph-mon[78593]: 1 slow requests, 0 included below; oldest blocked for > 206.103944 secs 2022-05-13T23:26:13.624 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:26:13 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-13T23:26:13.624 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:26:13 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-13T23:26:13.796 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:26:13 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:26:13.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-13T23:26:13.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-13T23:26:13.797 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-13T23:26:13.797 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-13T23:26:13.798 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-13T23:26:13.798 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-13T23:26:13.798 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-13T23:26:13.799 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-13T23:26:13.799 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-13T23:26:13.799 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-13T23:26:13.799 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-13T23:26:13.799 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-13T23:26:13.800 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-13T23:26:13.800 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-13T23:26:13.800 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-13T23:26:13.800 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-13T23:26:13.800 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-13T23:26:13.801 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:26:13.801 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:26:13 smithi047 ceph-mon[77147]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-13T23:26:13.801 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-13T23:26:13.801 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-13T23:26:13.802 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-13T23:26:13.802 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-13T23:26:13.802 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-13T23:26:13.803 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-13T23:26:13.803 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-13T23:26:13.803 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-13T23:26:13.803 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-13T23:26:13.803 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-13T23:26:13.804 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-13T23:26:13.804 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-13T23:26:13.804 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-13T23:26:13.804 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-13T23:26:13.805 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-13T23:26:13.805 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-13T23:26:13.805 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-13T23:26:13.805 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:26:13.805 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-13T23:26:13.806 INFO:journalctl@ceph.mon.b.smithi047.stdout: During handling of the above exception, another exception occurred: 2022-05-13T23:26:13.806 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-13T23:26:13.806 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-13T23:26:13.806 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-13T23:26:13.806 INFO:journalctl@ceph.mon.b.smithi047.stdout: return f(*arg) 2022-05-13T23:26:13.807 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-13T23:26:13.807 INFO:journalctl@ceph.mon.b.smithi047.stdout: self._write_client_files(client_files, host) 2022-05-13T23:26:13.807 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-13T23:26:13.807 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-13T23:26:13.807 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-13T23:26:13.809 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-13T23:26:13.810 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-13T23:26:13.810 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.event_loop.get_result(coro) 2022-05-13T23:26:13.810 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-13T23:26:13.810 INFO:journalctl@ceph.mon.b.smithi047.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-13T23:26:13.810 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-13T23:26:13.811 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.__get_result() 2022-05-13T23:26:13.811 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-13T23:26:13.811 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise self._exception 2022-05-13T23:26:13.811 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-13T23:26:13.812 INFO:journalctl@ceph.mon.b.smithi047.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-13T23:26:13.812 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-13T23:26:13.812 INFO:journalctl@ceph.mon.b.smithi047.stdout: conn = await self._remote_connection(host, addr) 2022-05-13T23:26:13.812 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-13T23:26:13.812 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise 2022-05-13T23:26:13.813 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-13T23:26:13.813 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.gen.throw(type, value, traceback) 2022-05-13T23:26:13.813 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-13T23:26:13.813 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OrchestratorError(msg) 2022-05-13T23:26:13.814 INFO:journalctl@ceph.mon.b.smithi047.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:26:13.814 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:26:13 smithi047 ceph-mon[77147]: 1 slow requests, 0 included below; oldest blocked for > 206.103944 secs 2022-05-13T23:26:13.814 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:26:13 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-13T23:26:13.814 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:26:13 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-13T23:26:14.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:26:14 smithi115 ceph-mon[78593]: pgmap v247: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 139 MiB data, 473 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 3 op/s; 10440/32430 objects degraded (32.192%) 2022-05-13T23:26:14.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:26:14 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:26:14.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:26:14 smithi115 ceph-mon[78593]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2022-05-13T23:26:14.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:26:14 smithi047 ceph-mon[77147]: pgmap v247: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 139 MiB data, 473 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 3 op/s; 10440/32430 objects degraded (32.192%) 2022-05-13T23:26:14.885 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:26:14 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:26:14.885 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:26:14 smithi047 ceph-mon[77147]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2022-05-13T23:26:16.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:26:16 smithi115 ceph-mon[78593]: pgmap v248: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 139 MiB data, 473 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s; 10440/32430 objects degraded (32.192%) 2022-05-13T23:26:16.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:26:16 smithi115 ceph-mon[78593]: 1 slow requests, 0 included below; oldest blocked for > 539.082875 secs 2022-05-13T23:26:16.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:26:16 smithi047 ceph-mon[77147]: pgmap v248: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 139 MiB data, 473 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s; 10440/32430 objects degraded (32.192%) 2022-05-13T23:26:16.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:26:16 smithi047 ceph-mon[77147]: 1 slow requests, 0 included below; oldest blocked for > 539.082875 secs 2022-05-13T23:26:18.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:26:18 smithi115 ceph-mon[78593]: pgmap v249: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 139 MiB data, 473 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 3 op/s; 10440/32430 objects degraded (32.192%) 2022-05-13T23:26:18.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:26:18 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:26:18.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:26:18 smithi115 ceph-mon[78593]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2022-05-13T23:26:18.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:26:18 smithi115 ceph-mon[78593]: 1 slow requests, 0 included below; oldest blocked for > 211.104082 secs 2022-05-13T23:26:18.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:26:18 smithi047 ceph-mon[77147]: pgmap v249: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 139 MiB data, 473 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 3 op/s; 10440/32430 objects degraded (32.192%) 2022-05-13T23:26:18.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:26:18 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:26:18.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:26:18 smithi047 ceph-mon[77147]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2022-05-13T23:26:18.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:26:18 smithi047 ceph-mon[77147]: 1 slow requests, 0 included below; oldest blocked for > 211.104082 secs 2022-05-13T23:26:20.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:26:20 smithi115 ceph-mon[78593]: pgmap v250: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 139 MiB data, 473 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s; 10440/32430 objects degraded (32.192%) 2022-05-13T23:26:20.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:26:20 smithi047 ceph-mon[77147]: pgmap v250: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 139 MiB data, 473 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s; 10440/32430 objects degraded (32.192%) 2022-05-13T23:26:21.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:26:21 smithi115 ceph-mon[78593]: 1 slow requests, 0 included below; oldest blocked for > 544.082995 secs 2022-05-13T23:26:21.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:26:21 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:26:21.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:26:21 smithi115 ceph-mon[78593]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2022-05-13T23:26:21.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:26:21 smithi047 ceph-mon[77147]: 1 slow requests, 0 included below; oldest blocked for > 544.082995 secs 2022-05-13T23:26:21.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:26:21 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:26:21.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:26:21 smithi047 ceph-mon[77147]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2022-05-13T23:26:22.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:26:22 smithi115 ceph-mon[78593]: pgmap v251: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 139 MiB data, 473 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 3 op/s; 10440/32430 objects degraded (32.192%) 2022-05-13T23:26:22.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:26:22 smithi047 ceph-mon[77147]: pgmap v251: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 139 MiB data, 473 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 3 op/s; 10440/32430 objects degraded (32.192%) 2022-05-13T23:26:23.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:26:23 smithi115 ceph-mon[78593]: 1 slow requests, 0 included below; oldest blocked for > 216.104186 secs 2022-05-13T23:26:23.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:26:23 smithi047 ceph-mon[77147]: 1 slow requests, 0 included below; oldest blocked for > 216.104186 secs 2022-05-13T23:26:24.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:26:24 smithi115 ceph-mon[78593]: pgmap v252: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 139 MiB data, 473 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 3 op/s; 10440/32430 objects degraded (32.192%) 2022-05-13T23:26:24.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:26:24 smithi047 ceph-mon[77147]: pgmap v252: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 139 MiB data, 473 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 3 op/s; 10440/32430 objects degraded (32.192%) 2022-05-13T23:26:26.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:26:26 smithi115 ceph-mon[78593]: pgmap v253: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 139 MiB data, 473 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s; 10440/32430 objects degraded (32.192%) 2022-05-13T23:26:26.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:26:26 smithi115 ceph-mon[78593]: 1 slow requests, 0 included below; oldest blocked for > 549.083112 secs 2022-05-13T23:26:26.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:26:26 smithi047 ceph-mon[77147]: pgmap v253: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 139 MiB data, 473 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s; 10440/32430 objects degraded (32.192%) 2022-05-13T23:26:26.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:26:26 smithi047 ceph-mon[77147]: 1 slow requests, 0 included below; oldest blocked for > 549.083112 secs 2022-05-13T23:26:28.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:26:28 smithi115 ceph-mon[78593]: pgmap v254: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 139 MiB data, 473 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 3 op/s; 10440/32430 objects degraded (32.192%) 2022-05-13T23:26:28.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:26:28 smithi115 ceph-mon[78593]: 1 slow requests, 0 included below; oldest blocked for > 221.104314 secs 2022-05-13T23:26:28.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:26:28 smithi047 ceph-mon[77147]: pgmap v254: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 139 MiB data, 473 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 3 op/s; 10440/32430 objects degraded (32.192%) 2022-05-13T23:26:28.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:26:28 smithi047 ceph-mon[77147]: 1 slow requests, 0 included below; oldest blocked for > 221.104314 secs 2022-05-13T23:26:30.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:26:30 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:26:30.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:26:30 smithi115 ceph-mon[78593]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2022-05-13T23:26:30.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:26:30 smithi115 ceph-mon[78593]: pgmap v255: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 139 MiB data, 473 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s; 10440/32430 objects degraded (32.192%) 2022-05-13T23:26:30.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:26:30 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:26:30.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:26:30 smithi047 ceph-mon[77147]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2022-05-13T23:26:30.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:26:30 smithi047 ceph-mon[77147]: pgmap v255: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 139 MiB data, 473 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s; 10440/32430 objects degraded (32.192%) 2022-05-13T23:26:31.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:26:31 smithi115 ceph-mon[78593]: 1 slow requests, 0 included below; oldest blocked for > 554.083230 secs 2022-05-13T23:26:31.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:26:31 smithi047 ceph-mon[77147]: 1 slow requests, 0 included below; oldest blocked for > 554.083230 secs 2022-05-13T23:26:32.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:26:32 smithi115 ceph-mon[78593]: pgmap v256: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 139 MiB data, 473 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 3 op/s; 10440/32430 objects degraded (32.192%) 2022-05-13T23:26:32.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:26:32 smithi047 ceph-mon[77147]: pgmap v256: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 139 MiB data, 473 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 3 op/s; 10440/32430 objects degraded (32.192%) 2022-05-13T23:26:33.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:26:33 smithi115 ceph-mon[78593]: 1 slow requests, 0 included below; oldest blocked for > 226.104419 secs 2022-05-13T23:26:33.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:26:33 smithi047 ceph-mon[77147]: 1 slow requests, 0 included below; oldest blocked for > 226.104419 secs 2022-05-13T23:26:34.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:26:34 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:26:34.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:26:34 smithi115 ceph-mon[78593]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2022-05-13T23:26:34.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:26:34 smithi115 ceph-mon[78593]: pgmap v257: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 139 MiB data, 473 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 3 op/s; 10440/32430 objects degraded (32.192%) 2022-05-13T23:26:34.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:26:34 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:26:34.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:26:34 smithi047 ceph-mon[77147]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2022-05-13T23:26:34.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:26:34 smithi047 ceph-mon[77147]: pgmap v257: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 139 MiB data, 473 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 3 op/s; 10440/32430 objects degraded (32.192%) 2022-05-13T23:26:36.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:26:36 smithi047 ceph-mon[77147]: pgmap v258: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 139 MiB data, 473 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s; 10440/32430 objects degraded (32.192%) 2022-05-13T23:26:36.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:26:36 smithi047 ceph-mon[77147]: 1 slow requests, 0 included below; oldest blocked for > 559.083367 secs 2022-05-13T23:26:36.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:26:36 smithi115 ceph-mon[78593]: pgmap v258: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 139 MiB data, 473 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s; 10440/32430 objects degraded (32.192%) 2022-05-13T23:26:36.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:26:36 smithi115 ceph-mon[78593]: 1 slow requests, 0 included below; oldest blocked for > 559.083367 secs 2022-05-13T23:26:38.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:26:38 smithi047 ceph-mon[77147]: pgmap v259: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 139 MiB data, 473 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 3 op/s; 10440/32430 objects degraded (32.192%) 2022-05-13T23:26:38.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:26:38 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:26:38.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:26:38 smithi047 ceph-mon[77147]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2022-05-13T23:26:38.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:26:38 smithi047 ceph-mon[77147]: 1 slow requests, 0 included below; oldest blocked for > 231.104569 secs 2022-05-13T23:26:38.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:26:38 smithi115 ceph-mon[78593]: pgmap v259: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 139 MiB data, 473 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 3 op/s; 10440/32430 objects degraded (32.192%) 2022-05-13T23:26:38.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:26:38 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:26:38.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:26:38 smithi115 ceph-mon[78593]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2022-05-13T23:26:38.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:26:38 smithi115 ceph-mon[78593]: 1 slow requests, 0 included below; oldest blocked for > 231.104569 secs 2022-05-13T23:26:40.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:26:40 smithi047 ceph-mon[77147]: pgmap v260: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 139 MiB data, 473 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s; 10440/32430 objects degraded (32.192%) 2022-05-13T23:26:40.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:26:40 smithi115 ceph-mon[78593]: pgmap v260: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 139 MiB data, 473 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s; 10440/32430 objects degraded (32.192%) 2022-05-13T23:26:41.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:26:41 smithi047 ceph-mon[77147]: 1 slow requests, 0 included below; oldest blocked for > 564.083512 secs 2022-05-13T23:26:41.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:26:41 smithi115 ceph-mon[78593]: 1 slow requests, 0 included below; oldest blocked for > 564.083512 secs 2022-05-13T23:26:42.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:26:42 smithi047 ceph-mon[77147]: pgmap v261: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 139 MiB data, 473 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 3 op/s; 10440/32430 objects degraded (32.192%) 2022-05-13T23:26:42.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:26:42 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:26:42.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:26:42 smithi047 ceph-mon[77147]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2022-05-13T23:26:42.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:26:42 smithi115 ceph-mon[78593]: pgmap v261: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 139 MiB data, 473 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 3 op/s; 10440/32430 objects degraded (32.192%) 2022-05-13T23:26:42.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:26:42 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:26:42.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:26:42 smithi115 ceph-mon[78593]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2022-05-13T23:26:43.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:26:43 smithi047 ceph-mon[77147]: 1 slow requests, 0 included below; oldest blocked for > 236.104687 secs 2022-05-13T23:26:43.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:26:43 smithi115 ceph-mon[78593]: 1 slow requests, 0 included below; oldest blocked for > 236.104687 secs 2022-05-13T23:26:44.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:26:44 smithi047 ceph-mon[77147]: pgmap v262: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 139 MiB data, 473 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 3 op/s; 10440/32430 objects degraded (32.192%) 2022-05-13T23:26:44.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:26:44 smithi115 ceph-mon[78593]: pgmap v262: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 139 MiB data, 473 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 3 op/s; 10440/32430 objects degraded (32.192%) 2022-05-13T23:26:46.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:26:46 smithi115 ceph-mon[78593]: pgmap v263: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 139 MiB data, 473 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s; 10440/32430 objects degraded (32.192%) 2022-05-13T23:26:46.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:26:46 smithi115 ceph-mon[78593]: 1 slow requests, 0 included below; oldest blocked for > 569.083628 secs 2022-05-13T23:26:46.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:26:46 smithi047 ceph-mon[77147]: pgmap v263: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 139 MiB data, 473 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s; 10440/32430 objects degraded (32.192%) 2022-05-13T23:26:46.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:26:46 smithi047 ceph-mon[77147]: 1 slow requests, 0 included below; oldest blocked for > 569.083628 secs 2022-05-13T23:26:48.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:26:48 smithi115 ceph-mon[78593]: pgmap v264: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 139 MiB data, 473 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 3 op/s; 10440/32430 objects degraded (32.192%) 2022-05-13T23:26:48.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:26:48 smithi115 ceph-mon[78593]: 1 slow requests, 0 included below; oldest blocked for > 241.104861 secs 2022-05-13T23:26:48.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:26:48 smithi047 ceph-mon[77147]: pgmap v264: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 139 MiB data, 473 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 3 op/s; 10440/32430 objects degraded (32.192%) 2022-05-13T23:26:48.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:26:48 smithi047 ceph-mon[77147]: 1 slow requests, 0 included below; oldest blocked for > 241.104861 secs 2022-05-13T23:26:50.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:26:50 smithi115 ceph-mon[78593]: pgmap v265: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 139 MiB data, 473 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s; 10440/32430 objects degraded (32.192%) 2022-05-13T23:26:50.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:26:50 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:26:50.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:26:50 smithi115 ceph-mon[78593]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2022-05-13T23:26:50.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:26:50 smithi047 ceph-mon[77147]: pgmap v265: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 139 MiB data, 473 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s; 10440/32430 objects degraded (32.192%) 2022-05-13T23:26:50.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:26:50 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:26:50.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:26:50 smithi047 ceph-mon[77147]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2022-05-13T23:26:51.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:26:51 smithi115 ceph-mon[78593]: 1 slow requests, 0 included below; oldest blocked for > 574.083718 secs 2022-05-13T23:26:51.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:26:51 smithi047 ceph-mon[77147]: 1 slow requests, 0 included below; oldest blocked for > 574.083718 secs 2022-05-13T23:26:52.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:26:52 smithi115 ceph-mon[78593]: pgmap v266: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 139 MiB data, 473 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 3 op/s; 10440/32430 objects degraded (32.192%) 2022-05-13T23:26:52.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:26:52 smithi047 ceph-mon[77147]: pgmap v266: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 139 MiB data, 473 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 3 op/s; 10440/32430 objects degraded (32.192%) 2022-05-13T23:26:53.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:26:53 smithi115 ceph-mon[78593]: 1 slow requests, 0 included below; oldest blocked for > 246.105035 secs 2022-05-13T23:26:53.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:26:53 smithi047 ceph-mon[77147]: 1 slow requests, 0 included below; oldest blocked for > 246.105035 secs 2022-05-13T23:26:54.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:26:54 smithi115 ceph-mon[78593]: pgmap v267: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 139 MiB data, 473 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 3 op/s; 10440/32430 objects degraded (32.192%) 2022-05-13T23:26:54.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:26:54 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:26:54.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:26:54 smithi115 ceph-mon[78593]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2022-05-13T23:26:54.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:26:54 smithi047 ceph-mon[77147]: pgmap v267: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 139 MiB data, 473 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 3 op/s; 10440/32430 objects degraded (32.192%) 2022-05-13T23:26:54.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:26:54 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:26:54.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:26:54 smithi047 ceph-mon[77147]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2022-05-13T23:26:56.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:26:56 smithi115 ceph-mon[78593]: pgmap v268: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 139 MiB data, 473 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s; 10440/32430 objects degraded (32.192%) 2022-05-13T23:26:56.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:26:56 smithi115 ceph-mon[78593]: 1 slow requests, 0 included below; oldest blocked for > 579.083852 secs 2022-05-13T23:26:56.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:26:56 smithi047 ceph-mon[77147]: pgmap v268: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 139 MiB data, 473 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s; 10440/32430 objects degraded (32.192%) 2022-05-13T23:26:56.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:26:56 smithi047 ceph-mon[77147]: 1 slow requests, 0 included below; oldest blocked for > 579.083852 secs 2022-05-13T23:26:57.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:26:57 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:26:57.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:26:57 smithi115 ceph-mon[78593]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2022-05-13T23:26:57.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:26:57 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:26:57.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:26:57 smithi047 ceph-mon[77147]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2022-05-13T23:26:58.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:26:58 smithi115 ceph-mon[78593]: pgmap v269: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 139 MiB data, 473 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 3 op/s; 10440/32430 objects degraded (32.192%) 2022-05-13T23:26:58.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:26:58 smithi115 ceph-mon[78593]: 1 slow requests, 0 included below; oldest blocked for > 251.105163 secs 2022-05-13T23:26:58.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:26:58 smithi047 ceph-mon[77147]: pgmap v269: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 139 MiB data, 473 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 3 op/s; 10440/32430 objects degraded (32.192%) 2022-05-13T23:26:58.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:26:58 smithi047 ceph-mon[77147]: 1 slow requests, 0 included below; oldest blocked for > 251.105163 secs 2022-05-13T23:27:00.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:27:00 smithi115 ceph-mon[78593]: pgmap v270: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 139 MiB data, 473 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s; 10440/32430 objects degraded (32.192%) 2022-05-13T23:27:00.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:27:00 smithi047 ceph-mon[77147]: pgmap v270: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 139 MiB data, 473 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s; 10440/32430 objects degraded (32.192%) 2022-05-13T23:27:01.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:27:01 smithi115 ceph-mon[78593]: 1 slow requests, 0 included below; oldest blocked for > 584.083976 secs 2022-05-13T23:27:01.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:27:01 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:27:01.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:27:01 smithi115 ceph-mon[78593]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2022-05-13T23:27:01.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:27:01 smithi047 ceph-mon[77147]: 1 slow requests, 0 included below; oldest blocked for > 584.083976 secs 2022-05-13T23:27:01.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:27:01 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:27:01.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:27:01 smithi047 ceph-mon[77147]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2022-05-13T23:27:02.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:27:02 smithi115 ceph-mon[78593]: pgmap v271: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 139 MiB data, 473 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 3 op/s; 10440/32430 objects degraded (32.192%) 2022-05-13T23:27:02.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:27:02 smithi047 ceph-mon[77147]: pgmap v271: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 139 MiB data, 473 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 3 op/s; 10440/32430 objects degraded (32.192%) 2022-05-13T23:27:03.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:27:03 smithi115 ceph-mon[78593]: 1 slow requests, 0 included below; oldest blocked for > 256.105272 secs 2022-05-13T23:27:03.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:27:03 smithi047 ceph-mon[77147]: 1 slow requests, 0 included below; oldest blocked for > 256.105272 secs 2022-05-13T23:27:04.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:27:04 smithi115 ceph-mon[78593]: pgmap v272: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 139 MiB data, 473 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 3 op/s; 10440/32430 objects degraded (32.192%) 2022-05-13T23:27:04.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:27:04 smithi047 ceph-mon[77147]: pgmap v272: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 139 MiB data, 473 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 3 op/s; 10440/32430 objects degraded (32.192%) 2022-05-13T23:27:06.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:27:06 smithi047 ceph-mon[77147]: pgmap v273: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 139 MiB data, 473 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s; 10440/32430 objects degraded (32.192%) 2022-05-13T23:27:06.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:27:06 smithi047 ceph-mon[77147]: 1 slow requests, 0 included below; oldest blocked for > 589.084093 secs 2022-05-13T23:27:06.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:27:06 smithi115 ceph-mon[78593]: pgmap v273: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 139 MiB data, 473 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s; 10440/32430 objects degraded (32.192%) 2022-05-13T23:27:06.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:27:06 smithi115 ceph-mon[78593]: 1 slow requests, 0 included below; oldest blocked for > 589.084093 secs 2022-05-13T23:27:08.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:27:08 smithi047 ceph-mon[77147]: pgmap v274: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 139 MiB data, 473 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 3 op/s; 10440/32430 objects degraded (32.192%) 2022-05-13T23:27:08.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:27:08 smithi047 ceph-mon[77147]: 1 slow requests, 0 included below; oldest blocked for > 261.105411 secs 2022-05-13T23:27:08.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:27:08 smithi115 ceph-mon[78593]: pgmap v274: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 139 MiB data, 473 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 3 op/s; 10440/32430 objects degraded (32.192%) 2022-05-13T23:27:08.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:27:08 smithi115 ceph-mon[78593]: 1 slow requests, 0 included below; oldest blocked for > 261.105411 secs 2022-05-13T23:27:10.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:27:10 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:27:10.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:27:10 smithi115 ceph-mon[78593]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2022-05-13T23:27:10.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:27:10 smithi115 ceph-mon[78593]: pgmap v275: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 139 MiB data, 473 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s; 10440/32430 objects degraded (32.192%) 2022-05-13T23:27:10.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:27:10 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:27:10.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:27:10 smithi047 ceph-mon[77147]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2022-05-13T23:27:10.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:27:10 smithi047 ceph-mon[77147]: pgmap v275: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 139 MiB data, 473 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s; 10440/32430 objects degraded (32.192%) 2022-05-13T23:27:11.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:27:11 smithi115 ceph-mon[78593]: 1 slow requests, 0 included below; oldest blocked for > 594.084231 secs 2022-05-13T23:27:11.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:27:11 smithi047 ceph-mon[77147]: 1 slow requests, 0 included below; oldest blocked for > 594.084231 secs 2022-05-13T23:27:12.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:27:12 smithi115 ceph-mon[78593]: pgmap v276: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 139 MiB data, 473 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 3 op/s; 10440/32430 objects degraded (32.192%) 2022-05-13T23:27:12.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:27:12 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-13T23:27:12.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:27:12 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-13T23:27:12.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:27:12 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-13T23:27:12.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:27:12 smithi047 ceph-mon[77147]: pgmap v276: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 139 MiB data, 473 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 3 op/s; 10440/32430 objects degraded (32.192%) 2022-05-13T23:27:12.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:27:12 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-13T23:27:12.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:27:12 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-13T23:27:12.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:27:12 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-13T23:27:13.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:27:13 smithi115 ceph-mon[78593]: 1 slow requests, 0 included below; oldest blocked for > 266.105537 secs 2022-05-13T23:27:13.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:27:13 smithi047 ceph-mon[77147]: 1 slow requests, 0 included below; oldest blocked for > 266.105537 secs 2022-05-13T23:27:14.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:27:14 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:27:14.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:27:14 smithi115 ceph-mon[78593]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2022-05-13T23:27:14.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:27:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-13T23:27:14.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:27:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-13T23:27:14.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:27:14 smithi115 ceph-mon[78593]: pgmap v277: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 139 MiB data, 473 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 3 op/s; 10440/32430 objects degraded (32.192%) 2022-05-13T23:27:14.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:27:14 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:27:14.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:27:14 smithi047 ceph-mon[77147]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2022-05-13T23:27:14.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:27:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-13T23:27:14.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:27:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-13T23:27:14.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:27:14 smithi047 ceph-mon[77147]: pgmap v277: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 139 MiB data, 473 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 3 op/s; 10440/32430 objects degraded (32.192%) 2022-05-13T23:27:15.541 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:27:15 smithi047 conmon[78521]: 2022-05-13T23:27:15.185+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:27:15.542 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:27:15 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-13T23:27:15.542 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:27:15 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-13T23:27:15.542 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:27:15 smithi047 conmon[78521]: yield 2022-05-13T23:27:15.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:27:15 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-13T23:27:15.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:27:15 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-13T23:27:15.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:27:15 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-13T23:27:15.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:27:15 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-13T23:27:15.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:27:15 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-13T23:27:15.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:27:15 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-13T23:27:15.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:27:15 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-13T23:27:15.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:27:15 smithi047 conmon[78521]: raise exceptions[0] 2022-05-13T23:27:15.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:27:15 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-13T23:27:15.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:27:15 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-13T23:27:15.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:27:15 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-13T23:27:15.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:27:15 smithi047 conmon[78521]: return (yield from fut) 2022-05-13T23:27:15.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:27:15 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-13T23:27:15.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:27:15 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-13T23:27:15.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:27:15 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:27:16.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:27:16 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:27:16.542 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-13T23:27:16.542 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-13T23:27:16.542 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-13T23:27:16.542 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-13T23:27:16.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-13T23:27:16.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-13T23:27:16.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-13T23:27:16.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-13T23:27:16.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-13T23:27:16.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-13T23:27:16.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-13T23:27:16.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-13T23:27:16.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-13T23:27:16.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-13T23:27:16.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-13T23:27:16.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-13T23:27:16.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-13T23:27:16.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:27:16.548 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:27:16 smithi047 ceph-mon[77147]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-13T23:27:16.549 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:27:16 smithi047 ceph-mon[77147]: pgmap v278: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 139 MiB data, 473 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s; 10440/32430 objects degraded (32.192%) 2022-05-13T23:27:16.549 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:27:16 smithi047 ceph-mon[77147]: 1 slow requests, 0 included below; oldest blocked for > 599.084354 secs 2022-05-13T23:27:16.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:27:16 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:27:16.608 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-13T23:27:16.608 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-13T23:27:16.609 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-13T23:27:16.609 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-13T23:27:16.609 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-13T23:27:16.609 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-13T23:27:16.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-13T23:27:16.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-13T23:27:16.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-13T23:27:16.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-13T23:27:16.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-13T23:27:16.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-13T23:27:16.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-13T23:27:16.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-13T23:27:16.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-13T23:27:16.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-13T23:27:16.613 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-13T23:27:16.613 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:27:16.613 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:27:16 smithi115 ceph-mon[78593]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-13T23:27:16.613 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:27:16 smithi115 ceph-mon[78593]: pgmap v278: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 139 MiB data, 473 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s; 10440/32430 objects degraded (32.192%) 2022-05-13T23:27:16.614 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:27:16 smithi115 ceph-mon[78593]: 1 slow requests, 0 included below; oldest blocked for > 599.084354 secs 2022-05-13T23:27:18.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:27:18 smithi047 ceph-mon[77147]: pgmap v279: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 139 MiB data, 473 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 3 op/s; 10440/32430 objects degraded (32.192%) 2022-05-13T23:27:18.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:27:18 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:27:18.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:27:18 smithi047 ceph-mon[77147]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2022-05-13T23:27:18.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:27:18 smithi047 ceph-mon[77147]: 1 slow requests, 0 included below; oldest blocked for > 271.105662 secs 2022-05-13T23:27:18.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:27:18 smithi047 conmon[78521]: 2022-05-13T23:27:18.256+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:27:18.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:27:18 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-13T23:27:18.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:27:18 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-13T23:27:18.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:27:18 smithi047 conmon[78521]: yield 2022-05-13T23:27:18.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:27:18 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-13T23:27:18.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:27:18 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-13T23:27:18.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:27:18 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-13T23:27:18.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:27:18 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-13T23:27:18.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:27:18 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-13T23:27:18.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:27:18 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-13T23:27:18.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:27:18 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-13T23:27:18.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:27:18 smithi047 conmon[78521]: raise exceptions[0] 2022-05-13T23:27:18.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:27:18 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-13T23:27:18.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:27:18 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-13T23:27:18.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:27:18 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-13T23:27:18.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:27:18 smithi047 conmon[78521]: return (yield from fut) 2022-05-13T23:27:18.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:27:18 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-13T23:27:18.547 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:27:18 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-13T23:27:18.547 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:27:18 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:27:18.547 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:27:18 smithi047 conmon[78521]: 2022-05-13T23:27:18.257+0000 7f5340f7a700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-13T23:27:18.547 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:27:18 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-13T23:27:18.547 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:27:18 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-13T23:27:18.548 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:27:18 smithi047 conmon[78521]: yield 2022-05-13T23:27:18.548 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:27:18 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-13T23:27:18.548 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:27:18 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-13T23:27:18.548 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:27:18 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-13T23:27:18.548 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:27:18 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-13T23:27:18.549 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:27:18 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-13T23:27:18.549 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:27:18 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-13T23:27:18.549 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:27:18 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-13T23:27:18.549 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:27:18 smithi047 conmon[78521]: raise exceptions[0] 2022-05-13T23:27:18.550 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:27:18 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-13T23:27:18.550 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:27:18 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-13T23:27:18.550 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:27:18 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-13T23:27:18.550 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:27:18 smithi047 conmon[78521]: return (yield from fut) 2022-05-13T23:27:18.551 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:27:18 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-13T23:27:18.551 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:27:18 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-13T23:27:18.551 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:27:18 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:27:18.551 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:27:18 smithi047 conmon[78521]: 2022-05-13T23:27:18.551 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:27:18 smithi047 conmon[78521]: During handling of the above exception, another exception occurred: 2022-05-13T23:27:18.552 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:27:18 smithi047 conmon[78521]: 2022-05-13T23:27:18.552 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:27:18 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-13T23:27:18.552 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:27:18 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-13T23:27:18.552 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:27:18 smithi047 conmon[78521]: return f(*arg) 2022-05-13T23:27:18.553 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:27:18 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-13T23:27:18.553 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:27:18 smithi047 conmon[78521]: self._write_client_files(client_files, host) 2022-05-13T23:27:18.553 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:27:18 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-13T23:27:18.553 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:27:18 smithi047 conmon[78521]: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-13T23:27:18.553 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:27:18 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-13T23:27:18.554 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:27:18 smithi047 conmon[78521]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-13T23:27:18.554 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:27:18 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-13T23:27:18.554 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:27:18 smithi047 conmon[78521]: return self.event_loop.get_result(coro) 2022-05-13T23:27:18.554 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:27:18 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-13T23:27:18.555 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:27:18 smithi047 conmon[78521]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-13T23:27:18.555 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:27:18 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-13T23:27:18.555 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:27:18 smithi047 conmon[78521]: return self.__get_result() 2022-05-13T23:27:18.555 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:27:18 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-13T23:27:18.555 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:27:18 smithi047 conmon[78521]: raise self._exception 2022-05-13T23:27:18.556 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:27:18 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-13T23:27:18.556 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:27:18 smithi047 conmon[78521]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-13T23:27:18.556 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:27:18 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-13T23:27:18.556 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:27:18 smithi047 conmon[78521]: conn = await self._remote_connection(host, addr) 2022-05-13T23:27:18.557 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:27:18 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-13T23:27:18.557 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:27:18 smithi047 conmon[78521]: raise 2022-05-13T23:27:18.557 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:27:18 smithi047 conmon[78521]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-13T23:27:18.557 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:27:18 smithi047 conmon[78521]: self.gen.throw(type, value, traceback) 2022-05-13T23:27:18.558 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:27:18 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-13T23:27:18.558 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:27:18 smithi047 conmon[78521]: raise OrchestratorError(msg) 2022-05-13T23:27:18.558 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:27:18 smithi047 conmon[78521]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:27:18.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:27:18 smithi115 ceph-mon[78593]: pgmap v279: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 139 MiB data, 473 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 3 op/s; 10440/32430 objects degraded (32.192%) 2022-05-13T23:27:18.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:27:18 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:27:18.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:27:18 smithi115 ceph-mon[78593]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2022-05-13T23:27:18.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:27:18 smithi115 ceph-mon[78593]: 1 slow requests, 0 included below; oldest blocked for > 271.105662 secs 2022-05-13T23:27:19.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:27:19 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:27:19.608 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-13T23:27:19.609 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-13T23:27:19.609 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-13T23:27:19.609 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-13T23:27:19.609 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-13T23:27:19.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-13T23:27:19.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-13T23:27:19.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-13T23:27:19.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-13T23:27:19.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-13T23:27:19.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-13T23:27:19.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-13T23:27:19.615 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-13T23:27:19.616 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-13T23:27:19.616 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-13T23:27:19.616 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-13T23:27:19.617 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-13T23:27:19.617 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:27:19.617 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:27:19 smithi115 ceph-mon[78593]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-13T23:27:19.618 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-13T23:27:19.618 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-13T23:27:19.618 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-13T23:27:19.619 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-13T23:27:19.619 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-13T23:27:19.619 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-13T23:27:19.620 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-13T23:27:19.620 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-13T23:27:19.620 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-13T23:27:19.621 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-13T23:27:19.621 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-13T23:27:19.621 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-13T23:27:19.622 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-13T23:27:19.622 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-13T23:27:19.623 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-13T23:27:19.623 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-13T23:27:19.623 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-13T23:27:19.624 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:27:19.624 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-13T23:27:19.625 INFO:journalctl@ceph.mon.c.smithi115.stdout: During handling of the above exception, another exception occurred: 2022-05-13T23:27:19.625 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-13T23:27:19.625 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-13T23:27:19.626 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-13T23:27:19.626 INFO:journalctl@ceph.mon.c.smithi115.stdout: return f(*arg) 2022-05-13T23:27:19.626 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-13T23:27:19.627 INFO:journalctl@ceph.mon.c.smithi115.stdout: self._write_client_files(client_files, host) 2022-05-13T23:27:19.627 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-13T23:27:19.627 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-13T23:27:19.628 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-13T23:27:19.628 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-13T23:27:19.629 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-13T23:27:19.629 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.event_loop.get_result(coro) 2022-05-13T23:27:19.630 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-13T23:27:19.630 INFO:journalctl@ceph.mon.c.smithi115.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-13T23:27:19.630 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-13T23:27:19.631 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.__get_result() 2022-05-13T23:27:19.631 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-13T23:27:19.632 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise self._exception 2022-05-13T23:27:19.632 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-13T23:27:19.632 INFO:journalctl@ceph.mon.c.smithi115.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-13T23:27:19.633 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-13T23:27:19.633 INFO:journalctl@ceph.mon.c.smithi115.stdout: conn = await self._remote_connection(host, addr) 2022-05-13T23:27:19.634 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-13T23:27:19.634 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise 2022-05-13T23:27:19.635 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-13T23:27:19.635 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.gen.throw(type, value, traceback) 2022-05-13T23:27:19.635 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-13T23:27:19.636 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OrchestratorError(msg) 2022-05-13T23:27:19.636 INFO:journalctl@ceph.mon.c.smithi115.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:27:19.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:27:19 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:27:19.792 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-13T23:27:19.792 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-13T23:27:19.792 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-13T23:27:19.793 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-13T23:27:19.793 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-13T23:27:19.793 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-13T23:27:19.793 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-13T23:27:19.793 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-13T23:27:19.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-13T23:27:19.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-13T23:27:19.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-13T23:27:19.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-13T23:27:19.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-13T23:27:19.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-13T23:27:19.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-13T23:27:19.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-13T23:27:19.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-13T23:27:19.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:27:19.796 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:27:19 smithi047 ceph-mon[77147]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-13T23:27:19.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-13T23:27:19.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-13T23:27:19.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-13T23:27:19.797 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-13T23:27:19.797 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-13T23:27:19.797 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-13T23:27:19.797 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-13T23:27:19.798 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-13T23:27:19.798 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-13T23:27:19.798 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-13T23:27:19.798 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-13T23:27:19.799 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-13T23:27:19.799 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-13T23:27:19.799 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-13T23:27:19.800 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-13T23:27:19.800 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-13T23:27:19.800 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-13T23:27:19.801 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:27:19.801 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-13T23:27:19.801 INFO:journalctl@ceph.mon.b.smithi047.stdout: During handling of the above exception, another exception occurred: 2022-05-13T23:27:19.801 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-13T23:27:19.801 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-13T23:27:19.802 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-13T23:27:19.802 INFO:journalctl@ceph.mon.b.smithi047.stdout: return f(*arg) 2022-05-13T23:27:19.802 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-13T23:27:19.803 INFO:journalctl@ceph.mon.b.smithi047.stdout: self._write_client_files(client_files, host) 2022-05-13T23:27:19.803 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-13T23:27:19.803 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-13T23:27:19.804 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-13T23:27:19.804 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-13T23:27:19.804 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-13T23:27:19.804 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.event_loop.get_result(coro) 2022-05-13T23:27:19.805 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-13T23:27:19.805 INFO:journalctl@ceph.mon.b.smithi047.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-13T23:27:19.805 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-13T23:27:19.805 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.__get_result() 2022-05-13T23:27:19.806 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-13T23:27:19.806 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise self._exception 2022-05-13T23:27:19.806 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-13T23:27:19.806 INFO:journalctl@ceph.mon.b.smithi047.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-13T23:27:19.806 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-13T23:27:19.807 INFO:journalctl@ceph.mon.b.smithi047.stdout: conn = await self._remote_connection(host, addr) 2022-05-13T23:27:19.807 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-13T23:27:19.807 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise 2022-05-13T23:27:19.807 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-13T23:27:19.808 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.gen.throw(type, value, traceback) 2022-05-13T23:27:19.808 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-13T23:27:19.809 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OrchestratorError(msg) 2022-05-13T23:27:19.809 INFO:journalctl@ceph.mon.b.smithi047.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:27:20.614 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:27:20 smithi115 ceph-mon[78593]: pgmap v280: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 139 MiB data, 473 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s; 10440/32430 objects degraded (32.192%) 2022-05-13T23:27:20.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:27:20 smithi047 ceph-mon[77147]: pgmap v280: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 139 MiB data, 473 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s; 10440/32430 objects degraded (32.192%) 2022-05-13T23:27:21.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:27:21 smithi115 ceph-mon[78593]: 1 slow requests, 0 included below; oldest blocked for > 604.084473 secs 2022-05-13T23:27:21.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:27:21 smithi047 ceph-mon[77147]: 1 slow requests, 0 included below; oldest blocked for > 604.084473 secs 2022-05-13T23:27:22.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:27:22 smithi115 ceph-mon[78593]: pgmap v281: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 139 MiB data, 473 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 3 op/s; 10440/32430 objects degraded (32.192%) 2022-05-13T23:27:22.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:27:22 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:27:22.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:27:22 smithi115 ceph-mon[78593]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2022-05-13T23:27:22.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:27:22 smithi047 ceph-mon[77147]: pgmap v281: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 139 MiB data, 473 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 3 op/s; 10440/32430 objects degraded (32.192%) 2022-05-13T23:27:22.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:27:22 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:27:22.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:27:22 smithi047 ceph-mon[77147]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2022-05-13T23:27:23.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:27:23 smithi115 ceph-mon[78593]: 1 slow requests, 0 included below; oldest blocked for > 276.105789 secs 2022-05-13T23:27:23.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:27:23 smithi047 ceph-mon[77147]: 1 slow requests, 0 included below; oldest blocked for > 276.105789 secs 2022-05-13T23:27:24.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:27:24 smithi115 ceph-mon[78593]: pgmap v282: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 139 MiB data, 473 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 3 op/s; 10440/32430 objects degraded (32.192%) 2022-05-13T23:27:24.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:27:24 smithi047 ceph-mon[77147]: pgmap v282: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 139 MiB data, 473 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 3 op/s; 10440/32430 objects degraded (32.192%) 2022-05-13T23:27:26.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:27:26 smithi115 ceph-mon[78593]: pgmap v283: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 139 MiB data, 473 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s; 10440/32430 objects degraded (32.192%) 2022-05-13T23:27:26.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:27:26 smithi115 ceph-mon[78593]: 1 slow requests, 0 included below; oldest blocked for > 609.084588 secs 2022-05-13T23:27:26.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:27:26 smithi047 ceph-mon[77147]: pgmap v283: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 139 MiB data, 473 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s; 10440/32430 objects degraded (32.192%) 2022-05-13T23:27:26.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:27:26 smithi047 ceph-mon[77147]: 1 slow requests, 0 included below; oldest blocked for > 609.084588 secs 2022-05-13T23:27:28.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:27:28 smithi115 ceph-mon[78593]: pgmap v284: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 139 MiB data, 473 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 3 op/s; 10440/32430 objects degraded (32.192%) 2022-05-13T23:27:28.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:27:28 smithi115 ceph-mon[78593]: 1 slow requests, 0 included below; oldest blocked for > 281.105925 secs 2022-05-13T23:27:28.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:27:28 smithi047 ceph-mon[77147]: pgmap v284: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 139 MiB data, 473 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 3 op/s; 10440/32430 objects degraded (32.192%) 2022-05-13T23:27:28.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:27:28 smithi047 ceph-mon[77147]: 1 slow requests, 0 included below; oldest blocked for > 281.105925 secs 2022-05-13T23:27:30.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:27:30 smithi115 ceph-mon[78593]: pgmap v285: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 139 MiB data, 473 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s; 10440/32430 objects degraded (32.192%) 2022-05-13T23:27:30.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:27:30 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:27:30.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:27:30 smithi115 ceph-mon[78593]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2022-05-13T23:27:30.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:27:30 smithi047 ceph-mon[77147]: pgmap v285: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 139 MiB data, 473 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s; 10440/32430 objects degraded (32.192%) 2022-05-13T23:27:30.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:27:30 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:27:30.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:27:30 smithi047 ceph-mon[77147]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2022-05-13T23:27:31.613 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:27:31 smithi115 ceph-mon[78593]: 1 slow requests, 0 included below; oldest blocked for > 614.084708 secs 2022-05-13T23:27:31.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:27:31 smithi047 ceph-mon[77147]: 1 slow requests, 0 included below; oldest blocked for > 614.084708 secs 2022-05-13T23:27:32.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:27:32 smithi115 ceph-mon[78593]: pgmap v286: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 139 MiB data, 473 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 3 op/s; 10440/32430 objects degraded (32.192%) 2022-05-13T23:27:32.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:27:32 smithi047 ceph-mon[77147]: pgmap v286: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 139 MiB data, 473 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 3 op/s; 10440/32430 objects degraded (32.192%) 2022-05-13T23:27:33.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:27:33 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:27:33.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:27:33 smithi115 ceph-mon[78593]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2022-05-13T23:27:33.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:27:33 smithi115 ceph-mon[78593]: 1 slow requests, 0 included below; oldest blocked for > 286.106061 secs 2022-05-13T23:27:33.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:27:33 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:27:33.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:27:33 smithi047 ceph-mon[77147]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2022-05-13T23:27:33.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:27:33 smithi047 ceph-mon[77147]: 1 slow requests, 0 included below; oldest blocked for > 286.106061 secs 2022-05-13T23:27:34.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:27:34 smithi115 ceph-mon[78593]: pgmap v287: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 139 MiB data, 473 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 3 op/s; 10440/32430 objects degraded (32.192%) 2022-05-13T23:27:34.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:27:34 smithi047 ceph-mon[77147]: pgmap v287: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 139 MiB data, 473 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 3 op/s; 10440/32430 objects degraded (32.192%) 2022-05-13T23:27:36.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:27:36 smithi115 ceph-mon[78593]: pgmap v288: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 139 MiB data, 473 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s; 10440/32430 objects degraded (32.192%) 2022-05-13T23:27:36.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:27:36 smithi115 ceph-mon[78593]: 1 slow requests, 0 included below; oldest blocked for > 619.084822 secs 2022-05-13T23:27:36.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:27:36 smithi047 ceph-mon[77147]: pgmap v288: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 139 MiB data, 473 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s; 10440/32430 objects degraded (32.192%) 2022-05-13T23:27:36.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:27:36 smithi047 ceph-mon[77147]: 1 slow requests, 0 included below; oldest blocked for > 619.084822 secs 2022-05-13T23:27:37.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:27:37 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:27:37.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:27:37 smithi115 ceph-mon[78593]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2022-05-13T23:27:37.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:27:37 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:27:37.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:27:37 smithi047 ceph-mon[77147]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2022-05-13T23:27:38.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:27:38 smithi047 ceph-mon[77147]: pgmap v289: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 139 MiB data, 473 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 3 op/s; 10440/32430 objects degraded (32.192%) 2022-05-13T23:27:38.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:27:38 smithi047 ceph-mon[77147]: 1 slow requests, 0 included below; oldest blocked for > 291.106212 secs 2022-05-13T23:27:38.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:27:38 smithi115 ceph-mon[78593]: pgmap v289: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 139 MiB data, 473 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 3 op/s; 10440/32430 objects degraded (32.192%) 2022-05-13T23:27:38.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:27:38 smithi115 ceph-mon[78593]: 1 slow requests, 0 included below; oldest blocked for > 291.106212 secs 2022-05-13T23:27:40.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:27:40 smithi047 ceph-mon[77147]: pgmap v290: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 139 MiB data, 473 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s; 10440/32430 objects degraded (32.192%) 2022-05-13T23:27:40.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:27:40 smithi115 ceph-mon[78593]: pgmap v290: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 139 MiB data, 473 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s; 10440/32430 objects degraded (32.192%) 2022-05-13T23:27:41.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:27:41 smithi047 ceph-mon[77147]: 1 slow requests, 0 included below; oldest blocked for > 624.084945 secs 2022-05-13T23:27:41.806 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:27:41 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:27:41.807 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:27:41 smithi047 ceph-mon[77147]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2022-05-13T23:27:41.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:27:41 smithi115 ceph-mon[78593]: 1 slow requests, 0 included below; oldest blocked for > 624.084945 secs 2022-05-13T23:27:41.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:27:41 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:27:41.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:27:41 smithi115 ceph-mon[78593]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2022-05-13T23:27:42.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:27:42 smithi047 ceph-mon[77147]: pgmap v291: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 139 MiB data, 473 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 3 op/s; 10440/32430 objects degraded (32.192%) 2022-05-13T23:27:42.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:27:42 smithi115 ceph-mon[78593]: pgmap v291: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 139 MiB data, 473 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 3 op/s; 10440/32430 objects degraded (32.192%) 2022-05-13T23:27:43.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:27:43 smithi047 ceph-mon[77147]: 1 slow requests, 0 included below; oldest blocked for > 296.106341 secs 2022-05-13T23:27:43.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:27:43 smithi047 ceph-mon[77147]: Marking osd.0 out (has been down for 602 seconds) 2022-05-13T23:27:43.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:27:43 smithi047 ceph-mon[77147]: Marking osd.9 out (has been down for 602 seconds) 2022-05-13T23:27:43.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:27:43 smithi047 ceph-mon[77147]: Health check update: 2 osds down (OSD_DOWN) 2022-05-13T23:27:43.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:27:43 smithi047 ceph-mon[77147]: osdmap e93: 12 total, 8 up, 10 in 2022-05-13T23:27:43.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:27:43 smithi115 ceph-mon[78593]: 1 slow requests, 0 included below; oldest blocked for > 296.106341 secs 2022-05-13T23:27:43.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:27:43 smithi115 ceph-mon[78593]: Marking osd.0 out (has been down for 602 seconds) 2022-05-13T23:27:43.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:27:43 smithi115 ceph-mon[78593]: Marking osd.9 out (has been down for 602 seconds) 2022-05-13T23:27:43.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:27:43 smithi115 ceph-mon[78593]: Health check update: 2 osds down (OSD_DOWN) 2022-05-13T23:27:43.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:27:43 smithi115 ceph-mon[78593]: osdmap e93: 12 total, 8 up, 10 in 2022-05-13T23:27:44.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:27:44 smithi047 ceph-mon[77147]: pgmap v293: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 139 MiB data, 473 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 3 op/s; 10440/32430 objects degraded (32.192%) 2022-05-13T23:27:44.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:27:44 smithi047 ceph-mon[77147]: osdmap e94: 12 total, 8 up, 10 in 2022-05-13T23:27:44.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:27:44 smithi115 ceph-mon[78593]: pgmap v293: 97 pgs: 21 undersized+degraded+peered, 46 active+undersized+degraded, 1 unknown, 1 undersized+peered, 28 active+clean; 139 MiB data, 473 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 3 op/s; 10440/32430 objects degraded (32.192%) 2022-05-13T23:27:44.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:27:44 smithi115 ceph-mon[78593]: osdmap e94: 12 total, 8 up, 10 in 2022-05-13T23:27:46.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:27:46 smithi115 ceph-mon[78593]: osdmap e95: 12 total, 8 up, 10 in 2022-05-13T23:27:46.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:27:46 smithi115 ceph-mon[78593]: pgmap v296: 97 pgs: 32 peering, 9 undersized+degraded+peered, 27 active+undersized+degraded, 1 unknown, 28 active+clean; 139 MiB data, 473 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 2 op/s; 5179/32430 objects degraded (15.970%) 2022-05-13T23:27:46.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:27:46 smithi115 ceph-mon[78593]: 1 slow requests, 0 included below; oldest blocked for > 629.085027 secs 2022-05-13T23:27:46.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:27:46 smithi047 ceph-mon[77147]: osdmap e95: 12 total, 8 up, 10 in 2022-05-13T23:27:46.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:27:46 smithi047 ceph-mon[77147]: pgmap v296: 97 pgs: 32 peering, 9 undersized+degraded+peered, 27 active+undersized+degraded, 1 unknown, 28 active+clean; 139 MiB data, 473 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 2 op/s; 5179/32430 objects degraded (15.970%) 2022-05-13T23:27:46.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:27:46 smithi047 ceph-mon[77147]: 1 slow requests, 0 included below; oldest blocked for > 629.085027 secs 2022-05-13T23:27:47.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:27:47 smithi115 ceph-mon[78593]: Health check update: Reduced data availability: 23 pgs inactive, 31 pgs peering (PG_AVAILABILITY) 2022-05-13T23:27:47.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:27:47 smithi115 ceph-mon[78593]: Health check update: Degraded data redundancy: 5179/32430 objects degraded (15.970%), 36 pgs degraded, 36 pgs undersized (PG_DEGRADED) 2022-05-13T23:27:47.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:27:47 smithi047 ceph-mon[77147]: Health check update: Reduced data availability: 23 pgs inactive, 31 pgs peering (PG_AVAILABILITY) 2022-05-13T23:27:47.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:27:47 smithi047 ceph-mon[77147]: Health check update: Degraded data redundancy: 5179/32430 objects degraded (15.970%), 36 pgs degraded, 36 pgs undersized (PG_DEGRADED) 2022-05-13T23:27:48.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:27:48 smithi047 ceph-mon[77147]: pgmap v297: 97 pgs: 32 peering, 9 undersized+degraded+peered, 27 active+undersized+degraded, 1 unknown, 28 active+clean; 139 MiB data, 473 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 2 op/s; 5179/32430 objects degraded (15.970%) 2022-05-13T23:27:48.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:27:48 smithi047 ceph-mon[77147]: 1 slow requests, 0 included below; oldest blocked for > 301.106422 secs 2022-05-13T23:27:48.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:27:48 smithi047 ceph-mon[77147]: Marking osd.3 out (has been down for 604 seconds) 2022-05-13T23:27:48.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:27:48 smithi047 ceph-mon[77147]: Marking osd.6 out (has been down for 604 seconds) 2022-05-13T23:27:48.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:27:48 smithi047 ceph-mon[77147]: Health check cleared: OSD_DOWN (was: 2 osds down) 2022-05-13T23:27:48.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:27:48 smithi047 ceph-mon[77147]: Health check cleared: OSD_HOST_DOWN (was: 1 host (4 osds) down) 2022-05-13T23:27:48.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:27:48 smithi047 ceph-mon[77147]: osdmap e96: 12 total, 8 up, 8 in 2022-05-13T23:27:48.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:27:48 smithi115 ceph-mon[78593]: pgmap v297: 97 pgs: 32 peering, 9 undersized+degraded+peered, 27 active+undersized+degraded, 1 unknown, 28 active+clean; 139 MiB data, 473 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 2 op/s; 5179/32430 objects degraded (15.970%) 2022-05-13T23:27:48.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:27:48 smithi115 ceph-mon[78593]: 1 slow requests, 0 included below; oldest blocked for > 301.106422 secs 2022-05-13T23:27:48.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:27:48 smithi115 ceph-mon[78593]: Marking osd.3 out (has been down for 604 seconds) 2022-05-13T23:27:48.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:27:48 smithi115 ceph-mon[78593]: Marking osd.6 out (has been down for 604 seconds) 2022-05-13T23:27:48.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:27:48 smithi115 ceph-mon[78593]: Health check cleared: OSD_DOWN (was: 2 osds down) 2022-05-13T23:27:48.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:27:48 smithi115 ceph-mon[78593]: Health check cleared: OSD_HOST_DOWN (was: 1 host (4 osds) down) 2022-05-13T23:27:48.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:27:48 smithi115 ceph-mon[78593]: osdmap e96: 12 total, 8 up, 8 in 2022-05-13T23:27:50.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:27:50 smithi115 ceph-mon[78593]: osdmap e97: 12 total, 8 up, 8 in 2022-05-13T23:27:50.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:27:50 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:27:50.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:27:50 smithi115 ceph-mon[78593]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2022-05-13T23:27:50.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:27:50 smithi115 ceph-mon[78593]: pgmap v300: 97 pgs: 3 active+recovering+undersized+degraded+remapped+wait, 3 active+undersized+remapped+wait, 8 active+recovery_wait+undersized+degraded+remapped+wait, 28 active+clean, 1 unknown, 47 peering, 7 active+clean+wait; 135 MiB data, 484 MiB used, 715 GiB / 715 GiB avail; 1719/32409 objects degraded (5.304%); 1.7 MiB/s, 1.52k keys/s, 169 objects/s recovering 2022-05-13T23:27:50.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:27:50 smithi047 ceph-mon[77147]: osdmap e97: 12 total, 8 up, 8 in 2022-05-13T23:27:50.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:27:50 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:27:50.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:27:50 smithi047 ceph-mon[77147]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2022-05-13T23:27:50.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:27:50 smithi047 ceph-mon[77147]: pgmap v300: 97 pgs: 3 active+recovering+undersized+degraded+remapped+wait, 3 active+undersized+remapped+wait, 8 active+recovery_wait+undersized+degraded+remapped+wait, 28 active+clean, 1 unknown, 47 peering, 7 active+clean+wait; 135 MiB data, 484 MiB used, 715 GiB / 715 GiB avail; 1719/32409 objects degraded (5.304%); 1.7 MiB/s, 1.52k keys/s, 169 objects/s recovering 2022-05-13T23:27:51.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:27:51 smithi115 ceph-mon[78593]: osdmap e98: 12 total, 8 up, 8 in 2022-05-13T23:27:51.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:27:51 smithi115 ceph-mon[78593]: 1 slow requests, 0 included below; oldest blocked for > 634.085116 secs 2022-05-13T23:27:51.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:27:51 smithi047 ceph-mon[77147]: osdmap e98: 12 total, 8 up, 8 in 2022-05-13T23:27:51.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:27:51 smithi047 ceph-mon[77147]: 1 slow requests, 0 included below; oldest blocked for > 634.085116 secs 2022-05-13T23:27:52.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:27:52 smithi047 ceph-mon[77147]: osdmap e99: 12 total, 8 up, 8 in 2022-05-13T23:27:52.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:27:52 smithi047 ceph-mon[77147]: pgmap v303: 97 pgs: 3 active+recovering+undersized+degraded+remapped+wait, 3 active+undersized+remapped+wait, 8 active+recovery_wait+undersized+degraded+remapped+wait, 28 active+clean, 1 unknown, 47 peering, 7 active+clean+wait; 135 MiB data, 484 MiB used, 715 GiB / 715 GiB avail; 1719/32409 objects degraded (5.304%); 2.2 MiB/s, 1.96k keys/s, 218 objects/s recovering 2022-05-13T23:27:52.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:27:52 smithi115 ceph-mon[78593]: osdmap e99: 12 total, 8 up, 8 in 2022-05-13T23:27:52.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:27:52 smithi115 ceph-mon[78593]: pgmap v303: 97 pgs: 3 active+recovering+undersized+degraded+remapped+wait, 3 active+undersized+remapped+wait, 8 active+recovery_wait+undersized+degraded+remapped+wait, 28 active+clean, 1 unknown, 47 peering, 7 active+clean+wait; 135 MiB data, 484 MiB used, 715 GiB / 715 GiB avail; 1719/32409 objects degraded (5.304%); 2.2 MiB/s, 1.96k keys/s, 218 objects/s recovering 2022-05-13T23:27:53.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:27:53 smithi047 ceph-mon[77147]: osdmap e100: 12 total, 8 up, 8 in 2022-05-13T23:27:53.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:27:53 smithi047 ceph-mon[77147]: 1 slow requests, 0 included below; oldest blocked for > 306.106515 secs 2022-05-13T23:27:53.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:27:53 smithi047 ceph-mon[77147]: Health check update: Reduced data availability: 10 pgs inactive (PG_AVAILABILITY) 2022-05-13T23:27:53.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:27:53 smithi047 ceph-mon[77147]: Health check update: Degraded data redundancy: 1719/32409 objects degraded (5.304%), 11 pgs degraded (PG_DEGRADED) 2022-05-13T23:27:53.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:27:53 smithi115 ceph-mon[78593]: osdmap e100: 12 total, 8 up, 8 in 2022-05-13T23:27:53.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:27:53 smithi115 ceph-mon[78593]: 1 slow requests, 0 included below; oldest blocked for > 306.106515 secs 2022-05-13T23:27:53.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:27:53 smithi115 ceph-mon[78593]: Health check update: Reduced data availability: 10 pgs inactive (PG_AVAILABILITY) 2022-05-13T23:27:53.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:27:53 smithi115 ceph-mon[78593]: Health check update: Degraded data redundancy: 1719/32409 objects degraded (5.304%), 11 pgs degraded (PG_DEGRADED) 2022-05-13T23:27:54.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:27:54 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:27:54.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:27:54 smithi047 ceph-mon[77147]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2022-05-13T23:27:54.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:27:54 smithi047 ceph-mon[77147]: pgmap v305: 97 pgs: 3 active+recovering+undersized+degraded+remapped+wait, 3 active+undersized+remapped+wait, 8 active+recovery_wait+undersized+degraded+remapped+wait, 28 active+clean, 1 unknown, 47 peering, 7 active+clean+wait; 135 MiB data, 484 MiB used, 715 GiB / 715 GiB avail; 1719/32409 objects degraded (5.304%) 2022-05-13T23:27:54.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:27:54 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:27:54.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:27:54 smithi115 ceph-mon[78593]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2022-05-13T23:27:54.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:27:54 smithi115 ceph-mon[78593]: pgmap v305: 97 pgs: 3 active+recovering+undersized+degraded+remapped+wait, 3 active+undersized+remapped+wait, 8 active+recovery_wait+undersized+degraded+remapped+wait, 28 active+clean, 1 unknown, 47 peering, 7 active+clean+wait; 135 MiB data, 484 MiB used, 715 GiB / 715 GiB avail; 1719/32409 objects degraded (5.304%) 2022-05-13T23:27:56.614 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:27:56 smithi115 ceph-mon[78593]: pgmap v306: 97 pgs: 1 active+recovery_wait+degraded, 1 active+recovering+undersized+degraded+remapped+wait, 42 active+recovery_wait+undersized+degraded+remapped+wait, 4 active+recovery_wait+degraded+wait, 28 active+clean, 1 unknown, 20 active+clean+wait; 139 MiB data, 517 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 684 KiB/s wr, 4 op/s; 9220/32430 objects degraded (28.430%); 2.1 MiB/s, 1.39k keys/s, 22 objects/s recovering 2022-05-13T23:27:56.615 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:27:56 smithi115 ceph-mon[78593]: 1 slow requests, 0 included below; oldest blocked for > 639.085201 secs 2022-05-13T23:27:56.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:27:56 smithi047 ceph-mon[77147]: pgmap v306: 97 pgs: 1 active+recovery_wait+degraded, 1 active+recovering+undersized+degraded+remapped+wait, 42 active+recovery_wait+undersized+degraded+remapped+wait, 4 active+recovery_wait+degraded+wait, 28 active+clean, 1 unknown, 20 active+clean+wait; 139 MiB data, 517 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 684 KiB/s wr, 4 op/s; 9220/32430 objects degraded (28.430%); 2.1 MiB/s, 1.39k keys/s, 22 objects/s recovering 2022-05-13T23:27:56.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:27:56 smithi047 ceph-mon[77147]: 1 slow requests, 0 included below; oldest blocked for > 639.085201 secs 2022-05-13T23:27:58.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:27:58 smithi115 ceph-mon[78593]: pgmap v307: 97 pgs: 1 active+recovery_wait+degraded, 1 active+recovering+undersized+degraded+remapped+wait, 42 active+recovery_wait+undersized+degraded+remapped+wait, 4 active+recovery_wait+degraded+wait, 28 active+clean, 1 unknown, 20 active+clean+wait; 139 MiB data, 517 MiB used, 715 GiB / 715 GiB avail; 859 B/s rd, 574 KiB/s wr, 3 op/s; 9220/32430 objects degraded (28.430%); 1.7 MiB/s, 1.17k keys/s, 18 objects/s recovering 2022-05-13T23:27:58.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:27:58 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:27:58.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:27:58 smithi115 ceph-mon[78593]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2022-05-13T23:27:58.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:27:58 smithi115 ceph-mon[78593]: 1 slow requests, 0 included below; oldest blocked for > 311.106607 secs 2022-05-13T23:27:58.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:27:58 smithi115 ceph-mon[78593]: Health check update: Reduced data availability: 1 pg inactive (PG_AVAILABILITY) 2022-05-13T23:27:58.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:27:58 smithi115 ceph-mon[78593]: Health check update: Degraded data redundancy: 9220/32430 objects degraded (28.430%), 48 pgs degraded (PG_DEGRADED) 2022-05-13T23:27:58.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:27:58 smithi047 ceph-mon[77147]: pgmap v307: 97 pgs: 1 active+recovery_wait+degraded, 1 active+recovering+undersized+degraded+remapped+wait, 42 active+recovery_wait+undersized+degraded+remapped+wait, 4 active+recovery_wait+degraded+wait, 28 active+clean, 1 unknown, 20 active+clean+wait; 139 MiB data, 517 MiB used, 715 GiB / 715 GiB avail; 859 B/s rd, 574 KiB/s wr, 3 op/s; 9220/32430 objects degraded (28.430%); 1.7 MiB/s, 1.17k keys/s, 18 objects/s recovering 2022-05-13T23:27:58.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:27:58 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:27:58.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:27:58 smithi047 ceph-mon[77147]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2022-05-13T23:27:58.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:27:58 smithi047 ceph-mon[77147]: 1 slow requests, 0 included below; oldest blocked for > 311.106607 secs 2022-05-13T23:27:58.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:27:58 smithi047 ceph-mon[77147]: Health check update: Reduced data availability: 1 pg inactive (PG_AVAILABILITY) 2022-05-13T23:27:58.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:27:58 smithi047 ceph-mon[77147]: Health check update: Degraded data redundancy: 9220/32430 objects degraded (28.430%), 48 pgs degraded (PG_DEGRADED) 2022-05-13T23:28:00.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:28:00 smithi115 ceph-mon[78593]: pgmap v308: 97 pgs: 15 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped+wait, 2 active+recovery_wait+degraded, 27 active+recovery_wait+undersized+degraded+remapped+wait, 3 active+recovery_wait+degraded+wait, 39 active+clean, 1 unknown, 9 active+clean+wait; 139 MiB data, 517 MiB used, 715 GiB / 715 GiB avail; 288 KiB/s rd, 510 KiB/s wr, 7 op/s; 9102/32430 objects degraded (28.067%); 1.5 MiB/s, 1.03k keys/s, 24 objects/s recovering 2022-05-13T23:28:00.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:28:00 smithi047 ceph-mon[77147]: pgmap v308: 97 pgs: 15 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped+wait, 2 active+recovery_wait+degraded, 27 active+recovery_wait+undersized+degraded+remapped+wait, 3 active+recovery_wait+degraded+wait, 39 active+clean, 1 unknown, 9 active+clean+wait; 139 MiB data, 517 MiB used, 715 GiB / 715 GiB avail; 288 KiB/s rd, 510 KiB/s wr, 7 op/s; 9102/32430 objects degraded (28.067%); 1.5 MiB/s, 1.03k keys/s, 24 objects/s recovering 2022-05-13T23:28:01.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:28:01 smithi115 ceph-mon[78593]: MDS health message cleared (mds.?): 1 slow requests are blocked > 180 secs 2022-05-13T23:28:01.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:28:01 smithi047 ceph-mon[77147]: MDS health message cleared (mds.?): 1 slow requests are blocked > 180 secs 2022-05-13T23:28:02.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:28:02 smithi115 ceph-mon[78593]: pgmap v309: 97 pgs: 15 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped+wait, 2 active+recovery_wait+degraded, 27 active+recovery_wait+undersized+degraded+remapped+wait, 3 active+recovery_wait+degraded+wait, 39 active+clean, 1 unknown, 9 active+clean+wait; 139 MiB data, 517 MiB used, 715 GiB / 715 GiB avail; 233 KiB/s rd, 414 KiB/s wr, 6 op/s; 9102/32430 objects degraded (28.067%); 1.3 MiB/s, 834 keys/s, 19 objects/s recovering 2022-05-13T23:28:02.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:28:02 smithi115 ceph-mon[78593]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2022-05-13T23:28:02.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:28:02 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:28:02.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:28:02 smithi115 ceph-mon[78593]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2022-05-13T23:28:02.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:28:02 smithi047 ceph-mon[77147]: pgmap v309: 97 pgs: 15 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped+wait, 2 active+recovery_wait+degraded, 27 active+recovery_wait+undersized+degraded+remapped+wait, 3 active+recovery_wait+degraded+wait, 39 active+clean, 1 unknown, 9 active+clean+wait; 139 MiB data, 517 MiB used, 715 GiB / 715 GiB avail; 233 KiB/s rd, 414 KiB/s wr, 6 op/s; 9102/32430 objects degraded (28.067%); 1.3 MiB/s, 834 keys/s, 19 objects/s recovering 2022-05-13T23:28:02.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:28:02 smithi047 ceph-mon[77147]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2022-05-13T23:28:02.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:28:02 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:28:02.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:28:02 smithi047 ceph-mon[77147]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2022-05-13T23:28:03.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:28:03 smithi115 ceph-mon[78593]: 1 slow requests, 0 included below; oldest blocked for > 316.106685 secs 2022-05-13T23:28:03.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:28:03 smithi115 ceph-mon[78593]: Health check update: Degraded data redundancy: 9102/32430 objects degraded (28.067%), 48 pgs degraded (PG_DEGRADED) 2022-05-13T23:28:03.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:28:03 smithi115 ceph-mon[78593]: osdmap e101: 12 total, 8 up, 8 in 2022-05-13T23:28:03.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:28:03 smithi047 ceph-mon[77147]: 1 slow requests, 0 included below; oldest blocked for > 316.106685 secs 2022-05-13T23:28:03.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:28:03 smithi047 ceph-mon[77147]: Health check update: Degraded data redundancy: 9102/32430 objects degraded (28.067%), 48 pgs degraded (PG_DEGRADED) 2022-05-13T23:28:03.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:28:03 smithi047 ceph-mon[77147]: osdmap e101: 12 total, 8 up, 8 in 2022-05-13T23:28:04.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:28:04 smithi115 ceph-mon[78593]: pgmap v311: 97 pgs: 15 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped+wait, 2 active+recovery_wait+degraded, 27 active+recovery_wait+undersized+degraded+remapped+wait, 3 active+recovery_wait+degraded+wait, 39 active+clean, 1 unknown, 9 active+clean+wait; 139 MiB data, 517 MiB used, 715 GiB / 715 GiB avail; 233 KiB/s rd, 414 KiB/s wr, 6 op/s; 9102/32430 objects degraded (28.067%); 1.3 MiB/s, 834 keys/s, 19 objects/s recovering 2022-05-13T23:28:04.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:28:04 smithi115 ceph-mon[78593]: osdmap e102: 12 total, 8 up, 8 in 2022-05-13T23:28:04.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:28:04 smithi047 ceph-mon[77147]: pgmap v311: 97 pgs: 15 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped+wait, 2 active+recovery_wait+degraded, 27 active+recovery_wait+undersized+degraded+remapped+wait, 3 active+recovery_wait+degraded+wait, 39 active+clean, 1 unknown, 9 active+clean+wait; 139 MiB data, 517 MiB used, 715 GiB / 715 GiB avail; 233 KiB/s rd, 414 KiB/s wr, 6 op/s; 9102/32430 objects degraded (28.067%); 1.3 MiB/s, 834 keys/s, 19 objects/s recovering 2022-05-13T23:28:04.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:28:04 smithi047 ceph-mon[77147]: osdmap e102: 12 total, 8 up, 8 in 2022-05-13T23:28:06.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:28:06 smithi047 ceph-mon[77147]: osdmap e103: 12 total, 8 up, 8 in 2022-05-13T23:28:06.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:28:06 smithi047 ceph-mon[77147]: pgmap v314: 97 pgs: 2 peering, 3 active+recovering+undersized+degraded+remapped, 3 active+undersized+remapped, 35 active+recovery_wait+undersized+degraded+remapped, 4 active+recovery_wait+degraded, 49 active+clean, 1 unknown; 144 MiB data, 568 MiB used, 715 GiB / 715 GiB avail; 758 KiB/s rd, 803 KiB/s wr, 366 op/s; 6919/32457 objects degraded (21.317%); 4.1 MiB/s, 12 keys/s, 204 objects/s recovering 2022-05-13T23:28:06.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:28:06 smithi115 ceph-mon[78593]: osdmap e103: 12 total, 8 up, 8 in 2022-05-13T23:28:06.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:28:06 smithi115 ceph-mon[78593]: pgmap v314: 97 pgs: 2 peering, 3 active+recovering+undersized+degraded+remapped, 3 active+undersized+remapped, 35 active+recovery_wait+undersized+degraded+remapped, 4 active+recovery_wait+degraded, 49 active+clean, 1 unknown; 144 MiB data, 568 MiB used, 715 GiB / 715 GiB avail; 758 KiB/s rd, 803 KiB/s wr, 366 op/s; 6919/32457 objects degraded (21.317%); 4.1 MiB/s, 12 keys/s, 204 objects/s recovering 2022-05-13T23:28:07.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:28:07 smithi047 ceph-mon[77147]: osdmap e104: 12 total, 8 up, 8 in 2022-05-13T23:28:07.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:28:07 smithi047 ceph-mon[77147]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2022-05-13T23:28:07.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:28:07 smithi115 ceph-mon[78593]: osdmap e104: 12 total, 8 up, 8 in 2022-05-13T23:28:07.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:28:07 smithi115 ceph-mon[78593]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2022-05-13T23:28:08.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:28:08 smithi047 ceph-mon[77147]: pgmap v316: 97 pgs: 2 peering, 3 active+recovering+undersized+degraded+remapped, 3 active+undersized+remapped, 35 active+recovery_wait+undersized+degraded+remapped, 4 active+recovery_wait+degraded, 49 active+clean, 1 unknown; 144 MiB data, 568 MiB used, 715 GiB / 715 GiB avail; 1.1 MiB/s rd, 1.1 MiB/s wr, 528 op/s; 6919/32457 objects degraded (21.317%); 5.9 MiB/s, 17 keys/s, 295 objects/s recovering 2022-05-13T23:28:08.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:28:08 smithi047 ceph-mon[77147]: 1 slow requests, 0 included below; oldest blocked for > 321.106803 secs 2022-05-13T23:28:08.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:28:08 smithi047 ceph-mon[77147]: Health check update: Degraded data redundancy: 6919/32457 objects degraded (21.317%), 42 pgs degraded (PG_DEGRADED) 2022-05-13T23:28:08.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:28:08 smithi115 ceph-mon[78593]: pgmap v316: 97 pgs: 2 peering, 3 active+recovering+undersized+degraded+remapped, 3 active+undersized+remapped, 35 active+recovery_wait+undersized+degraded+remapped, 4 active+recovery_wait+degraded, 49 active+clean, 1 unknown; 144 MiB data, 568 MiB used, 715 GiB / 715 GiB avail; 1.1 MiB/s rd, 1.1 MiB/s wr, 528 op/s; 6919/32457 objects degraded (21.317%); 5.9 MiB/s, 17 keys/s, 295 objects/s recovering 2022-05-13T23:28:08.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:28:08 smithi115 ceph-mon[78593]: 1 slow requests, 0 included below; oldest blocked for > 321.106803 secs 2022-05-13T23:28:08.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:28:08 smithi115 ceph-mon[78593]: Health check update: Degraded data redundancy: 6919/32457 objects degraded (21.317%), 42 pgs degraded (PG_DEGRADED) 2022-05-13T23:28:10.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:28:10 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:28:10.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:28:10 smithi047 ceph-mon[77147]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2022-05-13T23:28:10.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:28:10 smithi047 ceph-mon[77147]: pgmap v317: 97 pgs: 2 active+recovering+undersized+degraded+remapped, 31 active+recovery_wait+undersized+degraded+remapped, 4 active+recovery_wait+degraded, 59 active+clean, 1 unknown; 145 MiB data, 609 MiB used, 715 GiB / 715 GiB avail; 771 KiB/s rd, 940 KiB/s wr, 379 op/s; 5953/32457 objects degraded (18.341%); 5.4 MiB/s, 12 keys/s, 346 objects/s recovering 2022-05-13T23:28:10.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:28:10 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:28:10.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:28:10 smithi115 ceph-mon[78593]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2022-05-13T23:28:10.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:28:10 smithi115 ceph-mon[78593]: pgmap v317: 97 pgs: 2 active+recovering+undersized+degraded+remapped, 31 active+recovery_wait+undersized+degraded+remapped, 4 active+recovery_wait+degraded, 59 active+clean, 1 unknown; 145 MiB data, 609 MiB used, 715 GiB / 715 GiB avail; 771 KiB/s rd, 940 KiB/s wr, 379 op/s; 5953/32457 objects degraded (18.341%); 5.4 MiB/s, 12 keys/s, 346 objects/s recovering 2022-05-13T23:28:12.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:28:12 smithi115 ceph-mon[78593]: pgmap v318: 97 pgs: 2 active+recovering+undersized+degraded+remapped, 31 active+recovery_wait+undersized+degraded+remapped, 4 active+recovery_wait+degraded, 59 active+clean, 1 unknown; 145 MiB data, 609 MiB used, 715 GiB / 715 GiB avail; 11 KiB/s rd, 115 KiB/s wr, 10 op/s; 5953/32457 objects degraded (18.341%); 1.1 MiB/s, 118 objects/s recovering 2022-05-13T23:28:12.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:28:12 smithi047 ceph-mon[77147]: pgmap v318: 97 pgs: 2 active+recovering+undersized+degraded+remapped, 31 active+recovery_wait+undersized+degraded+remapped, 4 active+recovery_wait+degraded, 59 active+clean, 1 unknown; 145 MiB data, 609 MiB used, 715 GiB / 715 GiB avail; 11 KiB/s rd, 115 KiB/s wr, 10 op/s; 5953/32457 objects degraded (18.341%); 1.1 MiB/s, 118 objects/s recovering 2022-05-13T23:28:13.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:28:13 smithi115 ceph-mon[78593]: Health check update: Degraded data redundancy: 5953/32457 objects degraded (18.341%), 37 pgs degraded (PG_DEGRADED) 2022-05-13T23:28:13.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:28:13 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-13T23:28:13.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:28:13 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-13T23:28:13.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:28:13 smithi047 ceph-mon[77147]: Health check update: Degraded data redundancy: 5953/32457 objects degraded (18.341%), 37 pgs degraded (PG_DEGRADED) 2022-05-13T23:28:13.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:28:13 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-13T23:28:13.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:28:13 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-13T23:28:14.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:28:14 smithi115 ceph-mon[78593]: pgmap v319: 97 pgs: 2 active+recovering+undersized+degraded+remapped, 31 active+recovery_wait+undersized+degraded+remapped, 4 active+recovery_wait+degraded, 59 active+clean, 1 unknown; 145 MiB data, 609 MiB used, 715 GiB / 715 GiB avail; 9.5 KiB/s rd, 101 KiB/s wr, 9 op/s; 5953/32457 objects degraded (18.341%); 1020 KiB/s, 104 objects/s recovering 2022-05-13T23:28:14.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:28:14 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:28:14.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:28:14 smithi115 ceph-mon[78593]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2022-05-13T23:28:14.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:28:14 smithi047 ceph-mon[77147]: pgmap v319: 97 pgs: 2 active+recovering+undersized+degraded+remapped, 31 active+recovery_wait+undersized+degraded+remapped, 4 active+recovery_wait+degraded, 59 active+clean, 1 unknown; 145 MiB data, 609 MiB used, 715 GiB / 715 GiB avail; 9.5 KiB/s rd, 101 KiB/s wr, 9 op/s; 5953/32457 objects degraded (18.341%); 1020 KiB/s, 104 objects/s recovering 2022-05-13T23:28:14.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:28:14 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:28:14.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:28:14 smithi047 ceph-mon[77147]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2022-05-13T23:28:16.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:28:16 smithi115 ceph-mon[78593]: pgmap v320: 97 pgs: 2 active+recovering+undersized+degraded+remapped, 31 active+recovery_wait+undersized+degraded+remapped, 4 active+recovery_wait+degraded, 59 active+clean, 1 unknown; 145 MiB data, 616 MiB used, 715 GiB / 715 GiB avail; 17 KiB/s rd, 96 KiB/s wr, 16 op/s; 5737/32457 objects degraded (17.676%); 874 KiB/s, 96 objects/s recovering 2022-05-13T23:28:16.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:28:16 smithi115 ceph-mon[78593]: osdmap e105: 12 total, 8 up, 8 in 2022-05-13T23:28:16.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:28:16 smithi115 ceph-mon[78593]: MDS health message cleared (mds.?): 1 slow requests are blocked > 180 secs 2022-05-13T23:28:16.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:28:16 smithi047 ceph-mon[77147]: pgmap v320: 97 pgs: 2 active+recovering+undersized+degraded+remapped, 31 active+recovery_wait+undersized+degraded+remapped, 4 active+recovery_wait+degraded, 59 active+clean, 1 unknown; 145 MiB data, 616 MiB used, 715 GiB / 715 GiB avail; 17 KiB/s rd, 96 KiB/s wr, 16 op/s; 5737/32457 objects degraded (17.676%); 874 KiB/s, 96 objects/s recovering 2022-05-13T23:28:16.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:28:16 smithi047 ceph-mon[77147]: osdmap e105: 12 total, 8 up, 8 in 2022-05-13T23:28:16.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:28:16 smithi047 ceph-mon[77147]: MDS health message cleared (mds.?): 1 slow requests are blocked > 180 secs 2022-05-13T23:28:17.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:28:17 smithi115 ceph-mon[78593]: Health check cleared: MDS_SLOW_REQUEST (was: 1 MDSs report slow requests) 2022-05-13T23:28:17.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:28:17 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6836/2940957881,v1:172.21.15.115:6837/2940957881] up:active 2022-05-13T23:28:17.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:28:17 smithi115 ceph-mon[78593]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2022-05-13T23:28:17.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:28:17 smithi115 ceph-mon[78593]: osdmap e106: 12 total, 8 up, 8 in 2022-05-13T23:28:17.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:28:17 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:28:17.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:28:17 smithi115 ceph-mon[78593]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2022-05-13T23:28:17.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:28:17 smithi047 ceph-mon[77147]: Health check cleared: MDS_SLOW_REQUEST (was: 1 MDSs report slow requests) 2022-05-13T23:28:17.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:28:17 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6836/2940957881,v1:172.21.15.115:6837/2940957881] up:active 2022-05-13T23:28:17.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:28:17 smithi047 ceph-mon[77147]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2022-05-13T23:28:17.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:28:17 smithi047 ceph-mon[77147]: osdmap e106: 12 total, 8 up, 8 in 2022-05-13T23:28:17.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:28:17 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:28:17.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:28:17 smithi047 ceph-mon[77147]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2022-05-13T23:28:18.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:28:18 smithi115 ceph-mon[78593]: pgmap v323: 97 pgs: 2 active+recovering+undersized+degraded+remapped, 31 active+recovery_wait+undersized+degraded+remapped, 4 active+recovery_wait+degraded, 59 active+clean, 1 unknown; 145 MiB data, 616 MiB used, 715 GiB / 715 GiB avail; 12 KiB/s rd, 17 KiB/s wr, 11 op/s; 5737/32457 objects degraded (17.676%); 54 KiB/s, 14 objects/s recovering 2022-05-13T23:28:18.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:28:18 smithi115 ceph-mon[78593]: osdmap e107: 12 total, 8 up, 8 in 2022-05-13T23:28:18.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:28:18 smithi115 ceph-mon[78593]: Health check update: Degraded data redundancy: 5737/32457 objects degraded (17.676%), 37 pgs degraded (PG_DEGRADED) 2022-05-13T23:28:18.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:28:18 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-13T23:28:18.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:28:18 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-13T23:28:18.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:28:18 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-13T23:28:18.741 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:28:18 smithi047 ceph-mon[77147]: pgmap v323: 97 pgs: 2 active+recovering+undersized+degraded+remapped, 31 active+recovery_wait+undersized+degraded+remapped, 4 active+recovery_wait+degraded, 59 active+clean, 1 unknown; 145 MiB data, 616 MiB used, 715 GiB / 715 GiB avail; 12 KiB/s rd, 17 KiB/s wr, 11 op/s; 5737/32457 objects degraded (17.676%); 54 KiB/s, 14 objects/s recovering 2022-05-13T23:28:18.741 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:28:18 smithi047 ceph-mon[77147]: osdmap e107: 12 total, 8 up, 8 in 2022-05-13T23:28:18.741 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:28:18 smithi047 ceph-mon[77147]: Health check update: Degraded data redundancy: 5737/32457 objects degraded (17.676%), 37 pgs degraded (PG_DEGRADED) 2022-05-13T23:28:18.742 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:28:18 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-13T23:28:18.742 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:28:18 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-13T23:28:18.742 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:28:18 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-13T23:28:20.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:28:19 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' 2022-05-13T23:28:20.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:28:19 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' 2022-05-13T23:28:20.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:28:19 smithi047 ceph-mon[77147]: pgmap v325: 97 pgs: 3 active+recovering+undersized+degraded+remapped, 28 active+recovery_wait+undersized+degraded+remapped, 4 active+recovery_wait+degraded, 61 active+clean, 1 unknown; 145 MiB data, 623 MiB used, 715 GiB / 715 GiB avail; 21 KiB/s rd, 25 KiB/s wr, 22 op/s; 5456/32457 objects degraded (16.810%); 185 KiB/s, 47 objects/s recovering 2022-05-13T23:28:20.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:28:19 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' 2022-05-13T23:28:20.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:28:19 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' 2022-05-13T23:28:20.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:28:19 smithi115 ceph-mon[78593]: pgmap v325: 97 pgs: 3 active+recovering+undersized+degraded+remapped, 28 active+recovery_wait+undersized+degraded+remapped, 4 active+recovery_wait+degraded, 61 active+clean, 1 unknown; 145 MiB data, 623 MiB used, 715 GiB / 715 GiB avail; 21 KiB/s rd, 25 KiB/s wr, 22 op/s; 5456/32457 objects degraded (16.810%); 185 KiB/s, 47 objects/s recovering 2022-05-13T23:28:21.754 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:28:21 smithi047 conmon[78521]: 2022-05-13T23:28:21.360+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:28:21.754 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:28:21 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-13T23:28:21.754 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:28:21 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-13T23:28:21.755 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:28:21 smithi047 conmon[78521]: yield 2022-05-13T23:28:21.755 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:28:21 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-13T23:28:21.755 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:28:21 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-13T23:28:21.756 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:28:21 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-13T23:28:21.756 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:28:21 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-13T23:28:21.756 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:28:21 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-13T23:28:21.756 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:28:21 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-13T23:28:21.757 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:28:21 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-13T23:28:21.757 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:28:21 smithi047 conmon[78521]: raise exceptions[0] 2022-05-13T23:28:21.757 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:28:21 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-13T23:28:21.758 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:28:21 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-13T23:28:21.758 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:28:21 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-13T23:28:21.758 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:28:21 smithi047 conmon[78521]: return (yield from fut) 2022-05-13T23:28:21.758 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:28:21 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-13T23:28:21.759 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:28:21 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-13T23:28:21.759 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:28:21 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:28:22.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:28:21 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:28:22.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:28:21 smithi047 ceph-mon[77147]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2022-05-13T23:28:22.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:28:21 smithi047 ceph-mon[77147]: pgmap v326: 97 pgs: 3 active+recovering+undersized+degraded+remapped, 28 active+recovery_wait+undersized+degraded+remapped, 4 active+recovery_wait+degraded, 61 active+clean, 1 unknown; 145 MiB data, 623 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 3.2 KiB/s wr, 6 op/s; 5456/32457 objects degraded (16.810%); 113 KiB/s, 28 objects/s recovering 2022-05-13T23:28:22.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:28:21 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' 2022-05-13T23:28:22.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:28:21 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:28:22.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-13T23:28:22.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-13T23:28:22.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-13T23:28:22.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-13T23:28:22.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-13T23:28:22.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-13T23:28:22.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-13T23:28:22.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-13T23:28:22.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-13T23:28:22.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-13T23:28:22.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-13T23:28:22.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-13T23:28:22.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-13T23:28:22.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-13T23:28:22.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-13T23:28:22.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-13T23:28:22.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-13T23:28:22.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:28:22.047 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:28:21 smithi047 ceph-mon[77147]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-13T23:28:22.047 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:28:21 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' 2022-05-13T23:28:22.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:28:21 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:28:22.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:28:21 smithi115 ceph-mon[78593]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2022-05-13T23:28:22.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:28:21 smithi115 ceph-mon[78593]: pgmap v326: 97 pgs: 3 active+recovering+undersized+degraded+remapped, 28 active+recovery_wait+undersized+degraded+remapped, 4 active+recovery_wait+degraded, 61 active+clean, 1 unknown; 145 MiB data, 623 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 3.2 KiB/s wr, 6 op/s; 5456/32457 objects degraded (16.810%); 113 KiB/s, 28 objects/s recovering 2022-05-13T23:28:22.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:28:21 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' 2022-05-13T23:28:22.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:28:21 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:28:22.109 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-13T23:28:22.109 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-13T23:28:22.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-13T23:28:22.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-13T23:28:22.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-13T23:28:22.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-13T23:28:22.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-13T23:28:22.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-13T23:28:22.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-13T23:28:22.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-13T23:28:22.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-13T23:28:22.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-13T23:28:22.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-13T23:28:22.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-13T23:28:22.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-13T23:28:22.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-13T23:28:22.113 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-13T23:28:22.113 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:28:22.113 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:28:21 smithi115 ceph-mon[78593]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-13T23:28:22.113 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:28:21 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' 2022-05-13T23:28:23.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:28:22 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-05-13T23:28:23.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:28:22 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-05-13T23:28:23.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:28:22 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-05-13T23:28:23.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:28:22 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-05-13T23:28:23.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:28:22 smithi047 ceph-mon[77147]: Adjusting osd_memory_target on smithi115 to 197.0M 2022-05-13T23:28:23.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:28:22 smithi047 ceph-mon[77147]: Unable to set osd_memory_target on smithi115 to 206629785: error parsing value: Value '206629785' is below minimum 939524096 2022-05-13T23:28:23.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:28:22 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-05-13T23:28:23.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:28:22 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-05-13T23:28:23.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:28:22 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-05-13T23:28:23.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:28:22 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-05-13T23:28:23.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:28:22 smithi047 ceph-mon[77147]: Adjusting osd_memory_target on smithi047 to 198.1M 2022-05-13T23:28:23.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:28:22 smithi047 ceph-mon[77147]: Unable to set osd_memory_target on smithi047 to 207755878: error parsing value: Value '207755878' is below minimum 939524096 2022-05-13T23:28:23.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:28:22 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-05-13T23:28:23.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:28:22 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-05-13T23:28:23.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:28:22 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-05-13T23:28:23.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:28:22 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-05-13T23:28:23.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:28:22 smithi115 ceph-mon[78593]: Adjusting osd_memory_target on smithi115 to 197.0M 2022-05-13T23:28:23.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:28:22 smithi115 ceph-mon[78593]: Unable to set osd_memory_target on smithi115 to 206629785: error parsing value: Value '206629785' is below minimum 939524096 2022-05-13T23:28:23.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:28:22 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-05-13T23:28:23.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:28:22 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-05-13T23:28:23.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:28:22 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-05-13T23:28:23.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:28:22 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-05-13T23:28:23.112 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:28:22 smithi115 ceph-mon[78593]: Adjusting osd_memory_target on smithi047 to 198.1M 2022-05-13T23:28:23.112 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:28:22 smithi115 ceph-mon[78593]: Unable to set osd_memory_target on smithi047 to 207755878: error parsing value: Value '207755878' is below minimum 939524096 2022-05-13T23:28:24.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:28:23 smithi047 ceph-mon[77147]: Health check update: Degraded data redundancy: 5456/32457 objects degraded (16.810%), 35 pgs degraded (PG_DEGRADED) 2022-05-13T23:28:24.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:28:23 smithi047 ceph-mon[77147]: pgmap v327: 97 pgs: 3 active+recovering+undersized+degraded+remapped, 28 active+recovery_wait+undersized+degraded+remapped, 4 active+recovery_wait+degraded, 61 active+clean, 1 unknown; 145 MiB data, 623 MiB used, 715 GiB / 715 GiB avail; 3.9 KiB/s rd, 2.4 KiB/s wr, 5 op/s; 5456/32457 objects degraded (16.810%); 85 KiB/s, 21 objects/s recovering 2022-05-13T23:28:24.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:28:23 smithi115 ceph-mon[78593]: Health check update: Degraded data redundancy: 5456/32457 objects degraded (16.810%), 35 pgs degraded (PG_DEGRADED) 2022-05-13T23:28:24.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:28:23 smithi115 ceph-mon[78593]: pgmap v327: 97 pgs: 3 active+recovering+undersized+degraded+remapped, 28 active+recovery_wait+undersized+degraded+remapped, 4 active+recovery_wait+degraded, 61 active+clean, 1 unknown; 145 MiB data, 623 MiB used, 715 GiB / 715 GiB avail; 3.9 KiB/s rd, 2.4 KiB/s wr, 5 op/s; 5456/32457 objects degraded (16.810%); 85 KiB/s, 21 objects/s recovering 2022-05-13T23:28:24.792 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:28:24 smithi047 conmon[78521]: 2022-05-13T23:28:24.431+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:28:24.792 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:28:24 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-13T23:28:24.792 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:28:24 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-13T23:28:24.793 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:28:24 smithi047 conmon[78521]: yield 2022-05-13T23:28:24.793 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:28:24 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-13T23:28:24.793 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:28:24 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-13T23:28:24.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:28:24 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-13T23:28:24.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:28:24 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-13T23:28:24.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:28:24 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-13T23:28:24.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:28:24 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-13T23:28:24.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:28:24 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-13T23:28:24.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:28:24 smithi047 conmon[78521]: raise exceptions[0] 2022-05-13T23:28:24.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:28:24 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-13T23:28:24.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:28:24 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-13T23:28:24.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:28:24 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-13T23:28:24.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:28:24 smithi047 conmon[78521]: return (yield from fut) 2022-05-13T23:28:24.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:28:24 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-13T23:28:24.797 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:28:24 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-13T23:28:24.797 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:28:24 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:28:24.797 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:28:24 smithi047 conmon[78521]: 2022-05-13T23:28:24.431+0000 7f5340779700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-13T23:28:24.798 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:28:24 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-13T23:28:24.798 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:28:24 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-13T23:28:24.798 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:28:24 smithi047 conmon[78521]: yield 2022-05-13T23:28:24.798 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:28:24 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-13T23:28:24.798 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:28:24 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-13T23:28:24.799 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:28:24 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-13T23:28:24.799 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:28:24 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-13T23:28:24.799 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:28:24 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-13T23:28:24.799 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:28:24 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-13T23:28:24.799 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:28:24 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-13T23:28:24.800 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:28:24 smithi047 conmon[78521]: raise exceptions[0] 2022-05-13T23:28:24.800 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:28:24 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-13T23:28:24.801 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:28:24 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-13T23:28:24.801 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:28:24 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-13T23:28:24.801 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:28:24 smithi047 conmon[78521]: return (yield from fut) 2022-05-13T23:28:24.802 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:28:24 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-13T23:28:24.802 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:28:24 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-13T23:28:24.802 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:28:24 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:28:24.802 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:28:24 smithi047 conmon[78521]: 2022-05-13T23:28:24.803 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:28:24 smithi047 conmon[78521]: During handling of the above exception, another exception occurred: 2022-05-13T23:28:24.803 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:28:24 smithi047 conmon[78521]: 2022-05-13T23:28:24.803 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:28:24 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-13T23:28:24.803 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:28:24 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-13T23:28:24.803 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:28:24 smithi047 conmon[78521]: return f(*arg) 2022-05-13T23:28:24.804 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:28:24 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-13T23:28:24.804 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:28:24 smithi047 conmon[78521]: self._write_client_files(client_files, host) 2022-05-13T23:28:24.804 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:28:24 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-13T23:28:24.804 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:28:24 smithi047 conmon[78521]: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-13T23:28:24.804 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:28:24 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-13T23:28:24.805 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:28:24 smithi047 conmon[78521]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-13T23:28:24.805 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:28:24 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-13T23:28:24.805 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:28:24 smithi047 conmon[78521]: return self.event_loop.get_result(coro) 2022-05-13T23:28:24.805 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:28:24 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-13T23:28:24.806 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:28:24 smithi047 conmon[78521]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-13T23:28:24.806 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:28:24 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-13T23:28:24.806 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:28:24 smithi047 conmon[78521]: return self.__get_result() 2022-05-13T23:28:24.806 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:28:24 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-13T23:28:24.806 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:28:24 smithi047 conmon[78521]: raise self._exception 2022-05-13T23:28:24.807 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:28:24 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-13T23:28:24.807 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:28:24 smithi047 conmon[78521]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-13T23:28:24.807 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:28:24 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-13T23:28:24.807 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:28:24 smithi047 conmon[78521]: conn = await self._remote_connection(host, addr) 2022-05-13T23:28:24.807 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:28:24 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-13T23:28:24.808 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:28:24 smithi047 conmon[78521]: raise 2022-05-13T23:28:24.808 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:28:24 smithi047 conmon[78521]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-13T23:28:24.808 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:28:24 smithi047 conmon[78521]: self.gen.throw(type, value, traceback) 2022-05-13T23:28:24.808 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:28:24 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-13T23:28:24.809 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:28:24 smithi047 conmon[78521]: raise OrchestratorError(msg) 2022-05-13T23:28:24.809 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:28:24 smithi047 conmon[78521]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:28:26.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:28:26 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:28:26.358 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-13T23:28:26.359 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-13T23:28:26.359 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-13T23:28:26.359 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-13T23:28:26.359 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-13T23:28:26.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-13T23:28:26.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-13T23:28:26.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-13T23:28:26.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-13T23:28:26.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-13T23:28:26.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-13T23:28:26.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-13T23:28:26.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-13T23:28:26.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-13T23:28:26.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-13T23:28:26.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-13T23:28:26.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-13T23:28:26.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:28:26.362 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:28:26 smithi115 ceph-mon[78593]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-13T23:28:26.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-13T23:28:26.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-13T23:28:26.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-13T23:28:26.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-13T23:28:26.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-13T23:28:26.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-13T23:28:26.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-13T23:28:26.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-13T23:28:26.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-13T23:28:26.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-13T23:28:26.366 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-13T23:28:26.366 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-13T23:28:26.366 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-13T23:28:26.366 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-13T23:28:26.366 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-13T23:28:26.367 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-13T23:28:26.367 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-13T23:28:26.367 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:28:26.367 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-13T23:28:26.368 INFO:journalctl@ceph.mon.c.smithi115.stdout: During handling of the above exception, another exception occurred: 2022-05-13T23:28:26.368 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-13T23:28:26.368 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-13T23:28:26.368 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-13T23:28:26.368 INFO:journalctl@ceph.mon.c.smithi115.stdout: return f(*arg) 2022-05-13T23:28:26.369 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-13T23:28:26.369 INFO:journalctl@ceph.mon.c.smithi115.stdout: self._write_client_files(client_files, host) 2022-05-13T23:28:26.369 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-13T23:28:26.369 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-13T23:28:26.370 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-13T23:28:26.370 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-13T23:28:26.370 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-13T23:28:26.370 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.event_loop.get_result(coro) 2022-05-13T23:28:26.371 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-13T23:28:26.371 INFO:journalctl@ceph.mon.c.smithi115.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-13T23:28:26.371 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-13T23:28:26.371 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.__get_result() 2022-05-13T23:28:26.371 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-13T23:28:26.372 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise self._exception 2022-05-13T23:28:26.372 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-13T23:28:26.372 INFO:journalctl@ceph.mon.c.smithi115.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-13T23:28:26.372 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-13T23:28:26.373 INFO:journalctl@ceph.mon.c.smithi115.stdout: conn = await self._remote_connection(host, addr) 2022-05-13T23:28:26.373 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-13T23:28:26.373 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise 2022-05-13T23:28:26.373 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-13T23:28:26.374 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.gen.throw(type, value, traceback) 2022-05-13T23:28:26.374 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-13T23:28:26.374 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OrchestratorError(msg) 2022-05-13T23:28:26.374 INFO:journalctl@ceph.mon.c.smithi115.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:28:26.374 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:28:26 smithi115 ceph-mon[78593]: osdmap e108: 12 total, 8 up, 8 in 2022-05-13T23:28:26.375 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:28:26 smithi115 ceph-mon[78593]: pgmap v329: 97 pgs: 3 active+recovering+undersized+degraded+remapped, 28 active+recovery_wait+undersized+degraded+remapped, 4 active+recovery_wait+degraded, 61 active+clean, 1 unknown; 145 MiB data, 630 MiB used, 715 GiB / 715 GiB avail; 8.9 KiB/s rd, 6.6 KiB/s wr, 10 op/s; 5201/32457 objects degraded (16.024%); 181 KiB/s, 42 objects/s recovering 2022-05-13T23:28:26.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:28:26 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:28:26.542 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-13T23:28:26.542 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-13T23:28:26.542 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-13T23:28:26.542 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-13T23:28:26.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-13T23:28:26.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-13T23:28:26.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-13T23:28:26.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-13T23:28:26.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-13T23:28:26.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-13T23:28:26.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-13T23:28:26.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-13T23:28:26.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-13T23:28:26.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-13T23:28:26.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-13T23:28:26.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-13T23:28:26.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-13T23:28:26.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:28:26.546 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:28:26 smithi047 ceph-mon[77147]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-13T23:28:26.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-13T23:28:26.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-13T23:28:26.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-13T23:28:26.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-13T23:28:26.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-13T23:28:26.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-13T23:28:26.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-13T23:28:26.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-13T23:28:26.548 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-13T23:28:26.548 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-13T23:28:26.548 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-13T23:28:26.548 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-13T23:28:26.548 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-13T23:28:26.549 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-13T23:28:26.549 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-13T23:28:26.549 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-13T23:28:26.549 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-13T23:28:26.550 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:28:26.550 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-13T23:28:26.550 INFO:journalctl@ceph.mon.b.smithi047.stdout: During handling of the above exception, another exception occurred: 2022-05-13T23:28:26.550 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-13T23:28:26.550 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-13T23:28:26.551 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-13T23:28:26.551 INFO:journalctl@ceph.mon.b.smithi047.stdout: return f(*arg) 2022-05-13T23:28:26.551 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-13T23:28:26.551 INFO:journalctl@ceph.mon.b.smithi047.stdout: self._write_client_files(client_files, host) 2022-05-13T23:28:26.551 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-13T23:28:26.552 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-13T23:28:26.552 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-13T23:28:26.552 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-13T23:28:26.552 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-13T23:28:26.553 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.event_loop.get_result(coro) 2022-05-13T23:28:26.553 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-13T23:28:26.553 INFO:journalctl@ceph.mon.b.smithi047.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-13T23:28:26.553 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-13T23:28:26.553 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.__get_result() 2022-05-13T23:28:26.554 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-13T23:28:26.554 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise self._exception 2022-05-13T23:28:26.554 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-13T23:28:26.554 INFO:journalctl@ceph.mon.b.smithi047.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-13T23:28:26.554 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-13T23:28:26.555 INFO:journalctl@ceph.mon.b.smithi047.stdout: conn = await self._remote_connection(host, addr) 2022-05-13T23:28:26.555 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-13T23:28:26.555 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise 2022-05-13T23:28:26.555 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-13T23:28:26.556 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.gen.throw(type, value, traceback) 2022-05-13T23:28:26.556 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-13T23:28:26.556 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OrchestratorError(msg) 2022-05-13T23:28:26.556 INFO:journalctl@ceph.mon.b.smithi047.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:28:26.556 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:28:26 smithi047 ceph-mon[77147]: osdmap e108: 12 total, 8 up, 8 in 2022-05-13T23:28:26.557 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:28:26 smithi047 ceph-mon[77147]: pgmap v329: 97 pgs: 3 active+recovering+undersized+degraded+remapped, 28 active+recovery_wait+undersized+degraded+remapped, 4 active+recovery_wait+degraded, 61 active+clean, 1 unknown; 145 MiB data, 630 MiB used, 715 GiB / 715 GiB avail; 8.9 KiB/s rd, 6.6 KiB/s wr, 10 op/s; 5201/32457 objects degraded (16.024%); 181 KiB/s, 42 objects/s recovering 2022-05-13T23:28:27.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:28:27 smithi047 ceph-mon[77147]: osdmap e109: 12 total, 8 up, 8 in 2022-05-13T23:28:27.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:28:27 smithi115 ceph-mon[78593]: osdmap e109: 12 total, 8 up, 8 in 2022-05-13T23:28:28.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:28:28 smithi047 ceph-mon[77147]: osdmap e110: 12 total, 8 up, 8 in 2022-05-13T23:28:28.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:28:28 smithi047 ceph-mon[77147]: pgmap v332: 97 pgs: 3 active+recovering+undersized+degraded+remapped, 28 active+recovery_wait+undersized+degraded+remapped, 4 active+recovery_wait+degraded, 61 active+clean, 1 unknown; 145 MiB data, 630 MiB used, 715 GiB / 715 GiB avail; 6.7 KiB/s rd, 5.7 KiB/s wr, 7 op/s; 5201/32457 objects degraded (16.024%); 128 KiB/s, 28 objects/s recovering 2022-05-13T23:28:28.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:28:28 smithi047 ceph-mon[77147]: Health check update: Degraded data redundancy: 5201/32457 objects degraded (16.024%), 35 pgs degraded (PG_DEGRADED) 2022-05-13T23:28:28.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:28:28 smithi115 ceph-mon[78593]: osdmap e110: 12 total, 8 up, 8 in 2022-05-13T23:28:28.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:28:28 smithi115 ceph-mon[78593]: pgmap v332: 97 pgs: 3 active+recovering+undersized+degraded+remapped, 28 active+recovery_wait+undersized+degraded+remapped, 4 active+recovery_wait+degraded, 61 active+clean, 1 unknown; 145 MiB data, 630 MiB used, 715 GiB / 715 GiB avail; 6.7 KiB/s rd, 5.7 KiB/s wr, 7 op/s; 5201/32457 objects degraded (16.024%); 128 KiB/s, 28 objects/s recovering 2022-05-13T23:28:28.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:28:28 smithi115 ceph-mon[78593]: Health check update: Degraded data redundancy: 5201/32457 objects degraded (16.024%), 35 pgs degraded (PG_DEGRADED) 2022-05-13T23:28:30.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:28:30 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:28:30.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:28:30 smithi047 ceph-mon[77147]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2022-05-13T23:28:30.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:28:30 smithi047 ceph-mon[77147]: pgmap v333: 97 pgs: 2 active+recovering+undersized+degraded+remapped, 26 active+recovery_wait+undersized+degraded+remapped, 4 active+recovery_wait+degraded, 64 active+clean, 1 unknown; 145 MiB data, 638 MiB used, 715 GiB / 715 GiB avail; 9.8 KiB/s rd, 7.0 KiB/s wr, 13 op/s; 4740/32457 objects degraded (14.604%); 436 KiB/s, 69 objects/s recovering 2022-05-13T23:28:30.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:28:30 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:28:30.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:28:30 smithi115 ceph-mon[78593]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2022-05-13T23:28:30.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:28:30 smithi115 ceph-mon[78593]: pgmap v333: 97 pgs: 2 active+recovering+undersized+degraded+remapped, 26 active+recovery_wait+undersized+degraded+remapped, 4 active+recovery_wait+degraded, 64 active+clean, 1 unknown; 145 MiB data, 638 MiB used, 715 GiB / 715 GiB avail; 9.8 KiB/s rd, 7.0 KiB/s wr, 13 op/s; 4740/32457 objects degraded (14.604%); 436 KiB/s, 69 objects/s recovering 2022-05-13T23:28:32.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:28:32 smithi047 ceph-mon[77147]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2022-05-13T23:28:32.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:28:32 smithi047 ceph-mon[77147]: pgmap v334: 97 pgs: 2 active+recovering+undersized+degraded+remapped, 26 active+recovery_wait+undersized+degraded+remapped, 4 active+recovery_wait+degraded, 64 active+clean, 1 unknown; 145 MiB data, 638 MiB used, 715 GiB / 715 GiB avail; 3.1 KiB/s rd, 1.3 KiB/s wr, 5 op/s; 4740/32457 objects degraded (14.604%); 300 KiB/s, 39 objects/s recovering 2022-05-13T23:28:32.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:28:32 smithi115 ceph-mon[78593]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2022-05-13T23:28:32.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:28:32 smithi115 ceph-mon[78593]: pgmap v334: 97 pgs: 2 active+recovering+undersized+degraded+remapped, 26 active+recovery_wait+undersized+degraded+remapped, 4 active+recovery_wait+degraded, 64 active+clean, 1 unknown; 145 MiB data, 638 MiB used, 715 GiB / 715 GiB avail; 3.1 KiB/s rd, 1.3 KiB/s wr, 5 op/s; 4740/32457 objects degraded (14.604%); 300 KiB/s, 39 objects/s recovering 2022-05-13T23:28:33.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:28:33 smithi047 ceph-mon[77147]: Health check update: Degraded data redundancy: 4740/32457 objects degraded (14.604%), 32 pgs degraded (PG_DEGRADED) 2022-05-13T23:28:33.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:28:33 smithi115 ceph-mon[78593]: Health check update: Degraded data redundancy: 4740/32457 objects degraded (14.604%), 32 pgs degraded (PG_DEGRADED) 2022-05-13T23:28:34.134 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:28:34 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:28:34.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:28:34 smithi047 ceph-mon[77147]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2022-05-13T23:28:34.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:28:34 smithi047 ceph-mon[77147]: pgmap v335: 97 pgs: 2 active+recovering+undersized+degraded+remapped, 26 active+recovery_wait+undersized+degraded+remapped, 4 active+recovery_wait+degraded, 64 active+clean, 1 unknown; 145 MiB data, 638 MiB used, 715 GiB / 715 GiB avail; 2.4 KiB/s rd, 1023 B/s wr, 4 op/s; 4740/32457 objects degraded (14.604%); 231 KiB/s, 30 objects/s recovering 2022-05-13T23:28:34.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:28:34 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:28:34.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:28:34 smithi115 ceph-mon[78593]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2022-05-13T23:28:34.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:28:34 smithi115 ceph-mon[78593]: pgmap v335: 97 pgs: 2 active+recovering+undersized+degraded+remapped, 26 active+recovery_wait+undersized+degraded+remapped, 4 active+recovery_wait+degraded, 64 active+clean, 1 unknown; 145 MiB data, 638 MiB used, 715 GiB / 715 GiB avail; 2.4 KiB/s rd, 1023 B/s wr, 4 op/s; 4740/32457 objects degraded (14.604%); 231 KiB/s, 30 objects/s recovering 2022-05-13T23:28:36.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:28:36 smithi115 ceph-mon[78593]: pgmap v336: 97 pgs: 2 active+recovering+undersized+degraded+remapped, 26 active+recovery_wait+undersized+degraded+remapped, 4 active+recovery_wait+degraded, 64 active+clean, 1 unknown; 146 MiB data, 649 MiB used, 715 GiB / 715 GiB avail; 49 KiB/s rd, 111 KiB/s wr, 9 op/s; 4504/32457 objects degraded (13.877%); 251 KiB/s, 39 objects/s recovering 2022-05-13T23:28:36.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:28:36 smithi047 ceph-mon[77147]: pgmap v336: 97 pgs: 2 active+recovering+undersized+degraded+remapped, 26 active+recovery_wait+undersized+degraded+remapped, 4 active+recovery_wait+degraded, 64 active+clean, 1 unknown; 146 MiB data, 649 MiB used, 715 GiB / 715 GiB avail; 49 KiB/s rd, 111 KiB/s wr, 9 op/s; 4504/32457 objects degraded (13.877%); 251 KiB/s, 39 objects/s recovering 2022-05-13T23:28:38.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:28:38 smithi115 ceph-mon[78593]: pgmap v337: 97 pgs: 2 active+recovering+undersized+degraded+remapped, 26 active+recovery_wait+undersized+degraded+remapped, 4 active+recovery_wait+degraded, 64 active+clean, 1 unknown; 146 MiB data, 649 MiB used, 715 GiB / 715 GiB avail; 45 KiB/s rd, 100 KiB/s wr, 8 op/s; 4504/32457 objects degraded (13.877%); 226 KiB/s, 35 objects/s recovering 2022-05-13T23:28:38.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:28:38 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:28:38.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:28:38 smithi115 ceph-mon[78593]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2022-05-13T23:28:38.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:28:38 smithi115 ceph-mon[78593]: osdmap e111: 12 total, 8 up, 8 in 2022-05-13T23:28:38.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:28:38 smithi115 ceph-mon[78593]: Health check update: Degraded data redundancy: 4504/32457 objects degraded (13.877%), 32 pgs degraded (PG_DEGRADED) 2022-05-13T23:28:38.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:28:38 smithi047 ceph-mon[77147]: pgmap v337: 97 pgs: 2 active+recovering+undersized+degraded+remapped, 26 active+recovery_wait+undersized+degraded+remapped, 4 active+recovery_wait+degraded, 64 active+clean, 1 unknown; 146 MiB data, 649 MiB used, 715 GiB / 715 GiB avail; 45 KiB/s rd, 100 KiB/s wr, 8 op/s; 4504/32457 objects degraded (13.877%); 226 KiB/s, 35 objects/s recovering 2022-05-13T23:28:38.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:28:38 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:28:38.795 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:28:38 smithi047 ceph-mon[77147]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2022-05-13T23:28:38.795 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:28:38 smithi047 ceph-mon[77147]: osdmap e111: 12 total, 8 up, 8 in 2022-05-13T23:28:38.795 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:28:38 smithi047 ceph-mon[77147]: Health check update: Degraded data redundancy: 4504/32457 objects degraded (13.877%), 32 pgs degraded (PG_DEGRADED) 2022-05-13T23:28:39.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:28:39 smithi115 ceph-mon[78593]: osdmap e112: 12 total, 8 up, 8 in 2022-05-13T23:28:39.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:28:39 smithi047 ceph-mon[77147]: osdmap e112: 12 total, 8 up, 8 in 2022-05-13T23:28:40.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:28:40 smithi115 ceph-mon[78593]: pgmap v340: 97 pgs: 2 active+recovering+undersized+degraded+remapped, 24 active+recovery_wait+undersized+degraded+remapped, 3 active+recovery_wait+degraded, 67 active+clean, 1 unknown; 146 MiB data, 651 MiB used, 715 GiB / 715 GiB avail; 56 KiB/s rd, 126 KiB/s wr, 10 op/s; 4298/32457 objects degraded (13.242%); 130 KiB/s, 7 keys/s, 29 objects/s recovering 2022-05-13T23:28:40.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:28:40 smithi047 ceph-mon[77147]: pgmap v340: 97 pgs: 2 active+recovering+undersized+degraded+remapped, 24 active+recovery_wait+undersized+degraded+remapped, 3 active+recovery_wait+degraded, 67 active+clean, 1 unknown; 146 MiB data, 651 MiB used, 715 GiB / 715 GiB avail; 56 KiB/s rd, 126 KiB/s wr, 10 op/s; 4298/32457 objects degraded (13.242%); 130 KiB/s, 7 keys/s, 29 objects/s recovering 2022-05-13T23:28:41.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:28:41 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:28:41.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:28:41 smithi115 ceph-mon[78593]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2022-05-13T23:28:41.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:28:41 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:28:41.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:28:41 smithi047 ceph-mon[77147]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2022-05-13T23:28:42.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:28:42 smithi115 ceph-mon[78593]: pgmap v341: 97 pgs: 2 active+recovering+undersized+degraded+remapped, 24 active+recovery_wait+undersized+degraded+remapped, 3 active+recovery_wait+degraded, 67 active+clean, 1 unknown; 146 MiB data, 651 MiB used, 715 GiB / 715 GiB avail; 56 KiB/s rd, 126 KiB/s wr, 10 op/s; 4298/32457 objects degraded (13.242%); 130 KiB/s, 7 keys/s, 29 objects/s recovering 2022-05-13T23:28:42.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:28:42 smithi047 ceph-mon[77147]: pgmap v341: 97 pgs: 2 active+recovering+undersized+degraded+remapped, 24 active+recovery_wait+undersized+degraded+remapped, 3 active+recovery_wait+degraded, 67 active+clean, 1 unknown; 146 MiB data, 651 MiB used, 715 GiB / 715 GiB avail; 56 KiB/s rd, 126 KiB/s wr, 10 op/s; 4298/32457 objects degraded (13.242%); 130 KiB/s, 7 keys/s, 29 objects/s recovering 2022-05-13T23:28:43.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:28:43 smithi115 ceph-mon[78593]: Health check update: Degraded data redundancy: 4298/32457 objects degraded (13.242%), 29 pgs degraded (PG_DEGRADED) 2022-05-13T23:28:43.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:28:43 smithi047 ceph-mon[77147]: Health check update: Degraded data redundancy: 4298/32457 objects degraded (13.242%), 29 pgs degraded (PG_DEGRADED) 2022-05-13T23:28:44.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:28:44 smithi115 ceph-mon[78593]: pgmap v342: 97 pgs: 2 active+recovering+undersized+degraded+remapped, 24 active+recovery_wait+undersized+degraded+remapped, 3 active+recovery_wait+degraded, 67 active+clean, 1 unknown; 146 MiB data, 651 MiB used, 715 GiB / 715 GiB avail; 1.9 KiB/s rd, 255 B/s wr, 3 op/s; 4298/32457 objects degraded (13.242%); 74 KiB/s, 7 keys/s, 14 objects/s recovering 2022-05-13T23:28:44.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:28:44 smithi047 ceph-mon[77147]: pgmap v342: 97 pgs: 2 active+recovering+undersized+degraded+remapped, 24 active+recovery_wait+undersized+degraded+remapped, 3 active+recovery_wait+degraded, 67 active+clean, 1 unknown; 146 MiB data, 651 MiB used, 715 GiB / 715 GiB avail; 1.9 KiB/s rd, 255 B/s wr, 3 op/s; 4298/32457 objects degraded (13.242%); 74 KiB/s, 7 keys/s, 14 objects/s recovering 2022-05-13T23:28:46.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:28:46 smithi115 ceph-mon[78593]: pgmap v343: 97 pgs: 2 active+recovering+undersized+degraded+remapped, 24 active+recovery_wait+undersized+degraded+remapped, 3 active+recovery_wait+degraded, 67 active+clean, 1 unknown; 146 MiB data, 652 MiB used, 715 GiB / 715 GiB avail; 4.2 KiB/s rd, 767 B/s wr, 7 op/s; 4125/32457 objects degraded (12.709%); 123 KiB/s, 7 keys/s, 28 objects/s recovering 2022-05-13T23:28:46.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:28:46 smithi047 ceph-mon[77147]: pgmap v343: 97 pgs: 2 active+recovering+undersized+degraded+remapped, 24 active+recovery_wait+undersized+degraded+remapped, 3 active+recovery_wait+degraded, 67 active+clean, 1 unknown; 146 MiB data, 652 MiB used, 715 GiB / 715 GiB avail; 4.2 KiB/s rd, 767 B/s wr, 7 op/s; 4125/32457 objects degraded (12.709%); 123 KiB/s, 7 keys/s, 28 objects/s recovering 2022-05-13T23:28:48.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:28:48 smithi115 ceph-mon[78593]: pgmap v344: 97 pgs: 2 active+recovering+undersized+degraded+remapped, 24 active+recovery_wait+undersized+degraded+remapped, 3 active+recovery_wait+degraded, 67 active+clean, 1 unknown; 146 MiB data, 652 MiB used, 715 GiB / 715 GiB avail; 3.4 KiB/s rd, 617 B/s wr, 6 op/s; 4125/32457 objects degraded (12.709%); 99 KiB/s, 5 keys/s, 23 objects/s recovering 2022-05-13T23:28:48.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:28:48 smithi115 ceph-mon[78593]: osdmap e113: 12 total, 8 up, 8 in 2022-05-13T23:28:48.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:28:48 smithi115 ceph-mon[78593]: Health check update: Degraded data redundancy: 4125/32457 objects degraded (12.709%), 29 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2022-05-13T23:28:48.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:28:48 smithi047 ceph-mon[77147]: pgmap v344: 97 pgs: 2 active+recovering+undersized+degraded+remapped, 24 active+recovery_wait+undersized+degraded+remapped, 3 active+recovery_wait+degraded, 67 active+clean, 1 unknown; 146 MiB data, 652 MiB used, 715 GiB / 715 GiB avail; 3.4 KiB/s rd, 617 B/s wr, 6 op/s; 4125/32457 objects degraded (12.709%); 99 KiB/s, 5 keys/s, 23 objects/s recovering 2022-05-13T23:28:48.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:28:48 smithi047 ceph-mon[77147]: osdmap e113: 12 total, 8 up, 8 in 2022-05-13T23:28:48.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:28:48 smithi047 ceph-mon[77147]: Health check update: Degraded data redundancy: 4125/32457 objects degraded (12.709%), 29 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2022-05-13T23:28:49.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:28:49 smithi115 ceph-mon[78593]: osdmap e114: 12 total, 8 up, 8 in 2022-05-13T23:28:49.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:28:49 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:28:49.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:28:49 smithi115 ceph-mon[78593]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2022-05-13T23:28:49.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:28:49 smithi047 ceph-mon[77147]: osdmap e114: 12 total, 8 up, 8 in 2022-05-13T23:28:49.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:28:49 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:28:49.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:28:49 smithi047 ceph-mon[77147]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2022-05-13T23:28:50.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:28:50 smithi115 ceph-mon[78593]: pgmap v347: 97 pgs: 2 active+recovering+undersized+degraded+remapped, 23 active+recovery_wait+undersized+degraded+remapped, 3 active+recovery_wait+degraded, 68 active+clean, 1 unknown; 146 MiB data, 655 MiB used, 715 GiB / 715 GiB avail; 4.2 KiB/s rd, 511 B/s wr, 7 op/s; 3929/32457 objects degraded (12.105%); 151 KiB/s, 28 objects/s recovering 2022-05-13T23:28:50.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:28:50 smithi047 ceph-mon[77147]: pgmap v347: 97 pgs: 2 active+recovering+undersized+degraded+remapped, 23 active+recovery_wait+undersized+degraded+remapped, 3 active+recovery_wait+degraded, 68 active+clean, 1 unknown; 146 MiB data, 655 MiB used, 715 GiB / 715 GiB avail; 4.2 KiB/s rd, 511 B/s wr, 7 op/s; 3929/32457 objects degraded (12.105%); 151 KiB/s, 28 objects/s recovering 2022-05-13T23:28:52.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:28:52 smithi047 ceph-mon[77147]: pgmap v348: 97 pgs: 2 active+recovering+undersized+degraded+remapped, 23 active+recovery_wait+undersized+degraded+remapped, 3 active+recovery_wait+degraded, 68 active+clean, 1 unknown; 146 MiB data, 655 MiB used, 715 GiB / 715 GiB avail; 4.2 KiB/s rd, 511 B/s wr, 7 op/s; 3929/32457 objects degraded (12.105%); 151 KiB/s, 28 objects/s recovering 2022-05-13T23:28:52.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:28:52 smithi115 ceph-mon[78593]: pgmap v348: 97 pgs: 2 active+recovering+undersized+degraded+remapped, 23 active+recovery_wait+undersized+degraded+remapped, 3 active+recovery_wait+degraded, 68 active+clean, 1 unknown; 146 MiB data, 655 MiB used, 715 GiB / 715 GiB avail; 4.2 KiB/s rd, 511 B/s wr, 7 op/s; 3929/32457 objects degraded (12.105%); 151 KiB/s, 28 objects/s recovering 2022-05-13T23:28:53.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:28:53 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:28:53.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:28:53 smithi047 ceph-mon[77147]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2022-05-13T23:28:53.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:28:53 smithi047 ceph-mon[77147]: Health check update: Degraded data redundancy: 3929/32457 objects degraded (12.105%), 28 pgs degraded, 25 pgs undersized (PG_DEGRADED) 2022-05-13T23:28:53.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:28:53 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:28:53.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:28:53 smithi115 ceph-mon[78593]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2022-05-13T23:28:53.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:28:53 smithi115 ceph-mon[78593]: Health check update: Degraded data redundancy: 3929/32457 objects degraded (12.105%), 28 pgs degraded, 25 pgs undersized (PG_DEGRADED) 2022-05-13T23:28:54.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:28:54 smithi047 ceph-mon[77147]: pgmap v349: 97 pgs: 2 active+recovering+undersized+degraded+remapped, 23 active+recovery_wait+undersized+degraded+remapped, 3 active+recovery_wait+degraded, 68 active+clean, 1 unknown; 146 MiB data, 655 MiB used, 715 GiB / 715 GiB avail; 1.9 KiB/s rd, 3 op/s; 3929/32457 objects degraded (12.105%); 102 KiB/s, 14 objects/s recovering 2022-05-13T23:28:54.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:28:54 smithi115 ceph-mon[78593]: pgmap v349: 97 pgs: 2 active+recovering+undersized+degraded+remapped, 23 active+recovery_wait+undersized+degraded+remapped, 3 active+recovery_wait+degraded, 68 active+clean, 1 unknown; 146 MiB data, 655 MiB used, 715 GiB / 715 GiB avail; 1.9 KiB/s rd, 3 op/s; 3929/32457 objects degraded (12.105%); 102 KiB/s, 14 objects/s recovering 2022-05-13T23:28:56.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:28:56 smithi047 ceph-mon[77147]: pgmap v350: 97 pgs: 2 active+recovering+undersized+degraded+remapped, 23 active+recovery_wait+undersized+degraded+remapped, 3 active+recovery_wait+degraded, 68 active+clean, 1 unknown; 146 MiB data, 657 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s; 3697/32457 objects degraded (11.390%); 151 KiB/s, 28 objects/s recovering 2022-05-13T23:28:56.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:28:56 smithi115 ceph-mon[78593]: pgmap v350: 97 pgs: 2 active+recovering+undersized+degraded+remapped, 23 active+recovery_wait+undersized+degraded+remapped, 3 active+recovery_wait+degraded, 68 active+clean, 1 unknown; 146 MiB data, 657 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s; 3697/32457 objects degraded (11.390%); 151 KiB/s, 28 objects/s recovering 2022-05-13T23:28:57.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:28:57 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:28:57.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:28:57 smithi047 ceph-mon[77147]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2022-05-13T23:28:57.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:28:57 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:28:57.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:28:57 smithi115 ceph-mon[78593]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2022-05-13T23:28:58.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:28:58 smithi047 ceph-mon[77147]: pgmap v351: 97 pgs: 2 active+recovering+undersized+degraded+remapped, 23 active+recovery_wait+undersized+degraded+remapped, 3 active+recovery_wait+degraded, 68 active+clean, 1 unknown; 146 MiB data, 657 MiB used, 715 GiB / 715 GiB avail; 3.0 KiB/s rd, 6 op/s; 3697/32457 objects degraded (11.390%); 121 KiB/s, 23 objects/s recovering 2022-05-13T23:28:58.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:28:58 smithi047 ceph-mon[77147]: Health check update: Degraded data redundancy: 3697/32457 objects degraded (11.390%), 28 pgs degraded, 25 pgs undersized (PG_DEGRADED) 2022-05-13T23:28:58.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:28:58 smithi115 ceph-mon[78593]: pgmap v351: 97 pgs: 2 active+recovering+undersized+degraded+remapped, 23 active+recovery_wait+undersized+degraded+remapped, 3 active+recovery_wait+degraded, 68 active+clean, 1 unknown; 146 MiB data, 657 MiB used, 715 GiB / 715 GiB avail; 3.0 KiB/s rd, 6 op/s; 3697/32457 objects degraded (11.390%); 121 KiB/s, 23 objects/s recovering 2022-05-13T23:28:58.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:28:58 smithi115 ceph-mon[78593]: Health check update: Degraded data redundancy: 3697/32457 objects degraded (11.390%), 28 pgs degraded, 25 pgs undersized (PG_DEGRADED) 2022-05-13T23:29:00.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:29:00 smithi047 ceph-mon[77147]: pgmap v352: 97 pgs: 2 active+recovering+undersized+degraded+remapped, 23 active+recovery_wait+undersized+degraded+remapped, 3 active+recovery_wait+degraded, 68 active+clean, 1 unknown; 146 MiB data, 658 MiB used, 715 GiB / 715 GiB avail; 4.1 KiB/s rd, 8 op/s; 3548/32457 objects degraded (10.931%); 142 KiB/s, 31 objects/s recovering 2022-05-13T23:29:00.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:29:00 smithi047 ceph-mon[77147]: osdmap e115: 12 total, 8 up, 8 in 2022-05-13T23:29:00.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:29:00 smithi115 ceph-mon[78593]: pgmap v352: 97 pgs: 2 active+recovering+undersized+degraded+remapped, 23 active+recovery_wait+undersized+degraded+remapped, 3 active+recovery_wait+degraded, 68 active+clean, 1 unknown; 146 MiB data, 658 MiB used, 715 GiB / 715 GiB avail; 4.1 KiB/s rd, 8 op/s; 3548/32457 objects degraded (10.931%); 142 KiB/s, 31 objects/s recovering 2022-05-13T23:29:00.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:29:00 smithi115 ceph-mon[78593]: osdmap e115: 12 total, 8 up, 8 in 2022-05-13T23:29:01.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:29:01 smithi047 ceph-mon[77147]: osdmap e116: 12 total, 8 up, 8 in 2022-05-13T23:29:01.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:29:01 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:29:01.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:29:01 smithi047 ceph-mon[77147]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2022-05-13T23:29:01.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:29:01 smithi115 ceph-mon[78593]: osdmap e116: 12 total, 8 up, 8 in 2022-05-13T23:29:01.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:29:01 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:29:01.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:29:01 smithi115 ceph-mon[78593]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2022-05-13T23:29:02.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:29:02 smithi047 ceph-mon[77147]: pgmap v355: 97 pgs: 2 active+recovering+undersized+degraded+remapped, 23 active+recovery_wait+undersized+degraded+remapped, 3 active+recovery_wait+degraded, 68 active+clean, 1 unknown; 146 MiB data, 658 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s; 3548/32457 objects degraded (10.931%); 92 KiB/s, 29 objects/s recovering 2022-05-13T23:29:02.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:29:02 smithi047 ceph-mon[77147]: osdmap e117: 12 total, 8 up, 8 in 2022-05-13T23:29:02.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:29:02 smithi115 ceph-mon[78593]: pgmap v355: 97 pgs: 2 active+recovering+undersized+degraded+remapped, 23 active+recovery_wait+undersized+degraded+remapped, 3 active+recovery_wait+degraded, 68 active+clean, 1 unknown; 146 MiB data, 658 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s; 3548/32457 objects degraded (10.931%); 92 KiB/s, 29 objects/s recovering 2022-05-13T23:29:02.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:29:02 smithi115 ceph-mon[78593]: osdmap e117: 12 total, 8 up, 8 in 2022-05-13T23:29:03.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:29:03 smithi047 ceph-mon[77147]: Health check update: Degraded data redundancy: 3548/32457 objects degraded (10.931%), 28 pgs degraded, 25 pgs undersized (PG_DEGRADED) 2022-05-13T23:29:03.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:29:03 smithi115 ceph-mon[78593]: Health check update: Degraded data redundancy: 3548/32457 objects degraded (10.931%), 28 pgs degraded, 25 pgs undersized (PG_DEGRADED) 2022-05-13T23:29:04.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:29:04 smithi047 ceph-mon[77147]: pgmap v357: 97 pgs: 2 active+recovering+undersized+degraded+remapped, 23 active+recovery_wait+undersized+degraded+remapped, 3 active+recovery_wait+degraded, 68 active+clean, 1 unknown; 146 MiB data, 658 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s; 3548/32457 objects degraded (10.931%); 58 KiB/s, 19 objects/s recovering 2022-05-13T23:29:04.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:29:04 smithi115 ceph-mon[78593]: pgmap v357: 97 pgs: 2 active+recovering+undersized+degraded+remapped, 23 active+recovery_wait+undersized+degraded+remapped, 3 active+recovery_wait+degraded, 68 active+clean, 1 unknown; 146 MiB data, 658 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s; 3548/32457 objects degraded (10.931%); 58 KiB/s, 19 objects/s recovering 2022-05-13T23:29:06.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:29:06 smithi047 ceph-mon[77147]: pgmap v358: 97 pgs: 3 active+recovering+undersized+degraded+remapped, 20 active+recovery_wait+undersized+degraded+remapped, 73 active+clean, 1 unknown; 146 MiB data, 664 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s; 3391/32457 objects degraded (10.448%); 782 KiB/s, 19 keys/s, 26 objects/s recovering 2022-05-13T23:29:06.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:29:06 smithi115 ceph-mon[78593]: pgmap v358: 97 pgs: 3 active+recovering+undersized+degraded+remapped, 20 active+recovery_wait+undersized+degraded+remapped, 73 active+clean, 1 unknown; 146 MiB data, 664 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s; 3391/32457 objects degraded (10.448%); 782 KiB/s, 19 keys/s, 26 objects/s recovering 2022-05-13T23:29:08.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:29:08 smithi047 ceph-mon[77147]: pgmap v359: 97 pgs: 3 active+recovering+undersized+degraded+remapped, 20 active+recovery_wait+undersized+degraded+remapped, 73 active+clean, 1 unknown; 146 MiB data, 664 MiB used, 715 GiB / 715 GiB avail; 1.9 KiB/s rd, 3 op/s; 3391/32457 objects degraded (10.448%); 594 KiB/s, 14 keys/s, 19 objects/s recovering 2022-05-13T23:29:08.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:29:08 smithi047 ceph-mon[77147]: Health check update: Degraded data redundancy: 3391/32457 objects degraded (10.448%), 23 pgs degraded, 23 pgs undersized (PG_DEGRADED) 2022-05-13T23:29:08.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:29:08 smithi115 ceph-mon[78593]: pgmap v359: 97 pgs: 3 active+recovering+undersized+degraded+remapped, 20 active+recovery_wait+undersized+degraded+remapped, 73 active+clean, 1 unknown; 146 MiB data, 664 MiB used, 715 GiB / 715 GiB avail; 1.9 KiB/s rd, 3 op/s; 3391/32457 objects degraded (10.448%); 594 KiB/s, 14 keys/s, 19 objects/s recovering 2022-05-13T23:29:08.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:29:08 smithi115 ceph-mon[78593]: Health check update: Degraded data redundancy: 3391/32457 objects degraded (10.448%), 23 pgs degraded, 23 pgs undersized (PG_DEGRADED) 2022-05-13T23:29:10.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:29:10 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:29:10.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:29:10 smithi047 ceph-mon[77147]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2022-05-13T23:29:10.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:29:10 smithi047 ceph-mon[77147]: pgmap v360: 97 pgs: 3 active+recovering+undersized+degraded+remapped, 20 active+recovery_wait+undersized+degraded+remapped, 73 active+clean, 1 unknown; 146 MiB data, 665 MiB used, 715 GiB / 715 GiB avail; 3.4 KiB/s rd, 6 op/s; 3217/32457 objects degraded (9.912%); 590 KiB/s, 12 keys/s, 37 objects/s recovering 2022-05-13T23:29:10.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:29:10 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:29:10.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:29:10 smithi115 ceph-mon[78593]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2022-05-13T23:29:10.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:29:10 smithi115 ceph-mon[78593]: pgmap v360: 97 pgs: 3 active+recovering+undersized+degraded+remapped, 20 active+recovery_wait+undersized+degraded+remapped, 73 active+clean, 1 unknown; 146 MiB data, 665 MiB used, 715 GiB / 715 GiB avail; 3.4 KiB/s rd, 6 op/s; 3217/32457 objects degraded (9.912%); 590 KiB/s, 12 keys/s, 37 objects/s recovering 2022-05-13T23:29:12.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:29:12 smithi115 ceph-mon[78593]: pgmap v361: 97 pgs: 3 active+recovering+undersized+degraded+remapped, 20 active+recovery_wait+undersized+degraded+remapped, 73 active+clean, 1 unknown; 146 MiB data, 665 MiB used, 715 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s; 3217/32457 objects degraded (9.912%); 524 KiB/s, 11 keys/s, 33 objects/s recovering 2022-05-13T23:29:12.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:29:12 smithi047 ceph-mon[77147]: pgmap v361: 97 pgs: 3 active+recovering+undersized+degraded+remapped, 20 active+recovery_wait+undersized+degraded+remapped, 73 active+clean, 1 unknown; 146 MiB data, 665 MiB used, 715 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s; 3217/32457 objects degraded (9.912%); 524 KiB/s, 11 keys/s, 33 objects/s recovering 2022-05-13T23:29:13.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:29:13 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:29:13.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:29:13 smithi115 ceph-mon[78593]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2022-05-13T23:29:13.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:29:13 smithi115 ceph-mon[78593]: Health check update: Degraded data redundancy: 3217/32457 objects degraded (9.912%), 23 pgs degraded, 23 pgs undersized (PG_DEGRADED) 2022-05-13T23:29:13.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:29:13 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-13T23:29:13.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:29:13 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-13T23:29:13.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:29:13 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:29:13.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:29:13 smithi047 ceph-mon[77147]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2022-05-13T23:29:13.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:29:13 smithi047 ceph-mon[77147]: Health check update: Degraded data redundancy: 3217/32457 objects degraded (9.912%), 23 pgs degraded, 23 pgs undersized (PG_DEGRADED) 2022-05-13T23:29:13.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:29:13 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-13T23:29:13.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:29:13 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-13T23:29:14.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:29:14 smithi115 ceph-mon[78593]: pgmap v362: 97 pgs: 3 active+recovering+undersized+degraded+remapped, 20 active+recovery_wait+undersized+degraded+remapped, 73 active+clean, 1 unknown; 146 MiB data, 665 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 5 op/s; 3217/32457 objects degraded (9.912%); 441 KiB/s, 9 keys/s, 27 objects/s recovering 2022-05-13T23:29:14.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:29:14 smithi115 ceph-mon[78593]: osdmap e118: 12 total, 8 up, 8 in 2022-05-13T23:29:14.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:29:14 smithi047 ceph-mon[77147]: pgmap v362: 97 pgs: 3 active+recovering+undersized+degraded+remapped, 20 active+recovery_wait+undersized+degraded+remapped, 73 active+clean, 1 unknown; 146 MiB data, 665 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 5 op/s; 3217/32457 objects degraded (9.912%); 441 KiB/s, 9 keys/s, 27 objects/s recovering 2022-05-13T23:29:14.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:29:14 smithi047 ceph-mon[77147]: osdmap e118: 12 total, 8 up, 8 in 2022-05-13T23:29:16.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:29:16 smithi047 ceph-mon[77147]: osdmap e119: 12 total, 8 up, 8 in 2022-05-13T23:29:16.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:29:16 smithi047 ceph-mon[77147]: pgmap v365: 97 pgs: 2 active+undersized+remapped, 3 active+recovering+undersized+degraded+remapped, 18 active+recovery_wait+undersized+degraded+remapped, 73 active+clean, 1 unknown; 146 MiB data, 671 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s; 2874/32457 objects degraded (8.855%); 267 KiB/s, 64 objects/s recovering 2022-05-13T23:29:16.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:29:16 smithi115 ceph-mon[78593]: osdmap e119: 12 total, 8 up, 8 in 2022-05-13T23:29:16.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:29:16 smithi115 ceph-mon[78593]: pgmap v365: 97 pgs: 2 active+undersized+remapped, 3 active+recovering+undersized+degraded+remapped, 18 active+recovery_wait+undersized+degraded+remapped, 73 active+clean, 1 unknown; 146 MiB data, 671 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s; 2874/32457 objects degraded (8.855%); 267 KiB/s, 64 objects/s recovering 2022-05-13T23:29:17.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:29:17 smithi047 ceph-mon[77147]: osdmap e120: 12 total, 8 up, 8 in 2022-05-13T23:29:17.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:29:17 smithi115 ceph-mon[78593]: osdmap e120: 12 total, 8 up, 8 in 2022-05-13T23:29:18.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:29:18 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:29:18.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:29:18 smithi047 ceph-mon[77147]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2022-05-13T23:29:18.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:29:18 smithi047 ceph-mon[77147]: pgmap v367: 97 pgs: 2 active+undersized+remapped, 3 active+recovering+undersized+degraded+remapped, 18 active+recovery_wait+undersized+degraded+remapped, 73 active+clean, 1 unknown; 146 MiB data, 671 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s; 2874/32457 objects degraded (8.855%); 264 KiB/s, 57 objects/s recovering 2022-05-13T23:29:18.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:29:18 smithi047 ceph-mon[77147]: Health check update: Degraded data redundancy: 2874/32457 objects degraded (8.855%), 21 pgs degraded, 23 pgs undersized (PG_DEGRADED) 2022-05-13T23:29:18.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:29:18 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:29:18.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:29:18 smithi115 ceph-mon[78593]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2022-05-13T23:29:18.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:29:18 smithi115 ceph-mon[78593]: pgmap v367: 97 pgs: 2 active+undersized+remapped, 3 active+recovering+undersized+degraded+remapped, 18 active+recovery_wait+undersized+degraded+remapped, 73 active+clean, 1 unknown; 146 MiB data, 671 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s; 2874/32457 objects degraded (8.855%); 264 KiB/s, 57 objects/s recovering 2022-05-13T23:29:18.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:29:18 smithi115 ceph-mon[78593]: Health check update: Degraded data redundancy: 2874/32457 objects degraded (8.855%), 21 pgs degraded, 23 pgs undersized (PG_DEGRADED) 2022-05-13T23:29:20.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:29:20 smithi115 ceph-mon[78593]: pgmap v368: 97 pgs: 2 active+recovering+undersized+degraded+remapped, 17 active+recovery_wait+undersized+degraded+remapped, 77 active+clean, 1 unknown; 146 MiB data, 673 MiB used, 715 GiB / 715 GiB avail; 5.0 KiB/s rd, 9 op/s; 2752/32457 objects degraded (8.479%); 333 KiB/s, 77 objects/s recovering 2022-05-13T23:29:20.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:29:20 smithi047 ceph-mon[77147]: pgmap v368: 97 pgs: 2 active+recovering+undersized+degraded+remapped, 17 active+recovery_wait+undersized+degraded+remapped, 77 active+clean, 1 unknown; 146 MiB data, 673 MiB used, 715 GiB / 715 GiB avail; 5.0 KiB/s rd, 9 op/s; 2752/32457 objects degraded (8.479%); 333 KiB/s, 77 objects/s recovering 2022-05-13T23:29:21.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:29:21 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:29:21.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:29:21 smithi115 ceph-mon[78593]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2022-05-13T23:29:21.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:29:21 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:29:21.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:29:21 smithi047 ceph-mon[77147]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2022-05-13T23:29:22.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:29:22 smithi115 ceph-mon[78593]: pgmap v369: 97 pgs: 2 active+recovering+undersized+degraded+remapped, 17 active+recovery_wait+undersized+degraded+remapped, 77 active+clean, 1 unknown; 146 MiB data, 673 MiB used, 715 GiB / 715 GiB avail; 2.1 KiB/s rd, 4 op/s; 2752/32457 objects degraded (8.479%); 57 KiB/s, 17 objects/s recovering 2022-05-13T23:29:22.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:29:22 smithi047 ceph-mon[77147]: pgmap v369: 97 pgs: 2 active+recovering+undersized+degraded+remapped, 17 active+recovery_wait+undersized+degraded+remapped, 77 active+clean, 1 unknown; 146 MiB data, 673 MiB used, 715 GiB / 715 GiB avail; 2.1 KiB/s rd, 4 op/s; 2752/32457 objects degraded (8.479%); 57 KiB/s, 17 objects/s recovering 2022-05-13T23:29:23.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:29:23 smithi115 ceph-mon[78593]: Health check update: Degraded data redundancy: 2752/32457 objects degraded (8.479%), 19 pgs degraded, 19 pgs undersized (PG_DEGRADED) 2022-05-13T23:29:23.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:29:23 smithi047 ceph-mon[77147]: Health check update: Degraded data redundancy: 2752/32457 objects degraded (8.479%), 19 pgs degraded, 19 pgs undersized (PG_DEGRADED) 2022-05-13T23:29:24.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:29:24 smithi115 ceph-mon[78593]: pgmap v370: 97 pgs: 2 active+recovering+undersized+degraded+remapped, 17 active+recovery_wait+undersized+degraded+remapped, 77 active+clean, 1 unknown; 146 MiB data, 673 MiB used, 715 GiB / 715 GiB avail; 1.8 KiB/s rd, 3 op/s; 2752/32457 objects degraded (8.479%); 50 KiB/s, 14 objects/s recovering 2022-05-13T23:29:24.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:29:24 smithi047 ceph-mon[77147]: pgmap v370: 97 pgs: 2 active+recovering+undersized+degraded+remapped, 17 active+recovery_wait+undersized+degraded+remapped, 77 active+clean, 1 unknown; 146 MiB data, 673 MiB used, 715 GiB / 715 GiB avail; 1.8 KiB/s rd, 3 op/s; 2752/32457 objects degraded (8.479%); 50 KiB/s, 14 objects/s recovering 2022-05-13T23:29:25.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:29:25 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-13T23:29:25.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:29:25 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-13T23:29:25.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:29:25 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-13T23:29:25.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:29:25 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-13T23:29:25.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:29:25 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-13T23:29:25.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:29:25 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-13T23:29:26.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:29:26 smithi115 ceph-mon[78593]: pgmap v371: 97 pgs: 2 active+recovering+undersized+degraded+remapped, 17 active+recovery_wait+undersized+degraded+remapped, 77 active+clean, 1 unknown; 146 MiB data, 674 MiB used, 715 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s; 2636/32457 objects degraded (8.122%); 80 KiB/s, 23 objects/s recovering 2022-05-13T23:29:26.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:29:26 smithi047 ceph-mon[77147]: pgmap v371: 97 pgs: 2 active+recovering+undersized+degraded+remapped, 17 active+recovery_wait+undersized+degraded+remapped, 77 active+clean, 1 unknown; 146 MiB data, 674 MiB used, 715 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s; 2636/32457 objects degraded (8.122%); 80 KiB/s, 23 objects/s recovering 2022-05-13T23:29:27.791 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:29:27 smithi047 conmon[78521]: 2022-05-13T23:29:27.535+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:29:27.792 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:29:27 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-13T23:29:27.792 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:29:27 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-13T23:29:27.792 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:29:27 smithi047 conmon[78521]: yield 2022-05-13T23:29:27.792 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:29:27 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-13T23:29:27.793 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:29:27 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-13T23:29:27.793 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:29:27 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-13T23:29:27.793 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:29:27 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-13T23:29:27.793 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:29:27 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-13T23:29:27.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:29:27 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-13T23:29:27.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:29:27 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-13T23:29:27.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:29:27 smithi047 conmon[78521]: raise exceptions[0] 2022-05-13T23:29:27.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:29:27 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-13T23:29:27.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:29:27 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-13T23:29:27.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:29:27 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-13T23:29:27.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:29:27 smithi047 conmon[78521]: return (yield from fut) 2022-05-13T23:29:27.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:29:27 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-13T23:29:27.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:29:27 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-13T23:29:27.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:29:27 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:29:28.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:29:28 smithi115 ceph-mon[78593]: pgmap v372: 97 pgs: 2 active+recovering+undersized+degraded+remapped, 17 active+recovery_wait+undersized+degraded+remapped, 77 active+clean, 1 unknown; 146 MiB data, 674 MiB used, 715 GiB / 715 GiB avail; 2.7 KiB/s rd, 5 op/s; 2636/32457 objects degraded (8.122%); 71 KiB/s, 21 objects/s recovering 2022-05-13T23:29:28.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:29:28 smithi115 ceph-mon[78593]: Health check update: Degraded data redundancy: 2636/32457 objects degraded (8.122%), 19 pgs degraded, 19 pgs undersized (PG_DEGRADED) 2022-05-13T23:29:28.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:29:28 smithi115 ceph-mon[78593]: osdmap e121: 12 total, 8 up, 8 in 2022-05-13T23:29:28.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:29:28 smithi047 ceph-mon[77147]: pgmap v372: 97 pgs: 2 active+recovering+undersized+degraded+remapped, 17 active+recovery_wait+undersized+degraded+remapped, 77 active+clean, 1 unknown; 146 MiB data, 674 MiB used, 715 GiB / 715 GiB avail; 2.7 KiB/s rd, 5 op/s; 2636/32457 objects degraded (8.122%); 71 KiB/s, 21 objects/s recovering 2022-05-13T23:29:28.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:29:28 smithi047 ceph-mon[77147]: Health check update: Degraded data redundancy: 2636/32457 objects degraded (8.122%), 19 pgs degraded, 19 pgs undersized (PG_DEGRADED) 2022-05-13T23:29:28.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:29:28 smithi047 ceph-mon[77147]: osdmap e121: 12 total, 8 up, 8 in 2022-05-13T23:29:29.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:29:29 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:29:29.608 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-13T23:29:29.608 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-13T23:29:29.609 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-13T23:29:29.609 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-13T23:29:29.609 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-13T23:29:29.609 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-13T23:29:29.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-13T23:29:29.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-13T23:29:29.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-13T23:29:29.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-13T23:29:29.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-13T23:29:29.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-13T23:29:29.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-13T23:29:29.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-13T23:29:29.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-13T23:29:29.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-13T23:29:29.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-13T23:29:29.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:29:29.612 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:29:29 smithi115 ceph-mon[78593]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-13T23:29:29.612 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:29:29 smithi115 ceph-mon[78593]: osdmap e122: 12 total, 8 up, 8 in 2022-05-13T23:29:29.613 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:29:29 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:29:29.613 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:29:29 smithi115 ceph-mon[78593]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2022-05-13T23:29:29.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:29:29 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:29:29.792 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-13T23:29:29.792 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-13T23:29:29.792 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-13T23:29:29.792 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-13T23:29:29.792 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-13T23:29:29.793 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-13T23:29:29.793 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-13T23:29:29.793 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-13T23:29:29.793 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-13T23:29:29.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-13T23:29:29.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-13T23:29:29.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-13T23:29:29.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-13T23:29:29.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-13T23:29:29.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-13T23:29:29.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-13T23:29:29.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-13T23:29:29.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:29:29.795 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:29:29 smithi047 ceph-mon[77147]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-13T23:29:29.796 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:29:29 smithi047 ceph-mon[77147]: osdmap e122: 12 total, 8 up, 8 in 2022-05-13T23:29:29.796 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:29:29 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:29:29.796 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:29:29 smithi047 ceph-mon[77147]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2022-05-13T23:29:30.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:29:30 smithi115 ceph-mon[78593]: pgmap v375: 97 pgs: 1 peering, 4 active+undersized+remapped, 3 active+recovering+undersized+degraded+remapped, 11 active+recovery_wait+undersized+degraded+remapped, 77 active+clean, 1 unknown; 146 MiB data, 680 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s; 2008/32457 objects degraded (6.187%); 510 KiB/s, 92 objects/s recovering 2022-05-13T23:29:30.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:29:30 smithi115 ceph-mon[78593]: osdmap e123: 12 total, 8 up, 8 in 2022-05-13T23:29:30.609 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:29:30 smithi047 ceph-mon[77147]: pgmap v375: 97 pgs: 1 peering, 4 active+undersized+remapped, 3 active+recovering+undersized+degraded+remapped, 11 active+recovery_wait+undersized+degraded+remapped, 77 active+clean, 1 unknown; 146 MiB data, 680 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s; 2008/32457 objects degraded (6.187%); 510 KiB/s, 92 objects/s recovering 2022-05-13T23:29:30.609 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:29:30 smithi047 ceph-mon[77147]: osdmap e123: 12 total, 8 up, 8 in 2022-05-13T23:29:31.042 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:29:30 smithi047 conmon[78521]: 2022-05-13T23:29:30.607+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:29:31.042 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:29:30 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-13T23:29:31.042 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:29:30 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-13T23:29:31.042 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:29:30 smithi047 conmon[78521]: yield 2022-05-13T23:29:31.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:29:30 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-13T23:29:31.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:29:30 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-13T23:29:31.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:29:30 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-13T23:29:31.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:29:30 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-13T23:29:31.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:29:30 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-13T23:29:31.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:29:30 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-13T23:29:31.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:29:30 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-13T23:29:31.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:29:30 smithi047 conmon[78521]: raise exceptions[0] 2022-05-13T23:29:31.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:29:30 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-13T23:29:31.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:29:30 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-13T23:29:31.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:29:30 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-13T23:29:31.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:29:30 smithi047 conmon[78521]: return (yield from fut) 2022-05-13T23:29:31.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:29:30 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-13T23:29:31.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:29:30 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-13T23:29:31.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:29:30 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:29:31.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:29:30 smithi047 conmon[78521]: 2022-05-13T23:29:30.607+0000 7f533ef76700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-13T23:29:31.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:29:30 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-13T23:29:31.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:29:30 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-13T23:29:31.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:29:30 smithi047 conmon[78521]: yield 2022-05-13T23:29:31.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:29:30 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-13T23:29:31.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:29:30 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-13T23:29:31.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:29:30 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-13T23:29:31.048 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:29:30 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-13T23:29:31.048 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:29:30 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-13T23:29:31.048 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:29:30 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-13T23:29:31.049 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:29:30 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-13T23:29:31.049 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:29:30 smithi047 conmon[78521]: raise exceptions[0] 2022-05-13T23:29:31.049 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:29:30 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-13T23:29:31.049 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:29:30 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-13T23:29:31.049 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:29:30 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-13T23:29:31.050 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:29:30 smithi047 conmon[78521]: return (yield from fut) 2022-05-13T23:29:31.050 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:29:30 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-13T23:29:31.050 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:29:30 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-13T23:29:31.050 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:29:30 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:29:31.050 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:29:30 smithi047 conmon[78521]: 2022-05-13T23:29:31.051 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:29:30 smithi047 conmon[78521]: During handling of the above exception, another exception occurred: 2022-05-13T23:29:31.051 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:29:30 smithi047 conmon[78521]: 2022-05-13T23:29:31.051 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:29:30 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-13T23:29:31.051 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:29:30 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-13T23:29:31.051 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:29:30 smithi047 conmon[78521]: return f(*arg) 2022-05-13T23:29:31.052 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:29:30 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-13T23:29:31.052 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:29:30 smithi047 conmon[78521]: self._write_client_files(client_files, host) 2022-05-13T23:29:31.052 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:29:30 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-13T23:29:31.052 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:29:30 smithi047 conmon[78521]: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-13T23:29:31.053 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:29:30 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-13T23:29:31.053 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:29:30 smithi047 conmon[78521]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-13T23:29:31.053 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:29:30 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-13T23:29:31.053 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:29:30 smithi047 conmon[78521]: return self.event_loop.get_result(coro) 2022-05-13T23:29:31.054 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:29:30 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-13T23:29:31.054 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:29:30 smithi047 conmon[78521]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-13T23:29:31.054 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:29:30 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-13T23:29:31.054 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:29:30 smithi047 conmon[78521]: return self.__get_result() 2022-05-13T23:29:31.055 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:29:30 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-13T23:29:31.055 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:29:30 smithi047 conmon[78521]: raise self._exception 2022-05-13T23:29:31.055 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:29:30 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-13T23:29:31.055 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:29:30 smithi047 conmon[78521]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-13T23:29:31.055 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:29:30 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-13T23:29:31.056 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:29:30 smithi047 conmon[78521]: conn = await self._remote_connection(host, addr) 2022-05-13T23:29:31.056 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:29:30 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-13T23:29:31.056 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:29:30 smithi047 conmon[78521]: raise 2022-05-13T23:29:31.056 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:29:30 smithi047 conmon[78521]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-13T23:29:31.057 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:29:30 smithi047 conmon[78521]: self.gen.throw(type, value, traceback) 2022-05-13T23:29:31.057 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:29:30 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-13T23:29:31.057 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:29:30 smithi047 conmon[78521]: raise OrchestratorError(msg) 2022-05-13T23:29:31.057 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:29:30 smithi047 conmon[78521]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:29:32.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:29:32 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:29:32.609 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-13T23:29:32.609 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-13T23:29:32.609 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-13T23:29:32.609 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-13T23:29:32.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-13T23:29:32.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-13T23:29:32.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-13T23:29:32.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-13T23:29:32.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-13T23:29:32.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-13T23:29:32.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-13T23:29:32.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-13T23:29:32.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-13T23:29:32.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-13T23:29:32.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-13T23:29:32.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-13T23:29:32.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-13T23:29:32.613 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:29:32.613 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:29:32 smithi115 ceph-mon[78593]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-13T23:29:32.613 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-13T23:29:32.613 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-13T23:29:32.613 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-13T23:29:32.614 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-13T23:29:32.614 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-13T23:29:32.614 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-13T23:29:32.614 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-13T23:29:32.615 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-13T23:29:32.615 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-13T23:29:32.615 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-13T23:29:32.615 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-13T23:29:32.615 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-13T23:29:32.616 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-13T23:29:32.616 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-13T23:29:32.616 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-13T23:29:32.616 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-13T23:29:32.616 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-13T23:29:32.617 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:29:32.617 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-13T23:29:32.617 INFO:journalctl@ceph.mon.c.smithi115.stdout: During handling of the above exception, another exception occurred: 2022-05-13T23:29:32.617 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-13T23:29:32.617 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-13T23:29:32.618 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-13T23:29:32.618 INFO:journalctl@ceph.mon.c.smithi115.stdout: return f(*arg) 2022-05-13T23:29:32.618 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-13T23:29:32.618 INFO:journalctl@ceph.mon.c.smithi115.stdout: self._write_client_files(client_files, host) 2022-05-13T23:29:32.619 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-13T23:29:32.619 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-13T23:29:32.619 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-13T23:29:32.619 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-13T23:29:32.619 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-13T23:29:32.620 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.event_loop.get_result(coro) 2022-05-13T23:29:32.620 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-13T23:29:32.620 INFO:journalctl@ceph.mon.c.smithi115.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-13T23:29:32.620 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-13T23:29:32.620 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.__get_result() 2022-05-13T23:29:32.621 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-13T23:29:32.621 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise self._exception 2022-05-13T23:29:32.621 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-13T23:29:32.621 INFO:journalctl@ceph.mon.c.smithi115.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-13T23:29:32.622 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-13T23:29:32.622 INFO:journalctl@ceph.mon.c.smithi115.stdout: conn = await self._remote_connection(host, addr) 2022-05-13T23:29:32.622 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-13T23:29:32.622 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise 2022-05-13T23:29:32.622 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-13T23:29:32.623 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.gen.throw(type, value, traceback) 2022-05-13T23:29:32.623 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-13T23:29:32.623 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OrchestratorError(msg) 2022-05-13T23:29:32.623 INFO:journalctl@ceph.mon.c.smithi115.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:29:32.623 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:29:32 smithi115 ceph-mon[78593]: pgmap v377: 97 pgs: 1 peering, 4 active+undersized+remapped, 3 active+recovering+undersized+degraded+remapped, 11 active+recovery_wait+undersized+degraded+remapped, 77 active+clean, 1 unknown; 146 MiB data, 680 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s; 2008/32457 objects degraded (6.187%); 616 KiB/s, 104 objects/s recovering 2022-05-13T23:29:32.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:29:32 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:29:32.792 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-13T23:29:32.792 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-13T23:29:32.792 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-13T23:29:32.792 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-13T23:29:32.793 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-13T23:29:32.793 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-13T23:29:32.793 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-13T23:29:32.793 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-13T23:29:32.793 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-13T23:29:32.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-13T23:29:32.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-13T23:29:32.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-13T23:29:32.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-13T23:29:32.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-13T23:29:32.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-13T23:29:32.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-13T23:29:32.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-13T23:29:32.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:29:32.796 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:29:32 smithi047 ceph-mon[77147]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-13T23:29:32.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-13T23:29:32.797 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-13T23:29:32.797 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-13T23:29:32.797 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-13T23:29:32.797 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-13T23:29:32.798 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-13T23:29:32.798 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-13T23:29:32.798 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-13T23:29:32.798 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-13T23:29:32.798 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-13T23:29:32.799 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-13T23:29:32.799 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-13T23:29:32.799 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-13T23:29:32.799 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-13T23:29:32.800 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-13T23:29:32.800 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-13T23:29:32.800 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-13T23:29:32.800 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:29:32.800 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-13T23:29:32.801 INFO:journalctl@ceph.mon.b.smithi047.stdout: During handling of the above exception, another exception occurred: 2022-05-13T23:29:32.801 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-13T23:29:32.801 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-13T23:29:32.801 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-13T23:29:32.801 INFO:journalctl@ceph.mon.b.smithi047.stdout: return f(*arg) 2022-05-13T23:29:32.802 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-13T23:29:32.802 INFO:journalctl@ceph.mon.b.smithi047.stdout: self._write_client_files(client_files, host) 2022-05-13T23:29:32.802 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-13T23:29:32.802 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-13T23:29:32.803 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-13T23:29:32.803 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-13T23:29:32.803 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-13T23:29:32.803 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.event_loop.get_result(coro) 2022-05-13T23:29:32.803 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-13T23:29:32.804 INFO:journalctl@ceph.mon.b.smithi047.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-13T23:29:32.804 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-13T23:29:32.804 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.__get_result() 2022-05-13T23:29:32.804 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-13T23:29:32.805 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise self._exception 2022-05-13T23:29:32.805 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-13T23:29:32.805 INFO:journalctl@ceph.mon.b.smithi047.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-13T23:29:32.805 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-13T23:29:32.806 INFO:journalctl@ceph.mon.b.smithi047.stdout: conn = await self._remote_connection(host, addr) 2022-05-13T23:29:32.806 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-13T23:29:32.806 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise 2022-05-13T23:29:32.806 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-13T23:29:32.806 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.gen.throw(type, value, traceback) 2022-05-13T23:29:32.807 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-13T23:29:32.807 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OrchestratorError(msg) 2022-05-13T23:29:32.807 INFO:journalctl@ceph.mon.b.smithi047.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:29:32.807 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:29:32 smithi047 ceph-mon[77147]: pgmap v377: 97 pgs: 1 peering, 4 active+undersized+remapped, 3 active+recovering+undersized+degraded+remapped, 11 active+recovery_wait+undersized+degraded+remapped, 77 active+clean, 1 unknown; 146 MiB data, 680 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s; 2008/32457 objects degraded (6.187%); 616 KiB/s, 104 objects/s recovering 2022-05-13T23:29:33.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:29:33 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:29:33.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:29:33 smithi115 ceph-mon[78593]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2022-05-13T23:29:33.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:29:33 smithi115 ceph-mon[78593]: Health check update: Degraded data redundancy: 2008/32457 objects degraded (6.187%), 14 pgs degraded, 18 pgs undersized (PG_DEGRADED) 2022-05-13T23:29:33.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:29:33 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:29:33.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:29:33 smithi047 ceph-mon[77147]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2022-05-13T23:29:33.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:29:33 smithi047 ceph-mon[77147]: Health check update: Degraded data redundancy: 2008/32457 objects degraded (6.187%), 14 pgs degraded, 18 pgs undersized (PG_DEGRADED) 2022-05-13T23:29:34.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:29:34 smithi115 ceph-mon[78593]: pgmap v378: 97 pgs: 1 peering, 4 active+undersized+remapped, 3 active+recovering+undersized+degraded+remapped, 11 active+recovery_wait+undersized+degraded+remapped, 77 active+clean, 1 unknown; 146 MiB data, 680 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s; 2008/32457 objects degraded (6.187%); 616 KiB/s, 104 objects/s recovering 2022-05-13T23:29:34.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:29:34 smithi047 ceph-mon[77147]: pgmap v378: 97 pgs: 1 peering, 4 active+undersized+remapped, 3 active+recovering+undersized+degraded+remapped, 11 active+recovery_wait+undersized+degraded+remapped, 77 active+clean, 1 unknown; 146 MiB data, 680 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s; 2008/32457 objects degraded (6.187%); 616 KiB/s, 104 objects/s recovering 2022-05-13T23:29:36.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:29:36 smithi115 ceph-mon[78593]: pgmap v379: 97 pgs: 3 active+recovering+undersized+degraded+remapped, 11 active+recovery_wait+undersized+degraded+remapped, 82 active+clean, 1 unknown; 146 MiB data, 682 MiB used, 715 GiB / 715 GiB avail; 4.2 KiB/s rd, 8 op/s; 1834/32457 objects degraded (5.651%); 659 KiB/s, 111 objects/s recovering 2022-05-13T23:29:36.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:29:36 smithi047 ceph-mon[77147]: pgmap v379: 97 pgs: 3 active+recovering+undersized+degraded+remapped, 11 active+recovery_wait+undersized+degraded+remapped, 82 active+clean, 1 unknown; 146 MiB data, 682 MiB used, 715 GiB / 715 GiB avail; 4.2 KiB/s rd, 8 op/s; 1834/32457 objects degraded (5.651%); 659 KiB/s, 111 objects/s recovering 2022-05-13T23:29:37.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:29:37 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:29:37.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:29:37 smithi047 ceph-mon[77147]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2022-05-13T23:29:37.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:29:37 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:29:37.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:29:37 smithi115 ceph-mon[78593]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2022-05-13T23:29:38.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:29:38 smithi047 ceph-mon[77147]: pgmap v380: 97 pgs: 3 active+recovering+undersized+degraded+remapped, 11 active+recovery_wait+undersized+degraded+remapped, 82 active+clean, 1 unknown; 146 MiB data, 682 MiB used, 715 GiB / 715 GiB avail; 1.8 KiB/s rd, 3 op/s; 1834/32457 objects degraded (5.651%); 126 KiB/s, 21 objects/s recovering 2022-05-13T23:29:38.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:29:38 smithi047 ceph-mon[77147]: Health check update: Degraded data redundancy: 1834/32457 objects degraded (5.651%), 14 pgs degraded, 14 pgs undersized (PG_DEGRADED) 2022-05-13T23:29:38.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:29:38 smithi115 ceph-mon[78593]: pgmap v380: 97 pgs: 3 active+recovering+undersized+degraded+remapped, 11 active+recovery_wait+undersized+degraded+remapped, 82 active+clean, 1 unknown; 146 MiB data, 682 MiB used, 715 GiB / 715 GiB avail; 1.8 KiB/s rd, 3 op/s; 1834/32457 objects degraded (5.651%); 126 KiB/s, 21 objects/s recovering 2022-05-13T23:29:38.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:29:38 smithi115 ceph-mon[78593]: Health check update: Degraded data redundancy: 1834/32457 objects degraded (5.651%), 14 pgs degraded, 14 pgs undersized (PG_DEGRADED) 2022-05-13T23:29:40.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:29:40 smithi047 ceph-mon[77147]: pgmap v381: 97 pgs: 3 active+recovering+undersized+degraded+remapped, 11 active+recovery_wait+undersized+degraded+remapped, 82 active+clean, 1 unknown; 146 MiB data, 684 MiB used, 715 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s; 1660/32457 objects degraded (5.114%); 170 KiB/s, 34 objects/s recovering 2022-05-13T23:29:40.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:29:40 smithi115 ceph-mon[78593]: pgmap v381: 97 pgs: 3 active+recovering+undersized+degraded+remapped, 11 active+recovery_wait+undersized+degraded+remapped, 82 active+clean, 1 unknown; 146 MiB data, 684 MiB used, 715 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s; 1660/32457 objects degraded (5.114%); 170 KiB/s, 34 objects/s recovering 2022-05-13T23:29:41.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:29:41 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:29:41.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:29:41 smithi047 ceph-mon[77147]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2022-05-13T23:29:41.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:29:41 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:29:41.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:29:41 smithi115 ceph-mon[78593]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2022-05-13T23:29:42.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:29:42 smithi047 ceph-mon[77147]: pgmap v382: 97 pgs: 3 active+recovering+undersized+degraded+remapped, 11 active+recovery_wait+undersized+degraded+remapped, 82 active+clean, 1 unknown; 146 MiB data, 684 MiB used, 715 GiB / 715 GiB avail; 2.7 KiB/s rd, 5 op/s; 1660/32457 objects degraded (5.114%); 153 KiB/s, 31 objects/s recovering 2022-05-13T23:29:42.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:29:42 smithi047 ceph-mon[77147]: osdmap e124: 12 total, 8 up, 8 in 2022-05-13T23:29:42.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:29:42 smithi115 ceph-mon[78593]: pgmap v382: 97 pgs: 3 active+recovering+undersized+degraded+remapped, 11 active+recovery_wait+undersized+degraded+remapped, 82 active+clean, 1 unknown; 146 MiB data, 684 MiB used, 715 GiB / 715 GiB avail; 2.7 KiB/s rd, 5 op/s; 1660/32457 objects degraded (5.114%); 153 KiB/s, 31 objects/s recovering 2022-05-13T23:29:42.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:29:42 smithi115 ceph-mon[78593]: osdmap e124: 12 total, 8 up, 8 in 2022-05-13T23:29:43.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:29:43 smithi047 ceph-mon[77147]: osdmap e125: 12 total, 8 up, 8 in 2022-05-13T23:29:43.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:29:43 smithi047 ceph-mon[77147]: Health check update: Degraded data redundancy: 1660/32457 objects degraded (5.114%), 14 pgs degraded, 14 pgs undersized (PG_DEGRADED) 2022-05-13T23:29:43.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:29:43 smithi047 ceph-mon[77147]: osdmap e126: 12 total, 8 up, 8 in 2022-05-13T23:29:43.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:29:43 smithi115 ceph-mon[78593]: osdmap e125: 12 total, 8 up, 8 in 2022-05-13T23:29:43.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:29:43 smithi115 ceph-mon[78593]: Health check update: Degraded data redundancy: 1660/32457 objects degraded (5.114%), 14 pgs degraded, 14 pgs undersized (PG_DEGRADED) 2022-05-13T23:29:43.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:29:43 smithi115 ceph-mon[78593]: osdmap e126: 12 total, 8 up, 8 in 2022-05-13T23:29:44.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:29:44 smithi047 ceph-mon[77147]: pgmap v386: 97 pgs: 3 active+recovering+undersized+degraded+remapped, 11 active+recovery_wait+undersized+degraded+remapped, 82 active+clean, 1 unknown; 146 MiB data, 684 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s; 1660/32457 objects degraded (5.114%); 112 KiB/s, 28 objects/s recovering 2022-05-13T23:29:44.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:29:44 smithi047 ceph-mon[77147]: osdmap e127: 12 total, 8 up, 8 in 2022-05-13T23:29:44.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:29:44 smithi115 ceph-mon[78593]: pgmap v386: 97 pgs: 3 active+recovering+undersized+degraded+remapped, 11 active+recovery_wait+undersized+degraded+remapped, 82 active+clean, 1 unknown; 146 MiB data, 684 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s; 1660/32457 objects degraded (5.114%); 112 KiB/s, 28 objects/s recovering 2022-05-13T23:29:44.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:29:44 smithi115 ceph-mon[78593]: osdmap e127: 12 total, 8 up, 8 in 2022-05-13T23:29:46.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:29:46 smithi047 ceph-mon[77147]: osdmap e128: 12 total, 8 up, 8 in 2022-05-13T23:29:46.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:29:46 smithi047 ceph-mon[77147]: pgmap v389: 97 pgs: 1 peering, 1 active+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 10 active+recovery_wait+undersized+degraded+remapped, 83 active+clean, 1 unknown; 146 MiB data, 690 MiB used, 715 GiB / 715 GiB avail; 3.8 KiB/s rd, 7 op/s; 1536/32457 objects degraded (4.732%); 227 KiB/s, 31 objects/s recovering 2022-05-13T23:29:46.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:29:46 smithi115 ceph-mon[78593]: osdmap e128: 12 total, 8 up, 8 in 2022-05-13T23:29:46.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:29:46 smithi115 ceph-mon[78593]: pgmap v389: 97 pgs: 1 peering, 1 active+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 10 active+recovery_wait+undersized+degraded+remapped, 83 active+clean, 1 unknown; 146 MiB data, 690 MiB used, 715 GiB / 715 GiB avail; 3.8 KiB/s rd, 7 op/s; 1536/32457 objects degraded (4.732%); 227 KiB/s, 31 objects/s recovering 2022-05-13T23:29:48.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:29:48 smithi115 ceph-mon[78593]: pgmap v390: 97 pgs: 1 peering, 1 active+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 10 active+recovery_wait+undersized+degraded+remapped, 83 active+clean, 1 unknown; 146 MiB data, 690 MiB used, 715 GiB / 715 GiB avail; 3.1 KiB/s rd, 6 op/s; 1536/32457 objects degraded (4.732%); 181 KiB/s, 25 objects/s recovering 2022-05-13T23:29:48.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:29:48 smithi115 ceph-mon[78593]: Health check update: Degraded data redundancy: 1536/32457 objects degraded (4.732%), 11 pgs degraded, 12 pgs undersized (PG_DEGRADED) 2022-05-13T23:29:48.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:29:48 smithi047 ceph-mon[77147]: pgmap v390: 97 pgs: 1 peering, 1 active+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 10 active+recovery_wait+undersized+degraded+remapped, 83 active+clean, 1 unknown; 146 MiB data, 690 MiB used, 715 GiB / 715 GiB avail; 3.1 KiB/s rd, 6 op/s; 1536/32457 objects degraded (4.732%); 181 KiB/s, 25 objects/s recovering 2022-05-13T23:29:48.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:29:48 smithi047 ceph-mon[77147]: Health check update: Degraded data redundancy: 1536/32457 objects degraded (4.732%), 11 pgs degraded, 12 pgs undersized (PG_DEGRADED) 2022-05-13T23:29:50.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:29:50 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:29:50.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:29:50 smithi047 ceph-mon[77147]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2022-05-13T23:29:50.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:29:50 smithi047 ceph-mon[77147]: pgmap v391: 97 pgs: 1 active+recovering+undersized+degraded+remapped, 10 active+recovery_wait+undersized+degraded+remapped, 85 active+clean, 1 unknown; 146 MiB data, 691 MiB used, 715 GiB / 715 GiB avail; 4.9 KiB/s rd, 9 op/s; 1478/32457 objects degraded (4.554%); 209 KiB/s, 29 objects/s recovering 2022-05-13T23:29:50.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:29:50 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:29:50.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:29:50 smithi115 ceph-mon[78593]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2022-05-13T23:29:50.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:29:50 smithi115 ceph-mon[78593]: pgmap v391: 97 pgs: 1 active+recovering+undersized+degraded+remapped, 10 active+recovery_wait+undersized+degraded+remapped, 85 active+clean, 1 unknown; 146 MiB data, 691 MiB used, 715 GiB / 715 GiB avail; 4.9 KiB/s rd, 9 op/s; 1478/32457 objects degraded (4.554%); 209 KiB/s, 29 objects/s recovering 2022-05-13T23:29:52.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:29:52 smithi115 ceph-mon[78593]: pgmap v392: 97 pgs: 1 active+recovering+undersized+degraded+remapped, 10 active+recovery_wait+undersized+degraded+remapped, 85 active+clean, 1 unknown; 146 MiB data, 691 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s; 1478/32457 objects degraded (4.554%); 161 KiB/s, 22 objects/s recovering 2022-05-13T23:29:52.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:29:52 smithi047 ceph-mon[77147]: pgmap v392: 97 pgs: 1 active+recovering+undersized+degraded+remapped, 10 active+recovery_wait+undersized+degraded+remapped, 85 active+clean, 1 unknown; 146 MiB data, 691 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s; 1478/32457 objects degraded (4.554%); 161 KiB/s, 22 objects/s recovering 2022-05-13T23:29:53.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:29:53 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:29:53.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:29:53 smithi115 ceph-mon[78593]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2022-05-13T23:29:53.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:29:53 smithi115 ceph-mon[78593]: Health check update: Degraded data redundancy: 1478/32457 objects degraded (4.554%), 11 pgs degraded, 11 pgs undersized (PG_DEGRADED) 2022-05-13T23:29:53.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:29:53 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:29:53.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:29:53 smithi047 ceph-mon[77147]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2022-05-13T23:29:53.795 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:29:53 smithi047 ceph-mon[77147]: Health check update: Degraded data redundancy: 1478/32457 objects degraded (4.554%), 11 pgs degraded, 11 pgs undersized (PG_DEGRADED) 2022-05-13T23:29:54.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:29:54 smithi115 ceph-mon[78593]: pgmap v393: 97 pgs: 1 active+recovering+undersized+degraded+remapped, 10 active+recovery_wait+undersized+degraded+remapped, 85 active+clean, 1 unknown; 146 MiB data, 691 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 3 op/s; 1478/32457 objects degraded (4.554%); 44 KiB/s, 6 objects/s recovering 2022-05-13T23:29:54.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:29:54 smithi047 ceph-mon[77147]: pgmap v393: 97 pgs: 1 active+recovering+undersized+degraded+remapped, 10 active+recovery_wait+undersized+degraded+remapped, 85 active+clean, 1 unknown; 146 MiB data, 691 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 3 op/s; 1478/32457 objects degraded (4.554%); 44 KiB/s, 6 objects/s recovering 2022-05-13T23:29:56.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:29:56 smithi115 ceph-mon[78593]: pgmap v394: 97 pgs: 1 active+recovering+undersized+degraded+remapped, 10 active+recovery_wait+undersized+degraded+remapped, 85 active+clean, 1 unknown; 146 MiB data, 691 MiB used, 715 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s; 1420/32457 objects degraded (4.375%); 61 KiB/s, 11 objects/s recovering 2022-05-13T23:29:56.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:29:56 smithi047 ceph-mon[77147]: pgmap v394: 97 pgs: 1 active+recovering+undersized+degraded+remapped, 10 active+recovery_wait+undersized+degraded+remapped, 85 active+clean, 1 unknown; 146 MiB data, 691 MiB used, 715 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s; 1420/32457 objects degraded (4.375%); 61 KiB/s, 11 objects/s recovering 2022-05-13T23:29:57.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:29:57 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:29:57.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:29:57 smithi115 ceph-mon[78593]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2022-05-13T23:29:57.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:29:57 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:29:57.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:29:57 smithi047 ceph-mon[77147]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2022-05-13T23:29:58.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:29:58 smithi115 ceph-mon[78593]: pgmap v395: 97 pgs: 1 active+recovering+undersized+degraded+remapped, 10 active+recovery_wait+undersized+degraded+remapped, 85 active+clean, 1 unknown; 146 MiB data, 691 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s; 1420/32457 objects degraded (4.375%); 52 KiB/s, 9 objects/s recovering 2022-05-13T23:29:58.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:29:58 smithi115 ceph-mon[78593]: Health check update: Degraded data redundancy: 1420/32457 objects degraded (4.375%), 11 pgs degraded, 11 pgs undersized (PG_DEGRADED) 2022-05-13T23:29:58.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:29:58 smithi115 ceph-mon[78593]: osdmap e129: 12 total, 8 up, 8 in 2022-05-13T23:29:58.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:29:58 smithi047 ceph-mon[77147]: pgmap v395: 97 pgs: 1 active+recovering+undersized+degraded+remapped, 10 active+recovery_wait+undersized+degraded+remapped, 85 active+clean, 1 unknown; 146 MiB data, 691 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s; 1420/32457 objects degraded (4.375%); 52 KiB/s, 9 objects/s recovering 2022-05-13T23:29:58.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:29:58 smithi047 ceph-mon[77147]: Health check update: Degraded data redundancy: 1420/32457 objects degraded (4.375%), 11 pgs degraded, 11 pgs undersized (PG_DEGRADED) 2022-05-13T23:29:58.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:29:58 smithi047 ceph-mon[77147]: osdmap e129: 12 total, 8 up, 8 in 2022-05-13T23:30:00.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:30:00 smithi047 ceph-mon[77147]: osdmap e130: 12 total, 8 up, 8 in 2022-05-13T23:30:00.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:30:00 smithi047 ceph-mon[77147]: pgmap v398: 97 pgs: 1 peering, 1 active+recovering+undersized+degraded+remapped, 9 active+recovery_wait+undersized+degraded+remapped, 85 active+clean, 1 unknown; 146 MiB data, 696 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s; 1362/32457 objects degraded (4.196%); 571 KiB/s, 14 objects/s recovering 2022-05-13T23:30:00.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:30:00 smithi047 ceph-mon[77147]: Health detail: HEALTH_WARN 1 MDSs report slow metadata IOs; 1/3 mons down, quorum b,c; Reduced data availability: 1 pg inactive; Degraded data redundancy: 1420/32457 objects degraded (4.375%), 11 pgs degraded, 11 pgs undersized 2022-05-13T23:30:00.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:30:00 smithi047 ceph-mon[77147]: [WRN] MDS_SLOW_METADATA_IO: 1 MDSs report slow metadata IOs 2022-05-13T23:30:00.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:30:00 smithi047 ceph-mon[77147]: mds.l(mds.0): 4 slow metadata IOs are blocked > 180 secs, oldest blocked for 433 secs 2022-05-13T23:30:00.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:30:00 smithi047 ceph-mon[77147]: [WRN] MON_DOWN: 1/3 mons down, quorum b,c 2022-05-13T23:30:00.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:30:00 smithi047 ceph-mon[77147]: mon.a (rank 0) addr [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] is down (out of quorum) 2022-05-13T23:30:00.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:30:00 smithi047 ceph-mon[77147]: [WRN] PG_AVAILABILITY: Reduced data availability: 1 pg inactive 2022-05-13T23:30:00.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:30:00 smithi047 ceph-mon[77147]: pg 2.1 is stuck inactive for 14m, current state unknown, last acting [] 2022-05-13T23:30:00.545 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:30:00 smithi047 ceph-mon[77147]: [WRN] PG_DEGRADED: Degraded data redundancy: 1420/32457 objects degraded (4.375%), 11 pgs degraded, 11 pgs undersized 2022-05-13T23:30:00.545 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:30:00 smithi047 ceph-mon[77147]: pg 3.1 is stuck undersized for 2m, current state active+recovery_wait+undersized+degraded+remapped, last acting [4,2] 2022-05-13T23:30:00.545 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:30:00 smithi047 ceph-mon[77147]: pg 3.3 is stuck undersized for 2m, current state active+recovery_wait+undersized+degraded+remapped, last acting [10,4] 2022-05-13T23:30:00.545 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:30:00 smithi047 ceph-mon[77147]: pg 3.7 is stuck undersized for 2m, current state active+recovery_wait+undersized+degraded+remapped, last acting [11,7] 2022-05-13T23:30:00.545 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:30:00 smithi047 ceph-mon[77147]: pg 3.c is stuck undersized for 2m, current state active+recovering+undersized+degraded+remapped, last acting [7,5] 2022-05-13T23:30:00.546 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:30:00 smithi047 ceph-mon[77147]: pg 3.13 is stuck undersized for 2m, current state active+recovery_wait+undersized+degraded+remapped, last acting [7,4] 2022-05-13T23:30:00.546 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:30:00 smithi047 ceph-mon[77147]: pg 3.1b is stuck undersized for 2m, current state active+recovery_wait+undersized+degraded+remapped, last acting [5,4] 2022-05-13T23:30:00.546 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:30:00 smithi047 ceph-mon[77147]: pg 3.2a is stuck undersized for 2m, current state active+recovery_wait+undersized+degraded+remapped, last acting [1,7] 2022-05-13T23:30:00.546 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:30:00 smithi047 ceph-mon[77147]: pg 3.2c is stuck undersized for 2m, current state active+recovery_wait+undersized+degraded+remapped, last acting [11,8] 2022-05-13T23:30:00.547 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:30:00 smithi047 ceph-mon[77147]: pg 3.31 is stuck undersized for 2m, current state active+recovery_wait+undersized+degraded+remapped, last acting [4,2] 2022-05-13T23:30:00.547 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:30:00 smithi047 ceph-mon[77147]: pg 3.33 is stuck undersized for 2m, current state active+recovery_wait+undersized+degraded+remapped, last acting [10,5] 2022-05-13T23:30:00.547 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:30:00 smithi047 ceph-mon[77147]: pg 3.3d is stuck undersized for 2m, current state active+recovery_wait+undersized+degraded+remapped, last acting [2,1] 2022-05-13T23:30:00.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:30:00 smithi115 ceph-mon[78593]: osdmap e130: 12 total, 8 up, 8 in 2022-05-13T23:30:00.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:30:00 smithi115 ceph-mon[78593]: pgmap v398: 97 pgs: 1 peering, 1 active+recovering+undersized+degraded+remapped, 9 active+recovery_wait+undersized+degraded+remapped, 85 active+clean, 1 unknown; 146 MiB data, 696 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s; 1362/32457 objects degraded (4.196%); 571 KiB/s, 14 objects/s recovering 2022-05-13T23:30:00.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:30:00 smithi115 ceph-mon[78593]: Health detail: HEALTH_WARN 1 MDSs report slow metadata IOs; 1/3 mons down, quorum b,c; Reduced data availability: 1 pg inactive; Degraded data redundancy: 1420/32457 objects degraded (4.375%), 11 pgs degraded, 11 pgs undersized 2022-05-13T23:30:00.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:30:00 smithi115 ceph-mon[78593]: [WRN] MDS_SLOW_METADATA_IO: 1 MDSs report slow metadata IOs 2022-05-13T23:30:00.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:30:00 smithi115 ceph-mon[78593]: mds.l(mds.0): 4 slow metadata IOs are blocked > 180 secs, oldest blocked for 433 secs 2022-05-13T23:30:00.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:30:00 smithi115 ceph-mon[78593]: [WRN] MON_DOWN: 1/3 mons down, quorum b,c 2022-05-13T23:30:00.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:30:00 smithi115 ceph-mon[78593]: mon.a (rank 0) addr [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] is down (out of quorum) 2022-05-13T23:30:00.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:30:00 smithi115 ceph-mon[78593]: [WRN] PG_AVAILABILITY: Reduced data availability: 1 pg inactive 2022-05-13T23:30:00.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:30:00 smithi115 ceph-mon[78593]: pg 2.1 is stuck inactive for 14m, current state unknown, last acting [] 2022-05-13T23:30:00.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:30:00 smithi115 ceph-mon[78593]: [WRN] PG_DEGRADED: Degraded data redundancy: 1420/32457 objects degraded (4.375%), 11 pgs degraded, 11 pgs undersized 2022-05-13T23:30:00.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:30:00 smithi115 ceph-mon[78593]: pg 3.1 is stuck undersized for 2m, current state active+recovery_wait+undersized+degraded+remapped, last acting [4,2] 2022-05-13T23:30:00.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:30:00 smithi115 ceph-mon[78593]: pg 3.3 is stuck undersized for 2m, current state active+recovery_wait+undersized+degraded+remapped, last acting [10,4] 2022-05-13T23:30:00.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:30:00 smithi115 ceph-mon[78593]: pg 3.7 is stuck undersized for 2m, current state active+recovery_wait+undersized+degraded+remapped, last acting [11,7] 2022-05-13T23:30:00.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:30:00 smithi115 ceph-mon[78593]: pg 3.c is stuck undersized for 2m, current state active+recovering+undersized+degraded+remapped, last acting [7,5] 2022-05-13T23:30:00.612 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:30:00 smithi115 ceph-mon[78593]: pg 3.13 is stuck undersized for 2m, current state active+recovery_wait+undersized+degraded+remapped, last acting [7,4] 2022-05-13T23:30:00.612 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:30:00 smithi115 ceph-mon[78593]: pg 3.1b is stuck undersized for 2m, current state active+recovery_wait+undersized+degraded+remapped, last acting [5,4] 2022-05-13T23:30:00.612 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:30:00 smithi115 ceph-mon[78593]: pg 3.2a is stuck undersized for 2m, current state active+recovery_wait+undersized+degraded+remapped, last acting [1,7] 2022-05-13T23:30:00.612 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:30:00 smithi115 ceph-mon[78593]: pg 3.2c is stuck undersized for 2m, current state active+recovery_wait+undersized+degraded+remapped, last acting [11,8] 2022-05-13T23:30:00.612 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:30:00 smithi115 ceph-mon[78593]: pg 3.31 is stuck undersized for 2m, current state active+recovery_wait+undersized+degraded+remapped, last acting [4,2] 2022-05-13T23:30:00.613 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:30:00 smithi115 ceph-mon[78593]: pg 3.33 is stuck undersized for 2m, current state active+recovery_wait+undersized+degraded+remapped, last acting [10,5] 2022-05-13T23:30:00.613 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:30:00 smithi115 ceph-mon[78593]: pg 3.3d is stuck undersized for 2m, current state active+recovery_wait+undersized+degraded+remapped, last acting [2,1] 2022-05-13T23:30:02.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:30:02 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:30:02.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:30:02 smithi047 ceph-mon[77147]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2022-05-13T23:30:02.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:30:02 smithi047 ceph-mon[77147]: pgmap v399: 97 pgs: 1 peering, 1 active+recovering+undersized+degraded+remapped, 9 active+recovery_wait+undersized+degraded+remapped, 85 active+clean, 1 unknown; 146 MiB data, 696 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s; 1362/32457 objects degraded (4.196%); 571 KiB/s, 14 objects/s recovering 2022-05-13T23:30:02.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:30:02 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:30:02.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:30:02 smithi115 ceph-mon[78593]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2022-05-13T23:30:02.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:30:02 smithi115 ceph-mon[78593]: pgmap v399: 97 pgs: 1 peering, 1 active+recovering+undersized+degraded+remapped, 9 active+recovery_wait+undersized+degraded+remapped, 85 active+clean, 1 unknown; 146 MiB data, 696 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s; 1362/32457 objects degraded (4.196%); 571 KiB/s, 14 objects/s recovering 2022-05-13T23:30:03.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:30:03 smithi047 ceph-mon[77147]: Health check update: Degraded data redundancy: 1362/32457 objects degraded (4.196%), 10 pgs degraded, 10 pgs undersized (PG_DEGRADED) 2022-05-13T23:30:03.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:30:03 smithi115 ceph-mon[78593]: Health check update: Degraded data redundancy: 1362/32457 objects degraded (4.196%), 10 pgs degraded, 10 pgs undersized (PG_DEGRADED) 2022-05-13T23:30:04.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:30:04 smithi047 ceph-mon[77147]: pgmap v400: 97 pgs: 1 peering, 1 active+recovering+undersized+degraded+remapped, 9 active+recovery_wait+undersized+degraded+remapped, 85 active+clean, 1 unknown; 146 MiB data, 696 MiB used, 715 GiB / 715 GiB avail; 1.9 KiB/s rd, 3 op/s; 1362/32457 objects degraded (4.196%); 543 KiB/s, 7 objects/s recovering 2022-05-13T23:30:04.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:30:04 smithi115 ceph-mon[78593]: pgmap v400: 97 pgs: 1 peering, 1 active+recovering+undersized+degraded+remapped, 9 active+recovery_wait+undersized+degraded+remapped, 85 active+clean, 1 unknown; 146 MiB data, 696 MiB used, 715 GiB / 715 GiB avail; 1.9 KiB/s rd, 3 op/s; 1362/32457 objects degraded (4.196%); 543 KiB/s, 7 objects/s recovering 2022-05-13T23:30:05.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:30:05 smithi047 ceph-mon[77147]: osdmap e131: 12 total, 8 up, 8 in 2022-05-13T23:30:05.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:30:05 smithi115 ceph-mon[78593]: osdmap e131: 12 total, 8 up, 8 in 2022-05-13T23:30:06.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:30:06 smithi047 ceph-mon[77147]: osdmap e132: 12 total, 8 up, 8 in 2022-05-13T23:30:06.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:30:06 smithi047 ceph-mon[77147]: pgmap v403: 97 pgs: 1 active+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 8 active+recovery_wait+undersized+degraded+remapped, 86 active+clean, 1 unknown; 146 MiB data, 697 MiB used, 715 GiB / 715 GiB avail; 2.4 KiB/s rd, 4 op/s; 1304/32457 objects degraded (4.018%); 47 KiB/s, 9 objects/s recovering 2022-05-13T23:30:06.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:30:06 smithi115 ceph-mon[78593]: osdmap e132: 12 total, 8 up, 8 in 2022-05-13T23:30:06.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:30:06 smithi115 ceph-mon[78593]: pgmap v403: 97 pgs: 1 active+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 8 active+recovery_wait+undersized+degraded+remapped, 86 active+clean, 1 unknown; 146 MiB data, 697 MiB used, 715 GiB / 715 GiB avail; 2.4 KiB/s rd, 4 op/s; 1304/32457 objects degraded (4.018%); 47 KiB/s, 9 objects/s recovering 2022-05-13T23:30:08.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:30:08 smithi115 ceph-mon[78593]: pgmap v404: 97 pgs: 1 active+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 8 active+recovery_wait+undersized+degraded+remapped, 86 active+clean, 1 unknown; 146 MiB data, 697 MiB used, 715 GiB / 715 GiB avail; 1.9 KiB/s rd, 3 op/s; 1304/32457 objects degraded (4.018%); 36 KiB/s, 7 objects/s recovering 2022-05-13T23:30:08.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:30:08 smithi115 ceph-mon[78593]: Health check update: Degraded data redundancy: 1304/32457 objects degraded (4.018%), 9 pgs degraded, 10 pgs undersized (PG_DEGRADED) 2022-05-13T23:30:08.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:30:08 smithi047 ceph-mon[77147]: pgmap v404: 97 pgs: 1 active+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 8 active+recovery_wait+undersized+degraded+remapped, 86 active+clean, 1 unknown; 146 MiB data, 697 MiB used, 715 GiB / 715 GiB avail; 1.9 KiB/s rd, 3 op/s; 1304/32457 objects degraded (4.018%); 36 KiB/s, 7 objects/s recovering 2022-05-13T23:30:08.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:30:08 smithi047 ceph-mon[77147]: Health check update: Degraded data redundancy: 1304/32457 objects degraded (4.018%), 9 pgs degraded, 10 pgs undersized (PG_DEGRADED) 2022-05-13T23:30:10.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:30:10 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:30:10.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:30:10 smithi047 ceph-mon[77147]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2022-05-13T23:30:10.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:30:10 smithi047 ceph-mon[77147]: pgmap v405: 97 pgs: 1 active+recovering+undersized+degraded+remapped, 8 active+recovery_wait+undersized+degraded+remapped, 87 active+clean, 1 unknown; 146 MiB data, 698 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s; 1246/32457 objects degraded (3.839%); 71 KiB/s, 14 objects/s recovering 2022-05-13T23:30:10.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:30:10 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:30:10.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:30:10 smithi115 ceph-mon[78593]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2022-05-13T23:30:10.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:30:10 smithi115 ceph-mon[78593]: pgmap v405: 97 pgs: 1 active+recovering+undersized+degraded+remapped, 8 active+recovery_wait+undersized+degraded+remapped, 87 active+clean, 1 unknown; 146 MiB data, 698 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s; 1246/32457 objects degraded (3.839%); 71 KiB/s, 14 objects/s recovering 2022-05-13T23:30:12.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:30:12 smithi115 ceph-mon[78593]: pgmap v406: 97 pgs: 1 active+recovering+undersized+degraded+remapped, 8 active+recovery_wait+undersized+degraded+remapped, 87 active+clean, 1 unknown; 146 MiB data, 698 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s; 1246/32457 objects degraded (3.839%); 71 KiB/s, 14 objects/s recovering 2022-05-13T23:30:12.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:30:12 smithi047 ceph-mon[77147]: pgmap v406: 97 pgs: 1 active+recovering+undersized+degraded+remapped, 8 active+recovery_wait+undersized+degraded+remapped, 87 active+clean, 1 unknown; 146 MiB data, 698 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s; 1246/32457 objects degraded (3.839%); 71 KiB/s, 14 objects/s recovering 2022-05-13T23:30:13.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:30:13 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:30:13.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:30:13 smithi115 ceph-mon[78593]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2022-05-13T23:30:13.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:30:13 smithi115 ceph-mon[78593]: Health check update: Degraded data redundancy: 1246/32457 objects degraded (3.839%), 9 pgs degraded, 9 pgs undersized (PG_DEGRADED) 2022-05-13T23:30:13.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:30:13 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-13T23:30:13.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:30:13 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-13T23:30:13.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:30:13 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:30:13.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:30:13 smithi047 ceph-mon[77147]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2022-05-13T23:30:13.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:30:13 smithi047 ceph-mon[77147]: Health check update: Degraded data redundancy: 1246/32457 objects degraded (3.839%), 9 pgs degraded, 9 pgs undersized (PG_DEGRADED) 2022-05-13T23:30:13.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:30:13 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-13T23:30:13.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:30:13 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-13T23:30:14.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:30:14 smithi115 ceph-mon[78593]: pgmap v407: 97 pgs: 1 active+recovering+undersized+degraded+remapped, 8 active+recovery_wait+undersized+degraded+remapped, 87 active+clean, 1 unknown; 146 MiB data, 698 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 3 op/s; 1246/32457 objects degraded (3.839%); 30 KiB/s, 6 objects/s recovering 2022-05-13T23:30:14.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:30:14 smithi047 ceph-mon[77147]: pgmap v407: 97 pgs: 1 active+recovering+undersized+degraded+remapped, 8 active+recovery_wait+undersized+degraded+remapped, 87 active+clean, 1 unknown; 146 MiB data, 698 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 3 op/s; 1246/32457 objects degraded (3.839%); 30 KiB/s, 6 objects/s recovering 2022-05-13T23:30:16.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:30:16 smithi115 ceph-mon[78593]: pgmap v408: 97 pgs: 1 active+recovering+undersized+degraded+remapped, 8 active+recovery_wait+undersized+degraded+remapped, 87 active+clean, 1 unknown; 146 MiB data, 698 MiB used, 715 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s; 1187/32457 objects degraded (3.657%); 53 KiB/s, 11 objects/s recovering 2022-05-13T23:30:16.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:30:16 smithi047 ceph-mon[77147]: pgmap v408: 97 pgs: 1 active+recovering+undersized+degraded+remapped, 8 active+recovery_wait+undersized+degraded+remapped, 87 active+clean, 1 unknown; 146 MiB data, 698 MiB used, 715 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s; 1187/32457 objects degraded (3.657%); 53 KiB/s, 11 objects/s recovering 2022-05-13T23:30:17.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:30:17 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:30:17.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:30:17 smithi047 ceph-mon[77147]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2022-05-13T23:30:17.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:30:17 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:30:17.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:30:17 smithi115 ceph-mon[78593]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2022-05-13T23:30:18.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:30:18 smithi047 ceph-mon[77147]: pgmap v409: 97 pgs: 1 active+recovering+undersized+degraded+remapped, 8 active+recovery_wait+undersized+degraded+remapped, 87 active+clean, 1 unknown; 146 MiB data, 698 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s; 1187/32457 objects degraded (3.657%); 45 KiB/s, 9 objects/s recovering 2022-05-13T23:30:18.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:30:18 smithi047 ceph-mon[77147]: osdmap e133: 12 total, 8 up, 8 in 2022-05-13T23:30:18.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:30:18 smithi047 ceph-mon[77147]: Health check update: Degraded data redundancy: 1187/32457 objects degraded (3.657%), 9 pgs degraded, 9 pgs undersized (PG_DEGRADED) 2022-05-13T23:30:18.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:30:18 smithi115 ceph-mon[78593]: pgmap v409: 97 pgs: 1 active+recovering+undersized+degraded+remapped, 8 active+recovery_wait+undersized+degraded+remapped, 87 active+clean, 1 unknown; 146 MiB data, 698 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s; 1187/32457 objects degraded (3.657%); 45 KiB/s, 9 objects/s recovering 2022-05-13T23:30:18.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:30:18 smithi115 ceph-mon[78593]: osdmap e133: 12 total, 8 up, 8 in 2022-05-13T23:30:18.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:30:18 smithi115 ceph-mon[78593]: Health check update: Degraded data redundancy: 1187/32457 objects degraded (3.657%), 9 pgs degraded, 9 pgs undersized (PG_DEGRADED) 2022-05-13T23:30:19.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:30:19 smithi047 ceph-mon[77147]: osdmap e134: 12 total, 8 up, 8 in 2022-05-13T23:30:19.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:30:19 smithi115 ceph-mon[78593]: osdmap e134: 12 total, 8 up, 8 in 2022-05-13T23:30:20.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:30:20 smithi047 ceph-mon[77147]: pgmap v412: 97 pgs: 1 active+recovering+undersized+degraded+remapped, 7 active+recovery_wait+undersized+degraded+remapped, 88 active+clean, 1 unknown; 146 MiB data, 699 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s; 1129/32457 objects degraded (3.478%); 60 KiB/s, 14 objects/s recovering 2022-05-13T23:30:20.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:30:20 smithi115 ceph-mon[78593]: pgmap v412: 97 pgs: 1 active+recovering+undersized+degraded+remapped, 7 active+recovery_wait+undersized+degraded+remapped, 88 active+clean, 1 unknown; 146 MiB data, 699 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s; 1129/32457 objects degraded (3.478%); 60 KiB/s, 14 objects/s recovering 2022-05-13T23:30:21.796 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:30:21 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:30:21.797 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:30:21 smithi047 ceph-mon[77147]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2022-05-13T23:30:21.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:30:21 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:30:21.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:30:21 smithi115 ceph-mon[78593]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2022-05-13T23:30:22.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:30:22 smithi047 ceph-mon[77147]: pgmap v413: 97 pgs: 1 active+recovering+undersized+degraded+remapped, 7 active+recovery_wait+undersized+degraded+remapped, 88 active+clean, 1 unknown; 146 MiB data, 699 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s; 1129/32457 objects degraded (3.478%); 60 KiB/s, 14 objects/s recovering 2022-05-13T23:30:22.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:30:22 smithi115 ceph-mon[78593]: pgmap v413: 97 pgs: 1 active+recovering+undersized+degraded+remapped, 7 active+recovery_wait+undersized+degraded+remapped, 88 active+clean, 1 unknown; 146 MiB data, 699 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s; 1129/32457 objects degraded (3.478%); 60 KiB/s, 14 objects/s recovering 2022-05-13T23:30:23.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:30:23 smithi047 ceph-mon[77147]: Health check update: Degraded data redundancy: 1129/32457 objects degraded (3.478%), 8 pgs degraded, 8 pgs undersized (PG_DEGRADED) 2022-05-13T23:30:23.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:30:23 smithi115 ceph-mon[78593]: Health check update: Degraded data redundancy: 1129/32457 objects degraded (3.478%), 8 pgs degraded, 8 pgs undersized (PG_DEGRADED) 2022-05-13T23:30:24.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:30:24 smithi047 ceph-mon[77147]: pgmap v414: 97 pgs: 1 active+recovering+undersized+degraded+remapped, 7 active+recovery_wait+undersized+degraded+remapped, 88 active+clean, 1 unknown; 146 MiB data, 699 MiB used, 715 GiB / 715 GiB avail; 1.9 KiB/s rd, 3 op/s; 1129/32457 objects degraded (3.478%); 27 KiB/s, 7 objects/s recovering 2022-05-13T23:30:24.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:30:24 smithi115 ceph-mon[78593]: pgmap v414: 97 pgs: 1 active+recovering+undersized+degraded+remapped, 7 active+recovery_wait+undersized+degraded+remapped, 88 active+clean, 1 unknown; 146 MiB data, 699 MiB used, 715 GiB / 715 GiB avail; 1.9 KiB/s rd, 3 op/s; 1129/32457 objects degraded (3.478%); 27 KiB/s, 7 objects/s recovering 2022-05-13T23:30:26.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:30:26 smithi047 ceph-mon[77147]: pgmap v415: 97 pgs: 1 active+recovering+undersized+degraded+remapped, 7 active+recovery_wait+undersized+degraded+remapped, 88 active+clean, 1 unknown; 146 MiB data, 699 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s; 1071/32457 objects degraded (3.300%); 59 KiB/s, 14 objects/s recovering 2022-05-13T23:30:26.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:30:26 smithi115 ceph-mon[78593]: pgmap v415: 97 pgs: 1 active+recovering+undersized+degraded+remapped, 7 active+recovery_wait+undersized+degraded+remapped, 88 active+clean, 1 unknown; 146 MiB data, 699 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s; 1071/32457 objects degraded (3.300%); 59 KiB/s, 14 objects/s recovering 2022-05-13T23:30:28.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:30:28 smithi047 ceph-mon[77147]: pgmap v416: 97 pgs: 1 active+recovering+undersized+degraded+remapped, 7 active+recovery_wait+undersized+degraded+remapped, 88 active+clean, 1 unknown; 146 MiB data, 699 MiB used, 715 GiB / 715 GiB avail; 3.0 KiB/s rd, 6 op/s; 1071/32457 objects degraded (3.300%); 48 KiB/s, 11 objects/s recovering 2022-05-13T23:30:28.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:30:28 smithi047 ceph-mon[77147]: Health check update: Degraded data redundancy: 1071/32457 objects degraded (3.300%), 8 pgs degraded, 8 pgs undersized (PG_DEGRADED) 2022-05-13T23:30:28.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:30:28 smithi115 ceph-mon[78593]: pgmap v416: 97 pgs: 1 active+recovering+undersized+degraded+remapped, 7 active+recovery_wait+undersized+degraded+remapped, 88 active+clean, 1 unknown; 146 MiB data, 699 MiB used, 715 GiB / 715 GiB avail; 3.0 KiB/s rd, 6 op/s; 1071/32457 objects degraded (3.300%); 48 KiB/s, 11 objects/s recovering 2022-05-13T23:30:28.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:30:28 smithi115 ceph-mon[78593]: Health check update: Degraded data redundancy: 1071/32457 objects degraded (3.300%), 8 pgs degraded, 8 pgs undersized (PG_DEGRADED) 2022-05-13T23:30:30.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:30:30 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:30:30.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:30:30 smithi047 ceph-mon[77147]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2022-05-13T23:30:30.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:30:30 smithi047 ceph-mon[77147]: pgmap v417: 97 pgs: 1 active+recovering+undersized+degraded+remapped, 7 active+recovery_wait+undersized+degraded+remapped, 88 active+clean, 1 unknown; 146 MiB data, 700 MiB used, 715 GiB / 715 GiB avail; 4.1 KiB/s rd, 8 op/s; 1013/32457 objects degraded (3.121%); 81 KiB/s, 15 objects/s recovering 2022-05-13T23:30:30.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:30:30 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:30:30.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:30:30 smithi115 ceph-mon[78593]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2022-05-13T23:30:30.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:30:30 smithi115 ceph-mon[78593]: pgmap v417: 97 pgs: 1 active+recovering+undersized+degraded+remapped, 7 active+recovery_wait+undersized+degraded+remapped, 88 active+clean, 1 unknown; 146 MiB data, 700 MiB used, 715 GiB / 715 GiB avail; 4.1 KiB/s rd, 8 op/s; 1013/32457 objects degraded (3.121%); 81 KiB/s, 15 objects/s recovering 2022-05-13T23:30:31.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:30:31 smithi047 ceph-mon[77147]: osdmap e135: 12 total, 8 up, 8 in 2022-05-13T23:30:31.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:30:31 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-13T23:30:31.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:30:31 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-13T23:30:31.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:30:31 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-13T23:30:31.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:30:31 smithi115 ceph-mon[78593]: osdmap e135: 12 total, 8 up, 8 in 2022-05-13T23:30:31.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:30:31 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-13T23:30:31.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:30:31 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-13T23:30:31.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:30:31 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-13T23:30:32.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:30:32 smithi047 ceph-mon[77147]: osdmap e136: 12 total, 8 up, 8 in 2022-05-13T23:30:32.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:30:32 smithi047 ceph-mon[77147]: pgmap v420: 97 pgs: 1 active+recovering+undersized+degraded+remapped, 7 active+recovery_wait+undersized+degraded+remapped, 88 active+clean, 1 unknown; 146 MiB data, 700 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s; 1013/32457 objects degraded (3.121%); 84 KiB/s, 14 objects/s recovering 2022-05-13T23:30:32.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:30:32 smithi115 ceph-mon[78593]: osdmap e136: 12 total, 8 up, 8 in 2022-05-13T23:30:32.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:30:32 smithi115 ceph-mon[78593]: pgmap v420: 97 pgs: 1 active+recovering+undersized+degraded+remapped, 7 active+recovery_wait+undersized+degraded+remapped, 88 active+clean, 1 unknown; 146 MiB data, 700 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s; 1013/32457 objects degraded (3.121%); 84 KiB/s, 14 objects/s recovering 2022-05-13T23:30:33.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:30:33 smithi047 ceph-mon[77147]: Health check update: Degraded data redundancy: 1013/32457 objects degraded (3.121%), 8 pgs degraded, 8 pgs undersized (PG_DEGRADED) 2022-05-13T23:30:33.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:30:33 smithi115 ceph-mon[78593]: Health check update: Degraded data redundancy: 1013/32457 objects degraded (3.121%), 8 pgs degraded, 8 pgs undersized (PG_DEGRADED) 2022-05-13T23:30:34.041 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:30:33 smithi047 conmon[78521]: 2022-05-13T23:30:33.712+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:30:34.042 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:30:33 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-13T23:30:34.042 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:30:33 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-13T23:30:34.042 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:30:33 smithi047 conmon[78521]: yield 2022-05-13T23:30:34.042 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:30:33 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-13T23:30:34.042 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:30:33 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-13T23:30:34.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:30:33 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-13T23:30:34.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:30:33 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-13T23:30:34.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:30:33 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-13T23:30:34.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:30:33 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-13T23:30:34.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:30:33 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-13T23:30:34.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:30:33 smithi047 conmon[78521]: raise exceptions[0] 2022-05-13T23:30:34.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:30:33 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-13T23:30:34.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:30:33 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-13T23:30:34.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:30:33 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-13T23:30:34.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:30:33 smithi047 conmon[78521]: return (yield from fut) 2022-05-13T23:30:34.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:30:33 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-13T23:30:34.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:30:33 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-13T23:30:34.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:30:33 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:30:34.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:30:34 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:30:34.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:30:34 smithi047 ceph-mon[77147]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2022-05-13T23:30:34.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:30:34 smithi047 ceph-mon[77147]: pgmap v421: 97 pgs: 1 active+recovering+undersized+degraded+remapped, 7 active+recovery_wait+undersized+degraded+remapped, 88 active+clean, 1 unknown; 146 MiB data, 700 MiB used, 715 GiB / 715 GiB avail; 1.9 KiB/s rd, 3 op/s; 1013/32457 objects degraded (3.121%); 52 KiB/s, 7 objects/s recovering 2022-05-13T23:30:34.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:30:34 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:30:34.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:30:34 smithi115 ceph-mon[78593]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2022-05-13T23:30:34.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:30:34 smithi115 ceph-mon[78593]: pgmap v421: 97 pgs: 1 active+recovering+undersized+degraded+remapped, 7 active+recovery_wait+undersized+degraded+remapped, 88 active+clean, 1 unknown; 146 MiB data, 700 MiB used, 715 GiB / 715 GiB avail; 1.9 KiB/s rd, 3 op/s; 1013/32457 objects degraded (3.121%); 52 KiB/s, 7 objects/s recovering 2022-05-13T23:30:35.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:30:35 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:30:35.541 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-13T23:30:35.542 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-13T23:30:35.542 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-13T23:30:35.542 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-13T23:30:35.542 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-13T23:30:35.542 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-13T23:30:35.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-13T23:30:35.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-13T23:30:35.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-13T23:30:35.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-13T23:30:35.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-13T23:30:35.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-13T23:30:35.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-13T23:30:35.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-13T23:30:35.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-13T23:30:35.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-13T23:30:35.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-13T23:30:35.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:30:35.545 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:30:35 smithi047 ceph-mon[77147]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-13T23:30:35.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:30:35 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:30:35.608 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-13T23:30:35.608 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-13T23:30:35.609 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-13T23:30:35.609 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-13T23:30:35.609 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-13T23:30:35.609 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-13T23:30:35.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-13T23:30:35.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-13T23:30:35.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-13T23:30:35.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-13T23:30:35.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-13T23:30:35.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-13T23:30:35.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-13T23:30:35.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-13T23:30:35.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-13T23:30:35.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-13T23:30:35.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-13T23:30:35.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:30:35.612 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:30:35 smithi115 ceph-mon[78593]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-13T23:30:36.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:30:36 smithi047 ceph-mon[77147]: pgmap v422: 97 pgs: 2 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 89 active+clean, 1 unknown; 146 MiB data, 703 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s; 902/32457 objects degraded (2.779%); 386 KiB/s, 21 objects/s recovering 2022-05-13T23:30:36.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:30:36 smithi115 ceph-mon[78593]: pgmap v422: 97 pgs: 2 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 89 active+clean, 1 unknown; 146 MiB data, 703 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s; 902/32457 objects degraded (2.779%); 386 KiB/s, 21 objects/s recovering 2022-05-13T23:30:37.041 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:30:36 smithi047 conmon[78521]: 2022-05-13T23:30:36.784+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:30:37.042 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:30:36 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-13T23:30:37.042 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:30:36 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-13T23:30:37.042 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:30:36 smithi047 conmon[78521]: yield 2022-05-13T23:30:37.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:30:36 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-13T23:30:37.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:30:36 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-13T23:30:37.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:30:36 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-13T23:30:37.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:30:36 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-13T23:30:37.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:30:36 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-13T23:30:37.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:30:36 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-13T23:30:37.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:30:36 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-13T23:30:37.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:30:36 smithi047 conmon[78521]: raise exceptions[0] 2022-05-13T23:30:37.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:30:36 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-13T23:30:37.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:30:36 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-13T23:30:37.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:30:36 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-13T23:30:37.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:30:36 smithi047 conmon[78521]: return (yield from fut) 2022-05-13T23:30:37.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:30:36 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-13T23:30:37.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:30:36 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-13T23:30:37.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:30:36 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:30:37.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:30:36 smithi047 conmon[78521]: 2022-05-13T23:30:36.784+0000 7f5345f84700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-13T23:30:37.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:30:36 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-13T23:30:37.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:30:36 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-13T23:30:37.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:30:36 smithi047 conmon[78521]: yield 2022-05-13T23:30:37.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:30:36 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-13T23:30:37.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:30:36 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-13T23:30:37.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:30:36 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-13T23:30:37.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:30:36 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-13T23:30:37.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:30:36 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-13T23:30:37.048 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:30:36 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-13T23:30:37.048 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:30:36 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-13T23:30:37.048 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:30:36 smithi047 conmon[78521]: raise exceptions[0] 2022-05-13T23:30:37.048 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:30:36 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-13T23:30:37.049 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:30:36 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-13T23:30:37.049 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:30:36 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-13T23:30:37.049 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:30:36 smithi047 conmon[78521]: return (yield from fut) 2022-05-13T23:30:37.049 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:30:36 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-13T23:30:37.049 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:30:36 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-13T23:30:37.050 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:30:36 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:30:37.050 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:30:36 smithi047 conmon[78521]: 2022-05-13T23:30:37.050 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:30:36 smithi047 conmon[78521]: During handling of the above exception, another exception occurred: 2022-05-13T23:30:37.050 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:30:36 smithi047 conmon[78521]: 2022-05-13T23:30:37.050 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:30:36 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-13T23:30:37.051 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:30:36 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-13T23:30:37.051 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:30:36 smithi047 conmon[78521]: return f(*arg) 2022-05-13T23:30:37.051 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:30:36 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-13T23:30:37.051 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:30:36 smithi047 conmon[78521]: self._write_client_files(client_files, host) 2022-05-13T23:30:37.052 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:30:36 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-13T23:30:37.052 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:30:36 smithi047 conmon[78521]: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-13T23:30:37.052 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:30:36 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-13T23:30:37.052 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:30:36 smithi047 conmon[78521]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-13T23:30:37.052 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:30:36 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-13T23:30:37.053 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:30:36 smithi047 conmon[78521]: return self.event_loop.get_result(coro) 2022-05-13T23:30:37.053 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:30:36 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-13T23:30:37.053 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:30:36 smithi047 conmon[78521]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-13T23:30:37.053 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:30:36 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-13T23:30:37.053 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:30:36 smithi047 conmon[78521]: return self.__get_result() 2022-05-13T23:30:37.054 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:30:36 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-13T23:30:37.054 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:30:36 smithi047 conmon[78521]: raise self._exception 2022-05-13T23:30:37.054 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:30:36 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-13T23:30:37.054 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:30:36 smithi047 conmon[78521]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-13T23:30:37.054 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:30:36 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-13T23:30:37.055 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:30:36 smithi047 conmon[78521]: conn = await self._remote_connection(host, addr) 2022-05-13T23:30:37.055 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:30:36 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-13T23:30:37.055 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:30:36 smithi047 conmon[78521]: raise 2022-05-13T23:30:37.055 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:30:36 smithi047 conmon[78521]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-13T23:30:37.056 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:30:36 smithi047 conmon[78521]: self.gen.throw(type, value, traceback) 2022-05-13T23:30:37.056 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:30:36 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-13T23:30:37.056 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:30:36 smithi047 conmon[78521]: raise OrchestratorError(msg) 2022-05-13T23:30:37.056 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:30:36 smithi047 conmon[78521]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:30:38.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:30:38 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:30:38.542 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-13T23:30:38.542 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-13T23:30:38.542 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-13T23:30:38.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-13T23:30:38.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-13T23:30:38.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-13T23:30:38.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-13T23:30:38.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-13T23:30:38.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-13T23:30:38.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-13T23:30:38.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-13T23:30:38.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-13T23:30:38.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-13T23:30:38.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-13T23:30:38.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-13T23:30:38.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-13T23:30:38.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-13T23:30:38.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:30:38.547 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:30:38 smithi047 ceph-mon[77147]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-13T23:30:38.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-13T23:30:38.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-13T23:30:38.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-13T23:30:38.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-13T23:30:38.548 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-13T23:30:38.548 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-13T23:30:38.548 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-13T23:30:38.548 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-13T23:30:38.549 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-13T23:30:38.549 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-13T23:30:38.549 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-13T23:30:38.549 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-13T23:30:38.550 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-13T23:30:38.550 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-13T23:30:38.550 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-13T23:30:38.550 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-13T23:30:38.551 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-13T23:30:38.551 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:30:38.551 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-13T23:30:38.551 INFO:journalctl@ceph.mon.b.smithi047.stdout: During handling of the above exception, another exception occurred: 2022-05-13T23:30:38.551 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-13T23:30:38.552 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-13T23:30:38.552 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-13T23:30:38.552 INFO:journalctl@ceph.mon.b.smithi047.stdout: return f(*arg) 2022-05-13T23:30:38.552 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-13T23:30:38.552 INFO:journalctl@ceph.mon.b.smithi047.stdout: self._write_client_files(client_files, host) 2022-05-13T23:30:38.553 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-13T23:30:38.553 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-13T23:30:38.553 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-13T23:30:38.553 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-13T23:30:38.554 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-13T23:30:38.554 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.event_loop.get_result(coro) 2022-05-13T23:30:38.554 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-13T23:30:38.554 INFO:journalctl@ceph.mon.b.smithi047.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-13T23:30:38.554 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-13T23:30:38.555 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.__get_result() 2022-05-13T23:30:38.555 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-13T23:30:38.555 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise self._exception 2022-05-13T23:30:38.555 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-13T23:30:38.556 INFO:journalctl@ceph.mon.b.smithi047.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-13T23:30:38.556 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-13T23:30:38.557 INFO:journalctl@ceph.mon.b.smithi047.stdout: conn = await self._remote_connection(host, addr) 2022-05-13T23:30:38.557 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-13T23:30:38.557 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise 2022-05-13T23:30:38.557 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-13T23:30:38.558 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.gen.throw(type, value, traceback) 2022-05-13T23:30:38.558 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-13T23:30:38.558 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OrchestratorError(msg) 2022-05-13T23:30:38.558 INFO:journalctl@ceph.mon.b.smithi047.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:30:38.559 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:30:38 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:30:38.559 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:30:38 smithi047 ceph-mon[77147]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2022-05-13T23:30:38.559 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:30:38 smithi047 ceph-mon[77147]: pgmap v423: 97 pgs: 2 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 89 active+clean, 1 unknown; 146 MiB data, 703 MiB used, 715 GiB / 715 GiB avail; 1.9 KiB/s rd, 3 op/s; 902/32457 objects degraded (2.779%); 334 KiB/s, 13 objects/s recovering 2022-05-13T23:30:38.559 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:30:38 smithi047 ceph-mon[77147]: Health check update: Degraded data redundancy: 902/32457 objects degraded (2.779%), 7 pgs degraded, 7 pgs undersized (PG_DEGRADED) 2022-05-13T23:30:38.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:30:38 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:30:38.608 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-13T23:30:38.608 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-13T23:30:38.609 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-13T23:30:38.609 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-13T23:30:38.609 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-13T23:30:38.609 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-13T23:30:38.609 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-13T23:30:38.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-13T23:30:38.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-13T23:30:38.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-13T23:30:38.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-13T23:30:38.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-13T23:30:38.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-13T23:30:38.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-13T23:30:38.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-13T23:30:38.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-13T23:30:38.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-13T23:30:38.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:30:38.612 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:30:38 smithi115 ceph-mon[78593]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-13T23:30:38.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-13T23:30:38.613 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-13T23:30:38.613 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-13T23:30:38.613 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-13T23:30:38.613 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-13T23:30:38.613 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-13T23:30:38.614 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-13T23:30:38.614 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-13T23:30:38.614 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-13T23:30:38.614 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-13T23:30:38.615 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-13T23:30:38.615 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-13T23:30:38.615 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-13T23:30:38.615 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-13T23:30:38.615 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-13T23:30:38.616 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-13T23:30:38.616 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-13T23:30:38.616 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:30:38.616 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-13T23:30:38.617 INFO:journalctl@ceph.mon.c.smithi115.stdout: During handling of the above exception, another exception occurred: 2022-05-13T23:30:38.617 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-13T23:30:38.617 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-13T23:30:38.617 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-13T23:30:38.617 INFO:journalctl@ceph.mon.c.smithi115.stdout: return f(*arg) 2022-05-13T23:30:38.618 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-13T23:30:38.618 INFO:journalctl@ceph.mon.c.smithi115.stdout: self._write_client_files(client_files, host) 2022-05-13T23:30:38.618 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-13T23:30:38.618 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-13T23:30:38.618 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-13T23:30:38.619 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-13T23:30:38.619 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-13T23:30:38.619 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.event_loop.get_result(coro) 2022-05-13T23:30:38.619 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-13T23:30:38.620 INFO:journalctl@ceph.mon.c.smithi115.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-13T23:30:38.620 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-13T23:30:38.620 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.__get_result() 2022-05-13T23:30:38.620 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-13T23:30:38.620 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise self._exception 2022-05-13T23:30:38.621 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-13T23:30:38.621 INFO:journalctl@ceph.mon.c.smithi115.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-13T23:30:38.621 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-13T23:30:38.621 INFO:journalctl@ceph.mon.c.smithi115.stdout: conn = await self._remote_connection(host, addr) 2022-05-13T23:30:38.622 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-13T23:30:38.622 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise 2022-05-13T23:30:38.622 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-13T23:30:38.623 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.gen.throw(type, value, traceback) 2022-05-13T23:30:38.623 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-13T23:30:38.623 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OrchestratorError(msg) 2022-05-13T23:30:38.623 INFO:journalctl@ceph.mon.c.smithi115.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:30:38.624 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:30:38 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:30:38.624 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:30:38 smithi115 ceph-mon[78593]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2022-05-13T23:30:38.624 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:30:38 smithi115 ceph-mon[78593]: pgmap v423: 97 pgs: 2 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 89 active+clean, 1 unknown; 146 MiB data, 703 MiB used, 715 GiB / 715 GiB avail; 1.9 KiB/s rd, 3 op/s; 902/32457 objects degraded (2.779%); 334 KiB/s, 13 objects/s recovering 2022-05-13T23:30:38.624 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:30:38 smithi115 ceph-mon[78593]: Health check update: Degraded data redundancy: 902/32457 objects degraded (2.779%), 7 pgs degraded, 7 pgs undersized (PG_DEGRADED) 2022-05-13T23:30:40.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:30:40 smithi115 ceph-mon[78593]: pgmap v424: 97 pgs: 2 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 89 active+clean, 1 unknown; 146 MiB data, 704 MiB used, 715 GiB / 715 GiB avail; 3.3 KiB/s rd, 6 op/s; 784/32457 objects degraded (2.416%); 338 KiB/s, 24 objects/s recovering 2022-05-13T23:30:40.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:30:40 smithi047 ceph-mon[77147]: pgmap v424: 97 pgs: 2 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 89 active+clean, 1 unknown; 146 MiB data, 704 MiB used, 715 GiB / 715 GiB avail; 3.3 KiB/s rd, 6 op/s; 784/32457 objects degraded (2.416%); 338 KiB/s, 24 objects/s recovering 2022-05-13T23:30:41.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:30:41 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:30:41.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:30:41 smithi115 ceph-mon[78593]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2022-05-13T23:30:41.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:30:41 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:30:41.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:30:41 smithi047 ceph-mon[77147]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2022-05-13T23:30:42.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:30:42 smithi047 ceph-mon[77147]: pgmap v425: 97 pgs: 2 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 89 active+clean, 1 unknown; 146 MiB data, 704 MiB used, 715 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s; 784/32457 objects degraded (2.416%); 305 KiB/s, 22 objects/s recovering 2022-05-13T23:30:42.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:30:42 smithi115 ceph-mon[78593]: pgmap v425: 97 pgs: 2 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 89 active+clean, 1 unknown; 146 MiB data, 704 MiB used, 715 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s; 784/32457 objects degraded (2.416%); 305 KiB/s, 22 objects/s recovering 2022-05-13T23:30:43.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:30:43 smithi115 conmon[90393]: 2022-05-13T23:30:43.335+0000 7fa085a08700 -1 osd.5 137 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:30:43.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:30:43 smithi115 ceph-mon[78593]: osdmap e137: 12 total, 8 up, 8 in 2022-05-13T23:30:43.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:30:43 smithi115 ceph-mon[78593]: Health check update: Degraded data redundancy: 784/32457 objects degraded (2.416%), 7 pgs degraded, 7 pgs undersized (PG_DEGRADED) 2022-05-13T23:30:43.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:30:43 smithi047 ceph-mon[77147]: osdmap e137: 12 total, 8 up, 8 in 2022-05-13T23:30:43.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:30:43 smithi047 ceph-mon[77147]: Health check update: Degraded data redundancy: 784/32457 objects degraded (2.416%), 7 pgs degraded, 7 pgs undersized (PG_DEGRADED) 2022-05-13T23:30:44.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:30:44 smithi115 conmon[90393]: 2022-05-13T23:30:44.351+0000 7fa085a08700 -1 osd.5 138 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:30:44.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:30:44 smithi115 ceph-mon[78593]: pgmap v427: 97 pgs: 2 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 89 active+clean, 1 unknown; 146 MiB data, 704 MiB used, 715 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s; 784/32457 objects degraded (2.416%); 310 KiB/s, 22 objects/s recovering 2022-05-13T23:30:44.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:30:44 smithi115 ceph-mon[78593]: 4 slow requests (by type [ 'queued for pg' : 4 ] most affected pool [ 'cephfs_metadata' : 4 ]) 2022-05-13T23:30:44.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:30:44 smithi115 ceph-mon[78593]: osdmap e138: 12 total, 8 up, 8 in 2022-05-13T23:30:44.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:30:44 smithi047 ceph-mon[77147]: pgmap v427: 97 pgs: 2 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 89 active+clean, 1 unknown; 146 MiB data, 704 MiB used, 715 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s; 784/32457 objects degraded (2.416%); 310 KiB/s, 22 objects/s recovering 2022-05-13T23:30:44.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:30:44 smithi047 ceph-mon[77147]: 4 slow requests (by type [ 'queued for pg' : 4 ] most affected pool [ 'cephfs_metadata' : 4 ]) 2022-05-13T23:30:44.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:30:44 smithi047 ceph-mon[77147]: osdmap e138: 12 total, 8 up, 8 in 2022-05-13T23:30:45.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:30:45 smithi047 ceph-mon[77147]: 4 slow requests (by type [ 'queued for pg' : 4 ] most affected pool [ 'cephfs_metadata' : 4 ]) 2022-05-13T23:30:45.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:30:45 smithi047 ceph-mon[77147]: osdmap e139: 12 total, 8 up, 8 in 2022-05-13T23:30:45.863 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:30:45 smithi115 ceph-mon[78593]: 4 slow requests (by type [ 'queued for pg' : 4 ] most affected pool [ 'cephfs_metadata' : 4 ]) 2022-05-13T23:30:45.864 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:30:45 smithi115 ceph-mon[78593]: osdmap e139: 12 total, 8 up, 8 in 2022-05-13T23:30:45.864 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:30:45 smithi115 conmon[90393]: 2022-05-13T23:30:45.368+0000 7fa085a08700 -1 osd.5 139 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:30:46.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:30:46 smithi115 conmon[90393]: 2022-05-13T23:30:46.355+0000 7fa085a08700 -1 osd.5 140 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:30:46.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:30:46 smithi115 ceph-mon[78593]: pgmap v430: 97 pgs: 2 active+recovering+undersized+degraded+remapped, 1 active+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 90 active+clean, 1 unknown; 146 MiB data, 706 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s; 663/32457 objects degraded (2.043%); 116 KiB/s, 20 objects/s recovering 2022-05-13T23:30:46.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:30:46 smithi115 ceph-mon[78593]: 4 slow requests (by type [ 'queued for pg' : 4 ] most affected pool [ 'cephfs_metadata' : 4 ]) 2022-05-13T23:30:46.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:30:46 smithi115 ceph-mon[78593]: Health check failed: 4 slow ops, oldest one blocked for 180 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:30:46.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:30:46 smithi115 ceph-mon[78593]: osdmap e140: 12 total, 8 up, 8 in 2022-05-13T23:30:46.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:30:46 smithi047 ceph-mon[77147]: pgmap v430: 97 pgs: 2 active+recovering+undersized+degraded+remapped, 1 active+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 90 active+clean, 1 unknown; 146 MiB data, 706 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s; 663/32457 objects degraded (2.043%); 116 KiB/s, 20 objects/s recovering 2022-05-13T23:30:46.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:30:46 smithi047 ceph-mon[77147]: 4 slow requests (by type [ 'queued for pg' : 4 ] most affected pool [ 'cephfs_metadata' : 4 ]) 2022-05-13T23:30:46.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:30:46 smithi047 ceph-mon[77147]: Health check failed: 4 slow ops, oldest one blocked for 180 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:30:46.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:30:46 smithi047 ceph-mon[77147]: osdmap e140: 12 total, 8 up, 8 in 2022-05-13T23:30:47.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:30:47 smithi047 ceph-mon[77147]: 4 slow requests (by type [ 'queued for pg' : 4 ] most affected pool [ 'cephfs_metadata' : 4 ]) 2022-05-13T23:30:47.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:30:47 smithi047 ceph-mon[77147]: osdmap e141: 12 total, 8 up, 8 in 2022-05-13T23:30:47.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:30:47 smithi115 conmon[90393]: 2022-05-13T23:30:47.365+0000 7fa085a08700 -1 osd.5 141 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:30:47.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:30:47 smithi115 ceph-mon[78593]: 4 slow requests (by type [ 'queued for pg' : 4 ] most affected pool [ 'cephfs_metadata' : 4 ]) 2022-05-13T23:30:47.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:30:47 smithi115 ceph-mon[78593]: osdmap e141: 12 total, 8 up, 8 in 2022-05-13T23:30:48.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:30:48 smithi115 conmon[90393]: 2022-05-13T23:30:48.351+0000 7fa085a08700 -1 osd.5 141 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:30:48.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:30:48 smithi115 ceph-mon[78593]: pgmap v433: 97 pgs: 2 active+recovering+undersized+degraded+remapped, 1 active+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 90 active+clean, 1 unknown; 146 MiB data, 706 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s; 663/32457 objects degraded (2.043%); 174 KiB/s, 30 objects/s recovering 2022-05-13T23:30:48.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:30:48 smithi115 ceph-mon[78593]: 4 slow requests (by type [ 'queued for pg' : 4 ] most affected pool [ 'cephfs_metadata' : 4 ]) 2022-05-13T23:30:48.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:30:48 smithi115 ceph-mon[78593]: Health check update: Degraded data redundancy: 663/32457 objects degraded (2.043%), 5 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2022-05-13T23:30:48.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:30:48 smithi047 ceph-mon[77147]: pgmap v433: 97 pgs: 2 active+recovering+undersized+degraded+remapped, 1 active+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 90 active+clean, 1 unknown; 146 MiB data, 706 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s; 663/32457 objects degraded (2.043%); 174 KiB/s, 30 objects/s recovering 2022-05-13T23:30:48.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:30:48 smithi047 ceph-mon[77147]: 4 slow requests (by type [ 'queued for pg' : 4 ] most affected pool [ 'cephfs_metadata' : 4 ]) 2022-05-13T23:30:48.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:30:48 smithi047 ceph-mon[77147]: Health check update: Degraded data redundancy: 663/32457 objects degraded (2.043%), 5 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2022-05-13T23:30:49.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:30:49 smithi115 conmon[90393]: 2022-05-13T23:30:49.333+0000 7fa085a08700 -1 osd.5 141 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:30:49.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:30:49 smithi115 ceph-mon[78593]: 4 slow requests (by type [ 'queued for pg' : 4 ] most affected pool [ 'cephfs_metadata' : 4 ]) 2022-05-13T23:30:49.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:30:49 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:30:49.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:30:49 smithi115 ceph-mon[78593]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2022-05-13T23:30:49.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:30:49 smithi047 ceph-mon[77147]: 4 slow requests (by type [ 'queued for pg' : 4 ] most affected pool [ 'cephfs_metadata' : 4 ]) 2022-05-13T23:30:49.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:30:49 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:30:49.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:30:49 smithi047 ceph-mon[77147]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2022-05-13T23:30:50.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:30:50 smithi115 conmon[90393]: 2022-05-13T23:30:50.306+0000 7fa085a08700 -1 osd.5 141 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:30:50.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:30:50 smithi115 ceph-mon[78593]: pgmap v434: 97 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 92 active+clean, 1 unknown; 146 MiB data, 709 MiB used, 715 GiB / 715 GiB avail; 5.1 KiB/s rd, 10 op/s; 437/32457 objects degraded (1.346%); 337 KiB/s, 58 objects/s recovering 2022-05-13T23:30:50.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:30:50 smithi115 ceph-mon[78593]: 4 slow requests (by type [ 'queued for pg' : 4 ] most affected pool [ 'cephfs_metadata' : 4 ]) 2022-05-13T23:30:50.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:30:50 smithi047 ceph-mon[77147]: pgmap v434: 97 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 92 active+clean, 1 unknown; 146 MiB data, 709 MiB used, 715 GiB / 715 GiB avail; 5.1 KiB/s rd, 10 op/s; 437/32457 objects degraded (1.346%); 337 KiB/s, 58 objects/s recovering 2022-05-13T23:30:50.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:30:50 smithi047 ceph-mon[77147]: 4 slow requests (by type [ 'queued for pg' : 4 ] most affected pool [ 'cephfs_metadata' : 4 ]) 2022-05-13T23:30:51.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:30:51 smithi115 conmon[90393]: 2022-05-13T23:30:51.320+0000 7fa085a08700 -1 osd.5 141 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:30:51.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:30:51 smithi115 ceph-mon[78593]: 4 slow requests (by type [ 'queued for pg' : 4 ] most affected pool [ 'cephfs_metadata' : 4 ]) 2022-05-13T23:30:51.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:30:51 smithi047 ceph-mon[77147]: 4 slow requests (by type [ 'queued for pg' : 4 ] most affected pool [ 'cephfs_metadata' : 4 ]) 2022-05-13T23:30:52.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:30:52 smithi115 conmon[90393]: 2022-05-13T23:30:52.355+0000 7fa085a08700 -1 osd.5 141 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:30:52.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:30:52 smithi115 ceph-mon[78593]: pgmap v435: 97 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 92 active+clean, 1 unknown; 146 MiB data, 709 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 4 op/s; 437/32457 objects degraded (1.346%); 189 KiB/s, 32 objects/s recovering 2022-05-13T23:30:52.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:30:52 smithi115 ceph-mon[78593]: 4 slow requests (by type [ 'queued for pg' : 4 ] most affected pool [ 'cephfs_metadata' : 4 ]) 2022-05-13T23:30:52.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:30:52 smithi047 ceph-mon[77147]: pgmap v435: 97 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 92 active+clean, 1 unknown; 146 MiB data, 709 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 4 op/s; 437/32457 objects degraded (1.346%); 189 KiB/s, 32 objects/s recovering 2022-05-13T23:30:52.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:30:52 smithi047 ceph-mon[77147]: 4 slow requests (by type [ 'queued for pg' : 4 ] most affected pool [ 'cephfs_metadata' : 4 ]) 2022-05-13T23:30:53.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:30:53 smithi115 conmon[90393]: 2022-05-13T23:30:53.317+0000 7fa085a08700 -1 osd.5 141 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:30:53.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:30:53 smithi115 ceph-mon[78593]: 4 slow requests (by type [ 'queued for pg' : 4 ] most affected pool [ 'cephfs_metadata' : 4 ]) 2022-05-13T23:30:53.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:30:53 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:30:53.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:30:53 smithi115 ceph-mon[78593]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2022-05-13T23:30:53.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:30:53 smithi115 ceph-mon[78593]: Health check update: Degraded data redundancy: 437/32457 objects degraded (1.346%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2022-05-13T23:30:53.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:30:53 smithi115 ceph-mon[78593]: Health check update: 4 slow ops, oldest one blocked for 185 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:30:53.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:30:53 smithi047 ceph-mon[77147]: 4 slow requests (by type [ 'queued for pg' : 4 ] most affected pool [ 'cephfs_metadata' : 4 ]) 2022-05-13T23:30:53.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:30:53 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:30:53.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:30:53 smithi047 ceph-mon[77147]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2022-05-13T23:30:53.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:30:53 smithi047 ceph-mon[77147]: Health check update: Degraded data redundancy: 437/32457 objects degraded (1.346%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2022-05-13T23:30:53.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:30:53 smithi047 ceph-mon[77147]: Health check update: 4 slow ops, oldest one blocked for 185 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:30:54.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:30:54 smithi115 conmon[90393]: 2022-05-13T23:30:54.323+0000 7fa085a08700 -1 osd.5 141 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:30:54.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:30:54 smithi115 ceph-mon[78593]: pgmap v436: 97 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 92 active+clean, 1 unknown; 146 MiB data, 709 MiB used, 715 GiB / 715 GiB avail; 1.9 KiB/s rd, 3 op/s; 437/32457 objects degraded (1.346%); 163 KiB/s, 28 objects/s recovering 2022-05-13T23:30:54.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:30:54 smithi115 ceph-mon[78593]: 4 slow requests (by type [ 'queued for pg' : 4 ] most affected pool [ 'cephfs_metadata' : 4 ]) 2022-05-13T23:30:54.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:30:54 smithi047 ceph-mon[77147]: pgmap v436: 97 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 92 active+clean, 1 unknown; 146 MiB data, 709 MiB used, 715 GiB / 715 GiB avail; 1.9 KiB/s rd, 3 op/s; 437/32457 objects degraded (1.346%); 163 KiB/s, 28 objects/s recovering 2022-05-13T23:30:54.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:30:54 smithi047 ceph-mon[77147]: 4 slow requests (by type [ 'queued for pg' : 4 ] most affected pool [ 'cephfs_metadata' : 4 ]) 2022-05-13T23:30:55.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:30:55 smithi115 ceph-mon[78593]: 4 slow requests (by type [ 'queued for pg' : 4 ] most affected pool [ 'cephfs_metadata' : 4 ]) 2022-05-13T23:30:55.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:30:55 smithi115 conmon[90393]: 2022-05-13T23:30:55.287+0000 7fa085a08700 -1 osd.5 141 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:30:55.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:30:55 smithi047 ceph-mon[77147]: 4 slow requests (by type [ 'queued for pg' : 4 ] most affected pool [ 'cephfs_metadata' : 4 ]) 2022-05-13T23:30:56.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:30:56 smithi115 conmon[90393]: 2022-05-13T23:30:56.283+0000 7fa085a08700 -1 osd.5 141 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:30:56.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:30:56 smithi115 ceph-mon[78593]: 4 slow requests (by type [ 'queued for pg' : 4 ] most affected pool [ 'cephfs_metadata' : 4 ]) 2022-05-13T23:30:56.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:30:56 smithi115 ceph-mon[78593]: pgmap v437: 97 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 92 active+clean, 1 unknown; 146 MiB data, 709 MiB used, 715 GiB / 715 GiB avail; 3.0 KiB/s rd, 6 op/s; 379/32457 objects degraded (1.168%); 161 KiB/s, 28 objects/s recovering 2022-05-13T23:30:56.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:30:56 smithi047 ceph-mon[77147]: 4 slow requests (by type [ 'queued for pg' : 4 ] most affected pool [ 'cephfs_metadata' : 4 ]) 2022-05-13T23:30:56.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:30:56 smithi047 ceph-mon[77147]: pgmap v437: 97 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 92 active+clean, 1 unknown; 146 MiB data, 709 MiB used, 715 GiB / 715 GiB avail; 3.0 KiB/s rd, 6 op/s; 379/32457 objects degraded (1.168%); 161 KiB/s, 28 objects/s recovering 2022-05-13T23:30:57.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:30:57 smithi115 conmon[90393]: 2022-05-13T23:30:57.323+0000 7fa085a08700 -1 osd.5 142 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:30:57.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:30:57 smithi115 ceph-mon[78593]: 4 slow requests (by type [ 'queued for pg' : 4 ] most affected pool [ 'cephfs_metadata' : 4 ]) 2022-05-13T23:30:57.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:30:57 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:30:57.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:30:57 smithi115 ceph-mon[78593]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2022-05-13T23:30:57.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:30:57 smithi115 ceph-mon[78593]: osdmap e142: 12 total, 8 up, 8 in 2022-05-13T23:30:57.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:30:57 smithi047 ceph-mon[77147]: 4 slow requests (by type [ 'queued for pg' : 4 ] most affected pool [ 'cephfs_metadata' : 4 ]) 2022-05-13T23:30:57.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:30:57 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:30:57.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:30:57 smithi047 ceph-mon[77147]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2022-05-13T23:30:57.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:30:57 smithi047 ceph-mon[77147]: osdmap e142: 12 total, 8 up, 8 in 2022-05-13T23:30:58.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:30:58 smithi115 conmon[90393]: 2022-05-13T23:30:58.279+0000 7fa085a08700 -1 osd.5 143 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:30:58.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:30:58 smithi115 ceph-mon[78593]: pgmap v439: 97 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 92 active+clean, 1 unknown; 146 MiB data, 709 MiB used, 715 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s; 379/32457 objects degraded (1.168%); 159 KiB/s, 28 objects/s recovering 2022-05-13T23:30:58.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:30:58 smithi115 ceph-mon[78593]: 4 slow requests (by type [ 'queued for pg' : 4 ] most affected pool [ 'cephfs_metadata' : 4 ]) 2022-05-13T23:30:58.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:30:58 smithi115 ceph-mon[78593]: osdmap e143: 12 total, 8 up, 8 in 2022-05-13T23:30:58.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:30:58 smithi115 ceph-mon[78593]: Health check update: Degraded data redundancy: 379/32457 objects degraded (1.168%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2022-05-13T23:30:58.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:30:58 smithi115 ceph-mon[78593]: Health check update: 4 slow ops, oldest one blocked for 190 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:30:58.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:30:58 smithi047 ceph-mon[77147]: pgmap v439: 97 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 92 active+clean, 1 unknown; 146 MiB data, 709 MiB used, 715 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s; 379/32457 objects degraded (1.168%); 159 KiB/s, 28 objects/s recovering 2022-05-13T23:30:58.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:30:58 smithi047 ceph-mon[77147]: 4 slow requests (by type [ 'queued for pg' : 4 ] most affected pool [ 'cephfs_metadata' : 4 ]) 2022-05-13T23:30:58.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:30:58 smithi047 ceph-mon[77147]: osdmap e143: 12 total, 8 up, 8 in 2022-05-13T23:30:58.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:30:58 smithi047 ceph-mon[77147]: Health check update: Degraded data redundancy: 379/32457 objects degraded (1.168%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2022-05-13T23:30:58.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:30:58 smithi047 ceph-mon[77147]: Health check update: 4 slow ops, oldest one blocked for 190 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:30:59.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:30:59 smithi115 conmon[90393]: 2022-05-13T23:30:59.309+0000 7fa085a08700 -1 osd.5 143 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:30:59.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:30:59 smithi115 ceph-mon[78593]: 4 slow requests (by type [ 'queued for pg' : 4 ] most affected pool [ 'cephfs_metadata' : 4 ]) 2022-05-13T23:30:59.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:30:59 smithi047 ceph-mon[77147]: 4 slow requests (by type [ 'queued for pg' : 4 ] most affected pool [ 'cephfs_metadata' : 4 ]) 2022-05-13T23:31:00.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:31:00 smithi115 conmon[90393]: 2022-05-13T23:31:00.286+0000 7fa085a08700 -1 osd.5 143 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:31:00.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:31:00 smithi115 ceph-mon[78593]: pgmap v441: 97 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 93 active+clean, 1 unknown; 146 MiB data, 710 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s; 322/32457 objects degraded (0.992%); 60 KiB/s, 14 objects/s recovering 2022-05-13T23:31:00.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:31:00 smithi115 ceph-mon[78593]: 4 slow requests (by type [ 'queued for pg' : 4 ] most affected pool [ 'cephfs_metadata' : 4 ]) 2022-05-13T23:31:00.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:31:00 smithi047 ceph-mon[77147]: pgmap v441: 97 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 93 active+clean, 1 unknown; 146 MiB data, 710 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s; 322/32457 objects degraded (0.992%); 60 KiB/s, 14 objects/s recovering 2022-05-13T23:31:00.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:31:00 smithi047 ceph-mon[77147]: 4 slow requests (by type [ 'queued for pg' : 4 ] most affected pool [ 'cephfs_metadata' : 4 ]) 2022-05-13T23:31:01.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:31:01 smithi115 conmon[90393]: 2022-05-13T23:31:01.261+0000 7fa085a08700 -1 osd.5 143 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:31:01.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:31:01 smithi115 ceph-mon[78593]: 4 slow requests (by type [ 'queued for pg' : 4 ] most affected pool [ 'cephfs_metadata' : 4 ]) 2022-05-13T23:31:01.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:31:01 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:31:01.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:31:01 smithi115 ceph-mon[78593]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2022-05-13T23:31:01.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:31:01 smithi047 ceph-mon[77147]: 4 slow requests (by type [ 'queued for pg' : 4 ] most affected pool [ 'cephfs_metadata' : 4 ]) 2022-05-13T23:31:01.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:31:01 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:31:01.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:31:01 smithi047 ceph-mon[77147]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2022-05-13T23:31:02.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:31:02 smithi115 conmon[90393]: 2022-05-13T23:31:02.250+0000 7fa085a08700 -1 osd.5 143 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:31:02.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:31:02 smithi115 ceph-mon[78593]: 4 slow requests (by type [ 'queued for pg' : 4 ] most affected pool [ 'cephfs_metadata' : 4 ]) 2022-05-13T23:31:02.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:31:02 smithi115 ceph-mon[78593]: pgmap v442: 97 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 93 active+clean, 1 unknown; 146 MiB data, 710 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s; 322/32457 objects degraded (0.992%); 60 KiB/s, 14 objects/s recovering 2022-05-13T23:31:02.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:31:02 smithi047 ceph-mon[77147]: 4 slow requests (by type [ 'queued for pg' : 4 ] most affected pool [ 'cephfs_metadata' : 4 ]) 2022-05-13T23:31:02.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:31:02 smithi047 ceph-mon[77147]: pgmap v442: 97 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 93 active+clean, 1 unknown; 146 MiB data, 710 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s; 322/32457 objects degraded (0.992%); 60 KiB/s, 14 objects/s recovering 2022-05-13T23:31:03.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:31:03 smithi115 conmon[90393]: 2022-05-13T23:31:03.295+0000 7fa085a08700 -1 osd.5 143 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:31:03.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:31:03 smithi115 ceph-mon[78593]: 4 slow requests (by type [ 'queued for pg' : 4 ] most affected pool [ 'cephfs_metadata' : 4 ]) 2022-05-13T23:31:03.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:31:03 smithi115 ceph-mon[78593]: Health check update: Degraded data redundancy: 322/32457 objects degraded (0.992%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2022-05-13T23:31:03.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:31:03 smithi115 ceph-mon[78593]: Health check update: 4 slow ops, oldest one blocked for 195 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:31:03.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:31:03 smithi115 ceph-mon[78593]: pgmap v443: 97 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 93 active+clean, 1 unknown; 146 MiB data, 710 MiB used, 715 GiB / 715 GiB avail; 1.9 KiB/s rd, 3 op/s; 322/32457 objects degraded (0.992%); 24 KiB/s, 7 objects/s recovering 2022-05-13T23:31:03.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:31:03 smithi047 ceph-mon[77147]: 4 slow requests (by type [ 'queued for pg' : 4 ] most affected pool [ 'cephfs_metadata' : 4 ]) 2022-05-13T23:31:03.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:31:03 smithi047 ceph-mon[77147]: Health check update: Degraded data redundancy: 322/32457 objects degraded (0.992%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2022-05-13T23:31:03.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:31:03 smithi047 ceph-mon[77147]: Health check update: 4 slow ops, oldest one blocked for 195 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:31:03.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:31:03 smithi047 ceph-mon[77147]: pgmap v443: 97 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 93 active+clean, 1 unknown; 146 MiB data, 710 MiB used, 715 GiB / 715 GiB avail; 1.9 KiB/s rd, 3 op/s; 322/32457 objects degraded (0.992%); 24 KiB/s, 7 objects/s recovering 2022-05-13T23:31:04.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:31:04 smithi115 conmon[90393]: 2022-05-13T23:31:04.253+0000 7fa085a08700 -1 osd.5 143 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:31:04.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:31:04 smithi115 ceph-mon[78593]: 4 slow requests (by type [ 'queued for pg' : 4 ] most affected pool [ 'cephfs_metadata' : 4 ]) 2022-05-13T23:31:04.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:31:04 smithi047 ceph-mon[77147]: 4 slow requests (by type [ 'queued for pg' : 4 ] most affected pool [ 'cephfs_metadata' : 4 ]) 2022-05-13T23:31:05.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:31:05 smithi115 conmon[90393]: 2022-05-13T23:31:05.246+0000 7fa085a08700 -1 osd.5 143 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:31:05.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:31:05 smithi115 ceph-mon[78593]: 4 slow requests (by type [ 'queued for pg' : 4 ] most affected pool [ 'cephfs_metadata' : 4 ]) 2022-05-13T23:31:05.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:31:05 smithi115 ceph-mon[78593]: pgmap v444: 97 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 93 active+clean, 1 unknown; 146 MiB data, 710 MiB used, 715 GiB / 715 GiB avail; 3.4 KiB/s rd, 6 op/s; 264/32457 objects degraded (0.813%); 49 KiB/s, 12 objects/s recovering 2022-05-13T23:31:05.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:31:05 smithi047 ceph-mon[77147]: 4 slow requests (by type [ 'queued for pg' : 4 ] most affected pool [ 'cephfs_metadata' : 4 ]) 2022-05-13T23:31:05.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:31:05 smithi047 ceph-mon[77147]: pgmap v444: 97 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 93 active+clean, 1 unknown; 146 MiB data, 710 MiB used, 715 GiB / 715 GiB avail; 3.4 KiB/s rd, 6 op/s; 264/32457 objects degraded (0.813%); 49 KiB/s, 12 objects/s recovering 2022-05-13T23:31:06.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:31:06 smithi115 conmon[90393]: 2022-05-13T23:31:06.242+0000 7fa085a08700 -1 osd.5 143 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:31:06.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:31:06 smithi115 ceph-mon[78593]: 4 slow requests (by type [ 'queued for pg' : 4 ] most affected pool [ 'cephfs_metadata' : 4 ]) 2022-05-13T23:31:06.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:31:06 smithi047 ceph-mon[77147]: 4 slow requests (by type [ 'queued for pg' : 4 ] most affected pool [ 'cephfs_metadata' : 4 ]) 2022-05-13T23:31:07.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:31:07 smithi115 conmon[90393]: 2022-05-13T23:31:07.212+0000 7fa085a08700 -1 osd.5 143 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:31:07.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:31:07 smithi115 ceph-mon[78593]: 4 slow requests (by type [ 'queued for pg' : 4 ] most affected pool [ 'cephfs_metadata' : 4 ]) 2022-05-13T23:31:07.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:31:07 smithi115 ceph-mon[78593]: pgmap v445: 97 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 93 active+clean, 1 unknown; 146 MiB data, 710 MiB used, 715 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s; 264/32457 objects degraded (0.813%); 43 KiB/s, 11 objects/s recovering 2022-05-13T23:31:07.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:31:07 smithi047 ceph-mon[77147]: 4 slow requests (by type [ 'queued for pg' : 4 ] most affected pool [ 'cephfs_metadata' : 4 ]) 2022-05-13T23:31:07.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:31:07 smithi047 ceph-mon[77147]: pgmap v445: 97 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 93 active+clean, 1 unknown; 146 MiB data, 710 MiB used, 715 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s; 264/32457 objects degraded (0.813%); 43 KiB/s, 11 objects/s recovering 2022-05-13T23:31:08.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:31:08 smithi115 conmon[90393]: 2022-05-13T23:31:08.215+0000 7fa085a08700 -1 osd.5 143 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:31:08.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:31:08 smithi115 ceph-mon[78593]: 4 slow requests (by type [ 'queued for pg' : 4 ] most affected pool [ 'cephfs_metadata' : 4 ]) 2022-05-13T23:31:08.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:31:08 smithi115 ceph-mon[78593]: Health check update: Degraded data redundancy: 264/32457 objects degraded (0.813%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2022-05-13T23:31:08.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:31:08 smithi115 ceph-mon[78593]: Health check update: 4 slow ops, oldest one blocked for 200 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:31:08.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:31:08 smithi047 ceph-mon[77147]: 4 slow requests (by type [ 'queued for pg' : 4 ] most affected pool [ 'cephfs_metadata' : 4 ]) 2022-05-13T23:31:08.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:31:08 smithi047 ceph-mon[77147]: Health check update: Degraded data redundancy: 264/32457 objects degraded (0.813%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2022-05-13T23:31:08.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:31:08 smithi047 ceph-mon[77147]: Health check update: 4 slow ops, oldest one blocked for 200 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:31:09.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:31:09 smithi115 conmon[90393]: 2022-05-13T23:31:09.213+0000 7fa085a08700 -1 osd.5 143 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:31:09.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:31:09 smithi115 ceph-mon[78593]: 4 slow requests (by type [ 'queued for pg' : 4 ] most affected pool [ 'cephfs_metadata' : 4 ]) 2022-05-13T23:31:09.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:31:09 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:31:09.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:31:09 smithi115 ceph-mon[78593]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2022-05-13T23:31:09.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:31:09 smithi115 ceph-mon[78593]: pgmap v446: 97 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 93 active+clean, 1 unknown; 146 MiB data, 711 MiB used, 715 GiB / 715 GiB avail; 3.8 KiB/s rd, 7 op/s; 205/32457 objects degraded (0.632%); 62 KiB/s, 14 objects/s recovering 2022-05-13T23:31:09.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:31:09 smithi047 ceph-mon[77147]: 4 slow requests (by type [ 'queued for pg' : 4 ] most affected pool [ 'cephfs_metadata' : 4 ]) 2022-05-13T23:31:09.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:31:09 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:31:09.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:31:09 smithi047 ceph-mon[77147]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2022-05-13T23:31:09.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:31:09 smithi047 ceph-mon[77147]: pgmap v446: 97 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 93 active+clean, 1 unknown; 146 MiB data, 711 MiB used, 715 GiB / 715 GiB avail; 3.8 KiB/s rd, 7 op/s; 205/32457 objects degraded (0.632%); 62 KiB/s, 14 objects/s recovering 2022-05-13T23:31:10.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:31:10 smithi115 conmon[90393]: 2022-05-13T23:31:10.243+0000 7fa085a08700 -1 osd.5 143 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:31:10.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:31:10 smithi115 ceph-mon[78593]: 4 slow requests (by type [ 'queued for pg' : 4 ] most affected pool [ 'cephfs_metadata' : 4 ]) 2022-05-13T23:31:10.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:31:10 smithi115 ceph-mon[78593]: osdmap e144: 12 total, 8 up, 8 in 2022-05-13T23:31:10.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:31:10 smithi047 ceph-mon[77147]: 4 slow requests (by type [ 'queued for pg' : 4 ] most affected pool [ 'cephfs_metadata' : 4 ]) 2022-05-13T23:31:10.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:31:10 smithi047 ceph-mon[77147]: osdmap e144: 12 total, 8 up, 8 in 2022-05-13T23:31:11.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:31:11 smithi115 conmon[90393]: 2022-05-13T23:31:11.266+0000 7fa085a08700 -1 osd.5 144 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:31:11.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:31:11 smithi115 ceph-mon[78593]: 4 slow requests (by type [ 'queued for pg' : 4 ] most affected pool [ 'cephfs_metadata' : 4 ]) 2022-05-13T23:31:11.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:31:11 smithi115 ceph-mon[78593]: osdmap e145: 12 total, 8 up, 8 in 2022-05-13T23:31:11.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:31:11 smithi115 ceph-mon[78593]: pgmap v449: 97 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 93 active+clean, 1 unknown; 146 MiB data, 711 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s; 205/32457 objects degraded (0.632%); 68 KiB/s, 14 objects/s recovering 2022-05-13T23:31:11.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:31:11 smithi047 ceph-mon[77147]: 4 slow requests (by type [ 'queued for pg' : 4 ] most affected pool [ 'cephfs_metadata' : 4 ]) 2022-05-13T23:31:11.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:31:11 smithi047 ceph-mon[77147]: osdmap e145: 12 total, 8 up, 8 in 2022-05-13T23:31:11.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:31:11 smithi047 ceph-mon[77147]: pgmap v449: 97 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 93 active+clean, 1 unknown; 146 MiB data, 711 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s; 205/32457 objects degraded (0.632%); 68 KiB/s, 14 objects/s recovering 2022-05-13T23:31:12.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:31:12 smithi115 conmon[90393]: 2022-05-13T23:31:12.265+0000 7fa085a08700 -1 osd.5 145 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:31:12.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:31:12 smithi115 ceph-mon[78593]: 4 slow requests (by type [ 'queued for pg' : 4 ] most affected pool [ 'cephfs_metadata' : 4 ]) 2022-05-13T23:31:12.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:31:12 smithi047 ceph-mon[77147]: 4 slow requests (by type [ 'queued for pg' : 4 ] most affected pool [ 'cephfs_metadata' : 4 ]) 2022-05-13T23:31:13.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:31:13 smithi115 conmon[90393]: 2022-05-13T23:31:13.278+0000 7fa085a08700 -1 osd.5 145 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:31:13.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:31:13 smithi115 ceph-mon[78593]: 4 slow requests (by type [ 'queued for pg' : 4 ] most affected pool [ 'cephfs_metadata' : 4 ]) 2022-05-13T23:31:13.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:31:13 smithi115 ceph-mon[78593]: Health check update: Degraded data redundancy: 205/32457 objects degraded (0.632%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2022-05-13T23:31:13.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:31:13 smithi115 ceph-mon[78593]: Health check update: 4 slow ops, oldest one blocked for 205 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:31:13.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:31:13 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-13T23:31:13.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:31:13 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:31:13.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:31:13 smithi115 ceph-mon[78593]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2022-05-13T23:31:13.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:31:13 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-13T23:31:13.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:31:13 smithi115 ceph-mon[78593]: pgmap v450: 97 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 93 active+clean, 1 unknown; 146 MiB data, 711 MiB used, 715 GiB / 715 GiB avail; 1.9 KiB/s rd, 3 op/s; 205/32457 objects degraded (0.632%); 39 KiB/s, 7 objects/s recovering 2022-05-13T23:31:13.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:31:13 smithi047 ceph-mon[77147]: 4 slow requests (by type [ 'queued for pg' : 4 ] most affected pool [ 'cephfs_metadata' : 4 ]) 2022-05-13T23:31:13.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:31:13 smithi047 ceph-mon[77147]: Health check update: Degraded data redundancy: 205/32457 objects degraded (0.632%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2022-05-13T23:31:13.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:31:13 smithi047 ceph-mon[77147]: Health check update: 4 slow ops, oldest one blocked for 205 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:31:13.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:31:13 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-13T23:31:13.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:31:13 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:31:13.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:31:13 smithi047 ceph-mon[77147]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2022-05-13T23:31:13.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:31:13 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-13T23:31:13.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:31:13 smithi047 ceph-mon[77147]: pgmap v450: 97 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 93 active+clean, 1 unknown; 146 MiB data, 711 MiB used, 715 GiB / 715 GiB avail; 1.9 KiB/s rd, 3 op/s; 205/32457 objects degraded (0.632%); 39 KiB/s, 7 objects/s recovering 2022-05-13T23:31:14.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:31:14 smithi115 ceph-mon[78593]: 4 slow requests (by type [ 'queued for pg' : 4 ] most affected pool [ 'cephfs_metadata' : 4 ]) 2022-05-13T23:31:14.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:31:14 smithi115 conmon[90393]: 2022-05-13T23:31:14.287+0000 7fa085a08700 -1 osd.5 145 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:31:14.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:31:14 smithi047 ceph-mon[77147]: 4 slow requests (by type [ 'queued for pg' : 4 ] most affected pool [ 'cephfs_metadata' : 4 ]) 2022-05-13T23:31:15.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:31:15 smithi115 conmon[90393]: 2022-05-13T23:31:15.238+0000 7fa085a08700 -1 osd.5 145 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:31:15.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:31:15 smithi115 ceph-mon[78593]: 4 slow requests (by type [ 'queued for pg' : 4 ] most affected pool [ 'cephfs_metadata' : 4 ]) 2022-05-13T23:31:15.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:31:15 smithi115 ceph-mon[78593]: pgmap v451: 97 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 94 active+clean, 1 unknown; 146 MiB data, 712 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s; 149/32457 objects degraded (0.459%); 69 KiB/s, 14 objects/s recovering 2022-05-13T23:31:15.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:31:15 smithi047 ceph-mon[77147]: 4 slow requests (by type [ 'queued for pg' : 4 ] most affected pool [ 'cephfs_metadata' : 4 ]) 2022-05-13T23:31:15.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:31:15 smithi047 ceph-mon[77147]: pgmap v451: 97 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 94 active+clean, 1 unknown; 146 MiB data, 712 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s; 149/32457 objects degraded (0.459%); 69 KiB/s, 14 objects/s recovering 2022-05-13T23:31:16.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:31:16 smithi115 conmon[90393]: 2022-05-13T23:31:16.208+0000 7fa085a08700 -1 osd.5 145 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:31:16.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:31:16 smithi115 ceph-mon[78593]: 4 slow requests (by type [ 'queued for pg' : 4 ] most affected pool [ 'cephfs_metadata' : 4 ]) 2022-05-13T23:31:16.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:31:16 smithi115 ceph-mon[78593]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2022-05-13T23:31:16.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:31:16 smithi047 ceph-mon[77147]: 4 slow requests (by type [ 'queued for pg' : 4 ] most affected pool [ 'cephfs_metadata' : 4 ]) 2022-05-13T23:31:16.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:31:16 smithi047 ceph-mon[77147]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2022-05-13T23:31:17.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:31:17 smithi115 conmon[90393]: 2022-05-13T23:31:17.222+0000 7fa085a08700 -1 osd.5 145 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:31:17.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:31:17 smithi115 ceph-mon[78593]: 6 slow requests (by type [ 'queued for pg' : 6 ] most affected pool [ 'cephfs_metadata' : 6 ]) 2022-05-13T23:31:17.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:31:17 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:31:17.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:31:17 smithi115 ceph-mon[78593]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2022-05-13T23:31:17.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:31:17 smithi115 ceph-mon[78593]: pgmap v452: 97 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 94 active+clean, 1 unknown; 146 MiB data, 712 MiB used, 715 GiB / 715 GiB avail; 1.9 KiB/s rd, 3 op/s; 149/32457 objects degraded (0.459%); 31 KiB/s, 6 objects/s recovering 2022-05-13T23:31:17.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:31:17 smithi047 ceph-mon[77147]: 6 slow requests (by type [ 'queued for pg' : 6 ] most affected pool [ 'cephfs_metadata' : 6 ]) 2022-05-13T23:31:17.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:31:17 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:31:17.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:31:17 smithi047 ceph-mon[77147]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2022-05-13T23:31:17.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:31:17 smithi047 ceph-mon[77147]: pgmap v452: 97 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 94 active+clean, 1 unknown; 146 MiB data, 712 MiB used, 715 GiB / 715 GiB avail; 1.9 KiB/s rd, 3 op/s; 149/32457 objects degraded (0.459%); 31 KiB/s, 6 objects/s recovering 2022-05-13T23:31:18.468 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:31:18 smithi115 conmon[90393]: 2022-05-13T23:31:18.202+0000 7fa085a08700 -1 osd.5 145 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:31:18.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:31:18 smithi047 ceph-mon[77147]: 6 slow requests (by type [ 'queued for pg' : 6 ] most affected pool [ 'cephfs_metadata' : 6 ]) 2022-05-13T23:31:18.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:31:18 smithi047 ceph-mon[77147]: Health check update: Degraded data redundancy: 149/32457 objects degraded (0.459%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2022-05-13T23:31:18.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:31:18 smithi047 ceph-mon[77147]: Health check update: 4 slow ops, oldest one blocked for 210 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:31:18.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:31:18 smithi115 ceph-mon[78593]: 6 slow requests (by type [ 'queued for pg' : 6 ] most affected pool [ 'cephfs_metadata' : 6 ]) 2022-05-13T23:31:18.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:31:18 smithi115 ceph-mon[78593]: Health check update: Degraded data redundancy: 149/32457 objects degraded (0.459%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2022-05-13T23:31:18.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:31:18 smithi115 ceph-mon[78593]: Health check update: 4 slow ops, oldest one blocked for 210 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:31:19.472 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:31:19 smithi115 conmon[90393]: 2022-05-13T23:31:19.196+0000 7fa085a08700 -1 osd.5 145 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:31:19.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:31:19 smithi047 ceph-mon[77147]: 6 slow requests (by type [ 'queued for pg' : 6 ] most affected pool [ 'cephfs_metadata' : 6 ]) 2022-05-13T23:31:19.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:31:19 smithi047 ceph-mon[77147]: pgmap v453: 97 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 94 active+clean, 1 unknown; 146 MiB data, 712 MiB used, 715 GiB / 715 GiB avail; 3.3 KiB/s rd, 6 op/s; 91/32457 objects degraded (0.280%); 54 KiB/s, 12 objects/s recovering 2022-05-13T23:31:19.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:31:19 smithi115 ceph-mon[78593]: 6 slow requests (by type [ 'queued for pg' : 6 ] most affected pool [ 'cephfs_metadata' : 6 ]) 2022-05-13T23:31:19.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:31:19 smithi115 ceph-mon[78593]: pgmap v453: 97 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 94 active+clean, 1 unknown; 146 MiB data, 712 MiB used, 715 GiB / 715 GiB avail; 3.3 KiB/s rd, 6 op/s; 91/32457 objects degraded (0.280%); 54 KiB/s, 12 objects/s recovering 2022-05-13T23:31:20.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:31:20 smithi115 conmon[90393]: 2022-05-13T23:31:20.236+0000 7fa085a08700 -1 osd.5 145 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:31:20.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:31:20 smithi115 ceph-mon[78593]: 6 slow requests (by type [ 'queued for pg' : 6 ] most affected pool [ 'cephfs_metadata' : 6 ]) 2022-05-13T23:31:20.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:31:20 smithi047 ceph-mon[77147]: 6 slow requests (by type [ 'queued for pg' : 6 ] most affected pool [ 'cephfs_metadata' : 6 ]) 2022-05-13T23:31:21.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:31:21 smithi115 conmon[90393]: 2022-05-13T23:31:21.272+0000 7fa085a08700 -1 osd.5 145 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:31:21.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:31:21 smithi115 ceph-mon[78593]: 6 slow requests (by type [ 'queued for pg' : 6 ] most affected pool [ 'cephfs_metadata' : 6 ]) 2022-05-13T23:31:21.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:31:21 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:31:21.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:31:21 smithi115 ceph-mon[78593]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2022-05-13T23:31:21.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:31:21 smithi115 ceph-mon[78593]: pgmap v454: 97 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 94 active+clean, 1 unknown; 146 MiB data, 712 MiB used, 715 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s; 91/32457 objects degraded (0.280%); 48 KiB/s, 11 objects/s recovering 2022-05-13T23:31:21.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:31:21 smithi047 ceph-mon[77147]: 6 slow requests (by type [ 'queued for pg' : 6 ] most affected pool [ 'cephfs_metadata' : 6 ]) 2022-05-13T23:31:21.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:31:21 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:31:21.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:31:21 smithi047 ceph-mon[77147]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2022-05-13T23:31:21.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:31:21 smithi047 ceph-mon[77147]: pgmap v454: 97 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 94 active+clean, 1 unknown; 146 MiB data, 712 MiB used, 715 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s; 91/32457 objects degraded (0.280%); 48 KiB/s, 11 objects/s recovering 2022-05-13T23:31:22.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:31:22 smithi115 conmon[90393]: 2022-05-13T23:31:22.232+0000 7fa085a08700 -1 osd.5 145 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:31:22.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:31:22 smithi115 ceph-mon[78593]: 6 slow requests (by type [ 'queued for pg' : 6 ] most affected pool [ 'cephfs_metadata' : 6 ]) 2022-05-13T23:31:22.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:31:22 smithi047 ceph-mon[77147]: 6 slow requests (by type [ 'queued for pg' : 6 ] most affected pool [ 'cephfs_metadata' : 6 ]) 2022-05-13T23:31:23.487 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:31:23 smithi115 conmon[90393]: 2022-05-13T23:31:23.186+0000 7fa085a08700 -1 osd.5 146 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:31:23.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:31:23 smithi047 ceph-mon[77147]: 6 slow requests (by type [ 'queued for pg' : 6 ] most affected pool [ 'cephfs_metadata' : 6 ]) 2022-05-13T23:31:23.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:31:23 smithi047 ceph-mon[77147]: osdmap e146: 12 total, 8 up, 8 in 2022-05-13T23:31:23.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:31:23 smithi047 ceph-mon[77147]: Health check update: Degraded data redundancy: 91/32457 objects degraded (0.280%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2022-05-13T23:31:23.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:31:23 smithi047 ceph-mon[77147]: Health check update: 6 slow ops, oldest one blocked for 215 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:31:23.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:31:23 smithi047 ceph-mon[77147]: pgmap v456: 97 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 94 active+clean, 1 unknown; 146 MiB data, 712 MiB used, 715 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s; 91/32457 objects degraded (0.280%); 49 KiB/s, 11 objects/s recovering 2022-05-13T23:31:23.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:31:23 smithi115 ceph-mon[78593]: 6 slow requests (by type [ 'queued for pg' : 6 ] most affected pool [ 'cephfs_metadata' : 6 ]) 2022-05-13T23:31:23.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:31:23 smithi115 ceph-mon[78593]: osdmap e146: 12 total, 8 up, 8 in 2022-05-13T23:31:23.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:31:23 smithi115 ceph-mon[78593]: Health check update: Degraded data redundancy: 91/32457 objects degraded (0.280%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2022-05-13T23:31:23.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:31:23 smithi115 ceph-mon[78593]: Health check update: 6 slow ops, oldest one blocked for 215 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:31:23.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:31:23 smithi115 ceph-mon[78593]: pgmap v456: 97 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 94 active+clean, 1 unknown; 146 MiB data, 712 MiB used, 715 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s; 91/32457 objects degraded (0.280%); 49 KiB/s, 11 objects/s recovering 2022-05-13T23:31:24.493 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:31:24 smithi115 conmon[90393]: 2022-05-13T23:31:24.228+0000 7fa085a08700 -1 osd.5 147 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:31:24.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:31:24 smithi047 ceph-mon[77147]: 6 slow requests (by type [ 'queued for pg' : 6 ] most affected pool [ 'cephfs_metadata' : 6 ]) 2022-05-13T23:31:24.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:31:24 smithi047 ceph-mon[77147]: osdmap e147: 12 total, 8 up, 8 in 2022-05-13T23:31:24.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:31:24 smithi115 ceph-mon[78593]: 6 slow requests (by type [ 'queued for pg' : 6 ] most affected pool [ 'cephfs_metadata' : 6 ]) 2022-05-13T23:31:24.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:31:24 smithi115 ceph-mon[78593]: osdmap e147: 12 total, 8 up, 8 in 2022-05-13T23:31:25.499 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:31:25 smithi115 conmon[90393]: 2022-05-13T23:31:25.232+0000 7fa085a08700 -1 osd.5 147 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:31:25.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:31:25 smithi047 ceph-mon[77147]: 6 slow requests (by type [ 'queued for pg' : 6 ] most affected pool [ 'cephfs_metadata' : 6 ]) 2022-05-13T23:31:25.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:31:25 smithi047 ceph-mon[77147]: pgmap v458: 97 pgs: 1 active+recovering+undersized+degraded+remapped, 95 active+clean, 1 unknown; 146 MiB data, 713 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s; 34/32457 objects degraded (0.105%); 61 KiB/s, 14 objects/s recovering 2022-05-13T23:31:25.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:31:25 smithi115 ceph-mon[78593]: 6 slow requests (by type [ 'queued for pg' : 6 ] most affected pool [ 'cephfs_metadata' : 6 ]) 2022-05-13T23:31:25.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:31:25 smithi115 ceph-mon[78593]: pgmap v458: 97 pgs: 1 active+recovering+undersized+degraded+remapped, 95 active+clean, 1 unknown; 146 MiB data, 713 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s; 34/32457 objects degraded (0.105%); 61 KiB/s, 14 objects/s recovering 2022-05-13T23:31:26.498 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:31:26 smithi115 conmon[90393]: 2022-05-13T23:31:26.226+0000 7fa085a08700 -1 osd.5 147 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:31:26.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:31:26 smithi047 ceph-mon[77147]: 6 slow requests (by type [ 'queued for pg' : 6 ] most affected pool [ 'cephfs_metadata' : 6 ]) 2022-05-13T23:31:26.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:31:26 smithi115 ceph-mon[78593]: 6 slow requests (by type [ 'queued for pg' : 6 ] most affected pool [ 'cephfs_metadata' : 6 ]) 2022-05-13T23:31:27.508 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:31:27 smithi115 conmon[90393]: 2022-05-13T23:31:27.202+0000 7fa085a08700 -1 osd.5 147 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:31:27.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:31:27 smithi047 ceph-mon[77147]: 6 slow requests (by type [ 'queued for pg' : 6 ] most affected pool [ 'cephfs_metadata' : 6 ]) 2022-05-13T23:31:27.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:31:27 smithi047 ceph-mon[77147]: pgmap v459: 97 pgs: 1 active+recovering+undersized+degraded+remapped, 95 active+clean, 1 unknown; 146 MiB data, 713 MiB used, 715 GiB / 715 GiB avail; 1.9 KiB/s rd, 3 op/s; 34/32457 objects degraded (0.105%); 30 KiB/s, 7 objects/s recovering 2022-05-13T23:31:27.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:31:27 smithi115 ceph-mon[78593]: 6 slow requests (by type [ 'queued for pg' : 6 ] most affected pool [ 'cephfs_metadata' : 6 ]) 2022-05-13T23:31:27.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:31:27 smithi115 ceph-mon[78593]: pgmap v459: 97 pgs: 1 active+recovering+undersized+degraded+remapped, 95 active+clean, 1 unknown; 146 MiB data, 713 MiB used, 715 GiB / 715 GiB avail; 1.9 KiB/s rd, 3 op/s; 34/32457 objects degraded (0.105%); 30 KiB/s, 7 objects/s recovering 2022-05-13T23:31:28.509 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:31:28 smithi115 conmon[90393]: 2022-05-13T23:31:28.224+0000 7fa085a08700 -1 osd.5 148 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:31:28.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:31:28 smithi047 ceph-mon[77147]: 6 slow requests (by type [ 'queued for pg' : 6 ] most affected pool [ 'cephfs_metadata' : 6 ]) 2022-05-13T23:31:28.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:31:28 smithi047 ceph-mon[77147]: osdmap e148: 12 total, 8 up, 8 in 2022-05-13T23:31:28.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:31:28 smithi047 ceph-mon[77147]: Health check update: Degraded data redundancy: 34/32457 objects degraded (0.105%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2022-05-13T23:31:28.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:31:28 smithi047 ceph-mon[77147]: Health check update: 6 slow ops, oldest one blocked for 220 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:31:28.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:31:28 smithi115 ceph-mon[78593]: 6 slow requests (by type [ 'queued for pg' : 6 ] most affected pool [ 'cephfs_metadata' : 6 ]) 2022-05-13T23:31:28.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:31:28 smithi115 ceph-mon[78593]: osdmap e148: 12 total, 8 up, 8 in 2022-05-13T23:31:28.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:31:28 smithi115 ceph-mon[78593]: Health check update: Degraded data redundancy: 34/32457 objects degraded (0.105%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2022-05-13T23:31:28.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:31:28 smithi115 ceph-mon[78593]: Health check update: 6 slow ops, oldest one blocked for 220 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:31:29.511 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:31:29 smithi115 conmon[90393]: 2022-05-13T23:31:29.175+0000 7fa085a08700 -1 osd.5 149 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:31:29.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:31:29 smithi047 ceph-mon[77147]: 6 slow requests (by type [ 'queued for pg' : 6 ] most affected pool [ 'cephfs_metadata' : 6 ]) 2022-05-13T23:31:29.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:31:29 smithi047 ceph-mon[77147]: osdmap e149: 12 total, 8 up, 8 in 2022-05-13T23:31:29.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:31:29 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:31:29.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:31:29 smithi047 ceph-mon[77147]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2022-05-13T23:31:29.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:31:29 smithi047 ceph-mon[77147]: pgmap v462: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 5.0 KiB/s rd, 9 op/s; 65 KiB/s, 15 objects/s recovering 2022-05-13T23:31:29.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:31:29 smithi115 ceph-mon[78593]: 6 slow requests (by type [ 'queued for pg' : 6 ] most affected pool [ 'cephfs_metadata' : 6 ]) 2022-05-13T23:31:29.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:31:29 smithi115 ceph-mon[78593]: osdmap e149: 12 total, 8 up, 8 in 2022-05-13T23:31:29.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:31:29 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:31:29.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:31:29 smithi115 ceph-mon[78593]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2022-05-13T23:31:29.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:31:29 smithi115 ceph-mon[78593]: pgmap v462: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 5.0 KiB/s rd, 9 op/s; 65 KiB/s, 15 objects/s recovering 2022-05-13T23:31:30.512 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:31:30 smithi115 conmon[90393]: 2022-05-13T23:31:30.152+0000 7fa085a08700 -1 osd.5 149 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:31:30.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:31:30 smithi047 ceph-mon[77147]: 6 slow requests (by type [ 'queued for pg' : 6 ] most affected pool [ 'cephfs_metadata' : 6 ]) 2022-05-13T23:31:30.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:31:30 smithi047 ceph-mon[77147]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 34/32457 objects degraded (0.105%), 1 pg degraded, 1 pg undersized) 2022-05-13T23:31:30.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:31:30 smithi115 ceph-mon[78593]: 6 slow requests (by type [ 'queued for pg' : 6 ] most affected pool [ 'cephfs_metadata' : 6 ]) 2022-05-13T23:31:30.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:31:30 smithi115 ceph-mon[78593]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 34/32457 objects degraded (0.105%), 1 pg degraded, 1 pg undersized) 2022-05-13T23:31:31.517 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:31:31 smithi115 conmon[90393]: 2022-05-13T23:31:31.120+0000 7fa085a08700 -1 osd.5 149 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:31:31.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:31:31 smithi047 ceph-mon[77147]: 6 slow requests (by type [ 'queued for pg' : 6 ] most affected pool [ 'cephfs_metadata' : 6 ]) 2022-05-13T23:31:31.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:31:31 smithi047 ceph-mon[77147]: pgmap v463: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.8 KiB/s rd, 7 op/s; 50 KiB/s, 11 objects/s recovering 2022-05-13T23:31:31.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:31:31 smithi115 ceph-mon[78593]: 6 slow requests (by type [ 'queued for pg' : 6 ] most affected pool [ 'cephfs_metadata' : 6 ]) 2022-05-13T23:31:31.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:31:31 smithi115 ceph-mon[78593]: pgmap v463: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.8 KiB/s rd, 7 op/s; 50 KiB/s, 11 objects/s recovering 2022-05-13T23:31:32.518 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:31:32 smithi115 conmon[90393]: 2022-05-13T23:31:32.113+0000 7fa085a08700 -1 osd.5 149 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:31:32.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:31:32 smithi047 ceph-mon[77147]: 6 slow requests (by type [ 'queued for pg' : 6 ] most affected pool [ 'cephfs_metadata' : 6 ]) 2022-05-13T23:31:32.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:31:32 smithi115 ceph-mon[78593]: 6 slow requests (by type [ 'queued for pg' : 6 ] most affected pool [ 'cephfs_metadata' : 6 ]) 2022-05-13T23:31:33.523 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:31:33 smithi115 conmon[90393]: 2022-05-13T23:31:33.114+0000 7fa085a08700 -1 osd.5 149 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:31:33.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:31:33 smithi047 ceph-mon[77147]: 6 slow requests (by type [ 'queued for pg' : 6 ] most affected pool [ 'cephfs_metadata' : 6 ]) 2022-05-13T23:31:33.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:31:33 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:31:33.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:31:33 smithi047 ceph-mon[77147]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2022-05-13T23:31:33.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:31:33 smithi047 ceph-mon[77147]: Health check update: 6 slow ops, oldest one blocked for 225 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:31:33.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:31:33 smithi047 ceph-mon[77147]: pgmap v464: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 1.9 KiB/s rd, 3 op/s; 19 KiB/s, 4 objects/s recovering 2022-05-13T23:31:33.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:31:33 smithi115 ceph-mon[78593]: 6 slow requests (by type [ 'queued for pg' : 6 ] most affected pool [ 'cephfs_metadata' : 6 ]) 2022-05-13T23:31:33.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:31:33 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:31:33.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:31:33 smithi115 ceph-mon[78593]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2022-05-13T23:31:33.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:31:33 smithi115 ceph-mon[78593]: Health check update: 6 slow ops, oldest one blocked for 225 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:31:33.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:31:33 smithi115 ceph-mon[78593]: pgmap v464: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 1.9 KiB/s rd, 3 op/s; 19 KiB/s, 4 objects/s recovering 2022-05-13T23:31:34.525 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:31:34 smithi115 conmon[90393]: 2022-05-13T23:31:34.151+0000 7fa085a08700 -1 osd.5 149 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:31:34.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:31:34 smithi047 ceph-mon[77147]: 6 slow requests (by type [ 'queued for pg' : 6 ] most affected pool [ 'cephfs_metadata' : 6 ]) 2022-05-13T23:31:34.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:31:34 smithi115 ceph-mon[78593]: 6 slow requests (by type [ 'queued for pg' : 6 ] most affected pool [ 'cephfs_metadata' : 6 ]) 2022-05-13T23:31:35.529 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:31:35 smithi115 conmon[90393]: 2022-05-13T23:31:35.187+0000 7fa085a08700 -1 osd.5 149 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:31:35.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:31:35 smithi047 ceph-mon[77147]: 6 slow requests (by type [ 'queued for pg' : 6 ] most affected pool [ 'cephfs_metadata' : 6 ]) 2022-05-13T23:31:35.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:31:35 smithi047 ceph-mon[77147]: pgmap v465: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s; 19 KiB/s, 4 objects/s recovering 2022-05-13T23:31:35.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:31:35 smithi115 ceph-mon[78593]: 6 slow requests (by type [ 'queued for pg' : 6 ] most affected pool [ 'cephfs_metadata' : 6 ]) 2022-05-13T23:31:35.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:31:35 smithi115 ceph-mon[78593]: pgmap v465: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s; 19 KiB/s, 4 objects/s recovering 2022-05-13T23:31:36.534 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:31:36 smithi115 conmon[90393]: 2022-05-13T23:31:36.232+0000 7fa085a08700 -1 osd.5 149 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:31:36.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:31:36 smithi047 ceph-mon[77147]: 6 slow requests (by type [ 'queued for pg' : 6 ] most affected pool [ 'cephfs_metadata' : 6 ]) 2022-05-13T23:31:36.842 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:31:36 smithi115 ceph-mon[78593]: 6 slow requests (by type [ 'queued for pg' : 6 ] most affected pool [ 'cephfs_metadata' : 6 ]) 2022-05-13T23:31:37.539 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:31:37 smithi115 conmon[90393]: 2022-05-13T23:31:37.272+0000 7fa085a08700 -1 osd.5 149 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:31:37.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:31:37 smithi047 ceph-mon[77147]: 8 slow requests (by type [ 'queued for pg' : 8 ] most affected pool [ 'cephfs_metadata' : 8 ]) 2022-05-13T23:31:37.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:31:37 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:31:37.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:31:37 smithi047 ceph-mon[77147]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2022-05-13T23:31:37.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:31:37 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-13T23:31:37.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:31:37 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-13T23:31:37.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:31:37 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-13T23:31:37.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:31:37 smithi047 ceph-mon[77147]: pgmap v466: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.1 KiB/s rd, 6 op/s; 16 KiB/s, 3 objects/s recovering 2022-05-13T23:31:37.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:31:37 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:31:37.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-13T23:31:37.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-13T23:31:37.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-13T23:31:37.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-13T23:31:37.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-13T23:31:37.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-13T23:31:37.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-13T23:31:37.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-13T23:31:37.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-13T23:31:37.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-13T23:31:37.797 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-13T23:31:37.797 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-13T23:31:37.797 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-13T23:31:37.797 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-13T23:31:37.798 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-13T23:31:37.798 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-13T23:31:37.798 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-13T23:31:37.798 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:31:37.798 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:31:37 smithi047 ceph-mon[77147]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-13T23:31:37.799 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:31:37 smithi047 conmon[78521]: 2022-05-13T23:31:37.391+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:31:37.799 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:31:37 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-13T23:31:37.799 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:31:37 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-13T23:31:37.800 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:31:37 smithi047 conmon[78521]: yield 2022-05-13T23:31:37.800 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:31:37 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-13T23:31:37.800 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:31:37 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-13T23:31:37.800 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:31:37 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-13T23:31:37.801 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:31:37 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-13T23:31:37.801 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:31:37 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-13T23:31:37.801 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:31:37 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-13T23:31:37.801 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:31:37 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-13T23:31:37.801 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:31:37 smithi047 conmon[78521]: raise exceptions[0] 2022-05-13T23:31:37.802 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:31:37 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-13T23:31:37.802 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:31:37 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-13T23:31:37.802 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:31:37 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-13T23:31:37.802 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:31:37 smithi047 conmon[78521]: return (yield from fut) 2022-05-13T23:31:37.803 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:31:37 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-13T23:31:37.803 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:31:37 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-13T23:31:37.803 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:31:37 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:31:37.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:31:37 smithi115 ceph-mon[78593]: 8 slow requests (by type [ 'queued for pg' : 8 ] most affected pool [ 'cephfs_metadata' : 8 ]) 2022-05-13T23:31:37.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:31:37 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:31:37.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:31:37 smithi115 ceph-mon[78593]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2022-05-13T23:31:37.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:31:37 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-13T23:31:37.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:31:37 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-13T23:31:37.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:31:37 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-13T23:31:37.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:31:37 smithi115 ceph-mon[78593]: pgmap v466: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.1 KiB/s rd, 6 op/s; 16 KiB/s, 3 objects/s recovering 2022-05-13T23:31:37.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:31:37 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:31:37.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-13T23:31:37.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-13T23:31:37.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-13T23:31:37.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-13T23:31:37.863 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-13T23:31:37.863 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-13T23:31:37.863 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-13T23:31:37.864 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-13T23:31:37.865 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-13T23:31:37.865 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-13T23:31:37.866 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-13T23:31:37.866 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-13T23:31:37.866 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-13T23:31:37.866 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-13T23:31:37.866 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-13T23:31:37.867 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-13T23:31:37.867 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-13T23:31:37.867 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:31:37.867 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:31:37 smithi115 ceph-mon[78593]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-13T23:31:38.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:31:38 smithi115 conmon[90393]: 2022-05-13T23:31:38.290+0000 7fa085a08700 -1 osd.5 149 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:31:38.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:31:38 smithi115 ceph-mon[78593]: 8 slow requests (by type [ 'queued for pg' : 8 ] most affected pool [ 'cephfs_metadata' : 8 ]) 2022-05-13T23:31:38.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:31:38 smithi115 ceph-mon[78593]: Health check update: 6 slow ops, oldest one blocked for 230 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:31:38.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:31:38 smithi047 ceph-mon[77147]: 8 slow requests (by type [ 'queued for pg' : 8 ] most affected pool [ 'cephfs_metadata' : 8 ]) 2022-05-13T23:31:38.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:31:38 smithi047 ceph-mon[77147]: Health check update: 6 slow ops, oldest one blocked for 230 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:31:39.545 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:31:39 smithi115 conmon[90393]: 2022-05-13T23:31:39.247+0000 7fa085a08700 -1 osd.5 149 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:31:39.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:31:39 smithi115 ceph-mon[78593]: 8 slow requests (by type [ 'queued for pg' : 8 ] most affected pool [ 'cephfs_metadata' : 8 ]) 2022-05-13T23:31:39.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:31:39 smithi115 ceph-mon[78593]: pgmap v467: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 4.2 KiB/s rd, 8 op/s; 14 KiB/s, 3 objects/s recovering 2022-05-13T23:31:40.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:31:39 smithi047 ceph-mon[77147]: 8 slow requests (by type [ 'queued for pg' : 8 ] most affected pool [ 'cephfs_metadata' : 8 ]) 2022-05-13T23:31:40.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:31:39 smithi047 ceph-mon[77147]: pgmap v467: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 4.2 KiB/s rd, 8 op/s; 14 KiB/s, 3 objects/s recovering 2022-05-13T23:31:40.551 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:31:40 smithi115 conmon[90393]: 2022-05-13T23:31:40.254+0000 7fa085a08700 -1 osd.5 149 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:31:40.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:31:40 smithi047 ceph-mon[77147]: 8 slow requests (by type [ 'queued for pg' : 8 ] most affected pool [ 'cephfs_metadata' : 8 ]) 2022-05-13T23:31:40.792 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:31:40 smithi047 conmon[78521]: 2022-05-13T23:31:40.463+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:31:40.792 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:31:40 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-13T23:31:40.792 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:31:40 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-13T23:31:40.793 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:31:40 smithi047 conmon[78521]: yield 2022-05-13T23:31:40.793 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:31:40 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-13T23:31:40.793 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:31:40 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-13T23:31:40.793 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:31:40 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-13T23:31:40.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:31:40 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-13T23:31:40.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:31:40 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-13T23:31:40.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:31:40 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-13T23:31:40.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:31:40 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-13T23:31:40.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:31:40 smithi047 conmon[78521]: raise exceptions[0] 2022-05-13T23:31:40.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:31:40 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-13T23:31:40.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:31:40 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-13T23:31:40.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:31:40 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-13T23:31:40.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:31:40 smithi047 conmon[78521]: return (yield from fut) 2022-05-13T23:31:40.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:31:40 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-13T23:31:40.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:31:40 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-13T23:31:40.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:31:40 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:31:40.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:31:40 smithi047 conmon[78521]: 2022-05-13T23:31:40.464+0000 7f5340f7a700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-13T23:31:40.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:31:40 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-13T23:31:40.797 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:31:40 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-13T23:31:40.797 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:31:40 smithi047 conmon[78521]: yield 2022-05-13T23:31:40.797 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:31:40 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-13T23:31:40.797 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:31:40 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-13T23:31:40.798 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:31:40 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-13T23:31:40.798 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:31:40 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-13T23:31:40.799 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:31:40 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-13T23:31:40.799 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:31:40 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-13T23:31:40.799 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:31:40 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-13T23:31:40.799 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:31:40 smithi047 conmon[78521]: raise exceptions[0] 2022-05-13T23:31:40.800 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:31:40 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-13T23:31:40.800 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:31:40 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-13T23:31:40.800 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:31:40 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-13T23:31:40.800 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:31:40 smithi047 conmon[78521]: return (yield from fut) 2022-05-13T23:31:40.801 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:31:40 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-13T23:31:40.801 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:31:40 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-13T23:31:40.801 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:31:40 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:31:40.801 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:31:40 smithi047 conmon[78521]: 2022-05-13T23:31:40.802 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:31:40 smithi047 conmon[78521]: During handling of the above exception, another exception occurred: 2022-05-13T23:31:40.802 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:31:40 smithi047 conmon[78521]: 2022-05-13T23:31:40.802 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:31:40 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-13T23:31:40.803 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:31:40 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-13T23:31:40.803 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:31:40 smithi047 conmon[78521]: return f(*arg) 2022-05-13T23:31:40.803 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:31:40 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-13T23:31:40.804 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:31:40 smithi047 conmon[78521]: self._write_client_files(client_files, host) 2022-05-13T23:31:40.805 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:31:40 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-13T23:31:40.805 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:31:40 smithi047 conmon[78521]: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-13T23:31:40.805 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:31:40 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-13T23:31:40.805 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:31:40 smithi047 conmon[78521]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-13T23:31:40.806 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:31:40 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-13T23:31:40.806 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:31:40 smithi047 conmon[78521]: return self.event_loop.get_result(coro) 2022-05-13T23:31:40.806 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:31:40 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-13T23:31:40.806 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:31:40 smithi047 conmon[78521]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-13T23:31:40.806 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:31:40 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-13T23:31:40.807 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:31:40 smithi047 conmon[78521]: return self.__get_result() 2022-05-13T23:31:40.807 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:31:40 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-13T23:31:40.807 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:31:40 smithi047 conmon[78521]: raise self._exception 2022-05-13T23:31:40.807 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:31:40 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-13T23:31:40.808 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:31:40 smithi047 conmon[78521]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-13T23:31:40.809 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:31:40 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-13T23:31:40.809 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:31:40 smithi047 conmon[78521]: conn = await self._remote_connection(host, addr) 2022-05-13T23:31:40.809 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:31:40 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-13T23:31:40.810 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:31:40 smithi047 conmon[78521]: raise 2022-05-13T23:31:40.810 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:31:40 smithi047 conmon[78521]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-13T23:31:40.810 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:31:40 smithi047 conmon[78521]: self.gen.throw(type, value, traceback) 2022-05-13T23:31:40.810 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:31:40 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-13T23:31:40.810 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:31:40 smithi047 conmon[78521]: raise OrchestratorError(msg) 2022-05-13T23:31:40.811 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:31:40 smithi047 conmon[78521]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:31:40.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:31:40 smithi115 ceph-mon[78593]: 8 slow requests (by type [ 'queued for pg' : 8 ] most affected pool [ 'cephfs_metadata' : 8 ]) 2022-05-13T23:31:41.552 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:31:41 smithi115 conmon[90393]: 2022-05-13T23:31:41.253+0000 7fa085a08700 -1 osd.5 149 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:31:41.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:31:41 smithi115 ceph-mon[78593]: 8 slow requests (by type [ 'queued for pg' : 8 ] most affected pool [ 'cephfs_metadata' : 8 ]) 2022-05-13T23:31:41.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:31:41 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:31:41.859 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-13T23:31:41.859 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-13T23:31:41.859 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-13T23:31:41.860 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-13T23:31:41.860 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-13T23:31:41.860 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-13T23:31:41.860 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-13T23:31:41.860 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-13T23:31:41.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-13T23:31:41.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-13T23:31:41.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-13T23:31:41.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-13T23:31:41.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-13T23:31:41.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-13T23:31:41.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-13T23:31:41.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-13T23:31:41.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-13T23:31:41.863 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:31:41.863 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:31:41 smithi115 ceph-mon[78593]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-13T23:31:41.863 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-13T23:31:41.863 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-13T23:31:41.863 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-13T23:31:41.864 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-13T23:31:41.864 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-13T23:31:41.864 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-13T23:31:41.864 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-13T23:31:41.864 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-13T23:31:41.865 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-13T23:31:41.865 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-13T23:31:41.865 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-13T23:31:41.865 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-13T23:31:41.866 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-13T23:31:41.866 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-13T23:31:41.866 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-13T23:31:41.866 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-13T23:31:41.866 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-13T23:31:41.867 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:31:41.867 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-13T23:31:41.867 INFO:journalctl@ceph.mon.c.smithi115.stdout: During handling of the above exception, another exception occurred: 2022-05-13T23:31:41.867 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-13T23:31:41.867 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-13T23:31:41.868 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-13T23:31:41.868 INFO:journalctl@ceph.mon.c.smithi115.stdout: return f(*arg) 2022-05-13T23:31:41.868 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-13T23:31:41.868 INFO:journalctl@ceph.mon.c.smithi115.stdout: self._write_client_files(client_files, host) 2022-05-13T23:31:41.869 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-13T23:31:41.869 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-13T23:31:41.869 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-13T23:31:41.869 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-13T23:31:41.870 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-13T23:31:41.870 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.event_loop.get_result(coro) 2022-05-13T23:31:41.870 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-13T23:31:41.870 INFO:journalctl@ceph.mon.c.smithi115.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-13T23:31:41.870 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-13T23:31:41.871 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.__get_result() 2022-05-13T23:31:41.871 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-13T23:31:41.871 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise self._exception 2022-05-13T23:31:41.871 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-13T23:31:41.871 INFO:journalctl@ceph.mon.c.smithi115.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-13T23:31:41.872 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-13T23:31:41.872 INFO:journalctl@ceph.mon.c.smithi115.stdout: conn = await self._remote_connection(host, addr) 2022-05-13T23:31:41.872 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-13T23:31:41.872 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise 2022-05-13T23:31:41.872 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-13T23:31:41.873 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.gen.throw(type, value, traceback) 2022-05-13T23:31:41.873 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-13T23:31:41.873 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OrchestratorError(msg) 2022-05-13T23:31:41.873 INFO:journalctl@ceph.mon.c.smithi115.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:31:41.873 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:31:41 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:31:41.874 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:31:41 smithi115 ceph-mon[78593]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2022-05-13T23:31:41.874 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:31:41 smithi115 ceph-mon[78593]: pgmap v468: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:31:42.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:31:41 smithi047 ceph-mon[77147]: 8 slow requests (by type [ 'queued for pg' : 8 ] most affected pool [ 'cephfs_metadata' : 8 ]) 2022-05-13T23:31:42.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:31:41 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:31:42.042 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-13T23:31:42.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-13T23:31:42.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-13T23:31:42.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-13T23:31:42.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-13T23:31:42.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-13T23:31:42.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-13T23:31:42.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-13T23:31:42.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-13T23:31:42.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-13T23:31:42.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-13T23:31:42.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-13T23:31:42.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-13T23:31:42.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-13T23:31:42.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-13T23:31:42.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-13T23:31:42.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-13T23:31:42.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:31:42.046 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:31:41 smithi047 ceph-mon[77147]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-13T23:31:42.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-13T23:31:42.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-13T23:31:42.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-13T23:31:42.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-13T23:31:42.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-13T23:31:42.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-13T23:31:42.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-13T23:31:42.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-13T23:31:42.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-13T23:31:42.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-13T23:31:42.049 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-13T23:31:42.049 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-13T23:31:42.049 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-13T23:31:42.049 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-13T23:31:42.050 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-13T23:31:42.050 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-13T23:31:42.051 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-13T23:31:42.051 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:31:42.051 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-13T23:31:42.051 INFO:journalctl@ceph.mon.b.smithi047.stdout: During handling of the above exception, another exception occurred: 2022-05-13T23:31:42.052 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-13T23:31:42.052 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-13T23:31:42.052 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-13T23:31:42.053 INFO:journalctl@ceph.mon.b.smithi047.stdout: return f(*arg) 2022-05-13T23:31:42.053 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-13T23:31:42.053 INFO:journalctl@ceph.mon.b.smithi047.stdout: self._write_client_files(client_files, host) 2022-05-13T23:31:42.053 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-13T23:31:42.054 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-13T23:31:42.054 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-13T23:31:42.054 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-13T23:31:42.054 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-13T23:31:42.055 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.event_loop.get_result(coro) 2022-05-13T23:31:42.055 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-13T23:31:42.055 INFO:journalctl@ceph.mon.b.smithi047.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-13T23:31:42.055 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-13T23:31:42.056 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.__get_result() 2022-05-13T23:31:42.056 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-13T23:31:42.056 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise self._exception 2022-05-13T23:31:42.057 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-13T23:31:42.057 INFO:journalctl@ceph.mon.b.smithi047.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-13T23:31:42.057 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-13T23:31:42.057 INFO:journalctl@ceph.mon.b.smithi047.stdout: conn = await self._remote_connection(host, addr) 2022-05-13T23:31:42.058 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-13T23:31:42.058 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise 2022-05-13T23:31:42.058 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-13T23:31:42.058 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.gen.throw(type, value, traceback) 2022-05-13T23:31:42.059 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-13T23:31:42.059 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OrchestratorError(msg) 2022-05-13T23:31:42.059 INFO:journalctl@ceph.mon.b.smithi047.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:31:42.059 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:31:41 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:31:42.060 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:31:41 smithi047 ceph-mon[77147]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2022-05-13T23:31:42.060 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:31:41 smithi047 ceph-mon[77147]: pgmap v468: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:31:42.558 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:31:42 smithi115 conmon[90393]: 2022-05-13T23:31:42.252+0000 7fa085a08700 -1 osd.5 149 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:31:42.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:31:42 smithi115 ceph-mon[78593]: 8 slow requests (by type [ 'queued for pg' : 8 ] most affected pool [ 'cephfs_metadata' : 8 ]) 2022-05-13T23:31:43.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:31:42 smithi047 ceph-mon[77147]: 8 slow requests (by type [ 'queued for pg' : 8 ] most affected pool [ 'cephfs_metadata' : 8 ]) 2022-05-13T23:31:43.559 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:31:43 smithi115 conmon[90393]: 2022-05-13T23:31:43.217+0000 7fa085a08700 -1 osd.5 149 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:31:43.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:31:43 smithi115 ceph-mon[78593]: 8 slow requests (by type [ 'queued for pg' : 8 ] most affected pool [ 'cephfs_metadata' : 8 ]) 2022-05-13T23:31:43.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:31:43 smithi115 ceph-mon[78593]: Health check update: 8 slow ops, oldest one blocked for 235 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:31:43.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:31:43 smithi115 ceph-mon[78593]: pgmap v469: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:31:44.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:31:43 smithi047 ceph-mon[77147]: 8 slow requests (by type [ 'queued for pg' : 8 ] most affected pool [ 'cephfs_metadata' : 8 ]) 2022-05-13T23:31:44.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:31:43 smithi047 ceph-mon[77147]: Health check update: 8 slow ops, oldest one blocked for 235 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:31:44.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:31:43 smithi047 ceph-mon[77147]: pgmap v469: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:31:44.561 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:31:44 smithi115 conmon[90393]: 2022-05-13T23:31:44.200+0000 7fa085a08700 -1 osd.5 149 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:31:44.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:31:44 smithi115 ceph-mon[78593]: 8 slow requests (by type [ 'queued for pg' : 8 ] most affected pool [ 'cephfs_metadata' : 8 ]) 2022-05-13T23:31:45.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:31:44 smithi047 ceph-mon[77147]: 8 slow requests (by type [ 'queued for pg' : 8 ] most affected pool [ 'cephfs_metadata' : 8 ]) 2022-05-13T23:31:45.565 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:31:45 smithi115 conmon[90393]: 2022-05-13T23:31:45.210+0000 7fa085a08700 -1 osd.5 149 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:31:45.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:31:45 smithi115 ceph-mon[78593]: 8 slow requests (by type [ 'queued for pg' : 8 ] most affected pool [ 'cephfs_metadata' : 8 ]) 2022-05-13T23:31:45.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:31:45 smithi115 ceph-mon[78593]: pgmap v470: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:31:46.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:31:45 smithi047 ceph-mon[77147]: 8 slow requests (by type [ 'queued for pg' : 8 ] most affected pool [ 'cephfs_metadata' : 8 ]) 2022-05-13T23:31:46.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:31:45 smithi047 ceph-mon[77147]: pgmap v470: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:31:46.567 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:31:46 smithi115 conmon[90393]: 2022-05-13T23:31:46.258+0000 7fa085a08700 -1 osd.5 149 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:31:46.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:31:46 smithi115 ceph-mon[78593]: 8 slow requests (by type [ 'queued for pg' : 8 ] most affected pool [ 'cephfs_metadata' : 8 ]) 2022-05-13T23:31:47.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:31:46 smithi047 ceph-mon[77147]: 8 slow requests (by type [ 'queued for pg' : 8 ] most affected pool [ 'cephfs_metadata' : 8 ]) 2022-05-13T23:31:47.572 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:31:47 smithi115 conmon[90393]: 2022-05-13T23:31:47.255+0000 7fa085a08700 -1 osd.5 149 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:31:47.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:31:47 smithi115 ceph-mon[78593]: 8 slow requests (by type [ 'queued for pg' : 8 ] most affected pool [ 'cephfs_metadata' : 8 ]) 2022-05-13T23:31:47.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:31:47 smithi115 ceph-mon[78593]: pgmap v471: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:31:48.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:31:47 smithi047 ceph-mon[77147]: 8 slow requests (by type [ 'queued for pg' : 8 ] most affected pool [ 'cephfs_metadata' : 8 ]) 2022-05-13T23:31:48.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:31:47 smithi047 ceph-mon[77147]: pgmap v471: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:31:48.573 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:31:48 smithi115 conmon[90393]: 2022-05-13T23:31:48.286+0000 7fa085a08700 -1 osd.5 149 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:31:48.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:31:48 smithi115 ceph-mon[78593]: 8 slow requests (by type [ 'queued for pg' : 8 ] most affected pool [ 'cephfs_metadata' : 8 ]) 2022-05-13T23:31:48.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:31:48 smithi115 ceph-mon[78593]: Health check update: 8 slow ops, oldest one blocked for 240 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:31:49.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:31:48 smithi047 ceph-mon[77147]: 8 slow requests (by type [ 'queued for pg' : 8 ] most affected pool [ 'cephfs_metadata' : 8 ]) 2022-05-13T23:31:49.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:31:48 smithi047 ceph-mon[77147]: Health check update: 8 slow ops, oldest one blocked for 240 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:31:49.578 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:31:49 smithi115 conmon[90393]: 2022-05-13T23:31:49.244+0000 7fa085a08700 -1 osd.5 149 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:31:49.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:31:49 smithi115 ceph-mon[78593]: 8 slow requests (by type [ 'queued for pg' : 8 ] most affected pool [ 'cephfs_metadata' : 8 ]) 2022-05-13T23:31:49.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:31:49 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:31:49.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:31:49 smithi115 ceph-mon[78593]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2022-05-13T23:31:49.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:31:49 smithi115 ceph-mon[78593]: pgmap v472: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:31:50.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:31:49 smithi047 ceph-mon[77147]: 8 slow requests (by type [ 'queued for pg' : 8 ] most affected pool [ 'cephfs_metadata' : 8 ]) 2022-05-13T23:31:50.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:31:49 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:31:50.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:31:49 smithi047 ceph-mon[77147]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2022-05-13T23:31:50.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:31:49 smithi047 ceph-mon[77147]: pgmap v472: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:31:50.579 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:31:50 smithi115 conmon[90393]: 2022-05-13T23:31:50.229+0000 7fa085a08700 -1 osd.5 149 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:31:50.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:31:50 smithi115 ceph-mon[78593]: 8 slow requests (by type [ 'queued for pg' : 8 ] most affected pool [ 'cephfs_metadata' : 8 ]) 2022-05-13T23:31:51.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:31:50 smithi047 ceph-mon[77147]: 8 slow requests (by type [ 'queued for pg' : 8 ] most affected pool [ 'cephfs_metadata' : 8 ]) 2022-05-13T23:31:51.584 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:31:51 smithi115 conmon[90393]: 2022-05-13T23:31:51.208+0000 7fa085a08700 -1 osd.5 149 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:31:51.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:31:51 smithi115 ceph-mon[78593]: 8 slow requests (by type [ 'queued for pg' : 8 ] most affected pool [ 'cephfs_metadata' : 8 ]) 2022-05-13T23:31:51.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:31:51 smithi115 ceph-mon[78593]: pgmap v473: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:31:52.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:31:51 smithi047 ceph-mon[77147]: 8 slow requests (by type [ 'queued for pg' : 8 ] most affected pool [ 'cephfs_metadata' : 8 ]) 2022-05-13T23:31:52.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:31:51 smithi047 ceph-mon[77147]: pgmap v473: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:31:52.585 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:31:52 smithi115 conmon[90393]: 2022-05-13T23:31:52.226+0000 7fa085a08700 -1 osd.5 149 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:31:52.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:31:52 smithi115 ceph-mon[78593]: 8 slow requests (by type [ 'queued for pg' : 8 ] most affected pool [ 'cephfs_metadata' : 8 ]) 2022-05-13T23:31:53.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:31:52 smithi047 ceph-mon[77147]: 8 slow requests (by type [ 'queued for pg' : 8 ] most affected pool [ 'cephfs_metadata' : 8 ]) 2022-05-13T23:31:53.593 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:31:53 smithi115 conmon[90393]: 2022-05-13T23:31:53.224+0000 7fa085a08700 -1 osd.5 149 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:31:53.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:31:53 smithi115 ceph-mon[78593]: 8 slow requests (by type [ 'queued for pg' : 8 ] most affected pool [ 'cephfs_metadata' : 8 ]) 2022-05-13T23:31:53.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:31:53 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:31:53.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:31:53 smithi115 ceph-mon[78593]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2022-05-13T23:31:53.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:31:53 smithi115 ceph-mon[78593]: Health check update: 8 slow ops, oldest one blocked for 245 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:31:53.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:31:53 smithi115 ceph-mon[78593]: pgmap v474: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:31:54.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:31:53 smithi047 ceph-mon[77147]: 8 slow requests (by type [ 'queued for pg' : 8 ] most affected pool [ 'cephfs_metadata' : 8 ]) 2022-05-13T23:31:54.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:31:53 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:31:54.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:31:53 smithi047 ceph-mon[77147]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2022-05-13T23:31:54.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:31:53 smithi047 ceph-mon[77147]: Health check update: 8 slow ops, oldest one blocked for 245 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:31:54.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:31:53 smithi047 ceph-mon[77147]: pgmap v474: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:31:54.594 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:31:54 smithi115 conmon[90393]: 2022-05-13T23:31:54.193+0000 7fa085a08700 -1 osd.5 149 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:31:54.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:31:54 smithi115 ceph-mon[78593]: 8 slow requests (by type [ 'queued for pg' : 8 ] most affected pool [ 'cephfs_metadata' : 8 ]) 2022-05-13T23:31:55.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:31:54 smithi047 ceph-mon[77147]: 8 slow requests (by type [ 'queued for pg' : 8 ] most affected pool [ 'cephfs_metadata' : 8 ]) 2022-05-13T23:31:55.597 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:31:55 smithi115 conmon[90393]: 2022-05-13T23:31:55.192+0000 7fa085a08700 -1 osd.5 149 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:31:55.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:31:55 smithi115 ceph-mon[78593]: 8 slow requests (by type [ 'queued for pg' : 8 ] most affected pool [ 'cephfs_metadata' : 8 ]) 2022-05-13T23:31:55.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:31:55 smithi115 ceph-mon[78593]: pgmap v475: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:31:56.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:31:55 smithi047 ceph-mon[77147]: 8 slow requests (by type [ 'queued for pg' : 8 ] most affected pool [ 'cephfs_metadata' : 8 ]) 2022-05-13T23:31:56.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:31:55 smithi047 ceph-mon[77147]: pgmap v475: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:31:56.605 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:31:56 smithi115 conmon[90393]: 2022-05-13T23:31:56.216+0000 7fa085a08700 -1 osd.5 149 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:31:56.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:31:56 smithi115 ceph-mon[78593]: 8 slow requests (by type [ 'queued for pg' : 8 ] most affected pool [ 'cephfs_metadata' : 8 ]) 2022-05-13T23:31:57.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:31:56 smithi047 ceph-mon[77147]: 8 slow requests (by type [ 'queued for pg' : 8 ] most affected pool [ 'cephfs_metadata' : 8 ]) 2022-05-13T23:31:57.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:31:57 smithi115 conmon[90393]: 2022-05-13T23:31:57.177+0000 7fa085a08700 -1 osd.5 149 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:31:58.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:31:57 smithi047 ceph-mon[77147]: 8 slow requests (by type [ 'queued for pg' : 8 ] most affected pool [ 'cephfs_metadata' : 8 ]) 2022-05-13T23:31:58.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:31:57 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:31:58.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:31:57 smithi047 ceph-mon[77147]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2022-05-13T23:31:58.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:31:57 smithi047 ceph-mon[77147]: pgmap v476: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:31:58.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:31:57 smithi115 ceph-mon[78593]: 8 slow requests (by type [ 'queued for pg' : 8 ] most affected pool [ 'cephfs_metadata' : 8 ]) 2022-05-13T23:31:58.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:31:57 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:31:58.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:31:57 smithi115 ceph-mon[78593]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2022-05-13T23:31:58.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:31:57 smithi115 ceph-mon[78593]: pgmap v476: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:31:58.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:31:58 smithi115 conmon[90393]: 2022-05-13T23:31:58.186+0000 7fa085a08700 -1 osd.5 149 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:31:59.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:31:58 smithi047 ceph-mon[77147]: 8 slow requests (by type [ 'queued for pg' : 8 ] most affected pool [ 'cephfs_metadata' : 8 ]) 2022-05-13T23:31:59.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:31:58 smithi047 ceph-mon[77147]: Health check update: 8 slow ops, oldest one blocked for 250 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:31:59.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:31:58 smithi115 ceph-mon[78593]: 8 slow requests (by type [ 'queued for pg' : 8 ] most affected pool [ 'cephfs_metadata' : 8 ]) 2022-05-13T23:31:59.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:31:58 smithi115 ceph-mon[78593]: Health check update: 8 slow ops, oldest one blocked for 250 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:31:59.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:31:59 smithi115 conmon[90393]: 2022-05-13T23:31:59.169+0000 7fa085a08700 -1 osd.5 149 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:32:00.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:31:59 smithi047 ceph-mon[77147]: 8 slow requests (by type [ 'queued for pg' : 8 ] most affected pool [ 'cephfs_metadata' : 8 ]) 2022-05-13T23:32:00.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:31:59 smithi047 ceph-mon[77147]: pgmap v477: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:32:00.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:31:59 smithi115 ceph-mon[78593]: 8 slow requests (by type [ 'queued for pg' : 8 ] most affected pool [ 'cephfs_metadata' : 8 ]) 2022-05-13T23:32:00.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:31:59 smithi115 ceph-mon[78593]: pgmap v477: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:32:00.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:32:00 smithi115 conmon[90393]: 2022-05-13T23:32:00.145+0000 7fa085a08700 -1 osd.5 149 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:32:01.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:32:00 smithi047 ceph-mon[77147]: 8 slow requests (by type [ 'queued for pg' : 8 ] most affected pool [ 'cephfs_metadata' : 8 ]) 2022-05-13T23:32:01.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:32:00 smithi115 ceph-mon[78593]: 8 slow requests (by type [ 'queued for pg' : 8 ] most affected pool [ 'cephfs_metadata' : 8 ]) 2022-05-13T23:32:01.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:32:01 smithi115 conmon[90393]: 2022-05-13T23:32:01.187+0000 7fa085a08700 -1 osd.5 149 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:32:02.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:32:01 smithi047 ceph-mon[77147]: 8 slow requests (by type [ 'queued for pg' : 8 ] most affected pool [ 'cephfs_metadata' : 8 ]) 2022-05-13T23:32:02.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:32:01 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:32:02.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:32:01 smithi047 ceph-mon[77147]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2022-05-13T23:32:02.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:32:01 smithi047 ceph-mon[77147]: pgmap v478: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:32:02.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:32:01 smithi115 ceph-mon[78593]: 8 slow requests (by type [ 'queued for pg' : 8 ] most affected pool [ 'cephfs_metadata' : 8 ]) 2022-05-13T23:32:02.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:32:01 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:32:02.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:32:01 smithi115 ceph-mon[78593]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2022-05-13T23:32:02.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:32:01 smithi115 ceph-mon[78593]: pgmap v478: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:32:02.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:32:02 smithi115 conmon[90393]: 2022-05-13T23:32:02.164+0000 7fa085a08700 -1 osd.5 149 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:32:03.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:32:02 smithi047 ceph-mon[77147]: 8 slow requests (by type [ 'queued for pg' : 8 ] most affected pool [ 'cephfs_metadata' : 8 ]) 2022-05-13T23:32:03.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:32:02 smithi115 ceph-mon[78593]: 8 slow requests (by type [ 'queued for pg' : 8 ] most affected pool [ 'cephfs_metadata' : 8 ]) 2022-05-13T23:32:03.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:32:03 smithi115 conmon[90393]: 2022-05-13T23:32:03.213+0000 7fa085a08700 -1 osd.5 149 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:32:04.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:32:03 smithi047 ceph-mon[77147]: 8 slow requests (by type [ 'queued for pg' : 8 ] most affected pool [ 'cephfs_metadata' : 8 ]) 2022-05-13T23:32:04.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:32:03 smithi047 ceph-mon[77147]: Health check update: 8 slow ops, oldest one blocked for 255 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:32:04.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:32:03 smithi047 ceph-mon[77147]: pgmap v479: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:32:04.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:32:03 smithi115 ceph-mon[78593]: 8 slow requests (by type [ 'queued for pg' : 8 ] most affected pool [ 'cephfs_metadata' : 8 ]) 2022-05-13T23:32:04.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:32:03 smithi115 ceph-mon[78593]: Health check update: 8 slow ops, oldest one blocked for 255 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:32:04.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:32:03 smithi115 ceph-mon[78593]: pgmap v479: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:32:04.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:32:04 smithi115 conmon[90393]: 2022-05-13T23:32:04.189+0000 7fa085a08700 -1 osd.5 149 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:32:05.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:32:04 smithi047 ceph-mon[77147]: 8 slow requests (by type [ 'queued for pg' : 8 ] most affected pool [ 'cephfs_metadata' : 8 ]) 2022-05-13T23:32:05.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:32:04 smithi115 ceph-mon[78593]: 8 slow requests (by type [ 'queued for pg' : 8 ] most affected pool [ 'cephfs_metadata' : 8 ]) 2022-05-13T23:32:05.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:32:05 smithi115 conmon[90393]: 2022-05-13T23:32:05.216+0000 7fa085a08700 -1 osd.5 149 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:32:06.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:32:05 smithi047 ceph-mon[77147]: 8 slow requests (by type [ 'queued for pg' : 8 ] most affected pool [ 'cephfs_metadata' : 8 ]) 2022-05-13T23:32:06.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:32:05 smithi047 ceph-mon[77147]: pgmap v480: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:32:06.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:32:05 smithi115 ceph-mon[78593]: 8 slow requests (by type [ 'queued for pg' : 8 ] most affected pool [ 'cephfs_metadata' : 8 ]) 2022-05-13T23:32:06.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:32:05 smithi115 ceph-mon[78593]: pgmap v480: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:32:06.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:32:06 smithi115 conmon[90393]: 2022-05-13T23:32:06.253+0000 7fa085a08700 -1 osd.5 149 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:32:07.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:32:06 smithi047 ceph-mon[77147]: 8 slow requests (by type [ 'queued for pg' : 8 ] most affected pool [ 'cephfs_metadata' : 8 ]) 2022-05-13T23:32:07.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:32:06 smithi115 ceph-mon[78593]: 8 slow requests (by type [ 'queued for pg' : 8 ] most affected pool [ 'cephfs_metadata' : 8 ]) 2022-05-13T23:32:07.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:32:07 smithi115 conmon[90393]: 2022-05-13T23:32:07.277+0000 7fa085a08700 -1 osd.5 149 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:32:08.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:32:07 smithi047 ceph-mon[77147]: 8 slow requests (by type [ 'queued for pg' : 8 ] most affected pool [ 'cephfs_metadata' : 8 ]) 2022-05-13T23:32:08.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:32:07 smithi047 ceph-mon[77147]: pgmap v481: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:32:08.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:32:07 smithi115 ceph-mon[78593]: 8 slow requests (by type [ 'queued for pg' : 8 ] most affected pool [ 'cephfs_metadata' : 8 ]) 2022-05-13T23:32:08.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:32:07 smithi115 ceph-mon[78593]: pgmap v481: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:32:08.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:32:08 smithi115 conmon[90393]: 2022-05-13T23:32:08.250+0000 7fa085a08700 -1 osd.5 149 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:32:09.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:32:08 smithi047 ceph-mon[77147]: 8 slow requests (by type [ 'queued for pg' : 8 ] most affected pool [ 'cephfs_metadata' : 8 ]) 2022-05-13T23:32:09.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:32:08 smithi047 ceph-mon[77147]: Health check update: 8 slow ops, oldest one blocked for 260 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:32:09.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:32:08 smithi115 ceph-mon[78593]: 8 slow requests (by type [ 'queued for pg' : 8 ] most affected pool [ 'cephfs_metadata' : 8 ]) 2022-05-13T23:32:09.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:32:08 smithi115 ceph-mon[78593]: Health check update: 8 slow ops, oldest one blocked for 260 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:32:09.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:32:09 smithi115 conmon[90393]: 2022-05-13T23:32:09.206+0000 7fa085a08700 -1 osd.5 149 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:32:10.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:32:09 smithi047 ceph-mon[77147]: 8 slow requests (by type [ 'queued for pg' : 8 ] most affected pool [ 'cephfs_metadata' : 8 ]) 2022-05-13T23:32:10.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:32:09 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:32:10.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:32:09 smithi047 ceph-mon[77147]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2022-05-13T23:32:10.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:32:09 smithi047 ceph-mon[77147]: pgmap v482: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:32:10.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:32:09 smithi115 ceph-mon[78593]: 8 slow requests (by type [ 'queued for pg' : 8 ] most affected pool [ 'cephfs_metadata' : 8 ]) 2022-05-13T23:32:10.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:32:09 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:32:10.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:32:09 smithi115 ceph-mon[78593]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2022-05-13T23:32:10.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:32:09 smithi115 ceph-mon[78593]: pgmap v482: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:32:10.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:32:10 smithi115 conmon[90393]: 2022-05-13T23:32:10.244+0000 7fa085a08700 -1 osd.5 149 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:32:11.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:32:10 smithi047 ceph-mon[77147]: 8 slow requests (by type [ 'queued for pg' : 8 ] most affected pool [ 'cephfs_metadata' : 8 ]) 2022-05-13T23:32:11.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:32:10 smithi115 ceph-mon[78593]: 8 slow requests (by type [ 'queued for pg' : 8 ] most affected pool [ 'cephfs_metadata' : 8 ]) 2022-05-13T23:32:11.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:32:11 smithi115 conmon[90393]: 2022-05-13T23:32:11.222+0000 7fa085a08700 -1 osd.5 149 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:32:12.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:32:11 smithi047 ceph-mon[77147]: 8 slow requests (by type [ 'queued for pg' : 8 ] most affected pool [ 'cephfs_metadata' : 8 ]) 2022-05-13T23:32:12.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:32:11 smithi047 ceph-mon[77147]: pgmap v483: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:32:12.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:32:11 smithi115 ceph-mon[78593]: 8 slow requests (by type [ 'queued for pg' : 8 ] most affected pool [ 'cephfs_metadata' : 8 ]) 2022-05-13T23:32:12.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:32:11 smithi115 ceph-mon[78593]: pgmap v483: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:32:12.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:32:12 smithi115 conmon[90393]: 2022-05-13T23:32:12.252+0000 7fa085a08700 -1 osd.5 149 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:32:13.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:32:12 smithi047 ceph-mon[77147]: 8 slow requests (by type [ 'queued for pg' : 8 ] most affected pool [ 'cephfs_metadata' : 8 ]) 2022-05-13T23:32:13.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:32:12 smithi115 ceph-mon[78593]: 8 slow requests (by type [ 'queued for pg' : 8 ] most affected pool [ 'cephfs_metadata' : 8 ]) 2022-05-13T23:32:13.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:32:13 smithi115 conmon[90393]: 2022-05-13T23:32:13.262+0000 7fa085a08700 -1 osd.5 149 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:32:14.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:32:13 smithi047 ceph-mon[77147]: 8 slow requests (by type [ 'queued for pg' : 8 ] most affected pool [ 'cephfs_metadata' : 8 ]) 2022-05-13T23:32:14.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:32:13 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:32:14.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:32:13 smithi047 ceph-mon[77147]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2022-05-13T23:32:14.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:32:13 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-13T23:32:14.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:32:13 smithi047 ceph-mon[77147]: Health check update: 8 slow ops, oldest one blocked for 265 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:32:14.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:32:13 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-13T23:32:14.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:32:13 smithi047 ceph-mon[77147]: pgmap v484: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:32:14.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:32:13 smithi115 ceph-mon[78593]: 8 slow requests (by type [ 'queued for pg' : 8 ] most affected pool [ 'cephfs_metadata' : 8 ]) 2022-05-13T23:32:14.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:32:13 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:32:14.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:32:13 smithi115 ceph-mon[78593]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2022-05-13T23:32:14.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:32:13 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-13T23:32:14.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:32:13 smithi115 ceph-mon[78593]: Health check update: 8 slow ops, oldest one blocked for 265 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:32:14.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:32:13 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-13T23:32:14.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:32:13 smithi115 ceph-mon[78593]: pgmap v484: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:32:14.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:32:14 smithi115 conmon[90393]: 2022-05-13T23:32:14.256+0000 7fa085a08700 -1 osd.5 149 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:32:15.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:32:14 smithi047 ceph-mon[77147]: 8 slow requests (by type [ 'queued for pg' : 8 ] most affected pool [ 'cephfs_metadata' : 8 ]) 2022-05-13T23:32:15.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:32:14 smithi115 ceph-mon[78593]: 8 slow requests (by type [ 'queued for pg' : 8 ] most affected pool [ 'cephfs_metadata' : 8 ]) 2022-05-13T23:32:15.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:32:15 smithi115 conmon[90393]: 2022-05-13T23:32:15.266+0000 7fa085a08700 -1 osd.5 149 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:32:16.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:32:15 smithi047 ceph-mon[77147]: 8 slow requests (by type [ 'queued for pg' : 8 ] most affected pool [ 'cephfs_metadata' : 8 ]) 2022-05-13T23:32:16.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:32:15 smithi047 ceph-mon[77147]: pgmap v485: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:32:16.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:32:15 smithi115 ceph-mon[78593]: 8 slow requests (by type [ 'queued for pg' : 8 ] most affected pool [ 'cephfs_metadata' : 8 ]) 2022-05-13T23:32:16.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:32:15 smithi115 ceph-mon[78593]: pgmap v485: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:32:16.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:32:16 smithi115 conmon[90393]: 2022-05-13T23:32:16.228+0000 7fa085a08700 -1 osd.5 149 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:32:17.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:32:16 smithi047 ceph-mon[77147]: 8 slow requests (by type [ 'queued for pg' : 8 ] most affected pool [ 'cephfs_metadata' : 8 ]) 2022-05-13T23:32:17.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:32:16 smithi115 ceph-mon[78593]: 8 slow requests (by type [ 'queued for pg' : 8 ] most affected pool [ 'cephfs_metadata' : 8 ]) 2022-05-13T23:32:17.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:32:17 smithi115 conmon[90393]: 2022-05-13T23:32:17.182+0000 7fa085a08700 -1 osd.5 149 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:32:18.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:32:17 smithi047 ceph-mon[77147]: 8 slow requests (by type [ 'queued for pg' : 8 ] most affected pool [ 'cephfs_metadata' : 8 ]) 2022-05-13T23:32:18.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:32:17 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:32:18.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:32:17 smithi047 ceph-mon[77147]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2022-05-13T23:32:18.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:32:17 smithi047 ceph-mon[77147]: pgmap v486: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:32:18.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:32:17 smithi115 ceph-mon[78593]: 8 slow requests (by type [ 'queued for pg' : 8 ] most affected pool [ 'cephfs_metadata' : 8 ]) 2022-05-13T23:32:18.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:32:17 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:32:18.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:32:17 smithi115 ceph-mon[78593]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2022-05-13T23:32:18.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:32:17 smithi115 ceph-mon[78593]: pgmap v486: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:32:18.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:32:18 smithi115 conmon[90393]: 2022-05-13T23:32:18.180+0000 7fa085a08700 -1 osd.5 149 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:32:19.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:32:18 smithi047 ceph-mon[77147]: 8 slow requests (by type [ 'queued for pg' : 8 ] most affected pool [ 'cephfs_metadata' : 8 ]) 2022-05-13T23:32:19.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:32:18 smithi047 ceph-mon[77147]: Health check update: 8 slow ops, oldest one blocked for 270 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:32:19.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:32:18 smithi115 ceph-mon[78593]: 8 slow requests (by type [ 'queued for pg' : 8 ] most affected pool [ 'cephfs_metadata' : 8 ]) 2022-05-13T23:32:19.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:32:18 smithi115 ceph-mon[78593]: Health check update: 8 slow ops, oldest one blocked for 270 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:32:19.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:32:19 smithi115 conmon[90393]: 2022-05-13T23:32:19.230+0000 7fa085a08700 -1 osd.5 149 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:32:20.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:32:19 smithi047 ceph-mon[77147]: 8 slow requests (by type [ 'queued for pg' : 8 ] most affected pool [ 'cephfs_metadata' : 8 ]) 2022-05-13T23:32:20.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:32:19 smithi047 ceph-mon[77147]: pgmap v487: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:32:20.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:32:19 smithi115 ceph-mon[78593]: 8 slow requests (by type [ 'queued for pg' : 8 ] most affected pool [ 'cephfs_metadata' : 8 ]) 2022-05-13T23:32:20.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:32:19 smithi115 ceph-mon[78593]: pgmap v487: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:32:20.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:32:20 smithi115 conmon[90393]: 2022-05-13T23:32:20.273+0000 7fa085a08700 -1 osd.5 149 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:32:21.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:32:20 smithi047 ceph-mon[77147]: 8 slow requests (by type [ 'queued for pg' : 8 ] most affected pool [ 'cephfs_metadata' : 8 ]) 2022-05-13T23:32:21.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:32:20 smithi115 ceph-mon[78593]: 8 slow requests (by type [ 'queued for pg' : 8 ] most affected pool [ 'cephfs_metadata' : 8 ]) 2022-05-13T23:32:21.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:32:21 smithi115 conmon[90393]: 2022-05-13T23:32:21.225+0000 7fa085a08700 -1 osd.5 149 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:32:22.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:32:21 smithi047 ceph-mon[77147]: 8 slow requests (by type [ 'queued for pg' : 8 ] most affected pool [ 'cephfs_metadata' : 8 ]) 2022-05-13T23:32:22.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:32:21 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:32:22.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:32:21 smithi047 ceph-mon[77147]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2022-05-13T23:32:22.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:32:21 smithi047 ceph-mon[77147]: pgmap v488: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:32:22.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:32:21 smithi115 ceph-mon[78593]: 8 slow requests (by type [ 'queued for pg' : 8 ] most affected pool [ 'cephfs_metadata' : 8 ]) 2022-05-13T23:32:22.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:32:21 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:32:22.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:32:21 smithi115 ceph-mon[78593]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2022-05-13T23:32:22.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:32:21 smithi115 ceph-mon[78593]: pgmap v488: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:32:22.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:32:22 smithi115 conmon[90393]: 2022-05-13T23:32:22.177+0000 7fa085a08700 -1 osd.5 149 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:32:23.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:32:22 smithi047 ceph-mon[77147]: 8 slow requests (by type [ 'queued for pg' : 8 ] most affected pool [ 'cephfs_metadata' : 8 ]) 2022-05-13T23:32:23.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:32:22 smithi115 ceph-mon[78593]: 8 slow requests (by type [ 'queued for pg' : 8 ] most affected pool [ 'cephfs_metadata' : 8 ]) 2022-05-13T23:32:23.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:32:23 smithi115 conmon[90393]: 2022-05-13T23:32:23.156+0000 7fa085a08700 -1 osd.5 149 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:32:24.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:32:23 smithi047 ceph-mon[77147]: 8 slow requests (by type [ 'queued for pg' : 8 ] most affected pool [ 'cephfs_metadata' : 8 ]) 2022-05-13T23:32:24.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:32:23 smithi047 ceph-mon[77147]: Health check update: 8 slow ops, oldest one blocked for 275 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:32:24.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:32:23 smithi047 ceph-mon[77147]: pgmap v489: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:32:24.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:32:23 smithi115 ceph-mon[78593]: 8 slow requests (by type [ 'queued for pg' : 8 ] most affected pool [ 'cephfs_metadata' : 8 ]) 2022-05-13T23:32:24.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:32:23 smithi115 ceph-mon[78593]: Health check update: 8 slow ops, oldest one blocked for 275 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:32:24.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:32:23 smithi115 ceph-mon[78593]: pgmap v489: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:32:24.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:32:24 smithi115 conmon[90393]: 2022-05-13T23:32:24.134+0000 7fa085a08700 -1 osd.5 149 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:32:25.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:32:24 smithi047 ceph-mon[77147]: 8 slow requests (by type [ 'queued for pg' : 8 ] most affected pool [ 'cephfs_metadata' : 8 ]) 2022-05-13T23:32:25.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:32:24 smithi115 ceph-mon[78593]: 8 slow requests (by type [ 'queued for pg' : 8 ] most affected pool [ 'cephfs_metadata' : 8 ]) 2022-05-13T23:32:25.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:32:25 smithi115 conmon[90393]: 2022-05-13T23:32:25.175+0000 7fa085a08700 -1 osd.5 149 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:32:26.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:32:25 smithi047 ceph-mon[77147]: 8 slow requests (by type [ 'queued for pg' : 8 ] most affected pool [ 'cephfs_metadata' : 8 ]) 2022-05-13T23:32:26.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:32:25 smithi047 ceph-mon[77147]: pgmap v490: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:32:26.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:32:25 smithi115 ceph-mon[78593]: 8 slow requests (by type [ 'queued for pg' : 8 ] most affected pool [ 'cephfs_metadata' : 8 ]) 2022-05-13T23:32:26.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:32:25 smithi115 ceph-mon[78593]: pgmap v490: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:32:26.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:32:26 smithi115 conmon[90393]: 2022-05-13T23:32:26.171+0000 7fa085a08700 -1 osd.5 149 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:32:27.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:32:26 smithi047 ceph-mon[77147]: 8 slow requests (by type [ 'queued for pg' : 8 ] most affected pool [ 'cephfs_metadata' : 8 ]) 2022-05-13T23:32:27.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:32:26 smithi115 ceph-mon[78593]: 8 slow requests (by type [ 'queued for pg' : 8 ] most affected pool [ 'cephfs_metadata' : 8 ]) 2022-05-13T23:32:27.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:32:27 smithi115 conmon[90393]: 2022-05-13T23:32:27.144+0000 7fa085a08700 -1 osd.5 149 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:32:28.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:32:27 smithi047 ceph-mon[77147]: 8 slow requests (by type [ 'queued for pg' : 8 ] most affected pool [ 'cephfs_metadata' : 8 ]) 2022-05-13T23:32:28.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:32:27 smithi047 ceph-mon[77147]: pgmap v491: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:32:28.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:32:27 smithi115 ceph-mon[78593]: 8 slow requests (by type [ 'queued for pg' : 8 ] most affected pool [ 'cephfs_metadata' : 8 ]) 2022-05-13T23:32:28.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:32:27 smithi115 ceph-mon[78593]: pgmap v491: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:32:28.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:32:28 smithi115 conmon[90393]: 2022-05-13T23:32:28.108+0000 7fa085a08700 -1 osd.5 149 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:32:29.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:32:28 smithi047 ceph-mon[77147]: 8 slow requests (by type [ 'queued for pg' : 8 ] most affected pool [ 'cephfs_metadata' : 8 ]) 2022-05-13T23:32:29.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:32:28 smithi047 ceph-mon[77147]: Health check update: 8 slow ops, oldest one blocked for 280 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:32:29.081 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:32:28 smithi115 ceph-mon[78593]: 8 slow requests (by type [ 'queued for pg' : 8 ] most affected pool [ 'cephfs_metadata' : 8 ]) 2022-05-13T23:32:29.082 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:32:28 smithi115 ceph-mon[78593]: Health check update: 8 slow ops, oldest one blocked for 280 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:32:29.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:32:29 smithi115 conmon[90393]: 2022-05-13T23:32:29.080+0000 7fa085a08700 -1 osd.5 149 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:32:30.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:32:29 smithi047 ceph-mon[77147]: 8 slow requests (by type [ 'queued for pg' : 8 ] most affected pool [ 'cephfs_metadata' : 8 ]) 2022-05-13T23:32:30.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:32:29 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:32:30.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:32:29 smithi047 ceph-mon[77147]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2022-05-13T23:32:30.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:32:29 smithi047 ceph-mon[77147]: pgmap v492: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:32:30.099 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:32:29 smithi115 ceph-mon[78593]: 8 slow requests (by type [ 'queued for pg' : 8 ] most affected pool [ 'cephfs_metadata' : 8 ]) 2022-05-13T23:32:30.099 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:32:29 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:32:30.099 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:32:29 smithi115 ceph-mon[78593]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2022-05-13T23:32:30.100 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:32:29 smithi115 ceph-mon[78593]: pgmap v492: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:32:30.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:32:30 smithi115 conmon[90393]: 2022-05-13T23:32:30.097+0000 7fa085a08700 -1 osd.5 149 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:32:31.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:32:30 smithi047 ceph-mon[77147]: 8 slow requests (by type [ 'queued for pg' : 8 ] most affected pool [ 'cephfs_metadata' : 8 ]) 2022-05-13T23:32:31.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:32:30 smithi115 ceph-mon[78593]: 8 slow requests (by type [ 'queued for pg' : 8 ] most affected pool [ 'cephfs_metadata' : 8 ]) 2022-05-13T23:32:31.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:32:31 smithi115 conmon[90393]: 2022-05-13T23:32:31.117+0000 7fa085a08700 -1 osd.5 149 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:32:32.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:32:31 smithi047 ceph-mon[77147]: 8 slow requests (by type [ 'queued for pg' : 8 ] most affected pool [ 'cephfs_metadata' : 8 ]) 2022-05-13T23:32:32.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:32:31 smithi047 ceph-mon[77147]: pgmap v493: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:32:32.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:32:31 smithi115 ceph-mon[78593]: 8 slow requests (by type [ 'queued for pg' : 8 ] most affected pool [ 'cephfs_metadata' : 8 ]) 2022-05-13T23:32:32.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:32:31 smithi115 ceph-mon[78593]: pgmap v493: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:32:32.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:32:32 smithi115 conmon[90393]: 2022-05-13T23:32:32.144+0000 7fa085a08700 -1 osd.5 149 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:32:33.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:32:32 smithi047 ceph-mon[77147]: 8 slow requests (by type [ 'queued for pg' : 8 ] most affected pool [ 'cephfs_metadata' : 8 ]) 2022-05-13T23:32:33.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:32:32 smithi115 ceph-mon[78593]: 8 slow requests (by type [ 'queued for pg' : 8 ] most affected pool [ 'cephfs_metadata' : 8 ]) 2022-05-13T23:32:33.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:32:33 smithi115 conmon[90393]: 2022-05-13T23:32:33.140+0000 7fa085a08700 -1 osd.5 149 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:32:34.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:32:33 smithi047 ceph-mon[77147]: 8 slow requests (by type [ 'queued for pg' : 8 ] most affected pool [ 'cephfs_metadata' : 8 ]) 2022-05-13T23:32:34.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:32:33 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:32:34.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:32:33 smithi047 ceph-mon[77147]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2022-05-13T23:32:34.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:32:33 smithi047 ceph-mon[77147]: Health check update: 8 slow ops, oldest one blocked for 285 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:32:34.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:32:33 smithi047 ceph-mon[77147]: pgmap v494: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:32:34.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:32:33 smithi115 ceph-mon[78593]: 8 slow requests (by type [ 'queued for pg' : 8 ] most affected pool [ 'cephfs_metadata' : 8 ]) 2022-05-13T23:32:34.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:32:33 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:32:34.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:32:33 smithi115 ceph-mon[78593]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2022-05-13T23:32:34.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:32:33 smithi115 ceph-mon[78593]: Health check update: 8 slow ops, oldest one blocked for 285 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:32:34.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:32:33 smithi115 ceph-mon[78593]: pgmap v494: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:32:34.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:32:34 smithi115 conmon[90393]: 2022-05-13T23:32:34.177+0000 7fa085a08700 -1 osd.5 149 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:32:35.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:32:34 smithi047 ceph-mon[77147]: 8 slow requests (by type [ 'queued for pg' : 8 ] most affected pool [ 'cephfs_metadata' : 8 ]) 2022-05-13T23:32:35.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:32:34 smithi115 ceph-mon[78593]: 8 slow requests (by type [ 'queued for pg' : 8 ] most affected pool [ 'cephfs_metadata' : 8 ]) 2022-05-13T23:32:35.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:32:35 smithi115 conmon[90393]: 2022-05-13T23:32:35.173+0000 7fa085a08700 -1 osd.5 149 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:32:36.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:32:35 smithi047 ceph-mon[77147]: 8 slow requests (by type [ 'queued for pg' : 8 ] most affected pool [ 'cephfs_metadata' : 8 ]) 2022-05-13T23:32:36.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:32:35 smithi047 ceph-mon[77147]: pgmap v495: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:32:36.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:32:35 smithi115 ceph-mon[78593]: 8 slow requests (by type [ 'queued for pg' : 8 ] most affected pool [ 'cephfs_metadata' : 8 ]) 2022-05-13T23:32:36.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:32:35 smithi115 ceph-mon[78593]: pgmap v495: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:32:36.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:32:36 smithi115 conmon[90393]: 2022-05-13T23:32:36.177+0000 7fa085a08700 -1 osd.5 149 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:32:37.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:32:36 smithi047 ceph-mon[77147]: 8 slow requests (by type [ 'queued for pg' : 8 ] most affected pool [ 'cephfs_metadata' : 8 ]) 2022-05-13T23:32:37.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:32:36 smithi115 ceph-mon[78593]: 8 slow requests (by type [ 'queued for pg' : 8 ] most affected pool [ 'cephfs_metadata' : 8 ]) 2022-05-13T23:32:37.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:32:37 smithi115 conmon[90393]: 2022-05-13T23:32:37.159+0000 7fa085a08700 -1 osd.5 149 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:32:37.740 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:32:37 smithi115 ceph-mon[78593]: 8 slow requests (by type [ 'queued for pg' : 8 ] most affected pool [ 'cephfs_metadata' : 8 ]) 2022-05-13T23:32:38.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:32:37 smithi047 ceph-mon[77147]: 8 slow requests (by type [ 'queued for pg' : 8 ] most affected pool [ 'cephfs_metadata' : 8 ]) 2022-05-13T23:32:38.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:32:37 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:32:38.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:32:37 smithi047 ceph-mon[77147]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2022-05-13T23:32:38.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:32:37 smithi047 ceph-mon[77147]: pgmap v496: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:32:38.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:32:37 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:32:38.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:32:37 smithi115 ceph-mon[78593]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2022-05-13T23:32:38.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:32:37 smithi115 ceph-mon[78593]: pgmap v496: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:32:38.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:32:38 smithi115 conmon[90393]: 2022-05-13T23:32:38.121+0000 7fa085a08700 -1 osd.5 149 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:32:39.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:32:38 smithi047 ceph-mon[77147]: 8 slow requests (by type [ 'queued for pg' : 8 ] most affected pool [ 'cephfs_metadata' : 8 ]) 2022-05-13T23:32:39.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:32:38 smithi047 ceph-mon[77147]: Health check update: 8 slow ops, oldest one blocked for 290 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:32:39.097 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:32:38 smithi115 ceph-mon[78593]: 8 slow requests (by type [ 'queued for pg' : 8 ] most affected pool [ 'cephfs_metadata' : 8 ]) 2022-05-13T23:32:39.098 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:32:38 smithi115 ceph-mon[78593]: Health check update: 8 slow ops, oldest one blocked for 290 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:32:39.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:32:39 smithi115 conmon[90393]: 2022-05-13T23:32:39.096+0000 7fa085a08700 -1 osd.5 149 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:32:40.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:32:39 smithi047 ceph-mon[77147]: 8 slow requests (by type [ 'queued for pg' : 8 ] most affected pool [ 'cephfs_metadata' : 8 ]) 2022-05-13T23:32:40.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:32:39 smithi047 ceph-mon[77147]: pgmap v497: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:32:40.106 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:32:39 smithi115 ceph-mon[78593]: 8 slow requests (by type [ 'queued for pg' : 8 ] most affected pool [ 'cephfs_metadata' : 8 ]) 2022-05-13T23:32:40.106 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:32:39 smithi115 ceph-mon[78593]: pgmap v497: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:32:40.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:32:40 smithi115 conmon[90393]: 2022-05-13T23:32:40.105+0000 7fa085a08700 -1 osd.5 149 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:32:40.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:32:40 smithi047 ceph-mon[77147]: 8 slow requests (by type [ 'queued for pg' : 8 ] most affected pool [ 'cephfs_metadata' : 8 ]) 2022-05-13T23:32:40.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:32:40 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-13T23:32:40.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:32:40 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-13T23:32:40.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:32:40 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-13T23:32:40.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:32:40 smithi115 ceph-mon[78593]: 8 slow requests (by type [ 'queued for pg' : 8 ] most affected pool [ 'cephfs_metadata' : 8 ]) 2022-05-13T23:32:40.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:32:40 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-13T23:32:40.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:32:40 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-13T23:32:40.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:32:40 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-13T23:32:41.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:32:41 smithi115 conmon[90393]: 2022-05-13T23:32:41.087+0000 7fa085a08700 -1 osd.5 149 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:32:42.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:32:41 smithi047 ceph-mon[77147]: 8 slow requests (by type [ 'queued for pg' : 8 ] most affected pool [ 'cephfs_metadata' : 8 ]) 2022-05-13T23:32:42.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:32:41 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:32:42.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:32:41 smithi047 ceph-mon[77147]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2022-05-13T23:32:42.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:32:41 smithi047 ceph-mon[77147]: pgmap v498: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:32:42.055 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:32:41 smithi115 ceph-mon[78593]: 8 slow requests (by type [ 'queued for pg' : 8 ] most affected pool [ 'cephfs_metadata' : 8 ]) 2022-05-13T23:32:42.055 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:32:41 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:32:42.056 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:32:41 smithi115 ceph-mon[78593]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2022-05-13T23:32:42.056 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:32:41 smithi115 ceph-mon[78593]: pgmap v498: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:32:42.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:32:42 smithi115 conmon[90393]: 2022-05-13T23:32:42.054+0000 7fa085a08700 -1 osd.5 149 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:32:43.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:32:42 smithi047 ceph-mon[77147]: 8 slow requests (by type [ 'queued for pg' : 8 ] most affected pool [ 'cephfs_metadata' : 8 ]) 2022-05-13T23:32:43.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:32:42 smithi047 ceph-mon[77147]: 8 slow requests (by type [ 'queued for pg' : 8 ] most affected pool [ 'cephfs_metadata' : 8 ]) 2022-05-13T23:32:43.092 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:32:42 smithi115 ceph-mon[78593]: 8 slow requests (by type [ 'queued for pg' : 8 ] most affected pool [ 'cephfs_metadata' : 8 ]) 2022-05-13T23:32:43.093 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:32:42 smithi115 ceph-mon[78593]: 8 slow requests (by type [ 'queued for pg' : 8 ] most affected pool [ 'cephfs_metadata' : 8 ]) 2022-05-13T23:32:43.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:32:43 smithi115 conmon[90393]: 2022-05-13T23:32:43.091+0000 7fa085a08700 -1 osd.5 149 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:32:44.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:32:43 smithi047 ceph-mon[77147]: Health check update: 8 slow ops, oldest one blocked for 296 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:32:44.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:32:43 smithi047 ceph-mon[77147]: pgmap v499: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:32:44.042 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:32:43 smithi047 conmon[78521]: 2022-05-13T23:32:43.567+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:32:44.042 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:32:43 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-13T23:32:44.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:32:43 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-13T23:32:44.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:32:43 smithi047 conmon[78521]: yield 2022-05-13T23:32:44.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:32:43 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-13T23:32:44.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:32:43 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-13T23:32:44.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:32:43 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-13T23:32:44.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:32:43 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-13T23:32:44.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:32:43 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-13T23:32:44.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:32:43 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-13T23:32:44.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:32:43 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-13T23:32:44.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:32:43 smithi047 conmon[78521]: raise exceptions[0] 2022-05-13T23:32:44.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:32:43 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-13T23:32:44.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:32:43 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-13T23:32:44.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:32:43 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-13T23:32:44.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:32:43 smithi047 conmon[78521]: return (yield from fut) 2022-05-13T23:32:44.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:32:43 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-13T23:32:44.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:32:43 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-13T23:32:44.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:32:43 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:32:44.092 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:32:43 smithi115 ceph-mon[78593]: Health check update: 8 slow ops, oldest one blocked for 296 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:32:44.092 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:32:43 smithi115 ceph-mon[78593]: pgmap v499: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:32:44.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:32:44 smithi115 conmon[90393]: 2022-05-13T23:32:44.091+0000 7fa085a08700 -1 osd.5 149 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:32:45.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:32:44 smithi047 ceph-mon[77147]: 8 slow requests (by type [ 'queued for pg' : 8 ] most affected pool [ 'cephfs_metadata' : 8 ]) 2022-05-13T23:32:45.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:32:44 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:32:45.042 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-13T23:32:45.042 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-13T23:32:45.042 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-13T23:32:45.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-13T23:32:45.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-13T23:32:45.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-13T23:32:45.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-13T23:32:45.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-13T23:32:45.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-13T23:32:45.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-13T23:32:45.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-13T23:32:45.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-13T23:32:45.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-13T23:32:45.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-13T23:32:45.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-13T23:32:45.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-13T23:32:45.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-13T23:32:45.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:32:45.046 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:32:44 smithi047 ceph-mon[77147]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-13T23:32:45.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:32:44 smithi115 ceph-mon[78593]: 8 slow requests (by type [ 'queued for pg' : 8 ] most affected pool [ 'cephfs_metadata' : 8 ]) 2022-05-13T23:32:45.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:32:44 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:32:45.109 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-13T23:32:45.109 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-13T23:32:45.109 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-13T23:32:45.109 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-13T23:32:45.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-13T23:32:45.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-13T23:32:45.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-13T23:32:45.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-13T23:32:45.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-13T23:32:45.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-13T23:32:45.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-13T23:32:45.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-13T23:32:45.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-13T23:32:45.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-13T23:32:45.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-13T23:32:45.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-13T23:32:45.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-13T23:32:45.113 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:32:45.113 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:32:44 smithi115 ceph-mon[78593]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-13T23:32:45.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:32:45 smithi115 conmon[90393]: 2022-05-13T23:32:45.122+0000 7fa085a08700 -1 osd.5 149 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:32:46.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:32:45 smithi047 ceph-mon[77147]: 8 slow requests (by type [ 'queued for pg' : 8 ] most affected pool [ 'cephfs_metadata' : 8 ]) 2022-05-13T23:32:46.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:32:45 smithi047 ceph-mon[77147]: pgmap v500: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:32:46.090 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:32:45 smithi115 ceph-mon[78593]: 8 slow requests (by type [ 'queued for pg' : 8 ] most affected pool [ 'cephfs_metadata' : 8 ]) 2022-05-13T23:32:46.090 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:32:45 smithi115 ceph-mon[78593]: pgmap v500: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:32:46.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:32:46 smithi115 conmon[90393]: 2022-05-13T23:32:46.089+0000 7fa085a08700 -1 osd.5 149 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:32:47.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:32:46 smithi047 ceph-mon[77147]: 8 slow requests (by type [ 'queued for pg' : 8 ] most affected pool [ 'cephfs_metadata' : 8 ]) 2022-05-13T23:32:47.042 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:32:46 smithi047 conmon[78521]: 2022-05-13T23:32:46.639+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:32:47.042 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:32:46 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-13T23:32:47.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:32:46 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-13T23:32:47.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:32:46 smithi047 conmon[78521]: yield 2022-05-13T23:32:47.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:32:46 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-13T23:32:47.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:32:46 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-13T23:32:47.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:32:46 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-13T23:32:47.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:32:46 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-13T23:32:47.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:32:46 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-13T23:32:47.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:32:46 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-13T23:32:47.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:32:46 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-13T23:32:47.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:32:46 smithi047 conmon[78521]: raise exceptions[0] 2022-05-13T23:32:47.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:32:46 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-13T23:32:47.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:32:46 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-13T23:32:47.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:32:46 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-13T23:32:47.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:32:46 smithi047 conmon[78521]: return (yield from fut) 2022-05-13T23:32:47.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:32:46 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-13T23:32:47.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:32:46 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-13T23:32:47.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:32:46 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:32:47.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:32:46 smithi047 conmon[78521]: 2022-05-13T23:32:46.640+0000 7f5340779700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-13T23:32:47.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:32:46 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-13T23:32:47.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:32:46 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-13T23:32:47.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:32:46 smithi047 conmon[78521]: yield 2022-05-13T23:32:47.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:32:46 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-13T23:32:47.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:32:46 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-13T23:32:47.048 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:32:46 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-13T23:32:47.048 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:32:46 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-13T23:32:47.048 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:32:46 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-13T23:32:47.048 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:32:46 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-13T23:32:47.048 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:32:46 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-13T23:32:47.049 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:32:46 smithi047 conmon[78521]: raise exceptions[0] 2022-05-13T23:32:47.049 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:32:46 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-13T23:32:47.049 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:32:46 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-13T23:32:47.049 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:32:46 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-13T23:32:47.050 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:32:46 smithi047 conmon[78521]: return (yield from fut) 2022-05-13T23:32:47.050 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:32:46 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-13T23:32:47.050 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:32:46 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-13T23:32:47.050 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:32:46 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:32:47.050 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:32:46 smithi047 conmon[78521]: 2022-05-13T23:32:47.051 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:32:46 smithi047 conmon[78521]: During handling of the above exception, another exception occurred: 2022-05-13T23:32:47.051 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:32:46 smithi047 conmon[78521]: 2022-05-13T23:32:47.051 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:32:46 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-13T23:32:47.051 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:32:46 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-13T23:32:47.052 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:32:46 smithi047 conmon[78521]: return f(*arg) 2022-05-13T23:32:47.052 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:32:46 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-13T23:32:47.052 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:32:46 smithi047 conmon[78521]: self._write_client_files(client_files, host) 2022-05-13T23:32:47.052 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:32:46 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-13T23:32:47.052 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:32:46 smithi047 conmon[78521]: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-13T23:32:47.053 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:32:46 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-13T23:32:47.053 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:32:46 smithi047 conmon[78521]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-13T23:32:47.053 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:32:46 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-13T23:32:47.053 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:32:46 smithi047 conmon[78521]: return self.event_loop.get_result(coro) 2022-05-13T23:32:47.054 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:32:46 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-13T23:32:47.054 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:32:46 smithi047 conmon[78521]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-13T23:32:47.054 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:32:46 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-13T23:32:47.054 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:32:46 smithi047 conmon[78521]: return self.__get_result() 2022-05-13T23:32:47.054 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:32:46 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-13T23:32:47.055 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:32:46 smithi047 conmon[78521]: raise self._exception 2022-05-13T23:32:47.055 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:32:46 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-13T23:32:47.055 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:32:46 smithi047 conmon[78521]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-13T23:32:47.055 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:32:46 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-13T23:32:47.056 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:32:46 smithi047 conmon[78521]: conn = await self._remote_connection(host, addr) 2022-05-13T23:32:47.056 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:32:46 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-13T23:32:47.056 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:32:46 smithi047 conmon[78521]: raise 2022-05-13T23:32:47.056 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:32:46 smithi047 conmon[78521]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-13T23:32:47.056 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:32:46 smithi047 conmon[78521]: self.gen.throw(type, value, traceback) 2022-05-13T23:32:47.057 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:32:46 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-13T23:32:47.057 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:32:46 smithi047 conmon[78521]: raise OrchestratorError(msg) 2022-05-13T23:32:47.057 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:32:46 smithi047 conmon[78521]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:32:47.077 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:32:46 smithi115 ceph-mon[78593]: 8 slow requests (by type [ 'queued for pg' : 8 ] most affected pool [ 'cephfs_metadata' : 8 ]) 2022-05-13T23:32:47.362 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:32:47 smithi115 conmon[90393]: 2022-05-13T23:32:47.075+0000 7fa085a08700 -1 osd.5 149 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:32:48.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:32:47 smithi047 ceph-mon[77147]: 8 slow requests (by type [ 'queued for pg' : 8 ] most affected pool [ 'cephfs_metadata' : 8 ]) 2022-05-13T23:32:48.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:32:47 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:32:48.042 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-13T23:32:48.042 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-13T23:32:48.042 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-13T23:32:48.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-13T23:32:48.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-13T23:32:48.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-13T23:32:48.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-13T23:32:48.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-13T23:32:48.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-13T23:32:48.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-13T23:32:48.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-13T23:32:48.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-13T23:32:48.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-13T23:32:48.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-13T23:32:48.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-13T23:32:48.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-13T23:32:48.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-13T23:32:48.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:32:48.046 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:32:47 smithi047 ceph-mon[77147]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-13T23:32:48.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-13T23:32:48.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-13T23:32:48.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-13T23:32:48.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-13T23:32:48.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-13T23:32:48.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-13T23:32:48.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-13T23:32:48.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-13T23:32:48.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-13T23:32:48.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-13T23:32:48.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-13T23:32:48.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-13T23:32:48.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-13T23:32:48.049 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-13T23:32:48.049 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-13T23:32:48.049 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-13T23:32:48.049 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-13T23:32:48.049 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:32:48.050 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-13T23:32:48.050 INFO:journalctl@ceph.mon.b.smithi047.stdout: During handling of the above exception, another exception occurred: 2022-05-13T23:32:48.050 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-13T23:32:48.050 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-13T23:32:48.050 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-13T23:32:48.051 INFO:journalctl@ceph.mon.b.smithi047.stdout: return f(*arg) 2022-05-13T23:32:48.051 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-13T23:32:48.051 INFO:journalctl@ceph.mon.b.smithi047.stdout: self._write_client_files(client_files, host) 2022-05-13T23:32:48.051 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-13T23:32:48.051 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-13T23:32:48.052 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-13T23:32:48.052 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-13T23:32:48.052 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-13T23:32:48.052 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.event_loop.get_result(coro) 2022-05-13T23:32:48.052 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-13T23:32:48.053 INFO:journalctl@ceph.mon.b.smithi047.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-13T23:32:48.053 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-13T23:32:48.053 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.__get_result() 2022-05-13T23:32:48.053 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-13T23:32:48.054 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise self._exception 2022-05-13T23:32:48.054 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-13T23:32:48.054 INFO:journalctl@ceph.mon.b.smithi047.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-13T23:32:48.054 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-13T23:32:48.054 INFO:journalctl@ceph.mon.b.smithi047.stdout: conn = await self._remote_connection(host, addr) 2022-05-13T23:32:48.055 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-13T23:32:48.055 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise 2022-05-13T23:32:48.055 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-13T23:32:48.055 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.gen.throw(type, value, traceback) 2022-05-13T23:32:48.055 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-13T23:32:48.056 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OrchestratorError(msg) 2022-05-13T23:32:48.056 INFO:journalctl@ceph.mon.b.smithi047.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:32:48.056 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:32:47 smithi047 ceph-mon[77147]: pgmap v501: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:32:48.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:32:47 smithi115 ceph-mon[78593]: 8 slow requests (by type [ 'queued for pg' : 8 ] most affected pool [ 'cephfs_metadata' : 8 ]) 2022-05-13T23:32:48.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:32:47 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:32:48.109 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-13T23:32:48.109 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-13T23:32:48.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-13T23:32:48.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-13T23:32:48.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-13T23:32:48.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-13T23:32:48.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-13T23:32:48.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-13T23:32:48.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-13T23:32:48.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-13T23:32:48.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-13T23:32:48.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-13T23:32:48.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-13T23:32:48.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-13T23:32:48.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-13T23:32:48.113 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-13T23:32:48.113 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-13T23:32:48.113 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:32:48.113 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:32:47 smithi115 ceph-mon[78593]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-13T23:32:48.113 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-13T23:32:48.114 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-13T23:32:48.114 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-13T23:32:48.114 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-13T23:32:48.114 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-13T23:32:48.115 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-13T23:32:48.115 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-13T23:32:48.115 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-13T23:32:48.115 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-13T23:32:48.115 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-13T23:32:48.116 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-13T23:32:48.116 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-13T23:32:48.116 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-13T23:32:48.116 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-13T23:32:48.116 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-13T23:32:48.117 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-13T23:32:48.117 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-13T23:32:48.117 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:32:48.117 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-13T23:32:48.118 INFO:journalctl@ceph.mon.c.smithi115.stdout: During handling of the above exception, another exception occurred: 2022-05-13T23:32:48.118 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-13T23:32:48.118 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-13T23:32:48.118 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-13T23:32:48.119 INFO:journalctl@ceph.mon.c.smithi115.stdout: return f(*arg) 2022-05-13T23:32:48.119 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-13T23:32:48.119 INFO:journalctl@ceph.mon.c.smithi115.stdout: self._write_client_files(client_files, host) 2022-05-13T23:32:48.119 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-13T23:32:48.119 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-13T23:32:48.120 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-13T23:32:48.120 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-13T23:32:48.120 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-13T23:32:48.120 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.event_loop.get_result(coro) 2022-05-13T23:32:48.120 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-13T23:32:48.121 INFO:journalctl@ceph.mon.c.smithi115.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-13T23:32:48.121 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-13T23:32:48.121 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.__get_result() 2022-05-13T23:32:48.121 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-13T23:32:48.122 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise self._exception 2022-05-13T23:32:48.122 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-13T23:32:48.122 INFO:journalctl@ceph.mon.c.smithi115.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-13T23:32:48.122 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-13T23:32:48.122 INFO:journalctl@ceph.mon.c.smithi115.stdout: conn = await self._remote_connection(host, addr) 2022-05-13T23:32:48.123 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-13T23:32:48.123 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise 2022-05-13T23:32:48.123 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-13T23:32:48.123 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.gen.throw(type, value, traceback) 2022-05-13T23:32:48.124 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-13T23:32:48.124 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OrchestratorError(msg) 2022-05-13T23:32:48.124 INFO:journalctl@ceph.mon.c.smithi115.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:32:48.124 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:32:47 smithi115 ceph-mon[78593]: pgmap v501: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:32:48.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:32:48 smithi115 conmon[90393]: 2022-05-13T23:32:48.111+0000 7fa085a08700 -1 osd.5 149 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:32:49.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:32:48 smithi047 ceph-mon[77147]: 8 slow requests (by type [ 'queued for pg' : 8 ] most affected pool [ 'cephfs_metadata' : 8 ]) 2022-05-13T23:32:49.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:32:48 smithi047 ceph-mon[77147]: Health check update: 8 slow ops, oldest one blocked for 301 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:32:49.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:32:48 smithi115 ceph-mon[78593]: 8 slow requests (by type [ 'queued for pg' : 8 ] most affected pool [ 'cephfs_metadata' : 8 ]) 2022-05-13T23:32:49.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:32:48 smithi115 ceph-mon[78593]: Health check update: 8 slow ops, oldest one blocked for 301 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:32:49.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:32:49 smithi115 conmon[90393]: 2022-05-13T23:32:49.125+0000 7fa085a08700 -1 osd.5 149 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:32:50.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:32:49 smithi047 ceph-mon[77147]: 8 slow requests (by type [ 'queued for pg' : 8 ] most affected pool [ 'cephfs_metadata' : 8 ]) 2022-05-13T23:32:50.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:32:49 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:32:50.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:32:49 smithi047 ceph-mon[77147]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2022-05-13T23:32:50.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:32:49 smithi047 ceph-mon[77147]: pgmap v502: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:32:50.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:32:49 smithi115 ceph-mon[78593]: 8 slow requests (by type [ 'queued for pg' : 8 ] most affected pool [ 'cephfs_metadata' : 8 ]) 2022-05-13T23:32:50.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:32:49 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:32:50.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:32:49 smithi115 ceph-mon[78593]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2022-05-13T23:32:50.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:32:49 smithi115 ceph-mon[78593]: pgmap v502: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:32:50.537 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:32:50 smithi115 conmon[90393]: 2022-05-13T23:32:50.164+0000 7fa085a08700 -1 osd.5 149 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:32:50.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:32:50 smithi115 ceph-mon[78593]: 8 slow requests (by type [ 'queued for pg' : 8 ] most affected pool [ 'cephfs_metadata' : 8 ]) 2022-05-13T23:32:50.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:32:50 smithi115 ceph-mon[78593]: daemon mds.f restarted 2022-05-13T23:32:50.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:32:50 smithi115 ceph-mon[78593]: osdmap e150: 12 total, 8 up, 8 in 2022-05-13T23:32:50.859 INFO:journalctl@ceph.mds.f.smithi115.stdout:May 13 23:32:50 smithi115 conmon[105998]: 2022-05-13T23:32:50.536+0000 7f46c78ef700 -1 mds.2.journaler.mdlog(rw) _finish_write_head got (110) Connection timed out 2022-05-13T23:32:50.860 INFO:journalctl@ceph.mds.f.smithi115.stdout:May 13 23:32:50 smithi115 conmon[105998]: 2022-05-13T23:32:50.536+0000 7f46c78ef700 -1 mds.2.journaler.mdlog(rw) handle_write_error (110) Connection timed out 2022-05-13T23:32:50.860 INFO:journalctl@ceph.mds.f.smithi115.stdout:May 13 23:32:50 smithi115 conmon[105998]: 2022-05-13T23:32:50.536+0000 7f46c78ef700 -1 MDSIOContextBase: failed with -110, restarting... 2022-05-13T23:32:50.860 INFO:journalctl@ceph.mds.f.smithi115.stdout:May 13 23:32:50 smithi115 conmon[105998]: -4> 2022-05-13T23:32:50.536+0000 7f46c78ef700 -1 mds.2.journaler.mdlog(rw) _finish_write_head got (110) Connection timed out 2022-05-13T23:32:50.860 INFO:journalctl@ceph.mds.f.smithi115.stdout:May 13 23:32:50 smithi115 conmon[105998]: -3> 2022-05-13T23:32:50.536+0000 7f46c78ef700 -1 mds.2.journaler.mdlog(rw) handle_write_error (110) Connection timed out 2022-05-13T23:32:50.861 INFO:journalctl@ceph.mds.f.smithi115.stdout:May 13 23:32:50 smithi115 conmon[105998]: -1> 2022-05-13T23:32:50.536+0000 7f46c78ef700 -1 MDSIOContextBase: failed with -110, restarting... 2022-05-13T23:32:50.861 INFO:journalctl@ceph.mds.f.smithi115.stdout:May 13 23:32:50 smithi115 conmon[105998]: ignoring --setuser ceph since I am not root 2022-05-13T23:32:50.861 INFO:journalctl@ceph.mds.f.smithi115.stdout:May 13 23:32:50 smithi115 conmon[105998]: ignoring --setgroup ceph since I am not root 2022-05-13T23:32:50.862 INFO:journalctl@ceph.mds.f.smithi115.stdout:May 13 23:32:50 smithi115 ceph-mds[106022]: -- 172.21.15.115:0/3107887026 <== mon.2 v2:172.21.15.115:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x562cad2fc340 con 0x562cad2d5000 2022-05-13T23:32:50.862 INFO:journalctl@ceph.mds.f.smithi115.stdout:May 13 23:32:50 smithi115 conmon[105998]: starting mds.f at 2022-05-13T23:32:51.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:32:50 smithi047 ceph-mon[77147]: 8 slow requests (by type [ 'queued for pg' : 8 ] most affected pool [ 'cephfs_metadata' : 8 ]) 2022-05-13T23:32:51.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:32:50 smithi047 ceph-mon[77147]: daemon mds.f restarted 2022-05-13T23:32:51.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:32:50 smithi047 ceph-mon[77147]: osdmap e150: 12 total, 8 up, 8 in 2022-05-13T23:32:51.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:32:51 smithi115 conmon[90393]: 2022-05-13T23:32:51.118+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:32:52.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:32:51 smithi047 ceph-mon[77147]: 8 slow requests (by type [ 'queued for pg' : 8 ] most affected pool [ 'cephfs_metadata' : 8 ]) 2022-05-13T23:32:52.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:32:51 smithi047 ceph-mon[77147]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2022-05-13T23:32:52.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:32:51 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6834/3300271029,v1:172.21.15.115:6835/3300271029] up:boot 2022-05-13T23:32:52.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:32:51 smithi047 ceph-mon[77147]: Standby daemon mds.f assigned to filesystem cephfs as rank 2 2022-05-13T23:32:52.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:32:51 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 failed 2022-05-13T23:32:52.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:32:51 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2022-05-13T23:32:52.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:32:51 smithi047 ceph-mon[77147]: fsmap cephfs:5/5 3 up:standby-replay 1 up:replay 4 up:active 2022-05-13T23:32:52.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:32:51 smithi047 ceph-mon[77147]: pgmap v504: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-05-13T23:32:52.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:32:51 smithi115 ceph-mon[78593]: 8 slow requests (by type [ 'queued for pg' : 8 ] most affected pool [ 'cephfs_metadata' : 8 ]) 2022-05-13T23:32:52.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:32:51 smithi115 ceph-mon[78593]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2022-05-13T23:32:52.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:32:51 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6834/3300271029,v1:172.21.15.115:6835/3300271029] up:boot 2022-05-13T23:32:52.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:32:51 smithi115 ceph-mon[78593]: Standby daemon mds.f assigned to filesystem cephfs as rank 2 2022-05-13T23:32:52.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:32:51 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 failed 2022-05-13T23:32:52.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:32:51 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2022-05-13T23:32:52.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:32:51 smithi115 ceph-mon[78593]: fsmap cephfs:5/5 3 up:standby-replay 1 up:replay 4 up:active 2022-05-13T23:32:52.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:32:51 smithi115 ceph-mon[78593]: pgmap v504: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-05-13T23:32:53.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:32:52 smithi047 ceph-mon[77147]: 8 slow requests (by type [ 'queued for pg' : 8 ] most affected pool [ 'cephfs_metadata' : 8 ]) 2022-05-13T23:32:53.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:32:52 smithi115 ceph-mon[78593]: 8 slow requests (by type [ 'queued for pg' : 8 ] most affected pool [ 'cephfs_metadata' : 8 ]) 2022-05-13T23:32:54.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:32:53 smithi047 ceph-mon[77147]: Health check update: 8 slow ops, oldest one blocked for 305 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:32:54.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:32:53 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:32:54.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:32:53 smithi047 ceph-mon[77147]: fsmap cephfs:5/5 3 up:standby-replay 1 up:replay 4 up:active 2022-05-13T23:32:54.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:32:53 smithi047 ceph-mon[77147]: pgmap v505: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-05-13T23:32:54.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:32:53 smithi115 ceph-mon[78593]: Health check update: 8 slow ops, oldest one blocked for 305 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:32:54.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:32:53 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:32:54.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:32:53 smithi115 ceph-mon[78593]: fsmap cephfs:5/5 3 up:standby-replay 1 up:replay 4 up:active 2022-05-13T23:32:54.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:32:53 smithi115 ceph-mon[78593]: pgmap v505: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-05-13T23:32:56.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:32:56 smithi047 ceph-mon[77147]: pgmap v506: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2022-05-13T23:32:56.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:32:56 smithi115 ceph-mon[78593]: pgmap v506: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2022-05-13T23:32:57.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:32:57 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:32:57.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:32:57 smithi047 ceph-mon[77147]: fsmap cephfs:5/5 3 up:standby-replay 1 up:replay 4 up:active 2022-05-13T23:32:57.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:32:57 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:32:57.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:32:57 smithi115 ceph-mon[78593]: fsmap cephfs:5/5 3 up:standby-replay 1 up:replay 4 up:active 2022-05-13T23:32:58.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:32:58 smithi047 ceph-mon[77147]: pgmap v507: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2022-05-13T23:32:58.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:32:58 smithi047 ceph-mon[77147]: Health check update: 0 slow ops, oldest one blocked for 310 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:32:58.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:32:58 smithi115 ceph-mon[78593]: pgmap v507: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2022-05-13T23:32:58.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:32:58 smithi115 ceph-mon[78593]: Health check update: 0 slow ops, oldest one blocked for 310 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:33:00.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:33:00 smithi047 ceph-mon[77147]: pgmap v508: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2022-05-13T23:33:00.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:33:00 smithi115 ceph-mon[78593]: pgmap v508: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2022-05-13T23:33:01.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:33:01 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:33:01.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:33:01 smithi047 ceph-mon[77147]: fsmap cephfs:5/5 3 up:standby-replay 1 up:replay 4 up:active 2022-05-13T23:33:01.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:33:01 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:33:01.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:33:01 smithi115 ceph-mon[78593]: fsmap cephfs:5/5 3 up:standby-replay 1 up:replay 4 up:active 2022-05-13T23:33:02.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:33:02 smithi047 ceph-mon[77147]: pgmap v509: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.4 KiB/s rd, 6 op/s 2022-05-13T23:33:02.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:33:02 smithi115 ceph-mon[78593]: pgmap v509: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.4 KiB/s rd, 6 op/s 2022-05-13T23:33:03.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:33:03 smithi047 ceph-mon[77147]: Health check update: 0 slow ops, oldest one blocked for 315 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:33:03.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:33:03 smithi115 ceph-mon[78593]: Health check update: 0 slow ops, oldest one blocked for 315 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:33:04.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:33:04 smithi047 ceph-mon[77147]: pgmap v510: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.1 KiB/s rd, 5 op/s 2022-05-13T23:33:04.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:33:04 smithi115 ceph-mon[78593]: pgmap v510: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.1 KiB/s rd, 5 op/s 2022-05-13T23:33:06.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:33:06 smithi047 ceph-mon[77147]: pgmap v511: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 4.3 KiB/s rd, 8 op/s 2022-05-13T23:33:06.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:33:06 smithi115 ceph-mon[78593]: pgmap v511: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 4.3 KiB/s rd, 8 op/s 2022-05-13T23:33:08.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:33:08 smithi047 ceph-mon[77147]: pgmap v512: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:33:08.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:33:08 smithi047 ceph-mon[77147]: Health check update: 0 slow ops, oldest one blocked for 320 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:33:08.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:33:08 smithi115 ceph-mon[78593]: pgmap v512: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:33:08.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:33:08 smithi115 ceph-mon[78593]: Health check update: 0 slow ops, oldest one blocked for 320 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:33:10.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:33:10 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:33:10.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:33:10 smithi047 ceph-mon[77147]: fsmap cephfs:5/5 3 up:standby-replay 1 up:replay 4 up:active 2022-05-13T23:33:10.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:33:10 smithi047 ceph-mon[77147]: pgmap v513: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:33:10.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:33:10 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:33:10.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:33:10 smithi115 ceph-mon[78593]: fsmap cephfs:5/5 3 up:standby-replay 1 up:replay 4 up:active 2022-05-13T23:33:10.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:33:10 smithi115 ceph-mon[78593]: pgmap v513: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:33:12.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:33:12 smithi047 ceph-mon[77147]: pgmap v514: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:33:12.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:33:12 smithi115 ceph-mon[78593]: pgmap v514: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:33:13.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:33:13 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:33:13.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:33:13 smithi047 ceph-mon[77147]: fsmap cephfs:5/5 3 up:standby-replay 1 up:replay 4 up:active 2022-05-13T23:33:13.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:33:13 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-13T23:33:13.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:33:13 smithi047 ceph-mon[77147]: Health check update: 0 slow ops, oldest one blocked for 325 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:33:13.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:33:13 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-13T23:33:13.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:33:13 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:33:13.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:33:13 smithi115 ceph-mon[78593]: fsmap cephfs:5/5 3 up:standby-replay 1 up:replay 4 up:active 2022-05-13T23:33:13.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:33:13 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-13T23:33:13.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:33:13 smithi115 ceph-mon[78593]: Health check update: 0 slow ops, oldest one blocked for 325 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:33:13.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:33:13 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-13T23:33:14.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:33:14 smithi047 ceph-mon[77147]: pgmap v515: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:33:14.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:33:14 smithi115 ceph-mon[78593]: pgmap v515: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:33:16.724 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:33:16 smithi047 ceph-mon[77147]: pgmap v516: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:33:16.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:33:16 smithi115 ceph-mon[78593]: pgmap v516: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:33:17.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:33:17 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:33:17.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:33:17 smithi047 ceph-mon[77147]: fsmap cephfs:5/5 3 up:standby-replay 1 up:replay 4 up:active 2022-05-13T23:33:17.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:33:17 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:33:17.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:33:17 smithi115 ceph-mon[78593]: fsmap cephfs:5/5 3 up:standby-replay 1 up:replay 4 up:active 2022-05-13T23:33:18.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:33:18 smithi047 ceph-mon[77147]: pgmap v517: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:33:18.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:33:18 smithi047 ceph-mon[77147]: Health check update: 0 slow ops, oldest one blocked for 330 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:33:18.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:33:18 smithi115 ceph-mon[78593]: pgmap v517: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:33:18.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:33:18 smithi115 ceph-mon[78593]: Health check update: 0 slow ops, oldest one blocked for 330 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:33:20.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:33:20 smithi047 ceph-mon[77147]: pgmap v518: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:33:20.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:33:20 smithi115 ceph-mon[78593]: pgmap v518: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:33:21.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:33:21 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:33:21.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:33:21 smithi047 ceph-mon[77147]: fsmap cephfs:5/5 3 up:standby-replay 1 up:replay 4 up:active 2022-05-13T23:33:21.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:33:21 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:33:21.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:33:21 smithi115 ceph-mon[78593]: fsmap cephfs:5/5 3 up:standby-replay 1 up:replay 4 up:active 2022-05-13T23:33:22.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:33:22 smithi047 ceph-mon[77147]: pgmap v519: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:33:22.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:33:22 smithi115 ceph-mon[78593]: pgmap v519: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:33:23.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:33:23 smithi047 ceph-mon[77147]: Health check update: 0 slow ops, oldest one blocked for 335 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:33:23.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:33:23 smithi115 ceph-mon[78593]: Health check update: 0 slow ops, oldest one blocked for 335 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:33:24.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:33:24 smithi047 ceph-mon[77147]: pgmap v520: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:33:24.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:33:24 smithi115 ceph-mon[78593]: pgmap v520: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:33:26.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:33:26 smithi047 ceph-mon[77147]: pgmap v521: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:33:26.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:33:26 smithi115 ceph-mon[78593]: pgmap v521: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:33:28.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:33:28 smithi047 ceph-mon[77147]: pgmap v522: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:33:28.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:33:28 smithi047 ceph-mon[77147]: Health check update: 0 slow ops, oldest one blocked for 340 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:33:28.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:33:28 smithi115 ceph-mon[78593]: pgmap v522: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:33:28.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:33:28 smithi115 ceph-mon[78593]: Health check update: 0 slow ops, oldest one blocked for 340 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:33:30.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:33:30 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:33:30.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:33:30 smithi047 ceph-mon[77147]: fsmap cephfs:5/5 3 up:standby-replay 1 up:replay 4 up:active 2022-05-13T23:33:30.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:33:30 smithi047 ceph-mon[77147]: pgmap v523: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:33:30.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:33:30 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:33:30.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:33:30 smithi115 ceph-mon[78593]: fsmap cephfs:5/5 3 up:standby-replay 1 up:replay 4 up:active 2022-05-13T23:33:30.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:33:30 smithi115 ceph-mon[78593]: pgmap v523: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:33:32.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:33:32 smithi047 ceph-mon[77147]: pgmap v524: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:33:32.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:33:32 smithi115 ceph-mon[78593]: pgmap v524: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:33:33.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:33:33 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:33:33.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:33:33 smithi047 ceph-mon[77147]: fsmap cephfs:5/5 3 up:standby-replay 1 up:replay 4 up:active 2022-05-13T23:33:33.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:33:33 smithi047 ceph-mon[77147]: Health check update: 0 slow ops, oldest one blocked for 345 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:33:33.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:33:33 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:33:33.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:33:33 smithi115 ceph-mon[78593]: fsmap cephfs:5/5 3 up:standby-replay 1 up:replay 4 up:active 2022-05-13T23:33:33.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:33:33 smithi115 ceph-mon[78593]: Health check update: 0 slow ops, oldest one blocked for 345 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:33:34.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:33:34 smithi047 ceph-mon[77147]: pgmap v525: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:33:34.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:33:34 smithi115 ceph-mon[78593]: pgmap v525: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:33:36.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:33:36 smithi047 ceph-mon[77147]: pgmap v526: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:33:36.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:33:36 smithi115 ceph-mon[78593]: pgmap v526: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:33:37.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:33:37 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:33:37.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:33:37 smithi047 ceph-mon[77147]: fsmap cephfs:5/5 3 up:standby-replay 1 up:replay 4 up:active 2022-05-13T23:33:37.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:33:37 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:33:37.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:33:37 smithi115 ceph-mon[78593]: fsmap cephfs:5/5 3 up:standby-replay 1 up:replay 4 up:active 2022-05-13T23:33:38.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:33:38 smithi047 ceph-mon[77147]: pgmap v527: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:33:38.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:33:38 smithi047 ceph-mon[77147]: Health check update: 0 slow ops, oldest one blocked for 350 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:33:38.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:33:38 smithi115 ceph-mon[78593]: pgmap v527: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:33:38.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:33:38 smithi115 ceph-mon[78593]: Health check update: 0 slow ops, oldest one blocked for 350 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:33:40.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:33:40 smithi047 ceph-mon[77147]: pgmap v528: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:33:40.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:33:40 smithi115 ceph-mon[78593]: pgmap v528: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:33:41.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:33:41 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:33:41.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:33:41 smithi047 ceph-mon[77147]: fsmap cephfs:5/5 3 up:standby-replay 1 up:replay 4 up:active 2022-05-13T23:33:41.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:33:41 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:33:41.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:33:41 smithi115 ceph-mon[78593]: fsmap cephfs:5/5 3 up:standby-replay 1 up:replay 4 up:active 2022-05-13T23:33:42.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:33:42 smithi047 ceph-mon[77147]: pgmap v529: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:33:42.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:33:42 smithi115 ceph-mon[78593]: pgmap v529: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:33:43.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:33:43 smithi047 ceph-mon[77147]: Health check update: 0 slow ops, oldest one blocked for 355 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:33:43.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:33:43 smithi115 ceph-mon[78593]: Health check update: 0 slow ops, oldest one blocked for 355 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:33:44.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:33:44 smithi047 ceph-mon[77147]: pgmap v530: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:33:44.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:33:44 smithi115 ceph-mon[78593]: pgmap v530: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:33:46.690 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:33:46 smithi047 ceph-mon[77147]: pgmap v531: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:33:46.691 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:33:46 smithi115 ceph-mon[78593]: pgmap v531: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:33:47.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:33:47 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-13T23:33:47.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:33:47 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-13T23:33:47.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:33:47 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-13T23:33:47.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:33:47 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-13T23:33:47.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:33:47 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-13T23:33:47.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:33:47 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-13T23:33:48.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:33:48 smithi047 ceph-mon[77147]: pgmap v532: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:33:48.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:33:48 smithi047 ceph-mon[77147]: Health check update: 0 slow ops, oldest one blocked for 360 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:33:48.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:33:48 smithi115 ceph-mon[78593]: pgmap v532: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:33:48.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:33:48 smithi115 ceph-mon[78593]: Health check update: 0 slow ops, oldest one blocked for 360 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:33:50.041 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:33:49 smithi047 conmon[78521]: 2022-05-13T23:33:49.744+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:33:50.042 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:33:49 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-13T23:33:50.042 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:33:49 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-13T23:33:50.042 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:33:49 smithi047 conmon[78521]: yield 2022-05-13T23:33:50.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:33:49 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-13T23:33:50.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:33:49 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-13T23:33:50.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:33:49 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-13T23:33:50.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:33:49 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-13T23:33:50.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:33:49 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-13T23:33:50.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:33:49 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-13T23:33:50.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:33:49 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-13T23:33:50.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:33:49 smithi047 conmon[78521]: raise exceptions[0] 2022-05-13T23:33:50.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:33:49 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-13T23:33:50.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:33:49 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-13T23:33:50.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:33:49 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-13T23:33:50.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:33:49 smithi047 conmon[78521]: return (yield from fut) 2022-05-13T23:33:50.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:33:49 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-13T23:33:50.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:33:49 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-13T23:33:50.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:33:49 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:33:50.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:33:50 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:33:50.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:33:50 smithi047 ceph-mon[77147]: fsmap cephfs:5/5 3 up:standby-replay 1 up:replay 4 up:active 2022-05-13T23:33:50.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:33:50 smithi047 ceph-mon[77147]: pgmap v533: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:33:50.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:33:50 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:33:50.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:33:50 smithi115 ceph-mon[78593]: fsmap cephfs:5/5 3 up:standby-replay 1 up:replay 4 up:active 2022-05-13T23:33:50.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:33:50 smithi115 ceph-mon[78593]: pgmap v533: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:33:51.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:33:51 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:33:51.542 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-13T23:33:51.542 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-13T23:33:51.542 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-13T23:33:51.542 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-13T23:33:51.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-13T23:33:51.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-13T23:33:51.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-13T23:33:51.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-13T23:33:51.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-13T23:33:51.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-13T23:33:51.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-13T23:33:51.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-13T23:33:51.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-13T23:33:51.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-13T23:33:51.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-13T23:33:51.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-13T23:33:51.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-13T23:33:51.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:33:51.546 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:33:51 smithi047 ceph-mon[77147]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-13T23:33:51.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:33:51 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:33:51.609 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-13T23:33:51.609 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-13T23:33:51.609 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-13T23:33:51.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-13T23:33:51.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-13T23:33:51.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-13T23:33:51.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-13T23:33:51.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-13T23:33:51.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-13T23:33:51.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-13T23:33:51.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-13T23:33:51.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-13T23:33:51.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-13T23:33:51.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-13T23:33:51.613 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-13T23:33:51.613 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-13T23:33:51.613 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-13T23:33:51.614 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:33:51.614 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:33:51 smithi115 ceph-mon[78593]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-13T23:33:52.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:33:52 smithi047 ceph-mon[77147]: pgmap v534: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:33:52.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:33:52 smithi115 ceph-mon[78593]: pgmap v534: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:33:53.180 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:33:52 smithi047 conmon[78521]: 2022-05-13T23:33:52.816+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:33:53.180 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:33:52 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-13T23:33:53.181 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:33:52 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-13T23:33:53.182 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:33:52 smithi047 conmon[78521]: yield 2022-05-13T23:33:53.182 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:33:52 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-13T23:33:53.182 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:33:52 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-13T23:33:53.182 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:33:52 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-13T23:33:53.183 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:33:52 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-13T23:33:53.183 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:33:52 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-13T23:33:53.183 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:33:52 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-13T23:33:53.183 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:33:52 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-13T23:33:53.184 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:33:52 smithi047 conmon[78521]: raise exceptions[0] 2022-05-13T23:33:53.184 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:33:52 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-13T23:33:53.184 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:33:52 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-13T23:33:53.184 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:33:52 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-13T23:33:53.184 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:33:52 smithi047 conmon[78521]: return (yield from fut) 2022-05-13T23:33:53.185 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:33:52 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-13T23:33:53.185 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:33:52 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-13T23:33:53.185 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:33:52 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:33:53.185 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:33:52 smithi047 conmon[78521]: 2022-05-13T23:33:52.817+0000 7f533ef76700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-13T23:33:53.186 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:33:52 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-13T23:33:53.186 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:33:52 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-13T23:33:53.186 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:33:52 smithi047 conmon[78521]: yield 2022-05-13T23:33:53.186 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:33:52 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-13T23:33:53.187 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:33:52 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-13T23:33:53.187 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:33:52 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-13T23:33:53.187 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:33:52 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-13T23:33:53.187 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:33:52 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-13T23:33:53.187 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:33:52 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-13T23:33:53.188 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:33:52 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-13T23:33:53.188 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:33:52 smithi047 conmon[78521]: raise exceptions[0] 2022-05-13T23:33:53.188 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:33:52 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-13T23:33:53.188 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:33:52 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-13T23:33:53.189 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:33:52 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-13T23:33:53.189 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:33:52 smithi047 conmon[78521]: return (yield from fut) 2022-05-13T23:33:53.189 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:33:52 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-13T23:33:53.189 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:33:52 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-13T23:33:53.190 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:33:52 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:33:53.190 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:33:52 smithi047 conmon[78521]: 2022-05-13T23:33:53.190 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:33:52 smithi047 conmon[78521]: During handling of the above exception, another exception occurred: 2022-05-13T23:33:53.190 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:33:52 smithi047 conmon[78521]: 2022-05-13T23:33:53.191 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:33:52 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-13T23:33:53.191 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:33:52 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-13T23:33:53.191 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:33:52 smithi047 conmon[78521]: return f(*arg) 2022-05-13T23:33:53.192 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:33:52 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-13T23:33:53.192 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:33:52 smithi047 conmon[78521]: self._write_client_files(client_files, host) 2022-05-13T23:33:53.192 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:33:52 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-13T23:33:53.193 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:33:52 smithi047 conmon[78521]: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-13T23:33:53.193 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:33:52 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-13T23:33:53.193 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:33:52 smithi047 conmon[78521]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-13T23:33:53.193 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:33:52 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-13T23:33:53.194 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:33:52 smithi047 conmon[78521]: return self.event_loop.get_result(coro) 2022-05-13T23:33:53.194 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:33:52 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-13T23:33:53.194 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:33:52 smithi047 conmon[78521]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-13T23:33:53.195 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:33:52 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-13T23:33:53.195 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:33:52 smithi047 conmon[78521]: return self.__get_result() 2022-05-13T23:33:53.195 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:33:52 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-13T23:33:53.196 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:33:52 smithi047 conmon[78521]: raise self._exception 2022-05-13T23:33:53.196 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:33:52 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-13T23:33:53.196 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:33:52 smithi047 conmon[78521]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-13T23:33:53.196 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:33:52 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-13T23:33:53.196 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:33:52 smithi047 conmon[78521]: conn = await self._remote_connection(host, addr) 2022-05-13T23:33:53.197 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:33:52 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-13T23:33:53.197 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:33:52 smithi047 conmon[78521]: raise 2022-05-13T23:33:53.197 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:33:52 smithi047 conmon[78521]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-13T23:33:53.198 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:33:52 smithi047 conmon[78521]: self.gen.throw(type, value, traceback) 2022-05-13T23:33:53.198 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:33:52 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-13T23:33:53.198 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:33:52 smithi047 conmon[78521]: raise OrchestratorError(msg) 2022-05-13T23:33:53.198 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:33:52 smithi047 conmon[78521]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:33:53.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:33:53 smithi047 ceph-mon[77147]: Health check update: 0 slow ops, oldest one blocked for 365 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:33:53.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:33:53 smithi115 ceph-mon[78593]: Health check update: 0 slow ops, oldest one blocked for 365 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:33:54.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:33:54 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:33:54.542 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-13T23:33:54.542 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-13T23:33:54.542 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-13T23:33:54.542 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-13T23:33:54.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-13T23:33:54.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-13T23:33:54.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-13T23:33:54.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-13T23:33:54.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-13T23:33:54.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-13T23:33:54.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-13T23:33:54.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-13T23:33:54.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-13T23:33:54.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-13T23:33:54.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-13T23:33:54.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-13T23:33:54.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-13T23:33:54.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:33:54.546 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:33:54 smithi047 ceph-mon[77147]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-13T23:33:54.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-13T23:33:54.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-13T23:33:54.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-13T23:33:54.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-13T23:33:54.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-13T23:33:54.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-13T23:33:54.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-13T23:33:54.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-13T23:33:54.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-13T23:33:54.548 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-13T23:33:54.548 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-13T23:33:54.548 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-13T23:33:54.548 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-13T23:33:54.549 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-13T23:33:54.549 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-13T23:33:54.549 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-13T23:33:54.549 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-13T23:33:54.549 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:33:54.550 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-13T23:33:54.550 INFO:journalctl@ceph.mon.b.smithi047.stdout: During handling of the above exception, another exception occurred: 2022-05-13T23:33:54.550 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-13T23:33:54.550 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-13T23:33:54.550 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-13T23:33:54.551 INFO:journalctl@ceph.mon.b.smithi047.stdout: return f(*arg) 2022-05-13T23:33:54.551 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-13T23:33:54.551 INFO:journalctl@ceph.mon.b.smithi047.stdout: self._write_client_files(client_files, host) 2022-05-13T23:33:54.551 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-13T23:33:54.552 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-13T23:33:54.552 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-13T23:33:54.552 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-13T23:33:54.552 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-13T23:33:54.552 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.event_loop.get_result(coro) 2022-05-13T23:33:54.553 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-13T23:33:54.553 INFO:journalctl@ceph.mon.b.smithi047.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-13T23:33:54.553 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-13T23:33:54.553 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.__get_result() 2022-05-13T23:33:54.553 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-13T23:33:54.554 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise self._exception 2022-05-13T23:33:54.554 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-13T23:33:54.554 INFO:journalctl@ceph.mon.b.smithi047.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-13T23:33:54.554 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-13T23:33:54.554 INFO:journalctl@ceph.mon.b.smithi047.stdout: conn = await self._remote_connection(host, addr) 2022-05-13T23:33:54.555 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-13T23:33:54.555 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise 2022-05-13T23:33:54.555 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-13T23:33:54.555 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.gen.throw(type, value, traceback) 2022-05-13T23:33:54.555 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-13T23:33:54.556 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OrchestratorError(msg) 2022-05-13T23:33:54.556 INFO:journalctl@ceph.mon.b.smithi047.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:33:54.556 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:33:54 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:33:54.556 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:33:54 smithi047 ceph-mon[77147]: fsmap cephfs:5/5 3 up:standby-replay 1 up:replay 4 up:active 2022-05-13T23:33:54.557 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:33:54 smithi047 ceph-mon[77147]: pgmap v535: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:33:54.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:33:54 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:33:54.609 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-13T23:33:54.609 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-13T23:33:54.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-13T23:33:54.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-13T23:33:54.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-13T23:33:54.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-13T23:33:54.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-13T23:33:54.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-13T23:33:54.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-13T23:33:54.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-13T23:33:54.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-13T23:33:54.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-13T23:33:54.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-13T23:33:54.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-13T23:33:54.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-13T23:33:54.613 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-13T23:33:54.613 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-13T23:33:54.613 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:33:54.614 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:33:54 smithi115 ceph-mon[78593]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-13T23:33:54.614 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-13T23:33:54.614 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-13T23:33:54.614 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-13T23:33:54.615 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-13T23:33:54.615 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-13T23:33:54.615 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-13T23:33:54.615 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-13T23:33:54.616 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-13T23:33:54.616 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-13T23:33:54.616 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-13T23:33:54.616 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-13T23:33:54.616 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-13T23:33:54.617 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-13T23:33:54.617 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-13T23:33:54.617 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-13T23:33:54.617 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-13T23:33:54.617 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-13T23:33:54.618 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:33:54.618 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-13T23:33:54.618 INFO:journalctl@ceph.mon.c.smithi115.stdout: During handling of the above exception, another exception occurred: 2022-05-13T23:33:54.618 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-13T23:33:54.618 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-13T23:33:54.619 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-13T23:33:54.619 INFO:journalctl@ceph.mon.c.smithi115.stdout: return f(*arg) 2022-05-13T23:33:54.619 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-13T23:33:54.619 INFO:journalctl@ceph.mon.c.smithi115.stdout: self._write_client_files(client_files, host) 2022-05-13T23:33:54.619 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-13T23:33:54.620 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-13T23:33:54.620 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-13T23:33:54.620 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-13T23:33:54.620 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-13T23:33:54.621 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.event_loop.get_result(coro) 2022-05-13T23:33:54.621 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-13T23:33:54.621 INFO:journalctl@ceph.mon.c.smithi115.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-13T23:33:54.621 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-13T23:33:54.622 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.__get_result() 2022-05-13T23:33:54.622 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-13T23:33:54.622 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise self._exception 2022-05-13T23:33:54.623 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-13T23:33:54.623 INFO:journalctl@ceph.mon.c.smithi115.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-13T23:33:54.624 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-13T23:33:54.624 INFO:journalctl@ceph.mon.c.smithi115.stdout: conn = await self._remote_connection(host, addr) 2022-05-13T23:33:54.624 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-13T23:33:54.625 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise 2022-05-13T23:33:54.625 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-13T23:33:54.626 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.gen.throw(type, value, traceback) 2022-05-13T23:33:54.626 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-13T23:33:54.626 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OrchestratorError(msg) 2022-05-13T23:33:54.627 INFO:journalctl@ceph.mon.c.smithi115.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:33:54.627 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:33:54 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:33:54.627 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:33:54 smithi115 ceph-mon[78593]: fsmap cephfs:5/5 3 up:standby-replay 1 up:replay 4 up:active 2022-05-13T23:33:54.628 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:33:54 smithi115 ceph-mon[78593]: pgmap v535: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:33:56.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:33:56 smithi047 ceph-mon[77147]: pgmap v536: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:33:56.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:33:56 smithi115 ceph-mon[78593]: pgmap v536: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:33:57.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:33:57 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:33:57.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:33:57 smithi047 ceph-mon[77147]: fsmap cephfs:5/5 3 up:standby-replay 1 up:replay 4 up:active 2022-05-13T23:33:57.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:33:57 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:33:57.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:33:57 smithi115 ceph-mon[78593]: fsmap cephfs:5/5 3 up:standby-replay 1 up:replay 4 up:active 2022-05-13T23:33:58.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:33:58 smithi047 ceph-mon[77147]: pgmap v537: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:33:58.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:33:58 smithi047 ceph-mon[77147]: Health check update: 0 slow ops, oldest one blocked for 371 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:33:58.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:33:58 smithi115 ceph-mon[78593]: pgmap v537: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:33:58.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:33:58 smithi115 ceph-mon[78593]: Health check update: 0 slow ops, oldest one blocked for 371 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:34:00.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:34:00 smithi047 ceph-mon[77147]: pgmap v538: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:34:00.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:34:00 smithi115 ceph-mon[78593]: pgmap v538: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:34:01.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:34:01 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:34:01.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:34:01 smithi047 ceph-mon[77147]: fsmap cephfs:5/5 3 up:standby-replay 1 up:replay 4 up:active 2022-05-13T23:34:01.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:34:01 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:34:01.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:34:01 smithi115 ceph-mon[78593]: fsmap cephfs:5/5 3 up:standby-replay 1 up:replay 4 up:active 2022-05-13T23:34:02.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:34:02 smithi047 ceph-mon[77147]: pgmap v539: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:34:02.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:34:02 smithi115 ceph-mon[78593]: pgmap v539: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:34:03.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:34:03 smithi047 ceph-mon[77147]: Health check update: 0 slow ops, oldest one blocked for 375 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:34:03.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:34:03 smithi115 ceph-mon[78593]: Health check update: 0 slow ops, oldest one blocked for 375 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:34:04.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:34:04 smithi047 ceph-mon[77147]: pgmap v540: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:34:04.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:34:04 smithi115 ceph-mon[78593]: pgmap v540: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:34:06.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:34:06 smithi047 ceph-mon[77147]: pgmap v541: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:34:06.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:34:06 smithi115 ceph-mon[78593]: pgmap v541: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:34:08.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:34:08 smithi047 ceph-mon[77147]: pgmap v542: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-05-13T23:34:08.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:34:08 smithi047 ceph-mon[77147]: Health check update: 0 slow ops, oldest one blocked for 380 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:34:08.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:34:08 smithi115 ceph-mon[78593]: pgmap v542: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-05-13T23:34:08.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:34:08 smithi115 ceph-mon[78593]: Health check update: 0 slow ops, oldest one blocked for 380 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:34:10.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:34:10 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:34:10.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:34:10 smithi047 ceph-mon[77147]: fsmap cephfs:5/5 3 up:standby-replay 1 up:replay 4 up:active 2022-05-13T23:34:10.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:34:10 smithi047 ceph-mon[77147]: pgmap v543: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:34:10.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:34:10 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:34:10.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:34:10 smithi115 ceph-mon[78593]: fsmap cephfs:5/5 3 up:standby-replay 1 up:replay 4 up:active 2022-05-13T23:34:10.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:34:10 smithi115 ceph-mon[78593]: pgmap v543: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:34:12.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:34:12 smithi047 ceph-mon[77147]: pgmap v544: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-05-13T23:34:12.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:34:12 smithi115 ceph-mon[78593]: pgmap v544: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-05-13T23:34:13.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:34:13 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:34:13.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:34:13 smithi047 ceph-mon[77147]: fsmap cephfs:5/5 3 up:standby-replay 1 up:replay 4 up:active 2022-05-13T23:34:13.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:34:13 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-13T23:34:13.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:34:13 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-13T23:34:13.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:34:13 smithi047 ceph-mon[77147]: Health check update: 0 slow ops, oldest one blocked for 385 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:34:13.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:34:13 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:34:13.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:34:13 smithi115 ceph-mon[78593]: fsmap cephfs:5/5 3 up:standby-replay 1 up:replay 4 up:active 2022-05-13T23:34:13.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:34:13 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-13T23:34:13.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:34:13 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-13T23:34:13.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:34:13 smithi115 ceph-mon[78593]: Health check update: 0 slow ops, oldest one blocked for 385 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:34:14.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:34:14 smithi047 ceph-mon[77147]: pgmap v545: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-05-13T23:34:14.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:34:14 smithi115 ceph-mon[78593]: pgmap v545: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-05-13T23:34:16.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:34:16 smithi047 ceph-mon[77147]: pgmap v546: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:34:16.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:34:16 smithi115 ceph-mon[78593]: pgmap v546: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:34:17.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:34:17 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:34:17.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:34:17 smithi047 ceph-mon[77147]: fsmap cephfs:5/5 3 up:standby-replay 1 up:replay 4 up:active 2022-05-13T23:34:17.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:34:17 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:34:17.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:34:17 smithi115 ceph-mon[78593]: fsmap cephfs:5/5 3 up:standby-replay 1 up:replay 4 up:active 2022-05-13T23:34:18.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:34:18 smithi047 ceph-mon[77147]: pgmap v547: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:34:18.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:34:18 smithi047 ceph-mon[77147]: Health check update: 0 slow ops, oldest one blocked for 390 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:34:18.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:34:18 smithi115 ceph-mon[78593]: pgmap v547: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:34:18.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:34:18 smithi115 ceph-mon[78593]: Health check update: 0 slow ops, oldest one blocked for 390 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:34:20.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:34:20 smithi047 ceph-mon[77147]: pgmap v548: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:34:20.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:34:20 smithi115 ceph-mon[78593]: pgmap v548: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:34:21.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:34:21 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:34:21.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:34:21 smithi047 ceph-mon[77147]: fsmap cephfs:5/5 3 up:standby-replay 1 up:replay 4 up:active 2022-05-13T23:34:21.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:34:21 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:34:21.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:34:21 smithi115 ceph-mon[78593]: fsmap cephfs:5/5 3 up:standby-replay 1 up:replay 4 up:active 2022-05-13T23:34:22.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:34:22 smithi047 ceph-mon[77147]: pgmap v549: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:34:22.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:34:22 smithi115 ceph-mon[78593]: pgmap v549: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:34:23.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:34:23 smithi047 ceph-mon[77147]: Health check update: 0 slow ops, oldest one blocked for 395 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:34:23.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:34:23 smithi115 ceph-mon[78593]: Health check update: 0 slow ops, oldest one blocked for 395 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:34:24.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:34:24 smithi047 ceph-mon[77147]: pgmap v550: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:34:24.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:34:24 smithi115 ceph-mon[78593]: pgmap v550: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:34:26.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:34:26 smithi047 ceph-mon[77147]: pgmap v551: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:34:26.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:34:26 smithi115 ceph-mon[78593]: pgmap v551: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:34:28.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:34:28 smithi047 ceph-mon[77147]: pgmap v552: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:34:28.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:34:28 smithi047 ceph-mon[77147]: Health check update: 0 slow ops, oldest one blocked for 400 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:34:28.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:34:28 smithi115 ceph-mon[78593]: pgmap v552: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:34:28.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:34:28 smithi115 ceph-mon[78593]: Health check update: 0 slow ops, oldest one blocked for 400 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:34:30.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:34:30 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:34:30.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:34:30 smithi047 ceph-mon[77147]: fsmap cephfs:5/5 3 up:standby-replay 1 up:replay 4 up:active 2022-05-13T23:34:30.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:34:30 smithi047 ceph-mon[77147]: pgmap v553: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:34:30.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:34:30 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:34:30.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:34:30 smithi115 ceph-mon[78593]: fsmap cephfs:5/5 3 up:standby-replay 1 up:replay 4 up:active 2022-05-13T23:34:30.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:34:30 smithi115 ceph-mon[78593]: pgmap v553: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:34:32.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:34:32 smithi047 ceph-mon[77147]: pgmap v554: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:34:32.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:34:32 smithi115 ceph-mon[78593]: pgmap v554: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:34:33.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:34:33 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:34:33.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:34:33 smithi047 ceph-mon[77147]: fsmap cephfs:5/5 3 up:standby-replay 1 up:replay 4 up:active 2022-05-13T23:34:33.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:34:33 smithi047 ceph-mon[77147]: Health check update: 0 slow ops, oldest one blocked for 405 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:34:33.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:34:33 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:34:33.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:34:33 smithi115 ceph-mon[78593]: fsmap cephfs:5/5 3 up:standby-replay 1 up:replay 4 up:active 2022-05-13T23:34:33.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:34:33 smithi115 ceph-mon[78593]: Health check update: 0 slow ops, oldest one blocked for 405 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:34:34.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:34:34 smithi047 ceph-mon[77147]: pgmap v555: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:34:34.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:34:34 smithi115 ceph-mon[78593]: pgmap v555: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:34:36.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:34:36 smithi047 ceph-mon[77147]: pgmap v556: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:34:36.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:34:36 smithi115 ceph-mon[78593]: pgmap v556: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:34:37.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:34:37 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:34:37.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:34:37 smithi047 ceph-mon[77147]: fsmap cephfs:5/5 3 up:standby-replay 1 up:replay 4 up:active 2022-05-13T23:34:37.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:34:37 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:34:37.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:34:37 smithi115 ceph-mon[78593]: fsmap cephfs:5/5 3 up:standby-replay 1 up:replay 4 up:active 2022-05-13T23:34:38.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:34:38 smithi047 ceph-mon[77147]: pgmap v557: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:34:38.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:34:38 smithi047 ceph-mon[77147]: Health check update: 0 slow ops, oldest one blocked for 410 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:34:38.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:34:38 smithi115 ceph-mon[78593]: pgmap v557: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:34:38.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:34:38 smithi115 ceph-mon[78593]: Health check update: 0 slow ops, oldest one blocked for 410 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:34:40.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:34:40 smithi047 ceph-mon[77147]: pgmap v558: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:34:40.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:34:40 smithi115 ceph-mon[78593]: pgmap v558: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:34:41.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:34:41 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:34:41.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:34:41 smithi047 ceph-mon[77147]: fsmap cephfs:5/5 3 up:standby-replay 1 up:replay 4 up:active 2022-05-13T23:34:41.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:34:41 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:34:41.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:34:41 smithi115 ceph-mon[78593]: fsmap cephfs:5/5 3 up:standby-replay 1 up:replay 4 up:active 2022-05-13T23:34:42.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:34:42 smithi047 ceph-mon[77147]: pgmap v559: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:34:42.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:34:42 smithi115 ceph-mon[78593]: pgmap v559: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:34:43.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:34:43 smithi047 ceph-mon[77147]: Health check update: 0 slow ops, oldest one blocked for 415 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:34:43.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:34:43 smithi115 ceph-mon[78593]: Health check update: 0 slow ops, oldest one blocked for 415 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:34:44.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:34:44 smithi047 ceph-mon[77147]: pgmap v560: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:34:44.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:34:44 smithi115 ceph-mon[78593]: pgmap v560: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:34:46.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:34:46 smithi047 ceph-mon[77147]: pgmap v561: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:34:46.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:34:46 smithi115 ceph-mon[78593]: pgmap v561: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:34:48.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:34:48 smithi047 ceph-mon[77147]: pgmap v562: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:34:48.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:34:48 smithi047 ceph-mon[77147]: Health check update: 0 slow ops, oldest one blocked for 420 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:34:48.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:34:48 smithi115 ceph-mon[78593]: pgmap v562: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:34:48.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:34:48 smithi115 ceph-mon[78593]: Health check update: 0 slow ops, oldest one blocked for 420 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:34:50.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:34:50 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:34:50.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:34:50 smithi047 ceph-mon[77147]: fsmap cephfs:5/5 3 up:standby-replay 1 up:replay 4 up:active 2022-05-13T23:34:50.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:34:50 smithi047 ceph-mon[77147]: pgmap v563: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:34:50.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:34:50 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:34:50.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:34:50 smithi115 ceph-mon[78593]: fsmap cephfs:5/5 3 up:standby-replay 1 up:replay 4 up:active 2022-05-13T23:34:50.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:34:50 smithi115 ceph-mon[78593]: pgmap v563: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:34:52.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:34:52 smithi047 ceph-mon[77147]: pgmap v564: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:34:52.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:34:52 smithi115 ceph-mon[78593]: pgmap v564: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:34:53.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:34:53 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:34:53.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:34:53 smithi115 ceph-mon[78593]: fsmap cephfs:5/5 3 up:standby-replay 1 up:replay 4 up:active 2022-05-13T23:34:53.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:34:53 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-13T23:34:53.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:34:53 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-13T23:34:53.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:34:53 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-13T23:34:53.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:34:53 smithi115 ceph-mon[78593]: Health check update: 0 slow ops, oldest one blocked for 425 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:34:53.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:34:53 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:34:53.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:34:53 smithi047 ceph-mon[77147]: fsmap cephfs:5/5 3 up:standby-replay 1 up:replay 4 up:active 2022-05-13T23:34:53.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:34:53 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-13T23:34:53.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:34:53 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-13T23:34:53.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:34:53 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-13T23:34:53.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:34:53 smithi047 ceph-mon[77147]: Health check update: 0 slow ops, oldest one blocked for 425 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:34:54.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:34:54 smithi047 ceph-mon[77147]: pgmap v565: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:34:54.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:34:54 smithi115 ceph-mon[78593]: pgmap v565: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:34:56.291 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:34:55 smithi047 conmon[78521]: 2022-05-13T23:34:55.919+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:34:56.292 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:34:55 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-13T23:34:56.292 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:34:55 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-13T23:34:56.292 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:34:55 smithi047 conmon[78521]: yield 2022-05-13T23:34:56.292 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:34:55 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-13T23:34:56.293 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:34:55 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-13T23:34:56.293 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:34:55 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-13T23:34:56.293 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:34:55 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-13T23:34:56.293 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:34:55 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-13T23:34:56.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:34:55 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-13T23:34:56.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:34:55 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-13T23:34:56.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:34:55 smithi047 conmon[78521]: raise exceptions[0] 2022-05-13T23:34:56.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:34:55 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-13T23:34:56.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:34:55 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-13T23:34:56.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:34:55 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-13T23:34:56.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:34:55 smithi047 conmon[78521]: return (yield from fut) 2022-05-13T23:34:56.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:34:55 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-13T23:34:56.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:34:55 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-13T23:34:56.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:34:55 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:34:56.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:34:56 smithi047 ceph-mon[77147]: pgmap v566: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:34:56.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:34:56 smithi115 ceph-mon[78593]: pgmap v566: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:34:57.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:34:57 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:34:57.791 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-13T23:34:57.792 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-13T23:34:57.792 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-13T23:34:57.792 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-13T23:34:57.793 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-13T23:34:57.793 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-13T23:34:57.793 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-13T23:34:57.793 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-13T23:34:57.793 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-13T23:34:57.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-13T23:34:57.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-13T23:34:57.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-13T23:34:57.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-13T23:34:57.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-13T23:34:57.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-13T23:34:57.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-13T23:34:57.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-13T23:34:57.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:34:57.796 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:34:57 smithi047 ceph-mon[77147]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-13T23:34:57.796 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:34:57 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:34:57.796 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:34:57 smithi047 ceph-mon[77147]: fsmap cephfs:5/5 3 up:standby-replay 1 up:replay 4 up:active 2022-05-13T23:34:57.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:34:57 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:34:57.858 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-13T23:34:57.858 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-13T23:34:57.859 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-13T23:34:57.859 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-13T23:34:57.859 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-13T23:34:57.859 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-13T23:34:57.859 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-13T23:34:57.860 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-13T23:34:57.860 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-13T23:34:57.860 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-13T23:34:57.860 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-13T23:34:57.860 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-13T23:34:57.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-13T23:34:57.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-13T23:34:57.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-13T23:34:57.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-13T23:34:57.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-13T23:34:57.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:34:57.862 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:34:57 smithi115 ceph-mon[78593]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-13T23:34:57.862 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:34:57 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:34:57.862 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:34:57 smithi115 ceph-mon[78593]: fsmap cephfs:5/5 3 up:standby-replay 1 up:replay 4 up:active 2022-05-13T23:34:58.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:34:58 smithi047 ceph-mon[77147]: pgmap v567: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:34:58.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:34:58 smithi047 ceph-mon[77147]: Health check update: 0 slow ops, oldest one blocked for 430 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:34:58.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:34:58 smithi115 ceph-mon[78593]: pgmap v567: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:34:58.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:34:58 smithi115 ceph-mon[78593]: Health check update: 0 slow ops, oldest one blocked for 430 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:34:59.291 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:34:58 smithi047 conmon[78521]: 2022-05-13T23:34:58.991+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:34:59.292 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:34:58 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-13T23:34:59.292 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:34:58 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-13T23:34:59.292 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:34:58 smithi047 conmon[78521]: yield 2022-05-13T23:34:59.293 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:34:58 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-13T23:34:59.293 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:34:58 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-13T23:34:59.293 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:34:58 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-13T23:34:59.293 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:34:58 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-13T23:34:59.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:34:58 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-13T23:34:59.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:34:58 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-13T23:34:59.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:34:58 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-13T23:34:59.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:34:58 smithi047 conmon[78521]: raise exceptions[0] 2022-05-13T23:34:59.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:34:58 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-13T23:34:59.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:34:58 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-13T23:34:59.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:34:58 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-13T23:34:59.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:34:58 smithi047 conmon[78521]: return (yield from fut) 2022-05-13T23:34:59.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:34:58 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-13T23:34:59.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:34:58 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-13T23:34:59.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:34:58 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:34:59.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:34:58 smithi047 conmon[78521]: 2022-05-13T23:34:58.992+0000 7f5345f84700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-13T23:34:59.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:34:58 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-13T23:34:59.297 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:34:58 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-13T23:34:59.297 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:34:58 smithi047 conmon[78521]: yield 2022-05-13T23:34:59.297 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:34:58 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-13T23:34:59.297 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:34:58 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-13T23:34:59.298 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:34:58 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-13T23:34:59.298 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:34:58 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-13T23:34:59.298 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:34:58 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-13T23:34:59.298 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:34:58 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-13T23:34:59.298 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:34:58 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-13T23:34:59.299 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:34:58 smithi047 conmon[78521]: raise exceptions[0] 2022-05-13T23:34:59.299 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:34:58 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-13T23:34:59.299 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:34:58 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-13T23:34:59.299 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:34:58 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-13T23:34:59.300 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:34:58 smithi047 conmon[78521]: return (yield from fut) 2022-05-13T23:34:59.300 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:34:58 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-13T23:34:59.300 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:34:58 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-13T23:34:59.300 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:34:58 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:34:59.300 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:34:58 smithi047 conmon[78521]: 2022-05-13T23:34:59.301 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:34:58 smithi047 conmon[78521]: During handling of the above exception, another exception occurred: 2022-05-13T23:34:59.301 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:34:58 smithi047 conmon[78521]: 2022-05-13T23:34:59.301 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:34:58 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-13T23:34:59.301 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:34:58 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-13T23:34:59.302 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:34:58 smithi047 conmon[78521]: return f(*arg) 2022-05-13T23:34:59.302 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:34:58 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-13T23:34:59.302 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:34:58 smithi047 conmon[78521]: self._write_client_files(client_files, host) 2022-05-13T23:34:59.302 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:34:58 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-13T23:34:59.302 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:34:58 smithi047 conmon[78521]: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-13T23:34:59.303 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:34:58 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-13T23:34:59.303 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:34:58 smithi047 conmon[78521]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-13T23:34:59.303 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:34:58 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-13T23:34:59.303 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:34:58 smithi047 conmon[78521]: return self.event_loop.get_result(coro) 2022-05-13T23:34:59.304 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:34:58 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-13T23:34:59.304 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:34:58 smithi047 conmon[78521]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-13T23:34:59.304 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:34:58 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-13T23:34:59.304 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:34:58 smithi047 conmon[78521]: return self.__get_result() 2022-05-13T23:34:59.304 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:34:58 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-13T23:34:59.305 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:34:58 smithi047 conmon[78521]: raise self._exception 2022-05-13T23:34:59.305 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:34:58 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-13T23:34:59.305 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:34:58 smithi047 conmon[78521]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-13T23:34:59.305 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:34:58 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-13T23:34:59.306 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:34:58 smithi047 conmon[78521]: conn = await self._remote_connection(host, addr) 2022-05-13T23:34:59.306 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:34:58 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-13T23:34:59.306 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:34:58 smithi047 conmon[78521]: raise 2022-05-13T23:34:59.306 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:34:58 smithi047 conmon[78521]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-13T23:34:59.306 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:34:58 smithi047 conmon[78521]: self.gen.throw(type, value, traceback) 2022-05-13T23:34:59.307 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:34:58 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-13T23:34:59.307 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:34:58 smithi047 conmon[78521]: raise OrchestratorError(msg) 2022-05-13T23:34:59.307 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:34:58 smithi047 conmon[78521]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:35:00.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:35:00 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:35:00.792 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-13T23:35:00.792 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-13T23:35:00.792 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-13T23:35:00.793 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-13T23:35:00.793 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-13T23:35:00.793 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-13T23:35:00.793 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-13T23:35:00.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-13T23:35:00.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-13T23:35:00.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-13T23:35:00.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-13T23:35:00.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-13T23:35:00.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-13T23:35:00.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-13T23:35:00.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-13T23:35:00.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-13T23:35:00.797 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-13T23:35:00.797 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:35:00.797 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:35:00 smithi047 ceph-mon[77147]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-13T23:35:00.798 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-13T23:35:00.798 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-13T23:35:00.798 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-13T23:35:00.799 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-13T23:35:00.799 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-13T23:35:00.799 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-13T23:35:00.800 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-13T23:35:00.800 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-13T23:35:00.800 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-13T23:35:00.801 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-13T23:35:00.801 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-13T23:35:00.801 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-13T23:35:00.802 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-13T23:35:00.802 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-13T23:35:00.802 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-13T23:35:00.803 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-13T23:35:00.803 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-13T23:35:00.803 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:35:00.804 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-13T23:35:00.804 INFO:journalctl@ceph.mon.b.smithi047.stdout: During handling of the above exception, another exception occurred: 2022-05-13T23:35:00.804 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-13T23:35:00.805 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-13T23:35:00.805 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-13T23:35:00.806 INFO:journalctl@ceph.mon.b.smithi047.stdout: return f(*arg) 2022-05-13T23:35:00.806 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-13T23:35:00.806 INFO:journalctl@ceph.mon.b.smithi047.stdout: self._write_client_files(client_files, host) 2022-05-13T23:35:00.807 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-13T23:35:00.807 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-13T23:35:00.807 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-13T23:35:00.808 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-13T23:35:00.808 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-13T23:35:00.808 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.event_loop.get_result(coro) 2022-05-13T23:35:00.809 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-13T23:35:00.809 INFO:journalctl@ceph.mon.b.smithi047.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-13T23:35:00.809 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-13T23:35:00.810 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.__get_result() 2022-05-13T23:35:00.810 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-13T23:35:00.810 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise self._exception 2022-05-13T23:35:00.811 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-13T23:35:00.811 INFO:journalctl@ceph.mon.b.smithi047.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-13T23:35:00.811 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-13T23:35:00.812 INFO:journalctl@ceph.mon.b.smithi047.stdout: conn = await self._remote_connection(host, addr) 2022-05-13T23:35:00.812 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-13T23:35:00.812 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise 2022-05-13T23:35:00.813 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-13T23:35:00.813 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.gen.throw(type, value, traceback) 2022-05-13T23:35:00.813 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-13T23:35:00.814 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OrchestratorError(msg) 2022-05-13T23:35:00.814 INFO:journalctl@ceph.mon.b.smithi047.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:35:00.814 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:35:00 smithi047 ceph-mon[77147]: pgmap v568: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:35:00.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:35:00 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:35:00.859 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-13T23:35:00.859 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-13T23:35:00.860 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-13T23:35:00.860 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-13T23:35:00.860 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-13T23:35:00.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-13T23:35:00.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-13T23:35:00.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-13T23:35:00.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-13T23:35:00.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-13T23:35:00.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-13T23:35:00.863 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-13T23:35:00.863 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-13T23:35:00.864 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-13T23:35:00.864 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-13T23:35:00.864 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-13T23:35:00.865 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-13T23:35:00.865 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:35:00.865 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:35:00 smithi115 ceph-mon[78593]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-13T23:35:00.866 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-13T23:35:00.866 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-13T23:35:00.866 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-13T23:35:00.867 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-13T23:35:00.867 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-13T23:35:00.867 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-13T23:35:00.867 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-13T23:35:00.868 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-13T23:35:00.868 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-13T23:35:00.868 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-13T23:35:00.868 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-13T23:35:00.869 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-13T23:35:00.869 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-13T23:35:00.869 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-13T23:35:00.869 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-13T23:35:00.870 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-13T23:35:00.870 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-13T23:35:00.870 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:35:00.870 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-13T23:35:00.870 INFO:journalctl@ceph.mon.c.smithi115.stdout: During handling of the above exception, another exception occurred: 2022-05-13T23:35:00.871 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-13T23:35:00.871 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-13T23:35:00.871 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-13T23:35:00.871 INFO:journalctl@ceph.mon.c.smithi115.stdout: return f(*arg) 2022-05-13T23:35:00.872 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-13T23:35:00.872 INFO:journalctl@ceph.mon.c.smithi115.stdout: self._write_client_files(client_files, host) 2022-05-13T23:35:00.872 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-13T23:35:00.872 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-13T23:35:00.873 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-13T23:35:00.873 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-13T23:35:00.873 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-13T23:35:00.873 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.event_loop.get_result(coro) 2022-05-13T23:35:00.874 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-13T23:35:00.874 INFO:journalctl@ceph.mon.c.smithi115.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-13T23:35:00.874 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-13T23:35:00.875 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.__get_result() 2022-05-13T23:35:00.875 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-13T23:35:00.876 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise self._exception 2022-05-13T23:35:00.876 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-13T23:35:00.877 INFO:journalctl@ceph.mon.c.smithi115.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-13T23:35:00.877 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-13T23:35:00.878 INFO:journalctl@ceph.mon.c.smithi115.stdout: conn = await self._remote_connection(host, addr) 2022-05-13T23:35:00.878 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-13T23:35:00.878 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise 2022-05-13T23:35:00.878 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-13T23:35:00.879 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.gen.throw(type, value, traceback) 2022-05-13T23:35:00.879 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-13T23:35:00.879 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OrchestratorError(msg) 2022-05-13T23:35:00.879 INFO:journalctl@ceph.mon.c.smithi115.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:35:00.880 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:35:00 smithi115 ceph-mon[78593]: pgmap v568: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:35:01.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:35:01 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:35:01.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:35:01 smithi047 ceph-mon[77147]: fsmap cephfs:5/5 3 up:standby-replay 1 up:replay 4 up:active 2022-05-13T23:35:01.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:35:01 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:35:01.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:35:01 smithi115 ceph-mon[78593]: fsmap cephfs:5/5 3 up:standby-replay 1 up:replay 4 up:active 2022-05-13T23:35:02.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:35:02 smithi047 ceph-mon[77147]: pgmap v569: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:35:02.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:35:02 smithi115 ceph-mon[78593]: pgmap v569: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:35:03.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:35:03 smithi047 ceph-mon[77147]: Health check update: 0 slow ops, oldest one blocked for 435 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:35:03.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:35:03 smithi115 ceph-mon[78593]: Health check update: 0 slow ops, oldest one blocked for 435 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:35:04.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:35:04 smithi047 ceph-mon[77147]: pgmap v570: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:35:04.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:35:04 smithi115 ceph-mon[78593]: pgmap v570: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:35:06.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:35:06 smithi047 ceph-mon[77147]: pgmap v571: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:35:06.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:35:06 smithi115 ceph-mon[78593]: pgmap v571: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:35:08.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:35:08 smithi047 ceph-mon[77147]: pgmap v572: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:35:08.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:35:08 smithi047 ceph-mon[77147]: Health check update: 0 slow ops, oldest one blocked for 440 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:35:08.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:35:08 smithi115 ceph-mon[78593]: pgmap v572: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:35:08.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:35:08 smithi115 ceph-mon[78593]: Health check update: 0 slow ops, oldest one blocked for 440 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:35:10.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:35:10 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:35:10.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:35:10 smithi047 ceph-mon[77147]: fsmap cephfs:5/5 3 up:standby-replay 1 up:replay 4 up:active 2022-05-13T23:35:10.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:35:10 smithi047 ceph-mon[77147]: pgmap v573: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:35:10.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:35:10 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:35:10.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:35:10 smithi115 ceph-mon[78593]: fsmap cephfs:5/5 3 up:standby-replay 1 up:replay 4 up:active 2022-05-13T23:35:10.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:35:10 smithi115 ceph-mon[78593]: pgmap v573: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:35:12.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:35:12 smithi047 ceph-mon[77147]: pgmap v574: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-05-13T23:35:12.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:35:12 smithi115 ceph-mon[78593]: pgmap v574: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-05-13T23:35:13.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:35:13 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:35:13.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:35:13 smithi047 ceph-mon[77147]: fsmap cephfs:5/5 3 up:standby-replay 1 up:replay 4 up:active 2022-05-13T23:35:13.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:35:13 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-13T23:35:13.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:35:13 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-13T23:35:13.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:35:13 smithi047 ceph-mon[77147]: Health check update: 0 slow ops, oldest one blocked for 445 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:35:13.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:35:13 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:35:13.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:35:13 smithi115 ceph-mon[78593]: fsmap cephfs:5/5 3 up:standby-replay 1 up:replay 4 up:active 2022-05-13T23:35:13.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:35:13 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-13T23:35:13.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:35:13 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-13T23:35:13.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:35:13 smithi115 ceph-mon[78593]: Health check update: 0 slow ops, oldest one blocked for 445 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:35:14.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:35:14 smithi047 ceph-mon[77147]: pgmap v575: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-05-13T23:35:14.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:35:14 smithi115 ceph-mon[78593]: pgmap v575: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-05-13T23:35:16.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:35:16 smithi047 ceph-mon[77147]: pgmap v576: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:35:16.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:35:16 smithi115 ceph-mon[78593]: pgmap v576: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:35:17.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:35:17 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:35:17.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:35:17 smithi047 ceph-mon[77147]: fsmap cephfs:5/5 3 up:standby-replay 1 up:replay 4 up:active 2022-05-13T23:35:17.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:35:17 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:35:17.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:35:17 smithi115 ceph-mon[78593]: fsmap cephfs:5/5 3 up:standby-replay 1 up:replay 4 up:active 2022-05-13T23:35:18.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:35:18 smithi047 ceph-mon[77147]: pgmap v577: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-05-13T23:35:18.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:35:18 smithi047 ceph-mon[77147]: Health check update: 0 slow ops, oldest one blocked for 450 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:35:18.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:35:18 smithi115 ceph-mon[78593]: pgmap v577: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-05-13T23:35:18.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:35:18 smithi115 ceph-mon[78593]: Health check update: 0 slow ops, oldest one blocked for 450 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:35:20.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:35:20 smithi047 ceph-mon[77147]: pgmap v578: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:35:20.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:35:20 smithi115 ceph-mon[78593]: pgmap v578: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:35:21.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:35:21 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:35:21.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:35:21 smithi047 ceph-mon[77147]: fsmap cephfs:5/5 3 up:standby-replay 1 up:replay 4 up:active 2022-05-13T23:35:21.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:35:21 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:35:21.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:35:21 smithi115 ceph-mon[78593]: fsmap cephfs:5/5 3 up:standby-replay 1 up:replay 4 up:active 2022-05-13T23:35:22.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:35:22 smithi047 ceph-mon[77147]: pgmap v579: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:35:22.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:35:22 smithi115 ceph-mon[78593]: pgmap v579: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:35:23.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:35:23 smithi047 ceph-mon[77147]: Health check update: 0 slow ops, oldest one blocked for 456 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:35:23.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:35:23 smithi115 ceph-mon[78593]: Health check update: 0 slow ops, oldest one blocked for 456 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:35:24.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:35:24 smithi047 ceph-mon[77147]: pgmap v580: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:35:24.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:35:24 smithi115 ceph-mon[78593]: pgmap v580: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:35:26.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:35:26 smithi047 ceph-mon[77147]: pgmap v581: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:35:26.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:35:26 smithi115 ceph-mon[78593]: pgmap v581: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:35:27.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:35:27 smithi047 ceph-mon[77147]: pgmap v582: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:35:27.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:35:27 smithi115 ceph-mon[78593]: pgmap v582: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:35:28.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:35:28 smithi047 ceph-mon[77147]: Health check update: 0 slow ops, oldest one blocked for 460 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:35:28.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:35:28 smithi115 ceph-mon[78593]: Health check update: 0 slow ops, oldest one blocked for 460 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:35:30.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:35:30 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:35:30.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:35:30 smithi047 ceph-mon[77147]: fsmap cephfs:5/5 3 up:standby-replay 1 up:replay 4 up:active 2022-05-13T23:35:30.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:35:30 smithi047 ceph-mon[77147]: pgmap v583: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:35:30.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:35:30 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:35:30.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:35:30 smithi115 ceph-mon[78593]: fsmap cephfs:5/5 3 up:standby-replay 1 up:replay 4 up:active 2022-05-13T23:35:30.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:35:30 smithi115 ceph-mon[78593]: pgmap v583: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:35:32.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:35:32 smithi047 ceph-mon[77147]: pgmap v584: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:35:32.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:35:32 smithi115 ceph-mon[78593]: pgmap v584: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:35:33.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:35:33 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:35:33.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:35:33 smithi047 ceph-mon[77147]: fsmap cephfs:5/5 3 up:standby-replay 1 up:replay 4 up:active 2022-05-13T23:35:33.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:35:33 smithi047 ceph-mon[77147]: Health check update: 0 slow ops, oldest one blocked for 465 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:35:33.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:35:33 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:35:33.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:35:33 smithi115 ceph-mon[78593]: fsmap cephfs:5/5 3 up:standby-replay 1 up:replay 4 up:active 2022-05-13T23:35:33.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:35:33 smithi115 ceph-mon[78593]: Health check update: 0 slow ops, oldest one blocked for 465 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:35:34.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:35:34 smithi047 ceph-mon[77147]: pgmap v585: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:35:34.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:35:34 smithi115 ceph-mon[78593]: pgmap v585: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:35:36.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:35:36 smithi047 ceph-mon[77147]: pgmap v586: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:35:36.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:35:36 smithi115 ceph-mon[78593]: pgmap v586: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:35:37.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:35:37 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:35:37.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:35:37 smithi047 ceph-mon[77147]: fsmap cephfs:5/5 3 up:standby-replay 1 up:replay 4 up:active 2022-05-13T23:35:37.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:35:37 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:35:37.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:35:37 smithi115 ceph-mon[78593]: fsmap cephfs:5/5 3 up:standby-replay 1 up:replay 4 up:active 2022-05-13T23:35:38.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:35:38 smithi047 ceph-mon[77147]: pgmap v587: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:35:38.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:35:38 smithi047 ceph-mon[77147]: Health check update: 0 slow ops, oldest one blocked for 470 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:35:38.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:35:38 smithi115 ceph-mon[78593]: pgmap v587: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:35:38.942 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:35:38 smithi115 ceph-mon[78593]: Health check update: 0 slow ops, oldest one blocked for 470 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:35:40.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:35:40 smithi047 ceph-mon[77147]: pgmap v588: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:35:40.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:35:40 smithi115 ceph-mon[78593]: pgmap v588: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:35:41.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:35:41 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:35:41.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:35:41 smithi047 ceph-mon[77147]: fsmap cephfs:5/5 3 up:standby-replay 1 up:replay 4 up:active 2022-05-13T23:35:41.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:35:41 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:35:41.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:35:41 smithi115 ceph-mon[78593]: fsmap cephfs:5/5 3 up:standby-replay 1 up:replay 4 up:active 2022-05-13T23:35:42.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:35:42 smithi047 ceph-mon[77147]: pgmap v589: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-05-13T23:35:42.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:35:42 smithi115 ceph-mon[78593]: pgmap v589: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-05-13T23:35:43.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:35:43 smithi047 ceph-mon[77147]: Health check update: 0 slow ops, oldest one blocked for 475 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:35:43.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:35:43 smithi115 ceph-mon[78593]: Health check update: 0 slow ops, oldest one blocked for 475 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:35:44.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:35:44 smithi047 ceph-mon[77147]: pgmap v590: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-05-13T23:35:44.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:35:44 smithi115 ceph-mon[78593]: pgmap v590: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-05-13T23:35:46.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:35:46 smithi047 ceph-mon[77147]: pgmap v591: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:35:46.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:35:46 smithi115 ceph-mon[78593]: pgmap v591: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:35:48.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:35:48 smithi047 ceph-mon[77147]: pgmap v592: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-05-13T23:35:48.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:35:48 smithi047 ceph-mon[77147]: Health check update: 0 slow ops, oldest one blocked for 480 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:35:48.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:35:48 smithi115 ceph-mon[78593]: pgmap v592: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-05-13T23:35:48.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:35:48 smithi115 ceph-mon[78593]: Health check update: 0 slow ops, oldest one blocked for 480 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:35:50.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:35:50 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:35:50.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:35:50 smithi047 ceph-mon[77147]: fsmap cephfs:5/5 3 up:standby-replay 1 up:replay 4 up:active 2022-05-13T23:35:50.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:35:50 smithi047 ceph-mon[77147]: pgmap v593: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:35:50.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:35:50 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:35:50.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:35:50 smithi115 ceph-mon[78593]: fsmap cephfs:5/5 3 up:standby-replay 1 up:replay 4 up:active 2022-05-13T23:35:50.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:35:50 smithi115 ceph-mon[78593]: pgmap v593: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:35:51.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:35:51 smithi115 conmon[90393]: 2022-05-13T23:35:51.227+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:35:52.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:35:52 smithi115 conmon[90393]: 2022-05-13T23:35:52.199+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:35:52.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:35:52 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:35:52.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:35:52 smithi115 ceph-mon[78593]: pgmap v594: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:35:52.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:35:52 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:35:52.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:35:52 smithi047 ceph-mon[77147]: pgmap v594: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:35:53.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:35:53 smithi115 conmon[90393]: 2022-05-13T23:35:53.224+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:35:53.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:35:53 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:35:53.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:35:53 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:35:53.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:35:53 smithi115 ceph-mon[78593]: fsmap cephfs:5/5 3 up:standby-replay 1 up:replay 4 up:active 2022-05-13T23:35:53.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:35:53 smithi115 ceph-mon[78593]: Health check update: 0 slow ops, oldest one blocked for 485 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:35:53.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:35:53 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:35:53.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:35:53 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:35:53.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:35:53 smithi047 ceph-mon[77147]: fsmap cephfs:5/5 3 up:standby-replay 1 up:replay 4 up:active 2022-05-13T23:35:53.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:35:53 smithi047 ceph-mon[77147]: Health check update: 0 slow ops, oldest one blocked for 485 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:35:54.454 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:35:54 smithi115 conmon[90393]: 2022-05-13T23:35:54.176+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:35:54.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:35:54 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:35:54.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:35:54 smithi047 ceph-mon[77147]: pgmap v595: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:35:54.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:35:54 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:35:54.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:35:54 smithi115 ceph-mon[78593]: pgmap v595: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:35:55.459 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:35:55 smithi115 conmon[90393]: 2022-05-13T23:35:55.131+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:35:55.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:35:55 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:35:55.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:35:55 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:35:56.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:35:56 smithi115 conmon[90393]: 2022-05-13T23:35:56.098+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:35:56.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:35:56 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:35:56.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:35:56 smithi047 ceph-mon[77147]: pgmap v596: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:35:56.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:35:56 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:35:56.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:35:56 smithi115 ceph-mon[78593]: pgmap v596: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:35:57.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:35:57 smithi115 conmon[90393]: 2022-05-13T23:35:57.104+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:35:57.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:35:57 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:35:57.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:35:57 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:35:57.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:35:57 smithi047 ceph-mon[77147]: fsmap cephfs:5/5 3 up:standby-replay 1 up:replay 4 up:active 2022-05-13T23:35:57.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:35:57 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:35:57.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:35:57 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:35:57.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:35:57 smithi115 ceph-mon[78593]: fsmap cephfs:5/5 3 up:standby-replay 1 up:replay 4 up:active 2022-05-13T23:35:58.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:35:58 smithi115 conmon[90393]: 2022-05-13T23:35:58.088+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:35:58.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:35:58 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:35:58.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:35:58 smithi047 ceph-mon[77147]: pgmap v597: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:35:58.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:35:58 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 490 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:35:58.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:35:58 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:35:58.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:35:58 smithi115 ceph-mon[78593]: pgmap v597: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:35:58.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:35:58 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 490 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:35:59.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:35:59 smithi115 conmon[90393]: 2022-05-13T23:35:59.049+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:35:59.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:35:59 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:35:59.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:35:59 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-13T23:35:59.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:35:59 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-13T23:35:59.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:35:59 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-13T23:35:59.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:35:59 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:35:59.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:35:59 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:35:59.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:35:59 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-13T23:35:59.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:35:59 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-13T23:35:59.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:35:59 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-13T23:35:59.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:35:59 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:36:00.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:36:00 smithi115 conmon[90393]: 2022-05-13T23:36:00.011+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:36:00.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:36:00 smithi047 ceph-mon[77147]: pgmap v598: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:36:00.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:36:00 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:36:00.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:36:00 smithi115 ceph-mon[78593]: pgmap v598: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:36:00.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:36:00 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:36:01.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:36:01 smithi115 conmon[90393]: 2022-05-13T23:36:01.014+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:36:01.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:36:01 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:36:01.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:36:01 smithi047 ceph-mon[77147]: fsmap cephfs:5/5 3 up:standby-replay 1 up:replay 4 up:active 2022-05-13T23:36:01.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:36:01 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:36:01.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:36:01 smithi047 ceph-mon[77147]: pgmap v599: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:36:01.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:36:01 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:36:01.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:36:01 smithi115 ceph-mon[78593]: fsmap cephfs:5/5 3 up:standby-replay 1 up:replay 4 up:active 2022-05-13T23:36:01.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:36:01 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:36:01.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:36:01 smithi115 ceph-mon[78593]: pgmap v599: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:36:02.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:36:02 smithi115 conmon[90393]: 2022-05-13T23:36:02.005+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:36:02.489 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:36:02 smithi047 conmon[78521]: 2022-05-13T23:36:02.096+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:36:02.489 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:36:02 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-13T23:36:02.490 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:36:02 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-13T23:36:02.490 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:36:02 smithi047 conmon[78521]: yield 2022-05-13T23:36:02.490 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:36:02 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-13T23:36:02.490 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:36:02 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-13T23:36:02.490 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:36:02 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-13T23:36:02.491 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:36:02 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-13T23:36:02.491 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:36:02 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-13T23:36:02.491 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:36:02 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-13T23:36:02.491 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:36:02 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-13T23:36:02.492 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:36:02 smithi047 conmon[78521]: raise exceptions[0] 2022-05-13T23:36:02.492 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:36:02 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-13T23:36:02.492 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:36:02 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-13T23:36:02.493 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:36:02 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-13T23:36:02.493 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:36:02 smithi047 conmon[78521]: return (yield from fut) 2022-05-13T23:36:02.493 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:36:02 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-13T23:36:02.493 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:36:02 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-13T23:36:02.494 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:36:02 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:36:02.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:36:02 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:36:02.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:36:02 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:36:02.792 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-13T23:36:02.792 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-13T23:36:02.792 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-13T23:36:02.792 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-13T23:36:02.793 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-13T23:36:02.793 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-13T23:36:02.793 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-13T23:36:02.793 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-13T23:36:02.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-13T23:36:02.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-13T23:36:02.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-13T23:36:02.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-13T23:36:02.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-13T23:36:02.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-13T23:36:02.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-13T23:36:02.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-13T23:36:02.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-13T23:36:02.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:36:02.796 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:36:02 smithi047 ceph-mon[77147]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-13T23:36:02.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:36:02 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:36:02.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:36:02 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:36:02.858 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-13T23:36:02.859 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-13T23:36:02.859 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-13T23:36:02.859 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-13T23:36:02.859 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-13T23:36:02.859 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-13T23:36:02.860 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-13T23:36:02.860 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-13T23:36:02.860 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-13T23:36:02.860 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-13T23:36:02.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-13T23:36:02.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-13T23:36:02.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-13T23:36:02.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-13T23:36:02.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-13T23:36:02.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-13T23:36:02.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-13T23:36:02.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:36:02.862 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:36:02 smithi115 ceph-mon[78593]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-13T23:36:03.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:36:03 smithi115 conmon[90393]: 2022-05-13T23:36:03.020+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:36:03.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:36:03 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:36:03.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:36:03 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 495 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:36:03.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:36:03 smithi047 ceph-mon[77147]: pgmap v600: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:36:03.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:36:03 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:36:03.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:36:03 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 495 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:36:03.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:36:03 smithi115 ceph-mon[78593]: pgmap v600: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:36:04.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:36:03 smithi115 conmon[90393]: 2022-05-13T23:36:03.992+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:36:04.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:36:04 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:36:04.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:36:04 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:36:05.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:36:05 smithi115 conmon[90393]: 2022-05-13T23:36:05.003+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:36:05.502 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:36:05 smithi047 conmon[78521]: 2022-05-13T23:36:05.168+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:36:05.502 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:36:05 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-13T23:36:05.502 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:36:05 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-13T23:36:05.502 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:36:05 smithi047 conmon[78521]: yield 2022-05-13T23:36:05.503 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:36:05 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-13T23:36:05.503 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:36:05 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-13T23:36:05.503 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:36:05 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-13T23:36:05.503 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:36:05 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-13T23:36:05.504 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:36:05 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-13T23:36:05.504 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:36:05 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-13T23:36:05.504 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:36:05 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-13T23:36:05.504 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:36:05 smithi047 conmon[78521]: raise exceptions[0] 2022-05-13T23:36:05.504 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:36:05 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-13T23:36:05.505 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:36:05 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-13T23:36:05.505 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:36:05 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-13T23:36:05.505 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:36:05 smithi047 conmon[78521]: return (yield from fut) 2022-05-13T23:36:05.505 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:36:05 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-13T23:36:05.506 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:36:05 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-13T23:36:05.506 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:36:05 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:36:05.506 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:36:05 smithi047 conmon[78521]: 2022-05-13T23:36:05.169+0000 7f5340f7a700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-13T23:36:05.506 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:36:05 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-13T23:36:05.506 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:36:05 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-13T23:36:05.507 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:36:05 smithi047 conmon[78521]: yield 2022-05-13T23:36:05.507 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:36:05 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-13T23:36:05.507 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:36:05 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-13T23:36:05.507 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:36:05 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-13T23:36:05.508 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:36:05 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-13T23:36:05.508 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:36:05 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-13T23:36:05.508 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:36:05 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-13T23:36:05.508 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:36:05 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-13T23:36:05.508 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:36:05 smithi047 conmon[78521]: raise exceptions[0] 2022-05-13T23:36:05.509 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:36:05 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-13T23:36:05.509 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:36:05 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-13T23:36:05.509 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:36:05 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-13T23:36:05.509 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:36:05 smithi047 conmon[78521]: return (yield from fut) 2022-05-13T23:36:05.510 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:36:05 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-13T23:36:05.510 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:36:05 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-13T23:36:05.510 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:36:05 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:36:05.511 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:36:05 smithi047 conmon[78521]: 2022-05-13T23:36:05.511 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:36:05 smithi047 conmon[78521]: During handling of the above exception, another exception occurred: 2022-05-13T23:36:05.511 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:36:05 smithi047 conmon[78521]: 2022-05-13T23:36:05.511 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:36:05 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-13T23:36:05.511 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:36:05 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-13T23:36:05.512 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:36:05 smithi047 conmon[78521]: return f(*arg) 2022-05-13T23:36:05.512 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:36:05 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-13T23:36:05.512 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:36:05 smithi047 conmon[78521]: self._write_client_files(client_files, host) 2022-05-13T23:36:05.512 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:36:05 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-13T23:36:05.513 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:36:05 smithi047 conmon[78521]: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-13T23:36:05.513 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:36:05 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-13T23:36:05.513 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:36:05 smithi047 conmon[78521]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-13T23:36:05.513 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:36:05 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-13T23:36:05.514 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:36:05 smithi047 conmon[78521]: return self.event_loop.get_result(coro) 2022-05-13T23:36:05.514 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:36:05 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-13T23:36:05.514 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:36:05 smithi047 conmon[78521]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-13T23:36:05.514 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:36:05 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-13T23:36:05.515 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:36:05 smithi047 conmon[78521]: return self.__get_result() 2022-05-13T23:36:05.515 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:36:05 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-13T23:36:05.515 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:36:05 smithi047 conmon[78521]: raise self._exception 2022-05-13T23:36:05.515 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:36:05 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-13T23:36:05.515 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:36:05 smithi047 conmon[78521]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-13T23:36:05.516 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:36:05 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-13T23:36:05.516 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:36:05 smithi047 conmon[78521]: conn = await self._remote_connection(host, addr) 2022-05-13T23:36:05.516 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:36:05 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-13T23:36:05.516 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:36:05 smithi047 conmon[78521]: raise 2022-05-13T23:36:05.517 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:36:05 smithi047 conmon[78521]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-13T23:36:05.517 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:36:05 smithi047 conmon[78521]: self.gen.throw(type, value, traceback) 2022-05-13T23:36:05.517 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:36:05 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-13T23:36:05.517 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:36:05 smithi047 conmon[78521]: raise OrchestratorError(msg) 2022-05-13T23:36:05.517 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:36:05 smithi047 conmon[78521]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:36:05.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:36:05 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:36:05.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:36:05 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:36:05.792 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-13T23:36:05.792 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-13T23:36:05.792 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-13T23:36:05.792 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-13T23:36:05.793 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-13T23:36:05.793 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-13T23:36:05.793 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-13T23:36:05.793 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-13T23:36:05.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-13T23:36:05.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-13T23:36:05.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-13T23:36:05.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-13T23:36:05.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-13T23:36:05.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-13T23:36:05.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-13T23:36:05.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-13T23:36:05.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-13T23:36:05.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:36:05.796 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:36:05 smithi047 ceph-mon[77147]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-13T23:36:05.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-13T23:36:05.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-13T23:36:05.797 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-13T23:36:05.797 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-13T23:36:05.797 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-13T23:36:05.797 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-13T23:36:05.798 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-13T23:36:05.798 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-13T23:36:05.798 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-13T23:36:05.798 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-13T23:36:05.798 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-13T23:36:05.799 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-13T23:36:05.799 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-13T23:36:05.799 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-13T23:36:05.799 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-13T23:36:05.800 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-13T23:36:05.800 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-13T23:36:05.800 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:36:05.800 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-13T23:36:05.801 INFO:journalctl@ceph.mon.b.smithi047.stdout: During handling of the above exception, another exception occurred: 2022-05-13T23:36:05.801 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-13T23:36:05.801 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-13T23:36:05.801 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-13T23:36:05.802 INFO:journalctl@ceph.mon.b.smithi047.stdout: return f(*arg) 2022-05-13T23:36:05.802 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-13T23:36:05.802 INFO:journalctl@ceph.mon.b.smithi047.stdout: self._write_client_files(client_files, host) 2022-05-13T23:36:05.802 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-13T23:36:05.802 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-13T23:36:05.803 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-13T23:36:05.803 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-13T23:36:05.803 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-13T23:36:05.803 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.event_loop.get_result(coro) 2022-05-13T23:36:05.804 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-13T23:36:05.804 INFO:journalctl@ceph.mon.b.smithi047.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-13T23:36:05.804 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-13T23:36:05.804 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.__get_result() 2022-05-13T23:36:05.805 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-13T23:36:05.805 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise self._exception 2022-05-13T23:36:05.805 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-13T23:36:05.805 INFO:journalctl@ceph.mon.b.smithi047.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-13T23:36:05.806 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-13T23:36:05.806 INFO:journalctl@ceph.mon.b.smithi047.stdout: conn = await self._remote_connection(host, addr) 2022-05-13T23:36:05.806 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-13T23:36:05.806 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise 2022-05-13T23:36:05.807 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-13T23:36:05.807 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.gen.throw(type, value, traceback) 2022-05-13T23:36:05.807 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-13T23:36:05.807 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OrchestratorError(msg) 2022-05-13T23:36:05.807 INFO:journalctl@ceph.mon.b.smithi047.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:36:05.808 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:36:05 smithi047 ceph-mon[77147]: pgmap v601: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:36:05.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:36:05 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:36:05.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:36:05 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:36:05.859 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-13T23:36:05.859 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-13T23:36:05.859 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-13T23:36:05.860 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-13T23:36:05.860 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-13T23:36:05.860 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-13T23:36:05.860 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-13T23:36:05.860 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-13T23:36:05.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-13T23:36:05.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-13T23:36:05.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-13T23:36:05.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-13T23:36:05.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-13T23:36:05.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-13T23:36:05.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-13T23:36:05.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-13T23:36:05.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-13T23:36:05.863 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:36:05.863 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:36:05 smithi115 ceph-mon[78593]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-13T23:36:05.863 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-13T23:36:05.863 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-13T23:36:05.863 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-13T23:36:05.864 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-13T23:36:05.864 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-13T23:36:05.864 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-13T23:36:05.864 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-13T23:36:05.865 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-13T23:36:05.865 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-13T23:36:05.865 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-13T23:36:05.865 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-13T23:36:05.866 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-13T23:36:05.866 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-13T23:36:05.866 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-13T23:36:05.866 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-13T23:36:05.866 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-13T23:36:05.867 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-13T23:36:05.867 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:36:05.867 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-13T23:36:05.867 INFO:journalctl@ceph.mon.c.smithi115.stdout: During handling of the above exception, another exception occurred: 2022-05-13T23:36:05.867 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-13T23:36:05.868 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-13T23:36:05.868 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-13T23:36:05.868 INFO:journalctl@ceph.mon.c.smithi115.stdout: return f(*arg) 2022-05-13T23:36:05.868 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-13T23:36:05.869 INFO:journalctl@ceph.mon.c.smithi115.stdout: self._write_client_files(client_files, host) 2022-05-13T23:36:05.869 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-13T23:36:05.869 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-13T23:36:05.869 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-13T23:36:05.869 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-13T23:36:05.870 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-13T23:36:05.870 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.event_loop.get_result(coro) 2022-05-13T23:36:05.870 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-13T23:36:05.870 INFO:journalctl@ceph.mon.c.smithi115.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-13T23:36:05.871 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-13T23:36:05.871 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.__get_result() 2022-05-13T23:36:05.871 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-13T23:36:05.871 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise self._exception 2022-05-13T23:36:05.871 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-13T23:36:05.872 INFO:journalctl@ceph.mon.c.smithi115.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-13T23:36:05.872 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-13T23:36:05.872 INFO:journalctl@ceph.mon.c.smithi115.stdout: conn = await self._remote_connection(host, addr) 2022-05-13T23:36:05.872 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-13T23:36:05.873 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise 2022-05-13T23:36:05.873 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-13T23:36:05.873 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.gen.throw(type, value, traceback) 2022-05-13T23:36:05.873 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-13T23:36:05.873 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OrchestratorError(msg) 2022-05-13T23:36:05.874 INFO:journalctl@ceph.mon.c.smithi115.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:36:05.874 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:36:05 smithi115 ceph-mon[78593]: pgmap v601: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:36:06.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:36:06 smithi115 conmon[90393]: 2022-05-13T23:36:06.009+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:36:06.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:36:06 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:36:06.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:36:06 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:36:07.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:36:07 smithi115 conmon[90393]: 2022-05-13T23:36:07.024+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:36:07.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:36:07 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:36:07.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:36:07 smithi047 ceph-mon[77147]: pgmap v602: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:36:07.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:36:07 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:36:07.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:36:07 smithi115 ceph-mon[78593]: pgmap v602: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:36:08.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:36:08 smithi115 conmon[90393]: 2022-05-13T23:36:08.058+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:36:08.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:36:08 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:36:08.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:36:08 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 500 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:36:08.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:36:08 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:36:08.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:36:08 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 500 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:36:09.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:36:09 smithi115 conmon[90393]: 2022-05-13T23:36:09.108+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:36:10.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:36:10 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:36:10.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:36:10 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:36:10.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:36:10 smithi047 ceph-mon[77147]: fsmap cephfs:5/5 3 up:standby-replay 1 up:replay 4 up:active 2022-05-13T23:36:10.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:36:10 smithi047 ceph-mon[77147]: pgmap v603: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:36:10.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:36:10 smithi115 conmon[90393]: 2022-05-13T23:36:10.157+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:36:10.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:36:10 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:36:10.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:36:10 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:36:10.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:36:10 smithi115 ceph-mon[78593]: fsmap cephfs:5/5 3 up:standby-replay 1 up:replay 4 up:active 2022-05-13T23:36:10.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:36:10 smithi115 ceph-mon[78593]: pgmap v603: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:36:11.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:36:11 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:36:11.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:36:11 smithi115 conmon[90393]: 2022-05-13T23:36:11.124+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:36:11.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:36:11 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:36:12.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:36:12 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:36:12.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:36:12 smithi047 ceph-mon[77147]: pgmap v604: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:36:12.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:36:12 smithi115 conmon[90393]: 2022-05-13T23:36:12.149+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:36:12.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:36:12 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:36:12.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:36:12 smithi115 ceph-mon[78593]: pgmap v604: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:36:13.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:36:13 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:36:13.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:36:13 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-13T23:36:13.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:36:13 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-13T23:36:13.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:36:13 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 506 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:36:13.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:36:13 smithi115 conmon[90393]: 2022-05-13T23:36:13.186+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:36:13.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:36:13 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:36:13.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:36:13 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-13T23:36:13.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:36:13 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-13T23:36:13.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:36:13 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 506 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:36:14.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:36:14 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:36:14.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:36:14 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:36:14.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:36:14 smithi047 ceph-mon[77147]: fsmap cephfs:5/5 3 up:standby-replay 1 up:replay 4 up:active 2022-05-13T23:36:14.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:36:14 smithi047 ceph-mon[77147]: pgmap v605: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:36:14.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:36:14 smithi115 conmon[90393]: 2022-05-13T23:36:14.225+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:36:14.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:36:14 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:36:14.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:36:14 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:36:14.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:36:14 smithi115 ceph-mon[78593]: fsmap cephfs:5/5 3 up:standby-replay 1 up:replay 4 up:active 2022-05-13T23:36:14.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:36:14 smithi115 ceph-mon[78593]: pgmap v605: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:36:15.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:36:15 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:36:15.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:36:15 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:36:15.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:36:15 smithi115 conmon[90393]: 2022-05-13T23:36:15.247+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:36:16.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:36:16 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:36:16.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:36:16 smithi047 ceph-mon[77147]: pgmap v606: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:36:16.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:36:16 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:36:16.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:36:16 smithi115 ceph-mon[78593]: pgmap v606: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:36:16.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:36:16 smithi115 conmon[90393]: 2022-05-13T23:36:16.250+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:36:17.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:36:17 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:36:17.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:36:17 smithi115 conmon[90393]: 2022-05-13T23:36:17.287+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:36:17.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:36:17 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:36:18.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:36:18 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:36:18.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:36:18 smithi047 ceph-mon[77147]: fsmap cephfs:5/5 3 up:standby-replay 1 up:replay 4 up:active 2022-05-13T23:36:18.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:36:18 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:36:18.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:36:18 smithi047 ceph-mon[77147]: pgmap v607: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:36:18.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:36:18 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 510 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:36:18.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:36:18 smithi115 conmon[90393]: 2022-05-13T23:36:18.330+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:36:18.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:36:18 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:36:18.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:36:18 smithi115 ceph-mon[78593]: fsmap cephfs:5/5 3 up:standby-replay 1 up:replay 4 up:active 2022-05-13T23:36:18.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:36:18 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:36:18.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:36:18 smithi115 ceph-mon[78593]: pgmap v607: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:36:18.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:36:18 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 510 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:36:19.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:36:19 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:36:19.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:36:19 smithi115 conmon[90393]: 2022-05-13T23:36:19.333+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:36:19.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:36:19 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:36:20.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:36:20 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:36:20.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:36:20 smithi047 ceph-mon[77147]: pgmap v608: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:36:20.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:36:20 smithi115 conmon[90393]: 2022-05-13T23:36:20.340+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:36:20.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:36:20 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:36:20.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:36:20 smithi115 ceph-mon[78593]: pgmap v608: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:36:21.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:36:21 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:36:21.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:36:21 smithi115 conmon[90393]: 2022-05-13T23:36:21.307+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:36:21.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:36:21 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:36:22.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:36:22 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:36:22.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:36:22 smithi047 ceph-mon[77147]: fsmap cephfs:5/5 3 up:standby-replay 1 up:replay 4 up:active 2022-05-13T23:36:22.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:36:22 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:36:22.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:36:22 smithi047 ceph-mon[77147]: pgmap v609: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:36:22.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:36:22 smithi115 conmon[90393]: 2022-05-13T23:36:22.273+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:36:22.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:36:22 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:36:22.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:36:22 smithi115 ceph-mon[78593]: fsmap cephfs:5/5 3 up:standby-replay 1 up:replay 4 up:active 2022-05-13T23:36:22.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:36:22 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:36:22.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:36:22 smithi115 ceph-mon[78593]: pgmap v609: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:36:23.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:36:23 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:36:23.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:36:23 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 515 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:36:23.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:36:23 smithi115 conmon[90393]: 2022-05-13T23:36:23.278+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:36:23.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:36:23 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:36:23.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:36:23 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 515 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:36:24.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:36:24 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:36:24.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:36:24 smithi047 ceph-mon[77147]: pgmap v610: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:36:24.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:36:24 smithi115 conmon[90393]: 2022-05-13T23:36:24.245+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:36:24.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:36:24 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:36:24.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:36:24 smithi115 ceph-mon[78593]: pgmap v610: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:36:25.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:36:25 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:36:25.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:36:25 smithi115 conmon[90393]: 2022-05-13T23:36:25.268+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:36:25.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:36:25 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:36:26.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:36:26 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:36:26.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:36:26 smithi047 ceph-mon[77147]: pgmap v611: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:36:26.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:36:26 smithi115 conmon[90393]: 2022-05-13T23:36:26.282+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:36:26.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:36:26 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:36:26.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:36:26 smithi115 ceph-mon[78593]: pgmap v611: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:36:27.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:36:27 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:36:27.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:36:27 smithi115 conmon[90393]: 2022-05-13T23:36:27.298+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:36:27.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:36:27 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:36:28.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:36:28 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:36:28.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:36:28 smithi047 ceph-mon[77147]: pgmap v612: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:36:28.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:36:28 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 520 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:36:28.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:36:28 smithi115 conmon[90393]: 2022-05-13T23:36:28.269+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:36:28.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:36:28 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:36:28.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:36:28 smithi115 ceph-mon[78593]: pgmap v612: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:36:28.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:36:28 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 520 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:36:29.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:36:29 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:36:29.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:36:29 smithi115 conmon[90393]: 2022-05-13T23:36:29.262+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:36:29.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:36:29 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:36:30.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:36:30 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:36:30.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:36:30 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:36:30.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:36:30 smithi047 ceph-mon[77147]: fsmap cephfs:5/5 3 up:standby-replay 1 up:replay 4 up:active 2022-05-13T23:36:30.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:36:30 smithi047 ceph-mon[77147]: pgmap v613: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:36:30.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:36:30 smithi115 conmon[90393]: 2022-05-13T23:36:30.304+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:36:30.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:36:30 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:36:30.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:36:30 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:36:30.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:36:30 smithi115 ceph-mon[78593]: fsmap cephfs:5/5 3 up:standby-replay 1 up:replay 4 up:active 2022-05-13T23:36:30.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:36:30 smithi115 ceph-mon[78593]: pgmap v613: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:36:31.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:36:31 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:36:31.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:36:31 smithi115 conmon[90393]: 2022-05-13T23:36:31.334+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:36:31.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:36:31 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:36:32.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:36:32 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:36:32.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:36:32 smithi047 ceph-mon[77147]: pgmap v614: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:36:32.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:36:32 smithi115 conmon[90393]: 2022-05-13T23:36:32.299+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:36:32.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:36:32 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:36:32.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:36:32 smithi115 ceph-mon[78593]: pgmap v614: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:36:33.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:36:33 smithi115 conmon[90393]: 2022-05-13T23:36:33.267+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:36:33.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:36:33 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:36:33.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:36:33 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 525 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:36:33.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:36:33 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:36:33.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:36:33 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 525 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:36:34.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:36:34 smithi115 conmon[90393]: 2022-05-13T23:36:34.264+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:36:34.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:36:34 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:36:34.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:36:34 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:36:34.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:36:34 smithi115 ceph-mon[78593]: fsmap cephfs:5/5 3 up:standby-replay 1 up:replay 4 up:active 2022-05-13T23:36:34.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:36:34 smithi115 ceph-mon[78593]: pgmap v615: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:36:34.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:36:34 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:36:34.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:36:34 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:36:34.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:36:34 smithi047 ceph-mon[77147]: fsmap cephfs:5/5 3 up:standby-replay 1 up:replay 4 up:active 2022-05-13T23:36:34.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:36:34 smithi047 ceph-mon[77147]: pgmap v615: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:36:35.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:36:35 smithi115 conmon[90393]: 2022-05-13T23:36:35.248+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:36:35.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:36:35 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:36:35.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:36:35 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:36:36.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:36:36 smithi115 conmon[90393]: 2022-05-13T23:36:36.282+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:36:36.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:36:36 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:36:36.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:36:36 smithi115 ceph-mon[78593]: pgmap v616: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:36:36.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:36:36 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:36:36.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:36:36 smithi047 ceph-mon[77147]: pgmap v616: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:36:37.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:36:37 smithi115 conmon[90393]: 2022-05-13T23:36:37.259+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:36:37.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:36:37 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:36:37.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:36:37 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:36:38.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:36:38 smithi115 conmon[90393]: 2022-05-13T23:36:38.307+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:36:38.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:36:38 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:36:38.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:36:38 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:36:38.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:36:38 smithi115 ceph-mon[78593]: fsmap cephfs:5/5 3 up:standby-replay 1 up:replay 4 up:active 2022-05-13T23:36:38.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:36:38 smithi115 ceph-mon[78593]: pgmap v617: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:36:38.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:36:38 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 530 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:36:38.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:36:38 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:36:38.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:36:38 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:36:38.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:36:38 smithi047 ceph-mon[77147]: fsmap cephfs:5/5 3 up:standby-replay 1 up:replay 4 up:active 2022-05-13T23:36:38.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:36:38 smithi047 ceph-mon[77147]: pgmap v617: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:36:38.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:36:38 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 530 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:36:39.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:36:39 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:36:39.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:36:39 smithi115 conmon[90393]: 2022-05-13T23:36:39.319+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:36:39.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:36:39 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:36:40.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:36:40 smithi115 conmon[90393]: 2022-05-13T23:36:40.306+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:36:40.612 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:36:40 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:36:40.613 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:36:40 smithi115 ceph-mon[78593]: pgmap v618: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:36:40.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:36:40 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:36:40.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:36:40 smithi047 ceph-mon[77147]: pgmap v618: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:36:41.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:36:41 smithi115 conmon[90393]: 2022-05-13T23:36:41.298+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:36:41.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:36:41 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:36:41.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:36:41 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:36:42.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:36:42 smithi115 conmon[90393]: 2022-05-13T23:36:42.296+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:36:42.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:36:42 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:36:42.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:36:42 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:36:42.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:36:42 smithi115 ceph-mon[78593]: fsmap cephfs:5/5 3 up:standby-replay 1 up:replay 4 up:active 2022-05-13T23:36:42.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:36:42 smithi115 ceph-mon[78593]: pgmap v619: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:36:42.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:36:42 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:36:42.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:36:42 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:36:42.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:36:42 smithi047 ceph-mon[77147]: fsmap cephfs:5/5 3 up:standby-replay 1 up:replay 4 up:active 2022-05-13T23:36:42.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:36:42 smithi047 ceph-mon[77147]: pgmap v619: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:36:43.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:36:43 smithi115 conmon[90393]: 2022-05-13T23:36:43.281+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:36:43.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:36:43 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:36:43.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:36:43 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 535 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:36:43.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:36:43 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:36:43.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:36:43 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 535 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:36:44.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:36:44 smithi115 conmon[90393]: 2022-05-13T23:36:44.290+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:36:44.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:36:44 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:36:44.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:36:44 smithi115 ceph-mon[78593]: pgmap v620: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:36:44.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:36:44 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:36:44.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:36:44 smithi047 ceph-mon[77147]: pgmap v620: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:36:45.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:36:45 smithi115 conmon[90393]: 2022-05-13T23:36:45.330+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:36:45.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:36:45 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:36:45.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:36:45 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:36:46.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:36:46 smithi115 conmon[90393]: 2022-05-13T23:36:46.367+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:36:46.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:36:46 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:36:46.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:36:46 smithi115 ceph-mon[78593]: pgmap v621: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:36:46.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:36:46 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:36:46.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:36:46 smithi047 ceph-mon[77147]: pgmap v621: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:36:47.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:36:47 smithi115 conmon[90393]: 2022-05-13T23:36:47.337+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:36:47.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:36:47 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:36:47.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:36:47 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:36:48.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:36:48 smithi115 conmon[90393]: 2022-05-13T23:36:48.345+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:36:48.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:36:48 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:36:48.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:36:48 smithi115 ceph-mon[78593]: pgmap v622: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:36:48.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:36:48 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 540 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:36:48.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:36:48 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:36:48.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:36:48 smithi047 ceph-mon[77147]: pgmap v622: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:36:48.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:36:48 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 540 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:36:49.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:36:49 smithi115 conmon[90393]: 2022-05-13T23:36:49.340+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:36:49.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:36:49 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:36:49.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:36:49 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:36:49.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:36:49 smithi115 ceph-mon[78593]: fsmap cephfs:5/5 3 up:standby-replay 1 up:replay 4 up:active 2022-05-13T23:36:49.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:36:49 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:36:49.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:36:49 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:36:49.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:36:49 smithi047 ceph-mon[77147]: fsmap cephfs:5/5 3 up:standby-replay 1 up:replay 4 up:active 2022-05-13T23:36:50.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:36:50 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:36:50.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:36:50 smithi047 ceph-mon[77147]: pgmap v623: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:36:50.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:36:50 smithi115 conmon[90393]: 2022-05-13T23:36:50.364+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:36:50.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:36:50 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:36:50.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:36:50 smithi115 ceph-mon[78593]: pgmap v623: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:36:51.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:36:51 smithi115 conmon[90393]: 2022-05-13T23:36:51.318+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:36:51.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:36:51 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:36:51.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:36:51 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:36:52.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:36:52 smithi115 conmon[90393]: 2022-05-13T23:36:52.326+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:36:52.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:36:52 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:36:52.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:36:52 smithi115 ceph-mon[78593]: pgmap v624: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:36:52.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:36:52 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:36:52.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:36:52 smithi047 ceph-mon[77147]: pgmap v624: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:36:53.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:36:53 smithi115 conmon[90393]: 2022-05-13T23:36:53.331+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:36:53.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:36:53 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:36:53.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:36:53 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:36:53.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:36:53 smithi115 ceph-mon[78593]: fsmap cephfs:5/5 3 up:standby-replay 1 up:replay 4 up:active 2022-05-13T23:36:53.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:36:53 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 545 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:36:53.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:36:53 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:36:53.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:36:53 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:36:53.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:36:53 smithi047 ceph-mon[77147]: fsmap cephfs:5/5 3 up:standby-replay 1 up:replay 4 up:active 2022-05-13T23:36:53.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:36:53 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 545 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:36:54.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:36:54 smithi115 conmon[90393]: 2022-05-13T23:36:54.324+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:36:54.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:36:54 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:36:54.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:36:54 smithi115 ceph-mon[78593]: pgmap v625: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:36:54.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:36:54 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:36:54.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:36:54 smithi047 ceph-mon[77147]: pgmap v625: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:36:55.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:36:55 smithi115 conmon[90393]: 2022-05-13T23:36:55.282+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:36:55.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:36:55 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:36:55.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:36:55 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:36:56.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:36:56 smithi115 conmon[90393]: 2022-05-13T23:36:56.298+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:36:56.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:36:56 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:36:56.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:36:56 smithi115 ceph-mon[78593]: pgmap v626: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:36:56.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:36:56 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:36:56.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:36:56 smithi047 ceph-mon[77147]: pgmap v626: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:36:57.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:36:57 smithi115 conmon[90393]: 2022-05-13T23:36:57.272+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:36:57.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:36:57 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:36:57.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:36:57 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:36:57.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:36:57 smithi115 ceph-mon[78593]: fsmap cephfs:5/5 3 up:standby-replay 1 up:replay 4 up:active 2022-05-13T23:36:57.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:36:57 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:36:57.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:36:57 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:36:57.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:36:57 smithi047 ceph-mon[77147]: fsmap cephfs:5/5 3 up:standby-replay 1 up:replay 4 up:active 2022-05-13T23:36:58.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:36:58 smithi115 conmon[90393]: 2022-05-13T23:36:58.240+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:36:58.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:36:58 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:36:58.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:36:58 smithi115 ceph-mon[78593]: pgmap v627: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:36:58.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:36:58 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 550 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:36:58.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:36:58 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:36:58.795 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:36:58 smithi047 ceph-mon[77147]: pgmap v627: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:36:58.795 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:36:58 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 550 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:36:59.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:36:59 smithi115 conmon[90393]: 2022-05-13T23:36:59.228+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:36:59.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:36:59 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:36:59.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:36:59 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:37:00.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:37:00 smithi115 conmon[90393]: 2022-05-13T23:37:00.223+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:37:00.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:37:00 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:37:00.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:37:00 smithi115 ceph-mon[78593]: pgmap v628: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:37:00.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:37:00 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:37:00.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:37:00 smithi047 ceph-mon[77147]: pgmap v628: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:37:01.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:37:01 smithi115 conmon[90393]: 2022-05-13T23:37:01.262+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:37:01.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:37:01 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:37:01.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:37:01 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:37:01.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:37:01 smithi115 ceph-mon[78593]: fsmap cephfs:5/5 3 up:standby-replay 1 up:replay 4 up:active 2022-05-13T23:37:01.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:37:01 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:37:01.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:37:01 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:37:01.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:37:01 smithi047 ceph-mon[77147]: fsmap cephfs:5/5 3 up:standby-replay 1 up:replay 4 up:active 2022-05-13T23:37:02.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:37:02 smithi115 conmon[90393]: 2022-05-13T23:37:02.259+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:37:02.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:37:02 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:37:02.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:37:02 smithi115 ceph-mon[78593]: pgmap v629: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:37:02.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:37:02 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:37:02.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:37:02 smithi047 ceph-mon[77147]: pgmap v629: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:37:03.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:37:03 smithi115 conmon[90393]: 2022-05-13T23:37:03.243+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:37:03.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:37:03 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:37:03.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:37:03 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 555 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:37:03.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:37:03 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:37:03.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:37:03 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 555 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:37:04.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:37:04 smithi115 conmon[90393]: 2022-05-13T23:37:04.194+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:37:04.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:37:04 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:37:04.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:37:04 smithi115 ceph-mon[78593]: pgmap v630: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:37:04.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:37:04 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:37:04.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:37:04 smithi047 ceph-mon[77147]: pgmap v630: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:37:05.410 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:37:05 smithi115 conmon[90393]: 2022-05-13T23:37:05.148+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:37:05.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:37:05 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:37:05.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:37:05 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-13T23:37:05.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:37:05 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-13T23:37:05.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:37:05 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-13T23:37:05.683 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:37:05 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:37:05.683 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:37:05 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-13T23:37:05.684 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:37:05 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-13T23:37:05.684 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:37:05 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-13T23:37:06.042 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:37:05 smithi047 conmon[78521]: 2022-05-13T23:37:05.648+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:37:06.042 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:37:05 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-13T23:37:06.042 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:37:05 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-13T23:37:06.042 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:37:05 smithi047 conmon[78521]: yield 2022-05-13T23:37:06.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:37:05 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-13T23:37:06.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:37:05 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-13T23:37:06.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:37:05 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-13T23:37:06.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:37:05 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-13T23:37:06.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:37:05 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-13T23:37:06.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:37:05 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-13T23:37:06.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:37:05 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-13T23:37:06.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:37:05 smithi047 conmon[78521]: raise exceptions[0] 2022-05-13T23:37:06.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:37:05 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-13T23:37:06.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:37:05 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-13T23:37:06.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:37:05 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-13T23:37:06.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:37:05 smithi047 conmon[78521]: return (yield from fut) 2022-05-13T23:37:06.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:37:05 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-13T23:37:06.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:37:05 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-13T23:37:06.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:37:05 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:37:06.415 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:37:06 smithi115 conmon[90393]: 2022-05-13T23:37:06.132+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:37:06.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:37:06 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:37:06.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:37:06 smithi047 ceph-mon[77147]: pgmap v631: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:37:06.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:37:06 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:37:06.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:37:06 smithi115 ceph-mon[78593]: pgmap v631: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:37:07.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:37:07 smithi115 conmon[90393]: 2022-05-13T23:37:07.101+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:37:07.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:37:07 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:37:07.791 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-13T23:37:07.792 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-13T23:37:07.792 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-13T23:37:07.792 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-13T23:37:07.793 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-13T23:37:07.793 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-13T23:37:07.793 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-13T23:37:07.793 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-13T23:37:07.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-13T23:37:07.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-13T23:37:07.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-13T23:37:07.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-13T23:37:07.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-13T23:37:07.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-13T23:37:07.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-13T23:37:07.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-13T23:37:07.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-13T23:37:07.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:37:07.796 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:37:07 smithi047 ceph-mon[77147]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-13T23:37:07.797 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:37:07 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:37:07.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:37:07 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:37:07.858 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-13T23:37:07.858 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-13T23:37:07.859 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-13T23:37:07.859 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-13T23:37:07.860 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-13T23:37:07.860 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-13T23:37:07.860 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-13T23:37:07.860 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-13T23:37:07.860 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-13T23:37:07.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-13T23:37:07.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-13T23:37:07.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-13T23:37:07.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-13T23:37:07.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-13T23:37:07.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-13T23:37:07.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-13T23:37:07.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-13T23:37:07.863 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:37:07.863 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:37:07 smithi115 ceph-mon[78593]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-13T23:37:07.863 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:37:07 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:37:08.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:37:08 smithi115 conmon[90393]: 2022-05-13T23:37:08.103+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:37:08.721 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:37:08 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:37:08.722 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:37:08 smithi047 ceph-mon[77147]: pgmap v632: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:37:08.722 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:37:08 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 560 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:37:08.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:37:08 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:37:08.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:37:08 smithi115 ceph-mon[78593]: pgmap v632: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:37:08.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:37:08 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 560 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:37:09.042 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:37:08 smithi047 conmon[78521]: 2022-05-13T23:37:08.720+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:37:09.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:37:08 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-13T23:37:09.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:37:08 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-13T23:37:09.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:37:08 smithi047 conmon[78521]: yield 2022-05-13T23:37:09.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:37:08 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-13T23:37:09.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:37:08 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-13T23:37:09.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:37:08 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-13T23:37:09.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:37:08 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-13T23:37:09.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:37:08 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-13T23:37:09.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:37:08 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-13T23:37:09.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:37:08 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-13T23:37:09.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:37:08 smithi047 conmon[78521]: raise exceptions[0] 2022-05-13T23:37:09.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:37:08 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-13T23:37:09.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:37:08 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-13T23:37:09.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:37:08 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-13T23:37:09.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:37:08 smithi047 conmon[78521]: return (yield from fut) 2022-05-13T23:37:09.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:37:08 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-13T23:37:09.048 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:37:08 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-13T23:37:09.048 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:37:08 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:37:09.048 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:37:08 smithi047 conmon[78521]: 2022-05-13T23:37:08.721+0000 7f5340779700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-13T23:37:09.048 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:37:08 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-13T23:37:09.048 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:37:08 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-13T23:37:09.049 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:37:08 smithi047 conmon[78521]: yield 2022-05-13T23:37:09.049 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:37:08 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-13T23:37:09.049 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:37:08 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-13T23:37:09.049 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:37:08 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-13T23:37:09.050 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:37:08 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-13T23:37:09.050 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:37:08 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-13T23:37:09.050 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:37:08 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-13T23:37:09.050 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:37:08 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-13T23:37:09.051 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:37:08 smithi047 conmon[78521]: raise exceptions[0] 2022-05-13T23:37:09.051 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:37:08 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-13T23:37:09.051 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:37:08 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-13T23:37:09.051 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:37:08 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-13T23:37:09.051 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:37:08 smithi047 conmon[78521]: return (yield from fut) 2022-05-13T23:37:09.052 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:37:08 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-13T23:37:09.052 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:37:08 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-13T23:37:09.052 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:37:08 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:37:09.052 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:37:08 smithi047 conmon[78521]: 2022-05-13T23:37:09.053 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:37:08 smithi047 conmon[78521]: During handling of the above exception, another exception occurred: 2022-05-13T23:37:09.053 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:37:08 smithi047 conmon[78521]: 2022-05-13T23:37:09.053 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:37:08 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-13T23:37:09.053 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:37:08 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-13T23:37:09.054 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:37:08 smithi047 conmon[78521]: return f(*arg) 2022-05-13T23:37:09.054 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:37:08 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-13T23:37:09.054 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:37:08 smithi047 conmon[78521]: self._write_client_files(client_files, host) 2022-05-13T23:37:09.054 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:37:08 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-13T23:37:09.055 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:37:08 smithi047 conmon[78521]: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-13T23:37:09.055 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:37:08 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-13T23:37:09.055 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:37:08 smithi047 conmon[78521]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-13T23:37:09.055 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:37:08 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-13T23:37:09.055 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:37:08 smithi047 conmon[78521]: return self.event_loop.get_result(coro) 2022-05-13T23:37:09.056 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:37:08 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-13T23:37:09.056 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:37:08 smithi047 conmon[78521]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-13T23:37:09.056 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:37:08 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-13T23:37:09.056 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:37:08 smithi047 conmon[78521]: return self.__get_result() 2022-05-13T23:37:09.057 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:37:08 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-13T23:37:09.057 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:37:08 smithi047 conmon[78521]: raise self._exception 2022-05-13T23:37:09.057 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:37:08 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-13T23:37:09.057 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:37:08 smithi047 conmon[78521]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-13T23:37:09.058 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:37:08 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-13T23:37:09.058 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:37:08 smithi047 conmon[78521]: conn = await self._remote_connection(host, addr) 2022-05-13T23:37:09.058 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:37:08 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-13T23:37:09.058 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:37:08 smithi047 conmon[78521]: raise 2022-05-13T23:37:09.058 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:37:08 smithi047 conmon[78521]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-13T23:37:09.059 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:37:08 smithi047 conmon[78521]: self.gen.throw(type, value, traceback) 2022-05-13T23:37:09.059 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:37:08 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-13T23:37:09.059 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:37:08 smithi047 conmon[78521]: raise OrchestratorError(msg) 2022-05-13T23:37:09.059 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:37:08 smithi047 conmon[78521]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:37:09.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:37:09 smithi115 conmon[90393]: 2022-05-13T23:37:09.053+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:37:09.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:37:09 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:37:09.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:37:09 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:37:09.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:37:09 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:37:09.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:37:09 smithi047 ceph-mon[77147]: fsmap cephfs:5/5 3 up:standby-replay 1 up:replay 4 up:active 2022-05-13T23:37:09.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:37:09 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:37:09.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:37:09 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:37:09.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:37:09 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:37:09.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:37:09 smithi115 ceph-mon[78593]: fsmap cephfs:5/5 3 up:standby-replay 1 up:replay 4 up:active 2022-05-13T23:37:10.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:37:10 smithi115 conmon[90393]: 2022-05-13T23:37:10.008+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:37:10.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:37:10 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:37:10.792 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-13T23:37:10.793 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-13T23:37:10.793 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-13T23:37:10.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-13T23:37:10.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-13T23:37:10.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-13T23:37:10.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-13T23:37:10.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-13T23:37:10.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-13T23:37:10.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-13T23:37:10.797 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-13T23:37:10.797 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-13T23:37:10.797 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-13T23:37:10.798 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-13T23:37:10.798 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-13T23:37:10.799 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-13T23:37:10.799 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-13T23:37:10.800 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:37:10.800 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:37:10 smithi047 ceph-mon[77147]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-13T23:37:10.800 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-13T23:37:10.801 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-13T23:37:10.801 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-13T23:37:10.802 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-13T23:37:10.802 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-13T23:37:10.803 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-13T23:37:10.803 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-13T23:37:10.803 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-13T23:37:10.804 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-13T23:37:10.804 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-13T23:37:10.805 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-13T23:37:10.805 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-13T23:37:10.806 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-13T23:37:10.806 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-13T23:37:10.806 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-13T23:37:10.807 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-13T23:37:10.807 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-13T23:37:10.807 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:37:10.808 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-13T23:37:10.808 INFO:journalctl@ceph.mon.b.smithi047.stdout: During handling of the above exception, another exception occurred: 2022-05-13T23:37:10.809 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-13T23:37:10.809 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-13T23:37:10.809 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-13T23:37:10.810 INFO:journalctl@ceph.mon.b.smithi047.stdout: return f(*arg) 2022-05-13T23:37:10.810 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-13T23:37:10.811 INFO:journalctl@ceph.mon.b.smithi047.stdout: self._write_client_files(client_files, host) 2022-05-13T23:37:10.811 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-13T23:37:10.811 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-13T23:37:10.812 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-13T23:37:10.812 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-13T23:37:10.812 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-13T23:37:10.813 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.event_loop.get_result(coro) 2022-05-13T23:37:10.813 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-13T23:37:10.814 INFO:journalctl@ceph.mon.b.smithi047.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-13T23:37:10.814 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-13T23:37:10.815 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.__get_result() 2022-05-13T23:37:10.815 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-13T23:37:10.815 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise self._exception 2022-05-13T23:37:10.816 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-13T23:37:10.816 INFO:journalctl@ceph.mon.b.smithi047.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-13T23:37:10.816 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-13T23:37:10.817 INFO:journalctl@ceph.mon.b.smithi047.stdout: conn = await self._remote_connection(host, addr) 2022-05-13T23:37:10.817 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-13T23:37:10.817 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise 2022-05-13T23:37:10.818 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-13T23:37:10.818 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.gen.throw(type, value, traceback) 2022-05-13T23:37:10.818 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-13T23:37:10.818 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OrchestratorError(msg) 2022-05-13T23:37:10.819 INFO:journalctl@ceph.mon.b.smithi047.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:37:10.819 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:37:10 smithi047 ceph-mon[77147]: pgmap v633: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:37:10.819 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:37:10 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:37:10.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:37:10 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:37:10.858 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-13T23:37:10.858 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-13T23:37:10.859 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-13T23:37:10.859 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-13T23:37:10.859 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-13T23:37:10.859 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-13T23:37:10.860 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-13T23:37:10.860 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-13T23:37:10.860 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-13T23:37:10.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-13T23:37:10.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-13T23:37:10.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-13T23:37:10.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-13T23:37:10.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-13T23:37:10.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-13T23:37:10.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-13T23:37:10.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-13T23:37:10.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:37:10.863 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:37:10 smithi115 ceph-mon[78593]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-13T23:37:10.863 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-13T23:37:10.863 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-13T23:37:10.863 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-13T23:37:10.864 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-13T23:37:10.864 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-13T23:37:10.864 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-13T23:37:10.864 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-13T23:37:10.864 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-13T23:37:10.865 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-13T23:37:10.865 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-13T23:37:10.865 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-13T23:37:10.865 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-13T23:37:10.866 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-13T23:37:10.866 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-13T23:37:10.866 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-13T23:37:10.866 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-13T23:37:10.866 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-13T23:37:10.867 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:37:10.867 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-13T23:37:10.867 INFO:journalctl@ceph.mon.c.smithi115.stdout: During handling of the above exception, another exception occurred: 2022-05-13T23:37:10.867 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-13T23:37:10.867 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-13T23:37:10.868 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-13T23:37:10.868 INFO:journalctl@ceph.mon.c.smithi115.stdout: return f(*arg) 2022-05-13T23:37:10.868 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-13T23:37:10.868 INFO:journalctl@ceph.mon.c.smithi115.stdout: self._write_client_files(client_files, host) 2022-05-13T23:37:10.868 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-13T23:37:10.869 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-13T23:37:10.869 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-13T23:37:10.869 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-13T23:37:10.869 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-13T23:37:10.870 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.event_loop.get_result(coro) 2022-05-13T23:37:10.870 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-13T23:37:10.870 INFO:journalctl@ceph.mon.c.smithi115.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-13T23:37:10.870 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-13T23:37:10.870 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.__get_result() 2022-05-13T23:37:10.871 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-13T23:37:10.871 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise self._exception 2022-05-13T23:37:10.871 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-13T23:37:10.871 INFO:journalctl@ceph.mon.c.smithi115.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-13T23:37:10.871 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-13T23:37:10.872 INFO:journalctl@ceph.mon.c.smithi115.stdout: conn = await self._remote_connection(host, addr) 2022-05-13T23:37:10.872 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-13T23:37:10.872 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise 2022-05-13T23:37:10.872 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-13T23:37:10.872 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.gen.throw(type, value, traceback) 2022-05-13T23:37:10.873 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-13T23:37:10.873 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OrchestratorError(msg) 2022-05-13T23:37:10.873 INFO:journalctl@ceph.mon.c.smithi115.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:37:10.873 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:37:10 smithi115 ceph-mon[78593]: pgmap v633: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:37:10.874 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:37:10 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:37:11.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:37:11 smithi115 conmon[90393]: 2022-05-13T23:37:11.051+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:37:11.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:37:11 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:37:11.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:37:11 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:37:12.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:37:12 smithi115 conmon[90393]: 2022-05-13T23:37:12.011+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:37:12.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:37:12 smithi047 ceph-mon[77147]: pgmap v634: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:37:12.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:37:12 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:37:12.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:37:12 smithi115 ceph-mon[78593]: pgmap v634: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:37:12.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:37:12 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:37:13.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:37:12 smithi115 conmon[90393]: 2022-05-13T23:37:12.990+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:37:13.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:37:13 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:37:13.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:37:13 smithi047 ceph-mon[77147]: fsmap cephfs:5/5 3 up:standby-replay 1 up:replay 4 up:active 2022-05-13T23:37:13.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:37:13 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:37:13.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:37:13 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-13T23:37:13.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:37:13 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-13T23:37:13.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:37:13 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 565 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:37:13.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:37:13 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:37:13.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:37:13 smithi115 ceph-mon[78593]: fsmap cephfs:5/5 3 up:standby-replay 1 up:replay 4 up:active 2022-05-13T23:37:13.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:37:13 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:37:13.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:37:13 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-13T23:37:13.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:37:13 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-13T23:37:13.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:37:13 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 565 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:37:14.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:37:14 smithi115 conmon[90393]: 2022-05-13T23:37:14.000+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:37:14.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:37:14 smithi047 ceph-mon[77147]: pgmap v635: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:37:14.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:37:14 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:37:14.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:37:14 smithi115 ceph-mon[78593]: pgmap v635: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:37:14.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:37:14 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:37:15.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:37:15 smithi115 conmon[90393]: 2022-05-13T23:37:15.005+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:37:15.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:37:15 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:37:15.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:37:15 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:37:16.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:37:15 smithi115 conmon[90393]: 2022-05-13T23:37:15.988+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:37:16.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:37:16 smithi047 ceph-mon[77147]: pgmap v636: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:37:16.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:37:16 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:37:16.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:37:16 smithi115 ceph-mon[78593]: pgmap v636: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:37:16.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:37:16 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:37:17.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:37:16 smithi115 conmon[90393]: 2022-05-13T23:37:16.990+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:37:17.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:37:17 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:37:17.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:37:17 smithi047 ceph-mon[77147]: fsmap cephfs:5/5 3 up:standby-replay 1 up:replay 4 up:active 2022-05-13T23:37:17.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:37:17 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:37:17.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:37:17 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:37:17.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:37:17 smithi115 ceph-mon[78593]: fsmap cephfs:5/5 3 up:standby-replay 1 up:replay 4 up:active 2022-05-13T23:37:17.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:37:17 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:37:18.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:37:17 smithi115 conmon[90393]: 2022-05-13T23:37:17.947+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:37:18.795 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:37:18 smithi047 ceph-mon[77147]: pgmap v637: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:37:18.795 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:37:18 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:37:18.796 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:37:18 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 570 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:37:18.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:37:18 smithi115 ceph-mon[78593]: pgmap v637: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:37:18.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:37:18 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:37:18.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:37:18 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 570 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:37:19.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:37:18 smithi115 conmon[90393]: 2022-05-13T23:37:18.939+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:37:19.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:37:19 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:37:19.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:37:19 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:37:20.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:37:19 smithi115 conmon[90393]: 2022-05-13T23:37:19.910+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:37:20.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:37:20 smithi047 ceph-mon[77147]: pgmap v638: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:37:20.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:37:20 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:37:20.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:37:20 smithi115 ceph-mon[78593]: pgmap v638: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:37:20.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:37:20 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:37:21.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:37:20 smithi115 conmon[90393]: 2022-05-13T23:37:20.879+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:37:21.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:37:21 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:37:21.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:37:21 smithi047 ceph-mon[77147]: fsmap cephfs:5/5 3 up:standby-replay 1 up:replay 4 up:active 2022-05-13T23:37:21.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:37:21 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:37:21.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:37:21 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:37:21.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:37:21 smithi115 ceph-mon[78593]: fsmap cephfs:5/5 3 up:standby-replay 1 up:replay 4 up:active 2022-05-13T23:37:21.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:37:21 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:37:22.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:37:21 smithi115 conmon[90393]: 2022-05-13T23:37:21.900+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:37:22.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:37:22 smithi047 ceph-mon[77147]: pgmap v639: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:37:22.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:37:22 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:37:22.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:37:22 smithi115 ceph-mon[78593]: pgmap v639: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:37:22.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:37:22 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:37:23.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:37:22 smithi115 conmon[90393]: 2022-05-13T23:37:22.862+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:37:23.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:37:23 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:37:23.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:37:23 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 575 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:37:23.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:37:23 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:37:23.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:37:23 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 575 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:37:24.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:37:23 smithi115 conmon[90393]: 2022-05-13T23:37:23.902+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:37:24.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:37:24 smithi047 ceph-mon[77147]: pgmap v640: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:37:24.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:37:24 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:37:24.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:37:24 smithi115 ceph-mon[78593]: pgmap v640: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:37:24.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:37:24 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:37:25.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:37:24 smithi115 conmon[90393]: 2022-05-13T23:37:24.951+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:37:25.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:37:25 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:37:25.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:37:25 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:37:26.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:37:25 smithi115 conmon[90393]: 2022-05-13T23:37:25.919+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:37:26.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:37:26 smithi047 ceph-mon[77147]: pgmap v641: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:37:26.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:37:26 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:37:26.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:37:26 smithi115 ceph-mon[78593]: pgmap v641: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:37:26.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:37:26 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:37:27.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:37:26 smithi115 conmon[90393]: 2022-05-13T23:37:26.902+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:37:27.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:37:27 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:37:27.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:37:27 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:37:28.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:37:27 smithi115 conmon[90393]: 2022-05-13T23:37:27.931+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:37:28.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:37:28 smithi047 ceph-mon[77147]: pgmap v642: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:37:28.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:37:28 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:37:28.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:37:28 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 580 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:37:28.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:37:28 smithi115 ceph-mon[78593]: pgmap v642: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:37:28.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:37:28 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:37:28.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:37:28 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 580 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:37:29.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:37:28 smithi115 conmon[90393]: 2022-05-13T23:37:28.952+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:37:29.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:37:29 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:37:29.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:37:29 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:37:29.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:37:29 smithi047 ceph-mon[77147]: fsmap cephfs:5/5 3 up:standby-replay 1 up:replay 4 up:active 2022-05-13T23:37:29.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:37:29 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:37:29.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:37:29 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:37:29.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:37:29 smithi115 ceph-mon[78593]: fsmap cephfs:5/5 3 up:standby-replay 1 up:replay 4 up:active 2022-05-13T23:37:30.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:37:29 smithi115 conmon[90393]: 2022-05-13T23:37:29.916+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:37:30.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:37:30 smithi047 ceph-mon[77147]: pgmap v643: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:37:30.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:37:30 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:37:30.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:37:30 smithi115 ceph-mon[78593]: pgmap v643: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:37:30.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:37:30 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:37:31.364 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:37:30 smithi115 conmon[90393]: 2022-05-13T23:37:30.887+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:37:31.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:37:31 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:37:31.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:37:31 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:37:32.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:37:31 smithi115 conmon[90393]: 2022-05-13T23:37:31.859+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:37:32.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:37:32 smithi047 ceph-mon[77147]: pgmap v644: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:37:32.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:37:32 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:37:32.817 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:37:32 smithi115 ceph-mon[78593]: pgmap v644: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:37:32.818 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:37:32 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:37:33.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:37:32 smithi115 conmon[90393]: 2022-05-13T23:37:32.815+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:37:33.770 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:37:33 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:37:33.771 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:37:33 smithi115 ceph-mon[78593]: fsmap cephfs:5/5 3 up:standby-replay 1 up:replay 4 up:active 2022-05-13T23:37:33.771 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:37:33 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:37:33.771 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:37:33 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 585 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:37:33.771 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:37:33 smithi115 ceph-mon[78593]: pgmap v645: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:37:33.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:37:33 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:37:33.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:37:33 smithi047 ceph-mon[77147]: fsmap cephfs:5/5 3 up:standby-replay 1 up:replay 4 up:active 2022-05-13T23:37:33.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:37:33 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:37:33.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:37:33 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 585 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:37:33.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:37:33 smithi047 ceph-mon[77147]: pgmap v645: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:37:34.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:37:33 smithi115 conmon[90393]: 2022-05-13T23:37:33.769+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:37:34.781 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:37:34 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:37:34.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:37:34 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:37:35.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:37:34 smithi115 conmon[90393]: 2022-05-13T23:37:34.780+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:37:35.770 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:37:35 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:37:35.770 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:37:35 smithi115 ceph-mon[78593]: pgmap v646: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:37:35.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:37:35 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:37:35.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:37:35 smithi047 ceph-mon[77147]: pgmap v646: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:37:36.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:37:35 smithi115 conmon[90393]: 2022-05-13T23:37:35.769+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:37:36.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:37:36 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:37:36.816 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:37:36 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:37:37.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:37:36 smithi115 conmon[90393]: 2022-05-13T23:37:36.816+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:37:37.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:37:37 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:37:37.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:37:37 smithi047 ceph-mon[77147]: fsmap cephfs:5/5 3 up:standby-replay 1 up:replay 4 up:active 2022-05-13T23:37:37.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:37:37 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:37:37.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:37:37 smithi047 ceph-mon[77147]: pgmap v647: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:37:37.822 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:37:37 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:37:37.822 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:37:37 smithi115 ceph-mon[78593]: fsmap cephfs:5/5 3 up:standby-replay 1 up:replay 4 up:active 2022-05-13T23:37:37.823 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:37:37 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:37:37.823 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:37:37 smithi115 ceph-mon[78593]: pgmap v647: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:37:38.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:37:37 smithi115 conmon[90393]: 2022-05-13T23:37:37.820+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:37:38.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:37:38 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:37:38.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:37:38 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 590 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:37:38.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:37:38 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:37:38.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:37:38 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 590 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:37:39.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:37:38 smithi115 conmon[90393]: 2022-05-13T23:37:38.860+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:37:39.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:37:39 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:37:39.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:37:39 smithi047 ceph-mon[77147]: pgmap v648: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:37:39.820 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:37:39 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:37:39.820 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:37:39 smithi115 ceph-mon[78593]: pgmap v648: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:37:40.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:37:39 smithi115 conmon[90393]: 2022-05-13T23:37:39.819+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:37:40.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:37:40 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:37:40.851 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:37:40 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:37:41.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:37:40 smithi115 conmon[90393]: 2022-05-13T23:37:40.850+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:37:41.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:37:41 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:37:41.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:37:41 smithi115 ceph-mon[78593]: fsmap cephfs:5/5 3 up:standby-replay 1 up:replay 4 up:active 2022-05-13T23:37:41.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:37:41 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:37:41.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:37:41 smithi115 ceph-mon[78593]: pgmap v649: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:37:41.860 INFO:journalctl@ceph.mds.l.smithi115.stdout:May 13 23:37:41 smithi115 conmon[107138]: 2022-05-13T23:37:41.685+0000 7f8deedbc700 -1 mds.0.journaler.pq(rw) _prezeroed got (110) Connection timed out 2022-05-13T23:37:41.861 INFO:journalctl@ceph.mds.l.smithi115.stdout:May 13 23:37:41 smithi115 conmon[107138]: 2022-05-13T23:37:41.685+0000 7f8deedbc700 -1 mds.0.journaler.pq(rw) handle_write_error (110) Connection timed out 2022-05-13T23:37:41.861 INFO:journalctl@ceph.mds.l.smithi115.stdout:May 13 23:37:41 smithi115 conmon[107138]: 2022-05-13T23:37:41.685+0000 7f8deedbc700 -1 mds.0.14 unhandled write error (110) Connection timed out, force readonly... 2022-05-13T23:37:42.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:37:41 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:37:42.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:37:41 smithi047 ceph-mon[77147]: fsmap cephfs:5/5 3 up:standby-replay 1 up:replay 4 up:active 2022-05-13T23:37:42.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:37:41 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:37:42.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:37:41 smithi047 ceph-mon[77147]: pgmap v649: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:37:42.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:37:41 smithi115 conmon[90393]: 2022-05-13T23:37:41.872+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:37:42.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:37:42 smithi115 ceph-mon[78593]: force file system read-only 2022-05-13T23:37:42.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:37:42 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:37:43.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:37:42 smithi047 ceph-mon[77147]: force file system read-only 2022-05-13T23:37:43.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:37:42 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:37:43.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:37:42 smithi115 conmon[90393]: 2022-05-13T23:37:42.888+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:37:43.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:37:43 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:37:43.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:37:43 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 595 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:37:43.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:37:43 smithi115 ceph-mon[78593]: pgmap v650: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:37:44.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:37:43 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:37:44.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:37:43 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 595 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:37:44.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:37:43 smithi047 ceph-mon[77147]: pgmap v650: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:37:44.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:37:43 smithi115 conmon[90393]: 2022-05-13T23:37:43.856+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:37:44.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:37:44 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:37:45.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:37:44 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:37:45.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:37:44 smithi115 conmon[90393]: 2022-05-13T23:37:44.892+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:37:45.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:37:45 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:37:45.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:37:45 smithi115 ceph-mon[78593]: pgmap v651: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:37:46.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:37:45 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:37:46.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:37:45 smithi047 ceph-mon[77147]: pgmap v651: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:37:46.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:37:45 smithi115 conmon[90393]: 2022-05-13T23:37:45.889+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:37:46.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:37:46 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:37:47.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:37:46 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:37:47.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:37:46 smithi115 conmon[90393]: 2022-05-13T23:37:46.877+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:37:47.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:37:47 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:37:47.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:37:47 smithi115 ceph-mon[78593]: pgmap v652: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:37:48.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:37:47 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:37:48.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:37:47 smithi047 ceph-mon[77147]: pgmap v652: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:37:48.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:37:47 smithi115 conmon[90393]: 2022-05-13T23:37:47.914+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:37:48.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:37:48 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:37:48.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:37:48 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 600 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:37:49.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:37:48 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:37:49.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:37:48 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 600 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:37:49.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:37:48 smithi115 conmon[90393]: 2022-05-13T23:37:48.922+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:37:49.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:37:49 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:37:49.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:37:49 smithi115 ceph-mon[78593]: Health check failed: 1 MDSs are read only (MDS_READ_ONLY) 2022-05-13T23:37:49.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:37:49 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:37:49.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:37:49 smithi115 ceph-mon[78593]: fsmap cephfs:5/5 3 up:standby-replay 1 up:replay 4 up:active 2022-05-13T23:37:49.862 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:37:49 smithi115 ceph-mon[78593]: pgmap v653: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:37:50.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:37:49 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:37:50.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:37:49 smithi047 ceph-mon[77147]: Health check failed: 1 MDSs are read only (MDS_READ_ONLY) 2022-05-13T23:37:50.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:37:49 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:37:50.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:37:49 smithi047 ceph-mon[77147]: fsmap cephfs:5/5 3 up:standby-replay 1 up:replay 4 up:active 2022-05-13T23:37:50.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:37:49 smithi047 ceph-mon[77147]: pgmap v653: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:37:50.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:37:49 smithi115 conmon[90393]: 2022-05-13T23:37:49.889+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:37:50.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:37:50 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:37:51.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:37:50 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:37:51.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:37:50 smithi115 conmon[90393]: 2022-05-13T23:37:50.920+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:37:51.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:37:51 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:37:51.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:37:51 smithi115 ceph-mon[78593]: pgmap v654: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:37:52.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:37:51 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:37:52.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:37:51 smithi047 ceph-mon[77147]: pgmap v654: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:37:52.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:37:51 smithi115 conmon[90393]: 2022-05-13T23:37:51.929+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:37:52.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:37:52 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:37:53.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:37:52 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:37:53.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:37:52 smithi115 conmon[90393]: 2022-05-13T23:37:52.906+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:37:53.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:37:53 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:37:53.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:37:53 smithi115 ceph-mon[78593]: fsmap cephfs:5/5 3 up:standby-replay 1 up:replay 4 up:active 2022-05-13T23:37:53.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:37:53 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:37:53.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:37:53 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 605 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:37:53.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:37:53 smithi115 ceph-mon[78593]: pgmap v655: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:37:54.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:37:53 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:37:54.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:37:53 smithi047 ceph-mon[77147]: fsmap cephfs:5/5 3 up:standby-replay 1 up:replay 4 up:active 2022-05-13T23:37:54.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:37:53 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:37:54.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:37:53 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 605 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:37:54.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:37:53 smithi047 ceph-mon[77147]: pgmap v655: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:37:54.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:37:53 smithi115 conmon[90393]: 2022-05-13T23:37:53.882+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:37:54.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:37:54 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:37:55.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:37:54 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:37:55.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:37:54 smithi115 conmon[90393]: 2022-05-13T23:37:54.867+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:37:55.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:37:55 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:37:55.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:37:55 smithi115 ceph-mon[78593]: pgmap v656: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:37:56.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:37:55 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:37:56.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:37:55 smithi047 ceph-mon[77147]: pgmap v656: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:37:56.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:37:55 smithi115 conmon[90393]: 2022-05-13T23:37:55.891+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:37:56.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:37:56 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:37:57.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:37:56 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:37:57.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:37:56 smithi115 conmon[90393]: 2022-05-13T23:37:56.869+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:37:57.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:37:57 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:37:57.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:37:57 smithi115 ceph-mon[78593]: fsmap cephfs:5/5 3 up:standby-replay 1 up:replay 4 up:active 2022-05-13T23:37:57.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:37:57 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:37:57.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:37:57 smithi115 ceph-mon[78593]: pgmap v657: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:37:58.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:37:57 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:37:58.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:37:57 smithi047 ceph-mon[77147]: fsmap cephfs:5/5 3 up:standby-replay 1 up:replay 4 up:active 2022-05-13T23:37:58.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:37:57 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:37:58.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:37:57 smithi047 ceph-mon[77147]: pgmap v657: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:37:58.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:37:57 smithi115 conmon[90393]: 2022-05-13T23:37:57.866+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:37:58.849 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:37:58 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:37:58.850 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:37:58 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 610 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:37:59.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:37:58 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:37:59.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:37:58 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 610 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:37:59.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:37:58 smithi115 conmon[90393]: 2022-05-13T23:37:58.847+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:37:59.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:37:59 smithi115 conmon[90393]: 2022-05-13T23:37:59.815+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:37:59.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:37:59 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:37:59.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:37:59 smithi115 ceph-mon[78593]: pgmap v658: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:38:00.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:37:59 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:38:00.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:37:59 smithi047 ceph-mon[77147]: pgmap v658: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:38:00.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:38:00 smithi115 conmon[90393]: 2022-05-13T23:38:00.769+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:38:00.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:38:00 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:38:01.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:38:00 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:38:02.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:38:01 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:38:02.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:38:01 smithi047 ceph-mon[77147]: fsmap cephfs:5/5 3 up:standby-replay 1 up:replay 4 up:active 2022-05-13T23:38:02.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:38:01 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:38:02.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:38:01 smithi047 ceph-mon[77147]: pgmap v659: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:38:02.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:38:01 smithi115 conmon[90393]: 2022-05-13T23:38:01.792+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:38:02.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:38:01 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:38:02.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:38:01 smithi115 ceph-mon[78593]: fsmap cephfs:5/5 3 up:standby-replay 1 up:replay 4 up:active 2022-05-13T23:38:02.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:38:01 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:38:02.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:38:01 smithi115 ceph-mon[78593]: pgmap v659: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:38:03.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:38:02 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:38:03.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:38:02 smithi115 conmon[90393]: 2022-05-13T23:38:02.749+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:38:03.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:38:02 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:38:04.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:38:03 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:38:04.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:38:03 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 615 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:38:04.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:38:03 smithi047 ceph-mon[77147]: pgmap v660: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:38:04.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:38:03 smithi115 conmon[90393]: 2022-05-13T23:38:03.726+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:38:04.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:38:03 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:38:04.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:38:03 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 615 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:38:04.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:38:03 smithi115 ceph-mon[78593]: pgmap v660: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:38:05.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:38:04 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:38:05.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:38:04 smithi115 conmon[90393]: 2022-05-13T23:38:04.764+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:38:05.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:38:04 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:38:06.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:38:05 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:38:06.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:38:05 smithi047 ceph-mon[77147]: pgmap v661: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:38:06.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:38:05 smithi115 conmon[90393]: 2022-05-13T23:38:05.719+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:38:06.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:38:05 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:38:06.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:38:05 smithi115 ceph-mon[78593]: pgmap v661: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:38:07.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:38:06 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:38:07.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:38:06 smithi115 conmon[90393]: 2022-05-13T23:38:06.713+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:38:07.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:38:06 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:38:08.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:38:07 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:38:08.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:38:07 smithi047 ceph-mon[77147]: pgmap v662: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:38:08.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:38:07 smithi115 conmon[90393]: 2022-05-13T23:38:07.754+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:38:08.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:38:07 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:38:08.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:38:07 smithi115 ceph-mon[78593]: pgmap v662: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:38:09.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:38:08 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:38:09.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:38:08 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 620 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:38:09.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:38:08 smithi115 conmon[90393]: 2022-05-13T23:38:08.760+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:38:09.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:38:08 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:38:09.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:38:08 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 620 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:38:10.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:38:09 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-13T23:38:10.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:38:09 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-13T23:38:10.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:38:09 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-13T23:38:10.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:38:09 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:38:10.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:38:09 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:38:10.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:38:09 smithi047 ceph-mon[77147]: fsmap cephfs:5/5 3 up:standby-replay 1 up:replay 4 up:active 2022-05-13T23:38:10.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:38:09 smithi047 ceph-mon[77147]: pgmap v663: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:38:10.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:38:09 smithi115 conmon[90393]: 2022-05-13T23:38:09.752+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:38:10.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:38:09 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-13T23:38:10.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:38:09 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-13T23:38:10.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:38:09 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-13T23:38:10.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:38:09 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:38:10.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:38:09 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:38:10.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:38:09 smithi115 ceph-mon[78593]: fsmap cephfs:5/5 3 up:standby-replay 1 up:replay 4 up:active 2022-05-13T23:38:10.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:38:09 smithi115 ceph-mon[78593]: pgmap v663: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:38:11.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:38:10 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:38:11.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:38:10 smithi115 conmon[90393]: 2022-05-13T23:38:10.780+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:38:11.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:38:10 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:38:12.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:38:11 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:38:12.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:38:11 smithi047 ceph-mon[77147]: pgmap v664: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:38:12.042 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:38:11 smithi047 conmon[78521]: 2022-05-13T23:38:11.823+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:38:12.042 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:38:11 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-13T23:38:12.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:38:11 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-13T23:38:12.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:38:11 smithi047 conmon[78521]: yield 2022-05-13T23:38:12.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:38:11 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-13T23:38:12.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:38:11 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-13T23:38:12.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:38:11 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-13T23:38:12.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:38:11 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-13T23:38:12.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:38:11 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-13T23:38:12.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:38:11 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-13T23:38:12.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:38:11 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-13T23:38:12.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:38:11 smithi047 conmon[78521]: raise exceptions[0] 2022-05-13T23:38:12.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:38:11 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-13T23:38:12.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:38:11 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-13T23:38:12.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:38:11 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-13T23:38:12.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:38:11 smithi047 conmon[78521]: return (yield from fut) 2022-05-13T23:38:12.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:38:11 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-13T23:38:12.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:38:11 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-13T23:38:12.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:38:11 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:38:12.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:38:11 smithi115 conmon[90393]: 2022-05-13T23:38:11.759+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:38:12.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:38:11 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:38:12.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:38:11 smithi115 ceph-mon[78593]: pgmap v664: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:38:13.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:38:12 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:38:13.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:38:12 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:38:13.042 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-13T23:38:13.042 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-13T23:38:13.042 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-13T23:38:13.042 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-13T23:38:13.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-13T23:38:13.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-13T23:38:13.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-13T23:38:13.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-13T23:38:13.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-13T23:38:13.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-13T23:38:13.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-13T23:38:13.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-13T23:38:13.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-13T23:38:13.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-13T23:38:13.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-13T23:38:13.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-13T23:38:13.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-13T23:38:13.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:38:13.046 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:38:12 smithi047 ceph-mon[77147]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-13T23:38:13.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:38:12 smithi115 conmon[90393]: 2022-05-13T23:38:12.776+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:38:13.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:38:12 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:38:13.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:38:12 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:38:13.109 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-13T23:38:13.109 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-13T23:38:13.109 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-13T23:38:13.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-13T23:38:13.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-13T23:38:13.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-13T23:38:13.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-13T23:38:13.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-13T23:38:13.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-13T23:38:13.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-13T23:38:13.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-13T23:38:13.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-13T23:38:13.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-13T23:38:13.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-13T23:38:13.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-13T23:38:13.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-13T23:38:13.113 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-13T23:38:13.113 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:38:13.113 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:38:12 smithi115 ceph-mon[78593]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-13T23:38:14.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:38:13 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:38:14.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:38:13 smithi047 ceph-mon[77147]: fsmap cephfs:5/5 3 up:standby-replay 1 up:replay 4 up:active 2022-05-13T23:38:14.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:38:13 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:38:14.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:38:13 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-13T23:38:14.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:38:13 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-13T23:38:14.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:38:13 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 625 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:38:14.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:38:13 smithi047 ceph-mon[77147]: pgmap v665: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:38:14.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:38:13 smithi115 conmon[90393]: 2022-05-13T23:38:13.797+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:38:14.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:38:13 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:38:14.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:38:13 smithi115 ceph-mon[78593]: fsmap cephfs:5/5 3 up:standby-replay 1 up:replay 4 up:active 2022-05-13T23:38:14.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:38:13 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:38:14.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:38:13 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-13T23:38:14.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:38:13 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-13T23:38:14.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:38:13 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 625 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:38:14.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:38:13 smithi115 ceph-mon[78593]: pgmap v665: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:38:14.897 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:38:14 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:38:15.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:38:14 smithi115 conmon[90393]: 2022-05-13T23:38:14.810+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:38:15.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:38:14 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:38:15.292 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:38:14 smithi047 conmon[78521]: 2022-05-13T23:38:14.895+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:38:15.292 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:38:14 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-13T23:38:15.292 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:38:14 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-13T23:38:15.292 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:38:14 smithi047 conmon[78521]: yield 2022-05-13T23:38:15.293 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:38:14 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-13T23:38:15.293 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:38:14 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-13T23:38:15.293 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:38:14 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-13T23:38:15.293 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:38:14 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-13T23:38:15.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:38:14 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-13T23:38:15.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:38:14 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-13T23:38:15.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:38:14 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-13T23:38:15.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:38:14 smithi047 conmon[78521]: raise exceptions[0] 2022-05-13T23:38:15.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:38:14 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-13T23:38:15.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:38:14 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-13T23:38:15.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:38:14 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-13T23:38:15.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:38:14 smithi047 conmon[78521]: return (yield from fut) 2022-05-13T23:38:15.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:38:14 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-13T23:38:15.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:38:14 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-13T23:38:15.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:38:14 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:38:15.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:38:14 smithi047 conmon[78521]: 2022-05-13T23:38:14.896+0000 7f533ef76700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-13T23:38:15.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:38:14 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-13T23:38:15.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:38:14 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-13T23:38:15.297 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:38:14 smithi047 conmon[78521]: yield 2022-05-13T23:38:15.297 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:38:14 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-13T23:38:15.297 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:38:14 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-13T23:38:15.297 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:38:14 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-13T23:38:15.297 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:38:14 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-13T23:38:15.298 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:38:14 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-13T23:38:15.298 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:38:14 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-13T23:38:15.298 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:38:14 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-13T23:38:15.298 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:38:14 smithi047 conmon[78521]: raise exceptions[0] 2022-05-13T23:38:15.299 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:38:14 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-13T23:38:15.299 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:38:14 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-13T23:38:15.299 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:38:14 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-13T23:38:15.299 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:38:14 smithi047 conmon[78521]: return (yield from fut) 2022-05-13T23:38:15.299 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:38:14 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-13T23:38:15.300 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:38:14 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-13T23:38:15.300 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:38:14 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:38:15.300 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:38:14 smithi047 conmon[78521]: 2022-05-13T23:38:15.300 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:38:14 smithi047 conmon[78521]: During handling of the above exception, another exception occurred: 2022-05-13T23:38:15.300 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:38:14 smithi047 conmon[78521]: 2022-05-13T23:38:15.301 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:38:14 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-13T23:38:15.301 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:38:14 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-13T23:38:15.301 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:38:14 smithi047 conmon[78521]: return f(*arg) 2022-05-13T23:38:15.301 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:38:14 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-13T23:38:15.302 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:38:14 smithi047 conmon[78521]: self._write_client_files(client_files, host) 2022-05-13T23:38:15.302 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:38:14 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-13T23:38:15.302 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:38:14 smithi047 conmon[78521]: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-13T23:38:15.302 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:38:14 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-13T23:38:15.302 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:38:14 smithi047 conmon[78521]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-13T23:38:15.303 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:38:14 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-13T23:38:15.303 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:38:14 smithi047 conmon[78521]: return self.event_loop.get_result(coro) 2022-05-13T23:38:15.303 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:38:14 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-13T23:38:15.303 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:38:14 smithi047 conmon[78521]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-13T23:38:15.303 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:38:14 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-13T23:38:15.304 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:38:14 smithi047 conmon[78521]: return self.__get_result() 2022-05-13T23:38:15.304 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:38:14 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-13T23:38:15.304 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:38:14 smithi047 conmon[78521]: raise self._exception 2022-05-13T23:38:15.304 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:38:14 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-13T23:38:15.304 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:38:14 smithi047 conmon[78521]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-13T23:38:15.305 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:38:14 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-13T23:38:15.305 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:38:14 smithi047 conmon[78521]: conn = await self._remote_connection(host, addr) 2022-05-13T23:38:15.305 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:38:14 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-13T23:38:15.305 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:38:14 smithi047 conmon[78521]: raise 2022-05-13T23:38:15.305 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:38:14 smithi047 conmon[78521]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-13T23:38:15.306 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:38:14 smithi047 conmon[78521]: self.gen.throw(type, value, traceback) 2022-05-13T23:38:15.306 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:38:14 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-13T23:38:15.306 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:38:14 smithi047 conmon[78521]: raise OrchestratorError(msg) 2022-05-13T23:38:15.306 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:38:14 smithi047 conmon[78521]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:38:16.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:38:15 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:38:16.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:38:15 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:38:16.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-13T23:38:16.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-13T23:38:16.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-13T23:38:16.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-13T23:38:16.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-13T23:38:16.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-13T23:38:16.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-13T23:38:16.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-13T23:38:16.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-13T23:38:16.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-13T23:38:16.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-13T23:38:16.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-13T23:38:16.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-13T23:38:16.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-13T23:38:16.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-13T23:38:16.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-13T23:38:16.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-13T23:38:16.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:38:16.048 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:38:15 smithi047 ceph-mon[77147]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-13T23:38:16.049 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-13T23:38:16.049 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-13T23:38:16.049 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-13T23:38:16.049 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-13T23:38:16.050 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-13T23:38:16.050 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-13T23:38:16.050 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-13T23:38:16.051 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-13T23:38:16.051 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-13T23:38:16.051 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-13T23:38:16.051 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-13T23:38:16.052 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-13T23:38:16.052 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-13T23:38:16.053 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-13T23:38:16.053 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-13T23:38:16.054 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-13T23:38:16.054 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-13T23:38:16.055 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:38:16.055 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-13T23:38:16.055 INFO:journalctl@ceph.mon.b.smithi047.stdout: During handling of the above exception, another exception occurred: 2022-05-13T23:38:16.056 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-13T23:38:16.056 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-13T23:38:16.056 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-13T23:38:16.057 INFO:journalctl@ceph.mon.b.smithi047.stdout: return f(*arg) 2022-05-13T23:38:16.057 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-13T23:38:16.057 INFO:journalctl@ceph.mon.b.smithi047.stdout: self._write_client_files(client_files, host) 2022-05-13T23:38:16.058 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-13T23:38:16.058 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-13T23:38:16.058 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-13T23:38:16.058 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-13T23:38:16.058 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-13T23:38:16.059 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.event_loop.get_result(coro) 2022-05-13T23:38:16.059 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-13T23:38:16.059 INFO:journalctl@ceph.mon.b.smithi047.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-13T23:38:16.059 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-13T23:38:16.060 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.__get_result() 2022-05-13T23:38:16.060 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-13T23:38:16.060 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise self._exception 2022-05-13T23:38:16.060 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-13T23:38:16.060 INFO:journalctl@ceph.mon.b.smithi047.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-13T23:38:16.061 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-13T23:38:16.061 INFO:journalctl@ceph.mon.b.smithi047.stdout: conn = await self._remote_connection(host, addr) 2022-05-13T23:38:16.061 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-13T23:38:16.061 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise 2022-05-13T23:38:16.062 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-13T23:38:16.062 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.gen.throw(type, value, traceback) 2022-05-13T23:38:16.062 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-13T23:38:16.062 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OrchestratorError(msg) 2022-05-13T23:38:16.063 INFO:journalctl@ceph.mon.b.smithi047.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:38:16.063 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:38:15 smithi047 ceph-mon[77147]: pgmap v666: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:38:16.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:38:15 smithi115 conmon[90393]: 2022-05-13T23:38:15.838+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:38:16.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:38:15 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:38:16.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:38:15 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:38:16.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-13T23:38:16.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-13T23:38:16.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-13T23:38:16.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-13T23:38:16.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-13T23:38:16.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-13T23:38:16.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-13T23:38:16.113 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-13T23:38:16.113 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-13T23:38:16.113 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-13T23:38:16.114 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-13T23:38:16.114 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-13T23:38:16.114 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-13T23:38:16.115 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-13T23:38:16.115 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-13T23:38:16.115 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-13T23:38:16.116 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-13T23:38:16.116 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:38:16.116 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:38:15 smithi115 ceph-mon[78593]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-13T23:38:16.116 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-13T23:38:16.117 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-13T23:38:16.117 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-13T23:38:16.117 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-13T23:38:16.117 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-13T23:38:16.118 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-13T23:38:16.118 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-13T23:38:16.118 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-13T23:38:16.118 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-13T23:38:16.119 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-13T23:38:16.119 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-13T23:38:16.119 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-13T23:38:16.120 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-13T23:38:16.120 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-13T23:38:16.120 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-13T23:38:16.121 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-13T23:38:16.121 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-13T23:38:16.122 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:38:16.122 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-13T23:38:16.122 INFO:journalctl@ceph.mon.c.smithi115.stdout: During handling of the above exception, another exception occurred: 2022-05-13T23:38:16.123 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-13T23:38:16.123 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-13T23:38:16.123 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-13T23:38:16.124 INFO:journalctl@ceph.mon.c.smithi115.stdout: return f(*arg) 2022-05-13T23:38:16.124 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-13T23:38:16.124 INFO:journalctl@ceph.mon.c.smithi115.stdout: self._write_client_files(client_files, host) 2022-05-13T23:38:16.125 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-13T23:38:16.125 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-13T23:38:16.126 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-13T23:38:16.126 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-13T23:38:16.126 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-13T23:38:16.127 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.event_loop.get_result(coro) 2022-05-13T23:38:16.127 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-13T23:38:16.127 INFO:journalctl@ceph.mon.c.smithi115.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-13T23:38:16.128 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-13T23:38:16.128 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.__get_result() 2022-05-13T23:38:16.128 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-13T23:38:16.129 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise self._exception 2022-05-13T23:38:16.129 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-13T23:38:16.130 INFO:journalctl@ceph.mon.c.smithi115.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-13T23:38:16.130 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-13T23:38:16.131 INFO:journalctl@ceph.mon.c.smithi115.stdout: conn = await self._remote_connection(host, addr) 2022-05-13T23:38:16.131 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-13T23:38:16.132 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise 2022-05-13T23:38:16.132 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-13T23:38:16.132 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.gen.throw(type, value, traceback) 2022-05-13T23:38:16.133 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-13T23:38:16.133 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OrchestratorError(msg) 2022-05-13T23:38:16.134 INFO:journalctl@ceph.mon.c.smithi115.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:38:16.134 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:38:15 smithi115 ceph-mon[78593]: pgmap v666: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:38:17.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:38:16 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:38:17.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:38:16 smithi115 conmon[90393]: 2022-05-13T23:38:16.830+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:38:17.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:38:16 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:38:18.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:38:17 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:38:18.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:38:17 smithi047 ceph-mon[77147]: fsmap cephfs:5/5 3 up:standby-replay 1 up:replay 4 up:active 2022-05-13T23:38:18.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:38:17 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:38:18.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:38:17 smithi047 ceph-mon[77147]: pgmap v667: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:38:18.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:38:17 smithi115 conmon[90393]: 2022-05-13T23:38:17.861+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:38:18.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:38:17 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:38:18.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:38:17 smithi115 ceph-mon[78593]: fsmap cephfs:5/5 3 up:standby-replay 1 up:replay 4 up:active 2022-05-13T23:38:18.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:38:17 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:38:18.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:38:17 smithi115 ceph-mon[78593]: pgmap v667: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:38:19.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:38:18 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:38:19.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:38:18 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 630 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:38:19.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:38:18 smithi115 conmon[90393]: 2022-05-13T23:38:18.846+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:38:19.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:38:18 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:38:19.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:38:18 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 630 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:38:20.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:38:19 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:38:20.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:38:19 smithi047 ceph-mon[77147]: pgmap v668: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:38:20.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:38:19 smithi115 conmon[90393]: 2022-05-13T23:38:19.889+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:38:20.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:38:19 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:38:20.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:38:19 smithi115 ceph-mon[78593]: pgmap v668: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:38:21.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:38:20 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:38:21.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:38:20 smithi115 conmon[90393]: 2022-05-13T23:38:20.901+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:38:21.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:38:20 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:38:22.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:38:21 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:38:22.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:38:21 smithi047 ceph-mon[77147]: fsmap cephfs:5/5 3 up:standby-replay 1 up:replay 4 up:active 2022-05-13T23:38:22.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:38:21 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:38:22.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:38:21 smithi047 ceph-mon[77147]: pgmap v669: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:38:22.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:38:21 smithi115 conmon[90393]: 2022-05-13T23:38:21.906+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:38:22.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:38:21 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:38:22.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:38:21 smithi115 ceph-mon[78593]: fsmap cephfs:5/5 3 up:standby-replay 1 up:replay 4 up:active 2022-05-13T23:38:22.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:38:21 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:38:22.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:38:21 smithi115 ceph-mon[78593]: pgmap v669: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:38:23.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:38:22 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:38:23.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:38:22 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:38:23.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:38:22 smithi115 conmon[90393]: 2022-05-13T23:38:22.898+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:38:24.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:38:23 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:38:24.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:38:23 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 635 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:38:24.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:38:23 smithi047 ceph-mon[77147]: pgmap v670: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:38:24.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:38:23 smithi115 conmon[90393]: 2022-05-13T23:38:23.873+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:38:24.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:38:23 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:38:24.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:38:23 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 635 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:38:24.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:38:23 smithi115 ceph-mon[78593]: pgmap v670: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:38:25.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:38:24 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:38:25.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:38:24 smithi115 conmon[90393]: 2022-05-13T23:38:24.923+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:38:25.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:38:24 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:38:25.956 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:38:25 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:38:25.957 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:38:25 smithi115 ceph-mon[78593]: pgmap v671: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:38:26.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:38:25 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:38:26.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:38:25 smithi047 ceph-mon[77147]: pgmap v671: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:38:26.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:38:25 smithi115 conmon[90393]: 2022-05-13T23:38:25.956+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:38:26.996 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:38:26 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:38:27.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:38:26 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:38:27.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:38:26 smithi115 conmon[90393]: 2022-05-13T23:38:26.995+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:38:27.960 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:38:27 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:38:27.960 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:38:27 smithi115 ceph-mon[78593]: pgmap v672: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:38:28.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:38:27 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:38:28.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:38:27 smithi047 ceph-mon[77147]: pgmap v672: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:38:28.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:38:27 smithi115 conmon[90393]: 2022-05-13T23:38:27.958+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:38:29.000 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:38:28 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:38:29.001 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:38:28 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 640 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:38:29.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:38:28 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:38:29.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:38:28 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 640 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:38:29.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:38:29 smithi115 conmon[90393]: 2022-05-13T23:38:28.999+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:38:29.997 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:38:29 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:38:29.997 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:38:29 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:38:29.998 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:38:29 smithi115 ceph-mon[78593]: fsmap cephfs:5/5 3 up:standby-replay 1 up:replay 4 up:active 2022-05-13T23:38:29.998 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:38:29 smithi115 ceph-mon[78593]: pgmap v673: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:38:30.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:38:29 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:38:30.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:38:29 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:38:30.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:38:29 smithi047 ceph-mon[77147]: fsmap cephfs:5/5 3 up:standby-replay 1 up:replay 4 up:active 2022-05-13T23:38:30.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:38:29 smithi047 ceph-mon[77147]: pgmap v673: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:38:30.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:38:29 smithi115 conmon[90393]: 2022-05-13T23:38:29.996+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:38:30.988 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:38:30 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:38:31.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:38:30 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:38:31.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:38:30 smithi115 conmon[90393]: 2022-05-13T23:38:30.987+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:38:32.009 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:38:31 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:38:32.010 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:38:31 smithi115 ceph-mon[78593]: pgmap v674: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:38:32.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:38:31 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:38:32.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:38:31 smithi047 ceph-mon[77147]: pgmap v674: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:38:32.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:38:32 smithi115 conmon[90393]: 2022-05-13T23:38:32.009+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:38:32.994 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:38:32 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:38:33.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:38:32 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:38:33.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:38:32 smithi115 conmon[90393]: 2022-05-13T23:38:32.992+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:38:34.027 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:38:33 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:38:34.028 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:38:33 smithi115 ceph-mon[78593]: fsmap cephfs:5/5 3 up:standby-replay 1 up:replay 4 up:active 2022-05-13T23:38:34.028 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:38:33 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:38:34.028 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:38:33 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 645 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:38:34.028 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:38:33 smithi115 ceph-mon[78593]: pgmap v675: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:38:34.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:38:33 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:38:34.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:38:33 smithi047 ceph-mon[77147]: fsmap cephfs:5/5 3 up:standby-replay 1 up:replay 4 up:active 2022-05-13T23:38:34.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:38:33 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:38:34.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:38:33 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 645 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:38:34.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:38:33 smithi047 ceph-mon[77147]: pgmap v675: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:38:34.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:38:34 smithi115 conmon[90393]: 2022-05-13T23:38:34.025+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:38:34.987 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:38:34 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:38:35.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:38:34 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:38:35.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:38:34 smithi115 conmon[90393]: 2022-05-13T23:38:34.986+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:38:35.993 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:38:35 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:38:35.994 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:38:35 smithi115 ceph-mon[78593]: pgmap v676: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:38:36.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:38:35 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:38:36.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:38:35 smithi047 ceph-mon[77147]: pgmap v676: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:38:36.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:38:35 smithi115 conmon[90393]: 2022-05-13T23:38:35.990+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:38:36.977 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:38:36 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:38:37.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:38:36 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:38:37.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:38:36 smithi115 conmon[90393]: 2022-05-13T23:38:36.975+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:38:38.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:38:37 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:38:38.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:38:37 smithi047 ceph-mon[77147]: fsmap cephfs:5/5 3 up:standby-replay 1 up:replay 4 up:active 2022-05-13T23:38:38.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:38:37 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:38:38.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:38:37 smithi047 ceph-mon[77147]: pgmap v677: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:38:38.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:38:37 smithi115 conmon[90393]: 2022-05-13T23:38:37.976+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:38:38.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:38:37 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:38:38.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:38:37 smithi115 ceph-mon[78593]: fsmap cephfs:5/5 3 up:standby-replay 1 up:replay 4 up:active 2022-05-13T23:38:38.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:38:37 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:38:38.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:38:37 smithi115 ceph-mon[78593]: pgmap v677: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:38:39.017 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:38:38 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:38:39.017 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:38:38 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 650 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:38:39.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:38:38 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:38:39.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:38:38 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 650 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:38:39.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:38:39 smithi115 conmon[90393]: 2022-05-13T23:38:39.015+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:38:40.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:38:39 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:38:40.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:38:39 smithi047 ceph-mon[77147]: pgmap v678: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:38:40.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:38:39 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:38:40.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:38:39 smithi115 ceph-mon[78593]: pgmap v678: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:38:40.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:38:40 smithi115 conmon[90393]: 2022-05-13T23:38:40.044+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:38:41.039 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:38:40 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:38:41.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:38:40 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:38:41.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:38:41 smithi115 conmon[90393]: 2022-05-13T23:38:41.037+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:38:42.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:38:41 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:38:42.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:38:41 smithi047 ceph-mon[77147]: fsmap cephfs:5/5 3 up:standby-replay 1 up:replay 4 up:active 2022-05-13T23:38:42.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:38:41 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:38:42.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:38:41 smithi047 ceph-mon[77147]: pgmap v679: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:38:42.061 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:38:41 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:38:42.061 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:38:41 smithi115 ceph-mon[78593]: fsmap cephfs:5/5 3 up:standby-replay 1 up:replay 4 up:active 2022-05-13T23:38:42.061 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:38:41 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:38:42.061 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:38:41 smithi115 ceph-mon[78593]: pgmap v679: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:38:42.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:38:42 smithi115 conmon[90393]: 2022-05-13T23:38:42.059+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:38:43.013 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:38:42 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:38:43.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:38:42 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:38:43.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:38:43 smithi115 conmon[90393]: 2022-05-13T23:38:43.011+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:38:44.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:38:43 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:38:44.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:38:43 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 655 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:38:44.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:38:43 smithi047 ceph-mon[77147]: pgmap v680: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:38:44.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:38:43 smithi115 conmon[90393]: 2022-05-13T23:38:43.982+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:38:44.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:38:43 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:38:44.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:38:43 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 655 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:38:44.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:38:43 smithi115 ceph-mon[78593]: pgmap v680: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:38:45.009 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:38:44 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:38:45.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:38:44 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:38:45.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:38:45 smithi115 conmon[90393]: 2022-05-13T23:38:45.007+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:38:46.019 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:38:45 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:38:46.020 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:38:45 smithi115 ceph-mon[78593]: pgmap v681: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:38:46.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:38:45 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:38:46.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:38:45 smithi047 ceph-mon[77147]: pgmap v681: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:38:46.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:38:46 smithi115 conmon[90393]: 2022-05-13T23:38:46.018+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:38:47.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:38:46 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:38:47.053 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:38:46 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:38:47.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:38:47 smithi115 conmon[90393]: 2022-05-13T23:38:47.051+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:38:48.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:38:47 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:38:48.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:38:47 smithi047 ceph-mon[77147]: pgmap v682: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:38:48.057 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:38:47 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:38:48.057 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:38:47 smithi115 ceph-mon[78593]: pgmap v682: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:38:48.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:38:48 smithi115 conmon[90393]: 2022-05-13T23:38:48.056+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:38:49.039 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:38:48 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:38:49.039 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:38:48 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 660 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:38:49.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:38:48 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:38:49.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:38:48 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 660 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:38:49.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:38:49 smithi115 conmon[90393]: 2022-05-13T23:38:49.037+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:38:50.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:38:49 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:38:50.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:38:49 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:38:50.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:38:49 smithi047 ceph-mon[77147]: fsmap cephfs:5/5 3 up:standby-replay 1 up:replay 4 up:active 2022-05-13T23:38:50.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:38:49 smithi047 ceph-mon[77147]: pgmap v683: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:38:50.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:38:50 smithi115 conmon[90393]: 2022-05-13T23:38:50.007+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:38:50.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:38:49 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:38:50.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:38:49 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:38:50.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:38:49 smithi115 ceph-mon[78593]: fsmap cephfs:5/5 3 up:standby-replay 1 up:replay 4 up:active 2022-05-13T23:38:50.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:38:49 smithi115 ceph-mon[78593]: pgmap v683: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:38:51.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:38:50 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:38:51.048 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:38:50 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:38:51.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:38:51 smithi115 conmon[90393]: 2022-05-13T23:38:51.045+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:38:52.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:38:51 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:38:52.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:38:51 smithi047 ceph-mon[77147]: pgmap v684: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:38:52.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:38:52 smithi115 conmon[90393]: 2022-05-13T23:38:52.006+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:38:52.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:38:51 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:38:52.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:38:51 smithi115 ceph-mon[78593]: pgmap v684: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:38:53.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:38:52 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:38:53.055 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:38:52 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:38:53.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:38:53 smithi115 conmon[90393]: 2022-05-13T23:38:53.054+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:38:54.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:38:53 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:38:54.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:38:53 smithi047 ceph-mon[77147]: fsmap cephfs:5/5 3 up:standby-replay 1 up:replay 4 up:active 2022-05-13T23:38:54.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:38:53 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:38:54.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:38:53 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 665 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:38:54.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:38:53 smithi047 ceph-mon[77147]: pgmap v685: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:38:54.050 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:38:53 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:38:54.051 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:38:53 smithi115 ceph-mon[78593]: fsmap cephfs:5/5 3 up:standby-replay 1 up:replay 4 up:active 2022-05-13T23:38:54.051 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:38:53 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:38:54.051 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:38:53 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 665 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:38:54.051 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:38:53 smithi115 ceph-mon[78593]: pgmap v685: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:38:54.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:38:54 smithi115 conmon[90393]: 2022-05-13T23:38:54.049+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:38:55.031 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:38:54 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:38:55.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:38:54 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:38:55.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:38:55 smithi115 conmon[90393]: 2022-05-13T23:38:55.030+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:38:56.049 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:38:55 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:38:56.049 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:38:55 smithi047 ceph-mon[77147]: pgmap v686: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:38:56.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:38:55 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:38:56.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:38:55 smithi115 ceph-mon[78593]: pgmap v686: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:38:56.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:38:56 smithi115 conmon[90393]: 2022-05-13T23:38:56.017+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:38:57.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:38:56 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:38:57.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:38:57 smithi115 conmon[90393]: 2022-05-13T23:38:57.015+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:38:57.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:38:56 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:38:58.060 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:38:57 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:38:58.061 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:38:57 smithi115 ceph-mon[78593]: fsmap cephfs:5/5 3 up:standby-replay 1 up:replay 4 up:active 2022-05-13T23:38:58.061 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:38:57 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:38:58.061 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:38:57 smithi115 ceph-mon[78593]: pgmap v687: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:38:58.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:38:57 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:38:58.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:38:57 smithi047 ceph-mon[77147]: fsmap cephfs:5/5 3 up:standby-replay 1 up:replay 4 up:active 2022-05-13T23:38:58.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:38:57 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:38:58.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:38:57 smithi047 ceph-mon[77147]: pgmap v687: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:38:58.360 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:38:58 smithi115 conmon[90393]: 2022-05-13T23:38:58.059+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:38:59.105 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:38:58 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:38:59.106 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:38:58 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 670 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:38:59.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:38:58 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:38:59.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:38:58 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 670 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:38:59.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:38:59 smithi115 conmon[90393]: 2022-05-13T23:38:59.103+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:39:00.076 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:38:59 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:39:00.076 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:38:59 smithi115 ceph-mon[78593]: pgmap v688: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:39:00.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:38:59 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:39:00.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:38:59 smithi047 ceph-mon[77147]: pgmap v688: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:39:00.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:39:00 smithi115 conmon[90393]: 2022-05-13T23:39:00.074+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:39:01.065 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:39:00 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:39:01.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:39:00 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:39:01.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:39:01 smithi115 conmon[90393]: 2022-05-13T23:39:01.063+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:39:02.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:39:02 smithi115 conmon[90393]: 2022-05-13T23:39:02.021+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:39:02.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:39:01 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:39:02.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:39:01 smithi115 ceph-mon[78593]: fsmap cephfs:5/5 3 up:standby-replay 1 up:replay 4 up:active 2022-05-13T23:39:02.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:39:01 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:39:02.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:39:01 smithi115 ceph-mon[78593]: pgmap v689: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:39:02.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:39:01 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:39:02.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:39:01 smithi047 ceph-mon[77147]: fsmap cephfs:5/5 3 up:standby-replay 1 up:replay 4 up:active 2022-05-13T23:39:02.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:39:01 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:39:02.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:39:01 smithi047 ceph-mon[77147]: pgmap v689: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:39:03.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:39:03 smithi115 conmon[90393]: 2022-05-13T23:39:03.018+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:39:03.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:39:02 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:39:03.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:39:02 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:39:04.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:39:04 smithi115 conmon[90393]: 2022-05-13T23:39:04.001+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:39:04.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:39:03 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:39:04.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:39:03 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 676 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:39:04.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:39:03 smithi115 ceph-mon[78593]: pgmap v690: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:39:04.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:39:03 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:39:04.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:39:03 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 676 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:39:04.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:39:03 smithi047 ceph-mon[77147]: pgmap v690: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:39:05.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:39:05 smithi115 conmon[90393]: 2022-05-13T23:39:05.014+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:39:05.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:39:04 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:39:05.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:39:04 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:39:06.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:39:06 smithi115 conmon[90393]: 2022-05-13T23:39:06.030+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:39:06.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:39:05 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:39:06.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:39:05 smithi115 ceph-mon[78593]: pgmap v691: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:39:06.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:39:05 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:39:06.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:39:05 smithi047 ceph-mon[77147]: pgmap v691: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:39:07.084 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:39:06 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:39:07.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:39:06 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:39:07.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:39:07 smithi115 conmon[90393]: 2022-05-13T23:39:07.081+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:39:08.093 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:39:07 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:39:08.094 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:39:07 smithi115 ceph-mon[78593]: pgmap v692: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:39:08.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:39:07 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:39:08.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:39:07 smithi047 ceph-mon[77147]: pgmap v692: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:39:08.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:39:08 smithi115 conmon[90393]: 2022-05-13T23:39:08.092+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:39:09.080 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:39:08 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:39:09.081 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:39:08 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 680 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:39:09.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:39:08 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:39:09.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:39:08 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 680 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:39:09.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:39:09 smithi115 conmon[90393]: 2022-05-13T23:39:09.079+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:39:10.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:39:09 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:39:10.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:39:09 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:39:10.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:39:09 smithi115 ceph-mon[78593]: fsmap cephfs:5/5 3 up:standby-replay 1 up:replay 4 up:active 2022-05-13T23:39:10.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:39:09 smithi115 ceph-mon[78593]: pgmap v693: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:39:10.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:39:09 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:39:10.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:39:09 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:39:10.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:39:09 smithi047 ceph-mon[77147]: fsmap cephfs:5/5 3 up:standby-replay 1 up:replay 4 up:active 2022-05-13T23:39:10.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:39:09 smithi047 ceph-mon[77147]: pgmap v693: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:39:10.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:39:10 smithi115 conmon[90393]: 2022-05-13T23:39:10.110+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:39:11.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:39:10 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:39:11.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:39:10 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:39:11.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:39:11 smithi115 conmon[90393]: 2022-05-13T23:39:11.140+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:39:12.106 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:39:11 smithi115 ceph-mon[78593]: pgmap v694: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:39:12.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:39:11 smithi047 ceph-mon[77147]: pgmap v694: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:39:12.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:39:12 smithi115 conmon[90393]: 2022-05-13T23:39:12.104+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:39:13.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:39:12 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:39:13.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:39:12 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:39:13.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:39:12 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:39:13.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:39:12 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:39:13.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:39:13 smithi115 conmon[90393]: 2022-05-13T23:39:13.145+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:39:14.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:39:13 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:39:14.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:39:13 smithi115 ceph-mon[78593]: fsmap cephfs:5/5 3 up:standby-replay 1 up:replay 4 up:active 2022-05-13T23:39:14.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:39:13 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-13T23:39:14.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:39:13 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-13T23:39:14.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:39:13 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 685 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:39:14.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:39:13 smithi115 ceph-mon[78593]: pgmap v695: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:39:14.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:39:13 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:39:14.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:39:13 smithi047 ceph-mon[77147]: fsmap cephfs:5/5 3 up:standby-replay 1 up:replay 4 up:active 2022-05-13T23:39:14.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:39:13 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-13T23:39:14.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:39:13 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-13T23:39:14.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:39:13 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 685 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:39:14.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:39:13 smithi047 ceph-mon[77147]: pgmap v695: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:39:14.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:39:14 smithi115 conmon[90393]: 2022-05-13T23:39:14.128+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:39:15.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:39:14 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:39:15.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:39:14 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:39:15.377 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:39:15 smithi115 conmon[90393]: 2022-05-13T23:39:15.107+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:39:15.791 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:39:15 smithi047 conmon[78521]: 2022-05-13T23:39:15.375+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:39:15.792 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:39:15 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-13T23:39:15.792 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:39:15 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-13T23:39:15.792 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:39:15 smithi047 conmon[78521]: yield 2022-05-13T23:39:15.792 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:39:15 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-13T23:39:15.792 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:39:15 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-13T23:39:15.793 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:39:15 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-13T23:39:15.793 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:39:15 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-13T23:39:15.793 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:39:15 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-13T23:39:15.793 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:39:15 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-13T23:39:15.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:39:15 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-13T23:39:15.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:39:15 smithi047 conmon[78521]: raise exceptions[0] 2022-05-13T23:39:15.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:39:15 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-13T23:39:15.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:39:15 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-13T23:39:15.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:39:15 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-13T23:39:15.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:39:15 smithi047 conmon[78521]: return (yield from fut) 2022-05-13T23:39:15.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:39:15 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-13T23:39:15.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:39:15 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-13T23:39:15.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:39:15 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:39:16.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:39:15 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:39:16.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:39:15 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-13T23:39:16.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:39:15 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-13T23:39:16.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:39:15 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-13T23:39:16.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:39:15 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:39:16.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:39:15 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' 2022-05-13T23:39:16.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:39:15 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' 2022-05-13T23:39:16.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:39:15 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:39:16.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-13T23:39:16.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-13T23:39:16.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-13T23:39:16.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-13T23:39:16.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-13T23:39:16.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-13T23:39:16.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-13T23:39:16.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-13T23:39:16.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-13T23:39:16.113 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-13T23:39:16.113 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-13T23:39:16.113 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-13T23:39:16.113 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-13T23:39:16.114 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-13T23:39:16.114 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-13T23:39:16.114 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-13T23:39:16.114 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-13T23:39:16.114 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:39:16.115 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:39:15 smithi115 ceph-mon[78593]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-13T23:39:16.115 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:39:15 smithi115 ceph-mon[78593]: pgmap v696: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:39:16.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:39:15 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:39:16.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:39:15 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-13T23:39:16.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:39:15 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-13T23:39:16.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:39:15 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-13T23:39:16.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:39:15 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:39:16.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:39:15 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' 2022-05-13T23:39:16.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:39:15 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' 2022-05-13T23:39:16.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:39:15 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:39:16.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-13T23:39:16.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-13T23:39:16.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-13T23:39:16.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-13T23:39:16.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-13T23:39:16.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-13T23:39:16.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-13T23:39:16.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-13T23:39:16.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-13T23:39:16.298 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-13T23:39:16.298 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-13T23:39:16.298 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-13T23:39:16.299 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-13T23:39:16.299 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-13T23:39:16.300 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-13T23:39:16.300 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-13T23:39:16.301 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-13T23:39:16.301 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:39:16.302 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:39:15 smithi047 ceph-mon[77147]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-13T23:39:16.302 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:39:15 smithi047 ceph-mon[77147]: pgmap v696: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:39:16.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:39:16 smithi115 conmon[90393]: 2022-05-13T23:39:16.124+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:39:17.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:39:17 smithi115 conmon[90393]: 2022-05-13T23:39:17.134+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:39:18.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:39:18 smithi115 conmon[90393]: 2022-05-13T23:39:18.087+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:39:18.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:39:17 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:39:18.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:39:17 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:39:18.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:39:17 smithi115 ceph-mon[78593]: fsmap cephfs:5/5 3 up:standby-replay 1 up:replay 4 up:active 2022-05-13T23:39:18.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:39:17 smithi115 ceph-mon[78593]: pgmap v697: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:39:18.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:39:17 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:39:18.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:39:17 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:39:18.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:39:17 smithi047 ceph-mon[77147]: fsmap cephfs:5/5 3 up:standby-replay 1 up:replay 4 up:active 2022-05-13T23:39:18.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:39:17 smithi047 ceph-mon[77147]: pgmap v697: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:39:18.791 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:39:18 smithi047 conmon[78521]: 2022-05-13T23:39:18.448+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:39:18.792 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:39:18 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-13T23:39:18.792 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:39:18 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-13T23:39:18.792 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:39:18 smithi047 conmon[78521]: yield 2022-05-13T23:39:18.792 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:39:18 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-13T23:39:18.793 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:39:18 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-13T23:39:18.793 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:39:18 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-13T23:39:18.793 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:39:18 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-13T23:39:18.793 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:39:18 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-13T23:39:18.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:39:18 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-13T23:39:18.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:39:18 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-13T23:39:18.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:39:18 smithi047 conmon[78521]: raise exceptions[0] 2022-05-13T23:39:18.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:39:18 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-13T23:39:18.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:39:18 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-13T23:39:18.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:39:18 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-13T23:39:18.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:39:18 smithi047 conmon[78521]: return (yield from fut) 2022-05-13T23:39:18.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:39:18 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-13T23:39:18.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:39:18 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-13T23:39:18.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:39:18 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:39:18.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:39:18 smithi047 conmon[78521]: 2022-05-13T23:39:18.448+0000 7f5345f84700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-13T23:39:18.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:39:18 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-13T23:39:18.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:39:18 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-13T23:39:18.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:39:18 smithi047 conmon[78521]: yield 2022-05-13T23:39:18.797 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:39:18 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-13T23:39:18.797 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:39:18 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-13T23:39:18.797 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:39:18 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-13T23:39:18.797 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:39:18 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-13T23:39:18.797 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:39:18 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-13T23:39:18.798 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:39:18 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-13T23:39:18.798 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:39:18 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-13T23:39:18.798 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:39:18 smithi047 conmon[78521]: raise exceptions[0] 2022-05-13T23:39:18.798 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:39:18 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-13T23:39:18.798 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:39:18 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-13T23:39:18.799 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:39:18 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-13T23:39:18.799 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:39:18 smithi047 conmon[78521]: return (yield from fut) 2022-05-13T23:39:18.799 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:39:18 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-13T23:39:18.799 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:39:18 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-13T23:39:18.800 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:39:18 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:39:18.800 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:39:18 smithi047 conmon[78521]: 2022-05-13T23:39:18.800 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:39:18 smithi047 conmon[78521]: During handling of the above exception, another exception occurred: 2022-05-13T23:39:18.800 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:39:18 smithi047 conmon[78521]: 2022-05-13T23:39:18.801 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:39:18 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-13T23:39:18.801 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:39:18 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-13T23:39:18.801 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:39:18 smithi047 conmon[78521]: return f(*arg) 2022-05-13T23:39:18.801 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:39:18 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-13T23:39:18.802 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:39:18 smithi047 conmon[78521]: self._write_client_files(client_files, host) 2022-05-13T23:39:18.802 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:39:18 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-13T23:39:18.802 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:39:18 smithi047 conmon[78521]: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-13T23:39:18.803 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:39:18 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-13T23:39:18.803 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:39:18 smithi047 conmon[78521]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-13T23:39:18.803 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:39:18 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-13T23:39:18.803 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:39:18 smithi047 conmon[78521]: return self.event_loop.get_result(coro) 2022-05-13T23:39:18.803 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:39:18 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-13T23:39:18.804 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:39:18 smithi047 conmon[78521]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-13T23:39:18.805 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:39:18 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-13T23:39:18.806 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:39:18 smithi047 conmon[78521]: return self.__get_result() 2022-05-13T23:39:18.806 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:39:18 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-13T23:39:18.807 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:39:18 smithi047 conmon[78521]: raise self._exception 2022-05-13T23:39:18.807 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:39:18 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-13T23:39:18.807 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:39:18 smithi047 conmon[78521]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-13T23:39:18.807 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:39:18 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-13T23:39:18.807 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:39:18 smithi047 conmon[78521]: conn = await self._remote_connection(host, addr) 2022-05-13T23:39:18.808 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:39:18 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-13T23:39:18.808 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:39:18 smithi047 conmon[78521]: raise 2022-05-13T23:39:18.808 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:39:18 smithi047 conmon[78521]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-13T23:39:18.808 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:39:18 smithi047 conmon[78521]: self.gen.throw(type, value, traceback) 2022-05-13T23:39:18.809 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:39:18 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-13T23:39:18.809 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:39:18 smithi047 conmon[78521]: raise OrchestratorError(msg) 2022-05-13T23:39:18.809 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:39:18 smithi047 conmon[78521]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:39:19.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:39:18 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:39:19.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:39:18 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' 2022-05-13T23:39:19.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:39:18 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' 2022-05-13T23:39:19.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:39:18 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:39:19.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:39:18 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 691 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:39:19.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:39:18 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-05-13T23:39:19.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:39:18 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-05-13T23:39:19.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:39:18 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-05-13T23:39:19.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:39:18 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-05-13T23:39:19.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:39:18 smithi047 ceph-mon[77147]: Adjusting osd_memory_target on smithi115 to 197.0M 2022-05-13T23:39:19.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:39:18 smithi047 ceph-mon[77147]: Unable to set osd_memory_target on smithi115 to 206629785: error parsing value: Value '206629785' is below minimum 939524096 2022-05-13T23:39:19.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:39:18 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:39:19.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-13T23:39:19.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-13T23:39:19.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-13T23:39:19.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-13T23:39:19.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-13T23:39:19.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-13T23:39:19.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-13T23:39:19.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-13T23:39:19.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-13T23:39:19.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-13T23:39:19.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-13T23:39:19.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-13T23:39:19.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-13T23:39:19.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-13T23:39:19.298 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-13T23:39:19.298 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-13T23:39:19.298 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-13T23:39:19.298 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:39:19.298 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:39:18 smithi047 ceph-mon[77147]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-13T23:39:19.299 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-13T23:39:19.299 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-13T23:39:19.299 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-13T23:39:19.299 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-13T23:39:19.299 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-13T23:39:19.300 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-13T23:39:19.300 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-13T23:39:19.300 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-13T23:39:19.300 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-13T23:39:19.301 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-13T23:39:19.301 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-13T23:39:19.301 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-13T23:39:19.301 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-13T23:39:19.301 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-13T23:39:19.302 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-13T23:39:19.302 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-13T23:39:19.302 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-13T23:39:19.302 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:39:19.302 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-13T23:39:19.303 INFO:journalctl@ceph.mon.b.smithi047.stdout: During handling of the above exception, another exception occurred: 2022-05-13T23:39:19.303 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-13T23:39:19.303 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-13T23:39:19.303 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-13T23:39:19.304 INFO:journalctl@ceph.mon.b.smithi047.stdout: return f(*arg) 2022-05-13T23:39:19.304 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-13T23:39:19.304 INFO:journalctl@ceph.mon.b.smithi047.stdout: self._write_client_files(client_files, host) 2022-05-13T23:39:19.304 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-13T23:39:19.304 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-13T23:39:19.305 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-13T23:39:19.305 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-13T23:39:19.305 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-13T23:39:19.305 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.event_loop.get_result(coro) 2022-05-13T23:39:19.305 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-13T23:39:19.306 INFO:journalctl@ceph.mon.b.smithi047.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-13T23:39:19.306 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-13T23:39:19.306 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.__get_result() 2022-05-13T23:39:19.306 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-13T23:39:19.307 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise self._exception 2022-05-13T23:39:19.307 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-13T23:39:19.307 INFO:journalctl@ceph.mon.b.smithi047.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-13T23:39:19.307 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-13T23:39:19.307 INFO:journalctl@ceph.mon.b.smithi047.stdout: conn = await self._remote_connection(host, addr) 2022-05-13T23:39:19.308 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-13T23:39:19.308 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise 2022-05-13T23:39:19.308 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-13T23:39:19.308 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.gen.throw(type, value, traceback) 2022-05-13T23:39:19.308 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-13T23:39:19.309 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OrchestratorError(msg) 2022-05-13T23:39:19.309 INFO:journalctl@ceph.mon.b.smithi047.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:39:19.309 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:39:18 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-05-13T23:39:19.309 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:39:18 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-05-13T23:39:19.310 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:39:18 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-05-13T23:39:19.310 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:39:18 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-05-13T23:39:19.310 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:39:18 smithi047 ceph-mon[77147]: Adjusting osd_memory_target on smithi047 to 198.1M 2022-05-13T23:39:19.310 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:39:18 smithi047 ceph-mon[77147]: Unable to set osd_memory_target on smithi047 to 207755878: error parsing value: Value '207755878' is below minimum 939524096 2022-05-13T23:39:19.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:39:19 smithi115 conmon[90393]: 2022-05-13T23:39:19.138+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:39:19.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:39:18 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:39:19.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:39:18 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' 2022-05-13T23:39:19.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:39:18 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' 2022-05-13T23:39:19.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:39:18 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:39:19.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:39:18 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 691 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:39:19.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:39:18 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-05-13T23:39:19.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:39:18 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-05-13T23:39:19.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:39:18 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-05-13T23:39:19.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:39:18 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-05-13T23:39:19.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:39:18 smithi115 ceph-mon[78593]: Adjusting osd_memory_target on smithi115 to 197.0M 2022-05-13T23:39:19.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:39:18 smithi115 ceph-mon[78593]: Unable to set osd_memory_target on smithi115 to 206629785: error parsing value: Value '206629785' is below minimum 939524096 2022-05-13T23:39:19.362 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:39:18 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:39:19.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-13T23:39:19.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-13T23:39:19.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-13T23:39:19.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-13T23:39:19.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-13T23:39:19.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-13T23:39:19.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-13T23:39:19.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-13T23:39:19.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-13T23:39:19.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-13T23:39:19.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-13T23:39:19.365 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-13T23:39:19.365 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-13T23:39:19.365 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-13T23:39:19.365 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-13T23:39:19.366 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-13T23:39:19.366 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-13T23:39:19.366 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:39:19.366 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:39:18 smithi115 ceph-mon[78593]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-13T23:39:19.367 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-13T23:39:19.367 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-13T23:39:19.367 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-13T23:39:19.367 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-13T23:39:19.367 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-13T23:39:19.368 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-13T23:39:19.368 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-13T23:39:19.368 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-13T23:39:19.369 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-13T23:39:19.369 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-13T23:39:19.369 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-13T23:39:19.369 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-13T23:39:19.370 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-13T23:39:19.370 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-13T23:39:19.370 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-13T23:39:19.370 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-13T23:39:19.371 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-13T23:39:19.371 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:39:19.371 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-13T23:39:19.371 INFO:journalctl@ceph.mon.c.smithi115.stdout: During handling of the above exception, another exception occurred: 2022-05-13T23:39:19.371 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-13T23:39:19.372 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-13T23:39:19.372 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-13T23:39:19.372 INFO:journalctl@ceph.mon.c.smithi115.stdout: return f(*arg) 2022-05-13T23:39:19.373 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-13T23:39:19.373 INFO:journalctl@ceph.mon.c.smithi115.stdout: self._write_client_files(client_files, host) 2022-05-13T23:39:19.374 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-13T23:39:19.374 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-13T23:39:19.374 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-13T23:39:19.374 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-13T23:39:19.375 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-13T23:39:19.375 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.event_loop.get_result(coro) 2022-05-13T23:39:19.375 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-13T23:39:19.375 INFO:journalctl@ceph.mon.c.smithi115.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-13T23:39:19.376 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-13T23:39:19.376 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.__get_result() 2022-05-13T23:39:19.376 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-13T23:39:19.376 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise self._exception 2022-05-13T23:39:19.377 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-13T23:39:19.377 INFO:journalctl@ceph.mon.c.smithi115.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-13T23:39:19.377 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-13T23:39:19.377 INFO:journalctl@ceph.mon.c.smithi115.stdout: conn = await self._remote_connection(host, addr) 2022-05-13T23:39:19.378 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-13T23:39:19.378 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise 2022-05-13T23:39:19.378 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-13T23:39:19.378 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.gen.throw(type, value, traceback) 2022-05-13T23:39:19.379 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-13T23:39:19.379 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OrchestratorError(msg) 2022-05-13T23:39:19.379 INFO:journalctl@ceph.mon.c.smithi115.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:39:19.379 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:39:18 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-05-13T23:39:19.379 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:39:18 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-05-13T23:39:19.380 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:39:18 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-05-13T23:39:19.380 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:39:18 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-05-13T23:39:19.380 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:39:18 smithi115 ceph-mon[78593]: Adjusting osd_memory_target on smithi047 to 198.1M 2022-05-13T23:39:19.380 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:39:18 smithi115 ceph-mon[78593]: Unable to set osd_memory_target on smithi047 to 207755878: error parsing value: Value '207755878' is below minimum 939524096 2022-05-13T23:39:20.297 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:39:19 smithi047 ceph-mon[77147]: pgmap v698: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:39:20.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:39:20 smithi115 conmon[90393]: 2022-05-13T23:39:20.094+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:39:20.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:39:19 smithi115 ceph-mon[78593]: pgmap v698: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:39:21.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:39:20 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:39:21.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:39:20 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:39:21.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:39:21 smithi115 conmon[90393]: 2022-05-13T23:39:21.098+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:39:21.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:39:20 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:39:21.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:39:20 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:39:22.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:39:21 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:39:22.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:39:21 smithi047 ceph-mon[77147]: fsmap cephfs:5/5 3 up:standby-replay 1 up:replay 4 up:active 2022-05-13T23:39:22.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:39:21 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:39:22.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:39:21 smithi047 ceph-mon[77147]: pgmap v699: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:39:22.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:39:22 smithi115 conmon[90393]: 2022-05-13T23:39:22.110+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:39:22.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:39:21 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:39:22.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:39:21 smithi115 ceph-mon[78593]: fsmap cephfs:5/5 3 up:standby-replay 1 up:replay 4 up:active 2022-05-13T23:39:22.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:39:21 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:39:22.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:39:21 smithi115 ceph-mon[78593]: pgmap v699: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:39:23.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:39:22 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:39:23.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:39:22 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:39:23.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:39:23 smithi115 conmon[90393]: 2022-05-13T23:39:23.082+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:39:24.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:39:23 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:39:24.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:39:23 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 696 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:39:24.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:39:23 smithi047 ceph-mon[77147]: pgmap v700: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:39:24.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:39:24 smithi115 conmon[90393]: 2022-05-13T23:39:24.048+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:39:24.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:39:23 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:39:24.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:39:23 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 696 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:39:24.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:39:23 smithi115 ceph-mon[78593]: pgmap v700: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:39:25.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:39:24 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:39:25.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:39:25 smithi115 conmon[90393]: 2022-05-13T23:39:25.035+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:39:25.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:39:24 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:39:26.296 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:39:25 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:39:26.296 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:39:25 smithi047 ceph-mon[77147]: pgmap v701: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:39:26.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:39:26 smithi115 conmon[90393]: 2022-05-13T23:39:26.073+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:39:26.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:39:25 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:39:26.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:39:25 smithi115 ceph-mon[78593]: pgmap v701: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:39:27.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:39:26 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:39:27.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:39:27 smithi115 conmon[90393]: 2022-05-13T23:39:27.076+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:39:27.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:39:26 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:39:28.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:39:27 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:39:28.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:39:27 smithi047 ceph-mon[77147]: pgmap v702: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:39:28.361 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:39:28 smithi115 conmon[90393]: 2022-05-13T23:39:28.034+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:39:28.362 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:39:27 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:39:28.362 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:39:27 smithi115 ceph-mon[78593]: pgmap v702: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:39:29.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:39:28 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:39:29.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:39:28 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 700 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:39:29.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:39:29 smithi115 conmon[90393]: 2022-05-13T23:39:29.054+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:39:29.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:39:28 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:39:29.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:39:28 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 700 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:39:30.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:39:29 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:39:30.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:39:29 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:39:30.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:39:29 smithi047 ceph-mon[77147]: fsmap cephfs:5/5 3 up:standby-replay 1 up:replay 4 up:active 2022-05-13T23:39:30.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:39:29 smithi047 ceph-mon[77147]: pgmap v703: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:39:30.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:39:30 smithi115 conmon[90393]: 2022-05-13T23:39:30.012+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:39:30.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:39:29 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:39:30.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:39:29 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:39:30.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:39:29 smithi115 ceph-mon[78593]: fsmap cephfs:5/5 3 up:standby-replay 1 up:replay 4 up:active 2022-05-13T23:39:30.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:39:29 smithi115 ceph-mon[78593]: pgmap v703: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:39:31.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:39:30 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:39:31.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:39:30 smithi115 conmon[90393]: 2022-05-13T23:39:30.986+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:39:31.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:39:30 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:39:32.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:39:31 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:39:32.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:39:31 smithi047 ceph-mon[77147]: pgmap v704: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:39:32.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:39:31 smithi115 conmon[90393]: 2022-05-13T23:39:31.974+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:39:32.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:39:31 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:39:32.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:39:31 smithi115 ceph-mon[78593]: pgmap v704: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:39:33.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:39:32 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:39:33.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:39:33 smithi115 conmon[90393]: 2022-05-13T23:39:33.010+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:39:33.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:39:32 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:39:34.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:39:33 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:39:34.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:39:33 smithi047 ceph-mon[77147]: fsmap cephfs:5/5 3 up:standby-replay 1 up:replay 4 up:active 2022-05-13T23:39:34.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:39:33 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:39:34.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:39:33 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 705 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:39:34.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:39:33 smithi047 ceph-mon[77147]: pgmap v705: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:39:34.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:39:34 smithi115 conmon[90393]: 2022-05-13T23:39:34.029+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:39:34.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:39:33 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:39:34.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:39:33 smithi115 ceph-mon[78593]: fsmap cephfs:5/5 3 up:standby-replay 1 up:replay 4 up:active 2022-05-13T23:39:34.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:39:33 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:39:34.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:39:33 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 705 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:39:34.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:39:33 smithi115 ceph-mon[78593]: pgmap v705: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:39:35.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:39:34 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:39:35.362 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:39:35 smithi115 conmon[90393]: 2022-05-13T23:39:35.075+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:39:35.362 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:39:34 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:39:36.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:39:35 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:39:36.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:39:35 smithi047 ceph-mon[77147]: pgmap v706: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:39:36.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:39:36 smithi115 conmon[90393]: 2022-05-13T23:39:36.105+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:39:36.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:39:35 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:39:36.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:39:35 smithi115 ceph-mon[78593]: pgmap v706: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:39:37.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:39:36 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:39:37.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:39:37 smithi115 conmon[90393]: 2022-05-13T23:39:37.062+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:39:37.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:39:36 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:39:38.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:39:37 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:39:38.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:39:37 smithi047 ceph-mon[77147]: fsmap cephfs:5/5 3 up:standby-replay 1 up:replay 4 up:active 2022-05-13T23:39:38.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:39:37 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:39:38.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:39:37 smithi047 ceph-mon[77147]: pgmap v707: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:39:38.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:39:38 smithi115 conmon[90393]: 2022-05-13T23:39:38.110+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:39:38.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:39:37 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:39:38.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:39:37 smithi115 ceph-mon[78593]: fsmap cephfs:5/5 3 up:standby-replay 1 up:replay 4 up:active 2022-05-13T23:39:38.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:39:37 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:39:38.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:39:37 smithi115 ceph-mon[78593]: pgmap v707: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:39:39.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:39:38 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:39:39.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:39:38 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 710 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:39:39.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:39:39 smithi115 conmon[90393]: 2022-05-13T23:39:39.067+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:39:39.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:39:38 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:39:39.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:39:38 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 710 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:39:40.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:39:40 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:39:40.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:39:40 smithi047 ceph-mon[77147]: pgmap v708: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:39:40.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:39:40 smithi115 conmon[90393]: 2022-05-13T23:39:40.088+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:39:40.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:39:40 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:39:40.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:39:40 smithi115 ceph-mon[78593]: pgmap v708: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:39:41.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:39:41 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:39:41.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:39:41 smithi115 conmon[90393]: 2022-05-13T23:39:41.109+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:39:41.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:39:41 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:39:42.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:39:42 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:39:42.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:39:42 smithi047 ceph-mon[77147]: fsmap cephfs:5/5 3 up:standby-replay 1 up:replay 4 up:active 2022-05-13T23:39:42.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:39:42 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:39:42.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:39:42 smithi047 ceph-mon[77147]: pgmap v709: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:39:42.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:39:42 smithi115 conmon[90393]: 2022-05-13T23:39:42.077+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:39:42.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:39:42 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:39:42.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:39:42 smithi115 ceph-mon[78593]: fsmap cephfs:5/5 3 up:standby-replay 1 up:replay 4 up:active 2022-05-13T23:39:42.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:39:42 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:39:42.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:39:42 smithi115 ceph-mon[78593]: pgmap v709: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:39:43.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:39:43 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:39:43.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:39:43 smithi115 conmon[90393]: 2022-05-13T23:39:43.121+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:39:43.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:39:43 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:39:44.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:39:44 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:39:44.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:39:44 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 715 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:39:44.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:39:44 smithi047 ceph-mon[77147]: pgmap v710: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:39:44.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:39:44 smithi115 conmon[90393]: 2022-05-13T23:39:44.143+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:39:44.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:39:44 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:39:44.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:39:44 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 715 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:39:44.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:39:44 smithi115 ceph-mon[78593]: pgmap v710: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:39:45.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:39:45 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:39:45.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:39:45 smithi115 conmon[90393]: 2022-05-13T23:39:45.159+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:39:45.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:39:45 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:39:46.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:39:46 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:39:46.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:39:46 smithi047 ceph-mon[77147]: pgmap v711: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:39:46.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:39:46 smithi115 conmon[90393]: 2022-05-13T23:39:46.194+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:39:46.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:39:46 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:39:46.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:39:46 smithi115 ceph-mon[78593]: pgmap v711: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:39:47.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:39:47 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:39:47.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:39:47 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:39:47.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:39:47 smithi115 conmon[90393]: 2022-05-13T23:39:47.166+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:39:48.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:39:48 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:39:48.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:39:48 smithi047 ceph-mon[77147]: pgmap v712: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:39:48.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:39:48 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 721 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:39:48.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:39:48 smithi115 conmon[90393]: 2022-05-13T23:39:48.167+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:39:48.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:39:48 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:39:48.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:39:48 smithi115 ceph-mon[78593]: pgmap v712: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:39:48.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:39:48 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 721 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:39:49.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:39:49 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:39:49.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:39:49 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:39:49.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:39:49 smithi115 conmon[90393]: 2022-05-13T23:39:49.127+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:39:49.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:39:49 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:39:49.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:39:49 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:39:50.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:39:50 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:39:50.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:39:50 smithi047 ceph-mon[77147]: fsmap cephfs:5/5 3 up:standby-replay 1 up:replay 4 up:active 2022-05-13T23:39:50.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:39:50 smithi047 ceph-mon[77147]: pgmap v713: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:39:50.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:39:50 smithi115 conmon[90393]: 2022-05-13T23:39:50.163+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:39:50.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:39:50 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:39:50.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:39:50 smithi115 ceph-mon[78593]: fsmap cephfs:5/5 3 up:standby-replay 1 up:replay 4 up:active 2022-05-13T23:39:50.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:39:50 smithi115 ceph-mon[78593]: pgmap v713: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:39:51.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:39:51 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:39:51.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:39:51 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:39:51.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:39:51 smithi115 conmon[90393]: 2022-05-13T23:39:51.125+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:39:51.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:39:51 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:39:51.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:39:51 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:39:52.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:39:52 smithi047 ceph-mon[77147]: pgmap v714: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:39:52.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:39:52 smithi115 conmon[90393]: 2022-05-13T23:39:52.138+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:39:52.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:39:52 smithi115 ceph-mon[78593]: pgmap v714: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:39:53.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:39:53 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:39:53.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:39:53 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 726 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:39:53.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:39:53 smithi115 conmon[90393]: 2022-05-13T23:39:53.141+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:39:53.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:39:53 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:39:53.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:39:53 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 726 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:39:54.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:39:54 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:39:54.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:39:54 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:39:54.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:39:54 smithi047 ceph-mon[77147]: fsmap cephfs:5/5 3 up:standby-replay 1 up:replay 4 up:active 2022-05-13T23:39:54.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:39:54 smithi047 ceph-mon[77147]: pgmap v715: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:39:54.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:39:54 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:39:54.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:39:54 smithi115 conmon[90393]: 2022-05-13T23:39:54.123+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:39:54.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:39:54 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:39:54.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:39:54 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:39:54.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:39:54 smithi115 ceph-mon[78593]: fsmap cephfs:5/5 3 up:standby-replay 1 up:replay 4 up:active 2022-05-13T23:39:54.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:39:54 smithi115 ceph-mon[78593]: pgmap v715: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:39:54.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:39:54 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:39:55.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:39:55 smithi115 conmon[90393]: 2022-05-13T23:39:55.152+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:39:56.515 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:39:56 smithi115 conmon[90393]: 2022-05-13T23:39:56.203+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:39:56.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:39:56 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:39:56.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:39:56 smithi047 ceph-mon[77147]: pgmap v716: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:39:56.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:39:56 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:39:56.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:39:56 smithi115 ceph-mon[78593]: pgmap v716: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:39:57.529 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:39:57 smithi115 conmon[90393]: 2022-05-13T23:39:57.234+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:39:57.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:39:57 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:39:57.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:39:57 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:39:57.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:39:57 smithi047 ceph-mon[77147]: fsmap cephfs:5/5 3 up:standby-replay 1 up:replay 4 up:active 2022-05-13T23:39:57.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:39:57 smithi047 ceph-mon[77147]: pgmap v717: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:39:57.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:39:57 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:39:57.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:39:57 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:39:57.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:39:57 smithi115 ceph-mon[78593]: fsmap cephfs:5/5 3 up:standby-replay 1 up:replay 4 up:active 2022-05-13T23:39:57.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:39:57 smithi115 ceph-mon[78593]: pgmap v717: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:39:58.528 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:39:58 smithi115 conmon[90393]: 2022-05-13T23:39:58.252+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:39:58.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:39:58 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:39:58.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:39:58 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 731 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:39:58.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:39:58 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:39:58.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:39:58 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 731 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:39:59.532 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:39:59 smithi115 conmon[90393]: 2022-05-13T23:39:59.214+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:39:59.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:39:59 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:39:59.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:39:59 smithi047 ceph-mon[77147]: pgmap v718: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:39:59.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:39:59 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:39:59.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:39:59 smithi115 ceph-mon[78593]: pgmap v718: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:40:00.538 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:40:00 smithi115 conmon[90393]: 2022-05-13T23:40:00.215+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:40:00.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:40:00 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:40:00.841 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:40:00 smithi047 ceph-mon[77147]: Health detail: HEALTH_WARN 1 filesystem is degraded; 1 MDSs are read only; 1 MDSs report slow metadata IOs; 1/3 mons down, quorum b,c; Reduced data availability: 1 pg inactive; 9 slow ops, oldest one blocked for 735 sec, osd.5 has slow ops 2022-05-13T23:40:00.841 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:40:00 smithi047 ceph-mon[77147]: [WRN] FS_DEGRADED: 1 filesystem is degraded 2022-05-13T23:40:00.842 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:40:00 smithi047 ceph-mon[77147]: fs cephfs is degraded 2022-05-13T23:40:00.842 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:40:00 smithi047 ceph-mon[77147]: [WRN] MDS_READ_ONLY: 1 MDSs are read only 2022-05-13T23:40:00.842 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:40:00 smithi047 ceph-mon[77147]: mds.l(mds.0): MDS in read-only mode 2022-05-13T23:40:00.842 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:40:00 smithi047 ceph-mon[77147]: [WRN] MDS_SLOW_METADATA_IO: 1 MDSs report slow metadata IOs 2022-05-13T23:40:00.843 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:40:00 smithi047 ceph-mon[77147]: mds.l(mds.0): 4 slow metadata IOs are blocked > 180 secs, oldest blocked for 1033 secs 2022-05-13T23:40:00.843 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:40:00 smithi047 ceph-mon[77147]: [WRN] MON_DOWN: 1/3 mons down, quorum b,c 2022-05-13T23:40:00.844 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:40:00 smithi047 ceph-mon[77147]: mon.a (rank 0) addr [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] is down (out of quorum) 2022-05-13T23:40:00.844 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:40:00 smithi047 ceph-mon[77147]: [WRN] PG_AVAILABILITY: Reduced data availability: 1 pg inactive 2022-05-13T23:40:00.844 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:40:00 smithi047 ceph-mon[77147]: pg 2.1 is stuck inactive for 24m, current state unknown, last acting [] 2022-05-13T23:40:00.844 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:40:00 smithi047 ceph-mon[77147]: [WRN] SLOW_OPS: 9 slow ops, oldest one blocked for 735 sec, osd.5 has slow ops 2022-05-13T23:40:00.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:40:00 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:40:00.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:40:00 smithi115 ceph-mon[78593]: Health detail: HEALTH_WARN 1 filesystem is degraded; 1 MDSs are read only; 1 MDSs report slow metadata IOs; 1/3 mons down, quorum b,c; Reduced data availability: 1 pg inactive; 9 slow ops, oldest one blocked for 735 sec, osd.5 has slow ops 2022-05-13T23:40:00.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:40:00 smithi115 ceph-mon[78593]: [WRN] FS_DEGRADED: 1 filesystem is degraded 2022-05-13T23:40:00.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:40:00 smithi115 ceph-mon[78593]: fs cephfs is degraded 2022-05-13T23:40:00.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:40:00 smithi115 ceph-mon[78593]: [WRN] MDS_READ_ONLY: 1 MDSs are read only 2022-05-13T23:40:00.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:40:00 smithi115 ceph-mon[78593]: mds.l(mds.0): MDS in read-only mode 2022-05-13T23:40:00.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:40:00 smithi115 ceph-mon[78593]: [WRN] MDS_SLOW_METADATA_IO: 1 MDSs report slow metadata IOs 2022-05-13T23:40:00.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:40:00 smithi115 ceph-mon[78593]: mds.l(mds.0): 4 slow metadata IOs are blocked > 180 secs, oldest blocked for 1033 secs 2022-05-13T23:40:00.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:40:00 smithi115 ceph-mon[78593]: [WRN] MON_DOWN: 1/3 mons down, quorum b,c 2022-05-13T23:40:00.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:40:00 smithi115 ceph-mon[78593]: mon.a (rank 0) addr [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] is down (out of quorum) 2022-05-13T23:40:00.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:40:00 smithi115 ceph-mon[78593]: [WRN] PG_AVAILABILITY: Reduced data availability: 1 pg inactive 2022-05-13T23:40:00.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:40:00 smithi115 ceph-mon[78593]: pg 2.1 is stuck inactive for 24m, current state unknown, last acting [] 2022-05-13T23:40:00.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:40:00 smithi115 ceph-mon[78593]: [WRN] SLOW_OPS: 9 slow ops, oldest one blocked for 735 sec, osd.5 has slow ops 2022-05-13T23:40:01.546 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:40:01 smithi115 conmon[90393]: 2022-05-13T23:40:01.181+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:40:01.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:40:01 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:40:01.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:40:01 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:40:01.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:40:01 smithi115 ceph-mon[78593]: fsmap cephfs:5/5 3 up:standby-replay 1 up:replay 4 up:active 2022-05-13T23:40:01.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:40:01 smithi115 ceph-mon[78593]: pgmap v719: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:40:02.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:40:01 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:40:02.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:40:01 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:40:02.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:40:01 smithi047 ceph-mon[77147]: fsmap cephfs:5/5 3 up:standby-replay 1 up:replay 4 up:active 2022-05-13T23:40:02.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:40:01 smithi047 ceph-mon[77147]: pgmap v719: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:40:02.545 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:40:02 smithi115 conmon[90393]: 2022-05-13T23:40:02.159+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:40:02.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:40:02 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:40:03.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:40:02 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:40:03.547 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:40:03 smithi115 conmon[90393]: 2022-05-13T23:40:03.117+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:40:03.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:40:03 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:40:03.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:40:03 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:40:03.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:40:03 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 735 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:40:03.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:40:03 smithi115 ceph-mon[78593]: pgmap v720: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:40:04.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:40:03 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:40:04.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:40:03 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:40:04.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:40:03 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 735 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:40:04.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:40:03 smithi047 ceph-mon[77147]: pgmap v720: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:40:04.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:40:04 smithi115 conmon[90393]: 2022-05-13T23:40:04.133+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:40:05.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:40:05 smithi115 conmon[90393]: 2022-05-13T23:40:05.102+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:40:05.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:40:05 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:40:05.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:40:05 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:40:05.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:40:05 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:40:05.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:40:05 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:40:06.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:40:06 smithi047 ceph-mon[77147]: pgmap v721: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:40:06.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:40:06 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:40:06.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:40:06 smithi115 conmon[90393]: 2022-05-13T23:40:06.113+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:40:06.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:40:06 smithi115 ceph-mon[78593]: pgmap v721: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:40:06.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:40:06 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:40:07.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:40:07 smithi115 conmon[90393]: 2022-05-13T23:40:07.157+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:40:08.519 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:40:08 smithi115 conmon[90393]: 2022-05-13T23:40:08.186+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:40:08.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:40:08 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:40:08.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:40:08 smithi047 ceph-mon[77147]: pgmap v722: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:40:08.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:40:08 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 741 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:40:08.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:40:08 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:40:08.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:40:08 smithi115 ceph-mon[78593]: pgmap v722: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:40:08.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:40:08 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 741 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:40:09.520 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:40:09 smithi115 conmon[90393]: 2022-05-13T23:40:09.180+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:40:09.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:40:09 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:40:09.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:40:09 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:40:09.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:40:09 smithi047 ceph-mon[77147]: fsmap cephfs:5/5 3 up:standby-replay 1 up:replay 4 up:active 2022-05-13T23:40:09.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:40:09 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:40:09.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:40:09 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:40:09.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:40:09 smithi115 ceph-mon[78593]: fsmap cephfs:5/5 3 up:standby-replay 1 up:replay 4 up:active 2022-05-13T23:40:10.521 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:40:10 smithi115 conmon[90393]: 2022-05-13T23:40:10.196+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:40:10.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:40:10 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:40:10.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:40:10 smithi047 ceph-mon[77147]: pgmap v723: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:40:10.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:40:10 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:40:10.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:40:10 smithi115 ceph-mon[78593]: pgmap v723: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:40:11.527 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:40:11 smithi115 conmon[90393]: 2022-05-13T23:40:11.235+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:40:11.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:40:11 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:40:11.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:40:11 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:40:12.528 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:40:12 smithi115 conmon[90393]: 2022-05-13T23:40:12.225+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:40:12.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:40:12 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:40:12.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:40:12 smithi047 ceph-mon[77147]: pgmap v724: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:40:12.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:40:12 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:40:12.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:40:12 smithi115 ceph-mon[78593]: pgmap v724: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:40:13.533 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:40:13 smithi115 conmon[90393]: 2022-05-13T23:40:13.268+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:40:13.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:40:13 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:40:13.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:40:13 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:40:13.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:40:13 smithi047 ceph-mon[77147]: fsmap cephfs:5/5 3 up:standby-replay 1 up:replay 4 up:active 2022-05-13T23:40:13.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:40:13 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-13T23:40:13.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:40:13 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-13T23:40:13.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:40:13 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 745 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:40:13.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:40:13 smithi047 ceph-mon[77147]: pgmap v725: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:40:13.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:40:13 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:40:13.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:40:13 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:40:13.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:40:13 smithi115 ceph-mon[78593]: fsmap cephfs:5/5 3 up:standby-replay 1 up:replay 4 up:active 2022-05-13T23:40:13.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:40:13 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-13T23:40:13.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:40:13 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-13T23:40:13.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:40:13 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 745 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:40:13.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:40:13 smithi115 ceph-mon[78593]: pgmap v725: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:40:14.534 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:40:14 smithi115 conmon[90393]: 2022-05-13T23:40:14.243+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:40:14.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:40:14 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:40:14.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:40:14 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:40:15.540 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:40:15 smithi115 conmon[90393]: 2022-05-13T23:40:15.250+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:40:15.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:40:15 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:40:15.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:40:15 smithi047 ceph-mon[77147]: pgmap v726: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:40:15.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:40:15 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:40:15.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:40:15 smithi115 ceph-mon[78593]: pgmap v726: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:40:16.546 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:40:16 smithi115 conmon[90393]: 2022-05-13T23:40:16.269+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:40:16.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:40:16 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:40:17.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:40:16 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:40:17.551 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:40:17 smithi115 conmon[90393]: 2022-05-13T23:40:17.229+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:40:17.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:40:17 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:40:17.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:40:17 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:40:17.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:40:17 smithi115 ceph-mon[78593]: fsmap cephfs:5/5 3 up:standby-replay 1 up:replay 4 up:active 2022-05-13T23:40:17.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:40:17 smithi115 ceph-mon[78593]: pgmap v727: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:40:18.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:40:17 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:40:18.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:40:17 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:40:18.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:40:17 smithi047 ceph-mon[77147]: fsmap cephfs:5/5 3 up:standby-replay 1 up:replay 4 up:active 2022-05-13T23:40:18.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:40:17 smithi047 ceph-mon[77147]: pgmap v727: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:40:18.513 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:40:18 smithi115 conmon[90393]: 2022-05-13T23:40:18.251+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:40:18.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:40:18 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:40:18.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:40:18 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 750 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:40:18.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:40:18 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-13T23:40:18.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:40:18 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-13T23:40:18.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:40:18 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-13T23:40:18.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:40:18 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:40:18.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:40:18 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 750 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:40:18.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:40:18 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-13T23:40:18.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:40:18 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-13T23:40:18.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:40:18 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-13T23:40:19.564 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:40:19 smithi115 conmon[90393]: 2022-05-13T23:40:19.219+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:40:19.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:40:19 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:40:19.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:40:19 smithi115 ceph-mon[78593]: pgmap v728: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:40:19.891 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:40:19 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:40:19.891 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:40:19 smithi047 ceph-mon[77147]: pgmap v728: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:40:20.291 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:40:19 smithi047 conmon[78521]: 2022-05-13T23:40:19.888+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:40:20.292 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:40:19 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-13T23:40:20.292 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:40:19 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-13T23:40:20.292 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:40:19 smithi047 conmon[78521]: yield 2022-05-13T23:40:20.292 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:40:19 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-13T23:40:20.292 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:40:19 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-13T23:40:20.293 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:40:19 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-13T23:40:20.293 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:40:19 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-13T23:40:20.293 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:40:19 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-13T23:40:20.293 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:40:19 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-13T23:40:20.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:40:19 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-13T23:40:20.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:40:19 smithi047 conmon[78521]: raise exceptions[0] 2022-05-13T23:40:20.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:40:19 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-13T23:40:20.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:40:19 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-13T23:40:20.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:40:19 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-13T23:40:20.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:40:19 smithi047 conmon[78521]: return (yield from fut) 2022-05-13T23:40:20.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:40:19 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-13T23:40:20.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:40:19 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-13T23:40:20.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:40:19 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:40:20.563 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:40:20 smithi115 conmon[90393]: 2022-05-13T23:40:20.255+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:40:20.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:40:20 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:40:20.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:40:20 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:40:20.860 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-13T23:40:20.860 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-13T23:40:20.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-13T23:40:20.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-13T23:40:20.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-13T23:40:20.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-13T23:40:20.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-13T23:40:20.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-13T23:40:20.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-13T23:40:20.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-13T23:40:20.863 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-13T23:40:20.863 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-13T23:40:20.863 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-13T23:40:20.863 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-13T23:40:20.863 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-13T23:40:20.864 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-13T23:40:20.864 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-13T23:40:20.864 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:40:20.865 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:40:20 smithi115 ceph-mon[78593]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-13T23:40:21.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:40:20 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:40:21.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:40:20 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:40:21.042 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-13T23:40:21.042 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-13T23:40:21.042 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-13T23:40:21.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-13T23:40:21.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-13T23:40:21.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-13T23:40:21.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-13T23:40:21.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-13T23:40:21.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-13T23:40:21.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-13T23:40:21.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-13T23:40:21.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-13T23:40:21.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-13T23:40:21.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-13T23:40:21.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-13T23:40:21.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-13T23:40:21.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-13T23:40:21.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:40:21.046 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:40:20 smithi047 ceph-mon[77147]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-13T23:40:21.570 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:40:21 smithi115 conmon[90393]: 2022-05-13T23:40:21.230+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:40:21.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:40:21 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:40:21.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:40:21 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:40:21.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:40:21 smithi115 ceph-mon[78593]: fsmap cephfs:5/5 3 up:standby-replay 1 up:replay 4 up:active 2022-05-13T23:40:21.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:40:21 smithi115 ceph-mon[78593]: pgmap v729: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:40:22.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:40:21 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:40:22.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:40:21 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:40:22.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:40:21 smithi047 ceph-mon[77147]: fsmap cephfs:5/5 3 up:standby-replay 1 up:replay 4 up:active 2022-05-13T23:40:22.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:40:21 smithi047 ceph-mon[77147]: pgmap v729: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:40:22.570 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:40:22 smithi115 conmon[90393]: 2022-05-13T23:40:22.187+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:40:22.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:40:22 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:40:22.961 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:40:22 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:40:23.292 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:40:22 smithi047 conmon[78521]: 2022-05-13T23:40:22.960+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:40:23.292 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:40:22 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-13T23:40:23.293 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:40:22 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-13T23:40:23.293 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:40:22 smithi047 conmon[78521]: yield 2022-05-13T23:40:23.293 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:40:22 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-13T23:40:23.293 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:40:22 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-13T23:40:23.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:40:22 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-13T23:40:23.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:40:22 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-13T23:40:23.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:40:22 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-13T23:40:23.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:40:22 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-13T23:40:23.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:40:22 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-13T23:40:23.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:40:22 smithi047 conmon[78521]: raise exceptions[0] 2022-05-13T23:40:23.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:40:22 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-13T23:40:23.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:40:22 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-13T23:40:23.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:40:22 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-13T23:40:23.297 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:40:22 smithi047 conmon[78521]: return (yield from fut) 2022-05-13T23:40:23.297 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:40:22 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-13T23:40:23.297 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:40:22 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-13T23:40:23.297 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:40:22 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:40:23.298 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:40:22 smithi047 conmon[78521]: 2022-05-13T23:40:22.961+0000 7f5340f7a700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-13T23:40:23.298 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:40:22 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-13T23:40:23.298 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:40:22 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-13T23:40:23.298 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:40:22 smithi047 conmon[78521]: yield 2022-05-13T23:40:23.298 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:40:22 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-13T23:40:23.299 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:40:22 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-13T23:40:23.299 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:40:22 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-13T23:40:23.299 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:40:22 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-13T23:40:23.299 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:40:22 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-13T23:40:23.300 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:40:22 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-13T23:40:23.300 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:40:22 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-13T23:40:23.300 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:40:22 smithi047 conmon[78521]: raise exceptions[0] 2022-05-13T23:40:23.300 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:40:22 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-13T23:40:23.300 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:40:22 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-13T23:40:23.301 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:40:22 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-13T23:40:23.301 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:40:22 smithi047 conmon[78521]: return (yield from fut) 2022-05-13T23:40:23.301 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:40:22 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-13T23:40:23.301 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:40:22 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-13T23:40:23.301 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:40:22 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:40:23.302 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:40:22 smithi047 conmon[78521]: 2022-05-13T23:40:23.302 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:40:22 smithi047 conmon[78521]: During handling of the above exception, another exception occurred: 2022-05-13T23:40:23.302 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:40:22 smithi047 conmon[78521]: 2022-05-13T23:40:23.302 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:40:22 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-13T23:40:23.303 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:40:22 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-13T23:40:23.303 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:40:22 smithi047 conmon[78521]: return f(*arg) 2022-05-13T23:40:23.303 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:40:22 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-13T23:40:23.303 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:40:22 smithi047 conmon[78521]: self._write_client_files(client_files, host) 2022-05-13T23:40:23.303 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:40:22 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-13T23:40:23.304 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:40:22 smithi047 conmon[78521]: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-13T23:40:23.304 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:40:22 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-13T23:40:23.304 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:40:22 smithi047 conmon[78521]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-13T23:40:23.304 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:40:22 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-13T23:40:23.305 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:40:22 smithi047 conmon[78521]: return self.event_loop.get_result(coro) 2022-05-13T23:40:23.305 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:40:22 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-13T23:40:23.305 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:40:22 smithi047 conmon[78521]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-13T23:40:23.305 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:40:22 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-13T23:40:23.305 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:40:22 smithi047 conmon[78521]: return self.__get_result() 2022-05-13T23:40:23.306 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:40:22 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-13T23:40:23.306 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:40:22 smithi047 conmon[78521]: raise self._exception 2022-05-13T23:40:23.306 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:40:22 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-13T23:40:23.306 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:40:22 smithi047 conmon[78521]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-13T23:40:23.306 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:40:22 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-13T23:40:23.307 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:40:22 smithi047 conmon[78521]: conn = await self._remote_connection(host, addr) 2022-05-13T23:40:23.307 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:40:22 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-13T23:40:23.307 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:40:22 smithi047 conmon[78521]: raise 2022-05-13T23:40:23.307 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:40:22 smithi047 conmon[78521]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-13T23:40:23.308 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:40:22 smithi047 conmon[78521]: self.gen.throw(type, value, traceback) 2022-05-13T23:40:23.308 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:40:22 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-13T23:40:23.308 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:40:22 smithi047 conmon[78521]: raise OrchestratorError(msg) 2022-05-13T23:40:23.308 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:40:22 smithi047 conmon[78521]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:40:23.573 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:40:23 smithi115 conmon[90393]: 2022-05-13T23:40:23.237+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:40:23.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:40:23 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:40:23.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:40:23 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:40:23.859 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-13T23:40:23.859 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-13T23:40:23.860 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-13T23:40:23.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-13T23:40:23.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-13T23:40:23.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-13T23:40:23.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-13T23:40:23.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-13T23:40:23.863 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-13T23:40:23.863 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-13T23:40:23.864 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-13T23:40:23.865 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-13T23:40:23.865 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-13T23:40:23.865 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-13T23:40:23.865 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-13T23:40:23.866 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-13T23:40:23.866 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-13T23:40:23.866 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:40:23.866 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:40:23 smithi115 ceph-mon[78593]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-13T23:40:23.867 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-13T23:40:23.867 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-13T23:40:23.867 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-13T23:40:23.867 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-13T23:40:23.867 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-13T23:40:23.868 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-13T23:40:23.868 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-13T23:40:23.868 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-13T23:40:23.869 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-13T23:40:23.869 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-13T23:40:23.869 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-13T23:40:23.869 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-13T23:40:23.870 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-13T23:40:23.870 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-13T23:40:23.870 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-13T23:40:23.870 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-13T23:40:23.870 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-13T23:40:23.871 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:40:23.871 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-13T23:40:23.871 INFO:journalctl@ceph.mon.c.smithi115.stdout: During handling of the above exception, another exception occurred: 2022-05-13T23:40:23.871 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-13T23:40:23.872 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-13T23:40:23.872 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-13T23:40:23.872 INFO:journalctl@ceph.mon.c.smithi115.stdout: return f(*arg) 2022-05-13T23:40:23.872 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-13T23:40:23.872 INFO:journalctl@ceph.mon.c.smithi115.stdout: self._write_client_files(client_files, host) 2022-05-13T23:40:23.873 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-13T23:40:23.873 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-13T23:40:23.873 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-13T23:40:23.873 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-13T23:40:23.874 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-13T23:40:23.874 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.event_loop.get_result(coro) 2022-05-13T23:40:23.874 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-13T23:40:23.874 INFO:journalctl@ceph.mon.c.smithi115.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-13T23:40:23.874 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-13T23:40:23.875 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.__get_result() 2022-05-13T23:40:23.875 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-13T23:40:23.875 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise self._exception 2022-05-13T23:40:23.875 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-13T23:40:23.876 INFO:journalctl@ceph.mon.c.smithi115.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-13T23:40:23.876 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-13T23:40:23.876 INFO:journalctl@ceph.mon.c.smithi115.stdout: conn = await self._remote_connection(host, addr) 2022-05-13T23:40:23.876 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-13T23:40:23.877 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise 2022-05-13T23:40:23.877 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-13T23:40:23.877 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.gen.throw(type, value, traceback) 2022-05-13T23:40:23.877 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-13T23:40:23.878 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OrchestratorError(msg) 2022-05-13T23:40:23.878 INFO:journalctl@ceph.mon.c.smithi115.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:40:23.878 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:40:23 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 755 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:40:23.878 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:40:23 smithi115 ceph-mon[78593]: pgmap v730: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:40:24.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:40:23 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:40:24.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:40:23 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:40:24.042 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-13T23:40:24.042 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-13T23:40:24.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-13T23:40:24.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-13T23:40:24.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-13T23:40:24.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-13T23:40:24.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-13T23:40:24.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-13T23:40:24.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-13T23:40:24.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-13T23:40:24.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-13T23:40:24.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-13T23:40:24.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-13T23:40:24.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-13T23:40:24.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-13T23:40:24.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-13T23:40:24.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-13T23:40:24.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:40:24.046 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:40:23 smithi047 ceph-mon[77147]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-13T23:40:24.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-13T23:40:24.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-13T23:40:24.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-13T23:40:24.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-13T23:40:24.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-13T23:40:24.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-13T23:40:24.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-13T23:40:24.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-13T23:40:24.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-13T23:40:24.049 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-13T23:40:24.049 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-13T23:40:24.050 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-13T23:40:24.050 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-13T23:40:24.050 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-13T23:40:24.051 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-13T23:40:24.051 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-13T23:40:24.051 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-13T23:40:24.051 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:40:24.052 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-13T23:40:24.052 INFO:journalctl@ceph.mon.b.smithi047.stdout: During handling of the above exception, another exception occurred: 2022-05-13T23:40:24.052 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-13T23:40:24.052 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-13T23:40:24.053 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-13T23:40:24.053 INFO:journalctl@ceph.mon.b.smithi047.stdout: return f(*arg) 2022-05-13T23:40:24.053 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-13T23:40:24.053 INFO:journalctl@ceph.mon.b.smithi047.stdout: self._write_client_files(client_files, host) 2022-05-13T23:40:24.054 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-13T23:40:24.054 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-13T23:40:24.054 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-13T23:40:24.054 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-13T23:40:24.055 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-13T23:40:24.055 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.event_loop.get_result(coro) 2022-05-13T23:40:24.055 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-13T23:40:24.056 INFO:journalctl@ceph.mon.b.smithi047.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-13T23:40:24.056 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-13T23:40:24.056 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.__get_result() 2022-05-13T23:40:24.056 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-13T23:40:24.056 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise self._exception 2022-05-13T23:40:24.057 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-13T23:40:24.057 INFO:journalctl@ceph.mon.b.smithi047.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-13T23:40:24.057 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-13T23:40:24.057 INFO:journalctl@ceph.mon.b.smithi047.stdout: conn = await self._remote_connection(host, addr) 2022-05-13T23:40:24.058 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-13T23:40:24.058 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise 2022-05-13T23:40:24.058 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-13T23:40:24.058 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.gen.throw(type, value, traceback) 2022-05-13T23:40:24.059 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-13T23:40:24.059 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OrchestratorError(msg) 2022-05-13T23:40:24.059 INFO:journalctl@ceph.mon.b.smithi047.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:40:24.059 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:40:23 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 755 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:40:24.060 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:40:23 smithi047 ceph-mon[77147]: pgmap v730: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:40:24.578 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:40:24 smithi115 conmon[90393]: 2022-05-13T23:40:24.201+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:40:24.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:40:24 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:40:25.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:40:24 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:40:25.582 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:40:25 smithi115 conmon[90393]: 2022-05-13T23:40:25.185+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:40:25.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:40:25 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:40:25.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:40:25 smithi115 ceph-mon[78593]: pgmap v731: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:40:26.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:40:25 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:40:26.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:40:25 smithi047 ceph-mon[77147]: pgmap v731: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:40:26.581 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:40:26 smithi115 conmon[90393]: 2022-05-13T23:40:26.165+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:40:26.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:40:26 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:40:27.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:40:26 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:40:27.589 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:40:27 smithi115 conmon[90393]: 2022-05-13T23:40:27.172+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:40:27.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:40:27 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:40:27.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:40:27 smithi115 ceph-mon[78593]: pgmap v732: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:40:28.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:40:27 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:40:28.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:40:27 smithi047 ceph-mon[77147]: pgmap v732: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:40:28.590 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:40:28 smithi115 conmon[90393]: 2022-05-13T23:40:28.216+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:40:28.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:40:28 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:40:28.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:40:28 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 760 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:40:29.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:40:28 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:40:29.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:40:28 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 760 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:40:29.595 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:40:29 smithi115 conmon[90393]: 2022-05-13T23:40:29.240+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:40:29.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:40:29 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:40:29.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:40:29 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:40:29.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:40:29 smithi115 ceph-mon[78593]: fsmap cephfs:5/5 3 up:standby-replay 1 up:replay 4 up:active 2022-05-13T23:40:29.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:40:29 smithi115 ceph-mon[78593]: pgmap v733: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:40:30.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:40:29 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:40:30.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:40:29 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:40:30.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:40:29 smithi047 ceph-mon[77147]: fsmap cephfs:5/5 3 up:standby-replay 1 up:replay 4 up:active 2022-05-13T23:40:30.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:40:29 smithi047 ceph-mon[77147]: pgmap v733: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:40:30.597 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:40:30 smithi115 conmon[90393]: 2022-05-13T23:40:30.252+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:40:30.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:40:30 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:40:31.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:40:30 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:40:31.600 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:40:31 smithi115 conmon[90393]: 2022-05-13T23:40:31.205+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:40:31.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:40:31 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:40:31.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:40:31 smithi115 ceph-mon[78593]: pgmap v734: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:40:32.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:40:31 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:40:32.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:40:31 smithi047 ceph-mon[77147]: pgmap v734: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:40:32.606 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:40:32 smithi115 conmon[90393]: 2022-05-13T23:40:32.218+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:40:32.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:40:32 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:40:33.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:40:32 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:40:33.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:40:33 smithi115 conmon[90393]: 2022-05-13T23:40:33.183+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:40:34.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:40:33 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:40:34.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:40:33 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:40:34.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:40:33 smithi047 ceph-mon[77147]: fsmap cephfs:5/5 3 up:standby-replay 1 up:replay 4 up:active 2022-05-13T23:40:34.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:40:33 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 765 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:40:34.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:40:33 smithi047 ceph-mon[77147]: pgmap v735: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:40:34.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:40:33 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:40:34.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:40:33 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:40:34.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:40:33 smithi115 ceph-mon[78593]: fsmap cephfs:5/5 3 up:standby-replay 1 up:replay 4 up:active 2022-05-13T23:40:34.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:40:33 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 765 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:40:34.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:40:33 smithi115 ceph-mon[78593]: pgmap v735: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:40:34.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:40:34 smithi115 conmon[90393]: 2022-05-13T23:40:34.163+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:40:35.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:40:34 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:40:35.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:40:34 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:40:35.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:40:35 smithi115 conmon[90393]: 2022-05-13T23:40:35.145+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:40:36.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:40:35 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:40:36.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:40:35 smithi047 ceph-mon[77147]: pgmap v736: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:40:36.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:40:35 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:40:36.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:40:35 smithi115 ceph-mon[78593]: pgmap v736: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:40:36.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:40:36 smithi115 conmon[90393]: 2022-05-13T23:40:36.140+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:40:37.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:40:36 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:40:37.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:40:36 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:40:37.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:40:37 smithi115 conmon[90393]: 2022-05-13T23:40:37.125+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:40:38.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:40:37 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:40:38.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:40:37 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:40:38.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:40:37 smithi047 ceph-mon[77147]: fsmap cephfs:5/5 3 up:standby-replay 1 up:replay 4 up:active 2022-05-13T23:40:38.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:40:37 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:40:38.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:40:37 smithi047 ceph-mon[77147]: pgmap v737: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:40:38.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:40:37 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:40:38.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:40:37 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:40:38.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:40:37 smithi115 ceph-mon[78593]: fsmap cephfs:5/5 3 up:standby-replay 1 up:replay 4 up:active 2022-05-13T23:40:38.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:40:37 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:40:38.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:40:37 smithi115 ceph-mon[78593]: pgmap v737: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:40:38.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:40:38 smithi115 conmon[90393]: 2022-05-13T23:40:38.137+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:40:39.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:40:38 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:40:39.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:40:38 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 770 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:40:39.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:40:38 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:40:39.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:40:38 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 770 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:40:39.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:40:39 smithi115 conmon[90393]: 2022-05-13T23:40:39.137+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:40:40.045 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:40:39 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:40:40.046 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:40:39 smithi047 ceph-mon[77147]: pgmap v738: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:40:40.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:40:39 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:40:40.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:40:39 smithi115 ceph-mon[78593]: pgmap v738: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:40:40.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:40:40 smithi115 conmon[90393]: 2022-05-13T23:40:40.138+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:40:41.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:40:41 smithi115 conmon[90393]: 2022-05-13T23:40:41.177+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:40:42.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:40:41 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:40:42.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:40:41 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:40:42.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:40:41 smithi047 ceph-mon[77147]: fsmap cephfs:5/5 3 up:standby-replay 1 up:replay 4 up:active 2022-05-13T23:40:42.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:40:41 smithi047 ceph-mon[77147]: pgmap v739: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:40:42.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:40:41 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:40:42.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:40:41 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:40:42.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:40:41 smithi115 ceph-mon[78593]: fsmap cephfs:5/5 3 up:standby-replay 1 up:replay 4 up:active 2022-05-13T23:40:42.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:40:41 smithi115 ceph-mon[78593]: pgmap v739: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:40:42.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:40:42 smithi115 conmon[90393]: 2022-05-13T23:40:42.145+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:40:43.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:40:42 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:40:43.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:40:42 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:40:43.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:40:43 smithi115 conmon[90393]: 2022-05-13T23:40:43.117+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:40:44.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:40:43 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:40:44.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:40:43 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:40:44.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:40:43 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 776 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:40:44.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:40:43 smithi047 ceph-mon[77147]: pgmap v740: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:40:44.093 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:40:43 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:40:44.094 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:40:43 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:40:44.094 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:40:43 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 776 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:40:44.094 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:40:43 smithi115 ceph-mon[78593]: pgmap v740: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:40:44.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:40:44 smithi115 conmon[90393]: 2022-05-13T23:40:44.092+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:40:45.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:40:44 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:40:45.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:40:44 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:40:45.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:40:45 smithi115 conmon[90393]: 2022-05-13T23:40:45.116+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:40:46.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:40:45 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:40:46.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:40:45 smithi047 ceph-mon[77147]: pgmap v741: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:40:46.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:40:45 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:40:46.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:40:45 smithi115 ceph-mon[78593]: pgmap v741: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:40:46.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:40:46 smithi115 conmon[90393]: 2022-05-13T23:40:46.126+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:40:47.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:40:46 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:40:47.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:40:46 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:40:47.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:40:47 smithi115 conmon[90393]: 2022-05-13T23:40:47.120+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:40:48.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:40:47 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:40:48.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:40:47 smithi047 ceph-mon[77147]: pgmap v742: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:40:48.072 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:40:47 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:40:48.073 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:40:47 smithi115 ceph-mon[78593]: pgmap v742: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:40:48.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:40:48 smithi115 conmon[90393]: 2022-05-13T23:40:48.071+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:40:49.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:40:48 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:40:49.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:40:48 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 781 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:40:49.079 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:40:48 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:40:49.079 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:40:48 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 781 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:40:49.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:40:49 smithi115 conmon[90393]: 2022-05-13T23:40:49.077+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:40:50.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:40:49 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:40:50.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:40:49 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:40:50.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:40:49 smithi047 ceph-mon[77147]: fsmap cephfs:5/5 3 up:standby-replay 1 up:replay 4 up:active 2022-05-13T23:40:50.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:40:49 smithi047 ceph-mon[77147]: pgmap v743: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:40:50.084 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:40:49 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:40:50.085 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:40:49 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:40:50.085 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:40:49 smithi115 ceph-mon[78593]: fsmap cephfs:5/5 3 up:standby-replay 1 up:replay 4 up:active 2022-05-13T23:40:50.086 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:40:49 smithi115 ceph-mon[78593]: pgmap v743: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:40:50.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:40:50 smithi115 conmon[90393]: 2022-05-13T23:40:50.083+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:40:51.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:40:50 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:40:51.098 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:40:50 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:40:51.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:40:51 smithi115 conmon[90393]: 2022-05-13T23:40:51.097+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:40:52.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:40:51 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:40:52.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:40:51 smithi047 ceph-mon[77147]: pgmap v744: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:40:52.051 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:40:51 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:40:52.052 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:40:51 smithi115 ceph-mon[78593]: pgmap v744: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:40:52.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:40:52 smithi115 conmon[90393]: 2022-05-13T23:40:52.050+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:40:53.011 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:40:52 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:40:53.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:40:52 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:40:53.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:40:53 smithi115 conmon[90393]: 2022-05-13T23:40:53.010+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:40:54.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:40:53 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:40:54.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:40:53 smithi047 ceph-mon[77147]: fsmap cephfs:5/5 3 up:standby-replay 1 up:replay 4 up:active 2022-05-13T23:40:54.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:40:53 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:40:54.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:40:53 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 785 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:40:54.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:40:53 smithi047 ceph-mon[77147]: pgmap v745: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:40:54.060 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:40:53 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:40:54.060 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:40:53 smithi115 ceph-mon[78593]: fsmap cephfs:5/5 3 up:standby-replay 1 up:replay 4 up:active 2022-05-13T23:40:54.060 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:40:53 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:40:54.060 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:40:53 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 785 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:40:54.061 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:40:53 smithi115 ceph-mon[78593]: pgmap v745: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:40:54.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:40:54 smithi115 conmon[90393]: 2022-05-13T23:40:54.058+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:40:55.022 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:40:54 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:40:55.045 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:40:54 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:40:55.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:40:55 smithi115 conmon[90393]: 2022-05-13T23:40:55.021+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:40:55.985 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:40:55 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:40:55.986 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:40:55 smithi115 ceph-mon[78593]: pgmap v746: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:40:56.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:40:55 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:40:56.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:40:55 smithi047 ceph-mon[77147]: pgmap v746: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:40:56.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:40:55 smithi115 conmon[90393]: 2022-05-13T23:40:55.984+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:40:57.015 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:40:56 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:40:57.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:40:56 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:40:57.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:40:57 smithi115 conmon[90393]: 2022-05-13T23:40:57.014+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:40:58.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:40:57 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:40:58.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:40:57 smithi047 ceph-mon[77147]: fsmap cephfs:5/5 3 up:standby-replay 1 up:replay 4 up:active 2022-05-13T23:40:58.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:40:57 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:40:58.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:40:57 smithi047 ceph-mon[77147]: pgmap v747: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:40:58.051 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:40:57 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:40:58.051 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:40:57 smithi115 ceph-mon[78593]: fsmap cephfs:5/5 3 up:standby-replay 1 up:replay 4 up:active 2022-05-13T23:40:58.052 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:40:57 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:40:58.052 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:40:57 smithi115 ceph-mon[78593]: pgmap v747: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:40:58.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:40:58 smithi115 conmon[90393]: 2022-05-13T23:40:58.050+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:40:59.026 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:40:58 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:40:59.027 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:40:58 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 790 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:40:59.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:40:58 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:40:59.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:40:58 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 790 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:40:59.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:40:59 smithi115 conmon[90393]: 2022-05-13T23:40:59.025+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:41:00.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:40:59 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:41:00.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:40:59 smithi115 ceph-mon[78593]: pgmap v748: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:41:00.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:40:59 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:41:00.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:40:59 smithi047 ceph-mon[77147]: pgmap v748: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:41:00.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:41:00 smithi115 conmon[90393]: 2022-05-13T23:41:00.040+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:41:01.022 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:41:00 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:41:01.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:41:00 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:41:01.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:41:01 smithi115 conmon[90393]: 2022-05-13T23:41:01.021+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:41:02.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:41:01 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:41:02.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:41:01 smithi047 ceph-mon[77147]: fsmap cephfs:5/5 3 up:standby-replay 1 up:replay 4 up:active 2022-05-13T23:41:02.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:41:01 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:41:02.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:41:01 smithi047 ceph-mon[77147]: pgmap v749: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:41:02.068 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:41:01 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:41:02.068 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:41:01 smithi115 ceph-mon[78593]: fsmap cephfs:5/5 3 up:standby-replay 1 up:replay 4 up:active 2022-05-13T23:41:02.069 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:41:01 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:41:02.069 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:41:01 smithi115 ceph-mon[78593]: pgmap v749: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:41:02.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:41:02 smithi115 conmon[90393]: 2022-05-13T23:41:02.067+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:41:03.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:41:02 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:41:03.067 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:41:02 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:41:03.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:41:03 smithi115 conmon[90393]: 2022-05-13T23:41:03.065+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:41:04.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:41:03 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:41:04.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:41:03 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 795 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:41:04.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:41:03 smithi047 ceph-mon[77147]: pgmap v750: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:41:04.091 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:41:03 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:41:04.091 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:41:03 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 795 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:41:04.091 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:41:03 smithi115 ceph-mon[78593]: pgmap v750: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:41:04.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:41:04 smithi115 conmon[90393]: 2022-05-13T23:41:04.090+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:41:05.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:41:04 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:41:05.088 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:41:04 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:41:05.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:41:05 smithi115 conmon[90393]: 2022-05-13T23:41:05.088+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:41:06.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:41:05 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:41:06.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:41:05 smithi047 ceph-mon[77147]: pgmap v751: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:41:06.075 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:41:05 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:41:06.075 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:41:05 smithi115 ceph-mon[78593]: pgmap v751: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:41:06.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:41:06 smithi115 conmon[90393]: 2022-05-13T23:41:06.074+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:41:07.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:41:06 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:41:07.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:41:06 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:41:07.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:41:07 smithi115 conmon[90393]: 2022-05-13T23:41:07.114+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:41:08.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:41:07 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:41:08.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:41:07 smithi047 ceph-mon[77147]: pgmap v752: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:41:08.089 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:41:07 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:41:08.090 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:41:07 smithi115 ceph-mon[78593]: pgmap v752: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:41:08.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:41:08 smithi115 conmon[90393]: 2022-05-13T23:41:08.088+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:41:09.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:41:08 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:41:09.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:41:08 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 801 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:41:09.049 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:41:08 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:41:09.049 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:41:08 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 801 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:41:09.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:41:09 smithi115 conmon[90393]: 2022-05-13T23:41:09.047+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:41:10.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:41:09 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:41:10.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:41:09 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:41:10.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:41:09 smithi047 ceph-mon[77147]: fsmap cephfs:5/5 3 up:standby-replay 1 up:replay 4 up:active 2022-05-13T23:41:10.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:41:09 smithi047 ceph-mon[77147]: pgmap v753: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:41:10.079 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:41:09 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:41:10.080 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:41:09 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:41:10.081 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:41:09 smithi115 ceph-mon[78593]: fsmap cephfs:5/5 3 up:standby-replay 1 up:replay 4 up:active 2022-05-13T23:41:10.081 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:41:09 smithi115 ceph-mon[78593]: pgmap v753: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:41:10.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:41:10 smithi115 conmon[90393]: 2022-05-13T23:41:10.078+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:41:11.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:41:10 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:41:11.082 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:41:10 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:41:11.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:41:11 smithi115 conmon[90393]: 2022-05-13T23:41:11.081+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:41:12.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:41:11 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:41:12.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:41:11 smithi047 ceph-mon[77147]: pgmap v754: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:41:12.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:41:11 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:41:12.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:41:11 smithi115 ceph-mon[78593]: pgmap v754: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:41:12.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:41:12 smithi115 conmon[90393]: 2022-05-13T23:41:12.122+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:41:13.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:41:12 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:41:13.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:41:12 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:41:13.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:41:13 smithi115 conmon[90393]: 2022-05-13T23:41:13.132+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:41:14.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:41:13 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:41:14.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:41:13 smithi047 ceph-mon[77147]: fsmap cephfs:5/5 3 up:standby-replay 1 up:replay 4 up:active 2022-05-13T23:41:14.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:41:13 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:41:14.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:41:13 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-13T23:41:14.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:41:13 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-13T23:41:14.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:41:13 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 805 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:41:14.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:41:13 smithi047 ceph-mon[77147]: pgmap v755: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:41:14.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:41:13 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:41:14.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:41:13 smithi115 ceph-mon[78593]: fsmap cephfs:5/5 3 up:standby-replay 1 up:replay 4 up:active 2022-05-13T23:41:14.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:41:13 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:41:14.112 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:41:13 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-13T23:41:14.112 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:41:13 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-13T23:41:14.112 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:41:13 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 805 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:41:14.112 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:41:13 smithi115 ceph-mon[78593]: pgmap v755: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:41:14.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:41:14 smithi115 conmon[90393]: 2022-05-13T23:41:14.158+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:41:15.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:41:15 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:41:15.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:41:15 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:41:15.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:41:15 smithi115 conmon[90393]: 2022-05-13T23:41:15.108+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:41:15.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:41:15 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:41:15.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:41:15 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:41:16.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:41:16 smithi047 ceph-mon[77147]: pgmap v756: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:41:16.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:41:16 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:41:16.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:41:16 smithi115 conmon[90393]: 2022-05-13T23:41:16.132+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:41:16.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:41:16 smithi115 ceph-mon[78593]: pgmap v756: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:41:16.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:41:16 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:41:17.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:41:17 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:41:17.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:41:17 smithi115 conmon[90393]: 2022-05-13T23:41:17.133+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:41:17.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:41:17 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:41:18.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:41:18 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:41:18.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:41:18 smithi047 ceph-mon[77147]: fsmap cephfs:5/5 3 up:standby-replay 1 up:replay 4 up:active 2022-05-13T23:41:18.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:41:18 smithi047 ceph-mon[77147]: pgmap v757: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:41:18.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:41:18 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:41:18.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:41:18 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 810 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:41:18.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:41:18 smithi115 conmon[90393]: 2022-05-13T23:41:18.123+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:41:18.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:41:18 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:41:18.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:41:18 smithi115 ceph-mon[78593]: fsmap cephfs:5/5 3 up:standby-replay 1 up:replay 4 up:active 2022-05-13T23:41:18.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:41:18 smithi115 ceph-mon[78593]: pgmap v757: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:41:18.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:41:18 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:41:18.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:41:18 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 810 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:41:19.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:41:19 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:41:19.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:41:19 smithi115 conmon[90393]: 2022-05-13T23:41:19.108+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:41:19.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:41:19 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:41:20.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:41:20 smithi115 conmon[90393]: 2022-05-13T23:41:20.076+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:41:20.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:41:20 smithi115 ceph-mon[78593]: pgmap v758: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:41:20.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:41:20 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:41:20.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:41:20 smithi047 ceph-mon[77147]: pgmap v758: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:41:20.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:41:20 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:41:21.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:41:21 smithi115 conmon[90393]: 2022-05-13T23:41:21.052+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:41:21.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:41:21 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:41:21.795 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:41:21 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:41:22.299 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:41:22 smithi115 conmon[90393]: 2022-05-13T23:41:22.011+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:41:22.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:41:22 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:41:22.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:41:22 smithi115 ceph-mon[78593]: fsmap cephfs:5/5 3 up:standby-replay 1 up:replay 4 up:active 2022-05-13T23:41:22.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:41:22 smithi115 ceph-mon[78593]: pgmap v759: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:41:22.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:41:22 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:41:22.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:41:22 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:41:22.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:41:22 smithi047 ceph-mon[77147]: fsmap cephfs:5/5 3 up:standby-replay 1 up:replay 4 up:active 2022-05-13T23:41:22.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:41:22 smithi047 ceph-mon[77147]: pgmap v759: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:41:22.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:41:22 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:41:23.302 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:41:23 smithi115 conmon[90393]: 2022-05-13T23:41:23.060+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:41:23.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:41:23 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-13T23:41:23.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:41:23 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-13T23:41:23.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:41:23 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-13T23:41:23.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:41:23 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:41:23.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:41:23 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 816 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:41:23.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:41:23 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-13T23:41:23.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:41:23 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-13T23:41:23.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:41:23 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-13T23:41:23.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:41:23 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:41:23.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:41:23 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 816 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:41:24.302 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:41:24 smithi115 conmon[90393]: 2022-05-13T23:41:24.010+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:41:24.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:41:24 smithi115 ceph-mon[78593]: pgmap v760: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:41:24.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:41:24 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:41:24.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:41:24 smithi047 ceph-mon[77147]: pgmap v760: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:41:24.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:41:24 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:41:24.792 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:41:24 smithi047 conmon[78521]: 2022-05-13T23:41:24.399+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:41:24.792 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:41:24 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-13T23:41:24.793 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:41:24 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-13T23:41:24.793 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:41:24 smithi047 conmon[78521]: yield 2022-05-13T23:41:24.793 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:41:24 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-13T23:41:24.793 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:41:24 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-13T23:41:24.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:41:24 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-13T23:41:24.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:41:24 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-13T23:41:24.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:41:24 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-13T23:41:24.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:41:24 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-13T23:41:24.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:41:24 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-13T23:41:24.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:41:24 smithi047 conmon[78521]: raise exceptions[0] 2022-05-13T23:41:24.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:41:24 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-13T23:41:24.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:41:24 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-13T23:41:24.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:41:24 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-13T23:41:24.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:41:24 smithi047 conmon[78521]: return (yield from fut) 2022-05-13T23:41:24.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:41:24 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-13T23:41:24.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:41:24 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-13T23:41:24.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:41:24 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:41:25.311 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:41:25 smithi115 conmon[90393]: 2022-05-13T23:41:25.003+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:41:25.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:41:25 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:41:25.608 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-13T23:41:25.608 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-13T23:41:25.609 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-13T23:41:25.609 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-13T23:41:25.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-13T23:41:25.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-13T23:41:25.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-13T23:41:25.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-13T23:41:25.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-13T23:41:25.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-13T23:41:25.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-13T23:41:25.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-13T23:41:25.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-13T23:41:25.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-13T23:41:25.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-13T23:41:25.613 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-13T23:41:25.613 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-13T23:41:25.613 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:41:25.613 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:41:25 smithi115 ceph-mon[78593]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-13T23:41:25.614 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:41:25 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:41:25.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:41:25 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:41:25.792 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-13T23:41:25.792 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-13T23:41:25.792 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-13T23:41:25.793 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-13T23:41:25.793 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-13T23:41:25.793 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-13T23:41:25.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-13T23:41:25.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-13T23:41:25.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-13T23:41:25.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-13T23:41:25.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-13T23:41:25.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-13T23:41:25.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-13T23:41:25.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-13T23:41:25.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-13T23:41:25.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-13T23:41:25.797 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-13T23:41:25.797 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:41:25.797 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:41:25 smithi047 ceph-mon[77147]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-13T23:41:25.797 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:41:25 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:41:26.315 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:41:26 smithi115 conmon[90393]: 2022-05-13T23:41:26.003+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:41:26.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:41:26 smithi115 ceph-mon[78593]: pgmap v761: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:41:26.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:41:26 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:41:26.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:41:26 smithi047 ceph-mon[77147]: pgmap v761: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:41:26.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:41:26 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:41:27.317 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:41:27 smithi115 conmon[90393]: 2022-05-13T23:41:27.008+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:41:27.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:41:27 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:41:27.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:41:27 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:41:27.792 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:41:27 smithi047 conmon[78521]: 2022-05-13T23:41:27.471+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:41:27.792 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:41:27 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-13T23:41:27.792 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:41:27 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-13T23:41:27.792 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:41:27 smithi047 conmon[78521]: yield 2022-05-13T23:41:27.793 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:41:27 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-13T23:41:27.793 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:41:27 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-13T23:41:27.793 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:41:27 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-13T23:41:27.793 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:41:27 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-13T23:41:27.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:41:27 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-13T23:41:27.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:41:27 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-13T23:41:27.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:41:27 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-13T23:41:27.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:41:27 smithi047 conmon[78521]: raise exceptions[0] 2022-05-13T23:41:27.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:41:27 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-13T23:41:27.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:41:27 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-13T23:41:27.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:41:27 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-13T23:41:27.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:41:27 smithi047 conmon[78521]: return (yield from fut) 2022-05-13T23:41:27.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:41:27 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-13T23:41:27.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:41:27 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-13T23:41:27.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:41:27 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:41:27.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:41:27 smithi047 conmon[78521]: 2022-05-13T23:41:27.472+0000 7f5340779700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-13T23:41:27.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:41:27 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-13T23:41:27.797 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:41:27 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-13T23:41:27.797 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:41:27 smithi047 conmon[78521]: yield 2022-05-13T23:41:27.797 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:41:27 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-13T23:41:27.797 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:41:27 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-13T23:41:27.798 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:41:27 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-13T23:41:27.798 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:41:27 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-13T23:41:27.798 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:41:27 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-13T23:41:27.798 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:41:27 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-13T23:41:27.799 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:41:27 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-13T23:41:27.799 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:41:27 smithi047 conmon[78521]: raise exceptions[0] 2022-05-13T23:41:27.799 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:41:27 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-13T23:41:27.799 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:41:27 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-13T23:41:27.800 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:41:27 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-13T23:41:27.800 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:41:27 smithi047 conmon[78521]: return (yield from fut) 2022-05-13T23:41:27.800 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:41:27 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-13T23:41:27.800 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:41:27 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-13T23:41:27.801 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:41:27 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:41:27.801 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:41:27 smithi047 conmon[78521]: 2022-05-13T23:41:27.801 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:41:27 smithi047 conmon[78521]: During handling of the above exception, another exception occurred: 2022-05-13T23:41:27.801 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:41:27 smithi047 conmon[78521]: 2022-05-13T23:41:27.801 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:41:27 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-13T23:41:27.802 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:41:27 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-13T23:41:27.802 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:41:27 smithi047 conmon[78521]: return f(*arg) 2022-05-13T23:41:27.802 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:41:27 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-13T23:41:27.802 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:41:27 smithi047 conmon[78521]: self._write_client_files(client_files, host) 2022-05-13T23:41:27.803 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:41:27 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-13T23:41:27.803 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:41:27 smithi047 conmon[78521]: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-13T23:41:27.803 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:41:27 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-13T23:41:27.803 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:41:27 smithi047 conmon[78521]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-13T23:41:27.804 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:41:27 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-13T23:41:27.804 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:41:27 smithi047 conmon[78521]: return self.event_loop.get_result(coro) 2022-05-13T23:41:27.804 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:41:27 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-13T23:41:27.804 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:41:27 smithi047 conmon[78521]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-13T23:41:27.804 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:41:27 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-13T23:41:27.805 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:41:27 smithi047 conmon[78521]: return self.__get_result() 2022-05-13T23:41:27.805 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:41:27 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-13T23:41:27.805 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:41:27 smithi047 conmon[78521]: raise self._exception 2022-05-13T23:41:27.805 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:41:27 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-13T23:41:27.806 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:41:27 smithi047 conmon[78521]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-13T23:41:27.806 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:41:27 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-13T23:41:27.806 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:41:27 smithi047 conmon[78521]: conn = await self._remote_connection(host, addr) 2022-05-13T23:41:27.806 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:41:27 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-13T23:41:27.807 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:41:27 smithi047 conmon[78521]: raise 2022-05-13T23:41:27.807 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:41:27 smithi047 conmon[78521]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-13T23:41:27.807 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:41:27 smithi047 conmon[78521]: self.gen.throw(type, value, traceback) 2022-05-13T23:41:27.807 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:41:27 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-13T23:41:27.808 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:41:27 smithi047 conmon[78521]: raise OrchestratorError(msg) 2022-05-13T23:41:27.808 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:41:27 smithi047 conmon[78521]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:41:28.320 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:41:28 smithi115 conmon[90393]: 2022-05-13T23:41:28.020+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:41:28.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:41:28 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:41:28.608 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-13T23:41:28.609 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-13T23:41:28.609 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-13T23:41:28.609 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-13T23:41:28.609 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-13T23:41:28.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-13T23:41:28.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-13T23:41:28.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-13T23:41:28.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-13T23:41:28.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-13T23:41:28.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-13T23:41:28.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-13T23:41:28.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-13T23:41:28.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-13T23:41:28.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-13T23:41:28.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-13T23:41:28.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-13T23:41:28.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:41:28.612 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:41:28 smithi115 ceph-mon[78593]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-13T23:41:28.613 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-13T23:41:28.613 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-13T23:41:28.613 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-13T23:41:28.613 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-13T23:41:28.613 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-13T23:41:28.614 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-13T23:41:28.614 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-13T23:41:28.614 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-13T23:41:28.614 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-13T23:41:28.615 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-13T23:41:28.615 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-13T23:41:28.615 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-13T23:41:28.615 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-13T23:41:28.615 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-13T23:41:28.616 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-13T23:41:28.616 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-13T23:41:28.616 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-13T23:41:28.616 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:41:28.617 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-13T23:41:28.617 INFO:journalctl@ceph.mon.c.smithi115.stdout: During handling of the above exception, another exception occurred: 2022-05-13T23:41:28.617 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-13T23:41:28.617 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-13T23:41:28.618 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-13T23:41:28.618 INFO:journalctl@ceph.mon.c.smithi115.stdout: return f(*arg) 2022-05-13T23:41:28.619 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-13T23:41:28.619 INFO:journalctl@ceph.mon.c.smithi115.stdout: self._write_client_files(client_files, host) 2022-05-13T23:41:28.619 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-13T23:41:28.619 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-13T23:41:28.619 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-13T23:41:28.620 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-13T23:41:28.620 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-13T23:41:28.620 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.event_loop.get_result(coro) 2022-05-13T23:41:28.620 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-13T23:41:28.621 INFO:journalctl@ceph.mon.c.smithi115.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-13T23:41:28.621 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-13T23:41:28.621 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.__get_result() 2022-05-13T23:41:28.621 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-13T23:41:28.621 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise self._exception 2022-05-13T23:41:28.622 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-13T23:41:28.622 INFO:journalctl@ceph.mon.c.smithi115.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-13T23:41:28.622 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-13T23:41:28.622 INFO:journalctl@ceph.mon.c.smithi115.stdout: conn = await self._remote_connection(host, addr) 2022-05-13T23:41:28.622 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-13T23:41:28.623 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise 2022-05-13T23:41:28.623 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-13T23:41:28.623 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.gen.throw(type, value, traceback) 2022-05-13T23:41:28.623 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-13T23:41:28.623 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OrchestratorError(msg) 2022-05-13T23:41:28.624 INFO:journalctl@ceph.mon.c.smithi115.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:41:28.624 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:41:28 smithi115 ceph-mon[78593]: pgmap v762: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:41:28.624 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:41:28 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:41:28.624 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:41:28 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 820 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:41:28.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:41:28 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:41:28.792 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-13T23:41:28.792 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-13T23:41:28.792 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-13T23:41:28.792 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-13T23:41:28.793 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-13T23:41:28.793 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-13T23:41:28.793 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-13T23:41:28.793 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-13T23:41:28.793 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-13T23:41:28.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-13T23:41:28.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-13T23:41:28.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-13T23:41:28.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-13T23:41:28.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-13T23:41:28.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-13T23:41:28.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-13T23:41:28.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-13T23:41:28.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:41:28.796 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:41:28 smithi047 ceph-mon[77147]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-13T23:41:28.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-13T23:41:28.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-13T23:41:28.797 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-13T23:41:28.797 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-13T23:41:28.797 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-13T23:41:28.797 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-13T23:41:28.798 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-13T23:41:28.798 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-13T23:41:28.798 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-13T23:41:28.798 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-13T23:41:28.798 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-13T23:41:28.799 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-13T23:41:28.799 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-13T23:41:28.799 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-13T23:41:28.799 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-13T23:41:28.800 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-13T23:41:28.800 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-13T23:41:28.800 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:41:28.800 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-13T23:41:28.800 INFO:journalctl@ceph.mon.b.smithi047.stdout: During handling of the above exception, another exception occurred: 2022-05-13T23:41:28.801 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-13T23:41:28.801 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-13T23:41:28.801 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-13T23:41:28.801 INFO:journalctl@ceph.mon.b.smithi047.stdout: return f(*arg) 2022-05-13T23:41:28.802 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-13T23:41:28.802 INFO:journalctl@ceph.mon.b.smithi047.stdout: self._write_client_files(client_files, host) 2022-05-13T23:41:28.802 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-13T23:41:28.802 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-13T23:41:28.802 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-13T23:41:28.803 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-13T23:41:28.803 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-13T23:41:28.803 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.event_loop.get_result(coro) 2022-05-13T23:41:28.803 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-13T23:41:28.804 INFO:journalctl@ceph.mon.b.smithi047.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-13T23:41:28.804 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-13T23:41:28.804 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.__get_result() 2022-05-13T23:41:28.804 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-13T23:41:28.804 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise self._exception 2022-05-13T23:41:28.805 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-13T23:41:28.805 INFO:journalctl@ceph.mon.b.smithi047.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-13T23:41:28.805 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-13T23:41:28.805 INFO:journalctl@ceph.mon.b.smithi047.stdout: conn = await self._remote_connection(host, addr) 2022-05-13T23:41:28.805 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-13T23:41:28.806 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise 2022-05-13T23:41:28.806 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-13T23:41:28.806 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.gen.throw(type, value, traceback) 2022-05-13T23:41:28.806 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-13T23:41:28.807 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OrchestratorError(msg) 2022-05-13T23:41:28.807 INFO:journalctl@ceph.mon.b.smithi047.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:41:28.807 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:41:28 smithi047 ceph-mon[77147]: pgmap v762: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:41:28.807 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:41:28 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:41:28.807 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:41:28 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 820 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:41:29.324 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:41:29 smithi115 conmon[90393]: 2022-05-13T23:41:29.065+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:41:29.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:41:29 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:41:29.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:41:29 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:41:30.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:41:30 smithi115 conmon[90393]: 2022-05-13T23:41:30.084+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:41:30.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:41:30 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:41:30.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:41:30 smithi115 ceph-mon[78593]: fsmap cephfs:5/5 3 up:standby-replay 1 up:replay 4 up:active 2022-05-13T23:41:30.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:41:30 smithi115 ceph-mon[78593]: pgmap v763: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:41:30.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:41:30 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:41:30.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:41:30 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:41:30.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:41:30 smithi047 ceph-mon[77147]: fsmap cephfs:5/5 3 up:standby-replay 1 up:replay 4 up:active 2022-05-13T23:41:30.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:41:30 smithi047 ceph-mon[77147]: pgmap v763: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:41:30.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:41:30 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:41:31.332 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:41:31 smithi115 conmon[90393]: 2022-05-13T23:41:31.066+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:41:31.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:41:31 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:41:31.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:41:31 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:41:32.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:41:32 smithi115 ceph-mon[78593]: pgmap v764: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:41:32.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:41:32 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:41:32.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:41:32 smithi115 conmon[90393]: 2022-05-13T23:41:32.096+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:41:32.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:41:32 smithi047 ceph-mon[77147]: pgmap v764: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:41:32.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:41:32 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:41:33.347 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:41:33 smithi115 conmon[90393]: 2022-05-13T23:41:33.077+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:41:33.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:41:33 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:41:33.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:41:33 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 825 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:41:33.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:41:33 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:41:33.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:41:33 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 825 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:41:34.351 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:41:34 smithi115 conmon[90393]: 2022-05-13T23:41:34.035+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:41:34.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:41:34 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:41:34.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:41:34 smithi115 ceph-mon[78593]: fsmap cephfs:5/5 3 up:standby-replay 1 up:replay 4 up:active 2022-05-13T23:41:34.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:41:34 smithi115 ceph-mon[78593]: pgmap v765: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:41:34.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:41:34 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:41:34.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:41:34 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:41:34.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:41:34 smithi047 ceph-mon[77147]: fsmap cephfs:5/5 3 up:standby-replay 1 up:replay 4 up:active 2022-05-13T23:41:34.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:41:34 smithi047 ceph-mon[77147]: pgmap v765: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:41:34.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:41:34 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:41:35.350 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:41:35 smithi115 conmon[90393]: 2022-05-13T23:41:35.013+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:41:35.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:41:35 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:41:35.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:41:35 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:41:36.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:41:36 smithi115 conmon[90393]: 2022-05-13T23:41:36.021+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:41:36.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:41:36 smithi047 ceph-mon[77147]: pgmap v766: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:41:36.795 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:41:36 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:41:36.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:41:36 smithi115 ceph-mon[78593]: pgmap v766: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:41:36.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:41:36 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:41:37.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:41:37 smithi115 conmon[90393]: 2022-05-13T23:41:37.070+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:41:37.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:41:37 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:41:37.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:41:37 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:41:38.366 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:41:38 smithi115 conmon[90393]: 2022-05-13T23:41:38.113+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:41:38.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:41:38 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:41:38.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:41:38 smithi047 ceph-mon[77147]: fsmap cephfs:5/5 3 up:standby-replay 1 up:replay 4 up:active 2022-05-13T23:41:38.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:41:38 smithi047 ceph-mon[77147]: pgmap v767: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:41:38.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:41:38 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:41:38.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:41:38 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 830 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:41:38.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:41:38 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:41:38.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:41:38 smithi115 ceph-mon[78593]: fsmap cephfs:5/5 3 up:standby-replay 1 up:replay 4 up:active 2022-05-13T23:41:38.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:41:38 smithi115 ceph-mon[78593]: pgmap v767: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:41:38.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:41:38 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:41:38.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:41:38 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 830 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:41:39.589 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:41:39 smithi115 conmon[90393]: 2022-05-13T23:41:39.157+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:41:39.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:41:39 smithi115 ceph-mon[78593]: pgmap v768: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:41:40.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:41:39 smithi047 ceph-mon[77147]: pgmap v768: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:41:40.594 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:41:40 smithi115 conmon[90393]: 2022-05-13T23:41:40.125+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:41:40.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:41:40 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:41:40.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:41:40 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:41:41.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:41:40 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:41:41.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:41:40 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:41:41.596 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:41:41 smithi115 conmon[90393]: 2022-05-13T23:41:41.137+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:41:41.869 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:41:41 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:41:41.870 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:41:41 smithi115 ceph-mon[78593]: fsmap cephfs:5/5 3 up:standby-replay 1 up:replay 4 up:active 2022-05-13T23:41:41.870 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:41:41 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:41:41.870 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:41:41 smithi115 ceph-mon[78593]: pgmap v769: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:41:42.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:41:41 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:41:42.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:41:41 smithi047 ceph-mon[77147]: fsmap cephfs:5/5 3 up:standby-replay 1 up:replay 4 up:active 2022-05-13T23:41:42.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:41:41 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:41:42.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:41:41 smithi047 ceph-mon[77147]: pgmap v769: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:41:42.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:41:42 smithi115 conmon[90393]: 2022-05-13T23:41:42.184+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:41:43.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:41:43 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:41:43.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:41:43 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:41:43.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:41:43 smithi115 conmon[90393]: 2022-05-13T23:41:43.212+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:41:44.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:41:44 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:41:44.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:41:44 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 835 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:41:44.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:41:44 smithi047 ceph-mon[77147]: pgmap v770: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:41:44.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:41:44 smithi115 conmon[90393]: 2022-05-13T23:41:44.254+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:41:44.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:41:44 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:41:44.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:41:44 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 835 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:41:44.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:41:44 smithi115 ceph-mon[78593]: pgmap v770: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:41:45.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:41:45 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:41:45.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:41:45 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:41:45.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:41:45 smithi115 conmon[90393]: 2022-05-13T23:41:45.247+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:41:46.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:41:46 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:41:46.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:41:46 smithi047 ceph-mon[77147]: pgmap v771: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:41:46.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:41:46 smithi115 conmon[90393]: 2022-05-13T23:41:46.282+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:41:46.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:41:46 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:41:46.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:41:46 smithi115 ceph-mon[78593]: pgmap v771: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:41:47.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:41:47 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:41:47.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:41:47 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:41:47.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:41:47 smithi115 conmon[90393]: 2022-05-13T23:41:47.269+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:41:48.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:41:48 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:41:48.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:41:48 smithi047 ceph-mon[77147]: pgmap v772: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:41:48.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:41:48 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 840 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:41:48.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:41:48 smithi115 conmon[90393]: 2022-05-13T23:41:48.249+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:41:48.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:41:48 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:41:48.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:41:48 smithi115 ceph-mon[78593]: pgmap v772: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:41:48.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:41:48 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 840 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:41:49.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:41:49 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:41:49.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:41:49 smithi115 conmon[90393]: 2022-05-13T23:41:49.282+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:41:49.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:41:49 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:41:50.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:41:50 smithi115 conmon[90393]: 2022-05-13T23:41:50.268+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:41:50.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:41:50 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:41:50.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:41:50 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:41:50.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:41:50 smithi115 ceph-mon[78593]: fsmap cephfs:5/5 3 up:standby-replay 1 up:replay 4 up:active 2022-05-13T23:41:50.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:41:50 smithi115 ceph-mon[78593]: pgmap v773: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:41:50.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:41:50 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:41:50.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:41:50 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:41:50.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:41:50 smithi047 ceph-mon[77147]: fsmap cephfs:5/5 3 up:standby-replay 1 up:replay 4 up:active 2022-05-13T23:41:50.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:41:50 smithi047 ceph-mon[77147]: pgmap v773: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:41:51.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:41:51 smithi115 conmon[90393]: 2022-05-13T23:41:51.316+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:41:51.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:41:51 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:41:51.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:41:51 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:41:52.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:41:52 smithi115 conmon[90393]: 2022-05-13T23:41:52.349+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:41:52.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:41:52 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:41:52.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:41:52 smithi115 ceph-mon[78593]: pgmap v774: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:41:52.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:41:52 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:41:52.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:41:52 smithi047 ceph-mon[77147]: pgmap v774: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:41:53.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:41:53 smithi115 conmon[90393]: 2022-05-13T23:41:53.331+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:41:53.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:41:53 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:41:53.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:41:53 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 845 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:41:53.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:41:53 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:41:53.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:41:53 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 845 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:41:54.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:41:54 smithi115 conmon[90393]: 2022-05-13T23:41:54.345+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:41:54.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:41:54 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:41:54.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:41:54 smithi115 ceph-mon[78593]: fsmap cephfs:5/5 3 up:standby-replay 1 up:replay 4 up:active 2022-05-13T23:41:54.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:41:54 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:41:54.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:41:54 smithi115 ceph-mon[78593]: pgmap v775: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:41:54.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:41:54 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:41:54.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:41:54 smithi047 ceph-mon[77147]: fsmap cephfs:5/5 3 up:standby-replay 1 up:replay 4 up:active 2022-05-13T23:41:54.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:41:54 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:41:54.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:41:54 smithi047 ceph-mon[77147]: pgmap v775: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:41:55.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:41:55 smithi115 conmon[90393]: 2022-05-13T23:41:55.371+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:41:55.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:41:55 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:41:55.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:41:55 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:41:56.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:41:56 smithi115 conmon[90393]: 2022-05-13T23:41:56.331+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:41:56.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:41:56 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:41:56.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:41:56 smithi115 ceph-mon[78593]: pgmap v776: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:41:56.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:41:56 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:41:56.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:41:56 smithi047 ceph-mon[77147]: pgmap v776: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:41:57.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:41:57 smithi115 conmon[90393]: 2022-05-13T23:41:57.290+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:41:57.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:41:57 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:41:57.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:41:57 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:41:58.610 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:41:58 smithi115 conmon[90393]: 2022-05-13T23:41:58.269+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:41:58.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:41:58 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:41:58.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:41:58 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:41:58.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:41:58 smithi115 ceph-mon[78593]: fsmap cephfs:5/5 3 up:standby-replay 1 up:replay 4 up:active 2022-05-13T23:41:58.612 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:41:58 smithi115 ceph-mon[78593]: pgmap v777: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:41:58.612 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:41:58 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 850 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:41:58.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:41:58 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:41:58.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:41:58 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:41:58.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:41:58 smithi047 ceph-mon[77147]: fsmap cephfs:5/5 3 up:standby-replay 1 up:replay 4 up:active 2022-05-13T23:41:58.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:41:58 smithi047 ceph-mon[77147]: pgmap v777: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:41:58.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:41:58 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 850 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:41:59.611 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:41:59 smithi115 conmon[90393]: 2022-05-13T23:41:59.239+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:41:59.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:41:59 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:41:59.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:41:59 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:42:00.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:42:00 smithi115 conmon[90393]: 2022-05-13T23:42:00.259+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:42:00.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:42:00 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:42:00.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:42:00 smithi115 ceph-mon[78593]: pgmap v778: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:42:00.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:42:00 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:42:00.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:42:00 smithi047 ceph-mon[77147]: pgmap v778: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:42:01.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:42:01 smithi115 conmon[90393]: 2022-05-13T23:42:01.283+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:42:01.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:42:01 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:42:01.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:42:01 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:42:02.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:42:02 smithi115 conmon[90393]: 2022-05-13T23:42:02.271+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:42:02.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:42:02 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:42:02.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:42:02 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:42:02.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:42:02 smithi115 ceph-mon[78593]: fsmap cephfs:5/5 3 up:standby-replay 1 up:replay 4 up:active 2022-05-13T23:42:02.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:42:02 smithi115 ceph-mon[78593]: pgmap v779: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:42:02.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:42:02 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:42:02.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:42:02 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:42:02.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:42:02 smithi047 ceph-mon[77147]: fsmap cephfs:5/5 3 up:standby-replay 1 up:replay 4 up:active 2022-05-13T23:42:02.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:42:02 smithi047 ceph-mon[77147]: pgmap v779: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:42:03.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:42:03 smithi115 conmon[90393]: 2022-05-13T23:42:03.307+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:42:03.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:42:03 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:42:03.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:42:03 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 855 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:42:03.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:42:03 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:42:03.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:42:03 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 855 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:42:04.610 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:42:04 smithi115 conmon[90393]: 2022-05-13T23:42:04.342+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:42:04.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:42:04 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:42:04.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:42:04 smithi115 ceph-mon[78593]: pgmap v780: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:42:04.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:42:04 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:42:04.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:42:04 smithi047 ceph-mon[77147]: pgmap v780: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:42:05.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:42:05 smithi115 conmon[90393]: 2022-05-13T23:42:05.323+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:42:05.612 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:42:05 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:42:05.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:42:05 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:42:06.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:42:06 smithi115 conmon[90393]: 2022-05-13T23:42:06.321+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:42:06.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:42:06 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:42:06.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:42:06 smithi115 ceph-mon[78593]: pgmap v781: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:42:06.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:42:06 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:42:06.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:42:06 smithi047 ceph-mon[77147]: pgmap v781: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:42:07.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:42:07 smithi115 conmon[90393]: 2022-05-13T23:42:07.350+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:42:07.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:42:07 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:42:07.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:42:07 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:42:08.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:42:08 smithi115 conmon[90393]: 2022-05-13T23:42:08.345+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:42:08.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:42:08 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:42:08.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:42:08 smithi115 ceph-mon[78593]: pgmap v782: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:42:08.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:42:08 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 860 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:42:08.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:42:08 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:42:08.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:42:08 smithi047 ceph-mon[77147]: pgmap v782: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:42:08.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:42:08 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 860 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:42:09.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:42:09 smithi115 conmon[90393]: 2022-05-13T23:42:09.306+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:42:09.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:42:09 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:42:09.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:42:09 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:42:10.610 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:42:10 smithi115 conmon[90393]: 2022-05-13T23:42:10.298+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:42:10.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:42:10 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:42:10.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:42:10 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:42:10.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:42:10 smithi115 ceph-mon[78593]: fsmap cephfs:5/5 3 up:standby-replay 1 up:replay 4 up:active 2022-05-13T23:42:10.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:42:10 smithi115 ceph-mon[78593]: pgmap v783: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:42:10.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:42:10 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:42:10.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:42:10 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:42:10.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:42:10 smithi047 ceph-mon[77147]: fsmap cephfs:5/5 3 up:standby-replay 1 up:replay 4 up:active 2022-05-13T23:42:10.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:42:10 smithi047 ceph-mon[77147]: pgmap v783: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:42:11.610 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:42:11 smithi115 conmon[90393]: 2022-05-13T23:42:11.317+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:42:11.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:42:11 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:42:11.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:42:11 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:42:12.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:42:12 smithi115 conmon[90393]: 2022-05-13T23:42:12.355+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:42:12.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:42:12 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:42:12.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:42:12 smithi115 ceph-mon[78593]: pgmap v784: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:42:12.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:42:12 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:42:12.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:42:12 smithi047 ceph-mon[77147]: pgmap v784: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:42:13.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:42:13 smithi115 conmon[90393]: 2022-05-13T23:42:13.320+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:42:13.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:42:13 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:42:13.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:42:13 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:42:13.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:42:13 smithi115 ceph-mon[78593]: fsmap cephfs:5/5 3 up:standby-replay 1 up:replay 4 up:active 2022-05-13T23:42:13.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:42:13 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-13T23:42:13.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:42:13 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-13T23:42:13.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:42:13 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 865 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:42:13.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:42:13 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:42:13.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:42:13 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:42:13.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:42:13 smithi047 ceph-mon[77147]: fsmap cephfs:5/5 3 up:standby-replay 1 up:replay 4 up:active 2022-05-13T23:42:13.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:42:13 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-13T23:42:13.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:42:13 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-13T23:42:13.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:42:13 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 865 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:42:14.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:42:14 smithi115 conmon[90393]: 2022-05-13T23:42:14.305+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:42:14.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:42:14 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:42:14.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:42:14 smithi115 ceph-mon[78593]: pgmap v785: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:42:14.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:42:14 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:42:14.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:42:14 smithi047 ceph-mon[77147]: pgmap v785: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:42:15.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:42:15 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:42:15.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:42:15 smithi115 conmon[90393]: 2022-05-13T23:42:15.303+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:42:15.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:42:15 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:42:16.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:42:16 smithi115 conmon[90393]: 2022-05-13T23:42:16.278+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:42:16.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:42:16 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:42:16.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:42:16 smithi115 ceph-mon[78593]: pgmap v786: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:42:16.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:42:16 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:42:16.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:42:16 smithi047 ceph-mon[77147]: pgmap v786: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:42:17.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:42:17 smithi115 conmon[90393]: 2022-05-13T23:42:17.301+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:42:17.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:42:17 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:42:17.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:42:17 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:42:17.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:42:17 smithi115 ceph-mon[78593]: fsmap cephfs:5/5 3 up:standby-replay 1 up:replay 4 up:active 2022-05-13T23:42:17.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:42:17 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:42:17.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:42:17 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:42:17.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:42:17 smithi047 ceph-mon[77147]: fsmap cephfs:5/5 3 up:standby-replay 1 up:replay 4 up:active 2022-05-13T23:42:18.610 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:42:18 smithi115 conmon[90393]: 2022-05-13T23:42:18.313+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:42:18.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:42:18 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:42:18.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:42:18 smithi115 ceph-mon[78593]: pgmap v787: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:42:18.612 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:42:18 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 870 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:42:18.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:42:18 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:42:18.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:42:18 smithi047 ceph-mon[77147]: pgmap v787: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:42:18.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:42:18 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 870 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:42:19.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:42:19 smithi115 conmon[90393]: 2022-05-13T23:42:19.323+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:42:19.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:42:19 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:42:19.799 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:42:19 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:42:20.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:42:20 smithi115 conmon[90393]: 2022-05-13T23:42:20.341+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:42:20.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:42:20 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:42:20.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:42:20 smithi115 ceph-mon[78593]: pgmap v788: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:42:20.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:42:20 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:42:20.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:42:20 smithi047 ceph-mon[77147]: pgmap v788: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:42:21.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:42:21 smithi115 conmon[90393]: 2022-05-13T23:42:21.306+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:42:21.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:42:21 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:42:21.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:42:21 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:42:21.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:42:21 smithi115 ceph-mon[78593]: fsmap cephfs:5/5 3 up:standby-replay 1 up:replay 4 up:active 2022-05-13T23:42:21.803 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:42:21 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:42:21.803 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:42:21 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:42:21.804 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:42:21 smithi047 ceph-mon[77147]: fsmap cephfs:5/5 3 up:standby-replay 1 up:replay 4 up:active 2022-05-13T23:42:22.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:42:22 smithi115 conmon[90393]: 2022-05-13T23:42:22.339+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:42:22.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:42:22 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:42:22.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:42:22 smithi115 ceph-mon[78593]: pgmap v789: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:42:22.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:42:22 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:42:22.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:42:22 smithi047 ceph-mon[77147]: pgmap v789: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:42:23.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:42:23 smithi115 conmon[90393]: 2022-05-13T23:42:23.294+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:42:23.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:42:23 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:42:23.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:42:23 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 875 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:42:23.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:42:23 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:42:23.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:42:23 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 875 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:42:24.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:42:24 smithi115 conmon[90393]: 2022-05-13T23:42:24.333+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:42:24.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:42:24 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:42:24.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:42:24 smithi115 ceph-mon[78593]: pgmap v790: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:42:24.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:42:24 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:42:24.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:42:24 smithi047 ceph-mon[77147]: pgmap v790: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:42:25.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:42:25 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:42:25.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:42:25 smithi115 conmon[90393]: 2022-05-13T23:42:25.379+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:42:25.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:42:25 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:42:26.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:42:26 smithi115 conmon[90393]: 2022-05-13T23:42:26.344+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:42:26.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:42:26 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:42:26.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:42:26 smithi115 ceph-mon[78593]: pgmap v791: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:42:26.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:42:26 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:42:26.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:42:26 smithi047 ceph-mon[77147]: pgmap v791: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:42:27.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:42:27 smithi115 conmon[90393]: 2022-05-13T23:42:27.347+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:42:27.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:42:27 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:42:27.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:42:27 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:42:28.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:42:28 smithi115 conmon[90393]: 2022-05-13T23:42:28.304+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:42:28.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:42:28 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:42:28.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:42:28 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-13T23:42:28.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:42:28 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-13T23:42:28.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:42:28 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-13T23:42:28.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:42:28 smithi115 ceph-mon[78593]: pgmap v792: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:42:28.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:42:28 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 880 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:42:28.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:42:28 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:42:28.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:42:28 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-13T23:42:28.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:42:28 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-13T23:42:28.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:42:28 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-13T23:42:28.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:42:28 smithi047 ceph-mon[77147]: pgmap v792: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:42:28.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:42:28 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 880 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:42:29.291 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:42:28 smithi047 conmon[78521]: 2022-05-13T23:42:28.912+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:42:29.292 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:42:28 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-13T23:42:29.292 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:42:28 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-13T23:42:29.292 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:42:28 smithi047 conmon[78521]: yield 2022-05-13T23:42:29.292 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:42:28 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-13T23:42:29.293 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:42:28 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-13T23:42:29.293 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:42:28 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-13T23:42:29.293 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:42:28 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-13T23:42:29.293 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:42:28 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-13T23:42:29.293 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:42:28 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-13T23:42:29.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:42:28 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-13T23:42:29.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:42:28 smithi047 conmon[78521]: raise exceptions[0] 2022-05-13T23:42:29.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:42:28 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-13T23:42:29.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:42:28 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-13T23:42:29.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:42:28 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-13T23:42:29.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:42:28 smithi047 conmon[78521]: return (yield from fut) 2022-05-13T23:42:29.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:42:28 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-13T23:42:29.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:42:28 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-13T23:42:29.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:42:28 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:42:29.611 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:42:29 smithi115 conmon[90393]: 2022-05-13T23:42:29.284+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:42:29.612 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:42:29 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:42:29.612 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:42:29 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:42:29.616 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:42:29 smithi115 ceph-mon[78593]: fsmap cephfs:5/5 3 up:standby-replay 1 up:replay 4 up:active 2022-05-13T23:42:29.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:42:29 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:42:29.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:42:29 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:42:29.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:42:29 smithi047 ceph-mon[77147]: fsmap cephfs:5/5 3 up:standby-replay 1 up:replay 4 up:active 2022-05-13T23:42:30.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:42:30 smithi115 conmon[90393]: 2022-05-13T23:42:30.243+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:42:30.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:42:30 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:42:30.609 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-13T23:42:30.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-13T23:42:30.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-13T23:42:30.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-13T23:42:30.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-13T23:42:30.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-13T23:42:30.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-13T23:42:30.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-13T23:42:30.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-13T23:42:30.613 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-13T23:42:30.613 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-13T23:42:30.613 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-13T23:42:30.613 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-13T23:42:30.613 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-13T23:42:30.614 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-13T23:42:30.614 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-13T23:42:30.615 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-13T23:42:30.615 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:42:30.615 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:42:30 smithi115 ceph-mon[78593]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-13T23:42:30.616 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:42:30 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:42:30.616 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:42:30 smithi115 ceph-mon[78593]: pgmap v793: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:42:30.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:42:30 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:42:30.791 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-13T23:42:30.792 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-13T23:42:30.792 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-13T23:42:30.792 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-13T23:42:30.792 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-13T23:42:30.793 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-13T23:42:30.793 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-13T23:42:30.793 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-13T23:42:30.793 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-13T23:42:30.793 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-13T23:42:30.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-13T23:42:30.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-13T23:42:30.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-13T23:42:30.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-13T23:42:30.797 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-13T23:42:30.797 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-13T23:42:30.798 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-13T23:42:30.799 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:42:30.799 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:42:30 smithi047 ceph-mon[77147]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-13T23:42:30.799 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:42:30 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:42:30.800 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:42:30 smithi047 ceph-mon[77147]: pgmap v793: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:42:31.611 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:42:31 smithi115 conmon[90393]: 2022-05-13T23:42:31.280+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:42:31.612 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:42:31 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:42:31.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:42:31 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:42:32.292 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:42:31 smithi047 conmon[78521]: 2022-05-13T23:42:31.984+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:42:32.293 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:42:31 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-13T23:42:32.293 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:42:31 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-13T23:42:32.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:42:31 smithi047 conmon[78521]: yield 2022-05-13T23:42:32.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:42:31 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-13T23:42:32.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:42:31 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-13T23:42:32.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:42:31 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-13T23:42:32.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:42:31 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-13T23:42:32.297 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:42:31 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-13T23:42:32.297 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:42:31 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-13T23:42:32.298 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:42:31 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-13T23:42:32.298 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:42:31 smithi047 conmon[78521]: raise exceptions[0] 2022-05-13T23:42:32.298 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:42:31 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-13T23:42:32.299 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:42:31 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-13T23:42:32.299 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:42:31 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-13T23:42:32.299 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:42:31 smithi047 conmon[78521]: return (yield from fut) 2022-05-13T23:42:32.299 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:42:31 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-13T23:42:32.300 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:42:31 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-13T23:42:32.300 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:42:31 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:42:32.300 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:42:31 smithi047 conmon[78521]: 2022-05-13T23:42:31.985+0000 7f533ef76700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-13T23:42:32.300 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:42:31 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-13T23:42:32.300 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:42:31 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-13T23:42:32.301 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:42:31 smithi047 conmon[78521]: yield 2022-05-13T23:42:32.301 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:42:31 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-13T23:42:32.302 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:42:31 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-13T23:42:32.302 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:42:31 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-13T23:42:32.302 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:42:31 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-13T23:42:32.303 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:42:31 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-13T23:42:32.303 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:42:31 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-13T23:42:32.304 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:42:31 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-13T23:42:32.304 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:42:31 smithi047 conmon[78521]: raise exceptions[0] 2022-05-13T23:42:32.305 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:42:31 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-13T23:42:32.305 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:42:31 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-13T23:42:32.306 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:42:31 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-13T23:42:32.306 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:42:31 smithi047 conmon[78521]: return (yield from fut) 2022-05-13T23:42:32.307 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:42:31 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-13T23:42:32.307 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:42:31 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-13T23:42:32.307 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:42:31 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:42:32.307 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:42:31 smithi047 conmon[78521]: 2022-05-13T23:42:32.308 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:42:31 smithi047 conmon[78521]: During handling of the above exception, another exception occurred: 2022-05-13T23:42:32.308 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:42:31 smithi047 conmon[78521]: 2022-05-13T23:42:32.308 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:42:31 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-13T23:42:32.308 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:42:31 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-13T23:42:32.308 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:42:31 smithi047 conmon[78521]: return f(*arg) 2022-05-13T23:42:32.309 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:42:31 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-13T23:42:32.309 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:42:31 smithi047 conmon[78521]: self._write_client_files(client_files, host) 2022-05-13T23:42:32.309 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:42:31 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-13T23:42:32.310 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:42:31 smithi047 conmon[78521]: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-13T23:42:32.311 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:42:31 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-13T23:42:32.311 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:42:31 smithi047 conmon[78521]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-13T23:42:32.311 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:42:31 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-13T23:42:32.311 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:42:31 smithi047 conmon[78521]: return self.event_loop.get_result(coro) 2022-05-13T23:42:32.311 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:42:31 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-13T23:42:32.312 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:42:31 smithi047 conmon[78521]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-13T23:42:32.312 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:42:31 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-13T23:42:32.312 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:42:31 smithi047 conmon[78521]: return self.__get_result() 2022-05-13T23:42:32.312 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:42:31 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-13T23:42:32.313 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:42:31 smithi047 conmon[78521]: raise self._exception 2022-05-13T23:42:32.313 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:42:31 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-13T23:42:32.313 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:42:31 smithi047 conmon[78521]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-13T23:42:32.313 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:42:31 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-13T23:42:32.314 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:42:31 smithi047 conmon[78521]: conn = await self._remote_connection(host, addr) 2022-05-13T23:42:32.314 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:42:31 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-13T23:42:32.314 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:42:31 smithi047 conmon[78521]: raise 2022-05-13T23:42:32.314 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:42:31 smithi047 conmon[78521]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-13T23:42:32.314 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:42:31 smithi047 conmon[78521]: self.gen.throw(type, value, traceback) 2022-05-13T23:42:32.315 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:42:31 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-13T23:42:32.315 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:42:31 smithi047 conmon[78521]: raise OrchestratorError(msg) 2022-05-13T23:42:32.315 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:42:31 smithi047 conmon[78521]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:42:32.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:42:32 smithi115 conmon[90393]: 2022-05-13T23:42:32.240+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:42:32.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:42:32 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:42:32.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:42:32 smithi115 ceph-mon[78593]: pgmap v794: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:42:32.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:42:32 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:42:32.796 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:42:32 smithi047 ceph-mon[77147]: pgmap v794: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:42:33.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:42:33 smithi115 conmon[90393]: 2022-05-13T23:42:33.254+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:42:33.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:42:33 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:42:33.609 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-13T23:42:33.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-13T23:42:33.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-13T23:42:33.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-13T23:42:33.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-13T23:42:33.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-13T23:42:33.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-13T23:42:33.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-13T23:42:33.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-13T23:42:33.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-13T23:42:33.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-13T23:42:33.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-13T23:42:33.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-13T23:42:33.613 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-13T23:42:33.613 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-13T23:42:33.614 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-13T23:42:33.614 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-13T23:42:33.614 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:42:33.614 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:42:33 smithi115 ceph-mon[78593]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-13T23:42:33.615 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-13T23:42:33.615 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-13T23:42:33.615 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-13T23:42:33.615 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-13T23:42:33.615 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-13T23:42:33.616 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-13T23:42:33.616 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-13T23:42:33.616 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-13T23:42:33.616 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-13T23:42:33.616 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-13T23:42:33.617 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-13T23:42:33.617 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-13T23:42:33.617 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-13T23:42:33.617 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-13T23:42:33.618 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-13T23:42:33.618 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-13T23:42:33.618 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-13T23:42:33.618 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:42:33.619 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-13T23:42:33.619 INFO:journalctl@ceph.mon.c.smithi115.stdout: During handling of the above exception, another exception occurred: 2022-05-13T23:42:33.620 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-13T23:42:33.621 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-13T23:42:33.621 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-13T23:42:33.621 INFO:journalctl@ceph.mon.c.smithi115.stdout: return f(*arg) 2022-05-13T23:42:33.621 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-13T23:42:33.622 INFO:journalctl@ceph.mon.c.smithi115.stdout: self._write_client_files(client_files, host) 2022-05-13T23:42:33.622 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-13T23:42:33.622 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-13T23:42:33.622 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-13T23:42:33.623 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-13T23:42:33.623 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-13T23:42:33.623 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.event_loop.get_result(coro) 2022-05-13T23:42:33.623 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-13T23:42:33.623 INFO:journalctl@ceph.mon.c.smithi115.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-13T23:42:33.624 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-13T23:42:33.624 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.__get_result() 2022-05-13T23:42:33.625 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-13T23:42:33.625 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise self._exception 2022-05-13T23:42:33.625 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-13T23:42:33.625 INFO:journalctl@ceph.mon.c.smithi115.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-13T23:42:33.626 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-13T23:42:33.626 INFO:journalctl@ceph.mon.c.smithi115.stdout: conn = await self._remote_connection(host, addr) 2022-05-13T23:42:33.626 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-13T23:42:33.626 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise 2022-05-13T23:42:33.626 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-13T23:42:33.627 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.gen.throw(type, value, traceback) 2022-05-13T23:42:33.627 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-13T23:42:33.627 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OrchestratorError(msg) 2022-05-13T23:42:33.628 INFO:journalctl@ceph.mon.c.smithi115.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:42:33.628 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:42:33 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:42:33.628 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:42:33 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:42:33.628 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:42:33 smithi115 ceph-mon[78593]: fsmap cephfs:5/5 3 up:standby-replay 1 up:replay 4 up:active 2022-05-13T23:42:33.628 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:42:33 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 885 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:42:33.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:42:33 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:42:33.792 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-13T23:42:33.792 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-13T23:42:33.793 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-13T23:42:33.793 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-13T23:42:33.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-13T23:42:33.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-13T23:42:33.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-13T23:42:33.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-13T23:42:33.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-13T23:42:33.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-13T23:42:33.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-13T23:42:33.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-13T23:42:33.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-13T23:42:33.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-13T23:42:33.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-13T23:42:33.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-13T23:42:33.797 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-13T23:42:33.797 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:42:33.797 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:42:33 smithi047 ceph-mon[77147]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-13T23:42:33.797 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-13T23:42:33.798 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-13T23:42:33.798 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-13T23:42:33.798 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-13T23:42:33.798 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-13T23:42:33.799 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-13T23:42:33.799 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-13T23:42:33.799 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-13T23:42:33.799 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-13T23:42:33.799 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-13T23:42:33.800 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-13T23:42:33.800 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-13T23:42:33.800 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-13T23:42:33.800 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-13T23:42:33.801 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-13T23:42:33.801 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-13T23:42:33.801 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-13T23:42:33.801 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:42:33.802 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-13T23:42:33.802 INFO:journalctl@ceph.mon.b.smithi047.stdout: During handling of the above exception, another exception occurred: 2022-05-13T23:42:33.802 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-13T23:42:33.802 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-13T23:42:33.802 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-13T23:42:33.803 INFO:journalctl@ceph.mon.b.smithi047.stdout: return f(*arg) 2022-05-13T23:42:33.803 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-13T23:42:33.803 INFO:journalctl@ceph.mon.b.smithi047.stdout: self._write_client_files(client_files, host) 2022-05-13T23:42:33.803 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-13T23:42:33.804 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-13T23:42:33.804 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-13T23:42:33.804 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-13T23:42:33.804 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-13T23:42:33.804 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.event_loop.get_result(coro) 2022-05-13T23:42:33.805 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-13T23:42:33.805 INFO:journalctl@ceph.mon.b.smithi047.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-13T23:42:33.805 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-13T23:42:33.805 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.__get_result() 2022-05-13T23:42:33.806 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-13T23:42:33.806 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise self._exception 2022-05-13T23:42:33.806 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-13T23:42:33.806 INFO:journalctl@ceph.mon.b.smithi047.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-13T23:42:33.807 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-13T23:42:33.807 INFO:journalctl@ceph.mon.b.smithi047.stdout: conn = await self._remote_connection(host, addr) 2022-05-13T23:42:33.807 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-13T23:42:33.807 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise 2022-05-13T23:42:33.807 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-13T23:42:33.808 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.gen.throw(type, value, traceback) 2022-05-13T23:42:33.808 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-13T23:42:33.808 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OrchestratorError(msg) 2022-05-13T23:42:33.808 INFO:journalctl@ceph.mon.b.smithi047.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:42:33.809 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:42:33 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:42:33.809 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:42:33 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:42:33.809 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:42:33 smithi047 ceph-mon[77147]: fsmap cephfs:5/5 3 up:standby-replay 1 up:replay 4 up:active 2022-05-13T23:42:33.809 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:42:33 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 885 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:42:34.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:42:34 smithi115 conmon[90393]: 2022-05-13T23:42:34.209+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:42:34.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:42:34 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:42:34.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:42:34 smithi115 ceph-mon[78593]: pgmap v795: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:42:34.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:42:34 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:42:34.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:42:34 smithi047 ceph-mon[77147]: pgmap v795: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:42:35.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:42:35 smithi115 conmon[90393]: 2022-05-13T23:42:35.225+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:42:35.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:42:35 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:42:35.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:42:35 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:42:36.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:42:36 smithi115 conmon[90393]: 2022-05-13T23:42:36.244+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:42:36.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:42:36 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:42:36.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:42:36 smithi115 ceph-mon[78593]: pgmap v796: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:42:36.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:42:36 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:42:36.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:42:36 smithi047 ceph-mon[77147]: pgmap v796: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:42:37.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:42:37 smithi115 conmon[90393]: 2022-05-13T23:42:37.236+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:42:37.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:42:37 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:42:37.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:42:37 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:42:37.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:42:37 smithi115 ceph-mon[78593]: fsmap cephfs:5/5 3 up:standby-replay 1 up:replay 4 up:active 2022-05-13T23:42:37.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:42:37 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:42:37.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:42:37 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:42:37.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:42:37 smithi047 ceph-mon[77147]: fsmap cephfs:5/5 3 up:standby-replay 1 up:replay 4 up:active 2022-05-13T23:42:38.610 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:42:38 smithi115 conmon[90393]: 2022-05-13T23:42:38.262+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:42:38.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:42:38 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:42:38.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:42:38 smithi115 ceph-mon[78593]: pgmap v797: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:42:38.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:42:38 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 890 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:42:38.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:42:38 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:42:38.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:42:38 smithi047 ceph-mon[77147]: pgmap v797: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:42:38.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:42:38 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 890 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:42:39.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:42:39 smithi115 conmon[90393]: 2022-05-13T23:42:39.240+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:42:39.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:42:39 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:42:39.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:42:39 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:42:40.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:42:40 smithi115 conmon[90393]: 2022-05-13T23:42:40.240+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:42:40.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:42:40 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:42:40.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:42:40 smithi115 ceph-mon[78593]: pgmap v798: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:42:40.795 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:42:40 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:42:40.799 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:42:40 smithi047 ceph-mon[77147]: pgmap v798: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:42:41.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:42:41 smithi115 conmon[90393]: 2022-05-13T23:42:41.225+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:42:41.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:42:41 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:42:41.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:42:41 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:42:41.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:42:41 smithi115 ceph-mon[78593]: fsmap cephfs:5/5 3 up:standby-replay 1 up:replay 4 up:active 2022-05-13T23:42:41.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:42:41 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:42:41.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:42:41 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:42:41.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:42:41 smithi047 ceph-mon[77147]: fsmap cephfs:5/5 3 up:standby-replay 1 up:replay 4 up:active 2022-05-13T23:42:42.480 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:42:42 smithi115 conmon[90393]: 2022-05-13T23:42:42.221+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:42:42.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:42:42 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:42:42.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:42:42 smithi047 ceph-mon[77147]: pgmap v799: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:42:42.863 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:42:42 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:42:42.863 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:42:42 smithi115 ceph-mon[78593]: pgmap v799: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:42:43.483 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:42:43 smithi115 conmon[90393]: 2022-05-13T23:42:43.207+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:42:43.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:42:43 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:42:43.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:42:43 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 895 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:42:43.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:42:43 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:42:43.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:42:43 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 895 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:42:44.492 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:42:44 smithi115 conmon[90393]: 2022-05-13T23:42:44.236+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:42:44.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:42:44 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:42:44.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:42:44 smithi047 ceph-mon[77147]: pgmap v800: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:42:44.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:42:44 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:42:44.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:42:44 smithi115 ceph-mon[78593]: pgmap v800: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:42:45.612 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:42:45 smithi115 conmon[90393]: 2022-05-13T23:42:45.258+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:42:45.613 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:42:45 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:42:45.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:42:45 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:42:46.499 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:42:46 smithi115 conmon[90393]: 2022-05-13T23:42:46.216+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:42:46.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:42:46 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:42:46.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:42:46 smithi047 ceph-mon[77147]: pgmap v801: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:42:46.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:42:46 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:42:46.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:42:46 smithi115 ceph-mon[78593]: pgmap v801: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:42:47.500 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:42:47 smithi115 conmon[90393]: 2022-05-13T23:42:47.186+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:42:47.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:42:47 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:42:47.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:42:47 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:42:48.508 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:42:48 smithi115 conmon[90393]: 2022-05-13T23:42:48.157+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:42:48.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:42:48 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:42:48.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:42:48 smithi047 ceph-mon[77147]: pgmap v802: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:42:48.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:42:48 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:42:48.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:42:48 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 900 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:42:48.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:42:48 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:42:48.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:42:48 smithi115 ceph-mon[78593]: pgmap v802: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:42:48.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:42:48 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:42:48.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:42:48 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 900 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:42:49.509 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:42:49 smithi115 conmon[90393]: 2022-05-13T23:42:49.146+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:42:49.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:42:49 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:42:49.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:42:49 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:42:49.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:42:49 smithi047 ceph-mon[77147]: fsmap cephfs:5/5 3 up:standby-replay 1 up:replay 4 up:active 2022-05-13T23:42:49.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:42:49 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:42:49.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:42:49 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:42:49.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:42:49 smithi115 ceph-mon[78593]: fsmap cephfs:5/5 3 up:standby-replay 1 up:replay 4 up:active 2022-05-13T23:42:50.514 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:42:50 smithi115 conmon[90393]: 2022-05-13T23:42:50.136+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:42:50.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:42:50 smithi047 ceph-mon[77147]: pgmap v803: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:42:50.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:42:50 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:42:50.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:42:50 smithi115 ceph-mon[78593]: pgmap v803: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:42:50.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:42:50 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:42:51.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:42:51 smithi115 conmon[90393]: 2022-05-13T23:42:51.105+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:42:51.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:42:51 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:42:51.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:42:51 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:42:52.520 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:42:52 smithi115 conmon[90393]: 2022-05-13T23:42:52.146+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:42:52.797 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:42:52 smithi047 ceph-mon[77147]: pgmap v804: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:42:52.798 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:42:52 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:42:52.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:42:52 smithi115 ceph-mon[78593]: pgmap v804: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:42:52.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:42:52 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:42:53.520 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:42:53 smithi115 conmon[90393]: 2022-05-13T23:42:53.173+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:42:53.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:42:53 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:42:53.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:42:53 smithi047 ceph-mon[77147]: fsmap cephfs:5/5 3 up:standby-replay 1 up:replay 4 up:active 2022-05-13T23:42:53.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:42:53 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 906 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:42:53.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:42:53 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:42:53.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:42:53 smithi115 ceph-mon[78593]: fsmap cephfs:5/5 3 up:standby-replay 1 up:replay 4 up:active 2022-05-13T23:42:53.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:42:53 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 906 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:42:54.523 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:42:54 smithi115 conmon[90393]: 2022-05-13T23:42:54.200+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:42:54.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:42:54 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:42:54.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:42:54 smithi047 ceph-mon[77147]: pgmap v805: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:42:54.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:42:54 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:42:54.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:42:54 smithi115 ceph-mon[78593]: pgmap v805: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:42:55.526 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:42:55 smithi115 conmon[90393]: 2022-05-13T23:42:55.160+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:42:55.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:42:55 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:42:55.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:42:55 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:42:55.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:42:55 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:42:55.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:42:55 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:42:56.534 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:42:56 smithi115 conmon[90393]: 2022-05-13T23:42:56.195+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:42:56.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:42:56 smithi047 ceph-mon[77147]: pgmap v806: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:42:56.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:42:56 smithi115 ceph-mon[78593]: pgmap v806: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:42:57.537 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:42:57 smithi115 conmon[90393]: 2022-05-13T23:42:57.221+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:42:57.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:42:57 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:42:57.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:42:57 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:42:57.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:42:57 smithi047 ceph-mon[77147]: fsmap cephfs:5/5 3 up:standby-replay 1 up:replay 4 up:active 2022-05-13T23:42:57.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:42:57 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:42:57.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:42:57 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:42:57.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:42:57 smithi115 ceph-mon[78593]: fsmap cephfs:5/5 3 up:standby-replay 1 up:replay 4 up:active 2022-05-13T23:42:58.541 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:42:58 smithi115 conmon[90393]: 2022-05-13T23:42:58.217+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:42:58.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:42:58 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:42:58.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:42:58 smithi047 ceph-mon[77147]: pgmap v807: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:42:58.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:42:58 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 910 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:42:58.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:42:58 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:42:58.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:42:58 smithi115 ceph-mon[78593]: pgmap v807: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:42:58.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:42:58 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 910 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:42:59.547 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:42:59 smithi115 conmon[90393]: 2022-05-13T23:42:59.251+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:42:59.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:42:59 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:43:00.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:42:59 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:43:00.550 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:43:00 smithi115 conmon[90393]: 2022-05-13T23:43:00.271+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:43:00.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:43:00 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:43:00.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:43:00 smithi115 ceph-mon[78593]: pgmap v808: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:43:01.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:43:00 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:43:01.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:43:00 smithi047 ceph-mon[77147]: pgmap v808: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:43:01.554 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:43:01 smithi115 conmon[90393]: 2022-05-13T23:43:01.262+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:43:01.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:43:01 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:43:01.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:43:01 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:43:01.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:43:01 smithi115 ceph-mon[78593]: fsmap cephfs:5/5 3 up:standby-replay 1 up:replay 4 up:active 2022-05-13T23:43:02.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:43:01 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:43:02.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:43:01 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:43:02.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:43:01 smithi047 ceph-mon[77147]: fsmap cephfs:5/5 3 up:standby-replay 1 up:replay 4 up:active 2022-05-13T23:43:02.561 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:43:02 smithi115 conmon[90393]: 2022-05-13T23:43:02.221+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:43:02.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:43:02 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:43:02.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:43:02 smithi115 ceph-mon[78593]: pgmap v809: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:43:03.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:43:02 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:43:03.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:43:02 smithi047 ceph-mon[77147]: pgmap v809: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:43:03.558 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:43:03 smithi115 conmon[90393]: 2022-05-13T23:43:03.177+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:43:03.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:43:03 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:43:03.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:43:03 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 915 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:43:03.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:43:03 smithi115 ceph-mon[78593]: pgmap v810: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:43:04.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:43:03 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:43:04.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:43:03 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 915 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:43:04.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:43:03 smithi047 ceph-mon[77147]: pgmap v810: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:43:04.562 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:43:04 smithi115 conmon[90393]: 2022-05-13T23:43:04.133+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:43:04.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:43:04 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:43:04.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:43:04 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:43:05.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:43:04 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:43:05.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:43:04 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:43:05.564 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:43:05 smithi115 conmon[90393]: 2022-05-13T23:43:05.142+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:43:05.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:43:05 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:43:05.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:43:05 smithi115 ceph-mon[78593]: pgmap v811: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:43:06.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:43:05 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:43:06.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:43:05 smithi047 ceph-mon[77147]: pgmap v811: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:43:06.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:43:06 smithi115 conmon[90393]: 2022-05-13T23:43:06.102+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:43:06.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:43:06 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:43:07.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:43:06 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:43:07.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:43:07 smithi115 conmon[90393]: 2022-05-13T23:43:07.064+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:43:07.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:43:07 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:43:07.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:43:07 smithi115 ceph-mon[78593]: pgmap v812: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:43:08.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:43:07 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:43:08.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:43:07 smithi047 ceph-mon[77147]: pgmap v812: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:43:08.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:43:08 smithi115 conmon[90393]: 2022-05-13T23:43:08.065+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:43:08.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:43:08 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:43:08.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:43:08 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 921 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:43:09.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:43:08 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:43:09.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:43:08 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 921 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:43:09.577 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:43:09 smithi115 conmon[90393]: 2022-05-13T23:43:09.115+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:43:09.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:43:09 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:43:09.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:43:09 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:43:09.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:43:09 smithi115 ceph-mon[78593]: fsmap cephfs:5/5 3 up:standby-replay 1 up:replay 4 up:active 2022-05-13T23:43:09.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:43:09 smithi115 ceph-mon[78593]: pgmap v813: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:43:10.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:43:09 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:43:10.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:43:09 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:43:10.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:43:09 smithi047 ceph-mon[77147]: fsmap cephfs:5/5 3 up:standby-replay 1 up:replay 4 up:active 2022-05-13T23:43:10.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:43:09 smithi047 ceph-mon[77147]: pgmap v813: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:43:10.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:43:10 smithi115 conmon[90393]: 2022-05-13T23:43:10.076+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:43:10.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:43:10 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:43:11.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:43:10 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:43:11.361 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:43:11 smithi115 conmon[90393]: 2022-05-13T23:43:11.099+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:43:11.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:43:11 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:43:11.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:43:11 smithi115 ceph-mon[78593]: pgmap v814: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:43:12.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:43:11 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:43:12.045 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:43:11 smithi047 ceph-mon[77147]: pgmap v814: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:43:12.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:43:12 smithi115 conmon[90393]: 2022-05-13T23:43:12.058+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:43:12.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:43:12 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:43:13.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:43:12 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:43:13.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:43:13 smithi115 conmon[90393]: 2022-05-13T23:43:13.105+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:43:13.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:43:13 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:43:13.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:43:13 smithi115 ceph-mon[78593]: fsmap cephfs:5/5 3 up:standby-replay 1 up:replay 4 up:active 2022-05-13T23:43:13.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:43:13 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:43:13.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:43:13 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-13T23:43:13.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:43:13 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-13T23:43:13.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:43:13 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 926 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:43:13.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:43:13 smithi115 ceph-mon[78593]: pgmap v815: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:43:14.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:43:13 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:43:14.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:43:13 smithi047 ceph-mon[77147]: fsmap cephfs:5/5 3 up:standby-replay 1 up:replay 4 up:active 2022-05-13T23:43:14.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:43:13 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:43:14.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:43:13 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-13T23:43:14.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:43:13 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-13T23:43:14.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:43:13 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 926 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:43:14.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:43:13 smithi047 ceph-mon[77147]: pgmap v815: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:43:14.591 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:43:14 smithi115 conmon[90393]: 2022-05-13T23:43:14.115+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:43:14.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:43:14 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:43:15.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:43:14 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:43:15.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:43:15 smithi115 conmon[90393]: 2022-05-13T23:43:15.085+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:43:15.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:43:15 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:43:15.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:43:15 smithi115 ceph-mon[78593]: pgmap v816: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:43:16.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:43:15 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:43:16.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:43:15 smithi047 ceph-mon[77147]: pgmap v816: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:43:16.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:43:16 smithi115 conmon[90393]: 2022-05-13T23:43:16.074+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:43:16.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:43:16 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:43:17.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:43:16 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:43:17.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:43:17 smithi115 conmon[90393]: 2022-05-13T23:43:17.105+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:43:17.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:43:17 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:43:17.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:43:17 smithi115 ceph-mon[78593]: fsmap cephfs:5/5 3 up:standby-replay 1 up:replay 4 up:active 2022-05-13T23:43:17.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:43:17 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:43:17.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:43:17 smithi115 ceph-mon[78593]: pgmap v817: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:43:18.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:43:17 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:43:18.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:43:17 smithi047 ceph-mon[77147]: fsmap cephfs:5/5 3 up:standby-replay 1 up:replay 4 up:active 2022-05-13T23:43:18.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:43:17 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:43:18.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:43:17 smithi047 ceph-mon[77147]: pgmap v817: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:43:18.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:43:18 smithi115 conmon[90393]: 2022-05-13T23:43:18.139+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:43:19.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:43:18 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:43:19.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:43:18 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 931 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:43:19.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:43:18 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:43:19.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:43:18 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 931 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:43:19.610 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:43:19 smithi115 conmon[90393]: 2022-05-13T23:43:19.182+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:43:20.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:43:19 smithi047 ceph-mon[77147]: pgmap v818: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:43:20.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:43:19 smithi115 ceph-mon[78593]: pgmap v818: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:43:20.611 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:43:20 smithi115 conmon[90393]: 2022-05-13T23:43:20.161+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:43:21.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:43:20 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:43:21.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:43:20 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:43:21.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:43:20 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:43:21.112 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:43:20 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:43:21.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:43:21 smithi115 conmon[90393]: 2022-05-13T23:43:21.170+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:43:22.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:43:21 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:43:22.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:43:21 smithi047 ceph-mon[77147]: fsmap cephfs:5/5 3 up:standby-replay 1 up:replay 4 up:active 2022-05-13T23:43:22.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:43:21 smithi047 ceph-mon[77147]: pgmap v819: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:43:22.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:43:21 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:43:22.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:43:21 smithi115 ceph-mon[78593]: fsmap cephfs:5/5 3 up:standby-replay 1 up:replay 4 up:active 2022-05-13T23:43:22.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:43:21 smithi115 ceph-mon[78593]: pgmap v819: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:43:22.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:43:22 smithi115 conmon[90393]: 2022-05-13T23:43:22.146+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:43:23.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:43:22 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:43:23.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:43:22 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:43:23.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:43:22 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:43:23.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:43:22 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:43:23.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:43:23 smithi115 conmon[90393]: 2022-05-13T23:43:23.184+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:43:24.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:43:23 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 935 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:43:24.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:43:23 smithi047 ceph-mon[77147]: pgmap v820: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:43:24.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:43:23 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 935 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:43:24.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:43:23 smithi115 ceph-mon[78593]: pgmap v820: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:43:24.620 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:43:24 smithi115 conmon[90393]: 2022-05-13T23:43:24.157+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:43:25.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:43:24 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:43:25.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:43:24 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:43:25.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:43:24 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:43:25.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:43:24 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:43:25.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:43:25 smithi115 conmon[90393]: 2022-05-13T23:43:25.142+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:43:26.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:43:25 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:43:26.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:43:25 smithi047 ceph-mon[77147]: pgmap v821: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:43:26.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:43:25 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:43:26.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:43:25 smithi115 ceph-mon[78593]: pgmap v821: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:43:26.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:43:26 smithi115 conmon[90393]: 2022-05-13T23:43:26.180+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:43:27.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:43:27 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:43:27.613 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:43:27 smithi115 conmon[90393]: 2022-05-13T23:43:27.185+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:43:27.615 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:43:27 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:43:28.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:43:28 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:43:28.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:43:28 smithi047 ceph-mon[77147]: pgmap v822: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:43:28.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:43:28 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:43:28.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:43:28 smithi115 conmon[90393]: 2022-05-13T23:43:28.139+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:43:28.614 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:43:28 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:43:28.614 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:43:28 smithi115 ceph-mon[78593]: pgmap v822: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:43:28.615 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:43:28 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:43:29.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:43:29 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 941 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:43:29.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:43:29 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:43:29.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:43:29 smithi115 conmon[90393]: 2022-05-13T23:43:29.120+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:43:29.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:43:29 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 941 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:43:29.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:43:29 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:43:30.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:43:30 smithi115 conmon[90393]: 2022-05-13T23:43:30.092+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:43:30.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:43:30 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:43:30.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:43:30 smithi115 ceph-mon[78593]: fsmap cephfs:5/5 3 up:standby-replay 1 up:replay 4 up:active 2022-05-13T23:43:30.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:43:30 smithi115 ceph-mon[78593]: pgmap v823: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:43:30.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:43:30 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:43:30.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:43:30 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:43:30.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:43:30 smithi047 ceph-mon[77147]: fsmap cephfs:5/5 3 up:standby-replay 1 up:replay 4 up:active 2022-05-13T23:43:30.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:43:30 smithi047 ceph-mon[77147]: pgmap v823: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:43:30.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:43:30 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:43:31.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:43:31 smithi115 conmon[90393]: 2022-05-13T23:43:31.090+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:43:31.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:43:31 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:43:31.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:43:31 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:43:32.314 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:43:32 smithi115 conmon[90393]: 2022-05-13T23:43:32.119+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:43:32.566 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:43:32 smithi115 ceph-mon[78593]: pgmap v824: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:43:32.566 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:43:32 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-13T23:43:32.566 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:43:32 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-13T23:43:32.567 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:43:32 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-13T23:43:32.567 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:43:32 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:43:32.577 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:43:32 smithi047 ceph-mon[77147]: pgmap v824: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:43:32.577 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:43:32 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-13T23:43:32.578 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:43:32 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-13T23:43:32.578 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:43:32 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-13T23:43:32.578 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:43:32 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:43:33.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:43:33 smithi115 conmon[90393]: 2022-05-13T23:43:33.104+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:43:33.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:43:33 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:43:33.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:43:33 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 946 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:43:33.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:43:33 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:43:33.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:43:33 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 946 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:43:33.542 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:43:33 smithi047 conmon[78521]: 2022-05-13T23:43:33.423+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:43:33.542 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:43:33 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-13T23:43:33.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:43:33 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-13T23:43:33.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:43:33 smithi047 conmon[78521]: yield 2022-05-13T23:43:33.549 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:43:33 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-13T23:43:33.549 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:43:33 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-13T23:43:33.549 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:43:33 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-13T23:43:33.550 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:43:33 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-13T23:43:33.550 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:43:33 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-13T23:43:33.550 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:43:33 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-13T23:43:33.550 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:43:33 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-13T23:43:33.551 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:43:33 smithi047 conmon[78521]: raise exceptions[0] 2022-05-13T23:43:33.551 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:43:33 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-13T23:43:33.551 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:43:33 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-13T23:43:33.551 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:43:33 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-13T23:43:33.552 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:43:33 smithi047 conmon[78521]: return (yield from fut) 2022-05-13T23:43:33.552 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:43:33 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-13T23:43:33.552 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:43:33 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-13T23:43:33.552 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:43:33 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:43:34.312 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:43:34 smithi115 conmon[90393]: 2022-05-13T23:43:34.137+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:43:34.549 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:43:34 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:43:34.549 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:43:34 smithi047 ceph-mon[77147]: fsmap cephfs:5/5 3 up:standby-replay 1 up:replay 4 up:active 2022-05-13T23:43:34.549 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:43:34 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:43:34.550 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-13T23:43:34.551 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-13T23:43:34.551 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-13T23:43:34.551 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-13T23:43:34.551 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-13T23:43:34.551 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-13T23:43:34.560 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-13T23:43:34.560 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-13T23:43:34.561 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-13T23:43:34.561 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-13T23:43:34.561 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-13T23:43:34.562 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-13T23:43:34.562 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-13T23:43:34.562 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-13T23:43:34.562 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-13T23:43:34.562 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-13T23:43:34.563 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-13T23:43:34.563 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:43:34.563 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:43:34 smithi047 ceph-mon[77147]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-13T23:43:34.563 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:43:34 smithi047 ceph-mon[77147]: pgmap v825: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:43:34.564 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:43:34 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:43:34.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:43:34 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:43:34.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:43:34 smithi115 ceph-mon[78593]: fsmap cephfs:5/5 3 up:standby-replay 1 up:replay 4 up:active 2022-05-13T23:43:34.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:43:34 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:43:34.609 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-13T23:43:34.609 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-13T23:43:34.609 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-13T23:43:34.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-13T23:43:34.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-13T23:43:34.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-13T23:43:34.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-13T23:43:34.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-13T23:43:34.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-13T23:43:34.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-13T23:43:34.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-13T23:43:34.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-13T23:43:34.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-13T23:43:34.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-13T23:43:34.613 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-13T23:43:34.613 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-13T23:43:34.613 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-13T23:43:34.613 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:43:34.614 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:43:34 smithi115 ceph-mon[78593]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-13T23:43:34.614 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:43:34 smithi115 ceph-mon[78593]: pgmap v825: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:43:34.614 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:43:34 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:43:35.471 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:43:35 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:43:35.472 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:43:35 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' 2022-05-13T23:43:35.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:43:35 smithi115 conmon[90393]: 2022-05-13T23:43:35.147+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:43:35.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:43:35 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:43:35.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:43:35 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' 2022-05-13T23:43:36.537 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:43:36 smithi115 conmon[90393]: 2022-05-13T23:43:36.122+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:43:36.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:43:36 smithi047 ceph-mon[77147]: Detected new or changed devices on smithi115 2022-05-13T23:43:36.795 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:43:36 smithi047 ceph-mon[77147]: pgmap v826: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:43:36.795 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:43:36 smithi047 ceph-mon[77147]: Detected new or changed devices on smithi047 2022-05-13T23:43:36.795 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:43:36 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' 2022-05-13T23:43:36.796 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:43:36 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:43:36.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:43:36 smithi047 conmon[78521]: 2022-05-13T23:43:36.495+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:43:36.797 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:43:36 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-13T23:43:36.798 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:43:36 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-13T23:43:36.798 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:43:36 smithi047 conmon[78521]: yield 2022-05-13T23:43:36.798 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:43:36 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-13T23:43:36.799 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:43:36 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-13T23:43:36.799 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:43:36 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-13T23:43:36.799 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:43:36 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-13T23:43:36.800 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:43:36 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-13T23:43:36.800 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:43:36 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-13T23:43:36.800 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:43:36 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-13T23:43:36.800 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:43:36 smithi047 conmon[78521]: raise exceptions[0] 2022-05-13T23:43:36.802 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:43:36 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-13T23:43:36.802 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:43:36 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-13T23:43:36.803 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:43:36 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-13T23:43:36.803 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:43:36 smithi047 conmon[78521]: return (yield from fut) 2022-05-13T23:43:36.803 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:43:36 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-13T23:43:36.804 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:43:36 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-13T23:43:36.804 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:43:36 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:43:36.804 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:43:36 smithi047 conmon[78521]: 2022-05-13T23:43:36.496+0000 7f533f777700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-13T23:43:36.804 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:43:36 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-13T23:43:36.805 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:43:36 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-13T23:43:36.805 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:43:36 smithi047 conmon[78521]: yield 2022-05-13T23:43:36.805 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:43:36 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-13T23:43:36.806 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:43:36 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-13T23:43:36.806 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:43:36 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-13T23:43:36.806 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:43:36 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-13T23:43:36.806 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:43:36 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-13T23:43:36.806 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:43:36 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-13T23:43:36.807 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:43:36 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-13T23:43:36.807 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:43:36 smithi047 conmon[78521]: raise exceptions[0] 2022-05-13T23:43:36.807 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:43:36 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-13T23:43:36.807 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:43:36 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-13T23:43:36.807 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:43:36 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-13T23:43:36.808 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:43:36 smithi047 conmon[78521]: return (yield from fut) 2022-05-13T23:43:36.808 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:43:36 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-13T23:43:36.808 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:43:36 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-13T23:43:36.808 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:43:36 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:43:36.809 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:43:36 smithi047 conmon[78521]: 2022-05-13T23:43:36.809 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:43:36 smithi047 conmon[78521]: During handling of the above exception, another exception occurred: 2022-05-13T23:43:36.822 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:43:36 smithi047 conmon[78521]: 2022-05-13T23:43:36.822 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:43:36 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-13T23:43:36.823 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:43:36 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-13T23:43:36.823 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:43:36 smithi047 conmon[78521]: return f(*arg) 2022-05-13T23:43:36.823 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:43:36 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-13T23:43:36.824 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:43:36 smithi047 conmon[78521]: self._write_client_files(client_files, host) 2022-05-13T23:43:36.824 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:43:36 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-13T23:43:36.824 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:43:36 smithi047 conmon[78521]: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-13T23:43:36.824 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:43:36 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-13T23:43:36.825 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:43:36 smithi047 conmon[78521]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-13T23:43:36.825 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:43:36 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-13T23:43:36.825 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:43:36 smithi047 conmon[78521]: return self.event_loop.get_result(coro) 2022-05-13T23:43:36.825 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:43:36 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-13T23:43:36.826 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:43:36 smithi047 conmon[78521]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-13T23:43:36.826 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:43:36 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-13T23:43:36.826 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:43:36 smithi047 conmon[78521]: return self.__get_result() 2022-05-13T23:43:36.826 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:43:36 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-13T23:43:36.826 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:43:36 smithi047 conmon[78521]: raise self._exception 2022-05-13T23:43:36.827 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:43:36 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-13T23:43:36.827 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:43:36 smithi047 conmon[78521]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-13T23:43:36.827 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:43:36 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-13T23:43:36.827 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:43:36 smithi047 conmon[78521]: conn = await self._remote_connection(host, addr) 2022-05-13T23:43:36.828 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:43:36 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-13T23:43:36.828 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:43:36 smithi047 conmon[78521]: raise 2022-05-13T23:43:36.828 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:43:36 smithi047 conmon[78521]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-13T23:43:36.828 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:43:36 smithi047 conmon[78521]: self.gen.throw(type, value, traceback) 2022-05-13T23:43:36.828 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:43:36 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-13T23:43:36.829 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:43:36 smithi047 conmon[78521]: raise OrchestratorError(msg) 2022-05-13T23:43:36.829 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:43:36 smithi047 conmon[78521]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:43:36.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:43:36 smithi115 ceph-mon[78593]: Detected new or changed devices on smithi115 2022-05-13T23:43:36.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:43:36 smithi115 ceph-mon[78593]: pgmap v826: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:43:36.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:43:36 smithi115 ceph-mon[78593]: Detected new or changed devices on smithi047 2022-05-13T23:43:36.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:43:36 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' 2022-05-13T23:43:36.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:43:36 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:43:37.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:43:37 smithi115 conmon[90393]: 2022-05-13T23:43:37.099+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:43:37.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:43:37 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:43:37.793 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-13T23:43:37.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-13T23:43:37.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-13T23:43:37.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-13T23:43:37.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-13T23:43:37.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-13T23:43:37.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-13T23:43:37.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-13T23:43:37.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-13T23:43:37.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-13T23:43:37.797 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-13T23:43:37.797 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-13T23:43:37.798 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-13T23:43:37.798 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-13T23:43:37.798 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-13T23:43:37.799 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-13T23:43:37.800 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-13T23:43:37.800 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:43:37.800 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:43:37 smithi047 ceph-mon[77147]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-13T23:43:37.801 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-13T23:43:37.801 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-13T23:43:37.801 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-13T23:43:37.802 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-13T23:43:37.802 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-13T23:43:37.802 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-13T23:43:37.802 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-13T23:43:37.803 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-13T23:43:37.803 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-13T23:43:37.803 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-13T23:43:37.803 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-13T23:43:37.804 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-13T23:43:37.804 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-13T23:43:37.804 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-13T23:43:37.804 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-13T23:43:37.805 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-13T23:43:37.805 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-13T23:43:37.806 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:43:37.806 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-13T23:43:37.806 INFO:journalctl@ceph.mon.b.smithi047.stdout: During handling of the above exception, another exception occurred: 2022-05-13T23:43:37.806 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-13T23:43:37.807 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-13T23:43:37.807 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-13T23:43:37.807 INFO:journalctl@ceph.mon.b.smithi047.stdout: return f(*arg) 2022-05-13T23:43:37.807 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-13T23:43:37.808 INFO:journalctl@ceph.mon.b.smithi047.stdout: self._write_client_files(client_files, host) 2022-05-13T23:43:37.808 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-13T23:43:37.809 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-13T23:43:37.809 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-13T23:43:37.809 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-13T23:43:37.810 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-13T23:43:37.810 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.event_loop.get_result(coro) 2022-05-13T23:43:37.811 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-13T23:43:37.811 INFO:journalctl@ceph.mon.b.smithi047.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-13T23:43:37.811 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-13T23:43:37.811 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.__get_result() 2022-05-13T23:43:37.812 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-13T23:43:37.812 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise self._exception 2022-05-13T23:43:37.812 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-13T23:43:37.812 INFO:journalctl@ceph.mon.b.smithi047.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-13T23:43:37.813 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-13T23:43:37.813 INFO:journalctl@ceph.mon.b.smithi047.stdout: conn = await self._remote_connection(host, addr) 2022-05-13T23:43:37.813 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-13T23:43:37.814 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise 2022-05-13T23:43:37.814 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-13T23:43:37.814 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.gen.throw(type, value, traceback) 2022-05-13T23:43:37.814 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-13T23:43:37.815 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OrchestratorError(msg) 2022-05-13T23:43:37.815 INFO:journalctl@ceph.mon.b.smithi047.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:43:37.815 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:43:37 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:43:37.815 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:43:37 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:43:37.816 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:43:37 smithi047 ceph-mon[77147]: fsmap cephfs:5/5 3 up:standby-replay 1 up:replay 4 up:active 2022-05-13T23:43:37.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:43:37 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:43:37.859 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-13T23:43:37.859 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-13T23:43:37.859 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-13T23:43:37.859 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-13T23:43:37.859 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-13T23:43:37.860 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-13T23:43:37.860 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-13T23:43:37.860 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-13T23:43:37.860 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-13T23:43:37.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-13T23:43:37.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-13T23:43:37.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-13T23:43:37.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-13T23:43:37.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-13T23:43:37.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-13T23:43:37.863 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-13T23:43:37.863 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-13T23:43:37.863 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:43:37.863 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:43:37 smithi115 ceph-mon[78593]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-13T23:43:37.864 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-13T23:43:37.864 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-13T23:43:37.864 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-13T23:43:37.864 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-13T23:43:37.865 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-13T23:43:37.865 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-13T23:43:37.865 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-13T23:43:37.865 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-13T23:43:37.866 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-13T23:43:37.866 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-13T23:43:37.866 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-13T23:43:37.866 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-13T23:43:37.866 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-13T23:43:37.867 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-13T23:43:37.867 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-13T23:43:37.867 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-13T23:43:37.867 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-13T23:43:37.868 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:43:37.868 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-13T23:43:37.868 INFO:journalctl@ceph.mon.c.smithi115.stdout: During handling of the above exception, another exception occurred: 2022-05-13T23:43:37.868 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-13T23:43:37.868 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-13T23:43:37.869 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-13T23:43:37.869 INFO:journalctl@ceph.mon.c.smithi115.stdout: return f(*arg) 2022-05-13T23:43:37.869 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-13T23:43:37.869 INFO:journalctl@ceph.mon.c.smithi115.stdout: self._write_client_files(client_files, host) 2022-05-13T23:43:37.870 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-13T23:43:37.870 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-13T23:43:37.870 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-13T23:43:37.870 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-13T23:43:37.870 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-13T23:43:37.871 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.event_loop.get_result(coro) 2022-05-13T23:43:37.871 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-13T23:43:37.871 INFO:journalctl@ceph.mon.c.smithi115.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-13T23:43:37.871 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-13T23:43:37.872 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.__get_result() 2022-05-13T23:43:37.872 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-13T23:43:37.872 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise self._exception 2022-05-13T23:43:37.872 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-13T23:43:37.872 INFO:journalctl@ceph.mon.c.smithi115.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-13T23:43:37.873 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-13T23:43:37.873 INFO:journalctl@ceph.mon.c.smithi115.stdout: conn = await self._remote_connection(host, addr) 2022-05-13T23:43:37.874 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-13T23:43:37.874 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise 2022-05-13T23:43:37.874 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-13T23:43:37.874 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.gen.throw(type, value, traceback) 2022-05-13T23:43:37.875 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-13T23:43:37.875 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OrchestratorError(msg) 2022-05-13T23:43:37.875 INFO:journalctl@ceph.mon.c.smithi115.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:43:37.875 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:43:37 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:43:37.876 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:43:37 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:43:37.876 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:43:37 smithi115 ceph-mon[78593]: fsmap cephfs:5/5 3 up:standby-replay 1 up:replay 4 up:active 2022-05-13T23:43:38.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:43:38 smithi115 conmon[90393]: 2022-05-13T23:43:38.060+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:43:38.804 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:43:38 smithi047 ceph-mon[77147]: pgmap v827: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:43:38.805 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:43:38 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:43:38.806 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:43:38 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 951 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:43:38.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:43:38 smithi115 ceph-mon[78593]: pgmap v827: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:43:38.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:43:38 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:43:38.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:43:38 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 951 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:43:39.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:43:39 smithi115 conmon[90393]: 2022-05-13T23:43:39.087+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:43:39.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:43:39 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:43:40.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:43:39 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:43:40.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:43:40 smithi115 conmon[90393]: 2022-05-13T23:43:40.051+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:43:40.864 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:43:40 smithi115 ceph-mon[78593]: pgmap v828: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:43:40.865 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:43:40 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:43:41.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:43:40 smithi047 ceph-mon[77147]: pgmap v828: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:43:41.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:43:40 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:43:41.361 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:43:41 smithi115 conmon[90393]: 2022-05-13T23:43:41.018+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:43:41.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:43:41 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:43:41.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:43:41 smithi115 ceph-mon[78593]: fsmap cephfs:5/5 3 up:standby-replay 1 up:replay 4 up:active 2022-05-13T23:43:41.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:43:41 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:43:42.045 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:43:41 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:43:42.045 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:43:41 smithi047 ceph-mon[77147]: fsmap cephfs:5/5 3 up:standby-replay 1 up:replay 4 up:active 2022-05-13T23:43:42.046 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:43:41 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:43:42.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:43:41 smithi115 conmon[90393]: 2022-05-13T23:43:41.971+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:43:42.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:43:42 smithi115 ceph-mon[78593]: pgmap v829: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:43:42.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:43:42 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:43:43.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:43:42 smithi047 ceph-mon[77147]: pgmap v829: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:43:43.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:43:42 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:43:43.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:43:42 smithi115 conmon[90393]: 2022-05-13T23:43:42.976+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:43:43.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:43:43 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:43:43.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:43:43 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 956 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:43:44.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:43:43 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:43:44.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:43:43 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 956 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:43:44.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:43:43 smithi115 conmon[90393]: 2022-05-13T23:43:43.932+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:43:44.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:43:44 smithi115 ceph-mon[78593]: pgmap v830: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:43:44.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:43:44 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:43:45.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:43:44 smithi047 ceph-mon[77147]: pgmap v830: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:43:45.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:43:44 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:43:45.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:43:44 smithi115 conmon[90393]: 2022-05-13T23:43:44.977+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:43:45.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:43:45 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:43:46.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:43:45 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:43:46.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:43:45 smithi115 conmon[90393]: 2022-05-13T23:43:45.962+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:43:46.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:43:46 smithi115 ceph-mon[78593]: pgmap v831: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:43:46.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:43:46 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:43:47.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:43:46 smithi047 ceph-mon[77147]: pgmap v831: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:43:47.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:43:46 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:43:47.361 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:43:46 smithi115 conmon[90393]: 2022-05-13T23:43:46.976+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:43:47.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:43:47 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:43:47.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:43:47 smithi115 ceph-mon[78593]: pgmap v832: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:43:48.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:43:47 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:43:48.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:43:47 smithi047 ceph-mon[77147]: pgmap v832: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:43:48.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:43:47 smithi115 conmon[90393]: 2022-05-13T23:43:47.996+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:43:48.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:43:48 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:43:48.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:43:48 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 960 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:43:49.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:43:48 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:43:49.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:43:48 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 960 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:43:49.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:43:49 smithi115 conmon[90393]: 2022-05-13T23:43:49.013+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:43:49.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:43:49 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:43:49.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:43:49 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:43:49.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:43:49 smithi115 ceph-mon[78593]: fsmap cephfs:5/5 3 up:standby-replay 1 up:replay 4 up:active 2022-05-13T23:43:49.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:43:49 smithi115 ceph-mon[78593]: pgmap v833: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:43:50.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:43:49 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:43:50.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:43:49 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:43:50.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:43:49 smithi047 ceph-mon[77147]: fsmap cephfs:5/5 3 up:standby-replay 1 up:replay 4 up:active 2022-05-13T23:43:50.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:43:49 smithi047 ceph-mon[77147]: pgmap v833: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:43:50.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:43:49 smithi115 conmon[90393]: 2022-05-13T23:43:49.989+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:43:50.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:43:50 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:43:51.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:43:50 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:43:51.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:43:51 smithi115 conmon[90393]: 2022-05-13T23:43:51.022+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:43:51.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:43:51 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:43:51.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:43:51 smithi115 ceph-mon[78593]: pgmap v834: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:43:52.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:43:51 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:43:52.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:43:51 smithi047 ceph-mon[77147]: pgmap v834: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:43:52.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:43:51 smithi115 conmon[90393]: 2022-05-13T23:43:51.999+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:43:52.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:43:52 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:43:53.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:43:52 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:43:53.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:43:52 smithi115 conmon[90393]: 2022-05-13T23:43:52.950+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:43:53.862 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:43:53 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:43:53.863 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:43:53 smithi115 ceph-mon[78593]: fsmap cephfs:5/5 3 up:standby-replay 1 up:replay 4 up:active 2022-05-13T23:43:53.863 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:43:53 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:43:53.864 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:43:53 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 965 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:43:53.864 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:43:53 smithi115 ceph-mon[78593]: pgmap v835: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:43:54.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:43:53 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:43:54.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:43:53 smithi047 ceph-mon[77147]: fsmap cephfs:5/5 3 up:standby-replay 1 up:replay 4 up:active 2022-05-13T23:43:54.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:43:53 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:43:54.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:43:53 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 965 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:43:54.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:43:53 smithi047 ceph-mon[77147]: pgmap v835: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:43:54.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:43:53 smithi115 conmon[90393]: 2022-05-13T23:43:53.990+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:43:54.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:43:54 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:43:55.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:43:54 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:43:55.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:43:55 smithi115 conmon[90393]: 2022-05-13T23:43:55.013+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:43:55.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:43:55 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:43:55.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:43:55 smithi115 ceph-mon[78593]: pgmap v836: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:43:56.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:43:55 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:43:56.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:43:55 smithi047 ceph-mon[77147]: pgmap v836: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:43:56.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:43:56 smithi115 conmon[90393]: 2022-05-13T23:43:56.034+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:43:56.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:43:56 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:43:57.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:43:56 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:43:57.360 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:43:57 smithi115 conmon[90393]: 2022-05-13T23:43:57.077+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:43:57.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:43:57 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:43:57.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:43:57 smithi115 ceph-mon[78593]: fsmap cephfs:5/5 3 up:standby-replay 1 up:replay 4 up:active 2022-05-13T23:43:57.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:43:57 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:43:57.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:43:57 smithi115 ceph-mon[78593]: pgmap v837: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:43:58.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:43:57 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:43:58.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:43:57 smithi047 ceph-mon[77147]: fsmap cephfs:5/5 3 up:standby-replay 1 up:replay 4 up:active 2022-05-13T23:43:58.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:43:57 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:43:58.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:43:57 smithi047 ceph-mon[77147]: pgmap v837: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:43:58.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:43:58 smithi115 conmon[90393]: 2022-05-13T23:43:58.106+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:43:58.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:43:58 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:43:58.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:43:58 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 970 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:43:59.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:43:58 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:43:59.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:43:58 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 970 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:43:59.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:43:59 smithi115 conmon[90393]: 2022-05-13T23:43:59.075+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:43:59.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:43:59 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:43:59.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:43:59 smithi115 ceph-mon[78593]: pgmap v838: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:44:00.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:43:59 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:44:00.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:43:59 smithi047 ceph-mon[77147]: pgmap v838: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:44:00.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:44:00 smithi115 conmon[90393]: 2022-05-13T23:44:00.098+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:44:01.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:44:00 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:44:01.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:44:00 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:44:01.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:44:01 smithi115 conmon[90393]: 2022-05-13T23:44:01.120+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:44:02.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:44:01 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:44:02.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:44:01 smithi047 ceph-mon[77147]: fsmap cephfs:5/5 3 up:standby-replay 1 up:replay 4 up:active 2022-05-13T23:44:02.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:44:01 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:44:02.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:44:01 smithi047 ceph-mon[77147]: pgmap v839: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:44:02.085 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:44:01 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:44:02.086 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:44:01 smithi115 ceph-mon[78593]: fsmap cephfs:5/5 3 up:standby-replay 1 up:replay 4 up:active 2022-05-13T23:44:02.086 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:44:01 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:44:02.086 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:44:01 smithi115 ceph-mon[78593]: pgmap v839: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:44:02.360 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:44:02 smithi115 conmon[90393]: 2022-05-13T23:44:02.078+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:44:03.049 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:44:02 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:44:03.088 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:44:02 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:44:03.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:44:03 smithi115 conmon[90393]: 2022-05-13T23:44:03.086+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:44:04.046 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:44:03 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:44:04.046 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:44:03 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 975 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:44:04.047 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:44:03 smithi047 ceph-mon[77147]: pgmap v840: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:44:04.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:44:03 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:44:04.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:44:03 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 975 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:44:04.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:44:03 smithi115 ceph-mon[78593]: pgmap v840: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:44:04.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:44:04 smithi115 conmon[90393]: 2022-05-13T23:44:04.113+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:44:05.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:44:04 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:44:05.101 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:44:04 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:44:05.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:44:05 smithi115 conmon[90393]: 2022-05-13T23:44:05.100+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:44:06.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:44:05 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:44:06.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:44:05 smithi047 ceph-mon[77147]: pgmap v841: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:44:06.090 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:44:05 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:44:06.091 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:44:05 smithi115 ceph-mon[78593]: pgmap v841: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:44:06.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:44:06 smithi115 conmon[90393]: 2022-05-13T23:44:06.089+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:44:07.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:44:06 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:44:07.114 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:44:06 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:44:07.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:44:07 smithi115 conmon[90393]: 2022-05-13T23:44:07.134+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:44:08.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:44:07 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:44:08.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:44:07 smithi047 ceph-mon[77147]: pgmap v842: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:44:08.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:44:07 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:44:08.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:44:07 smithi115 ceph-mon[78593]: pgmap v842: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:44:08.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:44:08 smithi115 conmon[90393]: 2022-05-13T23:44:08.111+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:44:09.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:44:08 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:44:09.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:44:08 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 981 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:44:09.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:44:08 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:44:09.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:44:08 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 981 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:44:09.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:44:09 smithi115 conmon[90393]: 2022-05-13T23:44:09.143+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:44:10.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:44:09 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:44:10.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:44:09 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:44:10.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:44:09 smithi047 ceph-mon[77147]: fsmap cephfs:5/5 3 up:standby-replay 1 up:replay 4 up:active 2022-05-13T23:44:10.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:44:09 smithi047 ceph-mon[77147]: pgmap v843: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:44:10.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:44:09 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:44:10.112 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:44:09 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:44:10.113 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:44:09 smithi115 ceph-mon[78593]: fsmap cephfs:5/5 3 up:standby-replay 1 up:replay 4 up:active 2022-05-13T23:44:10.113 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:44:09 smithi115 ceph-mon[78593]: pgmap v843: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:44:10.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:44:10 smithi115 conmon[90393]: 2022-05-13T23:44:10.118+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:44:11.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:44:10 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:44:11.116 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:44:10 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:44:11.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:44:11 smithi115 conmon[90393]: 2022-05-13T23:44:11.104+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:44:12.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:44:11 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:44:12.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:44:11 smithi047 ceph-mon[77147]: pgmap v844: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:44:12.105 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:44:11 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:44:12.105 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:44:11 smithi115 ceph-mon[78593]: pgmap v844: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:44:12.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:44:12 smithi115 conmon[90393]: 2022-05-13T23:44:12.102+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:44:13.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:44:12 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:44:13.078 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:44:12 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:44:13.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:44:13 smithi115 conmon[90393]: 2022-05-13T23:44:13.077+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:44:14.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:44:13 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:44:14.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:44:13 smithi047 ceph-mon[77147]: fsmap cephfs:5/5 3 up:standby-replay 1 up:replay 4 up:active 2022-05-13T23:44:14.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:44:13 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:44:14.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:44:13 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-13T23:44:14.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:44:13 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-13T23:44:14.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:44:13 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 986 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:44:14.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:44:13 smithi047 ceph-mon[77147]: pgmap v845: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:44:14.079 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:44:13 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:44:14.080 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:44:13 smithi115 ceph-mon[78593]: fsmap cephfs:5/5 3 up:standby-replay 1 up:replay 4 up:active 2022-05-13T23:44:14.080 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:44:13 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:44:14.080 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:44:13 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-13T23:44:14.080 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:44:13 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-13T23:44:14.080 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:44:13 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 986 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:44:14.081 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:44:13 smithi115 ceph-mon[78593]: pgmap v845: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:44:14.360 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:44:14 smithi115 conmon[90393]: 2022-05-13T23:44:14.077+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:44:15.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:44:14 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:44:15.067 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:44:14 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:44:15.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:44:15 smithi115 conmon[90393]: 2022-05-13T23:44:15.066+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:44:16.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:44:15 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:44:16.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:44:15 smithi047 ceph-mon[77147]: pgmap v846: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:44:16.087 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:44:15 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:44:16.087 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:44:15 smithi115 ceph-mon[78593]: pgmap v846: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:44:16.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:44:16 smithi115 conmon[90393]: 2022-05-13T23:44:16.085+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:44:17.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:44:16 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:44:17.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:44:16 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:44:17.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:44:17 smithi115 conmon[90393]: 2022-05-13T23:44:17.117+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:44:18.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:44:17 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:44:18.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:44:17 smithi047 ceph-mon[77147]: fsmap cephfs:5/5 3 up:standby-replay 1 up:replay 4 up:active 2022-05-13T23:44:18.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:44:17 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:44:18.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:44:17 smithi047 ceph-mon[77147]: pgmap v847: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:44:18.078 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:44:17 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:44:18.078 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:44:17 smithi115 ceph-mon[78593]: fsmap cephfs:5/5 3 up:standby-replay 1 up:replay 4 up:active 2022-05-13T23:44:18.078 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:44:17 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:44:18.079 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:44:17 smithi115 ceph-mon[78593]: pgmap v847: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:44:18.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:44:18 smithi115 conmon[90393]: 2022-05-13T23:44:18.076+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:44:19.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:44:18 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:44:19.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:44:18 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 990 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:44:19.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:44:18 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:44:19.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:44:18 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 990 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:44:19.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:44:19 smithi115 conmon[90393]: 2022-05-13T23:44:19.110+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:44:20.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:44:19 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:44:20.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:44:19 smithi047 ceph-mon[77147]: pgmap v848: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:44:20.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:44:19 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:44:20.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:44:19 smithi115 ceph-mon[78593]: pgmap v848: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:44:20.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:44:20 smithi115 conmon[90393]: 2022-05-13T23:44:20.133+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:44:21.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:44:20 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:44:21.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:44:20 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:44:21.612 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:44:21 smithi115 conmon[90393]: 2022-05-13T23:44:21.137+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:44:22.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:44:21 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:44:22.045 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:44:21 smithi047 ceph-mon[77147]: fsmap cephfs:5/5 3 up:standby-replay 1 up:replay 4 up:active 2022-05-13T23:44:22.046 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:44:21 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:44:22.046 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:44:21 smithi047 ceph-mon[77147]: pgmap v849: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:44:22.128 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:44:21 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:44:22.129 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:44:21 smithi115 ceph-mon[78593]: fsmap cephfs:5/5 3 up:standby-replay 1 up:replay 4 up:active 2022-05-13T23:44:22.129 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:44:21 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:44:22.129 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:44:21 smithi115 ceph-mon[78593]: pgmap v849: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:44:22.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:44:22 smithi115 conmon[90393]: 2022-05-13T23:44:22.139+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:44:23.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:44:22 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:44:23.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:44:22 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:44:23.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:44:23 smithi115 conmon[90393]: 2022-05-13T23:44:23.172+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:44:24.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:44:23 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:44:24.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:44:23 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 996 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:44:24.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:44:23 smithi047 ceph-mon[77147]: pgmap v850: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:44:24.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:44:23 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:44:24.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:44:23 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 996 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:44:24.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:44:23 smithi115 ceph-mon[78593]: pgmap v850: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:44:24.615 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:44:24 smithi115 conmon[90393]: 2022-05-13T23:44:24.155+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:44:25.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:44:24 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:44:25.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:44:24 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:44:25.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:44:25 smithi115 conmon[90393]: 2022-05-13T23:44:25.165+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:44:26.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:44:25 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:44:26.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:44:25 smithi047 ceph-mon[77147]: pgmap v851: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:44:26.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:44:25 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:44:26.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:44:25 smithi115 ceph-mon[78593]: pgmap v851: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:44:26.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:44:26 smithi115 conmon[90393]: 2022-05-13T23:44:26.182+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:44:27.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:44:27 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:44:27.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:44:27 smithi115 conmon[90393]: 2022-05-13T23:44:27.203+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:44:27.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:44:27 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:44:28.546 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:44:28 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:44:28.546 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:44:28 smithi047 ceph-mon[77147]: pgmap v852: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:44:28.546 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:44:28 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:44:28.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:44:28 smithi115 conmon[90393]: 2022-05-13T23:44:28.170+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:44:28.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:44:28 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:44:28.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:44:28 smithi115 ceph-mon[78593]: pgmap v852: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:44:28.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:44:28 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:44:29.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:44:29 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 1001 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:44:29.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:44:29 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:44:29.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:44:29 smithi115 conmon[90393]: 2022-05-13T23:44:29.130+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:44:29.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:44:29 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 1001 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:44:29.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:44:29 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:44:30.610 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:44:30 smithi115 conmon[90393]: 2022-05-13T23:44:30.147+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:44:30.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:44:30 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:44:30.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:44:30 smithi115 ceph-mon[78593]: fsmap cephfs:5/5 3 up:standby-replay 1 up:replay 4 up:active 2022-05-13T23:44:30.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:44:30 smithi115 ceph-mon[78593]: pgmap v853: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:44:30.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:44:30 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:44:30.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:44:30 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:44:30.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:44:30 smithi047 ceph-mon[77147]: fsmap cephfs:5/5 3 up:standby-replay 1 up:replay 4 up:active 2022-05-13T23:44:30.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:44:30 smithi047 ceph-mon[77147]: pgmap v853: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:44:30.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:44:30 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:44:31.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:44:31 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:44:31.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:44:31 smithi115 conmon[90393]: 2022-05-13T23:44:31.156+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:44:31.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:44:31 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:44:32.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:44:32 smithi115 conmon[90393]: 2022-05-13T23:44:32.153+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:44:32.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:44:32 smithi115 ceph-mon[78593]: pgmap v854: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:44:32.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:44:32 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:44:32.795 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:44:32 smithi047 ceph-mon[77147]: pgmap v854: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:44:32.795 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:44:32 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:44:33.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:44:33 smithi115 conmon[90393]: 2022-05-13T23:44:33.181+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:44:33.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:44:33 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:44:33.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:44:33 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 1006 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:44:33.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:44:33 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:44:33.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:44:33 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 1006 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:44:34.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:44:34 smithi115 conmon[90393]: 2022-05-13T23:44:34.172+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:44:34.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:44:34 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:44:34.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:44:34 smithi115 ceph-mon[78593]: fsmap cephfs:5/5 3 up:standby-replay 1 up:replay 4 up:active 2022-05-13T23:44:34.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:44:34 smithi115 ceph-mon[78593]: pgmap v855: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:44:34.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:44:34 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:44:34.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:44:34 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:44:34.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:44:34 smithi047 ceph-mon[77147]: fsmap cephfs:5/5 3 up:standby-replay 1 up:replay 4 up:active 2022-05-13T23:44:34.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:44:34 smithi047 ceph-mon[77147]: pgmap v855: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:44:34.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:44:34 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:44:35.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:44:35 smithi115 conmon[90393]: 2022-05-13T23:44:35.219+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:44:36.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:44:35 smithi047 ceph-mon[77147]: pgmap v856: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:44:36.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:44:35 smithi115 ceph-mon[78593]: pgmap v856: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:44:36.550 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:44:36 smithi115 conmon[90393]: 2022-05-13T23:44:36.229+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:44:36.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:44:36 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:44:36.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:44:36 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-13T23:44:36.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:44:36 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-13T23:44:36.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:44:36 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-13T23:44:37.026 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:44:36 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:44:37.027 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:44:36 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-13T23:44:37.027 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:44:36 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-13T23:44:37.027 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:44:36 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-13T23:44:37.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:44:37 smithi115 conmon[90393]: 2022-05-13T23:44:37.232+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:44:37.938 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:44:37 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:44:37.939 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:44:37 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:44:37.939 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:44:37 smithi047 ceph-mon[77147]: fsmap cephfs:5/5 3 up:standby-replay 1 up:replay 4 up:active 2022-05-13T23:44:37.939 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:44:37 smithi047 ceph-mon[77147]: pgmap v857: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:44:38.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:44:37 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:44:38.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:44:37 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:44:38.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:44:37 smithi115 ceph-mon[78593]: fsmap cephfs:5/5 3 up:standby-replay 1 up:replay 4 up:active 2022-05-13T23:44:38.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:44:37 smithi115 ceph-mon[78593]: pgmap v857: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:44:38.291 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:44:37 smithi047 conmon[78521]: 2022-05-13T23:44:37.935+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:44:38.291 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:44:37 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-13T23:44:38.292 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:44:37 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-13T23:44:38.292 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:44:37 smithi047 conmon[78521]: yield 2022-05-13T23:44:38.292 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:44:37 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-13T23:44:38.292 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:44:37 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-13T23:44:38.293 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:44:37 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-13T23:44:38.293 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:44:37 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-13T23:44:38.293 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:44:37 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-13T23:44:38.293 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:44:37 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-13T23:44:38.293 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:44:37 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-13T23:44:38.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:44:37 smithi047 conmon[78521]: raise exceptions[0] 2022-05-13T23:44:38.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:44:37 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-13T23:44:38.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:44:37 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-13T23:44:38.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:44:37 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-13T23:44:38.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:44:37 smithi047 conmon[78521]: return (yield from fut) 2022-05-13T23:44:38.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:44:37 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-13T23:44:38.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:44:37 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-13T23:44:38.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:44:37 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:44:38.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:44:38 smithi115 conmon[90393]: 2022-05-13T23:44:38.243+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:44:39.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:44:38 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:44:39.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:44:38 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:44:39.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-13T23:44:39.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-13T23:44:39.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-13T23:44:39.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-13T23:44:39.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-13T23:44:39.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-13T23:44:39.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-13T23:44:39.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-13T23:44:39.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-13T23:44:39.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-13T23:44:39.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-13T23:44:39.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-13T23:44:39.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-13T23:44:39.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-13T23:44:39.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-13T23:44:39.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-13T23:44:39.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-13T23:44:39.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:44:39.049 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:44:38 smithi047 ceph-mon[77147]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-13T23:44:39.049 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:44:38 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 1011 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:44:39.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:44:38 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:44:39.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:44:38 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:44:39.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-13T23:44:39.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-13T23:44:39.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-13T23:44:39.113 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-13T23:44:39.113 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-13T23:44:39.113 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-13T23:44:39.113 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-13T23:44:39.113 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-13T23:44:39.114 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-13T23:44:39.114 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-13T23:44:39.114 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-13T23:44:39.114 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-13T23:44:39.115 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-13T23:44:39.115 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-13T23:44:39.115 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-13T23:44:39.115 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-13T23:44:39.116 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-13T23:44:39.116 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:44:39.119 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:44:38 smithi115 ceph-mon[78593]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-13T23:44:39.122 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:44:38 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 1011 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:44:39.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:44:39 smithi115 conmon[90393]: 2022-05-13T23:44:39.272+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:44:40.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:44:39 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:44:40.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:44:39 smithi047 ceph-mon[77147]: pgmap v858: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:44:40.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:44:39 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:44:40.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:44:39 smithi115 ceph-mon[78593]: pgmap v858: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:44:40.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:44:40 smithi115 conmon[90393]: 2022-05-13T23:44:40.248+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:44:41.012 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:44:40 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:44:41.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:44:40 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:44:41.292 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:44:41 smithi047 conmon[78521]: 2022-05-13T23:44:41.007+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:44:41.292 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:44:41 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-13T23:44:41.293 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:44:41 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-13T23:44:41.293 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:44:41 smithi047 conmon[78521]: yield 2022-05-13T23:44:41.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:44:41 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-13T23:44:41.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:44:41 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-13T23:44:41.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:44:41 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-13T23:44:41.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:44:41 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-13T23:44:41.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:44:41 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-13T23:44:41.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:44:41 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-13T23:44:41.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:44:41 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-13T23:44:41.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:44:41 smithi047 conmon[78521]: raise exceptions[0] 2022-05-13T23:44:41.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:44:41 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-13T23:44:41.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:44:41 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-13T23:44:41.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:44:41 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-13T23:44:41.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:44:41 smithi047 conmon[78521]: return (yield from fut) 2022-05-13T23:44:41.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:44:41 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-13T23:44:41.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:44:41 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-13T23:44:41.297 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:44:41 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:44:41.297 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:44:41 smithi047 conmon[78521]: 2022-05-13T23:44:41.008+0000 7f5340f7a700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-13T23:44:41.297 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:44:41 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-13T23:44:41.297 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:44:41 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-13T23:44:41.298 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:44:41 smithi047 conmon[78521]: yield 2022-05-13T23:44:41.298 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:44:41 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-13T23:44:41.298 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:44:41 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-13T23:44:41.298 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:44:41 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-13T23:44:41.298 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:44:41 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-13T23:44:41.299 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:44:41 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-13T23:44:41.299 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:44:41 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-13T23:44:41.299 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:44:41 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-13T23:44:41.299 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:44:41 smithi047 conmon[78521]: raise exceptions[0] 2022-05-13T23:44:41.299 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:44:41 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-13T23:44:41.300 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:44:41 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-13T23:44:41.300 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:44:41 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-13T23:44:41.300 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:44:41 smithi047 conmon[78521]: return (yield from fut) 2022-05-13T23:44:41.300 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:44:41 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-13T23:44:41.301 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:44:41 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-13T23:44:41.301 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:44:41 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:44:41.301 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:44:41 smithi047 conmon[78521]: 2022-05-13T23:44:41.301 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:44:41 smithi047 conmon[78521]: During handling of the above exception, another exception occurred: 2022-05-13T23:44:41.301 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:44:41 smithi047 conmon[78521]: 2022-05-13T23:44:41.302 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:44:41 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-13T23:44:41.302 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:44:41 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-13T23:44:41.302 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:44:41 smithi047 conmon[78521]: return f(*arg) 2022-05-13T23:44:41.302 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:44:41 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-13T23:44:41.303 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:44:41 smithi047 conmon[78521]: self._write_client_files(client_files, host) 2022-05-13T23:44:41.303 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:44:41 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-13T23:44:41.304 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:44:41 smithi047 conmon[78521]: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-13T23:44:41.304 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:44:41 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-13T23:44:41.304 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:44:41 smithi047 conmon[78521]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-13T23:44:41.305 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:44:41 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-13T23:44:41.305 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:44:41 smithi047 conmon[78521]: return self.event_loop.get_result(coro) 2022-05-13T23:44:41.305 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:44:41 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-13T23:44:41.305 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:44:41 smithi047 conmon[78521]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-13T23:44:41.306 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:44:41 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-13T23:44:41.306 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:44:41 smithi047 conmon[78521]: return self.__get_result() 2022-05-13T23:44:41.306 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:44:41 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-13T23:44:41.306 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:44:41 smithi047 conmon[78521]: raise self._exception 2022-05-13T23:44:41.306 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:44:41 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-13T23:44:41.307 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:44:41 smithi047 conmon[78521]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-13T23:44:41.307 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:44:41 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-13T23:44:41.307 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:44:41 smithi047 conmon[78521]: conn = await self._remote_connection(host, addr) 2022-05-13T23:44:41.307 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:44:41 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-13T23:44:41.308 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:44:41 smithi047 conmon[78521]: raise 2022-05-13T23:44:41.308 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:44:41 smithi047 conmon[78521]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-13T23:44:41.308 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:44:41 smithi047 conmon[78521]: self.gen.throw(type, value, traceback) 2022-05-13T23:44:41.308 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:44:41 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-13T23:44:41.308 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:44:41 smithi047 conmon[78521]: raise OrchestratorError(msg) 2022-05-13T23:44:41.309 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:44:41 smithi047 conmon[78521]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:44:41.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:44:41 smithi115 conmon[90393]: 2022-05-13T23:44:41.254+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:44:42.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:44:41 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:44:42.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:44:41 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:44:42.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:44:41 smithi047 ceph-mon[77147]: fsmap cephfs:5/5 3 up:standby-replay 1 up:replay 4 up:active 2022-05-13T23:44:42.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:44:41 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:44:42.042 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-13T23:44:42.042 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-13T23:44:42.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-13T23:44:42.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-13T23:44:42.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-13T23:44:42.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-13T23:44:42.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-13T23:44:42.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-13T23:44:42.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-13T23:44:42.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-13T23:44:42.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-13T23:44:42.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-13T23:44:42.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-13T23:44:42.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-13T23:44:42.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-13T23:44:42.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-13T23:44:42.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-13T23:44:42.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:44:42.047 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:44:41 smithi047 ceph-mon[77147]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-13T23:44:42.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-13T23:44:42.049 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-13T23:44:42.050 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-13T23:44:42.050 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-13T23:44:42.050 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-13T23:44:42.050 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-13T23:44:42.051 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-13T23:44:42.051 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-13T23:44:42.051 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-13T23:44:42.051 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-13T23:44:42.052 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-13T23:44:42.052 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-13T23:44:42.052 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-13T23:44:42.053 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-13T23:44:42.053 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-13T23:44:42.053 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-13T23:44:42.053 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-13T23:44:42.053 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:44:42.054 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-13T23:44:42.054 INFO:journalctl@ceph.mon.b.smithi047.stdout: During handling of the above exception, another exception occurred: 2022-05-13T23:44:42.055 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-13T23:44:42.055 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-13T23:44:42.055 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-13T23:44:42.055 INFO:journalctl@ceph.mon.b.smithi047.stdout: return f(*arg) 2022-05-13T23:44:42.056 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-13T23:44:42.056 INFO:journalctl@ceph.mon.b.smithi047.stdout: self._write_client_files(client_files, host) 2022-05-13T23:44:42.056 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-13T23:44:42.056 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-13T23:44:42.059 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-13T23:44:42.059 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-13T23:44:42.059 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-13T23:44:42.059 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.event_loop.get_result(coro) 2022-05-13T23:44:42.060 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-13T23:44:42.060 INFO:journalctl@ceph.mon.b.smithi047.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-13T23:44:42.060 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-13T23:44:42.061 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.__get_result() 2022-05-13T23:44:42.061 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-13T23:44:42.061 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise self._exception 2022-05-13T23:44:42.061 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-13T23:44:42.061 INFO:journalctl@ceph.mon.b.smithi047.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-13T23:44:42.062 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-13T23:44:42.062 INFO:journalctl@ceph.mon.b.smithi047.stdout: conn = await self._remote_connection(host, addr) 2022-05-13T23:44:42.062 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-13T23:44:42.062 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise 2022-05-13T23:44:42.063 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-13T23:44:42.063 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.gen.throw(type, value, traceback) 2022-05-13T23:44:42.063 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-13T23:44:42.063 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OrchestratorError(msg) 2022-05-13T23:44:42.063 INFO:journalctl@ceph.mon.b.smithi047.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:44:42.064 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:44:41 smithi047 ceph-mon[77147]: pgmap v859: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:44:42.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:44:41 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:44:42.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:44:41 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:44:42.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:44:41 smithi115 ceph-mon[78593]: fsmap cephfs:5/5 3 up:standby-replay 1 up:replay 4 up:active 2022-05-13T23:44:42.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:44:41 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:44:42.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-13T23:44:42.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-13T23:44:42.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-13T23:44:42.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-13T23:44:42.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-13T23:44:42.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-13T23:44:42.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-13T23:44:42.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-13T23:44:42.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-13T23:44:42.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-13T23:44:42.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-13T23:44:42.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-13T23:44:42.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-13T23:44:42.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-13T23:44:42.113 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-13T23:44:42.113 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-13T23:44:42.113 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-13T23:44:42.114 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:44:42.114 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:44:41 smithi115 ceph-mon[78593]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-13T23:44:42.114 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-13T23:44:42.114 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-13T23:44:42.114 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-13T23:44:42.115 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-13T23:44:42.115 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-13T23:44:42.115 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-13T23:44:42.115 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-13T23:44:42.116 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-13T23:44:42.122 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-13T23:44:42.122 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-13T23:44:42.122 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-13T23:44:42.122 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-13T23:44:42.123 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-13T23:44:42.123 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-13T23:44:42.123 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-13T23:44:42.123 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-13T23:44:42.124 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-13T23:44:42.124 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:44:42.124 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-13T23:44:42.124 INFO:journalctl@ceph.mon.c.smithi115.stdout: During handling of the above exception, another exception occurred: 2022-05-13T23:44:42.124 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-13T23:44:42.125 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-13T23:44:42.125 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-13T23:44:42.126 INFO:journalctl@ceph.mon.c.smithi115.stdout: return f(*arg) 2022-05-13T23:44:42.126 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-13T23:44:42.126 INFO:journalctl@ceph.mon.c.smithi115.stdout: self._write_client_files(client_files, host) 2022-05-13T23:44:42.126 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-13T23:44:42.127 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-13T23:44:42.127 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-13T23:44:42.127 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-13T23:44:42.127 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-13T23:44:42.127 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.event_loop.get_result(coro) 2022-05-13T23:44:42.128 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-13T23:44:42.128 INFO:journalctl@ceph.mon.c.smithi115.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-13T23:44:42.128 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-13T23:44:42.128 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.__get_result() 2022-05-13T23:44:42.129 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-13T23:44:42.129 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise self._exception 2022-05-13T23:44:42.129 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-13T23:44:42.129 INFO:journalctl@ceph.mon.c.smithi115.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-13T23:44:42.130 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-13T23:44:42.130 INFO:journalctl@ceph.mon.c.smithi115.stdout: conn = await self._remote_connection(host, addr) 2022-05-13T23:44:42.130 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-13T23:44:42.130 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise 2022-05-13T23:44:42.131 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-13T23:44:42.131 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.gen.throw(type, value, traceback) 2022-05-13T23:44:42.131 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-13T23:44:42.131 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OrchestratorError(msg) 2022-05-13T23:44:42.132 INFO:journalctl@ceph.mon.c.smithi115.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:44:42.132 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:44:41 smithi115 ceph-mon[78593]: pgmap v859: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:44:42.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:44:42 smithi115 conmon[90393]: 2022-05-13T23:44:42.273+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:44:43.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:44:42 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:44:43.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:44:42 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:44:43.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:44:43 smithi115 conmon[90393]: 2022-05-13T23:44:43.237+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:44:44.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:44:43 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:44:44.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:44:43 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 1015 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:44:44.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:44:43 smithi047 ceph-mon[77147]: pgmap v860: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:44:44.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:44:43 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:44:44.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:44:43 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 1015 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:44:44.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:44:43 smithi115 ceph-mon[78593]: pgmap v860: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:44:44.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:44:44 smithi115 conmon[90393]: 2022-05-13T23:44:44.232+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:44:45.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:44:44 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:44:45.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:44:44 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:44:45.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:44:45 smithi115 conmon[90393]: 2022-05-13T23:44:45.258+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:44:46.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:44:45 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:44:46.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:44:45 smithi047 ceph-mon[77147]: pgmap v861: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:44:46.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:44:45 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:44:46.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:44:45 smithi115 ceph-mon[78593]: pgmap v861: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:44:46.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:44:46 smithi115 conmon[90393]: 2022-05-13T23:44:46.264+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:44:47.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:44:46 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:44:47.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:44:46 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:44:47.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:44:47 smithi115 conmon[90393]: 2022-05-13T23:44:47.288+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:44:48.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:44:47 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:44:48.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:44:47 smithi047 ceph-mon[77147]: pgmap v862: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:44:48.115 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:44:47 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:44:48.115 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:44:47 smithi115 ceph-mon[78593]: pgmap v862: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:44:48.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:44:48 smithi115 conmon[90393]: 2022-05-13T23:44:48.309+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:44:49.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:44:48 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:44:49.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:44:48 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 1020 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:44:49.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:44:48 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:44:49.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:44:48 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 1020 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:44:49.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:44:49 smithi115 conmon[90393]: 2022-05-13T23:44:49.318+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:44:50.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:44:49 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:44:50.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:44:49 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:44:50.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:44:49 smithi047 ceph-mon[77147]: fsmap cephfs:5/5 3 up:standby-replay 1 up:replay 4 up:active 2022-05-13T23:44:50.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:44:49 smithi047 ceph-mon[77147]: pgmap v863: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:44:50.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:44:49 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:44:50.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:44:49 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:44:50.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:44:49 smithi115 ceph-mon[78593]: fsmap cephfs:5/5 3 up:standby-replay 1 up:replay 4 up:active 2022-05-13T23:44:50.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:44:49 smithi115 ceph-mon[78593]: pgmap v863: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:44:50.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:44:50 smithi115 conmon[90393]: 2022-05-13T23:44:50.319+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:44:51.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:44:50 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:44:51.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:44:50 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:44:51.610 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:44:51 smithi115 conmon[90393]: 2022-05-13T23:44:51.330+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:44:52.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:44:51 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:44:52.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:44:51 smithi047 ceph-mon[77147]: pgmap v864: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:44:52.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:44:51 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:44:52.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:44:51 smithi115 ceph-mon[78593]: pgmap v864: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:44:52.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:44:52 smithi115 conmon[90393]: 2022-05-13T23:44:52.328+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:44:53.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:44:52 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:44:53.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:44:52 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:44:53.610 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:44:53 smithi115 conmon[90393]: 2022-05-13T23:44:53.340+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:44:54.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:44:53 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:44:54.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:44:53 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:44:54.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:44:53 smithi047 ceph-mon[77147]: fsmap cephfs:5/5 3 up:standby-replay 1 up:replay 4 up:active 2022-05-13T23:44:54.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:44:53 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 1025 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:44:54.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:44:53 smithi047 ceph-mon[77147]: pgmap v865: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:44:54.114 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:44:53 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:44:54.115 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:44:53 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:44:54.116 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:44:53 smithi115 ceph-mon[78593]: fsmap cephfs:5/5 3 up:standby-replay 1 up:replay 4 up:active 2022-05-13T23:44:54.116 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:44:53 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 1025 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:44:54.116 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:44:53 smithi115 ceph-mon[78593]: pgmap v865: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:44:54.680 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:44:54 smithi115 conmon[90393]: 2022-05-13T23:44:54.364+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:44:55.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:44:54 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:44:55.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:44:54 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:44:55.688 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:44:55 smithi115 conmon[90393]: 2022-05-13T23:44:55.384+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:44:56.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:44:55 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:44:56.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:44:55 smithi047 ceph-mon[77147]: pgmap v866: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:44:56.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:44:55 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:44:56.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:44:55 smithi115 ceph-mon[78593]: pgmap v866: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:44:56.690 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:44:56 smithi115 conmon[90393]: 2022-05-13T23:44:56.383+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:44:57.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:44:56 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:44:57.112 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:44:56 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:44:57.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:44:57 smithi115 conmon[90393]: 2022-05-13T23:44:57.343+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:44:58.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:44:57 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:44:58.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:44:57 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:44:58.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:44:57 smithi047 ceph-mon[77147]: fsmap cephfs:5/5 3 up:standby-replay 1 up:replay 4 up:active 2022-05-13T23:44:58.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:44:57 smithi047 ceph-mon[77147]: pgmap v867: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:44:58.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:44:57 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:44:58.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:44:57 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:44:58.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:44:57 smithi115 ceph-mon[78593]: fsmap cephfs:5/5 3 up:standby-replay 1 up:replay 4 up:active 2022-05-13T23:44:58.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:44:57 smithi115 ceph-mon[78593]: pgmap v867: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:44:58.702 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:44:58 smithi115 conmon[90393]: 2022-05-13T23:44:58.393+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:44:59.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:44:58 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:44:59.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:44:58 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 1030 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:44:59.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:44:58 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:44:59.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:44:58 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 1030 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:44:59.708 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:44:59 smithi115 conmon[90393]: 2022-05-13T23:44:59.382+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:45:00.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:44:59 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:45:00.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:44:59 smithi047 ceph-mon[77147]: pgmap v868: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:45:00.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:44:59 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:45:00.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:44:59 smithi115 ceph-mon[78593]: pgmap v868: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:45:00.713 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:45:00 smithi115 conmon[90393]: 2022-05-13T23:45:00.365+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:45:01.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:45:00 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:45:01.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:45:00 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:45:01.716 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:45:01 smithi115 conmon[90393]: 2022-05-13T23:45:01.364+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:45:02.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:45:01 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:45:02.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:45:01 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:45:02.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:45:01 smithi047 ceph-mon[77147]: fsmap cephfs:5/5 3 up:standby-replay 1 up:replay 4 up:active 2022-05-13T23:45:02.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:45:01 smithi047 ceph-mon[77147]: pgmap v869: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:45:02.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:45:01 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:45:02.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:45:01 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:45:02.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:45:01 smithi115 ceph-mon[78593]: fsmap cephfs:5/5 3 up:standby-replay 1 up:replay 4 up:active 2022-05-13T23:45:02.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:45:01 smithi115 ceph-mon[78593]: pgmap v869: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:45:02.721 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:45:02 smithi115 conmon[90393]: 2022-05-13T23:45:02.383+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:45:03.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:45:02 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:45:03.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:45:02 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:45:03.722 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:45:03 smithi115 conmon[90393]: 2022-05-13T23:45:03.376+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:45:04.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:45:03 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:45:04.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:45:03 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 1035 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:45:04.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:45:03 smithi047 ceph-mon[77147]: pgmap v870: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:45:04.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:45:03 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:45:04.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:45:03 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 1035 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:45:04.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:45:03 smithi115 ceph-mon[78593]: pgmap v870: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:45:04.726 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:45:04 smithi115 conmon[90393]: 2022-05-13T23:45:04.361+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:45:05.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:45:04 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:45:05.112 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:45:04 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:45:05.730 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:45:05 smithi115 conmon[90393]: 2022-05-13T23:45:05.366+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:45:06.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:45:05 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:45:06.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:45:05 smithi047 ceph-mon[77147]: pgmap v871: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:45:06.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:45:05 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:45:06.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:45:05 smithi115 ceph-mon[78593]: pgmap v871: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:45:06.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:45:06 smithi115 conmon[90393]: 2022-05-13T23:45:06.351+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:45:07.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:45:06 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:45:07.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:45:06 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:45:07.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:45:07 smithi115 conmon[90393]: 2022-05-13T23:45:07.308+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:45:08.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:45:07 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:45:08.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:45:07 smithi047 ceph-mon[77147]: pgmap v872: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:45:08.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:45:07 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:45:08.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:45:07 smithi115 ceph-mon[78593]: pgmap v872: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:45:08.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:45:08 smithi115 conmon[90393]: 2022-05-13T23:45:08.288+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:45:09.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:45:08 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:45:09.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:45:08 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 1040 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:45:09.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:45:08 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:45:09.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:45:08 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 1040 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:45:09.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:45:09 smithi115 conmon[90393]: 2022-05-13T23:45:09.314+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:45:10.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:45:09 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:45:10.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:45:09 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:45:10.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:45:09 smithi047 ceph-mon[77147]: fsmap cephfs:5/5 3 up:standby-replay 1 up:replay 4 up:active 2022-05-13T23:45:10.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:45:09 smithi047 ceph-mon[77147]: pgmap v873: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:45:10.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:45:09 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:45:10.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:45:09 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:45:10.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:45:09 smithi115 ceph-mon[78593]: fsmap cephfs:5/5 3 up:standby-replay 1 up:replay 4 up:active 2022-05-13T23:45:10.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:45:09 smithi115 ceph-mon[78593]: pgmap v873: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:45:10.610 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:45:10 smithi115 conmon[90393]: 2022-05-13T23:45:10.284+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:45:11.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:45:10 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:45:11.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:45:10 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:45:11.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:45:11 smithi115 conmon[90393]: 2022-05-13T23:45:11.244+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:45:12.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:45:11 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:45:12.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:45:11 smithi047 ceph-mon[77147]: pgmap v874: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:45:12.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:45:11 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:45:12.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:45:11 smithi115 ceph-mon[78593]: pgmap v874: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:45:12.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:45:12 smithi115 conmon[90393]: 2022-05-13T23:45:12.268+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:45:13.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:45:12 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:45:13.117 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:45:12 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:45:13.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:45:13 smithi115 conmon[90393]: 2022-05-13T23:45:13.318+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:45:14.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:45:13 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:45:14.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:45:13 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:45:14.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:45:13 smithi047 ceph-mon[77147]: fsmap cephfs:5/5 3 up:standby-replay 1 up:replay 4 up:active 2022-05-13T23:45:14.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:45:13 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-13T23:45:14.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:45:13 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-13T23:45:14.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:45:13 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 1045 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:45:14.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:45:13 smithi047 ceph-mon[77147]: pgmap v875: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:45:14.118 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:45:13 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:45:14.118 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:45:13 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:45:14.119 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:45:13 smithi115 ceph-mon[78593]: fsmap cephfs:5/5 3 up:standby-replay 1 up:replay 4 up:active 2022-05-13T23:45:14.119 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:45:13 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-13T23:45:14.119 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:45:13 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-13T23:45:14.123 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:45:13 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 1045 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:45:14.123 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:45:13 smithi115 ceph-mon[78593]: pgmap v875: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:45:14.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:45:14 smithi115 conmon[90393]: 2022-05-13T23:45:14.288+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:45:15.005 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:45:14 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:45:15.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:45:14 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:45:15.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:45:15 smithi115 conmon[90393]: 2022-05-13T23:45:15.299+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:45:16.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:45:15 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:45:16.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:45:15 smithi047 ceph-mon[77147]: pgmap v876: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:45:16.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:45:15 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:45:16.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:45:15 smithi115 ceph-mon[78593]: pgmap v876: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:45:16.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:45:16 smithi115 conmon[90393]: 2022-05-13T23:45:16.269+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:45:17.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:45:16 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:45:17.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:45:16 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:45:17.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:45:17 smithi115 conmon[90393]: 2022-05-13T23:45:17.261+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:45:18.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:45:17 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:45:18.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:45:17 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:45:18.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:45:17 smithi047 ceph-mon[77147]: fsmap cephfs:5/5 3 up:standby-replay 1 up:replay 4 up:active 2022-05-13T23:45:18.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:45:17 smithi047 ceph-mon[77147]: pgmap v877: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:45:18.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:45:17 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:45:18.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:45:17 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:45:18.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:45:17 smithi115 ceph-mon[78593]: fsmap cephfs:5/5 3 up:standby-replay 1 up:replay 4 up:active 2022-05-13T23:45:18.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:45:17 smithi115 ceph-mon[78593]: pgmap v877: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:45:18.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:45:18 smithi115 conmon[90393]: 2022-05-13T23:45:18.276+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:45:19.045 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:45:18 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:45:19.045 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:45:18 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 1050 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:45:19.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:45:18 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:45:19.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:45:18 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 1050 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:45:19.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:45:19 smithi115 conmon[90393]: 2022-05-13T23:45:19.283+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:45:20.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:45:19 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:45:20.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:45:19 smithi047 ceph-mon[77147]: pgmap v878: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:45:20.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:45:19 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:45:20.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:45:19 smithi115 ceph-mon[78593]: pgmap v878: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:45:20.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:45:20 smithi115 conmon[90393]: 2022-05-13T23:45:20.328+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:45:21.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:45:20 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:45:21.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:45:20 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:45:21.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:45:21 smithi115 conmon[90393]: 2022-05-13T23:45:21.337+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:45:22.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:45:21 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:45:22.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:45:21 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:45:22.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:45:21 smithi047 ceph-mon[77147]: fsmap cephfs:5/5 3 up:standby-replay 1 up:replay 4 up:active 2022-05-13T23:45:22.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:45:21 smithi047 ceph-mon[77147]: pgmap v879: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:45:22.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:45:21 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:45:22.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:45:21 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:45:22.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:45:21 smithi115 ceph-mon[78593]: fsmap cephfs:5/5 3 up:standby-replay 1 up:replay 4 up:active 2022-05-13T23:45:22.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:45:21 smithi115 ceph-mon[78593]: pgmap v879: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:45:22.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:45:22 smithi115 conmon[90393]: 2022-05-13T23:45:22.306+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:45:23.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:45:22 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:45:23.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:45:22 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:45:23.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:45:23 smithi115 conmon[90393]: 2022-05-13T23:45:23.300+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:45:24.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:45:23 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:45:24.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:45:23 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 1055 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:45:24.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:45:23 smithi047 ceph-mon[77147]: pgmap v880: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:45:24.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:45:23 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:45:24.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:45:23 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 1055 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:45:24.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:45:23 smithi115 ceph-mon[78593]: pgmap v880: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:45:24.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:45:24 smithi115 conmon[90393]: 2022-05-13T23:45:24.328+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:45:25.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:45:24 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:45:25.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:45:24 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:45:25.798 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:45:25 smithi115 conmon[90393]: 2022-05-13T23:45:25.362+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:45:26.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:45:25 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:45:26.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:45:25 smithi115 ceph-mon[78593]: pgmap v881: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:45:26.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:45:25 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:45:26.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:45:25 smithi047 ceph-mon[77147]: pgmap v881: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:45:26.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:45:26 smithi115 conmon[90393]: 2022-05-13T23:45:26.323+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:45:27.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:45:26 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:45:27.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:45:26 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:45:27.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:45:27 smithi115 conmon[90393]: 2022-05-13T23:45:27.317+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:45:28.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:45:27 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:45:28.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:45:27 smithi115 ceph-mon[78593]: pgmap v882: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:45:28.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:45:27 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:45:28.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:45:27 smithi047 ceph-mon[77147]: pgmap v882: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:45:28.610 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:45:28 smithi115 conmon[90393]: 2022-05-13T23:45:28.318+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:45:29.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:45:28 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:45:29.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:45:28 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 1060 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:45:29.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:45:28 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:45:29.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:45:28 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 1060 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:45:29.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:45:29 smithi115 conmon[90393]: 2022-05-13T23:45:29.333+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:45:30.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:45:29 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:45:30.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:45:29 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:45:30.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:45:29 smithi115 ceph-mon[78593]: fsmap cephfs:5/5 3 up:standby-replay 1 up:replay 4 up:active 2022-05-13T23:45:30.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:45:29 smithi115 ceph-mon[78593]: pgmap v883: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:45:30.295 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:45:29 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:45:30.295 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:45:29 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:45:30.295 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:45:29 smithi047 ceph-mon[77147]: fsmap cephfs:5/5 3 up:standby-replay 1 up:replay 4 up:active 2022-05-13T23:45:30.295 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:45:29 smithi047 ceph-mon[77147]: pgmap v883: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:45:30.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:45:30 smithi115 conmon[90393]: 2022-05-13T23:45:30.353+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:45:31.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:45:30 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:45:31.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:45:30 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:45:31.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:45:31 smithi115 conmon[90393]: 2022-05-13T23:45:31.334+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:45:32.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:45:31 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:45:32.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:45:31 smithi115 ceph-mon[78593]: pgmap v884: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:45:32.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:45:31 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:45:32.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:45:31 smithi047 ceph-mon[77147]: pgmap v884: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:45:32.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:45:32 smithi115 conmon[90393]: 2022-05-13T23:45:32.317+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:45:33.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:45:32 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:45:33.296 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:45:32 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:45:33.611 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:45:33 smithi115 conmon[90393]: 2022-05-13T23:45:33.328+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:45:34.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:45:33 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:45:34.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:45:33 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:45:34.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:45:33 smithi115 ceph-mon[78593]: fsmap cephfs:5/5 3 up:standby-replay 1 up:replay 4 up:active 2022-05-13T23:45:34.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:45:33 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 1065 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:45:34.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:45:33 smithi115 ceph-mon[78593]: pgmap v885: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:45:34.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:45:33 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:45:34.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:45:33 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:45:34.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:45:33 smithi047 ceph-mon[77147]: fsmap cephfs:5/5 3 up:standby-replay 1 up:replay 4 up:active 2022-05-13T23:45:34.297 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:45:33 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 1065 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:45:34.297 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:45:33 smithi047 ceph-mon[77147]: pgmap v885: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:45:34.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:45:34 smithi115 conmon[90393]: 2022-05-13T23:45:34.314+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:45:35.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:45:34 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:45:35.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:45:34 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:45:35.610 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:45:35 smithi115 conmon[90393]: 2022-05-13T23:45:35.349+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:45:36.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:45:35 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:45:36.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:45:35 smithi115 ceph-mon[78593]: pgmap v886: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:45:36.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:45:35 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:45:36.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:45:35 smithi047 ceph-mon[77147]: pgmap v886: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:45:36.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:45:36 smithi115 conmon[90393]: 2022-05-13T23:45:36.345+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:45:37.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:45:36 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:45:37.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:45:36 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:45:37.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:45:37 smithi115 conmon[90393]: 2022-05-13T23:45:37.313+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:45:38.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:45:37 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:45:38.112 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:45:37 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:45:38.112 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:45:37 smithi115 ceph-mon[78593]: fsmap cephfs:5/5 3 up:standby-replay 1 up:replay 4 up:active 2022-05-13T23:45:38.114 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:45:37 smithi115 ceph-mon[78593]: pgmap v887: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-05-13T23:45:38.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:45:37 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:45:38.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:45:37 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:45:38.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:45:37 smithi047 ceph-mon[77147]: fsmap cephfs:5/5 3 up:standby-replay 1 up:replay 4 up:active 2022-05-13T23:45:38.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:45:37 smithi047 ceph-mon[77147]: pgmap v887: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-05-13T23:45:38.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:45:38 smithi115 conmon[90393]: 2022-05-13T23:45:38.280+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:45:39.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:45:38 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:45:39.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:45:38 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 1070 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:45:39.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:45:38 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:45:39.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:45:38 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 1070 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:45:39.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:45:39 smithi115 conmon[90393]: 2022-05-13T23:45:39.280+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:45:40.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:45:39 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:45:40.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:45:39 smithi115 ceph-mon[78593]: pgmap v888: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:45:40.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:45:39 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:45:40.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:45:39 smithi047 ceph-mon[77147]: pgmap v888: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:45:40.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:45:40 smithi115 conmon[90393]: 2022-05-13T23:45:40.296+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:45:41.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:45:40 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:45:41.113 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:45:40 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:45:41.520 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:45:41 smithi115 conmon[90393]: 2022-05-13T23:45:41.299+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:45:42.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:45:41 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:45:42.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:45:41 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:45:42.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:45:41 smithi115 ceph-mon[78593]: fsmap cephfs:5/5 3 up:standby-replay 1 up:replay 4 up:active 2022-05-13T23:45:42.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:45:41 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-13T23:45:42.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:45:41 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-13T23:45:42.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:45:41 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-13T23:45:42.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:45:41 smithi115 ceph-mon[78593]: pgmap v889: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-05-13T23:45:42.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:45:41 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:45:42.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:45:41 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:45:42.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:45:41 smithi047 ceph-mon[77147]: fsmap cephfs:5/5 3 up:standby-replay 1 up:replay 4 up:active 2022-05-13T23:45:42.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:45:41 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-13T23:45:42.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:45:41 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-13T23:45:42.297 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:45:41 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-13T23:45:42.298 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:45:41 smithi047 ceph-mon[77147]: pgmap v889: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-05-13T23:45:42.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:45:42 smithi115 conmon[90393]: 2022-05-13T23:45:42.340+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:45:42.791 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:45:42 smithi047 conmon[78521]: 2022-05-13T23:45:42.448+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:45:42.792 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:45:42 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-13T23:45:42.792 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:45:42 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-13T23:45:42.793 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:45:42 smithi047 conmon[78521]: yield 2022-05-13T23:45:42.793 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:45:42 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-13T23:45:42.793 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:45:42 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-13T23:45:42.793 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:45:42 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-13T23:45:42.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:45:42 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-13T23:45:42.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:45:42 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-13T23:45:42.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:45:42 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-13T23:45:42.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:45:42 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-13T23:45:42.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:45:42 smithi047 conmon[78521]: raise exceptions[0] 2022-05-13T23:45:42.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:45:42 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-13T23:45:42.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:45:42 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-13T23:45:42.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:45:42 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-13T23:45:42.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:45:42 smithi047 conmon[78521]: return (yield from fut) 2022-05-13T23:45:42.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:45:42 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-13T23:45:42.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:45:42 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-13T23:45:42.797 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:45:42 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:45:43.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:45:42 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:45:43.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:45:42 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:45:43.109 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-13T23:45:43.109 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-13T23:45:43.109 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-13T23:45:43.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-13T23:45:43.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-13T23:45:43.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-13T23:45:43.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-13T23:45:43.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-13T23:45:43.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-13T23:45:43.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-13T23:45:43.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-13T23:45:43.113 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-13T23:45:43.113 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-13T23:45:43.113 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-13T23:45:43.114 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-13T23:45:43.114 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-13T23:45:43.114 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-13T23:45:43.114 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:45:43.115 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:45:42 smithi115 ceph-mon[78593]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-13T23:45:43.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:45:42 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:45:43.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:45:42 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:45:43.292 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-13T23:45:43.292 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-13T23:45:43.292 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-13T23:45:43.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-13T23:45:43.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-13T23:45:43.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-13T23:45:43.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-13T23:45:43.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-13T23:45:43.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-13T23:45:43.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-13T23:45:43.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-13T23:45:43.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-13T23:45:43.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-13T23:45:43.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-13T23:45:43.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-13T23:45:43.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-13T23:45:43.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-13T23:45:43.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:45:43.296 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:45:42 smithi047 ceph-mon[77147]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-13T23:45:43.611 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:45:43 smithi115 conmon[90393]: 2022-05-13T23:45:43.321+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:45:44.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:45:43 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:45:44.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:45:43 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 1075 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:45:44.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:45:43 smithi115 ceph-mon[78593]: pgmap v890: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-05-13T23:45:44.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:45:43 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:45:44.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:45:43 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 1075 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:45:44.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:45:43 smithi047 ceph-mon[77147]: pgmap v890: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-05-13T23:45:44.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:45:44 smithi115 conmon[90393]: 2022-05-13T23:45:44.313+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:45:45.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:45:44 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:45:45.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:45:44 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:45:45.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:45:45 smithi115 conmon[90393]: 2022-05-13T23:45:45.343+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:45:45.793 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:45:45 smithi047 conmon[78521]: 2022-05-13T23:45:45.520+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:45:45.793 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:45:45 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-13T23:45:45.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:45:45 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-13T23:45:45.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:45:45 smithi047 conmon[78521]: yield 2022-05-13T23:45:45.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:45:45 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-13T23:45:45.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:45:45 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-13T23:45:45.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:45:45 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-13T23:45:45.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:45:45 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-13T23:45:45.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:45:45 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-13T23:45:45.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:45:45 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-13T23:45:45.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:45:45 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-13T23:45:45.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:45:45 smithi047 conmon[78521]: raise exceptions[0] 2022-05-13T23:45:45.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:45:45 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-13T23:45:45.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:45:45 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-13T23:45:45.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:45:45 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-13T23:45:45.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:45:45 smithi047 conmon[78521]: return (yield from fut) 2022-05-13T23:45:45.797 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:45:45 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-13T23:45:45.797 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:45:45 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-13T23:45:45.797 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:45:45 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:45:45.797 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:45:45 smithi047 conmon[78521]: 2022-05-13T23:45:45.521+0000 7f5340779700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-13T23:45:45.798 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:45:45 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-13T23:45:45.798 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:45:45 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-13T23:45:45.798 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:45:45 smithi047 conmon[78521]: yield 2022-05-13T23:45:45.798 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:45:45 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-13T23:45:45.798 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:45:45 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-13T23:45:45.799 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:45:45 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-13T23:45:45.799 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:45:45 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-13T23:45:45.799 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:45:45 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-13T23:45:45.799 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:45:45 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-13T23:45:45.800 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:45:45 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-13T23:45:45.800 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:45:45 smithi047 conmon[78521]: raise exceptions[0] 2022-05-13T23:45:45.800 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:45:45 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-13T23:45:45.800 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:45:45 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-13T23:45:45.800 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:45:45 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-13T23:45:45.801 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:45:45 smithi047 conmon[78521]: return (yield from fut) 2022-05-13T23:45:45.801 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:45:45 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-13T23:45:45.801 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:45:45 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-13T23:45:45.801 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:45:45 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:45:45.801 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:45:45 smithi047 conmon[78521]: 2022-05-13T23:45:45.802 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:45:45 smithi047 conmon[78521]: During handling of the above exception, another exception occurred: 2022-05-13T23:45:45.803 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:45:45 smithi047 conmon[78521]: 2022-05-13T23:45:45.803 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:45:45 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-13T23:45:45.804 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:45:45 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-13T23:45:45.804 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:45:45 smithi047 conmon[78521]: return f(*arg) 2022-05-13T23:45:45.804 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:45:45 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-13T23:45:45.804 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:45:45 smithi047 conmon[78521]: self._write_client_files(client_files, host) 2022-05-13T23:45:45.805 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:45:45 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-13T23:45:45.805 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:45:45 smithi047 conmon[78521]: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-13T23:45:45.805 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:45:45 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-13T23:45:45.805 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:45:45 smithi047 conmon[78521]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-13T23:45:45.806 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:45:45 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-13T23:45:45.807 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:45:45 smithi047 conmon[78521]: return self.event_loop.get_result(coro) 2022-05-13T23:45:45.807 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:45:45 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-13T23:45:45.807 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:45:45 smithi047 conmon[78521]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-13T23:45:45.808 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:45:45 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-13T23:45:45.808 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:45:45 smithi047 conmon[78521]: return self.__get_result() 2022-05-13T23:45:45.808 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:45:45 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-13T23:45:45.808 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:45:45 smithi047 conmon[78521]: raise self._exception 2022-05-13T23:45:45.809 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:45:45 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-13T23:45:45.809 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:45:45 smithi047 conmon[78521]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-13T23:45:45.809 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:45:45 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-13T23:45:45.809 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:45:45 smithi047 conmon[78521]: conn = await self._remote_connection(host, addr) 2022-05-13T23:45:45.809 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:45:45 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-13T23:45:45.810 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:45:45 smithi047 conmon[78521]: raise 2022-05-13T23:45:45.810 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:45:45 smithi047 conmon[78521]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-13T23:45:45.810 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:45:45 smithi047 conmon[78521]: self.gen.throw(type, value, traceback) 2022-05-13T23:45:45.810 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:45:45 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-13T23:45:45.810 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:45:45 smithi047 conmon[78521]: raise OrchestratorError(msg) 2022-05-13T23:45:45.811 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:45:45 smithi047 conmon[78521]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:45:46.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:45:45 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:45:46.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:45:45 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:45:46.292 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-13T23:45:46.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-13T23:45:46.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-13T23:45:46.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-13T23:45:46.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-13T23:45:46.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-13T23:45:46.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-13T23:45:46.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-13T23:45:46.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-13T23:45:46.300 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-13T23:45:46.300 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-13T23:45:46.301 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-13T23:45:46.301 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-13T23:45:46.301 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-13T23:45:46.301 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-13T23:45:46.302 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-13T23:45:46.302 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-13T23:45:46.302 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:45:46.302 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:45:45 smithi047 ceph-mon[77147]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-13T23:45:46.302 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-13T23:45:46.303 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-13T23:45:46.303 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-13T23:45:46.303 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-13T23:45:46.303 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-13T23:45:46.304 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-13T23:45:46.304 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-13T23:45:46.304 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-13T23:45:46.304 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-13T23:45:46.305 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-13T23:45:46.305 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-13T23:45:46.306 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-13T23:45:46.306 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-13T23:45:46.306 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-13T23:45:46.307 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-13T23:45:46.307 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-13T23:45:46.307 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-13T23:45:46.307 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:45:46.308 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-13T23:45:46.308 INFO:journalctl@ceph.mon.b.smithi047.stdout: During handling of the above exception, another exception occurred: 2022-05-13T23:45:46.308 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-13T23:45:46.308 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-13T23:45:46.308 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-13T23:45:46.309 INFO:journalctl@ceph.mon.b.smithi047.stdout: return f(*arg) 2022-05-13T23:45:46.309 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-13T23:45:46.309 INFO:journalctl@ceph.mon.b.smithi047.stdout: self._write_client_files(client_files, host) 2022-05-13T23:45:46.309 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-13T23:45:46.310 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-13T23:45:46.310 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-13T23:45:46.310 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-13T23:45:46.311 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-13T23:45:46.311 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.event_loop.get_result(coro) 2022-05-13T23:45:46.311 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-13T23:45:46.311 INFO:journalctl@ceph.mon.b.smithi047.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-13T23:45:46.311 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-13T23:45:46.312 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.__get_result() 2022-05-13T23:45:46.313 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-13T23:45:46.314 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise self._exception 2022-05-13T23:45:46.314 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-13T23:45:46.315 INFO:journalctl@ceph.mon.b.smithi047.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-13T23:45:46.315 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-13T23:45:46.316 INFO:journalctl@ceph.mon.b.smithi047.stdout: conn = await self._remote_connection(host, addr) 2022-05-13T23:45:46.317 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-13T23:45:46.317 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise 2022-05-13T23:45:46.318 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-13T23:45:46.318 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.gen.throw(type, value, traceback) 2022-05-13T23:45:46.319 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-13T23:45:46.319 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OrchestratorError(msg) 2022-05-13T23:45:46.320 INFO:journalctl@ceph.mon.b.smithi047.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:45:46.320 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:45:45 smithi047 ceph-mon[77147]: pgmap v891: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:45:46.364 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:45:45 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:45:46.365 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:45:45 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:45:46.365 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-13T23:45:46.365 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-13T23:45:46.366 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-13T23:45:46.366 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-13T23:45:46.366 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-13T23:45:46.366 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-13T23:45:46.366 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-13T23:45:46.367 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-13T23:45:46.367 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-13T23:45:46.367 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-13T23:45:46.367 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-13T23:45:46.367 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-13T23:45:46.368 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-13T23:45:46.368 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-13T23:45:46.368 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-13T23:45:46.368 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-13T23:45:46.369 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-13T23:45:46.369 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:45:46.372 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:45:45 smithi115 ceph-mon[78593]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-13T23:45:46.373 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-13T23:45:46.374 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-13T23:45:46.375 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-13T23:45:46.375 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-13T23:45:46.376 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-13T23:45:46.376 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-13T23:45:46.376 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-13T23:45:46.377 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-13T23:45:46.377 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-13T23:45:46.378 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-13T23:45:46.400 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-13T23:45:46.401 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-13T23:45:46.401 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-13T23:45:46.401 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-13T23:45:46.402 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-13T23:45:46.402 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-13T23:45:46.402 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-13T23:45:46.403 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:45:46.403 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-13T23:45:46.403 INFO:journalctl@ceph.mon.c.smithi115.stdout: During handling of the above exception, another exception occurred: 2022-05-13T23:45:46.403 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-13T23:45:46.404 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-13T23:45:46.404 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-13T23:45:46.404 INFO:journalctl@ceph.mon.c.smithi115.stdout: return f(*arg) 2022-05-13T23:45:46.404 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-13T23:45:46.404 INFO:journalctl@ceph.mon.c.smithi115.stdout: self._write_client_files(client_files, host) 2022-05-13T23:45:46.405 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-13T23:45:46.405 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-13T23:45:46.405 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-13T23:45:46.406 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-13T23:45:46.406 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-13T23:45:46.409 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.event_loop.get_result(coro) 2022-05-13T23:45:46.409 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-13T23:45:46.409 INFO:journalctl@ceph.mon.c.smithi115.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-13T23:45:46.410 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-13T23:45:46.410 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.__get_result() 2022-05-13T23:45:46.410 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-13T23:45:46.410 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise self._exception 2022-05-13T23:45:46.411 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-13T23:45:46.411 INFO:journalctl@ceph.mon.c.smithi115.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-13T23:45:46.411 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-13T23:45:46.411 INFO:journalctl@ceph.mon.c.smithi115.stdout: conn = await self._remote_connection(host, addr) 2022-05-13T23:45:46.412 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-13T23:45:46.412 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise 2022-05-13T23:45:46.412 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-13T23:45:46.412 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.gen.throw(type, value, traceback) 2022-05-13T23:45:46.412 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-13T23:45:46.413 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OrchestratorError(msg) 2022-05-13T23:45:46.413 INFO:journalctl@ceph.mon.c.smithi115.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:45:46.413 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:45:45 smithi115 ceph-mon[78593]: pgmap v891: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:45:46.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:45:46 smithi115 conmon[90393]: 2022-05-13T23:45:46.375+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:45:47.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:45:46 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:45:47.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:45:46 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:45:47.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:45:47 smithi115 conmon[90393]: 2022-05-13T23:45:47.407+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:45:48.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:45:47 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:45:48.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:45:47 smithi047 ceph-mon[77147]: pgmap v892: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:45:48.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:45:47 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:45:48.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:45:47 smithi115 ceph-mon[78593]: pgmap v892: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:45:48.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:45:48 smithi115 conmon[90393]: 2022-05-13T23:45:48.401+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:45:49.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:45:48 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:45:49.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:45:48 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 1080 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:45:49.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:45:48 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:45:49.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:45:48 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 1080 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:45:49.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:45:49 smithi115 conmon[90393]: 2022-05-13T23:45:49.387+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:45:50.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:45:49 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:45:50.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:45:49 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:45:50.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:45:49 smithi047 ceph-mon[77147]: fsmap cephfs:5/5 3 up:standby-replay 1 up:replay 4 up:active 2022-05-13T23:45:50.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:45:49 smithi047 ceph-mon[77147]: pgmap v893: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:45:50.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:45:49 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:45:50.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:45:49 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:45:50.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:45:49 smithi115 ceph-mon[78593]: fsmap cephfs:5/5 3 up:standby-replay 1 up:replay 4 up:active 2022-05-13T23:45:50.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:45:49 smithi115 ceph-mon[78593]: pgmap v893: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:45:50.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:45:50 smithi115 conmon[90393]: 2022-05-13T23:45:50.382+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:45:51.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:45:50 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:45:51.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:45:50 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:45:51.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:45:51 smithi115 conmon[90393]: 2022-05-13T23:45:51.373+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:45:52.306 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:45:51 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:45:52.306 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:45:51 smithi047 ceph-mon[77147]: pgmap v894: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:45:52.328 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:45:51 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:45:52.328 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:45:51 smithi115 ceph-mon[78593]: pgmap v894: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:45:52.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:45:52 smithi115 conmon[90393]: 2022-05-13T23:45:52.326+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:45:53.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:45:52 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:45:53.323 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:45:52 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:45:53.626 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:45:53 smithi115 conmon[90393]: 2022-05-13T23:45:53.321+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:45:54.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:45:53 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:45:54.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:45:53 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:45:54.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:45:53 smithi047 ceph-mon[77147]: fsmap cephfs:5/5 3 up:standby-replay 1 up:replay 4 up:active 2022-05-13T23:45:54.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:45:53 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 1085 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:45:54.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:45:53 smithi047 ceph-mon[77147]: pgmap v895: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:45:54.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:45:53 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:45:54.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:45:53 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:45:54.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:45:53 smithi115 ceph-mon[78593]: fsmap cephfs:5/5 3 up:standby-replay 1 up:replay 4 up:active 2022-05-13T23:45:54.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:45:53 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 1085 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:45:54.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:45:53 smithi115 ceph-mon[78593]: pgmap v895: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:45:54.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:45:54 smithi115 conmon[90393]: 2022-05-13T23:45:54.360+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:45:55.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:45:54 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:45:55.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:45:54 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:45:55.869 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:45:55 smithi115 conmon[90393]: 2022-05-13T23:45:55.409+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:45:56.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:45:55 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:45:56.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:45:55 smithi047 ceph-mon[77147]: pgmap v896: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:45:56.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:45:55 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:45:56.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:45:55 smithi115 ceph-mon[78593]: pgmap v896: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:45:56.863 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:45:56 smithi115 conmon[90393]: 2022-05-13T23:45:56.361+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:45:57.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:45:56 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:45:57.334 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:45:56 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:45:57.617 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:45:57 smithi115 conmon[90393]: 2022-05-13T23:45:57.333+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:45:58.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:45:57 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:45:58.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:45:57 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:45:58.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:45:57 smithi047 ceph-mon[77147]: fsmap cephfs:5/5 3 up:standby-replay 1 up:replay 4 up:active 2022-05-13T23:45:58.295 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:45:57 smithi047 ceph-mon[77147]: pgmap v897: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:45:58.352 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:45:57 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:45:58.353 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:45:57 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:45:58.353 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:45:57 smithi115 ceph-mon[78593]: fsmap cephfs:5/5 3 up:standby-replay 1 up:replay 4 up:active 2022-05-13T23:45:58.353 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:45:57 smithi115 ceph-mon[78593]: pgmap v897: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:45:58.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:45:58 smithi115 conmon[90393]: 2022-05-13T23:45:58.351+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:45:59.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:45:58 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:45:59.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:45:58 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 1090 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:45:59.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:45:58 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:45:59.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:45:58 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 1090 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:45:59.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:45:59 smithi115 conmon[90393]: 2022-05-13T23:45:59.362+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:46:00.298 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:45:59 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:46:00.299 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:45:59 smithi047 ceph-mon[77147]: pgmap v898: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:46:00.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:45:59 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:46:00.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:45:59 smithi115 ceph-mon[78593]: pgmap v898: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:46:00.863 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:46:00 smithi115 conmon[90393]: 2022-05-13T23:46:00.358+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:46:01.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:46:00 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:46:01.322 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:46:00 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:46:01.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:46:01 smithi115 conmon[90393]: 2022-05-13T23:46:01.320+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:46:02.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:46:01 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:46:02.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:46:01 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:46:02.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:46:01 smithi047 ceph-mon[77147]: fsmap cephfs:5/5 3 up:standby-replay 1 up:replay 4 up:active 2022-05-13T23:46:02.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:46:01 smithi047 ceph-mon[77147]: pgmap v899: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:46:02.336 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:46:01 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:46:02.336 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:46:01 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:46:02.336 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:46:01 smithi115 ceph-mon[78593]: fsmap cephfs:5/5 3 up:standby-replay 1 up:replay 4 up:active 2022-05-13T23:46:02.337 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:46:01 smithi115 ceph-mon[78593]: pgmap v899: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:46:02.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:46:02 smithi115 conmon[90393]: 2022-05-13T23:46:02.334+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:46:03.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:46:02 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:46:03.300 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:46:02 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:46:03.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:46:03 smithi115 conmon[90393]: 2022-05-13T23:46:03.299+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:46:04.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:46:03 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:46:04.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:46:03 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 1095 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:46:04.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:46:03 smithi047 ceph-mon[77147]: pgmap v900: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:46:04.318 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:46:03 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:46:04.318 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:46:03 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 1095 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:46:04.318 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:46:03 smithi115 ceph-mon[78593]: pgmap v900: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:46:04.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:46:04 smithi115 conmon[90393]: 2022-05-13T23:46:04.315+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:46:05.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:46:04 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:46:05.324 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:46:04 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:46:05.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:46:05 smithi115 conmon[90393]: 2022-05-13T23:46:05.323+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:46:06.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:46:05 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:46:06.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:46:05 smithi047 ceph-mon[77147]: pgmap v901: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:46:06.320 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:46:05 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:46:06.321 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:46:05 smithi115 ceph-mon[78593]: pgmap v901: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:46:06.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:46:06 smithi115 conmon[90393]: 2022-05-13T23:46:06.317+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:46:07.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:46:06 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:46:07.321 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:46:06 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:46:07.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:46:07 smithi115 conmon[90393]: 2022-05-13T23:46:07.319+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:46:08.283 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:46:07 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:46:08.284 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:46:07 smithi115 ceph-mon[78593]: pgmap v902: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:46:08.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:46:07 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:46:08.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:46:07 smithi047 ceph-mon[77147]: pgmap v902: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:46:08.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:46:08 smithi115 conmon[90393]: 2022-05-13T23:46:08.282+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:46:09.282 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:46:08 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:46:09.282 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:46:08 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 1100 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:46:09.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:46:08 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:46:09.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:46:08 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 1100 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:46:09.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:46:09 smithi115 conmon[90393]: 2022-05-13T23:46:09.274+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:46:10.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:46:09 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:46:10.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:46:09 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:46:10.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:46:09 smithi047 ceph-mon[77147]: fsmap cephfs:5/5 3 up:standby-replay 1 up:replay 4 up:active 2022-05-13T23:46:10.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:46:09 smithi047 ceph-mon[77147]: pgmap v903: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:46:10.294 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:46:09 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:46:10.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:46:09 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:46:10.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:46:09 smithi115 ceph-mon[78593]: fsmap cephfs:5/5 3 up:standby-replay 1 up:replay 4 up:active 2022-05-13T23:46:10.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:46:09 smithi115 ceph-mon[78593]: pgmap v903: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:46:10.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:46:10 smithi115 conmon[90393]: 2022-05-13T23:46:10.290+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:46:11.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:46:10 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:46:11.336 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:46:10 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:46:11.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:46:11 smithi115 conmon[90393]: 2022-05-13T23:46:11.335+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:46:12.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:46:11 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:46:12.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:46:11 smithi047 ceph-mon[77147]: pgmap v904: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:46:12.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:46:11 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:46:12.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:46:11 smithi115 ceph-mon[78593]: pgmap v904: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:46:12.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:46:12 smithi115 conmon[90393]: 2022-05-13T23:46:12.372+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:46:13.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:46:12 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:46:13.342 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:46:12 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:46:13.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:46:13 smithi115 conmon[90393]: 2022-05-13T23:46:13.341+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:46:14.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:46:13 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:46:14.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:46:13 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:46:14.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:46:13 smithi047 ceph-mon[77147]: fsmap cephfs:5/5 3 up:standby-replay 1 up:replay 4 up:active 2022-05-13T23:46:14.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:46:13 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-13T23:46:14.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:46:13 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-13T23:46:14.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:46:13 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 1105 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:46:14.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:46:13 smithi047 ceph-mon[77147]: pgmap v905: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:46:14.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:46:13 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:46:14.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:46:13 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:46:14.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:46:13 smithi115 ceph-mon[78593]: fsmap cephfs:5/5 3 up:standby-replay 1 up:replay 4 up:active 2022-05-13T23:46:14.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:46:13 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-13T23:46:14.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:46:13 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-13T23:46:14.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:46:13 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 1105 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:46:14.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:46:13 smithi115 ceph-mon[78593]: pgmap v905: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:46:14.861 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:46:14 smithi115 conmon[90393]: 2022-05-13T23:46:14.367+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:46:15.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:46:14 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:46:15.338 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:46:14 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:46:15.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:46:15 smithi115 conmon[90393]: 2022-05-13T23:46:15.337+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:46:16.296 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:46:15 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:46:16.296 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:46:15 smithi047 ceph-mon[77147]: pgmap v906: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:46:16.335 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:46:15 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:46:16.336 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:46:15 smithi115 ceph-mon[78593]: pgmap v906: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:46:16.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:46:16 smithi115 conmon[90393]: 2022-05-13T23:46:16.332+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:46:17.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:46:16 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:46:17.343 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:46:16 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:46:17.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:46:17 smithi115 conmon[90393]: 2022-05-13T23:46:17.333+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:46:18.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:46:17 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:46:18.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:46:17 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:46:18.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:46:17 smithi047 ceph-mon[77147]: fsmap cephfs:5/5 3 up:standby-replay 1 up:replay 4 up:active 2022-05-13T23:46:18.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:46:17 smithi047 ceph-mon[77147]: pgmap v907: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:46:18.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:46:17 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:46:18.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:46:17 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:46:18.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:46:17 smithi115 ceph-mon[78593]: fsmap cephfs:5/5 3 up:standby-replay 1 up:replay 4 up:active 2022-05-13T23:46:18.362 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:46:17 smithi115 ceph-mon[78593]: pgmap v907: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:46:18.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:46:18 smithi115 conmon[90393]: 2022-05-13T23:46:18.361+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:46:19.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:46:18 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:46:19.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:46:18 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 1110 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:46:19.314 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:46:18 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:46:19.315 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:46:18 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 1110 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:46:19.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:46:19 smithi115 conmon[90393]: 2022-05-13T23:46:19.313+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:46:20.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:46:19 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:46:20.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:46:19 smithi047 ceph-mon[77147]: pgmap v908: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:46:20.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:46:19 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:46:20.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:46:19 smithi115 ceph-mon[78593]: pgmap v908: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:46:20.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:46:20 smithi115 conmon[90393]: 2022-05-13T23:46:20.293+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:46:21.259 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:46:20 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:46:21.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:46:20 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:46:21.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:46:21 smithi115 conmon[90393]: 2022-05-13T23:46:21.257+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:46:22.241 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:46:21 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:46:22.241 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:46:21 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:46:22.242 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:46:21 smithi115 ceph-mon[78593]: fsmap cephfs:5/5 3 up:standby-replay 1 up:replay 4 up:active 2022-05-13T23:46:22.242 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:46:21 smithi115 ceph-mon[78593]: pgmap v909: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:46:22.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:46:21 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:46:22.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:46:21 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:46:22.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:46:21 smithi047 ceph-mon[77147]: fsmap cephfs:5/5 3 up:standby-replay 1 up:replay 4 up:active 2022-05-13T23:46:22.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:46:21 smithi047 ceph-mon[77147]: pgmap v909: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:46:22.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:46:22 smithi115 conmon[90393]: 2022-05-13T23:46:22.239+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:46:23.266 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:46:22 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:46:23.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:46:22 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:46:23.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:46:23 smithi115 conmon[90393]: 2022-05-13T23:46:23.265+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:46:24.276 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:46:23 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:46:24.276 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:46:23 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 1115 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:46:24.276 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:46:23 smithi115 ceph-mon[78593]: pgmap v910: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:46:24.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:46:23 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:46:24.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:46:23 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 1115 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:46:24.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:46:23 smithi047 ceph-mon[77147]: pgmap v910: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:46:24.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:46:24 smithi115 conmon[90393]: 2022-05-13T23:46:24.272+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:46:25.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:46:24 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:46:25.311 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:46:24 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:46:25.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:46:25 smithi115 conmon[90393]: 2022-05-13T23:46:25.310+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:46:26.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:46:25 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:46:26.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:46:25 smithi047 ceph-mon[77147]: pgmap v911: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:46:26.317 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:46:25 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:46:26.317 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:46:25 smithi115 ceph-mon[78593]: pgmap v911: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:46:26.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:46:26 smithi115 conmon[90393]: 2022-05-13T23:46:26.314+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:46:27.288 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:46:26 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:46:27.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:46:26 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:46:27.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:46:27 smithi115 conmon[90393]: 2022-05-13T23:46:27.287+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:46:28.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:46:28 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:46:28.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:46:28 smithi047 ceph-mon[77147]: pgmap v912: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:46:28.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:46:28 smithi115 conmon[90393]: 2022-05-13T23:46:28.252+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:46:28.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:46:28 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:46:28.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:46:28 smithi115 ceph-mon[78593]: pgmap v912: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:46:29.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:46:29 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:46:29.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:46:29 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 1120 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:46:29.294 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:46:29 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:46:29.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:46:29 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 1120 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:46:29.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:46:29 smithi115 conmon[90393]: 2022-05-13T23:46:29.293+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:46:30.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:46:30 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:46:30.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:46:30 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:46:30.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:46:30 smithi047 ceph-mon[77147]: fsmap cephfs:5/5 3 up:standby-replay 1 up:replay 4 up:active 2022-05-13T23:46:30.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:46:30 smithi047 ceph-mon[77147]: pgmap v913: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:46:30.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:46:30 smithi115 conmon[90393]: 2022-05-13T23:46:30.250+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:46:30.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:46:30 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:46:30.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:46:30 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:46:30.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:46:30 smithi115 ceph-mon[78593]: fsmap cephfs:5/5 3 up:standby-replay 1 up:replay 4 up:active 2022-05-13T23:46:30.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:46:30 smithi115 ceph-mon[78593]: pgmap v913: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:46:31.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:46:31 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:46:31.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:46:31 smithi115 conmon[90393]: 2022-05-13T23:46:31.225+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:46:31.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:46:31 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:46:32.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:46:32 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:46:32.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:46:32 smithi047 ceph-mon[77147]: pgmap v914: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:46:32.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:46:32 smithi115 conmon[90393]: 2022-05-13T23:46:32.207+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:46:32.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:46:32 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:46:32.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:46:32 smithi115 ceph-mon[78593]: pgmap v914: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:46:33.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:46:33 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:46:33.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:46:33 smithi115 conmon[90393]: 2022-05-13T23:46:33.204+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:46:33.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:46:33 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:46:34.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:46:34 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:46:34.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:46:34 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:46:34.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:46:34 smithi047 ceph-mon[77147]: fsmap cephfs:5/5 3 up:standby-replay 1 up:replay 4 up:active 2022-05-13T23:46:34.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:46:34 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:46:34.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:46:34 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 1125 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:46:34.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:46:34 smithi047 ceph-mon[77147]: pgmap v915: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:46:34.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:46:34 smithi115 conmon[90393]: 2022-05-13T23:46:34.205+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:46:34.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:46:34 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:46:34.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:46:34 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:46:34.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:46:34 smithi115 ceph-mon[78593]: fsmap cephfs:5/5 3 up:standby-replay 1 up:replay 4 up:active 2022-05-13T23:46:34.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:46:34 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:46:34.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:46:34 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 1125 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:46:34.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:46:34 smithi115 ceph-mon[78593]: pgmap v915: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:46:35.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:46:35 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:46:35.362 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:46:35 smithi115 conmon[90393]: 2022-05-13T23:46:35.185+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:46:35.363 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:46:35 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:46:36.303 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:46:36 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:46:36.304 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:46:36 smithi047 ceph-mon[77147]: pgmap v916: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:46:36.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:46:36 smithi115 conmon[90393]: 2022-05-13T23:46:36.216+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:46:36.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:46:36 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:46:36.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:46:36 smithi115 ceph-mon[78593]: pgmap v916: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:46:37.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:46:37 smithi115 conmon[90393]: 2022-05-13T23:46:37.209+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:46:38.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:46:38 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:46:38.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:46:38 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:46:38.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:46:38 smithi047 ceph-mon[77147]: fsmap cephfs:5/5 3 up:standby-replay 1 up:replay 4 up:active 2022-05-13T23:46:38.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:46:38 smithi047 ceph-mon[77147]: pgmap v917: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:46:38.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:46:38 smithi115 conmon[90393]: 2022-05-13T23:46:38.217+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:46:38.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:46:38 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:46:38.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:46:38 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:46:38.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:46:38 smithi115 ceph-mon[78593]: fsmap cephfs:5/5 3 up:standby-replay 1 up:replay 4 up:active 2022-05-13T23:46:38.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:46:38 smithi115 ceph-mon[78593]: pgmap v917: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:46:39.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:46:39 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:46:39.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:46:39 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 1130 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:46:39.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:46:39 smithi115 conmon[90393]: 2022-05-13T23:46:39.266+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:46:39.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:46:39 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:46:39.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:46:39 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 1130 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:46:40.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:46:40 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:46:40.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:46:40 smithi047 ceph-mon[77147]: pgmap v918: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:46:40.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:46:40 smithi115 conmon[90393]: 2022-05-13T23:46:40.260+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:46:40.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:46:40 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:46:40.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:46:40 smithi115 ceph-mon[78593]: pgmap v918: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:46:41.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:46:41 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:46:41.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:46:41 smithi115 conmon[90393]: 2022-05-13T23:46:41.230+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:46:41.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:46:41 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:46:42.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:46:42 smithi115 conmon[90393]: 2022-05-13T23:46:42.214+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:46:42.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:46:42 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:46:42.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:46:42 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:46:42.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:46:42 smithi115 ceph-mon[78593]: fsmap cephfs:5/5 3 up:standby-replay 1 up:replay 4 up:active 2022-05-13T23:46:42.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:46:42 smithi115 ceph-mon[78593]: pgmap v919: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:46:42.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:46:42 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:46:42.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:46:42 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:46:42.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:46:42 smithi047 ceph-mon[77147]: fsmap cephfs:5/5 3 up:standby-replay 1 up:replay 4 up:active 2022-05-13T23:46:42.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:46:42 smithi047 ceph-mon[77147]: pgmap v919: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:46:43.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:46:43 smithi115 conmon[90393]: 2022-05-13T23:46:43.251+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:46:43.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:46:43 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:46:43.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:46:43 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:46:44.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:46:44 smithi115 conmon[90393]: 2022-05-13T23:46:44.222+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:46:44.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:46:44 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:46:44.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:46:44 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 1135 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:46:44.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:46:44 smithi115 ceph-mon[78593]: pgmap v920: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:46:44.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:46:44 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:46:44.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:46:44 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 1135 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:46:44.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:46:44 smithi047 ceph-mon[77147]: pgmap v920: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:46:45.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:46:45 smithi115 conmon[90393]: 2022-05-13T23:46:45.235+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:46:45.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:46:45 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:46:45.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:46:45 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:46:46.363 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:46:46 smithi115 conmon[90393]: 2022-05-13T23:46:46.233+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:46:46.364 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:46:46 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:46:46.364 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:46:46 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-13T23:46:46.365 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:46:46 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-13T23:46:46.366 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:46:46 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-13T23:46:46.366 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:46:46 smithi115 ceph-mon[78593]: pgmap v921: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:46:46.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:46:46 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:46:46.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:46:46 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-13T23:46:46.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:46:46 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-13T23:46:46.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:46:46 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-13T23:46:46.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:46:46 smithi047 ceph-mon[77147]: pgmap v921: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:46:47.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:46:47 smithi115 conmon[90393]: 2022-05-13T23:46:47.234+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:46:47.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:46:47 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:46:47.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:46:47 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:46:48.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:46:48 smithi115 conmon[90393]: 2022-05-13T23:46:48.283+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:46:48.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:46:48 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:46:48.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:46:48 smithi115 ceph-mon[78593]: pgmap v922: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:46:48.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:46:48 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:46:48.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:46:48 smithi047 ceph-mon[77147]: pgmap v922: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:46:49.042 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:46:48 smithi047 conmon[78521]: 2022-05-13T23:46:48.623+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:46:49.042 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:46:48 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-13T23:46:49.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:46:48 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-13T23:46:49.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:46:48 smithi047 conmon[78521]: yield 2022-05-13T23:46:49.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:46:48 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-13T23:46:49.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:46:48 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-13T23:46:49.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:46:48 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-13T23:46:49.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:46:48 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-13T23:46:49.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:46:48 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-13T23:46:49.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:46:48 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-13T23:46:49.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:46:48 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-13T23:46:49.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:46:48 smithi047 conmon[78521]: raise exceptions[0] 2022-05-13T23:46:49.049 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:46:48 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-13T23:46:49.050 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:46:48 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-13T23:46:49.050 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:46:48 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-13T23:46:49.051 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:46:48 smithi047 conmon[78521]: return (yield from fut) 2022-05-13T23:46:49.051 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:46:48 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-13T23:46:49.051 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:46:48 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-13T23:46:49.052 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:46:48 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:46:49.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:46:49 smithi115 conmon[90393]: 2022-05-13T23:46:49.277+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:46:49.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:46:49 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:46:49.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:46:49 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 1140 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:46:49.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:46:49 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:46:49.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:46:49 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 1140 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:46:50.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:46:50 smithi115 conmon[90393]: 2022-05-13T23:46:50.268+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:46:50.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:46:50 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:46:50.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:46:50 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:46:50.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-13T23:46:50.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-13T23:46:50.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-13T23:46:50.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-13T23:46:50.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-13T23:46:50.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-13T23:46:50.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-13T23:46:50.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-13T23:46:50.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-13T23:46:50.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-13T23:46:50.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-13T23:46:50.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-13T23:46:50.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-13T23:46:50.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-13T23:46:50.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-13T23:46:50.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-13T23:46:50.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-13T23:46:50.365 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:46:50.365 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:46:50 smithi115 ceph-mon[78593]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-13T23:46:50.366 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:46:50 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:46:50.366 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:46:50 smithi115 ceph-mon[78593]: fsmap cephfs:5/5 3 up:standby-replay 1 up:replay 4 up:active 2022-05-13T23:46:50.366 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:46:50 smithi115 ceph-mon[78593]: pgmap v923: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:46:50.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:46:50 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:46:50.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:46:50 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:46:50.542 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-13T23:46:50.542 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-13T23:46:50.542 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-13T23:46:50.542 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-13T23:46:50.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-13T23:46:50.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-13T23:46:50.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-13T23:46:50.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-13T23:46:50.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-13T23:46:50.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-13T23:46:50.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-13T23:46:50.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-13T23:46:50.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-13T23:46:50.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-13T23:46:50.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-13T23:46:50.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-13T23:46:50.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-13T23:46:50.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:46:50.546 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:46:50 smithi047 ceph-mon[77147]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-13T23:46:50.546 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:46:50 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:46:50.547 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:46:50 smithi047 ceph-mon[77147]: fsmap cephfs:5/5 3 up:standby-replay 1 up:replay 4 up:active 2022-05-13T23:46:50.547 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:46:50 smithi047 ceph-mon[77147]: pgmap v923: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:46:51.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:46:51 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:46:51.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:46:51 smithi115 conmon[90393]: 2022-05-13T23:46:51.220+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:46:51.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:46:51 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:46:52.042 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:46:51 smithi047 conmon[78521]: 2022-05-13T23:46:51.695+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:46:52.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:46:51 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-13T23:46:52.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:46:51 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-13T23:46:52.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:46:51 smithi047 conmon[78521]: yield 2022-05-13T23:46:52.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:46:51 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-13T23:46:52.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:46:51 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-13T23:46:52.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:46:51 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-13T23:46:52.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:46:51 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-13T23:46:52.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:46:51 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-13T23:46:52.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:46:51 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-13T23:46:52.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:46:51 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-13T23:46:52.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:46:51 smithi047 conmon[78521]: raise exceptions[0] 2022-05-13T23:46:52.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:46:51 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-13T23:46:52.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:46:51 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-13T23:46:52.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:46:51 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-13T23:46:52.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:46:51 smithi047 conmon[78521]: return (yield from fut) 2022-05-13T23:46:52.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:46:51 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-13T23:46:52.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:46:51 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-13T23:46:52.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:46:51 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:46:52.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:46:51 smithi047 conmon[78521]: 2022-05-13T23:46:51.696+0000 7f533ef76700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-13T23:46:52.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:46:51 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-13T23:46:52.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:46:51 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-13T23:46:52.048 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:46:51 smithi047 conmon[78521]: yield 2022-05-13T23:46:52.048 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:46:51 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-13T23:46:52.048 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:46:51 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-13T23:46:52.049 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:46:51 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-13T23:46:52.050 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:46:51 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-13T23:46:52.050 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:46:51 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-13T23:46:52.051 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:46:51 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-13T23:46:52.051 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:46:51 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-13T23:46:52.051 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:46:51 smithi047 conmon[78521]: raise exceptions[0] 2022-05-13T23:46:52.051 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:46:51 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-13T23:46:52.051 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:46:51 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-13T23:46:52.052 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:46:51 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-13T23:46:52.052 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:46:51 smithi047 conmon[78521]: return (yield from fut) 2022-05-13T23:46:52.052 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:46:51 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-13T23:46:52.052 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:46:51 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-13T23:46:52.053 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:46:51 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:46:52.053 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:46:51 smithi047 conmon[78521]: 2022-05-13T23:46:52.053 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:46:51 smithi047 conmon[78521]: During handling of the above exception, another exception occurred: 2022-05-13T23:46:52.053 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:46:51 smithi047 conmon[78521]: 2022-05-13T23:46:52.054 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:46:51 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-13T23:46:52.054 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:46:51 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-13T23:46:52.054 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:46:51 smithi047 conmon[78521]: return f(*arg) 2022-05-13T23:46:52.054 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:46:51 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-13T23:46:52.054 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:46:51 smithi047 conmon[78521]: self._write_client_files(client_files, host) 2022-05-13T23:46:52.055 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:46:51 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-13T23:46:52.055 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:46:51 smithi047 conmon[78521]: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-13T23:46:52.055 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:46:51 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-13T23:46:52.055 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:46:51 smithi047 conmon[78521]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-13T23:46:52.056 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:46:51 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-13T23:46:52.056 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:46:51 smithi047 conmon[78521]: return self.event_loop.get_result(coro) 2022-05-13T23:46:52.056 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:46:51 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-13T23:46:52.056 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:46:51 smithi047 conmon[78521]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-13T23:46:52.056 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:46:51 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-13T23:46:52.057 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:46:51 smithi047 conmon[78521]: return self.__get_result() 2022-05-13T23:46:52.060 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:46:51 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-13T23:46:52.060 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:46:51 smithi047 conmon[78521]: raise self._exception 2022-05-13T23:46:52.061 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:46:51 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-13T23:46:52.061 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:46:51 smithi047 conmon[78521]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-13T23:46:52.061 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:46:51 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-13T23:46:52.061 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:46:51 smithi047 conmon[78521]: conn = await self._remote_connection(host, addr) 2022-05-13T23:46:52.062 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:46:51 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-13T23:46:52.062 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:46:51 smithi047 conmon[78521]: raise 2022-05-13T23:46:52.062 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:46:51 smithi047 conmon[78521]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-13T23:46:52.062 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:46:51 smithi047 conmon[78521]: self.gen.throw(type, value, traceback) 2022-05-13T23:46:52.062 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:46:51 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-13T23:46:52.063 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:46:51 smithi047 conmon[78521]: raise OrchestratorError(msg) 2022-05-13T23:46:52.063 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:46:51 smithi047 conmon[78521]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:46:52.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:46:52 smithi115 conmon[90393]: 2022-05-13T23:46:52.261+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:46:52.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:46:52 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:46:52.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:46:52 smithi115 ceph-mon[78593]: pgmap v924: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:46:52.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:46:52 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:46:52.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:46:52 smithi047 ceph-mon[77147]: pgmap v924: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:46:53.362 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:46:53 smithi115 conmon[90393]: 2022-05-13T23:46:53.302+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:46:53.368 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:46:53 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:46:53.369 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:46:53 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:46:53.369 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-13T23:46:53.370 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-13T23:46:53.370 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-13T23:46:53.371 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-13T23:46:53.372 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-13T23:46:53.373 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-13T23:46:53.374 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-13T23:46:53.374 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-13T23:46:53.374 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-13T23:46:53.374 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-13T23:46:53.375 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-13T23:46:53.375 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-13T23:46:53.375 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-13T23:46:53.375 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-13T23:46:53.375 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-13T23:46:53.376 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-13T23:46:53.376 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-13T23:46:53.377 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:46:53.377 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:46:53 smithi115 ceph-mon[78593]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-13T23:46:53.377 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-13T23:46:53.378 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-13T23:46:53.378 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-13T23:46:53.378 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-13T23:46:53.378 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-13T23:46:53.378 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-13T23:46:53.379 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-13T23:46:53.379 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-13T23:46:53.379 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-13T23:46:53.379 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-13T23:46:53.380 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-13T23:46:53.382 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-13T23:46:53.382 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-13T23:46:53.382 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-13T23:46:53.383 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-13T23:46:53.383 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-13T23:46:53.383 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-13T23:46:53.383 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:46:53.384 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-13T23:46:53.384 INFO:journalctl@ceph.mon.c.smithi115.stdout: During handling of the above exception, another exception occurred: 2022-05-13T23:46:53.384 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-13T23:46:53.384 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-13T23:46:53.384 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-13T23:46:53.385 INFO:journalctl@ceph.mon.c.smithi115.stdout: return f(*arg) 2022-05-13T23:46:53.385 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-13T23:46:53.386 INFO:journalctl@ceph.mon.c.smithi115.stdout: self._write_client_files(client_files, host) 2022-05-13T23:46:53.386 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-13T23:46:53.386 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-13T23:46:53.386 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-13T23:46:53.387 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-13T23:46:53.387 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-13T23:46:53.388 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.event_loop.get_result(coro) 2022-05-13T23:46:53.388 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-13T23:46:53.388 INFO:journalctl@ceph.mon.c.smithi115.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-13T23:46:53.388 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-13T23:46:53.389 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.__get_result() 2022-05-13T23:46:53.389 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-13T23:46:53.389 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise self._exception 2022-05-13T23:46:53.389 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-13T23:46:53.389 INFO:journalctl@ceph.mon.c.smithi115.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-13T23:46:53.390 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-13T23:46:53.390 INFO:journalctl@ceph.mon.c.smithi115.stdout: conn = await self._remote_connection(host, addr) 2022-05-13T23:46:53.391 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-13T23:46:53.391 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise 2022-05-13T23:46:53.391 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-13T23:46:53.391 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.gen.throw(type, value, traceback) 2022-05-13T23:46:53.392 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-13T23:46:53.392 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OrchestratorError(msg) 2022-05-13T23:46:53.393 INFO:journalctl@ceph.mon.c.smithi115.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:46:53.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:46:53 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:46:53.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:46:53 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:46:53.542 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-13T23:46:53.542 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-13T23:46:53.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-13T23:46:53.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-13T23:46:53.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-13T23:46:53.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-13T23:46:53.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-13T23:46:53.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-13T23:46:53.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-13T23:46:53.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-13T23:46:53.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-13T23:46:53.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-13T23:46:53.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-13T23:46:53.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-13T23:46:53.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-13T23:46:53.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-13T23:46:53.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-13T23:46:53.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:46:53.547 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:46:53 smithi047 ceph-mon[77147]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-13T23:46:53.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-13T23:46:53.548 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-13T23:46:53.548 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-13T23:46:53.549 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-13T23:46:53.549 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-13T23:46:53.549 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-13T23:46:53.550 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-13T23:46:53.550 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-13T23:46:53.551 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-13T23:46:53.551 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-13T23:46:53.551 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-13T23:46:53.551 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-13T23:46:53.552 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-13T23:46:53.552 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-13T23:46:53.552 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-13T23:46:53.553 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-13T23:46:53.553 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-13T23:46:53.554 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:46:53.554 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-13T23:46:53.554 INFO:journalctl@ceph.mon.b.smithi047.stdout: During handling of the above exception, another exception occurred: 2022-05-13T23:46:53.554 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-13T23:46:53.554 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-13T23:46:53.555 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-13T23:46:53.555 INFO:journalctl@ceph.mon.b.smithi047.stdout: return f(*arg) 2022-05-13T23:46:53.556 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-13T23:46:53.556 INFO:journalctl@ceph.mon.b.smithi047.stdout: self._write_client_files(client_files, host) 2022-05-13T23:46:53.556 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-13T23:46:53.556 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-13T23:46:53.557 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-13T23:46:53.557 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-13T23:46:53.557 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-13T23:46:53.557 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.event_loop.get_result(coro) 2022-05-13T23:46:53.558 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-13T23:46:53.558 INFO:journalctl@ceph.mon.b.smithi047.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-13T23:46:53.558 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-13T23:46:53.559 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.__get_result() 2022-05-13T23:46:53.559 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-13T23:46:53.559 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise self._exception 2022-05-13T23:46:53.560 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-13T23:46:53.560 INFO:journalctl@ceph.mon.b.smithi047.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-13T23:46:53.560 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-13T23:46:53.560 INFO:journalctl@ceph.mon.b.smithi047.stdout: conn = await self._remote_connection(host, addr) 2022-05-13T23:46:53.561 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-13T23:46:53.561 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise 2022-05-13T23:46:53.561 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-13T23:46:53.561 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.gen.throw(type, value, traceback) 2022-05-13T23:46:53.562 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-13T23:46:53.562 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OrchestratorError(msg) 2022-05-13T23:46:53.562 INFO:journalctl@ceph.mon.b.smithi047.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:46:54.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:46:54 smithi115 conmon[90393]: 2022-05-13T23:46:54.296+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:46:54.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:46:54 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:46:54.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:46:54 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:46:54.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:46:54 smithi115 ceph-mon[78593]: fsmap cephfs:5/5 3 up:standby-replay 1 up:replay 4 up:active 2022-05-13T23:46:54.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:46:54 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 1145 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:46:54.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:46:54 smithi115 ceph-mon[78593]: pgmap v925: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:46:54.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:46:54 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:46:54.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:46:54 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:46:54.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:46:54 smithi047 ceph-mon[77147]: fsmap cephfs:5/5 3 up:standby-replay 1 up:replay 4 up:active 2022-05-13T23:46:54.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:46:54 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 1145 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:46:54.545 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:46:54 smithi047 ceph-mon[77147]: pgmap v925: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:46:55.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:46:55 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:46:55.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:46:55 smithi115 conmon[90393]: 2022-05-13T23:46:55.291+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:46:55.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:46:55 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:46:56.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:46:56 smithi115 conmon[90393]: 2022-05-13T23:46:56.317+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:46:56.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:46:56 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:46:56.362 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:46:56 smithi115 ceph-mon[78593]: pgmap v926: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:46:56.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:46:56 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:46:56.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:46:56 smithi047 ceph-mon[77147]: pgmap v926: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:46:57.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:46:57 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:46:57.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:46:57 smithi115 conmon[90393]: 2022-05-13T23:46:57.291+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:46:57.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:46:57 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:46:58.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:46:58 smithi115 conmon[90393]: 2022-05-13T23:46:58.253+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:46:58.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:46:58 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:46:58.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:46:58 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:46:58.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:46:58 smithi115 ceph-mon[78593]: fsmap cephfs:5/5 3 up:standby-replay 1 up:replay 4 up:active 2022-05-13T23:46:58.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:46:58 smithi115 ceph-mon[78593]: pgmap v927: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:46:58.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:46:58 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:46:58.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:46:58 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:46:58.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:46:58 smithi047 ceph-mon[77147]: fsmap cephfs:5/5 3 up:standby-replay 1 up:replay 4 up:active 2022-05-13T23:46:58.545 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:46:58 smithi047 ceph-mon[77147]: pgmap v927: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:46:59.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:46:59 smithi115 conmon[90393]: 2022-05-13T23:46:59.238+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:46:59.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:46:59 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:46:59.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:46:59 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 1150 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:46:59.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:46:59 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:46:59.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:46:59 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 1150 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:47:00.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:47:00 smithi115 conmon[90393]: 2022-05-13T23:47:00.251+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:47:00.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:47:00 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:47:00.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:47:00 smithi115 ceph-mon[78593]: pgmap v928: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-05-13T23:47:00.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:47:00 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:47:00.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:47:00 smithi047 ceph-mon[77147]: pgmap v928: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-05-13T23:47:01.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:47:01 smithi115 conmon[90393]: 2022-05-13T23:47:01.226+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:47:01.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:47:01 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:47:01.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:47:01 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:47:02.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:47:02 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:47:02.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:47:02 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:47:02.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:47:02 smithi047 ceph-mon[77147]: fsmap cephfs:5/5 3 up:standby-replay 1 up:replay 4 up:active 2022-05-13T23:47:02.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:47:02 smithi047 ceph-mon[77147]: pgmap v929: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.3 KiB/s rd, 4 op/s 2022-05-13T23:47:02.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:47:02 smithi115 conmon[90393]: 2022-05-13T23:47:02.257+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:47:02.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:47:02 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:47:02.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:47:02 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:47:02.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:47:02 smithi115 ceph-mon[78593]: fsmap cephfs:5/5 3 up:standby-replay 1 up:replay 4 up:active 2022-05-13T23:47:02.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:47:02 smithi115 ceph-mon[78593]: pgmap v929: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.3 KiB/s rd, 4 op/s 2022-05-13T23:47:03.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:47:03 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:47:03.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:47:03 smithi115 conmon[90393]: 2022-05-13T23:47:03.297+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:47:03.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:47:03 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:47:04.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:47:04 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:47:04.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:47:04 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 1155 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:47:04.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:47:04 smithi047 ceph-mon[77147]: pgmap v930: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.3 KiB/s rd, 4 op/s 2022-05-13T23:47:04.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:47:04 smithi115 conmon[90393]: 2022-05-13T23:47:04.313+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:47:04.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:47:04 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:47:04.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:47:04 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 1155 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:47:04.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:47:04 smithi115 ceph-mon[78593]: pgmap v930: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.3 KiB/s rd, 4 op/s 2022-05-13T23:47:05.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:47:05 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:47:05.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:47:05 smithi115 conmon[90393]: 2022-05-13T23:47:05.329+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:47:05.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:47:05 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:47:06.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:47:06 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:47:06.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:47:06 smithi047 ceph-mon[77147]: pgmap v931: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-05-13T23:47:06.614 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:47:06 smithi115 conmon[90393]: 2022-05-13T23:47:06.313+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:47:06.615 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:47:06 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:47:06.615 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:47:06 smithi115 ceph-mon[78593]: pgmap v931: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-05-13T23:47:07.554 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:47:07 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:47:07.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:47:07 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:47:07.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:47:07 smithi115 conmon[90393]: 2022-05-13T23:47:07.319+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:47:08.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:47:08 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:47:08.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:47:08 smithi047 ceph-mon[77147]: pgmap v932: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.3 KiB/s rd, 4 op/s 2022-05-13T23:47:08.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:47:08 smithi115 conmon[90393]: 2022-05-13T23:47:08.317+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:47:08.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:47:08 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:47:08.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:47:08 smithi115 ceph-mon[78593]: pgmap v932: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.3 KiB/s rd, 4 op/s 2022-05-13T23:47:09.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:47:09 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:47:09.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:47:09 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 1160 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:47:09.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:47:09 smithi115 conmon[90393]: 2022-05-13T23:47:09.318+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:47:09.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:47:09 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:47:09.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:47:09 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 1160 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:47:10.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:47:10 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:47:10.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:47:10 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:47:10.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:47:10 smithi047 ceph-mon[77147]: fsmap cephfs:5/5 3 up:standby-replay 1 up:replay 4 up:active 2022-05-13T23:47:10.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:47:10 smithi047 ceph-mon[77147]: pgmap v933: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-05-13T23:47:10.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:47:10 smithi115 conmon[90393]: 2022-05-13T23:47:10.341+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:47:10.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:47:10 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:47:10.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:47:10 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:47:10.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:47:10 smithi115 ceph-mon[78593]: fsmap cephfs:5/5 3 up:standby-replay 1 up:replay 4 up:active 2022-05-13T23:47:10.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:47:10 smithi115 ceph-mon[78593]: pgmap v933: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-05-13T23:47:11.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:47:11 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:47:11.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:47:11 smithi115 conmon[90393]: 2022-05-13T23:47:11.335+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:47:11.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:47:11 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:47:12.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:47:12 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:47:12.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:47:12 smithi047 ceph-mon[77147]: pgmap v934: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:47:12.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:47:12 smithi115 conmon[90393]: 2022-05-13T23:47:12.378+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:47:12.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:47:12 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:47:12.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:47:12 smithi115 ceph-mon[78593]: pgmap v934: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:47:13.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:47:13 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:47:13.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:47:13 smithi115 conmon[90393]: 2022-05-13T23:47:13.350+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:47:13.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:47:13 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:47:14.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:47:14 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:47:14.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:47:14 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:47:14.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:47:14 smithi047 ceph-mon[77147]: fsmap cephfs:5/5 3 up:standby-replay 1 up:replay 4 up:active 2022-05-13T23:47:14.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:47:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-13T23:47:14.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:47:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-13T23:47:14.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:47:14 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 1165 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:47:14.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:47:14 smithi047 ceph-mon[77147]: pgmap v935: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:47:14.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:47:14 smithi115 conmon[90393]: 2022-05-13T23:47:14.391+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:47:14.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:47:14 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:47:14.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:47:14 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:47:14.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:47:14 smithi115 ceph-mon[78593]: fsmap cephfs:5/5 3 up:standby-replay 1 up:replay 4 up:active 2022-05-13T23:47:14.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:47:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-13T23:47:14.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:47:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-13T23:47:14.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:47:14 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 1165 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:47:14.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:47:14 smithi115 ceph-mon[78593]: pgmap v935: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:47:15.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:47:15 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:47:15.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:47:15 smithi115 conmon[90393]: 2022-05-13T23:47:15.343+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:47:15.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:47:15 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:47:16.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:47:16 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:47:16.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:47:16 smithi115 conmon[90393]: 2022-05-13T23:47:16.328+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:47:16.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:47:16 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:47:17.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:47:17 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:47:17.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:47:17 smithi047 ceph-mon[77147]: pgmap v936: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:47:17.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:47:17 smithi115 conmon[90393]: 2022-05-13T23:47:17.318+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:47:17.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:47:17 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:47:17.612 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:47:17 smithi115 ceph-mon[78593]: pgmap v936: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:47:18.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:47:18 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:47:18.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:47:18 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:47:18.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:47:18 smithi047 ceph-mon[77147]: fsmap cephfs:5/5 3 up:standby-replay 1 up:replay 4 up:active 2022-05-13T23:47:18.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:47:18 smithi115 conmon[90393]: 2022-05-13T23:47:18.347+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:47:18.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:47:18 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:47:18.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:47:18 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:47:18.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:47:18 smithi115 ceph-mon[78593]: fsmap cephfs:5/5 3 up:standby-replay 1 up:replay 4 up:active 2022-05-13T23:47:19.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:47:19 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:47:19.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:47:19 smithi047 ceph-mon[77147]: pgmap v937: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:47:19.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:47:19 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 1170 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:47:19.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:47:19 smithi115 conmon[90393]: 2022-05-13T23:47:19.344+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:47:19.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:47:19 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:47:19.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:47:19 smithi115 ceph-mon[78593]: pgmap v937: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:47:19.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:47:19 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 1170 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:47:20.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:47:20 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:47:20.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:47:20 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:47:20.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:47:20 smithi115 conmon[90393]: 2022-05-13T23:47:20.315+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:47:21.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:47:21 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:47:21.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:47:21 smithi047 ceph-mon[77147]: pgmap v938: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:47:21.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:47:21 smithi115 conmon[90393]: 2022-05-13T23:47:21.284+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:47:21.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:47:21 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:47:21.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:47:21 smithi115 ceph-mon[78593]: pgmap v938: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:47:22.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:47:22 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:47:22.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:47:22 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:47:22.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:47:22 smithi047 ceph-mon[77147]: fsmap cephfs:5/5 3 up:standby-replay 1 up:replay 4 up:active 2022-05-13T23:47:22.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:47:22 smithi115 conmon[90393]: 2022-05-13T23:47:22.308+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:47:22.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:47:22 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:47:22.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:47:22 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:47:22.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:47:22 smithi115 ceph-mon[78593]: fsmap cephfs:5/5 3 up:standby-replay 1 up:replay 4 up:active 2022-05-13T23:47:23.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:47:23 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:47:23.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:47:23 smithi047 ceph-mon[77147]: pgmap v939: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:47:23.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:47:23 smithi115 conmon[90393]: 2022-05-13T23:47:23.271+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:47:23.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:47:23 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:47:23.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:47:23 smithi115 ceph-mon[78593]: pgmap v939: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:47:24.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:47:24 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:47:24.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:47:24 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 1175 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:47:24.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:47:24 smithi115 conmon[90393]: 2022-05-13T23:47:24.264+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:47:24.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:47:24 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:47:24.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:47:24 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 1175 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:47:25.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:47:25 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:47:25.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:47:25 smithi047 ceph-mon[77147]: pgmap v940: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:47:25.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:47:25 smithi115 conmon[90393]: 2022-05-13T23:47:25.251+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:47:25.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:47:25 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:47:25.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:47:25 smithi115 ceph-mon[78593]: pgmap v940: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:47:26.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:47:26 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:47:26.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:47:26 smithi115 conmon[90393]: 2022-05-13T23:47:26.266+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:47:26.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:47:26 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:47:27.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:47:27 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:47:27.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:47:27 smithi047 ceph-mon[77147]: pgmap v941: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:47:27.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:47:27 smithi115 conmon[90393]: 2022-05-13T23:47:27.270+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:47:27.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:47:27 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:47:27.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:47:27 smithi115 ceph-mon[78593]: pgmap v941: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:47:28.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:47:28 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:47:28.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:47:28 smithi115 conmon[90393]: 2022-05-13T23:47:28.275+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:47:28.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:47:28 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:47:29.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:47:29 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:47:29.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:47:29 smithi047 ceph-mon[77147]: pgmap v942: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:47:29.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:47:29 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 1180 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:47:29.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:47:29 smithi115 conmon[90393]: 2022-05-13T23:47:29.272+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:47:29.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:47:29 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:47:29.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:47:29 smithi115 ceph-mon[78593]: pgmap v942: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:47:29.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:47:29 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 1180 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:47:30.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:47:30 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:47:30.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:47:30 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:47:30.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:47:30 smithi047 ceph-mon[77147]: fsmap cephfs:5/5 3 up:standby-replay 1 up:replay 4 up:active 2022-05-13T23:47:30.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:47:30 smithi115 conmon[90393]: 2022-05-13T23:47:30.228+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:47:30.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:47:30 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:47:30.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:47:30 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:47:30.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:47:30 smithi115 ceph-mon[78593]: fsmap cephfs:5/5 3 up:standby-replay 1 up:replay 4 up:active 2022-05-13T23:47:31.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:47:31 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:47:31.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:47:31 smithi047 ceph-mon[77147]: pgmap v943: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:47:31.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:47:31 smithi115 conmon[90393]: 2022-05-13T23:47:31.213+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:47:31.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:47:31 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:47:31.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:47:31 smithi115 ceph-mon[78593]: pgmap v943: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:47:32.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:47:32 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:47:32.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:47:32 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:47:32.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:47:32 smithi115 conmon[90393]: 2022-05-13T23:47:32.218+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:47:32.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:47:32 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:47:32.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:47:32 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:47:33.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:47:33 smithi047 ceph-mon[77147]: pgmap v944: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:47:33.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:47:33 smithi115 conmon[90393]: 2022-05-13T23:47:33.240+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:47:33.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:47:33 smithi115 ceph-mon[78593]: pgmap v944: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:47:34.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:47:34 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:47:34.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:47:34 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:47:34.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:47:34 smithi047 ceph-mon[77147]: fsmap cephfs:5/5 3 up:standby-replay 1 up:replay 4 up:active 2022-05-13T23:47:34.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:47:34 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 1185 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:47:34.616 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:47:34 smithi115 conmon[90393]: 2022-05-13T23:47:34.232+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:47:34.616 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:47:34 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:47:34.617 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:47:34 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:47:34.617 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:47:34 smithi115 ceph-mon[78593]: fsmap cephfs:5/5 3 up:standby-replay 1 up:replay 4 up:active 2022-05-13T23:47:34.617 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:47:34 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 1185 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:47:35.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:47:35 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:47:35.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:47:35 smithi047 ceph-mon[77147]: pgmap v945: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:47:35.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:47:35 smithi115 conmon[90393]: 2022-05-13T23:47:35.256+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:47:35.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:47:35 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:47:35.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:47:35 smithi115 ceph-mon[78593]: pgmap v945: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:47:36.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:47:36 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:47:36.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:47:36 smithi115 conmon[90393]: 2022-05-13T23:47:36.306+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:47:36.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:47:36 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:47:37.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:47:37 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:47:37.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:47:37 smithi047 ceph-mon[77147]: pgmap v946: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:47:37.610 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:47:37 smithi115 conmon[90393]: 2022-05-13T23:47:37.341+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:47:37.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:47:37 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:47:37.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:47:37 smithi115 ceph-mon[78593]: pgmap v946: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:47:38.545 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:47:38 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:47:38.546 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:47:38 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:47:38.550 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:47:38 smithi047 ceph-mon[77147]: fsmap cephfs:5/5 3 up:standby-replay 1 up:replay 4 up:active 2022-05-13T23:47:38.551 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:47:38 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:47:38.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:47:38 smithi115 conmon[90393]: 2022-05-13T23:47:38.365+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.2.197:6 2.1 2.96a7dda1 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e150) 2022-05-13T23:47:38.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:47:38 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:47:38.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:47:38 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:47:38.613 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:47:38 smithi115 ceph-mon[78593]: fsmap cephfs:5/5 3 up:standby-replay 1 up:replay 4 up:active 2022-05-13T23:47:38.614 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:47:38 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:47:39.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:47:39 smithi047 ceph-mon[77147]: pgmap v947: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:47:39.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:47:39 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 1190 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:47:39.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:47:39 smithi047 ceph-mon[77147]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2022-05-13T23:47:39.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:47:39 smithi115 conmon[90393]: 2022-05-13T23:47:39.364+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:47:39.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:47:39 smithi115 ceph-mon[78593]: pgmap v947: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:47:39.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:47:39 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 1190 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:47:39.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:47:39 smithi115 ceph-mon[78593]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2022-05-13T23:47:40.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:47:40 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:47:40.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:47:40 smithi115 conmon[90393]: 2022-05-13T23:47:40.356+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:47:40.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:47:40 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:47:41.548 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:47:41 smithi047 ceph-mon[77147]: pgmap v948: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:47:41.552 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:47:41 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:47:41.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:47:41 smithi115 conmon[90393]: 2022-05-13T23:47:41.391+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:47:41.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:47:41 smithi115 ceph-mon[78593]: pgmap v948: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:47:41.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:47:41 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:47:42.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:47:42 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:47:42.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:47:42 smithi047 ceph-mon[77147]: fsmap cephfs:5/5 3 up:standby-replay 1 up:replay 4 up:active 2022-05-13T23:47:42.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:47:42 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:47:42.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:47:42 smithi115 conmon[90393]: 2022-05-13T23:47:42.411+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:47:42.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:47:42 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:47:42.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:47:42 smithi115 ceph-mon[78593]: fsmap cephfs:5/5 3 up:standby-replay 1 up:replay 4 up:active 2022-05-13T23:47:42.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:47:42 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:47:43.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:47:43 smithi047 ceph-mon[77147]: pgmap v949: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:47:43.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:47:43 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:47:43.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:47:43 smithi115 conmon[90393]: 2022-05-13T23:47:43.456+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:47:43.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:47:43 smithi115 ceph-mon[78593]: pgmap v949: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:47:43.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:47:43 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:47:44.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:47:44 smithi047 ceph-mon[77147]: Health check update: 1 slow ops, oldest one blocked for 1195 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:47:44.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:47:44 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:47:44.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:47:44 smithi115 conmon[90393]: 2022-05-13T23:47:44.469+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:47:44.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:47:44 smithi115 ceph-mon[78593]: Health check update: 1 slow ops, oldest one blocked for 1195 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:47:44.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:47:44 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:47:45.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:47:45 smithi115 conmon[90393]: 2022-05-13T23:47:45.457+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:47:45.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:47:45 smithi115 ceph-mon[78593]: pgmap v950: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:47:45.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:47:45 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:47:45.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:47:45 smithi047 ceph-mon[77147]: pgmap v950: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:47:45.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:47:45 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:47:46.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:47:46 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:47:46.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:47:46 smithi115 conmon[90393]: 2022-05-13T23:47:46.487+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:47:46.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:47:46 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:47:47.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:47:47 smithi047 ceph-mon[77147]: pgmap v951: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:47:47.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:47:47 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:47:47.864 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:47:47 smithi115 conmon[90393]: 2022-05-13T23:47:47.487+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:47:47.864 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:47:47 smithi115 ceph-mon[78593]: pgmap v951: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:47:47.865 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:47:47 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:47:48.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:47:48 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:47:48.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:47:48 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 1200 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:47:48.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:47:48 smithi115 conmon[90393]: 2022-05-13T23:47:48.518+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:47:48.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:47:48 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:47:48.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:47:48 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 1200 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:47:49.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:47:49 smithi047 ceph-mon[77147]: pgmap v952: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:47:49.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:47:49 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:47:49.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:47:49 smithi115 conmon[90393]: 2022-05-13T23:47:49.562+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:47:49.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:47:49 smithi115 ceph-mon[78593]: pgmap v952: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:47:49.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:47:49 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:47:50.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:47:50 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:47:50.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:47:50 smithi047 ceph-mon[77147]: fsmap cephfs:5/5 3 up:standby-replay 1 up:replay 4 up:active 2022-05-13T23:47:50.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:47:50 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:47:50.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:47:50 smithi115 conmon[90393]: 2022-05-13T23:47:50.546+0000 7fa085a08700 -1 osd.5 150 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:47:50.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:47:50 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:47:50.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:47:50 smithi115 ceph-mon[78593]: fsmap cephfs:5/5 3 up:standby-replay 1 up:replay 4 up:active 2022-05-13T23:47:50.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:47:50 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:47:51.425 INFO:journalctl@ceph.mds.f.smithi115.stdout:May 13 23:47:51 smithi115 conmon[105998]: 2022-05-13T23:47:51.159+0000 7f428c114700 -1 log_channel(cluster) log [ERR] : Error recovering journal 0x202: (110) Connection timed out 2022-05-13T23:47:51.425 INFO:journalctl@ceph.mds.f.smithi115.stdout:May 13 23:47:51 smithi115 conmon[105998]: -37> 2022-05-13T23:47:51.159+0000 7f428c114700 -1 log_channel(cluster) log [ERR] : Error recovering journal 0x202: (110) Connection timed out 2022-05-13T23:47:51.426 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:47:51 smithi115 ceph-mon[78593]: pgmap v953: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:47:51.426 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:47:51 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:47:51.426 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:47:51 smithi115 ceph-mon[78593]: Error recovering journal 0x202: (110) Connection timed out 2022-05-13T23:47:51.427 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:47:51 smithi115 ceph-mon[78593]: osdmap e151: 12 total, 8 up, 8 in 2022-05-13T23:47:51.754 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:47:51 smithi047 conmon[77123]: 2022-05-13T23:47:51.379+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 mds daemon damaged (MDS_DAMAGE) 2022-05-13T23:47:51.755 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:47:51 smithi047 ceph-mon[77147]: pgmap v953: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:47:51.755 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:47:51 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:47:51.755 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:47:51 smithi047 ceph-mon[77147]: Error recovering journal 0x202: (110) Connection timed out 2022-05-13T23:47:51.756 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:47:51 smithi047 ceph-mon[77147]: osdmap e151: 12 total, 8 up, 8 in 2022-05-13T23:47:51.756 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:47:51 smithi115 conmon[90393]: 2022-05-13T23:47:51.533+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:47:51.757 INFO:journalctl@ceph.mds.f.smithi115.stdout:May 13 23:47:51 smithi115 conmon[105998]: ignoring --setuser ceph since I am not root 2022-05-13T23:47:51.758 INFO:journalctl@ceph.mds.f.smithi115.stdout:May 13 23:47:51 smithi115 conmon[105998]: ignoring --setgroup ceph since I am not root 2022-05-13T23:47:51.758 INFO:journalctl@ceph.mds.f.smithi115.stdout:May 13 23:47:51 smithi115 ceph-mds[106022]: -- 172.21.15.115:0/100653601 <== mon.2 v2:172.21.15.115:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5635fc314340 con 0x5635fc2ecc00 2022-05-13T23:47:51.759 INFO:journalctl@ceph.mds.f.smithi115.stdout:May 13 23:47:51 smithi115 conmon[105998]: starting mds.f at 2022-05-13T23:47:52.546 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:47:52 smithi047 ceph-mon[77147]: Health check failed: 1 mds daemon damaged (MDS_DAMAGE) 2022-05-13T23:47:52.546 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:47:52 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6834/3300271029,v1:172.21.15.115:6835/3300271029] down:damaged 2022-05-13T23:47:52.546 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:47:52 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay, 1 damaged 2022-05-13T23:47:52.547 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:47:52 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:47:52.547 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:47:52 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-13T23:47:52.547 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:47:52 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-13T23:47:52.547 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:47:52 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-13T23:47:52.561 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:47:52 smithi115 ceph-mon[78593]: Health check failed: 1 mds daemon damaged (MDS_DAMAGE) 2022-05-13T23:47:52.562 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:47:52 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6834/3300271029,v1:172.21.15.115:6835/3300271029] down:damaged 2022-05-13T23:47:52.562 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:47:52 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay, 1 damaged 2022-05-13T23:47:52.562 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:47:52 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:47:52.563 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:47:52 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-13T23:47:52.563 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:47:52 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-13T23:47:52.563 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:47:52 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-13T23:47:52.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:47:52 smithi115 conmon[90393]: 2022-05-13T23:47:52.560+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:47:53.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:47:53 smithi047 ceph-mon[77147]: pgmap v955: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-05-13T23:47:53.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:47:53 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6834/2223119766,v1:172.21.15.115:6835/2223119766] up:boot 2022-05-13T23:47:53.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:47:53 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:47:53.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:47:53 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-13T23:47:53.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:47:53 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2022-05-13T23:47:53.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:47:53 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:47:53.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:47:53 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 1205 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:47:53.793 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:47:53 smithi047 conmon[78521]: 2022-05-13T23:47:53.583+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:47:53.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:47:53 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-13T23:47:53.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:47:53 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-13T23:47:53.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:47:53 smithi047 conmon[78521]: yield 2022-05-13T23:47:53.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:47:53 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-13T23:47:53.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:47:53 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-13T23:47:53.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:47:53 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-13T23:47:53.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:47:53 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-13T23:47:53.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:47:53 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-13T23:47:53.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:47:53 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-13T23:47:53.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:47:53 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-13T23:47:53.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:47:53 smithi047 conmon[78521]: raise exceptions[0] 2022-05-13T23:47:53.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:47:53 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-13T23:47:53.797 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:47:53 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-13T23:47:53.797 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:47:53 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-13T23:47:53.797 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:47:53 smithi047 conmon[78521]: return (yield from fut) 2022-05-13T23:47:53.798 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:47:53 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-13T23:47:53.798 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:47:53 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-13T23:47:53.798 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:47:53 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:47:53.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:47:53 smithi115 conmon[90393]: 2022-05-13T23:47:53.513+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:47:53.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:47:53 smithi115 ceph-mon[78593]: pgmap v955: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-05-13T23:47:53.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:47:53 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6834/2223119766,v1:172.21.15.115:6835/2223119766] up:boot 2022-05-13T23:47:53.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:47:53 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:47:53.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:47:53 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-13T23:47:53.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:47:53 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2022-05-13T23:47:53.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:47:53 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:47:53.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:47:53 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 1205 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:47:54.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:47:54 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:47:54.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:47:54 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:47:54.792 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-13T23:47:54.793 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-13T23:47:54.793 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-13T23:47:54.793 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-13T23:47:54.793 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-13T23:47:54.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-13T23:47:54.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-13T23:47:54.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-13T23:47:54.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-13T23:47:54.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-13T23:47:54.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-13T23:47:54.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-13T23:47:54.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-13T23:47:54.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-13T23:47:54.797 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-13T23:47:54.797 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-13T23:47:54.797 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-13T23:47:54.797 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:47:54.798 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:47:54 smithi047 ceph-mon[77147]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-13T23:47:54.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:47:54 smithi115 conmon[90393]: 2022-05-13T23:47:54.555+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:47:54.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:47:54 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:47:54.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:47:54 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:47:54.859 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-13T23:47:54.860 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-13T23:47:54.860 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-13T23:47:54.860 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-13T23:47:54.860 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-13T23:47:54.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-13T23:47:54.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-13T23:47:54.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-13T23:47:54.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-13T23:47:54.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-13T23:47:54.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-13T23:47:54.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-13T23:47:54.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-13T23:47:54.863 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-13T23:47:54.863 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-13T23:47:54.863 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-13T23:47:54.863 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-13T23:47:54.864 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:47:54.864 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:47:54 smithi115 ceph-mon[78593]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-13T23:47:55.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:47:55 smithi047 ceph-mon[77147]: pgmap v956: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-05-13T23:47:55.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:47:55 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:47:55.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:47:55 smithi115 conmon[90393]: 2022-05-13T23:47:55.599+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:47:55.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:47:55 smithi115 ceph-mon[78593]: pgmap v956: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-05-13T23:47:55.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:47:55 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:47:56.663 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:47:56 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:47:56.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:47:56 smithi115 conmon[90393]: 2022-05-13T23:47:56.617+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:47:56.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:47:56 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:47:57.041 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:47:56 smithi047 conmon[78521]: 2022-05-13T23:47:56.655+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:47:57.042 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:47:56 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-13T23:47:57.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:47:56 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-13T23:47:57.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:47:56 smithi047 conmon[78521]: yield 2022-05-13T23:47:57.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:47:56 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-13T23:47:57.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:47:56 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-13T23:47:57.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:47:56 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-13T23:47:57.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:47:56 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-13T23:47:57.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:47:56 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-13T23:47:57.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:47:56 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-13T23:47:57.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:47:56 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-13T23:47:57.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:47:56 smithi047 conmon[78521]: raise exceptions[0] 2022-05-13T23:47:57.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:47:56 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-13T23:47:57.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:47:56 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-13T23:47:57.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:47:56 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-13T23:47:57.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:47:56 smithi047 conmon[78521]: return (yield from fut) 2022-05-13T23:47:57.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:47:56 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-13T23:47:57.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:47:56 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-13T23:47:57.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:47:56 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:47:57.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:47:56 smithi047 conmon[78521]: 2022-05-13T23:47:56.656+0000 7f533f777700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-13T23:47:57.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:47:56 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-13T23:47:57.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:47:56 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-13T23:47:57.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:47:56 smithi047 conmon[78521]: yield 2022-05-13T23:47:57.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:47:56 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-13T23:47:57.048 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:47:56 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-13T23:47:57.048 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:47:56 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-13T23:47:57.048 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:47:56 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-13T23:47:57.049 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:47:56 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-13T23:47:57.049 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:47:56 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-13T23:47:57.049 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:47:56 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-13T23:47:57.049 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:47:56 smithi047 conmon[78521]: raise exceptions[0] 2022-05-13T23:47:57.050 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:47:56 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-13T23:47:57.050 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:47:56 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-13T23:47:57.050 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:47:56 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-13T23:47:57.050 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:47:56 smithi047 conmon[78521]: return (yield from fut) 2022-05-13T23:47:57.050 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:47:56 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-13T23:47:57.051 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:47:56 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-13T23:47:57.051 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:47:56 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:47:57.051 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:47:56 smithi047 conmon[78521]: 2022-05-13T23:47:57.051 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:47:56 smithi047 conmon[78521]: During handling of the above exception, another exception occurred: 2022-05-13T23:47:57.051 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:47:56 smithi047 conmon[78521]: 2022-05-13T23:47:57.052 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:47:56 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-13T23:47:57.052 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:47:56 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-13T23:47:57.052 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:47:56 smithi047 conmon[78521]: return f(*arg) 2022-05-13T23:47:57.053 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:47:56 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-13T23:47:57.053 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:47:56 smithi047 conmon[78521]: self._write_client_files(client_files, host) 2022-05-13T23:47:57.053 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:47:56 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-13T23:47:57.053 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:47:56 smithi047 conmon[78521]: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-13T23:47:57.054 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:47:56 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-13T23:47:57.054 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:47:56 smithi047 conmon[78521]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-13T23:47:57.054 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:47:56 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-13T23:47:57.054 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:47:56 smithi047 conmon[78521]: return self.event_loop.get_result(coro) 2022-05-13T23:47:57.055 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:47:56 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-13T23:47:57.055 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:47:56 smithi047 conmon[78521]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-13T23:47:57.055 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:47:56 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-13T23:47:57.055 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:47:56 smithi047 conmon[78521]: return self.__get_result() 2022-05-13T23:47:57.055 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:47:56 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-13T23:47:57.056 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:47:56 smithi047 conmon[78521]: raise self._exception 2022-05-13T23:47:57.056 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:47:56 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-13T23:47:57.056 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:47:56 smithi047 conmon[78521]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-13T23:47:57.056 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:47:56 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-13T23:47:57.056 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:47:56 smithi047 conmon[78521]: conn = await self._remote_connection(host, addr) 2022-05-13T23:47:57.057 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:47:56 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-13T23:47:57.057 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:47:56 smithi047 conmon[78521]: raise 2022-05-13T23:47:57.058 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:47:56 smithi047 conmon[78521]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-13T23:47:57.058 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:47:56 smithi047 conmon[78521]: self.gen.throw(type, value, traceback) 2022-05-13T23:47:57.059 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:47:56 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-13T23:47:57.059 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:47:56 smithi047 conmon[78521]: raise OrchestratorError(msg) 2022-05-13T23:47:57.059 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:47:56 smithi047 conmon[78521]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:47:57.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:47:57 smithi047 ceph-mon[77147]: pgmap v957: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-05-13T23:47:57.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:47:57 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:47:57.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:47:57 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-13T23:47:57.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:47:57 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:47:57.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:47:57 smithi115 conmon[90393]: 2022-05-13T23:47:57.622+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:47:57.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:47:57 smithi115 ceph-mon[78593]: pgmap v957: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-05-13T23:47:57.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:47:57 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:47:57.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:47:57 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-13T23:47:57.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:47:57 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:47:58.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:47:58 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:47:58.791 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-13T23:47:58.792 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-13T23:47:58.792 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-13T23:47:58.792 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-13T23:47:58.792 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-13T23:47:58.792 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-13T23:47:58.793 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-13T23:47:58.793 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-13T23:47:58.793 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-13T23:47:58.793 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-13T23:47:58.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-13T23:47:58.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-13T23:47:58.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-13T23:47:58.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-13T23:47:58.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-13T23:47:58.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-13T23:47:58.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-13T23:47:58.797 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:47:58.797 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:47:58 smithi047 ceph-mon[77147]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-13T23:47:58.797 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-13T23:47:58.798 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-13T23:47:58.798 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-13T23:47:58.799 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-13T23:47:58.802 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-13T23:47:58.803 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-13T23:47:58.803 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-13T23:47:58.804 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-13T23:47:58.804 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-13T23:47:58.804 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-13T23:47:58.805 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-13T23:47:58.805 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-13T23:47:58.806 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-13T23:47:58.806 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-13T23:47:58.806 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-13T23:47:58.807 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-13T23:47:58.807 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-13T23:47:58.807 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:47:58.808 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-13T23:47:58.809 INFO:journalctl@ceph.mon.b.smithi047.stdout: During handling of the above exception, another exception occurred: 2022-05-13T23:47:58.809 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-13T23:47:58.810 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-13T23:47:58.810 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-13T23:47:58.810 INFO:journalctl@ceph.mon.b.smithi047.stdout: return f(*arg) 2022-05-13T23:47:58.811 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-13T23:47:58.812 INFO:journalctl@ceph.mon.b.smithi047.stdout: self._write_client_files(client_files, host) 2022-05-13T23:47:58.812 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-13T23:47:58.813 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-13T23:47:58.813 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-13T23:47:58.813 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-13T23:47:58.813 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-13T23:47:58.814 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.event_loop.get_result(coro) 2022-05-13T23:47:58.814 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-13T23:47:58.814 INFO:journalctl@ceph.mon.b.smithi047.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-13T23:47:58.814 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-13T23:47:58.814 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.__get_result() 2022-05-13T23:47:58.815 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-13T23:47:58.815 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise self._exception 2022-05-13T23:47:58.815 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-13T23:47:58.815 INFO:journalctl@ceph.mon.b.smithi047.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-13T23:47:58.816 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-13T23:47:58.816 INFO:journalctl@ceph.mon.b.smithi047.stdout: conn = await self._remote_connection(host, addr) 2022-05-13T23:47:58.816 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-13T23:47:58.817 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise 2022-05-13T23:47:58.817 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-13T23:47:58.818 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.gen.throw(type, value, traceback) 2022-05-13T23:47:58.818 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-13T23:47:58.818 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OrchestratorError(msg) 2022-05-13T23:47:58.819 INFO:journalctl@ceph.mon.b.smithi047.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:47:58.820 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:47:58 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:47:58.820 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:47:58 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 1210 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:47:58.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:47:58 smithi115 conmon[90393]: 2022-05-13T23:47:58.618+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:47:58.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:47:58 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:47:58.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-13T23:47:58.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-13T23:47:58.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-13T23:47:58.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-13T23:47:58.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-13T23:47:58.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-13T23:47:58.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-13T23:47:58.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-13T23:47:58.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-13T23:47:58.863 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-13T23:47:58.863 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-13T23:47:58.863 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-13T23:47:58.863 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-13T23:47:58.863 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-13T23:47:58.864 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-13T23:47:58.864 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-13T23:47:58.864 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-13T23:47:58.864 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:47:58.865 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:47:58 smithi115 ceph-mon[78593]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-13T23:47:58.865 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-13T23:47:58.865 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-13T23:47:58.865 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-13T23:47:58.865 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-13T23:47:58.866 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-13T23:47:58.866 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-13T23:47:58.866 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-13T23:47:58.866 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-13T23:47:58.866 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-13T23:47:58.867 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-13T23:47:58.867 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-13T23:47:58.867 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-13T23:47:58.867 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-13T23:47:58.868 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-13T23:47:58.868 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-13T23:47:58.868 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-13T23:47:58.868 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-13T23:47:58.868 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:47:58.869 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-13T23:47:58.869 INFO:journalctl@ceph.mon.c.smithi115.stdout: During handling of the above exception, another exception occurred: 2022-05-13T23:47:58.870 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-13T23:47:58.875 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-13T23:47:58.875 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-13T23:47:58.876 INFO:journalctl@ceph.mon.c.smithi115.stdout: return f(*arg) 2022-05-13T23:47:58.876 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-13T23:47:58.876 INFO:journalctl@ceph.mon.c.smithi115.stdout: self._write_client_files(client_files, host) 2022-05-13T23:47:58.876 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-13T23:47:58.877 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-13T23:47:58.877 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-13T23:47:58.877 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-13T23:47:58.877 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-13T23:47:58.877 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.event_loop.get_result(coro) 2022-05-13T23:47:58.878 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-13T23:47:58.878 INFO:journalctl@ceph.mon.c.smithi115.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-13T23:47:58.878 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-13T23:47:58.878 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.__get_result() 2022-05-13T23:47:58.879 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-13T23:47:58.879 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise self._exception 2022-05-13T23:47:58.879 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-13T23:47:58.879 INFO:journalctl@ceph.mon.c.smithi115.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-13T23:47:58.879 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-13T23:47:58.880 INFO:journalctl@ceph.mon.c.smithi115.stdout: conn = await self._remote_connection(host, addr) 2022-05-13T23:47:58.880 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-13T23:47:58.880 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise 2022-05-13T23:47:58.880 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-13T23:47:58.881 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.gen.throw(type, value, traceback) 2022-05-13T23:47:58.881 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-13T23:47:58.881 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OrchestratorError(msg) 2022-05-13T23:47:58.881 INFO:journalctl@ceph.mon.c.smithi115.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:47:58.882 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:47:58 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:47:58.882 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:47:58 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 1210 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:47:59.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:47:59 smithi047 ceph-mon[77147]: pgmap v958: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-05-13T23:47:59.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:47:59 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:47:59.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:47:59 smithi115 conmon[90393]: 2022-05-13T23:47:59.586+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:47:59.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:47:59 smithi115 ceph-mon[78593]: pgmap v958: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-05-13T23:47:59.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:47:59 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:48:00.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:48:00 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:48:00.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:48:00 smithi115 conmon[90393]: 2022-05-13T23:48:00.610+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:48:00.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:48:00 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:48:01.804 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:48:01 smithi047 ceph-mon[77147]: pgmap v959: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-05-13T23:48:01.805 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:48:01 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:48:01.805 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:48:01 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-13T23:48:01.806 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:48:01 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:48:01.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:48:01 smithi115 conmon[90393]: 2022-05-13T23:48:01.561+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:48:01.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:48:01 smithi115 ceph-mon[78593]: pgmap v959: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-05-13T23:48:01.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:48:01 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:48:01.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:48:01 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-13T23:48:01.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:48:01 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:48:02.797 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:48:02 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:48:02.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:48:02 smithi115 conmon[90393]: 2022-05-13T23:48:02.537+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:48:02.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:48:02 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:48:03.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:48:03 smithi047 ceph-mon[77147]: pgmap v960: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-05-13T23:48:03.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:48:03 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:48:03.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:48:03 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 1215 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:48:03.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:48:03 smithi115 conmon[90393]: 2022-05-13T23:48:03.520+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:48:03.887 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:48:03 smithi115 ceph-mon[78593]: pgmap v960: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2022-05-13T23:48:03.887 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:48:03 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:48:03.888 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:48:03 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 1215 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:48:04.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:48:04 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:48:04.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:48:04 smithi115 conmon[90393]: 2022-05-13T23:48:04.518+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:48:04.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:48:04 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:48:05.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:48:05 smithi047 ceph-mon[77147]: pgmap v961: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:48:05.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:48:05 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:48:05.867 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:48:05 smithi115 conmon[90393]: 2022-05-13T23:48:05.536+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:48:05.867 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:48:05 smithi115 ceph-mon[78593]: pgmap v961: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:48:05.868 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:48:05 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:48:06.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:48:06 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:48:06.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:48:06 smithi115 conmon[90393]: 2022-05-13T23:48:06.496+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:48:06.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:48:06 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:48:07.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:48:07 smithi047 ceph-mon[77147]: pgmap v962: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:48:07.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:48:07 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:48:07.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:48:07 smithi115 conmon[90393]: 2022-05-13T23:48:07.532+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:48:07.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:48:07 smithi115 ceph-mon[78593]: pgmap v962: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:48:07.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:48:07 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:48:08.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:48:08 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:48:08.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:48:08 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 1220 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:48:08.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:48:08 smithi115 conmon[90393]: 2022-05-13T23:48:08.520+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:48:08.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:48:08 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:48:08.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:48:08 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 1220 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:48:09.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:48:09 smithi047 ceph-mon[77147]: pgmap v963: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:48:09.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:48:09 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:48:09.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:48:09 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:48:09.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:48:09 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-13T23:48:09.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:48:09 smithi115 conmon[90393]: 2022-05-13T23:48:09.549+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:48:09.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:48:09 smithi115 ceph-mon[78593]: pgmap v963: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:48:09.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:48:09 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:48:09.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:48:09 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:48:09.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:48:09 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-13T23:48:10.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:48:10 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:48:10.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:48:10 smithi115 conmon[90393]: 2022-05-13T23:48:10.595+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:48:10.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:48:10 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:48:11.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:48:11 smithi047 ceph-mon[77147]: pgmap v964: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:48:11.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:48:11 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:48:11.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:48:11 smithi115 conmon[90393]: 2022-05-13T23:48:11.589+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:48:11.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:48:11 smithi115 ceph-mon[78593]: pgmap v964: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:48:11.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:48:11 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:48:12.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:48:12 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:48:12.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:48:12 smithi115 conmon[90393]: 2022-05-13T23:48:12.630+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:48:12.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:48:12 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:48:13.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:48:13 smithi047 ceph-mon[77147]: pgmap v965: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:48:13.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:48:13 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:48:13.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:48:13 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-13T23:48:13.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:48:13 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:48:13.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:48:13 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-13T23:48:13.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:48:13 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-13T23:48:13.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:48:13 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 1225 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:48:13.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:48:13 smithi115 conmon[90393]: 2022-05-13T23:48:13.611+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:48:13.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:48:13 smithi115 ceph-mon[78593]: pgmap v965: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:48:13.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:48:13 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:48:13.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:48:13 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-13T23:48:13.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:48:13 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:48:13.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:48:13 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-13T23:48:13.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:48:13 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-13T23:48:13.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:48:13 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 1225 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:48:14.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:48:14 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:48:14.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:48:14 smithi115 conmon[90393]: 2022-05-13T23:48:14.577+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:48:14.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:48:14 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:48:15.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:48:15 smithi047 ceph-mon[77147]: pgmap v966: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:48:15.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:48:15 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:48:15.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:48:15 smithi115 conmon[90393]: 2022-05-13T23:48:15.560+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:48:15.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:48:15 smithi115 ceph-mon[78593]: pgmap v966: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:48:15.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:48:15 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:48:16.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:48:16 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:48:16.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:48:16 smithi115 conmon[90393]: 2022-05-13T23:48:16.597+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:48:16.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:48:16 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:48:17.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:48:17 smithi047 ceph-mon[77147]: pgmap v967: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:48:17.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:48:17 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:48:17.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:48:17 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-13T23:48:17.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:48:17 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:48:17.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:48:17 smithi115 conmon[90393]: 2022-05-13T23:48:17.551+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:48:17.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:48:17 smithi115 ceph-mon[78593]: pgmap v967: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:48:17.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:48:17 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:48:17.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:48:17 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-13T23:48:17.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:48:17 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:48:18.796 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:48:18 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:48:18.796 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:48:18 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 1230 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:48:18.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:48:18 smithi115 conmon[90393]: 2022-05-13T23:48:18.568+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:48:18.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:48:18 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:48:18.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:48:18 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 1230 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:48:19.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:48:19 smithi047 ceph-mon[77147]: pgmap v968: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:48:19.795 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:48:19 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:48:19.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:48:19 smithi115 conmon[90393]: 2022-05-13T23:48:19.612+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:48:19.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:48:19 smithi115 ceph-mon[78593]: pgmap v968: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:48:19.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:48:19 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:48:20.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:48:20 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:48:20.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:48:20 smithi115 conmon[90393]: 2022-05-13T23:48:20.568+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:48:20.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:48:20 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:48:21.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:48:21 smithi047 ceph-mon[77147]: pgmap v969: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:48:21.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:48:21 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:48:21.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:48:21 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-13T23:48:21.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:48:21 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:48:21.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:48:21 smithi115 conmon[90393]: 2022-05-13T23:48:21.579+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:48:21.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:48:21 smithi115 ceph-mon[78593]: pgmap v969: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:48:21.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:48:21 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:48:21.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:48:21 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-13T23:48:21.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:48:21 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:48:22.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:48:22 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:48:22.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:48:22 smithi115 conmon[90393]: 2022-05-13T23:48:22.581+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:48:22.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:48:22 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:48:23.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:48:23 smithi047 ceph-mon[77147]: pgmap v970: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:48:23.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:48:23 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:48:23.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:48:23 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 1235 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:48:23.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:48:23 smithi115 conmon[90393]: 2022-05-13T23:48:23.609+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:48:23.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:48:23 smithi115 ceph-mon[78593]: pgmap v970: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:48:23.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:48:23 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:48:23.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:48:23 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 1235 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:48:24.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:48:24 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:48:24.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:48:24 smithi115 conmon[90393]: 2022-05-13T23:48:24.626+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:48:24.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:48:24 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:48:25.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:48:25 smithi047 ceph-mon[77147]: pgmap v971: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:48:25.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:48:25 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:48:25.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:48:25 smithi115 conmon[90393]: 2022-05-13T23:48:25.668+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:48:25.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:48:25 smithi115 ceph-mon[78593]: pgmap v971: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:48:25.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:48:25 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:48:26.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:48:26 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:48:26.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:48:26 smithi115 conmon[90393]: 2022-05-13T23:48:26.629+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:48:26.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:48:26 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:48:27.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:48:27 smithi047 ceph-mon[77147]: pgmap v972: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:48:27.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:48:27 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:48:27.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:48:27 smithi115 conmon[90393]: 2022-05-13T23:48:27.588+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:48:27.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:48:27 smithi115 ceph-mon[78593]: pgmap v972: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:48:27.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:48:27 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:48:28.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:48:28 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:48:28.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:48:28 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 1240 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:48:28.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:48:28 smithi115 conmon[90393]: 2022-05-13T23:48:28.612+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:48:28.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:48:28 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:48:28.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:48:28 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 1240 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:48:29.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:48:29 smithi047 ceph-mon[77147]: pgmap v973: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:48:29.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:48:29 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:48:29.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:48:29 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:48:29.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:48:29 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-13T23:48:29.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:48:29 smithi115 conmon[90393]: 2022-05-13T23:48:29.651+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:48:29.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:48:29 smithi115 ceph-mon[78593]: pgmap v973: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:48:29.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:48:29 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:48:29.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:48:29 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:48:29.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:48:29 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-13T23:48:30.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:48:30 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:48:30.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:48:30 smithi115 conmon[90393]: 2022-05-13T23:48:30.667+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:48:30.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:48:30 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:48:31.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:48:31 smithi047 ceph-mon[77147]: pgmap v974: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:48:31.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:48:31 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:48:31.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:48:31 smithi115 conmon[90393]: 2022-05-13T23:48:31.675+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:48:31.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:48:31 smithi115 ceph-mon[78593]: pgmap v974: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:48:31.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:48:31 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:48:32.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:48:32 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:48:32.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:48:32 smithi115 conmon[90393]: 2022-05-13T23:48:32.692+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:48:32.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:48:32 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:48:33.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:48:33 smithi047 ceph-mon[77147]: pgmap v975: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:48:33.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:48:33 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:48:33.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:48:33 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-13T23:48:33.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:48:33 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:48:33.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:48:33 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 1245 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:48:33.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:48:33 smithi115 conmon[90393]: 2022-05-13T23:48:33.735+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:48:33.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:48:33 smithi115 ceph-mon[78593]: pgmap v975: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:48:33.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:48:33 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:48:33.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:48:33 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-13T23:48:33.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:48:33 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:48:33.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:48:33 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 1245 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:48:34.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:48:34 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:48:34.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:48:34 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:48:34.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:48:34 smithi115 conmon[90393]: 2022-05-13T23:48:34.712+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:48:35.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:48:35 smithi047 ceph-mon[77147]: pgmap v976: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:48:35.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:48:35 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:48:35.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:48:35 smithi115 conmon[90393]: 2022-05-13T23:48:35.666+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:48:35.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:48:35 smithi115 ceph-mon[78593]: pgmap v976: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:48:35.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:48:35 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:48:36.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:48:36 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:48:36.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:48:36 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:48:36.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:48:36 smithi115 conmon[90393]: 2022-05-13T23:48:36.687+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:48:37.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:48:37 smithi047 ceph-mon[77147]: pgmap v977: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:48:37.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:48:37 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:48:37.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:48:37 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-13T23:48:37.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:48:37 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:48:37.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:48:37 smithi115 conmon[90393]: 2022-05-13T23:48:37.675+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:48:37.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:48:37 smithi115 ceph-mon[78593]: pgmap v977: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:48:37.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:48:37 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:48:37.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:48:37 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-13T23:48:37.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:48:37 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:48:38.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:48:38 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:48:38.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:48:38 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 1250 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:48:38.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:48:38 smithi115 conmon[90393]: 2022-05-13T23:48:38.657+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:48:38.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:48:38 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:48:38.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:48:38 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 1250 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:48:39.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:48:39 smithi047 ceph-mon[77147]: pgmap v978: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:48:39.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:48:39 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:48:39.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:48:39 smithi115 conmon[90393]: 2022-05-13T23:48:39.689+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:48:39.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:48:39 smithi115 ceph-mon[78593]: pgmap v978: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:48:39.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:48:39 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:48:40.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:48:40 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:48:40.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:48:40 smithi115 conmon[90393]: 2022-05-13T23:48:40.725+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:48:41.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:48:40 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:48:41.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:48:41 smithi115 conmon[90393]: 2022-05-13T23:48:41.713+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:48:41.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:48:41 smithi115 ceph-mon[78593]: pgmap v979: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:48:41.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:48:41 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:48:41.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:48:41 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-13T23:48:41.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:48:41 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:48:42.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:48:41 smithi047 ceph-mon[77147]: pgmap v979: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:48:42.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:48:41 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:48:42.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:48:41 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-13T23:48:42.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:48:41 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:48:42.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:48:42 smithi115 conmon[90393]: 2022-05-13T23:48:42.706+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:48:42.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:48:42 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:48:43.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:48:42 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:48:43.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:48:43 smithi115 conmon[90393]: 2022-05-13T23:48:43.665+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:48:43.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:48:43 smithi115 ceph-mon[78593]: pgmap v980: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:48:43.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:48:43 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:48:43.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:48:43 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 1255 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:48:44.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:48:43 smithi047 ceph-mon[77147]: pgmap v980: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:48:44.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:48:43 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:48:44.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:48:43 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 1255 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:48:44.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:48:44 smithi115 conmon[90393]: 2022-05-13T23:48:44.656+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:48:44.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:48:44 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:48:45.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:48:44 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:48:45.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:48:45 smithi115 conmon[90393]: 2022-05-13T23:48:45.676+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:48:45.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:48:45 smithi115 ceph-mon[78593]: pgmap v981: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:48:45.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:48:45 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:48:46.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:48:45 smithi047 ceph-mon[77147]: pgmap v981: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:48:46.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:48:45 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:48:46.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:48:46 smithi115 conmon[90393]: 2022-05-13T23:48:46.677+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:48:46.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:48:46 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:48:47.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:48:46 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:48:47.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:48:47 smithi115 conmon[90393]: 2022-05-13T23:48:47.680+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:48:47.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:48:47 smithi115 ceph-mon[78593]: pgmap v982: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:48:47.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:48:47 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:48:48.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:48:47 smithi047 ceph-mon[77147]: pgmap v982: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:48:48.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:48:47 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:48:48.862 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:48:48 smithi115 conmon[90393]: 2022-05-13T23:48:48.666+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:48:48.863 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:48:48 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:48:48.863 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:48:48 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 1260 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:48:49.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:48:48 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:48:49.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:48:48 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 1260 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:48:49.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:48:49 smithi115 conmon[90393]: 2022-05-13T23:48:49.654+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:48:49.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:48:49 smithi115 ceph-mon[78593]: pgmap v983: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:48:49.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:48:49 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:48:49.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:48:49 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:48:49.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:48:49 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-13T23:48:50.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:48:49 smithi047 ceph-mon[77147]: pgmap v983: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:48:50.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:48:49 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:48:50.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:48:49 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:48:50.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:48:49 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-13T23:48:50.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:48:50 smithi115 conmon[90393]: 2022-05-13T23:48:50.650+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:48:50.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:48:50 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:48:51.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:48:50 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:48:51.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:48:51 smithi115 conmon[90393]: 2022-05-13T23:48:51.639+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:48:51.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:48:51 smithi115 ceph-mon[78593]: pgmap v984: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:48:51.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:48:51 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:48:52.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:48:51 smithi047 ceph-mon[77147]: pgmap v984: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:48:52.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:48:51 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:48:52.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:48:52 smithi115 conmon[90393]: 2022-05-13T23:48:52.631+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:48:52.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:48:52 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:48:53.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:48:52 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:48:53.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:48:53 smithi115 conmon[90393]: 2022-05-13T23:48:53.668+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:48:53.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:48:53 smithi115 ceph-mon[78593]: pgmap v985: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:48:53.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:48:53 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:48:53.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:48:53 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-13T23:48:53.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:48:53 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:48:53.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:48:53 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 1265 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:48:54.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:48:53 smithi047 ceph-mon[77147]: pgmap v985: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:48:54.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:48:53 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:48:54.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:48:53 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-13T23:48:54.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:48:53 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:48:54.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:48:53 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 1265 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:48:54.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:48:54 smithi115 conmon[90393]: 2022-05-13T23:48:54.621+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:48:54.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:48:54 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:48:55.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:48:54 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:48:55.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:48:55 smithi115 conmon[90393]: 2022-05-13T23:48:55.598+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:48:55.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:48:55 smithi115 ceph-mon[78593]: pgmap v986: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:48:55.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:48:55 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:48:56.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:48:55 smithi047 ceph-mon[77147]: pgmap v986: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:48:56.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:48:55 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:48:56.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:48:56 smithi115 conmon[90393]: 2022-05-13T23:48:56.648+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:48:56.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:48:56 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:48:57.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:48:56 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:48:57.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:48:57 smithi115 conmon[90393]: 2022-05-13T23:48:57.638+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:48:57.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:48:57 smithi115 ceph-mon[78593]: pgmap v987: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:48:57.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:48:57 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:48:57.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:48:57 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-13T23:48:57.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:48:57 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:48:57.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:48:57 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-13T23:48:57.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:48:57 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-13T23:48:57.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:48:57 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-13T23:48:57.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:48:57 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' 2022-05-13T23:48:57.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:48:57 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' 2022-05-13T23:48:58.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:48:57 smithi047 ceph-mon[77147]: pgmap v987: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:48:58.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:48:57 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:48:58.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:48:57 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-13T23:48:58.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:48:57 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:48:58.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:48:57 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-13T23:48:58.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:48:57 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-13T23:48:58.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:48:57 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-13T23:48:58.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:48:57 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' 2022-05-13T23:48:58.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:48:57 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' 2022-05-13T23:48:58.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:48:58 smithi115 conmon[90393]: 2022-05-13T23:48:58.619+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:48:58.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:48:58 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:48:58.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:48:58 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 1270 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:48:59.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:48:58 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:48:59.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:48:58 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 1270 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:48:59.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:48:59 smithi115 conmon[90393]: 2022-05-13T23:48:59.643+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:48:59.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:48:59 smithi115 ceph-mon[78593]: pgmap v988: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:48:59.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:48:59 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:49:00.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:48:59 smithi047 ceph-mon[77147]: pgmap v988: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:49:00.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:48:59 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:49:00.042 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:48:59 smithi047 conmon[78521]: 2022-05-13T23:48:59.760+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:49:00.042 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:48:59 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-13T23:49:00.042 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:48:59 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-13T23:49:00.042 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:48:59 smithi047 conmon[78521]: yield 2022-05-13T23:49:00.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:48:59 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-13T23:49:00.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:48:59 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-13T23:49:00.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:48:59 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-13T23:49:00.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:48:59 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-13T23:49:00.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:48:59 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-13T23:49:00.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:48:59 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-13T23:49:00.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:48:59 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-13T23:49:00.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:48:59 smithi047 conmon[78521]: raise exceptions[0] 2022-05-13T23:49:00.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:48:59 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-13T23:49:00.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:48:59 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-13T23:49:00.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:48:59 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-13T23:49:00.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:48:59 smithi047 conmon[78521]: return (yield from fut) 2022-05-13T23:49:00.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:48:59 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-13T23:49:00.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:48:59 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-13T23:49:00.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:48:59 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:49:01.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:49:00 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:49:01.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:49:00 smithi115 conmon[90393]: 2022-05-13T23:49:00.644+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:49:01.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:49:00 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:49:02.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:49:01 smithi047 ceph-mon[77147]: pgmap v989: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:49:02.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:49:01 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:49:02.041 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-13T23:49:02.042 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-13T23:49:02.042 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-13T23:49:02.042 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-13T23:49:02.042 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-13T23:49:02.042 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-13T23:49:02.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-13T23:49:02.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-13T23:49:02.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-13T23:49:02.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-13T23:49:02.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-13T23:49:02.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-13T23:49:02.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-13T23:49:02.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-13T23:49:02.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-13T23:49:02.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-13T23:49:02.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-13T23:49:02.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:49:02.045 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:49:01 smithi047 ceph-mon[77147]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-13T23:49:02.045 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:49:01 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:49:02.046 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:49:01 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-13T23:49:02.046 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:49:01 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:49:02.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:49:01 smithi115 conmon[90393]: 2022-05-13T23:49:01.644+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:49:02.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:49:01 smithi115 ceph-mon[78593]: pgmap v989: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:49:02.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:49:01 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:49:02.108 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-13T23:49:02.109 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-13T23:49:02.109 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-13T23:49:02.109 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-13T23:49:02.109 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-13T23:49:02.109 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-13T23:49:02.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-13T23:49:02.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-13T23:49:02.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-13T23:49:02.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-13T23:49:02.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-13T23:49:02.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-13T23:49:02.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-13T23:49:02.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-13T23:49:02.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-13T23:49:02.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-13T23:49:02.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-13T23:49:02.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:49:02.112 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:49:01 smithi115 ceph-mon[78593]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-13T23:49:02.112 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:49:01 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:49:02.113 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:49:01 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-13T23:49:02.113 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:49:01 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:49:02.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:49:02 smithi115 conmon[90393]: 2022-05-13T23:49:02.596+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:49:02.871 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:49:02 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:49:03.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:49:02 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:49:03.042 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:49:02 smithi047 conmon[78521]: 2022-05-13T23:49:02.832+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:49:03.042 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:49:02 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-13T23:49:03.042 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:49:02 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-13T23:49:03.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:49:02 smithi047 conmon[78521]: yield 2022-05-13T23:49:03.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:49:02 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-13T23:49:03.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:49:02 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-13T23:49:03.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:49:02 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-13T23:49:03.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:49:02 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-13T23:49:03.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:49:02 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-13T23:49:03.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:49:02 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-13T23:49:03.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:49:02 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-13T23:49:03.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:49:02 smithi047 conmon[78521]: raise exceptions[0] 2022-05-13T23:49:03.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:49:02 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-13T23:49:03.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:49:02 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-13T23:49:03.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:49:02 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-13T23:49:03.048 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:49:02 smithi047 conmon[78521]: return (yield from fut) 2022-05-13T23:49:03.048 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:49:02 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-13T23:49:03.048 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:49:02 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-13T23:49:03.049 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:49:02 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:49:03.049 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:49:02 smithi047 conmon[78521]: 2022-05-13T23:49:02.833+0000 7f5340f7a700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-13T23:49:03.050 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:49:02 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-13T23:49:03.050 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:49:02 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-13T23:49:03.050 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:49:02 smithi047 conmon[78521]: yield 2022-05-13T23:49:03.051 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:49:02 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-13T23:49:03.051 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:49:02 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-13T23:49:03.051 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:49:02 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-13T23:49:03.052 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:49:02 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-13T23:49:03.052 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:49:02 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-13T23:49:03.052 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:49:02 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-13T23:49:03.053 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:49:02 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-13T23:49:03.053 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:49:02 smithi047 conmon[78521]: raise exceptions[0] 2022-05-13T23:49:03.053 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:49:02 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-13T23:49:03.054 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:49:02 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-13T23:49:03.054 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:49:02 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-13T23:49:03.054 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:49:02 smithi047 conmon[78521]: return (yield from fut) 2022-05-13T23:49:03.054 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:49:02 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-13T23:49:03.054 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:49:02 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-13T23:49:03.055 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:49:02 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:49:03.055 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:49:02 smithi047 conmon[78521]: 2022-05-13T23:49:03.055 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:49:02 smithi047 conmon[78521]: During handling of the above exception, another exception occurred: 2022-05-13T23:49:03.056 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:49:02 smithi047 conmon[78521]: 2022-05-13T23:49:03.056 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:49:02 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-13T23:49:03.056 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:49:02 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-13T23:49:03.057 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:49:02 smithi047 conmon[78521]: return f(*arg) 2022-05-13T23:49:03.057 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:49:02 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-13T23:49:03.058 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:49:02 smithi047 conmon[78521]: self._write_client_files(client_files, host) 2022-05-13T23:49:03.058 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:49:02 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-13T23:49:03.059 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:49:02 smithi047 conmon[78521]: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-13T23:49:03.059 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:49:02 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-13T23:49:03.059 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:49:02 smithi047 conmon[78521]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-13T23:49:03.060 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:49:02 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-13T23:49:03.060 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:49:02 smithi047 conmon[78521]: return self.event_loop.get_result(coro) 2022-05-13T23:49:03.060 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:49:02 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-13T23:49:03.060 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:49:02 smithi047 conmon[78521]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-13T23:49:03.061 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:49:02 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-13T23:49:03.062 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:49:02 smithi047 conmon[78521]: return self.__get_result() 2022-05-13T23:49:03.062 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:49:02 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-13T23:49:03.062 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:49:02 smithi047 conmon[78521]: raise self._exception 2022-05-13T23:49:03.063 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:49:02 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-13T23:49:03.063 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:49:02 smithi047 conmon[78521]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-13T23:49:03.064 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:49:02 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-13T23:49:03.064 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:49:02 smithi047 conmon[78521]: conn = await self._remote_connection(host, addr) 2022-05-13T23:49:03.064 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:49:02 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-13T23:49:03.065 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:49:02 smithi047 conmon[78521]: raise 2022-05-13T23:49:03.065 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:49:02 smithi047 conmon[78521]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-13T23:49:03.065 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:49:02 smithi047 conmon[78521]: self.gen.throw(type, value, traceback) 2022-05-13T23:49:03.066 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:49:02 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-13T23:49:03.066 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:49:02 smithi047 conmon[78521]: raise OrchestratorError(msg) 2022-05-13T23:49:03.066 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:49:02 smithi047 conmon[78521]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:49:03.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:49:03 smithi115 conmon[90393]: 2022-05-13T23:49:03.560+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:49:03.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:49:03 smithi115 ceph-mon[78593]: pgmap v990: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:49:03.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:49:03 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:49:03.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:49:03 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:49:03.859 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-13T23:49:03.859 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-13T23:49:03.859 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-13T23:49:03.859 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-13T23:49:03.860 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-13T23:49:03.860 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-13T23:49:03.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-13T23:49:03.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-13T23:49:03.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-13T23:49:03.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-13T23:49:03.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-13T23:49:03.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-13T23:49:03.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-13T23:49:03.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-13T23:49:03.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-13T23:49:03.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-13T23:49:03.863 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-13T23:49:03.863 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:49:03.863 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:49:03 smithi115 ceph-mon[78593]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-13T23:49:03.863 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-13T23:49:03.863 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-13T23:49:03.864 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-13T23:49:03.864 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-13T23:49:03.864 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-13T23:49:03.864 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-13T23:49:03.864 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-13T23:49:03.865 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-13T23:49:03.865 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-13T23:49:03.865 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-13T23:49:03.865 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-13T23:49:03.866 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-13T23:49:03.866 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-13T23:49:03.866 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-13T23:49:03.866 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-13T23:49:03.866 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-13T23:49:03.867 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-13T23:49:03.867 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:49:03.867 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-13T23:49:03.867 INFO:journalctl@ceph.mon.c.smithi115.stdout: During handling of the above exception, another exception occurred: 2022-05-13T23:49:03.867 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-13T23:49:03.868 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-13T23:49:03.868 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-13T23:49:03.868 INFO:journalctl@ceph.mon.c.smithi115.stdout: return f(*arg) 2022-05-13T23:49:03.868 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-13T23:49:03.868 INFO:journalctl@ceph.mon.c.smithi115.stdout: self._write_client_files(client_files, host) 2022-05-13T23:49:03.869 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-13T23:49:03.869 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-13T23:49:03.869 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-13T23:49:03.869 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-13T23:49:03.870 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-13T23:49:03.870 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.event_loop.get_result(coro) 2022-05-13T23:49:03.870 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-13T23:49:03.870 INFO:journalctl@ceph.mon.c.smithi115.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-13T23:49:03.870 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-13T23:49:03.871 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.__get_result() 2022-05-13T23:49:03.871 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-13T23:49:03.871 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise self._exception 2022-05-13T23:49:03.871 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-13T23:49:03.871 INFO:journalctl@ceph.mon.c.smithi115.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-13T23:49:03.872 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-13T23:49:03.872 INFO:journalctl@ceph.mon.c.smithi115.stdout: conn = await self._remote_connection(host, addr) 2022-05-13T23:49:03.872 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-13T23:49:03.872 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise 2022-05-13T23:49:03.872 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-13T23:49:03.873 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.gen.throw(type, value, traceback) 2022-05-13T23:49:03.873 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-13T23:49:03.873 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OrchestratorError(msg) 2022-05-13T23:49:03.873 INFO:journalctl@ceph.mon.c.smithi115.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:49:03.874 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:49:03 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 1275 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:49:04.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:49:03 smithi047 ceph-mon[77147]: pgmap v990: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:49:04.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:49:03 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:49:04.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:49:03 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:49:04.042 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-13T23:49:04.042 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-13T23:49:04.042 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-13T23:49:04.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-13T23:49:04.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-13T23:49:04.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-13T23:49:04.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-13T23:49:04.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-13T23:49:04.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-13T23:49:04.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-13T23:49:04.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-13T23:49:04.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-13T23:49:04.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-13T23:49:04.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-13T23:49:04.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-13T23:49:04.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-13T23:49:04.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-13T23:49:04.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:49:04.046 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:49:03 smithi047 ceph-mon[77147]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-13T23:49:04.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-13T23:49:04.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-13T23:49:04.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-13T23:49:04.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-13T23:49:04.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-13T23:49:04.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-13T23:49:04.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-13T23:49:04.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-13T23:49:04.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-13T23:49:04.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-13T23:49:04.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-13T23:49:04.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-13T23:49:04.049 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-13T23:49:04.049 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-13T23:49:04.049 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-13T23:49:04.049 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-13T23:49:04.050 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-13T23:49:04.050 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:49:04.050 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-13T23:49:04.050 INFO:journalctl@ceph.mon.b.smithi047.stdout: During handling of the above exception, another exception occurred: 2022-05-13T23:49:04.050 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-13T23:49:04.051 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-13T23:49:04.051 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-13T23:49:04.051 INFO:journalctl@ceph.mon.b.smithi047.stdout: return f(*arg) 2022-05-13T23:49:04.051 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-13T23:49:04.052 INFO:journalctl@ceph.mon.b.smithi047.stdout: self._write_client_files(client_files, host) 2022-05-13T23:49:04.052 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-13T23:49:04.052 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-13T23:49:04.052 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-13T23:49:04.052 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-13T23:49:04.053 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-13T23:49:04.053 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.event_loop.get_result(coro) 2022-05-13T23:49:04.053 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-13T23:49:04.053 INFO:journalctl@ceph.mon.b.smithi047.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-13T23:49:04.053 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-13T23:49:04.054 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.__get_result() 2022-05-13T23:49:04.054 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-13T23:49:04.054 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise self._exception 2022-05-13T23:49:04.054 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-13T23:49:04.055 INFO:journalctl@ceph.mon.b.smithi047.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-13T23:49:04.055 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-13T23:49:04.055 INFO:journalctl@ceph.mon.b.smithi047.stdout: conn = await self._remote_connection(host, addr) 2022-05-13T23:49:04.055 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-13T23:49:04.055 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise 2022-05-13T23:49:04.056 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-13T23:49:04.056 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.gen.throw(type, value, traceback) 2022-05-13T23:49:04.056 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-13T23:49:04.056 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OrchestratorError(msg) 2022-05-13T23:49:04.056 INFO:journalctl@ceph.mon.b.smithi047.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:49:04.057 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:49:03 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 1275 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:49:04.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:49:04 smithi115 conmon[90393]: 2022-05-13T23:49:04.548+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:49:04.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:49:04 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:49:04.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:49:04 smithi115 ceph-mon[78593]: pgmap v991: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:49:05.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:49:04 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:49:05.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:49:04 smithi047 ceph-mon[77147]: pgmap v991: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:49:05.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:49:05 smithi115 conmon[90393]: 2022-05-13T23:49:05.589+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:49:05.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:49:05 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:49:06.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:49:05 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:49:07.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:49:06 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:49:07.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:49:06 smithi047 ceph-mon[77147]: pgmap v992: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:49:07.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:49:06 smithi115 conmon[90393]: 2022-05-13T23:49:06.631+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:49:07.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:49:06 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:49:07.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:49:06 smithi115 ceph-mon[78593]: pgmap v992: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:49:08.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:49:07 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:49:08.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:49:07 smithi115 conmon[90393]: 2022-05-13T23:49:07.658+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:49:08.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:49:07 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:49:09.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:49:08 smithi047 ceph-mon[77147]: pgmap v993: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:49:09.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:49:08 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:49:09.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:49:08 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 1280 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:49:09.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:49:08 smithi115 conmon[90393]: 2022-05-13T23:49:08.632+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:49:09.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:49:08 smithi115 ceph-mon[78593]: pgmap v993: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:49:09.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:49:08 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:49:09.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:49:08 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 1280 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:49:10.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:49:09 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:49:10.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:49:09 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:49:10.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:49:09 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-13T23:49:10.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:49:09 smithi115 conmon[90393]: 2022-05-13T23:49:09.665+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:49:10.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:49:09 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:49:10.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:49:09 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:49:10.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:49:09 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-13T23:49:11.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:49:10 smithi047 ceph-mon[77147]: pgmap v994: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:49:11.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:49:10 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:49:11.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:49:10 smithi115 conmon[90393]: 2022-05-13T23:49:10.672+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:49:11.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:49:10 smithi115 ceph-mon[78593]: pgmap v994: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:49:11.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:49:10 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:49:12.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:49:11 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:49:12.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:49:11 smithi115 conmon[90393]: 2022-05-13T23:49:11.661+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:49:12.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:49:11 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:49:13.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:49:12 smithi047 ceph-mon[77147]: pgmap v995: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:49:13.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:49:12 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:49:13.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:49:12 smithi115 conmon[90393]: 2022-05-13T23:49:12.677+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:49:13.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:49:12 smithi115 ceph-mon[78593]: pgmap v995: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:49:13.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:49:12 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:49:14.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:49:13 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:49:14.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:49:13 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-13T23:49:14.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:49:13 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:49:14.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:49:13 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-13T23:49:14.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:49:13 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-13T23:49:14.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:49:13 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 1285 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:49:14.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:49:13 smithi115 conmon[90393]: 2022-05-13T23:49:13.692+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:49:14.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:49:13 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:49:14.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:49:13 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-13T23:49:14.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:49:13 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:49:14.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:49:13 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-13T23:49:14.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:49:13 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-13T23:49:14.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:49:13 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 1285 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:49:15.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:49:14 smithi047 ceph-mon[77147]: pgmap v996: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:49:15.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:49:14 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:49:15.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:49:14 smithi115 ceph-mon[78593]: pgmap v996: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:49:15.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:49:14 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:49:15.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:49:14 smithi115 conmon[90393]: 2022-05-13T23:49:14.721+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:49:16.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:49:15 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:49:16.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:49:15 smithi115 conmon[90393]: 2022-05-13T23:49:15.690+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:49:16.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:49:15 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:49:17.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:49:16 smithi047 ceph-mon[77147]: pgmap v997: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:49:17.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:49:16 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:49:17.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:49:16 smithi115 conmon[90393]: 2022-05-13T23:49:16.705+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:49:17.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:49:16 smithi115 ceph-mon[78593]: pgmap v997: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:49:17.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:49:16 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:49:18.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:49:17 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:49:18.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:49:17 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-13T23:49:18.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:49:17 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:49:18.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:49:17 smithi115 conmon[90393]: 2022-05-13T23:49:17.685+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:49:18.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:49:17 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:49:18.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:49:17 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-13T23:49:18.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:49:17 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:49:19.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:49:18 smithi047 ceph-mon[77147]: pgmap v998: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:49:19.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:49:18 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:49:19.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:49:18 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 1290 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:49:19.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:49:18 smithi115 conmon[90393]: 2022-05-13T23:49:18.696+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:49:19.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:49:18 smithi115 ceph-mon[78593]: pgmap v998: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:49:19.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:49:18 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:49:19.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:49:18 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 1290 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:49:20.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:49:19 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:49:20.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:49:19 smithi115 conmon[90393]: 2022-05-13T23:49:19.677+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:49:20.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:49:19 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:49:21.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:49:20 smithi047 ceph-mon[77147]: pgmap v999: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:49:21.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:49:20 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:49:21.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:49:20 smithi115 conmon[90393]: 2022-05-13T23:49:20.637+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:49:21.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:49:20 smithi115 ceph-mon[78593]: pgmap v999: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:49:21.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:49:20 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:49:22.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:49:21 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:49:22.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:49:21 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:49:22.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:49:21 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-13T23:49:22.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:49:21 smithi115 conmon[90393]: 2022-05-13T23:49:21.618+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:49:22.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:49:21 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:49:22.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:49:21 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:49:22.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:49:21 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-13T23:49:23.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:49:22 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:49:23.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:49:22 smithi047 ceph-mon[77147]: pgmap v1000: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:49:23.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:49:22 smithi115 conmon[90393]: 2022-05-13T23:49:22.660+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:49:23.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:49:22 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:49:23.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:49:22 smithi115 ceph-mon[78593]: pgmap v1000: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:49:24.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:49:23 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:49:24.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:49:23 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 1295 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:49:24.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:49:23 smithi115 conmon[90393]: 2022-05-13T23:49:23.710+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:49:24.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:49:23 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:49:24.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:49:23 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 1295 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:49:25.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:49:24 smithi047 ceph-mon[77147]: pgmap v1001: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:49:25.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:49:24 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:49:25.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:49:24 smithi115 conmon[90393]: 2022-05-13T23:49:24.717+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:49:25.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:49:24 smithi115 ceph-mon[78593]: pgmap v1001: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:49:25.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:49:24 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:49:26.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:49:25 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:49:26.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:49:25 smithi115 conmon[90393]: 2022-05-13T23:49:25.674+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:49:26.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:49:25 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:49:27.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:49:26 smithi047 ceph-mon[77147]: pgmap v1002: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:49:27.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:49:26 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:49:27.113 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:49:26 smithi115 conmon[90393]: 2022-05-13T23:49:26.696+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:49:27.114 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:49:26 smithi115 ceph-mon[78593]: pgmap v1002: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:49:27.114 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:49:26 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:49:28.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:49:27 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:49:28.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:49:27 smithi115 conmon[90393]: 2022-05-13T23:49:27.656+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:49:28.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:49:27 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:49:29.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:49:28 smithi047 ceph-mon[77147]: pgmap v1003: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:49:29.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:49:28 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:49:29.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:49:28 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 1300 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:49:29.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:49:28 smithi115 conmon[90393]: 2022-05-13T23:49:28.658+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:49:29.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:49:28 smithi115 ceph-mon[78593]: pgmap v1003: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:49:29.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:49:28 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:49:29.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:49:28 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 1300 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:49:30.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:49:29 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:49:30.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:49:29 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:49:30.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:49:29 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-13T23:49:30.110 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:49:29 smithi115 conmon[90393]: 2022-05-13T23:49:29.644+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:49:30.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:49:29 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:49:30.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:49:29 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:49:30.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:49:29 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-13T23:49:31.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:49:30 smithi047 ceph-mon[77147]: pgmap v1004: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:49:31.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:49:30 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:49:31.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:49:30 smithi115 conmon[90393]: 2022-05-13T23:49:30.632+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:49:31.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:49:30 smithi115 ceph-mon[78593]: pgmap v1004: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:49:31.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:49:30 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:49:32.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:49:31 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:49:32.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:49:31 smithi115 conmon[90393]: 2022-05-13T23:49:31.615+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:49:32.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:49:31 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:49:33.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:49:32 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:49:33.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:49:32 smithi047 ceph-mon[77147]: pgmap v1005: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:49:33.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:49:32 smithi115 conmon[90393]: 2022-05-13T23:49:32.642+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:49:33.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:49:32 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:49:33.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:49:32 smithi115 ceph-mon[78593]: pgmap v1005: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:49:34.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:49:33 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:49:34.098 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:49:33 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:49:34.098 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:49:33 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-13T23:49:34.098 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:49:33 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 1305 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:49:34.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:49:33 smithi115 conmon[90393]: 2022-05-13T23:49:33.621+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:49:34.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:49:33 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:49:34.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:49:33 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:49:34.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:49:33 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-13T23:49:34.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:49:33 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 1305 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:49:35.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:49:34 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:49:35.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:49:34 smithi047 ceph-mon[77147]: pgmap v1006: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:49:35.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:49:34 smithi115 conmon[90393]: 2022-05-13T23:49:34.639+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:49:35.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:49:34 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:49:35.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:49:34 smithi115 ceph-mon[78593]: pgmap v1006: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:49:36.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:49:35 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:49:36.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:49:35 smithi115 conmon[90393]: 2022-05-13T23:49:35.638+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:49:36.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:49:35 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:49:37.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:49:36 smithi047 ceph-mon[77147]: pgmap v1007: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:49:37.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:49:36 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:49:37.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:49:36 smithi115 conmon[90393]: 2022-05-13T23:49:36.666+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:49:37.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:49:36 smithi115 ceph-mon[78593]: pgmap v1007: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:49:37.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:49:36 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:49:38.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:49:37 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:49:38.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:49:37 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:49:38.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:49:37 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-13T23:49:38.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:49:37 smithi115 conmon[90393]: 2022-05-13T23:49:37.658+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:49:38.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:49:37 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:49:38.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:49:37 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:49:38.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:49:37 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-13T23:49:39.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:49:38 smithi047 ceph-mon[77147]: pgmap v1008: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:49:39.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:49:38 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:49:39.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:49:38 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 1310 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:49:39.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:49:38 smithi115 conmon[90393]: 2022-05-13T23:49:38.702+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:49:39.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:49:38 smithi115 ceph-mon[78593]: pgmap v1008: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:49:39.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:49:38 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:49:39.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:49:38 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 1310 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:49:40.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:49:39 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:49:40.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:49:39 smithi115 conmon[90393]: 2022-05-13T23:49:39.688+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:49:40.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:49:39 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:49:41.046 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:49:40 smithi047 ceph-mon[77147]: pgmap v1009: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:49:41.046 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:49:40 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:49:41.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:49:40 smithi115 conmon[90393]: 2022-05-13T23:49:40.659+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:49:41.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:49:40 smithi115 ceph-mon[78593]: pgmap v1009: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:49:41.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:49:40 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:49:42.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:49:41 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:49:42.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:49:41 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:49:42.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:49:41 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-13T23:49:42.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:49:41 smithi115 conmon[90393]: 2022-05-13T23:49:41.704+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:49:42.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:49:41 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:49:42.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:49:41 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:49:42.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:49:41 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-13T23:49:43.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:49:42 smithi047 ceph-mon[77147]: pgmap v1010: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:49:43.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:49:42 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:49:43.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:49:42 smithi115 conmon[90393]: 2022-05-13T23:49:42.667+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:49:43.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:49:42 smithi115 ceph-mon[78593]: pgmap v1010: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:49:43.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:49:42 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:49:44.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:49:43 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:49:44.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:49:43 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 1315 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:49:44.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:49:43 smithi115 conmon[90393]: 2022-05-13T23:49:43.697+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:49:44.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:49:43 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:49:44.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:49:43 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 1315 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:49:45.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:49:44 smithi047 ceph-mon[77147]: pgmap v1011: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:49:45.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:49:44 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:49:45.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:49:44 smithi115 ceph-mon[78593]: pgmap v1011: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:49:45.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:49:44 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:49:45.111 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:49:44 smithi115 conmon[90393]: 2022-05-13T23:49:44.682+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:49:46.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:49:45 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:49:46.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:49:45 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:49:46.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:49:45 smithi115 conmon[90393]: 2022-05-13T23:49:45.723+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:49:47.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:49:46 smithi047 ceph-mon[77147]: pgmap v1012: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:49:47.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:49:46 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:49:47.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:49:46 smithi115 conmon[90393]: 2022-05-13T23:49:46.681+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:49:47.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:49:46 smithi115 ceph-mon[78593]: pgmap v1012: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:49:47.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:49:46 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:49:48.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:49:47 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:49:48.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:49:47 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:49:48.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:49:47 smithi115 conmon[90393]: 2022-05-13T23:49:47.716+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:49:49.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:49:48 smithi047 ceph-mon[77147]: pgmap v1013: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:49:49.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:49:48 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:49:49.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:49:48 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 1320 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:49:49.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:49:48 smithi115 conmon[90393]: 2022-05-13T23:49:48.692+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:49:49.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:49:48 smithi115 ceph-mon[78593]: pgmap v1013: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:49:49.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:49:48 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:49:49.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:49:48 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 1320 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:49:50.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:49:49 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:49:50.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:49:49 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:49:50.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:49:49 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-13T23:49:50.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:49:49 smithi115 conmon[90393]: 2022-05-13T23:49:49.657+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:49:50.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:49:49 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:49:50.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:49:49 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:49:50.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:49:49 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-13T23:49:51.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:49:50 smithi047 ceph-mon[77147]: pgmap v1014: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:49:51.045 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:49:50 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:49:51.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:49:50 smithi115 conmon[90393]: 2022-05-13T23:49:50.648+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:49:51.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:49:50 smithi115 ceph-mon[78593]: pgmap v1014: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:49:51.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:49:50 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:49:51.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:49:51 smithi115 conmon[90393]: 2022-05-13T23:49:51.601+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:49:51.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:49:51 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:49:52.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:49:51 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:49:53.110 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:49:52 smithi115 conmon[90393]: 2022-05-13T23:49:52.651+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:49:53.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:49:52 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:49:53.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:49:52 smithi115 ceph-mon[78593]: pgmap v1015: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:49:53.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:49:52 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:49:53.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:49:52 smithi047 ceph-mon[77147]: pgmap v1015: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:49:54.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:49:53 smithi115 conmon[90393]: 2022-05-13T23:49:53.637+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:49:54.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:49:53 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:49:54.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:49:53 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:49:54.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:49:53 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-13T23:49:54.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:49:53 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 1325 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:49:54.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:49:53 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:49:54.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:49:53 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:49:54.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:49:53 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-13T23:49:54.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:49:53 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 1325 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:49:55.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:49:54 smithi115 conmon[90393]: 2022-05-13T23:49:54.671+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:49:55.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:49:54 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:49:55.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:49:54 smithi115 ceph-mon[78593]: pgmap v1016: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:49:55.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:49:54 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:49:55.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:49:54 smithi047 ceph-mon[77147]: pgmap v1016: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:49:56.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:49:55 smithi115 conmon[90393]: 2022-05-13T23:49:55.690+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:49:56.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:49:55 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:49:56.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:49:55 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:49:57.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:49:56 smithi115 conmon[90393]: 2022-05-13T23:49:56.694+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:49:57.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:49:56 smithi115 ceph-mon[78593]: pgmap v1017: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:49:57.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:49:56 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:49:57.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:49:56 smithi047 ceph-mon[77147]: pgmap v1017: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:49:57.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:49:56 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:49:58.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:49:57 smithi115 conmon[90393]: 2022-05-13T23:49:57.715+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:49:58.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:49:57 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:49:58.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:49:57 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:49:58.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:49:57 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-13T23:49:58.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:49:57 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:49:58.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:49:57 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:49:58.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:49:57 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-13T23:49:59.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:49:58 smithi115 conmon[90393]: 2022-05-13T23:49:58.700+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:49:59.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:49:58 smithi115 ceph-mon[78593]: pgmap v1018: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:49:59.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:49:58 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:49:59.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:49:58 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 1330 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:49:59.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:49:58 smithi047 ceph-mon[77147]: pgmap v1018: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:49:59.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:49:58 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:49:59.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:49:58 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 1330 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:50:00.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:49:59 smithi115 conmon[90393]: 2022-05-13T23:49:59.705+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:50:00.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:49:59 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:50:00.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:49:59 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:50:00.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:50:00 smithi047 conmon[77123]: 2022-05-13T23:49:59.998+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : Health detail: HEALTH_ERR 1 filesystem is degraded; 1 mds daemon damaged; 1 MDSs are read only; 1 MDSs report slow metadata IOs; 1/3 mons down, quorum b,c; Reduced data availability: 1 pg inactive; 9 slow ops, oldest one blocked for 1335 sec, osd.5 has slow ops 2022-05-13T23:50:00.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:50:00 smithi047 conmon[77123]: 2022-05-13T23:49:59.998+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : [WRN] FS_DEGRADED: 1 filesystem is degraded 2022-05-13T23:50:00.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:50:00 smithi047 conmon[77123]: 2022-05-13T23:49:59.998+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : fs cephfs is degraded 2022-05-13T23:50:00.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:50:00 smithi047 conmon[77123]: 2022-05-13T23:49:59.998+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : [ERR] MDS_DAMAGE: 1 mds daemon damaged 2022-05-13T23:50:00.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:50:00 smithi047 conmon[77123]: 2022-05-13T23:49:59.998+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : fs cephfs mds.2 is damaged 2022-05-13T23:50:00.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:50:00 smithi047 conmon[77123]: 2022-05-13T23:49:59.998+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : [WRN] MDS_READ_ONLY: 1 MDSs are read only 2022-05-13T23:50:00.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:50:00 smithi047 conmon[77123]: 2022-05-13T23:49:59.998+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : mds.l(mds.0): MDS in read-only mode 2022-05-13T23:50:00.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:50:00 smithi047 conmon[77123]: 2022-05-13T23:49:59.998+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : [WRN] MDS_SLOW_METADATA_IO: 1 MDSs report slow metadata IOs 2022-05-13T23:50:00.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:50:00 smithi047 conmon[77123]: 2022-05-13T23:49:59.998+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : mds.l(mds.0): 4 slow metadata IOs are blocked > 180 secs, oldest blocked for 1633 secs 2022-05-13T23:50:00.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:50:00 smithi047 conmon[77123]: 2022-05-13T23:49:59.998+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : [WRN] MON_DOWN: 1/3 mons down, quorum b,c 2022-05-13T23:50:00.295 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:50:00 smithi047 conmon[77123]: 2022-05-13T23:49:59.998+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : mon.a (rank 0) addr [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] is down (out of quorum) 2022-05-13T23:50:00.295 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:50:00 smithi047 conmon[77123]: 2022-05-13T23:49:59.998+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : [WRN] PG_AVAILABILITY: Reduced data availability: 1 pg inactive 2022-05-13T23:50:00.295 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:50:00 smithi047 conmon[77123]: 2022-05-13T23:49:59.998+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : pg 2.1 is stuck inactive for 34m, current state unknown, last acting [] 2022-05-13T23:50:00.295 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:50:00 smithi047 conmon[77123]: 2022-05-13T23:49:59.998+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : [WRN] SLOW_OPS: 9 slow ops, oldest one blocked for 1335 sec, osd.5 has slow ops 2022-05-13T23:50:01.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:50:00 smithi115 conmon[90393]: 2022-05-13T23:50:00.712+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:50:01.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:50:00 smithi115 ceph-mon[78593]: pgmap v1019: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:50:01.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:50:00 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:50:01.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:50:00 smithi115 ceph-mon[78593]: Health detail: HEALTH_ERR 1 filesystem is degraded; 1 mds daemon damaged; 1 MDSs are read only; 1 MDSs report slow metadata IOs; 1/3 mons down, quorum b,c; Reduced data availability: 1 pg inactive; 9 slow ops, oldest one blocked for 1335 sec, osd.5 has slow ops 2022-05-13T23:50:01.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:50:00 smithi115 ceph-mon[78593]: [WRN] FS_DEGRADED: 1 filesystem is degraded 2022-05-13T23:50:01.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:50:00 smithi115 ceph-mon[78593]: fs cephfs is degraded 2022-05-13T23:50:01.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:50:00 smithi115 ceph-mon[78593]: [ERR] MDS_DAMAGE: 1 mds daemon damaged 2022-05-13T23:50:01.116 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:50:00 smithi115 ceph-mon[78593]: fs cephfs mds.2 is damaged 2022-05-13T23:50:01.117 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:50:00 smithi115 ceph-mon[78593]: [WRN] MDS_READ_ONLY: 1 MDSs are read only 2022-05-13T23:50:01.117 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:50:00 smithi115 ceph-mon[78593]: mds.l(mds.0): MDS in read-only mode 2022-05-13T23:50:01.117 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:50:00 smithi115 ceph-mon[78593]: [WRN] MDS_SLOW_METADATA_IO: 1 MDSs report slow metadata IOs 2022-05-13T23:50:01.118 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:50:00 smithi115 ceph-mon[78593]: mds.l(mds.0): 4 slow metadata IOs are blocked > 180 secs, oldest blocked for 1633 secs 2022-05-13T23:50:01.118 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:50:00 smithi115 ceph-mon[78593]: [WRN] MON_DOWN: 1/3 mons down, quorum b,c 2022-05-13T23:50:01.118 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:50:00 smithi115 ceph-mon[78593]: mon.a (rank 0) addr [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] is down (out of quorum) 2022-05-13T23:50:01.118 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:50:00 smithi115 ceph-mon[78593]: [WRN] PG_AVAILABILITY: Reduced data availability: 1 pg inactive 2022-05-13T23:50:01.118 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:50:00 smithi115 ceph-mon[78593]: pg 2.1 is stuck inactive for 34m, current state unknown, last acting [] 2022-05-13T23:50:01.119 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:50:00 smithi115 ceph-mon[78593]: [WRN] SLOW_OPS: 9 slow ops, oldest one blocked for 1335 sec, osd.5 has slow ops 2022-05-13T23:50:01.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:50:00 smithi047 ceph-mon[77147]: pgmap v1019: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:50:01.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:50:00 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:50:01.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:50:00 smithi047 ceph-mon[77147]: Health detail: HEALTH_ERR 1 filesystem is degraded; 1 mds daemon damaged; 1 MDSs are read only; 1 MDSs report slow metadata IOs; 1/3 mons down, quorum b,c; Reduced data availability: 1 pg inactive; 9 slow ops, oldest one blocked for 1335 sec, osd.5 has slow ops 2022-05-13T23:50:01.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:50:00 smithi047 ceph-mon[77147]: [WRN] FS_DEGRADED: 1 filesystem is degraded 2022-05-13T23:50:01.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:50:00 smithi047 ceph-mon[77147]: fs cephfs is degraded 2022-05-13T23:50:01.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:50:00 smithi047 ceph-mon[77147]: [ERR] MDS_DAMAGE: 1 mds daemon damaged 2022-05-13T23:50:01.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:50:00 smithi047 ceph-mon[77147]: fs cephfs mds.2 is damaged 2022-05-13T23:50:01.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:50:00 smithi047 ceph-mon[77147]: [WRN] MDS_READ_ONLY: 1 MDSs are read only 2022-05-13T23:50:01.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:50:00 smithi047 ceph-mon[77147]: mds.l(mds.0): MDS in read-only mode 2022-05-13T23:50:01.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:50:00 smithi047 ceph-mon[77147]: [WRN] MDS_SLOW_METADATA_IO: 1 MDSs report slow metadata IOs 2022-05-13T23:50:01.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:50:00 smithi047 ceph-mon[77147]: mds.l(mds.0): 4 slow metadata IOs are blocked > 180 secs, oldest blocked for 1633 secs 2022-05-13T23:50:01.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:50:00 smithi047 ceph-mon[77147]: [WRN] MON_DOWN: 1/3 mons down, quorum b,c 2022-05-13T23:50:01.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:50:00 smithi047 ceph-mon[77147]: mon.a (rank 0) addr [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] is down (out of quorum) 2022-05-13T23:50:01.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:50:00 smithi047 ceph-mon[77147]: [WRN] PG_AVAILABILITY: Reduced data availability: 1 pg inactive 2022-05-13T23:50:01.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:50:00 smithi047 ceph-mon[77147]: pg 2.1 is stuck inactive for 34m, current state unknown, last acting [] 2022-05-13T23:50:01.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:50:00 smithi047 ceph-mon[77147]: [WRN] SLOW_OPS: 9 slow ops, oldest one blocked for 1335 sec, osd.5 has slow ops 2022-05-13T23:50:02.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:50:01 smithi115 conmon[90393]: 2022-05-13T23:50:01.753+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:50:02.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:50:01 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:50:02.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:50:01 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:50:02.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:50:01 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-13T23:50:02.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:50:01 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:50:02.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:50:01 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:50:02.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:50:01 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-13T23:50:03.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:50:02 smithi115 conmon[90393]: 2022-05-13T23:50:02.789+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:50:03.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:50:02 smithi115 ceph-mon[78593]: pgmap v1020: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:50:03.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:50:02 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:50:03.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:50:02 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-13T23:50:03.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:50:02 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-13T23:50:03.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:50:02 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-13T23:50:03.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:50:02 smithi047 ceph-mon[77147]: pgmap v1020: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:50:03.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:50:02 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:50:03.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:50:02 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-13T23:50:03.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:50:02 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-13T23:50:03.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:50:02 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-13T23:50:04.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:50:03 smithi115 conmon[90393]: 2022-05-13T23:50:03.774+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:50:04.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:50:03 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:50:04.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:50:03 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' 2022-05-13T23:50:04.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:50:03 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' 2022-05-13T23:50:04.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:50:03 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 1335 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:50:04.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:50:03 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:50:04.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:50:03 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' 2022-05-13T23:50:04.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:50:03 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' 2022-05-13T23:50:04.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:50:03 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 1335 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:50:05.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:50:04 smithi115 conmon[90393]: 2022-05-13T23:50:04.772+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:50:05.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:50:04 smithi115 ceph-mon[78593]: pgmap v1021: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:50:05.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:50:04 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:50:05.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:50:04 smithi047 ceph-mon[77147]: pgmap v1021: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:50:05.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:50:04 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:50:06.100 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:50:05 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:50:06.100 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:50:05 smithi047 conmon[78521]: 2022-05-13T23:50:05.935+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:50:06.101 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:50:05 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-13T23:50:06.101 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:50:05 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-13T23:50:06.101 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:50:05 smithi047 conmon[78521]: yield 2022-05-13T23:50:06.102 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:50:05 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-13T23:50:06.102 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:50:05 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-13T23:50:06.102 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:50:05 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-13T23:50:06.102 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:50:05 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-13T23:50:06.103 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:50:05 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-13T23:50:06.103 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:50:05 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-13T23:50:06.103 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:50:05 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-13T23:50:06.103 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:50:05 smithi047 conmon[78521]: raise exceptions[0] 2022-05-13T23:50:06.103 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:50:05 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-13T23:50:06.104 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:50:05 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-13T23:50:06.104 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:50:05 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-13T23:50:06.104 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:50:05 smithi047 conmon[78521]: return (yield from fut) 2022-05-13T23:50:06.104 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:50:05 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-13T23:50:06.105 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:50:05 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-13T23:50:06.105 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:50:05 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:50:06.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:50:05 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:50:06.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:50:05 smithi115 conmon[90393]: 2022-05-13T23:50:05.775+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:50:07.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:50:06 smithi115 conmon[90393]: 2022-05-13T23:50:06.823+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:50:07.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:50:06 smithi115 ceph-mon[78593]: pgmap v1022: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:50:07.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:50:06 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:50:07.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:50:06 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' 2022-05-13T23:50:07.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:50:06 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:50:07.109 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-13T23:50:07.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-13T23:50:07.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-13T23:50:07.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-13T23:50:07.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-13T23:50:07.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-13T23:50:07.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-13T23:50:07.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-13T23:50:07.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-13T23:50:07.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-13T23:50:07.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-13T23:50:07.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-13T23:50:07.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-13T23:50:07.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-13T23:50:07.113 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-13T23:50:07.113 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-13T23:50:07.113 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-13T23:50:07.113 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:50:07.113 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:50:06 smithi115 ceph-mon[78593]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-13T23:50:07.114 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:50:06 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' 2022-05-13T23:50:07.114 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:50:06 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-05-13T23:50:07.114 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:50:06 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-05-13T23:50:07.114 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:50:06 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-05-13T23:50:07.115 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:50:06 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-05-13T23:50:07.115 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:50:06 smithi115 ceph-mon[78593]: Adjusting osd_memory_target on smithi115 to 197.0M 2022-05-13T23:50:07.115 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:50:06 smithi115 ceph-mon[78593]: Unable to set osd_memory_target on smithi115 to 206629785: error parsing value: Value '206629785' is below minimum 939524096 2022-05-13T23:50:07.115 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:50:06 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-05-13T23:50:07.116 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:50:06 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-05-13T23:50:07.116 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:50:06 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-05-13T23:50:07.117 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:50:06 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-05-13T23:50:07.117 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:50:06 smithi115 ceph-mon[78593]: Adjusting osd_memory_target on smithi047 to 198.1M 2022-05-13T23:50:07.117 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:50:06 smithi115 ceph-mon[78593]: Unable to set osd_memory_target on smithi047 to 207755878: error parsing value: Value '207755878' is below minimum 939524096 2022-05-13T23:50:07.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:50:06 smithi047 ceph-mon[77147]: pgmap v1022: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:50:07.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:50:06 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:50:07.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:50:06 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' 2022-05-13T23:50:07.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:50:06 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:50:07.292 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-13T23:50:07.292 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-13T23:50:07.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-13T23:50:07.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-13T23:50:07.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-13T23:50:07.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-13T23:50:07.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-13T23:50:07.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-13T23:50:07.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-13T23:50:07.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-13T23:50:07.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-13T23:50:07.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-13T23:50:07.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-13T23:50:07.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-13T23:50:07.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-13T23:50:07.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-13T23:50:07.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-13T23:50:07.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:50:07.297 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:50:06 smithi047 ceph-mon[77147]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-13T23:50:07.298 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:50:06 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' 2022-05-13T23:50:07.298 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:50:06 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-05-13T23:50:07.298 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:50:06 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-05-13T23:50:07.298 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:50:06 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-05-13T23:50:07.299 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:50:06 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-05-13T23:50:07.299 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:50:06 smithi047 ceph-mon[77147]: Adjusting osd_memory_target on smithi115 to 197.0M 2022-05-13T23:50:07.299 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:50:06 smithi047 ceph-mon[77147]: Unable to set osd_memory_target on smithi115 to 206629785: error parsing value: Value '206629785' is below minimum 939524096 2022-05-13T23:50:07.299 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:50:06 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-05-13T23:50:07.299 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:50:06 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-05-13T23:50:07.300 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:50:06 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-05-13T23:50:07.300 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:50:06 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-05-13T23:50:07.300 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:50:06 smithi047 ceph-mon[77147]: Adjusting osd_memory_target on smithi047 to 198.1M 2022-05-13T23:50:07.300 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:50:06 smithi047 ceph-mon[77147]: Unable to set osd_memory_target on smithi047 to 207755878: error parsing value: Value '207755878' is below minimum 939524096 2022-05-13T23:50:08.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:50:07 smithi115 conmon[90393]: 2022-05-13T23:50:07.781+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:50:08.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:50:07 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:50:08.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:50:07 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:50:09.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:50:08 smithi115 conmon[90393]: 2022-05-13T23:50:08.796+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:50:09.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:50:08 smithi115 ceph-mon[78593]: pgmap v1023: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:50:09.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:50:08 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:50:09.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:50:08 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 1340 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:50:09.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:50:08 smithi047 ceph-mon[77147]: pgmap v1023: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:50:09.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:50:08 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:50:09.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:50:08 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 1340 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:50:09.292 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:50:09 smithi047 conmon[78521]: 2022-05-13T23:50:09.007+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:50:09.293 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:50:09 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-13T23:50:09.293 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:50:09 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-13T23:50:09.293 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:50:09 smithi047 conmon[78521]: yield 2022-05-13T23:50:09.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:50:09 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-13T23:50:09.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:50:09 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-13T23:50:09.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:50:09 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-13T23:50:09.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:50:09 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-13T23:50:09.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:50:09 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-13T23:50:09.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:50:09 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-13T23:50:09.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:50:09 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-13T23:50:09.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:50:09 smithi047 conmon[78521]: raise exceptions[0] 2022-05-13T23:50:09.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:50:09 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-13T23:50:09.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:50:09 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-13T23:50:09.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:50:09 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-13T23:50:09.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:50:09 smithi047 conmon[78521]: return (yield from fut) 2022-05-13T23:50:09.297 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:50:09 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-13T23:50:09.297 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:50:09 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-13T23:50:09.297 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:50:09 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:50:09.298 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:50:09 smithi047 conmon[78521]: 2022-05-13T23:50:09.008+0000 7f5340779700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-13T23:50:09.298 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:50:09 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-13T23:50:09.298 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:50:09 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-13T23:50:09.299 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:50:09 smithi047 conmon[78521]: yield 2022-05-13T23:50:09.299 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:50:09 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-13T23:50:09.299 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:50:09 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-13T23:50:09.299 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:50:09 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-13T23:50:09.300 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:50:09 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-13T23:50:09.300 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:50:09 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-13T23:50:09.300 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:50:09 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-13T23:50:09.300 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:50:09 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-13T23:50:09.300 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:50:09 smithi047 conmon[78521]: raise exceptions[0] 2022-05-13T23:50:09.301 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:50:09 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-13T23:50:09.301 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:50:09 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-13T23:50:09.301 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:50:09 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-13T23:50:09.301 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:50:09 smithi047 conmon[78521]: return (yield from fut) 2022-05-13T23:50:09.302 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:50:09 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-13T23:50:09.302 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:50:09 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-13T23:50:09.302 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:50:09 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:50:09.302 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:50:09 smithi047 conmon[78521]: 2022-05-13T23:50:09.303 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:50:09 smithi047 conmon[78521]: During handling of the above exception, another exception occurred: 2022-05-13T23:50:09.303 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:50:09 smithi047 conmon[78521]: 2022-05-13T23:50:09.303 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:50:09 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-13T23:50:09.303 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:50:09 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-13T23:50:09.303 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:50:09 smithi047 conmon[78521]: return f(*arg) 2022-05-13T23:50:09.304 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:50:09 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-13T23:50:09.304 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:50:09 smithi047 conmon[78521]: self._write_client_files(client_files, host) 2022-05-13T23:50:09.304 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:50:09 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-13T23:50:09.304 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:50:09 smithi047 conmon[78521]: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-13T23:50:09.304 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:50:09 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-13T23:50:09.305 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:50:09 smithi047 conmon[78521]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-13T23:50:09.305 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:50:09 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-13T23:50:09.305 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:50:09 smithi047 conmon[78521]: return self.event_loop.get_result(coro) 2022-05-13T23:50:09.305 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:50:09 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-13T23:50:09.306 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:50:09 smithi047 conmon[78521]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-13T23:50:09.306 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:50:09 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-13T23:50:09.306 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:50:09 smithi047 conmon[78521]: return self.__get_result() 2022-05-13T23:50:09.306 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:50:09 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-13T23:50:09.307 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:50:09 smithi047 conmon[78521]: raise self._exception 2022-05-13T23:50:09.307 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:50:09 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-13T23:50:09.307 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:50:09 smithi047 conmon[78521]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-13T23:50:09.307 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:50:09 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-13T23:50:09.307 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:50:09 smithi047 conmon[78521]: conn = await self._remote_connection(host, addr) 2022-05-13T23:50:09.308 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:50:09 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-13T23:50:09.308 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:50:09 smithi047 conmon[78521]: raise 2022-05-13T23:50:09.308 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:50:09 smithi047 conmon[78521]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-13T23:50:09.308 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:50:09 smithi047 conmon[78521]: self.gen.throw(type, value, traceback) 2022-05-13T23:50:09.309 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:50:09 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-13T23:50:09.309 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:50:09 smithi047 conmon[78521]: raise OrchestratorError(msg) 2022-05-13T23:50:09.309 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:50:09 smithi047 conmon[78521]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:50:10.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:50:09 smithi115 conmon[90393]: 2022-05-13T23:50:09.777+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:50:10.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:50:09 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:50:10.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:50:09 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:50:10.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:50:09 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-13T23:50:10.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:50:09 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:50:10.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-13T23:50:10.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-13T23:50:10.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-13T23:50:10.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-13T23:50:10.113 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-13T23:50:10.113 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-13T23:50:10.114 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-13T23:50:10.114 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-13T23:50:10.114 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-13T23:50:10.115 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-13T23:50:10.115 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-13T23:50:10.115 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-13T23:50:10.116 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-13T23:50:10.116 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-13T23:50:10.116 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-13T23:50:10.117 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-13T23:50:10.117 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-13T23:50:10.117 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:50:10.118 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:50:09 smithi115 ceph-mon[78593]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-13T23:50:10.118 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-13T23:50:10.118 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-13T23:50:10.119 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-13T23:50:10.119 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-13T23:50:10.119 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-13T23:50:10.119 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-13T23:50:10.119 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-13T23:50:10.120 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-13T23:50:10.120 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-13T23:50:10.120 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-13T23:50:10.120 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-13T23:50:10.121 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-13T23:50:10.121 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-13T23:50:10.121 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-13T23:50:10.122 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-13T23:50:10.123 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-13T23:50:10.123 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-13T23:50:10.124 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:50:10.124 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-13T23:50:10.125 INFO:journalctl@ceph.mon.c.smithi115.stdout: During handling of the above exception, another exception occurred: 2022-05-13T23:50:10.125 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-13T23:50:10.125 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-13T23:50:10.125 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-13T23:50:10.126 INFO:journalctl@ceph.mon.c.smithi115.stdout: return f(*arg) 2022-05-13T23:50:10.126 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-13T23:50:10.126 INFO:journalctl@ceph.mon.c.smithi115.stdout: self._write_client_files(client_files, host) 2022-05-13T23:50:10.126 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-13T23:50:10.127 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-13T23:50:10.127 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-13T23:50:10.127 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-13T23:50:10.127 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-13T23:50:10.127 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.event_loop.get_result(coro) 2022-05-13T23:50:10.128 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-13T23:50:10.128 INFO:journalctl@ceph.mon.c.smithi115.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-13T23:50:10.128 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-13T23:50:10.128 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.__get_result() 2022-05-13T23:50:10.129 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-13T23:50:10.129 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise self._exception 2022-05-13T23:50:10.129 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-13T23:50:10.129 INFO:journalctl@ceph.mon.c.smithi115.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-13T23:50:10.130 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-13T23:50:10.130 INFO:journalctl@ceph.mon.c.smithi115.stdout: conn = await self._remote_connection(host, addr) 2022-05-13T23:50:10.130 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-13T23:50:10.130 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise 2022-05-13T23:50:10.130 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-13T23:50:10.131 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.gen.throw(type, value, traceback) 2022-05-13T23:50:10.131 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-13T23:50:10.131 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OrchestratorError(msg) 2022-05-13T23:50:10.131 INFO:journalctl@ceph.mon.c.smithi115.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:50:10.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:50:09 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:50:10.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:50:09 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:50:10.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:50:09 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-13T23:50:10.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:50:09 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:50:10.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-13T23:50:10.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-13T23:50:10.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-13T23:50:10.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-13T23:50:10.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-13T23:50:10.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-13T23:50:10.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-13T23:50:10.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-13T23:50:10.299 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-13T23:50:10.300 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-13T23:50:10.300 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-13T23:50:10.301 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-13T23:50:10.301 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-13T23:50:10.301 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-13T23:50:10.301 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-13T23:50:10.302 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-13T23:50:10.302 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-13T23:50:10.302 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:50:10.302 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:50:09 smithi047 ceph-mon[77147]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-13T23:50:10.303 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-13T23:50:10.303 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-13T23:50:10.303 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-13T23:50:10.303 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-13T23:50:10.304 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-13T23:50:10.304 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-13T23:50:10.304 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-13T23:50:10.304 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-13T23:50:10.305 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-13T23:50:10.305 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-13T23:50:10.305 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-13T23:50:10.305 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-13T23:50:10.306 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-13T23:50:10.306 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-13T23:50:10.306 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-13T23:50:10.306 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-13T23:50:10.307 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-13T23:50:10.307 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:50:10.307 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-13T23:50:10.307 INFO:journalctl@ceph.mon.b.smithi047.stdout: During handling of the above exception, another exception occurred: 2022-05-13T23:50:10.308 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-13T23:50:10.308 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-13T23:50:10.308 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-13T23:50:10.308 INFO:journalctl@ceph.mon.b.smithi047.stdout: return f(*arg) 2022-05-13T23:50:10.309 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-13T23:50:10.309 INFO:journalctl@ceph.mon.b.smithi047.stdout: self._write_client_files(client_files, host) 2022-05-13T23:50:10.309 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-13T23:50:10.309 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-13T23:50:10.310 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-13T23:50:10.310 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-13T23:50:10.310 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-13T23:50:10.310 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.event_loop.get_result(coro) 2022-05-13T23:50:10.311 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-13T23:50:10.311 INFO:journalctl@ceph.mon.b.smithi047.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-13T23:50:10.311 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-13T23:50:10.312 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.__get_result() 2022-05-13T23:50:10.312 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-13T23:50:10.312 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise self._exception 2022-05-13T23:50:10.313 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-13T23:50:10.313 INFO:journalctl@ceph.mon.b.smithi047.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-13T23:50:10.314 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-13T23:50:10.314 INFO:journalctl@ceph.mon.b.smithi047.stdout: conn = await self._remote_connection(host, addr) 2022-05-13T23:50:10.314 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-13T23:50:10.315 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise 2022-05-13T23:50:10.315 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-13T23:50:10.316 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.gen.throw(type, value, traceback) 2022-05-13T23:50:10.316 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-13T23:50:10.317 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OrchestratorError(msg) 2022-05-13T23:50:10.317 INFO:journalctl@ceph.mon.b.smithi047.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:50:11.112 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:50:10 smithi115 conmon[90393]: 2022-05-13T23:50:10.737+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:50:11.112 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:50:10 smithi115 ceph-mon[78593]: pgmap v1024: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:50:11.113 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:50:10 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:50:11.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:50:10 smithi047 ceph-mon[77147]: pgmap v1024: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:50:11.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:50:10 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:50:12.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:50:11 smithi115 conmon[90393]: 2022-05-13T23:50:11.711+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:50:12.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:50:11 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:50:12.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:50:11 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:50:13.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:50:12 smithi115 conmon[90393]: 2022-05-13T23:50:12.721+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:50:13.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:50:12 smithi115 ceph-mon[78593]: pgmap v1025: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:50:13.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:50:12 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:50:13.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:50:12 smithi047 ceph-mon[77147]: pgmap v1025: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:50:13.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:50:12 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:50:14.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:50:13 smithi115 conmon[90393]: 2022-05-13T23:50:13.754+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:50:14.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:50:13 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:50:14.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:50:13 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:50:14.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:50:13 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-13T23:50:14.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:50:13 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-13T23:50:14.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:50:13 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-13T23:50:14.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:50:13 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 1345 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:50:14.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:50:13 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:50:14.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:50:13 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:50:14.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:50:13 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-13T23:50:14.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:50:13 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-13T23:50:14.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:50:13 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-13T23:50:14.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:50:13 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 1345 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:50:15.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:50:14 smithi115 conmon[90393]: 2022-05-13T23:50:14.736+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:50:15.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:50:14 smithi115 ceph-mon[78593]: pgmap v1026: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:50:15.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:50:14 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:50:15.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:50:14 smithi047 ceph-mon[77147]: pgmap v1026: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:50:15.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:50:14 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:50:16.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:50:15 smithi115 conmon[90393]: 2022-05-13T23:50:15.716+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:50:16.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:50:15 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:50:16.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:50:15 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:50:17.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:50:16 smithi115 conmon[90393]: 2022-05-13T23:50:16.699+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:50:17.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:50:16 smithi115 ceph-mon[78593]: pgmap v1027: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:50:17.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:50:16 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:50:17.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:50:16 smithi047 ceph-mon[77147]: pgmap v1027: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:50:17.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:50:16 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:50:17.912 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:50:17 smithi115 conmon[90393]: 2022-05-13T23:50:17.658+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:50:18.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:50:17 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:50:18.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:50:17 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:50:18.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:50:17 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-13T23:50:18.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:50:17 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:50:18.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:50:17 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:50:18.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:50:17 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-13T23:50:18.913 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:50:18 smithi115 conmon[90393]: 2022-05-13T23:50:18.610+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:50:19.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:50:18 smithi047 ceph-mon[77147]: pgmap v1028: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:50:19.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:50:18 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:50:19.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:50:18 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 1350 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:50:19.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:50:18 smithi115 ceph-mon[78593]: pgmap v1028: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:50:19.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:50:18 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:50:19.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:50:18 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 1350 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:50:19.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:50:19 smithi115 conmon[90393]: 2022-05-13T23:50:19.563+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:50:20.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:50:19 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:50:20.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:50:19 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:50:20.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:50:20 smithi115 conmon[90393]: 2022-05-13T23:50:20.540+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:50:21.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:50:20 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:50:21.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:50:20 smithi047 ceph-mon[77147]: pgmap v1029: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:50:21.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:50:20 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:50:21.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:50:20 smithi115 ceph-mon[78593]: pgmap v1029: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:50:21.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:50:21 smithi115 conmon[90393]: 2022-05-13T23:50:21.575+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:50:22.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:50:21 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:50:22.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:50:21 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:50:22.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:50:21 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-13T23:50:22.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:50:21 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:50:22.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:50:21 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:50:22.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:50:21 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-13T23:50:22.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:50:22 smithi115 conmon[90393]: 2022-05-13T23:50:22.581+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:50:23.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:50:22 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:50:23.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:50:22 smithi047 ceph-mon[77147]: pgmap v1030: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:50:23.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:50:22 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:50:23.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:50:22 smithi115 ceph-mon[78593]: pgmap v1030: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:50:23.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:50:23 smithi115 conmon[90393]: 2022-05-13T23:50:23.585+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:50:24.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:50:23 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:50:24.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:50:23 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 1355 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:50:24.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:50:23 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:50:24.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:50:23 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 1355 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:50:24.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:50:24 smithi115 conmon[90393]: 2022-05-13T23:50:24.589+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:50:25.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:50:24 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:50:25.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:50:24 smithi047 ceph-mon[77147]: pgmap v1031: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:50:25.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:50:24 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:50:25.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:50:24 smithi115 ceph-mon[78593]: pgmap v1031: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:50:25.935 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:50:25 smithi115 conmon[90393]: 2022-05-13T23:50:25.609+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:50:26.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:50:25 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:50:26.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:50:25 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:50:26.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:50:26 smithi115 conmon[90393]: 2022-05-13T23:50:26.605+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:50:27.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:50:26 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:50:27.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:50:26 smithi047 ceph-mon[77147]: pgmap v1032: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:50:27.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:50:26 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:50:27.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:50:26 smithi115 ceph-mon[78593]: pgmap v1032: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:50:27.944 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:50:27 smithi115 conmon[90393]: 2022-05-13T23:50:27.638+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:50:28.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:50:27 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:50:28.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:50:27 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:50:28.944 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:50:28 smithi115 conmon[90393]: 2022-05-13T23:50:28.610+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:50:29.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:50:28 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:50:29.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:50:28 smithi047 ceph-mon[77147]: pgmap v1033: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:50:29.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:50:28 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 1360 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:50:29.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:50:28 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:50:29.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:50:28 smithi115 ceph-mon[78593]: pgmap v1033: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:50:29.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:50:28 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 1360 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:50:29.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:50:29 smithi115 conmon[90393]: 2022-05-13T23:50:29.586+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:50:30.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:50:29 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:50:30.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:50:29 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:50:30.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:50:29 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-13T23:50:30.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:50:29 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:50:30.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:50:29 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:50:30.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:50:29 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-13T23:50:30.957 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:50:30 smithi115 conmon[90393]: 2022-05-13T23:50:30.607+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:50:31.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:50:30 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:50:31.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:50:30 smithi047 ceph-mon[77147]: pgmap v1034: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:50:31.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:50:30 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:50:31.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:50:30 smithi115 ceph-mon[78593]: pgmap v1034: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:50:31.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:50:31 smithi115 conmon[90393]: 2022-05-13T23:50:31.567+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:50:32.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:50:31 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:50:32.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:50:31 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:50:32.861 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:50:32 smithi115 conmon[90393]: 2022-05-13T23:50:32.577+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:50:33.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:50:32 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:50:33.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:50:32 smithi047 ceph-mon[77147]: pgmap v1035: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:50:33.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:50:32 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:50:33.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:50:32 smithi115 ceph-mon[78593]: pgmap v1035: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:50:33.969 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:50:33 smithi115 conmon[90393]: 2022-05-13T23:50:33.614+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:50:34.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:50:33 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:50:34.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:50:33 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:50:34.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:50:33 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-13T23:50:34.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:50:33 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 1365 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:50:34.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:50:33 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:50:34.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:50:33 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:50:34.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:50:33 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-13T23:50:34.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:50:33 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 1365 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:50:34.970 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:50:34 smithi115 conmon[90393]: 2022-05-13T23:50:34.636+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:50:35.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:50:34 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:50:35.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:50:34 smithi047 ceph-mon[77147]: pgmap v1036: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:50:35.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:50:34 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:50:35.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:50:34 smithi115 ceph-mon[78593]: pgmap v1036: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:50:35.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:50:35 smithi115 conmon[90393]: 2022-05-13T23:50:35.590+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:50:36.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:50:35 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:50:36.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:50:35 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:50:36.983 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:50:36 smithi115 conmon[90393]: 2022-05-13T23:50:36.636+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:50:37.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:50:36 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:50:37.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:50:36 smithi047 ceph-mon[77147]: pgmap v1037: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:50:37.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:50:36 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:50:37.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:50:36 smithi115 ceph-mon[78593]: pgmap v1037: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:50:37.986 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:50:37 smithi115 conmon[90393]: 2022-05-13T23:50:37.664+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:50:38.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:50:37 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:50:38.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:50:37 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:50:38.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:50:37 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-13T23:50:38.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:50:37 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:50:38.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:50:37 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:50:38.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:50:37 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-13T23:50:38.987 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:50:38 smithi115 conmon[90393]: 2022-05-13T23:50:38.620+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:50:39.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:50:38 smithi047 ceph-mon[77147]: pgmap v1038: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:50:39.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:50:38 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:50:39.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:50:38 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 1370 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:50:39.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:50:38 smithi115 ceph-mon[78593]: pgmap v1038: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:50:39.369 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:50:38 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:50:39.369 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:50:38 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 1370 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:50:39.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:50:39 smithi115 conmon[90393]: 2022-05-13T23:50:39.585+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:50:40.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:50:39 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:50:40.362 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:50:39 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:50:40.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:50:40 smithi115 conmon[90393]: 2022-05-13T23:50:40.559+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:50:41.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:50:40 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:50:41.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:50:40 smithi047 ceph-mon[77147]: pgmap v1039: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:50:41.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:50:40 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:50:41.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:50:40 smithi115 ceph-mon[78593]: pgmap v1039: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:50:41.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:50:41 smithi115 conmon[90393]: 2022-05-13T23:50:41.517+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:50:42.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:50:42 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:50:42.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:50:42 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:50:42.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:50:42 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-13T23:50:42.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:50:42 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:50:42.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:50:42 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:50:42.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:50:42 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-13T23:50:42.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:50:42 smithi115 conmon[90393]: 2022-05-13T23:50:42.483+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:50:43.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:50:43 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:50:43.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:50:43 smithi047 ceph-mon[77147]: pgmap v1040: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:50:43.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:50:43 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:50:43.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:50:43 smithi115 ceph-mon[78593]: pgmap v1040: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:50:43.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:50:43 smithi115 conmon[90393]: 2022-05-13T23:50:43.497+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:50:44.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:50:44 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:50:44.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:50:44 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 1375 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:50:44.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:50:44 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:50:44.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:50:44 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 1375 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:50:44.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:50:44 smithi115 conmon[90393]: 2022-05-13T23:50:44.457+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:50:45.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:50:45 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:50:45.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:50:45 smithi047 ceph-mon[77147]: pgmap v1041: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:50:45.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:50:45 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:50:45.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:50:45 smithi115 ceph-mon[78593]: pgmap v1041: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:50:45.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:50:45 smithi115 conmon[90393]: 2022-05-13T23:50:45.407+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:50:46.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:50:46 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:50:46.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:50:46 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:50:46.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:50:46 smithi115 conmon[90393]: 2022-05-13T23:50:46.455+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:50:47.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:50:47 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:50:47.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:50:47 smithi047 ceph-mon[77147]: pgmap v1042: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:50:47.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:50:47 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:50:47.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:50:47 smithi115 ceph-mon[78593]: pgmap v1042: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:50:47.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:50:47 smithi115 conmon[90393]: 2022-05-13T23:50:47.490+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:50:48.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:50:48 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:50:48.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:50:48 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:50:48.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:50:48 smithi115 conmon[90393]: 2022-05-13T23:50:48.499+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:50:49.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:50:49 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:50:49.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:50:49 smithi047 ceph-mon[77147]: pgmap v1043: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:50:49.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:50:49 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 1380 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:50:49.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:50:49 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:50:49.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:50:49 smithi115 ceph-mon[78593]: pgmap v1043: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:50:49.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:50:49 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 1380 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:50:49.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:50:49 smithi115 conmon[90393]: 2022-05-13T23:50:49.520+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:50:50.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:50:50 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:50:50.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:50:50 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:50:50.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:50:50 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-13T23:50:50.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:50:50 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:50:50.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:50:50 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:50:50.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:50:50 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-13T23:50:50.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:50:50 smithi115 conmon[90393]: 2022-05-13T23:50:50.506+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:50:51.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:50:51 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:50:51.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:50:51 smithi047 ceph-mon[77147]: pgmap v1044: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:50:51.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:50:51 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:50:51.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:50:51 smithi115 ceph-mon[78593]: pgmap v1044: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:50:51.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:50:51 smithi115 conmon[90393]: 2022-05-13T23:50:51.500+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:50:52.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:50:52 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:50:52.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:50:52 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:50:52.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:50:52 smithi115 conmon[90393]: 2022-05-13T23:50:52.458+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:50:53.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:50:53 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:50:53.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:50:53 smithi047 ceph-mon[77147]: pgmap v1045: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:50:53.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:50:53 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:50:53.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:50:53 smithi115 ceph-mon[78593]: pgmap v1045: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:50:53.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:50:53 smithi115 conmon[90393]: 2022-05-13T23:50:53.484+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:50:54.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:50:54 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:50:54.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:50:54 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:50:54.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:50:54 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-13T23:50:54.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:50:54 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 1385 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:50:54.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:50:54 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:50:54.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:50:54 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:50:54.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:50:54 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-13T23:50:54.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:50:54 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 1385 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:50:54.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:50:54 smithi115 conmon[90393]: 2022-05-13T23:50:54.474+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:50:55.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:50:55 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:50:55.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:50:55 smithi115 ceph-mon[78593]: pgmap v1046: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:50:55.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:50:55 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:50:55.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:50:55 smithi047 ceph-mon[77147]: pgmap v1046: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:50:55.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:50:55 smithi115 conmon[90393]: 2022-05-13T23:50:55.522+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:50:56.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:50:56 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:50:56.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:50:56 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:50:56.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:50:56 smithi115 conmon[90393]: 2022-05-13T23:50:56.561+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:50:57.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:50:57 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:50:57.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:50:57 smithi115 ceph-mon[78593]: pgmap v1047: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:50:57.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:50:57 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:50:57.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:50:57 smithi047 ceph-mon[77147]: pgmap v1047: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:50:57.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:50:57 smithi115 conmon[90393]: 2022-05-13T23:50:57.551+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:50:58.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:50:58 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:50:58.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:50:58 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:50:58.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:50:58 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-13T23:50:58.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:50:58 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:50:58.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:50:58 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:50:58.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:50:58 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-13T23:50:58.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:50:58 smithi115 conmon[90393]: 2022-05-13T23:50:58.545+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:50:59.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:50:59 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:50:59.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:50:59 smithi115 ceph-mon[78593]: pgmap v1048: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:50:59.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:50:59 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 1390 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:50:59.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:50:59 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:50:59.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:50:59 smithi047 ceph-mon[77147]: pgmap v1048: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:50:59.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:50:59 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 1390 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:50:59.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:50:59 smithi115 conmon[90393]: 2022-05-13T23:50:59.538+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:51:00.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:51:00 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:51:00.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:51:00 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:51:00.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:51:00 smithi115 conmon[90393]: 2022-05-13T23:51:00.512+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:51:01.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:51:01 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:51:01.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:51:01 smithi115 ceph-mon[78593]: pgmap v1049: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:51:01.546 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:51:01 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:51:01.546 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:51:01 smithi047 ceph-mon[77147]: pgmap v1049: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:51:01.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:51:01 smithi115 conmon[90393]: 2022-05-13T23:51:01.481+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:51:02.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:51:02 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:51:02.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:51:02 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:51:02.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:51:02 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-13T23:51:02.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:51:02 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:51:02.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:51:02 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:51:02.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:51:02 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-13T23:51:02.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:51:02 smithi115 conmon[90393]: 2022-05-13T23:51:02.459+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:51:03.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:51:03 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:51:03.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:51:03 smithi115 ceph-mon[78593]: pgmap v1050: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:51:03.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:51:03 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:51:03.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:51:03 smithi047 ceph-mon[77147]: pgmap v1050: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:51:03.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:51:03 smithi115 conmon[90393]: 2022-05-13T23:51:03.445+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:51:04.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:51:04 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:51:04.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:51:04 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 1395 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:51:04.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:51:04 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:51:04.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:51:04 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 1395 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:51:04.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:51:04 smithi115 conmon[90393]: 2022-05-13T23:51:04.459+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:51:05.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:51:05 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:51:05.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:51:05 smithi115 ceph-mon[78593]: pgmap v1051: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:51:05.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:51:05 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:51:05.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:51:05 smithi047 ceph-mon[77147]: pgmap v1051: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:51:05.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:51:05 smithi115 conmon[90393]: 2022-05-13T23:51:05.441+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:51:06.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:51:06 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:51:06.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:51:06 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:51:06.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:51:06 smithi115 conmon[90393]: 2022-05-13T23:51:06.455+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:51:07.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:51:07 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:51:07.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:51:07 smithi115 ceph-mon[78593]: pgmap v1052: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:51:07.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:51:07 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:51:07.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:51:07 smithi047 ceph-mon[77147]: pgmap v1052: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:51:07.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:51:07 smithi115 conmon[90393]: 2022-05-13T23:51:07.479+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:51:08.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:51:08 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:51:08.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:51:08 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:51:08.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:51:08 smithi115 conmon[90393]: 2022-05-13T23:51:08.510+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:51:09.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:51:09 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:51:09.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:51:09 smithi115 ceph-mon[78593]: pgmap v1053: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:51:09.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:51:09 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 1400 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:51:09.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:51:09 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-13T23:51:09.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:51:09 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-13T23:51:09.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:51:09 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-13T23:51:09.529 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:51:09 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:51:09.530 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:51:09 smithi047 ceph-mon[77147]: pgmap v1053: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:51:09.530 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:51:09 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 1400 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:51:09.530 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:51:09 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-13T23:51:09.530 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:51:09 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-13T23:51:09.531 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:51:09 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-13T23:51:09.791 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:51:09 smithi047 conmon[78521]: 2022-05-13T23:51:09.615+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:51:09.791 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:51:09 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-13T23:51:09.792 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:51:09 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-13T23:51:09.792 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:51:09 smithi047 conmon[78521]: yield 2022-05-13T23:51:09.792 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:51:09 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-13T23:51:09.792 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:51:09 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-13T23:51:09.793 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:51:09 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-13T23:51:09.793 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:51:09 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-13T23:51:09.793 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:51:09 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-13T23:51:09.793 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:51:09 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-13T23:51:09.793 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:51:09 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-13T23:51:09.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:51:09 smithi047 conmon[78521]: raise exceptions[0] 2022-05-13T23:51:09.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:51:09 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-13T23:51:09.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:51:09 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-13T23:51:09.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:51:09 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-13T23:51:09.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:51:09 smithi047 conmon[78521]: return (yield from fut) 2022-05-13T23:51:09.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:51:09 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-13T23:51:09.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:51:09 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-13T23:51:09.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:51:09 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:51:09.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:51:09 smithi115 conmon[90393]: 2022-05-13T23:51:09.461+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:51:10.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:51:10 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:51:10.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:51:10 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:51:10.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:51:10 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-13T23:51:10.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:51:10 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:51:10.359 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-13T23:51:10.359 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-13T23:51:10.359 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-13T23:51:10.359 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-13T23:51:10.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-13T23:51:10.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-13T23:51:10.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-13T23:51:10.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-13T23:51:10.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-13T23:51:10.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-13T23:51:10.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-13T23:51:10.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-13T23:51:10.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-13T23:51:10.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-13T23:51:10.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-13T23:51:10.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-13T23:51:10.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-13T23:51:10.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:51:10.363 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:51:10 smithi115 ceph-mon[78593]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-13T23:51:10.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:51:10 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:51:10.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:51:10 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:51:10.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:51:10 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-13T23:51:10.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:51:10 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:51:10.542 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-13T23:51:10.542 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-13T23:51:10.542 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-13T23:51:10.542 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-13T23:51:10.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-13T23:51:10.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-13T23:51:10.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-13T23:51:10.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-13T23:51:10.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-13T23:51:10.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-13T23:51:10.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-13T23:51:10.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-13T23:51:10.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-13T23:51:10.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-13T23:51:10.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-13T23:51:10.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-13T23:51:10.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-13T23:51:10.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:51:10.546 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:51:10 smithi047 ceph-mon[77147]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-13T23:51:10.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:51:10 smithi115 conmon[90393]: 2022-05-13T23:51:10.416+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:51:11.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:51:11 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:51:11.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:51:11 smithi115 ceph-mon[78593]: pgmap v1054: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:51:11.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:51:11 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:51:11.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:51:11 smithi047 ceph-mon[77147]: pgmap v1054: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:51:11.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:51:11 smithi115 conmon[90393]: 2022-05-13T23:51:11.408+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:51:12.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:51:12 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:51:12.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:51:12 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:51:12.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:51:12 smithi115 conmon[90393]: 2022-05-13T23:51:12.453+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:51:13.041 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:51:12 smithi047 conmon[78521]: 2022-05-13T23:51:12.687+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:51:13.042 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:51:12 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-13T23:51:13.042 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:51:12 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-13T23:51:13.042 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:51:12 smithi047 conmon[78521]: yield 2022-05-13T23:51:13.042 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:51:12 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-13T23:51:13.042 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:51:12 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-13T23:51:13.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:51:12 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-13T23:51:13.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:51:12 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-13T23:51:13.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:51:12 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-13T23:51:13.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:51:12 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-13T23:51:13.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:51:12 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-13T23:51:13.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:51:12 smithi047 conmon[78521]: raise exceptions[0] 2022-05-13T23:51:13.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:51:12 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-13T23:51:13.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:51:12 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-13T23:51:13.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:51:12 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-13T23:51:13.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:51:12 smithi047 conmon[78521]: return (yield from fut) 2022-05-13T23:51:13.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:51:12 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-13T23:51:13.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:51:12 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-13T23:51:13.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:51:12 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:51:13.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:51:12 smithi047 conmon[78521]: 2022-05-13T23:51:12.688+0000 7f533ef76700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-13T23:51:13.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:51:12 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-13T23:51:13.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:51:12 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-13T23:51:13.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:51:12 smithi047 conmon[78521]: yield 2022-05-13T23:51:13.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:51:12 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-13T23:51:13.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:51:12 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-13T23:51:13.048 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:51:12 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-13T23:51:13.048 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:51:12 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-13T23:51:13.048 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:51:12 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-13T23:51:13.048 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:51:12 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-13T23:51:13.048 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:51:12 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-13T23:51:13.049 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:51:12 smithi047 conmon[78521]: raise exceptions[0] 2022-05-13T23:51:13.049 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:51:12 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-13T23:51:13.049 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:51:12 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-13T23:51:13.049 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:51:12 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-13T23:51:13.050 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:51:12 smithi047 conmon[78521]: return (yield from fut) 2022-05-13T23:51:13.050 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:51:12 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-13T23:51:13.050 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:51:12 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-13T23:51:13.050 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:51:12 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:51:13.050 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:51:12 smithi047 conmon[78521]: 2022-05-13T23:51:13.050 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:51:12 smithi047 conmon[78521]: During handling of the above exception, another exception occurred: 2022-05-13T23:51:13.051 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:51:12 smithi047 conmon[78521]: 2022-05-13T23:51:13.051 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:51:12 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-13T23:51:13.051 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:51:12 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-13T23:51:13.051 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:51:12 smithi047 conmon[78521]: return f(*arg) 2022-05-13T23:51:13.052 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:51:12 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-13T23:51:13.052 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:51:12 smithi047 conmon[78521]: self._write_client_files(client_files, host) 2022-05-13T23:51:13.052 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:51:12 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-13T23:51:13.052 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:51:12 smithi047 conmon[78521]: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-13T23:51:13.053 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:51:12 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-13T23:51:13.053 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:51:12 smithi047 conmon[78521]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-13T23:51:13.053 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:51:12 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-13T23:51:13.053 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:51:12 smithi047 conmon[78521]: return self.event_loop.get_result(coro) 2022-05-13T23:51:13.053 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:51:12 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-13T23:51:13.054 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:51:12 smithi047 conmon[78521]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-13T23:51:13.054 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:51:12 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-13T23:51:13.054 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:51:12 smithi047 conmon[78521]: return self.__get_result() 2022-05-13T23:51:13.054 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:51:12 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-13T23:51:13.055 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:51:12 smithi047 conmon[78521]: raise self._exception 2022-05-13T23:51:13.055 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:51:12 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-13T23:51:13.055 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:51:12 smithi047 conmon[78521]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-13T23:51:13.055 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:51:12 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-13T23:51:13.055 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:51:12 smithi047 conmon[78521]: conn = await self._remote_connection(host, addr) 2022-05-13T23:51:13.056 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:51:12 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-13T23:51:13.056 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:51:12 smithi047 conmon[78521]: raise 2022-05-13T23:51:13.056 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:51:12 smithi047 conmon[78521]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-13T23:51:13.056 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:51:12 smithi047 conmon[78521]: self.gen.throw(type, value, traceback) 2022-05-13T23:51:13.057 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:51:12 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-13T23:51:13.057 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:51:12 smithi047 conmon[78521]: raise OrchestratorError(msg) 2022-05-13T23:51:13.057 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:51:12 smithi047 conmon[78521]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:51:13.502 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:51:13 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:51:13.502 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:51:13 smithi115 ceph-mon[78593]: pgmap v1055: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:51:13.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:51:13 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:51:13.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:51:13 smithi047 ceph-mon[77147]: pgmap v1055: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:51:13.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:51:13 smithi115 conmon[90393]: 2022-05-13T23:51:13.502+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:51:14.529 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:51:14 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:51:14.529 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:51:14 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:51:14.530 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-13T23:51:14.530 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-13T23:51:14.530 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-13T23:51:14.530 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-13T23:51:14.531 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-13T23:51:14.531 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-13T23:51:14.531 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-13T23:51:14.531 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-13T23:51:14.532 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-13T23:51:14.532 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-13T23:51:14.532 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-13T23:51:14.533 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-13T23:51:14.533 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-13T23:51:14.533 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-13T23:51:14.533 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-13T23:51:14.534 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-13T23:51:14.534 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-13T23:51:14.534 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:51:14.534 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:51:14 smithi115 ceph-mon[78593]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-13T23:51:14.535 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-13T23:51:14.535 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-13T23:51:14.536 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-13T23:51:14.536 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-13T23:51:14.536 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-13T23:51:14.537 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-13T23:51:14.537 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-13T23:51:14.537 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-13T23:51:14.537 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-13T23:51:14.538 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-13T23:51:14.538 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-13T23:51:14.538 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-13T23:51:14.538 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-13T23:51:14.538 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-13T23:51:14.539 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-13T23:51:14.539 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-13T23:51:14.539 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-13T23:51:14.539 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:51:14.540 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-13T23:51:14.540 INFO:journalctl@ceph.mon.c.smithi115.stdout: During handling of the above exception, another exception occurred: 2022-05-13T23:51:14.540 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-13T23:51:14.540 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-13T23:51:14.540 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-13T23:51:14.541 INFO:journalctl@ceph.mon.c.smithi115.stdout: return f(*arg) 2022-05-13T23:51:14.541 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-13T23:51:14.541 INFO:journalctl@ceph.mon.c.smithi115.stdout: self._write_client_files(client_files, host) 2022-05-13T23:51:14.541 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-13T23:51:14.542 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-13T23:51:14.542 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-13T23:51:14.542 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-13T23:51:14.542 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-13T23:51:14.543 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.event_loop.get_result(coro) 2022-05-13T23:51:14.543 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-13T23:51:14.543 INFO:journalctl@ceph.mon.c.smithi115.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-13T23:51:14.543 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-13T23:51:14.543 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.__get_result() 2022-05-13T23:51:14.544 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-13T23:51:14.544 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise self._exception 2022-05-13T23:51:14.544 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-13T23:51:14.544 INFO:journalctl@ceph.mon.c.smithi115.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-13T23:51:14.544 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-13T23:51:14.545 INFO:journalctl@ceph.mon.c.smithi115.stdout: conn = await self._remote_connection(host, addr) 2022-05-13T23:51:14.545 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-13T23:51:14.545 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise 2022-05-13T23:51:14.546 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-13T23:51:14.546 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.gen.throw(type, value, traceback) 2022-05-13T23:51:14.546 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-13T23:51:14.546 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OrchestratorError(msg) 2022-05-13T23:51:14.547 INFO:journalctl@ceph.mon.c.smithi115.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:51:14.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:51:14 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:51:14.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:51:14 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-13T23:51:14.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:51:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-13T23:51:14.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:51:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-13T23:51:14.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:51:14 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 1405 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:51:14.549 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:51:14 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:51:14.549 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:51:14 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:51:14.549 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-13T23:51:14.549 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-13T23:51:14.550 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-13T23:51:14.550 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-13T23:51:14.550 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-13T23:51:14.551 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-13T23:51:14.551 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-13T23:51:14.551 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-13T23:51:14.551 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-13T23:51:14.551 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-13T23:51:14.552 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-13T23:51:14.552 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-13T23:51:14.552 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-13T23:51:14.552 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-13T23:51:14.553 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-13T23:51:14.553 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-13T23:51:14.553 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-13T23:51:14.553 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:51:14.553 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:51:14 smithi047 ceph-mon[77147]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-13T23:51:14.554 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-13T23:51:14.554 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-13T23:51:14.554 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-13T23:51:14.554 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-13T23:51:14.555 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-13T23:51:14.555 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-13T23:51:14.555 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-13T23:51:14.556 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-13T23:51:14.556 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-13T23:51:14.556 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-13T23:51:14.556 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-13T23:51:14.557 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-13T23:51:14.557 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-13T23:51:14.557 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-13T23:51:14.557 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-13T23:51:14.558 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-13T23:51:14.558 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-13T23:51:14.558 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:51:14.558 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-13T23:51:14.559 INFO:journalctl@ceph.mon.b.smithi047.stdout: During handling of the above exception, another exception occurred: 2022-05-13T23:51:14.559 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-13T23:51:14.559 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-13T23:51:14.559 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-13T23:51:14.559 INFO:journalctl@ceph.mon.b.smithi047.stdout: return f(*arg) 2022-05-13T23:51:14.560 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-13T23:51:14.560 INFO:journalctl@ceph.mon.b.smithi047.stdout: self._write_client_files(client_files, host) 2022-05-13T23:51:14.560 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-13T23:51:14.560 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-13T23:51:14.561 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-13T23:51:14.561 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-13T23:51:14.561 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-13T23:51:14.561 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.event_loop.get_result(coro) 2022-05-13T23:51:14.562 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-13T23:51:14.562 INFO:journalctl@ceph.mon.b.smithi047.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-13T23:51:14.562 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-13T23:51:14.562 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.__get_result() 2022-05-13T23:51:14.562 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-13T23:51:14.563 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise self._exception 2022-05-13T23:51:14.563 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-13T23:51:14.563 INFO:journalctl@ceph.mon.b.smithi047.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-13T23:51:14.563 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-13T23:51:14.564 INFO:journalctl@ceph.mon.b.smithi047.stdout: conn = await self._remote_connection(host, addr) 2022-05-13T23:51:14.564 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-13T23:51:14.564 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise 2022-05-13T23:51:14.564 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-13T23:51:14.564 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.gen.throw(type, value, traceback) 2022-05-13T23:51:14.565 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-13T23:51:14.565 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OrchestratorError(msg) 2022-05-13T23:51:14.565 INFO:journalctl@ceph.mon.b.smithi047.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:51:14.565 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:51:14 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:51:14.566 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:51:14 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-13T23:51:14.566 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:51:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-13T23:51:14.566 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:51:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-13T23:51:14.566 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:51:14 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 1405 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:51:14.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:51:14 smithi115 conmon[90393]: 2022-05-13T23:51:14.528+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:51:15.500 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:51:15 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:51:15.501 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:51:15 smithi115 ceph-mon[78593]: pgmap v1056: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:51:15.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:51:15 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:51:15.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:51:15 smithi047 ceph-mon[77147]: pgmap v1056: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:51:15.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:51:15 smithi115 conmon[90393]: 2022-05-13T23:51:15.499+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:51:16.478 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:51:16 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:51:16.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:51:16 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:51:16.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:51:16 smithi115 conmon[90393]: 2022-05-13T23:51:16.477+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:51:17.480 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:51:17 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:51:17.481 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:51:17 smithi115 ceph-mon[78593]: pgmap v1057: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:51:17.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:51:17 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:51:17.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:51:17 smithi047 ceph-mon[77147]: pgmap v1057: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:51:17.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:51:17 smithi115 conmon[90393]: 2022-05-13T23:51:17.478+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:51:18.471 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:51:18 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:51:18.471 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:51:18 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:51:18.471 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:51:18 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-13T23:51:18.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:51:18 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:51:18.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:51:18 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:51:18.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:51:18 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-13T23:51:18.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:51:18 smithi115 conmon[90393]: 2022-05-13T23:51:18.469+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:51:19.487 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:51:19 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:51:19.487 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:51:19 smithi115 ceph-mon[78593]: pgmap v1058: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:51:19.488 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:51:19 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 1410 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:51:19.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:51:19 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:51:19.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:51:19 smithi047 ceph-mon[77147]: pgmap v1058: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:51:19.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:51:19 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 1410 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:51:19.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:51:19 smithi115 conmon[90393]: 2022-05-13T23:51:19.486+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:51:20.529 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:51:20 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:51:20.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:51:20 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:51:20.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:51:20 smithi115 conmon[90393]: 2022-05-13T23:51:20.527+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:51:21.530 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:51:21 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:51:21.531 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:51:21 smithi115 ceph-mon[78593]: pgmap v1059: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:51:21.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:51:21 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:51:21.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:51:21 smithi047 ceph-mon[77147]: pgmap v1059: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:51:21.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:51:21 smithi115 conmon[90393]: 2022-05-13T23:51:21.529+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:51:22.531 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:51:22 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:51:22.531 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:51:22 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:51:22.531 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:51:22 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-13T23:51:22.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:51:22 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:51:22.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:51:22 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:51:22.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:51:22 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-13T23:51:22.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:51:22 smithi115 conmon[90393]: 2022-05-13T23:51:22.529+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:51:23.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:51:23 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:51:23.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:51:23 smithi047 ceph-mon[77147]: pgmap v1060: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:51:23.574 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:51:23 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:51:23.575 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:51:23 smithi115 ceph-mon[78593]: pgmap v1060: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:51:23.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:51:23 smithi115 conmon[90393]: 2022-05-13T23:51:23.573+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:51:24.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:51:24 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:51:24.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:51:24 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 1415 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:51:24.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:51:24 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:51:24.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:51:24 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 1415 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:51:25.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:51:24 smithi115 conmon[90393]: 2022-05-13T23:51:24.608+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:51:25.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:51:25 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:51:25.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:51:25 smithi047 ceph-mon[77147]: pgmap v1061: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:51:25.561 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:51:25 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:51:25.562 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:51:25 smithi115 ceph-mon[78593]: pgmap v1061: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:51:25.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:51:25 smithi115 conmon[90393]: 2022-05-13T23:51:25.560+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:51:26.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:51:26 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:51:26.598 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:51:26 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:51:26.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:51:26 smithi115 conmon[90393]: 2022-05-13T23:51:26.597+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:51:27.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:51:27 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:51:27.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:51:27 smithi047 ceph-mon[77147]: pgmap v1062: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:51:27.552 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:51:27 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:51:27.552 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:51:27 smithi115 ceph-mon[78593]: pgmap v1062: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:51:27.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:51:27 smithi115 conmon[90393]: 2022-05-13T23:51:27.550+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:51:28.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:51:28 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:51:28.593 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:51:28 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:51:28.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:51:28 smithi115 conmon[90393]: 2022-05-13T23:51:28.592+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:51:29.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:51:29 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:51:29.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:51:29 smithi047 ceph-mon[77147]: pgmap v1063: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:51:29.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:51:29 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 1420 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:51:29.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:51:29 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:51:29.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:51:29 smithi115 ceph-mon[78593]: pgmap v1063: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:51:29.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:51:29 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 1420 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:51:29.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:51:29 smithi115 conmon[90393]: 2022-05-13T23:51:29.547+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:51:30.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:51:30 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:51:30.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:51:30 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:51:30.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:51:30 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-13T23:51:30.561 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:51:30 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:51:30.561 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:51:30 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:51:30.561 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:51:30 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-13T23:51:30.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:51:30 smithi115 conmon[90393]: 2022-05-13T23:51:30.560+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:51:31.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:51:31 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:51:31.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:51:31 smithi047 ceph-mon[77147]: pgmap v1064: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:51:31.588 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:51:31 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:51:31.589 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:51:31 smithi115 ceph-mon[78593]: pgmap v1064: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:51:31.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:51:31 smithi115 conmon[90393]: 2022-05-13T23:51:31.587+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:51:32.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:51:32 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:51:32.558 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:51:32 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:51:32.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:51:32 smithi115 conmon[90393]: 2022-05-13T23:51:32.557+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:51:33.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:51:33 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:51:33.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:51:33 smithi047 ceph-mon[77147]: pgmap v1065: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:51:33.561 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:51:33 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:51:33.562 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:51:33 smithi115 ceph-mon[78593]: pgmap v1065: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:51:33.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:51:33 smithi115 conmon[90393]: 2022-05-13T23:51:33.559+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:51:34.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:51:34 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:51:34.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:51:34 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:51:34.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:51:34 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-13T23:51:34.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:51:34 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 1425 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:51:34.585 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:51:34 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:51:34.585 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:51:34 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:51:34.586 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:51:34 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-13T23:51:34.586 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:51:34 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 1425 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:51:34.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:51:34 smithi115 conmon[90393]: 2022-05-13T23:51:34.583+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:51:35.537 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:51:35 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:51:35.537 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:51:35 smithi115 ceph-mon[78593]: pgmap v1066: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:51:35.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:51:35 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:51:35.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:51:35 smithi047 ceph-mon[77147]: pgmap v1066: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:51:35.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:51:35 smithi115 conmon[90393]: 2022-05-13T23:51:35.535+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:51:36.511 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:51:36 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:51:36.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:51:36 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:51:36.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:51:36 smithi115 conmon[90393]: 2022-05-13T23:51:36.509+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:51:37.492 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:51:37 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:51:37.493 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:51:37 smithi115 ceph-mon[78593]: pgmap v1067: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:51:37.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:51:37 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:51:37.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:51:37 smithi047 ceph-mon[77147]: pgmap v1067: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:51:37.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:51:37 smithi115 conmon[90393]: 2022-05-13T23:51:37.491+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:51:38.505 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:51:38 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:51:38.505 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:51:38 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:51:38.505 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:51:38 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-13T23:51:38.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:51:38 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:51:38.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:51:38 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:51:38.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:51:38 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-13T23:51:38.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:51:38 smithi115 conmon[90393]: 2022-05-13T23:51:38.503+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:51:39.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:51:39 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:51:39.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:51:39 smithi047 ceph-mon[77147]: pgmap v1068: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:51:39.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:51:39 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 1430 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:51:39.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:51:39 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:51:39.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:51:39 smithi115 ceph-mon[78593]: pgmap v1068: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:51:39.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:51:39 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 1430 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:51:39.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:51:39 smithi115 conmon[90393]: 2022-05-13T23:51:39.455+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:51:40.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:51:40 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:51:40.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:51:40 smithi115 conmon[90393]: 2022-05-13T23:51:40.432+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:51:40.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:51:40 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:51:41.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:51:41 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:51:41.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:51:41 smithi047 ceph-mon[77147]: pgmap v1069: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:51:41.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:51:41 smithi115 conmon[90393]: 2022-05-13T23:51:41.447+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:51:41.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:51:41 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:51:41.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:51:41 smithi115 ceph-mon[78593]: pgmap v1069: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:51:42.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:51:42 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:51:42.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:51:42 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:51:42.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:51:42 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-13T23:51:42.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:51:42 smithi115 conmon[90393]: 2022-05-13T23:51:42.462+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:51:42.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:51:42 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:51:42.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:51:42 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:51:42.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:51:42 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-13T23:51:43.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:51:43 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:51:43.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:51:43 smithi047 ceph-mon[77147]: pgmap v1070: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:51:43.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:51:43 smithi115 conmon[90393]: 2022-05-13T23:51:43.477+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:51:43.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:51:43 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:51:43.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:51:43 smithi115 ceph-mon[78593]: pgmap v1070: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:51:44.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:51:44 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:51:44.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:51:44 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 1435 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:51:44.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:51:44 smithi115 conmon[90393]: 2022-05-13T23:51:44.449+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:51:44.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:51:44 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:51:44.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:51:44 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 1435 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:51:45.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:51:45 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:51:45.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:51:45 smithi047 ceph-mon[77147]: pgmap v1071: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:51:45.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:51:45 smithi115 conmon[90393]: 2022-05-13T23:51:45.439+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:51:45.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:51:45 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:51:45.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:51:45 smithi115 ceph-mon[78593]: pgmap v1071: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:51:46.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:51:46 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:51:46.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:51:46 smithi115 conmon[90393]: 2022-05-13T23:51:46.391+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:51:46.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:51:46 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:51:47.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:51:47 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:51:47.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:51:47 smithi047 ceph-mon[77147]: pgmap v1072: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:51:47.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:51:47 smithi115 conmon[90393]: 2022-05-13T23:51:47.383+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:51:47.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:51:47 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:51:47.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:51:47 smithi115 ceph-mon[78593]: pgmap v1072: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:51:48.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:51:48 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:51:48.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:51:48 smithi115 conmon[90393]: 2022-05-13T23:51:48.376+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:51:48.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:51:48 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:51:49.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:51:49 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:51:49.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:51:49 smithi047 ceph-mon[77147]: pgmap v1073: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:51:49.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:51:49 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 1440 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:51:49.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:51:49 smithi115 conmon[90393]: 2022-05-13T23:51:49.367+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:51:49.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:51:49 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:51:49.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:51:49 smithi115 ceph-mon[78593]: pgmap v1073: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:51:49.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:51:49 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 1440 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:51:50.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:51:50 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:51:50.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:51:50 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:51:50.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:51:50 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-13T23:51:50.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:51:50 smithi115 conmon[90393]: 2022-05-13T23:51:50.340+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:51:50.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:51:50 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:51:50.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:51:50 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:51:50.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:51:50 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-13T23:51:51.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:51:51 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:51:51.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:51:51 smithi047 ceph-mon[77147]: pgmap v1074: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:51:51.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:51:51 smithi115 conmon[90393]: 2022-05-13T23:51:51.331+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:51:51.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:51:51 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:51:51.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:51:51 smithi115 ceph-mon[78593]: pgmap v1074: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:51:52.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:51:52 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:51:52.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:51:52 smithi115 conmon[90393]: 2022-05-13T23:51:52.338+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:51:52.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:51:52 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:51:53.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:51:53 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:51:53.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:51:53 smithi047 ceph-mon[77147]: pgmap v1075: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:51:53.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:51:53 smithi115 conmon[90393]: 2022-05-13T23:51:53.302+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:51:53.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:51:53 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:51:53.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:51:53 smithi115 ceph-mon[78593]: pgmap v1075: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:51:54.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:51:54 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:51:54.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:51:54 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:51:54.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:51:54 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-13T23:51:54.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:51:54 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:51:54.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:51:54 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 1445 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:51:54.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:51:54 smithi115 conmon[90393]: 2022-05-13T23:51:54.343+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:51:54.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:51:54 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:51:54.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:51:54 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:51:54.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:51:54 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-13T23:51:54.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:51:54 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:51:54.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:51:54 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 1445 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:51:55.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:51:55 smithi047 ceph-mon[77147]: pgmap v1076: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:51:55.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:51:55 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:51:55.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:51:55 smithi115 conmon[90393]: 2022-05-13T23:51:55.380+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:51:55.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:51:55 smithi115 ceph-mon[78593]: pgmap v1076: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:51:55.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:51:55 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:51:56.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:51:56 smithi115 conmon[90393]: 2022-05-13T23:51:56.348+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:51:56.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:51:56 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:51:56.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:51:56 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:51:57.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:51:57 smithi047 ceph-mon[77147]: pgmap v1077: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:51:57.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:51:57 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:51:57.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:51:57 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:51:57.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:51:57 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-13T23:51:57.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:51:57 smithi115 conmon[90393]: 2022-05-13T23:51:57.357+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:51:57.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:51:57 smithi115 ceph-mon[78593]: pgmap v1077: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:51:57.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:51:57 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:51:57.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:51:57 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:51:57.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:51:57 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-13T23:51:58.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:51:58 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:51:58.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:51:58 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 1450 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:51:58.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:51:58 smithi115 conmon[90393]: 2022-05-13T23:51:58.386+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:51:58.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:51:58 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:51:58.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:51:58 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 1450 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:51:59.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:51:59 smithi047 ceph-mon[77147]: pgmap v1078: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:51:59.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:51:59 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:51:59.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:51:59 smithi115 conmon[90393]: 2022-05-13T23:51:59.426+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:51:59.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:51:59 smithi115 ceph-mon[78593]: pgmap v1078: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:51:59.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:51:59 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:52:00.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:52:00 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:52:00.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:52:00 smithi115 conmon[90393]: 2022-05-13T23:52:00.444+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:52:00.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:52:00 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:52:01.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:52:01 smithi047 ceph-mon[77147]: pgmap v1079: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:52:01.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:52:01 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:52:01.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:52:01 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-13T23:52:01.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:52:01 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:52:01.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:52:01 smithi115 conmon[90393]: 2022-05-13T23:52:01.487+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:52:01.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:52:01 smithi115 ceph-mon[78593]: pgmap v1079: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:52:01.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:52:01 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:52:01.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:52:01 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-13T23:52:01.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:52:01 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:52:02.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:52:02 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:52:02.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:52:02 smithi115 conmon[90393]: 2022-05-13T23:52:02.529+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:52:02.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:52:02 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:52:03.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:52:03 smithi047 ceph-mon[77147]: pgmap v1080: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:52:03.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:52:03 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:52:03.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:52:03 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 1455 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:52:03.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:52:03 smithi115 conmon[90393]: 2022-05-13T23:52:03.503+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:52:03.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:52:03 smithi115 ceph-mon[78593]: pgmap v1080: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:52:03.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:52:03 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:52:03.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:52:03 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 1455 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:52:04.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:52:04 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:52:04.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:52:04 smithi115 conmon[90393]: 2022-05-13T23:52:04.460+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:52:04.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:52:04 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:52:05.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:52:05 smithi047 ceph-mon[77147]: pgmap v1081: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:52:05.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:52:05 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:52:05.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:52:05 smithi115 conmon[90393]: 2022-05-13T23:52:05.494+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:52:05.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:52:05 smithi115 ceph-mon[78593]: pgmap v1081: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:52:05.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:52:05 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:52:06.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:52:06 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:52:06.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:52:06 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:52:06.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:52:06 smithi115 conmon[90393]: 2022-05-13T23:52:06.531+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:52:07.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:52:07 smithi047 ceph-mon[77147]: pgmap v1082: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:52:07.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:52:07 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:52:07.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:52:07 smithi115 conmon[90393]: 2022-05-13T23:52:07.539+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:52:07.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:52:07 smithi115 ceph-mon[78593]: pgmap v1082: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:52:07.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:52:07 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:52:08.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:52:08 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:52:08.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:52:08 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 1460 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:52:08.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:52:08 smithi115 conmon[90393]: 2022-05-13T23:52:08.509+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:52:08.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:52:08 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:52:08.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:52:08 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 1460 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:52:09.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:52:09 smithi047 ceph-mon[77147]: pgmap v1083: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:52:09.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:52:09 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:52:09.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:52:09 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-13T23:52:09.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:52:09 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:52:09.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:52:09 smithi115 conmon[90393]: 2022-05-13T23:52:09.520+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:52:09.998 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:52:09 smithi115 ceph-mon[78593]: pgmap v1083: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:52:09.998 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:52:09 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:52:09.998 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:52:09 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-13T23:52:09.998 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:52:09 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:52:10.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:52:10 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:52:10.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:52:10 smithi115 conmon[90393]: 2022-05-13T23:52:10.567+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:52:10.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:52:10 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:52:11.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:52:11 smithi047 ceph-mon[77147]: pgmap v1084: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:52:11.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:52:11 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:52:11.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:52:11 smithi115 conmon[90393]: 2022-05-13T23:52:11.551+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:52:11.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:52:11 smithi115 ceph-mon[78593]: pgmap v1084: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:52:11.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:52:11 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:52:12.735 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:52:12 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:52:12.736 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:52:12 smithi115 conmon[90393]: 2022-05-13T23:52:12.543+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:52:12.737 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:52:12 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:52:13.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:52:13 smithi047 ceph-mon[77147]: pgmap v1085: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:52:13.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:52:13 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:52:13.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:52:13 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-13T23:52:13.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:52:13 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:52:13.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:52:13 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-13T23:52:13.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:52:13 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-13T23:52:13.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:52:13 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-13T23:52:13.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:52:13 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-13T23:52:13.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:52:13 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-13T23:52:13.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:52:13 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 1465 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:52:13.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:52:13 smithi115 conmon[90393]: 2022-05-13T23:52:13.540+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:52:13.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:52:13 smithi115 ceph-mon[78593]: pgmap v1085: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:52:13.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:52:13 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:52:13.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:52:13 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-13T23:52:13.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:52:13 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:52:13.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:52:13 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-13T23:52:13.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:52:13 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-13T23:52:13.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:52:13 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-13T23:52:13.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:52:13 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-13T23:52:13.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:52:13 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-13T23:52:13.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:52:13 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 1465 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:52:14.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:52:14 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:52:14.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:52:14 smithi115 conmon[90393]: 2022-05-13T23:52:14.548+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:52:14.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:52:14 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:52:15.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:52:15 smithi047 ceph-mon[77147]: pgmap v1086: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:52:15.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:52:15 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:52:15.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:52:15 smithi115 conmon[90393]: 2022-05-13T23:52:15.521+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:52:15.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:52:15 smithi115 ceph-mon[78593]: pgmap v1086: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:52:15.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:52:15 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:52:16.291 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:52:15 smithi047 conmon[78521]: 2022-05-13T23:52:15.792+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:52:16.291 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:52:15 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-13T23:52:16.292 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:52:15 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-13T23:52:16.292 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:52:15 smithi047 conmon[78521]: yield 2022-05-13T23:52:16.292 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:52:15 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-13T23:52:16.292 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:52:15 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-13T23:52:16.293 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:52:15 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-13T23:52:16.293 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:52:15 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-13T23:52:16.293 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:52:15 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-13T23:52:16.293 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:52:15 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-13T23:52:16.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:52:15 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-13T23:52:16.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:52:15 smithi047 conmon[78521]: raise exceptions[0] 2022-05-13T23:52:16.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:52:15 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-13T23:52:16.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:52:15 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-13T23:52:16.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:52:15 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-13T23:52:16.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:52:15 smithi047 conmon[78521]: return (yield from fut) 2022-05-13T23:52:16.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:52:15 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-13T23:52:16.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:52:15 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-13T23:52:16.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:52:15 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:52:16.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:52:16 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:52:16.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:52:16 smithi115 conmon[90393]: 2022-05-13T23:52:16.562+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:52:16.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:52:16 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:52:17.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:52:17 smithi047 ceph-mon[77147]: pgmap v1087: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:52:17.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:52:17 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:52:17.791 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-13T23:52:17.792 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-13T23:52:17.792 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-13T23:52:17.792 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-13T23:52:17.792 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-13T23:52:17.793 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-13T23:52:17.793 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-13T23:52:17.793 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-13T23:52:17.793 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-13T23:52:17.793 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-13T23:52:17.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-13T23:52:17.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-13T23:52:17.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-13T23:52:17.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-13T23:52:17.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-13T23:52:17.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-13T23:52:17.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-13T23:52:17.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:52:17.795 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:52:17 smithi047 ceph-mon[77147]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-13T23:52:17.796 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:52:17 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:52:17.796 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:52:17 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-13T23:52:17.796 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:52:17 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:52:17.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:52:17 smithi115 conmon[90393]: 2022-05-13T23:52:17.522+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:52:17.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:52:17 smithi115 ceph-mon[78593]: pgmap v1087: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:52:17.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:52:17 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:52:17.859 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-13T23:52:17.859 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-13T23:52:17.859 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-13T23:52:17.859 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-13T23:52:17.860 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-13T23:52:17.860 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-13T23:52:17.860 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-13T23:52:17.860 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-13T23:52:17.860 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-13T23:52:17.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-13T23:52:17.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-13T23:52:17.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-13T23:52:17.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-13T23:52:17.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-13T23:52:17.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-13T23:52:17.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-13T23:52:17.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-13T23:52:17.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:52:17.863 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:52:17 smithi115 ceph-mon[78593]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-13T23:52:17.863 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:52:17 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:52:17.863 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:52:17 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-13T23:52:17.863 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:52:17 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:52:18.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:52:18 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:52:18.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:52:18 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 1470 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:52:18.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:52:18 smithi115 conmon[90393]: 2022-05-13T23:52:18.546+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:52:18.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:52:18 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:52:18.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:52:18 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 1470 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:52:19.291 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:52:18 smithi047 conmon[78521]: 2022-05-13T23:52:18.864+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:52:19.292 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:52:18 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-13T23:52:19.292 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:52:18 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-13T23:52:19.292 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:52:18 smithi047 conmon[78521]: yield 2022-05-13T23:52:19.292 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:52:18 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-13T23:52:19.293 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:52:18 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-13T23:52:19.293 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:52:18 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-13T23:52:19.293 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:52:18 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-13T23:52:19.293 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:52:18 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-13T23:52:19.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:52:18 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-13T23:52:19.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:52:18 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-13T23:52:19.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:52:18 smithi047 conmon[78521]: raise exceptions[0] 2022-05-13T23:52:19.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:52:18 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-13T23:52:19.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:52:18 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-13T23:52:19.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:52:18 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-13T23:52:19.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:52:18 smithi047 conmon[78521]: return (yield from fut) 2022-05-13T23:52:19.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:52:18 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-13T23:52:19.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:52:18 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-13T23:52:19.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:52:18 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:52:19.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:52:18 smithi047 conmon[78521]: 2022-05-13T23:52:18.865+0000 7f533f777700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-13T23:52:19.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:52:18 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-13T23:52:19.297 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:52:18 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-13T23:52:19.297 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:52:18 smithi047 conmon[78521]: yield 2022-05-13T23:52:19.297 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:52:18 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-13T23:52:19.297 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:52:18 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-13T23:52:19.298 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:52:18 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-13T23:52:19.298 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:52:18 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-13T23:52:19.298 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:52:18 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-13T23:52:19.298 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:52:18 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-13T23:52:19.298 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:52:18 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-13T23:52:19.299 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:52:18 smithi047 conmon[78521]: raise exceptions[0] 2022-05-13T23:52:19.299 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:52:18 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-13T23:52:19.299 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:52:18 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-13T23:52:19.299 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:52:18 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-13T23:52:19.300 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:52:18 smithi047 conmon[78521]: return (yield from fut) 2022-05-13T23:52:19.300 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:52:18 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-13T23:52:19.300 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:52:18 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-13T23:52:19.300 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:52:18 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:52:19.300 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:52:18 smithi047 conmon[78521]: 2022-05-13T23:52:19.301 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:52:18 smithi047 conmon[78521]: During handling of the above exception, another exception occurred: 2022-05-13T23:52:19.301 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:52:18 smithi047 conmon[78521]: 2022-05-13T23:52:19.301 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:52:18 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-13T23:52:19.301 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:52:18 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-13T23:52:19.302 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:52:18 smithi047 conmon[78521]: return f(*arg) 2022-05-13T23:52:19.302 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:52:18 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-13T23:52:19.302 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:52:18 smithi047 conmon[78521]: self._write_client_files(client_files, host) 2022-05-13T23:52:19.302 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:52:18 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-13T23:52:19.302 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:52:18 smithi047 conmon[78521]: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-13T23:52:19.303 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:52:18 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-13T23:52:19.303 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:52:18 smithi047 conmon[78521]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-13T23:52:19.303 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:52:18 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-13T23:52:19.303 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:52:18 smithi047 conmon[78521]: return self.event_loop.get_result(coro) 2022-05-13T23:52:19.304 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:52:18 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-13T23:52:19.304 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:52:18 smithi047 conmon[78521]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-13T23:52:19.304 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:52:18 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-13T23:52:19.304 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:52:18 smithi047 conmon[78521]: return self.__get_result() 2022-05-13T23:52:19.304 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:52:18 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-13T23:52:19.305 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:52:18 smithi047 conmon[78521]: raise self._exception 2022-05-13T23:52:19.305 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:52:18 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-13T23:52:19.305 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:52:18 smithi047 conmon[78521]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-13T23:52:19.305 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:52:18 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-13T23:52:19.306 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:52:18 smithi047 conmon[78521]: conn = await self._remote_connection(host, addr) 2022-05-13T23:52:19.306 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:52:18 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-13T23:52:19.306 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:52:18 smithi047 conmon[78521]: raise 2022-05-13T23:52:19.306 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:52:18 smithi047 conmon[78521]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-13T23:52:19.306 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:52:18 smithi047 conmon[78521]: self.gen.throw(type, value, traceback) 2022-05-13T23:52:19.307 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:52:18 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-13T23:52:19.307 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:52:18 smithi047 conmon[78521]: raise OrchestratorError(msg) 2022-05-13T23:52:19.307 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:52:18 smithi047 conmon[78521]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:52:19.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:52:19 smithi047 ceph-mon[77147]: pgmap v1088: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:52:19.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:52:19 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:52:19.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:52:19 smithi115 conmon[90393]: 2022-05-13T23:52:19.591+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:52:19.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:52:19 smithi115 ceph-mon[78593]: pgmap v1088: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:52:19.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:52:19 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:52:20.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:52:20 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:52:20.793 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-13T23:52:20.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-13T23:52:20.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-13T23:52:20.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-13T23:52:20.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-13T23:52:20.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-13T23:52:20.797 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-13T23:52:20.797 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-13T23:52:20.797 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-13T23:52:20.798 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-13T23:52:20.798 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-13T23:52:20.798 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-13T23:52:20.798 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-13T23:52:20.799 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-13T23:52:20.799 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-13T23:52:20.799 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-13T23:52:20.799 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-13T23:52:20.799 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:52:20.800 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:52:20 smithi047 ceph-mon[77147]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-13T23:52:20.800 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-13T23:52:20.800 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-13T23:52:20.801 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-13T23:52:20.801 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-13T23:52:20.801 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-13T23:52:20.801 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-13T23:52:20.802 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-13T23:52:20.802 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-13T23:52:20.802 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-13T23:52:20.802 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-13T23:52:20.803 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-13T23:52:20.803 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-13T23:52:20.803 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-13T23:52:20.803 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-13T23:52:20.803 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-13T23:52:20.804 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-13T23:52:20.804 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-13T23:52:20.804 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:52:20.805 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-13T23:52:20.806 INFO:journalctl@ceph.mon.b.smithi047.stdout: During handling of the above exception, another exception occurred: 2022-05-13T23:52:20.806 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-13T23:52:20.806 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-13T23:52:20.806 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-13T23:52:20.807 INFO:journalctl@ceph.mon.b.smithi047.stdout: return f(*arg) 2022-05-13T23:52:20.807 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-13T23:52:20.807 INFO:journalctl@ceph.mon.b.smithi047.stdout: self._write_client_files(client_files, host) 2022-05-13T23:52:20.808 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-13T23:52:20.808 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-13T23:52:20.808 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-13T23:52:20.809 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-13T23:52:20.809 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-13T23:52:20.809 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.event_loop.get_result(coro) 2022-05-13T23:52:20.810 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-13T23:52:20.810 INFO:journalctl@ceph.mon.b.smithi047.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-13T23:52:20.811 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-13T23:52:20.811 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.__get_result() 2022-05-13T23:52:20.811 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-13T23:52:20.812 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise self._exception 2022-05-13T23:52:20.812 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-13T23:52:20.813 INFO:journalctl@ceph.mon.b.smithi047.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-13T23:52:20.813 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-13T23:52:20.813 INFO:journalctl@ceph.mon.b.smithi047.stdout: conn = await self._remote_connection(host, addr) 2022-05-13T23:52:20.814 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-13T23:52:20.814 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise 2022-05-13T23:52:20.814 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-13T23:52:20.814 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.gen.throw(type, value, traceback) 2022-05-13T23:52:20.815 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-13T23:52:20.815 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OrchestratorError(msg) 2022-05-13T23:52:20.815 INFO:journalctl@ceph.mon.b.smithi047.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:52:20.816 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:52:20 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:52:20.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:52:20 smithi115 conmon[90393]: 2022-05-13T23:52:20.551+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:52:20.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:52:20 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:52:20.860 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-13T23:52:20.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-13T23:52:20.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-13T23:52:20.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-13T23:52:20.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-13T23:52:20.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-13T23:52:20.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-13T23:52:20.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-13T23:52:20.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-13T23:52:20.863 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-13T23:52:20.863 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-13T23:52:20.863 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-13T23:52:20.863 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-13T23:52:20.863 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-13T23:52:20.864 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-13T23:52:20.864 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-13T23:52:20.864 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-13T23:52:20.865 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:52:20.865 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:52:20 smithi115 ceph-mon[78593]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-13T23:52:20.865 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-13T23:52:20.866 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-13T23:52:20.867 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-13T23:52:20.867 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-13T23:52:20.868 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-13T23:52:20.868 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-13T23:52:20.868 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-13T23:52:20.868 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-13T23:52:20.869 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-13T23:52:20.869 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-13T23:52:20.870 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-13T23:52:20.870 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-13T23:52:20.870 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-13T23:52:20.870 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-13T23:52:20.871 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-13T23:52:20.871 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-13T23:52:20.871 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-13T23:52:20.872 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:52:20.872 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-13T23:52:20.872 INFO:journalctl@ceph.mon.c.smithi115.stdout: During handling of the above exception, another exception occurred: 2022-05-13T23:52:20.872 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-13T23:52:20.872 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-13T23:52:20.873 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-13T23:52:20.873 INFO:journalctl@ceph.mon.c.smithi115.stdout: return f(*arg) 2022-05-13T23:52:20.873 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-13T23:52:20.873 INFO:journalctl@ceph.mon.c.smithi115.stdout: self._write_client_files(client_files, host) 2022-05-13T23:52:20.874 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-13T23:52:20.874 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-13T23:52:20.874 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-13T23:52:20.875 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-13T23:52:20.875 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-13T23:52:20.875 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.event_loop.get_result(coro) 2022-05-13T23:52:20.876 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-13T23:52:20.876 INFO:journalctl@ceph.mon.c.smithi115.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-13T23:52:20.876 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-13T23:52:20.877 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.__get_result() 2022-05-13T23:52:20.877 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-13T23:52:20.877 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise self._exception 2022-05-13T23:52:20.877 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-13T23:52:20.878 INFO:journalctl@ceph.mon.c.smithi115.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-13T23:52:20.878 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-13T23:52:20.878 INFO:journalctl@ceph.mon.c.smithi115.stdout: conn = await self._remote_connection(host, addr) 2022-05-13T23:52:20.878 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-13T23:52:20.878 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise 2022-05-13T23:52:20.879 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-13T23:52:20.879 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.gen.throw(type, value, traceback) 2022-05-13T23:52:20.879 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-13T23:52:20.879 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OrchestratorError(msg) 2022-05-13T23:52:20.880 INFO:journalctl@ceph.mon.c.smithi115.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:52:20.880 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:52:20 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:52:21.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:52:21 smithi047 ceph-mon[77147]: pgmap v1089: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:52:21.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:52:21 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:52:21.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:52:21 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-13T23:52:21.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:52:21 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:52:21.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:52:21 smithi115 conmon[90393]: 2022-05-13T23:52:21.541+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:52:21.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:52:21 smithi115 ceph-mon[78593]: pgmap v1089: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:52:21.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:52:21 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:52:21.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:52:21 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-13T23:52:21.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:52:21 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:52:22.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:52:22 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:52:22.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:52:22 smithi115 conmon[90393]: 2022-05-13T23:52:22.565+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:52:22.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:52:22 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:52:23.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:52:23 smithi047 ceph-mon[77147]: pgmap v1090: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:52:23.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:52:23 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:52:23.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:52:23 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 1475 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:52:23.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:52:23 smithi115 conmon[90393]: 2022-05-13T23:52:23.567+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:52:23.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:52:23 smithi115 ceph-mon[78593]: pgmap v1090: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:52:23.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:52:23 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:52:23.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:52:23 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 1475 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:52:24.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:52:24 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:52:24.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:52:24 smithi115 conmon[90393]: 2022-05-13T23:52:24.584+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:52:24.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:52:24 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:52:25.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:52:25 smithi047 ceph-mon[77147]: pgmap v1091: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:52:25.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:52:25 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:52:25.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:52:25 smithi115 conmon[90393]: 2022-05-13T23:52:25.614+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:52:25.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:52:25 smithi115 ceph-mon[78593]: pgmap v1091: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:52:25.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:52:25 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:52:26.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:52:26 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:52:26.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:52:26 smithi115 conmon[90393]: 2022-05-13T23:52:26.657+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:52:26.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:52:26 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:52:27.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:52:27 smithi047 ceph-mon[77147]: pgmap v1092: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:52:27.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:52:27 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:52:27.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:52:27 smithi115 conmon[90393]: 2022-05-13T23:52:27.613+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:52:27.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:52:27 smithi115 ceph-mon[78593]: pgmap v1092: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:52:27.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:52:27 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:52:28.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:52:28 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:52:28.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:52:28 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 1480 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:52:28.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:52:28 smithi115 conmon[90393]: 2022-05-13T23:52:28.622+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:52:28.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:52:28 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:52:28.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:52:28 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 1480 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:52:29.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:52:29 smithi047 ceph-mon[77147]: pgmap v1093: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:52:29.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:52:29 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:52:29.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:52:29 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-13T23:52:29.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:52:29 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:52:29.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:52:29 smithi115 conmon[90393]: 2022-05-13T23:52:29.632+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:52:29.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:52:29 smithi115 ceph-mon[78593]: pgmap v1093: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:52:29.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:52:29 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:52:29.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:52:29 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-13T23:52:29.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:52:29 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:52:30.802 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:52:30 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:52:30.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:52:30 smithi115 conmon[90393]: 2022-05-13T23:52:30.665+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:52:30.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:52:30 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:52:31.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:52:31 smithi115 conmon[90393]: 2022-05-13T23:52:31.697+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:52:31.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:52:31 smithi115 ceph-mon[78593]: pgmap v1094: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:52:31.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:52:31 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:52:32.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:52:31 smithi047 ceph-mon[77147]: pgmap v1094: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:52:32.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:52:31 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:52:32.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:52:32 smithi115 conmon[90393]: 2022-05-13T23:52:32.678+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:52:32.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:52:32 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:52:33.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:52:32 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:52:33.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:52:33 smithi115 conmon[90393]: 2022-05-13T23:52:33.651+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:52:33.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:52:33 smithi115 ceph-mon[78593]: pgmap v1095: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:52:33.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:52:33 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:52:33.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:52:33 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-13T23:52:33.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:52:33 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:52:33.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:52:33 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 1485 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:52:34.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:52:33 smithi047 ceph-mon[77147]: pgmap v1095: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:52:34.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:52:33 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:52:34.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:52:33 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-13T23:52:34.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:52:33 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:52:34.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:52:33 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 1485 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:52:34.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:52:34 smithi115 conmon[90393]: 2022-05-13T23:52:34.624+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:52:34.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:52:34 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:52:35.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:52:34 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:52:35.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:52:35 smithi115 conmon[90393]: 2022-05-13T23:52:35.598+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:52:35.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:52:35 smithi115 ceph-mon[78593]: pgmap v1096: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:52:35.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:52:35 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:52:36.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:52:35 smithi047 ceph-mon[77147]: pgmap v1096: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:52:36.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:52:35 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:52:36.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:52:36 smithi115 conmon[90393]: 2022-05-13T23:52:36.578+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:52:36.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:52:36 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:52:37.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:52:36 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:52:37.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:52:37 smithi115 conmon[90393]: 2022-05-13T23:52:37.593+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:52:37.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:52:37 smithi115 ceph-mon[78593]: pgmap v1097: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:52:37.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:52:37 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:52:37.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:52:37 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-13T23:52:37.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:52:37 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:52:38.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:52:37 smithi047 ceph-mon[77147]: pgmap v1097: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:52:38.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:52:37 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:52:38.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:52:37 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-13T23:52:38.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:52:37 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:52:38.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:52:38 smithi115 conmon[90393]: 2022-05-13T23:52:38.559+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:52:38.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:52:38 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:52:38.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:52:38 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 1490 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:52:39.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:52:38 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:52:39.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:52:38 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 1490 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:52:39.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:52:39 smithi115 conmon[90393]: 2022-05-13T23:52:39.586+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:52:39.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:52:39 smithi115 ceph-mon[78593]: pgmap v1098: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:52:39.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:52:39 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:52:40.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:52:39 smithi047 ceph-mon[77147]: pgmap v1098: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:52:40.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:52:39 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:52:40.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:52:40 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:52:40.860 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:52:40 smithi115 conmon[90393]: 2022-05-13T23:52:40.567+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:52:41.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:52:40 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:52:41.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:52:41 smithi115 conmon[90393]: 2022-05-13T23:52:41.576+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:52:41.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:52:41 smithi115 ceph-mon[78593]: pgmap v1099: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:52:41.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:52:41 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:52:41.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:52:41 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:52:41.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:52:41 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-13T23:52:42.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:52:41 smithi047 ceph-mon[77147]: pgmap v1099: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:52:42.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:52:41 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:52:42.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:52:41 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:52:42.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:52:41 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-13T23:52:42.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:52:42 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:52:42.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:52:42 smithi115 conmon[90393]: 2022-05-13T23:52:42.590+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:52:43.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:52:42 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:52:43.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:52:43 smithi115 conmon[90393]: 2022-05-13T23:52:43.603+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:52:43.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:52:43 smithi115 ceph-mon[78593]: pgmap v1100: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:52:43.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:52:43 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:52:43.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:52:43 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 1495 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:52:44.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:52:43 smithi047 ceph-mon[77147]: pgmap v1100: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:52:44.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:52:43 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:52:44.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:52:43 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 1495 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:52:44.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:52:44 smithi115 conmon[90393]: 2022-05-13T23:52:44.617+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:52:44.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:52:44 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:52:45.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:52:44 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:52:45.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:52:45 smithi115 conmon[90393]: 2022-05-13T23:52:45.655+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:52:45.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:52:45 smithi115 ceph-mon[78593]: pgmap v1101: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:52:45.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:52:45 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:52:46.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:52:45 smithi047 ceph-mon[77147]: pgmap v1101: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:52:46.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:52:45 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:52:46.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:52:46 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:52:46.865 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:52:46 smithi115 conmon[90393]: 2022-05-13T23:52:46.629+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:52:46.865 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:52:46 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:52:47.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:52:47 smithi115 conmon[90393]: 2022-05-13T23:52:47.655+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:52:47.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:52:47 smithi115 ceph-mon[78593]: pgmap v1102: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:52:47.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:52:47 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:52:48.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:52:47 smithi047 ceph-mon[77147]: pgmap v1102: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:52:48.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:52:47 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:52:48.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:52:48 smithi115 conmon[90393]: 2022-05-13T23:52:48.688+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:52:48.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:52:48 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:52:48.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:52:48 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 1500 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:52:49.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:52:48 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:52:49.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:52:48 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 1500 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:52:49.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:52:49 smithi115 conmon[90393]: 2022-05-13T23:52:49.672+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:52:49.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:52:49 smithi115 ceph-mon[78593]: pgmap v1103: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:52:49.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:52:49 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:52:49.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:52:49 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-13T23:52:49.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:52:49 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:52:50.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:52:49 smithi047 ceph-mon[77147]: pgmap v1103: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:52:50.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:52:49 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:52:50.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:52:49 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-13T23:52:50.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:52:49 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:52:50.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:52:50 smithi115 conmon[90393]: 2022-05-13T23:52:50.684+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:52:50.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:52:50 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:52:51.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:52:50 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:52:52.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:52:51 smithi047 ceph-mon[77147]: pgmap v1104: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:52:52.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:52:51 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:52:52.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:52:51 smithi115 conmon[90393]: 2022-05-13T23:52:51.674+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:52:52.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:52:51 smithi115 ceph-mon[78593]: pgmap v1104: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:52:52.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:52:51 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:52:53.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:52:52 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:52:53.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:52:52 smithi115 conmon[90393]: 2022-05-13T23:52:52.645+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:52:53.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:52:52 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:52:54.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:52:53 smithi047 ceph-mon[77147]: pgmap v1105: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:52:54.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:52:53 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:52:54.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:52:53 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-13T23:52:54.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:52:53 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:52:54.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:52:53 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 1505 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:52:54.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:52:53 smithi115 conmon[90393]: 2022-05-13T23:52:53.689+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:52:54.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:52:53 smithi115 ceph-mon[78593]: pgmap v1105: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:52:54.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:52:53 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:52:54.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:52:53 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-13T23:52:54.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:52:53 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:52:54.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:52:53 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 1505 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:52:55.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:52:54 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:52:55.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:52:54 smithi115 conmon[90393]: 2022-05-13T23:52:54.659+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:52:55.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:52:54 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:52:56.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:52:55 smithi047 ceph-mon[77147]: pgmap v1106: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:52:56.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:52:55 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:52:56.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:52:55 smithi115 conmon[90393]: 2022-05-13T23:52:55.680+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:52:56.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:52:55 smithi115 ceph-mon[78593]: pgmap v1106: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:52:56.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:52:55 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:52:57.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:52:56 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:52:57.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:52:56 smithi115 conmon[90393]: 2022-05-13T23:52:56.668+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:52:57.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:52:56 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:52:58.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:52:57 smithi047 ceph-mon[77147]: pgmap v1107: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:52:58.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:52:57 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:52:58.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:52:57 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-13T23:52:58.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:52:57 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:52:58.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:52:57 smithi115 conmon[90393]: 2022-05-13T23:52:57.713+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:52:58.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:52:57 smithi115 ceph-mon[78593]: pgmap v1107: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:52:58.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:52:57 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:52:58.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:52:57 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-13T23:52:58.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:52:57 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:52:59.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:52:58 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:52:59.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:52:58 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 1510 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:52:59.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:52:58 smithi115 conmon[90393]: 2022-05-13T23:52:58.685+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:52:59.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:52:58 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:52:59.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:52:58 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 1510 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:53:00.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:52:59 smithi047 ceph-mon[77147]: pgmap v1108: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:53:00.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:52:59 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:53:00.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:52:59 smithi115 conmon[90393]: 2022-05-13T23:52:59.673+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:53:00.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:52:59 smithi115 ceph-mon[78593]: pgmap v1108: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:53:00.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:52:59 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:53:01.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:53:00 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:53:01.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:53:00 smithi115 conmon[90393]: 2022-05-13T23:53:00.649+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:53:01.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:53:00 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:53:02.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:53:01 smithi047 ceph-mon[77147]: pgmap v1109: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:53:02.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:53:01 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:53:02.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:53:01 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-13T23:53:02.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:53:01 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:53:02.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:53:01 smithi115 conmon[90393]: 2022-05-13T23:53:01.611+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:53:02.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:53:01 smithi115 ceph-mon[78593]: pgmap v1109: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:53:02.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:53:01 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:53:02.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:53:01 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-13T23:53:02.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:53:01 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:53:03.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:53:02 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:53:03.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:53:02 smithi115 conmon[90393]: 2022-05-13T23:53:02.629+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:53:03.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:53:02 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:53:03.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:53:03 smithi115 conmon[90393]: 2022-05-13T23:53:03.595+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:53:03.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:53:03 smithi115 ceph-mon[78593]: pgmap v1110: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:53:03.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:53:03 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:53:03.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:53:03 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 1515 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:53:04.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:53:03 smithi047 ceph-mon[77147]: pgmap v1110: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:53:04.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:53:03 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:53:04.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:53:03 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 1515 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:53:05.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:53:04 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:53:05.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:53:04 smithi115 conmon[90393]: 2022-05-13T23:53:04.612+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:53:05.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:53:04 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:53:06.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:53:05 smithi047 ceph-mon[77147]: pgmap v1111: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:53:06.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:53:05 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:53:06.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:53:05 smithi115 conmon[90393]: 2022-05-13T23:53:05.659+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:53:06.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:53:05 smithi115 ceph-mon[78593]: pgmap v1111: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:53:06.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:53:05 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:53:07.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:53:06 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:53:07.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:53:06 smithi115 conmon[90393]: 2022-05-13T23:53:06.619+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:53:07.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:53:06 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:53:07.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:53:07 smithi115 conmon[90393]: 2022-05-13T23:53:07.598+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:53:07.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:53:07 smithi115 ceph-mon[78593]: pgmap v1112: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:53:07.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:53:07 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:53:08.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:53:07 smithi047 ceph-mon[77147]: pgmap v1112: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:53:08.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:53:07 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:53:08.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:53:08 smithi115 conmon[90393]: 2022-05-13T23:53:08.595+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:53:08.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:53:08 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:53:08.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:53:08 smithi115 ceph-mon[78593]: pgmap v1113: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:53:08.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:53:08 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 1520 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:53:09.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:53:08 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:53:09.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:53:08 smithi047 ceph-mon[77147]: pgmap v1113: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:53:09.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:53:08 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 1520 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:53:10.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:53:09 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:53:10.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:53:09 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:53:10.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:53:09 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-13T23:53:10.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:53:09 smithi115 conmon[90393]: 2022-05-13T23:53:09.614+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:53:10.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:53:09 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:53:10.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:53:09 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:53:10.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:53:09 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-13T23:53:11.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:53:10 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:53:11.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:53:10 smithi047 ceph-mon[77147]: pgmap v1114: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:53:11.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:53:10 smithi115 conmon[90393]: 2022-05-13T23:53:10.662+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:53:11.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:53:10 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:53:11.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:53:10 smithi115 ceph-mon[78593]: pgmap v1114: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:53:12.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:53:11 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:53:12.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:53:11 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:53:12.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:53:11 smithi115 conmon[90393]: 2022-05-13T23:53:11.629+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:53:13.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:53:12 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:53:13.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:53:12 smithi047 ceph-mon[77147]: pgmap v1115: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:53:13.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:53:12 smithi115 conmon[90393]: 2022-05-13T23:53:12.653+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:53:13.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:53:12 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:53:13.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:53:12 smithi115 ceph-mon[78593]: pgmap v1115: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:53:14.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:53:13 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:53:14.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:53:13 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:53:14.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:53:13 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-13T23:53:14.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:53:13 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-13T23:53:14.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:53:13 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-13T23:53:14.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:53:13 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 1525 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:53:14.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:53:13 smithi115 conmon[90393]: 2022-05-13T23:53:13.661+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:53:14.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:53:13 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:53:14.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:53:13 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:53:14.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:53:13 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-13T23:53:14.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:53:13 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-13T23:53:14.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:53:13 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-13T23:53:14.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:53:13 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 1525 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:53:15.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:53:14 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:53:15.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:53:14 smithi047 ceph-mon[77147]: pgmap v1116: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:53:15.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:53:14 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:53:15.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:53:14 smithi115 ceph-mon[78593]: pgmap v1116: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:53:15.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:53:14 smithi115 conmon[90393]: 2022-05-13T23:53:14.708+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:53:16.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:53:15 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:53:16.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:53:15 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:53:16.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:53:15 smithi115 conmon[90393]: 2022-05-13T23:53:15.722+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:53:17.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:53:16 smithi047 ceph-mon[77147]: pgmap v1117: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:53:17.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:53:16 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:53:17.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:53:16 smithi115 ceph-mon[78593]: pgmap v1117: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:53:17.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:53:16 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:53:17.110 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:53:16 smithi115 conmon[90393]: 2022-05-13T23:53:16.769+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:53:18.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:53:17 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:53:18.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:53:17 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-13T23:53:18.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:53:17 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:53:18.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:53:17 smithi115 conmon[90393]: 2022-05-13T23:53:17.799+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:53:18.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:53:17 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:53:18.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:53:17 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-13T23:53:18.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:53:17 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:53:18.950 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:53:18 smithi047 ceph-mon[77147]: pgmap v1118: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:53:18.951 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:53:18 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:53:18.951 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:53:18 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 1530 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:53:18.952 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:53:18 smithi115 ceph-mon[78593]: pgmap v1118: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:53:18.952 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:53:18 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:53:18.952 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:53:18 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 1530 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:53:18.953 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:53:18 smithi115 conmon[90393]: 2022-05-13T23:53:18.774+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:53:19.701 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:53:19 smithi047 conmon[78521]: 2022-05-13T23:53:19.343+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:53:19.701 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:53:19 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-13T23:53:19.702 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:53:19 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-13T23:53:19.702 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:53:19 smithi047 conmon[78521]: yield 2022-05-13T23:53:19.702 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:53:19 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-13T23:53:19.702 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:53:19 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-13T23:53:19.703 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:53:19 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-13T23:53:19.703 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:53:19 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-13T23:53:19.703 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:53:19 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-13T23:53:19.703 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:53:19 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-13T23:53:19.704 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:53:19 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-13T23:53:19.704 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:53:19 smithi047 conmon[78521]: raise exceptions[0] 2022-05-13T23:53:19.704 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:53:19 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-13T23:53:19.705 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:53:19 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-13T23:53:19.705 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:53:19 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-13T23:53:19.705 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:53:19 smithi047 conmon[78521]: return (yield from fut) 2022-05-13T23:53:19.705 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:53:19 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-13T23:53:19.706 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:53:19 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-13T23:53:19.706 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:53:19 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:53:20.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:53:19 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:53:20.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:53:19 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-13T23:53:20.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:53:19 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-13T23:53:20.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:53:19 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-13T23:53:20.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:53:19 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:53:20.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-13T23:53:20.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-13T23:53:20.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-13T23:53:20.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-13T23:53:20.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-13T23:53:20.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-13T23:53:20.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-13T23:53:20.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-13T23:53:20.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-13T23:53:20.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-13T23:53:20.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-13T23:53:20.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-13T23:53:20.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-13T23:53:20.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-13T23:53:20.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-13T23:53:20.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-13T23:53:20.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-13T23:53:20.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:53:20.048 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:53:19 smithi047 ceph-mon[77147]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-13T23:53:20.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:53:19 smithi115 conmon[90393]: 2022-05-13T23:53:19.737+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:53:20.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:53:19 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:53:20.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:53:19 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-13T23:53:20.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:53:19 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-13T23:53:20.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:53:19 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-13T23:53:20.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:53:19 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:53:20.109 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-13T23:53:20.109 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-13T23:53:20.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-13T23:53:20.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-13T23:53:20.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-13T23:53:20.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-13T23:53:20.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-13T23:53:20.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-13T23:53:20.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-13T23:53:20.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-13T23:53:20.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-13T23:53:20.113 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-13T23:53:20.113 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-13T23:53:20.113 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-13T23:53:20.113 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-13T23:53:20.114 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-13T23:53:20.114 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-13T23:53:20.114 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:53:20.114 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:53:19 smithi115 ceph-mon[78593]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-13T23:53:21.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:53:20 smithi047 ceph-mon[77147]: pgmap v1119: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:53:21.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:53:20 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:53:21.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:53:20 smithi115 conmon[90393]: 2022-05-13T23:53:20.771+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:53:21.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:53:20 smithi115 ceph-mon[78593]: pgmap v1119: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:53:21.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:53:20 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:53:22.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:53:21 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:53:22.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:53:21 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-13T23:53:22.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:53:21 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:53:22.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:53:21 smithi115 conmon[90393]: 2022-05-13T23:53:21.806+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:53:22.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:53:21 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:53:22.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:53:21 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-13T23:53:22.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:53:21 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:53:22.761 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:53:22 smithi047 conmon[78521]: 2022-05-13T23:53:22.415+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:53:22.762 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:53:22 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-13T23:53:22.762 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:53:22 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-13T23:53:22.762 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:53:22 smithi047 conmon[78521]: yield 2022-05-13T23:53:22.762 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:53:22 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-13T23:53:22.762 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:53:22 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-13T23:53:22.763 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:53:22 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-13T23:53:22.763 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:53:22 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-13T23:53:22.763 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:53:22 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-13T23:53:22.763 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:53:22 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-13T23:53:22.764 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:53:22 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-13T23:53:22.764 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:53:22 smithi047 conmon[78521]: raise exceptions[0] 2022-05-13T23:53:22.764 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:53:22 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-13T23:53:22.764 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:53:22 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-13T23:53:22.764 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:53:22 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-13T23:53:22.765 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:53:22 smithi047 conmon[78521]: return (yield from fut) 2022-05-13T23:53:22.765 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:53:22 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-13T23:53:22.765 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:53:22 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-13T23:53:22.765 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:53:22 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:53:22.766 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:53:22 smithi047 conmon[78521]: 2022-05-13T23:53:22.416+0000 7f5340f7a700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-13T23:53:22.766 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:53:22 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-13T23:53:22.766 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:53:22 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-13T23:53:22.767 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:53:22 smithi047 conmon[78521]: yield 2022-05-13T23:53:22.767 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:53:22 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-13T23:53:22.767 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:53:22 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-13T23:53:22.769 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:53:22 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-13T23:53:22.769 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:53:22 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-13T23:53:22.770 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:53:22 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-13T23:53:22.770 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:53:22 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-13T23:53:22.771 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:53:22 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-13T23:53:22.771 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:53:22 smithi047 conmon[78521]: raise exceptions[0] 2022-05-13T23:53:22.771 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:53:22 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-13T23:53:22.771 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:53:22 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-13T23:53:22.772 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:53:22 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-13T23:53:22.772 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:53:22 smithi047 conmon[78521]: return (yield from fut) 2022-05-13T23:53:22.772 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:53:22 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-13T23:53:22.773 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:53:22 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-13T23:53:22.773 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:53:22 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:53:22.773 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:53:22 smithi047 conmon[78521]: 2022-05-13T23:53:22.773 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:53:22 smithi047 conmon[78521]: During handling of the above exception, another exception occurred: 2022-05-13T23:53:22.773 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:53:22 smithi047 conmon[78521]: 2022-05-13T23:53:22.774 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:53:22 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-13T23:53:22.774 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:53:22 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-13T23:53:22.774 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:53:22 smithi047 conmon[78521]: return f(*arg) 2022-05-13T23:53:22.774 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:53:22 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-13T23:53:22.775 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:53:22 smithi047 conmon[78521]: self._write_client_files(client_files, host) 2022-05-13T23:53:22.775 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:53:22 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-13T23:53:22.775 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:53:22 smithi047 conmon[78521]: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-13T23:53:22.775 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:53:22 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-13T23:53:22.776 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:53:22 smithi047 conmon[78521]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-13T23:53:22.776 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:53:22 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-13T23:53:22.776 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:53:22 smithi047 conmon[78521]: return self.event_loop.get_result(coro) 2022-05-13T23:53:22.776 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:53:22 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-13T23:53:22.776 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:53:22 smithi047 conmon[78521]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-13T23:53:22.777 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:53:22 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-13T23:53:22.777 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:53:22 smithi047 conmon[78521]: return self.__get_result() 2022-05-13T23:53:22.777 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:53:22 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-13T23:53:22.777 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:53:22 smithi047 conmon[78521]: raise self._exception 2022-05-13T23:53:22.778 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:53:22 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-13T23:53:22.781 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:53:22 smithi047 conmon[78521]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-13T23:53:22.781 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:53:22 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-13T23:53:22.781 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:53:22 smithi047 conmon[78521]: conn = await self._remote_connection(host, addr) 2022-05-13T23:53:22.781 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:53:22 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-13T23:53:22.782 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:53:22 smithi047 conmon[78521]: raise 2022-05-13T23:53:22.782 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:53:22 smithi047 conmon[78521]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-13T23:53:22.782 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:53:22 smithi047 conmon[78521]: self.gen.throw(type, value, traceback) 2022-05-13T23:53:22.782 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:53:22 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-13T23:53:22.783 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:53:22 smithi047 conmon[78521]: raise OrchestratorError(msg) 2022-05-13T23:53:22.783 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:53:22 smithi047 conmon[78521]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:53:23.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:53:22 smithi047 ceph-mon[77147]: pgmap v1120: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:53:23.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:53:22 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:53:23.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:53:22 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:53:23.042 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-13T23:53:23.042 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-13T23:53:23.042 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-13T23:53:23.042 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-13T23:53:23.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-13T23:53:23.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-13T23:53:23.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-13T23:53:23.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-13T23:53:23.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-13T23:53:23.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-13T23:53:23.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-13T23:53:23.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-13T23:53:23.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-13T23:53:23.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-13T23:53:23.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-13T23:53:23.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-13T23:53:23.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-13T23:53:23.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:53:23.046 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:53:22 smithi047 ceph-mon[77147]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-13T23:53:23.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-13T23:53:23.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-13T23:53:23.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-13T23:53:23.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-13T23:53:23.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-13T23:53:23.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-13T23:53:23.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-13T23:53:23.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-13T23:53:23.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-13T23:53:23.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-13T23:53:23.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-13T23:53:23.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-13T23:53:23.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-13T23:53:23.049 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-13T23:53:23.049 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-13T23:53:23.049 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-13T23:53:23.049 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-13T23:53:23.049 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:53:23.050 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-13T23:53:23.050 INFO:journalctl@ceph.mon.b.smithi047.stdout: During handling of the above exception, another exception occurred: 2022-05-13T23:53:23.050 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-13T23:53:23.050 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-13T23:53:23.050 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-13T23:53:23.051 INFO:journalctl@ceph.mon.b.smithi047.stdout: return f(*arg) 2022-05-13T23:53:23.051 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-13T23:53:23.051 INFO:journalctl@ceph.mon.b.smithi047.stdout: self._write_client_files(client_files, host) 2022-05-13T23:53:23.051 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-13T23:53:23.052 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-13T23:53:23.052 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-13T23:53:23.052 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-13T23:53:23.052 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-13T23:53:23.052 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.event_loop.get_result(coro) 2022-05-13T23:53:23.053 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-13T23:53:23.053 INFO:journalctl@ceph.mon.b.smithi047.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-13T23:53:23.053 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-13T23:53:23.053 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.__get_result() 2022-05-13T23:53:23.054 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-13T23:53:23.054 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise self._exception 2022-05-13T23:53:23.054 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-13T23:53:23.054 INFO:journalctl@ceph.mon.b.smithi047.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-13T23:53:23.054 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-13T23:53:23.055 INFO:journalctl@ceph.mon.b.smithi047.stdout: conn = await self._remote_connection(host, addr) 2022-05-13T23:53:23.055 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-13T23:53:23.055 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise 2022-05-13T23:53:23.056 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-13T23:53:23.056 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.gen.throw(type, value, traceback) 2022-05-13T23:53:23.059 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-13T23:53:23.060 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OrchestratorError(msg) 2022-05-13T23:53:23.060 INFO:journalctl@ceph.mon.b.smithi047.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:53:23.110 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:53:22 smithi115 conmon[90393]: 2022-05-13T23:53:22.776+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:53:23.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:53:22 smithi115 ceph-mon[78593]: pgmap v1120: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:53:23.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:53:22 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:53:23.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:53:22 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:53:23.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-13T23:53:23.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-13T23:53:23.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-13T23:53:23.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-13T23:53:23.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-13T23:53:23.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-13T23:53:23.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-13T23:53:23.113 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-13T23:53:23.113 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-13T23:53:23.113 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-13T23:53:23.113 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-13T23:53:23.114 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-13T23:53:23.114 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-13T23:53:23.114 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-13T23:53:23.114 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-13T23:53:23.114 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-13T23:53:23.115 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-13T23:53:23.116 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:53:23.116 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:53:22 smithi115 ceph-mon[78593]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-13T23:53:23.116 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-13T23:53:23.116 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-13T23:53:23.117 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-13T23:53:23.117 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-13T23:53:23.117 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-13T23:53:23.118 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-13T23:53:23.118 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-13T23:53:23.118 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-13T23:53:23.118 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-13T23:53:23.119 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-13T23:53:23.119 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-13T23:53:23.119 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-13T23:53:23.119 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-13T23:53:23.120 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-13T23:53:23.120 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-13T23:53:23.120 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-13T23:53:23.120 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-13T23:53:23.121 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:53:23.121 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-13T23:53:23.121 INFO:journalctl@ceph.mon.c.smithi115.stdout: During handling of the above exception, another exception occurred: 2022-05-13T23:53:23.121 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-13T23:53:23.121 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-13T23:53:23.122 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-13T23:53:23.122 INFO:journalctl@ceph.mon.c.smithi115.stdout: return f(*arg) 2022-05-13T23:53:23.122 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-13T23:53:23.122 INFO:journalctl@ceph.mon.c.smithi115.stdout: self._write_client_files(client_files, host) 2022-05-13T23:53:23.123 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-13T23:53:23.123 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-13T23:53:23.123 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-13T23:53:23.124 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-13T23:53:23.124 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-13T23:53:23.124 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.event_loop.get_result(coro) 2022-05-13T23:53:23.125 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-13T23:53:23.125 INFO:journalctl@ceph.mon.c.smithi115.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-13T23:53:23.125 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-13T23:53:23.125 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.__get_result() 2022-05-13T23:53:23.125 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-13T23:53:23.126 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise self._exception 2022-05-13T23:53:23.126 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-13T23:53:23.126 INFO:journalctl@ceph.mon.c.smithi115.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-13T23:53:23.126 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-13T23:53:23.127 INFO:journalctl@ceph.mon.c.smithi115.stdout: conn = await self._remote_connection(host, addr) 2022-05-13T23:53:23.127 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-13T23:53:23.127 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise 2022-05-13T23:53:23.127 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-13T23:53:23.128 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.gen.throw(type, value, traceback) 2022-05-13T23:53:23.128 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-13T23:53:23.128 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OrchestratorError(msg) 2022-05-13T23:53:23.129 INFO:journalctl@ceph.mon.c.smithi115.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:53:24.046 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:53:23 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:53:24.047 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:53:23 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 1535 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:53:24.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:53:23 smithi115 conmon[90393]: 2022-05-13T23:53:23.793+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:53:24.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:53:23 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:53:24.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:53:23 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 1535 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:53:25.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:53:24 smithi047 ceph-mon[77147]: pgmap v1121: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:53:25.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:53:24 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:53:25.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:53:24 smithi115 conmon[90393]: 2022-05-13T23:53:24.779+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:53:25.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:53:24 smithi115 ceph-mon[78593]: pgmap v1121: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:53:25.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:53:24 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:53:26.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:53:25 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:53:26.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:53:25 smithi115 conmon[90393]: 2022-05-13T23:53:25.797+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:53:26.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:53:25 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:53:27.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:53:26 smithi047 ceph-mon[77147]: pgmap v1122: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:53:27.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:53:26 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:53:27.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:53:26 smithi115 conmon[90393]: 2022-05-13T23:53:26.816+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:53:27.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:53:26 smithi115 ceph-mon[78593]: pgmap v1122: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:53:27.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:53:26 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:53:28.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:53:27 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:53:28.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:53:27 smithi115 conmon[90393]: 2022-05-13T23:53:27.783+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:53:28.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:53:27 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:53:29.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:53:28 smithi047 ceph-mon[77147]: pgmap v1123: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:53:29.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:53:28 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:53:29.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:53:28 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 1540 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:53:29.110 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:53:28 smithi115 conmon[90393]: 2022-05-13T23:53:28.760+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:53:29.115 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:53:28 smithi115 ceph-mon[78593]: pgmap v1123: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:53:29.115 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:53:28 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:53:29.116 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:53:28 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 1540 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:53:30.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:53:29 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:53:30.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:53:29 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:53:30.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:53:29 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-13T23:53:30.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:53:29 smithi115 conmon[90393]: 2022-05-13T23:53:29.766+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:53:30.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:53:29 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:53:30.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:53:29 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:53:30.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:53:29 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-13T23:53:31.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:53:30 smithi047 ceph-mon[77147]: pgmap v1124: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:53:31.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:53:30 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:53:31.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:53:30 smithi115 conmon[90393]: 2022-05-13T23:53:30.754+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:53:31.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:53:30 smithi115 ceph-mon[78593]: pgmap v1124: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:53:31.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:53:30 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:53:32.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:53:31 smithi115 conmon[90393]: 2022-05-13T23:53:31.747+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:53:32.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:53:31 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:53:32.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:53:31 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:53:33.110 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:53:32 smithi115 conmon[90393]: 2022-05-13T23:53:32.720+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:53:33.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:53:32 smithi115 ceph-mon[78593]: pgmap v1125: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:53:33.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:53:32 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:53:33.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:53:32 smithi047 ceph-mon[77147]: pgmap v1125: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:53:33.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:53:32 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:53:34.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:53:33 smithi115 conmon[90393]: 2022-05-13T23:53:33.767+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:53:34.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:53:33 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:53:34.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:53:33 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:53:34.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:53:33 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-13T23:53:34.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:53:33 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 1545 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:53:34.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:53:33 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:53:34.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:53:33 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:53:34.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:53:33 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-13T23:53:34.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:53:33 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 1545 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:53:35.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:53:34 smithi115 conmon[90393]: 2022-05-13T23:53:34.757+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:53:35.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:53:34 smithi115 ceph-mon[78593]: pgmap v1126: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:53:35.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:53:34 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:53:35.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:53:34 smithi047 ceph-mon[77147]: pgmap v1126: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:53:35.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:53:34 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:53:36.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:53:35 smithi115 conmon[90393]: 2022-05-13T23:53:35.794+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:53:36.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:53:35 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:53:36.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:53:35 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:53:37.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:53:36 smithi115 conmon[90393]: 2022-05-13T23:53:36.798+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:53:37.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:53:36 smithi115 ceph-mon[78593]: pgmap v1127: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:53:37.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:53:36 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:53:37.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:53:36 smithi047 ceph-mon[77147]: pgmap v1127: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:53:37.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:53:36 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:53:38.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:53:37 smithi115 conmon[90393]: 2022-05-13T23:53:37.775+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:53:38.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:53:37 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:53:38.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:53:37 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:53:38.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:53:37 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-13T23:53:38.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:53:37 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:53:38.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:53:37 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:53:38.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:53:37 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-13T23:53:39.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:53:38 smithi115 conmon[90393]: 2022-05-13T23:53:38.804+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:53:39.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:53:38 smithi115 ceph-mon[78593]: pgmap v1128: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:53:39.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:53:38 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:53:39.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:53:38 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 1550 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:53:39.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:53:38 smithi047 ceph-mon[77147]: pgmap v1128: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:53:39.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:53:38 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:53:39.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:53:38 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 1550 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:53:40.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:53:39 smithi115 conmon[90393]: 2022-05-13T23:53:39.813+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:53:40.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:53:39 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:53:40.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:53:39 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:53:41.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:53:40 smithi115 conmon[90393]: 2022-05-13T23:53:40.852+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:53:41.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:53:40 smithi115 ceph-mon[78593]: pgmap v1129: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:53:41.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:53:40 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:53:41.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:53:40 smithi047 ceph-mon[77147]: pgmap v1129: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:53:41.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:53:40 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:53:42.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:53:41 smithi115 conmon[90393]: 2022-05-13T23:53:41.861+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:53:42.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:53:41 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:53:42.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:53:41 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-13T23:53:42.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:53:41 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:53:42.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:53:41 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:53:42.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:53:41 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-13T23:53:42.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:53:41 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:53:43.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:53:42 smithi115 conmon[90393]: 2022-05-13T23:53:42.871+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:53:43.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:53:42 smithi115 ceph-mon[78593]: pgmap v1130: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:53:43.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:53:42 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:53:43.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:53:42 smithi047 ceph-mon[77147]: pgmap v1130: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:53:43.382 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:53:42 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:53:44.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:53:43 smithi115 conmon[90393]: 2022-05-13T23:53:43.917+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:53:44.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:53:43 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:53:44.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:53:43 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 1555 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:53:44.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:53:43 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:53:44.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:53:43 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 1555 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:53:45.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:53:44 smithi115 conmon[90393]: 2022-05-13T23:53:44.872+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:53:45.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:53:44 smithi115 ceph-mon[78593]: pgmap v1131: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:53:45.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:53:44 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:53:45.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:53:44 smithi047 ceph-mon[77147]: pgmap v1131: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:53:45.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:53:44 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:53:46.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:53:45 smithi115 conmon[90393]: 2022-05-13T23:53:45.825+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:53:46.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:53:45 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:53:46.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:53:45 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:53:47.111 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:53:46 smithi115 conmon[90393]: 2022-05-13T23:53:46.862+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:53:47.112 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:53:46 smithi115 ceph-mon[78593]: pgmap v1132: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:53:47.112 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:53:46 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:53:47.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:53:46 smithi047 ceph-mon[77147]: pgmap v1132: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:53:47.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:53:46 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:53:48.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:53:47 smithi115 conmon[90393]: 2022-05-13T23:53:47.824+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:53:48.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:53:47 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:53:48.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:53:47 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:53:49.112 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:53:48 smithi115 conmon[90393]: 2022-05-13T23:53:48.833+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:53:49.113 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:53:48 smithi115 ceph-mon[78593]: pgmap v1133: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:53:49.114 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:53:48 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:53:49.114 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:53:48 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 1560 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:53:49.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:53:48 smithi047 ceph-mon[77147]: pgmap v1133: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:53:49.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:53:48 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:53:49.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:53:48 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 1560 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:53:50.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:53:49 smithi115 conmon[90393]: 2022-05-13T23:53:49.785+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:53:50.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:53:49 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:53:50.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:53:49 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:53:50.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:53:49 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-13T23:53:50.297 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:53:49 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:53:50.298 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:53:49 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:53:50.298 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:53:49 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-13T23:53:51.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:53:50 smithi115 conmon[90393]: 2022-05-13T23:53:50.770+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:53:51.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:53:50 smithi115 ceph-mon[78593]: pgmap v1134: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:53:51.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:53:50 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:53:51.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:53:50 smithi047 ceph-mon[77147]: pgmap v1134: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:53:51.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:53:50 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:53:52.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:53:51 smithi115 conmon[90393]: 2022-05-13T23:53:51.770+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:53:52.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:53:51 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:53:52.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:53:51 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:53:53.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:53:52 smithi115 conmon[90393]: 2022-05-13T23:53:52.793+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:53:53.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:53:52 smithi115 ceph-mon[78593]: pgmap v1135: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:53:53.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:53:52 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:53:53.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:53:52 smithi047 ceph-mon[77147]: pgmap v1135: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:53:53.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:53:52 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:53:54.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:53:53 smithi115 conmon[90393]: 2022-05-13T23:53:53.802+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:53:54.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:53:53 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:53:54.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:53:53 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:53:54.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:53:53 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-13T23:53:54.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:53:53 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 1565 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:53:54.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:53:53 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:53:54.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:53:53 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:53:54.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:53:53 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-13T23:53:54.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:53:53 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 1565 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:53:55.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:53:54 smithi115 conmon[90393]: 2022-05-13T23:53:54.781+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:53:55.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:53:54 smithi115 ceph-mon[78593]: pgmap v1136: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:53:55.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:53:54 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:53:55.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:53:54 smithi047 ceph-mon[77147]: pgmap v1136: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:53:55.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:53:54 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:53:56.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:53:55 smithi115 conmon[90393]: 2022-05-13T23:53:55.785+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:53:56.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:53:55 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:53:56.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:53:55 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:53:57.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:53:56 smithi115 conmon[90393]: 2022-05-13T23:53:56.751+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:53:57.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:53:56 smithi115 ceph-mon[78593]: pgmap v1137: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:53:57.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:53:56 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:53:57.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:53:56 smithi047 ceph-mon[77147]: pgmap v1137: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:53:57.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:53:56 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:53:58.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:53:57 smithi115 conmon[90393]: 2022-05-13T23:53:57.712+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:53:58.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:53:57 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:53:58.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:53:57 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:53:58.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:53:57 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-13T23:53:58.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:53:57 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:53:58.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:53:57 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:53:58.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:53:57 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-13T23:53:59.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:53:58 smithi115 conmon[90393]: 2022-05-13T23:53:58.751+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:53:59.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:53:58 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:53:59.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:53:58 smithi115 ceph-mon[78593]: pgmap v1138: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:53:59.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:53:58 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 1570 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:53:59.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:53:58 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:53:59.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:53:58 smithi047 ceph-mon[77147]: pgmap v1138: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:53:59.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:53:58 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 1570 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:54:00.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:53:59 smithi115 conmon[90393]: 2022-05-13T23:53:59.704+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:54:00.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:53:59 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:54:00.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:53:59 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:54:01.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:54:00 smithi115 conmon[90393]: 2022-05-13T23:54:00.713+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:54:01.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:54:00 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:54:01.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:54:00 smithi115 ceph-mon[78593]: pgmap v1139: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:54:01.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:54:00 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:54:01.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:54:00 smithi047 ceph-mon[77147]: pgmap v1139: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:54:02.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:54:01 smithi115 conmon[90393]: 2022-05-13T23:54:01.704+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:54:02.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:54:01 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:54:02.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:54:01 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:54:02.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:54:01 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-13T23:54:02.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:54:01 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:54:02.295 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:54:01 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:54:02.295 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:54:01 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-13T23:54:03.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:54:02 smithi115 conmon[90393]: 2022-05-13T23:54:02.690+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:54:03.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:54:02 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:54:03.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:54:02 smithi115 ceph-mon[78593]: pgmap v1140: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:54:03.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:54:02 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:54:03.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:54:02 smithi047 ceph-mon[77147]: pgmap v1140: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:54:04.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:54:03 smithi115 conmon[90393]: 2022-05-13T23:54:03.694+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:54:04.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:54:03 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:54:04.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:54:03 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 1575 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:54:04.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:54:03 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:54:04.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:54:03 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 1575 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:54:05.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:54:04 smithi115 conmon[90393]: 2022-05-13T23:54:04.728+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:54:05.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:54:04 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:54:05.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:54:04 smithi115 ceph-mon[78593]: pgmap v1141: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:54:05.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:54:04 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:54:05.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:54:04 smithi047 ceph-mon[77147]: pgmap v1141: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:54:06.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:54:05 smithi115 conmon[90393]: 2022-05-13T23:54:05.743+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:54:06.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:54:05 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:54:06.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:54:05 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:54:07.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:54:06 smithi115 conmon[90393]: 2022-05-13T23:54:06.719+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:54:07.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:54:06 smithi115 ceph-mon[78593]: pgmap v1142: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:54:07.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:54:06 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:54:07.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:54:06 smithi047 ceph-mon[77147]: pgmap v1142: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:54:07.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:54:06 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:54:07.945 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:54:07 smithi115 conmon[90393]: 2022-05-13T23:54:07.686+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:54:08.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:54:07 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:54:08.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:54:07 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:54:08.946 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:54:08 smithi115 conmon[90393]: 2022-05-13T23:54:08.640+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:54:09.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:54:08 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:54:09.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:54:08 smithi047 ceph-mon[77147]: pgmap v1143: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:54:09.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:54:08 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 1580 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:54:09.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:54:08 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:54:09.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:54:08 smithi115 ceph-mon[78593]: pgmap v1143: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:54:09.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:54:08 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 1580 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:54:09.951 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:54:09 smithi115 conmon[90393]: 2022-05-13T23:54:09.682+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:54:10.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:54:09 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:54:10.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:54:09 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:54:10.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:54:09 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-13T23:54:10.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:54:09 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:54:10.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:54:09 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:54:10.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:54:09 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-13T23:54:10.953 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:54:10 smithi115 conmon[90393]: 2022-05-13T23:54:10.693+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:54:11.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:54:10 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:54:11.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:54:10 smithi047 ceph-mon[77147]: pgmap v1144: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:54:11.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:54:10 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:54:11.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:54:10 smithi115 ceph-mon[78593]: pgmap v1144: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:54:11.957 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:54:11 smithi115 conmon[90393]: 2022-05-13T23:54:11.671+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:54:12.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:54:11 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:54:12.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:54:11 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:54:12.963 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:54:12 smithi115 conmon[90393]: 2022-05-13T23:54:12.642+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:54:13.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:54:12 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:54:13.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:54:12 smithi047 ceph-mon[77147]: pgmap v1145: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:54:13.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:54:12 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:54:13.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:54:12 smithi115 ceph-mon[78593]: pgmap v1145: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:54:13.964 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:54:13 smithi115 conmon[90393]: 2022-05-13T23:54:13.682+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:54:14.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:54:13 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:54:14.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:54:13 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:54:14.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:54:13 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-13T23:54:14.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:54:13 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-13T23:54:14.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:54:13 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-13T23:54:14.295 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:54:13 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 1585 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:54:14.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:54:13 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:54:14.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:54:13 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:54:14.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:54:13 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-13T23:54:14.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:54:13 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-13T23:54:14.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:54:13 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-13T23:54:14.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:54:13 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 1585 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:54:14.970 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:54:14 smithi115 conmon[90393]: 2022-05-13T23:54:14.686+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:54:15.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:54:14 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:54:15.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:54:14 smithi047 ceph-mon[77147]: pgmap v1146: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:54:15.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:54:14 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:54:15.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:54:14 smithi115 ceph-mon[78593]: pgmap v1146: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:54:15.970 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:54:15 smithi115 conmon[90393]: 2022-05-13T23:54:15.669+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:54:16.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:54:15 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:54:16.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:54:15 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:54:16.979 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:54:16 smithi115 conmon[90393]: 2022-05-13T23:54:16.701+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:54:17.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:54:16 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:54:17.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:54:16 smithi047 ceph-mon[77147]: pgmap v1147: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:54:17.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:54:16 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:54:17.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:54:16 smithi115 ceph-mon[78593]: pgmap v1147: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:54:17.981 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:54:17 smithi115 conmon[90393]: 2022-05-13T23:54:17.653+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:54:18.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:54:17 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:54:18.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:54:17 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:54:18.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:54:17 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-13T23:54:18.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:54:17 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:54:18.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:54:17 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:54:18.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:54:17 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-13T23:54:18.983 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:54:18 smithi115 conmon[90393]: 2022-05-13T23:54:18.635+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:54:19.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:54:18 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:54:19.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:54:18 smithi047 ceph-mon[77147]: pgmap v1148: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:54:19.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:54:18 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 1590 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:54:19.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:54:18 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:54:19.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:54:18 smithi115 ceph-mon[78593]: pgmap v1148: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:54:19.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:54:18 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 1590 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:54:19.987 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:54:19 smithi115 conmon[90393]: 2022-05-13T23:54:19.680+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:54:20.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:54:19 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:54:20.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:54:19 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:54:20.988 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:54:20 smithi115 conmon[90393]: 2022-05-13T23:54:20.686+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:54:21.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:54:20 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:54:21.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:54:20 smithi047 ceph-mon[77147]: pgmap v1149: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:54:21.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:54:20 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:54:21.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:54:20 smithi115 ceph-mon[78593]: pgmap v1149: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:54:21.997 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:54:21 smithi115 conmon[90393]: 2022-05-13T23:54:21.732+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:54:22.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:54:21 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:54:22.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:54:21 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:54:22.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:54:21 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-13T23:54:22.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:54:21 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:54:22.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:54:21 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:54:22.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:54:21 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-13T23:54:22.998 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:54:22 smithi115 conmon[90393]: 2022-05-13T23:54:22.744+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:54:23.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:54:22 smithi047 ceph-mon[77147]: pgmap v1150: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:54:23.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:54:22 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:54:23.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:54:22 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-13T23:54:23.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:54:22 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-13T23:54:23.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:54:22 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-13T23:54:23.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:54:22 smithi115 ceph-mon[78593]: pgmap v1150: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:54:23.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:54:22 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:54:23.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:54:22 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-13T23:54:23.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:54:22 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-13T23:54:23.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:54:22 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-13T23:54:24.000 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:54:23 smithi115 conmon[90393]: 2022-05-13T23:54:23.719+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:54:24.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:54:23 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:54:24.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:54:23 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 1595 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:54:24.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:54:24 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:54:24.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:54:24 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 1595 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:54:25.001 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:54:24 smithi115 conmon[90393]: 2022-05-13T23:54:24.728+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:54:25.002 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:54:25 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:54:25.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:54:25 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:54:25.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:54:25 smithi047 ceph-mon[77147]: pgmap v1151: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:54:25.791 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:54:25 smithi047 conmon[78521]: 2022-05-13T23:54:25.520+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:54:25.791 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:54:25 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-13T23:54:25.791 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:54:25 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-13T23:54:25.792 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:54:25 smithi047 conmon[78521]: yield 2022-05-13T23:54:25.792 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:54:25 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-13T23:54:25.792 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:54:25 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-13T23:54:25.792 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:54:25 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-13T23:54:25.793 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:54:25 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-13T23:54:25.793 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:54:25 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-13T23:54:25.793 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:54:25 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-13T23:54:25.793 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:54:25 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-13T23:54:25.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:54:25 smithi047 conmon[78521]: raise exceptions[0] 2022-05-13T23:54:25.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:54:25 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-13T23:54:25.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:54:25 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-13T23:54:25.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:54:25 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-13T23:54:25.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:54:25 smithi047 conmon[78521]: return (yield from fut) 2022-05-13T23:54:25.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:54:25 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-13T23:54:25.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:54:25 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-13T23:54:25.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:54:25 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:54:26.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:54:25 smithi115 conmon[90393]: 2022-05-13T23:54:25.756+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:54:26.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:54:25 smithi115 ceph-mon[78593]: pgmap v1151: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:54:26.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:54:26 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:54:26.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:54:26 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:54:26.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-13T23:54:26.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-13T23:54:26.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-13T23:54:26.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-13T23:54:26.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-13T23:54:26.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-13T23:54:26.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-13T23:54:26.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-13T23:54:26.113 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-13T23:54:26.114 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-13T23:54:26.114 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-13T23:54:26.114 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-13T23:54:26.115 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-13T23:54:26.115 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-13T23:54:26.116 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-13T23:54:26.116 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-13T23:54:26.117 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-13T23:54:26.117 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:54:26.117 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:54:26 smithi115 ceph-mon[78593]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-13T23:54:26.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:54:26 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:54:26.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:54:26 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:54:26.292 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-13T23:54:26.292 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-13T23:54:26.292 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-13T23:54:26.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-13T23:54:26.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-13T23:54:26.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-13T23:54:26.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-13T23:54:26.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-13T23:54:26.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-13T23:54:26.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-13T23:54:26.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-13T23:54:26.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-13T23:54:26.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-13T23:54:26.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-13T23:54:26.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-13T23:54:26.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-13T23:54:26.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-13T23:54:26.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:54:26.296 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:54:26 smithi047 ceph-mon[77147]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-13T23:54:27.008 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:54:26 smithi115 conmon[90393]: 2022-05-13T23:54:26.745+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:54:27.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:54:27 smithi047 ceph-mon[77147]: pgmap v1152: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:54:27.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:54:27 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:54:27.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:54:27 smithi115 ceph-mon[78593]: pgmap v1152: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:54:27.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:54:27 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:54:28.015 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:54:27 smithi115 conmon[90393]: 2022-05-13T23:54:27.740+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:54:28.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:54:28 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:54:28.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:54:28 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:54:29.019 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:54:28 smithi047 conmon[78521]: 2022-05-13T23:54:28.592+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:54:29.019 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:54:28 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-13T23:54:29.019 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:54:28 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-13T23:54:29.020 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:54:28 smithi047 conmon[78521]: yield 2022-05-13T23:54:29.020 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:54:28 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-13T23:54:29.020 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:54:28 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-13T23:54:29.020 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:54:28 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-13T23:54:29.021 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:54:28 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-13T23:54:29.021 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:54:28 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-13T23:54:29.021 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:54:28 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-13T23:54:29.021 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:54:28 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-13T23:54:29.021 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:54:28 smithi047 conmon[78521]: raise exceptions[0] 2022-05-13T23:54:29.022 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:54:28 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-13T23:54:29.022 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:54:28 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-13T23:54:29.022 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:54:28 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-13T23:54:29.022 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:54:28 smithi047 conmon[78521]: return (yield from fut) 2022-05-13T23:54:29.023 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:54:28 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-13T23:54:29.023 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:54:28 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-13T23:54:29.023 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:54:28 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:54:29.023 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:54:28 smithi047 conmon[78521]: 2022-05-13T23:54:28.593+0000 7f5340779700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-13T23:54:29.024 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:54:28 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-13T23:54:29.024 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:54:28 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-13T23:54:29.025 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:54:28 smithi047 conmon[78521]: yield 2022-05-13T23:54:29.025 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:54:28 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-13T23:54:29.025 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:54:28 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-13T23:54:29.025 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:54:28 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-13T23:54:29.026 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:54:28 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-13T23:54:29.026 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:54:28 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-13T23:54:29.030 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:54:28 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-13T23:54:29.030 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:54:28 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-13T23:54:29.031 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:54:28 smithi047 conmon[78521]: raise exceptions[0] 2022-05-13T23:54:29.031 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:54:28 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-13T23:54:29.031 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:54:28 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-13T23:54:29.031 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:54:28 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-13T23:54:29.032 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:54:28 smithi047 conmon[78521]: return (yield from fut) 2022-05-13T23:54:29.032 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:54:28 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-13T23:54:29.033 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:54:28 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-13T23:54:29.033 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:54:28 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:54:29.034 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:54:28 smithi047 conmon[78521]: 2022-05-13T23:54:29.034 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:54:28 smithi047 conmon[78521]: During handling of the above exception, another exception occurred: 2022-05-13T23:54:29.034 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:54:28 smithi047 conmon[78521]: 2022-05-13T23:54:29.035 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:54:28 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-13T23:54:29.035 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:54:28 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-13T23:54:29.035 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:54:28 smithi047 conmon[78521]: return f(*arg) 2022-05-13T23:54:29.035 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:54:28 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-13T23:54:29.036 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:54:28 smithi047 conmon[78521]: self._write_client_files(client_files, host) 2022-05-13T23:54:29.036 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:54:28 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-13T23:54:29.036 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:54:28 smithi047 conmon[78521]: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-13T23:54:29.037 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:54:28 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-13T23:54:29.037 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:54:28 smithi047 conmon[78521]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-13T23:54:29.037 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:54:28 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-13T23:54:29.037 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:54:28 smithi047 conmon[78521]: return self.event_loop.get_result(coro) 2022-05-13T23:54:29.038 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:54:28 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-13T23:54:29.038 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:54:28 smithi047 conmon[78521]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-13T23:54:29.038 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:54:28 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-13T23:54:29.038 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:54:28 smithi047 conmon[78521]: return self.__get_result() 2022-05-13T23:54:29.039 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:54:28 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-13T23:54:29.039 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:54:28 smithi047 conmon[78521]: raise self._exception 2022-05-13T23:54:29.039 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:54:28 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-13T23:54:29.040 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:54:28 smithi047 conmon[78521]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-13T23:54:29.040 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:54:28 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-13T23:54:29.040 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:54:28 smithi047 conmon[78521]: conn = await self._remote_connection(host, addr) 2022-05-13T23:54:29.040 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:54:28 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-13T23:54:29.041 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:54:28 smithi047 conmon[78521]: raise 2022-05-13T23:54:29.041 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:54:28 smithi047 conmon[78521]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-13T23:54:29.041 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:54:28 smithi047 conmon[78521]: self.gen.throw(type, value, traceback) 2022-05-13T23:54:29.042 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:54:28 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-13T23:54:29.042 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:54:28 smithi047 conmon[78521]: raise OrchestratorError(msg) 2022-05-13T23:54:29.042 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:54:28 smithi047 conmon[78521]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:54:29.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:54:28 smithi115 conmon[90393]: 2022-05-13T23:54:28.776+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:54:29.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:54:29 smithi115 ceph-mon[78593]: pgmap v1153: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:54:29.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:54:29 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:54:29.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:54:29 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 1600 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:54:29.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:54:29 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:54:29.109 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-13T23:54:29.109 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-13T23:54:29.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-13T23:54:29.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-13T23:54:29.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-13T23:54:29.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-13T23:54:29.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-13T23:54:29.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-13T23:54:29.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-13T23:54:29.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-13T23:54:29.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-13T23:54:29.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-13T23:54:29.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-13T23:54:29.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-13T23:54:29.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-13T23:54:29.113 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-13T23:54:29.113 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-13T23:54:29.113 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:54:29.113 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:54:29 smithi115 ceph-mon[78593]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-13T23:54:29.113 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-13T23:54:29.114 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-13T23:54:29.114 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-13T23:54:29.114 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-13T23:54:29.114 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-13T23:54:29.115 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-13T23:54:29.115 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-13T23:54:29.115 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-13T23:54:29.115 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-13T23:54:29.116 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-13T23:54:29.116 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-13T23:54:29.116 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-13T23:54:29.116 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-13T23:54:29.116 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-13T23:54:29.117 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-13T23:54:29.117 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-13T23:54:29.117 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-13T23:54:29.117 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:54:29.118 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-13T23:54:29.118 INFO:journalctl@ceph.mon.c.smithi115.stdout: During handling of the above exception, another exception occurred: 2022-05-13T23:54:29.118 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-13T23:54:29.118 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-13T23:54:29.119 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-13T23:54:29.119 INFO:journalctl@ceph.mon.c.smithi115.stdout: return f(*arg) 2022-05-13T23:54:29.119 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-13T23:54:29.119 INFO:journalctl@ceph.mon.c.smithi115.stdout: self._write_client_files(client_files, host) 2022-05-13T23:54:29.119 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-13T23:54:29.120 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-13T23:54:29.120 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-13T23:54:29.120 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-13T23:54:29.120 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-13T23:54:29.121 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.event_loop.get_result(coro) 2022-05-13T23:54:29.121 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-13T23:54:29.121 INFO:journalctl@ceph.mon.c.smithi115.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-13T23:54:29.121 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-13T23:54:29.121 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.__get_result() 2022-05-13T23:54:29.122 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-13T23:54:29.122 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise self._exception 2022-05-13T23:54:29.122 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-13T23:54:29.122 INFO:journalctl@ceph.mon.c.smithi115.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-13T23:54:29.123 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-13T23:54:29.123 INFO:journalctl@ceph.mon.c.smithi115.stdout: conn = await self._remote_connection(host, addr) 2022-05-13T23:54:29.124 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-13T23:54:29.124 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise 2022-05-13T23:54:29.124 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-13T23:54:29.124 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.gen.throw(type, value, traceback) 2022-05-13T23:54:29.125 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-13T23:54:29.125 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OrchestratorError(msg) 2022-05-13T23:54:29.125 INFO:journalctl@ceph.mon.c.smithi115.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:54:29.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:54:29 smithi047 ceph-mon[77147]: pgmap v1153: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:54:29.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:54:29 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:54:29.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:54:29 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 1600 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:54:29.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:54:29 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:54:29.292 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-13T23:54:29.292 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-13T23:54:29.292 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-13T23:54:29.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-13T23:54:29.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-13T23:54:29.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-13T23:54:29.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-13T23:54:29.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-13T23:54:29.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-13T23:54:29.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-13T23:54:29.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-13T23:54:29.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-13T23:54:29.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-13T23:54:29.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-13T23:54:29.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-13T23:54:29.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-13T23:54:29.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-13T23:54:29.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:54:29.296 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:54:29 smithi047 ceph-mon[77147]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-13T23:54:29.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-13T23:54:29.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-13T23:54:29.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-13T23:54:29.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-13T23:54:29.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-13T23:54:29.298 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-13T23:54:29.298 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-13T23:54:29.298 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-13T23:54:29.298 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-13T23:54:29.299 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-13T23:54:29.299 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-13T23:54:29.299 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-13T23:54:29.299 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-13T23:54:29.299 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-13T23:54:29.300 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-13T23:54:29.300 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-13T23:54:29.300 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-13T23:54:29.301 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:54:29.301 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-13T23:54:29.301 INFO:journalctl@ceph.mon.b.smithi047.stdout: During handling of the above exception, another exception occurred: 2022-05-13T23:54:29.301 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-13T23:54:29.302 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-13T23:54:29.302 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-13T23:54:29.302 INFO:journalctl@ceph.mon.b.smithi047.stdout: return f(*arg) 2022-05-13T23:54:29.302 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-13T23:54:29.302 INFO:journalctl@ceph.mon.b.smithi047.stdout: self._write_client_files(client_files, host) 2022-05-13T23:54:29.303 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-13T23:54:29.303 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-13T23:54:29.303 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-13T23:54:29.303 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-13T23:54:29.303 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-13T23:54:29.304 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.event_loop.get_result(coro) 2022-05-13T23:54:29.304 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-13T23:54:29.304 INFO:journalctl@ceph.mon.b.smithi047.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-13T23:54:29.304 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-13T23:54:29.305 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.__get_result() 2022-05-13T23:54:29.305 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-13T23:54:29.305 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise self._exception 2022-05-13T23:54:29.305 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-13T23:54:29.305 INFO:journalctl@ceph.mon.b.smithi047.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-13T23:54:29.306 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-13T23:54:29.306 INFO:journalctl@ceph.mon.b.smithi047.stdout: conn = await self._remote_connection(host, addr) 2022-05-13T23:54:29.306 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-13T23:54:29.306 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise 2022-05-13T23:54:29.306 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-13T23:54:29.307 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.gen.throw(type, value, traceback) 2022-05-13T23:54:29.307 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-13T23:54:29.307 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OrchestratorError(msg) 2022-05-13T23:54:29.307 INFO:journalctl@ceph.mon.b.smithi047.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:54:30.026 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:54:29 smithi115 conmon[90393]: 2022-05-13T23:54:29.761+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:54:30.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:54:30 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:54:30.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:54:30 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:54:30.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:54:30 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-13T23:54:30.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:54:30 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:54:30.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:54:30 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:54:30.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:54:30 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-13T23:54:31.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:54:31 smithi115 ceph-mon[78593]: pgmap v1154: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:54:31.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:54:31 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:54:31.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:54:30 smithi115 conmon[90393]: 2022-05-13T23:54:30.794+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:54:31.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:54:31 smithi047 ceph-mon[77147]: pgmap v1154: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:54:31.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:54:31 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:54:32.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:54:32 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:54:32.110 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:54:31 smithi115 conmon[90393]: 2022-05-13T23:54:31.786+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:54:32.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:54:32 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:54:33.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:54:32 smithi115 conmon[90393]: 2022-05-13T23:54:32.820+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:54:33.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:54:33 smithi115 ceph-mon[78593]: pgmap v1155: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:54:33.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:54:33 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:54:33.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:54:33 smithi047 ceph-mon[77147]: pgmap v1155: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:54:33.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:54:33 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:54:34.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:54:33 smithi115 conmon[90393]: 2022-05-13T23:54:33.818+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:54:34.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:54:34 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:54:34.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:54:34 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:54:34.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:54:34 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-13T23:54:34.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:54:34 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 1605 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:54:34.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:54:34 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:54:34.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:54:34 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:54:34.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:54:34 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-13T23:54:34.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:54:34 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 1605 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:54:35.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:54:34 smithi115 conmon[90393]: 2022-05-13T23:54:34.808+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:54:35.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:54:35 smithi115 ceph-mon[78593]: pgmap v1156: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:54:35.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:54:35 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:54:35.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:54:35 smithi047 ceph-mon[77147]: pgmap v1156: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:54:35.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:54:35 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:54:36.045 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:54:35 smithi115 conmon[90393]: 2022-05-13T23:54:35.794+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:54:36.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:54:36 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:54:36.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:54:36 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:54:37.044 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:54:36 smithi115 conmon[90393]: 2022-05-13T23:54:36.783+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:54:37.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:54:37 smithi115 ceph-mon[78593]: pgmap v1157: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:54:37.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:54:37 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:54:37.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:54:37 smithi047 ceph-mon[77147]: pgmap v1157: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:54:37.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:54:37 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:54:38.059 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:54:37 smithi115 conmon[90393]: 2022-05-13T23:54:37.746+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:54:38.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:54:38 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:54:38.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:54:38 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:54:38.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:54:38 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-13T23:54:38.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:54:38 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:54:38.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:54:38 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:54:38.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:54:38 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-13T23:54:39.058 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:54:38 smithi115 conmon[90393]: 2022-05-13T23:54:38.758+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:54:39.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:54:39 smithi115 ceph-mon[78593]: pgmap v1158: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:54:39.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:54:39 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:54:39.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:54:39 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 1610 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:54:39.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:54:39 smithi047 ceph-mon[77147]: pgmap v1158: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:54:39.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:54:39 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:54:39.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:54:39 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 1610 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:54:40.057 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:54:39 smithi115 conmon[90393]: 2022-05-13T23:54:39.771+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:54:40.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:54:40 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:54:40.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:54:40 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:54:41.066 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:54:40 smithi115 conmon[90393]: 2022-05-13T23:54:40.731+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:54:41.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:54:41 smithi115 ceph-mon[78593]: pgmap v1159: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:54:41.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:54:41 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:54:41.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:54:41 smithi047 ceph-mon[77147]: pgmap v1159: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:54:41.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:54:41 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:54:42.071 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:54:41 smithi115 conmon[90393]: 2022-05-13T23:54:41.717+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:54:42.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:54:42 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:54:42.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:54:42 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:54:42.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:54:42 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-13T23:54:42.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:54:42 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:54:42.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:54:42 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:54:42.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:54:42 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-13T23:54:43.073 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:54:42 smithi115 conmon[90393]: 2022-05-13T23:54:42.753+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:54:43.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:54:43 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:54:43.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:54:43 smithi115 ceph-mon[78593]: pgmap v1160: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:54:43.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:54:43 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:54:43.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:54:43 smithi047 ceph-mon[77147]: pgmap v1160: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:54:44.074 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:54:43 smithi115 conmon[90393]: 2022-05-13T23:54:43.735+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:54:44.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:54:44 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:54:44.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:54:44 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 1615 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:54:44.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:54:44 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:54:44.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:54:44 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 1615 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:54:45.075 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:54:44 smithi115 conmon[90393]: 2022-05-13T23:54:44.747+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:54:45.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:54:45 smithi115 ceph-mon[78593]: pgmap v1161: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:54:45.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:54:45 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:54:45.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:54:45 smithi047 ceph-mon[77147]: pgmap v1161: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:54:45.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:54:45 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:54:46.081 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:54:45 smithi115 conmon[90393]: 2022-05-13T23:54:45.737+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:54:46.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:54:46 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:54:46.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:54:46 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:54:47.082 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:54:46 smithi115 conmon[90393]: 2022-05-13T23:54:46.744+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:54:47.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:54:47 smithi115 ceph-mon[78593]: pgmap v1162: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:54:47.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:54:47 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:54:47.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:54:47 smithi047 ceph-mon[77147]: pgmap v1162: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:54:47.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:54:47 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:54:48.087 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:54:47 smithi115 conmon[90393]: 2022-05-13T23:54:47.773+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:54:48.368 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:54:48 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:54:48.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:54:48 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:54:49.092 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:54:48 smithi115 conmon[90393]: 2022-05-13T23:54:48.726+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:54:49.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:54:49 smithi115 ceph-mon[78593]: pgmap v1163: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:54:49.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:54:49 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:54:49.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:54:49 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 1620 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:54:49.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:54:49 smithi047 ceph-mon[77147]: pgmap v1163: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:54:49.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:54:49 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:54:49.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:54:49 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 1620 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:54:50.097 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:54:49 smithi115 conmon[90393]: 2022-05-13T23:54:49.745+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:54:50.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:54:50 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:54:50.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:54:50 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:54:50.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:54:50 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-13T23:54:50.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:54:50 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:54:50.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:54:50 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:54:50.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:54:50 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-13T23:54:51.099 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:54:50 smithi115 conmon[90393]: 2022-05-13T23:54:50.706+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:54:51.368 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:54:51 smithi115 ceph-mon[78593]: pgmap v1164: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:54:51.369 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:54:51 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:54:51.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:54:51 smithi047 ceph-mon[77147]: pgmap v1164: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:54:51.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:54:51 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:54:52.105 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:54:51 smithi115 conmon[90393]: 2022-05-13T23:54:51.720+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:54:52.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:54:52 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:54:52.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:54:52 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:54:53.105 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:54:52 smithi115 conmon[90393]: 2022-05-13T23:54:52.757+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:54:53.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:54:53 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:54:53.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:54:53 smithi115 ceph-mon[78593]: pgmap v1165: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:54:53.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:54:53 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:54:53.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:54:53 smithi047 ceph-mon[77147]: pgmap v1165: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:54:54.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:54:53 smithi115 conmon[90393]: 2022-05-13T23:54:53.716+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:54:54.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:54:54 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:54:54.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:54:54 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:54:54.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:54:54 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-13T23:54:54.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:54:54 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 1625 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:54:54.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:54:54 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:54:54.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:54:54 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:54:54.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:54:54 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-13T23:54:54.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:54:54 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 1625 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:54:55.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:54:54 smithi115 conmon[90393]: 2022-05-13T23:54:54.720+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:54:55.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:54:55 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:54:55.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:54:55 smithi047 ceph-mon[77147]: pgmap v1166: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:54:55.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:54:55 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:54:55.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:54:55 smithi115 ceph-mon[78593]: pgmap v1166: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:54:56.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:54:55 smithi115 conmon[90393]: 2022-05-13T23:54:55.684+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:54:56.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:54:56 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:54:56.613 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:54:56 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:54:57.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:54:56 smithi115 conmon[90393]: 2022-05-13T23:54:56.714+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:54:57.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:54:57 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:54:57.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:54:57 smithi047 ceph-mon[77147]: pgmap v1167: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:54:57.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:54:57 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:54:57.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:54:57 smithi115 ceph-mon[78593]: pgmap v1167: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:54:58.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:54:57 smithi115 conmon[90393]: 2022-05-13T23:54:57.764+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:54:58.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:54:58 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:54:58.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:54:58 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:54:58.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:54:58 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-13T23:54:58.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:54:58 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:54:58.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:54:58 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:54:58.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:54:58 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-13T23:54:59.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:54:58 smithi115 conmon[90393]: 2022-05-13T23:54:58.729+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:54:59.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:54:59 smithi047 ceph-mon[77147]: pgmap v1168: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:54:59.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:54:59 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:54:59.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:54:59 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 1630 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:54:59.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:54:59 smithi115 ceph-mon[78593]: pgmap v1168: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:54:59.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:54:59 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:54:59.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:54:59 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 1630 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:55:00.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:54:59 smithi115 conmon[90393]: 2022-05-13T23:54:59.693+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:55:00.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:55:00 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:55:00.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:55:00 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:55:01.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:55:00 smithi115 conmon[90393]: 2022-05-13T23:55:00.713+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:55:01.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:55:01 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:55:01.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:55:01 smithi047 ceph-mon[77147]: pgmap v1169: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:55:01.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:55:01 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:55:01.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:55:01 smithi115 ceph-mon[78593]: pgmap v1169: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:55:02.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:55:01 smithi115 conmon[90393]: 2022-05-13T23:55:01.739+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:55:02.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:55:02 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:55:02.546 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:55:02 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:55:02.546 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:55:02 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-13T23:55:02.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:55:02 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:55:02.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:55:02 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:55:02.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:55:02 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-13T23:55:03.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:55:02 smithi115 conmon[90393]: 2022-05-13T23:55:02.739+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:55:03.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:55:03 smithi047 ceph-mon[77147]: pgmap v1170: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:55:03.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:55:03 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:55:03.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:55:03 smithi115 ceph-mon[78593]: pgmap v1170: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:55:03.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:55:03 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:55:04.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:55:03 smithi115 conmon[90393]: 2022-05-13T23:55:03.739+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:55:04.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:55:04 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:55:04.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:55:04 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 1635 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:55:04.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:55:04 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:55:04.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:55:04 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 1635 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:55:05.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:55:04 smithi115 conmon[90393]: 2022-05-13T23:55:04.784+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:55:05.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:55:05 smithi047 ceph-mon[77147]: pgmap v1171: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:55:05.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:55:05 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:55:05.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:55:05 smithi115 ceph-mon[78593]: pgmap v1171: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:55:05.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:55:05 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:55:06.113 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:55:05 smithi115 conmon[90393]: 2022-05-13T23:55:05.759+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:55:06.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:55:06 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:55:06.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:55:06 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:55:07.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:55:06 smithi115 conmon[90393]: 2022-05-13T23:55:06.802+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:55:07.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:55:07 smithi047 ceph-mon[77147]: pgmap v1172: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:55:07.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:55:07 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:55:07.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:55:07 smithi115 ceph-mon[78593]: pgmap v1172: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:55:07.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:55:07 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:55:08.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:55:07 smithi115 conmon[90393]: 2022-05-13T23:55:07.844+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:55:08.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:55:08 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:55:08.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:55:08 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:55:09.161 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:55:08 smithi115 conmon[90393]: 2022-05-13T23:55:08.868+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:55:09.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:55:09 smithi047 ceph-mon[77147]: pgmap v1173: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:55:09.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:55:09 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:55:09.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:55:09 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 1640 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:55:09.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:55:09 smithi115 ceph-mon[78593]: pgmap v1173: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:55:09.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:55:09 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:55:09.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:55:09 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 1640 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:55:10.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:55:09 smithi115 conmon[90393]: 2022-05-13T23:55:09.821+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:55:10.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:55:10 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:55:10.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:55:10 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:55:10.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:55:10 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-13T23:55:10.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:55:10 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:55:10.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:55:10 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:55:10.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:55:10 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-13T23:55:11.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:55:10 smithi115 conmon[90393]: 2022-05-13T23:55:10.852+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:55:11.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:55:11 smithi047 ceph-mon[77147]: pgmap v1174: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:55:11.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:55:11 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:55:11.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:55:11 smithi115 ceph-mon[78593]: pgmap v1174: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:55:11.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:55:11 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:55:12.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:55:11 smithi115 conmon[90393]: 2022-05-13T23:55:11.844+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:55:12.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:55:12 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:55:12.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:55:12 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:55:13.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:55:12 smithi115 conmon[90393]: 2022-05-13T23:55:12.811+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:55:13.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:55:13 smithi047 ceph-mon[77147]: pgmap v1175: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:55:13.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:55:13 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:55:13.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:55:13 smithi115 ceph-mon[78593]: pgmap v1175: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:55:13.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:55:13 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:55:14.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:55:13 smithi115 conmon[90393]: 2022-05-13T23:55:13.846+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:55:14.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:55:14 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:55:14.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:55:14 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:55:14.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:55:14 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-13T23:55:14.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:55:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-13T23:55:14.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:55:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-13T23:55:14.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:55:14 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 1645 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:55:14.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:55:14 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:55:14.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:55:14 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:55:14.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:55:14 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-13T23:55:14.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:55:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-13T23:55:14.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:55:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-13T23:55:14.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:55:14 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 1645 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:55:15.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:55:14 smithi115 conmon[90393]: 2022-05-13T23:55:14.852+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:55:15.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:55:15 smithi047 ceph-mon[77147]: pgmap v1176: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:55:15.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:55:15 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:55:15.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:55:15 smithi115 ceph-mon[78593]: pgmap v1176: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:55:15.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:55:15 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:55:16.193 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:55:15 smithi115 conmon[90393]: 2022-05-13T23:55:15.901+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:55:16.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:55:16 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:55:16.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:55:16 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:55:17.198 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:55:16 smithi115 conmon[90393]: 2022-05-13T23:55:16.925+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:55:17.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:55:17 smithi047 ceph-mon[77147]: pgmap v1177: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:55:17.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:55:17 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:55:17.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:55:17 smithi115 ceph-mon[78593]: pgmap v1177: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:55:17.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:55:17 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:55:18.206 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:55:17 smithi115 conmon[90393]: 2022-05-13T23:55:17.902+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:55:18.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:55:18 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:55:18.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:55:18 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:55:18.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:55:18 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-13T23:55:18.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:55:18 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:55:18.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:55:18 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:55:18.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:55:18 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-13T23:55:19.202 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:55:18 smithi115 conmon[90393]: 2022-05-13T23:55:18.926+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:55:19.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:55:19 smithi047 ceph-mon[77147]: pgmap v1178: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-05-13T23:55:19.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:55:19 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:55:19.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:55:19 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 1650 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:55:19.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:55:19 smithi115 ceph-mon[78593]: pgmap v1178: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-05-13T23:55:19.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:55:19 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:55:19.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:55:19 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 1650 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:55:20.207 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:55:19 smithi115 conmon[90393]: 2022-05-13T23:55:19.951+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:55:20.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:55:20 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:55:20.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:55:20 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:55:21.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:55:20 smithi115 conmon[90393]: 2022-05-13T23:55:20.987+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:55:21.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:55:21 smithi115 ceph-mon[78593]: pgmap v1179: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:55:21.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:55:21 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:55:21.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:55:21 smithi047 ceph-mon[77147]: pgmap v1179: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:55:21.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:55:21 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:55:22.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:55:21 smithi115 conmon[90393]: 2022-05-13T23:55:21.998+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:55:22.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:55:22 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:55:22.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:55:22 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:55:22.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:55:22 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-13T23:55:22.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:55:22 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:55:22.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:55:22 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:55:22.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:55:22 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-13T23:55:23.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:55:23 smithi115 conmon[90393]: 2022-05-13T23:55:23.028+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:55:23.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:55:23 smithi115 ceph-mon[78593]: pgmap v1180: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-05-13T23:55:23.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:55:23 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:55:23.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:55:23 smithi047 ceph-mon[77147]: pgmap v1180: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-05-13T23:55:23.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:55:23 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:55:24.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:55:24 smithi115 conmon[90393]: 2022-05-13T23:55:24.048+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:55:24.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:55:24 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:55:24.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:55:24 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 1655 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:55:24.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:55:24 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:55:24.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:55:24 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 1655 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:55:25.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:55:25 smithi115 conmon[90393]: 2022-05-13T23:55:25.056+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:55:25.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:55:25 smithi115 ceph-mon[78593]: pgmap v1181: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-05-13T23:55:25.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:55:25 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:55:25.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:55:25 smithi047 ceph-mon[77147]: pgmap v1181: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-05-13T23:55:25.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:55:25 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:55:26.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:55:26 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:55:26.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:55:26 smithi115 conmon[90393]: 2022-05-13T23:55:26.068+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:55:26.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:55:26 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:55:27.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:55:27 smithi115 conmon[90393]: 2022-05-13T23:55:27.034+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:55:27.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:55:27 smithi115 ceph-mon[78593]: pgmap v1182: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:55:27.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:55:27 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:55:27.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:55:27 smithi047 ceph-mon[77147]: pgmap v1182: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:55:27.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:55:27 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:55:28.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:55:28 smithi115 conmon[90393]: 2022-05-13T23:55:28.028+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:55:28.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:55:28 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:55:28.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:55:28 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:55:29.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:55:28 smithi115 conmon[90393]: 2022-05-13T23:55:28.989+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:55:29.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:55:29 smithi115 ceph-mon[78593]: pgmap v1183: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:55:29.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:55:29 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:55:29.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:55:29 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 1660 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:55:29.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:55:29 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-13T23:55:29.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:55:29 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-13T23:55:29.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:55:29 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-13T23:55:29.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:55:29 smithi047 ceph-mon[77147]: pgmap v1183: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:55:29.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:55:29 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:55:29.545 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:55:29 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 1660 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:55:29.545 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:55:29 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-13T23:55:29.545 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:55:29 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-13T23:55:29.545 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:55:29 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-13T23:55:30.239 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:55:29 smithi115 conmon[90393]: 2022-05-13T23:55:29.967+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:55:30.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:55:30 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:55:30.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:55:30 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:55:30.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:55:30 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-13T23:55:30.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:55:30 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:55:30.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:55:30 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:55:30.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:55:30 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-13T23:55:31.246 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:55:30 smithi115 conmon[90393]: 2022-05-13T23:55:30.957+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:55:31.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:55:31 smithi047 ceph-mon[77147]: pgmap v1184: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:55:31.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:55:31 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:55:31.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:55:31 smithi115 ceph-mon[78593]: pgmap v1184: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:55:31.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:55:31 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:55:32.041 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:55:31 smithi047 conmon[78521]: 2022-05-13T23:55:31.695+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:55:32.041 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:55:31 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-13T23:55:32.041 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:55:31 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-13T23:55:32.042 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:55:31 smithi047 conmon[78521]: yield 2022-05-13T23:55:32.042 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:55:31 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-13T23:55:32.042 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:55:31 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-13T23:55:32.042 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:55:31 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-13T23:55:32.042 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:55:31 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-13T23:55:32.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:55:31 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-13T23:55:32.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:55:31 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-13T23:55:32.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:55:31 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-13T23:55:32.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:55:31 smithi047 conmon[78521]: raise exceptions[0] 2022-05-13T23:55:32.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:55:31 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-13T23:55:32.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:55:31 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-13T23:55:32.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:55:31 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-13T23:55:32.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:55:31 smithi047 conmon[78521]: return (yield from fut) 2022-05-13T23:55:32.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:55:31 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-13T23:55:32.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:55:31 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-13T23:55:32.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:55:31 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:55:32.250 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:55:31 smithi115 conmon[90393]: 2022-05-13T23:55:31.959+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:55:32.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:55:32 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:55:32.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:55:32 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:55:33.257 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:55:32 smithi115 conmon[90393]: 2022-05-13T23:55:32.909+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:55:33.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:55:33 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:55:33.541 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-13T23:55:33.542 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-13T23:55:33.542 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-13T23:55:33.542 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-13T23:55:33.542 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-13T23:55:33.542 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-13T23:55:33.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-13T23:55:33.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-13T23:55:33.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-13T23:55:33.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-13T23:55:33.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-13T23:55:33.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-13T23:55:33.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-13T23:55:33.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-13T23:55:33.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-13T23:55:33.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-13T23:55:33.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-13T23:55:33.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:55:33.545 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:55:33 smithi047 ceph-mon[77147]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-13T23:55:33.545 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:55:33 smithi047 ceph-mon[77147]: pgmap v1185: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:55:33.546 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:55:33 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:55:33.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:55:33 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:55:33.608 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-13T23:55:33.608 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-13T23:55:33.609 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-13T23:55:33.609 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-13T23:55:33.609 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-13T23:55:33.609 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-13T23:55:33.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-13T23:55:33.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-13T23:55:33.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-13T23:55:33.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-13T23:55:33.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-13T23:55:33.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-13T23:55:33.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-13T23:55:33.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-13T23:55:33.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-13T23:55:33.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-13T23:55:33.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-13T23:55:33.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:55:33.612 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:55:33 smithi115 ceph-mon[78593]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-13T23:55:33.612 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:55:33 smithi115 ceph-mon[78593]: pgmap v1185: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:55:33.613 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:55:33 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:55:34.258 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:55:33 smithi115 conmon[90393]: 2022-05-13T23:55:33.893+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:55:34.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:55:34 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:55:34.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:55:34 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:55:34.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:55:34 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-13T23:55:34.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:55:34 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 1665 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:55:34.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:55:34 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:55:34.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:55:34 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:55:34.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:55:34 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-13T23:55:34.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:55:34 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 1665 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:55:35.042 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:55:34 smithi047 conmon[78521]: 2022-05-13T23:55:34.767+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:55:35.042 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:55:34 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-13T23:55:35.042 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:55:34 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-13T23:55:35.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:55:34 smithi047 conmon[78521]: yield 2022-05-13T23:55:35.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:55:34 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-13T23:55:35.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:55:34 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-13T23:55:35.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:55:34 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-13T23:55:35.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:55:34 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-13T23:55:35.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:55:34 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-13T23:55:35.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:55:34 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-13T23:55:35.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:55:34 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-13T23:55:35.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:55:34 smithi047 conmon[78521]: raise exceptions[0] 2022-05-13T23:55:35.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:55:34 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-13T23:55:35.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:55:34 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-13T23:55:35.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:55:34 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-13T23:55:35.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:55:34 smithi047 conmon[78521]: return (yield from fut) 2022-05-13T23:55:35.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:55:34 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-13T23:55:35.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:55:34 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-13T23:55:35.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:55:34 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:55:35.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:55:34 smithi047 conmon[78521]: 2022-05-13T23:55:34.768+0000 7f533ef76700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-13T23:55:35.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:55:34 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-13T23:55:35.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:55:34 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-13T23:55:35.048 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:55:34 smithi047 conmon[78521]: yield 2022-05-13T23:55:35.048 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:55:34 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-13T23:55:35.048 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:55:34 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-13T23:55:35.048 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:55:34 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-13T23:55:35.049 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:55:34 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-13T23:55:35.050 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:55:34 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-13T23:55:35.050 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:55:34 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-13T23:55:35.050 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:55:34 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-13T23:55:35.051 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:55:34 smithi047 conmon[78521]: raise exceptions[0] 2022-05-13T23:55:35.051 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:55:34 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-13T23:55:35.051 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:55:34 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-13T23:55:35.051 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:55:34 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-13T23:55:35.051 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:55:34 smithi047 conmon[78521]: return (yield from fut) 2022-05-13T23:55:35.052 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:55:34 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-13T23:55:35.052 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:55:34 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-13T23:55:35.052 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:55:34 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:55:35.052 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:55:34 smithi047 conmon[78521]: 2022-05-13T23:55:35.053 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:55:34 smithi047 conmon[78521]: During handling of the above exception, another exception occurred: 2022-05-13T23:55:35.053 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:55:34 smithi047 conmon[78521]: 2022-05-13T23:55:35.053 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:55:34 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-13T23:55:35.053 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:55:34 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-13T23:55:35.053 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:55:34 smithi047 conmon[78521]: return f(*arg) 2022-05-13T23:55:35.054 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:55:34 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-13T23:55:35.054 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:55:34 smithi047 conmon[78521]: self._write_client_files(client_files, host) 2022-05-13T23:55:35.054 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:55:34 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-13T23:55:35.054 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:55:34 smithi047 conmon[78521]: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-13T23:55:35.055 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:55:34 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-13T23:55:35.055 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:55:34 smithi047 conmon[78521]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-13T23:55:35.055 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:55:34 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-13T23:55:35.055 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:55:34 smithi047 conmon[78521]: return self.event_loop.get_result(coro) 2022-05-13T23:55:35.055 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:55:34 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-13T23:55:35.056 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:55:34 smithi047 conmon[78521]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-13T23:55:35.056 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:55:34 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-13T23:55:35.056 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:55:34 smithi047 conmon[78521]: return self.__get_result() 2022-05-13T23:55:35.056 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:55:34 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-13T23:55:35.056 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:55:34 smithi047 conmon[78521]: raise self._exception 2022-05-13T23:55:35.057 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:55:34 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-13T23:55:35.057 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:55:34 smithi047 conmon[78521]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-13T23:55:35.057 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:55:34 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-13T23:55:35.057 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:55:34 smithi047 conmon[78521]: conn = await self._remote_connection(host, addr) 2022-05-13T23:55:35.058 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:55:34 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-13T23:55:35.058 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:55:34 smithi047 conmon[78521]: raise 2022-05-13T23:55:35.060 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:55:34 smithi047 conmon[78521]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-13T23:55:35.060 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:55:34 smithi047 conmon[78521]: self.gen.throw(type, value, traceback) 2022-05-13T23:55:35.060 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:55:34 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-13T23:55:35.060 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:55:34 smithi047 conmon[78521]: raise OrchestratorError(msg) 2022-05-13T23:55:35.061 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:55:34 smithi047 conmon[78521]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:55:35.259 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:55:34 smithi115 conmon[90393]: 2022-05-13T23:55:34.929+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:55:35.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:55:35 smithi047 ceph-mon[77147]: pgmap v1186: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:55:35.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:55:35 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:55:35.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:55:35 smithi115 ceph-mon[78593]: pgmap v1186: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:55:35.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:55:35 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:55:36.264 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:55:35 smithi115 conmon[90393]: 2022-05-13T23:55:35.905+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:55:36.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:55:36 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:55:36.541 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-13T23:55:36.541 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-13T23:55:36.542 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-13T23:55:36.542 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-13T23:55:36.542 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-13T23:55:36.542 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-13T23:55:36.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-13T23:55:36.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-13T23:55:36.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-13T23:55:36.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-13T23:55:36.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-13T23:55:36.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-13T23:55:36.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-13T23:55:36.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-13T23:55:36.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-13T23:55:36.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-13T23:55:36.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-13T23:55:36.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:55:36.545 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:55:36 smithi047 ceph-mon[77147]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-13T23:55:36.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-13T23:55:36.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-13T23:55:36.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-13T23:55:36.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-13T23:55:36.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-13T23:55:36.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-13T23:55:36.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-13T23:55:36.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-13T23:55:36.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-13T23:55:36.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-13T23:55:36.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-13T23:55:36.548 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-13T23:55:36.548 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-13T23:55:36.548 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-13T23:55:36.548 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-13T23:55:36.549 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-13T23:55:36.549 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-13T23:55:36.549 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:55:36.549 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-13T23:55:36.549 INFO:journalctl@ceph.mon.b.smithi047.stdout: During handling of the above exception, another exception occurred: 2022-05-13T23:55:36.550 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-13T23:55:36.550 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-13T23:55:36.550 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-13T23:55:36.550 INFO:journalctl@ceph.mon.b.smithi047.stdout: return f(*arg) 2022-05-13T23:55:36.550 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-13T23:55:36.551 INFO:journalctl@ceph.mon.b.smithi047.stdout: self._write_client_files(client_files, host) 2022-05-13T23:55:36.551 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-13T23:55:36.551 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-13T23:55:36.551 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-13T23:55:36.552 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-13T23:55:36.552 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-13T23:55:36.552 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.event_loop.get_result(coro) 2022-05-13T23:55:36.552 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-13T23:55:36.552 INFO:journalctl@ceph.mon.b.smithi047.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-13T23:55:36.553 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-13T23:55:36.553 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.__get_result() 2022-05-13T23:55:36.553 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-13T23:55:36.553 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise self._exception 2022-05-13T23:55:36.553 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-13T23:55:36.554 INFO:journalctl@ceph.mon.b.smithi047.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-13T23:55:36.554 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-13T23:55:36.554 INFO:journalctl@ceph.mon.b.smithi047.stdout: conn = await self._remote_connection(host, addr) 2022-05-13T23:55:36.554 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-13T23:55:36.555 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise 2022-05-13T23:55:36.555 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-13T23:55:36.555 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.gen.throw(type, value, traceback) 2022-05-13T23:55:36.555 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-13T23:55:36.555 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OrchestratorError(msg) 2022-05-13T23:55:36.556 INFO:journalctl@ceph.mon.b.smithi047.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:55:36.556 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:55:36 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:55:36.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:55:36 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:55:36.608 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-13T23:55:36.608 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-13T23:55:36.608 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-13T23:55:36.609 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-13T23:55:36.609 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-13T23:55:36.609 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-13T23:55:36.609 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-13T23:55:36.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-13T23:55:36.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-13T23:55:36.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-13T23:55:36.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-13T23:55:36.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-13T23:55:36.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-13T23:55:36.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-13T23:55:36.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-13T23:55:36.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-13T23:55:36.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-13T23:55:36.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:55:36.612 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:55:36 smithi115 ceph-mon[78593]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-13T23:55:36.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-13T23:55:36.613 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-13T23:55:36.613 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-13T23:55:36.613 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-13T23:55:36.613 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-13T23:55:36.614 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-13T23:55:36.614 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-13T23:55:36.614 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-13T23:55:36.614 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-13T23:55:36.615 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-13T23:55:36.615 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-13T23:55:36.615 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-13T23:55:36.615 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-13T23:55:36.615 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-13T23:55:36.616 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-13T23:55:36.616 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-13T23:55:36.616 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-13T23:55:36.616 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:55:36.617 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-13T23:55:36.617 INFO:journalctl@ceph.mon.c.smithi115.stdout: During handling of the above exception, another exception occurred: 2022-05-13T23:55:36.617 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-13T23:55:36.617 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-13T23:55:36.618 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-13T23:55:36.618 INFO:journalctl@ceph.mon.c.smithi115.stdout: return f(*arg) 2022-05-13T23:55:36.618 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-13T23:55:36.618 INFO:journalctl@ceph.mon.c.smithi115.stdout: self._write_client_files(client_files, host) 2022-05-13T23:55:36.619 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-13T23:55:36.619 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-13T23:55:36.619 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-13T23:55:36.619 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-13T23:55:36.620 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-13T23:55:36.620 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.event_loop.get_result(coro) 2022-05-13T23:55:36.620 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-13T23:55:36.620 INFO:journalctl@ceph.mon.c.smithi115.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-13T23:55:36.621 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-13T23:55:36.621 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.__get_result() 2022-05-13T23:55:36.621 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-13T23:55:36.621 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise self._exception 2022-05-13T23:55:36.622 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-13T23:55:36.622 INFO:journalctl@ceph.mon.c.smithi115.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-13T23:55:36.622 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-13T23:55:36.622 INFO:journalctl@ceph.mon.c.smithi115.stdout: conn = await self._remote_connection(host, addr) 2022-05-13T23:55:36.623 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-13T23:55:36.623 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise 2022-05-13T23:55:36.623 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-13T23:55:36.623 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.gen.throw(type, value, traceback) 2022-05-13T23:55:36.624 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-13T23:55:36.624 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OrchestratorError(msg) 2022-05-13T23:55:36.624 INFO:journalctl@ceph.mon.c.smithi115.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:55:36.624 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:55:36 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:55:37.265 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:55:36 smithi115 conmon[90393]: 2022-05-13T23:55:36.864+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:55:37.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:55:37 smithi047 ceph-mon[77147]: pgmap v1187: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:55:37.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:55:37 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:55:37.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:55:37 smithi115 ceph-mon[78593]: pgmap v1187: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:55:37.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:55:37 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:55:38.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:55:37 smithi115 conmon[90393]: 2022-05-13T23:55:37.855+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:55:38.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:55:38 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:55:38.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:55:38 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:55:38.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:55:38 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-13T23:55:38.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:55:38 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:55:38.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:55:38 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:55:38.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:55:38 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-13T23:55:39.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:55:38 smithi115 conmon[90393]: 2022-05-13T23:55:38.850+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:55:39.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:55:39 smithi047 ceph-mon[77147]: pgmap v1188: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:55:39.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:55:39 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:55:39.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:55:39 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 1670 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:55:39.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:55:39 smithi115 ceph-mon[78593]: pgmap v1188: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:55:39.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:55:39 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:55:39.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:55:39 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 1670 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:55:40.280 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:55:39 smithi115 conmon[90393]: 2022-05-13T23:55:39.879+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:55:40.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:55:40 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:55:40.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:55:40 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:55:41.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:55:40 smithi115 conmon[90393]: 2022-05-13T23:55:40.835+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:55:41.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:55:41 smithi047 ceph-mon[77147]: pgmap v1189: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:55:41.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:55:41 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:55:41.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:55:41 smithi115 ceph-mon[78593]: pgmap v1189: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:55:41.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:55:41 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:55:42.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:55:41 smithi115 conmon[90393]: 2022-05-13T23:55:41.787+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:55:42.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:55:42 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:55:42.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:55:42 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:55:42.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:55:42 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-13T23:55:42.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:55:42 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:55:42.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:55:42 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:55:42.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:55:42 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-13T23:55:43.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:55:42 smithi115 conmon[90393]: 2022-05-13T23:55:42.797+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:55:43.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:55:43 smithi115 ceph-mon[78593]: pgmap v1190: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:55:43.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:55:43 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:55:43.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:55:43 smithi047 ceph-mon[77147]: pgmap v1190: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:55:43.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:55:43 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:55:44.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:55:43 smithi115 conmon[90393]: 2022-05-13T23:55:43.844+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:55:44.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:55:44 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:55:44.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:55:44 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 1675 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:55:44.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:55:44 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:55:44.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:55:44 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 1675 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:55:45.298 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:55:44 smithi115 conmon[90393]: 2022-05-13T23:55:44.863+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:55:45.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:55:45 smithi115 ceph-mon[78593]: pgmap v1191: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:55:45.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:55:45 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:55:45.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:55:45 smithi047 ceph-mon[77147]: pgmap v1191: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:55:45.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:55:45 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:55:46.301 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:55:45 smithi115 conmon[90393]: 2022-05-13T23:55:45.869+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:55:46.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:55:46 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:55:46.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:55:46 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:55:47.303 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:55:46 smithi115 conmon[90393]: 2022-05-13T23:55:46.889+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:55:47.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:55:47 smithi115 ceph-mon[78593]: pgmap v1192: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:55:47.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:55:47 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:55:47.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:55:47 smithi047 ceph-mon[77147]: pgmap v1192: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:55:47.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:55:47 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:55:48.310 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:55:47 smithi115 conmon[90393]: 2022-05-13T23:55:47.878+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:55:48.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:55:48 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:55:48.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:55:48 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:55:49.310 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:55:48 smithi115 conmon[90393]: 2022-05-13T23:55:48.875+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:55:49.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:55:49 smithi115 ceph-mon[78593]: pgmap v1193: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:55:49.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:55:49 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:55:49.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:55:49 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 1680 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:55:49.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:55:49 smithi047 ceph-mon[77147]: pgmap v1193: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:55:49.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:55:49 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:55:49.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:55:49 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 1680 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:55:50.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:55:49 smithi115 conmon[90393]: 2022-05-13T23:55:49.835+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:55:50.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:55:50 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:55:50.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:55:50 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:55:50.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:55:50 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-13T23:55:50.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:55:50 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:55:50.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:55:50 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:55:50.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:55:50 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-13T23:55:51.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:55:50 smithi115 conmon[90393]: 2022-05-13T23:55:50.843+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:55:51.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:55:51 smithi115 ceph-mon[78593]: pgmap v1194: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:55:51.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:55:51 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:55:51.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:55:51 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:55:51.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:55:51 smithi047 ceph-mon[77147]: pgmap v1194: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:55:51.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:55:51 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:55:51.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:55:51 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:55:52.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:55:51 smithi115 conmon[90393]: 2022-05-13T23:55:51.805+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:55:52.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:55:52 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:55:52.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:55:52 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:55:53.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:55:52 smithi115 conmon[90393]: 2022-05-13T23:55:52.821+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:55:53.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:55:53 smithi115 ceph-mon[78593]: pgmap v1195: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:55:53.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:55:53 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:55:53.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:55:53 smithi047 ceph-mon[77147]: pgmap v1195: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:55:53.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:55:53 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:55:54.335 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:55:53 smithi115 conmon[90393]: 2022-05-13T23:55:53.868+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:55:54.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:55:54 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:55:54.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:55:54 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-13T23:55:54.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:55:54 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 1685 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:55:54.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:55:54 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:55:54.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:55:54 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:55:54.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:55:54 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-13T23:55:54.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:55:54 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 1685 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:55:54.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:55:54 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:55:55.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:55:54 smithi115 conmon[90393]: 2022-05-13T23:55:54.913+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:55:55.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:55:55 smithi047 ceph-mon[77147]: pgmap v1196: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:55:55.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:55:55 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:55:55.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:55:55 smithi115 ceph-mon[78593]: pgmap v1196: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:55:55.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:55:55 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:55:56.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:55:55 smithi115 conmon[90393]: 2022-05-13T23:55:55.910+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:55:56.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:55:56 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:55:56.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:55:56 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:55:57.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:55:56 smithi115 conmon[90393]: 2022-05-13T23:55:56.949+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:55:57.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:55:57 smithi047 ceph-mon[77147]: pgmap v1197: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:55:57.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:55:57 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:55:57.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:55:57 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-13T23:55:57.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:55:57 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:55:57.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:55:57 smithi115 ceph-mon[78593]: pgmap v1197: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:55:57.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:55:57 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:55:57.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:55:57 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-13T23:55:57.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:55:57 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:55:58.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:55:57 smithi115 conmon[90393]: 2022-05-13T23:55:57.970+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:55:58.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:55:58 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:55:58.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:55:58 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 1690 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:55:58.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:55:58 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:55:58.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:55:58 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 1690 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:55:59.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:55:58 smithi115 conmon[90393]: 2022-05-13T23:55:58.989+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:55:59.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:55:59 smithi047 ceph-mon[77147]: pgmap v1198: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:55:59.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:55:59 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:55:59.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:55:59 smithi115 ceph-mon[78593]: pgmap v1198: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:55:59.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:55:59 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:56:00.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:56:00 smithi115 conmon[90393]: 2022-05-13T23:56:00.019+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:56:00.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:56:00 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:56:00.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:56:00 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:56:01.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:56:00 smithi115 conmon[90393]: 2022-05-13T23:56:00.995+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:56:01.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:56:01 smithi047 ceph-mon[77147]: pgmap v1199: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:56:01.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:56:01 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:56:01.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:56:01 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-13T23:56:01.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:56:01 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:56:01.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:56:01 smithi115 ceph-mon[78593]: pgmap v1199: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:56:01.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:56:01 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:56:01.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:56:01 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-13T23:56:01.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:56:01 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:56:02.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:56:02 smithi115 conmon[90393]: 2022-05-13T23:56:02.006+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:56:02.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:56:02 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:56:02.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:56:02 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:56:03.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:56:03 smithi115 conmon[90393]: 2022-05-13T23:56:03.028+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:56:03.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:56:03 smithi047 ceph-mon[77147]: pgmap v1200: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:56:03.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:56:03 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:56:03.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:56:03 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 1695 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:56:03.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:56:03 smithi115 ceph-mon[78593]: pgmap v1200: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:56:03.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:56:03 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:56:03.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:56:03 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 1695 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:56:04.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:56:03 smithi115 conmon[90393]: 2022-05-13T23:56:03.993+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:56:04.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:56:04 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:56:04.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:56:04 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:56:05.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:56:05 smithi115 conmon[90393]: 2022-05-13T23:56:05.022+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:56:05.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:56:05 smithi047 ceph-mon[77147]: pgmap v1201: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:56:05.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:56:05 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:56:05.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:56:05 smithi115 ceph-mon[78593]: pgmap v1201: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:56:05.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:56:05 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:56:06.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:56:06 smithi115 conmon[90393]: 2022-05-13T23:56:06.067+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:56:06.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:56:06 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:56:06.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:56:06 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:56:07.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:56:07 smithi115 conmon[90393]: 2022-05-13T23:56:07.088+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:56:07.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:56:07 smithi047 ceph-mon[77147]: pgmap v1202: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:56:07.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:56:07 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:56:07.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:56:07 smithi115 ceph-mon[78593]: pgmap v1202: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:56:07.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:56:07 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:56:08.363 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:56:08 smithi115 conmon[90393]: 2022-05-13T23:56:08.049+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:56:08.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:56:08 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:56:08.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:56:08 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 1700 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:56:08.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:56:08 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:56:08.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:56:08 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 1700 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:56:09.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:56:09 smithi115 conmon[90393]: 2022-05-13T23:56:09.026+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:56:09.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:56:09 smithi047 ceph-mon[77147]: pgmap v1203: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:56:09.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:56:09 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:56:09.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:56:09 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-13T23:56:09.795 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:56:09 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:56:09.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:56:09 smithi115 ceph-mon[78593]: pgmap v1203: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:56:09.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:56:09 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:56:09.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:56:09 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-13T23:56:09.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:56:09 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:56:10.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:56:09 smithi115 conmon[90393]: 2022-05-13T23:56:09.979+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:56:10.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:56:10 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:56:10.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:56:10 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:56:11.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:56:10 smithi115 conmon[90393]: 2022-05-13T23:56:10.935+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:56:11.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:56:11 smithi047 ceph-mon[77147]: pgmap v1204: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-05-13T23:56:11.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:56:11 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:56:11.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:56:11 smithi115 ceph-mon[78593]: pgmap v1204: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-05-13T23:56:11.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:56:11 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:56:12.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:56:11 smithi115 conmon[90393]: 2022-05-13T23:56:11.921+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:56:12.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:56:12 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:56:12.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:56:12 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:56:13.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:56:12 smithi115 conmon[90393]: 2022-05-13T23:56:12.940+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:56:13.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:56:13 smithi115 ceph-mon[78593]: pgmap v1205: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.3 KiB/s rd, 4 op/s 2022-05-13T23:56:13.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:56:13 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:56:13.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:56:13 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-13T23:56:13.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:56:13 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:56:13.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:56:13 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-13T23:56:13.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:56:13 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-13T23:56:13.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:56:13 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 1705 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:56:14.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:56:13 smithi047 ceph-mon[77147]: pgmap v1205: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.3 KiB/s rd, 4 op/s 2022-05-13T23:56:14.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:56:13 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:56:14.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:56:13 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-13T23:56:14.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:56:13 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:56:14.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:56:13 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-13T23:56:14.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:56:13 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-13T23:56:14.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:56:13 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 1705 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:56:14.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:56:13 smithi115 conmon[90393]: 2022-05-13T23:56:13.984+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:56:14.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:56:14 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:56:15.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:56:14 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:56:15.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:56:15 smithi115 conmon[90393]: 2022-05-13T23:56:15.013+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:56:15.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:56:15 smithi115 ceph-mon[78593]: pgmap v1206: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.3 KiB/s rd, 4 op/s 2022-05-13T23:56:15.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:56:15 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:56:16.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:56:15 smithi047 ceph-mon[77147]: pgmap v1206: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.3 KiB/s rd, 4 op/s 2022-05-13T23:56:16.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:56:15 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:56:16.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:56:15 smithi115 conmon[90393]: 2022-05-13T23:56:15.978+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:56:16.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:56:16 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:56:17.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:56:16 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:56:17.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:56:16 smithi115 conmon[90393]: 2022-05-13T23:56:16.986+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:56:17.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:56:17 smithi115 ceph-mon[78593]: pgmap v1207: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-05-13T23:56:17.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:56:17 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:56:17.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:56:17 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-13T23:56:17.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:56:17 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:56:18.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:56:17 smithi047 ceph-mon[77147]: pgmap v1207: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-05-13T23:56:18.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:56:17 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:56:18.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:56:17 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-13T23:56:18.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:56:17 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:56:18.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:56:18 smithi115 conmon[90393]: 2022-05-13T23:56:18.006+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:56:18.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:56:18 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:56:18.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:56:18 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 1710 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:56:19.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:56:18 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:56:19.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:56:18 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 1710 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:56:19.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:56:19 smithi115 conmon[90393]: 2022-05-13T23:56:19.054+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:56:19.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:56:19 smithi115 ceph-mon[78593]: pgmap v1208: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.3 KiB/s rd, 4 op/s 2022-05-13T23:56:19.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:56:19 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:56:20.047 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:56:19 smithi047 ceph-mon[77147]: pgmap v1208: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.3 KiB/s rd, 4 op/s 2022-05-13T23:56:20.048 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:56:19 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:56:20.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:56:20 smithi115 conmon[90393]: 2022-05-13T23:56:20.104+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:56:20.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:56:20 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:56:21.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:56:20 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:56:21.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:56:21 smithi115 conmon[90393]: 2022-05-13T23:56:21.101+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:56:21.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:56:21 smithi115 ceph-mon[78593]: pgmap v1209: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-05-13T23:56:21.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:56:21 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:56:21.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:56:21 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-13T23:56:21.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:56:21 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:56:22.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:56:21 smithi047 ceph-mon[77147]: pgmap v1209: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-05-13T23:56:22.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:56:21 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:56:22.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:56:21 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-13T23:56:22.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:56:21 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:56:22.574 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:56:22 smithi115 conmon[90393]: 2022-05-13T23:56:22.135+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:56:22.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:56:22 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:56:23.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:56:22 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:56:23.575 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:56:23 smithi115 conmon[90393]: 2022-05-13T23:56:23.118+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:56:23.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:56:23 smithi115 ceph-mon[78593]: pgmap v1210: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:56:23.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:56:23 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:56:23.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:56:23 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 1715 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:56:24.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:56:23 smithi047 ceph-mon[77147]: pgmap v1210: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:56:24.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:56:23 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:56:24.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:56:23 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 1715 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:56:24.582 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:56:24 smithi115 conmon[90393]: 2022-05-13T23:56:24.118+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:56:24.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:56:24 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:56:25.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:56:24 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:56:25.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:56:25 smithi115 conmon[90393]: 2022-05-13T23:56:25.101+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:56:25.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:56:25 smithi115 ceph-mon[78593]: pgmap v1211: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:56:25.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:56:25 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:56:26.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:56:25 smithi047 ceph-mon[77147]: pgmap v1211: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:56:26.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:56:25 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:56:26.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:56:26 smithi115 conmon[90393]: 2022-05-13T23:56:26.061+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:56:26.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:56:26 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:56:27.045 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:56:26 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:56:27.360 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:56:27 smithi115 conmon[90393]: 2022-05-13T23:56:27.055+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:56:27.874 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:56:27 smithi115 ceph-mon[78593]: pgmap v1212: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:56:27.874 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:56:27 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:56:28.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:56:27 smithi047 ceph-mon[77147]: pgmap v1212: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:56:28.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:56:27 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:56:28.366 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:56:28 smithi115 conmon[90393]: 2022-05-13T23:56:28.056+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:56:28.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:56:28 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:56:28.862 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:56:28 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 1720 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:56:29.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:56:28 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:56:29.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:56:28 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 1720 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:56:29.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:56:29 smithi115 conmon[90393]: 2022-05-13T23:56:29.009+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:56:29.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:56:29 smithi115 ceph-mon[78593]: pgmap v1213: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:56:29.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:56:29 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:56:29.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:56:29 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-13T23:56:29.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:56:29 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:56:30.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:56:29 smithi047 ceph-mon[77147]: pgmap v1213: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:56:30.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:56:29 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:56:30.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:56:29 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-13T23:56:30.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:56:29 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:56:30.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:56:30 smithi115 conmon[90393]: 2022-05-13T23:56:30.029+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:56:30.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:56:30 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:56:31.046 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:56:30 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:56:31.361 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:56:31 smithi115 conmon[90393]: 2022-05-13T23:56:31.072+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:56:31.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:56:31 smithi115 ceph-mon[78593]: pgmap v1214: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:56:31.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:56:31 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:56:32.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:56:31 smithi047 ceph-mon[77147]: pgmap v1214: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:56:32.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:56:31 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:56:32.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:56:32 smithi115 conmon[90393]: 2022-05-13T23:56:32.047+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:56:32.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:56:32 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:56:33.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:56:32 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:56:33.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:56:33 smithi115 conmon[90393]: 2022-05-13T23:56:33.039+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:56:34.006 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:56:33 smithi115 ceph-mon[78593]: pgmap v1215: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:56:34.006 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:56:33 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:56:34.007 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:56:33 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-13T23:56:34.007 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:56:33 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:56:34.007 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:56:33 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 1725 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:56:34.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:56:33 smithi047 ceph-mon[77147]: pgmap v1215: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:56:34.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:56:33 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:56:34.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:56:33 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-13T23:56:34.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:56:33 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:56:34.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:56:33 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 1725 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:56:34.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:56:34 smithi115 conmon[90393]: 2022-05-13T23:56:34.005+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:56:34.875 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:56:34 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:56:34.894 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:56:34 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:56:35.277 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:56:35 smithi115 conmon[90393]: 2022-05-13T23:56:35.013+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:56:35.541 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:56:35 smithi047 conmon[78521]: 2022-05-13T23:56:35.375+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:56:35.541 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:56:35 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-13T23:56:35.541 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:56:35 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-13T23:56:35.542 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:56:35 smithi047 conmon[78521]: yield 2022-05-13T23:56:35.542 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:56:35 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-13T23:56:35.542 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:56:35 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-13T23:56:35.542 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:56:35 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-13T23:56:35.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:56:35 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-13T23:56:35.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:56:35 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-13T23:56:35.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:56:35 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-13T23:56:35.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:56:35 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-13T23:56:35.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:56:35 smithi047 conmon[78521]: raise exceptions[0] 2022-05-13T23:56:35.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:56:35 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-13T23:56:35.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:56:35 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-13T23:56:35.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:56:35 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-13T23:56:35.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:56:35 smithi047 conmon[78521]: return (yield from fut) 2022-05-13T23:56:35.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:56:35 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-13T23:56:35.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:56:35 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-13T23:56:35.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:56:35 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:56:36.027 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:56:35 smithi115 ceph-mon[78593]: pgmap v1216: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:56:36.028 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:56:35 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-13T23:56:36.028 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:56:35 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-13T23:56:36.028 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:56:35 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-13T23:56:36.028 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:56:35 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:56:36.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:56:35 smithi047 ceph-mon[77147]: pgmap v1216: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:56:36.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:56:35 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-13T23:56:36.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:56:35 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-13T23:56:36.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:56:35 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-13T23:56:36.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:56:35 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:56:36.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:56:36 smithi115 conmon[90393]: 2022-05-13T23:56:36.026+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:56:36.982 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:56:36 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:56:36.982 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-13T23:56:36.983 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-13T23:56:36.983 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-13T23:56:36.983 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-13T23:56:36.983 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-13T23:56:36.984 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-13T23:56:36.984 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-13T23:56:36.984 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-13T23:56:36.984 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-13T23:56:36.984 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-13T23:56:36.985 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-13T23:56:36.985 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-13T23:56:36.985 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-13T23:56:36.985 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-13T23:56:36.986 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-13T23:56:36.986 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-13T23:56:36.986 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-13T23:56:36.986 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:56:36.986 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:56:36 smithi115 ceph-mon[78593]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-13T23:56:36.987 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:56:36 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:56:37.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:56:36 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:56:37.042 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-13T23:56:37.042 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-13T23:56:37.042 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-13T23:56:37.042 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-13T23:56:37.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-13T23:56:37.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-13T23:56:37.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-13T23:56:37.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-13T23:56:37.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-13T23:56:37.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-13T23:56:37.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-13T23:56:37.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-13T23:56:37.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-13T23:56:37.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-13T23:56:37.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-13T23:56:37.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-13T23:56:37.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-13T23:56:37.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:56:37.046 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:56:36 smithi047 ceph-mon[77147]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-13T23:56:37.046 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:56:36 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:56:37.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:56:36 smithi115 conmon[90393]: 2022-05-13T23:56:36.981+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:56:37.974 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:56:37 smithi115 ceph-mon[78593]: pgmap v1217: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:56:37.975 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:56:37 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:56:37.975 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:56:37 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-13T23:56:37.975 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:56:37 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:56:38.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:56:37 smithi047 ceph-mon[77147]: pgmap v1217: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:56:38.072 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:56:37 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:56:38.073 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:56:37 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-13T23:56:38.073 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:56:37 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:56:38.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:56:37 smithi115 conmon[90393]: 2022-05-13T23:56:37.968+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:56:38.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:56:38 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:56:38.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:56:38 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 1730 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:56:38.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:56:38 smithi047 conmon[78521]: 2022-05-13T23:56:38.448+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:56:38.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:56:38 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-13T23:56:38.797 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:56:38 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-13T23:56:38.797 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:56:38 smithi047 conmon[78521]: yield 2022-05-13T23:56:38.797 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:56:38 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-13T23:56:38.798 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:56:38 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-13T23:56:38.798 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:56:38 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-13T23:56:38.799 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:56:38 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-13T23:56:38.799 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:56:38 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-13T23:56:38.799 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:56:38 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-13T23:56:38.800 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:56:38 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-13T23:56:38.800 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:56:38 smithi047 conmon[78521]: raise exceptions[0] 2022-05-13T23:56:38.801 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:56:38 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-13T23:56:38.801 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:56:38 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-13T23:56:38.802 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:56:38 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-13T23:56:38.802 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:56:38 smithi047 conmon[78521]: return (yield from fut) 2022-05-13T23:56:38.803 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:56:38 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-13T23:56:38.803 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:56:38 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-13T23:56:38.803 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:56:38 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:56:38.804 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:56:38 smithi047 conmon[78521]: 2022-05-13T23:56:38.448+0000 7f533f777700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-13T23:56:38.804 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:56:38 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-13T23:56:38.804 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:56:38 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-13T23:56:38.805 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:56:38 smithi047 conmon[78521]: yield 2022-05-13T23:56:38.805 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:56:38 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-13T23:56:38.806 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:56:38 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-13T23:56:38.806 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:56:38 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-13T23:56:38.807 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:56:38 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-13T23:56:38.807 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:56:38 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-13T23:56:38.807 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:56:38 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-13T23:56:38.808 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:56:38 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-13T23:56:38.808 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:56:38 smithi047 conmon[78521]: raise exceptions[0] 2022-05-13T23:56:38.809 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:56:38 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-13T23:56:38.809 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:56:38 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-13T23:56:38.809 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:56:38 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-13T23:56:38.810 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:56:38 smithi047 conmon[78521]: return (yield from fut) 2022-05-13T23:56:38.810 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:56:38 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-13T23:56:38.811 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:56:38 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-13T23:56:38.811 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:56:38 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:56:38.811 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:56:38 smithi047 conmon[78521]: 2022-05-13T23:56:38.812 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:56:38 smithi047 conmon[78521]: During handling of the above exception, another exception occurred: 2022-05-13T23:56:38.812 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:56:38 smithi047 conmon[78521]: 2022-05-13T23:56:38.812 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:56:38 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-13T23:56:38.813 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:56:38 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-13T23:56:38.813 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:56:38 smithi047 conmon[78521]: return f(*arg) 2022-05-13T23:56:38.814 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:56:38 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-13T23:56:38.814 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:56:38 smithi047 conmon[78521]: self._write_client_files(client_files, host) 2022-05-13T23:56:38.814 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:56:38 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-13T23:56:38.815 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:56:38 smithi047 conmon[78521]: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-13T23:56:38.816 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:56:38 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-13T23:56:38.816 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:56:38 smithi047 conmon[78521]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-13T23:56:38.817 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:56:38 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-13T23:56:38.817 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:56:38 smithi047 conmon[78521]: return self.event_loop.get_result(coro) 2022-05-13T23:56:38.817 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:56:38 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-13T23:56:38.818 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:56:38 smithi047 conmon[78521]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-13T23:56:38.818 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:56:38 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-13T23:56:38.819 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:56:38 smithi047 conmon[78521]: return self.__get_result() 2022-05-13T23:56:38.819 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:56:38 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-13T23:56:38.819 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:56:38 smithi047 conmon[78521]: raise self._exception 2022-05-13T23:56:38.820 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:56:38 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-13T23:56:38.820 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:56:38 smithi047 conmon[78521]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-13T23:56:38.820 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:56:38 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-13T23:56:38.821 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:56:38 smithi047 conmon[78521]: conn = await self._remote_connection(host, addr) 2022-05-13T23:56:38.821 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:56:38 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-13T23:56:38.822 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:56:38 smithi047 conmon[78521]: raise 2022-05-13T23:56:38.822 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:56:38 smithi047 conmon[78521]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-13T23:56:38.822 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:56:38 smithi047 conmon[78521]: self.gen.throw(type, value, traceback) 2022-05-13T23:56:38.825 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:56:38 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-13T23:56:38.826 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:56:38 smithi047 conmon[78521]: raise OrchestratorError(msg) 2022-05-13T23:56:38.826 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:56:38 smithi047 conmon[78521]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:56:38.950 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:56:38 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:56:38.951 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:56:38 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 1730 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:56:39.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:56:38 smithi115 conmon[90393]: 2022-05-13T23:56:38.948+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:56:39.946 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:56:39 smithi115 ceph-mon[78593]: pgmap v1218: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:56:39.947 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:56:39 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:56:39.947 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-13T23:56:39.947 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-13T23:56:39.948 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-13T23:56:39.948 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-13T23:56:39.948 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-13T23:56:39.948 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-13T23:56:39.948 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-13T23:56:39.949 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-13T23:56:39.949 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-13T23:56:39.950 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-13T23:56:39.950 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-13T23:56:39.950 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-13T23:56:39.950 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-13T23:56:39.951 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-13T23:56:39.951 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-13T23:56:39.951 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-13T23:56:39.951 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-13T23:56:39.951 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:56:39.952 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:56:39 smithi115 ceph-mon[78593]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-13T23:56:39.952 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-13T23:56:39.952 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-13T23:56:39.952 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-13T23:56:39.952 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-13T23:56:39.953 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-13T23:56:39.953 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-13T23:56:39.953 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-13T23:56:39.954 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-13T23:56:39.954 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-13T23:56:39.954 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-13T23:56:39.954 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-13T23:56:39.955 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-13T23:56:39.955 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-13T23:56:39.955 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-13T23:56:39.955 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-13T23:56:39.956 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-13T23:56:39.956 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-13T23:56:39.956 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:56:39.956 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-13T23:56:39.956 INFO:journalctl@ceph.mon.c.smithi115.stdout: During handling of the above exception, another exception occurred: 2022-05-13T23:56:39.957 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-13T23:56:39.957 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-13T23:56:39.957 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-13T23:56:39.957 INFO:journalctl@ceph.mon.c.smithi115.stdout: return f(*arg) 2022-05-13T23:56:39.957 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-13T23:56:39.958 INFO:journalctl@ceph.mon.c.smithi115.stdout: self._write_client_files(client_files, host) 2022-05-13T23:56:39.958 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-13T23:56:39.958 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-13T23:56:39.958 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-13T23:56:39.959 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-13T23:56:39.959 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-13T23:56:39.959 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.event_loop.get_result(coro) 2022-05-13T23:56:39.959 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-13T23:56:39.959 INFO:journalctl@ceph.mon.c.smithi115.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-13T23:56:39.960 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-13T23:56:39.960 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.__get_result() 2022-05-13T23:56:39.960 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-13T23:56:39.960 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise self._exception 2022-05-13T23:56:39.960 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-13T23:56:39.961 INFO:journalctl@ceph.mon.c.smithi115.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-13T23:56:39.961 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-13T23:56:39.961 INFO:journalctl@ceph.mon.c.smithi115.stdout: conn = await self._remote_connection(host, addr) 2022-05-13T23:56:39.961 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-13T23:56:39.962 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise 2022-05-13T23:56:39.962 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-13T23:56:39.962 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.gen.throw(type, value, traceback) 2022-05-13T23:56:39.962 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-13T23:56:39.962 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OrchestratorError(msg) 2022-05-13T23:56:39.963 INFO:journalctl@ceph.mon.c.smithi115.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:56:39.963 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:56:39 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:56:40.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:56:39 smithi047 ceph-mon[77147]: pgmap v1218: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:56:40.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:56:39 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:56:40.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-13T23:56:40.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-13T23:56:40.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-13T23:56:40.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-13T23:56:40.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-13T23:56:40.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-13T23:56:40.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-13T23:56:40.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-13T23:56:40.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-13T23:56:40.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-13T23:56:40.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-13T23:56:40.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-13T23:56:40.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-13T23:56:40.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-13T23:56:40.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-13T23:56:40.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-13T23:56:40.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-13T23:56:40.049 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:56:40.049 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:56:39 smithi047 ceph-mon[77147]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-13T23:56:40.049 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-13T23:56:40.049 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-13T23:56:40.050 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-13T23:56:40.050 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-13T23:56:40.050 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-13T23:56:40.051 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-13T23:56:40.051 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-13T23:56:40.051 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-13T23:56:40.052 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-13T23:56:40.052 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-13T23:56:40.052 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-13T23:56:40.053 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-13T23:56:40.053 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-13T23:56:40.054 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-13T23:56:40.054 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-13T23:56:40.055 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-13T23:56:40.055 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-13T23:56:40.055 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:56:40.056 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-13T23:56:40.056 INFO:journalctl@ceph.mon.b.smithi047.stdout: During handling of the above exception, another exception occurred: 2022-05-13T23:56:40.056 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-13T23:56:40.057 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-13T23:56:40.057 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-13T23:56:40.057 INFO:journalctl@ceph.mon.b.smithi047.stdout: return f(*arg) 2022-05-13T23:56:40.058 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-13T23:56:40.058 INFO:journalctl@ceph.mon.b.smithi047.stdout: self._write_client_files(client_files, host) 2022-05-13T23:56:40.058 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-13T23:56:40.059 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-13T23:56:40.059 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-13T23:56:40.059 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-13T23:56:40.060 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-13T23:56:40.060 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.event_loop.get_result(coro) 2022-05-13T23:56:40.060 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-13T23:56:40.060 INFO:journalctl@ceph.mon.b.smithi047.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-13T23:56:40.061 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-13T23:56:40.061 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.__get_result() 2022-05-13T23:56:40.062 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-13T23:56:40.062 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise self._exception 2022-05-13T23:56:40.062 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-13T23:56:40.063 INFO:journalctl@ceph.mon.b.smithi047.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-13T23:56:40.063 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-13T23:56:40.063 INFO:journalctl@ceph.mon.b.smithi047.stdout: conn = await self._remote_connection(host, addr) 2022-05-13T23:56:40.064 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-13T23:56:40.064 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise 2022-05-13T23:56:40.064 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-13T23:56:40.065 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.gen.throw(type, value, traceback) 2022-05-13T23:56:40.065 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-13T23:56:40.065 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OrchestratorError(msg) 2022-05-13T23:56:40.066 INFO:journalctl@ceph.mon.b.smithi047.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:56:40.066 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:56:39 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:56:40.361 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:56:39 smithi115 conmon[90393]: 2022-05-13T23:56:39.937+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:56:40.927 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:56:40 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:56:41.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:56:40 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:56:41.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:56:40 smithi115 conmon[90393]: 2022-05-13T23:56:40.925+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:56:41.973 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:56:41 smithi115 ceph-mon[78593]: pgmap v1219: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:56:41.973 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:56:41 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:56:41.974 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:56:41 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-13T23:56:41.974 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:56:41 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:56:42.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:56:41 smithi047 ceph-mon[77147]: pgmap v1219: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:56:42.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:56:41 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:56:42.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:56:41 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-13T23:56:42.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:56:41 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:56:42.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:56:41 smithi115 conmon[90393]: 2022-05-13T23:56:41.971+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:56:42.924 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:56:42 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:56:43.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:56:42 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:56:43.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:56:42 smithi115 conmon[90393]: 2022-05-13T23:56:42.922+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:56:43.911 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:56:43 smithi115 ceph-mon[78593]: pgmap v1220: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:56:43.911 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:56:43 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:56:43.912 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:56:43 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 1735 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:56:44.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:56:43 smithi047 ceph-mon[77147]: pgmap v1220: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:56:44.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:56:43 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:56:44.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:56:43 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 1735 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:56:44.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:56:43 smithi115 conmon[90393]: 2022-05-13T23:56:43.910+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:56:45.047 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:56:44 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:56:45.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:56:44 smithi115 conmon[90393]: 2022-05-13T23:56:44.878+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:56:45.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:56:44 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:56:46.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:56:45 smithi047 ceph-mon[77147]: pgmap v1221: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:56:46.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:56:45 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:56:46.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:56:45 smithi115 conmon[90393]: 2022-05-13T23:56:45.838+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:56:46.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:56:45 smithi115 ceph-mon[78593]: pgmap v1221: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:56:46.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:56:45 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:56:47.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:56:46 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:56:47.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:56:46 smithi115 conmon[90393]: 2022-05-13T23:56:46.800+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:56:47.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:56:46 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:56:48.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:56:47 smithi047 ceph-mon[77147]: pgmap v1222: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:56:48.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:56:47 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:56:48.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:56:47 smithi115 conmon[90393]: 2022-05-13T23:56:47.777+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:56:48.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:56:47 smithi115 ceph-mon[78593]: pgmap v1222: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:56:48.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:56:47 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:56:49.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:56:48 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:56:49.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:56:48 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 1740 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:56:49.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:56:48 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:56:49.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:56:48 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 1740 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:56:49.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:56:48 smithi115 conmon[90393]: 2022-05-13T23:56:48.818+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:56:50.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:56:49 smithi047 ceph-mon[77147]: pgmap v1223: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:56:50.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:56:49 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:56:50.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:56:49 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-13T23:56:50.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:56:49 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:56:50.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:56:49 smithi115 conmon[90393]: 2022-05-13T23:56:49.781+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:56:50.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:56:49 smithi115 ceph-mon[78593]: pgmap v1223: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:56:50.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:56:49 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:56:50.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:56:49 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-13T23:56:50.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:56:49 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:56:51.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:56:50 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:56:51.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:56:50 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:56:51.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:56:50 smithi115 conmon[90393]: 2022-05-13T23:56:50.758+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:56:52.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:56:51 smithi047 ceph-mon[77147]: pgmap v1224: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:56:52.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:56:51 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:56:52.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:56:51 smithi115 conmon[90393]: 2022-05-13T23:56:51.806+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:56:52.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:56:51 smithi115 ceph-mon[78593]: pgmap v1224: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:56:52.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:56:51 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:56:53.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:56:52 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:56:53.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:56:52 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:56:53.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:56:52 smithi115 conmon[90393]: 2022-05-13T23:56:52.845+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:56:54.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:56:53 smithi047 ceph-mon[77147]: pgmap v1225: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:56:54.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:56:53 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:56:54.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:56:53 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-13T23:56:54.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:56:53 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:56:54.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:56:53 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 1745 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:56:54.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:56:53 smithi115 conmon[90393]: 2022-05-13T23:56:53.832+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:56:54.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:56:53 smithi115 ceph-mon[78593]: pgmap v1225: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:56:54.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:56:53 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:56:54.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:56:53 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-13T23:56:54.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:56:53 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:56:54.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:56:53 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 1745 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:56:55.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:56:54 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:56:55.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:56:54 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:56:55.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:56:54 smithi115 conmon[90393]: 2022-05-13T23:56:54.835+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:56:56.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:56:55 smithi047 ceph-mon[77147]: pgmap v1226: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:56:56.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:56:55 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:56:56.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:56:55 smithi115 ceph-mon[78593]: pgmap v1226: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:56:56.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:56:55 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:56:56.110 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:56:55 smithi115 conmon[90393]: 2022-05-13T23:56:55.869+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:56:57.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:56:56 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:56:57.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:56:56 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:56:57.111 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:56:56 smithi115 conmon[90393]: 2022-05-13T23:56:56.840+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:56:58.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:56:57 smithi047 ceph-mon[77147]: pgmap v1227: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:56:58.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:56:57 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:56:58.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:56:57 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-13T23:56:58.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:56:57 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:56:58.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:56:57 smithi115 conmon[90393]: 2022-05-13T23:56:57.840+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:56:58.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:56:57 smithi115 ceph-mon[78593]: pgmap v1227: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:56:58.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:56:57 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:56:58.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:56:57 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-13T23:56:58.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:56:57 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:56:59.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:56:58 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:56:59.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:56:58 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 1750 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:56:59.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:56:58 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:56:59.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:56:58 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 1750 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:56:59.112 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:56:58 smithi115 conmon[90393]: 2022-05-13T23:56:58.814+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:57:00.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:56:59 smithi047 ceph-mon[77147]: pgmap v1228: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:57:00.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:56:59 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:57:00.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:56:59 smithi115 conmon[90393]: 2022-05-13T23:56:59.840+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:57:00.112 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:56:59 smithi115 ceph-mon[78593]: pgmap v1228: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:57:00.112 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:56:59 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:57:01.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:57:00 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:57:01.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:57:00 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:57:01.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:57:00 smithi115 conmon[90393]: 2022-05-13T23:57:00.882+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:57:02.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:57:01 smithi047 ceph-mon[77147]: pgmap v1229: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:57:02.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:57:01 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:57:02.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:57:01 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-13T23:57:02.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:57:01 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:57:02.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:57:01 smithi115 conmon[90393]: 2022-05-13T23:57:01.860+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:57:02.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:57:01 smithi115 ceph-mon[78593]: pgmap v1229: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:57:02.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:57:01 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:57:02.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:57:01 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-13T23:57:02.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:57:01 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:57:03.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:57:02 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:57:03.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:57:02 smithi115 conmon[90393]: 2022-05-13T23:57:02.906+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:57:03.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:57:02 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:57:04.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:57:03 smithi047 ceph-mon[77147]: pgmap v1230: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:57:04.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:57:03 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:57:04.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:57:03 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 1755 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:57:04.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:57:03 smithi115 conmon[90393]: 2022-05-13T23:57:03.905+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:57:04.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:57:03 smithi115 ceph-mon[78593]: pgmap v1230: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:57:04.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:57:03 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:57:04.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:57:03 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 1755 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:57:05.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:57:04 smithi047 ceph-mon[77147]: pgmap v1231: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:57:05.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:57:04 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:57:05.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:57:04 smithi115 conmon[90393]: 2022-05-13T23:57:04.906+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:57:05.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:57:04 smithi115 ceph-mon[78593]: pgmap v1231: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:57:05.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:57:04 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:57:06.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:57:05 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:57:06.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:57:05 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:57:06.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:57:05 smithi115 conmon[90393]: 2022-05-13T23:57:05.905+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:57:07.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:57:06 smithi047 ceph-mon[77147]: pgmap v1232: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:57:07.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:57:06 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:57:07.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:57:06 smithi115 conmon[90393]: 2022-05-13T23:57:06.952+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:57:07.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:57:06 smithi115 ceph-mon[78593]: pgmap v1232: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:57:07.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:57:06 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:57:07.989 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:57:07 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:57:08.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:57:07 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:57:08.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:57:07 smithi115 conmon[90393]: 2022-05-13T23:57:07.987+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:57:08.986 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:57:08 smithi115 ceph-mon[78593]: pgmap v1233: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:57:08.987 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:57:08 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:57:08.987 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:57:08 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 1760 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:57:09.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:57:08 smithi047 ceph-mon[77147]: pgmap v1233: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:57:09.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:57:08 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:57:09.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:57:08 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 1760 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:57:09.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:57:08 smithi115 conmon[90393]: 2022-05-13T23:57:08.985+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:57:09.994 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:57:09 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:57:09.994 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:57:09 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-13T23:57:09.994 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:57:09 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:57:10.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:57:09 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:57:10.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:57:09 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-13T23:57:10.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:57:09 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:57:10.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:57:09 smithi115 conmon[90393]: 2022-05-13T23:57:09.990+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:57:11.010 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:57:10 smithi115 ceph-mon[78593]: pgmap v1234: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:57:11.010 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:57:10 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:57:11.047 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:57:10 smithi047 ceph-mon[77147]: pgmap v1234: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:57:11.048 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:57:10 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:57:11.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:57:11 smithi115 conmon[90393]: 2022-05-13T23:57:11.009+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:57:12.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:57:11 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:57:12.051 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:57:11 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:57:12.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:57:12 smithi115 conmon[90393]: 2022-05-13T23:57:12.050+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:57:13.008 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:57:12 smithi115 ceph-mon[78593]: pgmap v1235: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:57:13.009 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:57:12 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:57:13.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:57:12 smithi047 ceph-mon[77147]: pgmap v1235: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:57:13.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:57:12 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:57:13.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:57:13 smithi115 conmon[90393]: 2022-05-13T23:57:13.007+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:57:14.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:57:13 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:57:14.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:57:13 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-13T23:57:14.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:57:13 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:57:14.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:57:13 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-13T23:57:14.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:57:13 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-13T23:57:14.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:57:13 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 1765 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:57:14.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:57:13 smithi115 conmon[90393]: 2022-05-13T23:57:13.962+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:57:14.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:57:13 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:57:14.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:57:13 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-13T23:57:14.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:57:13 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:57:14.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:57:13 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-13T23:57:14.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:57:13 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-13T23:57:14.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:57:13 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 1765 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:57:15.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:57:14 smithi047 ceph-mon[77147]: pgmap v1236: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:57:15.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:57:14 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:57:15.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:57:14 smithi115 conmon[90393]: 2022-05-13T23:57:14.920+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:57:15.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:57:14 smithi115 ceph-mon[78593]: pgmap v1236: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:57:15.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:57:14 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:57:16.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:57:15 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:57:16.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:57:15 smithi115 conmon[90393]: 2022-05-13T23:57:15.954+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:57:16.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:57:15 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:57:17.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:57:16 smithi047 ceph-mon[77147]: pgmap v1237: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:57:17.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:57:16 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:57:17.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:57:16 smithi115 conmon[90393]: 2022-05-13T23:57:16.919+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:57:17.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:57:16 smithi115 ceph-mon[78593]: pgmap v1237: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:57:17.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:57:16 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:57:18.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:57:17 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:57:18.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:57:17 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-13T23:57:18.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:57:17 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:57:18.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:57:17 smithi115 conmon[90393]: 2022-05-13T23:57:17.924+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:57:18.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:57:17 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:57:18.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:57:17 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-13T23:57:18.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:57:17 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:57:19.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:57:18 smithi047 ceph-mon[77147]: pgmap v1238: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:57:19.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:57:18 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:57:19.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:57:18 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 1770 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:57:19.112 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:57:18 smithi115 conmon[90393]: 2022-05-13T23:57:18.898+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:57:19.113 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:57:18 smithi115 ceph-mon[78593]: pgmap v1238: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:57:19.113 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:57:18 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:57:19.114 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:57:18 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 1770 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:57:20.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:57:19 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:57:20.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:57:19 smithi115 conmon[90393]: 2022-05-13T23:57:19.927+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:57:20.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:57:19 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:57:21.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:57:20 smithi047 ceph-mon[77147]: pgmap v1239: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:57:21.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:57:20 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:57:21.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:57:20 smithi115 conmon[90393]: 2022-05-13T23:57:20.951+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:57:21.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:57:20 smithi115 ceph-mon[78593]: pgmap v1239: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:57:21.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:57:20 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:57:22.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:57:21 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:57:22.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:57:21 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-13T23:57:22.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:57:21 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:57:22.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:57:21 smithi115 conmon[90393]: 2022-05-13T23:57:21.967+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:57:22.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:57:21 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:57:22.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:57:21 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-13T23:57:22.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:57:21 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:57:23.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:57:22 smithi047 ceph-mon[77147]: pgmap v1240: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:57:23.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:57:22 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:57:23.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:57:22 smithi115 conmon[90393]: 2022-05-13T23:57:22.920+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:57:23.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:57:22 smithi115 ceph-mon[78593]: pgmap v1240: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:57:23.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:57:22 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:57:24.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:57:23 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:57:24.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:57:23 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 1775 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:57:24.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:57:23 smithi115 conmon[90393]: 2022-05-13T23:57:23.877+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:57:24.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:57:23 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:57:24.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:57:23 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 1775 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:57:25.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:57:24 smithi047 ceph-mon[77147]: pgmap v1241: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:57:25.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:57:24 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:57:25.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:57:24 smithi115 conmon[90393]: 2022-05-13T23:57:24.835+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:57:25.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:57:24 smithi115 ceph-mon[78593]: pgmap v1241: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:57:25.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:57:24 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:57:26.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:57:25 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:57:26.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:57:25 smithi115 conmon[90393]: 2022-05-13T23:57:25.884+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:57:26.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:57:25 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:57:27.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:57:26 smithi047 ceph-mon[77147]: pgmap v1242: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:57:27.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:57:26 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:57:27.110 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:57:26 smithi115 conmon[90393]: 2022-05-13T23:57:26.872+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:57:27.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:57:26 smithi115 ceph-mon[78593]: pgmap v1242: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:57:27.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:57:26 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:57:28.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:57:27 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:57:28.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:57:27 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:57:28.111 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:57:27 smithi115 conmon[90393]: 2022-05-13T23:57:27.873+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:57:29.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:57:28 smithi115 conmon[90393]: 2022-05-13T23:57:28.875+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:57:29.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:57:28 smithi115 ceph-mon[78593]: pgmap v1243: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:57:29.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:57:28 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:57:29.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:57:28 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 1780 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:57:29.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:57:28 smithi047 ceph-mon[77147]: pgmap v1243: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:57:29.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:57:28 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:57:29.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:57:28 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 1780 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:57:30.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:57:29 smithi115 conmon[90393]: 2022-05-13T23:57:29.898+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:57:30.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:57:29 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:57:30.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:57:29 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-13T23:57:30.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:57:29 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:57:30.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:57:29 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:57:30.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:57:29 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-13T23:57:30.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:57:29 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:57:31.110 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:57:30 smithi115 conmon[90393]: 2022-05-13T23:57:30.887+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:57:31.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:57:30 smithi115 ceph-mon[78593]: pgmap v1244: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:57:31.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:57:30 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:57:31.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:57:30 smithi047 ceph-mon[77147]: pgmap v1244: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:57:31.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:57:30 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:57:32.110 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:57:31 smithi115 conmon[90393]: 2022-05-13T23:57:31.887+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:57:32.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:57:31 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:57:32.301 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:57:31 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:57:33.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:57:32 smithi047 ceph-mon[77147]: pgmap v1245: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:57:33.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:57:32 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:57:33.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:57:32 smithi115 conmon[90393]: 2022-05-13T23:57:32.869+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:57:33.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:57:32 smithi115 ceph-mon[78593]: pgmap v1245: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:57:33.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:57:32 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:57:34.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:57:33 smithi115 conmon[90393]: 2022-05-13T23:57:33.834+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:57:34.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:57:33 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:57:34.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:57:33 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-13T23:57:34.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:57:33 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:57:34.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:57:33 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 1785 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:57:34.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:57:33 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:57:34.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:57:33 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-13T23:57:34.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:57:33 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:57:34.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:57:33 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 1785 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:57:35.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:57:34 smithi115 conmon[90393]: 2022-05-13T23:57:34.823+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:57:35.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:57:34 smithi115 ceph-mon[78593]: pgmap v1246: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:57:35.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:57:34 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:57:35.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:57:34 smithi047 ceph-mon[77147]: pgmap v1246: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:57:35.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:57:34 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:57:36.118 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:57:35 smithi115 conmon[90393]: 2022-05-13T23:57:35.782+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:57:36.119 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:57:35 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:57:36.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:57:35 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:57:37.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:57:36 smithi115 conmon[90393]: 2022-05-13T23:57:36.748+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:57:37.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:57:36 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:57:37.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:57:36 smithi115 ceph-mon[78593]: pgmap v1247: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:57:37.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:57:36 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:57:37.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:57:36 smithi047 ceph-mon[77147]: pgmap v1247: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:57:38.114 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:57:37 smithi115 conmon[90393]: 2022-05-13T23:57:37.791+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:57:38.115 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:57:37 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:57:38.115 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:57:37 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:57:38.115 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:57:37 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-13T23:57:38.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:57:37 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:57:38.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:57:37 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:57:38.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:57:37 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-13T23:57:39.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:57:38 smithi115 conmon[90393]: 2022-05-13T23:57:38.804+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:57:39.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:57:38 smithi115 ceph-mon[78593]: pgmap v1248: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:57:39.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:57:38 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:57:39.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:57:38 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-13T23:57:39.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:57:38 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-13T23:57:39.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:57:38 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-13T23:57:39.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:57:38 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 1790 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:57:39.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:57:38 smithi047 ceph-mon[77147]: pgmap v1248: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:57:39.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:57:38 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:57:39.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:57:38 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-13T23:57:39.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:57:38 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-13T23:57:39.295 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:57:38 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-13T23:57:39.295 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:57:38 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 1790 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:57:40.110 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:57:39 smithi115 conmon[90393]: 2022-05-13T23:57:39.787+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:57:40.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:57:39 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:57:40.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:57:39 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:57:41.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:57:40 smithi115 conmon[90393]: 2022-05-13T23:57:40.740+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:57:41.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:57:40 smithi115 ceph-mon[78593]: pgmap v1249: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:57:41.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:57:40 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:57:41.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:57:40 smithi047 ceph-mon[77147]: pgmap v1249: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:57:41.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:57:40 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:57:41.890 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:57:41 smithi047 conmon[78521]: 2022-05-13T23:57:41.552+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:57:41.891 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:57:41 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-13T23:57:41.891 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:57:41 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-13T23:57:41.891 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:57:41 smithi047 conmon[78521]: yield 2022-05-13T23:57:41.892 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:57:41 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-13T23:57:41.892 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:57:41 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-13T23:57:41.892 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:57:41 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-13T23:57:41.892 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:57:41 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-13T23:57:41.892 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:57:41 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-13T23:57:41.893 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:57:41 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-13T23:57:41.893 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:57:41 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-13T23:57:41.893 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:57:41 smithi047 conmon[78521]: raise exceptions[0] 2022-05-13T23:57:41.894 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:57:41 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-13T23:57:41.894 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:57:41 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-13T23:57:41.894 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:57:41 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-13T23:57:41.894 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:57:41 smithi047 conmon[78521]: return (yield from fut) 2022-05-13T23:57:41.894 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:57:41 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-13T23:57:41.895 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:57:41 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-13T23:57:41.895 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:57:41 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:57:42.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:57:41 smithi115 conmon[90393]: 2022-05-13T23:57:41.748+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:57:42.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:57:41 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:57:42.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:57:41 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:57:42.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:57:41 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-13T23:57:42.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:57:41 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:57:42.109 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-13T23:57:42.109 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-13T23:57:42.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-13T23:57:42.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-13T23:57:42.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-13T23:57:42.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-13T23:57:42.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-13T23:57:42.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-13T23:57:42.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-13T23:57:42.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-13T23:57:42.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-13T23:57:42.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-13T23:57:42.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-13T23:57:42.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-13T23:57:42.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-13T23:57:42.113 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-13T23:57:42.113 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-13T23:57:42.113 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:57:42.113 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:57:41 smithi115 ceph-mon[78593]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-13T23:57:42.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:57:41 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:57:42.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:57:41 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:57:42.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:57:41 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-13T23:57:42.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:57:41 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:57:42.292 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-13T23:57:42.292 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-13T23:57:42.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-13T23:57:42.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-13T23:57:42.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-13T23:57:42.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-13T23:57:42.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-13T23:57:42.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-13T23:57:42.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-13T23:57:42.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-13T23:57:42.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-13T23:57:42.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-13T23:57:42.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-13T23:57:42.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-13T23:57:42.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-13T23:57:42.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-13T23:57:42.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-13T23:57:42.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:57:42.297 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:57:41 smithi047 ceph-mon[77147]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-13T23:57:43.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:57:42 smithi115 conmon[90393]: 2022-05-13T23:57:42.774+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:57:43.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:57:42 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:57:43.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:57:42 smithi115 ceph-mon[78593]: pgmap v1250: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:57:43.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:57:42 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:57:43.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:57:42 smithi047 ceph-mon[77147]: pgmap v1250: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:57:44.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:57:43 smithi115 conmon[90393]: 2022-05-13T23:57:43.743+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:57:44.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:57:43 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:57:44.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:57:43 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 1795 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:57:44.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:57:43 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:57:44.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:57:43 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 1795 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:57:44.899 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:57:44 smithi047 conmon[78521]: 2022-05-13T23:57:44.624+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:57:44.900 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:57:44 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-13T23:57:44.900 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:57:44 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-13T23:57:44.900 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:57:44 smithi047 conmon[78521]: yield 2022-05-13T23:57:44.900 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:57:44 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-13T23:57:44.901 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:57:44 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-13T23:57:44.901 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:57:44 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-13T23:57:44.901 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:57:44 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-13T23:57:44.901 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:57:44 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-13T23:57:44.902 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:57:44 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-13T23:57:44.902 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:57:44 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-13T23:57:44.902 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:57:44 smithi047 conmon[78521]: raise exceptions[0] 2022-05-13T23:57:44.902 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:57:44 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-13T23:57:44.902 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:57:44 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-13T23:57:44.903 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:57:44 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-13T23:57:44.903 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:57:44 smithi047 conmon[78521]: return (yield from fut) 2022-05-13T23:57:44.904 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:57:44 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-13T23:57:44.904 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:57:44 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-13T23:57:44.904 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:57:44 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:57:44.904 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:57:44 smithi047 conmon[78521]: 2022-05-13T23:57:44.626+0000 7f5340f7a700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-13T23:57:44.905 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:57:44 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-13T23:57:44.905 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:57:44 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-13T23:57:44.905 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:57:44 smithi047 conmon[78521]: yield 2022-05-13T23:57:44.906 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:57:44 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-13T23:57:44.906 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:57:44 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-13T23:57:44.906 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:57:44 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-13T23:57:44.907 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:57:44 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-13T23:57:44.907 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:57:44 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-13T23:57:44.907 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:57:44 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-13T23:57:44.907 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:57:44 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-13T23:57:44.907 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:57:44 smithi047 conmon[78521]: raise exceptions[0] 2022-05-13T23:57:44.908 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:57:44 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-13T23:57:44.908 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:57:44 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-13T23:57:44.908 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:57:44 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-13T23:57:44.909 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:57:44 smithi047 conmon[78521]: return (yield from fut) 2022-05-13T23:57:44.909 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:57:44 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-13T23:57:44.909 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:57:44 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-13T23:57:44.910 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:57:44 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:57:44.910 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:57:44 smithi047 conmon[78521]: 2022-05-13T23:57:44.911 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:57:44 smithi047 conmon[78521]: During handling of the above exception, another exception occurred: 2022-05-13T23:57:44.911 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:57:44 smithi047 conmon[78521]: 2022-05-13T23:57:44.911 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:57:44 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-13T23:57:44.912 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:57:44 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-13T23:57:44.912 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:57:44 smithi047 conmon[78521]: return f(*arg) 2022-05-13T23:57:44.912 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:57:44 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-13T23:57:44.912 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:57:44 smithi047 conmon[78521]: self._write_client_files(client_files, host) 2022-05-13T23:57:44.912 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:57:44 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-13T23:57:44.913 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:57:44 smithi047 conmon[78521]: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-13T23:57:44.913 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:57:44 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-13T23:57:44.913 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:57:44 smithi047 conmon[78521]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-13T23:57:44.913 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:57:44 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-13T23:57:44.914 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:57:44 smithi047 conmon[78521]: return self.event_loop.get_result(coro) 2022-05-13T23:57:44.914 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:57:44 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-13T23:57:44.914 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:57:44 smithi047 conmon[78521]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-13T23:57:44.914 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:57:44 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-13T23:57:44.915 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:57:44 smithi047 conmon[78521]: return self.__get_result() 2022-05-13T23:57:44.915 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:57:44 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-13T23:57:44.915 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:57:44 smithi047 conmon[78521]: raise self._exception 2022-05-13T23:57:44.916 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:57:44 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-13T23:57:44.916 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:57:44 smithi047 conmon[78521]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-13T23:57:44.917 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:57:44 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-13T23:57:44.917 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:57:44 smithi047 conmon[78521]: conn = await self._remote_connection(host, addr) 2022-05-13T23:57:44.917 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:57:44 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-13T23:57:44.918 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:57:44 smithi047 conmon[78521]: raise 2022-05-13T23:57:44.918 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:57:44 smithi047 conmon[78521]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-13T23:57:44.918 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:57:44 smithi047 conmon[78521]: self.gen.throw(type, value, traceback) 2022-05-13T23:57:44.918 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:57:44 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-13T23:57:44.919 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:57:44 smithi047 conmon[78521]: raise OrchestratorError(msg) 2022-05-13T23:57:44.919 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:57:44 smithi047 conmon[78521]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:57:45.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:57:44 smithi115 conmon[90393]: 2022-05-13T23:57:44.737+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:57:45.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:57:44 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:57:45.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:57:44 smithi115 ceph-mon[78593]: pgmap v1251: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:57:45.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:57:44 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:57:45.109 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-13T23:57:45.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-13T23:57:45.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-13T23:57:45.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-13T23:57:45.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-13T23:57:45.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-13T23:57:45.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-13T23:57:45.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-13T23:57:45.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-13T23:57:45.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-13T23:57:45.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-13T23:57:45.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-13T23:57:45.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-13T23:57:45.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-13T23:57:45.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-13T23:57:45.113 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-13T23:57:45.113 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-13T23:57:45.113 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:57:45.113 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:57:44 smithi115 ceph-mon[78593]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-13T23:57:45.114 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-13T23:57:45.114 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-13T23:57:45.114 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-13T23:57:45.114 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-13T23:57:45.114 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-13T23:57:45.115 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-13T23:57:45.115 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-13T23:57:45.115 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-13T23:57:45.115 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-13T23:57:45.115 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-13T23:57:45.116 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-13T23:57:45.116 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-13T23:57:45.116 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-13T23:57:45.116 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-13T23:57:45.116 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-13T23:57:45.117 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-13T23:57:45.117 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-13T23:57:45.117 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:57:45.119 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-13T23:57:45.119 INFO:journalctl@ceph.mon.c.smithi115.stdout: During handling of the above exception, another exception occurred: 2022-05-13T23:57:45.120 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-13T23:57:45.120 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-13T23:57:45.121 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-13T23:57:45.121 INFO:journalctl@ceph.mon.c.smithi115.stdout: return f(*arg) 2022-05-13T23:57:45.122 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-13T23:57:45.122 INFO:journalctl@ceph.mon.c.smithi115.stdout: self._write_client_files(client_files, host) 2022-05-13T23:57:45.123 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-13T23:57:45.123 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-13T23:57:45.123 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-13T23:57:45.124 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-13T23:57:45.124 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-13T23:57:45.125 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.event_loop.get_result(coro) 2022-05-13T23:57:45.125 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-13T23:57:45.126 INFO:journalctl@ceph.mon.c.smithi115.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-13T23:57:45.127 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-13T23:57:45.128 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.__get_result() 2022-05-13T23:57:45.129 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-13T23:57:45.130 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise self._exception 2022-05-13T23:57:45.130 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-13T23:57:45.131 INFO:journalctl@ceph.mon.c.smithi115.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-13T23:57:45.131 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-13T23:57:45.132 INFO:journalctl@ceph.mon.c.smithi115.stdout: conn = await self._remote_connection(host, addr) 2022-05-13T23:57:45.133 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-13T23:57:45.133 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise 2022-05-13T23:57:45.134 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-13T23:57:45.134 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.gen.throw(type, value, traceback) 2022-05-13T23:57:45.134 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-13T23:57:45.135 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OrchestratorError(msg) 2022-05-13T23:57:45.135 INFO:journalctl@ceph.mon.c.smithi115.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:57:45.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:57:44 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:57:45.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:57:44 smithi047 ceph-mon[77147]: pgmap v1251: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:57:45.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:57:44 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:57:45.292 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-13T23:57:45.292 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-13T23:57:45.292 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-13T23:57:45.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-13T23:57:45.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-13T23:57:45.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-13T23:57:45.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-13T23:57:45.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-13T23:57:45.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-13T23:57:45.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-13T23:57:45.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-13T23:57:45.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-13T23:57:45.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-13T23:57:45.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-13T23:57:45.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-13T23:57:45.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-13T23:57:45.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-13T23:57:45.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:57:45.296 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:57:44 smithi047 ceph-mon[77147]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-13T23:57:45.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-13T23:57:45.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-13T23:57:45.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-13T23:57:45.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-13T23:57:45.298 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-13T23:57:45.298 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-13T23:57:45.298 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-13T23:57:45.298 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-13T23:57:45.298 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-13T23:57:45.299 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-13T23:57:45.299 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-13T23:57:45.299 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-13T23:57:45.299 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-13T23:57:45.300 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-13T23:57:45.302 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-13T23:57:45.302 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-13T23:57:45.302 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-13T23:57:45.303 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:57:45.303 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-13T23:57:45.303 INFO:journalctl@ceph.mon.b.smithi047.stdout: During handling of the above exception, another exception occurred: 2022-05-13T23:57:45.303 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-13T23:57:45.304 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-13T23:57:45.304 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-13T23:57:45.304 INFO:journalctl@ceph.mon.b.smithi047.stdout: return f(*arg) 2022-05-13T23:57:45.304 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-13T23:57:45.304 INFO:journalctl@ceph.mon.b.smithi047.stdout: self._write_client_files(client_files, host) 2022-05-13T23:57:45.305 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-13T23:57:45.305 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-13T23:57:45.305 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-13T23:57:45.305 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-13T23:57:45.306 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-13T23:57:45.306 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.event_loop.get_result(coro) 2022-05-13T23:57:45.306 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-13T23:57:45.306 INFO:journalctl@ceph.mon.b.smithi047.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-13T23:57:45.307 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-13T23:57:45.307 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.__get_result() 2022-05-13T23:57:45.307 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-13T23:57:45.307 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise self._exception 2022-05-13T23:57:45.308 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-13T23:57:45.308 INFO:journalctl@ceph.mon.b.smithi047.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-13T23:57:45.308 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-13T23:57:45.308 INFO:journalctl@ceph.mon.b.smithi047.stdout: conn = await self._remote_connection(host, addr) 2022-05-13T23:57:45.308 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-13T23:57:45.309 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise 2022-05-13T23:57:45.309 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-13T23:57:45.309 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.gen.throw(type, value, traceback) 2022-05-13T23:57:45.310 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-13T23:57:45.310 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OrchestratorError(msg) 2022-05-13T23:57:45.310 INFO:journalctl@ceph.mon.b.smithi047.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:57:46.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:57:45 smithi115 conmon[90393]: 2022-05-13T23:57:45.721+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:57:46.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:57:45 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:57:46.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:57:45 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:57:47.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:57:46 smithi115 conmon[90393]: 2022-05-13T23:57:46.733+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:57:47.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:57:46 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:57:47.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:57:46 smithi115 ceph-mon[78593]: pgmap v1252: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:57:47.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:57:46 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:57:47.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:57:46 smithi047 ceph-mon[77147]: pgmap v1252: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:57:48.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:57:47 smithi115 conmon[90393]: 2022-05-13T23:57:47.688+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:57:48.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:57:47 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:57:48.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:57:47 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:57:49.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:57:48 smithi115 conmon[90393]: 2022-05-13T23:57:48.725+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:57:49.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:57:48 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:57:49.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:57:48 smithi115 ceph-mon[78593]: pgmap v1253: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:57:49.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:57:48 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 1800 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:57:49.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:57:48 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:57:49.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:57:48 smithi047 ceph-mon[77147]: pgmap v1253: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:57:49.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:57:48 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 1800 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:57:50.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:57:49 smithi115 conmon[90393]: 2022-05-13T23:57:49.718+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:57:50.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:57:49 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:57:50.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:57:49 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:57:50.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:57:49 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-13T23:57:50.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:57:49 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:57:50.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:57:49 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:57:50.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:57:49 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-13T23:57:51.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:57:50 smithi115 conmon[90393]: 2022-05-13T23:57:50.747+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:57:51.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:57:50 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:57:51.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:57:50 smithi115 ceph-mon[78593]: pgmap v1254: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:57:51.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:57:50 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:57:51.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:57:50 smithi047 ceph-mon[77147]: pgmap v1254: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:57:52.110 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:57:51 smithi115 conmon[90393]: 2022-05-13T23:57:51.763+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:57:52.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:57:51 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:57:52.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:57:51 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:57:53.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:57:52 smithi115 conmon[90393]: 2022-05-13T23:57:52.790+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:57:53.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:57:52 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:57:53.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:57:52 smithi115 ceph-mon[78593]: pgmap v1255: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:57:53.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:57:52 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:57:53.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:57:52 smithi047 ceph-mon[77147]: pgmap v1255: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:57:54.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:57:53 smithi115 conmon[90393]: 2022-05-13T23:57:53.828+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:57:54.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:57:53 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:57:54.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:57:53 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:57:54.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:57:53 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-13T23:57:54.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:57:53 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 1805 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:57:54.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:57:53 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:57:54.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:57:53 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:57:54.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:57:53 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-13T23:57:54.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:57:53 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 1805 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:57:55.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:57:54 smithi047 ceph-mon[77147]: pgmap v1256: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:57:55.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:57:54 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:57:55.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:57:54 smithi115 conmon[90393]: 2022-05-13T23:57:54.875+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:57:55.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:57:54 smithi115 ceph-mon[78593]: pgmap v1256: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:57:55.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:57:54 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:57:56.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:57:55 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:57:56.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:57:55 smithi115 conmon[90393]: 2022-05-13T23:57:55.896+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:57:56.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:57:55 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:57:57.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:57:56 smithi115 conmon[90393]: 2022-05-13T23:57:56.848+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:57:57.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:57:56 smithi115 ceph-mon[78593]: pgmap v1257: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:57:57.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:57:56 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:57:57.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:57:56 smithi047 ceph-mon[77147]: pgmap v1257: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:57:57.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:57:56 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:57:58.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:57:57 smithi115 conmon[90393]: 2022-05-13T23:57:57.814+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:57:58.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:57:57 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:57:58.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:57:57 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:57:58.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:57:57 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-13T23:57:58.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:57:57 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:57:58.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:57:57 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:57:58.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:57:57 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-13T23:57:59.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:57:58 smithi115 conmon[90393]: 2022-05-13T23:57:58.800+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:57:59.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:57:58 smithi115 ceph-mon[78593]: pgmap v1258: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:57:59.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:57:58 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:57:59.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:57:58 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 1810 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:57:59.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:57:58 smithi047 ceph-mon[77147]: pgmap v1258: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:57:59.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:57:58 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:57:59.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:57:58 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 1810 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:58:00.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:57:59 smithi115 conmon[90393]: 2022-05-13T23:57:59.830+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:58:00.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:57:59 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:58:00.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:57:59 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:58:01.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:58:00 smithi115 conmon[90393]: 2022-05-13T23:58:00.837+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:58:01.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:58:00 smithi115 ceph-mon[78593]: pgmap v1259: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:58:01.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:58:00 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:58:01.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:58:00 smithi047 ceph-mon[77147]: pgmap v1259: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:58:01.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:58:00 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:58:02.110 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:58:01 smithi115 conmon[90393]: 2022-05-13T23:58:01.797+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:58:02.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:58:01 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:58:02.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:58:01 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:58:02.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:58:01 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-13T23:58:02.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:58:01 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:58:02.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:58:01 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:58:02.295 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:58:01 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-13T23:58:03.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:58:02 smithi115 conmon[90393]: 2022-05-13T23:58:02.752+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:58:03.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:58:02 smithi115 ceph-mon[78593]: pgmap v1260: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:58:03.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:58:02 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:58:03.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:58:02 smithi047 ceph-mon[77147]: pgmap v1260: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:58:03.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:58:02 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:58:03.972 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:58:03 smithi115 conmon[90393]: 2022-05-13T23:58:03.719+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:58:04.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:58:03 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:58:04.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:58:03 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 1815 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:58:04.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:58:03 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:58:04.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:58:03 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 1815 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:58:05.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:58:04 smithi115 conmon[90393]: 2022-05-13T23:58:04.738+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:58:05.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:58:04 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:58:05.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:58:04 smithi115 ceph-mon[78593]: pgmap v1261: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:58:05.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:58:04 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:58:05.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:58:04 smithi047 ceph-mon[77147]: pgmap v1261: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:58:06.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:58:05 smithi115 conmon[90393]: 2022-05-13T23:58:05.758+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:58:06.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:58:05 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:58:06.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:58:05 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:58:07.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:58:06 smithi115 conmon[90393]: 2022-05-13T23:58:06.749+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:58:07.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:58:06 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:58:07.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:58:06 smithi115 ceph-mon[78593]: pgmap v1262: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:58:07.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:58:06 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:58:07.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:58:06 smithi047 ceph-mon[77147]: pgmap v1262: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:58:08.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:58:07 smithi115 conmon[90393]: 2022-05-13T23:58:07.776+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:58:08.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:58:07 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:58:08.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:58:07 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:58:09.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:58:08 smithi115 conmon[90393]: 2022-05-13T23:58:08.788+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:58:09.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:58:08 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:58:09.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:58:08 smithi115 ceph-mon[78593]: pgmap v1263: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:58:09.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:58:08 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 1820 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:58:09.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:58:08 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:58:09.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:58:08 smithi047 ceph-mon[77147]: pgmap v1263: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:58:09.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:58:08 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 1820 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:58:10.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:58:09 smithi115 conmon[90393]: 2022-05-13T23:58:09.807+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:58:10.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:58:09 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:58:10.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:58:09 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:58:10.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:58:09 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-13T23:58:10.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:58:09 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:58:10.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:58:09 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:58:10.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:58:09 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-13T23:58:11.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:58:10 smithi115 conmon[90393]: 2022-05-13T23:58:10.796+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:58:11.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:58:10 smithi115 ceph-mon[78593]: pgmap v1264: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:58:11.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:58:10 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:58:11.305 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:58:10 smithi047 ceph-mon[77147]: pgmap v1264: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:58:11.306 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:58:10 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:58:12.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:58:11 smithi115 conmon[90393]: 2022-05-13T23:58:11.789+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:58:12.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:58:12 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:58:12.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:58:12 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:58:13.013 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:58:12 smithi115 conmon[90393]: 2022-05-13T23:58:12.759+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:58:13.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:58:13 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:58:13.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:58:13 smithi047 ceph-mon[77147]: pgmap v1265: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:58:13.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:58:13 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:58:13.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:58:13 smithi115 ceph-mon[78593]: pgmap v1265: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:58:14.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:58:13 smithi115 conmon[90393]: 2022-05-13T23:58:13.789+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:58:14.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:58:14 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:58:14.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:58:14 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:58:14.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:58:14 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-13T23:58:14.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:58:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-13T23:58:14.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:58:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-13T23:58:14.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:58:14 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 1825 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:58:14.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:58:14 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:58:14.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:58:14 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:58:14.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:58:14 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-13T23:58:14.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:58:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-13T23:58:14.295 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:58:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-13T23:58:14.295 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:58:14 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 1825 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:58:15.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:58:14 smithi115 conmon[90393]: 2022-05-13T23:58:14.769+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:58:15.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:58:15 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:58:15.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:58:15 smithi115 ceph-mon[78593]: pgmap v1266: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:58:15.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:58:15 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:58:15.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:58:15 smithi047 ceph-mon[77147]: pgmap v1266: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:58:16.021 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:58:15 smithi115 conmon[90393]: 2022-05-13T23:58:15.766+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:58:16.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:58:16 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:58:16.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:58:16 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:58:17.020 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:58:16 smithi115 conmon[90393]: 2022-05-13T23:58:16.741+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:58:17.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:58:17 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:58:17.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:58:17 smithi047 ceph-mon[77147]: pgmap v1267: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:58:17.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:58:17 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:58:17.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:58:17 smithi115 ceph-mon[78593]: pgmap v1267: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:58:18.034 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:58:17 smithi115 conmon[90393]: 2022-05-13T23:58:17.729+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:58:18.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:58:18 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:58:18.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:58:18 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:58:18.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:58:18 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-13T23:58:18.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:58:18 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:58:18.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:58:18 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:58:18.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:58:18 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-13T23:58:19.036 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:58:18 smithi115 conmon[90393]: 2022-05-13T23:58:18.687+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:58:19.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:58:19 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:58:19.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:58:19 smithi047 ceph-mon[77147]: pgmap v1268: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:58:19.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:58:19 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 1830 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:58:19.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:58:19 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:58:19.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:58:19 smithi115 ceph-mon[78593]: pgmap v1268: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:58:19.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:58:19 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 1830 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:58:20.038 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:58:19 smithi115 conmon[90393]: 2022-05-13T23:58:19.731+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:58:20.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:58:20 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:58:20.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:58:20 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:58:21.039 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:58:20 smithi115 conmon[90393]: 2022-05-13T23:58:20.723+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:58:21.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:58:21 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:58:21.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:58:21 smithi047 ceph-mon[77147]: pgmap v1269: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:58:21.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:58:21 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:58:21.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:58:21 smithi115 ceph-mon[78593]: pgmap v1269: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:58:22.048 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:58:21 smithi115 conmon[90393]: 2022-05-13T23:58:21.771+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:58:22.362 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:58:22 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:58:22.362 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:58:22 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:58:22.362 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:58:22 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-13T23:58:22.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:58:22 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:58:22.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:58:22 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:58:22.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:58:22 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-13T23:58:23.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:58:22 smithi115 conmon[90393]: 2022-05-13T23:58:22.821+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:58:23.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:58:23 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:58:23.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:58:23 smithi115 ceph-mon[78593]: pgmap v1270: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:58:23.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:58:23 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:58:23.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:58:23 smithi047 ceph-mon[77147]: pgmap v1270: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:58:24.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:58:23 smithi115 conmon[90393]: 2022-05-13T23:58:23.847+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:58:24.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:58:24 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:58:24.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:58:24 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 1835 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:58:24.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:58:24 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:58:24.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:58:24 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 1835 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:58:25.112 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:58:24 smithi115 conmon[90393]: 2022-05-13T23:58:24.821+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:58:25.124 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:58:25 smithi115 ceph-mon[78593]: pgmap v1271: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:58:25.124 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:58:25 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:58:25.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:58:25 smithi047 ceph-mon[77147]: pgmap v1271: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:58:25.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:58:25 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:58:26.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:58:25 smithi115 conmon[90393]: 2022-05-13T23:58:25.827+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:58:26.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:58:26 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:58:26.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:58:26 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:58:27.064 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:58:26 smithi115 conmon[90393]: 2022-05-13T23:58:26.812+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:58:27.065 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:58:27 smithi115 ceph-mon[78593]: pgmap v1272: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:58:27.065 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:58:27 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:58:27.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:58:27 smithi047 ceph-mon[77147]: pgmap v1272: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:58:27.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:58:27 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:58:28.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:58:28 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:58:28.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:58:27 smithi115 conmon[90393]: 2022-05-13T23:58:27.862+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:58:28.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:58:28 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:58:29.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:58:28 smithi115 conmon[90393]: 2022-05-13T23:58:28.902+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:58:29.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:58:29 smithi115 ceph-mon[78593]: pgmap v1273: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:58:29.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:58:29 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:58:29.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:58:29 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 1840 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:58:29.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:58:29 smithi047 ceph-mon[77147]: pgmap v1273: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:58:29.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:58:29 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:58:29.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:58:29 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 1840 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:58:30.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:58:29 smithi115 conmon[90393]: 2022-05-13T23:58:29.918+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:58:30.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:58:30 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:58:30.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:58:30 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:58:30.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:58:30 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-13T23:58:30.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:58:30 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:58:30.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:58:30 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:58:30.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:58:30 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-13T23:58:31.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:58:30 smithi115 conmon[90393]: 2022-05-13T23:58:30.890+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:58:31.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:58:31 smithi115 ceph-mon[78593]: pgmap v1274: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:58:31.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:58:31 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:58:31.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:58:31 smithi047 ceph-mon[77147]: pgmap v1274: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:58:31.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:58:31 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:58:32.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:58:31 smithi115 conmon[90393]: 2022-05-13T23:58:31.895+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:58:32.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:58:32 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:58:32.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:58:32 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:58:33.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:58:32 smithi115 conmon[90393]: 2022-05-13T23:58:32.888+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:58:33.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:58:33 smithi115 ceph-mon[78593]: pgmap v1275: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:58:33.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:58:33 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:58:33.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:58:33 smithi047 ceph-mon[77147]: pgmap v1275: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:58:33.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:58:33 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:58:34.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:58:33 smithi115 conmon[90393]: 2022-05-13T23:58:33.910+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:58:34.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:58:34 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:58:34.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:58:34 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:58:34.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:58:34 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-13T23:58:34.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:58:34 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 1845 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:58:34.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:58:34 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:58:34.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:58:34 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:58:34.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:58:34 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-13T23:58:34.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:58:34 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 1845 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:58:35.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:58:34 smithi115 conmon[90393]: 2022-05-13T23:58:34.884+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:58:35.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:58:35 smithi115 ceph-mon[78593]: pgmap v1276: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:58:35.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:58:35 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:58:35.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:58:35 smithi047 ceph-mon[77147]: pgmap v1276: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:58:35.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:58:35 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:58:36.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:58:35 smithi115 conmon[90393]: 2022-05-13T23:58:35.908+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:58:36.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:58:36 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:58:36.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:58:36 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:58:37.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:58:36 smithi115 conmon[90393]: 2022-05-13T23:58:36.912+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:58:37.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:58:37 smithi115 ceph-mon[78593]: pgmap v1277: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:58:37.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:58:37 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:58:37.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:58:37 smithi047 ceph-mon[77147]: pgmap v1277: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:58:37.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:58:37 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:58:38.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:58:37 smithi115 conmon[90393]: 2022-05-13T23:58:37.920+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:58:38.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:58:38 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:58:38.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:58:38 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:58:38.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:58:38 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-13T23:58:38.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:58:38 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:58:38.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:58:38 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:58:38.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:58:38 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-13T23:58:39.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:58:38 smithi115 conmon[90393]: 2022-05-13T23:58:38.926+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:58:39.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:58:39 smithi115 ceph-mon[78593]: pgmap v1278: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:58:39.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:58:39 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:58:39.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:58:39 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 1850 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:58:39.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:58:39 smithi047 ceph-mon[77147]: pgmap v1278: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:58:39.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:58:39 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:58:39.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:58:39 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 1850 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:58:40.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:58:40 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:58:40.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:58:39 smithi115 conmon[90393]: 2022-05-13T23:58:39.900+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:58:40.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:58:40 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:58:41.119 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:58:40 smithi115 conmon[90393]: 2022-05-13T23:58:40.868+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:58:41.120 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:58:41 smithi115 ceph-mon[78593]: pgmap v1279: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:58:41.120 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:58:41 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:58:41.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:58:41 smithi047 ceph-mon[77147]: pgmap v1279: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:58:41.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:58:41 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:58:42.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:58:41 smithi115 conmon[90393]: 2022-05-13T23:58:41.884+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:58:42.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:58:42 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:58:42.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:58:42 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:58:42.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:58:42 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-13T23:58:42.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:58:42 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:58:42.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:58:42 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:58:42.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:58:42 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-13T23:58:43.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:58:42 smithi115 conmon[90393]: 2022-05-13T23:58:42.897+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:58:43.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:58:43 smithi115 ceph-mon[78593]: pgmap v1280: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:58:43.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:58:43 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:58:43.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:58:43 smithi047 ceph-mon[77147]: pgmap v1280: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:58:43.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:58:43 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:58:44.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:58:43 smithi115 conmon[90393]: 2022-05-13T23:58:43.940+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:58:44.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:58:44 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:58:44.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:58:44 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 1855 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:58:44.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:58:44 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:58:44.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:58:44 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 1855 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:58:45.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:58:44 smithi115 conmon[90393]: 2022-05-13T23:58:44.940+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:58:45.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:58:45 smithi115 ceph-mon[78593]: pgmap v1281: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:58:45.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:58:45 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:58:45.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:58:45 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-13T23:58:45.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:58:45 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-13T23:58:45.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:58:45 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-13T23:58:45.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:58:45 smithi047 ceph-mon[77147]: pgmap v1281: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:58:45.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:58:45 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:58:45.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:58:45 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-13T23:58:45.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:58:45 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-13T23:58:45.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:58:45 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-13T23:58:45.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:58:45 smithi047 conmon[78521]: 2022-05-13T23:58:45.103+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:58:45.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:58:45 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-13T23:58:45.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:58:45 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-13T23:58:45.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:58:45 smithi047 conmon[78521]: yield 2022-05-13T23:58:45.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:58:45 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-13T23:58:45.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:58:45 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-13T23:58:45.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:58:45 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-13T23:58:45.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:58:45 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-13T23:58:45.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:58:45 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-13T23:58:45.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:58:45 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-13T23:58:45.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:58:45 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-13T23:58:45.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:58:45 smithi047 conmon[78521]: raise exceptions[0] 2022-05-13T23:58:45.547 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:58:45 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-13T23:58:45.547 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:58:45 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-13T23:58:45.547 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:58:45 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-13T23:58:45.547 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:58:45 smithi047 conmon[78521]: return (yield from fut) 2022-05-13T23:58:45.547 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:58:45 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-13T23:58:45.548 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:58:45 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-13T23:58:45.549 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:58:45 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:58:46.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:58:45 smithi115 conmon[90393]: 2022-05-13T23:58:45.914+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:58:46.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:58:46 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:58:46.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:58:46 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:58:46.359 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-13T23:58:46.359 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-13T23:58:46.359 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-13T23:58:46.359 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-13T23:58:46.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-13T23:58:46.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-13T23:58:46.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-13T23:58:46.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-13T23:58:46.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-13T23:58:46.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-13T23:58:46.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-13T23:58:46.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-13T23:58:46.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-13T23:58:46.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-13T23:58:46.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-13T23:58:46.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-13T23:58:46.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-13T23:58:46.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:58:46.364 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:58:46 smithi115 ceph-mon[78593]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-13T23:58:46.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:58:46 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:58:46.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:58:46 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:58:46.541 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-13T23:58:46.542 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-13T23:58:46.542 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-13T23:58:46.542 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-13T23:58:46.542 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-13T23:58:46.542 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-13T23:58:46.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-13T23:58:46.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-13T23:58:46.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-13T23:58:46.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-13T23:58:46.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-13T23:58:46.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-13T23:58:46.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-13T23:58:46.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-13T23:58:46.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-13T23:58:46.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-13T23:58:46.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-13T23:58:46.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:58:46.545 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:58:46 smithi047 ceph-mon[77147]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-13T23:58:47.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:58:46 smithi115 conmon[90393]: 2022-05-13T23:58:46.936+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:58:47.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:58:47 smithi115 ceph-mon[78593]: pgmap v1282: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:58:47.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:58:47 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:58:47.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:58:47 smithi047 ceph-mon[77147]: pgmap v1282: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:58:47.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:58:47 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:58:48.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:58:47 smithi115 conmon[90393]: 2022-05-13T23:58:47.912+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:58:48.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:58:48 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:58:48.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:58:48 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:58:48.541 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:58:48 smithi047 conmon[78521]: 2022-05-13T23:58:48.175+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:58:48.542 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:58:48 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-13T23:58:48.542 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:58:48 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-13T23:58:48.542 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:58:48 smithi047 conmon[78521]: yield 2022-05-13T23:58:48.542 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:58:48 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-13T23:58:48.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:58:48 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-13T23:58:48.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:58:48 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-13T23:58:48.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:58:48 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-13T23:58:48.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:58:48 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-13T23:58:48.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:58:48 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-13T23:58:48.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:58:48 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-13T23:58:48.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:58:48 smithi047 conmon[78521]: raise exceptions[0] 2022-05-13T23:58:48.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:58:48 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-13T23:58:48.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:58:48 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-13T23:58:48.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:58:48 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-13T23:58:48.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:58:48 smithi047 conmon[78521]: return (yield from fut) 2022-05-13T23:58:48.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:58:48 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-13T23:58:48.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:58:48 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-13T23:58:48.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:58:48 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:58:48.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:58:48 smithi047 conmon[78521]: 2022-05-13T23:58:48.176+0000 7f5340779700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-13T23:58:48.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:58:48 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-13T23:58:48.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:58:48 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-13T23:58:48.547 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:58:48 smithi047 conmon[78521]: yield 2022-05-13T23:58:48.547 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:58:48 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-13T23:58:48.547 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:58:48 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-13T23:58:48.547 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:58:48 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-13T23:58:48.548 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:58:48 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-13T23:58:48.548 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:58:48 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-13T23:58:48.548 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:58:48 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-13T23:58:48.548 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:58:48 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-13T23:58:48.548 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:58:48 smithi047 conmon[78521]: raise exceptions[0] 2022-05-13T23:58:48.549 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:58:48 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-13T23:58:48.549 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:58:48 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-13T23:58:48.549 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:58:48 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-13T23:58:48.549 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:58:48 smithi047 conmon[78521]: return (yield from fut) 2022-05-13T23:58:48.550 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:58:48 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-13T23:58:48.550 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:58:48 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-13T23:58:48.550 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:58:48 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:58:48.550 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:58:48 smithi047 conmon[78521]: 2022-05-13T23:58:48.550 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:58:48 smithi047 conmon[78521]: During handling of the above exception, another exception occurred: 2022-05-13T23:58:48.551 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:58:48 smithi047 conmon[78521]: 2022-05-13T23:58:48.551 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:58:48 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-13T23:58:48.551 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:58:48 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-13T23:58:48.551 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:58:48 smithi047 conmon[78521]: return f(*arg) 2022-05-13T23:58:48.552 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:58:48 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-13T23:58:48.552 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:58:48 smithi047 conmon[78521]: self._write_client_files(client_files, host) 2022-05-13T23:58:48.552 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:58:48 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-13T23:58:48.552 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:58:48 smithi047 conmon[78521]: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-13T23:58:48.552 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:58:48 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-13T23:58:48.553 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:58:48 smithi047 conmon[78521]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-13T23:58:48.553 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:58:48 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-13T23:58:48.553 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:58:48 smithi047 conmon[78521]: return self.event_loop.get_result(coro) 2022-05-13T23:58:48.553 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:58:48 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-13T23:58:48.554 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:58:48 smithi047 conmon[78521]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-13T23:58:48.554 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:58:48 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-13T23:58:48.554 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:58:48 smithi047 conmon[78521]: return self.__get_result() 2022-05-13T23:58:48.554 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:58:48 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-13T23:58:48.554 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:58:48 smithi047 conmon[78521]: raise self._exception 2022-05-13T23:58:48.555 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:58:48 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-13T23:58:48.555 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:58:48 smithi047 conmon[78521]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-13T23:58:48.555 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:58:48 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-13T23:58:48.555 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:58:48 smithi047 conmon[78521]: conn = await self._remote_connection(host, addr) 2022-05-13T23:58:48.555 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:58:48 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-13T23:58:48.557 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:58:48 smithi047 conmon[78521]: raise 2022-05-13T23:58:48.557 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:58:48 smithi047 conmon[78521]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-13T23:58:48.558 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:58:48 smithi047 conmon[78521]: self.gen.throw(type, value, traceback) 2022-05-13T23:58:48.558 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:58:48 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-13T23:58:48.558 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:58:48 smithi047 conmon[78521]: raise OrchestratorError(msg) 2022-05-13T23:58:48.559 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:58:48 smithi047 conmon[78521]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:58:49.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:58:48 smithi115 conmon[90393]: 2022-05-13T23:58:48.899+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:58:49.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:58:49 smithi115 ceph-mon[78593]: pgmap v1283: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:58:49.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:58:49 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:58:49.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:58:49 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:58:49.359 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-13T23:58:49.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-13T23:58:49.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-13T23:58:49.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-13T23:58:49.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-13T23:58:49.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-13T23:58:49.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-13T23:58:49.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-13T23:58:49.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-13T23:58:49.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-13T23:58:49.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-13T23:58:49.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-13T23:58:49.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-13T23:58:49.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-13T23:58:49.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-13T23:58:49.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-13T23:58:49.365 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-13T23:58:49.365 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:58:49.365 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:58:49 smithi115 ceph-mon[78593]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-13T23:58:49.365 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-13T23:58:49.366 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-13T23:58:49.366 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-13T23:58:49.366 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-13T23:58:49.366 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-13T23:58:49.366 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-13T23:58:49.367 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-13T23:58:49.367 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-13T23:58:49.367 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-13T23:58:49.367 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-13T23:58:49.367 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-13T23:58:49.368 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-13T23:58:49.368 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-13T23:58:49.368 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-13T23:58:49.369 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-13T23:58:49.369 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-13T23:58:49.369 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-13T23:58:49.369 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:58:49.370 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-13T23:58:49.370 INFO:journalctl@ceph.mon.c.smithi115.stdout: During handling of the above exception, another exception occurred: 2022-05-13T23:58:49.370 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-13T23:58:49.370 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-13T23:58:49.371 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-13T23:58:49.371 INFO:journalctl@ceph.mon.c.smithi115.stdout: return f(*arg) 2022-05-13T23:58:49.371 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-13T23:58:49.371 INFO:journalctl@ceph.mon.c.smithi115.stdout: self._write_client_files(client_files, host) 2022-05-13T23:58:49.371 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-13T23:58:49.372 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-13T23:58:49.372 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-13T23:58:49.372 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-13T23:58:49.372 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-13T23:58:49.372 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.event_loop.get_result(coro) 2022-05-13T23:58:49.373 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-13T23:58:49.373 INFO:journalctl@ceph.mon.c.smithi115.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-13T23:58:49.373 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-13T23:58:49.373 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.__get_result() 2022-05-13T23:58:49.373 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-13T23:58:49.374 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise self._exception 2022-05-13T23:58:49.374 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-13T23:58:49.374 INFO:journalctl@ceph.mon.c.smithi115.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-13T23:58:49.374 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-13T23:58:49.374 INFO:journalctl@ceph.mon.c.smithi115.stdout: conn = await self._remote_connection(host, addr) 2022-05-13T23:58:49.375 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-13T23:58:49.375 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise 2022-05-13T23:58:49.375 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-13T23:58:49.375 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.gen.throw(type, value, traceback) 2022-05-13T23:58:49.376 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-13T23:58:49.376 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OrchestratorError(msg) 2022-05-13T23:58:49.376 INFO:journalctl@ceph.mon.c.smithi115.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:58:49.376 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:58:49 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 1860 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:58:49.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:58:49 smithi047 ceph-mon[77147]: pgmap v1283: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:58:49.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:58:49 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:58:49.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:58:49 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:58:49.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-13T23:58:49.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-13T23:58:49.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-13T23:58:49.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-13T23:58:49.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-13T23:58:49.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-13T23:58:49.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-13T23:58:49.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-13T23:58:49.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-13T23:58:49.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-13T23:58:49.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-13T23:58:49.548 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-13T23:58:49.548 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-13T23:58:49.548 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-13T23:58:49.549 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-13T23:58:49.549 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-13T23:58:49.550 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-13T23:58:49.550 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:58:49.551 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:58:49 smithi047 ceph-mon[77147]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-13T23:58:49.551 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-13T23:58:49.552 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-13T23:58:49.552 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-13T23:58:49.553 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-13T23:58:49.553 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-13T23:58:49.553 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-13T23:58:49.554 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-13T23:58:49.554 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-13T23:58:49.555 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-13T23:58:49.555 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-13T23:58:49.556 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-13T23:58:49.556 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-13T23:58:49.557 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-13T23:58:49.557 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-13T23:58:49.557 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-13T23:58:49.558 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-13T23:58:49.558 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-13T23:58:49.559 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:58:49.559 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-13T23:58:49.559 INFO:journalctl@ceph.mon.b.smithi047.stdout: During handling of the above exception, another exception occurred: 2022-05-13T23:58:49.560 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-13T23:58:49.561 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-13T23:58:49.561 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-13T23:58:49.561 INFO:journalctl@ceph.mon.b.smithi047.stdout: return f(*arg) 2022-05-13T23:58:49.562 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-13T23:58:49.562 INFO:journalctl@ceph.mon.b.smithi047.stdout: self._write_client_files(client_files, host) 2022-05-13T23:58:49.562 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-13T23:58:49.562 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-13T23:58:49.562 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-13T23:58:49.563 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-13T23:58:49.563 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-13T23:58:49.563 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.event_loop.get_result(coro) 2022-05-13T23:58:49.563 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-13T23:58:49.564 INFO:journalctl@ceph.mon.b.smithi047.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-13T23:58:49.564 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-13T23:58:49.565 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.__get_result() 2022-05-13T23:58:49.565 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-13T23:58:49.565 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise self._exception 2022-05-13T23:58:49.566 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-13T23:58:49.566 INFO:journalctl@ceph.mon.b.smithi047.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-13T23:58:49.566 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-13T23:58:49.566 INFO:journalctl@ceph.mon.b.smithi047.stdout: conn = await self._remote_connection(host, addr) 2022-05-13T23:58:49.567 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-13T23:58:49.567 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise 2022-05-13T23:58:49.567 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-13T23:58:49.567 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.gen.throw(type, value, traceback) 2022-05-13T23:58:49.567 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-13T23:58:49.568 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OrchestratorError(msg) 2022-05-13T23:58:49.568 INFO:journalctl@ceph.mon.b.smithi047.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:58:49.568 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:58:49 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 1860 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:58:50.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:58:49 smithi115 conmon[90393]: 2022-05-13T23:58:49.940+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:58:50.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:58:50 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:58:50.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:58:50 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:58:50.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:58:50 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-13T23:58:50.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:58:50 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:58:50.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:58:50 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:58:50.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:58:50 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-13T23:58:51.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:58:50 smithi115 conmon[90393]: 2022-05-13T23:58:50.918+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:58:51.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:58:51 smithi115 ceph-mon[78593]: pgmap v1284: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:58:51.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:58:51 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:58:51.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:58:51 smithi047 ceph-mon[77147]: pgmap v1284: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:58:51.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:58:51 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:58:52.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:58:51 smithi115 conmon[90393]: 2022-05-13T23:58:51.929+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:58:52.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:58:52 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:58:52.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:58:52 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:58:53.170 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:58:52 smithi115 conmon[90393]: 2022-05-13T23:58:52.898+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:58:53.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:58:53 smithi047 ceph-mon[77147]: pgmap v1285: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:58:53.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:58:53 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:58:53.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:58:53 smithi115 ceph-mon[78593]: pgmap v1285: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:58:53.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:58:53 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:58:54.172 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:58:53 smithi115 conmon[90393]: 2022-05-13T23:58:53.865+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:58:54.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:58:54 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:58:54.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:58:54 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:58:54.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:58:54 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-13T23:58:54.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:58:54 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 1865 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:58:54.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:58:54 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:58:54.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:58:54 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:58:54.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:58:54 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-13T23:58:54.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:58:54 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 1865 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:58:55.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:58:54 smithi115 conmon[90393]: 2022-05-13T23:58:54.850+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:58:55.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:58:55 smithi047 ceph-mon[77147]: pgmap v1286: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:58:55.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:58:55 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:58:55.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:58:55 smithi115 ceph-mon[78593]: pgmap v1286: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:58:55.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:58:55 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:58:56.179 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:58:55 smithi115 conmon[90393]: 2022-05-13T23:58:55.880+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:58:56.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:58:56 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:58:56.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:58:56 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:58:57.184 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:58:56 smithi115 conmon[90393]: 2022-05-13T23:58:56.898+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:58:57.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:58:57 smithi047 ceph-mon[77147]: pgmap v1287: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:58:57.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:58:57 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:58:57.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:58:57 smithi115 ceph-mon[78593]: pgmap v1287: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:58:57.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:58:57 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:58:58.189 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:58:57 smithi115 conmon[90393]: 2022-05-13T23:58:57.933+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:58:58.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:58:58 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:58:58.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:58:58 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:58:58.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:58:58 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-13T23:58:58.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:58:58 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:58:58.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:58:58 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:58:58.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:58:58 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-13T23:58:59.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:58:58 smithi115 conmon[90393]: 2022-05-13T23:58:58.959+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:58:59.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:58:59 smithi115 ceph-mon[78593]: pgmap v1288: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:58:59.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:58:59 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:58:59.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:58:59 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 1870 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:58:59.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:58:59 smithi047 ceph-mon[77147]: pgmap v1288: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:58:59.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:58:59 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:58:59.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:58:59 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 1870 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:59:00.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:58:59 smithi115 conmon[90393]: 2022-05-13T23:58:59.998+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:59:00.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:59:00 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:59:00.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:59:00 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:59:01.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:59:01 smithi115 conmon[90393]: 2022-05-13T23:59:01.014+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:59:01.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:59:01 smithi115 ceph-mon[78593]: pgmap v1289: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:59:01.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:59:01 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:59:01.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:59:01 smithi047 ceph-mon[77147]: pgmap v1289: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:59:01.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:59:01 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:59:02.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:59:02 smithi115 conmon[90393]: 2022-05-13T23:59:02.014+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:59:02.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:59:02 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:59:02.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:59:02 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:59:02.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:59:02 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-13T23:59:02.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:59:02 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:59:02.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:59:02 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:59:02.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:59:02 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-13T23:59:03.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:59:03 smithi115 conmon[90393]: 2022-05-13T23:59:03.037+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:59:03.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:59:03 smithi115 ceph-mon[78593]: pgmap v1290: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:59:03.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:59:03 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:59:03.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:59:03 smithi047 ceph-mon[77147]: pgmap v1290: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:59:03.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:59:03 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:59:04.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:59:03 smithi115 conmon[90393]: 2022-05-13T23:59:03.997+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:59:04.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:59:04 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:59:04.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:59:04 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 1875 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:59:04.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:59:04 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:59:04.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:59:04 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 1875 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:59:05.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:59:05 smithi115 conmon[90393]: 2022-05-13T23:59:05.042+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:59:05.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:59:05 smithi115 ceph-mon[78593]: pgmap v1291: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:59:05.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:59:05 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:59:05.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:59:05 smithi047 ceph-mon[77147]: pgmap v1291: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:59:05.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:59:05 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:59:06.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:59:06 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:59:06.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:59:06 smithi115 conmon[90393]: 2022-05-13T23:59:06.005+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:59:06.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:59:06 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:59:07.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:59:07 smithi115 conmon[90393]: 2022-05-13T23:59:07.050+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:59:07.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:59:07 smithi115 ceph-mon[78593]: pgmap v1292: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:59:07.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:59:07 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:59:07.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:59:07 smithi047 ceph-mon[77147]: pgmap v1292: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:59:07.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:59:07 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:59:08.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:59:08 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:59:08.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:59:08 smithi115 conmon[90393]: 2022-05-13T23:59:08.086+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:59:08.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:59:08 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:59:09.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:59:09 smithi115 conmon[90393]: 2022-05-13T23:59:09.038+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:59:09.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:59:09 smithi115 ceph-mon[78593]: pgmap v1293: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:59:09.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:59:09 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:59:09.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:59:09 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 1880 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:59:09.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:59:09 smithi047 ceph-mon[77147]: pgmap v1293: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:59:09.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:59:09 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:59:09.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:59:09 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 1880 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:59:10.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:59:10 smithi115 conmon[90393]: 2022-05-13T23:59:10.061+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:59:10.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:59:10 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:59:10.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:59:10 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:59:10.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:59:10 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-13T23:59:10.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:59:10 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:59:10.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:59:10 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:59:10.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:59:10 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-13T23:59:11.360 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:59:11 smithi115 conmon[90393]: 2022-05-13T23:59:11.035+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:59:11.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:59:11 smithi115 ceph-mon[78593]: pgmap v1294: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:59:11.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:59:11 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:59:11.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:59:11 smithi047 ceph-mon[77147]: pgmap v1294: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:59:11.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:59:11 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:59:12.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:59:12 smithi115 conmon[90393]: 2022-05-13T23:59:12.063+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:59:12.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:59:12 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:59:12.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:59:12 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:59:13.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:59:13 smithi115 conmon[90393]: 2022-05-13T23:59:13.020+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:59:13.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:59:13 smithi115 ceph-mon[78593]: pgmap v1295: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:59:13.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:59:13 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:59:13.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:59:13 smithi047 ceph-mon[77147]: pgmap v1295: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:59:13.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:59:13 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:59:14.364 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:59:14 smithi115 conmon[90393]: 2022-05-13T23:59:14.045+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:59:14.366 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:59:14 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:59:14.366 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:59:14 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:59:14.366 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:59:14 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-13T23:59:14.366 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:59:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-13T23:59:14.367 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:59:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-13T23:59:14.367 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:59:14 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 1885 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:59:14.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:59:14 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:59:14.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:59:14 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:59:14.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:59:14 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-13T23:59:14.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:59:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-13T23:59:14.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:59:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-13T23:59:14.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:59:14 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 1885 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:59:15.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:59:15 smithi115 conmon[90393]: 2022-05-13T23:59:15.020+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:59:15.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:59:15 smithi115 ceph-mon[78593]: pgmap v1296: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:59:15.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:59:15 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:59:15.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:59:15 smithi047 ceph-mon[77147]: pgmap v1296: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:59:15.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:59:15 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:59:16.252 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:59:16 smithi115 conmon[90393]: 2022-05-13T23:59:15.999+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:59:16.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:59:16 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:59:16.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:59:16 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:59:17.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:59:17 smithi115 conmon[90393]: 2022-05-13T23:59:17.006+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:59:17.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:59:17 smithi115 ceph-mon[78593]: pgmap v1297: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:59:17.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:59:17 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:59:17.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:59:17 smithi047 ceph-mon[77147]: pgmap v1297: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:59:17.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:59:17 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:59:18.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:59:18 smithi115 conmon[90393]: 2022-05-13T23:59:18.050+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:59:18.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:59:18 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:59:18.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:59:18 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:59:18.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:59:18 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-13T23:59:18.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:59:18 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:59:18.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:59:18 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:59:18.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:59:18 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-13T23:59:19.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:59:19 smithi115 conmon[90393]: 2022-05-13T23:59:19.084+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:59:19.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:59:19 smithi115 ceph-mon[78593]: pgmap v1298: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:59:19.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:59:19 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:59:19.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:59:19 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 1890 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:59:19.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:59:19 smithi047 ceph-mon[77147]: pgmap v1298: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:59:19.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:59:19 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:59:19.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:59:19 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 1890 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:59:20.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:59:20 smithi115 conmon[90393]: 2022-05-13T23:59:20.095+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:59:20.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:59:20 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:59:20.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:59:20 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:59:21.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:59:21 smithi047 ceph-mon[77147]: pgmap v1299: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:59:21.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:59:21 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:59:21.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:59:21 smithi115 conmon[90393]: 2022-05-13T23:59:21.114+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:59:21.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:59:21 smithi115 ceph-mon[78593]: pgmap v1299: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:59:21.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:59:21 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:59:22.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:59:22 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:59:22.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:59:22 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:59:22.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:59:22 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-13T23:59:22.610 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:59:22 smithi115 conmon[90393]: 2022-05-13T23:59:22.143+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:59:22.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:59:22 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:59:22.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:59:22 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:59:22.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:59:22 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-13T23:59:23.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:59:23 smithi047 ceph-mon[77147]: pgmap v1300: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:59:23.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:59:23 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:59:23.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:59:23 smithi115 conmon[90393]: 2022-05-13T23:59:23.159+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:59:23.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:59:23 smithi115 ceph-mon[78593]: pgmap v1300: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:59:23.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:59:23 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:59:24.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:59:24 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:59:24.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:59:24 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 1895 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:59:24.613 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:59:24 smithi115 conmon[90393]: 2022-05-13T23:59:24.147+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:59:24.614 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:59:24 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:59:24.614 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:59:24 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 1895 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:59:25.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:59:25 smithi047 ceph-mon[77147]: pgmap v1301: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:59:25.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:59:25 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:59:25.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:59:25 smithi115 conmon[90393]: 2022-05-13T23:59:25.162+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:59:25.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:59:25 smithi115 ceph-mon[78593]: pgmap v1301: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:59:25.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:59:25 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:59:26.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:59:26 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:59:26.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:59:26 smithi115 conmon[90393]: 2022-05-13T23:59:26.197+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:59:26.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:59:26 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:59:27.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:59:27 smithi047 ceph-mon[77147]: pgmap v1302: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:59:27.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:59:27 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:59:27.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:59:27 smithi115 conmon[90393]: 2022-05-13T23:59:27.156+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:59:27.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:59:27 smithi115 ceph-mon[78593]: pgmap v1302: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:59:27.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:59:27 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:59:28.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:59:28 smithi115 conmon[90393]: 2022-05-13T23:59:28.182+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:59:28.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:59:28 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:59:28.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:59:28 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:59:29.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:59:29 smithi115 conmon[90393]: 2022-05-13T23:59:29.197+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:59:29.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:59:29 smithi115 ceph-mon[78593]: pgmap v1303: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:59:29.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:59:29 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:59:29.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:59:29 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 1900 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:59:29.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:59:29 smithi047 ceph-mon[77147]: pgmap v1303: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:59:29.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:59:29 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:59:29.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:59:29 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 1900 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:59:30.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:59:30 smithi115 conmon[90393]: 2022-05-13T23:59:30.238+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:59:30.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:59:30 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:59:30.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:59:30 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:59:30.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:59:30 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-13T23:59:30.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:59:30 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:59:30.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:59:30 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:59:30.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:59:30 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-13T23:59:31.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:59:31 smithi115 conmon[90393]: 2022-05-13T23:59:31.272+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:59:31.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:59:31 smithi115 ceph-mon[78593]: pgmap v1304: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:59:31.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:59:31 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:59:31.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:59:31 smithi047 ceph-mon[77147]: pgmap v1304: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:59:31.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:59:31 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:59:32.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:59:32 smithi115 conmon[90393]: 2022-05-13T23:59:32.272+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:59:32.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:59:32 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:59:32.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:59:32 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:59:33.610 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:59:33 smithi115 conmon[90393]: 2022-05-13T23:59:33.232+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:59:33.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:59:33 smithi115 ceph-mon[78593]: pgmap v1305: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:59:33.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:59:33 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:59:33.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:59:33 smithi047 ceph-mon[77147]: pgmap v1305: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:59:33.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:59:33 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:59:34.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:59:34 smithi115 conmon[90393]: 2022-05-13T23:59:34.271+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:59:34.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:59:34 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:59:34.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:59:34 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:59:34.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:59:34 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-13T23:59:34.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:59:34 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 1905 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:59:34.795 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:59:34 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:59:34.796 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:59:34 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:59:34.796 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:59:34 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-13T23:59:34.796 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:59:34 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 1905 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:59:35.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:59:35 smithi115 conmon[90393]: 2022-05-13T23:59:35.293+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:59:35.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:59:35 smithi115 ceph-mon[78593]: pgmap v1306: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:59:35.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:59:35 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:59:35.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:59:35 smithi047 ceph-mon[77147]: pgmap v1306: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:59:35.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:59:35 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:59:36.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:59:36 smithi115 conmon[90393]: 2022-05-13T23:59:36.337+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:59:36.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:59:36 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:59:36.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:59:36 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:59:37.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:59:37 smithi115 conmon[90393]: 2022-05-13T23:59:37.321+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:59:37.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:59:37 smithi115 ceph-mon[78593]: pgmap v1307: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:59:37.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:59:37 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:59:37.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:59:37 smithi047 ceph-mon[77147]: pgmap v1307: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:59:37.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:59:37 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:59:38.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:59:38 smithi115 conmon[90393]: 2022-05-13T23:59:38.292+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:59:38.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:59:38 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:59:38.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:59:38 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:59:38.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:59:38 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-13T23:59:38.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:59:38 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:59:38.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:59:38 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:59:38.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:59:38 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-13T23:59:39.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:59:39 smithi115 conmon[90393]: 2022-05-13T23:59:39.269+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:59:39.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:59:39 smithi115 ceph-mon[78593]: pgmap v1308: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:59:39.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:59:39 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:59:39.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:59:39 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 1910 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:59:39.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:59:39 smithi047 ceph-mon[77147]: pgmap v1308: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:59:39.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:59:39 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:59:39.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:59:39 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 1910 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:59:40.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:59:40 smithi115 conmon[90393]: 2022-05-13T23:59:40.262+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:59:40.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:59:40 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:59:40.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:59:40 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:59:41.614 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:59:41 smithi115 conmon[90393]: 2022-05-13T23:59:41.254+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:59:41.615 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:59:41 smithi115 ceph-mon[78593]: pgmap v1309: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:59:41.615 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:59:41 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:59:41.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:59:41 smithi047 ceph-mon[77147]: pgmap v1309: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:59:41.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:59:41 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:59:42.618 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:59:42 smithi115 conmon[90393]: 2022-05-13T23:59:42.267+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:59:42.619 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:59:42 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:59:42.619 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:59:42 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:59:42.620 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:59:42 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-13T23:59:42.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:59:42 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:59:42.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:59:42 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:59:42.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:59:42 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-13T23:59:43.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:59:43 smithi115 conmon[90393]: 2022-05-13T23:59:43.229+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:59:43.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:59:43 smithi115 ceph-mon[78593]: pgmap v1310: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:59:43.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:59:43 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:59:43.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:59:43 smithi047 ceph-mon[77147]: pgmap v1310: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:59:43.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:59:43 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:59:44.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:59:44 smithi115 conmon[90393]: 2022-05-13T23:59:44.232+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:59:44.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:59:44 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:59:44.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:59:44 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 1915 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:59:44.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:59:44 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:59:44.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:59:44 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 1915 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:59:45.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:59:45 smithi115 conmon[90393]: 2022-05-13T23:59:45.240+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:59:45.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:59:45 smithi115 ceph-mon[78593]: pgmap v1311: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:59:45.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:59:45 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:59:45.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:59:45 smithi047 ceph-mon[77147]: pgmap v1311: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:59:45.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:59:45 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:59:46.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:59:46 smithi115 conmon[90393]: 2022-05-13T23:59:46.281+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:59:46.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:59:46 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:59:46.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:59:46 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:59:47.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:59:47 smithi115 conmon[90393]: 2022-05-13T23:59:47.246+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:59:47.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:59:47 smithi115 ceph-mon[78593]: pgmap v1312: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:59:47.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:59:47 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:59:47.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:59:47 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:59:47.798 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:59:47 smithi047 ceph-mon[77147]: pgmap v1312: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:59:47.798 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:59:47 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:59:47.798 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:59:47 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:59:48.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:59:48 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-13T23:59:48.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:59:48 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-13T23:59:48.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:59:48 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-13T23:59:48.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:59:48 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:59:48.610 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:59:48 smithi115 conmon[90393]: 2022-05-13T23:59:48.262+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:59:48.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:59:48 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-13T23:59:48.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:59:48 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-13T23:59:48.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:59:48 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-13T23:59:48.612 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:59:48 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:59:49.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:59:49 smithi115 conmon[90393]: 2022-05-13T23:59:49.278+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:59:49.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:59:49 smithi115 ceph-mon[78593]: pgmap v1313: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:59:49.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:59:49 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 1920 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:59:49.617 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:59:49 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:59:49.795 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:59:49 smithi047 ceph-mon[77147]: pgmap v1313: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:59:49.795 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:59:49 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 1920 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:59:49.795 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:59:49 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:59:50.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:59:50 smithi115 conmon[90393]: 2022-05-13T23:59:50.275+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:59:50.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:59:50 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:59:50.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:59:50 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-13T23:59:50.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:59:50 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:59:50.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:59:50 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:59:50.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:59:50 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-13T23:59:50.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:59:50 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:59:51.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:59:51 smithi047 ceph-mon[77147]: pgmap v1314: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:59:51.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:59:51 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:59:51.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:59:51 smithi047 conmon[78521]: 2022-05-13T23:59:51.280+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:59:51.548 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:59:51 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-13T23:59:51.548 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:59:51 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-13T23:59:51.549 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:59:51 smithi047 conmon[78521]: yield 2022-05-13T23:59:51.549 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:59:51 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-13T23:59:51.549 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:59:51 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-13T23:59:51.554 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:59:51 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-13T23:59:51.555 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:59:51 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-13T23:59:51.555 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:59:51 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-13T23:59:51.555 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:59:51 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-13T23:59:51.555 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:59:51 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-13T23:59:51.556 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:59:51 smithi047 conmon[78521]: raise exceptions[0] 2022-05-13T23:59:51.556 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:59:51 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-13T23:59:51.556 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:59:51 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-13T23:59:51.556 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:59:51 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-13T23:59:51.557 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:59:51 smithi047 conmon[78521]: return (yield from fut) 2022-05-13T23:59:51.557 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:59:51 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-13T23:59:51.557 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:59:51 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-13T23:59:51.557 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:59:51 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:59:51.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:59:51 smithi115 conmon[90393]: 2022-05-13T23:59:51.277+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:59:51.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:59:51 smithi115 ceph-mon[78593]: pgmap v1314: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:59:51.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:59:51 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:59:52.610 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:59:52 smithi115 conmon[90393]: 2022-05-13T23:59:52.242+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:59:52.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:59:52 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:59:52.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-13T23:59:52.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-13T23:59:52.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-13T23:59:52.613 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-13T23:59:52.613 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-13T23:59:52.614 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-13T23:59:52.614 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-13T23:59:52.614 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-13T23:59:52.615 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-13T23:59:52.615 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-13T23:59:52.616 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-13T23:59:52.616 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-13T23:59:52.616 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-13T23:59:52.617 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-13T23:59:52.618 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-13T23:59:52.619 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-13T23:59:52.619 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-13T23:59:52.642 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:59:52.642 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:59:52 smithi115 ceph-mon[78593]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-13T23:59:52.643 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:59:52 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:59:52.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:59:52 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:59:52.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-13T23:59:52.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-13T23:59:52.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-13T23:59:52.797 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-13T23:59:52.798 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-13T23:59:52.798 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-13T23:59:52.799 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-13T23:59:52.800 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-13T23:59:52.801 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-13T23:59:52.801 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-13T23:59:52.802 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-13T23:59:52.815 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-13T23:59:52.815 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-13T23:59:52.816 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-13T23:59:52.816 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-13T23:59:52.816 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-13T23:59:52.817 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-13T23:59:52.818 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:59:52.818 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:59:52 smithi047 ceph-mon[77147]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-13T23:59:52.818 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:59:52 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:59:53.610 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:59:53 smithi115 conmon[90393]: 2022-05-13T23:59:53.229+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:59:53.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:59:53 smithi115 ceph-mon[78593]: pgmap v1315: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:59:53.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:59:53 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:59:53.802 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:59:53 smithi047 ceph-mon[77147]: pgmap v1315: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:59:53.803 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:59:53 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:59:54.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:59:54 smithi115 conmon[90393]: 2022-05-13T23:59:54.214+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:59:54.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:59:54 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:59:54.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:59:54 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-13T23:59:54.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:59:54 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 1925 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:59:54.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:59:54 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:59:54.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:59:54 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:59:54.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:59:54 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-13T23:59:54.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:59:54 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 1925 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:59:54.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:59:54 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:59:54.800 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:59:54 smithi047 conmon[78521]: 2022-05-13T23:59:54.352+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:59:54.800 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:59:54 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-13T23:59:54.800 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:59:54 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-13T23:59:54.801 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:59:54 smithi047 conmon[78521]: yield 2022-05-13T23:59:54.801 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:59:54 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-13T23:59:54.802 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:59:54 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-13T23:59:54.802 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:59:54 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-13T23:59:54.802 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:59:54 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-13T23:59:54.802 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:59:54 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-13T23:59:54.802 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:59:54 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-13T23:59:54.803 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:59:54 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-13T23:59:54.803 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:59:54 smithi047 conmon[78521]: raise exceptions[0] 2022-05-13T23:59:54.803 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:59:54 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-13T23:59:54.803 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:59:54 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-13T23:59:54.804 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:59:54 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-13T23:59:54.804 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:59:54 smithi047 conmon[78521]: return (yield from fut) 2022-05-13T23:59:54.804 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:59:54 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-13T23:59:54.804 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:59:54 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-13T23:59:54.804 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:59:54 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:59:54.805 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:59:54 smithi047 conmon[78521]: 2022-05-13T23:59:54.353+0000 7f533ef76700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-13T23:59:54.806 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:59:54 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-13T23:59:54.807 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:59:54 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-13T23:59:54.807 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:59:54 smithi047 conmon[78521]: yield 2022-05-13T23:59:54.807 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:59:54 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-13T23:59:54.808 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:59:54 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-13T23:59:54.808 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:59:54 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-13T23:59:54.808 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:59:54 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-13T23:59:54.808 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:59:54 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-13T23:59:54.809 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:59:54 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-13T23:59:54.809 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:59:54 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-13T23:59:54.809 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:59:54 smithi047 conmon[78521]: raise exceptions[0] 2022-05-13T23:59:54.809 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:59:54 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-13T23:59:54.809 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:59:54 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-13T23:59:54.810 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:59:54 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-13T23:59:54.810 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:59:54 smithi047 conmon[78521]: return (yield from fut) 2022-05-13T23:59:54.810 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:59:54 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-13T23:59:54.810 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:59:54 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-13T23:59:54.811 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:59:54 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:59:54.811 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:59:54 smithi047 conmon[78521]: 2022-05-13T23:59:54.811 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:59:54 smithi047 conmon[78521]: During handling of the above exception, another exception occurred: 2022-05-13T23:59:54.811 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:59:54 smithi047 conmon[78521]: 2022-05-13T23:59:54.812 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:59:54 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-13T23:59:54.812 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:59:54 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-13T23:59:54.812 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:59:54 smithi047 conmon[78521]: return f(*arg) 2022-05-13T23:59:54.813 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:59:54 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-13T23:59:54.813 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:59:54 smithi047 conmon[78521]: self._write_client_files(client_files, host) 2022-05-13T23:59:54.813 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:59:54 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-13T23:59:54.813 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:59:54 smithi047 conmon[78521]: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-13T23:59:54.814 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:59:54 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-13T23:59:54.814 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:59:54 smithi047 conmon[78521]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-13T23:59:54.814 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:59:54 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-13T23:59:54.814 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:59:54 smithi047 conmon[78521]: return self.event_loop.get_result(coro) 2022-05-13T23:59:54.814 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:59:54 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-13T23:59:54.815 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:59:54 smithi047 conmon[78521]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-13T23:59:54.815 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:59:54 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-13T23:59:54.815 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:59:54 smithi047 conmon[78521]: return self.__get_result() 2022-05-13T23:59:54.816 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:59:54 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-13T23:59:54.817 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:59:54 smithi047 conmon[78521]: raise self._exception 2022-05-13T23:59:54.817 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:59:54 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-13T23:59:54.817 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:59:54 smithi047 conmon[78521]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-13T23:59:54.817 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:59:54 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-13T23:59:54.817 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:59:54 smithi047 conmon[78521]: conn = await self._remote_connection(host, addr) 2022-05-13T23:59:54.818 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:59:54 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-13T23:59:54.818 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:59:54 smithi047 conmon[78521]: raise 2022-05-13T23:59:54.818 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:59:54 smithi047 conmon[78521]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-13T23:59:54.818 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:59:54 smithi047 conmon[78521]: self.gen.throw(type, value, traceback) 2022-05-13T23:59:54.819 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:59:54 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-13T23:59:54.819 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:59:54 smithi047 conmon[78521]: raise OrchestratorError(msg) 2022-05-13T23:59:54.819 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 13 23:59:54 smithi047 conmon[78521]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:59:55.533 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:59:55 smithi115 conmon[90393]: 2022-05-13T23:59:55.231+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:59:55.798 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:59:55 smithi047 ceph-mon[77147]: pgmap v1316: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:59:55.799 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:59:55 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:59:55.799 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-13T23:59:55.799 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-13T23:59:55.800 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-13T23:59:55.800 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-13T23:59:55.801 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-13T23:59:55.802 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-13T23:59:55.802 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-13T23:59:55.803 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-13T23:59:55.804 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-13T23:59:55.804 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-13T23:59:55.804 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-13T23:59:55.805 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-13T23:59:55.805 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-13T23:59:55.806 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-13T23:59:55.806 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-13T23:59:55.808 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-13T23:59:55.808 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-13T23:59:55.809 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:59:55.809 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:59:55 smithi047 ceph-mon[77147]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-13T23:59:55.810 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-13T23:59:55.810 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-13T23:59:55.810 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-13T23:59:55.811 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-13T23:59:55.811 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-13T23:59:55.812 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-13T23:59:55.812 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-13T23:59:55.812 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-13T23:59:55.813 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-13T23:59:55.813 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-13T23:59:55.814 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-13T23:59:55.814 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-13T23:59:55.814 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-13T23:59:55.815 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-13T23:59:55.815 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-13T23:59:55.816 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-13T23:59:55.816 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-13T23:59:55.816 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:59:55.817 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-13T23:59:55.820 INFO:journalctl@ceph.mon.b.smithi047.stdout: During handling of the above exception, another exception occurred: 2022-05-13T23:59:55.821 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-13T23:59:55.821 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-13T23:59:55.821 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-13T23:59:55.822 INFO:journalctl@ceph.mon.b.smithi047.stdout: return f(*arg) 2022-05-13T23:59:55.822 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-13T23:59:55.822 INFO:journalctl@ceph.mon.b.smithi047.stdout: self._write_client_files(client_files, host) 2022-05-13T23:59:55.822 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-13T23:59:55.822 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-13T23:59:55.823 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-13T23:59:55.823 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-13T23:59:55.823 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-13T23:59:55.823 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.event_loop.get_result(coro) 2022-05-13T23:59:55.824 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-13T23:59:55.824 INFO:journalctl@ceph.mon.b.smithi047.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-13T23:59:55.824 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-13T23:59:55.824 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.__get_result() 2022-05-13T23:59:55.824 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-13T23:59:55.825 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise self._exception 2022-05-13T23:59:55.825 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-13T23:59:55.825 INFO:journalctl@ceph.mon.b.smithi047.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-13T23:59:55.825 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-13T23:59:55.826 INFO:journalctl@ceph.mon.b.smithi047.stdout: conn = await self._remote_connection(host, addr) 2022-05-13T23:59:55.826 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-13T23:59:55.826 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise 2022-05-13T23:59:55.826 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-13T23:59:55.826 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.gen.throw(type, value, traceback) 2022-05-13T23:59:55.827 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-13T23:59:55.827 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OrchestratorError(msg) 2022-05-13T23:59:55.827 INFO:journalctl@ceph.mon.b.smithi047.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:59:55.827 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:59:55 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:59:55.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:59:55 smithi115 ceph-mon[78593]: pgmap v1316: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:59:55.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:59:55 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:59:55.860 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-13T23:59:55.860 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-13T23:59:55.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-13T23:59:55.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-13T23:59:55.867 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-13T23:59:55.874 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-13T23:59:55.874 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-13T23:59:55.875 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-13T23:59:55.875 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-13T23:59:55.875 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-13T23:59:55.875 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-13T23:59:55.876 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-13T23:59:55.876 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-13T23:59:55.876 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-13T23:59:55.876 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-13T23:59:55.876 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-13T23:59:55.877 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-13T23:59:55.877 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:59:55.877 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:59:55 smithi115 ceph-mon[78593]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-13T23:59:55.890 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-13T23:59:55.891 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-13T23:59:55.892 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-13T23:59:55.892 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-13T23:59:55.893 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-13T23:59:55.893 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-13T23:59:55.894 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-13T23:59:55.895 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-13T23:59:55.895 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-13T23:59:55.895 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-13T23:59:55.896 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-13T23:59:55.896 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-13T23:59:55.897 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-13T23:59:55.898 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-13T23:59:55.899 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-13T23:59:55.900 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-13T23:59:55.900 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-13T23:59:55.900 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:59:55.901 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-13T23:59:55.901 INFO:journalctl@ceph.mon.c.smithi115.stdout: During handling of the above exception, another exception occurred: 2022-05-13T23:59:55.902 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-13T23:59:55.902 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-13T23:59:55.903 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-13T23:59:55.905 INFO:journalctl@ceph.mon.c.smithi115.stdout: return f(*arg) 2022-05-13T23:59:55.905 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-13T23:59:55.906 INFO:journalctl@ceph.mon.c.smithi115.stdout: self._write_client_files(client_files, host) 2022-05-13T23:59:55.906 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-13T23:59:55.907 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-13T23:59:55.907 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-13T23:59:55.907 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-13T23:59:55.908 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-13T23:59:55.909 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.event_loop.get_result(coro) 2022-05-13T23:59:55.909 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-13T23:59:55.909 INFO:journalctl@ceph.mon.c.smithi115.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-13T23:59:55.910 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-13T23:59:55.910 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.__get_result() 2022-05-13T23:59:55.910 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-13T23:59:55.911 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise self._exception 2022-05-13T23:59:55.911 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-13T23:59:55.912 INFO:journalctl@ceph.mon.c.smithi115.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-13T23:59:55.912 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-13T23:59:55.912 INFO:journalctl@ceph.mon.c.smithi115.stdout: conn = await self._remote_connection(host, addr) 2022-05-13T23:59:55.913 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-13T23:59:55.913 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise 2022-05-13T23:59:55.913 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-13T23:59:55.914 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.gen.throw(type, value, traceback) 2022-05-13T23:59:55.915 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-13T23:59:55.916 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OrchestratorError(msg) 2022-05-13T23:59:55.916 INFO:journalctl@ceph.mon.c.smithi115.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-13T23:59:55.917 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:59:55 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:59:56.539 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:59:56 smithi115 conmon[90393]: 2022-05-13T23:59:56.264+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:59:56.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:59:56 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:59:56.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:59:56 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:59:57.543 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:59:57 smithi115 conmon[90393]: 2022-05-13T23:59:57.254+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:59:57.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:59:57 smithi047 ceph-mon[77147]: pgmap v1317: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:59:57.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:59:57 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:59:57.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:59:57 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-13T23:59:57.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:59:57 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:59:57.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:59:57 smithi115 ceph-mon[78593]: pgmap v1317: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-13T23:59:57.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:59:57 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-13T23:59:57.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:59:57 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-13T23:59:57.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:59:57 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:59:58.544 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:59:58 smithi115 conmon[90393]: 2022-05-13T23:59:58.258+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:59:58.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:59:58 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:59:58.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:59:58 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 1930 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:59:59.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:59:58 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-13T23:59:59.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:59:58 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 1930 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-13T23:59:59.547 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 13 23:59:59 smithi115 conmon[90393]: 2022-05-13T23:59:59.268+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-13T23:59:59.862 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:59:59 smithi115 ceph-mon[78593]: pgmap v1318: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-13T23:59:59.863 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 13 23:59:59 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:00:00.001 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:59:59 smithi047 ceph-mon[77147]: pgmap v1318: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:00:00.002 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 13 23:59:59 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:00:00.296 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:00:00 smithi047 conmon[77123]: 2022-05-13T23:59:59.999+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : Health detail: HEALTH_ERR 1 filesystem is degraded; 1 mds daemon damaged; 1 MDSs are read only; 1 MDSs report slow metadata IOs; 1/3 mons down, quorum b,c; Reduced data availability: 1 pg inactive; 9 slow ops, oldest one blocked for 1930 sec, osd.5 has slow ops 2022-05-14T00:00:00.297 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:00:00 smithi047 conmon[77123]: 2022-05-13T23:59:59.999+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : [WRN] FS_DEGRADED: 1 filesystem is degraded 2022-05-14T00:00:00.298 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:00:00 smithi047 conmon[77123]: 2022-05-13T23:59:59.999+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : fs cephfs is degraded 2022-05-14T00:00:00.298 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:00:00 smithi047 conmon[77123]: 2022-05-13T23:59:59.999+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : [ERR] MDS_DAMAGE: 1 mds daemon damaged 2022-05-14T00:00:00.298 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:00:00 smithi047 conmon[77123]: 2022-05-13T23:59:59.999+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : fs cephfs mds.2 is damaged 2022-05-14T00:00:00.299 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:00:00 smithi047 conmon[77123]: 2022-05-14T00:00:00.000+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : [WRN] MDS_READ_ONLY: 1 MDSs are read only 2022-05-14T00:00:00.299 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:00:00 smithi047 conmon[77123]: 2022-05-14T00:00:00.000+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : mds.l(mds.0): MDS in read-only mode 2022-05-14T00:00:00.300 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:00:00 smithi047 conmon[77123]: 2022-05-14T00:00:00.000+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : [WRN] MDS_SLOW_METADATA_IO: 1 MDSs report slow metadata IOs 2022-05-14T00:00:00.302 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:00:00 smithi047 conmon[77123]: 2022-05-14T00:00:00.000+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : mds.l(mds.0): 4 slow metadata IOs are blocked > 180 secs, oldest blocked for 2233 secs 2022-05-14T00:00:00.303 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:00:00 smithi047 conmon[77123]: 2022-05-14T00:00:00.000+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : [WRN] MON_DOWN: 1/3 mons down, quorum b,c 2022-05-14T00:00:00.303 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:00:00 smithi047 conmon[77123]: 2022-05-14T00:00:00.000+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : mon.a (rank 0) addr [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] is down (out of quorum) 2022-05-14T00:00:00.306 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:00:00 smithi047 conmon[77123]: 2022-05-14T00:00:00.000+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : [WRN] PG_AVAILABILITY: Reduced data availability: 1 pg inactive 2022-05-14T00:00:00.306 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:00:00 smithi047 conmon[77123]: 2022-05-14T00:00:00.000+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : pg 2.1 is stuck inactive for 44m, current state unknown, last acting [] 2022-05-14T00:00:00.306 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:00:00 smithi047 conmon[77123]: 2022-05-14T00:00:00.000+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : [WRN] SLOW_OPS: 9 slow ops, oldest one blocked for 1930 sec, osd.5 has slow ops 2022-05-14T00:00:00.551 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:00:00 smithi115 conmon[90393]: 2022-05-14T00:00:00.244+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:00:00.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:00:00 smithi115 ceph-mon[78593]: Health detail: HEALTH_ERR 1 filesystem is degraded; 1 mds daemon damaged; 1 MDSs are read only; 1 MDSs report slow metadata IOs; 1/3 mons down, quorum b,c; Reduced data availability: 1 pg inactive; 9 slow ops, oldest one blocked for 1930 sec, osd.5 has slow ops 2022-05-14T00:00:00.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:00:00 smithi115 ceph-mon[78593]: [WRN] FS_DEGRADED: 1 filesystem is degraded 2022-05-14T00:00:00.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:00:00 smithi115 ceph-mon[78593]: fs cephfs is degraded 2022-05-14T00:00:00.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:00:00 smithi115 ceph-mon[78593]: [ERR] MDS_DAMAGE: 1 mds daemon damaged 2022-05-14T00:00:00.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:00:00 smithi115 ceph-mon[78593]: fs cephfs mds.2 is damaged 2022-05-14T00:00:00.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:00:00 smithi115 ceph-mon[78593]: [WRN] MDS_READ_ONLY: 1 MDSs are read only 2022-05-14T00:00:00.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:00:00 smithi115 ceph-mon[78593]: mds.l(mds.0): MDS in read-only mode 2022-05-14T00:00:00.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:00:00 smithi115 ceph-mon[78593]: [WRN] MDS_SLOW_METADATA_IO: 1 MDSs report slow metadata IOs 2022-05-14T00:00:00.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:00:00 smithi115 ceph-mon[78593]: mds.l(mds.0): 4 slow metadata IOs are blocked > 180 secs, oldest blocked for 2233 secs 2022-05-14T00:00:00.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:00:00 smithi115 ceph-mon[78593]: [WRN] MON_DOWN: 1/3 mons down, quorum b,c 2022-05-14T00:00:00.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:00:00 smithi115 ceph-mon[78593]: mon.a (rank 0) addr [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] is down (out of quorum) 2022-05-14T00:00:00.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:00:00 smithi115 ceph-mon[78593]: [WRN] PG_AVAILABILITY: Reduced data availability: 1 pg inactive 2022-05-14T00:00:00.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:00:00 smithi115 ceph-mon[78593]: pg 2.1 is stuck inactive for 44m, current state unknown, last acting [] 2022-05-14T00:00:00.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:00:00 smithi115 ceph-mon[78593]: [WRN] SLOW_OPS: 9 slow ops, oldest one blocked for 1930 sec, osd.5 has slow ops 2022-05-14T00:00:00.862 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:00:00 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:00:01.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:00:00 smithi047 ceph-mon[77147]: Health detail: HEALTH_ERR 1 filesystem is degraded; 1 mds daemon damaged; 1 MDSs are read only; 1 MDSs report slow metadata IOs; 1/3 mons down, quorum b,c; Reduced data availability: 1 pg inactive; 9 slow ops, oldest one blocked for 1930 sec, osd.5 has slow ops 2022-05-14T00:00:01.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:00:00 smithi047 ceph-mon[77147]: [WRN] FS_DEGRADED: 1 filesystem is degraded 2022-05-14T00:00:01.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:00:00 smithi047 ceph-mon[77147]: fs cephfs is degraded 2022-05-14T00:00:01.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:00:00 smithi047 ceph-mon[77147]: [ERR] MDS_DAMAGE: 1 mds daemon damaged 2022-05-14T00:00:01.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:00:00 smithi047 ceph-mon[77147]: fs cephfs mds.2 is damaged 2022-05-14T00:00:01.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:00:00 smithi047 ceph-mon[77147]: [WRN] MDS_READ_ONLY: 1 MDSs are read only 2022-05-14T00:00:01.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:00:00 smithi047 ceph-mon[77147]: mds.l(mds.0): MDS in read-only mode 2022-05-14T00:00:01.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:00:00 smithi047 ceph-mon[77147]: [WRN] MDS_SLOW_METADATA_IO: 1 MDSs report slow metadata IOs 2022-05-14T00:00:01.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:00:00 smithi047 ceph-mon[77147]: mds.l(mds.0): 4 slow metadata IOs are blocked > 180 secs, oldest blocked for 2233 secs 2022-05-14T00:00:01.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:00:00 smithi047 ceph-mon[77147]: [WRN] MON_DOWN: 1/3 mons down, quorum b,c 2022-05-14T00:00:01.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:00:00 smithi047 ceph-mon[77147]: mon.a (rank 0) addr [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] is down (out of quorum) 2022-05-14T00:00:01.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:00:00 smithi047 ceph-mon[77147]: [WRN] PG_AVAILABILITY: Reduced data availability: 1 pg inactive 2022-05-14T00:00:01.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:00:00 smithi047 ceph-mon[77147]: pg 2.1 is stuck inactive for 44m, current state unknown, last acting [] 2022-05-14T00:00:01.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:00:00 smithi047 ceph-mon[77147]: [WRN] SLOW_OPS: 9 slow ops, oldest one blocked for 1930 sec, osd.5 has slow ops 2022-05-14T00:00:01.045 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:00:00 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:00:01.558 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:00:01 smithi115 conmon[90393]: 2022-05-14T00:00:01.239+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:00:01.870 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:00:01 smithi115 ceph-mon[78593]: pgmap v1319: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:00:01.870 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:00:01 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:00:01.870 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:00:01 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:00:01.871 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:00:01 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:00:02.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:00:01 smithi047 ceph-mon[77147]: pgmap v1319: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:00:02.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:00:01 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:00:02.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:00:01 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:00:02.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:00:01 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:00:02.559 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:00:02 smithi115 conmon[90393]: 2022-05-14T00:00:02.236+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:00:02.862 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:00:02 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:00:03.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:00:02 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:00:03.560 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:00:03 smithi115 conmon[90393]: 2022-05-14T00:00:03.223+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:00:03.866 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:00:03 smithi115 ceph-mon[78593]: pgmap v1320: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:00:03.866 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:00:03 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:00:03.867 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:00:03 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 1935 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:00:04.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:00:03 smithi047 ceph-mon[77147]: pgmap v1320: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:00:04.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:00:03 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:00:04.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:00:03 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 1935 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:00:04.574 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:00:04 smithi115 conmon[90393]: 2022-05-14T00:00:04.221+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:00:04.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:00:04 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:00:05.046 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:00:04 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:00:05.566 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:00:05 smithi115 conmon[90393]: 2022-05-14T00:00:05.258+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:00:05.862 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:00:05 smithi115 ceph-mon[78593]: pgmap v1321: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:00:05.862 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:00:05 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:00:06.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:00:05 smithi047 ceph-mon[77147]: pgmap v1321: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:00:06.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:00:05 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:00:06.570 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:00:06 smithi115 conmon[90393]: 2022-05-14T00:00:06.268+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:00:06.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:00:06 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:00:07.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:00:06 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:00:07.578 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:00:07 smithi115 conmon[90393]: 2022-05-14T00:00:07.283+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:00:07.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:00:07 smithi115 ceph-mon[78593]: pgmap v1322: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:00:07.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:00:07 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:00:08.059 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:00:07 smithi047 ceph-mon[77147]: pgmap v1322: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:00:08.098 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:00:07 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:00:08.593 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:00:08 smithi115 conmon[90393]: 2022-05-14T00:00:08.239+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:00:08.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:00:08 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:00:08.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:00:08 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 1940 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:00:08.892 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:00:08 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:00:08.893 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:00:08 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 1940 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:00:09.583 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:00:09 smithi115 conmon[90393]: 2022-05-14T00:00:09.248+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:00:09.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:00:09 smithi115 ceph-mon[78593]: pgmap v1323: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:00:09.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:00:09 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:00:09.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:00:09 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:00:09.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:00:09 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:00:10.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:00:09 smithi047 ceph-mon[77147]: pgmap v1323: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:00:10.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:00:09 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:00:10.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:00:09 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:00:10.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:00:09 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:00:10.587 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:00:10 smithi115 conmon[90393]: 2022-05-14T00:00:10.233+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:00:10.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:00:10 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:00:11.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:00:10 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:00:11.588 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:00:11 smithi115 conmon[90393]: 2022-05-14T00:00:11.251+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:00:11.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:00:11 smithi115 ceph-mon[78593]: pgmap v1324: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:00:11.862 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:00:11 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:00:12.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:00:11 smithi047 ceph-mon[77147]: pgmap v1324: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:00:12.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:00:11 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:00:12.595 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:00:12 smithi115 conmon[90393]: 2022-05-14T00:00:12.261+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:00:12.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:00:12 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:00:13.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:00:12 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:00:13.598 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:00:13 smithi115 conmon[90393]: 2022-05-14T00:00:13.257+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:00:13.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:00:13 smithi115 ceph-mon[78593]: pgmap v1325: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:00:13.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:00:13 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:00:13.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:00:13 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:00:13.862 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:00:13 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T00:00:13.862 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:00:13 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T00:00:13.863 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:00:13 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:00:13.864 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:00:13 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 1945 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:00:14.046 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:00:13 smithi047 ceph-mon[77147]: pgmap v1325: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:00:14.046 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:00:13 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:00:14.046 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:00:13 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:00:14.047 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:00:13 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T00:00:14.047 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:00:13 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T00:00:14.047 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:00:13 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:00:14.047 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:00:13 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 1945 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:00:14.606 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:00:14 smithi115 conmon[90393]: 2022-05-14T00:00:14.212+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:00:14.866 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:00:14 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:00:15.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:00:14 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:00:15.600 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:00:15 smithi115 conmon[90393]: 2022-05-14T00:00:15.183+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:00:15.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:00:15 smithi115 ceph-mon[78593]: pgmap v1326: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:00:15.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:00:15 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:00:16.046 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:00:15 smithi047 ceph-mon[77147]: pgmap v1326: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:00:16.048 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:00:15 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:00:16.614 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:00:16 smithi115 conmon[90393]: 2022-05-14T00:00:16.191+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:00:16.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:00:16 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:00:17.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:00:16 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:00:17.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:00:17 smithi115 conmon[90393]: 2022-05-14T00:00:17.202+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:00:17.862 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:00:17 smithi115 ceph-mon[78593]: pgmap v1327: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:00:17.862 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:00:17 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:00:17.862 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:00:17 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:00:17.863 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:00:17 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:00:18.045 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:00:17 smithi047 ceph-mon[77147]: pgmap v1327: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:00:18.046 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:00:17 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:00:18.046 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:00:17 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:00:18.046 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:00:17 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:00:18.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:00:18 smithi115 conmon[90393]: 2022-05-14T00:00:18.174+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:00:19.048 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:00:18 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:00:19.049 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:00:18 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 1950 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:00:19.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:00:18 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:00:19.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:00:18 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 1950 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:00:19.611 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:00:19 smithi115 conmon[90393]: 2022-05-14T00:00:19.168+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:00:20.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:00:19 smithi047 ceph-mon[77147]: pgmap v1328: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:00:20.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:00:19 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:00:20.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:00:19 smithi115 ceph-mon[78593]: pgmap v1328: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:00:20.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:00:19 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:00:20.616 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:00:20 smithi115 conmon[90393]: 2022-05-14T00:00:20.200+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:00:21.045 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:00:20 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:00:21.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:00:20 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:00:21.610 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:00:21 smithi115 conmon[90393]: 2022-05-14T00:00:21.240+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:00:22.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:00:21 smithi047 ceph-mon[77147]: pgmap v1329: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:00:22.047 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:00:21 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:00:22.048 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:00:21 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:00:22.048 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:00:21 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:00:22.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:00:21 smithi115 ceph-mon[78593]: pgmap v1329: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:00:22.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:00:21 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:00:22.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:00:21 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:00:22.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:00:21 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:00:22.610 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:00:22 smithi115 conmon[90393]: 2022-05-14T00:00:22.260+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:00:23.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:00:22 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:00:23.123 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:00:22 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:00:23.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:00:23 smithi115 conmon[90393]: 2022-05-14T00:00:23.253+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:00:24.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:00:23 smithi047 ceph-mon[77147]: pgmap v1330: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:00:24.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:00:23 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:00:24.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:00:23 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 1955 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:00:24.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:00:23 smithi115 ceph-mon[78593]: pgmap v1330: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:00:24.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:00:23 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:00:24.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:00:23 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 1955 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:00:24.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:00:24 smithi115 conmon[90393]: 2022-05-14T00:00:24.261+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:00:25.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:00:24 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:00:25.114 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:00:24 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:00:25.611 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:00:25 smithi115 conmon[90393]: 2022-05-14T00:00:25.229+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:00:26.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:00:25 smithi047 ceph-mon[77147]: pgmap v1331: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:00:26.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:00:25 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:00:26.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:00:25 smithi115 ceph-mon[78593]: pgmap v1331: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:00:26.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:00:25 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:00:26.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:00:26 smithi115 conmon[90393]: 2022-05-14T00:00:26.274+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:00:27.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:00:26 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:00:27.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:00:26 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:00:27.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:00:27 smithi115 conmon[90393]: 2022-05-14T00:00:27.241+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:00:28.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:00:27 smithi047 ceph-mon[77147]: pgmap v1332: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:00:28.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:00:27 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:00:28.130 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:00:27 smithi115 ceph-mon[78593]: pgmap v1332: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:00:28.130 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:00:27 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:00:28.615 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:00:28 smithi115 conmon[90393]: 2022-05-14T00:00:28.274+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:00:29.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:00:28 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:00:29.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:00:28 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 1960 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:00:29.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:00:28 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:00:29.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:00:28 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 1960 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:00:29.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:00:29 smithi115 conmon[90393]: 2022-05-14T00:00:29.289+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:00:30.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:00:29 smithi047 ceph-mon[77147]: pgmap v1333: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:00:30.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:00:29 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:00:30.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:00:29 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:00:30.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:00:29 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:00:30.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:00:29 smithi115 ceph-mon[78593]: pgmap v1333: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:00:30.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:00:29 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:00:30.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:00:29 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:00:30.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:00:29 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:00:30.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:00:30 smithi115 conmon[90393]: 2022-05-14T00:00:30.308+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:00:31.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:00:30 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:00:31.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:00:30 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:00:31.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:00:31 smithi115 conmon[90393]: 2022-05-14T00:00:31.316+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:00:32.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:00:31 smithi047 ceph-mon[77147]: pgmap v1334: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:00:32.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:00:31 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:00:32.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:00:31 smithi115 ceph-mon[78593]: pgmap v1334: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:00:32.112 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:00:31 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:00:32.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:00:32 smithi115 conmon[90393]: 2022-05-14T00:00:32.270+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:00:33.047 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:00:32 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:00:33.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:00:32 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:00:33.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:00:33 smithi115 conmon[90393]: 2022-05-14T00:00:33.295+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:00:34.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:00:33 smithi047 ceph-mon[77147]: pgmap v1335: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:00:34.101 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:00:33 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:00:34.102 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:00:33 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:00:34.103 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:00:33 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:00:34.103 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:00:33 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 1965 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:00:34.142 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:00:33 smithi115 ceph-mon[78593]: pgmap v1335: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:00:34.143 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:00:33 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:00:34.143 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:00:33 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:00:34.144 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:00:33 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:00:34.145 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:00:33 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 1965 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:00:34.616 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:00:34 smithi115 conmon[90393]: 2022-05-14T00:00:34.344+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:00:35.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:00:34 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:00:35.112 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:00:34 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:00:35.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:00:35 smithi115 conmon[90393]: 2022-05-14T00:00:35.335+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:00:36.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:00:35 smithi047 ceph-mon[77147]: pgmap v1336: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:00:36.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:00:35 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:00:36.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:00:35 smithi115 ceph-mon[78593]: pgmap v1336: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:00:36.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:00:35 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:00:36.670 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:00:36 smithi115 conmon[90393]: 2022-05-14T00:00:36.366+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:00:37.046 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:00:36 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:00:37.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:00:36 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:00:37.674 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:00:37 smithi115 conmon[90393]: 2022-05-14T00:00:37.388+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:00:38.046 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:00:37 smithi047 ceph-mon[77147]: pgmap v1337: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:00:38.046 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:00:37 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:00:38.046 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:00:37 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:00:38.118 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:00:37 smithi115 ceph-mon[78593]: pgmap v1337: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:00:38.118 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:00:37 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:00:38.118 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:00:37 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:00:38.861 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:00:38 smithi115 conmon[90393]: 2022-05-14T00:00:38.429+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:00:38.866 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:00:38 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:00:38.866 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:00:38 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 1970 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:00:39.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:00:38 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:00:39.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:00:38 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 1970 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:00:39.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:00:39 smithi115 conmon[90393]: 2022-05-14T00:00:39.442+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:00:39.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:00:39 smithi115 ceph-mon[78593]: pgmap v1338: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:00:39.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:00:39 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:00:40.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:00:39 smithi047 ceph-mon[77147]: pgmap v1338: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:00:40.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:00:39 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:00:40.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:00:40 smithi115 conmon[90393]: 2022-05-14T00:00:40.483+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:00:40.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:00:40 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:00:41.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:00:40 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:00:41.860 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:00:41 smithi115 conmon[90393]: 2022-05-14T00:00:41.533+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:00:41.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:00:41 smithi115 ceph-mon[78593]: pgmap v1339: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:00:41.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:00:41 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:00:41.862 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:00:41 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:00:41.862 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:00:41 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:00:42.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:00:41 smithi047 ceph-mon[77147]: pgmap v1339: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:00:42.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:00:41 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:00:42.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:00:41 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:00:42.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:00:41 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:00:42.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:00:42 smithi115 conmon[90393]: 2022-05-14T00:00:42.549+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:00:42.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:00:42 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:00:43.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:00:42 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:00:43.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:00:43 smithi115 conmon[90393]: 2022-05-14T00:00:43.582+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:00:43.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:00:43 smithi115 ceph-mon[78593]: pgmap v1340: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:00:43.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:00:43 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:00:43.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:00:43 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 1975 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:00:44.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:00:43 smithi047 ceph-mon[77147]: pgmap v1340: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:00:44.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:00:43 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:00:44.047 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:00:43 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 1975 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:00:44.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:00:44 smithi115 conmon[90393]: 2022-05-14T00:00:44.577+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:00:44.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:00:44 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:00:45.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:00:44 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:00:45.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:00:45 smithi115 conmon[90393]: 2022-05-14T00:00:45.564+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:00:45.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:00:45 smithi115 ceph-mon[78593]: pgmap v1341: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:00:45.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:00:45 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:00:46.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:00:45 smithi047 ceph-mon[77147]: pgmap v1341: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:00:46.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:00:45 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:00:46.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:00:46 smithi115 conmon[90393]: 2022-05-14T00:00:46.569+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:00:46.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:00:46 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:00:47.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:00:46 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:00:47.862 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:00:47 smithi115 conmon[90393]: 2022-05-14T00:00:47.555+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:00:47.863 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:00:47 smithi115 ceph-mon[78593]: pgmap v1342: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:00:47.863 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:00:47 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:00:48.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:00:47 smithi047 ceph-mon[77147]: pgmap v1342: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:00:48.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:00:47 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:00:48.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:00:48 smithi115 conmon[90393]: 2022-05-14T00:00:48.570+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:00:48.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:00:48 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:00:48.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:00:48 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 1980 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:00:49.058 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:00:48 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:00:49.058 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:00:48 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 1980 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:00:49.880 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:00:49 smithi115 conmon[90393]: 2022-05-14T00:00:49.545+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:00:49.881 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:00:49 smithi115 ceph-mon[78593]: pgmap v1343: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:00:49.881 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:00:49 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:00:49.882 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:00:49 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:00:49.882 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:00:49 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:00:50.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:00:49 smithi047 ceph-mon[77147]: pgmap v1343: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:00:50.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:00:49 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:00:50.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:00:49 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:00:50.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:00:49 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:00:50.862 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:00:50 smithi115 conmon[90393]: 2022-05-14T00:00:50.523+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:00:50.876 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:00:50 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:00:51.046 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:00:50 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:00:51.861 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:00:51 smithi115 conmon[90393]: 2022-05-14T00:00:51.481+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:00:51.863 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:00:51 smithi115 ceph-mon[78593]: pgmap v1344: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:00:51.863 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:00:51 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:00:52.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:00:51 smithi047 ceph-mon[77147]: pgmap v1344: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:00:52.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:00:51 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:00:52.724 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:00:52 smithi115 conmon[90393]: 2022-05-14T00:00:52.439+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:00:53.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:00:52 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:00:53.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:00:52 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:00:53.724 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:00:53 smithi115 conmon[90393]: 2022-05-14T00:00:53.449+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:00:54.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:00:53 smithi047 ceph-mon[77147]: pgmap v1345: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:00:54.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:00:53 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:00:54.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:00:53 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:00:54.047 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:00:53 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:00:54.048 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:00:53 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 1985 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:00:54.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:00:53 smithi115 ceph-mon[78593]: pgmap v1345: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:00:54.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:00:53 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:00:54.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:00:53 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:00:54.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:00:53 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:00:54.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:00:53 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 1985 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:00:54.728 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:00:54 smithi115 conmon[90393]: 2022-05-14T00:00:54.467+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:00:55.051 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:00:54 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:00:55.051 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:00:54 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T00:00:55.051 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:00:54 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T00:00:55.052 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:00:54 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T00:00:55.120 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:00:54 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:00:55.121 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:00:54 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T00:00:55.121 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:00:54 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T00:00:55.122 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:00:54 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T00:00:55.730 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:00:55 smithi115 conmon[90393]: 2022-05-14T00:00:55.468+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:00:55.730 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:00:55 smithi115 ceph-mon[78593]: pgmap v1346: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:00:55.731 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:00:55 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:00:55.731 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:00:55 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' 2022-05-14T00:00:55.731 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:00:55 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' 2022-05-14T00:00:56.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:00:55 smithi047 ceph-mon[77147]: pgmap v1346: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:00:56.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:00:55 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:00:56.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:00:55 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' 2022-05-14T00:00:56.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:00:55 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' 2022-05-14T00:00:56.734 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:00:56 smithi115 conmon[90393]: 2022-05-14T00:00:56.433+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:00:57.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:00:56 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:00:57.113 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:00:56 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:00:57.738 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:00:57 smithi115 conmon[90393]: 2022-05-14T00:00:57.398+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:00:57.738 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:00:57 smithi047 conmon[78521]: 2022-05-14T00:00:57.457+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:00:57.739 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:00:57 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T00:00:57.739 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:00:57 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:00:57.739 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:00:57 smithi047 conmon[78521]: yield 2022-05-14T00:00:57.739 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:00:57 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:00:57.740 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:00:57 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:00:57.740 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:00:57 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:00:57.740 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:00:57 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T00:00:57.740 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:00:57 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:00:57.740 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:00:57 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T00:00:57.741 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:00:57 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:00:57.741 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:00:57 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T00:00:57.741 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:00:57 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:00:57.742 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:00:57 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T00:00:57.742 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:00:57 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:00:57.742 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:00:57 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T00:00:57.742 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:00:57 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:00:57.743 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:00:57 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:00:57.743 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:00:57 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:00:58.054 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:00:57 smithi047 ceph-mon[77147]: pgmap v1347: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:00:58.054 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:00:57 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:00:58.054 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:00:57 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:00:58.055 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:00:57 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:00:58.055 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:00:57 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' 2022-05-14T00:00:58.055 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:00:57 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' 2022-05-14T00:00:58.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:00:57 smithi115 ceph-mon[78593]: pgmap v1347: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:00:58.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:00:57 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:00:58.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:00:57 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:00:58.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:00:57 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:00:58.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:00:57 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' 2022-05-14T00:00:58.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:00:57 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' 2022-05-14T00:00:58.749 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:00:58 smithi115 conmon[90393]: 2022-05-14T00:00:58.399+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:00:59.046 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:00:58 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:00:59.047 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:00:58 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:00:59.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T00:00:59.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:00:59.049 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T00:00:59.049 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:00:59.049 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:00:59.050 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:00:59.051 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T00:00:59.052 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:00:59.052 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T00:00:59.053 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:00:59.053 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T00:00:59.054 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:00:59.054 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T00:00:59.054 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:00:59.055 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T00:00:59.055 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:00:59.056 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:00:59.056 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:00:59.056 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:00:58 smithi047 ceph-mon[77147]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T00:00:59.057 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:00:58 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-05-14T00:00:59.057 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:00:58 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-05-14T00:00:59.057 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:00:58 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-05-14T00:00:59.058 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:00:58 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-05-14T00:00:59.058 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:00:58 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-05-14T00:00:59.058 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:00:58 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-05-14T00:00:59.059 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:00:58 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-05-14T00:00:59.059 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:00:58 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-05-14T00:00:59.060 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:00:58 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 1990 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:00:59.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:00:58 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:00:59.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:00:58 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:00:59.119 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T00:00:59.120 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:00:59.120 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T00:00:59.120 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:00:59.120 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:00:59.141 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:00:59.143 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T00:00:59.144 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:00:59.144 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T00:00:59.144 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:00:59.144 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T00:00:59.145 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:00:59.145 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T00:00:59.145 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:00:59.145 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T00:00:59.146 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:00:59.146 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:00:59.146 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:00:59.146 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:00:58 smithi115 ceph-mon[78593]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T00:00:59.147 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:00:58 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-05-14T00:00:59.147 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:00:58 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-05-14T00:00:59.147 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:00:58 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-05-14T00:00:59.147 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:00:58 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-05-14T00:00:59.148 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:00:58 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-05-14T00:00:59.148 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:00:58 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-05-14T00:00:59.148 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:00:58 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-05-14T00:00:59.148 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:00:58 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-05-14T00:00:59.148 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:00:58 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 1990 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:00:59.739 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:00:59 smithi115 conmon[90393]: 2022-05-14T00:00:59.359+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:01:00.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:00:59 smithi047 ceph-mon[77147]: pgmap v1348: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:01:00.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:00:59 smithi047 ceph-mon[77147]: Adjusting osd_memory_target on smithi115 to 197.0M 2022-05-14T00:01:00.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:00:59 smithi047 ceph-mon[77147]: Unable to set osd_memory_target on smithi115 to 206629785: error parsing value: Value '206629785' is below minimum 939524096 2022-05-14T00:01:00.045 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:00:59 smithi047 ceph-mon[77147]: Adjusting osd_memory_target on smithi047 to 198.1M 2022-05-14T00:01:00.046 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:00:59 smithi047 ceph-mon[77147]: Unable to set osd_memory_target on smithi047 to 207755878: error parsing value: Value '207755878' is below minimum 939524096 2022-05-14T00:01:00.053 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:00:59 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:01:00.053 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:00:59 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:01:00.112 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:00:59 smithi115 ceph-mon[78593]: pgmap v1348: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:01:00.130 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:00:59 smithi115 ceph-mon[78593]: Adjusting osd_memory_target on smithi115 to 197.0M 2022-05-14T00:01:00.131 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:00:59 smithi115 ceph-mon[78593]: Unable to set osd_memory_target on smithi115 to 206629785: error parsing value: Value '206629785' is below minimum 939524096 2022-05-14T00:01:00.132 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:00:59 smithi115 ceph-mon[78593]: Adjusting osd_memory_target on smithi047 to 198.1M 2022-05-14T00:01:00.136 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:00:59 smithi115 ceph-mon[78593]: Unable to set osd_memory_target on smithi047 to 207755878: error parsing value: Value '207755878' is below minimum 939524096 2022-05-14T00:01:00.137 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:00:59 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:01:00.159 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:00:59 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:01:00.792 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:01:00 smithi047 conmon[78521]: 2022-05-14T00:01:00.527+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:01:00.792 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:01:00 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T00:01:00.793 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:01:00 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:01:00.793 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:01:00 smithi047 conmon[78521]: yield 2022-05-14T00:01:00.793 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:01:00 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:01:00.793 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:01:00 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:01:00.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:01:00 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:01:00.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:01:00 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T00:01:00.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:01:00 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:01:00.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:01:00 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T00:01:00.797 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:01:00 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:01:00.797 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:01:00 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T00:01:00.797 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:01:00 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:01:00.797 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:01:00 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T00:01:00.797 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:01:00 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:01:00.798 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:01:00 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T00:01:00.798 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:01:00 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:01:00.798 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:01:00 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:01:00.798 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:01:00 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:01:00.799 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:01:00 smithi047 conmon[78521]: 2022-05-14T00:01:00.528+0000 7f533f777700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T00:01:00.799 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:01:00 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T00:01:00.799 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:01:00 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:01:00.799 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:01:00 smithi047 conmon[78521]: yield 2022-05-14T00:01:00.799 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:01:00 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:01:00.800 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:01:00 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:01:00.800 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:01:00 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:01:00.800 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:01:00 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T00:01:00.800 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:01:00 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:01:00.801 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:01:00 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T00:01:00.801 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:01:00 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:01:00.801 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:01:00 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T00:01:00.801 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:01:00 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:01:00.802 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:01:00 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T00:01:00.802 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:01:00 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:01:00.802 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:01:00 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T00:01:00.802 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:01:00 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:01:00.802 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:01:00 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:01:00.803 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:01:00 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:01:00.803 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:01:00 smithi047 conmon[78521]: 2022-05-14T00:01:00.803 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:01:00 smithi047 conmon[78521]: During handling of the above exception, another exception occurred: 2022-05-14T00:01:00.803 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:01:00 smithi047 conmon[78521]: 2022-05-14T00:01:00.804 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:01:00 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T00:01:00.804 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:01:00 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T00:01:00.804 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:01:00 smithi047 conmon[78521]: return f(*arg) 2022-05-14T00:01:00.804 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:01:00 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T00:01:00.804 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:01:00 smithi047 conmon[78521]: self._write_client_files(client_files, host) 2022-05-14T00:01:00.805 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:01:00 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T00:01:00.805 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:01:00 smithi047 conmon[78521]: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T00:01:00.805 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:01:00 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T00:01:00.805 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:01:00 smithi047 conmon[78521]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T00:01:00.806 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:01:00 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T00:01:00.806 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:01:00 smithi047 conmon[78521]: return self.event_loop.get_result(coro) 2022-05-14T00:01:00.806 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:01:00 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T00:01:00.806 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:01:00 smithi047 conmon[78521]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T00:01:00.806 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:01:00 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T00:01:00.807 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:01:00 smithi047 conmon[78521]: return self.__get_result() 2022-05-14T00:01:00.807 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:01:00 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T00:01:00.807 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:01:00 smithi047 conmon[78521]: raise self._exception 2022-05-14T00:01:00.807 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:01:00 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T00:01:00.807 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:01:00 smithi047 conmon[78521]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T00:01:00.808 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:01:00 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T00:01:00.808 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:01:00 smithi047 conmon[78521]: conn = await self._remote_connection(host, addr) 2022-05-14T00:01:00.808 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:01:00 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T00:01:00.808 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:01:00 smithi047 conmon[78521]: raise 2022-05-14T00:01:00.808 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:01:00 smithi047 conmon[78521]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T00:01:00.809 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:01:00 smithi047 conmon[78521]: self.gen.throw(type, value, traceback) 2022-05-14T00:01:00.809 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:01:00 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T00:01:00.809 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:01:00 smithi047 conmon[78521]: raise OrchestratorError(msg) 2022-05-14T00:01:00.809 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:01:00 smithi047 conmon[78521]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:01:00.860 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:01:00 smithi115 conmon[90393]: 2022-05-14T00:01:00.375+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:01:01.612 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:01:01 smithi115 conmon[90393]: 2022-05-14T00:01:01.345+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:01:02.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:01:01 smithi047 ceph-mon[77147]: pgmap v1349: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:01:02.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:01:01 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:01:02.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:01:01 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:01:02.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T00:01:02.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:01:02.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T00:01:02.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:01:02.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:01:02.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:01:02.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T00:01:02.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:01:02.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T00:01:02.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:01:02.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T00:01:02.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:01:02.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T00:01:02.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:01:02.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T00:01:02.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:01:02.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:01:02.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:01:02.047 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:01:01 smithi047 ceph-mon[77147]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T00:01:02.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T00:01:02.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:01:02.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T00:01:02.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:01:02.049 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:01:02.049 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:01:02.049 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T00:01:02.050 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:01:02.050 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T00:01:02.050 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:01:02.051 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T00:01:02.054 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:01:02.055 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T00:01:02.055 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:01:02.055 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T00:01:02.055 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:01:02.056 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:01:02.056 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:01:02.056 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T00:01:02.056 INFO:journalctl@ceph.mon.b.smithi047.stdout: During handling of the above exception, another exception occurred: 2022-05-14T00:01:02.057 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T00:01:02.057 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T00:01:02.057 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T00:01:02.057 INFO:journalctl@ceph.mon.b.smithi047.stdout: return f(*arg) 2022-05-14T00:01:02.057 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T00:01:02.058 INFO:journalctl@ceph.mon.b.smithi047.stdout: self._write_client_files(client_files, host) 2022-05-14T00:01:02.058 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T00:01:02.058 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T00:01:02.058 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T00:01:02.058 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T00:01:02.059 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T00:01:02.059 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.event_loop.get_result(coro) 2022-05-14T00:01:02.059 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T00:01:02.059 INFO:journalctl@ceph.mon.b.smithi047.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T00:01:02.060 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T00:01:02.060 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.__get_result() 2022-05-14T00:01:02.060 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T00:01:02.060 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise self._exception 2022-05-14T00:01:02.060 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T00:01:02.061 INFO:journalctl@ceph.mon.b.smithi047.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T00:01:02.061 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T00:01:02.061 INFO:journalctl@ceph.mon.b.smithi047.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T00:01:02.062 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T00:01:02.062 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise 2022-05-14T00:01:02.062 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T00:01:02.063 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.gen.throw(type, value, traceback) 2022-05-14T00:01:02.063 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T00:01:02.063 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OrchestratorError(msg) 2022-05-14T00:01:02.063 INFO:journalctl@ceph.mon.b.smithi047.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:01:02.064 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:01:01 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:01:02.064 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:01:01 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:01:02.064 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:01:01 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:01:02.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:01:01 smithi115 ceph-mon[78593]: pgmap v1349: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:01:02.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:01:01 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:01:02.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:01:01 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:01:02.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T00:01:02.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:01:02.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T00:01:02.113 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:01:02.113 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:01:02.113 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:01:02.114 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T00:01:02.115 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:01:02.115 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T00:01:02.115 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:01:02.116 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T00:01:02.116 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:01:02.116 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T00:01:02.116 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:01:02.116 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T00:01:02.117 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:01:02.117 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:01:02.117 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:01:02.117 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:01:01 smithi115 ceph-mon[78593]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T00:01:02.118 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T00:01:02.118 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:01:02.118 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T00:01:02.118 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:01:02.118 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:01:02.119 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:01:02.119 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T00:01:02.119 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:01:02.119 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T00:01:02.119 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:01:02.120 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T00:01:02.120 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:01:02.120 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T00:01:02.120 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:01:02.121 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T00:01:02.121 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:01:02.121 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:01:02.121 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:01:02.121 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T00:01:02.122 INFO:journalctl@ceph.mon.c.smithi115.stdout: During handling of the above exception, another exception occurred: 2022-05-14T00:01:02.122 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T00:01:02.122 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T00:01:02.122 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T00:01:02.122 INFO:journalctl@ceph.mon.c.smithi115.stdout: return f(*arg) 2022-05-14T00:01:02.123 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T00:01:02.123 INFO:journalctl@ceph.mon.c.smithi115.stdout: self._write_client_files(client_files, host) 2022-05-14T00:01:02.123 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T00:01:02.123 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T00:01:02.123 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T00:01:02.124 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T00:01:02.124 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T00:01:02.124 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.event_loop.get_result(coro) 2022-05-14T00:01:02.125 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T00:01:02.125 INFO:journalctl@ceph.mon.c.smithi115.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T00:01:02.126 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T00:01:02.126 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.__get_result() 2022-05-14T00:01:02.126 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T00:01:02.127 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise self._exception 2022-05-14T00:01:02.127 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T00:01:02.127 INFO:journalctl@ceph.mon.c.smithi115.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T00:01:02.128 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T00:01:02.128 INFO:journalctl@ceph.mon.c.smithi115.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T00:01:02.128 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T00:01:02.129 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise 2022-05-14T00:01:02.129 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T00:01:02.130 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.gen.throw(type, value, traceback) 2022-05-14T00:01:02.130 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T00:01:02.130 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OrchestratorError(msg) 2022-05-14T00:01:02.131 INFO:journalctl@ceph.mon.c.smithi115.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:01:02.131 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:01:01 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:01:02.131 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:01:01 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:01:02.131 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:01:01 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:01:02.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:01:02 smithi115 conmon[90393]: 2022-05-14T00:01:02.331+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:01:03.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:01:02 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:01:03.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:01:02 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:01:03.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:01:03 smithi115 conmon[90393]: 2022-05-14T00:01:03.303+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:01:04.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:01:03 smithi047 ceph-mon[77147]: pgmap v1350: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:01:04.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:01:03 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:01:04.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:01:03 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 1995 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:01:04.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:01:03 smithi115 ceph-mon[78593]: pgmap v1350: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:01:04.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:01:03 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:01:04.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:01:03 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 1995 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:01:04.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:01:04 smithi115 conmon[90393]: 2022-05-14T00:01:04.321+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:01:05.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:01:04 smithi047 ceph-mon[77147]: pgmap v1351: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:01:05.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:01:04 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:01:05.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:01:04 smithi115 ceph-mon[78593]: pgmap v1351: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:01:05.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:01:04 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:01:05.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:01:05 smithi115 conmon[90393]: 2022-05-14T00:01:05.334+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:01:06.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:01:05 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:01:06.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:01:05 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:01:06.610 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:01:06 smithi115 conmon[90393]: 2022-05-14T00:01:06.335+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:01:07.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:01:06 smithi047 ceph-mon[77147]: pgmap v1352: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:01:07.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:01:06 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:01:07.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:01:06 smithi115 ceph-mon[78593]: pgmap v1352: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:01:07.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:01:06 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:01:07.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:01:07 smithi115 conmon[90393]: 2022-05-14T00:01:07.336+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:01:08.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:01:07 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:01:08.112 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:01:07 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:01:08.772 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:01:08 smithi115 conmon[90393]: 2022-05-14T00:01:08.378+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:01:09.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:01:08 smithi047 ceph-mon[77147]: pgmap v1353: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:01:09.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:01:08 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 2000 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:01:09.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:01:08 smithi115 ceph-mon[78593]: pgmap v1353: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:01:09.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:01:08 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 2000 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:01:09.773 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:01:09 smithi115 conmon[90393]: 2022-05-14T00:01:09.370+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:01:10.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:01:09 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:01:10.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:01:09 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:01:10.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:01:09 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:01:10.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:01:09 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:01:10.114 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:01:09 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:01:10.114 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:01:09 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:01:10.114 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:01:09 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:01:10.114 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:01:09 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:01:10.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:01:10 smithi115 conmon[90393]: 2022-05-14T00:01:10.349+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:01:11.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:01:10 smithi047 ceph-mon[77147]: pgmap v1354: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:01:11.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:01:10 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:01:11.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:01:10 smithi115 ceph-mon[78593]: pgmap v1354: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:01:11.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:01:10 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:01:11.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:01:11 smithi115 conmon[90393]: 2022-05-14T00:01:11.303+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:01:12.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:01:11 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:01:12.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:01:11 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:01:12.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:01:12 smithi115 conmon[90393]: 2022-05-14T00:01:12.266+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:01:13.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:01:12 smithi047 ceph-mon[77147]: pgmap v1355: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:01:13.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:01:12 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:01:13.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:01:12 smithi115 ceph-mon[78593]: pgmap v1355: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:01:13.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:01:12 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:01:13.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:01:13 smithi115 conmon[90393]: 2022-05-14T00:01:13.278+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:01:14.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:01:13 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:01:14.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:01:13 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:01:14.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:01:13 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T00:01:14.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:01:13 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T00:01:14.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:01:13 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:01:14.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:01:13 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 2005 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:01:14.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:01:13 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:01:14.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:01:13 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:01:14.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:01:13 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T00:01:14.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:01:13 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T00:01:14.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:01:13 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:01:14.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:01:13 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 2005 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:01:14.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:01:14 smithi115 conmon[90393]: 2022-05-14T00:01:14.299+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:01:15.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:01:14 smithi047 ceph-mon[77147]: pgmap v1356: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:01:15.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:01:14 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:01:15.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:01:14 smithi115 ceph-mon[78593]: pgmap v1356: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:01:15.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:01:14 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:01:15.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:01:15 smithi115 conmon[90393]: 2022-05-14T00:01:15.333+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:01:16.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:01:15 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:01:16.298 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:01:15 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:01:16.610 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:01:16 smithi115 conmon[90393]: 2022-05-14T00:01:16.307+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:01:17.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:01:16 smithi115 ceph-mon[78593]: pgmap v1357: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:01:17.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:01:16 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:01:17.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:01:16 smithi047 ceph-mon[77147]: pgmap v1357: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:01:17.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:01:16 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:01:17.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:01:17 smithi115 conmon[90393]: 2022-05-14T00:01:17.329+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:01:18.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:01:17 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:01:18.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:01:17 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:01:18.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:01:17 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:01:18.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:01:17 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:01:18.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:01:17 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:01:18.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:01:17 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:01:18.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:01:18 smithi115 conmon[90393]: 2022-05-14T00:01:18.330+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:01:19.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:01:18 smithi115 ceph-mon[78593]: pgmap v1358: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:01:19.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:01:18 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:01:19.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:01:18 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 2010 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:01:19.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:01:18 smithi047 ceph-mon[77147]: pgmap v1358: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:01:19.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:01:18 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:01:19.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:01:18 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 2010 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:01:19.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:01:19 smithi115 conmon[90393]: 2022-05-14T00:01:19.338+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:01:20.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:01:19 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:01:20.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:01:19 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:01:20.809 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:01:20 smithi115 conmon[90393]: 2022-05-14T00:01:20.370+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:01:21.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:01:20 smithi115 ceph-mon[78593]: pgmap v1359: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:01:21.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:01:20 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:01:21.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:01:20 smithi047 ceph-mon[77147]: pgmap v1359: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:01:21.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:01:20 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:01:21.812 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:01:21 smithi115 conmon[90393]: 2022-05-14T00:01:21.382+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:01:22.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:01:21 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:01:22.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:01:21 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:01:22.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:01:21 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:01:22.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:01:21 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:01:22.818 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:01:22 smithi115 conmon[90393]: 2022-05-14T00:01:22.378+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:01:23.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:01:22 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:01:23.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:01:22 smithi115 ceph-mon[78593]: pgmap v1360: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:01:23.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:01:22 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:01:23.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:01:22 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:01:23.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:01:22 smithi047 ceph-mon[77147]: pgmap v1360: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:01:23.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:01:22 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:01:23.816 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:01:23 smithi115 conmon[90393]: 2022-05-14T00:01:23.421+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:01:24.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:01:23 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 2015 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:01:24.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:01:23 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 2015 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:01:24.822 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:01:24 smithi115 conmon[90393]: 2022-05-14T00:01:24.388+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:01:25.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:01:24 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:01:25.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:01:24 smithi115 ceph-mon[78593]: pgmap v1361: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:01:25.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:01:24 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:01:25.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:01:24 smithi047 ceph-mon[77147]: pgmap v1361: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:01:25.824 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:01:25 smithi115 conmon[90393]: 2022-05-14T00:01:25.422+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:01:26.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:01:25 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:01:26.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:01:25 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:01:26.831 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:01:26 smithi115 conmon[90393]: 2022-05-14T00:01:26.389+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:01:27.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:01:26 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:01:27.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:01:26 smithi115 ceph-mon[78593]: pgmap v1362: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:01:27.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:01:26 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:01:27.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:01:26 smithi047 ceph-mon[77147]: pgmap v1362: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:01:27.832 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:01:27 smithi115 conmon[90393]: 2022-05-14T00:01:27.403+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:01:27.833 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:01:27 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:01:28.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:01:27 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:01:28.837 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:01:28 smithi115 conmon[90393]: 2022-05-14T00:01:28.439+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:01:29.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:01:28 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:01:29.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:01:28 smithi115 ceph-mon[78593]: pgmap v1363: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:01:29.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:01:28 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 2020 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:01:29.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:01:28 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:01:29.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:01:28 smithi047 ceph-mon[77147]: pgmap v1363: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:01:29.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:01:28 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 2020 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:01:29.838 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:01:29 smithi115 conmon[90393]: 2022-05-14T00:01:29.432+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:01:30.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:01:29 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:01:30.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:01:29 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:01:30.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:01:29 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:01:30.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:01:29 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:01:30.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:01:29 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:01:30.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:01:29 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:01:30.843 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:01:30 smithi115 conmon[90393]: 2022-05-14T00:01:30.392+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:01:31.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:01:30 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:01:31.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:01:30 smithi115 ceph-mon[78593]: pgmap v1364: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:01:31.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:01:30 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:01:31.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:01:30 smithi047 ceph-mon[77147]: pgmap v1364: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:01:31.851 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:01:31 smithi115 conmon[90393]: 2022-05-14T00:01:31.369+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:01:32.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:01:31 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:01:32.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:01:31 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:01:32.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:01:31 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:01:32.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:01:31 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:01:32.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:01:32 smithi115 conmon[90393]: 2022-05-14T00:01:32.336+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:01:33.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:01:32 smithi115 ceph-mon[78593]: pgmap v1365: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:01:33.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:01:32 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:01:33.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:01:32 smithi047 ceph-mon[77147]: pgmap v1365: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:01:33.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:01:32 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:01:33.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:01:33 smithi115 conmon[90393]: 2022-05-14T00:01:33.302+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:01:34.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:01:33 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:01:34.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:01:33 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:01:34.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:01:33 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:01:34.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:01:33 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 2025 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:01:34.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:01:33 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:01:34.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:01:33 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:01:34.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:01:33 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:01:34.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:01:33 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 2025 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:01:34.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:01:34 smithi115 conmon[90393]: 2022-05-14T00:01:34.268+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:01:35.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:01:34 smithi115 ceph-mon[78593]: pgmap v1366: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:01:35.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:01:34 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:01:35.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:01:34 smithi047 ceph-mon[77147]: pgmap v1366: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:01:35.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:01:34 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:01:35.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:01:35 smithi115 conmon[90393]: 2022-05-14T00:01:35.294+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:01:36.266 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:01:35 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:01:36.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:01:35 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:01:36.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:01:36 smithi115 conmon[90393]: 2022-05-14T00:01:36.265+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:01:37.252 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:01:36 smithi115 ceph-mon[78593]: pgmap v1367: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:01:37.253 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:01:36 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:01:37.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:01:36 smithi047 ceph-mon[77147]: pgmap v1367: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:01:37.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:01:36 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:01:37.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:01:37 smithi115 conmon[90393]: 2022-05-14T00:01:37.250+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:01:38.261 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:01:37 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:01:38.262 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:01:37 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:01:38.262 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:01:37 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:01:38.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:01:37 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:01:38.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:01:37 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:01:38.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:01:37 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:01:38.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:01:38 smithi115 conmon[90393]: 2022-05-14T00:01:38.260+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:01:39.243 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:01:38 smithi115 ceph-mon[78593]: pgmap v1368: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:01:39.244 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:01:38 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:01:39.245 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:01:38 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 2030 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:01:39.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:01:38 smithi047 ceph-mon[77147]: pgmap v1368: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:01:39.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:01:38 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:01:39.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:01:38 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 2030 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:01:39.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:01:39 smithi115 conmon[90393]: 2022-05-14T00:01:39.242+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:01:40.281 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:01:39 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:01:40.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:01:39 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:01:40.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:01:40 smithi115 conmon[90393]: 2022-05-14T00:01:40.280+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:01:41.246 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:01:40 smithi115 ceph-mon[78593]: pgmap v1369: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:01:41.247 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:01:40 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:01:41.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:01:40 smithi047 ceph-mon[77147]: pgmap v1369: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:01:41.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:01:40 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:01:41.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:01:41 smithi115 conmon[90393]: 2022-05-14T00:01:41.245+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:01:42.209 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:01:41 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:01:42.210 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:01:41 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:01:42.210 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:01:41 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:01:42.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:01:41 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:01:42.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:01:41 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:01:42.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:01:41 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:01:42.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:01:42 smithi115 conmon[90393]: 2022-05-14T00:01:42.208+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:01:43.202 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:01:42 smithi115 ceph-mon[78593]: pgmap v1370: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:01:43.203 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:01:42 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:01:43.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:01:42 smithi047 ceph-mon[77147]: pgmap v1370: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:01:43.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:01:42 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:01:43.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:01:43 smithi115 conmon[90393]: 2022-05-14T00:01:43.201+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:01:44.198 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:01:43 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:01:44.198 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:01:43 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 2035 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:01:44.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:01:43 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:01:44.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:01:43 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 2035 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:01:44.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:01:44 smithi115 conmon[90393]: 2022-05-14T00:01:44.196+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:01:45.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:01:44 smithi047 ceph-mon[77147]: pgmap v1371: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:01:45.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:01:44 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:01:45.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:01:45 smithi115 conmon[90393]: 2022-05-14T00:01:45.178+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:01:45.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:01:44 smithi115 ceph-mon[78593]: pgmap v1371: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:01:45.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:01:44 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:01:46.202 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:01:45 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:01:46.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:01:45 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:01:46.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:01:46 smithi115 conmon[90393]: 2022-05-14T00:01:46.200+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:01:47.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:01:46 smithi047 ceph-mon[77147]: pgmap v1372: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:01:47.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:01:46 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:01:47.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:01:47 smithi115 conmon[90393]: 2022-05-14T00:01:47.173+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:01:47.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:01:46 smithi115 ceph-mon[78593]: pgmap v1372: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:01:47.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:01:46 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:01:48.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:01:47 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:01:48.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:01:48 smithi115 conmon[90393]: 2022-05-14T00:01:48.182+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:01:48.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:01:47 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:01:49.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:01:48 smithi047 ceph-mon[77147]: pgmap v1373: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:01:49.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:01:48 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:01:49.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:01:48 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 2040 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:01:49.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:01:49 smithi115 conmon[90393]: 2022-05-14T00:01:49.171+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:01:49.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:01:48 smithi115 ceph-mon[78593]: pgmap v1373: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:01:49.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:01:48 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:01:49.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:01:48 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 2040 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:01:50.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:01:49 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:01:50.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:01:49 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:01:50.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:01:49 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:01:50.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:01:50 smithi115 conmon[90393]: 2022-05-14T00:01:50.128+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:01:50.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:01:49 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:01:50.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:01:49 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:01:50.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:01:49 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:01:51.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:01:50 smithi047 ceph-mon[77147]: pgmap v1374: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:01:51.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:01:50 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:01:51.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:01:51 smithi115 conmon[90393]: 2022-05-14T00:01:51.153+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:01:51.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:01:50 smithi115 ceph-mon[78593]: pgmap v1374: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:01:51.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:01:50 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:01:52.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:01:51 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:01:52.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:01:52 smithi115 conmon[90393]: 2022-05-14T00:01:52.159+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:01:52.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:01:51 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:01:53.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:01:52 smithi047 ceph-mon[77147]: pgmap v1375: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:01:53.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:01:52 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:01:53.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:01:53 smithi115 conmon[90393]: 2022-05-14T00:01:53.160+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:01:53.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:01:52 smithi115 ceph-mon[78593]: pgmap v1375: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:01:53.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:01:52 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:01:54.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:01:53 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:01:54.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:01:53 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:01:54.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:01:53 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:01:54.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:01:53 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 2045 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:01:54.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:01:54 smithi115 conmon[90393]: 2022-05-14T00:01:54.137+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:01:54.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:01:53 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:01:54.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:01:53 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:01:54.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:01:53 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:01:54.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:01:53 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 2045 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:01:55.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:01:54 smithi047 ceph-mon[77147]: pgmap v1376: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:01:55.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:01:54 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:01:55.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:01:55 smithi115 conmon[90393]: 2022-05-14T00:01:55.132+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:01:55.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:01:54 smithi115 ceph-mon[78593]: pgmap v1376: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:01:55.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:01:54 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:01:56.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:01:55 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:01:56.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:01:56 smithi115 conmon[90393]: 2022-05-14T00:01:56.111+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:01:56.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:01:55 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:01:57.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:01:56 smithi047 ceph-mon[77147]: pgmap v1377: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:01:57.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:01:56 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:01:57.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:01:57 smithi115 conmon[90393]: 2022-05-14T00:01:57.124+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:01:57.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:01:56 smithi115 ceph-mon[78593]: pgmap v1377: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:01:57.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:01:56 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:01:58.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:01:57 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:01:58.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:01:57 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:01:58.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:01:57 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:01:58.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:01:58 smithi115 conmon[90393]: 2022-05-14T00:01:58.091+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:01:58.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:01:57 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:01:58.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:01:57 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:01:58.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:01:57 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:01:59.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:01:58 smithi047 ceph-mon[77147]: pgmap v1378: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:01:59.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:01:58 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:01:59.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:01:58 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 2050 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:01:59.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:01:59 smithi115 conmon[90393]: 2022-05-14T00:01:59.089+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:01:59.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:01:58 smithi115 ceph-mon[78593]: pgmap v1378: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:01:59.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:01:58 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:01:59.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:01:58 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 2050 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:02:00.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:02:00 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:02:00.361 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:02:00 smithi115 conmon[90393]: 2022-05-14T00:02:00.093+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:02:00.362 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:02:00 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:02:01.005 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:02:01 smithi047 ceph-mon[77147]: pgmap v1379: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:02:01.005 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:02:01 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:02:01.006 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:02:01 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T00:02:01.006 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:02:01 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T00:02:01.007 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:02:01 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T00:02:01.291 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:02:01 smithi047 conmon[78521]: 2022-05-14T00:02:01.135+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:02:01.291 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:02:01 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T00:02:01.292 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:02:01 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:02:01.292 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:02:01 smithi047 conmon[78521]: yield 2022-05-14T00:02:01.292 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:02:01 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:02:01.292 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:02:01 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:02:01.293 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:02:01 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:02:01.293 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:02:01 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T00:02:01.293 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:02:01 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:02:01.293 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:02:01 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T00:02:01.293 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:02:01 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:02:01.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:02:01 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T00:02:01.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:02:01 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:02:01.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:02:01 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T00:02:01.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:02:01 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:02:01.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:02:01 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T00:02:01.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:02:01 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:02:01.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:02:01 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:02:01.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:02:01 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:02:01.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:02:01 smithi115 conmon[90393]: 2022-05-14T00:02:01.071+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:02:01.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:02:01 smithi115 ceph-mon[78593]: pgmap v1379: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:02:01.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:02:01 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:02:01.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:02:01 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T00:02:01.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:02:01 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T00:02:01.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:02:01 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T00:02:02.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:02:02 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:02:02.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:02:02 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:02:02.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:02:02 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:02:02.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:02:02 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:02:02.292 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T00:02:02.292 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:02:02.292 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T00:02:02.292 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:02:02.292 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:02:02.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:02:02.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T00:02:02.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:02:02.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T00:02:02.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:02:02.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T00:02:02.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:02:02.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T00:02:02.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:02:02.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T00:02:02.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:02:02.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:02:02.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:02:02.295 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:02:02 smithi047 ceph-mon[77147]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T00:02:02.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:02:02 smithi115 conmon[90393]: 2022-05-14T00:02:02.120+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:02:02.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:02:02 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:02:02.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:02:02 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:02:02.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:02:02 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:02:02.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:02:02 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:02:02.359 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T00:02:02.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:02:02.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T00:02:02.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:02:02.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:02:02.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:02:02.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T00:02:02.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:02:02.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T00:02:02.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:02:02.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T00:02:02.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:02:02.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T00:02:02.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:02:02.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T00:02:02.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:02:02.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:02:02.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:02:02.364 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:02:02 smithi115 ceph-mon[78593]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T00:02:03.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:02:03 smithi047 ceph-mon[77147]: pgmap v1380: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:02:03.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:02:03 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:02:03.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:02:03 smithi115 conmon[90393]: 2022-05-14T00:02:03.124+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:02:03.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:02:03 smithi115 ceph-mon[78593]: pgmap v1380: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:02:03.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:02:03 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:02:04.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:02:04 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:02:04.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:02:04 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 2056 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:02:04.292 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:02:04 smithi047 conmon[78521]: 2022-05-14T00:02:04.207+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:02:04.292 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:02:04 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T00:02:04.292 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:02:04 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:02:04.293 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:02:04 smithi047 conmon[78521]: yield 2022-05-14T00:02:04.293 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:02:04 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:02:04.293 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:02:04 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:02:04.293 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:02:04 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:02:04.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:02:04 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T00:02:04.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:02:04 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:02:04.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:02:04 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T00:02:04.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:02:04 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:02:04.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:02:04 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T00:02:04.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:02:04 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:02:04.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:02:04 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T00:02:04.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:02:04 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:02:04.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:02:04 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T00:02:04.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:02:04 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:02:04.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:02:04 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:02:04.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:02:04 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:02:04.297 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:02:04 smithi047 conmon[78521]: 2022-05-14T00:02:04.209+0000 7f5340f7a700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T00:02:04.297 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:02:04 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T00:02:04.297 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:02:04 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:02:04.297 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:02:04 smithi047 conmon[78521]: yield 2022-05-14T00:02:04.298 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:02:04 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:02:04.298 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:02:04 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:02:04.298 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:02:04 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:02:04.298 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:02:04 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T00:02:04.298 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:02:04 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:02:04.299 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:02:04 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T00:02:04.299 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:02:04 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:02:04.299 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:02:04 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T00:02:04.299 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:02:04 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:02:04.300 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:02:04 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T00:02:04.300 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:02:04 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:02:04.300 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:02:04 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T00:02:04.300 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:02:04 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:02:04.300 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:02:04 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:02:04.301 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:02:04 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:02:04.301 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:02:04 smithi047 conmon[78521]: 2022-05-14T00:02:04.301 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:02:04 smithi047 conmon[78521]: During handling of the above exception, another exception occurred: 2022-05-14T00:02:04.301 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:02:04 smithi047 conmon[78521]: 2022-05-14T00:02:04.302 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:02:04 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T00:02:04.302 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:02:04 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T00:02:04.302 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:02:04 smithi047 conmon[78521]: return f(*arg) 2022-05-14T00:02:04.303 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:02:04 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T00:02:04.303 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:02:04 smithi047 conmon[78521]: self._write_client_files(client_files, host) 2022-05-14T00:02:04.303 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:02:04 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T00:02:04.303 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:02:04 smithi047 conmon[78521]: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T00:02:04.303 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:02:04 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T00:02:04.304 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:02:04 smithi047 conmon[78521]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T00:02:04.304 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:02:04 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T00:02:04.304 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:02:04 smithi047 conmon[78521]: return self.event_loop.get_result(coro) 2022-05-14T00:02:04.305 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:02:04 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T00:02:04.305 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:02:04 smithi047 conmon[78521]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T00:02:04.305 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:02:04 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T00:02:04.305 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:02:04 smithi047 conmon[78521]: return self.__get_result() 2022-05-14T00:02:04.306 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:02:04 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T00:02:04.306 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:02:04 smithi047 conmon[78521]: raise self._exception 2022-05-14T00:02:04.306 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:02:04 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T00:02:04.306 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:02:04 smithi047 conmon[78521]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T00:02:04.307 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:02:04 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T00:02:04.307 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:02:04 smithi047 conmon[78521]: conn = await self._remote_connection(host, addr) 2022-05-14T00:02:04.307 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:02:04 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T00:02:04.307 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:02:04 smithi047 conmon[78521]: raise 2022-05-14T00:02:04.307 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:02:04 smithi047 conmon[78521]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T00:02:04.308 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:02:04 smithi047 conmon[78521]: self.gen.throw(type, value, traceback) 2022-05-14T00:02:04.308 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:02:04 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T00:02:04.308 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:02:04 smithi047 conmon[78521]: raise OrchestratorError(msg) 2022-05-14T00:02:04.308 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:02:04 smithi047 conmon[78521]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:02:04.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:02:04 smithi115 conmon[90393]: 2022-05-14T00:02:04.081+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:02:04.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:02:04 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:02:04.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:02:04 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 2056 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:02:05.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:02:05 smithi047 ceph-mon[77147]: pgmap v1381: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:02:05.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:02:05 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:02:05.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:02:05 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:02:05.292 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T00:02:05.292 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:02:05.292 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T00:02:05.292 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:02:05.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:02:05.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:02:05.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T00:02:05.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:02:05.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T00:02:05.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:02:05.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T00:02:05.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:02:05.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T00:02:05.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:02:05.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T00:02:05.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:02:05.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:02:05.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:02:05.296 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:02:05 smithi047 ceph-mon[77147]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T00:02:05.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T00:02:05.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:02:05.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T00:02:05.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:02:05.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:02:05.298 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:02:05.298 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T00:02:05.298 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:02:05.298 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T00:02:05.298 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:02:05.299 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T00:02:05.299 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:02:05.299 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T00:02:05.299 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:02:05.299 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T00:02:05.300 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:02:05.300 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:02:05.300 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:02:05.300 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T00:02:05.300 INFO:journalctl@ceph.mon.b.smithi047.stdout: During handling of the above exception, another exception occurred: 2022-05-14T00:02:05.301 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T00:02:05.301 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T00:02:05.301 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T00:02:05.301 INFO:journalctl@ceph.mon.b.smithi047.stdout: return f(*arg) 2022-05-14T00:02:05.302 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T00:02:05.302 INFO:journalctl@ceph.mon.b.smithi047.stdout: self._write_client_files(client_files, host) 2022-05-14T00:02:05.302 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T00:02:05.302 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T00:02:05.302 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T00:02:05.303 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T00:02:05.303 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T00:02:05.303 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.event_loop.get_result(coro) 2022-05-14T00:02:05.303 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T00:02:05.303 INFO:journalctl@ceph.mon.b.smithi047.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T00:02:05.304 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T00:02:05.304 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.__get_result() 2022-05-14T00:02:05.304 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T00:02:05.304 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise self._exception 2022-05-14T00:02:05.305 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T00:02:05.305 INFO:journalctl@ceph.mon.b.smithi047.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T00:02:05.305 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T00:02:05.306 INFO:journalctl@ceph.mon.b.smithi047.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T00:02:05.306 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T00:02:05.306 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise 2022-05-14T00:02:05.306 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T00:02:05.307 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.gen.throw(type, value, traceback) 2022-05-14T00:02:05.307 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T00:02:05.307 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OrchestratorError(msg) 2022-05-14T00:02:05.307 INFO:journalctl@ceph.mon.b.smithi047.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:02:05.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:02:05 smithi115 conmon[90393]: 2022-05-14T00:02:05.110+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:02:05.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:02:05 smithi115 ceph-mon[78593]: pgmap v1381: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:02:05.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:02:05 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:02:05.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:02:05 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:02:05.359 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T00:02:05.359 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:02:05.359 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T00:02:05.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:02:05.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:02:05.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:02:05.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T00:02:05.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:02:05.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T00:02:05.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:02:05.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T00:02:05.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:02:05.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T00:02:05.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:02:05.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T00:02:05.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:02:05.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:02:05.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:02:05.363 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:02:05 smithi115 ceph-mon[78593]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T00:02:05.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T00:02:05.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:02:05.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T00:02:05.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:02:05.366 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:02:05.366 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:02:05.366 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T00:02:05.366 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:02:05.367 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T00:02:05.367 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:02:05.367 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T00:02:05.367 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:02:05.368 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T00:02:05.368 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:02:05.368 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T00:02:05.368 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:02:05.369 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:02:05.369 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:02:05.369 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T00:02:05.369 INFO:journalctl@ceph.mon.c.smithi115.stdout: During handling of the above exception, another exception occurred: 2022-05-14T00:02:05.369 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T00:02:05.370 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T00:02:05.370 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T00:02:05.370 INFO:journalctl@ceph.mon.c.smithi115.stdout: return f(*arg) 2022-05-14T00:02:05.370 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T00:02:05.370 INFO:journalctl@ceph.mon.c.smithi115.stdout: self._write_client_files(client_files, host) 2022-05-14T00:02:05.371 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T00:02:05.371 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T00:02:05.371 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T00:02:05.371 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T00:02:05.371 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T00:02:05.372 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.event_loop.get_result(coro) 2022-05-14T00:02:05.372 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T00:02:05.372 INFO:journalctl@ceph.mon.c.smithi115.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T00:02:05.372 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T00:02:05.373 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.__get_result() 2022-05-14T00:02:05.373 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T00:02:05.373 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise self._exception 2022-05-14T00:02:05.373 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T00:02:05.374 INFO:journalctl@ceph.mon.c.smithi115.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T00:02:05.374 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T00:02:05.374 INFO:journalctl@ceph.mon.c.smithi115.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T00:02:05.375 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T00:02:05.375 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise 2022-05-14T00:02:05.375 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T00:02:05.375 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.gen.throw(type, value, traceback) 2022-05-14T00:02:05.376 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T00:02:05.376 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OrchestratorError(msg) 2022-05-14T00:02:05.377 INFO:journalctl@ceph.mon.c.smithi115.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:02:06.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:02:06 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:02:06.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:02:06 smithi115 conmon[90393]: 2022-05-14T00:02:06.098+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:02:06.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:02:06 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:02:07.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:02:07 smithi047 ceph-mon[77147]: pgmap v1382: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:02:07.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:02:07 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:02:07.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:02:07 smithi115 conmon[90393]: 2022-05-14T00:02:07.113+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:02:07.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:02:07 smithi115 ceph-mon[78593]: pgmap v1382: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:02:07.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:02:07 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:02:08.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:02:08 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:02:08.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:02:08 smithi115 conmon[90393]: 2022-05-14T00:02:08.084+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:02:08.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:02:08 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:02:09.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:02:09 smithi047 ceph-mon[77147]: pgmap v1383: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:02:09.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:02:09 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:02:09.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:02:09 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 2060 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:02:09.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:02:09 smithi115 conmon[90393]: 2022-05-14T00:02:09.064+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:02:09.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:02:09 smithi115 ceph-mon[78593]: pgmap v1383: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:02:09.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:02:09 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:02:09.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:02:09 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 2060 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:02:10.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:02:10 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:02:10.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:02:10 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:02:10.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:02:10 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:02:10.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:02:10 smithi115 conmon[90393]: 2022-05-14T00:02:10.104+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:02:10.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:02:10 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:02:10.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:02:10 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:02:10.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:02:10 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:02:11.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:02:11 smithi047 ceph-mon[77147]: pgmap v1384: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:02:11.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:02:11 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:02:11.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:02:11 smithi115 conmon[90393]: 2022-05-14T00:02:11.105+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:02:11.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:02:11 smithi115 ceph-mon[78593]: pgmap v1384: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:02:11.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:02:11 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:02:12.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:02:12 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:02:12.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:02:12 smithi115 conmon[90393]: 2022-05-14T00:02:12.097+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:02:12.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:02:12 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:02:13.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:02:13 smithi115 ceph-mon[78593]: pgmap v1385: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:02:13.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:02:13 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:02:13.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:02:13 smithi115 conmon[90393]: 2022-05-14T00:02:13.119+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:02:13.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:02:13 smithi047 ceph-mon[77147]: pgmap v1385: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:02:13.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:02:13 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:02:14.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:02:14 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:02:14.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:02:14 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:02:14.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:02:14 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:02:14.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:02:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T00:02:14.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:02:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T00:02:14.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:02:14 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 2065 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:02:14.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:02:14 smithi115 conmon[90393]: 2022-05-14T00:02:14.131+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:02:14.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:02:14 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:02:14.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:02:14 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:02:14.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:02:14 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:02:14.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:02:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T00:02:14.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:02:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T00:02:14.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:02:14 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 2065 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:02:15.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:02:15 smithi115 ceph-mon[78593]: pgmap v1386: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:02:15.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:02:15 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:02:15.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:02:15 smithi115 conmon[90393]: 2022-05-14T00:02:15.169+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:02:15.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:02:15 smithi047 ceph-mon[77147]: pgmap v1386: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:02:15.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:02:15 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:02:16.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:02:16 smithi115 conmon[90393]: 2022-05-14T00:02:16.161+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:02:16.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:02:16 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:02:16.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:02:16 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:02:17.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:02:17 smithi115 conmon[90393]: 2022-05-14T00:02:17.167+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:02:17.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:02:17 smithi115 ceph-mon[78593]: pgmap v1387: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:02:17.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:02:17 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:02:17.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:02:17 smithi047 ceph-mon[77147]: pgmap v1387: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:02:17.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:02:17 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:02:18.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:02:18 smithi115 conmon[90393]: 2022-05-14T00:02:18.188+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:02:18.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:02:18 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:02:18.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:02:18 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:02:18.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:02:18 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:02:18.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:02:18 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:02:18.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:02:18 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:02:18.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:02:18 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:02:19.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:02:19 smithi115 conmon[90393]: 2022-05-14T00:02:19.197+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:02:19.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:02:19 smithi115 ceph-mon[78593]: pgmap v1388: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:02:19.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:02:19 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:02:19.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:02:19 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 2070 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:02:19.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:02:19 smithi047 ceph-mon[77147]: pgmap v1388: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:02:19.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:02:19 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:02:19.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:02:19 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 2070 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:02:20.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:02:20 smithi115 conmon[90393]: 2022-05-14T00:02:20.177+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:02:20.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:02:20 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:02:20.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:02:20 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:02:21.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:02:21 smithi115 conmon[90393]: 2022-05-14T00:02:21.182+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:02:21.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:02:21 smithi115 ceph-mon[78593]: pgmap v1389: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:02:21.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:02:21 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:02:21.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:02:21 smithi047 ceph-mon[77147]: pgmap v1389: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:02:21.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:02:21 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:02:22.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:02:22 smithi115 conmon[90393]: 2022-05-14T00:02:22.167+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:02:22.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:02:22 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:02:22.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:02:22 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:02:22.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:02:22 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:02:22.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:02:22 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:02:22.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:02:22 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:02:22.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:02:22 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:02:23.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:02:23 smithi115 conmon[90393]: 2022-05-14T00:02:23.144+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:02:23.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:02:23 smithi115 ceph-mon[78593]: pgmap v1390: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:02:23.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:02:23 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:02:23.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:02:23 smithi047 ceph-mon[77147]: pgmap v1390: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:02:23.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:02:23 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:02:24.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:02:24 smithi115 conmon[90393]: 2022-05-14T00:02:24.117+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:02:24.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:02:24 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:02:24.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:02:24 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 2075 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:02:24.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:02:24 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:02:24.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:02:24 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 2075 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:02:25.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:02:25 smithi115 conmon[90393]: 2022-05-14T00:02:25.129+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:02:25.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:02:25 smithi115 ceph-mon[78593]: pgmap v1391: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:02:25.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:02:25 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:02:25.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:02:25 smithi047 ceph-mon[77147]: pgmap v1391: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:02:25.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:02:25 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:02:26.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:02:26 smithi115 conmon[90393]: 2022-05-14T00:02:26.172+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:02:26.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:02:26 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:02:26.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:02:26 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:02:27.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:02:27 smithi115 conmon[90393]: 2022-05-14T00:02:27.152+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:02:27.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:02:27 smithi115 ceph-mon[78593]: pgmap v1392: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:02:27.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:02:27 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:02:27.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:02:27 smithi047 ceph-mon[77147]: pgmap v1392: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:02:27.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:02:27 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:02:28.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:02:28 smithi115 conmon[90393]: 2022-05-14T00:02:28.159+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:02:28.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:02:28 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:02:28.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:02:28 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:02:29.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:02:29 smithi115 conmon[90393]: 2022-05-14T00:02:29.136+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:02:29.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:02:29 smithi115 ceph-mon[78593]: pgmap v1393: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:02:29.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:02:29 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:02:29.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:02:29 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 2080 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:02:29.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:02:29 smithi047 ceph-mon[77147]: pgmap v1393: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:02:29.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:02:29 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:02:29.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:02:29 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 2080 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:02:30.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:02:30 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:02:30.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:02:30 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:02:30.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:02:30 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:02:30.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:02:30 smithi115 conmon[90393]: 2022-05-14T00:02:30.126+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:02:30.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:02:30 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:02:30.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:02:30 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:02:30.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:02:30 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:02:31.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:02:31 smithi115 conmon[90393]: 2022-05-14T00:02:31.102+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:02:31.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:02:31 smithi115 ceph-mon[78593]: pgmap v1394: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:02:31.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:02:31 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:02:31.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:02:31 smithi047 ceph-mon[77147]: pgmap v1394: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:02:31.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:02:31 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:02:32.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:02:32 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:02:32.617 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:02:32 smithi115 conmon[90393]: 2022-05-14T00:02:32.125+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:02:32.618 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:02:32 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:02:33.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:02:33 smithi115 conmon[90393]: 2022-05-14T00:02:33.092+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:02:33.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:02:33 smithi115 ceph-mon[78593]: pgmap v1395: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:02:33.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:02:33 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:02:33.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:02:33 smithi047 ceph-mon[77147]: pgmap v1395: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:02:33.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:02:33 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:02:34.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:02:34 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:02:34.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:02:34 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:02:34.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:02:34 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:02:34.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:02:34 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 2085 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:02:34.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:02:34 smithi115 conmon[90393]: 2022-05-14T00:02:34.129+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:02:34.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:02:34 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:02:34.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:02:34 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:02:34.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:02:34 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:02:34.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:02:34 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 2085 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:02:35.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:02:35 smithi047 ceph-mon[77147]: pgmap v1396: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:02:35.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:02:35 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:02:35.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:02:35 smithi115 conmon[90393]: 2022-05-14T00:02:35.170+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:02:35.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:02:35 smithi115 ceph-mon[78593]: pgmap v1396: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:02:35.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:02:35 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:02:36.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:02:36 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:02:36.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:02:36 smithi115 conmon[90393]: 2022-05-14T00:02:36.164+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:02:36.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:02:36 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:02:37.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:02:37 smithi047 ceph-mon[77147]: pgmap v1397: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:02:37.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:02:37 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:02:37.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:02:37 smithi115 conmon[90393]: 2022-05-14T00:02:37.134+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:02:37.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:02:37 smithi115 ceph-mon[78593]: pgmap v1397: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:02:37.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:02:37 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:02:38.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:02:38 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:02:38.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:02:38 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:02:38.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:02:38 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:02:38.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:02:38 smithi115 conmon[90393]: 2022-05-14T00:02:38.126+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:02:38.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:02:38 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:02:38.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:02:38 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:02:38.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:02:38 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:02:39.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:02:39 smithi047 ceph-mon[77147]: pgmap v1398: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:02:39.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:02:39 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:02:39.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:02:39 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 2090 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:02:39.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:02:39 smithi115 conmon[90393]: 2022-05-14T00:02:39.118+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:02:39.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:02:39 smithi115 ceph-mon[78593]: pgmap v1398: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:02:39.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:02:39 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:02:39.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:02:39 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 2090 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:02:40.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:02:40 smithi115 conmon[90393]: 2022-05-14T00:02:40.082+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:02:40.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:02:40 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:02:40.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:02:40 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:02:41.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:02:41 smithi115 conmon[90393]: 2022-05-14T00:02:41.066+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:02:41.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:02:41 smithi115 ceph-mon[78593]: pgmap v1399: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:02:41.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:02:41 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:02:41.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:02:41 smithi047 ceph-mon[77147]: pgmap v1399: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:02:41.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:02:41 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:02:42.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:02:42 smithi115 conmon[90393]: 2022-05-14T00:02:42.025+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:02:42.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:02:42 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:02:42.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:02:42 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:02:42.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:02:42 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:02:42.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:02:42 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:02:42.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:02:42 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:02:42.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:02:42 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:02:43.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:02:42 smithi115 conmon[90393]: 2022-05-14T00:02:42.979+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:02:43.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:02:43 smithi115 ceph-mon[78593]: pgmap v1400: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:02:43.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:02:43 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:02:43.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:02:43 smithi047 ceph-mon[77147]: pgmap v1400: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:02:43.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:02:43 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:02:44.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:02:43 smithi115 conmon[90393]: 2022-05-14T00:02:43.937+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:02:44.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:02:44 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:02:44.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:02:44 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 2095 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:02:44.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:02:44 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:02:44.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:02:44 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 2095 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:02:45.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:02:44 smithi115 conmon[90393]: 2022-05-14T00:02:44.933+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:02:45.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:02:45 smithi115 ceph-mon[78593]: pgmap v1401: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:02:45.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:02:45 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:02:45.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:02:45 smithi047 ceph-mon[77147]: pgmap v1401: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:02:45.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:02:45 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:02:46.171 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:02:45 smithi115 conmon[90393]: 2022-05-14T00:02:45.910+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:02:46.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:02:46 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:02:46.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:02:46 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:02:47.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:02:46 smithi115 conmon[90393]: 2022-05-14T00:02:46.936+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:02:47.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:02:47 smithi115 ceph-mon[78593]: pgmap v1402: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:02:47.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:02:47 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:02:47.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:02:47 smithi047 ceph-mon[77147]: pgmap v1402: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:02:47.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:02:47 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:02:48.181 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:02:47 smithi115 conmon[90393]: 2022-05-14T00:02:47.921+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:02:48.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:02:48 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:02:48.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:02:48 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:02:49.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:02:48 smithi115 conmon[90393]: 2022-05-14T00:02:48.940+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:02:49.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:02:49 smithi115 ceph-mon[78593]: pgmap v1403: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:02:49.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:02:49 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:02:49.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:02:49 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 2100 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:02:49.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:02:49 smithi047 ceph-mon[77147]: pgmap v1403: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:02:49.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:02:49 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:02:49.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:02:49 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 2100 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:02:50.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:02:49 smithi115 conmon[90393]: 2022-05-14T00:02:49.981+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:02:50.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:02:50 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:02:50.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:02:50 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:02:50.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:02:50 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:02:50.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:02:50 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:02:50.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:02:50 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:02:50.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:02:50 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:02:51.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:02:51 smithi115 conmon[90393]: 2022-05-14T00:02:51.002+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:02:51.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:02:51 smithi115 ceph-mon[78593]: pgmap v1404: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:02:51.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:02:51 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:02:51.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:02:51 smithi047 ceph-mon[77147]: pgmap v1404: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:02:51.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:02:51 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:02:52.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:02:52 smithi115 conmon[90393]: 2022-05-14T00:02:52.041+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:02:52.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:02:52 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:02:52.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:02:52 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:02:53.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:02:53 smithi115 conmon[90393]: 2022-05-14T00:02:53.090+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:02:53.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:02:53 smithi115 ceph-mon[78593]: pgmap v1405: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:02:53.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:02:53 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:02:53.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:02:53 smithi047 ceph-mon[77147]: pgmap v1405: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:02:53.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:02:53 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:02:54.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:02:54 smithi115 conmon[90393]: 2022-05-14T00:02:54.083+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:02:54.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:02:54 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:02:54.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:02:54 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:02:54.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:02:54 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:02:54.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:02:54 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 2105 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:02:54.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:02:54 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:02:54.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:02:54 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:02:54.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:02:54 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:02:54.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:02:54 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 2105 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:02:55.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:02:55 smithi115 conmon[90393]: 2022-05-14T00:02:55.063+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:02:55.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:02:55 smithi115 ceph-mon[78593]: pgmap v1406: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:02:55.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:02:55 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:02:55.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:02:55 smithi047 ceph-mon[77147]: pgmap v1406: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:02:55.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:02:55 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:02:56.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:02:56 smithi115 conmon[90393]: 2022-05-14T00:02:56.065+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:02:56.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:02:56 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:02:56.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:02:56 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:02:57.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:02:57 smithi115 conmon[90393]: 2022-05-14T00:02:57.064+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:02:57.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:02:57 smithi115 ceph-mon[78593]: pgmap v1407: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:02:57.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:02:57 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:02:57.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:02:57 smithi047 ceph-mon[77147]: pgmap v1407: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:02:57.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:02:57 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:02:58.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:02:58 smithi115 conmon[90393]: 2022-05-14T00:02:58.071+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:02:58.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:02:58 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:02:58.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:02:58 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:02:58.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:02:58 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:02:58.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:02:58 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:02:58.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:02:58 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:02:58.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:02:58 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:02:59.232 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:02:59 smithi047 ceph-mon[77147]: pgmap v1408: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:02:59.233 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:02:59 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:02:59.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:02:59 smithi115 conmon[90393]: 2022-05-14T00:02:59.084+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:02:59.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:02:59 smithi115 ceph-mon[78593]: pgmap v1408: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:02:59.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:02:59 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:02:59.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:02:59 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 2110 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:02:59.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:02:59 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 2110 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:03:00.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:03:00 smithi115 conmon[90393]: 2022-05-14T00:03:00.068+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:03:00.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:03:00 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:03:00.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:03:00 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:03:01.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:03:01 smithi115 conmon[90393]: 2022-05-14T00:03:01.072+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:03:01.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:03:01 smithi115 ceph-mon[78593]: pgmap v1409: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:03:01.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:03:01 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:03:01.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:03:01 smithi047 ceph-mon[77147]: pgmap v1409: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:03:01.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:03:01 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:03:02.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:03:02 smithi115 conmon[90393]: 2022-05-14T00:03:02.080+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:03:02.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:03:02 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:03:02.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:03:02 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:03:02.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:03:02 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:03:02.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:03:02 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:03:02.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:03:02 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:03:02.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:03:02 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:03:03.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:03:03 smithi047 ceph-mon[77147]: pgmap v1410: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:03:03.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:03:03 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:03:03.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:03:03 smithi115 conmon[90393]: 2022-05-14T00:03:03.130+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:03:03.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:03:03 smithi115 ceph-mon[78593]: pgmap v1410: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:03:03.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:03:03 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:03:04.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:03:04 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:03:04.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:03:04 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 2115 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:03:04.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:03:04 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T00:03:04.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:03:04 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T00:03:04.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:03:04 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T00:03:04.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:03:04 smithi115 conmon[90393]: 2022-05-14T00:03:04.107+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:03:04.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:03:04 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:03:04.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:03:04 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 2115 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:03:04.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:03:04 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T00:03:04.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:03:04 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T00:03:04.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:03:04 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T00:03:05.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:03:05 smithi115 ceph-mon[78593]: pgmap v1411: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:03:05.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:03:05 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:03:05.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:03:05 smithi115 conmon[90393]: 2022-05-14T00:03:05.093+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:03:05.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:03:05 smithi047 ceph-mon[77147]: pgmap v1411: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:03:05.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:03:05 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:03:06.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:03:06 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:03:06.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:03:06 smithi115 conmon[90393]: 2022-05-14T00:03:06.056+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:03:06.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:03:06 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:03:07.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:03:07 smithi115 conmon[90393]: 2022-05-14T00:03:07.097+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:03:07.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:03:07 smithi115 ceph-mon[78593]: pgmap v1412: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:03:07.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:03:07 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:03:07.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:03:07 smithi047 ceph-mon[77147]: pgmap v1412: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:03:07.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:03:07 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:03:07.542 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:03:07 smithi047 conmon[78521]: 2022-05-14T00:03:07.312+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:03:07.542 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:03:07 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T00:03:07.542 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:03:07 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:03:07.542 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:03:07 smithi047 conmon[78521]: yield 2022-05-14T00:03:07.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:03:07 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:03:07.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:03:07 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:03:07.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:03:07 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:03:07.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:03:07 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T00:03:07.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:03:07 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:03:07.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:03:07 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T00:03:07.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:03:07 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:03:07.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:03:07 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T00:03:07.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:03:07 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:03:07.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:03:07 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T00:03:07.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:03:07 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:03:07.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:03:07 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T00:03:07.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:03:07 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:03:07.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:03:07 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:03:07.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:03:07 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:03:08.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:03:08 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:03:08.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:03:08 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:03:08.542 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T00:03:08.542 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:03:08.542 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T00:03:08.542 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:03:08.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:03:08.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:03:08.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T00:03:08.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:03:08.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T00:03:08.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:03:08.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T00:03:08.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:03:08.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T00:03:08.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:03:08.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T00:03:08.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:03:08.549 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:03:08.550 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:03:08.550 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:03:08 smithi047 ceph-mon[77147]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T00:03:08.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:03:08 smithi115 conmon[90393]: 2022-05-14T00:03:08.143+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:03:08.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:03:08 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:03:08.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:03:08 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:03:08.609 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T00:03:08.609 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:03:08.609 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T00:03:08.609 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:03:08.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:03:08.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:03:08.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T00:03:08.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:03:08.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T00:03:08.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:03:08.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T00:03:08.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:03:08.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T00:03:08.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:03:08.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T00:03:08.613 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:03:08.613 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:03:08.613 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:03:08.614 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:03:08 smithi115 ceph-mon[78593]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T00:03:09.367 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:03:09 smithi115 conmon[90393]: 2022-05-14T00:03:09.103+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:03:09.367 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:03:09 smithi115 ceph-mon[78593]: pgmap v1413: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:03:09.367 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:03:09 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:03:09.368 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:03:09 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 2121 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:03:09.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:03:09 smithi047 ceph-mon[77147]: pgmap v1413: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:03:09.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:03:09 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:03:09.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:03:09 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 2121 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:03:10.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:03:10 smithi115 conmon[90393]: 2022-05-14T00:03:10.094+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:03:10.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:03:10 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:03:10.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:03:10 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:03:10.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:03:10 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:03:10.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:03:10 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:03:10.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:03:10 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:03:10.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:03:10 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:03:10.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:03:10 smithi047 conmon[78521]: 2022-05-14T00:03:10.385+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:03:10.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:03:10 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T00:03:10.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:03:10 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:03:10.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:03:10 smithi047 conmon[78521]: yield 2022-05-14T00:03:10.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:03:10 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:03:10.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:03:10 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:03:10.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:03:10 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:03:10.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:03:10 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T00:03:10.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:03:10 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:03:10.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:03:10 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T00:03:10.547 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:03:10 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:03:10.547 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:03:10 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T00:03:10.547 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:03:10 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:03:10.548 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:03:10 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T00:03:10.549 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:03:10 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:03:10.550 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:03:10 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T00:03:10.550 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:03:10 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:03:10.550 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:03:10 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:03:10.551 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:03:10 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:03:10.551 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:03:10 smithi047 conmon[78521]: 2022-05-14T00:03:10.385+0000 7f5344781700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T00:03:10.552 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:03:10 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T00:03:10.553 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:03:10 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:03:10.554 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:03:10 smithi047 conmon[78521]: yield 2022-05-14T00:03:10.554 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:03:10 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:03:10.555 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:03:10 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:03:10.555 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:03:10 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:03:10.556 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:03:10 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T00:03:10.556 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:03:10 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:03:10.556 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:03:10 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T00:03:10.557 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:03:10 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:03:10.558 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:03:10 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T00:03:10.558 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:03:10 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:03:10.558 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:03:10 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T00:03:10.559 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:03:10 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:03:10.559 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:03:10 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T00:03:10.560 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:03:10 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:03:10.560 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:03:10 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:03:10.560 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:03:10 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:03:10.561 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:03:10 smithi047 conmon[78521]: 2022-05-14T00:03:10.561 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:03:10 smithi047 conmon[78521]: During handling of the above exception, another exception occurred: 2022-05-14T00:03:10.563 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:03:10 smithi047 conmon[78521]: 2022-05-14T00:03:10.563 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:03:10 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T00:03:10.563 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:03:10 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T00:03:10.564 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:03:10 smithi047 conmon[78521]: return f(*arg) 2022-05-14T00:03:10.564 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:03:10 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T00:03:10.564 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:03:10 smithi047 conmon[78521]: self._write_client_files(client_files, host) 2022-05-14T00:03:10.564 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:03:10 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T00:03:10.564 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:03:10 smithi047 conmon[78521]: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T00:03:10.565 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:03:10 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T00:03:10.565 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:03:10 smithi047 conmon[78521]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T00:03:10.565 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:03:10 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T00:03:10.566 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:03:10 smithi047 conmon[78521]: return self.event_loop.get_result(coro) 2022-05-14T00:03:10.566 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:03:10 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T00:03:10.566 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:03:10 smithi047 conmon[78521]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T00:03:10.567 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:03:10 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T00:03:10.567 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:03:10 smithi047 conmon[78521]: return self.__get_result() 2022-05-14T00:03:10.567 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:03:10 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T00:03:10.568 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:03:10 smithi047 conmon[78521]: raise self._exception 2022-05-14T00:03:10.568 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:03:10 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T00:03:10.568 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:03:10 smithi047 conmon[78521]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T00:03:10.568 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:03:10 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T00:03:10.569 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:03:10 smithi047 conmon[78521]: conn = await self._remote_connection(host, addr) 2022-05-14T00:03:10.569 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:03:10 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T00:03:10.569 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:03:10 smithi047 conmon[78521]: raise 2022-05-14T00:03:10.569 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:03:10 smithi047 conmon[78521]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T00:03:10.570 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:03:10 smithi047 conmon[78521]: self.gen.throw(type, value, traceback) 2022-05-14T00:03:10.570 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:03:10 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T00:03:10.570 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:03:10 smithi047 conmon[78521]: raise OrchestratorError(msg) 2022-05-14T00:03:10.570 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:03:10 smithi047 conmon[78521]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:03:11.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:03:11 smithi115 conmon[90393]: 2022-05-14T00:03:11.052+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:03:11.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:03:11 smithi115 ceph-mon[78593]: pgmap v1414: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:03:11.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:03:11 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:03:11.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:03:11 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:03:11.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T00:03:11.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:03:11.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T00:03:11.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:03:11.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:03:11.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:03:11.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T00:03:11.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:03:11.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T00:03:11.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:03:11.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T00:03:11.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:03:11.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T00:03:11.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:03:11.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T00:03:11.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:03:11.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:03:11.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:03:11.363 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:03:11 smithi115 ceph-mon[78593]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T00:03:11.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T00:03:11.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:03:11.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T00:03:11.365 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:03:11.365 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:03:11.366 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:03:11.366 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T00:03:11.366 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:03:11.366 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T00:03:11.366 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:03:11.367 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T00:03:11.367 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:03:11.367 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T00:03:11.367 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:03:11.368 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T00:03:11.368 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:03:11.368 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:03:11.368 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:03:11.369 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T00:03:11.369 INFO:journalctl@ceph.mon.c.smithi115.stdout: During handling of the above exception, another exception occurred: 2022-05-14T00:03:11.370 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T00:03:11.371 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T00:03:11.371 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T00:03:11.371 INFO:journalctl@ceph.mon.c.smithi115.stdout: return f(*arg) 2022-05-14T00:03:11.372 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T00:03:11.372 INFO:journalctl@ceph.mon.c.smithi115.stdout: self._write_client_files(client_files, host) 2022-05-14T00:03:11.372 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T00:03:11.372 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T00:03:11.373 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T00:03:11.373 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T00:03:11.373 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T00:03:11.373 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.event_loop.get_result(coro) 2022-05-14T00:03:11.374 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T00:03:11.374 INFO:journalctl@ceph.mon.c.smithi115.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T00:03:11.374 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T00:03:11.374 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.__get_result() 2022-05-14T00:03:11.374 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T00:03:11.375 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise self._exception 2022-05-14T00:03:11.375 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T00:03:11.375 INFO:journalctl@ceph.mon.c.smithi115.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T00:03:11.376 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T00:03:11.376 INFO:journalctl@ceph.mon.c.smithi115.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T00:03:11.376 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T00:03:11.376 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise 2022-05-14T00:03:11.377 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T00:03:11.377 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.gen.throw(type, value, traceback) 2022-05-14T00:03:11.377 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T00:03:11.377 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OrchestratorError(msg) 2022-05-14T00:03:11.378 INFO:journalctl@ceph.mon.c.smithi115.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:03:11.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:03:11 smithi047 ceph-mon[77147]: pgmap v1414: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:03:11.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:03:11 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:03:11.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:03:11 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:03:11.542 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T00:03:11.542 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:03:11.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T00:03:11.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:03:11.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:03:11.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:03:11.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T00:03:11.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:03:11.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T00:03:11.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:03:11.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T00:03:11.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:03:11.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T00:03:11.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:03:11.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T00:03:11.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:03:11.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:03:11.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:03:11.546 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:03:11 smithi047 ceph-mon[77147]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T00:03:11.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T00:03:11.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:03:11.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T00:03:11.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:03:11.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:03:11.548 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:03:11.548 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T00:03:11.548 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:03:11.549 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T00:03:11.549 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:03:11.549 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T00:03:11.549 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:03:11.550 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T00:03:11.550 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:03:11.550 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T00:03:11.550 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:03:11.550 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:03:11.551 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:03:11.551 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T00:03:11.551 INFO:journalctl@ceph.mon.b.smithi047.stdout: During handling of the above exception, another exception occurred: 2022-05-14T00:03:11.551 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T00:03:11.551 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T00:03:11.552 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T00:03:11.552 INFO:journalctl@ceph.mon.b.smithi047.stdout: return f(*arg) 2022-05-14T00:03:11.552 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T00:03:11.552 INFO:journalctl@ceph.mon.b.smithi047.stdout: self._write_client_files(client_files, host) 2022-05-14T00:03:11.553 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T00:03:11.553 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T00:03:11.554 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T00:03:11.554 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T00:03:11.554 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T00:03:11.554 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.event_loop.get_result(coro) 2022-05-14T00:03:11.554 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T00:03:11.555 INFO:journalctl@ceph.mon.b.smithi047.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T00:03:11.555 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T00:03:11.555 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.__get_result() 2022-05-14T00:03:11.556 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T00:03:11.556 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise self._exception 2022-05-14T00:03:11.556 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T00:03:11.556 INFO:journalctl@ceph.mon.b.smithi047.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T00:03:11.557 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T00:03:11.557 INFO:journalctl@ceph.mon.b.smithi047.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T00:03:11.557 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T00:03:11.557 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise 2022-05-14T00:03:11.558 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T00:03:11.558 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.gen.throw(type, value, traceback) 2022-05-14T00:03:11.558 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T00:03:11.558 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OrchestratorError(msg) 2022-05-14T00:03:11.559 INFO:journalctl@ceph.mon.b.smithi047.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:03:12.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:03:12 smithi115 conmon[90393]: 2022-05-14T00:03:12.049+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:03:12.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:03:12 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:03:12.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:03:12 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:03:13.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:03:13 smithi115 conmon[90393]: 2022-05-14T00:03:13.066+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:03:13.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:03:13 smithi115 ceph-mon[78593]: pgmap v1415: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:03:13.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:03:13 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:03:13.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:03:13 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T00:03:13.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:03:13 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T00:03:13.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:03:13 smithi047 ceph-mon[77147]: pgmap v1415: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:03:13.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:03:13 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:03:13.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:03:13 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T00:03:13.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:03:13 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T00:03:14.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:03:14 smithi115 conmon[90393]: 2022-05-14T00:03:14.085+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:03:14.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:03:14 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:03:14.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:03:14 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:03:14.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:03:14 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:03:14.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:03:14 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 2126 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:03:14.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:03:14 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:03:14.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:03:14 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:03:14.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:03:14 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:03:14.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:03:14 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 2126 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:03:15.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:03:15 smithi047 ceph-mon[77147]: pgmap v1416: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:03:15.582 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:03:15 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:03:15.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:03:15 smithi115 conmon[90393]: 2022-05-14T00:03:15.109+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:03:15.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:03:15 smithi115 ceph-mon[78593]: pgmap v1416: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:03:15.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:03:15 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:03:16.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:03:16 smithi115 conmon[90393]: 2022-05-14T00:03:16.090+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:03:16.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:03:16 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:03:16.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:03:16 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:03:17.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:03:17 smithi115 conmon[90393]: 2022-05-14T00:03:17.077+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:03:17.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:03:17 smithi115 ceph-mon[78593]: pgmap v1417: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:03:17.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:03:17 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:03:17.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:03:17 smithi047 ceph-mon[77147]: pgmap v1417: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:03:17.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:03:17 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:03:18.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:03:18 smithi115 conmon[90393]: 2022-05-14T00:03:18.100+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:03:18.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:03:18 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:03:18.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:03:18 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:03:18.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:03:18 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:03:18.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:03:18 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:03:18.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:03:18 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:03:18.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:03:18 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:03:19.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:03:19 smithi115 conmon[90393]: 2022-05-14T00:03:19.103+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:03:19.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:03:19 smithi115 ceph-mon[78593]: pgmap v1418: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:03:19.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:03:19 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:03:19.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:03:19 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 2130 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:03:19.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:03:19 smithi047 ceph-mon[77147]: pgmap v1418: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:03:19.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:03:19 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:03:19.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:03:19 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 2130 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:03:20.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:03:20 smithi115 conmon[90393]: 2022-05-14T00:03:20.089+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:03:20.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:03:20 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:03:20.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:03:20 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:03:21.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:03:21 smithi115 conmon[90393]: 2022-05-14T00:03:21.064+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:03:21.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:03:21 smithi115 ceph-mon[78593]: pgmap v1419: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:03:21.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:03:21 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:03:21.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:03:21 smithi047 ceph-mon[77147]: pgmap v1419: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:03:21.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:03:21 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:03:22.321 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:03:22 smithi115 conmon[90393]: 2022-05-14T00:03:22.028+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:03:22.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:03:22 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:03:22.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:03:22 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:03:22.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:03:22 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:03:22.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:03:22 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:03:22.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:03:22 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:03:22.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:03:22 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:03:23.323 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:03:22 smithi115 conmon[90393]: 2022-05-14T00:03:22.984+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:03:23.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:03:23 smithi115 ceph-mon[78593]: pgmap v1420: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:03:23.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:03:23 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:03:23.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:03:23 smithi047 ceph-mon[77147]: pgmap v1420: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:03:23.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:03:23 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:03:24.324 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:03:24 smithi115 conmon[90393]: 2022-05-14T00:03:24.026+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:03:24.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:03:24 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:03:24.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:03:24 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 2136 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:03:24.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:03:24 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:03:24.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:03:24 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 2136 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:03:25.325 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:03:24 smithi115 conmon[90393]: 2022-05-14T00:03:24.998+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:03:25.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:03:25 smithi115 ceph-mon[78593]: pgmap v1421: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:03:25.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:03:25 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:03:25.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:03:25 smithi047 ceph-mon[77147]: pgmap v1421: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:03:25.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:03:25 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:03:26.330 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:03:25 smithi115 conmon[90393]: 2022-05-14T00:03:25.973+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:03:26.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:03:26 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:03:26.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:03:26 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:03:27.332 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:03:26 smithi115 conmon[90393]: 2022-05-14T00:03:26.963+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:03:27.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:03:27 smithi115 ceph-mon[78593]: pgmap v1422: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:03:27.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:03:27 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:03:27.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:03:27 smithi047 ceph-mon[77147]: pgmap v1422: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:03:27.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:03:27 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:03:28.339 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:03:27 smithi115 conmon[90393]: 2022-05-14T00:03:27.947+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:03:28.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:03:28 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:03:28.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:03:28 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:03:29.338 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:03:28 smithi115 conmon[90393]: 2022-05-14T00:03:28.959+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:03:29.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:03:29 smithi115 ceph-mon[78593]: pgmap v1423: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:03:29.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:03:29 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:03:29.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:03:29 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 2140 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:03:29.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:03:29 smithi047 ceph-mon[77147]: pgmap v1423: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:03:29.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:03:29 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:03:29.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:03:29 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 2140 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:03:30.349 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:03:29 smithi115 conmon[90393]: 2022-05-14T00:03:29.941+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:03:30.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:03:30 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:03:30.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:03:30 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:03:30.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:03:30 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:03:30.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:03:30 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:03:30.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:03:30 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:03:30.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:03:30 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:03:31.351 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:03:30 smithi115 conmon[90393]: 2022-05-14T00:03:30.956+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:03:31.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:03:31 smithi115 ceph-mon[78593]: pgmap v1424: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:03:31.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:03:31 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:03:31.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:03:31 smithi047 ceph-mon[77147]: pgmap v1424: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:03:31.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:03:31 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:03:32.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:03:32 smithi115 conmon[90393]: 2022-05-14T00:03:32.000+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:03:32.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:03:32 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:03:32.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:03:32 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:03:33.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:03:32 smithi115 conmon[90393]: 2022-05-14T00:03:32.958+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:03:33.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:03:33 smithi047 ceph-mon[77147]: pgmap v1425: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:03:33.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:03:33 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:03:33.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:03:33 smithi115 ceph-mon[78593]: pgmap v1425: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:03:33.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:03:33 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:03:34.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:03:34 smithi115 conmon[90393]: 2022-05-14T00:03:34.001+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:03:34.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:03:34 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:03:34.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:03:34 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:03:34.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:03:34 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:03:34.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:03:34 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 2145 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:03:34.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:03:34 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:03:34.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:03:34 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:03:34.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:03:34 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:03:34.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:03:34 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 2145 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:03:35.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:03:35 smithi115 conmon[90393]: 2022-05-14T00:03:35.012+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:03:35.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:03:35 smithi047 ceph-mon[77147]: pgmap v1426: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:03:35.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:03:35 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:03:35.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:03:35 smithi115 ceph-mon[78593]: pgmap v1426: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:03:35.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:03:35 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:03:36.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:03:35 smithi115 conmon[90393]: 2022-05-14T00:03:35.987+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:03:36.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:03:36 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:03:36.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:03:36 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:03:37.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:03:37 smithi115 conmon[90393]: 2022-05-14T00:03:37.027+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:03:37.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:03:37 smithi047 ceph-mon[77147]: pgmap v1427: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:03:37.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:03:37 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:03:37.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:03:37 smithi115 ceph-mon[78593]: pgmap v1427: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:03:37.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:03:37 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:03:38.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:03:38 smithi115 conmon[90393]: 2022-05-14T00:03:38.072+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:03:38.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:03:38 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:03:38.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:03:38 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:03:38.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:03:38 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:03:38.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:03:38 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:03:38.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:03:38 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:03:38.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:03:38 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:03:39.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:03:39 smithi115 conmon[90393]: 2022-05-14T00:03:39.025+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:03:39.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:03:39 smithi047 ceph-mon[77147]: pgmap v1428: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:03:39.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:03:39 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:03:39.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:03:39 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 2150 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:03:39.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:03:39 smithi115 ceph-mon[78593]: pgmap v1428: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:03:39.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:03:39 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:03:39.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:03:39 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 2150 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:03:40.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:03:39 smithi115 conmon[90393]: 2022-05-14T00:03:39.979+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:03:40.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:03:40 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:03:40.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:03:40 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:03:41.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:03:40 smithi115 conmon[90393]: 2022-05-14T00:03:40.991+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:03:41.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:03:41 smithi047 ceph-mon[77147]: pgmap v1429: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:03:41.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:03:41 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:03:41.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:03:41 smithi115 ceph-mon[78593]: pgmap v1429: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:03:41.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:03:41 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:03:42.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:03:42 smithi115 conmon[90393]: 2022-05-14T00:03:42.015+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:03:42.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:03:42 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:03:42.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:03:42 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:03:42.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:03:42 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:03:42.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:03:42 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:03:42.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:03:42 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:03:42.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:03:42 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:03:43.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:03:42 smithi115 conmon[90393]: 2022-05-14T00:03:42.976+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:03:43.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:03:43 smithi047 ceph-mon[77147]: pgmap v1430: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:03:43.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:03:43 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:03:43.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:03:43 smithi115 ceph-mon[78593]: pgmap v1430: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:03:43.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:03:43 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:03:44.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:03:43 smithi115 conmon[90393]: 2022-05-14T00:03:43.980+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:03:44.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:03:44 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:03:44.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:03:44 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 2155 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:03:44.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:03:44 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:03:44.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:03:44 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 2155 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:03:45.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:03:44 smithi115 conmon[90393]: 2022-05-14T00:03:44.938+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:03:45.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:03:45 smithi047 ceph-mon[77147]: pgmap v1431: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:03:45.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:03:45 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:03:45.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:03:45 smithi115 ceph-mon[78593]: pgmap v1431: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:03:45.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:03:45 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:03:46.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:03:45 smithi115 conmon[90393]: 2022-05-14T00:03:45.923+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:03:46.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:03:46 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:03:46.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:03:46 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:03:47.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:03:46 smithi115 conmon[90393]: 2022-05-14T00:03:46.919+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:03:47.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:03:47 smithi047 ceph-mon[77147]: pgmap v1432: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:03:47.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:03:47 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:03:47.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:03:47 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:03:47.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:03:47 smithi115 ceph-mon[78593]: pgmap v1432: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:03:47.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:03:47 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:03:47.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:03:47 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:03:48.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:03:47 smithi115 conmon[90393]: 2022-05-14T00:03:47.884+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:03:48.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:03:48 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:03:48.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:03:48 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:03:49.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:03:48 smithi115 conmon[90393]: 2022-05-14T00:03:48.860+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:03:49.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:03:49 smithi047 ceph-mon[77147]: pgmap v1433: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:03:49.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:03:49 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 2160 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:03:49.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:03:49 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:03:49.815 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:03:49 smithi115 ceph-mon[78593]: pgmap v1433: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:03:49.815 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:03:49 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 2160 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:03:49.815 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:03:49 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:03:50.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:03:49 smithi115 conmon[90393]: 2022-05-14T00:03:49.813+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:03:50.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:03:50 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:03:50.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:03:50 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:03:50.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:03:50 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:03:50.853 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:03:50 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:03:50.854 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:03:50 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:03:50.854 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:03:50 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:03:51.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:03:50 smithi115 conmon[90393]: 2022-05-14T00:03:50.852+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:03:51.798 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:03:51 smithi047 ceph-mon[77147]: pgmap v1434: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:03:51.798 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:03:51 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:03:51.829 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:03:51 smithi115 ceph-mon[78593]: pgmap v1434: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:03:51.829 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:03:51 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:03:52.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:03:51 smithi115 conmon[90393]: 2022-05-14T00:03:51.827+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:03:52.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:03:52 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:03:52.841 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:03:52 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:03:53.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:03:52 smithi115 conmon[90393]: 2022-05-14T00:03:52.840+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:03:53.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:03:53 smithi047 ceph-mon[77147]: pgmap v1435: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:03:53.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:03:53 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:03:53.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:03:53 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:03:53.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:03:53 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:03:53.827 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:03:53 smithi115 ceph-mon[78593]: pgmap v1435: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:03:53.828 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:03:53 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:03:53.828 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:03:53 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:03:53.828 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:03:53 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:03:54.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:03:53 smithi115 conmon[90393]: 2022-05-14T00:03:53.826+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:03:54.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:03:54 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 2165 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:03:54.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:03:54 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:03:54.832 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:03:54 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 2165 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:03:54.832 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:03:54 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:03:55.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:03:54 smithi115 conmon[90393]: 2022-05-14T00:03:54.831+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:03:55.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:03:55 smithi115 conmon[90393]: 2022-05-14T00:03:55.784+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:03:55.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:03:55 smithi115 ceph-mon[78593]: pgmap v1436: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:03:55.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:03:55 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:03:56.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:03:55 smithi047 ceph-mon[77147]: pgmap v1436: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:03:56.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:03:55 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:03:56.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:03:56 smithi115 conmon[90393]: 2022-05-14T00:03:56.815+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:03:56.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:03:56 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:03:57.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:03:56 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:03:57.852 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:03:57 smithi115 ceph-mon[78593]: pgmap v1437: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:03:57.853 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:03:57 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:03:57.853 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:03:57 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:03:57.853 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:03:57 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:03:58.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:03:57 smithi047 ceph-mon[77147]: pgmap v1437: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:03:58.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:03:57 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:03:58.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:03:57 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:03:58.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:03:57 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:03:58.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:03:57 smithi115 conmon[90393]: 2022-05-14T00:03:57.851+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:03:58.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:03:58 smithi115 conmon[90393]: 2022-05-14T00:03:58.842+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:03:58.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:03:58 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:03:58.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:03:58 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 2170 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:03:59.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:03:58 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:03:59.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:03:58 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 2170 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:03:59.856 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:03:59 smithi115 ceph-mon[78593]: pgmap v1438: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:03:59.856 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:03:59 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:04:00.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:03:59 smithi047 ceph-mon[77147]: pgmap v1438: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:04:00.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:03:59 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:04:00.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:03:59 smithi115 conmon[90393]: 2022-05-14T00:03:59.854+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:04:00.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:04:00 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:04:01.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:04:00 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:04:01.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:04:00 smithi115 conmon[90393]: 2022-05-14T00:04:00.866+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:04:01.901 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:04:01 smithi115 ceph-mon[78593]: pgmap v1439: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:04:01.902 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:04:01 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:04:01.903 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:04:01 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:04:01.903 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:04:01 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:04:02.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:04:01 smithi047 ceph-mon[77147]: pgmap v1439: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:04:02.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:04:01 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:04:02.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:04:01 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:04:02.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:04:01 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:04:02.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:04:01 smithi115 conmon[90393]: 2022-05-14T00:04:01.899+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:04:02.869 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:04:02 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:04:03.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:04:02 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:04:03.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:04:02 smithi115 conmon[90393]: 2022-05-14T00:04:02.868+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:04:03.906 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:04:03 smithi115 ceph-mon[78593]: pgmap v1440: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:04:03.907 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:04:03 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:04:03.907 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:04:03 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 2175 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:04:04.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:04:03 smithi047 ceph-mon[77147]: pgmap v1440: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:04:04.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:04:03 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:04:04.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:04:03 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 2175 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:04:04.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:04:03 smithi115 conmon[90393]: 2022-05-14T00:04:03.905+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:04:04.913 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:04:04 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:04:05.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:04:04 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:04:05.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:04:04 smithi115 conmon[90393]: 2022-05-14T00:04:04.912+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:04:05.923 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:04:05 smithi115 ceph-mon[78593]: pgmap v1441: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:04:05.924 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:04:05 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:04:06.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:04:05 smithi047 ceph-mon[77147]: pgmap v1441: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:04:06.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:04:05 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:04:06.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:04:05 smithi115 conmon[90393]: 2022-05-14T00:04:05.922+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:04:06.929 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:04:06 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:04:07.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:04:06 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:04:07.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:04:06 smithi115 conmon[90393]: 2022-05-14T00:04:06.928+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:04:07.930 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:04:07 smithi115 ceph-mon[78593]: pgmap v1442: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:04:07.931 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:04:07 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:04:08.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:04:07 smithi047 ceph-mon[77147]: pgmap v1442: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:04:08.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:04:07 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:04:08.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:04:07 smithi115 conmon[90393]: 2022-05-14T00:04:07.929+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:04:08.892 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:04:08 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:04:08.892 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:04:08 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 2180 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:04:09.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:04:08 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:04:09.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:04:08 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 2180 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:04:09.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:04:08 smithi115 conmon[90393]: 2022-05-14T00:04:08.891+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:04:10.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:04:09 smithi047 ceph-mon[77147]: pgmap v1443: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:04:10.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:04:09 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:04:10.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:04:09 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:04:10.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:04:09 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:04:10.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:04:09 smithi115 conmon[90393]: 2022-05-14T00:04:09.842+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:04:10.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:04:09 smithi115 ceph-mon[78593]: pgmap v1443: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:04:10.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:04:09 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:04:10.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:04:09 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:04:10.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:04:09 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:04:10.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:04:10 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:04:10.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:04:10 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T00:04:10.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:04:10 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T00:04:10.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:04:10 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T00:04:10.825 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:04:10 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:04:10.825 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:04:10 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T00:04:10.825 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:04:10 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T00:04:10.825 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:04:10 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T00:04:11.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:04:10 smithi115 conmon[90393]: 2022-05-14T00:04:10.824+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:04:11.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:04:10 smithi047 conmon[78521]: 2022-05-14T00:04:10.863+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:04:11.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:04:10 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T00:04:11.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:04:10 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:04:11.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:04:10 smithi047 conmon[78521]: yield 2022-05-14T00:04:11.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:04:10 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:04:11.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:04:10 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:04:11.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:04:10 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:04:11.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:04:10 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T00:04:11.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:04:10 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:04:11.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:04:10 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T00:04:11.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:04:10 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:04:11.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:04:10 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T00:04:11.297 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:04:10 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:04:11.297 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:04:10 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T00:04:11.297 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:04:10 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:04:11.297 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:04:10 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T00:04:11.298 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:04:10 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:04:11.298 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:04:10 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:04:11.298 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:04:10 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:04:12.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:04:11 smithi047 ceph-mon[77147]: pgmap v1444: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:04:12.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:04:11 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:04:12.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:04:11 smithi115 conmon[90393]: 2022-05-14T00:04:11.783+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:04:12.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:04:11 smithi115 ceph-mon[78593]: pgmap v1444: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:04:12.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:04:11 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:04:13.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:04:12 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:04:13.041 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T00:04:13.041 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:04:13.041 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T00:04:13.042 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:04:13.042 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:04:13.042 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:04:13.042 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T00:04:13.042 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:04:13.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T00:04:13.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:04:13.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T00:04:13.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:04:13.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T00:04:13.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:04:13.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T00:04:13.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:04:13.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:04:13.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:04:13.045 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:04:12 smithi047 ceph-mon[77147]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T00:04:13.045 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:04:12 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:04:13.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:04:12 smithi115 conmon[90393]: 2022-05-14T00:04:12.762+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:04:13.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:04:12 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:04:13.109 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T00:04:13.109 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:04:13.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T00:04:13.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:04:13.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:04:13.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:04:13.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T00:04:13.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:04:13.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T00:04:13.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:04:13.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T00:04:13.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:04:13.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T00:04:13.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:04:13.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T00:04:13.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:04:13.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:04:13.113 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:04:13.113 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:04:12 smithi115 ceph-mon[78593]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T00:04:13.113 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:04:12 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:04:13.937 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:04:13 smithi047 ceph-mon[77147]: pgmap v1445: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:04:13.937 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:04:13 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:04:13.938 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:04:13 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:04:13.938 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:04:13 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:04:13.938 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:04:13 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T00:04:13.938 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:04:13 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T00:04:13.939 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:04:13 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 2185 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:04:14.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:04:13 smithi115 conmon[90393]: 2022-05-14T00:04:13.730+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:04:14.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:04:13 smithi115 ceph-mon[78593]: pgmap v1445: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:04:14.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:04:13 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:04:14.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:04:13 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:04:14.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:04:13 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:04:14.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:04:13 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T00:04:14.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:04:13 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T00:04:14.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:04:13 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 2185 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:04:14.291 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:04:13 smithi047 conmon[78521]: 2022-05-14T00:04:13.935+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:04:14.292 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:04:13 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T00:04:14.292 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:04:13 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:04:14.292 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:04:13 smithi047 conmon[78521]: yield 2022-05-14T00:04:14.292 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:04:13 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:04:14.293 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:04:13 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:04:14.293 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:04:13 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:04:14.293 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:04:13 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T00:04:14.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:04:13 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:04:14.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:04:13 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T00:04:14.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:04:13 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:04:14.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:04:13 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T00:04:14.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:04:13 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:04:14.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:04:13 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T00:04:14.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:04:13 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:04:14.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:04:13 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T00:04:14.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:04:13 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:04:14.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:04:13 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:04:14.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:04:13 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:04:14.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:04:13 smithi047 conmon[78521]: 2022-05-14T00:04:13.936+0000 7f533ef76700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T00:04:14.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:04:13 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T00:04:14.297 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:04:13 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:04:14.297 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:04:13 smithi047 conmon[78521]: yield 2022-05-14T00:04:14.297 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:04:13 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:04:14.297 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:04:13 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:04:14.297 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:04:13 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:04:14.298 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:04:13 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T00:04:14.298 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:04:13 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:04:14.298 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:04:13 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T00:04:14.298 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:04:13 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:04:14.299 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:04:13 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T00:04:14.299 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:04:13 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:04:14.299 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:04:13 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T00:04:14.299 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:04:13 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:04:14.299 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:04:13 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T00:04:14.300 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:04:13 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:04:14.300 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:04:13 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:04:14.300 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:04:13 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:04:14.300 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:04:13 smithi047 conmon[78521]: 2022-05-14T00:04:14.300 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:04:13 smithi047 conmon[78521]: During handling of the above exception, another exception occurred: 2022-05-14T00:04:14.301 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:04:13 smithi047 conmon[78521]: 2022-05-14T00:04:14.301 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:04:13 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T00:04:14.301 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:04:13 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T00:04:14.301 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:04:13 smithi047 conmon[78521]: return f(*arg) 2022-05-14T00:04:14.302 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:04:13 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T00:04:14.302 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:04:13 smithi047 conmon[78521]: self._write_client_files(client_files, host) 2022-05-14T00:04:14.302 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:04:13 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T00:04:14.302 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:04:13 smithi047 conmon[78521]: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T00:04:14.303 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:04:13 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T00:04:14.303 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:04:13 smithi047 conmon[78521]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T00:04:14.303 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:04:13 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T00:04:14.303 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:04:13 smithi047 conmon[78521]: return self.event_loop.get_result(coro) 2022-05-14T00:04:14.303 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:04:13 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T00:04:14.304 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:04:13 smithi047 conmon[78521]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T00:04:14.304 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:04:13 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T00:04:14.304 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:04:13 smithi047 conmon[78521]: return self.__get_result() 2022-05-14T00:04:14.304 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:04:13 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T00:04:14.304 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:04:13 smithi047 conmon[78521]: raise self._exception 2022-05-14T00:04:14.305 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:04:13 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T00:04:14.305 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:04:13 smithi047 conmon[78521]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T00:04:14.305 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:04:13 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T00:04:14.305 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:04:13 smithi047 conmon[78521]: conn = await self._remote_connection(host, addr) 2022-05-14T00:04:14.306 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:04:13 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T00:04:14.306 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:04:13 smithi047 conmon[78521]: raise 2022-05-14T00:04:14.306 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:04:13 smithi047 conmon[78521]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T00:04:14.306 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:04:13 smithi047 conmon[78521]: self.gen.throw(type, value, traceback) 2022-05-14T00:04:14.306 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:04:13 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T00:04:14.307 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:04:13 smithi047 conmon[78521]: raise OrchestratorError(msg) 2022-05-14T00:04:14.307 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:04:13 smithi047 conmon[78521]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:04:15.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:04:14 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:04:15.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:04:14 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:04:15.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:04:14 smithi115 conmon[90393]: 2022-05-14T00:04:14.719+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:04:16.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:04:15 smithi047 ceph-mon[77147]: pgmap v1446: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:04:16.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:04:15 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:04:16.041 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T00:04:16.042 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:04:16.042 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T00:04:16.042 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:04:16.042 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:04:16.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:04:16.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T00:04:16.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:04:16.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T00:04:16.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:04:16.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T00:04:16.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:04:16.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T00:04:16.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:04:16.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T00:04:16.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:04:16.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:04:16.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:04:16.046 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:04:15 smithi047 ceph-mon[77147]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T00:04:16.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T00:04:16.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:04:16.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T00:04:16.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:04:16.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:04:16.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:04:16.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T00:04:16.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:04:16.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T00:04:16.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:04:16.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T00:04:16.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:04:16.049 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T00:04:16.049 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:04:16.049 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T00:04:16.049 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:04:16.050 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:04:16.050 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:04:16.050 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T00:04:16.050 INFO:journalctl@ceph.mon.b.smithi047.stdout: During handling of the above exception, another exception occurred: 2022-05-14T00:04:16.050 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T00:04:16.051 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T00:04:16.051 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T00:04:16.051 INFO:journalctl@ceph.mon.b.smithi047.stdout: return f(*arg) 2022-05-14T00:04:16.051 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T00:04:16.051 INFO:journalctl@ceph.mon.b.smithi047.stdout: self._write_client_files(client_files, host) 2022-05-14T00:04:16.052 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T00:04:16.052 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T00:04:16.052 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T00:04:16.052 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T00:04:16.053 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T00:04:16.053 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.event_loop.get_result(coro) 2022-05-14T00:04:16.053 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T00:04:16.053 INFO:journalctl@ceph.mon.b.smithi047.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T00:04:16.053 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T00:04:16.054 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.__get_result() 2022-05-14T00:04:16.054 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T00:04:16.054 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise self._exception 2022-05-14T00:04:16.054 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T00:04:16.055 INFO:journalctl@ceph.mon.b.smithi047.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T00:04:16.055 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T00:04:16.055 INFO:journalctl@ceph.mon.b.smithi047.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T00:04:16.055 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T00:04:16.056 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise 2022-05-14T00:04:16.056 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T00:04:16.056 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.gen.throw(type, value, traceback) 2022-05-14T00:04:16.056 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T00:04:16.057 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OrchestratorError(msg) 2022-05-14T00:04:16.057 INFO:journalctl@ceph.mon.b.smithi047.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:04:16.057 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:04:15 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:04:16.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:04:15 smithi115 conmon[90393]: 2022-05-14T00:04:15.762+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:04:16.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:04:15 smithi115 ceph-mon[78593]: pgmap v1446: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:04:16.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:04:15 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:04:16.109 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T00:04:16.109 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:04:16.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T00:04:16.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:04:16.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:04:16.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:04:16.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T00:04:16.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:04:16.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T00:04:16.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:04:16.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T00:04:16.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:04:16.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T00:04:16.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:04:16.113 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T00:04:16.113 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:04:16.113 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:04:16.113 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:04:16.113 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:04:15 smithi115 ceph-mon[78593]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T00:04:16.114 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T00:04:16.114 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:04:16.115 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T00:04:16.115 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:04:16.115 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:04:16.115 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:04:16.115 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T00:04:16.116 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:04:16.116 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T00:04:16.116 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:04:16.116 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T00:04:16.116 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:04:16.117 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T00:04:16.117 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:04:16.117 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T00:04:16.117 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:04:16.118 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:04:16.118 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:04:16.118 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T00:04:16.118 INFO:journalctl@ceph.mon.c.smithi115.stdout: During handling of the above exception, another exception occurred: 2022-05-14T00:04:16.119 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T00:04:16.119 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T00:04:16.119 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T00:04:16.119 INFO:journalctl@ceph.mon.c.smithi115.stdout: return f(*arg) 2022-05-14T00:04:16.119 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T00:04:16.120 INFO:journalctl@ceph.mon.c.smithi115.stdout: self._write_client_files(client_files, host) 2022-05-14T00:04:16.120 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T00:04:16.120 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T00:04:16.120 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T00:04:16.120 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T00:04:16.121 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T00:04:16.121 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.event_loop.get_result(coro) 2022-05-14T00:04:16.121 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T00:04:16.121 INFO:journalctl@ceph.mon.c.smithi115.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T00:04:16.122 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T00:04:16.122 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.__get_result() 2022-05-14T00:04:16.122 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T00:04:16.122 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise self._exception 2022-05-14T00:04:16.123 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T00:04:16.123 INFO:journalctl@ceph.mon.c.smithi115.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T00:04:16.123 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T00:04:16.123 INFO:journalctl@ceph.mon.c.smithi115.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T00:04:16.124 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T00:04:16.124 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise 2022-05-14T00:04:16.124 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T00:04:16.124 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.gen.throw(type, value, traceback) 2022-05-14T00:04:16.124 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T00:04:16.125 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OrchestratorError(msg) 2022-05-14T00:04:16.125 INFO:journalctl@ceph.mon.c.smithi115.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:04:16.125 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:04:15 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:04:17.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:04:16 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:04:17.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:04:16 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:04:17.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:04:16 smithi115 conmon[90393]: 2022-05-14T00:04:16.745+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:04:18.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:04:17 smithi047 ceph-mon[77147]: pgmap v1447: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:04:18.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:04:17 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:04:18.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:04:17 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:04:18.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:04:17 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:04:18.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:04:17 smithi115 conmon[90393]: 2022-05-14T00:04:17.731+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:04:18.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:04:17 smithi115 ceph-mon[78593]: pgmap v1447: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:04:18.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:04:17 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:04:18.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:04:17 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:04:18.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:04:17 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:04:19.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:04:18 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:04:19.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:04:18 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 2190 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:04:19.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:04:18 smithi115 conmon[90393]: 2022-05-14T00:04:18.779+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:04:19.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:04:18 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:04:19.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:04:18 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 2190 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:04:20.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:04:19 smithi047 ceph-mon[77147]: pgmap v1448: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:04:20.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:04:19 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:04:20.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:04:19 smithi115 conmon[90393]: 2022-05-14T00:04:19.827+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:04:20.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:04:19 smithi115 ceph-mon[78593]: pgmap v1448: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:04:20.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:04:19 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:04:21.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:04:20 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:04:21.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:04:20 smithi115 conmon[90393]: 2022-05-14T00:04:20.829+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:04:21.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:04:20 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:04:22.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:04:21 smithi047 ceph-mon[77147]: pgmap v1449: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:04:22.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:04:21 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:04:22.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:04:21 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:04:22.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:04:21 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:04:22.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:04:21 smithi115 conmon[90393]: 2022-05-14T00:04:21.828+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:04:22.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:04:21 smithi115 ceph-mon[78593]: pgmap v1449: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:04:22.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:04:21 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:04:22.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:04:21 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:04:22.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:04:21 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:04:23.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:04:22 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:04:23.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:04:22 smithi115 conmon[90393]: 2022-05-14T00:04:22.801+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:04:23.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:04:22 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:04:24.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:04:23 smithi047 ceph-mon[77147]: pgmap v1450: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:04:24.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:04:23 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:04:24.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:04:23 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 2195 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:04:24.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:04:23 smithi115 conmon[90393]: 2022-05-14T00:04:23.761+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:04:24.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:04:23 smithi115 ceph-mon[78593]: pgmap v1450: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:04:24.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:04:23 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:04:24.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:04:23 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 2195 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:04:25.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:04:24 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:04:25.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:04:24 smithi115 conmon[90393]: 2022-05-14T00:04:24.778+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:04:25.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:04:24 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:04:26.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:04:25 smithi047 ceph-mon[77147]: pgmap v1451: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:04:26.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:04:25 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:04:26.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:04:25 smithi115 conmon[90393]: 2022-05-14T00:04:25.731+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:04:26.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:04:25 smithi115 ceph-mon[78593]: pgmap v1451: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:04:26.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:04:25 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:04:27.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:04:26 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:04:27.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:04:26 smithi115 conmon[90393]: 2022-05-14T00:04:26.721+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:04:27.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:04:26 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:04:28.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:04:27 smithi047 ceph-mon[77147]: pgmap v1452: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:04:28.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:04:27 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:04:28.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:04:27 smithi115 conmon[90393]: 2022-05-14T00:04:27.704+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:04:28.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:04:27 smithi115 ceph-mon[78593]: pgmap v1452: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:04:28.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:04:27 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:04:29.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:04:28 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:04:29.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:04:28 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 2200 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:04:29.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:04:28 smithi115 conmon[90393]: 2022-05-14T00:04:28.700+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:04:29.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:04:28 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:04:29.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:04:28 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 2200 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:04:30.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:04:29 smithi047 ceph-mon[77147]: pgmap v1453: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:04:30.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:04:29 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:04:30.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:04:29 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:04:30.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:04:29 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:04:30.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:04:29 smithi115 conmon[90393]: 2022-05-14T00:04:29.678+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:04:30.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:04:29 smithi115 ceph-mon[78593]: pgmap v1453: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:04:30.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:04:29 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:04:30.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:04:29 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:04:30.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:04:29 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:04:31.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:04:30 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:04:31.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:04:30 smithi115 conmon[90393]: 2022-05-14T00:04:30.724+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:04:31.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:04:30 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:04:32.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:04:31 smithi047 ceph-mon[77147]: pgmap v1454: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:04:32.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:04:31 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:04:32.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:04:31 smithi115 conmon[90393]: 2022-05-14T00:04:31.710+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:04:32.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:04:31 smithi115 ceph-mon[78593]: pgmap v1454: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:04:32.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:04:31 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:04:33.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:04:32 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:04:33.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:04:32 smithi115 conmon[90393]: 2022-05-14T00:04:32.666+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:04:33.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:04:32 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:04:34.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:04:33 smithi047 ceph-mon[77147]: pgmap v1455: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:04:34.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:04:33 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:04:34.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:04:33 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:04:34.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:04:33 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:04:34.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:04:33 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 2205 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:04:34.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:04:33 smithi115 conmon[90393]: 2022-05-14T00:04:33.636+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:04:34.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:04:33 smithi115 ceph-mon[78593]: pgmap v1455: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:04:34.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:04:33 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:04:34.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:04:33 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:04:34.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:04:33 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:04:34.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:04:33 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 2205 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:04:35.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:04:34 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:04:35.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:04:34 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:04:35.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:04:34 smithi115 conmon[90393]: 2022-05-14T00:04:34.618+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:04:35.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:04:35 smithi115 conmon[90393]: 2022-05-14T00:04:35.572+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:04:35.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:04:35 smithi115 ceph-mon[78593]: pgmap v1456: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:04:35.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:04:35 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:04:36.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:04:35 smithi047 ceph-mon[77147]: pgmap v1456: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:04:36.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:04:35 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:04:36.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:04:36 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:04:36.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:04:36 smithi115 conmon[90393]: 2022-05-14T00:04:36.542+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:04:37.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:04:36 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:04:37.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:04:37 smithi115 ceph-mon[78593]: pgmap v1457: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:04:37.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:04:37 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:04:37.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:04:37 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:04:37.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:04:37 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:04:37.867 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:04:37 smithi115 conmon[90393]: 2022-05-14T00:04:37.591+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:04:38.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:04:37 smithi047 ceph-mon[77147]: pgmap v1457: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:04:38.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:04:37 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:04:38.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:04:37 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:04:38.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:04:37 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:04:38.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:04:38 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:04:38.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:04:38 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 2210 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:04:38.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:04:38 smithi115 conmon[90393]: 2022-05-14T00:04:38.583+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:04:39.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:04:38 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:04:39.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:04:38 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 2210 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:04:39.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:04:39 smithi115 ceph-mon[78593]: pgmap v1458: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:04:39.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:04:39 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:04:39.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:04:39 smithi115 conmon[90393]: 2022-05-14T00:04:39.572+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:04:40.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:04:39 smithi047 ceph-mon[77147]: pgmap v1458: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:04:40.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:04:39 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:04:40.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:04:40 smithi115 conmon[90393]: 2022-05-14T00:04:40.546+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:04:40.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:04:40 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:04:41.045 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:04:40 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:04:41.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:04:41 smithi115 conmon[90393]: 2022-05-14T00:04:41.511+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:04:41.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:04:41 smithi115 ceph-mon[78593]: pgmap v1459: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:04:41.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:04:41 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:04:41.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:04:41 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:04:41.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:04:41 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:04:42.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:04:41 smithi047 ceph-mon[77147]: pgmap v1459: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:04:42.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:04:41 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:04:42.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:04:41 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:04:42.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:04:41 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:04:42.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:04:42 smithi115 conmon[90393]: 2022-05-14T00:04:42.523+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:04:42.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:04:42 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:04:43.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:04:42 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:04:43.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:04:43 smithi115 conmon[90393]: 2022-05-14T00:04:43.552+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:04:43.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:04:43 smithi115 ceph-mon[78593]: pgmap v1460: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:04:43.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:04:43 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:04:43.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:04:43 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 2215 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:04:44.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:04:43 smithi047 ceph-mon[77147]: pgmap v1460: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:04:44.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:04:43 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:04:44.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:04:43 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 2215 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:04:44.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:04:44 smithi115 conmon[90393]: 2022-05-14T00:04:44.592+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:04:44.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:04:44 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:04:45.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:04:44 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:04:46.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:04:45 smithi047 ceph-mon[77147]: pgmap v1461: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:04:46.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:04:45 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:04:46.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:04:45 smithi115 conmon[90393]: 2022-05-14T00:04:45.626+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:04:46.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:04:45 smithi115 ceph-mon[78593]: pgmap v1461: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:04:46.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:04:45 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:04:47.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:04:46 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:04:47.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:04:46 smithi115 conmon[90393]: 2022-05-14T00:04:46.676+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:04:47.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:04:46 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:04:48.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:04:47 smithi047 ceph-mon[77147]: pgmap v1462: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:04:48.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:04:47 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:04:48.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:04:47 smithi115 conmon[90393]: 2022-05-14T00:04:47.651+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:04:48.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:04:47 smithi115 ceph-mon[78593]: pgmap v1462: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:04:48.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:04:47 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:04:49.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:04:48 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:04:49.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:04:48 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 2220 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:04:49.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:04:48 smithi115 conmon[90393]: 2022-05-14T00:04:48.642+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:04:49.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:04:48 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:04:49.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:04:48 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 2220 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:04:50.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:04:49 smithi047 ceph-mon[77147]: pgmap v1463: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:04:50.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:04:49 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:04:50.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:04:49 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:04:50.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:04:49 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:04:50.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:04:49 smithi115 conmon[90393]: 2022-05-14T00:04:49.645+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:04:50.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:04:49 smithi115 ceph-mon[78593]: pgmap v1463: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:04:50.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:04:49 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:04:50.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:04:49 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:04:50.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:04:49 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:04:51.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:04:50 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:04:51.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:04:50 smithi115 conmon[90393]: 2022-05-14T00:04:50.631+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:04:51.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:04:50 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:04:52.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:04:51 smithi047 ceph-mon[77147]: pgmap v1464: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:04:52.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:04:51 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:04:52.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:04:51 smithi115 conmon[90393]: 2022-05-14T00:04:51.611+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:04:52.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:04:51 smithi115 ceph-mon[78593]: pgmap v1464: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:04:52.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:04:51 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:04:52.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:04:52 smithi115 conmon[90393]: 2022-05-14T00:04:52.563+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:04:52.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:04:52 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:04:53.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:04:52 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:04:54.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:04:53 smithi047 ceph-mon[77147]: pgmap v1465: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:04:54.045 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:04:53 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:04:54.045 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:04:53 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:04:54.046 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:04:53 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:04:54.046 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:04:53 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 2225 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:04:54.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:04:53 smithi115 conmon[90393]: 2022-05-14T00:04:53.609+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:04:54.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:04:53 smithi115 ceph-mon[78593]: pgmap v1465: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:04:54.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:04:53 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:04:54.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:04:53 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:04:54.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:04:53 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:04:54.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:04:53 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 2225 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:04:55.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:04:54 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:04:55.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:04:54 smithi115 conmon[90393]: 2022-05-14T00:04:54.626+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:04:55.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:04:54 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:04:55.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:04:55 smithi115 conmon[90393]: 2022-05-14T00:04:55.586+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:04:55.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:04:55 smithi115 ceph-mon[78593]: pgmap v1466: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:04:55.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:04:55 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:04:56.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:04:55 smithi047 ceph-mon[77147]: pgmap v1466: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:04:56.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:04:55 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:04:56.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:04:56 smithi115 conmon[90393]: 2022-05-14T00:04:56.555+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:04:56.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:04:56 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:04:57.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:04:56 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:04:57.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:04:57 smithi115 conmon[90393]: 2022-05-14T00:04:57.553+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:04:57.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:04:57 smithi115 ceph-mon[78593]: pgmap v1467: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:04:57.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:04:57 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:04:57.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:04:57 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:04:57.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:04:57 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:04:58.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:04:57 smithi047 ceph-mon[77147]: pgmap v1467: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:04:58.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:04:57 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:04:58.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:04:57 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:04:58.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:04:57 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:04:58.793 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:04:58 smithi115 conmon[90393]: 2022-05-14T00:04:58.535+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:04:59.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:04:58 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:04:59.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:04:58 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 2230 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:04:59.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:04:58 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:04:59.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:04:58 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 2230 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:04:59.795 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:04:59 smithi115 conmon[90393]: 2022-05-14T00:04:59.544+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:05:00.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:04:59 smithi115 ceph-mon[78593]: pgmap v1468: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:05:00.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:04:59 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:05:00.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:04:59 smithi047 ceph-mon[77147]: pgmap v1468: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:05:00.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:04:59 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:05:00.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:05:00 smithi115 conmon[90393]: 2022-05-14T00:05:00.584+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:05:00.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:05:00 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:05:00.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:05:00 smithi115 ceph-mon[78593]: pgmap v1469: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:05:01.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:05:00 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:05:01.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:05:00 smithi047 ceph-mon[77147]: pgmap v1469: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:05:01.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:05:01 smithi115 conmon[90393]: 2022-05-14T00:05:01.557+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:05:01.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:05:01 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:05:01.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:05:01 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:05:01.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:05:01 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:05:02.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:05:01 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:05:02.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:05:01 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:05:02.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:05:01 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:05:02.860 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:05:02 smithi115 conmon[90393]: 2022-05-14T00:05:02.595+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:05:02.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:05:02 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:05:02.862 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:05:02 smithi115 ceph-mon[78593]: pgmap v1470: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:05:03.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:05:02 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:05:03.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:05:02 smithi047 ceph-mon[77147]: pgmap v1470: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:05:04.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:05:03 smithi115 conmon[90393]: 2022-05-14T00:05:03.640+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:05:04.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:05:03 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:05:04.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:05:03 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 2235 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:05:04.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:05:03 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:05:04.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:05:03 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 2235 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:05:05.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:05:04 smithi115 conmon[90393]: 2022-05-14T00:05:04.612+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:05:05.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:05:04 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:05:05.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:05:04 smithi115 ceph-mon[78593]: pgmap v1471: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:05:05.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:05:04 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:05:05.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:05:04 smithi047 ceph-mon[77147]: pgmap v1471: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:05:05.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:05:05 smithi115 conmon[90393]: 2022-05-14T00:05:05.567+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:05:05.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:05:05 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:05:06.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:05:05 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:05:06.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:05:06 smithi115 conmon[90393]: 2022-05-14T00:05:06.593+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:05:06.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:05:06 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:05:06.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:05:06 smithi115 ceph-mon[78593]: pgmap v1472: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:05:07.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:05:06 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:05:07.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:05:06 smithi047 ceph-mon[77147]: pgmap v1472: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:05:07.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:05:07 smithi115 conmon[90393]: 2022-05-14T00:05:07.593+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:05:07.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:05:07 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:05:08.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:05:07 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:05:08.825 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:05:08 smithi115 conmon[90393]: 2022-05-14T00:05:08.570+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:05:09.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:05:08 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:05:09.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:05:08 smithi115 ceph-mon[78593]: pgmap v1473: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:05:09.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:05:08 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 2240 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:05:09.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:05:08 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:05:09.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:05:08 smithi047 ceph-mon[77147]: pgmap v1473: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:05:09.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:05:08 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 2240 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:05:09.827 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:05:09 smithi115 conmon[90393]: 2022-05-14T00:05:09.533+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:05:10.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:05:09 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:05:10.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:05:09 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:05:10.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:05:09 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:05:10.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:05:09 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:05:10.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:05:09 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:05:10.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:05:09 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:05:10.831 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:05:10 smithi115 conmon[90393]: 2022-05-14T00:05:10.531+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:05:11.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:05:10 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:05:11.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:05:10 smithi115 ceph-mon[78593]: pgmap v1474: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:05:11.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:05:10 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:05:11.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:05:10 smithi047 ceph-mon[77147]: pgmap v1474: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:05:11.833 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:05:11 smithi115 conmon[90393]: 2022-05-14T00:05:11.518+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:05:12.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:05:11 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:05:12.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:05:11 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:05:12.839 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:05:12 smithi115 conmon[90393]: 2022-05-14T00:05:12.565+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:05:13.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:05:12 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:05:13.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:05:12 smithi115 ceph-mon[78593]: pgmap v1475: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:05:13.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:05:12 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:05:13.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:05:12 smithi047 ceph-mon[77147]: pgmap v1475: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:05:13.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:05:13 smithi115 conmon[90393]: 2022-05-14T00:05:13.591+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:05:13.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:05:13 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:05:13.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:05:13 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:05:13.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:05:13 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:05:13.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:05:13 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T00:05:13.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:05:13 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T00:05:13.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:05:13 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 2245 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:05:14.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:05:13 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:05:14.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:05:13 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:05:14.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:05:13 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:05:14.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:05:13 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T00:05:14.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:05:13 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T00:05:14.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:05:13 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 2245 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:05:14.846 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:05:14 smithi115 conmon[90393]: 2022-05-14T00:05:14.558+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:05:15.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:05:14 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:05:15.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:05:14 smithi115 ceph-mon[78593]: pgmap v1476: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:05:15.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:05:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T00:05:15.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:05:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T00:05:15.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:05:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T00:05:15.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:05:14 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:05:15.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:05:14 smithi047 ceph-mon[77147]: pgmap v1476: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:05:15.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:05:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T00:05:15.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:05:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T00:05:15.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:05:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T00:05:15.845 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:05:15 smithi115 conmon[90393]: 2022-05-14T00:05:15.526+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:05:16.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:05:15 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:05:16.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:05:15 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:05:16.852 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:05:16 smithi115 conmon[90393]: 2022-05-14T00:05:16.575+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:05:17.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:05:16 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:05:17.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:05:16 smithi115 ceph-mon[78593]: pgmap v1477: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:05:17.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:05:16 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:05:17.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:05:16 smithi047 ceph-mon[77147]: pgmap v1477: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:05:17.291 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:05:17 smithi047 conmon[78521]: 2022-05-14T00:05:17.039+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:05:17.292 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:05:17 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T00:05:17.292 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:05:17 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:05:17.293 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:05:17 smithi047 conmon[78521]: yield 2022-05-14T00:05:17.293 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:05:17 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:05:17.293 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:05:17 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:05:17.293 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:05:17 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:05:17.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:05:17 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T00:05:17.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:05:17 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:05:17.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:05:17 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T00:05:17.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:05:17 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:05:17.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:05:17 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T00:05:17.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:05:17 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:05:17.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:05:17 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T00:05:17.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:05:17 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:05:17.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:05:17 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T00:05:17.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:05:17 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:05:17.297 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:05:17 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:05:17.297 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:05:17 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:05:18.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:05:17 smithi115 conmon[90393]: 2022-05-14T00:05:17.610+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:05:18.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:05:17 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:05:18.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:05:17 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:05:18.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:05:17 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:05:18.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:05:17 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:05:18.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T00:05:18.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:05:18.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T00:05:18.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:05:18.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:05:18.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:05:18.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T00:05:18.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:05:18.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T00:05:18.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:05:18.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T00:05:18.113 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:05:18.113 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T00:05:18.114 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:05:18.114 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T00:05:18.114 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:05:18.114 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:05:18.115 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:05:18.115 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:05:17 smithi115 ceph-mon[78593]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T00:05:18.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:05:17 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:05:18.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:05:17 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:05:18.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:05:17 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:05:18.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:05:17 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:05:18.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T00:05:18.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:05:18.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T00:05:18.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:05:18.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:05:18.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:05:18.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T00:05:18.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:05:18.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T00:05:18.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:05:18.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T00:05:18.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:05:18.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T00:05:18.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:05:18.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T00:05:18.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:05:18.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:05:18.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:05:18.298 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:05:17 smithi047 ceph-mon[77147]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T00:05:18.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:05:18 smithi115 conmon[90393]: 2022-05-14T00:05:18.591+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:05:19.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:05:18 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:05:19.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:05:18 smithi047 ceph-mon[77147]: pgmap v1478: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:05:19.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:05:18 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 2250 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:05:19.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:05:18 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:05:19.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:05:18 smithi115 ceph-mon[78593]: pgmap v1478: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:05:19.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:05:18 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 2250 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:05:20.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:05:19 smithi115 conmon[90393]: 2022-05-14T00:05:19.611+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:05:20.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:05:19 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:05:20.112 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:05:19 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:05:20.541 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:05:20 smithi047 conmon[78521]: 2022-05-14T00:05:20.112+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:05:20.542 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:05:20 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T00:05:20.542 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:05:20 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:05:20.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:05:20 smithi047 conmon[78521]: yield 2022-05-14T00:05:20.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:05:20 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:05:20.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:05:20 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:05:20.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:05:20 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:05:20.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:05:20 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T00:05:20.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:05:20 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:05:20.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:05:20 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T00:05:20.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:05:20 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:05:20.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:05:20 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T00:05:20.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:05:20 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:05:20.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:05:20 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T00:05:20.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:05:20 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:05:20.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:05:20 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T00:05:20.547 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:05:20 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:05:20.547 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:05:20 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:05:20.548 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:05:20 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:05:20.548 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:05:20 smithi047 conmon[78521]: 2022-05-14T00:05:20.112+0000 7f533f777700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T00:05:20.548 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:05:20 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T00:05:20.549 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:05:20 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:05:20.549 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:05:20 smithi047 conmon[78521]: yield 2022-05-14T00:05:20.550 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:05:20 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:05:20.550 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:05:20 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:05:20.551 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:05:20 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:05:20.551 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:05:20 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T00:05:20.552 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:05:20 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:05:20.552 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:05:20 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T00:05:20.553 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:05:20 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:05:20.553 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:05:20 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T00:05:20.554 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:05:20 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:05:20.554 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:05:20 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T00:05:20.555 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:05:20 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:05:20.556 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:05:20 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T00:05:20.556 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:05:20 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:05:20.557 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:05:20 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:05:20.559 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:05:20 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:05:20.560 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:05:20 smithi047 conmon[78521]: 2022-05-14T00:05:20.560 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:05:20 smithi047 conmon[78521]: During handling of the above exception, another exception occurred: 2022-05-14T00:05:20.561 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:05:20 smithi047 conmon[78521]: 2022-05-14T00:05:20.561 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:05:20 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T00:05:20.561 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:05:20 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T00:05:20.561 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:05:20 smithi047 conmon[78521]: return f(*arg) 2022-05-14T00:05:20.562 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:05:20 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T00:05:20.562 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:05:20 smithi047 conmon[78521]: self._write_client_files(client_files, host) 2022-05-14T00:05:20.562 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:05:20 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T00:05:20.565 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:05:20 smithi047 conmon[78521]: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T00:05:20.565 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:05:20 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T00:05:20.565 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:05:20 smithi047 conmon[78521]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T00:05:20.565 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:05:20 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T00:05:20.566 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:05:20 smithi047 conmon[78521]: return self.event_loop.get_result(coro) 2022-05-14T00:05:20.566 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:05:20 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T00:05:20.566 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:05:20 smithi047 conmon[78521]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T00:05:20.566 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:05:20 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T00:05:20.567 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:05:20 smithi047 conmon[78521]: return self.__get_result() 2022-05-14T00:05:20.567 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:05:20 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T00:05:20.567 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:05:20 smithi047 conmon[78521]: raise self._exception 2022-05-14T00:05:20.567 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:05:20 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T00:05:20.567 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:05:20 smithi047 conmon[78521]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T00:05:20.568 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:05:20 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T00:05:20.568 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:05:20 smithi047 conmon[78521]: conn = await self._remote_connection(host, addr) 2022-05-14T00:05:20.568 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:05:20 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T00:05:20.568 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:05:20 smithi047 conmon[78521]: raise 2022-05-14T00:05:20.569 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:05:20 smithi047 conmon[78521]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T00:05:20.569 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:05:20 smithi047 conmon[78521]: self.gen.throw(type, value, traceback) 2022-05-14T00:05:20.569 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:05:20 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T00:05:20.569 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:05:20 smithi047 conmon[78521]: raise OrchestratorError(msg) 2022-05-14T00:05:20.569 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:05:20 smithi047 conmon[78521]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:05:20.867 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:05:20 smithi115 conmon[90393]: 2022-05-14T00:05:20.586+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:05:21.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:05:20 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:05:21.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:05:20 smithi047 ceph-mon[77147]: pgmap v1479: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:05:21.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:05:20 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:05:21.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T00:05:21.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:05:21.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T00:05:21.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:05:21.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:05:21.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:05:21.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T00:05:21.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:05:21.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T00:05:21.298 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:05:21.298 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T00:05:21.299 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:05:21.299 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T00:05:21.299 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:05:21.300 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T00:05:21.300 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:05:21.300 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:05:21.301 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:05:21.301 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:05:20 smithi047 ceph-mon[77147]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T00:05:21.301 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T00:05:21.301 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:05:21.302 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T00:05:21.302 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:05:21.302 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:05:21.302 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:05:21.303 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T00:05:21.303 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:05:21.303 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T00:05:21.304 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:05:21.304 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T00:05:21.304 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:05:21.305 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T00:05:21.305 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:05:21.305 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T00:05:21.305 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:05:21.305 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:05:21.306 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:05:21.306 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T00:05:21.306 INFO:journalctl@ceph.mon.b.smithi047.stdout: During handling of the above exception, another exception occurred: 2022-05-14T00:05:21.307 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T00:05:21.307 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T00:05:21.307 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T00:05:21.307 INFO:journalctl@ceph.mon.b.smithi047.stdout: return f(*arg) 2022-05-14T00:05:21.308 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T00:05:21.308 INFO:journalctl@ceph.mon.b.smithi047.stdout: self._write_client_files(client_files, host) 2022-05-14T00:05:21.308 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T00:05:21.309 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T00:05:21.309 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T00:05:21.309 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T00:05:21.309 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T00:05:21.310 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.event_loop.get_result(coro) 2022-05-14T00:05:21.310 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T00:05:21.310 INFO:journalctl@ceph.mon.b.smithi047.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T00:05:21.310 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T00:05:21.310 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.__get_result() 2022-05-14T00:05:21.311 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T00:05:21.311 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise self._exception 2022-05-14T00:05:21.311 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T00:05:21.311 INFO:journalctl@ceph.mon.b.smithi047.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T00:05:21.311 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T00:05:21.312 INFO:journalctl@ceph.mon.b.smithi047.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T00:05:21.312 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T00:05:21.312 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise 2022-05-14T00:05:21.312 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T00:05:21.313 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.gen.throw(type, value, traceback) 2022-05-14T00:05:21.313 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T00:05:21.313 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OrchestratorError(msg) 2022-05-14T00:05:21.313 INFO:journalctl@ceph.mon.b.smithi047.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:05:21.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:05:20 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:05:21.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:05:20 smithi115 ceph-mon[78593]: pgmap v1479: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:05:21.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:05:20 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:05:21.359 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T00:05:21.359 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:05:21.359 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T00:05:21.359 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:05:21.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:05:21.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:05:21.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T00:05:21.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:05:21.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T00:05:21.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:05:21.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T00:05:21.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:05:21.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T00:05:21.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:05:21.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T00:05:21.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:05:21.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:05:21.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:05:21.363 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:05:20 smithi115 ceph-mon[78593]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T00:05:21.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T00:05:21.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:05:21.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T00:05:21.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:05:21.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:05:21.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:05:21.365 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T00:05:21.365 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:05:21.365 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T00:05:21.365 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:05:21.365 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T00:05:21.366 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:05:21.366 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T00:05:21.366 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:05:21.366 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T00:05:21.366 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:05:21.367 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:05:21.367 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:05:21.367 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T00:05:21.367 INFO:journalctl@ceph.mon.c.smithi115.stdout: During handling of the above exception, another exception occurred: 2022-05-14T00:05:21.368 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T00:05:21.368 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T00:05:21.368 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T00:05:21.368 INFO:journalctl@ceph.mon.c.smithi115.stdout: return f(*arg) 2022-05-14T00:05:21.369 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T00:05:21.369 INFO:journalctl@ceph.mon.c.smithi115.stdout: self._write_client_files(client_files, host) 2022-05-14T00:05:21.369 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T00:05:21.370 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T00:05:21.370 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T00:05:21.370 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T00:05:21.370 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T00:05:21.370 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.event_loop.get_result(coro) 2022-05-14T00:05:21.371 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T00:05:21.371 INFO:journalctl@ceph.mon.c.smithi115.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T00:05:21.371 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T00:05:21.371 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.__get_result() 2022-05-14T00:05:21.372 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T00:05:21.372 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise self._exception 2022-05-14T00:05:21.372 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T00:05:21.372 INFO:journalctl@ceph.mon.c.smithi115.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T00:05:21.372 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T00:05:21.373 INFO:journalctl@ceph.mon.c.smithi115.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T00:05:21.373 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T00:05:21.373 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise 2022-05-14T00:05:21.373 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T00:05:21.374 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.gen.throw(type, value, traceback) 2022-05-14T00:05:21.374 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T00:05:21.374 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OrchestratorError(msg) 2022-05-14T00:05:21.374 INFO:journalctl@ceph.mon.c.smithi115.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:05:21.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:05:21 smithi115 conmon[90393]: 2022-05-14T00:05:21.588+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:05:22.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:05:21 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:05:22.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:05:21 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:05:22.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:05:21 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:05:22.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:05:21 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:05:22.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:05:21 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:05:22.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:05:21 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:05:23.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:05:22 smithi115 conmon[90393]: 2022-05-14T00:05:22.634+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:05:23.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:05:22 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:05:23.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:05:22 smithi115 ceph-mon[78593]: pgmap v1480: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:05:23.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:05:22 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:05:23.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:05:22 smithi047 ceph-mon[77147]: pgmap v1480: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:05:24.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:05:23 smithi115 conmon[90393]: 2022-05-14T00:05:23.679+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:05:24.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:05:23 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:05:24.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:05:23 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 2255 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:05:24.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:05:23 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:05:24.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:05:23 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 2255 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:05:25.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:05:24 smithi115 conmon[90393]: 2022-05-14T00:05:24.642+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:05:25.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:05:24 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:05:25.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:05:24 smithi115 ceph-mon[78593]: pgmap v1481: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:05:25.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:05:24 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:05:25.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:05:24 smithi047 ceph-mon[77147]: pgmap v1481: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:05:26.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:05:25 smithi115 conmon[90393]: 2022-05-14T00:05:25.629+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:05:26.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:05:25 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:05:26.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:05:25 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:05:27.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:05:26 smithi115 conmon[90393]: 2022-05-14T00:05:26.674+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:05:27.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:05:26 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:05:27.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:05:26 smithi115 ceph-mon[78593]: pgmap v1482: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:05:27.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:05:26 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:05:27.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:05:26 smithi047 ceph-mon[77147]: pgmap v1482: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:05:28.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:05:27 smithi115 conmon[90393]: 2022-05-14T00:05:27.656+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:05:28.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:05:27 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:05:28.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:05:27 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:05:28.890 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:05:28 smithi115 conmon[90393]: 2022-05-14T00:05:28.610+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:05:29.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:05:28 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:05:29.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:05:28 smithi047 ceph-mon[77147]: pgmap v1483: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:05:29.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:05:28 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 2260 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:05:29.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:05:28 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:05:29.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:05:28 smithi115 ceph-mon[78593]: pgmap v1483: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:05:29.362 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:05:28 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 2260 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:05:30.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:05:29 smithi115 conmon[90393]: 2022-05-14T00:05:29.658+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:05:30.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:05:29 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:05:30.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:05:29 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:05:30.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:05:29 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:05:30.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:05:29 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:05:30.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:05:29 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:05:30.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:05:29 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:05:30.898 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:05:30 smithi115 conmon[90393]: 2022-05-14T00:05:30.620+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:05:31.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:05:30 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:05:31.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:05:30 smithi047 ceph-mon[77147]: pgmap v1484: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:05:31.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:05:30 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:05:31.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:05:30 smithi115 ceph-mon[78593]: pgmap v1484: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:05:32.110 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:05:31 smithi115 conmon[90393]: 2022-05-14T00:05:31.662+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:05:32.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:05:31 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:05:32.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:05:31 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:05:32.903 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:05:32 smithi115 conmon[90393]: 2022-05-14T00:05:32.641+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:05:33.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:05:32 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:05:33.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:05:32 smithi047 ceph-mon[77147]: pgmap v1485: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:05:33.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:05:32 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:05:33.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:05:32 smithi115 ceph-mon[78593]: pgmap v1485: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:05:33.909 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:05:33 smithi115 conmon[90393]: 2022-05-14T00:05:33.649+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:05:34.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:05:33 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:05:34.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:05:33 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:05:34.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:05:33 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:05:34.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:05:33 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 2265 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:05:34.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:05:33 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:05:34.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:05:33 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:05:34.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:05:33 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:05:34.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:05:33 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 2265 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:05:34.910 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:05:34 smithi115 conmon[90393]: 2022-05-14T00:05:34.618+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:05:35.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:05:34 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:05:35.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:05:34 smithi047 ceph-mon[77147]: pgmap v1486: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:05:35.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:05:34 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:05:35.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:05:34 smithi115 ceph-mon[78593]: pgmap v1486: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:05:35.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:05:35 smithi115 conmon[90393]: 2022-05-14T00:05:35.577+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:05:36.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:05:35 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:05:36.402 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:05:35 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:05:36.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:05:36 smithi115 conmon[90393]: 2022-05-14T00:05:36.534+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:05:37.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:05:36 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:05:37.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:05:36 smithi047 ceph-mon[77147]: pgmap v1487: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:05:37.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:05:36 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:05:37.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:05:36 smithi115 ceph-mon[78593]: pgmap v1487: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:05:37.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:05:37 smithi115 conmon[90393]: 2022-05-14T00:05:37.554+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:05:38.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:05:37 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:05:38.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:05:37 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:05:38.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:05:37 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:05:38.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:05:37 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:05:38.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:05:37 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:05:38.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:05:37 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:05:38.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:05:38 smithi115 conmon[90393]: 2022-05-14T00:05:38.509+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:05:39.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:05:38 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:05:39.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:05:38 smithi047 ceph-mon[77147]: pgmap v1488: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:05:39.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:05:38 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 2270 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:05:39.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:05:38 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:05:39.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:05:38 smithi115 ceph-mon[78593]: pgmap v1488: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:05:39.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:05:38 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 2270 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:05:39.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:05:39 smithi115 conmon[90393]: 2022-05-14T00:05:39.480+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:05:40.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:05:39 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:05:40.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:05:39 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:05:40.861 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:05:40 smithi115 conmon[90393]: 2022-05-14T00:05:40.452+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:05:41.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:05:40 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:05:41.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:05:40 smithi047 ceph-mon[77147]: pgmap v1489: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:05:41.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:05:40 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:05:41.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:05:40 smithi115 ceph-mon[78593]: pgmap v1489: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:05:41.862 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:05:41 smithi115 conmon[90393]: 2022-05-14T00:05:41.448+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:05:42.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:05:41 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:05:42.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:05:41 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:05:42.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:05:41 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:05:42.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:05:41 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:05:42.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:05:41 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:05:42.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:05:41 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:05:42.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:05:42 smithi115 conmon[90393]: 2022-05-14T00:05:42.496+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:05:43.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:05:42 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:05:43.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:05:42 smithi047 ceph-mon[77147]: pgmap v1490: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:05:43.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:05:42 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:05:43.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:05:42 smithi115 ceph-mon[78593]: pgmap v1490: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:05:43.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:05:43 smithi115 conmon[90393]: 2022-05-14T00:05:43.546+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:05:44.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:05:43 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:05:44.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:05:43 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 2275 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:05:44.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:05:43 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:05:44.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:05:43 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 2275 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:05:44.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:05:44 smithi115 conmon[90393]: 2022-05-14T00:05:44.526+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:05:45.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:05:44 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:05:45.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:05:44 smithi047 ceph-mon[77147]: pgmap v1491: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:05:45.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:05:44 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:05:45.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:05:44 smithi115 ceph-mon[78593]: pgmap v1491: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:05:45.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:05:45 smithi115 conmon[90393]: 2022-05-14T00:05:45.546+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:05:46.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:05:45 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:05:46.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:05:45 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:05:46.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:05:46 smithi115 conmon[90393]: 2022-05-14T00:05:46.529+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:05:47.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:05:46 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:05:47.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:05:46 smithi047 ceph-mon[77147]: pgmap v1492: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:05:47.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:05:46 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:05:47.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:05:46 smithi115 ceph-mon[78593]: pgmap v1492: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:05:47.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:05:47 smithi115 conmon[90393]: 2022-05-14T00:05:47.537+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:05:48.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:05:47 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:05:48.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:05:47 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:05:48.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:05:48 smithi115 conmon[90393]: 2022-05-14T00:05:48.575+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:05:49.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:05:48 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:05:49.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:05:48 smithi047 ceph-mon[77147]: pgmap v1493: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:05:49.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:05:48 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 2280 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:05:49.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:05:48 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:05:49.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:05:48 smithi115 ceph-mon[78593]: pgmap v1493: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:05:49.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:05:48 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 2280 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:05:49.959 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:05:49 smithi115 conmon[90393]: 2022-05-14T00:05:49.620+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:05:50.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:05:49 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:05:50.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:05:49 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:05:50.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:05:49 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:05:50.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:05:49 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:05:50.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:05:49 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:05:50.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:05:49 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:05:50.964 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:05:50 smithi115 conmon[90393]: 2022-05-14T00:05:50.650+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:05:51.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:05:50 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:05:51.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:05:50 smithi047 ceph-mon[77147]: pgmap v1494: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:05:51.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:05:50 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:05:51.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:05:50 smithi115 ceph-mon[78593]: pgmap v1494: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:05:51.966 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:05:51 smithi115 conmon[90393]: 2022-05-14T00:05:51.612+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:05:52.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:05:51 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:05:52.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:05:51 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:05:52.967 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:05:52 smithi115 conmon[90393]: 2022-05-14T00:05:52.641+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:05:53.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:05:52 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:05:53.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:05:52 smithi047 ceph-mon[77147]: pgmap v1495: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:05:53.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:05:52 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:05:53.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:05:52 smithi115 ceph-mon[78593]: pgmap v1495: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:05:53.973 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:05:53 smithi115 conmon[90393]: 2022-05-14T00:05:53.671+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:05:54.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:05:53 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:05:54.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:05:53 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:05:54.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:05:53 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:05:54.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:05:53 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 2285 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:05:54.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:05:53 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:05:54.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:05:53 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:05:54.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:05:53 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:05:54.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:05:53 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 2285 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:05:54.975 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:05:54 smithi115 conmon[90393]: 2022-05-14T00:05:54.708+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:05:55.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:05:54 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:05:55.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:05:54 smithi047 ceph-mon[77147]: pgmap v1496: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:05:55.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:05:54 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:05:55.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:05:54 smithi115 ceph-mon[78593]: pgmap v1496: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:05:56.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:05:55 smithi115 conmon[90393]: 2022-05-14T00:05:55.737+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:05:56.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:05:55 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:05:56.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:05:55 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:05:56.981 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:05:56 smithi115 conmon[90393]: 2022-05-14T00:05:56.718+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:05:57.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:05:56 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:05:57.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:05:56 smithi047 ceph-mon[77147]: pgmap v1497: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:05:57.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:05:56 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:05:57.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:05:56 smithi115 ceph-mon[78593]: pgmap v1497: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:05:58.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:05:57 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:05:58.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:05:57 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:05:58.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:05:57 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:05:58.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:05:57 smithi115 conmon[90393]: 2022-05-14T00:05:57.760+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:05:58.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:05:57 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:05:58.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:05:57 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:05:58.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:05:57 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:05:59.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:05:58 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:05:59.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:05:58 smithi115 ceph-mon[78593]: pgmap v1498: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:05:59.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:05:58 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 2290 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:05:59.110 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:05:58 smithi115 conmon[90393]: 2022-05-14T00:05:58.782+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:05:59.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:05:58 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:05:59.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:05:58 smithi047 ceph-mon[77147]: pgmap v1498: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:05:59.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:05:58 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 2290 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:06:00.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:05:59 smithi115 conmon[90393]: 2022-05-14T00:05:59.785+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:06:00.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:05:59 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:06:00.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:05:59 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:06:01.002 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:06:00 smithi115 conmon[90393]: 2022-05-14T00:06:00.747+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:06:01.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:06:01 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:06:01.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:06:01 smithi047 ceph-mon[77147]: pgmap v1499: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:06:01.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:06:01 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:06:01.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:06:01 smithi115 ceph-mon[78593]: pgmap v1499: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:06:02.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:06:02 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:06:02.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:06:02 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:06:02.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:06:02 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:06:02.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:06:01 smithi115 conmon[90393]: 2022-05-14T00:06:01.789+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:06:02.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:06:02 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:06:02.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:06:02 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:06:02.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:06:02 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:06:03.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:06:02 smithi115 conmon[90393]: 2022-05-14T00:06:02.792+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:06:03.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:06:03 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:06:03.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:06:03 smithi115 ceph-mon[78593]: pgmap v1500: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:06:03.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:06:03 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:06:03.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:06:03 smithi047 ceph-mon[77147]: pgmap v1500: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:06:04.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:06:03 smithi115 conmon[90393]: 2022-05-14T00:06:03.834+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:06:04.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:06:04 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:06:04.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:06:04 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 2295 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:06:04.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:06:04 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:06:04.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:06:04 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 2295 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:06:05.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:06:04 smithi115 conmon[90393]: 2022-05-14T00:06:04.825+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:06:05.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:06:05 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:06:05.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:06:05 smithi115 ceph-mon[78593]: pgmap v1501: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:06:05.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:06:05 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:06:05.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:06:05 smithi047 ceph-mon[77147]: pgmap v1501: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:06:06.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:06:06 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:06:06.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:06:05 smithi115 conmon[90393]: 2022-05-14T00:06:05.837+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:06:06.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:06:06 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:06:07.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:06:06 smithi115 conmon[90393]: 2022-05-14T00:06:06.850+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:06:07.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:06:07 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:06:07.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:06:07 smithi115 ceph-mon[78593]: pgmap v1502: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:06:07.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:06:07 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:06:07.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:06:07 smithi047 ceph-mon[77147]: pgmap v1502: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:06:08.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:06:08 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:06:08.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:06:07 smithi115 conmon[90393]: 2022-05-14T00:06:07.821+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:06:08.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:06:08 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:06:09.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:06:08 smithi115 conmon[90393]: 2022-05-14T00:06:08.782+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:06:09.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:06:09 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:06:09.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:06:09 smithi115 ceph-mon[78593]: pgmap v1503: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:06:09.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:06:09 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 2300 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:06:09.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:06:09 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:06:09.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:06:09 smithi047 ceph-mon[77147]: pgmap v1503: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:06:09.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:06:09 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 2300 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:06:10.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:06:09 smithi115 conmon[90393]: 2022-05-14T00:06:09.820+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:06:10.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:06:10 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:06:10.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:06:10 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:06:10.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:06:10 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:06:10.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:06:10 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:06:10.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:06:10 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:06:10.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:06:10 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:06:11.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:06:10 smithi115 conmon[90393]: 2022-05-14T00:06:10.846+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:06:11.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:06:11 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:06:11.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:06:11 smithi115 ceph-mon[78593]: pgmap v1504: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:06:11.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:06:11 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:06:11.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:06:11 smithi047 ceph-mon[77147]: pgmap v1504: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:06:12.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:06:12 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:06:12.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:06:11 smithi115 conmon[90393]: 2022-05-14T00:06:11.841+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:06:12.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:06:12 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:06:13.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:06:12 smithi115 conmon[90393]: 2022-05-14T00:06:12.810+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:06:13.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:06:13 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:06:13.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:06:13 smithi115 ceph-mon[78593]: pgmap v1505: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:06:13.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:06:13 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:06:13.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:06:13 smithi047 ceph-mon[77147]: pgmap v1505: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:06:14.049 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:06:13 smithi115 conmon[90393]: 2022-05-14T00:06:13.761+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:06:14.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:06:14 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:06:14.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:06:14 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:06:14.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:06:14 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:06:14.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:06:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T00:06:14.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:06:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T00:06:14.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:06:14 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 2305 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:06:14.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:06:14 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:06:14.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:06:14 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:06:14.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:06:14 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:06:14.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:06:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T00:06:14.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:06:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T00:06:14.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:06:14 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 2305 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:06:15.051 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:06:14 smithi115 conmon[90393]: 2022-05-14T00:06:14.712+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:06:15.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:06:15 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:06:15.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:06:15 smithi115 ceph-mon[78593]: pgmap v1506: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:06:15.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:06:15 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:06:15.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:06:15 smithi047 ceph-mon[77147]: pgmap v1506: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:06:16.056 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:06:15 smithi115 conmon[90393]: 2022-05-14T00:06:15.718+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:06:16.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:06:16 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:06:16.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:06:16 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:06:17.055 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:06:16 smithi115 conmon[90393]: 2022-05-14T00:06:16.703+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:06:17.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:06:17 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:06:17.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:06:17 smithi115 ceph-mon[78593]: pgmap v1507: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:06:17.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:06:17 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:06:17.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:06:17 smithi047 ceph-mon[77147]: pgmap v1507: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:06:18.067 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:06:17 smithi115 conmon[90393]: 2022-05-14T00:06:17.718+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:06:18.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:06:18 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:06:18.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:06:18 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:06:18.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:06:18 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:06:18.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:06:18 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:06:18.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:06:18 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:06:18.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:06:18 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:06:19.068 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:06:18 smithi115 conmon[90393]: 2022-05-14T00:06:18.720+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:06:19.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:06:19 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:06:19.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:06:19 smithi115 ceph-mon[78593]: pgmap v1508: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:06:19.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:06:19 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 2310 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:06:19.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:06:19 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:06:19.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:06:19 smithi047 ceph-mon[77147]: pgmap v1508: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:06:19.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:06:19 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 2310 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:06:20.073 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:06:19 smithi115 conmon[90393]: 2022-05-14T00:06:19.751+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:06:20.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:06:20 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:06:20.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:06:20 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:06:21.074 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:06:20 smithi115 conmon[90393]: 2022-05-14T00:06:20.736+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:06:21.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:06:21 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:06:21.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:06:21 smithi115 ceph-mon[78593]: pgmap v1509: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:06:21.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:06:21 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T00:06:21.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:06:21 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T00:06:21.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:06:21 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T00:06:21.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:06:21 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:06:21.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:06:21 smithi047 ceph-mon[77147]: pgmap v1509: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:06:21.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:06:21 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T00:06:21.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:06:21 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T00:06:21.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:06:21 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T00:06:22.083 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:06:21 smithi115 conmon[90393]: 2022-05-14T00:06:21.715+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:06:22.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:06:22 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:06:22.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:06:22 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:06:22.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:06:22 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:06:22.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:06:22 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:06:22.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:06:22 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:06:22.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:06:22 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:06:23.085 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:06:22 smithi115 conmon[90393]: 2022-05-14T00:06:22.671+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:06:23.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:06:23 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:06:23.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:06:23 smithi115 ceph-mon[78593]: pgmap v1510: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:06:23.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:06:23 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:06:23.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:06:23 smithi047 ceph-mon[77147]: pgmap v1510: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:06:23.541 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:06:23 smithi047 conmon[78521]: 2022-05-14T00:06:23.216+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:06:23.542 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:06:23 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T00:06:23.542 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:06:23 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:06:23.542 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:06:23 smithi047 conmon[78521]: yield 2022-05-14T00:06:23.542 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:06:23 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:06:23.542 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:06:23 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:06:23.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:06:23 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:06:23.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:06:23 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T00:06:23.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:06:23 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:06:23.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:06:23 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T00:06:23.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:06:23 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:06:23.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:06:23 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T00:06:23.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:06:23 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:06:23.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:06:23 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T00:06:23.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:06:23 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:06:23.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:06:23 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T00:06:23.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:06:23 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:06:23.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:06:23 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:06:23.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:06:23 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:06:24.086 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:06:23 smithi115 conmon[90393]: 2022-05-14T00:06:23.720+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:06:24.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:06:24 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:06:24.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:06:24 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:06:24.358 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T00:06:24.358 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:06:24.359 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T00:06:24.359 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:06:24.359 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:06:24.359 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:06:24.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T00:06:24.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:06:24.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T00:06:24.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:06:24.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T00:06:24.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:06:24.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T00:06:24.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:06:24.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T00:06:24.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:06:24.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:06:24.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:06:24.363 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:06:24 smithi115 ceph-mon[78593]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T00:06:24.363 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:06:24 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 2315 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:06:24.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:06:24 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:06:24.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:06:24 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:06:24.541 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T00:06:24.541 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:06:24.542 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T00:06:24.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:06:24.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:06:24.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:06:24.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T00:06:24.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:06:24.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T00:06:24.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:06:24.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T00:06:24.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:06:24.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T00:06:24.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:06:24.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T00:06:24.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:06:24.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:06:24.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:06:24.546 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:06:24 smithi047 ceph-mon[77147]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T00:06:24.546 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:06:24 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 2315 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:06:25.088 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:06:24 smithi115 conmon[90393]: 2022-05-14T00:06:24.753+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:06:25.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:06:25 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:06:25.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:06:25 smithi115 ceph-mon[78593]: pgmap v1511: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:06:25.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:06:25 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:06:25.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:06:25 smithi047 ceph-mon[77147]: pgmap v1511: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:06:26.093 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:06:25 smithi115 conmon[90393]: 2022-05-14T00:06:25.722+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:06:26.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:06:26 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:06:26.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:06:26 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:06:26.541 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:06:26 smithi047 conmon[78521]: 2022-05-14T00:06:26.288+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:06:26.542 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:06:26 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T00:06:26.542 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:06:26 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:06:26.542 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:06:26 smithi047 conmon[78521]: yield 2022-05-14T00:06:26.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:06:26 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:06:26.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:06:26 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:06:26.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:06:26 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:06:26.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:06:26 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T00:06:26.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:06:26 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:06:26.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:06:26 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T00:06:26.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:06:26 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:06:26.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:06:26 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T00:06:26.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:06:26 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:06:26.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:06:26 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T00:06:26.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:06:26 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:06:26.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:06:26 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T00:06:26.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:06:26 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:06:26.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:06:26 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:06:26.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:06:26 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:06:26.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:06:26 smithi047 conmon[78521]: 2022-05-14T00:06:26.289+0000 7f5344f82700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T00:06:26.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:06:26 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T00:06:26.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:06:26 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:06:26.547 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:06:26 smithi047 conmon[78521]: yield 2022-05-14T00:06:26.547 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:06:26 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:06:26.547 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:06:26 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:06:26.547 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:06:26 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:06:26.548 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:06:26 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T00:06:26.548 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:06:26 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:06:26.548 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:06:26 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T00:06:26.548 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:06:26 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:06:26.548 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:06:26 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T00:06:26.549 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:06:26 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:06:26.549 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:06:26 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T00:06:26.549 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:06:26 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:06:26.549 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:06:26 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T00:06:26.549 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:06:26 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:06:26.550 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:06:26 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:06:26.550 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:06:26 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:06:26.550 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:06:26 smithi047 conmon[78521]: 2022-05-14T00:06:26.550 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:06:26 smithi047 conmon[78521]: During handling of the above exception, another exception occurred: 2022-05-14T00:06:26.551 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:06:26 smithi047 conmon[78521]: 2022-05-14T00:06:26.551 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:06:26 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T00:06:26.551 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:06:26 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T00:06:26.551 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:06:26 smithi047 conmon[78521]: return f(*arg) 2022-05-14T00:06:26.551 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:06:26 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T00:06:26.552 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:06:26 smithi047 conmon[78521]: self._write_client_files(client_files, host) 2022-05-14T00:06:26.552 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:06:26 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T00:06:26.552 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:06:26 smithi047 conmon[78521]: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T00:06:26.552 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:06:26 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T00:06:26.553 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:06:26 smithi047 conmon[78521]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T00:06:26.553 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:06:26 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T00:06:26.553 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:06:26 smithi047 conmon[78521]: return self.event_loop.get_result(coro) 2022-05-14T00:06:26.553 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:06:26 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T00:06:26.553 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:06:26 smithi047 conmon[78521]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T00:06:26.554 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:06:26 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T00:06:26.554 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:06:26 smithi047 conmon[78521]: return self.__get_result() 2022-05-14T00:06:26.554 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:06:26 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T00:06:26.554 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:06:26 smithi047 conmon[78521]: raise self._exception 2022-05-14T00:06:26.554 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:06:26 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T00:06:26.555 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:06:26 smithi047 conmon[78521]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T00:06:26.555 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:06:26 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T00:06:26.555 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:06:26 smithi047 conmon[78521]: conn = await self._remote_connection(host, addr) 2022-05-14T00:06:26.555 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:06:26 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T00:06:26.556 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:06:26 smithi047 conmon[78521]: raise 2022-05-14T00:06:26.556 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:06:26 smithi047 conmon[78521]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T00:06:26.556 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:06:26 smithi047 conmon[78521]: self.gen.throw(type, value, traceback) 2022-05-14T00:06:26.556 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:06:26 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T00:06:26.556 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:06:26 smithi047 conmon[78521]: raise OrchestratorError(msg) 2022-05-14T00:06:26.557 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:06:26 smithi047 conmon[78521]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:06:27.095 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:06:26 smithi115 conmon[90393]: 2022-05-14T00:06:26.760+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:06:27.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:06:27 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:06:27.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:06:27 smithi115 ceph-mon[78593]: pgmap v1512: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:06:27.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:06:27 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:06:27.358 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T00:06:27.358 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:06:27.359 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T00:06:27.359 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:06:27.359 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:06:27.359 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:06:27.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T00:06:27.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:06:27.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T00:06:27.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:06:27.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T00:06:27.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:06:27.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T00:06:27.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:06:27.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T00:06:27.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:06:27.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:06:27.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:06:27.363 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:06:27 smithi115 ceph-mon[78593]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T00:06:27.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T00:06:27.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:06:27.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T00:06:27.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:06:27.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:06:27.365 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:06:27.365 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T00:06:27.365 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:06:27.366 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T00:06:27.366 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:06:27.366 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T00:06:27.366 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:06:27.366 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T00:06:27.367 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:06:27.367 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T00:06:27.367 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:06:27.367 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:06:27.367 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:06:27.368 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T00:06:27.368 INFO:journalctl@ceph.mon.c.smithi115.stdout: During handling of the above exception, another exception occurred: 2022-05-14T00:06:27.368 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T00:06:27.368 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T00:06:27.368 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T00:06:27.369 INFO:journalctl@ceph.mon.c.smithi115.stdout: return f(*arg) 2022-05-14T00:06:27.369 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T00:06:27.369 INFO:journalctl@ceph.mon.c.smithi115.stdout: self._write_client_files(client_files, host) 2022-05-14T00:06:27.369 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T00:06:27.370 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T00:06:27.370 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T00:06:27.370 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T00:06:27.371 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T00:06:27.371 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.event_loop.get_result(coro) 2022-05-14T00:06:27.371 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T00:06:27.371 INFO:journalctl@ceph.mon.c.smithi115.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T00:06:27.371 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T00:06:27.372 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.__get_result() 2022-05-14T00:06:27.372 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T00:06:27.372 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise self._exception 2022-05-14T00:06:27.372 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T00:06:27.372 INFO:journalctl@ceph.mon.c.smithi115.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T00:06:27.373 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T00:06:27.373 INFO:journalctl@ceph.mon.c.smithi115.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T00:06:27.373 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T00:06:27.373 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise 2022-05-14T00:06:27.373 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T00:06:27.374 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.gen.throw(type, value, traceback) 2022-05-14T00:06:27.374 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T00:06:27.374 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OrchestratorError(msg) 2022-05-14T00:06:27.374 INFO:journalctl@ceph.mon.c.smithi115.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:06:27.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:06:27 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:06:27.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:06:27 smithi047 ceph-mon[77147]: pgmap v1512: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:06:27.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:06:27 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:06:27.542 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T00:06:27.542 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:06:27.542 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T00:06:27.542 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:06:27.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:06:27.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:06:27.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T00:06:27.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:06:27.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T00:06:27.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:06:27.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T00:06:27.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:06:27.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T00:06:27.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:06:27.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T00:06:27.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:06:27.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:06:27.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:06:27.546 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:06:27 smithi047 ceph-mon[77147]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T00:06:27.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T00:06:27.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:06:27.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T00:06:27.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:06:27.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:06:27.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:06:27.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T00:06:27.548 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:06:27.548 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T00:06:27.548 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:06:27.548 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T00:06:27.549 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:06:27.549 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T00:06:27.549 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:06:27.549 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T00:06:27.549 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:06:27.550 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:06:27.550 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:06:27.550 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T00:06:27.550 INFO:journalctl@ceph.mon.b.smithi047.stdout: During handling of the above exception, another exception occurred: 2022-05-14T00:06:27.550 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T00:06:27.551 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T00:06:27.551 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T00:06:27.551 INFO:journalctl@ceph.mon.b.smithi047.stdout: return f(*arg) 2022-05-14T00:06:27.552 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T00:06:27.552 INFO:journalctl@ceph.mon.b.smithi047.stdout: self._write_client_files(client_files, host) 2022-05-14T00:06:27.552 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T00:06:27.552 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T00:06:27.553 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T00:06:27.553 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T00:06:27.553 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T00:06:27.553 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.event_loop.get_result(coro) 2022-05-14T00:06:27.553 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T00:06:27.554 INFO:journalctl@ceph.mon.b.smithi047.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T00:06:27.554 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T00:06:27.554 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.__get_result() 2022-05-14T00:06:27.554 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T00:06:27.555 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise self._exception 2022-05-14T00:06:27.555 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T00:06:27.555 INFO:journalctl@ceph.mon.b.smithi047.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T00:06:27.555 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T00:06:27.555 INFO:journalctl@ceph.mon.b.smithi047.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T00:06:27.556 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T00:06:27.556 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise 2022-05-14T00:06:27.556 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T00:06:27.556 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.gen.throw(type, value, traceback) 2022-05-14T00:06:27.556 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T00:06:27.557 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OrchestratorError(msg) 2022-05-14T00:06:27.557 INFO:journalctl@ceph.mon.b.smithi047.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:06:28.101 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:06:27 smithi115 conmon[90393]: 2022-05-14T00:06:27.781+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:06:28.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:06:28 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:06:28.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:06:28 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:06:29.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:06:28 smithi115 conmon[90393]: 2022-05-14T00:06:28.825+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:06:29.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:06:29 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:06:29.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:06:29 smithi115 ceph-mon[78593]: pgmap v1513: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:06:29.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:06:29 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 2320 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:06:29.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:06:29 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:06:29.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:06:29 smithi047 ceph-mon[77147]: pgmap v1513: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:06:29.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:06:29 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 2320 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:06:30.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:06:29 smithi115 conmon[90393]: 2022-05-14T00:06:29.816+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:06:30.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:06:30 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:06:30.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:06:30 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:06:30.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:06:30 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:06:30.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:06:30 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:06:30.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:06:30 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:06:30.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:06:30 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:06:31.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:06:30 smithi115 conmon[90393]: 2022-05-14T00:06:30.815+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:06:31.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:06:31 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:06:31.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:06:31 smithi047 ceph-mon[77147]: pgmap v1514: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:06:31.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:06:31 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:06:31.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:06:31 smithi115 ceph-mon[78593]: pgmap v1514: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:06:32.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:06:31 smithi115 conmon[90393]: 2022-05-14T00:06:31.803+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:06:32.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:06:32 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:06:32.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:06:32 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:06:33.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:06:32 smithi115 conmon[90393]: 2022-05-14T00:06:32.849+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:06:33.546 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:06:33 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:06:33.546 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:06:33 smithi047 ceph-mon[77147]: pgmap v1515: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:06:33.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:06:33 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:06:33.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:06:33 smithi115 ceph-mon[78593]: pgmap v1515: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:06:34.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:06:33 smithi115 conmon[90393]: 2022-05-14T00:06:33.845+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:06:34.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:06:34 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:06:34.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:06:34 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:06:34.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:06:34 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:06:34.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:06:34 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 2325 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:06:34.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:06:34 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:06:34.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:06:34 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:06:34.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:06:34 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:06:34.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:06:34 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 2325 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:06:35.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:06:34 smithi115 conmon[90393]: 2022-05-14T00:06:34.878+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:06:35.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:06:35 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:06:35.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:06:35 smithi115 ceph-mon[78593]: pgmap v1516: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:06:35.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:06:35 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:06:35.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:06:35 smithi047 ceph-mon[77147]: pgmap v1516: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:06:36.130 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:06:35 smithi115 conmon[90393]: 2022-05-14T00:06:35.874+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:06:36.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:06:36 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:06:36.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:06:36 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:06:37.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:06:36 smithi115 conmon[90393]: 2022-05-14T00:06:36.917+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:06:37.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:06:37 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:06:37.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:06:37 smithi115 ceph-mon[78593]: pgmap v1517: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:06:37.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:06:37 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:06:37.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:06:37 smithi047 ceph-mon[77147]: pgmap v1517: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:06:38.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:06:37 smithi115 conmon[90393]: 2022-05-14T00:06:37.960+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:06:38.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:06:38 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:06:38.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:06:38 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:06:38.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:06:38 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:06:38.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:06:38 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:06:38.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:06:38 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:06:38.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:06:38 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:06:39.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:06:38 smithi115 conmon[90393]: 2022-05-14T00:06:38.979+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:06:39.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:06:39 smithi115 ceph-mon[78593]: pgmap v1518: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:06:39.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:06:39 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:06:39.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:06:39 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 2330 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:06:39.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:06:39 smithi047 ceph-mon[77147]: pgmap v1518: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:06:39.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:06:39 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:06:39.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:06:39 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 2330 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:06:40.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:06:39 smithi115 conmon[90393]: 2022-05-14T00:06:39.937+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:06:40.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:06:40 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:06:40.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:06:40 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:06:41.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:06:40 smithi115 conmon[90393]: 2022-05-14T00:06:40.931+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:06:41.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:06:41 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:06:41.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:06:41 smithi115 ceph-mon[78593]: pgmap v1519: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:06:41.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:06:41 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:06:41.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:06:41 smithi047 ceph-mon[77147]: pgmap v1519: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:06:42.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:06:41 smithi115 conmon[90393]: 2022-05-14T00:06:41.919+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:06:42.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:06:42 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:06:42.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:06:42 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:06:42.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:06:42 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:06:42.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:06:42 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:06:42.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:06:42 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:06:42.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:06:42 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:06:43.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:06:42 smithi115 conmon[90393]: 2022-05-14T00:06:42.945+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:06:43.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:06:43 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:06:43.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:06:43 smithi115 ceph-mon[78593]: pgmap v1520: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:06:43.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:06:43 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:06:43.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:06:43 smithi047 ceph-mon[77147]: pgmap v1520: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:06:44.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:06:43 smithi115 conmon[90393]: 2022-05-14T00:06:43.941+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:06:44.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:06:44 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:06:44.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:06:44 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 2335 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:06:44.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:06:44 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:06:44.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:06:44 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 2335 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:06:45.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:06:44 smithi115 conmon[90393]: 2022-05-14T00:06:44.962+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:06:45.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:06:45 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:06:45.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:06:45 smithi115 ceph-mon[78593]: pgmap v1521: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:06:45.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:06:45 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:06:45.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:06:45 smithi047 ceph-mon[77147]: pgmap v1521: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:06:46.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:06:46 smithi115 conmon[90393]: 2022-05-14T00:06:46.011+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:06:46.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:06:46 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:06:46.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:06:46 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:06:47.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:06:46 smithi115 conmon[90393]: 2022-05-14T00:06:46.963+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:06:47.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:06:47 smithi115 ceph-mon[78593]: pgmap v1522: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:06:47.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:06:47 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:06:47.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:06:47 smithi047 ceph-mon[77147]: pgmap v1522: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:06:47.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:06:47 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:06:48.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:06:47 smithi115 conmon[90393]: 2022-05-14T00:06:47.986+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:06:48.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:06:48 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:06:48.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:06:48 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:06:49.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:06:49 smithi115 conmon[90393]: 2022-05-14T00:06:49.009+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:06:49.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:06:49 smithi115 ceph-mon[78593]: pgmap v1523: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:06:49.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:06:49 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:06:49.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:06:49 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 2340 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:06:49.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:06:49 smithi047 ceph-mon[77147]: pgmap v1523: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:06:49.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:06:49 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:06:49.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:06:49 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 2340 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:06:50.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:06:49 smithi115 conmon[90393]: 2022-05-14T00:06:49.986+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:06:50.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:06:50 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:06:50.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:06:50 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:06:50.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:06:50 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:06:50.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:06:50 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:06:50.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:06:50 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:06:50.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:06:50 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:06:51.362 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:06:51 smithi115 conmon[90393]: 2022-05-14T00:06:51.023+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:06:51.362 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:06:51 smithi115 ceph-mon[78593]: pgmap v1524: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:06:51.363 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:06:51 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:06:51.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:06:51 smithi047 ceph-mon[77147]: pgmap v1524: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:06:51.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:06:51 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:06:52.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:06:52 smithi115 conmon[90393]: 2022-05-14T00:06:52.062+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:06:52.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:06:52 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:06:52.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:06:52 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:06:53.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:06:53 smithi115 conmon[90393]: 2022-05-14T00:06:53.066+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:06:53.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:06:53 smithi115 ceph-mon[78593]: pgmap v1525: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-05-14T00:06:53.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:06:53 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:06:53.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:06:53 smithi047 ceph-mon[77147]: pgmap v1525: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-05-14T00:06:53.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:06:53 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:06:54.361 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:06:54 smithi115 conmon[90393]: 2022-05-14T00:06:54.058+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:06:54.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:06:54 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:06:54.362 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:06:54 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:06:54.362 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:06:54 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:06:54.362 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:06:54 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 2345 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:06:54.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:06:54 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:06:54.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:06:54 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:06:54.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:06:54 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:06:54.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:06:54 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 2345 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:06:55.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:06:55 smithi115 conmon[90393]: 2022-05-14T00:06:55.090+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:06:55.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:06:55 smithi115 ceph-mon[78593]: pgmap v1526: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-05-14T00:06:55.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:06:55 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:06:55.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:06:55 smithi047 ceph-mon[77147]: pgmap v1526: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-05-14T00:06:55.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:06:55 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:06:56.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:06:56 smithi115 conmon[90393]: 2022-05-14T00:06:56.047+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:06:56.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:06:56 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:06:56.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:06:56 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:06:57.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:06:57 smithi115 conmon[90393]: 2022-05-14T00:06:57.073+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:06:57.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:06:57 smithi115 ceph-mon[78593]: pgmap v1527: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:06:57.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:06:57 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:06:57.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:06:57 smithi047 ceph-mon[77147]: pgmap v1527: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:06:57.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:06:57 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:06:58.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:06:58 smithi115 conmon[90393]: 2022-05-14T00:06:58.077+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:06:58.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:06:58 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:06:58.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:06:58 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:06:58.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:06:58 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:06:58.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:06:58 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:06:58.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:06:58 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:06:58.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:06:58 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:06:59.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:06:59 smithi115 conmon[90393]: 2022-05-14T00:06:59.064+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:06:59.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:06:59 smithi115 ceph-mon[78593]: pgmap v1528: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-05-14T00:06:59.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:06:59 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:06:59.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:06:59 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 2350 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:06:59.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:06:59 smithi047 ceph-mon[77147]: pgmap v1528: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-05-14T00:06:59.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:06:59 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:06:59.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:06:59 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 2350 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:07:00.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:07:00 smithi115 conmon[90393]: 2022-05-14T00:07:00.039+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:07:00.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:07:00 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:07:00.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:07:00 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:07:01.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:07:01 smithi115 conmon[90393]: 2022-05-14T00:07:01.007+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:07:01.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:07:01 smithi115 ceph-mon[78593]: pgmap v1529: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:07:01.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:07:01 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:07:01.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:07:01 smithi047 ceph-mon[77147]: pgmap v1529: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:07:01.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:07:01 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:07:02.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:07:01 smithi115 conmon[90393]: 2022-05-14T00:07:01.989+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:07:02.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:07:02 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:07:02.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:07:02 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:07:02.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:07:02 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:07:02.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:07:02 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:07:02.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:07:02 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:07:02.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:07:02 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:07:03.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:07:03 smithi115 conmon[90393]: 2022-05-14T00:07:03.024+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:07:03.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:07:03 smithi115 ceph-mon[78593]: pgmap v1530: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:07:03.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:07:03 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:07:03.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:07:03 smithi047 ceph-mon[77147]: pgmap v1530: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:07:03.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:07:03 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:07:04.360 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:07:04 smithi115 conmon[90393]: 2022-05-14T00:07:04.005+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:07:04.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:07:04 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:07:04.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:07:04 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 2355 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:07:04.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:07:04 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:07:04.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:07:04 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 2355 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:07:05.234 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:07:04 smithi115 conmon[90393]: 2022-05-14T00:07:04.982+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:07:05.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:07:05 smithi047 ceph-mon[77147]: pgmap v1531: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:07:05.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:07:05 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:07:05.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:07:05 smithi115 ceph-mon[78593]: pgmap v1531: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:07:05.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:07:05 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:07:06.239 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:07:05 smithi115 conmon[90393]: 2022-05-14T00:07:05.955+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:07:06.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:07:06 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:07:06.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:07:06 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:07:07.240 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:07:06 smithi115 conmon[90393]: 2022-05-14T00:07:06.989+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:07:07.241 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:07:07 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:07:07.241 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:07:07 smithi115 ceph-mon[78593]: pgmap v1532: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:07:07.546 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:07:07 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:07:07.547 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:07:07 smithi047 ceph-mon[77147]: pgmap v1532: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:07:08.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:07:08 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:07:08.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:07:08 smithi115 conmon[90393]: 2022-05-14T00:07:08.016+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:07:08.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:07:08 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:07:09.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:07:09 smithi115 conmon[90393]: 2022-05-14T00:07:09.036+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:07:09.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:07:09 smithi115 ceph-mon[78593]: pgmap v1533: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:07:09.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:07:09 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:07:09.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:07:09 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 2360 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:07:09.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:07:09 smithi047 ceph-mon[77147]: pgmap v1533: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:07:09.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:07:09 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:07:09.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:07:09 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 2360 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:07:10.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:07:10 smithi115 conmon[90393]: 2022-05-14T00:07:10.032+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:07:10.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:07:10 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:07:10.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:07:10 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:07:10.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:07:10 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:07:10.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:07:10 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:07:10.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:07:10 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:07:10.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:07:10 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:07:11.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:07:11 smithi115 conmon[90393]: 2022-05-14T00:07:11.058+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:07:11.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:07:11 smithi115 ceph-mon[78593]: pgmap v1534: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:07:11.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:07:11 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:07:11.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:07:11 smithi047 ceph-mon[77147]: pgmap v1534: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:07:11.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:07:11 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:07:12.265 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:07:12 smithi115 conmon[90393]: 2022-05-14T00:07:12.011+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:07:12.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:07:12 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:07:12.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:07:12 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:07:13.267 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:07:13 smithi115 conmon[90393]: 2022-05-14T00:07:13.004+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:07:13.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:07:13 smithi047 ceph-mon[77147]: pgmap v1535: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:07:13.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:07:13 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:07:13.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:07:13 smithi115 ceph-mon[78593]: pgmap v1535: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:07:13.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:07:13 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:07:14.272 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:07:14 smithi115 conmon[90393]: 2022-05-14T00:07:14.010+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:07:14.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:07:14 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:07:14.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:07:14 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:07:14.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:07:14 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:07:14.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:07:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T00:07:14.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:07:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T00:07:14.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:07:14 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 2365 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:07:14.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:07:14 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:07:14.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:07:14 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:07:14.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:07:14 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:07:14.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:07:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T00:07:14.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:07:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T00:07:14.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:07:14 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 2365 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:07:15.274 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:07:15 smithi115 conmon[90393]: 2022-05-14T00:07:14.999+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:07:15.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:07:15 smithi047 ceph-mon[77147]: pgmap v1536: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:07:15.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:07:15 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:07:15.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:07:15 smithi115 ceph-mon[78593]: pgmap v1536: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:07:15.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:07:15 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:07:16.279 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:07:15 smithi115 conmon[90393]: 2022-05-14T00:07:15.980+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:07:16.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:07:16 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:07:16.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:07:16 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:07:17.281 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:07:16 smithi115 conmon[90393]: 2022-05-14T00:07:16.942+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:07:17.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:07:17 smithi047 ceph-mon[77147]: pgmap v1537: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:07:17.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:07:17 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:07:17.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:07:17 smithi115 ceph-mon[78593]: pgmap v1537: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:07:17.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:07:17 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:07:18.292 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:07:17 smithi115 conmon[90393]: 2022-05-14T00:07:17.931+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:07:18.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:07:18 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:07:18.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:07:18 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:07:18.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:07:18 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:07:18.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:07:18 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:07:18.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:07:18 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:07:18.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:07:18 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:07:19.294 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:07:18 smithi115 conmon[90393]: 2022-05-14T00:07:18.924+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:07:19.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:07:19 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:07:19.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:07:19 smithi115 ceph-mon[78593]: pgmap v1538: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:07:19.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:07:19 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 2370 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:07:19.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:07:19 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:07:19.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:07:19 smithi047 ceph-mon[77147]: pgmap v1538: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:07:19.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:07:19 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 2370 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:07:20.301 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:07:19 smithi115 conmon[90393]: 2022-05-14T00:07:19.879+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:07:20.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:07:20 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:07:20.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:07:20 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:07:21.301 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:07:20 smithi115 conmon[90393]: 2022-05-14T00:07:20.868+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:07:21.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:07:21 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:07:21.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:07:21 smithi115 ceph-mon[78593]: pgmap v1539: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-05-14T00:07:21.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:07:21 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:07:21.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:07:21 smithi047 ceph-mon[77147]: pgmap v1539: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-05-14T00:07:22.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:07:21 smithi115 conmon[90393]: 2022-05-14T00:07:21.835+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:07:22.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:07:22 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:07:22.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:07:22 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:07:22.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:07:22 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:07:22.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:07:22 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:07:22.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:07:22 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:07:22.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:07:22 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:07:23.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:07:22 smithi115 conmon[90393]: 2022-05-14T00:07:22.842+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:07:23.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:07:23 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:07:23.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:07:23 smithi115 ceph-mon[78593]: pgmap v1540: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.3 KiB/s rd, 4 op/s 2022-05-14T00:07:23.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:07:23 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:07:23.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:07:23 smithi047 ceph-mon[77147]: pgmap v1540: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.3 KiB/s rd, 4 op/s 2022-05-14T00:07:24.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:07:23 smithi115 conmon[90393]: 2022-05-14T00:07:23.841+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:07:24.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:07:24 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:07:24.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:07:24 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 2375 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:07:24.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:07:24 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:07:24.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:07:24 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 2375 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:07:25.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:07:24 smithi115 conmon[90393]: 2022-05-14T00:07:24.842+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:07:25.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:07:25 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:07:25.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:07:25 smithi115 ceph-mon[78593]: pgmap v1541: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.3 KiB/s rd, 4 op/s 2022-05-14T00:07:25.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:07:25 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:07:25.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:07:25 smithi047 ceph-mon[77147]: pgmap v1541: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.3 KiB/s rd, 4 op/s 2022-05-14T00:07:26.321 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:07:25 smithi115 conmon[90393]: 2022-05-14T00:07:25.860+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:07:26.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:07:26 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:07:26.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:07:26 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T00:07:26.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:07:26 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T00:07:26.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:07:26 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T00:07:26.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:07:26 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:07:26.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:07:26 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T00:07:26.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:07:26 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T00:07:26.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:07:26 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T00:07:27.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:07:26 smithi115 conmon[90393]: 2022-05-14T00:07:26.822+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:07:27.291 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:07:26 smithi047 conmon[78521]: 2022-05-14T00:07:26.895+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:07:27.291 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:07:26 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T00:07:27.292 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:07:26 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:07:27.292 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:07:26 smithi047 conmon[78521]: yield 2022-05-14T00:07:27.292 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:07:26 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:07:27.292 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:07:26 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:07:27.293 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:07:26 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:07:27.293 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:07:26 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T00:07:27.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:07:26 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:07:27.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:07:26 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T00:07:27.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:07:26 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:07:27.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:07:26 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T00:07:27.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:07:26 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:07:27.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:07:26 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T00:07:27.297 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:07:26 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:07:27.297 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:07:26 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T00:07:27.298 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:07:26 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:07:27.298 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:07:26 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:07:27.299 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:07:26 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:07:27.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:07:27 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:07:27.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:07:27 smithi115 ceph-mon[78593]: pgmap v1542: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-05-14T00:07:27.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:07:27 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:07:27.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:07:27 smithi047 ceph-mon[77147]: pgmap v1542: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-05-14T00:07:28.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:07:27 smithi115 conmon[90393]: 2022-05-14T00:07:27.830+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:07:28.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:07:28 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:07:28.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:07:28 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:07:28.609 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T00:07:28.609 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:07:28.609 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T00:07:28.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:07:28.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:07:28.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:07:28.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T00:07:28.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:07:28.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T00:07:28.613 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:07:28.613 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T00:07:28.614 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:07:28.614 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T00:07:28.614 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:07:28.615 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T00:07:28.615 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:07:28.615 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:07:28.615 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:07:28.616 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:07:28 smithi115 ceph-mon[78593]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T00:07:28.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:07:28 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:07:28.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:07:28 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:07:28.791 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T00:07:28.791 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:07:28.792 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T00:07:28.792 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:07:28.792 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:07:28.792 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:07:28.793 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T00:07:28.793 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:07:28.793 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T00:07:28.793 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:07:28.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T00:07:28.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:07:28.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T00:07:28.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:07:28.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T00:07:28.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:07:28.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:07:28.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:07:28.795 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:07:28 smithi047 ceph-mon[77147]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T00:07:29.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:07:28 smithi115 conmon[90393]: 2022-05-14T00:07:28.827+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:07:29.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:07:29 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:07:29.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:07:29 smithi115 ceph-mon[78593]: pgmap v1543: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.3 KiB/s rd, 4 op/s 2022-05-14T00:07:29.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:07:29 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 2380 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:07:29.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:07:29 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:07:29.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:07:29 smithi047 ceph-mon[77147]: pgmap v1543: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.3 KiB/s rd, 4 op/s 2022-05-14T00:07:29.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:07:29 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 2380 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:07:30.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:07:29 smithi115 conmon[90393]: 2022-05-14T00:07:29.854+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:07:30.291 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:07:29 smithi047 conmon[78521]: 2022-05-14T00:07:29.967+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:07:30.292 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:07:29 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T00:07:30.292 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:07:29 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:07:30.292 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:07:29 smithi047 conmon[78521]: yield 2022-05-14T00:07:30.292 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:07:29 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:07:30.292 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:07:29 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:07:30.293 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:07:29 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:07:30.293 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:07:29 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T00:07:30.293 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:07:29 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:07:30.293 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:07:29 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T00:07:30.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:07:29 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:07:30.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:07:29 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T00:07:30.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:07:29 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:07:30.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:07:29 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T00:07:30.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:07:29 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:07:30.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:07:29 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T00:07:30.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:07:29 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:07:30.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:07:29 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:07:30.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:07:29 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:07:30.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:07:29 smithi047 conmon[78521]: 2022-05-14T00:07:29.968+0000 7f5344781700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T00:07:30.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:07:29 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T00:07:30.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:07:29 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:07:30.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:07:29 smithi047 conmon[78521]: yield 2022-05-14T00:07:30.297 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:07:29 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:07:30.297 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:07:29 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:07:30.297 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:07:29 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:07:30.297 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:07:29 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T00:07:30.298 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:07:29 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:07:30.298 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:07:29 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T00:07:30.298 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:07:29 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:07:30.299 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:07:29 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T00:07:30.299 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:07:29 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:07:30.299 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:07:29 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T00:07:30.299 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:07:29 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:07:30.300 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:07:29 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T00:07:30.300 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:07:29 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:07:30.300 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:07:29 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:07:30.300 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:07:29 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:07:30.301 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:07:29 smithi047 conmon[78521]: 2022-05-14T00:07:30.301 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:07:29 smithi047 conmon[78521]: During handling of the above exception, another exception occurred: 2022-05-14T00:07:30.301 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:07:29 smithi047 conmon[78521]: 2022-05-14T00:07:30.301 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:07:29 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T00:07:30.302 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:07:29 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T00:07:30.302 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:07:29 smithi047 conmon[78521]: return f(*arg) 2022-05-14T00:07:30.302 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:07:29 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T00:07:30.302 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:07:29 smithi047 conmon[78521]: self._write_client_files(client_files, host) 2022-05-14T00:07:30.303 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:07:29 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T00:07:30.303 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:07:29 smithi047 conmon[78521]: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T00:07:30.303 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:07:29 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T00:07:30.304 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:07:29 smithi047 conmon[78521]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T00:07:30.304 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:07:29 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T00:07:30.304 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:07:29 smithi047 conmon[78521]: return self.event_loop.get_result(coro) 2022-05-14T00:07:30.304 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:07:29 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T00:07:30.305 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:07:29 smithi047 conmon[78521]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T00:07:30.305 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:07:29 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T00:07:30.305 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:07:29 smithi047 conmon[78521]: return self.__get_result() 2022-05-14T00:07:30.305 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:07:29 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T00:07:30.306 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:07:29 smithi047 conmon[78521]: raise self._exception 2022-05-14T00:07:30.306 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:07:29 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T00:07:30.306 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:07:29 smithi047 conmon[78521]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T00:07:30.306 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:07:29 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T00:07:30.307 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:07:29 smithi047 conmon[78521]: conn = await self._remote_connection(host, addr) 2022-05-14T00:07:30.307 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:07:29 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T00:07:30.307 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:07:29 smithi047 conmon[78521]: raise 2022-05-14T00:07:30.307 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:07:29 smithi047 conmon[78521]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T00:07:30.307 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:07:29 smithi047 conmon[78521]: self.gen.throw(type, value, traceback) 2022-05-14T00:07:30.308 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:07:29 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T00:07:30.308 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:07:29 smithi047 conmon[78521]: raise OrchestratorError(msg) 2022-05-14T00:07:30.308 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:07:29 smithi047 conmon[78521]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:07:30.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:07:30 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:07:30.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:07:30 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:07:30.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:07:30 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:07:30.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:07:30 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:07:30.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:07:30 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:07:30.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:07:30 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:07:31.340 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:07:30 smithi115 conmon[90393]: 2022-05-14T00:07:30.885+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:07:31.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:07:31 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:07:31.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:07:31 smithi115 ceph-mon[78593]: pgmap v1544: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-05-14T00:07:31.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:07:31 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:07:31.609 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T00:07:31.609 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:07:31.609 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T00:07:31.609 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:07:31.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:07:31.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:07:31.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T00:07:31.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:07:31.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T00:07:31.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:07:31.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T00:07:31.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:07:31.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T00:07:31.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:07:31.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T00:07:31.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:07:31.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:07:31.613 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:07:31.613 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:07:31 smithi115 ceph-mon[78593]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T00:07:31.613 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T00:07:31.613 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:07:31.613 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T00:07:31.614 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:07:31.614 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:07:31.614 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:07:31.614 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T00:07:31.615 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:07:31.615 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T00:07:31.615 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:07:31.615 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T00:07:31.615 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:07:31.616 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T00:07:31.616 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:07:31.616 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T00:07:31.616 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:07:31.617 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:07:31.617 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:07:31.617 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T00:07:31.617 INFO:journalctl@ceph.mon.c.smithi115.stdout: During handling of the above exception, another exception occurred: 2022-05-14T00:07:31.618 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T00:07:31.618 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T00:07:31.618 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T00:07:31.618 INFO:journalctl@ceph.mon.c.smithi115.stdout: return f(*arg) 2022-05-14T00:07:31.618 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T00:07:31.619 INFO:journalctl@ceph.mon.c.smithi115.stdout: self._write_client_files(client_files, host) 2022-05-14T00:07:31.619 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T00:07:31.619 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T00:07:31.619 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T00:07:31.626 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T00:07:31.627 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T00:07:31.627 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.event_loop.get_result(coro) 2022-05-14T00:07:31.627 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T00:07:31.627 INFO:journalctl@ceph.mon.c.smithi115.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T00:07:31.628 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T00:07:31.628 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.__get_result() 2022-05-14T00:07:31.628 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T00:07:31.628 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise self._exception 2022-05-14T00:07:31.628 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T00:07:31.629 INFO:journalctl@ceph.mon.c.smithi115.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T00:07:31.629 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T00:07:31.629 INFO:journalctl@ceph.mon.c.smithi115.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T00:07:31.629 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T00:07:31.630 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise 2022-05-14T00:07:31.630 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T00:07:31.630 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.gen.throw(type, value, traceback) 2022-05-14T00:07:31.630 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T00:07:31.630 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OrchestratorError(msg) 2022-05-14T00:07:31.631 INFO:journalctl@ceph.mon.c.smithi115.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:07:31.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:07:31 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:07:31.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:07:31 smithi047 ceph-mon[77147]: pgmap v1544: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-05-14T00:07:31.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:07:31 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:07:31.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T00:07:31.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:07:31.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T00:07:31.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:07:31.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:07:31.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:07:31.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T00:07:31.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:07:31.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T00:07:31.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:07:31.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T00:07:31.797 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:07:31.797 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T00:07:31.797 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:07:31.797 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T00:07:31.798 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:07:31.798 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:07:31.799 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:07:31.799 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:07:31 smithi047 ceph-mon[77147]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T00:07:31.799 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T00:07:31.799 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:07:31.800 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T00:07:31.800 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:07:31.800 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:07:31.800 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:07:31.801 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T00:07:31.801 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:07:31.801 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T00:07:31.801 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:07:31.801 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T00:07:31.802 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:07:31.802 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T00:07:31.802 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:07:31.802 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T00:07:31.803 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:07:31.803 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:07:31.803 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:07:31.803 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T00:07:31.803 INFO:journalctl@ceph.mon.b.smithi047.stdout: During handling of the above exception, another exception occurred: 2022-05-14T00:07:31.804 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T00:07:31.804 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T00:07:31.804 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T00:07:31.804 INFO:journalctl@ceph.mon.b.smithi047.stdout: return f(*arg) 2022-05-14T00:07:31.804 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T00:07:31.805 INFO:journalctl@ceph.mon.b.smithi047.stdout: self._write_client_files(client_files, host) 2022-05-14T00:07:31.805 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T00:07:31.805 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T00:07:31.805 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T00:07:31.806 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T00:07:31.806 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T00:07:31.806 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.event_loop.get_result(coro) 2022-05-14T00:07:31.806 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T00:07:31.806 INFO:journalctl@ceph.mon.b.smithi047.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T00:07:31.807 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T00:07:31.807 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.__get_result() 2022-05-14T00:07:31.807 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T00:07:31.807 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise self._exception 2022-05-14T00:07:31.807 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T00:07:31.808 INFO:journalctl@ceph.mon.b.smithi047.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T00:07:31.808 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T00:07:31.808 INFO:journalctl@ceph.mon.b.smithi047.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T00:07:31.808 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T00:07:31.809 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise 2022-05-14T00:07:31.809 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T00:07:31.809 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.gen.throw(type, value, traceback) 2022-05-14T00:07:31.809 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T00:07:31.810 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OrchestratorError(msg) 2022-05-14T00:07:31.810 INFO:journalctl@ceph.mon.b.smithi047.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:07:32.348 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:07:31 smithi115 conmon[90393]: 2022-05-14T00:07:31.922+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:07:32.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:07:32 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:07:32.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:07:32 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:07:33.348 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:07:32 smithi115 conmon[90393]: 2022-05-14T00:07:32.880+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:07:33.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:07:33 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:07:33.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:07:33 smithi115 ceph-mon[78593]: pgmap v1545: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:07:33.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:07:33 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:07:33.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:07:33 smithi047 ceph-mon[77147]: pgmap v1545: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:07:34.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:07:33 smithi115 conmon[90393]: 2022-05-14T00:07:33.852+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:07:34.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:07:34 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:07:34.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:07:34 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:07:34.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:07:34 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:07:34.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:07:34 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 2385 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:07:34.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:07:34 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:07:34.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:07:34 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:07:34.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:07:34 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:07:34.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:07:34 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 2385 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:07:35.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:07:34 smithi115 conmon[90393]: 2022-05-14T00:07:34.834+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:07:35.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:07:35 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:07:35.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:07:35 smithi115 ceph-mon[78593]: pgmap v1546: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:07:35.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:07:35 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:07:35.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:07:35 smithi047 ceph-mon[77147]: pgmap v1546: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:07:36.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:07:35 smithi115 conmon[90393]: 2022-05-14T00:07:35.864+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:07:36.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:07:36 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:07:36.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:07:36 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:07:37.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:07:36 smithi115 conmon[90393]: 2022-05-14T00:07:36.887+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:07:37.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:07:37 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:07:37.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:07:37 smithi047 ceph-mon[77147]: pgmap v1547: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:07:37.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:07:37 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:07:37.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:07:37 smithi115 ceph-mon[78593]: pgmap v1547: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:07:38.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:07:37 smithi115 conmon[90393]: 2022-05-14T00:07:37.878+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:07:38.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:07:38 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:07:38.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:07:38 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:07:38.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:07:38 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:07:38.835 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:07:38 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:07:38.836 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:07:38 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:07:38.836 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:07:38 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:07:39.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:07:38 smithi115 conmon[90393]: 2022-05-14T00:07:38.834+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:07:39.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:07:39 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:07:39.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:07:39 smithi047 ceph-mon[77147]: pgmap v1548: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:07:39.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:07:39 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 2390 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:07:39.838 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:07:39 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:07:39.839 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:07:39 smithi115 ceph-mon[78593]: pgmap v1548: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:07:39.839 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:07:39 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 2390 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:07:40.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:07:39 smithi115 conmon[90393]: 2022-05-14T00:07:39.837+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:07:40.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:07:40 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:07:40.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:07:40 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:07:41.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:07:40 smithi115 conmon[90393]: 2022-05-14T00:07:40.795+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:07:41.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:07:41 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:07:41.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:07:41 smithi047 ceph-mon[77147]: pgmap v1549: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:07:41.821 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:07:41 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:07:41.821 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:07:41 smithi115 ceph-mon[78593]: pgmap v1549: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:07:42.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:07:41 smithi115 conmon[90393]: 2022-05-14T00:07:41.820+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:07:42.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:07:42 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:07:42.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:07:42 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:07:42.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:07:42 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:07:42.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:07:42 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:07:42.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:07:42 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:07:42.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:07:42 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:07:43.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:07:42 smithi115 conmon[90393]: 2022-05-14T00:07:42.862+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:07:43.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:07:43 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:07:43.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:07:43 smithi047 ceph-mon[77147]: pgmap v1550: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:07:43.851 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:07:43 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:07:43.851 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:07:43 smithi115 ceph-mon[78593]: pgmap v1550: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:07:44.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:07:43 smithi115 conmon[90393]: 2022-05-14T00:07:43.850+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:07:44.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:07:44 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:07:44.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:07:44 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 2395 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:07:44.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:07:44 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:07:44.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:07:44 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 2395 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:07:45.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:07:44 smithi115 conmon[90393]: 2022-05-14T00:07:44.884+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:07:45.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:07:45 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:07:45.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:07:45 smithi047 ceph-mon[77147]: pgmap v1551: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:07:45.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:07:45 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:07:45.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:07:45 smithi115 ceph-mon[78593]: pgmap v1551: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:07:46.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:07:45 smithi115 conmon[90393]: 2022-05-14T00:07:45.878+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:07:46.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:07:46 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:07:46.845 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:07:46 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:07:47.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:07:46 smithi115 conmon[90393]: 2022-05-14T00:07:46.844+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:07:47.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:07:47 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:07:47.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:07:47 smithi047 ceph-mon[77147]: pgmap v1552: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:07:47.812 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:07:47 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:07:47.812 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:07:47 smithi115 ceph-mon[78593]: pgmap v1552: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:07:48.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:07:47 smithi115 conmon[90393]: 2022-05-14T00:07:47.811+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:07:48.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:07:48 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:07:48.817 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:07:48 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:07:49.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:07:48 smithi115 conmon[90393]: 2022-05-14T00:07:48.816+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:07:49.786 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:07:49 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:07:49.786 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:07:49 smithi115 ceph-mon[78593]: pgmap v1553: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:07:49.787 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:07:49 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 2400 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:07:49.799 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:07:49 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:07:49.800 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:07:49 smithi047 ceph-mon[77147]: pgmap v1553: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:07:49.800 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:07:49 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 2400 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:07:50.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:07:49 smithi115 conmon[90393]: 2022-05-14T00:07:49.785+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:07:50.765 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:07:50 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:07:50.766 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:07:50 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:07:50.766 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:07:50 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:07:50.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:07:50 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:07:50.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:07:50 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:07:50.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:07:50 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:07:51.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:07:50 smithi115 conmon[90393]: 2022-05-14T00:07:50.765+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:07:51.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:07:51 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:07:51.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:07:51 smithi047 ceph-mon[77147]: pgmap v1554: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:07:51.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:07:51 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:07:51.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:07:51 smithi115 ceph-mon[78593]: pgmap v1554: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:07:52.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:07:51 smithi115 conmon[90393]: 2022-05-14T00:07:51.791+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:07:52.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:07:52 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:07:52.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:07:52 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:07:53.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:07:52 smithi115 conmon[90393]: 2022-05-14T00:07:52.791+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:07:53.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:07:53 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:07:53.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:07:53 smithi047 ceph-mon[77147]: pgmap v1555: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:07:53.829 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:07:53 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:07:53.830 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:07:53 smithi115 ceph-mon[78593]: pgmap v1555: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:07:54.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:07:53 smithi115 conmon[90393]: 2022-05-14T00:07:53.828+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:07:54.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:07:54 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:07:54.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:07:54 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:07:54.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:07:54 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:07:54.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:07:54 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 2405 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:07:54.811 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:07:54 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:07:54.812 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:07:54 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:07:54.812 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:07:54 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:07:54.812 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:07:54 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 2405 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:07:55.110 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:07:54 smithi115 conmon[90393]: 2022-05-14T00:07:54.811+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:07:55.778 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:07:55 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:07:55.779 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:07:55 smithi115 ceph-mon[78593]: pgmap v1556: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:07:55.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:07:55 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:07:55.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:07:55 smithi047 ceph-mon[77147]: pgmap v1556: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:07:56.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:07:55 smithi115 conmon[90393]: 2022-05-14T00:07:55.778+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:07:56.755 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:07:56 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:07:56.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:07:56 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:07:57.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:07:56 smithi115 conmon[90393]: 2022-05-14T00:07:56.754+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:07:57.713 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:07:57 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:07:57.714 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:07:57 smithi115 ceph-mon[78593]: pgmap v1557: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:07:57.714 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:07:57 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:07:57.714 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:07:57 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:07:57.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:07:57 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:07:57.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:07:57 smithi047 ceph-mon[77147]: pgmap v1557: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:07:57.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:07:57 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:07:57.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:07:57 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:07:58.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:07:57 smithi115 conmon[90393]: 2022-05-14T00:07:57.712+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:07:58.753 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:07:58 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:07:58.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:07:58 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:07:59.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:07:58 smithi115 conmon[90393]: 2022-05-14T00:07:58.748+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:07:59.717 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:07:59 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:07:59.717 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:07:59 smithi115 ceph-mon[78593]: pgmap v1558: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:07:59.718 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:07:59 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 2410 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:07:59.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:07:59 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:07:59.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:07:59 smithi047 ceph-mon[77147]: pgmap v1558: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:07:59.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:07:59 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 2410 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:08:00.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:07:59 smithi115 conmon[90393]: 2022-05-14T00:07:59.716+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:08:00.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:08:00 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:08:00.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:08:00 smithi115 conmon[90393]: 2022-05-14T00:08:00.680+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:08:00.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:08:00 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:08:01.714 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:08:01 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:08:01.714 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:08:01 smithi115 ceph-mon[78593]: pgmap v1559: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:08:01.714 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:08:01 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:08:01.715 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:08:01 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:08:01.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:08:01 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:08:01.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:08:01 smithi047 ceph-mon[77147]: pgmap v1559: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:08:01.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:08:01 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:08:01.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:08:01 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:08:02.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:08:01 smithi115 conmon[90393]: 2022-05-14T00:08:01.713+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:08:02.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:08:02 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:08:02.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:08:02 smithi115 conmon[90393]: 2022-05-14T00:08:02.685+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:08:02.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:08:02 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:08:03.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:08:03 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:08:03.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:08:03 smithi047 ceph-mon[77147]: pgmap v1560: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:08:03.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:08:03 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:08:03.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:08:03 smithi115 conmon[90393]: 2022-05-14T00:08:03.650+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:08:03.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:08:03 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:08:03.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:08:03 smithi115 ceph-mon[78593]: pgmap v1560: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:08:03.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:08:03 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:08:04.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:08:04 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 2415 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:08:04.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:08:04 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:08:04.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:08:04 smithi115 conmon[90393]: 2022-05-14T00:08:04.639+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:08:04.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:08:04 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 2415 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:08:04.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:08:04 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:08:06.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:08:05 smithi047 ceph-mon[77147]: pgmap v1561: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:08:06.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:08:05 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:08:06.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:08:05 smithi115 conmon[90393]: 2022-05-14T00:08:05.675+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:08:06.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:08:05 smithi115 ceph-mon[78593]: pgmap v1561: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:08:06.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:08:05 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:08:07.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:08:06 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:08:07.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:08:06 smithi115 conmon[90393]: 2022-05-14T00:08:06.626+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:08:07.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:08:06 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:08:07.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:08:07 smithi115 conmon[90393]: 2022-05-14T00:08:07.582+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:08:07.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:08:07 smithi115 ceph-mon[78593]: pgmap v1562: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:08:07.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:08:07 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:08:08.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:08:07 smithi047 ceph-mon[77147]: pgmap v1562: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:08:08.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:08:07 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:08:08.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:08:08 smithi115 conmon[90393]: 2022-05-14T00:08:08.564+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:08:08.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:08:08 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:08:08.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:08:08 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 2420 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:08:09.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:08:08 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:08:09.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:08:08 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 2420 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:08:09.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:08:09 smithi115 conmon[90393]: 2022-05-14T00:08:09.593+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:08:09.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:08:09 smithi115 ceph-mon[78593]: pgmap v1563: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:08:09.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:08:09 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:08:09.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:08:09 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:08:09.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:08:09 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:08:10.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:08:09 smithi047 ceph-mon[77147]: pgmap v1563: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:08:10.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:08:09 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:08:10.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:08:09 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:08:10.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:08:09 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:08:11.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:08:10 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:08:11.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:08:10 smithi115 conmon[90393]: 2022-05-14T00:08:10.635+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:08:11.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:08:10 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:08:12.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:08:11 smithi047 ceph-mon[77147]: pgmap v1564: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:08:12.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:08:11 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:08:12.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:08:11 smithi115 conmon[90393]: 2022-05-14T00:08:11.636+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:08:12.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:08:11 smithi115 ceph-mon[78593]: pgmap v1564: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:08:12.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:08:11 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:08:13.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:08:12 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:08:13.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:08:12 smithi115 conmon[90393]: 2022-05-14T00:08:12.660+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:08:13.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:08:12 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:08:14.041 INFO:journalctl@ceph.osd.1.smithi047.stdout:May 14 00:08:13 smithi047 sudo[118325]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2022-05-14T00:08:14.041 INFO:journalctl@ceph.osd.1.smithi047.stdout:May 14 00:08:13 smithi047 sudo[118325]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2022-05-14T00:08:14.041 INFO:journalctl@ceph.osd.1.smithi047.stdout:May 14 00:08:13 smithi047 sudo[118325]: pam_unix(sudo:session): session opened for user root by (uid=0) 2022-05-14T00:08:14.041 INFO:journalctl@ceph.osd.1.smithi047.stdout:May 14 00:08:13 smithi047 sudo[118325]: pam_unix(sudo:session): session closed for user root 2022-05-14T00:08:14.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:08:13 smithi047 ceph-mon[77147]: pgmap v1565: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:08:14.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:08:13 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:08:14.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:08:13 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:08:14.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:08:13 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:08:14.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:08:13 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T00:08:14.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:08:13 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T00:08:14.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:08:13 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 2425 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:08:14.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:08:13 smithi115 conmon[90393]: 2022-05-14T00:08:13.679+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:08:14.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:08:13 smithi115 ceph-mon[78593]: pgmap v1565: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:08:14.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:08:13 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:08:14.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:08:13 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:08:14.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:08:13 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:08:14.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:08:13 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T00:08:14.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:08:13 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T00:08:14.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:08:13 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 2425 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:08:14.540 INFO:journalctl@ceph.osd.1.smithi047.stdout:May 14 00:08:14 smithi047 sudo[118328]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2022-05-14T00:08:14.541 INFO:journalctl@ceph.osd.1.smithi047.stdout:May 14 00:08:14 smithi047 sudo[118328]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2022-05-14T00:08:14.541 INFO:journalctl@ceph.osd.1.smithi047.stdout:May 14 00:08:14 smithi047 sudo[118328]: pam_unix(sudo:session): session opened for user root by (uid=0) 2022-05-14T00:08:14.541 INFO:journalctl@ceph.osd.1.smithi047.stdout:May 14 00:08:14 smithi047 sudo[118328]: pam_unix(sudo:session): session closed for user root 2022-05-14T00:08:14.599 INFO:journalctl@ceph.osd.2.smithi115.stdout:May 14 00:08:14 smithi115 sudo[120318]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2022-05-14T00:08:14.599 INFO:journalctl@ceph.osd.2.smithi115.stdout:May 14 00:08:14 smithi115 sudo[120318]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2022-05-14T00:08:14.599 INFO:journalctl@ceph.osd.2.smithi115.stdout:May 14 00:08:14 smithi115 sudo[120318]: pam_unix(sudo:session): session opened for user root by (uid=0) 2022-05-14T00:08:14.599 INFO:journalctl@ceph.osd.2.smithi115.stdout:May 14 00:08:14 smithi115 sudo[120318]: pam_unix(sudo:session): session closed for user root 2022-05-14T00:08:14.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:08:14 smithi115 conmon[90393]: 2022-05-14T00:08:14.633+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:08:14.858 INFO:journalctl@ceph.osd.2.smithi115.stdout:May 14 00:08:14 smithi115 sudo[120321]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2022-05-14T00:08:14.858 INFO:journalctl@ceph.osd.2.smithi115.stdout:May 14 00:08:14 smithi115 sudo[120321]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2022-05-14T00:08:14.858 INFO:journalctl@ceph.osd.2.smithi115.stdout:May 14 00:08:14 smithi115 sudo[120321]: pam_unix(sudo:session): session opened for user root by (uid=0) 2022-05-14T00:08:14.859 INFO:journalctl@ceph.osd.2.smithi115.stdout:May 14 00:08:14 smithi115 sudo[120321]: pam_unix(sudo:session): session closed for user root 2022-05-14T00:08:15.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:08:15 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:08:15.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:08:15 smithi115 ceph-mon[78593]: pgmap v1566: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:08:15.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:08:15 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' 2022-05-14T00:08:15.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:08:15 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' 2022-05-14T00:08:15.521 INFO:journalctl@ceph.osd.4.smithi047.stdout:May 14 00:08:15 smithi047 sudo[118331]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2022-05-14T00:08:15.522 INFO:journalctl@ceph.osd.4.smithi047.stdout:May 14 00:08:15 smithi047 sudo[118331]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2022-05-14T00:08:15.522 INFO:journalctl@ceph.osd.4.smithi047.stdout:May 14 00:08:15 smithi047 sudo[118331]: pam_unix(sudo:session): session opened for user root by (uid=0) 2022-05-14T00:08:15.522 INFO:journalctl@ceph.osd.4.smithi047.stdout:May 14 00:08:15 smithi047 sudo[118331]: pam_unix(sudo:session): session closed for user root 2022-05-14T00:08:15.523 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:08:15 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:08:15.523 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:08:15 smithi047 ceph-mon[77147]: pgmap v1566: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:08:15.523 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:08:15 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' 2022-05-14T00:08:15.523 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:08:15 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' 2022-05-14T00:08:15.791 INFO:journalctl@ceph.osd.4.smithi047.stdout:May 14 00:08:15 smithi047 sudo[118334]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2022-05-14T00:08:15.791 INFO:journalctl@ceph.osd.4.smithi047.stdout:May 14 00:08:15 smithi047 sudo[118334]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2022-05-14T00:08:15.792 INFO:journalctl@ceph.osd.4.smithi047.stdout:May 14 00:08:15 smithi047 sudo[118334]: pam_unix(sudo:session): session opened for user root by (uid=0) 2022-05-14T00:08:15.792 INFO:journalctl@ceph.osd.4.smithi047.stdout:May 14 00:08:15 smithi047 sudo[118334]: pam_unix(sudo:session): session closed for user root 2022-05-14T00:08:15.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:08:15 smithi115 conmon[90393]: 2022-05-14T00:08:15.599+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:08:15.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:08:15 smithi115 sudo[120324]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2022-05-14T00:08:15.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:08:15 smithi115 sudo[120324]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2022-05-14T00:08:15.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:08:15 smithi115 sudo[120324]: pam_unix(sudo:session): session opened for user root by (uid=0) 2022-05-14T00:08:15.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:08:15 smithi115 sudo[120324]: pam_unix(sudo:session): session closed for user root 2022-05-14T00:08:16.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:08:16 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:08:16.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:08:16 smithi115 sudo[120327]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2022-05-14T00:08:16.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:08:16 smithi115 sudo[120327]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2022-05-14T00:08:16.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:08:16 smithi115 sudo[120327]: pam_unix(sudo:session): session opened for user root by (uid=0) 2022-05-14T00:08:16.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:08:16 smithi115 sudo[120327]: pam_unix(sudo:session): session closed for user root 2022-05-14T00:08:16.480 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:08:16 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:08:16.791 INFO:journalctl@ceph.osd.7.smithi047.stdout:May 14 00:08:16 smithi047 sudo[118337]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2022-05-14T00:08:16.791 INFO:journalctl@ceph.osd.7.smithi047.stdout:May 14 00:08:16 smithi047 sudo[118337]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2022-05-14T00:08:16.792 INFO:journalctl@ceph.osd.7.smithi047.stdout:May 14 00:08:16 smithi047 sudo[118337]: pam_unix(sudo:session): session opened for user root by (uid=0) 2022-05-14T00:08:16.792 INFO:journalctl@ceph.osd.7.smithi047.stdout:May 14 00:08:16 smithi047 sudo[118337]: pam_unix(sudo:session): session closed for user root 2022-05-14T00:08:16.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:08:16 smithi115 conmon[90393]: 2022-05-14T00:08:16.577+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:08:17.291 INFO:journalctl@ceph.osd.7.smithi047.stdout:May 14 00:08:16 smithi047 sudo[118340]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2022-05-14T00:08:17.291 INFO:journalctl@ceph.osd.7.smithi047.stdout:May 14 00:08:16 smithi047 sudo[118340]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2022-05-14T00:08:17.291 INFO:journalctl@ceph.osd.7.smithi047.stdout:May 14 00:08:16 smithi047 sudo[118340]: pam_unix(sudo:session): session opened for user root by (uid=0) 2022-05-14T00:08:17.291 INFO:journalctl@ceph.osd.7.smithi047.stdout:May 14 00:08:16 smithi047 sudo[118340]: pam_unix(sudo:session): session closed for user root 2022-05-14T00:08:17.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:08:17 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:08:17.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:08:17 smithi047 ceph-mon[77147]: pgmap v1567: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 21 KiB/s rd, 33 KiB/s wr, 11 op/s 2022-05-14T00:08:17.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:08:17 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:08:17.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:08:17 smithi115 ceph-mon[78593]: pgmap v1567: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 21 KiB/s rd, 33 KiB/s wr, 11 op/s 2022-05-14T00:08:17.358 INFO:journalctl@ceph.osd.8.smithi115.stdout:May 14 00:08:17 smithi115 sudo[120330]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2022-05-14T00:08:17.358 INFO:journalctl@ceph.osd.8.smithi115.stdout:May 14 00:08:17 smithi115 sudo[120330]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2022-05-14T00:08:17.359 INFO:journalctl@ceph.osd.8.smithi115.stdout:May 14 00:08:17 smithi115 sudo[120330]: pam_unix(sudo:session): session opened for user root by (uid=0) 2022-05-14T00:08:17.359 INFO:journalctl@ceph.osd.8.smithi115.stdout:May 14 00:08:17 smithi115 sudo[120330]: pam_unix(sudo:session): session closed for user root 2022-05-14T00:08:17.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:08:17 smithi115 conmon[90393]: 2022-05-14T00:08:17.576+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:08:17.858 INFO:journalctl@ceph.osd.8.smithi115.stdout:May 14 00:08:17 smithi115 sudo[120333]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2022-05-14T00:08:17.859 INFO:journalctl@ceph.osd.8.smithi115.stdout:May 14 00:08:17 smithi115 sudo[120333]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2022-05-14T00:08:17.859 INFO:journalctl@ceph.osd.8.smithi115.stdout:May 14 00:08:17 smithi115 sudo[120333]: pam_unix(sudo:session): session opened for user root by (uid=0) 2022-05-14T00:08:17.862 INFO:journalctl@ceph.osd.8.smithi115.stdout:May 14 00:08:17 smithi115 sudo[120333]: pam_unix(sudo:session): session closed for user root 2022-05-14T00:08:18.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:08:18 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:08:18.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:08:18 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:08:18.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:08:18 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:08:18.292 INFO:journalctl@ceph.osd.10.smithi047.stdout:May 14 00:08:17 smithi047 sudo[118343]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2022-05-14T00:08:18.292 INFO:journalctl@ceph.osd.10.smithi047.stdout:May 14 00:08:17 smithi047 sudo[118343]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2022-05-14T00:08:18.292 INFO:journalctl@ceph.osd.10.smithi047.stdout:May 14 00:08:17 smithi047 sudo[118343]: pam_unix(sudo:session): session opened for user root by (uid=0) 2022-05-14T00:08:18.292 INFO:journalctl@ceph.osd.10.smithi047.stdout:May 14 00:08:17 smithi047 sudo[118343]: pam_unix(sudo:session): session closed for user root 2022-05-14T00:08:18.583 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:08:18 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:08:18.584 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:08:18 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:08:18.584 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:08:18 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:08:18.790 INFO:journalctl@ceph.osd.10.smithi047.stdout:May 14 00:08:18 smithi047 sudo[118346]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2022-05-14T00:08:18.791 INFO:journalctl@ceph.osd.10.smithi047.stdout:May 14 00:08:18 smithi047 sudo[118346]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2022-05-14T00:08:18.791 INFO:journalctl@ceph.osd.10.smithi047.stdout:May 14 00:08:18 smithi047 sudo[118346]: pam_unix(sudo:session): session opened for user root by (uid=0) 2022-05-14T00:08:18.791 INFO:journalctl@ceph.osd.10.smithi047.stdout:May 14 00:08:18 smithi047 sudo[118346]: pam_unix(sudo:session): session closed for user root 2022-05-14T00:08:18.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:08:18 smithi115 conmon[90393]: 2022-05-14T00:08:18.582+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:08:18.858 INFO:journalctl@ceph.osd.11.smithi115.stdout:May 14 00:08:18 smithi115 sudo[120336]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2022-05-14T00:08:18.858 INFO:journalctl@ceph.osd.11.smithi115.stdout:May 14 00:08:18 smithi115 sudo[120336]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2022-05-14T00:08:18.858 INFO:journalctl@ceph.osd.11.smithi115.stdout:May 14 00:08:18 smithi115 sudo[120336]: pam_unix(sudo:session): session opened for user root by (uid=0) 2022-05-14T00:08:18.859 INFO:journalctl@ceph.osd.11.smithi115.stdout:May 14 00:08:18 smithi115 sudo[120336]: pam_unix(sudo:session): session closed for user root 2022-05-14T00:08:19.110 INFO:journalctl@ceph.osd.11.smithi115.stdout:May 14 00:08:18 smithi115 sudo[120339]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2022-05-14T00:08:19.110 INFO:journalctl@ceph.osd.11.smithi115.stdout:May 14 00:08:18 smithi115 sudo[120339]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2022-05-14T00:08:19.110 INFO:journalctl@ceph.osd.11.smithi115.stdout:May 14 00:08:18 smithi115 sudo[120339]: pam_unix(sudo:session): session opened for user root by (uid=0) 2022-05-14T00:08:19.111 INFO:journalctl@ceph.osd.11.smithi115.stdout:May 14 00:08:18 smithi115 sudo[120339]: pam_unix(sudo:session): session closed for user root 2022-05-14T00:08:19.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:08:19 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:08:19.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:08:19 smithi047 ceph-mon[77147]: pgmap v1568: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 33 KiB/s wr, 8 op/s 2022-05-14T00:08:19.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:08:19 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 2430 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:08:19.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:08:19 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:08:19.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:08:19 smithi115 ceph-mon[78593]: pgmap v1568: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 33 KiB/s wr, 8 op/s 2022-05-14T00:08:19.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:08:19 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 2430 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:08:20.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:08:19 smithi115 conmon[90393]: 2022-05-14T00:08:19.625+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:08:20.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:08:20 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:08:20.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:08:20 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:08:21.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:08:20 smithi115 conmon[90393]: 2022-05-14T00:08:20.665+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:08:21.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:08:21 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:08:21.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:08:21 smithi047 ceph-mon[77147]: pgmap v1569: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 22 KiB/s rd, 65 KiB/s wr, 13 op/s 2022-05-14T00:08:21.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:08:21 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:08:21.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:08:21 smithi115 ceph-mon[78593]: pgmap v1569: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 22 KiB/s rd, 65 KiB/s wr, 13 op/s 2022-05-14T00:08:22.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:08:21 smithi115 conmon[90393]: 2022-05-14T00:08:21.696+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:08:22.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:08:22 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:08:22.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:08:22 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:08:22.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:08:22 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:08:22.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:08:22 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:08:22.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:08:22 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:08:22.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:08:22 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:08:23.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:08:22 smithi115 conmon[90393]: 2022-05-14T00:08:22.690+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:08:23.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:08:23 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:08:23.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:08:23 smithi047 ceph-mon[77147]: pgmap v1570: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 21 KiB/s rd, 65 KiB/s wr, 10 op/s 2022-05-14T00:08:23.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:08:23 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:08:23.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:08:23 smithi115 ceph-mon[78593]: pgmap v1570: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 21 KiB/s rd, 65 KiB/s wr, 10 op/s 2022-05-14T00:08:24.110 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:08:23 smithi115 conmon[90393]: 2022-05-14T00:08:23.650+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:08:24.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:08:24 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:08:24.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:08:24 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 2435 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:08:24.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:08:24 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:08:24.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:08:24 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 2435 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:08:25.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:08:24 smithi115 conmon[90393]: 2022-05-14T00:08:24.648+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:08:25.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:08:25 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:08:25.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:08:25 smithi047 ceph-mon[77147]: pgmap v1571: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 21 KiB/s rd, 65 KiB/s wr, 10 op/s 2022-05-14T00:08:25.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:08:25 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:08:25.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:08:25 smithi115 ceph-mon[78593]: pgmap v1571: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 21 KiB/s rd, 65 KiB/s wr, 10 op/s 2022-05-14T00:08:26.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:08:25 smithi115 conmon[90393]: 2022-05-14T00:08:25.616+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:08:26.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:08:26 smithi047 sudo[118349]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2022-05-14T00:08:26.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:08:26 smithi047 sudo[118349]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2022-05-14T00:08:26.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:08:26 smithi047 sudo[118349]: pam_unix(sudo:session): session opened for user root by (uid=0) 2022-05-14T00:08:26.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:08:26 smithi047 sudo[118349]: pam_unix(sudo:session): session closed for user root 2022-05-14T00:08:26.978 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:08:26 smithi115 conmon[90393]: 2022-05-14T00:08:26.653+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:08:26.978 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:08:26 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:08:26.979 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:08:26 smithi115 ceph-mon[78593]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2022-05-14T00:08:27.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:08:26 smithi047 sudo[118352]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2022-05-14T00:08:27.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:08:26 smithi047 sudo[118352]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2022-05-14T00:08:27.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:08:26 smithi047 sudo[118352]: pam_unix(sudo:session): session opened for user root by (uid=0) 2022-05-14T00:08:27.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:08:26 smithi047 sudo[118352]: pam_unix(sudo:session): session closed for user root 2022-05-14T00:08:27.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:08:26 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:08:27.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:08:26 smithi047 ceph-mon[77147]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2022-05-14T00:08:27.233 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:08:26 smithi115 sudo[120342]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2022-05-14T00:08:27.234 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:08:26 smithi115 sudo[120342]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2022-05-14T00:08:27.234 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:08:26 smithi115 sudo[120342]: pam_unix(sudo:session): session opened for user root by (uid=0) 2022-05-14T00:08:27.234 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:08:26 smithi115 sudo[120342]: pam_unix(sudo:session): session closed for user root 2022-05-14T00:08:27.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:08:27 smithi115 sudo[120345]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2022-05-14T00:08:27.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:08:27 smithi115 sudo[120345]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2022-05-14T00:08:27.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:08:27 smithi115 sudo[120345]: pam_unix(sudo:session): session opened for user root by (uid=0) 2022-05-14T00:08:27.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:08:27 smithi115 sudo[120345]: pam_unix(sudo:session): session closed for user root 2022-05-14T00:08:28.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:08:27 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:08:28.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:08:27 smithi047 ceph-mon[77147]: pgmap v1572: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 22 KiB/s rd, 65 KiB/s wr, 13 op/s 2022-05-14T00:08:28.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:08:27 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:08:28.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:08:27 smithi047 ceph-mon[77147]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2022-05-14T00:08:28.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:08:27 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-05-14T00:08:28.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:08:27 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-05-14T00:08:28.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:08:27 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-05-14T00:08:28.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:08:27 smithi047 ceph-mon[77147]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2022-05-14T00:08:28.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:08:27 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-05-14T00:08:28.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:08:27 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-05-14T00:08:28.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:08:27 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-05-14T00:08:28.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:08:27 smithi047 ceph-mon[77147]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2022-05-14T00:08:28.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:08:27 smithi115 conmon[90393]: 2022-05-14T00:08:27.688+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:08:28.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:08:27 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:08:28.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:08:27 smithi115 ceph-mon[78593]: pgmap v1572: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 22 KiB/s rd, 65 KiB/s wr, 13 op/s 2022-05-14T00:08:28.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:08:27 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:08:28.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:08:27 smithi115 ceph-mon[78593]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2022-05-14T00:08:28.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:08:27 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-05-14T00:08:28.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:08:27 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-05-14T00:08:28.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:08:27 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-05-14T00:08:28.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:08:27 smithi115 ceph-mon[78593]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2022-05-14T00:08:28.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:08:27 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-05-14T00:08:28.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:08:27 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-05-14T00:08:28.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:08:27 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-05-14T00:08:28.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:08:27 smithi115 ceph-mon[78593]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2022-05-14T00:08:29.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:08:28 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:08:29.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:08:28 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 2440 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:08:29.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:08:28 smithi115 conmon[90393]: 2022-05-14T00:08:28.654+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:08:29.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:08:28 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:08:29.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:08:28 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 2440 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:08:30.015 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:08:29 smithi047 ceph-mon[77147]: pgmap v1573: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 3.2 KiB/s rd, 33 KiB/s wr, 7 op/s 2022-05-14T00:08:30.015 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:08:29 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:08:30.016 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:08:29 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:08:30.016 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:08:29 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:08:30.017 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:08:29 smithi115 conmon[90393]: 2022-05-14T00:08:29.626+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:08:30.017 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:08:29 smithi115 ceph-mon[78593]: pgmap v1573: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 3.2 KiB/s rd, 33 KiB/s wr, 7 op/s 2022-05-14T00:08:30.017 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:08:29 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:08:30.017 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:08:29 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:08:30.018 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:08:29 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:08:30.751 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:08:30 smithi115 conmon[90393]: 2022-05-14T00:08:30.608+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:08:31.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:08:30 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:08:31.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:08:30 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T00:08:31.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:08:30 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T00:08:31.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:08:30 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T00:08:31.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:08:30 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:08:31.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:08:30 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T00:08:31.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:08:30 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T00:08:31.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:08:30 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T00:08:31.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:08:31 smithi115 conmon[90393]: 2022-05-14T00:08:31.562+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:08:31.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:08:31 smithi115 ceph-mon[78593]: pgmap v1574: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 54 KiB/s wr, 11 op/s 2022-05-14T00:08:31.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:08:31 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:08:32.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:08:31 smithi047 ceph-mon[77147]: pgmap v1574: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 54 KiB/s wr, 11 op/s 2022-05-14T00:08:32.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:08:31 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:08:32.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:08:32 smithi115 conmon[90393]: 2022-05-14T00:08:32.580+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:08:32.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:08:32 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:08:33.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:08:32 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:08:33.541 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:08:33 smithi047 conmon[78521]: 2022-05-14T00:08:33.072+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:08:33.541 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:08:33 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T00:08:33.542 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:08:33 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:08:33.542 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:08:33 smithi047 conmon[78521]: yield 2022-05-14T00:08:33.542 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:08:33 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:08:33.542 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:08:33 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:08:33.542 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:08:33 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:08:33.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:08:33 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T00:08:33.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:08:33 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:08:33.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:08:33 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T00:08:33.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:08:33 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:08:33.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:08:33 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T00:08:33.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:08:33 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:08:33.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:08:33 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T00:08:33.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:08:33 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:08:33.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:08:33 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T00:08:33.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:08:33 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:08:33.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:08:33 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:08:33.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:08:33 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:08:33.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:08:33 smithi115 conmon[90393]: 2022-05-14T00:08:33.538+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:08:33.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:08:33 smithi115 ceph-mon[78593]: pgmap v1575: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 3.2 KiB/s rd, 22 KiB/s wr, 6 op/s 2022-05-14T00:08:33.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:08:33 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:08:33.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:08:33 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:08:33.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:08:33 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:08:33.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:08:33 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 2445 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:08:34.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:08:33 smithi047 ceph-mon[77147]: pgmap v1575: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 3.2 KiB/s rd, 22 KiB/s wr, 6 op/s 2022-05-14T00:08:34.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:08:33 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:08:34.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:08:33 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:08:34.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:08:33 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:08:34.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:08:33 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 2445 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:08:34.762 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:08:34 smithi115 conmon[90393]: 2022-05-14T00:08:34.495+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:08:35.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:08:34 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:08:35.041 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T00:08:35.041 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:08:35.042 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T00:08:35.042 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:08:35.042 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:08:35.042 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:08:35.042 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T00:08:35.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:08:35.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T00:08:35.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:08:35.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T00:08:35.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:08:35.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T00:08:35.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:08:35.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T00:08:35.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:08:35.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:08:35.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:08:35.045 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:08:34 smithi047 ceph-mon[77147]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T00:08:35.045 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:08:34 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:08:35.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:08:34 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:08:35.108 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T00:08:35.108 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:08:35.109 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T00:08:35.109 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:08:35.109 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:08:35.109 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:08:35.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T00:08:35.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:08:35.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T00:08:35.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:08:35.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T00:08:35.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:08:35.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T00:08:35.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:08:35.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T00:08:35.113 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:08:35.113 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:08:35.114 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:08:35.114 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:08:34 smithi115 ceph-mon[78593]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T00:08:35.114 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:08:34 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:08:35.764 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:08:35 smithi115 conmon[90393]: 2022-05-14T00:08:35.508+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:08:36.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:08:35 smithi047 ceph-mon[77147]: pgmap v1576: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 3.2 KiB/s rd, 22 KiB/s wr, 6 op/s 2022-05-14T00:08:36.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:08:35 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:08:36.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:08:35 smithi115 ceph-mon[78593]: pgmap v1576: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 3.2 KiB/s rd, 22 KiB/s wr, 6 op/s 2022-05-14T00:08:36.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:08:35 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:08:36.541 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:08:36 smithi047 conmon[78521]: 2022-05-14T00:08:36.144+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:08:36.542 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:08:36 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T00:08:36.542 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:08:36 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:08:36.542 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:08:36 smithi047 conmon[78521]: yield 2022-05-14T00:08:36.542 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:08:36 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:08:36.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:08:36 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:08:36.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:08:36 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:08:36.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:08:36 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T00:08:36.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:08:36 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:08:36.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:08:36 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T00:08:36.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:08:36 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:08:36.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:08:36 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T00:08:36.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:08:36 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:08:36.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:08:36 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T00:08:36.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:08:36 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:08:36.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:08:36 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T00:08:36.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:08:36 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:08:36.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:08:36 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:08:36.547 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:08:36 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:08:36.547 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:08:36 smithi047 conmon[78521]: 2022-05-14T00:08:36.145+0000 7f533ef76700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T00:08:36.547 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:08:36 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T00:08:36.547 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:08:36 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:08:36.547 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:08:36 smithi047 conmon[78521]: yield 2022-05-14T00:08:36.548 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:08:36 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:08:36.548 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:08:36 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:08:36.548 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:08:36 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:08:36.548 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:08:36 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T00:08:36.548 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:08:36 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:08:36.549 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:08:36 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T00:08:36.549 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:08:36 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:08:36.549 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:08:36 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T00:08:36.549 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:08:36 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:08:36.550 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:08:36 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T00:08:36.550 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:08:36 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:08:36.550 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:08:36 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T00:08:36.550 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:08:36 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:08:36.550 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:08:36 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:08:36.551 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:08:36 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:08:36.551 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:08:36 smithi047 conmon[78521]: 2022-05-14T00:08:36.551 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:08:36 smithi047 conmon[78521]: During handling of the above exception, another exception occurred: 2022-05-14T00:08:36.551 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:08:36 smithi047 conmon[78521]: 2022-05-14T00:08:36.551 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:08:36 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T00:08:36.552 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:08:36 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T00:08:36.552 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:08:36 smithi047 conmon[78521]: return f(*arg) 2022-05-14T00:08:36.552 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:08:36 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T00:08:36.552 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:08:36 smithi047 conmon[78521]: self._write_client_files(client_files, host) 2022-05-14T00:08:36.553 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:08:36 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T00:08:36.553 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:08:36 smithi047 conmon[78521]: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T00:08:36.553 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:08:36 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T00:08:36.553 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:08:36 smithi047 conmon[78521]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T00:08:36.553 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:08:36 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T00:08:36.554 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:08:36 smithi047 conmon[78521]: return self.event_loop.get_result(coro) 2022-05-14T00:08:36.554 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:08:36 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T00:08:36.554 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:08:36 smithi047 conmon[78521]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T00:08:36.554 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:08:36 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T00:08:36.554 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:08:36 smithi047 conmon[78521]: return self.__get_result() 2022-05-14T00:08:36.555 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:08:36 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T00:08:36.555 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:08:36 smithi047 conmon[78521]: raise self._exception 2022-05-14T00:08:36.556 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:08:36 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T00:08:36.556 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:08:36 smithi047 conmon[78521]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T00:08:36.556 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:08:36 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T00:08:36.557 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:08:36 smithi047 conmon[78521]: conn = await self._remote_connection(host, addr) 2022-05-14T00:08:36.557 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:08:36 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T00:08:36.557 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:08:36 smithi047 conmon[78521]: raise 2022-05-14T00:08:36.557 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:08:36 smithi047 conmon[78521]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T00:08:36.558 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:08:36 smithi047 conmon[78521]: self.gen.throw(type, value, traceback) 2022-05-14T00:08:36.558 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:08:36 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T00:08:36.558 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:08:36 smithi047 conmon[78521]: raise OrchestratorError(msg) 2022-05-14T00:08:36.558 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:08:36 smithi047 conmon[78521]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:08:36.769 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:08:36 smithi115 conmon[90393]: 2022-05-14T00:08:36.508+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:08:37.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:08:36 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:08:37.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:08:36 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:08:37.771 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:08:37 smithi115 conmon[90393]: 2022-05-14T00:08:37.460+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:08:38.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:08:37 smithi047 ceph-mon[77147]: pgmap v1577: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 4.5 KiB/s rd, 22 KiB/s wr, 9 op/s 2022-05-14T00:08:38.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:08:37 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:08:38.042 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T00:08:38.042 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:08:38.042 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T00:08:38.042 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:08:38.042 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:08:38.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:08:38.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T00:08:38.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:08:38.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T00:08:38.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:08:38.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T00:08:38.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:08:38.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T00:08:38.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:08:38.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T00:08:38.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:08:38.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:08:38.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:08:38.048 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:08:37 smithi047 ceph-mon[77147]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T00:08:38.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T00:08:38.049 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:08:38.050 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T00:08:38.050 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:08:38.050 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:08:38.050 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:08:38.050 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T00:08:38.051 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:08:38.051 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T00:08:38.051 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:08:38.052 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T00:08:38.052 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:08:38.052 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T00:08:38.053 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:08:38.053 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T00:08:38.053 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:08:38.053 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:08:38.054 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:08:38.054 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T00:08:38.054 INFO:journalctl@ceph.mon.b.smithi047.stdout: During handling of the above exception, another exception occurred: 2022-05-14T00:08:38.055 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T00:08:38.055 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T00:08:38.055 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T00:08:38.055 INFO:journalctl@ceph.mon.b.smithi047.stdout: return f(*arg) 2022-05-14T00:08:38.056 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T00:08:38.056 INFO:journalctl@ceph.mon.b.smithi047.stdout: self._write_client_files(client_files, host) 2022-05-14T00:08:38.056 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T00:08:38.056 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T00:08:38.056 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T00:08:38.057 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T00:08:38.057 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T00:08:38.057 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.event_loop.get_result(coro) 2022-05-14T00:08:38.057 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T00:08:38.058 INFO:journalctl@ceph.mon.b.smithi047.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T00:08:38.058 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T00:08:38.059 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.__get_result() 2022-05-14T00:08:38.059 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T00:08:38.060 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise self._exception 2022-05-14T00:08:38.060 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T00:08:38.061 INFO:journalctl@ceph.mon.b.smithi047.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T00:08:38.062 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T00:08:38.062 INFO:journalctl@ceph.mon.b.smithi047.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T00:08:38.063 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T00:08:38.063 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise 2022-05-14T00:08:38.064 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T00:08:38.064 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.gen.throw(type, value, traceback) 2022-05-14T00:08:38.065 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T00:08:38.065 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OrchestratorError(msg) 2022-05-14T00:08:38.065 INFO:journalctl@ceph.mon.b.smithi047.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:08:38.066 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:08:37 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:08:38.066 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:08:37 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:08:38.067 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:08:37 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:08:38.067 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:08:37 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:08:38.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:08:37 smithi115 ceph-mon[78593]: pgmap v1577: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 4.5 KiB/s rd, 22 KiB/s wr, 9 op/s 2022-05-14T00:08:38.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:08:37 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:08:38.108 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T00:08:38.109 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:08:38.109 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T00:08:38.109 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:08:38.109 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:08:38.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:08:38.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T00:08:38.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:08:38.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T00:08:38.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:08:38.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T00:08:38.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:08:38.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T00:08:38.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:08:38.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T00:08:38.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:08:38.113 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:08:38.113 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:08:38.113 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:08:37 smithi115 ceph-mon[78593]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T00:08:38.113 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T00:08:38.114 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:08:38.114 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T00:08:38.114 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:08:38.114 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:08:38.114 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:08:38.115 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T00:08:38.115 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:08:38.115 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T00:08:38.115 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:08:38.116 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T00:08:38.116 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:08:38.116 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T00:08:38.116 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:08:38.117 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T00:08:38.117 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:08:38.117 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:08:38.117 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:08:38.117 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T00:08:38.118 INFO:journalctl@ceph.mon.c.smithi115.stdout: During handling of the above exception, another exception occurred: 2022-05-14T00:08:38.118 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T00:08:38.118 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T00:08:38.118 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T00:08:38.119 INFO:journalctl@ceph.mon.c.smithi115.stdout: return f(*arg) 2022-05-14T00:08:38.119 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T00:08:38.119 INFO:journalctl@ceph.mon.c.smithi115.stdout: self._write_client_files(client_files, host) 2022-05-14T00:08:38.119 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T00:08:38.119 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T00:08:38.120 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T00:08:38.120 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T00:08:38.120 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T00:08:38.120 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.event_loop.get_result(coro) 2022-05-14T00:08:38.121 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T00:08:38.121 INFO:journalctl@ceph.mon.c.smithi115.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T00:08:38.121 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T00:08:38.121 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.__get_result() 2022-05-14T00:08:38.122 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T00:08:38.122 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise self._exception 2022-05-14T00:08:38.122 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T00:08:38.122 INFO:journalctl@ceph.mon.c.smithi115.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T00:08:38.122 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T00:08:38.123 INFO:journalctl@ceph.mon.c.smithi115.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T00:08:38.123 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T00:08:38.123 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise 2022-05-14T00:08:38.123 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T00:08:38.123 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.gen.throw(type, value, traceback) 2022-05-14T00:08:38.124 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T00:08:38.124 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OrchestratorError(msg) 2022-05-14T00:08:38.124 INFO:journalctl@ceph.mon.c.smithi115.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:08:38.124 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:08:37 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:08:38.125 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:08:37 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:08:38.125 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:08:37 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:08:38.125 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:08:37 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:08:38.775 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:08:38 smithi115 conmon[90393]: 2022-05-14T00:08:38.436+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:08:39.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:08:38 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:08:39.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:08:38 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 2450 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:08:39.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:08:38 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:08:39.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:08:38 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 2450 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:08:39.776 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:08:39 smithi115 conmon[90393]: 2022-05-14T00:08:39.409+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:08:40.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:08:39 smithi047 ceph-mon[77147]: pgmap v1578: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 3.2 KiB/s rd, 22 KiB/s wr, 6 op/s 2022-05-14T00:08:40.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:08:39 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:08:40.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:08:39 smithi115 ceph-mon[78593]: pgmap v1578: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 3.2 KiB/s rd, 22 KiB/s wr, 6 op/s 2022-05-14T00:08:40.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:08:39 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:08:40.782 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:08:40 smithi115 conmon[90393]: 2022-05-14T00:08:40.441+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:08:41.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:08:40 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:08:41.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:08:40 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:08:41.783 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:08:41 smithi115 conmon[90393]: 2022-05-14T00:08:41.463+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:08:42.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:08:41 smithi047 ceph-mon[77147]: pgmap v1579: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 4.5 KiB/s rd, 22 KiB/s wr, 9 op/s 2022-05-14T00:08:42.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:08:41 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:08:42.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:08:41 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:08:42.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:08:41 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:08:42.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:08:41 smithi115 ceph-mon[78593]: pgmap v1579: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 4.5 KiB/s rd, 22 KiB/s wr, 9 op/s 2022-05-14T00:08:42.153 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:08:41 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:08:42.154 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:08:41 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:08:42.154 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:08:41 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:08:42.788 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:08:42 smithi115 conmon[90393]: 2022-05-14T00:08:42.427+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:08:43.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:08:42 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:08:43.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:08:42 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:08:43.790 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:08:43 smithi115 conmon[90393]: 2022-05-14T00:08:43.424+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:08:44.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:08:43 smithi047 ceph-mon[77147]: pgmap v1580: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:08:44.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:08:43 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:08:44.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:08:43 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 2455 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:08:44.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:08:43 smithi115 ceph-mon[78593]: pgmap v1580: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:08:44.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:08:43 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:08:44.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:08:43 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 2455 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:08:44.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:08:44 smithi115 conmon[90393]: 2022-05-14T00:08:44.469+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:08:45.792 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:08:45 smithi115 conmon[90393]: 2022-05-14T00:08:45.480+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:08:46.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:08:45 smithi115 ceph-mon[78593]: pgmap v1581: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:08:46.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:08:45 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:08:46.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:08:45 smithi047 ceph-mon[77147]: pgmap v1581: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:08:46.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:08:45 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:08:46.799 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:08:46 smithi115 conmon[90393]: 2022-05-14T00:08:46.441+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:08:47.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:08:46 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:08:47.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:08:46 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:08:47.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:08:46 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:08:47.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:08:46 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:08:47.799 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:08:47 smithi115 conmon[90393]: 2022-05-14T00:08:47.438+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:08:48.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:08:47 smithi115 ceph-mon[78593]: pgmap v1582: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:08:48.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:08:47 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:08:48.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:08:47 smithi047 ceph-mon[77147]: pgmap v1582: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:08:48.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:08:47 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:08:48.803 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:08:48 smithi115 conmon[90393]: 2022-05-14T00:08:48.423+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:08:48.804 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:08:48 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:08:48.805 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:08:48 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 2460 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:08:49.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:08:48 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:08:49.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:08:48 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 2460 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:08:49.804 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:08:49 smithi115 conmon[90393]: 2022-05-14T00:08:49.409+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:08:50.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:08:49 smithi115 ceph-mon[78593]: pgmap v1583: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:08:50.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:08:49 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:08:50.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:08:49 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:08:50.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:08:49 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:08:50.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:08:49 smithi047 ceph-mon[77147]: pgmap v1583: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:08:50.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:08:49 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:08:50.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:08:49 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:08:50.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:08:49 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:08:50.811 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:08:50 smithi115 conmon[90393]: 2022-05-14T00:08:50.375+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:08:51.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:08:50 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:08:51.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:08:50 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:08:51.811 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:08:51 smithi115 conmon[90393]: 2022-05-14T00:08:51.418+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:08:52.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:08:51 smithi115 ceph-mon[78593]: pgmap v1584: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:08:52.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:08:51 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:08:52.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:08:51 smithi047 ceph-mon[77147]: pgmap v1584: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:08:52.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:08:51 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:08:52.818 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:08:52 smithi115 conmon[90393]: 2022-05-14T00:08:52.412+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:08:53.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:08:52 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:08:53.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:08:52 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:08:53.819 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:08:53 smithi115 conmon[90393]: 2022-05-14T00:08:53.376+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:08:54.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:08:53 smithi115 ceph-mon[78593]: pgmap v1585: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:08:54.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:08:53 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:08:54.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:08:53 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:08:54.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:08:53 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:08:54.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:08:53 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 2465 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:08:54.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:08:53 smithi047 ceph-mon[77147]: pgmap v1585: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:08:54.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:08:53 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:08:54.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:08:53 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:08:54.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:08:53 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:08:54.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:08:53 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 2465 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:08:54.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:08:54 smithi115 conmon[90393]: 2022-05-14T00:08:54.345+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:08:55.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:08:54 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:08:55.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:08:54 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:08:55.824 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:08:55 smithi115 conmon[90393]: 2022-05-14T00:08:55.358+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:08:56.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:08:55 smithi115 ceph-mon[78593]: pgmap v1586: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:08:56.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:08:55 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:08:56.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:08:55 smithi047 ceph-mon[77147]: pgmap v1586: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:08:56.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:08:55 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:08:56.829 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:08:56 smithi115 conmon[90393]: 2022-05-14T00:08:56.407+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:08:57.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:08:56 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:08:57.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:08:56 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:08:57.831 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:08:57 smithi115 conmon[90393]: 2022-05-14T00:08:57.456+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:08:58.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:08:57 smithi115 ceph-mon[78593]: pgmap v1587: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:08:58.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:08:57 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:08:58.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:08:57 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:08:58.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:08:57 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:08:58.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:08:57 smithi047 ceph-mon[77147]: pgmap v1587: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:08:58.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:08:57 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:08:58.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:08:57 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:08:58.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:08:57 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:08:58.839 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:08:58 smithi115 conmon[90393]: 2022-05-14T00:08:58.448+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:08:59.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:08:58 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:08:59.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:08:58 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 2470 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:08:59.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:08:58 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:08:59.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:08:58 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 2470 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:08:59.839 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:08:59 smithi115 conmon[90393]: 2022-05-14T00:08:59.447+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:09:00.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:08:59 smithi115 ceph-mon[78593]: pgmap v1588: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:09:00.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:08:59 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:09:00.296 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:08:59 smithi047 ceph-mon[77147]: pgmap v1588: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:09:00.296 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:08:59 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:09:00.847 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:09:00 smithi115 conmon[90393]: 2022-05-14T00:09:00.411+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:09:01.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:09:00 smithi115 ceph-mon[78593]: pgmap v1589: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:09:01.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:09:00 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:09:01.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:09:00 smithi047 ceph-mon[77147]: pgmap v1589: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:09:01.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:09:00 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:09:01.846 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:09:01 smithi115 conmon[90393]: 2022-05-14T00:09:01.457+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:09:02.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:09:01 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:09:02.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:09:01 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:09:02.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:09:01 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:09:02.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:09:01 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:09:02.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:09:01 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:09:02.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:09:01 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:09:02.853 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:09:02 smithi115 conmon[90393]: 2022-05-14T00:09:02.466+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:09:03.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:09:02 smithi115 ceph-mon[78593]: pgmap v1590: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:09:03.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:09:02 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:09:03.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:09:02 smithi047 ceph-mon[77147]: pgmap v1590: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:09:03.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:09:02 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:09:03.854 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:09:03 smithi115 conmon[90393]: 2022-05-14T00:09:03.463+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:09:04.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:09:03 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:09:04.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:09:03 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 2475 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:09:04.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:09:03 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:09:04.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:09:03 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 2475 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:09:04.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:09:04 smithi115 conmon[90393]: 2022-05-14T00:09:04.466+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:09:05.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:09:04 smithi047 ceph-mon[77147]: pgmap v1591: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:09:05.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:09:04 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:09:05.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:09:04 smithi115 ceph-mon[78593]: pgmap v1591: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:09:05.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:09:04 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:09:05.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:09:05 smithi115 conmon[90393]: 2022-05-14T00:09:05.436+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:09:06.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:09:05 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:09:06.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:09:05 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:09:06.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:09:06 smithi115 conmon[90393]: 2022-05-14T00:09:06.395+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:09:07.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:09:06 smithi047 ceph-mon[77147]: pgmap v1592: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:09:07.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:09:06 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:09:07.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:09:06 smithi115 ceph-mon[78593]: pgmap v1592: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:09:07.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:09:06 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:09:07.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:09:07 smithi115 conmon[90393]: 2022-05-14T00:09:07.407+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:09:08.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:09:07 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:09:08.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:09:07 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:09:08.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:09:08 smithi115 conmon[90393]: 2022-05-14T00:09:08.395+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:09:09.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:09:08 smithi047 ceph-mon[77147]: pgmap v1593: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:09:09.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:09:08 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:09:09.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:09:08 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 2480 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:09:09.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:09:08 smithi115 ceph-mon[78593]: pgmap v1593: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:09:09.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:09:08 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:09:09.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:09:08 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 2480 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:09:09.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:09:09 smithi115 conmon[90393]: 2022-05-14T00:09:09.368+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:09:10.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:09:09 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:09:10.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:09:09 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:09:10.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:09:09 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:09:10.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:09:09 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:09:10.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:09:09 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:09:10.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:09:09 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:09:10.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:09:10 smithi115 conmon[90393]: 2022-05-14T00:09:10.399+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:09:11.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:09:10 smithi047 ceph-mon[77147]: pgmap v1594: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:09:11.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:09:10 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:09:11.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:09:10 smithi115 ceph-mon[78593]: pgmap v1594: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:09:11.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:09:10 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:09:11.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:09:11 smithi115 conmon[90393]: 2022-05-14T00:09:11.368+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:09:12.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:09:11 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:09:12.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:09:11 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:09:12.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:09:12 smithi115 conmon[90393]: 2022-05-14T00:09:12.417+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:09:13.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:09:12 smithi047 ceph-mon[77147]: pgmap v1595: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:09:13.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:09:12 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:09:13.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:09:12 smithi115 ceph-mon[78593]: pgmap v1595: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:09:13.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:09:12 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:09:13.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:09:13 smithi115 conmon[90393]: 2022-05-14T00:09:13.374+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:09:14.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:09:13 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:09:14.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:09:13 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:09:14.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:09:13 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T00:09:14.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:09:13 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T00:09:14.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:09:13 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:09:14.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:09:13 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 2485 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:09:14.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:09:13 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:09:14.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:09:13 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:09:14.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:09:13 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T00:09:14.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:09:13 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T00:09:14.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:09:13 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:09:14.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:09:13 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 2485 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:09:14.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:09:14 smithi115 conmon[90393]: 2022-05-14T00:09:14.421+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:09:15.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:09:14 smithi047 ceph-mon[77147]: pgmap v1596: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:09:15.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:09:14 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:09:15.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:09:14 smithi115 ceph-mon[78593]: pgmap v1596: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:09:15.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:09:14 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:09:15.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:09:15 smithi115 conmon[90393]: 2022-05-14T00:09:15.459+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:09:16.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:09:15 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:09:16.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:09:15 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:09:16.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:09:16 smithi115 conmon[90393]: 2022-05-14T00:09:16.418+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:09:17.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:09:16 smithi047 ceph-mon[77147]: pgmap v1597: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:09:17.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:09:16 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:09:17.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:09:16 smithi115 ceph-mon[78593]: pgmap v1597: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:09:17.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:09:16 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:09:17.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:09:17 smithi115 conmon[90393]: 2022-05-14T00:09:17.408+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:09:18.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:09:17 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:09:18.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:09:17 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:09:18.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:09:17 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:09:18.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:09:17 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:09:18.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:09:17 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:09:18.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:09:17 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:09:18.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:09:18 smithi115 conmon[90393]: 2022-05-14T00:09:18.431+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:09:19.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:09:18 smithi047 ceph-mon[77147]: pgmap v1598: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:09:19.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:09:18 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:09:19.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:09:18 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 2490 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:09:19.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:09:18 smithi115 ceph-mon[78593]: pgmap v1598: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:09:19.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:09:18 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:09:19.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:09:18 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 2490 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:09:19.860 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:09:19 smithi115 conmon[90393]: 2022-05-14T00:09:19.383+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:09:20.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:09:19 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:09:20.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:09:19 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:09:20.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:09:20 smithi115 conmon[90393]: 2022-05-14T00:09:20.421+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:09:21.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:09:20 smithi047 ceph-mon[77147]: pgmap v1599: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:09:21.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:09:20 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:09:21.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:09:20 smithi115 ceph-mon[78593]: pgmap v1599: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:09:21.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:09:20 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:09:21.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:09:21 smithi115 conmon[90393]: 2022-05-14T00:09:21.441+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:09:22.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:09:21 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:09:22.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:09:21 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:09:22.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:09:21 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:09:22.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:09:21 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:09:22.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:09:21 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:09:22.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:09:21 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:09:22.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:09:22 smithi115 conmon[90393]: 2022-05-14T00:09:22.457+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:09:23.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:09:22 smithi047 ceph-mon[77147]: pgmap v1600: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:09:23.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:09:22 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:09:23.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:09:22 smithi115 ceph-mon[78593]: pgmap v1600: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:09:23.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:09:22 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:09:23.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:09:23 smithi115 conmon[90393]: 2022-05-14T00:09:23.411+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:09:24.303 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:09:23 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:09:24.304 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:09:23 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 2495 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:09:24.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:09:23 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:09:24.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:09:23 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 2495 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:09:24.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:09:24 smithi115 conmon[90393]: 2022-05-14T00:09:24.376+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:09:25.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:09:24 smithi047 ceph-mon[77147]: pgmap v1601: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:09:25.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:09:24 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:09:25.332 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:09:24 smithi115 ceph-mon[78593]: pgmap v1601: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:09:25.332 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:09:24 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:09:25.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:09:25 smithi115 conmon[90393]: 2022-05-14T00:09:25.330+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:09:26.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:09:25 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:09:26.335 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:09:25 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:09:26.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:09:26 smithi115 conmon[90393]: 2022-05-14T00:09:26.335+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:09:27.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:09:26 smithi047 ceph-mon[77147]: pgmap v1602: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:09:27.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:09:26 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:09:27.344 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:09:26 smithi115 ceph-mon[78593]: pgmap v1602: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:09:27.344 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:09:26 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:09:27.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:09:27 smithi115 conmon[90393]: 2022-05-14T00:09:27.343+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:09:28.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:09:27 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:09:28.300 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:09:27 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:09:28.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:09:28 smithi115 conmon[90393]: 2022-05-14T00:09:28.299+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:09:29.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:09:28 smithi047 ceph-mon[77147]: pgmap v1603: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:09:29.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:09:28 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:09:29.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:09:28 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 2500 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:09:29.299 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:09:28 smithi115 ceph-mon[78593]: pgmap v1603: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:09:29.299 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:09:28 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:09:29.300 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:09:28 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 2500 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:09:29.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:09:29 smithi115 conmon[90393]: 2022-05-14T00:09:29.298+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:09:30.251 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:09:29 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:09:30.251 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:09:29 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:09:30.251 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:09:29 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:09:30.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:09:29 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:09:30.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:09:29 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:09:30.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:09:29 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:09:30.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:09:30 smithi115 conmon[90393]: 2022-05-14T00:09:30.250+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:09:31.222 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:09:30 smithi115 ceph-mon[78593]: pgmap v1604: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:09:31.222 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:09:30 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:09:31.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:09:30 smithi047 ceph-mon[77147]: pgmap v1604: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:09:31.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:09:30 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:09:31.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:09:31 smithi115 conmon[90393]: 2022-05-14T00:09:31.220+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:09:32.238 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:09:31 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:09:32.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:09:31 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:09:32.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:09:32 smithi115 conmon[90393]: 2022-05-14T00:09:32.236+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:09:33.253 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:09:32 smithi115 ceph-mon[78593]: pgmap v1605: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:09:33.254 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:09:32 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:09:33.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:09:32 smithi047 ceph-mon[77147]: pgmap v1605: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:09:33.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:09:32 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:09:33.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:09:33 smithi115 conmon[90393]: 2022-05-14T00:09:33.252+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:09:34.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:09:33 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:09:34.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:09:33 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:09:34.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:09:33 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:09:34.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:09:33 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 2505 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:09:34.294 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:09:33 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:09:34.294 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:09:33 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:09:34.294 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:09:33 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:09:34.294 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:09:33 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 2505 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:09:34.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:09:34 smithi115 conmon[90393]: 2022-05-14T00:09:34.291+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:09:35.262 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:09:34 smithi115 ceph-mon[78593]: pgmap v1606: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:09:35.262 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:09:34 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:09:35.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:09:34 smithi047 ceph-mon[77147]: pgmap v1606: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:09:35.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:09:34 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:09:35.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:09:35 smithi115 conmon[90393]: 2022-05-14T00:09:35.261+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:09:36.229 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:09:35 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:09:36.229 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:09:35 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:09:36.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:09:36 smithi115 conmon[90393]: 2022-05-14T00:09:36.292+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:09:36.959 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:09:36 smithi047 conmon[78521]: 2022-05-14T00:09:36.623+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:09:36.959 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:09:36 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T00:09:36.959 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:09:36 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:09:36.959 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:09:36 smithi047 conmon[78521]: yield 2022-05-14T00:09:36.960 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:09:36 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:09:36.960 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:09:36 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:09:36.960 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:09:36 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:09:36.960 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:09:36 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T00:09:36.961 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:09:36 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:09:36.961 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:09:36 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T00:09:36.961 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:09:36 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:09:36.961 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:09:36 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T00:09:36.962 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:09:36 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:09:36.962 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:09:36 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T00:09:36.962 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:09:36 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:09:36.962 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:09:36 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T00:09:36.962 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:09:36 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:09:36.963 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:09:36 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:09:36.963 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:09:36 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:09:37.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:09:36 smithi047 ceph-mon[77147]: pgmap v1607: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:09:37.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:09:36 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T00:09:37.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:09:36 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T00:09:37.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:09:36 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T00:09:37.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:09:36 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:09:37.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:09:36 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:09:37.292 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T00:09:37.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:09:37.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T00:09:37.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:09:37.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:09:37.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:09:37.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T00:09:37.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:09:37.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T00:09:37.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:09:37.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T00:09:37.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:09:37.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T00:09:37.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:09:37.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T00:09:37.299 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:09:37.299 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:09:37.299 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:09:37.299 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:09:36 smithi047 ceph-mon[77147]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T00:09:37.341 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:09:36 smithi115 ceph-mon[78593]: pgmap v1607: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:09:37.341 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:09:36 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T00:09:37.342 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:09:36 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T00:09:37.342 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:09:36 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T00:09:37.342 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:09:36 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:09:37.343 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:09:36 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:09:37.343 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T00:09:37.343 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:09:37.343 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T00:09:37.344 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:09:37.344 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:09:37.344 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:09:37.344 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T00:09:37.345 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:09:37.345 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T00:09:37.345 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:09:37.345 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T00:09:37.346 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:09:37.346 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T00:09:37.346 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:09:37.346 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T00:09:37.347 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:09:37.347 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:09:37.347 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:09:37.347 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:09:36 smithi115 ceph-mon[78593]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T00:09:37.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:09:37 smithi115 conmon[90393]: 2022-05-14T00:09:37.340+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:09:38.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:09:37 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:09:38.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:09:37 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:09:38.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:09:37 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:09:38.303 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:09:37 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:09:38.304 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:09:37 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:09:38.304 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:09:37 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:09:38.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:09:38 smithi115 conmon[90393]: 2022-05-14T00:09:38.302+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:09:39.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:09:38 smithi047 ceph-mon[77147]: pgmap v1608: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:09:39.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:09:38 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:09:39.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:09:38 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 2510 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:09:39.348 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:09:38 smithi115 ceph-mon[78593]: pgmap v1608: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:09:39.348 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:09:38 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:09:39.348 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:09:38 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 2510 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:09:39.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:09:39 smithi115 conmon[90393]: 2022-05-14T00:09:39.347+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:09:39.966 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:09:39 smithi047 conmon[78521]: 2022-05-14T00:09:39.695+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:09:39.966 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:09:39 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T00:09:39.967 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:09:39 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:09:39.967 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:09:39 smithi047 conmon[78521]: yield 2022-05-14T00:09:39.967 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:09:39 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:09:39.967 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:09:39 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:09:39.968 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:09:39 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:09:39.968 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:09:39 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T00:09:39.968 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:09:39 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:09:39.968 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:09:39 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T00:09:39.968 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:09:39 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:09:39.969 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:09:39 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T00:09:39.969 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:09:39 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:09:39.969 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:09:39 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T00:09:39.969 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:09:39 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:09:39.970 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:09:39 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T00:09:39.970 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:09:39 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:09:39.970 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:09:39 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:09:39.970 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:09:39 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:09:39.971 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:09:39 smithi047 conmon[78521]: 2022-05-14T00:09:39.696+0000 7f533f777700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T00:09:39.971 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:09:39 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T00:09:39.971 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:09:39 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:09:39.972 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:09:39 smithi047 conmon[78521]: yield 2022-05-14T00:09:39.972 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:09:39 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:09:39.972 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:09:39 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:09:39.973 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:09:39 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:09:39.974 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:09:39 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T00:09:39.974 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:09:39 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:09:39.974 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:09:39 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T00:09:39.975 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:09:39 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:09:39.975 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:09:39 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T00:09:39.976 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:09:39 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:09:39.976 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:09:39 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T00:09:39.976 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:09:39 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:09:39.976 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:09:39 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T00:09:39.976 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:09:39 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:09:39.977 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:09:39 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:09:39.977 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:09:39 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:09:39.977 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:09:39 smithi047 conmon[78521]: 2022-05-14T00:09:39.977 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:09:39 smithi047 conmon[78521]: During handling of the above exception, another exception occurred: 2022-05-14T00:09:39.978 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:09:39 smithi047 conmon[78521]: 2022-05-14T00:09:39.978 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:09:39 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T00:09:39.978 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:09:39 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T00:09:39.978 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:09:39 smithi047 conmon[78521]: return f(*arg) 2022-05-14T00:09:39.979 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:09:39 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T00:09:39.979 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:09:39 smithi047 conmon[78521]: self._write_client_files(client_files, host) 2022-05-14T00:09:39.979 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:09:39 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T00:09:39.979 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:09:39 smithi047 conmon[78521]: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T00:09:39.980 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:09:39 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T00:09:39.980 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:09:39 smithi047 conmon[78521]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T00:09:39.980 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:09:39 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T00:09:39.980 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:09:39 smithi047 conmon[78521]: return self.event_loop.get_result(coro) 2022-05-14T00:09:39.980 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:09:39 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T00:09:39.981 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:09:39 smithi047 conmon[78521]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T00:09:39.981 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:09:39 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T00:09:39.981 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:09:39 smithi047 conmon[78521]: return self.__get_result() 2022-05-14T00:09:39.981 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:09:39 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T00:09:39.982 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:09:39 smithi047 conmon[78521]: raise self._exception 2022-05-14T00:09:39.982 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:09:39 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T00:09:39.982 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:09:39 smithi047 conmon[78521]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T00:09:39.983 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:09:39 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T00:09:39.983 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:09:39 smithi047 conmon[78521]: conn = await self._remote_connection(host, addr) 2022-05-14T00:09:39.984 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:09:39 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T00:09:39.985 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:09:39 smithi047 conmon[78521]: raise 2022-05-14T00:09:39.985 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:09:39 smithi047 conmon[78521]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T00:09:39.985 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:09:39 smithi047 conmon[78521]: self.gen.throw(type, value, traceback) 2022-05-14T00:09:39.986 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:09:39 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T00:09:39.986 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:09:39 smithi047 conmon[78521]: raise OrchestratorError(msg) 2022-05-14T00:09:39.986 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:09:39 smithi047 conmon[78521]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:09:40.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:09:39 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:09:40.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:09:39 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:09:40.292 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T00:09:40.292 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:09:40.292 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T00:09:40.292 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:09:40.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:09:40.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:09:40.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T00:09:40.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:09:40.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T00:09:40.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:09:40.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T00:09:40.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:09:40.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T00:09:40.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:09:40.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T00:09:40.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:09:40.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:09:40.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:09:40.296 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:09:39 smithi047 ceph-mon[77147]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T00:09:40.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T00:09:40.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:09:40.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T00:09:40.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:09:40.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:09:40.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:09:40.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T00:09:40.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:09:40.298 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T00:09:40.298 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:09:40.298 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T00:09:40.298 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:09:40.298 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T00:09:40.299 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:09:40.299 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T00:09:40.299 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:09:40.299 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:09:40.299 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:09:40.300 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T00:09:40.300 INFO:journalctl@ceph.mon.b.smithi047.stdout: During handling of the above exception, another exception occurred: 2022-05-14T00:09:40.300 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T00:09:40.300 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T00:09:40.300 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T00:09:40.301 INFO:journalctl@ceph.mon.b.smithi047.stdout: return f(*arg) 2022-05-14T00:09:40.301 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T00:09:40.301 INFO:journalctl@ceph.mon.b.smithi047.stdout: self._write_client_files(client_files, host) 2022-05-14T00:09:40.301 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T00:09:40.302 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T00:09:40.302 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T00:09:40.302 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T00:09:40.302 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T00:09:40.302 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.event_loop.get_result(coro) 2022-05-14T00:09:40.303 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T00:09:40.303 INFO:journalctl@ceph.mon.b.smithi047.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T00:09:40.303 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T00:09:40.303 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.__get_result() 2022-05-14T00:09:40.303 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T00:09:40.304 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise self._exception 2022-05-14T00:09:40.304 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T00:09:40.304 INFO:journalctl@ceph.mon.b.smithi047.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T00:09:40.304 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T00:09:40.304 INFO:journalctl@ceph.mon.b.smithi047.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T00:09:40.305 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T00:09:40.305 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise 2022-05-14T00:09:40.305 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T00:09:40.305 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.gen.throw(type, value, traceback) 2022-05-14T00:09:40.306 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T00:09:40.306 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OrchestratorError(msg) 2022-05-14T00:09:40.306 INFO:journalctl@ceph.mon.b.smithi047.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:09:40.353 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:09:39 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:09:40.354 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:09:39 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:09:40.354 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T00:09:40.354 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:09:40.355 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T00:09:40.355 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:09:40.356 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:09:40.357 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:09:40.357 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T00:09:40.358 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:09:40.358 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T00:09:40.359 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:09:40.359 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T00:09:40.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:09:40.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T00:09:40.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:09:40.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T00:09:40.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:09:40.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:09:40.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:09:40.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:09:39 smithi115 ceph-mon[78593]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T00:09:40.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T00:09:40.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:09:40.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T00:09:40.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:09:40.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:09:40.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:09:40.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T00:09:40.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:09:40.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T00:09:40.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:09:40.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T00:09:40.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:09:40.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T00:09:40.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:09:40.365 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T00:09:40.365 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:09:40.365 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:09:40.365 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:09:40.365 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T00:09:40.366 INFO:journalctl@ceph.mon.c.smithi115.stdout: During handling of the above exception, another exception occurred: 2022-05-14T00:09:40.366 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T00:09:40.366 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T00:09:40.366 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T00:09:40.367 INFO:journalctl@ceph.mon.c.smithi115.stdout: return f(*arg) 2022-05-14T00:09:40.367 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T00:09:40.367 INFO:journalctl@ceph.mon.c.smithi115.stdout: self._write_client_files(client_files, host) 2022-05-14T00:09:40.367 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T00:09:40.367 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T00:09:40.368 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T00:09:40.368 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T00:09:40.368 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T00:09:40.368 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.event_loop.get_result(coro) 2022-05-14T00:09:40.369 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T00:09:40.369 INFO:journalctl@ceph.mon.c.smithi115.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T00:09:40.369 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T00:09:40.369 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.__get_result() 2022-05-14T00:09:40.369 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T00:09:40.370 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise self._exception 2022-05-14T00:09:40.370 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T00:09:40.370 INFO:journalctl@ceph.mon.c.smithi115.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T00:09:40.370 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T00:09:40.371 INFO:journalctl@ceph.mon.c.smithi115.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T00:09:40.371 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T00:09:40.371 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise 2022-05-14T00:09:40.371 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T00:09:40.371 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.gen.throw(type, value, traceback) 2022-05-14T00:09:40.372 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T00:09:40.372 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OrchestratorError(msg) 2022-05-14T00:09:40.372 INFO:journalctl@ceph.mon.c.smithi115.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:09:40.611 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:09:40 smithi115 conmon[90393]: 2022-05-14T00:09:40.352+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:09:41.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:09:40 smithi047 ceph-mon[77147]: pgmap v1609: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:09:41.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:09:40 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:09:41.341 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:09:40 smithi115 ceph-mon[78593]: pgmap v1609: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:09:41.341 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:09:40 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:09:41.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:09:41 smithi115 conmon[90393]: 2022-05-14T00:09:41.340+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:09:42.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:09:41 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:09:42.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:09:41 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:09:42.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:09:41 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:09:42.318 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:09:41 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:09:42.319 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:09:41 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:09:42.319 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:09:41 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:09:42.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:09:42 smithi115 conmon[90393]: 2022-05-14T00:09:42.317+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:09:43.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:09:42 smithi047 ceph-mon[77147]: pgmap v1610: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:09:43.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:09:42 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:09:43.319 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:09:42 smithi115 ceph-mon[78593]: pgmap v1610: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:09:43.319 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:09:42 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:09:43.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:09:43 smithi115 conmon[90393]: 2022-05-14T00:09:43.318+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:09:44.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:09:43 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:09:44.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:09:43 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 2515 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:09:44.314 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:09:43 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:09:44.314 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:09:43 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 2515 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:09:44.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:09:44 smithi115 conmon[90393]: 2022-05-14T00:09:44.314+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:09:45.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:09:44 smithi047 ceph-mon[77147]: pgmap v1611: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:09:45.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:09:44 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:09:45.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:09:44 smithi115 ceph-mon[78593]: pgmap v1611: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:09:45.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:09:44 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:09:45.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:09:45 smithi115 conmon[90393]: 2022-05-14T00:09:45.358+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:09:46.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:09:45 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:09:46.341 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:09:45 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:09:46.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:09:46 smithi115 conmon[90393]: 2022-05-14T00:09:46.341+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:09:47.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:09:46 smithi047 ceph-mon[77147]: pgmap v1612: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:09:47.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:09:46 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:09:47.318 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:09:46 smithi115 ceph-mon[78593]: pgmap v1612: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:09:47.319 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:09:46 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:09:47.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:09:47 smithi115 conmon[90393]: 2022-05-14T00:09:47.318+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:09:48.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:09:47 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:09:48.324 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:09:47 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:09:48.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:09:48 smithi115 conmon[90393]: 2022-05-14T00:09:48.323+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:09:49.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:09:48 smithi047 ceph-mon[77147]: pgmap v1613: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:09:49.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:09:48 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:09:49.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:09:48 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 2520 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:09:49.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:09:48 smithi115 ceph-mon[78593]: pgmap v1613: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:09:49.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:09:48 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:09:49.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:09:48 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 2520 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:09:49.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:09:49 smithi115 conmon[90393]: 2022-05-14T00:09:49.359+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:09:50.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:09:49 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:09:50.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:09:49 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:09:50.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:09:49 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:09:50.352 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:09:49 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:09:50.352 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:09:49 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:09:50.353 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:09:49 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:09:50.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:09:50 smithi115 conmon[90393]: 2022-05-14T00:09:50.351+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:09:51.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:09:51 smithi047 ceph-mon[77147]: pgmap v1614: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:09:51.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:09:51 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:09:51.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:09:51 smithi115 ceph-mon[78593]: pgmap v1614: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:09:51.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:09:51 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:09:51.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:09:51 smithi115 conmon[90393]: 2022-05-14T00:09:51.384+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:09:52.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:09:52 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:09:52.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:09:52 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:09:52.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:09:52 smithi115 conmon[90393]: 2022-05-14T00:09:52.376+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:09:53.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:09:53 smithi047 ceph-mon[77147]: pgmap v1615: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:09:53.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:09:53 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:09:53.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:09:53 smithi115 ceph-mon[78593]: pgmap v1615: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:09:53.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:09:53 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:09:53.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:09:53 smithi115 conmon[90393]: 2022-05-14T00:09:53.387+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:09:54.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:09:54 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:09:54.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:09:54 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:09:54.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:09:54 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:09:54.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:09:54 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 2525 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:09:54.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:09:54 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:09:54.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:09:54 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:09:54.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:09:54 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:09:54.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:09:54 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 2525 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:09:54.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:09:54 smithi115 conmon[90393]: 2022-05-14T00:09:54.412+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:09:55.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:09:55 smithi047 ceph-mon[77147]: pgmap v1616: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:09:55.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:09:55 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:09:55.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:09:55 smithi115 ceph-mon[78593]: pgmap v1616: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:09:55.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:09:55 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:09:57.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:09:57 smithi047 ceph-mon[77147]: pgmap v1617: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:09:57.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:09:57 smithi115 ceph-mon[78593]: pgmap v1617: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:09:57.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:09:57 smithi115 conmon[90393]: 2022-05-14T00:09:57.374+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:09:58.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:09:58 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:09:58.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:09:58 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:09:58.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:09:58 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:09:58.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:09:58 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:09:58.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:09:58 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:09:58.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:09:58 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:09:58.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:09:58 smithi115 conmon[90393]: 2022-05-14T00:09:58.371+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:09:59.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:09:59 smithi047 ceph-mon[77147]: pgmap v1618: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:09:59.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:09:59 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:09:59.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:09:59 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 2530 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:09:59.338 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:09:59 smithi115 ceph-mon[78593]: pgmap v1618: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:09:59.338 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:09:59 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:09:59.338 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:09:59 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 2530 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:09:59.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:09:59 smithi115 conmon[90393]: 2022-05-14T00:09:59.337+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:10:00.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:10:00 smithi047 conmon[77123]: 2022-05-14T00:09:59.999+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : Health detail: HEALTH_ERR 1 filesystem is degraded; 1 mds daemon damaged; 1 MDSs are read only; 1 MDSs report slow metadata IOs; 1/3 mons down, quorum b,c; Reduced data availability: 1 pg inactive; 9 slow ops, oldest one blocked for 2530 sec, osd.5 has slow ops 2022-05-14T00:10:00.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:10:00 smithi047 conmon[77123]: 2022-05-14T00:09:59.999+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : [WRN] FS_DEGRADED: 1 filesystem is degraded 2022-05-14T00:10:00.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:10:00 smithi047 conmon[77123]: 2022-05-14T00:09:59.999+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : fs cephfs is degraded 2022-05-14T00:10:00.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:10:00 smithi047 conmon[77123]: 2022-05-14T00:09:59.999+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : [ERR] MDS_DAMAGE: 1 mds daemon damaged 2022-05-14T00:10:00.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:10:00 smithi047 conmon[77123]: 2022-05-14T00:09:59.999+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : fs cephfs mds.2 is damaged 2022-05-14T00:10:00.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:10:00 smithi047 conmon[77123]: 2022-05-14T00:09:59.999+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : [WRN] MDS_READ_ONLY: 1 MDSs are read only 2022-05-14T00:10:00.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:10:00 smithi047 conmon[77123]: 2022-05-14T00:10:00.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:10:00 smithi047 conmon[77123]: 2022-05-14T00:09:59.999+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : mds.l(mds.0): MDS in read-only mode 2022-05-14T00:10:00.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:10:00 smithi047 conmon[77123]: 2022-05-14T00:09:59.999+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : [WRN] MDS_SLOW_METADATA_IO: 1 MDSs report slow metadata IOs 2022-05-14T00:10:00.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:10:00 smithi047 conmon[77123]: 2022-05-14T00:09:59.999+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : mds.l(mds.0): 4 slow metadata IOs are blocked > 180 secs, oldest blocked for 2833 secs 2022-05-14T00:10:00.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:10:00 smithi047 conmon[77123]: 2022-05-14T00:09:59.999+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : [WRN] MON_DOWN: 1/3 mons down, quorum b,c 2022-05-14T00:10:00.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:10:00 smithi047 conmon[77123]: 2022-05-14T00:09:59.999+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : mon.a (rank 0) addr [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] is down (out of quorum) 2022-05-14T00:10:00.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:10:00 smithi047 conmon[77123]: 2022-05-14T00:09:59.999+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : [WRN] PG_AVAILABILITY: Reduced data availability: 1 pg inactive 2022-05-14T00:10:00.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:10:00 smithi047 conmon[77123]: 2022-05-14T00:09:59.999+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : pg 2.1 is stuck inactive for 54m, current state unknown, last acting [] 2022-05-14T00:10:00.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:10:00 smithi047 conmon[77123]: 2022-05-14T00:10:00.295 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:10:00 smithi047 conmon[77123]: 2022-05-14T00:09:59.999+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : [WRN] SLOW_OPS: 9 slow ops, oldest one blocked for 2530 sec, osd.5 has slow ops 2022-05-14T00:10:00.295 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:10:00 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:10:00.295 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:10:00 smithi047 ceph-mon[77147]: Health detail: HEALTH_ERR 1 filesystem is degraded; 1 mds daemon damaged; 1 MDSs are read only; 1 MDSs report slow metadata IOs; 1/3 mons down, quorum b,c; Reduced data availability: 1 pg inactive; 9 slow ops, oldest one blocked for 2530 sec, osd.5 has slow ops 2022-05-14T00:10:00.295 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:10:00 smithi047 ceph-mon[77147]: [WRN] FS_DEGRADED: 1 filesystem is degraded 2022-05-14T00:10:00.296 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:10:00 smithi047 ceph-mon[77147]: fs cephfs is degraded 2022-05-14T00:10:00.296 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:10:00 smithi047 ceph-mon[77147]: [ERR] MDS_DAMAGE: 1 mds daemon damaged 2022-05-14T00:10:00.296 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:10:00 smithi047 ceph-mon[77147]: fs cephfs mds.2 is damaged 2022-05-14T00:10:00.296 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:10:00 smithi047 ceph-mon[77147]: [WRN] MDS_READ_ONLY: 1 MDSs are read only 2022-05-14T00:10:00.296 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:10:00 smithi047 ceph-mon[77147]: mds.l(mds.0): MDS in read-only mode 2022-05-14T00:10:00.297 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:10:00 smithi047 ceph-mon[77147]: [WRN] MDS_SLOW_METADATA_IO: 1 MDSs report slow metadata IOs 2022-05-14T00:10:00.297 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:10:00 smithi047 ceph-mon[77147]: mds.l(mds.0): 4 slow metadata IOs are blocked > 180 secs, oldest blocked for 2833 secs 2022-05-14T00:10:00.297 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:10:00 smithi047 ceph-mon[77147]: [WRN] MON_DOWN: 1/3 mons down, quorum b,c 2022-05-14T00:10:00.297 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:10:00 smithi047 ceph-mon[77147]: mon.a (rank 0) addr [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] is down (out of quorum) 2022-05-14T00:10:00.298 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:10:00 smithi047 ceph-mon[77147]: [WRN] PG_AVAILABILITY: Reduced data availability: 1 pg inactive 2022-05-14T00:10:00.298 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:10:00 smithi047 ceph-mon[77147]: pg 2.1 is stuck inactive for 54m, current state unknown, last acting [] 2022-05-14T00:10:00.298 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:10:00 smithi047 ceph-mon[77147]: [WRN] SLOW_OPS: 9 slow ops, oldest one blocked for 2530 sec, osd.5 has slow ops 2022-05-14T00:10:00.347 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:10:00 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:10:00.347 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:10:00 smithi115 ceph-mon[78593]: Health detail: HEALTH_ERR 1 filesystem is degraded; 1 mds daemon damaged; 1 MDSs are read only; 1 MDSs report slow metadata IOs; 1/3 mons down, quorum b,c; Reduced data availability: 1 pg inactive; 9 slow ops, oldest one blocked for 2530 sec, osd.5 has slow ops 2022-05-14T00:10:00.349 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:10:00 smithi115 ceph-mon[78593]: [WRN] FS_DEGRADED: 1 filesystem is degraded 2022-05-14T00:10:00.350 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:10:00 smithi115 ceph-mon[78593]: fs cephfs is degraded 2022-05-14T00:10:00.350 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:10:00 smithi115 ceph-mon[78593]: [ERR] MDS_DAMAGE: 1 mds daemon damaged 2022-05-14T00:10:00.350 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:10:00 smithi115 ceph-mon[78593]: fs cephfs mds.2 is damaged 2022-05-14T00:10:00.350 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:10:00 smithi115 ceph-mon[78593]: [WRN] MDS_READ_ONLY: 1 MDSs are read only 2022-05-14T00:10:00.350 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:10:00 smithi115 ceph-mon[78593]: mds.l(mds.0): MDS in read-only mode 2022-05-14T00:10:00.351 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:10:00 smithi115 ceph-mon[78593]: [WRN] MDS_SLOW_METADATA_IO: 1 MDSs report slow metadata IOs 2022-05-14T00:10:00.351 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:10:00 smithi115 ceph-mon[78593]: mds.l(mds.0): 4 slow metadata IOs are blocked > 180 secs, oldest blocked for 2833 secs 2022-05-14T00:10:00.351 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:10:00 smithi115 ceph-mon[78593]: [WRN] MON_DOWN: 1/3 mons down, quorum b,c 2022-05-14T00:10:00.351 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:10:00 smithi115 ceph-mon[78593]: mon.a (rank 0) addr [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] is down (out of quorum) 2022-05-14T00:10:00.352 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:10:00 smithi115 ceph-mon[78593]: [WRN] PG_AVAILABILITY: Reduced data availability: 1 pg inactive 2022-05-14T00:10:00.352 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:10:00 smithi115 ceph-mon[78593]: pg 2.1 is stuck inactive for 54m, current state unknown, last acting [] 2022-05-14T00:10:00.352 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:10:00 smithi115 ceph-mon[78593]: [WRN] SLOW_OPS: 9 slow ops, oldest one blocked for 2530 sec, osd.5 has slow ops 2022-05-14T00:10:00.613 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:10:00 smithi115 conmon[90393]: 2022-05-14T00:10:00.344+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:10:01.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:10:01 smithi047 ceph-mon[77147]: pgmap v1619: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:10:01.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:10:01 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:10:01.301 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:10:01 smithi115 ceph-mon[78593]: pgmap v1619: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:10:01.302 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:10:01 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:10:01.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:10:01 smithi115 conmon[90393]: 2022-05-14T00:10:01.300+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:10:02.306 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:10:02 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:10:02.306 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:10:02 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:10:02.307 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:10:02 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:10:02.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:10:02 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:10:02.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:10:02 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:10:02.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:10:02 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:10:02.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:10:02 smithi115 conmon[90393]: 2022-05-14T00:10:02.305+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:10:03.310 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:10:03 smithi115 ceph-mon[78593]: pgmap v1620: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:10:03.311 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:10:03 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:10:03.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:10:03 smithi047 ceph-mon[77147]: pgmap v1620: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:10:03.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:10:03 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:10:03.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:10:03 smithi115 conmon[90393]: 2022-05-14T00:10:03.309+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:10:04.330 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:10:04 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:10:04.331 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:10:04 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 2535 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:10:04.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:10:04 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:10:04.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:10:04 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 2535 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:10:04.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:10:04 smithi115 conmon[90393]: 2022-05-14T00:10:04.329+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:10:05.302 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:10:05 smithi115 ceph-mon[78593]: pgmap v1621: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:10:05.302 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:10:05 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:10:05.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:10:05 smithi047 ceph-mon[77147]: pgmap v1621: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:10:05.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:10:05 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:10:05.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:10:05 smithi115 conmon[90393]: 2022-05-14T00:10:05.300+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:10:06.323 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:10:06 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:10:06.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:10:06 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:10:06.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:10:06 smithi115 conmon[90393]: 2022-05-14T00:10:06.322+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:10:07.318 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:10:07 smithi115 ceph-mon[78593]: pgmap v1622: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:10:07.320 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:10:07 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:10:07.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:10:07 smithi047 ceph-mon[77147]: pgmap v1622: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:10:07.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:10:07 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:10:07.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:10:07 smithi115 conmon[90393]: 2022-05-14T00:10:07.317+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:10:08.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:10:08 smithi115 conmon[90393]: 2022-05-14T00:10:08.308+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:10:08.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:10:08 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:10:08.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:10:08 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:10:09.332 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:10:09 smithi115 ceph-mon[78593]: pgmap v1623: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:10:09.333 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:10:09 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:10:09.333 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:10:09 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 2540 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:10:09.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:10:09 smithi047 ceph-mon[77147]: pgmap v1623: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:10:09.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:10:09 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:10:09.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:10:09 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 2540 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:10:09.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:10:09 smithi115 conmon[90393]: 2022-05-14T00:10:09.331+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:10:10.339 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:10:10 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:10:10.340 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:10:10 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:10:10.340 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:10:10 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:10:10.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:10:10 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:10:10.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:10:10 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:10:10.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:10:10 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:10:10.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:10:10 smithi115 conmon[90393]: 2022-05-14T00:10:10.338+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:10:11.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:10:11 smithi115 ceph-mon[78593]: pgmap v1624: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:10:11.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:10:11 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:10:11.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:10:11 smithi047 ceph-mon[77147]: pgmap v1624: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:10:11.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:10:11 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:10:11.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:10:11 smithi115 conmon[90393]: 2022-05-14T00:10:11.387+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:10:12.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:10:12 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:10:12.548 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:10:12 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:10:12.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:10:12 smithi115 conmon[90393]: 2022-05-14T00:10:12.428+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:10:13.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:10:13 smithi115 ceph-mon[78593]: pgmap v1625: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:10:13.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:10:13 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:10:13.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:10:13 smithi047 ceph-mon[77147]: pgmap v1625: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:10:13.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:10:13 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:10:13.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:10:13 smithi115 conmon[90393]: 2022-05-14T00:10:13.438+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:10:14.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:10:14 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:10:14.447 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:10:14 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:10:14.447 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:10:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T00:10:14.447 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:10:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T00:10:14.447 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:10:14 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:10:14.448 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:10:14 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 2545 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:10:14.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:10:14 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:10:14.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:10:14 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:10:14.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:10:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T00:10:14.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:10:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T00:10:14.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:10:14 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:10:14.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:10:14 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 2545 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:10:14.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:10:14 smithi115 conmon[90393]: 2022-05-14T00:10:14.447+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:10:15.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:10:15 smithi115 ceph-mon[78593]: pgmap v1626: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:10:15.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:10:15 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:10:15.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:10:15 smithi047 ceph-mon[77147]: pgmap v1626: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:10:15.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:10:15 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:10:15.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:10:15 smithi115 conmon[90393]: 2022-05-14T00:10:15.447+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:10:16.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:10:16 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:10:16.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:10:16 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:10:16.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:10:16 smithi115 conmon[90393]: 2022-05-14T00:10:16.418+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:10:17.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:10:17 smithi115 ceph-mon[78593]: pgmap v1627: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:10:17.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:10:17 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:10:17.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:10:17 smithi047 ceph-mon[77147]: pgmap v1627: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:10:17.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:10:17 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:10:17.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:10:17 smithi115 conmon[90393]: 2022-05-14T00:10:17.374+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:10:18.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:10:18 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:10:18.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:10:18 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:10:18.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:10:18 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:10:18.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:10:18 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:10:18.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:10:18 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:10:18.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:10:18 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:10:18.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:10:18 smithi115 conmon[90393]: 2022-05-14T00:10:18.389+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:10:19.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:10:19 smithi115 ceph-mon[78593]: pgmap v1628: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:10:19.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:10:19 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:10:19.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:10:19 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 2550 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:10:19.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:10:19 smithi047 ceph-mon[77147]: pgmap v1628: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:10:19.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:10:19 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:10:19.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:10:19 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 2550 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:10:19.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:10:19 smithi115 conmon[90393]: 2022-05-14T00:10:19.439+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:10:20.392 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:10:20 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:10:20.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:10:20 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:10:20.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:10:20 smithi115 conmon[90393]: 2022-05-14T00:10:20.390+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:10:21.366 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:10:21 smithi115 ceph-mon[78593]: pgmap v1629: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:10:21.366 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:10:21 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:10:21.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:10:21 smithi047 ceph-mon[77147]: pgmap v1629: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:10:21.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:10:21 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:10:21.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:10:21 smithi115 conmon[90393]: 2022-05-14T00:10:21.364+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:10:22.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:10:22 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:10:22.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:10:22 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:10:22.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:10:22 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:10:22.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:10:22 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:10:22.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:10:22 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:10:22.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:10:22 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:10:22.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:10:22 smithi115 conmon[90393]: 2022-05-14T00:10:22.359+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:10:23.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:10:23 smithi047 ceph-mon[77147]: pgmap v1630: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:10:23.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:10:23 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:10:23.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:10:23 smithi115 ceph-mon[78593]: pgmap v1630: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:10:23.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:10:23 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:10:23.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:10:23 smithi115 conmon[90393]: 2022-05-14T00:10:23.366+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:10:24.389 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:10:24 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:10:24.389 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:10:24 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 2555 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:10:24.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:10:24 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:10:24.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:10:24 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 2555 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:10:24.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:10:24 smithi115 conmon[90393]: 2022-05-14T00:10:24.388+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:10:25.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:10:25 smithi047 ceph-mon[77147]: pgmap v1631: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:10:25.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:10:25 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:10:25.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:10:25 smithi115 ceph-mon[78593]: pgmap v1631: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:10:25.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:10:25 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:10:25.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:10:25 smithi115 conmon[90393]: 2022-05-14T00:10:25.372+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:10:26.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:10:26 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:10:26.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:10:26 smithi115 conmon[90393]: 2022-05-14T00:10:26.337+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:10:26.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:10:26 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:10:27.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:10:27 smithi047 ceph-mon[77147]: pgmap v1632: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:10:27.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:10:27 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:10:27.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:10:27 smithi115 ceph-mon[78593]: pgmap v1632: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:10:27.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:10:27 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:10:27.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:10:27 smithi115 conmon[90393]: 2022-05-14T00:10:27.324+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:10:28.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:10:28 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:10:28.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:10:28 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:10:28.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:10:28 smithi115 conmon[90393]: 2022-05-14T00:10:28.308+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:10:29.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:10:29 smithi047 ceph-mon[77147]: pgmap v1633: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:10:29.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:10:29 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:10:29.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:10:29 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 2560 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:10:29.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:10:29 smithi115 conmon[90393]: 2022-05-14T00:10:29.273+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:10:29.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:10:29 smithi115 ceph-mon[78593]: pgmap v1633: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:10:29.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:10:29 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:10:29.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:10:29 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 2560 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:10:30.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:10:30 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:10:30.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:10:30 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:10:30.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:10:30 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:10:30.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:10:30 smithi115 conmon[90393]: 2022-05-14T00:10:30.307+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:10:30.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:10:30 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:10:30.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:10:30 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:10:30.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:10:30 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:10:31.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:10:31 smithi047 ceph-mon[77147]: pgmap v1634: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:10:31.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:10:31 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:10:31.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:10:31 smithi115 conmon[90393]: 2022-05-14T00:10:31.318+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:10:31.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:10:31 smithi115 ceph-mon[78593]: pgmap v1634: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:10:31.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:10:31 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:10:32.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:10:32 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:10:32.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:10:32 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:10:32.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:10:32 smithi115 conmon[90393]: 2022-05-14T00:10:32.325+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:10:33.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:10:33 smithi047 ceph-mon[77147]: pgmap v1635: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:10:33.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:10:33 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:10:33.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:10:33 smithi115 conmon[90393]: 2022-05-14T00:10:33.366+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:10:33.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:10:33 smithi115 ceph-mon[78593]: pgmap v1635: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:10:33.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:10:33 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:10:34.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:10:34 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:10:34.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:10:34 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:10:34.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:10:34 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:10:34.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:10:34 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 2565 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:10:34.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:10:34 smithi115 conmon[90393]: 2022-05-14T00:10:34.384+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:10:34.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:10:34 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:10:34.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:10:34 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:10:34.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:10:34 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:10:34.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:10:34 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 2565 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:10:35.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:10:35 smithi047 ceph-mon[77147]: pgmap v1636: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:10:35.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:10:35 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:10:35.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:10:35 smithi115 ceph-mon[78593]: pgmap v1636: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:10:35.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:10:35 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:10:35.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:10:35 smithi115 conmon[90393]: 2022-05-14T00:10:35.415+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:10:36.461 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:10:36 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:10:36.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:10:36 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:10:36.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:10:36 smithi115 conmon[90393]: 2022-05-14T00:10:36.461+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:10:37.438 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:10:37 smithi115 ceph-mon[78593]: pgmap v1637: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:10:37.439 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:10:37 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:10:37.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:10:37 smithi047 ceph-mon[77147]: pgmap v1637: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:10:37.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:10:37 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:10:37.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:10:37 smithi115 conmon[90393]: 2022-05-14T00:10:37.437+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:10:38.485 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:10:38 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:10:38.485 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:10:38 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:10:38.486 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:10:38 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:10:38.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:10:38 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:10:38.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:10:38 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:10:38.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:10:38 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:10:38.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:10:38 smithi115 conmon[90393]: 2022-05-14T00:10:38.484+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:10:39.518 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:10:39 smithi115 ceph-mon[78593]: pgmap v1638: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:10:39.518 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:10:39 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:10:39.518 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:10:39 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 2570 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:10:39.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:10:39 smithi047 ceph-mon[77147]: pgmap v1638: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:10:39.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:10:39 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:10:39.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:10:39 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 2570 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:10:39.774 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:10:39 smithi115 conmon[90393]: 2022-05-14T00:10:39.517+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:10:40.470 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:10:40 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T00:10:40.471 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:10:40 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T00:10:40.471 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:10:40 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T00:10:40.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:10:40 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T00:10:40.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:10:40 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T00:10:40.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:10:40 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T00:10:40.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:10:40 smithi115 conmon[90393]: 2022-05-14T00:10:40.469+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:10:41.504 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:10:41 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:10:41.504 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:10:41 smithi115 ceph-mon[78593]: pgmap v1639: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:10:41.505 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:10:41 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:10:41.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:10:41 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:10:41.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:10:41 smithi047 ceph-mon[77147]: pgmap v1639: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:10:41.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:10:41 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:10:41.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:10:41 smithi115 conmon[90393]: 2022-05-14T00:10:41.503+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:10:42.495 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:10:42 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:10:42.496 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:10:42 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:10:42.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:10:42 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:10:42.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:10:42 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:10:42.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:10:42 smithi115 conmon[90393]: 2022-05-14T00:10:42.495+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:10:43.202 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:10:42 smithi047 conmon[78521]: 2022-05-14T00:10:42.799+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:10:43.202 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:10:42 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T00:10:43.203 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:10:42 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:10:43.203 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:10:42 smithi047 conmon[78521]: yield 2022-05-14T00:10:43.203 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:10:42 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:10:43.203 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:10:42 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:10:43.204 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:10:42 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:10:43.204 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:10:42 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T00:10:43.204 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:10:42 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:10:43.204 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:10:42 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T00:10:43.204 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:10:42 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:10:43.205 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:10:42 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T00:10:43.205 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:10:42 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:10:43.206 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:10:42 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T00:10:43.206 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:10:42 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:10:43.206 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:10:42 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T00:10:43.206 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:10:42 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:10:43.207 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:10:42 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:10:43.207 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:10:42 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:10:43.501 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:10:43 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:10:43.502 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:10:43 smithi115 ceph-mon[78593]: pgmap v1640: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:10:43.502 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:10:43 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:10:43.502 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T00:10:43.503 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:10:43.503 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T00:10:43.503 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:10:43.503 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:10:43.503 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:10:43.504 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T00:10:43.504 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:10:43.504 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T00:10:43.504 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:10:43.505 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T00:10:43.505 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:10:43.505 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T00:10:43.505 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:10:43.505 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T00:10:43.506 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:10:43.506 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:10:43.506 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:10:43.506 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:10:43 smithi115 ceph-mon[78593]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T00:10:43.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:10:43 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:10:43.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:10:43 smithi047 ceph-mon[77147]: pgmap v1640: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:10:43.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:10:43 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:10:43.541 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T00:10:43.542 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:10:43.542 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T00:10:43.542 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:10:43.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:10:43.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:10:43.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T00:10:43.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:10:43.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T00:10:43.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:10:43.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T00:10:43.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:10:43.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T00:10:43.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:10:43.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T00:10:43.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:10:43.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:10:43.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:10:43.546 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:10:43 smithi047 ceph-mon[77147]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T00:10:43.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:10:43 smithi115 conmon[90393]: 2022-05-14T00:10:43.501+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:10:44.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:10:44 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:10:44.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:10:44 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 2575 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:10:44.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:10:44 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:10:44.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:10:44 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 2575 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:10:44.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:10:44 smithi115 conmon[90393]: 2022-05-14T00:10:44.543+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:10:45.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:10:45 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:10:45.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:10:45 smithi047 ceph-mon[77147]: pgmap v1641: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:10:45.579 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:10:45 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:10:45.579 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:10:45 smithi115 ceph-mon[78593]: pgmap v1641: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:10:45.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:10:45 smithi115 conmon[90393]: 2022-05-14T00:10:45.578+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:10:46.211 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:10:45 smithi047 conmon[78521]: 2022-05-14T00:10:45.871+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:10:46.211 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:10:45 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T00:10:46.211 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:10:45 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:10:46.211 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:10:45 smithi047 conmon[78521]: yield 2022-05-14T00:10:46.212 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:10:45 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:10:46.212 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:10:45 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:10:46.212 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:10:45 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:10:46.212 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:10:45 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T00:10:46.213 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:10:45 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:10:46.213 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:10:45 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T00:10:46.214 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:10:45 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:10:46.215 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:10:45 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T00:10:46.215 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:10:45 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:10:46.215 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:10:45 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T00:10:46.215 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:10:45 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:10:46.216 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:10:45 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T00:10:46.216 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:10:45 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:10:46.216 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:10:45 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:10:46.216 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:10:45 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:10:46.217 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:10:45 smithi047 conmon[78521]: 2022-05-14T00:10:45.872+0000 7f5344f82700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T00:10:46.217 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:10:45 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T00:10:46.217 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:10:45 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:10:46.217 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:10:45 smithi047 conmon[78521]: yield 2022-05-14T00:10:46.217 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:10:45 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:10:46.218 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:10:45 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:10:46.218 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:10:45 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:10:46.218 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:10:45 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T00:10:46.218 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:10:45 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:10:46.219 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:10:45 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T00:10:46.219 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:10:45 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:10:46.219 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:10:45 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T00:10:46.219 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:10:45 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:10:46.219 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:10:45 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T00:10:46.220 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:10:45 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:10:46.220 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:10:45 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T00:10:46.220 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:10:45 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:10:46.220 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:10:45 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:10:46.221 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:10:45 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:10:46.221 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:10:45 smithi047 conmon[78521]: 2022-05-14T00:10:46.221 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:10:45 smithi047 conmon[78521]: During handling of the above exception, another exception occurred: 2022-05-14T00:10:46.221 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:10:45 smithi047 conmon[78521]: 2022-05-14T00:10:46.221 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:10:45 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T00:10:46.222 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:10:45 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T00:10:46.222 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:10:45 smithi047 conmon[78521]: return f(*arg) 2022-05-14T00:10:46.222 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:10:45 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T00:10:46.222 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:10:45 smithi047 conmon[78521]: self._write_client_files(client_files, host) 2022-05-14T00:10:46.223 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:10:45 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T00:10:46.223 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:10:45 smithi047 conmon[78521]: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T00:10:46.223 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:10:45 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T00:10:46.224 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:10:45 smithi047 conmon[78521]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T00:10:46.224 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:10:45 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T00:10:46.224 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:10:45 smithi047 conmon[78521]: return self.event_loop.get_result(coro) 2022-05-14T00:10:46.225 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:10:45 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T00:10:46.225 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:10:45 smithi047 conmon[78521]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T00:10:46.225 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:10:45 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T00:10:46.225 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:10:45 smithi047 conmon[78521]: return self.__get_result() 2022-05-14T00:10:46.226 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:10:45 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T00:10:46.226 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:10:45 smithi047 conmon[78521]: raise self._exception 2022-05-14T00:10:46.226 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:10:45 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T00:10:46.227 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:10:45 smithi047 conmon[78521]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T00:10:46.227 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:10:45 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T00:10:46.227 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:10:45 smithi047 conmon[78521]: conn = await self._remote_connection(host, addr) 2022-05-14T00:10:46.227 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:10:45 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T00:10:46.228 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:10:45 smithi047 conmon[78521]: raise 2022-05-14T00:10:46.228 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:10:45 smithi047 conmon[78521]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T00:10:46.228 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:10:45 smithi047 conmon[78521]: self.gen.throw(type, value, traceback) 2022-05-14T00:10:46.228 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:10:45 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T00:10:46.229 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:10:45 smithi047 conmon[78521]: raise OrchestratorError(msg) 2022-05-14T00:10:46.229 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:10:45 smithi047 conmon[78521]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:10:46.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:10:46 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:10:46.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:10:46 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:10:47.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:10:46 smithi115 conmon[90393]: 2022-05-14T00:10:46.609+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:10:47.547 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:10:47 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:10:47.548 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:10:47 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:10:47.548 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T00:10:47.548 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:10:47.549 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T00:10:47.549 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:10:47.549 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:10:47.549 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:10:47.550 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T00:10:47.550 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:10:47.550 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T00:10:47.550 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:10:47.551 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T00:10:47.551 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:10:47.551 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T00:10:47.551 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:10:47.552 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T00:10:47.552 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:10:47.552 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:10:47.553 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:10:47.553 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:10:47 smithi047 ceph-mon[77147]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T00:10:47.553 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T00:10:47.553 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:10:47.554 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T00:10:47.554 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:10:47.554 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:10:47.555 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:10:47.555 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T00:10:47.556 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:10:47.556 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T00:10:47.556 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:10:47.556 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T00:10:47.557 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:10:47.557 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T00:10:47.557 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:10:47.557 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T00:10:47.558 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:10:47.558 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:10:47.558 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:10:47.559 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T00:10:47.559 INFO:journalctl@ceph.mon.b.smithi047.stdout: During handling of the above exception, another exception occurred: 2022-05-14T00:10:47.559 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T00:10:47.559 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T00:10:47.560 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T00:10:47.560 INFO:journalctl@ceph.mon.b.smithi047.stdout: return f(*arg) 2022-05-14T00:10:47.561 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T00:10:47.561 INFO:journalctl@ceph.mon.b.smithi047.stdout: self._write_client_files(client_files, host) 2022-05-14T00:10:47.561 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T00:10:47.562 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T00:10:47.562 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T00:10:47.562 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T00:10:47.562 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T00:10:47.563 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.event_loop.get_result(coro) 2022-05-14T00:10:47.563 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T00:10:47.563 INFO:journalctl@ceph.mon.b.smithi047.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T00:10:47.564 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T00:10:47.564 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.__get_result() 2022-05-14T00:10:47.564 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T00:10:47.565 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise self._exception 2022-05-14T00:10:47.565 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T00:10:47.565 INFO:journalctl@ceph.mon.b.smithi047.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T00:10:47.566 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T00:10:47.566 INFO:journalctl@ceph.mon.b.smithi047.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T00:10:47.566 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T00:10:47.566 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise 2022-05-14T00:10:47.567 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T00:10:47.567 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.gen.throw(type, value, traceback) 2022-05-14T00:10:47.567 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T00:10:47.568 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OrchestratorError(msg) 2022-05-14T00:10:47.568 INFO:journalctl@ceph.mon.b.smithi047.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:10:47.568 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:10:47 smithi047 ceph-mon[77147]: pgmap v1642: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:10:47.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:10:47 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:10:47.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:10:47 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:10:47.608 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T00:10:47.608 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:10:47.608 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T00:10:47.609 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:10:47.609 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:10:47.609 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:10:47.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T00:10:47.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:10:47.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T00:10:47.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:10:47.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T00:10:47.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:10:47.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T00:10:47.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:10:47.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T00:10:47.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:10:47.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:10:47.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:10:47.612 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:10:47 smithi115 ceph-mon[78593]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T00:10:47.613 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T00:10:47.613 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:10:47.613 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T00:10:47.613 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:10:47.614 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:10:47.614 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:10:47.614 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T00:10:47.614 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:10:47.615 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T00:10:47.615 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:10:47.615 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T00:10:47.615 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:10:47.615 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T00:10:47.616 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:10:47.616 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T00:10:47.616 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:10:47.616 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:10:47.617 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:10:47.617 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T00:10:47.617 INFO:journalctl@ceph.mon.c.smithi115.stdout: During handling of the above exception, another exception occurred: 2022-05-14T00:10:47.617 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T00:10:47.618 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T00:10:47.618 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T00:10:47.618 INFO:journalctl@ceph.mon.c.smithi115.stdout: return f(*arg) 2022-05-14T00:10:47.618 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T00:10:47.618 INFO:journalctl@ceph.mon.c.smithi115.stdout: self._write_client_files(client_files, host) 2022-05-14T00:10:47.619 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T00:10:47.619 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T00:10:47.619 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T00:10:47.619 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T00:10:47.620 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T00:10:47.620 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.event_loop.get_result(coro) 2022-05-14T00:10:47.620 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T00:10:47.620 INFO:journalctl@ceph.mon.c.smithi115.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T00:10:47.621 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T00:10:47.621 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.__get_result() 2022-05-14T00:10:47.621 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T00:10:47.621 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise self._exception 2022-05-14T00:10:47.621 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T00:10:47.622 INFO:journalctl@ceph.mon.c.smithi115.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T00:10:47.622 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T00:10:47.622 INFO:journalctl@ceph.mon.c.smithi115.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T00:10:47.622 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T00:10:47.623 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise 2022-05-14T00:10:47.623 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T00:10:47.623 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.gen.throw(type, value, traceback) 2022-05-14T00:10:47.623 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T00:10:47.623 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OrchestratorError(msg) 2022-05-14T00:10:47.624 INFO:journalctl@ceph.mon.c.smithi115.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:10:47.624 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:10:47 smithi115 ceph-mon[78593]: pgmap v1642: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:10:48.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:10:47 smithi115 conmon[90393]: 2022-05-14T00:10:47.622+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:10:48.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:10:48 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:10:48.605 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:10:48 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:10:48.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:10:48 smithi115 conmon[90393]: 2022-05-14T00:10:48.604+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:10:49.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:10:49 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:10:49.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:10:49 smithi047 ceph-mon[77147]: pgmap v1643: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:10:49.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:10:49 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 2580 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:10:49.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:10:49 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:10:49.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:10:49 smithi115 ceph-mon[78593]: pgmap v1643: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:10:49.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:10:49 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 2580 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:10:50.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:10:49 smithi115 conmon[90393]: 2022-05-14T00:10:49.642+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:10:50.550 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:10:50 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:10:50.550 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:10:50 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:10:50.550 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:10:50 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:10:50.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:10:50 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:10:50.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:10:50 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:10:50.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:10:50 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:10:51.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:10:50 smithi115 conmon[90393]: 2022-05-14T00:10:50.638+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:10:51.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:10:51 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:10:51.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:10:51 smithi047 ceph-mon[77147]: pgmap v1644: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:10:51.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:10:51 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:10:51.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:10:51 smithi115 ceph-mon[78593]: pgmap v1644: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:10:52.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:10:51 smithi115 conmon[90393]: 2022-05-14T00:10:51.673+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:10:52.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:10:52 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:10:52.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:10:52 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:10:53.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:10:52 smithi115 conmon[90393]: 2022-05-14T00:10:52.662+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:10:53.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:10:53 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:10:53.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:10:53 smithi047 ceph-mon[77147]: pgmap v1645: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:10:53.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:10:53 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:10:53.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:10:53 smithi115 ceph-mon[78593]: pgmap v1645: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:10:54.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:10:53 smithi115 conmon[90393]: 2022-05-14T00:10:53.615+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:10:54.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:10:54 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:10:54.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:10:54 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:10:54.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:10:54 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:10:54.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:10:54 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 2585 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:10:54.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:10:54 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:10:54.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:10:54 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:10:54.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:10:54 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:10:54.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:10:54 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 2585 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:10:55.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:10:54 smithi115 conmon[90393]: 2022-05-14T00:10:54.622+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:10:55.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:10:55 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:10:55.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:10:55 smithi047 ceph-mon[77147]: pgmap v1646: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:10:55.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:10:55 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:10:55.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:10:55 smithi115 ceph-mon[78593]: pgmap v1646: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:10:56.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:10:55 smithi115 conmon[90393]: 2022-05-14T00:10:55.652+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:10:56.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:10:56 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:10:56.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:10:56 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:10:57.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:10:56 smithi115 conmon[90393]: 2022-05-14T00:10:56.624+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:10:57.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:10:57 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:10:57.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:10:57 smithi047 ceph-mon[77147]: pgmap v1647: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:10:57.590 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:10:57 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:10:57.590 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:10:57 smithi115 ceph-mon[78593]: pgmap v1647: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:10:57.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:10:57 smithi115 conmon[90393]: 2022-05-14T00:10:57.589+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:10:58.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:10:58 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:10:58.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:10:58 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:10:58.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:10:58 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:10:58.567 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:10:58 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:10:58.567 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:10:58 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:10:58.567 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:10:58 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:10:58.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:10:58 smithi115 conmon[90393]: 2022-05-14T00:10:58.566+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:10:59.536 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:10:59 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:10:59.536 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:10:59 smithi115 ceph-mon[78593]: pgmap v1648: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:10:59.536 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:10:59 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 2590 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:10:59.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:10:59 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:10:59.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:10:59 smithi047 ceph-mon[77147]: pgmap v1648: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:10:59.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:10:59 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 2590 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:10:59.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:10:59 smithi115 conmon[90393]: 2022-05-14T00:10:59.534+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:11:00.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:11:00 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:11:00.556 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:11:00 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:11:00.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:11:00 smithi115 conmon[90393]: 2022-05-14T00:11:00.555+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:11:01.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:11:01 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:11:01.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:11:01 smithi047 ceph-mon[77147]: pgmap v1649: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:11:01.554 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:11:01 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:11:01.555 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:11:01 smithi115 ceph-mon[78593]: pgmap v1649: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:11:01.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:11:01 smithi115 conmon[90393]: 2022-05-14T00:11:01.553+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:11:02.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:11:02 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:11:02.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:11:02 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:11:02.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:11:02 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:11:02.574 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:11:02 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:11:02.575 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:11:02 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:11:02.575 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:11:02 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:11:02.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:11:02 smithi115 conmon[90393]: 2022-05-14T00:11:02.572+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:11:03.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:11:03 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:11:03.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:11:03 smithi047 ceph-mon[77147]: pgmap v1650: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:11:03.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:11:03 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:11:03.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:11:03 smithi115 ceph-mon[78593]: pgmap v1650: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:11:03.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:11:03 smithi115 conmon[90393]: 2022-05-14T00:11:03.546+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:11:04.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:11:04 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:11:04.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:11:04 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 2595 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:11:04.576 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:11:04 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:11:04.577 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:11:04 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 2595 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:11:04.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:11:04 smithi115 conmon[90393]: 2022-05-14T00:11:04.575+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:11:05.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:11:05 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:11:05.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:11:05 smithi047 ceph-mon[77147]: pgmap v1651: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:11:05.570 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:11:05 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:11:05.570 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:11:05 smithi115 ceph-mon[78593]: pgmap v1651: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:11:05.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:11:05 smithi115 conmon[90393]: 2022-05-14T00:11:05.568+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:11:06.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:11:06 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:11:06.551 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:11:06 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:11:06.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:11:06 smithi115 conmon[90393]: 2022-05-14T00:11:06.548+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:11:07.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:11:07 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:11:07.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:11:07 smithi047 ceph-mon[77147]: pgmap v1652: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:11:07.556 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:11:07 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:11:07.556 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:11:07 smithi115 ceph-mon[78593]: pgmap v1652: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:11:07.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:11:07 smithi115 conmon[90393]: 2022-05-14T00:11:07.549+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:11:08.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:11:08 smithi115 conmon[90393]: 2022-05-14T00:11:08.526+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:11:08.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:11:08 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:11:08.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:11:08 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:11:09.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:11:09 smithi115 conmon[90393]: 2022-05-14T00:11:09.510+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:11:09.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:11:09 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:11:09.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:11:09 smithi115 ceph-mon[78593]: pgmap v1653: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:11:09.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:11:09 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 2600 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:11:09.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:11:09 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:11:09.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:11:09 smithi047 ceph-mon[77147]: pgmap v1653: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:11:09.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:11:09 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 2600 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:11:10.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:11:10 smithi115 conmon[90393]: 2022-05-14T00:11:10.513+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:11:10.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:11:10 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:11:10.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:11:10 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:11:10.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:11:10 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:11:10.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:11:10 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:11:10.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:11:10 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:11:10.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:11:10 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:11:11.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:11:11 smithi115 conmon[90393]: 2022-05-14T00:11:11.468+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:11:11.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:11:11 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:11:11.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:11:11 smithi115 ceph-mon[78593]: pgmap v1654: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:11:11.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:11:11 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:11:11.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:11:11 smithi047 ceph-mon[77147]: pgmap v1654: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:11:12.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:11:12 smithi115 conmon[90393]: 2022-05-14T00:11:12.465+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:11:12.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:11:12 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:11:12.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:11:12 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:11:12.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:11:12 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:11:12.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:11:12 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:11:13.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:11:13 smithi115 conmon[90393]: 2022-05-14T00:11:13.419+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:11:13.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:11:13 smithi115 ceph-mon[78593]: pgmap v1655: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:11:13.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:11:13 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:11:13.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:11:13 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T00:11:13.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:11:13 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T00:11:13.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:11:13 smithi047 ceph-mon[77147]: pgmap v1655: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:11:13.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:11:13 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:11:13.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:11:13 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T00:11:13.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:11:13 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T00:11:14.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:11:14 smithi115 conmon[90393]: 2022-05-14T00:11:14.389+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:11:14.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:11:14 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:11:14.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:11:14 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:11:14.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:11:14 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:11:14.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:11:14 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 2605 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:11:14.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:11:14 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:11:14.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:11:14 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:11:14.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:11:14 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:11:14.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:11:14 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 2605 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:11:15.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:11:15 smithi115 conmon[90393]: 2022-05-14T00:11:15.437+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:11:15.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:11:15 smithi115 ceph-mon[78593]: pgmap v1656: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:11:15.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:11:15 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:11:15.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:11:15 smithi047 ceph-mon[77147]: pgmap v1656: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:11:15.852 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:11:15 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:11:16.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:11:16 smithi115 conmon[90393]: 2022-05-14T00:11:16.454+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:11:16.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:11:16 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:11:16.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:11:16 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:11:17.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:11:17 smithi115 conmon[90393]: 2022-05-14T00:11:17.408+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:11:17.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:11:17 smithi115 ceph-mon[78593]: pgmap v1657: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:11:17.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:11:17 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:11:17.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:11:17 smithi047 ceph-mon[77147]: pgmap v1657: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:11:17.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:11:17 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:11:18.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:11:18 smithi115 conmon[90393]: 2022-05-14T00:11:18.370+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:11:18.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:11:18 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:11:18.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:11:18 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:11:18.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:11:18 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:11:18.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:11:18 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:11:18.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:11:18 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:11:18.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:11:18 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:11:19.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:11:19 smithi115 conmon[90393]: 2022-05-14T00:11:19.353+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:11:19.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:11:19 smithi115 ceph-mon[78593]: pgmap v1658: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:11:19.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:11:19 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:11:19.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:11:19 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 2610 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:11:19.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:11:19 smithi047 ceph-mon[77147]: pgmap v1658: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:11:19.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:11:19 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:11:19.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:11:19 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 2610 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:11:20.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:11:20 smithi115 conmon[90393]: 2022-05-14T00:11:20.382+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:11:20.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:11:20 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:11:20.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:11:20 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:11:21.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:11:21 smithi115 conmon[90393]: 2022-05-14T00:11:21.406+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:11:21.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:11:21 smithi115 ceph-mon[78593]: pgmap v1659: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:11:21.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:11:21 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:11:21.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:11:21 smithi047 ceph-mon[77147]: pgmap v1659: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:11:21.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:11:21 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:11:22.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:11:22 smithi115 conmon[90393]: 2022-05-14T00:11:22.421+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:11:22.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:11:22 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:11:22.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:11:22 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:11:22.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:11:22 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:11:22.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:11:22 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:11:22.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:11:22 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:11:22.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:11:22 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:11:23.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:11:23 smithi115 conmon[90393]: 2022-05-14T00:11:23.459+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:11:23.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:11:23 smithi115 ceph-mon[78593]: pgmap v1660: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:11:23.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:11:23 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:11:23.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:11:23 smithi047 ceph-mon[77147]: pgmap v1660: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:11:23.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:11:23 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:11:24.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:11:24 smithi115 conmon[90393]: 2022-05-14T00:11:24.444+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:11:24.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:11:24 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:11:24.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:11:24 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 2615 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:11:24.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:11:24 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:11:24.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:11:24 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 2615 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:11:25.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:11:25 smithi115 conmon[90393]: 2022-05-14T00:11:25.448+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:11:25.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:11:25 smithi115 ceph-mon[78593]: pgmap v1661: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:11:25.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:11:25 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:11:25.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:11:25 smithi047 ceph-mon[77147]: pgmap v1661: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:11:25.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:11:25 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:11:26.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:11:26 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:11:26.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:11:26 smithi115 conmon[90393]: 2022-05-14T00:11:26.409+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:11:26.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:11:26 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:11:27.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:11:27 smithi047 ceph-mon[77147]: pgmap v1662: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:11:27.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:11:27 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:11:27.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:11:27 smithi115 conmon[90393]: 2022-05-14T00:11:27.424+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:11:27.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:11:27 smithi115 ceph-mon[78593]: pgmap v1662: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:11:27.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:11:27 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:11:28.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:11:28 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:11:28.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:11:28 smithi115 conmon[90393]: 2022-05-14T00:11:28.464+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:11:28.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:11:28 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:11:29.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:11:29 smithi047 ceph-mon[77147]: pgmap v1663: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:11:29.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:11:29 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:11:29.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:11:29 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 2620 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:11:29.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:11:29 smithi115 conmon[90393]: 2022-05-14T00:11:29.421+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:11:29.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:11:29 smithi115 ceph-mon[78593]: pgmap v1663: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:11:29.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:11:29 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:11:29.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:11:29 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 2620 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:11:30.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:11:30 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:11:30.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:11:30 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:11:30.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:11:30 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:11:30.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:11:30 smithi115 conmon[90393]: 2022-05-14T00:11:30.462+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:11:30.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:11:30 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:11:30.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:11:30 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:11:30.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:11:30 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:11:31.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:11:31 smithi047 ceph-mon[77147]: pgmap v1664: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:11:31.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:11:31 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:11:31.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:11:31 smithi115 conmon[90393]: 2022-05-14T00:11:31.415+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:11:31.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:11:31 smithi115 ceph-mon[78593]: pgmap v1664: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:11:31.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:11:31 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:11:32.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:11:32 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:11:32.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:11:32 smithi115 conmon[90393]: 2022-05-14T00:11:32.407+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:11:32.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:11:32 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:11:33.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:11:33 smithi047 ceph-mon[77147]: pgmap v1665: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:11:33.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:11:33 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:11:33.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:11:33 smithi115 conmon[90393]: 2022-05-14T00:11:33.445+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:11:33.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:11:33 smithi115 ceph-mon[78593]: pgmap v1665: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:11:33.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:11:33 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:11:34.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:11:34 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:11:34.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:11:34 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:11:34.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:11:34 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:11:34.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:11:34 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 2625 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:11:34.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:11:34 smithi115 conmon[90393]: 2022-05-14T00:11:34.414+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:11:34.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:11:34 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:11:34.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:11:34 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:11:34.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:11:34 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:11:34.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:11:34 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 2625 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:11:35.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:11:35 smithi047 ceph-mon[77147]: pgmap v1666: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:11:35.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:11:35 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:11:35.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:11:35 smithi115 conmon[90393]: 2022-05-14T00:11:35.369+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:11:35.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:11:35 smithi115 ceph-mon[78593]: pgmap v1666: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:11:35.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:11:35 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:11:36.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:11:36 smithi115 conmon[90393]: 2022-05-14T00:11:36.334+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:11:36.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:11:36 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:11:36.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:11:36 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:11:37.799 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:11:37 smithi047 ceph-mon[77147]: pgmap v1667: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:11:37.800 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:11:37 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:11:37.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:11:37 smithi115 conmon[90393]: 2022-05-14T00:11:37.361+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:11:37.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:11:37 smithi115 ceph-mon[78593]: pgmap v1667: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:11:37.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:11:37 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:11:38.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:11:38 smithi115 conmon[90393]: 2022-05-14T00:11:38.349+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:11:38.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:11:38 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:11:38.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:11:38 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:11:38.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:11:38 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:11:38.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:11:38 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:11:38.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:11:38 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:11:38.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:11:38 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:11:39.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:11:39 smithi047 ceph-mon[77147]: pgmap v1668: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:11:39.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:11:39 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:11:39.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:11:39 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 2630 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:11:39.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:11:39 smithi115 conmon[90393]: 2022-05-14T00:11:39.391+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:11:39.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:11:39 smithi115 ceph-mon[78593]: pgmap v1668: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:11:39.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:11:39 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:11:39.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:11:39 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 2630 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:11:40.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:11:40 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:11:40.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:11:40 smithi115 conmon[90393]: 2022-05-14T00:11:40.413+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:11:40.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:11:40 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:11:41.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:11:41 smithi047 ceph-mon[77147]: pgmap v1669: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:11:41.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:11:41 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:11:41.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:11:41 smithi115 conmon[90393]: 2022-05-14T00:11:41.442+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:11:41.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:11:41 smithi115 ceph-mon[78593]: pgmap v1669: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:11:41.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:11:41 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:11:42.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:11:42 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:11:42.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:11:42 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:11:42.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:11:42 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:11:42.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:11:42 smithi115 conmon[90393]: 2022-05-14T00:11:42.472+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:11:42.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:11:42 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:11:42.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:11:42 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:11:42.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:11:42 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:11:43.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:11:43 smithi047 ceph-mon[77147]: pgmap v1670: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:11:43.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:11:43 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:11:43.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:11:43 smithi115 conmon[90393]: 2022-05-14T00:11:43.427+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:11:43.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:11:43 smithi115 ceph-mon[78593]: pgmap v1670: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:11:43.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:11:43 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:11:44.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:11:44 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:11:44.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:11:44 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 2635 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:11:44.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:11:44 smithi115 conmon[90393]: 2022-05-14T00:11:44.441+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:11:44.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:11:44 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:11:44.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:11:44 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 2635 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:11:45.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:11:45 smithi047 ceph-mon[77147]: pgmap v1671: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:11:45.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:11:45 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:11:45.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:11:45 smithi115 conmon[90393]: 2022-05-14T00:11:45.471+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:11:45.865 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:11:45 smithi115 ceph-mon[78593]: pgmap v1671: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:11:45.865 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:11:45 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:11:46.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:11:46 smithi115 conmon[90393]: 2022-05-14T00:11:46.436+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:11:46.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:11:46 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:11:46.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:11:46 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T00:11:46.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:11:46 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T00:11:46.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:11:46 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T00:11:46.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:11:46 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' 2022-05-14T00:11:46.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:11:46 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' 2022-05-14T00:11:46.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:11:46 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:11:46.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:11:46 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T00:11:46.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:11:46 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T00:11:46.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:11:46 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T00:11:46.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:11:46 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' 2022-05-14T00:11:46.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:11:46 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' 2022-05-14T00:11:47.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:11:47 smithi047 ceph-mon[77147]: pgmap v1672: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:11:47.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:11:47 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:11:47.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:11:47 smithi115 conmon[90393]: 2022-05-14T00:11:47.427+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:11:47.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:11:47 smithi115 ceph-mon[78593]: pgmap v1672: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:11:47.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:11:47 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:11:48.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:11:48 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:11:48.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:11:48 smithi115 conmon[90393]: 2022-05-14T00:11:48.398+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:11:48.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:11:48 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:11:49.291 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:11:48 smithi047 conmon[78521]: 2022-05-14T00:11:48.976+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:11:49.291 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:11:48 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T00:11:49.292 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:11:48 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:11:49.292 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:11:48 smithi047 conmon[78521]: yield 2022-05-14T00:11:49.292 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:11:48 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:11:49.292 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:11:48 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:11:49.292 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:11:48 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:11:49.293 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:11:48 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T00:11:49.293 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:11:48 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:11:49.293 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:11:48 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T00:11:49.293 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:11:48 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:11:49.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:11:48 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T00:11:49.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:11:48 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:11:49.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:11:48 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T00:11:49.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:11:48 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:11:49.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:11:48 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T00:11:49.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:11:48 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:11:49.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:11:48 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:11:49.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:11:48 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:11:49.444 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:11:49 smithi115 conmon[90393]: 2022-05-14T00:11:49.373+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:11:49.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:11:49 smithi047 ceph-mon[77147]: pgmap v1673: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:11:49.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:11:49 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:11:49.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:11:49 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 2640 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:11:49.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:11:49 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' 2022-05-14T00:11:49.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:11:49 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' 2022-05-14T00:11:49.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:11:49 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-05-14T00:11:49.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:11:49 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-05-14T00:11:49.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:11:49 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-05-14T00:11:49.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:11:49 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-05-14T00:11:49.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:11:49 smithi115 ceph-mon[78593]: pgmap v1673: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:11:49.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:11:49 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:11:49.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:11:49 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 2640 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:11:49.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:11:49 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' 2022-05-14T00:11:49.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:11:49 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' 2022-05-14T00:11:49.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:11:49 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-05-14T00:11:49.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:11:49 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-05-14T00:11:49.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:11:49 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-05-14T00:11:49.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:11:49 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-05-14T00:11:50.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:11:50 smithi115 conmon[90393]: 2022-05-14T00:11:50.351+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:11:50.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:11:50 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:11:50.609 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T00:11:50.609 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:11:50.609 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T00:11:50.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:11:50.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:11:50.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:11:50.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T00:11:50.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:11:50.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T00:11:50.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:11:50.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T00:11:50.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:11:50.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T00:11:50.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:11:50.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T00:11:50.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:11:50.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:11:50.613 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:11:50.613 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:11:50 smithi115 ceph-mon[78593]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T00:11:50.613 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:11:50 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:11:50.614 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:11:50 smithi115 ceph-mon[78593]: Adjusting osd_memory_target on smithi047 to 198.1M 2022-05-14T00:11:50.614 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:11:50 smithi115 ceph-mon[78593]: Unable to set osd_memory_target on smithi047 to 207755878: error parsing value: Value '207755878' is below minimum 939524096 2022-05-14T00:11:50.614 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:11:50 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:11:50.615 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:11:50 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:11:50.615 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:11:50 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-05-14T00:11:50.615 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:11:50 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-05-14T00:11:50.615 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:11:50 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-05-14T00:11:50.615 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:11:50 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-05-14T00:11:50.616 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:11:50 smithi115 ceph-mon[78593]: Adjusting osd_memory_target on smithi115 to 197.0M 2022-05-14T00:11:50.616 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:11:50 smithi115 ceph-mon[78593]: Unable to set osd_memory_target on smithi115 to 206629785: error parsing value: Value '206629785' is below minimum 939524096 2022-05-14T00:11:50.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:11:50 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:11:50.791 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T00:11:50.792 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:11:50.792 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T00:11:50.793 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:11:50.793 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:11:50.793 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:11:50.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T00:11:50.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:11:50.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T00:11:50.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:11:50.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T00:11:50.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:11:50.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T00:11:50.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:11:50.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T00:11:50.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:11:50.797 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:11:50.797 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:11:50.797 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:11:50 smithi047 ceph-mon[77147]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T00:11:50.797 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:11:50 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:11:50.798 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:11:50 smithi047 ceph-mon[77147]: Adjusting osd_memory_target on smithi047 to 198.1M 2022-05-14T00:11:50.798 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:11:50 smithi047 ceph-mon[77147]: Unable to set osd_memory_target on smithi047 to 207755878: error parsing value: Value '207755878' is below minimum 939524096 2022-05-14T00:11:50.798 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:11:50 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:11:50.798 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:11:50 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:11:50.798 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:11:50 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-05-14T00:11:50.799 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:11:50 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-05-14T00:11:50.799 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:11:50 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-05-14T00:11:50.799 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:11:50 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-05-14T00:11:50.799 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:11:50 smithi047 ceph-mon[77147]: Adjusting osd_memory_target on smithi115 to 197.0M 2022-05-14T00:11:50.800 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:11:50 smithi047 ceph-mon[77147]: Unable to set osd_memory_target on smithi115 to 206629785: error parsing value: Value '206629785' is below minimum 939524096 2022-05-14T00:11:51.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:11:51 smithi047 ceph-mon[77147]: pgmap v1674: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:11:51.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:11:51 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:11:51.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:11:51 smithi115 conmon[90393]: 2022-05-14T00:11:51.358+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:11:51.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:11:51 smithi115 ceph-mon[78593]: pgmap v1674: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:11:51.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:11:51 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:11:52.456 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:11:52 smithi047 conmon[78521]: 2022-05-14T00:11:52.048+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:11:52.457 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:11:52 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T00:11:52.457 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:11:52 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:11:52.458 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:11:52 smithi047 conmon[78521]: yield 2022-05-14T00:11:52.458 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:11:52 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:11:52.458 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:11:52 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:11:52.459 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:11:52 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:11:52.459 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:11:52 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T00:11:52.460 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:11:52 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:11:52.460 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:11:52 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T00:11:52.460 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:11:52 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:11:52.461 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:11:52 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T00:11:52.461 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:11:52 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:11:52.461 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:11:52 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T00:11:52.462 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:11:52 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:11:52.462 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:11:52 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T00:11:52.463 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:11:52 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:11:52.463 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:11:52 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:11:52.464 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:11:52 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:11:52.464 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:11:52 smithi047 conmon[78521]: 2022-05-14T00:11:52.049+0000 7f5344781700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T00:11:52.464 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:11:52 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T00:11:52.465 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:11:52 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:11:52.465 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:11:52 smithi047 conmon[78521]: yield 2022-05-14T00:11:52.466 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:11:52 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:11:52.466 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:11:52 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:11:52.466 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:11:52 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:11:52.467 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:11:52 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T00:11:52.467 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:11:52 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:11:52.467 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:11:52 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T00:11:52.468 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:11:52 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:11:52.468 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:11:52 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T00:11:52.469 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:11:52 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:11:52.469 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:11:52 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T00:11:52.469 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:11:52 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:11:52.470 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:11:52 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T00:11:52.470 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:11:52 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:11:52.471 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:11:52 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:11:52.471 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:11:52 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:11:52.471 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:11:52 smithi047 conmon[78521]: 2022-05-14T00:11:52.472 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:11:52 smithi047 conmon[78521]: During handling of the above exception, another exception occurred: 2022-05-14T00:11:52.472 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:11:52 smithi047 conmon[78521]: 2022-05-14T00:11:52.473 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:11:52 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T00:11:52.473 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:11:52 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T00:11:52.473 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:11:52 smithi047 conmon[78521]: return f(*arg) 2022-05-14T00:11:52.474 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:11:52 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T00:11:52.474 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:11:52 smithi047 conmon[78521]: self._write_client_files(client_files, host) 2022-05-14T00:11:52.475 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:11:52 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T00:11:52.475 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:11:52 smithi047 conmon[78521]: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T00:11:52.475 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:11:52 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T00:11:52.476 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:11:52 smithi047 conmon[78521]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T00:11:52.476 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:11:52 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T00:11:52.476 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:11:52 smithi047 conmon[78521]: return self.event_loop.get_result(coro) 2022-05-14T00:11:52.477 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:11:52 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T00:11:52.477 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:11:52 smithi047 conmon[78521]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T00:11:52.477 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:11:52 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T00:11:52.478 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:11:52 smithi047 conmon[78521]: return self.__get_result() 2022-05-14T00:11:52.478 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:11:52 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T00:11:52.479 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:11:52 smithi047 conmon[78521]: raise self._exception 2022-05-14T00:11:52.479 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:11:52 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T00:11:52.479 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:11:52 smithi047 conmon[78521]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T00:11:52.480 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:11:52 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T00:11:52.480 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:11:52 smithi047 conmon[78521]: conn = await self._remote_connection(host, addr) 2022-05-14T00:11:52.480 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:11:52 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T00:11:52.481 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:11:52 smithi047 conmon[78521]: raise 2022-05-14T00:11:52.481 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:11:52 smithi047 conmon[78521]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T00:11:52.481 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:11:52 smithi047 conmon[78521]: self.gen.throw(type, value, traceback) 2022-05-14T00:11:52.482 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:11:52 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T00:11:52.482 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:11:52 smithi047 conmon[78521]: raise OrchestratorError(msg) 2022-05-14T00:11:52.483 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:11:52 smithi047 conmon[78521]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:11:52.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:11:52 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:11:52.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:11:52 smithi115 conmon[90393]: 2022-05-14T00:11:52.398+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:11:52.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:11:52 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:11:53.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:11:53 smithi047 ceph-mon[77147]: pgmap v1675: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:11:53.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:11:53 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:11:53.791 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T00:11:53.792 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:11:53.792 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T00:11:53.792 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:11:53.792 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:11:53.792 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:11:53.793 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T00:11:53.793 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:11:53.793 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T00:11:53.793 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:11:53.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T00:11:53.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:11:53.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T00:11:53.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:11:53.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T00:11:53.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:11:53.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:11:53.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:11:53.795 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:11:53 smithi047 ceph-mon[77147]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T00:11:53.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T00:11:53.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:11:53.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T00:11:53.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:11:53.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:11:53.797 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:11:53.797 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T00:11:53.797 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:11:53.797 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T00:11:53.798 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:11:53.798 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T00:11:53.798 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:11:53.798 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T00:11:53.798 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:11:53.799 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T00:11:53.799 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:11:53.799 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:11:53.799 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:11:53.800 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T00:11:53.800 INFO:journalctl@ceph.mon.b.smithi047.stdout: During handling of the above exception, another exception occurred: 2022-05-14T00:11:53.800 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T00:11:53.800 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T00:11:53.800 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T00:11:53.801 INFO:journalctl@ceph.mon.b.smithi047.stdout: return f(*arg) 2022-05-14T00:11:53.801 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T00:11:53.801 INFO:journalctl@ceph.mon.b.smithi047.stdout: self._write_client_files(client_files, host) 2022-05-14T00:11:53.801 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T00:11:53.802 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T00:11:53.802 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T00:11:53.802 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T00:11:53.802 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T00:11:53.803 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.event_loop.get_result(coro) 2022-05-14T00:11:53.803 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T00:11:53.803 INFO:journalctl@ceph.mon.b.smithi047.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T00:11:53.803 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T00:11:53.803 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.__get_result() 2022-05-14T00:11:53.804 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T00:11:53.804 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise self._exception 2022-05-14T00:11:53.804 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T00:11:53.804 INFO:journalctl@ceph.mon.b.smithi047.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T00:11:53.804 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T00:11:53.805 INFO:journalctl@ceph.mon.b.smithi047.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T00:11:53.805 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T00:11:53.805 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise 2022-05-14T00:11:53.805 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T00:11:53.806 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.gen.throw(type, value, traceback) 2022-05-14T00:11:53.806 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T00:11:53.806 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OrchestratorError(msg) 2022-05-14T00:11:53.806 INFO:journalctl@ceph.mon.b.smithi047.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:11:53.807 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:11:53 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:11:53.807 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:11:53 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:11:53.807 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:11:53 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:11:53.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:11:53 smithi115 conmon[90393]: 2022-05-14T00:11:53.447+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:11:53.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:11:53 smithi115 ceph-mon[78593]: pgmap v1675: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:11:53.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:11:53 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:11:53.859 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T00:11:53.859 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:11:53.859 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T00:11:53.860 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:11:53.860 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:11:53.860 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:11:53.860 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T00:11:53.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:11:53.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T00:11:53.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:11:53.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T00:11:53.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:11:53.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T00:11:53.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:11:53.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T00:11:53.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:11:53.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:11:53.863 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:11:53.863 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:11:53 smithi115 ceph-mon[78593]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T00:11:53.863 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T00:11:53.863 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:11:53.864 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T00:11:53.864 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:11:53.864 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:11:53.865 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:11:53.865 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T00:11:53.865 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:11:53.865 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T00:11:53.865 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:11:53.866 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T00:11:53.866 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:11:53.866 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T00:11:53.866 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:11:53.867 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T00:11:53.867 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:11:53.867 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:11:53.867 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:11:53.867 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T00:11:53.868 INFO:journalctl@ceph.mon.c.smithi115.stdout: During handling of the above exception, another exception occurred: 2022-05-14T00:11:53.868 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T00:11:53.868 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T00:11:53.868 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T00:11:53.869 INFO:journalctl@ceph.mon.c.smithi115.stdout: return f(*arg) 2022-05-14T00:11:53.869 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T00:11:53.869 INFO:journalctl@ceph.mon.c.smithi115.stdout: self._write_client_files(client_files, host) 2022-05-14T00:11:53.869 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T00:11:53.869 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T00:11:53.870 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T00:11:53.870 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T00:11:53.870 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T00:11:53.870 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.event_loop.get_result(coro) 2022-05-14T00:11:53.871 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T00:11:53.871 INFO:journalctl@ceph.mon.c.smithi115.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T00:11:53.871 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T00:11:53.871 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.__get_result() 2022-05-14T00:11:53.871 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T00:11:53.872 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise self._exception 2022-05-14T00:11:53.872 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T00:11:53.872 INFO:journalctl@ceph.mon.c.smithi115.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T00:11:53.872 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T00:11:53.872 INFO:journalctl@ceph.mon.c.smithi115.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T00:11:53.873 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T00:11:53.873 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise 2022-05-14T00:11:53.873 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T00:11:53.873 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.gen.throw(type, value, traceback) 2022-05-14T00:11:53.874 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T00:11:53.874 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OrchestratorError(msg) 2022-05-14T00:11:53.874 INFO:journalctl@ceph.mon.c.smithi115.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:11:53.874 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:11:53 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:11:53.874 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:11:53 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:11:53.875 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:11:53 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:11:54.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:11:54 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:11:54.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:11:54 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 2645 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:11:54.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:11:54 smithi115 conmon[90393]: 2022-05-14T00:11:54.497+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:11:54.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:11:54 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:11:54.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:11:54 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 2645 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:11:55.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:11:55 smithi047 ceph-mon[77147]: pgmap v1676: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:11:55.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:11:55 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:11:55.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:11:55 smithi115 conmon[90393]: 2022-05-14T00:11:55.526+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:11:55.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:11:55 smithi115 ceph-mon[78593]: pgmap v1676: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:11:55.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:11:55 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:11:56.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:11:56 smithi115 conmon[90393]: 2022-05-14T00:11:56.498+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:11:57.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:11:57 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:11:57.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:11:57 smithi047 ceph-mon[77147]: pgmap v1677: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:11:57.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:11:57 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:11:57.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:11:57 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:11:57.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:11:57 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:11:57.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:11:57 smithi115 conmon[90393]: 2022-05-14T00:11:57.487+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:11:57.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:11:57 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:11:57.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:11:57 smithi115 ceph-mon[78593]: pgmap v1677: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:11:57.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:11:57 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:11:57.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:11:57 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:11:57.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:11:57 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:11:58.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:11:58 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:11:58.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:11:58 smithi115 conmon[90393]: 2022-05-14T00:11:58.503+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:11:58.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:11:58 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:11:59.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:11:59 smithi047 ceph-mon[77147]: pgmap v1678: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:11:59.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:11:59 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 2650 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:11:59.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:11:59 smithi115 conmon[90393]: 2022-05-14T00:11:59.512+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:11:59.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:11:59 smithi115 ceph-mon[78593]: pgmap v1678: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:11:59.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:11:59 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 2650 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:12:00.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:12:00 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:12:00.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:12:00 smithi115 conmon[90393]: 2022-05-14T00:12:00.560+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:12:00.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:12:00 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:12:01.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:12:01 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:12:01.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:12:01 smithi047 ceph-mon[77147]: pgmap v1679: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:12:01.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:12:01 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:12:01.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:12:01 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:12:01.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:12:01 smithi115 conmon[90393]: 2022-05-14T00:12:01.534+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:12:01.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:12:01 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:12:01.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:12:01 smithi115 ceph-mon[78593]: pgmap v1679: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:12:01.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:12:01 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:12:01.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:12:01 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:12:02.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:12:02 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:12:02.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:12:02 smithi115 conmon[90393]: 2022-05-14T00:12:02.500+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:12:02.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:12:02 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:12:03.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:12:03 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:12:03.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:12:03 smithi047 ceph-mon[77147]: pgmap v1680: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:12:03.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:12:03 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:12:03.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:12:03 smithi115 conmon[90393]: 2022-05-14T00:12:03.506+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:12:03.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:12:03 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:12:03.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:12:03 smithi115 ceph-mon[78593]: pgmap v1680: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:12:03.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:12:03 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:12:04.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:12:04 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 2655 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:12:04.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:12:04 smithi115 conmon[90393]: 2022-05-14T00:12:04.539+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:12:04.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:12:04 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 2655 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:12:05.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:12:05 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:12:05.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:12:05 smithi047 ceph-mon[77147]: pgmap v1681: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:12:05.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:12:05 smithi115 conmon[90393]: 2022-05-14T00:12:05.558+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:12:05.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:12:05 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:12:05.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:12:05 smithi115 ceph-mon[78593]: pgmap v1681: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:12:06.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:12:06 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:12:06.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:12:06 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:12:06.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:12:06 smithi115 conmon[90393]: 2022-05-14T00:12:06.533+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:12:07.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:12:07 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:12:07.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:12:07 smithi047 ceph-mon[77147]: pgmap v1682: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:12:07.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:12:07 smithi115 conmon[90393]: 2022-05-14T00:12:07.582+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:12:07.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:12:07 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:12:07.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:12:07 smithi115 ceph-mon[78593]: pgmap v1682: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:12:08.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:12:08 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:12:08.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:12:08 smithi115 conmon[90393]: 2022-05-14T00:12:08.560+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:12:08.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:12:08 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:12:09.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:12:09 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:12:09.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:12:09 smithi047 ceph-mon[77147]: pgmap v1683: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:12:09.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:12:09 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:12:09.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:12:09 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:12:09.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:12:09 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 2660 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:12:09.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:12:09 smithi115 conmon[90393]: 2022-05-14T00:12:09.602+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:12:09.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:12:09 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:12:09.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:12:09 smithi115 ceph-mon[78593]: pgmap v1683: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:12:09.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:12:09 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:12:09.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:12:09 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:12:09.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:12:09 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 2660 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:12:10.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:12:10 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:12:10.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:12:10 smithi115 conmon[90393]: 2022-05-14T00:12:10.608+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:12:10.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:12:10 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:12:11.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:12:11 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:12:11.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:12:11 smithi047 ceph-mon[77147]: pgmap v1684: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:12:11.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:12:11 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:12:11.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:12:11 smithi115 conmon[90393]: 2022-05-14T00:12:11.610+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:12:11.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:12:11 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:12:11.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:12:11 smithi115 ceph-mon[78593]: pgmap v1684: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:12:11.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:12:11 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:12:12.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:12:12 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:12:12.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:12:12 smithi115 conmon[90393]: 2022-05-14T00:12:12.595+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:12:12.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:12:12 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:12:13.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:12:13 smithi047 ceph-mon[77147]: pgmap v1685: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:12:13.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:12:13 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:12:13.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:12:13 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:12:13.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:12:13 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:12:13.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:12:13 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T00:12:13.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:12:13 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T00:12:13.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:12:13 smithi115 conmon[90393]: 2022-05-14T00:12:13.642+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:12:13.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:12:13 smithi115 ceph-mon[78593]: pgmap v1685: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:12:13.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:12:13 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:12:13.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:12:13 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:12:13.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:12:13 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:12:13.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:12:13 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T00:12:13.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:12:13 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T00:12:14.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:12:14 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:12:14.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:12:14 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 2665 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:12:14.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:12:14 smithi115 conmon[90393]: 2022-05-14T00:12:14.594+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:12:14.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:12:14 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:12:14.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:12:14 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 2665 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:12:15.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:12:15 smithi115 conmon[90393]: 2022-05-14T00:12:15.555+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:12:15.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:12:15 smithi115 ceph-mon[78593]: pgmap v1686: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:12:15.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:12:15 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:12:16.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:12:15 smithi047 ceph-mon[77147]: pgmap v1686: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:12:16.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:12:15 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:12:16.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:12:16 smithi115 conmon[90393]: 2022-05-14T00:12:16.596+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:12:16.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:12:16 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:12:17.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:12:16 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:12:17.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:12:17 smithi115 conmon[90393]: 2022-05-14T00:12:17.555+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:12:17.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:12:17 smithi115 ceph-mon[78593]: pgmap v1687: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:12:17.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:12:17 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:12:17.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:12:17 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:12:17.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:12:17 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:12:18.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:12:17 smithi047 ceph-mon[77147]: pgmap v1687: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:12:18.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:12:17 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:12:18.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:12:17 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:12:18.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:12:17 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:12:18.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:12:18 smithi115 conmon[90393]: 2022-05-14T00:12:18.518+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:12:18.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:12:18 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:12:18.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:12:18 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 2670 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:12:19.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:12:18 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:12:19.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:12:18 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 2670 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:12:19.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:12:19 smithi115 conmon[90393]: 2022-05-14T00:12:19.506+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:12:19.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:12:19 smithi115 ceph-mon[78593]: pgmap v1688: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:12:19.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:12:19 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:12:19.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:12:19 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:12:20.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:12:19 smithi047 ceph-mon[77147]: pgmap v1688: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:12:20.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:12:19 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:12:20.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:12:19 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:12:20.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:12:20 smithi115 conmon[90393]: 2022-05-14T00:12:20.511+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:12:21.726 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:12:21 smithi115 conmon[90393]: 2022-05-14T00:12:21.468+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:12:22.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:12:21 smithi047 ceph-mon[77147]: pgmap v1689: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:12:22.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:12:21 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:12:22.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:12:21 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:12:22.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:12:21 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:12:22.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:12:21 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:12:22.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:12:21 smithi115 ceph-mon[78593]: pgmap v1689: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:12:22.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:12:21 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:12:22.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:12:21 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:12:22.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:12:21 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:12:22.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:12:21 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:12:22.731 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:12:22 smithi115 conmon[90393]: 2022-05-14T00:12:22.424+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:12:23.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:12:22 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:12:23.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:12:22 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:12:23.732 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:12:23 smithi115 conmon[90393]: 2022-05-14T00:12:23.420+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:12:24.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:12:23 smithi047 ceph-mon[77147]: pgmap v1690: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:12:24.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:12:23 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:12:24.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:12:23 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 2675 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:12:24.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:12:23 smithi115 ceph-mon[78593]: pgmap v1690: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:12:24.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:12:23 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:12:24.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:12:23 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 2675 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:12:24.737 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:12:24 smithi115 conmon[90393]: 2022-05-14T00:12:24.376+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:12:25.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:12:24 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:12:25.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:12:24 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:12:25.739 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:12:25 smithi115 conmon[90393]: 2022-05-14T00:12:25.359+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:12:26.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:12:25 smithi047 ceph-mon[77147]: pgmap v1691: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:12:26.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:12:25 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:12:26.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:12:25 smithi115 ceph-mon[78593]: pgmap v1691: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:12:26.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:12:25 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:12:26.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:12:26 smithi115 conmon[90393]: 2022-05-14T00:12:26.341+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:12:27.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:12:26 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:12:27.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:12:26 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:12:27.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:12:27 smithi115 conmon[90393]: 2022-05-14T00:12:27.336+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:12:28.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:12:27 smithi047 ceph-mon[77147]: pgmap v1692: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:12:28.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:12:27 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:12:28.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:12:27 smithi115 ceph-mon[78593]: pgmap v1692: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:12:28.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:12:27 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:12:28.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:12:28 smithi115 conmon[90393]: 2022-05-14T00:12:28.301+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:12:29.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:12:28 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:12:29.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:12:28 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 2680 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:12:29.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:12:28 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:12:29.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:12:28 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 2680 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:12:29.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:12:29 smithi115 conmon[90393]: 2022-05-14T00:12:29.303+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:12:30.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:12:29 smithi047 ceph-mon[77147]: pgmap v1693: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:12:30.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:12:29 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:12:30.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:12:29 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:12:30.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:12:29 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:12:30.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:12:29 smithi115 ceph-mon[78593]: pgmap v1693: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:12:30.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:12:29 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:12:30.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:12:29 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:12:30.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:12:29 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:12:30.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:12:30 smithi115 conmon[90393]: 2022-05-14T00:12:30.347+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:12:31.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:12:30 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:12:31.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:12:30 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:12:31.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:12:31 smithi115 conmon[90393]: 2022-05-14T00:12:31.322+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:12:32.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:12:31 smithi047 ceph-mon[77147]: pgmap v1694: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:12:32.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:12:31 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:12:32.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:12:31 smithi115 ceph-mon[78593]: pgmap v1694: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:12:32.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:12:31 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:12:32.767 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:12:32 smithi115 conmon[90393]: 2022-05-14T00:12:32.360+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:12:33.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:12:32 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:12:33.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:12:32 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:12:33.770 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:12:33 smithi115 conmon[90393]: 2022-05-14T00:12:33.358+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:12:34.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:12:33 smithi047 ceph-mon[77147]: pgmap v1695: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:12:34.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:12:33 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:12:34.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:12:33 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:12:34.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:12:33 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:12:34.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:12:33 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 2685 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:12:34.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:12:33 smithi115 ceph-mon[78593]: pgmap v1695: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:12:34.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:12:33 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:12:34.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:12:33 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:12:34.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:12:33 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:12:34.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:12:33 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 2685 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:12:34.773 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:12:34 smithi115 conmon[90393]: 2022-05-14T00:12:34.362+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:12:35.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:12:34 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:12:35.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:12:34 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:12:35.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:12:35 smithi115 conmon[90393]: 2022-05-14T00:12:35.323+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:12:36.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:12:35 smithi047 ceph-mon[77147]: pgmap v1696: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:12:36.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:12:35 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:12:36.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:12:35 smithi115 ceph-mon[78593]: pgmap v1696: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:12:36.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:12:35 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:12:36.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:12:36 smithi115 conmon[90393]: 2022-05-14T00:12:36.344+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:12:37.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:12:36 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:12:37.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:12:36 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:12:37.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:12:37 smithi115 conmon[90393]: 2022-05-14T00:12:37.347+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:12:38.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:12:37 smithi047 ceph-mon[77147]: pgmap v1697: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:12:38.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:12:37 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:12:38.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:12:37 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:12:38.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:12:37 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:12:38.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:12:37 smithi115 ceph-mon[78593]: pgmap v1697: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:12:38.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:12:37 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:12:38.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:12:37 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:12:38.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:12:37 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:12:38.785 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:12:38 smithi115 conmon[90393]: 2022-05-14T00:12:38.378+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:12:39.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:12:38 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:12:39.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:12:38 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 2690 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:12:39.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:12:38 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:12:39.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:12:38 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 2690 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:12:39.786 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:12:39 smithi115 conmon[90393]: 2022-05-14T00:12:39.384+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:12:40.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:12:39 smithi047 ceph-mon[77147]: pgmap v1698: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:12:40.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:12:39 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:12:40.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:12:39 smithi115 ceph-mon[78593]: pgmap v1698: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:12:40.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:12:39 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:12:40.792 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:12:40 smithi115 conmon[90393]: 2022-05-14T00:12:40.431+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:12:41.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:12:40 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:12:41.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:12:40 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:12:41.793 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:12:41 smithi115 conmon[90393]: 2022-05-14T00:12:41.408+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:12:42.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:12:41 smithi115 ceph-mon[78593]: pgmap v1699: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:12:42.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:12:41 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:12:42.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:12:41 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:12:42.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:12:41 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:12:42.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:12:41 smithi047 ceph-mon[77147]: pgmap v1699: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:12:42.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:12:41 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:12:42.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:12:41 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:12:42.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:12:41 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:12:42.800 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:12:42 smithi115 conmon[90393]: 2022-05-14T00:12:42.411+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:12:43.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:12:42 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:12:43.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:12:42 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:12:43.799 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:12:43 smithi115 conmon[90393]: 2022-05-14T00:12:43.427+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:12:44.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:12:43 smithi115 ceph-mon[78593]: pgmap v1700: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:12:44.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:12:43 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:12:44.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:12:43 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 2695 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:12:44.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:12:43 smithi047 ceph-mon[77147]: pgmap v1700: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:12:44.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:12:43 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:12:44.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:12:43 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 2695 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:12:44.804 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:12:44 smithi115 conmon[90393]: 2022-05-14T00:12:44.438+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:12:45.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:12:44 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:12:45.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:12:44 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:12:45.806 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:12:45 smithi115 conmon[90393]: 2022-05-14T00:12:45.484+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:12:46.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:12:45 smithi115 ceph-mon[78593]: pgmap v1701: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:12:46.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:12:45 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:12:46.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:12:45 smithi047 ceph-mon[77147]: pgmap v1701: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:12:46.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:12:45 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:12:46.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:12:46 smithi115 conmon[90393]: 2022-05-14T00:12:46.516+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:12:47.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:12:47 smithi115 ceph-mon[78593]: pgmap v1702: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:12:47.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:12:47 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:12:47.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:12:47 smithi115 conmon[90393]: 2022-05-14T00:12:47.565+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:12:48.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:12:47 smithi047 ceph-mon[77147]: pgmap v1702: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:12:48.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:12:47 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:12:48.813 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:12:48 smithi115 conmon[90393]: 2022-05-14T00:12:48.562+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:12:49.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:12:48 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:12:49.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:12:48 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 2700 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:12:49.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:12:48 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:12:49.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:12:48 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 2700 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:12:49.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:12:49 smithi115 conmon[90393]: 2022-05-14T00:12:49.602+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:12:49.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:12:49 smithi115 ceph-mon[78593]: pgmap v1703: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:12:49.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:12:49 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:12:49.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:12:49 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:12:49.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:12:49 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:12:50.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:12:49 smithi047 ceph-mon[77147]: pgmap v1703: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:12:50.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:12:49 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:12:50.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:12:49 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:12:50.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:12:49 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:12:51.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:12:50 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:12:51.112 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:12:50 smithi115 conmon[90393]: 2022-05-14T00:12:50.618+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:12:51.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:12:50 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:12:52.096 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:12:51 smithi047 ceph-mon[77147]: pgmap v1704: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:12:52.096 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:12:51 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:12:52.097 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:12:51 smithi115 conmon[90393]: 2022-05-14T00:12:51.615+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:12:52.097 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:12:51 smithi115 ceph-mon[78593]: pgmap v1704: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:12:52.098 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:12:51 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:12:52.824 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:12:52 smithi115 conmon[90393]: 2022-05-14T00:12:52.637+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:12:53.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:12:52 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:12:53.149 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:12:52 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T00:12:53.149 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:12:52 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T00:12:53.149 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:12:52 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T00:12:53.150 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:12:52 smithi047 conmon[78521]: 2022-05-14T00:12:52.655+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:12:53.150 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:12:52 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T00:12:53.151 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:12:52 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:12:53.151 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:12:52 smithi047 conmon[78521]: yield 2022-05-14T00:12:53.151 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:12:52 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:12:53.152 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:12:52 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:12:53.152 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:12:52 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:12:53.152 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:12:52 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T00:12:53.153 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:12:52 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:12:53.153 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:12:52 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T00:12:53.153 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:12:52 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:12:53.154 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:12:52 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T00:12:53.154 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:12:52 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:12:53.154 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:12:52 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T00:12:53.154 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:12:52 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:12:53.155 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:12:52 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T00:12:53.156 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:12:52 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:12:53.156 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:12:52 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:12:53.157 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:12:52 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:12:53.158 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:12:52 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:12:53.158 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:12:52 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T00:12:53.159 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:12:52 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T00:12:53.159 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:12:52 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T00:12:54.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:12:53 smithi115 conmon[90393]: 2022-05-14T00:12:53.622+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:12:54.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:12:53 smithi115 ceph-mon[78593]: pgmap v1705: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:12:54.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:12:53 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:12:54.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:12:53 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:12:54.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T00:12:54.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:12:54.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T00:12:54.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:12:54.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:12:54.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:12:54.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T00:12:54.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:12:54.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T00:12:54.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:12:54.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T00:12:54.113 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:12:54.113 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T00:12:54.113 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:12:54.113 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T00:12:54.114 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:12:54.114 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:12:54.114 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:12:54.115 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:12:53 smithi115 ceph-mon[78593]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T00:12:54.115 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:12:53 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:12:54.115 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:12:53 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:12:54.115 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:12:53 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 2705 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:12:54.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:12:53 smithi047 ceph-mon[77147]: pgmap v1705: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:12:54.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:12:53 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:12:54.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:12:53 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:12:54.291 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T00:12:54.292 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:12:54.292 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T00:12:54.292 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:12:54.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:12:54.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:12:54.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T00:12:54.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:12:54.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T00:12:54.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:12:54.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T00:12:54.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:12:54.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T00:12:54.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:12:54.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T00:12:54.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:12:54.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:12:54.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:12:54.297 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:12:53 smithi047 ceph-mon[77147]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T00:12:54.297 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:12:53 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:12:54.297 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:12:53 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:12:54.297 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:12:53 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 2705 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:12:55.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:12:54 smithi115 conmon[90393]: 2022-05-14T00:12:54.615+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:12:55.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:12:54 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:12:55.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:12:54 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:12:55.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:12:55 smithi115 conmon[90393]: 2022-05-14T00:12:55.583+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:12:55.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:12:55 smithi115 ceph-mon[78593]: pgmap v1706: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:12:55.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:12:55 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:12:56.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:12:55 smithi047 ceph-mon[77147]: pgmap v1706: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:12:56.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:12:55 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:12:56.042 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:12:55 smithi047 conmon[78521]: 2022-05-14T00:12:55.727+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:12:56.042 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:12:55 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T00:12:56.042 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:12:55 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:12:56.042 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:12:55 smithi047 conmon[78521]: yield 2022-05-14T00:12:56.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:12:55 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:12:56.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:12:55 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:12:56.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:12:55 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:12:56.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:12:55 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T00:12:56.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:12:55 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:12:56.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:12:55 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T00:12:56.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:12:55 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:12:56.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:12:55 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T00:12:56.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:12:55 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:12:56.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:12:55 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T00:12:56.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:12:55 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:12:56.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:12:55 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T00:12:56.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:12:55 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:12:56.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:12:55 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:12:56.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:12:55 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:12:56.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:12:55 smithi047 conmon[78521]: 2022-05-14T00:12:55.729+0000 7f533ef76700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T00:12:56.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:12:55 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T00:12:56.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:12:55 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:12:56.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:12:55 smithi047 conmon[78521]: yield 2022-05-14T00:12:56.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:12:55 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:12:56.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:12:55 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:12:56.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:12:55 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:12:56.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:12:55 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T00:12:56.048 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:12:55 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:12:56.048 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:12:55 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T00:12:56.048 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:12:55 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:12:56.048 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:12:55 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T00:12:56.049 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:12:55 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:12:56.049 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:12:55 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T00:12:56.049 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:12:55 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:12:56.049 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:12:55 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T00:12:56.049 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:12:55 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:12:56.050 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:12:55 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:12:56.050 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:12:55 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:12:56.050 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:12:55 smithi047 conmon[78521]: 2022-05-14T00:12:56.050 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:12:55 smithi047 conmon[78521]: During handling of the above exception, another exception occurred: 2022-05-14T00:12:56.050 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:12:55 smithi047 conmon[78521]: 2022-05-14T00:12:56.051 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:12:55 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T00:12:56.051 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:12:55 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T00:12:56.051 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:12:55 smithi047 conmon[78521]: return f(*arg) 2022-05-14T00:12:56.051 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:12:55 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T00:12:56.052 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:12:55 smithi047 conmon[78521]: self._write_client_files(client_files, host) 2022-05-14T00:12:56.052 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:12:55 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T00:12:56.052 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:12:55 smithi047 conmon[78521]: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T00:12:56.052 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:12:55 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T00:12:56.052 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:12:55 smithi047 conmon[78521]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T00:12:56.053 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:12:55 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T00:12:56.053 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:12:55 smithi047 conmon[78521]: return self.event_loop.get_result(coro) 2022-05-14T00:12:56.053 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:12:55 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T00:12:56.053 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:12:55 smithi047 conmon[78521]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T00:12:56.054 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:12:55 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T00:12:56.054 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:12:55 smithi047 conmon[78521]: return self.__get_result() 2022-05-14T00:12:56.054 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:12:55 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T00:12:56.054 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:12:55 smithi047 conmon[78521]: raise self._exception 2022-05-14T00:12:56.054 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:12:55 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T00:12:56.055 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:12:55 smithi047 conmon[78521]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T00:12:56.055 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:12:55 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T00:12:56.055 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:12:55 smithi047 conmon[78521]: conn = await self._remote_connection(host, addr) 2022-05-14T00:12:56.055 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:12:55 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T00:12:56.056 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:12:55 smithi047 conmon[78521]: raise 2022-05-14T00:12:56.056 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:12:55 smithi047 conmon[78521]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T00:12:56.056 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:12:55 smithi047 conmon[78521]: self.gen.throw(type, value, traceback) 2022-05-14T00:12:56.056 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:12:55 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T00:12:56.056 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:12:55 smithi047 conmon[78521]: raise OrchestratorError(msg) 2022-05-14T00:12:56.057 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:12:55 smithi047 conmon[78521]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:12:56.839 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:12:56 smithi115 conmon[90393]: 2022-05-14T00:12:56.541+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:12:57.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:12:56 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:12:57.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:12:56 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:12:57.108 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T00:12:57.108 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:12:57.108 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T00:12:57.109 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:12:57.109 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:12:57.109 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:12:57.109 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T00:12:57.109 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:12:57.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T00:12:57.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:12:57.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T00:12:57.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:12:57.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T00:12:57.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:12:57.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T00:12:57.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:12:57.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:12:57.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:12:57.112 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:12:56 smithi115 ceph-mon[78593]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T00:12:57.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T00:12:57.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:12:57.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T00:12:57.113 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:12:57.113 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:12:57.113 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:12:57.113 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T00:12:57.114 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:12:57.114 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T00:12:57.114 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:12:57.114 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T00:12:57.114 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:12:57.115 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T00:12:57.115 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:12:57.115 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T00:12:57.115 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:12:57.115 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:12:57.116 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:12:57.116 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T00:12:57.116 INFO:journalctl@ceph.mon.c.smithi115.stdout: During handling of the above exception, another exception occurred: 2022-05-14T00:12:57.116 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T00:12:57.116 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T00:12:57.117 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T00:12:57.121 INFO:journalctl@ceph.mon.c.smithi115.stdout: return f(*arg) 2022-05-14T00:12:57.121 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T00:12:57.121 INFO:journalctl@ceph.mon.c.smithi115.stdout: self._write_client_files(client_files, host) 2022-05-14T00:12:57.121 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T00:12:57.122 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T00:12:57.122 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T00:12:57.122 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T00:12:57.122 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T00:12:57.122 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.event_loop.get_result(coro) 2022-05-14T00:12:57.123 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T00:12:57.123 INFO:journalctl@ceph.mon.c.smithi115.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T00:12:57.123 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T00:12:57.123 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.__get_result() 2022-05-14T00:12:57.123 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T00:12:57.124 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise self._exception 2022-05-14T00:12:57.124 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T00:12:57.124 INFO:journalctl@ceph.mon.c.smithi115.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T00:12:57.124 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T00:12:57.125 INFO:journalctl@ceph.mon.c.smithi115.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T00:12:57.125 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T00:12:57.125 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise 2022-05-14T00:12:57.125 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T00:12:57.125 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.gen.throw(type, value, traceback) 2022-05-14T00:12:57.126 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T00:12:57.126 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OrchestratorError(msg) 2022-05-14T00:12:57.126 INFO:journalctl@ceph.mon.c.smithi115.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:12:57.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:12:56 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:12:57.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:12:56 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:12:57.292 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T00:12:57.292 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:12:57.292 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T00:12:57.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:12:57.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:12:57.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:12:57.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T00:12:57.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:12:57.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T00:12:57.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:12:57.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T00:12:57.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:12:57.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T00:12:57.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:12:57.298 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T00:12:57.298 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:12:57.299 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:12:57.299 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:12:57.300 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:12:56 smithi047 ceph-mon[77147]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T00:12:57.300 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T00:12:57.300 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:12:57.301 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T00:12:57.301 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:12:57.302 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:12:57.302 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:12:57.303 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T00:12:57.303 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:12:57.303 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T00:12:57.304 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:12:57.304 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T00:12:57.304 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:12:57.305 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T00:12:57.305 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:12:57.306 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T00:12:57.306 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:12:57.307 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:12:57.307 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:12:57.307 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T00:12:57.307 INFO:journalctl@ceph.mon.b.smithi047.stdout: During handling of the above exception, another exception occurred: 2022-05-14T00:12:57.308 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T00:12:57.308 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T00:12:57.308 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T00:12:57.308 INFO:journalctl@ceph.mon.b.smithi047.stdout: return f(*arg) 2022-05-14T00:12:57.308 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T00:12:57.309 INFO:journalctl@ceph.mon.b.smithi047.stdout: self._write_client_files(client_files, host) 2022-05-14T00:12:57.309 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T00:12:57.309 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T00:12:57.309 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T00:12:57.310 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T00:12:57.310 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T00:12:57.310 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.event_loop.get_result(coro) 2022-05-14T00:12:57.311 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T00:12:57.311 INFO:journalctl@ceph.mon.b.smithi047.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T00:12:57.311 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T00:12:57.311 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.__get_result() 2022-05-14T00:12:57.312 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T00:12:57.312 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise self._exception 2022-05-14T00:12:57.312 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T00:12:57.312 INFO:journalctl@ceph.mon.b.smithi047.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T00:12:57.312 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T00:12:57.313 INFO:journalctl@ceph.mon.b.smithi047.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T00:12:57.313 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T00:12:57.313 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise 2022-05-14T00:12:57.314 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T00:12:57.314 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.gen.throw(type, value, traceback) 2022-05-14T00:12:57.314 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T00:12:57.314 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OrchestratorError(msg) 2022-05-14T00:12:57.314 INFO:journalctl@ceph.mon.b.smithi047.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:12:57.840 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:12:57 smithi115 conmon[90393]: 2022-05-14T00:12:57.577+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:12:58.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:12:57 smithi115 ceph-mon[78593]: pgmap v1707: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:12:58.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:12:57 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:12:58.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:12:57 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:12:58.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:12:57 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:12:58.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:12:57 smithi047 ceph-mon[77147]: pgmap v1707: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:12:58.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:12:57 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:12:58.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:12:57 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:12:58.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:12:57 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:12:58.847 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:12:58 smithi115 conmon[90393]: 2022-05-14T00:12:58.527+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:12:59.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:12:58 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:12:59.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:12:58 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 2710 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:12:59.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:12:58 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:12:59.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:12:58 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 2710 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:12:59.846 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:12:59 smithi115 conmon[90393]: 2022-05-14T00:12:59.480+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:13:00.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:12:59 smithi115 ceph-mon[78593]: pgmap v1708: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:13:00.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:12:59 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:13:00.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:12:59 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:13:00.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:12:59 smithi047 ceph-mon[77147]: pgmap v1708: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:13:00.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:12:59 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:13:00.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:12:59 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:13:00.852 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:13:00 smithi115 conmon[90393]: 2022-05-14T00:13:00.439+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:13:01.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:13:00 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:13:01.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:13:00 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:13:01.853 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:13:01 smithi115 conmon[90393]: 2022-05-14T00:13:01.456+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:13:02.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:13:01 smithi115 ceph-mon[78593]: pgmap v1709: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:13:02.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:13:01 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:13:02.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:13:01 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:13:02.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:13:01 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:13:02.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:13:01 smithi047 ceph-mon[77147]: pgmap v1709: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:13:02.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:13:01 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:13:02.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:13:01 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:13:02.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:13:01 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:13:02.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:13:02 smithi115 conmon[90393]: 2022-05-14T00:13:02.415+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:13:03.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:13:02 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:13:03.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:13:02 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:13:03.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:13:03 smithi115 conmon[90393]: 2022-05-14T00:13:03.392+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:13:04.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:13:03 smithi047 ceph-mon[77147]: pgmap v1710: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:13:04.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:13:03 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:13:04.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:13:03 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 2715 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:13:04.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:13:03 smithi115 ceph-mon[78593]: pgmap v1710: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:13:04.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:13:03 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:13:04.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:13:03 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 2715 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:13:04.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:13:04 smithi115 conmon[90393]: 2022-05-14T00:13:04.405+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:13:05.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:13:04 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:13:05.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:13:04 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:13:05.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:13:05 smithi115 conmon[90393]: 2022-05-14T00:13:05.428+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:13:06.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:13:05 smithi047 ceph-mon[77147]: pgmap v1711: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:13:06.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:13:05 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:13:06.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:13:05 smithi115 ceph-mon[78593]: pgmap v1711: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:13:06.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:13:05 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:13:06.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:13:06 smithi115 conmon[90393]: 2022-05-14T00:13:06.450+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:13:07.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:13:06 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:13:07.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:13:06 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:13:07.861 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:13:07 smithi115 conmon[90393]: 2022-05-14T00:13:07.402+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:13:08.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:13:07 smithi047 ceph-mon[77147]: pgmap v1712: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:13:08.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:13:07 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:13:08.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:13:07 smithi115 ceph-mon[78593]: pgmap v1712: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:13:08.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:13:07 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:13:08.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:13:08 smithi115 conmon[90393]: 2022-05-14T00:13:08.367+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:13:09.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:13:08 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:13:09.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:13:08 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 2720 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:13:09.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:13:08 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:13:09.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:13:08 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 2720 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:13:09.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:13:09 smithi115 conmon[90393]: 2022-05-14T00:13:09.387+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:13:10.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:13:09 smithi047 ceph-mon[77147]: pgmap v1713: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:13:10.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:13:09 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:13:10.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:13:09 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:13:10.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:13:09 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:13:10.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:13:09 smithi115 ceph-mon[78593]: pgmap v1713: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:13:10.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:13:09 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:13:10.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:13:09 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:13:10.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:13:09 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:13:10.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:13:10 smithi115 conmon[90393]: 2022-05-14T00:13:10.383+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:13:11.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:13:10 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:13:11.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:13:10 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:13:11.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:13:11 smithi115 conmon[90393]: 2022-05-14T00:13:11.422+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:13:12.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:13:11 smithi047 ceph-mon[77147]: pgmap v1714: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:13:12.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:13:11 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:13:12.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:13:11 smithi115 ceph-mon[78593]: pgmap v1714: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:13:12.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:13:11 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:13:12.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:13:12 smithi115 conmon[90393]: 2022-05-14T00:13:12.459+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:13:13.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:13:12 smithi047 ceph-mon[77147]: pgmap v1715: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:13:13.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:13:12 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:13:13.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:13:12 smithi115 ceph-mon[78593]: pgmap v1715: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:13:13.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:13:12 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:13:13.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:13:13 smithi115 conmon[90393]: 2022-05-14T00:13:13.442+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:13:14.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:13:13 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:13:14.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:13:13 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:13:14.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:13:13 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T00:13:14.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:13:13 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T00:13:14.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:13:13 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:13:14.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:13:13 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 2725 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:13:14.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:13:13 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:13:14.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:13:13 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:13:14.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:13:13 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T00:13:14.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:13:13 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T00:13:14.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:13:13 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:13:14.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:13:13 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 2725 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:13:14.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:13:14 smithi115 conmon[90393]: 2022-05-14T00:13:14.406+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:13:15.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:13:14 smithi047 ceph-mon[77147]: pgmap v1716: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:13:15.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:13:14 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:13:15.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:13:14 smithi115 ceph-mon[78593]: pgmap v1716: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:13:15.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:13:14 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:13:15.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:13:15 smithi115 conmon[90393]: 2022-05-14T00:13:15.378+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:13:16.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:13:15 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:13:16.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:13:15 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:13:16.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:13:16 smithi115 conmon[90393]: 2022-05-14T00:13:16.421+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:13:17.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:13:16 smithi047 ceph-mon[77147]: pgmap v1717: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:13:17.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:13:16 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:13:17.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:13:16 smithi115 ceph-mon[78593]: pgmap v1717: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:13:17.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:13:16 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:13:17.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:13:17 smithi115 conmon[90393]: 2022-05-14T00:13:17.453+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:13:18.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:13:17 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:13:18.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:13:17 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:13:18.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:13:17 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:13:18.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:13:17 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:13:18.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:13:17 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:13:18.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:13:17 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:13:18.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:13:18 smithi115 conmon[90393]: 2022-05-14T00:13:18.411+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:13:19.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:13:18 smithi047 ceph-mon[77147]: pgmap v1718: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:13:19.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:13:18 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:13:19.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:13:18 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 2730 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:13:19.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:13:18 smithi115 ceph-mon[78593]: pgmap v1718: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:13:19.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:13:18 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:13:19.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:13:18 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 2730 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:13:19.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:13:19 smithi115 conmon[90393]: 2022-05-14T00:13:19.365+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:13:20.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:13:19 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:13:20.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:13:19 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:13:20.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:13:20 smithi115 conmon[90393]: 2022-05-14T00:13:20.372+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:13:21.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:13:20 smithi047 ceph-mon[77147]: pgmap v1719: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:13:21.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:13:20 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:13:21.340 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:13:20 smithi115 ceph-mon[78593]: pgmap v1719: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:13:21.340 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:13:20 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:13:21.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:13:21 smithi115 conmon[90393]: 2022-05-14T00:13:21.339+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:13:22.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:13:21 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:13:22.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:13:21 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:13:22.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:13:21 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:13:22.341 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:13:21 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:13:22.342 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:13:21 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:13:22.342 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:13:21 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:13:22.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:13:22 smithi115 conmon[90393]: 2022-05-14T00:13:22.341+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:13:23.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:13:22 smithi047 ceph-mon[77147]: pgmap v1720: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:13:23.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:13:22 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:13:23.306 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:13:22 smithi115 ceph-mon[78593]: pgmap v1720: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:13:23.307 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:13:22 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:13:23.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:13:23 smithi115 conmon[90393]: 2022-05-14T00:13:23.306+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:13:24.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:13:23 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:13:24.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:13:23 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 2735 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:13:24.354 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:13:23 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:13:24.355 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:13:23 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 2735 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:13:24.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:13:24 smithi115 conmon[90393]: 2022-05-14T00:13:24.353+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:13:25.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:13:24 smithi047 ceph-mon[77147]: pgmap v1721: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:13:25.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:13:24 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:13:25.346 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:13:24 smithi115 ceph-mon[78593]: pgmap v1721: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:13:25.347 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:13:24 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:13:25.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:13:25 smithi115 conmon[90393]: 2022-05-14T00:13:25.346+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:13:26.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:13:25 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:13:26.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:13:25 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:13:26.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:13:26 smithi115 conmon[90393]: 2022-05-14T00:13:26.361+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:13:27.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:13:26 smithi047 ceph-mon[77147]: pgmap v1722: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:13:27.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:13:26 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:13:27.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:13:26 smithi115 ceph-mon[78593]: pgmap v1722: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:13:27.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:13:26 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:13:27.860 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:13:27 smithi115 conmon[90393]: 2022-05-14T00:13:27.393+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:13:28.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:13:27 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:13:28.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:13:27 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:13:28.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:13:28 smithi115 conmon[90393]: 2022-05-14T00:13:28.396+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:13:29.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:13:28 smithi047 ceph-mon[77147]: pgmap v1723: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:13:29.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:13:28 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:13:29.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:13:28 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 2740 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:13:29.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:13:28 smithi115 ceph-mon[78593]: pgmap v1723: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:13:29.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:13:28 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:13:29.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:13:28 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 2740 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:13:29.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:13:29 smithi115 conmon[90393]: 2022-05-14T00:13:29.396+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:13:30.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:13:29 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:13:30.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:13:29 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:13:30.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:13:29 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:13:30.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:13:29 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:13:30.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:13:29 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:13:30.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:13:29 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:13:30.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:13:30 smithi115 conmon[90393]: 2022-05-14T00:13:30.373+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:13:31.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:13:30 smithi047 ceph-mon[77147]: pgmap v1724: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:13:31.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:13:30 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:13:31.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:13:30 smithi115 ceph-mon[78593]: pgmap v1724: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:13:31.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:13:30 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:13:31.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:13:31 smithi115 conmon[90393]: 2022-05-14T00:13:31.375+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:13:32.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:13:31 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:13:32.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:13:31 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:13:32.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:13:32 smithi115 conmon[90393]: 2022-05-14T00:13:32.409+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:13:33.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:13:32 smithi047 ceph-mon[77147]: pgmap v1725: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:13:33.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:13:32 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:13:33.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:13:32 smithi115 ceph-mon[78593]: pgmap v1725: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:13:33.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:13:32 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:13:33.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:13:33 smithi115 conmon[90393]: 2022-05-14T00:13:33.453+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:13:34.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:13:33 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:13:34.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:13:33 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:13:34.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:13:33 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:13:34.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:13:33 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 2745 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:13:34.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:13:33 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:13:34.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:13:33 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:13:34.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:13:33 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:13:34.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:13:33 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 2745 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:13:34.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:13:34 smithi115 conmon[90393]: 2022-05-14T00:13:34.503+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:13:35.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:13:34 smithi047 ceph-mon[77147]: pgmap v1726: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:13:35.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:13:34 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:13:35.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:13:34 smithi115 ceph-mon[78593]: pgmap v1726: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:13:35.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:13:34 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:13:35.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:13:35 smithi115 conmon[90393]: 2022-05-14T00:13:35.459+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:13:36.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:13:35 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:13:36.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:13:35 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:13:36.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:13:36 smithi115 conmon[90393]: 2022-05-14T00:13:36.488+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:13:37.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:13:36 smithi047 ceph-mon[77147]: pgmap v1727: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:13:37.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:13:36 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:13:37.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:13:36 smithi115 ceph-mon[78593]: pgmap v1727: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:13:37.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:13:36 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:13:37.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:13:37 smithi115 conmon[90393]: 2022-05-14T00:13:37.526+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:13:38.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:13:37 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:13:38.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:13:37 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:13:38.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:13:37 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:13:38.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:13:37 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:13:38.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:13:38 smithi115 conmon[90393]: 2022-05-14T00:13:38.560+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:13:39.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:13:38 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:13:39.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:13:38 smithi047 ceph-mon[77147]: pgmap v1728: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:13:39.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:13:38 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 2750 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:13:39.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:13:38 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:13:39.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:13:38 smithi115 ceph-mon[78593]: pgmap v1728: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:13:39.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:13:38 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 2750 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:13:39.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:13:39 smithi115 conmon[90393]: 2022-05-14T00:13:39.521+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:13:40.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:13:39 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:13:40.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:13:39 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:13:40.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:13:39 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:13:40.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:13:39 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:13:40.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:13:40 smithi115 conmon[90393]: 2022-05-14T00:13:40.544+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:13:41.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:13:40 smithi047 ceph-mon[77147]: pgmap v1729: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:13:41.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:13:40 smithi115 ceph-mon[78593]: pgmap v1729: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:13:41.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:13:41 smithi115 conmon[90393]: 2022-05-14T00:13:41.555+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:13:42.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:13:41 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:13:42.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:13:41 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:13:42.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:13:41 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:13:42.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:13:41 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:13:42.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:13:41 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:13:42.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:13:41 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:13:42.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:13:42 smithi115 conmon[90393]: 2022-05-14T00:13:42.537+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:13:43.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:13:42 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:13:43.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:13:42 smithi047 ceph-mon[77147]: pgmap v1730: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:13:43.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:13:42 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:13:43.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:13:42 smithi115 ceph-mon[78593]: pgmap v1730: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:13:43.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:13:43 smithi115 conmon[90393]: 2022-05-14T00:13:43.570+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:13:44.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:13:43 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:13:44.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:13:43 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 2755 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:13:44.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:13:43 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:13:44.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:13:43 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 2755 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:13:44.994 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:13:44 smithi115 conmon[90393]: 2022-05-14T00:13:44.620+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:13:45.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:13:44 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:13:45.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:13:44 smithi047 ceph-mon[77147]: pgmap v1731: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:13:45.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:13:44 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:13:45.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:13:44 smithi115 ceph-mon[78593]: pgmap v1731: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:13:45.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:13:45 smithi115 conmon[90393]: 2022-05-14T00:13:45.575+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:13:46.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:13:45 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:13:46.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:13:45 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:13:46.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:13:46 smithi115 conmon[90393]: 2022-05-14T00:13:46.566+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:13:47.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:13:46 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:13:47.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:13:46 smithi047 ceph-mon[77147]: pgmap v1732: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:13:47.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:13:47 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:13:47.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:13:47 smithi115 ceph-mon[78593]: pgmap v1732: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:13:47.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:13:47 smithi115 conmon[90393]: 2022-05-14T00:13:47.592+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:13:48.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:13:48 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:13:48.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:13:48 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:13:48.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:13:48 smithi115 conmon[90393]: 2022-05-14T00:13:48.557+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:13:49.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:13:49 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:13:49.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:13:49 smithi047 ceph-mon[77147]: pgmap v1733: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:13:49.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:13:49 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 2760 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:13:49.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:13:49 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:13:49.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:13:49 smithi115 ceph-mon[78593]: pgmap v1733: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:13:49.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:13:49 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 2760 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:13:49.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:13:49 smithi115 conmon[90393]: 2022-05-14T00:13:49.546+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:13:50.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:13:50 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:13:50.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:13:50 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:13:50.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:13:50 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:13:50.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:13:50 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:13:50.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:13:50 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:13:50.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:13:50 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:13:50.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:13:50 smithi115 conmon[90393]: 2022-05-14T00:13:50.513+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:13:51.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:13:51 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:13:51.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:13:51 smithi047 ceph-mon[77147]: pgmap v1734: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:13:51.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:13:51 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:13:51.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:13:51 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:13:51.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:13:51 smithi115 ceph-mon[78593]: pgmap v1734: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:13:51.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:13:51 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:13:51.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:13:51 smithi115 conmon[90393]: 2022-05-14T00:13:51.558+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:13:52.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:13:52 smithi115 conmon[90393]: 2022-05-14T00:13:52.574+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:13:53.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:13:53 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:13:53.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:13:53 smithi047 ceph-mon[77147]: pgmap v1735: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:13:53.574 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:13:53 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:13:53.575 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:13:53 smithi115 ceph-mon[78593]: pgmap v1735: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:13:53.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:13:53 smithi115 conmon[90393]: 2022-05-14T00:13:53.573+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:13:54.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:13:54 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:13:54.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:13:54 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:13:54.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:13:54 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:13:54.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:13:54 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 2765 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:13:54.576 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:13:54 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:13:54.576 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:13:54 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:13:54.577 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:13:54 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:13:54.577 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:13:54 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 2765 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:13:54.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:13:54 smithi115 conmon[90393]: 2022-05-14T00:13:54.575+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:13:55.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:13:55 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:13:55.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:13:55 smithi047 ceph-mon[77147]: pgmap v1736: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:13:55.557 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:13:55 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:13:55.557 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:13:55 smithi115 ceph-mon[78593]: pgmap v1736: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:13:55.805 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:13:55 smithi115 conmon[90393]: 2022-05-14T00:13:55.554+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:13:56.520 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:13:56 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:13:56.520 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:13:56 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T00:13:56.521 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:13:56 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T00:13:56.521 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:13:56 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T00:13:56.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:13:56 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:13:56.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:13:56 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T00:13:56.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:13:56 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T00:13:56.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:13:56 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T00:13:56.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:13:56 smithi115 conmon[90393]: 2022-05-14T00:13:56.519+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:13:57.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:13:57 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:13:57.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:13:57 smithi047 ceph-mon[77147]: pgmap v1737: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:13:57.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:13:57 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:13:57.331 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:13:57 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:13:57.332 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:13:57 smithi115 ceph-mon[78593]: pgmap v1737: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:13:57.332 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:13:57 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:13:57.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:13:57 smithi115 conmon[90393]: 2022-05-14T00:13:57.523+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:13:58.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:13:58 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:13:58.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:13:58 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:13:58.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:13:58 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:13:58.573 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:13:58 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:13:58.573 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:13:58 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:13:58.573 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:13:58 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:13:58.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:13:58 smithi115 conmon[90393]: 2022-05-14T00:13:58.567+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:13:58.906 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:13:58 smithi047 conmon[78521]: 2022-05-14T00:13:58.833+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:13:58.907 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:13:58 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T00:13:58.907 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:13:58 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:13:58.907 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:13:58 smithi047 conmon[78521]: yield 2022-05-14T00:13:58.907 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:13:58 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:13:58.908 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:13:58 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:13:58.908 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:13:58 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:13:58.908 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:13:58 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T00:13:58.908 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:13:58 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:13:58.908 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:13:58 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T00:13:58.909 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:13:58 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:13:58.909 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:13:58 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T00:13:58.909 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:13:58 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:13:58.909 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:13:58 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T00:13:58.910 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:13:58 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:13:58.910 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:13:58 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T00:13:58.910 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:13:58 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:13:58.910 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:13:58 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:13:58.910 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:13:58 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:13:59.535 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:13:59 smithi115 ceph-mon[78593]: pgmap v1738: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:13:59.536 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:13:59 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 2770 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:13:59.536 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:13:59 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:13:59.537 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T00:13:59.537 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:13:59.537 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T00:13:59.538 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:13:59.538 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:13:59.539 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:13:59.539 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T00:13:59.539 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:13:59.540 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T00:13:59.540 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:13:59.540 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T00:13:59.540 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:13:59.541 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T00:13:59.541 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:13:59.541 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T00:13:59.542 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:13:59.542 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:13:59.543 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:13:59.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:13:59 smithi115 ceph-mon[78593]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T00:13:59.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:13:59 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' 2022-05-14T00:13:59.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:13:59 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' 2022-05-14T00:13:59.547 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:13:59 smithi047 ceph-mon[77147]: pgmap v1738: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:13:59.548 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:13:59 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 2770 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:13:59.548 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:13:59 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:13:59.549 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T00:13:59.549 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:13:59.549 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T00:13:59.549 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:13:59.550 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:13:59.550 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:13:59.550 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T00:13:59.551 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:13:59.551 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T00:13:59.551 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:13:59.551 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T00:13:59.552 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:13:59.552 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T00:13:59.552 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:13:59.552 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T00:13:59.553 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:13:59.553 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:13:59.553 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:13:59.553 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:13:59 smithi047 ceph-mon[77147]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T00:13:59.554 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:13:59 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' 2022-05-14T00:13:59.554 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:13:59 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' 2022-05-14T00:13:59.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:13:59 smithi115 conmon[90393]: 2022-05-14T00:13:59.533+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:14:00.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:14:00 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:14:00.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:14:00 smithi047 ceph-mon[77147]: Detected new or changed devices on smithi115 2022-05-14T00:14:00.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:14:00 smithi047 ceph-mon[77147]: Detected new or changed devices on smithi047 2022-05-14T00:14:00.563 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:14:00 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:14:00.563 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:14:00 smithi115 ceph-mon[78593]: Detected new or changed devices on smithi115 2022-05-14T00:14:00.563 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:14:00 smithi115 ceph-mon[78593]: Detected new or changed devices on smithi047 2022-05-14T00:14:00.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:14:00 smithi115 conmon[90393]: 2022-05-14T00:14:00.562+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:14:01.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:14:01 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:14:01.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:14:01 smithi047 ceph-mon[77147]: pgmap v1739: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:14:01.544 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:14:01 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:14:01.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:14:01 smithi115 ceph-mon[78593]: pgmap v1739: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:14:01.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:14:01 smithi115 conmon[90393]: 2022-05-14T00:14:01.544+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:14:02.171 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:14:01 smithi047 conmon[78521]: 2022-05-14T00:14:01.904+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:14:02.172 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:14:01 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T00:14:02.172 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:14:01 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:14:02.173 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:14:01 smithi047 conmon[78521]: yield 2022-05-14T00:14:02.173 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:14:01 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:14:02.173 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:14:01 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:14:02.173 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:14:01 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:14:02.174 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:14:01 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T00:14:02.174 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:14:01 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:14:02.176 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:14:01 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T00:14:02.176 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:14:01 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:14:02.177 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:14:01 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T00:14:02.177 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:14:01 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:14:02.177 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:14:01 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T00:14:02.178 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:14:01 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:14:02.178 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:14:01 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T00:14:02.179 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:14:01 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:14:02.179 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:14:01 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:14:02.179 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:14:01 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:14:02.180 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:14:01 smithi047 conmon[78521]: 2022-05-14T00:14:01.905+0000 7f533f777700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T00:14:02.180 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:14:01 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T00:14:02.180 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:14:01 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:14:02.181 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:14:01 smithi047 conmon[78521]: yield 2022-05-14T00:14:02.181 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:14:01 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:14:02.181 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:14:01 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:14:02.182 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:14:01 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:14:02.182 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:14:01 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T00:14:02.182 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:14:01 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:14:02.182 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:14:01 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T00:14:02.182 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:14:01 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:14:02.183 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:14:01 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T00:14:02.183 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:14:01 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:14:02.183 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:14:01 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T00:14:02.183 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:14:01 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:14:02.183 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:14:01 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T00:14:02.184 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:14:01 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:14:02.184 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:14:01 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:14:02.184 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:14:01 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:14:02.184 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:14:01 smithi047 conmon[78521]: 2022-05-14T00:14:02.185 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:14:01 smithi047 conmon[78521]: During handling of the above exception, another exception occurred: 2022-05-14T00:14:02.185 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:14:01 smithi047 conmon[78521]: 2022-05-14T00:14:02.185 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:14:01 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T00:14:02.185 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:14:01 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T00:14:02.186 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:14:01 smithi047 conmon[78521]: return f(*arg) 2022-05-14T00:14:02.186 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:14:01 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T00:14:02.186 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:14:01 smithi047 conmon[78521]: self._write_client_files(client_files, host) 2022-05-14T00:14:02.186 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:14:01 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T00:14:02.187 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:14:01 smithi047 conmon[78521]: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T00:14:02.187 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:14:01 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T00:14:02.187 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:14:01 smithi047 conmon[78521]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T00:14:02.187 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:14:01 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T00:14:02.187 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:14:01 smithi047 conmon[78521]: return self.event_loop.get_result(coro) 2022-05-14T00:14:02.188 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:14:01 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T00:14:02.188 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:14:01 smithi047 conmon[78521]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T00:14:02.188 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:14:01 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T00:14:02.188 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:14:01 smithi047 conmon[78521]: return self.__get_result() 2022-05-14T00:14:02.188 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:14:01 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T00:14:02.189 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:14:01 smithi047 conmon[78521]: raise self._exception 2022-05-14T00:14:02.189 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:14:01 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T00:14:02.189 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:14:01 smithi047 conmon[78521]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T00:14:02.189 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:14:01 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T00:14:02.190 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:14:01 smithi047 conmon[78521]: conn = await self._remote_connection(host, addr) 2022-05-14T00:14:02.190 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:14:01 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T00:14:02.190 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:14:01 smithi047 conmon[78521]: raise 2022-05-14T00:14:02.190 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:14:01 smithi047 conmon[78521]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T00:14:02.191 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:14:01 smithi047 conmon[78521]: self.gen.throw(type, value, traceback) 2022-05-14T00:14:02.191 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:14:01 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T00:14:02.191 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:14:01 smithi047 conmon[78521]: raise OrchestratorError(msg) 2022-05-14T00:14:02.191 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:14:01 smithi047 conmon[78521]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:14:02.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:14:02 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:14:02.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:14:02 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:14:02.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:14:02 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:14:02.552 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:14:02 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:14:02.552 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:14:02 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:14:02.553 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:14:02 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:14:02.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:14:02 smithi115 conmon[90393]: 2022-05-14T00:14:02.551+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:14:03.505 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:14:03 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:14:03.505 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:14:03 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:14:03.506 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T00:14:03.506 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:14:03.506 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T00:14:03.506 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:14:03.507 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:14:03.507 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:14:03.507 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T00:14:03.507 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:14:03.507 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T00:14:03.508 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:14:03.508 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T00:14:03.508 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:14:03.508 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T00:14:03.508 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:14:03.509 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T00:14:03.509 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:14:03.509 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:14:03.509 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:14:03.510 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:14:03 smithi115 ceph-mon[78593]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T00:14:03.510 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T00:14:03.510 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:14:03.510 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T00:14:03.510 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:14:03.511 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:14:03.511 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:14:03.511 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T00:14:03.511 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:14:03.511 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T00:14:03.512 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:14:03.512 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T00:14:03.512 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:14:03.512 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T00:14:03.512 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:14:03.513 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T00:14:03.513 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:14:03.513 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:14:03.513 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:14:03.514 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T00:14:03.514 INFO:journalctl@ceph.mon.c.smithi115.stdout: During handling of the above exception, another exception occurred: 2022-05-14T00:14:03.514 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T00:14:03.514 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T00:14:03.514 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T00:14:03.515 INFO:journalctl@ceph.mon.c.smithi115.stdout: return f(*arg) 2022-05-14T00:14:03.515 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T00:14:03.515 INFO:journalctl@ceph.mon.c.smithi115.stdout: self._write_client_files(client_files, host) 2022-05-14T00:14:03.515 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T00:14:03.515 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T00:14:03.516 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T00:14:03.516 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T00:14:03.516 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T00:14:03.516 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.event_loop.get_result(coro) 2022-05-14T00:14:03.517 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T00:14:03.517 INFO:journalctl@ceph.mon.c.smithi115.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T00:14:03.517 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T00:14:03.517 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.__get_result() 2022-05-14T00:14:03.517 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T00:14:03.518 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise self._exception 2022-05-14T00:14:03.518 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T00:14:03.518 INFO:journalctl@ceph.mon.c.smithi115.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T00:14:03.518 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T00:14:03.518 INFO:journalctl@ceph.mon.c.smithi115.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T00:14:03.519 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T00:14:03.519 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise 2022-05-14T00:14:03.519 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T00:14:03.519 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.gen.throw(type, value, traceback) 2022-05-14T00:14:03.520 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T00:14:03.520 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OrchestratorError(msg) 2022-05-14T00:14:03.520 INFO:journalctl@ceph.mon.c.smithi115.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:14:03.520 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:14:03 smithi115 ceph-mon[78593]: pgmap v1740: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:14:03.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:14:03 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:14:03.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:14:03 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:14:03.541 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T00:14:03.542 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:14:03.542 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T00:14:03.542 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:14:03.542 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:14:03.542 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:14:03.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T00:14:03.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:14:03.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T00:14:03.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:14:03.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T00:14:03.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:14:03.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T00:14:03.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:14:03.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T00:14:03.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:14:03.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:14:03.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:14:03.545 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:14:03 smithi047 ceph-mon[77147]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T00:14:03.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T00:14:03.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:14:03.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T00:14:03.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:14:03.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:14:03.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:14:03.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T00:14:03.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:14:03.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T00:14:03.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:14:03.548 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T00:14:03.548 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:14:03.548 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T00:14:03.548 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:14:03.548 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T00:14:03.549 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:14:03.549 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:14:03.549 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:14:03.549 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T00:14:03.549 INFO:journalctl@ceph.mon.b.smithi047.stdout: During handling of the above exception, another exception occurred: 2022-05-14T00:14:03.550 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T00:14:03.550 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T00:14:03.550 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T00:14:03.550 INFO:journalctl@ceph.mon.b.smithi047.stdout: return f(*arg) 2022-05-14T00:14:03.551 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T00:14:03.551 INFO:journalctl@ceph.mon.b.smithi047.stdout: self._write_client_files(client_files, host) 2022-05-14T00:14:03.551 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T00:14:03.551 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T00:14:03.551 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T00:14:03.552 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T00:14:03.552 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T00:14:03.552 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.event_loop.get_result(coro) 2022-05-14T00:14:03.552 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T00:14:03.553 INFO:journalctl@ceph.mon.b.smithi047.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T00:14:03.553 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T00:14:03.553 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.__get_result() 2022-05-14T00:14:03.553 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T00:14:03.553 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise self._exception 2022-05-14T00:14:03.554 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T00:14:03.554 INFO:journalctl@ceph.mon.b.smithi047.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T00:14:03.554 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T00:14:03.554 INFO:journalctl@ceph.mon.b.smithi047.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T00:14:03.554 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T00:14:03.555 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise 2022-05-14T00:14:03.555 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T00:14:03.555 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.gen.throw(type, value, traceback) 2022-05-14T00:14:03.555 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T00:14:03.556 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OrchestratorError(msg) 2022-05-14T00:14:03.556 INFO:journalctl@ceph.mon.b.smithi047.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:14:03.556 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:14:03 smithi047 ceph-mon[77147]: pgmap v1740: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:14:03.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:14:03 smithi115 conmon[90393]: 2022-05-14T00:14:03.503+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:14:04.483 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:14:04 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:14:04.483 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:14:04 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:14:04.483 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:14:04 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 2775 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:14:04.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:14:04 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:14:04.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:14:04 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:14:04.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:14:04 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 2775 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:14:04.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:14:04 smithi115 conmon[90393]: 2022-05-14T00:14:04.482+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:14:05.518 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:14:05 smithi115 ceph-mon[78593]: pgmap v1741: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:14:05.519 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:14:05 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:14:05.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:14:05 smithi047 ceph-mon[77147]: pgmap v1741: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:14:05.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:14:05 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:14:05.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:14:05 smithi115 conmon[90393]: 2022-05-14T00:14:05.517+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:14:06.490 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:14:06 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:14:06.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:14:06 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:14:06.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:14:06 smithi115 conmon[90393]: 2022-05-14T00:14:06.490+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:14:07.517 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:14:07 smithi115 ceph-mon[78593]: pgmap v1742: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:14:07.517 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:14:07 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:14:07.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:14:07 smithi047 ceph-mon[77147]: pgmap v1742: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:14:07.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:14:07 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:14:07.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:14:07 smithi115 conmon[90393]: 2022-05-14T00:14:07.515+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:14:08.534 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:14:08 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:14:08.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:14:08 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:14:08.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:14:08 smithi115 conmon[90393]: 2022-05-14T00:14:08.533+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:14:09.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:14:09 smithi047 ceph-mon[77147]: pgmap v1743: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:14:09.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:14:09 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 2780 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:14:09.560 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:14:09 smithi115 ceph-mon[78593]: pgmap v1743: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:14:09.561 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:14:09 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 2780 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:14:09.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:14:09 smithi115 conmon[90393]: 2022-05-14T00:14:09.559+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:14:10.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:14:10 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:14:10.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:14:10 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:14:10.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:14:10 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:14:10.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:14:10 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:14:10.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:14:10 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:14:10.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:14:10 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:14:10.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:14:10 smithi115 conmon[90393]: 2022-05-14T00:14:10.541+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:14:11.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:14:11 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:14:11.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:14:11 smithi047 ceph-mon[77147]: pgmap v1744: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:14:11.572 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:14:11 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:14:11.572 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:14:11 smithi115 ceph-mon[78593]: pgmap v1744: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:14:11.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:14:11 smithi115 conmon[90393]: 2022-05-14T00:14:11.571+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:14:12.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:14:12 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:14:12.555 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:14:12 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:14:12.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:14:12 smithi115 conmon[90393]: 2022-05-14T00:14:12.554+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:14:13.529 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:14:13 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:14:13.530 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:14:13 smithi115 ceph-mon[78593]: pgmap v1745: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:14:13.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:14:13 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:14:13.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:14:13 smithi047 ceph-mon[77147]: pgmap v1745: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:14:13.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:14:13 smithi115 conmon[90393]: 2022-05-14T00:14:13.528+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:14:14.496 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:14:14 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:14:14.496 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:14:14 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:14:14.497 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:14:14 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:14:14.497 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:14:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T00:14:14.497 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:14:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T00:14:14.497 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:14:14 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 2785 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:14:14.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:14:14 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:14:14.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:14:14 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:14:14.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:14:14 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:14:14.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:14:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T00:14:14.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:14:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T00:14:14.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:14:14 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 2785 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:14:14.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:14:14 smithi115 conmon[90393]: 2022-05-14T00:14:14.495+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:14:15.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:14:15 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:14:15.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:14:15 smithi047 ceph-mon[77147]: pgmap v1746: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:14:15.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:14:15 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:14:15.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:14:15 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:14:15.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:14:15 smithi115 ceph-mon[78593]: pgmap v1746: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:14:15.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:14:15 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:14:15.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:14:15 smithi115 conmon[90393]: 2022-05-14T00:14:15.541+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:14:16.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:14:16 smithi115 conmon[90393]: 2022-05-14T00:14:16.516+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:14:17.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:14:17 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:14:17.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:14:17 smithi047 ceph-mon[77147]: pgmap v1747: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:14:17.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:14:17 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:14:17.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:14:17 smithi115 conmon[90393]: 2022-05-14T00:14:17.468+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:14:17.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:14:17 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:14:17.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:14:17 smithi115 ceph-mon[78593]: pgmap v1747: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:14:17.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:14:17 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:14:18.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:14:18 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:14:18.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:14:18 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:14:18.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:14:18 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:14:18.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:14:18 smithi115 conmon[90393]: 2022-05-14T00:14:18.445+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:14:18.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:14:18 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:14:18.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:14:18 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:14:18.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:14:18 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:14:19.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:14:19 smithi047 ceph-mon[77147]: pgmap v1748: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:14:19.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:14:19 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:14:19.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:14:19 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 2790 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:14:19.610 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:14:19 smithi115 conmon[90393]: 2022-05-14T00:14:19.425+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:14:19.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:14:19 smithi115 ceph-mon[78593]: pgmap v1748: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:14:19.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:14:19 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:14:19.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:14:19 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 2790 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:14:20.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:14:20 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:14:20.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:14:20 smithi115 conmon[90393]: 2022-05-14T00:14:20.424+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:14:20.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:14:20 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:14:21.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:14:21 smithi047 ceph-mon[77147]: pgmap v1749: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:14:21.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:14:21 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:14:21.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:14:21 smithi115 conmon[90393]: 2022-05-14T00:14:21.463+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:14:21.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:14:21 smithi115 ceph-mon[78593]: pgmap v1749: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:14:21.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:14:21 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:14:22.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:14:22 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:14:22.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:14:22 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:14:22.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:14:22 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:14:22.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:14:22 smithi115 conmon[90393]: 2022-05-14T00:14:22.419+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:14:22.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:14:22 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:14:22.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:14:22 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:14:22.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:14:22 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:14:23.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:14:23 smithi047 ceph-mon[77147]: pgmap v1750: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:14:23.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:14:23 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:14:23.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:14:23 smithi115 conmon[90393]: 2022-05-14T00:14:23.421+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:14:23.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:14:23 smithi115 ceph-mon[78593]: pgmap v1750: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:14:23.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:14:23 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:14:24.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:14:24 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:14:24.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:14:24 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 2795 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:14:24.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:14:24 smithi115 conmon[90393]: 2022-05-14T00:14:24.398+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:14:24.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:14:24 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:14:24.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:14:24 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 2795 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:14:25.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:14:25 smithi047 ceph-mon[77147]: pgmap v1751: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:14:25.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:14:25 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:14:25.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:14:25 smithi115 conmon[90393]: 2022-05-14T00:14:25.408+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:14:25.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:14:25 smithi115 ceph-mon[78593]: pgmap v1751: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:14:25.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:14:25 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:14:26.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:14:26 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:14:26.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:14:26 smithi115 conmon[90393]: 2022-05-14T00:14:26.383+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:14:26.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:14:26 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:14:27.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:14:27 smithi047 ceph-mon[77147]: pgmap v1752: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:14:27.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:14:27 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:14:27.610 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:14:27 smithi115 conmon[90393]: 2022-05-14T00:14:27.398+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:14:27.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:14:27 smithi115 ceph-mon[78593]: pgmap v1752: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:14:27.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:14:27 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:14:28.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:14:28 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:14:28.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:14:28 smithi115 conmon[90393]: 2022-05-14T00:14:28.384+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:14:28.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:14:28 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:14:29.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:14:29 smithi047 ceph-mon[77147]: pgmap v1753: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:14:29.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:14:29 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:14:29.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:14:29 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 2800 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:14:29.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:14:29 smithi115 conmon[90393]: 2022-05-14T00:14:29.342+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:14:29.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:14:29 smithi115 ceph-mon[78593]: pgmap v1753: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:14:29.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:14:29 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:14:29.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:14:29 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 2800 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:14:30.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:14:30 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:14:30.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:14:30 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:14:30.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:14:30 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:14:30.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:14:30 smithi115 conmon[90393]: 2022-05-14T00:14:30.304+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:14:30.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:14:30 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:14:30.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:14:30 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:14:30.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:14:30 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:14:31.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:14:31 smithi047 ceph-mon[77147]: pgmap v1754: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:14:31.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:14:31 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:14:31.612 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:14:31 smithi115 conmon[90393]: 2022-05-14T00:14:31.264+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:14:31.612 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:14:31 smithi115 ceph-mon[78593]: pgmap v1754: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:14:31.613 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:14:31 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:14:32.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:14:32 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:14:32.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:14:32 smithi115 conmon[90393]: 2022-05-14T00:14:32.311+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:14:32.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:14:32 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:14:33.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:14:33 smithi047 ceph-mon[77147]: pgmap v1755: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:14:33.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:14:33 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:14:33.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:14:33 smithi115 conmon[90393]: 2022-05-14T00:14:33.291+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:14:33.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:14:33 smithi115 ceph-mon[78593]: pgmap v1755: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:14:33.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:14:33 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:14:34.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:14:34 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:14:34.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:14:34 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:14:34.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:14:34 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:14:34.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:14:34 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 2805 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:14:34.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:14:34 smithi115 conmon[90393]: 2022-05-14T00:14:34.245+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:14:34.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:14:34 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:14:34.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:14:34 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:14:34.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:14:34 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:14:34.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:14:34 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 2805 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:14:35.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:14:35 smithi115 conmon[90393]: 2022-05-14T00:14:35.255+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:14:35.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:14:35 smithi115 ceph-mon[78593]: pgmap v1756: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:14:35.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:14:35 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:14:35.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:14:35 smithi047 ceph-mon[77147]: pgmap v1756: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:14:35.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:14:35 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:14:36.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:14:36 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:14:36.610 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:14:36 smithi115 conmon[90393]: 2022-05-14T00:14:36.213+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:14:36.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:14:36 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:14:37.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:14:37 smithi115 conmon[90393]: 2022-05-14T00:14:37.201+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:14:37.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:14:37 smithi115 ceph-mon[78593]: pgmap v1757: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:14:37.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:14:37 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:14:37.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:14:37 smithi047 ceph-mon[77147]: pgmap v1757: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:14:37.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:14:37 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:14:38.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:14:38 smithi115 conmon[90393]: 2022-05-14T00:14:38.185+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:14:38.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:14:38 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:14:38.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:14:38 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:14:38.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:14:38 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:14:38.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:14:38 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:14:38.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:14:38 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:14:38.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:14:38 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:14:39.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:14:39 smithi115 conmon[90393]: 2022-05-14T00:14:39.175+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:14:39.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:14:39 smithi115 ceph-mon[78593]: pgmap v1758: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:14:39.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:14:39 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:14:39.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:14:39 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 2810 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:14:39.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:14:39 smithi047 ceph-mon[77147]: pgmap v1758: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:14:39.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:14:39 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:14:39.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:14:39 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 2810 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:14:40.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:14:40 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:14:40.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:14:40 smithi115 conmon[90393]: 2022-05-14T00:14:40.162+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:14:40.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:14:40 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:14:41.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:14:41 smithi115 conmon[90393]: 2022-05-14T00:14:41.158+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:14:41.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:14:41 smithi115 ceph-mon[78593]: pgmap v1759: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:14:41.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:14:41 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:14:41.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:14:41 smithi047 ceph-mon[77147]: pgmap v1759: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:14:41.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:14:41 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:14:42.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:14:42 smithi115 conmon[90393]: 2022-05-14T00:14:42.158+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:14:42.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:14:42 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:14:42.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:14:42 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:14:42.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:14:42 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:14:42.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:14:42 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:14:42.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:14:42 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:14:42.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:14:42 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:14:43.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:14:43 smithi115 conmon[90393]: 2022-05-14T00:14:43.122+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:14:43.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:14:43 smithi115 ceph-mon[78593]: pgmap v1760: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:14:43.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:14:43 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:14:43.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:14:43 smithi047 ceph-mon[77147]: pgmap v1760: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:14:43.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:14:43 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:14:44.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:14:44 smithi115 conmon[90393]: 2022-05-14T00:14:44.158+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:14:44.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:14:44 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:14:44.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:14:44 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 2816 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:14:44.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:14:44 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:14:44.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:14:44 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 2816 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:14:45.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:14:45 smithi115 conmon[90393]: 2022-05-14T00:14:45.201+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:14:45.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:14:45 smithi115 ceph-mon[78593]: pgmap v1761: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:14:45.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:14:45 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:14:45.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:14:45 smithi047 ceph-mon[77147]: pgmap v1761: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:14:45.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:14:45 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:14:46.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:14:46 smithi115 conmon[90393]: 2022-05-14T00:14:46.214+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:14:46.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:14:46 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:14:46.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:14:46 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:14:47.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:14:47 smithi115 conmon[90393]: 2022-05-14T00:14:47.245+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:14:47.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:14:47 smithi115 ceph-mon[78593]: pgmap v1762: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:14:47.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:14:47 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:14:47.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:14:47 smithi047 ceph-mon[77147]: pgmap v1762: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:14:47.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:14:47 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:14:48.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:14:48 smithi115 conmon[90393]: 2022-05-14T00:14:48.272+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:14:48.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:14:48 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:14:48.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:14:48 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:14:49.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:14:49 smithi115 conmon[90393]: 2022-05-14T00:14:49.285+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:14:49.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:14:49 smithi115 ceph-mon[78593]: pgmap v1763: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:14:49.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:14:49 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:14:49.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:14:49 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 2821 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:14:49.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:14:49 smithi047 ceph-mon[77147]: pgmap v1763: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:14:49.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:14:49 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:14:49.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:14:49 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 2821 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:14:50.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:14:50 smithi115 conmon[90393]: 2022-05-14T00:14:50.333+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:14:50.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:14:50 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:14:50.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:14:50 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:14:50.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:14:50 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:14:50.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:14:50 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:14:50.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:14:50 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:14:50.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:14:50 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:14:51.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:14:51 smithi115 conmon[90393]: 2022-05-14T00:14:51.337+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:14:51.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:14:51 smithi115 ceph-mon[78593]: pgmap v1764: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:14:51.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:14:51 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:14:51.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:14:51 smithi047 ceph-mon[77147]: pgmap v1764: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:14:51.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:14:51 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:14:52.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:14:52 smithi115 conmon[90393]: 2022-05-14T00:14:52.349+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:14:52.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:14:52 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:14:52.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:14:52 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:14:53.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:14:53 smithi047 ceph-mon[77147]: pgmap v1765: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:14:53.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:14:53 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:14:53.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:14:53 smithi115 conmon[90393]: 2022-05-14T00:14:53.377+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:14:53.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:14:53 smithi115 ceph-mon[78593]: pgmap v1765: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:14:53.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:14:53 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:14:54.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:14:54 smithi115 conmon[90393]: 2022-05-14T00:14:54.331+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:14:54.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:14:54 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:14:54.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:14:54 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:14:54.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:14:54 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:14:54.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:14:54 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 2825 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:14:54.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:14:54 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:14:54.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:14:54 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:14:54.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:14:54 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:14:54.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:14:54 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 2825 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:14:55.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:14:55 smithi115 conmon[90393]: 2022-05-14T00:14:55.313+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:14:55.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:14:55 smithi115 ceph-mon[78593]: pgmap v1766: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:14:55.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:14:55 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:14:55.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:14:55 smithi047 ceph-mon[77147]: pgmap v1766: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:14:55.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:14:55 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:14:56.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:14:56 smithi115 conmon[90393]: 2022-05-14T00:14:56.295+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:14:56.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:14:56 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:14:56.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:14:56 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:14:57.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:14:57 smithi115 conmon[90393]: 2022-05-14T00:14:57.267+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:14:57.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:14:57 smithi115 ceph-mon[78593]: pgmap v1767: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:14:57.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:14:57 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:14:57.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:14:57 smithi047 ceph-mon[77147]: pgmap v1767: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:14:57.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:14:57 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:14:58.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:14:58 smithi115 conmon[90393]: 2022-05-14T00:14:58.280+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:14:58.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:14:58 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:14:58.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:14:58 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:14:58.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:14:58 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:14:58.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:14:58 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:14:58.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:14:58 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:14:58.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:14:58 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:14:59.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:14:59 smithi115 conmon[90393]: 2022-05-14T00:14:59.326+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:14:59.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:14:59 smithi115 ceph-mon[78593]: pgmap v1768: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:14:59.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:14:59 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:14:59.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:14:59 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 2830 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:14:59.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:14:59 smithi047 ceph-mon[77147]: pgmap v1768: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:14:59.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:14:59 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:14:59.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:14:59 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 2830 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:15:00.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:15:00 smithi115 conmon[90393]: 2022-05-14T00:15:00.295+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:15:00.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:15:00 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:15:00.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:15:00 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:15:01.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:15:01 smithi115 conmon[90393]: 2022-05-14T00:15:01.265+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:15:01.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:15:01 smithi115 ceph-mon[78593]: pgmap v1769: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:15:01.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:15:01 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:15:01.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:15:01 smithi047 ceph-mon[77147]: pgmap v1769: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:15:01.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:15:01 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:15:02.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:15:02 smithi115 conmon[90393]: 2022-05-14T00:15:02.289+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:15:02.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:15:02 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:15:02.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:15:02 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:15:02.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:15:02 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:15:02.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:15:02 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T00:15:02.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:15:02 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T00:15:02.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:15:02 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T00:15:02.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:15:02 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:15:02.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:15:02 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:15:02.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:15:02 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:15:02.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:15:02 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T00:15:02.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:15:02 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T00:15:02.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:15:02 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T00:15:02.793 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:15:02 smithi047 conmon[78521]: 2022-05-14T00:15:02.384+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:15:02.793 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:15:02 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T00:15:02.793 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:15:02 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:15:02.793 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:15:02 smithi047 conmon[78521]: yield 2022-05-14T00:15:02.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:15:02 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:15:02.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:15:02 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:15:02.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:15:02 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:15:02.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:15:02 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T00:15:02.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:15:02 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:15:02.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:15:02 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T00:15:02.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:15:02 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:15:02.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:15:02 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T00:15:02.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:15:02 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:15:02.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:15:02 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T00:15:02.800 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:15:02 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:15:02.800 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:15:02 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T00:15:02.800 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:15:02 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:15:02.801 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:15:02 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:15:02.801 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:15:02 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:15:03.614 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:15:03 smithi115 conmon[90393]: 2022-05-14T00:15:03.268+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:15:03.614 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:15:03 smithi115 ceph-mon[78593]: pgmap v1770: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:15:03.615 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:15:03 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:15:03.615 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:15:03 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:15:03.616 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T00:15:03.616 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:15:03.616 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T00:15:03.616 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:15:03.617 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:15:03.617 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:15:03.617 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T00:15:03.617 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:15:03.618 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T00:15:03.618 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:15:03.618 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T00:15:03.618 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:15:03.618 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T00:15:03.619 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:15:03.619 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T00:15:03.619 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:15:03.619 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:15:03.619 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:15:03.620 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:15:03 smithi115 ceph-mon[78593]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T00:15:03.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:15:03 smithi047 ceph-mon[77147]: pgmap v1770: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:15:03.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:15:03 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:15:03.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:15:03 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:15:03.792 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T00:15:03.792 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:15:03.792 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T00:15:03.793 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:15:03.793 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:15:03.793 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:15:03.793 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T00:15:03.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:15:03.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T00:15:03.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:15:03.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T00:15:03.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:15:03.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T00:15:03.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:15:03.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T00:15:03.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:15:03.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:15:03.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:15:03.796 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:15:03 smithi047 ceph-mon[77147]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T00:15:04.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:15:04 smithi115 conmon[90393]: 2022-05-14T00:15:04.233+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:15:04.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:15:04 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:15:04.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:15:04 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 2835 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:15:04.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:15:04 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:15:04.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:15:04 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 2835 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:15:05.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:15:05 smithi115 conmon[90393]: 2022-05-14T00:15:05.187+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:15:05.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:15:05 smithi115 ceph-mon[78593]: pgmap v1771: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:15:05.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:15:05 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:15:05.796 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:15:05 smithi047 ceph-mon[77147]: pgmap v1771: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:15:05.796 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:15:05 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:15:05.797 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:15:05 smithi047 conmon[78521]: 2022-05-14T00:15:05.455+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:15:05.797 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:15:05 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T00:15:05.797 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:15:05 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:15:05.798 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:15:05 smithi047 conmon[78521]: yield 2022-05-14T00:15:05.798 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:15:05 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:15:05.798 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:15:05 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:15:05.798 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:15:05 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:15:05.799 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:15:05 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T00:15:05.799 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:15:05 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:15:05.799 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:15:05 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T00:15:05.799 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:15:05 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:15:05.799 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:15:05 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T00:15:05.800 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:15:05 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:15:05.800 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:15:05 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T00:15:05.800 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:15:05 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:15:05.800 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:15:05 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T00:15:05.800 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:15:05 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:15:05.801 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:15:05 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:15:05.803 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:15:05 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:15:05.803 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:15:05 smithi047 conmon[78521]: 2022-05-14T00:15:05.456+0000 7f5344f82700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T00:15:05.804 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:15:05 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T00:15:05.804 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:15:05 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:15:05.804 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:15:05 smithi047 conmon[78521]: yield 2022-05-14T00:15:05.804 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:15:05 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:15:05.805 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:15:05 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:15:05.805 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:15:05 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:15:05.805 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:15:05 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T00:15:05.805 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:15:05 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:15:05.805 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:15:05 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T00:15:05.806 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:15:05 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:15:05.806 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:15:05 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T00:15:05.806 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:15:05 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:15:05.806 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:15:05 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T00:15:05.806 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:15:05 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:15:05.807 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:15:05 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T00:15:05.807 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:15:05 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:15:05.807 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:15:05 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:15:05.807 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:15:05 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:15:05.807 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:15:05 smithi047 conmon[78521]: 2022-05-14T00:15:05.808 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:15:05 smithi047 conmon[78521]: During handling of the above exception, another exception occurred: 2022-05-14T00:15:05.808 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:15:05 smithi047 conmon[78521]: 2022-05-14T00:15:05.808 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:15:05 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T00:15:05.808 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:15:05 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T00:15:05.809 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:15:05 smithi047 conmon[78521]: return f(*arg) 2022-05-14T00:15:05.809 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:15:05 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T00:15:05.809 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:15:05 smithi047 conmon[78521]: self._write_client_files(client_files, host) 2022-05-14T00:15:05.809 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:15:05 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T00:15:05.810 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:15:05 smithi047 conmon[78521]: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T00:15:05.810 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:15:05 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T00:15:05.811 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:15:05 smithi047 conmon[78521]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T00:15:05.811 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:15:05 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T00:15:05.811 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:15:05 smithi047 conmon[78521]: return self.event_loop.get_result(coro) 2022-05-14T00:15:05.812 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:15:05 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T00:15:05.812 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:15:05 smithi047 conmon[78521]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T00:15:05.812 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:15:05 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T00:15:05.812 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:15:05 smithi047 conmon[78521]: return self.__get_result() 2022-05-14T00:15:05.812 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:15:05 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T00:15:05.813 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:15:05 smithi047 conmon[78521]: raise self._exception 2022-05-14T00:15:05.815 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:15:05 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T00:15:05.816 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:15:05 smithi047 conmon[78521]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T00:15:05.816 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:15:05 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T00:15:05.816 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:15:05 smithi047 conmon[78521]: conn = await self._remote_connection(host, addr) 2022-05-14T00:15:05.816 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:15:05 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T00:15:05.816 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:15:05 smithi047 conmon[78521]: raise 2022-05-14T00:15:05.817 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:15:05 smithi047 conmon[78521]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T00:15:05.817 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:15:05 smithi047 conmon[78521]: self.gen.throw(type, value, traceback) 2022-05-14T00:15:05.817 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:15:05 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T00:15:05.817 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:15:05 smithi047 conmon[78521]: raise OrchestratorError(msg) 2022-05-14T00:15:05.820 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:15:05 smithi047 conmon[78521]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:15:06.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:15:06 smithi115 conmon[90393]: 2022-05-14T00:15:06.166+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:15:06.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:15:06 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:15:06.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:15:06 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:15:06.609 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T00:15:06.609 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:15:06.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T00:15:06.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:15:06.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:15:06.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:15:06.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T00:15:06.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:15:06.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T00:15:06.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:15:06.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T00:15:06.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:15:06.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T00:15:06.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:15:06.613 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T00:15:06.613 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:15:06.613 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:15:06.613 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:15:06.614 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:15:06 smithi115 ceph-mon[78593]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T00:15:06.614 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T00:15:06.614 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:15:06.615 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T00:15:06.615 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:15:06.615 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:15:06.617 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:15:06.618 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T00:15:06.618 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:15:06.618 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T00:15:06.618 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:15:06.619 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T00:15:06.619 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:15:06.619 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T00:15:06.619 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:15:06.620 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T00:15:06.620 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:15:06.620 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:15:06.620 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:15:06.621 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T00:15:06.622 INFO:journalctl@ceph.mon.c.smithi115.stdout: During handling of the above exception, another exception occurred: 2022-05-14T00:15:06.622 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T00:15:06.622 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T00:15:06.623 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T00:15:06.623 INFO:journalctl@ceph.mon.c.smithi115.stdout: return f(*arg) 2022-05-14T00:15:06.623 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T00:15:06.624 INFO:journalctl@ceph.mon.c.smithi115.stdout: self._write_client_files(client_files, host) 2022-05-14T00:15:06.624 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T00:15:06.624 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T00:15:06.624 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T00:15:06.625 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T00:15:06.625 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T00:15:06.625 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.event_loop.get_result(coro) 2022-05-14T00:15:06.626 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T00:15:06.626 INFO:journalctl@ceph.mon.c.smithi115.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T00:15:06.626 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T00:15:06.627 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.__get_result() 2022-05-14T00:15:06.627 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T00:15:06.627 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise self._exception 2022-05-14T00:15:06.627 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T00:15:06.628 INFO:journalctl@ceph.mon.c.smithi115.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T00:15:06.628 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T00:15:06.628 INFO:journalctl@ceph.mon.c.smithi115.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T00:15:06.628 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T00:15:06.628 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise 2022-05-14T00:15:06.629 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T00:15:06.630 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.gen.throw(type, value, traceback) 2022-05-14T00:15:06.630 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T00:15:06.631 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OrchestratorError(msg) 2022-05-14T00:15:06.631 INFO:journalctl@ceph.mon.c.smithi115.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:15:06.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:15:06 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:15:06.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:15:06 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:15:06.791 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T00:15:06.792 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:15:06.792 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T00:15:06.792 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:15:06.792 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:15:06.792 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:15:06.793 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T00:15:06.793 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:15:06.793 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T00:15:06.793 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:15:06.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T00:15:06.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:15:06.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T00:15:06.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:15:06.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T00:15:06.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:15:06.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:15:06.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:15:06.795 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:15:06 smithi047 ceph-mon[77147]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T00:15:06.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T00:15:06.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:15:06.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T00:15:06.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:15:06.797 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:15:06.797 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:15:06.797 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T00:15:06.797 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:15:06.797 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T00:15:06.798 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:15:06.798 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T00:15:06.798 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:15:06.798 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T00:15:06.799 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:15:06.799 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T00:15:06.799 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:15:06.800 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:15:06.800 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:15:06.800 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T00:15:06.801 INFO:journalctl@ceph.mon.b.smithi047.stdout: During handling of the above exception, another exception occurred: 2022-05-14T00:15:06.801 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T00:15:06.801 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T00:15:06.806 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T00:15:06.806 INFO:journalctl@ceph.mon.b.smithi047.stdout: return f(*arg) 2022-05-14T00:15:06.806 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T00:15:06.806 INFO:journalctl@ceph.mon.b.smithi047.stdout: self._write_client_files(client_files, host) 2022-05-14T00:15:06.807 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T00:15:06.807 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T00:15:06.807 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T00:15:06.807 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T00:15:06.807 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T00:15:06.808 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.event_loop.get_result(coro) 2022-05-14T00:15:06.808 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T00:15:06.808 INFO:journalctl@ceph.mon.b.smithi047.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T00:15:06.808 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T00:15:06.808 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.__get_result() 2022-05-14T00:15:06.809 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T00:15:06.809 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise self._exception 2022-05-14T00:15:06.809 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T00:15:06.809 INFO:journalctl@ceph.mon.b.smithi047.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T00:15:06.810 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T00:15:06.810 INFO:journalctl@ceph.mon.b.smithi047.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T00:15:06.810 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T00:15:06.810 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise 2022-05-14T00:15:06.811 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T00:15:06.811 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.gen.throw(type, value, traceback) 2022-05-14T00:15:06.811 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T00:15:06.811 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OrchestratorError(msg) 2022-05-14T00:15:06.811 INFO:journalctl@ceph.mon.b.smithi047.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:15:07.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:15:07 smithi115 conmon[90393]: 2022-05-14T00:15:07.183+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:15:07.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:15:07 smithi115 ceph-mon[78593]: pgmap v1772: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:15:07.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:15:07 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:15:07.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:15:07 smithi047 ceph-mon[77147]: pgmap v1772: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:15:07.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:15:07 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:15:08.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:15:08 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:15:08.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:15:08 smithi115 conmon[90393]: 2022-05-14T00:15:08.173+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:15:08.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:15:08 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:15:09.610 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:15:09 smithi115 conmon[90393]: 2022-05-14T00:15:09.209+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:15:09.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:15:09 smithi115 ceph-mon[78593]: pgmap v1773: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:15:09.612 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:15:09 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:15:09.612 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:15:09 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 2840 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:15:09.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:15:09 smithi047 ceph-mon[77147]: pgmap v1773: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:15:09.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:15:09 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:15:09.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:15:09 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 2840 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:15:10.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:15:10 smithi115 conmon[90393]: 2022-05-14T00:15:10.202+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:15:10.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:15:10 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:15:10.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:15:10 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:15:10.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:15:10 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:15:10.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:15:10 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:15:10.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:15:10 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:15:10.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:15:10 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:15:11.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:15:11 smithi115 conmon[90393]: 2022-05-14T00:15:11.218+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:15:11.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:15:11 smithi115 ceph-mon[78593]: pgmap v1774: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:15:11.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:15:11 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:15:11.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:15:11 smithi047 ceph-mon[77147]: pgmap v1774: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:15:11.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:15:11 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:15:12.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:15:12 smithi115 conmon[90393]: 2022-05-14T00:15:12.221+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:15:12.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:15:12 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:15:12.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:15:12 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:15:13.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:15:13 smithi115 conmon[90393]: 2022-05-14T00:15:13.212+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:15:13.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:15:13 smithi115 ceph-mon[78593]: pgmap v1775: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:15:13.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:15:13 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:15:13.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:15:13 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T00:15:13.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:15:13 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T00:15:13.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:15:13 smithi047 ceph-mon[77147]: pgmap v1775: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:15:13.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:15:13 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:15:13.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:15:13 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T00:15:13.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:15:13 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T00:15:14.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:15:14 smithi115 conmon[90393]: 2022-05-14T00:15:14.204+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:15:14.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:15:14 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:15:14.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:15:14 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:15:14.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:15:14 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:15:14.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:15:14 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 2845 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:15:14.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:15:14 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:15:14.795 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:15:14 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:15:14.795 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:15:14 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:15:14.800 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:15:14 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 2845 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:15:15.440 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:15:15 smithi115 conmon[90393]: 2022-05-14T00:15:15.181+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:15:15.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:15:15 smithi047 ceph-mon[77147]: pgmap v1776: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:15:15.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:15:15 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:15:15.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:15:15 smithi115 ceph-mon[78593]: pgmap v1776: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:15:15.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:15:15 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:15:16.448 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:15:16 smithi115 conmon[90393]: 2022-05-14T00:15:16.167+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:15:16.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:15:16 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:15:16.862 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:15:16 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:15:17.451 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:15:17 smithi115 conmon[90393]: 2022-05-14T00:15:17.192+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:15:17.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:15:17 smithi047 ceph-mon[77147]: pgmap v1777: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:15:17.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:15:17 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:15:17.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:15:17 smithi115 ceph-mon[78593]: pgmap v1777: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:15:17.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:15:17 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:15:18.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:15:18 smithi115 conmon[90393]: 2022-05-14T00:15:18.206+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:15:18.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:15:18 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:15:18.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:15:18 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:15:18.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:15:18 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:15:18.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:15:18 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:15:18.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:15:18 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:15:18.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:15:18 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:15:19.458 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:15:19 smithi115 conmon[90393]: 2022-05-14T00:15:19.169+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:15:19.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:15:19 smithi047 ceph-mon[77147]: pgmap v1778: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:15:19.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:15:19 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:15:19.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:15:19 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 2850 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:15:19.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:15:19 smithi115 ceph-mon[78593]: pgmap v1778: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:15:19.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:15:19 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:15:19.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:15:19 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 2850 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:15:20.462 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:15:20 smithi115 conmon[90393]: 2022-05-14T00:15:20.166+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:15:20.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:15:20 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:15:20.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:15:20 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:15:21.464 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:15:21 smithi115 conmon[90393]: 2022-05-14T00:15:21.210+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:15:21.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:15:21 smithi047 ceph-mon[77147]: pgmap v1779: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:15:21.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:15:21 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:15:21.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:15:21 smithi115 ceph-mon[78593]: pgmap v1779: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:15:21.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:15:21 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:15:22.466 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:15:22 smithi115 conmon[90393]: 2022-05-14T00:15:22.167+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:15:22.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:15:22 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:15:22.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:15:22 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:15:22.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:15:22 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:15:22.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:15:22 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:15:22.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:15:22 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:15:22.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:15:22 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:15:23.471 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:15:23 smithi115 conmon[90393]: 2022-05-14T00:15:23.131+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:15:23.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:15:23 smithi047 ceph-mon[77147]: pgmap v1780: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:15:23.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:15:23 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:15:23.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:15:23 smithi115 ceph-mon[78593]: pgmap v1780: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:15:23.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:15:23 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:15:24.473 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:15:24 smithi115 conmon[90393]: 2022-05-14T00:15:24.125+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:15:24.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:15:24 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:15:24.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:15:24 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 2856 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:15:24.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:15:24 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:15:24.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:15:24 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 2856 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:15:25.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:15:25 smithi115 conmon[90393]: 2022-05-14T00:15:25.083+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:15:25.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:15:25 smithi047 ceph-mon[77147]: pgmap v1781: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:15:25.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:15:25 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:15:25.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:15:25 smithi115 ceph-mon[78593]: pgmap v1781: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:15:25.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:15:25 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:15:26.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:15:26 smithi115 conmon[90393]: 2022-05-14T00:15:26.063+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:15:26.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:15:26 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:15:26.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:15:26 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:15:27.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:15:27 smithi115 conmon[90393]: 2022-05-14T00:15:27.079+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:15:27.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:15:27 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:15:27.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:15:27 smithi047 ceph-mon[77147]: pgmap v1782: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:15:27.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:15:27 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:15:27.862 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:15:27 smithi115 ceph-mon[78593]: pgmap v1782: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:15:28.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:15:28 smithi115 conmon[90393]: 2022-05-14T00:15:28.092+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:15:28.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:15:28 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:15:28.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:15:28 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:15:29.488 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:15:29 smithi115 conmon[90393]: 2022-05-14T00:15:29.139+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:15:29.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:15:29 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:15:29.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:15:29 smithi047 ceph-mon[77147]: pgmap v1783: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:15:29.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:15:29 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:15:29.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:15:29 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:15:29.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:15:29 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 2861 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:15:29.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:15:29 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:15:29.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:15:29 smithi115 ceph-mon[78593]: pgmap v1783: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:15:29.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:15:29 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:15:29.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:15:29 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:15:29.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:15:29 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 2861 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:15:30.492 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:15:30 smithi115 conmon[90393]: 2022-05-14T00:15:30.114+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:15:30.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:15:30 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:15:30.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:15:30 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:15:31.494 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:15:31 smithi115 conmon[90393]: 2022-05-14T00:15:31.139+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:15:31.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:15:31 smithi047 ceph-mon[77147]: pgmap v1784: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:15:31.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:15:31 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:15:31.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:15:31 smithi115 ceph-mon[78593]: pgmap v1784: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:15:31.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:15:31 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:15:32.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:15:32 smithi115 conmon[90393]: 2022-05-14T00:15:32.099+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:15:32.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:15:32 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:15:32.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:15:32 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:15:33.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:15:33 smithi115 conmon[90393]: 2022-05-14T00:15:33.077+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:15:33.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:15:33 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:15:33.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:15:33 smithi047 ceph-mon[77147]: pgmap v1785: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:15:33.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:15:33 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:15:33.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:15:33 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:15:33.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:15:33 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:15:33.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:15:33 smithi115 ceph-mon[78593]: pgmap v1785: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:15:33.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:15:33 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:15:33.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:15:33 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:15:34.501 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:15:34 smithi115 conmon[90393]: 2022-05-14T00:15:34.108+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:15:34.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:15:34 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:15:34.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:15:34 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 2866 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:15:34.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:15:34 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:15:34.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:15:34 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 2866 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:15:35.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:15:35 smithi115 conmon[90393]: 2022-05-14T00:15:35.095+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:15:35.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:15:35 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:15:35.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:15:35 smithi047 ceph-mon[77147]: pgmap v1786: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:15:35.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:15:35 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:15:35.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:15:35 smithi115 ceph-mon[78593]: pgmap v1786: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:15:36.509 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:15:36 smithi115 conmon[90393]: 2022-05-14T00:15:36.129+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:15:36.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:15:36 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:15:36.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:15:36 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:15:37.510 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:15:37 smithi115 conmon[90393]: 2022-05-14T00:15:37.173+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:15:37.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:15:37 smithi047 ceph-mon[77147]: pgmap v1787: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:15:37.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:15:37 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:15:37.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:15:37 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:15:37.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:15:37 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:15:37.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:15:37 smithi115 ceph-mon[78593]: pgmap v1787: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:15:37.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:15:37 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:15:37.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:15:37 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:15:37.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:15:37 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:15:38.514 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:15:38 smithi115 conmon[90393]: 2022-05-14T00:15:38.150+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:15:38.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:15:38 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:15:38.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:15:38 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:15:39.516 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:15:39 smithi115 conmon[90393]: 2022-05-14T00:15:39.197+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:15:39.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:15:39 smithi047 ceph-mon[77147]: pgmap v1788: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:15:39.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:15:39 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:15:39.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:15:39 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 2871 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:15:39.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:15:39 smithi115 ceph-mon[78593]: pgmap v1788: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:15:39.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:15:39 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:15:39.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:15:39 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 2871 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:15:40.521 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:15:40 smithi115 conmon[90393]: 2022-05-14T00:15:40.158+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:15:40.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:15:40 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:15:40.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:15:40 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:15:41.522 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:15:41 smithi115 conmon[90393]: 2022-05-14T00:15:41.162+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:15:41.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:15:41 smithi047 ceph-mon[77147]: pgmap v1789: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:15:41.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:15:41 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:15:41.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:15:41 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:15:41.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:15:41 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:15:41.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:15:41 smithi115 ceph-mon[78593]: pgmap v1789: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:15:41.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:15:41 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:15:41.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:15:41 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:15:41.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:15:41 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:15:42.529 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:15:42 smithi115 conmon[90393]: 2022-05-14T00:15:42.125+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:15:42.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:15:42 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:15:42.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:15:42 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:15:43.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:15:43 smithi115 conmon[90393]: 2022-05-14T00:15:43.093+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:15:43.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:15:43 smithi047 ceph-mon[77147]: pgmap v1790: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:15:43.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:15:43 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:15:43.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:15:43 smithi115 ceph-mon[78593]: pgmap v1790: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:15:43.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:15:43 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:15:44.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:15:44 smithi115 conmon[90393]: 2022-05-14T00:15:44.084+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:15:44.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:15:44 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:15:44.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:15:44 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 2875 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:15:44.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:15:44 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:15:44.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:15:44 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 2875 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:15:45.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:15:45 smithi115 conmon[90393]: 2022-05-14T00:15:45.054+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:15:45.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:15:45 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:15:45.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:15:45 smithi047 ceph-mon[77147]: pgmap v1791: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:15:45.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:15:45 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:15:45.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:15:45 smithi115 ceph-mon[78593]: pgmap v1791: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:15:46.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:15:46 smithi115 conmon[90393]: 2022-05-14T00:15:46.023+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:15:46.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:15:46 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:15:46.865 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:15:46 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:15:47.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:15:47 smithi115 conmon[90393]: 2022-05-14T00:15:47.029+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:15:47.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:15:47 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:15:47.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:15:47 smithi047 ceph-mon[77147]: pgmap v1792: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:15:47.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:15:47 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:15:47.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:15:47 smithi115 ceph-mon[78593]: pgmap v1792: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:15:48.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:15:48 smithi115 conmon[90393]: 2022-05-14T00:15:48.061+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:15:48.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:15:48 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:15:49.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:15:48 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:15:49.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:15:49 smithi115 conmon[90393]: 2022-05-14T00:15:49.058+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:15:49.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:15:49 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:15:49.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:15:49 smithi115 ceph-mon[78593]: pgmap v1793: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:15:49.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:15:49 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:15:49.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:15:49 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:15:49.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:15:49 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 2880 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:15:50.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:15:49 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:15:50.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:15:49 smithi047 ceph-mon[77147]: pgmap v1793: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:15:50.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:15:49 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:15:50.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:15:49 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:15:50.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:15:49 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 2880 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:15:50.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:15:50 smithi115 conmon[90393]: 2022-05-14T00:15:50.067+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:15:50.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:15:50 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:15:51.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:15:50 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:15:51.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:15:51 smithi115 conmon[90393]: 2022-05-14T00:15:51.032+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:15:51.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:15:51 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:15:51.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:15:51 smithi115 ceph-mon[78593]: pgmap v1794: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:15:51.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:15:51 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:15:52.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:15:51 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:15:52.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:15:51 smithi047 ceph-mon[77147]: pgmap v1794: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:15:52.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:15:51 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:15:52.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:15:52 smithi115 conmon[90393]: 2022-05-14T00:15:52.056+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:15:52.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:15:52 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:15:53.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:15:52 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:15:53.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:15:53 smithi115 conmon[90393]: 2022-05-14T00:15:53.080+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:15:53.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:15:53 smithi115 ceph-mon[78593]: pgmap v1795: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:15:53.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:15:53 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:15:53.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:15:53 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:15:53.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:15:53 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:15:54.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:15:53 smithi047 ceph-mon[77147]: pgmap v1795: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:15:54.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:15:53 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:15:54.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:15:53 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:15:54.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:15:53 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:15:54.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:15:54 smithi115 conmon[90393]: 2022-05-14T00:15:54.069+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:15:54.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:15:54 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 2885 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:15:54.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:15:54 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:15:55.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:15:54 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 2885 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:15:55.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:15:54 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:15:55.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:15:55 smithi115 conmon[90393]: 2022-05-14T00:15:55.082+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:15:56.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:15:55 smithi047 ceph-mon[77147]: pgmap v1796: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:15:56.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:15:55 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:15:56.072 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:15:55 smithi115 ceph-mon[78593]: pgmap v1796: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:15:56.072 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:15:55 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:15:56.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:15:56 smithi115 conmon[90393]: 2022-05-14T00:15:56.071+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:15:57.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:15:56 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:15:57.078 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:15:56 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:15:57.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:15:57 smithi115 conmon[90393]: 2022-05-14T00:15:57.076+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:15:58.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:15:57 smithi047 ceph-mon[77147]: pgmap v1797: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:15:58.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:15:57 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:15:58.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:15:57 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:15:58.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:15:57 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:15:58.071 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:15:57 smithi115 ceph-mon[78593]: pgmap v1797: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:15:58.071 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:15:57 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:15:58.072 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:15:57 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:15:58.072 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:15:57 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:15:58.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:15:58 smithi115 conmon[90393]: 2022-05-14T00:15:58.071+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:15:59.026 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:15:58 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:15:59.026 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:15:58 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 2890 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:15:59.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:15:58 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:15:59.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:15:58 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 2890 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:15:59.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:15:59 smithi115 conmon[90393]: 2022-05-14T00:15:59.025+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:16:00.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:15:59 smithi047 ceph-mon[77147]: pgmap v1798: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:16:00.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:15:59 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:16:00.114 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:15:59 smithi115 ceph-mon[78593]: pgmap v1798: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:16:00.115 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:15:59 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:16:00.116 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:15:59 smithi115 conmon[90393]: 2022-05-14T00:15:59.988+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:16:01.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:16:00 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:16:01.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:16:00 smithi115 conmon[90393]: 2022-05-14T00:16:00.981+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:16:01.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:16:00 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:16:02.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:16:01 smithi047 ceph-mon[77147]: pgmap v1799: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:16:02.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:16:01 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:16:02.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:16:01 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:16:02.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:16:01 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:16:02.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:16:01 smithi115 conmon[90393]: 2022-05-14T00:16:01.975+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:16:02.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:16:01 smithi115 ceph-mon[78593]: pgmap v1799: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:16:02.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:16:01 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:16:02.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:16:01 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:16:02.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:16:01 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:16:03.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:16:02 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:16:03.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:16:02 smithi115 conmon[90393]: 2022-05-14T00:16:02.946+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:16:03.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:16:02 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:16:04.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:16:03 smithi047 ceph-mon[77147]: pgmap v1800: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:16:04.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:16:03 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:16:04.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:16:03 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 2895 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:16:04.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:16:03 smithi115 conmon[90393]: 2022-05-14T00:16:03.949+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:16:04.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:16:03 smithi115 ceph-mon[78593]: pgmap v1800: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:16:04.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:16:03 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:16:04.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:16:03 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 2895 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:16:05.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:16:04 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:16:05.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:16:04 smithi115 conmon[90393]: 2022-05-14T00:16:04.979+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:16:05.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:16:04 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:16:06.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:16:05 smithi047 ceph-mon[77147]: pgmap v1801: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:16:06.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:16:05 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:16:06.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:16:05 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T00:16:06.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:16:05 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T00:16:06.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:16:05 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T00:16:06.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:16:05 smithi115 conmon[90393]: 2022-05-14T00:16:05.963+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:16:06.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:16:05 smithi115 ceph-mon[78593]: pgmap v1801: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:16:06.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:16:05 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:16:06.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:16:05 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T00:16:06.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:16:05 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T00:16:06.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:16:05 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T00:16:07.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:16:06 smithi115 conmon[90393]: 2022-05-14T00:16:06.969+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:16:07.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:16:06 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:16:07.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:16:06 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:16:08.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:16:08 smithi115 conmon[90393]: 2022-05-14T00:16:08.006+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:16:08.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:16:07 smithi115 ceph-mon[78593]: pgmap v1802: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:16:08.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:16:07 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:16:08.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:16:07 smithi047 ceph-mon[77147]: pgmap v1802: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:16:08.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:16:07 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:16:09.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:16:08 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:16:09.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:16:08 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 2900 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:16:09.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:16:08 smithi047 conmon[78521]: 2022-05-14T00:16:08.559+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:16:09.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:16:08 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T00:16:09.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:16:08 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:16:09.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:16:08 smithi047 conmon[78521]: yield 2022-05-14T00:16:09.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:16:08 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:16:09.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:16:08 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:16:09.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:16:08 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:16:09.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:16:08 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T00:16:09.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:16:08 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:16:09.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:16:08 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T00:16:09.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:16:08 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:16:09.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:16:08 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T00:16:09.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:16:08 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:16:09.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:16:08 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T00:16:09.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:16:08 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:16:09.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:16:08 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T00:16:09.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:16:08 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:16:09.048 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:16:08 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:16:09.048 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:16:08 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:16:09.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:16:08 smithi115 conmon[90393]: 2022-05-14T00:16:08.990+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:16:09.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:16:08 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:16:09.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:16:08 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 2900 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:16:10.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:16:09 smithi115 conmon[90393]: 2022-05-14T00:16:09.989+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:16:10.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:16:09 smithi115 ceph-mon[78593]: pgmap v1803: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:16:10.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:16:09 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:16:10.109 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T00:16:10.109 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:16:10.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T00:16:10.117 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:16:10.118 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:16:10.118 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:16:10.118 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T00:16:10.118 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:16:10.118 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T00:16:10.119 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:16:10.119 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T00:16:10.119 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:16:10.119 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T00:16:10.120 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:16:10.120 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T00:16:10.120 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:16:10.120 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:16:10.120 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:16:10.121 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:16:09 smithi115 ceph-mon[78593]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T00:16:10.121 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:16:09 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:16:10.121 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:16:09 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:16:10.121 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:16:09 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:16:10.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:16:09 smithi047 ceph-mon[77147]: pgmap v1803: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:16:10.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:16:09 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:16:10.291 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T00:16:10.291 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:16:10.292 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T00:16:10.292 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:16:10.292 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:16:10.292 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:16:10.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T00:16:10.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:16:10.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T00:16:10.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:16:10.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T00:16:10.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:16:10.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T00:16:10.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:16:10.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T00:16:10.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:16:10.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:16:10.298 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:16:10.298 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:16:09 smithi047 ceph-mon[77147]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T00:16:10.298 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:16:09 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:16:10.298 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:16:09 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:16:10.298 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:16:09 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:16:11.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:16:11 smithi115 conmon[90393]: 2022-05-14T00:16:11.018+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:16:11.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:16:10 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:16:11.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:16:10 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:16:12.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:16:11 smithi047 ceph-mon[77147]: pgmap v1804: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:16:12.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:16:11 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:16:12.042 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:16:11 smithi047 conmon[78521]: 2022-05-14T00:16:11.631+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:16:12.042 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:16:11 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T00:16:12.042 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:16:11 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:16:12.042 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:16:11 smithi047 conmon[78521]: yield 2022-05-14T00:16:12.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:16:11 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:16:12.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:16:11 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:16:12.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:16:11 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:16:12.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:16:11 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T00:16:12.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:16:11 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:16:12.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:16:11 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T00:16:12.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:16:11 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:16:12.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:16:11 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T00:16:12.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:16:11 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:16:12.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:16:11 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T00:16:12.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:16:11 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:16:12.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:16:11 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T00:16:12.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:16:11 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:16:12.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:16:11 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:16:12.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:16:11 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:16:12.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:16:11 smithi047 conmon[78521]: 2022-05-14T00:16:11.633+0000 7f5344781700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T00:16:12.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:16:11 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T00:16:12.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:16:11 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:16:12.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:16:11 smithi047 conmon[78521]: yield 2022-05-14T00:16:12.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:16:11 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:16:12.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:16:11 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:16:12.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:16:11 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:16:12.048 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:16:11 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T00:16:12.048 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:16:11 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:16:12.048 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:16:11 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T00:16:12.050 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:16:11 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:16:12.051 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:16:11 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T00:16:12.051 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:16:11 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:16:12.052 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:16:11 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T00:16:12.052 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:16:11 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:16:12.053 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:16:11 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T00:16:12.053 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:16:11 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:16:12.053 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:16:11 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:16:12.053 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:16:11 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:16:12.054 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:16:11 smithi047 conmon[78521]: 2022-05-14T00:16:12.054 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:16:11 smithi047 conmon[78521]: During handling of the above exception, another exception occurred: 2022-05-14T00:16:12.054 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:16:11 smithi047 conmon[78521]: 2022-05-14T00:16:12.054 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:16:11 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T00:16:12.054 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:16:11 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T00:16:12.055 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:16:11 smithi047 conmon[78521]: return f(*arg) 2022-05-14T00:16:12.055 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:16:11 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T00:16:12.056 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:16:11 smithi047 conmon[78521]: self._write_client_files(client_files, host) 2022-05-14T00:16:12.056 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:16:11 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T00:16:12.056 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:16:11 smithi047 conmon[78521]: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T00:16:12.056 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:16:11 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T00:16:12.056 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:16:11 smithi047 conmon[78521]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T00:16:12.057 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:16:11 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T00:16:12.057 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:16:11 smithi047 conmon[78521]: return self.event_loop.get_result(coro) 2022-05-14T00:16:12.057 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:16:11 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T00:16:12.058 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:16:11 smithi047 conmon[78521]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T00:16:12.058 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:16:11 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T00:16:12.058 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:16:11 smithi047 conmon[78521]: return self.__get_result() 2022-05-14T00:16:12.058 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:16:11 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T00:16:12.058 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:16:11 smithi047 conmon[78521]: raise self._exception 2022-05-14T00:16:12.059 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:16:11 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T00:16:12.059 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:16:11 smithi047 conmon[78521]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T00:16:12.059 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:16:11 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T00:16:12.060 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:16:11 smithi047 conmon[78521]: conn = await self._remote_connection(host, addr) 2022-05-14T00:16:12.060 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:16:11 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T00:16:12.060 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:16:11 smithi047 conmon[78521]: raise 2022-05-14T00:16:12.060 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:16:11 smithi047 conmon[78521]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T00:16:12.060 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:16:11 smithi047 conmon[78521]: self.gen.throw(type, value, traceback) 2022-05-14T00:16:12.061 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:16:11 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T00:16:12.061 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:16:11 smithi047 conmon[78521]: raise OrchestratorError(msg) 2022-05-14T00:16:12.061 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:16:11 smithi047 conmon[78521]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:16:12.063 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:16:11 smithi115 ceph-mon[78593]: pgmap v1804: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:16:12.064 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:16:11 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:16:12.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:16:12 smithi115 conmon[90393]: 2022-05-14T00:16:12.062+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:16:13.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:16:13 smithi115 conmon[90393]: 2022-05-14T00:16:13.021+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:16:13.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:16:12 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:16:13.108 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T00:16:13.108 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:16:13.109 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T00:16:13.109 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:16:13.109 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:16:13.109 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:16:13.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T00:16:13.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:16:13.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T00:16:13.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:16:13.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T00:16:13.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:16:13.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T00:16:13.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:16:13.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T00:16:13.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:16:13.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:16:13.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:16:13.112 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:16:12 smithi115 ceph-mon[78593]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T00:16:13.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T00:16:13.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:16:13.113 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T00:16:13.113 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:16:13.113 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:16:13.113 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:16:13.114 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T00:16:13.114 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:16:13.114 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T00:16:13.114 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:16:13.114 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T00:16:13.115 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:16:13.115 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T00:16:13.115 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:16:13.115 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T00:16:13.115 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:16:13.116 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:16:13.116 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:16:13.116 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T00:16:13.116 INFO:journalctl@ceph.mon.c.smithi115.stdout: During handling of the above exception, another exception occurred: 2022-05-14T00:16:13.117 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T00:16:13.117 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T00:16:13.117 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T00:16:13.117 INFO:journalctl@ceph.mon.c.smithi115.stdout: return f(*arg) 2022-05-14T00:16:13.118 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T00:16:13.118 INFO:journalctl@ceph.mon.c.smithi115.stdout: self._write_client_files(client_files, host) 2022-05-14T00:16:13.118 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T00:16:13.118 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T00:16:13.118 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T00:16:13.119 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T00:16:13.119 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T00:16:13.119 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.event_loop.get_result(coro) 2022-05-14T00:16:13.119 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T00:16:13.119 INFO:journalctl@ceph.mon.c.smithi115.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T00:16:13.120 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T00:16:13.120 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.__get_result() 2022-05-14T00:16:13.120 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T00:16:13.120 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise self._exception 2022-05-14T00:16:13.120 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T00:16:13.121 INFO:journalctl@ceph.mon.c.smithi115.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T00:16:13.121 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T00:16:13.121 INFO:journalctl@ceph.mon.c.smithi115.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T00:16:13.121 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T00:16:13.122 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise 2022-05-14T00:16:13.122 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T00:16:13.122 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.gen.throw(type, value, traceback) 2022-05-14T00:16:13.122 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T00:16:13.122 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OrchestratorError(msg) 2022-05-14T00:16:13.123 INFO:journalctl@ceph.mon.c.smithi115.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:16:13.123 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:16:12 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:16:13.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:16:12 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:16:13.291 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T00:16:13.291 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:16:13.292 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T00:16:13.292 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:16:13.292 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:16:13.292 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:16:13.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T00:16:13.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:16:13.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T00:16:13.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:16:13.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T00:16:13.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:16:13.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T00:16:13.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:16:13.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T00:16:13.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:16:13.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:16:13.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:16:13.295 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:16:12 smithi047 ceph-mon[77147]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T00:16:13.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T00:16:13.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:16:13.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T00:16:13.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:16:13.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:16:13.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:16:13.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T00:16:13.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:16:13.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T00:16:13.298 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:16:13.298 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T00:16:13.298 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:16:13.298 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T00:16:13.299 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:16:13.299 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T00:16:13.299 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:16:13.299 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:16:13.299 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:16:13.300 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T00:16:13.300 INFO:journalctl@ceph.mon.b.smithi047.stdout: During handling of the above exception, another exception occurred: 2022-05-14T00:16:13.300 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T00:16:13.301 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T00:16:13.301 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T00:16:13.301 INFO:journalctl@ceph.mon.b.smithi047.stdout: return f(*arg) 2022-05-14T00:16:13.302 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T00:16:13.302 INFO:journalctl@ceph.mon.b.smithi047.stdout: self._write_client_files(client_files, host) 2022-05-14T00:16:13.302 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T00:16:13.302 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T00:16:13.303 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T00:16:13.303 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T00:16:13.303 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T00:16:13.303 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.event_loop.get_result(coro) 2022-05-14T00:16:13.304 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T00:16:13.304 INFO:journalctl@ceph.mon.b.smithi047.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T00:16:13.304 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T00:16:13.305 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.__get_result() 2022-05-14T00:16:13.305 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T00:16:13.305 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise self._exception 2022-05-14T00:16:13.305 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T00:16:13.306 INFO:journalctl@ceph.mon.b.smithi047.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T00:16:13.306 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T00:16:13.306 INFO:journalctl@ceph.mon.b.smithi047.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T00:16:13.306 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T00:16:13.307 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise 2022-05-14T00:16:13.307 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T00:16:13.307 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.gen.throw(type, value, traceback) 2022-05-14T00:16:13.307 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T00:16:13.307 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OrchestratorError(msg) 2022-05-14T00:16:13.308 INFO:journalctl@ceph.mon.b.smithi047.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:16:13.308 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:16:12 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:16:14.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:16:13 smithi115 conmon[90393]: 2022-05-14T00:16:13.997+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:16:14.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:16:13 smithi115 ceph-mon[78593]: pgmap v1805: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:16:14.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:16:13 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:16:14.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:16:13 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:16:14.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:16:13 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:16:14.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:16:13 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T00:16:14.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:16:13 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T00:16:14.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:16:13 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 2905 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:16:14.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:16:13 smithi047 ceph-mon[77147]: pgmap v1805: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:16:14.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:16:13 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:16:14.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:16:13 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:16:14.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:16:13 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:16:14.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:16:13 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T00:16:14.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:16:13 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T00:16:14.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:16:13 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 2905 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:16:15.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:16:14 smithi115 conmon[90393]: 2022-05-14T00:16:14.970+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:16:15.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:16:14 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:16:15.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:16:14 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:16:16.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:16:15 smithi115 conmon[90393]: 2022-05-14T00:16:15.973+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:16:16.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:16:15 smithi115 ceph-mon[78593]: pgmap v1806: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:16:16.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:16:15 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:16:16.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:16:15 smithi047 ceph-mon[77147]: pgmap v1806: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:16:16.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:16:15 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:16:17.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:16:16 smithi115 conmon[90393]: 2022-05-14T00:16:16.969+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:16:17.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:16:16 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:16:17.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:16:16 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:16:18.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:16:17 smithi115 conmon[90393]: 2022-05-14T00:16:17.988+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:16:18.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:16:17 smithi115 ceph-mon[78593]: pgmap v1807: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:16:18.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:16:17 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:16:18.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:16:17 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:16:18.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:16:17 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:16:18.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:16:17 smithi047 ceph-mon[77147]: pgmap v1807: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:16:18.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:16:17 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:16:18.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:16:17 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:16:18.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:16:17 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:16:19.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:16:18 smithi115 conmon[90393]: 2022-05-14T00:16:18.963+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:16:19.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:16:18 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:16:19.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:16:18 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 2910 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:16:19.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:16:18 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:16:19.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:16:18 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 2910 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:16:20.110 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:16:19 smithi115 conmon[90393]: 2022-05-14T00:16:19.973+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:16:20.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:16:19 smithi115 ceph-mon[78593]: pgmap v1808: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:16:20.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:16:19 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:16:20.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:16:19 smithi047 ceph-mon[77147]: pgmap v1808: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:16:20.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:16:19 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:16:21.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:16:20 smithi115 conmon[90393]: 2022-05-14T00:16:20.964+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:16:21.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:16:20 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:16:21.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:16:20 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:16:22.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:16:21 smithi115 conmon[90393]: 2022-05-14T00:16:21.926+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:16:22.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:16:21 smithi115 ceph-mon[78593]: pgmap v1809: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:16:22.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:16:21 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:16:22.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:16:21 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:16:22.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:16:21 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:16:22.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:16:21 smithi047 ceph-mon[77147]: pgmap v1809: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:16:22.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:16:21 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:16:22.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:16:21 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:16:22.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:16:21 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:16:23.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:16:22 smithi115 conmon[90393]: 2022-05-14T00:16:22.884+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:16:23.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:16:22 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:16:23.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:16:22 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:16:24.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:16:23 smithi115 conmon[90393]: 2022-05-14T00:16:23.874+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:16:24.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:16:23 smithi115 ceph-mon[78593]: pgmap v1810: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:16:24.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:16:23 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:16:24.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:16:23 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 2915 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:16:24.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:16:23 smithi047 ceph-mon[77147]: pgmap v1810: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:16:24.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:16:23 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:16:24.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:16:23 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 2915 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:16:25.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:16:24 smithi115 conmon[90393]: 2022-05-14T00:16:24.837+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:16:25.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:16:24 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:16:25.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:16:24 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:16:26.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:16:25 smithi115 conmon[90393]: 2022-05-14T00:16:25.813+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:16:26.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:16:25 smithi115 ceph-mon[78593]: pgmap v1811: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:16:26.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:16:25 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:16:26.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:16:25 smithi047 ceph-mon[77147]: pgmap v1811: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:16:26.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:16:25 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:16:27.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:16:26 smithi115 conmon[90393]: 2022-05-14T00:16:26.820+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:16:27.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:16:26 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:16:27.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:16:26 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:16:28.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:16:27 smithi115 conmon[90393]: 2022-05-14T00:16:27.819+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:16:28.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:16:27 smithi115 ceph-mon[78593]: pgmap v1812: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:16:28.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:16:27 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:16:28.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:16:27 smithi047 ceph-mon[77147]: pgmap v1812: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:16:28.347 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:16:27 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:16:29.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:16:28 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:16:29.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:16:28 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 2920 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:16:29.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:16:28 smithi115 conmon[90393]: 2022-05-14T00:16:28.867+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:16:29.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:16:28 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:16:29.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:16:28 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 2920 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:16:30.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:16:29 smithi047 ceph-mon[77147]: pgmap v1813: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:16:30.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:16:29 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:16:30.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:16:29 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:16:30.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:16:29 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:16:30.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:16:29 smithi115 conmon[90393]: 2022-05-14T00:16:29.892+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:16:30.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:16:29 smithi115 ceph-mon[78593]: pgmap v1813: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:16:30.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:16:29 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:16:30.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:16:29 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:16:30.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:16:29 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:16:31.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:16:30 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:16:31.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:16:30 smithi115 conmon[90393]: 2022-05-14T00:16:30.906+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:16:31.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:16:30 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:16:32.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:16:31 smithi047 ceph-mon[77147]: pgmap v1814: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:16:32.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:16:31 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:16:32.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:16:31 smithi115 conmon[90393]: 2022-05-14T00:16:31.919+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:16:32.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:16:31 smithi115 ceph-mon[78593]: pgmap v1814: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:16:32.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:16:31 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:16:33.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:16:32 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:16:33.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:16:32 smithi115 conmon[90393]: 2022-05-14T00:16:32.963+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:16:33.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:16:32 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:16:34.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:16:33 smithi047 ceph-mon[77147]: pgmap v1815: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:16:34.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:16:33 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:16:34.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:16:33 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:16:34.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:16:33 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:16:34.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:16:33 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 2925 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:16:34.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:16:33 smithi115 conmon[90393]: 2022-05-14T00:16:33.932+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:16:34.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:16:33 smithi115 ceph-mon[78593]: pgmap v1815: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:16:34.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:16:33 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:16:34.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:16:33 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:16:34.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:16:33 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:16:34.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:16:33 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 2925 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:16:35.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:16:34 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:16:35.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:16:34 smithi115 conmon[90393]: 2022-05-14T00:16:34.981+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:16:35.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:16:34 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:16:36.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:16:35 smithi047 ceph-mon[77147]: pgmap v1816: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:16:36.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:16:35 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:16:36.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:16:35 smithi115 conmon[90393]: 2022-05-14T00:16:35.981+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:16:36.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:16:35 smithi115 ceph-mon[78593]: pgmap v1816: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:16:36.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:16:35 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:16:37.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:16:36 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:16:37.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:16:36 smithi115 conmon[90393]: 2022-05-14T00:16:36.998+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:16:37.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:16:36 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:16:38.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:16:37 smithi047 ceph-mon[77147]: pgmap v1817: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:16:38.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:16:37 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:16:38.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:16:37 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:16:38.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:16:37 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:16:38.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:16:38 smithi115 conmon[90393]: 2022-05-14T00:16:38.005+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:16:38.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:16:37 smithi115 ceph-mon[78593]: pgmap v1817: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:16:38.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:16:37 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:16:38.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:16:37 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:16:38.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:16:37 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:16:39.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:16:38 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:16:39.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:16:38 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 2930 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:16:39.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:16:38 smithi115 conmon[90393]: 2022-05-14T00:16:38.977+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:16:39.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:16:38 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:16:39.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:16:38 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 2930 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:16:40.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:16:39 smithi047 ceph-mon[77147]: pgmap v1818: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:16:40.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:16:39 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:16:40.363 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:16:39 smithi115 conmon[90393]: 2022-05-14T00:16:39.963+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:16:40.363 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:16:39 smithi115 ceph-mon[78593]: pgmap v1818: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:16:40.364 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:16:39 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:16:41.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:16:40 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:16:41.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:16:40 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:16:41.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:16:40 smithi115 conmon[90393]: 2022-05-14T00:16:40.940+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:16:42.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:16:41 smithi047 ceph-mon[77147]: pgmap v1819: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:16:42.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:16:41 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:16:42.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:16:41 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:16:42.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:16:41 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:16:42.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:16:41 smithi115 conmon[90393]: 2022-05-14T00:16:41.970+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:16:42.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:16:41 smithi115 ceph-mon[78593]: pgmap v1819: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:16:42.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:16:41 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:16:42.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:16:41 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:16:42.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:16:41 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:16:43.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:16:42 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:16:43.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:16:43 smithi115 conmon[90393]: 2022-05-14T00:16:43.005+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:16:43.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:16:42 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:16:44.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:16:43 smithi047 ceph-mon[77147]: pgmap v1820: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:16:44.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:16:43 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:16:44.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:16:43 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 2935 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:16:44.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:16:43 smithi115 conmon[90393]: 2022-05-14T00:16:43.965+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:16:44.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:16:43 smithi115 ceph-mon[78593]: pgmap v1820: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:16:44.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:16:43 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:16:44.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:16:43 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 2935 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:16:45.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:16:44 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:16:45.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:16:44 smithi115 conmon[90393]: 2022-05-14T00:16:44.991+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:16:45.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:16:44 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:16:46.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:16:45 smithi047 ceph-mon[77147]: pgmap v1821: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:16:46.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:16:45 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:16:46.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:16:46 smithi115 conmon[90393]: 2022-05-14T00:16:46.016+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:16:46.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:16:45 smithi115 ceph-mon[78593]: pgmap v1821: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:16:46.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:16:45 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:16:47.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:16:46 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:16:47.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:16:47 smithi115 conmon[90393]: 2022-05-14T00:16:47.057+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:16:47.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:16:46 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:16:48.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:16:47 smithi047 ceph-mon[77147]: pgmap v1822: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:16:48.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:16:47 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:16:48.360 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:16:48 smithi115 conmon[90393]: 2022-05-14T00:16:48.065+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:16:48.363 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:16:47 smithi115 ceph-mon[78593]: pgmap v1822: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:16:48.363 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:16:47 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:16:49.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:16:48 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:16:49.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:16:48 smithi047 ceph-mon[77147]: pgmap v1823: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:16:49.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:16:48 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 2940 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:16:49.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:16:49 smithi115 conmon[90393]: 2022-05-14T00:16:49.051+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:16:49.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:16:48 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:16:49.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:16:48 smithi115 ceph-mon[78593]: pgmap v1823: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:16:49.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:16:48 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 2940 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:16:50.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:16:49 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:16:50.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:16:49 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:16:50.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:16:49 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:16:50.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:16:50 smithi115 conmon[90393]: 2022-05-14T00:16:50.099+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:16:50.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:16:49 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:16:50.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:16:49 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:16:50.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:16:49 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:16:51.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:16:50 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:16:51.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:16:50 smithi047 ceph-mon[77147]: pgmap v1824: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:16:51.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:16:51 smithi115 conmon[90393]: 2022-05-14T00:16:51.142+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:16:51.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:16:50 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:16:51.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:16:50 smithi115 ceph-mon[78593]: pgmap v1824: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:16:52.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:16:51 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:16:52.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:16:52 smithi115 conmon[90393]: 2022-05-14T00:16:52.186+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:16:52.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:16:51 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:16:53.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:16:52 smithi047 ceph-mon[77147]: pgmap v1825: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:16:53.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:16:52 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:16:53.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:16:53 smithi115 conmon[90393]: 2022-05-14T00:16:53.180+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:16:53.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:16:52 smithi115 ceph-mon[78593]: pgmap v1825: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:16:53.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:16:52 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:16:54.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:16:53 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:16:54.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:16:53 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:16:54.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:16:53 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:16:54.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:16:53 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 2945 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:16:54.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:16:54 smithi115 conmon[90393]: 2022-05-14T00:16:54.191+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:16:54.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:16:53 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:16:54.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:16:53 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:16:54.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:16:53 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:16:54.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:16:53 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 2945 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:16:55.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:16:54 smithi047 ceph-mon[77147]: pgmap v1826: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:16:55.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:16:54 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:16:55.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:16:55 smithi115 conmon[90393]: 2022-05-14T00:16:55.160+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:16:55.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:16:54 smithi115 ceph-mon[78593]: pgmap v1826: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:16:55.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:16:54 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:16:56.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:16:55 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:16:56.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:16:55 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:16:56.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:16:56 smithi115 conmon[90393]: 2022-05-14T00:16:56.128+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:16:57.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:16:56 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:16:57.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:16:56 smithi047 ceph-mon[77147]: pgmap v1827: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:16:57.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:16:57 smithi115 conmon[90393]: 2022-05-14T00:16:57.124+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:16:57.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:16:56 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:16:57.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:16:56 smithi115 ceph-mon[78593]: pgmap v1827: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:16:58.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:16:57 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:16:58.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:16:57 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:16:58.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:16:57 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:16:58.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:16:58 smithi115 conmon[90393]: 2022-05-14T00:16:58.118+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:16:58.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:16:57 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:16:58.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:16:57 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:16:58.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:16:57 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:16:59.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:16:58 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:16:59.398 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:16:58 smithi047 ceph-mon[77147]: pgmap v1828: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:16:59.399 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:16:58 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 2950 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:16:59.399 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:16:59 smithi115 conmon[90393]: 2022-05-14T00:16:59.099+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:16:59.400 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:16:58 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:16:59.400 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:16:58 smithi115 ceph-mon[78593]: pgmap v1828: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:16:59.400 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:16:58 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 2950 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:17:00.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:16:59 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:17:00.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:16:59 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:17:00.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:17:00 smithi115 conmon[90393]: 2022-05-14T00:17:00.131+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:17:01.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:17:00 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:17:01.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:17:00 smithi047 ceph-mon[77147]: pgmap v1829: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:17:01.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:17:01 smithi115 conmon[90393]: 2022-05-14T00:17:01.173+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:17:01.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:17:00 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:17:01.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:17:00 smithi115 ceph-mon[78593]: pgmap v1829: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:17:02.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:17:01 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:17:02.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:17:01 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:17:02.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:17:01 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:17:02.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:17:02 smithi115 conmon[90393]: 2022-05-14T00:17:02.141+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:17:02.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:17:01 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:17:02.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:17:01 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:17:02.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:17:01 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:17:03.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:17:02 smithi047 ceph-mon[77147]: pgmap v1830: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:17:03.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:17:02 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:17:03.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:17:03 smithi115 conmon[90393]: 2022-05-14T00:17:03.157+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:17:03.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:17:02 smithi115 ceph-mon[78593]: pgmap v1830: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:17:03.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:17:02 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:17:04.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:17:03 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:17:04.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:17:03 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 2956 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:17:04.362 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:17:04 smithi115 conmon[90393]: 2022-05-14T00:17:04.175+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:17:04.363 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:17:03 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:17:04.363 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:17:03 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 2956 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:17:05.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:17:04 smithi047 ceph-mon[77147]: pgmap v1831: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:17:05.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:17:04 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:17:05.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:17:05 smithi115 conmon[90393]: 2022-05-14T00:17:05.220+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:17:05.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:17:04 smithi115 ceph-mon[78593]: pgmap v1831: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:17:05.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:17:04 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:17:06.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:17:05 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:17:06.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:17:06 smithi115 conmon[90393]: 2022-05-14T00:17:06.222+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:17:06.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:17:05 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:17:07.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:17:06 smithi047 ceph-mon[77147]: pgmap v1832: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:17:07.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:17:06 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:17:07.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:17:07 smithi115 conmon[90393]: 2022-05-14T00:17:07.224+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:17:07.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:17:06 smithi115 ceph-mon[78593]: pgmap v1832: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:17:07.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:17:06 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:17:08.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:17:07 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:17:08.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:17:08 smithi115 conmon[90393]: 2022-05-14T00:17:08.239+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:17:08.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:17:07 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:17:09.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:17:08 smithi047 ceph-mon[77147]: pgmap v1833: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:17:09.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:17:08 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:17:09.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:17:08 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 2961 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:17:09.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:17:09 smithi115 conmon[90393]: 2022-05-14T00:17:09.205+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:17:09.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:17:08 smithi115 ceph-mon[78593]: pgmap v1833: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:17:09.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:17:08 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:17:09.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:17:08 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 2961 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:17:10.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:17:09 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:17:10.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:17:09 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:17:10.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:17:09 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:17:10.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:17:10 smithi115 conmon[90393]: 2022-05-14T00:17:10.231+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:17:10.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:17:09 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:17:10.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:17:09 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:17:10.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:17:09 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:17:11.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:17:11 smithi047 ceph-mon[77147]: pgmap v1834: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:17:11.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:17:11 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:17:11.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:17:11 smithi115 conmon[90393]: 2022-05-14T00:17:11.188+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:17:11.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:17:11 smithi115 ceph-mon[78593]: pgmap v1834: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:17:11.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:17:11 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:17:12.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:17:12 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:17:12.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:17:12 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T00:17:12.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:17:12 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T00:17:12.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:17:12 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T00:17:12.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:17:12 smithi115 conmon[90393]: 2022-05-14T00:17:12.187+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:17:12.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:17:12 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:17:12.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:17:12 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T00:17:12.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:17:12 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T00:17:12.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:17:12 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T00:17:13.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:17:13 smithi047 ceph-mon[77147]: pgmap v1835: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:17:13.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:17:13 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:17:13.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:17:13 smithi115 conmon[90393]: 2022-05-14T00:17:13.233+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:17:13.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:17:13 smithi115 ceph-mon[78593]: pgmap v1835: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:17:13.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:17:13 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:17:14.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:17:14 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:17:14.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:17:14 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:17:14.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:17:14 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:17:14.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:17:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T00:17:14.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:17:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T00:17:14.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:17:14 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 2965 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:17:14.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:17:14 smithi115 conmon[90393]: 2022-05-14T00:17:14.192+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:17:14.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:17:14 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:17:14.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:17:14 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:17:14.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:17:14 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:17:14.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:17:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T00:17:14.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:17:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T00:17:14.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:17:14 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 2965 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:17:15.016 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:17:14 smithi047 conmon[78521]: 2022-05-14T00:17:14.736+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:17:15.016 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:17:14 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T00:17:15.017 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:17:14 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:17:15.017 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:17:14 smithi047 conmon[78521]: yield 2022-05-14T00:17:15.018 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:17:14 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:17:15.018 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:17:14 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:17:15.018 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:17:14 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:17:15.019 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:17:14 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T00:17:15.019 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:17:14 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:17:15.020 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:17:14 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T00:17:15.020 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:17:14 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:17:15.020 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:17:14 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T00:17:15.021 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:17:14 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:17:15.022 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:17:14 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T00:17:15.022 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:17:14 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:17:15.022 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:17:14 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T00:17:15.023 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:17:14 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:17:15.023 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:17:14 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:17:15.023 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:17:14 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:17:15.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:17:15 smithi047 ceph-mon[77147]: pgmap v1836: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:17:15.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:17:15 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:17:15.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:17:15 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:17:15.292 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T00:17:15.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:17:15.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T00:17:15.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:17:15.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:17:15.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:17:15.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T00:17:15.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:17:15.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T00:17:15.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:17:15.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T00:17:15.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:17:15.299 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T00:17:15.299 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:17:15.300 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T00:17:15.300 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:17:15.300 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:17:15.300 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:17:15.300 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:17:15 smithi047 ceph-mon[77147]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T00:17:15.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:17:15 smithi115 conmon[90393]: 2022-05-14T00:17:15.206+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:17:15.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:17:15 smithi115 ceph-mon[78593]: pgmap v1836: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:17:15.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:17:15 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:17:15.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:17:15 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:17:15.359 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T00:17:15.359 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:17:15.359 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T00:17:15.359 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:17:15.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:17:15.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:17:15.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T00:17:15.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:17:15.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T00:17:15.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:17:15.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T00:17:15.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:17:15.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T00:17:15.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:17:15.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T00:17:15.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:17:15.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:17:15.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:17:15.364 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:17:15 smithi115 ceph-mon[78593]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T00:17:16.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:17:16 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:17:16.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:17:16 smithi115 conmon[90393]: 2022-05-14T00:17:16.224+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:17:16.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:17:16 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:17:17.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:17:17 smithi047 ceph-mon[77147]: pgmap v1837: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:17:17.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:17:17 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:17:17.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:17:17 smithi115 conmon[90393]: 2022-05-14T00:17:17.256+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:17:17.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:17:17 smithi115 ceph-mon[78593]: pgmap v1837: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:17:17.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:17:17 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:17:18.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:17:18 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:17:18.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:17:18 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:17:18.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:17:18 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:17:18.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:17:18 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:17:18.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T00:17:18.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:17:18.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T00:17:18.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:17:18.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:17:18.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:17:18.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T00:17:18.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:17:18.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T00:17:18.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:17:18.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T00:17:18.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:17:18.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T00:17:18.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:17:18.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T00:17:18.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:17:18.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:17:18.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:17:18.297 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:17:18 smithi047 ceph-mon[77147]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T00:17:18.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T00:17:18.298 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:17:18.298 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T00:17:18.298 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:17:18.298 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:17:18.299 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:17:18.299 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T00:17:18.300 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:17:18.300 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T00:17:18.300 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:17:18.300 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T00:17:18.301 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:17:18.301 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T00:17:18.301 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:17:18.301 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T00:17:18.302 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:17:18.303 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:17:18.303 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:17:18.303 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T00:17:18.303 INFO:journalctl@ceph.mon.b.smithi047.stdout: During handling of the above exception, another exception occurred: 2022-05-14T00:17:18.304 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T00:17:18.304 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T00:17:18.304 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T00:17:18.305 INFO:journalctl@ceph.mon.b.smithi047.stdout: return f(*arg) 2022-05-14T00:17:18.305 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T00:17:18.305 INFO:journalctl@ceph.mon.b.smithi047.stdout: self._write_client_files(client_files, host) 2022-05-14T00:17:18.305 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T00:17:18.306 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T00:17:18.306 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T00:17:18.306 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T00:17:18.306 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T00:17:18.307 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.event_loop.get_result(coro) 2022-05-14T00:17:18.307 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T00:17:18.307 INFO:journalctl@ceph.mon.b.smithi047.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T00:17:18.307 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T00:17:18.308 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.__get_result() 2022-05-14T00:17:18.308 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T00:17:18.309 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise self._exception 2022-05-14T00:17:18.309 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T00:17:18.310 INFO:journalctl@ceph.mon.b.smithi047.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T00:17:18.310 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T00:17:18.310 INFO:journalctl@ceph.mon.b.smithi047.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T00:17:18.311 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T00:17:18.311 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise 2022-05-14T00:17:18.311 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T00:17:18.311 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.gen.throw(type, value, traceback) 2022-05-14T00:17:18.312 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T00:17:18.312 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OrchestratorError(msg) 2022-05-14T00:17:18.312 INFO:journalctl@ceph.mon.b.smithi047.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:17:18.313 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:17:17 smithi047 conmon[78521]: 2022-05-14T00:17:17.808+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:17:18.313 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:17:17 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T00:17:18.313 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:17:17 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:17:18.314 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:17:17 smithi047 conmon[78521]: yield 2022-05-14T00:17:18.314 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:17:17 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:17:18.314 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:17:17 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:17:18.314 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:17:17 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:17:18.314 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:17:17 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T00:17:18.315 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:17:17 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:17:18.315 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:17:17 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T00:17:18.315 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:17:17 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:17:18.315 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:17:17 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T00:17:18.316 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:17:17 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:17:18.316 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:17:17 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T00:17:18.316 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:17:17 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:17:18.316 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:17:17 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T00:17:18.317 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:17:17 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:17:18.317 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:17:17 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:17:18.317 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:17:17 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:17:18.318 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:17:17 smithi047 conmon[78521]: 2022-05-14T00:17:17.809+0000 7f533ef76700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T00:17:18.318 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:17:17 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T00:17:18.318 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:17:17 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:17:18.318 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:17:17 smithi047 conmon[78521]: yield 2022-05-14T00:17:18.318 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:17:17 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:17:18.319 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:17:17 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:17:18.319 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:17:17 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:17:18.319 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:17:17 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T00:17:18.319 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:17:17 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:17:18.320 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:17:17 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T00:17:18.320 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:17:17 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:17:18.321 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:17:17 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T00:17:18.321 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:17:17 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:17:18.321 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:17:17 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T00:17:18.321 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:17:17 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:17:18.321 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:17:17 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T00:17:18.322 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:17:17 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:17:18.322 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:17:17 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:17:18.323 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:17:17 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:17:18.323 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:17:17 smithi047 conmon[78521]: 2022-05-14T00:17:18.323 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:17:17 smithi047 conmon[78521]: During handling of the above exception, another exception occurred: 2022-05-14T00:17:18.323 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:17:17 smithi047 conmon[78521]: 2022-05-14T00:17:18.324 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:17:17 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T00:17:18.324 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:17:17 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T00:17:18.324 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:17:17 smithi047 conmon[78521]: return f(*arg) 2022-05-14T00:17:18.324 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:17:17 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T00:17:18.325 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:17:17 smithi047 conmon[78521]: self._write_client_files(client_files, host) 2022-05-14T00:17:18.325 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:17:17 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T00:17:18.325 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:17:17 smithi047 conmon[78521]: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T00:17:18.325 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:17:17 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T00:17:18.326 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:17:17 smithi047 conmon[78521]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T00:17:18.326 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:17:17 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T00:17:18.326 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:17:17 smithi047 conmon[78521]: return self.event_loop.get_result(coro) 2022-05-14T00:17:18.326 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:17:17 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T00:17:18.327 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:17:17 smithi047 conmon[78521]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T00:17:18.327 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:17:17 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T00:17:18.327 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:17:17 smithi047 conmon[78521]: return self.__get_result() 2022-05-14T00:17:18.328 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:17:17 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T00:17:18.328 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:17:17 smithi047 conmon[78521]: raise self._exception 2022-05-14T00:17:18.328 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:17:17 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T00:17:18.328 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:17:17 smithi047 conmon[78521]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T00:17:18.329 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:17:17 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T00:17:18.329 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:17:17 smithi047 conmon[78521]: conn = await self._remote_connection(host, addr) 2022-05-14T00:17:18.329 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:17:17 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T00:17:18.329 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:17:17 smithi047 conmon[78521]: raise 2022-05-14T00:17:18.329 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:17:17 smithi047 conmon[78521]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T00:17:18.330 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:17:17 smithi047 conmon[78521]: self.gen.throw(type, value, traceback) 2022-05-14T00:17:18.330 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:17:17 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T00:17:18.330 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:17:17 smithi047 conmon[78521]: raise OrchestratorError(msg) 2022-05-14T00:17:18.330 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:17:17 smithi047 conmon[78521]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:17:18.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:17:18 smithi115 conmon[90393]: 2022-05-14T00:17:18.233+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:17:18.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:17:18 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:17:18.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:17:18 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:17:18.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:17:18 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:17:18.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:17:18 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:17:18.359 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T00:17:18.359 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:17:18.359 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T00:17:18.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:17:18.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:17:18.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:17:18.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T00:17:18.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:17:18.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T00:17:18.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:17:18.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T00:17:18.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:17:18.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T00:17:18.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:17:18.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T00:17:18.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:17:18.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:17:18.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:17:18.363 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:17:18 smithi115 ceph-mon[78593]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T00:17:18.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T00:17:18.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:17:18.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T00:17:18.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:17:18.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:17:18.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:17:18.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T00:17:18.365 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:17:18.365 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T00:17:18.365 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:17:18.365 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T00:17:18.365 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:17:18.366 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T00:17:18.366 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:17:18.366 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T00:17:18.366 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:17:18.367 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:17:18.367 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:17:18.367 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T00:17:18.367 INFO:journalctl@ceph.mon.c.smithi115.stdout: During handling of the above exception, another exception occurred: 2022-05-14T00:17:18.367 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T00:17:18.368 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T00:17:18.368 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T00:17:18.368 INFO:journalctl@ceph.mon.c.smithi115.stdout: return f(*arg) 2022-05-14T00:17:18.368 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T00:17:18.369 INFO:journalctl@ceph.mon.c.smithi115.stdout: self._write_client_files(client_files, host) 2022-05-14T00:17:18.369 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T00:17:18.369 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T00:17:18.369 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T00:17:18.370 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T00:17:18.370 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T00:17:18.370 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.event_loop.get_result(coro) 2022-05-14T00:17:18.370 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T00:17:18.370 INFO:journalctl@ceph.mon.c.smithi115.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T00:17:18.371 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T00:17:18.371 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.__get_result() 2022-05-14T00:17:18.371 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T00:17:18.371 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise self._exception 2022-05-14T00:17:18.372 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T00:17:18.372 INFO:journalctl@ceph.mon.c.smithi115.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T00:17:18.372 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T00:17:18.372 INFO:journalctl@ceph.mon.c.smithi115.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T00:17:18.372 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T00:17:18.373 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise 2022-05-14T00:17:18.373 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T00:17:18.373 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.gen.throw(type, value, traceback) 2022-05-14T00:17:18.374 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T00:17:18.374 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OrchestratorError(msg) 2022-05-14T00:17:18.374 INFO:journalctl@ceph.mon.c.smithi115.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:17:19.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:17:19 smithi047 ceph-mon[77147]: pgmap v1838: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:17:19.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:17:19 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:17:19.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:17:19 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 2970 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:17:19.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:17:19 smithi115 conmon[90393]: 2022-05-14T00:17:19.220+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:17:19.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:17:19 smithi115 ceph-mon[78593]: pgmap v1838: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:17:19.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:17:19 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:17:19.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:17:19 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 2970 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:17:20.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:17:20 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:17:20.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:17:20 smithi115 conmon[90393]: 2022-05-14T00:17:20.268+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:17:20.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:17:20 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:17:21.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:17:21 smithi047 ceph-mon[77147]: pgmap v1839: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:17:21.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:17:21 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:17:21.313 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:17:21 smithi115 ceph-mon[78593]: pgmap v1839: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:17:21.314 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:17:21 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:17:21.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:17:21 smithi115 conmon[90393]: 2022-05-14T00:17:21.312+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:17:22.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:17:22 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:17:22.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:17:22 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:17:22.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:17:22 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:17:22.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:17:22 smithi115 conmon[90393]: 2022-05-14T00:17:22.268+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:17:22.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:17:22 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:17:22.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:17:22 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:17:22.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:17:22 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:17:23.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:17:23 smithi115 conmon[90393]: 2022-05-14T00:17:23.276+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:17:23.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:17:23 smithi115 ceph-mon[78593]: pgmap v1840: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:17:23.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:17:23 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:17:23.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:17:23 smithi047 ceph-mon[77147]: pgmap v1840: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:17:23.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:17:23 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:17:24.314 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:17:24 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:17:24.314 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:17:24 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 2975 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:17:24.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:17:24 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:17:24.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:17:24 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 2975 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:17:24.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:17:24 smithi115 conmon[90393]: 2022-05-14T00:17:24.313+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:17:25.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:17:25 smithi115 ceph-mon[78593]: pgmap v1841: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:17:25.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:17:25 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:17:25.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:17:25 smithi047 ceph-mon[77147]: pgmap v1841: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:17:25.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:17:25 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:17:25.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:17:25 smithi115 conmon[90393]: 2022-05-14T00:17:25.357+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:17:26.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:17:26 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:17:26.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:17:26 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:17:26.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:17:26 smithi115 conmon[90393]: 2022-05-14T00:17:26.397+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:17:27.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:17:27 smithi115 ceph-mon[78593]: pgmap v1842: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:17:27.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:17:27 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:17:27.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:17:27 smithi047 ceph-mon[77147]: pgmap v1842: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:17:27.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:17:27 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:17:27.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:17:27 smithi115 conmon[90393]: 2022-05-14T00:17:27.419+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:17:28.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:17:28 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:17:28.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:17:28 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:17:28.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:17:28 smithi115 conmon[90393]: 2022-05-14T00:17:28.438+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:17:29.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:17:29 smithi115 ceph-mon[78593]: pgmap v1843: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:17:29.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:17:29 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:17:29.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:17:29 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 2980 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:17:29.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:17:29 smithi047 ceph-mon[77147]: pgmap v1843: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:17:29.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:17:29 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:17:29.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:17:29 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 2980 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:17:29.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:17:29 smithi115 conmon[90393]: 2022-05-14T00:17:29.468+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:17:30.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:17:30 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:17:30.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:17:30 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:17:30.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:17:30 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:17:30.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:17:30 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:17:30.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:17:30 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:17:30.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:17:30 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:17:30.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:17:30 smithi115 conmon[90393]: 2022-05-14T00:17:30.434+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:17:31.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:17:31 smithi115 ceph-mon[78593]: pgmap v1844: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:17:31.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:17:31 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:17:31.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:17:31 smithi047 ceph-mon[77147]: pgmap v1844: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:17:31.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:17:31 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:17:31.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:17:31 smithi115 conmon[90393]: 2022-05-14T00:17:31.409+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:17:32.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:17:32 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:17:32.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:17:32 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:17:32.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:17:32 smithi115 conmon[90393]: 2022-05-14T00:17:32.402+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:17:33.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:17:33 smithi115 ceph-mon[78593]: pgmap v1845: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:17:33.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:17:33 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:17:33.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:17:33 smithi047 ceph-mon[77147]: pgmap v1845: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:17:33.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:17:33 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:17:33.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:17:33 smithi115 conmon[90393]: 2022-05-14T00:17:33.394+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:17:34.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:17:34 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:17:34.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:17:34 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:17:34.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:17:34 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:17:34.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:17:34 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 2985 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:17:34.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:17:34 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:17:34.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:17:34 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:17:34.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:17:34 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:17:34.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:17:34 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 2985 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:17:34.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:17:34 smithi115 conmon[90393]: 2022-05-14T00:17:34.371+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:17:35.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:17:35 smithi115 ceph-mon[78593]: pgmap v1846: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:17:35.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:17:35 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:17:35.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:17:35 smithi047 ceph-mon[77147]: pgmap v1846: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:17:35.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:17:35 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:17:35.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:17:35 smithi115 conmon[90393]: 2022-05-14T00:17:35.378+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:17:36.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:17:36 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:17:36.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:17:36 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:17:36.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:17:36 smithi115 conmon[90393]: 2022-05-14T00:17:36.402+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:17:37.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:17:37 smithi115 ceph-mon[78593]: pgmap v1847: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:17:37.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:17:37 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:17:37.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:17:37 smithi047 ceph-mon[77147]: pgmap v1847: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:17:37.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:17:37 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:17:37.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:17:37 smithi115 conmon[90393]: 2022-05-14T00:17:37.405+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:17:38.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:17:38 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:17:38.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:17:38 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:17:38.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:17:38 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:17:38.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:17:38 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:17:38.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:17:38 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:17:38.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:17:38 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:17:38.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:17:38 smithi115 conmon[90393]: 2022-05-14T00:17:38.376+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:17:39.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:17:39 smithi115 ceph-mon[78593]: pgmap v1848: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:17:39.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:17:39 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:17:39.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:17:39 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 2990 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:17:39.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:17:39 smithi047 ceph-mon[77147]: pgmap v1848: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:17:39.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:17:39 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:17:39.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:17:39 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 2990 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:17:39.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:17:39 smithi115 conmon[90393]: 2022-05-14T00:17:39.398+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:17:40.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:17:40 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:17:40.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:17:40 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:17:40.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:17:40 smithi115 conmon[90393]: 2022-05-14T00:17:40.437+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:17:41.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:17:41 smithi115 ceph-mon[78593]: pgmap v1849: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:17:41.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:17:41 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:17:41.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:17:41 smithi047 ceph-mon[77147]: pgmap v1849: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:17:41.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:17:41 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:17:41.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:17:41 smithi115 conmon[90393]: 2022-05-14T00:17:41.428+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:17:42.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:17:42 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:17:42.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:17:42 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:17:42.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:17:42 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:17:42.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:17:42 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:17:42.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:17:42 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:17:42.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:17:42 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:17:42.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:17:42 smithi115 conmon[90393]: 2022-05-14T00:17:42.461+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:17:43.450 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:17:43 smithi115 ceph-mon[78593]: pgmap v1850: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:17:43.450 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:17:43 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:17:43.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:17:43 smithi047 ceph-mon[77147]: pgmap v1850: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:17:43.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:17:43 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:17:43.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:17:43 smithi115 conmon[90393]: 2022-05-14T00:17:43.450+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:17:44.404 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:17:44 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:17:44.404 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:17:44 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 2995 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:17:44.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:17:44 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:17:44.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:17:44 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 2995 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:17:44.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:17:44 smithi115 conmon[90393]: 2022-05-14T00:17:44.403+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:17:45.387 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:17:45 smithi115 ceph-mon[78593]: pgmap v1851: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:17:45.388 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:17:45 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:17:45.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:17:45 smithi047 ceph-mon[77147]: pgmap v1851: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:17:45.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:17:45 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:17:45.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:17:45 smithi115 conmon[90393]: 2022-05-14T00:17:45.386+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:17:46.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:17:46 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:17:46.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:17:46 smithi115 conmon[90393]: 2022-05-14T00:17:46.337+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:17:46.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:17:46 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:17:47.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:17:47 smithi047 ceph-mon[77147]: pgmap v1852: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:17:47.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:17:47 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:17:47.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:17:47 smithi115 conmon[90393]: 2022-05-14T00:17:47.323+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:17:47.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:17:47 smithi115 ceph-mon[78593]: pgmap v1852: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:17:47.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:17:47 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:17:48.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:17:48 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:17:48.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:17:48 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:17:48.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:17:48 smithi115 conmon[90393]: 2022-05-14T00:17:48.300+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:17:49.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:17:49 smithi047 ceph-mon[77147]: pgmap v1853: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:17:49.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:17:49 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:17:49.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:17:49 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 3000 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:17:49.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:17:49 smithi115 conmon[90393]: 2022-05-14T00:17:49.281+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:17:49.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:17:49 smithi115 ceph-mon[78593]: pgmap v1853: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:17:49.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:17:49 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:17:49.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:17:49 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 3000 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:17:50.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:17:50 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:17:50.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:17:50 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:17:50.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:17:50 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:17:50.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:17:50 smithi115 conmon[90393]: 2022-05-14T00:17:50.305+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:17:50.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:17:50 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:17:50.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:17:50 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:17:50.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:17:50 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:17:51.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:17:51 smithi047 ceph-mon[77147]: pgmap v1854: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:17:51.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:17:51 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:17:51.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:17:51 smithi115 conmon[90393]: 2022-05-14T00:17:51.265+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:17:51.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:17:51 smithi115 ceph-mon[78593]: pgmap v1854: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:17:51.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:17:51 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:17:52.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:17:52 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:17:52.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:17:52 smithi115 conmon[90393]: 2022-05-14T00:17:52.238+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:17:52.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:17:52 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:17:53.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:17:53 smithi047 ceph-mon[77147]: pgmap v1855: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:17:53.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:17:53 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:17:53.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:17:53 smithi115 conmon[90393]: 2022-05-14T00:17:53.275+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:17:53.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:17:53 smithi115 ceph-mon[78593]: pgmap v1855: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:17:53.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:17:53 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:17:54.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:17:54 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:17:54.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:17:54 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:17:54.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:17:54 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:17:54.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:17:54 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 3005 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:17:54.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:17:54 smithi115 conmon[90393]: 2022-05-14T00:17:54.252+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:17:54.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:17:54 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:17:54.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:17:54 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:17:54.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:17:54 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:17:54.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:17:54 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 3005 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:17:55.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:17:55 smithi047 ceph-mon[77147]: pgmap v1856: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:17:55.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:17:55 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:17:55.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:17:55 smithi115 conmon[90393]: 2022-05-14T00:17:55.226+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:17:55.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:17:55 smithi115 ceph-mon[78593]: pgmap v1856: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:17:55.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:17:55 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:17:56.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:17:56 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:17:56.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:17:56 smithi115 conmon[90393]: 2022-05-14T00:17:56.227+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:17:56.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:17:56 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:17:57.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:17:57 smithi047 ceph-mon[77147]: pgmap v1857: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:17:57.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:17:57 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:17:57.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:17:57 smithi115 conmon[90393]: 2022-05-14T00:17:57.180+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:17:57.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:17:57 smithi115 ceph-mon[78593]: pgmap v1857: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:17:57.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:17:57 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:17:58.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:17:58 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:17:58.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:17:58 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:17:58.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:17:58 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:17:58.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:17:58 smithi115 conmon[90393]: 2022-05-14T00:17:58.219+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:17:58.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:17:58 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:17:58.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:17:58 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:17:58.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:17:58 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:17:59.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:17:59 smithi047 ceph-mon[77147]: pgmap v1858: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:17:59.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:17:59 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:17:59.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:17:59 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 3010 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:17:59.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:17:59 smithi115 conmon[90393]: 2022-05-14T00:17:59.268+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:17:59.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:17:59 smithi115 ceph-mon[78593]: pgmap v1858: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:17:59.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:17:59 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:17:59.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:17:59 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 3010 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:18:00.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:18:00 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:18:00.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:18:00 smithi115 conmon[90393]: 2022-05-14T00:18:00.284+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:18:00.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:18:00 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:18:01.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:18:01 smithi047 ceph-mon[77147]: pgmap v1859: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:18:01.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:18:01 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:18:01.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:18:01 smithi115 conmon[90393]: 2022-05-14T00:18:01.245+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:18:01.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:18:01 smithi115 ceph-mon[78593]: pgmap v1859: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:18:01.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:18:01 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:18:02.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:18:02 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:18:02.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:18:02 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:18:02.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:18:02 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:18:02.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:18:02 smithi115 conmon[90393]: 2022-05-14T00:18:02.274+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:18:02.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:18:02 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:18:02.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:18:02 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:18:02.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:18:02 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:18:03.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:18:03 smithi047 ceph-mon[77147]: pgmap v1860: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:18:03.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:18:03 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:18:03.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:18:03 smithi115 conmon[90393]: 2022-05-14T00:18:03.233+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:18:03.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:18:03 smithi115 ceph-mon[78593]: pgmap v1860: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:18:03.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:18:03 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:18:04.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:18:04 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:18:04.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:18:04 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 3015 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:18:04.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:18:04 smithi115 conmon[90393]: 2022-05-14T00:18:04.230+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:18:04.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:18:04 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:18:04.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:18:04 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 3015 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:18:05.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:18:05 smithi047 ceph-mon[77147]: pgmap v1861: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:18:05.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:18:05 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:18:05.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:18:05 smithi115 conmon[90393]: 2022-05-14T00:18:05.258+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:18:05.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:18:05 smithi115 ceph-mon[78593]: pgmap v1861: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:18:05.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:18:05 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:18:06.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:18:06 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:18:06.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:18:06 smithi115 conmon[90393]: 2022-05-14T00:18:06.233+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:18:06.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:18:06 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:18:07.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:18:07 smithi047 ceph-mon[77147]: pgmap v1862: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:18:07.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:18:07 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:18:07.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:18:07 smithi115 conmon[90393]: 2022-05-14T00:18:07.279+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:18:07.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:18:07 smithi115 ceph-mon[78593]: pgmap v1862: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:18:07.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:18:07 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:18:08.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:18:08 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:18:08.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:18:08 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:18:08.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:18:08 smithi115 conmon[90393]: 2022-05-14T00:18:08.307+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:18:09.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:18:09 smithi047 ceph-mon[77147]: pgmap v1863: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:18:09.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:18:09 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:18:09.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:18:09 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 3020 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:18:09.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:18:09 smithi115 conmon[90393]: 2022-05-14T00:18:09.275+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:18:09.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:18:09 smithi115 ceph-mon[78593]: pgmap v1863: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:18:09.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:18:09 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:18:09.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:18:09 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 3020 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:18:10.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:18:10 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:18:10.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:18:10 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:18:10.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:18:10 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:18:10.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:18:10 smithi115 conmon[90393]: 2022-05-14T00:18:10.229+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:18:10.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:18:10 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:18:10.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:18:10 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:18:10.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:18:10 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:18:11.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:18:11 smithi047 ceph-mon[77147]: pgmap v1864: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:18:11.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:18:11 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:18:11.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:18:11 smithi115 conmon[90393]: 2022-05-14T00:18:11.227+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:18:11.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:18:11 smithi115 ceph-mon[78593]: pgmap v1864: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:18:11.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:18:11 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:18:12.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:18:12 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:18:12.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:18:12 smithi115 conmon[90393]: 2022-05-14T00:18:12.251+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:18:12.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:18:12 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:18:13.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:18:13 smithi047 ceph-mon[77147]: pgmap v1865: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:18:13.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:18:13 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:18:13.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:18:13 smithi115 conmon[90393]: 2022-05-14T00:18:13.235+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:18:13.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:18:13 smithi115 ceph-mon[78593]: pgmap v1865: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:18:13.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:18:13 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:18:14.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:18:14 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:18:14.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:18:14 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:18:14.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:18:14 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:18:14.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:18:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T00:18:14.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:18:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T00:18:14.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:18:14 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 3025 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:18:14.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:18:14 smithi115 conmon[90393]: 2022-05-14T00:18:14.223+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:18:14.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:18:14 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:18:14.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:18:14 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:18:14.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:18:14 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:18:14.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:18:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T00:18:14.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:18:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T00:18:14.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:18:14 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 3025 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:18:15.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:18:15 smithi047 ceph-mon[77147]: pgmap v1866: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 6 op/s 2022-05-14T00:18:15.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:18:15 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:18:15.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:18:15 smithi115 conmon[90393]: 2022-05-14T00:18:15.195+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:18:15.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:18:15 smithi115 ceph-mon[78593]: pgmap v1866: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 6 op/s 2022-05-14T00:18:15.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:18:15 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:18:16.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:18:16 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:18:16.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:18:16 smithi115 conmon[90393]: 2022-05-14T00:18:16.227+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:18:16.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:18:16 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:18:17.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:18:17 smithi047 ceph-mon[77147]: pgmap v1867: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 6 op/s 2022-05-14T00:18:17.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:18:17 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:18:17.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:18:17 smithi115 conmon[90393]: 2022-05-14T00:18:17.215+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:18:17.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:18:17 smithi115 ceph-mon[78593]: pgmap v1867: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 6 op/s 2022-05-14T00:18:17.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:18:17 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:18:18.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:18:18 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:18:18.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:18:18 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:18:18.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:18:18 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:18:18.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:18:18 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T00:18:18.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:18:18 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T00:18:18.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:18:18 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T00:18:18.542 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:18:18 smithi047 conmon[78521]: 2022-05-14T00:18:18.415+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:18:18.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:18:18 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T00:18:18.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:18:18 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:18:18.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:18:18 smithi047 conmon[78521]: yield 2022-05-14T00:18:18.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:18:18 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:18:18.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:18:18 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:18:18.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:18:18 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:18:18.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:18:18 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T00:18:18.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:18:18 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:18:18.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:18:18 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T00:18:18.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:18:18 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:18:18.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:18:18 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T00:18:18.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:18:18 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:18:18.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:18:18 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T00:18:18.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:18:18 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:18:18.547 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:18:18 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T00:18:18.547 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:18:18 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:18:18.547 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:18:18 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:18:18.547 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:18:18 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:18:18.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:18:18 smithi115 conmon[90393]: 2022-05-14T00:18:18.261+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:18:18.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:18:18 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:18:18.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:18:18 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:18:18.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:18:18 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:18:18.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:18:18 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T00:18:18.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:18:18 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T00:18:18.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:18:18 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T00:18:19.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:18:19 smithi047 ceph-mon[77147]: pgmap v1868: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 4 op/s 2022-05-14T00:18:19.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:18:19 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:18:19.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:18:19 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:18:19.542 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T00:18:19.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:18:19.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T00:18:19.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:18:19.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:18:19.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:18:19.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T00:18:19.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:18:19.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T00:18:19.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:18:19.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T00:18:19.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:18:19.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T00:18:19.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:18:19.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T00:18:19.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:18:19.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:18:19.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:18:19.548 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:18:19 smithi047 ceph-mon[77147]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T00:18:19.548 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:18:19 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 3030 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:18:19.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:18:19 smithi115 conmon[90393]: 2022-05-14T00:18:19.232+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:18:19.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:18:19 smithi115 ceph-mon[78593]: pgmap v1868: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 4 op/s 2022-05-14T00:18:19.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:18:19 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:18:19.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:18:19 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:18:19.609 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T00:18:19.609 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:18:19.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T00:18:19.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:18:19.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:18:19.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:18:19.613 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T00:18:19.613 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:18:19.613 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T00:18:19.614 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:18:19.614 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T00:18:19.614 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:18:19.615 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T00:18:19.615 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:18:19.615 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T00:18:19.616 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:18:19.616 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:18:19.616 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:18:19.616 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:18:19 smithi115 ceph-mon[78593]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T00:18:19.617 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:18:19 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 3030 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:18:20.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:18:20 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:18:20.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:18:20 smithi115 conmon[90393]: 2022-05-14T00:18:20.238+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:18:20.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:18:20 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:18:21.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:18:21 smithi047 ceph-mon[77147]: pgmap v1869: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 6 op/s 2022-05-14T00:18:21.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:18:21 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:18:21.542 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:18:21 smithi047 conmon[78521]: 2022-05-14T00:18:21.487+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:18:21.542 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:18:21 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T00:18:21.542 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:18:21 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:18:21.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:18:21 smithi047 conmon[78521]: yield 2022-05-14T00:18:21.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:18:21 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:18:21.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:18:21 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:18:21.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:18:21 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:18:21.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:18:21 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T00:18:21.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:18:21 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:18:21.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:18:21 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T00:18:21.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:18:21 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:18:21.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:18:21 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T00:18:21.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:18:21 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:18:21.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:18:21 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T00:18:21.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:18:21 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:18:21.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:18:21 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T00:18:21.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:18:21 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:18:21.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:18:21 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:18:21.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:18:21 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:18:21.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:18:21 smithi047 conmon[78521]: 2022-05-14T00:18:21.488+0000 7f533f777700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T00:18:21.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:18:21 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T00:18:21.547 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:18:21 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:18:21.547 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:18:21 smithi047 conmon[78521]: yield 2022-05-14T00:18:21.547 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:18:21 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:18:21.547 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:18:21 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:18:21.548 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:18:21 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:18:21.548 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:18:21 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T00:18:21.548 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:18:21 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:18:21.548 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:18:21 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T00:18:21.548 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:18:21 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:18:21.549 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:18:21 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T00:18:21.549 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:18:21 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:18:21.549 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:18:21 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T00:18:21.549 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:18:21 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:18:21.550 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:18:21 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T00:18:21.550 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:18:21 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:18:21.550 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:18:21 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:18:21.550 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:18:21 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:18:21.550 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:18:21 smithi047 conmon[78521]: 2022-05-14T00:18:21.551 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:18:21 smithi047 conmon[78521]: During handling of the above exception, another exception occurred: 2022-05-14T00:18:21.551 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:18:21 smithi047 conmon[78521]: 2022-05-14T00:18:21.551 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:18:21 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T00:18:21.551 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:18:21 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T00:18:21.552 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:18:21 smithi047 conmon[78521]: return f(*arg) 2022-05-14T00:18:21.552 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:18:21 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T00:18:21.552 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:18:21 smithi047 conmon[78521]: self._write_client_files(client_files, host) 2022-05-14T00:18:21.552 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:18:21 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T00:18:21.552 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:18:21 smithi047 conmon[78521]: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T00:18:21.553 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:18:21 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T00:18:21.553 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:18:21 smithi047 conmon[78521]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T00:18:21.553 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:18:21 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T00:18:21.553 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:18:21 smithi047 conmon[78521]: return self.event_loop.get_result(coro) 2022-05-14T00:18:21.554 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:18:21 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T00:18:21.554 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:18:21 smithi047 conmon[78521]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T00:18:21.554 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:18:21 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T00:18:21.554 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:18:21 smithi047 conmon[78521]: return self.__get_result() 2022-05-14T00:18:21.554 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:18:21 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T00:18:21.555 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:18:21 smithi047 conmon[78521]: raise self._exception 2022-05-14T00:18:21.555 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:18:21 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T00:18:21.555 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:18:21 smithi047 conmon[78521]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T00:18:21.555 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:18:21 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T00:18:21.556 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:18:21 smithi047 conmon[78521]: conn = await self._remote_connection(host, addr) 2022-05-14T00:18:21.556 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:18:21 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T00:18:21.556 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:18:21 smithi047 conmon[78521]: raise 2022-05-14T00:18:21.556 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:18:21 smithi047 conmon[78521]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T00:18:21.557 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:18:21 smithi047 conmon[78521]: self.gen.throw(type, value, traceback) 2022-05-14T00:18:21.557 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:18:21 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T00:18:21.558 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:18:21 smithi047 conmon[78521]: raise OrchestratorError(msg) 2022-05-14T00:18:21.558 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:18:21 smithi047 conmon[78521]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:18:21.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:18:21 smithi115 conmon[90393]: 2022-05-14T00:18:21.218+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:18:21.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:18:21 smithi115 ceph-mon[78593]: pgmap v1869: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 6 op/s 2022-05-14T00:18:21.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:18:21 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:18:22.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:18:22 smithi115 conmon[90393]: 2022-05-14T00:18:22.212+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:18:22.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:18:22 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:18:22.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:18:22 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:18:22.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:18:22 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:18:22.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:18:22 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:18:22.609 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T00:18:22.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:18:22.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T00:18:22.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:18:22.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:18:22.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:18:22.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T00:18:22.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:18:22.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T00:18:22.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:18:22.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T00:18:22.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:18:22.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T00:18:22.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:18:22.613 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T00:18:22.613 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:18:22.613 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:18:22.613 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:18:22.614 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:18:22 smithi115 ceph-mon[78593]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T00:18:22.614 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T00:18:22.614 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:18:22.614 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T00:18:22.614 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:18:22.615 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:18:22.615 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:18:22.615 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T00:18:22.615 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:18:22.616 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T00:18:22.616 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:18:22.616 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T00:18:22.616 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:18:22.616 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T00:18:22.617 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:18:22.617 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T00:18:22.617 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:18:22.617 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:18:22.618 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:18:22.618 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T00:18:22.618 INFO:journalctl@ceph.mon.c.smithi115.stdout: During handling of the above exception, another exception occurred: 2022-05-14T00:18:22.618 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T00:18:22.619 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T00:18:22.619 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T00:18:22.619 INFO:journalctl@ceph.mon.c.smithi115.stdout: return f(*arg) 2022-05-14T00:18:22.619 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T00:18:22.619 INFO:journalctl@ceph.mon.c.smithi115.stdout: self._write_client_files(client_files, host) 2022-05-14T00:18:22.620 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T00:18:22.620 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T00:18:22.620 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T00:18:22.620 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T00:18:22.621 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T00:18:22.621 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.event_loop.get_result(coro) 2022-05-14T00:18:22.621 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T00:18:22.621 INFO:journalctl@ceph.mon.c.smithi115.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T00:18:22.622 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T00:18:22.622 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.__get_result() 2022-05-14T00:18:22.622 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T00:18:22.622 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise self._exception 2022-05-14T00:18:22.622 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T00:18:22.623 INFO:journalctl@ceph.mon.c.smithi115.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T00:18:22.623 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T00:18:22.623 INFO:journalctl@ceph.mon.c.smithi115.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T00:18:22.623 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T00:18:22.624 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise 2022-05-14T00:18:22.624 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T00:18:22.624 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.gen.throw(type, value, traceback) 2022-05-14T00:18:22.624 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T00:18:22.624 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OrchestratorError(msg) 2022-05-14T00:18:22.625 INFO:journalctl@ceph.mon.c.smithi115.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:18:22.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:18:22 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:18:22.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:18:22 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:18:22.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:18:22 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:18:22.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:18:22 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:18:22.792 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T00:18:22.792 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:18:22.792 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T00:18:22.792 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:18:22.793 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:18:22.793 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:18:22.793 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T00:18:22.793 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:18:22.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T00:18:22.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:18:22.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T00:18:22.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:18:22.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T00:18:22.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:18:22.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T00:18:22.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:18:22.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:18:22.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:18:22.796 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:18:22 smithi047 ceph-mon[77147]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T00:18:22.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T00:18:22.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:18:22.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T00:18:22.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:18:22.797 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:18:22.797 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:18:22.797 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T00:18:22.797 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:18:22.798 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T00:18:22.798 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:18:22.798 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T00:18:22.798 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:18:22.798 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T00:18:22.799 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:18:22.799 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T00:18:22.799 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:18:22.799 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:18:22.799 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:18:22.800 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T00:18:22.800 INFO:journalctl@ceph.mon.b.smithi047.stdout: During handling of the above exception, another exception occurred: 2022-05-14T00:18:22.800 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T00:18:22.800 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T00:18:22.800 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T00:18:22.801 INFO:journalctl@ceph.mon.b.smithi047.stdout: return f(*arg) 2022-05-14T00:18:22.801 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T00:18:22.801 INFO:journalctl@ceph.mon.b.smithi047.stdout: self._write_client_files(client_files, host) 2022-05-14T00:18:22.801 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T00:18:22.802 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T00:18:22.802 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T00:18:22.802 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T00:18:22.802 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T00:18:22.802 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.event_loop.get_result(coro) 2022-05-14T00:18:22.803 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T00:18:22.803 INFO:journalctl@ceph.mon.b.smithi047.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T00:18:22.803 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T00:18:22.804 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.__get_result() 2022-05-14T00:18:22.804 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T00:18:22.804 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise self._exception 2022-05-14T00:18:22.804 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T00:18:22.805 INFO:journalctl@ceph.mon.b.smithi047.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T00:18:22.805 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T00:18:22.805 INFO:journalctl@ceph.mon.b.smithi047.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T00:18:22.805 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T00:18:22.806 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise 2022-05-14T00:18:22.806 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T00:18:22.806 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.gen.throw(type, value, traceback) 2022-05-14T00:18:22.806 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T00:18:22.807 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OrchestratorError(msg) 2022-05-14T00:18:22.807 INFO:journalctl@ceph.mon.b.smithi047.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:18:23.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:18:23 smithi115 conmon[90393]: 2022-05-14T00:18:23.204+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:18:23.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:18:23 smithi115 ceph-mon[78593]: pgmap v1870: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 4 op/s 2022-05-14T00:18:23.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:18:23 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:18:23.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:18:23 smithi047 ceph-mon[77147]: pgmap v1870: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 4 op/s 2022-05-14T00:18:23.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:18:23 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:18:24.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:18:24 smithi115 conmon[90393]: 2022-05-14T00:18:24.206+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:18:24.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:18:24 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:18:24.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:18:24 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 3035 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:18:24.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:18:24 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:18:24.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:18:24 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 3035 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:18:25.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:18:25 smithi115 conmon[90393]: 2022-05-14T00:18:25.175+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:18:25.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:18:25 smithi115 ceph-mon[78593]: pgmap v1871: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 6 op/s 2022-05-14T00:18:25.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:18:25 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:18:25.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:18:25 smithi047 ceph-mon[77147]: pgmap v1871: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 6 op/s 2022-05-14T00:18:25.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:18:25 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:18:26.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:18:26 smithi115 conmon[90393]: 2022-05-14T00:18:26.215+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:18:26.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:18:26 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:18:26.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:18:26 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:18:27.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:18:27 smithi115 conmon[90393]: 2022-05-14T00:18:27.230+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:18:27.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:18:27 smithi115 ceph-mon[78593]: pgmap v1872: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:18:27.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:18:27 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:18:27.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:18:27 smithi047 ceph-mon[77147]: pgmap v1872: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:18:27.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:18:27 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:18:28.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:18:28 smithi115 conmon[90393]: 2022-05-14T00:18:28.253+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:18:28.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:18:28 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:18:28.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:18:28 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:18:29.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:18:29 smithi115 conmon[90393]: 2022-05-14T00:18:29.258+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:18:29.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:18:29 smithi115 ceph-mon[78593]: pgmap v1873: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:18:29.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:18:29 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:18:29.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:18:29 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 3040 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:18:29.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:18:29 smithi047 ceph-mon[77147]: pgmap v1873: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:18:29.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:18:29 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:18:29.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:18:29 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 3040 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:18:30.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:18:30 smithi115 conmon[90393]: 2022-05-14T00:18:30.288+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:18:30.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:18:30 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:18:30.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:18:30 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:18:30.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:18:30 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:18:30.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:18:30 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:18:30.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:18:30 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:18:30.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:18:30 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:18:31.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:18:31 smithi115 conmon[90393]: 2022-05-14T00:18:31.255+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:18:31.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:18:31 smithi115 ceph-mon[78593]: pgmap v1874: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:18:31.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:18:31 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:18:31.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:18:31 smithi047 ceph-mon[77147]: pgmap v1874: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:18:31.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:18:31 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:18:32.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:18:32 smithi115 conmon[90393]: 2022-05-14T00:18:32.225+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:18:32.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:18:32 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:18:32.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:18:32 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:18:33.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:18:33 smithi115 conmon[90393]: 2022-05-14T00:18:33.185+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:18:33.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:18:33 smithi115 ceph-mon[78593]: pgmap v1875: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:18:33.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:18:33 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:18:33.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:18:33 smithi047 ceph-mon[77147]: pgmap v1875: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:18:33.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:18:33 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:18:34.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:18:34 smithi115 conmon[90393]: 2022-05-14T00:18:34.168+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:18:34.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:18:34 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:18:34.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:18:34 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:18:34.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:18:34 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:18:34.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:18:34 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 3045 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:18:34.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:18:34 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:18:34.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:18:34 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:18:34.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:18:34 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:18:34.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:18:34 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 3045 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:18:35.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:18:35 smithi115 conmon[90393]: 2022-05-14T00:18:35.145+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:18:35.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:18:35 smithi115 ceph-mon[78593]: pgmap v1876: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:18:35.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:18:35 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:18:35.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:18:35 smithi047 ceph-mon[77147]: pgmap v1876: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:18:35.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:18:35 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:18:36.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:18:36 smithi115 conmon[90393]: 2022-05-14T00:18:36.123+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:18:36.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:18:36 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:18:36.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:18:36 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:18:37.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:18:37 smithi115 conmon[90393]: 2022-05-14T00:18:37.127+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:18:37.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:18:37 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:18:37.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:18:37 smithi115 ceph-mon[78593]: pgmap v1877: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:18:37.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:18:37 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:18:37.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:18:37 smithi047 ceph-mon[77147]: pgmap v1877: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:18:38.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:18:38 smithi115 conmon[90393]: 2022-05-14T00:18:38.120+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:18:38.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:18:38 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:18:38.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:18:38 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:18:38.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:18:38 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:18:38.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:18:38 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:18:38.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:18:38 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:18:38.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:18:38 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:18:39.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:18:39 smithi115 conmon[90393]: 2022-05-14T00:18:39.133+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:18:39.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:18:39 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:18:39.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:18:39 smithi115 ceph-mon[78593]: pgmap v1878: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:18:39.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:18:39 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 3050 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:18:39.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:18:39 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:18:39.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:18:39 smithi047 ceph-mon[77147]: pgmap v1878: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:18:39.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:18:39 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 3050 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:18:40.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:18:40 smithi115 conmon[90393]: 2022-05-14T00:18:40.158+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:18:40.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:18:40 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:18:40.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:18:40 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:18:41.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:18:41 smithi115 conmon[90393]: 2022-05-14T00:18:41.121+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:18:41.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:18:41 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:18:41.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:18:41 smithi115 ceph-mon[78593]: pgmap v1879: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:18:41.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:18:41 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:18:41.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:18:41 smithi047 ceph-mon[77147]: pgmap v1879: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:18:42.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:18:42 smithi115 conmon[90393]: 2022-05-14T00:18:42.135+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:18:42.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:18:42 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:18:42.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:18:42 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:18:42.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:18:42 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:18:42.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:18:42 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:18:42.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:18:42 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:18:42.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:18:42 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:18:43.373 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:18:43 smithi115 conmon[90393]: 2022-05-14T00:18:43.121+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:18:43.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:18:43 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:18:43.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:18:43 smithi047 ceph-mon[77147]: pgmap v1880: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:18:43.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:18:43 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:18:43.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:18:43 smithi115 ceph-mon[78593]: pgmap v1880: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:18:44.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:18:44 smithi115 conmon[90393]: 2022-05-14T00:18:44.163+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:18:44.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:18:44 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:18:44.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:18:44 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 3055 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:18:44.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:18:44 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:18:44.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:18:44 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 3055 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:18:45.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:18:45 smithi115 conmon[90393]: 2022-05-14T00:18:45.131+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:18:45.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:18:45 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:18:45.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:18:45 smithi115 ceph-mon[78593]: pgmap v1881: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:18:45.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:18:45 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:18:45.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:18:45 smithi047 ceph-mon[77147]: pgmap v1881: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:18:46.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:18:46 smithi115 conmon[90393]: 2022-05-14T00:18:46.175+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:18:46.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:18:46 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:18:46.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:18:46 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:18:47.383 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:18:47 smithi115 conmon[90393]: 2022-05-14T00:18:47.127+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:18:47.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:18:47 smithi047 ceph-mon[77147]: pgmap v1882: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:18:47.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:18:47 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:18:47.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:18:47 smithi115 ceph-mon[78593]: pgmap v1882: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:18:47.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:18:47 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:18:48.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:18:48 smithi115 conmon[90393]: 2022-05-14T00:18:48.085+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:18:48.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:18:48 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:18:48.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:18:48 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:18:49.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:18:49 smithi115 conmon[90393]: 2022-05-14T00:18:49.083+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:18:49.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:18:49 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:18:49.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:18:49 smithi047 ceph-mon[77147]: pgmap v1883: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:18:49.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:18:49 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 3060 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:18:49.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:18:49 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:18:49.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:18:49 smithi115 ceph-mon[78593]: pgmap v1883: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:18:49.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:18:49 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 3060 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:18:50.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:18:50 smithi115 conmon[90393]: 2022-05-14T00:18:50.042+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:18:50.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:18:50 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:18:50.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:18:50 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:18:50.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:18:50 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:18:50.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:18:50 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:18:50.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:18:50 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:18:50.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:18:50 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:18:51.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:18:51 smithi115 conmon[90393]: 2022-05-14T00:18:51.041+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:18:51.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:18:51 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:18:51.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:18:51 smithi047 ceph-mon[77147]: pgmap v1884: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:18:51.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:18:51 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:18:51.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:18:51 smithi115 ceph-mon[78593]: pgmap v1884: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:18:52.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:18:52 smithi115 conmon[90393]: 2022-05-14T00:18:52.043+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:18:52.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:18:52 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:18:52.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:18:52 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:18:53.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:18:53 smithi115 conmon[90393]: 2022-05-14T00:18:53.093+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:18:53.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:18:53 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:18:53.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:18:53 smithi047 ceph-mon[77147]: pgmap v1885: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:18:53.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:18:53 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:18:53.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:18:53 smithi115 ceph-mon[78593]: pgmap v1885: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:18:54.415 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:18:54 smithi115 conmon[90393]: 2022-05-14T00:18:54.119+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:18:54.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:18:54 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:18:54.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:18:54 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:18:54.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:18:54 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:18:54.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:18:54 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 3064 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:18:54.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:18:54 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:18:54.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:18:54 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:18:54.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:18:54 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:18:54.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:18:54 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 3064 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:18:55.416 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:18:55 smithi115 conmon[90393]: 2022-05-14T00:18:55.114+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:18:55.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:18:55 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:18:55.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:18:55 smithi047 ceph-mon[77147]: pgmap v1886: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:18:55.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:18:55 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:18:55.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:18:55 smithi115 ceph-mon[78593]: pgmap v1886: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:18:56.421 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:18:56 smithi115 conmon[90393]: 2022-05-14T00:18:56.160+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:18:56.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:18:56 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:18:56.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:18:56 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:18:57.429 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:18:57 smithi115 conmon[90393]: 2022-05-14T00:18:57.110+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:18:57.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:18:57 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:18:57.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:18:57 smithi047 ceph-mon[77147]: pgmap v1887: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:18:57.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:18:57 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:18:57.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:18:57 smithi115 ceph-mon[78593]: pgmap v1887: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:18:58.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:18:58 smithi115 conmon[90393]: 2022-05-14T00:18:58.078+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:18:58.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:18:58 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:18:58.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:18:58 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:18:58.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:18:58 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:18:58.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:18:58 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:18:58.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:18:58 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:18:58.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:18:58 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:18:59.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:18:59 smithi115 conmon[90393]: 2022-05-14T00:18:59.085+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:18:59.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:18:59 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:18:59.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:18:59 smithi047 ceph-mon[77147]: pgmap v1888: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:18:59.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:18:59 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 3070 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:18:59.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:18:59 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:18:59.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:18:59 smithi115 ceph-mon[78593]: pgmap v1888: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:18:59.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:18:59 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 3070 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:19:00.444 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:19:00 smithi115 conmon[90393]: 2022-05-14T00:19:00.108+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:19:00.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:19:00 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:19:00.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:19:00 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:19:01.443 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:19:01 smithi115 conmon[90393]: 2022-05-14T00:19:01.148+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:19:01.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:19:01 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:19:01.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:19:01 smithi047 ceph-mon[77147]: pgmap v1889: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:19:01.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:19:01 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:19:01.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:19:01 smithi115 ceph-mon[78593]: pgmap v1889: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:19:02.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:19:02 smithi115 conmon[90393]: 2022-05-14T00:19:02.105+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:19:02.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:19:02 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:19:02.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:19:02 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:19:02.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:19:02 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:19:02.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:19:02 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:19:02.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:19:02 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:19:02.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:19:02 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:19:03.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:19:03 smithi115 conmon[90393]: 2022-05-14T00:19:03.101+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:19:03.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:19:03 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:19:03.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:19:03 smithi047 ceph-mon[77147]: pgmap v1890: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:19:03.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:19:03 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:19:03.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:19:03 smithi115 ceph-mon[78593]: pgmap v1890: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:19:04.457 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:19:04 smithi115 conmon[90393]: 2022-05-14T00:19:04.133+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:19:04.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:19:04 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:19:04.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:19:04 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 3074 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:19:04.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:19:04 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:19:04.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:19:04 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 3074 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:19:05.459 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:19:05 smithi115 conmon[90393]: 2022-05-14T00:19:05.138+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:19:05.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:19:05 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:19:05.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:19:05 smithi047 ceph-mon[77147]: pgmap v1891: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:19:05.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:19:05 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:19:05.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:19:05 smithi115 ceph-mon[78593]: pgmap v1891: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:19:06.465 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:19:06 smithi115 conmon[90393]: 2022-05-14T00:19:06.146+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:19:06.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:19:06 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:19:06.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:19:06 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:19:07.466 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:19:07 smithi115 conmon[90393]: 2022-05-14T00:19:07.149+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:19:07.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:19:07 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:19:07.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:19:07 smithi047 ceph-mon[77147]: pgmap v1892: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:19:07.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:19:07 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:19:07.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:19:07 smithi115 ceph-mon[78593]: pgmap v1892: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:19:08.471 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:19:08 smithi115 conmon[90393]: 2022-05-14T00:19:08.151+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:19:08.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:19:08 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:19:08.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:19:08 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:19:09.476 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:19:09 smithi115 conmon[90393]: 2022-05-14T00:19:09.128+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:19:09.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:19:09 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:19:09.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:19:09 smithi047 ceph-mon[77147]: pgmap v1893: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:19:09.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:19:09 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 3080 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:19:09.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:19:09 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:19:09.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:19:09 smithi115 ceph-mon[78593]: pgmap v1893: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:19:09.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:19:09 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 3080 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:19:10.479 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:19:10 smithi115 conmon[90393]: 2022-05-14T00:19:10.139+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:19:10.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:19:10 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:19:10.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:19:10 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:19:10.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:19:10 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:19:10.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:19:10 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:19:10.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:19:10 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:19:10.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:19:10 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:19:11.480 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:19:11 smithi115 conmon[90393]: 2022-05-14T00:19:11.135+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:19:11.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:19:11 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:19:11.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:19:11 smithi047 ceph-mon[77147]: pgmap v1894: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:19:11.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:19:11 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:19:11.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:19:11 smithi115 ceph-mon[78593]: pgmap v1894: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:19:12.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:19:12 smithi115 conmon[90393]: 2022-05-14T00:19:12.101+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:19:12.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:19:12 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:19:12.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:19:12 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:19:13.487 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:19:13 smithi115 conmon[90393]: 2022-05-14T00:19:13.127+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:19:13.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:19:13 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:19:13.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:19:13 smithi047 ceph-mon[77147]: pgmap v1895: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:19:13.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:19:13 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:19:13.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:19:13 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:19:13.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:19:13 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T00:19:13.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:19:13 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T00:19:13.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:19:13 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:19:13.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:19:13 smithi115 ceph-mon[78593]: pgmap v1895: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:19:13.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:19:13 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:19:13.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:19:13 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:19:13.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:19:13 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T00:19:13.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:19:13 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T00:19:14.489 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:19:14 smithi115 conmon[90393]: 2022-05-14T00:19:14.135+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:19:14.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:19:14 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:19:14.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:19:14 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 3085 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:19:14.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:19:14 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:19:14.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:19:14 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 3085 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:19:15.490 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:19:15 smithi115 conmon[90393]: 2022-05-14T00:19:15.181+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:19:15.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:19:15 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:19:15.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:19:15 smithi047 ceph-mon[77147]: pgmap v1896: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:19:15.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:19:15 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:19:15.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:19:15 smithi115 ceph-mon[78593]: pgmap v1896: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:19:16.496 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:19:16 smithi115 conmon[90393]: 2022-05-14T00:19:16.147+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:19:16.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:19:16 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:19:16.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:19:16 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:19:17.497 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:19:17 smithi115 conmon[90393]: 2022-05-14T00:19:17.169+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:19:17.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:19:17 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:19:17.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:19:17 smithi047 ceph-mon[77147]: pgmap v1897: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:19:17.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:19:17 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:19:17.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:19:17 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:19:17.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:19:17 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:19:17.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:19:17 smithi115 ceph-mon[78593]: pgmap v1897: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:19:17.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:19:17 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:19:17.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:19:17 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:19:18.501 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:19:18 smithi115 conmon[90393]: 2022-05-14T00:19:18.208+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:19:18.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:19:18 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:19:18.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:19:18 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:19:19.503 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:19:19 smithi115 conmon[90393]: 2022-05-14T00:19:19.189+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:19:19.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:19:19 smithi047 ceph-mon[77147]: pgmap v1898: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:19:19.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:19:19 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:19:19.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:19:19 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 3090 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:19:19.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:19:19 smithi115 ceph-mon[78593]: pgmap v1898: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:19:19.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:19:19 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:19:19.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:19:19 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 3090 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:19:20.508 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:19:20 smithi115 conmon[90393]: 2022-05-14T00:19:20.179+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:19:20.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:19:20 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:19:20.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:19:20 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:19:21.509 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:19:21 smithi115 conmon[90393]: 2022-05-14T00:19:21.207+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:19:21.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:19:21 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:19:21.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:19:21 smithi047 ceph-mon[77147]: pgmap v1899: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:19:21.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:19:21 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:19:21.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:19:21 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:19:21.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:19:21 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T00:19:21.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:19:21 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T00:19:21.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:19:21 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T00:19:21.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:19:21 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:19:21.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:19:21 smithi115 ceph-mon[78593]: pgmap v1899: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:19:21.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:19:21 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:19:21.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:19:21 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:19:21.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:19:21 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T00:19:21.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:19:21 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T00:19:21.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:19:21 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T00:19:22.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:19:22 smithi115 conmon[90393]: 2022-05-14T00:19:22.178+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:19:22.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:19:22 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:19:22.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:19:22 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' 2022-05-14T00:19:22.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:19:22 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' 2022-05-14T00:19:22.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:19:22 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:19:22.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:19:22 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' 2022-05-14T00:19:22.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:19:22 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' 2022-05-14T00:19:23.514 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:19:23 smithi115 conmon[90393]: 2022-05-14T00:19:23.206+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:19:23.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:19:23 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:19:23.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:19:23 smithi047 ceph-mon[77147]: pgmap v1900: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:19:23.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:19:23 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:19:23.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:19:23 smithi115 ceph-mon[78593]: pgmap v1900: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:19:24.520 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:19:24 smithi115 conmon[90393]: 2022-05-14T00:19:24.216+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:19:24.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:19:24 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:19:24.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:19:24 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 3095 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:19:24.793 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:19:24 smithi047 conmon[78521]: 2022-05-14T00:19:24.591+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:19:24.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:19:24 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T00:19:24.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:19:24 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:19:24.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:19:24 smithi047 conmon[78521]: yield 2022-05-14T00:19:24.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:19:24 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:19:24.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:19:24 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:19:24.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:19:24 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:19:24.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:19:24 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T00:19:24.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:19:24 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:19:24.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:19:24 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T00:19:24.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:19:24 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:19:24.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:19:24 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T00:19:24.797 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:19:24 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:19:24.797 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:19:24 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T00:19:24.797 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:19:24 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:19:24.798 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:19:24 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T00:19:24.798 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:19:24 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:19:24.798 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:19:24 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:19:24.798 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:19:24 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:19:24.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:19:24 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:19:24.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:19:24 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 3095 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:19:25.518 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:19:25 smithi115 conmon[90393]: 2022-05-14T00:19:25.236+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:19:25.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:19:25 smithi047 ceph-mon[77147]: pgmap v1901: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:19:25.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:19:25 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:19:25.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:19:25 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:19:25.791 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T00:19:25.791 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:19:25.792 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T00:19:25.792 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:19:25.792 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:19:25.792 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:19:25.792 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T00:19:25.793 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:19:25.793 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T00:19:25.793 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:19:25.793 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T00:19:25.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:19:25.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T00:19:25.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:19:25.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T00:19:25.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:19:25.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:19:25.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:19:25.795 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:19:25 smithi047 ceph-mon[77147]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T00:19:25.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:19:25 smithi115 ceph-mon[78593]: pgmap v1901: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:19:25.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:19:25 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:19:25.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:19:25 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:19:25.858 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T00:19:25.858 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:19:25.859 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T00:19:25.859 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:19:25.859 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:19:25.859 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:19:25.860 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T00:19:25.860 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:19:25.860 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T00:19:25.860 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:19:25.860 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T00:19:25.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:19:25.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T00:19:25.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:19:25.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T00:19:25.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:19:25.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:19:25.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:19:25.862 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:19:25 smithi115 ceph-mon[78593]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T00:19:26.522 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:19:26 smithi115 conmon[90393]: 2022-05-14T00:19:26.263+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:19:26.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:19:26 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:19:26.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:19:26 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:19:27.524 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:19:27 smithi115 conmon[90393]: 2022-05-14T00:19:27.217+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:19:27.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:19:27 smithi047 ceph-mon[77147]: pgmap v1902: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:19:27.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:19:27 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:19:27.792 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:19:27 smithi047 conmon[78521]: 2022-05-14T00:19:27.663+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:19:27.792 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:19:27 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T00:19:27.793 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:19:27 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:19:27.793 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:19:27 smithi047 conmon[78521]: yield 2022-05-14T00:19:27.793 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:19:27 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:19:27.793 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:19:27 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:19:27.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:19:27 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:19:27.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:19:27 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T00:19:27.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:19:27 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:19:27.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:19:27 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T00:19:27.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:19:27 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:19:27.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:19:27 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T00:19:27.797 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:19:27 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:19:27.798 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:19:27 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T00:19:27.798 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:19:27 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:19:27.798 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:19:27 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T00:19:27.798 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:19:27 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:19:27.799 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:19:27 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:19:27.799 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:19:27 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:19:27.799 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:19:27 smithi047 conmon[78521]: 2022-05-14T00:19:27.664+0000 7f5344f82700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T00:19:27.799 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:19:27 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T00:19:27.800 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:19:27 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:19:27.800 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:19:27 smithi047 conmon[78521]: yield 2022-05-14T00:19:27.800 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:19:27 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:19:27.801 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:19:27 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:19:27.801 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:19:27 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:19:27.801 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:19:27 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T00:19:27.802 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:19:27 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:19:27.802 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:19:27 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T00:19:27.802 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:19:27 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:19:27.802 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:19:27 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T00:19:27.803 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:19:27 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:19:27.803 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:19:27 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T00:19:27.803 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:19:27 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:19:27.804 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:19:27 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T00:19:27.804 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:19:27 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:19:27.804 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:19:27 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:19:27.805 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:19:27 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:19:27.805 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:19:27 smithi047 conmon[78521]: 2022-05-14T00:19:27.805 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:19:27 smithi047 conmon[78521]: During handling of the above exception, another exception occurred: 2022-05-14T00:19:27.806 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:19:27 smithi047 conmon[78521]: 2022-05-14T00:19:27.806 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:19:27 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T00:19:27.807 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:19:27 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T00:19:27.807 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:19:27 smithi047 conmon[78521]: return f(*arg) 2022-05-14T00:19:27.807 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:19:27 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T00:19:27.808 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:19:27 smithi047 conmon[78521]: self._write_client_files(client_files, host) 2022-05-14T00:19:27.808 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:19:27 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T00:19:27.808 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:19:27 smithi047 conmon[78521]: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T00:19:27.808 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:19:27 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T00:19:27.808 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:19:27 smithi047 conmon[78521]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T00:19:27.809 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:19:27 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T00:19:27.809 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:19:27 smithi047 conmon[78521]: return self.event_loop.get_result(coro) 2022-05-14T00:19:27.809 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:19:27 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T00:19:27.809 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:19:27 smithi047 conmon[78521]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T00:19:27.810 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:19:27 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T00:19:27.810 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:19:27 smithi047 conmon[78521]: return self.__get_result() 2022-05-14T00:19:27.810 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:19:27 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T00:19:27.810 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:19:27 smithi047 conmon[78521]: raise self._exception 2022-05-14T00:19:27.811 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:19:27 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T00:19:27.811 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:19:27 smithi047 conmon[78521]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T00:19:27.811 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:19:27 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T00:19:27.811 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:19:27 smithi047 conmon[78521]: conn = await self._remote_connection(host, addr) 2022-05-14T00:19:27.811 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:19:27 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T00:19:27.812 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:19:27 smithi047 conmon[78521]: raise 2022-05-14T00:19:27.812 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:19:27 smithi047 conmon[78521]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T00:19:27.812 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:19:27 smithi047 conmon[78521]: self.gen.throw(type, value, traceback) 2022-05-14T00:19:27.812 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:19:27 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T00:19:27.813 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:19:27 smithi047 conmon[78521]: raise OrchestratorError(msg) 2022-05-14T00:19:27.813 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:19:27 smithi047 conmon[78521]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:19:27.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:19:27 smithi115 ceph-mon[78593]: pgmap v1902: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:19:27.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:19:27 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:19:28.530 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:19:28 smithi115 conmon[90393]: 2022-05-14T00:19:28.182+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:19:28.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:19:28 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:19:28.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:19:28 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:19:28.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T00:19:28.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:19:28.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T00:19:28.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:19:28.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:19:28.797 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:19:28.797 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T00:19:28.797 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:19:28.798 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T00:19:28.798 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:19:28.798 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T00:19:28.798 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:19:28.799 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T00:19:28.799 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:19:28.799 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T00:19:28.799 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:19:28.800 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:19:28.800 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:19:28.800 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:19:28 smithi047 ceph-mon[77147]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T00:19:28.800 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T00:19:28.800 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:19:28.801 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T00:19:28.802 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:19:28.802 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:19:28.802 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:19:28.802 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T00:19:28.803 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:19:28.803 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T00:19:28.803 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:19:28.803 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T00:19:28.803 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:19:28.804 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T00:19:28.804 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:19:28.804 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T00:19:28.804 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:19:28.805 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:19:28.805 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:19:28.805 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T00:19:28.805 INFO:journalctl@ceph.mon.b.smithi047.stdout: During handling of the above exception, another exception occurred: 2022-05-14T00:19:28.806 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T00:19:28.806 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T00:19:28.806 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T00:19:28.806 INFO:journalctl@ceph.mon.b.smithi047.stdout: return f(*arg) 2022-05-14T00:19:28.807 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T00:19:28.807 INFO:journalctl@ceph.mon.b.smithi047.stdout: self._write_client_files(client_files, host) 2022-05-14T00:19:28.807 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T00:19:28.807 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T00:19:28.807 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T00:19:28.808 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T00:19:28.808 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T00:19:28.808 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.event_loop.get_result(coro) 2022-05-14T00:19:28.808 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T00:19:28.809 INFO:journalctl@ceph.mon.b.smithi047.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T00:19:28.809 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T00:19:28.809 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.__get_result() 2022-05-14T00:19:28.809 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T00:19:28.809 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise self._exception 2022-05-14T00:19:28.810 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T00:19:28.810 INFO:journalctl@ceph.mon.b.smithi047.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T00:19:28.810 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T00:19:28.810 INFO:journalctl@ceph.mon.b.smithi047.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T00:19:28.811 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T00:19:28.811 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise 2022-05-14T00:19:28.811 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T00:19:28.811 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.gen.throw(type, value, traceback) 2022-05-14T00:19:28.811 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T00:19:28.812 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OrchestratorError(msg) 2022-05-14T00:19:28.812 INFO:journalctl@ceph.mon.b.smithi047.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:19:28.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:19:28 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:19:28.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:19:28 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:19:28.858 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T00:19:28.859 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:19:28.859 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T00:19:28.860 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:19:28.860 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:19:28.860 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:19:28.860 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T00:19:28.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:19:28.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T00:19:28.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:19:28.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T00:19:28.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:19:28.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T00:19:28.863 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:19:28.863 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T00:19:28.863 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:19:28.864 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:19:28.864 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:19:28.864 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:19:28 smithi115 ceph-mon[78593]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T00:19:28.865 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T00:19:28.865 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:19:28.865 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T00:19:28.865 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:19:28.866 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:19:28.866 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:19:28.866 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T00:19:28.867 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:19:28.867 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T00:19:28.867 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:19:28.868 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T00:19:28.868 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:19:28.868 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T00:19:28.868 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:19:28.869 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T00:19:28.869 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:19:28.869 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:19:28.870 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:19:28.870 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T00:19:28.870 INFO:journalctl@ceph.mon.c.smithi115.stdout: During handling of the above exception, another exception occurred: 2022-05-14T00:19:28.871 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T00:19:28.871 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T00:19:28.871 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T00:19:28.871 INFO:journalctl@ceph.mon.c.smithi115.stdout: return f(*arg) 2022-05-14T00:19:28.872 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T00:19:28.872 INFO:journalctl@ceph.mon.c.smithi115.stdout: self._write_client_files(client_files, host) 2022-05-14T00:19:28.872 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T00:19:28.873 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T00:19:28.873 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T00:19:28.873 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T00:19:28.874 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T00:19:28.874 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.event_loop.get_result(coro) 2022-05-14T00:19:28.874 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T00:19:28.874 INFO:journalctl@ceph.mon.c.smithi115.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T00:19:28.875 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T00:19:28.875 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.__get_result() 2022-05-14T00:19:28.875 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T00:19:28.876 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise self._exception 2022-05-14T00:19:28.876 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T00:19:28.876 INFO:journalctl@ceph.mon.c.smithi115.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T00:19:28.877 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T00:19:28.877 INFO:journalctl@ceph.mon.c.smithi115.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T00:19:28.877 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T00:19:28.878 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise 2022-05-14T00:19:28.878 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T00:19:28.878 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.gen.throw(type, value, traceback) 2022-05-14T00:19:28.879 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T00:19:28.879 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OrchestratorError(msg) 2022-05-14T00:19:28.879 INFO:journalctl@ceph.mon.c.smithi115.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:19:29.531 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:19:29 smithi115 conmon[90393]: 2022-05-14T00:19:29.159+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:19:29.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:19:29 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:19:29.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:19:29 smithi047 ceph-mon[77147]: pgmap v1903: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:19:29.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:19:29 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:19:29.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:19:29 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:19:29.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:19:29 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 3100 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:19:29.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:19:29 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:19:29.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:19:29 smithi115 ceph-mon[78593]: pgmap v1903: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:19:29.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:19:29 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:19:29.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:19:29 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:19:29.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:19:29 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 3100 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:19:30.536 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:19:30 smithi115 conmon[90393]: 2022-05-14T00:19:30.184+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:19:30.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:19:30 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:19:30.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:19:30 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:19:31.537 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:19:31 smithi115 conmon[90393]: 2022-05-14T00:19:31.193+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:19:31.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:19:31 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:19:31.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:19:31 smithi047 ceph-mon[77147]: pgmap v1904: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:19:31.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:19:31 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:19:31.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:19:31 smithi115 ceph-mon[78593]: pgmap v1904: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:19:32.545 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:19:32 smithi115 conmon[90393]: 2022-05-14T00:19:32.160+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:19:33.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:19:32 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:19:33.550 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:19:32 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:19:33.551 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:19:33 smithi115 conmon[90393]: 2022-05-14T00:19:33.146+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:19:33.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:19:33 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:19:33.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:19:33 smithi115 ceph-mon[78593]: pgmap v1905: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:19:33.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:19:33 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:19:33.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:19:33 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:19:34.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:19:33 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:19:34.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:19:33 smithi047 ceph-mon[77147]: pgmap v1905: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:19:34.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:19:33 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:19:34.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:19:33 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:19:34.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:19:34 smithi115 conmon[90393]: 2022-05-14T00:19:34.104+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:19:34.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:19:34 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:19:34.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:19:34 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 3105 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:19:35.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:19:34 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:19:35.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:19:34 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 3105 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:19:35.549 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:19:35 smithi115 conmon[90393]: 2022-05-14T00:19:35.136+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:19:35.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:19:35 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:19:35.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:19:35 smithi115 ceph-mon[78593]: pgmap v1906: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:19:36.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:19:35 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:19:36.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:19:35 smithi047 ceph-mon[77147]: pgmap v1906: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:19:36.556 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:19:36 smithi115 conmon[90393]: 2022-05-14T00:19:36.119+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:19:36.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:19:36 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:19:37.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:19:36 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:19:37.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:19:37 smithi115 conmon[90393]: 2022-05-14T00:19:37.087+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:19:37.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:19:37 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:19:37.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:19:37 smithi115 ceph-mon[78593]: pgmap v1907: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:19:37.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:19:37 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:19:37.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:19:37 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:19:38.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:19:37 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:19:38.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:19:37 smithi047 ceph-mon[77147]: pgmap v1907: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:19:38.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:19:37 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:19:38.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:19:37 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:19:38.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:19:38 smithi115 conmon[90393]: 2022-05-14T00:19:38.063+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:19:38.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:19:38 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:19:39.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:19:38 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:19:39.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:19:39 smithi115 conmon[90393]: 2022-05-14T00:19:39.059+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:19:39.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:19:39 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:19:39.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:19:39 smithi115 ceph-mon[78593]: pgmap v1908: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:19:39.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:19:39 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 3110 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:19:40.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:19:39 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:19:40.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:19:39 smithi047 ceph-mon[77147]: pgmap v1908: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:19:40.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:19:39 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 3110 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:19:40.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:19:40 smithi115 conmon[90393]: 2022-05-14T00:19:40.029+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:19:40.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:19:40 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:19:41.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:19:40 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:19:41.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:19:40 smithi115 conmon[90393]: 2022-05-14T00:19:40.991+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:19:41.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:19:41 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:19:41.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:19:41 smithi115 ceph-mon[78593]: pgmap v1909: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:19:41.873 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:19:41 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:19:41.873 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:19:41 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:19:42.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:19:41 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:19:42.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:19:41 smithi047 ceph-mon[77147]: pgmap v1909: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:19:42.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:19:41 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:19:42.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:19:41 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:19:42.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:19:42 smithi115 conmon[90393]: 2022-05-14T00:19:42.007+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:19:42.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:19:42 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:19:43.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:19:42 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:19:43.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:19:42 smithi115 conmon[90393]: 2022-05-14T00:19:42.967+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:19:43.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:19:43 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:19:43.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:19:43 smithi115 ceph-mon[78593]: pgmap v1910: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:19:44.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:19:43 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:19:44.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:19:43 smithi047 ceph-mon[77147]: pgmap v1910: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:19:44.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:19:43 smithi115 conmon[90393]: 2022-05-14T00:19:43.948+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:19:44.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:19:44 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:19:44.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:19:44 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 3114 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:19:45.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:19:44 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:19:45.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:19:44 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 3114 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:19:45.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:19:44 smithi115 conmon[90393]: 2022-05-14T00:19:44.945+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:19:45.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:19:45 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:19:45.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:19:45 smithi115 ceph-mon[78593]: pgmap v1911: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:19:46.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:19:45 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:19:46.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:19:45 smithi047 ceph-mon[77147]: pgmap v1911: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:19:46.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:19:45 smithi115 conmon[90393]: 2022-05-14T00:19:45.942+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:19:46.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:19:46 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:19:47.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:19:46 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:19:47.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:19:46 smithi115 conmon[90393]: 2022-05-14T00:19:46.938+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:19:47.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:19:47 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:19:47.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:19:47 smithi115 ceph-mon[78593]: pgmap v1912: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:19:48.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:19:47 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:19:48.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:19:47 smithi047 ceph-mon[77147]: pgmap v1912: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:19:48.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:19:47 smithi115 conmon[90393]: 2022-05-14T00:19:47.934+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:19:48.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:19:48 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:19:49.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:19:48 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:19:49.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:19:48 smithi115 conmon[90393]: 2022-05-14T00:19:48.948+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:19:49.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:19:49 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:19:49.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:19:49 smithi115 ceph-mon[78593]: pgmap v1913: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:19:49.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:19:49 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:19:49.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:19:49 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:19:49.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:19:49 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 3120 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:19:50.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:19:49 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:19:50.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:19:49 smithi047 ceph-mon[77147]: pgmap v1913: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:19:50.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:19:49 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:19:50.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:19:49 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:19:50.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:19:49 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 3120 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:19:50.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:19:49 smithi115 conmon[90393]: 2022-05-14T00:19:49.909+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:19:50.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:19:50 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:19:51.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:19:50 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:19:51.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:19:50 smithi115 conmon[90393]: 2022-05-14T00:19:50.909+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:19:51.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:19:51 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:19:51.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:19:51 smithi115 ceph-mon[78593]: pgmap v1914: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:19:51.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:19:51 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:19:52.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:19:51 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:19:52.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:19:51 smithi047 ceph-mon[77147]: pgmap v1914: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:19:52.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:19:51 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:19:52.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:19:51 smithi115 conmon[90393]: 2022-05-14T00:19:51.867+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:19:52.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:19:52 smithi115 conmon[90393]: 2022-05-14T00:19:52.848+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:19:52.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:19:52 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:19:53.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:19:52 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:19:53.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:19:53 smithi115 ceph-mon[78593]: pgmap v1915: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:19:53.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:19:53 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:19:53.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:19:53 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:19:53.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:19:53 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:19:54.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:19:53 smithi047 ceph-mon[77147]: pgmap v1915: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:19:54.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:19:53 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:19:54.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:19:53 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:19:54.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:19:53 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:19:54.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:19:53 smithi115 conmon[90393]: 2022-05-14T00:19:53.859+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:19:54.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:19:54 smithi115 conmon[90393]: 2022-05-14T00:19:54.827+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:19:54.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:19:54 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 3125 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:19:54.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:19:54 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:19:55.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:19:54 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 3125 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:19:55.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:19:54 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:19:56.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:19:55 smithi115 conmon[90393]: 2022-05-14T00:19:55.784+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:19:56.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:19:55 smithi115 ceph-mon[78593]: pgmap v1916: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:19:56.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:19:55 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:19:56.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:19:55 smithi047 ceph-mon[77147]: pgmap v1916: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:19:56.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:19:55 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:19:57.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:19:56 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:19:57.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:19:56 smithi115 conmon[90393]: 2022-05-14T00:19:56.782+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:19:57.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:19:56 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:19:58.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:19:57 smithi115 conmon[90393]: 2022-05-14T00:19:57.808+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:19:58.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:19:57 smithi115 ceph-mon[78593]: pgmap v1917: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:19:58.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:19:57 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:19:58.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:19:57 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:19:58.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:19:57 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:19:58.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:19:57 smithi047 ceph-mon[77147]: pgmap v1917: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:19:58.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:19:57 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:19:58.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:19:57 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:19:58.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:19:57 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:19:59.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:19:58 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:19:59.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:19:58 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 3130 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:19:59.110 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:19:58 smithi115 conmon[90393]: 2022-05-14T00:19:58.818+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:19:59.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:19:58 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:19:59.300 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:19:58 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 3130 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:20:00.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:19:59 smithi115 conmon[90393]: 2022-05-14T00:19:59.795+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:20:00.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:19:59 smithi115 ceph-mon[78593]: pgmap v1918: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:20:00.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:19:59 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:20:00.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:19:59 smithi047 ceph-mon[77147]: pgmap v1918: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:20:00.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:19:59 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:20:00.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:20:00 smithi047 conmon[77123]: 2022-05-14T00:19:59.999+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : Health detail: HEALTH_ERR 1 filesystem is degraded; 1 mds daemon damaged; 1 MDSs are read only; 1 MDSs report slow metadata IOs; 1/3 mons down, quorum b,c; Reduced data availability: 1 pg inactive; 9 slow ops, oldest one blocked for 3130 sec, osd.5 has slow ops 2022-05-14T00:20:00.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:20:00 smithi047 conmon[77123]: 2022-05-14T00:19:59.999+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : [WRN] FS_DEGRADED: 1 filesystem is degraded 2022-05-14T00:20:00.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:20:00 smithi047 conmon[77123]: 2022-05-14T00:19:59.999+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : fs cephfs is degraded 2022-05-14T00:20:00.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:20:00 smithi047 conmon[77123]: 2022-05-14T00:19:59.999+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : [ERR] MDS_DAMAGE: 1 mds daemon damaged 2022-05-14T00:20:00.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:20:00 smithi047 conmon[77123]: 2022-05-14T00:19:59.999+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : fs cephfs mds.2 is damaged 2022-05-14T00:20:00.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:20:00 smithi047 conmon[77123]: 2022-05-14T00:19:59.999+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : [WRN] MDS_READ_ONLY: 1 MDSs are read only 2022-05-14T00:20:00.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:20:00 smithi047 conmon[77123]: 2022-05-14T00:19:59.999+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : mds.l(mds.0): MDS in read-only mode 2022-05-14T00:20:00.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:20:00 smithi047 conmon[77123]: 2022-05-14T00:19:59.999+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : [WRN] MDS_SLOW_METADATA_IO: 1 MDSs report slow metadata IOs 2022-05-14T00:20:00.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:20:00 smithi047 conmon[77123]: 2022-05-14T00:19:59.999+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : mds.l(mds.0): 4 slow metadata IOs are blocked > 180 secs, oldest blocked for 3433 secs 2022-05-14T00:20:00.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:20:00 smithi047 conmon[77123]: 2022-05-14T00:19:59.999+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : [WRN] MON_DOWN: 1/3 mons down, quorum b,c 2022-05-14T00:20:00.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:20:00 smithi047 conmon[77123]: 2022-05-14T00:19:59.999+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : mon.a (rank 0) addr [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] is down (out of quorum) 2022-05-14T00:20:00.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:20:00 smithi047 conmon[77123]: 2022-05-14T00:19:59.999+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : [WRN] PG_AVAILABILITY: Reduced data availability: 1 pg inactive 2022-05-14T00:20:00.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:20:00 smithi047 conmon[77123]: 2022-05-14T00:19:59.999+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : pg 2.1 is stuck inactive for 64m, current state unknown, last acting [] 2022-05-14T00:20:00.295 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:20:00 smithi047 conmon[77123]: 2022-05-14T00:19:59.999+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : [WRN] SLOW_OPS: 9 slow ops, oldest one blocked for 3130 sec, osd.5 has slow ops 2022-05-14T00:20:01.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:20:00 smithi115 conmon[90393]: 2022-05-14T00:20:00.840+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:20:01.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:20:00 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:20:01.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:20:00 smithi115 ceph-mon[78593]: Health detail: HEALTH_ERR 1 filesystem is degraded; 1 mds daemon damaged; 1 MDSs are read only; 1 MDSs report slow metadata IOs; 1/3 mons down, quorum b,c; Reduced data availability: 1 pg inactive; 9 slow ops, oldest one blocked for 3130 sec, osd.5 has slow ops 2022-05-14T00:20:01.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:20:00 smithi115 ceph-mon[78593]: [WRN] FS_DEGRADED: 1 filesystem is degraded 2022-05-14T00:20:01.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:20:00 smithi115 ceph-mon[78593]: fs cephfs is degraded 2022-05-14T00:20:01.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:20:00 smithi115 ceph-mon[78593]: [ERR] MDS_DAMAGE: 1 mds daemon damaged 2022-05-14T00:20:01.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:20:00 smithi115 ceph-mon[78593]: fs cephfs mds.2 is damaged 2022-05-14T00:20:01.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:20:00 smithi115 ceph-mon[78593]: [WRN] MDS_READ_ONLY: 1 MDSs are read only 2022-05-14T00:20:01.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:20:00 smithi115 ceph-mon[78593]: mds.l(mds.0): MDS in read-only mode 2022-05-14T00:20:01.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:20:00 smithi115 ceph-mon[78593]: [WRN] MDS_SLOW_METADATA_IO: 1 MDSs report slow metadata IOs 2022-05-14T00:20:01.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:20:00 smithi115 ceph-mon[78593]: mds.l(mds.0): 4 slow metadata IOs are blocked > 180 secs, oldest blocked for 3433 secs 2022-05-14T00:20:01.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:20:00 smithi115 ceph-mon[78593]: [WRN] MON_DOWN: 1/3 mons down, quorum b,c 2022-05-14T00:20:01.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:20:00 smithi115 ceph-mon[78593]: mon.a (rank 0) addr [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] is down (out of quorum) 2022-05-14T00:20:01.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:20:00 smithi115 ceph-mon[78593]: [WRN] PG_AVAILABILITY: Reduced data availability: 1 pg inactive 2022-05-14T00:20:01.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:20:00 smithi115 ceph-mon[78593]: pg 2.1 is stuck inactive for 64m, current state unknown, last acting [] 2022-05-14T00:20:01.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:20:00 smithi115 ceph-mon[78593]: [WRN] SLOW_OPS: 9 slow ops, oldest one blocked for 3130 sec, osd.5 has slow ops 2022-05-14T00:20:01.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:20:00 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:20:01.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:20:00 smithi047 ceph-mon[77147]: Health detail: HEALTH_ERR 1 filesystem is degraded; 1 mds daemon damaged; 1 MDSs are read only; 1 MDSs report slow metadata IOs; 1/3 mons down, quorum b,c; Reduced data availability: 1 pg inactive; 9 slow ops, oldest one blocked for 3130 sec, osd.5 has slow ops 2022-05-14T00:20:01.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:20:00 smithi047 ceph-mon[77147]: [WRN] FS_DEGRADED: 1 filesystem is degraded 2022-05-14T00:20:01.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:20:00 smithi047 ceph-mon[77147]: fs cephfs is degraded 2022-05-14T00:20:01.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:20:00 smithi047 ceph-mon[77147]: [ERR] MDS_DAMAGE: 1 mds daemon damaged 2022-05-14T00:20:01.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:20:00 smithi047 ceph-mon[77147]: fs cephfs mds.2 is damaged 2022-05-14T00:20:01.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:20:00 smithi047 ceph-mon[77147]: [WRN] MDS_READ_ONLY: 1 MDSs are read only 2022-05-14T00:20:01.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:20:00 smithi047 ceph-mon[77147]: mds.l(mds.0): MDS in read-only mode 2022-05-14T00:20:01.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:20:00 smithi047 ceph-mon[77147]: [WRN] MDS_SLOW_METADATA_IO: 1 MDSs report slow metadata IOs 2022-05-14T00:20:01.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:20:00 smithi047 ceph-mon[77147]: mds.l(mds.0): 4 slow metadata IOs are blocked > 180 secs, oldest blocked for 3433 secs 2022-05-14T00:20:01.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:20:00 smithi047 ceph-mon[77147]: [WRN] MON_DOWN: 1/3 mons down, quorum b,c 2022-05-14T00:20:01.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:20:00 smithi047 ceph-mon[77147]: mon.a (rank 0) addr [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] is down (out of quorum) 2022-05-14T00:20:01.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:20:00 smithi047 ceph-mon[77147]: [WRN] PG_AVAILABILITY: Reduced data availability: 1 pg inactive 2022-05-14T00:20:01.295 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:20:00 smithi047 ceph-mon[77147]: pg 2.1 is stuck inactive for 64m, current state unknown, last acting [] 2022-05-14T00:20:01.295 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:20:00 smithi047 ceph-mon[77147]: [WRN] SLOW_OPS: 9 slow ops, oldest one blocked for 3130 sec, osd.5 has slow ops 2022-05-14T00:20:02.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:20:01 smithi115 conmon[90393]: 2022-05-14T00:20:01.800+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:20:02.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:20:01 smithi115 ceph-mon[78593]: pgmap v1919: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:20:02.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:20:01 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:20:02.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:20:01 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:20:02.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:20:01 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:20:02.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:20:01 smithi047 ceph-mon[77147]: pgmap v1919: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:20:02.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:20:01 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:20:02.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:20:01 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:20:02.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:20:01 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:20:03.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:20:02 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:20:03.110 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:20:02 smithi115 conmon[90393]: 2022-05-14T00:20:02.795+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:20:03.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:20:02 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:20:04.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:20:03 smithi115 conmon[90393]: 2022-05-14T00:20:03.842+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:20:04.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:20:03 smithi115 ceph-mon[78593]: pgmap v1920: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:20:04.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:20:03 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:20:04.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:20:03 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 3135 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:20:04.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:20:03 smithi047 ceph-mon[77147]: pgmap v1920: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:20:04.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:20:03 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:20:04.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:20:03 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 3135 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:20:05.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:20:04 smithi115 conmon[90393]: 2022-05-14T00:20:04.825+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:20:05.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:20:04 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:20:05.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:20:04 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:20:06.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:20:05 smithi115 conmon[90393]: 2022-05-14T00:20:05.840+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:20:06.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:20:05 smithi115 ceph-mon[78593]: pgmap v1921: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:20:06.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:20:05 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:20:06.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:20:05 smithi047 ceph-mon[77147]: pgmap v1921: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:20:06.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:20:05 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:20:07.110 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:20:06 smithi115 conmon[90393]: 2022-05-14T00:20:06.827+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:20:07.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:20:06 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:20:07.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:20:06 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:20:08.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:20:07 smithi115 conmon[90393]: 2022-05-14T00:20:07.834+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:20:08.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:20:07 smithi115 ceph-mon[78593]: pgmap v1922: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:20:08.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:20:07 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:20:08.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:20:07 smithi047 ceph-mon[77147]: pgmap v1922: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:20:08.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:20:07 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:20:09.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:20:08 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:20:09.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:20:08 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 3140 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:20:09.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:20:08 smithi115 conmon[90393]: 2022-05-14T00:20:08.865+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:20:09.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:20:08 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:20:09.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:20:08 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 3140 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:20:10.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:20:09 smithi047 ceph-mon[77147]: pgmap v1923: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:20:10.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:20:09 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:20:10.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:20:09 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:20:10.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:20:09 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:20:10.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:20:09 smithi115 conmon[90393]: 2022-05-14T00:20:09.907+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:20:10.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:20:09 smithi115 ceph-mon[78593]: pgmap v1923: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:20:10.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:20:09 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:20:10.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:20:09 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:20:10.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:20:09 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:20:11.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:20:10 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:20:11.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:20:10 smithi115 conmon[90393]: 2022-05-14T00:20:10.884+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:20:11.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:20:10 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:20:12.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:20:11 smithi115 conmon[90393]: 2022-05-14T00:20:11.854+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:20:12.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:20:11 smithi115 ceph-mon[78593]: pgmap v1924: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:20:12.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:20:11 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:20:12.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:20:11 smithi047 ceph-mon[77147]: pgmap v1924: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:20:12.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:20:11 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:20:13.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:20:12 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:20:13.111 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:20:12 smithi115 conmon[90393]: 2022-05-14T00:20:12.809+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:20:13.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:20:12 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:20:14.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:20:13 smithi115 conmon[90393]: 2022-05-14T00:20:13.776+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:20:14.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:20:13 smithi115 ceph-mon[78593]: pgmap v1925: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:20:14.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:20:13 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:20:14.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:20:13 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:20:14.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:20:13 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:20:14.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:20:13 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T00:20:14.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:20:13 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T00:20:14.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:20:13 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 3145 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:20:14.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:20:13 smithi047 ceph-mon[77147]: pgmap v1925: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:20:14.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:20:13 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:20:14.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:20:13 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:20:14.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:20:13 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:20:14.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:20:13 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T00:20:14.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:20:13 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T00:20:14.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:20:13 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 3145 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:20:15.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:20:14 smithi115 conmon[90393]: 2022-05-14T00:20:14.817+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:20:15.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:20:14 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:20:15.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:20:14 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:20:16.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:20:15 smithi115 conmon[90393]: 2022-05-14T00:20:15.792+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:20:16.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:20:15 smithi115 ceph-mon[78593]: pgmap v1926: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:20:16.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:20:15 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:20:16.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:20:15 smithi047 ceph-mon[77147]: pgmap v1926: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:20:16.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:20:15 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:20:17.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:20:16 smithi115 conmon[90393]: 2022-05-14T00:20:16.752+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:20:17.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:20:16 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:20:17.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:20:16 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:20:18.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:20:17 smithi115 conmon[90393]: 2022-05-14T00:20:17.784+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:20:18.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:20:17 smithi115 ceph-mon[78593]: pgmap v1927: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:20:18.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:20:17 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:20:18.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:20:17 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:20:18.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:20:17 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:20:18.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:20:17 smithi047 ceph-mon[77147]: pgmap v1927: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:20:18.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:20:17 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:20:18.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:20:17 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:20:18.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:20:17 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:20:19.112 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:20:18 smithi115 conmon[90393]: 2022-05-14T00:20:18.820+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:20:19.113 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:20:18 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:20:19.113 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:20:18 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 3150 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:20:19.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:20:18 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:20:19.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:20:18 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 3150 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:20:20.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:20:19 smithi047 ceph-mon[77147]: pgmap v1928: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:20:20.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:20:19 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:20:20.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:20:19 smithi115 conmon[90393]: 2022-05-14T00:20:19.862+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:20:20.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:20:19 smithi115 ceph-mon[78593]: pgmap v1928: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:20:20.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:20:19 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:20:21.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:20:20 smithi115 conmon[90393]: 2022-05-14T00:20:20.835+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:20:21.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:20:20 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:20:21.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:20:20 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:20:22.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:20:21 smithi115 conmon[90393]: 2022-05-14T00:20:21.835+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:20:22.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:20:21 smithi115 ceph-mon[78593]: pgmap v1929: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:20:22.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:20:21 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:20:22.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:20:21 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:20:22.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:20:21 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:20:22.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:20:21 smithi047 ceph-mon[77147]: pgmap v1929: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:20:22.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:20:21 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:20:22.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:20:21 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:20:22.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:20:21 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:20:23.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:20:22 smithi115 conmon[90393]: 2022-05-14T00:20:22.852+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:20:23.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:20:22 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:20:23.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:20:22 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:20:24.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:20:23 smithi047 ceph-mon[77147]: pgmap v1930: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:20:24.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:20:23 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:20:24.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:20:23 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 3155 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:20:24.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:20:23 smithi115 conmon[90393]: 2022-05-14T00:20:23.863+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:20:24.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:20:23 smithi115 ceph-mon[78593]: pgmap v1930: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:20:24.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:20:23 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:20:24.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:20:23 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 3155 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:20:25.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:20:24 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:20:25.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:20:24 smithi115 conmon[90393]: 2022-05-14T00:20:24.899+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:20:25.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:20:24 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:20:26.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:20:25 smithi047 ceph-mon[77147]: pgmap v1931: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:20:26.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:20:25 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:20:26.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:20:25 smithi115 conmon[90393]: 2022-05-14T00:20:25.924+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:20:26.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:20:25 smithi115 ceph-mon[78593]: pgmap v1931: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:20:26.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:20:25 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:20:27.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:20:26 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:20:27.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:20:26 smithi115 conmon[90393]: 2022-05-14T00:20:26.889+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:20:27.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:20:26 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:20:28.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:20:27 smithi047 ceph-mon[77147]: pgmap v1932: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:20:28.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:20:27 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:20:28.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:20:27 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T00:20:28.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:20:27 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T00:20:28.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:20:27 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T00:20:28.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:20:27 smithi115 conmon[90393]: 2022-05-14T00:20:27.880+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:20:28.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:20:27 smithi115 ceph-mon[78593]: pgmap v1932: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:20:28.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:20:27 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:20:28.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:20:27 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T00:20:28.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:20:27 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T00:20:28.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:20:27 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T00:20:28.541 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:20:28 smithi047 conmon[78521]: 2022-05-14T00:20:28.144+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:20:28.541 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:20:28 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T00:20:28.541 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:20:28 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:20:28.541 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:20:28 smithi047 conmon[78521]: yield 2022-05-14T00:20:28.542 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:20:28 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:20:28.542 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:20:28 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:20:28.542 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:20:28 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:20:28.542 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:20:28 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T00:20:28.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:20:28 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:20:28.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:20:28 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T00:20:28.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:20:28 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:20:28.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:20:28 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T00:20:28.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:20:28 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:20:28.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:20:28 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T00:20:28.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:20:28 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:20:28.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:20:28 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T00:20:28.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:20:28 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:20:28.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:20:28 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:20:28.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:20:28 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:20:29.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:20:28 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:20:29.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:20:28 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 3160 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:20:29.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:20:28 smithi115 conmon[90393]: 2022-05-14T00:20:28.862+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:20:29.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:20:28 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:20:29.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:20:28 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 3160 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:20:30.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:20:29 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:20:30.109 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T00:20:30.109 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:20:30.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T00:20:30.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:20:30.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:20:30.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:20:30.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T00:20:30.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:20:30.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T00:20:30.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:20:30.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T00:20:30.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:20:30.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T00:20:30.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:20:30.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T00:20:30.113 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:20:30.113 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:20:30.113 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:20:30.114 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:20:29 smithi115 ceph-mon[78593]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T00:20:30.114 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:20:29 smithi115 ceph-mon[78593]: pgmap v1933: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:20:30.114 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:20:29 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:20:30.114 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:20:29 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:20:30.114 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:20:29 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:20:30.115 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:20:29 smithi115 conmon[90393]: 2022-05-14T00:20:29.813+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:20:30.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:20:29 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:20:30.291 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T00:20:30.291 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:20:30.291 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T00:20:30.292 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:20:30.292 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:20:30.292 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:20:30.292 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T00:20:30.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:20:30.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T00:20:30.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:20:30.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T00:20:30.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:20:30.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T00:20:30.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:20:30.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T00:20:30.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:20:30.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:20:30.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:20:30.295 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:20:29 smithi047 ceph-mon[77147]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T00:20:30.295 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:20:29 smithi047 ceph-mon[77147]: pgmap v1933: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:20:30.296 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:20:29 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:20:30.296 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:20:29 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:20:30.296 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:20:29 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:20:31.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:20:30 smithi115 conmon[90393]: 2022-05-14T00:20:30.765+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:20:31.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:20:30 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:20:31.217 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:20:30 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:20:31.541 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:20:31 smithi047 conmon[78521]: 2022-05-14T00:20:31.216+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:20:31.541 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:20:31 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T00:20:31.541 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:20:31 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:20:31.542 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:20:31 smithi047 conmon[78521]: yield 2022-05-14T00:20:31.542 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:20:31 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:20:31.542 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:20:31 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:20:31.542 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:20:31 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:20:31.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:20:31 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T00:20:31.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:20:31 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:20:31.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:20:31 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T00:20:31.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:20:31 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:20:31.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:20:31 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T00:20:31.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:20:31 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:20:31.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:20:31 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T00:20:31.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:20:31 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:20:31.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:20:31 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T00:20:31.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:20:31 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:20:31.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:20:31 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:20:31.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:20:31 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:20:31.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:20:31 smithi047 conmon[78521]: 2022-05-14T00:20:31.217+0000 7f5344781700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T00:20:31.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:20:31 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T00:20:31.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:20:31 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:20:31.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:20:31 smithi047 conmon[78521]: yield 2022-05-14T00:20:31.547 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:20:31 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:20:31.547 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:20:31 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:20:31.547 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:20:31 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:20:31.547 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:20:31 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T00:20:31.547 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:20:31 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:20:31.548 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:20:31 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T00:20:31.548 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:20:31 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:20:31.548 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:20:31 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T00:20:31.548 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:20:31 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:20:31.549 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:20:31 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T00:20:31.549 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:20:31 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:20:31.549 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:20:31 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T00:20:31.549 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:20:31 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:20:31.549 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:20:31 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:20:31.550 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:20:31 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:20:31.550 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:20:31 smithi047 conmon[78521]: 2022-05-14T00:20:31.550 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:20:31 smithi047 conmon[78521]: During handling of the above exception, another exception occurred: 2022-05-14T00:20:31.550 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:20:31 smithi047 conmon[78521]: 2022-05-14T00:20:31.551 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:20:31 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T00:20:31.551 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:20:31 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T00:20:31.551 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:20:31 smithi047 conmon[78521]: return f(*arg) 2022-05-14T00:20:31.551 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:20:31 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T00:20:31.552 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:20:31 smithi047 conmon[78521]: self._write_client_files(client_files, host) 2022-05-14T00:20:31.552 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:20:31 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T00:20:31.552 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:20:31 smithi047 conmon[78521]: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T00:20:31.552 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:20:31 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T00:20:31.552 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:20:31 smithi047 conmon[78521]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T00:20:31.553 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:20:31 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T00:20:31.553 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:20:31 smithi047 conmon[78521]: return self.event_loop.get_result(coro) 2022-05-14T00:20:31.553 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:20:31 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T00:20:31.553 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:20:31 smithi047 conmon[78521]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T00:20:31.553 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:20:31 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T00:20:31.554 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:20:31 smithi047 conmon[78521]: return self.__get_result() 2022-05-14T00:20:31.554 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:20:31 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T00:20:31.554 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:20:31 smithi047 conmon[78521]: raise self._exception 2022-05-14T00:20:31.554 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:20:31 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T00:20:31.555 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:20:31 smithi047 conmon[78521]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T00:20:31.555 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:20:31 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T00:20:31.555 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:20:31 smithi047 conmon[78521]: conn = await self._remote_connection(host, addr) 2022-05-14T00:20:31.555 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:20:31 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T00:20:31.555 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:20:31 smithi047 conmon[78521]: raise 2022-05-14T00:20:31.556 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:20:31 smithi047 conmon[78521]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T00:20:31.556 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:20:31 smithi047 conmon[78521]: self.gen.throw(type, value, traceback) 2022-05-14T00:20:31.556 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:20:31 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T00:20:31.556 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:20:31 smithi047 conmon[78521]: raise OrchestratorError(msg) 2022-05-14T00:20:31.556 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:20:31 smithi047 conmon[78521]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:20:32.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:20:31 smithi115 conmon[90393]: 2022-05-14T00:20:31.807+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:20:32.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:20:31 smithi115 ceph-mon[78593]: pgmap v1934: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:20:32.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:20:31 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:20:32.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:20:31 smithi047 ceph-mon[77147]: pgmap v1934: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:20:32.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:20:31 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:20:33.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:20:32 smithi115 conmon[90393]: 2022-05-14T00:20:32.774+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:20:33.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:20:32 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:20:33.108 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T00:20:33.109 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:20:33.109 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T00:20:33.109 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:20:33.109 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:20:33.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:20:33.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T00:20:33.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:20:33.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T00:20:33.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:20:33.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T00:20:33.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:20:33.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T00:20:33.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:20:33.113 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T00:20:33.113 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:20:33.113 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:20:33.113 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:20:33.114 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:20:32 smithi115 ceph-mon[78593]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T00:20:33.114 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T00:20:33.114 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:20:33.114 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T00:20:33.114 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:20:33.115 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:20:33.115 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:20:33.115 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T00:20:33.115 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:20:33.115 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T00:20:33.116 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:20:33.116 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T00:20:33.116 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:20:33.116 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T00:20:33.117 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:20:33.117 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T00:20:33.117 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:20:33.117 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:20:33.118 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:20:33.118 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T00:20:33.118 INFO:journalctl@ceph.mon.c.smithi115.stdout: During handling of the above exception, another exception occurred: 2022-05-14T00:20:33.119 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T00:20:33.119 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T00:20:33.119 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T00:20:33.119 INFO:journalctl@ceph.mon.c.smithi115.stdout: return f(*arg) 2022-05-14T00:20:33.120 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T00:20:33.120 INFO:journalctl@ceph.mon.c.smithi115.stdout: self._write_client_files(client_files, host) 2022-05-14T00:20:33.120 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T00:20:33.120 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T00:20:33.121 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T00:20:33.121 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T00:20:33.121 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T00:20:33.121 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.event_loop.get_result(coro) 2022-05-14T00:20:33.122 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T00:20:33.122 INFO:journalctl@ceph.mon.c.smithi115.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T00:20:33.122 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T00:20:33.122 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.__get_result() 2022-05-14T00:20:33.122 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T00:20:33.123 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise self._exception 2022-05-14T00:20:33.123 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T00:20:33.123 INFO:journalctl@ceph.mon.c.smithi115.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T00:20:33.123 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T00:20:33.123 INFO:journalctl@ceph.mon.c.smithi115.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T00:20:33.124 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T00:20:33.125 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise 2022-05-14T00:20:33.125 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T00:20:33.125 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.gen.throw(type, value, traceback) 2022-05-14T00:20:33.125 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T00:20:33.126 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OrchestratorError(msg) 2022-05-14T00:20:33.126 INFO:journalctl@ceph.mon.c.smithi115.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:20:33.126 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:20:32 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:20:33.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:20:32 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:20:33.291 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T00:20:33.291 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:20:33.291 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T00:20:33.292 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:20:33.292 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:20:33.292 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:20:33.292 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T00:20:33.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:20:33.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T00:20:33.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:20:33.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T00:20:33.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:20:33.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T00:20:33.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:20:33.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T00:20:33.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:20:33.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:20:33.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:20:33.295 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:20:32 smithi047 ceph-mon[77147]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T00:20:33.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T00:20:33.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:20:33.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T00:20:33.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:20:33.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:20:33.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:20:33.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T00:20:33.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:20:33.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T00:20:33.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:20:33.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T00:20:33.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:20:33.298 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T00:20:33.298 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:20:33.298 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T00:20:33.298 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:20:33.298 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:20:33.299 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:20:33.299 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T00:20:33.299 INFO:journalctl@ceph.mon.b.smithi047.stdout: During handling of the above exception, another exception occurred: 2022-05-14T00:20:33.299 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T00:20:33.299 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T00:20:33.300 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T00:20:33.300 INFO:journalctl@ceph.mon.b.smithi047.stdout: return f(*arg) 2022-05-14T00:20:33.300 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T00:20:33.300 INFO:journalctl@ceph.mon.b.smithi047.stdout: self._write_client_files(client_files, host) 2022-05-14T00:20:33.300 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T00:20:33.301 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T00:20:33.301 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T00:20:33.301 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T00:20:33.301 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T00:20:33.302 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.event_loop.get_result(coro) 2022-05-14T00:20:33.302 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T00:20:33.302 INFO:journalctl@ceph.mon.b.smithi047.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T00:20:33.302 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T00:20:33.302 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.__get_result() 2022-05-14T00:20:33.303 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T00:20:33.303 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise self._exception 2022-05-14T00:20:33.303 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T00:20:33.303 INFO:journalctl@ceph.mon.b.smithi047.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T00:20:33.303 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T00:20:33.304 INFO:journalctl@ceph.mon.b.smithi047.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T00:20:33.304 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T00:20:33.304 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise 2022-05-14T00:20:33.304 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T00:20:33.304 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.gen.throw(type, value, traceback) 2022-05-14T00:20:33.305 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T00:20:33.305 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OrchestratorError(msg) 2022-05-14T00:20:33.305 INFO:journalctl@ceph.mon.b.smithi047.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:20:33.305 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:20:32 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:20:34.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:20:33 smithi115 conmon[90393]: 2022-05-14T00:20:33.801+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:20:34.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:20:33 smithi115 ceph-mon[78593]: pgmap v1935: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:20:34.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:20:33 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:20:34.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:20:33 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:20:34.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:20:33 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:20:34.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:20:33 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 3165 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:20:34.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:20:33 smithi047 ceph-mon[77147]: pgmap v1935: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:20:34.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:20:33 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:20:34.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:20:33 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:20:34.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:20:33 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:20:34.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:20:33 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 3165 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:20:35.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:20:34 smithi115 conmon[90393]: 2022-05-14T00:20:34.772+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:20:35.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:20:34 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:20:35.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:20:34 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:20:36.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:20:35 smithi115 conmon[90393]: 2022-05-14T00:20:35.783+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:20:36.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:20:35 smithi115 ceph-mon[78593]: pgmap v1936: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:20:36.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:20:35 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:20:36.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:20:35 smithi047 ceph-mon[77147]: pgmap v1936: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:20:36.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:20:35 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:20:37.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:20:36 smithi115 conmon[90393]: 2022-05-14T00:20:36.785+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:20:37.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:20:36 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:20:37.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:20:36 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:20:38.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:20:37 smithi115 conmon[90393]: 2022-05-14T00:20:37.754+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:20:38.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:20:37 smithi115 ceph-mon[78593]: pgmap v1937: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:20:38.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:20:37 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:20:38.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:20:37 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:20:38.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:20:37 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:20:38.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:20:37 smithi047 ceph-mon[77147]: pgmap v1937: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:20:38.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:20:37 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:20:38.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:20:37 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:20:38.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:20:37 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:20:39.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:20:38 smithi115 conmon[90393]: 2022-05-14T00:20:38.787+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:20:39.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:20:38 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:20:39.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:20:38 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 3170 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:20:39.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:20:38 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:20:39.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:20:38 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 3170 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:20:40.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:20:39 smithi115 conmon[90393]: 2022-05-14T00:20:39.770+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:20:40.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:20:39 smithi115 ceph-mon[78593]: pgmap v1938: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:20:40.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:20:39 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:20:40.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:20:39 smithi047 ceph-mon[77147]: pgmap v1938: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:20:40.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:20:39 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:20:41.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:20:40 smithi115 conmon[90393]: 2022-05-14T00:20:40.767+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:20:41.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:20:40 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:20:41.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:20:40 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:20:42.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:20:41 smithi115 conmon[90393]: 2022-05-14T00:20:41.816+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:20:42.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:20:41 smithi115 ceph-mon[78593]: pgmap v1939: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:20:42.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:20:41 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:20:42.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:20:41 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:20:42.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:20:41 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:20:42.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:20:41 smithi047 ceph-mon[77147]: pgmap v1939: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:20:42.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:20:41 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:20:42.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:20:41 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:20:42.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:20:41 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:20:43.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:20:42 smithi115 conmon[90393]: 2022-05-14T00:20:42.770+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:20:43.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:20:42 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:20:43.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:20:42 smithi115 ceph-mon[78593]: pgmap v1940: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:20:43.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:20:42 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:20:43.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:20:42 smithi047 ceph-mon[77147]: pgmap v1940: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:20:44.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:20:43 smithi115 conmon[90393]: 2022-05-14T00:20:43.798+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:20:44.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:20:43 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:20:44.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:20:43 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 3175 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:20:44.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:20:43 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:20:44.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:20:43 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 3175 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:20:45.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:20:44 smithi115 conmon[90393]: 2022-05-14T00:20:44.766+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:20:45.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:20:44 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:20:45.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:20:44 smithi115 ceph-mon[78593]: pgmap v1941: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:20:45.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:20:44 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:20:45.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:20:44 smithi047 ceph-mon[77147]: pgmap v1941: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:20:46.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:20:45 smithi115 conmon[90393]: 2022-05-14T00:20:45.803+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:20:46.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:20:45 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:20:46.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:20:45 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:20:47.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:20:46 smithi115 conmon[90393]: 2022-05-14T00:20:46.811+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:20:47.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:20:46 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:20:47.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:20:46 smithi115 ceph-mon[78593]: pgmap v1942: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:20:47.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:20:46 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:20:47.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:20:46 smithi047 ceph-mon[77147]: pgmap v1942: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:20:48.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:20:47 smithi115 conmon[90393]: 2022-05-14T00:20:47.851+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:20:48.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:20:47 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:20:48.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:20:47 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:20:49.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:20:48 smithi115 conmon[90393]: 2022-05-14T00:20:48.816+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:20:49.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:20:48 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:20:49.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:20:48 smithi115 ceph-mon[78593]: pgmap v1943: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:20:49.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:20:48 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 3180 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:20:49.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:20:48 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:20:49.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:20:48 smithi047 ceph-mon[77147]: pgmap v1943: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:20:49.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:20:48 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 3180 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:20:50.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:20:49 smithi115 conmon[90393]: 2022-05-14T00:20:49.769+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:20:50.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:20:49 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:20:50.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:20:49 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:20:50.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:20:49 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:20:50.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:20:49 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:20:50.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:20:49 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:20:50.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:20:49 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:20:51.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:20:50 smithi115 conmon[90393]: 2022-05-14T00:20:50.747+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:20:51.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:20:50 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:20:51.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:20:50 smithi115 ceph-mon[78593]: pgmap v1944: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:20:51.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:20:50 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:20:51.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:20:50 smithi047 ceph-mon[77147]: pgmap v1944: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:20:51.998 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:20:51 smithi115 conmon[90393]: 2022-05-14T00:20:51.718+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:20:52.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:20:51 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:20:52.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:20:51 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:20:53.003 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:20:52 smithi115 conmon[90393]: 2022-05-14T00:20:52.697+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:20:53.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:20:53 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:20:53.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:20:53 smithi047 ceph-mon[77147]: pgmap v1945: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:20:53.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:20:53 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:20:53.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:20:53 smithi115 ceph-mon[78593]: pgmap v1945: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:20:54.003 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:20:53 smithi115 conmon[90393]: 2022-05-14T00:20:53.731+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:20:54.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:20:54 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:20:54.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:20:54 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:20:54.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:20:54 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:20:54.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:20:54 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 3185 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:20:54.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:20:54 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:20:54.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:20:54 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:20:54.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:20:54 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:20:54.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:20:54 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 3185 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:20:55.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:20:54 smithi115 conmon[90393]: 2022-05-14T00:20:54.776+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:20:55.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:20:55 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:20:55.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:20:55 smithi115 ceph-mon[78593]: pgmap v1946: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:20:55.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:20:55 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:20:55.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:20:55 smithi047 ceph-mon[77147]: pgmap v1946: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:20:56.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:20:55 smithi115 conmon[90393]: 2022-05-14T00:20:55.817+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:20:56.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:20:56 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:20:56.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:20:56 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:20:57.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:20:56 smithi115 conmon[90393]: 2022-05-14T00:20:56.773+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:20:57.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:20:57 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:20:57.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:20:57 smithi115 ceph-mon[78593]: pgmap v1947: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:20:57.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:20:57 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:20:57.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:20:57 smithi047 ceph-mon[77147]: pgmap v1947: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:20:58.016 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:20:57 smithi115 conmon[90393]: 2022-05-14T00:20:57.738+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:20:58.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:20:58 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:20:58.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:20:58 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:20:58.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:20:58 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:20:58.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:20:58 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:20:58.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:20:58 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:20:58.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:20:58 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:20:59.020 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:20:58 smithi115 conmon[90393]: 2022-05-14T00:20:58.749+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:20:59.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:20:59 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:20:59.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:20:59 smithi047 ceph-mon[77147]: pgmap v1948: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:20:59.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:20:59 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 3190 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:20:59.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:20:59 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:20:59.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:20:59 smithi115 ceph-mon[78593]: pgmap v1948: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:20:59.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:20:59 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 3190 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:21:00.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:20:59 smithi115 conmon[90393]: 2022-05-14T00:20:59.797+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:21:00.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:21:00 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:21:00.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:21:00 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:21:01.027 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:21:00 smithi115 conmon[90393]: 2022-05-14T00:21:00.774+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:21:01.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:21:01 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:21:01.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:21:01 smithi047 ceph-mon[77147]: pgmap v1949: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:21:01.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:21:01 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:21:01.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:21:01 smithi115 ceph-mon[78593]: pgmap v1949: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:21:02.029 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:21:01 smithi115 conmon[90393]: 2022-05-14T00:21:01.768+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:21:02.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:21:02 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:21:02.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:21:02 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:21:02.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:21:02 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:21:02.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:21:02 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:21:02.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:21:02 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:21:02.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:21:02 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:21:03.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:21:02 smithi115 conmon[90393]: 2022-05-14T00:21:02.802+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:21:03.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:21:03 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:21:03.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:21:03 smithi115 ceph-mon[78593]: pgmap v1950: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:21:03.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:21:03 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:21:03.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:21:03 smithi047 ceph-mon[77147]: pgmap v1950: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:21:04.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:21:03 smithi115 conmon[90393]: 2022-05-14T00:21:03.849+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:21:04.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:21:04 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:21:04.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:21:04 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 3195 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:21:04.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:21:04 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:21:04.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:21:04 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 3195 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:21:05.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:21:05 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:21:05.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:21:05 smithi047 ceph-mon[77147]: pgmap v1951: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:21:05.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:21:04 smithi115 conmon[90393]: 2022-05-14T00:21:04.863+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:21:05.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:21:05 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:21:05.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:21:05 smithi115 ceph-mon[78593]: pgmap v1951: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:21:06.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:21:05 smithi115 conmon[90393]: 2022-05-14T00:21:05.871+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:21:06.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:21:06 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:21:06.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:21:06 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:21:07.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:21:06 smithi115 conmon[90393]: 2022-05-14T00:21:06.888+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:21:07.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:21:07 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:21:07.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:21:07 smithi115 ceph-mon[78593]: pgmap v1952: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:21:07.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:21:07 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:21:07.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:21:07 smithi047 ceph-mon[77147]: pgmap v1952: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:21:08.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:21:07 smithi115 conmon[90393]: 2022-05-14T00:21:07.933+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:21:08.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:21:08 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:21:08.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:21:08 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:21:09.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:21:08 smithi115 conmon[90393]: 2022-05-14T00:21:08.901+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:21:09.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:21:09 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:21:09.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:21:09 smithi115 ceph-mon[78593]: pgmap v1953: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:21:09.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:21:09 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 3200 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:21:09.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:21:09 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:21:09.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:21:09 smithi047 ceph-mon[77147]: pgmap v1953: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:21:09.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:21:09 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 3200 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:21:10.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:21:09 smithi115 conmon[90393]: 2022-05-14T00:21:09.856+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:21:10.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:21:10 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:21:10.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:21:10 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:21:10.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:21:10 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:21:10.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:21:10 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:21:10.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:21:10 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:21:10.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:21:10 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:21:11.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:21:10 smithi115 conmon[90393]: 2022-05-14T00:21:10.859+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:21:11.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:21:11 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:21:11.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:21:11 smithi115 ceph-mon[78593]: pgmap v1954: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:21:11.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:21:11 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:21:11.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:21:11 smithi047 ceph-mon[77147]: pgmap v1954: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:21:12.361 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:21:11 smithi115 conmon[90393]: 2022-05-14T00:21:11.864+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:21:12.362 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:21:12 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:21:12.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:21:12 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:21:13.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:21:12 smithi115 conmon[90393]: 2022-05-14T00:21:12.846+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:21:13.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:21:13 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:21:13.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:21:13 smithi115 ceph-mon[78593]: pgmap v1955: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:21:13.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:21:13 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:21:13.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:21:13 smithi047 ceph-mon[77147]: pgmap v1955: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:21:14.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:21:13 smithi115 conmon[90393]: 2022-05-14T00:21:13.851+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:21:14.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:21:14 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:21:14.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:21:14 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:21:14.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:21:14 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:21:14.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:21:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T00:21:14.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:21:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T00:21:14.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:21:14 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 3205 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:21:14.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:21:14 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:21:14.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:21:14 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:21:14.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:21:14 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:21:14.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:21:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T00:21:14.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:21:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T00:21:14.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:21:14 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 3205 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:21:15.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:21:14 smithi115 conmon[90393]: 2022-05-14T00:21:14.847+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:21:15.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:21:15 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:21:15.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:21:15 smithi115 ceph-mon[78593]: pgmap v1956: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:21:15.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:21:15 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:21:15.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:21:15 smithi047 ceph-mon[77147]: pgmap v1956: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:21:16.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:21:15 smithi115 conmon[90393]: 2022-05-14T00:21:15.880+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:21:16.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:21:16 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:21:16.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:21:16 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:21:17.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:21:16 smithi115 conmon[90393]: 2022-05-14T00:21:16.894+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:21:17.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:21:17 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:21:17.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:21:17 smithi115 ceph-mon[78593]: pgmap v1957: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:21:17.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:21:17 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:21:17.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:21:17 smithi047 ceph-mon[77147]: pgmap v1957: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:21:18.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:21:17 smithi115 conmon[90393]: 2022-05-14T00:21:17.928+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:21:18.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:21:18 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:21:18.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:21:18 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:21:18.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:21:18 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:21:18.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:21:18 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:21:18.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:21:18 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:21:18.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:21:18 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:21:19.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:21:18 smithi115 conmon[90393]: 2022-05-14T00:21:18.903+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:21:19.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:21:19 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:21:19.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:21:19 smithi115 ceph-mon[78593]: pgmap v1958: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:21:19.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:21:19 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 3210 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:21:19.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:21:19 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:21:19.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:21:19 smithi047 ceph-mon[77147]: pgmap v1958: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:21:19.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:21:19 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 3210 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:21:20.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:21:19 smithi115 conmon[90393]: 2022-05-14T00:21:19.932+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:21:20.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:21:20 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:21:20.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:21:20 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:21:21.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:21:20 smithi115 conmon[90393]: 2022-05-14T00:21:20.933+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:21:21.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:21:21 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:21:21.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:21:21 smithi115 ceph-mon[78593]: pgmap v1959: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:21:21.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:21:21 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:21:21.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:21:21 smithi047 ceph-mon[77147]: pgmap v1959: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:21:22.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:21:21 smithi115 conmon[90393]: 2022-05-14T00:21:21.966+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:21:22.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:21:22 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:21:22.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:21:22 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:21:22.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:21:22 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:21:22.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:21:22 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:21:22.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:21:22 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:21:22.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:21:22 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:21:23.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:21:22 smithi115 conmon[90393]: 2022-05-14T00:21:22.999+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:21:23.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:21:23 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:21:23.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:21:23 smithi115 ceph-mon[78593]: pgmap v1960: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:21:23.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:21:23 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:21:23.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:21:23 smithi047 ceph-mon[77147]: pgmap v1960: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:21:24.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:21:23 smithi115 conmon[90393]: 2022-05-14T00:21:23.977+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:21:24.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:21:24 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:21:24.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:21:24 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 3215 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:21:24.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:21:24 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:21:24.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:21:24 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 3215 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:21:25.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:21:24 smithi115 conmon[90393]: 2022-05-14T00:21:24.970+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:21:25.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:21:25 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:21:25.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:21:25 smithi115 ceph-mon[78593]: pgmap v1961: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:21:25.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:21:25 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:21:25.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:21:25 smithi047 ceph-mon[77147]: pgmap v1961: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:21:26.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:21:25 smithi115 conmon[90393]: 2022-05-14T00:21:25.986+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:21:26.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:21:26 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:21:26.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:21:26 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:21:27.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:21:26 smithi115 conmon[90393]: 2022-05-14T00:21:26.979+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:21:27.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:21:27 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:21:27.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:21:27 smithi115 ceph-mon[78593]: pgmap v1962: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:21:27.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:21:27 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:21:27.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:21:27 smithi047 ceph-mon[77147]: pgmap v1962: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:21:28.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:21:27 smithi115 conmon[90393]: 2022-05-14T00:21:27.959+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:21:28.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:21:28 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:21:28.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:21:28 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:21:29.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:21:29 smithi115 conmon[90393]: 2022-05-14T00:21:29.008+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:21:29.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:21:29 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:21:29.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:21:29 smithi115 ceph-mon[78593]: pgmap v1963: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:21:29.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:21:29 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 3220 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:21:29.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:21:29 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:21:29.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:21:29 smithi047 ceph-mon[77147]: pgmap v1963: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:21:29.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:21:29 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 3220 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:21:30.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:21:30 smithi115 conmon[90393]: 2022-05-14T00:21:30.049+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:21:30.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:21:30 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:21:30.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:21:30 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:21:30.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:21:30 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:21:30.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:21:30 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:21:30.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:21:30 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:21:30.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:21:30 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:21:31.347 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:21:31 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:21:31.348 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:21:31 smithi115 ceph-mon[78593]: pgmap v1964: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:21:31.348 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:21:31 smithi115 conmon[90393]: 2022-05-14T00:21:31.074+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:21:31.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:21:31 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:21:31.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:21:31 smithi047 ceph-mon[77147]: pgmap v1964: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:21:32.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:21:32 smithi115 conmon[90393]: 2022-05-14T00:21:32.067+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:21:32.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:21:32 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:21:32.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:21:32 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T00:21:32.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:21:32 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T00:21:32.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:21:32 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T00:21:32.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:21:32 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:21:32.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:21:32 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T00:21:32.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:21:32 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T00:21:32.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:21:32 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T00:21:33.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:21:33 smithi115 conmon[90393]: 2022-05-14T00:21:33.025+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:21:33.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:21:33 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:21:33.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:21:33 smithi115 ceph-mon[78593]: pgmap v1965: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:21:33.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:21:33 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:21:33.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:21:33 smithi047 ceph-mon[77147]: pgmap v1965: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:21:34.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:21:33 smithi115 conmon[90393]: 2022-05-14T00:21:33.988+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:21:34.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:21:34 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:21:34.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:21:34 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:21:34.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:21:34 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:21:34.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:21:34 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 3225 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:21:34.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:21:34 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:21:34.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:21:34 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:21:34.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:21:34 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:21:34.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:21:34 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 3225 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:21:34.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:21:34 smithi047 conmon[78521]: 2022-05-14T00:21:34.319+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:21:34.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:21:34 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T00:21:34.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:21:34 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:21:34.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:21:34 smithi047 conmon[78521]: yield 2022-05-14T00:21:34.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:21:34 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:21:34.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:21:34 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:21:34.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:21:34 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:21:34.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:21:34 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T00:21:34.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:21:34 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:21:34.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:21:34 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T00:21:34.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:21:34 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:21:34.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:21:34 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T00:21:34.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:21:34 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:21:34.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:21:34 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T00:21:34.547 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:21:34 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:21:34.547 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:21:34 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T00:21:34.548 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:21:34 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:21:34.548 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:21:34 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:21:34.548 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:21:34 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:21:35.365 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:21:35 smithi115 conmon[90393]: 2022-05-14T00:21:35.020+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:21:35.366 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:21:35 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:21:35.366 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:21:35 smithi115 ceph-mon[78593]: pgmap v1966: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:21:35.366 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:21:35 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:21:35.367 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T00:21:35.367 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:21:35.367 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T00:21:35.367 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:21:35.368 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:21:35.368 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:21:35.368 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T00:21:35.368 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:21:35.368 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T00:21:35.369 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:21:35.369 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T00:21:35.369 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:21:35.369 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T00:21:35.370 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:21:35.370 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T00:21:35.370 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:21:35.370 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:21:35.370 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:21:35.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:21:35 smithi115 ceph-mon[78593]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T00:21:35.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:21:35 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:21:35.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:21:35 smithi047 ceph-mon[77147]: pgmap v1966: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:21:35.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:21:35 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:21:35.541 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T00:21:35.542 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:21:35.542 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T00:21:35.542 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:21:35.542 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:21:35.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:21:35.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T00:21:35.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:21:35.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T00:21:35.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:21:35.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T00:21:35.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:21:35.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T00:21:35.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:21:35.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T00:21:35.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:21:35.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:21:35.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:21:35.546 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:21:35 smithi047 ceph-mon[77147]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T00:21:36.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:21:35 smithi115 conmon[90393]: 2022-05-14T00:21:35.997+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:21:36.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:21:36 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:21:36.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:21:36 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:21:37.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:21:37 smithi115 conmon[90393]: 2022-05-14T00:21:37.009+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:21:37.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:21:37 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:21:37.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:21:37 smithi115 ceph-mon[78593]: pgmap v1967: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:21:37.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:21:37 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:21:37.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:21:37 smithi047 ceph-mon[77147]: pgmap v1967: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:21:37.542 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:21:37 smithi047 conmon[78521]: 2022-05-14T00:21:37.391+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:21:37.542 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:21:37 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T00:21:37.542 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:21:37 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:21:37.542 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:21:37 smithi047 conmon[78521]: yield 2022-05-14T00:21:37.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:21:37 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:21:37.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:21:37 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:21:37.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:21:37 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:21:37.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:21:37 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T00:21:37.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:21:37 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:21:37.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:21:37 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T00:21:37.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:21:37 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:21:37.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:21:37 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T00:21:37.547 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:21:37 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:21:37.547 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:21:37 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T00:21:37.547 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:21:37 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:21:37.547 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:21:37 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T00:21:37.548 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:21:37 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:21:37.548 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:21:37 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:21:37.548 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:21:37 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:21:37.548 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:21:37 smithi047 conmon[78521]: 2022-05-14T00:21:37.392+0000 7f533ef76700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T00:21:37.549 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:21:37 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T00:21:37.549 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:21:37 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:21:37.550 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:21:37 smithi047 conmon[78521]: yield 2022-05-14T00:21:37.550 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:21:37 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:21:37.551 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:21:37 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:21:37.551 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:21:37 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:21:37.551 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:21:37 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T00:21:37.552 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:21:37 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:21:37.552 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:21:37 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T00:21:37.552 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:21:37 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:21:37.553 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:21:37 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T00:21:37.553 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:21:37 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:21:37.553 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:21:37 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T00:21:37.556 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:21:37 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:21:37.556 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:21:37 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T00:21:37.557 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:21:37 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:21:37.557 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:21:37 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:21:37.557 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:21:37 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:21:37.558 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:21:37 smithi047 conmon[78521]: 2022-05-14T00:21:37.558 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:21:37 smithi047 conmon[78521]: During handling of the above exception, another exception occurred: 2022-05-14T00:21:37.558 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:21:37 smithi047 conmon[78521]: 2022-05-14T00:21:37.559 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:21:37 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T00:21:37.559 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:21:37 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T00:21:37.560 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:21:37 smithi047 conmon[78521]: return f(*arg) 2022-05-14T00:21:37.560 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:21:37 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T00:21:37.561 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:21:37 smithi047 conmon[78521]: self._write_client_files(client_files, host) 2022-05-14T00:21:37.562 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:21:37 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T00:21:37.562 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:21:37 smithi047 conmon[78521]: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T00:21:37.563 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:21:37 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T00:21:37.563 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:21:37 smithi047 conmon[78521]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T00:21:37.564 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:21:37 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T00:21:37.564 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:21:37 smithi047 conmon[78521]: return self.event_loop.get_result(coro) 2022-05-14T00:21:37.564 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:21:37 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T00:21:37.565 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:21:37 smithi047 conmon[78521]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T00:21:37.568 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:21:37 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T00:21:37.568 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:21:37 smithi047 conmon[78521]: return self.__get_result() 2022-05-14T00:21:37.569 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:21:37 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T00:21:37.569 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:21:37 smithi047 conmon[78521]: raise self._exception 2022-05-14T00:21:37.570 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:21:37 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T00:21:37.570 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:21:37 smithi047 conmon[78521]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T00:21:37.571 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:21:37 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T00:21:37.572 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:21:37 smithi047 conmon[78521]: conn = await self._remote_connection(host, addr) 2022-05-14T00:21:37.573 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:21:37 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T00:21:37.573 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:21:37 smithi047 conmon[78521]: raise 2022-05-14T00:21:37.574 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:21:37 smithi047 conmon[78521]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T00:21:37.574 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:21:37 smithi047 conmon[78521]: self.gen.throw(type, value, traceback) 2022-05-14T00:21:37.574 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:21:37 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T00:21:37.575 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:21:37 smithi047 conmon[78521]: raise OrchestratorError(msg) 2022-05-14T00:21:37.575 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:21:37 smithi047 conmon[78521]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:21:38.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:21:38 smithi115 conmon[90393]: 2022-05-14T00:21:38.038+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:21:38.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:21:38 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:21:38.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:21:38 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:21:38.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:21:38 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:21:38.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:21:38 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:21:38.359 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T00:21:38.359 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:21:38.359 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T00:21:38.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:21:38.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:21:38.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:21:38.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T00:21:38.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:21:38.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T00:21:38.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:21:38.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T00:21:38.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:21:38.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T00:21:38.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:21:38.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T00:21:38.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:21:38.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:21:38.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:21:38.363 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:21:38 smithi115 ceph-mon[78593]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T00:21:38.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T00:21:38.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:21:38.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T00:21:38.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:21:38.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:21:38.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:21:38.365 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T00:21:38.365 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:21:38.365 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T00:21:38.366 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:21:38.366 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T00:21:38.366 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:21:38.370 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T00:21:38.371 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:21:38.371 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T00:21:38.372 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:21:38.372 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:21:38.373 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:21:38.373 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T00:21:38.374 INFO:journalctl@ceph.mon.c.smithi115.stdout: During handling of the above exception, another exception occurred: 2022-05-14T00:21:38.374 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T00:21:38.374 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T00:21:38.375 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T00:21:38.375 INFO:journalctl@ceph.mon.c.smithi115.stdout: return f(*arg) 2022-05-14T00:21:38.375 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T00:21:38.376 INFO:journalctl@ceph.mon.c.smithi115.stdout: self._write_client_files(client_files, host) 2022-05-14T00:21:38.376 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T00:21:38.377 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T00:21:38.377 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T00:21:38.377 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T00:21:38.378 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T00:21:38.378 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.event_loop.get_result(coro) 2022-05-14T00:21:38.378 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T00:21:38.378 INFO:journalctl@ceph.mon.c.smithi115.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T00:21:38.379 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T00:21:38.379 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.__get_result() 2022-05-14T00:21:38.379 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T00:21:38.379 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise self._exception 2022-05-14T00:21:38.380 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T00:21:38.380 INFO:journalctl@ceph.mon.c.smithi115.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T00:21:38.380 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T00:21:38.381 INFO:journalctl@ceph.mon.c.smithi115.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T00:21:38.381 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T00:21:38.382 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise 2022-05-14T00:21:38.382 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T00:21:38.382 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.gen.throw(type, value, traceback) 2022-05-14T00:21:38.383 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T00:21:38.383 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OrchestratorError(msg) 2022-05-14T00:21:38.383 INFO:journalctl@ceph.mon.c.smithi115.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:21:38.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:21:38 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:21:38.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:21:38 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:21:38.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:21:38 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:21:38.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:21:38 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:21:38.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T00:21:38.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:21:38.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T00:21:38.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:21:38.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:21:38.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:21:38.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T00:21:38.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:21:38.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T00:21:38.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:21:38.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T00:21:38.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:21:38.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T00:21:38.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:21:38.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T00:21:38.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:21:38.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:21:38.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:21:38.547 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:21:38 smithi047 ceph-mon[77147]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T00:21:38.548 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T00:21:38.548 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:21:38.548 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T00:21:38.548 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:21:38.550 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:21:38.550 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:21:38.550 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T00:21:38.550 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:21:38.551 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T00:21:38.551 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:21:38.551 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T00:21:38.551 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:21:38.552 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T00:21:38.552 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:21:38.552 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T00:21:38.553 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:21:38.553 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:21:38.553 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:21:38.553 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T00:21:38.554 INFO:journalctl@ceph.mon.b.smithi047.stdout: During handling of the above exception, another exception occurred: 2022-05-14T00:21:38.554 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T00:21:38.554 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T00:21:38.554 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T00:21:38.554 INFO:journalctl@ceph.mon.b.smithi047.stdout: return f(*arg) 2022-05-14T00:21:38.555 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T00:21:38.555 INFO:journalctl@ceph.mon.b.smithi047.stdout: self._write_client_files(client_files, host) 2022-05-14T00:21:38.555 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T00:21:38.555 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T00:21:38.556 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T00:21:38.556 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T00:21:38.556 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T00:21:38.556 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.event_loop.get_result(coro) 2022-05-14T00:21:38.556 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T00:21:38.557 INFO:journalctl@ceph.mon.b.smithi047.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T00:21:38.557 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T00:21:38.557 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.__get_result() 2022-05-14T00:21:38.558 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T00:21:38.558 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise self._exception 2022-05-14T00:21:38.559 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T00:21:38.559 INFO:journalctl@ceph.mon.b.smithi047.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T00:21:38.559 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T00:21:38.559 INFO:journalctl@ceph.mon.b.smithi047.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T00:21:38.560 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T00:21:38.560 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise 2022-05-14T00:21:38.560 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T00:21:38.560 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.gen.throw(type, value, traceback) 2022-05-14T00:21:38.560 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T00:21:38.561 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OrchestratorError(msg) 2022-05-14T00:21:38.562 INFO:journalctl@ceph.mon.b.smithi047.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:21:39.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:21:38 smithi115 conmon[90393]: 2022-05-14T00:21:38.991+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:21:39.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:21:39 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:21:39.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:21:39 smithi115 ceph-mon[78593]: pgmap v1968: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:21:39.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:21:39 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 3230 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:21:39.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:21:39 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:21:39.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:21:39 smithi047 ceph-mon[77147]: pgmap v1968: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:21:39.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:21:39 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 3230 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:21:40.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:21:40 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:21:40.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:21:39 smithi115 conmon[90393]: 2022-05-14T00:21:39.983+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:21:40.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:21:40 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:21:41.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:21:40 smithi115 conmon[90393]: 2022-05-14T00:21:40.973+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:21:41.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:21:41 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:21:41.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:21:41 smithi115 ceph-mon[78593]: pgmap v1969: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:21:41.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:21:41 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:21:41.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:21:41 smithi047 ceph-mon[77147]: pgmap v1969: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:21:42.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:21:41 smithi115 conmon[90393]: 2022-05-14T00:21:41.935+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:21:42.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:21:42 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:21:42.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:21:42 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:21:42.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:21:42 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:21:42.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:21:42 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:21:42.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:21:42 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:21:42.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:21:42 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:21:43.166 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:21:42 smithi115 conmon[90393]: 2022-05-14T00:21:42.911+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:21:43.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:21:43 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:21:43.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:21:43 smithi047 ceph-mon[77147]: pgmap v1970: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:21:43.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:21:43 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:21:43.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:21:43 smithi115 ceph-mon[78593]: pgmap v1970: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:21:44.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:21:44 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:21:44.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:21:44 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 3235 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:21:44.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:21:43 smithi115 conmon[90393]: 2022-05-14T00:21:43.929+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:21:44.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:21:44 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:21:44.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:21:44 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 3235 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:21:45.172 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:21:44 smithi115 conmon[90393]: 2022-05-14T00:21:44.897+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:21:45.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:21:45 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:21:45.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:21:45 smithi047 ceph-mon[77147]: pgmap v1971: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:21:45.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:21:45 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:21:45.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:21:45 smithi115 ceph-mon[78593]: pgmap v1971: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:21:46.175 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:21:45 smithi115 conmon[90393]: 2022-05-14T00:21:45.891+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:21:46.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:21:46 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:21:46.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:21:46 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:21:47.179 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:21:46 smithi115 conmon[90393]: 2022-05-14T00:21:46.891+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:21:47.545 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:21:47 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:21:47.546 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:21:47 smithi047 ceph-mon[77147]: pgmap v1972: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:21:47.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:21:47 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:21:47.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:21:47 smithi115 ceph-mon[78593]: pgmap v1972: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:21:48.180 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:21:47 smithi115 conmon[90393]: 2022-05-14T00:21:47.880+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:21:48.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:21:48 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:21:48.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:21:48 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:21:49.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:21:48 smithi115 conmon[90393]: 2022-05-14T00:21:48.831+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:21:49.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:21:49 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:21:49.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:21:49 smithi047 ceph-mon[77147]: pgmap v1973: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:21:49.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:21:49 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 3240 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:21:49.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:21:49 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:21:49.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:21:49 smithi115 ceph-mon[78593]: pgmap v1973: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:21:49.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:21:49 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 3240 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:21:50.186 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:21:49 smithi115 conmon[90393]: 2022-05-14T00:21:49.873+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:21:50.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:21:50 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:21:50.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:21:50 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:21:50.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:21:50 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:21:50.618 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:21:50 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:21:50.619 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:21:50 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:21:50.619 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:21:50 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:21:51.191 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:21:50 smithi115 conmon[90393]: 2022-05-14T00:21:50.919+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:21:51.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:21:51 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:21:51.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:21:51 smithi047 ceph-mon[77147]: pgmap v1974: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:21:51.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:21:51 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:21:51.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:21:51 smithi115 ceph-mon[78593]: pgmap v1974: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:21:52.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:21:52 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:21:52.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:21:51 smithi115 conmon[90393]: 2022-05-14T00:21:51.960+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:21:52.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:21:52 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:21:53.201 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:21:52 smithi115 conmon[90393]: 2022-05-14T00:21:52.922+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:21:53.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:21:53 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:21:53.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:21:53 smithi047 ceph-mon[77147]: pgmap v1975: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:21:53.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:21:53 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:21:53.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:21:53 smithi115 ceph-mon[78593]: pgmap v1975: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:21:54.201 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:21:53 smithi115 conmon[90393]: 2022-05-14T00:21:53.881+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:21:54.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:21:54 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:21:54.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:21:54 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:21:54.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:21:54 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:21:54.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:21:54 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 3245 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:21:54.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:21:54 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:21:54.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:21:54 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:21:54.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:21:54 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:21:54.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:21:54 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 3245 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:21:55.208 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:21:54 smithi115 conmon[90393]: 2022-05-14T00:21:54.866+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:21:55.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:21:55 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:21:55.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:21:55 smithi047 ceph-mon[77147]: pgmap v1976: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:21:55.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:21:55 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:21:55.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:21:55 smithi115 ceph-mon[78593]: pgmap v1976: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:21:56.206 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:21:55 smithi115 conmon[90393]: 2022-05-14T00:21:55.875+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:21:56.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:21:56 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:21:56.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:21:56 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:21:57.214 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:21:56 smithi115 conmon[90393]: 2022-05-14T00:21:56.899+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:21:57.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:21:57 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:21:57.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:21:57 smithi047 ceph-mon[77147]: pgmap v1977: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:21:57.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:21:57 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:21:57.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:21:57 smithi115 ceph-mon[78593]: pgmap v1977: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:21:58.213 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:21:57 smithi115 conmon[90393]: 2022-05-14T00:21:57.904+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:21:58.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:21:58 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:21:58.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:21:58 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:21:58.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:21:58 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:21:58.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:21:58 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:21:58.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:21:58 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:21:58.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:21:58 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:21:59.217 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:21:58 smithi115 conmon[90393]: 2022-05-14T00:21:58.943+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:21:59.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:21:59 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:21:59.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:21:59 smithi047 ceph-mon[77147]: pgmap v1978: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:21:59.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:21:59 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 3250 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:21:59.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:21:59 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:21:59.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:21:59 smithi115 ceph-mon[78593]: pgmap v1978: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:21:59.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:21:59 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 3250 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:22:00.219 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:21:59 smithi115 conmon[90393]: 2022-05-14T00:21:59.924+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:22:00.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:22:00 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:22:00.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:22:00 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:22:01.227 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:22:00 smithi115 conmon[90393]: 2022-05-14T00:22:00.879+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:22:01.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:22:01 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:22:01.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:22:01 smithi047 ceph-mon[77147]: pgmap v1979: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:22:01.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:22:01 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:22:01.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:22:01 smithi115 ceph-mon[78593]: pgmap v1979: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:22:02.225 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:22:01 smithi115 conmon[90393]: 2022-05-14T00:22:01.862+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:22:02.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:22:02 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:22:02.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:22:02 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:22:02.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:22:02 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:22:02.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:22:02 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:22:02.612 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:22:02 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:22:02.614 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:22:02 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:22:03.230 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:22:02 smithi115 conmon[90393]: 2022-05-14T00:22:02.896+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:22:03.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:22:03 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:22:03.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:22:03 smithi047 ceph-mon[77147]: pgmap v1980: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:22:03.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:22:03 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:22:03.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:22:03 smithi115 ceph-mon[78593]: pgmap v1980: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:22:04.231 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:22:03 smithi115 conmon[90393]: 2022-05-14T00:22:03.922+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:22:04.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:22:04 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:22:04.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:22:04 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 3255 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:22:04.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:22:04 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:22:04.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:22:04 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 3255 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:22:05.236 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:22:04 smithi115 conmon[90393]: 2022-05-14T00:22:04.894+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:22:05.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:22:05 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:22:05.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:22:05 smithi047 ceph-mon[77147]: pgmap v1981: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:22:05.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:22:05 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:22:05.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:22:05 smithi115 ceph-mon[78593]: pgmap v1981: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:22:06.242 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:22:05 smithi115 conmon[90393]: 2022-05-14T00:22:05.872+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:22:06.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:22:06 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:22:06.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:22:06 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:22:07.243 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:22:06 smithi115 conmon[90393]: 2022-05-14T00:22:06.881+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:22:07.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:22:07 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:22:07.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:22:07 smithi047 ceph-mon[77147]: pgmap v1982: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:22:07.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:22:07 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:22:07.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:22:07 smithi115 ceph-mon[78593]: pgmap v1982: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:22:08.250 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:22:07 smithi115 conmon[90393]: 2022-05-14T00:22:07.922+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:22:08.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:22:08 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:22:08.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:22:08 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:22:09.249 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:22:08 smithi115 conmon[90393]: 2022-05-14T00:22:08.886+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:22:09.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:22:09 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:22:09.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:22:09 smithi047 ceph-mon[77147]: pgmap v1983: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:22:09.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:22:09 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 3260 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:22:09.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:22:09 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:22:09.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:22:09 smithi115 ceph-mon[78593]: pgmap v1983: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:22:09.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:22:09 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 3260 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:22:10.258 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:22:09 smithi115 conmon[90393]: 2022-05-14T00:22:09.936+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:22:10.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:22:10 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:22:10.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:22:10 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:22:10.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:22:10 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:22:10.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:22:10 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:22:10.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:22:10 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:22:10.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:22:10 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:22:11.260 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:22:10 smithi115 conmon[90393]: 2022-05-14T00:22:10.943+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:22:11.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:22:11 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:22:11.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:22:11 smithi047 ceph-mon[77147]: pgmap v1984: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:22:11.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:22:11 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:22:11.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:22:11 smithi115 ceph-mon[78593]: pgmap v1984: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:22:12.265 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:22:11 smithi115 conmon[90393]: 2022-05-14T00:22:11.963+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:22:12.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:22:12 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:22:12.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:22:12 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:22:13.270 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:22:12 smithi115 conmon[90393]: 2022-05-14T00:22:12.942+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:22:13.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:22:13 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:22:13.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:22:13 smithi047 ceph-mon[77147]: pgmap v1985: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:22:13.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:22:13 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:22:13.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:22:13 smithi115 ceph-mon[78593]: pgmap v1985: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:22:14.273 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:22:13 smithi115 conmon[90393]: 2022-05-14T00:22:13.963+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:22:14.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:22:14 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:22:14.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:22:14 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:22:14.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:22:14 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:22:14.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:22:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T00:22:14.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:22:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T00:22:14.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:22:14 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 3265 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:22:14.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:22:14 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:22:14.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:22:14 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:22:14.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:22:14 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:22:14.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:22:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T00:22:14.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:22:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T00:22:14.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:22:14 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 3265 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:22:15.274 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:22:14 smithi115 conmon[90393]: 2022-05-14T00:22:14.941+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:22:15.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:22:15 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:22:15.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:22:15 smithi047 ceph-mon[77147]: pgmap v1986: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:22:15.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:22:15 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:22:15.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:22:15 smithi115 ceph-mon[78593]: pgmap v1986: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:22:16.282 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:22:15 smithi115 conmon[90393]: 2022-05-14T00:22:15.911+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:22:16.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:22:16 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:22:16.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:22:16 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:22:17.285 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:22:16 smithi115 conmon[90393]: 2022-05-14T00:22:16.944+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:22:17.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:22:17 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:22:17.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:22:17 smithi047 ceph-mon[77147]: pgmap v1987: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:22:17.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:22:17 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:22:17.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:22:17 smithi115 ceph-mon[78593]: pgmap v1987: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:22:18.289 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:22:17 smithi115 conmon[90393]: 2022-05-14T00:22:17.935+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:22:18.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:22:18 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:22:18.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:22:18 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:22:18.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:22:18 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:22:18.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:22:18 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:22:18.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:22:18 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:22:18.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:22:18 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:22:19.291 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:22:18 smithi115 conmon[90393]: 2022-05-14T00:22:18.910+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:22:19.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:22:19 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:22:19.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:22:19 smithi115 ceph-mon[78593]: pgmap v1988: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:22:19.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:22:19 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 3270 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:22:19.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:22:19 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:22:19.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:22:19 smithi047 ceph-mon[77147]: pgmap v1988: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:22:19.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:22:19 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 3270 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:22:20.297 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:22:19 smithi115 conmon[90393]: 2022-05-14T00:22:19.928+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:22:20.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:22:20 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:22:20.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:22:20 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:22:21.298 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:22:20 smithi115 conmon[90393]: 2022-05-14T00:22:20.890+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:22:21.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:22:21 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:22:21.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:22:21 smithi115 ceph-mon[78593]: pgmap v1989: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:22:21.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:22:21 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:22:21.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:22:21 smithi047 ceph-mon[77147]: pgmap v1989: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:22:22.307 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:22:21 smithi115 conmon[90393]: 2022-05-14T00:22:21.893+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:22:22.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:22:22 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:22:22.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:22:22 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:22:22.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:22:22 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:22:22.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:22:22 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:22:22.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:22:22 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:22:22.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:22:22 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:22:23.308 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:22:22 smithi115 conmon[90393]: 2022-05-14T00:22:22.876+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:22:23.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:22:23 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:22:23.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:22:23 smithi115 ceph-mon[78593]: pgmap v1990: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:22:23.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:22:23 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:22:23.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:22:23 smithi047 ceph-mon[77147]: pgmap v1990: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:22:24.310 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:22:23 smithi115 conmon[90393]: 2022-05-14T00:22:23.896+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:22:24.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:22:24 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:22:24.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:22:24 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 3275 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:22:24.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:22:24 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:22:24.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:22:24 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 3275 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:22:25.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:22:24 smithi115 conmon[90393]: 2022-05-14T00:22:24.850+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:22:25.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:22:25 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:22:25.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:22:25 smithi115 ceph-mon[78593]: pgmap v1991: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:22:25.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:22:25 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:22:25.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:22:25 smithi047 ceph-mon[77147]: pgmap v1991: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:22:26.313 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:22:25 smithi115 conmon[90393]: 2022-05-14T00:22:25.895+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:22:26.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:22:26 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:22:26.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:22:26 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:22:27.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:22:26 smithi115 conmon[90393]: 2022-05-14T00:22:26.851+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:22:27.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:22:27 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:22:27.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:22:27 smithi115 ceph-mon[78593]: pgmap v1992: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:22:27.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:22:27 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:22:27.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:22:27 smithi047 ceph-mon[77147]: pgmap v1992: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:22:28.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:22:27 smithi115 conmon[90393]: 2022-05-14T00:22:27.843+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:22:28.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:22:28 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:22:28.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:22:28 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:22:29.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:22:28 smithi115 conmon[90393]: 2022-05-14T00:22:28.795+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:22:29.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:22:29 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:22:29.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:22:29 smithi115 ceph-mon[78593]: pgmap v1993: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:22:29.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:22:29 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 3280 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:22:29.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:22:29 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:22:29.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:22:29 smithi047 ceph-mon[77147]: pgmap v1993: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:22:29.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:22:29 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 3280 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:22:30.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:22:29 smithi115 conmon[90393]: 2022-05-14T00:22:29.756+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:22:30.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:22:30 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:22:30.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:22:30 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:22:30.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:22:30 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:22:30.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:22:30 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:22:30.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:22:30 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:22:30.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:22:30 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:22:31.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:22:30 smithi115 conmon[90393]: 2022-05-14T00:22:30.804+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:22:31.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:22:31 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:22:31.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:22:31 smithi115 ceph-mon[78593]: pgmap v1994: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:22:31.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:22:31 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:22:31.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:22:31 smithi047 ceph-mon[77147]: pgmap v1994: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:22:32.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:22:31 smithi115 conmon[90393]: 2022-05-14T00:22:31.839+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:22:32.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:22:32 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:22:32.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:22:32 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:22:33.343 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:22:32 smithi115 conmon[90393]: 2022-05-14T00:22:32.879+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:22:33.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:22:33 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:22:33.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:22:33 smithi115 ceph-mon[78593]: pgmap v1995: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:22:33.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:22:33 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:22:33.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:22:33 smithi047 ceph-mon[77147]: pgmap v1995: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:22:34.349 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:22:33 smithi115 conmon[90393]: 2022-05-14T00:22:33.904+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:22:34.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:22:34 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:22:34.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:22:34 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:22:34.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:22:34 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:22:34.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:22:34 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 3285 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:22:34.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:22:34 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:22:34.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:22:34 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:22:34.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:22:34 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:22:34.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:22:34 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 3285 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:22:35.349 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:22:34 smithi115 conmon[90393]: 2022-05-14T00:22:34.903+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:22:35.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:22:35 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:22:35.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:22:35 smithi115 ceph-mon[78593]: pgmap v1996: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:22:35.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:22:35 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:22:35.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:22:35 smithi047 ceph-mon[77147]: pgmap v1996: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:22:36.354 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:22:35 smithi115 conmon[90393]: 2022-05-14T00:22:35.920+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:22:36.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:22:36 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:22:36.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:22:36 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:22:37.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:22:36 smithi115 conmon[90393]: 2022-05-14T00:22:36.874+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:22:37.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:22:37 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:22:37.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:22:37 smithi047 ceph-mon[77147]: pgmap v1997: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:22:37.852 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:22:37 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:22:37.852 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:22:37 smithi115 ceph-mon[78593]: pgmap v1997: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:22:38.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:22:37 smithi115 conmon[90393]: 2022-05-14T00:22:37.906+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:22:38.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:22:38 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:22:38.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:22:38 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:22:38.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:22:38 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:22:38.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:22:38 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T00:22:38.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:22:38 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T00:22:38.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:22:38 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T00:22:38.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:22:38 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' 2022-05-14T00:22:38.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:22:38 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' 2022-05-14T00:22:38.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:22:38 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:22:38.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:22:38 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:22:38.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:22:38 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:22:38.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:22:38 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T00:22:38.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:22:38 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T00:22:38.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:22:38 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T00:22:38.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:22:38 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' 2022-05-14T00:22:38.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:22:38 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' 2022-05-14T00:22:39.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:22:38 smithi115 conmon[90393]: 2022-05-14T00:22:38.898+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:22:39.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:22:39 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:22:39.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:22:39 smithi047 ceph-mon[77147]: pgmap v1998: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:22:39.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:22:39 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 3290 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:22:39.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:22:39 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:22:39.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:22:39 smithi115 ceph-mon[78593]: pgmap v1998: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:22:39.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:22:39 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 3290 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:22:40.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:22:39 smithi115 conmon[90393]: 2022-05-14T00:22:39.939+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:22:40.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:22:40 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:22:40.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:22:40 smithi047 conmon[78521]: 2022-05-14T00:22:40.496+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:22:40.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:22:40 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T00:22:40.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:22:40 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:22:40.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:22:40 smithi047 conmon[78521]: yield 2022-05-14T00:22:40.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:22:40 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:22:40.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:22:40 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:22:40.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:22:40 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:22:40.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:22:40 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T00:22:40.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:22:40 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:22:40.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:22:40 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T00:22:40.797 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:22:40 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:22:40.797 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:22:40 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T00:22:40.798 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:22:40 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:22:40.798 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:22:40 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T00:22:40.798 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:22:40 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:22:40.798 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:22:40 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T00:22:40.799 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:22:40 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:22:40.799 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:22:40 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:22:40.799 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:22:40 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:22:40.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:22:40 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:22:41.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:22:40 smithi115 conmon[90393]: 2022-05-14T00:22:40.894+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:22:41.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:22:41 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:22:41.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:22:41 smithi047 ceph-mon[77147]: pgmap v1999: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:22:41.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:22:41 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' 2022-05-14T00:22:41.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:22:41 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' 2022-05-14T00:22:41.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:22:41 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:22:41.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T00:22:41.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:22:41.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T00:22:41.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:22:41.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:22:41.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:22:41.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T00:22:41.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:22:41.797 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T00:22:41.797 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:22:41.797 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T00:22:41.797 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:22:41.798 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T00:22:41.798 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:22:41.798 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T00:22:41.798 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:22:41.798 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:22:41.799 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:22:41.799 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:22:41 smithi047 ceph-mon[77147]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T00:22:41.799 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:22:41 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-05-14T00:22:41.800 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:22:41 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-05-14T00:22:41.800 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:22:41 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-05-14T00:22:41.800 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:22:41 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-05-14T00:22:41.800 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:22:41 smithi047 ceph-mon[77147]: Adjusting osd_memory_target on smithi115 to 197.0M 2022-05-14T00:22:41.801 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:22:41 smithi047 ceph-mon[77147]: Unable to set osd_memory_target on smithi115 to 206629785: error parsing value: Value '206629785' is below minimum 939524096 2022-05-14T00:22:41.801 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:22:41 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-05-14T00:22:41.801 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:22:41 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-05-14T00:22:41.801 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:22:41 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-05-14T00:22:41.801 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:22:41 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-05-14T00:22:41.802 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:22:41 smithi047 ceph-mon[77147]: Adjusting osd_memory_target on smithi047 to 198.1M 2022-05-14T00:22:41.802 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:22:41 smithi047 ceph-mon[77147]: Unable to set osd_memory_target on smithi047 to 207755878: error parsing value: Value '207755878' is below minimum 939524096 2022-05-14T00:22:41.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:22:41 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:22:41.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:22:41 smithi115 ceph-mon[78593]: pgmap v1999: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:22:41.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:22:41 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' 2022-05-14T00:22:41.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:22:41 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' 2022-05-14T00:22:41.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:22:41 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:22:41.859 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T00:22:41.859 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:22:41.860 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T00:22:41.860 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:22:41.860 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:22:41.860 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:22:41.860 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T00:22:41.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:22:41.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T00:22:41.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:22:41.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T00:22:41.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:22:41.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T00:22:41.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:22:41.863 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T00:22:41.863 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:22:41.863 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:22:41.863 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:22:41.863 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:22:41 smithi115 ceph-mon[78593]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T00:22:41.864 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:22:41 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-05-14T00:22:41.864 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:22:41 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-05-14T00:22:41.864 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:22:41 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-05-14T00:22:41.864 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:22:41 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-05-14T00:22:41.865 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:22:41 smithi115 ceph-mon[78593]: Adjusting osd_memory_target on smithi115 to 197.0M 2022-05-14T00:22:41.865 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:22:41 smithi115 ceph-mon[78593]: Unable to set osd_memory_target on smithi115 to 206629785: error parsing value: Value '206629785' is below minimum 939524096 2022-05-14T00:22:41.865 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:22:41 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-05-14T00:22:41.865 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:22:41 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-05-14T00:22:41.866 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:22:41 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-05-14T00:22:41.866 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:22:41 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-05-14T00:22:41.866 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:22:41 smithi115 ceph-mon[78593]: Adjusting osd_memory_target on smithi047 to 198.1M 2022-05-14T00:22:41.866 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:22:41 smithi115 ceph-mon[78593]: Unable to set osd_memory_target on smithi047 to 207755878: error parsing value: Value '207755878' is below minimum 939524096 2022-05-14T00:22:42.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:22:41 smithi115 conmon[90393]: 2022-05-14T00:22:41.882+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:22:42.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:22:42 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:22:42.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:22:42 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:22:42.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:22:42 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:22:42.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:22:42 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:22:42.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:22:42 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:22:42.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:22:42 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:22:43.362 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:22:42 smithi115 conmon[90393]: 2022-05-14T00:22:42.921+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:22:43.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:22:43 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:22:43.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:22:43 smithi047 ceph-mon[77147]: pgmap v2000: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:22:43.793 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:22:43 smithi047 conmon[78521]: 2022-05-14T00:22:43.568+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:22:43.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:22:43 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T00:22:43.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:22:43 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:22:43.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:22:43 smithi047 conmon[78521]: yield 2022-05-14T00:22:43.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:22:43 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:22:43.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:22:43 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:22:43.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:22:43 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:22:43.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:22:43 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T00:22:43.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:22:43 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:22:43.797 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:22:43 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T00:22:43.797 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:22:43 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:22:43.797 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:22:43 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T00:22:43.798 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:22:43 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:22:43.799 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:22:43 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T00:22:43.799 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:22:43 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:22:43.799 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:22:43 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T00:22:43.799 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:22:43 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:22:43.800 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:22:43 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:22:43.800 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:22:43 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:22:43.800 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:22:43 smithi047 conmon[78521]: 2022-05-14T00:22:43.569+0000 7f533f777700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T00:22:43.800 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:22:43 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T00:22:43.801 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:22:43 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:22:43.801 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:22:43 smithi047 conmon[78521]: yield 2022-05-14T00:22:43.801 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:22:43 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:22:43.801 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:22:43 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:22:43.802 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:22:43 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:22:43.802 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:22:43 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T00:22:43.802 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:22:43 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:22:43.802 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:22:43 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T00:22:43.803 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:22:43 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:22:43.803 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:22:43 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T00:22:43.803 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:22:43 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:22:43.803 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:22:43 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T00:22:43.804 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:22:43 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:22:43.804 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:22:43 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T00:22:43.804 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:22:43 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:22:43.804 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:22:43 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:22:43.804 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:22:43 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:22:43.805 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:22:43 smithi047 conmon[78521]: 2022-05-14T00:22:43.806 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:22:43 smithi047 conmon[78521]: During handling of the above exception, another exception occurred: 2022-05-14T00:22:43.806 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:22:43 smithi047 conmon[78521]: 2022-05-14T00:22:43.807 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:22:43 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T00:22:43.807 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:22:43 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T00:22:43.807 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:22:43 smithi047 conmon[78521]: return f(*arg) 2022-05-14T00:22:43.807 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:22:43 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T00:22:43.808 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:22:43 smithi047 conmon[78521]: self._write_client_files(client_files, host) 2022-05-14T00:22:43.808 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:22:43 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T00:22:43.808 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:22:43 smithi047 conmon[78521]: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T00:22:43.811 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:22:43 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T00:22:43.811 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:22:43 smithi047 conmon[78521]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T00:22:43.812 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:22:43 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T00:22:43.812 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:22:43 smithi047 conmon[78521]: return self.event_loop.get_result(coro) 2022-05-14T00:22:43.813 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:22:43 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T00:22:43.813 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:22:43 smithi047 conmon[78521]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T00:22:43.814 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:22:43 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T00:22:43.814 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:22:43 smithi047 conmon[78521]: return self.__get_result() 2022-05-14T00:22:43.814 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:22:43 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T00:22:43.814 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:22:43 smithi047 conmon[78521]: raise self._exception 2022-05-14T00:22:43.815 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:22:43 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T00:22:43.816 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:22:43 smithi047 conmon[78521]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T00:22:43.816 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:22:43 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T00:22:43.817 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:22:43 smithi047 conmon[78521]: conn = await self._remote_connection(host, addr) 2022-05-14T00:22:43.817 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:22:43 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T00:22:43.818 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:22:43 smithi047 conmon[78521]: raise 2022-05-14T00:22:43.818 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:22:43 smithi047 conmon[78521]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T00:22:43.818 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:22:43 smithi047 conmon[78521]: self.gen.throw(type, value, traceback) 2022-05-14T00:22:43.819 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:22:43 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T00:22:43.819 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:22:43 smithi047 conmon[78521]: raise OrchestratorError(msg) 2022-05-14T00:22:43.819 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:22:43 smithi047 conmon[78521]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:22:43.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:22:43 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:22:43.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:22:43 smithi115 ceph-mon[78593]: pgmap v2000: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:22:44.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:22:43 smithi115 conmon[90393]: 2022-05-14T00:22:43.896+0000 7fa085a08700 -1 osd.5 151 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:22:44.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:22:44 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:22:44.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:22:44 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:22:44.792 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T00:22:44.793 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:22:44.793 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T00:22:44.793 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:22:44.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:22:44.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:22:44.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T00:22:44.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:22:44.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T00:22:44.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:22:44.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T00:22:44.797 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:22:44.798 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T00:22:44.798 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:22:44.799 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T00:22:44.799 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:22:44.799 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:22:44.800 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:22:44.801 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:22:44 smithi047 ceph-mon[77147]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T00:22:44.801 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T00:22:44.801 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:22:44.802 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T00:22:44.802 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:22:44.802 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:22:44.803 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:22:44.803 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T00:22:44.803 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:22:44.803 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T00:22:44.804 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:22:44.804 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T00:22:44.804 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:22:44.804 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T00:22:44.805 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:22:44.805 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T00:22:44.805 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:22:44.805 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:22:44.806 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:22:44.806 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T00:22:44.806 INFO:journalctl@ceph.mon.b.smithi047.stdout: During handling of the above exception, another exception occurred: 2022-05-14T00:22:44.806 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T00:22:44.807 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T00:22:44.807 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T00:22:44.807 INFO:journalctl@ceph.mon.b.smithi047.stdout: return f(*arg) 2022-05-14T00:22:44.807 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T00:22:44.808 INFO:journalctl@ceph.mon.b.smithi047.stdout: self._write_client_files(client_files, host) 2022-05-14T00:22:44.808 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T00:22:44.808 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T00:22:44.808 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T00:22:44.808 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T00:22:44.809 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T00:22:44.809 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.event_loop.get_result(coro) 2022-05-14T00:22:44.809 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T00:22:44.809 INFO:journalctl@ceph.mon.b.smithi047.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T00:22:44.810 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T00:22:44.810 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.__get_result() 2022-05-14T00:22:44.810 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T00:22:44.810 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise self._exception 2022-05-14T00:22:44.810 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T00:22:44.811 INFO:journalctl@ceph.mon.b.smithi047.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T00:22:44.811 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T00:22:44.811 INFO:journalctl@ceph.mon.b.smithi047.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T00:22:44.811 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T00:22:44.812 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise 2022-05-14T00:22:44.812 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T00:22:44.812 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.gen.throw(type, value, traceback) 2022-05-14T00:22:44.812 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T00:22:44.813 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OrchestratorError(msg) 2022-05-14T00:22:44.813 INFO:journalctl@ceph.mon.b.smithi047.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:22:44.813 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:22:44 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 3295 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:22:44.813 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:22:44 smithi047 ceph-mon[77147]: osdmap e152: 12 total, 8 up, 8 in 2022-05-14T00:22:44.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:22:44 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:22:44.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:22:44 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:22:44.858 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T00:22:44.859 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:22:44.859 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T00:22:44.859 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:22:44.859 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:22:44.859 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:22:44.860 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T00:22:44.860 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:22:44.860 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T00:22:44.860 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:22:44.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T00:22:44.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:22:44.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T00:22:44.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:22:44.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T00:22:44.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:22:44.863 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:22:44.863 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:22:44.863 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:22:44 smithi115 ceph-mon[78593]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T00:22:44.863 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T00:22:44.863 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:22:44.864 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T00:22:44.864 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:22:44.864 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:22:44.864 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:22:44.865 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T00:22:44.865 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:22:44.865 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T00:22:44.865 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:22:44.866 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T00:22:44.866 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:22:44.866 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T00:22:44.866 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:22:44.866 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T00:22:44.867 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:22:44.867 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:22:44.867 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:22:44.868 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T00:22:44.868 INFO:journalctl@ceph.mon.c.smithi115.stdout: During handling of the above exception, another exception occurred: 2022-05-14T00:22:44.868 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T00:22:44.868 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T00:22:44.868 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T00:22:44.869 INFO:journalctl@ceph.mon.c.smithi115.stdout: return f(*arg) 2022-05-14T00:22:44.869 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T00:22:44.869 INFO:journalctl@ceph.mon.c.smithi115.stdout: self._write_client_files(client_files, host) 2022-05-14T00:22:44.869 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T00:22:44.870 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T00:22:44.870 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T00:22:44.870 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T00:22:44.870 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T00:22:44.870 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.event_loop.get_result(coro) 2022-05-14T00:22:44.871 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T00:22:44.871 INFO:journalctl@ceph.mon.c.smithi115.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T00:22:44.871 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T00:22:44.871 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.__get_result() 2022-05-14T00:22:44.872 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T00:22:44.872 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise self._exception 2022-05-14T00:22:44.872 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T00:22:44.873 INFO:journalctl@ceph.mon.c.smithi115.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T00:22:44.873 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T00:22:44.873 INFO:journalctl@ceph.mon.c.smithi115.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T00:22:44.873 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T00:22:44.874 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise 2022-05-14T00:22:44.874 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T00:22:44.874 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.gen.throw(type, value, traceback) 2022-05-14T00:22:44.874 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T00:22:44.874 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OrchestratorError(msg) 2022-05-14T00:22:44.875 INFO:journalctl@ceph.mon.c.smithi115.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:22:44.875 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:22:44 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 3295 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:22:44.875 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:22:44 smithi115 ceph-mon[78593]: osdmap e152: 12 total, 8 up, 8 in 2022-05-14T00:22:45.361 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:22:44 smithi115 conmon[90393]: 2022-05-14T00:22:44.911+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:22:45.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:22:45 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:22:45.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:22:45 smithi047 ceph-mon[77147]: pgmap v2002: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-05-14T00:22:45.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:22:45 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:22:45.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:22:45 smithi115 ceph-mon[78593]: pgmap v2002: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-05-14T00:22:46.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:22:45 smithi115 conmon[90393]: 2022-05-14T00:22:45.880+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:22:46.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:22:46 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:22:46.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:22:46 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:22:47.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:22:46 smithi115 conmon[90393]: 2022-05-14T00:22:46.925+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:22:47.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:22:47 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:22:47.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:22:47 smithi047 ceph-mon[77147]: pgmap v2003: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-05-14T00:22:47.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:22:47 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:22:47.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:22:47 smithi115 ceph-mon[78593]: pgmap v2003: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-05-14T00:22:48.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:22:47 smithi115 conmon[90393]: 2022-05-14T00:22:47.931+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:22:48.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:22:48 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:22:48.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:22:48 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:22:49.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:22:48 smithi115 conmon[90393]: 2022-05-14T00:22:48.934+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:22:49.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:22:49 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:22:49.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:22:49 smithi047 ceph-mon[77147]: pgmap v2004: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-05-14T00:22:49.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:22:49 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:22:49.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:22:49 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:22:49.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:22:49 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 3300 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:22:49.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:22:49 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:22:49.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:22:49 smithi115 ceph-mon[78593]: pgmap v2004: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-05-14T00:22:49.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:22:49 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:22:49.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:22:49 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:22:49.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:22:49 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 3300 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:22:50.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:22:49 smithi115 conmon[90393]: 2022-05-14T00:22:49.892+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:22:50.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:22:50 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:22:50.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:22:50 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:22:51.363 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:22:50 smithi115 conmon[90393]: 2022-05-14T00:22:50.916+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:22:51.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:22:51 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:22:51.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:22:51 smithi047 ceph-mon[77147]: pgmap v2005: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-05-14T00:22:51.877 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:22:51 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:22:51.878 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:22:51 smithi115 ceph-mon[78593]: pgmap v2005: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-05-14T00:22:52.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:22:51 smithi115 conmon[90393]: 2022-05-14T00:22:51.961+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:22:52.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:22:52 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:22:52.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:22:52 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:22:53.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:22:52 smithi115 conmon[90393]: 2022-05-14T00:22:52.936+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:22:53.800 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:22:53 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:22:53.801 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:22:53 smithi047 ceph-mon[77147]: pgmap v2006: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-05-14T00:22:53.801 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:22:53 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:22:53.801 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:22:53 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:22:53.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:22:53 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:22:53.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:22:53 smithi115 ceph-mon[78593]: pgmap v2006: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2022-05-14T00:22:53.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:22:53 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:22:53.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:22:53 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:22:54.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:22:53 smithi115 conmon[90393]: 2022-05-14T00:22:53.979+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:22:54.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:22:54 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:22:54.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:22:54 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 3305 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:22:54.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:22:54 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:22:54.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:22:54 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 3305 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:22:55.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:22:55 smithi115 conmon[90393]: 2022-05-14T00:22:55.004+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:22:55.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:22:55 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:22:55.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:22:55 smithi047 ceph-mon[77147]: pgmap v2007: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 4.3 KiB/s rd, 8 op/s 2022-05-14T00:22:55.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:22:55 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:22:55.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:22:55 smithi115 ceph-mon[78593]: pgmap v2007: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 4.3 KiB/s rd, 8 op/s 2022-05-14T00:22:56.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:22:56 smithi115 conmon[90393]: 2022-05-14T00:22:56.000+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:22:56.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:22:56 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:22:56.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:22:56 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:22:57.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:22:57 smithi115 conmon[90393]: 2022-05-14T00:22:57.014+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:22:57.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:22:57 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:22:57.800 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:22:57 smithi047 ceph-mon[77147]: pgmap v2008: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:22:57.800 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:22:57 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:22:57.801 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:22:57 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:22:57.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:22:57 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:22:57.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:22:57 smithi115 ceph-mon[78593]: pgmap v2008: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:22:57.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:22:57 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:22:57.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:22:57 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:22:58.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:22:58 smithi115 conmon[90393]: 2022-05-14T00:22:58.010+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:22:58.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:22:58 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:22:58.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:22:58 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:22:59.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:22:59 smithi115 conmon[90393]: 2022-05-14T00:22:59.036+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:22:59.796 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:22:59 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:22:59.797 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:22:59 smithi047 ceph-mon[77147]: pgmap v2009: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:22:59.797 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:22:59 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 3310 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:22:59.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:22:59 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:22:59.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:22:59 smithi115 ceph-mon[78593]: pgmap v2009: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:22:59.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:22:59 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 3310 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:23:00.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:23:00 smithi115 conmon[90393]: 2022-05-14T00:23:00.063+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:23:00.795 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:23:00 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:23:00.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:23:00 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:23:01.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:23:01 smithi115 conmon[90393]: 2022-05-14T00:23:01.095+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:23:01.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:23:01 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:23:01.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:23:01 smithi047 ceph-mon[77147]: pgmap v2010: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:23:01.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:23:01 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:23:01.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:23:01 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:23:01.862 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:23:01 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:23:01.862 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:23:01 smithi115 ceph-mon[78593]: pgmap v2010: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:23:01.863 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:23:01 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:23:01.863 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:23:01 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:23:02.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:23:02 smithi115 conmon[90393]: 2022-05-14T00:23:02.070+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:23:02.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:23:02 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:23:02.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:23:02 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:23:03.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:23:03 smithi115 conmon[90393]: 2022-05-14T00:23:03.032+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:23:03.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:23:03 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:23:03.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:23:03 smithi047 ceph-mon[77147]: pgmap v2011: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:23:03.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:23:03 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:23:03.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:23:03 smithi115 ceph-mon[78593]: pgmap v2011: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:23:04.360 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:23:03 smithi115 conmon[90393]: 2022-05-14T00:23:03.986+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:23:04.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:23:04 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:23:04.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:23:04 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 3315 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:23:04.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:23:04 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:23:04.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:23:04 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 3315 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:23:05.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:23:05 smithi115 conmon[90393]: 2022-05-14T00:23:05.006+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:23:05.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:23:05 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:23:05.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:23:05 smithi047 ceph-mon[77147]: pgmap v2012: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:23:05.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:23:05 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:23:05.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:23:05 smithi115 ceph-mon[78593]: pgmap v2012: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:23:06.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:23:05 smithi115 conmon[90393]: 2022-05-14T00:23:05.997+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:23:06.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:23:06 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:23:06.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:23:06 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:23:07.360 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:23:07 smithi115 conmon[90393]: 2022-05-14T00:23:07.042+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:23:07.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:23:07 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:23:07.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:23:07 smithi047 ceph-mon[77147]: pgmap v2013: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:23:07.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:23:07 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:23:07.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:23:07 smithi115 ceph-mon[78593]: pgmap v2013: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:23:08.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:23:08 smithi115 conmon[90393]: 2022-05-14T00:23:08.057+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:23:08.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:23:08 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:23:09.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:23:08 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:23:09.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:23:09 smithi115 conmon[90393]: 2022-05-14T00:23:09.082+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:23:09.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:23:09 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:23:09.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:23:09 smithi115 ceph-mon[78593]: pgmap v2014: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:23:09.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:23:09 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:23:09.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:23:09 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:23:09.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:23:09 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 3320 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:23:10.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:23:09 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:23:10.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:23:09 smithi047 ceph-mon[77147]: pgmap v2014: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:23:10.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:23:09 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:23:10.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:23:09 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:23:10.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:23:09 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 3320 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:23:10.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:23:10 smithi115 conmon[90393]: 2022-05-14T00:23:10.033+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:23:10.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:23:10 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:23:11.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:23:10 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:23:11.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:23:11 smithi115 conmon[90393]: 2022-05-14T00:23:11.030+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:23:11.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:23:11 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:23:11.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:23:11 smithi115 ceph-mon[78593]: pgmap v2015: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:23:12.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:23:11 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:23:12.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:23:11 smithi047 ceph-mon[77147]: pgmap v2015: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:23:12.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:23:12 smithi115 conmon[90393]: 2022-05-14T00:23:12.038+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:23:12.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:23:12 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:23:13.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:23:12 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:23:13.360 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:23:13 smithi115 conmon[90393]: 2022-05-14T00:23:13.074+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:23:13.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:23:13 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:23:13.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:23:13 smithi115 ceph-mon[78593]: pgmap v2016: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:23:13.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:23:13 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:23:13.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:23:13 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:23:13.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:23:13 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T00:23:13.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:23:13 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T00:23:14.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:23:13 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:23:14.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:23:13 smithi047 ceph-mon[77147]: pgmap v2016: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:23:14.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:23:13 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:23:14.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:23:13 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:23:14.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:23:13 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T00:23:14.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:23:13 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T00:23:14.361 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:23:14 smithi115 conmon[90393]: 2022-05-14T00:23:14.063+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:23:14.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:23:14 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:23:14.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:23:14 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 3324 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:23:15.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:23:14 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:23:15.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:23:14 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 3324 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:23:15.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:23:15 smithi115 conmon[90393]: 2022-05-14T00:23:15.073+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:23:15.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:23:15 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:23:15.863 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:23:15 smithi115 ceph-mon[78593]: pgmap v2017: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:23:16.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:23:15 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:23:16.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:23:15 smithi047 ceph-mon[77147]: pgmap v2017: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:23:16.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:23:16 smithi115 conmon[90393]: 2022-05-14T00:23:16.037+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:23:16.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:23:16 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:23:17.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:23:16 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:23:17.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:23:17 smithi115 conmon[90393]: 2022-05-14T00:23:17.084+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:23:17.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:23:17 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:23:17.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:23:17 smithi115 ceph-mon[78593]: pgmap v2018: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:23:17.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:23:17 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:23:17.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:23:17 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:23:18.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:23:17 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:23:18.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:23:17 smithi047 ceph-mon[77147]: pgmap v2018: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:23:18.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:23:17 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:23:18.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:23:17 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:23:18.574 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:23:18 smithi115 conmon[90393]: 2022-05-14T00:23:18.129+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:23:18.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:23:18 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:23:19.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:23:18 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:23:19.586 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:23:19 smithi115 conmon[90393]: 2022-05-14T00:23:19.158+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:23:19.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:23:19 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:23:19.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:23:19 smithi115 ceph-mon[78593]: pgmap v2019: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:23:19.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:23:19 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 3329 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:23:20.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:23:19 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:23:20.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:23:19 smithi047 ceph-mon[77147]: pgmap v2019: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:23:20.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:23:19 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 3329 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:23:20.581 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:23:20 smithi115 conmon[90393]: 2022-05-14T00:23:20.165+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:23:20.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:23:20 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:23:21.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:23:20 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:23:21.582 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:23:21 smithi115 conmon[90393]: 2022-05-14T00:23:21.154+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:23:21.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:23:21 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:23:21.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:23:21 smithi115 ceph-mon[78593]: pgmap v2020: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:23:21.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:23:21 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:23:21.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:23:21 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:23:22.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:23:21 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:23:22.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:23:21 smithi047 ceph-mon[77147]: pgmap v2020: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:23:22.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:23:21 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:23:22.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:23:21 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:23:22.588 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:23:22 smithi115 conmon[90393]: 2022-05-14T00:23:22.126+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:23:22.862 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:23:22 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:23:23.045 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:23:22 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:23:23.589 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:23:23 smithi115 conmon[90393]: 2022-05-14T00:23:23.154+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:23:23.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:23:23 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:23:23.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:23:23 smithi115 ceph-mon[78593]: pgmap v2021: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:23:24.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:23:23 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:23:24.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:23:23 smithi047 ceph-mon[77147]: pgmap v2021: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:23:24.591 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:23:24 smithi115 conmon[90393]: 2022-05-14T00:23:24.174+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:23:24.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:23:24 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:23:24.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:23:24 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 3335 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:23:25.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:23:24 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:23:25.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:23:24 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 3335 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:23:25.594 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:23:25 smithi115 conmon[90393]: 2022-05-14T00:23:25.218+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:23:25.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:23:25 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:23:25.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:23:25 smithi115 ceph-mon[78593]: pgmap v2022: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:23:26.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:23:25 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:23:26.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:23:25 smithi047 ceph-mon[77147]: pgmap v2022: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:23:26.600 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:23:26 smithi115 conmon[90393]: 2022-05-14T00:23:26.210+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:23:26.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:23:26 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:23:27.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:23:26 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:23:27.598 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:23:27 smithi115 conmon[90393]: 2022-05-14T00:23:27.239+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:23:27.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:23:27 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:23:27.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:23:27 smithi115 ceph-mon[78593]: pgmap v2023: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:23:28.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:23:27 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:23:28.045 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:23:27 smithi047 ceph-mon[77147]: pgmap v2023: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:23:28.605 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:23:28 smithi115 conmon[90393]: 2022-05-14T00:23:28.272+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:23:28.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:23:28 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:23:29.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:23:28 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:23:29.605 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:23:29 smithi115 conmon[90393]: 2022-05-14T00:23:29.283+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:23:29.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:23:29 smithi115 ceph-mon[78593]: pgmap v2024: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:23:29.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:23:29 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:23:29.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:23:29 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:23:29.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:23:29 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:23:29.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:23:29 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 3340 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:23:30.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:23:29 smithi047 ceph-mon[77147]: pgmap v2024: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:23:30.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:23:29 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:23:30.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:23:29 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:23:30.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:23:29 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:23:30.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:23:29 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 3340 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:23:30.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:23:30 smithi115 conmon[90393]: 2022-05-14T00:23:30.236+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:23:31.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:23:30 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:23:31.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:23:30 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:23:31.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:23:31 smithi115 conmon[90393]: 2022-05-14T00:23:31.283+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:23:32.045 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:23:31 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:23:32.045 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:23:31 smithi047 ceph-mon[77147]: pgmap v2025: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:23:32.112 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:23:31 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:23:32.113 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:23:31 smithi115 ceph-mon[78593]: pgmap v2025: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:23:32.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:23:32 smithi115 conmon[90393]: 2022-05-14T00:23:32.308+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:23:33.046 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:23:32 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:23:33.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:23:32 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:23:33.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:23:33 smithi115 conmon[90393]: 2022-05-14T00:23:33.289+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:23:34.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:23:33 smithi047 ceph-mon[77147]: pgmap v2026: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:23:34.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:23:33 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:23:34.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:23:33 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:23:34.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:23:33 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:23:34.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:23:33 smithi115 ceph-mon[78593]: pgmap v2026: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:23:34.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:23:33 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:23:34.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:23:33 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:23:34.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:23:33 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:23:34.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:23:34 smithi115 conmon[90393]: 2022-05-14T00:23:34.310+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:23:35.052 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:23:34 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:23:35.053 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:23:34 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 3345 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:23:35.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:23:34 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:23:35.822 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:23:34 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 3345 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:23:35.823 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:23:35 smithi115 conmon[90393]: 2022-05-14T00:23:35.270+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:23:36.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:23:35 smithi047 ceph-mon[77147]: pgmap v2027: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:23:36.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:23:35 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:23:36.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:23:35 smithi115 ceph-mon[78593]: pgmap v2027: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:23:36.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:23:35 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:23:36.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:23:36 smithi115 conmon[90393]: 2022-05-14T00:23:36.238+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:23:37.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:23:36 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:23:37.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:23:36 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:23:37.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:23:37 smithi115 conmon[90393]: 2022-05-14T00:23:37.272+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:23:38.048 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:23:37 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:23:38.049 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:23:37 smithi047 ceph-mon[77147]: pgmap v2028: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:23:38.049 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:23:37 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:23:38.049 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:23:37 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:23:38.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:23:37 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:23:38.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:23:37 smithi115 ceph-mon[78593]: pgmap v2028: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:23:38.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:23:37 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:23:38.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:23:37 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:23:38.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:23:38 smithi115 conmon[90393]: 2022-05-14T00:23:38.307+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:23:39.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:23:38 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:23:39.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:23:38 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:23:39.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:23:39 smithi115 conmon[90393]: 2022-05-14T00:23:39.315+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:23:40.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:23:39 smithi047 ceph-mon[77147]: pgmap v2029: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:23:40.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:23:39 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:23:40.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:23:39 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 3350 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:23:40.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:23:39 smithi115 ceph-mon[78593]: pgmap v2029: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:23:40.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:23:39 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:23:40.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:23:39 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 3350 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:23:40.610 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:23:40 smithi115 conmon[90393]: 2022-05-14T00:23:40.316+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:23:41.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:23:40 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:23:41.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:23:40 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:23:41.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:23:41 smithi115 conmon[90393]: 2022-05-14T00:23:41.346+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:23:42.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:23:41 smithi047 ceph-mon[77147]: pgmap v2030: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:23:42.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:23:41 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:23:42.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:23:41 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:23:42.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:23:41 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:23:42.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:23:41 smithi115 ceph-mon[78593]: pgmap v2030: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:23:42.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:23:41 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:23:42.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:23:41 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:23:42.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:23:41 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:23:42.645 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:23:42 smithi115 conmon[90393]: 2022-05-14T00:23:42.356+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:23:43.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:23:42 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:23:43.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:23:42 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:23:43.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:23:43 smithi115 conmon[90393]: 2022-05-14T00:23:43.344+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:23:44.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:23:43 smithi047 ceph-mon[77147]: pgmap v2031: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:23:44.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:23:43 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:23:44.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:23:43 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:23:44.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:23:43 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T00:23:44.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:23:43 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T00:23:44.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:23:43 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T00:23:44.075 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:23:43 smithi115 ceph-mon[78593]: pgmap v2031: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:23:44.075 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:23:43 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:23:44.075 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:23:43 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:23:44.075 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:23:43 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T00:23:44.076 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:23:43 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T00:23:44.076 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:23:43 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T00:23:44.541 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:23:44 smithi047 conmon[78521]: 2022-05-14T00:23:44.175+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:23:44.541 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:23:44 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T00:23:44.541 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:23:44 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:23:44.542 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:23:44 smithi047 conmon[78521]: yield 2022-05-14T00:23:44.542 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:23:44 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:23:44.542 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:23:44 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:23:44.542 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:23:44 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:23:44.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:23:44 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T00:23:44.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:23:44 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:23:44.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:23:44 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T00:23:44.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:23:44 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:23:44.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:23:44 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T00:23:44.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:23:44 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:23:44.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:23:44 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T00:23:44.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:23:44 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:23:44.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:23:44 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T00:23:44.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:23:44 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:23:44.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:23:44 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:23:44.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:23:44 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:23:44.654 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:23:44 smithi115 conmon[90393]: 2022-05-14T00:23:44.372+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:23:45.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:23:44 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 3355 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:23:45.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:23:44 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:23:45.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:23:44 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 3355 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:23:45.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:23:44 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:23:45.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:23:45 smithi115 conmon[90393]: 2022-05-14T00:23:45.410+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:23:46.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:23:45 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:23:46.291 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T00:23:46.292 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:23:46.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T00:23:46.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:23:46.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:23:46.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:23:46.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T00:23:46.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:23:46.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T00:23:46.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:23:46.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T00:23:46.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:23:46.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T00:23:46.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:23:46.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T00:23:46.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:23:46.298 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:23:46.298 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:23:46.298 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:23:45 smithi047 ceph-mon[77147]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T00:23:46.299 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:23:45 smithi047 ceph-mon[77147]: pgmap v2032: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:23:46.299 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:23:45 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:23:46.368 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:23:45 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:23:46.369 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T00:23:46.369 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:23:46.370 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T00:23:46.370 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:23:46.371 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:23:46.371 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:23:46.372 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T00:23:46.372 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:23:46.373 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T00:23:46.374 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:23:46.380 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T00:23:46.380 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:23:46.380 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T00:23:46.381 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:23:46.381 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T00:23:46.381 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:23:46.383 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:23:46.383 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:23:46.383 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:23:45 smithi115 ceph-mon[78593]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T00:23:46.384 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:23:45 smithi115 ceph-mon[78593]: pgmap v2032: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:23:46.384 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:23:45 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:23:46.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:23:46 smithi115 conmon[90393]: 2022-05-14T00:23:46.445+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:23:47.249 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:23:46 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:23:47.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:23:46 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:23:47.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:23:47 smithi047 conmon[78521]: 2022-05-14T00:23:47.247+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:23:47.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:23:47 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T00:23:47.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:23:47 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:23:47.547 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:23:47 smithi047 conmon[78521]: yield 2022-05-14T00:23:47.554 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:23:47 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:23:47.554 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:23:47 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:23:47.554 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:23:47 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:23:47.555 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:23:47 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T00:23:47.555 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:23:47 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:23:47.555 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:23:47 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T00:23:47.555 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:23:47 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:23:47.555 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:23:47 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T00:23:47.556 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:23:47 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:23:47.556 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:23:47 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T00:23:47.556 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:23:47 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:23:47.567 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:23:47 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T00:23:47.568 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:23:47 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:23:47.568 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:23:47 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:23:47.569 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:23:47 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:23:47.569 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:23:47 smithi047 conmon[78521]: 2022-05-14T00:23:47.248+0000 7f5344f82700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T00:23:47.570 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:23:47 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T00:23:47.571 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:23:47 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:23:47.571 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:23:47 smithi047 conmon[78521]: yield 2022-05-14T00:23:47.572 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:23:47 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:23:47.572 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:23:47 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:23:47.573 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:23:47 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:23:47.573 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:23:47 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T00:23:47.573 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:23:47 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:23:47.574 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:23:47 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T00:23:47.574 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:23:47 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:23:47.575 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:23:47 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T00:23:47.575 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:23:47 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:23:47.576 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:23:47 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T00:23:47.576 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:23:47 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:23:47.577 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:23:47 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T00:23:47.578 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:23:47 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:23:47.578 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:23:47 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:23:47.579 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:23:47 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:23:47.579 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:23:47 smithi047 conmon[78521]: 2022-05-14T00:23:47.580 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:23:47 smithi047 conmon[78521]: During handling of the above exception, another exception occurred: 2022-05-14T00:23:47.580 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:23:47 smithi047 conmon[78521]: 2022-05-14T00:23:47.581 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:23:47 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T00:23:47.585 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:23:47 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T00:23:47.586 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:23:47 smithi047 conmon[78521]: return f(*arg) 2022-05-14T00:23:47.586 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:23:47 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T00:23:47.586 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:23:47 smithi047 conmon[78521]: self._write_client_files(client_files, host) 2022-05-14T00:23:47.587 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:23:47 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T00:23:47.587 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:23:47 smithi047 conmon[78521]: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T00:23:47.590 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:23:47 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T00:23:47.590 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:23:47 smithi047 conmon[78521]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T00:23:47.591 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:23:47 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T00:23:47.591 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:23:47 smithi047 conmon[78521]: return self.event_loop.get_result(coro) 2022-05-14T00:23:47.591 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:23:47 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T00:23:47.592 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:23:47 smithi047 conmon[78521]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T00:23:47.592 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:23:47 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T00:23:47.592 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:23:47 smithi047 conmon[78521]: return self.__get_result() 2022-05-14T00:23:47.592 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:23:47 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T00:23:47.592 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:23:47 smithi047 conmon[78521]: raise self._exception 2022-05-14T00:23:47.593 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:23:47 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T00:23:47.593 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:23:47 smithi047 conmon[78521]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T00:23:47.593 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:23:47 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T00:23:47.593 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:23:47 smithi047 conmon[78521]: conn = await self._remote_connection(host, addr) 2022-05-14T00:23:47.594 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:23:47 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T00:23:47.594 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:23:47 smithi047 conmon[78521]: raise 2022-05-14T00:23:47.594 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:23:47 smithi047 conmon[78521]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T00:23:47.594 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:23:47 smithi047 conmon[78521]: self.gen.throw(type, value, traceback) 2022-05-14T00:23:47.594 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:23:47 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T00:23:47.596 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:23:47 smithi047 conmon[78521]: raise OrchestratorError(msg) 2022-05-14T00:23:47.596 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:23:47 smithi047 conmon[78521]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:23:47.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:23:47 smithi115 conmon[90393]: 2022-05-14T00:23:47.460+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:23:48.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:23:47 smithi047 ceph-mon[77147]: pgmap v2033: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:23:48.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:23:47 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:23:48.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:23:47 smithi115 ceph-mon[78593]: pgmap v2033: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:23:48.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:23:47 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:23:48.865 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:23:48 smithi115 conmon[90393]: 2022-05-14T00:23:48.478+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:23:49.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:23:48 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:23:49.291 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T00:23:49.291 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:23:49.291 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T00:23:49.292 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:23:49.292 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:23:49.292 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:23:49.292 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T00:23:49.292 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:23:49.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T00:23:49.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:23:49.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T00:23:49.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:23:49.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T00:23:49.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:23:49.298 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T00:23:49.298 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:23:49.298 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:23:49.298 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:23:49.299 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:23:48 smithi047 ceph-mon[77147]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T00:23:49.299 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T00:23:49.299 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:23:49.299 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T00:23:49.300 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:23:49.300 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:23:49.300 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:23:49.300 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T00:23:49.301 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:23:49.301 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T00:23:49.303 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:23:49.303 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T00:23:49.303 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:23:49.303 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T00:23:49.304 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:23:49.304 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T00:23:49.304 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:23:49.305 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:23:49.305 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:23:49.305 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T00:23:49.305 INFO:journalctl@ceph.mon.b.smithi047.stdout: During handling of the above exception, another exception occurred: 2022-05-14T00:23:49.306 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T00:23:49.306 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T00:23:49.306 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T00:23:49.306 INFO:journalctl@ceph.mon.b.smithi047.stdout: return f(*arg) 2022-05-14T00:23:49.307 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T00:23:49.307 INFO:journalctl@ceph.mon.b.smithi047.stdout: self._write_client_files(client_files, host) 2022-05-14T00:23:49.308 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T00:23:49.309 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T00:23:49.309 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T00:23:49.309 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T00:23:49.311 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T00:23:49.311 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.event_loop.get_result(coro) 2022-05-14T00:23:49.312 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T00:23:49.312 INFO:journalctl@ceph.mon.b.smithi047.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T00:23:49.312 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T00:23:49.312 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.__get_result() 2022-05-14T00:23:49.313 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T00:23:49.313 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise self._exception 2022-05-14T00:23:49.313 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T00:23:49.313 INFO:journalctl@ceph.mon.b.smithi047.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T00:23:49.314 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T00:23:49.314 INFO:journalctl@ceph.mon.b.smithi047.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T00:23:49.314 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T00:23:49.314 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise 2022-05-14T00:23:49.315 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T00:23:49.315 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.gen.throw(type, value, traceback) 2022-05-14T00:23:49.315 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T00:23:49.315 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OrchestratorError(msg) 2022-05-14T00:23:49.315 INFO:journalctl@ceph.mon.b.smithi047.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:23:49.316 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:23:48 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:23:49.316 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:23:48 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 3360 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:23:49.363 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:23:48 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:23:49.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T00:23:49.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:23:49.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T00:23:49.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:23:49.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:23:49.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:23:49.365 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T00:23:49.365 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:23:49.375 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T00:23:49.375 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:23:49.375 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T00:23:49.375 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:23:49.376 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T00:23:49.376 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:23:49.376 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T00:23:49.376 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:23:49.377 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:23:49.377 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:23:49.378 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:23:48 smithi115 ceph-mon[78593]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T00:23:49.378 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T00:23:49.379 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:23:49.379 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T00:23:49.380 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:23:49.380 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:23:49.380 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:23:49.380 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T00:23:49.381 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:23:49.381 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T00:23:49.381 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:23:49.381 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T00:23:49.381 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:23:49.382 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T00:23:49.382 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:23:49.382 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T00:23:49.382 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:23:49.383 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:23:49.383 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:23:49.384 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T00:23:49.384 INFO:journalctl@ceph.mon.c.smithi115.stdout: During handling of the above exception, another exception occurred: 2022-05-14T00:23:49.384 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T00:23:49.385 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T00:23:49.385 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T00:23:49.385 INFO:journalctl@ceph.mon.c.smithi115.stdout: return f(*arg) 2022-05-14T00:23:49.386 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T00:23:49.386 INFO:journalctl@ceph.mon.c.smithi115.stdout: self._write_client_files(client_files, host) 2022-05-14T00:23:49.386 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T00:23:49.387 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T00:23:49.387 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T00:23:49.387 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T00:23:49.388 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T00:23:49.388 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.event_loop.get_result(coro) 2022-05-14T00:23:49.388 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T00:23:49.388 INFO:journalctl@ceph.mon.c.smithi115.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T00:23:49.389 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T00:23:49.389 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.__get_result() 2022-05-14T00:23:49.390 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T00:23:49.390 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise self._exception 2022-05-14T00:23:49.390 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T00:23:49.390 INFO:journalctl@ceph.mon.c.smithi115.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T00:23:49.391 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T00:23:49.391 INFO:journalctl@ceph.mon.c.smithi115.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T00:23:49.391 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T00:23:49.391 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise 2022-05-14T00:23:49.391 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T00:23:49.392 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.gen.throw(type, value, traceback) 2022-05-14T00:23:49.392 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T00:23:49.392 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OrchestratorError(msg) 2022-05-14T00:23:49.392 INFO:journalctl@ceph.mon.c.smithi115.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:23:49.392 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:23:48 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:23:49.393 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:23:48 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 3360 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:23:49.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:23:49 smithi115 conmon[90393]: 2022-05-14T00:23:49.429+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:23:50.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:23:49 smithi047 ceph-mon[77147]: pgmap v2034: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:23:50.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:23:49 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:23:50.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:23:49 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:23:50.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:23:49 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:23:50.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:23:49 smithi115 ceph-mon[78593]: pgmap v2034: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:23:50.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:23:49 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:23:50.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:23:49 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:23:50.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:23:49 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:23:50.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:23:50 smithi115 conmon[90393]: 2022-05-14T00:23:50.473+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:23:51.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:23:50 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:23:51.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:23:50 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:23:51.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:23:51 smithi115 conmon[90393]: 2022-05-14T00:23:51.504+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:23:52.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:23:51 smithi047 ceph-mon[77147]: pgmap v2035: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:23:52.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:23:51 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:23:52.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:23:51 smithi115 ceph-mon[78593]: pgmap v2035: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:23:52.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:23:51 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:23:52.862 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:23:52 smithi115 conmon[90393]: 2022-05-14T00:23:52.489+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:23:53.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:23:52 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:23:53.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:23:52 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:23:53.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:23:53 smithi115 conmon[90393]: 2022-05-14T00:23:53.539+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:23:54.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:23:53 smithi047 ceph-mon[77147]: pgmap v2036: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:23:54.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:23:53 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:23:54.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:23:53 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:23:54.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:23:53 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:23:54.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:23:53 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 3365 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:23:54.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:23:53 smithi115 ceph-mon[78593]: pgmap v2036: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:23:54.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:23:53 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:23:54.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:23:53 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:23:54.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:23:53 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:23:54.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:23:53 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 3365 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:23:54.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:23:54 smithi115 conmon[90393]: 2022-05-14T00:23:54.526+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:23:55.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:23:54 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:23:55.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:23:54 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:23:55.863 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:23:55 smithi115 conmon[90393]: 2022-05-14T00:23:55.562+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:23:56.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:23:55 smithi047 ceph-mon[77147]: pgmap v2037: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:23:56.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:23:55 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:23:56.366 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:23:55 smithi115 ceph-mon[78593]: pgmap v2037: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:23:56.366 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:23:55 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:23:56.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:23:56 smithi115 conmon[90393]: 2022-05-14T00:23:56.594+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:23:57.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:23:56 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:23:57.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:23:56 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:23:57.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:23:57 smithi115 conmon[90393]: 2022-05-14T00:23:57.586+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:23:58.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:23:57 smithi047 ceph-mon[77147]: pgmap v2038: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:23:58.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:23:57 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:23:58.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:23:57 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:23:58.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:23:57 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:23:58.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:23:57 smithi115 ceph-mon[78593]: pgmap v2038: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:23:58.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:23:57 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:23:58.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:23:57 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:23:58.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:23:57 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:23:58.934 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:23:58 smithi115 conmon[90393]: 2022-05-14T00:23:58.610+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:23:59.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:23:58 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:23:59.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:23:58 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 3370 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:23:59.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:23:58 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:23:59.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:23:58 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 3370 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:23:59.927 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:23:59 smithi115 conmon[90393]: 2022-05-14T00:23:59.660+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:24:00.295 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:23:59 smithi047 ceph-mon[77147]: pgmap v2039: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:24:00.365 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:23:59 smithi115 ceph-mon[78593]: pgmap v2039: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:24:00.932 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:24:00 smithi115 conmon[90393]: 2022-05-14T00:24:00.621+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:24:00.932 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:24:00 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:24:01.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:24:00 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:24:01.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:24:00 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:24:01.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:24:00 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:24:01.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:24:01 smithi115 conmon[90393]: 2022-05-14T00:24:01.584+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:24:02.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:24:01 smithi047 ceph-mon[77147]: pgmap v2040: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:24:02.295 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:24:01 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:24:02.295 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:24:01 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:24:02.295 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:24:01 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:24:02.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:24:01 smithi115 ceph-mon[78593]: pgmap v2040: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:24:02.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:24:01 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:24:02.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:24:01 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:24:02.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:24:01 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:24:02.864 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:24:02 smithi115 conmon[90393]: 2022-05-14T00:24:02.545+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:24:03.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:24:02 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:24:03.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:24:02 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:24:03.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:24:03 smithi115 conmon[90393]: 2022-05-14T00:24:03.532+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:24:04.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:24:03 smithi047 ceph-mon[77147]: pgmap v2041: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:24:04.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:24:03 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:24:04.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:24:03 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 3375 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:24:04.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:24:03 smithi115 ceph-mon[78593]: pgmap v2041: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:24:04.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:24:03 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:24:04.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:24:03 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 3375 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:24:04.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:24:04 smithi115 conmon[90393]: 2022-05-14T00:24:04.558+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:24:05.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:24:04 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:24:05.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:24:04 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:24:05.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:24:05 smithi115 conmon[90393]: 2022-05-14T00:24:05.518+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:24:06.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:24:05 smithi047 ceph-mon[77147]: pgmap v2042: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:24:06.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:24:05 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:24:06.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:24:05 smithi115 ceph-mon[78593]: pgmap v2042: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:24:06.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:24:05 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:24:06.861 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:24:06 smithi115 conmon[90393]: 2022-05-14T00:24:06.495+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:24:07.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:24:06 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:24:07.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:24:06 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:24:07.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:24:07 smithi115 conmon[90393]: 2022-05-14T00:24:07.541+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:24:08.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:24:07 smithi047 ceph-mon[77147]: pgmap v2043: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:24:08.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:24:07 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:24:08.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:24:07 smithi115 ceph-mon[78593]: pgmap v2043: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:24:08.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:24:07 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:24:08.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:24:08 smithi115 conmon[90393]: 2022-05-14T00:24:08.585+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:24:09.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:24:08 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:24:09.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:24:08 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 3380 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:24:09.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:24:08 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:24:09.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:24:08 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 3380 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:24:09.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:24:09 smithi115 conmon[90393]: 2022-05-14T00:24:09.566+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:24:10.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:24:09 smithi047 ceph-mon[77147]: pgmap v2044: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:24:10.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:24:09 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:24:10.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:24:09 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:24:10.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:24:09 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:24:10.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:24:09 smithi115 ceph-mon[78593]: pgmap v2044: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:24:10.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:24:09 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:24:10.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:24:09 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:24:10.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:24:09 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:24:10.863 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:24:10 smithi115 conmon[90393]: 2022-05-14T00:24:10.533+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:24:11.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:24:10 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:24:11.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:24:10 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:24:11.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:24:11 smithi115 conmon[90393]: 2022-05-14T00:24:11.514+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:24:12.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:24:11 smithi047 ceph-mon[77147]: pgmap v2045: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:24:12.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:24:11 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:24:12.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:24:11 smithi115 ceph-mon[78593]: pgmap v2045: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:24:12.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:24:11 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:24:12.862 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:24:12 smithi115 conmon[90393]: 2022-05-14T00:24:12.564+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:24:13.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:24:12 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:24:13.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:24:12 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:24:13.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:24:13 smithi115 conmon[90393]: 2022-05-14T00:24:13.555+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:24:14.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:24:13 smithi047 ceph-mon[77147]: pgmap v2046: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:24:14.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:24:13 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:24:14.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:24:13 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:24:14.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:24:13 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T00:24:14.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:24:13 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T00:24:14.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:24:13 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:24:14.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:24:13 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 3385 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:24:14.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:24:13 smithi115 ceph-mon[78593]: pgmap v2046: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:24:14.362 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:24:13 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:24:14.362 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:24:13 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:24:14.362 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:24:13 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T00:24:14.363 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:24:13 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T00:24:14.363 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:24:13 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:24:14.363 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:24:13 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 3385 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:24:14.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:24:14 smithi115 conmon[90393]: 2022-05-14T00:24:14.525+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:24:15.302 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:24:14 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:24:15.362 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:24:14 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:24:15.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:24:15 smithi115 conmon[90393]: 2022-05-14T00:24:15.553+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:24:16.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:24:15 smithi047 ceph-mon[77147]: pgmap v2047: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:24:16.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:24:15 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:24:16.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:24:15 smithi115 ceph-mon[78593]: pgmap v2047: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:24:16.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:24:15 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:24:16.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:24:16 smithi115 conmon[90393]: 2022-05-14T00:24:16.569+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:24:17.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:24:16 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:24:17.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:24:16 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:24:17.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:24:17 smithi115 conmon[90393]: 2022-05-14T00:24:17.561+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:24:18.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:24:17 smithi047 ceph-mon[77147]: pgmap v2048: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:24:18.298 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:24:17 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:24:18.298 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:24:17 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:24:18.298 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:24:17 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:24:18.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:24:17 smithi115 ceph-mon[78593]: pgmap v2048: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:24:18.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:24:17 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:24:18.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:24:17 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:24:18.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:24:17 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:24:18.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:24:18 smithi115 conmon[90393]: 2022-05-14T00:24:18.571+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:24:19.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:24:18 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:24:19.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:24:18 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 3390 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:24:19.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:24:18 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:24:19.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:24:18 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 3390 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:24:19.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:24:19 smithi115 conmon[90393]: 2022-05-14T00:24:19.596+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:24:20.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:24:19 smithi047 ceph-mon[77147]: pgmap v2049: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:24:20.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:24:19 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:24:20.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:24:19 smithi115 ceph-mon[78593]: pgmap v2049: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:24:20.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:24:19 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:24:20.998 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:24:20 smithi115 conmon[90393]: 2022-05-14T00:24:20.635+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:24:21.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:24:20 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:24:21.362 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:24:20 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:24:22.005 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:24:21 smithi115 conmon[90393]: 2022-05-14T00:24:21.681+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:24:22.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:24:21 smithi047 ceph-mon[77147]: pgmap v2050: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:24:22.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:24:21 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:24:22.295 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:24:21 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:24:22.362 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:24:21 smithi115 ceph-mon[78593]: pgmap v2050: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:24:22.363 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:24:21 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:24:22.363 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:24:21 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:24:23.004 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:24:22 smithi115 conmon[90393]: 2022-05-14T00:24:22.652+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:24:23.295 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:24:23 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:24:23.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:24:23 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:24:24.006 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:24:23 smithi115 conmon[90393]: 2022-05-14T00:24:23.620+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:24:24.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:24:24 smithi047 ceph-mon[77147]: pgmap v2051: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:24:24.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:24:24 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:24:24.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:24:24 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:24:24.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:24:24 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 3395 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:24:24.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:24:24 smithi115 ceph-mon[78593]: pgmap v2051: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:24:24.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:24:24 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:24:24.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:24:24 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:24:24.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:24:24 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 3395 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:24:24.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:24:24 smithi115 conmon[90393]: 2022-05-14T00:24:24.588+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:24:25.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:24:25 smithi047 ceph-mon[77147]: pgmap v2052: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:24:25.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:24:25 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:24:25.362 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:24:25 smithi115 ceph-mon[78593]: pgmap v2052: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:24:25.373 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:24:25 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:24:26.015 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:24:25 smithi115 conmon[90393]: 2022-05-14T00:24:25.611+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:24:26.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:24:26 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:24:26.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:24:26 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:24:26.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:24:26 smithi115 conmon[90393]: 2022-05-14T00:24:26.603+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:24:27.297 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:24:27 smithi047 ceph-mon[77147]: pgmap v2053: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:24:27.297 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:24:27 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:24:27.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:24:27 smithi115 ceph-mon[78593]: pgmap v2053: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:24:27.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:24:27 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:24:28.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:24:27 smithi115 conmon[90393]: 2022-05-14T00:24:27.650+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:24:29.110 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:24:28 smithi115 conmon[90393]: 2022-05-14T00:24:28.652+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:24:29.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:24:29 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:24:29.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:24:29 smithi115 ceph-mon[78593]: pgmap v2054: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:24:29.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:24:29 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 3400 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:24:29.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:24:29 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:24:29.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:24:29 smithi047 ceph-mon[77147]: pgmap v2054: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:24:29.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:24:29 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 3400 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:24:30.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:24:29 smithi115 conmon[90393]: 2022-05-14T00:24:29.623+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:24:30.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:24:30 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:24:30.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:24:30 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:24:30.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:24:30 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:24:30.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:24:30 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:24:30.818 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:24:30 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:24:30.819 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:24:30 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:24:30.819 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:24:30 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:24:30.819 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:24:30 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:24:31.114 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:24:30 smithi115 conmon[90393]: 2022-05-14T00:24:30.651+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:24:31.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:24:31 smithi115 ceph-mon[78593]: pgmap v2055: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:24:31.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:24:31 smithi047 ceph-mon[77147]: pgmap v2055: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:24:32.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:24:31 smithi115 conmon[90393]: 2022-05-14T00:24:31.656+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:24:32.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:24:32 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:24:32.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:24:32 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:24:33.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:24:32 smithi115 conmon[90393]: 2022-05-14T00:24:32.706+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:24:33.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:24:33 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:24:33.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:24:33 smithi115 ceph-mon[78593]: pgmap v2056: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:24:33.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:24:33 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:24:33.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:24:33 smithi047 ceph-mon[77147]: pgmap v2056: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:24:34.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:24:33 smithi115 conmon[90393]: 2022-05-14T00:24:33.707+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:24:34.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:24:34 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:24:34.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:24:34 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:24:34.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:24:34 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:24:34.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:24:34 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 3405 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:24:34.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:24:34 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:24:34.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:24:34 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:24:34.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:24:34 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:24:34.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:24:34 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 3405 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:24:35.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:24:34 smithi115 conmon[90393]: 2022-05-14T00:24:34.719+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:24:35.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:24:35 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:24:35.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:24:35 smithi115 ceph-mon[78593]: pgmap v2057: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:24:35.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:24:35 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:24:35.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:24:35 smithi047 ceph-mon[77147]: pgmap v2057: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:24:36.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:24:35 smithi115 conmon[90393]: 2022-05-14T00:24:35.677+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:24:36.677 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:24:36 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:24:36.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:24:36 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:24:37.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:24:36 smithi115 conmon[90393]: 2022-05-14T00:24:36.676+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:24:37.690 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:24:37 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:24:37.690 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:24:37 smithi115 ceph-mon[78593]: pgmap v2058: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:24:37.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:24:37 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:24:37.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:24:37 smithi047 ceph-mon[77147]: pgmap v2058: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:24:38.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:24:37 smithi115 conmon[90393]: 2022-05-14T00:24:37.689+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:24:38.712 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:24:38 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:24:38.712 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:24:38 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:24:38.713 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:24:38 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:24:38.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:24:38 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:24:38.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:24:38 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:24:38.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:24:38 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:24:39.114 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:24:38 smithi115 conmon[90393]: 2022-05-14T00:24:38.711+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:24:39.723 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:24:39 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:24:39.723 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:24:39 smithi115 ceph-mon[78593]: pgmap v2059: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:24:39.724 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:24:39 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 3410 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:24:39.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:24:39 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:24:39.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:24:39 smithi047 ceph-mon[77147]: pgmap v2059: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:24:39.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:24:39 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 3410 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:24:40.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:24:39 smithi115 conmon[90393]: 2022-05-14T00:24:39.721+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:24:40.740 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:24:40 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:24:40.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:24:40 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:24:41.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:24:40 smithi115 conmon[90393]: 2022-05-14T00:24:40.740+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:24:41.740 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:24:41 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:24:41.740 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:24:41 smithi115 ceph-mon[78593]: pgmap v2060: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:24:41.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:24:41 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:24:41.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:24:41 smithi047 ceph-mon[77147]: pgmap v2060: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:24:42.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:24:41 smithi115 conmon[90393]: 2022-05-14T00:24:41.739+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:24:42.776 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:24:42 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:24:42.777 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:24:42 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:24:42.777 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:24:42 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:24:42.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:24:42 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:24:42.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:24:42 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:24:42.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:24:42 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:24:43.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:24:42 smithi115 conmon[90393]: 2022-05-14T00:24:42.775+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:24:43.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:24:43 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:24:43.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:24:43 smithi047 ceph-mon[77147]: pgmap v2061: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:24:43.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:24:43 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:24:43.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:24:43 smithi115 ceph-mon[78593]: pgmap v2061: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:24:44.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:24:43 smithi115 conmon[90393]: 2022-05-14T00:24:43.791+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:24:44.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:24:44 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:24:44.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:24:44 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 3415 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:24:44.846 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:24:44 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:24:44.846 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:24:44 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 3415 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:24:45.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:24:44 smithi115 conmon[90393]: 2022-05-14T00:24:44.831+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:24:45.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:24:45 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:24:45.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:24:45 smithi047 ceph-mon[77147]: pgmap v2062: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:24:45.841 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:24:45 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:24:45.841 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:24:45 smithi115 ceph-mon[78593]: pgmap v2062: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:24:46.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:24:45 smithi115 conmon[90393]: 2022-05-14T00:24:45.836+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:24:46.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:24:46 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:24:46.818 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:24:46 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:24:47.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:24:46 smithi115 conmon[90393]: 2022-05-14T00:24:46.817+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:24:47.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:24:47 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:24:47.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:24:47 smithi115 ceph-mon[78593]: pgmap v2063: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:24:47.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:24:47 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T00:24:47.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:24:47 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T00:24:47.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:24:47 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T00:24:47.770 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:24:47 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:24:47.770 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:24:47 smithi047 ceph-mon[77147]: pgmap v2063: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:24:47.771 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:24:47 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T00:24:47.771 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:24:47 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T00:24:47.771 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:24:47 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T00:24:48.110 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:24:47 smithi115 conmon[90393]: 2022-05-14T00:24:47.821+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:24:48.790 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:24:48 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:24:48.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:24:48 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:24:49.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:24:48 smithi115 conmon[90393]: 2022-05-14T00:24:48.790+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:24:49.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:24:49 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:24:49.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:24:49 smithi047 ceph-mon[77147]: pgmap v2064: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:24:49.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:24:49 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 3420 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:24:49.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:24:49 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:24:49.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:24:49 smithi115 ceph-mon[78593]: pgmap v2064: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:24:49.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:24:49 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 3420 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:24:50.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:24:49 smithi115 conmon[90393]: 2022-05-14T00:24:49.794+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:24:50.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:24:50 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:24:50.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:24:50 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:24:50.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:24:50 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:24:50.793 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:24:50 smithi047 conmon[78521]: 2022-05-14T00:24:50.351+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:24:50.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:24:50 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T00:24:50.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:24:50 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:24:50.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:24:50 smithi047 conmon[78521]: yield 2022-05-14T00:24:50.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:24:50 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:24:50.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:24:50 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:24:50.798 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:24:50 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:24:50.798 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:24:50 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T00:24:50.798 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:24:50 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:24:50.800 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:24:50 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T00:24:50.801 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:24:50 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:24:50.801 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:24:50 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T00:24:50.801 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:24:50 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:24:50.802 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:24:50 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T00:24:50.802 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:24:50 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:24:50.803 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:24:50 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T00:24:50.804 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:24:50 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:24:50.804 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:24:50 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:24:50.806 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:24:50 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:24:50.812 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:24:50 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:24:50.813 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:24:50 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:24:50.819 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:24:50 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:24:51.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:24:50 smithi115 conmon[90393]: 2022-05-14T00:24:50.810+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:24:51.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:24:51 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:24:51.795 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:24:51 smithi047 ceph-mon[77147]: pgmap v2065: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:24:51.795 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:24:51 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:24:51.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T00:24:51.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:24:51.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T00:24:51.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:24:51.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:24:51.797 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:24:51.797 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T00:24:51.797 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:24:51.797 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T00:24:51.798 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:24:51.798 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T00:24:51.798 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:24:51.802 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T00:24:51.803 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:24:51.804 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T00:24:51.804 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:24:51.805 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:24:51.805 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:24:51.806 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:24:51 smithi047 ceph-mon[77147]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T00:24:51.838 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:24:51 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:24:51.838 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:24:51 smithi115 ceph-mon[78593]: pgmap v2065: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:24:51.839 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:24:51 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:24:51.839 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T00:24:51.839 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:24:51.840 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T00:24:51.840 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:24:51.840 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:24:51.840 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:24:51.841 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T00:24:51.841 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:24:51.842 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T00:24:51.842 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:24:51.842 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T00:24:51.842 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:24:51.842 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T00:24:51.843 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:24:51.843 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T00:24:51.843 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:24:51.843 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:24:51.843 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:24:51.844 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:24:51 smithi115 ceph-mon[78593]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T00:24:52.110 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:24:51 smithi115 conmon[90393]: 2022-05-14T00:24:51.831+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:24:52.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:24:52 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:24:52.814 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:24:52 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:24:53.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:24:52 smithi115 conmon[90393]: 2022-05-14T00:24:52.812+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:24:53.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:24:53 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:24:53.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:24:53 smithi115 ceph-mon[78593]: pgmap v2066: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:24:53.796 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:24:53 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:24:53.797 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:24:53 smithi047 ceph-mon[77147]: pgmap v2066: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:24:53.798 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:24:53 smithi047 conmon[78521]: 2022-05-14T00:24:53.423+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:24:53.800 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:24:53 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T00:24:53.800 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:24:53 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:24:53.801 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:24:53 smithi047 conmon[78521]: yield 2022-05-14T00:24:53.801 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:24:53 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:24:53.802 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:24:53 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:24:53.802 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:24:53 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:24:53.802 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:24:53 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T00:24:53.803 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:24:53 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:24:53.806 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:24:53 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T00:24:53.806 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:24:53 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:24:53.807 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:24:53 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T00:24:53.807 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:24:53 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:24:53.807 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:24:53 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T00:24:53.808 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:24:53 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:24:53.809 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:24:53 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T00:24:53.812 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:24:53 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:24:53.813 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:24:53 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:24:53.814 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:24:53 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:24:53.814 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:24:53 smithi047 conmon[78521]: 2022-05-14T00:24:53.424+0000 7f5344781700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T00:24:53.814 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:24:53 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T00:24:53.816 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:24:53 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:24:53.816 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:24:53 smithi047 conmon[78521]: yield 2022-05-14T00:24:53.817 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:24:53 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:24:53.818 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:24:53 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:24:53.818 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:24:53 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:24:53.819 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:24:53 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T00:24:53.819 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:24:53 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:24:53.819 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:24:53 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T00:24:53.820 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:24:53 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:24:53.820 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:24:53 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T00:24:53.820 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:24:53 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:24:53.821 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:24:53 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T00:24:53.821 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:24:53 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:24:53.822 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:24:53 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T00:24:53.822 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:24:53 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:24:53.822 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:24:53 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:24:53.823 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:24:53 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:24:53.823 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:24:53 smithi047 conmon[78521]: 2022-05-14T00:24:53.823 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:24:53 smithi047 conmon[78521]: During handling of the above exception, another exception occurred: 2022-05-14T00:24:53.823 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:24:53 smithi047 conmon[78521]: 2022-05-14T00:24:53.824 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:24:53 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T00:24:53.824 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:24:53 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T00:24:53.824 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:24:53 smithi047 conmon[78521]: return f(*arg) 2022-05-14T00:24:53.825 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:24:53 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T00:24:53.825 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:24:53 smithi047 conmon[78521]: self._write_client_files(client_files, host) 2022-05-14T00:24:53.825 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:24:53 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T00:24:53.826 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:24:53 smithi047 conmon[78521]: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T00:24:53.826 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:24:53 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T00:24:53.826 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:24:53 smithi047 conmon[78521]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T00:24:53.826 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:24:53 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T00:24:53.827 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:24:53 smithi047 conmon[78521]: return self.event_loop.get_result(coro) 2022-05-14T00:24:53.827 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:24:53 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T00:24:53.827 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:24:53 smithi047 conmon[78521]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T00:24:53.827 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:24:53 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T00:24:53.827 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:24:53 smithi047 conmon[78521]: return self.__get_result() 2022-05-14T00:24:53.828 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:24:53 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T00:24:53.831 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:24:53 smithi047 conmon[78521]: raise self._exception 2022-05-14T00:24:53.831 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:24:53 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T00:24:53.831 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:24:53 smithi047 conmon[78521]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T00:24:53.832 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:24:53 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T00:24:53.832 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:24:53 smithi047 conmon[78521]: conn = await self._remote_connection(host, addr) 2022-05-14T00:24:53.833 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:24:53 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T00:24:53.833 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:24:53 smithi047 conmon[78521]: raise 2022-05-14T00:24:53.833 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:24:53 smithi047 conmon[78521]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T00:24:53.834 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:24:53 smithi047 conmon[78521]: self.gen.throw(type, value, traceback) 2022-05-14T00:24:53.834 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:24:53 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T00:24:53.835 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:24:53 smithi047 conmon[78521]: raise OrchestratorError(msg) 2022-05-14T00:24:53.835 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:24:53 smithi047 conmon[78521]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:24:54.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:24:53 smithi115 conmon[90393]: 2022-05-14T00:24:53.794+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:24:54.747 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:24:54 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:24:54.748 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:24:54 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:24:54.748 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T00:24:54.749 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:24:54.749 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T00:24:54.749 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:24:54.750 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:24:54.750 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:24:54.750 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T00:24:54.750 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:24:54.751 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T00:24:54.751 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:24:54.751 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T00:24:54.751 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:24:54.752 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T00:24:54.753 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:24:54.754 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T00:24:54.757 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:24:54.757 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:24:54.758 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:24:54.758 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:24:54 smithi115 ceph-mon[78593]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T00:24:54.758 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T00:24:54.758 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:24:54.759 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T00:24:54.759 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:24:54.759 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:24:54.759 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:24:54.760 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T00:24:54.760 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:24:54.760 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T00:24:54.761 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:24:54.761 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T00:24:54.761 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:24:54.761 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T00:24:54.762 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:24:54.762 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T00:24:54.762 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:24:54.762 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:24:54.762 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:24:54.763 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T00:24:54.763 INFO:journalctl@ceph.mon.c.smithi115.stdout: During handling of the above exception, another exception occurred: 2022-05-14T00:24:54.763 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T00:24:54.763 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T00:24:54.763 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T00:24:54.764 INFO:journalctl@ceph.mon.c.smithi115.stdout: return f(*arg) 2022-05-14T00:24:54.764 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T00:24:54.764 INFO:journalctl@ceph.mon.c.smithi115.stdout: self._write_client_files(client_files, host) 2022-05-14T00:24:54.764 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T00:24:54.765 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T00:24:54.765 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T00:24:54.765 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T00:24:54.766 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T00:24:54.766 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.event_loop.get_result(coro) 2022-05-14T00:24:54.766 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T00:24:54.766 INFO:journalctl@ceph.mon.c.smithi115.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T00:24:54.767 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T00:24:54.767 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.__get_result() 2022-05-14T00:24:54.767 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T00:24:54.767 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise self._exception 2022-05-14T00:24:54.768 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T00:24:54.768 INFO:journalctl@ceph.mon.c.smithi115.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T00:24:54.768 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T00:24:54.769 INFO:journalctl@ceph.mon.c.smithi115.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T00:24:54.770 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T00:24:54.770 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise 2022-05-14T00:24:54.771 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T00:24:54.771 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.gen.throw(type, value, traceback) 2022-05-14T00:24:54.771 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T00:24:54.772 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OrchestratorError(msg) 2022-05-14T00:24:54.772 INFO:journalctl@ceph.mon.c.smithi115.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:24:54.772 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:24:54 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:24:54.772 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:24:54 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:24:54.773 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:24:54 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 3425 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:24:54.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:24:54 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:24:54.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:24:54 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:24:54.792 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T00:24:54.792 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:24:54.792 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T00:24:54.793 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:24:54.793 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:24:54.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:24:54.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T00:24:54.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:24:54.797 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T00:24:54.797 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:24:54.797 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T00:24:54.797 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:24:54.798 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T00:24:54.798 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:24:54.798 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T00:24:54.798 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:24:54.799 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:24:54.799 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:24:54.801 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:24:54 smithi047 ceph-mon[77147]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T00:24:54.801 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T00:24:54.801 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:24:54.801 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T00:24:54.802 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:24:54.802 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:24:54.802 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:24:54.802 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T00:24:54.803 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:24:54.803 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T00:24:54.803 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:24:54.803 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T00:24:54.804 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:24:54.804 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T00:24:54.804 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:24:54.804 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T00:24:54.805 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:24:54.805 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:24:54.806 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:24:54.806 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T00:24:54.806 INFO:journalctl@ceph.mon.b.smithi047.stdout: During handling of the above exception, another exception occurred: 2022-05-14T00:24:54.806 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T00:24:54.807 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T00:24:54.807 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T00:24:54.807 INFO:journalctl@ceph.mon.b.smithi047.stdout: return f(*arg) 2022-05-14T00:24:54.807 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T00:24:54.808 INFO:journalctl@ceph.mon.b.smithi047.stdout: self._write_client_files(client_files, host) 2022-05-14T00:24:54.808 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T00:24:54.809 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T00:24:54.809 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T00:24:54.809 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T00:24:54.810 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T00:24:54.810 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.event_loop.get_result(coro) 2022-05-14T00:24:54.810 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T00:24:54.810 INFO:journalctl@ceph.mon.b.smithi047.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T00:24:54.811 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T00:24:54.812 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.__get_result() 2022-05-14T00:24:54.813 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T00:24:54.813 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise self._exception 2022-05-14T00:24:54.814 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T00:24:54.814 INFO:journalctl@ceph.mon.b.smithi047.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T00:24:54.814 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T00:24:54.814 INFO:journalctl@ceph.mon.b.smithi047.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T00:24:54.815 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T00:24:54.815 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise 2022-05-14T00:24:54.815 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T00:24:54.815 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.gen.throw(type, value, traceback) 2022-05-14T00:24:54.816 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T00:24:54.816 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OrchestratorError(msg) 2022-05-14T00:24:54.817 INFO:journalctl@ceph.mon.b.smithi047.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:24:54.817 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:24:54 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:24:54.817 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:24:54 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:24:54.817 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:24:54 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 3425 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:24:55.110 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:24:54 smithi115 conmon[90393]: 2022-05-14T00:24:54.747+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:24:55.711 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:24:55 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:24:55.712 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:24:55 smithi115 ceph-mon[78593]: pgmap v2067: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:24:55.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:24:55 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:24:55.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:24:55 smithi047 ceph-mon[77147]: pgmap v2067: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:24:56.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:24:55 smithi115 conmon[90393]: 2022-05-14T00:24:55.711+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:24:56.707 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:24:56 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:24:56.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:24:56 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:24:57.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:24:56 smithi115 conmon[90393]: 2022-05-14T00:24:56.706+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:24:57.736 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:24:57 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:24:57.737 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:24:57 smithi115 ceph-mon[78593]: pgmap v2068: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:24:57.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:24:57 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:24:57.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:24:57 smithi047 ceph-mon[77147]: pgmap v2068: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:24:58.118 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:24:57 smithi115 conmon[90393]: 2022-05-14T00:24:57.735+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:24:58.766 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:24:58 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:24:58.766 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:24:58 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:24:58.767 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:24:58 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:24:58.803 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:24:58 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:24:58.804 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:24:58 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:24:58.804 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:24:58 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:24:59.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:24:58 smithi115 conmon[90393]: 2022-05-14T00:24:58.765+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:24:59.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:24:59 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:24:59.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:24:59 smithi047 ceph-mon[77147]: pgmap v2069: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:24:59.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:24:59 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 3430 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:24:59.801 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:24:59 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:24:59.802 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:24:59 smithi115 ceph-mon[78593]: pgmap v2069: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:24:59.802 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:24:59 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 3430 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:25:00.117 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:24:59 smithi115 conmon[90393]: 2022-05-14T00:24:59.801+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:25:00.782 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:25:00 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:25:00.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:25:00 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:25:01.112 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:25:00 smithi115 conmon[90393]: 2022-05-14T00:25:00.780+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:25:01.752 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:25:01 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:25:01.753 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:25:01 smithi115 ceph-mon[78593]: pgmap v2070: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:25:01.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:25:01 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:25:01.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:25:01 smithi047 ceph-mon[77147]: pgmap v2070: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:25:02.111 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:25:01 smithi115 conmon[90393]: 2022-05-14T00:25:01.747+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:25:02.760 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:25:02 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:25:02.760 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:25:02 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:25:02.761 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:25:02 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:25:02.795 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:25:02 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:25:02.796 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:25:02 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:25:02.796 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:25:02 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:25:03.113 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:25:02 smithi115 conmon[90393]: 2022-05-14T00:25:02.759+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:25:03.756 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:25:03 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:25:03.757 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:25:03 smithi115 ceph-mon[78593]: pgmap v2071: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:25:03.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:25:03 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:25:03.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:25:03 smithi047 ceph-mon[77147]: pgmap v2071: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:25:04.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:25:03 smithi115 conmon[90393]: 2022-05-14T00:25:03.753+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:25:04.780 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:25:04 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:25:04.780 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:25:04 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 3435 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:25:04.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:25:04 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:25:04.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:25:04 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 3435 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:25:05.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:25:04 smithi115 conmon[90393]: 2022-05-14T00:25:04.779+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:25:05.796 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:25:05 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:25:05.797 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:25:05 smithi047 ceph-mon[77147]: pgmap v2072: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:25:05.806 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:25:05 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:25:05.807 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:25:05 smithi115 ceph-mon[78593]: pgmap v2072: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:25:06.110 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:25:05 smithi115 conmon[90393]: 2022-05-14T00:25:05.801+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:25:06.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:25:06 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:25:06.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:25:06 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:25:07.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:25:07 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:25:07.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:25:07 smithi047 ceph-mon[77147]: pgmap v2073: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:25:07.833 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:25:07 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:25:07.833 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:25:07 smithi115 ceph-mon[78593]: pgmap v2073: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:25:08.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:25:07 smithi115 conmon[90393]: 2022-05-14T00:25:07.830+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:25:09.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:25:08 smithi115 conmon[90393]: 2022-05-14T00:25:08.784+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:25:09.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:25:09 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:25:09.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:25:09 smithi047 ceph-mon[77147]: pgmap v2074: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:25:09.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:25:09 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 3440 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:25:09.830 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:25:09 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:25:09.831 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:25:09 smithi115 ceph-mon[78593]: pgmap v2074: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:25:09.831 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:25:09 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 3440 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:25:10.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:25:09 smithi115 conmon[90393]: 2022-05-14T00:25:09.828+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:25:10.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:25:10 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:25:10.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:25:10 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:25:10.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:25:10 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:25:10.852 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:25:10 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:25:10.852 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:25:10 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:25:10.852 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:25:10 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:25:11.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:25:10 smithi115 conmon[90393]: 2022-05-14T00:25:10.851+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:25:11.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:25:11 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:25:11.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:25:11 smithi047 ceph-mon[77147]: pgmap v2075: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:25:11.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:25:11 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:25:11.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:25:11 smithi115 ceph-mon[78593]: pgmap v2075: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:25:12.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:25:11 smithi115 conmon[90393]: 2022-05-14T00:25:11.900+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:25:12.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:25:12 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:25:12.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:25:12 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:25:13.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:25:12 smithi115 conmon[90393]: 2022-05-14T00:25:12.853+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:25:13.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:25:13 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:25:13.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:25:13 smithi047 ceph-mon[77147]: pgmap v2076: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:25:13.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:25:13 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:25:13.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:25:13 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:25:13.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:25:13 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T00:25:13.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:25:13 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T00:25:13.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:25:13 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:25:13.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:25:13 smithi115 ceph-mon[78593]: pgmap v2076: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:25:13.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:25:13 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:25:13.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:25:13 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:25:13.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:25:13 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T00:25:13.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:25:13 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T00:25:14.360 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:25:13 smithi115 conmon[90393]: 2022-05-14T00:25:13.886+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:25:14.798 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:25:14 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:25:14.799 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:25:14 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 3445 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:25:14.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:25:14 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:25:14.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:25:14 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 3445 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:25:15.370 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:25:14 smithi115 conmon[90393]: 2022-05-14T00:25:14.884+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:25:15.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:25:15 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:25:15.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:25:15 smithi047 ceph-mon[77147]: pgmap v2077: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:25:15.862 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:25:15 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:25:15.862 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:25:15 smithi115 ceph-mon[78593]: pgmap v2077: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:25:16.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:25:15 smithi115 conmon[90393]: 2022-05-14T00:25:15.889+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:25:16.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:25:16 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:25:16.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:25:16 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:25:17.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:25:16 smithi115 conmon[90393]: 2022-05-14T00:25:16.930+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:25:17.806 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:25:17 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:25:17.811 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:25:17 smithi047 ceph-mon[77147]: pgmap v2078: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:25:17.812 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:25:17 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:25:17.812 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:25:17 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:25:17.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:25:17 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:25:17.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:25:17 smithi115 ceph-mon[78593]: pgmap v2078: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:25:17.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:25:17 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:25:17.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:25:17 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:25:18.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:25:17 smithi115 conmon[90393]: 2022-05-14T00:25:17.931+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:25:18.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:25:18 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:25:18.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:25:18 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:25:19.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:25:18 smithi115 conmon[90393]: 2022-05-14T00:25:18.890+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:25:19.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:25:19 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:25:19.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:25:19 smithi047 ceph-mon[77147]: pgmap v2079: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:25:19.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:25:19 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 3450 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:25:19.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:25:19 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:25:19.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:25:19 smithi115 ceph-mon[78593]: pgmap v2079: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:25:19.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:25:19 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 3450 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:25:20.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:25:19 smithi115 conmon[90393]: 2022-05-14T00:25:19.883+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:25:20.798 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:25:20 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:25:20.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:25:20 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:25:21.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:25:20 smithi115 conmon[90393]: 2022-05-14T00:25:20.910+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:25:21.795 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:25:21 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:25:21.796 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:25:21 smithi047 ceph-mon[77147]: pgmap v2080: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:25:21.796 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:25:21 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:25:21.796 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:25:21 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:25:21.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:25:21 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:25:21.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:25:21 smithi115 ceph-mon[78593]: pgmap v2080: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:25:21.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:25:21 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:25:21.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:25:21 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:25:22.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:25:21 smithi115 conmon[90393]: 2022-05-14T00:25:21.892+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:25:22.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:25:22 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:25:22.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:25:22 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:25:23.361 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:25:22 smithi115 conmon[90393]: 2022-05-14T00:25:22.925+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:25:23.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:25:23 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:25:23.795 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:25:23 smithi047 ceph-mon[77147]: pgmap v2081: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:25:23.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:25:23 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:25:23.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:25:23 smithi115 ceph-mon[78593]: pgmap v2081: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:25:24.361 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:25:23 smithi115 conmon[90393]: 2022-05-14T00:25:23.945+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:25:24.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:25:24 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:25:24.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:25:24 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 3455 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:25:24.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:25:24 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:25:24.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:25:24 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 3455 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:25:25.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:25:24 smithi115 conmon[90393]: 2022-05-14T00:25:24.925+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:25:25.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:25:25 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:25:25.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:25:25 smithi047 ceph-mon[77147]: pgmap v2082: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:25:25.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:25:25 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:25:25.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:25:25 smithi115 ceph-mon[78593]: pgmap v2082: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:25:26.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:25:25 smithi115 conmon[90393]: 2022-05-14T00:25:25.924+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:25:26.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:25:26 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:25:26.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:25:26 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:25:27.360 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:25:26 smithi115 conmon[90393]: 2022-05-14T00:25:26.883+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:25:27.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:25:27 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:25:27.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:25:27 smithi047 ceph-mon[77147]: pgmap v2083: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:25:27.854 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:25:27 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:25:27.862 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:25:27 smithi115 ceph-mon[78593]: pgmap v2083: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:25:28.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:25:27 smithi115 conmon[90393]: 2022-05-14T00:25:27.850+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:25:28.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:25:28 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:25:28.817 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:25:28 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:25:29.111 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:25:28 smithi115 conmon[90393]: 2022-05-14T00:25:28.816+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:25:29.798 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:25:29 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:25:29.798 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:25:29 smithi047 ceph-mon[77147]: pgmap v2084: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:25:29.798 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:25:29 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:25:29.799 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:25:29 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:25:29.799 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:25:29 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 3460 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:25:29.830 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:25:29 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:25:29.830 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:25:29 smithi115 ceph-mon[78593]: pgmap v2084: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:25:29.830 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:25:29 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:25:29.830 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:25:29 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:25:29.831 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:25:29 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 3460 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:25:30.118 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:25:29 smithi115 conmon[90393]: 2022-05-14T00:25:29.822+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:25:30.818 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:25:30 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:25:31.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:25:30 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:25:31.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:25:30 smithi115 conmon[90393]: 2022-05-14T00:25:30.803+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:25:31.808 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:25:31 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:25:31.808 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:25:31 smithi115 ceph-mon[78593]: pgmap v2085: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:25:32.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:25:31 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:25:32.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:25:31 smithi047 ceph-mon[77147]: pgmap v2085: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:25:32.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:25:31 smithi115 conmon[90393]: 2022-05-14T00:25:31.807+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:25:32.804 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:25:32 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:25:33.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:25:32 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:25:33.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:25:32 smithi115 conmon[90393]: 2022-05-14T00:25:32.803+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:25:33.809 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:25:33 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:25:33.809 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:25:33 smithi115 ceph-mon[78593]: pgmap v2086: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:25:33.809 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:25:33 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:25:33.810 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:25:33 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:25:34.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:25:33 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:25:34.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:25:33 smithi047 ceph-mon[77147]: pgmap v2086: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:25:34.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:25:33 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:25:34.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:25:33 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:25:34.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:25:33 smithi115 conmon[90393]: 2022-05-14T00:25:33.807+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:25:34.809 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:25:34 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:25:34.809 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:25:34 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 3465 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:25:34.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:25:34 smithi115 conmon[90393]: 2022-05-14T00:25:34.800+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:25:34.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:25:34 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:25:34.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:25:34 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 3465 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:25:35.827 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:25:35 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:25:35.828 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:25:35 smithi047 ceph-mon[77147]: pgmap v2087: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:25:35.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:25:35 smithi115 conmon[90393]: 2022-05-14T00:25:35.756+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:25:35.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:25:35 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:25:35.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:25:35 smithi115 ceph-mon[78593]: pgmap v2087: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:25:36.817 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:25:36 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:25:36.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:25:36 smithi115 conmon[90393]: 2022-05-14T00:25:36.765+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:25:36.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:25:36 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:25:37.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:25:37 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:25:37.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:25:37 smithi047 ceph-mon[77147]: pgmap v2088: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:25:37.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:25:37 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:25:37.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:25:37 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:25:37.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:25:37 smithi115 conmon[90393]: 2022-05-14T00:25:37.763+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:25:37.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:25:37 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:25:37.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:25:37 smithi115 ceph-mon[78593]: pgmap v2088: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:25:37.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:25:37 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:25:37.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:25:37 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:25:38.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:25:38 smithi115 conmon[90393]: 2022-05-14T00:25:38.808+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:25:38.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:25:38 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:25:38.955 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:25:38 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:25:39.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:25:39 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:25:39.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:25:39 smithi115 ceph-mon[78593]: pgmap v2089: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:25:39.862 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:25:39 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 3470 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:25:40.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:25:39 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:25:40.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:25:39 smithi047 ceph-mon[77147]: pgmap v2089: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:25:40.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:25:39 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 3470 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:25:40.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:25:39 smithi115 conmon[90393]: 2022-05-14T00:25:39.857+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:25:40.841 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:25:40 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:25:41.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:25:40 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:25:41.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:25:40 smithi115 conmon[90393]: 2022-05-14T00:25:40.840+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:25:41.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:25:41 smithi115 conmon[90393]: 2022-05-14T00:25:41.799+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:25:41.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:25:41 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:25:41.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:25:41 smithi115 ceph-mon[78593]: pgmap v2090: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:25:41.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:25:41 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:25:41.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:25:41 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:25:42.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:25:41 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:25:42.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:25:41 smithi047 ceph-mon[77147]: pgmap v2090: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:25:42.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:25:41 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:25:42.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:25:41 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:25:42.834 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:25:42 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:25:43.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:25:42 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:25:43.112 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:25:42 smithi115 conmon[90393]: 2022-05-14T00:25:42.833+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:25:43.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:25:43 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:25:43.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:25:43 smithi115 ceph-mon[78593]: pgmap v2091: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:25:43.860 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:25:43 smithi115 conmon[90393]: 2022-05-14T00:25:43.788+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:25:44.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:25:43 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:25:44.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:25:43 smithi047 ceph-mon[77147]: pgmap v2091: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:25:44.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:25:44 smithi115 conmon[90393]: 2022-05-14T00:25:44.786+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:25:44.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:25:44 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:25:44.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:25:44 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 3475 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:25:45.046 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:25:44 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:25:45.047 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:25:44 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 3475 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:25:45.860 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:25:45 smithi115 conmon[90393]: 2022-05-14T00:25:45.798+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:25:45.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:25:45 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:25:45.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:25:45 smithi115 ceph-mon[78593]: pgmap v2092: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:25:46.047 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:25:45 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:25:46.047 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:25:45 smithi047 ceph-mon[77147]: pgmap v2092: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:25:46.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:25:46 smithi115 conmon[90393]: 2022-05-14T00:25:46.819+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:25:46.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:25:46 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:25:47.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:25:46 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:25:47.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:25:47 smithi115 conmon[90393]: 2022-05-14T00:25:47.789+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:25:47.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:25:47 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:25:47.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:25:47 smithi115 ceph-mon[78593]: pgmap v2093: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:25:47.963 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:25:47 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:25:47.964 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:25:47 smithi047 ceph-mon[77147]: pgmap v2093: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:25:48.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:25:48 smithi115 conmon[90393]: 2022-05-14T00:25:48.818+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:25:48.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:25:48 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:25:49.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:25:48 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:25:49.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:25:49 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:25:49.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:25:49 smithi115 ceph-mon[78593]: pgmap v2094: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:25:49.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:25:49 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:25:49.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:25:49 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:25:49.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:25:49 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 3480 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:25:50.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:25:49 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:25:50.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:25:49 smithi047 ceph-mon[77147]: pgmap v2094: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:25:50.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:25:49 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:25:50.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:25:49 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:25:50.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:25:49 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 3480 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:25:50.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:25:49 smithi115 conmon[90393]: 2022-05-14T00:25:49.867+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:25:50.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:25:50 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:25:51.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:25:50 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:25:51.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:25:50 smithi115 conmon[90393]: 2022-05-14T00:25:50.895+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:25:51.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:25:51 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:25:51.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:25:51 smithi115 ceph-mon[78593]: pgmap v2095: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:25:52.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:25:51 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:25:52.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:25:51 smithi047 ceph-mon[77147]: pgmap v2095: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:25:52.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:25:51 smithi115 conmon[90393]: 2022-05-14T00:25:51.878+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:25:52.878 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:25:52 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:25:53.045 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:25:52 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:25:53.364 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:25:52 smithi115 conmon[90393]: 2022-05-14T00:25:52.871+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:25:53.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:25:53 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:25:53.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:25:53 smithi047 ceph-mon[77147]: pgmap v2096: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:25:53.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:25:53 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:25:53.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:25:53 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:25:53.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:25:53 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T00:25:53.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:25:53 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T00:25:53.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:25:53 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T00:25:53.864 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:25:53 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:25:53.864 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:25:53 smithi115 ceph-mon[78593]: pgmap v2096: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:25:53.865 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:25:53 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:25:53.865 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:25:53 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:25:53.865 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:25:53 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T00:25:53.865 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:25:53 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T00:25:53.866 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:25:53 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T00:25:54.290 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:25:53 smithi047 conmon[78521]: 2022-05-14T00:25:53.904+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:25:54.291 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:25:53 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T00:25:54.291 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:25:53 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:25:54.291 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:25:53 smithi047 conmon[78521]: yield 2022-05-14T00:25:54.291 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:25:53 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:25:54.292 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:25:53 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:25:54.292 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:25:53 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:25:54.292 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:25:53 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T00:25:54.292 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:25:53 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:25:54.293 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:25:53 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T00:25:54.293 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:25:53 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:25:54.297 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:25:53 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T00:25:54.297 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:25:53 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:25:54.297 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:25:53 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T00:25:54.298 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:25:53 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:25:54.298 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:25:53 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T00:25:54.298 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:25:53 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:25:54.298 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:25:53 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:25:54.299 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:25:53 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:25:54.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:25:53 smithi115 conmon[90393]: 2022-05-14T00:25:53.907+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:25:54.944 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:25:54 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:25:54.944 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:25:54 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 3485 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:25:54.944 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:25:54 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:25:54.944 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T00:25:54.945 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:25:54.945 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T00:25:54.945 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:25:54.945 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:25:54.946 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:25:54.946 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T00:25:54.946 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:25:54.946 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T00:25:54.947 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:25:54.947 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T00:25:54.948 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:25:54.948 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T00:25:54.948 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:25:54.948 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T00:25:54.948 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:25:54.949 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:25:54.949 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:25:54.949 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:25:54 smithi115 ceph-mon[78593]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T00:25:55.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:25:54 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:25:55.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:25:54 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 3485 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:25:55.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:25:54 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:25:55.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T00:25:55.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:25:55.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T00:25:55.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:25:55.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:25:55.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:25:55.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T00:25:55.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:25:55.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T00:25:55.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:25:55.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T00:25:55.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:25:55.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T00:25:55.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:25:55.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T00:25:55.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:25:55.049 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:25:55.049 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:25:55.049 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:25:54 smithi047 ceph-mon[77147]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T00:25:55.360 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:25:54 smithi115 conmon[90393]: 2022-05-14T00:25:54.943+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:25:55.991 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:25:55 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:25:55.992 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:25:55 smithi115 ceph-mon[78593]: pgmap v2097: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:25:56.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:25:55 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:25:56.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:25:55 smithi047 ceph-mon[77147]: pgmap v2097: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:25:56.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:25:55 smithi115 conmon[90393]: 2022-05-14T00:25:55.991+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:25:56.981 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:25:56 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:25:56.981 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:25:56 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:25:57.291 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:25:56 smithi047 conmon[78521]: 2022-05-14T00:25:56.976+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:25:57.291 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:25:56 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T00:25:57.292 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:25:56 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:25:57.292 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:25:56 smithi047 conmon[78521]: yield 2022-05-14T00:25:57.292 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:25:56 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:25:57.292 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:25:56 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:25:57.293 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:25:56 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:25:57.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:25:56 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T00:25:57.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:25:56 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:25:57.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:25:56 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T00:25:57.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:25:56 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:25:57.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:25:56 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T00:25:57.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:25:56 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:25:57.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:25:56 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T00:25:57.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:25:56 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:25:57.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:25:56 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T00:25:57.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:25:56 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:25:57.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:25:56 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:25:57.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:25:56 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:25:57.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:25:56 smithi047 conmon[78521]: 2022-05-14T00:25:56.977+0000 7f533ef76700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T00:25:57.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:25:56 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T00:25:57.297 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:25:56 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:25:57.297 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:25:56 smithi047 conmon[78521]: yield 2022-05-14T00:25:57.297 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:25:56 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:25:57.297 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:25:56 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:25:57.297 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:25:56 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:25:57.298 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:25:56 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T00:25:57.298 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:25:56 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:25:57.298 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:25:56 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T00:25:57.298 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:25:56 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:25:57.299 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:25:56 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T00:25:57.299 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:25:56 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:25:57.300 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:25:56 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T00:25:57.300 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:25:56 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:25:57.301 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:25:56 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T00:25:57.301 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:25:56 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:25:57.301 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:25:56 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:25:57.301 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:25:56 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:25:57.302 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:25:56 smithi047 conmon[78521]: 2022-05-14T00:25:57.302 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:25:56 smithi047 conmon[78521]: During handling of the above exception, another exception occurred: 2022-05-14T00:25:57.302 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:25:56 smithi047 conmon[78521]: 2022-05-14T00:25:57.302 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:25:56 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T00:25:57.303 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:25:56 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T00:25:57.303 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:25:56 smithi047 conmon[78521]: return f(*arg) 2022-05-14T00:25:57.303 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:25:56 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T00:25:57.303 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:25:56 smithi047 conmon[78521]: self._write_client_files(client_files, host) 2022-05-14T00:25:57.303 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:25:56 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T00:25:57.304 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:25:56 smithi047 conmon[78521]: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T00:25:57.304 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:25:56 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T00:25:57.304 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:25:56 smithi047 conmon[78521]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T00:25:57.304 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:25:56 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T00:25:57.305 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:25:56 smithi047 conmon[78521]: return self.event_loop.get_result(coro) 2022-05-14T00:25:57.305 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:25:56 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T00:25:57.306 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:25:56 smithi047 conmon[78521]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T00:25:57.306 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:25:56 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T00:25:57.307 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:25:56 smithi047 conmon[78521]: return self.__get_result() 2022-05-14T00:25:57.307 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:25:56 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T00:25:57.307 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:25:56 smithi047 conmon[78521]: raise self._exception 2022-05-14T00:25:57.308 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:25:56 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T00:25:57.308 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:25:56 smithi047 conmon[78521]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T00:25:57.308 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:25:56 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T00:25:57.308 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:25:56 smithi047 conmon[78521]: conn = await self._remote_connection(host, addr) 2022-05-14T00:25:57.309 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:25:56 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T00:25:57.309 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:25:56 smithi047 conmon[78521]: raise 2022-05-14T00:25:57.309 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:25:56 smithi047 conmon[78521]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T00:25:57.309 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:25:56 smithi047 conmon[78521]: self.gen.throw(type, value, traceback) 2022-05-14T00:25:57.309 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:25:56 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T00:25:57.310 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:25:56 smithi047 conmon[78521]: raise OrchestratorError(msg) 2022-05-14T00:25:57.310 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:25:56 smithi047 conmon[78521]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:25:57.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:25:56 smithi115 conmon[90393]: 2022-05-14T00:25:56.978+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:25:57.946 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:25:57 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:25:57.946 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:25:57 smithi115 ceph-mon[78593]: pgmap v2098: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:25:57.947 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:25:57 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:25:57.947 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:25:57 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:25:57.947 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:25:57 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:25:57.947 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T00:25:57.948 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:25:57.948 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T00:25:57.948 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:25:57.948 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:25:57.949 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:25:57.949 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T00:25:57.950 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:25:57.951 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T00:25:57.951 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:25:57.951 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T00:25:57.952 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:25:57.952 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T00:25:57.952 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:25:57.952 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T00:25:57.953 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:25:57.953 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:25:57.953 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:25:57.953 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:25:57 smithi115 ceph-mon[78593]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T00:25:57.954 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T00:25:57.954 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:25:57.954 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T00:25:57.954 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:25:57.955 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:25:57.955 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:25:57.955 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T00:25:57.955 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:25:57.955 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T00:25:57.956 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:25:57.956 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T00:25:57.956 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:25:57.956 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T00:25:57.957 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:25:57.957 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T00:25:57.957 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:25:57.957 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:25:57.958 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:25:57.958 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T00:25:57.958 INFO:journalctl@ceph.mon.c.smithi115.stdout: During handling of the above exception, another exception occurred: 2022-05-14T00:25:57.959 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T00:25:57.959 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T00:25:57.959 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T00:25:57.959 INFO:journalctl@ceph.mon.c.smithi115.stdout: return f(*arg) 2022-05-14T00:25:57.960 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T00:25:57.960 INFO:journalctl@ceph.mon.c.smithi115.stdout: self._write_client_files(client_files, host) 2022-05-14T00:25:57.960 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T00:25:57.960 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T00:25:57.961 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T00:25:57.961 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T00:25:57.961 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T00:25:57.962 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.event_loop.get_result(coro) 2022-05-14T00:25:57.962 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T00:25:57.964 INFO:journalctl@ceph.mon.c.smithi115.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T00:25:57.964 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T00:25:57.964 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.__get_result() 2022-05-14T00:25:57.964 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T00:25:57.965 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise self._exception 2022-05-14T00:25:57.965 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T00:25:57.965 INFO:journalctl@ceph.mon.c.smithi115.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T00:25:57.965 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T00:25:57.966 INFO:journalctl@ceph.mon.c.smithi115.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T00:25:57.966 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T00:25:57.966 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise 2022-05-14T00:25:57.966 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T00:25:57.967 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.gen.throw(type, value, traceback) 2022-05-14T00:25:57.967 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T00:25:57.967 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OrchestratorError(msg) 2022-05-14T00:25:57.967 INFO:journalctl@ceph.mon.c.smithi115.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:25:58.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:25:57 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:25:58.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:25:57 smithi047 ceph-mon[77147]: pgmap v2098: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:25:58.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:25:57 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:25:58.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:25:57 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:25:58.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:25:57 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:25:58.042 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T00:25:58.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:25:58.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T00:25:58.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:25:58.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:25:58.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:25:58.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T00:25:58.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:25:58.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T00:25:58.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:25:58.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T00:25:58.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:25:58.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T00:25:58.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:25:58.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T00:25:58.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:25:58.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:25:58.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:25:58.047 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:25:57 smithi047 ceph-mon[77147]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T00:25:58.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T00:25:58.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:25:58.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T00:25:58.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:25:58.051 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:25:58.051 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:25:58.051 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T00:25:58.051 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:25:58.053 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T00:25:58.053 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:25:58.054 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T00:25:58.054 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:25:58.054 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T00:25:58.054 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:25:58.055 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T00:25:58.055 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:25:58.055 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:25:58.056 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:25:58.056 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T00:25:58.057 INFO:journalctl@ceph.mon.b.smithi047.stdout: During handling of the above exception, another exception occurred: 2022-05-14T00:25:58.057 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T00:25:58.057 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T00:25:58.058 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T00:25:58.058 INFO:journalctl@ceph.mon.b.smithi047.stdout: return f(*arg) 2022-05-14T00:25:58.058 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T00:25:58.058 INFO:journalctl@ceph.mon.b.smithi047.stdout: self._write_client_files(client_files, host) 2022-05-14T00:25:58.058 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T00:25:58.059 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T00:25:58.059 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T00:25:58.059 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T00:25:58.059 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T00:25:58.060 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.event_loop.get_result(coro) 2022-05-14T00:25:58.060 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T00:25:58.060 INFO:journalctl@ceph.mon.b.smithi047.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T00:25:58.060 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T00:25:58.061 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.__get_result() 2022-05-14T00:25:58.062 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T00:25:58.062 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise self._exception 2022-05-14T00:25:58.063 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T00:25:58.063 INFO:journalctl@ceph.mon.b.smithi047.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T00:25:58.063 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T00:25:58.064 INFO:journalctl@ceph.mon.b.smithi047.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T00:25:58.064 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T00:25:58.064 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise 2022-05-14T00:25:58.065 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T00:25:58.065 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.gen.throw(type, value, traceback) 2022-05-14T00:25:58.065 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T00:25:58.066 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OrchestratorError(msg) 2022-05-14T00:25:58.066 INFO:journalctl@ceph.mon.b.smithi047.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:25:58.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:25:57 smithi115 conmon[90393]: 2022-05-14T00:25:57.944+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:25:58.976 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:25:58 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:25:59.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:25:58 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:25:59.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:25:58 smithi115 conmon[90393]: 2022-05-14T00:25:58.975+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:26:00.021 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:25:59 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:26:00.022 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:25:59 smithi115 ceph-mon[78593]: pgmap v2099: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:26:00.022 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:25:59 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 3490 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:26:00.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:25:59 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:26:00.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:25:59 smithi047 ceph-mon[77147]: pgmap v2099: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:26:00.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:25:59 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 3490 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:26:00.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:26:00 smithi115 conmon[90393]: 2022-05-14T00:26:00.019+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:26:01.026 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:26:00 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:26:01.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:26:00 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:26:01.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:26:01 smithi115 conmon[90393]: 2022-05-14T00:26:01.021+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:26:02.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:26:01 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:26:02.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:26:01 smithi047 ceph-mon[77147]: pgmap v2100: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:26:02.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:26:01 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:26:02.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:26:01 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:26:02.071 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:26:01 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:26:02.071 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:26:01 smithi115 ceph-mon[78593]: pgmap v2100: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:26:02.071 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:26:01 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:26:02.072 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:26:01 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:26:02.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:26:02 smithi115 conmon[90393]: 2022-05-14T00:26:02.070+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:26:03.024 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:26:02 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:26:03.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:26:02 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:26:03.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:26:03 smithi115 conmon[90393]: 2022-05-14T00:26:03.023+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:26:04.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:26:03 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:26:04.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:26:03 smithi047 ceph-mon[77147]: pgmap v2101: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:26:04.069 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:26:03 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:26:04.069 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:26:03 smithi115 ceph-mon[78593]: pgmap v2101: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:26:04.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:26:04 smithi115 conmon[90393]: 2022-05-14T00:26:04.067+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:26:05.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:26:04 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:26:05.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:26:04 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 3495 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:26:05.048 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:26:04 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:26:05.049 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:26:04 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 3495 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:26:05.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:26:05 smithi115 conmon[90393]: 2022-05-14T00:26:05.047+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:26:06.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:26:05 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:26:06.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:26:05 smithi047 ceph-mon[77147]: pgmap v2102: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:26:06.090 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:26:05 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:26:06.091 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:26:05 smithi115 ceph-mon[78593]: pgmap v2102: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:26:06.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:26:06 smithi115 conmon[90393]: 2022-05-14T00:26:06.088+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:26:07.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:26:06 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:26:07.075 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:26:06 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:26:07.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:26:07 smithi115 conmon[90393]: 2022-05-14T00:26:07.074+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:26:08.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:26:07 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:26:08.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:26:07 smithi047 ceph-mon[77147]: pgmap v2103: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:26:08.059 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:26:07 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:26:08.060 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:26:07 smithi115 ceph-mon[78593]: pgmap v2103: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:26:08.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:26:08 smithi115 conmon[90393]: 2022-05-14T00:26:08.058+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:26:09.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:26:08 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:26:09.099 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:26:08 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:26:09.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:26:09 smithi115 conmon[90393]: 2022-05-14T00:26:09.098+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:26:10.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:26:09 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:26:10.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:26:09 smithi047 ceph-mon[77147]: pgmap v2104: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:26:10.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:26:09 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:26:10.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:26:09 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:26:10.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:26:09 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 3500 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:26:10.059 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:26:09 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:26:10.059 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:26:09 smithi115 ceph-mon[78593]: pgmap v2104: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:26:10.060 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:26:09 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:26:10.060 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:26:09 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:26:10.061 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:26:09 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 3500 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:26:10.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:26:10 smithi115 conmon[90393]: 2022-05-14T00:26:10.058+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:26:11.030 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:26:10 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:26:11.046 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:26:10 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:26:11.365 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:26:11 smithi115 conmon[90393]: 2022-05-14T00:26:11.028+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:26:12.046 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:26:11 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:26:12.046 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:26:11 smithi047 ceph-mon[77147]: pgmap v2105: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:26:12.078 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:26:11 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:26:12.078 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:26:11 smithi115 ceph-mon[78593]: pgmap v2105: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:26:12.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:26:12 smithi115 conmon[90393]: 2022-05-14T00:26:12.075+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:26:13.032 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:26:12 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:26:13.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:26:12 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:26:13.362 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:26:13 smithi115 conmon[90393]: 2022-05-14T00:26:13.030+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:26:13.994 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:26:13 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:26:13.994 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:26:13 smithi115 ceph-mon[78593]: pgmap v2106: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:26:13.994 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:26:13 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:26:13.995 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:26:13 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:26:13.995 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:26:13 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T00:26:13.995 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:26:13 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T00:26:14.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:26:13 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:26:14.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:26:13 smithi047 ceph-mon[77147]: pgmap v2106: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:26:14.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:26:13 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:26:14.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:26:13 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:26:14.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:26:13 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T00:26:14.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:26:13 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T00:26:14.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:26:13 smithi115 conmon[90393]: 2022-05-14T00:26:13.985+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:26:15.008 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:26:14 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:26:15.009 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:26:14 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 3504 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:26:15.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:26:14 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:26:15.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:26:14 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 3504 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:26:15.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:26:15 smithi115 conmon[90393]: 2022-05-14T00:26:15.007+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:26:15.972 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:26:15 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:26:15.972 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:26:15 smithi115 ceph-mon[78593]: pgmap v2107: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:26:16.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:26:15 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:26:16.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:26:15 smithi047 ceph-mon[77147]: pgmap v2107: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:26:16.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:26:15 smithi115 conmon[90393]: 2022-05-14T00:26:15.971+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:26:16.950 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:26:16 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:26:16.950 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:26:16 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:26:17.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:26:16 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:26:17.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:26:16 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:26:17.362 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:26:16 smithi115 conmon[90393]: 2022-05-14T00:26:16.948+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:26:18.045 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:26:17 smithi047 ceph-mon[77147]: pgmap v2108: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:26:18.046 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:26:17 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:26:18.046 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:26:17 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:26:18.046 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:26:17 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:26:18.110 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:26:17 smithi115 conmon[90393]: 2022-05-14T00:26:17.903+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:26:18.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:26:17 smithi115 ceph-mon[78593]: pgmap v2108: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:26:18.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:26:17 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:26:18.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:26:17 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:26:18.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:26:17 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:26:19.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:26:18 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:26:19.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:26:18 smithi115 conmon[90393]: 2022-05-14T00:26:18.860+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:26:19.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:26:18 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:26:20.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:26:19 smithi047 ceph-mon[77147]: pgmap v2109: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:26:20.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:26:19 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:26:20.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:26:19 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 3510 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:26:20.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:26:19 smithi115 conmon[90393]: 2022-05-14T00:26:19.849+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:26:20.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:26:19 smithi115 ceph-mon[78593]: pgmap v2109: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:26:20.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:26:19 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:26:20.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:26:19 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 3510 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:26:21.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:26:20 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:26:21.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:26:20 smithi115 conmon[90393]: 2022-05-14T00:26:20.814+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:26:21.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:26:20 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:26:22.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:26:21 smithi047 ceph-mon[77147]: pgmap v2110: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:26:22.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:26:21 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:26:22.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:26:21 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:26:22.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:26:21 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:26:22.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:26:21 smithi115 conmon[90393]: 2022-05-14T00:26:21.793+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:26:22.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:26:21 smithi115 ceph-mon[78593]: pgmap v2110: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:26:22.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:26:21 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:26:22.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:26:21 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:26:22.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:26:21 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:26:23.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:26:22 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:26:23.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:26:22 smithi115 conmon[90393]: 2022-05-14T00:26:22.832+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:26:23.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:26:22 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:26:24.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:26:23 smithi047 ceph-mon[77147]: pgmap v2111: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:26:24.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:26:23 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:26:24.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:26:23 smithi115 conmon[90393]: 2022-05-14T00:26:23.855+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:26:24.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:26:23 smithi115 ceph-mon[78593]: pgmap v2111: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:26:24.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:26:23 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:26:25.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:26:24 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:26:25.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:26:24 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 3515 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:26:25.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:26:24 smithi115 conmon[90393]: 2022-05-14T00:26:24.893+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:26:25.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:26:24 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:26:25.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:26:24 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 3515 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:26:26.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:26:25 smithi047 ceph-mon[77147]: pgmap v2112: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:26:26.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:26:25 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:26:26.362 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:26:25 smithi115 conmon[90393]: 2022-05-14T00:26:25.859+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:26:26.363 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:26:25 smithi115 ceph-mon[78593]: pgmap v2112: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:26:26.363 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:26:25 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:26:27.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:26:26 smithi115 conmon[90393]: 2022-05-14T00:26:26.844+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:26:27.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:26:26 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:26:27.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:26:26 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:26:28.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:26:27 smithi115 conmon[90393]: 2022-05-14T00:26:27.810+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:26:28.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:26:27 smithi115 ceph-mon[78593]: pgmap v2113: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:26:28.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:26:27 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:26:28.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:26:27 smithi047 ceph-mon[77147]: pgmap v2113: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:26:28.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:26:27 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:26:29.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:26:28 smithi115 conmon[90393]: 2022-05-14T00:26:28.846+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:26:29.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:26:28 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:26:29.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:26:28 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 3520 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:26:29.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:26:28 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:26:29.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:26:28 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 3520 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:26:30.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:26:29 smithi047 ceph-mon[77147]: pgmap v2114: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:26:30.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:26:29 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:26:30.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:26:29 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:26:30.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:26:29 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:26:30.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:26:29 smithi115 conmon[90393]: 2022-05-14T00:26:29.876+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:26:30.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:26:29 smithi115 ceph-mon[78593]: pgmap v2114: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:26:30.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:26:29 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:26:30.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:26:29 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:26:30.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:26:29 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:26:31.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:26:30 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:26:31.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:26:30 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:26:31.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:26:30 smithi115 conmon[90393]: 2022-05-14T00:26:30.924+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:26:32.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:26:31 smithi047 ceph-mon[77147]: pgmap v2115: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:26:32.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:26:31 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:26:32.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:26:31 smithi115 conmon[90393]: 2022-05-14T00:26:31.906+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:26:32.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:26:31 smithi115 ceph-mon[78593]: pgmap v2115: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:26:32.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:26:31 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:26:33.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:26:32 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:26:33.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:26:32 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:26:33.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:26:32 smithi115 conmon[90393]: 2022-05-14T00:26:32.955+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:26:34.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:26:33 smithi047 ceph-mon[77147]: pgmap v2116: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:26:34.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:26:33 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:26:34.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:26:33 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:26:34.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:26:33 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:26:34.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:26:33 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 3525 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:26:34.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:26:33 smithi115 conmon[90393]: 2022-05-14T00:26:33.986+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:26:34.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:26:33 smithi115 ceph-mon[78593]: pgmap v2116: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:26:34.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:26:33 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:26:34.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:26:33 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:26:34.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:26:33 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:26:34.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:26:33 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 3525 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:26:35.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:26:34 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:26:35.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:26:34 smithi115 conmon[90393]: 2022-05-14T00:26:34.958+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:26:35.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:26:34 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:26:36.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:26:35 smithi047 ceph-mon[77147]: pgmap v2117: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:26:36.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:26:35 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:26:36.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:26:35 smithi115 ceph-mon[78593]: pgmap v2117: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:26:36.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:26:35 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:26:36.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:26:35 smithi115 conmon[90393]: 2022-05-14T00:26:35.945+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:26:37.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:26:36 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:26:37.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:26:36 smithi115 conmon[90393]: 2022-05-14T00:26:36.978+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:26:37.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:26:36 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:26:38.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:26:37 smithi047 ceph-mon[77147]: pgmap v2118: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:26:38.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:26:37 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:26:38.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:26:37 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:26:38.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:26:37 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:26:38.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:26:38 smithi115 conmon[90393]: 2022-05-14T00:26:38.011+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:26:38.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:26:37 smithi115 ceph-mon[78593]: pgmap v2118: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:26:38.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:26:37 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:26:38.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:26:37 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:26:38.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:26:37 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:26:39.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:26:38 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:26:39.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:26:38 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 3530 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:26:39.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:26:39 smithi115 conmon[90393]: 2022-05-14T00:26:39.043+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:26:39.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:26:38 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:26:39.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:26:38 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 3530 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:26:40.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:26:39 smithi047 ceph-mon[77147]: pgmap v2119: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:26:40.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:26:39 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:26:40.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:26:40 smithi115 conmon[90393]: 2022-05-14T00:26:40.070+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:26:40.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:26:39 smithi115 ceph-mon[78593]: pgmap v2119: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:26:40.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:26:39 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:26:41.297 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:26:40 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:26:41.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:26:40 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:26:41.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:26:41 smithi115 conmon[90393]: 2022-05-14T00:26:41.063+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:26:42.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:26:41 smithi047 ceph-mon[77147]: pgmap v2120: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:26:42.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:26:41 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:26:42.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:26:41 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:26:42.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:26:41 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:26:42.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:26:42 smithi115 conmon[90393]: 2022-05-14T00:26:42.069+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:26:42.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:26:41 smithi115 ceph-mon[78593]: pgmap v2120: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:26:42.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:26:41 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:26:42.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:26:41 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:26:42.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:26:41 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:26:43.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:26:42 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:26:43.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:26:43 smithi115 conmon[90393]: 2022-05-14T00:26:43.025+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:26:43.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:26:42 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:26:44.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:26:43 smithi047 ceph-mon[77147]: pgmap v2121: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:26:44.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:26:43 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:26:44.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:26:43 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 3535 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:26:44.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:26:44 smithi115 conmon[90393]: 2022-05-14T00:26:44.015+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:26:44.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:26:43 smithi115 ceph-mon[78593]: pgmap v2121: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:26:44.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:26:43 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:26:44.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:26:43 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 3535 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:26:45.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:26:44 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:26:45.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:26:44 smithi115 conmon[90393]: 2022-05-14T00:26:44.967+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:26:45.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:26:44 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:26:46.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:26:45 smithi047 ceph-mon[77147]: pgmap v2122: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:26:46.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:26:45 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:26:46.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:26:46 smithi115 conmon[90393]: 2022-05-14T00:26:46.017+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:26:46.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:26:45 smithi115 ceph-mon[78593]: pgmap v2122: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:26:46.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:26:45 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:26:47.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:26:46 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:26:47.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:26:47 smithi115 conmon[90393]: 2022-05-14T00:26:47.033+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:26:47.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:26:46 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:26:48.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:26:47 smithi047 ceph-mon[77147]: pgmap v2123: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:26:48.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:26:47 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:26:48.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:26:48 smithi115 conmon[90393]: 2022-05-14T00:26:48.019+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:26:48.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:26:47 smithi115 ceph-mon[78593]: pgmap v2123: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:26:48.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:26:47 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:26:49.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:26:48 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:26:49.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:26:48 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 3540 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:26:49.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:26:49 smithi115 conmon[90393]: 2022-05-14T00:26:49.023+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:26:49.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:26:48 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:26:49.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:26:48 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 3540 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:26:50.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:26:49 smithi047 ceph-mon[77147]: pgmap v2124: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:26:50.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:26:49 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:26:50.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:26:49 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:26:50.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:26:49 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:26:50.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:26:50 smithi115 conmon[90393]: 2022-05-14T00:26:50.023+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:26:50.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:26:49 smithi115 ceph-mon[78593]: pgmap v2124: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:26:50.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:26:49 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:26:50.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:26:49 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:26:50.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:26:49 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:26:51.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:26:50 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:26:51.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:26:50 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:26:52.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:26:51 smithi047 ceph-mon[77147]: pgmap v2125: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:26:52.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:26:52 smithi115 conmon[90393]: 2022-05-14T00:26:52.048+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:26:52.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:26:51 smithi115 ceph-mon[78593]: pgmap v2125: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:26:53.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:26:53 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:26:53.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:26:53 smithi115 conmon[90393]: 2022-05-14T00:26:53.023+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:26:53.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:26:53 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:26:54.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:26:54 smithi047 ceph-mon[77147]: pgmap v2126: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:26:54.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:26:54 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:26:54.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:26:54 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:26:54.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:26:54 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:26:54.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:26:54 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 3545 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:26:54.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:26:54 smithi115 conmon[90393]: 2022-05-14T00:26:54.070+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:26:54.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:26:54 smithi115 ceph-mon[78593]: pgmap v2126: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:26:54.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:26:54 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:26:54.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:26:54 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:26:54.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:26:54 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:26:54.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:26:54 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 3545 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:26:55.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:26:55 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:26:55.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:26:55 smithi115 conmon[90393]: 2022-05-14T00:26:55.045+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:26:55.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:26:55 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:26:56.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:26:56 smithi047 ceph-mon[77147]: pgmap v2127: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:26:56.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:26:56 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:26:56.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:26:56 smithi115 conmon[90393]: 2022-05-14T00:26:56.007+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:26:56.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:26:56 smithi115 ceph-mon[78593]: pgmap v2127: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:26:56.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:26:56 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:26:57.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:26:57 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:26:57.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:26:57 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T00:26:57.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:26:57 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T00:26:57.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:26:57 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T00:26:57.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:26:57 smithi115 conmon[90393]: 2022-05-14T00:26:57.032+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:26:57.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:26:57 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:26:57.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:26:57 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T00:26:57.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:26:57 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T00:26:57.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:26:57 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T00:26:58.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:26:58 smithi047 ceph-mon[77147]: pgmap v2128: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:26:58.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:26:58 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:26:58.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:26:58 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:26:58.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:26:58 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:26:58.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:26:58 smithi115 conmon[90393]: 2022-05-14T00:26:58.000+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:26:58.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:26:58 smithi115 ceph-mon[78593]: pgmap v2128: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:26:58.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:26:58 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:26:58.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:26:58 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:26:58.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:26:58 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:26:59.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:26:59 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:26:59.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:26:59 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 3550 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:26:59.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:26:59 smithi115 conmon[90393]: 2022-05-14T00:26:59.035+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:26:59.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:26:59 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:26:59.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:26:59 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 3550 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:27:00.296 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:27:00 smithi047 ceph-mon[77147]: pgmap v2129: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:27:00.297 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:27:00 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:27:00.297 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:27:00 smithi047 conmon[78521]: 2022-05-14T00:27:00.079+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:27:00.297 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:27:00 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T00:27:00.298 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:27:00 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:27:00.298 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:27:00 smithi047 conmon[78521]: yield 2022-05-14T00:27:00.298 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:27:00 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:27:00.298 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:27:00 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:27:00.299 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:27:00 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:27:00.299 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:27:00 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T00:27:00.299 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:27:00 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:27:00.299 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:27:00 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T00:27:00.299 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:27:00 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:27:00.300 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:27:00 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T00:27:00.300 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:27:00 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:27:00.300 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:27:00 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T00:27:00.300 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:27:00 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:27:00.301 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:27:00 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T00:27:00.301 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:27:00 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:27:00.301 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:27:00 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:27:00.301 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:27:00 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:27:00.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:27:00 smithi115 conmon[90393]: 2022-05-14T00:27:00.001+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:27:00.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:27:00 smithi115 ceph-mon[78593]: pgmap v2129: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:27:00.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:27:00 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:27:01.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:27:01 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:27:01.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:27:01 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:27:01.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:27:01 smithi115 conmon[90393]: 2022-05-14T00:27:01.037+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:27:02.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:27:02 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:27:02.956 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T00:27:02.957 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:27:02.957 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T00:27:02.957 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:27:02.957 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:27:02.958 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:27:02.958 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T00:27:02.958 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:27:02.958 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T00:27:02.959 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:27:02.959 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T00:27:02.959 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:27:02.959 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T00:27:02.959 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:27:02.960 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T00:27:02.960 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:27:02.960 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:27:02.961 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:27:02.961 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:27:02 smithi047 ceph-mon[77147]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T00:27:02.962 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:27:02 smithi047 ceph-mon[77147]: pgmap v2130: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:27:02.962 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:27:02 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:27:02.962 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:27:02 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:27:02.963 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:27:02 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:27:02.964 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:27:02 smithi115 conmon[90393]: 2022-05-14T00:27:02.017+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:27:02.964 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:27:02 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:27:02.964 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T00:27:02.964 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:27:02.965 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T00:27:02.965 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:27:02.965 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:27:02.965 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:27:02.966 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T00:27:02.966 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:27:02.966 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T00:27:02.966 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:27:02.966 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T00:27:02.967 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:27:02.967 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T00:27:02.967 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:27:02.967 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T00:27:02.968 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:27:02.968 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:27:02.968 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:27:02.969 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:27:02 smithi115 ceph-mon[78593]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T00:27:02.969 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:27:02 smithi115 ceph-mon[78593]: pgmap v2130: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:27:02.969 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:27:02 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:27:02.969 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:27:02 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:27:02.969 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:27:02 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:27:03.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:27:03 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:27:03.292 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:27:03 smithi047 conmon[78521]: 2022-05-14T00:27:03.151+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:27:03.292 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:27:03 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T00:27:03.293 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:27:03 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:27:03.293 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:27:03 smithi047 conmon[78521]: yield 2022-05-14T00:27:03.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:27:03 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:27:03.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:27:03 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:27:03.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:27:03 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:27:03.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:27:03 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T00:27:03.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:27:03 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:27:03.297 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:27:03 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T00:27:03.297 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:27:03 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:27:03.298 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:27:03 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T00:27:03.298 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:27:03 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:27:03.298 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:27:03 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T00:27:03.299 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:27:03 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:27:03.299 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:27:03 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T00:27:03.299 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:27:03 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:27:03.300 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:27:03 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:27:03.300 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:27:03 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:27:03.300 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:27:03 smithi047 conmon[78521]: 2022-05-14T00:27:03.152+0000 7f533f777700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T00:27:03.300 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:27:03 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T00:27:03.301 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:27:03 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:27:03.301 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:27:03 smithi047 conmon[78521]: yield 2022-05-14T00:27:03.301 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:27:03 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:27:03.301 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:27:03 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:27:03.302 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:27:03 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:27:03.302 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:27:03 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T00:27:03.302 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:27:03 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:27:03.303 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:27:03 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T00:27:03.303 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:27:03 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:27:03.303 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:27:03 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T00:27:03.304 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:27:03 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:27:03.304 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:27:03 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T00:27:03.305 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:27:03 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:27:03.305 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:27:03 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T00:27:03.305 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:27:03 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:27:03.305 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:27:03 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:27:03.306 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:27:03 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:27:03.306 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:27:03 smithi047 conmon[78521]: 2022-05-14T00:27:03.306 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:27:03 smithi047 conmon[78521]: During handling of the above exception, another exception occurred: 2022-05-14T00:27:03.306 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:27:03 smithi047 conmon[78521]: 2022-05-14T00:27:03.306 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:27:03 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T00:27:03.307 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:27:03 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T00:27:03.307 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:27:03 smithi047 conmon[78521]: return f(*arg) 2022-05-14T00:27:03.308 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:27:03 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T00:27:03.308 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:27:03 smithi047 conmon[78521]: self._write_client_files(client_files, host) 2022-05-14T00:27:03.308 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:27:03 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T00:27:03.308 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:27:03 smithi047 conmon[78521]: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T00:27:03.309 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:27:03 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T00:27:03.309 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:27:03 smithi047 conmon[78521]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T00:27:03.309 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:27:03 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T00:27:03.309 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:27:03 smithi047 conmon[78521]: return self.event_loop.get_result(coro) 2022-05-14T00:27:03.309 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:27:03 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T00:27:03.310 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:27:03 smithi047 conmon[78521]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T00:27:03.310 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:27:03 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T00:27:03.310 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:27:03 smithi047 conmon[78521]: return self.__get_result() 2022-05-14T00:27:03.310 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:27:03 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T00:27:03.311 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:27:03 smithi047 conmon[78521]: raise self._exception 2022-05-14T00:27:03.311 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:27:03 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T00:27:03.311 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:27:03 smithi047 conmon[78521]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T00:27:03.311 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:27:03 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T00:27:03.312 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:27:03 smithi047 conmon[78521]: conn = await self._remote_connection(host, addr) 2022-05-14T00:27:03.312 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:27:03 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T00:27:03.312 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:27:03 smithi047 conmon[78521]: raise 2022-05-14T00:27:03.312 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:27:03 smithi047 conmon[78521]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T00:27:03.313 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:27:03 smithi047 conmon[78521]: self.gen.throw(type, value, traceback) 2022-05-14T00:27:03.313 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:27:03 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T00:27:03.313 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:27:03 smithi047 conmon[78521]: raise OrchestratorError(msg) 2022-05-14T00:27:03.313 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:27:03 smithi047 conmon[78521]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:27:03.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:27:03 smithi115 conmon[90393]: 2022-05-14T00:27:03.030+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:27:03.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:27:03 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:27:04.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:27:04 smithi047 ceph-mon[77147]: pgmap v2131: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:27:04.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:27:04 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:27:04.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:27:04 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:27:04.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T00:27:04.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:27:04.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T00:27:04.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:27:04.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:27:04.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:27:04.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T00:27:04.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:27:04.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T00:27:04.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:27:04.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T00:27:04.298 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:27:04.298 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T00:27:04.298 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:27:04.299 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T00:27:04.299 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:27:04.300 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:27:04.301 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:27:04.301 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:27:04 smithi047 ceph-mon[77147]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T00:27:04.302 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T00:27:04.302 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:27:04.303 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T00:27:04.303 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:27:04.303 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:27:04.304 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:27:04.304 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T00:27:04.305 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:27:04.305 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T00:27:04.306 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:27:04.306 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T00:27:04.307 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:27:04.307 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T00:27:04.308 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:27:04.309 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T00:27:04.309 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:27:04.309 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:27:04.309 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:27:04.310 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T00:27:04.310 INFO:journalctl@ceph.mon.b.smithi047.stdout: During handling of the above exception, another exception occurred: 2022-05-14T00:27:04.310 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T00:27:04.310 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T00:27:04.311 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T00:27:04.311 INFO:journalctl@ceph.mon.b.smithi047.stdout: return f(*arg) 2022-05-14T00:27:04.312 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T00:27:04.312 INFO:journalctl@ceph.mon.b.smithi047.stdout: self._write_client_files(client_files, host) 2022-05-14T00:27:04.312 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T00:27:04.313 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T00:27:04.313 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T00:27:04.313 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T00:27:04.313 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T00:27:04.314 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.event_loop.get_result(coro) 2022-05-14T00:27:04.314 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T00:27:04.314 INFO:journalctl@ceph.mon.b.smithi047.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T00:27:04.314 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T00:27:04.314 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.__get_result() 2022-05-14T00:27:04.315 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T00:27:04.315 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise self._exception 2022-05-14T00:27:04.315 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T00:27:04.315 INFO:journalctl@ceph.mon.b.smithi047.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T00:27:04.316 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T00:27:04.316 INFO:journalctl@ceph.mon.b.smithi047.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T00:27:04.316 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T00:27:04.316 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise 2022-05-14T00:27:04.316 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T00:27:04.317 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.gen.throw(type, value, traceback) 2022-05-14T00:27:04.317 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T00:27:04.335 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OrchestratorError(msg) 2022-05-14T00:27:04.336 INFO:journalctl@ceph.mon.b.smithi047.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:27:04.336 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:27:04 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 3555 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:27:04.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:27:03 smithi115 conmon[90393]: 2022-05-14T00:27:03.993+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:27:04.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:27:04 smithi115 ceph-mon[78593]: pgmap v2131: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:27:04.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:27:04 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:27:04.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:27:04 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:27:04.359 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T00:27:04.359 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:27:04.359 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T00:27:04.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:27:04.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:27:04.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:27:04.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T00:27:04.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:27:04.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T00:27:04.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:27:04.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T00:27:04.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:27:04.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T00:27:04.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:27:04.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T00:27:04.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:27:04.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:27:04.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:27:04.363 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:27:04 smithi115 ceph-mon[78593]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T00:27:04.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T00:27:04.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:27:04.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T00:27:04.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:27:04.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:27:04.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:27:04.365 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T00:27:04.365 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:27:04.365 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T00:27:04.365 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:27:04.366 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T00:27:04.366 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:27:04.366 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T00:27:04.366 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:27:04.366 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T00:27:04.367 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:27:04.367 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:27:04.367 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:27:04.367 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T00:27:04.367 INFO:journalctl@ceph.mon.c.smithi115.stdout: During handling of the above exception, another exception occurred: 2022-05-14T00:27:04.368 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T00:27:04.368 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T00:27:04.368 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T00:27:04.368 INFO:journalctl@ceph.mon.c.smithi115.stdout: return f(*arg) 2022-05-14T00:27:04.368 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T00:27:04.369 INFO:journalctl@ceph.mon.c.smithi115.stdout: self._write_client_files(client_files, host) 2022-05-14T00:27:04.369 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T00:27:04.369 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T00:27:04.369 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T00:27:04.370 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T00:27:04.370 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T00:27:04.370 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.event_loop.get_result(coro) 2022-05-14T00:27:04.370 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T00:27:04.371 INFO:journalctl@ceph.mon.c.smithi115.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T00:27:04.371 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T00:27:04.371 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.__get_result() 2022-05-14T00:27:04.371 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T00:27:04.371 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise self._exception 2022-05-14T00:27:04.372 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T00:27:04.372 INFO:journalctl@ceph.mon.c.smithi115.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T00:27:04.372 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T00:27:04.372 INFO:journalctl@ceph.mon.c.smithi115.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T00:27:04.372 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T00:27:04.373 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise 2022-05-14T00:27:04.373 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T00:27:04.373 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.gen.throw(type, value, traceback) 2022-05-14T00:27:04.373 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T00:27:04.373 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OrchestratorError(msg) 2022-05-14T00:27:04.374 INFO:journalctl@ceph.mon.c.smithi115.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:27:04.374 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:27:04 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 3555 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:27:05.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:27:05 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:27:05.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:27:05 smithi047 ceph-mon[77147]: pgmap v2132: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:27:05.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:27:04 smithi115 conmon[90393]: 2022-05-14T00:27:04.992+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:27:05.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:27:05 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:27:05.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:27:05 smithi115 ceph-mon[78593]: pgmap v2132: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:27:06.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:27:05 smithi115 conmon[90393]: 2022-05-14T00:27:05.974+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:27:06.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:27:06 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:27:06.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:27:06 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:27:07.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:27:06 smithi115 conmon[90393]: 2022-05-14T00:27:06.948+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:27:07.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:27:07 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:27:07.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:27:07 smithi115 ceph-mon[78593]: pgmap v2133: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:27:07.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:27:07 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:27:07.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:27:07 smithi047 ceph-mon[77147]: pgmap v2133: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:27:08.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:27:07 smithi115 conmon[90393]: 2022-05-14T00:27:07.957+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:27:08.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:27:08 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:27:08.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:27:08 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:27:09.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:27:08 smithi115 conmon[90393]: 2022-05-14T00:27:08.991+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:27:09.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:27:09 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:27:09.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:27:09 smithi115 ceph-mon[78593]: pgmap v2134: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:27:09.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:27:09 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 3560 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:27:09.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:27:09 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:27:09.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:27:09 smithi047 ceph-mon[77147]: pgmap v2134: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:27:09.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:27:09 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 3560 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:27:10.360 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:27:09 smithi115 conmon[90393]: 2022-05-14T00:27:09.950+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:27:10.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:27:10 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:27:10.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:27:10 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:27:10.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:27:10 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:27:10.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:27:10 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:27:10.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:27:10 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:27:10.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:27:10 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:27:11.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:27:10 smithi115 conmon[90393]: 2022-05-14T00:27:10.905+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:27:11.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:27:11 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:27:11.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:27:11 smithi115 ceph-mon[78593]: pgmap v2135: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:27:11.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:27:11 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:27:11.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:27:11 smithi047 ceph-mon[77147]: pgmap v2135: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:27:12.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:27:11 smithi115 conmon[90393]: 2022-05-14T00:27:11.912+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:27:12.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:27:12 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:27:12.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:27:12 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:27:13.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:27:12 smithi115 conmon[90393]: 2022-05-14T00:27:12.891+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:27:13.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:27:13 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:27:13.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:27:13 smithi115 ceph-mon[78593]: pgmap v2136: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:27:13.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:27:13 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:27:13.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:27:13 smithi047 ceph-mon[77147]: pgmap v2136: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:27:14.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:27:13 smithi115 conmon[90393]: 2022-05-14T00:27:13.864+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:27:14.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:27:14 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:27:14.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:27:14 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:27:14.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:27:14 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:27:14.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:27:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T00:27:14.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:27:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T00:27:14.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:27:14 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 3565 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:27:14.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:27:14 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:27:14.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:27:14 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:27:14.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:27:14 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:27:14.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:27:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T00:27:14.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:27:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T00:27:14.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:27:14 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 3565 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:27:15.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:27:14 smithi115 conmon[90393]: 2022-05-14T00:27:14.899+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:27:15.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:27:15 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:27:15.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:27:15 smithi115 ceph-mon[78593]: pgmap v2137: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:27:15.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:27:15 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:27:15.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:27:15 smithi047 ceph-mon[77147]: pgmap v2137: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:27:16.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:27:15 smithi115 conmon[90393]: 2022-05-14T00:27:15.941+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:27:16.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:27:16 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:27:16.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:27:16 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:27:17.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:27:16 smithi115 conmon[90393]: 2022-05-14T00:27:16.933+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:27:17.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:27:17 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:27:17.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:27:17 smithi115 ceph-mon[78593]: pgmap v2138: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:27:17.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:27:17 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:27:17.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:27:17 smithi047 ceph-mon[77147]: pgmap v2138: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:27:18.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:27:17 smithi115 conmon[90393]: 2022-05-14T00:27:17.889+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:27:18.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:27:18 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:27:18.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:27:18 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:27:18.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:27:18 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:27:18.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:27:18 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:27:18.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:27:18 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:27:18.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:27:18 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:27:19.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:27:18 smithi115 conmon[90393]: 2022-05-14T00:27:18.866+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:27:19.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:27:19 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:27:19.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:27:19 smithi115 ceph-mon[78593]: pgmap v2139: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:27:19.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:27:19 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 3570 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:27:19.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:27:19 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:27:19.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:27:19 smithi047 ceph-mon[77147]: pgmap v2139: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:27:19.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:27:19 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 3570 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:27:20.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:27:19 smithi115 conmon[90393]: 2022-05-14T00:27:19.843+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:27:20.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:27:20 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:27:20.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:27:20 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:27:21.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:27:20 smithi115 conmon[90393]: 2022-05-14T00:27:20.874+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:27:21.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:27:21 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:27:21.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:27:21 smithi115 ceph-mon[78593]: pgmap v2140: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:27:21.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:27:21 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:27:21.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:27:21 smithi047 ceph-mon[77147]: pgmap v2140: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:27:22.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:27:21 smithi115 conmon[90393]: 2022-05-14T00:27:21.861+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:27:22.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:27:22 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:27:22.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:27:22 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:27:22.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:27:22 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:27:22.555 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:27:22 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:27:22.555 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:27:22 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:27:22.556 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:27:22 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:27:23.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:27:22 smithi115 conmon[90393]: 2022-05-14T00:27:22.900+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:27:23.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:27:23 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:27:23.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:27:23 smithi115 ceph-mon[78593]: pgmap v2141: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:27:23.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:27:23 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:27:23.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:27:23 smithi047 ceph-mon[77147]: pgmap v2141: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:27:24.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:27:23 smithi115 conmon[90393]: 2022-05-14T00:27:23.889+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:27:24.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:27:24 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:27:24.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:27:24 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 3575 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:27:24.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:27:24 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:27:24.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:27:24 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 3575 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:27:25.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:27:24 smithi115 conmon[90393]: 2022-05-14T00:27:24.922+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:27:25.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:27:25 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:27:25.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:27:25 smithi115 ceph-mon[78593]: pgmap v2142: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:27:25.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:27:25 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:27:25.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:27:25 smithi047 ceph-mon[77147]: pgmap v2142: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:27:26.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:27:25 smithi115 conmon[90393]: 2022-05-14T00:27:25.909+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:27:26.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:27:26 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:27:26.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:27:26 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:27:27.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:27:26 smithi115 conmon[90393]: 2022-05-14T00:27:26.893+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:27:27.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:27:27 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:27:27.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:27:27 smithi115 ceph-mon[78593]: pgmap v2143: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:27:27.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:27:27 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:27:27.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:27:27 smithi047 ceph-mon[77147]: pgmap v2143: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:27:28.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:27:27 smithi115 conmon[90393]: 2022-05-14T00:27:27.934+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:27:28.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:27:28 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:27:28.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:27:28 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:27:29.363 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:27:28 smithi115 conmon[90393]: 2022-05-14T00:27:28.907+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:27:29.364 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:27:29 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:27:29.365 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:27:29 smithi115 ceph-mon[78593]: pgmap v2144: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:27:29.365 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:27:29 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 3580 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:27:29.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:27:29 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:27:29.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:27:29 smithi047 ceph-mon[77147]: pgmap v2144: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:27:29.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:27:29 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 3580 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:27:30.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:27:29 smithi115 conmon[90393]: 2022-05-14T00:27:29.870+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:27:30.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:27:30 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:27:30.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:27:30 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:27:30.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:27:30 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:27:30.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:27:30 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:27:30.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:27:30 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:27:30.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:27:30 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:27:31.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:27:30 smithi115 conmon[90393]: 2022-05-14T00:27:30.904+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:27:31.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:27:31 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:27:31.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:27:31 smithi115 ceph-mon[78593]: pgmap v2145: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:27:31.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:27:31 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:27:31.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:27:31 smithi047 ceph-mon[77147]: pgmap v2145: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:27:32.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:27:31 smithi115 conmon[90393]: 2022-05-14T00:27:31.932+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:27:32.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:27:32 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:27:32.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:27:32 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:27:33.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:27:32 smithi115 conmon[90393]: 2022-05-14T00:27:32.955+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:27:33.833 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:27:33 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:27:33.834 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:27:33 smithi115 ceph-mon[78593]: pgmap v2146: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:27:33.835 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:27:33 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:27:33.835 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:27:33 smithi047 ceph-mon[77147]: pgmap v2146: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:27:34.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:27:33 smithi115 conmon[90393]: 2022-05-14T00:27:33.952+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:27:34.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:27:34 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:27:34.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:27:34 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:27:34.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:27:34 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:27:34.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:27:34 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 3585 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:27:34.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:27:34 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:27:34.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:27:34 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:27:34.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:27:34 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:27:34.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:27:34 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 3585 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:27:35.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:27:34 smithi115 conmon[90393]: 2022-05-14T00:27:34.997+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:27:35.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:27:35 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:27:35.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:27:35 smithi115 ceph-mon[78593]: pgmap v2147: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:27:35.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:27:35 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:27:35.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:27:35 smithi047 ceph-mon[77147]: pgmap v2147: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:27:36.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:27:35 smithi115 conmon[90393]: 2022-05-14T00:27:35.959+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:27:36.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:27:36 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:27:36.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:27:36 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:27:37.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:27:36 smithi115 conmon[90393]: 2022-05-14T00:27:36.949+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:27:37.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:27:37 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:27:37.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:27:37 smithi115 ceph-mon[78593]: pgmap v2148: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:27:37.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:27:37 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:27:37.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:27:37 smithi047 ceph-mon[77147]: pgmap v2148: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:27:38.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:27:37 smithi115 conmon[90393]: 2022-05-14T00:27:37.973+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:27:38.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:27:38 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:27:38.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:27:38 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:27:38.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:27:38 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:27:38.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:27:38 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:27:38.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:27:38 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:27:38.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:27:38 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:27:39.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:27:38 smithi115 conmon[90393]: 2022-05-14T00:27:38.960+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:27:39.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:27:39 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:27:39.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:27:39 smithi115 ceph-mon[78593]: pgmap v2149: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:27:39.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:27:39 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 3590 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:27:39.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:27:39 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:27:39.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:27:39 smithi047 ceph-mon[77147]: pgmap v2149: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:27:39.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:27:39 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 3590 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:27:40.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:27:39 smithi115 conmon[90393]: 2022-05-14T00:27:39.994+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:27:40.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:27:40 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:27:40.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:27:40 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:27:41.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:27:40 smithi115 conmon[90393]: 2022-05-14T00:27:40.966+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:27:41.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:27:41 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:27:41.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:27:41 smithi115 ceph-mon[78593]: pgmap v2150: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 7 op/s 2022-05-14T00:27:41.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:27:41 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:27:41.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:27:41 smithi047 ceph-mon[77147]: pgmap v2150: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 7 op/s 2022-05-14T00:27:42.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:27:41 smithi115 conmon[90393]: 2022-05-14T00:27:41.975+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:27:42.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:27:42 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:27:42.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:27:42 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:27:42.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:27:42 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:27:42.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:27:42 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:27:42.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:27:42 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:27:42.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:27:42 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:27:43.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:27:42 smithi115 conmon[90393]: 2022-05-14T00:27:42.948+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:27:43.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:27:43 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:27:43.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:27:43 smithi115 ceph-mon[78593]: pgmap v2151: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 4 op/s 2022-05-14T00:27:43.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:27:43 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:27:43.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:27:43 smithi047 ceph-mon[77147]: pgmap v2151: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 4 op/s 2022-05-14T00:27:44.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:27:44 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:27:44.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:27:44 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 3595 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:27:44.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:27:43 smithi115 conmon[90393]: 2022-05-14T00:27:43.919+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:27:44.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:27:44 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:27:44.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:27:44 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 3595 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:27:45.169 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:27:44 smithi115 conmon[90393]: 2022-05-14T00:27:44.880+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:27:45.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:27:45 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:27:45.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:27:45 smithi047 ceph-mon[77147]: pgmap v2152: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 6 op/s 2022-05-14T00:27:45.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:27:45 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:27:45.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:27:45 smithi115 ceph-mon[78593]: pgmap v2152: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 6 op/s 2022-05-14T00:27:46.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:27:45 smithi115 conmon[90393]: 2022-05-14T00:27:45.834+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:27:46.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:27:46 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:27:46.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:27:46 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:27:47.175 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:27:46 smithi115 conmon[90393]: 2022-05-14T00:27:46.883+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:27:47.546 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:27:47 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:27:47.546 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:27:47 smithi047 ceph-mon[77147]: pgmap v2153: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 4 op/s 2022-05-14T00:27:47.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:27:47 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:27:47.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:27:47 smithi115 ceph-mon[78593]: pgmap v2153: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 4 op/s 2022-05-14T00:27:48.176 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:27:47 smithi115 conmon[90393]: 2022-05-14T00:27:47.872+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:27:48.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:27:48 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:27:48.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:27:48 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:27:49.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:27:48 smithi115 conmon[90393]: 2022-05-14T00:27:48.850+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:27:49.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:27:49 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:27:49.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:27:49 smithi047 ceph-mon[77147]: pgmap v2154: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 4 op/s 2022-05-14T00:27:49.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:27:49 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 3600 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:27:49.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:27:49 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:27:49.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:27:49 smithi115 ceph-mon[78593]: pgmap v2154: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 4 op/s 2022-05-14T00:27:49.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:27:49 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 3600 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:27:50.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:27:49 smithi115 conmon[90393]: 2022-05-14T00:27:49.846+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:27:50.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:27:50 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:27:50.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:27:50 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:27:50.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:27:50 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:27:50.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:27:50 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:27:50.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:27:50 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:27:50.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:27:50 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:27:51.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:27:50 smithi115 conmon[90393]: 2022-05-14T00:27:50.838+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:27:51.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:27:51 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:27:51.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:27:51 smithi047 ceph-mon[77147]: pgmap v2155: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 6 op/s 2022-05-14T00:27:51.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:27:51 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:27:51.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:27:51 smithi115 ceph-mon[78593]: pgmap v2155: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 6 op/s 2022-05-14T00:27:52.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:27:51 smithi115 conmon[90393]: 2022-05-14T00:27:51.835+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:27:52.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:27:52 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:27:52.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:27:52 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:27:53.195 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:27:52 smithi115 conmon[90393]: 2022-05-14T00:27:52.875+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:27:53.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:27:53 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:27:53.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:27:53 smithi047 ceph-mon[77147]: pgmap v2156: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:27:53.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:27:53 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:27:53.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:27:53 smithi115 ceph-mon[78593]: pgmap v2156: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:27:54.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:27:53 smithi115 conmon[90393]: 2022-05-14T00:27:53.837+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:27:54.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:27:54 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:27:54.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:27:54 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:27:54.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:27:54 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:27:54.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:27:54 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 3605 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:27:54.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:27:54 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:27:54.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:27:54 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:27:54.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:27:54 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:27:54.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:27:54 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 3605 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:27:55.201 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:27:54 smithi115 conmon[90393]: 2022-05-14T00:27:54.880+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:27:55.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:27:55 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:27:55.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:27:55 smithi047 ceph-mon[77147]: pgmap v2157: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:27:55.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:27:55 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:27:55.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:27:55 smithi115 ceph-mon[78593]: pgmap v2157: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:27:56.202 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:27:55 smithi115 conmon[90393]: 2022-05-14T00:27:55.914+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:27:56.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:27:56 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:27:56.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:27:56 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:27:57.208 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:27:56 smithi115 conmon[90393]: 2022-05-14T00:27:56.901+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:27:57.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:27:57 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:27:57.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:27:57 smithi047 ceph-mon[77147]: pgmap v2158: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:27:57.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:27:57 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:27:57.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:27:57 smithi115 ceph-mon[78593]: pgmap v2158: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:27:58.209 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:27:57 smithi115 conmon[90393]: 2022-05-14T00:27:57.941+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:27:58.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:27:58 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:27:58.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:27:58 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:27:58.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:27:58 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:27:58.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:27:58 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:27:58.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:27:58 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:27:58.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:27:58 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:27:59.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:27:58 smithi115 conmon[90393]: 2022-05-14T00:27:58.971+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:27:59.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:27:59 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:27:59.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:27:59 smithi115 ceph-mon[78593]: pgmap v2159: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:27:59.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:27:59 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 3610 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:27:59.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:27:59 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:27:59.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:27:59 smithi047 ceph-mon[77147]: pgmap v2159: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:27:59.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:27:59 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 3610 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:28:00.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:27:59 smithi115 conmon[90393]: 2022-05-14T00:27:59.975+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:28:00.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:28:00 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:28:00.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:28:00 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:28:01.221 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:28:00 smithi115 conmon[90393]: 2022-05-14T00:28:00.943+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:28:01.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:28:01 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:28:01.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:28:01 smithi047 ceph-mon[77147]: pgmap v2160: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:28:01.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:28:01 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:28:01.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:28:01 smithi115 ceph-mon[78593]: pgmap v2160: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:28:02.222 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:28:01 smithi115 conmon[90393]: 2022-05-14T00:28:01.942+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:28:02.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:28:02 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:28:02.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:28:02 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:28:02.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:28:02 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:28:02.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:28:02 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:28:02.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:28:02 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:28:02.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:28:02 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:28:03.225 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:28:02 smithi115 conmon[90393]: 2022-05-14T00:28:02.962+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:28:03.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:28:03 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:28:03.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:28:03 smithi047 ceph-mon[77147]: pgmap v2161: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:28:03.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:28:03 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T00:28:03.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:28:03 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T00:28:03.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:28:03 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T00:28:03.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:28:03 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:28:03.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:28:03 smithi115 ceph-mon[78593]: pgmap v2161: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:28:03.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:28:03 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T00:28:03.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:28:03 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T00:28:03.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:28:03 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T00:28:04.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:28:04 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:28:04.501 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:28:04 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 3615 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:28:04.501 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:28:04 smithi115 conmon[90393]: 2022-05-14T00:28:04.006+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:28:04.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:28:04 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:28:04.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:28:04 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 3615 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:28:05.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:28:05 smithi115 conmon[90393]: 2022-05-14T00:28:05.049+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:28:05.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:28:05 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:28:05.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:28:05 smithi115 ceph-mon[78593]: pgmap v2162: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:28:05.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:28:05 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:28:05.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:28:05 smithi047 ceph-mon[77147]: pgmap v2162: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:28:06.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:28:06 smithi115 conmon[90393]: 2022-05-14T00:28:06.072+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:28:06.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:28:06 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:28:06.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:28:06 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:28:06.541 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:28:06 smithi047 conmon[78521]: 2022-05-14T00:28:06.256+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:28:06.541 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:28:06 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T00:28:06.542 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:28:06 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:28:06.542 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:28:06 smithi047 conmon[78521]: yield 2022-05-14T00:28:06.542 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:28:06 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:28:06.542 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:28:06 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:28:06.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:28:06 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:28:06.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:28:06 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T00:28:06.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:28:06 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:28:06.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:28:06 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T00:28:06.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:28:06 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:28:06.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:28:06 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T00:28:06.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:28:06 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:28:06.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:28:06 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T00:28:06.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:28:06 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:28:06.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:28:06 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T00:28:06.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:28:06 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:28:06.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:28:06 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:28:06.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:28:06 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:28:07.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:28:07 smithi115 conmon[90393]: 2022-05-14T00:28:07.088+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:28:07.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:28:07 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:28:07.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:28:07 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:28:07.358 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T00:28:07.359 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:28:07.359 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T00:28:07.359 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:28:07.359 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:28:07.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:28:07.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T00:28:07.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:28:07.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T00:28:07.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:28:07.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T00:28:07.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:28:07.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T00:28:07.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:28:07.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T00:28:07.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:28:07.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:28:07.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:28:07.362 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:28:07 smithi115 ceph-mon[78593]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T00:28:07.363 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:28:07 smithi115 ceph-mon[78593]: pgmap v2163: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:28:07.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:28:07 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:28:07.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:28:07 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:28:07.541 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T00:28:07.542 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:28:07.542 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T00:28:07.542 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:28:07.542 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:28:07.542 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:28:07.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T00:28:07.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:28:07.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T00:28:07.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:28:07.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T00:28:07.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:28:07.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T00:28:07.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:28:07.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T00:28:07.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:28:07.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:28:07.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:28:07.545 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:28:07 smithi047 ceph-mon[77147]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T00:28:07.546 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:28:07 smithi047 ceph-mon[77147]: pgmap v2163: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:28:08.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:28:08 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:28:08.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:28:08 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:28:08.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:28:08 smithi115 conmon[90393]: 2022-05-14T00:28:08.113+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:28:09.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:28:09 smithi115 conmon[90393]: 2022-05-14T00:28:09.081+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:28:09.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:28:09 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:28:09.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:28:09 smithi115 ceph-mon[78593]: pgmap v2164: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:28:09.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:28:09 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 3620 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:28:09.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:28:09 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:28:09.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:28:09 smithi047 ceph-mon[77147]: pgmap v2164: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:28:09.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:28:09 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 3620 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:28:09.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:28:09 smithi047 conmon[78521]: 2022-05-14T00:28:09.328+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:28:09.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:28:09 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T00:28:09.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:28:09 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:28:09.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:28:09 smithi047 conmon[78521]: yield 2022-05-14T00:28:09.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:28:09 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:28:09.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:28:09 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:28:09.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:28:09 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:28:09.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:28:09 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T00:28:09.547 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:28:09 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:28:09.547 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:28:09 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T00:28:09.547 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:28:09 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:28:09.548 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:28:09 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T00:28:09.548 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:28:09 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:28:09.549 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:28:09 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T00:28:09.550 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:28:09 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:28:09.550 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:28:09 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T00:28:09.551 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:28:09 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:28:09.551 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:28:09 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:28:09.552 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:28:09 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:28:09.552 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:28:09 smithi047 conmon[78521]: 2022-05-14T00:28:09.329+0000 7f5344f82700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T00:28:09.553 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:28:09 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T00:28:09.553 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:28:09 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:28:09.554 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:28:09 smithi047 conmon[78521]: yield 2022-05-14T00:28:09.554 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:28:09 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:28:09.555 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:28:09 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:28:09.555 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:28:09 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:28:09.556 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:28:09 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T00:28:09.556 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:28:09 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:28:09.556 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:28:09 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T00:28:09.556 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:28:09 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:28:09.557 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:28:09 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T00:28:09.557 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:28:09 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:28:09.557 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:28:09 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T00:28:09.557 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:28:09 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:28:09.557 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:28:09 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T00:28:09.558 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:28:09 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:28:09.558 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:28:09 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:28:09.558 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:28:09 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:28:09.559 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:28:09 smithi047 conmon[78521]: 2022-05-14T00:28:09.559 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:28:09 smithi047 conmon[78521]: During handling of the above exception, another exception occurred: 2022-05-14T00:28:09.559 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:28:09 smithi047 conmon[78521]: 2022-05-14T00:28:09.560 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:28:09 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T00:28:09.560 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:28:09 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T00:28:09.560 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:28:09 smithi047 conmon[78521]: return f(*arg) 2022-05-14T00:28:09.561 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:28:09 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T00:28:09.561 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:28:09 smithi047 conmon[78521]: self._write_client_files(client_files, host) 2022-05-14T00:28:09.561 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:28:09 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T00:28:09.561 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:28:09 smithi047 conmon[78521]: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T00:28:09.562 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:28:09 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T00:28:09.562 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:28:09 smithi047 conmon[78521]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T00:28:09.562 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:28:09 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T00:28:09.562 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:28:09 smithi047 conmon[78521]: return self.event_loop.get_result(coro) 2022-05-14T00:28:09.563 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:28:09 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T00:28:09.563 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:28:09 smithi047 conmon[78521]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T00:28:09.563 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:28:09 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T00:28:09.563 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:28:09 smithi047 conmon[78521]: return self.__get_result() 2022-05-14T00:28:09.564 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:28:09 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T00:28:09.564 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:28:09 smithi047 conmon[78521]: raise self._exception 2022-05-14T00:28:09.564 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:28:09 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T00:28:09.564 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:28:09 smithi047 conmon[78521]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T00:28:09.564 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:28:09 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T00:28:09.565 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:28:09 smithi047 conmon[78521]: conn = await self._remote_connection(host, addr) 2022-05-14T00:28:09.565 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:28:09 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T00:28:09.565 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:28:09 smithi047 conmon[78521]: raise 2022-05-14T00:28:09.565 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:28:09 smithi047 conmon[78521]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T00:28:09.566 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:28:09 smithi047 conmon[78521]: self.gen.throw(type, value, traceback) 2022-05-14T00:28:09.566 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:28:09 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T00:28:09.567 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:28:09 smithi047 conmon[78521]: raise OrchestratorError(msg) 2022-05-14T00:28:09.567 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:28:09 smithi047 conmon[78521]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:28:10.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:28:10 smithi115 conmon[90393]: 2022-05-14T00:28:10.072+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:28:10.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:28:10 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:28:10.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:28:10 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:28:10.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:28:10 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:28:10.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:28:10 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:28:10.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T00:28:10.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:28:10.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T00:28:10.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:28:10.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:28:10.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:28:10.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T00:28:10.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:28:10.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T00:28:10.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:28:10.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T00:28:10.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:28:10.365 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T00:28:10.366 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:28:10.367 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T00:28:10.367 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:28:10.368 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:28:10.368 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:28:10.368 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:28:10 smithi115 ceph-mon[78593]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T00:28:10.369 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T00:28:10.370 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:28:10.370 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T00:28:10.371 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:28:10.371 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:28:10.372 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:28:10.373 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T00:28:10.373 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:28:10.374 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T00:28:10.374 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:28:10.374 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T00:28:10.375 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:28:10.375 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T00:28:10.375 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:28:10.375 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T00:28:10.376 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:28:10.376 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:28:10.376 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:28:10.377 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T00:28:10.377 INFO:journalctl@ceph.mon.c.smithi115.stdout: During handling of the above exception, another exception occurred: 2022-05-14T00:28:10.377 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T00:28:10.377 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T00:28:10.378 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T00:28:10.378 INFO:journalctl@ceph.mon.c.smithi115.stdout: return f(*arg) 2022-05-14T00:28:10.378 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T00:28:10.378 INFO:journalctl@ceph.mon.c.smithi115.stdout: self._write_client_files(client_files, host) 2022-05-14T00:28:10.379 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T00:28:10.379 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T00:28:10.379 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T00:28:10.380 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T00:28:10.380 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T00:28:10.381 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.event_loop.get_result(coro) 2022-05-14T00:28:10.381 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T00:28:10.382 INFO:journalctl@ceph.mon.c.smithi115.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T00:28:10.382 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T00:28:10.383 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.__get_result() 2022-05-14T00:28:10.383 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T00:28:10.384 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise self._exception 2022-05-14T00:28:10.384 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T00:28:10.385 INFO:journalctl@ceph.mon.c.smithi115.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T00:28:10.385 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T00:28:10.386 INFO:journalctl@ceph.mon.c.smithi115.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T00:28:10.386 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T00:28:10.386 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise 2022-05-14T00:28:10.386 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T00:28:10.387 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.gen.throw(type, value, traceback) 2022-05-14T00:28:10.387 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T00:28:10.387 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OrchestratorError(msg) 2022-05-14T00:28:10.388 INFO:journalctl@ceph.mon.c.smithi115.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:28:10.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:28:10 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:28:10.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:28:10 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:28:10.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:28:10 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:28:10.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:28:10 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:28:10.542 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T00:28:10.542 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:28:10.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T00:28:10.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:28:10.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:28:10.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:28:10.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T00:28:10.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:28:10.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T00:28:10.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:28:10.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T00:28:10.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:28:10.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T00:28:10.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:28:10.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T00:28:10.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:28:10.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:28:10.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:28:10.546 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:28:10 smithi047 ceph-mon[77147]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T00:28:10.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T00:28:10.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:28:10.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T00:28:10.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:28:10.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:28:10.548 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:28:10.548 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T00:28:10.548 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:28:10.548 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T00:28:10.549 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:28:10.549 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T00:28:10.549 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:28:10.549 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T00:28:10.550 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:28:10.550 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T00:28:10.550 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:28:10.550 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:28:10.551 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:28:10.551 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T00:28:10.551 INFO:journalctl@ceph.mon.b.smithi047.stdout: During handling of the above exception, another exception occurred: 2022-05-14T00:28:10.552 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T00:28:10.552 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T00:28:10.552 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T00:28:10.552 INFO:journalctl@ceph.mon.b.smithi047.stdout: return f(*arg) 2022-05-14T00:28:10.553 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T00:28:10.553 INFO:journalctl@ceph.mon.b.smithi047.stdout: self._write_client_files(client_files, host) 2022-05-14T00:28:10.553 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T00:28:10.553 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T00:28:10.554 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T00:28:10.554 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T00:28:10.554 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T00:28:10.554 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.event_loop.get_result(coro) 2022-05-14T00:28:10.554 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T00:28:10.555 INFO:journalctl@ceph.mon.b.smithi047.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T00:28:10.555 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T00:28:10.555 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.__get_result() 2022-05-14T00:28:10.555 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T00:28:10.556 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise self._exception 2022-05-14T00:28:10.556 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T00:28:10.556 INFO:journalctl@ceph.mon.b.smithi047.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T00:28:10.557 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T00:28:10.557 INFO:journalctl@ceph.mon.b.smithi047.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T00:28:10.557 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T00:28:10.557 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise 2022-05-14T00:28:10.558 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T00:28:10.558 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.gen.throw(type, value, traceback) 2022-05-14T00:28:10.558 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T00:28:10.558 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OrchestratorError(msg) 2022-05-14T00:28:10.559 INFO:journalctl@ceph.mon.b.smithi047.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:28:11.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:28:11 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:28:11.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:28:11 smithi047 ceph-mon[77147]: pgmap v2165: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:28:11.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:28:11 smithi115 conmon[90393]: 2022-05-14T00:28:11.115+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:28:11.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:28:11 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:28:11.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:28:11 smithi115 ceph-mon[78593]: pgmap v2165: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:28:12.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:28:12 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:28:12.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:28:12 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:28:12.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:28:12 smithi115 conmon[90393]: 2022-05-14T00:28:12.163+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:28:13.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:28:13 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:28:13.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:28:13 smithi047 ceph-mon[77147]: pgmap v2166: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:28:13.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:28:13 smithi115 conmon[90393]: 2022-05-14T00:28:13.183+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:28:13.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:28:13 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:28:13.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:28:13 smithi115 ceph-mon[78593]: pgmap v2166: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:28:14.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:28:14 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:28:14.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:28:14 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:28:14.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:28:14 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:28:14.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:28:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T00:28:14.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:28:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T00:28:14.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:28:14 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 3625 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:28:14.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:28:14 smithi115 conmon[90393]: 2022-05-14T00:28:14.205+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:28:14.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:28:14 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:28:14.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:28:14 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:28:14.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:28:14 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:28:14.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:28:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T00:28:14.612 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:28:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T00:28:14.613 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:28:14 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 3625 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:28:15.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:28:15 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:28:15.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:28:15 smithi047 ceph-mon[77147]: pgmap v2167: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:28:15.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:28:15 smithi115 conmon[90393]: 2022-05-14T00:28:15.239+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:28:15.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:28:15 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:28:15.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:28:15 smithi115 ceph-mon[78593]: pgmap v2167: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:28:16.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:28:16 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:28:16.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:28:16 smithi115 conmon[90393]: 2022-05-14T00:28:16.246+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:28:16.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:28:16 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:28:17.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:28:17 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:28:17.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:28:17 smithi047 ceph-mon[77147]: pgmap v2168: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:28:17.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:28:17 smithi115 conmon[90393]: 2022-05-14T00:28:17.269+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:28:17.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:28:17 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:28:17.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:28:17 smithi115 ceph-mon[78593]: pgmap v2168: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:28:18.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:28:18 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:28:18.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:28:18 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:28:18.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:28:18 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:28:18.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:28:18 smithi115 conmon[90393]: 2022-05-14T00:28:18.317+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:28:18.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:28:18 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:28:18.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:28:18 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:28:18.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:28:18 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:28:19.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:28:19 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:28:19.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:28:19 smithi047 ceph-mon[77147]: pgmap v2169: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:28:19.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:28:19 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 3630 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:28:19.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:28:19 smithi115 conmon[90393]: 2022-05-14T00:28:19.329+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:28:19.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:28:19 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:28:19.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:28:19 smithi115 ceph-mon[78593]: pgmap v2169: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:28:19.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:28:19 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 3630 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:28:20.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:28:20 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:28:20.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:28:20 smithi115 conmon[90393]: 2022-05-14T00:28:20.373+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:28:20.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:28:20 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:28:21.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:28:21 smithi047 ceph-mon[77147]: pgmap v2170: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:28:21.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:28:21 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:28:21.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:28:21 smithi115 conmon[90393]: 2022-05-14T00:28:21.392+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:28:21.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:28:21 smithi115 ceph-mon[78593]: pgmap v2170: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:28:21.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:28:21 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:28:22.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:28:22 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:28:22.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:28:22 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:28:22.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:28:22 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:28:22.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:28:22 smithi115 conmon[90393]: 2022-05-14T00:28:22.435+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:28:22.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:28:22 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:28:22.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:28:22 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:28:22.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:28:22 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:28:23.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:28:23 smithi115 conmon[90393]: 2022-05-14T00:28:23.475+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:28:23.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:28:23 smithi115 ceph-mon[78593]: pgmap v2171: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:28:23.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:28:23 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:28:23.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:28:23 smithi047 ceph-mon[77147]: pgmap v2171: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:28:23.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:28:23 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:28:24.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:28:24 smithi115 conmon[90393]: 2022-05-14T00:28:24.426+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:28:24.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:28:24 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:28:24.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:28:24 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 3635 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:28:24.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:28:24 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:28:24.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:28:24 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 3635 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:28:25.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:28:25 smithi115 conmon[90393]: 2022-05-14T00:28:25.446+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:28:25.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:28:25 smithi115 ceph-mon[78593]: pgmap v2172: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:28:25.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:28:25 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:28:25.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:28:25 smithi047 ceph-mon[77147]: pgmap v2172: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:28:25.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:28:25 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:28:26.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:28:26 smithi115 conmon[90393]: 2022-05-14T00:28:26.465+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:28:26.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:28:26 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:28:26.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:28:26 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:28:27.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:28:27 smithi115 conmon[90393]: 2022-05-14T00:28:27.482+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:28:27.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:28:27 smithi115 ceph-mon[78593]: pgmap v2173: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:28:27.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:28:27 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:28:27.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:28:27 smithi047 ceph-mon[77147]: pgmap v2173: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:28:27.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:28:27 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:28:28.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:28:28 smithi115 conmon[90393]: 2022-05-14T00:28:28.499+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:28:28.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:28:28 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:28:28.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:28:28 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:28:29.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:28:29 smithi115 conmon[90393]: 2022-05-14T00:28:29.456+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:28:29.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:28:29 smithi115 ceph-mon[78593]: pgmap v2174: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:28:29.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:28:29 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:28:29.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:28:29 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 3640 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:28:29.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:28:29 smithi047 ceph-mon[77147]: pgmap v2174: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:28:29.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:28:29 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:28:29.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:28:29 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 3640 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:28:30.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:28:30 smithi115 conmon[90393]: 2022-05-14T00:28:30.409+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:28:30.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:28:30 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:28:30.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:28:30 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:28:30.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:28:30 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:28:30.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:28:30 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:28:30.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:28:30 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:28:30.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:28:30 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:28:31.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:28:31 smithi115 conmon[90393]: 2022-05-14T00:28:31.365+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:28:31.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:28:31 smithi115 ceph-mon[78593]: pgmap v2175: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:28:31.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:28:31 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:28:31.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:28:31 smithi047 ceph-mon[77147]: pgmap v2175: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:28:31.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:28:31 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:28:32.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:28:32 smithi115 conmon[90393]: 2022-05-14T00:28:32.383+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:28:32.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:28:32 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:28:32.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:28:32 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:28:33.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:28:33 smithi115 conmon[90393]: 2022-05-14T00:28:33.398+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:28:33.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:28:33 smithi115 ceph-mon[78593]: pgmap v2176: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:28:33.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:28:33 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:28:33.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:28:33 smithi047 ceph-mon[77147]: pgmap v2176: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:28:33.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:28:33 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:28:34.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:28:34 smithi115 conmon[90393]: 2022-05-14T00:28:34.357+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:28:34.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:28:34 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:28:34.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:28:34 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:28:34.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:28:34 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:28:34.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:28:34 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 3645 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:28:34.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:28:34 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:28:34.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:28:34 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:28:34.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:28:34 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:28:34.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:28:34 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 3645 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:28:35.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:28:35 smithi115 conmon[90393]: 2022-05-14T00:28:35.352+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:28:35.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:28:35 smithi115 ceph-mon[78593]: pgmap v2177: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:28:35.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:28:35 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:28:35.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:28:35 smithi047 ceph-mon[77147]: pgmap v2177: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:28:35.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:28:35 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:28:36.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:28:36 smithi115 conmon[90393]: 2022-05-14T00:28:36.358+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:28:36.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:28:36 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:28:36.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:28:36 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:28:37.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:28:37 smithi115 conmon[90393]: 2022-05-14T00:28:37.322+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:28:37.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:28:37 smithi115 ceph-mon[78593]: pgmap v2178: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:28:37.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:28:37 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:28:37.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:28:37 smithi047 ceph-mon[77147]: pgmap v2178: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:28:37.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:28:37 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:28:38.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:28:38 smithi115 conmon[90393]: 2022-05-14T00:28:38.309+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:28:38.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:28:38 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:28:38.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:28:38 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:28:38.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:28:38 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:28:38.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:28:38 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:28:38.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:28:38 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:28:38.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:28:38 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:28:39.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:28:39 smithi115 conmon[90393]: 2022-05-14T00:28:39.305+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:28:39.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:28:39 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:28:39.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:28:39 smithi115 ceph-mon[78593]: pgmap v2179: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:28:39.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:28:39 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 3650 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:28:39.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:28:39 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:28:39.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:28:39 smithi047 ceph-mon[77147]: pgmap v2179: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:28:39.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:28:39 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 3650 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:28:40.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:28:40 smithi115 conmon[90393]: 2022-05-14T00:28:40.257+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:28:40.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:28:40 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:28:40.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:28:40 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:28:41.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:28:41 smithi115 conmon[90393]: 2022-05-14T00:28:41.228+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:28:41.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:28:41 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:28:41.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:28:41 smithi115 ceph-mon[78593]: pgmap v2180: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:28:41.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:28:41 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:28:41.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:28:41 smithi047 ceph-mon[77147]: pgmap v2180: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:28:42.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:28:42 smithi115 conmon[90393]: 2022-05-14T00:28:42.251+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:28:42.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:28:42 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:28:42.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:28:42 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:28:42.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:28:42 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:28:42.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:28:42 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:28:42.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:28:42 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:28:42.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:28:42 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:28:43.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:28:43 smithi115 conmon[90393]: 2022-05-14T00:28:43.241+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:28:43.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:28:43 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:28:43.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:28:43 smithi115 ceph-mon[78593]: pgmap v2181: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:28:43.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:28:43 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:28:43.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:28:43 smithi047 ceph-mon[77147]: pgmap v2181: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:28:44.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:28:44 smithi115 conmon[90393]: 2022-05-14T00:28:44.199+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:28:44.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:28:44 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:28:44.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:28:44 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 3655 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:28:44.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:28:44 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:28:44.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:28:44 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 3655 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:28:45.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:28:45 smithi115 conmon[90393]: 2022-05-14T00:28:45.199+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:28:45.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:28:45 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:28:45.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:28:45 smithi115 ceph-mon[78593]: pgmap v2182: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:28:45.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:28:45 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:28:45.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:28:45 smithi047 ceph-mon[77147]: pgmap v2182: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:28:46.421 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:28:46 smithi115 conmon[90393]: 2022-05-14T00:28:46.167+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:28:46.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:28:46 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:28:46.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:28:46 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:28:47.424 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:28:47 smithi115 conmon[90393]: 2022-05-14T00:28:47.135+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:28:47.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:28:47 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:28:47.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:28:47 smithi047 ceph-mon[77147]: pgmap v2183: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:28:47.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:28:47 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:28:47.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:28:47 smithi115 ceph-mon[78593]: pgmap v2183: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:28:48.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:28:48 smithi115 conmon[90393]: 2022-05-14T00:28:48.101+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:28:48.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:28:48 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:28:48.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:28:48 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:28:49.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:28:49 smithi115 conmon[90393]: 2022-05-14T00:28:49.099+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:28:49.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:28:49 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:28:49.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:28:49 smithi047 ceph-mon[77147]: pgmap v2184: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:28:49.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:28:49 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 3660 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:28:49.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:28:49 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:28:49.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:28:49 smithi115 ceph-mon[78593]: pgmap v2184: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:28:49.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:28:49 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 3660 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:28:50.439 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:28:50 smithi115 conmon[90393]: 2022-05-14T00:28:50.115+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:28:50.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:28:50 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:28:50.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:28:50 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:28:50.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:28:50 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:28:50.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:28:50 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:28:50.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:28:50 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:28:50.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:28:50 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:28:51.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:28:51 smithi115 conmon[90393]: 2022-05-14T00:28:51.090+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:28:51.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:28:51 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:28:51.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:28:51 smithi047 ceph-mon[77147]: pgmap v2185: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:28:51.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:28:51 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:28:51.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:28:51 smithi115 ceph-mon[78593]: pgmap v2185: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:28:52.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:28:52 smithi115 conmon[90393]: 2022-05-14T00:28:52.058+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:28:52.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:28:52 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:28:52.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:28:52 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:28:53.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:28:53 smithi115 conmon[90393]: 2022-05-14T00:28:53.054+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:28:53.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:28:53 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:28:53.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:28:53 smithi047 ceph-mon[77147]: pgmap v2186: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:28:53.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:28:53 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:28:53.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:28:53 smithi115 ceph-mon[78593]: pgmap v2186: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:28:54.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:28:54 smithi115 conmon[90393]: 2022-05-14T00:28:54.043+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:28:54.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:28:54 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:28:54.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:28:54 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:28:54.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:28:54 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:28:54.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:28:54 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 3665 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:28:54.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:28:54 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:28:54.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:28:54 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:28:54.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:28:54 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:28:54.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:28:54 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 3665 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:28:55.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:28:55 smithi115 conmon[90393]: 2022-05-14T00:28:55.063+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:28:55.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:28:55 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:28:55.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:28:55 smithi047 ceph-mon[77147]: pgmap v2187: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:28:55.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:28:55 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:28:55.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:28:55 smithi115 ceph-mon[78593]: pgmap v2187: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:28:56.459 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:28:56 smithi115 conmon[90393]: 2022-05-14T00:28:56.107+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:28:56.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:28:56 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:28:56.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:28:56 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:28:57.460 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:28:57 smithi115 conmon[90393]: 2022-05-14T00:28:57.118+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:28:57.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:28:57 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:28:57.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:28:57 smithi047 ceph-mon[77147]: pgmap v2188: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:28:57.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:28:57 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:28:57.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:28:57 smithi115 ceph-mon[78593]: pgmap v2188: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:28:58.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:28:58 smithi115 conmon[90393]: 2022-05-14T00:28:58.094+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:28:58.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:28:58 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:28:58.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:28:58 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:28:58.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:28:58 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:28:58.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:28:58 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:28:58.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:28:58 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:28:58.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:28:58 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:28:59.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:28:59 smithi115 conmon[90393]: 2022-05-14T00:28:59.081+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:28:59.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:28:59 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:28:59.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:28:59 smithi047 ceph-mon[77147]: pgmap v2189: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:28:59.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:28:59 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 3670 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:28:59.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:28:59 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:28:59.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:28:59 smithi115 ceph-mon[78593]: pgmap v2189: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:28:59.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:28:59 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 3670 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:29:00.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:29:00 smithi115 conmon[90393]: 2022-05-14T00:29:00.032+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:29:00.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:29:00 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:29:00.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:29:00 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:29:01.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:29:01 smithi115 conmon[90393]: 2022-05-14T00:29:01.009+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:29:01.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:29:01 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:29:01.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:29:01 smithi047 ceph-mon[77147]: pgmap v2190: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:29:01.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:29:01 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:29:01.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:29:01 smithi115 ceph-mon[78593]: pgmap v2190: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:29:02.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:29:02 smithi115 conmon[90393]: 2022-05-14T00:29:02.029+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:29:02.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:29:02 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:29:02.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:29:02 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:29:02.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:29:02 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:29:02.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:29:02 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:29:02.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:29:02 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:29:02.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:29:02 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:29:03.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:29:03 smithi115 conmon[90393]: 2022-05-14T00:29:03.043+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:29:03.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:29:03 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:29:03.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:29:03 smithi047 ceph-mon[77147]: pgmap v2191: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:29:03.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:29:03 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:29:03.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:29:03 smithi115 ceph-mon[78593]: pgmap v2191: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:29:04.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:29:04 smithi115 conmon[90393]: 2022-05-14T00:29:04.020+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:29:04.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:29:04 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:29:04.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:29:04 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 3675 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:29:04.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:29:04 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:29:04.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:29:04 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 3675 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:29:05.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:29:05 smithi115 conmon[90393]: 2022-05-14T00:29:05.034+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:29:05.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:29:05 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:29:05.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:29:05 smithi047 ceph-mon[77147]: pgmap v2192: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:29:05.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:29:05 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:29:05.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:29:05 smithi115 ceph-mon[78593]: pgmap v2192: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:29:06.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:29:06 smithi115 conmon[90393]: 2022-05-14T00:29:06.041+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:29:06.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:29:06 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:29:06.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:29:06 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:29:07.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:29:07 smithi115 conmon[90393]: 2022-05-14T00:29:07.025+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:29:07.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:29:07 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:29:07.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:29:07 smithi047 ceph-mon[77147]: pgmap v2193: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:29:07.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:29:07 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:29:07.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:29:07 smithi115 ceph-mon[78593]: pgmap v2193: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:29:08.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:29:08 smithi115 conmon[90393]: 2022-05-14T00:29:08.004+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:29:08.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:29:08 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:29:08.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:29:08 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:29:09.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:29:09 smithi115 conmon[90393]: 2022-05-14T00:29:09.028+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:29:09.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:29:09 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:29:09.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:29:09 smithi047 ceph-mon[77147]: pgmap v2194: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:29:09.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:29:09 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:29:09.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:29:09 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:29:09.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:29:09 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 3680 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:29:09.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:29:09 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T00:29:09.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:29:09 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T00:29:09.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:29:09 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T00:29:09.839 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:29:09 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:29:09.839 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:29:09 smithi115 ceph-mon[78593]: pgmap v2194: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:29:09.840 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:29:09 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:29:09.840 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:29:09 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:29:09.840 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:29:09 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 3680 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:29:09.841 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:29:09 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T00:29:09.841 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:29:09 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T00:29:09.841 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:29:09 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T00:29:10.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:29:10 smithi115 conmon[90393]: 2022-05-14T00:29:10.002+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:29:10.291 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:29:09 smithi047 conmon[78521]: 2022-05-14T00:29:09.936+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:29:10.291 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:29:09 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T00:29:10.291 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:29:09 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:29:10.292 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:29:09 smithi047 conmon[78521]: yield 2022-05-14T00:29:10.292 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:29:09 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:29:10.292 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:29:09 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:29:10.292 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:29:09 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:29:10.293 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:29:09 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T00:29:10.293 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:29:09 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:29:10.293 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:29:09 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T00:29:10.293 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:29:09 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:29:10.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:29:09 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T00:29:10.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:29:09 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:29:10.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:29:09 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T00:29:10.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:29:09 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:29:10.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:29:09 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T00:29:10.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:29:09 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:29:10.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:29:09 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:29:10.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:29:09 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:29:10.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:29:10 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:29:10.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:29:10 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:29:10.858 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T00:29:10.858 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:29:10.858 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T00:29:10.859 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:29:10.859 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:29:10.859 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:29:10.859 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T00:29:10.859 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:29:10.860 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T00:29:10.860 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:29:10.860 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T00:29:10.860 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:29:10.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T00:29:10.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:29:10.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T00:29:10.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:29:10.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:29:10.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:29:10.862 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:29:10 smithi115 ceph-mon[78593]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T00:29:11.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:29:10 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:29:11.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:29:10 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:29:11.041 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T00:29:11.041 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:29:11.042 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T00:29:11.042 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:29:11.042 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:29:11.042 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:29:11.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T00:29:11.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:29:11.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T00:29:11.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:29:11.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T00:29:11.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:29:11.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T00:29:11.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:29:11.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T00:29:11.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:29:11.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:29:11.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:29:11.045 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:29:10 smithi047 ceph-mon[77147]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T00:29:11.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:29:10 smithi115 conmon[90393]: 2022-05-14T00:29:10.956+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:29:11.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:29:11 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:29:11.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:29:11 smithi115 ceph-mon[78593]: pgmap v2195: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:29:12.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:29:11 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:29:12.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:29:11 smithi047 ceph-mon[77147]: pgmap v2195: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:29:12.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:29:11 smithi115 conmon[90393]: 2022-05-14T00:29:11.956+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:29:12.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:29:12 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:29:13.008 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:29:12 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:29:13.291 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:29:13 smithi047 conmon[78521]: 2022-05-14T00:29:13.007+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:29:13.291 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:29:13 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T00:29:13.291 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:29:13 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:29:13.292 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:29:13 smithi047 conmon[78521]: yield 2022-05-14T00:29:13.292 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:29:13 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:29:13.292 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:29:13 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:29:13.292 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:29:13 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:29:13.293 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:29:13 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T00:29:13.293 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:29:13 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:29:13.293 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:29:13 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T00:29:13.293 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:29:13 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:29:13.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:29:13 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T00:29:13.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:29:13 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:29:13.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:29:13 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T00:29:13.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:29:13 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:29:13.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:29:13 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T00:29:13.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:29:13 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:29:13.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:29:13 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:29:13.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:29:13 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:29:13.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:29:13 smithi047 conmon[78521]: 2022-05-14T00:29:13.008+0000 7f5344781700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T00:29:13.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:29:13 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T00:29:13.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:29:13 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:29:13.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:29:13 smithi047 conmon[78521]: yield 2022-05-14T00:29:13.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:29:13 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:29:13.297 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:29:13 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:29:13.297 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:29:13 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:29:13.297 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:29:13 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T00:29:13.297 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:29:13 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:29:13.298 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:29:13 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T00:29:13.298 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:29:13 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:29:13.298 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:29:13 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T00:29:13.298 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:29:13 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:29:13.299 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:29:13 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T00:29:13.299 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:29:13 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:29:13.299 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:29:13 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T00:29:13.299 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:29:13 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:29:13.300 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:29:13 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:29:13.300 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:29:13 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:29:13.300 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:29:13 smithi047 conmon[78521]: 2022-05-14T00:29:13.300 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:29:13 smithi047 conmon[78521]: During handling of the above exception, another exception occurred: 2022-05-14T00:29:13.301 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:29:13 smithi047 conmon[78521]: 2022-05-14T00:29:13.301 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:29:13 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T00:29:13.301 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:29:13 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T00:29:13.301 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:29:13 smithi047 conmon[78521]: return f(*arg) 2022-05-14T00:29:13.301 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:29:13 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T00:29:13.302 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:29:13 smithi047 conmon[78521]: self._write_client_files(client_files, host) 2022-05-14T00:29:13.302 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:29:13 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T00:29:13.302 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:29:13 smithi047 conmon[78521]: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T00:29:13.302 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:29:13 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T00:29:13.303 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:29:13 smithi047 conmon[78521]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T00:29:13.303 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:29:13 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T00:29:13.303 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:29:13 smithi047 conmon[78521]: return self.event_loop.get_result(coro) 2022-05-14T00:29:13.303 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:29:13 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T00:29:13.304 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:29:13 smithi047 conmon[78521]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T00:29:13.304 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:29:13 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T00:29:13.304 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:29:13 smithi047 conmon[78521]: return self.__get_result() 2022-05-14T00:29:13.304 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:29:13 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T00:29:13.304 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:29:13 smithi047 conmon[78521]: raise self._exception 2022-05-14T00:29:13.305 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:29:13 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T00:29:13.305 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:29:13 smithi047 conmon[78521]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T00:29:13.305 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:29:13 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T00:29:13.306 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:29:13 smithi047 conmon[78521]: conn = await self._remote_connection(host, addr) 2022-05-14T00:29:13.306 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:29:13 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T00:29:13.306 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:29:13 smithi047 conmon[78521]: raise 2022-05-14T00:29:13.306 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:29:13 smithi047 conmon[78521]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T00:29:13.307 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:29:13 smithi047 conmon[78521]: self.gen.throw(type, value, traceback) 2022-05-14T00:29:13.307 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:29:13 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T00:29:13.307 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:29:13 smithi047 conmon[78521]: raise OrchestratorError(msg) 2022-05-14T00:29:13.307 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:29:13 smithi047 conmon[78521]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:29:13.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:29:12 smithi115 conmon[90393]: 2022-05-14T00:29:12.924+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:29:13.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:29:13 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:29:13.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:29:13 smithi115 ceph-mon[78593]: pgmap v2196: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:29:13.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:29:13 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:29:13.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:29:13 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:29:13.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:29:13 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:29:13.859 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T00:29:13.860 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:29:13.860 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T00:29:13.860 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:29:13.860 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:29:13.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:29:13.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T00:29:13.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:29:13.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T00:29:13.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:29:13.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T00:29:13.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:29:13.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T00:29:13.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:29:13.863 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T00:29:13.863 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:29:13.863 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:29:13.863 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:29:13.863 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:29:13 smithi115 ceph-mon[78593]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T00:29:13.864 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T00:29:13.864 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:29:13.864 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T00:29:13.864 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:29:13.864 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:29:13.865 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:29:13.865 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T00:29:13.865 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:29:13.865 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T00:29:13.866 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:29:13.866 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T00:29:13.866 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:29:13.866 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T00:29:13.867 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:29:13.867 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T00:29:13.867 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:29:13.867 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:29:13.868 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:29:13.868 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T00:29:13.868 INFO:journalctl@ceph.mon.c.smithi115.stdout: During handling of the above exception, another exception occurred: 2022-05-14T00:29:13.868 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T00:29:13.869 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T00:29:13.869 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T00:29:13.869 INFO:journalctl@ceph.mon.c.smithi115.stdout: return f(*arg) 2022-05-14T00:29:13.869 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T00:29:13.870 INFO:journalctl@ceph.mon.c.smithi115.stdout: self._write_client_files(client_files, host) 2022-05-14T00:29:13.870 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T00:29:13.870 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T00:29:13.870 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T00:29:13.870 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T00:29:13.871 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T00:29:13.871 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.event_loop.get_result(coro) 2022-05-14T00:29:13.871 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T00:29:13.871 INFO:journalctl@ceph.mon.c.smithi115.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T00:29:13.871 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T00:29:13.872 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.__get_result() 2022-05-14T00:29:13.872 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T00:29:13.872 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise self._exception 2022-05-14T00:29:13.872 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T00:29:13.872 INFO:journalctl@ceph.mon.c.smithi115.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T00:29:13.873 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T00:29:13.873 INFO:journalctl@ceph.mon.c.smithi115.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T00:29:13.873 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T00:29:13.873 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise 2022-05-14T00:29:13.874 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T00:29:13.874 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.gen.throw(type, value, traceback) 2022-05-14T00:29:13.874 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T00:29:13.874 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OrchestratorError(msg) 2022-05-14T00:29:13.875 INFO:journalctl@ceph.mon.c.smithi115.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:29:13.875 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:29:13 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T00:29:13.875 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:29:13 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T00:29:14.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:29:13 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:29:14.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:29:13 smithi047 ceph-mon[77147]: pgmap v2196: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:29:14.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:29:13 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:29:14.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:29:13 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:29:14.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:29:13 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:29:14.042 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T00:29:14.042 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:29:14.042 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T00:29:14.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:29:14.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:29:14.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:29:14.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T00:29:14.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:29:14.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T00:29:14.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:29:14.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T00:29:14.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:29:14.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T00:29:14.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:29:14.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T00:29:14.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:29:14.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:29:14.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:29:14.046 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:29:13 smithi047 ceph-mon[77147]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T00:29:14.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T00:29:14.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:29:14.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T00:29:14.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:29:14.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:29:14.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:29:14.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T00:29:14.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:29:14.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T00:29:14.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:29:14.049 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T00:29:14.049 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:29:14.049 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T00:29:14.049 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:29:14.049 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T00:29:14.050 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:29:14.050 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:29:14.050 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:29:14.050 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T00:29:14.051 INFO:journalctl@ceph.mon.b.smithi047.stdout: During handling of the above exception, another exception occurred: 2022-05-14T00:29:14.051 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T00:29:14.051 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T00:29:14.051 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T00:29:14.051 INFO:journalctl@ceph.mon.b.smithi047.stdout: return f(*arg) 2022-05-14T00:29:14.052 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T00:29:14.052 INFO:journalctl@ceph.mon.b.smithi047.stdout: self._write_client_files(client_files, host) 2022-05-14T00:29:14.052 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T00:29:14.052 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T00:29:14.053 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T00:29:14.053 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T00:29:14.053 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T00:29:14.053 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.event_loop.get_result(coro) 2022-05-14T00:29:14.053 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T00:29:14.054 INFO:journalctl@ceph.mon.b.smithi047.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T00:29:14.054 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T00:29:14.054 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.__get_result() 2022-05-14T00:29:14.054 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T00:29:14.055 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise self._exception 2022-05-14T00:29:14.055 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T00:29:14.055 INFO:journalctl@ceph.mon.b.smithi047.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T00:29:14.055 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T00:29:14.055 INFO:journalctl@ceph.mon.b.smithi047.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T00:29:14.056 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T00:29:14.056 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise 2022-05-14T00:29:14.056 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T00:29:14.056 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.gen.throw(type, value, traceback) 2022-05-14T00:29:14.056 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T00:29:14.057 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OrchestratorError(msg) 2022-05-14T00:29:14.057 INFO:journalctl@ceph.mon.b.smithi047.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:29:14.057 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:29:13 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T00:29:14.057 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:29:13 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T00:29:14.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:29:13 smithi115 conmon[90393]: 2022-05-14T00:29:13.892+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:29:14.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:29:14 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:29:14.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:29:14 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 3685 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:29:15.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:29:14 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:29:15.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:29:14 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 3685 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:29:15.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:29:14 smithi115 conmon[90393]: 2022-05-14T00:29:14.877+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:29:15.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:29:15 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:29:15.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:29:15 smithi115 ceph-mon[78593]: pgmap v2197: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:29:16.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:29:15 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:29:16.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:29:15 smithi047 ceph-mon[77147]: pgmap v2197: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:29:16.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:29:15 smithi115 conmon[90393]: 2022-05-14T00:29:15.920+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:29:16.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:29:16 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:29:17.067 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:29:16 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:29:17.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:29:16 smithi115 conmon[90393]: 2022-05-14T00:29:16.924+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:29:17.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:29:17 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:29:17.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:29:17 smithi115 ceph-mon[78593]: pgmap v2198: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:29:17.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:29:17 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:29:17.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:29:17 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:29:18.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:29:17 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:29:18.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:29:17 smithi047 ceph-mon[77147]: pgmap v2198: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:29:18.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:29:17 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:29:18.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:29:17 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:29:18.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:29:17 smithi115 conmon[90393]: 2022-05-14T00:29:17.917+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:29:18.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:29:18 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:29:19.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:29:18 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:29:19.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:29:18 smithi115 conmon[90393]: 2022-05-14T00:29:18.884+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:29:19.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:29:19 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:29:19.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:29:19 smithi115 ceph-mon[78593]: pgmap v2199: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:29:19.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:29:19 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 3690 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:29:20.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:29:19 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:29:20.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:29:19 smithi047 ceph-mon[77147]: pgmap v2199: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:29:20.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:29:19 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 3690 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:29:20.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:29:19 smithi115 conmon[90393]: 2022-05-14T00:29:19.892+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:29:20.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:29:20 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:29:21.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:29:20 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:29:21.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:29:20 smithi115 conmon[90393]: 2022-05-14T00:29:20.869+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:29:21.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:29:21 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:29:21.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:29:21 smithi115 ceph-mon[78593]: pgmap v2200: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:29:21.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:29:21 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:29:21.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:29:21 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:29:22.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:29:21 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:29:22.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:29:21 smithi047 ceph-mon[77147]: pgmap v2200: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:29:22.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:29:21 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:29:22.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:29:21 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:29:22.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:29:21 smithi115 conmon[90393]: 2022-05-14T00:29:21.867+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:29:22.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:29:22 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:29:23.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:29:22 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:29:23.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:29:22 smithi115 conmon[90393]: 2022-05-14T00:29:22.914+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:29:23.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:29:23 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:29:23.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:29:23 smithi115 ceph-mon[78593]: pgmap v2201: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:29:24.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:29:23 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:29:24.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:29:23 smithi047 ceph-mon[77147]: pgmap v2201: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:29:24.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:29:23 smithi115 conmon[90393]: 2022-05-14T00:29:23.939+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:29:24.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:29:24 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:29:24.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:29:24 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 3695 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:29:25.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:29:24 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:29:25.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:29:24 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 3695 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:29:25.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:29:24 smithi115 conmon[90393]: 2022-05-14T00:29:24.894+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:29:25.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:29:25 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:29:25.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:29:25 smithi115 ceph-mon[78593]: pgmap v2202: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:29:26.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:29:25 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:29:26.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:29:25 smithi047 ceph-mon[77147]: pgmap v2202: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:29:26.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:29:25 smithi115 conmon[90393]: 2022-05-14T00:29:25.874+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:29:26.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:29:26 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:29:27.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:29:26 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:29:27.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:29:26 smithi115 conmon[90393]: 2022-05-14T00:29:26.867+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:29:27.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:29:27 smithi115 conmon[90393]: 2022-05-14T00:29:27.839+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:29:27.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:29:27 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:29:27.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:29:27 smithi115 ceph-mon[78593]: pgmap v2203: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:29:28.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:29:27 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:29:28.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:29:27 smithi047 ceph-mon[77147]: pgmap v2203: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:29:29.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:29:28 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:29:29.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:29:28 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:29:29.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:29:28 smithi115 conmon[90393]: 2022-05-14T00:29:28.819+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:29:30.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:29:29 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:29:30.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:29:29 smithi047 ceph-mon[77147]: pgmap v2204: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:29:30.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:29:29 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:29:30.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:29:29 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:29:30.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:29:29 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 3700 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:29:30.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:29:29 smithi115 conmon[90393]: 2022-05-14T00:29:29.842+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:29:30.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:29:29 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:29:30.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:29:29 smithi115 ceph-mon[78593]: pgmap v2204: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:29:30.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:29:29 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:29:30.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:29:29 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:29:30.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:29:29 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 3700 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:29:31.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:29:30 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:29:31.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:29:30 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:29:31.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:29:30 smithi115 conmon[90393]: 2022-05-14T00:29:30.825+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:29:32.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:29:31 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:29:32.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:29:31 smithi047 ceph-mon[77147]: pgmap v2205: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:29:32.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:29:31 smithi115 conmon[90393]: 2022-05-14T00:29:31.838+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:29:32.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:29:31 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:29:32.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:29:31 smithi115 ceph-mon[78593]: pgmap v2205: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:29:33.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:29:32 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:29:33.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:29:32 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:29:33.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:29:32 smithi115 conmon[90393]: 2022-05-14T00:29:32.823+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:29:34.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:29:33 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:29:34.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:29:33 smithi047 ceph-mon[77147]: pgmap v2206: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:29:34.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:29:33 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:29:34.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:29:33 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:29:34.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:29:33 smithi115 conmon[90393]: 2022-05-14T00:29:33.779+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:29:34.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:29:33 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:29:34.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:29:33 smithi115 ceph-mon[78593]: pgmap v2206: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:29:34.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:29:33 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:29:34.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:29:33 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:29:35.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:29:34 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:29:35.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:29:34 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 3705 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:29:35.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:29:34 smithi115 conmon[90393]: 2022-05-14T00:29:34.731+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:29:35.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:29:34 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:29:35.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:29:34 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 3705 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:29:36.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:29:35 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:29:36.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:29:35 smithi047 ceph-mon[77147]: pgmap v2207: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:29:36.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:29:35 smithi115 conmon[90393]: 2022-05-14T00:29:35.725+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:29:36.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:29:35 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:29:36.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:29:35 smithi115 ceph-mon[78593]: pgmap v2207: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:29:37.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:29:36 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:29:37.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:29:36 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:29:37.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:29:36 smithi115 conmon[90393]: 2022-05-14T00:29:36.722+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:29:38.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:29:37 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:29:38.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:29:37 smithi047 ceph-mon[77147]: pgmap v2208: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:29:38.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:29:37 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:29:38.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:29:37 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:29:38.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:29:37 smithi115 conmon[90393]: 2022-05-14T00:29:37.738+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:29:38.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:29:37 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:29:38.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:29:37 smithi115 ceph-mon[78593]: pgmap v2208: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:29:38.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:29:37 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:29:38.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:29:37 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:29:39.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:29:38 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:29:39.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:29:38 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:29:39.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:29:38 smithi115 conmon[90393]: 2022-05-14T00:29:38.752+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:29:40.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:29:39 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:29:40.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:29:39 smithi047 ceph-mon[77147]: pgmap v2209: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:29:40.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:29:39 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 3710 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:29:40.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:29:39 smithi115 conmon[90393]: 2022-05-14T00:29:39.729+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:29:40.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:29:39 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:29:40.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:29:39 smithi115 ceph-mon[78593]: pgmap v2209: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:29:40.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:29:39 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 3710 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:29:41.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:29:40 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:29:41.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:29:40 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:29:41.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:29:40 smithi115 conmon[90393]: 2022-05-14T00:29:40.726+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:29:42.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:29:41 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:29:42.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:29:41 smithi047 ceph-mon[77147]: pgmap v2210: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:29:42.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:29:41 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:29:42.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:29:41 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:29:42.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:29:41 smithi115 conmon[90393]: 2022-05-14T00:29:41.755+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:29:42.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:29:41 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:29:42.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:29:41 smithi115 ceph-mon[78593]: pgmap v2210: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:29:42.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:29:41 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:29:42.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:29:41 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:29:43.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:29:42 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:29:43.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:29:42 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:29:43.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:29:42 smithi115 conmon[90393]: 2022-05-14T00:29:42.722+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:29:44.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:29:43 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:29:44.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:29:43 smithi047 ceph-mon[77147]: pgmap v2211: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:29:44.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:29:43 smithi115 conmon[90393]: 2022-05-14T00:29:43.698+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:29:44.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:29:43 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:29:44.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:29:43 smithi115 ceph-mon[78593]: pgmap v2211: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:29:45.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:29:44 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:29:45.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:29:44 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:29:45.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:29:44 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 3715 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:29:45.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:29:44 smithi115 conmon[90393]: 2022-05-14T00:29:44.726+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:29:45.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:29:44 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:29:45.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:29:44 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:29:45.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:29:44 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 3715 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:29:46.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:29:45 smithi115 conmon[90393]: 2022-05-14T00:29:45.721+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:29:46.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:29:45 smithi115 ceph-mon[78593]: pgmap v2212: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:29:46.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:29:45 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:29:46.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:29:45 smithi047 ceph-mon[77147]: pgmap v2212: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:29:46.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:29:45 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:29:47.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:29:46 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:29:47.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:29:46 smithi115 conmon[90393]: 2022-05-14T00:29:46.727+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:29:47.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:29:46 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:29:48.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:29:47 smithi115 conmon[90393]: 2022-05-14T00:29:47.722+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:29:48.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:29:47 smithi115 ceph-mon[78593]: pgmap v2213: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:29:48.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:29:47 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:29:48.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:29:47 smithi047 ceph-mon[77147]: pgmap v2213: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:29:48.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:29:47 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:29:48.969 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:29:48 smithi115 conmon[90393]: 2022-05-14T00:29:48.676+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:29:49.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:29:48 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:29:49.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:29:48 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:29:49.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:29:48 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 3720 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:29:49.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:29:48 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:29:49.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:29:48 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:29:49.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:29:48 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 3720 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:29:49.970 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:29:49 smithi115 conmon[90393]: 2022-05-14T00:29:49.706+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:29:50.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:29:49 smithi047 ceph-mon[77147]: pgmap v2214: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:29:50.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:29:49 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:29:50.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:29:49 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:29:50.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:29:49 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:29:50.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:29:49 smithi115 ceph-mon[78593]: pgmap v2214: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:29:50.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:29:49 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:29:50.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:29:49 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:29:50.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:29:49 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:29:50.975 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:29:50 smithi115 conmon[90393]: 2022-05-14T00:29:50.677+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:29:50.975 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:29:50 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:29:51.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:29:50 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:29:51.976 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:29:51 smithi115 conmon[90393]: 2022-05-14T00:29:51.697+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:29:52.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:29:51 smithi047 ceph-mon[77147]: pgmap v2215: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:29:52.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:29:51 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:29:52.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:29:51 smithi115 ceph-mon[78593]: pgmap v2215: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:29:52.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:29:51 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:29:52.982 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:29:52 smithi115 conmon[90393]: 2022-05-14T00:29:52.652+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:29:53.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:29:52 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:29:53.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:29:52 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:29:53.985 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:29:53 smithi115 conmon[90393]: 2022-05-14T00:29:53.669+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:29:54.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:29:53 smithi047 ceph-mon[77147]: pgmap v2216: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:29:54.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:29:53 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:29:54.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:29:53 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:29:54.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:29:53 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:29:54.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:29:53 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 3725 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:29:54.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:29:53 smithi115 ceph-mon[78593]: pgmap v2216: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:29:54.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:29:53 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:29:54.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:29:53 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:29:54.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:29:53 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:29:54.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:29:53 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 3725 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:29:54.987 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:29:54 smithi115 conmon[90393]: 2022-05-14T00:29:54.638+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:29:55.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:29:54 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:29:55.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:29:54 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:29:55.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:29:55 smithi115 conmon[90393]: 2022-05-14T00:29:55.597+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:29:56.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:29:55 smithi047 ceph-mon[77147]: pgmap v2217: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:29:56.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:29:55 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:29:56.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:29:55 smithi115 ceph-mon[78593]: pgmap v2217: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:29:56.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:29:55 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:29:56.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:29:56 smithi115 conmon[90393]: 2022-05-14T00:29:56.599+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:29:57.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:29:56 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:29:57.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:29:56 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:29:57.999 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:29:57 smithi115 conmon[90393]: 2022-05-14T00:29:57.635+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:29:58.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:29:57 smithi047 ceph-mon[77147]: pgmap v2218: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:29:58.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:29:57 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:29:58.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:29:57 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:29:58.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:29:57 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:29:58.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:29:57 smithi115 ceph-mon[78593]: pgmap v2218: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:29:58.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:29:57 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:29:58.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:29:57 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:29:58.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:29:57 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:29:58.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:29:58 smithi115 conmon[90393]: 2022-05-14T00:29:58.589+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:29:59.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:29:58 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:29:59.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:29:58 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 3730 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:29:59.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:29:58 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:29:59.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:29:59 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 3730 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:29:59.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:29:59 smithi115 conmon[90393]: 2022-05-14T00:29:59.578+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:30:00.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:30:00 smithi047 conmon[77123]: 2022-05-14T00:29:59.998+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : Health detail: HEALTH_ERR 1 filesystem is degraded; 1 mds daemon damaged; 1 MDSs are read only; 1 MDSs report slow metadata IOs; 1/3 mons down, quorum b,c; Reduced data availability: 1 pg inactive; 9 slow ops, oldest one blocked for 3730 sec, osd.5 has slow ops 2022-05-14T00:30:00.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:30:00 smithi047 conmon[77123]: 2022-05-14T00:29:59.998+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : [WRN] FS_DEGRADED: 1 filesystem is degraded 2022-05-14T00:30:00.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:30:00 smithi047 conmon[77123]: 2022-05-14T00:29:59.998+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : fs cephfs is degraded 2022-05-14T00:30:00.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:30:00 smithi047 conmon[77123]: 2022-05-14T00:29:59.998+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : [ERR] MDS_DAMAGE: 1 mds daemon damaged 2022-05-14T00:30:00.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:30:00 smithi047 conmon[77123]: 2022-05-14T00:29:59.998+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : fs cephfs mds.2 is damaged 2022-05-14T00:30:00.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:30:00 smithi047 conmon[77123]: 2022-05-14T00:29:59.998+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : [WRN] MDS_READ_ONLY: 1 MDSs are read only 2022-05-14T00:30:00.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:30:00 smithi047 conmon[77123]: 2022-05-14T00:29:59.998+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : mds.l(mds.0): MDS in read-only mode 2022-05-14T00:30:00.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:30:00 smithi047 conmon[77123]: 2022-05-14T00:29:59.998+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : [WRN] MDS_SLOW_METADATA_IO: 1 MDSs report slow metadata IOs 2022-05-14T00:30:00.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:30:00 smithi047 conmon[77123]: 2022-05-14T00:29:59.998+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : mds.l(mds.0): 4 slow metadata IOs are blocked > 180 secs, oldest blocked for 4033 secs 2022-05-14T00:30:00.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:30:00 smithi047 conmon[77123]: 2022-05-14T00:29:59.998+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : [WRN] MON_DOWN: 1/3 mons down, quorum b,c 2022-05-14T00:30:00.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:30:00 smithi047 conmon[77123]: 2022-05-14T00:29:59.998+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : mon.a (rank 0) addr [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] is down (out of quorum) 2022-05-14T00:30:00.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:30:00 smithi047 conmon[77123]: 2022-05-14T00:29:59.998+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : [WRN] PG_AVAILABILITY: Reduced data availability: 1 pg inactive 2022-05-14T00:30:00.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:30:00 smithi047 conmon[77123]: 2022-05-14T00:29:59.998+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : pg 2.1 is stuck inactive for 74m, current state unknown, last acting [] 2022-05-14T00:30:00.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:30:00 smithi047 conmon[77123]: 2022-05-14T00:29:59.998+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : [WRN] SLOW_OPS: 9 slow ops, oldest one blocked for 3730 sec, osd.5 has slow ops 2022-05-14T00:30:00.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:30:00 smithi047 ceph-mon[77147]: pgmap v2219: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:30:00.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:30:00 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:30:00.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:30:00 smithi115 ceph-mon[78593]: pgmap v2219: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:30:00.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:30:00 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:30:00.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:30:00 smithi115 conmon[90393]: 2022-05-14T00:30:00.543+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:30:01.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:30:01 smithi047 ceph-mon[77147]: Health detail: HEALTH_ERR 1 filesystem is degraded; 1 mds daemon damaged; 1 MDSs are read only; 1 MDSs report slow metadata IOs; 1/3 mons down, quorum b,c; Reduced data availability: 1 pg inactive; 9 slow ops, oldest one blocked for 3730 sec, osd.5 has slow ops 2022-05-14T00:30:01.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:30:01 smithi047 ceph-mon[77147]: [WRN] FS_DEGRADED: 1 filesystem is degraded 2022-05-14T00:30:01.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:30:01 smithi047 ceph-mon[77147]: fs cephfs is degraded 2022-05-14T00:30:01.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:30:01 smithi047 ceph-mon[77147]: [ERR] MDS_DAMAGE: 1 mds daemon damaged 2022-05-14T00:30:01.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:30:01 smithi047 ceph-mon[77147]: fs cephfs mds.2 is damaged 2022-05-14T00:30:01.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:30:01 smithi047 ceph-mon[77147]: [WRN] MDS_READ_ONLY: 1 MDSs are read only 2022-05-14T00:30:01.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:30:01 smithi047 ceph-mon[77147]: mds.l(mds.0): MDS in read-only mode 2022-05-14T00:30:01.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:30:01 smithi047 ceph-mon[77147]: [WRN] MDS_SLOW_METADATA_IO: 1 MDSs report slow metadata IOs 2022-05-14T00:30:01.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:30:01 smithi047 ceph-mon[77147]: mds.l(mds.0): 4 slow metadata IOs are blocked > 180 secs, oldest blocked for 4033 secs 2022-05-14T00:30:01.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:30:01 smithi047 ceph-mon[77147]: [WRN] MON_DOWN: 1/3 mons down, quorum b,c 2022-05-14T00:30:01.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:30:01 smithi047 ceph-mon[77147]: mon.a (rank 0) addr [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] is down (out of quorum) 2022-05-14T00:30:01.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:30:01 smithi047 ceph-mon[77147]: [WRN] PG_AVAILABILITY: Reduced data availability: 1 pg inactive 2022-05-14T00:30:01.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:30:01 smithi047 ceph-mon[77147]: pg 2.1 is stuck inactive for 74m, current state unknown, last acting [] 2022-05-14T00:30:01.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:30:01 smithi047 ceph-mon[77147]: [WRN] SLOW_OPS: 9 slow ops, oldest one blocked for 3730 sec, osd.5 has slow ops 2022-05-14T00:30:01.295 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:30:01 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:30:01.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:30:01 smithi115 ceph-mon[78593]: Health detail: HEALTH_ERR 1 filesystem is degraded; 1 mds daemon damaged; 1 MDSs are read only; 1 MDSs report slow metadata IOs; 1/3 mons down, quorum b,c; Reduced data availability: 1 pg inactive; 9 slow ops, oldest one blocked for 3730 sec, osd.5 has slow ops 2022-05-14T00:30:01.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:30:01 smithi115 ceph-mon[78593]: [WRN] FS_DEGRADED: 1 filesystem is degraded 2022-05-14T00:30:01.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:30:01 smithi115 ceph-mon[78593]: fs cephfs is degraded 2022-05-14T00:30:01.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:30:01 smithi115 ceph-mon[78593]: [ERR] MDS_DAMAGE: 1 mds daemon damaged 2022-05-14T00:30:01.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:30:01 smithi115 ceph-mon[78593]: fs cephfs mds.2 is damaged 2022-05-14T00:30:01.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:30:01 smithi115 ceph-mon[78593]: [WRN] MDS_READ_ONLY: 1 MDSs are read only 2022-05-14T00:30:01.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:30:01 smithi115 ceph-mon[78593]: mds.l(mds.0): MDS in read-only mode 2022-05-14T00:30:01.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:30:01 smithi115 ceph-mon[78593]: [WRN] MDS_SLOW_METADATA_IO: 1 MDSs report slow metadata IOs 2022-05-14T00:30:01.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:30:01 smithi115 ceph-mon[78593]: mds.l(mds.0): 4 slow metadata IOs are blocked > 180 secs, oldest blocked for 4033 secs 2022-05-14T00:30:01.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:30:01 smithi115 ceph-mon[78593]: [WRN] MON_DOWN: 1/3 mons down, quorum b,c 2022-05-14T00:30:01.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:30:01 smithi115 ceph-mon[78593]: mon.a (rank 0) addr [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] is down (out of quorum) 2022-05-14T00:30:01.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:30:01 smithi115 ceph-mon[78593]: [WRN] PG_AVAILABILITY: Reduced data availability: 1 pg inactive 2022-05-14T00:30:01.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:30:01 smithi115 ceph-mon[78593]: pg 2.1 is stuck inactive for 74m, current state unknown, last acting [] 2022-05-14T00:30:01.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:30:01 smithi115 ceph-mon[78593]: [WRN] SLOW_OPS: 9 slow ops, oldest one blocked for 3730 sec, osd.5 has slow ops 2022-05-14T00:30:01.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:30:01 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:30:01.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:30:01 smithi115 conmon[90393]: 2022-05-14T00:30:01.500+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:30:02.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:30:02 smithi047 ceph-mon[77147]: pgmap v2220: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:30:02.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:30:02 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:30:02.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:30:02 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:30:02.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:30:02 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:30:02.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:30:02 smithi115 ceph-mon[78593]: pgmap v2220: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:30:02.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:30:02 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:30:02.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:30:02 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:30:02.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:30:02 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:30:02.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:30:02 smithi115 conmon[90393]: 2022-05-14T00:30:02.463+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:30:03.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:30:03 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:30:03.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:30:03 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:30:03.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:30:03 smithi115 conmon[90393]: 2022-05-14T00:30:03.479+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:30:04.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:30:04 smithi047 ceph-mon[77147]: pgmap v2221: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:30:04.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:30:04 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:30:04.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:30:04 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 3735 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:30:04.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:30:04 smithi115 ceph-mon[78593]: pgmap v2221: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:30:04.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:30:04 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:30:04.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:30:04 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 3735 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:30:04.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:30:04 smithi115 conmon[90393]: 2022-05-14T00:30:04.502+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:30:05.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:30:05 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:30:05.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:30:05 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:30:05.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:30:05 smithi115 conmon[90393]: 2022-05-14T00:30:05.459+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:30:06.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:30:06 smithi047 ceph-mon[77147]: pgmap v2222: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:30:06.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:30:06 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:30:06.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:30:06 smithi115 ceph-mon[78593]: pgmap v2222: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:30:06.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:30:06 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:30:06.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:30:06 smithi115 conmon[90393]: 2022-05-14T00:30:06.469+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:30:07.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:30:07 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:30:07.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:30:07 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:30:07.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:30:07 smithi115 conmon[90393]: 2022-05-14T00:30:07.497+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:30:08.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:30:08 smithi047 ceph-mon[77147]: pgmap v2223: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:30:08.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:30:08 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:30:08.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:30:08 smithi115 ceph-mon[78593]: pgmap v2223: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:30:08.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:30:08 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:30:08.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:30:08 smithi115 conmon[90393]: 2022-05-14T00:30:08.541+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:30:09.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:30:09 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:30:09.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:30:09 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 3740 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:30:09.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:30:09 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:30:09.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:30:09 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 3740 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:30:09.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:30:09 smithi115 conmon[90393]: 2022-05-14T00:30:09.512+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:30:10.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:30:10 smithi047 ceph-mon[77147]: pgmap v2224: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:30:10.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:30:10 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:30:10.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:30:10 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:30:10.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:30:10 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:30:10.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:30:10 smithi115 ceph-mon[78593]: pgmap v2224: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:30:10.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:30:10 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:30:10.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:30:10 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:30:10.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:30:10 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:30:10.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:30:10 smithi115 conmon[90393]: 2022-05-14T00:30:10.560+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:30:11.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:30:11 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:30:11.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:30:11 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:30:11.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:30:11 smithi115 conmon[90393]: 2022-05-14T00:30:11.569+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:30:12.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:30:12 smithi047 ceph-mon[77147]: pgmap v2225: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:30:12.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:30:12 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:30:12.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:30:12 smithi115 ceph-mon[78593]: pgmap v2225: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:30:12.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:30:12 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:30:12.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:30:12 smithi115 conmon[90393]: 2022-05-14T00:30:12.526+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:30:13.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:30:13 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:30:13.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:30:13 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T00:30:13.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:30:13 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T00:30:13.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:30:13 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T00:30:13.534 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:30:13 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:30:13.534 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:30:13 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T00:30:13.535 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:30:13 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T00:30:13.535 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:30:13 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T00:30:13.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:30:13 smithi115 conmon[90393]: 2022-05-14T00:30:13.514+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:30:14.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:30:14 smithi115 ceph-mon[78593]: pgmap v2226: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:30:14.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:30:14 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:30:14.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:30:14 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:30:14.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:30:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T00:30:14.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:30:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T00:30:14.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:30:14 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:30:14.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:30:14 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 3745 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:30:14.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:30:14 smithi047 ceph-mon[77147]: pgmap v2226: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:30:14.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:30:14 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:30:14.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:30:14 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:30:14.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:30:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T00:30:14.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:30:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T00:30:14.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:30:14 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:30:14.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:30:14 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 3745 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:30:14.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:30:14 smithi115 conmon[90393]: 2022-05-14T00:30:14.515+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:30:15.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:30:15 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:30:15.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:30:15 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:30:15.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:30:15 smithi115 conmon[90393]: 2022-05-14T00:30:15.557+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:30:16.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:30:16 smithi115 ceph-mon[78593]: pgmap v2227: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:30:16.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:30:16 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:30:16.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:30:16 smithi047 ceph-mon[77147]: pgmap v2227: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:30:16.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:30:16 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:30:16.541 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:30:16 smithi047 conmon[78521]: 2022-05-14T00:30:16.111+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:30:16.541 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:30:16 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T00:30:16.542 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:30:16 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:30:16.542 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:30:16 smithi047 conmon[78521]: yield 2022-05-14T00:30:16.542 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:30:16 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:30:16.542 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:30:16 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:30:16.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:30:16 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:30:16.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:30:16 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T00:30:16.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:30:16 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:30:16.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:30:16 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T00:30:16.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:30:16 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:30:16.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:30:16 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T00:30:16.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:30:16 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:30:16.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:30:16 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T00:30:16.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:30:16 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:30:16.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:30:16 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T00:30:16.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:30:16 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:30:16.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:30:16 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:30:16.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:30:16 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:30:16.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:30:16 smithi115 conmon[90393]: 2022-05-14T00:30:16.564+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:30:17.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:30:17 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:30:17.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:30:17 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:30:17.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:30:17 smithi115 conmon[90393]: 2022-05-14T00:30:17.597+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:30:18.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:30:18 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:30:18.358 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T00:30:18.358 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:30:18.358 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T00:30:18.358 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:30:18.359 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:30:18.359 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:30:18.359 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T00:30:18.359 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:30:18.359 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T00:30:18.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:30:18.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T00:30:18.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:30:18.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T00:30:18.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:30:18.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T00:30:18.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:30:18.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:30:18.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:30:18.363 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:30:18 smithi115 ceph-mon[78593]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T00:30:18.363 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:30:18 smithi115 ceph-mon[78593]: pgmap v2228: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:30:18.363 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:30:18 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:30:18.364 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:30:18 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:30:18.364 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:30:18 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:30:18.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:30:18 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:30:18.551 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T00:30:18.552 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:30:18.553 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T00:30:18.553 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:30:18.553 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:30:18.554 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:30:18.554 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T00:30:18.555 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:30:18.555 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T00:30:18.555 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:30:18.556 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T00:30:18.556 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:30:18.557 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T00:30:18.557 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:30:18.557 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T00:30:18.558 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:30:18.558 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:30:18.558 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:30:18.559 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:30:18 smithi047 ceph-mon[77147]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T00:30:18.559 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:30:18 smithi047 ceph-mon[77147]: pgmap v2228: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:30:18.559 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:30:18 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:30:18.560 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:30:18 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:30:18.560 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:30:18 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:30:18.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:30:18 smithi115 conmon[90393]: 2022-05-14T00:30:18.594+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:30:19.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:30:19 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:30:19.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:30:19 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 3750 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:30:19.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:30:19 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:30:19.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:30:19 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 3750 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:30:19.542 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:30:19 smithi047 conmon[78521]: 2022-05-14T00:30:19.183+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:30:19.542 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:30:19 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T00:30:19.542 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:30:19 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:30:19.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:30:19 smithi047 conmon[78521]: yield 2022-05-14T00:30:19.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:30:19 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:30:19.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:30:19 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:30:19.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:30:19 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:30:19.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:30:19 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T00:30:19.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:30:19 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:30:19.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:30:19 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T00:30:19.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:30:19 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:30:19.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:30:19 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T00:30:19.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:30:19 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:30:19.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:30:19 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T00:30:19.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:30:19 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:30:19.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:30:19 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T00:30:19.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:30:19 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:30:19.547 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:30:19 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:30:19.547 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:30:19 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:30:19.547 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:30:19 smithi047 conmon[78521]: 2022-05-14T00:30:19.184+0000 7f533ef76700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T00:30:19.547 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:30:19 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T00:30:19.548 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:30:19 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:30:19.548 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:30:19 smithi047 conmon[78521]: yield 2022-05-14T00:30:19.548 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:30:19 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:30:19.548 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:30:19 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:30:19.549 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:30:19 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:30:19.549 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:30:19 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T00:30:19.549 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:30:19 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:30:19.549 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:30:19 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T00:30:19.550 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:30:19 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:30:19.550 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:30:19 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T00:30:19.550 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:30:19 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:30:19.550 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:30:19 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T00:30:19.550 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:30:19 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:30:19.551 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:30:19 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T00:30:19.551 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:30:19 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:30:19.551 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:30:19 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:30:19.551 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:30:19 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:30:19.552 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:30:19 smithi047 conmon[78521]: 2022-05-14T00:30:19.552 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:30:19 smithi047 conmon[78521]: During handling of the above exception, another exception occurred: 2022-05-14T00:30:19.552 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:30:19 smithi047 conmon[78521]: 2022-05-14T00:30:19.552 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:30:19 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T00:30:19.552 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:30:19 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T00:30:19.553 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:30:19 smithi047 conmon[78521]: return f(*arg) 2022-05-14T00:30:19.553 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:30:19 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T00:30:19.553 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:30:19 smithi047 conmon[78521]: self._write_client_files(client_files, host) 2022-05-14T00:30:19.553 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:30:19 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T00:30:19.554 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:30:19 smithi047 conmon[78521]: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T00:30:19.554 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:30:19 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T00:30:19.554 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:30:19 smithi047 conmon[78521]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T00:30:19.554 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:30:19 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T00:30:19.555 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:30:19 smithi047 conmon[78521]: return self.event_loop.get_result(coro) 2022-05-14T00:30:19.555 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:30:19 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T00:30:19.555 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:30:19 smithi047 conmon[78521]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T00:30:19.555 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:30:19 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T00:30:19.555 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:30:19 smithi047 conmon[78521]: return self.__get_result() 2022-05-14T00:30:19.556 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:30:19 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T00:30:19.556 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:30:19 smithi047 conmon[78521]: raise self._exception 2022-05-14T00:30:19.556 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:30:19 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T00:30:19.556 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:30:19 smithi047 conmon[78521]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T00:30:19.556 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:30:19 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T00:30:19.557 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:30:19 smithi047 conmon[78521]: conn = await self._remote_connection(host, addr) 2022-05-14T00:30:19.557 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:30:19 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T00:30:19.557 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:30:19 smithi047 conmon[78521]: raise 2022-05-14T00:30:19.557 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:30:19 smithi047 conmon[78521]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T00:30:19.558 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:30:19 smithi047 conmon[78521]: self.gen.throw(type, value, traceback) 2022-05-14T00:30:19.558 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:30:19 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T00:30:19.558 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:30:19 smithi047 conmon[78521]: raise OrchestratorError(msg) 2022-05-14T00:30:19.558 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:30:19 smithi047 conmon[78521]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:30:19.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:30:19 smithi115 conmon[90393]: 2022-05-14T00:30:19.581+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:30:20.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:30:20 smithi115 ceph-mon[78593]: pgmap v2229: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:30:20.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:30:20 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:30:20.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:30:20 smithi047 ceph-mon[77147]: pgmap v2229: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:30:20.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:30:20 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:30:20.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:30:20 smithi115 conmon[90393]: 2022-05-14T00:30:20.547+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:30:21.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:30:21 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:30:21.358 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T00:30:21.358 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:30:21.358 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T00:30:21.358 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:30:21.359 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:30:21.359 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:30:21.359 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T00:30:21.359 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:30:21.359 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T00:30:21.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:30:21.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T00:30:21.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:30:21.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T00:30:21.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:30:21.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T00:30:21.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:30:21.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:30:21.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:30:21.362 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:30:21 smithi115 ceph-mon[78593]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T00:30:21.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T00:30:21.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:30:21.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T00:30:21.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:30:21.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:30:21.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:30:21.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T00:30:21.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:30:21.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T00:30:21.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:30:21.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T00:30:21.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:30:21.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T00:30:21.365 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:30:21.365 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T00:30:21.365 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:30:21.365 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:30:21.365 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:30:21.366 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T00:30:21.366 INFO:journalctl@ceph.mon.c.smithi115.stdout: During handling of the above exception, another exception occurred: 2022-05-14T00:30:21.366 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T00:30:21.366 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T00:30:21.367 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T00:30:21.367 INFO:journalctl@ceph.mon.c.smithi115.stdout: return f(*arg) 2022-05-14T00:30:21.367 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T00:30:21.367 INFO:journalctl@ceph.mon.c.smithi115.stdout: self._write_client_files(client_files, host) 2022-05-14T00:30:21.367 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T00:30:21.368 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T00:30:21.368 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T00:30:21.368 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T00:30:21.368 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T00:30:21.368 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.event_loop.get_result(coro) 2022-05-14T00:30:21.369 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T00:30:21.369 INFO:journalctl@ceph.mon.c.smithi115.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T00:30:21.369 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T00:30:21.369 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.__get_result() 2022-05-14T00:30:21.370 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T00:30:21.370 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise self._exception 2022-05-14T00:30:21.370 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T00:30:21.370 INFO:journalctl@ceph.mon.c.smithi115.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T00:30:21.370 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T00:30:21.371 INFO:journalctl@ceph.mon.c.smithi115.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T00:30:21.371 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T00:30:21.371 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise 2022-05-14T00:30:21.371 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T00:30:21.371 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.gen.throw(type, value, traceback) 2022-05-14T00:30:21.372 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T00:30:21.372 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OrchestratorError(msg) 2022-05-14T00:30:21.372 INFO:journalctl@ceph.mon.c.smithi115.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:30:21.372 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:30:21 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:30:21.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:30:21 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:30:21.541 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T00:30:21.541 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:30:21.541 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T00:30:21.542 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:30:21.542 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:30:21.542 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:30:21.542 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T00:30:21.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:30:21.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T00:30:21.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:30:21.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T00:30:21.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:30:21.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T00:30:21.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:30:21.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T00:30:21.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:30:21.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:30:21.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:30:21.545 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:30:21 smithi047 ceph-mon[77147]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T00:30:21.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T00:30:21.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:30:21.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T00:30:21.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:30:21.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:30:21.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:30:21.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T00:30:21.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:30:21.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T00:30:21.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:30:21.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T00:30:21.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:30:21.548 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T00:30:21.548 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:30:21.548 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T00:30:21.548 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:30:21.549 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:30:21.549 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:30:21.549 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T00:30:21.550 INFO:journalctl@ceph.mon.b.smithi047.stdout: During handling of the above exception, another exception occurred: 2022-05-14T00:30:21.550 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T00:30:21.550 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T00:30:21.550 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T00:30:21.551 INFO:journalctl@ceph.mon.b.smithi047.stdout: return f(*arg) 2022-05-14T00:30:21.551 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T00:30:21.551 INFO:journalctl@ceph.mon.b.smithi047.stdout: self._write_client_files(client_files, host) 2022-05-14T00:30:21.551 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T00:30:21.551 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T00:30:21.552 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T00:30:21.552 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T00:30:21.552 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T00:30:21.552 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.event_loop.get_result(coro) 2022-05-14T00:30:21.552 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T00:30:21.553 INFO:journalctl@ceph.mon.b.smithi047.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T00:30:21.553 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T00:30:21.553 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.__get_result() 2022-05-14T00:30:21.553 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T00:30:21.554 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise self._exception 2022-05-14T00:30:21.554 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T00:30:21.554 INFO:journalctl@ceph.mon.b.smithi047.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T00:30:21.554 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T00:30:21.554 INFO:journalctl@ceph.mon.b.smithi047.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T00:30:21.555 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T00:30:21.555 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise 2022-05-14T00:30:21.555 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T00:30:21.555 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.gen.throw(type, value, traceback) 2022-05-14T00:30:21.556 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T00:30:21.556 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OrchestratorError(msg) 2022-05-14T00:30:21.556 INFO:journalctl@ceph.mon.b.smithi047.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:30:21.556 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:30:21 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:30:21.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:30:21 smithi115 conmon[90393]: 2022-05-14T00:30:21.557+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:30:22.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:30:22 smithi115 ceph-mon[78593]: pgmap v2230: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:30:22.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:30:22 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:30:22.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:30:22 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:30:22.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:30:22 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:30:22.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:30:22 smithi047 ceph-mon[77147]: pgmap v2230: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:30:22.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:30:22 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:30:22.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:30:22 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:30:22.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:30:22 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:30:22.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:30:22 smithi115 conmon[90393]: 2022-05-14T00:30:22.518+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:30:23.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:30:23 smithi115 ceph-mon[78593]: pgmap v2231: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:30:23.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:30:23 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:30:23.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:30:23 smithi047 ceph-mon[77147]: pgmap v2231: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:30:23.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:30:23 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:30:23.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:30:23 smithi115 conmon[90393]: 2022-05-14T00:30:23.490+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:30:24.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:30:24 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:30:24.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:30:24 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 3755 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:30:24.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:30:24 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:30:24.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:30:24 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 3755 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:30:24.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:30:24 smithi115 conmon[90393]: 2022-05-14T00:30:24.505+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:30:25.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:30:25 smithi115 ceph-mon[78593]: pgmap v2232: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:30:25.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:30:25 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:30:25.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:30:25 smithi047 ceph-mon[77147]: pgmap v2232: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:30:25.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:30:25 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:30:25.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:30:25 smithi115 conmon[90393]: 2022-05-14T00:30:25.455+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:30:26.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:30:26 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:30:26.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:30:26 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:30:26.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:30:26 smithi115 conmon[90393]: 2022-05-14T00:30:26.409+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:30:27.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:30:27 smithi115 ceph-mon[78593]: pgmap v2233: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:30:27.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:30:27 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:30:27.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:30:27 smithi047 ceph-mon[77147]: pgmap v2233: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:30:27.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:30:27 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:30:27.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:30:27 smithi115 conmon[90393]: 2022-05-14T00:30:27.362+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:30:28.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:30:28 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:30:28.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:30:28 smithi115 conmon[90393]: 2022-05-14T00:30:28.319+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:30:28.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:30:28 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:30:29.346 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:30:29 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:30:29.347 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:30:29 smithi115 ceph-mon[78593]: pgmap v2234: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:30:29.347 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:30:29 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 3760 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:30:29.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:30:29 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:30:29.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:30:29 smithi047 ceph-mon[77147]: pgmap v2234: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:30:29.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:30:29 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 3760 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:30:29.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:30:29 smithi115 conmon[90393]: 2022-05-14T00:30:29.345+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:30:30.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:30:30 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:30:30.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:30:30 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:30:30.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:30:30 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:30:30.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:30:30 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:30:30.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:30:30 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:30:30.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:30:30 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:30:30.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:30:30 smithi115 conmon[90393]: 2022-05-14T00:30:30.379+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:30:31.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:30:31 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:30:31.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:30:31 smithi115 ceph-mon[78593]: pgmap v2235: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:30:31.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:30:31 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:30:31.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:30:31 smithi047 ceph-mon[77147]: pgmap v2235: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:30:31.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:30:31 smithi115 conmon[90393]: 2022-05-14T00:30:31.365+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:30:32.361 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:30:32 smithi115 conmon[90393]: 2022-05-14T00:30:32.324+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:30:32.362 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:30:32 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:30:32.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:30:32 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:30:33.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:30:33 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:30:33.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:30:33 smithi047 ceph-mon[77147]: pgmap v2236: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:30:33.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:30:33 smithi115 conmon[90393]: 2022-05-14T00:30:33.353+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:30:33.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:30:33 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:30:33.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:30:33 smithi115 ceph-mon[78593]: pgmap v2236: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:30:34.365 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:30:34 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:30:34.380 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:30:34 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:30:34.381 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:30:34 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:30:34.382 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:30:34 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 3765 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:30:34.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:30:34 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:30:34.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:30:34 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:30:34.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:30:34 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:30:34.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:30:34 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 3765 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:30:34.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:30:34 smithi115 conmon[90393]: 2022-05-14T00:30:34.365+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:30:35.379 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:30:35 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:30:35.379 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:30:35 smithi115 ceph-mon[78593]: pgmap v2237: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:30:35.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:30:35 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:30:35.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:30:35 smithi047 ceph-mon[77147]: pgmap v2237: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:30:35.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:30:35 smithi115 conmon[90393]: 2022-05-14T00:30:35.378+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:30:36.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:30:36 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:30:36.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:30:36 smithi115 conmon[90393]: 2022-05-14T00:30:36.339+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:30:36.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:30:36 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:30:37.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:30:37 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:30:37.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:30:37 smithi047 ceph-mon[77147]: pgmap v2238: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:30:37.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:30:37 smithi115 conmon[90393]: 2022-05-14T00:30:37.295+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:30:37.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:30:37 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:30:37.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:30:37 smithi115 ceph-mon[78593]: pgmap v2238: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:30:38.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:30:38 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:30:38.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:30:38 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:30:38.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:30:38 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:30:38.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:30:38 smithi115 conmon[90393]: 2022-05-14T00:30:38.261+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:30:38.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:30:38 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:30:38.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:30:38 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:30:38.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:30:38 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:30:39.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:30:39 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:30:39.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:30:39 smithi047 ceph-mon[77147]: pgmap v2239: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:30:39.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:30:39 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 3770 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:30:39.612 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:30:39 smithi115 conmon[90393]: 2022-05-14T00:30:39.307+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:30:39.614 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:30:39 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:30:39.614 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:30:39 smithi115 ceph-mon[78593]: pgmap v2239: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:30:39.615 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:30:39 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 3770 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:30:40.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:30:40 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:30:40.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:30:40 smithi115 conmon[90393]: 2022-05-14T00:30:40.304+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:30:40.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:30:40 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:30:41.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:30:41 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:30:41.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:30:41 smithi047 ceph-mon[77147]: pgmap v2240: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:30:41.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:30:41 smithi115 conmon[90393]: 2022-05-14T00:30:41.304+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:30:41.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:30:41 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:30:41.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:30:41 smithi115 ceph-mon[78593]: pgmap v2240: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:30:42.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:30:42 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:30:42.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:30:42 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:30:42.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:30:42 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:30:42.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:30:42 smithi115 conmon[90393]: 2022-05-14T00:30:42.341+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:30:42.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:30:42 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:30:42.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:30:42 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:30:42.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:30:42 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:30:43.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:30:43 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:30:43.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:30:43 smithi047 ceph-mon[77147]: pgmap v2241: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:30:43.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:30:43 smithi115 conmon[90393]: 2022-05-14T00:30:43.300+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:30:43.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:30:43 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:30:43.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:30:43 smithi115 ceph-mon[78593]: pgmap v2241: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:30:44.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:30:44 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:30:44.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:30:44 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 3775 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:30:44.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:30:44 smithi115 conmon[90393]: 2022-05-14T00:30:44.334+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:30:44.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:30:44 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:30:44.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:30:44 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 3775 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:30:45.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:30:45 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:30:45.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:30:45 smithi047 ceph-mon[77147]: pgmap v2242: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:30:45.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:30:45 smithi115 conmon[90393]: 2022-05-14T00:30:45.298+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:30:45.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:30:45 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:30:45.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:30:45 smithi115 ceph-mon[78593]: pgmap v2242: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:30:46.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:30:46 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:30:46.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:30:46 smithi115 conmon[90393]: 2022-05-14T00:30:46.267+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:30:46.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:30:46 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:30:47.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:30:47 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:30:47.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:30:47 smithi047 ceph-mon[77147]: pgmap v2243: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:30:47.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:30:47 smithi115 conmon[90393]: 2022-05-14T00:30:47.317+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:30:47.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:30:47 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:30:47.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:30:47 smithi115 ceph-mon[78593]: pgmap v2243: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:30:48.549 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:30:48 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:30:48.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:30:48 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:30:48.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:30:48 smithi115 conmon[90393]: 2022-05-14T00:30:48.319+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:30:49.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:30:49 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:30:49.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:30:49 smithi047 ceph-mon[77147]: pgmap v2244: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:30:49.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:30:49 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 3780 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:30:49.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:30:49 smithi115 conmon[90393]: 2022-05-14T00:30:49.279+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:30:49.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:30:49 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:30:49.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:30:49 smithi115 ceph-mon[78593]: pgmap v2244: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:30:49.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:30:49 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 3780 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:30:50.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:30:50 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:30:50.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:30:50 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:30:50.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:30:50 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:30:50.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:30:50 smithi115 conmon[90393]: 2022-05-14T00:30:50.310+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:30:50.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:30:50 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:30:50.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:30:50 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:30:50.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:30:50 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:30:51.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:30:51 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:30:51.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:30:51 smithi047 ceph-mon[77147]: pgmap v2245: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:30:51.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:30:51 smithi115 conmon[90393]: 2022-05-14T00:30:51.295+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:30:51.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:30:51 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:30:51.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:30:51 smithi115 ceph-mon[78593]: pgmap v2245: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:30:52.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:30:52 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:30:52.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:30:52 smithi115 conmon[90393]: 2022-05-14T00:30:52.339+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:30:52.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:30:52 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:30:53.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:30:53 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:30:53.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:30:53 smithi047 ceph-mon[77147]: pgmap v2246: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:30:53.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:30:53 smithi115 conmon[90393]: 2022-05-14T00:30:53.362+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:30:53.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:30:53 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:30:53.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:30:53 smithi115 ceph-mon[78593]: pgmap v2246: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:30:54.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:30:54 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:30:54.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:30:54 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:30:54.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:30:54 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:30:54.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:30:54 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 3785 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:30:54.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:30:54 smithi115 conmon[90393]: 2022-05-14T00:30:54.349+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:30:54.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:30:54 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:30:54.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:30:54 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:30:54.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:30:54 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:30:54.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:30:54 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 3785 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:30:55.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:30:55 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:30:55.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:30:55 smithi047 ceph-mon[77147]: pgmap v2247: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:30:55.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:30:55 smithi115 conmon[90393]: 2022-05-14T00:30:55.365+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:30:55.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:30:55 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:30:55.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:30:55 smithi115 ceph-mon[78593]: pgmap v2247: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:30:56.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:30:56 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:30:56.615 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:30:56 smithi115 conmon[90393]: 2022-05-14T00:30:56.344+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:30:56.616 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:30:56 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:30:57.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:30:57 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:30:57.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:30:57 smithi047 ceph-mon[77147]: pgmap v2248: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:30:57.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:30:57 smithi115 conmon[90393]: 2022-05-14T00:30:57.376+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:30:57.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:30:57 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:30:57.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:30:57 smithi115 ceph-mon[78593]: pgmap v2248: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:30:58.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:30:58 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:30:58.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:30:58 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:30:58.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:30:58 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:30:58.610 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:30:58 smithi115 conmon[90393]: 2022-05-14T00:30:58.381+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:30:58.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:30:58 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:30:58.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:30:58 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:30:58.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:30:58 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:30:59.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:30:59 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:30:59.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:30:59 smithi047 ceph-mon[77147]: pgmap v2249: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:30:59.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:30:59 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 3790 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:30:59.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:30:59 smithi115 conmon[90393]: 2022-05-14T00:30:59.339+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:30:59.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:30:59 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:30:59.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:30:59 smithi115 ceph-mon[78593]: pgmap v2249: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:30:59.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:30:59 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 3790 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:31:00.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:31:00 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:31:00.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:31:00 smithi115 conmon[90393]: 2022-05-14T00:31:00.318+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:31:00.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:31:00 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:31:01.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:31:01 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:31:01.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:31:01 smithi047 ceph-mon[77147]: pgmap v2250: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:31:01.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:31:01 smithi115 conmon[90393]: 2022-05-14T00:31:01.316+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:31:01.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:31:01 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:31:01.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:31:01 smithi115 ceph-mon[78593]: pgmap v2250: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:31:02.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:31:02 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:31:02.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:31:02 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:31:02.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:31:02 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:31:02.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:31:02 smithi115 conmon[90393]: 2022-05-14T00:31:02.323+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:31:02.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:31:02 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:31:02.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:31:02 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:31:02.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:31:02 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:31:03.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:31:03 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:31:03.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:31:03 smithi047 ceph-mon[77147]: pgmap v2251: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:31:03.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:31:03 smithi115 conmon[90393]: 2022-05-14T00:31:03.360+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:31:03.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:31:03 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:31:03.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:31:03 smithi115 ceph-mon[78593]: pgmap v2251: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:31:04.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:31:04 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:31:04.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:31:04 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 3795 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:31:04.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:31:04 smithi115 conmon[90393]: 2022-05-14T00:31:04.316+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:31:04.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:31:04 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:31:04.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:31:04 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 3795 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:31:05.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:31:05 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:31:05.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:31:05 smithi047 ceph-mon[77147]: pgmap v2252: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:31:05.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:31:05 smithi115 conmon[90393]: 2022-05-14T00:31:05.284+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:31:05.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:31:05 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:31:05.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:31:05 smithi115 ceph-mon[78593]: pgmap v2252: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:31:06.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:31:06 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:31:06.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:31:06 smithi115 conmon[90393]: 2022-05-14T00:31:06.235+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:31:06.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:31:06 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:31:07.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:31:07 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:31:07.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:31:07 smithi047 ceph-mon[77147]: pgmap v2253: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:31:07.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:31:07 smithi115 conmon[90393]: 2022-05-14T00:31:07.264+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:31:07.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:31:07 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:31:07.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:31:07 smithi115 ceph-mon[78593]: pgmap v2253: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:31:08.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:31:08 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:31:08.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:31:08 smithi115 conmon[90393]: 2022-05-14T00:31:08.220+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:31:08.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:31:08 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:31:09.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:31:09 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:31:09.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:31:09 smithi047 ceph-mon[77147]: pgmap v2254: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:31:09.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:31:09 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 3800 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:31:09.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:31:09 smithi115 conmon[90393]: 2022-05-14T00:31:09.237+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:31:09.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:31:09 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:31:09.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:31:09 smithi115 ceph-mon[78593]: pgmap v2254: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:31:09.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:31:09 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 3800 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:31:10.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:31:10 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:31:10.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:31:10 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:31:10.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:31:10 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:31:10.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:31:10 smithi115 conmon[90393]: 2022-05-14T00:31:10.277+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:31:10.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:31:10 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:31:10.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:31:10 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:31:10.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:31:10 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:31:11.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:31:11 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:31:11.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:31:11 smithi047 ceph-mon[77147]: pgmap v2255: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:31:11.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:31:11 smithi115 conmon[90393]: 2022-05-14T00:31:11.295+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:31:11.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:31:11 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:31:11.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:31:11 smithi115 ceph-mon[78593]: pgmap v2255: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:31:12.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:31:12 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:31:12.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:31:12 smithi115 conmon[90393]: 2022-05-14T00:31:12.267+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:31:12.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:31:12 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:31:13.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:31:13 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:31:13.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:31:13 smithi047 ceph-mon[77147]: pgmap v2256: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:31:13.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:31:13 smithi115 conmon[90393]: 2022-05-14T00:31:13.299+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:31:13.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:31:13 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:31:13.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:31:13 smithi115 ceph-mon[78593]: pgmap v2256: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:31:14.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:31:14 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:31:14.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:31:14 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:31:14.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:31:14 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:31:14.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:31:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T00:31:14.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:31:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T00:31:14.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:31:14 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 3805 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:31:14.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:31:14 smithi115 conmon[90393]: 2022-05-14T00:31:14.282+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:31:14.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:31:14 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:31:14.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:31:14 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:31:14.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:31:14 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:31:14.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:31:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T00:31:14.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:31:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T00:31:14.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:31:14 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 3805 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:31:15.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:31:15 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:31:15.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:31:15 smithi047 ceph-mon[77147]: pgmap v2257: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:31:15.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:31:15 smithi115 conmon[90393]: 2022-05-14T00:31:15.253+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:31:15.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:31:15 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:31:15.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:31:15 smithi115 ceph-mon[78593]: pgmap v2257: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:31:16.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:31:16 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:31:16.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:31:16 smithi115 conmon[90393]: 2022-05-14T00:31:16.293+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:31:16.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:31:16 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:31:17.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:31:17 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:31:17.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:31:17 smithi047 ceph-mon[77147]: pgmap v2258: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:31:17.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:31:17 smithi115 conmon[90393]: 2022-05-14T00:31:17.335+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:31:17.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:31:17 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:31:17.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:31:17 smithi115 ceph-mon[78593]: pgmap v2258: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:31:18.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:31:18 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:31:18.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:31:18 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:31:18.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:31:18 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:31:18.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:31:18 smithi115 conmon[90393]: 2022-05-14T00:31:18.335+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:31:18.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:31:18 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:31:18.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:31:18 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:31:18.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:31:18 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:31:19.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:31:19 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:31:19.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:31:19 smithi047 ceph-mon[77147]: pgmap v2259: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:31:19.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:31:19 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 3810 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:31:19.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:31:19 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T00:31:19.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:31:19 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T00:31:19.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:31:19 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T00:31:19.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:31:19 smithi115 conmon[90393]: 2022-05-14T00:31:19.327+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:31:19.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:31:19 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:31:19.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:31:19 smithi115 ceph-mon[78593]: pgmap v2259: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:31:19.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:31:19 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 3810 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:31:19.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:31:19 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T00:31:19.620 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:31:19 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T00:31:19.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:31:19 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T00:31:20.041 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:31:19 smithi047 conmon[78521]: 2022-05-14T00:31:19.664+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:31:20.042 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:31:19 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T00:31:20.042 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:31:19 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:31:20.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:31:19 smithi047 conmon[78521]: yield 2022-05-14T00:31:20.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:31:19 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:31:20.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:31:19 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:31:20.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:31:19 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:31:20.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:31:19 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T00:31:20.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:31:19 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:31:20.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:31:19 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T00:31:20.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:31:19 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:31:20.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:31:19 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T00:31:20.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:31:19 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:31:20.048 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:31:19 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T00:31:20.048 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:31:19 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:31:20.049 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:31:19 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T00:31:20.049 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:31:19 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:31:20.050 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:31:19 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:31:20.050 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:31:19 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:31:20.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:31:20 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:31:20.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:31:20 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:31:20.541 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T00:31:20.541 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:31:20.541 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T00:31:20.542 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:31:20.542 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:31:20.542 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:31:20.542 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T00:31:20.542 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:31:20.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T00:31:20.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:31:20.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T00:31:20.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:31:20.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T00:31:20.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:31:20.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T00:31:20.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:31:20.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:31:20.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:31:20.545 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:31:20 smithi047 ceph-mon[77147]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T00:31:20.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:31:20 smithi115 conmon[90393]: 2022-05-14T00:31:20.374+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:31:20.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:31:20 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:31:20.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:31:20 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:31:20.608 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T00:31:20.608 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:31:20.609 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T00:31:20.609 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:31:20.609 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:31:20.609 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:31:20.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T00:31:20.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:31:20.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T00:31:20.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:31:20.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T00:31:20.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:31:20.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T00:31:20.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:31:20.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T00:31:20.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:31:20.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:31:20.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:31:20.612 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:31:20 smithi115 ceph-mon[78593]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T00:31:21.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:31:21 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:31:21.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:31:21 smithi047 ceph-mon[77147]: pgmap v2260: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:31:21.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:31:21 smithi115 conmon[90393]: 2022-05-14T00:31:21.348+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:31:21.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:31:21 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:31:21.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:31:21 smithi115 ceph-mon[78593]: pgmap v2260: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:31:22.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:31:22 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:31:22.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:31:22 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:31:22.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:31:22 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:31:22.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:31:22 smithi115 conmon[90393]: 2022-05-14T00:31:22.352+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:31:22.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:31:22 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:31:22.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:31:22 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:31:22.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:31:22 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:31:23.041 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:31:22 smithi047 conmon[78521]: 2022-05-14T00:31:22.736+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:31:23.042 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:31:22 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T00:31:23.042 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:31:22 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:31:23.042 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:31:22 smithi047 conmon[78521]: yield 2022-05-14T00:31:23.042 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:31:22 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:31:23.042 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:31:22 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:31:23.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:31:22 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:31:23.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:31:22 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T00:31:23.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:31:22 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:31:23.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:31:22 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T00:31:23.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:31:22 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:31:23.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:31:22 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T00:31:23.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:31:22 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:31:23.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:31:22 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T00:31:23.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:31:22 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:31:23.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:31:22 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T00:31:23.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:31:22 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:31:23.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:31:22 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:31:23.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:31:22 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:31:23.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:31:22 smithi047 conmon[78521]: 2022-05-14T00:31:22.737+0000 7f533f777700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T00:31:23.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:31:22 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T00:31:23.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:31:22 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:31:23.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:31:22 smithi047 conmon[78521]: yield 2022-05-14T00:31:23.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:31:22 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:31:23.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:31:22 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:31:23.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:31:22 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:31:23.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:31:22 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T00:31:23.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:31:22 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:31:23.048 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:31:22 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T00:31:23.048 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:31:22 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:31:23.048 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:31:22 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T00:31:23.048 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:31:22 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:31:23.049 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:31:22 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T00:31:23.049 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:31:22 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:31:23.049 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:31:22 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T00:31:23.049 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:31:22 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:31:23.050 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:31:22 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:31:23.050 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:31:22 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:31:23.050 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:31:22 smithi047 conmon[78521]: 2022-05-14T00:31:23.050 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:31:22 smithi047 conmon[78521]: During handling of the above exception, another exception occurred: 2022-05-14T00:31:23.050 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:31:22 smithi047 conmon[78521]: 2022-05-14T00:31:23.051 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:31:22 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T00:31:23.051 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:31:22 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T00:31:23.051 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:31:22 smithi047 conmon[78521]: return f(*arg) 2022-05-14T00:31:23.051 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:31:22 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T00:31:23.052 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:31:22 smithi047 conmon[78521]: self._write_client_files(client_files, host) 2022-05-14T00:31:23.052 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:31:22 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T00:31:23.052 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:31:22 smithi047 conmon[78521]: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T00:31:23.052 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:31:22 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T00:31:23.052 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:31:22 smithi047 conmon[78521]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T00:31:23.053 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:31:22 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T00:31:23.053 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:31:22 smithi047 conmon[78521]: return self.event_loop.get_result(coro) 2022-05-14T00:31:23.053 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:31:22 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T00:31:23.053 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:31:22 smithi047 conmon[78521]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T00:31:23.054 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:31:22 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T00:31:23.054 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:31:22 smithi047 conmon[78521]: return self.__get_result() 2022-05-14T00:31:23.054 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:31:22 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T00:31:23.054 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:31:22 smithi047 conmon[78521]: raise self._exception 2022-05-14T00:31:23.054 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:31:22 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T00:31:23.055 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:31:22 smithi047 conmon[78521]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T00:31:23.055 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:31:22 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T00:31:23.055 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:31:22 smithi047 conmon[78521]: conn = await self._remote_connection(host, addr) 2022-05-14T00:31:23.055 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:31:22 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T00:31:23.056 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:31:22 smithi047 conmon[78521]: raise 2022-05-14T00:31:23.056 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:31:22 smithi047 conmon[78521]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T00:31:23.056 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:31:22 smithi047 conmon[78521]: self.gen.throw(type, value, traceback) 2022-05-14T00:31:23.056 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:31:22 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T00:31:23.056 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:31:22 smithi047 conmon[78521]: raise OrchestratorError(msg) 2022-05-14T00:31:23.057 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:31:22 smithi047 conmon[78521]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:31:23.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:31:23 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:31:23.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:31:23 smithi047 ceph-mon[77147]: pgmap v2261: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:31:23.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:31:23 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:31:23.542 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T00:31:23.542 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:31:23.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T00:31:23.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:31:23.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:31:23.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:31:23.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T00:31:23.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:31:23.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T00:31:23.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:31:23.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T00:31:23.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:31:23.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T00:31:23.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:31:23.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T00:31:23.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:31:23.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:31:23.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:31:23.547 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:31:23 smithi047 ceph-mon[77147]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T00:31:23.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T00:31:23.548 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:31:23.548 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T00:31:23.548 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:31:23.548 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:31:23.549 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:31:23.549 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T00:31:23.549 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:31:23.550 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T00:31:23.550 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:31:23.550 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T00:31:23.550 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:31:23.551 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T00:31:23.551 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:31:23.551 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T00:31:23.552 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:31:23.552 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:31:23.552 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:31:23.553 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T00:31:23.553 INFO:journalctl@ceph.mon.b.smithi047.stdout: During handling of the above exception, another exception occurred: 2022-05-14T00:31:23.553 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T00:31:23.554 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T00:31:23.554 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T00:31:23.554 INFO:journalctl@ceph.mon.b.smithi047.stdout: return f(*arg) 2022-05-14T00:31:23.554 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T00:31:23.555 INFO:journalctl@ceph.mon.b.smithi047.stdout: self._write_client_files(client_files, host) 2022-05-14T00:31:23.555 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T00:31:23.555 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T00:31:23.556 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T00:31:23.556 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T00:31:23.557 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T00:31:23.557 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.event_loop.get_result(coro) 2022-05-14T00:31:23.558 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T00:31:23.558 INFO:journalctl@ceph.mon.b.smithi047.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T00:31:23.559 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T00:31:23.559 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.__get_result() 2022-05-14T00:31:23.560 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T00:31:23.560 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise self._exception 2022-05-14T00:31:23.561 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T00:31:23.561 INFO:journalctl@ceph.mon.b.smithi047.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T00:31:23.561 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T00:31:23.562 INFO:journalctl@ceph.mon.b.smithi047.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T00:31:23.562 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T00:31:23.563 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise 2022-05-14T00:31:23.563 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T00:31:23.563 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.gen.throw(type, value, traceback) 2022-05-14T00:31:23.564 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T00:31:23.564 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OrchestratorError(msg) 2022-05-14T00:31:23.565 INFO:journalctl@ceph.mon.b.smithi047.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:31:23.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:31:23 smithi115 conmon[90393]: 2022-05-14T00:31:23.322+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:31:23.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:31:23 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:31:23.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:31:23 smithi115 ceph-mon[78593]: pgmap v2261: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:31:23.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:31:23 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:31:23.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T00:31:23.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:31:23.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T00:31:23.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:31:23.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:31:23.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:31:23.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T00:31:23.613 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:31:23.613 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T00:31:23.615 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:31:23.615 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T00:31:23.615 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:31:23.615 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T00:31:23.616 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:31:23.616 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T00:31:23.616 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:31:23.616 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:31:23.617 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:31:23.617 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:31:23 smithi115 ceph-mon[78593]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T00:31:23.618 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T00:31:23.618 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:31:23.618 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T00:31:23.618 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:31:23.618 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:31:23.619 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:31:23.619 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T00:31:23.619 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:31:23.620 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T00:31:23.620 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:31:23.620 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T00:31:23.621 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:31:23.621 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T00:31:23.621 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:31:23.621 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T00:31:23.622 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:31:23.622 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:31:23.624 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:31:23.624 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T00:31:23.625 INFO:journalctl@ceph.mon.c.smithi115.stdout: During handling of the above exception, another exception occurred: 2022-05-14T00:31:23.626 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T00:31:23.626 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T00:31:23.627 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T00:31:23.627 INFO:journalctl@ceph.mon.c.smithi115.stdout: return f(*arg) 2022-05-14T00:31:23.628 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T00:31:23.628 INFO:journalctl@ceph.mon.c.smithi115.stdout: self._write_client_files(client_files, host) 2022-05-14T00:31:23.629 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T00:31:23.629 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T00:31:23.630 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T00:31:23.630 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T00:31:23.631 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T00:31:23.631 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.event_loop.get_result(coro) 2022-05-14T00:31:23.632 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T00:31:23.632 INFO:journalctl@ceph.mon.c.smithi115.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T00:31:23.633 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T00:31:23.633 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.__get_result() 2022-05-14T00:31:23.633 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T00:31:23.634 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise self._exception 2022-05-14T00:31:23.634 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T00:31:23.634 INFO:journalctl@ceph.mon.c.smithi115.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T00:31:23.634 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T00:31:23.635 INFO:journalctl@ceph.mon.c.smithi115.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T00:31:23.635 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T00:31:23.635 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise 2022-05-14T00:31:23.635 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T00:31:23.636 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.gen.throw(type, value, traceback) 2022-05-14T00:31:23.636 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T00:31:23.636 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OrchestratorError(msg) 2022-05-14T00:31:23.636 INFO:journalctl@ceph.mon.c.smithi115.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:31:24.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:31:24 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:31:24.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:31:24 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 3815 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:31:24.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:31:24 smithi115 conmon[90393]: 2022-05-14T00:31:24.318+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:31:24.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:31:24 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:31:24.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:31:24 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 3815 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:31:25.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:31:25 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:31:25.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:31:25 smithi047 ceph-mon[77147]: pgmap v2262: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:31:25.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:31:25 smithi115 conmon[90393]: 2022-05-14T00:31:25.340+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:31:25.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:31:25 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:31:25.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:31:25 smithi115 ceph-mon[78593]: pgmap v2262: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:31:26.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:31:26 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:31:26.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:31:26 smithi115 conmon[90393]: 2022-05-14T00:31:26.372+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:31:26.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:31:26 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:31:27.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:31:27 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:31:27.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:31:27 smithi047 ceph-mon[77147]: pgmap v2263: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:31:27.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:31:27 smithi115 conmon[90393]: 2022-05-14T00:31:27.385+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:31:27.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:31:27 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:31:27.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:31:27 smithi115 ceph-mon[78593]: pgmap v2263: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:31:28.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:31:28 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:31:28.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:31:28 smithi115 conmon[90393]: 2022-05-14T00:31:28.392+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:31:28.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:31:28 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:31:29.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:31:29 smithi115 conmon[90393]: 2022-05-14T00:31:29.373+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:31:29.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:31:29 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:31:29.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:31:29 smithi115 ceph-mon[78593]: pgmap v2264: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:31:29.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:31:29 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 3820 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:31:29.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:31:29 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:31:29.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:31:29 smithi047 ceph-mon[77147]: pgmap v2264: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:31:29.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:31:29 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 3820 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:31:30.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:31:30 smithi115 conmon[90393]: 2022-05-14T00:31:30.398+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:31:30.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:31:30 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:31:30.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:31:30 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:31:30.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:31:30 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:31:30.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:31:30 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:31:30.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:31:30 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:31:30.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:31:30 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:31:31.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:31:31 smithi115 conmon[90393]: 2022-05-14T00:31:31.434+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:31:31.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:31:31 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:31:31.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:31:31 smithi115 ceph-mon[78593]: pgmap v2265: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:31:31.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:31:31 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:31:31.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:31:31 smithi047 ceph-mon[77147]: pgmap v2265: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:31:32.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:31:32 smithi115 conmon[90393]: 2022-05-14T00:31:32.456+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.2.197:6 2.1 2.96a7dda1 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e150) 2022-05-14T00:31:32.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:31:32 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:31:32.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:31:32 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:31:33.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:31:33 smithi115 conmon[90393]: 2022-05-14T00:31:33.486+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:31:33.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:31:33 smithi115 ceph-mon[78593]: pgmap v2266: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:31:33.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:31:33 smithi115 ceph-mon[78593]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2022-05-14T00:31:33.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:31:33 smithi047 ceph-mon[77147]: pgmap v2266: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:31:33.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:31:33 smithi047 ceph-mon[77147]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2022-05-14T00:31:34.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:31:34 smithi115 conmon[90393]: 2022-05-14T00:31:34.461+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:31:34.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:31:34 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:31:34.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:31:34 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:31:34.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:31:34 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:31:34.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:31:34 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 3825 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:31:34.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:31:34 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:31:34.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:31:34 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:31:34.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:31:34 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:31:34.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:31:34 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 3825 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:31:35.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:31:35 smithi115 conmon[90393]: 2022-05-14T00:31:35.434+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:31:35.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:31:35 smithi115 ceph-mon[78593]: pgmap v2267: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:31:35.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:31:35 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:31:35.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:31:35 smithi047 ceph-mon[77147]: pgmap v2267: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:31:35.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:31:35 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:31:36.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:31:36 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:31:36.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:31:36 smithi115 conmon[90393]: 2022-05-14T00:31:36.388+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:31:36.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:31:36 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:31:37.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:31:37 smithi115 conmon[90393]: 2022-05-14T00:31:37.430+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:31:37.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:31:37 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:31:37.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:31:37 smithi115 ceph-mon[78593]: pgmap v2268: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:31:37.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:31:37 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:31:37.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:31:37 smithi047 ceph-mon[77147]: pgmap v2268: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:31:38.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:31:38 smithi115 conmon[90393]: 2022-05-14T00:31:38.423+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:31:38.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:31:38 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:31:38.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:31:38 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:31:38.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:31:38 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:31:38.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:31:38 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:31:38.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:31:38 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:31:38.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:31:38 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:31:39.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:31:39 smithi115 conmon[90393]: 2022-05-14T00:31:39.399+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:31:39.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:31:39 smithi115 ceph-mon[78593]: pgmap v2269: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:31:39.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:31:39 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:31:39.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:31:39 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 3830 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:31:39.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:31:39 smithi047 ceph-mon[77147]: pgmap v2269: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:31:39.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:31:39 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:31:39.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:31:39 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 3830 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:31:40.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:31:40 smithi115 conmon[90393]: 2022-05-14T00:31:40.367+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:31:40.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:31:40 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:31:40.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:31:40 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:31:41.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:31:41 smithi115 conmon[90393]: 2022-05-14T00:31:41.324+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:31:41.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:31:41 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:31:41.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:31:41 smithi115 ceph-mon[78593]: pgmap v2270: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:31:41.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:31:41 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:31:41.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:31:41 smithi047 ceph-mon[77147]: pgmap v2270: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:31:42.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:31:42 smithi115 conmon[90393]: 2022-05-14T00:31:42.335+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:31:42.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:31:42 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:31:42.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:31:42 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:31:42.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:31:42 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:31:42.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:31:42 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:31:42.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:31:42 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:31:42.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:31:42 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:31:43.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:31:43 smithi115 conmon[90393]: 2022-05-14T00:31:43.304+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:31:43.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:31:43 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:31:43.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:31:43 smithi115 ceph-mon[78593]: pgmap v2271: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:31:43.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:31:43 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:31:43.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:31:43 smithi047 ceph-mon[77147]: pgmap v2271: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:31:44.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:31:44 smithi115 conmon[90393]: 2022-05-14T00:31:44.265+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:31:44.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:31:44 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:31:44.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:31:44 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 3835 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:31:44.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:31:44 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:31:44.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:31:44 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 3835 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:31:45.610 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:31:45 smithi115 conmon[90393]: 2022-05-14T00:31:45.301+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:31:45.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:31:45 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:31:45.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:31:45 smithi115 ceph-mon[78593]: pgmap v2272: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:31:45.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:31:45 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:31:45.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:31:45 smithi047 ceph-mon[77147]: pgmap v2272: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:31:46.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:31:46 smithi115 conmon[90393]: 2022-05-14T00:31:46.267+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:31:46.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:31:46 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:31:46.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:31:46 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:31:47.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:31:47 smithi115 conmon[90393]: 2022-05-14T00:31:47.234+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:31:47.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:31:47 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:31:47.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:31:47 smithi115 ceph-mon[78593]: pgmap v2273: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:31:47.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:31:47 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:31:47.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:31:47 smithi047 ceph-mon[77147]: pgmap v2273: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:31:48.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:31:48 smithi115 conmon[90393]: 2022-05-14T00:31:48.251+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:31:48.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:31:48 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:31:48.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:31:48 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:31:49.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:31:49 smithi115 conmon[90393]: 2022-05-14T00:31:49.288+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:31:49.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:31:49 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:31:49.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:31:49 smithi115 ceph-mon[78593]: pgmap v2274: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:31:49.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:31:49 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 3840 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:31:49.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:31:49 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:31:49.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:31:49 smithi047 ceph-mon[77147]: pgmap v2274: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:31:49.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:31:49 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 3840 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:31:50.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:31:50 smithi115 conmon[90393]: 2022-05-14T00:31:50.256+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:31:50.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:31:50 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:31:50.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:31:50 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:31:50.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:31:50 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:31:50.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:31:50 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:31:50.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:31:50 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:31:50.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:31:50 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:31:51.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:31:51 smithi115 conmon[90393]: 2022-05-14T00:31:51.273+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:31:51.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:31:51 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:31:51.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:31:51 smithi115 ceph-mon[78593]: pgmap v2275: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:31:51.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:31:51 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:31:51.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:31:51 smithi047 ceph-mon[77147]: pgmap v2275: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:31:52.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:31:52 smithi115 conmon[90393]: 2022-05-14T00:31:52.322+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:31:52.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:31:52 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:31:52.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:31:52 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:31:53.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:31:53 smithi115 conmon[90393]: 2022-05-14T00:31:53.294+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:31:53.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:31:53 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:31:53.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:31:53 smithi115 ceph-mon[78593]: pgmap v2276: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:31:53.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:31:53 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:31:53.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:31:53 smithi047 ceph-mon[77147]: pgmap v2276: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:31:54.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:31:54 smithi115 conmon[90393]: 2022-05-14T00:31:54.273+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:31:54.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:31:54 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:31:54.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:31:54 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:31:54.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:31:54 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:31:54.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:31:54 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 3845 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:31:54.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:31:54 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:31:54.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:31:54 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:31:54.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:31:54 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:31:54.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:31:54 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 3845 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:31:55.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:31:55 smithi115 conmon[90393]: 2022-05-14T00:31:55.299+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:31:55.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:31:55 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:31:55.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:31:55 smithi115 ceph-mon[78593]: pgmap v2277: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:31:55.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:31:55 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:31:55.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:31:55 smithi047 ceph-mon[77147]: pgmap v2277: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:31:56.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:31:56 smithi115 conmon[90393]: 2022-05-14T00:31:56.338+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:31:56.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:31:56 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:31:56.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:31:56 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:31:57.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:31:57 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:31:57.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:31:57 smithi047 ceph-mon[77147]: pgmap v2278: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:31:57.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:31:57 smithi115 conmon[90393]: 2022-05-14T00:31:57.366+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:31:57.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:31:57 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:31:57.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:31:57 smithi115 ceph-mon[78593]: pgmap v2278: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:31:58.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:31:58 smithi115 conmon[90393]: 2022-05-14T00:31:58.354+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:31:58.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:31:58 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:31:58.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:31:58 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:31:58.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:31:58 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:31:58.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:31:58 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:31:58.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:31:58 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:31:58.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:31:58 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:31:59.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:31:59 smithi115 conmon[90393]: 2022-05-14T00:31:59.350+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:31:59.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:31:59 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:31:59.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:31:59 smithi115 ceph-mon[78593]: pgmap v2279: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:31:59.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:31:59 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 3850 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:31:59.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:31:59 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:31:59.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:31:59 smithi047 ceph-mon[77147]: pgmap v2279: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:31:59.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:31:59 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 3850 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:32:00.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:32:00 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:32:00.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:32:00 smithi115 conmon[90393]: 2022-05-14T00:32:00.382+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:32:00.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:32:00 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:32:01.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:32:01 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:32:01.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:32:01 smithi047 ceph-mon[77147]: pgmap v2280: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:32:01.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:32:01 smithi115 conmon[90393]: 2022-05-14T00:32:01.393+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:32:01.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:32:01 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:32:01.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:32:01 smithi115 ceph-mon[78593]: pgmap v2280: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:32:02.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:32:02 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:32:02.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:32:02 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:32:02.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:32:02 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:32:02.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:32:02 smithi115 conmon[90393]: 2022-05-14T00:32:02.424+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:32:02.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:32:02 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:32:02.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:32:02 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:32:02.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:32:02 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:32:03.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:32:03 smithi047 ceph-mon[77147]: pgmap v2281: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:32:03.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:32:03 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:32:03.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:32:03 smithi115 conmon[90393]: 2022-05-14T00:32:03.399+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:32:03.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:32:03 smithi115 ceph-mon[78593]: pgmap v2281: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:32:03.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:32:03 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:32:04.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:32:04 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:32:04.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:32:04 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 3855 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:32:04.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:32:04 smithi115 conmon[90393]: 2022-05-14T00:32:04.375+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:32:04.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:32:04 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:32:04.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:32:04 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 3855 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:32:05.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:32:05 smithi115 conmon[90393]: 2022-05-14T00:32:05.336+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:32:05.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:32:05 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:32:05.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:32:05 smithi115 ceph-mon[78593]: pgmap v2282: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:32:05.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:32:05 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:32:05.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:32:05 smithi047 ceph-mon[77147]: pgmap v2282: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:32:06.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:32:06 smithi115 conmon[90393]: 2022-05-14T00:32:06.290+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:32:06.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:32:06 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:32:06.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:32:06 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:32:07.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:32:07 smithi115 conmon[90393]: 2022-05-14T00:32:07.255+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:32:07.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:32:07 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:32:07.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:32:07 smithi115 ceph-mon[78593]: pgmap v2283: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:32:07.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:32:07 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:32:07.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:32:07 smithi047 ceph-mon[77147]: pgmap v2283: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:32:08.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:32:08 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:32:08.610 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:32:08 smithi115 conmon[90393]: 2022-05-14T00:32:08.273+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:32:08.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:32:08 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:32:09.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:32:09 smithi115 conmon[90393]: 2022-05-14T00:32:09.302+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:32:09.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:32:09 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:32:09.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:32:09 smithi115 ceph-mon[78593]: pgmap v2284: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:32:09.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:32:09 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 3860 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:32:09.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:32:09 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:32:09.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:32:09 smithi047 ceph-mon[77147]: pgmap v2284: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:32:09.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:32:09 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 3860 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:32:10.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:32:10 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:32:10.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:32:10 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:32:10.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:32:10 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:32:10.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:32:10 smithi115 conmon[90393]: 2022-05-14T00:32:10.324+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:32:10.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:32:10 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:32:10.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:32:10 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:32:10.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:32:10 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:32:11.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:32:11 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:32:11.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:32:11 smithi115 ceph-mon[78593]: pgmap v2285: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:32:11.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:32:11 smithi115 conmon[90393]: 2022-05-14T00:32:11.305+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:32:11.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:32:11 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:32:11.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:32:11 smithi047 ceph-mon[77147]: pgmap v2285: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:32:12.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:32:12 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:32:12.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:32:12 smithi115 conmon[90393]: 2022-05-14T00:32:12.287+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:32:12.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:32:12 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:32:13.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:32:13 smithi115 conmon[90393]: 2022-05-14T00:32:13.297+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:32:13.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:32:13 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:32:13.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:32:13 smithi115 ceph-mon[78593]: pgmap v2286: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:32:13.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:32:13 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T00:32:13.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:32:13 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T00:32:13.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:32:13 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:32:13.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:32:13 smithi047 ceph-mon[77147]: pgmap v2286: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:32:13.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:32:13 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T00:32:13.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:32:13 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T00:32:14.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:32:14 smithi115 conmon[90393]: 2022-05-14T00:32:14.277+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:32:14.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:32:14 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:32:14.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:32:14 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:32:14.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:32:14 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:32:14.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:32:14 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 3865 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:32:14.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:32:14 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:32:14.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:32:14 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:32:14.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:32:14 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:32:14.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:32:14 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 3865 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:32:15.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:32:15 smithi115 conmon[90393]: 2022-05-14T00:32:15.243+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:32:15.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:32:15 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:32:15.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:32:15 smithi115 ceph-mon[78593]: pgmap v2287: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:32:15.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:32:15 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:32:15.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:32:15 smithi047 ceph-mon[77147]: pgmap v2287: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:32:16.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:32:16 smithi115 conmon[90393]: 2022-05-14T00:32:16.230+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:32:16.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:32:16 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:32:16.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:32:16 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:32:17.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:32:17 smithi115 conmon[90393]: 2022-05-14T00:32:17.275+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:32:17.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:32:17 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:32:17.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:32:17 smithi115 ceph-mon[78593]: pgmap v2288: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:32:17.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:32:17 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:32:17.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:32:17 smithi047 ceph-mon[77147]: pgmap v2288: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:32:18.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:32:18 smithi115 conmon[90393]: 2022-05-14T00:32:18.253+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:32:18.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:32:18 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:32:18.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:32:18 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:32:18.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:32:18 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:32:18.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:32:18 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:32:18.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:32:18 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:32:18.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:32:18 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:32:19.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:32:19 smithi115 conmon[90393]: 2022-05-14T00:32:19.231+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:32:19.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:32:19 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:32:19.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:32:19 smithi115 ceph-mon[78593]: pgmap v2289: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:32:19.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:32:19 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 3870 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:32:19.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:32:19 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:32:19.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:32:19 smithi047 ceph-mon[77147]: pgmap v2289: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:32:19.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:32:19 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 3870 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:32:20.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:32:20 smithi115 conmon[90393]: 2022-05-14T00:32:20.257+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:32:20.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:32:20 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:32:20.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:32:20 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:32:21.485 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:32:21 smithi115 conmon[90393]: 2022-05-14T00:32:21.221+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:32:21.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:32:21 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:32:21.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:32:21 smithi047 ceph-mon[77147]: pgmap v2290: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:32:21.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:32:21 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:32:21.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:32:21 smithi115 ceph-mon[78593]: pgmap v2290: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:32:22.489 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:32:22 smithi115 conmon[90393]: 2022-05-14T00:32:22.177+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:32:22.783 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:32:22 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:32:22.784 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:32:22 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:32:22.784 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:32:22 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:32:22.785 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:32:22 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:32:22.785 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:32:22 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:32:22.786 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:32:22 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:32:23.491 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:32:23 smithi115 conmon[90393]: 2022-05-14T00:32:23.178+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:32:23.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:32:23 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:32:23.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:32:23 smithi047 ceph-mon[77147]: pgmap v2291: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:32:23.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:32:23 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T00:32:23.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:32:23 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T00:32:23.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:32:23 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T00:32:23.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:32:23 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:32:23.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:32:23 smithi115 ceph-mon[78593]: pgmap v2291: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:32:23.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:32:23 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T00:32:23.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:32:23 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T00:32:23.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:32:23 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T00:32:24.492 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:32:24 smithi115 conmon[90393]: 2022-05-14T00:32:24.202+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:32:24.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:32:24 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:32:24.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:32:24 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 3875 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:32:24.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:32:24 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:32:24.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:32:24 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 3875 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:32:25.493 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:32:25 smithi115 conmon[90393]: 2022-05-14T00:32:25.238+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:32:25.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:32:25 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:32:25.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:32:25 smithi047 ceph-mon[77147]: pgmap v2292: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:32:25.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:32:25 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:32:25.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:32:25 smithi115 ceph-mon[78593]: pgmap v2292: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:32:26.291 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:32:25 smithi047 conmon[78521]: 2022-05-14T00:32:25.839+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:32:26.291 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:32:25 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T00:32:26.291 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:32:25 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:32:26.292 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:32:25 smithi047 conmon[78521]: yield 2022-05-14T00:32:26.292 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:32:25 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:32:26.292 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:32:25 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:32:26.292 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:32:25 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:32:26.293 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:32:25 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T00:32:26.293 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:32:25 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:32:26.293 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:32:25 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T00:32:26.293 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:32:25 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:32:26.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:32:25 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T00:32:26.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:32:25 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:32:26.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:32:25 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T00:32:26.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:32:25 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:32:26.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:32:25 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T00:32:26.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:32:25 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:32:26.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:32:25 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:32:26.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:32:25 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:32:26.498 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:32:26 smithi115 conmon[90393]: 2022-05-14T00:32:26.190+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:32:26.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:32:26 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:32:26.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:32:26 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:32:26.791 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T00:32:26.791 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:32:26.792 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T00:32:26.792 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:32:26.792 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:32:26.792 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:32:26.793 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T00:32:26.793 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:32:26.793 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T00:32:26.793 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:32:26.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T00:32:26.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:32:26.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T00:32:26.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:32:26.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T00:32:26.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:32:26.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:32:26.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:32:26.796 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:32:26 smithi047 ceph-mon[77147]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T00:32:26.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:32:26 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:32:26.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:32:26 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:32:26.858 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T00:32:26.858 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:32:26.859 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T00:32:26.859 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:32:26.859 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:32:26.859 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:32:26.860 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T00:32:26.860 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:32:26.860 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T00:32:26.860 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:32:26.860 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T00:32:26.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:32:26.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T00:32:26.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:32:26.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T00:32:26.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:32:26.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:32:26.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:32:26.862 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:32:26 smithi115 ceph-mon[78593]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T00:32:27.500 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:32:27 smithi115 conmon[90393]: 2022-05-14T00:32:27.218+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:32:27.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:32:27 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:32:27.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:32:27 smithi047 ceph-mon[77147]: pgmap v2293: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:32:27.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:32:27 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:32:27.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:32:27 smithi115 ceph-mon[78593]: pgmap v2293: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:32:28.504 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:32:28 smithi115 conmon[90393]: 2022-05-14T00:32:28.221+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:32:28.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:32:28 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:32:28.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:32:28 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:32:29.291 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:32:28 smithi047 conmon[78521]: 2022-05-14T00:32:28.911+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:32:29.292 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:32:28 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T00:32:29.292 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:32:28 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:32:29.292 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:32:28 smithi047 conmon[78521]: yield 2022-05-14T00:32:29.293 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:32:28 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:32:29.293 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:32:28 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:32:29.293 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:32:28 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:32:29.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:32:28 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T00:32:29.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:32:28 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:32:29.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:32:28 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T00:32:29.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:32:28 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:32:29.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:32:28 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T00:32:29.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:32:28 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:32:29.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:32:28 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T00:32:29.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:32:28 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:32:29.297 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:32:28 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T00:32:29.297 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:32:28 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:32:29.298 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:32:28 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:32:29.298 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:32:28 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:32:29.298 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:32:28 smithi047 conmon[78521]: 2022-05-14T00:32:28.912+0000 7f5340f7a700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T00:32:29.299 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:32:28 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T00:32:29.299 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:32:28 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:32:29.300 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:32:28 smithi047 conmon[78521]: yield 2022-05-14T00:32:29.300 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:32:28 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:32:29.300 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:32:28 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:32:29.301 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:32:28 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:32:29.301 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:32:28 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T00:32:29.302 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:32:28 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:32:29.302 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:32:28 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T00:32:29.302 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:32:28 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:32:29.303 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:32:28 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T00:32:29.303 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:32:28 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:32:29.304 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:32:28 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T00:32:29.304 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:32:28 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:32:29.304 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:32:28 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T00:32:29.305 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:32:28 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:32:29.305 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:32:28 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:32:29.305 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:32:28 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:32:29.306 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:32:28 smithi047 conmon[78521]: 2022-05-14T00:32:29.306 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:32:28 smithi047 conmon[78521]: During handling of the above exception, another exception occurred: 2022-05-14T00:32:29.306 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:32:28 smithi047 conmon[78521]: 2022-05-14T00:32:29.306 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:32:28 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T00:32:29.307 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:32:28 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T00:32:29.307 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:32:28 smithi047 conmon[78521]: return f(*arg) 2022-05-14T00:32:29.307 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:32:28 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T00:32:29.308 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:32:28 smithi047 conmon[78521]: self._write_client_files(client_files, host) 2022-05-14T00:32:29.308 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:32:28 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T00:32:29.308 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:32:28 smithi047 conmon[78521]: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T00:32:29.308 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:32:28 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T00:32:29.309 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:32:28 smithi047 conmon[78521]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T00:32:29.309 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:32:28 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T00:32:29.309 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:32:28 smithi047 conmon[78521]: return self.event_loop.get_result(coro) 2022-05-14T00:32:29.310 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:32:28 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T00:32:29.310 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:32:28 smithi047 conmon[78521]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T00:32:29.310 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:32:28 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T00:32:29.311 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:32:28 smithi047 conmon[78521]: return self.__get_result() 2022-05-14T00:32:29.311 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:32:28 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T00:32:29.311 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:32:28 smithi047 conmon[78521]: raise self._exception 2022-05-14T00:32:29.312 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:32:28 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T00:32:29.312 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:32:28 smithi047 conmon[78521]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T00:32:29.312 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:32:28 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T00:32:29.313 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:32:28 smithi047 conmon[78521]: conn = await self._remote_connection(host, addr) 2022-05-14T00:32:29.313 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:32:28 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T00:32:29.314 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:32:28 smithi047 conmon[78521]: raise 2022-05-14T00:32:29.314 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:32:28 smithi047 conmon[78521]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T00:32:29.314 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:32:28 smithi047 conmon[78521]: self.gen.throw(type, value, traceback) 2022-05-14T00:32:29.315 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:32:28 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T00:32:29.315 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:32:28 smithi047 conmon[78521]: raise OrchestratorError(msg) 2022-05-14T00:32:29.316 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:32:28 smithi047 conmon[78521]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:32:29.510 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:32:29 smithi115 conmon[90393]: 2022-05-14T00:32:29.251+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:32:29.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:32:29 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:32:29.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:32:29 smithi047 ceph-mon[77147]: pgmap v2294: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:32:29.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:32:29 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:32:29.792 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T00:32:29.792 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:32:29.792 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T00:32:29.793 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:32:29.793 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:32:29.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:32:29.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T00:32:29.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:32:29.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T00:32:29.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:32:29.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T00:32:29.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:32:29.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T00:32:29.797 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:32:29.798 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T00:32:29.799 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:32:29.799 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:32:29.799 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:32:29.800 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:32:29 smithi047 ceph-mon[77147]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T00:32:29.800 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T00:32:29.800 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:32:29.800 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T00:32:29.801 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:32:29.801 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:32:29.801 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:32:29.801 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T00:32:29.802 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:32:29.803 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T00:32:29.803 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:32:29.803 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T00:32:29.803 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:32:29.804 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T00:32:29.804 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:32:29.804 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T00:32:29.804 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:32:29.805 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:32:29.805 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:32:29.805 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T00:32:29.806 INFO:journalctl@ceph.mon.b.smithi047.stdout: During handling of the above exception, another exception occurred: 2022-05-14T00:32:29.806 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T00:32:29.807 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T00:32:29.807 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T00:32:29.807 INFO:journalctl@ceph.mon.b.smithi047.stdout: return f(*arg) 2022-05-14T00:32:29.807 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T00:32:29.808 INFO:journalctl@ceph.mon.b.smithi047.stdout: self._write_client_files(client_files, host) 2022-05-14T00:32:29.808 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T00:32:29.808 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T00:32:29.808 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T00:32:29.808 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T00:32:29.809 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T00:32:29.809 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.event_loop.get_result(coro) 2022-05-14T00:32:29.809 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T00:32:29.809 INFO:journalctl@ceph.mon.b.smithi047.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T00:32:29.810 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T00:32:29.810 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.__get_result() 2022-05-14T00:32:29.810 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T00:32:29.810 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise self._exception 2022-05-14T00:32:29.810 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T00:32:29.811 INFO:journalctl@ceph.mon.b.smithi047.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T00:32:29.811 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T00:32:29.811 INFO:journalctl@ceph.mon.b.smithi047.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T00:32:29.811 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T00:32:29.812 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise 2022-05-14T00:32:29.812 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T00:32:29.812 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.gen.throw(type, value, traceback) 2022-05-14T00:32:29.812 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T00:32:29.813 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OrchestratorError(msg) 2022-05-14T00:32:29.814 INFO:journalctl@ceph.mon.b.smithi047.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:32:29.814 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:32:29 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 3880 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:32:29.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:32:29 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:32:29.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:32:29 smithi115 ceph-mon[78593]: pgmap v2294: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:32:29.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:32:29 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:32:29.859 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T00:32:29.859 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:32:29.859 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T00:32:29.859 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:32:29.860 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:32:29.860 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:32:29.860 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T00:32:29.860 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:32:29.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T00:32:29.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:32:29.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T00:32:29.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:32:29.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T00:32:29.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:32:29.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T00:32:29.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:32:29.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:32:29.863 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:32:29.863 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:32:29 smithi115 ceph-mon[78593]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T00:32:29.863 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T00:32:29.863 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:32:29.864 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T00:32:29.864 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:32:29.864 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:32:29.864 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:32:29.864 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T00:32:29.865 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:32:29.865 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T00:32:29.865 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:32:29.865 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T00:32:29.866 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:32:29.867 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T00:32:29.868 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:32:29.868 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T00:32:29.868 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:32:29.868 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:32:29.868 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:32:29.869 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T00:32:29.869 INFO:journalctl@ceph.mon.c.smithi115.stdout: During handling of the above exception, another exception occurred: 2022-05-14T00:32:29.869 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T00:32:29.869 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T00:32:29.870 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T00:32:29.870 INFO:journalctl@ceph.mon.c.smithi115.stdout: return f(*arg) 2022-05-14T00:32:29.870 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T00:32:29.871 INFO:journalctl@ceph.mon.c.smithi115.stdout: self._write_client_files(client_files, host) 2022-05-14T00:32:29.871 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T00:32:29.871 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T00:32:29.871 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T00:32:29.872 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T00:32:29.872 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T00:32:29.872 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.event_loop.get_result(coro) 2022-05-14T00:32:29.873 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T00:32:29.873 INFO:journalctl@ceph.mon.c.smithi115.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T00:32:29.873 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T00:32:29.873 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.__get_result() 2022-05-14T00:32:29.874 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T00:32:29.874 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise self._exception 2022-05-14T00:32:29.874 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T00:32:29.874 INFO:journalctl@ceph.mon.c.smithi115.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T00:32:29.875 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T00:32:29.875 INFO:journalctl@ceph.mon.c.smithi115.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T00:32:29.875 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T00:32:29.876 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise 2022-05-14T00:32:29.876 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T00:32:29.876 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.gen.throw(type, value, traceback) 2022-05-14T00:32:29.876 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T00:32:29.877 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OrchestratorError(msg) 2022-05-14T00:32:29.877 INFO:journalctl@ceph.mon.c.smithi115.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:32:29.878 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:32:29 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 3880 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:32:30.515 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:32:30 smithi115 conmon[90393]: 2022-05-14T00:32:30.246+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:32:30.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:32:30 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:32:30.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:32:30 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:32:30.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:32:30 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:32:30.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:32:30 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:32:30.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:32:30 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:32:30.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:32:30 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:32:31.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:32:31 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:32:31.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:32:31 smithi115 ceph-mon[78593]: pgmap v2295: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:32:31.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:32:31 smithi115 conmon[90393]: 2022-05-14T00:32:31.289+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:32:31.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:32:31 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:32:31.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:32:31 smithi047 ceph-mon[77147]: pgmap v2295: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:32:32.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:32:32 smithi115 conmon[90393]: 2022-05-14T00:32:32.282+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:32:32.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:32:32 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:32:32.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:32:32 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:32:33.525 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:32:33 smithi115 conmon[90393]: 2022-05-14T00:32:33.241+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:32:33.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:32:33 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:32:33.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:32:33 smithi047 ceph-mon[77147]: pgmap v2296: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:32:33.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:32:33 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:32:33.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:32:33 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:32:33.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:32:33 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:32:33.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:32:33 smithi115 ceph-mon[78593]: pgmap v2296: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:32:33.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:32:33 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:32:33.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:32:33 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:32:34.526 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:32:34 smithi115 conmon[90393]: 2022-05-14T00:32:34.252+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:32:34.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:32:34 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:32:34.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:32:34 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 3885 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:32:34.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:32:34 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:32:34.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:32:34 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 3885 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:32:35.528 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:32:35 smithi115 conmon[90393]: 2022-05-14T00:32:35.249+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:32:35.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:32:35 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:32:35.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:32:35 smithi047 ceph-mon[77147]: pgmap v2297: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:32:35.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:32:35 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:32:35.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:32:35 smithi115 ceph-mon[78593]: pgmap v2297: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:32:36.535 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:32:36 smithi115 conmon[90393]: 2022-05-14T00:32:36.256+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:32:36.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:32:36 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:32:36.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:32:36 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:32:37.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:32:37 smithi115 conmon[90393]: 2022-05-14T00:32:37.293+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:32:37.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:32:37 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:32:37.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:32:37 smithi115 ceph-mon[78593]: pgmap v2298: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:32:37.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:32:37 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:32:37.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:32:37 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:32:37.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:32:37 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:32:37.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:32:37 smithi047 ceph-mon[77147]: pgmap v2298: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:32:37.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:32:37 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:32:37.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:32:37 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:32:38.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:32:38 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:32:38.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:32:38 smithi115 conmon[90393]: 2022-05-14T00:32:38.292+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:32:38.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:32:38 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:32:39.541 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:32:39 smithi115 conmon[90393]: 2022-05-14T00:32:39.267+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:32:39.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:32:39 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:32:39.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:32:39 smithi115 ceph-mon[78593]: pgmap v2299: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:32:39.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:32:39 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 3890 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:32:40.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:32:39 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:32:40.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:32:39 smithi047 ceph-mon[77147]: pgmap v2299: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:32:40.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:32:39 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 3890 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:32:40.546 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:32:40 smithi115 conmon[90393]: 2022-05-14T00:32:40.296+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:32:40.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:32:40 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:32:41.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:32:40 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:32:41.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:32:41 smithi115 conmon[90393]: 2022-05-14T00:32:41.301+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:32:41.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:32:41 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:32:41.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:32:41 smithi115 ceph-mon[78593]: pgmap v2300: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:32:41.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:32:41 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:32:41.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:32:41 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:32:42.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:32:41 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:32:42.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:32:41 smithi047 ceph-mon[77147]: pgmap v2300: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:32:42.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:32:41 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:32:42.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:32:41 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:32:42.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:32:42 smithi115 conmon[90393]: 2022-05-14T00:32:42.324+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:32:42.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:32:42 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:32:43.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:32:42 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:32:43.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:32:43 smithi115 conmon[90393]: 2022-05-14T00:32:43.320+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:32:43.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:32:43 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:32:43.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:32:43 smithi115 ceph-mon[78593]: pgmap v2301: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:32:44.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:32:43 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:32:44.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:32:43 smithi047 ceph-mon[77147]: pgmap v2301: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:32:44.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:32:44 smithi115 conmon[90393]: 2022-05-14T00:32:44.355+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:32:44.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:32:44 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:32:44.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:32:44 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 3895 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:32:45.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:32:44 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:32:45.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:32:44 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 3895 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:32:45.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:32:45 smithi115 conmon[90393]: 2022-05-14T00:32:45.398+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:32:45.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:32:45 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:32:45.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:32:45 smithi115 ceph-mon[78593]: pgmap v2302: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:32:46.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:32:45 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:32:46.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:32:45 smithi047 ceph-mon[77147]: pgmap v2302: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:32:46.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:32:46 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:32:46.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:32:46 smithi115 conmon[90393]: 2022-05-14T00:32:46.405+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:32:47.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:32:46 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:32:47.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:32:47 smithi115 conmon[90393]: 2022-05-14T00:32:47.401+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:32:47.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:32:47 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:32:47.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:32:47 smithi115 ceph-mon[78593]: pgmap v2303: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:32:48.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:32:47 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:32:48.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:32:47 smithi047 ceph-mon[77147]: pgmap v2303: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:32:48.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:32:48 smithi115 conmon[90393]: 2022-05-14T00:32:48.381+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:32:48.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:32:48 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:32:49.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:32:48 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:32:49.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:32:49 smithi115 conmon[90393]: 2022-05-14T00:32:49.336+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:32:49.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:32:49 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:32:49.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:32:49 smithi115 ceph-mon[78593]: pgmap v2304: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:32:49.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:32:49 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:32:49.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:32:49 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:32:49.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:32:49 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 3900 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:32:50.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:32:49 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:32:50.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:32:49 smithi047 ceph-mon[77147]: pgmap v2304: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:32:50.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:32:49 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:32:50.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:32:49 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:32:50.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:32:49 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 3900 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:32:50.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:32:50 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:32:50.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:32:50 smithi115 conmon[90393]: 2022-05-14T00:32:50.378+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:32:51.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:32:50 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:32:51.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:32:51 smithi115 conmon[90393]: 2022-05-14T00:32:51.404+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:32:51.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:32:51 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:32:51.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:32:51 smithi115 ceph-mon[78593]: pgmap v2305: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:32:52.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:32:51 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:32:52.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:32:51 smithi047 ceph-mon[77147]: pgmap v2305: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:32:52.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:32:52 smithi115 conmon[90393]: 2022-05-14T00:32:52.359+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:32:52.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:32:52 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:32:53.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:32:52 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:32:53.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:32:53 smithi115 conmon[90393]: 2022-05-14T00:32:53.396+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:32:53.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:32:53 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:32:53.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:32:53 smithi115 ceph-mon[78593]: pgmap v2306: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:32:53.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:32:53 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:32:53.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:32:53 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:32:54.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:32:53 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:32:54.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:32:53 smithi047 ceph-mon[77147]: pgmap v2306: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:32:54.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:32:53 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:32:54.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:32:53 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:32:54.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:32:54 smithi115 conmon[90393]: 2022-05-14T00:32:54.386+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:32:54.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:32:54 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:32:54.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:32:54 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 3905 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:32:55.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:32:54 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:32:55.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:32:54 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 3905 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:32:55.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:32:55 smithi115 conmon[90393]: 2022-05-14T00:32:55.417+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:32:55.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:32:55 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:32:55.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:32:55 smithi115 ceph-mon[78593]: pgmap v2307: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:32:56.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:32:55 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:32:56.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:32:55 smithi047 ceph-mon[77147]: pgmap v2307: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:32:56.591 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:32:56 smithi115 conmon[90393]: 2022-05-14T00:32:56.444+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:32:56.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:32:56 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:32:57.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:32:56 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:32:57.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:32:57 smithi115 conmon[90393]: 2022-05-14T00:32:57.414+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:32:57.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:32:57 smithi115 ceph-mon[78593]: pgmap v2308: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:32:57.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:32:57 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:32:57.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:32:57 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:32:57.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:32:57 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:32:58.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:32:57 smithi047 ceph-mon[77147]: pgmap v2308: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:32:58.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:32:57 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:32:58.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:32:57 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:32:58.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:32:57 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:32:58.861 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:32:58 smithi115 conmon[90393]: 2022-05-14T00:32:58.442+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:32:58.862 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:32:58 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:32:59.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:32:58 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:32:59.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:32:59 smithi115 conmon[90393]: 2022-05-14T00:32:59.482+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:32:59.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:32:59 smithi115 ceph-mon[78593]: pgmap v2309: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:32:59.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:32:59 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:32:59.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:32:59 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 3910 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:33:00.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:32:59 smithi047 ceph-mon[77147]: pgmap v2309: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:33:00.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:32:59 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:33:00.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:32:59 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 3910 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:33:00.861 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:33:00 smithi115 conmon[90393]: 2022-05-14T00:33:00.491+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:33:00.862 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:33:00 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:33:01.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:33:00 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:33:01.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:33:01 smithi115 conmon[90393]: 2022-05-14T00:33:01.536+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:33:01.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:33:01 smithi115 ceph-mon[78593]: pgmap v2310: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:33:01.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:33:01 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:33:01.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:33:01 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:33:01.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:33:01 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:33:02.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:33:01 smithi047 ceph-mon[77147]: pgmap v2310: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:33:02.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:33:01 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:33:02.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:33:01 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:33:02.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:33:01 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:33:02.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:33:02 smithi115 conmon[90393]: 2022-05-14T00:33:02.553+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:33:02.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:33:02 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:33:03.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:33:02 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:33:03.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:33:03 smithi115 conmon[90393]: 2022-05-14T00:33:03.528+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:33:03.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:33:03 smithi115 ceph-mon[78593]: pgmap v2311: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:33:03.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:33:03 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:33:04.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:33:03 smithi047 ceph-mon[77147]: pgmap v2311: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:33:04.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:33:03 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:33:04.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:33:04 smithi115 conmon[90393]: 2022-05-14T00:33:04.570+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:33:04.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:33:04 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:33:04.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:33:04 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 3915 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:33:05.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:33:04 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:33:05.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:33:04 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 3915 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:33:05.860 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:33:05 smithi115 conmon[90393]: 2022-05-14T00:33:05.582+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:33:05.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:33:05 smithi115 ceph-mon[78593]: pgmap v2312: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:33:05.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:33:05 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:33:06.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:33:05 smithi047 ceph-mon[77147]: pgmap v2312: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:33:06.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:33:05 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:33:06.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:33:06 smithi115 conmon[90393]: 2022-05-14T00:33:06.560+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:33:06.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:33:06 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:33:07.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:33:06 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:33:07.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:33:07 smithi115 conmon[90393]: 2022-05-14T00:33:07.521+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:33:07.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:33:07 smithi115 ceph-mon[78593]: pgmap v2313: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:33:07.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:33:07 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:33:08.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:33:07 smithi047 ceph-mon[77147]: pgmap v2313: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:33:08.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:33:07 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:33:08.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:33:08 smithi115 conmon[90393]: 2022-05-14T00:33:08.538+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:33:08.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:33:08 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:33:09.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:33:08 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:33:09.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:33:09 smithi115 conmon[90393]: 2022-05-14T00:33:09.498+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:33:09.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:33:09 smithi115 ceph-mon[78593]: pgmap v2314: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:33:09.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:33:09 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:33:09.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:33:09 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:33:09.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:33:09 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:33:09.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:33:09 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 3920 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:33:10.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:33:09 smithi047 ceph-mon[77147]: pgmap v2314: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:33:10.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:33:09 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:33:10.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:33:09 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:33:10.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:33:09 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:33:10.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:33:09 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 3920 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:33:10.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:33:10 smithi115 conmon[90393]: 2022-05-14T00:33:10.523+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:33:10.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:33:10 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:33:11.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:33:10 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:33:11.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:33:11 smithi115 conmon[90393]: 2022-05-14T00:33:11.499+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:33:12.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:33:11 smithi047 ceph-mon[77147]: pgmap v2315: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:33:12.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:33:11 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:33:12.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:33:11 smithi115 ceph-mon[78593]: pgmap v2315: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:33:12.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:33:11 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:33:12.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:33:12 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:33:12.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:33:12 smithi115 conmon[90393]: 2022-05-14T00:33:12.480+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:33:13.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:33:12 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:33:13.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:33:13 smithi115 conmon[90393]: 2022-05-14T00:33:13.454+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:33:13.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:33:13 smithi115 ceph-mon[78593]: pgmap v2316: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:33:13.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:33:13 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:33:13.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:33:13 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:33:13.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:33:13 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:33:13.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:33:13 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T00:33:13.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:33:13 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T00:33:14.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:33:13 smithi047 ceph-mon[77147]: pgmap v2316: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:33:14.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:33:13 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:33:14.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:33:13 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:33:14.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:33:13 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:33:14.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:33:13 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T00:33:14.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:33:13 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T00:33:14.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:33:14 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:33:14.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:33:14 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 3925 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:33:14.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:33:14 smithi115 conmon[90393]: 2022-05-14T00:33:14.438+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:33:15.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:33:14 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:33:15.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:33:14 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 3925 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:33:15.751 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:33:15 smithi115 ceph-mon[78593]: pgmap v2317: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:33:15.752 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:33:15 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:33:15.752 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:33:15 smithi115 conmon[90393]: 2022-05-14T00:33:15.439+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:33:16.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:33:15 smithi047 ceph-mon[77147]: pgmap v2317: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:33:16.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:33:15 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:33:16.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:33:16 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:33:16.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:33:16 smithi115 conmon[90393]: 2022-05-14T00:33:16.412+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:33:17.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:33:16 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:33:17.643 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:33:17 smithi115 conmon[90393]: 2022-05-14T00:33:17.388+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:33:17.991 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:33:17 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:33:17.992 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:33:17 smithi115 ceph-mon[78593]: pgmap v2318: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:33:17.992 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:33:17 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:33:17.992 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:33:17 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:33:18.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:33:17 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:33:18.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:33:17 smithi047 ceph-mon[77147]: pgmap v2318: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:33:18.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:33:17 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:33:18.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:33:17 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:33:18.648 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:33:18 smithi115 conmon[90393]: 2022-05-14T00:33:18.381+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:33:18.649 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:33:18 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:33:19.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:33:18 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:33:19.649 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:33:19 smithi115 conmon[90393]: 2022-05-14T00:33:19.383+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:33:20.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:33:19 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:33:20.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:33:19 smithi047 ceph-mon[77147]: pgmap v2319: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:33:20.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:33:19 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 3930 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:33:20.112 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:33:19 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:33:20.112 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:33:19 smithi115 ceph-mon[78593]: pgmap v2319: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:33:20.113 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:33:19 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 3930 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:33:20.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:33:20 smithi115 conmon[90393]: 2022-05-14T00:33:20.428+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:33:21.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:33:20 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:33:21.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:33:20 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:33:21.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:33:21 smithi115 conmon[90393]: 2022-05-14T00:33:21.456+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:33:21.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:33:21 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:33:21.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:33:21 smithi115 ceph-mon[78593]: pgmap v2320: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:33:21.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:33:21 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:33:21.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:33:21 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:33:22.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:33:21 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:33:22.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:33:21 smithi047 ceph-mon[77147]: pgmap v2320: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:33:22.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:33:21 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:33:22.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:33:21 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:33:22.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:33:22 smithi115 conmon[90393]: 2022-05-14T00:33:22.502+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:33:22.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:33:22 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:33:23.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:33:22 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:33:23.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:33:23 smithi115 conmon[90393]: 2022-05-14T00:33:23.525+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:33:23.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:33:23 smithi115 ceph-mon[78593]: pgmap v2321: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:33:23.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:33:23 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:33:24.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:33:23 smithi047 ceph-mon[77147]: pgmap v2321: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:33:24.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:33:23 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:33:24.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:33:24 smithi115 conmon[90393]: 2022-05-14T00:33:24.492+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:33:24.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:33:24 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:33:24.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:33:24 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 3935 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:33:25.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:33:24 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:33:25.403 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:33:24 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 3935 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:33:25.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:33:25 smithi115 conmon[90393]: 2022-05-14T00:33:25.528+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:33:25.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:33:25 smithi115 ceph-mon[78593]: pgmap v2322: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:33:25.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:33:25 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:33:26.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:33:25 smithi047 ceph-mon[77147]: pgmap v2322: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:33:26.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:33:25 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:33:26.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:33:26 smithi115 conmon[90393]: 2022-05-14T00:33:26.504+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:33:26.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:33:26 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:33:27.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:33:26 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:33:27.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:33:27 smithi115 conmon[90393]: 2022-05-14T00:33:27.548+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:33:27.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:33:27 smithi115 ceph-mon[78593]: pgmap v2323: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:33:27.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:33:27 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:33:28.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:33:27 smithi047 ceph-mon[77147]: pgmap v2323: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:33:28.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:33:27 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:33:28.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:33:28 smithi115 conmon[90393]: 2022-05-14T00:33:28.564+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:33:28.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:33:28 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:33:28.958 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:33:28 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:33:29.679 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:33:29 smithi115 conmon[90393]: 2022-05-14T00:33:29.571+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:33:30.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:33:29 smithi047 ceph-mon[77147]: pgmap v2324: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:33:30.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:33:29 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:33:30.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:33:29 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:33:30.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:33:29 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:33:30.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:33:29 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T00:33:30.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:33:29 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T00:33:30.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:33:29 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T00:33:30.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:33:29 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 3940 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:33:30.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:33:29 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' 2022-05-14T00:33:30.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:33:29 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' 2022-05-14T00:33:30.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:33:29 smithi115 ceph-mon[78593]: pgmap v2324: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:33:30.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:33:29 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:33:30.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:33:29 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:33:30.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:33:29 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:33:30.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:33:29 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T00:33:30.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:33:29 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T00:33:30.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:33:29 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T00:33:30.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:33:29 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 3940 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:33:30.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:33:29 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' 2022-05-14T00:33:30.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:33:29 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' 2022-05-14T00:33:30.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:33:30 smithi115 conmon[90393]: 2022-05-14T00:33:30.586+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:33:30.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:33:30 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:33:31.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:33:30 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:33:31.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:33:31 smithi115 conmon[90393]: 2022-05-14T00:33:31.579+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:33:31.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:33:31 smithi115 ceph-mon[78593]: pgmap v2325: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:33:31.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:33:31 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:33:31.969 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:33:31 smithi047 ceph-mon[77147]: pgmap v2325: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:33:31.970 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:33:31 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:33:32.291 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:33:32 smithi047 conmon[78521]: 2022-05-14T00:33:32.015+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:33:32.291 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:33:32 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T00:33:32.292 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:33:32 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:33:32.292 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:33:32 smithi047 conmon[78521]: yield 2022-05-14T00:33:32.293 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:33:32 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:33:32.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:33:32 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:33:32.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:33:32 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:33:32.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:33:32 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T00:33:32.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:33:32 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:33:32.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:33:32 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T00:33:32.297 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:33:32 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:33:32.297 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:33:32 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T00:33:32.299 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:33:32 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:33:32.299 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:33:32 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T00:33:32.300 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:33:32 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:33:32.301 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:33:32 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T00:33:32.302 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:33:32 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:33:32.302 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:33:32 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:33:32.303 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:33:32 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:33:32.684 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:33:32 smithi115 conmon[90393]: 2022-05-14T00:33:32.579+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:33:32.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:33:32 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:33:32.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:33:32 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' 2022-05-14T00:33:32.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:33:32 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' 2022-05-14T00:33:32.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:33:32 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:33:32.793 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T00:33:32.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:33:32.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T00:33:32.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:33:32.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:33:32.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:33:32.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T00:33:32.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:33:32.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T00:33:32.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:33:32.797 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T00:33:32.797 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:33:32.798 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T00:33:32.798 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:33:32.798 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T00:33:32.798 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:33:32.799 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:33:32.799 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:33:32.799 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:33:32 smithi047 ceph-mon[77147]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T00:33:32.799 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:33:32 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-05-14T00:33:32.800 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:33:32 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-05-14T00:33:32.800 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:33:32 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-05-14T00:33:32.800 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:33:32 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-05-14T00:33:32.800 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:33:32 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-05-14T00:33:32.801 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:33:32 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-05-14T00:33:32.801 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:33:32 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-05-14T00:33:32.801 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:33:32 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-05-14T00:33:33.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:33:32 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:33:33.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:33:32 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' 2022-05-14T00:33:33.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:33:32 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' 2022-05-14T00:33:33.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:33:32 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:33:33.108 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T00:33:33.109 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:33:33.109 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T00:33:33.109 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:33:33.109 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:33:33.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:33:33.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T00:33:33.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:33:33.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T00:33:33.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:33:33.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T00:33:33.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:33:33.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T00:33:33.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:33:33.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T00:33:33.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:33:33.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:33:33.113 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:33:33.113 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:33:32 smithi115 ceph-mon[78593]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T00:33:33.113 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:33:32 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-05-14T00:33:33.113 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:33:32 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-05-14T00:33:33.113 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:33:32 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-05-14T00:33:33.114 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:33:32 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-05-14T00:33:33.114 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:33:32 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-05-14T00:33:33.114 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:33:32 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-05-14T00:33:33.114 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:33:32 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-05-14T00:33:33.115 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:33:32 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-05-14T00:33:33.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:33:33 smithi115 conmon[90393]: 2022-05-14T00:33:33.575+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:33:33.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:33:33 smithi115 ceph-mon[78593]: Adjusting osd_memory_target on smithi115 to 197.0M 2022-05-14T00:33:33.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:33:33 smithi115 ceph-mon[78593]: Unable to set osd_memory_target on smithi115 to 206629785: error parsing value: Value '206629785' is below minimum 939524096 2022-05-14T00:33:33.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:33:33 smithi115 ceph-mon[78593]: pgmap v2326: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:33:33.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:33:33 smithi115 ceph-mon[78593]: Adjusting osd_memory_target on smithi047 to 198.1M 2022-05-14T00:33:33.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:33:33 smithi115 ceph-mon[78593]: Unable to set osd_memory_target on smithi047 to 207755878: error parsing value: Value '207755878' is below minimum 939524096 2022-05-14T00:33:33.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:33:33 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:33:33.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:33:33 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:33:33.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:33:33 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:33:34.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:33:33 smithi047 ceph-mon[77147]: Adjusting osd_memory_target on smithi115 to 197.0M 2022-05-14T00:33:34.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:33:33 smithi047 ceph-mon[77147]: Unable to set osd_memory_target on smithi115 to 206629785: error parsing value: Value '206629785' is below minimum 939524096 2022-05-14T00:33:34.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:33:33 smithi047 ceph-mon[77147]: pgmap v2326: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:33:34.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:33:33 smithi047 ceph-mon[77147]: Adjusting osd_memory_target on smithi047 to 198.1M 2022-05-14T00:33:34.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:33:33 smithi047 ceph-mon[77147]: Unable to set osd_memory_target on smithi047 to 207755878: error parsing value: Value '207755878' is below minimum 939524096 2022-05-14T00:33:34.046 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:33:33 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:33:34.046 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:33:33 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:33:34.047 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:33:33 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:33:34.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:33:34 smithi115 conmon[90393]: 2022-05-14T00:33:34.536+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:33:34.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:33:34 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:33:34.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:33:34 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 3945 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:33:35.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:33:34 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:33:35.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:33:34 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 3945 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:33:35.541 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:33:35 smithi047 conmon[78521]: 2022-05-14T00:33:35.088+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:33:35.541 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:33:35 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T00:33:35.541 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:33:35 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:33:35.542 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:33:35 smithi047 conmon[78521]: yield 2022-05-14T00:33:35.542 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:33:35 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:33:35.542 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:33:35 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:33:35.542 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:33:35 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:33:35.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:33:35 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T00:33:35.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:33:35 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:33:35.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:33:35 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T00:33:35.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:33:35 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:33:35.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:33:35 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T00:33:35.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:33:35 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:33:35.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:33:35 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T00:33:35.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:33:35 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:33:35.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:33:35 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T00:33:35.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:33:35 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:33:35.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:33:35 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:33:35.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:33:35 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:33:35.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:33:35 smithi047 conmon[78521]: 2022-05-14T00:33:35.089+0000 7f5344781700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T00:33:35.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:33:35 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T00:33:35.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:33:35 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:33:35.547 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:33:35 smithi047 conmon[78521]: yield 2022-05-14T00:33:35.548 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:33:35 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:33:35.548 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:33:35 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:33:35.548 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:33:35 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:33:35.548 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:33:35 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T00:33:35.549 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:33:35 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:33:35.549 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:33:35 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T00:33:35.549 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:33:35 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:33:35.550 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:33:35 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T00:33:35.550 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:33:35 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:33:35.550 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:33:35 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T00:33:35.551 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:33:35 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:33:35.551 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:33:35 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T00:33:35.551 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:33:35 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:33:35.551 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:33:35 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:33:35.552 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:33:35 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:33:35.552 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:33:35 smithi047 conmon[78521]: 2022-05-14T00:33:35.553 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:33:35 smithi047 conmon[78521]: During handling of the above exception, another exception occurred: 2022-05-14T00:33:35.555 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:33:35 smithi047 conmon[78521]: 2022-05-14T00:33:35.555 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:33:35 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T00:33:35.555 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:33:35 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T00:33:35.555 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:33:35 smithi047 conmon[78521]: return f(*arg) 2022-05-14T00:33:35.556 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:33:35 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T00:33:35.557 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:33:35 smithi047 conmon[78521]: self._write_client_files(client_files, host) 2022-05-14T00:33:35.557 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:33:35 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T00:33:35.558 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:33:35 smithi047 conmon[78521]: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T00:33:35.558 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:33:35 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T00:33:35.559 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:33:35 smithi047 conmon[78521]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T00:33:35.559 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:33:35 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T00:33:35.559 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:33:35 smithi047 conmon[78521]: return self.event_loop.get_result(coro) 2022-05-14T00:33:35.559 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:33:35 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T00:33:35.560 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:33:35 smithi047 conmon[78521]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T00:33:35.560 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:33:35 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T00:33:35.560 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:33:35 smithi047 conmon[78521]: return self.__get_result() 2022-05-14T00:33:35.560 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:33:35 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T00:33:35.561 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:33:35 smithi047 conmon[78521]: raise self._exception 2022-05-14T00:33:35.561 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:33:35 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T00:33:35.561 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:33:35 smithi047 conmon[78521]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T00:33:35.562 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:33:35 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T00:33:35.562 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:33:35 smithi047 conmon[78521]: conn = await self._remote_connection(host, addr) 2022-05-14T00:33:35.562 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:33:35 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T00:33:35.563 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:33:35 smithi047 conmon[78521]: raise 2022-05-14T00:33:35.563 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:33:35 smithi047 conmon[78521]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T00:33:35.564 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:33:35 smithi047 conmon[78521]: self.gen.throw(type, value, traceback) 2022-05-14T00:33:35.564 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:33:35 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T00:33:35.564 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:33:35 smithi047 conmon[78521]: raise OrchestratorError(msg) 2022-05-14T00:33:35.564 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:33:35 smithi047 conmon[78521]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:33:35.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:33:35 smithi115 conmon[90393]: 2022-05-14T00:33:35.570+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:33:35.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:33:35 smithi115 ceph-mon[78593]: pgmap v2327: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:33:35.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:33:35 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:33:35.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:33:35 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:33:35.859 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T00:33:35.859 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:33:35.859 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T00:33:35.859 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:33:35.860 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:33:35.860 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:33:35.860 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T00:33:35.860 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:33:35.860 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T00:33:35.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:33:35.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T00:33:35.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:33:35.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T00:33:35.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:33:35.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T00:33:35.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:33:35.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:33:35.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:33:35.863 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:33:35 smithi115 ceph-mon[78593]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T00:33:35.863 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T00:33:35.863 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:33:35.863 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T00:33:35.863 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:33:35.864 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:33:35.864 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:33:35.864 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T00:33:35.864 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:33:35.864 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T00:33:35.865 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:33:35.865 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T00:33:35.865 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:33:35.865 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T00:33:35.866 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:33:35.866 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T00:33:35.866 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:33:35.866 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:33:35.866 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:33:35.867 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T00:33:35.867 INFO:journalctl@ceph.mon.c.smithi115.stdout: During handling of the above exception, another exception occurred: 2022-05-14T00:33:35.867 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T00:33:35.867 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T00:33:35.868 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T00:33:35.868 INFO:journalctl@ceph.mon.c.smithi115.stdout: return f(*arg) 2022-05-14T00:33:35.868 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T00:33:35.868 INFO:journalctl@ceph.mon.c.smithi115.stdout: self._write_client_files(client_files, host) 2022-05-14T00:33:35.868 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T00:33:35.869 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T00:33:35.869 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T00:33:35.869 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T00:33:35.870 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T00:33:35.870 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.event_loop.get_result(coro) 2022-05-14T00:33:35.870 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T00:33:35.870 INFO:journalctl@ceph.mon.c.smithi115.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T00:33:35.871 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T00:33:35.871 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.__get_result() 2022-05-14T00:33:35.871 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T00:33:35.871 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise self._exception 2022-05-14T00:33:35.871 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T00:33:35.872 INFO:journalctl@ceph.mon.c.smithi115.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T00:33:35.872 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T00:33:35.872 INFO:journalctl@ceph.mon.c.smithi115.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T00:33:35.872 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T00:33:35.873 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise 2022-05-14T00:33:35.873 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T00:33:35.873 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.gen.throw(type, value, traceback) 2022-05-14T00:33:35.873 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T00:33:35.873 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OrchestratorError(msg) 2022-05-14T00:33:35.874 INFO:journalctl@ceph.mon.c.smithi115.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:33:36.046 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:33:35 smithi047 ceph-mon[77147]: pgmap v2327: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:33:36.046 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:33:35 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:33:36.047 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:33:35 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:33:36.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T00:33:36.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:33:36.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T00:33:36.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:33:36.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:33:36.049 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:33:36.049 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T00:33:36.049 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:33:36.049 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T00:33:36.050 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:33:36.050 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T00:33:36.050 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:33:36.050 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T00:33:36.050 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:33:36.051 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T00:33:36.051 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:33:36.051 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:33:36.051 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:33:36.052 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:33:35 smithi047 ceph-mon[77147]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T00:33:36.052 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T00:33:36.052 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:33:36.052 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T00:33:36.052 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:33:36.053 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:33:36.053 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:33:36.053 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T00:33:36.053 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:33:36.054 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T00:33:36.054 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:33:36.054 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T00:33:36.054 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:33:36.054 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T00:33:36.055 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:33:36.055 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T00:33:36.055 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:33:36.055 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:33:36.056 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:33:36.056 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T00:33:36.056 INFO:journalctl@ceph.mon.b.smithi047.stdout: During handling of the above exception, another exception occurred: 2022-05-14T00:33:36.056 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T00:33:36.056 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T00:33:36.057 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T00:33:36.058 INFO:journalctl@ceph.mon.b.smithi047.stdout: return f(*arg) 2022-05-14T00:33:36.058 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T00:33:36.059 INFO:journalctl@ceph.mon.b.smithi047.stdout: self._write_client_files(client_files, host) 2022-05-14T00:33:36.059 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T00:33:36.060 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T00:33:36.063 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T00:33:36.063 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T00:33:36.063 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T00:33:36.064 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.event_loop.get_result(coro) 2022-05-14T00:33:36.068 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T00:33:36.069 INFO:journalctl@ceph.mon.b.smithi047.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T00:33:36.069 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T00:33:36.070 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.__get_result() 2022-05-14T00:33:36.070 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T00:33:36.070 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise self._exception 2022-05-14T00:33:36.071 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T00:33:36.071 INFO:journalctl@ceph.mon.b.smithi047.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T00:33:36.072 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T00:33:36.072 INFO:journalctl@ceph.mon.b.smithi047.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T00:33:36.073 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T00:33:36.073 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise 2022-05-14T00:33:36.074 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T00:33:36.074 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.gen.throw(type, value, traceback) 2022-05-14T00:33:36.074 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T00:33:36.075 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OrchestratorError(msg) 2022-05-14T00:33:36.075 INFO:journalctl@ceph.mon.b.smithi047.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:33:36.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:33:36 smithi115 conmon[90393]: 2022-05-14T00:33:36.527+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:33:36.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:33:36 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:33:37.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:33:36 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:33:37.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:33:37 smithi115 conmon[90393]: 2022-05-14T00:33:37.512+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:33:37.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:33:37 smithi115 ceph-mon[78593]: pgmap v2328: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:33:37.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:33:37 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:33:37.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:33:37 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:33:37.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:33:37 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:33:38.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:33:37 smithi047 ceph-mon[77147]: pgmap v2328: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:33:38.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:33:37 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:33:38.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:33:37 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:33:38.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:33:37 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:33:38.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:33:38 smithi115 conmon[90393]: 2022-05-14T00:33:38.533+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:33:38.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:33:38 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:33:39.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:33:38 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:33:39.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:33:39 smithi115 conmon[90393]: 2022-05-14T00:33:39.503+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:33:39.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:33:39 smithi115 ceph-mon[78593]: pgmap v2329: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:33:39.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:33:39 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:33:39.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:33:39 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 3950 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:33:40.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:33:39 smithi047 ceph-mon[77147]: pgmap v2329: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:33:40.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:33:39 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:33:40.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:33:39 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 3950 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:33:40.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:33:40 smithi115 conmon[90393]: 2022-05-14T00:33:40.482+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:33:40.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:33:40 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:33:41.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:33:40 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:33:41.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:33:41 smithi115 conmon[90393]: 2022-05-14T00:33:41.468+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:33:41.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:33:41 smithi115 ceph-mon[78593]: pgmap v2330: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:33:41.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:33:41 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:33:41.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:33:41 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:33:41.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:33:41 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:33:42.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:33:41 smithi047 ceph-mon[77147]: pgmap v2330: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:33:42.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:33:41 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:33:42.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:33:41 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:33:42.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:33:41 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:33:42.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:33:42 smithi115 conmon[90393]: 2022-05-14T00:33:42.489+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:33:42.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:33:42 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:33:43.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:33:42 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:33:43.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:33:43 smithi115 conmon[90393]: 2022-05-14T00:33:43.509+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:33:43.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:33:43 smithi115 ceph-mon[78593]: pgmap v2331: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:33:43.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:33:43 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:33:44.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:33:43 smithi047 ceph-mon[77147]: pgmap v2331: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:33:44.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:33:43 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:33:44.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:33:44 smithi115 conmon[90393]: 2022-05-14T00:33:44.485+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:33:44.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:33:44 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:33:44.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:33:44 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 3955 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:33:45.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:33:44 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:33:45.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:33:44 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 3955 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:33:45.722 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:33:45 smithi115 conmon[90393]: 2022-05-14T00:33:45.453+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:33:46.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:33:45 smithi047 ceph-mon[77147]: pgmap v2332: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:33:46.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:33:45 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:33:46.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:33:45 smithi115 ceph-mon[78593]: pgmap v2332: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:33:46.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:33:45 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:33:46.723 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:33:46 smithi115 conmon[90393]: 2022-05-14T00:33:46.459+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:33:47.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:33:46 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:33:47.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:33:46 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:33:47.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:33:47 smithi115 conmon[90393]: 2022-05-14T00:33:47.495+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:33:47.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:33:47 smithi115 ceph-mon[78593]: pgmap v2333: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:33:47.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:33:47 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:33:48.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:33:47 smithi047 ceph-mon[77147]: pgmap v2333: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:33:48.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:33:47 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:33:48.732 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:33:48 smithi115 conmon[90393]: 2022-05-14T00:33:48.479+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:33:49.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:33:48 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:33:49.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:33:48 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:33:49.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:33:49 smithi115 conmon[90393]: 2022-05-14T00:33:49.517+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:33:49.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:33:49 smithi115 ceph-mon[78593]: pgmap v2334: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:33:49.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:33:49 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:33:49.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:33:49 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:33:49.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:33:49 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:33:49.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:33:49 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 3960 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:33:50.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:33:49 smithi047 ceph-mon[77147]: pgmap v2334: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:33:50.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:33:49 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:33:50.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:33:49 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:33:50.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:33:49 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:33:50.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:33:49 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 3960 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:33:50.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:33:50 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:33:50.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:33:50 smithi115 conmon[90393]: 2022-05-14T00:33:50.495+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:33:51.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:33:50 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:33:51.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:33:51 smithi115 conmon[90393]: 2022-05-14T00:33:51.496+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:33:51.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:33:51 smithi115 ceph-mon[78593]: pgmap v2335: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:33:51.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:33:51 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:33:52.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:33:51 smithi047 ceph-mon[77147]: pgmap v2335: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:33:52.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:33:51 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:33:52.743 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:33:52 smithi115 conmon[90393]: 2022-05-14T00:33:52.487+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:33:53.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:33:52 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:33:53.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:33:52 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:33:53.744 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:33:53 smithi115 conmon[90393]: 2022-05-14T00:33:53.462+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:33:54.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:33:53 smithi047 ceph-mon[77147]: pgmap v2336: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:33:54.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:33:53 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:33:54.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:33:53 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:33:54.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:33:53 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:33:54.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:33:53 smithi115 ceph-mon[78593]: pgmap v2336: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:33:54.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:33:53 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:33:54.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:33:53 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:33:54.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:33:53 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:33:54.746 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:33:54 smithi115 conmon[90393]: 2022-05-14T00:33:54.451+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:33:55.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:33:54 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:33:55.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:33:54 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 3965 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:33:55.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:33:54 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:33:55.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:33:54 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 3965 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:33:55.748 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:33:55 smithi115 conmon[90393]: 2022-05-14T00:33:55.437+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:33:56.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:33:55 smithi047 ceph-mon[77147]: pgmap v2337: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:33:56.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:33:55 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:33:56.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:33:55 smithi115 ceph-mon[78593]: pgmap v2337: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:33:56.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:33:55 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:33:56.753 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:33:56 smithi115 conmon[90393]: 2022-05-14T00:33:56.487+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:33:57.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:33:56 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:33:57.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:33:56 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:33:57.755 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:33:57 smithi115 conmon[90393]: 2022-05-14T00:33:57.495+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:33:58.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:33:57 smithi047 ceph-mon[77147]: pgmap v2338: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:33:58.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:33:57 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:33:58.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:33:57 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:33:58.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:33:57 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:33:58.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:33:57 smithi115 ceph-mon[78593]: pgmap v2338: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:33:58.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:33:57 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:33:58.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:33:57 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:33:58.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:33:57 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:33:58.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:33:58 smithi115 conmon[90393]: 2022-05-14T00:33:58.511+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:33:58.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:33:58 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:33:59.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:33:58 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:33:59.763 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:33:59 smithi115 conmon[90393]: 2022-05-14T00:33:59.503+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:34:00.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:33:59 smithi047 ceph-mon[77147]: pgmap v2339: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:34:00.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:33:59 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:34:00.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:33:59 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 3970 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:34:00.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:33:59 smithi115 ceph-mon[78593]: pgmap v2339: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:34:00.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:33:59 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:34:00.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:33:59 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 3970 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:34:00.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:34:00 smithi115 conmon[90393]: 2022-05-14T00:34:00.544+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:34:00.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:34:00 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:34:01.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:34:00 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:34:01.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:34:01 smithi115 conmon[90393]: 2022-05-14T00:34:01.593+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:34:01.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:34:01 smithi115 ceph-mon[78593]: pgmap v2340: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:34:01.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:34:01 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:34:01.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:34:01 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:34:01.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:34:01 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:34:01.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:34:01 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:34:02.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:34:01 smithi047 ceph-mon[77147]: pgmap v2340: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:34:02.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:34:01 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:34:02.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:34:01 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:34:02.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:34:01 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:34:02.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:34:01 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:34:02.861 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:34:02 smithi115 conmon[90393]: 2022-05-14T00:34:02.569+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:34:02.862 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:34:02 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:34:03.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:34:02 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:34:03.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:34:03 smithi115 conmon[90393]: 2022-05-14T00:34:03.554+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:34:03.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:34:03 smithi115 ceph-mon[78593]: pgmap v2341: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:34:03.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:34:03 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:34:04.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:34:03 smithi047 ceph-mon[77147]: pgmap v2341: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:34:04.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:34:03 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:34:04.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:34:04 smithi115 conmon[90393]: 2022-05-14T00:34:04.546+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:34:04.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:34:04 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 3975 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:34:04.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:34:04 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:34:05.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:34:04 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 3975 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:34:05.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:34:04 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:34:05.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:34:05 smithi115 conmon[90393]: 2022-05-14T00:34:05.549+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:34:06.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:34:06 smithi047 ceph-mon[77147]: pgmap v2342: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:34:06.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:34:06 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:34:06.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:34:06 smithi115 ceph-mon[78593]: pgmap v2342: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:34:06.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:34:06 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:34:06.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:34:06 smithi115 conmon[90393]: 2022-05-14T00:34:06.531+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:34:07.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:34:07 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:34:07.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:34:07 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:34:07.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:34:07 smithi115 conmon[90393]: 2022-05-14T00:34:07.496+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:34:08.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:34:08 smithi047 ceph-mon[77147]: pgmap v2343: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:34:08.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:34:08 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:34:08.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:34:08 smithi115 ceph-mon[78593]: pgmap v2343: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:34:08.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:34:08 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:34:08.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:34:08 smithi115 conmon[90393]: 2022-05-14T00:34:08.479+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:34:09.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:34:09 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:34:09.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:34:09 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 3980 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:34:09.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:34:09 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:34:09.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:34:09 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 3980 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:34:09.861 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:34:09 smithi115 conmon[90393]: 2022-05-14T00:34:09.431+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:34:10.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:34:10 smithi047 ceph-mon[77147]: pgmap v2344: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:34:10.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:34:10 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:34:10.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:34:10 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:34:10.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:34:10 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:34:10.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:34:10 smithi115 ceph-mon[78593]: pgmap v2344: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:34:10.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:34:10 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:34:10.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:34:10 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:34:10.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:34:10 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:34:10.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:34:10 smithi115 conmon[90393]: 2022-05-14T00:34:10.421+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:34:11.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:34:11 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:34:11.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:34:11 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:34:11.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:34:11 smithi115 conmon[90393]: 2022-05-14T00:34:11.463+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:34:12.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:34:12 smithi115 ceph-mon[78593]: pgmap v2345: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:34:12.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:34:12 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:34:12.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:34:12 smithi047 ceph-mon[77147]: pgmap v2345: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:34:12.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:34:12 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:34:12.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:34:12 smithi115 conmon[90393]: 2022-05-14T00:34:12.463+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:34:13.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:34:13 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:34:13.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:34:13 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:34:13.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:34:13 smithi115 conmon[90393]: 2022-05-14T00:34:13.419+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:34:14.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:34:14 smithi115 ceph-mon[78593]: pgmap v2346: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:34:14.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:34:14 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:34:14.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:34:14 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:34:14.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:34:14 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:34:14.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:34:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T00:34:14.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:34:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T00:34:14.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:34:14 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 3985 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:34:14.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:34:14 smithi047 ceph-mon[77147]: pgmap v2346: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:34:14.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:34:14 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:34:14.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:34:14 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:34:14.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:34:14 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:34:14.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:34:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T00:34:14.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:34:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T00:34:14.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:34:14 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 3985 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:34:14.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:34:14 smithi115 conmon[90393]: 2022-05-14T00:34:14.462+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:34:15.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:34:15 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:34:15.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:34:15 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:34:15.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:34:15 smithi115 conmon[90393]: 2022-05-14T00:34:15.439+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:34:16.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:34:16 smithi115 ceph-mon[78593]: pgmap v2347: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:34:16.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:34:16 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:34:16.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:34:16 smithi047 ceph-mon[77147]: pgmap v2347: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:34:16.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:34:16 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:34:16.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:34:16 smithi115 conmon[90393]: 2022-05-14T00:34:16.423+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:34:17.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:34:17 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:34:17.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:34:17 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:34:17.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:34:17 smithi115 conmon[90393]: 2022-05-14T00:34:17.445+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:34:18.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:34:18 smithi115 ceph-mon[78593]: pgmap v2348: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:34:18.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:34:18 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:34:18.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:34:18 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:34:18.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:34:18 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:34:18.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:34:18 smithi047 ceph-mon[77147]: pgmap v2348: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:34:18.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:34:18 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:34:18.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:34:18 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:34:18.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:34:18 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:34:18.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:34:18 smithi115 conmon[90393]: 2022-05-14T00:34:18.463+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:34:19.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:34:19 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:34:19.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:34:19 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 3990 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:34:19.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:34:19 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:34:19.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:34:19 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 3990 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:34:19.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:34:19 smithi115 conmon[90393]: 2022-05-14T00:34:19.476+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:34:20.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:34:20 smithi115 ceph-mon[78593]: pgmap v2349: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:34:20.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:34:20 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:34:20.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:34:20 smithi047 ceph-mon[77147]: pgmap v2349: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:34:20.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:34:20 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:34:20.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:34:20 smithi115 conmon[90393]: 2022-05-14T00:34:20.462+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:34:21.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:34:21 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:34:21.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:34:21 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:34:21.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:34:21 smithi115 conmon[90393]: 2022-05-14T00:34:21.472+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:34:22.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:34:22 smithi115 ceph-mon[78593]: pgmap v2350: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:34:22.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:34:22 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:34:22.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:34:22 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:34:22.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:34:22 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:34:22.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:34:22 smithi047 ceph-mon[77147]: pgmap v2350: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:34:22.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:34:22 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:34:22.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:34:22 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:34:22.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:34:22 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:34:22.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:34:22 smithi115 conmon[90393]: 2022-05-14T00:34:22.499+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:34:23.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:34:23 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:34:23.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:34:23 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:34:23.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:34:23 smithi115 conmon[90393]: 2022-05-14T00:34:23.515+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:34:24.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:34:24 smithi115 ceph-mon[78593]: pgmap v2351: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:34:24.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:34:24 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:34:24.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:34:24 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 3995 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:34:24.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:34:24 smithi047 ceph-mon[77147]: pgmap v2351: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:34:24.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:34:24 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:34:24.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:34:24 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 3995 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:34:24.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:34:24 smithi115 conmon[90393]: 2022-05-14T00:34:24.558+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:34:25.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:34:25 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:34:25.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:34:25 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:34:25.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:34:25 smithi115 conmon[90393]: 2022-05-14T00:34:25.520+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:34:26.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:34:26 smithi115 ceph-mon[78593]: pgmap v2352: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:34:26.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:34:26 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:34:26.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:34:26 smithi047 ceph-mon[77147]: pgmap v2352: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:34:26.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:34:26 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:34:26.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:34:26 smithi115 conmon[90393]: 2022-05-14T00:34:26.513+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:34:27.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:34:27 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:34:27.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:34:27 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:34:27.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:34:27 smithi115 conmon[90393]: 2022-05-14T00:34:27.472+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:34:28.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:34:28 smithi115 ceph-mon[78593]: pgmap v2353: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:34:28.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:34:28 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:34:28.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:34:28 smithi047 ceph-mon[77147]: pgmap v2353: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:34:28.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:34:28 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:34:28.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:34:28 smithi115 conmon[90393]: 2022-05-14T00:34:28.495+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:34:29.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:34:29 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:34:29.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:34:29 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 4000 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:34:29.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:34:29 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:34:29.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:34:29 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 4000 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:34:29.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:34:29 smithi115 conmon[90393]: 2022-05-14T00:34:29.511+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:34:30.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:34:30 smithi115 ceph-mon[78593]: pgmap v2354: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:34:30.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:34:30 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:34:30.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:34:30 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:34:30.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:34:30 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:34:30.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:34:30 smithi047 ceph-mon[77147]: pgmap v2354: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:34:30.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:34:30 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:34:30.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:34:30 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:34:30.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:34:30 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:34:30.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:34:30 smithi115 conmon[90393]: 2022-05-14T00:34:30.560+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:34:31.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:34:31 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:34:31.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:34:31 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:34:31.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:34:31 smithi115 conmon[90393]: 2022-05-14T00:34:31.576+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:34:32.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:34:32 smithi047 ceph-mon[77147]: pgmap v2355: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:34:32.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:34:32 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:34:32.597 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:34:32 smithi115 ceph-mon[78593]: pgmap v2355: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:34:32.597 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:34:32 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:34:32.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:34:32 smithi115 conmon[90393]: 2022-05-14T00:34:32.595+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:34:33.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:34:33 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:34:33.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:34:33 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:34:34.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:34:33 smithi115 conmon[90393]: 2022-05-14T00:34:33.614+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:34:34.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:34:34 smithi047 ceph-mon[77147]: pgmap v2356: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:34:34.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:34:34 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:34:34.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:34:34 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:34:34.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:34:34 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:34:34.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:34:34 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 4005 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:34:34.602 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:34:34 smithi115 ceph-mon[78593]: pgmap v2356: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:34:34.603 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:34:34 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:34:34.603 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:34:34 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:34:34.603 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:34:34 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:34:34.604 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:34:34 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 4005 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:34:34.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:34:34 smithi115 conmon[90393]: 2022-05-14T00:34:34.601+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:34:35.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:34:35 smithi047 ceph-mon[77147]: pgmap v2357: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:34:35.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:34:35 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:34:35.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:34:35 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T00:34:35.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:34:35 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T00:34:35.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:34:35 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T00:34:35.586 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:34:35 smithi115 ceph-mon[78593]: pgmap v2357: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:34:35.587 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:34:35 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:34:35.587 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:34:35 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T00:34:35.587 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:34:35 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T00:34:35.588 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:34:35 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T00:34:35.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:34:35 smithi115 conmon[90393]: 2022-05-14T00:34:35.584+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:34:36.041 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:34:35 smithi047 conmon[78521]: 2022-05-14T00:34:35.696+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:34:36.041 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:34:35 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T00:34:36.042 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:34:35 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:34:36.042 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:34:35 smithi047 conmon[78521]: yield 2022-05-14T00:34:36.042 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:34:35 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:34:36.042 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:34:35 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:34:36.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:34:35 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:34:36.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:34:35 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T00:34:36.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:34:35 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:34:36.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:34:35 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T00:34:36.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:34:35 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:34:36.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:34:35 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T00:34:36.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:34:35 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:34:36.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:34:35 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T00:34:36.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:34:35 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:34:36.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:34:35 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T00:34:36.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:34:35 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:34:36.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:34:35 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:34:36.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:34:35 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:34:36.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:34:36 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:34:36.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:34:36 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:34:36.541 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T00:34:36.541 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:34:36.542 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T00:34:36.542 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:34:36.542 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:34:36.542 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:34:36.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T00:34:36.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:34:36.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T00:34:36.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:34:36.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T00:34:36.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:34:36.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T00:34:36.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:34:36.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T00:34:36.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:34:36.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:34:36.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:34:36.545 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:34:36 smithi047 ceph-mon[77147]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T00:34:36.590 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:34:36 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:34:36.591 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:34:36 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:34:36.591 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T00:34:36.591 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:34:36.591 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T00:34:36.592 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:34:36.592 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:34:36.592 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:34:36.592 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T00:34:36.593 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:34:36.593 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T00:34:36.593 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:34:36.593 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T00:34:36.593 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:34:36.594 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T00:34:36.594 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:34:36.594 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T00:34:36.594 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:34:36.595 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:34:36.595 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:34:36.595 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:34:36 smithi115 ceph-mon[78593]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T00:34:36.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:34:36 smithi115 conmon[90393]: 2022-05-14T00:34:36.589+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:34:37.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:34:37 smithi047 ceph-mon[77147]: pgmap v2358: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:34:37.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:34:37 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:34:37.560 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:34:37 smithi115 ceph-mon[78593]: pgmap v2358: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:34:37.561 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:34:37 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:34:37.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:34:37 smithi115 conmon[90393]: 2022-05-14T00:34:37.558+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:34:38.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:34:38 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:34:38.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:34:38 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:34:38.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:34:38 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:34:38.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:34:38 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:34:38.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:34:38 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:34:38.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:34:38 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:34:38.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:34:38 smithi115 conmon[90393]: 2022-05-14T00:34:38.541+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:34:39.042 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:34:38 smithi047 conmon[78521]: 2022-05-14T00:34:38.768+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:34:39.042 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:34:38 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T00:34:39.042 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:34:38 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:34:39.042 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:34:38 smithi047 conmon[78521]: yield 2022-05-14T00:34:39.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:34:38 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:34:39.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:34:38 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:34:39.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:34:38 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:34:39.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:34:38 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T00:34:39.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:34:38 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:34:39.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:34:38 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T00:34:39.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:34:38 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:34:39.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:34:38 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T00:34:39.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:34:38 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:34:39.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:34:38 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T00:34:39.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:34:38 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:34:39.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:34:38 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T00:34:39.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:34:38 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:34:39.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:34:38 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:34:39.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:34:38 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:34:39.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:34:38 smithi047 conmon[78521]: 2022-05-14T00:34:38.769+0000 7f533ef76700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T00:34:39.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:34:38 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T00:34:39.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:34:38 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:34:39.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:34:38 smithi047 conmon[78521]: yield 2022-05-14T00:34:39.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:34:38 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:34:39.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:34:38 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:34:39.048 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:34:38 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:34:39.048 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:34:38 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T00:34:39.048 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:34:38 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:34:39.048 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:34:38 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T00:34:39.048 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:34:38 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:34:39.049 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:34:38 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T00:34:39.049 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:34:38 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:34:39.053 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:34:38 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T00:34:39.054 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:34:38 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:34:39.054 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:34:38 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T00:34:39.054 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:34:38 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:34:39.054 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:34:38 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:34:39.055 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:34:38 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:34:39.055 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:34:38 smithi047 conmon[78521]: 2022-05-14T00:34:39.055 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:34:38 smithi047 conmon[78521]: During handling of the above exception, another exception occurred: 2022-05-14T00:34:39.055 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:34:38 smithi047 conmon[78521]: 2022-05-14T00:34:39.056 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:34:38 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T00:34:39.056 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:34:38 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T00:34:39.056 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:34:38 smithi047 conmon[78521]: return f(*arg) 2022-05-14T00:34:39.056 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:34:38 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T00:34:39.057 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:34:38 smithi047 conmon[78521]: self._write_client_files(client_files, host) 2022-05-14T00:34:39.057 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:34:38 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T00:34:39.057 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:34:38 smithi047 conmon[78521]: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T00:34:39.057 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:34:38 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T00:34:39.058 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:34:38 smithi047 conmon[78521]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T00:34:39.058 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:34:38 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T00:34:39.058 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:34:38 smithi047 conmon[78521]: return self.event_loop.get_result(coro) 2022-05-14T00:34:39.058 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:34:38 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T00:34:39.059 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:34:38 smithi047 conmon[78521]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T00:34:39.059 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:34:38 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T00:34:39.060 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:34:38 smithi047 conmon[78521]: return self.__get_result() 2022-05-14T00:34:39.060 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:34:38 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T00:34:39.061 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:34:38 smithi047 conmon[78521]: raise self._exception 2022-05-14T00:34:39.061 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:34:38 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T00:34:39.061 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:34:38 smithi047 conmon[78521]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T00:34:39.061 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:34:38 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T00:34:39.062 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:34:38 smithi047 conmon[78521]: conn = await self._remote_connection(host, addr) 2022-05-14T00:34:39.062 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:34:38 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T00:34:39.062 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:34:38 smithi047 conmon[78521]: raise 2022-05-14T00:34:39.062 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:34:38 smithi047 conmon[78521]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T00:34:39.063 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:34:38 smithi047 conmon[78521]: self.gen.throw(type, value, traceback) 2022-05-14T00:34:39.063 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:34:38 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T00:34:39.063 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:34:38 smithi047 conmon[78521]: raise OrchestratorError(msg) 2022-05-14T00:34:39.063 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:34:38 smithi047 conmon[78521]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:34:39.493 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:34:39 smithi115 ceph-mon[78593]: pgmap v2359: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:34:39.494 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:34:39 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:34:39.494 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:34:39 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:34:39.495 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T00:34:39.495 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:34:39.495 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T00:34:39.496 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:34:39.496 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:34:39.496 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:34:39.497 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T00:34:39.497 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:34:39.498 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T00:34:39.498 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:34:39.499 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T00:34:39.499 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:34:39.499 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T00:34:39.500 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:34:39.500 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T00:34:39.500 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:34:39.501 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:34:39.501 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:34:39.501 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:34:39 smithi115 ceph-mon[78593]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T00:34:39.501 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T00:34:39.501 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:34:39.502 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T00:34:39.502 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:34:39.502 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:34:39.502 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:34:39.503 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T00:34:39.503 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:34:39.503 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T00:34:39.503 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:34:39.503 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T00:34:39.504 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:34:39.504 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T00:34:39.504 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:34:39.504 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T00:34:39.504 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:34:39.505 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:34:39.505 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:34:39.505 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T00:34:39.505 INFO:journalctl@ceph.mon.c.smithi115.stdout: During handling of the above exception, another exception occurred: 2022-05-14T00:34:39.505 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T00:34:39.506 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T00:34:39.506 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T00:34:39.506 INFO:journalctl@ceph.mon.c.smithi115.stdout: return f(*arg) 2022-05-14T00:34:39.506 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T00:34:39.507 INFO:journalctl@ceph.mon.c.smithi115.stdout: self._write_client_files(client_files, host) 2022-05-14T00:34:39.507 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T00:34:39.507 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T00:34:39.508 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T00:34:39.508 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T00:34:39.508 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T00:34:39.508 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.event_loop.get_result(coro) 2022-05-14T00:34:39.509 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T00:34:39.509 INFO:journalctl@ceph.mon.c.smithi115.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T00:34:39.509 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T00:34:39.510 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.__get_result() 2022-05-14T00:34:39.510 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T00:34:39.511 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise self._exception 2022-05-14T00:34:39.511 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T00:34:39.511 INFO:journalctl@ceph.mon.c.smithi115.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T00:34:39.511 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T00:34:39.511 INFO:journalctl@ceph.mon.c.smithi115.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T00:34:39.512 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T00:34:39.512 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise 2022-05-14T00:34:39.512 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T00:34:39.512 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.gen.throw(type, value, traceback) 2022-05-14T00:34:39.513 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T00:34:39.513 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OrchestratorError(msg) 2022-05-14T00:34:39.513 INFO:journalctl@ceph.mon.c.smithi115.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:34:39.514 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:34:39 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 4010 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:34:39.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:34:39 smithi047 ceph-mon[77147]: pgmap v2359: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:34:39.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:34:39 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:34:39.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:34:39 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:34:39.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T00:34:39.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:34:39.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T00:34:39.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:34:39.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:34:39.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:34:39.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T00:34:39.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:34:39.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T00:34:39.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:34:39.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T00:34:39.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:34:39.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T00:34:39.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:34:39.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T00:34:39.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:34:39.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:34:39.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:34:39.547 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:34:39 smithi047 ceph-mon[77147]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T00:34:39.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T00:34:39.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:34:39.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T00:34:39.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:34:39.548 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:34:39.548 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:34:39.548 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T00:34:39.548 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:34:39.549 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T00:34:39.549 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:34:39.550 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T00:34:39.550 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:34:39.550 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T00:34:39.550 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:34:39.551 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T00:34:39.551 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:34:39.551 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:34:39.551 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:34:39.551 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T00:34:39.552 INFO:journalctl@ceph.mon.b.smithi047.stdout: During handling of the above exception, another exception occurred: 2022-05-14T00:34:39.552 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T00:34:39.552 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T00:34:39.552 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T00:34:39.553 INFO:journalctl@ceph.mon.b.smithi047.stdout: return f(*arg) 2022-05-14T00:34:39.553 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T00:34:39.553 INFO:journalctl@ceph.mon.b.smithi047.stdout: self._write_client_files(client_files, host) 2022-05-14T00:34:39.553 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T00:34:39.554 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T00:34:39.554 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T00:34:39.554 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T00:34:39.554 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T00:34:39.555 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.event_loop.get_result(coro) 2022-05-14T00:34:39.555 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T00:34:39.555 INFO:journalctl@ceph.mon.b.smithi047.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T00:34:39.555 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T00:34:39.555 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.__get_result() 2022-05-14T00:34:39.556 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T00:34:39.556 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise self._exception 2022-05-14T00:34:39.556 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T00:34:39.556 INFO:journalctl@ceph.mon.b.smithi047.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T00:34:39.557 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T00:34:39.557 INFO:journalctl@ceph.mon.b.smithi047.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T00:34:39.557 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T00:34:39.557 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise 2022-05-14T00:34:39.557 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T00:34:39.558 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.gen.throw(type, value, traceback) 2022-05-14T00:34:39.558 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T00:34:39.558 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OrchestratorError(msg) 2022-05-14T00:34:39.558 INFO:journalctl@ceph.mon.b.smithi047.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:34:39.559 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:34:39 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 4010 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:34:39.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:34:39 smithi115 conmon[90393]: 2022-05-14T00:34:39.493+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:34:40.518 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:34:40 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:34:40.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:34:40 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:34:40.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:34:40 smithi115 conmon[90393]: 2022-05-14T00:34:40.515+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:34:41.503 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:34:41 smithi115 ceph-mon[78593]: pgmap v2360: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:34:41.503 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:34:41 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:34:41.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:34:41 smithi047 ceph-mon[77147]: pgmap v2360: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:34:41.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:34:41 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:34:41.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:34:41 smithi115 conmon[90393]: 2022-05-14T00:34:41.501+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:34:42.503 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:34:42 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:34:42.504 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:34:42 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:34:42.504 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:34:42 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:34:42.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:34:42 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:34:42.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:34:42 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:34:42.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:34:42 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:34:42.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:34:42 smithi115 conmon[90393]: 2022-05-14T00:34:42.502+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:34:43.519 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:34:43 smithi115 ceph-mon[78593]: pgmap v2361: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:34:43.520 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:34:43 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:34:43.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:34:43 smithi047 ceph-mon[77147]: pgmap v2361: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:34:43.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:34:43 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:34:43.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:34:43 smithi115 conmon[90393]: 2022-05-14T00:34:43.519+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:34:44.507 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:34:44 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:34:44.508 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:34:44 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 4015 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:34:44.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:34:44 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:34:44.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:34:44 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 4015 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:34:44.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:34:44 smithi115 conmon[90393]: 2022-05-14T00:34:44.507+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:34:45.501 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:34:45 smithi115 ceph-mon[78593]: pgmap v2362: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:34:45.502 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:34:45 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:34:45.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:34:45 smithi047 ceph-mon[77147]: pgmap v2362: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:34:45.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:34:45 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:34:45.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:34:45 smithi115 conmon[90393]: 2022-05-14T00:34:45.501+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:34:46.503 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:34:46 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:34:46.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:34:46 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:34:46.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:34:46 smithi115 conmon[90393]: 2022-05-14T00:34:46.502+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:34:47.530 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:34:47 smithi115 ceph-mon[78593]: pgmap v2363: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:34:47.530 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:34:47 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:34:47.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:34:47 smithi047 ceph-mon[77147]: pgmap v2363: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:34:47.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:34:47 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:34:47.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:34:47 smithi115 conmon[90393]: 2022-05-14T00:34:47.529+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:34:48.538 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:34:48 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:34:48.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:34:48 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:34:48.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:34:48 smithi115 conmon[90393]: 2022-05-14T00:34:48.538+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:34:49.518 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:34:49 smithi115 ceph-mon[78593]: pgmap v2364: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:34:49.519 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:34:49 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:34:49.519 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:34:49 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 4020 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:34:49.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:34:49 smithi047 ceph-mon[77147]: pgmap v2364: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:34:49.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:34:49 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:34:49.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:34:49 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 4020 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:34:49.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:34:49 smithi115 conmon[90393]: 2022-05-14T00:34:49.518+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:34:50.539 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:34:50 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:34:50.540 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:34:50 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:34:50.540 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:34:50 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:34:50.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:34:50 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:34:50.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:34:50 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:34:50.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:34:50 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:34:50.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:34:50 smithi115 conmon[90393]: 2022-05-14T00:34:50.539+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:34:51.515 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:34:51 smithi115 ceph-mon[78593]: pgmap v2365: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:34:51.517 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:34:51 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:34:51.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:34:51 smithi047 ceph-mon[77147]: pgmap v2365: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:34:51.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:34:51 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:34:51.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:34:51 smithi115 conmon[90393]: 2022-05-14T00:34:51.515+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:34:52.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:34:52 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:34:52.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:34:52 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:34:52.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:34:52 smithi115 conmon[90393]: 2022-05-14T00:34:52.547+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:34:53.501 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:34:53 smithi115 ceph-mon[78593]: pgmap v2366: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:34:53.502 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:34:53 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:34:53.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:34:53 smithi047 ceph-mon[77147]: pgmap v2366: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:34:53.545 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:34:53 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:34:53.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:34:53 smithi115 conmon[90393]: 2022-05-14T00:34:53.499+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:34:54.464 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:34:54 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:34:54.464 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:34:54 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:34:54.464 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:34:54 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:34:54.464 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:34:54 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 4025 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:34:54.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:34:54 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:34:54.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:34:54 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:34:54.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:34:54 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:34:54.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:34:54 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 4025 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:34:54.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:34:54 smithi115 conmon[90393]: 2022-05-14T00:34:54.462+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:34:55.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:34:55 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:34:55.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:34:55 smithi047 ceph-mon[77147]: pgmap v2367: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:34:55.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:34:55 smithi115 conmon[90393]: 2022-05-14T00:34:55.421+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:34:55.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:34:55 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:34:55.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:34:55 smithi115 ceph-mon[78593]: pgmap v2367: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:34:56.438 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:34:56 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:34:56.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:34:56 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:34:56.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:34:56 smithi115 conmon[90393]: 2022-05-14T00:34:56.437+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:34:57.472 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:34:57 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:34:57.473 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:34:57 smithi115 ceph-mon[78593]: pgmap v2368: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:34:57.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:34:57 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:34:57.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:34:57 smithi047 ceph-mon[77147]: pgmap v2368: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:34:57.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:34:57 smithi115 conmon[90393]: 2022-05-14T00:34:57.471+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:34:58.508 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:34:58 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:34:58.509 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:34:58 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:34:58.510 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:34:58 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:34:58.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:34:58 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:34:58.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:34:58 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:34:58.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:34:58 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:34:58.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:34:58 smithi115 conmon[90393]: 2022-05-14T00:34:58.506+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:34:59.515 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:34:59 smithi115 ceph-mon[78593]: pgmap v2369: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:34:59.516 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:34:59 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:34:59.516 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:34:59 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 4030 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:34:59.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:34:59 smithi047 ceph-mon[77147]: pgmap v2369: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:34:59.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:34:59 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:34:59.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:34:59 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 4030 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:34:59.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:34:59 smithi115 conmon[90393]: 2022-05-14T00:34:59.514+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:35:00.526 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:35:00 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:35:00.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:35:00 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:35:00.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:35:00 smithi115 conmon[90393]: 2022-05-14T00:35:00.525+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:35:01.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:35:01 smithi115 ceph-mon[78593]: pgmap v2370: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:35:01.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:35:01 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:35:01.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:35:01 smithi047 ceph-mon[77147]: pgmap v2370: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:35:01.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:35:01 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:35:01.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:35:01 smithi115 conmon[90393]: 2022-05-14T00:35:01.540+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:35:02.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:35:02 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:35:02.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:35:02 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:35:02.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:35:02 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:35:02.560 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:35:02 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:35:02.561 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:35:02 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:35:02.561 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:35:02 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:35:02.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:35:02 smithi115 conmon[90393]: 2022-05-14T00:35:02.559+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:35:03.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:35:03 smithi047 ceph-mon[77147]: pgmap v2371: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:35:03.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:35:03 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:35:03.553 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:35:03 smithi115 ceph-mon[78593]: pgmap v2371: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:35:03.553 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:35:03 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:35:03.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:35:03 smithi115 conmon[90393]: 2022-05-14T00:35:03.551+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:35:04.524 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:35:04 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:35:04.524 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:35:04 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 4035 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:35:04.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:35:04 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:35:04.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:35:04 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 4035 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:35:04.860 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:35:04 smithi115 conmon[90393]: 2022-05-14T00:35:04.523+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:35:05.476 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:35:05 smithi115 ceph-mon[78593]: pgmap v2372: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:35:05.477 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:35:05 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:35:05.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:35:05 smithi047 ceph-mon[77147]: pgmap v2372: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:35:05.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:35:05 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:35:05.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:35:05 smithi115 conmon[90393]: 2022-05-14T00:35:05.475+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:35:06.482 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:35:06 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:35:06.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:35:06 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:35:06.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:35:06 smithi115 conmon[90393]: 2022-05-14T00:35:06.481+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:35:07.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:35:07 smithi047 ceph-mon[77147]: pgmap v2373: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:35:07.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:35:07 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:35:07.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:35:07 smithi115 ceph-mon[78593]: pgmap v2373: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:35:07.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:35:07 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:35:07.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:35:07 smithi115 conmon[90393]: 2022-05-14T00:35:07.464+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:35:08.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:35:08 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:35:08.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:35:08 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:35:08.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:35:08 smithi115 conmon[90393]: 2022-05-14T00:35:08.426+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:35:09.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:35:09 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:35:09.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:35:09 smithi047 ceph-mon[77147]: pgmap v2374: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:35:09.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:35:09 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 4040 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:35:09.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:35:09 smithi115 conmon[90393]: 2022-05-14T00:35:09.403+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:35:09.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:35:09 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:35:09.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:35:09 smithi115 ceph-mon[78593]: pgmap v2374: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:35:09.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:35:09 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 4040 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:35:10.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:35:10 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:35:10.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:35:10 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:35:10.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:35:10 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:35:10.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:35:10 smithi115 conmon[90393]: 2022-05-14T00:35:10.423+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:35:10.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:35:10 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:35:10.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:35:10 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:35:10.612 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:35:10 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:35:11.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:35:11 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:35:11.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:35:11 smithi047 ceph-mon[77147]: pgmap v2375: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:35:11.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:35:11 smithi115 conmon[90393]: 2022-05-14T00:35:11.400+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:35:11.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:35:11 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:35:11.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:35:11 smithi115 ceph-mon[78593]: pgmap v2375: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:35:12.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:35:12 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:35:12.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:35:12 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:35:12.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:35:12 smithi115 conmon[90393]: 2022-05-14T00:35:12.368+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:35:13.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:35:13 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:35:13.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:35:13 smithi047 ceph-mon[77147]: pgmap v2376: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:35:13.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:35:13 smithi115 conmon[90393]: 2022-05-14T00:35:13.337+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:35:13.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:35:13 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:35:13.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:35:13 smithi115 ceph-mon[78593]: pgmap v2376: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:35:14.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:35:14 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:35:14.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:35:14 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:35:14.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:35:14 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:35:14.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:35:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T00:35:14.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:35:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T00:35:14.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:35:14 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 4045 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:35:14.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:35:14 smithi115 conmon[90393]: 2022-05-14T00:35:14.329+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:35:14.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:35:14 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:35:14.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:35:14 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:35:14.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:35:14 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:35:14.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:35:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T00:35:14.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:35:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T00:35:14.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:35:14 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 4045 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:35:15.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:35:15 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:35:15.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:35:15 smithi047 ceph-mon[77147]: pgmap v2377: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:35:15.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:35:15 smithi115 conmon[90393]: 2022-05-14T00:35:15.366+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:35:15.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:35:15 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:35:15.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:35:15 smithi115 ceph-mon[78593]: pgmap v2377: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:35:16.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:35:16 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:35:16.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:35:16 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:35:16.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:35:16 smithi115 conmon[90393]: 2022-05-14T00:35:16.371+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:35:17.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:35:17 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:35:17.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:35:17 smithi047 ceph-mon[77147]: pgmap v2378: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:35:17.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:35:17 smithi115 conmon[90393]: 2022-05-14T00:35:17.411+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:35:17.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:35:17 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:35:17.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:35:17 smithi115 ceph-mon[78593]: pgmap v2378: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:35:18.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:35:18 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:35:18.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:35:18 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:35:18.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:35:18 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:35:18.610 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:35:18 smithi115 conmon[90393]: 2022-05-14T00:35:18.371+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:35:18.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:35:18 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:35:18.613 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:35:18 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:35:18.613 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:35:18 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:35:19.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:35:19 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:35:19.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:35:19 smithi047 ceph-mon[77147]: pgmap v2379: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:35:19.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:35:19 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 4050 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:35:19.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:35:19 smithi115 conmon[90393]: 2022-05-14T00:35:19.344+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:35:19.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:35:19 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:35:19.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:35:19 smithi115 ceph-mon[78593]: pgmap v2379: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:35:19.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:35:19 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 4050 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:35:20.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:35:20 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:35:20.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:35:20 smithi115 conmon[90393]: 2022-05-14T00:35:20.333+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:35:20.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:35:20 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:35:21.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:35:21 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:35:21.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:35:21 smithi047 ceph-mon[77147]: pgmap v2380: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:35:21.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:35:21 smithi115 conmon[90393]: 2022-05-14T00:35:21.378+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:35:21.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:35:21 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:35:21.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:35:21 smithi115 ceph-mon[78593]: pgmap v2380: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:35:22.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:35:22 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:35:22.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:35:22 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:35:22.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:35:22 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:35:22.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:35:22 smithi115 conmon[90393]: 2022-05-14T00:35:22.412+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:35:22.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:35:22 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:35:22.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:35:22 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:35:22.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:35:22 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:35:23.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:35:23 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:35:23.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:35:23 smithi047 ceph-mon[77147]: pgmap v2381: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:35:23.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:35:23 smithi115 conmon[90393]: 2022-05-14T00:35:23.379+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:35:23.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:35:23 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:35:23.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:35:23 smithi115 ceph-mon[78593]: pgmap v2381: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:35:24.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:35:24 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:35:24.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:35:24 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 4055 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:35:24.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:35:24 smithi115 conmon[90393]: 2022-05-14T00:35:24.367+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:35:24.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:35:24 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:35:24.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:35:24 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 4055 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:35:25.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:35:25 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:35:25.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:35:25 smithi047 ceph-mon[77147]: pgmap v2382: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:35:25.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:35:25 smithi115 conmon[90393]: 2022-05-14T00:35:25.357+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:35:25.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:35:25 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:35:25.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:35:25 smithi115 ceph-mon[78593]: pgmap v2382: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:35:26.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:35:26 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:35:26.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:35:26 smithi115 conmon[90393]: 2022-05-14T00:35:26.356+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:35:26.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:35:26 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:35:27.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:35:27 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:35:27.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:35:27 smithi047 ceph-mon[77147]: pgmap v2383: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:35:27.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:35:27 smithi115 conmon[90393]: 2022-05-14T00:35:27.349+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:35:27.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:35:27 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:35:27.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:35:27 smithi115 ceph-mon[78593]: pgmap v2383: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:35:28.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:35:28 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:35:28.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:35:28 smithi115 conmon[90393]: 2022-05-14T00:35:28.341+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:35:28.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:35:28 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:35:29.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:35:29 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:35:29.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:35:29 smithi047 ceph-mon[77147]: pgmap v2384: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:35:29.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:35:29 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 4060 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:35:29.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:35:29 smithi115 conmon[90393]: 2022-05-14T00:35:29.355+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:35:29.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:35:29 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:35:29.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:35:29 smithi115 ceph-mon[78593]: pgmap v2384: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:35:29.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:35:29 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 4060 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:35:30.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:35:30 smithi115 conmon[90393]: 2022-05-14T00:35:30.391+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:35:30.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:35:30 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:35:30.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:35:30 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:35:30.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:35:30 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:35:30.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:35:30 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:35:30.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:35:30 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:35:30.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:35:30 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:35:31.610 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:35:31 smithi115 conmon[90393]: 2022-05-14T00:35:31.407+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:35:31.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:35:31 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:35:31.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:35:31 smithi115 ceph-mon[78593]: pgmap v2385: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:35:31.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:35:31 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:35:31.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:35:31 smithi047 ceph-mon[77147]: pgmap v2385: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:35:32.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:35:32 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:35:32.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:35:32 smithi115 conmon[90393]: 2022-05-14T00:35:32.379+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:35:32.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:35:32 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:35:33.611 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:35:33 smithi115 conmon[90393]: 2022-05-14T00:35:33.403+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:35:33.612 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:35:33 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:35:33.612 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:35:33 smithi115 ceph-mon[78593]: pgmap v2386: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:35:33.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:35:33 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:35:33.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:35:33 smithi047 ceph-mon[77147]: pgmap v2386: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:35:34.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:35:34 smithi115 conmon[90393]: 2022-05-14T00:35:34.361+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:35:34.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:35:34 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:35:34.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:35:34 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:35:34.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:35:34 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:35:34.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:35:34 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 4065 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:35:34.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:35:34 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:35:34.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:35:34 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:35:34.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:35:34 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:35:34.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:35:34 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 4065 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:35:35.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:35:35 smithi115 conmon[90393]: 2022-05-14T00:35:35.321+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:35:35.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:35:35 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:35:35.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:35:35 smithi115 ceph-mon[78593]: pgmap v2387: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:35:35.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:35:35 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:35:35.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:35:35 smithi047 ceph-mon[77147]: pgmap v2387: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:35:36.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:35:36 smithi115 conmon[90393]: 2022-05-14T00:35:36.290+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:35:36.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:35:36 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:35:36.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:35:36 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:35:37.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:35:37 smithi115 conmon[90393]: 2022-05-14T00:35:37.318+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:35:37.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:35:37 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:35:37.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:35:37 smithi115 ceph-mon[78593]: pgmap v2388: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:35:37.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:35:37 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:35:37.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:35:37 smithi047 ceph-mon[77147]: pgmap v2388: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:35:38.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:35:38 smithi115 conmon[90393]: 2022-05-14T00:35:38.367+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:35:38.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:35:38 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:35:38.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:35:38 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:35:38.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:35:38 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:35:38.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:35:38 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:35:38.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:35:38 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:35:38.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:35:38 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:35:39.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:35:39 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:35:39.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:35:39 smithi047 ceph-mon[77147]: pgmap v2389: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:35:39.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:35:39 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T00:35:39.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:35:39 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T00:35:39.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:35:39 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T00:35:39.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:35:39 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 4070 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:35:39.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:35:39 smithi115 conmon[90393]: 2022-05-14T00:35:39.345+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:35:39.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:35:39 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:35:39.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:35:39 smithi115 ceph-mon[78593]: pgmap v2389: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:35:39.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:35:39 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T00:35:39.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:35:39 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T00:35:39.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:35:39 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T00:35:39.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:35:39 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 4070 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:35:40.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:35:40 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:35:40.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:35:40 smithi115 conmon[90393]: 2022-05-14T00:35:40.376+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:35:40.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:35:40 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:35:41.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:35:41 smithi115 conmon[90393]: 2022-05-14T00:35:41.364+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:35:41.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:35:41 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:35:41.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:35:41 smithi115 ceph-mon[78593]: pgmap v2390: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:35:41.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:35:41 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:35:41.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:35:41 smithi047 ceph-mon[77147]: pgmap v2390: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:35:42.291 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:35:41 smithi047 conmon[78521]: 2022-05-14T00:35:41.871+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:35:42.291 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:35:41 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T00:35:42.291 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:35:41 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:35:42.291 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:35:41 smithi047 conmon[78521]: yield 2022-05-14T00:35:42.292 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:35:41 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:35:42.292 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:35:41 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:35:42.292 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:35:41 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:35:42.292 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:35:41 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T00:35:42.293 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:35:41 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:35:42.293 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:35:41 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T00:35:42.293 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:35:41 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:35:42.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:35:41 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T00:35:42.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:35:41 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:35:42.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:35:41 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T00:35:42.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:35:41 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:35:42.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:35:41 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T00:35:42.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:35:41 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:35:42.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:35:41 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:35:42.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:35:41 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:35:42.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:35:42 smithi115 conmon[90393]: 2022-05-14T00:35:42.357+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:35:42.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:35:42 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:35:42.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:35:42 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:35:42.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:35:42 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:35:42.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:35:42 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:35:42.609 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T00:35:42.609 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:35:42.609 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T00:35:42.609 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:35:42.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:35:42.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:35:42.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T00:35:42.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:35:42.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T00:35:42.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:35:42.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T00:35:42.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:35:42.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T00:35:42.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:35:42.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T00:35:42.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:35:42.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:35:42.613 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:35:42.613 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:35:42 smithi115 ceph-mon[78593]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T00:35:42.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:35:42 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:35:42.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:35:42 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:35:42.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:35:42 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:35:42.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:35:42 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:35:42.792 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T00:35:42.792 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:35:42.792 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T00:35:42.792 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:35:42.792 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:35:42.793 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:35:42.793 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T00:35:42.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:35:42.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T00:35:42.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:35:42.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T00:35:42.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:35:42.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T00:35:42.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:35:42.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T00:35:42.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:35:42.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:35:42.797 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:35:42.797 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:35:42 smithi047 ceph-mon[77147]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T00:35:43.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:35:43 smithi115 conmon[90393]: 2022-05-14T00:35:43.343+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:35:43.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:35:43 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:35:43.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:35:43 smithi115 ceph-mon[78593]: pgmap v2391: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:35:43.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:35:43 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:35:43.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:35:43 smithi047 ceph-mon[77147]: pgmap v2391: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:35:44.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:35:44 smithi115 conmon[90393]: 2022-05-14T00:35:44.388+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:35:44.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:35:44 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:35:44.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:35:44 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 4075 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:35:44.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:35:44 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:35:44.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:35:44 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 4075 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:35:45.291 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:35:44 smithi047 conmon[78521]: 2022-05-14T00:35:44.943+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:35:45.291 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:35:44 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T00:35:45.292 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:35:44 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:35:45.292 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:35:44 smithi047 conmon[78521]: yield 2022-05-14T00:35:45.292 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:35:44 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:35:45.292 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:35:44 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:35:45.292 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:35:44 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:35:45.293 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:35:44 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T00:35:45.293 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:35:44 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:35:45.293 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:35:44 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T00:35:45.293 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:35:44 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:35:45.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:35:44 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T00:35:45.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:35:44 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:35:45.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:35:44 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T00:35:45.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:35:44 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:35:45.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:35:44 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T00:35:45.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:35:44 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:35:45.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:35:44 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:35:45.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:35:44 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:35:45.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:35:44 smithi047 conmon[78521]: 2022-05-14T00:35:44.944+0000 7f533f777700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T00:35:45.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:35:44 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T00:35:45.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:35:44 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:35:45.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:35:44 smithi047 conmon[78521]: yield 2022-05-14T00:35:45.297 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:35:44 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:35:45.297 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:35:44 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:35:45.297 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:35:44 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:35:45.297 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:35:44 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T00:35:45.298 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:35:44 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:35:45.298 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:35:44 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T00:35:45.298 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:35:44 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:35:45.298 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:35:44 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T00:35:45.299 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:35:44 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:35:45.299 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:35:44 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T00:35:45.299 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:35:44 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:35:45.299 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:35:44 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T00:35:45.299 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:35:44 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:35:45.300 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:35:44 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:35:45.300 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:35:44 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:35:45.300 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:35:44 smithi047 conmon[78521]: 2022-05-14T00:35:45.300 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:35:44 smithi047 conmon[78521]: During handling of the above exception, another exception occurred: 2022-05-14T00:35:45.301 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:35:44 smithi047 conmon[78521]: 2022-05-14T00:35:45.301 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:35:44 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T00:35:45.301 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:35:44 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T00:35:45.301 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:35:44 smithi047 conmon[78521]: return f(*arg) 2022-05-14T00:35:45.302 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:35:44 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T00:35:45.302 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:35:44 smithi047 conmon[78521]: self._write_client_files(client_files, host) 2022-05-14T00:35:45.302 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:35:44 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T00:35:45.302 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:35:44 smithi047 conmon[78521]: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T00:35:45.302 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:35:44 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T00:35:45.303 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:35:44 smithi047 conmon[78521]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T00:35:45.303 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:35:44 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T00:35:45.303 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:35:44 smithi047 conmon[78521]: return self.event_loop.get_result(coro) 2022-05-14T00:35:45.303 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:35:44 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T00:35:45.304 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:35:44 smithi047 conmon[78521]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T00:35:45.304 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:35:44 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T00:35:45.304 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:35:44 smithi047 conmon[78521]: return self.__get_result() 2022-05-14T00:35:45.305 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:35:44 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T00:35:45.305 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:35:44 smithi047 conmon[78521]: raise self._exception 2022-05-14T00:35:45.305 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:35:44 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T00:35:45.305 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:35:44 smithi047 conmon[78521]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T00:35:45.306 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:35:44 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T00:35:45.306 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:35:44 smithi047 conmon[78521]: conn = await self._remote_connection(host, addr) 2022-05-14T00:35:45.306 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:35:44 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T00:35:45.306 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:35:44 smithi047 conmon[78521]: raise 2022-05-14T00:35:45.307 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:35:44 smithi047 conmon[78521]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T00:35:45.307 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:35:44 smithi047 conmon[78521]: self.gen.throw(type, value, traceback) 2022-05-14T00:35:45.307 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:35:44 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T00:35:45.307 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:35:44 smithi047 conmon[78521]: raise OrchestratorError(msg) 2022-05-14T00:35:45.308 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:35:44 smithi047 conmon[78521]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:35:45.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:35:45 smithi115 conmon[90393]: 2022-05-14T00:35:45.397+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:35:45.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:35:45 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:35:45.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:35:45 smithi115 ceph-mon[78593]: pgmap v2392: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:35:45.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:35:45 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:35:45.609 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T00:35:45.609 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:35:45.609 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T00:35:45.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:35:45.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:35:45.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:35:45.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T00:35:45.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:35:45.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T00:35:45.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:35:45.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T00:35:45.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:35:45.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T00:35:45.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:35:45.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T00:35:45.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:35:45.613 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:35:45.613 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:35:45.613 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:35:45 smithi115 ceph-mon[78593]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T00:35:45.613 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T00:35:45.614 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:35:45.614 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T00:35:45.614 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:35:45.614 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:35:45.615 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:35:45.615 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T00:35:45.615 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:35:45.615 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T00:35:45.616 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:35:45.616 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T00:35:45.616 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:35:45.616 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T00:35:45.617 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:35:45.617 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T00:35:45.617 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:35:45.618 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:35:45.618 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:35:45.618 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T00:35:45.618 INFO:journalctl@ceph.mon.c.smithi115.stdout: During handling of the above exception, another exception occurred: 2022-05-14T00:35:45.618 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T00:35:45.619 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T00:35:45.619 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T00:35:45.619 INFO:journalctl@ceph.mon.c.smithi115.stdout: return f(*arg) 2022-05-14T00:35:45.619 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T00:35:45.619 INFO:journalctl@ceph.mon.c.smithi115.stdout: self._write_client_files(client_files, host) 2022-05-14T00:35:45.620 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T00:35:45.620 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T00:35:45.620 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T00:35:45.620 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T00:35:45.621 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T00:35:45.621 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.event_loop.get_result(coro) 2022-05-14T00:35:45.621 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T00:35:45.621 INFO:journalctl@ceph.mon.c.smithi115.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T00:35:45.621 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T00:35:45.622 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.__get_result() 2022-05-14T00:35:45.622 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T00:35:45.622 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise self._exception 2022-05-14T00:35:45.623 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T00:35:45.623 INFO:journalctl@ceph.mon.c.smithi115.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T00:35:45.623 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T00:35:45.623 INFO:journalctl@ceph.mon.c.smithi115.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T00:35:45.624 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T00:35:45.624 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise 2022-05-14T00:35:45.624 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T00:35:45.624 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.gen.throw(type, value, traceback) 2022-05-14T00:35:45.625 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T00:35:45.625 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OrchestratorError(msg) 2022-05-14T00:35:45.625 INFO:journalctl@ceph.mon.c.smithi115.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:35:45.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:35:45 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:35:45.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:35:45 smithi047 ceph-mon[77147]: pgmap v2392: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:35:45.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:35:45 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:35:45.791 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T00:35:45.792 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:35:45.792 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T00:35:45.792 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:35:45.793 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:35:45.793 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:35:45.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T00:35:45.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:35:45.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T00:35:45.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:35:45.797 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T00:35:45.798 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:35:45.798 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T00:35:45.799 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:35:45.800 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T00:35:45.800 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:35:45.801 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:35:45.801 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:35:45.801 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:35:45 smithi047 ceph-mon[77147]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T00:35:45.802 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T00:35:45.802 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:35:45.803 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T00:35:45.803 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:35:45.803 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:35:45.804 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:35:45.804 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T00:35:45.805 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:35:45.805 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T00:35:45.805 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:35:45.806 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T00:35:45.806 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:35:45.807 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T00:35:45.807 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:35:45.807 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T00:35:45.808 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:35:45.808 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:35:45.809 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:35:45.810 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T00:35:45.810 INFO:journalctl@ceph.mon.b.smithi047.stdout: During handling of the above exception, another exception occurred: 2022-05-14T00:35:45.810 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T00:35:45.811 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T00:35:45.811 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T00:35:45.812 INFO:journalctl@ceph.mon.b.smithi047.stdout: return f(*arg) 2022-05-14T00:35:45.812 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T00:35:45.812 INFO:journalctl@ceph.mon.b.smithi047.stdout: self._write_client_files(client_files, host) 2022-05-14T00:35:45.813 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T00:35:45.813 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T00:35:45.814 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T00:35:45.814 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T00:35:45.814 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T00:35:45.814 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.event_loop.get_result(coro) 2022-05-14T00:35:45.815 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T00:35:45.815 INFO:journalctl@ceph.mon.b.smithi047.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T00:35:45.815 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T00:35:45.815 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.__get_result() 2022-05-14T00:35:45.816 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T00:35:45.816 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise self._exception 2022-05-14T00:35:45.816 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T00:35:45.816 INFO:journalctl@ceph.mon.b.smithi047.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T00:35:45.816 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T00:35:45.817 INFO:journalctl@ceph.mon.b.smithi047.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T00:35:45.817 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T00:35:45.817 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise 2022-05-14T00:35:45.818 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T00:35:45.818 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.gen.throw(type, value, traceback) 2022-05-14T00:35:45.818 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T00:35:45.818 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OrchestratorError(msg) 2022-05-14T00:35:45.819 INFO:journalctl@ceph.mon.b.smithi047.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:35:46.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:35:46 smithi115 conmon[90393]: 2022-05-14T00:35:46.424+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:35:46.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:35:46 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:35:46.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:35:46 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:35:47.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:35:47 smithi115 conmon[90393]: 2022-05-14T00:35:47.427+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:35:47.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:35:47 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:35:47.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:35:47 smithi115 ceph-mon[78593]: pgmap v2393: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:35:47.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:35:47 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:35:47.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:35:47 smithi047 ceph-mon[77147]: pgmap v2393: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:35:48.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:35:48 smithi115 conmon[90393]: 2022-05-14T00:35:48.406+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:35:48.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:35:48 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:35:48.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:35:48 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:35:49.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:35:49 smithi115 conmon[90393]: 2022-05-14T00:35:49.455+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:35:49.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:35:49 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:35:49.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:35:49 smithi115 ceph-mon[78593]: pgmap v2394: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:35:49.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:35:49 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 4080 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:35:49.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:35:49 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:35:49.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:35:49 smithi047 ceph-mon[77147]: pgmap v2394: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:35:49.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:35:49 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 4080 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:35:50.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:35:50 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:35:50.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:35:50 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:35:50.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:35:50 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:35:50.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:35:50 smithi115 conmon[90393]: 2022-05-14T00:35:50.421+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:35:50.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:35:50 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:35:50.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:35:50 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:35:50.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:35:50 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:35:51.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:35:51 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:35:51.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:35:51 smithi047 ceph-mon[77147]: pgmap v2395: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:35:51.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:35:51 smithi115 conmon[90393]: 2022-05-14T00:35:51.393+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:35:51.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:35:51 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:35:51.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:35:51 smithi115 ceph-mon[78593]: pgmap v2395: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:35:52.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:35:52 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:35:52.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:35:52 smithi115 conmon[90393]: 2022-05-14T00:35:52.439+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:35:52.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:35:52 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:35:53.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:35:53 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:35:53.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:35:53 smithi047 ceph-mon[77147]: pgmap v2396: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:35:53.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:35:53 smithi115 conmon[90393]: 2022-05-14T00:35:53.424+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:35:53.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:35:53 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:35:53.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:35:53 smithi115 ceph-mon[78593]: pgmap v2396: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:35:54.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:35:54 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:35:54.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:35:54 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:35:54.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:35:54 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:35:54.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:35:54 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 4085 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:35:54.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:35:54 smithi115 conmon[90393]: 2022-05-14T00:35:54.428+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:35:54.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:35:54 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:35:54.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:35:54 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:35:54.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:35:54 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:35:54.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:35:54 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 4085 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:35:55.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:35:55 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:35:55.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:35:55 smithi047 ceph-mon[77147]: pgmap v2397: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:35:55.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:35:55 smithi115 conmon[90393]: 2022-05-14T00:35:55.390+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:35:55.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:35:55 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:35:55.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:35:55 smithi115 ceph-mon[78593]: pgmap v2397: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:35:56.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:35:56 smithi115 conmon[90393]: 2022-05-14T00:35:56.347+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:35:56.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:35:56 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:35:56.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:35:56 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:35:57.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:35:57 smithi115 conmon[90393]: 2022-05-14T00:35:57.343+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:35:57.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:35:57 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:35:57.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:35:57 smithi115 ceph-mon[78593]: pgmap v2398: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:35:57.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:35:57 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:35:57.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:35:57 smithi047 ceph-mon[77147]: pgmap v2398: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:35:58.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:35:58 smithi115 conmon[90393]: 2022-05-14T00:35:58.336+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:35:58.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:35:58 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:35:58.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:35:58 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:35:58.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:35:58 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:35:58.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:35:58 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:35:58.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:35:58 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:35:58.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:35:58 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:35:59.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:35:59 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:35:59.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:35:59 smithi047 ceph-mon[77147]: pgmap v2399: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:35:59.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:35:59 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 4090 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:35:59.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:35:59 smithi115 conmon[90393]: 2022-05-14T00:35:59.381+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:35:59.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:35:59 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:35:59.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:35:59 smithi115 ceph-mon[78593]: pgmap v2399: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:35:59.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:35:59 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 4090 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:36:00.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:36:00 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:36:00.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:36:00 smithi115 conmon[90393]: 2022-05-14T00:36:00.431+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:36:00.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:36:00 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:36:01.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:36:01 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:36:02.219 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:36:01 smithi047 ceph-mon[77147]: pgmap v2400: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:36:02.220 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:36:01 smithi115 conmon[90393]: 2022-05-14T00:36:01.385+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:36:02.221 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:36:01 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:36:02.221 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:36:01 smithi115 ceph-mon[78593]: pgmap v2400: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:36:02.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:36:02 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:36:02.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:36:02 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:36:02.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:36:02 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:36:02.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:36:02 smithi115 conmon[90393]: 2022-05-14T00:36:02.416+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:36:02.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:36:02 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:36:02.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:36:02 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:36:02.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:36:02 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:36:03.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:36:03 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:36:03.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:36:03 smithi047 ceph-mon[77147]: pgmap v2401: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:36:03.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:36:03 smithi115 conmon[90393]: 2022-05-14T00:36:03.391+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:36:03.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:36:03 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:36:03.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:36:03 smithi115 ceph-mon[78593]: pgmap v2401: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:36:04.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:36:04 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:36:04.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:36:04 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 4095 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:36:04.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:36:04 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:36:04.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:36:04 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 4095 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:36:04.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:36:04 smithi115 conmon[90393]: 2022-05-14T00:36:04.380+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:36:05.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:36:05 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:36:05.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:36:05 smithi047 ceph-mon[77147]: pgmap v2402: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:36:05.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:36:05 smithi115 conmon[90393]: 2022-05-14T00:36:05.402+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:36:05.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:36:05 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:36:05.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:36:05 smithi115 ceph-mon[78593]: pgmap v2402: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:36:06.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:36:06 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:36:06.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:36:06 smithi115 conmon[90393]: 2022-05-14T00:36:06.422+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:36:06.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:36:06 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:36:07.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:36:07 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:36:07.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:36:07 smithi047 ceph-mon[77147]: pgmap v2403: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:36:07.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:36:07 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:36:07.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:36:07 smithi115 ceph-mon[78593]: pgmap v2403: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:36:07.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:36:07 smithi115 conmon[90393]: 2022-05-14T00:36:07.469+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:36:08.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:36:08 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:36:08.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:36:08 smithi115 conmon[90393]: 2022-05-14T00:36:08.513+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:36:08.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:36:08 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:36:09.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:36:09 smithi047 ceph-mon[77147]: pgmap v2404: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:36:09.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:36:09 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:36:09.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:36:09 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 4100 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:36:09.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:36:09 smithi115 conmon[90393]: 2022-05-14T00:36:09.548+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:36:09.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:36:09 smithi115 ceph-mon[78593]: pgmap v2404: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:36:09.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:36:09 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:36:09.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:36:09 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 4100 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:36:10.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:36:10 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:36:10.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:36:10 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:36:10.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:36:10 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:36:10.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:36:10 smithi115 conmon[90393]: 2022-05-14T00:36:10.571+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:36:10.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:36:10 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:36:10.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:36:10 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:36:10.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:36:10 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:36:11.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:36:11 smithi047 ceph-mon[77147]: pgmap v2405: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:36:11.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:36:11 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:36:11.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:36:11 smithi115 conmon[90393]: 2022-05-14T00:36:11.557+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:36:11.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:36:11 smithi115 ceph-mon[78593]: pgmap v2405: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:36:11.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:36:11 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:36:12.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:36:12 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:36:12.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:36:12 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:36:12.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:36:12 smithi115 conmon[90393]: 2022-05-14T00:36:12.572+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:36:13.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:36:13 smithi047 ceph-mon[77147]: pgmap v2406: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:36:13.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:36:13 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:36:13.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:36:13 smithi115 ceph-mon[78593]: pgmap v2406: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:36:13.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:36:13 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:36:13.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:36:13 smithi115 conmon[90393]: 2022-05-14T00:36:13.535+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:36:14.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:36:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T00:36:14.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:36:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T00:36:14.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:36:14 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:36:14.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:36:14 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:36:14.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:36:14 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:36:14.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:36:14 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 4105 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:36:14.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:36:14 smithi115 conmon[90393]: 2022-05-14T00:36:14.562+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:36:14.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:36:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T00:36:14.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:36:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T00:36:14.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:36:14 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:36:14.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:36:14 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:36:14.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:36:14 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:36:14.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:36:14 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 4105 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:36:15.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:36:15 smithi047 ceph-mon[77147]: pgmap v2407: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:36:15.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:36:15 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:36:15.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:36:15 smithi115 ceph-mon[78593]: pgmap v2407: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:36:15.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:36:15 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:36:15.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:36:15 smithi115 conmon[90393]: 2022-05-14T00:36:15.611+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:36:16.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:36:16 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:36:16.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:36:16 smithi115 conmon[90393]: 2022-05-14T00:36:16.594+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:36:16.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:36:16 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:36:17.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:36:17 smithi047 ceph-mon[77147]: pgmap v2408: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:36:17.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:36:17 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:36:17.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:36:17 smithi115 conmon[90393]: 2022-05-14T00:36:17.622+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:36:17.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:36:17 smithi115 ceph-mon[78593]: pgmap v2408: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:36:17.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:36:17 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:36:18.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:36:18 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:36:18.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:36:18 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:36:18.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:36:18 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:36:18.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:36:18 smithi115 conmon[90393]: 2022-05-14T00:36:18.579+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:36:18.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:36:18 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:36:18.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:36:18 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:36:18.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:36:18 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:36:19.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:36:19 smithi047 ceph-mon[77147]: pgmap v2409: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:36:19.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:36:19 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:36:19.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:36:19 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 4110 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:36:19.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:36:19 smithi115 conmon[90393]: 2022-05-14T00:36:19.565+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:36:19.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:36:19 smithi115 ceph-mon[78593]: pgmap v2409: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:36:19.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:36:19 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:36:19.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:36:19 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 4110 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:36:20.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:36:20 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:36:20.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:36:20 smithi115 conmon[90393]: 2022-05-14T00:36:20.558+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:36:20.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:36:20 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:36:21.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:36:21 smithi047 ceph-mon[77147]: pgmap v2410: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:36:21.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:36:21 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:36:21.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:36:21 smithi115 conmon[90393]: 2022-05-14T00:36:21.554+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:36:21.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:36:21 smithi115 ceph-mon[78593]: pgmap v2410: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:36:21.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:36:21 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:36:22.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:36:22 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:36:22.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:36:22 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:36:22.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:36:22 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:36:22.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:36:22 smithi115 conmon[90393]: 2022-05-14T00:36:22.519+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:36:22.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:36:22 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:36:22.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:36:22 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:36:22.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:36:22 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:36:23.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:36:23 smithi047 ceph-mon[77147]: pgmap v2411: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:36:23.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:36:23 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:36:23.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:36:23 smithi115 conmon[90393]: 2022-05-14T00:36:23.507+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:36:23.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:36:23 smithi115 ceph-mon[78593]: pgmap v2411: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:36:23.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:36:23 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:36:24.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:36:24 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:36:24.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:36:24 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 4115 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:36:24.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:36:24 smithi115 conmon[90393]: 2022-05-14T00:36:24.481+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:36:24.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:36:24 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:36:24.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:36:24 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 4115 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:36:25.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:36:25 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:36:25.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:36:25 smithi047 ceph-mon[77147]: pgmap v2412: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:36:25.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:36:25 smithi115 conmon[90393]: 2022-05-14T00:36:25.475+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:36:25.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:36:25 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:36:25.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:36:25 smithi115 ceph-mon[78593]: pgmap v2412: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:36:26.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:36:26 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:36:26.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:36:26 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:36:26.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:36:26 smithi115 conmon[90393]: 2022-05-14T00:36:26.437+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:36:27.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:36:27 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:36:27.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:36:27 smithi047 ceph-mon[77147]: pgmap v2413: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:36:27.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:36:27 smithi115 conmon[90393]: 2022-05-14T00:36:27.425+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:36:27.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:36:27 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:36:27.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:36:27 smithi115 ceph-mon[78593]: pgmap v2413: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:36:28.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:36:28 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:36:28.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:36:28 smithi115 conmon[90393]: 2022-05-14T00:36:28.440+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:36:28.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:36:28 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:36:29.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:36:29 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:36:29.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:36:29 smithi047 ceph-mon[77147]: pgmap v2414: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:36:29.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:36:29 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 4120 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:36:29.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:36:29 smithi115 conmon[90393]: 2022-05-14T00:36:29.439+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:36:29.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:36:29 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:36:29.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:36:29 smithi115 ceph-mon[78593]: pgmap v2414: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:36:29.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:36:29 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 4120 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:36:30.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:36:30 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:36:30.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:36:30 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:36:30.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:36:30 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:36:30.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:36:30 smithi115 conmon[90393]: 2022-05-14T00:36:30.434+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:36:30.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:36:30 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:36:30.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:36:30 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:36:30.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:36:30 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:36:31.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:36:31 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:36:31.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:36:31 smithi047 ceph-mon[77147]: pgmap v2415: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:36:31.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:36:31 smithi115 conmon[90393]: 2022-05-14T00:36:31.419+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:36:31.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:36:31 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:36:31.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:36:31 smithi115 ceph-mon[78593]: pgmap v2415: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:36:32.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:36:32 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:36:32.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:36:32 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:36:32.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:36:32 smithi115 conmon[90393]: 2022-05-14T00:36:32.415+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:36:33.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:36:33 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:36:33.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:36:33 smithi047 ceph-mon[77147]: pgmap v2416: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:36:33.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:36:33 smithi115 conmon[90393]: 2022-05-14T00:36:33.397+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:36:33.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:36:33 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:36:33.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:36:33 smithi115 ceph-mon[78593]: pgmap v2416: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:36:34.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:36:34 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:36:34.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:36:34 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:36:34.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:36:34 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:36:34.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:36:34 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 4125 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:36:34.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:36:34 smithi115 conmon[90393]: 2022-05-14T00:36:34.439+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:36:34.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:36:34 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:36:34.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:36:34 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:36:34.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:36:34 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:36:34.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:36:34 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 4125 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:36:35.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:36:35 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:36:35.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:36:35 smithi047 ceph-mon[77147]: pgmap v2417: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:36:35.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:36:35 smithi115 conmon[90393]: 2022-05-14T00:36:35.451+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:36:35.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:36:35 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:36:35.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:36:35 smithi115 ceph-mon[78593]: pgmap v2417: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:36:36.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:36:36 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:36:36.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:36:36 smithi115 conmon[90393]: 2022-05-14T00:36:36.500+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:36:36.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:36:36 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:36:37.797 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:36:37 smithi047 ceph-mon[77147]: pgmap v2418: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:36:37.798 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:36:37 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:36:37.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:36:37 smithi115 conmon[90393]: 2022-05-14T00:36:37.506+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:36:37.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:36:37 smithi115 ceph-mon[78593]: pgmap v2418: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:36:37.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:36:37 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:36:38.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:36:38 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:36:38.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:36:38 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:36:38.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:36:38 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:36:38.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:36:38 smithi115 conmon[90393]: 2022-05-14T00:36:38.505+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:36:38.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:36:38 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:36:38.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:36:38 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:36:38.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:36:38 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:36:39.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:36:39 smithi047 ceph-mon[77147]: pgmap v2419: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:36:39.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:36:39 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:36:39.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:36:39 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 4130 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:36:39.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:36:39 smithi115 conmon[90393]: 2022-05-14T00:36:39.533+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:36:39.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:36:39 smithi115 ceph-mon[78593]: pgmap v2419: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:36:39.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:36:39 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:36:39.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:36:39 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 4130 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:36:40.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:36:40 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:36:40.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:36:40 smithi115 conmon[90393]: 2022-05-14T00:36:40.519+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:36:40.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:36:40 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:36:41.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:36:41 smithi047 ceph-mon[77147]: pgmap v2420: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:36:41.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:36:41 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:36:41.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:36:41 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:36:41.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:36:41 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:36:41.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:36:41 smithi115 conmon[90393]: 2022-05-14T00:36:41.549+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:36:41.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:36:41 smithi115 ceph-mon[78593]: pgmap v2420: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:36:41.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:36:41 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:36:41.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:36:41 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:36:41.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:36:41 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:36:42.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:36:42 smithi115 conmon[90393]: 2022-05-14T00:36:42.555+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:36:42.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:36:42 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:36:43.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:36:42 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:36:43.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:36:43 smithi115 conmon[90393]: 2022-05-14T00:36:43.541+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:36:43.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:36:43 smithi115 ceph-mon[78593]: pgmap v2421: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:36:43.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:36:43 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:36:44.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:36:43 smithi047 ceph-mon[77147]: pgmap v2421: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:36:44.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:36:43 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:36:44.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:36:44 smithi115 conmon[90393]: 2022-05-14T00:36:44.522+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:36:44.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:36:44 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:36:44.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:36:44 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 4135 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:36:45.000 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:36:44 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:36:45.001 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:36:44 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 4135 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:36:45.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:36:45 smithi047 ceph-mon[77147]: pgmap v2422: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:36:45.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:36:45 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:36:45.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:36:45 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T00:36:45.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:36:45 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T00:36:45.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:36:45 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T00:36:45.792 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:36:45 smithi047 conmon[78521]: 2022-05-14T00:36:45.424+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:36:45.792 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:36:45 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T00:36:45.792 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:36:45 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:36:45.793 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:36:45 smithi047 conmon[78521]: yield 2022-05-14T00:36:45.793 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:36:45 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:36:45.793 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:36:45 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:36:45.793 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:36:45 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:36:45.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:36:45 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T00:36:45.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:36:45 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:36:45.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:36:45 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T00:36:45.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:36:45 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:36:45.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:36:45 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T00:36:45.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:36:45 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:36:45.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:36:45 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T00:36:45.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:36:45 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:36:45.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:36:45 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T00:36:45.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:36:45 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:36:45.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:36:45 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:36:45.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:36:45 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:36:45.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:36:45 smithi115 conmon[90393]: 2022-05-14T00:36:45.485+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:36:45.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:36:45 smithi115 ceph-mon[78593]: pgmap v2422: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:36:45.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:36:45 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:36:45.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:36:45 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T00:36:45.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:36:45 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T00:36:45.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:36:45 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T00:36:46.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:36:46 smithi115 conmon[90393]: 2022-05-14T00:36:46.529+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:36:46.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:36:46 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:36:46.858 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T00:36:46.858 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:36:46.858 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T00:36:46.859 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:36:46.859 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:36:46.859 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:36:46.859 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T00:36:46.859 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:36:46.860 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T00:36:46.860 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:36:46.860 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T00:36:46.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:36:46.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T00:36:46.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:36:46.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T00:36:46.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:36:46.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:36:46.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:36:46.862 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:36:46 smithi115 ceph-mon[78593]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T00:36:46.863 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:36:46 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:36:47.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:36:46 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:36:47.042 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T00:36:47.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:36:47.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T00:36:47.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:36:47.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:36:47.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:36:47.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T00:36:47.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:36:47.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T00:36:47.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:36:47.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T00:36:47.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:36:47.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T00:36:47.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:36:47.049 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T00:36:47.049 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:36:47.050 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:36:47.050 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:36:47.051 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:36:46 smithi047 ceph-mon[77147]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T00:36:47.051 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:36:46 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:36:47.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:36:47 smithi115 conmon[90393]: 2022-05-14T00:36:47.504+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:36:47.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:36:47 smithi115 ceph-mon[78593]: pgmap v2423: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:36:47.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:36:47 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:36:48.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:36:47 smithi047 ceph-mon[77147]: pgmap v2423: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:36:48.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:36:47 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:36:48.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:36:48 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:36:48.791 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:36:48 smithi047 conmon[78521]: 2022-05-14T00:36:48.496+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:36:48.792 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:36:48 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T00:36:48.792 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:36:48 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:36:48.792 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:36:48 smithi047 conmon[78521]: yield 2022-05-14T00:36:48.792 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:36:48 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:36:48.792 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:36:48 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:36:48.793 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:36:48 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:36:48.793 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:36:48 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T00:36:48.793 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:36:48 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:36:48.793 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:36:48 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T00:36:48.793 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:36:48 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:36:48.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:36:48 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T00:36:48.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:36:48 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:36:48.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:36:48 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T00:36:48.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:36:48 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:36:48.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:36:48 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T00:36:48.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:36:48 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:36:48.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:36:48 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:36:48.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:36:48 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:36:48.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:36:48 smithi047 conmon[78521]: 2022-05-14T00:36:48.497+0000 7f5340f7a700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T00:36:48.797 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:36:48 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T00:36:48.797 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:36:48 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:36:48.797 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:36:48 smithi047 conmon[78521]: yield 2022-05-14T00:36:48.798 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:36:48 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:36:48.799 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:36:48 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:36:48.799 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:36:48 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:36:48.800 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:36:48 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T00:36:48.800 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:36:48 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:36:48.800 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:36:48 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T00:36:48.800 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:36:48 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:36:48.801 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:36:48 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T00:36:48.801 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:36:48 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:36:48.801 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:36:48 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T00:36:48.801 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:36:48 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:36:48.802 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:36:48 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T00:36:48.802 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:36:48 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:36:48.802 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:36:48 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:36:48.802 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:36:48 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:36:48.802 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:36:48 smithi047 conmon[78521]: 2022-05-14T00:36:48.803 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:36:48 smithi047 conmon[78521]: During handling of the above exception, another exception occurred: 2022-05-14T00:36:48.803 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:36:48 smithi047 conmon[78521]: 2022-05-14T00:36:48.803 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:36:48 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T00:36:48.803 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:36:48 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T00:36:48.804 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:36:48 smithi047 conmon[78521]: return f(*arg) 2022-05-14T00:36:48.804 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:36:48 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T00:36:48.804 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:36:48 smithi047 conmon[78521]: self._write_client_files(client_files, host) 2022-05-14T00:36:48.804 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:36:48 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T00:36:48.804 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:36:48 smithi047 conmon[78521]: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T00:36:48.805 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:36:48 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T00:36:48.805 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:36:48 smithi047 conmon[78521]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T00:36:48.806 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:36:48 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T00:36:48.806 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:36:48 smithi047 conmon[78521]: return self.event_loop.get_result(coro) 2022-05-14T00:36:48.807 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:36:48 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T00:36:48.807 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:36:48 smithi047 conmon[78521]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T00:36:48.807 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:36:48 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T00:36:48.807 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:36:48 smithi047 conmon[78521]: return self.__get_result() 2022-05-14T00:36:48.807 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:36:48 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T00:36:48.808 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:36:48 smithi047 conmon[78521]: raise self._exception 2022-05-14T00:36:48.808 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:36:48 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T00:36:48.808 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:36:48 smithi047 conmon[78521]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T00:36:48.808 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:36:48 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T00:36:48.808 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:36:48 smithi047 conmon[78521]: conn = await self._remote_connection(host, addr) 2022-05-14T00:36:48.809 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:36:48 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T00:36:48.809 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:36:48 smithi047 conmon[78521]: raise 2022-05-14T00:36:48.810 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:36:48 smithi047 conmon[78521]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T00:36:48.810 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:36:48 smithi047 conmon[78521]: self.gen.throw(type, value, traceback) 2022-05-14T00:36:48.810 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:36:48 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T00:36:48.810 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:36:48 smithi047 conmon[78521]: raise OrchestratorError(msg) 2022-05-14T00:36:48.811 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:36:48 smithi047 conmon[78521]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:36:48.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:36:48 smithi115 conmon[90393]: 2022-05-14T00:36:48.534+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:36:48.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:36:48 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:36:49.862 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:36:49 smithi115 conmon[90393]: 2022-05-14T00:36:49.578+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:36:49.862 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:36:49 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:36:49.863 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T00:36:49.863 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:36:49.863 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T00:36:49.863 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:36:49.864 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:36:49.864 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:36:49.864 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T00:36:49.864 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:36:49.864 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T00:36:49.865 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:36:49.865 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T00:36:49.865 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:36:49.865 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T00:36:49.866 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:36:49.866 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T00:36:49.866 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:36:49.866 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:36:49.867 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:36:49.867 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:36:49 smithi115 ceph-mon[78593]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T00:36:49.867 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T00:36:49.867 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:36:49.868 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T00:36:49.868 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:36:49.868 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:36:49.868 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:36:49.868 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T00:36:49.869 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:36:49.869 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T00:36:49.869 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:36:49.869 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T00:36:49.870 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:36:49.870 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T00:36:49.870 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:36:49.870 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T00:36:49.871 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:36:49.871 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:36:49.871 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:36:49.871 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T00:36:49.871 INFO:journalctl@ceph.mon.c.smithi115.stdout: During handling of the above exception, another exception occurred: 2022-05-14T00:36:49.872 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T00:36:49.872 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T00:36:49.873 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T00:36:49.873 INFO:journalctl@ceph.mon.c.smithi115.stdout: return f(*arg) 2022-05-14T00:36:49.873 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T00:36:49.873 INFO:journalctl@ceph.mon.c.smithi115.stdout: self._write_client_files(client_files, host) 2022-05-14T00:36:49.874 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T00:36:49.874 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T00:36:49.874 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T00:36:49.874 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T00:36:49.875 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T00:36:49.875 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.event_loop.get_result(coro) 2022-05-14T00:36:49.885 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T00:36:49.885 INFO:journalctl@ceph.mon.c.smithi115.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T00:36:49.886 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T00:36:49.886 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.__get_result() 2022-05-14T00:36:49.886 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T00:36:49.886 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise self._exception 2022-05-14T00:36:49.886 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T00:36:49.887 INFO:journalctl@ceph.mon.c.smithi115.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T00:36:49.887 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T00:36:49.887 INFO:journalctl@ceph.mon.c.smithi115.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T00:36:49.887 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T00:36:49.888 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise 2022-05-14T00:36:49.888 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T00:36:49.888 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.gen.throw(type, value, traceback) 2022-05-14T00:36:49.888 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T00:36:49.888 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OrchestratorError(msg) 2022-05-14T00:36:49.889 INFO:journalctl@ceph.mon.c.smithi115.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:36:49.889 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:36:49 smithi115 ceph-mon[78593]: pgmap v2424: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:36:49.889 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:36:49 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:36:49.891 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:36:49 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:36:49.892 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:36:49 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:36:49.892 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:36:49 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 4140 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:36:50.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:36:49 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:36:50.041 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T00:36:50.041 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:36:50.042 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T00:36:50.042 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:36:50.042 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:36:50.042 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:36:50.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T00:36:50.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:36:50.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T00:36:50.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:36:50.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T00:36:50.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:36:50.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T00:36:50.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:36:50.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T00:36:50.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:36:50.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:36:50.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:36:50.046 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:36:49 smithi047 ceph-mon[77147]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T00:36:50.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T00:36:50.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:36:50.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T00:36:50.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:36:50.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:36:50.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:36:50.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T00:36:50.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:36:50.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T00:36:50.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:36:50.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T00:36:50.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:36:50.049 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T00:36:50.049 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:36:50.049 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T00:36:50.049 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:36:50.049 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:36:50.050 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:36:50.050 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T00:36:50.050 INFO:journalctl@ceph.mon.b.smithi047.stdout: During handling of the above exception, another exception occurred: 2022-05-14T00:36:50.050 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T00:36:50.051 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T00:36:50.051 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T00:36:50.051 INFO:journalctl@ceph.mon.b.smithi047.stdout: return f(*arg) 2022-05-14T00:36:50.051 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T00:36:50.051 INFO:journalctl@ceph.mon.b.smithi047.stdout: self._write_client_files(client_files, host) 2022-05-14T00:36:50.052 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T00:36:50.052 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T00:36:50.052 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T00:36:50.052 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T00:36:50.053 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T00:36:50.053 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.event_loop.get_result(coro) 2022-05-14T00:36:50.053 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T00:36:50.053 INFO:journalctl@ceph.mon.b.smithi047.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T00:36:50.053 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T00:36:50.054 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.__get_result() 2022-05-14T00:36:50.054 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T00:36:50.054 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise self._exception 2022-05-14T00:36:50.054 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T00:36:50.054 INFO:journalctl@ceph.mon.b.smithi047.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T00:36:50.055 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T00:36:50.055 INFO:journalctl@ceph.mon.b.smithi047.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T00:36:50.055 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T00:36:50.056 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise 2022-05-14T00:36:50.056 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T00:36:50.056 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.gen.throw(type, value, traceback) 2022-05-14T00:36:50.056 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T00:36:50.057 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OrchestratorError(msg) 2022-05-14T00:36:50.057 INFO:journalctl@ceph.mon.b.smithi047.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:36:50.057 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:36:49 smithi047 ceph-mon[77147]: pgmap v2424: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:36:50.057 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:36:49 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:36:50.058 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:36:49 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:36:50.058 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:36:49 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:36:50.058 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:36:49 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 4140 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:36:50.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:36:50 smithi115 conmon[90393]: 2022-05-14T00:36:50.598+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:36:50.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:36:50 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:36:51.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:36:50 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:36:51.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:36:51 smithi115 conmon[90393]: 2022-05-14T00:36:51.616+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:36:51.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:36:51 smithi115 ceph-mon[78593]: pgmap v2425: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:36:51.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:36:51 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:36:52.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:36:51 smithi047 ceph-mon[77147]: pgmap v2425: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:36:52.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:36:51 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:36:52.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:36:52 smithi115 conmon[90393]: 2022-05-14T00:36:52.606+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:36:52.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:36:52 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:36:53.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:36:52 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:36:53.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:36:53 smithi115 conmon[90393]: 2022-05-14T00:36:53.627+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:36:53.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:36:53 smithi115 ceph-mon[78593]: pgmap v2426: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:36:53.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:36:53 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:36:53.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:36:53 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:36:53.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:36:53 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:36:54.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:36:53 smithi047 ceph-mon[77147]: pgmap v2426: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:36:54.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:36:53 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:36:54.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:36:53 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:36:54.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:36:53 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:36:54.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:36:54 smithi115 conmon[90393]: 2022-05-14T00:36:54.653+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:36:54.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:36:54 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:36:54.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:36:54 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 4145 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:36:55.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:36:54 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:36:55.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:36:54 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 4145 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:36:55.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:36:55 smithi115 conmon[90393]: 2022-05-14T00:36:55.617+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:36:55.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:36:55 smithi115 ceph-mon[78593]: pgmap v2427: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:36:55.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:36:55 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:36:56.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:36:55 smithi047 ceph-mon[77147]: pgmap v2427: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:36:56.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:36:55 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:36:56.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:36:56 smithi115 conmon[90393]: 2022-05-14T00:36:56.569+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:36:56.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:36:56 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:36:57.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:36:56 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:36:57.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:36:57 smithi115 conmon[90393]: 2022-05-14T00:36:57.581+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:36:57.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:36:57 smithi115 ceph-mon[78593]: pgmap v2428: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:36:57.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:36:57 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:36:57.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:36:57 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:36:57.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:36:57 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:36:58.047 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:36:57 smithi047 ceph-mon[77147]: pgmap v2428: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:36:58.047 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:36:57 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:36:58.047 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:36:57 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:36:58.047 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:36:57 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:36:58.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:36:58 smithi115 conmon[90393]: 2022-05-14T00:36:58.609+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:36:58.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:36:58 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:36:59.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:36:58 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:36:59.861 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:36:59 smithi115 conmon[90393]: 2022-05-14T00:36:59.586+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:36:59.862 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:36:59 smithi115 ceph-mon[78593]: pgmap v2429: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:36:59.862 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:36:59 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:36:59.862 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:36:59 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 4150 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:37:00.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:36:59 smithi047 ceph-mon[77147]: pgmap v2429: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:37:00.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:36:59 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:37:00.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:36:59 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 4150 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:37:00.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:37:00 smithi115 conmon[90393]: 2022-05-14T00:37:00.606+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:37:00.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:37:00 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:37:01.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:37:00 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:37:01.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:37:01 smithi115 conmon[90393]: 2022-05-14T00:37:01.569+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:37:01.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:37:01 smithi115 ceph-mon[78593]: pgmap v2430: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:37:01.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:37:01 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:37:01.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:37:01 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:37:01.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:37:01 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:37:02.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:37:01 smithi047 ceph-mon[77147]: pgmap v2430: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:37:02.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:37:01 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:37:02.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:37:01 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:37:02.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:37:01 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:37:02.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:37:02 smithi115 conmon[90393]: 2022-05-14T00:37:02.593+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:37:02.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:37:02 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:37:03.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:37:02 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:37:03.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:37:03 smithi115 conmon[90393]: 2022-05-14T00:37:03.565+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:37:03.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:37:03 smithi115 ceph-mon[78593]: pgmap v2431: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:37:03.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:37:03 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:37:04.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:37:03 smithi047 ceph-mon[77147]: pgmap v2431: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:37:04.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:37:03 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:37:04.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:37:04 smithi115 conmon[90393]: 2022-05-14T00:37:04.577+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:37:04.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:37:04 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:37:04.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:37:04 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 4155 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:37:05.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:37:04 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:37:05.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:37:04 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 4155 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:37:05.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:37:05 smithi115 conmon[90393]: 2022-05-14T00:37:05.563+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:37:05.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:37:05 smithi115 ceph-mon[78593]: pgmap v2432: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:37:05.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:37:05 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:37:06.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:37:05 smithi047 ceph-mon[77147]: pgmap v2432: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:37:06.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:37:05 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:37:07.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:37:06 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:37:07.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:37:06 smithi115 conmon[90393]: 2022-05-14T00:37:06.613+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:37:07.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:37:06 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:37:08.046 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:37:07 smithi047 ceph-mon[77147]: pgmap v2433: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:37:08.046 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:37:07 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:37:08.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:37:07 smithi115 conmon[90393]: 2022-05-14T00:37:07.611+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:37:08.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:37:07 smithi115 ceph-mon[78593]: pgmap v2433: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:37:08.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:37:07 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:37:09.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:37:08 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:37:09.111 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:37:08 smithi115 conmon[90393]: 2022-05-14T00:37:08.630+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:37:09.112 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:37:08 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:37:10.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:37:09 smithi047 ceph-mon[77147]: pgmap v2434: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:37:10.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:37:09 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:37:10.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:37:09 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:37:10.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:37:09 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:37:10.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:37:09 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 4160 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:37:10.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:37:09 smithi115 conmon[90393]: 2022-05-14T00:37:09.672+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:37:10.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:37:09 smithi115 ceph-mon[78593]: pgmap v2434: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:37:10.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:37:09 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:37:10.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:37:09 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:37:10.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:37:09 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:37:10.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:37:09 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 4160 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:37:11.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:37:10 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:37:11.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:37:10 smithi115 conmon[90393]: 2022-05-14T00:37:10.691+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:37:11.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:37:10 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:37:12.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:37:11 smithi047 ceph-mon[77147]: pgmap v2435: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:37:12.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:37:11 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:37:12.119 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:37:11 smithi115 conmon[90393]: 2022-05-14T00:37:11.698+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:37:12.119 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:37:11 smithi115 ceph-mon[78593]: pgmap v2435: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:37:12.120 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:37:11 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:37:13.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:37:12 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:37:13.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:37:12 smithi115 conmon[90393]: 2022-05-14T00:37:12.686+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:37:13.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:37:12 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:37:14.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:37:13 smithi047 ceph-mon[77147]: pgmap v2436: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:37:14.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:37:13 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:37:14.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:37:13 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:37:14.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:37:13 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:37:14.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:37:13 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T00:37:14.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:37:13 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T00:37:14.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:37:13 smithi115 conmon[90393]: 2022-05-14T00:37:13.732+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:37:14.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:37:13 smithi115 ceph-mon[78593]: pgmap v2436: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:37:14.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:37:13 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:37:14.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:37:13 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:37:14.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:37:13 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:37:14.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:37:13 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T00:37:14.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:37:13 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T00:37:15.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:37:14 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:37:15.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:37:14 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 4165 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:37:15.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:37:14 smithi115 conmon[90393]: 2022-05-14T00:37:14.778+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:37:15.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:37:14 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:37:15.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:37:14 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 4165 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:37:16.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:37:15 smithi047 ceph-mon[77147]: pgmap v2437: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:37:16.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:37:15 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:37:16.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:37:15 smithi115 conmon[90393]: 2022-05-14T00:37:15.824+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:37:16.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:37:15 smithi115 ceph-mon[78593]: pgmap v2437: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:37:16.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:37:15 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:37:17.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:37:16 smithi115 conmon[90393]: 2022-05-14T00:37:16.797+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:37:18.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:37:17 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:37:18.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:37:17 smithi047 ceph-mon[77147]: pgmap v2438: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:37:18.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:37:17 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:37:18.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:37:17 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:37:18.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:37:17 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:37:18.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:37:17 smithi115 conmon[90393]: 2022-05-14T00:37:17.810+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:37:18.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:37:17 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:37:18.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:37:17 smithi115 ceph-mon[78593]: pgmap v2438: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:37:18.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:37:17 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:37:18.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:37:17 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:37:18.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:37:17 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:37:19.113 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:37:18 smithi115 conmon[90393]: 2022-05-14T00:37:18.787+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:37:20.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:37:19 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:37:20.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:37:19 smithi047 ceph-mon[77147]: pgmap v2439: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:37:20.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:37:19 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:37:20.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:37:19 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 4170 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:37:20.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:37:19 smithi115 conmon[90393]: 2022-05-14T00:37:19.829+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:37:20.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:37:19 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:37:20.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:37:19 smithi115 ceph-mon[78593]: pgmap v2439: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:37:20.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:37:19 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:37:20.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:37:19 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 4170 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:37:21.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:37:20 smithi115 conmon[90393]: 2022-05-14T00:37:20.786+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:37:22.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:37:21 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:37:22.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:37:21 smithi047 ceph-mon[77147]: pgmap v2440: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:37:22.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:37:21 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:37:22.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:37:21 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:37:22.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:37:21 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:37:22.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:37:21 smithi115 conmon[90393]: 2022-05-14T00:37:21.819+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:37:22.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:37:21 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:37:22.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:37:21 smithi115 ceph-mon[78593]: pgmap v2440: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:37:22.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:37:21 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:37:22.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:37:21 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:37:22.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:37:21 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:37:23.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:37:22 smithi115 conmon[90393]: 2022-05-14T00:37:22.786+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:37:24.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:37:23 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:37:24.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:37:23 smithi047 ceph-mon[77147]: pgmap v2441: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:37:24.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:37:23 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:37:24.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:37:23 smithi115 conmon[90393]: 2022-05-14T00:37:23.812+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:37:24.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:37:23 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:37:24.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:37:23 smithi115 ceph-mon[78593]: pgmap v2441: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:37:24.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:37:23 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:37:25.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:37:24 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 4175 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:37:25.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:37:24 smithi115 conmon[90393]: 2022-05-14T00:37:24.833+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:37:25.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:37:24 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 4175 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:37:26.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:37:25 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:37:26.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:37:25 smithi047 ceph-mon[77147]: pgmap v2442: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:37:26.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:37:25 smithi115 conmon[90393]: 2022-05-14T00:37:25.880+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:37:26.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:37:25 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:37:26.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:37:25 smithi115 ceph-mon[78593]: pgmap v2442: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:37:27.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:37:26 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:37:27.110 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:37:26 smithi115 conmon[90393]: 2022-05-14T00:37:26.856+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:37:27.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:37:26 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:37:28.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:37:27 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:37:28.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:37:27 smithi047 ceph-mon[77147]: pgmap v2443: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:37:28.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:37:27 smithi115 conmon[90393]: 2022-05-14T00:37:27.883+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:37:28.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:37:27 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:37:28.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:37:27 smithi115 ceph-mon[78593]: pgmap v2443: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:37:29.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:37:28 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:37:29.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:37:28 smithi115 conmon[90393]: 2022-05-14T00:37:28.900+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:37:29.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:37:28 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:37:29.948 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:37:29 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:37:29.948 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:37:29 smithi115 ceph-mon[78593]: pgmap v2444: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:37:29.949 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:37:29 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:37:29.949 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:37:29 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:37:29.949 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:37:29 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 4180 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:37:30.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:37:29 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:37:30.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:37:29 smithi047 ceph-mon[77147]: pgmap v2444: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:37:30.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:37:29 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:37:30.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:37:29 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:37:30.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:37:29 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 4180 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:37:30.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:37:29 smithi115 conmon[90393]: 2022-05-14T00:37:29.948+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:37:30.968 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:37:30 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:37:31.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:37:30 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:37:31.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:37:30 smithi115 conmon[90393]: 2022-05-14T00:37:30.967+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:37:31.980 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:37:31 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:37:31.980 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:37:31 smithi115 ceph-mon[78593]: pgmap v2445: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:37:32.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:37:31 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:37:32.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:37:31 smithi047 ceph-mon[77147]: pgmap v2445: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:37:32.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:37:31 smithi115 conmon[90393]: 2022-05-14T00:37:31.979+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:37:32.958 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:37:32 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:37:33.045 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:37:32 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:37:33.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:37:32 smithi115 conmon[90393]: 2022-05-14T00:37:32.958+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:37:34.006 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:37:33 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:37:34.006 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:37:33 smithi115 ceph-mon[78593]: pgmap v2446: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:37:34.006 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:37:33 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:37:34.006 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:37:33 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:37:34.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:37:33 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:37:34.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:37:33 smithi047 ceph-mon[77147]: pgmap v2446: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:37:34.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:37:33 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:37:34.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:37:33 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:37:34.360 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:37:34 smithi115 conmon[90393]: 2022-05-14T00:37:34.004+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:37:34.987 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:37:34 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:37:34.988 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:37:34 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 4185 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:37:35.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:37:34 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:37:35.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:37:34 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 4185 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:37:35.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:37:34 smithi115 conmon[90393]: 2022-05-14T00:37:34.985+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:37:35.971 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:37:35 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:37:35.971 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:37:35 smithi115 ceph-mon[78593]: pgmap v2447: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:37:36.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:37:35 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:37:36.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:37:35 smithi047 ceph-mon[77147]: pgmap v2447: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:37:36.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:37:35 smithi115 conmon[90393]: 2022-05-14T00:37:35.970+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:37:37.007 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:37:36 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:37:37.045 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:37:36 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:37:37.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:37:37 smithi115 conmon[90393]: 2022-05-14T00:37:37.006+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:37:37.960 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:37:37 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:37:37.960 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:37:37 smithi115 ceph-mon[78593]: pgmap v2448: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:37:37.961 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:37:37 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:37:37.961 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:37:37 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:37:38.046 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:37:37 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:37:38.046 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:37:37 smithi047 ceph-mon[77147]: pgmap v2448: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:37:38.046 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:37:37 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:37:38.046 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:37:37 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:37:38.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:37:37 smithi115 conmon[90393]: 2022-05-14T00:37:37.959+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:37:39.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:37:38 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:37:39.117 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:37:38 smithi115 conmon[90393]: 2022-05-14T00:37:38.916+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:37:39.118 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:37:38 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:37:40.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:37:39 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:37:40.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:37:39 smithi047 ceph-mon[77147]: pgmap v2449: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:37:40.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:37:39 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 4190 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:37:40.111 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:37:39 smithi115 conmon[90393]: 2022-05-14T00:37:39.909+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:37:40.112 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:37:39 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:37:40.113 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:37:39 smithi115 ceph-mon[78593]: pgmap v2449: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:37:40.113 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:37:39 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 4190 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:37:41.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:37:40 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:37:41.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:37:40 smithi115 conmon[90393]: 2022-05-14T00:37:40.914+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:37:41.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:37:40 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:37:42.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:37:41 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:37:42.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:37:41 smithi047 ceph-mon[77147]: pgmap v2450: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:37:42.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:37:41 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:37:42.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:37:41 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:37:42.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:37:41 smithi115 conmon[90393]: 2022-05-14T00:37:41.898+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:37:42.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:37:41 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:37:42.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:37:41 smithi115 ceph-mon[78593]: pgmap v2450: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:37:42.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:37:41 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:37:42.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:37:41 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:37:43.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:37:42 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:37:43.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:37:42 smithi115 conmon[90393]: 2022-05-14T00:37:42.886+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:37:43.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:37:42 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:37:44.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:37:43 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:37:44.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:37:43 smithi047 ceph-mon[77147]: pgmap v2451: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:37:44.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:37:43 smithi115 conmon[90393]: 2022-05-14T00:37:43.896+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:37:44.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:37:43 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:37:44.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:37:43 smithi115 ceph-mon[78593]: pgmap v2451: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:37:45.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:37:44 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:37:45.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:37:44 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 4195 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:37:45.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:37:44 smithi115 conmon[90393]: 2022-05-14T00:37:44.916+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:37:45.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:37:44 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:37:45.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:37:44 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 4195 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:37:46.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:37:45 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:37:46.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:37:45 smithi047 ceph-mon[77147]: pgmap v2452: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 7 op/s 2022-05-14T00:37:46.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:37:45 smithi115 conmon[90393]: 2022-05-14T00:37:45.917+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:37:46.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:37:45 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:37:46.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:37:45 smithi115 ceph-mon[78593]: pgmap v2452: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 7 op/s 2022-05-14T00:37:47.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:37:46 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:37:47.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:37:46 smithi115 conmon[90393]: 2022-05-14T00:37:46.877+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:37:47.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:37:46 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:37:48.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:37:47 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:37:48.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:37:47 smithi047 ceph-mon[77147]: pgmap v2453: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 4 op/s 2022-05-14T00:37:48.110 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:37:47 smithi115 conmon[90393]: 2022-05-14T00:37:47.856+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:37:48.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:37:47 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:37:48.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:37:47 smithi115 ceph-mon[78593]: pgmap v2453: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 4 op/s 2022-05-14T00:37:48.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:37:48 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:37:48.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:37:48 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T00:37:48.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:37:48 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T00:37:48.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:37:48 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T00:37:49.018 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:37:48 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:37:49.019 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:37:48 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T00:37:49.019 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:37:48 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T00:37:49.019 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:37:48 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T00:37:49.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:37:48 smithi115 conmon[90393]: 2022-05-14T00:37:48.871+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:37:50.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:37:49 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:37:50.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:37:49 smithi047 ceph-mon[77147]: pgmap v2454: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 4 op/s 2022-05-14T00:37:50.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:37:49 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:37:50.045 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:37:49 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:37:50.045 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:37:49 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 4200 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:37:50.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:37:49 smithi115 conmon[90393]: 2022-05-14T00:37:49.841+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:37:50.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:37:49 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:37:50.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:37:49 smithi115 ceph-mon[78593]: pgmap v2454: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 4 op/s 2022-05-14T00:37:50.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:37:49 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:37:50.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:37:49 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:37:50.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:37:49 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 4200 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:37:51.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:37:50 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:37:51.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:37:50 smithi115 conmon[90393]: 2022-05-14T00:37:50.852+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:37:51.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:37:50 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:37:52.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:37:51 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:37:52.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:37:51 smithi047 ceph-mon[77147]: pgmap v2455: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 7 op/s 2022-05-14T00:37:52.041 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:37:51 smithi047 conmon[78521]: 2022-05-14T00:37:51.599+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:37:52.042 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:37:51 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T00:37:52.042 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:37:51 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:37:52.042 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:37:51 smithi047 conmon[78521]: yield 2022-05-14T00:37:52.042 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:37:51 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:37:52.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:37:51 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:37:52.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:37:51 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:37:52.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:37:51 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T00:37:52.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:37:51 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:37:52.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:37:51 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T00:37:52.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:37:51 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:37:52.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:37:51 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T00:37:52.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:37:51 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:37:52.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:37:51 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T00:37:52.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:37:51 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:37:52.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:37:51 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T00:37:52.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:37:51 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:37:52.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:37:51 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:37:52.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:37:51 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:37:52.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:37:51 smithi115 conmon[90393]: 2022-05-14T00:37:51.888+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:37:52.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:37:51 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:37:52.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:37:51 smithi115 ceph-mon[78593]: pgmap v2455: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 7 op/s 2022-05-14T00:37:53.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:37:52 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:37:53.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:37:52 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:37:53.042 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T00:37:53.042 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:37:53.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T00:37:53.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:37:53.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:37:53.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:37:53.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T00:37:53.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:37:53.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T00:37:53.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:37:53.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T00:37:53.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:37:53.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T00:37:53.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:37:53.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T00:37:53.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:37:53.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:37:53.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:37:53.049 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:37:52 smithi047 ceph-mon[77147]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T00:37:53.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:37:52 smithi115 conmon[90393]: 2022-05-14T00:37:52.898+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:37:53.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:37:52 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:37:53.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:37:52 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:37:53.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T00:37:53.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:37:53.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T00:37:53.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:37:53.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:37:53.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:37:53.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T00:37:53.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:37:53.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T00:37:53.113 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:37:53.117 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T00:37:53.118 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:37:53.118 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T00:37:53.118 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:37:53.119 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T00:37:53.119 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:37:53.119 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:37:53.120 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:37:53.120 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:37:52 smithi115 ceph-mon[78593]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T00:37:54.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:37:53 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:37:54.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:37:53 smithi047 ceph-mon[77147]: pgmap v2456: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 4 op/s 2022-05-14T00:37:54.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:37:53 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:37:54.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:37:53 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:37:54.110 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:37:53 smithi115 conmon[90393]: 2022-05-14T00:37:53.852+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:37:54.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:37:53 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:37:54.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:37:53 smithi115 ceph-mon[78593]: pgmap v2456: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 4 op/s 2022-05-14T00:37:54.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:37:53 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:37:54.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:37:53 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:37:55.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:37:54 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:37:55.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:37:54 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 4205 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:37:55.042 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:37:54 smithi047 conmon[78521]: 2022-05-14T00:37:54.671+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:37:55.042 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:37:54 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T00:37:55.042 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:37:54 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:37:55.042 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:37:54 smithi047 conmon[78521]: yield 2022-05-14T00:37:55.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:37:54 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:37:55.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:37:54 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:37:55.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:37:54 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:37:55.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:37:54 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T00:37:55.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:37:54 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:37:55.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:37:54 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T00:37:55.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:37:54 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:37:55.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:37:54 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T00:37:55.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:37:54 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:37:55.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:37:54 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T00:37:55.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:37:54 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:37:55.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:37:54 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T00:37:55.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:37:54 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:37:55.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:37:54 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:37:55.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:37:54 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:37:55.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:37:54 smithi047 conmon[78521]: 2022-05-14T00:37:54.672+0000 7f5344781700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T00:37:55.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:37:54 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T00:37:55.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:37:54 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:37:55.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:37:54 smithi047 conmon[78521]: yield 2022-05-14T00:37:55.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:37:54 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:37:55.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:37:54 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:37:55.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:37:54 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:37:55.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:37:54 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T00:37:55.048 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:37:54 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:37:55.048 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:37:54 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T00:37:55.048 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:37:54 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:37:55.048 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:37:54 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T00:37:55.048 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:37:54 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:37:55.049 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:37:54 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T00:37:55.049 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:37:54 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:37:55.050 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:37:54 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T00:37:55.050 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:37:54 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:37:55.050 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:37:54 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:37:55.050 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:37:54 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:37:55.051 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:37:54 smithi047 conmon[78521]: 2022-05-14T00:37:55.051 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:37:54 smithi047 conmon[78521]: During handling of the above exception, another exception occurred: 2022-05-14T00:37:55.051 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:37:54 smithi047 conmon[78521]: 2022-05-14T00:37:55.051 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:37:54 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T00:37:55.051 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:37:54 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T00:37:55.052 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:37:54 smithi047 conmon[78521]: return f(*arg) 2022-05-14T00:37:55.052 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:37:54 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T00:37:55.052 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:37:54 smithi047 conmon[78521]: self._write_client_files(client_files, host) 2022-05-14T00:37:55.053 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:37:54 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T00:37:55.053 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:37:54 smithi047 conmon[78521]: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T00:37:55.053 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:37:54 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T00:37:55.053 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:37:54 smithi047 conmon[78521]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T00:37:55.054 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:37:54 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T00:37:55.054 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:37:54 smithi047 conmon[78521]: return self.event_loop.get_result(coro) 2022-05-14T00:37:55.054 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:37:54 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T00:37:55.054 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:37:54 smithi047 conmon[78521]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T00:37:55.054 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:37:54 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T00:37:55.055 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:37:54 smithi047 conmon[78521]: return self.__get_result() 2022-05-14T00:37:55.055 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:37:54 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T00:37:55.055 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:37:54 smithi047 conmon[78521]: raise self._exception 2022-05-14T00:37:55.055 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:37:54 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T00:37:55.055 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:37:54 smithi047 conmon[78521]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T00:37:55.056 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:37:54 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T00:37:55.056 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:37:54 smithi047 conmon[78521]: conn = await self._remote_connection(host, addr) 2022-05-14T00:37:55.056 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:37:54 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T00:37:55.056 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:37:54 smithi047 conmon[78521]: raise 2022-05-14T00:37:55.057 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:37:54 smithi047 conmon[78521]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T00:37:55.057 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:37:54 smithi047 conmon[78521]: self.gen.throw(type, value, traceback) 2022-05-14T00:37:55.057 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:37:54 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T00:37:55.057 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:37:54 smithi047 conmon[78521]: raise OrchestratorError(msg) 2022-05-14T00:37:55.057 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:37:54 smithi047 conmon[78521]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:37:55.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:37:54 smithi115 conmon[90393]: 2022-05-14T00:37:54.884+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:37:55.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:37:54 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:37:55.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:37:54 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 4205 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:37:56.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:37:55 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:37:56.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:37:55 smithi047 ceph-mon[77147]: pgmap v2457: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 7 op/s 2022-05-14T00:37:56.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:37:55 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:37:56.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T00:37:56.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:37:56.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T00:37:56.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:37:56.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:37:56.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:37:56.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T00:37:56.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:37:56.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T00:37:56.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:37:56.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T00:37:56.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:37:56.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T00:37:56.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:37:56.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T00:37:56.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:37:56.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:37:56.049 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:37:56.049 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:37:55 smithi047 ceph-mon[77147]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T00:37:56.049 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T00:37:56.049 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:37:56.050 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T00:37:56.050 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:37:56.050 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:37:56.050 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:37:56.050 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T00:37:56.051 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:37:56.051 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T00:37:56.051 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:37:56.051 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T00:37:56.052 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:37:56.052 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T00:37:56.052 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:37:56.052 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T00:37:56.053 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:37:56.053 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:37:56.053 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:37:56.053 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T00:37:56.053 INFO:journalctl@ceph.mon.b.smithi047.stdout: During handling of the above exception, another exception occurred: 2022-05-14T00:37:56.054 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T00:37:56.054 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T00:37:56.054 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T00:37:56.054 INFO:journalctl@ceph.mon.b.smithi047.stdout: return f(*arg) 2022-05-14T00:37:56.054 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T00:37:56.055 INFO:journalctl@ceph.mon.b.smithi047.stdout: self._write_client_files(client_files, host) 2022-05-14T00:37:56.056 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T00:37:56.056 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T00:37:56.056 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T00:37:56.056 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T00:37:56.057 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T00:37:56.057 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.event_loop.get_result(coro) 2022-05-14T00:37:56.057 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T00:37:56.057 INFO:journalctl@ceph.mon.b.smithi047.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T00:37:56.058 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T00:37:56.058 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.__get_result() 2022-05-14T00:37:56.058 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T00:37:56.058 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise self._exception 2022-05-14T00:37:56.058 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T00:37:56.059 INFO:journalctl@ceph.mon.b.smithi047.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T00:37:56.059 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T00:37:56.059 INFO:journalctl@ceph.mon.b.smithi047.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T00:37:56.059 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T00:37:56.059 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise 2022-05-14T00:37:56.060 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T00:37:56.060 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.gen.throw(type, value, traceback) 2022-05-14T00:37:56.060 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T00:37:56.060 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OrchestratorError(msg) 2022-05-14T00:37:56.061 INFO:journalctl@ceph.mon.b.smithi047.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:37:56.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:37:55 smithi115 conmon[90393]: 2022-05-14T00:37:55.903+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:37:56.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:37:55 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:37:56.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:37:55 smithi115 ceph-mon[78593]: pgmap v2457: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 7 op/s 2022-05-14T00:37:56.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:37:55 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:37:56.109 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T00:37:56.109 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:37:56.109 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T00:37:56.109 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:37:56.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:37:56.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:37:56.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T00:37:56.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:37:56.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T00:37:56.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:37:56.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T00:37:56.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:37:56.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T00:37:56.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:37:56.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T00:37:56.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:37:56.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:37:56.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:37:56.113 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:37:55 smithi115 ceph-mon[78593]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T00:37:56.113 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T00:37:56.113 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:37:56.113 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T00:37:56.113 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:37:56.114 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:37:56.114 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:37:56.114 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T00:37:56.114 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:37:56.115 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T00:37:56.115 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:37:56.115 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T00:37:56.115 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:37:56.115 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T00:37:56.116 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:37:56.116 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T00:37:56.116 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:37:56.116 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:37:56.117 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:37:56.117 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T00:37:56.117 INFO:journalctl@ceph.mon.c.smithi115.stdout: During handling of the above exception, another exception occurred: 2022-05-14T00:37:56.117 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T00:37:56.117 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T00:37:56.118 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T00:37:56.118 INFO:journalctl@ceph.mon.c.smithi115.stdout: return f(*arg) 2022-05-14T00:37:56.118 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T00:37:56.118 INFO:journalctl@ceph.mon.c.smithi115.stdout: self._write_client_files(client_files, host) 2022-05-14T00:37:56.119 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T00:37:56.119 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T00:37:56.119 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T00:37:56.119 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T00:37:56.119 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T00:37:56.120 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.event_loop.get_result(coro) 2022-05-14T00:37:56.120 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T00:37:56.120 INFO:journalctl@ceph.mon.c.smithi115.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T00:37:56.120 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T00:37:56.120 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.__get_result() 2022-05-14T00:37:56.121 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T00:37:56.121 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise self._exception 2022-05-14T00:37:56.121 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T00:37:56.121 INFO:journalctl@ceph.mon.c.smithi115.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T00:37:56.122 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T00:37:56.122 INFO:journalctl@ceph.mon.c.smithi115.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T00:37:56.122 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T00:37:56.122 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise 2022-05-14T00:37:56.122 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T00:37:56.123 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.gen.throw(type, value, traceback) 2022-05-14T00:37:56.123 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T00:37:56.123 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OrchestratorError(msg) 2022-05-14T00:37:56.123 INFO:journalctl@ceph.mon.c.smithi115.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:37:57.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:37:56 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:37:57.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:37:56 smithi115 conmon[90393]: 2022-05-14T00:37:56.899+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:37:57.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:37:56 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:37:58.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:37:57 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:37:58.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:37:57 smithi047 ceph-mon[77147]: pgmap v2458: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:37:58.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:37:57 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:37:58.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:37:57 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:37:58.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:37:57 smithi115 conmon[90393]: 2022-05-14T00:37:57.860+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:37:58.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:37:57 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:37:58.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:37:57 smithi115 ceph-mon[78593]: pgmap v2458: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:37:58.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:37:57 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:37:58.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:37:57 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:37:59.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:37:58 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:37:59.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:37:58 smithi115 conmon[90393]: 2022-05-14T00:37:58.820+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:37:59.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:37:58 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:38:00.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:37:59 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:38:00.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:37:59 smithi047 ceph-mon[77147]: pgmap v2459: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:38:00.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:37:59 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 4210 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:38:00.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:37:59 smithi115 conmon[90393]: 2022-05-14T00:37:59.845+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:38:00.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:37:59 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:38:00.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:37:59 smithi115 ceph-mon[78593]: pgmap v2459: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:38:00.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:37:59 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 4210 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:38:01.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:38:00 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:38:01.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:38:00 smithi115 conmon[90393]: 2022-05-14T00:38:00.895+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:38:01.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:38:00 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:38:02.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:38:01 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:38:02.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:38:01 smithi047 ceph-mon[77147]: pgmap v2460: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:38:02.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:38:01 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:38:02.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:38:01 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:38:02.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:38:01 smithi115 conmon[90393]: 2022-05-14T00:38:01.846+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:38:02.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:38:01 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:38:02.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:38:01 smithi115 ceph-mon[78593]: pgmap v2460: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:38:02.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:38:01 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:38:02.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:38:01 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:38:03.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:38:02 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:38:03.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:38:02 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:38:03.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:38:02 smithi115 conmon[90393]: 2022-05-14T00:38:02.818+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:38:04.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:38:03 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:38:04.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:38:03 smithi047 ceph-mon[77147]: pgmap v2461: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:38:04.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:38:03 smithi115 conmon[90393]: 2022-05-14T00:38:03.837+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:38:04.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:38:03 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:38:04.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:38:03 smithi115 ceph-mon[78593]: pgmap v2461: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:38:05.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:38:04 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:38:05.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:38:04 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 4215 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:38:05.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:38:04 smithi115 conmon[90393]: 2022-05-14T00:38:04.831+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:38:05.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:38:04 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:38:05.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:38:04 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 4215 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:38:06.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:38:05 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:38:06.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:38:05 smithi047 ceph-mon[77147]: pgmap v2462: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:38:06.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:38:05 smithi115 conmon[90393]: 2022-05-14T00:38:05.833+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:38:06.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:38:05 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:38:06.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:38:05 smithi115 ceph-mon[78593]: pgmap v2462: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:38:07.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:38:06 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:38:07.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:38:06 smithi115 conmon[90393]: 2022-05-14T00:38:06.877+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:38:07.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:38:06 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:38:08.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:38:07 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:38:08.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:38:07 smithi047 ceph-mon[77147]: pgmap v2463: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:38:08.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:38:07 smithi115 conmon[90393]: 2022-05-14T00:38:07.886+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:38:08.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:38:07 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:38:08.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:38:07 smithi115 ceph-mon[78593]: pgmap v2463: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:38:09.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:38:08 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:38:09.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:38:08 smithi115 conmon[90393]: 2022-05-14T00:38:08.934+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:38:09.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:38:08 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:38:10.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:38:09 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:38:10.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:38:09 smithi047 ceph-mon[77147]: pgmap v2464: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:38:10.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:38:09 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:38:10.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:38:09 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:38:10.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:38:09 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 4220 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:38:10.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:38:09 smithi115 conmon[90393]: 2022-05-14T00:38:09.907+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:38:10.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:38:09 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:38:10.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:38:09 smithi115 ceph-mon[78593]: pgmap v2464: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:38:10.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:38:09 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:38:10.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:38:09 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:38:10.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:38:09 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 4220 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:38:11.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:38:10 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:38:11.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:38:10 smithi115 conmon[90393]: 2022-05-14T00:38:10.953+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:38:11.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:38:10 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:38:12.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:38:11 smithi115 conmon[90393]: 2022-05-14T00:38:11.960+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:38:12.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:38:11 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:38:12.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:38:11 smithi115 ceph-mon[78593]: pgmap v2465: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:38:12.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:38:11 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:38:12.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:38:11 smithi047 ceph-mon[77147]: pgmap v2465: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:38:13.114 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:38:12 smithi115 conmon[90393]: 2022-05-14T00:38:12.980+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:38:13.115 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:38:12 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:38:13.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:38:12 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:38:14.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:38:13 smithi115 conmon[90393]: 2022-05-14T00:38:13.988+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:38:14.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:38:13 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:38:14.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:38:13 smithi115 ceph-mon[78593]: pgmap v2466: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:38:14.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:38:13 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:38:14.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:38:13 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:38:14.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:38:13 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T00:38:14.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:38:13 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T00:38:14.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:38:13 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:38:14.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:38:13 smithi047 ceph-mon[77147]: pgmap v2466: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:38:14.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:38:13 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:38:14.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:38:13 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:38:14.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:38:13 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T00:38:14.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:38:13 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T00:38:15.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:38:15 smithi115 conmon[90393]: 2022-05-14T00:38:15.023+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:38:15.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:38:14 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:38:15.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:38:14 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 4225 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:38:15.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:38:14 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:38:15.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:38:14 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 4225 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:38:16.114 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:38:15 smithi115 conmon[90393]: 2022-05-14T00:38:15.989+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:38:16.114 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:38:15 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:38:16.114 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:38:15 smithi115 ceph-mon[78593]: pgmap v2467: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:38:16.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:38:15 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:38:16.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:38:15 smithi047 ceph-mon[77147]: pgmap v2467: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:38:17.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:38:16 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:38:17.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:38:17 smithi115 conmon[90393]: 2022-05-14T00:38:17.007+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:38:17.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:38:16 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:38:18.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:38:17 smithi115 conmon[90393]: 2022-05-14T00:38:17.989+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:38:18.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:38:17 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:38:18.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:38:17 smithi115 ceph-mon[78593]: pgmap v2468: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:38:18.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:38:17 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:38:18.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:38:17 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:38:18.297 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:38:17 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:38:18.297 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:38:17 smithi047 ceph-mon[77147]: pgmap v2468: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:38:18.297 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:38:17 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:38:18.298 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:38:17 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:38:19.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:38:18 smithi115 conmon[90393]: 2022-05-14T00:38:18.985+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:38:19.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:38:18 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:38:19.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:38:18 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:38:19.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:38:18 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:38:19.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:38:18 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:38:20.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:38:19 smithi115 conmon[90393]: 2022-05-14T00:38:19.995+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:38:20.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:38:19 smithi115 ceph-mon[78593]: pgmap v2469: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:38:20.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:38:19 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:38:20.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:38:19 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 4230 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:38:20.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:38:19 smithi047 ceph-mon[77147]: pgmap v2469: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:38:20.295 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:38:19 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:38:20.295 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:38:19 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 4230 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:38:21.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:38:21 smithi115 conmon[90393]: 2022-05-14T00:38:21.026+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:38:21.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:38:20 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:38:21.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:38:20 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:38:22.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:38:22 smithi115 conmon[90393]: 2022-05-14T00:38:22.003+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:38:22.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:38:21 smithi115 ceph-mon[78593]: pgmap v2470: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:38:22.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:38:21 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:38:22.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:38:21 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:38:22.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:38:21 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:38:22.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:38:21 smithi047 ceph-mon[77147]: pgmap v2470: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:38:22.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:38:21 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:38:22.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:38:21 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:38:22.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:38:21 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:38:23.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:38:23 smithi115 conmon[90393]: 2022-05-14T00:38:23.021+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:38:23.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:38:22 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:38:23.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:38:22 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:38:24.110 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:38:24 smithi115 conmon[90393]: 2022-05-14T00:38:24.033+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:38:24.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:38:23 smithi115 ceph-mon[78593]: pgmap v2471: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:38:24.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:38:23 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:38:24.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:38:23 smithi047 ceph-mon[77147]: pgmap v2471: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:38:24.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:38:23 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:38:25.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:38:24 smithi115 conmon[90393]: 2022-05-14T00:38:24.993+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:38:25.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:38:24 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:38:25.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:38:24 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 4235 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:38:25.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:38:24 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:38:25.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:38:24 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 4235 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:38:26.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:38:25 smithi115 conmon[90393]: 2022-05-14T00:38:25.978+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:38:26.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:38:26 smithi115 ceph-mon[78593]: pgmap v2472: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:38:26.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:38:26 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:38:26.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:38:26 smithi047 ceph-mon[77147]: pgmap v2472: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:38:26.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:38:26 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:38:27.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:38:26 smithi115 conmon[90393]: 2022-05-14T00:38:26.932+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:38:27.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:38:27 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:38:27.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:38:27 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:38:28.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:38:27 smithi115 conmon[90393]: 2022-05-14T00:38:27.946+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:38:28.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:38:28 smithi115 ceph-mon[78593]: pgmap v2473: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:38:28.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:38:28 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:38:28.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:38:28 smithi047 ceph-mon[77147]: pgmap v2473: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:38:28.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:38:28 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:38:29.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:38:28 smithi115 conmon[90393]: 2022-05-14T00:38:28.909+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:38:29.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:38:29 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:38:29.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:38:29 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 4240 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:38:29.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:38:29 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:38:29.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:38:29 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 4240 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:38:30.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:38:29 smithi115 conmon[90393]: 2022-05-14T00:38:29.938+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:38:30.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:38:30 smithi115 ceph-mon[78593]: pgmap v2474: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:38:30.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:38:30 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:38:30.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:38:30 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:38:30.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:38:30 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:38:30.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:38:30 smithi047 ceph-mon[77147]: pgmap v2474: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:38:30.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:38:30 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:38:30.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:38:30 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:38:30.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:38:30 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:38:31.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:38:30 smithi115 conmon[90393]: 2022-05-14T00:38:30.960+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:38:31.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:38:31 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:38:31.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:38:31 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:38:32.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:38:31 smithi115 conmon[90393]: 2022-05-14T00:38:31.926+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:38:32.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:38:32 smithi115 ceph-mon[78593]: pgmap v2475: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:38:32.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:38:32 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:38:32.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:38:32 smithi047 ceph-mon[77147]: pgmap v2475: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:38:32.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:38:32 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:38:33.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:38:32 smithi115 conmon[90393]: 2022-05-14T00:38:32.897+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:38:33.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:38:33 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:38:33.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:38:33 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:38:34.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:38:33 smithi115 conmon[90393]: 2022-05-14T00:38:33.860+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:38:34.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:38:34 smithi115 ceph-mon[78593]: pgmap v2476: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:38:34.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:38:34 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:38:34.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:38:34 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:38:34.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:38:34 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:38:34.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:38:34 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 4245 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:38:34.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:38:34 smithi047 ceph-mon[77147]: pgmap v2476: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:38:34.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:38:34 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:38:34.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:38:34 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:38:34.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:38:34 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:38:34.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:38:34 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 4245 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:38:35.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:38:34 smithi115 conmon[90393]: 2022-05-14T00:38:34.837+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:38:35.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:38:35 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:38:35.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:38:35 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:38:36.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:38:35 smithi115 conmon[90393]: 2022-05-14T00:38:35.827+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:38:36.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:38:36 smithi047 ceph-mon[77147]: pgmap v2477: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:38:36.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:38:36 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:38:36.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:38:36 smithi115 ceph-mon[78593]: pgmap v2477: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:38:36.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:38:36 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:38:37.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:38:36 smithi115 conmon[90393]: 2022-05-14T00:38:36.844+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:38:37.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:38:37 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:38:37.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:38:37 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:38:38.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:38:37 smithi115 conmon[90393]: 2022-05-14T00:38:37.834+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:38:38.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:38:38 smithi047 ceph-mon[77147]: pgmap v2478: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:38:38.545 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:38:38 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:38:38.545 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:38:38 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:38:38.545 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:38:38 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:38:38.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:38:38 smithi115 ceph-mon[78593]: pgmap v2478: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:38:38.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:38:38 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:38:38.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:38:38 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:38:38.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:38:38 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:38:39.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:38:38 smithi115 conmon[90393]: 2022-05-14T00:38:38.834+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:38:39.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:38:39 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:38:39.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:38:39 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 4250 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:38:39.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:38:39 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:38:39.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:38:39 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 4250 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:38:40.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:38:39 smithi115 conmon[90393]: 2022-05-14T00:38:39.789+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:38:40.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:38:40 smithi047 ceph-mon[77147]: pgmap v2479: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:38:40.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:38:40 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:38:40.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:38:40 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:38:40.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:38:40 smithi115 ceph-mon[78593]: pgmap v2479: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:38:40.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:38:40 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:38:40.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:38:40 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:38:41.110 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:38:40 smithi115 conmon[90393]: 2022-05-14T00:38:40.776+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:38:41.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:38:41 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:38:41.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:38:41 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:38:42.110 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:38:41 smithi115 conmon[90393]: 2022-05-14T00:38:41.783+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:38:42.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:38:42 smithi047 ceph-mon[77147]: pgmap v2480: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:38:42.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:38:42 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:38:42.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:38:42 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:38:42.545 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:38:42 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:38:42.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:38:42 smithi115 ceph-mon[78593]: pgmap v2480: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:38:42.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:38:42 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:38:42.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:38:42 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:38:42.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:38:42 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:38:43.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:38:42 smithi115 conmon[90393]: 2022-05-14T00:38:42.739+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:38:43.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:38:43 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:38:43.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:38:43 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:38:44.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:38:43 smithi115 conmon[90393]: 2022-05-14T00:38:43.709+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:38:44.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:38:44 smithi047 ceph-mon[77147]: pgmap v2481: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:38:44.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:38:44 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:38:44.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:38:44 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 4255 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:38:44.614 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:38:44 smithi115 ceph-mon[78593]: pgmap v2481: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:38:44.615 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:38:44 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:38:44.615 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:38:44 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 4255 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:38:45.111 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:38:44 smithi115 conmon[90393]: 2022-05-14T00:38:44.702+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:38:45.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:38:45 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:38:45.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:38:45 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:38:46.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:38:45 smithi115 conmon[90393]: 2022-05-14T00:38:45.714+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:38:46.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:38:46 smithi047 ceph-mon[77147]: pgmap v2482: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:38:46.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:38:46 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:38:46.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:38:46 smithi115 ceph-mon[78593]: pgmap v2482: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:38:46.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:38:46 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:38:47.115 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:38:46 smithi115 conmon[90393]: 2022-05-14T00:38:46.700+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:38:47.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:38:47 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:38:47.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:38:47 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:38:48.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:38:47 smithi115 conmon[90393]: 2022-05-14T00:38:47.745+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:38:48.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:38:48 smithi047 ceph-mon[77147]: pgmap v2483: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:38:48.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:38:48 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:38:48.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:38:48 smithi115 ceph-mon[78593]: pgmap v2483: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:38:48.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:38:48 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:38:49.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:38:48 smithi115 conmon[90393]: 2022-05-14T00:38:48.728+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:38:49.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:38:49 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:38:49.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:38:49 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 4260 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:38:49.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:38:49 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:38:49.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:38:49 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 4260 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:38:50.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:38:49 smithi115 conmon[90393]: 2022-05-14T00:38:49.727+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:38:50.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:38:50 smithi047 ceph-mon[77147]: pgmap v2484: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:38:50.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:38:50 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:38:50.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:38:50 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:38:50.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:38:50 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:38:50.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:38:50 smithi115 ceph-mon[78593]: pgmap v2484: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:38:50.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:38:50 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:38:50.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:38:50 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:38:50.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:38:50 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:38:51.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:38:50 smithi115 conmon[90393]: 2022-05-14T00:38:50.728+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:38:51.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:38:51 smithi047 ceph-mon[77147]: pgmap v2485: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:38:51.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:38:51 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:38:51.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:38:51 smithi115 ceph-mon[78593]: pgmap v2485: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:38:51.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:38:51 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:38:52.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:38:51 smithi115 conmon[90393]: 2022-05-14T00:38:51.702+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:38:52.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:38:52 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:38:52.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:38:52 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:38:53.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:38:52 smithi115 conmon[90393]: 2022-05-14T00:38:52.736+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:38:53.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:38:53 smithi047 ceph-mon[77147]: pgmap v2486: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:38:53.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:38:53 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:38:53.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:38:53 smithi115 ceph-mon[78593]: pgmap v2486: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:38:53.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:38:53 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:38:54.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:38:53 smithi115 conmon[90393]: 2022-05-14T00:38:53.754+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:38:54.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:38:54 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:38:54.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:38:54 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:38:54.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:38:54 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:38:54.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:38:54 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 4265 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:38:54.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:38:54 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:38:54.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:38:54 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:38:54.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:38:54 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:38:54.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:38:54 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 4265 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:38:55.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:38:54 smithi115 conmon[90393]: 2022-05-14T00:38:54.708+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:38:55.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:38:55 smithi047 ceph-mon[77147]: pgmap v2487: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:38:55.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:38:55 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T00:38:55.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:38:55 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T00:38:55.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:38:55 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T00:38:55.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:38:55 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:38:55.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:38:55 smithi115 ceph-mon[78593]: pgmap v2487: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:38:55.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:38:55 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T00:38:55.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:38:55 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T00:38:55.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:38:55 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T00:38:55.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:38:55 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:38:56.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:38:55 smithi115 conmon[90393]: 2022-05-14T00:38:55.668+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:38:56.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:38:56 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:38:56.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:38:56 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:38:57.111 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:38:56 smithi115 conmon[90393]: 2022-05-14T00:38:56.696+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:38:57.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:38:57 smithi047 ceph-mon[77147]: pgmap v2488: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:38:57.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:38:57 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:38:57.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:38:57 smithi115 ceph-mon[78593]: pgmap v2488: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:38:57.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:38:57 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:38:58.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:38:57 smithi047 conmon[78521]: 2022-05-14T00:38:57.775+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:38:58.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:38:57 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T00:38:58.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:38:57 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:38:58.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:38:57 smithi047 conmon[78521]: yield 2022-05-14T00:38:58.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:38:57 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:38:58.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:38:57 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:38:58.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:38:57 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:38:58.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:38:57 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T00:38:58.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:38:57 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:38:58.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:38:57 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T00:38:58.048 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:38:57 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:38:58.048 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:38:57 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T00:38:58.048 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:38:57 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:38:58.049 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:38:57 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T00:38:58.049 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:38:57 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:38:58.049 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:38:57 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T00:38:58.050 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:38:57 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:38:58.051 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:38:57 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:38:58.051 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:38:57 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:38:58.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:38:57 smithi115 conmon[90393]: 2022-05-14T00:38:57.733+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:38:58.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:38:58 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:38:58.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:38:58 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:38:58.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:38:58 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:38:58.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:38:58 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:38:58.542 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T00:38:58.542 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:38:58.542 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T00:38:58.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:38:58.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:38:58.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:38:58.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T00:38:58.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:38:58.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T00:38:58.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:38:58.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T00:38:58.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:38:58.548 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T00:38:58.548 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:38:58.548 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T00:38:58.548 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:38:58.548 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:38:58.549 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:38:58.549 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:38:58 smithi047 ceph-mon[77147]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T00:38:58.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:38:58 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:38:58.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:38:58 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:38:58.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:38:58 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:38:58.612 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:38:58 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:38:58.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T00:38:58.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:38:58.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T00:38:58.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:38:58.613 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:38:58.613 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:38:58.613 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T00:38:58.613 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:38:58.613 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T00:38:58.614 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:38:58.614 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T00:38:58.614 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:38:58.614 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T00:38:58.615 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:38:58.615 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T00:38:58.615 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:38:58.615 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:38:58.615 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:38:58.616 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:38:58 smithi115 ceph-mon[78593]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T00:38:59.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:38:58 smithi115 conmon[90393]: 2022-05-14T00:38:58.701+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:38:59.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:38:59 smithi047 ceph-mon[77147]: pgmap v2489: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:38:59.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:38:59 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:38:59.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:38:59 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 4270 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:38:59.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:38:59 smithi115 ceph-mon[78593]: pgmap v2489: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:38:59.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:38:59 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:38:59.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:38:59 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 4270 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:39:00.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:38:59 smithi115 conmon[90393]: 2022-05-14T00:38:59.681+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:39:00.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:39:00 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:39:00.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:39:00 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:39:01.110 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:39:00 smithi115 conmon[90393]: 2022-05-14T00:39:00.684+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:39:01.190 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:39:00 smithi047 conmon[78521]: 2022-05-14T00:39:00.848+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:39:01.191 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:39:00 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T00:39:01.191 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:39:00 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:39:01.192 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:39:00 smithi047 conmon[78521]: yield 2022-05-14T00:39:01.192 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:39:00 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:39:01.193 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:39:00 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:39:01.193 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:39:00 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:39:01.194 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:39:00 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T00:39:01.194 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:39:00 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:39:01.195 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:39:00 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T00:39:01.195 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:39:00 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:39:01.195 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:39:00 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T00:39:01.196 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:39:00 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:39:01.196 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:39:00 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T00:39:01.197 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:39:00 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:39:01.197 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:39:00 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T00:39:01.198 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:39:00 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:39:01.198 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:39:00 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:39:01.199 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:39:00 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:39:01.199 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:39:00 smithi047 conmon[78521]: 2022-05-14T00:39:00.848+0000 7f533ef76700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T00:39:01.199 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:39:00 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T00:39:01.200 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:39:00 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:39:01.200 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:39:00 smithi047 conmon[78521]: yield 2022-05-14T00:39:01.201 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:39:00 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:39:01.201 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:39:00 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:39:01.202 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:39:00 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:39:01.202 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:39:00 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T00:39:01.203 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:39:00 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:39:01.203 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:39:00 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T00:39:01.203 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:39:00 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:39:01.204 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:39:00 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T00:39:01.204 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:39:00 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:39:01.205 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:39:00 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T00:39:01.205 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:39:00 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:39:01.206 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:39:00 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T00:39:01.206 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:39:00 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:39:01.208 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:39:00 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:39:01.208 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:39:00 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:39:01.208 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:39:00 smithi047 conmon[78521]: 2022-05-14T00:39:01.208 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:39:00 smithi047 conmon[78521]: During handling of the above exception, another exception occurred: 2022-05-14T00:39:01.209 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:39:00 smithi047 conmon[78521]: 2022-05-14T00:39:01.209 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:39:00 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T00:39:01.209 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:39:00 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T00:39:01.209 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:39:00 smithi047 conmon[78521]: return f(*arg) 2022-05-14T00:39:01.210 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:39:00 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T00:39:01.210 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:39:00 smithi047 conmon[78521]: self._write_client_files(client_files, host) 2022-05-14T00:39:01.210 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:39:00 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T00:39:01.210 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:39:00 smithi047 conmon[78521]: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T00:39:01.210 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:39:00 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T00:39:01.211 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:39:00 smithi047 conmon[78521]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T00:39:01.211 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:39:00 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T00:39:01.211 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:39:00 smithi047 conmon[78521]: return self.event_loop.get_result(coro) 2022-05-14T00:39:01.211 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:39:00 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T00:39:01.212 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:39:00 smithi047 conmon[78521]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T00:39:01.212 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:39:00 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T00:39:01.212 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:39:00 smithi047 conmon[78521]: return self.__get_result() 2022-05-14T00:39:01.213 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:39:00 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T00:39:01.214 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:39:00 smithi047 conmon[78521]: raise self._exception 2022-05-14T00:39:01.214 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:39:00 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T00:39:01.214 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:39:00 smithi047 conmon[78521]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T00:39:01.215 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:39:00 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T00:39:01.215 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:39:00 smithi047 conmon[78521]: conn = await self._remote_connection(host, addr) 2022-05-14T00:39:01.215 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:39:00 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T00:39:01.215 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:39:00 smithi047 conmon[78521]: raise 2022-05-14T00:39:01.216 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:39:00 smithi047 conmon[78521]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T00:39:01.216 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:39:00 smithi047 conmon[78521]: self.gen.throw(type, value, traceback) 2022-05-14T00:39:01.216 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:39:00 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T00:39:01.216 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:39:00 smithi047 conmon[78521]: raise OrchestratorError(msg) 2022-05-14T00:39:01.217 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:39:00 smithi047 conmon[78521]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:39:01.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:39:01 smithi047 ceph-mon[77147]: pgmap v2490: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:39:01.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:39:01 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:39:01.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:39:01 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:39:01.542 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T00:39:01.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:39:01.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T00:39:01.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:39:01.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:39:01.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:39:01.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T00:39:01.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:39:01.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T00:39:01.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:39:01.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T00:39:01.548 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:39:01.548 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T00:39:01.549 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:39:01.550 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T00:39:01.550 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:39:01.550 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:39:01.551 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:39:01.551 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:39:01 smithi047 ceph-mon[77147]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T00:39:01.552 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T00:39:01.552 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:39:01.553 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T00:39:01.553 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:39:01.554 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:39:01.554 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:39:01.555 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T00:39:01.555 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:39:01.555 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T00:39:01.556 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:39:01.556 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T00:39:01.557 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:39:01.557 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T00:39:01.558 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:39:01.558 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T00:39:01.560 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:39:01.560 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:39:01.560 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:39:01.560 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T00:39:01.561 INFO:journalctl@ceph.mon.b.smithi047.stdout: During handling of the above exception, another exception occurred: 2022-05-14T00:39:01.561 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T00:39:01.561 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T00:39:01.561 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T00:39:01.561 INFO:journalctl@ceph.mon.b.smithi047.stdout: return f(*arg) 2022-05-14T00:39:01.562 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T00:39:01.562 INFO:journalctl@ceph.mon.b.smithi047.stdout: self._write_client_files(client_files, host) 2022-05-14T00:39:01.562 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T00:39:01.562 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T00:39:01.563 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T00:39:01.563 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T00:39:01.563 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T00:39:01.563 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.event_loop.get_result(coro) 2022-05-14T00:39:01.563 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T00:39:01.564 INFO:journalctl@ceph.mon.b.smithi047.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T00:39:01.564 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T00:39:01.564 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.__get_result() 2022-05-14T00:39:01.564 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T00:39:01.565 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise self._exception 2022-05-14T00:39:01.565 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T00:39:01.565 INFO:journalctl@ceph.mon.b.smithi047.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T00:39:01.565 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T00:39:01.566 INFO:journalctl@ceph.mon.b.smithi047.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T00:39:01.566 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T00:39:01.566 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise 2022-05-14T00:39:01.566 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T00:39:01.566 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.gen.throw(type, value, traceback) 2022-05-14T00:39:01.567 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T00:39:01.567 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OrchestratorError(msg) 2022-05-14T00:39:01.567 INFO:journalctl@ceph.mon.b.smithi047.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:39:01.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:39:01 smithi115 ceph-mon[78593]: pgmap v2490: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:39:01.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:39:01 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:39:01.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:39:01 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:39:01.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T00:39:01.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:39:01.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T00:39:01.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:39:01.613 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:39:01.613 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:39:01.613 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T00:39:01.617 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:39:01.618 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T00:39:01.618 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:39:01.618 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T00:39:01.618 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:39:01.619 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T00:39:01.619 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:39:01.619 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T00:39:01.620 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:39:01.620 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:39:01.620 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:39:01.620 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:39:01 smithi115 ceph-mon[78593]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T00:39:01.620 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T00:39:01.621 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:39:01.621 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T00:39:01.622 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:39:01.622 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:39:01.622 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:39:01.622 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T00:39:01.622 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:39:01.623 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T00:39:01.623 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:39:01.623 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T00:39:01.623 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:39:01.623 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T00:39:01.624 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:39:01.624 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T00:39:01.624 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:39:01.624 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:39:01.625 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:39:01.625 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T00:39:01.625 INFO:journalctl@ceph.mon.c.smithi115.stdout: During handling of the above exception, another exception occurred: 2022-05-14T00:39:01.625 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T00:39:01.625 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T00:39:01.626 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T00:39:01.626 INFO:journalctl@ceph.mon.c.smithi115.stdout: return f(*arg) 2022-05-14T00:39:01.626 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T00:39:01.626 INFO:journalctl@ceph.mon.c.smithi115.stdout: self._write_client_files(client_files, host) 2022-05-14T00:39:01.627 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T00:39:01.627 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T00:39:01.627 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T00:39:01.627 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T00:39:01.627 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T00:39:01.628 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.event_loop.get_result(coro) 2022-05-14T00:39:01.628 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T00:39:01.628 INFO:journalctl@ceph.mon.c.smithi115.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T00:39:01.628 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T00:39:01.628 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.__get_result() 2022-05-14T00:39:01.629 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T00:39:01.629 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise self._exception 2022-05-14T00:39:01.630 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T00:39:01.630 INFO:journalctl@ceph.mon.c.smithi115.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T00:39:01.631 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T00:39:01.631 INFO:journalctl@ceph.mon.c.smithi115.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T00:39:01.631 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T00:39:01.631 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise 2022-05-14T00:39:01.632 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T00:39:01.632 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.gen.throw(type, value, traceback) 2022-05-14T00:39:01.632 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T00:39:01.633 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OrchestratorError(msg) 2022-05-14T00:39:01.633 INFO:journalctl@ceph.mon.c.smithi115.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:39:02.115 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:39:01 smithi115 conmon[90393]: 2022-05-14T00:39:01.689+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:39:02.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:39:02 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:39:02.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:39:02 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:39:02.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:39:02 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:39:02.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:39:02 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:39:02.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:39:02 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:39:02.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:39:02 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:39:03.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:39:02 smithi115 conmon[90393]: 2022-05-14T00:39:02.640+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:39:03.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:39:03 smithi047 ceph-mon[77147]: pgmap v2491: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:39:03.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:39:03 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:39:03.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:39:03 smithi115 ceph-mon[78593]: pgmap v2491: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:39:03.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:39:03 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:39:04.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:39:03 smithi115 conmon[90393]: 2022-05-14T00:39:03.685+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:39:04.546 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:39:04 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:39:04.546 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:39:04 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 4275 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:39:04.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:39:04 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:39:04.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:39:04 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 4275 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:39:05.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:39:04 smithi115 conmon[90393]: 2022-05-14T00:39:04.666+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:39:05.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:39:05 smithi047 ceph-mon[77147]: pgmap v2492: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:39:05.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:39:05 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:39:05.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:39:05 smithi115 ceph-mon[78593]: pgmap v2492: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:39:05.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:39:05 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:39:06.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:39:05 smithi115 conmon[90393]: 2022-05-14T00:39:05.671+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:39:06.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:39:06 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:39:06.615 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:39:06 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:39:07.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:39:06 smithi115 conmon[90393]: 2022-05-14T00:39:06.650+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:39:07.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:39:07 smithi047 ceph-mon[77147]: pgmap v2493: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:39:07.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:39:07 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:39:07.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:39:07 smithi115 ceph-mon[78593]: pgmap v2493: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:39:07.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:39:07 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:39:08.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:39:07 smithi115 conmon[90393]: 2022-05-14T00:39:07.697+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:39:08.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:39:08 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:39:08.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:39:08 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:39:09.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:39:08 smithi115 conmon[90393]: 2022-05-14T00:39:08.679+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:39:09.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:39:09 smithi047 ceph-mon[77147]: pgmap v2494: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:39:09.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:39:09 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:39:09.546 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:39:09 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 4280 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:39:09.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:39:09 smithi115 ceph-mon[78593]: pgmap v2494: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:39:09.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:39:09 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:39:09.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:39:09 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 4280 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:39:10.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:39:09 smithi115 conmon[90393]: 2022-05-14T00:39:09.689+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:39:10.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:39:10 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:39:10.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:39:10 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:39:10.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:39:10 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:39:10.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:39:10 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:39:10.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:39:10 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:39:10.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:39:10 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:39:11.110 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:39:10 smithi115 conmon[90393]: 2022-05-14T00:39:10.719+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:39:11.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:39:11 smithi047 ceph-mon[77147]: pgmap v2495: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:39:11.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:39:11 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:39:11.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:39:11 smithi115 ceph-mon[78593]: pgmap v2495: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:39:11.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:39:11 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:39:12.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:39:11 smithi115 conmon[90393]: 2022-05-14T00:39:11.752+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:39:12.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:39:12 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:39:12.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:39:12 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:39:13.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:39:12 smithi115 conmon[90393]: 2022-05-14T00:39:12.716+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:39:13.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:39:13 smithi047 ceph-mon[77147]: pgmap v2496: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:39:13.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:39:13 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:39:13.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:39:13 smithi115 ceph-mon[78593]: pgmap v2496: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:39:13.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:39:13 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:39:14.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:39:13 smithi115 conmon[90393]: 2022-05-14T00:39:13.738+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:39:14.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:39:14 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:39:14.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:39:14 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:39:14.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:39:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T00:39:14.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:39:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T00:39:14.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:39:14 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:39:14.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:39:14 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 4285 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:39:14.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:39:14 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:39:14.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:39:14 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:39:14.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:39:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T00:39:14.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:39:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T00:39:14.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:39:14 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:39:14.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:39:14 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 4285 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:39:15.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:39:14 smithi115 conmon[90393]: 2022-05-14T00:39:14.744+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:39:15.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:39:15 smithi047 ceph-mon[77147]: pgmap v2497: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:39:15.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:39:15 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:39:15.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:39:15 smithi115 ceph-mon[78593]: pgmap v2497: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:39:15.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:39:15 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:39:16.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:39:15 smithi115 conmon[90393]: 2022-05-14T00:39:15.715+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:39:16.546 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:39:16 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:39:16.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:39:16 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:39:17.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:39:16 smithi115 conmon[90393]: 2022-05-14T00:39:16.703+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:39:17.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:39:17 smithi047 ceph-mon[77147]: pgmap v2498: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:39:17.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:39:17 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:39:17.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:39:17 smithi115 ceph-mon[78593]: pgmap v2498: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:39:17.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:39:17 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:39:18.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:39:17 smithi115 conmon[90393]: 2022-05-14T00:39:17.725+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:39:18.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:39:18 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:39:18.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:39:18 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:39:18.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:39:18 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:39:18.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:39:18 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:39:18.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:39:18 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:39:18.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:39:18 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:39:19.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:39:18 smithi115 conmon[90393]: 2022-05-14T00:39:18.682+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:39:19.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:39:19 smithi047 ceph-mon[77147]: pgmap v2499: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:39:19.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:39:19 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:39:19.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:39:19 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 4290 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:39:19.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:39:19 smithi115 ceph-mon[78593]: pgmap v2499: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:39:19.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:39:19 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:39:19.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:39:19 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 4290 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:39:20.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:39:19 smithi115 conmon[90393]: 2022-05-14T00:39:19.671+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:39:20.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:39:20 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:39:20.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:39:20 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:39:21.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:39:20 smithi115 conmon[90393]: 2022-05-14T00:39:20.671+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:39:21.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:39:21 smithi047 ceph-mon[77147]: pgmap v2500: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:39:21.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:39:21 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:39:21.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:39:21 smithi115 ceph-mon[78593]: pgmap v2500: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:39:21.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:39:21 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:39:22.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:39:21 smithi115 conmon[90393]: 2022-05-14T00:39:21.670+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:39:22.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:39:22 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:39:22.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:39:22 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:39:22.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:39:22 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:39:22.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:39:22 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:39:22.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:39:22 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:39:22.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:39:22 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:39:23.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:39:22 smithi115 conmon[90393]: 2022-05-14T00:39:22.699+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:39:23.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:39:23 smithi047 ceph-mon[77147]: pgmap v2501: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:39:23.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:39:23 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:39:23.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:39:23 smithi115 ceph-mon[78593]: pgmap v2501: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:39:23.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:39:23 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:39:24.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:39:23 smithi115 conmon[90393]: 2022-05-14T00:39:23.722+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:39:24.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:39:24 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:39:24.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:39:24 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 4295 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:39:24.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:39:24 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:39:24.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:39:24 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 4295 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:39:25.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:39:24 smithi115 conmon[90393]: 2022-05-14T00:39:24.716+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:39:25.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:39:25 smithi047 ceph-mon[77147]: pgmap v2502: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:39:25.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:39:25 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:39:25.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:39:25 smithi115 ceph-mon[78593]: pgmap v2502: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:39:25.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:39:25 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:39:26.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:39:25 smithi115 conmon[90393]: 2022-05-14T00:39:25.765+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:39:26.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:39:26 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:39:26.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:39:26 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:39:27.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:39:26 smithi115 conmon[90393]: 2022-05-14T00:39:26.804+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:39:27.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:39:27 smithi047 ceph-mon[77147]: pgmap v2503: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:39:27.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:39:27 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:39:27.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:39:27 smithi115 ceph-mon[78593]: pgmap v2503: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:39:27.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:39:27 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:39:28.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:39:27 smithi115 conmon[90393]: 2022-05-14T00:39:27.844+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:39:29.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:39:28 smithi115 conmon[90393]: 2022-05-14T00:39:28.825+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:39:29.844 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:39:29 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:39:29.844 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:39:29 smithi115 ceph-mon[78593]: pgmap v2504: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:39:29.844 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:39:29 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:39:29.845 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:39:29 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:39:29.845 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:39:29 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 4300 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:39:30.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:39:29 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:39:30.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:39:29 smithi047 ceph-mon[77147]: pgmap v2504: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:39:30.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:39:29 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:39:30.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:39:29 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:39:30.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:39:29 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 4300 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:39:30.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:39:29 smithi115 conmon[90393]: 2022-05-14T00:39:29.843+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:39:30.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:39:30 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:39:30.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:39:30 smithi115 conmon[90393]: 2022-05-14T00:39:30.834+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:39:31.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:39:30 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:39:31.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:39:31 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:39:31.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:39:31 smithi115 ceph-mon[78593]: pgmap v2505: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:39:32.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:39:31 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:39:32.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:39:31 smithi047 ceph-mon[77147]: pgmap v2505: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:39:32.360 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:39:31 smithi115 conmon[90393]: 2022-05-14T00:39:31.861+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:39:32.842 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:39:32 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:39:33.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:39:32 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:39:33.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:39:32 smithi115 conmon[90393]: 2022-05-14T00:39:32.841+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:39:33.847 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:39:33 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:39:33.848 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:39:33 smithi115 ceph-mon[78593]: pgmap v2506: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:39:33.848 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:39:33 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:39:33.848 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:39:33 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:39:34.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:39:33 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:39:34.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:39:33 smithi047 ceph-mon[77147]: pgmap v2506: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:39:34.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:39:33 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:39:34.045 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:39:33 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:39:34.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:39:33 smithi115 conmon[90393]: 2022-05-14T00:39:33.846+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:39:34.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:39:34 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:39:34.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:39:34 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 4305 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:39:35.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:39:34 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:39:35.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:39:34 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 4305 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:39:35.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:39:34 smithi115 conmon[90393]: 2022-05-14T00:39:34.873+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:39:35.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:39:35 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:39:35.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:39:35 smithi115 ceph-mon[78593]: pgmap v2507: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:39:35.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:39:35 smithi115 conmon[90393]: 2022-05-14T00:39:35.841+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:39:36.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:39:35 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:39:36.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:39:35 smithi047 ceph-mon[77147]: pgmap v2507: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:39:36.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:39:36 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:39:36.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:39:36 smithi115 conmon[90393]: 2022-05-14T00:39:36.840+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:39:37.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:39:36 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:39:37.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:39:37 smithi115 conmon[90393]: 2022-05-14T00:39:37.829+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:39:37.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:39:37 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:39:37.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:39:37 smithi115 ceph-mon[78593]: pgmap v2508: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:39:37.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:39:37 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:39:37.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:39:37 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:39:38.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:39:37 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:39:38.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:39:37 smithi047 ceph-mon[77147]: pgmap v2508: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:39:38.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:39:37 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:39:38.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:39:37 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:39:38.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:39:38 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:39:38.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:39:38 smithi115 conmon[90393]: 2022-05-14T00:39:38.832+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:39:39.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:39:38 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:39:39.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:39:39 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:39:39.862 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:39:39 smithi115 ceph-mon[78593]: pgmap v2509: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:39:39.862 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:39:39 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 4310 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:39:40.047 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:39:39 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:39:40.048 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:39:39 smithi047 ceph-mon[77147]: pgmap v2509: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:39:40.048 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:39:39 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 4310 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:39:40.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:39:39 smithi115 conmon[90393]: 2022-05-14T00:39:39.858+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:39:41.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:39:40 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:39:41.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:39:40 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:39:41.111 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:39:40 smithi115 conmon[90393]: 2022-05-14T00:39:40.844+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:39:41.883 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:39:41 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:39:41.884 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:39:41 smithi115 ceph-mon[78593]: pgmap v2510: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:39:41.884 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:39:41 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:39:41.884 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:39:41 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:39:42.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:39:41 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:39:42.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:39:41 smithi047 ceph-mon[77147]: pgmap v2510: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:39:42.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:39:41 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:39:42.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:39:41 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:39:42.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:39:41 smithi115 conmon[90393]: 2022-05-14T00:39:41.882+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:39:42.900 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:39:42 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:39:43.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:39:42 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:39:43.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:39:42 smithi115 conmon[90393]: 2022-05-14T00:39:42.898+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:39:43.918 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:39:43 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:39:43.918 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:39:43 smithi115 ceph-mon[78593]: pgmap v2511: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:39:44.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:39:43 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:39:44.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:39:43 smithi047 ceph-mon[77147]: pgmap v2511: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:39:44.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:39:43 smithi115 conmon[90393]: 2022-05-14T00:39:43.917+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:39:44.914 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:39:44 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:39:44.915 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:39:44 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 4315 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:39:45.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:39:44 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:39:45.045 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:39:44 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 4315 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:39:45.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:39:44 smithi115 conmon[90393]: 2022-05-14T00:39:44.914+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:39:45.935 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:39:45 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:39:45.936 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:39:45 smithi115 ceph-mon[78593]: pgmap v2512: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:39:46.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:39:45 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:39:46.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:39:45 smithi047 ceph-mon[77147]: pgmap v2512: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:39:46.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:39:45 smithi115 conmon[90393]: 2022-05-14T00:39:45.934+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:39:46.965 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:39:46 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:39:47.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:39:46 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:39:47.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:39:46 smithi115 conmon[90393]: 2022-05-14T00:39:46.964+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:39:47.916 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:39:47 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:39:47.917 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:39:47 smithi115 ceph-mon[78593]: pgmap v2513: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:39:48.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:39:47 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:39:48.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:39:47 smithi047 ceph-mon[77147]: pgmap v2513: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:39:48.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:39:47 smithi115 conmon[90393]: 2022-05-14T00:39:47.915+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:39:48.913 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:39:48 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:39:49.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:39:48 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:39:49.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:39:48 smithi115 conmon[90393]: 2022-05-14T00:39:48.912+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:39:49.893 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:39:49 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:39:49.894 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:39:49 smithi115 ceph-mon[78593]: pgmap v2514: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:39:49.894 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:39:49 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:39:49.895 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:39:49 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:39:49.895 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:39:49 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 4320 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:39:50.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:39:49 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:39:50.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:39:49 smithi047 ceph-mon[77147]: pgmap v2514: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:39:50.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:39:49 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:39:50.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:39:49 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:39:50.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:39:49 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 4320 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:39:50.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:39:49 smithi115 conmon[90393]: 2022-05-14T00:39:49.892+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:39:50.910 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:39:50 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:39:51.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:39:50 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:39:51.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:39:50 smithi115 conmon[90393]: 2022-05-14T00:39:50.909+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:39:51.954 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:39:51 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:39:51.954 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:39:51 smithi115 ceph-mon[78593]: pgmap v2515: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:39:52.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:39:51 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:39:52.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:39:51 smithi047 ceph-mon[77147]: pgmap v2515: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:39:52.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:39:51 smithi115 conmon[90393]: 2022-05-14T00:39:51.953+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:39:52.968 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:39:52 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:39:53.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:39:52 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:39:53.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:39:52 smithi115 conmon[90393]: 2022-05-14T00:39:52.967+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:39:53.946 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:39:53 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:39:53.946 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:39:53 smithi115 ceph-mon[78593]: pgmap v2516: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:39:53.946 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:39:53 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:39:53.947 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:39:53 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:39:54.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:39:53 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:39:54.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:39:53 smithi047 ceph-mon[77147]: pgmap v2516: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:39:54.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:39:53 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:39:54.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:39:53 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:39:54.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:39:53 smithi115 conmon[90393]: 2022-05-14T00:39:53.944+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:39:54.987 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:39:54 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:39:54.987 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:39:54 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 4325 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:39:55.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:39:54 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:39:55.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:39:54 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 4325 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:39:55.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:39:54 smithi115 conmon[90393]: 2022-05-14T00:39:54.986+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:39:55.989 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:39:55 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:39:55.989 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:39:55 smithi115 ceph-mon[78593]: pgmap v2517: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:39:56.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:39:55 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:39:56.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:39:55 smithi047 ceph-mon[77147]: pgmap v2517: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:39:56.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:39:55 smithi115 conmon[90393]: 2022-05-14T00:39:55.988+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:39:57.022 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:39:56 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:39:57.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:39:56 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:39:57.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:39:57 smithi115 conmon[90393]: 2022-05-14T00:39:57.021+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:39:57.990 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:39:57 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:39:57.990 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:39:57 smithi115 ceph-mon[78593]: pgmap v2518: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:39:57.991 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:39:57 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:39:57.991 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:39:57 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:39:58.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:39:57 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:39:58.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:39:57 smithi047 ceph-mon[77147]: pgmap v2518: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:39:58.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:39:57 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:39:58.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:39:57 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:39:58.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:39:57 smithi115 conmon[90393]: 2022-05-14T00:39:57.989+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:39:58.940 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:39:58 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:39:59.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:39:58 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:39:59.360 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:39:58 smithi115 conmon[90393]: 2022-05-14T00:39:58.940+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:39:59.929 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:39:59 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:39:59.929 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:39:59 smithi115 ceph-mon[78593]: pgmap v2519: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:39:59.930 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:39:59 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 4330 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:40:00.002 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:39:59 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:40:00.002 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:39:59 smithi047 ceph-mon[77147]: pgmap v2519: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:40:00.002 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:39:59 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 4330 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:40:00.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:40:00 smithi047 conmon[77123]: 2022-05-14T00:39:59.999+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : Health detail: HEALTH_ERR 1 filesystem is degraded; 1 mds daemon damaged; 1 MDSs are read only; 1 MDSs report slow metadata IOs; 1/3 mons down, quorum b,c; Reduced data availability: 1 pg inactive; 9 slow ops, oldest one blocked for 4330 sec, osd.5 has slow ops 2022-05-14T00:40:00.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:40:00 smithi047 conmon[77123]: 2022-05-14T00:39:59.999+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : [WRN] FS_DEGRADED: 1 filesystem is degraded 2022-05-14T00:40:00.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:40:00 smithi047 conmon[77123]: 2022-05-14T00:39:59.999+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : fs cephfs is degraded 2022-05-14T00:40:00.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:40:00 smithi047 conmon[77123]: 2022-05-14T00:39:59.999+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : [ERR] MDS_DAMAGE: 1 mds daemon damaged 2022-05-14T00:40:00.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:40:00 smithi047 conmon[77123]: 2022-05-14T00:39:59.999+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : fs cephfs mds.2 is damaged 2022-05-14T00:40:00.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:40:00 smithi047 conmon[77123]: 2022-05-14T00:39:59.999+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : [WRN] MDS_READ_ONLY: 1 MDSs are read only 2022-05-14T00:40:00.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:40:00 smithi047 conmon[77123]: 2022-05-14T00:39:59.999+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : mds.l(mds.0): MDS in read-only mode 2022-05-14T00:40:00.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:40:00 smithi047 conmon[77123]: 2022-05-14T00:39:59.999+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : [WRN] MDS_SLOW_METADATA_IO: 1 MDSs report slow metadata IOs 2022-05-14T00:40:00.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:40:00 smithi047 conmon[77123]: 2022-05-14T00:39:59.999+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : mds.l(mds.0): 4 slow metadata IOs are blocked > 180 secs, oldest blocked for 4633 secs 2022-05-14T00:40:00.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:40:00 smithi047 conmon[77123]: 2022-05-14T00:39:59.999+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : [WRN] MON_DOWN: 1/3 mons down, quorum b,c 2022-05-14T00:40:00.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:40:00 smithi047 conmon[77123]: 2022-05-14T00:40:00.000+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : mon.a (rank 0) addr [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] is down (out of quorum) 2022-05-14T00:40:00.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:40:00 smithi047 conmon[77123]: 2022-05-14T00:40:00.000+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : [WRN] PG_AVAILABILITY: Reduced data availability: 1 pg inactive 2022-05-14T00:40:00.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:40:00 smithi047 conmon[77123]: 2022-05-14T00:40:00.000+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : pg 2.1 is stuck inactive for 84m, current state unknown, last acting [] 2022-05-14T00:40:00.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:40:00 smithi047 conmon[77123]: 2022-05-14T00:40:00.000+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : [WRN] SLOW_OPS: 9 slow ops, oldest one blocked for 4330 sec, osd.5 has slow ops 2022-05-14T00:40:00.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:39:59 smithi115 conmon[90393]: 2022-05-14T00:39:59.928+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:40:00.924 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:40:00 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:40:00.925 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:40:00 smithi047 ceph-mon[77147]: Health detail: HEALTH_ERR 1 filesystem is degraded; 1 mds daemon damaged; 1 MDSs are read only; 1 MDSs report slow metadata IOs; 1/3 mons down, quorum b,c; Reduced data availability: 1 pg inactive; 9 slow ops, oldest one blocked for 4330 sec, osd.5 has slow ops 2022-05-14T00:40:00.925 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:40:00 smithi047 ceph-mon[77147]: [WRN] FS_DEGRADED: 1 filesystem is degraded 2022-05-14T00:40:00.925 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:40:00 smithi047 ceph-mon[77147]: fs cephfs is degraded 2022-05-14T00:40:00.925 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:40:00 smithi047 ceph-mon[77147]: [ERR] MDS_DAMAGE: 1 mds daemon damaged 2022-05-14T00:40:00.926 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:40:00 smithi047 ceph-mon[77147]: fs cephfs mds.2 is damaged 2022-05-14T00:40:00.926 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:40:00 smithi047 ceph-mon[77147]: [WRN] MDS_READ_ONLY: 1 MDSs are read only 2022-05-14T00:40:00.926 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:40:00 smithi047 ceph-mon[77147]: mds.l(mds.0): MDS in read-only mode 2022-05-14T00:40:00.926 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:40:00 smithi047 ceph-mon[77147]: [WRN] MDS_SLOW_METADATA_IO: 1 MDSs report slow metadata IOs 2022-05-14T00:40:00.926 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:40:00 smithi047 ceph-mon[77147]: mds.l(mds.0): 4 slow metadata IOs are blocked > 180 secs, oldest blocked for 4633 secs 2022-05-14T00:40:00.927 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:40:00 smithi047 ceph-mon[77147]: [WRN] MON_DOWN: 1/3 mons down, quorum b,c 2022-05-14T00:40:00.927 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:40:00 smithi047 ceph-mon[77147]: mon.a (rank 0) addr [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] is down (out of quorum) 2022-05-14T00:40:00.927 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:40:00 smithi047 ceph-mon[77147]: [WRN] PG_AVAILABILITY: Reduced data availability: 1 pg inactive 2022-05-14T00:40:00.927 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:40:00 smithi047 ceph-mon[77147]: pg 2.1 is stuck inactive for 84m, current state unknown, last acting [] 2022-05-14T00:40:00.928 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:40:00 smithi047 ceph-mon[77147]: [WRN] SLOW_OPS: 9 slow ops, oldest one blocked for 4330 sec, osd.5 has slow ops 2022-05-14T00:40:00.928 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:40:00 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:40:00.928 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:40:00 smithi115 ceph-mon[78593]: Health detail: HEALTH_ERR 1 filesystem is degraded; 1 mds daemon damaged; 1 MDSs are read only; 1 MDSs report slow metadata IOs; 1/3 mons down, quorum b,c; Reduced data availability: 1 pg inactive; 9 slow ops, oldest one blocked for 4330 sec, osd.5 has slow ops 2022-05-14T00:40:00.929 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:40:00 smithi115 ceph-mon[78593]: [WRN] FS_DEGRADED: 1 filesystem is degraded 2022-05-14T00:40:00.929 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:40:00 smithi115 ceph-mon[78593]: fs cephfs is degraded 2022-05-14T00:40:00.929 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:40:00 smithi115 ceph-mon[78593]: [ERR] MDS_DAMAGE: 1 mds daemon damaged 2022-05-14T00:40:00.929 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:40:00 smithi115 ceph-mon[78593]: fs cephfs mds.2 is damaged 2022-05-14T00:40:00.930 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:40:00 smithi115 ceph-mon[78593]: [WRN] MDS_READ_ONLY: 1 MDSs are read only 2022-05-14T00:40:00.930 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:40:00 smithi115 ceph-mon[78593]: mds.l(mds.0): MDS in read-only mode 2022-05-14T00:40:00.930 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:40:00 smithi115 ceph-mon[78593]: [WRN] MDS_SLOW_METADATA_IO: 1 MDSs report slow metadata IOs 2022-05-14T00:40:00.930 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:40:00 smithi115 ceph-mon[78593]: mds.l(mds.0): 4 slow metadata IOs are blocked > 180 secs, oldest blocked for 4633 secs 2022-05-14T00:40:00.930 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:40:00 smithi115 ceph-mon[78593]: [WRN] MON_DOWN: 1/3 mons down, quorum b,c 2022-05-14T00:40:00.931 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:40:00 smithi115 ceph-mon[78593]: mon.a (rank 0) addr [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] is down (out of quorum) 2022-05-14T00:40:00.931 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:40:00 smithi115 ceph-mon[78593]: [WRN] PG_AVAILABILITY: Reduced data availability: 1 pg inactive 2022-05-14T00:40:00.931 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:40:00 smithi115 ceph-mon[78593]: pg 2.1 is stuck inactive for 84m, current state unknown, last acting [] 2022-05-14T00:40:00.931 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:40:00 smithi115 ceph-mon[78593]: [WRN] SLOW_OPS: 9 slow ops, oldest one blocked for 4330 sec, osd.5 has slow ops 2022-05-14T00:40:01.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:40:00 smithi115 conmon[90393]: 2022-05-14T00:40:00.959+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:40:01.674 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:40:01 smithi047 conmon[78521]: 2022-05-14T00:40:01.456+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:40:01.675 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:40:01 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T00:40:01.675 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:40:01 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:40:01.675 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:40:01 smithi047 conmon[78521]: yield 2022-05-14T00:40:01.675 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:40:01 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:40:01.676 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:40:01 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:40:01.676 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:40:01 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:40:01.676 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:40:01 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T00:40:01.676 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:40:01 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:40:01.677 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:40:01 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T00:40:01.677 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:40:01 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:40:01.677 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:40:01 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T00:40:01.677 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:40:01 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:40:01.677 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:40:01 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T00:40:01.678 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:40:01 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:40:01.678 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:40:01 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T00:40:01.678 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:40:01 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:40:01.678 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:40:01 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:40:01.678 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:40:01 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:40:01.947 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:40:01 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:40:01.948 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:40:01 smithi115 ceph-mon[78593]: pgmap v2520: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:40:01.948 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:40:01 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:40:01.948 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:40:01 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:40:01.948 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:40:01 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T00:40:01.949 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:40:01 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T00:40:01.949 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:40:01 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T00:40:02.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:40:01 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:40:02.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:40:01 smithi047 ceph-mon[77147]: pgmap v2520: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:40:02.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:40:01 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:40:02.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:40:01 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:40:02.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:40:01 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T00:40:02.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:40:01 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T00:40:02.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:40:01 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T00:40:02.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:40:01 smithi115 conmon[90393]: 2022-05-14T00:40:01.946+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:40:02.987 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:40:02 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:40:02.987 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:40:02 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:40:02.988 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T00:40:02.988 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:40:02.988 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T00:40:02.989 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:40:02.989 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:40:02.989 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:40:02.989 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T00:40:02.990 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:40:02.990 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T00:40:02.990 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:40:02.990 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T00:40:02.990 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:40:02.991 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T00:40:02.991 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:40:02.991 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T00:40:02.991 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:40:02.992 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:40:02.992 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:40:02.992 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:40:02 smithi115 ceph-mon[78593]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T00:40:03.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:40:02 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:40:03.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:40:02 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:40:03.041 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T00:40:03.042 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:40:03.042 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T00:40:03.042 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:40:03.042 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:40:03.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:40:03.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T00:40:03.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:40:03.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T00:40:03.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:40:03.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T00:40:03.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:40:03.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T00:40:03.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:40:03.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T00:40:03.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:40:03.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:40:03.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:40:03.045 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:40:02 smithi047 ceph-mon[77147]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T00:40:03.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:40:02 smithi115 conmon[90393]: 2022-05-14T00:40:02.987+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:40:03.975 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:40:03 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:40:03.975 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:40:03 smithi115 ceph-mon[78593]: pgmap v2521: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:40:04.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:40:03 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:40:04.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:40:03 smithi047 ceph-mon[77147]: pgmap v2521: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:40:04.362 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:40:03 smithi115 conmon[90393]: 2022-05-14T00:40:03.974+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:40:04.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:40:04 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:40:04.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:40:04 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 4335 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:40:04.792 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:40:04 smithi047 conmon[78521]: 2022-05-14T00:40:04.528+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:40:04.792 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:40:04 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T00:40:04.793 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:40:04 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:40:04.793 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:40:04 smithi047 conmon[78521]: yield 2022-05-14T00:40:04.793 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:40:04 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:40:04.793 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:40:04 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:40:04.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:40:04 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:40:04.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:40:04 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T00:40:04.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:40:04 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:40:04.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:40:04 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T00:40:04.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:40:04 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:40:04.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:40:04 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T00:40:04.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:40:04 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:40:04.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:40:04 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T00:40:04.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:40:04 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:40:04.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:40:04 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T00:40:04.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:40:04 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:40:04.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:40:04 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:40:04.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:40:04 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:40:04.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:40:04 smithi047 conmon[78521]: 2022-05-14T00:40:04.529+0000 7f533f777700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T00:40:04.797 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:40:04 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T00:40:04.797 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:40:04 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:40:04.797 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:40:04 smithi047 conmon[78521]: yield 2022-05-14T00:40:04.797 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:40:04 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:40:04.797 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:40:04 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:40:04.798 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:40:04 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:40:04.798 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:40:04 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T00:40:04.798 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:40:04 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:40:04.798 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:40:04 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T00:40:04.798 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:40:04 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:40:04.799 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:40:04 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T00:40:04.799 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:40:04 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:40:04.799 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:40:04 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T00:40:04.799 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:40:04 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:40:04.800 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:40:04 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T00:40:04.800 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:40:04 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:40:04.800 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:40:04 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:40:04.800 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:40:04 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:40:04.800 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:40:04 smithi047 conmon[78521]: 2022-05-14T00:40:04.801 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:40:04 smithi047 conmon[78521]: During handling of the above exception, another exception occurred: 2022-05-14T00:40:04.801 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:40:04 smithi047 conmon[78521]: 2022-05-14T00:40:04.801 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:40:04 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T00:40:04.801 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:40:04 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T00:40:04.802 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:40:04 smithi047 conmon[78521]: return f(*arg) 2022-05-14T00:40:04.802 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:40:04 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T00:40:04.802 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:40:04 smithi047 conmon[78521]: self._write_client_files(client_files, host) 2022-05-14T00:40:04.802 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:40:04 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T00:40:04.802 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:40:04 smithi047 conmon[78521]: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T00:40:04.803 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:40:04 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T00:40:04.803 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:40:04 smithi047 conmon[78521]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T00:40:04.803 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:40:04 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T00:40:04.803 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:40:04 smithi047 conmon[78521]: return self.event_loop.get_result(coro) 2022-05-14T00:40:04.803 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:40:04 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T00:40:04.804 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:40:04 smithi047 conmon[78521]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T00:40:04.804 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:40:04 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T00:40:04.804 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:40:04 smithi047 conmon[78521]: return self.__get_result() 2022-05-14T00:40:04.804 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:40:04 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T00:40:04.805 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:40:04 smithi047 conmon[78521]: raise self._exception 2022-05-14T00:40:04.805 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:40:04 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T00:40:04.805 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:40:04 smithi047 conmon[78521]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T00:40:04.805 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:40:04 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T00:40:04.805 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:40:04 smithi047 conmon[78521]: conn = await self._remote_connection(host, addr) 2022-05-14T00:40:04.806 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:40:04 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T00:40:04.806 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:40:04 smithi047 conmon[78521]: raise 2022-05-14T00:40:04.806 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:40:04 smithi047 conmon[78521]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T00:40:04.806 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:40:04 smithi047 conmon[78521]: self.gen.throw(type, value, traceback) 2022-05-14T00:40:04.807 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:40:04 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T00:40:04.807 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:40:04 smithi047 conmon[78521]: raise OrchestratorError(msg) 2022-05-14T00:40:04.807 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:40:04 smithi047 conmon[78521]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:40:04.964 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:40:04 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:40:04.964 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:40:04 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 4335 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:40:05.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:40:04 smithi115 conmon[90393]: 2022-05-14T00:40:04.963+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:40:05.962 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:40:05 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:40:05.962 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:40:05 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:40:05.963 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T00:40:05.963 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:40:05.963 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T00:40:05.963 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:40:05.963 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:40:05.964 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:40:05.964 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T00:40:05.964 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:40:05.964 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T00:40:05.964 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:40:05.965 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T00:40:05.965 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:40:05.965 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T00:40:05.966 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:40:05.966 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T00:40:05.967 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:40:05.967 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:40:05.967 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:40:05.967 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:40:05 smithi115 ceph-mon[78593]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T00:40:05.967 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T00:40:05.968 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:40:05.968 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T00:40:05.968 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:40:05.968 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:40:05.968 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:40:05.969 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T00:40:05.969 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:40:05.969 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T00:40:05.969 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:40:05.970 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T00:40:05.970 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:40:05.970 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T00:40:05.970 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:40:05.971 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T00:40:05.971 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:40:05.971 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:40:05.971 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:40:05.971 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T00:40:05.972 INFO:journalctl@ceph.mon.c.smithi115.stdout: During handling of the above exception, another exception occurred: 2022-05-14T00:40:05.972 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T00:40:05.972 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T00:40:05.972 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T00:40:05.972 INFO:journalctl@ceph.mon.c.smithi115.stdout: return f(*arg) 2022-05-14T00:40:05.973 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T00:40:05.973 INFO:journalctl@ceph.mon.c.smithi115.stdout: self._write_client_files(client_files, host) 2022-05-14T00:40:05.973 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T00:40:05.973 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T00:40:05.974 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T00:40:05.974 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T00:40:05.974 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T00:40:05.974 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.event_loop.get_result(coro) 2022-05-14T00:40:05.975 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T00:40:05.975 INFO:journalctl@ceph.mon.c.smithi115.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T00:40:05.975 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T00:40:05.975 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.__get_result() 2022-05-14T00:40:05.976 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T00:40:05.976 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise self._exception 2022-05-14T00:40:05.976 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T00:40:05.976 INFO:journalctl@ceph.mon.c.smithi115.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T00:40:05.977 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T00:40:05.977 INFO:journalctl@ceph.mon.c.smithi115.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T00:40:05.977 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T00:40:05.977 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise 2022-05-14T00:40:05.978 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T00:40:05.978 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.gen.throw(type, value, traceback) 2022-05-14T00:40:05.978 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T00:40:05.978 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OrchestratorError(msg) 2022-05-14T00:40:05.979 INFO:journalctl@ceph.mon.c.smithi115.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:40:05.979 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:40:05 smithi115 ceph-mon[78593]: pgmap v2522: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:40:06.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:40:05 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:40:06.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:40:05 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:40:06.042 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T00:40:06.042 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:40:06.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T00:40:06.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:40:06.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:40:06.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:40:06.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T00:40:06.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:40:06.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T00:40:06.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:40:06.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T00:40:06.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:40:06.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T00:40:06.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:40:06.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T00:40:06.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:40:06.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:40:06.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:40:06.046 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:40:05 smithi047 ceph-mon[77147]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T00:40:06.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T00:40:06.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:40:06.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T00:40:06.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:40:06.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:40:06.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:40:06.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T00:40:06.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:40:06.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T00:40:06.049 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:40:06.049 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T00:40:06.049 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:40:06.049 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T00:40:06.049 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:40:06.050 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T00:40:06.050 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:40:06.050 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:40:06.050 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:40:06.051 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T00:40:06.051 INFO:journalctl@ceph.mon.b.smithi047.stdout: During handling of the above exception, another exception occurred: 2022-05-14T00:40:06.051 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T00:40:06.051 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T00:40:06.051 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T00:40:06.052 INFO:journalctl@ceph.mon.b.smithi047.stdout: return f(*arg) 2022-05-14T00:40:06.052 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T00:40:06.052 INFO:journalctl@ceph.mon.b.smithi047.stdout: self._write_client_files(client_files, host) 2022-05-14T00:40:06.053 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T00:40:06.053 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T00:40:06.053 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T00:40:06.054 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T00:40:06.054 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T00:40:06.054 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.event_loop.get_result(coro) 2022-05-14T00:40:06.054 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T00:40:06.054 INFO:journalctl@ceph.mon.b.smithi047.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T00:40:06.055 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T00:40:06.055 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.__get_result() 2022-05-14T00:40:06.055 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T00:40:06.055 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise self._exception 2022-05-14T00:40:06.055 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T00:40:06.056 INFO:journalctl@ceph.mon.b.smithi047.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T00:40:06.056 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T00:40:06.056 INFO:journalctl@ceph.mon.b.smithi047.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T00:40:06.056 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T00:40:06.056 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise 2022-05-14T00:40:06.057 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T00:40:06.057 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.gen.throw(type, value, traceback) 2022-05-14T00:40:06.058 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T00:40:06.058 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OrchestratorError(msg) 2022-05-14T00:40:06.058 INFO:journalctl@ceph.mon.b.smithi047.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:40:06.058 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:40:05 smithi047 ceph-mon[77147]: pgmap v2522: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:40:06.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:40:05 smithi115 conmon[90393]: 2022-05-14T00:40:05.959+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:40:07.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:40:06 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:40:07.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:40:06 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:40:07.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:40:06 smithi115 conmon[90393]: 2022-05-14T00:40:06.923+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:40:08.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:40:07 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:40:08.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:40:07 smithi047 ceph-mon[77147]: pgmap v2523: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:40:08.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:40:07 smithi115 conmon[90393]: 2022-05-14T00:40:07.906+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:40:08.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:40:07 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:40:08.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:40:07 smithi115 ceph-mon[78593]: pgmap v2523: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:40:09.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:40:08 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:40:09.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:40:08 smithi115 conmon[90393]: 2022-05-14T00:40:08.906+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:40:09.257 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:40:08 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:40:10.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:40:09 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:40:10.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:40:09 smithi047 ceph-mon[77147]: pgmap v2524: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:40:10.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:40:09 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:40:10.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:40:09 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:40:10.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:40:09 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 4340 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:40:10.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:40:09 smithi115 conmon[90393]: 2022-05-14T00:40:09.907+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:40:10.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:40:09 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:40:10.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:40:09 smithi115 ceph-mon[78593]: pgmap v2524: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:40:10.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:40:09 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:40:10.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:40:09 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:40:10.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:40:09 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 4340 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:40:11.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:40:10 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:40:11.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:40:10 smithi115 conmon[90393]: 2022-05-14T00:40:10.858+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:40:11.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:40:10 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:40:12.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:40:11 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:40:12.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:40:11 smithi047 ceph-mon[77147]: pgmap v2525: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:40:12.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:40:11 smithi115 conmon[90393]: 2022-05-14T00:40:11.876+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:40:12.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:40:11 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:40:12.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:40:11 smithi115 ceph-mon[78593]: pgmap v2525: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:40:13.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:40:12 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:40:13.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:40:12 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:40:13.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:40:12 smithi115 conmon[90393]: 2022-05-14T00:40:12.877+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:40:14.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:40:13 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:40:14.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:40:13 smithi047 ceph-mon[77147]: pgmap v2526: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:40:14.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:40:13 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:40:14.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:40:13 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:40:14.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:40:13 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T00:40:14.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:40:13 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T00:40:14.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:40:13 smithi115 conmon[90393]: 2022-05-14T00:40:13.916+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:40:14.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:40:13 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:40:14.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:40:13 smithi115 ceph-mon[78593]: pgmap v2526: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:40:14.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:40:13 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:40:14.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:40:13 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:40:14.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:40:13 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T00:40:14.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:40:13 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T00:40:15.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:40:14 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:40:15.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:40:14 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 4345 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:40:15.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:40:14 smithi115 conmon[90393]: 2022-05-14T00:40:14.927+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:40:15.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:40:14 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:40:15.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:40:14 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 4345 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:40:15.970 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:40:15 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:40:15.970 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:40:15 smithi115 ceph-mon[78593]: pgmap v2527: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:40:16.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:40:15 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:40:16.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:40:15 smithi047 ceph-mon[77147]: pgmap v2527: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:40:16.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:40:15 smithi115 conmon[90393]: 2022-05-14T00:40:15.968+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:40:17.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:40:16 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:40:17.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:40:16 smithi115 conmon[90393]: 2022-05-14T00:40:16.922+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:40:17.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:40:16 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:40:18.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:40:17 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:40:18.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:40:17 smithi047 ceph-mon[77147]: pgmap v2528: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:40:18.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:40:17 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:40:18.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:40:17 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:40:18.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:40:17 smithi115 conmon[90393]: 2022-05-14T00:40:17.918+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:40:18.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:40:17 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:40:18.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:40:17 smithi115 ceph-mon[78593]: pgmap v2528: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:40:18.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:40:17 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:40:18.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:40:17 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:40:19.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:40:18 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:40:19.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:40:18 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:40:19.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:40:18 smithi115 conmon[90393]: 2022-05-14T00:40:18.960+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:40:20.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:40:19 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:40:20.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:40:19 smithi047 ceph-mon[77147]: pgmap v2529: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:40:20.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:40:19 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 4350 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:40:20.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:40:19 smithi115 conmon[90393]: 2022-05-14T00:40:19.916+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:40:20.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:40:19 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:40:20.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:40:19 smithi115 ceph-mon[78593]: pgmap v2529: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:40:20.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:40:19 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 4350 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:40:21.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:40:20 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:40:21.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:40:20 smithi115 conmon[90393]: 2022-05-14T00:40:20.874+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:40:21.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:40:20 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:40:22.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:40:21 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:40:22.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:40:21 smithi047 ceph-mon[77147]: pgmap v2530: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:40:22.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:40:21 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:40:22.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:40:21 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:40:22.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:40:21 smithi115 conmon[90393]: 2022-05-14T00:40:21.892+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:40:22.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:40:21 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:40:22.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:40:21 smithi115 ceph-mon[78593]: pgmap v2530: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:40:22.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:40:21 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:40:22.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:40:21 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:40:23.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:40:22 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:40:23.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:40:22 smithi115 conmon[90393]: 2022-05-14T00:40:22.901+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:40:23.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:40:22 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:40:24.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:40:23 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:40:24.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:40:23 smithi047 ceph-mon[77147]: pgmap v2531: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:40:24.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:40:23 smithi115 conmon[90393]: 2022-05-14T00:40:23.874+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:40:24.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:40:23 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:40:24.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:40:23 smithi115 ceph-mon[78593]: pgmap v2531: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:40:25.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:40:24 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:40:25.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:40:24 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 4355 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:40:25.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:40:24 smithi115 conmon[90393]: 2022-05-14T00:40:24.881+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:40:25.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:40:24 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:40:25.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:40:24 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 4355 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:40:26.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:40:25 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:40:26.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:40:25 smithi047 ceph-mon[77147]: pgmap v2532: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:40:26.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:40:25 smithi115 conmon[90393]: 2022-05-14T00:40:25.901+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:40:26.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:40:25 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:40:26.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:40:25 smithi115 ceph-mon[78593]: pgmap v2532: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:40:27.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:40:26 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:40:27.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:40:26 smithi115 conmon[90393]: 2022-05-14T00:40:26.880+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:40:27.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:40:26 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:40:28.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:40:27 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:40:28.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:40:27 smithi047 ceph-mon[77147]: pgmap v2533: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:40:28.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:40:27 smithi115 conmon[90393]: 2022-05-14T00:40:27.922+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:40:28.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:40:27 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:40:28.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:40:27 smithi115 ceph-mon[78593]: pgmap v2533: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:40:29.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:40:28 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:40:29.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:40:28 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:40:29.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:40:28 smithi115 conmon[90393]: 2022-05-14T00:40:28.891+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:40:30.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:40:29 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:40:30.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:40:29 smithi047 ceph-mon[77147]: pgmap v2534: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:40:30.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:40:29 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:40:30.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:40:29 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:40:30.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:40:29 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 4360 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:40:30.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:40:29 smithi115 conmon[90393]: 2022-05-14T00:40:29.906+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:40:30.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:40:29 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:40:30.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:40:29 smithi115 ceph-mon[78593]: pgmap v2534: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:40:30.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:40:29 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:40:30.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:40:29 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:40:30.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:40:29 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 4360 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:40:31.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:40:30 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:40:31.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:40:30 smithi115 conmon[90393]: 2022-05-14T00:40:30.865+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:40:31.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:40:30 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:40:32.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:40:31 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:40:32.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:40:31 smithi047 ceph-mon[77147]: pgmap v2535: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:40:32.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:40:31 smithi115 conmon[90393]: 2022-05-14T00:40:31.875+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:40:32.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:40:31 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:40:32.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:40:31 smithi115 ceph-mon[78593]: pgmap v2535: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:40:33.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:40:32 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:40:33.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:40:32 smithi115 conmon[90393]: 2022-05-14T00:40:32.861+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:40:33.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:40:32 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:40:34.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:40:33 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:40:34.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:40:33 smithi047 ceph-mon[77147]: pgmap v2536: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:40:34.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:40:33 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:40:34.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:40:33 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:40:34.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:40:33 smithi115 conmon[90393]: 2022-05-14T00:40:33.905+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:40:34.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:40:33 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:40:34.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:40:33 smithi115 ceph-mon[78593]: pgmap v2536: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:40:34.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:40:33 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:40:34.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:40:33 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:40:35.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:40:34 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:40:35.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:40:34 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 4365 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:40:35.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:40:34 smithi115 conmon[90393]: 2022-05-14T00:40:34.864+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:40:35.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:40:34 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:40:35.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:40:34 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 4365 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:40:36.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:40:35 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:40:36.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:40:35 smithi047 ceph-mon[77147]: pgmap v2537: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:40:36.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:40:35 smithi115 conmon[90393]: 2022-05-14T00:40:35.825+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:40:36.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:40:35 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:40:36.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:40:35 smithi115 ceph-mon[78593]: pgmap v2537: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:40:37.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:40:36 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:40:37.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:40:36 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:40:37.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:40:36 smithi115 conmon[90393]: 2022-05-14T00:40:36.831+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:40:37.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:40:36 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:40:37.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:40:36 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:40:38.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:40:37 smithi047 ceph-mon[77147]: pgmap v2538: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:40:38.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:40:37 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:40:38.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:40:37 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:40:38.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:40:37 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:40:38.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:40:37 smithi115 conmon[90393]: 2022-05-14T00:40:37.824+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:40:38.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:40:37 smithi115 ceph-mon[78593]: pgmap v2538: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:40:38.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:40:37 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:40:38.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:40:37 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:40:38.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:40:37 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:40:39.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:40:38 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:40:39.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:40:38 smithi115 conmon[90393]: 2022-05-14T00:40:38.832+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:40:39.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:40:38 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:40:40.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:40:39 smithi047 ceph-mon[77147]: pgmap v2539: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:40:40.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:40:39 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:40:40.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:40:39 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 4370 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:40:40.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:40:39 smithi115 conmon[90393]: 2022-05-14T00:40:39.851+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:40:40.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:40:39 smithi115 ceph-mon[78593]: pgmap v2539: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:40:40.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:40:39 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:40:40.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:40:39 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 4370 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:40:41.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:40:40 smithi115 conmon[90393]: 2022-05-14T00:40:40.837+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:40:42.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:40:41 smithi115 conmon[90393]: 2022-05-14T00:40:41.826+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:40:42.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:40:41 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:40:42.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:40:41 smithi115 ceph-mon[78593]: pgmap v2540: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:40:42.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:40:41 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:40:42.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:40:41 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:40:42.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:40:41 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:40:42.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:40:41 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:40:42.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:40:41 smithi047 ceph-mon[77147]: pgmap v2540: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:40:42.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:40:41 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:40:42.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:40:41 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:40:42.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:40:41 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:40:43.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:40:42 smithi115 conmon[90393]: 2022-05-14T00:40:42.835+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:40:43.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:40:42 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:40:43.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:40:42 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:40:44.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:40:43 smithi115 conmon[90393]: 2022-05-14T00:40:43.824+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:40:44.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:40:43 smithi115 ceph-mon[78593]: pgmap v2541: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:40:44.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:40:43 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:40:44.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:40:43 smithi047 ceph-mon[77147]: pgmap v2541: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:40:44.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:40:43 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:40:45.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:40:44 smithi115 conmon[90393]: 2022-05-14T00:40:44.854+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:40:45.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:40:44 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:40:45.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:40:44 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 4375 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:40:45.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:40:44 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:40:45.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:40:44 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 4375 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:40:46.117 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:40:45 smithi115 conmon[90393]: 2022-05-14T00:40:45.861+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:40:46.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:40:46 smithi047 ceph-mon[77147]: pgmap v2542: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:40:46.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:40:46 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:40:46.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:40:46 smithi115 ceph-mon[78593]: pgmap v2542: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:40:46.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:40:46 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:40:47.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:40:46 smithi115 conmon[90393]: 2022-05-14T00:40:46.905+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:40:47.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:40:47 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:40:47.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:40:47 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:40:48.122 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:40:47 smithi115 conmon[90393]: 2022-05-14T00:40:47.865+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:40:48.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:40:48 smithi047 ceph-mon[77147]: pgmap v2543: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:40:48.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:40:48 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:40:48.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:40:48 smithi115 ceph-mon[78593]: pgmap v2543: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:40:48.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:40:48 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:40:49.126 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:40:48 smithi115 conmon[90393]: 2022-05-14T00:40:48.860+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:40:49.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:40:49 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:40:49.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:40:49 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 4380 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:40:49.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:40:49 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:40:49.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:40:49 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 4380 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:40:50.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:40:49 smithi115 conmon[90393]: 2022-05-14T00:40:49.881+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:40:50.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:40:50 smithi115 ceph-mon[78593]: pgmap v2544: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:40:50.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:40:50 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:40:50.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:40:50 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:40:50.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:40:50 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:40:50.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:40:50 smithi047 ceph-mon[77147]: pgmap v2544: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:40:50.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:40:50 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:40:50.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:40:50 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:40:50.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:40:50 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:40:51.134 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:40:50 smithi115 conmon[90393]: 2022-05-14T00:40:50.920+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:40:51.134 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:40:51 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:40:51.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:40:51 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:40:52.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:40:52 smithi115 ceph-mon[78593]: pgmap v2545: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:40:52.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:40:52 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:40:52.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:40:51 smithi115 conmon[90393]: 2022-05-14T00:40:51.953+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:40:52.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:40:52 smithi047 ceph-mon[77147]: pgmap v2545: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:40:52.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:40:52 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:40:53.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:40:53 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:40:53.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:40:52 smithi115 conmon[90393]: 2022-05-14T00:40:52.946+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:40:53.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:40:53 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:40:54.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:40:53 smithi115 conmon[90393]: 2022-05-14T00:40:53.991+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:40:54.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:40:54 smithi115 ceph-mon[78593]: pgmap v2546: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:40:54.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:40:54 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:40:54.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:40:54 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:40:54.362 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:40:54 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:40:54.362 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:40:54 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 4385 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:40:54.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:40:54 smithi047 ceph-mon[77147]: pgmap v2546: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:40:54.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:40:54 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:40:54.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:40:54 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:40:54.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:40:54 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:40:54.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:40:54 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 4385 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:40:55.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:40:55 smithi115 conmon[90393]: 2022-05-14T00:40:55.040+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:40:55.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:40:55 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:40:55.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:40:55 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:40:56.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:40:56 smithi115 conmon[90393]: 2022-05-14T00:40:56.066+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:40:56.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:40:56 smithi115 ceph-mon[78593]: pgmap v2547: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:40:56.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:40:56 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:40:56.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:40:56 smithi047 ceph-mon[77147]: pgmap v2547: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:40:56.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:40:56 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:40:57.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:40:57 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:40:57.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:40:57 smithi115 conmon[90393]: 2022-05-14T00:40:57.110+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:40:57.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:40:57 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:40:58.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:40:58 smithi115 conmon[90393]: 2022-05-14T00:40:58.062+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:40:58.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:40:58 smithi115 ceph-mon[78593]: pgmap v2548: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:40:58.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:40:58 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:40:58.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:40:58 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:40:58.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:40:58 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:40:58.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:40:58 smithi047 ceph-mon[77147]: pgmap v2548: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:40:58.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:40:58 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:40:58.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:40:58 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:40:58.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:40:58 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:40:59.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:40:59 smithi115 conmon[90393]: 2022-05-14T00:40:59.077+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:40:59.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:40:59 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:40:59.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:40:59 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 4390 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:40:59.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:40:59 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:40:59.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:40:59 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 4390 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:41:00.362 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:41:00 smithi115 conmon[90393]: 2022-05-14T00:41:00.052+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:41:00.363 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:41:00 smithi115 ceph-mon[78593]: pgmap v2549: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:41:00.363 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:41:00 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:41:00.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:41:00 smithi047 ceph-mon[77147]: pgmap v2549: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:41:00.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:41:00 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:41:01.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:41:01 smithi115 conmon[90393]: 2022-05-14T00:41:01.037+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:41:01.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:41:01 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:41:01.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:41:01 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:41:02.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:41:02 smithi115 conmon[90393]: 2022-05-14T00:41:02.024+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:41:02.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:41:02 smithi115 ceph-mon[78593]: pgmap v2550: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:41:02.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:41:02 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:41:02.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:41:02 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:41:02.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:41:02 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:41:02.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:41:02 smithi047 ceph-mon[77147]: pgmap v2550: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:41:02.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:41:02 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:41:02.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:41:02 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:41:02.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:41:02 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:41:03.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:41:03 smithi115 conmon[90393]: 2022-05-14T00:41:03.005+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:41:03.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:41:03 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:41:03.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:41:03 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:41:04.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:41:03 smithi115 conmon[90393]: 2022-05-14T00:41:03.972+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:41:04.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:41:04 smithi115 ceph-mon[78593]: pgmap v2551: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:41:04.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:41:04 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:41:04.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:41:04 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 4395 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:41:04.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:41:04 smithi047 ceph-mon[77147]: pgmap v2551: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:41:04.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:41:04 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:41:04.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:41:04 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 4395 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:41:05.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:41:04 smithi115 conmon[90393]: 2022-05-14T00:41:04.954+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:41:05.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:41:05 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:41:05.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:41:05 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T00:41:05.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:41:05 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T00:41:05.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:41:05 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T00:41:05.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:41:05 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:41:05.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:41:05 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T00:41:05.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:41:05 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T00:41:05.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:41:05 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T00:41:06.179 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:41:05 smithi115 conmon[90393]: 2022-05-14T00:41:05.905+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:41:06.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:41:06 smithi047 ceph-mon[77147]: pgmap v2552: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:41:06.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:41:06 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:41:06.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:41:06 smithi115 ceph-mon[78593]: pgmap v2552: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:41:06.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:41:06 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:41:07.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:41:06 smithi115 conmon[90393]: 2022-05-14T00:41:06.944+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:41:07.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:41:07 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:41:07.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:41:07 smithi115 ceph-mon[78593]: pgmap v2553: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:41:07.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:41:07 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:41:07.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:41:07 smithi047 ceph-mon[77147]: pgmap v2553: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:41:08.041 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:41:07 smithi047 conmon[78521]: 2022-05-14T00:41:07.631+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:41:08.041 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:41:07 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T00:41:08.041 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:41:07 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:41:08.042 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:41:07 smithi047 conmon[78521]: yield 2022-05-14T00:41:08.042 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:41:07 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:41:08.042 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:41:07 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:41:08.042 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:41:07 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:41:08.042 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:41:07 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T00:41:08.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:41:07 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:41:08.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:41:07 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T00:41:08.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:41:07 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:41:08.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:41:07 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T00:41:08.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:41:07 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:41:08.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:41:07 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T00:41:08.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:41:07 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:41:08.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:41:07 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T00:41:08.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:41:07 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:41:08.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:41:07 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:41:08.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:41:07 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:41:08.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:41:07 smithi115 conmon[90393]: 2022-05-14T00:41:07.961+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:41:08.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:41:08 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:41:08.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:41:08 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:41:08.359 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T00:41:08.359 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:41:08.359 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T00:41:08.359 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:41:08.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:41:08.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:41:08.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T00:41:08.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:41:08.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T00:41:08.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:41:08.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T00:41:08.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:41:08.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T00:41:08.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:41:08.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T00:41:08.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:41:08.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:41:08.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:41:08.363 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:41:08 smithi115 ceph-mon[78593]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T00:41:08.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:41:08 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:41:08.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:41:08 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:41:08.542 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T00:41:08.542 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:41:08.542 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T00:41:08.542 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:41:08.542 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:41:08.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:41:08.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T00:41:08.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:41:08.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T00:41:08.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:41:08.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T00:41:08.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:41:08.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T00:41:08.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:41:08.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T00:41:08.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:41:08.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:41:08.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:41:08.547 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:41:08 smithi047 ceph-mon[77147]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T00:41:09.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:41:09 smithi115 conmon[90393]: 2022-05-14T00:41:09.011+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:41:09.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:41:09 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:41:09.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:41:09 smithi115 ceph-mon[78593]: pgmap v2554: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:41:09.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:41:09 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 4400 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:41:09.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:41:09 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:41:09.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:41:09 smithi047 ceph-mon[77147]: pgmap v2554: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:41:09.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:41:09 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 4400 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:41:10.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:41:10 smithi115 conmon[90393]: 2022-05-14T00:41:10.059+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:41:10.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:41:10 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:41:10.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:41:10 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:41:10.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:41:10 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:41:10.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:41:10 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:41:10.545 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:41:10 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:41:10.546 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:41:10 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:41:11.041 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:41:10 smithi047 conmon[78521]: 2022-05-14T00:41:10.703+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:41:11.042 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:41:10 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T00:41:11.042 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:41:10 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:41:11.042 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:41:10 smithi047 conmon[78521]: yield 2022-05-14T00:41:11.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:41:10 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:41:11.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:41:10 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:41:11.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:41:10 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:41:11.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:41:10 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T00:41:11.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:41:10 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:41:11.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:41:10 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T00:41:11.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:41:10 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:41:11.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:41:10 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T00:41:11.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:41:10 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:41:11.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:41:10 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T00:41:11.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:41:10 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:41:11.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:41:10 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T00:41:11.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:41:10 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:41:11.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:41:10 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:41:11.048 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:41:10 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:41:11.048 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:41:10 smithi047 conmon[78521]: 2022-05-14T00:41:10.704+0000 7f5340f7a700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T00:41:11.048 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:41:10 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T00:41:11.048 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:41:10 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:41:11.048 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:41:10 smithi047 conmon[78521]: yield 2022-05-14T00:41:11.049 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:41:10 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:41:11.049 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:41:10 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:41:11.049 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:41:10 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:41:11.049 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:41:10 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T00:41:11.049 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:41:10 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:41:11.050 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:41:10 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T00:41:11.050 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:41:10 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:41:11.050 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:41:10 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T00:41:11.050 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:41:10 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:41:11.051 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:41:10 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T00:41:11.051 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:41:10 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:41:11.051 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:41:10 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T00:41:11.051 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:41:10 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:41:11.051 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:41:10 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:41:11.052 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:41:10 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:41:11.052 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:41:10 smithi047 conmon[78521]: 2022-05-14T00:41:11.052 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:41:10 smithi047 conmon[78521]: During handling of the above exception, another exception occurred: 2022-05-14T00:41:11.052 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:41:10 smithi047 conmon[78521]: 2022-05-14T00:41:11.053 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:41:10 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T00:41:11.053 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:41:10 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T00:41:11.053 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:41:10 smithi047 conmon[78521]: return f(*arg) 2022-05-14T00:41:11.053 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:41:10 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T00:41:11.053 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:41:10 smithi047 conmon[78521]: self._write_client_files(client_files, host) 2022-05-14T00:41:11.054 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:41:10 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T00:41:11.054 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:41:10 smithi047 conmon[78521]: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T00:41:11.054 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:41:10 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T00:41:11.054 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:41:10 smithi047 conmon[78521]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T00:41:11.055 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:41:10 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T00:41:11.055 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:41:10 smithi047 conmon[78521]: return self.event_loop.get_result(coro) 2022-05-14T00:41:11.055 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:41:10 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T00:41:11.055 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:41:10 smithi047 conmon[78521]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T00:41:11.055 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:41:10 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T00:41:11.056 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:41:10 smithi047 conmon[78521]: return self.__get_result() 2022-05-14T00:41:11.056 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:41:10 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T00:41:11.056 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:41:10 smithi047 conmon[78521]: raise self._exception 2022-05-14T00:41:11.056 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:41:10 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T00:41:11.056 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:41:10 smithi047 conmon[78521]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T00:41:11.057 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:41:10 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T00:41:11.057 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:41:10 smithi047 conmon[78521]: conn = await self._remote_connection(host, addr) 2022-05-14T00:41:11.057 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:41:10 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T00:41:11.057 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:41:10 smithi047 conmon[78521]: raise 2022-05-14T00:41:11.058 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:41:10 smithi047 conmon[78521]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T00:41:11.058 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:41:10 smithi047 conmon[78521]: self.gen.throw(type, value, traceback) 2022-05-14T00:41:11.058 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:41:10 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T00:41:11.058 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:41:10 smithi047 conmon[78521]: raise OrchestratorError(msg) 2022-05-14T00:41:11.058 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:41:10 smithi047 conmon[78521]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:41:11.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:41:11 smithi115 conmon[90393]: 2022-05-14T00:41:11.016+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:41:11.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:41:11 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:41:11.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:41:11 smithi115 ceph-mon[78593]: pgmap v2555: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:41:11.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:41:11 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:41:11.359 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T00:41:11.359 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:41:11.359 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T00:41:11.359 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:41:11.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:41:11.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:41:11.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T00:41:11.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:41:11.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T00:41:11.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:41:11.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T00:41:11.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:41:11.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T00:41:11.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:41:11.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T00:41:11.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:41:11.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:41:11.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:41:11.363 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:41:11 smithi115 ceph-mon[78593]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T00:41:11.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T00:41:11.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:41:11.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T00:41:11.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:41:11.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:41:11.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:41:11.365 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T00:41:11.365 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:41:11.365 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T00:41:11.365 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:41:11.365 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T00:41:11.366 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:41:11.366 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T00:41:11.366 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:41:11.367 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T00:41:11.367 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:41:11.367 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:41:11.367 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:41:11.368 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T00:41:11.368 INFO:journalctl@ceph.mon.c.smithi115.stdout: During handling of the above exception, another exception occurred: 2022-05-14T00:41:11.368 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T00:41:11.368 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T00:41:11.369 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T00:41:11.369 INFO:journalctl@ceph.mon.c.smithi115.stdout: return f(*arg) 2022-05-14T00:41:11.369 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T00:41:11.370 INFO:journalctl@ceph.mon.c.smithi115.stdout: self._write_client_files(client_files, host) 2022-05-14T00:41:11.370 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T00:41:11.370 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T00:41:11.371 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T00:41:11.371 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T00:41:11.371 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T00:41:11.372 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.event_loop.get_result(coro) 2022-05-14T00:41:11.372 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T00:41:11.372 INFO:journalctl@ceph.mon.c.smithi115.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T00:41:11.372 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T00:41:11.372 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.__get_result() 2022-05-14T00:41:11.373 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T00:41:11.373 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise self._exception 2022-05-14T00:41:11.373 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T00:41:11.373 INFO:journalctl@ceph.mon.c.smithi115.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T00:41:11.374 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T00:41:11.374 INFO:journalctl@ceph.mon.c.smithi115.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T00:41:11.374 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T00:41:11.374 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise 2022-05-14T00:41:11.375 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T00:41:11.375 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.gen.throw(type, value, traceback) 2022-05-14T00:41:11.375 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T00:41:11.375 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OrchestratorError(msg) 2022-05-14T00:41:11.376 INFO:journalctl@ceph.mon.c.smithi115.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:41:11.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:41:11 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:41:11.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:41:11 smithi047 ceph-mon[77147]: pgmap v2555: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:41:11.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:41:11 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:41:11.542 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T00:41:11.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:41:11.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T00:41:11.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:41:11.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:41:11.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:41:11.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T00:41:11.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:41:11.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T00:41:11.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:41:11.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T00:41:11.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:41:11.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T00:41:11.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:41:11.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T00:41:11.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:41:11.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:41:11.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:41:11.547 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:41:11 smithi047 ceph-mon[77147]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T00:41:11.548 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T00:41:11.548 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:41:11.548 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T00:41:11.549 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:41:11.549 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:41:11.549 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:41:11.549 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T00:41:11.550 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:41:11.550 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T00:41:11.550 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:41:11.550 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T00:41:11.550 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:41:11.551 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T00:41:11.551 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:41:11.551 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T00:41:11.551 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:41:11.552 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:41:11.552 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:41:11.552 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T00:41:11.552 INFO:journalctl@ceph.mon.b.smithi047.stdout: During handling of the above exception, another exception occurred: 2022-05-14T00:41:11.553 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T00:41:11.553 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T00:41:11.553 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T00:41:11.554 INFO:journalctl@ceph.mon.b.smithi047.stdout: return f(*arg) 2022-05-14T00:41:11.554 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T00:41:11.555 INFO:journalctl@ceph.mon.b.smithi047.stdout: self._write_client_files(client_files, host) 2022-05-14T00:41:11.555 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T00:41:11.555 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T00:41:11.555 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T00:41:11.556 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T00:41:11.556 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T00:41:11.556 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.event_loop.get_result(coro) 2022-05-14T00:41:11.556 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T00:41:11.557 INFO:journalctl@ceph.mon.b.smithi047.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T00:41:11.557 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T00:41:11.557 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.__get_result() 2022-05-14T00:41:11.557 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T00:41:11.558 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise self._exception 2022-05-14T00:41:11.558 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T00:41:11.558 INFO:journalctl@ceph.mon.b.smithi047.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T00:41:11.558 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T00:41:11.558 INFO:journalctl@ceph.mon.b.smithi047.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T00:41:11.559 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T00:41:11.559 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise 2022-05-14T00:41:11.559 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T00:41:11.559 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.gen.throw(type, value, traceback) 2022-05-14T00:41:11.560 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T00:41:11.560 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OrchestratorError(msg) 2022-05-14T00:41:11.560 INFO:journalctl@ceph.mon.b.smithi047.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:41:12.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:41:11 smithi115 conmon[90393]: 2022-05-14T00:41:11.996+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:41:12.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:41:12 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:41:12.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:41:12 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:41:13.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:41:12 smithi115 conmon[90393]: 2022-05-14T00:41:12.980+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:41:13.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:41:13 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:41:13.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:41:13 smithi115 ceph-mon[78593]: pgmap v2556: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:41:13.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:41:13 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:41:13.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:41:13 smithi047 ceph-mon[77147]: pgmap v2556: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:41:14.208 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:41:13 smithi115 conmon[90393]: 2022-05-14T00:41:13.942+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:41:14.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:41:14 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:41:14.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:41:14 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:41:14.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:41:14 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:41:14.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:41:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T00:41:14.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:41:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T00:41:14.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:41:14 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 4405 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:41:14.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:41:14 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:41:14.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:41:14 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:41:14.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:41:14 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:41:14.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:41:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T00:41:14.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:41:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T00:41:14.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:41:14 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 4405 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:41:15.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:41:14 smithi115 conmon[90393]: 2022-05-14T00:41:14.988+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:41:15.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:41:15 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:41:15.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:41:15 smithi115 ceph-mon[78593]: pgmap v2557: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:41:15.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:41:15 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:41:15.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:41:15 smithi047 ceph-mon[77147]: pgmap v2557: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:41:16.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:41:15 smithi115 conmon[90393]: 2022-05-14T00:41:15.974+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:41:16.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:41:16 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:41:16.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:41:16 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:41:17.218 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:41:16 smithi115 conmon[90393]: 2022-05-14T00:41:16.944+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:41:17.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:41:17 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:41:17.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:41:17 smithi047 ceph-mon[77147]: pgmap v2558: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:41:17.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:41:17 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:41:17.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:41:17 smithi115 ceph-mon[78593]: pgmap v2558: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:41:18.219 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:41:17 smithi115 conmon[90393]: 2022-05-14T00:41:17.912+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:41:18.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:41:18 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:41:18.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:41:18 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:41:18.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:41:18 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:41:18.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:41:18 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:41:18.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:41:18 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:41:18.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:41:18 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:41:19.223 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:41:18 smithi115 conmon[90393]: 2022-05-14T00:41:18.879+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:41:19.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:41:19 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:41:19.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:41:19 smithi047 ceph-mon[77147]: pgmap v2559: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:41:19.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:41:19 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 4410 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:41:19.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:41:19 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:41:19.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:41:19 smithi115 ceph-mon[78593]: pgmap v2559: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:41:19.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:41:19 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 4410 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:41:20.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:41:19 smithi115 conmon[90393]: 2022-05-14T00:41:19.833+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:41:20.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:41:20 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:41:20.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:41:20 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:41:20.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:41:20 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:41:20.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:41:20 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:41:21.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:41:20 smithi115 conmon[90393]: 2022-05-14T00:41:20.839+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:41:21.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:41:21 smithi047 ceph-mon[77147]: pgmap v2560: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:41:21.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:41:21 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:41:21.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:41:21 smithi115 ceph-mon[78593]: pgmap v2560: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:41:21.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:41:21 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:41:22.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:41:21 smithi115 conmon[90393]: 2022-05-14T00:41:21.806+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:41:22.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:41:22 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:41:22.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:41:22 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:41:22.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:41:22 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:41:22.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:41:22 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:41:22.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:41:22 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:41:22.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:41:22 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:41:23.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:41:22 smithi115 conmon[90393]: 2022-05-14T00:41:22.786+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:41:23.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:41:23 smithi047 ceph-mon[77147]: pgmap v2561: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:41:23.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:41:23 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:41:23.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:41:23 smithi115 ceph-mon[78593]: pgmap v2561: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:41:23.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:41:23 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:41:24.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:41:23 smithi115 conmon[90393]: 2022-05-14T00:41:23.754+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:41:24.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:41:24 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:41:24.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:41:24 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 4415 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:41:24.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:41:24 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:41:24.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:41:24 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 4415 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:41:25.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:41:24 smithi115 conmon[90393]: 2022-05-14T00:41:24.774+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:41:25.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:41:25 smithi047 ceph-mon[77147]: pgmap v2562: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:41:25.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:41:25 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:41:25.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:41:25 smithi115 ceph-mon[78593]: pgmap v2562: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:41:25.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:41:25 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:41:26.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:41:25 smithi115 conmon[90393]: 2022-05-14T00:41:25.757+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:41:26.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:41:26 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:41:26.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:41:26 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:41:27.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:41:26 smithi115 conmon[90393]: 2022-05-14T00:41:26.792+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:41:27.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:41:27 smithi047 ceph-mon[77147]: pgmap v2563: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:41:27.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:41:27 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:41:27.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:41:27 smithi115 ceph-mon[78593]: pgmap v2563: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:41:27.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:41:27 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:41:28.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:41:27 smithi115 conmon[90393]: 2022-05-14T00:41:27.783+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:41:28.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:41:28 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:41:28.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:41:28 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:41:29.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:41:28 smithi115 conmon[90393]: 2022-05-14T00:41:28.777+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:41:29.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:41:29 smithi047 ceph-mon[77147]: pgmap v2564: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:41:29.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:41:29 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:41:29.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:41:29 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 4420 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:41:29.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:41:29 smithi115 ceph-mon[78593]: pgmap v2564: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:41:29.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:41:29 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:41:29.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:41:29 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 4420 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:41:30.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:41:29 smithi115 conmon[90393]: 2022-05-14T00:41:29.772+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:41:30.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:41:30 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:41:30.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:41:30 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:41:30.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:41:30 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:41:30.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:41:30 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:41:30.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:41:30 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:41:30.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:41:30 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:41:31.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:41:30 smithi115 conmon[90393]: 2022-05-14T00:41:30.725+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:41:31.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:41:31 smithi047 ceph-mon[77147]: pgmap v2565: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:41:31.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:41:31 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:41:31.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:41:31 smithi115 ceph-mon[78593]: pgmap v2565: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:41:31.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:41:31 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:41:32.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:41:31 smithi115 conmon[90393]: 2022-05-14T00:41:31.751+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:41:32.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:41:32 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:41:32.617 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:41:32 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:41:33.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:41:32 smithi115 conmon[90393]: 2022-05-14T00:41:32.760+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:41:33.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:41:33 smithi047 ceph-mon[77147]: pgmap v2566: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:41:33.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:41:33 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:41:33.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:41:33 smithi115 ceph-mon[78593]: pgmap v2566: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:41:33.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:41:33 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:41:34.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:41:33 smithi115 conmon[90393]: 2022-05-14T00:41:33.745+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:41:34.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:41:34 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:41:34.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:41:34 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:41:34.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:41:34 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:41:34.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:41:34 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 4425 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:41:34.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:41:34 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:41:34.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:41:34 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:41:34.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:41:34 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:41:34.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:41:34 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 4425 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:41:35.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:41:34 smithi115 conmon[90393]: 2022-05-14T00:41:34.761+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:41:35.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:41:35 smithi047 ceph-mon[77147]: pgmap v2567: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:41:35.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:41:35 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:41:35.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:41:35 smithi115 ceph-mon[78593]: pgmap v2567: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:41:35.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:41:35 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:41:36.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:41:35 smithi115 conmon[90393]: 2022-05-14T00:41:35.770+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:41:36.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:41:36 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:41:36.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:41:36 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:41:37.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:41:36 smithi115 conmon[90393]: 2022-05-14T00:41:36.754+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:41:37.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:41:37 smithi047 ceph-mon[77147]: pgmap v2568: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:41:37.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:41:37 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:41:37.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:41:37 smithi115 ceph-mon[78593]: pgmap v2568: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:41:37.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:41:37 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:41:38.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:41:37 smithi115 conmon[90393]: 2022-05-14T00:41:37.767+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:41:38.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:41:38 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:41:38.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:41:38 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:41:38.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:41:38 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:41:38.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:41:38 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:41:38.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:41:38 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:41:38.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:41:38 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:41:39.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:41:38 smithi115 conmon[90393]: 2022-05-14T00:41:38.733+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:41:39.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:41:39 smithi047 ceph-mon[77147]: pgmap v2569: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:41:39.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:41:39 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:41:39.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:41:39 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 4430 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:41:39.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:41:39 smithi115 ceph-mon[78593]: pgmap v2569: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:41:39.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:41:39 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:41:39.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:41:39 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 4430 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:41:40.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:41:39 smithi115 conmon[90393]: 2022-05-14T00:41:39.731+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:41:40.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:41:40 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:41:40.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:41:40 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:41:41.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:41:40 smithi115 conmon[90393]: 2022-05-14T00:41:40.770+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:41:41.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:41:41 smithi115 ceph-mon[78593]: pgmap v2570: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:41:41.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:41:41 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:41:41.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:41:41 smithi047 ceph-mon[77147]: pgmap v2570: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:41:41.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:41:41 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:41:42.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:41:41 smithi115 conmon[90393]: 2022-05-14T00:41:41.733+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:41:42.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:41:42 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:41:42.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:41:42 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:41:42.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:41:42 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:41:42.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:41:42 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:41:42.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:41:42 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:41:42.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:41:42 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:41:43.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:41:42 smithi115 conmon[90393]: 2022-05-14T00:41:42.768+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:41:43.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:41:43 smithi115 ceph-mon[78593]: pgmap v2571: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:41:43.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:41:43 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:41:43.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:41:43 smithi047 ceph-mon[77147]: pgmap v2571: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:41:43.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:41:43 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:41:44.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:41:43 smithi115 conmon[90393]: 2022-05-14T00:41:43.792+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:41:44.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:41:44 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:41:44.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:41:44 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 4435 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:41:44.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:41:44 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:41:44.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:41:44 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 4435 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:41:45.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:41:44 smithi115 conmon[90393]: 2022-05-14T00:41:44.811+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:41:45.612 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:41:45 smithi115 ceph-mon[78593]: pgmap v2572: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:41:45.613 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:41:45 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:41:45.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:41:45 smithi047 ceph-mon[77147]: pgmap v2572: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:41:45.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:41:45 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:41:46.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:41:45 smithi115 conmon[90393]: 2022-05-14T00:41:45.857+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:41:47.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:41:46 smithi115 conmon[90393]: 2022-05-14T00:41:46.890+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:41:48.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:41:47 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:41:48.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:41:47 smithi047 ceph-mon[77147]: pgmap v2573: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:41:48.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:41:47 smithi115 conmon[90393]: 2022-05-14T00:41:47.851+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:41:48.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:41:47 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:41:48.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:41:47 smithi115 ceph-mon[78593]: pgmap v2573: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:41:49.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:41:48 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:41:49.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:41:48 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:41:49.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:41:48 smithi115 conmon[90393]: 2022-05-14T00:41:48.830+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:41:49.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:41:48 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:41:49.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:41:48 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:41:50.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:41:49 smithi047 ceph-mon[77147]: pgmap v2574: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:41:50.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:41:49 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:41:50.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:41:49 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:41:50.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:41:49 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:41:50.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:41:49 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 4440 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:41:50.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:41:49 smithi115 conmon[90393]: 2022-05-14T00:41:49.812+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:41:50.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:41:49 smithi115 ceph-mon[78593]: pgmap v2574: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:41:50.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:41:49 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:41:50.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:41:49 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:41:50.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:41:49 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:41:50.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:41:49 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 4440 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:41:51.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:41:50 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:41:51.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:41:50 smithi115 conmon[90393]: 2022-05-14T00:41:50.790+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:41:51.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:41:50 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:41:52.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:41:51 smithi047 ceph-mon[77147]: pgmap v2575: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:41:52.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:41:51 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:41:52.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:41:51 smithi115 conmon[90393]: 2022-05-14T00:41:51.816+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:41:52.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:41:51 smithi115 ceph-mon[78593]: pgmap v2575: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:41:52.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:41:51 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:41:53.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:41:52 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:41:53.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:41:52 smithi115 conmon[90393]: 2022-05-14T00:41:52.816+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:41:53.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:41:52 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:41:54.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:41:53 smithi047 ceph-mon[77147]: pgmap v2576: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:41:54.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:41:53 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:41:54.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:41:53 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:41:54.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:41:53 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:41:54.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:41:53 smithi115 conmon[90393]: 2022-05-14T00:41:53.824+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:41:54.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:41:53 smithi115 ceph-mon[78593]: pgmap v2576: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:41:54.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:41:53 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:41:54.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:41:53 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:41:54.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:41:53 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:41:55.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:41:54 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:41:55.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:41:54 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 4445 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:41:55.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:41:54 smithi115 conmon[90393]: 2022-05-14T00:41:54.824+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:41:55.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:41:54 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:41:55.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:41:54 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 4445 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:41:56.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:41:55 smithi047 ceph-mon[77147]: pgmap v2577: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:41:56.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:41:55 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:41:56.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:41:55 smithi115 conmon[90393]: 2022-05-14T00:41:55.810+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:41:56.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:41:55 smithi115 ceph-mon[78593]: pgmap v2577: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:41:56.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:41:55 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:41:57.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:41:56 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:41:57.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:41:56 smithi115 conmon[90393]: 2022-05-14T00:41:56.801+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:41:57.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:41:56 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:41:58.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:41:57 smithi047 ceph-mon[77147]: pgmap v2578: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:41:58.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:41:57 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:41:58.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:41:57 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:41:58.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:41:57 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:41:58.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:41:57 smithi115 conmon[90393]: 2022-05-14T00:41:57.826+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:41:58.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:41:57 smithi115 ceph-mon[78593]: pgmap v2578: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:41:58.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:41:57 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:41:58.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:41:57 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:41:58.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:41:57 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:41:59.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:41:58 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:41:59.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:41:58 smithi115 conmon[90393]: 2022-05-14T00:41:58.797+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:41:59.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:41:58 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:42:00.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:41:59 smithi047 ceph-mon[77147]: pgmap v2579: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:42:00.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:41:59 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:42:00.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:41:59 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 4450 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:42:00.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:41:59 smithi115 conmon[90393]: 2022-05-14T00:41:59.757+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:42:00.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:41:59 smithi115 ceph-mon[78593]: pgmap v2579: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:42:00.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:41:59 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:42:00.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:41:59 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 4450 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:42:01.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:42:00 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:42:01.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:42:00 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:42:01.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:42:00 smithi115 conmon[90393]: 2022-05-14T00:42:00.768+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:42:02.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:42:01 smithi047 ceph-mon[77147]: pgmap v2580: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:42:02.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:42:01 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:42:02.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:42:01 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:42:02.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:42:01 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:42:02.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:42:01 smithi115 conmon[90393]: 2022-05-14T00:42:01.772+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:42:02.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:42:01 smithi115 ceph-mon[78593]: pgmap v2580: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:42:02.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:42:01 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:42:02.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:42:01 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:42:02.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:42:01 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:42:03.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:42:02 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:42:03.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:42:02 smithi115 conmon[90393]: 2022-05-14T00:42:02.730+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:42:03.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:42:02 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:42:04.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:42:03 smithi047 ceph-mon[77147]: pgmap v2581: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:42:04.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:42:03 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:42:04.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:42:03 smithi115 conmon[90393]: 2022-05-14T00:42:03.764+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:42:04.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:42:03 smithi115 ceph-mon[78593]: pgmap v2581: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:42:04.114 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:42:03 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:42:05.046 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:42:04 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:42:05.046 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:42:04 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 4455 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:42:05.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:42:04 smithi115 conmon[90393]: 2022-05-14T00:42:04.812+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:42:05.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:42:04 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:42:05.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:42:04 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 4455 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:42:06.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:42:05 smithi047 ceph-mon[77147]: pgmap v2582: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:42:06.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:42:05 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:42:06.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:42:05 smithi115 conmon[90393]: 2022-05-14T00:42:05.769+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:42:06.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:42:05 smithi115 ceph-mon[78593]: pgmap v2582: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:42:06.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:42:05 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:42:07.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:42:06 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:42:07.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:42:06 smithi115 conmon[90393]: 2022-05-14T00:42:06.810+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:42:07.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:42:06 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:42:08.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:42:07 smithi047 ceph-mon[77147]: pgmap v2583: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:42:08.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:42:07 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:42:08.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:42:07 smithi115 conmon[90393]: 2022-05-14T00:42:07.855+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:42:08.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:42:07 smithi115 ceph-mon[78593]: pgmap v2583: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:42:08.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:42:07 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:42:09.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:42:08 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:42:09.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:42:08 smithi115 conmon[90393]: 2022-05-14T00:42:08.815+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:42:09.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:42:08 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:42:10.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:42:09 smithi047 ceph-mon[77147]: pgmap v2584: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:42:10.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:42:09 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:42:10.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:42:09 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:42:10.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:42:09 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:42:10.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:42:09 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 4460 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:42:10.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:42:09 smithi115 conmon[90393]: 2022-05-14T00:42:09.836+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:42:10.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:42:09 smithi115 ceph-mon[78593]: pgmap v2584: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:42:10.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:42:09 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:42:10.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:42:09 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:42:10.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:42:09 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:42:10.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:42:09 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 4460 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:42:11.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:42:10 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:42:11.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:42:10 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T00:42:11.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:42:10 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T00:42:11.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:42:10 smithi115 conmon[90393]: 2022-05-14T00:42:10.878+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:42:11.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:42:10 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:42:11.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:42:10 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T00:42:11.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:42:10 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T00:42:11.541 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:42:11 smithi047 conmon[78521]: 2022-05-14T00:42:11.184+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:42:11.541 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:42:11 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T00:42:11.541 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:42:11 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:42:11.542 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:42:11 smithi047 conmon[78521]: yield 2022-05-14T00:42:11.542 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:42:11 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:42:11.542 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:42:11 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:42:11.542 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:42:11 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:42:11.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:42:11 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T00:42:11.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:42:11 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:42:11.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:42:11 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T00:42:11.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:42:11 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:42:11.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:42:11 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T00:42:11.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:42:11 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:42:11.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:42:11 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T00:42:11.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:42:11 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:42:11.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:42:11 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T00:42:11.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:42:11 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:42:11.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:42:11 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:42:11.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:42:11 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:42:12.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:42:11 smithi047 ceph-mon[77147]: pgmap v2585: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:42:12.086 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:42:11 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T00:42:12.086 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:42:11 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:42:12.086 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T00:42:12.087 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:42:12.087 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T00:42:12.087 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:42:12.087 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:42:12.088 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:42:12.088 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T00:42:12.088 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:42:12.088 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T00:42:12.088 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:42:12.089 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T00:42:12.089 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:42:12.089 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T00:42:12.089 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:42:12.090 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T00:42:12.090 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:42:12.090 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:42:12.090 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:42:12.090 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:42:11 smithi047 ceph-mon[77147]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T00:42:12.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:42:11 smithi115 conmon[90393]: 2022-05-14T00:42:11.889+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:42:12.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:42:11 smithi115 ceph-mon[78593]: pgmap v2585: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:42:12.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:42:11 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T00:42:12.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:42:11 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:42:12.109 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T00:42:12.109 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:42:12.109 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T00:42:12.109 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:42:12.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:42:12.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:42:12.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T00:42:12.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:42:12.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T00:42:12.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:42:12.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T00:42:12.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:42:12.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T00:42:12.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:42:12.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T00:42:12.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:42:12.113 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:42:12.114 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:42:12.115 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:42:11 smithi115 ceph-mon[78593]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T00:42:13.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:42:12 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:42:13.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:42:12 smithi115 conmon[90393]: 2022-05-14T00:42:12.850+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:42:13.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:42:12 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:42:14.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:42:13 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:42:14.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:42:13 smithi047 ceph-mon[77147]: pgmap v2586: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:42:14.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:42:13 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:42:14.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:42:13 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:42:14.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:42:13 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:42:14.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:42:13 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T00:42:14.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:42:13 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T00:42:14.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:42:13 smithi115 conmon[90393]: 2022-05-14T00:42:13.817+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:42:14.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:42:13 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:42:14.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:42:13 smithi115 ceph-mon[78593]: pgmap v2586: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:42:14.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:42:13 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:42:14.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:42:13 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:42:14.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:42:13 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:42:14.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:42:13 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T00:42:14.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:42:13 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T00:42:14.541 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:42:14 smithi047 conmon[78521]: 2022-05-14T00:42:14.256+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:42:14.541 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:42:14 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T00:42:14.542 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:42:14 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:42:14.542 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:42:14 smithi047 conmon[78521]: yield 2022-05-14T00:42:14.542 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:42:14 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:42:14.542 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:42:14 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:42:14.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:42:14 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:42:14.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:42:14 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T00:42:14.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:42:14 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:42:14.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:42:14 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T00:42:14.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:42:14 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:42:14.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:42:14 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T00:42:14.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:42:14 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:42:14.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:42:14 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T00:42:14.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:42:14 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:42:14.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:42:14 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T00:42:14.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:42:14 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:42:14.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:42:14 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:42:14.547 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:42:14 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:42:14.547 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:42:14 smithi047 conmon[78521]: 2022-05-14T00:42:14.257+0000 7f5344781700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T00:42:14.547 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:42:14 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T00:42:14.547 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:42:14 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:42:14.547 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:42:14 smithi047 conmon[78521]: yield 2022-05-14T00:42:14.548 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:42:14 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:42:14.548 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:42:14 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:42:14.548 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:42:14 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:42:14.548 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:42:14 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T00:42:14.549 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:42:14 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:42:14.549 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:42:14 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T00:42:14.549 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:42:14 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:42:14.549 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:42:14 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T00:42:14.549 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:42:14 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:42:14.550 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:42:14 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T00:42:14.550 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:42:14 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:42:14.550 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:42:14 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T00:42:14.550 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:42:14 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:42:14.551 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:42:14 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:42:14.551 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:42:14 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:42:14.551 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:42:14 smithi047 conmon[78521]: 2022-05-14T00:42:14.551 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:42:14 smithi047 conmon[78521]: During handling of the above exception, another exception occurred: 2022-05-14T00:42:14.551 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:42:14 smithi047 conmon[78521]: 2022-05-14T00:42:14.552 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:42:14 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T00:42:14.552 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:42:14 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T00:42:14.552 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:42:14 smithi047 conmon[78521]: return f(*arg) 2022-05-14T00:42:14.552 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:42:14 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T00:42:14.552 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:42:14 smithi047 conmon[78521]: self._write_client_files(client_files, host) 2022-05-14T00:42:14.553 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:42:14 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T00:42:14.553 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:42:14 smithi047 conmon[78521]: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T00:42:14.553 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:42:14 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T00:42:14.553 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:42:14 smithi047 conmon[78521]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T00:42:14.554 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:42:14 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T00:42:14.554 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:42:14 smithi047 conmon[78521]: return self.event_loop.get_result(coro) 2022-05-14T00:42:14.554 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:42:14 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T00:42:14.554 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:42:14 smithi047 conmon[78521]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T00:42:14.555 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:42:14 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T00:42:14.555 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:42:14 smithi047 conmon[78521]: return self.__get_result() 2022-05-14T00:42:14.555 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:42:14 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T00:42:14.555 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:42:14 smithi047 conmon[78521]: raise self._exception 2022-05-14T00:42:14.555 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:42:14 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T00:42:14.556 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:42:14 smithi047 conmon[78521]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T00:42:14.556 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:42:14 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T00:42:14.556 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:42:14 smithi047 conmon[78521]: conn = await self._remote_connection(host, addr) 2022-05-14T00:42:14.556 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:42:14 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T00:42:14.557 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:42:14 smithi047 conmon[78521]: raise 2022-05-14T00:42:14.557 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:42:14 smithi047 conmon[78521]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T00:42:14.557 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:42:14 smithi047 conmon[78521]: self.gen.throw(type, value, traceback) 2022-05-14T00:42:14.557 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:42:14 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T00:42:14.558 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:42:14 smithi047 conmon[78521]: raise OrchestratorError(msg) 2022-05-14T00:42:14.558 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:42:14 smithi047 conmon[78521]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:42:15.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:42:14 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:42:15.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:42:14 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 4465 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:42:15.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:42:14 smithi115 conmon[90393]: 2022-05-14T00:42:14.843+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:42:15.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:42:14 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:42:15.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:42:14 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 4465 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:42:16.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:42:15 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:42:16.041 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T00:42:16.041 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:42:16.042 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T00:42:16.042 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:42:16.042 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:42:16.042 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:42:16.042 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T00:42:16.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:42:16.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T00:42:16.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:42:16.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T00:42:16.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:42:16.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T00:42:16.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:42:16.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T00:42:16.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:42:16.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:42:16.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:42:16.045 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:42:15 smithi047 ceph-mon[77147]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T00:42:16.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T00:42:16.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:42:16.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T00:42:16.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:42:16.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:42:16.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:42:16.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T00:42:16.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:42:16.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T00:42:16.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:42:16.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T00:42:16.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:42:16.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T00:42:16.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:42:16.049 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T00:42:16.049 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:42:16.049 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:42:16.050 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:42:16.050 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T00:42:16.050 INFO:journalctl@ceph.mon.b.smithi047.stdout: During handling of the above exception, another exception occurred: 2022-05-14T00:42:16.050 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T00:42:16.051 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T00:42:16.051 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T00:42:16.051 INFO:journalctl@ceph.mon.b.smithi047.stdout: return f(*arg) 2022-05-14T00:42:16.051 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T00:42:16.052 INFO:journalctl@ceph.mon.b.smithi047.stdout: self._write_client_files(client_files, host) 2022-05-14T00:42:16.052 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T00:42:16.052 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T00:42:16.052 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T00:42:16.053 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T00:42:16.053 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T00:42:16.053 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.event_loop.get_result(coro) 2022-05-14T00:42:16.053 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T00:42:16.054 INFO:journalctl@ceph.mon.b.smithi047.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T00:42:16.054 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T00:42:16.054 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.__get_result() 2022-05-14T00:42:16.054 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T00:42:16.054 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise self._exception 2022-05-14T00:42:16.055 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T00:42:16.055 INFO:journalctl@ceph.mon.b.smithi047.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T00:42:16.055 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T00:42:16.055 INFO:journalctl@ceph.mon.b.smithi047.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T00:42:16.056 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T00:42:16.056 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise 2022-05-14T00:42:16.057 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T00:42:16.057 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.gen.throw(type, value, traceback) 2022-05-14T00:42:16.057 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T00:42:16.058 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OrchestratorError(msg) 2022-05-14T00:42:16.058 INFO:journalctl@ceph.mon.b.smithi047.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:42:16.058 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:42:15 smithi047 ceph-mon[77147]: pgmap v2587: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:42:16.058 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:42:15 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:42:16.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:42:15 smithi115 conmon[90393]: 2022-05-14T00:42:15.866+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:42:16.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:42:15 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:42:16.109 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T00:42:16.109 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:42:16.109 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T00:42:16.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:42:16.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:42:16.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:42:16.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T00:42:16.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:42:16.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T00:42:16.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:42:16.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T00:42:16.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:42:16.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T00:42:16.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:42:16.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T00:42:16.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:42:16.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:42:16.113 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:42:16.113 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:42:15 smithi115 ceph-mon[78593]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T00:42:16.113 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T00:42:16.114 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:42:16.114 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T00:42:16.114 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:42:16.115 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:42:16.115 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:42:16.116 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T00:42:16.116 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:42:16.116 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T00:42:16.116 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:42:16.117 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T00:42:16.117 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:42:16.117 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T00:42:16.117 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:42:16.118 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T00:42:16.118 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:42:16.118 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:42:16.118 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:42:16.118 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T00:42:16.119 INFO:journalctl@ceph.mon.c.smithi115.stdout: During handling of the above exception, another exception occurred: 2022-05-14T00:42:16.119 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T00:42:16.119 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T00:42:16.119 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T00:42:16.119 INFO:journalctl@ceph.mon.c.smithi115.stdout: return f(*arg) 2022-05-14T00:42:16.120 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T00:42:16.120 INFO:journalctl@ceph.mon.c.smithi115.stdout: self._write_client_files(client_files, host) 2022-05-14T00:42:16.120 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T00:42:16.120 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T00:42:16.121 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T00:42:16.121 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T00:42:16.121 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T00:42:16.121 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.event_loop.get_result(coro) 2022-05-14T00:42:16.121 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T00:42:16.122 INFO:journalctl@ceph.mon.c.smithi115.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T00:42:16.122 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T00:42:16.122 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.__get_result() 2022-05-14T00:42:16.122 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T00:42:16.123 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise self._exception 2022-05-14T00:42:16.123 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T00:42:16.123 INFO:journalctl@ceph.mon.c.smithi115.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T00:42:16.123 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T00:42:16.123 INFO:journalctl@ceph.mon.c.smithi115.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T00:42:16.124 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T00:42:16.124 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise 2022-05-14T00:42:16.124 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T00:42:16.124 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.gen.throw(type, value, traceback) 2022-05-14T00:42:16.124 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T00:42:16.125 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OrchestratorError(msg) 2022-05-14T00:42:16.125 INFO:journalctl@ceph.mon.c.smithi115.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:42:16.125 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:42:15 smithi115 ceph-mon[78593]: pgmap v2587: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:42:16.125 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:42:15 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:42:17.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:42:16 smithi115 conmon[90393]: 2022-05-14T00:42:16.868+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:42:18.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:42:17 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:42:18.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:42:17 smithi047 ceph-mon[77147]: pgmap v2588: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:42:18.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:42:17 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:42:18.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:42:17 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:42:18.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:42:17 smithi115 conmon[90393]: 2022-05-14T00:42:17.892+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:42:18.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:42:17 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:42:18.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:42:17 smithi115 ceph-mon[78593]: pgmap v2588: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:42:18.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:42:17 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:42:18.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:42:17 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:42:19.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:42:18 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:42:19.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:42:18 smithi115 conmon[90393]: 2022-05-14T00:42:18.929+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:42:19.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:42:18 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:42:20.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:42:19 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:42:20.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:42:19 smithi047 ceph-mon[77147]: pgmap v2589: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:42:20.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:42:19 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 4470 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:42:20.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:42:19 smithi115 conmon[90393]: 2022-05-14T00:42:19.972+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:42:20.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:42:19 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:42:20.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:42:19 smithi115 ceph-mon[78593]: pgmap v2589: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:42:20.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:42:19 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 4470 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:42:21.004 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:42:20 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:42:21.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:42:20 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:42:21.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:42:20 smithi115 conmon[90393]: 2022-05-14T00:42:20.998+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:42:22.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:42:21 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:42:22.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:42:21 smithi047 ceph-mon[77147]: pgmap v2590: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:42:22.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:42:21 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:42:22.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:42:21 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:42:22.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:42:21 smithi115 conmon[90393]: 2022-05-14T00:42:21.972+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:42:22.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:42:21 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:42:22.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:42:21 smithi115 ceph-mon[78593]: pgmap v2590: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:42:22.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:42:21 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:42:22.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:42:21 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:42:23.046 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:42:22 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:42:23.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:42:22 smithi115 conmon[90393]: 2022-05-14T00:42:22.963+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:42:23.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:42:22 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:42:24.008 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:42:23 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:42:24.009 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:42:23 smithi115 ceph-mon[78593]: pgmap v2591: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:42:24.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:42:23 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:42:24.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:42:23 smithi047 ceph-mon[77147]: pgmap v2591: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:42:24.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:42:24 smithi115 conmon[90393]: 2022-05-14T00:42:24.007+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:42:25.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:42:24 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:42:25.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:42:24 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 4475 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:42:25.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:42:24 smithi115 conmon[90393]: 2022-05-14T00:42:24.977+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:42:25.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:42:24 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:42:25.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:42:24 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 4475 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:42:26.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:42:25 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:42:26.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:42:25 smithi047 ceph-mon[77147]: pgmap v2592: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:42:26.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:42:25 smithi115 conmon[90393]: 2022-05-14T00:42:25.972+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:42:26.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:42:25 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:42:26.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:42:25 smithi115 ceph-mon[78593]: pgmap v2592: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:42:27.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:42:26 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:42:27.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:42:26 smithi115 conmon[90393]: 2022-05-14T00:42:26.943+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:42:27.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:42:26 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:42:28.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:42:27 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:42:28.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:42:27 smithi047 ceph-mon[77147]: pgmap v2593: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:42:28.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:42:27 smithi115 conmon[90393]: 2022-05-14T00:42:27.941+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:42:28.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:42:27 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:42:28.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:42:27 smithi115 ceph-mon[78593]: pgmap v2593: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:42:29.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:42:28 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:42:29.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:42:28 smithi115 conmon[90393]: 2022-05-14T00:42:28.895+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:42:29.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:42:28 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:42:30.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:42:29 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:42:30.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:42:29 smithi047 ceph-mon[77147]: pgmap v2594: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:42:30.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:42:29 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:42:30.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:42:29 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:42:30.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:42:29 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 4480 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:42:30.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:42:29 smithi115 conmon[90393]: 2022-05-14T00:42:29.923+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:42:30.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:42:29 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:42:30.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:42:29 smithi115 ceph-mon[78593]: pgmap v2594: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:42:30.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:42:29 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:42:30.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:42:29 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:42:30.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:42:29 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 4480 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:42:31.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:42:30 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:42:31.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:42:30 smithi115 conmon[90393]: 2022-05-14T00:42:30.933+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:42:31.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:42:30 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:42:32.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:42:31 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:42:32.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:42:31 smithi047 ceph-mon[77147]: pgmap v2595: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:42:32.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:42:31 smithi115 conmon[90393]: 2022-05-14T00:42:31.900+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:42:32.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:42:31 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:42:32.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:42:31 smithi115 ceph-mon[78593]: pgmap v2595: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:42:33.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:42:32 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:42:33.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:42:32 smithi115 conmon[90393]: 2022-05-14T00:42:32.918+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:42:33.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:42:32 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:42:34.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:42:33 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:42:34.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:42:33 smithi047 ceph-mon[77147]: pgmap v2596: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:42:34.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:42:33 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:42:34.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:42:33 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:42:34.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:42:33 smithi115 conmon[90393]: 2022-05-14T00:42:33.891+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:42:34.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:42:33 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:42:34.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:42:33 smithi115 ceph-mon[78593]: pgmap v2596: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:42:34.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:42:33 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:42:34.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:42:33 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:42:35.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:42:34 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:42:35.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:42:34 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 4485 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:42:35.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:42:34 smithi115 conmon[90393]: 2022-05-14T00:42:34.918+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:42:35.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:42:34 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:42:35.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:42:34 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 4485 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:42:36.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:42:35 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:42:36.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:42:35 smithi047 ceph-mon[77147]: pgmap v2597: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:42:36.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:42:35 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:42:36.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:42:35 smithi115 ceph-mon[78593]: pgmap v2597: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:42:36.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:42:35 smithi115 conmon[90393]: 2022-05-14T00:42:35.926+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:42:37.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:42:36 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:42:37.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:42:36 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:42:37.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:42:36 smithi115 conmon[90393]: 2022-05-14T00:42:36.891+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:42:38.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:42:37 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:42:38.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:42:37 smithi047 ceph-mon[77147]: pgmap v2598: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:42:38.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:42:37 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:42:38.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:42:37 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:42:38.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:42:37 smithi115 conmon[90393]: 2022-05-14T00:42:37.893+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:42:38.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:42:37 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:42:38.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:42:37 smithi115 ceph-mon[78593]: pgmap v2598: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:42:38.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:42:37 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:42:38.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:42:37 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:42:39.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:42:38 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:42:39.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:42:38 smithi115 conmon[90393]: 2022-05-14T00:42:38.871+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:42:39.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:42:38 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:42:40.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:42:39 smithi115 conmon[90393]: 2022-05-14T00:42:39.875+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:42:40.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:42:39 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:42:40.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:42:39 smithi115 ceph-mon[78593]: pgmap v2599: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:42:40.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:42:39 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 4490 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:42:40.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:42:39 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:42:40.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:42:39 smithi047 ceph-mon[77147]: pgmap v2599: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:42:40.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:42:39 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 4490 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:42:41.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:42:40 smithi115 conmon[90393]: 2022-05-14T00:42:40.838+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:42:41.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:42:40 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:42:41.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:42:40 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:42:42.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:42:41 smithi115 conmon[90393]: 2022-05-14T00:42:41.873+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:42:42.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:42:41 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:42:42.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:42:41 smithi115 ceph-mon[78593]: pgmap v2600: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:42:42.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:42:41 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:42:42.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:42:41 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:42:42.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:42:41 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:42:42.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:42:41 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:42:42.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:42:41 smithi047 ceph-mon[77147]: pgmap v2600: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:42:42.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:42:41 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:42:42.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:42:41 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:42:42.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:42:41 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:42:43.363 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:42:42 smithi115 conmon[90393]: 2022-05-14T00:42:42.888+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:42:44.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:42:43 smithi115 conmon[90393]: 2022-05-14T00:42:43.933+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:42:44.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:42:43 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:42:44.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:42:43 smithi115 ceph-mon[78593]: pgmap v2601: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:42:44.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:42:43 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:42:44.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:42:43 smithi047 ceph-mon[77147]: pgmap v2601: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:42:45.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:42:44 smithi115 conmon[90393]: 2022-05-14T00:42:44.932+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:42:45.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:42:44 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:42:45.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:42:44 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 4495 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:42:45.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:42:44 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:42:45.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:42:44 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 4495 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:42:46.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:42:45 smithi115 conmon[90393]: 2022-05-14T00:42:45.915+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:42:46.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:42:45 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:42:46.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:42:45 smithi115 ceph-mon[78593]: pgmap v2602: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:42:46.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:42:45 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:42:46.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:42:45 smithi047 ceph-mon[77147]: pgmap v2602: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:42:47.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:42:46 smithi115 conmon[90393]: 2022-05-14T00:42:46.959+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:42:47.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:42:46 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:42:47.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:42:46 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:42:48.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:42:47 smithi115 conmon[90393]: 2022-05-14T00:42:47.953+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:42:48.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:42:47 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:42:48.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:42:47 smithi115 ceph-mon[78593]: pgmap v2603: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:42:48.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:42:47 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:42:48.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:42:47 smithi047 ceph-mon[77147]: pgmap v2603: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:42:49.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:42:49 smithi115 conmon[90393]: 2022-05-14T00:42:49.000+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:42:49.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:42:48 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:42:49.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:42:48 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:42:50.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:42:49 smithi115 conmon[90393]: 2022-05-14T00:42:49.969+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:42:50.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:42:49 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:42:50.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:42:49 smithi115 ceph-mon[78593]: pgmap v2604: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:42:50.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:42:49 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:42:50.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:42:49 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:42:50.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:42:49 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 4500 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:42:50.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:42:49 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:42:50.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:42:49 smithi047 ceph-mon[77147]: pgmap v2604: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:42:50.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:42:49 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:42:50.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:42:49 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:42:50.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:42:49 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 4500 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:42:51.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:42:51 smithi115 conmon[90393]: 2022-05-14T00:42:50.999+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:42:51.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:42:50 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:42:51.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:42:50 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:42:52.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:42:51 smithi115 conmon[90393]: 2022-05-14T00:42:51.964+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:42:52.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:42:51 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:42:52.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:42:51 smithi115 ceph-mon[78593]: pgmap v2605: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:42:52.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:42:51 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:42:52.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:42:51 smithi047 ceph-mon[77147]: pgmap v2605: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:42:53.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:42:52 smithi115 conmon[90393]: 2022-05-14T00:42:52.961+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:42:53.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:42:52 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:42:53.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:42:52 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:42:54.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:42:54 smithi115 conmon[90393]: 2022-05-14T00:42:54.008+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:42:54.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:42:53 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:42:54.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:42:53 smithi115 ceph-mon[78593]: pgmap v2606: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:42:54.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:42:53 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:42:54.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:42:53 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:42:54.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:42:53 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:42:54.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:42:53 smithi047 ceph-mon[77147]: pgmap v2606: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:42:54.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:42:53 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:42:54.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:42:53 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:42:55.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:42:55 smithi115 conmon[90393]: 2022-05-14T00:42:55.038+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:42:55.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:42:54 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:42:55.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:42:54 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 4505 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:42:55.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:42:54 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:42:55.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:42:54 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 4505 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:42:56.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:42:56 smithi115 conmon[90393]: 2022-05-14T00:42:56.060+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:42:56.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:42:55 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:42:56.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:42:55 smithi115 ceph-mon[78593]: pgmap v2607: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:42:56.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:42:55 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:42:56.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:42:55 smithi047 ceph-mon[77147]: pgmap v2607: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:42:57.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:42:57 smithi115 conmon[90393]: 2022-05-14T00:42:57.054+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:42:57.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:42:56 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:42:57.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:42:56 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:42:58.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:42:58 smithi115 conmon[90393]: 2022-05-14T00:42:58.063+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:42:58.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:42:57 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:42:58.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:42:57 smithi115 ceph-mon[78593]: pgmap v2608: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:42:58.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:42:57 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:42:58.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:42:57 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:42:58.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:42:57 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:42:58.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:42:57 smithi047 ceph-mon[77147]: pgmap v2608: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:42:58.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:42:57 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:42:58.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:42:57 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:42:59.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:42:59 smithi115 conmon[90393]: 2022-05-14T00:42:59.081+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:42:59.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:42:58 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:42:59.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:42:58 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:43:00.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:43:00 smithi115 conmon[90393]: 2022-05-14T00:43:00.044+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:43:00.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:42:59 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:43:00.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:42:59 smithi115 ceph-mon[78593]: pgmap v2609: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:43:00.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:42:59 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 4510 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:43:00.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:42:59 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:43:00.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:42:59 smithi047 ceph-mon[77147]: pgmap v2609: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:43:00.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:42:59 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 4510 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:43:01.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:43:00 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:43:01.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:43:01 smithi115 conmon[90393]: 2022-05-14T00:43:01.046+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:43:01.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:43:00 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:43:02.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:43:02 smithi115 conmon[90393]: 2022-05-14T00:43:02.028+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:43:02.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:43:01 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:43:02.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:43:01 smithi115 ceph-mon[78593]: pgmap v2610: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:43:02.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:43:01 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:43:02.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:43:01 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:43:02.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:43:01 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:43:02.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:43:01 smithi047 ceph-mon[77147]: pgmap v2610: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:43:02.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:43:01 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:43:02.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:43:01 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:43:03.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:43:02 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:43:03.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:43:02 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:43:03.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:43:03 smithi115 conmon[90393]: 2022-05-14T00:43:03.054+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:43:04.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:43:03 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:43:04.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:43:03 smithi047 ceph-mon[77147]: pgmap v2611: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:43:04.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:43:04 smithi115 conmon[90393]: 2022-05-14T00:43:04.061+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:43:04.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:43:03 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:43:04.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:43:03 smithi115 ceph-mon[78593]: pgmap v2611: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:43:05.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:43:04 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:43:05.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:43:04 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 4515 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:43:05.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:43:05 smithi115 conmon[90393]: 2022-05-14T00:43:05.024+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:43:05.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:43:04 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:43:05.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:43:04 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 4515 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:43:06.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:43:05 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:43:06.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:43:05 smithi047 ceph-mon[77147]: pgmap v2612: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:43:06.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:43:05 smithi115 conmon[90393]: 2022-05-14T00:43:05.976+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:43:06.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:43:05 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:43:06.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:43:05 smithi115 ceph-mon[78593]: pgmap v2612: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:43:07.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:43:06 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:43:07.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:43:06 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:43:07.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:43:06 smithi115 conmon[90393]: 2022-05-14T00:43:06.936+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:43:08.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:43:07 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:43:08.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:43:07 smithi047 ceph-mon[77147]: pgmap v2613: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:43:08.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:43:07 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:43:08.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:43:07 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:43:08.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:43:07 smithi115 ceph-mon[78593]: pgmap v2613: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:43:08.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:43:07 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:43:08.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:43:07 smithi115 conmon[90393]: 2022-05-14T00:43:07.905+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:43:09.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:43:08 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:43:09.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:43:08 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:43:09.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:43:08 smithi115 conmon[90393]: 2022-05-14T00:43:08.919+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:43:10.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:43:09 smithi047 ceph-mon[77147]: pgmap v2614: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:43:10.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:43:09 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:43:10.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:43:09 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:43:10.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:43:09 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:43:10.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:43:09 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 4520 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:43:10.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:43:09 smithi115 conmon[90393]: 2022-05-14T00:43:09.943+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:43:10.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:43:09 smithi115 ceph-mon[78593]: pgmap v2614: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:43:10.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:43:09 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:43:10.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:43:09 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:43:10.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:43:09 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:43:10.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:43:09 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 4520 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:43:11.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:43:10 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:43:11.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:43:10 smithi115 conmon[90393]: 2022-05-14T00:43:10.971+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:43:11.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:43:10 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:43:12.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:43:11 smithi047 ceph-mon[77147]: pgmap v2615: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:43:12.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:43:11 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:43:12.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:43:11 smithi115 conmon[90393]: 2022-05-14T00:43:11.960+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:43:12.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:43:11 smithi115 ceph-mon[78593]: pgmap v2615: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:43:12.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:43:11 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:43:13.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:43:12 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:43:13.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:43:12 smithi115 conmon[90393]: 2022-05-14T00:43:12.960+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:43:13.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:43:12 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:43:14.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:43:13 smithi047 ceph-mon[77147]: pgmap v2616: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:43:14.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:43:13 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:43:14.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:43:13 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:43:14.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:43:13 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:43:14.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:43:13 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T00:43:14.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:43:13 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T00:43:14.304 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:43:14 smithi115 conmon[90393]: 2022-05-14T00:43:14.008+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:43:14.305 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:43:13 smithi115 ceph-mon[78593]: pgmap v2616: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:43:14.305 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:43:13 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:43:14.305 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:43:13 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:43:14.305 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:43:13 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:43:14.306 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:43:13 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T00:43:14.306 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:43:13 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T00:43:15.034 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:43:14 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:43:15.034 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:43:14 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 4525 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:43:15.034 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:43:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T00:43:15.035 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:43:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T00:43:15.035 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:43:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T00:43:15.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:43:14 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:43:15.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:43:14 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 4525 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:43:15.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:43:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T00:43:15.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:43:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T00:43:15.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:43:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T00:43:15.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:43:15 smithi115 conmon[90393]: 2022-05-14T00:43:15.033+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:43:16.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:43:16 smithi115 conmon[90393]: 2022-05-14T00:43:16.003+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:43:16.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:43:16 smithi115 ceph-mon[78593]: pgmap v2617: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:43:16.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:43:16 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:43:16.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:43:16 smithi047 ceph-mon[77147]: pgmap v2617: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:43:16.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:43:16 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:43:17.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:43:16 smithi115 conmon[90393]: 2022-05-14T00:43:16.958+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:43:17.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:43:17 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:43:17.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:43:17 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:43:17.541 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:43:17 smithi047 conmon[78521]: 2022-05-14T00:43:17.359+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:43:17.542 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:43:17 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T00:43:17.542 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:43:17 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:43:17.542 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:43:17 smithi047 conmon[78521]: yield 2022-05-14T00:43:17.542 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:43:17 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:43:17.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:43:17 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:43:17.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:43:17 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:43:17.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:43:17 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T00:43:17.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:43:17 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:43:17.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:43:17 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T00:43:17.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:43:17 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:43:17.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:43:17 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T00:43:17.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:43:17 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:43:17.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:43:17 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T00:43:17.548 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:43:17 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:43:17.548 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:43:17 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T00:43:17.548 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:43:17 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:43:17.549 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:43:17 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:43:17.549 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:43:17 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:43:18.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:43:17 smithi115 conmon[90393]: 2022-05-14T00:43:17.921+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:43:18.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:43:18 smithi115 ceph-mon[78593]: pgmap v2618: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:43:18.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:43:18 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:43:18.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:43:18 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:43:18.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:43:18 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:43:18.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:43:18 smithi047 ceph-mon[77147]: pgmap v2618: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:43:18.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:43:18 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:43:18.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:43:18 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:43:18.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:43:18 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:43:19.158 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:43:18 smithi115 conmon[90393]: 2022-05-14T00:43:18.903+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:43:19.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:43:19 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:43:19.541 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T00:43:19.541 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:43:19.542 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T00:43:19.542 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:43:19.542 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:43:19.542 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:43:19.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T00:43:19.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:43:19.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T00:43:19.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:43:19.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T00:43:19.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:43:19.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T00:43:19.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:43:19.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T00:43:19.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:43:19.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:43:19.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:43:19.545 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:43:19 smithi047 ceph-mon[77147]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T00:43:19.545 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:43:19 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:43:19.546 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:43:19 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 4530 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:43:19.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:43:19 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:43:19.608 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T00:43:19.608 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:43:19.608 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T00:43:19.609 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:43:19.609 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:43:19.609 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:43:19.609 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T00:43:19.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:43:19.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T00:43:19.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:43:19.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T00:43:19.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:43:19.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T00:43:19.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:43:19.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T00:43:19.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:43:19.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:43:19.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:43:19.612 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:43:19 smithi115 ceph-mon[78593]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T00:43:19.612 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:43:19 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:43:19.612 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:43:19 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 4530 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:43:20.160 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:43:19 smithi115 conmon[90393]: 2022-05-14T00:43:19.879+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:43:20.433 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:43:20 smithi047 ceph-mon[77147]: pgmap v2619: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:43:20.433 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:43:20 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:43:20.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:43:20 smithi115 ceph-mon[78593]: pgmap v2619: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:43:20.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:43:20 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:43:20.792 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:43:20 smithi047 conmon[78521]: 2022-05-14T00:43:20.431+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:43:20.792 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:43:20 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T00:43:20.793 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:43:20 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:43:20.793 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:43:20 smithi047 conmon[78521]: yield 2022-05-14T00:43:20.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:43:20 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:43:20.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:43:20 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:43:20.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:43:20 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:43:20.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:43:20 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T00:43:20.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:43:20 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:43:20.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:43:20 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T00:43:20.797 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:43:20 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:43:20.798 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:43:20 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T00:43:20.798 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:43:20 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:43:20.798 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:43:20 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T00:43:20.799 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:43:20 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:43:20.799 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:43:20 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T00:43:20.800 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:43:20 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:43:20.800 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:43:20 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:43:20.802 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:43:20 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:43:20.802 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:43:20 smithi047 conmon[78521]: 2022-05-14T00:43:20.432+0000 7f533ef76700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T00:43:20.802 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:43:20 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T00:43:20.803 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:43:20 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:43:20.803 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:43:20 smithi047 conmon[78521]: yield 2022-05-14T00:43:20.803 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:43:20 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:43:20.803 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:43:20 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:43:20.803 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:43:20 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:43:20.804 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:43:20 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T00:43:20.804 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:43:20 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:43:20.804 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:43:20 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T00:43:20.804 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:43:20 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:43:20.804 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:43:20 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T00:43:20.805 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:43:20 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:43:20.805 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:43:20 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T00:43:20.805 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:43:20 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:43:20.805 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:43:20 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T00:43:20.806 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:43:20 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:43:20.806 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:43:20 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:43:20.806 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:43:20 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:43:20.806 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:43:20 smithi047 conmon[78521]: 2022-05-14T00:43:20.806 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:43:20 smithi047 conmon[78521]: During handling of the above exception, another exception occurred: 2022-05-14T00:43:20.807 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:43:20 smithi047 conmon[78521]: 2022-05-14T00:43:20.807 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:43:20 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T00:43:20.807 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:43:20 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T00:43:20.807 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:43:20 smithi047 conmon[78521]: return f(*arg) 2022-05-14T00:43:20.808 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:43:20 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T00:43:20.808 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:43:20 smithi047 conmon[78521]: self._write_client_files(client_files, host) 2022-05-14T00:43:20.808 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:43:20 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T00:43:20.808 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:43:20 smithi047 conmon[78521]: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T00:43:20.808 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:43:20 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T00:43:20.809 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:43:20 smithi047 conmon[78521]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T00:43:20.809 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:43:20 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T00:43:20.810 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:43:20 smithi047 conmon[78521]: return self.event_loop.get_result(coro) 2022-05-14T00:43:20.810 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:43:20 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T00:43:20.810 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:43:20 smithi047 conmon[78521]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T00:43:20.810 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:43:20 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T00:43:20.810 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:43:20 smithi047 conmon[78521]: return self.__get_result() 2022-05-14T00:43:20.811 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:43:20 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T00:43:20.811 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:43:20 smithi047 conmon[78521]: raise self._exception 2022-05-14T00:43:20.811 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:43:20 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T00:43:20.811 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:43:20 smithi047 conmon[78521]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T00:43:20.811 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:43:20 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T00:43:20.812 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:43:20 smithi047 conmon[78521]: conn = await self._remote_connection(host, addr) 2022-05-14T00:43:20.813 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:43:20 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T00:43:20.813 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:43:20 smithi047 conmon[78521]: raise 2022-05-14T00:43:20.813 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:43:20 smithi047 conmon[78521]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T00:43:20.813 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:43:20 smithi047 conmon[78521]: self.gen.throw(type, value, traceback) 2022-05-14T00:43:20.814 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:43:20 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T00:43:20.814 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:43:20 smithi047 conmon[78521]: raise OrchestratorError(msg) 2022-05-14T00:43:20.814 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:43:20 smithi047 conmon[78521]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:43:21.165 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:43:20 smithi115 conmon[90393]: 2022-05-14T00:43:20.870+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:43:21.545 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:43:21 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:43:21.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:43:21 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:43:22.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:43:21 smithi115 conmon[90393]: 2022-05-14T00:43:21.843+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:43:22.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:43:22 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:43:22.541 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T00:43:22.542 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:43:22.542 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T00:43:22.542 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:43:22.542 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:43:22.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:43:22.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T00:43:22.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:43:22.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T00:43:22.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:43:22.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T00:43:22.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:43:22.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T00:43:22.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:43:22.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T00:43:22.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:43:22.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:43:22.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:43:22.546 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:43:22 smithi047 ceph-mon[77147]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T00:43:22.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T00:43:22.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:43:22.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T00:43:22.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:43:22.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:43:22.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:43:22.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T00:43:22.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:43:22.548 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T00:43:22.548 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:43:22.548 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T00:43:22.548 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:43:22.549 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T00:43:22.549 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:43:22.549 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T00:43:22.549 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:43:22.549 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:43:22.550 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:43:22.550 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T00:43:22.550 INFO:journalctl@ceph.mon.b.smithi047.stdout: During handling of the above exception, another exception occurred: 2022-05-14T00:43:22.550 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T00:43:22.551 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T00:43:22.551 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T00:43:22.551 INFO:journalctl@ceph.mon.b.smithi047.stdout: return f(*arg) 2022-05-14T00:43:22.551 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T00:43:22.551 INFO:journalctl@ceph.mon.b.smithi047.stdout: self._write_client_files(client_files, host) 2022-05-14T00:43:22.552 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T00:43:22.552 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T00:43:22.552 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T00:43:22.553 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T00:43:22.553 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T00:43:22.553 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.event_loop.get_result(coro) 2022-05-14T00:43:22.553 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T00:43:22.554 INFO:journalctl@ceph.mon.b.smithi047.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T00:43:22.554 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T00:43:22.554 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.__get_result() 2022-05-14T00:43:22.554 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T00:43:22.554 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise self._exception 2022-05-14T00:43:22.555 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T00:43:22.555 INFO:journalctl@ceph.mon.b.smithi047.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T00:43:22.555 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T00:43:22.555 INFO:journalctl@ceph.mon.b.smithi047.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T00:43:22.556 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T00:43:22.556 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise 2022-05-14T00:43:22.556 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T00:43:22.556 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.gen.throw(type, value, traceback) 2022-05-14T00:43:22.556 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T00:43:22.557 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OrchestratorError(msg) 2022-05-14T00:43:22.557 INFO:journalctl@ceph.mon.b.smithi047.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:43:22.557 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:43:22 smithi047 ceph-mon[77147]: pgmap v2620: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:43:22.557 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:43:22 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:43:22.558 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:43:22 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:43:22.558 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:43:22 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:43:22.558 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:43:22 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:43:22.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:43:22 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:43:22.609 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T00:43:22.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:43:22.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T00:43:22.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:43:22.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:43:22.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:43:22.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T00:43:22.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:43:22.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T00:43:22.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:43:22.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T00:43:22.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:43:22.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T00:43:22.613 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:43:22.613 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T00:43:22.613 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:43:22.613 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:43:22.613 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:43:22.614 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:43:22 smithi115 ceph-mon[78593]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T00:43:22.614 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T00:43:22.615 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:43:22.615 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T00:43:22.616 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:43:22.616 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:43:22.617 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:43:22.617 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T00:43:22.617 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:43:22.618 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T00:43:22.618 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:43:22.618 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T00:43:22.618 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:43:22.619 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T00:43:22.619 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:43:22.619 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T00:43:22.619 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:43:22.620 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:43:22.620 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:43:22.620 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T00:43:22.620 INFO:journalctl@ceph.mon.c.smithi115.stdout: During handling of the above exception, another exception occurred: 2022-05-14T00:43:22.620 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T00:43:22.621 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T00:43:22.621 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T00:43:22.621 INFO:journalctl@ceph.mon.c.smithi115.stdout: return f(*arg) 2022-05-14T00:43:22.621 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T00:43:22.622 INFO:journalctl@ceph.mon.c.smithi115.stdout: self._write_client_files(client_files, host) 2022-05-14T00:43:22.622 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T00:43:22.622 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T00:43:22.622 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T00:43:22.622 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T00:43:22.623 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T00:43:22.623 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.event_loop.get_result(coro) 2022-05-14T00:43:22.623 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T00:43:22.623 INFO:journalctl@ceph.mon.c.smithi115.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T00:43:22.623 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T00:43:22.624 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.__get_result() 2022-05-14T00:43:22.624 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T00:43:22.624 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise self._exception 2022-05-14T00:43:22.624 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T00:43:22.625 INFO:journalctl@ceph.mon.c.smithi115.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T00:43:22.625 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T00:43:22.625 INFO:journalctl@ceph.mon.c.smithi115.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T00:43:22.625 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T00:43:22.625 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise 2022-05-14T00:43:22.626 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T00:43:22.626 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.gen.throw(type, value, traceback) 2022-05-14T00:43:22.626 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T00:43:22.626 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OrchestratorError(msg) 2022-05-14T00:43:22.627 INFO:journalctl@ceph.mon.c.smithi115.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:43:22.627 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:43:22 smithi115 ceph-mon[78593]: pgmap v2620: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:43:22.627 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:43:22 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:43:22.627 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:43:22 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:43:22.627 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:43:22 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:43:22.628 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:43:22 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:43:23.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:43:22 smithi115 conmon[90393]: 2022-05-14T00:43:22.818+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:43:23.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:43:23 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:43:23.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:43:23 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:43:24.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:43:23 smithi115 conmon[90393]: 2022-05-14T00:43:23.820+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:43:24.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:43:24 smithi047 ceph-mon[77147]: pgmap v2621: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:43:24.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:43:24 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:43:24.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:43:24 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 4535 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:43:24.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:43:24 smithi115 ceph-mon[78593]: pgmap v2621: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:43:24.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:43:24 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:43:24.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:43:24 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 4535 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:43:25.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:43:24 smithi115 conmon[90393]: 2022-05-14T00:43:24.847+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:43:25.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:43:25 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:43:25.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:43:25 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:43:26.181 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:43:25 smithi115 conmon[90393]: 2022-05-14T00:43:25.860+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:43:26.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:43:26 smithi047 ceph-mon[77147]: pgmap v2622: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:43:26.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:43:26 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:43:26.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:43:26 smithi115 ceph-mon[78593]: pgmap v2622: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:43:26.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:43:26 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:43:27.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:43:26 smithi115 conmon[90393]: 2022-05-14T00:43:26.907+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:43:28.182 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:43:27 smithi115 conmon[90393]: 2022-05-14T00:43:27.865+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:43:28.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:43:28 smithi047 ceph-mon[77147]: pgmap v2623: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:43:28.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:43:28 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:43:28.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:43:28 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:43:28.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:43:28 smithi115 ceph-mon[78593]: pgmap v2623: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:43:28.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:43:28 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:43:28.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:43:28 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:43:29.186 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:43:28 smithi115 conmon[90393]: 2022-05-14T00:43:28.910+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:43:29.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:43:29 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 4540 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:43:29.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:43:29 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 4540 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:43:30.188 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:43:29 smithi115 conmon[90393]: 2022-05-14T00:43:29.891+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:43:30.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:43:30 smithi047 ceph-mon[77147]: pgmap v2624: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:43:30.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:43:30 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:43:30.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:43:30 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:43:30.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:43:30 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:43:30.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:43:30 smithi115 ceph-mon[78593]: pgmap v2624: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:43:30.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:43:30 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:43:30.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:43:30 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:43:30.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:43:30 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:43:31.192 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:43:30 smithi115 conmon[90393]: 2022-05-14T00:43:30.893+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:43:31.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:43:31 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:43:31.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:43:31 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:43:32.194 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:43:31 smithi115 conmon[90393]: 2022-05-14T00:43:31.931+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:43:32.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:43:32 smithi047 ceph-mon[77147]: pgmap v2625: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:43:32.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:43:32 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:43:32.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:43:32 smithi115 ceph-mon[78593]: pgmap v2625: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:43:32.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:43:32 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:43:33.199 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:43:32 smithi115 conmon[90393]: 2022-05-14T00:43:32.938+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:43:33.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:43:33 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:43:33.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:43:33 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:43:34.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:43:33 smithi115 conmon[90393]: 2022-05-14T00:43:33.954+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:43:34.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:43:34 smithi115 ceph-mon[78593]: pgmap v2626: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:43:34.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:43:34 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:43:34.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:43:34 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:43:34.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:43:34 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:43:34.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:43:34 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 4545 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:43:34.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:43:34 smithi047 ceph-mon[77147]: pgmap v2626: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:43:34.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:43:34 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:43:34.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:43:34 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:43:34.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:43:34 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:43:34.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:43:34 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 4545 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:43:35.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:43:35 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:43:35.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:43:34 smithi115 conmon[90393]: 2022-05-14T00:43:34.974+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:43:35.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:43:35 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:43:36.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:43:36 smithi115 ceph-mon[78593]: pgmap v2627: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:43:36.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:43:36 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:43:36.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:43:36 smithi115 conmon[90393]: 2022-05-14T00:43:36.000+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:43:36.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:43:36 smithi047 ceph-mon[77147]: pgmap v2627: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:43:36.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:43:36 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:43:37.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:43:37 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:43:37.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:43:37 smithi115 ceph-mon[78593]: pgmap v2628: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:43:37.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:43:36 smithi115 conmon[90393]: 2022-05-14T00:43:36.997+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:43:37.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:43:37 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:43:37.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:43:37 smithi047 ceph-mon[77147]: pgmap v2628: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:43:38.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:43:37 smithi115 conmon[90393]: 2022-05-14T00:43:37.985+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:43:38.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:43:38 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:43:38.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:43:38 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:43:38.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:43:38 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:43:38.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:43:38 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:43:38.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:43:38 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:43:38.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:43:38 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:43:39.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:43:38 smithi115 conmon[90393]: 2022-05-14T00:43:38.974+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:43:39.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:43:39 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:43:39.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:43:39 smithi115 ceph-mon[78593]: pgmap v2629: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:43:39.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:43:39 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 4550 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:43:39.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:43:39 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:43:39.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:43:39 smithi047 ceph-mon[77147]: pgmap v2629: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:43:39.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:43:39 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 4550 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:43:40.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:43:40 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:43:40.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:43:40 smithi115 conmon[90393]: 2022-05-14T00:43:40.017+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:43:40.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:43:40 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:43:41.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:43:41 smithi115 conmon[90393]: 2022-05-14T00:43:41.017+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:43:41.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:43:41 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:43:41.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:43:41 smithi115 ceph-mon[78593]: pgmap v2630: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:43:41.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:43:41 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:43:41.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:43:41 smithi047 ceph-mon[77147]: pgmap v2630: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:43:42.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:43:42 smithi115 conmon[90393]: 2022-05-14T00:43:42.016+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:43:42.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:43:42 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:43:42.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:43:42 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:43:42.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:43:42 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:43:42.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:43:42 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:43:42.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:43:42 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:43:42.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:43:42 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:43:43.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:43:43 smithi115 conmon[90393]: 2022-05-14T00:43:43.001+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:43:43.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:43:43 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:43:43.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:43:43 smithi115 ceph-mon[78593]: pgmap v2631: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:43:43.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:43:43 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:43:43.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:43:43 smithi047 ceph-mon[77147]: pgmap v2631: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:43:44.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:43:44 smithi115 conmon[90393]: 2022-05-14T00:43:44.045+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:43:44.425 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:43:44 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:43:44.426 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:43:44 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 4555 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:43:44.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:43:44 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:43:44.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:43:44 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 4555 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:43:45.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:43:45 smithi115 conmon[90393]: 2022-05-14T00:43:45.031+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:43:45.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:43:45 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:43:45.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:43:45 smithi115 ceph-mon[78593]: pgmap v2632: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:43:45.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:43:45 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:43:45.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:43:45 smithi047 ceph-mon[77147]: pgmap v2632: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:43:46.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:43:46 smithi115 conmon[90393]: 2022-05-14T00:43:46.017+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:43:46.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:43:46 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:43:46.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:43:46 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:43:47.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:43:47 smithi115 conmon[90393]: 2022-05-14T00:43:47.058+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:43:47.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:43:47 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:43:47.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:43:47 smithi115 ceph-mon[78593]: pgmap v2633: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:43:47.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:43:47 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:43:47.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:43:47 smithi047 ceph-mon[77147]: pgmap v2633: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:43:48.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:43:48 smithi115 conmon[90393]: 2022-05-14T00:43:48.033+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:43:48.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:43:48 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:43:48.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:43:48 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:43:49.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:43:49 smithi115 conmon[90393]: 2022-05-14T00:43:49.068+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:43:49.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:43:49 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:43:49.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:43:49 smithi115 ceph-mon[78593]: pgmap v2634: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:43:49.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:43:49 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 4560 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:43:49.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:43:49 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:43:49.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:43:49 smithi047 ceph-mon[77147]: pgmap v2634: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:43:49.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:43:49 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 4560 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:43:50.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:43:50 smithi115 conmon[90393]: 2022-05-14T00:43:50.048+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:43:50.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:43:50 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:43:50.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:43:50 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:43:50.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:43:50 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:43:50.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:43:50 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:43:50.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:43:50 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:43:50.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:43:50 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:43:51.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:43:51 smithi115 conmon[90393]: 2022-05-14T00:43:51.059+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:43:51.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:43:51 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:43:51.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:43:51 smithi115 ceph-mon[78593]: pgmap v2635: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:43:51.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:43:51 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:43:51.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:43:51 smithi047 ceph-mon[77147]: pgmap v2635: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:43:52.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:43:52 smithi115 conmon[90393]: 2022-05-14T00:43:52.028+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:43:52.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:43:52 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:43:52.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:43:52 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:43:53.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:43:53 smithi115 conmon[90393]: 2022-05-14T00:43:53.031+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:43:53.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:43:53 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:43:53.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:43:53 smithi115 ceph-mon[78593]: pgmap v2636: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:43:53.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:43:53 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:43:53.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:43:53 smithi047 ceph-mon[77147]: pgmap v2636: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:43:54.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:43:54 smithi115 conmon[90393]: 2022-05-14T00:43:54.017+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:43:54.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:43:54 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:43:54.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:43:54 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:43:54.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:43:54 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:43:54.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:43:54 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 4565 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:43:54.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:43:54 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:43:54.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:43:54 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:43:54.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:43:54 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:43:54.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:43:54 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 4565 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:43:55.271 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:43:55 smithi115 conmon[90393]: 2022-05-14T00:43:55.007+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:43:55.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:43:55 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:43:55.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:43:55 smithi047 ceph-mon[77147]: pgmap v2637: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:43:55.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:43:55 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:43:55.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:43:55 smithi115 ceph-mon[78593]: pgmap v2637: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:43:56.271 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:43:55 smithi115 conmon[90393]: 2022-05-14T00:43:55.964+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:43:56.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:43:56 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:43:56.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:43:56 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:43:57.278 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:43:57 smithi115 conmon[90393]: 2022-05-14T00:43:57.005+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:43:57.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:43:57 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:43:57.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:43:57 smithi047 ceph-mon[77147]: pgmap v2638: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:43:57.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:43:57 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:43:57.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:43:57 smithi115 ceph-mon[78593]: pgmap v2638: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:43:58.277 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:43:57 smithi115 conmon[90393]: 2022-05-14T00:43:57.996+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:43:58.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:43:58 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:43:58.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:43:58 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:43:58.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:43:58 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:43:58.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:43:58 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:43:58.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:43:58 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:43:58.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:43:58 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:43:59.284 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:43:59 smithi115 conmon[90393]: 2022-05-14T00:43:59.030+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:43:59.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:43:59 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:43:59.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:43:59 smithi047 ceph-mon[77147]: pgmap v2639: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:43:59.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:43:59 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 4570 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:43:59.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:43:59 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:43:59.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:43:59 smithi115 ceph-mon[78593]: pgmap v2639: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:43:59.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:43:59 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 4570 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:44:00.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:44:00 smithi115 conmon[90393]: 2022-05-14T00:44:00.073+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:44:00.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:44:00 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:44:00.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:44:00 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:44:01.291 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:44:01 smithi115 conmon[90393]: 2022-05-14T00:44:01.027+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:44:01.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:44:01 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:44:01.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:44:01 smithi115 ceph-mon[78593]: pgmap v2640: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:44:01.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:44:01 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:44:01.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:44:01 smithi047 ceph-mon[77147]: pgmap v2640: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:44:02.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:44:02 smithi115 conmon[90393]: 2022-05-14T00:44:02.064+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:44:02.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:44:02 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:44:02.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:44:02 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:44:02.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:44:02 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:44:02.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:44:02 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:44:02.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:44:02 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:44:02.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:44:02 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:44:03.296 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:44:03 smithi115 conmon[90393]: 2022-05-14T00:44:03.023+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:44:03.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:44:03 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:44:03.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:44:03 smithi115 ceph-mon[78593]: pgmap v2641: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:44:03.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:44:03 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:44:03.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:44:03 smithi047 ceph-mon[77147]: pgmap v2641: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:44:04.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:44:04 smithi115 conmon[90393]: 2022-05-14T00:44:04.052+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:44:04.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:44:04 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:44:04.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:44:04 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 4575 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:44:04.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:44:04 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:44:04.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:44:04 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 4575 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:44:05.303 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:44:05 smithi115 conmon[90393]: 2022-05-14T00:44:05.025+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:44:05.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:44:05 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:44:05.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:44:05 smithi115 ceph-mon[78593]: pgmap v2642: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:44:05.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:44:05 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:44:05.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:44:05 smithi047 ceph-mon[77147]: pgmap v2642: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:44:06.307 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:44:06 smithi115 conmon[90393]: 2022-05-14T00:44:06.013+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:44:06.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:44:06 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:44:06.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:44:06 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:44:07.310 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:44:07 smithi115 conmon[90393]: 2022-05-14T00:44:07.027+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:44:07.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:44:07 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:44:07.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:44:07 smithi115 ceph-mon[78593]: pgmap v2643: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:44:07.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:44:07 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:44:07.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:44:07 smithi047 ceph-mon[77147]: pgmap v2643: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:44:08.311 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:44:08 smithi115 conmon[90393]: 2022-05-14T00:44:08.029+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:44:08.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:44:08 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:44:08.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:44:08 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:44:09.315 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:44:09 smithi115 conmon[90393]: 2022-05-14T00:44:09.030+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:44:09.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:44:09 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:44:09.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:44:09 smithi115 ceph-mon[78593]: pgmap v2644: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:44:09.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:44:09 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 4580 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:44:09.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:44:09 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:44:09.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:44:09 smithi047 ceph-mon[77147]: pgmap v2644: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:44:09.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:44:09 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 4580 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:44:10.316 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:44:10 smithi115 conmon[90393]: 2022-05-14T00:44:10.060+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:44:10.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:44:10 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:44:10.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:44:10 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:44:10.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:44:10 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:44:10.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:44:10 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:44:10.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:44:10 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:44:10.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:44:10 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:44:11.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:44:11 smithi115 conmon[90393]: 2022-05-14T00:44:11.077+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:44:11.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:44:11 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:44:11.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:44:11 smithi115 ceph-mon[78593]: pgmap v2645: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:44:11.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:44:11 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:44:11.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:44:11 smithi047 ceph-mon[77147]: pgmap v2645: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:44:12.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:44:12 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:44:12.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:44:12 smithi115 conmon[90393]: 2022-05-14T00:44:12.126+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:44:12.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:44:12 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:44:13.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:44:13 smithi115 conmon[90393]: 2022-05-14T00:44:13.090+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:44:13.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:44:13 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:44:13.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:44:13 smithi115 ceph-mon[78593]: pgmap v2646: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:44:13.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:44:13 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:44:13.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:44:13 smithi047 ceph-mon[77147]: pgmap v2646: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:44:14.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:44:14 smithi115 conmon[90393]: 2022-05-14T00:44:14.128+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:44:14.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:44:14 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:44:14.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:44:14 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:44:14.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:44:14 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:44:14.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:44:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T00:44:14.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:44:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T00:44:14.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:44:14 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 4585 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:44:14.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:44:14 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:44:14.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:44:14 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:44:14.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:44:14 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:44:14.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:44:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T00:44:14.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:44:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T00:44:14.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:44:14 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 4585 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:44:15.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:44:15 smithi115 conmon[90393]: 2022-05-14T00:44:15.134+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:44:15.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:44:15 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:44:15.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:44:15 smithi115 ceph-mon[78593]: pgmap v2647: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:44:15.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:44:15 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:44:15.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:44:15 smithi047 ceph-mon[77147]: pgmap v2647: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:44:16.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:44:16 smithi115 conmon[90393]: 2022-05-14T00:44:16.135+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:44:16.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:44:16 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:44:16.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:44:16 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:44:17.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:44:17 smithi115 conmon[90393]: 2022-05-14T00:44:17.111+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:44:17.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:44:17 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:44:17.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:44:17 smithi115 ceph-mon[78593]: pgmap v2648: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:44:17.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:44:17 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:44:17.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:44:17 smithi047 ceph-mon[77147]: pgmap v2648: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:44:18.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:44:18 smithi115 conmon[90393]: 2022-05-14T00:44:18.159+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:44:18.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:44:18 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:44:18.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:44:18 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:44:18.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:44:18 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:44:18.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:44:18 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:44:18.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:44:18 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:44:18.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:44:18 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:44:19.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:44:19 smithi115 conmon[90393]: 2022-05-14T00:44:19.131+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:44:19.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:44:19 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:44:19.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:44:19 smithi115 ceph-mon[78593]: pgmap v2649: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:44:19.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:44:19 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 4591 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:44:19.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:44:19 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:44:19.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:44:19 smithi047 ceph-mon[77147]: pgmap v2649: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:44:19.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:44:19 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 4591 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:44:20.479 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:44:20 smithi115 conmon[90393]: 2022-05-14T00:44:20.118+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:44:20.480 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:44:20 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:44:20.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:44:20 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:44:21.354 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:44:21 smithi115 conmon[90393]: 2022-05-14T00:44:21.120+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:44:21.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:44:21 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:44:21.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:44:21 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T00:44:21.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:44:21 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T00:44:21.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:44:21 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T00:44:21.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:44:21 smithi115 ceph-mon[78593]: pgmap v2650: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:44:21.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:44:21 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' 2022-05-14T00:44:21.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:44:21 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' 2022-05-14T00:44:21.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:44:21 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:44:21.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:44:21 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T00:44:21.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:44:21 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T00:44:21.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:44:21 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T00:44:21.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:44:21 smithi047 ceph-mon[77147]: pgmap v2650: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:44:21.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:44:21 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' 2022-05-14T00:44:21.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:44:21 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' 2022-05-14T00:44:22.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:44:22 smithi115 conmon[90393]: 2022-05-14T00:44:22.165+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:44:22.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:44:22 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:44:22.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:44:22 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:44:22.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:44:22 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:44:22.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:44:22 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:44:22.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:44:22 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:44:22.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:44:22 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:44:23.449 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:44:23 smithi115 conmon[90393]: 2022-05-14T00:44:23.117+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:44:23.449 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:44:23 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:44:23.449 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:44:23 smithi115 ceph-mon[78593]: pgmap v2651: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:44:23.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:44:23 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:44:23.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:44:23 smithi047 ceph-mon[77147]: pgmap v2651: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:44:23.791 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:44:23 smithi047 conmon[78521]: 2022-05-14T00:44:23.536+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:44:23.792 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:44:23 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T00:44:23.792 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:44:23 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:44:23.792 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:44:23 smithi047 conmon[78521]: yield 2022-05-14T00:44:23.792 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:44:23 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:44:23.792 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:44:23 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:44:23.793 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:44:23 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:44:23.793 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:44:23 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T00:44:23.793 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:44:23 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:44:23.793 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:44:23 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T00:44:23.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:44:23 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:44:23.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:44:23 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T00:44:23.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:44:23 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:44:23.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:44:23 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T00:44:23.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:44:23 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:44:23.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:44:23 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T00:44:23.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:44:23 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:44:23.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:44:23 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:44:23.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:44:23 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:44:24.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:44:24 smithi115 conmon[90393]: 2022-05-14T00:44:24.092+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:44:24.709 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:44:24 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:44:24.710 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:44:24 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' 2022-05-14T00:44:24.710 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:44:24 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' 2022-05-14T00:44:24.710 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:44:24 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:44:24.710 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T00:44:24.710 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:44:24.711 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T00:44:24.711 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:44:24.711 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:44:24.711 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:44:24.712 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T00:44:24.712 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:44:24.712 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T00:44:24.713 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:44:24.713 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T00:44:24.713 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:44:24.713 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T00:44:24.713 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:44:24.714 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T00:44:24.714 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:44:24.714 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:44:24.714 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:44:24.714 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:44:24 smithi115 ceph-mon[78593]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T00:44:24.715 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:44:24 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 4596 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:44:24.715 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:44:24 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:44:24.716 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:44:24 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' 2022-05-14T00:44:24.716 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:44:24 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' 2022-05-14T00:44:24.716 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:44:24 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:44:24.716 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T00:44:24.716 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:44:24.717 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T00:44:24.717 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:44:24.717 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:44:24.718 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:44:24.718 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T00:44:24.718 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:44:24.718 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T00:44:24.718 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:44:24.719 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T00:44:24.719 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:44:24.719 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T00:44:24.719 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:44:24.720 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T00:44:24.720 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:44:24.720 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:44:24.720 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:44:24.720 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:44:24 smithi047 ceph-mon[77147]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T00:44:24.721 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:44:24 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 4596 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:44:25.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:44:25 smithi115 conmon[90393]: 2022-05-14T00:44:25.079+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:44:25.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:44:25 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:44:25.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:44:25 smithi047 ceph-mon[77147]: pgmap v2652: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:44:25.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:44:25 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:44:25.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:44:25 smithi115 ceph-mon[78593]: pgmap v2652: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:44:26.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:44:26 smithi115 conmon[90393]: 2022-05-14T00:44:26.080+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:44:26.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:44:26 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:44:26.791 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:44:26 smithi047 conmon[78521]: 2022-05-14T00:44:26.607+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:44:26.791 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:44:26 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T00:44:26.792 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:44:26 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:44:26.792 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:44:26 smithi047 conmon[78521]: yield 2022-05-14T00:44:26.792 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:44:26 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:44:26.792 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:44:26 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:44:26.793 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:44:26 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:44:26.793 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:44:26 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T00:44:26.793 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:44:26 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:44:26.793 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:44:26 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T00:44:26.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:44:26 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:44:26.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:44:26 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T00:44:26.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:44:26 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:44:26.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:44:26 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T00:44:26.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:44:26 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:44:26.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:44:26 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T00:44:26.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:44:26 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:44:26.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:44:26 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:44:26.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:44:26 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:44:26.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:44:26 smithi047 conmon[78521]: 2022-05-14T00:44:26.608+0000 7f533f777700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T00:44:26.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:44:26 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T00:44:26.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:44:26 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:44:26.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:44:26 smithi047 conmon[78521]: yield 2022-05-14T00:44:26.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:44:26 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:44:26.800 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:44:26 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:44:26.800 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:44:26 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:44:26.800 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:44:26 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T00:44:26.800 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:44:26 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:44:26.800 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:44:26 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T00:44:26.801 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:44:26 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:44:26.801 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:44:26 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T00:44:26.801 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:44:26 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:44:26.801 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:44:26 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T00:44:26.802 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:44:26 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:44:26.802 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:44:26 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T00:44:26.802 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:44:26 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:44:26.802 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:44:26 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:44:26.802 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:44:26 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:44:26.803 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:44:26 smithi047 conmon[78521]: 2022-05-14T00:44:26.803 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:44:26 smithi047 conmon[78521]: During handling of the above exception, another exception occurred: 2022-05-14T00:44:26.803 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:44:26 smithi047 conmon[78521]: 2022-05-14T00:44:26.803 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:44:26 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T00:44:26.804 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:44:26 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T00:44:26.804 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:44:26 smithi047 conmon[78521]: return f(*arg) 2022-05-14T00:44:26.804 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:44:26 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T00:44:26.804 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:44:26 smithi047 conmon[78521]: self._write_client_files(client_files, host) 2022-05-14T00:44:26.804 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:44:26 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T00:44:26.805 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:44:26 smithi047 conmon[78521]: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T00:44:26.805 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:44:26 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T00:44:26.805 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:44:26 smithi047 conmon[78521]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T00:44:26.806 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:44:26 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T00:44:26.806 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:44:26 smithi047 conmon[78521]: return self.event_loop.get_result(coro) 2022-05-14T00:44:26.806 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:44:26 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T00:44:26.807 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:44:26 smithi047 conmon[78521]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T00:44:26.807 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:44:26 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T00:44:26.807 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:44:26 smithi047 conmon[78521]: return self.__get_result() 2022-05-14T00:44:26.808 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:44:26 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T00:44:26.808 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:44:26 smithi047 conmon[78521]: raise self._exception 2022-05-14T00:44:26.808 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:44:26 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T00:44:26.809 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:44:26 smithi047 conmon[78521]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T00:44:26.809 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:44:26 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T00:44:26.810 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:44:26 smithi047 conmon[78521]: conn = await self._remote_connection(host, addr) 2022-05-14T00:44:26.810 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:44:26 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T00:44:26.811 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:44:26 smithi047 conmon[78521]: raise 2022-05-14T00:44:26.811 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:44:26 smithi047 conmon[78521]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T00:44:26.811 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:44:26 smithi047 conmon[78521]: self.gen.throw(type, value, traceback) 2022-05-14T00:44:26.812 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:44:26 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T00:44:26.812 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:44:26 smithi047 conmon[78521]: raise OrchestratorError(msg) 2022-05-14T00:44:26.813 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:44:26 smithi047 conmon[78521]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:44:26.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:44:26 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:44:27.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:44:27 smithi115 conmon[90393]: 2022-05-14T00:44:27.049+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:44:28.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:44:27 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:44:28.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:44:27 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:44:28.041 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T00:44:28.042 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:44:28.042 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T00:44:28.042 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:44:28.042 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:44:28.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:44:28.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T00:44:28.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:44:28.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T00:44:28.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:44:28.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T00:44:28.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:44:28.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T00:44:28.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:44:28.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T00:44:28.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:44:28.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:44:28.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:44:28.046 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:44:27 smithi047 ceph-mon[77147]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T00:44:28.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T00:44:28.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:44:28.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T00:44:28.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:44:28.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:44:28.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:44:28.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T00:44:28.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:44:28.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T00:44:28.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:44:28.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T00:44:28.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:44:28.049 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T00:44:28.049 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:44:28.049 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T00:44:28.049 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:44:28.049 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:44:28.050 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:44:28.050 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T00:44:28.050 INFO:journalctl@ceph.mon.b.smithi047.stdout: During handling of the above exception, another exception occurred: 2022-05-14T00:44:28.050 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T00:44:28.051 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T00:44:28.051 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T00:44:28.051 INFO:journalctl@ceph.mon.b.smithi047.stdout: return f(*arg) 2022-05-14T00:44:28.051 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T00:44:28.051 INFO:journalctl@ceph.mon.b.smithi047.stdout: self._write_client_files(client_files, host) 2022-05-14T00:44:28.052 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T00:44:28.052 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T00:44:28.052 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T00:44:28.052 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T00:44:28.052 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T00:44:28.053 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.event_loop.get_result(coro) 2022-05-14T00:44:28.053 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T00:44:28.053 INFO:journalctl@ceph.mon.b.smithi047.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T00:44:28.053 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T00:44:28.054 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.__get_result() 2022-05-14T00:44:28.054 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T00:44:28.054 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise self._exception 2022-05-14T00:44:28.054 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T00:44:28.054 INFO:journalctl@ceph.mon.b.smithi047.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T00:44:28.055 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T00:44:28.055 INFO:journalctl@ceph.mon.b.smithi047.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T00:44:28.055 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T00:44:28.055 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise 2022-05-14T00:44:28.055 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T00:44:28.056 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.gen.throw(type, value, traceback) 2022-05-14T00:44:28.056 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T00:44:28.056 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OrchestratorError(msg) 2022-05-14T00:44:28.056 INFO:journalctl@ceph.mon.b.smithi047.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:44:28.057 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:44:27 smithi047 ceph-mon[77147]: pgmap v2653: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:44:28.057 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:44:27 smithi047 ceph-mon[77147]: Detected new or changed devices on smithi115 2022-05-14T00:44:28.057 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:44:27 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' 2022-05-14T00:44:28.057 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:44:27 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-05-14T00:44:28.057 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:44:27 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-05-14T00:44:28.058 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:44:27 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-05-14T00:44:28.058 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:44:27 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-05-14T00:44:28.058 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:44:27 smithi047 ceph-mon[77147]: Adjusting osd_memory_target on smithi115 to 197.0M 2022-05-14T00:44:28.058 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:44:27 smithi047 ceph-mon[77147]: Unable to set osd_memory_target on smithi115 to 206629785: error parsing value: Value '206629785' is below minimum 939524096 2022-05-14T00:44:28.059 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:44:27 smithi047 ceph-mon[77147]: Detected new or changed devices on smithi047 2022-05-14T00:44:28.059 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:44:27 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' 2022-05-14T00:44:28.059 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:44:27 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-05-14T00:44:28.059 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:44:27 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-05-14T00:44:28.059 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:44:27 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-05-14T00:44:28.060 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:44:27 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-05-14T00:44:28.060 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:44:27 smithi047 ceph-mon[77147]: Adjusting osd_memory_target on smithi047 to 198.1M 2022-05-14T00:44:28.060 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:44:27 smithi047 ceph-mon[77147]: Unable to set osd_memory_target on smithi047 to 207755878: error parsing value: Value '207755878' is below minimum 939524096 2022-05-14T00:44:28.063 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:44:27 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:44:28.063 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:44:27 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:44:28.064 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T00:44:28.064 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:44:28.064 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T00:44:28.064 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:44:28.064 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:44:28.065 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:44:28.065 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T00:44:28.065 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:44:28.065 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T00:44:28.066 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:44:28.066 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T00:44:28.066 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:44:28.066 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T00:44:28.066 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:44:28.067 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T00:44:28.067 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:44:28.067 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:44:28.067 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:44:28.068 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:44:27 smithi115 ceph-mon[78593]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T00:44:28.068 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T00:44:28.068 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:44:28.068 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T00:44:28.068 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:44:28.069 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:44:28.069 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:44:28.069 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T00:44:28.069 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:44:28.069 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T00:44:28.070 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:44:28.070 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T00:44:28.070 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:44:28.070 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T00:44:28.071 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:44:28.071 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T00:44:28.071 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:44:28.071 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:44:28.071 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:44:28.072 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T00:44:28.072 INFO:journalctl@ceph.mon.c.smithi115.stdout: During handling of the above exception, another exception occurred: 2022-05-14T00:44:28.072 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T00:44:28.072 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T00:44:28.072 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T00:44:28.073 INFO:journalctl@ceph.mon.c.smithi115.stdout: return f(*arg) 2022-05-14T00:44:28.073 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T00:44:28.073 INFO:journalctl@ceph.mon.c.smithi115.stdout: self._write_client_files(client_files, host) 2022-05-14T00:44:28.073 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T00:44:28.074 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T00:44:28.074 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T00:44:28.074 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T00:44:28.074 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T00:44:28.075 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.event_loop.get_result(coro) 2022-05-14T00:44:28.075 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T00:44:28.075 INFO:journalctl@ceph.mon.c.smithi115.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T00:44:28.075 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T00:44:28.075 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.__get_result() 2022-05-14T00:44:28.076 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T00:44:28.076 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise self._exception 2022-05-14T00:44:28.076 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T00:44:28.076 INFO:journalctl@ceph.mon.c.smithi115.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T00:44:28.076 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T00:44:28.077 INFO:journalctl@ceph.mon.c.smithi115.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T00:44:28.077 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T00:44:28.077 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise 2022-05-14T00:44:28.077 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T00:44:28.078 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.gen.throw(type, value, traceback) 2022-05-14T00:44:28.078 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T00:44:28.078 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OrchestratorError(msg) 2022-05-14T00:44:28.078 INFO:journalctl@ceph.mon.c.smithi115.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:44:28.078 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:44:27 smithi115 ceph-mon[78593]: pgmap v2653: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:44:28.079 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:44:27 smithi115 ceph-mon[78593]: Detected new or changed devices on smithi115 2022-05-14T00:44:28.079 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:44:27 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' 2022-05-14T00:44:28.079 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:44:27 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-05-14T00:44:28.079 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:44:27 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-05-14T00:44:28.080 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:44:27 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-05-14T00:44:28.080 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:44:27 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-05-14T00:44:28.080 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:44:27 smithi115 ceph-mon[78593]: Adjusting osd_memory_target on smithi115 to 197.0M 2022-05-14T00:44:28.080 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:44:27 smithi115 ceph-mon[78593]: Unable to set osd_memory_target on smithi115 to 206629785: error parsing value: Value '206629785' is below minimum 939524096 2022-05-14T00:44:28.080 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:44:27 smithi115 ceph-mon[78593]: Detected new or changed devices on smithi047 2022-05-14T00:44:28.081 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:44:27 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' 2022-05-14T00:44:28.081 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:44:27 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-05-14T00:44:28.081 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:44:27 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-05-14T00:44:28.081 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:44:27 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-05-14T00:44:28.082 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:44:27 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-05-14T00:44:28.082 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:44:27 smithi115 ceph-mon[78593]: Adjusting osd_memory_target on smithi047 to 198.1M 2022-05-14T00:44:28.082 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:44:27 smithi115 ceph-mon[78593]: Unable to set osd_memory_target on smithi047 to 207755878: error parsing value: Value '207755878' is below minimum 939524096 2022-05-14T00:44:28.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:44:28 smithi115 conmon[90393]: 2022-05-14T00:44:28.062+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:44:29.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:44:28 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:44:29.082 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:44:28 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:44:29.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:44:29 smithi115 conmon[90393]: 2022-05-14T00:44:29.081+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:44:30.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:44:29 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:44:30.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:44:29 smithi047 ceph-mon[77147]: pgmap v2654: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:44:30.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:44:29 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:44:30.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:44:29 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:44:30.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:44:29 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 4601 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:44:30.055 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:44:29 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:44:30.055 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:44:29 smithi115 ceph-mon[78593]: pgmap v2654: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:44:30.055 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:44:29 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:44:30.056 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:44:29 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:44:30.056 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:44:29 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 4601 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:44:30.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:44:30 smithi115 conmon[90393]: 2022-05-14T00:44:30.054+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:44:31.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:44:30 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:44:31.096 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:44:30 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:44:31.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:44:31 smithi115 conmon[90393]: 2022-05-14T00:44:31.096+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:44:32.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:44:31 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:44:32.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:44:31 smithi047 ceph-mon[77147]: pgmap v2655: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:44:32.063 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:44:31 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:44:32.064 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:44:31 smithi115 ceph-mon[78593]: pgmap v2655: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:44:32.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:44:32 smithi115 conmon[90393]: 2022-05-14T00:44:32.062+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:44:33.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:44:32 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:44:33.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:44:32 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:44:33.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:44:33 smithi115 conmon[90393]: 2022-05-14T00:44:33.041+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:44:34.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:44:33 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:44:34.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:44:33 smithi047 ceph-mon[77147]: pgmap v2656: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:44:34.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:44:33 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:44:34.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:44:33 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:44:34.061 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:44:33 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:44:34.062 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:44:33 smithi115 ceph-mon[78593]: pgmap v2656: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:44:34.062 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:44:33 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:44:34.062 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:44:33 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:44:34.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:44:34 smithi115 conmon[90393]: 2022-05-14T00:44:34.061+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:44:35.040 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:44:34 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:44:35.040 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:44:34 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 4605 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:44:35.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:44:34 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:44:35.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:44:34 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 4605 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:44:35.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:44:35 smithi115 conmon[90393]: 2022-05-14T00:44:35.039+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:44:36.018 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:44:35 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:44:36.018 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:44:35 smithi115 ceph-mon[78593]: pgmap v2657: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:44:36.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:44:35 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:44:36.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:44:35 smithi047 ceph-mon[77147]: pgmap v2657: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:44:36.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:44:36 smithi115 conmon[90393]: 2022-05-14T00:44:36.017+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:44:37.024 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:44:36 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:44:37.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:44:36 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:44:37.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:44:37 smithi115 conmon[90393]: 2022-05-14T00:44:37.024+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:44:38.027 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:44:37 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:44:38.028 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:44:37 smithi115 ceph-mon[78593]: pgmap v2658: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:44:38.028 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:44:37 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:44:38.029 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:44:37 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:44:38.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:44:37 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:44:38.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:44:37 smithi047 ceph-mon[77147]: pgmap v2658: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:44:38.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:44:37 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:44:38.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:44:37 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:44:38.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:44:38 smithi115 conmon[90393]: 2022-05-14T00:44:38.026+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:44:39.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:44:38 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:44:39.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:44:38 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:44:39.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:44:39 smithi115 conmon[90393]: 2022-05-14T00:44:39.044+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:44:40.012 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:44:39 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:44:40.013 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:44:39 smithi115 ceph-mon[78593]: pgmap v2659: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:44:40.013 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:44:39 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 4610 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:44:40.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:44:39 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:44:40.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:44:39 smithi047 ceph-mon[77147]: pgmap v2659: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:44:40.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:44:39 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 4610 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:44:40.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:44:40 smithi115 conmon[90393]: 2022-05-14T00:44:40.012+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:44:41.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:44:40 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:44:41.060 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:44:40 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:44:41.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:44:41 smithi115 conmon[90393]: 2022-05-14T00:44:41.060+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:44:42.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:44:41 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:44:42.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:44:41 smithi047 ceph-mon[77147]: pgmap v2660: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:44:42.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:44:41 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:44:42.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:44:41 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:44:42.099 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:44:41 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:44:42.100 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:44:41 smithi115 ceph-mon[78593]: pgmap v2660: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:44:42.100 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:44:41 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:44:42.100 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:44:41 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:44:42.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:44:42 smithi115 conmon[90393]: 2022-05-14T00:44:42.098+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:44:43.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:44:42 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:44:43.075 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:44:42 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:44:43.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:44:43 smithi115 conmon[90393]: 2022-05-14T00:44:43.074+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:44:44.046 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:44:43 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:44:44.046 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:44:43 smithi047 ceph-mon[77147]: pgmap v2661: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:44:44.074 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:44:43 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:44:44.074 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:44:43 smithi115 ceph-mon[78593]: pgmap v2661: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:44:44.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:44:44 smithi115 conmon[90393]: 2022-05-14T00:44:44.072+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:44:45.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:44:44 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:44:45.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:44:44 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 4615 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:44:45.104 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:44:44 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:44:45.105 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:44:44 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 4615 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:44:45.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:44:45 smithi115 conmon[90393]: 2022-05-14T00:44:45.103+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:44:46.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:44:45 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:44:46.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:44:45 smithi047 ceph-mon[77147]: pgmap v2662: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:44:46.064 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:44:45 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:44:46.064 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:44:45 smithi115 ceph-mon[78593]: pgmap v2662: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:44:46.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:44:46 smithi115 conmon[90393]: 2022-05-14T00:44:46.062+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:44:47.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:44:46 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:44:47.056 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:44:46 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:44:47.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:44:47 smithi115 conmon[90393]: 2022-05-14T00:44:47.055+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:44:48.011 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:44:47 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:44:48.011 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:44:47 smithi115 ceph-mon[78593]: pgmap v2663: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:44:48.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:44:47 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:44:48.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:44:47 smithi047 ceph-mon[77147]: pgmap v2663: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:44:48.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:44:48 smithi115 conmon[90393]: 2022-05-14T00:44:48.009+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:44:49.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:44:48 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:44:49.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:44:48 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:44:49.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:44:49 smithi115 conmon[90393]: 2022-05-14T00:44:49.044+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:44:50.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:44:49 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:44:50.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:44:49 smithi047 ceph-mon[77147]: pgmap v2664: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:44:50.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:44:49 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:44:50.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:44:49 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:44:50.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:44:49 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 4620 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:44:50.094 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:44:49 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:44:50.095 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:44:49 smithi115 ceph-mon[78593]: pgmap v2664: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:44:50.095 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:44:49 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:44:50.095 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:44:49 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:44:50.095 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:44:49 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 4620 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:44:50.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:44:50 smithi115 conmon[90393]: 2022-05-14T00:44:50.093+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:44:51.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:44:50 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:44:51.088 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:44:50 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:44:51.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:44:51 smithi115 conmon[90393]: 2022-05-14T00:44:51.086+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:44:52.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:44:51 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:44:52.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:44:51 smithi047 ceph-mon[77147]: pgmap v2665: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:44:52.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:44:51 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:44:52.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:44:51 smithi115 ceph-mon[78593]: pgmap v2665: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:44:52.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:44:52 smithi115 conmon[90393]: 2022-05-14T00:44:52.132+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:44:53.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:44:52 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:44:53.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:44:52 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:44:53.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:44:53 smithi115 conmon[90393]: 2022-05-14T00:44:53.152+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:44:54.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:44:53 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:44:54.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:44:53 smithi047 ceph-mon[77147]: pgmap v2666: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:44:54.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:44:53 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:44:54.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:44:53 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:44:54.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:44:53 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:44:54.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:44:53 smithi115 ceph-mon[78593]: pgmap v2666: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:44:54.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:44:53 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:44:54.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:44:53 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:44:54.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:44:54 smithi115 conmon[90393]: 2022-05-14T00:44:54.182+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:44:55.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:44:54 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:44:55.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:44:54 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 4625 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:44:55.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:44:54 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:44:55.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:44:54 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 4625 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:44:55.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:44:55 smithi115 conmon[90393]: 2022-05-14T00:44:55.209+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:44:56.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:44:55 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:44:56.151 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:44:55 smithi047 ceph-mon[77147]: pgmap v2667: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:44:56.152 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:44:55 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:44:56.152 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:44:55 smithi115 ceph-mon[78593]: pgmap v2667: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:44:56.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:44:56 smithi115 conmon[90393]: 2022-05-14T00:44:56.226+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:44:57.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:44:56 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:44:57.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:44:56 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:44:57.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:44:57 smithi115 conmon[90393]: 2022-05-14T00:44:57.187+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:44:58.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:44:57 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:44:58.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:44:57 smithi047 ceph-mon[77147]: pgmap v2668: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:44:58.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:44:57 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:44:58.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:44:57 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:44:58.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:44:57 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:44:58.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:44:57 smithi115 ceph-mon[78593]: pgmap v2668: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:44:58.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:44:57 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:44:58.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:44:57 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:44:58.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:44:58 smithi115 conmon[90393]: 2022-05-14T00:44:58.172+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:44:59.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:44:58 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:44:59.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:44:58 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:44:59.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:44:59 smithi115 conmon[90393]: 2022-05-14T00:44:59.189+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:45:00.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:44:59 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:45:00.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:44:59 smithi047 ceph-mon[77147]: pgmap v2669: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:45:00.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:44:59 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 4630 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:45:00.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:44:59 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:45:00.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:44:59 smithi115 ceph-mon[78593]: pgmap v2669: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:45:00.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:44:59 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 4630 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:45:00.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:45:00 smithi115 conmon[90393]: 2022-05-14T00:45:00.145+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:45:01.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:45:00 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:45:01.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:45:00 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:45:01.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:45:01 smithi115 conmon[90393]: 2022-05-14T00:45:01.174+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:45:02.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:45:01 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:45:02.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:45:01 smithi047 ceph-mon[77147]: pgmap v2670: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:45:02.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:45:01 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:45:02.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:45:01 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:45:02.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:45:01 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:45:02.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:45:01 smithi115 ceph-mon[78593]: pgmap v2670: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:45:02.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:45:01 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:45:02.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:45:01 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:45:02.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:45:02 smithi115 conmon[90393]: 2022-05-14T00:45:02.172+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:45:03.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:45:02 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:45:03.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:45:02 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:45:03.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:45:03 smithi115 conmon[90393]: 2022-05-14T00:45:03.219+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:45:04.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:45:03 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:45:04.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:45:03 smithi047 ceph-mon[77147]: pgmap v2671: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:45:04.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:45:03 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:45:04.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:45:03 smithi115 ceph-mon[78593]: pgmap v2671: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:45:04.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:45:04 smithi115 conmon[90393]: 2022-05-14T00:45:04.212+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:45:05.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:45:04 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:45:05.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:45:04 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 4635 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:45:05.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:45:04 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:45:05.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:45:04 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 4635 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:45:05.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:45:05 smithi115 conmon[90393]: 2022-05-14T00:45:05.208+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:45:06.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:45:05 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:45:06.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:45:05 smithi047 ceph-mon[77147]: pgmap v2672: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:45:06.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:45:05 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:45:06.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:45:05 smithi115 ceph-mon[78593]: pgmap v2672: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:45:06.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:45:06 smithi115 conmon[90393]: 2022-05-14T00:45:06.204+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:45:07.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:45:06 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:45:07.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:45:06 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:45:07.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:45:07 smithi115 conmon[90393]: 2022-05-14T00:45:07.209+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:45:08.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:45:07 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:45:08.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:45:07 smithi047 ceph-mon[77147]: pgmap v2673: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:45:08.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:45:07 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:45:08.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:45:07 smithi115 ceph-mon[78593]: pgmap v2673: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:45:08.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:45:08 smithi115 conmon[90393]: 2022-05-14T00:45:08.228+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:45:09.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:45:08 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:45:09.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:45:08 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:45:09.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:45:09 smithi115 conmon[90393]: 2022-05-14T00:45:09.184+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:45:10.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:45:09 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:45:10.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:45:09 smithi047 ceph-mon[77147]: pgmap v2674: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:45:10.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:45:09 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:45:10.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:45:09 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:45:10.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:45:09 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 4640 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:45:10.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:45:09 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:45:10.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:45:09 smithi115 ceph-mon[78593]: pgmap v2674: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:45:10.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:45:09 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:45:10.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:45:09 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:45:10.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:45:09 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 4640 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:45:10.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:45:10 smithi115 conmon[90393]: 2022-05-14T00:45:10.154+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:45:11.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:45:10 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:45:11.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:45:10 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:45:11.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:45:11 smithi115 conmon[90393]: 2022-05-14T00:45:11.127+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:45:12.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:45:11 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:45:12.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:45:11 smithi047 ceph-mon[77147]: pgmap v2675: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:45:12.106 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:45:11 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:45:12.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:45:11 smithi115 ceph-mon[78593]: pgmap v2675: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:45:12.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:45:12 smithi115 conmon[90393]: 2022-05-14T00:45:12.106+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:45:13.064 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:45:12 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:45:13.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:45:12 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:45:13.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:45:13 smithi115 conmon[90393]: 2022-05-14T00:45:13.063+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:45:14.085 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:45:13 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:45:14.086 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:45:13 smithi115 ceph-mon[78593]: pgmap v2676: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:45:14.086 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:45:13 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:45:14.086 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:45:13 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:45:14.086 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:45:13 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T00:45:14.087 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:45:13 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T00:45:14.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:45:13 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:45:14.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:45:13 smithi047 ceph-mon[77147]: pgmap v2676: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:45:14.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:45:13 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:45:14.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:45:13 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:45:14.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:45:13 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T00:45:14.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:45:13 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T00:45:14.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:45:14 smithi115 conmon[90393]: 2022-05-14T00:45:14.084+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:45:15.096 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:45:14 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:45:15.096 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:45:14 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 4645 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:45:15.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:45:14 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:45:15.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:45:14 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 4645 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:45:15.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:45:15 smithi115 conmon[90393]: 2022-05-14T00:45:15.095+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:45:16.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:45:15 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:45:16.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:45:15 smithi115 ceph-mon[78593]: pgmap v2677: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:45:16.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:45:15 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:45:16.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:45:15 smithi047 ceph-mon[77147]: pgmap v2677: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:45:16.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:45:16 smithi115 conmon[90393]: 2022-05-14T00:45:16.135+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:45:17.093 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:45:16 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:45:17.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:45:16 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:45:17.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:45:17 smithi115 conmon[90393]: 2022-05-14T00:45:17.092+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:45:18.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:45:17 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:45:18.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:45:17 smithi115 ceph-mon[78593]: pgmap v2678: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:45:18.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:45:17 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:45:18.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:45:17 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:45:18.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:45:17 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:45:18.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:45:17 smithi047 ceph-mon[77147]: pgmap v2678: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:45:18.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:45:17 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:45:18.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:45:17 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:45:18.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:45:18 smithi115 conmon[90393]: 2022-05-14T00:45:18.139+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:45:19.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:45:18 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:45:19.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:45:18 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:45:19.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:45:19 smithi115 conmon[90393]: 2022-05-14T00:45:19.145+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:45:20.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:45:19 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:45:20.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:45:19 smithi115 ceph-mon[78593]: pgmap v2679: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:45:20.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:45:19 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 4650 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:45:20.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:45:19 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:45:20.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:45:19 smithi047 ceph-mon[77147]: pgmap v2679: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:45:20.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:45:19 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 4650 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:45:20.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:45:20 smithi115 conmon[90393]: 2022-05-14T00:45:20.125+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:45:21.088 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:45:20 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:45:21.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:45:20 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:45:21.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:45:21 smithi115 conmon[90393]: 2022-05-14T00:45:21.087+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:45:22.091 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:45:21 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:45:22.092 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:45:21 smithi115 ceph-mon[78593]: pgmap v2680: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:45:22.092 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:45:21 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:45:22.092 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:45:21 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:45:22.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:45:21 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:45:22.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:45:21 smithi047 ceph-mon[77147]: pgmap v2680: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:45:22.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:45:21 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:45:22.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:45:21 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:45:22.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:45:22 smithi115 conmon[90393]: 2022-05-14T00:45:22.090+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:45:23.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:45:22 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:45:23.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:45:22 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:45:23.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:45:23 smithi115 conmon[90393]: 2022-05-14T00:45:23.137+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:45:24.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:45:23 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:45:24.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:45:23 smithi115 ceph-mon[78593]: pgmap v2681: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:45:24.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:45:23 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:45:24.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:45:23 smithi047 ceph-mon[77147]: pgmap v2681: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:45:24.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:45:24 smithi115 conmon[90393]: 2022-05-14T00:45:24.150+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:45:25.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:45:24 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:45:25.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:45:24 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 4656 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:45:25.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:45:24 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:45:25.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:45:24 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 4656 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:45:25.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:45:25 smithi115 conmon[90393]: 2022-05-14T00:45:25.122+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:45:26.100 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:45:25 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:45:26.100 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:45:25 smithi115 ceph-mon[78593]: pgmap v2682: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:45:26.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:45:25 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:45:26.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:45:25 smithi047 ceph-mon[77147]: pgmap v2682: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:45:26.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:45:26 smithi115 conmon[90393]: 2022-05-14T00:45:26.099+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:45:27.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:45:26 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:45:27.217 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:45:26 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:45:27.540 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:45:27 smithi047 conmon[78521]: 2022-05-14T00:45:27.216+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:45:27.541 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:45:27 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T00:45:27.541 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:45:27 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:45:27.541 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:45:27 smithi047 conmon[78521]: yield 2022-05-14T00:45:27.542 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:45:27 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:45:27.542 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:45:27 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:45:27.542 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:45:27 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:45:27.542 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:45:27 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T00:45:27.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:45:27 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:45:27.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:45:27 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T00:45:27.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:45:27 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:45:27.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:45:27 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T00:45:27.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:45:27 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:45:27.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:45:27 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T00:45:27.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:45:27 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:45:27.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:45:27 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T00:45:27.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:45:27 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:45:27.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:45:27 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:45:27.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:45:27 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:45:27.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:45:27 smithi115 conmon[90393]: 2022-05-14T00:45:27.113+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:45:28.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:45:27 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:45:28.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:45:27 smithi115 ceph-mon[78593]: pgmap v2683: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:45:28.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:45:27 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T00:45:28.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:45:27 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T00:45:28.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:45:27 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T00:45:28.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:45:27 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:45:28.109 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T00:45:28.109 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:45:28.109 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T00:45:28.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:45:28.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:45:28.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:45:28.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T00:45:28.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:45:28.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T00:45:28.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:45:28.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T00:45:28.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:45:28.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T00:45:28.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:45:28.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T00:45:28.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:45:28.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:45:28.113 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:45:28.113 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:45:27 smithi115 ceph-mon[78593]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T00:45:28.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:45:27 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:45:28.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:45:27 smithi047 ceph-mon[77147]: pgmap v2683: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:45:28.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:45:27 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T00:45:28.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:45:27 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T00:45:28.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:45:27 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T00:45:28.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:45:27 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:45:28.292 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T00:45:28.292 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:45:28.292 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T00:45:28.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:45:28.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:45:28.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:45:28.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T00:45:28.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:45:28.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T00:45:28.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:45:28.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T00:45:28.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:45:28.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T00:45:28.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:45:28.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T00:45:28.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:45:28.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:45:28.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:45:28.296 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:45:27 smithi047 ceph-mon[77147]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T00:45:28.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:45:28 smithi115 conmon[90393]: 2022-05-14T00:45:28.133+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:45:29.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:45:28 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:45:29.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:45:28 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:45:29.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:45:29 smithi115 conmon[90393]: 2022-05-14T00:45:29.143+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:45:30.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:45:29 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:45:30.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:45:29 smithi115 ceph-mon[78593]: pgmap v2684: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:45:30.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:45:29 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:45:30.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:45:29 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:45:30.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:45:29 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 4661 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:45:30.289 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:45:29 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:45:30.289 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:45:29 smithi047 ceph-mon[77147]: pgmap v2684: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:45:30.289 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:45:29 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:45:30.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:45:29 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:45:30.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:45:29 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 4661 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:45:30.541 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:45:30 smithi047 conmon[78521]: 2022-05-14T00:45:30.288+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:45:30.542 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:45:30 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T00:45:30.542 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:45:30 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:45:30.542 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:45:30 smithi047 conmon[78521]: yield 2022-05-14T00:45:30.542 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:45:30 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:45:30.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:45:30 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:45:30.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:45:30 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:45:30.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:45:30 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T00:45:30.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:45:30 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:45:30.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:45:30 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T00:45:30.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:45:30 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:45:30.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:45:30 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T00:45:30.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:45:30 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:45:30.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:45:30 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T00:45:30.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:45:30 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:45:30.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:45:30 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T00:45:30.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:45:30 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:45:30.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:45:30 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:45:30.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:45:30 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:45:30.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:45:30 smithi047 conmon[78521]: 2022-05-14T00:45:30.289+0000 7f5340f7a700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T00:45:30.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:45:30 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T00:45:30.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:45:30 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:45:30.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:45:30 smithi047 conmon[78521]: yield 2022-05-14T00:45:30.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:45:30 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:45:30.547 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:45:30 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:45:30.547 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:45:30 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:45:30.547 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:45:30 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T00:45:30.547 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:45:30 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:45:30.547 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:45:30 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T00:45:30.548 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:45:30 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:45:30.548 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:45:30 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T00:45:30.548 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:45:30 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:45:30.548 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:45:30 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T00:45:30.549 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:45:30 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:45:30.549 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:45:30 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T00:45:30.549 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:45:30 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:45:30.549 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:45:30 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:45:30.549 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:45:30 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:45:30.550 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:45:30 smithi047 conmon[78521]: 2022-05-14T00:45:30.550 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:45:30 smithi047 conmon[78521]: During handling of the above exception, another exception occurred: 2022-05-14T00:45:30.550 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:45:30 smithi047 conmon[78521]: 2022-05-14T00:45:30.550 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:45:30 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T00:45:30.551 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:45:30 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T00:45:30.551 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:45:30 smithi047 conmon[78521]: return f(*arg) 2022-05-14T00:45:30.551 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:45:30 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T00:45:30.551 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:45:30 smithi047 conmon[78521]: self._write_client_files(client_files, host) 2022-05-14T00:45:30.551 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:45:30 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T00:45:30.552 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:45:30 smithi047 conmon[78521]: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T00:45:30.552 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:45:30 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T00:45:30.552 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:45:30 smithi047 conmon[78521]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T00:45:30.552 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:45:30 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T00:45:30.553 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:45:30 smithi047 conmon[78521]: return self.event_loop.get_result(coro) 2022-05-14T00:45:30.553 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:45:30 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T00:45:30.553 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:45:30 smithi047 conmon[78521]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T00:45:30.553 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:45:30 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T00:45:30.554 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:45:30 smithi047 conmon[78521]: return self.__get_result() 2022-05-14T00:45:30.554 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:45:30 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T00:45:30.554 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:45:30 smithi047 conmon[78521]: raise self._exception 2022-05-14T00:45:30.554 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:45:30 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T00:45:30.555 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:45:30 smithi047 conmon[78521]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T00:45:30.555 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:45:30 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T00:45:30.555 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:45:30 smithi047 conmon[78521]: conn = await self._remote_connection(host, addr) 2022-05-14T00:45:30.555 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:45:30 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T00:45:30.555 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:45:30 smithi047 conmon[78521]: raise 2022-05-14T00:45:30.556 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:45:30 smithi047 conmon[78521]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T00:45:30.556 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:45:30 smithi047 conmon[78521]: self.gen.throw(type, value, traceback) 2022-05-14T00:45:30.556 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:45:30 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T00:45:30.556 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:45:30 smithi047 conmon[78521]: raise OrchestratorError(msg) 2022-05-14T00:45:30.557 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:45:30 smithi047 conmon[78521]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:45:30.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:45:30 smithi115 conmon[90393]: 2022-05-14T00:45:30.114+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:45:31.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:45:31 smithi115 conmon[90393]: 2022-05-14T00:45:31.089+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:45:31.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:45:30 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:45:31.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:45:30 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:45:32.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:45:31 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:45:32.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:45:31 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:45:32.108 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T00:45:32.109 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:45:32.109 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T00:45:32.109 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:45:32.109 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:45:32.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:45:32.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T00:45:32.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:45:32.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T00:45:32.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:45:32.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T00:45:32.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:45:32.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T00:45:32.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:45:32.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T00:45:32.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:45:32.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:45:32.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:45:32.113 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:45:31 smithi115 ceph-mon[78593]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T00:45:32.113 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T00:45:32.113 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:45:32.113 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T00:45:32.114 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:45:32.114 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:45:32.114 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:45:32.114 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T00:45:32.115 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:45:32.115 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T00:45:32.115 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:45:32.115 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T00:45:32.116 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:45:32.116 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T00:45:32.116 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:45:32.116 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T00:45:32.116 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:45:32.117 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:45:32.117 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:45:32.117 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T00:45:32.117 INFO:journalctl@ceph.mon.c.smithi115.stdout: During handling of the above exception, another exception occurred: 2022-05-14T00:45:32.118 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T00:45:32.118 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T00:45:32.118 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T00:45:32.118 INFO:journalctl@ceph.mon.c.smithi115.stdout: return f(*arg) 2022-05-14T00:45:32.118 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T00:45:32.119 INFO:journalctl@ceph.mon.c.smithi115.stdout: self._write_client_files(client_files, host) 2022-05-14T00:45:32.119 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T00:45:32.119 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T00:45:32.119 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T00:45:32.120 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T00:45:32.120 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T00:45:32.120 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.event_loop.get_result(coro) 2022-05-14T00:45:32.120 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T00:45:32.121 INFO:journalctl@ceph.mon.c.smithi115.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T00:45:32.121 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T00:45:32.121 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.__get_result() 2022-05-14T00:45:32.121 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T00:45:32.122 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise self._exception 2022-05-14T00:45:32.122 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T00:45:32.122 INFO:journalctl@ceph.mon.c.smithi115.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T00:45:32.122 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T00:45:32.122 INFO:journalctl@ceph.mon.c.smithi115.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T00:45:32.123 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T00:45:32.123 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise 2022-05-14T00:45:32.123 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T00:45:32.123 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.gen.throw(type, value, traceback) 2022-05-14T00:45:32.124 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T00:45:32.124 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OrchestratorError(msg) 2022-05-14T00:45:32.124 INFO:journalctl@ceph.mon.c.smithi115.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:45:32.125 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:45:31 smithi115 ceph-mon[78593]: pgmap v2685: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:45:32.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:45:31 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:45:32.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:45:31 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:45:32.291 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T00:45:32.292 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:45:32.292 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T00:45:32.292 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:45:32.292 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:45:32.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:45:32.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T00:45:32.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:45:32.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T00:45:32.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:45:32.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T00:45:32.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:45:32.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T00:45:32.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:45:32.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T00:45:32.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:45:32.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:45:32.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:45:32.297 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:45:31 smithi047 ceph-mon[77147]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T00:45:32.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T00:45:32.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:45:32.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T00:45:32.298 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:45:32.298 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:45:32.298 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:45:32.298 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T00:45:32.298 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:45:32.299 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T00:45:32.299 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:45:32.299 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T00:45:32.299 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:45:32.299 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T00:45:32.300 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:45:32.300 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T00:45:32.300 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:45:32.300 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:45:32.300 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:45:32.301 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T00:45:32.301 INFO:journalctl@ceph.mon.b.smithi047.stdout: During handling of the above exception, another exception occurred: 2022-05-14T00:45:32.301 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T00:45:32.301 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T00:45:32.302 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T00:45:32.302 INFO:journalctl@ceph.mon.b.smithi047.stdout: return f(*arg) 2022-05-14T00:45:32.302 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T00:45:32.302 INFO:journalctl@ceph.mon.b.smithi047.stdout: self._write_client_files(client_files, host) 2022-05-14T00:45:32.302 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T00:45:32.303 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T00:45:32.303 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T00:45:32.303 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T00:45:32.303 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T00:45:32.303 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.event_loop.get_result(coro) 2022-05-14T00:45:32.304 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T00:45:32.304 INFO:journalctl@ceph.mon.b.smithi047.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T00:45:32.304 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T00:45:32.304 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.__get_result() 2022-05-14T00:45:32.305 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T00:45:32.305 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise self._exception 2022-05-14T00:45:32.305 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T00:45:32.305 INFO:journalctl@ceph.mon.b.smithi047.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T00:45:32.305 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T00:45:32.306 INFO:journalctl@ceph.mon.b.smithi047.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T00:45:32.306 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T00:45:32.306 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise 2022-05-14T00:45:32.306 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T00:45:32.307 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.gen.throw(type, value, traceback) 2022-05-14T00:45:32.307 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T00:45:32.307 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OrchestratorError(msg) 2022-05-14T00:45:32.307 INFO:journalctl@ceph.mon.b.smithi047.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:45:32.308 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:45:31 smithi047 ceph-mon[77147]: pgmap v2685: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:45:32.612 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:45:32 smithi115 conmon[90393]: 2022-05-14T00:45:32.131+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:45:33.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:45:32 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:45:33.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:45:33 smithi115 conmon[90393]: 2022-05-14T00:45:33.088+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:45:33.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:45:32 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:45:34.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:45:33 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:45:34.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:45:33 smithi115 ceph-mon[78593]: pgmap v2686: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:45:34.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:45:33 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:45:34.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:45:33 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:45:34.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:45:33 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:45:34.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:45:33 smithi047 ceph-mon[77147]: pgmap v2686: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:45:34.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:45:33 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:45:34.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:45:33 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:45:34.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:45:34 smithi115 conmon[90393]: 2022-05-14T00:45:34.119+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:45:35.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:45:34 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:45:35.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:45:34 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 4666 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:45:35.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:45:35 smithi115 conmon[90393]: 2022-05-14T00:45:35.098+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:45:35.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:45:34 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:45:35.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:45:34 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 4666 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:45:36.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:45:36 smithi115 conmon[90393]: 2022-05-14T00:45:36.086+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:45:36.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:45:35 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:45:36.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:45:35 smithi115 ceph-mon[78593]: pgmap v2687: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:45:36.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:45:35 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:45:36.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:45:35 smithi047 ceph-mon[77147]: pgmap v2687: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:45:37.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:45:36 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:45:37.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:45:37 smithi115 conmon[90393]: 2022-05-14T00:45:37.073+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:45:37.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:45:36 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:45:38.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:45:37 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:45:38.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:45:37 smithi047 ceph-mon[77147]: pgmap v2688: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:45:38.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:45:37 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:45:38.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:45:37 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:45:38.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:45:38 smithi115 conmon[90393]: 2022-05-14T00:45:38.069+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:45:38.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:45:37 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:45:38.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:45:37 smithi115 ceph-mon[78593]: pgmap v2688: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:45:38.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:45:37 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:45:38.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:45:37 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:45:39.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:45:38 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:45:39.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:45:39 smithi115 conmon[90393]: 2022-05-14T00:45:39.107+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:45:39.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:45:38 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:45:40.141 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:45:39 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:45:40.142 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:45:39 smithi115 ceph-mon[78593]: pgmap v2689: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:45:40.142 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:45:39 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 4671 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:45:40.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:45:39 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:45:40.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:45:39 smithi047 ceph-mon[77147]: pgmap v2689: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:45:40.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:45:39 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 4671 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:45:40.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:45:40 smithi115 conmon[90393]: 2022-05-14T00:45:40.140+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:45:41.152 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:45:40 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:45:41.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:45:40 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:45:41.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:45:41 smithi115 conmon[90393]: 2022-05-14T00:45:41.151+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:45:42.198 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:45:41 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:45:42.235 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:45:41 smithi115 ceph-mon[78593]: pgmap v2690: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:45:42.236 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:45:41 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:45:42.236 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:45:41 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:45:42.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:45:41 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:45:42.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:45:41 smithi047 ceph-mon[77147]: pgmap v2690: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:45:42.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:45:41 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:45:42.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:45:41 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:45:42.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:45:42 smithi115 conmon[90393]: 2022-05-14T00:45:42.196+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:45:43.228 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:45:42 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:45:43.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:45:42 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:45:43.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:45:43 smithi115 conmon[90393]: 2022-05-14T00:45:43.226+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:45:44.230 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:45:43 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:45:44.231 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:45:43 smithi115 ceph-mon[78593]: pgmap v2691: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:45:44.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:45:43 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:45:44.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:45:43 smithi047 ceph-mon[77147]: pgmap v2691: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:45:44.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:45:44 smithi115 conmon[90393]: 2022-05-14T00:45:44.229+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:45:45.269 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:45:44 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:45:45.269 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:45:44 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 4676 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:45:45.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:45:44 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:45:45.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:45:44 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 4676 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:45:45.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:45:45 smithi115 conmon[90393]: 2022-05-14T00:45:45.268+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:45:46.265 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:45:45 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:45:46.266 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:45:45 smithi115 ceph-mon[78593]: pgmap v2692: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:45:46.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:45:45 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:45:46.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:45:45 smithi047 ceph-mon[77147]: pgmap v2692: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:45:46.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:45:46 smithi115 conmon[90393]: 2022-05-14T00:45:46.264+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:45:47.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:45:46 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:45:47.300 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:45:46 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:45:47.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:45:47 smithi115 conmon[90393]: 2022-05-14T00:45:47.299+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:45:48.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:45:47 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:45:48.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:45:47 smithi047 ceph-mon[77147]: pgmap v2693: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:45:48.312 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:45:47 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:45:48.312 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:45:47 smithi115 ceph-mon[78593]: pgmap v2693: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:45:48.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:45:48 smithi115 conmon[90393]: 2022-05-14T00:45:48.310+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:45:49.274 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:45:48 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:45:49.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:45:48 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:45:49.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:45:49 smithi115 conmon[90393]: 2022-05-14T00:45:49.272+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:45:50.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:45:49 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:45:50.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:45:49 smithi047 ceph-mon[77147]: pgmap v2694: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:45:50.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:45:49 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:45:50.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:45:49 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:45:50.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:45:49 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 4680 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:45:50.306 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:45:49 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:45:50.307 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:45:49 smithi115 ceph-mon[78593]: pgmap v2694: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:45:50.307 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:45:49 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:45:50.307 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:45:49 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:45:50.308 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:45:49 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 4680 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:45:50.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:45:50 smithi115 conmon[90393]: 2022-05-14T00:45:50.305+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:45:51.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:45:50 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:45:51.341 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:45:50 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:45:51.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:45:51 smithi115 conmon[90393]: 2022-05-14T00:45:51.340+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:45:52.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:45:51 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:45:52.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:45:51 smithi047 ceph-mon[77147]: pgmap v2695: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:45:52.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:45:51 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:45:52.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:45:51 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:45:52.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:45:51 smithi115 ceph-mon[78593]: pgmap v2695: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:45:52.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:45:51 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:45:52.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:45:52 smithi115 conmon[90393]: 2022-05-14T00:45:52.373+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:45:53.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:45:52 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:45:53.346 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:45:52 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:45:53.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:45:53 smithi115 conmon[90393]: 2022-05-14T00:45:53.346+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:45:54.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:45:53 smithi047 ceph-mon[77147]: pgmap v2696: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:45:54.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:45:53 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:45:54.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:45:53 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:45:54.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:45:53 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:45:54.306 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:45:53 smithi115 ceph-mon[78593]: pgmap v2696: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:45:54.306 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:45:53 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:45:54.307 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:45:53 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:45:54.307 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:45:53 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:45:54.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:45:54 smithi115 conmon[90393]: 2022-05-14T00:45:54.305+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:45:55.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:45:54 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 4685 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:45:55.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:45:54 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:45:55.327 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:45:54 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 4685 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:45:55.328 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:45:54 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:45:55.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:45:55 smithi115 conmon[90393]: 2022-05-14T00:45:55.326+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:45:56.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:45:56 smithi047 ceph-mon[77147]: pgmap v2697: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:45:56.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:45:56 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:45:56.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:45:56 smithi115 conmon[90393]: 2022-05-14T00:45:56.308+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:45:56.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:45:56 smithi115 ceph-mon[78593]: pgmap v2697: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:45:56.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:45:56 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:45:57.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:45:57 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:45:57.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:45:57 smithi115 conmon[90393]: 2022-05-14T00:45:57.266+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:45:57.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:45:57 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:45:58.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:45:58 smithi047 ceph-mon[77147]: pgmap v2698: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:45:58.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:45:58 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:45:58.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:45:58 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:45:58.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:45:58 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:45:58.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:45:58 smithi115 conmon[90393]: 2022-05-14T00:45:58.290+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:45:58.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:45:58 smithi115 ceph-mon[78593]: pgmap v2698: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:45:58.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:45:58 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:45:58.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:45:58 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:45:58.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:45:58 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:45:59.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:45:59 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:45:59.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:45:59 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 4690 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:45:59.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:45:59 smithi115 conmon[90393]: 2022-05-14T00:45:59.254+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:45:59.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:45:59 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:45:59.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:45:59 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 4690 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:46:00.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:46:00 smithi047 ceph-mon[77147]: pgmap v2699: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:46:00.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:46:00 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:46:00.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:46:00 smithi115 conmon[90393]: 2022-05-14T00:46:00.227+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:46:00.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:46:00 smithi115 ceph-mon[78593]: pgmap v2699: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:46:00.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:46:00 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:46:01.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:46:01 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:46:01.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:46:01 smithi115 conmon[90393]: 2022-05-14T00:46:01.211+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:46:01.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:46:01 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:46:02.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:46:02 smithi047 ceph-mon[77147]: pgmap v2700: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:46:02.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:46:02 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:46:02.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:46:02 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:46:02.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:46:02 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:46:02.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:46:02 smithi115 conmon[90393]: 2022-05-14T00:46:02.195+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:46:02.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:46:02 smithi115 ceph-mon[78593]: pgmap v2700: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:46:02.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:46:02 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:46:02.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:46:02 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:46:02.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:46:02 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:46:03.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:46:03 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:46:03.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:46:03 smithi115 conmon[90393]: 2022-05-14T00:46:03.157+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:46:03.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:46:03 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:46:04.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:46:04 smithi047 ceph-mon[77147]: pgmap v2701: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:46:04.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:46:04 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:46:04.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:46:04 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 4695 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:46:04.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:46:04 smithi115 conmon[90393]: 2022-05-14T00:46:04.144+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:46:04.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:46:04 smithi115 ceph-mon[78593]: pgmap v2701: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:46:04.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:46:04 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:46:04.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:46:04 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 4695 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:46:05.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:46:05 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:46:05.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:46:05 smithi115 conmon[90393]: 2022-05-14T00:46:05.149+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:46:05.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:46:05 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:46:06.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:46:06 smithi047 ceph-mon[77147]: pgmap v2702: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:46:06.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:46:06 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:46:06.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:46:06 smithi115 conmon[90393]: 2022-05-14T00:46:06.152+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:46:06.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:46:06 smithi115 ceph-mon[78593]: pgmap v2702: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:46:06.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:46:06 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:46:07.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:46:07 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:46:07.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:46:07 smithi115 conmon[90393]: 2022-05-14T00:46:07.192+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:46:07.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:46:07 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:46:08.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:46:08 smithi047 ceph-mon[77147]: pgmap v2703: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:46:08.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:46:08 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:46:08.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:46:08 smithi115 conmon[90393]: 2022-05-14T00:46:08.231+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:46:08.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:46:08 smithi115 ceph-mon[78593]: pgmap v2703: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:46:08.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:46:08 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:46:09.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:46:09 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:46:09.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:46:09 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 4701 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:46:09.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:46:09 smithi115 conmon[90393]: 2022-05-14T00:46:09.242+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:46:09.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:46:09 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:46:09.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:46:09 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 4701 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:46:10.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:46:10 smithi047 ceph-mon[77147]: pgmap v2704: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:46:10.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:46:10 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:46:10.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:46:10 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:46:10.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:46:10 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:46:10.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:46:10 smithi115 conmon[90393]: 2022-05-14T00:46:10.238+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:46:10.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:46:10 smithi115 ceph-mon[78593]: pgmap v2704: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:46:10.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:46:10 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:46:10.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:46:10 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:46:10.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:46:10 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:46:11.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:46:11 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:46:11.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:46:11 smithi115 conmon[90393]: 2022-05-14T00:46:11.262+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:46:11.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:46:11 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:46:12.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:46:12 smithi047 ceph-mon[77147]: pgmap v2705: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:46:12.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:46:12 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:46:12.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:46:12 smithi115 conmon[90393]: 2022-05-14T00:46:12.277+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:46:12.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:46:12 smithi115 ceph-mon[78593]: pgmap v2705: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:46:12.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:46:12 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:46:13.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:46:13 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:46:13.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:46:13 smithi047 ceph-mon[77147]: pgmap v2706: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:46:13.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:46:13 smithi115 conmon[90393]: 2022-05-14T00:46:13.257+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:46:13.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:46:13 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:46:13.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:46:13 smithi115 ceph-mon[78593]: pgmap v2706: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:46:14.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:46:14 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:46:14.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:46:14 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:46:14.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:46:14 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:46:14.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:46:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T00:46:14.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:46:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T00:46:14.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:46:14 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 4705 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:46:14.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:46:14 smithi115 conmon[90393]: 2022-05-14T00:46:14.270+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:46:14.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:46:14 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:46:14.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:46:14 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:46:14.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:46:14 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:46:14.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:46:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T00:46:14.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:46:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T00:46:14.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:46:14 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 4705 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:46:15.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:46:15 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:46:15.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:46:15 smithi047 ceph-mon[77147]: pgmap v2707: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:46:15.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:46:15 smithi115 conmon[90393]: 2022-05-14T00:46:15.309+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:46:15.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:46:15 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:46:15.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:46:15 smithi115 ceph-mon[78593]: pgmap v2707: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:46:16.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:46:16 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:46:16.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:46:16 smithi115 conmon[90393]: 2022-05-14T00:46:16.320+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:46:16.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:46:16 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:46:17.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:46:17 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:46:17.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:46:17 smithi047 ceph-mon[77147]: pgmap v2708: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:46:17.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:46:17 smithi115 conmon[90393]: 2022-05-14T00:46:17.361+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:46:17.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:46:17 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:46:17.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:46:17 smithi115 ceph-mon[78593]: pgmap v2708: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:46:18.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:46:18 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:46:18.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:46:18 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:46:18.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:46:18 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:46:18.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:46:18 smithi115 conmon[90393]: 2022-05-14T00:46:18.410+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:46:18.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:46:18 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:46:18.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:46:18 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:46:18.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:46:18 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:46:19.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:46:19 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:46:19.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:46:19 smithi047 ceph-mon[77147]: pgmap v2709: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:46:19.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:46:19 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 4710 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:46:19.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:46:19 smithi115 conmon[90393]: 2022-05-14T00:46:19.374+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:46:19.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:46:19 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:46:19.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:46:19 smithi115 ceph-mon[78593]: pgmap v2709: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:46:19.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:46:19 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 4710 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:46:20.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:46:20 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:46:20.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:46:20 smithi115 conmon[90393]: 2022-05-14T00:46:20.383+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:46:20.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:46:20 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:46:21.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:46:21 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:46:21.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:46:21 smithi047 ceph-mon[77147]: pgmap v2710: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:46:21.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:46:21 smithi115 conmon[90393]: 2022-05-14T00:46:21.395+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:46:21.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:46:21 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:46:21.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:46:21 smithi115 ceph-mon[78593]: pgmap v2710: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:46:22.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:46:22 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:46:22.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:46:22 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:46:22.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:46:22 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:46:22.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:46:22 smithi115 conmon[90393]: 2022-05-14T00:46:22.427+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:46:22.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:46:22 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:46:22.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:46:22 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:46:22.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:46:22 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:46:23.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:46:23 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:46:23.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:46:23 smithi047 ceph-mon[77147]: pgmap v2711: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:46:23.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:46:23 smithi115 conmon[90393]: 2022-05-14T00:46:23.462+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:46:23.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:46:23 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:46:23.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:46:23 smithi115 ceph-mon[78593]: pgmap v2711: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:46:24.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:46:24 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:46:24.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:46:24 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 4715 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:46:24.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:46:24 smithi115 conmon[90393]: 2022-05-14T00:46:24.464+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:46:24.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:46:24 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:46:24.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:46:24 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 4715 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:46:25.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:46:25 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:46:25.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:46:25 smithi047 ceph-mon[77147]: pgmap v2712: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:46:25.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:46:25 smithi115 conmon[90393]: 2022-05-14T00:46:25.477+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:46:25.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:46:25 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:46:25.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:46:25 smithi115 ceph-mon[78593]: pgmap v2712: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:46:26.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:46:26 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:46:26.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:46:26 smithi115 conmon[90393]: 2022-05-14T00:46:26.496+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:46:26.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:46:26 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:46:27.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:46:27 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:46:27.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:46:27 smithi047 ceph-mon[77147]: pgmap v2713: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:46:27.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:46:27 smithi115 conmon[90393]: 2022-05-14T00:46:27.536+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:46:27.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:46:27 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:46:27.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:46:27 smithi115 ceph-mon[78593]: pgmap v2713: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:46:28.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:46:28 smithi115 conmon[90393]: 2022-05-14T00:46:28.492+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:46:28.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:46:28 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:46:28.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:46:28 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:46:29.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:46:29 smithi115 conmon[90393]: 2022-05-14T00:46:29.508+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:46:29.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:46:29 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:46:29.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:46:29 smithi115 ceph-mon[78593]: pgmap v2714: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:46:29.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:46:29 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 4720 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:46:29.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:46:29 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:46:29.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:46:29 smithi047 ceph-mon[77147]: pgmap v2714: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:46:29.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:46:29 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 4720 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:46:30.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:46:30 smithi115 conmon[90393]: 2022-05-14T00:46:30.533+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:46:30.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:46:30 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:46:30.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:46:30 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:46:30.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:46:30 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:46:30.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:46:30 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T00:46:30.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:46:30 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T00:46:30.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:46:30 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T00:46:30.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:46:30 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:46:30.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:46:30 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:46:30.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:46:30 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:46:30.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:46:30 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T00:46:30.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:46:30 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T00:46:30.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:46:30 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T00:46:31.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:46:31 smithi115 conmon[90393]: 2022-05-14T00:46:31.547+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:46:31.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:46:31 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:46:31.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:46:31 smithi115 ceph-mon[78593]: pgmap v2715: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:46:31.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:46:31 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:46:31.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:46:31 smithi047 ceph-mon[77147]: pgmap v2715: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:46:32.560 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:46:32 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:46:32.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:46:32 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:46:32.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:46:32 smithi115 conmon[90393]: 2022-05-14T00:46:32.559+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:46:33.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:46:33 smithi115 conmon[90393]: 2022-05-14T00:46:33.521+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:46:33.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:46:33 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:46:33.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:46:33 smithi115 ceph-mon[78593]: pgmap v2716: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:46:33.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:46:33 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:46:33.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:46:33 smithi047 ceph-mon[77147]: pgmap v2716: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:46:33.792 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:46:33 smithi047 conmon[78521]: 2022-05-14T00:46:33.391+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:46:33.792 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:46:33 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T00:46:33.792 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:46:33 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:46:33.792 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:46:33 smithi047 conmon[78521]: yield 2022-05-14T00:46:33.793 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:46:33 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:46:33.793 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:46:33 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:46:33.793 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:46:33 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:46:33.793 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:46:33 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T00:46:33.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:46:33 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:46:33.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:46:33 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T00:46:33.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:46:33 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:46:33.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:46:33 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T00:46:33.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:46:33 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:46:33.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:46:33 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T00:46:33.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:46:33 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:46:33.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:46:33 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T00:46:33.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:46:33 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:46:33.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:46:33 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:46:33.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:46:33 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:46:34.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:46:34 smithi115 conmon[90393]: 2022-05-14T00:46:34.496+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:46:34.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:46:34 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:46:34.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:46:34 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:46:34.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:46:34 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:46:34.609 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T00:46:34.609 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:46:34.609 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T00:46:34.609 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:46:34.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:46:34.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:46:34.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T00:46:34.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:46:34.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T00:46:34.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:46:34.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T00:46:34.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:46:34.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T00:46:34.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:46:34.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T00:46:34.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:46:34.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:46:34.613 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:46:34.613 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:46:34 smithi115 ceph-mon[78593]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T00:46:34.613 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:46:34 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:46:34.613 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:46:34 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 4725 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:46:34.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:46:34 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:46:34.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:46:34 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:46:34.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:46:34 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:46:34.791 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T00:46:34.792 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:46:34.792 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T00:46:34.792 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:46:34.792 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:46:34.792 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:46:34.793 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T00:46:34.793 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:46:34.793 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T00:46:34.793 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:46:34.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T00:46:34.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:46:34.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T00:46:34.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:46:34.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T00:46:34.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:46:34.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:46:34.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:46:34.795 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:46:34 smithi047 ceph-mon[77147]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T00:46:34.796 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:46:34 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:46:34.796 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:46:34 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 4725 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:46:35.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:46:35 smithi115 conmon[90393]: 2022-05-14T00:46:35.466+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:46:35.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:46:35 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:46:35.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:46:35 smithi115 ceph-mon[78593]: pgmap v2717: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:46:35.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:46:35 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:46:35.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:46:35 smithi047 ceph-mon[77147]: pgmap v2717: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:46:36.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:46:36 smithi115 conmon[90393]: 2022-05-14T00:46:36.493+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:46:36.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:46:36 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:46:36.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:46:36 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:46:36.792 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:46:36 smithi047 conmon[78521]: 2022-05-14T00:46:36.463+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:46:36.792 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:46:36 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T00:46:36.792 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:46:36 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:46:36.792 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:46:36 smithi047 conmon[78521]: yield 2022-05-14T00:46:36.793 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:46:36 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:46:36.793 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:46:36 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:46:36.793 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:46:36 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:46:36.793 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:46:36 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T00:46:36.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:46:36 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:46:36.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:46:36 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T00:46:36.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:46:36 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:46:36.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:46:36 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T00:46:36.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:46:36 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:46:36.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:46:36 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T00:46:36.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:46:36 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:46:36.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:46:36 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T00:46:36.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:46:36 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:46:36.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:46:36 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:46:36.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:46:36 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:46:36.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:46:36 smithi047 conmon[78521]: 2022-05-14T00:46:36.464+0000 7f5344781700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T00:46:36.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:46:36 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T00:46:36.797 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:46:36 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:46:36.797 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:46:36 smithi047 conmon[78521]: yield 2022-05-14T00:46:36.797 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:46:36 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:46:36.797 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:46:36 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:46:36.797 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:46:36 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:46:36.798 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:46:36 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T00:46:36.798 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:46:36 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:46:36.798 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:46:36 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T00:46:36.798 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:46:36 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:46:36.798 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:46:36 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T00:46:36.799 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:46:36 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:46:36.799 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:46:36 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T00:46:36.799 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:46:36 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:46:36.800 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:46:36 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T00:46:36.800 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:46:36 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:46:36.800 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:46:36 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:46:36.800 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:46:36 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:46:36.800 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:46:36 smithi047 conmon[78521]: 2022-05-14T00:46:36.801 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:46:36 smithi047 conmon[78521]: During handling of the above exception, another exception occurred: 2022-05-14T00:46:36.801 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:46:36 smithi047 conmon[78521]: 2022-05-14T00:46:36.801 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:46:36 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T00:46:36.801 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:46:36 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T00:46:36.802 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:46:36 smithi047 conmon[78521]: return f(*arg) 2022-05-14T00:46:36.802 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:46:36 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T00:46:36.802 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:46:36 smithi047 conmon[78521]: self._write_client_files(client_files, host) 2022-05-14T00:46:36.802 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:46:36 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T00:46:36.803 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:46:36 smithi047 conmon[78521]: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T00:46:36.803 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:46:36 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T00:46:36.803 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:46:36 smithi047 conmon[78521]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T00:46:36.803 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:46:36 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T00:46:36.803 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:46:36 smithi047 conmon[78521]: return self.event_loop.get_result(coro) 2022-05-14T00:46:36.804 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:46:36 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T00:46:36.804 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:46:36 smithi047 conmon[78521]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T00:46:36.804 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:46:36 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T00:46:36.804 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:46:36 smithi047 conmon[78521]: return self.__get_result() 2022-05-14T00:46:36.804 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:46:36 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T00:46:36.805 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:46:36 smithi047 conmon[78521]: raise self._exception 2022-05-14T00:46:36.805 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:46:36 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T00:46:36.805 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:46:36 smithi047 conmon[78521]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T00:46:36.805 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:46:36 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T00:46:36.806 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:46:36 smithi047 conmon[78521]: conn = await self._remote_connection(host, addr) 2022-05-14T00:46:36.806 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:46:36 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T00:46:36.806 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:46:36 smithi047 conmon[78521]: raise 2022-05-14T00:46:36.806 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:46:36 smithi047 conmon[78521]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T00:46:36.806 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:46:36 smithi047 conmon[78521]: self.gen.throw(type, value, traceback) 2022-05-14T00:46:36.807 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:46:36 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T00:46:36.807 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:46:36 smithi047 conmon[78521]: raise OrchestratorError(msg) 2022-05-14T00:46:36.807 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:46:36 smithi047 conmon[78521]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:46:37.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:46:37 smithi115 conmon[90393]: 2022-05-14T00:46:37.532+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:46:37.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:46:37 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:46:37.609 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T00:46:37.609 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:46:37.609 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T00:46:37.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:46:37.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:46:37.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:46:37.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T00:46:37.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:46:37.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T00:46:37.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:46:37.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T00:46:37.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:46:37.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T00:46:37.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:46:37.613 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T00:46:37.613 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:46:37.614 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:46:37.614 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:46:37.614 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:46:37 smithi115 ceph-mon[78593]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T00:46:37.615 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T00:46:37.615 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:46:37.615 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T00:46:37.615 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:46:37.615 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:46:37.616 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:46:37.616 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T00:46:37.616 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:46:37.616 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T00:46:37.617 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:46:37.617 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T00:46:37.617 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:46:37.617 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T00:46:37.617 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:46:37.618 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T00:46:37.618 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:46:37.618 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:46:37.618 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:46:37.619 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T00:46:37.619 INFO:journalctl@ceph.mon.c.smithi115.stdout: During handling of the above exception, another exception occurred: 2022-05-14T00:46:37.619 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T00:46:37.619 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T00:46:37.620 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T00:46:37.620 INFO:journalctl@ceph.mon.c.smithi115.stdout: return f(*arg) 2022-05-14T00:46:37.620 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T00:46:37.620 INFO:journalctl@ceph.mon.c.smithi115.stdout: self._write_client_files(client_files, host) 2022-05-14T00:46:37.621 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T00:46:37.621 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T00:46:37.621 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T00:46:37.621 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T00:46:37.622 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T00:46:37.622 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.event_loop.get_result(coro) 2022-05-14T00:46:37.622 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T00:46:37.622 INFO:journalctl@ceph.mon.c.smithi115.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T00:46:37.622 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T00:46:37.623 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.__get_result() 2022-05-14T00:46:37.623 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T00:46:37.623 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise self._exception 2022-05-14T00:46:37.623 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T00:46:37.624 INFO:journalctl@ceph.mon.c.smithi115.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T00:46:37.624 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T00:46:37.624 INFO:journalctl@ceph.mon.c.smithi115.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T00:46:37.624 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T00:46:37.624 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise 2022-05-14T00:46:37.625 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T00:46:37.625 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.gen.throw(type, value, traceback) 2022-05-14T00:46:37.625 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T00:46:37.625 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OrchestratorError(msg) 2022-05-14T00:46:37.626 INFO:journalctl@ceph.mon.c.smithi115.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:46:37.626 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:46:37 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:46:37.626 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:46:37 smithi115 ceph-mon[78593]: pgmap v2718: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:46:37.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:46:37 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:46:37.791 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T00:46:37.791 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:46:37.791 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T00:46:37.792 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:46:37.792 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:46:37.792 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:46:37.792 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T00:46:37.793 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:46:37.793 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T00:46:37.793 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:46:37.793 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T00:46:37.793 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:46:37.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T00:46:37.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:46:37.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T00:46:37.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:46:37.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:46:37.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:46:37.795 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:46:37 smithi047 ceph-mon[77147]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T00:46:37.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T00:46:37.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:46:37.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T00:46:37.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:46:37.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:46:37.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:46:37.797 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T00:46:37.797 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:46:37.797 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T00:46:37.797 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:46:37.798 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T00:46:37.798 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:46:37.798 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T00:46:37.799 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:46:37.799 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T00:46:37.799 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:46:37.799 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:46:37.799 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:46:37.800 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T00:46:37.800 INFO:journalctl@ceph.mon.b.smithi047.stdout: During handling of the above exception, another exception occurred: 2022-05-14T00:46:37.800 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T00:46:37.800 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T00:46:37.801 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T00:46:37.801 INFO:journalctl@ceph.mon.b.smithi047.stdout: return f(*arg) 2022-05-14T00:46:37.801 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T00:46:37.801 INFO:journalctl@ceph.mon.b.smithi047.stdout: self._write_client_files(client_files, host) 2022-05-14T00:46:37.801 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T00:46:37.802 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T00:46:37.802 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T00:46:37.802 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T00:46:37.802 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T00:46:37.802 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.event_loop.get_result(coro) 2022-05-14T00:46:37.803 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T00:46:37.803 INFO:journalctl@ceph.mon.b.smithi047.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T00:46:37.803 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T00:46:37.803 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.__get_result() 2022-05-14T00:46:37.804 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T00:46:37.804 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise self._exception 2022-05-14T00:46:37.804 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T00:46:37.804 INFO:journalctl@ceph.mon.b.smithi047.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T00:46:37.804 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T00:46:37.805 INFO:journalctl@ceph.mon.b.smithi047.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T00:46:37.805 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T00:46:37.805 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise 2022-05-14T00:46:37.805 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T00:46:37.806 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.gen.throw(type, value, traceback) 2022-05-14T00:46:37.806 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T00:46:37.806 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OrchestratorError(msg) 2022-05-14T00:46:37.806 INFO:journalctl@ceph.mon.b.smithi047.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:46:37.806 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:46:37 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:46:37.807 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:46:37 smithi047 ceph-mon[77147]: pgmap v2718: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:46:38.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:46:38 smithi115 conmon[90393]: 2022-05-14T00:46:38.546+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:46:38.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:46:38 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:46:38.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:46:38 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:46:38.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:46:38 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:46:38.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:46:38 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:46:38.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:46:38 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:46:38.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:46:38 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:46:39.594 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:46:39 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:46:39.595 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:46:39 smithi115 ceph-mon[78593]: pgmap v2719: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:46:39.595 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:46:39 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 4730 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:46:39.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:46:39 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:46:39.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:46:39 smithi047 ceph-mon[77147]: pgmap v2719: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:46:39.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:46:39 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 4730 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:46:39.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:46:39 smithi115 conmon[90393]: 2022-05-14T00:46:39.593+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:46:40.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:46:40 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:46:40.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:46:40 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:46:41.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:46:40 smithi115 conmon[90393]: 2022-05-14T00:46:40.637+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:46:41.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:46:41 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:46:41.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:46:41 smithi115 ceph-mon[78593]: pgmap v2720: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:46:41.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:46:41 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:46:41.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:46:41 smithi047 ceph-mon[77147]: pgmap v2720: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:46:42.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:46:41 smithi115 conmon[90393]: 2022-05-14T00:46:41.653+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:46:42.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:46:42 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:46:42.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:46:42 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:46:42.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:46:42 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:46:42.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:46:42 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:46:42.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:46:42 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:46:42.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:46:42 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:46:43.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:46:42 smithi115 conmon[90393]: 2022-05-14T00:46:42.607+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:46:43.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:46:43 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:46:43.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:46:43 smithi115 ceph-mon[78593]: pgmap v2721: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:46:43.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:46:43 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:46:43.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:46:43 smithi047 ceph-mon[77147]: pgmap v2721: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:46:44.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:46:43 smithi115 conmon[90393]: 2022-05-14T00:46:43.654+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:46:44.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:46:44 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:46:44.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:46:44 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 4735 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:46:44.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:46:44 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:46:44.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:46:44 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 4735 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:46:45.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:46:44 smithi115 conmon[90393]: 2022-05-14T00:46:44.693+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:46:45.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:46:45 smithi115 ceph-mon[78593]: pgmap v2722: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:46:45.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:46:45 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:46:45.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:46:45 smithi047 ceph-mon[77147]: pgmap v2722: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:46:45.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:46:45 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:46:46.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:46:45 smithi115 conmon[90393]: 2022-05-14T00:46:45.658+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:46:46.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:46:46 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:46:46.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:46:46 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:46:47.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:46:46 smithi115 conmon[90393]: 2022-05-14T00:46:46.666+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:46:47.699 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:46:47 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:46:47.699 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:46:47 smithi115 ceph-mon[78593]: pgmap v2723: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:46:47.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:46:47 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:46:47.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:46:47 smithi047 ceph-mon[77147]: pgmap v2723: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:46:48.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:46:47 smithi115 conmon[90393]: 2022-05-14T00:46:47.698+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:46:48.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:46:48 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:46:48.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:46:48 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:46:49.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:46:48 smithi115 conmon[90393]: 2022-05-14T00:46:48.695+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:46:49.656 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:46:49 smithi115 ceph-mon[78593]: pgmap v2724: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:46:49.656 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:46:49 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:46:49.657 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:46:49 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 4740 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:46:49.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:46:49 smithi047 ceph-mon[77147]: pgmap v2724: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:46:49.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:46:49 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:46:49.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:46:49 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 4740 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:46:50.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:46:49 smithi115 conmon[90393]: 2022-05-14T00:46:49.655+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:46:50.703 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:46:50 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:46:50.704 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:46:50 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:46:50.704 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:46:50 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:46:50.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:46:50 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:46:50.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:46:50 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:46:50.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:46:50 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:46:51.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:46:50 smithi115 conmon[90393]: 2022-05-14T00:46:50.702+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:46:51.751 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:46:51 smithi115 ceph-mon[78593]: pgmap v2725: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:46:51.751 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:46:51 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:46:51.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:46:51 smithi047 ceph-mon[77147]: pgmap v2725: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:46:51.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:46:51 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:46:52.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:46:51 smithi115 conmon[90393]: 2022-05-14T00:46:51.749+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:46:52.768 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:46:52 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:46:52.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:46:52 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:46:53.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:46:52 smithi115 conmon[90393]: 2022-05-14T00:46:52.767+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:46:53.731 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:46:53 smithi115 ceph-mon[78593]: pgmap v2726: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:46:53.732 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:46:53 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:46:53.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:46:53 smithi047 ceph-mon[77147]: pgmap v2726: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:46:53.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:46:53 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:46:54.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:46:53 smithi115 conmon[90393]: 2022-05-14T00:46:53.730+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:46:54.698 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:46:54 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:46:54.698 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:46:54 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:46:54.698 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:46:54 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:46:54.699 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:46:54 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 4745 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:46:54.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:46:54 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:46:54.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:46:54 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:46:54.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:46:54 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:46:54.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:46:54 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 4745 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:46:55.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:46:54 smithi115 conmon[90393]: 2022-05-14T00:46:54.697+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:46:55.709 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:46:55 smithi115 ceph-mon[78593]: pgmap v2727: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:46:55.710 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:46:55 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:46:55.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:46:55 smithi047 ceph-mon[77147]: pgmap v2727: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:46:55.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:46:55 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:46:56.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:46:55 smithi115 conmon[90393]: 2022-05-14T00:46:55.708+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:46:56.694 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:46:56 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:46:56.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:46:56 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:46:57.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:46:56 smithi115 conmon[90393]: 2022-05-14T00:46:56.693+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:46:57.708 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:46:57 smithi115 ceph-mon[78593]: pgmap v2728: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:46:57.708 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:46:57 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:46:57.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:46:57 smithi047 ceph-mon[77147]: pgmap v2728: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:46:57.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:46:57 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:46:58.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:46:57 smithi115 conmon[90393]: 2022-05-14T00:46:57.707+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:46:58.669 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:46:58 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:46:58.669 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:46:58 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:46:58.670 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:46:58 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:46:58.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:46:58 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:46:58.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:46:58 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:46:58.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:46:58 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:46:59.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:46:58 smithi115 conmon[90393]: 2022-05-14T00:46:58.668+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:46:59.651 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:46:59 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:46:59.651 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:46:59 smithi115 ceph-mon[78593]: pgmap v2729: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:46:59.651 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:46:59 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 4750 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:46:59.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:46:59 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:46:59.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:46:59 smithi047 ceph-mon[77147]: pgmap v2729: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:46:59.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:46:59 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 4750 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:47:00.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:46:59 smithi115 conmon[90393]: 2022-05-14T00:46:59.650+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:47:00.647 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:47:00 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:47:00.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:47:00 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:47:01.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:47:00 smithi115 conmon[90393]: 2022-05-14T00:47:00.646+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:47:01.691 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:47:01 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:47:01.692 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:47:01 smithi115 ceph-mon[78593]: pgmap v2730: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:47:01.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:47:01 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:47:01.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:47:01 smithi047 ceph-mon[77147]: pgmap v2730: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:47:02.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:47:01 smithi115 conmon[90393]: 2022-05-14T00:47:01.690+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:47:02.656 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:47:02 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:47:02.656 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:47:02 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:47:02.657 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:47:02 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:47:02.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:47:02 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:47:02.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:47:02 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:47:02.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:47:02 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:47:03.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:47:02 smithi115 conmon[90393]: 2022-05-14T00:47:02.655+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:47:03.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:47:03 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:47:03.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:47:03 smithi047 ceph-mon[77147]: pgmap v2731: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:47:03.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:47:03 smithi115 conmon[90393]: 2022-05-14T00:47:03.639+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:47:03.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:47:03 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:47:03.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:47:03 smithi115 ceph-mon[78593]: pgmap v2731: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:47:04.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:47:04 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:47:04.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:47:04 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 4755 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:47:04.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:47:04 smithi115 conmon[90393]: 2022-05-14T00:47:04.624+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:47:04.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:47:04 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:47:04.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:47:04 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 4755 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:47:05.665 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:47:05 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:47:05.665 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:47:05 smithi115 ceph-mon[78593]: pgmap v2732: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:47:05.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:47:05 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:47:05.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:47:05 smithi047 ceph-mon[77147]: pgmap v2732: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:47:06.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:47:05 smithi115 conmon[90393]: 2022-05-14T00:47:05.664+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:47:06.714 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:47:06 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:47:06.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:47:06 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:47:07.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:47:06 smithi115 conmon[90393]: 2022-05-14T00:47:06.713+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:47:07.717 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:47:07 smithi115 ceph-mon[78593]: pgmap v2733: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:47:07.717 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:47:07 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:47:07.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:47:07 smithi047 ceph-mon[77147]: pgmap v2733: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:47:07.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:47:07 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:47:08.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:47:07 smithi115 conmon[90393]: 2022-05-14T00:47:07.715+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:47:08.708 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:47:08 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:47:08.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:47:08 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:47:09.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:47:08 smithi115 conmon[90393]: 2022-05-14T00:47:08.707+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:47:09.741 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:47:09 smithi115 ceph-mon[78593]: pgmap v2734: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:47:09.742 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:47:09 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:47:09.742 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:47:09 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 4760 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:47:09.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:47:09 smithi047 ceph-mon[77147]: pgmap v2734: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:47:09.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:47:09 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:47:09.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:47:09 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 4760 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:47:10.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:47:09 smithi115 conmon[90393]: 2022-05-14T00:47:09.740+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:47:10.708 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:47:10 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:47:10.708 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:47:10 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:47:10.708 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:47:10 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:47:10.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:47:10 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:47:10.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:47:10 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:47:10.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:47:10 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:47:11.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:47:10 smithi115 conmon[90393]: 2022-05-14T00:47:10.707+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:47:11.745 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:47:11 smithi115 ceph-mon[78593]: pgmap v2735: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:47:11.746 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:47:11 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:47:11.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:47:11 smithi047 ceph-mon[77147]: pgmap v2735: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:47:11.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:47:11 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:47:12.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:47:11 smithi115 conmon[90393]: 2022-05-14T00:47:11.745+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:47:12.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:47:12 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:47:12.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:47:12 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:47:13.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:47:12 smithi115 conmon[90393]: 2022-05-14T00:47:12.791+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:47:13.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:47:13 smithi047 ceph-mon[77147]: pgmap v2736: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:47:13.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:47:13 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:47:13.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:47:13 smithi115 ceph-mon[78593]: pgmap v2736: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:47:13.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:47:13 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:47:14.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:47:13 smithi115 conmon[90393]: 2022-05-14T00:47:13.794+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:47:14.756 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:47:14 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:47:14.756 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:47:14 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:47:14.757 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:47:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T00:47:14.757 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:47:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T00:47:14.757 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:47:14 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:47:14.757 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:47:14 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 4765 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:47:14.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:47:14 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:47:14.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:47:14 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:47:14.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:47:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T00:47:14.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:47:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T00:47:14.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:47:14 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:47:14.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:47:14 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 4765 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:47:15.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:47:14 smithi115 conmon[90393]: 2022-05-14T00:47:14.755+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:47:15.767 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:47:15 smithi115 ceph-mon[78593]: pgmap v2737: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:47:15.767 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:47:15 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:47:15.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:47:15 smithi047 ceph-mon[77147]: pgmap v2737: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:47:15.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:47:15 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:47:16.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:47:15 smithi115 conmon[90393]: 2022-05-14T00:47:15.765+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:47:16.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:47:16 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:47:16.813 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:47:16 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:47:17.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:47:16 smithi115 conmon[90393]: 2022-05-14T00:47:16.812+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:47:17.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:47:17 smithi047 ceph-mon[77147]: pgmap v2738: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:47:17.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:47:17 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:47:17.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:47:17 smithi115 ceph-mon[78593]: pgmap v2738: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:47:17.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:47:17 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:47:18.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:47:17 smithi115 conmon[90393]: 2022-05-14T00:47:17.860+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:47:18.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:47:18 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:47:18.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:47:18 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:47:18.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:47:18 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:47:18.839 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:47:18 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:47:18.840 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:47:18 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:47:18.840 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:47:18 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:47:19.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:47:18 smithi115 conmon[90393]: 2022-05-14T00:47:18.839+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:47:19.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:47:19 smithi047 ceph-mon[77147]: pgmap v2739: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:47:19.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:47:19 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:47:19.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:47:19 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 4770 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:47:19.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:47:19 smithi115 ceph-mon[78593]: pgmap v2739: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:47:19.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:47:19 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:47:19.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:47:19 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 4770 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:47:20.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:47:19 smithi115 conmon[90393]: 2022-05-14T00:47:19.888+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:47:20.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:47:20 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:47:20.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:47:20 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:47:21.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:47:20 smithi115 conmon[90393]: 2022-05-14T00:47:20.895+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:47:21.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:47:21 smithi047 ceph-mon[77147]: pgmap v2740: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:47:21.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:47:21 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:47:21.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:47:21 smithi115 ceph-mon[78593]: pgmap v2740: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:47:21.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:47:21 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:47:22.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:47:21 smithi115 conmon[90393]: 2022-05-14T00:47:21.886+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:47:22.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:47:22 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:47:22.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:47:22 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:47:22.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:47:22 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:47:22.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:47:22 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:47:22.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:47:22 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:47:22.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:47:22 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:47:23.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:47:22 smithi115 conmon[90393]: 2022-05-14T00:47:22.915+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:47:23.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:47:23 smithi047 ceph-mon[77147]: pgmap v2741: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:47:23.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:47:23 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:47:23.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:47:23 smithi115 ceph-mon[78593]: pgmap v2741: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:47:23.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:47:23 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:47:24.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:47:23 smithi115 conmon[90393]: 2022-05-14T00:47:23.945+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:47:24.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:47:24 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 4775 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:47:24.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:47:24 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 4775 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:47:25.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:47:24 smithi115 conmon[90393]: 2022-05-14T00:47:24.995+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:47:25.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:47:25 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:47:25.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:47:25 smithi047 ceph-mon[77147]: pgmap v2742: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:47:25.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:47:25 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:47:25.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:47:25 smithi115 ceph-mon[78593]: pgmap v2742: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:47:26.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:47:25 smithi115 conmon[90393]: 2022-05-14T00:47:25.959+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:47:26.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:47:26 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:47:26.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:47:26 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:47:27.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:47:26 smithi115 conmon[90393]: 2022-05-14T00:47:26.911+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:47:27.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:47:27 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:47:27.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:47:27 smithi047 ceph-mon[77147]: pgmap v2743: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:47:27.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:47:27 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:47:27.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:47:27 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:47:27.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:47:27 smithi115 ceph-mon[78593]: pgmap v2743: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:47:27.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:47:27 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:47:28.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:47:27 smithi115 conmon[90393]: 2022-05-14T00:47:27.919+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:47:29.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:47:28 smithi115 conmon[90393]: 2022-05-14T00:47:28.874+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:47:29.865 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:47:29 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:47:29.865 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:47:29 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:47:29.865 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:47:29 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:47:29.866 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:47:29 smithi115 ceph-mon[78593]: pgmap v2744: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:47:29.866 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:47:29 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:47:29.866 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:47:29 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 4780 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:47:30.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:47:29 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:47:30.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:47:29 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:47:30.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:47:29 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:47:30.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:47:29 smithi047 ceph-mon[77147]: pgmap v2744: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:47:30.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:47:29 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:47:30.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:47:29 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 4780 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:47:30.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:47:29 smithi115 conmon[90393]: 2022-05-14T00:47:29.863+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:47:30.891 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:47:30 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:47:31.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:47:30 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:47:31.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:47:30 smithi115 conmon[90393]: 2022-05-14T00:47:30.890+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:47:31.928 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:47:31 smithi115 ceph-mon[78593]: pgmap v2745: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:47:31.928 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:47:31 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:47:32.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:47:31 smithi047 ceph-mon[77147]: pgmap v2745: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:47:32.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:47:31 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:47:32.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:47:31 smithi115 conmon[90393]: 2022-05-14T00:47:31.927+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:47:33.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:47:32 smithi115 conmon[90393]: 2022-05-14T00:47:32.935+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:47:33.904 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:47:33 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:47:33.904 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:47:33 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:47:33.904 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:47:33 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:47:33.905 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:47:33 smithi115 ceph-mon[78593]: pgmap v2746: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:47:34.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:47:33 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:47:34.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:47:33 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:47:34.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:47:33 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:47:34.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:47:33 smithi047 ceph-mon[77147]: pgmap v2746: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:47:34.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:47:33 smithi115 conmon[90393]: 2022-05-14T00:47:33.902+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:47:34.896 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:47:34 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:47:34.897 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:47:34 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:47:34.897 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:47:34 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 4785 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:47:35.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:47:34 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:47:35.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:47:34 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:47:35.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:47:34 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 4785 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:47:35.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:47:34 smithi115 conmon[90393]: 2022-05-14T00:47:34.895+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:47:35.891 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:47:35 smithi115 ceph-mon[78593]: pgmap v2747: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:47:35.891 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:47:35 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:47:36.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:47:35 smithi047 ceph-mon[77147]: pgmap v2747: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:47:36.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:47:35 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:47:36.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:47:35 smithi115 conmon[90393]: 2022-05-14T00:47:35.890+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:47:36.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:47:36 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:47:36.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:47:36 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T00:47:36.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:47:36 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T00:47:36.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:47:36 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T00:47:36.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:47:36 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:47:36.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:47:36 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T00:47:36.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:47:36 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T00:47:36.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:47:36 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T00:47:37.291 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:47:36 smithi047 conmon[78521]: 2022-05-14T00:47:36.943+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:47:37.291 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:47:36 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T00:47:37.291 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:47:36 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:47:37.292 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:47:36 smithi047 conmon[78521]: yield 2022-05-14T00:47:37.292 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:47:36 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:47:37.293 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:47:36 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:47:37.293 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:47:36 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:47:37.293 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:47:36 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T00:47:37.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:47:36 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:47:37.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:47:36 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T00:47:37.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:47:36 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:47:37.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:47:36 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T00:47:37.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:47:36 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:47:37.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:47:36 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T00:47:37.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:47:36 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:47:37.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:47:36 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T00:47:37.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:47:36 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:47:37.297 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:47:36 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:47:37.297 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:47:36 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:47:37.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:47:36 smithi115 conmon[90393]: 2022-05-14T00:47:36.908+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:47:38.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:47:37 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:47:38.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:47:37 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:47:38.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:47:37 smithi047 ceph-mon[77147]: pgmap v2748: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:47:38.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:47:37 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:47:38.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:47:37 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:47:38.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T00:47:38.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:47:38.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T00:47:38.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:47:38.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:47:38.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:47:38.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T00:47:38.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:47:38.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T00:47:38.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:47:38.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T00:47:38.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:47:38.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T00:47:38.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:47:38.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T00:47:38.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:47:38.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:47:38.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:47:38.047 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:47:37 smithi047 ceph-mon[77147]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T00:47:38.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:47:37 smithi115 conmon[90393]: 2022-05-14T00:47:37.874+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:47:38.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:47:37 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:47:38.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:47:37 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:47:38.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:47:37 smithi115 ceph-mon[78593]: pgmap v2748: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:47:38.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:47:37 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:47:38.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:47:37 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:47:38.109 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T00:47:38.109 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:47:38.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T00:47:38.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:47:38.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:47:38.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:47:38.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T00:47:38.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:47:38.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T00:47:38.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:47:38.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T00:47:38.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:47:38.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T00:47:38.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:47:38.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T00:47:38.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:47:38.113 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:47:38.113 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:47:38.113 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:47:37 smithi115 ceph-mon[78593]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T00:47:39.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:47:38 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:47:39.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:47:38 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:47:39.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:47:38 smithi115 conmon[90393]: 2022-05-14T00:47:38.874+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:47:39.902 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:47:39 smithi115 ceph-mon[78593]: pgmap v2749: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:47:39.903 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:47:39 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:47:39.903 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:47:39 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 4790 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:47:40.017 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:47:39 smithi047 ceph-mon[77147]: pgmap v2749: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:47:40.017 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:47:39 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:47:40.017 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:47:39 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 4790 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:47:40.291 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:47:40 smithi047 conmon[78521]: 2022-05-14T00:47:40.016+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:47:40.291 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:47:40 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T00:47:40.291 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:47:40 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:47:40.292 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:47:40 smithi047 conmon[78521]: yield 2022-05-14T00:47:40.292 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:47:40 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:47:40.292 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:47:40 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:47:40.292 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:47:40 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:47:40.293 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:47:40 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T00:47:40.293 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:47:40 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:47:40.293 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:47:40 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T00:47:40.293 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:47:40 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:47:40.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:47:40 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T00:47:40.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:47:40 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:47:40.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:47:40 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T00:47:40.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:47:40 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:47:40.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:47:40 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T00:47:40.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:47:40 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:47:40.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:47:40 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:47:40.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:47:40 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:47:40.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:47:40 smithi047 conmon[78521]: 2022-05-14T00:47:40.016+0000 7f5345f84700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T00:47:40.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:47:40 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T00:47:40.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:47:40 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:47:40.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:47:40 smithi047 conmon[78521]: yield 2022-05-14T00:47:40.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:47:40 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:47:40.297 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:47:40 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:47:40.297 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:47:40 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:47:40.297 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:47:40 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T00:47:40.297 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:47:40 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:47:40.298 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:47:40 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T00:47:40.298 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:47:40 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:47:40.298 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:47:40 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T00:47:40.298 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:47:40 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:47:40.298 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:47:40 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T00:47:40.299 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:47:40 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:47:40.299 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:47:40 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T00:47:40.299 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:47:40 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:47:40.299 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:47:40 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:47:40.300 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:47:40 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:47:40.300 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:47:40 smithi047 conmon[78521]: 2022-05-14T00:47:40.300 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:47:40 smithi047 conmon[78521]: During handling of the above exception, another exception occurred: 2022-05-14T00:47:40.300 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:47:40 smithi047 conmon[78521]: 2022-05-14T00:47:40.301 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:47:40 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T00:47:40.301 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:47:40 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T00:47:40.301 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:47:40 smithi047 conmon[78521]: return f(*arg) 2022-05-14T00:47:40.301 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:47:40 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T00:47:40.302 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:47:40 smithi047 conmon[78521]: self._write_client_files(client_files, host) 2022-05-14T00:47:40.302 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:47:40 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T00:47:40.302 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:47:40 smithi047 conmon[78521]: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T00:47:40.302 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:47:40 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T00:47:40.303 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:47:40 smithi047 conmon[78521]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T00:47:40.303 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:47:40 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T00:47:40.303 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:47:40 smithi047 conmon[78521]: return self.event_loop.get_result(coro) 2022-05-14T00:47:40.303 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:47:40 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T00:47:40.303 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:47:40 smithi047 conmon[78521]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T00:47:40.304 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:47:40 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T00:47:40.304 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:47:40 smithi047 conmon[78521]: return self.__get_result() 2022-05-14T00:47:40.304 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:47:40 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T00:47:40.304 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:47:40 smithi047 conmon[78521]: raise self._exception 2022-05-14T00:47:40.305 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:47:40 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T00:47:40.305 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:47:40 smithi047 conmon[78521]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T00:47:40.305 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:47:40 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T00:47:40.305 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:47:40 smithi047 conmon[78521]: conn = await self._remote_connection(host, addr) 2022-05-14T00:47:40.305 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:47:40 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T00:47:40.306 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:47:40 smithi047 conmon[78521]: raise 2022-05-14T00:47:40.306 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:47:40 smithi047 conmon[78521]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T00:47:40.306 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:47:40 smithi047 conmon[78521]: self.gen.throw(type, value, traceback) 2022-05-14T00:47:40.306 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:47:40 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T00:47:40.307 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:47:40 smithi047 conmon[78521]: raise OrchestratorError(msg) 2022-05-14T00:47:40.307 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:47:40 smithi047 conmon[78521]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:47:40.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:47:39 smithi115 conmon[90393]: 2022-05-14T00:47:39.901+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:47:41.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:47:40 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:47:41.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:47:40 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:47:41.042 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T00:47:41.042 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:47:41.042 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T00:47:41.042 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:47:41.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:47:41.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:47:41.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T00:47:41.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:47:41.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T00:47:41.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:47:41.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T00:47:41.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:47:41.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T00:47:41.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:47:41.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T00:47:41.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:47:41.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:47:41.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:47:41.048 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:47:40 smithi047 ceph-mon[77147]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T00:47:41.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T00:47:41.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:47:41.049 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T00:47:41.049 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:47:41.049 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:47:41.049 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:47:41.050 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T00:47:41.050 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:47:41.050 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T00:47:41.050 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:47:41.050 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T00:47:41.051 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:47:41.051 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T00:47:41.051 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:47:41.051 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T00:47:41.052 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:47:41.052 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:47:41.052 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:47:41.052 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T00:47:41.052 INFO:journalctl@ceph.mon.b.smithi047.stdout: During handling of the above exception, another exception occurred: 2022-05-14T00:47:41.053 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T00:47:41.053 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T00:47:41.053 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T00:47:41.053 INFO:journalctl@ceph.mon.b.smithi047.stdout: return f(*arg) 2022-05-14T00:47:41.054 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T00:47:41.054 INFO:journalctl@ceph.mon.b.smithi047.stdout: self._write_client_files(client_files, host) 2022-05-14T00:47:41.054 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T00:47:41.054 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T00:47:41.054 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T00:47:41.055 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T00:47:41.055 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T00:47:41.055 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.event_loop.get_result(coro) 2022-05-14T00:47:41.055 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T00:47:41.055 INFO:journalctl@ceph.mon.b.smithi047.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T00:47:41.056 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T00:47:41.056 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.__get_result() 2022-05-14T00:47:41.056 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T00:47:41.056 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise self._exception 2022-05-14T00:47:41.056 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T00:47:41.057 INFO:journalctl@ceph.mon.b.smithi047.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T00:47:41.057 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T00:47:41.057 INFO:journalctl@ceph.mon.b.smithi047.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T00:47:41.057 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T00:47:41.058 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise 2022-05-14T00:47:41.058 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T00:47:41.058 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.gen.throw(type, value, traceback) 2022-05-14T00:47:41.058 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T00:47:41.058 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OrchestratorError(msg) 2022-05-14T00:47:41.059 INFO:journalctl@ceph.mon.b.smithi047.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:47:41.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:47:40 smithi115 conmon[90393]: 2022-05-14T00:47:40.863+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:47:41.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:47:40 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:47:41.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:47:40 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:47:41.109 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T00:47:41.109 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:47:41.109 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T00:47:41.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:47:41.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:47:41.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:47:41.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T00:47:41.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:47:41.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T00:47:41.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:47:41.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T00:47:41.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:47:41.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T00:47:41.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:47:41.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T00:47:41.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:47:41.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:47:41.113 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:47:41.113 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:47:40 smithi115 ceph-mon[78593]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T00:47:41.113 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T00:47:41.113 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:47:41.114 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T00:47:41.114 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:47:41.114 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:47:41.114 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:47:41.114 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T00:47:41.115 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:47:41.115 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T00:47:41.115 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:47:41.115 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T00:47:41.115 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:47:41.116 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T00:47:41.116 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:47:41.116 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T00:47:41.116 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:47:41.116 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:47:41.117 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:47:41.117 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T00:47:41.117 INFO:journalctl@ceph.mon.c.smithi115.stdout: During handling of the above exception, another exception occurred: 2022-05-14T00:47:41.118 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T00:47:41.119 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T00:47:41.120 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T00:47:41.120 INFO:journalctl@ceph.mon.c.smithi115.stdout: return f(*arg) 2022-05-14T00:47:41.120 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T00:47:41.120 INFO:journalctl@ceph.mon.c.smithi115.stdout: self._write_client_files(client_files, host) 2022-05-14T00:47:41.120 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T00:47:41.121 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T00:47:41.121 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T00:47:41.121 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T00:47:41.121 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T00:47:41.121 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.event_loop.get_result(coro) 2022-05-14T00:47:41.122 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T00:47:41.122 INFO:journalctl@ceph.mon.c.smithi115.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T00:47:41.122 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T00:47:41.122 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.__get_result() 2022-05-14T00:47:41.123 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T00:47:41.123 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise self._exception 2022-05-14T00:47:41.123 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T00:47:41.123 INFO:journalctl@ceph.mon.c.smithi115.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T00:47:41.123 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T00:47:41.124 INFO:journalctl@ceph.mon.c.smithi115.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T00:47:41.124 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T00:47:41.124 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise 2022-05-14T00:47:41.124 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T00:47:41.124 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.gen.throw(type, value, traceback) 2022-05-14T00:47:41.125 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T00:47:41.125 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OrchestratorError(msg) 2022-05-14T00:47:41.125 INFO:journalctl@ceph.mon.c.smithi115.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:47:42.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:47:41 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:47:42.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:47:41 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:47:42.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:47:41 smithi047 ceph-mon[77147]: pgmap v2750: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:47:42.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:47:41 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:47:42.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:47:41 smithi115 conmon[90393]: 2022-05-14T00:47:41.882+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:47:42.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:47:41 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:47:42.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:47:41 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:47:42.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:47:41 smithi115 ceph-mon[78593]: pgmap v2750: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:47:42.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:47:41 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:47:43.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:47:42 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:47:43.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:47:42 smithi115 conmon[90393]: 2022-05-14T00:47:42.888+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:47:43.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:47:42 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:47:44.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:47:43 smithi047 ceph-mon[77147]: pgmap v2751: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:47:44.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:47:43 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:47:44.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:47:43 smithi115 conmon[90393]: 2022-05-14T00:47:43.853+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:47:44.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:47:43 smithi115 ceph-mon[78593]: pgmap v2751: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:47:44.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:47:43 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:47:45.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:47:44 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:47:45.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:47:44 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 4795 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:47:45.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:47:44 smithi115 conmon[90393]: 2022-05-14T00:47:44.871+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:47:45.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:47:44 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:47:45.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:47:44 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 4795 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:47:46.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:47:45 smithi047 ceph-mon[77147]: pgmap v2752: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:47:46.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:47:45 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:47:46.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:47:45 smithi115 conmon[90393]: 2022-05-14T00:47:45.864+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:47:46.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:47:45 smithi115 ceph-mon[78593]: pgmap v2752: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:47:46.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:47:45 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:47:47.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:47:46 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:47:47.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:47:46 smithi115 conmon[90393]: 2022-05-14T00:47:46.886+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:47:47.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:47:46 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:47:47.931 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:47:47 smithi115 ceph-mon[78593]: pgmap v2753: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:47:47.932 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:47:47 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:47:48.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:47:47 smithi047 ceph-mon[77147]: pgmap v2753: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:47:48.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:47:47 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:47:48.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:47:47 smithi115 conmon[90393]: 2022-05-14T00:47:47.931+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:47:49.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:47:48 smithi115 conmon[90393]: 2022-05-14T00:47:48.945+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:47:50.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:47:49 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:47:50.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:47:49 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:47:50.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:47:49 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:47:50.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:47:49 smithi047 ceph-mon[77147]: pgmap v2754: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:47:50.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:47:49 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 4800 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:47:50.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:47:49 smithi115 conmon[90393]: 2022-05-14T00:47:49.930+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:47:50.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:47:49 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:47:50.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:47:49 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:47:50.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:47:49 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:47:50.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:47:49 smithi115 ceph-mon[78593]: pgmap v2754: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:47:50.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:47:49 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 4800 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:47:50.973 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:47:50 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:47:51.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:47:50 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:47:51.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:47:50 smithi115 conmon[90393]: 2022-05-14T00:47:50.971+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:47:51.946 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:47:51 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:47:51.946 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:47:51 smithi115 ceph-mon[78593]: pgmap v2755: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:47:52.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:47:51 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:47:52.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:47:51 smithi047 ceph-mon[77147]: pgmap v2755: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:47:52.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:47:51 smithi115 conmon[90393]: 2022-05-14T00:47:51.945+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:47:53.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:47:52 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:47:53.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:47:52 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:47:53.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:47:52 smithi115 conmon[90393]: 2022-05-14T00:47:52.901+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:47:54.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:47:53 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:47:54.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:47:53 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:47:54.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:47:53 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:47:54.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:47:53 smithi047 ceph-mon[77147]: pgmap v2756: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:47:54.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:47:53 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:47:54.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:47:53 smithi115 conmon[90393]: 2022-05-14T00:47:53.929+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:47:54.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:47:53 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:47:54.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:47:53 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:47:54.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:47:53 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:47:54.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:47:53 smithi115 ceph-mon[78593]: pgmap v2756: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:47:54.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:47:53 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:47:55.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:47:54 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 4805 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:47:55.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:47:54 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 4805 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:47:55.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:47:54 smithi115 conmon[90393]: 2022-05-14T00:47:54.885+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:47:56.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:47:55 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:47:56.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:47:55 smithi047 ceph-mon[77147]: pgmap v2757: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:47:56.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:47:55 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:47:56.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:47:55 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:47:56.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:47:55 smithi115 ceph-mon[78593]: pgmap v2757: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:47:56.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:47:55 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:47:56.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:47:55 smithi115 conmon[90393]: 2022-05-14T00:47:55.904+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:47:57.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:47:56 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:47:57.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:47:56 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:47:57.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:47:56 smithi115 conmon[90393]: 2022-05-14T00:47:56.940+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:47:58.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:47:57 smithi047 ceph-mon[77147]: pgmap v2758: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:47:58.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:47:57 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:47:58.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:47:57 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:47:58.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:47:57 smithi115 conmon[90393]: 2022-05-14T00:47:57.957+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:47:58.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:47:57 smithi115 ceph-mon[78593]: pgmap v2758: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:47:58.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:47:57 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:47:58.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:47:57 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:47:59.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:47:58 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:47:59.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:47:58 smithi115 conmon[90393]: 2022-05-14T00:47:58.959+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:47:59.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:47:58 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:47:59.976 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:47:59 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:47:59.977 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:47:59 smithi115 ceph-mon[78593]: pgmap v2759: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:47:59.977 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:47:59 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 4810 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:48:00.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:47:59 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:48:00.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:47:59 smithi047 ceph-mon[77147]: pgmap v2759: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:48:00.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:47:59 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 4810 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:48:00.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:47:59 smithi115 conmon[90393]: 2022-05-14T00:47:59.976+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:48:01.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:48:00 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:48:01.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:48:00 smithi115 conmon[90393]: 2022-05-14T00:48:00.937+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:48:01.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:48:00 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:48:02.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:48:01 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:48:02.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:48:01 smithi047 ceph-mon[77147]: pgmap v2760: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:48:02.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:48:01 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:48:02.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:48:01 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:48:02.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:48:01 smithi115 conmon[90393]: 2022-05-14T00:48:01.968+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:48:02.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:48:01 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:48:02.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:48:01 smithi115 ceph-mon[78593]: pgmap v2760: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:48:02.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:48:01 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:48:02.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:48:01 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:48:03.016 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:48:02 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:48:03.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:48:02 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:48:03.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:48:03 smithi115 conmon[90393]: 2022-05-14T00:48:03.015+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:48:04.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:48:03 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:48:04.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:48:03 smithi047 ceph-mon[77147]: pgmap v2761: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:48:04.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:48:03 smithi115 conmon[90393]: 2022-05-14T00:48:03.976+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:48:04.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:48:03 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:48:04.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:48:03 smithi115 ceph-mon[78593]: pgmap v2761: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:48:05.025 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:48:04 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:48:05.025 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:48:04 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 4815 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:48:05.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:48:04 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:48:05.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:48:04 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 4815 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:48:05.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:48:05 smithi115 conmon[90393]: 2022-05-14T00:48:05.024+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:48:06.034 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:48:05 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:48:06.035 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:48:05 smithi115 ceph-mon[78593]: pgmap v2762: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:48:06.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:48:05 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:48:06.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:48:05 smithi047 ceph-mon[77147]: pgmap v2762: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:48:06.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:48:06 smithi115 conmon[90393]: 2022-05-14T00:48:06.033+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:48:07.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:48:06 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:48:07.079 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:48:06 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:48:07.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:48:07 smithi115 conmon[90393]: 2022-05-14T00:48:07.079+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:48:08.034 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:48:07 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:48:08.034 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:48:07 smithi115 ceph-mon[78593]: pgmap v2763: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:48:08.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:48:07 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:48:08.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:48:07 smithi047 ceph-mon[77147]: pgmap v2763: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:48:08.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:48:08 smithi115 conmon[90393]: 2022-05-14T00:48:08.033+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:48:09.009 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:48:08 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:48:09.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:48:08 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:48:09.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:48:09 smithi115 conmon[90393]: 2022-05-14T00:48:09.008+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:48:10.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:48:09 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:48:10.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:48:09 smithi047 ceph-mon[77147]: pgmap v2764: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:48:10.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:48:09 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:48:10.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:48:09 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:48:10.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:48:09 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 4820 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:48:10.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:48:09 smithi115 conmon[90393]: 2022-05-14T00:48:09.962+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:48:10.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:48:09 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:48:10.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:48:09 smithi115 ceph-mon[78593]: pgmap v2764: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:48:10.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:48:09 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:48:10.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:48:09 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:48:10.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:48:09 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 4820 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:48:11.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:48:10 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:48:11.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:48:10 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:48:11.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:48:10 smithi115 conmon[90393]: 2022-05-14T00:48:10.961+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:48:12.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:48:11 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:48:12.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:48:11 smithi047 ceph-mon[77147]: pgmap v2765: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:48:12.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:48:11 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:48:12.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:48:11 smithi115 ceph-mon[78593]: pgmap v2765: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:48:12.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:48:11 smithi115 conmon[90393]: 2022-05-14T00:48:11.927+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:48:13.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:48:12 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:48:13.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:48:12 smithi115 conmon[90393]: 2022-05-14T00:48:12.899+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:48:13.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:48:12 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:48:14.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:48:13 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:48:14.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:48:13 smithi047 ceph-mon[77147]: pgmap v2766: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:48:14.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:48:13 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:48:14.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:48:13 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:48:14.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:48:13 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:48:14.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:48:13 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T00:48:14.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:48:13 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T00:48:14.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:48:13 smithi115 conmon[90393]: 2022-05-14T00:48:13.888+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:48:14.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:48:13 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:48:14.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:48:13 smithi115 ceph-mon[78593]: pgmap v2766: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:48:14.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:48:13 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:48:14.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:48:13 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:48:14.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:48:13 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:48:14.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:48:13 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T00:48:14.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:48:13 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T00:48:15.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:48:14 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:48:15.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:48:14 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 4825 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:48:15.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:48:14 smithi115 conmon[90393]: 2022-05-14T00:48:14.935+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:48:15.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:48:14 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:48:15.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:48:14 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 4825 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:48:16.222 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:48:15 smithi115 conmon[90393]: 2022-05-14T00:48:15.910+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:48:16.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:48:16 smithi047 ceph-mon[77147]: pgmap v2767: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 7 op/s 2022-05-14T00:48:16.552 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:48:16 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:48:16.553 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:48:16 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:48:16.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:48:16 smithi115 ceph-mon[78593]: pgmap v2767: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 7 op/s 2022-05-14T00:48:16.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:48:16 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:48:16.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:48:16 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:48:17.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:48:16 smithi115 conmon[90393]: 2022-05-14T00:48:16.940+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:48:18.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:48:17 smithi115 conmon[90393]: 2022-05-14T00:48:17.985+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:48:18.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:48:18 smithi115 ceph-mon[78593]: pgmap v2768: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 4 op/s 2022-05-14T00:48:18.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:48:18 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:48:18.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:48:18 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:48:18.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:48:18 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:48:18.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:48:18 smithi047 ceph-mon[77147]: pgmap v2768: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 4 op/s 2022-05-14T00:48:18.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:48:18 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:48:18.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:48:18 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:48:18.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:48:18 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:48:19.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:48:19 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:48:19.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:48:19 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 4830 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:48:19.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:48:19 smithi115 conmon[90393]: 2022-05-14T00:48:19.019+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:48:19.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:48:19 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:48:19.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:48:19 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 4830 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:48:20.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:48:20 smithi115 conmon[90393]: 2022-05-14T00:48:20.040+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:48:20.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:48:20 smithi115 ceph-mon[78593]: pgmap v2769: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 4 op/s 2022-05-14T00:48:20.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:48:20 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:48:20.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:48:20 smithi047 ceph-mon[77147]: pgmap v2769: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 4 op/s 2022-05-14T00:48:20.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:48:20 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:48:21.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:48:21 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:48:21.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:48:21 smithi115 conmon[90393]: 2022-05-14T00:48:21.063+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:48:21.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:48:21 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:48:22.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:48:22 smithi115 conmon[90393]: 2022-05-14T00:48:22.014+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:48:22.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:48:22 smithi115 ceph-mon[78593]: pgmap v2770: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 7 op/s 2022-05-14T00:48:22.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:48:22 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:48:22.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:48:22 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:48:22.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:48:22 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:48:22.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:48:22 smithi047 ceph-mon[77147]: pgmap v2770: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 7 op/s 2022-05-14T00:48:22.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:48:22 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:48:22.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:48:22 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:48:22.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:48:22 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:48:23.246 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:48:22 smithi115 conmon[90393]: 2022-05-14T00:48:22.965+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:48:23.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:48:23 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:48:23.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:48:23 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:48:24.248 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:48:23 smithi115 conmon[90393]: 2022-05-14T00:48:23.951+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:48:24.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:48:24 smithi047 ceph-mon[77147]: pgmap v2771: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 4 op/s 2022-05-14T00:48:24.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:48:24 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:48:24.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:48:24 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 4835 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:48:24.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:48:24 smithi115 ceph-mon[78593]: pgmap v2771: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 4 op/s 2022-05-14T00:48:24.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:48:24 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:48:24.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:48:24 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 4835 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:48:25.253 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:48:24 smithi115 conmon[90393]: 2022-05-14T00:48:24.990+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:48:25.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:48:25 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:48:25.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:48:25 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:48:26.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:48:26 smithi115 conmon[90393]: 2022-05-14T00:48:26.022+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:48:26.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:48:26 smithi115 ceph-mon[78593]: pgmap v2772: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 7 op/s 2022-05-14T00:48:26.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:48:26 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:48:26.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:48:26 smithi047 ceph-mon[77147]: pgmap v2772: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 7 op/s 2022-05-14T00:48:26.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:48:26 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:48:27.259 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:48:26 smithi115 conmon[90393]: 2022-05-14T00:48:26.983+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:48:27.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:48:27 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:48:27.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:48:27 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:48:28.261 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:48:27 smithi115 conmon[90393]: 2022-05-14T00:48:27.997+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:48:28.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:48:28 smithi047 ceph-mon[77147]: pgmap v2773: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:48:28.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:48:28 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:48:28.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:48:28 smithi115 ceph-mon[78593]: pgmap v2773: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:48:28.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:48:28 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:48:29.266 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:48:28 smithi115 conmon[90393]: 2022-05-14T00:48:28.991+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:48:29.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:48:29 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:48:29.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:48:29 smithi047 ceph-mon[77147]: pgmap v2774: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:48:29.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:48:29 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 4840 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:48:29.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:48:29 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:48:29.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:48:29 smithi115 ceph-mon[78593]: pgmap v2774: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:48:29.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:48:29 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 4840 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:48:30.267 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:48:29 smithi115 conmon[90393]: 2022-05-14T00:48:29.979+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:48:30.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:48:30 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:48:30.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:48:30 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:48:30.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:48:30 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:48:30.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:48:30 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:48:30.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:48:30 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:48:30.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:48:30 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:48:31.274 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:48:30 smithi115 conmon[90393]: 2022-05-14T00:48:30.978+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:48:31.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:48:31 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:48:31.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:48:31 smithi047 ceph-mon[77147]: pgmap v2775: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:48:31.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:48:31 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:48:31.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:48:31 smithi115 ceph-mon[78593]: pgmap v2775: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:48:32.273 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:48:31 smithi115 conmon[90393]: 2022-05-14T00:48:31.946+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:48:32.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:48:32 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:48:32.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:48:32 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:48:33.279 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:48:32 smithi115 conmon[90393]: 2022-05-14T00:48:32.991+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:48:33.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:48:33 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:48:33.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:48:33 smithi047 ceph-mon[77147]: pgmap v2776: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:48:33.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:48:33 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:48:33.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:48:33 smithi115 ceph-mon[78593]: pgmap v2776: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:48:34.280 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:48:34 smithi115 conmon[90393]: 2022-05-14T00:48:34.002+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:48:34.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:48:34 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:48:34.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:48:34 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:48:34.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:48:34 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:48:34.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:48:34 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 4845 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:48:34.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:48:34 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:48:34.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:48:34 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:48:34.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:48:34 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:48:34.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:48:34 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 4845 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:48:35.285 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:48:34 smithi115 conmon[90393]: 2022-05-14T00:48:34.962+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:48:35.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:48:35 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:48:35.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:48:35 smithi047 ceph-mon[77147]: pgmap v2777: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:48:35.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:48:35 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:48:35.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:48:35 smithi115 ceph-mon[78593]: pgmap v2777: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:48:36.286 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:48:35 smithi115 conmon[90393]: 2022-05-14T00:48:35.953+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:48:36.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:48:36 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:48:36.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:48:36 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:48:37.291 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:48:36 smithi115 conmon[90393]: 2022-05-14T00:48:36.977+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:48:37.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:48:37 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:48:37.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:48:37 smithi115 ceph-mon[78593]: pgmap v2778: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:48:37.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:48:37 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:48:37.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:48:37 smithi047 ceph-mon[77147]: pgmap v2778: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:48:38.292 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:48:37 smithi115 conmon[90393]: 2022-05-14T00:48:37.961+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:48:38.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:48:38 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:48:38.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:48:38 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:48:38.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:48:38 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:48:38.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:48:38 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:48:38.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:48:38 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:48:38.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:48:38 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:48:39.296 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:48:38 smithi115 conmon[90393]: 2022-05-14T00:48:38.973+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:48:39.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:48:39 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:48:39.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:48:39 smithi115 ceph-mon[78593]: pgmap v2779: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:48:39.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:48:39 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 4850 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:48:39.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:48:39 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:48:39.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:48:39 smithi047 ceph-mon[77147]: pgmap v2779: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:48:39.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:48:39 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 4850 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:48:40.298 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:48:39 smithi115 conmon[90393]: 2022-05-14T00:48:39.936+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:48:40.538 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:48:40 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:48:40.538 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:48:40 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T00:48:40.539 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:48:40 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T00:48:40.539 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:48:40 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T00:48:40.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:48:40 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:48:40.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:48:40 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T00:48:40.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:48:40 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T00:48:40.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:48:40 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T00:48:41.305 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:48:40 smithi115 conmon[90393]: 2022-05-14T00:48:40.919+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:48:41.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:48:41 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:48:41.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:48:41 smithi115 ceph-mon[78593]: pgmap v2780: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:48:41.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:48:41 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:48:41.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:48:41 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:48:41.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:48:41 smithi047 ceph-mon[77147]: pgmap v2780: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:48:41.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:48:41 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:48:42.309 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:48:41 smithi115 conmon[90393]: 2022-05-14T00:48:41.950+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:48:42.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:48:42 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:48:42.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:48:42 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:48:42.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:48:42 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:48:42.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:48:42 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:48:43.309 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:48:42 smithi115 conmon[90393]: 2022-05-14T00:48:42.970+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:48:43.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:48:43 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:48:43.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:48:43 smithi047 ceph-mon[77147]: pgmap v2781: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:48:43.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:48:43 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:48:43.542 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T00:48:43.542 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:48:43.542 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T00:48:43.542 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:48:43.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:48:43.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:48:43.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T00:48:43.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:48:43.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T00:48:43.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:48:43.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T00:48:43.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:48:43.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T00:48:43.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:48:43.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T00:48:43.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:48:43.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:48:43.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:48:43.546 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:48:43 smithi047 ceph-mon[77147]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T00:48:43.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:48:43 smithi047 conmon[78521]: 2022-05-14T00:48:43.120+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:48:43.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:48:43 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T00:48:43.547 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:48:43 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:48:43.547 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:48:43 smithi047 conmon[78521]: yield 2022-05-14T00:48:43.548 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:48:43 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:48:43.548 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:48:43 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:48:43.548 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:48:43 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:48:43.549 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:48:43 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T00:48:43.549 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:48:43 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:48:43.549 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:48:43 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T00:48:43.549 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:48:43 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:48:43.550 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:48:43 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T00:48:43.550 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:48:43 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:48:43.550 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:48:43 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T00:48:43.550 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:48:43 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:48:43.551 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:48:43 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T00:48:43.551 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:48:43 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:48:43.551 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:48:43 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:48:43.551 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:48:43 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:48:43.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:48:43 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:48:43.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:48:43 smithi115 ceph-mon[78593]: pgmap v2781: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:48:43.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:48:43 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:48:43.608 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T00:48:43.608 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:48:43.608 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T00:48:43.609 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:48:43.609 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:48:43.609 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:48:43.609 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T00:48:43.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:48:43.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T00:48:43.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:48:43.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T00:48:43.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:48:43.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T00:48:43.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:48:43.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T00:48:43.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:48:43.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:48:43.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:48:43.612 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:48:43 smithi115 ceph-mon[78593]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T00:48:44.310 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:48:43 smithi115 conmon[90393]: 2022-05-14T00:48:43.964+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:48:44.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:48:44 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:48:44.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:48:44 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 4855 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:48:44.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:48:44 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:48:44.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:48:44 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 4855 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:48:45.315 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:48:45 smithi115 conmon[90393]: 2022-05-14T00:48:45.013+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:48:45.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:48:45 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:48:45.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:48:45 smithi115 ceph-mon[78593]: pgmap v2782: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:48:45.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:48:45 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:48:45.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:48:45 smithi047 ceph-mon[77147]: pgmap v2782: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:48:46.319 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:48:45 smithi115 conmon[90393]: 2022-05-14T00:48:45.988+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:48:46.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:48:46 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:48:46.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:48:46 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:48:46.542 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T00:48:46.542 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:48:46.542 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T00:48:46.542 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:48:46.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:48:46.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:48:46.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T00:48:46.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:48:46.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T00:48:46.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:48:46.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T00:48:46.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:48:46.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T00:48:46.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:48:46.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T00:48:46.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:48:46.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:48:46.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:48:46.546 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:48:46 smithi047 ceph-mon[77147]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T00:48:46.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T00:48:46.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:48:46.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T00:48:46.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:48:46.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:48:46.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:48:46.548 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T00:48:46.548 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:48:46.548 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T00:48:46.548 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:48:46.549 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T00:48:46.549 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:48:46.549 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T00:48:46.549 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:48:46.549 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T00:48:46.550 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:48:46.550 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:48:46.550 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:48:46.550 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T00:48:46.551 INFO:journalctl@ceph.mon.b.smithi047.stdout: During handling of the above exception, another exception occurred: 2022-05-14T00:48:46.551 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T00:48:46.551 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T00:48:46.551 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T00:48:46.552 INFO:journalctl@ceph.mon.b.smithi047.stdout: return f(*arg) 2022-05-14T00:48:46.552 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T00:48:46.552 INFO:journalctl@ceph.mon.b.smithi047.stdout: self._write_client_files(client_files, host) 2022-05-14T00:48:46.552 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T00:48:46.553 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T00:48:46.553 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T00:48:46.553 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T00:48:46.553 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T00:48:46.554 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.event_loop.get_result(coro) 2022-05-14T00:48:46.554 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T00:48:46.554 INFO:journalctl@ceph.mon.b.smithi047.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T00:48:46.554 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T00:48:46.554 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.__get_result() 2022-05-14T00:48:46.555 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T00:48:46.555 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise self._exception 2022-05-14T00:48:46.555 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T00:48:46.555 INFO:journalctl@ceph.mon.b.smithi047.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T00:48:46.556 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T00:48:46.556 INFO:journalctl@ceph.mon.b.smithi047.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T00:48:46.556 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T00:48:46.556 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise 2022-05-14T00:48:46.556 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T00:48:46.557 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.gen.throw(type, value, traceback) 2022-05-14T00:48:46.557 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T00:48:46.557 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OrchestratorError(msg) 2022-05-14T00:48:46.557 INFO:journalctl@ceph.mon.b.smithi047.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:48:46.558 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:48:46 smithi047 conmon[78521]: 2022-05-14T00:48:46.191+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:48:46.559 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:48:46 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T00:48:46.559 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:48:46 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:48:46.559 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:48:46 smithi047 conmon[78521]: yield 2022-05-14T00:48:46.559 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:48:46 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:48:46.559 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:48:46 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:48:46.560 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:48:46 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:48:46.560 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:48:46 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T00:48:46.560 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:48:46 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:48:46.560 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:48:46 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T00:48:46.561 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:48:46 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:48:46.561 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:48:46 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T00:48:46.561 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:48:46 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:48:46.561 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:48:46 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T00:48:46.561 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:48:46 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:48:46.562 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:48:46 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T00:48:46.562 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:48:46 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:48:46.562 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:48:46 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:48:46.562 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:48:46 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:48:46.563 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:48:46 smithi047 conmon[78521]: 2022-05-14T00:48:46.192+0000 7f533ff78700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T00:48:46.563 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:48:46 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T00:48:46.563 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:48:46 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:48:46.563 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:48:46 smithi047 conmon[78521]: yield 2022-05-14T00:48:46.564 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:48:46 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:48:46.564 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:48:46 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:48:46.564 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:48:46 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:48:46.564 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:48:46 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T00:48:46.565 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:48:46 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:48:46.565 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:48:46 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T00:48:46.565 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:48:46 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:48:46.565 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:48:46 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T00:48:46.566 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:48:46 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:48:46.566 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:48:46 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T00:48:46.566 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:48:46 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:48:46.566 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:48:46 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T00:48:46.567 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:48:46 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:48:46.567 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:48:46 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:48:46.567 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:48:46 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:48:46.567 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:48:46 smithi047 conmon[78521]: 2022-05-14T00:48:46.568 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:48:46 smithi047 conmon[78521]: During handling of the above exception, another exception occurred: 2022-05-14T00:48:46.568 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:48:46 smithi047 conmon[78521]: 2022-05-14T00:48:46.568 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:48:46 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T00:48:46.568 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:48:46 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T00:48:46.568 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:48:46 smithi047 conmon[78521]: return f(*arg) 2022-05-14T00:48:46.569 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:48:46 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T00:48:46.569 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:48:46 smithi047 conmon[78521]: self._write_client_files(client_files, host) 2022-05-14T00:48:46.569 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:48:46 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T00:48:46.569 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:48:46 smithi047 conmon[78521]: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T00:48:46.570 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:48:46 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T00:48:46.570 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:48:46 smithi047 conmon[78521]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T00:48:46.570 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:48:46 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T00:48:46.570 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:48:46 smithi047 conmon[78521]: return self.event_loop.get_result(coro) 2022-05-14T00:48:46.571 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:48:46 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T00:48:46.571 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:48:46 smithi047 conmon[78521]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T00:48:46.571 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:48:46 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T00:48:46.571 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:48:46 smithi047 conmon[78521]: return self.__get_result() 2022-05-14T00:48:46.571 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:48:46 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T00:48:46.572 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:48:46 smithi047 conmon[78521]: raise self._exception 2022-05-14T00:48:46.572 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:48:46 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T00:48:46.572 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:48:46 smithi047 conmon[78521]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T00:48:46.572 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:48:46 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T00:48:46.573 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:48:46 smithi047 conmon[78521]: conn = await self._remote_connection(host, addr) 2022-05-14T00:48:46.573 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:48:46 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T00:48:46.573 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:48:46 smithi047 conmon[78521]: raise 2022-05-14T00:48:46.573 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:48:46 smithi047 conmon[78521]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T00:48:46.573 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:48:46 smithi047 conmon[78521]: self.gen.throw(type, value, traceback) 2022-05-14T00:48:46.574 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:48:46 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T00:48:46.574 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:48:46 smithi047 conmon[78521]: raise OrchestratorError(msg) 2022-05-14T00:48:46.574 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:48:46 smithi047 conmon[78521]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:48:46.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:48:46 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:48:46.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:48:46 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:48:46.608 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T00:48:46.608 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:48:46.609 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T00:48:46.609 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:48:46.609 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:48:46.609 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:48:46.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T00:48:46.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:48:46.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T00:48:46.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:48:46.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T00:48:46.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:48:46.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T00:48:46.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:48:46.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T00:48:46.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:48:46.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:48:46.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:48:46.612 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:48:46 smithi115 ceph-mon[78593]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T00:48:46.613 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T00:48:46.613 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:48:46.613 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T00:48:46.613 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:48:46.614 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:48:46.614 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:48:46.614 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T00:48:46.614 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:48:46.615 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T00:48:46.615 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:48:46.615 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T00:48:46.615 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:48:46.615 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T00:48:46.616 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:48:46.616 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T00:48:46.616 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:48:46.616 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:48:46.617 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:48:46.617 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T00:48:46.617 INFO:journalctl@ceph.mon.c.smithi115.stdout: During handling of the above exception, another exception occurred: 2022-05-14T00:48:46.617 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T00:48:46.617 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T00:48:46.618 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T00:48:46.618 INFO:journalctl@ceph.mon.c.smithi115.stdout: return f(*arg) 2022-05-14T00:48:46.618 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T00:48:46.618 INFO:journalctl@ceph.mon.c.smithi115.stdout: self._write_client_files(client_files, host) 2022-05-14T00:48:46.619 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T00:48:46.619 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T00:48:46.619 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T00:48:46.619 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T00:48:46.619 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T00:48:46.620 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.event_loop.get_result(coro) 2022-05-14T00:48:46.620 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T00:48:46.620 INFO:journalctl@ceph.mon.c.smithi115.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T00:48:46.620 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T00:48:46.621 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.__get_result() 2022-05-14T00:48:46.621 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T00:48:46.621 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise self._exception 2022-05-14T00:48:46.621 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T00:48:46.621 INFO:journalctl@ceph.mon.c.smithi115.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T00:48:46.622 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T00:48:46.622 INFO:journalctl@ceph.mon.c.smithi115.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T00:48:46.622 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T00:48:46.622 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise 2022-05-14T00:48:46.623 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T00:48:46.623 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.gen.throw(type, value, traceback) 2022-05-14T00:48:46.623 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T00:48:46.623 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OrchestratorError(msg) 2022-05-14T00:48:46.624 INFO:journalctl@ceph.mon.c.smithi115.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:48:47.324 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:48:46 smithi115 conmon[90393]: 2022-05-14T00:48:46.950+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:48:47.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:48:47 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:48:47.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:48:47 smithi115 ceph-mon[78593]: pgmap v2783: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:48:47.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:48:47 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:48:47.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:48:47 smithi047 ceph-mon[77147]: pgmap v2783: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:48:48.323 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:48:48 smithi115 conmon[90393]: 2022-05-14T00:48:48.000+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:48:48.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:48:48 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:48:48.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:48:48 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:48:49.332 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:48:49 smithi115 conmon[90393]: 2022-05-14T00:48:49.005+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:48:49.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:48:49 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:48:49.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:48:49 smithi115 ceph-mon[78593]: pgmap v2784: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:48:49.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:48:49 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 4860 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:48:49.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:48:49 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:48:49.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:48:49 smithi047 ceph-mon[77147]: pgmap v2784: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:48:49.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:48:49 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 4860 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:48:50.332 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:48:50 smithi115 conmon[90393]: 2022-05-14T00:48:50.008+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:48:50.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:48:50 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:48:50.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:48:50 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:48:50.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:48:50 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:48:50.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:48:50 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:48:50.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:48:50 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:48:50.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:48:50 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:48:51.336 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:48:51 smithi115 conmon[90393]: 2022-05-14T00:48:51.026+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:48:51.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:48:51 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:48:51.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:48:51 smithi115 ceph-mon[78593]: pgmap v2785: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:48:51.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:48:51 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:48:51.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:48:51 smithi047 ceph-mon[77147]: pgmap v2785: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:48:52.338 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:48:52 smithi115 conmon[90393]: 2022-05-14T00:48:52.000+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:48:52.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:48:52 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:48:52.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:48:52 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:48:53.344 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:48:52 smithi115 conmon[90393]: 2022-05-14T00:48:52.990+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:48:53.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:48:53 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:48:53.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:48:53 smithi115 ceph-mon[78593]: pgmap v2786: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:48:53.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:48:53 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:48:53.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:48:53 smithi047 ceph-mon[77147]: pgmap v2786: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:48:54.344 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:48:53 smithi115 conmon[90393]: 2022-05-14T00:48:53.955+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:48:54.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:48:54 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:48:54.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:48:54 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:48:54.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:48:54 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:48:54.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:48:54 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 4865 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:48:54.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:48:54 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:48:54.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:48:54 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:48:54.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:48:54 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:48:54.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:48:54 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 4865 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:48:55.348 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:48:54 smithi115 conmon[90393]: 2022-05-14T00:48:54.935+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:48:55.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:48:55 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:48:55.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:48:55 smithi115 ceph-mon[78593]: pgmap v2787: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:48:55.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:48:55 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:48:55.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:48:55 smithi047 ceph-mon[77147]: pgmap v2787: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:48:56.350 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:48:55 smithi115 conmon[90393]: 2022-05-14T00:48:55.895+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:48:56.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:48:56 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:48:56.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:48:56 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:48:56.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:48:56 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:48:56.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:48:56 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:48:57.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:48:56 smithi115 conmon[90393]: 2022-05-14T00:48:56.938+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:48:57.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:48:57 smithi047 ceph-mon[77147]: pgmap v2788: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:48:57.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:48:57 smithi115 ceph-mon[78593]: pgmap v2788: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:48:58.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:48:57 smithi115 conmon[90393]: 2022-05-14T00:48:57.914+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:48:58.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:48:58 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:48:58.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:48:58 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:48:58.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:48:58 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:48:58.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:48:58 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:48:58.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:48:58 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:48:58.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:48:58 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:48:58.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:48:58 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:48:58.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:48:58 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:48:59.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:48:58 smithi115 conmon[90393]: 2022-05-14T00:48:58.923+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:48:59.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:48:59 smithi047 ceph-mon[77147]: pgmap v2789: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:48:59.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:48:59 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:48:59.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:48:59 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 4870 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:48:59.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:48:59 smithi115 ceph-mon[78593]: pgmap v2789: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:48:59.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:48:59 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:48:59.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:48:59 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 4870 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:49:00.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:48:59 smithi115 conmon[90393]: 2022-05-14T00:48:59.916+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:49:00.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:49:00 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:49:00.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:49:00 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:49:01.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:49:00 smithi115 conmon[90393]: 2022-05-14T00:49:00.897+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:49:01.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:49:01 smithi047 ceph-mon[77147]: pgmap v2790: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:49:01.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:49:01 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:49:01.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:49:01 smithi115 ceph-mon[78593]: pgmap v2790: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:49:01.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:49:01 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:49:02.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:49:01 smithi115 conmon[90393]: 2022-05-14T00:49:01.860+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:49:02.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:49:02 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:49:02.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:49:02 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:49:02.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:49:02 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:49:02.846 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:49:02 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:49:02.847 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:49:02 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:49:02.847 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:49:02 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:49:03.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:49:02 smithi115 conmon[90393]: 2022-05-14T00:49:02.846+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:49:03.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:49:03 smithi047 ceph-mon[77147]: pgmap v2791: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:49:03.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:49:03 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:49:03.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:49:03 smithi115 ceph-mon[78593]: pgmap v2791: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:49:03.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:49:03 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:49:04.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:49:03 smithi115 conmon[90393]: 2022-05-14T00:49:03.873+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:49:04.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:49:04 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:49:04.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:49:04 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 4875 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:49:04.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:49:04 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:49:04.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:49:04 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 4875 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:49:05.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:49:04 smithi115 conmon[90393]: 2022-05-14T00:49:04.877+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:49:05.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:49:05 smithi047 ceph-mon[77147]: pgmap v2792: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:49:05.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:49:05 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:49:05.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:49:05 smithi115 ceph-mon[78593]: pgmap v2792: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:49:05.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:49:05 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:49:06.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:49:05 smithi115 conmon[90393]: 2022-05-14T00:49:05.901+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:49:06.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:49:06 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:49:06.853 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:49:06 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:49:07.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:49:06 smithi115 conmon[90393]: 2022-05-14T00:49:06.852+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:49:07.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:49:07 smithi047 ceph-mon[77147]: pgmap v2793: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:49:07.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:49:07 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:49:07.804 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:49:07 smithi115 ceph-mon[78593]: pgmap v2793: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:49:07.804 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:49:07 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:49:08.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:49:07 smithi115 conmon[90393]: 2022-05-14T00:49:07.803+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:49:08.761 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:49:08 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:49:08.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:49:08 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:49:09.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:49:08 smithi115 conmon[90393]: 2022-05-14T00:49:08.761+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:49:09.728 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:49:09 smithi115 ceph-mon[78593]: pgmap v2794: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:49:09.729 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:49:09 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:49:09.729 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:49:09 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 4880 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:49:09.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:49:09 smithi047 ceph-mon[77147]: pgmap v2794: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:49:09.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:49:09 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:49:09.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:49:09 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 4880 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:49:10.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:49:09 smithi115 conmon[90393]: 2022-05-14T00:49:09.727+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:49:10.765 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:49:10 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:49:10.765 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:49:10 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:49:10.765 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:49:10 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:49:10.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:49:10 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:49:10.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:49:10 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:49:10.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:49:10 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:49:11.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:49:10 smithi115 conmon[90393]: 2022-05-14T00:49:10.763+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:49:11.750 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:49:11 smithi115 ceph-mon[78593]: pgmap v2795: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:49:11.750 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:49:11 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:49:11.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:49:11 smithi047 ceph-mon[77147]: pgmap v2795: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:49:11.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:49:11 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:49:12.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:49:11 smithi115 conmon[90393]: 2022-05-14T00:49:11.745+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:49:12.770 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:49:12 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:49:12.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:49:12 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:49:13.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:49:12 smithi115 conmon[90393]: 2022-05-14T00:49:12.769+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:49:13.782 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:49:13 smithi115 ceph-mon[78593]: pgmap v2796: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:49:13.783 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:49:13 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:49:13.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:49:13 smithi047 ceph-mon[77147]: pgmap v2796: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:49:13.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:49:13 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:49:14.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:49:13 smithi115 conmon[90393]: 2022-05-14T00:49:13.781+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:49:14.739 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:49:14 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:49:14.739 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:49:14 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:49:14.740 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:49:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T00:49:14.740 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:49:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T00:49:14.740 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:49:14 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:49:14.740 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:49:14 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 4885 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:49:14.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:49:14 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:49:14.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:49:14 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:49:14.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:49:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T00:49:14.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:49:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T00:49:14.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:49:14 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:49:14.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:49:14 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 4885 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:49:15.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:49:14 smithi115 conmon[90393]: 2022-05-14T00:49:14.738+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:49:15.728 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:49:15 smithi115 ceph-mon[78593]: pgmap v2797: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:49:15.728 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:49:15 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:49:15.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:49:15 smithi047 ceph-mon[77147]: pgmap v2797: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:49:15.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:49:15 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:49:16.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:49:15 smithi115 conmon[90393]: 2022-05-14T00:49:15.727+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:49:16.698 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:49:16 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:49:16.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:49:16 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:49:17.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:49:16 smithi115 conmon[90393]: 2022-05-14T00:49:16.697+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:49:17.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:49:17 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:49:17.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:49:17 smithi047 ceph-mon[77147]: pgmap v2798: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:49:17.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:49:17 smithi115 conmon[90393]: 2022-05-14T00:49:17.660+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:49:17.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:49:17 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:49:17.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:49:17 smithi115 ceph-mon[78593]: pgmap v2798: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:49:18.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:49:18 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:49:18.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:49:18 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:49:18.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:49:18 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:49:18.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:49:18 smithi115 conmon[90393]: 2022-05-14T00:49:18.640+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:49:18.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:49:18 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:49:18.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:49:18 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:49:18.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:49:18 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:49:19.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:49:19 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:49:19.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:49:19 smithi047 ceph-mon[77147]: pgmap v2799: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:49:19.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:49:19 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 4890 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:49:19.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:49:19 smithi115 conmon[90393]: 2022-05-14T00:49:19.632+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:49:19.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:49:19 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:49:19.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:49:19 smithi115 ceph-mon[78593]: pgmap v2799: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:49:19.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:49:19 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 4890 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:49:20.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:49:20 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:49:20.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:49:20 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:49:20.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:49:20 smithi115 conmon[90393]: 2022-05-14T00:49:20.629+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:49:21.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:49:21 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:49:21.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:49:21 smithi047 ceph-mon[77147]: pgmap v2800: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:49:21.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:49:21 smithi115 conmon[90393]: 2022-05-14T00:49:21.595+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:49:21.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:49:21 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:49:21.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:49:21 smithi115 ceph-mon[78593]: pgmap v2800: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:49:22.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:49:22 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:49:22.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:49:22 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:49:22.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:49:22 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:49:22.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:49:22 smithi115 conmon[90393]: 2022-05-14T00:49:22.581+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:49:22.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:49:22 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:49:22.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:49:22 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:49:22.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:49:22 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:49:23.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:49:23 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:49:23.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:49:23 smithi047 ceph-mon[77147]: pgmap v2801: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:49:23.870 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:49:23 smithi115 conmon[90393]: 2022-05-14T00:49:23.558+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:49:23.870 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:49:23 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:49:23.871 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:49:23 smithi115 ceph-mon[78593]: pgmap v2801: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:49:24.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:49:24 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:49:24.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:49:24 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 4895 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:49:24.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:49:24 smithi115 conmon[90393]: 2022-05-14T00:49:24.564+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:49:24.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:49:24 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:49:24.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:49:24 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 4895 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:49:25.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:49:25 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:49:25.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:49:25 smithi047 ceph-mon[77147]: pgmap v2802: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:49:25.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:49:25 smithi115 conmon[90393]: 2022-05-14T00:49:25.556+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:49:25.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:49:25 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:49:25.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:49:25 smithi115 ceph-mon[78593]: pgmap v2802: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:49:26.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:49:26 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:49:26.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:49:26 smithi115 conmon[90393]: 2022-05-14T00:49:26.594+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:49:26.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:49:26 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:49:27.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:49:27 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:49:27.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:49:27 smithi047 ceph-mon[77147]: pgmap v2803: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:49:27.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:49:27 smithi115 conmon[90393]: 2022-05-14T00:49:27.640+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:49:27.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:49:27 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:49:27.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:49:27 smithi115 ceph-mon[78593]: pgmap v2803: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:49:28.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:49:28 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:49:28.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:49:28 smithi115 conmon[90393]: 2022-05-14T00:49:28.607+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:49:28.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:49:28 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:49:29.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:49:29 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:49:29.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:49:29 smithi047 ceph-mon[77147]: pgmap v2804: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:49:29.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:49:29 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 4900 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:49:29.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:49:29 smithi115 conmon[90393]: 2022-05-14T00:49:29.608+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:49:29.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:49:29 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:49:29.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:49:29 smithi115 ceph-mon[78593]: pgmap v2804: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:49:29.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:49:29 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 4900 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:49:30.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:49:30 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:49:30.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:49:30 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:49:30.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:49:30 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:49:30.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:49:30 smithi115 conmon[90393]: 2022-05-14T00:49:30.638+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:49:30.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:49:30 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:49:30.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:49:30 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:49:30.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:49:30 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:49:31.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:49:31 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:49:31.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:49:31 smithi047 ceph-mon[77147]: pgmap v2805: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:49:31.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:49:31 smithi115 conmon[90393]: 2022-05-14T00:49:31.636+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:49:31.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:49:31 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:49:31.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:49:31 smithi115 ceph-mon[78593]: pgmap v2805: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:49:32.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:49:32 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:49:32.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:49:32 smithi115 conmon[90393]: 2022-05-14T00:49:32.685+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:49:32.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:49:32 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:49:33.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:49:33 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:49:33.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:49:33 smithi047 ceph-mon[77147]: pgmap v2806: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:49:33.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:49:33 smithi115 conmon[90393]: 2022-05-14T00:49:33.711+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:49:33.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:49:33 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:49:33.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:49:33 smithi115 ceph-mon[78593]: pgmap v2806: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:49:34.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:49:34 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:49:34.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:49:34 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:49:34.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:49:34 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:49:34.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:49:34 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 4905 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:49:34.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:49:34 smithi115 conmon[90393]: 2022-05-14T00:49:34.721+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:49:34.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:49:34 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:49:34.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:49:34 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:49:34.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:49:34 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:49:34.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:49:34 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 4905 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:49:35.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:49:35 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:49:35.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:49:35 smithi047 ceph-mon[77147]: pgmap v2807: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:49:35.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:49:35 smithi115 conmon[90393]: 2022-05-14T00:49:35.695+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:49:35.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:49:35 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:49:35.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:49:35 smithi115 ceph-mon[78593]: pgmap v2807: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:49:36.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:49:36 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:49:36.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:49:36 smithi115 conmon[90393]: 2022-05-14T00:49:36.684+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:49:36.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:49:36 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:49:37.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:49:37 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:49:37.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:49:37 smithi047 ceph-mon[77147]: pgmap v2808: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:49:37.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:49:37 smithi115 conmon[90393]: 2022-05-14T00:49:37.678+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:49:37.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:49:37 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:49:37.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:49:37 smithi115 ceph-mon[78593]: pgmap v2808: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:49:38.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:49:38 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:49:38.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:49:38 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:49:38.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:49:38 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:49:38.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:49:38 smithi115 conmon[90393]: 2022-05-14T00:49:38.641+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:49:38.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:49:38 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:49:38.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:49:38 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:49:38.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:49:38 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:49:39.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:49:39 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:49:39.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:49:39 smithi047 ceph-mon[77147]: pgmap v2809: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:49:39.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:49:39 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 4910 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:49:39.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:49:39 smithi115 conmon[90393]: 2022-05-14T00:49:39.679+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:49:39.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:49:39 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:49:39.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:49:39 smithi115 ceph-mon[78593]: pgmap v2809: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:49:39.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:49:39 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 4910 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:49:40.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:49:40 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:49:40.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:49:40 smithi115 conmon[90393]: 2022-05-14T00:49:40.664+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:49:40.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:49:40 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:49:41.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:49:41 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:49:41.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:49:41 smithi047 ceph-mon[77147]: pgmap v2810: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:49:41.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:49:41 smithi115 conmon[90393]: 2022-05-14T00:49:41.641+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:49:41.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:49:41 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:49:41.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:49:41 smithi115 ceph-mon[78593]: pgmap v2810: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:49:42.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:49:42 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:49:42.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:49:42 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:49:42.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:49:42 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:49:42.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:49:42 smithi115 conmon[90393]: 2022-05-14T00:49:42.591+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:49:42.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:49:42 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:49:42.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:49:42 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:49:42.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:49:42 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:49:43.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:49:43 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:49:43.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:49:43 smithi047 ceph-mon[77147]: pgmap v2811: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:49:43.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:49:43 smithi115 conmon[90393]: 2022-05-14T00:49:43.574+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:49:43.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:49:43 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:49:43.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:49:43 smithi115 ceph-mon[78593]: pgmap v2811: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:49:44.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:49:44 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:49:44.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:49:44 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 4915 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:49:44.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:49:44 smithi115 conmon[90393]: 2022-05-14T00:49:44.586+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:49:44.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:49:44 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:49:44.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:49:44 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 4915 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:49:45.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:49:45 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:49:45.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:49:45 smithi047 ceph-mon[77147]: pgmap v2812: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:49:45.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:49:45 smithi115 conmon[90393]: 2022-05-14T00:49:45.565+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:49:45.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:49:45 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:49:45.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:49:45 smithi115 ceph-mon[78593]: pgmap v2812: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:49:46.515 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:49:46 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:49:46.516 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:49:46 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T00:49:46.516 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:49:46 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T00:49:46.516 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:49:46 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T00:49:46.785 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:49:46 smithi115 conmon[90393]: 2022-05-14T00:49:46.585+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:49:46.785 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:49:46 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:49:46.785 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:49:46 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T00:49:46.786 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:49:46 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T00:49:46.786 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:49:46 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T00:49:47.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:49:47 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:49:47.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:49:47 smithi047 ceph-mon[77147]: pgmap v2813: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:49:47.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:49:47 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' 2022-05-14T00:49:47.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:49:47 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' 2022-05-14T00:49:47.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:49:47 smithi115 conmon[90393]: 2022-05-14T00:49:47.566+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:49:47.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:49:47 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:49:47.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:49:47 smithi115 ceph-mon[78593]: pgmap v2813: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:49:47.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:49:47 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' 2022-05-14T00:49:47.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:49:47 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' 2022-05-14T00:49:48.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:49:48 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:49:48.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:49:48 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:49:48.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:49:48 smithi115 conmon[90393]: 2022-05-14T00:49:48.520+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:49:49.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:49:49 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:49:49.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:49:49 smithi047 ceph-mon[77147]: pgmap v2814: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:49:49.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:49:49 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 4920 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:49:49.792 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:49:49 smithi047 conmon[78521]: 2022-05-14T00:49:49.295+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:49:49.792 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:49:49 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T00:49:49.792 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:49:49 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:49:49.793 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:49:49 smithi047 conmon[78521]: yield 2022-05-14T00:49:49.793 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:49:49 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:49:49.793 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:49:49 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:49:49.793 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:49:49 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:49:49.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:49:49 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T00:49:49.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:49:49 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:49:49.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:49:49 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T00:49:49.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:49:49 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:49:49.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:49:49 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T00:49:49.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:49:49 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:49:49.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:49:49 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T00:49:49.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:49:49 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:49:49.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:49:49 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T00:49:49.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:49:49 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:49:49.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:49:49 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:49:49.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:49:49 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:49:49.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:49:49 smithi115 conmon[90393]: 2022-05-14T00:49:49.562+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:49:49.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:49:49 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:49:49.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:49:49 smithi115 ceph-mon[78593]: pgmap v2814: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:49:49.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:49:49 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 4920 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:49:50.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:49:50 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:49:50.791 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T00:49:50.791 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:49:50.792 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T00:49:50.792 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:49:50.792 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:49:50.792 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:49:50.793 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T00:49:50.793 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:49:50.793 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T00:49:50.793 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:49:50.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T00:49:50.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:49:50.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T00:49:50.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:49:50.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T00:49:50.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:49:50.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:49:50.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:49:50.795 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:49:50 smithi047 ceph-mon[77147]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T00:49:50.796 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:49:50 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:49:50.796 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:49:50 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:49:50.796 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:49:50 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:49:50.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:49:50 smithi115 conmon[90393]: 2022-05-14T00:49:50.585+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:49:50.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:49:50 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:49:50.858 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T00:49:50.858 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:49:50.859 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T00:49:50.859 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:49:50.859 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:49:50.859 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:49:50.860 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T00:49:50.860 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:49:50.860 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T00:49:50.860 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:49:50.860 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T00:49:50.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:49:50.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T00:49:50.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:49:50.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T00:49:50.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:49:50.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:49:50.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:49:50.862 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:49:50 smithi115 ceph-mon[78593]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T00:49:50.862 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:49:50 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:49:50.863 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:49:50 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:49:50.863 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:49:50 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:49:51.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:49:51 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:49:51.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:49:51 smithi047 ceph-mon[77147]: pgmap v2815: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:49:51.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:49:51 smithi115 conmon[90393]: 2022-05-14T00:49:51.545+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:49:51.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:49:51 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:49:51.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:49:51 smithi115 ceph-mon[78593]: pgmap v2815: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:49:52.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:49:52 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:49:52.791 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:49:52 smithi047 conmon[78521]: 2022-05-14T00:49:52.367+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:49:52.791 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:49:52 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T00:49:52.792 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:49:52 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:49:52.792 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:49:52 smithi047 conmon[78521]: yield 2022-05-14T00:49:52.792 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:49:52 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:49:52.792 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:49:52 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:49:52.793 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:49:52 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:49:52.793 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:49:52 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T00:49:52.793 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:49:52 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:49:52.793 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:49:52 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T00:49:52.793 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:49:52 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:49:52.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:49:52 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T00:49:52.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:49:52 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:49:52.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:49:52 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T00:49:52.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:49:52 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:49:52.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:49:52 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T00:49:52.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:49:52 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:49:52.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:49:52 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:49:52.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:49:52 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:49:52.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:49:52 smithi047 conmon[78521]: 2022-05-14T00:49:52.368+0000 7f5340f7a700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T00:49:52.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:49:52 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T00:49:52.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:49:52 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:49:52.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:49:52 smithi047 conmon[78521]: yield 2022-05-14T00:49:52.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:49:52 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:49:52.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:49:52 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:49:52.797 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:49:52 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:49:52.797 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:49:52 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T00:49:52.797 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:49:52 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:49:52.797 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:49:52 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T00:49:52.797 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:49:52 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:49:52.798 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:49:52 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T00:49:52.798 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:49:52 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:49:52.798 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:49:52 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T00:49:52.798 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:49:52 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:49:52.799 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:49:52 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T00:49:52.799 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:49:52 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:49:52.799 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:49:52 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:49:52.799 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:49:52 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:49:52.799 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:49:52 smithi047 conmon[78521]: 2022-05-14T00:49:52.800 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:49:52 smithi047 conmon[78521]: During handling of the above exception, another exception occurred: 2022-05-14T00:49:52.800 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:49:52 smithi047 conmon[78521]: 2022-05-14T00:49:52.800 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:49:52 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T00:49:52.800 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:49:52 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T00:49:52.800 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:49:52 smithi047 conmon[78521]: return f(*arg) 2022-05-14T00:49:52.801 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:49:52 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T00:49:52.801 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:49:52 smithi047 conmon[78521]: self._write_client_files(client_files, host) 2022-05-14T00:49:52.801 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:49:52 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T00:49:52.801 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:49:52 smithi047 conmon[78521]: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T00:49:52.802 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:49:52 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T00:49:52.802 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:49:52 smithi047 conmon[78521]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T00:49:52.802 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:49:52 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T00:49:52.802 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:49:52 smithi047 conmon[78521]: return self.event_loop.get_result(coro) 2022-05-14T00:49:52.802 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:49:52 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T00:49:52.803 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:49:52 smithi047 conmon[78521]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T00:49:52.803 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:49:52 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T00:49:52.803 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:49:52 smithi047 conmon[78521]: return self.__get_result() 2022-05-14T00:49:52.803 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:49:52 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T00:49:52.803 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:49:52 smithi047 conmon[78521]: raise self._exception 2022-05-14T00:49:52.804 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:49:52 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T00:49:52.804 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:49:52 smithi047 conmon[78521]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T00:49:52.804 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:49:52 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T00:49:52.804 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:49:52 smithi047 conmon[78521]: conn = await self._remote_connection(host, addr) 2022-05-14T00:49:52.804 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:49:52 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T00:49:52.805 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:49:52 smithi047 conmon[78521]: raise 2022-05-14T00:49:52.805 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:49:52 smithi047 conmon[78521]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T00:49:52.805 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:49:52 smithi047 conmon[78521]: self.gen.throw(type, value, traceback) 2022-05-14T00:49:52.805 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:49:52 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T00:49:52.805 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:49:52 smithi047 conmon[78521]: raise OrchestratorError(msg) 2022-05-14T00:49:52.806 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:49:52 smithi047 conmon[78521]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:49:52.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:49:52 smithi115 conmon[90393]: 2022-05-14T00:49:52.499+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:49:52.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:49:52 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:49:53.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:49:53 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:49:53.792 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T00:49:53.792 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:49:53.792 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T00:49:53.793 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:49:53.793 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:49:53.793 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:49:53.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T00:49:53.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:49:53.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T00:49:53.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:49:53.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T00:49:53.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:49:53.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T00:49:53.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:49:53.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T00:49:53.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:49:53.797 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:49:53.797 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:49:53.797 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:49:53 smithi047 ceph-mon[77147]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T00:49:53.798 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T00:49:53.798 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:49:53.798 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T00:49:53.798 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:49:53.798 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:49:53.799 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:49:53.799 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T00:49:53.799 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:49:53.799 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T00:49:53.800 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:49:53.800 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T00:49:53.800 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:49:53.800 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T00:49:53.801 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:49:53.801 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T00:49:53.802 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:49:53.802 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:49:53.802 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:49:53.803 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T00:49:53.803 INFO:journalctl@ceph.mon.b.smithi047.stdout: During handling of the above exception, another exception occurred: 2022-05-14T00:49:53.803 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T00:49:53.804 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T00:49:53.804 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T00:49:53.804 INFO:journalctl@ceph.mon.b.smithi047.stdout: return f(*arg) 2022-05-14T00:49:53.804 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T00:49:53.805 INFO:journalctl@ceph.mon.b.smithi047.stdout: self._write_client_files(client_files, host) 2022-05-14T00:49:53.805 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T00:49:53.805 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T00:49:53.805 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T00:49:53.806 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T00:49:53.806 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T00:49:53.806 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.event_loop.get_result(coro) 2022-05-14T00:49:53.806 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T00:49:53.806 INFO:journalctl@ceph.mon.b.smithi047.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T00:49:53.807 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T00:49:53.807 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.__get_result() 2022-05-14T00:49:53.807 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T00:49:53.807 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise self._exception 2022-05-14T00:49:53.808 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T00:49:53.808 INFO:journalctl@ceph.mon.b.smithi047.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T00:49:53.808 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T00:49:53.809 INFO:journalctl@ceph.mon.b.smithi047.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T00:49:53.809 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T00:49:53.809 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise 2022-05-14T00:49:53.809 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T00:49:53.810 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.gen.throw(type, value, traceback) 2022-05-14T00:49:53.810 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T00:49:53.810 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OrchestratorError(msg) 2022-05-14T00:49:53.810 INFO:journalctl@ceph.mon.b.smithi047.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:49:53.811 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:49:53 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:49:53.811 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:49:53 smithi047 ceph-mon[77147]: pgmap v2816: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:49:53.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:49:53 smithi115 conmon[90393]: 2022-05-14T00:49:53.524+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:49:53.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:49:53 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:49:53.858 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T00:49:53.859 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:49:53.859 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T00:49:53.859 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:49:53.859 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:49:53.860 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:49:53.860 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T00:49:53.860 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:49:53.860 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T00:49:53.860 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:49:53.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T00:49:53.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:49:53.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T00:49:53.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:49:53.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T00:49:53.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:49:53.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:49:53.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:49:53.862 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:49:53 smithi115 ceph-mon[78593]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T00:49:53.863 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T00:49:53.863 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:49:53.863 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T00:49:53.863 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:49:53.864 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:49:53.864 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:49:53.864 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T00:49:53.864 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:49:53.864 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T00:49:53.865 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:49:53.865 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T00:49:53.865 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:49:53.866 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T00:49:53.866 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:49:53.866 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T00:49:53.866 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:49:53.866 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:49:53.867 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:49:53.867 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T00:49:53.867 INFO:journalctl@ceph.mon.c.smithi115.stdout: During handling of the above exception, another exception occurred: 2022-05-14T00:49:53.867 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T00:49:53.868 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T00:49:53.868 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T00:49:53.868 INFO:journalctl@ceph.mon.c.smithi115.stdout: return f(*arg) 2022-05-14T00:49:53.868 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T00:49:53.868 INFO:journalctl@ceph.mon.c.smithi115.stdout: self._write_client_files(client_files, host) 2022-05-14T00:49:53.869 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T00:49:53.869 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T00:49:53.869 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T00:49:53.869 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T00:49:53.870 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T00:49:53.870 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.event_loop.get_result(coro) 2022-05-14T00:49:53.870 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T00:49:53.870 INFO:journalctl@ceph.mon.c.smithi115.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T00:49:53.870 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T00:49:53.871 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.__get_result() 2022-05-14T00:49:53.871 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T00:49:53.871 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise self._exception 2022-05-14T00:49:53.871 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T00:49:53.871 INFO:journalctl@ceph.mon.c.smithi115.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T00:49:53.872 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T00:49:53.872 INFO:journalctl@ceph.mon.c.smithi115.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T00:49:53.872 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T00:49:53.872 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise 2022-05-14T00:49:53.873 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T00:49:53.873 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.gen.throw(type, value, traceback) 2022-05-14T00:49:53.873 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T00:49:53.874 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OrchestratorError(msg) 2022-05-14T00:49:53.874 INFO:journalctl@ceph.mon.c.smithi115.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:49:53.874 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:49:53 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:49:53.874 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:49:53 smithi115 ceph-mon[78593]: pgmap v2816: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:49:54.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:49:54 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:49:54.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:49:54 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:49:54.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:49:54 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:49:54.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:49:54 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 4925 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:49:54.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:49:54 smithi115 conmon[90393]: 2022-05-14T00:49:54.477+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:49:54.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:49:54 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:49:54.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:49:54 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:49:54.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:49:54 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:49:54.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:49:54 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 4925 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:49:55.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:49:55 smithi115 conmon[90393]: 2022-05-14T00:49:55.472+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:49:55.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:49:55 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:49:55.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:49:55 smithi115 ceph-mon[78593]: pgmap v2817: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:49:56.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:49:55 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:49:56.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:49:55 smithi047 ceph-mon[77147]: pgmap v2817: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:49:56.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:49:56 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:49:56.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:49:56 smithi115 conmon[90393]: 2022-05-14T00:49:56.487+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:49:57.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:49:56 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:49:57.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:49:57 smithi115 conmon[90393]: 2022-05-14T00:49:57.446+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:49:57.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:49:57 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:49:57.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:49:57 smithi115 ceph-mon[78593]: pgmap v2818: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:49:58.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:49:57 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:49:58.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:49:57 smithi047 ceph-mon[77147]: pgmap v2818: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:49:58.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:49:58 smithi115 conmon[90393]: 2022-05-14T00:49:58.478+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:49:58.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:49:58 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:49:58.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:49:58 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:49:58.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:49:58 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:49:59.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:49:58 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:49:59.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:49:58 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:49:59.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:49:58 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:49:59.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:49:59 smithi115 conmon[90393]: 2022-05-14T00:49:59.513+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:49:59.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:49:59 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:49:59.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:49:59 smithi115 ceph-mon[78593]: pgmap v2819: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:49:59.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:49:59 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 4930 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:50:00.000 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:49:59 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:50:00.000 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:49:59 smithi047 ceph-mon[77147]: pgmap v2819: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:50:00.000 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:49:59 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 4930 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:50:00.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:50:00 smithi047 conmon[77123]: 2022-05-14T00:49:59.999+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : Health detail: HEALTH_ERR 1 filesystem is degraded; 1 mds daemon damaged; 1 MDSs are read only; 1 MDSs report slow metadata IOs; 1/3 mons down, quorum b,c; Reduced data availability: 1 pg inactive; 9 slow ops, oldest one blocked for 4930 sec, osd.5 has slow ops 2022-05-14T00:50:00.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:50:00 smithi047 conmon[77123]: 2022-05-14T00:49:59.999+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : [WRN] FS_DEGRADED: 1 filesystem is degraded 2022-05-14T00:50:00.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:50:00 smithi047 conmon[77123]: 2022-05-14T00:49:59.999+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : fs cephfs is degraded 2022-05-14T00:50:00.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:50:00 smithi047 conmon[77123]: 2022-05-14T00:49:59.999+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : [ERR] MDS_DAMAGE: 1 mds daemon damaged 2022-05-14T00:50:00.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:50:00 smithi047 conmon[77123]: 2022-05-14T00:49:59.999+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : fs cephfs mds.2 is damaged 2022-05-14T00:50:00.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:50:00 smithi047 conmon[77123]: 2022-05-14T00:49:59.999+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : [WRN] MDS_READ_ONLY: 1 MDSs are read only 2022-05-14T00:50:00.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:50:00 smithi047 conmon[77123]: 2022-05-14T00:49:59.999+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : mds.l(mds.0): MDS in read-only mode 2022-05-14T00:50:00.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:50:00 smithi047 conmon[77123]: 2022-05-14T00:49:59.999+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : [WRN] MDS_SLOW_METADATA_IO: 1 MDSs report slow metadata IOs 2022-05-14T00:50:00.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:50:00 smithi047 conmon[77123]: 2022-05-14T00:49:59.999+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : mds.l(mds.0): 4 slow metadata IOs are blocked > 180 secs, oldest blocked for 5233 secs 2022-05-14T00:50:00.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:50:00 smithi047 conmon[77123]: 2022-05-14T00:49:59.999+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : [WRN] MON_DOWN: 1/3 mons down, quorum b,c 2022-05-14T00:50:00.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:50:00 smithi047 conmon[77123]: 2022-05-14T00:49:59.999+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : mon.a (rank 0) addr [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] is down (out of quorum) 2022-05-14T00:50:00.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:50:00 smithi047 conmon[77123]: 2022-05-14T00:49:59.999+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : [WRN] PG_AVAILABILITY: Reduced data availability: 1 pg inactive 2022-05-14T00:50:00.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:50:00 smithi047 conmon[77123]: 2022-05-14T00:49:59.999+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : pg 2.1 is stuck inactive for 94m, current state unknown, last acting [] 2022-05-14T00:50:00.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:50:00 smithi047 conmon[77123]: 2022-05-14T00:49:59.999+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : [WRN] SLOW_OPS: 9 slow ops, oldest one blocked for 4930 sec, osd.5 has slow ops 2022-05-14T00:50:00.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:50:00 smithi115 conmon[90393]: 2022-05-14T00:50:00.530+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:50:00.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:50:00 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:50:00.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:50:00 smithi115 ceph-mon[78593]: Health detail: HEALTH_ERR 1 filesystem is degraded; 1 mds daemon damaged; 1 MDSs are read only; 1 MDSs report slow metadata IOs; 1/3 mons down, quorum b,c; Reduced data availability: 1 pg inactive; 9 slow ops, oldest one blocked for 4930 sec, osd.5 has slow ops 2022-05-14T00:50:00.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:50:00 smithi115 ceph-mon[78593]: [WRN] FS_DEGRADED: 1 filesystem is degraded 2022-05-14T00:50:00.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:50:00 smithi115 ceph-mon[78593]: fs cephfs is degraded 2022-05-14T00:50:00.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:50:00 smithi115 ceph-mon[78593]: [ERR] MDS_DAMAGE: 1 mds daemon damaged 2022-05-14T00:50:00.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:50:00 smithi115 ceph-mon[78593]: fs cephfs mds.2 is damaged 2022-05-14T00:50:00.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:50:00 smithi115 ceph-mon[78593]: [WRN] MDS_READ_ONLY: 1 MDSs are read only 2022-05-14T00:50:00.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:50:00 smithi115 ceph-mon[78593]: mds.l(mds.0): MDS in read-only mode 2022-05-14T00:50:00.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:50:00 smithi115 ceph-mon[78593]: [WRN] MDS_SLOW_METADATA_IO: 1 MDSs report slow metadata IOs 2022-05-14T00:50:00.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:50:00 smithi115 ceph-mon[78593]: mds.l(mds.0): 4 slow metadata IOs are blocked > 180 secs, oldest blocked for 5233 secs 2022-05-14T00:50:00.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:50:00 smithi115 ceph-mon[78593]: [WRN] MON_DOWN: 1/3 mons down, quorum b,c 2022-05-14T00:50:00.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:50:00 smithi115 ceph-mon[78593]: mon.a (rank 0) addr [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] is down (out of quorum) 2022-05-14T00:50:00.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:50:00 smithi115 ceph-mon[78593]: [WRN] PG_AVAILABILITY: Reduced data availability: 1 pg inactive 2022-05-14T00:50:00.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:50:00 smithi115 ceph-mon[78593]: pg 2.1 is stuck inactive for 94m, current state unknown, last acting [] 2022-05-14T00:50:00.862 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:50:00 smithi115 ceph-mon[78593]: [WRN] SLOW_OPS: 9 slow ops, oldest one blocked for 4930 sec, osd.5 has slow ops 2022-05-14T00:50:01.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:50:00 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:50:01.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:50:00 smithi047 ceph-mon[77147]: Health detail: HEALTH_ERR 1 filesystem is degraded; 1 mds daemon damaged; 1 MDSs are read only; 1 MDSs report slow metadata IOs; 1/3 mons down, quorum b,c; Reduced data availability: 1 pg inactive; 9 slow ops, oldest one blocked for 4930 sec, osd.5 has slow ops 2022-05-14T00:50:01.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:50:00 smithi047 ceph-mon[77147]: [WRN] FS_DEGRADED: 1 filesystem is degraded 2022-05-14T00:50:01.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:50:00 smithi047 ceph-mon[77147]: fs cephfs is degraded 2022-05-14T00:50:01.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:50:00 smithi047 ceph-mon[77147]: [ERR] MDS_DAMAGE: 1 mds daemon damaged 2022-05-14T00:50:01.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:50:00 smithi047 ceph-mon[77147]: fs cephfs mds.2 is damaged 2022-05-14T00:50:01.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:50:00 smithi047 ceph-mon[77147]: [WRN] MDS_READ_ONLY: 1 MDSs are read only 2022-05-14T00:50:01.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:50:00 smithi047 ceph-mon[77147]: mds.l(mds.0): MDS in read-only mode 2022-05-14T00:50:01.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:50:00 smithi047 ceph-mon[77147]: [WRN] MDS_SLOW_METADATA_IO: 1 MDSs report slow metadata IOs 2022-05-14T00:50:01.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:50:00 smithi047 ceph-mon[77147]: mds.l(mds.0): 4 slow metadata IOs are blocked > 180 secs, oldest blocked for 5233 secs 2022-05-14T00:50:01.045 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:50:00 smithi047 ceph-mon[77147]: [WRN] MON_DOWN: 1/3 mons down, quorum b,c 2022-05-14T00:50:01.045 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:50:00 smithi047 ceph-mon[77147]: mon.a (rank 0) addr [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] is down (out of quorum) 2022-05-14T00:50:01.045 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:50:00 smithi047 ceph-mon[77147]: [WRN] PG_AVAILABILITY: Reduced data availability: 1 pg inactive 2022-05-14T00:50:01.046 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:50:00 smithi047 ceph-mon[77147]: pg 2.1 is stuck inactive for 94m, current state unknown, last acting [] 2022-05-14T00:50:01.046 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:50:00 smithi047 ceph-mon[77147]: [WRN] SLOW_OPS: 9 slow ops, oldest one blocked for 4930 sec, osd.5 has slow ops 2022-05-14T00:50:01.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:50:01 smithi115 conmon[90393]: 2022-05-14T00:50:01.558+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:50:01.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:50:01 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:50:01.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:50:01 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:50:01.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:50:01 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:50:01.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:50:01 smithi115 ceph-mon[78593]: pgmap v2820: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:50:02.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:50:01 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:50:02.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:50:01 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:50:02.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:50:01 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:50:02.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:50:01 smithi047 ceph-mon[77147]: pgmap v2820: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:50:02.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:50:02 smithi115 conmon[90393]: 2022-05-14T00:50:02.560+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:50:02.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:50:02 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:50:03.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:50:02 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:50:03.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:50:03 smithi115 conmon[90393]: 2022-05-14T00:50:03.586+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:50:03.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:50:03 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:50:03.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:50:03 smithi115 ceph-mon[78593]: pgmap v2821: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:50:04.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:50:03 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:50:04.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:50:03 smithi047 ceph-mon[77147]: pgmap v2821: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:50:04.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:50:04 smithi115 conmon[90393]: 2022-05-14T00:50:04.622+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:50:04.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:50:04 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:50:04.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:50:04 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 4935 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:50:05.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:50:04 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:50:05.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:50:04 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 4935 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:50:05.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:50:05 smithi115 conmon[90393]: 2022-05-14T00:50:05.584+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:50:05.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:50:05 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:50:05.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:50:05 smithi115 ceph-mon[78593]: pgmap v2822: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:50:06.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:50:05 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:50:06.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:50:05 smithi047 ceph-mon[77147]: pgmap v2822: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:50:06.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:50:06 smithi115 conmon[90393]: 2022-05-14T00:50:06.615+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:50:06.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:50:06 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:50:07.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:50:06 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:50:07.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:50:07 smithi115 conmon[90393]: 2022-05-14T00:50:07.654+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:50:07.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:50:07 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:50:07.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:50:07 smithi115 ceph-mon[78593]: pgmap v2823: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:50:08.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:50:07 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:50:08.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:50:07 smithi047 ceph-mon[77147]: pgmap v2823: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:50:08.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:50:08 smithi115 conmon[90393]: 2022-05-14T00:50:08.673+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:50:08.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:50:08 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:50:09.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:50:08 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:50:09.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:50:09 smithi115 conmon[90393]: 2022-05-14T00:50:09.655+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:50:09.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:50:09 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:50:09.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:50:09 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:50:09.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:50:09 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:50:09.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:50:09 smithi115 ceph-mon[78593]: pgmap v2824: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:50:09.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:50:09 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 4940 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:50:10.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:50:09 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:50:10.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:50:09 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:50:10.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:50:09 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:50:10.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:50:09 smithi047 ceph-mon[77147]: pgmap v2824: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:50:10.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:50:09 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 4940 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:50:10.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:50:10 smithi115 conmon[90393]: 2022-05-14T00:50:10.677+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:50:10.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:50:10 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:50:11.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:50:10 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:50:12.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:50:11 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:50:12.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:50:11 smithi047 ceph-mon[77147]: pgmap v2825: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:50:12.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:50:11 smithi115 conmon[90393]: 2022-05-14T00:50:11.649+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:50:12.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:50:11 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:50:12.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:50:11 smithi115 ceph-mon[78593]: pgmap v2825: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:50:13.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:50:12 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:50:13.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:50:12 smithi115 conmon[90393]: 2022-05-14T00:50:12.631+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:50:13.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:50:12 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:50:14.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:50:13 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:50:14.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:50:13 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:50:14.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:50:13 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:50:14.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:50:13 smithi047 ceph-mon[77147]: pgmap v2826: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:50:14.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:50:13 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T00:50:14.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:50:13 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T00:50:14.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:50:13 smithi115 conmon[90393]: 2022-05-14T00:50:13.616+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:50:14.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:50:13 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:50:14.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:50:13 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:50:14.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:50:13 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:50:14.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:50:13 smithi115 ceph-mon[78593]: pgmap v2826: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:50:14.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:50:13 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T00:50:14.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:50:13 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T00:50:15.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:50:14 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:50:15.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:50:14 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 4945 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:50:15.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:50:14 smithi115 conmon[90393]: 2022-05-14T00:50:14.654+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:50:15.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:50:14 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:50:15.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:50:14 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 4945 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:50:16.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:50:15 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:50:16.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:50:15 smithi047 ceph-mon[77147]: pgmap v2827: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:50:16.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:50:15 smithi115 conmon[90393]: 2022-05-14T00:50:15.687+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:50:16.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:50:15 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:50:16.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:50:15 smithi115 ceph-mon[78593]: pgmap v2827: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:50:17.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:50:16 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:50:17.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:50:16 smithi115 conmon[90393]: 2022-05-14T00:50:16.643+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:50:17.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:50:16 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:50:18.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:50:17 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:50:18.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:50:17 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:50:18.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:50:17 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:50:18.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:50:17 smithi047 ceph-mon[77147]: pgmap v2828: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:50:18.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:50:17 smithi115 conmon[90393]: 2022-05-14T00:50:17.692+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:50:18.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:50:17 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:50:18.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:50:17 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:50:18.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:50:17 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:50:18.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:50:17 smithi115 ceph-mon[78593]: pgmap v2828: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:50:19.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:50:18 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:50:19.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:50:18 smithi115 conmon[90393]: 2022-05-14T00:50:18.665+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:50:19.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:50:18 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:50:20.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:50:19 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:50:20.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:50:19 smithi047 ceph-mon[77147]: pgmap v2829: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:50:20.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:50:19 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 4950 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:50:20.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:50:19 smithi115 conmon[90393]: 2022-05-14T00:50:19.636+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:50:20.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:50:19 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:50:20.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:50:19 smithi115 ceph-mon[78593]: pgmap v2829: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:50:20.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:50:19 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 4950 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:50:21.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:50:20 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:50:21.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:50:20 smithi115 conmon[90393]: 2022-05-14T00:50:20.660+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:50:21.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:50:20 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:50:22.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:50:21 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:50:22.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:50:21 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:50:22.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:50:21 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:50:22.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:50:21 smithi047 ceph-mon[77147]: pgmap v2830: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:50:22.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:50:21 smithi115 conmon[90393]: 2022-05-14T00:50:21.649+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:50:22.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:50:21 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:50:22.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:50:21 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:50:22.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:50:21 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:50:22.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:50:21 smithi115 ceph-mon[78593]: pgmap v2830: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:50:23.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:50:22 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:50:23.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:50:22 smithi115 conmon[90393]: 2022-05-14T00:50:22.627+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:50:23.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:50:22 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:50:23.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:50:23 smithi115 conmon[90393]: 2022-05-14T00:50:23.591+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:50:23.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:50:23 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:50:23.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:50:23 smithi115 ceph-mon[78593]: pgmap v2831: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:50:24.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:50:23 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:50:24.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:50:23 smithi047 ceph-mon[77147]: pgmap v2831: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:50:24.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:50:24 smithi115 conmon[90393]: 2022-05-14T00:50:24.600+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:50:24.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:50:24 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:50:24.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:50:24 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 4955 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:50:25.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:50:24 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:50:25.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:50:24 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 4955 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:50:25.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:50:25 smithi115 conmon[90393]: 2022-05-14T00:50:25.573+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:50:25.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:50:25 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:50:25.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:50:25 smithi115 ceph-mon[78593]: pgmap v2832: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:50:26.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:50:25 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:50:26.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:50:25 smithi047 ceph-mon[77147]: pgmap v2832: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:50:26.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:50:26 smithi115 conmon[90393]: 2022-05-14T00:50:26.566+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:50:26.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:50:26 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:50:27.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:50:26 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:50:27.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:50:27 smithi115 conmon[90393]: 2022-05-14T00:50:27.572+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:50:27.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:50:27 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:50:27.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:50:27 smithi115 ceph-mon[78593]: pgmap v2833: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:50:28.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:50:27 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:50:28.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:50:27 smithi047 ceph-mon[77147]: pgmap v2833: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:50:28.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:50:28 smithi115 conmon[90393]: 2022-05-14T00:50:28.566+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:50:28.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:50:28 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:50:29.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:50:28 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:50:29.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:50:29 smithi115 conmon[90393]: 2022-05-14T00:50:29.583+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:50:29.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:50:29 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:50:29.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:50:29 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:50:29.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:50:29 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:50:29.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:50:29 smithi115 ceph-mon[78593]: pgmap v2834: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:50:29.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:50:29 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 4960 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:50:30.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:50:29 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:50:30.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:50:29 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:50:30.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:50:29 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:50:30.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:50:29 smithi047 ceph-mon[77147]: pgmap v2834: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:50:30.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:50:29 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 4960 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:50:31.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:50:30 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:50:31.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:50:30 smithi115 conmon[90393]: 2022-05-14T00:50:30.627+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:50:31.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:50:30 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:50:32.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:50:31 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:50:32.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:50:31 smithi047 ceph-mon[77147]: pgmap v2835: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:50:32.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:50:31 smithi115 conmon[90393]: 2022-05-14T00:50:31.612+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:50:32.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:50:31 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:50:32.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:50:31 smithi115 ceph-mon[78593]: pgmap v2835: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:50:33.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:50:32 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:50:33.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:50:32 smithi115 conmon[90393]: 2022-05-14T00:50:32.655+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:50:33.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:50:32 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:50:34.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:50:33 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:50:34.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:50:33 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:50:34.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:50:33 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:50:34.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:50:33 smithi047 ceph-mon[77147]: pgmap v2836: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:50:34.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:50:33 smithi115 conmon[90393]: 2022-05-14T00:50:33.650+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:50:34.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:50:33 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:50:34.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:50:33 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:50:34.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:50:33 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:50:34.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:50:33 smithi115 ceph-mon[78593]: pgmap v2836: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:50:35.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:50:34 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:50:35.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:50:34 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 4965 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:50:35.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:50:34 smithi115 conmon[90393]: 2022-05-14T00:50:34.675+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:50:35.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:50:34 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:50:35.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:50:34 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 4965 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:50:36.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:50:35 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:50:36.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:50:35 smithi047 ceph-mon[77147]: pgmap v2837: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:50:36.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:50:35 smithi115 conmon[90393]: 2022-05-14T00:50:35.659+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:50:36.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:50:35 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:50:36.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:50:35 smithi115 ceph-mon[78593]: pgmap v2837: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:50:37.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:50:36 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:50:37.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:50:36 smithi115 conmon[90393]: 2022-05-14T00:50:36.643+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:50:37.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:50:36 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:50:38.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:50:37 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:50:38.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:50:37 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:50:38.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:50:37 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:50:38.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:50:37 smithi047 ceph-mon[77147]: pgmap v2838: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:50:38.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:50:37 smithi115 conmon[90393]: 2022-05-14T00:50:37.647+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:50:38.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:50:37 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:50:38.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:50:37 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:50:38.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:50:37 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:50:38.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:50:37 smithi115 ceph-mon[78593]: pgmap v2838: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:50:39.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:50:38 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:50:39.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:50:38 smithi115 conmon[90393]: 2022-05-14T00:50:38.660+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:50:39.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:50:38 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:50:40.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:50:39 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:50:40.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:50:39 smithi047 ceph-mon[77147]: pgmap v2839: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:50:40.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:50:39 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 4970 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:50:40.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:50:39 smithi115 conmon[90393]: 2022-05-14T00:50:39.647+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:50:40.199 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:50:39 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:50:40.199 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:50:39 smithi115 ceph-mon[78593]: pgmap v2839: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:50:40.199 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:50:39 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 4970 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:50:41.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:50:40 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:50:41.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:50:40 smithi115 conmon[90393]: 2022-05-14T00:50:40.670+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:50:41.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:50:40 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:50:42.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:50:41 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:50:42.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:50:41 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:50:42.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:50:41 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:50:42.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:50:41 smithi047 ceph-mon[77147]: pgmap v2840: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:50:42.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:50:41 smithi115 conmon[90393]: 2022-05-14T00:50:41.698+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:50:42.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:50:41 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:50:42.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:50:41 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:50:42.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:50:41 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:50:42.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:50:41 smithi115 ceph-mon[78593]: pgmap v2840: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:50:43.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:50:42 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:50:43.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:50:42 smithi115 conmon[90393]: 2022-05-14T00:50:42.716+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:50:43.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:50:42 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:50:44.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:50:43 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:50:44.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:50:43 smithi047 ceph-mon[77147]: pgmap v2841: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:50:44.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:50:43 smithi115 conmon[90393]: 2022-05-14T00:50:43.698+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:50:44.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:50:43 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:50:44.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:50:43 smithi115 ceph-mon[78593]: pgmap v2841: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:50:45.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:50:44 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:50:45.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:50:44 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 4975 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:50:45.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:50:44 smithi115 conmon[90393]: 2022-05-14T00:50:44.720+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:50:45.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:50:44 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:50:45.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:50:44 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 4975 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:50:46.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:50:45 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:50:46.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:50:45 smithi047 ceph-mon[77147]: pgmap v2842: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:50:46.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:50:45 smithi115 conmon[90393]: 2022-05-14T00:50:45.723+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:50:46.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:50:45 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:50:46.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:50:45 smithi115 ceph-mon[78593]: pgmap v2842: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:50:47.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:50:46 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:50:47.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:50:46 smithi115 conmon[90393]: 2022-05-14T00:50:46.698+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:50:47.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:50:46 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:50:48.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:50:47 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:50:48.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:50:47 smithi047 ceph-mon[77147]: pgmap v2843: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:50:48.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:50:47 smithi115 conmon[90393]: 2022-05-14T00:50:47.681+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:50:48.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:50:47 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:50:48.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:50:47 smithi115 ceph-mon[78593]: pgmap v2843: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:50:49.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:50:48 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:50:49.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:50:48 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:50:49.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:50:48 smithi115 conmon[90393]: 2022-05-14T00:50:48.729+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:50:50.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:50:49 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:50:50.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:50:49 smithi047 ceph-mon[77147]: pgmap v2844: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:50:50.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:50:49 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:50:50.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:50:49 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:50:50.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:50:49 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 4980 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:50:50.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:50:49 smithi115 conmon[90393]: 2022-05-14T00:50:49.755+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:50:50.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:50:49 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:50:50.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:50:49 smithi115 ceph-mon[78593]: pgmap v2844: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:50:50.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:50:49 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:50:50.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:50:49 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:50:50.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:50:49 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 4980 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:50:51.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:50:50 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:50:51.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:50:50 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:50:51.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:50:50 smithi115 conmon[90393]: 2022-05-14T00:50:50.733+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:50:52.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:50:51 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:50:52.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:50:51 smithi047 ceph-mon[77147]: pgmap v2845: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:50:52.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:50:51 smithi115 conmon[90393]: 2022-05-14T00:50:51.704+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:50:52.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:50:51 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:50:52.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:50:51 smithi115 ceph-mon[78593]: pgmap v2845: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:50:53.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:50:52 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:50:53.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:50:52 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T00:50:53.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:50:52 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T00:50:53.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:50:52 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T00:50:53.042 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:50:52 smithi047 conmon[78521]: 2022-05-14T00:50:52.976+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:50:53.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:50:52 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T00:50:53.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:50:52 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:50:53.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:50:52 smithi047 conmon[78521]: yield 2022-05-14T00:50:53.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:50:52 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:50:53.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:50:52 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:50:53.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:50:52 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:50:53.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:50:52 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T00:50:53.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:50:52 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:50:53.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:50:52 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T00:50:53.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:50:52 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:50:53.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:50:52 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T00:50:53.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:50:52 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:50:53.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:50:52 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T00:50:53.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:50:52 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:50:53.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:50:52 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T00:50:53.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:50:52 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:50:53.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:50:52 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:50:53.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:50:52 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:50:53.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:50:52 smithi115 conmon[90393]: 2022-05-14T00:50:52.753+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:50:53.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:50:52 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:50:53.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:50:52 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T00:50:53.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:50:52 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T00:50:53.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:50:52 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T00:50:54.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:50:53 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:50:54.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:50:53 smithi047 ceph-mon[77147]: pgmap v2846: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:50:54.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:50:53 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:50:54.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:50:53 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:50:54.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:50:53 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:50:54.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T00:50:54.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:50:54.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T00:50:54.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:50:54.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:50:54.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:50:54.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T00:50:54.049 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:50:54.050 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T00:50:54.050 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:50:54.050 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T00:50:54.051 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:50:54.051 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T00:50:54.051 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:50:54.051 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T00:50:54.052 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:50:54.052 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:50:54.052 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:50:54.052 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:50:53 smithi047 ceph-mon[77147]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T00:50:54.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:50:53 smithi115 conmon[90393]: 2022-05-14T00:50:53.753+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:50:54.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:50:53 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:50:54.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:50:53 smithi115 ceph-mon[78593]: pgmap v2846: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:50:54.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:50:53 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:50:54.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:50:53 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:50:54.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:50:53 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:50:54.109 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T00:50:54.109 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:50:54.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T00:50:54.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:50:54.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:50:54.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:50:54.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T00:50:54.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:50:54.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T00:50:54.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:50:54.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T00:50:54.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:50:54.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T00:50:54.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:50:54.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T00:50:54.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:50:54.113 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:50:54.113 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:50:54.113 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:50:53 smithi115 ceph-mon[78593]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T00:50:55.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:50:54 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:50:55.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:50:54 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 4985 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:50:55.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:50:54 smithi115 conmon[90393]: 2022-05-14T00:50:54.787+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:50:55.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:50:54 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:50:55.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:50:54 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 4985 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:50:56.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:50:55 smithi047 ceph-mon[77147]: pgmap v2847: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:50:56.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:50:55 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:50:56.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:50:55 smithi115 conmon[90393]: 2022-05-14T00:50:55.764+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:50:56.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:50:55 smithi115 ceph-mon[78593]: pgmap v2847: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:50:56.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:50:55 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:50:56.541 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:50:56 smithi047 conmon[78521]: 2022-05-14T00:50:56.048+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:50:56.542 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:50:56 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T00:50:56.542 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:50:56 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:50:56.542 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:50:56 smithi047 conmon[78521]: yield 2022-05-14T00:50:56.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:50:56 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:50:56.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:50:56 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:50:56.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:50:56 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:50:56.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:50:56 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T00:50:56.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:50:56 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:50:56.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:50:56 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T00:50:56.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:50:56 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:50:56.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:50:56 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T00:50:56.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:50:56 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:50:56.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:50:56 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T00:50:56.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:50:56 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:50:56.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:50:56 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T00:50:56.547 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:50:56 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:50:56.547 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:50:56 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:50:56.548 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:50:56 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:50:56.548 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:50:56 smithi047 conmon[78521]: 2022-05-14T00:50:56.049+0000 7f5344781700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T00:50:56.549 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:50:56 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T00:50:56.549 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:50:56 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:50:56.549 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:50:56 smithi047 conmon[78521]: yield 2022-05-14T00:50:56.550 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:50:56 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:50:56.550 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:50:56 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:50:56.550 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:50:56 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:50:56.551 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:50:56 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T00:50:56.551 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:50:56 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:50:56.551 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:50:56 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T00:50:56.552 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:50:56 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:50:56.552 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:50:56 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T00:50:56.552 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:50:56 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:50:56.552 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:50:56 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T00:50:56.553 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:50:56 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:50:56.553 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:50:56 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T00:50:56.553 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:50:56 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:50:56.554 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:50:56 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:50:56.554 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:50:56 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:50:56.554 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:50:56 smithi047 conmon[78521]: 2022-05-14T00:50:56.554 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:50:56 smithi047 conmon[78521]: During handling of the above exception, another exception occurred: 2022-05-14T00:50:56.555 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:50:56 smithi047 conmon[78521]: 2022-05-14T00:50:56.555 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:50:56 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T00:50:56.556 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:50:56 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T00:50:56.556 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:50:56 smithi047 conmon[78521]: return f(*arg) 2022-05-14T00:50:56.557 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:50:56 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T00:50:56.557 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:50:56 smithi047 conmon[78521]: self._write_client_files(client_files, host) 2022-05-14T00:50:56.557 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:50:56 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T00:50:56.558 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:50:56 smithi047 conmon[78521]: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T00:50:56.558 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:50:56 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T00:50:56.558 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:50:56 smithi047 conmon[78521]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T00:50:56.558 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:50:56 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T00:50:56.559 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:50:56 smithi047 conmon[78521]: return self.event_loop.get_result(coro) 2022-05-14T00:50:56.559 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:50:56 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T00:50:56.559 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:50:56 smithi047 conmon[78521]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T00:50:56.559 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:50:56 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T00:50:56.560 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:50:56 smithi047 conmon[78521]: return self.__get_result() 2022-05-14T00:50:56.560 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:50:56 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T00:50:56.560 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:50:56 smithi047 conmon[78521]: raise self._exception 2022-05-14T00:50:56.560 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:50:56 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T00:50:56.561 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:50:56 smithi047 conmon[78521]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T00:50:56.561 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:50:56 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T00:50:56.561 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:50:56 smithi047 conmon[78521]: conn = await self._remote_connection(host, addr) 2022-05-14T00:50:56.562 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:50:56 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T00:50:56.563 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:50:56 smithi047 conmon[78521]: raise 2022-05-14T00:50:56.563 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:50:56 smithi047 conmon[78521]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T00:50:56.564 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:50:56 smithi047 conmon[78521]: self.gen.throw(type, value, traceback) 2022-05-14T00:50:56.564 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:50:56 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T00:50:56.565 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:50:56 smithi047 conmon[78521]: raise OrchestratorError(msg) 2022-05-14T00:50:56.565 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:50:56 smithi047 conmon[78521]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:50:57.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:50:56 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:50:57.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:50:56 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:50:57.041 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T00:50:57.042 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:50:57.042 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T00:50:57.042 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:50:57.042 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:50:57.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:50:57.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T00:50:57.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:50:57.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T00:50:57.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:50:57.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T00:50:57.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:50:57.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T00:50:57.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:50:57.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T00:50:57.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:50:57.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:50:57.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:50:57.046 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:50:56 smithi047 ceph-mon[77147]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T00:50:57.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T00:50:57.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:50:57.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T00:50:57.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:50:57.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:50:57.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:50:57.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T00:50:57.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:50:57.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T00:50:57.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:50:57.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T00:50:57.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:50:57.049 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T00:50:57.049 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:50:57.050 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T00:50:57.050 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:50:57.050 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:50:57.051 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:50:57.051 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T00:50:57.051 INFO:journalctl@ceph.mon.b.smithi047.stdout: During handling of the above exception, another exception occurred: 2022-05-14T00:50:57.052 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T00:50:57.052 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T00:50:57.052 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T00:50:57.052 INFO:journalctl@ceph.mon.b.smithi047.stdout: return f(*arg) 2022-05-14T00:50:57.053 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T00:50:57.053 INFO:journalctl@ceph.mon.b.smithi047.stdout: self._write_client_files(client_files, host) 2022-05-14T00:50:57.053 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T00:50:57.053 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T00:50:57.054 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T00:50:57.054 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T00:50:57.054 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T00:50:57.054 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.event_loop.get_result(coro) 2022-05-14T00:50:57.055 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T00:50:57.055 INFO:journalctl@ceph.mon.b.smithi047.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T00:50:57.055 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T00:50:57.055 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.__get_result() 2022-05-14T00:50:57.055 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T00:50:57.056 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise self._exception 2022-05-14T00:50:57.056 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T00:50:57.056 INFO:journalctl@ceph.mon.b.smithi047.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T00:50:57.056 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T00:50:57.057 INFO:journalctl@ceph.mon.b.smithi047.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T00:50:57.057 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T00:50:57.057 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise 2022-05-14T00:50:57.057 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T00:50:57.058 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.gen.throw(type, value, traceback) 2022-05-14T00:50:57.058 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T00:50:57.058 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OrchestratorError(msg) 2022-05-14T00:50:57.058 INFO:journalctl@ceph.mon.b.smithi047.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:50:57.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:50:56 smithi115 conmon[90393]: 2022-05-14T00:50:56.783+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:50:57.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:50:56 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:50:57.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:50:56 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:50:57.109 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T00:50:57.109 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:50:57.109 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T00:50:57.109 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:50:57.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:50:57.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:50:57.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T00:50:57.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:50:57.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T00:50:57.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:50:57.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T00:50:57.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:50:57.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T00:50:57.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:50:57.114 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T00:50:57.114 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:50:57.114 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:50:57.114 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:50:57.114 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:50:56 smithi115 ceph-mon[78593]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T00:50:57.115 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T00:50:57.115 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:50:57.115 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T00:50:57.115 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:50:57.116 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:50:57.116 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:50:57.116 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T00:50:57.116 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:50:57.116 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T00:50:57.117 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:50:57.117 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T00:50:57.117 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:50:57.117 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T00:50:57.118 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:50:57.118 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T00:50:57.118 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:50:57.118 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:50:57.118 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:50:57.119 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T00:50:57.119 INFO:journalctl@ceph.mon.c.smithi115.stdout: During handling of the above exception, another exception occurred: 2022-05-14T00:50:57.119 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T00:50:57.119 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T00:50:57.120 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T00:50:57.120 INFO:journalctl@ceph.mon.c.smithi115.stdout: return f(*arg) 2022-05-14T00:50:57.120 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T00:50:57.120 INFO:journalctl@ceph.mon.c.smithi115.stdout: self._write_client_files(client_files, host) 2022-05-14T00:50:57.120 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T00:50:57.121 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T00:50:57.121 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T00:50:57.121 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T00:50:57.121 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T00:50:57.121 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.event_loop.get_result(coro) 2022-05-14T00:50:57.122 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T00:50:57.122 INFO:journalctl@ceph.mon.c.smithi115.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T00:50:57.123 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T00:50:57.123 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.__get_result() 2022-05-14T00:50:57.123 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T00:50:57.124 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise self._exception 2022-05-14T00:50:57.124 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T00:50:57.124 INFO:journalctl@ceph.mon.c.smithi115.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T00:50:57.125 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T00:50:57.125 INFO:journalctl@ceph.mon.c.smithi115.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T00:50:57.126 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T00:50:57.126 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise 2022-05-14T00:50:57.126 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T00:50:57.126 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.gen.throw(type, value, traceback) 2022-05-14T00:50:57.127 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T00:50:57.127 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OrchestratorError(msg) 2022-05-14T00:50:57.127 INFO:journalctl@ceph.mon.c.smithi115.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:50:58.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:50:57 smithi047 ceph-mon[77147]: pgmap v2848: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:50:58.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:50:57 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:50:58.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:50:57 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:50:58.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:50:57 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:50:58.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:50:57 smithi115 conmon[90393]: 2022-05-14T00:50:57.790+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:50:58.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:50:57 smithi115 ceph-mon[78593]: pgmap v2848: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:50:58.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:50:57 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:50:58.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:50:57 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:50:58.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:50:57 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:50:59.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:50:58 smithi115 conmon[90393]: 2022-05-14T00:50:58.740+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:50:59.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:50:58 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:50:59.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:50:58 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:51:00.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:50:59 smithi115 conmon[90393]: 2022-05-14T00:50:59.776+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:51:00.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:50:59 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:51:00.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:50:59 smithi115 ceph-mon[78593]: pgmap v2849: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:51:00.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:50:59 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 4990 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:51:00.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:50:59 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:51:00.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:50:59 smithi047 ceph-mon[77147]: pgmap v2849: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:51:00.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:50:59 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 4990 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:51:01.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:51:00 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:51:01.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:51:00 smithi115 conmon[90393]: 2022-05-14T00:51:00.825+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:51:01.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:51:00 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:51:02.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:51:01 smithi115 conmon[90393]: 2022-05-14T00:51:01.828+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:51:02.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:51:01 smithi115 ceph-mon[78593]: pgmap v2850: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:51:02.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:51:01 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:51:02.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:51:01 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:51:02.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:51:01 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:51:02.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:51:01 smithi047 ceph-mon[77147]: pgmap v2850: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:51:02.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:51:01 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:51:02.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:51:01 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:51:02.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:51:01 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:51:03.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:51:02 smithi115 conmon[90393]: 2022-05-14T00:51:02.816+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:51:03.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:51:02 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:51:03.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:51:02 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:51:04.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:51:03 smithi115 conmon[90393]: 2022-05-14T00:51:03.811+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:51:04.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:51:03 smithi115 ceph-mon[78593]: pgmap v2851: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:51:04.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:51:03 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:51:04.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:51:03 smithi047 ceph-mon[77147]: pgmap v2851: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:51:04.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:51:03 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:51:05.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:51:04 smithi115 conmon[90393]: 2022-05-14T00:51:04.783+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:51:05.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:51:04 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:51:05.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:51:04 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 4995 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:51:05.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:51:04 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:51:05.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:51:04 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 4995 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:51:06.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:51:05 smithi115 conmon[90393]: 2022-05-14T00:51:05.797+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:51:06.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:51:05 smithi115 ceph-mon[78593]: pgmap v2852: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:51:06.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:51:05 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:51:06.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:51:05 smithi047 ceph-mon[77147]: pgmap v2852: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:51:06.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:51:05 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:51:07.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:51:06 smithi115 conmon[90393]: 2022-05-14T00:51:06.771+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:51:07.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:51:06 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:51:07.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:51:06 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:51:08.110 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:51:07 smithi115 conmon[90393]: 2022-05-14T00:51:07.778+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:51:08.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:51:07 smithi115 ceph-mon[78593]: pgmap v2853: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:51:08.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:51:07 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:51:08.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:51:07 smithi047 ceph-mon[77147]: pgmap v2853: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:51:08.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:51:07 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:51:09.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:51:08 smithi115 conmon[90393]: 2022-05-14T00:51:08.734+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:51:09.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:51:08 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:51:09.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:51:08 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:51:10.113 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:51:09 smithi115 conmon[90393]: 2022-05-14T00:51:09.710+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:51:10.113 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:51:09 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:51:10.113 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:51:09 smithi115 ceph-mon[78593]: pgmap v2854: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:51:10.114 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:51:09 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:51:10.114 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:51:09 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:51:10.114 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:51:09 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 5000 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:51:10.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:51:09 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:51:10.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:51:09 smithi047 ceph-mon[77147]: pgmap v2854: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:51:10.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:51:09 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:51:10.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:51:09 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:51:10.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:51:09 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 5000 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:51:11.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:51:10 smithi115 conmon[90393]: 2022-05-14T00:51:10.699+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:51:11.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:51:10 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:51:11.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:51:10 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:51:12.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:51:11 smithi115 conmon[90393]: 2022-05-14T00:51:11.682+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:51:12.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:51:11 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:51:12.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:51:11 smithi115 ceph-mon[78593]: pgmap v2855: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:51:12.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:51:11 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:51:12.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:51:11 smithi047 ceph-mon[77147]: pgmap v2855: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:51:13.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:51:12 smithi115 conmon[90393]: 2022-05-14T00:51:12.673+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:51:13.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:51:12 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:51:13.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:51:12 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:51:14.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:51:13 smithi115 conmon[90393]: 2022-05-14T00:51:13.654+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:51:14.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:51:13 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:51:14.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:51:13 smithi115 ceph-mon[78593]: pgmap v2856: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:51:14.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:51:13 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:51:14.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:51:13 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:51:14.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:51:13 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T00:51:14.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:51:13 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T00:51:14.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:51:13 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:51:14.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:51:13 smithi047 ceph-mon[77147]: pgmap v2856: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:51:14.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:51:13 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:51:14.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:51:13 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:51:14.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:51:13 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T00:51:14.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:51:13 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T00:51:15.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:51:14 smithi115 conmon[90393]: 2022-05-14T00:51:14.640+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:51:15.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:51:14 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:51:15.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:51:14 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 5005 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:51:15.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:51:14 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:51:15.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:51:14 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 5005 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:51:16.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:51:15 smithi115 conmon[90393]: 2022-05-14T00:51:15.625+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:51:16.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:51:15 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:51:16.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:51:15 smithi115 ceph-mon[78593]: pgmap v2857: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:51:16.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:51:15 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:51:16.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:51:15 smithi047 ceph-mon[77147]: pgmap v2857: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:51:17.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:51:16 smithi115 conmon[90393]: 2022-05-14T00:51:16.653+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:51:17.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:51:16 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:51:17.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:51:16 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:51:17.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:51:17 smithi115 conmon[90393]: 2022-05-14T00:51:17.606+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:51:17.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:51:17 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:51:17.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:51:17 smithi115 ceph-mon[78593]: pgmap v2858: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:51:17.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:51:17 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:51:17.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:51:17 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:51:18.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:51:17 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:51:18.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:51:17 smithi047 ceph-mon[77147]: pgmap v2858: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:51:18.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:51:17 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:51:18.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:51:17 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:51:18.852 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:51:18 smithi115 conmon[90393]: 2022-05-14T00:51:18.600+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:51:19.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:51:18 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:51:19.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:51:18 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:51:20.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:51:19 smithi115 conmon[90393]: 2022-05-14T00:51:19.632+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:51:20.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:51:19 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:51:20.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:51:19 smithi115 ceph-mon[78593]: pgmap v2859: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:51:20.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:51:19 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 5010 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:51:20.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:51:19 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:51:20.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:51:19 smithi047 ceph-mon[77147]: pgmap v2859: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:51:20.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:51:19 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 5010 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:51:21.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:51:20 smithi115 conmon[90393]: 2022-05-14T00:51:20.676+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:51:21.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:51:20 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:51:21.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:51:20 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:51:22.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:51:21 smithi115 conmon[90393]: 2022-05-14T00:51:21.662+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:51:22.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:51:21 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:51:22.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:51:21 smithi115 ceph-mon[78593]: pgmap v2860: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:51:22.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:51:21 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:51:22.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:51:21 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:51:22.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:51:21 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:51:22.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:51:21 smithi047 ceph-mon[77147]: pgmap v2860: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:51:22.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:51:21 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:51:22.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:51:21 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:51:23.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:51:22 smithi115 conmon[90393]: 2022-05-14T00:51:22.643+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:51:23.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:51:22 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:51:23.298 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:51:22 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:51:23.866 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:51:23 smithi115 conmon[90393]: 2022-05-14T00:51:23.614+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:51:24.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:51:23 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:51:24.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:51:23 smithi047 ceph-mon[77147]: pgmap v2861: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:51:24.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:51:23 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:51:24.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:51:23 smithi115 ceph-mon[78593]: pgmap v2861: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:51:25.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:51:24 smithi115 conmon[90393]: 2022-05-14T00:51:24.645+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:51:25.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:51:24 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:51:25.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:51:24 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 5015 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:51:25.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:51:24 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:51:25.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:51:24 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 5015 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:51:26.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:51:25 smithi115 conmon[90393]: 2022-05-14T00:51:25.655+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:51:26.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:51:25 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:51:26.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:51:25 smithi115 ceph-mon[78593]: pgmap v2862: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:51:26.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:51:25 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:51:26.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:51:25 smithi047 ceph-mon[77147]: pgmap v2862: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:51:26.876 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:51:26 smithi115 conmon[90393]: 2022-05-14T00:51:26.619+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:51:27.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:51:26 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:51:27.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:51:26 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:51:27.876 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:51:27 smithi115 conmon[90393]: 2022-05-14T00:51:27.621+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:51:28.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:51:27 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:51:28.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:51:27 smithi047 ceph-mon[77147]: pgmap v2863: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:51:28.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:51:27 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:51:28.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:51:27 smithi115 ceph-mon[78593]: pgmap v2863: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:51:29.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:51:28 smithi115 conmon[90393]: 2022-05-14T00:51:28.646+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:51:29.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:51:28 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:51:29.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:51:28 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:51:30.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:51:29 smithi115 conmon[90393]: 2022-05-14T00:51:29.665+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:51:30.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:51:29 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:51:30.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:51:29 smithi115 ceph-mon[78593]: pgmap v2864: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:51:30.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:51:29 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:51:30.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:51:29 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:51:30.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:51:29 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 5020 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:51:30.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:51:29 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:51:30.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:51:29 smithi047 ceph-mon[77147]: pgmap v2864: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:51:30.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:51:29 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:51:30.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:51:29 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:51:30.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:51:29 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 5020 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:51:31.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:51:30 smithi115 conmon[90393]: 2022-05-14T00:51:30.666+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:51:31.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:51:30 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:51:31.295 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:51:30 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:51:31.890 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:51:31 smithi115 conmon[90393]: 2022-05-14T00:51:31.617+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:51:32.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:51:31 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:51:32.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:51:31 smithi047 ceph-mon[77147]: pgmap v2865: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:51:32.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:51:31 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:51:32.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:51:31 smithi115 ceph-mon[78593]: pgmap v2865: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:51:32.895 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:51:32 smithi115 conmon[90393]: 2022-05-14T00:51:32.644+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:51:33.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:51:32 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:51:33.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:51:32 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:51:33.897 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:51:33 smithi115 conmon[90393]: 2022-05-14T00:51:33.637+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:51:34.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:51:33 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:51:34.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:51:33 smithi047 ceph-mon[77147]: pgmap v2866: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:51:34.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:51:33 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:51:34.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:51:33 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:51:34.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:51:33 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:51:34.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:51:33 smithi115 ceph-mon[78593]: pgmap v2866: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:51:34.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:51:33 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:51:34.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:51:33 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:51:34.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:51:34 smithi115 conmon[90393]: 2022-05-14T00:51:34.604+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:51:35.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:51:34 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:51:35.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:51:34 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 5025 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:51:35.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:51:34 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:51:35.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:51:34 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 5025 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:51:35.900 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:51:35 smithi115 conmon[90393]: 2022-05-14T00:51:35.649+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:51:35.901 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:51:35 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:51:35.901 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:51:35 smithi115 ceph-mon[78593]: pgmap v2867: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:51:36.296 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:51:35 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:51:36.297 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:51:35 smithi047 ceph-mon[77147]: pgmap v2867: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:51:37.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:51:36 smithi115 conmon[90393]: 2022-05-14T00:51:36.656+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:51:37.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:51:36 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:51:37.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:51:36 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:51:37.906 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:51:37 smithi115 conmon[90393]: 2022-05-14T00:51:37.643+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:51:38.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:51:37 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:51:38.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:51:37 smithi047 ceph-mon[77147]: pgmap v2868: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:51:38.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:51:37 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:51:38.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:51:37 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:51:38.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:51:37 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:51:38.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:51:37 smithi115 ceph-mon[78593]: pgmap v2868: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:51:38.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:51:37 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:51:38.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:51:37 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:51:38.911 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:51:38 smithi115 conmon[90393]: 2022-05-14T00:51:38.631+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:51:39.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:51:38 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:51:39.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:51:38 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:51:39.913 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:51:39 smithi115 conmon[90393]: 2022-05-14T00:51:39.635+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:51:40.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:51:39 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:51:40.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:51:39 smithi047 ceph-mon[77147]: pgmap v2869: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:51:40.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:51:39 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 5030 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:51:40.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:51:39 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:51:40.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:51:39 smithi115 ceph-mon[78593]: pgmap v2869: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:51:40.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:51:39 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 5030 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:51:40.918 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:51:40 smithi115 conmon[90393]: 2022-05-14T00:51:40.626+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:51:41.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:51:40 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:51:41.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:51:40 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:51:41.919 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:51:41 smithi115 conmon[90393]: 2022-05-14T00:51:41.639+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:51:42.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:51:41 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:51:42.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:51:41 smithi047 ceph-mon[77147]: pgmap v2870: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:51:42.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:51:41 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:51:42.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:51:41 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:51:42.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:51:41 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:51:42.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:51:41 smithi115 ceph-mon[78593]: pgmap v2870: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:51:42.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:51:41 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:51:42.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:51:41 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:51:42.924 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:51:42 smithi115 conmon[90393]: 2022-05-14T00:51:42.645+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:51:43.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:51:42 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:51:43.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:51:42 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:51:43.926 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:51:43 smithi115 conmon[90393]: 2022-05-14T00:51:43.669+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:51:44.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:51:43 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:51:44.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:51:43 smithi047 ceph-mon[77147]: pgmap v2871: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:51:44.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:51:43 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:51:44.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:51:43 smithi115 ceph-mon[78593]: pgmap v2871: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:51:44.927 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:51:44 smithi115 conmon[90393]: 2022-05-14T00:51:44.619+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:51:45.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:51:44 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:51:45.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:51:44 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 5035 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:51:45.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:51:44 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:51:45.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:51:44 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 5035 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:51:45.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:51:45 smithi115 conmon[90393]: 2022-05-14T00:51:45.580+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:51:46.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:51:45 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:51:46.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:51:45 smithi047 ceph-mon[77147]: pgmap v2872: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:51:46.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:51:45 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:51:46.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:51:45 smithi115 ceph-mon[78593]: pgmap v2872: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:51:46.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:51:46 smithi115 conmon[90393]: 2022-05-14T00:51:46.572+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:51:47.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:51:46 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:51:47.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:51:46 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:51:47.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:51:47 smithi115 conmon[90393]: 2022-05-14T00:51:47.584+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:51:48.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:51:47 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:51:48.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:51:47 smithi047 ceph-mon[77147]: pgmap v2873: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:51:48.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:51:47 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:51:48.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:51:47 smithi115 ceph-mon[78593]: pgmap v2873: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:51:48.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:51:48 smithi115 conmon[90393]: 2022-05-14T00:51:48.575+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:51:49.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:51:48 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:51:49.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:51:48 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:51:49.861 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:51:49 smithi115 conmon[90393]: 2022-05-14T00:51:49.567+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:51:50.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:51:49 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:51:50.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:51:49 smithi047 ceph-mon[77147]: pgmap v2874: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:51:50.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:51:49 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:51:50.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:51:49 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:51:50.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:51:49 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 5040 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:51:50.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:51:49 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:51:50.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:51:49 smithi115 ceph-mon[78593]: pgmap v2874: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:51:50.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:51:49 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:51:50.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:51:49 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:51:50.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:51:49 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 5040 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:51:50.946 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:51:50 smithi115 conmon[90393]: 2022-05-14T00:51:50.611+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:51:51.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:51:50 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:51:51.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:51:50 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:51:51.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:51:51 smithi115 conmon[90393]: 2022-05-14T00:51:51.605+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:51:52.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:51:51 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:51:52.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:51:51 smithi047 ceph-mon[77147]: pgmap v2875: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:51:52.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:51:51 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:51:52.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:51:51 smithi115 ceph-mon[78593]: pgmap v2875: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:51:52.954 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:51:52 smithi115 conmon[90393]: 2022-05-14T00:51:52.607+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:51:53.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:51:52 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:51:53.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:51:52 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:51:53.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:51:53 smithi115 conmon[90393]: 2022-05-14T00:51:53.592+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:51:54.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:51:53 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:51:54.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:51:53 smithi047 ceph-mon[77147]: pgmap v2876: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:51:54.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:51:53 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:51:54.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:51:53 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:51:54.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:51:53 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:51:54.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:51:53 smithi115 ceph-mon[78593]: pgmap v2876: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:51:54.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:51:53 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:51:54.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:51:53 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:51:54.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:51:54 smithi115 conmon[90393]: 2022-05-14T00:51:54.594+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:51:55.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:51:54 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:51:55.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:51:54 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 5045 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:51:55.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:51:54 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:51:55.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:51:54 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 5045 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:51:55.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:51:55 smithi115 conmon[90393]: 2022-05-14T00:51:55.563+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:51:56.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:51:55 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:51:56.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:51:55 smithi047 ceph-mon[77147]: pgmap v2877: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:51:56.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:51:55 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:51:56.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:51:55 smithi115 ceph-mon[78593]: pgmap v2877: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:51:56.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:51:56 smithi115 conmon[90393]: 2022-05-14T00:51:56.541+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:51:57.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:51:56 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:51:57.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:51:56 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T00:51:57.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:51:56 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T00:51:57.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:51:56 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T00:51:57.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:51:56 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:51:57.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:51:56 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T00:51:57.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:51:56 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T00:51:57.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:51:56 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T00:51:57.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:51:57 smithi115 conmon[90393]: 2022-05-14T00:51:57.554+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:51:58.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:51:57 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:51:58.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:51:57 smithi047 ceph-mon[77147]: pgmap v2878: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:51:58.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:51:57 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:51:58.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:51:57 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:51:58.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:51:57 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:51:58.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:51:57 smithi115 ceph-mon[78593]: pgmap v2878: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:51:58.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:51:57 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:51:58.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:51:57 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:51:58.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:51:58 smithi115 conmon[90393]: 2022-05-14T00:51:58.521+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:51:59.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:51:58 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:51:59.291 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:51:59 smithi047 conmon[78521]: 2022-05-14T00:51:59.151+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:51:59.292 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:51:59 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T00:51:59.292 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:51:59 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:51:59.292 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:51:59 smithi047 conmon[78521]: yield 2022-05-14T00:51:59.292 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:51:59 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:51:59.293 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:51:59 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:51:59.293 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:51:59 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:51:59.293 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:51:59 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T00:51:59.293 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:51:59 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:51:59.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:51:59 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T00:51:59.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:51:59 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:51:59.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:51:59 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T00:51:59.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:51:59 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:51:59.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:51:59 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T00:51:59.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:51:59 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:51:59.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:51:59 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T00:51:59.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:51:59 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:51:59.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:51:59 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:51:59.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:51:59 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:51:59.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:51:58 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:51:59.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:51:59 smithi115 conmon[90393]: 2022-05-14T00:51:59.560+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:52:00.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:51:59 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:52:00.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:51:59 smithi047 ceph-mon[77147]: pgmap v2879: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:52:00.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:51:59 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:52:00.292 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T00:52:00.292 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:52:00.292 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T00:52:00.292 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:52:00.292 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:52:00.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:52:00.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T00:52:00.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:52:00.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T00:52:00.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:52:00.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T00:52:00.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:52:00.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T00:52:00.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:52:00.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T00:52:00.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:52:00.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:52:00.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:52:00.296 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:51:59 smithi047 ceph-mon[77147]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T00:52:00.296 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:51:59 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 5050 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:52:00.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:51:59 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:52:00.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:51:59 smithi115 ceph-mon[78593]: pgmap v2879: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:52:00.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:51:59 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:52:00.358 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T00:52:00.359 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:52:00.359 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T00:52:00.359 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:52:00.359 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:52:00.359 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:52:00.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T00:52:00.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:52:00.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T00:52:00.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:52:00.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T00:52:00.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:52:00.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T00:52:00.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:52:00.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T00:52:00.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:52:00.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:52:00.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:52:00.362 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:51:59 smithi115 ceph-mon[78593]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T00:52:00.363 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:51:59 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 5050 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:52:00.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:52:00 smithi115 conmon[90393]: 2022-05-14T00:52:00.525+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:52:01.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:52:00 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:52:01.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:52:00 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:52:01.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:52:00 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:52:01.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:52:00 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:52:01.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:52:01 smithi115 conmon[90393]: 2022-05-14T00:52:01.567+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:52:02.227 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:52:01 smithi047 ceph-mon[77147]: pgmap v2880: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:52:02.227 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:52:01 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:52:02.228 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:52:01 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:52:02.228 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:52:01 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:52:02.229 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:52:02 smithi047 conmon[78521]: 2022-05-14T00:52:02.223+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:52:02.229 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:52:02 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T00:52:02.229 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:52:02 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:52:02.229 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:52:02 smithi047 conmon[78521]: yield 2022-05-14T00:52:02.230 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:52:02 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:52:02.230 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:52:02 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:52:02.230 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:52:02 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:52:02.230 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:52:02 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T00:52:02.230 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:52:02 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:52:02.231 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:52:02 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T00:52:02.231 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:52:02 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:52:02.231 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:52:02 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T00:52:02.231 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:52:02 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:52:02.232 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:52:02 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T00:52:02.232 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:52:02 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:52:02.232 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:52:02 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T00:52:02.232 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:52:02 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:52:02.232 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:52:02 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:52:02.233 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:52:02 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:52:02.233 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:52:02 smithi047 conmon[78521]: 2022-05-14T00:52:02.224+0000 7f5345f84700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T00:52:02.233 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:52:02 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T00:52:02.233 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:52:02 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:52:02.233 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:52:02 smithi047 conmon[78521]: yield 2022-05-14T00:52:02.234 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:52:02 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:52:02.234 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:52:02 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:52:02.234 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:52:02 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:52:02.234 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:52:02 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T00:52:02.235 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:52:02 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:52:02.235 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:52:02 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T00:52:02.235 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:52:02 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:52:02.235 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:52:02 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T00:52:02.235 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:52:02 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:52:02.236 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:52:02 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T00:52:02.236 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:52:02 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:52:02.236 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:52:02 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T00:52:02.236 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:52:02 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:52:02.236 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:52:02 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:52:02.237 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:52:02 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:52:02.237 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:52:02 smithi047 conmon[78521]: 2022-05-14T00:52:02.237 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:52:02 smithi047 conmon[78521]: During handling of the above exception, another exception occurred: 2022-05-14T00:52:02.237 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:52:02 smithi047 conmon[78521]: 2022-05-14T00:52:02.238 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:52:02 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T00:52:02.238 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:52:02 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T00:52:02.238 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:52:02 smithi047 conmon[78521]: return f(*arg) 2022-05-14T00:52:02.238 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:52:02 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T00:52:02.238 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:52:02 smithi047 conmon[78521]: self._write_client_files(client_files, host) 2022-05-14T00:52:02.239 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:52:02 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T00:52:02.239 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:52:02 smithi047 conmon[78521]: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T00:52:02.239 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:52:02 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T00:52:02.239 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:52:02 smithi047 conmon[78521]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T00:52:02.239 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:52:02 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T00:52:02.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:52:01 smithi115 ceph-mon[78593]: pgmap v2880: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:52:02.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:52:01 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:52:02.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:52:01 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:52:02.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:52:01 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:52:02.540 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:52:02 smithi047 conmon[78521]: return self.event_loop.get_result(coro) 2022-05-14T00:52:02.541 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:52:02 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T00:52:02.541 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:52:02 smithi047 conmon[78521]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T00:52:02.541 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:52:02 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T00:52:02.542 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:52:02 smithi047 conmon[78521]: return self.__get_result() 2022-05-14T00:52:02.542 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:52:02 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T00:52:02.542 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:52:02 smithi047 conmon[78521]: raise self._exception 2022-05-14T00:52:02.542 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:52:02 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T00:52:02.542 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:52:02 smithi047 conmon[78521]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T00:52:02.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:52:02 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T00:52:02.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:52:02 smithi047 conmon[78521]: conn = await self._remote_connection(host, addr) 2022-05-14T00:52:02.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:52:02 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T00:52:02.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:52:02 smithi047 conmon[78521]: raise 2022-05-14T00:52:02.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:52:02 smithi047 conmon[78521]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T00:52:02.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:52:02 smithi047 conmon[78521]: self.gen.throw(type, value, traceback) 2022-05-14T00:52:02.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:52:02 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T00:52:02.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:52:02 smithi047 conmon[78521]: raise OrchestratorError(msg) 2022-05-14T00:52:02.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:52:02 smithi047 conmon[78521]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:52:02.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:52:02 smithi115 conmon[90393]: 2022-05-14T00:52:02.604+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:52:03.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:52:02 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:52:03.291 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T00:52:03.291 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:52:03.292 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T00:52:03.292 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:52:03.292 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:52:03.292 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:52:03.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T00:52:03.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:52:03.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T00:52:03.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:52:03.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T00:52:03.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:52:03.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T00:52:03.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:52:03.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T00:52:03.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:52:03.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:52:03.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:52:03.295 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:52:02 smithi047 ceph-mon[77147]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T00:52:03.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T00:52:03.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:52:03.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T00:52:03.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:52:03.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:52:03.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:52:03.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T00:52:03.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:52:03.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T00:52:03.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:52:03.298 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T00:52:03.298 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:52:03.298 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T00:52:03.298 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:52:03.299 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T00:52:03.299 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:52:03.299 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:52:03.299 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:52:03.299 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T00:52:03.300 INFO:journalctl@ceph.mon.b.smithi047.stdout: During handling of the above exception, another exception occurred: 2022-05-14T00:52:03.300 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T00:52:03.300 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T00:52:03.300 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T00:52:03.301 INFO:journalctl@ceph.mon.b.smithi047.stdout: return f(*arg) 2022-05-14T00:52:03.301 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T00:52:03.301 INFO:journalctl@ceph.mon.b.smithi047.stdout: self._write_client_files(client_files, host) 2022-05-14T00:52:03.301 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T00:52:03.302 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T00:52:03.302 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T00:52:03.302 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T00:52:03.302 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T00:52:03.302 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.event_loop.get_result(coro) 2022-05-14T00:52:03.303 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T00:52:03.303 INFO:journalctl@ceph.mon.b.smithi047.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T00:52:03.303 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T00:52:03.303 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.__get_result() 2022-05-14T00:52:03.303 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T00:52:03.304 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise self._exception 2022-05-14T00:52:03.304 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T00:52:03.304 INFO:journalctl@ceph.mon.b.smithi047.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T00:52:03.304 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T00:52:03.304 INFO:journalctl@ceph.mon.b.smithi047.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T00:52:03.305 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T00:52:03.305 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise 2022-05-14T00:52:03.305 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T00:52:03.305 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.gen.throw(type, value, traceback) 2022-05-14T00:52:03.306 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T00:52:03.306 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OrchestratorError(msg) 2022-05-14T00:52:03.306 INFO:journalctl@ceph.mon.b.smithi047.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:52:03.306 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:52:02 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:52:03.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:52:02 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:52:03.358 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T00:52:03.358 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:52:03.358 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T00:52:03.359 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:52:03.359 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:52:03.359 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:52:03.359 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T00:52:03.359 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:52:03.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T00:52:03.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:52:03.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T00:52:03.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:52:03.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T00:52:03.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:52:03.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T00:52:03.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:52:03.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:52:03.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:52:03.362 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:52:02 smithi115 ceph-mon[78593]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T00:52:03.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T00:52:03.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:52:03.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T00:52:03.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:52:03.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:52:03.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:52:03.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T00:52:03.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:52:03.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T00:52:03.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:52:03.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T00:52:03.365 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:52:03.365 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T00:52:03.365 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:52:03.365 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T00:52:03.366 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:52:03.366 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:52:03.366 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:52:03.366 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T00:52:03.366 INFO:journalctl@ceph.mon.c.smithi115.stdout: During handling of the above exception, another exception occurred: 2022-05-14T00:52:03.367 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T00:52:03.367 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T00:52:03.367 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T00:52:03.367 INFO:journalctl@ceph.mon.c.smithi115.stdout: return f(*arg) 2022-05-14T00:52:03.368 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T00:52:03.368 INFO:journalctl@ceph.mon.c.smithi115.stdout: self._write_client_files(client_files, host) 2022-05-14T00:52:03.368 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T00:52:03.369 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T00:52:03.369 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T00:52:03.369 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T00:52:03.369 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T00:52:03.369 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.event_loop.get_result(coro) 2022-05-14T00:52:03.370 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T00:52:03.370 INFO:journalctl@ceph.mon.c.smithi115.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T00:52:03.370 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T00:52:03.370 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.__get_result() 2022-05-14T00:52:03.371 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T00:52:03.371 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise self._exception 2022-05-14T00:52:03.371 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T00:52:03.371 INFO:journalctl@ceph.mon.c.smithi115.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T00:52:03.371 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T00:52:03.372 INFO:journalctl@ceph.mon.c.smithi115.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T00:52:03.372 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T00:52:03.372 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise 2022-05-14T00:52:03.372 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T00:52:03.373 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.gen.throw(type, value, traceback) 2022-05-14T00:52:03.373 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T00:52:03.373 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OrchestratorError(msg) 2022-05-14T00:52:03.374 INFO:journalctl@ceph.mon.c.smithi115.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:52:03.374 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:52:02 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:52:03.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:52:03 smithi115 conmon[90393]: 2022-05-14T00:52:03.590+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:52:04.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:52:03 smithi047 ceph-mon[77147]: pgmap v2881: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:52:04.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:52:03 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:52:04.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:52:03 smithi115 ceph-mon[78593]: pgmap v2881: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:52:04.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:52:03 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:52:04.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:52:04 smithi115 conmon[90393]: 2022-05-14T00:52:04.601+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:52:05.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:52:04 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 5055 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:52:05.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:52:04 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:52:05.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:52:04 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 5055 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:52:05.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:52:04 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:52:05.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:52:05 smithi115 conmon[90393]: 2022-05-14T00:52:05.575+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:52:06.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:52:06 smithi047 ceph-mon[77147]: pgmap v2882: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:52:06.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:52:06 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:52:06.590 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:52:06 smithi115 ceph-mon[78593]: pgmap v2882: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:52:06.591 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:52:06 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:52:06.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:52:06 smithi115 conmon[90393]: 2022-05-14T00:52:06.589+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:52:07.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:52:07 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:52:07.583 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:52:07 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:52:07.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:52:07 smithi115 conmon[90393]: 2022-05-14T00:52:07.582+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:52:08.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:52:08 smithi047 ceph-mon[77147]: pgmap v2883: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:52:08.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:52:08 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:52:08.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:52:08 smithi115 conmon[90393]: 2022-05-14T00:52:08.534+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:52:08.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:52:08 smithi115 ceph-mon[78593]: pgmap v2883: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:52:08.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:52:08 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:52:09.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:52:09 smithi115 conmon[90393]: 2022-05-14T00:52:09.520+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:52:09.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:52:09 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:52:09.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:52:09 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 5060 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:52:09.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:52:09 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:52:09.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:52:09 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 5060 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:52:10.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:52:10 smithi115 conmon[90393]: 2022-05-14T00:52:10.495+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:52:10.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:52:10 smithi115 ceph-mon[78593]: pgmap v2884: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:52:10.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:52:10 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:52:10.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:52:10 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:52:10.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:52:10 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:52:10.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:52:10 smithi047 ceph-mon[77147]: pgmap v2884: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:52:10.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:52:10 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:52:10.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:52:10 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:52:10.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:52:10 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:52:11.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:52:11 smithi115 conmon[90393]: 2022-05-14T00:52:11.507+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:52:11.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:52:11 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:52:11.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:52:11 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:52:12.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:52:12 smithi115 conmon[90393]: 2022-05-14T00:52:12.535+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:52:12.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:52:12 smithi115 ceph-mon[78593]: pgmap v2885: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:52:12.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:52:12 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:52:12.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:52:12 smithi047 ceph-mon[77147]: pgmap v2885: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:52:12.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:52:12 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:52:13.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:52:13 smithi115 conmon[90393]: 2022-05-14T00:52:13.552+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:52:13.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:52:13 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:52:13.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:52:13 smithi115 ceph-mon[78593]: pgmap v2886: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:52:13.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:52:13 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:52:13.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:52:13 smithi047 ceph-mon[77147]: pgmap v2886: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:52:14.570 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:52:14 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:52:14.570 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:52:14 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:52:14.570 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:52:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T00:52:14.571 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:52:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T00:52:14.571 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:52:14 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:52:14.571 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:52:14 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 5065 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:52:14.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:52:14 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:52:14.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:52:14 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:52:14.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:52:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T00:52:14.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:52:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T00:52:14.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:52:14 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:52:14.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:52:14 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 5065 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:52:14.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:52:14 smithi115 conmon[90393]: 2022-05-14T00:52:14.568+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:52:15.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:52:15 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:52:15.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:52:15 smithi115 ceph-mon[78593]: pgmap v2887: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:52:15.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:52:15 smithi115 conmon[90393]: 2022-05-14T00:52:15.543+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:52:15.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:52:15 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:52:15.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:52:15 smithi047 ceph-mon[77147]: pgmap v2887: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:52:16.588 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:52:16 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:52:16.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:52:16 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:52:16.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:52:16 smithi115 conmon[90393]: 2022-05-14T00:52:16.587+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:52:17.575 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:52:17 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:52:17.576 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:52:17 smithi115 ceph-mon[78593]: pgmap v2888: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:52:17.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:52:17 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:52:17.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:52:17 smithi047 ceph-mon[77147]: pgmap v2888: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:52:17.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:52:17 smithi115 conmon[90393]: 2022-05-14T00:52:17.574+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:52:18.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:52:18 smithi115 conmon[90393]: 2022-05-14T00:52:18.538+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:52:18.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:52:18 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:52:18.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:52:18 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:52:18.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:52:18 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:52:18.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:52:18 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:52:18.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:52:18 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:52:18.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:52:18 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:52:19.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:52:19 smithi115 conmon[90393]: 2022-05-14T00:52:19.553+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:52:19.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:52:19 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:52:19.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:52:19 smithi115 ceph-mon[78593]: pgmap v2889: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:52:19.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:52:19 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 5070 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:52:19.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:52:19 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:52:19.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:52:19 smithi047 ceph-mon[77147]: pgmap v2889: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:52:19.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:52:19 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 5070 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:52:20.599 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:52:20 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:52:20.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:52:20 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:52:20.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:52:20 smithi115 conmon[90393]: 2022-05-14T00:52:20.598+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:52:21.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:52:21 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:52:21.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:52:21 smithi115 ceph-mon[78593]: pgmap v2890: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:52:21.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:52:21 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:52:21.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:52:21 smithi047 ceph-mon[77147]: pgmap v2890: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:52:22.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:52:21 smithi115 conmon[90393]: 2022-05-14T00:52:21.612+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:52:22.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:52:22 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:52:22.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:52:22 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:52:22.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:52:22 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:52:22.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:52:22 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:52:22.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:52:22 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:52:22.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:52:22 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:52:23.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:52:22 smithi115 conmon[90393]: 2022-05-14T00:52:22.643+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:52:23.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:52:23 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:52:23.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:52:23 smithi115 ceph-mon[78593]: pgmap v2891: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:52:23.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:52:23 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:52:23.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:52:23 smithi047 ceph-mon[77147]: pgmap v2891: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:52:24.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:52:23 smithi115 conmon[90393]: 2022-05-14T00:52:23.657+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:52:24.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:52:24 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:52:24.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:52:24 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 5075 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:52:24.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:52:24 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:52:24.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:52:24 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 5075 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:52:25.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:52:24 smithi115 conmon[90393]: 2022-05-14T00:52:24.625+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:52:25.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:52:25 smithi115 conmon[90393]: 2022-05-14T00:52:25.576+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:52:25.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:52:25 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:52:25.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:52:25 smithi115 ceph-mon[78593]: pgmap v2892: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:52:25.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:52:25 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:52:25.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:52:25 smithi047 ceph-mon[77147]: pgmap v2892: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:52:26.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:52:26 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:52:26.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:52:26 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:52:27.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:52:26 smithi115 conmon[90393]: 2022-05-14T00:52:26.623+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:52:27.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:52:27 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:52:27.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:52:27 smithi115 ceph-mon[78593]: pgmap v2893: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:52:27.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:52:27 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:52:27.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:52:27 smithi047 ceph-mon[77147]: pgmap v2893: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:52:28.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:52:27 smithi115 conmon[90393]: 2022-05-14T00:52:27.662+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:52:28.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:52:28 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:52:28.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:52:28 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:52:29.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:52:28 smithi115 conmon[90393]: 2022-05-14T00:52:28.696+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:52:29.683 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:52:29 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:52:29.684 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:52:29 smithi115 ceph-mon[78593]: pgmap v2894: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:52:29.684 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:52:29 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 5080 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:52:29.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:52:29 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:52:29.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:52:29 smithi047 ceph-mon[77147]: pgmap v2894: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:52:29.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:52:29 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 5080 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:52:30.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:52:29 smithi115 conmon[90393]: 2022-05-14T00:52:29.683+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:52:30.713 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:52:30 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:52:30.713 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:52:30 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:52:30.713 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:52:30 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:52:30.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:52:30 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:52:30.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:52:30 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:52:30.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:52:30 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:52:31.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:52:30 smithi115 conmon[90393]: 2022-05-14T00:52:30.712+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:52:31.666 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:52:31 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:52:31.666 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:52:31 smithi115 ceph-mon[78593]: pgmap v2895: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:52:31.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:52:31 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:52:31.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:52:31 smithi047 ceph-mon[77147]: pgmap v2895: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:52:32.112 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:52:31 smithi115 conmon[90393]: 2022-05-14T00:52:31.666+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:52:32.618 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:52:32 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:52:32.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:52:32 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:52:33.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:52:32 smithi115 conmon[90393]: 2022-05-14T00:52:32.617+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:52:33.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:52:33 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:52:33.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:52:33 smithi047 ceph-mon[77147]: pgmap v2896: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:52:33.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:52:33 smithi115 conmon[90393]: 2022-05-14T00:52:33.578+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:52:33.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:52:33 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:52:33.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:52:33 smithi115 ceph-mon[78593]: pgmap v2896: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:52:34.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:52:34 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:52:34.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:52:34 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:52:34.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:52:34 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:52:34.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:52:34 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 5085 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:52:34.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:52:34 smithi115 conmon[90393]: 2022-05-14T00:52:34.545+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:52:34.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:52:34 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:52:34.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:52:34 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:52:34.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:52:34 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:52:34.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:52:34 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 5085 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:52:35.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:52:35 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:52:35.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:52:35 smithi047 ceph-mon[77147]: pgmap v2897: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:52:35.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:52:35 smithi115 conmon[90393]: 2022-05-14T00:52:35.516+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:52:35.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:52:35 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:52:35.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:52:35 smithi115 ceph-mon[78593]: pgmap v2897: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:52:36.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:52:36 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:52:36.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:52:36 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:52:36.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:52:36 smithi115 conmon[90393]: 2022-05-14T00:52:36.505+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:52:37.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:52:37 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:52:37.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:52:37 smithi047 ceph-mon[77147]: pgmap v2898: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:52:37.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:52:37 smithi115 conmon[90393]: 2022-05-14T00:52:37.463+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:52:37.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:52:37 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:52:38.006 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:52:37 smithi115 ceph-mon[78593]: pgmap v2898: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:52:38.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:52:38 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:52:38.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:52:38 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:52:38.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:52:38 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:52:38.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:52:38 smithi115 conmon[90393]: 2022-05-14T00:52:38.487+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:52:38.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:52:38 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:52:38.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:52:38 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:52:38.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:52:38 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:52:39.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:52:39 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:52:39.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:52:39 smithi047 ceph-mon[77147]: pgmap v2899: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:52:39.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:52:39 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 5090 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:52:39.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:52:39 smithi115 conmon[90393]: 2022-05-14T00:52:39.505+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:52:39.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:52:39 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:52:39.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:52:39 smithi115 ceph-mon[78593]: pgmap v2899: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:52:39.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:52:39 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 5090 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:52:40.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:52:40 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:52:40.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:52:40 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:52:40.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:52:40 smithi115 conmon[90393]: 2022-05-14T00:52:40.471+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:52:41.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:52:41 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:52:41.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:52:41 smithi047 ceph-mon[77147]: pgmap v2900: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:52:41.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:52:41 smithi115 conmon[90393]: 2022-05-14T00:52:41.488+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:52:41.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:52:41 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:52:41.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:52:41 smithi115 ceph-mon[78593]: pgmap v2900: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:52:42.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:52:42 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:52:42.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:52:42 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:52:42.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:52:42 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:52:42.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:52:42 smithi115 conmon[90393]: 2022-05-14T00:52:42.439+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:52:42.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:52:42 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:52:42.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:52:42 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:52:42.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:52:42 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:52:43.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:52:43 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:52:43.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:52:43 smithi047 ceph-mon[77147]: pgmap v2901: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:52:43.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:52:43 smithi115 conmon[90393]: 2022-05-14T00:52:43.480+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:52:43.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:52:43 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:52:43.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:52:43 smithi115 ceph-mon[78593]: pgmap v2901: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:52:44.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:52:44 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:52:44.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:52:44 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 5095 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:52:44.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:52:44 smithi115 conmon[90393]: 2022-05-14T00:52:44.471+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:52:44.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:52:44 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:52:44.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:52:44 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 5095 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:52:45.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:52:45 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:52:45.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:52:45 smithi047 ceph-mon[77147]: pgmap v2902: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:52:45.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:52:45 smithi115 conmon[90393]: 2022-05-14T00:52:45.439+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:52:45.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:52:45 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:52:45.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:52:45 smithi115 ceph-mon[78593]: pgmap v2902: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:52:46.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:52:46 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:52:46.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:52:46 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:52:46.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:52:46 smithi115 conmon[90393]: 2022-05-14T00:52:46.404+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:52:47.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:52:47 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:52:47.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:52:47 smithi047 ceph-mon[77147]: pgmap v2903: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:52:47.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:52:47 smithi115 conmon[90393]: 2022-05-14T00:52:47.426+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:52:47.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:52:47 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:52:47.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:52:47 smithi115 ceph-mon[78593]: pgmap v2903: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:52:48.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:52:48 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:52:48.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:52:48 smithi115 conmon[90393]: 2022-05-14T00:52:48.476+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:52:48.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:52:48 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:52:49.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:52:49 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:52:49.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:52:49 smithi047 ceph-mon[77147]: pgmap v2904: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:52:49.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:52:49 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 5100 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:52:49.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:52:49 smithi115 conmon[90393]: 2022-05-14T00:52:49.433+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:52:49.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:52:49 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:52:49.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:52:49 smithi115 ceph-mon[78593]: pgmap v2904: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:52:49.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:52:49 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 5100 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:52:50.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:52:50 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:52:50.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:52:50 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:52:50.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:52:50 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:52:50.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:52:50 smithi115 conmon[90393]: 2022-05-14T00:52:50.414+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:52:50.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:52:50 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:52:50.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:52:50 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:52:50.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:52:50 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:52:51.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:52:51 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:52:51.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:52:51 smithi047 ceph-mon[77147]: pgmap v2905: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:52:51.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:52:51 smithi115 conmon[90393]: 2022-05-14T00:52:51.461+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:52:51.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:52:51 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:52:51.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:52:51 smithi115 ceph-mon[78593]: pgmap v2905: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:52:52.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:52:52 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:52:52.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:52:52 smithi115 conmon[90393]: 2022-05-14T00:52:52.467+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:52:52.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:52:52 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:52:53.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:52:53 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:52:53.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:52:53 smithi047 ceph-mon[77147]: pgmap v2906: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:52:53.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:52:53 smithi115 conmon[90393]: 2022-05-14T00:52:53.503+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:52:53.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:52:53 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:52:53.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:52:53 smithi115 ceph-mon[78593]: pgmap v2906: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:52:54.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:52:54 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:52:54.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:52:54 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:52:54.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:52:54 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:52:54.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:52:54 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 5105 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:52:54.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:52:54 smithi115 conmon[90393]: 2022-05-14T00:52:54.469+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:52:54.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:52:54 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:52:54.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:52:54 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:52:54.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:52:54 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:52:54.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:52:54 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 5105 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:52:55.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:52:55 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:52:55.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:52:55 smithi047 ceph-mon[77147]: pgmap v2907: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:52:55.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:52:55 smithi115 conmon[90393]: 2022-05-14T00:52:55.467+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:52:55.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:52:55 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:52:55.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:52:55 smithi115 ceph-mon[78593]: pgmap v2907: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:52:56.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:52:56 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:52:56.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:52:56 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:52:56.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:52:56 smithi115 conmon[90393]: 2022-05-14T00:52:56.454+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:52:57.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:52:57 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:52:57.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:52:57 smithi047 ceph-mon[77147]: pgmap v2908: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:52:57.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:52:57 smithi115 conmon[90393]: 2022-05-14T00:52:57.460+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:52:57.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:52:57 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:52:57.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:52:57 smithi115 ceph-mon[78593]: pgmap v2908: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:52:58.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:52:58 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:52:58.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:52:58 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:52:58.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:52:58 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:52:58.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:52:58 smithi115 conmon[90393]: 2022-05-14T00:52:58.413+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:52:58.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:52:58 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:52:58.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:52:58 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:52:58.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:52:58 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:52:59.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:52:59 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:52:59.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:52:59 smithi047 ceph-mon[77147]: pgmap v2909: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:52:59.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:52:59 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 5110 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:52:59.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:52:59 smithi115 conmon[90393]: 2022-05-14T00:52:59.416+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:52:59.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:52:59 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:52:59.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:52:59 smithi115 ceph-mon[78593]: pgmap v2909: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:52:59.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:52:59 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 5110 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:53:00.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:53:00 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:53:00.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:53:00 smithi115 conmon[90393]: 2022-05-14T00:53:00.447+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:53:00.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:53:00 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:53:01.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:53:01 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:53:01.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:53:01 smithi047 ceph-mon[77147]: pgmap v2910: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:53:01.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:53:01 smithi115 conmon[90393]: 2022-05-14T00:53:01.491+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:53:01.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:53:01 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:53:01.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:53:01 smithi115 ceph-mon[78593]: pgmap v2910: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:53:02.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:53:02 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:53:02.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:53:02 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:53:02.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:53:02 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:53:02.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:53:02 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T00:53:02.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:53:02 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T00:53:02.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:53:02 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T00:53:02.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:53:02 smithi115 conmon[90393]: 2022-05-14T00:53:02.512+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:53:02.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:53:02 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:53:02.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:53:02 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:53:02.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:53:02 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:53:02.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:53:02 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T00:53:02.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:53:02 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T00:53:02.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:53:02 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T00:53:03.041 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:53:02 smithi047 conmon[78521]: 2022-05-14T00:53:02.703+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:53:03.041 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:53:02 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T00:53:03.041 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:53:02 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:53:03.042 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:53:02 smithi047 conmon[78521]: yield 2022-05-14T00:53:03.042 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:53:02 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:53:03.042 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:53:02 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:53:03.042 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:53:02 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:53:03.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:53:02 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T00:53:03.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:53:02 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:53:03.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:53:02 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T00:53:03.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:53:02 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:53:03.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:53:02 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T00:53:03.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:53:02 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:53:03.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:53:02 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T00:53:03.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:53:02 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:53:03.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:53:02 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T00:53:03.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:53:02 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:53:03.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:53:02 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:53:03.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:53:02 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:53:03.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:53:03 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:53:03.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:53:03 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:53:03.791 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T00:53:03.792 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:53:03.792 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T00:53:03.792 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:53:03.792 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:53:03.793 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:53:03.793 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T00:53:03.793 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:53:03.793 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T00:53:03.793 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:53:03.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T00:53:03.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:53:03.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T00:53:03.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:53:03.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T00:53:03.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:53:03.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:53:03.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:53:03.795 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:53:03 smithi047 ceph-mon[77147]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T00:53:03.796 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:53:03 smithi047 ceph-mon[77147]: pgmap v2911: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:53:03.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:53:03 smithi115 conmon[90393]: 2022-05-14T00:53:03.471+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:53:03.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:53:03 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:53:03.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:53:03 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:53:03.858 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T00:53:03.859 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:53:03.859 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T00:53:03.859 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:53:03.859 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:53:03.859 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:53:03.860 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T00:53:03.860 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:53:03.860 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T00:53:03.860 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:53:03.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T00:53:03.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:53:03.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T00:53:03.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:53:03.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T00:53:03.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:53:03.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:53:03.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:53:03.863 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:53:03 smithi115 ceph-mon[78593]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T00:53:03.863 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:53:03 smithi115 ceph-mon[78593]: pgmap v2911: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:53:04.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:53:04 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:53:04.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:53:04 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 5115 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:53:04.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:53:04 smithi115 conmon[90393]: 2022-05-14T00:53:04.449+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:53:04.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:53:04 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:53:04.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:53:04 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 5115 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:53:05.777 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:53:05 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:53:05.777 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:53:05 smithi047 ceph-mon[77147]: pgmap v2912: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:53:05.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:53:05 smithi115 conmon[90393]: 2022-05-14T00:53:05.461+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:53:05.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:53:05 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:53:05.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:53:05 smithi115 ceph-mon[78593]: pgmap v2912: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:53:06.042 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:53:05 smithi047 conmon[78521]: 2022-05-14T00:53:05.776+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:53:06.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:53:05 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T00:53:06.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:53:05 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:53:06.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:53:05 smithi047 conmon[78521]: yield 2022-05-14T00:53:06.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:53:05 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:53:06.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:53:05 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:53:06.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:53:05 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:53:06.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:53:05 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T00:53:06.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:53:05 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:53:06.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:53:05 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T00:53:06.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:53:05 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:53:06.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:53:05 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T00:53:06.048 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:53:05 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:53:06.048 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:53:05 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T00:53:06.048 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:53:05 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:53:06.048 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:53:05 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T00:53:06.049 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:53:05 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:53:06.049 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:53:05 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:53:06.049 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:53:05 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:53:06.050 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:53:05 smithi047 conmon[78521]: 2022-05-14T00:53:05.776+0000 7f533ff78700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T00:53:06.050 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:53:05 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T00:53:06.050 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:53:05 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:53:06.050 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:53:05 smithi047 conmon[78521]: yield 2022-05-14T00:53:06.051 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:53:05 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:53:06.051 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:53:05 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:53:06.051 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:53:05 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:53:06.051 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:53:05 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T00:53:06.052 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:53:05 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:53:06.052 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:53:05 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T00:53:06.052 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:53:05 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:53:06.052 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:53:05 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T00:53:06.052 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:53:05 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:53:06.053 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:53:05 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T00:53:06.053 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:53:05 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:53:06.053 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:53:05 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T00:53:06.053 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:53:05 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:53:06.054 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:53:05 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:53:06.054 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:53:05 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:53:06.054 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:53:05 smithi047 conmon[78521]: 2022-05-14T00:53:06.054 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:53:05 smithi047 conmon[78521]: During handling of the above exception, another exception occurred: 2022-05-14T00:53:06.054 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:53:05 smithi047 conmon[78521]: 2022-05-14T00:53:06.055 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:53:05 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T00:53:06.055 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:53:05 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T00:53:06.055 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:53:05 smithi047 conmon[78521]: return f(*arg) 2022-05-14T00:53:06.056 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:53:05 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T00:53:06.056 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:53:05 smithi047 conmon[78521]: self._write_client_files(client_files, host) 2022-05-14T00:53:06.057 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:53:05 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T00:53:06.057 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:53:05 smithi047 conmon[78521]: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T00:53:06.058 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:53:05 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T00:53:06.058 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:53:05 smithi047 conmon[78521]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T00:53:06.058 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:53:05 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T00:53:06.059 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:53:05 smithi047 conmon[78521]: return self.event_loop.get_result(coro) 2022-05-14T00:53:06.059 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:53:05 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T00:53:06.060 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:53:05 smithi047 conmon[78521]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T00:53:06.060 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:53:05 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T00:53:06.061 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:53:05 smithi047 conmon[78521]: return self.__get_result() 2022-05-14T00:53:06.061 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:53:05 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T00:53:06.061 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:53:05 smithi047 conmon[78521]: raise self._exception 2022-05-14T00:53:06.062 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:53:05 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T00:53:06.062 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:53:05 smithi047 conmon[78521]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T00:53:06.063 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:53:05 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T00:53:06.063 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:53:05 smithi047 conmon[78521]: conn = await self._remote_connection(host, addr) 2022-05-14T00:53:06.063 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:53:05 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T00:53:06.064 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:53:05 smithi047 conmon[78521]: raise 2022-05-14T00:53:06.064 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:53:05 smithi047 conmon[78521]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T00:53:06.065 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:53:05 smithi047 conmon[78521]: self.gen.throw(type, value, traceback) 2022-05-14T00:53:06.065 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:53:05 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T00:53:06.065 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:53:05 smithi047 conmon[78521]: raise OrchestratorError(msg) 2022-05-14T00:53:06.066 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:53:05 smithi047 conmon[78521]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:53:06.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:53:06 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:53:06.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:53:06 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:53:06.791 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T00:53:06.792 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:53:06.792 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T00:53:06.792 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:53:06.792 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:53:06.793 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:53:06.793 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T00:53:06.793 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:53:06.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T00:53:06.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:53:06.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T00:53:06.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:53:06.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T00:53:06.797 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:53:06.798 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T00:53:06.798 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:53:06.798 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:53:06.798 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:53:06.798 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:53:06 smithi047 ceph-mon[77147]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T00:53:06.799 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T00:53:06.799 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:53:06.799 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T00:53:06.799 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:53:06.800 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:53:06.800 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:53:06.800 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T00:53:06.800 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:53:06.800 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T00:53:06.801 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:53:06.801 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T00:53:06.801 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:53:06.801 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T00:53:06.802 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:53:06.802 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T00:53:06.802 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:53:06.802 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:53:06.802 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:53:06.803 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T00:53:06.803 INFO:journalctl@ceph.mon.b.smithi047.stdout: During handling of the above exception, another exception occurred: 2022-05-14T00:53:06.803 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T00:53:06.803 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T00:53:06.804 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T00:53:06.804 INFO:journalctl@ceph.mon.b.smithi047.stdout: return f(*arg) 2022-05-14T00:53:06.804 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T00:53:06.804 INFO:journalctl@ceph.mon.b.smithi047.stdout: self._write_client_files(client_files, host) 2022-05-14T00:53:06.804 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T00:53:06.805 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T00:53:06.805 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T00:53:06.805 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T00:53:06.805 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T00:53:06.806 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.event_loop.get_result(coro) 2022-05-14T00:53:06.806 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T00:53:06.806 INFO:journalctl@ceph.mon.b.smithi047.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T00:53:06.806 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T00:53:06.807 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.__get_result() 2022-05-14T00:53:06.807 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T00:53:06.807 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise self._exception 2022-05-14T00:53:06.807 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T00:53:06.807 INFO:journalctl@ceph.mon.b.smithi047.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T00:53:06.808 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T00:53:06.808 INFO:journalctl@ceph.mon.b.smithi047.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T00:53:06.808 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T00:53:06.808 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise 2022-05-14T00:53:06.808 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T00:53:06.809 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.gen.throw(type, value, traceback) 2022-05-14T00:53:06.809 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T00:53:06.809 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OrchestratorError(msg) 2022-05-14T00:53:06.809 INFO:journalctl@ceph.mon.b.smithi047.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:53:06.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:53:06 smithi115 conmon[90393]: 2022-05-14T00:53:06.503+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:53:06.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:53:06 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:53:06.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:53:06 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:53:06.859 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T00:53:06.859 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:53:06.859 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T00:53:06.860 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:53:06.860 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:53:06.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:53:06.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T00:53:06.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:53:06.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T00:53:06.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:53:06.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T00:53:06.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:53:06.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T00:53:06.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:53:06.863 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T00:53:06.863 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:53:06.863 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:53:06.863 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:53:06.864 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:53:06 smithi115 ceph-mon[78593]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T00:53:06.864 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T00:53:06.864 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:53:06.864 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T00:53:06.864 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:53:06.865 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:53:06.865 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:53:06.865 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T00:53:06.865 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:53:06.865 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T00:53:06.866 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:53:06.866 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T00:53:06.866 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:53:06.866 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T00:53:06.867 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:53:06.867 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T00:53:06.867 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:53:06.867 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:53:06.867 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:53:06.868 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T00:53:06.868 INFO:journalctl@ceph.mon.c.smithi115.stdout: During handling of the above exception, another exception occurred: 2022-05-14T00:53:06.868 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T00:53:06.869 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T00:53:06.869 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T00:53:06.869 INFO:journalctl@ceph.mon.c.smithi115.stdout: return f(*arg) 2022-05-14T00:53:06.869 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T00:53:06.870 INFO:journalctl@ceph.mon.c.smithi115.stdout: self._write_client_files(client_files, host) 2022-05-14T00:53:06.871 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T00:53:06.871 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T00:53:06.871 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T00:53:06.871 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T00:53:06.872 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T00:53:06.872 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.event_loop.get_result(coro) 2022-05-14T00:53:06.872 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T00:53:06.872 INFO:journalctl@ceph.mon.c.smithi115.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T00:53:06.873 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T00:53:06.873 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.__get_result() 2022-05-14T00:53:06.873 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T00:53:06.873 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise self._exception 2022-05-14T00:53:06.874 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T00:53:06.874 INFO:journalctl@ceph.mon.c.smithi115.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T00:53:06.874 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T00:53:06.874 INFO:journalctl@ceph.mon.c.smithi115.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T00:53:06.875 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T00:53:06.875 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise 2022-05-14T00:53:06.875 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T00:53:06.875 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.gen.throw(type, value, traceback) 2022-05-14T00:53:06.875 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T00:53:06.876 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OrchestratorError(msg) 2022-05-14T00:53:06.876 INFO:journalctl@ceph.mon.c.smithi115.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:53:07.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:53:07 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:53:07.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:53:07 smithi047 ceph-mon[77147]: pgmap v2913: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:53:07.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:53:07 smithi115 conmon[90393]: 2022-05-14T00:53:07.475+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:53:07.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:53:07 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:53:07.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:53:07 smithi115 ceph-mon[78593]: pgmap v2913: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:53:08.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:53:08 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:53:08.861 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:53:08 smithi115 conmon[90393]: 2022-05-14T00:53:08.460+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:53:08.862 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:53:08 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:53:09.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:53:09 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:53:09.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:53:09 smithi047 ceph-mon[77147]: pgmap v2914: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:53:09.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:53:09 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 5120 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:53:09.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:53:09 smithi115 conmon[90393]: 2022-05-14T00:53:09.442+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:53:09.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:53:09 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:53:09.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:53:09 smithi115 ceph-mon[78593]: pgmap v2914: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:53:09.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:53:09 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 5120 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:53:10.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:53:10 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:53:10.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:53:10 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:53:10.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:53:10 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:53:10.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:53:10 smithi115 conmon[90393]: 2022-05-14T00:53:10.419+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:53:10.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:53:10 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:53:10.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:53:10 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:53:10.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:53:10 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:53:11.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:53:11 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:53:11.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:53:11 smithi047 ceph-mon[77147]: pgmap v2915: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:53:11.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:53:11 smithi115 conmon[90393]: 2022-05-14T00:53:11.448+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:53:11.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:53:11 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:53:11.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:53:11 smithi115 ceph-mon[78593]: pgmap v2915: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:53:12.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:53:12 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:53:12.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:53:12 smithi115 conmon[90393]: 2022-05-14T00:53:12.460+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:53:12.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:53:12 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:53:13.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:53:13 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:53:13.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:53:13 smithi047 ceph-mon[77147]: pgmap v2916: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:53:13.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:53:13 smithi115 conmon[90393]: 2022-05-14T00:53:13.497+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:53:13.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:53:13 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:53:13.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:53:13 smithi115 ceph-mon[78593]: pgmap v2916: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:53:14.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:53:14 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:53:14.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:53:14 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:53:14.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:53:14 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:53:14.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:53:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T00:53:14.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:53:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T00:53:14.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:53:14 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 5125 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:53:14.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:53:14 smithi115 conmon[90393]: 2022-05-14T00:53:14.497+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:53:14.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:53:14 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:53:14.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:53:14 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:53:14.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:53:14 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:53:14.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:53:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T00:53:14.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:53:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T00:53:14.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:53:14 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 5125 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:53:15.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:53:15 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:53:15.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:53:15 smithi047 ceph-mon[77147]: pgmap v2917: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:53:15.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:53:15 smithi115 conmon[90393]: 2022-05-14T00:53:15.475+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:53:15.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:53:15 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:53:15.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:53:15 smithi115 ceph-mon[78593]: pgmap v2917: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:53:16.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:53:16 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:53:16.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:53:16 smithi115 conmon[90393]: 2022-05-14T00:53:16.461+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:53:16.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:53:16 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:53:17.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:53:17 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:53:17.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:53:17 smithi047 ceph-mon[77147]: pgmap v2918: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:53:17.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:53:17 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:53:17.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:53:17 smithi115 ceph-mon[78593]: pgmap v2918: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:53:17.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:53:17 smithi115 conmon[90393]: 2022-05-14T00:53:17.472+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:53:18.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:53:18 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:53:18.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:53:18 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:53:18.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:53:18 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:53:18.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:53:18 smithi115 conmon[90393]: 2022-05-14T00:53:18.470+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:53:18.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:53:18 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:53:18.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:53:18 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:53:18.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:53:18 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:53:19.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:53:19 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:53:19.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:53:19 smithi047 ceph-mon[77147]: pgmap v2919: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:53:19.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:53:19 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 5130 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:53:19.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:53:19 smithi115 conmon[90393]: 2022-05-14T00:53:19.479+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:53:19.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:53:19 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:53:19.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:53:19 smithi115 ceph-mon[78593]: pgmap v2919: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:53:19.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:53:19 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 5130 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:53:20.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:53:20 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:53:20.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:53:20 smithi115 conmon[90393]: 2022-05-14T00:53:20.497+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:53:20.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:53:20 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:53:21.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:53:21 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:53:21.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:53:21 smithi047 ceph-mon[77147]: pgmap v2920: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:53:21.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:53:21 smithi115 conmon[90393]: 2022-05-14T00:53:21.466+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:53:21.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:53:21 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:53:21.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:53:21 smithi115 ceph-mon[78593]: pgmap v2920: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:53:22.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:53:22 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:53:22.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:53:22 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:53:22.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:53:22 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:53:22.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:53:22 smithi115 conmon[90393]: 2022-05-14T00:53:22.506+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:53:22.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:53:22 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:53:22.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:53:22 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:53:22.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:53:22 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:53:23.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:53:23 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:53:23.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:53:23 smithi047 ceph-mon[77147]: pgmap v2921: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:53:23.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:53:23 smithi115 conmon[90393]: 2022-05-14T00:53:23.552+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:53:23.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:53:23 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:53:23.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:53:23 smithi115 ceph-mon[78593]: pgmap v2921: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:53:24.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:53:24 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:53:24.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:53:24 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 5135 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:53:24.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:53:24 smithi115 conmon[90393]: 2022-05-14T00:53:24.518+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:53:24.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:53:24 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:53:24.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:53:24 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 5135 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:53:25.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:53:25 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:53:25.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:53:25 smithi047 ceph-mon[77147]: pgmap v2922: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:53:25.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:53:25 smithi115 conmon[90393]: 2022-05-14T00:53:25.533+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:53:25.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:53:25 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:53:25.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:53:25 smithi115 ceph-mon[78593]: pgmap v2922: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:53:26.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:53:26 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:53:26.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:53:26 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:53:26.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:53:26 smithi115 conmon[90393]: 2022-05-14T00:53:26.518+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:53:27.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:53:27 smithi115 conmon[90393]: 2022-05-14T00:53:27.538+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:53:27.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:53:27 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:53:27.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:53:27 smithi115 ceph-mon[78593]: pgmap v2923: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:53:28.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:53:27 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:53:28.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:53:27 smithi047 ceph-mon[77147]: pgmap v2923: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:53:28.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:53:28 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:53:28.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:53:28 smithi115 conmon[90393]: 2022-05-14T00:53:28.533+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:53:29.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:53:28 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:53:29.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:53:29 smithi115 conmon[90393]: 2022-05-14T00:53:29.549+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:53:29.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:53:29 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:53:29.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:53:29 smithi115 ceph-mon[78593]: pgmap v2924: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:53:29.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:53:29 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 5140 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:53:30.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:53:29 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:53:30.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:53:29 smithi047 ceph-mon[77147]: pgmap v2924: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:53:30.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:53:29 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 5140 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:53:30.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:53:30 smithi115 conmon[90393]: 2022-05-14T00:53:30.507+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:53:30.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:53:30 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:53:30.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:53:30 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:53:30.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:53:30 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:53:31.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:53:30 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:53:31.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:53:30 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:53:31.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:53:30 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:53:31.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:53:31 smithi115 conmon[90393]: 2022-05-14T00:53:31.538+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:53:31.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:53:31 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:53:31.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:53:31 smithi115 ceph-mon[78593]: pgmap v2925: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:53:32.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:53:31 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:53:32.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:53:31 smithi047 ceph-mon[77147]: pgmap v2925: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:53:32.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:53:32 smithi115 conmon[90393]: 2022-05-14T00:53:32.497+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:53:32.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:53:32 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:53:33.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:53:32 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:53:33.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:53:33 smithi115 conmon[90393]: 2022-05-14T00:53:33.458+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:53:33.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:53:33 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:53:33.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:53:33 smithi115 ceph-mon[78593]: pgmap v2926: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:53:34.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:53:33 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:53:34.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:53:33 smithi047 ceph-mon[77147]: pgmap v2926: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:53:34.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:53:34 smithi115 conmon[90393]: 2022-05-14T00:53:34.470+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:53:34.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:53:34 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:53:34.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:53:34 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:53:34.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:53:34 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:53:34.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:53:34 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 5145 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:53:35.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:53:34 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:53:35.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:53:34 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:53:35.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:53:34 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:53:35.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:53:34 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 5145 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:53:35.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:53:35 smithi115 conmon[90393]: 2022-05-14T00:53:35.500+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:53:35.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:53:35 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:53:35.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:53:35 smithi115 ceph-mon[78593]: pgmap v2927: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:53:36.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:53:35 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:53:36.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:53:35 smithi047 ceph-mon[77147]: pgmap v2927: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:53:36.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:53:36 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:53:36.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:53:36 smithi115 conmon[90393]: 2022-05-14T00:53:36.475+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:53:37.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:53:36 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:53:37.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:53:37 smithi115 conmon[90393]: 2022-05-14T00:53:37.431+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:53:37.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:53:37 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:53:37.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:53:37 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:53:37.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:53:37 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:53:37.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:53:37 smithi115 ceph-mon[78593]: pgmap v2928: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:53:38.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:53:37 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:53:38.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:53:37 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:53:38.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:53:37 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:53:38.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:53:37 smithi047 ceph-mon[77147]: pgmap v2928: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:53:38.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:53:38 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:53:38.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:53:38 smithi115 conmon[90393]: 2022-05-14T00:53:38.448+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:53:39.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:53:38 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:53:39.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:53:39 smithi115 conmon[90393]: 2022-05-14T00:53:39.482+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:53:39.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:53:39 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:53:39.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:53:39 smithi115 ceph-mon[78593]: pgmap v2929: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:53:39.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:53:39 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 5150 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:53:40.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:53:39 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:53:40.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:53:39 smithi047 ceph-mon[77147]: pgmap v2929: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:53:40.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:53:39 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 5150 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:53:40.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:53:40 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:53:40.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:53:40 smithi115 conmon[90393]: 2022-05-14T00:53:40.511+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:53:41.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:53:40 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:53:41.862 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:53:41 smithi115 conmon[90393]: 2022-05-14T00:53:41.505+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:53:41.862 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:53:41 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:53:41.863 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:53:41 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:53:41.864 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:53:41 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:53:41.864 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:53:41 smithi115 ceph-mon[78593]: pgmap v2930: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:53:42.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:53:41 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:53:42.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:53:41 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:53:42.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:53:41 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:53:42.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:53:41 smithi047 ceph-mon[77147]: pgmap v2930: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:53:42.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:53:42 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:53:42.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:53:42 smithi115 conmon[90393]: 2022-05-14T00:53:42.496+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:53:43.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:53:42 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:53:43.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:53:43 smithi115 conmon[90393]: 2022-05-14T00:53:43.496+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:53:43.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:53:43 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:53:43.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:53:43 smithi115 ceph-mon[78593]: pgmap v2931: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:53:44.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:53:43 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:53:44.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:53:43 smithi047 ceph-mon[77147]: pgmap v2931: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:53:44.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:53:44 smithi115 conmon[90393]: 2022-05-14T00:53:44.545+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:53:44.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:53:44 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:53:44.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:53:44 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 5155 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:53:45.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:53:44 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:53:45.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:53:44 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 5155 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:53:45.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:53:45 smithi115 conmon[90393]: 2022-05-14T00:53:45.512+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:53:45.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:53:45 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:53:45.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:53:45 smithi115 ceph-mon[78593]: pgmap v2932: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:53:46.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:53:45 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:53:46.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:53:45 smithi047 ceph-mon[77147]: pgmap v2932: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:53:46.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:53:46 smithi115 conmon[90393]: 2022-05-14T00:53:46.505+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:53:46.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:53:46 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:53:47.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:53:46 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:53:47.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:53:47 smithi115 conmon[90393]: 2022-05-14T00:53:47.480+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:53:47.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:53:47 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:53:47.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:53:47 smithi115 ceph-mon[78593]: pgmap v2933: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:53:48.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:53:47 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:53:48.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:53:47 smithi047 ceph-mon[77147]: pgmap v2933: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:53:48.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:53:48 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:53:48.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:53:48 smithi115 conmon[90393]: 2022-05-14T00:53:48.483+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:53:49.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:53:48 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:53:49.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:53:49 smithi115 conmon[90393]: 2022-05-14T00:53:49.461+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:53:49.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:53:49 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:53:49.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:53:49 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:53:49.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:53:49 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:53:49.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:53:49 smithi115 ceph-mon[78593]: pgmap v2934: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:53:49.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:53:49 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 5160 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:53:50.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:53:49 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:53:50.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:53:49 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:53:50.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:53:49 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:53:50.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:53:49 smithi047 ceph-mon[77147]: pgmap v2934: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:53:50.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:53:49 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 5160 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:53:50.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:53:50 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:53:50.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:53:50 smithi115 conmon[90393]: 2022-05-14T00:53:50.478+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:53:51.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:53:50 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:53:51.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:53:51 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:53:51.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:53:51 smithi115 ceph-mon[78593]: pgmap v2935: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:53:51.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:53:51 smithi115 conmon[90393]: 2022-05-14T00:53:51.459+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:53:52.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:53:51 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:53:52.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:53:51 smithi047 ceph-mon[77147]: pgmap v2935: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:53:52.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:53:52 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:53:52.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:53:52 smithi115 conmon[90393]: 2022-05-14T00:53:52.461+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:53:53.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:53:52 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:53:53.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:53:53 smithi115 conmon[90393]: 2022-05-14T00:53:53.437+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:53:53.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:53:53 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:53:53.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:53:53 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:53:53.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:53:53 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:53:53.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:53:53 smithi115 ceph-mon[78593]: pgmap v2936: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:53:54.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:53:53 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:53:54.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:53:53 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:53:54.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:53:53 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:53:54.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:53:53 smithi047 ceph-mon[77147]: pgmap v2936: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:53:54.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:53:54 smithi115 conmon[90393]: 2022-05-14T00:53:54.445+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:53:54.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:53:54 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:53:54.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:53:54 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 5165 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:53:55.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:53:54 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:53:55.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:53:54 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 5165 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:53:55.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:53:55 smithi115 conmon[90393]: 2022-05-14T00:53:55.417+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:53:55.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:53:55 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:53:55.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:53:55 smithi115 ceph-mon[78593]: pgmap v2937: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:53:56.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:53:55 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:53:56.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:53:55 smithi047 ceph-mon[77147]: pgmap v2937: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:53:56.664 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:53:56 smithi115 conmon[90393]: 2022-05-14T00:53:56.403+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:53:57.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:53:56 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:53:57.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:53:56 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:53:57.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:53:57 smithi115 conmon[90393]: 2022-05-14T00:53:57.450+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:53:57.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:53:57 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:53:57.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:53:57 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:53:57.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:53:57 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:53:57.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:53:57 smithi115 ceph-mon[78593]: pgmap v2938: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:53:58.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:53:57 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:53:58.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:53:57 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:53:58.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:53:57 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:53:58.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:53:57 smithi047 ceph-mon[77147]: pgmap v2938: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:53:58.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:53:58 smithi115 conmon[90393]: 2022-05-14T00:53:58.443+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:53:58.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:53:58 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:53:59.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:53:58 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:53:59.681 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:53:59 smithi115 conmon[90393]: 2022-05-14T00:53:59.427+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:54:00.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:53:59 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:54:00.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:53:59 smithi047 ceph-mon[77147]: pgmap v2939: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:54:00.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:53:59 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 5170 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:54:00.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:53:59 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:54:00.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:53:59 smithi115 ceph-mon[78593]: pgmap v2939: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:54:00.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:53:59 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 5170 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:54:00.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:54:00 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:54:00.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:54:00 smithi115 conmon[90393]: 2022-05-14T00:54:00.472+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:54:01.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:54:00 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:54:01.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:54:01 smithi115 conmon[90393]: 2022-05-14T00:54:01.508+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:54:01.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:54:01 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:54:01.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:54:01 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:54:01.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:54:01 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:54:01.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:54:01 smithi115 ceph-mon[78593]: pgmap v2940: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:54:02.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:54:01 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:54:02.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:54:01 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:54:02.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:54:01 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:54:02.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:54:01 smithi047 ceph-mon[77147]: pgmap v2940: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:54:02.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:54:02 smithi115 conmon[90393]: 2022-05-14T00:54:02.476+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:54:02.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:54:02 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:54:03.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:54:02 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:54:03.698 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:54:03 smithi115 conmon[90393]: 2022-05-14T00:54:03.444+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:54:04.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:54:03 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:54:04.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:54:03 smithi047 ceph-mon[77147]: pgmap v2941: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:54:04.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:54:03 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:54:04.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:54:03 smithi115 ceph-mon[78593]: pgmap v2941: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:54:04.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:54:04 smithi115 conmon[90393]: 2022-05-14T00:54:04.487+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:54:04.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:54:04 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:54:04.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:54:04 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 5175 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:54:05.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:54:04 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:54:05.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:54:04 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 5175 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:54:05.822 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:54:05 smithi115 conmon[90393]: 2022-05-14T00:54:05.490+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:54:05.823 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:54:05 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:54:05.823 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:54:05 smithi115 ceph-mon[78593]: pgmap v2942: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:54:06.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:54:05 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:54:06.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:54:05 smithi047 ceph-mon[77147]: pgmap v2942: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:54:06.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:54:06 smithi115 conmon[90393]: 2022-05-14T00:54:06.507+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:54:07.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:54:06 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:54:07.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:54:06 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T00:54:07.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:54:06 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T00:54:07.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:54:06 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T00:54:07.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:54:06 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:54:07.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:54:06 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T00:54:07.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:54:06 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T00:54:07.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:54:06 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T00:54:07.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:54:07 smithi115 conmon[90393]: 2022-05-14T00:54:07.501+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:54:07.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:54:07 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:54:07.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:54:07 smithi115 ceph-mon[78593]: pgmap v2943: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:54:08.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:54:07 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:54:08.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:54:07 smithi047 ceph-mon[77147]: pgmap v2943: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:54:08.713 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:54:08 smithi115 conmon[90393]: 2022-05-14T00:54:08.454+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:54:09.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:54:08 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:54:09.041 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:54:08 smithi047 conmon[78521]: 2022-05-14T00:54:08.880+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:54:09.041 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:54:08 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T00:54:09.042 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:54:08 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:54:09.042 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:54:08 smithi047 conmon[78521]: yield 2022-05-14T00:54:09.042 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:54:08 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:54:09.042 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:54:08 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:54:09.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:54:08 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:54:09.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:54:08 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T00:54:09.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:54:08 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:54:09.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:54:08 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T00:54:09.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:54:08 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:54:09.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:54:08 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T00:54:09.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:54:08 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:54:09.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:54:08 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T00:54:09.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:54:08 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:54:09.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:54:08 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T00:54:09.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:54:08 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:54:09.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:54:08 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:54:09.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:54:08 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:54:09.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:54:08 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:54:09.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:54:09 smithi115 conmon[90393]: 2022-05-14T00:54:09.496+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:54:09.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:54:09 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:54:09.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:54:09 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:54:09.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:54:09 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:54:09.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:54:09 smithi115 ceph-mon[78593]: pgmap v2944: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:54:09.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:54:09 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:54:09.859 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T00:54:09.859 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:54:09.859 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T00:54:09.860 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:54:09.860 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:54:09.860 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:54:09.860 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T00:54:09.860 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:54:09.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T00:54:09.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:54:09.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T00:54:09.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:54:09.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T00:54:09.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:54:09.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T00:54:09.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:54:09.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:54:09.863 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:54:09.863 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:54:09 smithi115 ceph-mon[78593]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T00:54:09.863 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:54:09 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 5180 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:54:10.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:54:09 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:54:10.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:54:09 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:54:10.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:54:09 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:54:10.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:54:09 smithi047 ceph-mon[77147]: pgmap v2944: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:54:10.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:54:09 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:54:10.042 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T00:54:10.042 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:54:10.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T00:54:10.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:54:10.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:54:10.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:54:10.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T00:54:10.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:54:10.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T00:54:10.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:54:10.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T00:54:10.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:54:10.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T00:54:10.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:54:10.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T00:54:10.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:54:10.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:54:10.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:54:10.046 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:54:09 smithi047 ceph-mon[77147]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T00:54:10.046 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:54:09 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 5180 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:54:10.723 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:54:10 smithi115 conmon[90393]: 2022-05-14T00:54:10.459+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:54:11.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:54:10 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:54:11.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:54:10 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:54:11.729 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:54:11 smithi115 conmon[90393]: 2022-05-14T00:54:11.473+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:54:12.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:54:11 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:54:12.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:54:11 smithi047 ceph-mon[77147]: pgmap v2945: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:54:12.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:54:11 smithi047 conmon[78521]: 2022-05-14T00:54:11.951+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:54:12.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:54:11 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T00:54:12.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:54:11 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:54:12.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:54:11 smithi047 conmon[78521]: yield 2022-05-14T00:54:12.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:54:11 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:54:12.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:54:11 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:54:12.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:54:11 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:54:12.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:54:11 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T00:54:12.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:54:11 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:54:12.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:54:11 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T00:54:12.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:54:11 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:54:12.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:54:11 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T00:54:12.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:54:11 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:54:12.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:54:11 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T00:54:12.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:54:11 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:54:12.048 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:54:11 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T00:54:12.048 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:54:11 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:54:12.048 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:54:11 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:54:12.048 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:54:11 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:54:12.048 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:54:11 smithi047 conmon[78521]: 2022-05-14T00:54:11.952+0000 7f5340f7a700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T00:54:12.049 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:54:11 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T00:54:12.049 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:54:11 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:54:12.049 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:54:11 smithi047 conmon[78521]: yield 2022-05-14T00:54:12.049 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:54:11 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:54:12.050 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:54:11 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:54:12.050 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:54:11 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:54:12.050 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:54:11 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T00:54:12.050 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:54:11 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:54:12.051 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:54:11 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T00:54:12.051 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:54:11 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:54:12.051 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:54:11 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T00:54:12.051 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:54:11 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:54:12.051 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:54:11 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T00:54:12.052 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:54:11 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:54:12.052 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:54:11 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T00:54:12.052 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:54:11 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:54:12.052 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:54:11 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:54:12.053 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:54:11 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:54:12.053 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:54:11 smithi047 conmon[78521]: 2022-05-14T00:54:12.053 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:54:11 smithi047 conmon[78521]: During handling of the above exception, another exception occurred: 2022-05-14T00:54:12.053 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:54:11 smithi047 conmon[78521]: 2022-05-14T00:54:12.054 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:54:11 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T00:54:12.054 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:54:11 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T00:54:12.054 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:54:11 smithi047 conmon[78521]: return f(*arg) 2022-05-14T00:54:12.054 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:54:11 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T00:54:12.055 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:54:11 smithi047 conmon[78521]: self._write_client_files(client_files, host) 2022-05-14T00:54:12.055 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:54:11 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T00:54:12.055 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:54:11 smithi047 conmon[78521]: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T00:54:12.055 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:54:11 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T00:54:12.056 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:54:11 smithi047 conmon[78521]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T00:54:12.056 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:54:11 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T00:54:12.056 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:54:11 smithi047 conmon[78521]: return self.event_loop.get_result(coro) 2022-05-14T00:54:12.057 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:54:11 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T00:54:12.057 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:54:11 smithi047 conmon[78521]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T00:54:12.058 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:54:11 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T00:54:12.058 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:54:11 smithi047 conmon[78521]: return self.__get_result() 2022-05-14T00:54:12.059 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:54:11 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T00:54:12.059 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:54:11 smithi047 conmon[78521]: raise self._exception 2022-05-14T00:54:12.059 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:54:11 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T00:54:12.059 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:54:11 smithi047 conmon[78521]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T00:54:12.060 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:54:11 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T00:54:12.060 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:54:11 smithi047 conmon[78521]: conn = await self._remote_connection(host, addr) 2022-05-14T00:54:12.060 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:54:11 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T00:54:12.060 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:54:11 smithi047 conmon[78521]: raise 2022-05-14T00:54:12.060 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:54:11 smithi047 conmon[78521]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T00:54:12.061 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:54:11 smithi047 conmon[78521]: self.gen.throw(type, value, traceback) 2022-05-14T00:54:12.061 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:54:11 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T00:54:12.061 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:54:11 smithi047 conmon[78521]: raise OrchestratorError(msg) 2022-05-14T00:54:12.061 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:54:11 smithi047 conmon[78521]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:54:12.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:54:11 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:54:12.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:54:11 smithi115 ceph-mon[78593]: pgmap v2945: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:54:12.730 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:54:12 smithi115 conmon[90393]: 2022-05-14T00:54:12.459+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:54:13.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:54:12 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:54:13.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:54:12 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:54:13.041 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T00:54:13.042 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:54:13.042 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T00:54:13.042 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:54:13.042 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:54:13.042 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:54:13.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T00:54:13.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:54:13.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T00:54:13.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:54:13.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T00:54:13.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:54:13.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T00:54:13.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:54:13.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T00:54:13.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:54:13.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:54:13.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:54:13.045 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:54:12 smithi047 ceph-mon[77147]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T00:54:13.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T00:54:13.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:54:13.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T00:54:13.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:54:13.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:54:13.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:54:13.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T00:54:13.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:54:13.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T00:54:13.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:54:13.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T00:54:13.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:54:13.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T00:54:13.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:54:13.049 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T00:54:13.049 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:54:13.049 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:54:13.049 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:54:13.049 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T00:54:13.050 INFO:journalctl@ceph.mon.b.smithi047.stdout: During handling of the above exception, another exception occurred: 2022-05-14T00:54:13.050 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T00:54:13.050 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T00:54:13.051 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T00:54:13.051 INFO:journalctl@ceph.mon.b.smithi047.stdout: return f(*arg) 2022-05-14T00:54:13.051 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T00:54:13.051 INFO:journalctl@ceph.mon.b.smithi047.stdout: self._write_client_files(client_files, host) 2022-05-14T00:54:13.051 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T00:54:13.052 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T00:54:13.052 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T00:54:13.052 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T00:54:13.052 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T00:54:13.053 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.event_loop.get_result(coro) 2022-05-14T00:54:13.053 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T00:54:13.053 INFO:journalctl@ceph.mon.b.smithi047.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T00:54:13.053 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T00:54:13.054 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.__get_result() 2022-05-14T00:54:13.054 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T00:54:13.054 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise self._exception 2022-05-14T00:54:13.055 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T00:54:13.055 INFO:journalctl@ceph.mon.b.smithi047.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T00:54:13.055 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T00:54:13.055 INFO:journalctl@ceph.mon.b.smithi047.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T00:54:13.055 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T00:54:13.056 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise 2022-05-14T00:54:13.057 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T00:54:13.058 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.gen.throw(type, value, traceback) 2022-05-14T00:54:13.058 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T00:54:13.058 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OrchestratorError(msg) 2022-05-14T00:54:13.058 INFO:journalctl@ceph.mon.b.smithi047.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:54:13.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:54:12 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:54:13.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:54:12 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:54:13.108 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T00:54:13.108 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:54:13.109 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T00:54:13.109 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:54:13.109 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:54:13.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:54:13.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T00:54:13.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:54:13.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T00:54:13.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:54:13.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T00:54:13.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:54:13.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T00:54:13.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:54:13.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T00:54:13.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:54:13.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:54:13.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:54:13.113 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:54:12 smithi115 ceph-mon[78593]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T00:54:13.113 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T00:54:13.113 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:54:13.113 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T00:54:13.113 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:54:13.114 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:54:13.114 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:54:13.114 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T00:54:13.114 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:54:13.115 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T00:54:13.115 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:54:13.115 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T00:54:13.115 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:54:13.116 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T00:54:13.116 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:54:13.116 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T00:54:13.116 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:54:13.117 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:54:13.117 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:54:13.117 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T00:54:13.117 INFO:journalctl@ceph.mon.c.smithi115.stdout: During handling of the above exception, another exception occurred: 2022-05-14T00:54:13.117 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T00:54:13.118 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T00:54:13.118 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T00:54:13.118 INFO:journalctl@ceph.mon.c.smithi115.stdout: return f(*arg) 2022-05-14T00:54:13.119 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T00:54:13.120 INFO:journalctl@ceph.mon.c.smithi115.stdout: self._write_client_files(client_files, host) 2022-05-14T00:54:13.120 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T00:54:13.120 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T00:54:13.120 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T00:54:13.121 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T00:54:13.121 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T00:54:13.121 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.event_loop.get_result(coro) 2022-05-14T00:54:13.121 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T00:54:13.121 INFO:journalctl@ceph.mon.c.smithi115.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T00:54:13.122 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T00:54:13.122 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.__get_result() 2022-05-14T00:54:13.122 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T00:54:13.122 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise self._exception 2022-05-14T00:54:13.123 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T00:54:13.123 INFO:journalctl@ceph.mon.c.smithi115.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T00:54:13.123 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T00:54:13.123 INFO:journalctl@ceph.mon.c.smithi115.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T00:54:13.123 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T00:54:13.124 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise 2022-05-14T00:54:13.124 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T00:54:13.124 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.gen.throw(type, value, traceback) 2022-05-14T00:54:13.124 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T00:54:13.124 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OrchestratorError(msg) 2022-05-14T00:54:13.125 INFO:journalctl@ceph.mon.c.smithi115.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:54:13.740 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:54:13 smithi115 conmon[90393]: 2022-05-14T00:54:13.462+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:54:14.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:54:13 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:54:14.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:54:13 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:54:14.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:54:13 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:54:14.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:54:13 smithi047 ceph-mon[77147]: pgmap v2946: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:54:14.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:54:13 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T00:54:14.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:54:13 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T00:54:14.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:54:13 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:54:14.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:54:13 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:54:14.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:54:13 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:54:14.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:54:13 smithi115 ceph-mon[78593]: pgmap v2946: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:54:14.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:54:13 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T00:54:14.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:54:13 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T00:54:14.741 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:54:14 smithi115 conmon[90393]: 2022-05-14T00:54:14.455+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:54:15.045 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:54:14 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:54:15.045 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:54:14 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 5185 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:54:15.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:54:14 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:54:15.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:54:14 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 5185 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:54:15.747 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:54:15 smithi115 conmon[90393]: 2022-05-14T00:54:15.412+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:54:16.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:54:15 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:54:16.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:54:15 smithi047 ceph-mon[77147]: pgmap v2947: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:54:16.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:54:15 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:54:16.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:54:15 smithi115 ceph-mon[78593]: pgmap v2947: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:54:16.748 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:54:16 smithi115 conmon[90393]: 2022-05-14T00:54:16.444+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:54:17.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:54:16 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:54:17.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:54:16 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:54:17.759 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:54:17 smithi115 conmon[90393]: 2022-05-14T00:54:17.414+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:54:18.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:54:17 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:54:18.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:54:17 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:54:18.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:54:17 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:54:18.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:54:17 smithi047 ceph-mon[77147]: pgmap v2948: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:54:18.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:54:17 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:54:18.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:54:17 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:54:18.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:54:17 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:54:18.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:54:17 smithi115 ceph-mon[78593]: pgmap v2948: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:54:18.758 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:54:18 smithi115 conmon[90393]: 2022-05-14T00:54:18.382+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:54:19.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:54:18 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:54:19.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:54:18 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:54:19.767 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:54:19 smithi115 conmon[90393]: 2022-05-14T00:54:19.404+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:54:20.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:54:19 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:54:20.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:54:19 smithi047 ceph-mon[77147]: pgmap v2949: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:54:20.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:54:19 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 5190 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:54:20.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:54:19 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:54:20.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:54:19 smithi115 ceph-mon[78593]: pgmap v2949: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:54:20.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:54:19 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 5190 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:54:20.773 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:54:20 smithi115 conmon[90393]: 2022-05-14T00:54:20.391+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:54:21.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:54:20 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:54:21.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:54:20 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:54:21.776 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:54:21 smithi115 conmon[90393]: 2022-05-14T00:54:21.417+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:54:22.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:54:21 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:54:22.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:54:21 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:54:22.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:54:21 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:54:22.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:54:21 smithi047 ceph-mon[77147]: pgmap v2950: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:54:22.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:54:21 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:54:22.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:54:21 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:54:22.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:54:21 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:54:22.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:54:21 smithi115 ceph-mon[78593]: pgmap v2950: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:54:22.778 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:54:22 smithi115 conmon[90393]: 2022-05-14T00:54:22.389+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:54:23.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:54:22 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:54:23.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:54:22 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:54:23.782 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:54:23 smithi115 conmon[90393]: 2022-05-14T00:54:23.360+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:54:24.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:54:23 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:54:24.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:54:23 smithi047 ceph-mon[77147]: pgmap v2951: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:54:24.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:54:23 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:54:24.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:54:23 smithi115 ceph-mon[78593]: pgmap v2951: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:54:24.785 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:54:24 smithi115 conmon[90393]: 2022-05-14T00:54:24.405+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:54:25.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:54:24 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:54:25.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:54:24 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 5195 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:54:25.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:54:24 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:54:25.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:54:24 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 5195 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:54:25.790 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:54:25 smithi115 conmon[90393]: 2022-05-14T00:54:25.441+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:54:26.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:54:25 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:54:26.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:54:25 smithi047 ceph-mon[77147]: pgmap v2952: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:54:26.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:54:25 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:54:26.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:54:25 smithi115 ceph-mon[78593]: pgmap v2952: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:54:26.791 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:54:26 smithi115 conmon[90393]: 2022-05-14T00:54:26.425+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:54:27.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:54:26 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:54:27.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:54:26 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:54:27.798 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:54:27 smithi115 conmon[90393]: 2022-05-14T00:54:27.453+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:54:28.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:54:27 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:54:28.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:54:27 smithi115 ceph-mon[78593]: pgmap v2953: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:54:28.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:54:27 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:54:28.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:54:27 smithi047 ceph-mon[77147]: pgmap v2953: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:54:28.798 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:54:28 smithi115 conmon[90393]: 2022-05-14T00:54:28.437+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:54:29.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:54:28 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:54:29.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:54:28 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:54:29.813 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:54:29 smithi115 conmon[90393]: 2022-05-14T00:54:29.391+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:54:30.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:54:29 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:54:30.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:54:29 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:54:30.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:54:29 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:54:30.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:54:29 smithi115 ceph-mon[78593]: pgmap v2954: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:54:30.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:54:29 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 5200 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:54:30.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:54:29 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:54:30.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:54:29 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:54:30.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:54:29 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:54:30.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:54:29 smithi047 ceph-mon[77147]: pgmap v2954: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:54:30.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:54:29 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 5200 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:54:30.812 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:54:30 smithi115 conmon[90393]: 2022-05-14T00:54:30.358+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:54:31.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:54:30 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:54:31.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:54:30 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:54:31.820 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:54:31 smithi115 conmon[90393]: 2022-05-14T00:54:31.357+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:54:32.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:54:31 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:54:32.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:54:31 smithi115 ceph-mon[78593]: pgmap v2955: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:54:32.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:54:31 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:54:32.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:54:31 smithi047 ceph-mon[77147]: pgmap v2955: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:54:32.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:54:32 smithi115 conmon[90393]: 2022-05-14T00:54:32.343+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:54:33.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:54:32 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:54:33.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:54:32 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:54:33.827 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:54:33 smithi115 conmon[90393]: 2022-05-14T00:54:33.368+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:54:34.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:54:33 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:54:34.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:54:33 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:54:34.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:54:33 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:54:34.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:54:33 smithi115 ceph-mon[78593]: pgmap v2956: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:54:34.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:54:33 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:54:34.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:54:33 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:54:34.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:54:33 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:54:34.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:54:33 smithi047 ceph-mon[77147]: pgmap v2956: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:54:34.834 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:54:34 smithi115 conmon[90393]: 2022-05-14T00:54:34.399+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:54:35.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:54:34 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:54:35.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:54:34 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 5205 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:54:35.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:54:34 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:54:35.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:54:34 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 5205 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:54:35.836 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:54:35 smithi115 conmon[90393]: 2022-05-14T00:54:35.357+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:54:36.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:54:35 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:54:36.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:54:35 smithi115 ceph-mon[78593]: pgmap v2957: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:54:36.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:54:35 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:54:36.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:54:35 smithi047 ceph-mon[77147]: pgmap v2957: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:54:36.841 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:54:36 smithi115 conmon[90393]: 2022-05-14T00:54:36.406+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:54:37.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:54:36 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:54:37.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:54:36 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:54:37.843 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:54:37 smithi115 conmon[90393]: 2022-05-14T00:54:37.408+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:54:38.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:54:37 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:54:38.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:54:37 smithi115 ceph-mon[78593]: pgmap v2958: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:54:38.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:54:37 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:54:38.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:54:37 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:54:38.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:54:37 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:54:38.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:54:37 smithi047 ceph-mon[77147]: pgmap v2958: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:54:38.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:54:37 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:54:38.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:54:37 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:54:38.849 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:54:38 smithi115 conmon[90393]: 2022-05-14T00:54:38.445+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:54:39.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:54:38 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:54:39.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:54:38 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:54:39.849 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:54:39 smithi115 conmon[90393]: 2022-05-14T00:54:39.463+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:54:40.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:54:39 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:54:40.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:54:39 smithi115 ceph-mon[78593]: pgmap v2959: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:54:40.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:54:39 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 5210 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:54:40.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:54:39 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:54:40.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:54:39 smithi047 ceph-mon[77147]: pgmap v2959: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:54:40.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:54:39 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 5210 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:54:40.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:54:40 smithi115 conmon[90393]: 2022-05-14T00:54:40.440+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:54:41.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:54:40 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:54:41.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:54:40 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:54:41.856 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:54:41 smithi115 conmon[90393]: 2022-05-14T00:54:41.421+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:54:42.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:54:41 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:54:42.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:54:41 smithi115 ceph-mon[78593]: pgmap v2960: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:54:42.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:54:41 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:54:42.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:54:41 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:54:42.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:54:41 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:54:42.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:54:41 smithi047 ceph-mon[77147]: pgmap v2960: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:54:42.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:54:41 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:54:42.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:54:41 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:54:42.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:54:42 smithi115 conmon[90393]: 2022-05-14T00:54:42.404+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:54:43.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:54:42 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:54:43.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:54:42 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:54:43.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:54:43 smithi115 conmon[90393]: 2022-05-14T00:54:43.386+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:54:44.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:54:43 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:54:44.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:54:43 smithi047 ceph-mon[77147]: pgmap v2961: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:54:44.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:54:43 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:54:44.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:54:43 smithi115 ceph-mon[78593]: pgmap v2961: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:54:44.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:54:44 smithi115 conmon[90393]: 2022-05-14T00:54:44.380+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:54:45.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:54:44 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:54:45.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:54:44 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 5215 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:54:45.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:54:44 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:54:45.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:54:44 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 5215 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:54:45.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:54:45 smithi115 conmon[90393]: 2022-05-14T00:54:45.421+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:54:46.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:54:45 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:54:46.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:54:45 smithi047 ceph-mon[77147]: pgmap v2962: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:54:46.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:54:45 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:54:46.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:54:45 smithi115 ceph-mon[78593]: pgmap v2962: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:54:46.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:54:46 smithi115 conmon[90393]: 2022-05-14T00:54:46.402+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:54:47.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:54:46 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:54:47.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:54:46 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:54:47.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:54:47 smithi115 conmon[90393]: 2022-05-14T00:54:47.367+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:54:48.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:54:47 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:54:48.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:54:47 smithi047 ceph-mon[77147]: pgmap v2963: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:54:48.349 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:54:47 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:54:48.350 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:54:47 smithi115 ceph-mon[78593]: pgmap v2963: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:54:48.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:54:48 smithi115 conmon[90393]: 2022-05-14T00:54:48.348+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:54:49.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:54:48 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:54:49.342 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:54:48 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:54:49.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:54:49 smithi115 conmon[90393]: 2022-05-14T00:54:49.341+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:54:50.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:54:49 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:54:50.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:54:49 smithi047 ceph-mon[77147]: pgmap v2964: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:54:50.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:54:49 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:54:50.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:54:49 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:54:50.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:54:49 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 5220 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:54:50.311 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:54:49 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:54:50.312 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:54:49 smithi115 ceph-mon[78593]: pgmap v2964: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:54:50.312 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:54:49 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:54:50.312 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:54:49 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:54:50.312 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:54:49 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 5220 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:54:50.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:54:50 smithi115 conmon[90393]: 2022-05-14T00:54:50.310+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:54:51.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:54:50 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:54:51.329 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:54:50 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:54:51.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:54:51 smithi115 conmon[90393]: 2022-05-14T00:54:51.328+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:54:52.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:54:51 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:54:52.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:54:51 smithi047 ceph-mon[77147]: pgmap v2965: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:54:52.339 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:54:51 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:54:52.340 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:54:51 smithi115 ceph-mon[78593]: pgmap v2965: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:54:52.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:54:52 smithi115 conmon[90393]: 2022-05-14T00:54:52.338+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:54:53.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:54:52 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:54:53.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:54:52 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:54:53.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:54:53 smithi115 conmon[90393]: 2022-05-14T00:54:53.294+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:54:54.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:54:53 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:54:54.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:54:53 smithi047 ceph-mon[77147]: pgmap v2966: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:54:54.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:54:53 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:54:54.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:54:53 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:54:54.302 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:54:53 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:54:54.302 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:54:53 smithi115 ceph-mon[78593]: pgmap v2966: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:54:54.302 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:54:53 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:54:54.303 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:54:53 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:54:54.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:54:54 smithi115 conmon[90393]: 2022-05-14T00:54:54.300+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:54:55.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:54:54 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:54:55.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:54:54 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 5225 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:54:55.293 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:54:54 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:54:55.294 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:54:54 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 5225 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:54:55.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:54:55 smithi115 conmon[90393]: 2022-05-14T00:54:55.293+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:54:56.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:54:55 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:54:56.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:54:55 smithi047 ceph-mon[77147]: pgmap v2967: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:54:56.338 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:54:55 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:54:56.338 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:54:55 smithi115 ceph-mon[78593]: pgmap v2967: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:54:56.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:54:56 smithi115 conmon[90393]: 2022-05-14T00:54:56.336+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:54:57.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:54:56 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:54:57.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:54:56 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:54:57.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:54:57 smithi115 conmon[90393]: 2022-05-14T00:54:57.382+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:54:58.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:54:57 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:54:58.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:54:57 smithi047 ceph-mon[77147]: pgmap v2968: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:54:58.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:54:57 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:54:58.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:54:57 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:54:58.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:54:57 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:54:58.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:54:57 smithi115 ceph-mon[78593]: pgmap v2968: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:54:58.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:54:57 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:54:58.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:54:57 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:54:58.863 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:54:58 smithi115 conmon[90393]: 2022-05-14T00:54:58.373+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:54:59.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:54:58 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:54:59.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:54:58 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:54:59.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:54:59 smithi115 conmon[90393]: 2022-05-14T00:54:59.420+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:55:00.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:54:59 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:55:00.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:54:59 smithi047 ceph-mon[77147]: pgmap v2969: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:55:00.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:54:59 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 5230 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:55:00.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:54:59 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:55:00.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:54:59 smithi115 ceph-mon[78593]: pgmap v2969: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:55:00.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:54:59 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 5230 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:55:00.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:55:00 smithi115 conmon[90393]: 2022-05-14T00:55:00.414+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:55:01.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:55:00 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:55:01.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:55:00 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:55:01.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:55:01 smithi115 conmon[90393]: 2022-05-14T00:55:01.419+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:55:02.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:55:01 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:55:02.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:55:01 smithi047 ceph-mon[77147]: pgmap v2970: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:55:02.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:55:01 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:55:02.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:55:01 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:55:02.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:55:01 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:55:02.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:55:01 smithi115 ceph-mon[78593]: pgmap v2970: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:55:02.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:55:01 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:55:02.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:55:01 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:55:02.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:55:02 smithi115 conmon[90393]: 2022-05-14T00:55:02.464+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:55:03.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:55:02 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:55:03.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:55:02 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:55:03.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:55:03 smithi115 conmon[90393]: 2022-05-14T00:55:03.435+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:55:04.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:55:03 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:55:04.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:55:03 smithi047 ceph-mon[77147]: pgmap v2971: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:55:04.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:55:03 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:55:04.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:55:03 smithi115 ceph-mon[78593]: pgmap v2971: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:55:04.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:55:04 smithi115 conmon[90393]: 2022-05-14T00:55:04.395+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:55:05.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:55:04 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:55:05.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:55:04 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 5235 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:55:05.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:55:04 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:55:05.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:55:04 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 5235 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:55:05.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:55:05 smithi115 conmon[90393]: 2022-05-14T00:55:05.402+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:55:06.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:55:05 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:55:06.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:55:05 smithi047 ceph-mon[77147]: pgmap v2972: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:55:06.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:55:05 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:55:06.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:55:05 smithi115 ceph-mon[78593]: pgmap v2972: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:55:06.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:55:06 smithi115 conmon[90393]: 2022-05-14T00:55:06.447+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:55:07.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:55:06 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:55:07.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:55:06 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:55:07.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:55:07 smithi115 conmon[90393]: 2022-05-14T00:55:07.455+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:55:08.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:55:07 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:55:08.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:55:07 smithi047 ceph-mon[77147]: pgmap v2973: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:55:08.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:55:07 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:55:08.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:55:07 smithi115 ceph-mon[78593]: pgmap v2973: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:55:08.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:55:08 smithi115 conmon[90393]: 2022-05-14T00:55:08.408+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:55:09.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:55:08 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:55:09.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:55:08 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:55:09.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:55:09 smithi115 conmon[90393]: 2022-05-14T00:55:09.440+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:55:10.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:55:09 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:55:10.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:55:09 smithi047 ceph-mon[77147]: pgmap v2974: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:55:10.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:55:09 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:55:10.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:55:09 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:55:10.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:55:09 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 5240 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:55:10.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:55:09 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:55:10.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:55:09 smithi115 ceph-mon[78593]: pgmap v2974: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:55:10.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:55:09 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:55:10.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:55:09 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:55:10.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:55:09 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 5240 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:55:10.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:55:10 smithi115 conmon[90393]: 2022-05-14T00:55:10.416+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:55:11.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:55:10 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:55:11.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:55:10 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:55:11.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:55:11 smithi115 conmon[90393]: 2022-05-14T00:55:11.441+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:55:12.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:55:11 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:55:12.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:55:11 smithi047 ceph-mon[77147]: pgmap v2975: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:55:12.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:55:11 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:55:12.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:55:11 smithi115 ceph-mon[78593]: pgmap v2975: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:55:12.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:55:12 smithi115 conmon[90393]: 2022-05-14T00:55:12.471+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:55:13.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:55:12 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:55:13.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:55:12 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T00:55:13.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:55:12 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T00:55:13.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:55:12 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T00:55:13.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:55:12 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' 2022-05-14T00:55:13.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:55:12 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' 2022-05-14T00:55:13.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:55:12 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:55:13.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:55:12 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T00:55:13.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:55:12 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T00:55:13.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:55:12 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T00:55:13.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:55:12 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' 2022-05-14T00:55:13.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:55:12 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' 2022-05-14T00:55:13.862 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:55:13 smithi115 conmon[90393]: 2022-05-14T00:55:13.441+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:55:14.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:55:13 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:55:14.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:55:13 smithi047 ceph-mon[77147]: pgmap v2976: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:55:14.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:55:13 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:55:14.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:55:13 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:55:14.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:55:13 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T00:55:14.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:55:13 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T00:55:14.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:55:13 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:55:14.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:55:13 smithi115 ceph-mon[78593]: pgmap v2976: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:55:14.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:55:13 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:55:14.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:55:13 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:55:14.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:55:13 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T00:55:14.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:55:13 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T00:55:14.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:55:14 smithi115 conmon[90393]: 2022-05-14T00:55:14.453+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:55:15.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:55:14 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:55:15.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:55:14 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 5245 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:55:15.291 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:55:15 smithi047 conmon[78521]: 2022-05-14T00:55:15.056+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:55:15.292 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:55:15 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T00:55:15.292 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:55:15 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:55:15.293 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:55:15 smithi047 conmon[78521]: yield 2022-05-14T00:55:15.293 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:55:15 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:55:15.293 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:55:15 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:55:15.293 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:55:15 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:55:15.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:55:15 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T00:55:15.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:55:15 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:55:15.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:55:15 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T00:55:15.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:55:15 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:55:15.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:55:15 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T00:55:15.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:55:15 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:55:15.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:55:15 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T00:55:15.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:55:15 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:55:15.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:55:15 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T00:55:15.297 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:55:15 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:55:15.298 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:55:15 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:55:15.298 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:55:15 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:55:15.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:55:14 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:55:15.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:55:14 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 5245 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:55:15.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:55:15 smithi115 conmon[90393]: 2022-05-14T00:55:15.462+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:55:16.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:55:15 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:55:16.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:55:15 smithi047 ceph-mon[77147]: pgmap v2977: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:55:16.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:55:15 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' 2022-05-14T00:55:16.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:55:15 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' 2022-05-14T00:55:16.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:55:15 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:55:16.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T00:55:16.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:55:16.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T00:55:16.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:55:16.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:55:16.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:55:16.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T00:55:16.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:55:16.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T00:55:16.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:55:16.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T00:55:16.298 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:55:16.298 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T00:55:16.299 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:55:16.299 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T00:55:16.300 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:55:16.300 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:55:16.300 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:55:16.301 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:55:15 smithi047 ceph-mon[77147]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T00:55:16.301 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:55:15 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-05-14T00:55:16.302 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:55:15 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-05-14T00:55:16.302 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:55:15 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-05-14T00:55:16.303 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:55:15 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-05-14T00:55:16.303 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:55:15 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-05-14T00:55:16.303 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:55:15 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-05-14T00:55:16.304 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:55:15 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-05-14T00:55:16.304 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:55:15 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-05-14T00:55:16.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:55:15 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:55:16.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:55:15 smithi115 ceph-mon[78593]: pgmap v2977: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:55:16.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:55:15 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' 2022-05-14T00:55:16.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:55:15 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' 2022-05-14T00:55:16.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:55:15 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:55:16.359 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T00:55:16.359 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:55:16.359 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T00:55:16.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:55:16.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:55:16.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:55:16.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T00:55:16.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:55:16.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T00:55:16.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:55:16.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T00:55:16.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:55:16.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T00:55:16.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:55:16.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T00:55:16.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:55:16.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:55:16.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:55:16.364 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:55:15 smithi115 ceph-mon[78593]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T00:55:16.364 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:55:15 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-05-14T00:55:16.364 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:55:15 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-05-14T00:55:16.364 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:55:15 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-05-14T00:55:16.365 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:55:15 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-05-14T00:55:16.365 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:55:15 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-05-14T00:55:16.366 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:55:15 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-05-14T00:55:16.367 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:55:15 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-05-14T00:55:16.367 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:55:15 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-05-14T00:55:16.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:55:16 smithi115 conmon[90393]: 2022-05-14T00:55:16.416+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:55:17.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:55:16 smithi047 ceph-mon[77147]: Adjusting osd_memory_target on smithi115 to 197.0M 2022-05-14T00:55:17.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:55:16 smithi047 ceph-mon[77147]: Unable to set osd_memory_target on smithi115 to 206629785: error parsing value: Value '206629785' is below minimum 939524096 2022-05-14T00:55:17.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:55:16 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:55:17.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:55:16 smithi047 ceph-mon[77147]: Adjusting osd_memory_target on smithi047 to 198.1M 2022-05-14T00:55:17.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:55:16 smithi047 ceph-mon[77147]: Unable to set osd_memory_target on smithi047 to 207755878: error parsing value: Value '207755878' is below minimum 939524096 2022-05-14T00:55:17.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:55:16 smithi115 ceph-mon[78593]: Adjusting osd_memory_target on smithi115 to 197.0M 2022-05-14T00:55:17.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:55:16 smithi115 ceph-mon[78593]: Unable to set osd_memory_target on smithi115 to 206629785: error parsing value: Value '206629785' is below minimum 939524096 2022-05-14T00:55:17.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:55:16 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:55:17.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:55:16 smithi115 ceph-mon[78593]: Adjusting osd_memory_target on smithi047 to 198.1M 2022-05-14T00:55:17.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:55:16 smithi115 ceph-mon[78593]: Unable to set osd_memory_target on smithi047 to 207755878: error parsing value: Value '207755878' is below minimum 939524096 2022-05-14T00:55:17.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:55:17 smithi115 conmon[90393]: 2022-05-14T00:55:17.393+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:55:18.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:55:17 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:55:18.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:55:17 smithi047 ceph-mon[77147]: pgmap v2978: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:55:18.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:55:17 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:55:18.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:55:17 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:55:18.292 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:55:18 smithi047 conmon[78521]: 2022-05-14T00:55:18.127+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:55:18.293 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:55:18 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T00:55:18.293 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:55:18 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:55:18.293 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:55:18 smithi047 conmon[78521]: yield 2022-05-14T00:55:18.293 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:55:18 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:55:18.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:55:18 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:55:18.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:55:18 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:55:18.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:55:18 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T00:55:18.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:55:18 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:55:18.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:55:18 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T00:55:18.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:55:18 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:55:18.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:55:18 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T00:55:18.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:55:18 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:55:18.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:55:18 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T00:55:18.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:55:18 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:55:18.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:55:18 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T00:55:18.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:55:18 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:55:18.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:55:18 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:55:18.297 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:55:18 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:55:18.297 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:55:18 smithi047 conmon[78521]: 2022-05-14T00:55:18.128+0000 7f5344781700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T00:55:18.297 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:55:18 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T00:55:18.298 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:55:18 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:55:18.298 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:55:18 smithi047 conmon[78521]: yield 2022-05-14T00:55:18.298 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:55:18 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:55:18.298 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:55:18 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:55:18.298 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:55:18 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:55:18.299 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:55:18 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T00:55:18.299 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:55:18 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:55:18.299 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:55:18 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T00:55:18.299 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:55:18 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:55:18.299 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:55:18 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T00:55:18.300 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:55:18 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:55:18.302 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:55:18 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T00:55:18.303 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:55:18 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:55:18.303 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:55:18 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T00:55:18.303 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:55:18 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:55:18.303 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:55:18 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:55:18.304 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:55:18 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:55:18.304 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:55:18 smithi047 conmon[78521]: 2022-05-14T00:55:18.304 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:55:18 smithi047 conmon[78521]: During handling of the above exception, another exception occurred: 2022-05-14T00:55:18.305 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:55:18 smithi047 conmon[78521]: 2022-05-14T00:55:18.305 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:55:18 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T00:55:18.305 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:55:18 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T00:55:18.306 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:55:18 smithi047 conmon[78521]: return f(*arg) 2022-05-14T00:55:18.306 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:55:18 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T00:55:18.306 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:55:18 smithi047 conmon[78521]: self._write_client_files(client_files, host) 2022-05-14T00:55:18.307 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:55:18 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T00:55:18.307 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:55:18 smithi047 conmon[78521]: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T00:55:18.308 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:55:18 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T00:55:18.308 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:55:18 smithi047 conmon[78521]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T00:55:18.309 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:55:18 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T00:55:18.309 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:55:18 smithi047 conmon[78521]: return self.event_loop.get_result(coro) 2022-05-14T00:55:18.310 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:55:18 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T00:55:18.310 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:55:18 smithi047 conmon[78521]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T00:55:18.311 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:55:18 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T00:55:18.311 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:55:18 smithi047 conmon[78521]: return self.__get_result() 2022-05-14T00:55:18.312 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:55:18 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T00:55:18.312 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:55:18 smithi047 conmon[78521]: raise self._exception 2022-05-14T00:55:18.313 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:55:18 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T00:55:18.313 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:55:18 smithi047 conmon[78521]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T00:55:18.313 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:55:18 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T00:55:18.314 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:55:18 smithi047 conmon[78521]: conn = await self._remote_connection(host, addr) 2022-05-14T00:55:18.314 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:55:18 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T00:55:18.315 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:55:18 smithi047 conmon[78521]: raise 2022-05-14T00:55:18.315 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:55:18 smithi047 conmon[78521]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T00:55:18.316 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:55:18 smithi047 conmon[78521]: self.gen.throw(type, value, traceback) 2022-05-14T00:55:18.316 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:55:18 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T00:55:18.316 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:55:18 smithi047 conmon[78521]: raise OrchestratorError(msg) 2022-05-14T00:55:18.317 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:55:18 smithi047 conmon[78521]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:55:18.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:55:17 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:55:18.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:55:17 smithi115 ceph-mon[78593]: pgmap v2978: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:55:18.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:55:17 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:55:18.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:55:17 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:55:18.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:55:18 smithi115 conmon[90393]: 2022-05-14T00:55:18.359+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:55:19.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:55:18 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:55:19.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:55:18 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:55:19.292 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T00:55:19.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:55:19.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T00:55:19.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:55:19.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:55:19.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:55:19.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T00:55:19.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:55:19.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T00:55:19.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:55:19.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T00:55:19.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:55:19.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T00:55:19.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:55:19.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T00:55:19.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:55:19.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:55:19.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:55:19.298 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:55:18 smithi047 ceph-mon[77147]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T00:55:19.298 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T00:55:19.298 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:55:19.298 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T00:55:19.299 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:55:19.299 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:55:19.299 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:55:19.299 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T00:55:19.299 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:55:19.300 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T00:55:19.300 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:55:19.301 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T00:55:19.301 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:55:19.301 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T00:55:19.301 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:55:19.301 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T00:55:19.302 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:55:19.302 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:55:19.302 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:55:19.302 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T00:55:19.303 INFO:journalctl@ceph.mon.b.smithi047.stdout: During handling of the above exception, another exception occurred: 2022-05-14T00:55:19.303 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T00:55:19.303 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T00:55:19.303 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T00:55:19.303 INFO:journalctl@ceph.mon.b.smithi047.stdout: return f(*arg) 2022-05-14T00:55:19.304 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T00:55:19.304 INFO:journalctl@ceph.mon.b.smithi047.stdout: self._write_client_files(client_files, host) 2022-05-14T00:55:19.304 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T00:55:19.304 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T00:55:19.305 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T00:55:19.305 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T00:55:19.305 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T00:55:19.305 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.event_loop.get_result(coro) 2022-05-14T00:55:19.305 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T00:55:19.306 INFO:journalctl@ceph.mon.b.smithi047.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T00:55:19.306 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T00:55:19.306 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.__get_result() 2022-05-14T00:55:19.306 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T00:55:19.307 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise self._exception 2022-05-14T00:55:19.307 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T00:55:19.307 INFO:journalctl@ceph.mon.b.smithi047.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T00:55:19.307 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T00:55:19.307 INFO:journalctl@ceph.mon.b.smithi047.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T00:55:19.308 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T00:55:19.308 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise 2022-05-14T00:55:19.308 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T00:55:19.308 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.gen.throw(type, value, traceback) 2022-05-14T00:55:19.309 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T00:55:19.309 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OrchestratorError(msg) 2022-05-14T00:55:19.309 INFO:journalctl@ceph.mon.b.smithi047.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:55:19.351 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:55:18 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:55:19.351 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:55:18 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:55:19.352 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T00:55:19.352 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:55:19.352 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T00:55:19.353 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:55:19.353 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:55:19.353 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:55:19.354 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T00:55:19.354 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:55:19.354 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T00:55:19.354 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:55:19.354 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T00:55:19.355 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:55:19.355 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T00:55:19.355 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:55:19.355 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T00:55:19.355 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:55:19.356 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:55:19.356 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:55:19.356 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:55:18 smithi115 ceph-mon[78593]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T00:55:19.356 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T00:55:19.357 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:55:19.357 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T00:55:19.357 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:55:19.357 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:55:19.358 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:55:19.358 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T00:55:19.358 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:55:19.358 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T00:55:19.358 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:55:19.359 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T00:55:19.359 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:55:19.359 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T00:55:19.359 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:55:19.359 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T00:55:19.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:55:19.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:55:19.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:55:19.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T00:55:19.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: During handling of the above exception, another exception occurred: 2022-05-14T00:55:19.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T00:55:19.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T00:55:19.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T00:55:19.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: return f(*arg) 2022-05-14T00:55:19.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T00:55:19.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: self._write_client_files(client_files, host) 2022-05-14T00:55:19.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T00:55:19.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T00:55:19.365 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T00:55:19.365 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T00:55:19.365 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T00:55:19.365 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.event_loop.get_result(coro) 2022-05-14T00:55:19.366 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T00:55:19.366 INFO:journalctl@ceph.mon.c.smithi115.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T00:55:19.366 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T00:55:19.366 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.__get_result() 2022-05-14T00:55:19.367 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T00:55:19.367 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise self._exception 2022-05-14T00:55:19.368 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T00:55:19.368 INFO:journalctl@ceph.mon.c.smithi115.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T00:55:19.368 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T00:55:19.369 INFO:journalctl@ceph.mon.c.smithi115.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T00:55:19.369 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T00:55:19.369 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise 2022-05-14T00:55:19.369 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T00:55:19.370 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.gen.throw(type, value, traceback) 2022-05-14T00:55:19.370 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T00:55:19.370 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OrchestratorError(msg) 2022-05-14T00:55:19.370 INFO:journalctl@ceph.mon.c.smithi115.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:55:19.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:55:19 smithi115 conmon[90393]: 2022-05-14T00:55:19.350+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:55:20.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:55:19 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:55:20.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:55:19 smithi047 ceph-mon[77147]: pgmap v2979: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:55:20.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:55:19 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 5250 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:55:20.322 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:55:19 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:55:20.322 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:55:19 smithi115 ceph-mon[78593]: pgmap v2979: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:55:20.323 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:55:19 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 5250 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:55:20.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:55:20 smithi115 conmon[90393]: 2022-05-14T00:55:20.322+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:55:21.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:55:20 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:55:21.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:55:20 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:55:21.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:55:21 smithi115 conmon[90393]: 2022-05-14T00:55:21.372+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:55:22.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:55:21 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:55:22.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:55:21 smithi047 ceph-mon[77147]: pgmap v2980: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:55:22.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:55:21 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:55:22.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:55:21 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:55:22.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:55:21 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:55:22.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:55:21 smithi115 ceph-mon[78593]: pgmap v2980: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:55:22.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:55:21 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:55:22.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:55:21 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:55:22.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:55:22 smithi115 conmon[90393]: 2022-05-14T00:55:22.415+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:55:23.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:55:22 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:55:23.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:55:22 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:55:23.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:55:23 smithi115 conmon[90393]: 2022-05-14T00:55:23.431+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:55:24.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:55:23 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:55:24.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:55:23 smithi047 ceph-mon[77147]: pgmap v2981: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:55:24.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:55:23 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:55:24.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:55:23 smithi115 ceph-mon[78593]: pgmap v2981: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:55:24.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:55:24 smithi115 conmon[90393]: 2022-05-14T00:55:24.399+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:55:25.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:55:24 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:55:25.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:55:24 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 5255 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:55:25.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:55:24 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:55:25.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:55:24 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 5255 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:55:25.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:55:25 smithi115 conmon[90393]: 2022-05-14T00:55:25.395+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:55:26.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:55:26 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:55:26.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:55:26 smithi047 ceph-mon[77147]: pgmap v2982: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:55:26.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:55:26 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:55:26.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:55:26 smithi115 ceph-mon[78593]: pgmap v2982: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:55:26.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:55:26 smithi115 conmon[90393]: 2022-05-14T00:55:26.406+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:55:27.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:55:27 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:55:27.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:55:27 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:55:27.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:55:27 smithi115 conmon[90393]: 2022-05-14T00:55:27.428+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:55:28.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:55:28 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:55:28.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:55:28 smithi047 ceph-mon[77147]: pgmap v2983: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:55:28.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:55:28 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:55:28.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:55:28 smithi115 ceph-mon[78593]: pgmap v2983: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:55:28.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:55:28 smithi115 conmon[90393]: 2022-05-14T00:55:28.404+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:55:29.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:55:29 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:55:29.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:55:29 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:55:29.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:55:29 smithi115 conmon[90393]: 2022-05-14T00:55:29.415+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:55:30.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:55:30 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:55:30.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:55:30 smithi047 ceph-mon[77147]: pgmap v2984: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:55:30.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:55:30 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:55:30.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:55:30 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:55:30.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:55:30 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 5260 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:55:30.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:55:30 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:55:30.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:55:30 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:55:30.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:55:30 smithi115 ceph-mon[78593]: pgmap v2984: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:55:30.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:55:30 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:55:30.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:55:30 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:55:30.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:55:30 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 5260 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:55:30.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:55:30 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:55:30.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:55:30 smithi115 conmon[90393]: 2022-05-14T00:55:30.440+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:55:31.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:55:31 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:55:31.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:55:31 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:55:31.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:55:31 smithi115 conmon[90393]: 2022-05-14T00:55:31.483+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:55:32.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:55:32 smithi047 ceph-mon[77147]: pgmap v2985: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:55:32.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:55:32 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:55:32.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:55:32 smithi115 ceph-mon[78593]: pgmap v2985: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:55:32.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:55:32 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:55:32.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:55:32 smithi115 conmon[90393]: 2022-05-14T00:55:32.463+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:55:33.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:55:33 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:55:33.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:55:33 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:55:33.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:55:33 smithi115 conmon[90393]: 2022-05-14T00:55:33.455+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:55:34.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:55:34 smithi047 ceph-mon[77147]: pgmap v2986: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:55:34.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:55:34 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:55:34.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:55:34 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:55:34.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:55:34 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:55:34.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:55:34 smithi115 ceph-mon[78593]: pgmap v2986: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:55:34.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:55:34 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:55:34.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:55:34 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:55:34.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:55:34 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:55:34.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:55:34 smithi115 conmon[90393]: 2022-05-14T00:55:34.502+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:55:35.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:55:35 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 5265 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:55:35.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:55:35 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:55:35.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:55:35 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 5265 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:55:35.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:55:35 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:55:35.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:55:35 smithi115 conmon[90393]: 2022-05-14T00:55:35.518+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:55:36.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:55:36 smithi047 ceph-mon[77147]: pgmap v2987: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:55:36.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:55:36 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:55:36.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:55:36 smithi115 ceph-mon[78593]: pgmap v2987: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:55:36.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:55:36 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:55:36.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:55:36 smithi115 conmon[90393]: 2022-05-14T00:55:36.553+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:55:37.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:55:37 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:55:37.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:55:37 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:55:37.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:55:37 smithi115 conmon[90393]: 2022-05-14T00:55:37.528+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:55:38.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:55:38 smithi047 ceph-mon[77147]: pgmap v2988: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:55:38.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:55:38 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:55:38.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:55:38 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:55:38.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:55:38 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:55:38.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:55:38 smithi115 ceph-mon[78593]: pgmap v2988: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:55:38.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:55:38 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:55:38.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:55:38 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:55:38.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:55:38 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:55:38.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:55:38 smithi115 conmon[90393]: 2022-05-14T00:55:38.521+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:55:39.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:55:39 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:55:39.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:55:39 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:55:39.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:55:39 smithi115 conmon[90393]: 2022-05-14T00:55:39.500+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:55:40.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:55:40 smithi115 ceph-mon[78593]: pgmap v2989: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:55:40.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:55:40 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 5270 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:55:40.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:55:40 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:55:40.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:55:40 smithi047 ceph-mon[77147]: pgmap v2989: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:55:40.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:55:40 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 5270 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:55:40.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:55:40 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:55:40.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:55:40 smithi115 conmon[90393]: 2022-05-14T00:55:40.489+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:55:41.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:55:41 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:55:41.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:55:41 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:55:41.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:55:41 smithi115 conmon[90393]: 2022-05-14T00:55:41.465+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:55:42.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:55:42 smithi115 ceph-mon[78593]: pgmap v2990: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:55:42.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:55:42 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:55:42.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:55:42 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:55:42.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:55:42 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:55:42.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:55:42 smithi047 ceph-mon[77147]: pgmap v2990: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:55:42.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:55:42 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:55:42.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:55:42 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:55:42.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:55:42 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:55:42.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:55:42 smithi115 conmon[90393]: 2022-05-14T00:55:42.421+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:55:43.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:55:43 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:55:43.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:55:43 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:55:43.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:55:43 smithi115 conmon[90393]: 2022-05-14T00:55:43.411+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:55:44.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:55:44 smithi115 ceph-mon[78593]: pgmap v2991: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:55:44.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:55:44 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:55:44.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:55:44 smithi047 ceph-mon[77147]: pgmap v2991: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:55:44.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:55:44 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:55:44.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:55:44 smithi115 conmon[90393]: 2022-05-14T00:55:44.450+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:55:45.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:55:45 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 5275 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:55:45.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:55:45 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:55:45.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:55:45 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 5275 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:55:45.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:55:45 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:55:45.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:55:45 smithi115 conmon[90393]: 2022-05-14T00:55:45.410+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:55:46.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:55:46 smithi115 ceph-mon[78593]: pgmap v2992: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:55:46.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:55:46 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:55:46.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:55:46 smithi047 ceph-mon[77147]: pgmap v2992: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:55:46.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:55:46 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:55:46.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:55:46 smithi115 conmon[90393]: 2022-05-14T00:55:46.411+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:55:47.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:55:47 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:55:47.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:55:47 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:55:47.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:55:47 smithi115 conmon[90393]: 2022-05-14T00:55:47.435+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:55:48.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:55:48 smithi115 ceph-mon[78593]: pgmap v2993: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:55:48.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:55:48 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:55:48.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:55:48 smithi047 ceph-mon[77147]: pgmap v2993: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:55:48.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:55:48 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:55:48.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:55:48 smithi115 conmon[90393]: 2022-05-14T00:55:48.453+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:55:49.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:55:49 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:55:49.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:55:49 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:55:49.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:55:49 smithi115 conmon[90393]: 2022-05-14T00:55:49.499+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:55:50.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:55:50 smithi115 ceph-mon[78593]: pgmap v2994: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:55:50.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:55:50 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:55:50.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:55:50 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:55:50.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:55:50 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 5280 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:55:50.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:55:50 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:55:50.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:55:50 smithi047 ceph-mon[77147]: pgmap v2994: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:55:50.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:55:50 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:55:50.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:55:50 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:55:50.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:55:50 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 5280 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:55:50.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:55:50 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:55:50.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:55:50 smithi115 conmon[90393]: 2022-05-14T00:55:50.518+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:55:51.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:55:51 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:55:51.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:55:51 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:55:51.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:55:51 smithi115 conmon[90393]: 2022-05-14T00:55:51.516+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:55:52.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:55:52 smithi115 ceph-mon[78593]: pgmap v2995: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:55:52.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:55:52 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:55:52.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:55:52 smithi047 ceph-mon[77147]: pgmap v2995: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:55:52.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:55:52 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:55:52.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:55:52 smithi115 conmon[90393]: 2022-05-14T00:55:52.519+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:55:53.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:55:53 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:55:53.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:55:53 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:55:53.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:55:53 smithi115 conmon[90393]: 2022-05-14T00:55:53.554+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:55:54.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:55:54 smithi115 ceph-mon[78593]: pgmap v2996: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:55:54.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:55:54 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:55:54.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:55:54 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:55:54.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:55:54 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:55:54.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:55:54 smithi047 ceph-mon[77147]: pgmap v2996: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:55:54.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:55:54 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:55:54.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:55:54 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:55:54.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:55:54 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:55:54.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:55:54 smithi115 conmon[90393]: 2022-05-14T00:55:54.578+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:55:55.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:55:55 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 5285 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:55:55.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:55:55 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:55:55.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:55:55 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 5285 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:55:55.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:55:55 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:55:55.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:55:55 smithi115 conmon[90393]: 2022-05-14T00:55:55.565+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:55:56.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:55:56 smithi115 ceph-mon[78593]: pgmap v2997: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:55:56.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:55:56 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:55:56.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:55:56 smithi047 ceph-mon[77147]: pgmap v2997: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:55:56.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:55:56 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:55:56.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:55:56 smithi115 conmon[90393]: 2022-05-14T00:55:56.560+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:55:57.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:55:57 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:55:57.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:55:57 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:55:57.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:55:57 smithi115 conmon[90393]: 2022-05-14T00:55:57.523+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:55:58.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:55:58 smithi115 ceph-mon[78593]: pgmap v2998: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:55:58.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:55:58 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:55:58.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:55:58 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:55:58.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:55:58 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:55:58.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:55:58 smithi047 ceph-mon[77147]: pgmap v2998: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:55:58.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:55:58 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:55:58.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:55:58 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:55:58.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:55:58 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:55:58.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:55:58 smithi115 conmon[90393]: 2022-05-14T00:55:58.494+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:55:59.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:55:59 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:55:59.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:55:59 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:55:59.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:55:59 smithi115 conmon[90393]: 2022-05-14T00:55:59.465+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:56:00.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:56:00 smithi115 ceph-mon[78593]: pgmap v2999: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:56:00.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:56:00 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 5290 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:56:00.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:56:00 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:56:00.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:56:00 smithi047 ceph-mon[77147]: pgmap v2999: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:56:00.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:56:00 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 5290 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:56:00.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:56:00 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:56:00.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:56:00 smithi115 conmon[90393]: 2022-05-14T00:56:00.469+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:56:01.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:56:01 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:56:01.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:56:01 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:56:01.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:56:01 smithi115 conmon[90393]: 2022-05-14T00:56:01.490+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:56:02.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:56:02 smithi115 ceph-mon[78593]: pgmap v3000: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:56:02.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:56:02 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:56:02.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:56:02 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:56:02.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:56:02 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:56:02.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:56:02 smithi047 ceph-mon[77147]: pgmap v3000: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:56:02.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:56:02 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:56:02.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:56:02 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:56:02.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:56:02 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:56:02.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:56:02 smithi115 conmon[90393]: 2022-05-14T00:56:02.456+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:56:03.445 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:56:03 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:56:03.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:56:03 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:56:03.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:56:03 smithi115 conmon[90393]: 2022-05-14T00:56:03.444+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:56:04.419 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:56:04 smithi115 ceph-mon[78593]: pgmap v3001: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:56:04.420 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:56:04 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:56:04.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:56:04 smithi047 ceph-mon[77147]: pgmap v3001: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:56:04.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:56:04 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:56:04.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:56:04 smithi115 conmon[90393]: 2022-05-14T00:56:04.418+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:56:05.387 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:56:05 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 5295 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:56:05.388 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:56:05 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:56:05.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:56:05 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 5295 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:56:05.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:56:05 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:56:05.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:56:05 smithi115 conmon[90393]: 2022-05-14T00:56:05.386+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:56:06.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:56:06 smithi047 ceph-mon[77147]: pgmap v3002: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:56:06.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:56:06 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:56:06.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:56:06 smithi115 conmon[90393]: 2022-05-14T00:56:06.357+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:56:06.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:56:06 smithi115 ceph-mon[78593]: pgmap v3002: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:56:06.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:56:06 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:56:07.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:56:07 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:56:07.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:56:07 smithi115 conmon[90393]: 2022-05-14T00:56:07.324+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:56:07.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:56:07 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:56:08.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:56:08 smithi047 ceph-mon[77147]: pgmap v3003: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:56:08.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:56:08 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:56:08.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:56:08 smithi115 conmon[90393]: 2022-05-14T00:56:08.307+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:56:08.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:56:08 smithi115 ceph-mon[78593]: pgmap v3003: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:56:08.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:56:08 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:56:09.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:56:09 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:56:09.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:56:09 smithi115 conmon[90393]: 2022-05-14T00:56:09.350+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:56:09.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:56:09 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:56:10.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:56:10 smithi047 ceph-mon[77147]: pgmap v3004: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:56:10.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:56:10 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:56:10.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:56:10 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:56:10.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:56:10 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 5300 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:56:10.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:56:10 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:56:10.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:56:10 smithi115 conmon[90393]: 2022-05-14T00:56:10.321+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:56:10.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:56:10 smithi115 ceph-mon[78593]: pgmap v3004: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:56:10.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:56:10 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:56:10.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:56:10 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:56:10.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:56:10 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 5300 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:56:10.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:56:10 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:56:11.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:56:11 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:56:11.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:56:11 smithi115 conmon[90393]: 2022-05-14T00:56:11.337+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:56:11.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:56:11 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:56:12.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:56:12 smithi047 ceph-mon[77147]: pgmap v3005: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:56:12.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:56:12 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:56:12.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:56:12 smithi115 conmon[90393]: 2022-05-14T00:56:12.384+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:56:12.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:56:12 smithi115 ceph-mon[78593]: pgmap v3005: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:56:12.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:56:12 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:56:13.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:56:13 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:56:13.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:56:13 smithi115 conmon[90393]: 2022-05-14T00:56:13.377+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:56:13.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:56:13 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:56:14.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:56:14 smithi047 ceph-mon[77147]: pgmap v3006: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:56:14.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:56:14 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:56:14.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:56:14 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:56:14.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:56:14 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:56:14.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:56:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T00:56:14.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:56:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T00:56:14.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:56:14 smithi115 conmon[90393]: 2022-05-14T00:56:14.343+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:56:14.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:56:14 smithi115 ceph-mon[78593]: pgmap v3006: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:56:14.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:56:14 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:56:14.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:56:14 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:56:14.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:56:14 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:56:14.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:56:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T00:56:14.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:56:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T00:56:15.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:56:15 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 5305 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:56:15.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:56:15 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:56:15.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:56:15 smithi115 conmon[90393]: 2022-05-14T00:56:15.328+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:56:15.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:56:15 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 5305 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:56:15.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:56:15 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:56:16.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:56:16 smithi047 ceph-mon[77147]: pgmap v3007: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:56:16.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:56:16 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:56:16.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:56:16 smithi115 conmon[90393]: 2022-05-14T00:56:16.353+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:56:16.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:56:16 smithi115 ceph-mon[78593]: pgmap v3007: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:56:16.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:56:16 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:56:17.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:56:17 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:56:17.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:56:17 smithi115 conmon[90393]: 2022-05-14T00:56:17.384+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:56:17.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:56:17 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:56:18.417 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:56:18 smithi115 ceph-mon[78593]: pgmap v3008: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:56:18.417 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:56:18 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:56:18.417 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:56:18 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:56:18.418 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:56:18 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:56:18.418 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:56:18 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T00:56:18.418 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:56:18 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T00:56:18.418 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:56:18 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T00:56:18.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:56:18 smithi047 ceph-mon[77147]: pgmap v3008: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:56:18.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:56:18 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:56:18.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:56:18 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:56:18.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:56:18 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:56:18.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:56:18 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T00:56:18.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:56:18 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T00:56:18.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:56:18 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T00:56:18.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:56:18 smithi115 conmon[90393]: 2022-05-14T00:56:18.416+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:56:19.041 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:56:18 smithi047 conmon[78521]: 2022-05-14T00:56:18.736+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:56:19.041 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:56:18 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T00:56:19.041 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:56:18 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:56:19.042 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:56:18 smithi047 conmon[78521]: yield 2022-05-14T00:56:19.042 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:56:18 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:56:19.042 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:56:18 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:56:19.042 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:56:18 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:56:19.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:56:18 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T00:56:19.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:56:18 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:56:19.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:56:18 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T00:56:19.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:56:18 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:56:19.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:56:18 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T00:56:19.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:56:18 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:56:19.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:56:18 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T00:56:19.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:56:18 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:56:19.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:56:18 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T00:56:19.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:56:18 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:56:19.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:56:18 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:56:19.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:56:18 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:56:19.406 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:56:19 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:56:19.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:56:19 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:56:19.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:56:19 smithi115 conmon[90393]: 2022-05-14T00:56:19.404+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:56:20.449 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:56:20 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:56:20.450 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T00:56:20.450 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:56:20.450 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T00:56:20.451 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:56:20.451 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:56:20.451 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:56:20.452 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T00:56:20.453 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:56:20.453 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T00:56:20.453 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:56:20.454 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T00:56:20.454 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:56:20.454 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T00:56:20.454 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:56:20.455 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T00:56:20.455 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:56:20.455 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:56:20.455 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:56:20.455 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:56:20 smithi115 ceph-mon[78593]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T00:56:20.456 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:56:20 smithi115 ceph-mon[78593]: pgmap v3009: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:56:20.456 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:56:20 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 5310 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:56:20.456 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:56:20 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:56:20.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:56:20 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:56:20.541 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T00:56:20.541 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:56:20.542 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T00:56:20.542 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:56:20.542 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:56:20.542 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:56:20.542 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T00:56:20.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:56:20.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T00:56:20.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:56:20.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T00:56:20.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:56:20.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T00:56:20.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:56:20.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T00:56:20.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:56:20.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:56:20.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:56:20.545 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:56:20 smithi047 ceph-mon[77147]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T00:56:20.545 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:56:20 smithi047 ceph-mon[77147]: pgmap v3009: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:56:20.545 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:56:20 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 5310 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:56:20.546 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:56:20 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:56:20.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:56:20 smithi115 conmon[90393]: 2022-05-14T00:56:20.448+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:56:21.478 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:56:21 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:56:21.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:56:21 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:56:21.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:56:21 smithi115 conmon[90393]: 2022-05-14T00:56:21.477+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:56:22.163 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:56:21 smithi047 conmon[78521]: 2022-05-14T00:56:21.808+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:56:22.163 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:56:21 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T00:56:22.164 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:56:21 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:56:22.164 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:56:21 smithi047 conmon[78521]: yield 2022-05-14T00:56:22.164 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:56:21 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:56:22.164 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:56:21 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:56:22.164 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:56:21 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:56:22.165 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:56:21 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T00:56:22.165 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:56:21 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:56:22.165 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:56:21 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T00:56:22.165 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:56:21 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:56:22.166 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:56:21 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T00:56:22.166 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:56:21 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:56:22.166 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:56:21 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T00:56:22.166 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:56:21 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:56:22.166 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:56:21 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T00:56:22.167 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:56:21 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:56:22.167 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:56:21 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:56:22.167 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:56:21 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:56:22.167 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:56:21 smithi047 conmon[78521]: 2022-05-14T00:56:21.809+0000 7f5345f84700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T00:56:22.168 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:56:21 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T00:56:22.168 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:56:21 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:56:22.168 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:56:21 smithi047 conmon[78521]: yield 2022-05-14T00:56:22.168 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:56:21 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:56:22.168 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:56:21 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:56:22.169 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:56:21 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:56:22.169 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:56:21 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T00:56:22.169 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:56:21 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:56:22.169 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:56:21 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T00:56:22.170 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:56:21 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:56:22.170 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:56:21 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T00:56:22.170 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:56:21 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:56:22.170 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:56:21 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T00:56:22.170 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:56:21 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:56:22.171 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:56:21 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T00:56:22.171 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:56:21 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:56:22.171 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:56:21 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:56:22.171 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:56:21 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:56:22.171 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:56:21 smithi047 conmon[78521]: 2022-05-14T00:56:22.172 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:56:21 smithi047 conmon[78521]: During handling of the above exception, another exception occurred: 2022-05-14T00:56:22.172 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:56:21 smithi047 conmon[78521]: 2022-05-14T00:56:22.172 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:56:21 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T00:56:22.172 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:56:21 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T00:56:22.172 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:56:21 smithi047 conmon[78521]: return f(*arg) 2022-05-14T00:56:22.173 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:56:21 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T00:56:22.173 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:56:21 smithi047 conmon[78521]: self._write_client_files(client_files, host) 2022-05-14T00:56:22.173 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:56:21 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T00:56:22.173 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:56:21 smithi047 conmon[78521]: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T00:56:22.174 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:56:21 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T00:56:22.174 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:56:21 smithi047 conmon[78521]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T00:56:22.174 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:56:21 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T00:56:22.174 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:56:21 smithi047 conmon[78521]: return self.event_loop.get_result(coro) 2022-05-14T00:56:22.174 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:56:21 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T00:56:22.175 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:56:21 smithi047 conmon[78521]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T00:56:22.175 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:56:21 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T00:56:22.175 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:56:21 smithi047 conmon[78521]: return self.__get_result() 2022-05-14T00:56:22.175 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:56:21 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T00:56:22.175 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:56:21 smithi047 conmon[78521]: raise self._exception 2022-05-14T00:56:22.176 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:56:21 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T00:56:22.176 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:56:21 smithi047 conmon[78521]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T00:56:22.176 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:56:21 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T00:56:22.176 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:56:21 smithi047 conmon[78521]: conn = await self._remote_connection(host, addr) 2022-05-14T00:56:22.177 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:56:21 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T00:56:22.177 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:56:21 smithi047 conmon[78521]: raise 2022-05-14T00:56:22.177 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:56:21 smithi047 conmon[78521]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T00:56:22.177 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:56:21 smithi047 conmon[78521]: self.gen.throw(type, value, traceback) 2022-05-14T00:56:22.177 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:56:21 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T00:56:22.178 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:56:21 smithi047 conmon[78521]: raise OrchestratorError(msg) 2022-05-14T00:56:22.178 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:56:21 smithi047 conmon[78521]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:56:22.442 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:56:22 smithi115 ceph-mon[78593]: pgmap v3010: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:56:22.442 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:56:22 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:56:22.442 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:56:22 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:56:22.442 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:56:22 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:56:22.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:56:22 smithi047 ceph-mon[77147]: pgmap v3010: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:56:22.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:56:22 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:56:22.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:56:22 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:56:22.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:56:22 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:56:22.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:56:22 smithi115 conmon[90393]: 2022-05-14T00:56:22.440+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:56:23.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:56:23 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:56:23.542 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T00:56:23.542 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:56:23.542 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T00:56:23.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:56:23.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:56:23.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:56:23.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T00:56:23.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:56:23.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T00:56:23.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:56:23.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T00:56:23.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:56:23.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T00:56:23.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:56:23.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T00:56:23.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:56:23.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:56:23.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:56:23.546 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:56:23 smithi047 ceph-mon[77147]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T00:56:23.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T00:56:23.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:56:23.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T00:56:23.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:56:23.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:56:23.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:56:23.548 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T00:56:23.548 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:56:23.548 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T00:56:23.548 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:56:23.548 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T00:56:23.549 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:56:23.549 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T00:56:23.549 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:56:23.549 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T00:56:23.550 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:56:23.550 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:56:23.550 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:56:23.550 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T00:56:23.550 INFO:journalctl@ceph.mon.b.smithi047.stdout: During handling of the above exception, another exception occurred: 2022-05-14T00:56:23.551 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T00:56:23.551 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T00:56:23.551 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T00:56:23.551 INFO:journalctl@ceph.mon.b.smithi047.stdout: return f(*arg) 2022-05-14T00:56:23.551 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T00:56:23.552 INFO:journalctl@ceph.mon.b.smithi047.stdout: self._write_client_files(client_files, host) 2022-05-14T00:56:23.552 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T00:56:23.552 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T00:56:23.552 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T00:56:23.553 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T00:56:23.553 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T00:56:23.553 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.event_loop.get_result(coro) 2022-05-14T00:56:23.553 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T00:56:23.553 INFO:journalctl@ceph.mon.b.smithi047.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T00:56:23.554 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T00:56:23.554 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.__get_result() 2022-05-14T00:56:23.554 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T00:56:23.554 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise self._exception 2022-05-14T00:56:23.554 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T00:56:23.555 INFO:journalctl@ceph.mon.b.smithi047.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T00:56:23.555 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T00:56:23.555 INFO:journalctl@ceph.mon.b.smithi047.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T00:56:23.555 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T00:56:23.556 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise 2022-05-14T00:56:23.556 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T00:56:23.556 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.gen.throw(type, value, traceback) 2022-05-14T00:56:23.556 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T00:56:23.556 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OrchestratorError(msg) 2022-05-14T00:56:23.557 INFO:journalctl@ceph.mon.b.smithi047.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:56:23.557 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:56:23 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:56:23.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:56:23 smithi115 conmon[90393]: 2022-05-14T00:56:23.413+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:56:23.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:56:23 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:56:23.609 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T00:56:23.609 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:56:23.609 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T00:56:23.609 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:56:23.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:56:23.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:56:23.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T00:56:23.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:56:23.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T00:56:23.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:56:23.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T00:56:23.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:56:23.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T00:56:23.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:56:23.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T00:56:23.613 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:56:23.613 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:56:23.614 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:56:23.614 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:56:23 smithi115 ceph-mon[78593]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T00:56:23.615 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T00:56:23.615 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:56:23.615 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T00:56:23.616 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:56:23.616 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:56:23.617 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:56:23.617 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T00:56:23.617 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:56:23.617 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T00:56:23.617 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:56:23.618 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T00:56:23.618 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:56:23.618 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T00:56:23.618 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:56:23.619 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T00:56:23.619 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:56:23.619 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:56:23.619 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:56:23.619 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T00:56:23.620 INFO:journalctl@ceph.mon.c.smithi115.stdout: During handling of the above exception, another exception occurred: 2022-05-14T00:56:23.620 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T00:56:23.620 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T00:56:23.620 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T00:56:23.620 INFO:journalctl@ceph.mon.c.smithi115.stdout: return f(*arg) 2022-05-14T00:56:23.621 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T00:56:23.621 INFO:journalctl@ceph.mon.c.smithi115.stdout: self._write_client_files(client_files, host) 2022-05-14T00:56:23.621 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T00:56:23.621 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T00:56:23.621 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T00:56:23.622 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T00:56:23.622 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T00:56:23.622 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.event_loop.get_result(coro) 2022-05-14T00:56:23.622 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T00:56:23.622 INFO:journalctl@ceph.mon.c.smithi115.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T00:56:23.623 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T00:56:23.623 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.__get_result() 2022-05-14T00:56:23.623 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T00:56:23.624 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise self._exception 2022-05-14T00:56:23.625 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T00:56:23.625 INFO:journalctl@ceph.mon.c.smithi115.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T00:56:23.625 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T00:56:23.625 INFO:journalctl@ceph.mon.c.smithi115.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T00:56:23.626 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T00:56:23.626 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise 2022-05-14T00:56:23.626 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T00:56:23.626 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.gen.throw(type, value, traceback) 2022-05-14T00:56:23.626 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T00:56:23.627 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OrchestratorError(msg) 2022-05-14T00:56:23.627 INFO:journalctl@ceph.mon.c.smithi115.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:56:23.627 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:56:23 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:56:24.461 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:56:24 smithi115 ceph-mon[78593]: pgmap v3011: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:56:24.462 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:56:24 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:56:24.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:56:24 smithi047 ceph-mon[77147]: pgmap v3011: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:56:24.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:56:24 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:56:24.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:56:24 smithi115 conmon[90393]: 2022-05-14T00:56:24.460+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:56:25.423 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:56:25 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 5315 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:56:25.424 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:56:25 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:56:25.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:56:25 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 5315 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:56:25.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:56:25 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:56:25.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:56:25 smithi115 conmon[90393]: 2022-05-14T00:56:25.422+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:56:26.457 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:56:26 smithi115 ceph-mon[78593]: pgmap v3012: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:56:26.458 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:56:26 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:56:26.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:56:26 smithi047 ceph-mon[77147]: pgmap v3012: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:56:26.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:56:26 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:56:26.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:56:26 smithi115 conmon[90393]: 2022-05-14T00:56:26.455+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:56:27.439 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:56:27 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:56:27.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:56:27 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:56:27.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:56:27 smithi115 conmon[90393]: 2022-05-14T00:56:27.437+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:56:28.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:56:28 smithi047 ceph-mon[77147]: pgmap v3013: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:56:28.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:56:28 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:56:28.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:56:28 smithi115 conmon[90393]: 2022-05-14T00:56:28.404+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:56:28.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:56:28 smithi115 ceph-mon[78593]: pgmap v3013: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:56:28.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:56:28 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:56:29.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:56:29 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:56:29.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:56:29 smithi115 conmon[90393]: 2022-05-14T00:56:29.388+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:56:29.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:56:29 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:56:30.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:56:30 smithi047 ceph-mon[77147]: pgmap v3014: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:56:30.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:56:30 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:56:30.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:56:30 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:56:30.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:56:30 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 5320 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:56:30.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:56:30 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:56:30.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:56:30 smithi115 conmon[90393]: 2022-05-14T00:56:30.346+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:56:30.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:56:30 smithi115 ceph-mon[78593]: pgmap v3014: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:56:30.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:56:30 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:56:30.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:56:30 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:56:30.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:56:30 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 5320 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:56:30.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:56:30 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:56:31.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:56:31 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:56:31.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:56:31 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:56:31.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:56:31 smithi115 conmon[90393]: 2022-05-14T00:56:31.376+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:56:32.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:56:32 smithi047 ceph-mon[77147]: pgmap v3015: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:56:32.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:56:32 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:56:32.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:56:32 smithi115 conmon[90393]: 2022-05-14T00:56:32.356+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:56:32.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:56:32 smithi115 ceph-mon[78593]: pgmap v3015: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:56:32.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:56:32 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:56:33.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:56:33 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:56:33.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:56:33 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:56:33.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:56:33 smithi115 conmon[90393]: 2022-05-14T00:56:33.336+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:56:34.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:56:34 smithi047 ceph-mon[77147]: pgmap v3016: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:56:34.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:56:34 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:56:34.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:56:34 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:56:34.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:56:34 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:56:34.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:56:34 smithi115 conmon[90393]: 2022-05-14T00:56:34.340+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:56:34.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:56:34 smithi115 ceph-mon[78593]: pgmap v3016: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:56:34.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:56:34 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:56:34.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:56:34 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:56:34.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:56:34 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:56:35.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:56:35 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:56:35.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:56:35 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 5325 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:56:35.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:56:35 smithi115 conmon[90393]: 2022-05-14T00:56:35.340+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:56:35.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:56:35 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:56:35.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:56:35 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 5325 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:56:36.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:56:36 smithi047 ceph-mon[77147]: pgmap v3017: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:56:36.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:56:36 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:56:36.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:56:36 smithi115 conmon[90393]: 2022-05-14T00:56:36.341+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:56:36.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:56:36 smithi115 ceph-mon[78593]: pgmap v3017: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:56:36.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:56:36 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:56:37.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:56:37 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:56:37.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:56:37 smithi115 conmon[90393]: 2022-05-14T00:56:37.304+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:56:37.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:56:37 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:56:38.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:56:38 smithi047 ceph-mon[77147]: pgmap v3018: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:56:38.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:56:38 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:56:38.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:56:38 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:56:38.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:56:38 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:56:38.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:56:38 smithi115 conmon[90393]: 2022-05-14T00:56:38.310+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:56:38.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:56:38 smithi115 ceph-mon[78593]: pgmap v3018: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:56:38.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:56:38 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:56:38.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:56:38 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:56:38.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:56:38 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:56:39.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:56:39 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:56:39.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:56:39 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:56:39.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:56:39 smithi115 conmon[90393]: 2022-05-14T00:56:39.351+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:56:40.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:56:40 smithi047 ceph-mon[77147]: pgmap v3019: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:56:40.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:56:40 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 5330 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:56:40.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:56:40 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:56:40.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:56:40 smithi115 conmon[90393]: 2022-05-14T00:56:40.363+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:56:40.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:56:40 smithi115 ceph-mon[78593]: pgmap v3019: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:56:40.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:56:40 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 5330 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:56:40.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:56:40 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:56:41.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:56:41 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:56:41.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:56:41 smithi115 conmon[90393]: 2022-05-14T00:56:41.318+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:56:41.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:56:41 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:56:42.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:56:42 smithi047 ceph-mon[77147]: pgmap v3020: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:56:42.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:56:42 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:56:42.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:56:42 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:56:42.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:56:42 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:56:42.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:56:42 smithi115 conmon[90393]: 2022-05-14T00:56:42.313+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:56:42.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:56:42 smithi115 ceph-mon[78593]: pgmap v3020: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:56:42.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:56:42 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:56:42.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:56:42 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:56:42.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:56:42 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:56:43.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:56:43 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:56:43.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:56:43 smithi115 conmon[90393]: 2022-05-14T00:56:43.278+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:56:43.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:56:43 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:56:44.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:56:44 smithi047 ceph-mon[77147]: pgmap v3021: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:56:44.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:56:44 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:56:44.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:56:44 smithi115 ceph-mon[78593]: pgmap v3021: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:56:44.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:56:44 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:56:44.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:56:44 smithi115 conmon[90393]: 2022-05-14T00:56:44.248+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:56:45.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:56:45 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:56:45.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:56:45 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 5335 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:56:45.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:56:45 smithi115 conmon[90393]: 2022-05-14T00:56:45.224+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:56:45.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:56:45 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:56:45.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:56:45 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 5335 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:56:46.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:56:46 smithi047 ceph-mon[77147]: pgmap v3022: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:56:46.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:56:46 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:56:46.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:56:46 smithi115 conmon[90393]: 2022-05-14T00:56:46.195+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:56:46.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:56:46 smithi115 ceph-mon[78593]: pgmap v3022: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:56:46.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:56:46 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:56:47.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:56:47 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:56:47.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:56:47 smithi115 conmon[90393]: 2022-05-14T00:56:47.207+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:56:47.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:56:47 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:56:48.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:56:48 smithi047 ceph-mon[77147]: pgmap v3023: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:56:48.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:56:48 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:56:48.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:56:48 smithi115 conmon[90393]: 2022-05-14T00:56:48.168+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:56:48.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:56:48 smithi115 ceph-mon[78593]: pgmap v3023: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:56:48.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:56:48 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:56:49.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:56:49 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:56:49.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:56:49 smithi115 conmon[90393]: 2022-05-14T00:56:49.129+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:56:49.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:56:49 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:56:50.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:56:50 smithi115 conmon[90393]: 2022-05-14T00:56:50.144+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:56:50.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:56:50 smithi115 ceph-mon[78593]: pgmap v3024: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:56:50.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:56:50 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:56:50.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:56:50 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:56:50.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:56:50 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:56:50.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:56:50 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 5340 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:56:50.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:56:50 smithi047 ceph-mon[77147]: pgmap v3024: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:56:50.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:56:50 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:56:50.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:56:50 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:56:50.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:56:50 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:56:50.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:56:50 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 5340 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:56:51.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:56:51 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:56:51.591 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:56:51 smithi115 conmon[90393]: 2022-05-14T00:56:51.172+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:56:51.592 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:56:51 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:56:52.401 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:56:52 smithi115 conmon[90393]: 2022-05-14T00:56:52.125+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:56:52.402 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:56:52 smithi115 ceph-mon[78593]: pgmap v3025: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:56:52.402 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:56:52 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:56:52.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:56:52 smithi047 ceph-mon[77147]: pgmap v3025: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:56:52.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:56:52 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:56:53.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:56:53 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:56:53.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:56:53 smithi115 conmon[90393]: 2022-05-14T00:56:53.146+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:56:53.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:56:53 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:56:54.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:56:54 smithi047 ceph-mon[77147]: pgmap v3026: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:56:54.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:56:54 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:56:54.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:56:54 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:56:54.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:56:54 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:56:54.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:56:54 smithi115 conmon[90393]: 2022-05-14T00:56:54.150+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:56:54.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:56:54 smithi115 ceph-mon[78593]: pgmap v3026: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:56:54.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:56:54 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:56:54.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:56:54 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:56:54.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:56:54 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:56:55.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:56:55 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:56:55.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:56:55 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 5345 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:56:55.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:56:55 smithi115 conmon[90393]: 2022-05-14T00:56:55.145+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:56:55.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:56:55 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:56:55.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:56:55 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 5345 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:56:56.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:56:56 smithi047 ceph-mon[77147]: pgmap v3027: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:56:56.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:56:56 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:56:56.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:56:56 smithi115 conmon[90393]: 2022-05-14T00:56:56.119+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:56:56.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:56:56 smithi115 ceph-mon[78593]: pgmap v3027: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:56:56.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:56:56 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:56:57.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:56:57 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:56:57.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:56:57 smithi115 conmon[90393]: 2022-05-14T00:56:57.132+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:56:57.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:56:57 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:56:58.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:56:58 smithi047 ceph-mon[77147]: pgmap v3028: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:56:58.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:56:58 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:56:58.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:56:58 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:56:58.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:56:58 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:56:58.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:56:58 smithi115 conmon[90393]: 2022-05-14T00:56:58.137+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:56:58.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:56:58 smithi115 ceph-mon[78593]: pgmap v3028: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:56:58.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:56:58 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:56:58.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:56:58 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:56:58.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:56:58 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:56:59.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:56:59 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:56:59.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:56:59 smithi115 conmon[90393]: 2022-05-14T00:56:59.166+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:56:59.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:56:59 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:57:00.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:57:00 smithi047 ceph-mon[77147]: pgmap v3029: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:57:00.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:57:00 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:57:00.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:57:00 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 5350 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:57:00.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:57:00 smithi115 conmon[90393]: 2022-05-14T00:57:00.202+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:57:00.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:57:00 smithi115 ceph-mon[78593]: pgmap v3029: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:57:00.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:57:00 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:57:00.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:57:00 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 5350 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:57:01.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:57:01 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:57:01.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:57:01 smithi115 conmon[90393]: 2022-05-14T00:57:01.158+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:57:01.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:57:01 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:57:02.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:57:02 smithi047 ceph-mon[77147]: pgmap v3030: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:57:02.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:57:02 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:57:02.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:57:02 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:57:02.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:57:02 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:57:02.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:57:02 smithi115 conmon[90393]: 2022-05-14T00:57:02.166+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:57:02.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:57:02 smithi115 ceph-mon[78593]: pgmap v3030: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:57:02.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:57:02 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:57:02.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:57:02 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:57:02.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:57:02 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:57:03.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:57:03 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:57:03.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:57:03 smithi115 conmon[90393]: 2022-05-14T00:57:03.132+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:57:03.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:57:03 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:57:04.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:57:04 smithi047 ceph-mon[77147]: pgmap v3031: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:57:04.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:57:04 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:57:04.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:57:04 smithi115 conmon[90393]: 2022-05-14T00:57:04.176+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:57:04.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:57:04 smithi115 ceph-mon[78593]: pgmap v3031: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:57:04.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:57:04 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:57:05.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:57:05 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:57:05.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:57:05 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 5355 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:57:05.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:57:05 smithi115 conmon[90393]: 2022-05-14T00:57:05.135+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:57:05.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:57:05 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:57:05.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:57:05 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 5355 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:57:06.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:57:06 smithi115 conmon[90393]: 2022-05-14T00:57:06.090+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:57:06.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:57:06 smithi115 ceph-mon[78593]: pgmap v3032: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:57:06.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:57:06 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:57:06.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:57:06 smithi047 ceph-mon[77147]: pgmap v3032: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:57:06.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:57:06 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:57:07.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:57:07 smithi115 conmon[90393]: 2022-05-14T00:57:07.106+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:57:07.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:57:07 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:57:07.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:57:07 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:57:08.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:57:08 smithi115 conmon[90393]: 2022-05-14T00:57:08.136+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:57:08.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:57:08 smithi115 ceph-mon[78593]: pgmap v3033: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:57:08.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:57:08 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:57:08.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:57:08 smithi047 ceph-mon[77147]: pgmap v3033: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:57:08.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:57:08 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:57:09.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:57:09 smithi115 conmon[90393]: 2022-05-14T00:57:09.098+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:57:09.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:57:09 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:57:09.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:57:09 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:57:10.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:57:10 smithi115 conmon[90393]: 2022-05-14T00:57:10.113+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:57:10.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:57:10 smithi115 ceph-mon[78593]: pgmap v3034: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:57:10.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:57:10 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:57:10.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:57:10 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:57:10.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:57:10 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:57:10.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:57:10 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 5360 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:57:10.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:57:10 smithi047 ceph-mon[77147]: pgmap v3034: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:57:10.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:57:10 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:57:10.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:57:10 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:57:10.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:57:10 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:57:10.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:57:10 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 5360 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:57:11.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:57:11 smithi115 conmon[90393]: 2022-05-14T00:57:11.092+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:57:11.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:57:11 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:57:11.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:57:11 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:57:12.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:57:12 smithi115 conmon[90393]: 2022-05-14T00:57:12.064+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:57:12.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:57:12 smithi115 ceph-mon[78593]: pgmap v3035: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:57:12.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:57:12 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:57:12.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:57:12 smithi047 ceph-mon[77147]: pgmap v3035: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:57:12.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:57:12 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:57:13.315 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:57:13 smithi115 conmon[90393]: 2022-05-14T00:57:13.046+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:57:13.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:57:13 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:57:13.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:57:13 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:57:14.316 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:57:14 smithi115 conmon[90393]: 2022-05-14T00:57:14.048+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:57:14.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:57:14 smithi115 ceph-mon[78593]: pgmap v3036: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:57:14.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:57:14 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:57:14.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:57:14 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:57:14.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:57:14 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:57:14.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:57:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T00:57:14.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:57:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T00:57:14.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:57:14 smithi047 ceph-mon[77147]: pgmap v3036: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:57:14.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:57:14 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:57:14.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:57:14 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:57:14.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:57:14 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:57:14.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:57:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T00:57:14.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:57:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T00:57:15.318 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:57:15 smithi115 conmon[90393]: 2022-05-14T00:57:15.010+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:57:15.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:57:15 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:57:15.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:57:15 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 5365 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:57:15.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:57:15 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:57:15.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:57:15 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:57:15.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:57:15 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 5365 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:57:15.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:57:15 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:57:16.319 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:57:15 smithi115 conmon[90393]: 2022-05-14T00:57:15.963+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:57:16.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:57:16 smithi115 ceph-mon[78593]: pgmap v3037: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:57:16.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:57:16 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:57:16.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:57:16 smithi047 ceph-mon[77147]: pgmap v3037: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:57:16.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:57:16 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:57:17.325 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:57:16 smithi115 conmon[90393]: 2022-05-14T00:57:16.919+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:57:17.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:57:17 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:57:17.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:57:17 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:57:18.326 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:57:17 smithi115 conmon[90393]: 2022-05-14T00:57:17.913+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:57:18.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:57:18 smithi115 ceph-mon[78593]: pgmap v3038: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:57:18.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:57:18 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:57:18.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:57:18 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:57:18.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:57:18 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:57:18.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:57:18 smithi047 ceph-mon[77147]: pgmap v3038: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:57:18.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:57:18 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:57:18.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:57:18 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:57:18.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:57:18 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:57:19.331 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:57:18 smithi115 conmon[90393]: 2022-05-14T00:57:18.956+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:57:19.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:57:19 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:57:19.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:57:19 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:57:20.333 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:57:19 smithi115 conmon[90393]: 2022-05-14T00:57:19.911+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:57:20.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:57:20 smithi115 ceph-mon[78593]: pgmap v3039: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:57:20.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:57:20 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 5370 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:57:20.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:57:20 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:57:20.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:57:20 smithi047 ceph-mon[77147]: pgmap v3039: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:57:20.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:57:20 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 5370 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:57:20.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:57:20 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:57:21.338 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:57:20 smithi115 conmon[90393]: 2022-05-14T00:57:20.920+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:57:21.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:57:21 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:57:21.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:57:21 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:57:22.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:57:21 smithi115 conmon[90393]: 2022-05-14T00:57:21.949+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:57:22.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:57:22 smithi115 ceph-mon[78593]: pgmap v3040: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:57:22.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:57:22 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:57:22.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:57:22 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:57:22.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:57:22 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T00:57:22.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:57:22 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T00:57:22.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:57:22 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T00:57:22.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:57:22 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:57:22.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:57:22 smithi047 ceph-mon[77147]: pgmap v3040: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:57:22.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:57:22 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:57:22.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:57:22 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:57:22.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:57:22 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T00:57:22.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:57:22 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T00:57:22.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:57:22 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T00:57:22.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:57:22 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:57:23.344 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:57:22 smithi115 conmon[90393]: 2022-05-14T00:57:22.987+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:57:23.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:57:23 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:57:23.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:57:23 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:57:24.345 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:57:24 smithi115 conmon[90393]: 2022-05-14T00:57:24.005+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:57:24.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:57:24 smithi115 ceph-mon[78593]: pgmap v3041: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:57:24.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:57:24 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:57:24.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:57:24 smithi047 ceph-mon[77147]: pgmap v3041: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:57:24.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:57:24 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:57:25.291 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:57:24 smithi047 conmon[78521]: 2022-05-14T00:57:24.911+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:57:25.291 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:57:24 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T00:57:25.291 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:57:24 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:57:25.292 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:57:24 smithi047 conmon[78521]: yield 2022-05-14T00:57:25.292 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:57:24 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:57:25.292 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:57:24 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:57:25.292 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:57:24 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:57:25.293 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:57:24 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T00:57:25.293 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:57:24 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:57:25.293 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:57:24 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T00:57:25.293 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:57:24 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:57:25.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:57:24 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T00:57:25.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:57:24 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:57:25.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:57:24 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T00:57:25.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:57:24 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:57:25.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:57:24 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T00:57:25.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:57:24 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:57:25.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:57:24 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:57:25.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:57:24 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:57:25.347 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:57:24 smithi115 conmon[90393]: 2022-05-14T00:57:24.988+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:57:25.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:57:25 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 5375 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:57:25.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:57:25 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:57:25.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:57:25 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 5375 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:57:25.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:57:25 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:57:26.351 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:57:25 smithi115 conmon[90393]: 2022-05-14T00:57:25.987+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:57:26.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:57:26 smithi115 ceph-mon[78593]: pgmap v3042: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:57:26.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:57:26 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:57:26.608 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T00:57:26.609 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:57:26.609 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T00:57:26.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:57:26.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:57:26.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:57:26.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T00:57:26.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:57:26.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T00:57:26.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:57:26.613 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T00:57:26.613 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:57:26.614 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T00:57:26.614 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:57:26.614 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T00:57:26.615 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:57:26.615 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:57:26.616 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:57:26.616 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:57:26 smithi115 ceph-mon[78593]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T00:57:26.616 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:57:26 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:57:26.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:57:26 smithi047 ceph-mon[77147]: pgmap v3042: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:57:26.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:57:26 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:57:26.791 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T00:57:26.792 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:57:26.792 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T00:57:26.792 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:57:26.792 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:57:26.792 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:57:26.793 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T00:57:26.793 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:57:26.793 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T00:57:26.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:57:26.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T00:57:26.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:57:26.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T00:57:26.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:57:26.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T00:57:26.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:57:26.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:57:26.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:57:26.796 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:57:26 smithi047 ceph-mon[77147]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T00:57:26.796 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:57:26 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:57:27.354 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:57:27 smithi115 conmon[90393]: 2022-05-14T00:57:27.004+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:57:27.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:57:27 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:57:27.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:57:27 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:57:28.291 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:57:27 smithi047 conmon[78521]: 2022-05-14T00:57:27.984+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:57:28.292 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:57:27 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T00:57:28.292 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:57:27 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:57:28.292 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:57:27 smithi047 conmon[78521]: yield 2022-05-14T00:57:28.292 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:57:27 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:57:28.292 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:57:27 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:57:28.293 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:57:27 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:57:28.293 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:57:27 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T00:57:28.293 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:57:27 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:57:28.293 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:57:27 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T00:57:28.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:57:27 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:57:28.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:57:27 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T00:57:28.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:57:27 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:57:28.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:57:27 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T00:57:28.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:57:27 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:57:28.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:57:27 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T00:57:28.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:57:27 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:57:28.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:57:27 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:57:28.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:57:27 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:57:28.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:57:27 smithi047 conmon[78521]: 2022-05-14T00:57:27.985+0000 7f533ff78700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T00:57:28.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:57:27 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T00:57:28.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:57:27 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:57:28.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:57:27 smithi047 conmon[78521]: yield 2022-05-14T00:57:28.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:57:27 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:57:28.297 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:57:27 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:57:28.297 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:57:27 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:57:28.297 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:57:27 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T00:57:28.297 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:57:27 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:57:28.297 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:57:27 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T00:57:28.298 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:57:27 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:57:28.298 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:57:27 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T00:57:28.298 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:57:27 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:57:28.298 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:57:27 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T00:57:28.298 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:57:27 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:57:28.299 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:57:27 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T00:57:28.299 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:57:27 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:57:28.299 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:57:27 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:57:28.299 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:57:27 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:57:28.299 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:57:27 smithi047 conmon[78521]: 2022-05-14T00:57:28.300 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:57:27 smithi047 conmon[78521]: During handling of the above exception, another exception occurred: 2022-05-14T00:57:28.300 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:57:27 smithi047 conmon[78521]: 2022-05-14T00:57:28.300 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:57:27 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T00:57:28.300 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:57:27 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T00:57:28.301 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:57:27 smithi047 conmon[78521]: return f(*arg) 2022-05-14T00:57:28.301 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:57:27 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T00:57:28.301 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:57:27 smithi047 conmon[78521]: self._write_client_files(client_files, host) 2022-05-14T00:57:28.301 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:57:27 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T00:57:28.301 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:57:27 smithi047 conmon[78521]: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T00:57:28.302 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:57:27 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T00:57:28.302 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:57:27 smithi047 conmon[78521]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T00:57:28.302 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:57:27 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T00:57:28.302 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:57:27 smithi047 conmon[78521]: return self.event_loop.get_result(coro) 2022-05-14T00:57:28.302 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:57:27 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T00:57:28.303 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:57:27 smithi047 conmon[78521]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T00:57:28.303 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:57:27 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T00:57:28.303 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:57:27 smithi047 conmon[78521]: return self.__get_result() 2022-05-14T00:57:28.303 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:57:27 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T00:57:28.304 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:57:27 smithi047 conmon[78521]: raise self._exception 2022-05-14T00:57:28.304 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:57:27 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T00:57:28.304 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:57:27 smithi047 conmon[78521]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T00:57:28.304 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:57:27 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T00:57:28.304 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:57:27 smithi047 conmon[78521]: conn = await self._remote_connection(host, addr) 2022-05-14T00:57:28.305 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:57:27 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T00:57:28.305 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:57:27 smithi047 conmon[78521]: raise 2022-05-14T00:57:28.305 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:57:27 smithi047 conmon[78521]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T00:57:28.305 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:57:27 smithi047 conmon[78521]: self.gen.throw(type, value, traceback) 2022-05-14T00:57:28.305 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:57:27 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T00:57:28.306 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:57:27 smithi047 conmon[78521]: raise OrchestratorError(msg) 2022-05-14T00:57:28.306 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:57:27 smithi047 conmon[78521]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:57:28.355 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:57:28 smithi115 conmon[90393]: 2022-05-14T00:57:28.043+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:57:28.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:57:28 smithi115 ceph-mon[78593]: pgmap v3043: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:57:28.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:57:28 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:57:28.608 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T00:57:28.608 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:57:28.609 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T00:57:28.609 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:57:28.609 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:57:28.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:57:28.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T00:57:28.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:57:28.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T00:57:28.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:57:28.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T00:57:28.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:57:28.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T00:57:28.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:57:28.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T00:57:28.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:57:28.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:57:28.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:57:28.612 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:57:28 smithi115 ceph-mon[78593]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T00:57:28.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T00:57:28.613 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:57:28.613 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T00:57:28.613 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:57:28.613 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:57:28.614 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:57:28.614 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T00:57:28.614 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:57:28.614 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T00:57:28.615 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:57:28.615 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T00:57:28.615 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:57:28.615 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T00:57:28.615 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:57:28.616 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T00:57:28.616 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:57:28.616 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:57:28.616 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:57:28.616 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T00:57:28.617 INFO:journalctl@ceph.mon.c.smithi115.stdout: During handling of the above exception, another exception occurred: 2022-05-14T00:57:28.617 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T00:57:28.617 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T00:57:28.617 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T00:57:28.618 INFO:journalctl@ceph.mon.c.smithi115.stdout: return f(*arg) 2022-05-14T00:57:28.618 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T00:57:28.618 INFO:journalctl@ceph.mon.c.smithi115.stdout: self._write_client_files(client_files, host) 2022-05-14T00:57:28.618 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T00:57:28.618 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T00:57:28.619 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T00:57:28.619 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T00:57:28.619 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T00:57:28.619 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.event_loop.get_result(coro) 2022-05-14T00:57:28.620 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T00:57:28.620 INFO:journalctl@ceph.mon.c.smithi115.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T00:57:28.620 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T00:57:28.620 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.__get_result() 2022-05-14T00:57:28.620 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T00:57:28.621 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise self._exception 2022-05-14T00:57:28.621 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T00:57:28.621 INFO:journalctl@ceph.mon.c.smithi115.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T00:57:28.621 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T00:57:28.622 INFO:journalctl@ceph.mon.c.smithi115.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T00:57:28.622 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T00:57:28.622 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise 2022-05-14T00:57:28.622 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T00:57:28.622 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.gen.throw(type, value, traceback) 2022-05-14T00:57:28.623 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T00:57:28.623 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OrchestratorError(msg) 2022-05-14T00:57:28.623 INFO:journalctl@ceph.mon.c.smithi115.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:57:28.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:57:28 smithi047 ceph-mon[77147]: pgmap v3043: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:57:28.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:57:28 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:57:28.791 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T00:57:28.792 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:57:28.792 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T00:57:28.792 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:57:28.792 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:57:28.793 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:57:28.793 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T00:57:28.793 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:57:28.793 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T00:57:28.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:57:28.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T00:57:28.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:57:28.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T00:57:28.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:57:28.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T00:57:28.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:57:28.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:57:28.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:57:28.796 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:57:28 smithi047 ceph-mon[77147]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T00:57:28.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T00:57:28.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:57:28.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T00:57:28.797 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:57:28.797 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:57:28.797 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:57:28.797 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T00:57:28.799 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:57:28.799 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T00:57:28.800 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:57:28.800 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T00:57:28.800 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:57:28.800 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T00:57:28.801 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:57:28.801 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T00:57:28.801 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:57:28.801 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:57:28.801 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:57:28.802 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T00:57:28.802 INFO:journalctl@ceph.mon.b.smithi047.stdout: During handling of the above exception, another exception occurred: 2022-05-14T00:57:28.802 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T00:57:28.802 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T00:57:28.803 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T00:57:28.803 INFO:journalctl@ceph.mon.b.smithi047.stdout: return f(*arg) 2022-05-14T00:57:28.803 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T00:57:28.803 INFO:journalctl@ceph.mon.b.smithi047.stdout: self._write_client_files(client_files, host) 2022-05-14T00:57:28.804 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T00:57:28.804 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T00:57:28.804 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T00:57:28.804 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T00:57:28.804 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T00:57:28.805 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.event_loop.get_result(coro) 2022-05-14T00:57:28.805 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T00:57:28.805 INFO:journalctl@ceph.mon.b.smithi047.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T00:57:28.805 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T00:57:28.806 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.__get_result() 2022-05-14T00:57:28.806 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T00:57:28.806 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise self._exception 2022-05-14T00:57:28.807 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T00:57:28.807 INFO:journalctl@ceph.mon.b.smithi047.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T00:57:28.807 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T00:57:28.807 INFO:journalctl@ceph.mon.b.smithi047.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T00:57:28.807 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T00:57:28.808 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise 2022-05-14T00:57:28.808 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T00:57:28.808 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.gen.throw(type, value, traceback) 2022-05-14T00:57:28.808 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T00:57:28.808 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OrchestratorError(msg) 2022-05-14T00:57:28.809 INFO:journalctl@ceph.mon.b.smithi047.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:57:29.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:57:29 smithi115 conmon[90393]: 2022-05-14T00:57:29.001+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:57:29.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:57:29 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:57:29.851 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:57:29 smithi047 ceph-mon[77147]: pgmap v3044: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:57:29.851 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:57:29 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:57:29.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:57:29 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:57:29.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:57:29 smithi115 ceph-mon[78593]: pgmap v3044: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:57:29.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:57:29 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:57:30.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:57:29 smithi115 conmon[90393]: 2022-05-14T00:57:29.998+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:57:30.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:57:30 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 5380 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:57:30.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:57:30 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:57:30.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:57:30 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:57:30.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:57:30 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:57:30.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:57:30 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 5380 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:57:30.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:57:30 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:57:30.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:57:30 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:57:30.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:57:30 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:57:31.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:57:30 smithi115 conmon[90393]: 2022-05-14T00:57:30.959+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:57:31.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:57:31 smithi047 ceph-mon[77147]: pgmap v3045: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:57:31.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:57:31 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:57:31.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:57:31 smithi115 ceph-mon[78593]: pgmap v3045: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:57:31.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:57:31 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:57:32.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:57:31 smithi115 conmon[90393]: 2022-05-14T00:57:31.967+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:57:32.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:57:32 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:57:32.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:57:32 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:57:33.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:57:32 smithi115 conmon[90393]: 2022-05-14T00:57:32.932+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:57:33.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:57:33 smithi047 ceph-mon[77147]: pgmap v3046: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:57:33.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:57:33 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:57:33.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:57:33 smithi115 ceph-mon[78593]: pgmap v3046: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:57:33.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:57:33 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:57:34.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:57:33 smithi115 conmon[90393]: 2022-05-14T00:57:33.980+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:57:34.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:57:34 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:57:34.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:57:34 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:57:34.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:57:34 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:57:34.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:57:34 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 5385 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:57:34.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:57:34 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:57:34.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:57:34 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:57:34.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:57:34 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:57:34.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:57:34 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 5385 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:57:35.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:57:34 smithi115 conmon[90393]: 2022-05-14T00:57:34.986+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:57:35.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:57:35 smithi047 ceph-mon[77147]: pgmap v3047: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:57:35.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:57:35 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:57:35.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:57:35 smithi115 ceph-mon[78593]: pgmap v3047: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:57:35.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:57:35 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:57:36.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:57:36 smithi115 conmon[90393]: 2022-05-14T00:57:36.012+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:57:36.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:57:36 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:57:36.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:57:36 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:57:37.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:57:37 smithi115 conmon[90393]: 2022-05-14T00:57:37.060+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:57:37.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:57:37 smithi047 ceph-mon[77147]: pgmap v3048: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:57:37.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:57:37 smithi115 ceph-mon[78593]: pgmap v3048: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:57:38.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:57:38 smithi115 conmon[90393]: 2022-05-14T00:57:38.095+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:57:38.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:57:38 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:57:38.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:57:38 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:57:38.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:57:38 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:57:38.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:57:38 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:57:38.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:57:38 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:57:38.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:57:38 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:57:39.403 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:57:39 smithi115 conmon[90393]: 2022-05-14T00:57:39.140+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:57:39.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:57:39 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:57:39.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:57:39 smithi047 ceph-mon[77147]: pgmap v3049: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:57:39.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:57:39 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 5390 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:57:39.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:57:39 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:57:39.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:57:39 smithi115 ceph-mon[78593]: pgmap v3049: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:57:39.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:57:39 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 5390 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:57:40.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:57:40 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:57:40.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:57:40 smithi115 conmon[90393]: 2022-05-14T00:57:40.159+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:57:40.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:57:40 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:57:41.411 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:57:41 smithi115 conmon[90393]: 2022-05-14T00:57:41.138+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:57:41.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:57:41 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:57:41.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:57:41 smithi047 ceph-mon[77147]: pgmap v3050: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:57:41.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:57:41 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:57:41.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:57:41 smithi115 ceph-mon[78593]: pgmap v3050: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:57:42.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:57:42 smithi115 conmon[90393]: 2022-05-14T00:57:42.104+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:57:42.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:57:42 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:57:42.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:57:42 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:57:42.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:57:42 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:57:42.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:57:42 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:57:42.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:57:42 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:57:42.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:57:42 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:57:43.416 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:57:43 smithi115 conmon[90393]: 2022-05-14T00:57:43.109+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:57:43.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:57:43 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:57:43.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:57:43 smithi047 ceph-mon[77147]: pgmap v3051: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:57:43.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:57:43 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:57:43.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:57:43 smithi115 ceph-mon[78593]: pgmap v3051: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:57:44.418 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:57:44 smithi115 conmon[90393]: 2022-05-14T00:57:44.138+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:57:44.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:57:44 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:57:44.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:57:44 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 5395 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:57:44.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:57:44 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:57:44.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:57:44 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 5395 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:57:45.423 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:57:45 smithi115 conmon[90393]: 2022-05-14T00:57:45.138+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:57:45.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:57:45 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:57:45.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:57:45 smithi047 ceph-mon[77147]: pgmap v3052: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:57:45.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:57:45 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:57:45.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:57:45 smithi115 ceph-mon[78593]: pgmap v3052: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:57:46.425 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:57:46 smithi115 conmon[90393]: 2022-05-14T00:57:46.153+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:57:46.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:57:46 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:57:46.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:57:46 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:57:47.430 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:57:47 smithi115 conmon[90393]: 2022-05-14T00:57:47.132+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:57:47.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:57:47 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:57:47.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:57:47 smithi047 ceph-mon[77147]: pgmap v3053: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:57:47.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:57:47 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:57:47.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:57:47 smithi115 ceph-mon[78593]: pgmap v3053: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:57:48.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:57:48 smithi115 conmon[90393]: 2022-05-14T00:57:48.086+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:57:48.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:57:48 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:57:48.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:57:48 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:57:49.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:57:49 smithi115 conmon[90393]: 2022-05-14T00:57:49.046+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:57:49.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:57:49 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:57:49.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:57:49 smithi047 ceph-mon[77147]: pgmap v3054: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:57:49.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:57:49 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 5400 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:57:49.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:57:49 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:57:49.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:57:49 smithi115 ceph-mon[78593]: pgmap v3054: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:57:49.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:57:49 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 5400 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:57:50.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:57:50 smithi115 conmon[90393]: 2022-05-14T00:57:50.060+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:57:50.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:57:50 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:57:50.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:57:50 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:57:50.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:57:50 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:57:50.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:57:50 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:57:50.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:57:50 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:57:50.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:57:50 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:57:51.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:57:51 smithi115 conmon[90393]: 2022-05-14T00:57:51.106+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:57:51.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:57:51 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:57:51.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:57:51 smithi047 ceph-mon[77147]: pgmap v3055: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:57:51.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:57:51 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:57:51.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:57:51 smithi115 ceph-mon[78593]: pgmap v3055: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:57:52.443 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:57:52 smithi115 conmon[90393]: 2022-05-14T00:57:52.115+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:57:52.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:57:52 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:57:52.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:57:52 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:57:53.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:57:53 smithi115 conmon[90393]: 2022-05-14T00:57:53.100+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:57:53.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:57:53 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:57:53.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:57:53 smithi047 ceph-mon[77147]: pgmap v3056: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:57:53.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:57:53 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:57:53.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:57:53 smithi115 ceph-mon[78593]: pgmap v3056: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:57:54.451 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:57:54 smithi115 conmon[90393]: 2022-05-14T00:57:54.123+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:57:54.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:57:54 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:57:54.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:57:54 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:57:54.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:57:54 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:57:54.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:57:54 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 5405 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:57:54.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:57:54 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:57:54.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:57:54 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:57:54.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:57:54 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:57:54.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:57:54 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 5405 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:57:55.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:57:55 smithi115 conmon[90393]: 2022-05-14T00:57:55.078+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:57:55.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:57:55 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:57:55.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:57:55 smithi047 ceph-mon[77147]: pgmap v3057: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:57:55.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:57:55 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:57:55.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:57:55 smithi115 ceph-mon[78593]: pgmap v3057: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:57:56.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:57:56 smithi115 conmon[90393]: 2022-05-14T00:57:56.048+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:57:56.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:57:56 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:57:56.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:57:56 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:57:57.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:57:57 smithi115 conmon[90393]: 2022-05-14T00:57:57.070+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:57:57.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:57:57 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:57:57.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:57:57 smithi047 ceph-mon[77147]: pgmap v3058: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:57:57.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:57:57 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:57:57.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:57:57 smithi115 ceph-mon[78593]: pgmap v3058: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:57:58.464 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:57:58 smithi115 conmon[90393]: 2022-05-14T00:57:58.107+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:57:58.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:57:58 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:57:58.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:57:58 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:57:58.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:57:58 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:57:58.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:57:58 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:57:58.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:57:58 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:57:58.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:57:58 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:57:59.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:57:59 smithi115 conmon[90393]: 2022-05-14T00:57:59.084+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:57:59.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:57:59 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:57:59.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:57:59 smithi047 ceph-mon[77147]: pgmap v3059: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:57:59.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:57:59 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 5410 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:57:59.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:57:59 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:57:59.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:57:59 smithi115 ceph-mon[78593]: pgmap v3059: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:57:59.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:57:59 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 5410 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:58:00.469 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:58:00 smithi115 conmon[90393]: 2022-05-14T00:58:00.123+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:58:00.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:58:00 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:58:00.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:58:00 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:58:01.477 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:58:01 smithi115 conmon[90393]: 2022-05-14T00:58:01.128+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:58:01.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:58:01 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:58:01.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:58:01 smithi047 ceph-mon[77147]: pgmap v3060: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:58:01.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:58:01 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:58:01.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:58:01 smithi115 ceph-mon[78593]: pgmap v3060: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:58:02.476 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:58:02 smithi115 conmon[90393]: 2022-05-14T00:58:02.117+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:58:02.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:58:02 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:58:02.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:58:02 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:58:02.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:58:02 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:58:02.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:58:02 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:58:02.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:58:02 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:58:02.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:58:02 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:58:03.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:58:03 smithi115 conmon[90393]: 2022-05-14T00:58:03.097+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:58:03.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:58:03 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:58:03.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:58:03 smithi047 ceph-mon[77147]: pgmap v3061: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:58:03.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:58:03 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:58:03.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:58:03 smithi115 ceph-mon[78593]: pgmap v3061: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:58:04.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:58:04 smithi115 conmon[90393]: 2022-05-14T00:58:04.059+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:58:04.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:58:04 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:58:04.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:58:04 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 5415 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:58:04.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:58:04 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:58:04.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:58:04 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 5415 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:58:05.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:58:05 smithi115 conmon[90393]: 2022-05-14T00:58:05.032+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:58:05.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:58:05 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:58:05.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:58:05 smithi047 ceph-mon[77147]: pgmap v3062: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:58:05.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:58:05 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:58:05.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:58:05 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:58:05.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:58:05 smithi115 ceph-mon[78593]: pgmap v3062: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:58:05.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:58:05 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:58:06.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:58:05 smithi115 conmon[90393]: 2022-05-14T00:58:05.984+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:58:06.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:58:06 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:58:06.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:58:06 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:58:07.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:58:06 smithi115 conmon[90393]: 2022-05-14T00:58:06.949+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:58:07.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:58:07 smithi047 ceph-mon[77147]: pgmap v3063: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:58:07.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:58:07 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:58:07.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:58:07 smithi115 ceph-mon[78593]: pgmap v3063: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:58:07.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:58:07 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:58:08.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:58:07 smithi115 conmon[90393]: 2022-05-14T00:58:07.942+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:58:08.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:58:08 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:58:08.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:58:08 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:58:09.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:58:08 smithi115 conmon[90393]: 2022-05-14T00:58:08.948+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:58:09.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:58:09 smithi047 ceph-mon[77147]: pgmap v3064: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:58:09.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:58:09 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:58:09.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:58:09 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 5420 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:58:09.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:58:09 smithi115 ceph-mon[78593]: pgmap v3064: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:58:09.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:58:09 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:58:09.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:58:09 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 5420 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:58:10.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:58:09 smithi115 conmon[90393]: 2022-05-14T00:58:09.948+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:58:10.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:58:10 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:58:10.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:58:10 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:58:10.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:58:10 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:58:10.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:58:10 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:58:10.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:58:10 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:58:10.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:58:10 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:58:11.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:58:10 smithi115 conmon[90393]: 2022-05-14T00:58:10.938+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:58:11.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:58:11 smithi047 ceph-mon[77147]: pgmap v3065: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:58:11.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:58:11 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:58:11.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:58:11 smithi115 ceph-mon[78593]: pgmap v3065: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:58:11.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:58:11 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:58:12.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:58:11 smithi115 conmon[90393]: 2022-05-14T00:58:11.968+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:58:12.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:58:12 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:58:12.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:58:12 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:58:13.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:58:12 smithi115 conmon[90393]: 2022-05-14T00:58:12.942+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:58:13.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:58:13 smithi047 ceph-mon[77147]: pgmap v3066: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:58:13.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:58:13 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:58:13.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:58:13 smithi115 ceph-mon[78593]: pgmap v3066: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:58:13.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:58:13 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:58:14.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:58:13 smithi115 conmon[90393]: 2022-05-14T00:58:13.952+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:58:14.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:58:14 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:58:14.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:58:14 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:58:14.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:58:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T00:58:14.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:58:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T00:58:14.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:58:14 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:58:14.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:58:14 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 5425 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:58:14.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:58:14 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:58:14.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:58:14 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:58:14.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:58:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T00:58:14.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:58:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T00:58:14.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:58:14 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:58:14.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:58:14 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 5425 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:58:15.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:58:14 smithi115 conmon[90393]: 2022-05-14T00:58:14.915+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:58:15.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:58:15 smithi047 ceph-mon[77147]: pgmap v3067: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:58:15.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:58:15 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:58:15.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:58:15 smithi115 ceph-mon[78593]: pgmap v3067: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:58:15.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:58:15 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:58:16.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:58:15 smithi115 conmon[90393]: 2022-05-14T00:58:15.893+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:58:16.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:58:16 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:58:16.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:58:16 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:58:17.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:58:16 smithi115 conmon[90393]: 2022-05-14T00:58:16.941+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:58:17.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:58:17 smithi047 ceph-mon[77147]: pgmap v3068: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:58:17.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:58:17 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:58:17.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:58:17 smithi115 ceph-mon[78593]: pgmap v3068: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:58:17.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:58:17 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:58:18.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:58:17 smithi115 conmon[90393]: 2022-05-14T00:58:17.899+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:58:18.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:58:18 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:58:18.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:58:18 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:58:18.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:58:18 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:58:18.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:58:18 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:58:18.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:58:18 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:58:18.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:58:18 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:58:19.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:58:18 smithi115 conmon[90393]: 2022-05-14T00:58:18.859+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:58:19.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:58:19 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:58:19.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:58:19 smithi047 ceph-mon[77147]: pgmap v3069: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:58:19.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:58:19 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 5430 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:58:19.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:58:19 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:58:19.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:58:19 smithi115 ceph-mon[78593]: pgmap v3069: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:58:19.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:58:19 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 5430 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:58:20.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:58:19 smithi115 conmon[90393]: 2022-05-14T00:58:19.871+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:58:20.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:58:20 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:58:20.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:58:20 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:58:21.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:58:20 smithi115 conmon[90393]: 2022-05-14T00:58:20.893+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:58:21.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:58:21 smithi047 ceph-mon[77147]: pgmap v3070: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:58:21.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:58:21 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:58:21.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:58:21 smithi115 ceph-mon[78593]: pgmap v3070: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:58:21.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:58:21 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:58:22.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:58:21 smithi115 conmon[90393]: 2022-05-14T00:58:21.914+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:58:22.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:58:22 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:58:22.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:58:22 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:58:22.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:58:22 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:58:22.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:58:22 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:58:22.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:58:22 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:58:22.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:58:22 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:58:23.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:58:22 smithi115 conmon[90393]: 2022-05-14T00:58:22.895+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:58:23.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:58:23 smithi115 ceph-mon[78593]: pgmap v3071: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:58:23.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:58:23 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:58:24.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:58:23 smithi047 ceph-mon[77147]: pgmap v3071: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:58:24.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:58:23 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:58:24.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:58:23 smithi115 conmon[90393]: 2022-05-14T00:58:23.880+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:58:24.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:58:24 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:58:24.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:58:24 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 5435 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:58:25.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:58:24 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:58:25.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:58:24 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 5435 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:58:25.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:58:24 smithi115 conmon[90393]: 2022-05-14T00:58:24.911+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:58:25.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:58:25 smithi115 ceph-mon[78593]: pgmap v3072: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:58:25.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:58:25 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:58:26.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:58:25 smithi047 ceph-mon[77147]: pgmap v3072: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:58:26.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:58:25 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:58:26.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:58:25 smithi115 conmon[90393]: 2022-05-14T00:58:25.881+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:58:26.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:58:26 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:58:27.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:58:26 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:58:27.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:58:26 smithi115 conmon[90393]: 2022-05-14T00:58:26.916+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:58:27.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:58:27 smithi115 ceph-mon[78593]: pgmap v3073: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:58:27.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:58:27 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:58:28.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:58:27 smithi047 ceph-mon[77147]: pgmap v3073: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:58:28.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:58:27 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:58:28.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:58:27 smithi115 conmon[90393]: 2022-05-14T00:58:27.899+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:58:28.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:58:28 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:58:28.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:58:28 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T00:58:28.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:58:28 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T00:58:28.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:58:28 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T00:58:28.792 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:58:28 smithi047 conmon[78521]: 2022-05-14T00:58:28.464+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:58:28.792 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:58:28 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T00:58:28.793 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:58:28 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:58:28.793 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:58:28 smithi047 conmon[78521]: yield 2022-05-14T00:58:28.793 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:58:28 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:58:28.793 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:58:28 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:58:28.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:58:28 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:58:28.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:58:28 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T00:58:28.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:58:28 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:58:28.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:58:28 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T00:58:28.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:58:28 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:58:28.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:58:28 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T00:58:28.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:58:28 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:58:28.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:58:28 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T00:58:28.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:58:28 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:58:28.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:58:28 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T00:58:28.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:58:28 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:58:28.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:58:28 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:58:28.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:58:28 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:58:28.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:58:28 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:58:28.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:58:28 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T00:58:28.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:58:28 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T00:58:28.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:58:28 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T00:58:29.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:58:28 smithi115 conmon[90393]: 2022-05-14T00:58:28.890+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:58:29.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:58:29 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:58:29.858 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T00:58:29.858 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:58:29.858 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T00:58:29.858 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:58:29.859 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:58:29.859 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:58:29.859 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T00:58:29.859 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:58:29.860 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T00:58:29.860 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:58:29.860 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T00:58:29.860 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:58:29.860 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T00:58:29.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:58:29.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T00:58:29.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:58:29.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:58:29.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:58:29.862 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:58:29 smithi115 ceph-mon[78593]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T00:58:29.862 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:58:29 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:58:29.862 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:58:29 smithi115 ceph-mon[78593]: pgmap v3074: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:58:29.862 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:58:29 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 5440 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:58:30.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:58:29 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:58:30.041 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T00:58:30.041 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:58:30.042 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T00:58:30.042 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:58:30.042 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:58:30.042 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:58:30.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T00:58:30.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:58:30.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T00:58:30.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:58:30.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T00:58:30.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:58:30.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T00:58:30.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:58:30.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T00:58:30.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:58:30.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:58:30.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:58:30.045 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:58:29 smithi047 ceph-mon[77147]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T00:58:30.046 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:58:29 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:58:30.046 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:58:29 smithi047 ceph-mon[77147]: pgmap v3074: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:58:30.046 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:58:29 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 5440 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:58:30.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:58:29 smithi115 conmon[90393]: 2022-05-14T00:58:29.931+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:58:30.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:58:30 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:58:30.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:58:30 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:58:30.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:58:30 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:58:31.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:58:30 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:58:31.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:58:30 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:58:31.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:58:30 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:58:31.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:58:30 smithi115 conmon[90393]: 2022-05-14T00:58:30.922+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:58:31.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:58:31 smithi047 ceph-mon[77147]: pgmap v3075: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:58:31.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:58:31 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:58:31.792 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:58:31 smithi047 conmon[78521]: 2022-05-14T00:58:31.536+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:58:31.792 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:58:31 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T00:58:31.792 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:58:31 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:58:31.792 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:58:31 smithi047 conmon[78521]: yield 2022-05-14T00:58:31.793 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:58:31 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:58:31.793 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:58:31 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:58:31.793 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:58:31 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:58:31.793 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:58:31 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T00:58:31.793 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:58:31 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:58:31.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:58:31 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T00:58:31.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:58:31 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:58:31.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:58:31 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T00:58:31.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:58:31 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:58:31.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:58:31 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T00:58:31.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:58:31 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:58:31.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:58:31 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T00:58:31.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:58:31 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:58:31.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:58:31 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:58:31.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:58:31 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:58:31.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:58:31 smithi047 conmon[78521]: 2022-05-14T00:58:31.537+0000 7f5340f7a700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T00:58:31.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:58:31 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T00:58:31.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:58:31 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:58:31.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:58:31 smithi047 conmon[78521]: yield 2022-05-14T00:58:31.797 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:58:31 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:58:31.797 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:58:31 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:58:31.797 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:58:31 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:58:31.797 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:58:31 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T00:58:31.798 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:58:31 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:58:31.798 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:58:31 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T00:58:31.798 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:58:31 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:58:31.798 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:58:31 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T00:58:31.798 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:58:31 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:58:31.799 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:58:31 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T00:58:31.799 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:58:31 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:58:31.799 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:58:31 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T00:58:31.799 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:58:31 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:58:31.800 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:58:31 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:58:31.800 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:58:31 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:58:31.800 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:58:31 smithi047 conmon[78521]: 2022-05-14T00:58:31.800 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:58:31 smithi047 conmon[78521]: During handling of the above exception, another exception occurred: 2022-05-14T00:58:31.800 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:58:31 smithi047 conmon[78521]: 2022-05-14T00:58:31.801 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:58:31 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T00:58:31.801 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:58:31 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T00:58:31.801 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:58:31 smithi047 conmon[78521]: return f(*arg) 2022-05-14T00:58:31.801 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:58:31 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T00:58:31.802 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:58:31 smithi047 conmon[78521]: self._write_client_files(client_files, host) 2022-05-14T00:58:31.802 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:58:31 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T00:58:31.802 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:58:31 smithi047 conmon[78521]: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T00:58:31.802 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:58:31 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T00:58:31.802 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:58:31 smithi047 conmon[78521]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T00:58:31.803 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:58:31 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T00:58:31.803 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:58:31 smithi047 conmon[78521]: return self.event_loop.get_result(coro) 2022-05-14T00:58:31.803 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:58:31 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T00:58:31.803 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:58:31 smithi047 conmon[78521]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T00:58:31.804 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:58:31 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T00:58:31.804 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:58:31 smithi047 conmon[78521]: return self.__get_result() 2022-05-14T00:58:31.804 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:58:31 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T00:58:31.804 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:58:31 smithi047 conmon[78521]: raise self._exception 2022-05-14T00:58:31.804 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:58:31 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T00:58:31.805 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:58:31 smithi047 conmon[78521]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T00:58:31.805 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:58:31 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T00:58:31.805 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:58:31 smithi047 conmon[78521]: conn = await self._remote_connection(host, addr) 2022-05-14T00:58:31.805 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:58:31 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T00:58:31.806 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:58:31 smithi047 conmon[78521]: raise 2022-05-14T00:58:31.806 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:58:31 smithi047 conmon[78521]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T00:58:31.806 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:58:31 smithi047 conmon[78521]: self.gen.throw(type, value, traceback) 2022-05-14T00:58:31.806 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:58:31 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T00:58:31.806 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:58:31 smithi047 conmon[78521]: raise OrchestratorError(msg) 2022-05-14T00:58:31.807 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:58:31 smithi047 conmon[78521]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:58:31.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:58:31 smithi115 ceph-mon[78593]: pgmap v3075: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:58:31.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:58:31 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:58:32.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:58:31 smithi115 conmon[90393]: 2022-05-14T00:58:31.962+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:58:32.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:58:32 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:58:32.858 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T00:58:32.858 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:58:32.858 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T00:58:32.858 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:58:32.859 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:58:32.859 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:58:32.859 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T00:58:32.859 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:58:32.860 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T00:58:32.860 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:58:32.860 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T00:58:32.860 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:58:32.860 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T00:58:32.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:58:32.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T00:58:32.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:58:32.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:58:32.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:58:32.862 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:58:32 smithi115 ceph-mon[78593]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T00:58:32.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T00:58:32.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:58:32.863 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T00:58:32.863 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:58:32.863 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:58:32.864 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:58:32.864 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T00:58:32.864 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:58:32.864 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T00:58:32.865 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:58:32.865 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T00:58:32.865 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:58:32.865 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T00:58:32.866 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:58:32.866 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T00:58:32.866 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:58:32.866 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:58:32.866 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:58:32.867 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T00:58:32.867 INFO:journalctl@ceph.mon.c.smithi115.stdout: During handling of the above exception, another exception occurred: 2022-05-14T00:58:32.867 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T00:58:32.867 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T00:58:32.867 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T00:58:32.868 INFO:journalctl@ceph.mon.c.smithi115.stdout: return f(*arg) 2022-05-14T00:58:32.868 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T00:58:32.868 INFO:journalctl@ceph.mon.c.smithi115.stdout: self._write_client_files(client_files, host) 2022-05-14T00:58:32.868 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T00:58:32.869 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T00:58:32.869 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T00:58:32.869 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T00:58:32.869 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T00:58:32.869 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.event_loop.get_result(coro) 2022-05-14T00:58:32.870 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T00:58:32.870 INFO:journalctl@ceph.mon.c.smithi115.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T00:58:32.870 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T00:58:32.870 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.__get_result() 2022-05-14T00:58:32.870 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T00:58:32.871 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise self._exception 2022-05-14T00:58:32.871 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T00:58:32.871 INFO:journalctl@ceph.mon.c.smithi115.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T00:58:32.871 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T00:58:32.872 INFO:journalctl@ceph.mon.c.smithi115.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T00:58:32.872 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T00:58:32.873 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise 2022-05-14T00:58:32.873 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T00:58:32.873 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.gen.throw(type, value, traceback) 2022-05-14T00:58:32.873 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T00:58:32.873 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OrchestratorError(msg) 2022-05-14T00:58:32.874 INFO:journalctl@ceph.mon.c.smithi115.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:58:32.874 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:58:32 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:58:33.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:58:32 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:58:33.041 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T00:58:33.042 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:58:33.042 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T00:58:33.042 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:58:33.042 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:58:33.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:58:33.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T00:58:33.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:58:33.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T00:58:33.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:58:33.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T00:58:33.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:58:33.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T00:58:33.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:58:33.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T00:58:33.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:58:33.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:58:33.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:58:33.046 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:58:32 smithi047 ceph-mon[77147]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T00:58:33.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T00:58:33.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:58:33.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T00:58:33.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:58:33.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:58:33.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:58:33.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T00:58:33.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:58:33.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T00:58:33.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:58:33.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T00:58:33.049 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:58:33.049 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T00:58:33.049 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:58:33.049 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T00:58:33.050 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:58:33.050 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:58:33.050 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:58:33.050 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T00:58:33.050 INFO:journalctl@ceph.mon.b.smithi047.stdout: During handling of the above exception, another exception occurred: 2022-05-14T00:58:33.051 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T00:58:33.051 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T00:58:33.051 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T00:58:33.051 INFO:journalctl@ceph.mon.b.smithi047.stdout: return f(*arg) 2022-05-14T00:58:33.051 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T00:58:33.052 INFO:journalctl@ceph.mon.b.smithi047.stdout: self._write_client_files(client_files, host) 2022-05-14T00:58:33.052 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T00:58:33.052 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T00:58:33.053 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T00:58:33.053 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T00:58:33.053 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T00:58:33.053 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.event_loop.get_result(coro) 2022-05-14T00:58:33.054 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T00:58:33.054 INFO:journalctl@ceph.mon.b.smithi047.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T00:58:33.054 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T00:58:33.054 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.__get_result() 2022-05-14T00:58:33.055 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T00:58:33.055 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise self._exception 2022-05-14T00:58:33.055 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T00:58:33.055 INFO:journalctl@ceph.mon.b.smithi047.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T00:58:33.055 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T00:58:33.056 INFO:journalctl@ceph.mon.b.smithi047.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T00:58:33.056 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T00:58:33.056 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise 2022-05-14T00:58:33.056 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T00:58:33.057 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.gen.throw(type, value, traceback) 2022-05-14T00:58:33.057 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T00:58:33.057 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OrchestratorError(msg) 2022-05-14T00:58:33.057 INFO:journalctl@ceph.mon.b.smithi047.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:58:33.058 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:58:32 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:58:33.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:58:32 smithi115 conmon[90393]: 2022-05-14T00:58:32.931+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:58:33.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:58:33 smithi115 ceph-mon[78593]: pgmap v3076: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:58:33.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:58:33 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:58:34.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:58:33 smithi047 ceph-mon[77147]: pgmap v3076: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:58:34.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:58:33 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:58:34.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:58:33 smithi115 conmon[90393]: 2022-05-14T00:58:33.972+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:58:34.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:58:34 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:58:34.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:58:34 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:58:34.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:58:34 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:58:34.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:58:34 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 5445 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:58:35.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:58:34 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:58:35.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:58:34 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:58:35.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:58:34 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:58:35.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:58:34 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 5445 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:58:35.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:58:34 smithi115 conmon[90393]: 2022-05-14T00:58:34.977+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:58:35.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:58:35 smithi115 ceph-mon[78593]: pgmap v3077: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:58:35.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:58:35 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:58:36.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:58:35 smithi047 ceph-mon[77147]: pgmap v3077: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:58:36.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:58:35 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:58:36.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:58:35 smithi115 conmon[90393]: 2022-05-14T00:58:35.968+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:58:36.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:58:36 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:58:37.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:58:36 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:58:37.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:58:36 smithi115 conmon[90393]: 2022-05-14T00:58:36.947+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:58:37.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:58:37 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:58:37.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:58:37 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:58:37.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:58:37 smithi115 ceph-mon[78593]: pgmap v3078: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:58:37.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:58:37 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:58:38.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:58:37 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:58:38.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:58:37 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:58:38.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:58:37 smithi047 ceph-mon[77147]: pgmap v3078: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:58:38.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:58:37 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:58:38.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:58:37 smithi115 conmon[90393]: 2022-05-14T00:58:37.945+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:58:38.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:58:38 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:58:39.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:58:38 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:58:39.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:58:38 smithi115 conmon[90393]: 2022-05-14T00:58:38.900+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:58:39.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:58:39 smithi115 ceph-mon[78593]: pgmap v3079: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:58:39.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:58:39 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:58:39.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:58:39 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 5450 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:58:40.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:58:39 smithi047 ceph-mon[77147]: pgmap v3079: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:58:40.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:58:39 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:58:40.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:58:39 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 5450 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:58:40.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:58:39 smithi115 conmon[90393]: 2022-05-14T00:58:39.856+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:58:40.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:58:40 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:58:40.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:58:40 smithi115 conmon[90393]: 2022-05-14T00:58:40.843+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:58:41.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:58:40 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:58:42.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:58:41 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:58:42.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:58:41 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:58:42.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:58:41 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:58:42.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:58:41 smithi047 ceph-mon[77147]: pgmap v3080: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:58:42.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:58:41 smithi115 conmon[90393]: 2022-05-14T00:58:41.810+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:58:42.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:58:41 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:58:42.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:58:41 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:58:42.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:58:41 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:58:42.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:58:41 smithi115 ceph-mon[78593]: pgmap v3080: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:58:43.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:58:42 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:58:43.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:58:42 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:58:43.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:58:42 smithi115 conmon[90393]: 2022-05-14T00:58:42.776+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:58:44.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:58:43 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:58:44.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:58:43 smithi047 ceph-mon[77147]: pgmap v3081: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-05-14T00:58:44.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:58:43 smithi115 conmon[90393]: 2022-05-14T00:58:43.769+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:58:44.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:58:43 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:58:44.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:58:43 smithi115 ceph-mon[78593]: pgmap v3081: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.4 KiB/s rd, 4 op/s 2022-05-14T00:58:45.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:58:44 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:58:45.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:58:44 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 5455 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:58:45.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:58:44 smithi115 conmon[90393]: 2022-05-14T00:58:44.788+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:58:45.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:58:44 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:58:45.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:58:44 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 5455 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:58:46.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:58:45 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:58:46.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:58:45 smithi047 ceph-mon[77147]: pgmap v3082: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 7 op/s 2022-05-14T00:58:46.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:58:45 smithi115 conmon[90393]: 2022-05-14T00:58:45.795+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:58:46.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:58:45 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:58:46.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:58:45 smithi115 ceph-mon[78593]: pgmap v3082: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 7 op/s 2022-05-14T00:58:47.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:58:46 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:58:47.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:58:46 smithi115 conmon[90393]: 2022-05-14T00:58:46.830+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:58:47.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:58:46 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:58:48.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:58:47 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:58:48.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:58:47 smithi047 ceph-mon[77147]: pgmap v3083: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 4 op/s 2022-05-14T00:58:48.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:58:47 smithi115 conmon[90393]: 2022-05-14T00:58:47.837+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:58:48.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:58:47 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:58:48.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:58:47 smithi115 ceph-mon[78593]: pgmap v3083: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 4 op/s 2022-05-14T00:58:49.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:58:48 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:58:49.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:58:48 smithi115 conmon[90393]: 2022-05-14T00:58:48.799+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:58:49.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:58:48 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:58:50.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:58:49 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:58:50.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:58:49 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:58:50.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:58:49 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:58:50.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:58:49 smithi047 ceph-mon[77147]: pgmap v3084: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 4 op/s 2022-05-14T00:58:50.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:58:49 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 5460 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:58:50.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:58:49 smithi115 conmon[90393]: 2022-05-14T00:58:49.791+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:58:50.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:58:49 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:58:50.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:58:49 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:58:50.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:58:49 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:58:50.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:58:49 smithi115 ceph-mon[78593]: pgmap v3084: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 4 op/s 2022-05-14T00:58:50.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:58:49 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 5460 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:58:51.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:58:50 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:58:51.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:58:50 smithi115 conmon[90393]: 2022-05-14T00:58:50.765+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:58:51.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:58:50 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:58:52.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:58:51 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:58:52.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:58:51 smithi047 ceph-mon[77147]: pgmap v3085: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 7 op/s 2022-05-14T00:58:52.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:58:51 smithi115 conmon[90393]: 2022-05-14T00:58:51.727+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:58:52.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:58:51 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:58:52.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:58:51 smithi115 ceph-mon[78593]: pgmap v3085: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 7 op/s 2022-05-14T00:58:53.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:58:52 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:58:53.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:58:52 smithi115 conmon[90393]: 2022-05-14T00:58:52.682+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:58:53.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:58:52 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:58:54.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:58:53 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:58:54.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:58:53 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:58:54.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:58:53 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:58:54.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:58:53 smithi047 ceph-mon[77147]: pgmap v3086: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.3 KiB/s rd, 4 op/s 2022-05-14T00:58:54.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:58:53 smithi115 conmon[90393]: 2022-05-14T00:58:53.661+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:58:54.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:58:53 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:58:54.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:58:53 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:58:54.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:58:53 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:58:54.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:58:53 smithi115 ceph-mon[78593]: pgmap v3086: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.3 KiB/s rd, 4 op/s 2022-05-14T00:58:55.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:58:54 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:58:55.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:58:54 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 5465 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:58:55.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:58:54 smithi115 conmon[90393]: 2022-05-14T00:58:54.648+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:58:55.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:58:54 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:58:55.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:58:54 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 5465 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:58:56.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:58:55 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:58:56.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:58:55 smithi047 ceph-mon[77147]: pgmap v3087: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-05-14T00:58:56.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:58:55 smithi115 conmon[90393]: 2022-05-14T00:58:55.644+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:58:56.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:58:55 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:58:56.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:58:55 smithi115 ceph-mon[78593]: pgmap v3087: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.6 KiB/s rd, 7 op/s 2022-05-14T00:58:57.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:58:56 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:58:57.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:58:56 smithi115 conmon[90393]: 2022-05-14T00:58:56.617+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:58:57.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:58:56 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:58:57.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:58:57 smithi115 conmon[90393]: 2022-05-14T00:58:57.588+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:58:57.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:58:57 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:58:57.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:58:57 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:58:57.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:58:57 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:58:57.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:58:57 smithi115 ceph-mon[78593]: pgmap v3088: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:58:58.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:58:57 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:58:58.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:58:57 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:58:58.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:58:57 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:58:58.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:58:57 smithi047 ceph-mon[77147]: pgmap v3088: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:58:59.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:58:58 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:58:59.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:58:58 smithi115 conmon[90393]: 2022-05-14T00:58:58.610+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:58:59.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:58:58 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:59:00.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:58:59 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:59:00.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:58:59 smithi047 ceph-mon[77147]: pgmap v3089: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:59:00.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:58:59 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 5470 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:59:00.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:58:59 smithi115 conmon[90393]: 2022-05-14T00:58:59.615+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:59:00.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:58:59 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:59:00.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:58:59 smithi115 ceph-mon[78593]: pgmap v3089: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:59:00.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:58:59 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 5470 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:59:00.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:59:00 smithi115 conmon[90393]: 2022-05-14T00:59:00.579+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:59:00.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:59:00 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:59:01.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:59:00 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:59:01.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:59:01 smithi115 conmon[90393]: 2022-05-14T00:59:01.544+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:59:01.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:59:01 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:59:01.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:59:01 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:59:01.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:59:01 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:59:01.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:59:01 smithi115 ceph-mon[78593]: pgmap v3090: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:59:02.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:59:01 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:59:02.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:59:01 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:59:02.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:59:01 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:59:02.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:59:01 smithi047 ceph-mon[77147]: pgmap v3090: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:59:02.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:59:02 smithi115 conmon[90393]: 2022-05-14T00:59:02.543+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:59:02.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:59:02 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:59:03.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:59:02 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:59:03.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:59:03 smithi115 conmon[90393]: 2022-05-14T00:59:03.524+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:59:03.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:59:03 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:59:03.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:59:03 smithi115 ceph-mon[78593]: pgmap v3091: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:59:04.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:59:03 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:59:04.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:59:03 smithi047 ceph-mon[77147]: pgmap v3091: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:59:04.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:59:04 smithi115 conmon[90393]: 2022-05-14T00:59:04.490+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:59:04.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:59:04 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:59:04.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:59:04 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 5475 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:59:05.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:59:04 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:59:05.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:59:04 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 5475 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:59:05.709 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:59:05 smithi115 conmon[90393]: 2022-05-14T00:59:05.455+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:59:06.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:59:05 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:59:06.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:59:05 smithi047 ceph-mon[77147]: pgmap v3092: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:59:06.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:59:05 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:59:06.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:59:05 smithi115 ceph-mon[78593]: pgmap v3092: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:59:06.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:59:06 smithi115 conmon[90393]: 2022-05-14T00:59:06.468+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:59:06.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:59:06 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:59:07.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:59:06 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:59:07.713 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:59:07 smithi115 conmon[90393]: 2022-05-14T00:59:07.435+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:59:08.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:59:07 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:59:08.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:59:07 smithi047 ceph-mon[77147]: pgmap v3093: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:59:08.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:59:07 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:59:08.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:59:07 smithi115 ceph-mon[78593]: pgmap v3093: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:59:08.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:59:08 smithi115 conmon[90393]: 2022-05-14T00:59:08.465+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:59:08.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:59:08 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:59:09.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:59:08 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:59:09.723 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:59:09 smithi115 conmon[90393]: 2022-05-14T00:59:09.431+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:59:10.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:59:09 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:59:10.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:59:09 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:59:10.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:59:09 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:59:10.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:59:09 smithi047 ceph-mon[77147]: pgmap v3094: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:59:10.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:59:09 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 5480 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:59:10.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:59:09 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:59:10.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:59:09 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:59:10.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:59:09 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:59:10.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:59:09 smithi115 ceph-mon[78593]: pgmap v3094: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:59:10.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:59:09 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 5480 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:59:10.725 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:59:10 smithi115 conmon[90393]: 2022-05-14T00:59:10.465+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:59:11.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:59:10 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:59:11.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:59:10 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:59:11.730 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:59:11 smithi115 conmon[90393]: 2022-05-14T00:59:11.416+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:59:12.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:59:11 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:59:12.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:59:11 smithi047 ceph-mon[77147]: pgmap v3095: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:59:12.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:59:11 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:59:12.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:59:11 smithi115 ceph-mon[78593]: pgmap v3095: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:59:12.737 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:59:12 smithi115 conmon[90393]: 2022-05-14T00:59:12.385+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:59:13.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:59:12 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:59:13.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:59:12 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:59:13.743 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:59:13 smithi115 conmon[90393]: 2022-05-14T00:59:13.391+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:59:14.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:59:13 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:59:14.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:59:13 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:59:14.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:59:13 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:59:14.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:59:13 smithi047 ceph-mon[77147]: pgmap v3096: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:59:14.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:59:13 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T00:59:14.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:59:13 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T00:59:14.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:59:13 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:59:14.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:59:13 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:59:14.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:59:13 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:59:14.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:59:13 smithi115 ceph-mon[78593]: pgmap v3096: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:59:14.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:59:13 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T00:59:14.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:59:13 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T00:59:14.741 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:59:14 smithi115 conmon[90393]: 2022-05-14T00:59:14.402+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:59:15.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:59:14 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:59:15.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:59:14 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 5485 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:59:15.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:59:14 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:59:15.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:59:14 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 5485 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:59:15.746 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:59:15 smithi115 conmon[90393]: 2022-05-14T00:59:15.449+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:59:16.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:59:15 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:59:16.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:59:15 smithi047 ceph-mon[77147]: pgmap v3097: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:59:16.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:59:15 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:59:16.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:59:15 smithi115 ceph-mon[78593]: pgmap v3097: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:59:16.752 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:59:16 smithi115 conmon[90393]: 2022-05-14T00:59:16.495+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:59:17.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:59:16 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:59:17.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:59:16 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:59:17.753 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:59:17 smithi115 conmon[90393]: 2022-05-14T00:59:17.453+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:59:18.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:59:17 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:59:18.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:59:17 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:59:18.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:59:17 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:59:18.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:59:17 smithi047 ceph-mon[77147]: pgmap v3098: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:59:18.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:59:17 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:59:18.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:59:17 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:59:18.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:59:17 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:59:18.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:59:17 smithi115 ceph-mon[78593]: pgmap v3098: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:59:18.759 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:59:18 smithi115 conmon[90393]: 2022-05-14T00:59:18.492+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:59:19.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:59:18 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:59:19.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:59:18 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:59:19.764 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:59:19 smithi115 conmon[90393]: 2022-05-14T00:59:19.493+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:59:20.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:59:19 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:59:20.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:59:19 smithi047 ceph-mon[77147]: pgmap v3099: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:59:20.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:59:19 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 5490 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:59:20.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:59:19 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:59:20.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:59:19 smithi115 ceph-mon[78593]: pgmap v3099: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:59:20.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:59:19 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 5490 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:59:20.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:59:20 smithi115 conmon[90393]: 2022-05-14T00:59:20.514+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:59:20.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:59:20 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:59:21.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:59:20 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:59:21.775 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:59:21 smithi115 conmon[90393]: 2022-05-14T00:59:21.468+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:59:22.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:59:21 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:59:22.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:59:21 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:59:22.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:59:21 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:59:22.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:59:21 smithi047 ceph-mon[77147]: pgmap v3100: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:59:22.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:59:21 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:59:22.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:59:21 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:59:22.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:59:21 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:59:22.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:59:21 smithi115 ceph-mon[78593]: pgmap v3100: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:59:22.777 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:59:22 smithi115 conmon[90393]: 2022-05-14T00:59:22.484+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:59:23.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:59:22 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:59:23.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:59:22 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:59:23.784 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:59:23 smithi115 conmon[90393]: 2022-05-14T00:59:23.466+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:59:24.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:59:23 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:59:24.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:59:23 smithi047 ceph-mon[77147]: pgmap v3101: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:59:24.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:59:23 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:59:24.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:59:23 smithi115 ceph-mon[78593]: pgmap v3101: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:59:24.786 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:59:24 smithi115 conmon[90393]: 2022-05-14T00:59:24.514+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:59:25.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:59:24 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:59:25.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:59:24 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 5495 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:59:25.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:59:24 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:59:25.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:59:24 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 5495 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:59:25.790 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:59:25 smithi115 conmon[90393]: 2022-05-14T00:59:25.506+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:59:26.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:59:25 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:59:26.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:59:25 smithi047 ceph-mon[77147]: pgmap v3102: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:59:26.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:59:25 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:59:26.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:59:25 smithi115 ceph-mon[78593]: pgmap v3102: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:59:26.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:59:26 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:59:26.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:59:26 smithi115 conmon[90393]: 2022-05-14T00:59:26.554+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:59:27.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:59:26 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:59:27.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:59:27 smithi115 conmon[90393]: 2022-05-14T00:59:27.563+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:59:27.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:59:27 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:59:27.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:59:27 smithi115 ceph-mon[78593]: pgmap v3103: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:59:28.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:59:27 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:59:28.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:59:27 smithi047 ceph-mon[77147]: pgmap v3103: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:59:28.796 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:59:28 smithi115 conmon[90393]: 2022-05-14T00:59:28.515+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:59:29.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:59:28 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:59:29.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:59:28 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:59:29.807 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:59:29 smithi115 conmon[90393]: 2022-05-14T00:59:29.471+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:59:30.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:59:29 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:59:30.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:59:29 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:59:30.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:59:29 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:59:30.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:59:29 smithi115 ceph-mon[78593]: pgmap v3104: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:59:30.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:59:29 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 5500 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:59:30.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:59:29 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:59:30.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:59:29 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:59:30.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:59:29 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:59:30.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:59:29 smithi047 ceph-mon[77147]: pgmap v3104: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:59:30.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:59:29 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 5500 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:59:30.809 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:59:30 smithi115 conmon[90393]: 2022-05-14T00:59:30.475+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:59:31.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:59:30 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:59:31.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:59:30 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:59:31.812 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:59:31 smithi115 conmon[90393]: 2022-05-14T00:59:31.456+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:59:32.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:59:31 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:59:32.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:59:31 smithi047 ceph-mon[77147]: pgmap v3105: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:59:32.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:59:31 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T00:59:32.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:59:31 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T00:59:32.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:59:31 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T00:59:32.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:59:31 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:59:32.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:59:31 smithi115 ceph-mon[78593]: pgmap v3105: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:59:32.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:59:31 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T00:59:32.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:59:31 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T00:59:32.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:59:31 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T00:59:32.818 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:59:32 smithi115 conmon[90393]: 2022-05-14T00:59:32.473+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:59:33.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:59:32 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:59:33.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:59:32 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:59:33.823 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:59:33 smithi115 conmon[90393]: 2022-05-14T00:59:33.433+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:59:34.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:59:33 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:59:34.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:59:33 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:59:34.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:59:33 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:59:34.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:59:33 smithi115 ceph-mon[78593]: pgmap v3106: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:59:34.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:59:33 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:59:34.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:59:33 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:59:34.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:59:33 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:59:34.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:59:33 smithi047 ceph-mon[77147]: pgmap v3106: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:59:34.825 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:59:34 smithi115 conmon[90393]: 2022-05-14T00:59:34.433+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:59:35.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:59:34 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:59:35.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:59:34 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 5505 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:59:35.042 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:59:34 smithi047 conmon[78521]: 2022-05-14T00:59:34.639+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:59:35.042 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:59:34 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T00:59:35.042 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:59:34 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:59:35.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:59:34 smithi047 conmon[78521]: yield 2022-05-14T00:59:35.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:59:34 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:59:35.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:59:34 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:59:35.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:59:34 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:59:35.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:59:34 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T00:59:35.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:59:34 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:59:35.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:59:34 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T00:59:35.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:59:34 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:59:35.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:59:34 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T00:59:35.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:59:34 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:59:35.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:59:34 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T00:59:35.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:59:34 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:59:35.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:59:34 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T00:59:35.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:59:34 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:59:35.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:59:34 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:59:35.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:59:34 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:59:35.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:59:34 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:59:35.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:59:34 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 5505 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:59:35.832 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:59:35 smithi115 conmon[90393]: 2022-05-14T00:59:35.459+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:59:36.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:59:35 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:59:36.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:59:35 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:59:36.108 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T00:59:36.108 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:59:36.108 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T00:59:36.109 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:59:36.109 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:59:36.109 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:59:36.109 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T00:59:36.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:59:36.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T00:59:36.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:59:36.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T00:59:36.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:59:36.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T00:59:36.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:59:36.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T00:59:36.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:59:36.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:59:36.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:59:36.112 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:59:35 smithi115 ceph-mon[78593]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T00:59:36.112 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:59:35 smithi115 ceph-mon[78593]: pgmap v3107: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:59:36.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:59:35 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:59:36.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:59:35 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:59:36.291 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T00:59:36.292 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:59:36.292 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T00:59:36.292 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:59:36.292 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:59:36.292 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:59:36.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T00:59:36.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:59:36.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T00:59:36.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:59:36.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T00:59:36.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:59:36.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T00:59:36.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:59:36.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T00:59:36.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:59:36.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:59:36.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:59:36.295 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:59:35 smithi047 ceph-mon[77147]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T00:59:36.295 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:59:35 smithi047 ceph-mon[77147]: pgmap v3107: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:59:36.836 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:59:36 smithi115 conmon[90393]: 2022-05-14T00:59:36.501+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:59:37.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:59:36 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:59:37.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:59:36 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:59:37.840 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:59:37 smithi115 conmon[90393]: 2022-05-14T00:59:37.532+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:59:38.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:59:37 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:59:38.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:59:37 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:59:38.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:59:37 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:59:38.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:59:37 smithi047 ceph-mon[77147]: pgmap v3108: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:59:38.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:59:37 smithi047 conmon[78521]: 2022-05-14T00:59:37.711+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:59:38.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:59:37 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T00:59:38.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:59:37 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:59:38.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:59:37 smithi047 conmon[78521]: yield 2022-05-14T00:59:38.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:59:37 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:59:38.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:59:37 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:59:38.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:59:37 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:59:38.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:59:37 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T00:59:38.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:59:37 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:59:38.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:59:37 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T00:59:38.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:59:37 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:59:38.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:59:37 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T00:59:38.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:59:37 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:59:38.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:59:37 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T00:59:38.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:59:37 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:59:38.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:59:37 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T00:59:38.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:59:37 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:59:38.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:59:37 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:59:38.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:59:37 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:59:38.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:59:37 smithi047 conmon[78521]: 2022-05-14T00:59:37.712+0000 7f5344781700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T00:59:38.048 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:59:37 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T00:59:38.048 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:59:37 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:59:38.048 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:59:37 smithi047 conmon[78521]: yield 2022-05-14T00:59:38.048 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:59:37 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:59:38.049 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:59:37 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:59:38.049 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:59:37 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:59:38.049 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:59:37 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T00:59:38.049 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:59:37 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:59:38.049 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:59:37 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T00:59:38.050 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:59:37 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:59:38.050 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:59:37 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T00:59:38.050 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:59:37 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:59:38.050 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:59:37 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T00:59:38.050 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:59:37 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:59:38.051 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:59:37 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T00:59:38.051 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:59:37 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:59:38.051 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:59:37 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:59:38.051 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:59:37 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:59:38.051 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:59:37 smithi047 conmon[78521]: 2022-05-14T00:59:38.052 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:59:37 smithi047 conmon[78521]: During handling of the above exception, another exception occurred: 2022-05-14T00:59:38.052 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:59:37 smithi047 conmon[78521]: 2022-05-14T00:59:38.052 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:59:37 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T00:59:38.052 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:59:37 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T00:59:38.052 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:59:37 smithi047 conmon[78521]: return f(*arg) 2022-05-14T00:59:38.053 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:59:37 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T00:59:38.053 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:59:37 smithi047 conmon[78521]: self._write_client_files(client_files, host) 2022-05-14T00:59:38.053 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:59:37 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T00:59:38.053 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:59:37 smithi047 conmon[78521]: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T00:59:38.054 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:59:37 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T00:59:38.054 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:59:37 smithi047 conmon[78521]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T00:59:38.054 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:59:37 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T00:59:38.054 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:59:37 smithi047 conmon[78521]: return self.event_loop.get_result(coro) 2022-05-14T00:59:38.054 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:59:37 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T00:59:38.055 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:59:37 smithi047 conmon[78521]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T00:59:38.055 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:59:37 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T00:59:38.055 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:59:37 smithi047 conmon[78521]: return self.__get_result() 2022-05-14T00:59:38.055 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:59:37 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T00:59:38.055 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:59:37 smithi047 conmon[78521]: raise self._exception 2022-05-14T00:59:38.056 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:59:37 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T00:59:38.056 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:59:37 smithi047 conmon[78521]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T00:59:38.056 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:59:37 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T00:59:38.056 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:59:37 smithi047 conmon[78521]: conn = await self._remote_connection(host, addr) 2022-05-14T00:59:38.056 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:59:37 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T00:59:38.057 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:59:37 smithi047 conmon[78521]: raise 2022-05-14T00:59:38.057 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:59:37 smithi047 conmon[78521]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T00:59:38.057 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:59:37 smithi047 conmon[78521]: self.gen.throw(type, value, traceback) 2022-05-14T00:59:38.057 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:59:37 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T00:59:38.058 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:59:37 smithi047 conmon[78521]: raise OrchestratorError(msg) 2022-05-14T00:59:38.058 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 00:59:37 smithi047 conmon[78521]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:59:38.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:59:37 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:59:38.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:59:37 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:59:38.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:59:37 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:59:38.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:59:37 smithi115 ceph-mon[78593]: pgmap v3108: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:59:38.842 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:59:38 smithi115 conmon[90393]: 2022-05-14T00:59:38.581+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:59:39.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:59:38 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:59:39.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:59:38 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:59:39.109 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T00:59:39.109 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:59:39.109 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T00:59:39.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:59:39.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:59:39.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:59:39.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T00:59:39.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:59:39.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T00:59:39.113 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:59:39.113 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T00:59:39.114 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:59:39.114 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T00:59:39.114 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:59:39.114 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T00:59:39.115 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:59:39.115 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:59:39.115 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:59:39.115 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:59:38 smithi115 ceph-mon[78593]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T00:59:39.115 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T00:59:39.116 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:59:39.116 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T00:59:39.116 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:59:39.116 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:59:39.117 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:59:39.117 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T00:59:39.117 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:59:39.117 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T00:59:39.117 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:59:39.118 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T00:59:39.118 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:59:39.118 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T00:59:39.118 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:59:39.118 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T00:59:39.119 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:59:39.119 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:59:39.119 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:59:39.119 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T00:59:39.120 INFO:journalctl@ceph.mon.c.smithi115.stdout: During handling of the above exception, another exception occurred: 2022-05-14T00:59:39.120 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T00:59:39.120 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T00:59:39.120 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T00:59:39.120 INFO:journalctl@ceph.mon.c.smithi115.stdout: return f(*arg) 2022-05-14T00:59:39.121 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T00:59:39.121 INFO:journalctl@ceph.mon.c.smithi115.stdout: self._write_client_files(client_files, host) 2022-05-14T00:59:39.121 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T00:59:39.121 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T00:59:39.121 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T00:59:39.122 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T00:59:39.122 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T00:59:39.122 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.event_loop.get_result(coro) 2022-05-14T00:59:39.122 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T00:59:39.123 INFO:journalctl@ceph.mon.c.smithi115.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T00:59:39.123 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T00:59:39.123 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.__get_result() 2022-05-14T00:59:39.123 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T00:59:39.123 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise self._exception 2022-05-14T00:59:39.124 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T00:59:39.124 INFO:journalctl@ceph.mon.c.smithi115.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T00:59:39.124 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T00:59:39.124 INFO:journalctl@ceph.mon.c.smithi115.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T00:59:39.124 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T00:59:39.125 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise 2022-05-14T00:59:39.125 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T00:59:39.125 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.gen.throw(type, value, traceback) 2022-05-14T00:59:39.125 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T00:59:39.126 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OrchestratorError(msg) 2022-05-14T00:59:39.126 INFO:journalctl@ceph.mon.c.smithi115.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:59:39.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:59:38 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:59:39.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:59:38 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:59:39.292 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T00:59:39.292 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:59:39.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T00:59:39.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:59:39.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:59:39.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:59:39.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T00:59:39.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:59:39.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T00:59:39.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:59:39.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T00:59:39.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:59:39.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T00:59:39.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:59:39.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T00:59:39.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:59:39.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:59:39.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:59:39.297 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:59:38 smithi047 ceph-mon[77147]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T00:59:39.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T00:59:39.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T00:59:39.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T00:59:39.298 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T00:59:39.298 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T00:59:39.298 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T00:59:39.298 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T00:59:39.298 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T00:59:39.299 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T00:59:39.299 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T00:59:39.299 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T00:59:39.299 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T00:59:39.299 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T00:59:39.300 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T00:59:39.300 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T00:59:39.300 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T00:59:39.300 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T00:59:39.300 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:59:39.301 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T00:59:39.301 INFO:journalctl@ceph.mon.b.smithi047.stdout: During handling of the above exception, another exception occurred: 2022-05-14T00:59:39.301 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T00:59:39.301 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T00:59:39.301 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T00:59:39.302 INFO:journalctl@ceph.mon.b.smithi047.stdout: return f(*arg) 2022-05-14T00:59:39.302 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T00:59:39.302 INFO:journalctl@ceph.mon.b.smithi047.stdout: self._write_client_files(client_files, host) 2022-05-14T00:59:39.302 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T00:59:39.302 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T00:59:39.303 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T00:59:39.303 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T00:59:39.303 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T00:59:39.303 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.event_loop.get_result(coro) 2022-05-14T00:59:39.303 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T00:59:39.304 INFO:journalctl@ceph.mon.b.smithi047.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T00:59:39.304 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T00:59:39.304 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.__get_result() 2022-05-14T00:59:39.304 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T00:59:39.304 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise self._exception 2022-05-14T00:59:39.305 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T00:59:39.305 INFO:journalctl@ceph.mon.b.smithi047.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T00:59:39.305 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T00:59:39.305 INFO:journalctl@ceph.mon.b.smithi047.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T00:59:39.306 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T00:59:39.306 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise 2022-05-14T00:59:39.306 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T00:59:39.306 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.gen.throw(type, value, traceback) 2022-05-14T00:59:39.306 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T00:59:39.307 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OrchestratorError(msg) 2022-05-14T00:59:39.307 INFO:journalctl@ceph.mon.b.smithi047.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T00:59:40.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:59:39 smithi115 conmon[90393]: 2022-05-14T00:59:39.608+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:59:40.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:59:39 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:59:40.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:59:39 smithi115 ceph-mon[78593]: pgmap v3109: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:59:40.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:59:39 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 5510 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:59:40.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:59:39 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:59:40.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:59:39 smithi047 ceph-mon[77147]: pgmap v3109: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:59:40.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:59:39 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 5510 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:59:40.849 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:59:40 smithi115 conmon[90393]: 2022-05-14T00:59:40.598+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:59:41.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:59:40 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:59:41.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:59:40 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:59:42.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:59:41 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:59:42.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:59:41 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:59:42.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:59:41 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:59:42.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:59:41 smithi115 ceph-mon[78593]: pgmap v3110: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:59:42.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:59:41 smithi115 conmon[90393]: 2022-05-14T00:59:41.637+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:59:42.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:59:41 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:59:42.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:59:41 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:59:42.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:59:41 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:59:42.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:59:41 smithi047 ceph-mon[77147]: pgmap v3110: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:59:43.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:59:42 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:59:43.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:59:42 smithi115 conmon[90393]: 2022-05-14T00:59:42.664+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:59:43.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:59:42 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:59:44.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:59:43 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:59:44.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:59:43 smithi115 ceph-mon[78593]: pgmap v3111: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:59:44.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:59:43 smithi115 conmon[90393]: 2022-05-14T00:59:43.637+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:59:44.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:59:43 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:59:44.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:59:43 smithi047 ceph-mon[77147]: pgmap v3111: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:59:45.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:59:44 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:59:45.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:59:44 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 5515 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:59:45.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:59:44 smithi115 conmon[90393]: 2022-05-14T00:59:44.667+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:59:45.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:59:44 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:59:45.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:59:44 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 5515 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:59:46.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:59:45 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:59:46.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:59:45 smithi115 ceph-mon[78593]: pgmap v3112: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:59:46.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:59:45 smithi115 conmon[90393]: 2022-05-14T00:59:45.696+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:59:46.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:59:45 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:59:46.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:59:45 smithi047 ceph-mon[77147]: pgmap v3112: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:59:47.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:59:46 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:59:47.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:59:46 smithi115 conmon[90393]: 2022-05-14T00:59:46.653+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:59:47.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:59:46 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:59:48.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:59:47 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:59:48.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:59:47 smithi115 ceph-mon[78593]: pgmap v3113: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:59:48.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:59:47 smithi115 conmon[90393]: 2022-05-14T00:59:47.666+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:59:48.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:59:47 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:59:48.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:59:47 smithi047 ceph-mon[77147]: pgmap v3113: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:59:49.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:59:48 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:59:49.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:59:48 smithi115 conmon[90393]: 2022-05-14T00:59:48.650+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:59:49.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:59:48 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:59:50.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:59:49 smithi115 conmon[90393]: 2022-05-14T00:59:49.692+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:59:50.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:59:49 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:59:50.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:59:49 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:59:50.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:59:49 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:59:50.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:59:49 smithi115 ceph-mon[78593]: pgmap v3114: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:59:50.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:59:49 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 5520 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:59:50.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:59:49 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:59:50.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:59:49 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:59:50.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:59:49 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:59:50.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:59:49 smithi047 ceph-mon[77147]: pgmap v3114: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:59:50.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:59:49 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 5520 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:59:51.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:59:50 smithi115 conmon[90393]: 2022-05-14T00:59:50.700+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:59:51.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:59:50 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:59:51.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:59:50 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:59:52.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:59:51 smithi115 conmon[90393]: 2022-05-14T00:59:51.678+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:59:52.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:59:51 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:59:52.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:59:51 smithi115 ceph-mon[78593]: pgmap v3115: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:59:52.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:59:51 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:59:52.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:59:51 smithi047 ceph-mon[77147]: pgmap v3115: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:59:53.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:59:52 smithi115 conmon[90393]: 2022-05-14T00:59:52.654+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:59:53.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:59:52 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:59:53.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:59:52 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:59:54.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:59:53 smithi115 conmon[90393]: 2022-05-14T00:59:53.688+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:59:54.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:59:53 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:59:54.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:59:53 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:59:54.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:59:53 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:59:54.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:59:53 smithi115 ceph-mon[78593]: pgmap v3116: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:59:54.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:59:53 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:59:54.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:59:53 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:59:54.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:59:53 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:59:54.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:59:53 smithi047 ceph-mon[77147]: pgmap v3116: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:59:55.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:59:54 smithi115 conmon[90393]: 2022-05-14T00:59:54.669+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:59:55.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:59:54 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:59:55.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:59:54 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 5525 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:59:55.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:59:54 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:59:55.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:59:54 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 5525 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T00:59:55.911 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:59:55 smithi115 conmon[90393]: 2022-05-14T00:59:55.653+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:59:56.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:59:55 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:59:56.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:59:55 smithi047 ceph-mon[77147]: pgmap v3117: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:59:56.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:59:55 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:59:56.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:59:55 smithi115 ceph-mon[78593]: pgmap v3117: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T00:59:57.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:59:56 smithi115 conmon[90393]: 2022-05-14T00:59:56.678+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:59:57.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:59:56 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:59:57.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:59:56 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:59:57.924 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:59:57 smithi115 conmon[90393]: 2022-05-14T00:59:57.644+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:59:58.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:59:57 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:59:58.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:59:57 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:59:58.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:59:57 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:59:58.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:59:57 smithi047 ceph-mon[77147]: pgmap v3118: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:59:58.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:59:57 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:59:58.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:59:57 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T00:59:58.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:59:57 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T00:59:58.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:59:57 smithi115 ceph-mon[78593]: pgmap v3118: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T00:59:58.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:59:58 smithi115 conmon[90393]: 2022-05-14T00:59:58.601+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T00:59:59.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:59:58 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:59:59.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:59:58 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T00:59:59.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 00:59:59 smithi115 conmon[90393]: 2022-05-14T00:59:59.582+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:00:00.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:59:59 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:00:00.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:59:59 smithi047 ceph-mon[77147]: pgmap v3119: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:00:00.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 00:59:59 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 5530 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:00:00.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:00:00 smithi047 conmon[77123]: 2022-05-14T00:59:59.999+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : Health detail: HEALTH_ERR 1 filesystem is degraded; 1 mds daemon damaged; 1 MDSs are read only; 1 MDSs report slow metadata IOs; 1/3 mons down, quorum b,c; Reduced data availability: 1 pg inactive; 9 slow ops, oldest one blocked for 5530 sec, osd.5 has slow ops 2022-05-14T01:00:00.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:00:00 smithi047 conmon[77123]: 2022-05-14T00:59:59.999+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : [WRN] FS_DEGRADED: 1 filesystem is degraded 2022-05-14T01:00:00.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:00:00 smithi047 conmon[77123]: 2022-05-14T00:59:59.999+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : fs cephfs is degraded 2022-05-14T01:00:00.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:00:00 smithi047 conmon[77123]: 2022-05-14T00:59:59.999+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : [ERR] MDS_DAMAGE: 1 mds daemon damaged 2022-05-14T01:00:00.295 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:00:00 smithi047 conmon[77123]: 2022-05-14T00:59:59.999+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : fs cephfs mds.2 is damaged 2022-05-14T01:00:00.295 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:00:00 smithi047 conmon[77123]: 2022-05-14T00:59:59.999+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : [WRN] MDS_READ_ONLY: 1 MDSs are read only 2022-05-14T01:00:00.296 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:00:00 smithi047 conmon[77123]: 2022-05-14T00:59:59.999+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : mds.l(mds.0): MDS in read-only mode 2022-05-14T01:00:00.296 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:00:00 smithi047 conmon[77123]: 2022-05-14T00:59:59.999+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : [WRN] MDS_SLOW_METADATA_IO: 1 MDSs report slow metadata IOs 2022-05-14T01:00:00.296 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:00:00 smithi047 conmon[77123]: 2022-05-14T00:59:59.999+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : mds.l(mds.0): 4 slow metadata IOs are blocked > 180 secs, oldest blocked for 5833 secs 2022-05-14T01:00:00.297 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:00:00 smithi047 conmon[77123]: 2022-05-14T00:59:59.999+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : [WRN] MON_DOWN: 1/3 mons down, quorum b,c 2022-05-14T01:00:00.297 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:00:00 smithi047 conmon[77123]: 2022-05-14T00:59:59.999+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : mon.a (rank 0) addr [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] is down (out of quorum) 2022-05-14T01:00:00.298 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:00:00 smithi047 conmon[77123]: 2022-05-14T00:59:59.999+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : [WRN] PG_AVAILABILITY: Reduced data availability: 1 pg inactive 2022-05-14T01:00:00.298 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:00:00 smithi047 conmon[77123]: 2022-05-14T00:59:59.999+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : pg 2.1 is stuck inactive for 104m, current state unknown, last acting [] 2022-05-14T01:00:00.299 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:00:00 smithi047 conmon[77123]: 2022-05-14T00:59:59.999+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : [WRN] SLOW_OPS: 9 slow ops, oldest one blocked for 5530 sec, osd.5 has slow ops 2022-05-14T01:00:00.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:59:59 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:00:00.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:59:59 smithi115 ceph-mon[78593]: pgmap v3119: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:00:00.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 00:59:59 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 5530 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:00:00.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:00:00 smithi115 conmon[90393]: 2022-05-14T01:00:00.552+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:00:01.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:00:00 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:00:01.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:00:00 smithi047 ceph-mon[77147]: Health detail: HEALTH_ERR 1 filesystem is degraded; 1 mds daemon damaged; 1 MDSs are read only; 1 MDSs report slow metadata IOs; 1/3 mons down, quorum b,c; Reduced data availability: 1 pg inactive; 9 slow ops, oldest one blocked for 5530 sec, osd.5 has slow ops 2022-05-14T01:00:01.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:00:00 smithi047 ceph-mon[77147]: [WRN] FS_DEGRADED: 1 filesystem is degraded 2022-05-14T01:00:01.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:00:00 smithi047 ceph-mon[77147]: fs cephfs is degraded 2022-05-14T01:00:01.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:00:00 smithi047 ceph-mon[77147]: [ERR] MDS_DAMAGE: 1 mds daemon damaged 2022-05-14T01:00:01.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:00:00 smithi047 ceph-mon[77147]: fs cephfs mds.2 is damaged 2022-05-14T01:00:01.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:00:00 smithi047 ceph-mon[77147]: [WRN] MDS_READ_ONLY: 1 MDSs are read only 2022-05-14T01:00:01.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:00:00 smithi047 ceph-mon[77147]: mds.l(mds.0): MDS in read-only mode 2022-05-14T01:00:01.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:00:00 smithi047 ceph-mon[77147]: [WRN] MDS_SLOW_METADATA_IO: 1 MDSs report slow metadata IOs 2022-05-14T01:00:01.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:00:00 smithi047 ceph-mon[77147]: mds.l(mds.0): 4 slow metadata IOs are blocked > 180 secs, oldest blocked for 5833 secs 2022-05-14T01:00:01.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:00:00 smithi047 ceph-mon[77147]: [WRN] MON_DOWN: 1/3 mons down, quorum b,c 2022-05-14T01:00:01.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:00:00 smithi047 ceph-mon[77147]: mon.a (rank 0) addr [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] is down (out of quorum) 2022-05-14T01:00:01.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:00:00 smithi047 ceph-mon[77147]: [WRN] PG_AVAILABILITY: Reduced data availability: 1 pg inactive 2022-05-14T01:00:01.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:00:00 smithi047 ceph-mon[77147]: pg 2.1 is stuck inactive for 104m, current state unknown, last acting [] 2022-05-14T01:00:01.295 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:00:00 smithi047 ceph-mon[77147]: [WRN] SLOW_OPS: 9 slow ops, oldest one blocked for 5530 sec, osd.5 has slow ops 2022-05-14T01:00:01.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:00:00 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:00:01.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:00:00 smithi115 ceph-mon[78593]: Health detail: HEALTH_ERR 1 filesystem is degraded; 1 mds daemon damaged; 1 MDSs are read only; 1 MDSs report slow metadata IOs; 1/3 mons down, quorum b,c; Reduced data availability: 1 pg inactive; 9 slow ops, oldest one blocked for 5530 sec, osd.5 has slow ops 2022-05-14T01:00:01.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:00:00 smithi115 ceph-mon[78593]: [WRN] FS_DEGRADED: 1 filesystem is degraded 2022-05-14T01:00:01.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:00:00 smithi115 ceph-mon[78593]: fs cephfs is degraded 2022-05-14T01:00:01.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:00:00 smithi115 ceph-mon[78593]: [ERR] MDS_DAMAGE: 1 mds daemon damaged 2022-05-14T01:00:01.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:00:00 smithi115 ceph-mon[78593]: fs cephfs mds.2 is damaged 2022-05-14T01:00:01.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:00:00 smithi115 ceph-mon[78593]: [WRN] MDS_READ_ONLY: 1 MDSs are read only 2022-05-14T01:00:01.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:00:00 smithi115 ceph-mon[78593]: mds.l(mds.0): MDS in read-only mode 2022-05-14T01:00:01.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:00:00 smithi115 ceph-mon[78593]: [WRN] MDS_SLOW_METADATA_IO: 1 MDSs report slow metadata IOs 2022-05-14T01:00:01.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:00:00 smithi115 ceph-mon[78593]: mds.l(mds.0): 4 slow metadata IOs are blocked > 180 secs, oldest blocked for 5833 secs 2022-05-14T01:00:01.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:00:00 smithi115 ceph-mon[78593]: [WRN] MON_DOWN: 1/3 mons down, quorum b,c 2022-05-14T01:00:01.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:00:00 smithi115 ceph-mon[78593]: mon.a (rank 0) addr [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] is down (out of quorum) 2022-05-14T01:00:01.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:00:00 smithi115 ceph-mon[78593]: [WRN] PG_AVAILABILITY: Reduced data availability: 1 pg inactive 2022-05-14T01:00:01.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:00:00 smithi115 ceph-mon[78593]: pg 2.1 is stuck inactive for 104m, current state unknown, last acting [] 2022-05-14T01:00:01.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:00:00 smithi115 ceph-mon[78593]: [WRN] SLOW_OPS: 9 slow ops, oldest one blocked for 5530 sec, osd.5 has slow ops 2022-05-14T01:00:01.861 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:00:01 smithi115 conmon[90393]: 2022-05-14T01:00:01.580+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:00:02.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:00:01 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:00:02.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:00:01 smithi047 ceph-mon[77147]: pgmap v3120: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:00:02.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:00:01 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:00:02.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:00:01 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:00:02.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:00:01 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:00:02.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:00:01 smithi115 ceph-mon[78593]: pgmap v3120: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:00:02.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:00:01 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:00:02.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:00:01 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:00:02.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:00:02 smithi115 conmon[90393]: 2022-05-14T01:00:02.539+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:00:03.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:00:02 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:00:03.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:00:02 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:00:03.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:00:03 smithi115 conmon[90393]: 2022-05-14T01:00:03.497+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:00:04.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:00:03 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:00:04.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:00:03 smithi047 ceph-mon[77147]: pgmap v3121: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:00:04.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:00:03 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:00:04.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:00:03 smithi115 ceph-mon[78593]: pgmap v3121: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:00:04.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:00:04 smithi115 conmon[90393]: 2022-05-14T01:00:04.480+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:00:05.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:00:04 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:00:05.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:00:04 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 5535 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:00:05.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:00:04 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:00:05.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:00:04 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 5535 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:00:05.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:00:05 smithi115 conmon[90393]: 2022-05-14T01:00:05.491+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:00:06.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:00:05 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:00:06.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:00:05 smithi047 ceph-mon[77147]: pgmap v3122: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:00:06.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:00:05 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:00:06.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:00:05 smithi115 ceph-mon[78593]: pgmap v3122: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:00:06.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:00:06 smithi115 conmon[90393]: 2022-05-14T01:00:06.503+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:00:07.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:00:06 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:00:07.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:00:06 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:00:07.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:00:07 smithi115 conmon[90393]: 2022-05-14T01:00:07.500+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:00:08.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:00:07 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:00:08.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:00:07 smithi047 ceph-mon[77147]: pgmap v3123: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:00:08.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:00:07 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:00:08.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:00:07 smithi115 ceph-mon[78593]: pgmap v3123: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:00:08.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:00:08 smithi115 conmon[90393]: 2022-05-14T01:00:08.484+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:00:09.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:00:08 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:00:09.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:00:08 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:00:09.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:00:09 smithi115 conmon[90393]: 2022-05-14T01:00:09.483+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:00:10.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:00:09 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:00:10.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:00:09 smithi047 ceph-mon[77147]: pgmap v3124: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:00:10.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:00:09 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:00:10.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:00:09 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:00:10.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:00:09 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 5540 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:00:10.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:00:09 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:00:10.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:00:09 smithi115 ceph-mon[78593]: pgmap v3124: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:00:10.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:00:09 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:00:10.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:00:09 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:00:10.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:00:09 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 5540 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:00:10.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:00:10 smithi115 conmon[90393]: 2022-05-14T01:00:10.481+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:00:11.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:00:10 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:00:11.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:00:10 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:00:11.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:00:11 smithi115 conmon[90393]: 2022-05-14T01:00:11.438+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:00:12.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:00:11 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:00:12.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:00:11 smithi047 ceph-mon[77147]: pgmap v3125: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:00:12.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:00:11 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:00:12.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:00:11 smithi115 ceph-mon[78593]: pgmap v3125: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:00:12.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:00:12 smithi115 conmon[90393]: 2022-05-14T01:00:12.433+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:00:13.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:00:12 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:00:13.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:00:12 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:00:13.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:00:13 smithi115 conmon[90393]: 2022-05-14T01:00:13.441+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:00:14.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:00:13 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:00:14.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:00:13 smithi047 ceph-mon[77147]: pgmap v3126: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:00:14.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:00:13 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:00:14.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:00:13 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:00:14.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:00:13 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T01:00:14.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:00:13 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T01:00:14.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:00:13 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:00:14.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:00:13 smithi115 ceph-mon[78593]: pgmap v3126: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:00:14.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:00:13 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:00:14.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:00:13 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:00:14.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:00:13 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T01:00:14.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:00:13 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T01:00:14.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:00:14 smithi115 conmon[90393]: 2022-05-14T01:00:14.394+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:00:15.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:00:14 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:00:15.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:00:14 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 5545 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:00:15.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:00:14 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:00:15.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:00:14 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 5545 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:00:15.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:00:15 smithi115 conmon[90393]: 2022-05-14T01:00:15.401+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:00:16.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:00:15 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:00:16.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:00:15 smithi047 ceph-mon[77147]: pgmap v3127: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:00:16.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:00:15 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:00:16.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:00:15 smithi115 ceph-mon[78593]: pgmap v3127: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:00:16.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:00:16 smithi115 conmon[90393]: 2022-05-14T01:00:16.391+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:00:17.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:00:16 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:00:17.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:00:16 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:00:17.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:00:17 smithi115 conmon[90393]: 2022-05-14T01:00:17.360+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:00:18.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:00:17 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:00:18.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:00:17 smithi047 ceph-mon[77147]: pgmap v3128: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:00:18.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:00:17 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:00:18.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:00:17 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:00:18.315 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:00:17 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:00:18.315 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:00:17 smithi115 ceph-mon[78593]: pgmap v3128: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:00:18.315 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:00:17 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:00:18.316 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:00:17 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:00:18.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:00:18 smithi115 conmon[90393]: 2022-05-14T01:00:18.314+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:00:19.270 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:00:18 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:00:19.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:00:18 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:00:19.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:00:19 smithi115 conmon[90393]: 2022-05-14T01:00:19.269+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:00:20.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:00:19 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:00:20.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:00:19 smithi047 ceph-mon[77147]: pgmap v3129: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:00:20.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:00:19 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 5550 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:00:20.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:00:19 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:00:20.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:00:19 smithi115 ceph-mon[78593]: pgmap v3129: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:00:20.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:00:19 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 5550 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:00:21.279 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:00:20 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:00:21.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:00:20 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:00:21.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:00:21 smithi115 conmon[90393]: 2022-05-14T01:00:21.279+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:00:22.244 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:00:21 smithi115 ceph-mon[78593]: pgmap v3130: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:00:22.246 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:00:21 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:00:22.246 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:00:21 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:00:22.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:00:21 smithi047 ceph-mon[77147]: pgmap v3130: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:00:22.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:00:21 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:00:22.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:00:21 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:00:22.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:00:22 smithi115 conmon[90393]: 2022-05-14T01:00:22.244+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:00:23.271 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:00:22 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:00:23.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:00:22 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:00:23.610 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:00:23 smithi115 conmon[90393]: 2022-05-14T01:00:23.271+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:00:24.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:00:23 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:00:24.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:00:23 smithi047 ceph-mon[77147]: pgmap v3131: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:00:24.315 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:00:23 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:00:24.316 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:00:24 smithi115 ceph-mon[78593]: pgmap v3131: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:00:24.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:00:24 smithi115 conmon[90393]: 2022-05-14T01:00:24.315+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:00:25.281 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:00:25 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:00:25.282 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:00:25 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 5555 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:00:25.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:00:25 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:00:25.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:00:25 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 5555 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:00:25.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:00:25 smithi115 conmon[90393]: 2022-05-14T01:00:25.280+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:00:26.267 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:00:26 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:00:26.267 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:00:26 smithi115 ceph-mon[78593]: pgmap v3132: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:00:26.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:00:26 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:00:26.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:00:26 smithi047 ceph-mon[77147]: pgmap v3132: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:00:26.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:00:26 smithi115 conmon[90393]: 2022-05-14T01:00:26.266+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:00:27.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:00:27 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:00:27.309 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:00:27 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:00:27.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:00:27 smithi115 conmon[90393]: 2022-05-14T01:00:27.308+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:00:28.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:00:28 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:00:28.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:00:28 smithi047 ceph-mon[77147]: pgmap v3133: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:00:28.345 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:00:28 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:00:28.346 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:00:28 smithi115 ceph-mon[78593]: pgmap v3133: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:00:28.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:00:28 smithi115 conmon[90393]: 2022-05-14T01:00:28.345+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:00:29.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:00:29 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:00:29.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:00:29 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:00:29.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:00:29 smithi115 conmon[90393]: 2022-05-14T01:00:29.384+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:00:30.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:00:30 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:00:30.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:00:30 smithi047 ceph-mon[77147]: pgmap v3134: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:00:30.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:00:30 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:00:30.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:00:30 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:00:30.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:00:30 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 5560 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:00:30.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:00:30 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:00:30.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:00:30 smithi115 ceph-mon[78593]: pgmap v3134: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:00:30.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:00:30 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:00:30.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:00:30 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:00:30.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:00:30 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 5560 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:00:30.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:00:30 smithi115 conmon[90393]: 2022-05-14T01:00:30.397+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:00:31.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:00:31 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:00:31.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:00:31 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:00:31.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:00:31 smithi115 conmon[90393]: 2022-05-14T01:00:31.434+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:00:32.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:00:32 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:00:32.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:00:32 smithi047 ceph-mon[77147]: pgmap v3135: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:00:32.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:00:32 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:00:32.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:00:32 smithi115 ceph-mon[78593]: pgmap v3135: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:00:32.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:00:32 smithi115 conmon[90393]: 2022-05-14T01:00:32.412+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:00:33.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:00:33 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:00:33.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:00:33 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:00:33.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:00:33 smithi115 conmon[90393]: 2022-05-14T01:00:33.411+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:00:34.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:00:34 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:00:34.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:00:34 smithi047 ceph-mon[77147]: pgmap v3136: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:00:34.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:00:34 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:00:34.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:00:34 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:00:34.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:00:34 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:00:34.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:00:34 smithi115 ceph-mon[78593]: pgmap v3136: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:00:34.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:00:34 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:00:34.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:00:34 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:00:34.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:00:34 smithi115 conmon[90393]: 2022-05-14T01:00:34.416+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:00:35.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:00:35 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:00:35.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:00:35 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 5565 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:00:35.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:00:35 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:00:35.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:00:35 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 5565 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:00:35.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:00:35 smithi115 conmon[90393]: 2022-05-14T01:00:35.455+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:00:36.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:00:36 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:00:36.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:00:36 smithi047 ceph-mon[77147]: pgmap v3137: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:00:36.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:00:36 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:00:36.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:00:36 smithi115 ceph-mon[78593]: pgmap v3137: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:00:36.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:00:36 smithi115 conmon[90393]: 2022-05-14T01:00:36.473+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:00:37.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:00:37 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:00:37.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:00:37 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:00:37.762 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:00:37 smithi115 conmon[90393]: 2022-05-14T01:00:37.461+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:00:38.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:00:38 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:00:38.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:00:38 smithi047 ceph-mon[77147]: pgmap v3138: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:00:38.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:00:38 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:00:38.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:00:38 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:00:38.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:00:38 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T01:00:38.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:00:38 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T01:00:38.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:00:38 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T01:00:38.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:00:38 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:00:38.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:00:38 smithi115 ceph-mon[78593]: pgmap v3138: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:00:38.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:00:38 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:00:38.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:00:38 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:00:38.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:00:38 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T01:00:38.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:00:38 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T01:00:38.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:00:38 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T01:00:38.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:00:38 smithi115 conmon[90393]: 2022-05-14T01:00:38.475+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:00:39.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:00:39 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:00:39.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:00:39 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:00:39.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:00:39 smithi115 conmon[90393]: 2022-05-14T01:00:39.492+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:00:40.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:00:40 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:00:40.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:00:40 smithi115 ceph-mon[78593]: pgmap v3139: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:00:40.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:00:40 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 5570 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:00:40.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:00:40 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:00:40.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:00:40 smithi047 ceph-mon[77147]: pgmap v3139: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:00:40.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:00:40 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 5570 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:00:40.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:00:40 smithi115 conmon[90393]: 2022-05-14T01:00:40.538+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:00:41.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:00:41 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:00:41.292 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:00:40 smithi047 conmon[78521]: 2022-05-14T01:00:40.815+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:00:41.292 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:00:40 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T01:00:41.292 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:00:40 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:00:41.292 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:00:40 smithi047 conmon[78521]: yield 2022-05-14T01:00:41.292 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:00:40 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:00:41.293 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:00:40 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:00:41.293 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:00:40 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:00:41.293 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:00:40 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T01:00:41.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:00:40 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:00:41.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:00:40 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T01:00:41.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:00:40 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:00:41.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:00:40 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T01:00:41.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:00:40 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:00:41.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:00:40 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T01:00:41.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:00:40 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:00:41.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:00:40 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T01:00:41.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:00:40 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:00:41.297 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:00:40 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:00:41.297 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:00:40 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:00:41.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:00:41 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:00:41.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:00:41 smithi115 conmon[90393]: 2022-05-14T01:00:41.557+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:00:42.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:00:42 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:00:42.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:00:42 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:00:42.358 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T01:00:42.358 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:00:42.359 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T01:00:42.359 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:00:42.359 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:00:42.359 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:00:42.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T01:00:42.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:00:42.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T01:00:42.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:00:42.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T01:00:42.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:00:42.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T01:00:42.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:00:42.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T01:00:42.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:00:42.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:00:42.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:00:42.363 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:00:42 smithi115 ceph-mon[78593]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T01:00:42.363 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:00:42 smithi115 ceph-mon[78593]: pgmap v3140: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:00:42.363 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:00:42 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:00:42.363 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:00:42 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:00:42.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:00:42 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:00:42.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:00:42 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:00:42.542 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T01:00:42.542 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:00:42.542 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T01:00:42.542 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:00:42.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:00:42.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:00:42.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T01:00:42.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:00:42.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T01:00:42.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:00:42.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T01:00:42.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:00:42.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T01:00:42.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:00:42.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T01:00:42.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:00:42.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:00:42.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:00:42.547 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:00:42 smithi047 ceph-mon[77147]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T01:00:42.547 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:00:42 smithi047 ceph-mon[77147]: pgmap v3140: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:00:42.548 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:00:42 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:00:42.548 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:00:42 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:00:42.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:00:42 smithi115 conmon[90393]: 2022-05-14T01:00:42.532+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:00:43.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:00:43 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:00:43.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:00:43 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:00:43.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:00:43 smithi115 conmon[90393]: 2022-05-14T01:00:43.549+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:00:44.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:00:44 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:00:44.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:00:44 smithi047 ceph-mon[77147]: pgmap v3141: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:00:44.292 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:00:43 smithi047 conmon[78521]: 2022-05-14T01:00:43.887+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:00:44.292 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:00:43 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T01:00:44.292 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:00:43 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:00:44.292 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:00:43 smithi047 conmon[78521]: yield 2022-05-14T01:00:44.293 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:00:43 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:00:44.293 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:00:43 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:00:44.293 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:00:43 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:00:44.293 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:00:43 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T01:00:44.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:00:43 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:00:44.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:00:43 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T01:00:44.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:00:43 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:00:44.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:00:43 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T01:00:44.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:00:43 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:00:44.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:00:43 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T01:00:44.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:00:43 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:00:44.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:00:43 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T01:00:44.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:00:43 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:00:44.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:00:43 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:00:44.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:00:43 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:00:44.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:00:43 smithi047 conmon[78521]: 2022-05-14T01:00:43.888+0000 7f5345f84700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T01:00:44.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:00:43 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T01:00:44.297 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:00:43 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:00:44.297 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:00:43 smithi047 conmon[78521]: yield 2022-05-14T01:00:44.297 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:00:43 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:00:44.298 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:00:43 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:00:44.298 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:00:43 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:00:44.298 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:00:43 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T01:00:44.298 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:00:43 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:00:44.299 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:00:43 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T01:00:44.299 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:00:43 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:00:44.299 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:00:43 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T01:00:44.299 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:00:43 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:00:44.299 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:00:43 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T01:00:44.300 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:00:43 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:00:44.300 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:00:43 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T01:00:44.300 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:00:43 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:00:44.300 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:00:43 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:00:44.301 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:00:43 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:00:44.301 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:00:43 smithi047 conmon[78521]: 2022-05-14T01:00:44.301 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:00:43 smithi047 conmon[78521]: During handling of the above exception, another exception occurred: 2022-05-14T01:00:44.301 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:00:43 smithi047 conmon[78521]: 2022-05-14T01:00:44.302 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:00:43 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T01:00:44.302 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:00:43 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T01:00:44.302 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:00:43 smithi047 conmon[78521]: return f(*arg) 2022-05-14T01:00:44.302 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:00:43 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T01:00:44.302 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:00:43 smithi047 conmon[78521]: self._write_client_files(client_files, host) 2022-05-14T01:00:44.303 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:00:43 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T01:00:44.303 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:00:43 smithi047 conmon[78521]: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T01:00:44.303 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:00:43 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T01:00:44.303 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:00:43 smithi047 conmon[78521]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T01:00:44.303 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:00:43 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T01:00:44.304 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:00:43 smithi047 conmon[78521]: return self.event_loop.get_result(coro) 2022-05-14T01:00:44.304 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:00:43 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T01:00:44.304 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:00:43 smithi047 conmon[78521]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T01:00:44.304 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:00:43 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T01:00:44.305 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:00:43 smithi047 conmon[78521]: return self.__get_result() 2022-05-14T01:00:44.305 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:00:43 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T01:00:44.305 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:00:43 smithi047 conmon[78521]: raise self._exception 2022-05-14T01:00:44.305 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:00:43 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T01:00:44.306 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:00:43 smithi047 conmon[78521]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T01:00:44.306 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:00:43 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T01:00:44.306 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:00:43 smithi047 conmon[78521]: conn = await self._remote_connection(host, addr) 2022-05-14T01:00:44.306 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:00:43 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T01:00:44.306 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:00:43 smithi047 conmon[78521]: raise 2022-05-14T01:00:44.307 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:00:43 smithi047 conmon[78521]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T01:00:44.307 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:00:43 smithi047 conmon[78521]: self.gen.throw(type, value, traceback) 2022-05-14T01:00:44.307 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:00:43 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T01:00:44.307 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:00:43 smithi047 conmon[78521]: raise OrchestratorError(msg) 2022-05-14T01:00:44.307 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:00:43 smithi047 conmon[78521]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:00:44.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:00:44 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:00:44.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:00:44 smithi115 ceph-mon[78593]: pgmap v3141: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:00:44.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:00:44 smithi115 conmon[90393]: 2022-05-14T01:00:44.512+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:00:45.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:00:45 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:00:45.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:00:45 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:00:45.358 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T01:00:45.358 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:00:45.359 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T01:00:45.359 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:00:45.359 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:00:45.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:00:45.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T01:00:45.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:00:45.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T01:00:45.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:00:45.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T01:00:45.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:00:45.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T01:00:45.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:00:45.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T01:00:45.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:00:45.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:00:45.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:00:45.363 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:00:45 smithi115 ceph-mon[78593]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T01:00:45.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T01:00:45.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:00:45.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T01:00:45.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:00:45.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:00:45.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:00:45.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T01:00:45.365 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:00:45.365 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T01:00:45.365 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:00:45.365 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T01:00:45.366 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:00:45.366 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T01:00:45.366 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:00:45.366 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T01:00:45.366 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:00:45.367 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:00:45.367 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:00:45.367 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T01:00:45.367 INFO:journalctl@ceph.mon.c.smithi115.stdout: During handling of the above exception, another exception occurred: 2022-05-14T01:00:45.367 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T01:00:45.368 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T01:00:45.368 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T01:00:45.368 INFO:journalctl@ceph.mon.c.smithi115.stdout: return f(*arg) 2022-05-14T01:00:45.368 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T01:00:45.369 INFO:journalctl@ceph.mon.c.smithi115.stdout: self._write_client_files(client_files, host) 2022-05-14T01:00:45.369 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T01:00:45.369 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T01:00:45.369 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T01:00:45.370 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T01:00:45.370 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T01:00:45.370 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.event_loop.get_result(coro) 2022-05-14T01:00:45.370 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T01:00:45.371 INFO:journalctl@ceph.mon.c.smithi115.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T01:00:45.371 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T01:00:45.371 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.__get_result() 2022-05-14T01:00:45.371 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T01:00:45.371 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise self._exception 2022-05-14T01:00:45.372 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T01:00:45.372 INFO:journalctl@ceph.mon.c.smithi115.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T01:00:45.372 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T01:00:45.372 INFO:journalctl@ceph.mon.c.smithi115.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T01:00:45.372 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T01:00:45.373 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise 2022-05-14T01:00:45.373 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T01:00:45.373 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.gen.throw(type, value, traceback) 2022-05-14T01:00:45.373 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T01:00:45.374 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OrchestratorError(msg) 2022-05-14T01:00:45.374 INFO:journalctl@ceph.mon.c.smithi115.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:00:45.374 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:00:45 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 5575 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:00:45.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:00:45 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:00:45.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:00:45 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:00:45.542 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T01:00:45.542 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:00:45.542 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T01:00:45.542 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:00:45.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:00:45.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:00:45.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T01:00:45.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:00:45.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T01:00:45.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:00:45.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T01:00:45.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:00:45.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T01:00:45.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:00:45.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T01:00:45.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:00:45.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:00:45.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:00:45.546 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:00:45 smithi047 ceph-mon[77147]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T01:00:45.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T01:00:45.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:00:45.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T01:00:45.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:00:45.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:00:45.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:00:45.548 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T01:00:45.548 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:00:45.548 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T01:00:45.548 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:00:45.548 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T01:00:45.549 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:00:45.549 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T01:00:45.549 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:00:45.549 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T01:00:45.550 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:00:45.550 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:00:45.550 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:00:45.550 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T01:00:45.551 INFO:journalctl@ceph.mon.b.smithi047.stdout: During handling of the above exception, another exception occurred: 2022-05-14T01:00:45.551 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T01:00:45.551 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T01:00:45.551 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T01:00:45.551 INFO:journalctl@ceph.mon.b.smithi047.stdout: return f(*arg) 2022-05-14T01:00:45.552 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T01:00:45.552 INFO:journalctl@ceph.mon.b.smithi047.stdout: self._write_client_files(client_files, host) 2022-05-14T01:00:45.552 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T01:00:45.552 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T01:00:45.553 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T01:00:45.553 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T01:00:45.553 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T01:00:45.553 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.event_loop.get_result(coro) 2022-05-14T01:00:45.554 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T01:00:45.554 INFO:journalctl@ceph.mon.b.smithi047.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T01:00:45.554 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T01:00:45.554 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.__get_result() 2022-05-14T01:00:45.554 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T01:00:45.555 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise self._exception 2022-05-14T01:00:45.555 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T01:00:45.555 INFO:journalctl@ceph.mon.b.smithi047.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T01:00:45.555 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T01:00:45.556 INFO:journalctl@ceph.mon.b.smithi047.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T01:00:45.556 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T01:00:45.556 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise 2022-05-14T01:00:45.556 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T01:00:45.556 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.gen.throw(type, value, traceback) 2022-05-14T01:00:45.557 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T01:00:45.557 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OrchestratorError(msg) 2022-05-14T01:00:45.557 INFO:journalctl@ceph.mon.b.smithi047.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:00:45.557 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:00:45 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 5575 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:00:45.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:00:45 smithi115 conmon[90393]: 2022-05-14T01:00:45.464+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:00:46.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:00:46 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:00:46.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:00:46 smithi115 ceph-mon[78593]: pgmap v3142: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:00:46.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:00:46 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:00:46.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:00:46 smithi047 ceph-mon[77147]: pgmap v3142: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:00:46.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:00:46 smithi115 conmon[90393]: 2022-05-14T01:00:46.427+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:00:47.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:00:47 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:00:47.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:00:47 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:00:47.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:00:47 smithi115 conmon[90393]: 2022-05-14T01:00:47.410+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:00:48.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:00:48 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:00:48.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:00:48 smithi115 ceph-mon[78593]: pgmap v3143: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:00:48.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:00:48 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:00:48.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:00:48 smithi047 ceph-mon[77147]: pgmap v3143: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:00:48.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:00:48 smithi115 conmon[90393]: 2022-05-14T01:00:48.429+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:00:49.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:00:49 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:00:49.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:00:49 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:00:49.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:00:49 smithi115 conmon[90393]: 2022-05-14T01:00:49.401+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:00:50.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:00:50 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:00:50.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:00:50 smithi115 ceph-mon[78593]: pgmap v3144: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:00:50.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:00:50 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:00:50.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:00:50 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:00:50.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:00:50 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 5580 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:00:50.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:00:50 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:00:50.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:00:50 smithi047 ceph-mon[77147]: pgmap v3144: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:00:50.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:00:50 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:00:50.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:00:50 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:00:50.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:00:50 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 5580 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:00:50.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:00:50 smithi115 conmon[90393]: 2022-05-14T01:00:50.417+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:00:51.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:00:51 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:00:51.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:00:51 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:00:51.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:00:51 smithi115 conmon[90393]: 2022-05-14T01:00:51.436+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:00:52.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:00:52 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:00:52.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:00:52 smithi115 ceph-mon[78593]: pgmap v3145: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:00:52.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:00:52 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:00:52.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:00:52 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:00:52.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:00:52 smithi047 ceph-mon[77147]: pgmap v3145: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:00:52.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:00:52 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:00:52.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:00:52 smithi115 conmon[90393]: 2022-05-14T01:00:52.427+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:00:53.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:00:53 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:00:53.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:00:53 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:00:53.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:00:53 smithi115 conmon[90393]: 2022-05-14T01:00:53.452+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:00:54.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:00:54 smithi115 ceph-mon[78593]: pgmap v3146: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:00:54.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:00:54 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:00:54.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:00:54 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:00:54.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:00:54 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:00:54.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:00:54 smithi047 ceph-mon[77147]: pgmap v3146: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:00:54.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:00:54 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:00:54.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:00:54 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:00:54.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:00:54 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:00:54.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:00:54 smithi115 conmon[90393]: 2022-05-14T01:00:54.438+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:00:55.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:00:55 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 5585 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:00:55.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:00:55 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:00:55.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:00:55 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 5585 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:00:55.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:00:55 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:00:55.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:00:55 smithi115 conmon[90393]: 2022-05-14T01:00:55.450+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:00:56.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:00:56 smithi115 ceph-mon[78593]: pgmap v3147: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:00:56.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:00:56 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:00:56.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:00:56 smithi047 ceph-mon[77147]: pgmap v3147: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:00:56.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:00:56 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:00:56.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:00:56 smithi115 conmon[90393]: 2022-05-14T01:00:56.486+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:00:57.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:00:57 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:00:57.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:00:57 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:00:57.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:00:57 smithi115 conmon[90393]: 2022-05-14T01:00:57.441+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:00:58.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:00:58 smithi115 ceph-mon[78593]: pgmap v3148: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:00:58.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:00:58 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:00:58.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:00:58 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:00:58.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:00:58 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:00:58.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:00:58 smithi047 ceph-mon[77147]: pgmap v3148: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:00:58.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:00:58 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:00:58.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:00:58 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:00:58.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:00:58 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:00:58.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:00:58 smithi115 conmon[90393]: 2022-05-14T01:00:58.424+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:00:59.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:00:59 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:00:59.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:00:59 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:00:59.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:00:59 smithi115 conmon[90393]: 2022-05-14T01:00:59.417+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:01:00.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:01:00 smithi115 ceph-mon[78593]: pgmap v3149: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:01:00.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:01:00 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 5590 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:01:00.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:01:00 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:01:00.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:01:00 smithi047 ceph-mon[77147]: pgmap v3149: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:01:00.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:01:00 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 5590 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:01:00.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:01:00 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:01:00.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:01:00 smithi115 conmon[90393]: 2022-05-14T01:01:00.440+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:01:01.459 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:01:01 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:01:01.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:01:01 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:01:01.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:01:01 smithi115 conmon[90393]: 2022-05-14T01:01:01.458+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:01:02.435 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:01:02 smithi115 ceph-mon[78593]: pgmap v3150: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:01:02.435 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:01:02 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:01:02.435 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:01:02 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:01:02.435 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:01:02 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:01:02.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:01:02 smithi047 ceph-mon[77147]: pgmap v3150: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:01:02.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:01:02 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:01:02.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:01:02 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:01:02.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:01:02 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:01:02.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:01:02 smithi115 conmon[90393]: 2022-05-14T01:01:02.433+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:01:03.430 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:01:03 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:01:03.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:01:03 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:01:03.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:01:03 smithi115 conmon[90393]: 2022-05-14T01:01:03.429+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:01:04.385 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:01:04 smithi115 ceph-mon[78593]: pgmap v3151: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:01:04.386 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:01:04 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:01:04.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:01:04 smithi047 ceph-mon[77147]: pgmap v3151: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:01:04.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:01:04 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:01:04.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:01:04 smithi115 conmon[90393]: 2022-05-14T01:01:04.384+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:01:05.392 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:01:05 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:01:05.393 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:01:05 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 5595 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:01:05.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:01:05 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:01:05.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:01:05 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 5595 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:01:05.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:01:05 smithi115 conmon[90393]: 2022-05-14T01:01:05.392+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:01:06.382 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:01:06 smithi115 ceph-mon[78593]: pgmap v3152: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:01:06.382 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:01:06 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:01:06.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:01:06 smithi047 ceph-mon[77147]: pgmap v3152: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:01:06.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:01:06 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:01:06.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:01:06 smithi115 conmon[90393]: 2022-05-14T01:01:06.381+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:01:07.417 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:01:07 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:01:07.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:01:07 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:01:07.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:01:07 smithi115 conmon[90393]: 2022-05-14T01:01:07.416+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:01:08.461 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:01:08 smithi115 ceph-mon[78593]: pgmap v3153: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:01:08.461 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:01:08 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:01:08.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:01:08 smithi047 ceph-mon[77147]: pgmap v3153: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:01:08.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:01:08 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:01:08.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:01:08 smithi115 conmon[90393]: 2022-05-14T01:01:08.460+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:01:09.426 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:01:09 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:01:09.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:01:09 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:01:09.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:01:09 smithi115 conmon[90393]: 2022-05-14T01:01:09.425+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:01:10.421 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:01:10 smithi115 ceph-mon[78593]: pgmap v3154: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:01:10.421 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:01:10 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:01:10.422 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:01:10 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:01:10.422 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:01:10 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 5600 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:01:10.422 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:01:10 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:01:10.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:01:10 smithi047 ceph-mon[77147]: pgmap v3154: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:01:10.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:01:10 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:01:10.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:01:10 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:01:10.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:01:10 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 5600 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:01:10.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:01:10 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:01:10.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:01:10 smithi115 conmon[90393]: 2022-05-14T01:01:10.410+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:01:11.402 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:01:11 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:01:11.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:01:11 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:01:11.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:01:11 smithi115 conmon[90393]: 2022-05-14T01:01:11.401+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:01:12.425 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:01:12 smithi115 ceph-mon[78593]: pgmap v3155: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:01:12.425 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:01:12 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:01:12.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:01:12 smithi047 ceph-mon[77147]: pgmap v3155: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:01:12.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:01:12 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:01:12.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:01:12 smithi115 conmon[90393]: 2022-05-14T01:01:12.424+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:01:13.456 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:01:13 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:01:13.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:01:13 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:01:13.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:01:13 smithi115 conmon[90393]: 2022-05-14T01:01:13.455+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:01:14.445 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:01:14 smithi115 ceph-mon[78593]: pgmap v3156: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:01:14.445 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:01:14 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:01:14.446 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:01:14 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:01:14.446 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:01:14 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:01:14.446 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:01:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T01:01:14.446 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:01:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T01:01:14.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:01:14 smithi047 ceph-mon[77147]: pgmap v3156: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:01:14.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:01:14 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:01:14.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:01:14 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:01:14.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:01:14 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:01:14.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:01:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T01:01:14.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:01:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T01:01:14.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:01:14 smithi115 conmon[90393]: 2022-05-14T01:01:14.444+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:01:15.470 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:01:15 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 5605 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:01:15.470 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:01:15 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:01:15.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:01:15 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 5605 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:01:15.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:01:15 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:01:15.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:01:15 smithi115 conmon[90393]: 2022-05-14T01:01:15.469+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:01:16.499 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:01:16 smithi115 ceph-mon[78593]: pgmap v3157: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:01:16.500 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:01:16 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:01:16.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:01:16 smithi047 ceph-mon[77147]: pgmap v3157: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:01:16.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:01:16 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:01:16.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:01:16 smithi115 conmon[90393]: 2022-05-14T01:01:16.498+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:01:17.464 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:01:17 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:01:17.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:01:17 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:01:17.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:01:17 smithi115 conmon[90393]: 2022-05-14T01:01:17.463+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:01:18.501 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:01:18 smithi115 ceph-mon[78593]: pgmap v3158: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:01:18.502 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:01:18 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:01:18.502 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:01:18 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:01:18.502 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:01:18 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:01:18.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:01:18 smithi047 ceph-mon[77147]: pgmap v3158: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:01:18.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:01:18 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:01:18.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:01:18 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:01:18.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:01:18 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:01:18.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:01:18 smithi115 conmon[90393]: 2022-05-14T01:01:18.493+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:01:19.482 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:01:19 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:01:19.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:01:19 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:01:19.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:01:19 smithi115 conmon[90393]: 2022-05-14T01:01:19.481+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:01:20.455 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:01:20 smithi115 ceph-mon[78593]: pgmap v3159: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:01:20.455 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:01:20 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 5610 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:01:20.456 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:01:20 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:01:20.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:01:20 smithi047 ceph-mon[77147]: pgmap v3159: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:01:20.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:01:20 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 5610 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:01:20.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:01:20 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:01:20.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:01:20 smithi115 conmon[90393]: 2022-05-14T01:01:20.454+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:01:21.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:01:21 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:01:21.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:01:21 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:01:21.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:01:21 smithi115 conmon[90393]: 2022-05-14T01:01:21.416+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:01:22.441 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:01:22 smithi115 ceph-mon[78593]: pgmap v3160: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:01:22.441 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:01:22 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:01:22.441 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:01:22 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:01:22.442 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:01:22 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:01:22.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:01:22 smithi047 ceph-mon[77147]: pgmap v3160: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:01:22.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:01:22 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:01:22.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:01:22 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:01:22.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:01:22 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:01:22.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:01:22 smithi115 conmon[90393]: 2022-05-14T01:01:22.439+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:01:23.438 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:01:23 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:01:23.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:01:23 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:01:23.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:01:23 smithi115 conmon[90393]: 2022-05-14T01:01:23.437+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:01:24.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:01:24 smithi047 ceph-mon[77147]: pgmap v3161: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:01:24.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:01:24 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:01:24.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:01:24 smithi115 conmon[90393]: 2022-05-14T01:01:24.393+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:01:24.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:01:24 smithi115 ceph-mon[78593]: pgmap v3161: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:01:24.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:01:24 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:01:25.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:01:25 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:01:25.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:01:25 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 5615 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:01:25.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:01:25 smithi115 conmon[90393]: 2022-05-14T01:01:25.361+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:01:25.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:01:25 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:01:25.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:01:25 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 5615 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:01:26.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:01:26 smithi047 ceph-mon[77147]: pgmap v3162: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:01:26.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:01:26 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:01:26.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:01:26 smithi115 conmon[90393]: 2022-05-14T01:01:26.324+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:01:26.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:01:26 smithi115 ceph-mon[78593]: pgmap v3162: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:01:26.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:01:26 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:01:27.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:01:27 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:01:27.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:01:27 smithi115 conmon[90393]: 2022-05-14T01:01:27.287+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:01:27.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:01:27 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:01:28.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:01:28 smithi047 ceph-mon[77147]: pgmap v3163: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:01:28.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:01:28 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:01:28.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:01:28 smithi115 conmon[90393]: 2022-05-14T01:01:28.301+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:01:28.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:01:28 smithi115 ceph-mon[78593]: pgmap v3163: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:01:28.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:01:28 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:01:29.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:01:29 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:01:29.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:01:29 smithi115 conmon[90393]: 2022-05-14T01:01:29.253+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:01:29.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:01:29 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:01:30.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:01:30 smithi047 ceph-mon[77147]: pgmap v3164: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:01:30.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:01:30 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:01:30.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:01:30 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:01:30.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:01:30 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:01:30.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:01:30 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 5620 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:01:30.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:01:30 smithi115 conmon[90393]: 2022-05-14T01:01:30.230+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:01:30.675 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:01:30 smithi115 ceph-mon[78593]: pgmap v3164: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:01:30.675 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:01:30 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:01:30.675 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:01:30 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:01:30.675 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:01:30 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:01:30.676 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:01:30 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 5620 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:01:31.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:01:31 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:01:31.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:01:31 smithi115 conmon[90393]: 2022-05-14T01:01:31.228+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:01:31.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:01:31 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:01:32.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:01:32 smithi047 ceph-mon[77147]: pgmap v3165: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:01:32.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:01:32 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:01:32.613 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:01:32 smithi115 conmon[90393]: 2022-05-14T01:01:32.215+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:01:32.614 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:01:32 smithi115 ceph-mon[78593]: pgmap v3165: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:01:32.614 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:01:32 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:01:33.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:01:33 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:01:33.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:01:33 smithi115 conmon[90393]: 2022-05-14T01:01:33.262+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:01:33.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:01:33 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:01:34.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:01:34 smithi047 ceph-mon[77147]: pgmap v3166: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:01:34.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:01:34 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:01:34.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:01:34 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:01:34.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:01:34 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:01:34.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:01:34 smithi115 conmon[90393]: 2022-05-14T01:01:34.238+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:01:34.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:01:34 smithi115 ceph-mon[78593]: pgmap v3166: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:01:34.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:01:34 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:01:34.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:01:34 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:01:34.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:01:34 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:01:35.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:01:35 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:01:35.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:01:35 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 5625 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:01:35.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:01:35 smithi115 conmon[90393]: 2022-05-14T01:01:35.286+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:01:35.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:01:35 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:01:35.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:01:35 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 5625 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:01:36.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:01:36 smithi047 ceph-mon[77147]: pgmap v3167: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:01:36.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:01:36 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:01:36.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:01:36 smithi115 conmon[90393]: 2022-05-14T01:01:36.272+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:01:36.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:01:36 smithi115 ceph-mon[78593]: pgmap v3167: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:01:36.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:01:36 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:01:37.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:01:37 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:01:37.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:01:37 smithi115 conmon[90393]: 2022-05-14T01:01:37.305+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:01:37.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:01:37 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:01:38.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:01:38 smithi047 ceph-mon[77147]: pgmap v3168: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:01:38.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:01:38 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:01:38.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:01:38 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:01:38.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:01:38 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:01:38.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:01:38 smithi115 conmon[90393]: 2022-05-14T01:01:38.343+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:01:38.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:01:38 smithi115 ceph-mon[78593]: pgmap v3168: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:01:38.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:01:38 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:01:38.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:01:38 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:01:38.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:01:38 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:01:39.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:01:39 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:01:39.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:01:39 smithi115 conmon[90393]: 2022-05-14T01:01:39.315+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:01:39.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:01:39 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:01:40.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:01:40 smithi047 ceph-mon[77147]: pgmap v3169: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:01:40.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:01:40 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:01:40.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:01:40 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 5630 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:01:40.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:01:40 smithi115 conmon[90393]: 2022-05-14T01:01:40.297+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:01:40.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:01:40 smithi115 ceph-mon[78593]: pgmap v3169: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:01:40.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:01:40 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:01:40.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:01:40 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 5630 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:01:41.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:01:41 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:01:41.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:01:41 smithi115 conmon[90393]: 2022-05-14T01:01:41.275+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:01:41.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:01:41 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:01:42.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:01:42 smithi047 ceph-mon[77147]: pgmap v3170: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:01:42.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:01:42 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:01:42.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:01:42 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:01:42.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:01:42 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:01:42.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:01:42 smithi115 conmon[90393]: 2022-05-14T01:01:42.256+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:01:42.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:01:42 smithi115 ceph-mon[78593]: pgmap v3170: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:01:42.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:01:42 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:01:42.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:01:42 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:01:42.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:01:42 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:01:43.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:01:43 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:01:43.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:01:43 smithi115 conmon[90393]: 2022-05-14T01:01:43.251+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:01:43.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:01:43 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:01:44.497 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:01:44 smithi047 ceph-mon[77147]: pgmap v3171: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:01:44.498 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:01:44 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:01:44.498 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:01:44 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T01:01:44.499 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:01:44 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T01:01:44.499 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:01:44 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T01:01:44.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:01:44 smithi115 conmon[90393]: 2022-05-14T01:01:44.244+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:01:44.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:01:44 smithi115 ceph-mon[78593]: pgmap v3171: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:01:44.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:01:44 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:01:44.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:01:44 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T01:01:44.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:01:44 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T01:01:44.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:01:44 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T01:01:44.791 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:01:44 smithi047 conmon[78521]: 2022-05-14T01:01:44.496+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:01:44.791 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:01:44 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T01:01:44.792 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:01:44 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:01:44.792 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:01:44 smithi047 conmon[78521]: yield 2022-05-14T01:01:44.792 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:01:44 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:01:44.792 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:01:44 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:01:44.793 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:01:44 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:01:44.793 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:01:44 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T01:01:44.793 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:01:44 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:01:44.793 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:01:44 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T01:01:44.793 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:01:44 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:01:44.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:01:44 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T01:01:44.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:01:44 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:01:44.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:01:44 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T01:01:44.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:01:44 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:01:44.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:01:44 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T01:01:44.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:01:44 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:01:44.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:01:44 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:01:44.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:01:44 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:01:45.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:01:45 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:01:45.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:01:45 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 5635 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:01:45.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:01:45 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:01:45.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:01:45 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 5635 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:01:45.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:01:45 smithi115 conmon[90393]: 2022-05-14T01:01:45.204+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:01:46.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:01:46 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:01:46.541 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T01:01:46.541 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:01:46.542 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T01:01:46.542 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:01:46.542 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:01:46.542 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:01:46.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T01:01:46.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:01:46.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T01:01:46.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:01:46.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T01:01:46.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:01:46.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T01:01:46.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:01:46.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T01:01:46.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:01:46.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:01:46.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:01:46.545 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:01:46 smithi047 ceph-mon[77147]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T01:01:46.545 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:01:46 smithi047 ceph-mon[77147]: pgmap v3172: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:01:46.546 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:01:46 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:01:46.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:01:46 smithi115 conmon[90393]: 2022-05-14T01:01:46.165+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:01:46.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:01:46 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:01:46.608 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T01:01:46.608 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:01:46.609 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T01:01:46.609 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:01:46.609 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:01:46.609 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:01:46.609 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T01:01:46.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:01:46.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T01:01:46.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:01:46.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T01:01:46.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:01:46.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T01:01:46.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:01:46.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T01:01:46.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:01:46.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:01:46.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:01:46.612 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:01:46 smithi115 ceph-mon[78593]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T01:01:46.612 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:01:46 smithi115 ceph-mon[78593]: pgmap v3172: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:01:46.612 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:01:46 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:01:47.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:01:47 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:01:47.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:01:47 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:01:47.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:01:47 smithi115 conmon[90393]: 2022-05-14T01:01:47.179+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:01:48.041 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:01:47 smithi047 conmon[78521]: 2022-05-14T01:01:47.568+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:01:48.042 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:01:47 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T01:01:48.042 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:01:47 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:01:48.042 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:01:47 smithi047 conmon[78521]: yield 2022-05-14T01:01:48.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:01:47 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:01:48.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:01:47 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:01:48.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:01:47 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:01:48.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:01:47 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T01:01:48.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:01:47 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:01:48.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:01:47 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T01:01:48.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:01:47 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:01:48.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:01:47 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T01:01:48.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:01:47 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:01:48.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:01:47 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T01:01:48.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:01:47 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:01:48.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:01:47 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T01:01:48.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:01:47 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:01:48.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:01:47 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:01:48.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:01:47 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:01:48.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:01:47 smithi047 conmon[78521]: 2022-05-14T01:01:47.569+0000 7f533ff78700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T01:01:48.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:01:47 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T01:01:48.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:01:47 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:01:48.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:01:47 smithi047 conmon[78521]: yield 2022-05-14T01:01:48.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:01:47 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:01:48.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:01:47 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:01:48.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:01:47 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:01:48.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:01:47 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T01:01:48.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:01:47 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:01:48.048 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:01:47 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T01:01:48.048 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:01:47 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:01:48.048 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:01:47 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T01:01:48.048 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:01:47 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:01:48.048 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:01:47 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T01:01:48.049 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:01:47 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:01:48.049 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:01:47 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T01:01:48.049 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:01:47 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:01:48.049 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:01:47 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:01:48.050 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:01:47 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:01:48.050 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:01:47 smithi047 conmon[78521]: 2022-05-14T01:01:48.050 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:01:47 smithi047 conmon[78521]: During handling of the above exception, another exception occurred: 2022-05-14T01:01:48.050 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:01:47 smithi047 conmon[78521]: 2022-05-14T01:01:48.050 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:01:47 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T01:01:48.051 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:01:47 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T01:01:48.051 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:01:47 smithi047 conmon[78521]: return f(*arg) 2022-05-14T01:01:48.051 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:01:47 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T01:01:48.051 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:01:47 smithi047 conmon[78521]: self._write_client_files(client_files, host) 2022-05-14T01:01:48.051 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:01:47 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T01:01:48.052 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:01:47 smithi047 conmon[78521]: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T01:01:48.052 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:01:47 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T01:01:48.052 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:01:47 smithi047 conmon[78521]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T01:01:48.052 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:01:47 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T01:01:48.052 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:01:47 smithi047 conmon[78521]: return self.event_loop.get_result(coro) 2022-05-14T01:01:48.053 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:01:47 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T01:01:48.053 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:01:47 smithi047 conmon[78521]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T01:01:48.053 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:01:47 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T01:01:48.053 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:01:47 smithi047 conmon[78521]: return self.__get_result() 2022-05-14T01:01:48.053 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:01:47 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T01:01:48.054 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:01:47 smithi047 conmon[78521]: raise self._exception 2022-05-14T01:01:48.054 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:01:47 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T01:01:48.054 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:01:47 smithi047 conmon[78521]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T01:01:48.054 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:01:47 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T01:01:48.055 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:01:47 smithi047 conmon[78521]: conn = await self._remote_connection(host, addr) 2022-05-14T01:01:48.055 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:01:47 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T01:01:48.055 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:01:47 smithi047 conmon[78521]: raise 2022-05-14T01:01:48.055 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:01:47 smithi047 conmon[78521]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T01:01:48.055 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:01:47 smithi047 conmon[78521]: self.gen.throw(type, value, traceback) 2022-05-14T01:01:48.056 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:01:47 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T01:01:48.056 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:01:47 smithi047 conmon[78521]: raise OrchestratorError(msg) 2022-05-14T01:01:48.056 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:01:47 smithi047 conmon[78521]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:01:48.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:01:48 smithi047 ceph-mon[77147]: pgmap v3173: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:01:48.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:01:48 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:01:48.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:01:48 smithi115 ceph-mon[78593]: pgmap v3173: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:01:48.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:01:48 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:01:48.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:01:48 smithi115 conmon[90393]: 2022-05-14T01:01:48.177+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:01:49.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:01:49 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:01:49.542 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T01:01:49.542 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:01:49.542 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T01:01:49.542 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:01:49.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:01:49.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:01:49.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T01:01:49.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:01:49.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T01:01:49.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:01:49.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T01:01:49.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:01:49.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T01:01:49.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:01:49.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T01:01:49.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:01:49.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:01:49.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:01:49.546 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:01:49 smithi047 ceph-mon[77147]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T01:01:49.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T01:01:49.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:01:49.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T01:01:49.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:01:49.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:01:49.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:01:49.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T01:01:49.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:01:49.548 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T01:01:49.548 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:01:49.548 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T01:01:49.548 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:01:49.548 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T01:01:49.549 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:01:49.549 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T01:01:49.549 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:01:49.550 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:01:49.550 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:01:49.550 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T01:01:49.551 INFO:journalctl@ceph.mon.b.smithi047.stdout: During handling of the above exception, another exception occurred: 2022-05-14T01:01:49.551 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T01:01:49.551 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T01:01:49.551 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T01:01:49.551 INFO:journalctl@ceph.mon.b.smithi047.stdout: return f(*arg) 2022-05-14T01:01:49.552 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T01:01:49.552 INFO:journalctl@ceph.mon.b.smithi047.stdout: self._write_client_files(client_files, host) 2022-05-14T01:01:49.552 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T01:01:49.552 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T01:01:49.553 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T01:01:49.553 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T01:01:49.553 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T01:01:49.553 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.event_loop.get_result(coro) 2022-05-14T01:01:49.554 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T01:01:49.554 INFO:journalctl@ceph.mon.b.smithi047.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T01:01:49.554 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T01:01:49.554 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.__get_result() 2022-05-14T01:01:49.555 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T01:01:49.555 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise self._exception 2022-05-14T01:01:49.556 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T01:01:49.556 INFO:journalctl@ceph.mon.b.smithi047.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T01:01:49.556 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T01:01:49.556 INFO:journalctl@ceph.mon.b.smithi047.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T01:01:49.557 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T01:01:49.557 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise 2022-05-14T01:01:49.557 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T01:01:49.557 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.gen.throw(type, value, traceback) 2022-05-14T01:01:49.558 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T01:01:49.558 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OrchestratorError(msg) 2022-05-14T01:01:49.558 INFO:journalctl@ceph.mon.b.smithi047.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:01:49.558 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:01:49 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:01:49.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:01:49 smithi115 conmon[90393]: 2022-05-14T01:01:49.160+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:01:49.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:01:49 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:01:49.608 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T01:01:49.609 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:01:49.609 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T01:01:49.609 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:01:49.609 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:01:49.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:01:49.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T01:01:49.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:01:49.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T01:01:49.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:01:49.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T01:01:49.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:01:49.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T01:01:49.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:01:49.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T01:01:49.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:01:49.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:01:49.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:01:49.612 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:01:49 smithi115 ceph-mon[78593]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T01:01:49.613 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T01:01:49.613 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:01:49.613 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T01:01:49.614 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:01:49.614 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:01:49.614 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:01:49.615 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T01:01:49.615 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:01:49.615 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T01:01:49.615 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:01:49.615 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T01:01:49.616 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:01:49.616 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T01:01:49.616 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:01:49.616 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T01:01:49.616 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:01:49.617 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:01:49.617 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:01:49.617 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T01:01:49.617 INFO:journalctl@ceph.mon.c.smithi115.stdout: During handling of the above exception, another exception occurred: 2022-05-14T01:01:49.618 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T01:01:49.618 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T01:01:49.618 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T01:01:49.618 INFO:journalctl@ceph.mon.c.smithi115.stdout: return f(*arg) 2022-05-14T01:01:49.619 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T01:01:49.619 INFO:journalctl@ceph.mon.c.smithi115.stdout: self._write_client_files(client_files, host) 2022-05-14T01:01:49.619 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T01:01:49.619 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T01:01:49.619 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T01:01:49.620 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T01:01:49.620 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T01:01:49.620 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.event_loop.get_result(coro) 2022-05-14T01:01:49.620 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T01:01:49.620 INFO:journalctl@ceph.mon.c.smithi115.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T01:01:49.621 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T01:01:49.621 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.__get_result() 2022-05-14T01:01:49.621 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T01:01:49.622 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise self._exception 2022-05-14T01:01:49.622 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T01:01:49.622 INFO:journalctl@ceph.mon.c.smithi115.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T01:01:49.622 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T01:01:49.622 INFO:journalctl@ceph.mon.c.smithi115.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T01:01:49.623 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T01:01:49.623 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise 2022-05-14T01:01:49.623 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T01:01:49.623 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.gen.throw(type, value, traceback) 2022-05-14T01:01:49.623 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T01:01:49.624 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OrchestratorError(msg) 2022-05-14T01:01:49.624 INFO:journalctl@ceph.mon.c.smithi115.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:01:49.624 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:01:49 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:01:50.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:01:50 smithi047 ceph-mon[77147]: pgmap v3174: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:01:50.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:01:50 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:01:50.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:01:50 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:01:50.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:01:50 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:01:50.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:01:50 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 5640 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:01:50.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:01:50 smithi115 conmon[90393]: 2022-05-14T01:01:50.166+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:01:50.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:01:50 smithi115 ceph-mon[78593]: pgmap v3174: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:01:50.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:01:50 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:01:50.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:01:50 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:01:50.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:01:50 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:01:50.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:01:50 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 5640 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:01:51.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:01:51 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:01:51.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:01:51 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:01:51.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:01:51 smithi115 conmon[90393]: 2022-05-14T01:01:51.182+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:01:52.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:01:52 smithi047 ceph-mon[77147]: pgmap v3175: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:01:52.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:01:52 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:01:52.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:01:52 smithi115 conmon[90393]: 2022-05-14T01:01:52.172+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:01:52.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:01:52 smithi115 ceph-mon[78593]: pgmap v3175: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:01:52.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:01:52 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:01:53.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:01:53 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:01:53.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:01:53 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:01:53.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:01:53 smithi115 conmon[90393]: 2022-05-14T01:01:53.146+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:01:54.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:01:54 smithi047 ceph-mon[77147]: pgmap v3176: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:01:54.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:01:54 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:01:54.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:01:54 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:01:54.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:01:54 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:01:54.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:01:54 smithi115 conmon[90393]: 2022-05-14T01:01:54.121+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:01:54.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:01:54 smithi115 ceph-mon[78593]: pgmap v3176: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:01:54.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:01:54 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:01:54.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:01:54 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:01:54.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:01:54 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:01:55.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:01:55 smithi115 conmon[90393]: 2022-05-14T01:01:55.094+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:01:55.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:01:55 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:01:55.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:01:55 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 5645 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:01:55.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:01:55 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:01:55.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:01:55 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 5645 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:01:56.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:01:56 smithi115 conmon[90393]: 2022-05-14T01:01:56.072+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:01:56.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:01:56 smithi115 ceph-mon[78593]: pgmap v3177: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:01:56.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:01:56 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:01:56.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:01:56 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:01:56.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:01:56 smithi047 ceph-mon[77147]: pgmap v3177: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:01:56.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:01:56 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:01:56.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:01:56 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:01:57.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:01:57 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:01:57.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:01:57 smithi115 conmon[90393]: 2022-05-14T01:01:57.082+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:01:57.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:01:57 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:01:58.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:01:58 smithi115 conmon[90393]: 2022-05-14T01:01:58.066+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:01:58.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:01:58 smithi115 ceph-mon[78593]: pgmap v3178: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:01:58.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:01:58 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:01:58.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:01:58 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:01:58.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:01:58 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:01:58.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:01:58 smithi047 ceph-mon[77147]: pgmap v3178: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:01:58.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:01:58 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:01:58.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:01:58 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:01:58.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:01:58 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:01:59.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:01:59 smithi115 conmon[90393]: 2022-05-14T01:01:59.113+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:02:00.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:02:00 smithi047 ceph-mon[77147]: pgmap v3179: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:02:00.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:02:00 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:02:00.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:02:00 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 5650 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:02:00.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:02:00 smithi115 conmon[90393]: 2022-05-14T01:02:00.106+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:02:00.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:02:00 smithi115 ceph-mon[78593]: pgmap v3179: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:02:00.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:02:00 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:02:00.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:02:00 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 5650 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:02:01.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:02:01 smithi115 conmon[90393]: 2022-05-14T01:02:01.067+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:02:01.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:02:01 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:02:01.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:02:01 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:02:01.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:02:01 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:02:01.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:02:01 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:02:02.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:02:02 smithi115 conmon[90393]: 2022-05-14T01:02:02.099+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:02:02.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:02:02 smithi115 ceph-mon[78593]: pgmap v3180: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:02:02.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:02:02 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:02:02.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:02:02 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:02:02.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:02:02 smithi047 ceph-mon[77147]: pgmap v3180: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:02:02.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:02:02 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:02:02.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:02:02 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:02:03.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:02:03 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:02:03.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:02:03 smithi115 conmon[90393]: 2022-05-14T01:02:03.141+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:02:03.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:02:03 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:02:04.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:02:04 smithi115 conmon[90393]: 2022-05-14T01:02:04.143+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:02:04.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:02:04 smithi115 ceph-mon[78593]: pgmap v3181: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:02:04.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:02:04 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:02:04.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:02:04 smithi047 ceph-mon[77147]: pgmap v3181: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:02:04.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:02:04 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:02:05.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:02:05 smithi115 conmon[90393]: 2022-05-14T01:02:05.103+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:02:05.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:02:05 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:02:05.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:02:05 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 5654 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:02:05.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:02:05 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:02:05.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:02:05 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 5654 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:02:06.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:02:06 smithi115 conmon[90393]: 2022-05-14T01:02:06.113+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:02:06.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:02:06 smithi115 ceph-mon[78593]: pgmap v3182: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:02:06.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:02:06 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:02:06.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:02:06 smithi047 ceph-mon[77147]: pgmap v3182: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:02:06.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:02:06 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:02:07.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:02:07 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:02:07.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:02:07 smithi115 conmon[90393]: 2022-05-14T01:02:07.120+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:02:07.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:02:07 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:02:08.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:02:08 smithi115 conmon[90393]: 2022-05-14T01:02:08.096+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:02:08.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:02:08 smithi115 ceph-mon[78593]: pgmap v3183: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:02:08.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:02:08 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:02:08.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:02:08 smithi047 ceph-mon[77147]: pgmap v3183: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:02:08.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:02:08 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:02:09.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:02:09 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:02:09.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:02:09 smithi115 conmon[90393]: 2022-05-14T01:02:09.107+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:02:09.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:02:09 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:02:10.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:02:10 smithi115 conmon[90393]: 2022-05-14T01:02:10.124+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:02:10.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:02:10 smithi115 ceph-mon[78593]: pgmap v3184: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:02:10.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:02:10 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:02:10.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:02:10 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:02:10.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:02:10 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:02:10.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:02:10 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 5660 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:02:10.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:02:10 smithi047 ceph-mon[77147]: pgmap v3184: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:02:10.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:02:10 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:02:10.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:02:10 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:02:10.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:02:10 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:02:10.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:02:10 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 5660 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:02:11.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:02:11 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:02:11.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:02:11 smithi115 conmon[90393]: 2022-05-14T01:02:11.140+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:02:11.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:02:11 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:02:12.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:02:12 smithi115 conmon[90393]: 2022-05-14T01:02:12.182+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:02:12.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:02:12 smithi115 ceph-mon[78593]: pgmap v3185: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:02:12.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:02:12 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:02:12.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:02:12 smithi047 ceph-mon[77147]: pgmap v3185: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:02:12.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:02:12 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:02:13.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:02:13 smithi115 conmon[90393]: 2022-05-14T01:02:13.200+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:02:13.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:02:13 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:02:13.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:02:13 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:02:14.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:02:14 smithi115 conmon[90393]: 2022-05-14T01:02:14.240+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:02:14.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:02:14 smithi115 ceph-mon[78593]: pgmap v3186: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:02:14.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:02:14 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:02:14.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:02:14 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:02:14.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:02:14 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:02:14.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:02:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T01:02:14.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:02:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T01:02:14.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:02:14 smithi047 ceph-mon[77147]: pgmap v3186: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:02:14.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:02:14 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:02:14.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:02:14 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:02:14.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:02:14 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:02:14.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:02:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T01:02:14.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:02:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T01:02:15.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:02:15 smithi115 conmon[90393]: 2022-05-14T01:02:15.235+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:02:15.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:02:15 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:02:15.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:02:15 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 5665 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:02:15.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:02:15 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:02:15.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:02:15 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 5665 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:02:16.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:02:16 smithi115 ceph-mon[78593]: pgmap v3187: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:02:16.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:02:16 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:02:16.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:02:16 smithi115 conmon[90393]: 2022-05-14T01:02:16.254+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:02:16.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:02:16 smithi047 ceph-mon[77147]: pgmap v3187: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:02:16.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:02:16 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:02:17.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:02:17 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:02:17.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:02:17 smithi115 conmon[90393]: 2022-05-14T01:02:17.303+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:02:17.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:02:17 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:02:18.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:02:18 smithi115 conmon[90393]: 2022-05-14T01:02:18.262+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:02:18.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:02:18 smithi115 ceph-mon[78593]: pgmap v3188: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:02:18.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:02:18 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:02:18.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:02:18 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:02:18.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:02:18 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:02:18.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:02:18 smithi047 ceph-mon[77147]: pgmap v3188: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:02:18.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:02:18 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:02:18.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:02:18 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:02:18.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:02:18 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:02:19.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:02:19 smithi115 conmon[90393]: 2022-05-14T01:02:19.311+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:02:19.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:02:19 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:02:19.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:02:19 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:02:20.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:02:20 smithi115 conmon[90393]: 2022-05-14T01:02:20.281+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:02:20.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:02:20 smithi115 ceph-mon[78593]: pgmap v3189: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:02:20.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:02:20 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:02:20.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:02:20 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 5670 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:02:20.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:02:20 smithi047 ceph-mon[77147]: pgmap v3189: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:02:20.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:02:20 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:02:20.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:02:20 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 5670 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:02:21.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:02:21 smithi115 conmon[90393]: 2022-05-14T01:02:21.283+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:02:21.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:02:21 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:02:21.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:02:21 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:02:22.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:02:22 smithi115 conmon[90393]: 2022-05-14T01:02:22.325+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:02:22.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:02:22 smithi115 ceph-mon[78593]: pgmap v3190: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:02:22.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:02:22 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:02:22.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:02:22 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:02:22.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:02:22 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:02:22.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:02:22 smithi047 ceph-mon[77147]: pgmap v3190: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:02:22.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:02:22 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:02:22.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:02:22 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:02:22.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:02:22 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:02:23.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:02:23 smithi115 conmon[90393]: 2022-05-14T01:02:23.295+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:02:23.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:02:23 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:02:23.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:02:23 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:02:24.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:02:24 smithi115 conmon[90393]: 2022-05-14T01:02:24.285+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:02:24.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:02:24 smithi115 ceph-mon[78593]: pgmap v3191: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:02:24.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:02:24 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:02:24.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:02:24 smithi047 ceph-mon[77147]: pgmap v3191: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:02:24.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:02:24 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:02:25.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:02:25 smithi115 conmon[90393]: 2022-05-14T01:02:25.319+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:02:25.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:02:25 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:02:25.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:02:25 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 5675 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:02:25.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:02:25 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:02:25.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:02:25 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 5675 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:02:26.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:02:26 smithi115 conmon[90393]: 2022-05-14T01:02:26.350+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:02:26.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:02:26 smithi115 ceph-mon[78593]: pgmap v3192: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:02:26.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:02:26 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:02:26.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:02:26 smithi047 ceph-mon[77147]: pgmap v3192: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:02:26.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:02:26 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:02:27.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:02:27 smithi115 conmon[90393]: 2022-05-14T01:02:27.325+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:02:27.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:02:27 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:02:27.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:02:27 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:02:28.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:02:28 smithi115 conmon[90393]: 2022-05-14T01:02:28.311+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:02:28.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:02:28 smithi115 ceph-mon[78593]: pgmap v3193: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:02:28.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:02:28 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:02:28.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:02:28 smithi047 ceph-mon[77147]: pgmap v3193: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:02:28.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:02:28 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:02:29.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:02:29 smithi115 conmon[90393]: 2022-05-14T01:02:29.328+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:02:29.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:02:29 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:02:29.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:02:29 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:02:30.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:02:30 smithi115 conmon[90393]: 2022-05-14T01:02:30.304+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:02:30.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:02:30 smithi115 ceph-mon[78593]: pgmap v3194: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:02:30.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:02:30 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:02:30.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:02:30 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:02:30.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:02:30 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:02:30.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:02:30 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 5680 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:02:30.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:02:30 smithi047 ceph-mon[77147]: pgmap v3194: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:02:30.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:02:30 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:02:30.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:02:30 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:02:30.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:02:30 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:02:30.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:02:30 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 5680 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:02:31.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:02:31 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:02:31.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:02:31 smithi115 conmon[90393]: 2022-05-14T01:02:31.316+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:02:31.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:02:31 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:02:32.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:02:32 smithi115 conmon[90393]: 2022-05-14T01:02:32.286+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:02:32.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:02:32 smithi115 ceph-mon[78593]: pgmap v3195: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:02:32.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:02:32 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:02:32.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:02:32 smithi047 ceph-mon[77147]: pgmap v3195: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:02:32.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:02:32 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:02:33.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:02:33 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:02:33.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:02:33 smithi115 conmon[90393]: 2022-05-14T01:02:33.271+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:02:33.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:02:33 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:02:34.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:02:34 smithi115 conmon[90393]: 2022-05-14T01:02:34.288+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:02:34.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:02:34 smithi115 ceph-mon[78593]: pgmap v3196: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:02:34.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:02:34 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:02:34.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:02:34 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:02:34.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:02:34 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:02:34.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:02:34 smithi047 ceph-mon[77147]: pgmap v3196: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:02:34.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:02:34 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:02:34.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:02:34 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:02:34.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:02:34 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:02:35.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:02:35 smithi115 conmon[90393]: 2022-05-14T01:02:35.319+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:02:35.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:02:35 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:02:35.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:02:35 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 5685 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:02:35.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:02:35 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:02:35.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:02:35 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 5685 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:02:36.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:02:36 smithi115 conmon[90393]: 2022-05-14T01:02:36.347+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:02:36.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:02:36 smithi115 ceph-mon[78593]: pgmap v3197: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:02:36.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:02:36 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:02:36.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:02:36 smithi047 ceph-mon[77147]: pgmap v3197: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:02:36.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:02:36 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:02:37.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:02:37 smithi115 conmon[90393]: 2022-05-14T01:02:37.338+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:02:37.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:02:37 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:02:37.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:02:37 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:02:38.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:02:38 smithi115 conmon[90393]: 2022-05-14T01:02:38.342+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:02:38.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:02:38 smithi115 ceph-mon[78593]: pgmap v3198: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:02:38.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:02:38 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:02:38.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:02:38 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:02:38.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:02:38 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:02:38.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:02:38 smithi047 ceph-mon[77147]: pgmap v3198: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:02:38.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:02:38 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:02:38.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:02:38 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:02:38.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:02:38 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:02:39.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:02:39 smithi115 conmon[90393]: 2022-05-14T01:02:39.313+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:02:39.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:02:39 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:02:39.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:02:39 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:02:40.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:02:40 smithi115 conmon[90393]: 2022-05-14T01:02:40.279+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:02:40.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:02:40 smithi115 ceph-mon[78593]: pgmap v3199: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:02:40.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:02:40 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:02:40.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:02:40 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 5690 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:02:40.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:02:40 smithi047 ceph-mon[77147]: pgmap v3199: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:02:40.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:02:40 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:02:40.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:02:40 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 5690 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:02:41.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:02:41 smithi115 conmon[90393]: 2022-05-14T01:02:41.256+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:02:41.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:02:41 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:02:41.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:02:41 smithi115 ceph-mon[78593]: pgmap v3200: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:02:41.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:02:41 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:02:41.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:02:41 smithi047 ceph-mon[77147]: pgmap v3200: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:02:42.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:02:42 smithi115 conmon[90393]: 2022-05-14T01:02:42.212+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:02:42.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:02:42 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:02:42.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:02:42 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:02:42.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:02:42 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:02:42.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:02:42 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:02:42.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:02:42 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:02:42.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:02:42 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:02:43.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:02:43 smithi115 conmon[90393]: 2022-05-14T01:02:43.238+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:02:43.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:02:43 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:02:43.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:02:43 smithi115 ceph-mon[78593]: pgmap v3201: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:02:43.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:02:43 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:02:43.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:02:43 smithi047 ceph-mon[77147]: pgmap v3201: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:02:44.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:02:44 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:02:44.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:02:44 smithi115 conmon[90393]: 2022-05-14T01:02:44.262+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:02:44.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:02:44 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:02:45.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:02:45 smithi115 conmon[90393]: 2022-05-14T01:02:45.254+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:02:45.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:02:45 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:02:45.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:02:45 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 5695 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:02:45.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:02:45 smithi115 ceph-mon[78593]: pgmap v3202: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:02:45.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:02:45 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:02:45.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:02:45 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 5695 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:02:45.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:02:45 smithi047 ceph-mon[77147]: pgmap v3202: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:02:46.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:02:46 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:02:46.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:02:46 smithi115 conmon[90393]: 2022-05-14T01:02:46.223+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:02:46.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:02:46 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:02:47.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:02:47 smithi115 conmon[90393]: 2022-05-14T01:02:47.221+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:02:47.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:02:47 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:02:47.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:02:47 smithi115 ceph-mon[78593]: pgmap v3203: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:02:47.698 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:02:47 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:02:47.699 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:02:47 smithi047 ceph-mon[77147]: pgmap v3203: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:02:48.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:02:48 smithi115 conmon[90393]: 2022-05-14T01:02:48.201+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:02:48.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:02:48 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:02:48.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:02:48 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T01:02:48.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:02:48 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T01:02:48.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:02:48 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T01:02:48.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:02:48 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:02:48.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:02:48 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T01:02:48.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:02:48 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T01:02:48.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:02:48 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T01:02:49.457 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:02:49 smithi115 conmon[90393]: 2022-05-14T01:02:49.154+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:02:49.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:02:49 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:02:49.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:02:49 smithi047 ceph-mon[77147]: pgmap v3204: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:02:49.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:02:49 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 5700 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:02:49.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:02:49 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:02:49.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:02:49 smithi115 ceph-mon[78593]: pgmap v3204: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:02:49.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:02:49 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 5700 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:02:50.457 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:02:50 smithi115 conmon[90393]: 2022-05-14T01:02:50.202+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:02:50.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:02:50 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:02:50.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:02:50 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:02:50.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:02:50 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:02:50.792 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:02:50 smithi047 conmon[78521]: 2022-05-14T01:02:50.671+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:02:50.792 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:02:50 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T01:02:50.792 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:02:50 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:02:50.793 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:02:50 smithi047 conmon[78521]: yield 2022-05-14T01:02:50.793 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:02:50 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:02:50.793 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:02:50 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:02:50.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:02:50 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:02:50.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:02:50 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T01:02:50.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:02:50 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:02:50.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:02:50 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T01:02:50.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:02:50 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:02:50.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:02:50 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T01:02:50.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:02:50 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:02:50.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:02:50 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T01:02:50.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:02:50 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:02:50.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:02:50 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T01:02:50.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:02:50 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:02:50.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:02:50 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:02:50.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:02:50 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:02:50.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:02:50 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:02:50.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:02:50 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:02:50.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:02:50 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:02:51.462 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:02:51 smithi115 conmon[90393]: 2022-05-14T01:02:51.189+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:02:51.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:02:51 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:02:51.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:02:51 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:02:51.791 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T01:02:51.792 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:02:51.792 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T01:02:51.792 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:02:51.792 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:02:51.792 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:02:51.793 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T01:02:51.793 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:02:51.793 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T01:02:51.793 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:02:51.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T01:02:51.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:02:51.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T01:02:51.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:02:51.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T01:02:51.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:02:51.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:02:51.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:02:51.795 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:02:51 smithi047 ceph-mon[77147]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T01:02:51.796 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:02:51 smithi047 ceph-mon[77147]: pgmap v3205: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:02:51.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:02:51 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:02:51.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:02:51 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:02:51.858 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T01:02:51.858 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:02:51.859 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T01:02:51.859 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:02:51.859 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:02:51.859 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:02:51.859 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T01:02:51.860 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:02:51.860 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T01:02:51.860 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:02:51.860 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T01:02:51.860 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:02:51.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T01:02:51.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:02:51.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T01:02:51.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:02:51.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:02:51.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:02:51.862 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:02:51 smithi115 ceph-mon[78593]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T01:02:51.862 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:02:51 smithi115 ceph-mon[78593]: pgmap v3205: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:02:52.463 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:02:52 smithi115 conmon[90393]: 2022-05-14T01:02:52.165+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:02:52.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:02:52 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:02:52.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:02:52 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:02:53.468 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:02:53 smithi115 conmon[90393]: 2022-05-14T01:02:53.198+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:02:53.745 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:02:53 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:02:53.745 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:02:53 smithi047 ceph-mon[77147]: pgmap v3206: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:02:53.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:02:53 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:02:53.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:02:53 smithi115 ceph-mon[78593]: pgmap v3206: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:02:54.041 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:02:53 smithi047 conmon[78521]: 2022-05-14T01:02:53.743+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:02:54.041 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:02:53 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T01:02:54.042 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:02:53 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:02:54.042 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:02:53 smithi047 conmon[78521]: yield 2022-05-14T01:02:54.042 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:02:53 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:02:54.042 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:02:53 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:02:54.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:02:53 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:02:54.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:02:53 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T01:02:54.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:02:53 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:02:54.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:02:53 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T01:02:54.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:02:53 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:02:54.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:02:53 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T01:02:54.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:02:53 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:02:54.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:02:53 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T01:02:54.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:02:53 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:02:54.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:02:53 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T01:02:54.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:02:53 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:02:54.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:02:53 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:02:54.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:02:53 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:02:54.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:02:53 smithi047 conmon[78521]: 2022-05-14T01:02:53.744+0000 7f5340f7a700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T01:02:54.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:02:53 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T01:02:54.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:02:53 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:02:54.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:02:53 smithi047 conmon[78521]: yield 2022-05-14T01:02:54.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:02:53 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:02:54.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:02:53 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:02:54.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:02:53 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:02:54.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:02:53 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T01:02:54.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:02:53 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:02:54.048 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:02:53 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T01:02:54.048 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:02:53 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:02:54.048 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:02:53 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T01:02:54.048 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:02:53 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:02:54.048 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:02:53 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T01:02:54.049 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:02:53 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:02:54.049 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:02:53 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T01:02:54.049 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:02:53 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:02:54.049 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:02:53 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:02:54.050 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:02:53 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:02:54.050 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:02:53 smithi047 conmon[78521]: 2022-05-14T01:02:54.050 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:02:53 smithi047 conmon[78521]: During handling of the above exception, another exception occurred: 2022-05-14T01:02:54.050 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:02:53 smithi047 conmon[78521]: 2022-05-14T01:02:54.051 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:02:53 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T01:02:54.051 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:02:53 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T01:02:54.051 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:02:53 smithi047 conmon[78521]: return f(*arg) 2022-05-14T01:02:54.052 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:02:53 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T01:02:54.052 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:02:53 smithi047 conmon[78521]: self._write_client_files(client_files, host) 2022-05-14T01:02:54.052 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:02:53 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T01:02:54.052 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:02:53 smithi047 conmon[78521]: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T01:02:54.053 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:02:53 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T01:02:54.053 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:02:53 smithi047 conmon[78521]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T01:02:54.053 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:02:53 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T01:02:54.053 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:02:53 smithi047 conmon[78521]: return self.event_loop.get_result(coro) 2022-05-14T01:02:54.054 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:02:53 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T01:02:54.054 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:02:53 smithi047 conmon[78521]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T01:02:54.054 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:02:53 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T01:02:54.054 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:02:53 smithi047 conmon[78521]: return self.__get_result() 2022-05-14T01:02:54.055 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:02:53 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T01:02:54.055 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:02:53 smithi047 conmon[78521]: raise self._exception 2022-05-14T01:02:54.055 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:02:53 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T01:02:54.056 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:02:53 smithi047 conmon[78521]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T01:02:54.056 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:02:53 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T01:02:54.056 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:02:53 smithi047 conmon[78521]: conn = await self._remote_connection(host, addr) 2022-05-14T01:02:54.056 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:02:53 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T01:02:54.056 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:02:53 smithi047 conmon[78521]: raise 2022-05-14T01:02:54.057 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:02:53 smithi047 conmon[78521]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T01:02:54.057 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:02:53 smithi047 conmon[78521]: self.gen.throw(type, value, traceback) 2022-05-14T01:02:54.057 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:02:53 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T01:02:54.057 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:02:53 smithi047 conmon[78521]: raise OrchestratorError(msg) 2022-05-14T01:02:54.058 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:02:53 smithi047 conmon[78521]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:02:54.470 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:02:54 smithi115 conmon[90393]: 2022-05-14T01:02:54.187+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:02:54.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:02:54 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:02:54.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:02:54 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:02:54.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:02:54 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:02:54.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:02:54 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:02:54.792 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T01:02:54.792 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:02:54.792 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T01:02:54.793 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:02:54.793 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:02:54.793 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:02:54.793 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T01:02:54.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:02:54.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T01:02:54.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:02:54.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T01:02:54.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:02:54.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T01:02:54.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:02:54.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T01:02:54.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:02:54.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:02:54.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:02:54.796 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:02:54 smithi047 ceph-mon[77147]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T01:02:54.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T01:02:54.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:02:54.797 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T01:02:54.797 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:02:54.797 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:02:54.797 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:02:54.797 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T01:02:54.798 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:02:54.798 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T01:02:54.798 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:02:54.798 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T01:02:54.799 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:02:54.799 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T01:02:54.799 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:02:54.799 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T01:02:54.799 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:02:54.800 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:02:54.800 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:02:54.800 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T01:02:54.800 INFO:journalctl@ceph.mon.b.smithi047.stdout: During handling of the above exception, another exception occurred: 2022-05-14T01:02:54.800 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T01:02:54.801 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T01:02:54.801 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T01:02:54.801 INFO:journalctl@ceph.mon.b.smithi047.stdout: return f(*arg) 2022-05-14T01:02:54.801 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T01:02:54.802 INFO:journalctl@ceph.mon.b.smithi047.stdout: self._write_client_files(client_files, host) 2022-05-14T01:02:54.802 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T01:02:54.802 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T01:02:54.802 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T01:02:54.802 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T01:02:54.803 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T01:02:54.803 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.event_loop.get_result(coro) 2022-05-14T01:02:54.803 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T01:02:54.803 INFO:journalctl@ceph.mon.b.smithi047.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T01:02:54.804 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T01:02:54.804 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.__get_result() 2022-05-14T01:02:54.804 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T01:02:54.804 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise self._exception 2022-05-14T01:02:54.804 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T01:02:54.805 INFO:journalctl@ceph.mon.b.smithi047.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T01:02:54.805 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T01:02:54.805 INFO:journalctl@ceph.mon.b.smithi047.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T01:02:54.805 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T01:02:54.805 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise 2022-05-14T01:02:54.806 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T01:02:54.806 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.gen.throw(type, value, traceback) 2022-05-14T01:02:54.806 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T01:02:54.806 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OrchestratorError(msg) 2022-05-14T01:02:54.806 INFO:journalctl@ceph.mon.b.smithi047.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:02:54.807 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:02:54 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 5705 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:02:54.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:02:54 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:02:54.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:02:54 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:02:54.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:02:54 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:02:54.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:02:54 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:02:54.859 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T01:02:54.859 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:02:54.859 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T01:02:54.859 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:02:54.860 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:02:54.860 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:02:54.860 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T01:02:54.860 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:02:54.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T01:02:54.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:02:54.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T01:02:54.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:02:54.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T01:02:54.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:02:54.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T01:02:54.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:02:54.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:02:54.863 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:02:54.863 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:02:54 smithi115 ceph-mon[78593]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T01:02:54.863 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T01:02:54.863 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:02:54.864 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T01:02:54.864 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:02:54.864 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:02:54.864 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:02:54.864 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T01:02:54.865 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:02:54.865 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T01:02:54.865 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:02:54.865 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T01:02:54.866 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:02:54.866 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T01:02:54.866 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:02:54.866 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T01:02:54.866 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:02:54.867 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:02:54.867 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:02:54.867 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T01:02:54.867 INFO:journalctl@ceph.mon.c.smithi115.stdout: During handling of the above exception, another exception occurred: 2022-05-14T01:02:54.867 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T01:02:54.868 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T01:02:54.868 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T01:02:54.868 INFO:journalctl@ceph.mon.c.smithi115.stdout: return f(*arg) 2022-05-14T01:02:54.868 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T01:02:54.868 INFO:journalctl@ceph.mon.c.smithi115.stdout: self._write_client_files(client_files, host) 2022-05-14T01:02:54.869 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T01:02:54.869 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T01:02:54.869 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T01:02:54.869 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T01:02:54.870 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T01:02:54.870 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.event_loop.get_result(coro) 2022-05-14T01:02:54.870 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T01:02:54.870 INFO:journalctl@ceph.mon.c.smithi115.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T01:02:54.871 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T01:02:54.871 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.__get_result() 2022-05-14T01:02:54.871 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T01:02:54.871 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise self._exception 2022-05-14T01:02:54.871 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T01:02:54.872 INFO:journalctl@ceph.mon.c.smithi115.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T01:02:54.872 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T01:02:54.872 INFO:journalctl@ceph.mon.c.smithi115.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T01:02:54.872 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T01:02:54.872 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise 2022-05-14T01:02:54.873 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T01:02:54.873 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.gen.throw(type, value, traceback) 2022-05-14T01:02:54.873 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T01:02:54.873 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OrchestratorError(msg) 2022-05-14T01:02:54.874 INFO:journalctl@ceph.mon.c.smithi115.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:02:54.874 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:02:54 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 5705 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:02:55.475 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:02:55 smithi115 conmon[90393]: 2022-05-14T01:02:55.148+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:02:55.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:02:55 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:02:55.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:02:55 smithi047 ceph-mon[77147]: pgmap v3207: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:02:55.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:02:55 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:02:55.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:02:55 smithi115 ceph-mon[78593]: pgmap v3207: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:02:56.477 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:02:56 smithi115 conmon[90393]: 2022-05-14T01:02:56.123+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:02:56.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:02:56 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:02:56.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:02:56 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:02:57.482 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:02:57 smithi115 conmon[90393]: 2022-05-14T01:02:57.167+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:02:57.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:02:57 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:02:57.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:02:57 smithi047 ceph-mon[77147]: pgmap v3208: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:02:57.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:02:57 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:02:57.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:02:57 smithi115 ceph-mon[78593]: pgmap v3208: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:02:58.483 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:02:58 smithi115 conmon[90393]: 2022-05-14T01:02:58.216+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:02:58.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:02:58 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:02:58.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:02:58 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:02:58.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:02:58 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:02:58.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:02:58 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:02:58.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:02:58 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:02:58.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:02:58 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:02:59.488 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:02:59 smithi115 conmon[90393]: 2022-05-14T01:02:59.182+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:02:59.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:02:59 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:02:59.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:02:59 smithi047 ceph-mon[77147]: pgmap v3209: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:02:59.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:02:59 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 5710 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:02:59.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:02:59 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:02:59.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:02:59 smithi115 ceph-mon[78593]: pgmap v3209: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:02:59.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:02:59 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 5710 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:03:00.489 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:03:00 smithi115 conmon[90393]: 2022-05-14T01:03:00.158+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:03:00.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:03:00 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:03:00.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:03:00 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:03:01.497 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:03:01 smithi115 conmon[90393]: 2022-05-14T01:03:01.196+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:03:01.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:03:01 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:03:01.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:03:01 smithi047 ceph-mon[77147]: pgmap v3210: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:03:01.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:03:01 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:03:01.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:03:01 smithi115 ceph-mon[78593]: pgmap v3210: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:03:02.496 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:03:02 smithi115 conmon[90393]: 2022-05-14T01:03:02.225+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:03:02.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:03:02 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:03:02.814 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:03:02 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:03:02.815 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:03:02 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:03:02.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:03:02 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:03:02.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:03:02 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:03:02.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:03:02 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:03:03.500 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:03:03 smithi115 conmon[90393]: 2022-05-14T01:03:03.180+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:03:03.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:03:03 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:03:03.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:03:03 smithi047 ceph-mon[77147]: pgmap v3211: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:03:03.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:03:03 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:03:03.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:03:03 smithi115 ceph-mon[78593]: pgmap v3211: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:03:04.502 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:03:04 smithi115 conmon[90393]: 2022-05-14T01:03:04.181+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:03:04.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:03:04 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:03:04.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:03:04 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 5715 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:03:04.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:03:04 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:03:04.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:03:04 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 5715 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:03:05.508 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:03:05 smithi115 conmon[90393]: 2022-05-14T01:03:05.134+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:03:05.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:03:05 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:03:05.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:03:05 smithi047 ceph-mon[77147]: pgmap v3212: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:03:05.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:03:05 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:03:05.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:03:05 smithi115 ceph-mon[78593]: pgmap v3212: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:03:06.509 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:03:06 smithi115 conmon[90393]: 2022-05-14T01:03:06.108+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:03:06.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:03:06 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:03:06.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:03:06 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:03:07.514 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:03:07 smithi115 conmon[90393]: 2022-05-14T01:03:07.129+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:03:07.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:03:07 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:03:07.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:03:07 smithi047 ceph-mon[77147]: pgmap v3213: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:03:07.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:03:07 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:03:07.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:03:07 smithi115 ceph-mon[78593]: pgmap v3213: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:03:08.515 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:03:08 smithi115 conmon[90393]: 2022-05-14T01:03:08.122+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:03:08.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:03:08 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:03:08.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:03:08 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:03:09.523 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:03:09 smithi115 conmon[90393]: 2022-05-14T01:03:09.161+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:03:09.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:03:09 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:03:09.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:03:09 smithi047 ceph-mon[77147]: pgmap v3214: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:03:09.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:03:09 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 5720 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:03:09.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:03:09 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:03:09.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:03:09 smithi115 ceph-mon[78593]: pgmap v3214: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:03:09.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:03:09 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 5720 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:03:10.523 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:03:10 smithi115 conmon[90393]: 2022-05-14T01:03:10.143+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:03:10.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:03:10 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:03:10.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:03:10 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:03:10.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:03:10 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:03:10.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:03:10 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:03:10.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:03:10 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:03:10.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:03:10 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:03:11.527 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:03:11 smithi115 conmon[90393]: 2022-05-14T01:03:11.120+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:03:11.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:03:11 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:03:11.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:03:11 smithi047 ceph-mon[77147]: pgmap v3215: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:03:11.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:03:11 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:03:11.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:03:11 smithi115 ceph-mon[78593]: pgmap v3215: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:03:12.530 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:03:12 smithi115 conmon[90393]: 2022-05-14T01:03:12.142+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:03:12.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:03:12 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:03:12.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:03:12 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:03:13.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:03:13 smithi115 conmon[90393]: 2022-05-14T01:03:13.102+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:03:13.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:03:13 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:03:13.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:03:13 smithi047 ceph-mon[77147]: pgmap v3216: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:03:13.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:03:13 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:03:13.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:03:13 smithi115 ceph-mon[78593]: pgmap v3216: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:03:14.536 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:03:14 smithi115 conmon[90393]: 2022-05-14T01:03:14.118+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:03:14.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:03:14 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:03:14.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:03:14 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:03:14.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:03:14 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:03:14.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:03:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T01:03:14.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:03:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T01:03:14.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:03:14 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 5725 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:03:14.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:03:14 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:03:14.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:03:14 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:03:14.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:03:14 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:03:14.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:03:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T01:03:14.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:03:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T01:03:14.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:03:14 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 5725 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:03:15.541 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:03:15 smithi115 conmon[90393]: 2022-05-14T01:03:15.149+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:03:15.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:03:15 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:03:15.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:03:15 smithi115 ceph-mon[78593]: pgmap v3217: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:03:16.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:03:15 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:03:16.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:03:15 smithi047 ceph-mon[77147]: pgmap v3217: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:03:16.543 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:03:16 smithi115 conmon[90393]: 2022-05-14T01:03:16.161+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:03:16.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:03:16 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:03:17.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:03:16 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:03:17.549 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:03:17 smithi115 conmon[90393]: 2022-05-14T01:03:17.142+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:03:17.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:03:17 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:03:17.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:03:17 smithi115 ceph-mon[78593]: pgmap v3218: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:03:18.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:03:17 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:03:18.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:03:17 smithi047 ceph-mon[77147]: pgmap v3218: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:03:18.550 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:03:18 smithi115 conmon[90393]: 2022-05-14T01:03:18.117+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:03:18.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:03:18 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:03:18.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:03:18 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:03:18.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:03:18 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:03:19.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:03:18 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:03:19.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:03:18 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:03:19.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:03:18 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:03:19.554 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:03:19 smithi115 conmon[90393]: 2022-05-14T01:03:19.152+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:03:19.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:03:19 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:03:19.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:03:19 smithi115 ceph-mon[78593]: pgmap v3219: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:03:19.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:03:19 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 5730 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:03:20.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:03:19 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:03:20.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:03:19 smithi047 ceph-mon[77147]: pgmap v3219: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:03:20.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:03:19 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 5730 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:03:20.556 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:03:20 smithi115 conmon[90393]: 2022-05-14T01:03:20.200+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:03:20.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:03:20 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:03:21.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:03:20 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:03:21.561 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:03:21 smithi115 conmon[90393]: 2022-05-14T01:03:21.201+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:03:21.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:03:21 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:03:21.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:03:21 smithi115 ceph-mon[78593]: pgmap v3220: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:03:22.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:03:21 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:03:22.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:03:21 smithi047 ceph-mon[77147]: pgmap v3220: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:03:22.562 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:03:22 smithi115 conmon[90393]: 2022-05-14T01:03:22.156+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:03:22.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:03:22 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:03:22.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:03:22 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:03:22.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:03:22 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:03:23.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:03:22 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:03:23.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:03:22 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:03:23.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:03:22 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:03:23.567 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:03:23 smithi115 conmon[90393]: 2022-05-14T01:03:23.153+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:03:23.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:03:23 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:03:23.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:03:23 smithi115 ceph-mon[78593]: pgmap v3221: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:03:24.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:03:23 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:03:24.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:03:23 smithi047 ceph-mon[77147]: pgmap v3221: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:03:24.568 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:03:24 smithi115 conmon[90393]: 2022-05-14T01:03:24.201+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:03:24.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:03:24 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:03:24.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:03:24 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 5735 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:03:25.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:03:24 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:03:25.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:03:24 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 5735 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:03:25.573 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:03:25 smithi115 conmon[90393]: 2022-05-14T01:03:25.198+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:03:25.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:03:25 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:03:25.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:03:25 smithi115 ceph-mon[78593]: pgmap v3222: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:03:26.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:03:25 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:03:26.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:03:25 smithi047 ceph-mon[77147]: pgmap v3222: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:03:26.575 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:03:26 smithi115 conmon[90393]: 2022-05-14T01:03:26.181+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:03:26.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:03:26 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:03:27.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:03:26 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:03:27.580 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:03:27 smithi115 conmon[90393]: 2022-05-14T01:03:27.136+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:03:27.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:03:27 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:03:27.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:03:27 smithi115 ceph-mon[78593]: pgmap v3223: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:03:28.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:03:27 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:03:28.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:03:27 smithi047 ceph-mon[77147]: pgmap v3223: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:03:28.581 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:03:28 smithi115 conmon[90393]: 2022-05-14T01:03:28.118+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:03:28.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:03:28 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:03:29.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:03:28 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:03:29.586 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:03:29 smithi115 conmon[90393]: 2022-05-14T01:03:29.141+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:03:29.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:03:29 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:03:29.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:03:29 smithi115 ceph-mon[78593]: pgmap v3224: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:03:29.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:03:29 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 5740 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:03:30.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:03:29 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:03:30.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:03:29 smithi047 ceph-mon[77147]: pgmap v3224: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:03:30.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:03:29 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 5740 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:03:30.587 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:03:30 smithi115 conmon[90393]: 2022-05-14T01:03:30.138+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:03:30.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:03:30 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:03:30.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:03:30 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:03:30.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:03:30 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:03:31.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:03:30 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:03:31.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:03:30 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:03:31.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:03:30 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:03:31.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:03:31 smithi115 conmon[90393]: 2022-05-14T01:03:31.099+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:03:31.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:03:31 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:03:31.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:03:31 smithi115 ceph-mon[78593]: pgmap v3225: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:03:31.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:03:31 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:03:32.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:03:31 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:03:32.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:03:31 smithi047 ceph-mon[77147]: pgmap v3225: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:03:32.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:03:31 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:03:32.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:03:32 smithi115 conmon[90393]: 2022-05-14T01:03:32.080+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:03:32.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:03:32 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:03:33.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:03:32 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:03:33.599 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:03:33 smithi115 conmon[90393]: 2022-05-14T01:03:33.123+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:03:33.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:03:33 smithi115 ceph-mon[78593]: pgmap v3226: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:03:34.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:03:33 smithi047 ceph-mon[77147]: pgmap v3226: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:03:34.600 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:03:34 smithi115 conmon[90393]: 2022-05-14T01:03:34.118+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:03:34.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:03:34 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:03:34.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:03:34 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:03:34.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:03:34 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:03:34.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:03:34 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 5745 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:03:35.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:03:34 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:03:35.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:03:34 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:03:35.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:03:34 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:03:35.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:03:34 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 5745 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:03:35.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:03:35 smithi115 conmon[90393]: 2022-05-14T01:03:35.091+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:03:35.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:03:35 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:03:35.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:03:35 smithi115 ceph-mon[78593]: pgmap v3227: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:03:35.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:03:35 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:03:36.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:03:35 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:03:36.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:03:35 smithi047 ceph-mon[77147]: pgmap v3227: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:03:36.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:03:35 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:03:36.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:03:36 smithi115 conmon[90393]: 2022-05-14T01:03:36.106+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:03:37.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:03:37 smithi115 conmon[90393]: 2022-05-14T01:03:37.088+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:03:38.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:03:37 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:03:38.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:03:37 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:03:38.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:03:37 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:03:38.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:03:37 smithi047 ceph-mon[77147]: pgmap v3228: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:03:38.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:03:37 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:03:38.067 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:03:37 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:03:38.068 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:03:37 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:03:38.068 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:03:37 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:03:38.068 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:03:37 smithi115 ceph-mon[78593]: pgmap v3228: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:03:38.068 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:03:37 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:03:38.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:03:38 smithi115 conmon[90393]: 2022-05-14T01:03:38.066+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:03:39.028 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:03:38 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:03:39.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:03:38 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:03:39.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:03:39 smithi115 conmon[90393]: 2022-05-14T01:03:39.027+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:03:40.018 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:03:39 smithi115 ceph-mon[78593]: pgmap v3229: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:03:40.018 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:03:39 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:03:40.018 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:03:39 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 5750 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:03:40.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:03:39 smithi047 ceph-mon[77147]: pgmap v3229: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:03:40.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:03:39 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:03:40.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:03:39 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 5750 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:03:40.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:03:40 smithi115 conmon[90393]: 2022-05-14T01:03:40.016+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:03:41.030 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:03:40 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:03:41.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:03:40 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:03:41.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:03:41 smithi115 conmon[90393]: 2022-05-14T01:03:41.030+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:03:42.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:03:41 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:03:42.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:03:41 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:03:42.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:03:41 smithi047 ceph-mon[77147]: pgmap v3230: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:03:42.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:03:41 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:03:42.050 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:03:41 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:03:42.050 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:03:41 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:03:42.051 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:03:41 smithi115 ceph-mon[78593]: pgmap v3230: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:03:42.051 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:03:41 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:03:42.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:03:42 smithi115 conmon[90393]: 2022-05-14T01:03:42.049+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:03:43.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:03:42 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:03:43.060 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:03:42 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:03:43.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:03:43 smithi115 conmon[90393]: 2022-05-14T01:03:43.059+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:03:44.024 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:03:43 smithi115 ceph-mon[78593]: pgmap v3231: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:03:44.025 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:03:43 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:03:44.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:03:43 smithi047 ceph-mon[77147]: pgmap v3231: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:03:44.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:03:43 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:03:44.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:03:44 smithi115 conmon[90393]: 2022-05-14T01:03:44.023+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:03:44.975 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:03:44 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:03:44.975 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:03:44 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 5755 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:03:45.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:03:44 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:03:45.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:03:44 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 5755 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:03:45.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:03:44 smithi115 conmon[90393]: 2022-05-14T01:03:44.974+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:03:46.003 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:03:45 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:03:46.003 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:03:45 smithi115 ceph-mon[78593]: pgmap v3232: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:03:46.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:03:45 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:03:46.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:03:45 smithi047 ceph-mon[77147]: pgmap v3232: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:03:46.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:03:46 smithi115 conmon[90393]: 2022-05-14T01:03:46.001+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:03:46.956 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:03:46 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:03:47.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:03:46 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:03:47.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:03:46 smithi115 conmon[90393]: 2022-05-14T01:03:46.955+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:03:47.907 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:03:47 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:03:47.907 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:03:47 smithi115 ceph-mon[78593]: pgmap v3233: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:03:48.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:03:47 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:03:48.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:03:47 smithi047 ceph-mon[77147]: pgmap v3233: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:03:48.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:03:47 smithi115 conmon[90393]: 2022-05-14T01:03:47.906+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:03:48.934 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:03:48 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:03:49.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:03:48 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:03:49.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:03:48 smithi115 conmon[90393]: 2022-05-14T01:03:48.933+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:03:49.947 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:03:49 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:03:49.947 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:03:49 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:03:49.948 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:03:49 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:03:49.948 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:03:49 smithi115 ceph-mon[78593]: pgmap v3234: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:03:49.948 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:03:49 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 5760 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:03:50.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:03:49 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:03:50.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:03:49 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:03:50.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:03:49 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:03:50.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:03:49 smithi047 ceph-mon[77147]: pgmap v3234: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:03:50.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:03:49 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 5760 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:03:50.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:03:49 smithi115 conmon[90393]: 2022-05-14T01:03:49.946+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:03:50.934 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:03:50 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:03:51.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:03:50 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:03:51.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:03:50 smithi115 conmon[90393]: 2022-05-14T01:03:50.933+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:03:51.935 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:03:51 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:03:51.935 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:03:51 smithi115 ceph-mon[78593]: pgmap v3235: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:03:52.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:03:51 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:03:52.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:03:51 smithi047 ceph-mon[77147]: pgmap v3235: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:03:52.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:03:51 smithi115 conmon[90393]: 2022-05-14T01:03:51.933+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:03:53.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:03:52 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:03:53.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:03:52 smithi115 conmon[90393]: 2022-05-14T01:03:52.908+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:03:53.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:03:52 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:03:54.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:03:53 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:03:54.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:03:53 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:03:54.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:03:53 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:03:54.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:03:53 smithi047 ceph-mon[77147]: pgmap v3236: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:03:54.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:03:53 smithi115 conmon[90393]: 2022-05-14T01:03:53.919+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:03:54.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:03:53 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:03:54.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:03:53 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:03:54.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:03:53 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:03:54.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:03:53 smithi115 ceph-mon[78593]: pgmap v3236: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:03:54.541 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:03:54 smithi047 conmon[78521]: 2022-05-14T01:03:54.223+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:03:54.541 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:03:54 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T01:03:54.541 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:03:54 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:03:54.542 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:03:54 smithi047 conmon[78521]: yield 2022-05-14T01:03:54.542 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:03:54 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:03:54.542 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:03:54 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:03:54.542 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:03:54 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:03:54.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:03:54 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T01:03:54.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:03:54 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:03:54.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:03:54 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T01:03:54.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:03:54 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:03:54.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:03:54 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T01:03:54.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:03:54 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:03:54.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:03:54 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T01:03:54.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:03:54 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:03:54.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:03:54 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T01:03:54.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:03:54 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:03:54.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:03:54 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:03:54.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:03:54 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:03:55.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:03:54 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T01:03:55.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:03:54 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T01:03:55.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:03:54 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T01:03:55.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:03:54 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:03:55.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:03:54 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:03:55.042 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T01:03:55.042 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:03:55.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T01:03:55.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:03:55.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:03:55.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:03:55.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T01:03:55.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:03:55.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T01:03:55.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:03:55.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T01:03:55.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:03:55.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T01:03:55.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:03:55.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T01:03:55.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:03:55.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:03:55.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:03:55.047 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:03:54 smithi047 ceph-mon[77147]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T01:03:55.047 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:03:54 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 5765 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:03:55.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:03:54 smithi115 conmon[90393]: 2022-05-14T01:03:54.930+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:03:55.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:03:54 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T01:03:55.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:03:54 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T01:03:55.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:03:54 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T01:03:55.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:03:54 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:03:55.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:03:54 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:03:55.109 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T01:03:55.109 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:03:55.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T01:03:55.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:03:55.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:03:55.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:03:55.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T01:03:55.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:03:55.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T01:03:55.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:03:55.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T01:03:55.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:03:55.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T01:03:55.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:03:55.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T01:03:55.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:03:55.113 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:03:55.113 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:03:55.113 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:03:54 smithi115 ceph-mon[78593]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T01:03:55.113 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:03:54 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 5765 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:03:56.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:03:55 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:03:56.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:03:55 smithi047 ceph-mon[77147]: pgmap v3237: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:03:56.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:03:55 smithi115 conmon[90393]: 2022-05-14T01:03:55.881+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:03:56.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:03:55 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:03:56.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:03:55 smithi115 ceph-mon[78593]: pgmap v3237: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:03:57.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:03:56 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:03:57.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:03:56 smithi115 conmon[90393]: 2022-05-14T01:03:56.924+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:03:57.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:03:56 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:03:57.700 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:03:57 smithi047 conmon[78521]: 2022-05-14T01:03:57.296+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:03:57.700 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:03:57 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T01:03:57.700 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:03:57 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:03:57.701 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:03:57 smithi047 conmon[78521]: yield 2022-05-14T01:03:57.701 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:03:57 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:03:57.701 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:03:57 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:03:57.701 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:03:57 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:03:57.702 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:03:57 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T01:03:57.702 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:03:57 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:03:57.702 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:03:57 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T01:03:57.702 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:03:57 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:03:57.702 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:03:57 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T01:03:57.703 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:03:57 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:03:57.703 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:03:57 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T01:03:57.703 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:03:57 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:03:57.703 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:03:57 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T01:03:57.703 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:03:57 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:03:57.704 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:03:57 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:03:57.704 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:03:57 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:03:57.704 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:03:57 smithi047 conmon[78521]: 2022-05-14T01:03:57.297+0000 7f5344781700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T01:03:57.705 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:03:57 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T01:03:57.705 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:03:57 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:03:57.705 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:03:57 smithi047 conmon[78521]: yield 2022-05-14T01:03:57.705 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:03:57 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:03:57.706 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:03:57 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:03:57.706 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:03:57 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:03:57.706 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:03:57 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T01:03:57.706 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:03:57 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:03:57.706 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:03:57 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T01:03:57.707 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:03:57 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:03:57.707 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:03:57 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T01:03:57.707 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:03:57 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:03:57.707 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:03:57 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T01:03:57.707 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:03:57 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:03:57.708 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:03:57 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T01:03:57.708 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:03:57 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:03:57.708 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:03:57 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:03:57.708 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:03:57 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:03:57.708 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:03:57 smithi047 conmon[78521]: 2022-05-14T01:03:57.709 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:03:57 smithi047 conmon[78521]: During handling of the above exception, another exception occurred: 2022-05-14T01:03:57.709 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:03:57 smithi047 conmon[78521]: 2022-05-14T01:03:57.709 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:03:57 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T01:03:57.709 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:03:57 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T01:03:57.710 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:03:57 smithi047 conmon[78521]: return f(*arg) 2022-05-14T01:03:57.710 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:03:57 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T01:03:57.710 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:03:57 smithi047 conmon[78521]: self._write_client_files(client_files, host) 2022-05-14T01:03:57.710 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:03:57 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T01:03:57.710 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:03:57 smithi047 conmon[78521]: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T01:03:57.711 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:03:57 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T01:03:57.711 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:03:57 smithi047 conmon[78521]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T01:03:57.711 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:03:57 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T01:03:57.711 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:03:57 smithi047 conmon[78521]: return self.event_loop.get_result(coro) 2022-05-14T01:03:57.711 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:03:57 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T01:03:57.712 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:03:57 smithi047 conmon[78521]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T01:03:57.712 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:03:57 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T01:03:57.712 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:03:57 smithi047 conmon[78521]: return self.__get_result() 2022-05-14T01:03:57.713 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:03:57 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T01:03:57.713 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:03:57 smithi047 conmon[78521]: raise self._exception 2022-05-14T01:03:57.714 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:03:57 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T01:03:57.714 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:03:57 smithi047 conmon[78521]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T01:03:57.715 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:03:57 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T01:03:57.715 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:03:57 smithi047 conmon[78521]: conn = await self._remote_connection(host, addr) 2022-05-14T01:03:57.715 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:03:57 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T01:03:57.716 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:03:57 smithi047 conmon[78521]: raise 2022-05-14T01:03:57.716 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:03:57 smithi047 conmon[78521]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T01:03:57.717 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:03:57 smithi047 conmon[78521]: self.gen.throw(type, value, traceback) 2022-05-14T01:03:57.717 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:03:57 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T01:03:57.717 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:03:57 smithi047 conmon[78521]: raise OrchestratorError(msg) 2022-05-14T01:03:57.718 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:03:57 smithi047 conmon[78521]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:03:58.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:03:57 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:03:58.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:03:57 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:03:58.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:03:57 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:03:58.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:03:57 smithi047 ceph-mon[77147]: pgmap v3238: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:03:58.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:03:57 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:03:58.042 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T01:03:58.042 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:03:58.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T01:03:58.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:03:58.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:03:58.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:03:58.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T01:03:58.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:03:58.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T01:03:58.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:03:58.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T01:03:58.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:03:58.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T01:03:58.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:03:58.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T01:03:58.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:03:58.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:03:58.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:03:58.046 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:03:57 smithi047 ceph-mon[77147]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T01:03:58.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T01:03:58.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:03:58.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T01:03:58.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:03:58.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:03:58.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:03:58.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T01:03:58.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:03:58.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T01:03:58.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:03:58.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T01:03:58.049 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:03:58.049 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T01:03:58.049 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:03:58.049 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T01:03:58.049 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:03:58.050 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:03:58.050 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:03:58.050 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T01:03:58.050 INFO:journalctl@ceph.mon.b.smithi047.stdout: During handling of the above exception, another exception occurred: 2022-05-14T01:03:58.050 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T01:03:58.051 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T01:03:58.051 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T01:03:58.051 INFO:journalctl@ceph.mon.b.smithi047.stdout: return f(*arg) 2022-05-14T01:03:58.051 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T01:03:58.051 INFO:journalctl@ceph.mon.b.smithi047.stdout: self._write_client_files(client_files, host) 2022-05-14T01:03:58.052 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T01:03:58.052 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T01:03:58.052 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T01:03:58.052 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T01:03:58.053 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T01:03:58.053 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.event_loop.get_result(coro) 2022-05-14T01:03:58.053 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T01:03:58.053 INFO:journalctl@ceph.mon.b.smithi047.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T01:03:58.053 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T01:03:58.054 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.__get_result() 2022-05-14T01:03:58.054 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T01:03:58.054 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise self._exception 2022-05-14T01:03:58.054 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T01:03:58.054 INFO:journalctl@ceph.mon.b.smithi047.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T01:03:58.055 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T01:03:58.055 INFO:journalctl@ceph.mon.b.smithi047.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T01:03:58.055 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T01:03:58.055 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise 2022-05-14T01:03:58.055 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T01:03:58.056 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.gen.throw(type, value, traceback) 2022-05-14T01:03:58.056 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T01:03:58.056 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OrchestratorError(msg) 2022-05-14T01:03:58.056 INFO:journalctl@ceph.mon.b.smithi047.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:03:58.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:03:57 smithi115 conmon[90393]: 2022-05-14T01:03:57.948+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:03:58.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:03:57 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:03:58.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:03:57 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:03:58.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:03:57 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:03:58.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:03:57 smithi115 ceph-mon[78593]: pgmap v3238: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:03:58.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:03:57 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:03:58.109 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T01:03:58.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:03:58.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T01:03:58.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:03:58.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:03:58.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:03:58.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T01:03:58.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:03:58.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T01:03:58.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:03:58.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T01:03:58.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:03:58.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T01:03:58.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:03:58.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T01:03:58.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:03:58.113 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:03:58.113 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:03:58.113 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:03:57 smithi115 ceph-mon[78593]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T01:03:58.113 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T01:03:58.113 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:03:58.114 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T01:03:58.114 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:03:58.114 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:03:58.114 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:03:58.115 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T01:03:58.115 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:03:58.115 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T01:03:58.115 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:03:58.115 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T01:03:58.116 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:03:58.116 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T01:03:58.116 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:03:58.116 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T01:03:58.116 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:03:58.117 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:03:58.117 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:03:58.117 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T01:03:58.117 INFO:journalctl@ceph.mon.c.smithi115.stdout: During handling of the above exception, another exception occurred: 2022-05-14T01:03:58.117 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T01:03:58.118 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T01:03:58.118 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T01:03:58.118 INFO:journalctl@ceph.mon.c.smithi115.stdout: return f(*arg) 2022-05-14T01:03:58.118 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T01:03:58.118 INFO:journalctl@ceph.mon.c.smithi115.stdout: self._write_client_files(client_files, host) 2022-05-14T01:03:58.119 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T01:03:58.119 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T01:03:58.119 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T01:03:58.119 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T01:03:58.119 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T01:03:58.120 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.event_loop.get_result(coro) 2022-05-14T01:03:58.120 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T01:03:58.120 INFO:journalctl@ceph.mon.c.smithi115.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T01:03:58.120 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T01:03:58.120 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.__get_result() 2022-05-14T01:03:58.121 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T01:03:58.121 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise self._exception 2022-05-14T01:03:58.121 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T01:03:58.121 INFO:journalctl@ceph.mon.c.smithi115.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T01:03:58.122 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T01:03:58.122 INFO:journalctl@ceph.mon.c.smithi115.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T01:03:58.122 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T01:03:58.122 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise 2022-05-14T01:03:58.122 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T01:03:58.123 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.gen.throw(type, value, traceback) 2022-05-14T01:03:58.123 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T01:03:58.123 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OrchestratorError(msg) 2022-05-14T01:03:58.123 INFO:journalctl@ceph.mon.c.smithi115.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:03:59.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:03:58 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:03:59.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:03:58 smithi115 conmon[90393]: 2022-05-14T01:03:58.943+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:03:59.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:03:58 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:04:00.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:03:59 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:04:00.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:03:59 smithi047 ceph-mon[77147]: pgmap v3239: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:04:00.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:03:59 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 5770 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:04:00.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:03:59 smithi115 conmon[90393]: 2022-05-14T01:03:59.925+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:04:00.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:03:59 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:04:00.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:03:59 smithi115 ceph-mon[78593]: pgmap v3239: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:04:00.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:03:59 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 5770 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:04:01.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:04:00 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:04:01.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:04:00 smithi115 conmon[90393]: 2022-05-14T01:04:00.884+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:04:01.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:04:00 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:04:02.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:04:01 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:04:02.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:04:01 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:04:02.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:04:01 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:04:02.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:04:01 smithi047 ceph-mon[77147]: pgmap v3240: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:04:02.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:04:01 smithi115 conmon[90393]: 2022-05-14T01:04:01.872+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:04:02.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:04:01 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:04:02.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:04:01 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:04:02.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:04:01 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:04:02.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:04:01 smithi115 ceph-mon[78593]: pgmap v3240: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:04:03.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:04:02 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:04:03.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:04:02 smithi115 conmon[90393]: 2022-05-14T01:04:02.825+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:04:03.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:04:02 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:04:04.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:04:03 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:04:04.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:04:03 smithi047 ceph-mon[77147]: pgmap v3241: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:04:04.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:04:03 smithi115 conmon[90393]: 2022-05-14T01:04:03.830+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:04:04.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:04:03 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:04:04.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:04:03 smithi115 ceph-mon[78593]: pgmap v3241: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:04:05.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:04:04 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:04:05.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:04:04 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 5775 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:04:05.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:04:04 smithi115 conmon[90393]: 2022-05-14T01:04:04.868+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:04:05.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:04:04 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:04:05.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:04:04 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 5775 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:04:06.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:04:05 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:04:06.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:04:05 smithi047 ceph-mon[77147]: pgmap v3242: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:04:06.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:04:05 smithi115 conmon[90393]: 2022-05-14T01:04:05.843+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:04:06.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:04:05 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:04:06.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:04:05 smithi115 ceph-mon[78593]: pgmap v3242: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:04:07.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:04:06 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:04:07.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:04:06 smithi115 conmon[90393]: 2022-05-14T01:04:06.838+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:04:07.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:04:06 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:04:08.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:04:07 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:04:08.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:04:07 smithi047 ceph-mon[77147]: pgmap v3243: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:04:08.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:04:07 smithi115 conmon[90393]: 2022-05-14T01:04:07.861+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:04:08.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:04:07 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:04:08.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:04:07 smithi115 ceph-mon[78593]: pgmap v3243: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:04:09.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:04:08 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:04:09.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:04:08 smithi115 conmon[90393]: 2022-05-14T01:04:08.867+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:04:09.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:04:08 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:04:10.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:04:09 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:04:10.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:04:09 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:04:10.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:04:09 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:04:10.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:04:09 smithi047 ceph-mon[77147]: pgmap v3244: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:04:10.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:04:09 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 5780 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:04:10.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:04:09 smithi115 conmon[90393]: 2022-05-14T01:04:09.883+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:04:10.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:04:09 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:04:10.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:04:09 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:04:10.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:04:09 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:04:10.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:04:09 smithi115 ceph-mon[78593]: pgmap v3244: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:04:10.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:04:09 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 5780 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:04:11.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:04:10 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:04:11.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:04:10 smithi115 conmon[90393]: 2022-05-14T01:04:10.929+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:04:11.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:04:10 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:04:12.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:04:11 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:04:12.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:04:11 smithi047 ceph-mon[77147]: pgmap v3245: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:04:12.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:04:11 smithi115 conmon[90393]: 2022-05-14T01:04:11.881+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:04:12.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:04:11 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:04:12.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:04:11 smithi115 ceph-mon[78593]: pgmap v3245: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:04:13.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:04:12 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:04:13.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:04:12 smithi115 conmon[90393]: 2022-05-14T01:04:12.849+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:04:13.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:04:12 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:04:14.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:04:13 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:04:14.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:04:13 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:04:14.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:04:13 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:04:14.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:04:13 smithi047 ceph-mon[77147]: pgmap v3246: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:04:14.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:04:13 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T01:04:14.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:04:13 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T01:04:14.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:04:13 smithi115 conmon[90393]: 2022-05-14T01:04:13.840+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:04:14.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:04:13 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:04:14.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:04:13 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:04:14.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:04:13 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:04:14.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:04:13 smithi115 ceph-mon[78593]: pgmap v3246: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:04:14.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:04:13 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T01:04:14.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:04:13 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T01:04:15.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:04:14 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:04:15.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:04:14 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 5785 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:04:15.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:04:14 smithi115 conmon[90393]: 2022-05-14T01:04:14.834+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:04:15.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:04:14 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:04:15.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:04:14 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 5785 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:04:16.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:04:15 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:04:16.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:04:15 smithi047 ceph-mon[77147]: pgmap v3247: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:04:16.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:04:15 smithi115 conmon[90393]: 2022-05-14T01:04:15.812+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:04:16.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:04:15 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:04:16.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:04:15 smithi115 ceph-mon[78593]: pgmap v3247: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:04:17.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:04:16 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:04:17.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:04:16 smithi115 conmon[90393]: 2022-05-14T01:04:16.799+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:04:17.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:04:16 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:04:18.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:04:17 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:04:18.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:04:17 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:04:18.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:04:17 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:04:18.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:04:17 smithi047 ceph-mon[77147]: pgmap v3248: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:04:18.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:04:17 smithi115 conmon[90393]: 2022-05-14T01:04:17.779+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:04:18.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:04:17 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:04:18.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:04:17 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:04:18.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:04:17 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:04:18.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:04:17 smithi115 ceph-mon[78593]: pgmap v3248: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:04:19.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:04:18 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:04:19.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:04:18 smithi115 conmon[90393]: 2022-05-14T01:04:18.743+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:04:19.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:04:18 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:04:20.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:04:19 smithi115 conmon[90393]: 2022-05-14T01:04:19.729+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:04:20.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:04:19 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:04:20.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:04:19 smithi115 ceph-mon[78593]: pgmap v3249: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:04:20.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:04:19 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 5790 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:04:20.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:04:19 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:04:20.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:04:19 smithi047 ceph-mon[77147]: pgmap v3249: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:04:20.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:04:19 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 5790 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:04:21.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:04:20 smithi115 conmon[90393]: 2022-05-14T01:04:20.702+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:04:21.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:04:20 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:04:21.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:04:20 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:04:22.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:04:21 smithi115 conmon[90393]: 2022-05-14T01:04:21.710+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:04:22.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:04:21 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:04:22.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:04:21 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:04:22.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:04:21 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:04:22.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:04:21 smithi115 ceph-mon[78593]: pgmap v3250: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:04:22.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:04:21 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:04:22.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:04:21 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:04:22.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:04:21 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:04:22.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:04:21 smithi047 ceph-mon[77147]: pgmap v3250: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:04:23.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:04:22 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:04:23.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:04:22 smithi115 conmon[90393]: 2022-05-14T01:04:22.705+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:04:23.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:04:22 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:04:24.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:04:23 smithi115 conmon[90393]: 2022-05-14T01:04:23.744+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:04:24.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:04:23 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:04:24.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:04:23 smithi115 ceph-mon[78593]: pgmap v3251: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:04:24.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:04:23 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:04:24.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:04:23 smithi047 ceph-mon[77147]: pgmap v3251: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:04:25.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:04:24 smithi115 conmon[90393]: 2022-05-14T01:04:24.727+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:04:25.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:04:24 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:04:25.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:04:24 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 5795 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:04:25.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:04:24 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:04:25.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:04:24 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 5795 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:04:26.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:04:25 smithi115 conmon[90393]: 2022-05-14T01:04:25.756+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:04:26.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:04:25 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:04:26.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:04:25 smithi115 ceph-mon[78593]: pgmap v3252: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:04:26.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:04:25 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:04:26.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:04:25 smithi047 ceph-mon[77147]: pgmap v3252: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:04:27.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:04:26 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:04:27.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:04:26 smithi115 conmon[90393]: 2022-05-14T01:04:26.759+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:04:27.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:04:26 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:04:28.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:04:27 smithi115 conmon[90393]: 2022-05-14T01:04:27.765+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:04:28.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:04:27 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:04:28.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:04:27 smithi115 ceph-mon[78593]: pgmap v3253: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:04:28.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:04:27 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:04:28.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:04:27 smithi047 ceph-mon[77147]: pgmap v3253: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:04:29.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:04:28 smithi115 conmon[90393]: 2022-05-14T01:04:28.806+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:04:29.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:04:28 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:04:29.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:04:28 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:04:30.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:04:29 smithi115 conmon[90393]: 2022-05-14T01:04:29.849+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:04:30.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:04:29 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:04:30.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:04:29 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:04:30.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:04:29 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:04:30.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:04:29 smithi115 ceph-mon[78593]: pgmap v3254: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:04:30.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:04:29 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 5800 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:04:30.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:04:29 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:04:30.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:04:29 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:04:30.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:04:29 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:04:30.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:04:29 smithi047 ceph-mon[77147]: pgmap v3254: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:04:30.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:04:29 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 5800 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:04:31.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:04:30 smithi115 conmon[90393]: 2022-05-14T01:04:30.806+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:04:31.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:04:30 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:04:31.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:04:30 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:04:32.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:04:31 smithi115 conmon[90393]: 2022-05-14T01:04:31.775+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:04:32.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:04:31 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:04:32.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:04:31 smithi115 ceph-mon[78593]: pgmap v3255: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:04:32.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:04:31 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:04:32.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:04:31 smithi047 ceph-mon[77147]: pgmap v3255: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:04:33.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:04:32 smithi115 conmon[90393]: 2022-05-14T01:04:32.825+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:04:33.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:04:32 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:04:33.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:04:32 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:04:34.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:04:33 smithi115 conmon[90393]: 2022-05-14T01:04:33.859+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:04:34.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:04:33 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:04:34.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:04:33 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:04:34.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:04:33 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:04:34.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:04:33 smithi115 ceph-mon[78593]: pgmap v3256: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:04:34.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:04:33 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:04:34.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:04:33 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:04:34.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:04:33 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:04:34.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:04:33 smithi047 ceph-mon[77147]: pgmap v3256: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:04:35.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:04:34 smithi115 conmon[90393]: 2022-05-14T01:04:34.906+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:04:35.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:04:34 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:04:35.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:04:34 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 5805 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:04:35.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:04:34 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:04:35.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:04:34 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 5805 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:04:36.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:04:35 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:04:36.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:04:35 smithi047 ceph-mon[77147]: pgmap v3257: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:04:36.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:04:35 smithi115 conmon[90393]: 2022-05-14T01:04:35.874+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:04:36.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:04:35 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:04:36.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:04:35 smithi115 ceph-mon[78593]: pgmap v3257: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:04:37.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:04:36 smithi115 conmon[90393]: 2022-05-14T01:04:36.845+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:04:37.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:04:36 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:04:37.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:04:36 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:04:38.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:04:37 smithi115 conmon[90393]: 2022-05-14T01:04:37.828+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:04:38.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:04:37 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:04:38.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:04:37 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:04:38.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:04:37 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:04:38.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:04:37 smithi115 ceph-mon[78593]: pgmap v3258: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:04:38.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:04:37 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:04:38.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:04:37 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:04:38.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:04:37 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:04:38.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:04:37 smithi047 ceph-mon[77147]: pgmap v3258: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:04:39.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:04:38 smithi115 conmon[90393]: 2022-05-14T01:04:38.811+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:04:39.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:04:38 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:04:39.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:04:38 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:04:40.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:04:39 smithi115 conmon[90393]: 2022-05-14T01:04:39.852+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:04:40.121 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:04:39 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:04:40.121 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:04:39 smithi115 ceph-mon[78593]: pgmap v3259: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:04:40.121 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:04:39 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 5810 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:04:40.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:04:39 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:04:40.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:04:39 smithi047 ceph-mon[77147]: pgmap v3259: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:04:40.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:04:39 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 5810 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:04:41.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:04:40 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:04:41.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:04:40 smithi115 conmon[90393]: 2022-05-14T01:04:40.868+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:04:41.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:04:40 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:04:42.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:04:41 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:04:42.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:04:41 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:04:42.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:04:41 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:04:42.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:04:41 smithi047 ceph-mon[77147]: pgmap v3260: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:04:42.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:04:41 smithi115 conmon[90393]: 2022-05-14T01:04:41.866+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:04:42.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:04:41 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:04:42.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:04:41 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:04:42.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:04:41 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:04:42.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:04:41 smithi115 ceph-mon[78593]: pgmap v3260: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:04:43.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:04:42 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:04:43.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:04:42 smithi115 conmon[90393]: 2022-05-14T01:04:42.875+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:04:43.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:04:42 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:04:44.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:04:43 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:04:44.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:04:43 smithi047 ceph-mon[77147]: pgmap v3261: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:04:44.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:04:43 smithi115 conmon[90393]: 2022-05-14T01:04:43.915+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:04:44.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:04:43 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:04:44.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:04:43 smithi115 ceph-mon[78593]: pgmap v3261: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:04:45.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:04:44 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:04:45.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:04:44 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 5815 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:04:45.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:04:44 smithi115 conmon[90393]: 2022-05-14T01:04:44.946+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:04:45.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:04:44 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:04:45.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:04:44 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 5815 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:04:46.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:04:45 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:04:46.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:04:45 smithi047 ceph-mon[77147]: pgmap v3262: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:04:46.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:04:45 smithi115 conmon[90393]: 2022-05-14T01:04:45.915+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:04:46.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:04:45 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:04:46.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:04:45 smithi115 ceph-mon[78593]: pgmap v3262: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:04:47.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:04:46 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:04:47.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:04:46 smithi115 conmon[90393]: 2022-05-14T01:04:46.914+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:04:47.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:04:46 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:04:48.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:04:47 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:04:48.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:04:47 smithi047 ceph-mon[77147]: pgmap v3263: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:04:48.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:04:47 smithi115 conmon[90393]: 2022-05-14T01:04:47.894+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:04:48.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:04:47 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:04:48.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:04:47 smithi115 ceph-mon[78593]: pgmap v3263: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:04:49.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:04:48 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:04:49.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:04:48 smithi115 conmon[90393]: 2022-05-14T01:04:48.911+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:04:49.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:04:48 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:04:50.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:04:49 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:04:50.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:04:49 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:04:50.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:04:49 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:04:50.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:04:49 smithi047 ceph-mon[77147]: pgmap v3264: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:04:50.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:04:49 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 5820 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:04:50.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:04:49 smithi115 conmon[90393]: 2022-05-14T01:04:49.875+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:04:50.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:04:49 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:04:50.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:04:49 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:04:50.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:04:49 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:04:50.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:04:49 smithi115 ceph-mon[78593]: pgmap v3264: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:04:50.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:04:49 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 5820 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:04:51.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:04:50 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:04:51.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:04:50 smithi115 conmon[90393]: 2022-05-14T01:04:50.878+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:04:51.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:04:50 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:04:52.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:04:51 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:04:52.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:04:51 smithi047 ceph-mon[77147]: pgmap v3265: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:04:52.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:04:51 smithi115 conmon[90393]: 2022-05-14T01:04:51.895+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:04:52.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:04:51 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:04:52.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:04:51 smithi115 ceph-mon[78593]: pgmap v3265: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:04:53.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:04:52 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:04:53.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:04:52 smithi115 conmon[90393]: 2022-05-14T01:04:52.928+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:04:53.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:04:52 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:04:54.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:04:53 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:04:54.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:04:53 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:04:54.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:04:53 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:04:54.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:04:53 smithi047 ceph-mon[77147]: pgmap v3266: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:04:54.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:04:53 smithi115 conmon[90393]: 2022-05-14T01:04:53.936+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:04:54.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:04:53 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:04:54.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:04:53 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:04:54.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:04:53 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:04:54.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:04:53 smithi115 ceph-mon[78593]: pgmap v3266: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:04:55.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:04:54 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:04:55.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:04:54 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 5825 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:04:55.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:04:54 smithi115 conmon[90393]: 2022-05-14T01:04:54.981+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:04:55.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:04:54 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:04:55.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:04:54 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 5825 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:04:56.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:04:55 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:04:56.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:04:55 smithi047 ceph-mon[77147]: pgmap v3267: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:04:56.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:04:55 smithi115 conmon[90393]: 2022-05-14T01:04:55.947+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:04:56.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:04:55 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:04:56.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:04:55 smithi115 ceph-mon[78593]: pgmap v3267: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:04:57.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:04:56 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:04:57.342 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:04:56 smithi115 conmon[90393]: 2022-05-14T01:04:56.994+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:04:57.343 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:04:56 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:04:58.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:04:57 smithi115 conmon[90393]: 2022-05-14T01:04:57.967+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:04:58.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:04:57 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:04:58.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:04:57 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:04:58.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:04:57 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:04:58.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:04:57 smithi115 ceph-mon[78593]: pgmap v3268: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:04:58.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:04:57 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T01:04:58.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:04:57 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T01:04:58.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:04:57 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T01:04:58.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:04:57 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:04:58.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:04:57 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:04:58.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:04:57 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:04:58.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:04:57 smithi047 ceph-mon[77147]: pgmap v3268: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:04:58.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:04:57 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T01:04:58.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:04:57 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T01:04:58.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:04:57 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T01:04:59.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:04:58 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:04:59.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:04:58 smithi115 conmon[90393]: 2022-05-14T01:04:58.999+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:04:59.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:04:58 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:05:00.296 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:04:59 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:05:00.297 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:04:59 smithi047 ceph-mon[77147]: pgmap v3269: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:05:00.297 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:04:59 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 5830 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:05:00.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:05:00 smithi115 conmon[90393]: 2022-05-14T01:05:00.010+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:05:00.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:04:59 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:05:00.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:04:59 smithi115 ceph-mon[78593]: pgmap v3269: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:05:00.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:04:59 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 5830 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:05:00.791 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:05:00 smithi047 conmon[78521]: 2022-05-14T01:05:00.400+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:05:00.791 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:05:00 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T01:05:00.792 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:05:00 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:05:00.792 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:05:00 smithi047 conmon[78521]: yield 2022-05-14T01:05:00.792 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:05:00 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:05:00.792 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:05:00 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:05:00.792 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:05:00 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:05:00.793 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:05:00 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T01:05:00.793 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:05:00 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:05:00.793 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:05:00 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T01:05:00.793 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:05:00 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:05:00.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:05:00 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T01:05:00.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:05:00 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:05:00.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:05:00 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T01:05:00.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:05:00 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:05:00.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:05:00 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T01:05:00.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:05:00 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:05:00.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:05:00 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:05:00.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:05:00 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:05:01.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:05:00 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:05:01.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:05:00 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:05:01.292 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T01:05:01.292 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:05:01.292 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T01:05:01.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:05:01.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:05:01.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:05:01.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T01:05:01.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:05:01.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T01:05:01.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:05:01.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T01:05:01.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:05:01.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T01:05:01.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:05:01.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T01:05:01.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:05:01.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:05:01.298 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:05:01.298 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:05:00 smithi047 ceph-mon[77147]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T01:05:01.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:05:00 smithi115 conmon[90393]: 2022-05-14T01:05:00.967+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:05:01.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:05:00 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:05:01.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:05:00 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:05:01.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T01:05:01.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:05:01.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T01:05:01.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:05:01.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:05:01.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:05:01.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T01:05:01.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:05:01.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T01:05:01.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:05:01.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T01:05:01.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:05:01.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T01:05:01.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:05:01.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T01:05:01.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:05:01.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:05:01.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:05:01.364 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:05:00 smithi115 ceph-mon[78593]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T01:05:02.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:05:01 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:05:02.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:05:01 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:05:02.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:05:01 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:05:02.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:05:01 smithi047 ceph-mon[77147]: pgmap v3270: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:05:02.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:05:01 smithi115 conmon[90393]: 2022-05-14T01:05:01.920+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:05:02.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:05:01 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:05:02.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:05:01 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:05:02.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:05:01 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:05:02.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:05:01 smithi115 ceph-mon[78593]: pgmap v3270: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:05:03.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:05:02 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:05:03.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:05:02 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:05:03.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:05:02 smithi115 conmon[90393]: 2022-05-14T01:05:02.903+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:05:03.791 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:05:03 smithi047 conmon[78521]: 2022-05-14T01:05:03.472+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:05:03.792 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:05:03 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T01:05:03.792 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:05:03 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:05:03.792 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:05:03 smithi047 conmon[78521]: yield 2022-05-14T01:05:03.793 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:05:03 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:05:03.793 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:05:03 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:05:03.793 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:05:03 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:05:03.793 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:05:03 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T01:05:03.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:05:03 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:05:03.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:05:03 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T01:05:03.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:05:03 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:05:03.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:05:03 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T01:05:03.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:05:03 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:05:03.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:05:03 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T01:05:03.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:05:03 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:05:03.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:05:03 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T01:05:03.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:05:03 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:05:03.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:05:03 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:05:03.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:05:03 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:05:03.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:05:03 smithi047 conmon[78521]: 2022-05-14T01:05:03.472+0000 7f5345f84700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T01:05:03.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:05:03 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T01:05:03.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:05:03 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:05:03.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:05:03 smithi047 conmon[78521]: yield 2022-05-14T01:05:03.797 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:05:03 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:05:03.797 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:05:03 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:05:03.797 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:05:03 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:05:03.797 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:05:03 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T01:05:03.798 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:05:03 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:05:03.798 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:05:03 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T01:05:03.798 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:05:03 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:05:03.798 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:05:03 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T01:05:03.798 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:05:03 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:05:03.799 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:05:03 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T01:05:03.799 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:05:03 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:05:03.799 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:05:03 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T01:05:03.799 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:05:03 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:05:03.799 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:05:03 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:05:03.800 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:05:03 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:05:03.800 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:05:03 smithi047 conmon[78521]: 2022-05-14T01:05:03.800 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:05:03 smithi047 conmon[78521]: During handling of the above exception, another exception occurred: 2022-05-14T01:05:03.800 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:05:03 smithi047 conmon[78521]: 2022-05-14T01:05:03.801 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:05:03 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T01:05:03.801 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:05:03 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T01:05:03.801 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:05:03 smithi047 conmon[78521]: return f(*arg) 2022-05-14T01:05:03.802 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:05:03 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T01:05:03.802 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:05:03 smithi047 conmon[78521]: self._write_client_files(client_files, host) 2022-05-14T01:05:03.802 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:05:03 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T01:05:03.802 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:05:03 smithi047 conmon[78521]: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T01:05:03.802 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:05:03 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T01:05:03.803 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:05:03 smithi047 conmon[78521]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T01:05:03.803 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:05:03 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T01:05:03.803 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:05:03 smithi047 conmon[78521]: return self.event_loop.get_result(coro) 2022-05-14T01:05:03.804 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:05:03 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T01:05:03.804 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:05:03 smithi047 conmon[78521]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T01:05:03.804 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:05:03 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T01:05:03.804 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:05:03 smithi047 conmon[78521]: return self.__get_result() 2022-05-14T01:05:03.804 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:05:03 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T01:05:03.805 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:05:03 smithi047 conmon[78521]: raise self._exception 2022-05-14T01:05:03.805 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:05:03 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T01:05:03.805 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:05:03 smithi047 conmon[78521]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T01:05:03.805 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:05:03 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T01:05:03.806 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:05:03 smithi047 conmon[78521]: conn = await self._remote_connection(host, addr) 2022-05-14T01:05:03.806 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:05:03 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T01:05:03.806 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:05:03 smithi047 conmon[78521]: raise 2022-05-14T01:05:03.806 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:05:03 smithi047 conmon[78521]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T01:05:03.806 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:05:03 smithi047 conmon[78521]: self.gen.throw(type, value, traceback) 2022-05-14T01:05:03.807 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:05:03 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T01:05:03.807 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:05:03 smithi047 conmon[78521]: raise OrchestratorError(msg) 2022-05-14T01:05:03.807 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:05:03 smithi047 conmon[78521]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:05:04.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:05:03 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:05:04.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:05:03 smithi047 ceph-mon[77147]: pgmap v3271: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:05:04.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:05:03 smithi115 conmon[90393]: 2022-05-14T01:05:03.907+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:05:04.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:05:03 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:05:04.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:05:03 smithi115 ceph-mon[78593]: pgmap v3271: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:05:05.298 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:05:04 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:05:05.299 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T01:05:05.299 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:05:05.299 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T01:05:05.299 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:05:05.300 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:05:05.300 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:05:05.300 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T01:05:05.300 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:05:05.300 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T01:05:05.301 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:05:05.301 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T01:05:05.301 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:05:05.301 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T01:05:05.302 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:05:05.302 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T01:05:05.302 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:05:05.302 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:05:05.303 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:05:05.303 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:05:04 smithi047 ceph-mon[77147]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T01:05:05.303 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T01:05:05.303 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:05:05.304 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T01:05:05.304 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:05:05.304 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:05:05.304 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:05:05.304 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T01:05:05.305 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:05:05.305 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T01:05:05.305 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:05:05.305 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T01:05:05.306 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:05:05.306 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T01:05:05.306 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:05:05.306 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T01:05:05.307 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:05:05.307 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:05:05.307 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:05:05.307 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T01:05:05.308 INFO:journalctl@ceph.mon.b.smithi047.stdout: During handling of the above exception, another exception occurred: 2022-05-14T01:05:05.308 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T01:05:05.308 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T01:05:05.308 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T01:05:05.309 INFO:journalctl@ceph.mon.b.smithi047.stdout: return f(*arg) 2022-05-14T01:05:05.309 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T01:05:05.309 INFO:journalctl@ceph.mon.b.smithi047.stdout: self._write_client_files(client_files, host) 2022-05-14T01:05:05.309 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T01:05:05.310 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T01:05:05.310 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T01:05:05.310 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T01:05:05.310 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T01:05:05.311 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.event_loop.get_result(coro) 2022-05-14T01:05:05.311 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T01:05:05.311 INFO:journalctl@ceph.mon.b.smithi047.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T01:05:05.312 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T01:05:05.312 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.__get_result() 2022-05-14T01:05:05.312 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T01:05:05.312 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise self._exception 2022-05-14T01:05:05.312 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T01:05:05.313 INFO:journalctl@ceph.mon.b.smithi047.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T01:05:05.313 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T01:05:05.314 INFO:journalctl@ceph.mon.b.smithi047.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T01:05:05.314 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T01:05:05.314 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise 2022-05-14T01:05:05.315 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T01:05:05.315 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.gen.throw(type, value, traceback) 2022-05-14T01:05:05.316 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T01:05:05.316 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OrchestratorError(msg) 2022-05-14T01:05:05.317 INFO:journalctl@ceph.mon.b.smithi047.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:05:05.317 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:05:04 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:05:05.318 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:05:04 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 5835 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:05:05.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:05:04 smithi115 conmon[90393]: 2022-05-14T01:05:04.925+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:05:05.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:05:04 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:05:05.358 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T01:05:05.359 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:05:05.359 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T01:05:05.359 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:05:05.359 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:05:05.359 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:05:05.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T01:05:05.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:05:05.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T01:05:05.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:05:05.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T01:05:05.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:05:05.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T01:05:05.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:05:05.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T01:05:05.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:05:05.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:05:05.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:05:05.363 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:05:04 smithi115 ceph-mon[78593]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T01:05:05.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T01:05:05.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:05:05.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T01:05:05.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:05:05.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:05:05.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:05:05.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T01:05:05.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:05:05.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T01:05:05.365 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:05:05.365 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T01:05:05.365 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:05:05.365 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T01:05:05.366 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:05:05.366 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T01:05:05.366 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:05:05.366 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:05:05.366 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:05:05.367 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T01:05:05.367 INFO:journalctl@ceph.mon.c.smithi115.stdout: During handling of the above exception, another exception occurred: 2022-05-14T01:05:05.367 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T01:05:05.367 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T01:05:05.367 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T01:05:05.368 INFO:journalctl@ceph.mon.c.smithi115.stdout: return f(*arg) 2022-05-14T01:05:05.368 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T01:05:05.368 INFO:journalctl@ceph.mon.c.smithi115.stdout: self._write_client_files(client_files, host) 2022-05-14T01:05:05.368 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T01:05:05.368 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T01:05:05.369 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T01:05:05.369 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T01:05:05.369 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T01:05:05.369 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.event_loop.get_result(coro) 2022-05-14T01:05:05.370 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T01:05:05.370 INFO:journalctl@ceph.mon.c.smithi115.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T01:05:05.370 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T01:05:05.370 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.__get_result() 2022-05-14T01:05:05.370 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T01:05:05.371 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise self._exception 2022-05-14T01:05:05.371 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T01:05:05.371 INFO:journalctl@ceph.mon.c.smithi115.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T01:05:05.371 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T01:05:05.371 INFO:journalctl@ceph.mon.c.smithi115.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T01:05:05.372 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T01:05:05.372 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise 2022-05-14T01:05:05.372 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T01:05:05.372 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.gen.throw(type, value, traceback) 2022-05-14T01:05:05.373 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T01:05:05.373 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OrchestratorError(msg) 2022-05-14T01:05:05.373 INFO:journalctl@ceph.mon.c.smithi115.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:05:05.373 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:05:04 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:05:05.373 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:05:04 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 5835 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:05:06.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:05:05 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:05:06.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:05:05 smithi047 ceph-mon[77147]: pgmap v3272: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:05:06.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:05:05 smithi115 conmon[90393]: 2022-05-14T01:05:05.930+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:05:06.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:05:05 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:05:06.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:05:05 smithi115 ceph-mon[78593]: pgmap v3272: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:05:07.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:05:06 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:05:07.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:05:06 smithi115 conmon[90393]: 2022-05-14T01:05:06.949+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:05:07.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:05:06 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:05:08.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:05:07 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:05:08.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:05:07 smithi047 ceph-mon[77147]: pgmap v3273: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:05:08.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:05:07 smithi115 conmon[90393]: 2022-05-14T01:05:07.986+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:05:08.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:05:07 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:05:08.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:05:07 smithi115 ceph-mon[78593]: pgmap v3273: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:05:09.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:05:08 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:05:09.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:05:08 smithi115 conmon[90393]: 2022-05-14T01:05:08.943+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:05:09.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:05:08 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:05:10.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:05:09 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:05:10.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:05:09 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:05:10.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:05:09 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:05:10.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:05:09 smithi047 ceph-mon[77147]: pgmap v3274: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:05:10.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:05:09 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 5840 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:05:10.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:05:09 smithi115 conmon[90393]: 2022-05-14T01:05:09.910+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:05:10.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:05:09 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:05:10.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:05:09 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:05:10.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:05:09 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:05:10.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:05:09 smithi115 ceph-mon[78593]: pgmap v3274: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:05:10.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:05:09 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 5840 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:05:11.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:05:11 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:05:11.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:05:10 smithi115 conmon[90393]: 2022-05-14T01:05:10.876+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:05:11.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:05:11 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:05:12.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:05:11 smithi115 conmon[90393]: 2022-05-14T01:05:11.846+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:05:12.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:05:12 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:05:12.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:05:12 smithi115 ceph-mon[78593]: pgmap v3275: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:05:12.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:05:12 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:05:12.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:05:12 smithi047 ceph-mon[77147]: pgmap v3275: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:05:13.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:05:12 smithi115 conmon[90393]: 2022-05-14T01:05:12.805+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:05:13.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:05:13 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:05:13.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:05:13 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:05:14.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:05:13 smithi115 conmon[90393]: 2022-05-14T01:05:13.796+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:05:14.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:05:14 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:05:14.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:05:14 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:05:14.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:05:14 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:05:14.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:05:14 smithi115 ceph-mon[78593]: pgmap v3276: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:05:14.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:05:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T01:05:14.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:05:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T01:05:14.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:05:14 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:05:14.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:05:14 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:05:14.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:05:14 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:05:14.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:05:14 smithi047 ceph-mon[77147]: pgmap v3276: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:05:14.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:05:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T01:05:14.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:05:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T01:05:15.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:05:14 smithi115 conmon[90393]: 2022-05-14T01:05:14.782+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:05:15.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:05:15 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:05:15.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:05:15 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 5845 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:05:15.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:05:15 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:05:15.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:05:15 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 5845 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:05:16.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:05:15 smithi115 conmon[90393]: 2022-05-14T01:05:15.805+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:05:16.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:05:16 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:05:16.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:05:16 smithi115 ceph-mon[78593]: pgmap v3277: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:05:16.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:05:16 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:05:16.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:05:16 smithi047 ceph-mon[77147]: pgmap v3277: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:05:17.032 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:05:16 smithi115 conmon[90393]: 2022-05-14T01:05:16.773+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:05:17.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:05:17 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:05:17.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:05:17 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:05:18.033 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:05:17 smithi115 conmon[90393]: 2022-05-14T01:05:17.724+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:05:18.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:05:18 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:05:18.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:05:18 smithi047 ceph-mon[77147]: pgmap v3278: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:05:18.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:05:18 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:05:18.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:05:18 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:05:18.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:05:18 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:05:18.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:05:18 smithi115 ceph-mon[78593]: pgmap v3278: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:05:18.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:05:18 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:05:18.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:05:18 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:05:19.038 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:05:18 smithi115 conmon[90393]: 2022-05-14T01:05:18.690+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:05:19.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:05:19 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:05:19.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:05:19 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:05:20.040 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:05:19 smithi115 conmon[90393]: 2022-05-14T01:05:19.738+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:05:20.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:05:20 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:05:20.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:05:20 smithi047 ceph-mon[77147]: pgmap v3279: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:05:20.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:05:20 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 5850 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:05:20.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:05:20 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:05:20.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:05:20 smithi115 ceph-mon[78593]: pgmap v3279: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:05:20.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:05:20 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 5850 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:05:21.046 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:05:20 smithi115 conmon[90393]: 2022-05-14T01:05:20.722+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:05:21.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:05:21 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:05:21.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:05:21 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:05:22.047 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:05:21 smithi115 conmon[90393]: 2022-05-14T01:05:21.700+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:05:22.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:05:22 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:05:22.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:05:22 smithi115 ceph-mon[78593]: pgmap v3280: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:05:22.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:05:22 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:05:22.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:05:22 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:05:22.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:05:22 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:05:22.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:05:22 smithi047 ceph-mon[77147]: pgmap v3280: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:05:22.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:05:22 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:05:22.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:05:22 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:05:23.052 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:05:22 smithi115 conmon[90393]: 2022-05-14T01:05:22.710+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:05:23.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:05:23 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:05:23.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:05:23 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:05:24.053 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:05:23 smithi115 conmon[90393]: 2022-05-14T01:05:23.747+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:05:24.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:05:24 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:05:24.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:05:24 smithi115 ceph-mon[78593]: pgmap v3281: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:05:24.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:05:24 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:05:24.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:05:24 smithi047 ceph-mon[77147]: pgmap v3281: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:05:25.055 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:05:24 smithi115 conmon[90393]: 2022-05-14T01:05:24.717+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:05:25.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:05:25 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:05:25.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:05:25 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 5855 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:05:25.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:05:25 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:05:25.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:05:25 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 5855 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:05:26.056 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:05:25 smithi115 conmon[90393]: 2022-05-14T01:05:25.740+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:05:26.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:05:26 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:05:26.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:05:26 smithi115 ceph-mon[78593]: pgmap v3282: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:05:26.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:05:26 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:05:26.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:05:26 smithi047 ceph-mon[77147]: pgmap v3282: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:05:27.063 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:05:26 smithi115 conmon[90393]: 2022-05-14T01:05:26.691+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:05:27.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:05:27 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:05:27.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:05:27 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:05:28.063 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:05:27 smithi115 conmon[90393]: 2022-05-14T01:05:27.664+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:05:28.064 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:05:28 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:05:28.064 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:05:28 smithi115 ceph-mon[78593]: pgmap v3283: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:05:28.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:05:28 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:05:28.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:05:28 smithi047 ceph-mon[77147]: pgmap v3283: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:05:29.068 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:05:28 smithi115 conmon[90393]: 2022-05-14T01:05:28.651+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:05:29.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:05:29 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:05:29.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:05:29 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:05:30.069 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:05:29 smithi115 conmon[90393]: 2022-05-14T01:05:29.696+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:05:30.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:05:30 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:05:30.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:05:30 smithi115 ceph-mon[78593]: pgmap v3284: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:05:30.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:05:30 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:05:30.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:05:30 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:05:30.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:05:30 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 5860 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:05:30.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:05:30 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:05:30.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:05:30 smithi047 ceph-mon[77147]: pgmap v3284: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:05:30.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:05:30 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:05:30.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:05:30 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:05:30.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:05:30 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 5860 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:05:31.075 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:05:30 smithi115 conmon[90393]: 2022-05-14T01:05:30.700+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:05:31.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:05:31 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:05:31.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:05:31 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:05:32.076 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:05:31 smithi115 conmon[90393]: 2022-05-14T01:05:31.658+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:05:32.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:05:32 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:05:32.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:05:32 smithi115 ceph-mon[78593]: pgmap v3285: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:05:32.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:05:32 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:05:32.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:05:32 smithi047 ceph-mon[77147]: pgmap v3285: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:05:33.082 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:05:32 smithi115 conmon[90393]: 2022-05-14T01:05:32.699+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:05:33.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:05:33 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:05:33.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:05:33 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:05:34.083 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:05:33 smithi115 conmon[90393]: 2022-05-14T01:05:33.728+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:05:34.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:05:34 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:05:34.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:05:34 smithi115 ceph-mon[78593]: pgmap v3286: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:05:34.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:05:34 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:05:34.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:05:34 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:05:34.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:05:34 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:05:34.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:05:34 smithi047 ceph-mon[77147]: pgmap v3286: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:05:34.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:05:34 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:05:34.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:05:34 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:05:35.084 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:05:34 smithi115 conmon[90393]: 2022-05-14T01:05:34.716+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:05:35.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:05:35 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:05:35.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:05:35 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 5865 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:05:35.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:05:35 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:05:35.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:05:35 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 5865 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:05:36.085 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:05:35 smithi115 conmon[90393]: 2022-05-14T01:05:35.693+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:05:36.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:05:36 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:05:36.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:05:36 smithi115 ceph-mon[78593]: pgmap v3287: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:05:36.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:05:36 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:05:36.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:05:36 smithi047 ceph-mon[77147]: pgmap v3287: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:05:37.093 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:05:36 smithi115 conmon[90393]: 2022-05-14T01:05:36.700+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:05:37.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:05:37 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:05:37.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:05:37 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:05:38.092 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:05:37 smithi115 conmon[90393]: 2022-05-14T01:05:37.676+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:05:38.093 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:05:38 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:05:38.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:05:38 smithi115 ceph-mon[78593]: pgmap v3288: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:05:38.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:05:38 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:05:38.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:05:38 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:05:38.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:05:38 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:05:38.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:05:38 smithi047 ceph-mon[77147]: pgmap v3288: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:05:38.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:05:38 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:05:38.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:05:38 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:05:39.097 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:05:38 smithi115 conmon[90393]: 2022-05-14T01:05:38.681+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:05:39.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:05:39 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:05:39.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:05:39 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:05:40.099 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:05:39 smithi115 conmon[90393]: 2022-05-14T01:05:39.636+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:05:40.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:05:40 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:05:40.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:05:40 smithi115 ceph-mon[78593]: pgmap v3289: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:05:40.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:05:40 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 5870 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:05:40.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:05:40 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:05:40.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:05:40 smithi047 ceph-mon[77147]: pgmap v3289: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:05:40.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:05:40 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 5870 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:05:40.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:05:40 smithi115 conmon[90393]: 2022-05-14T01:05:40.593+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:05:41.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:05:41 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:05:41.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:05:41 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:05:42.105 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:05:41 smithi115 conmon[90393]: 2022-05-14T01:05:41.629+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:05:42.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:05:42 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:05:42.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:05:42 smithi115 ceph-mon[78593]: pgmap v3290: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:05:42.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:05:42 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:05:42.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:05:42 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:05:42.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:05:42 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:05:42.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:05:42 smithi047 ceph-mon[77147]: pgmap v3290: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:05:42.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:05:42 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:05:42.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:05:42 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:05:43.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:05:42 smithi115 conmon[90393]: 2022-05-14T01:05:42.614+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:05:43.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:05:43 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:05:43.592 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:05:43 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:05:43.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:05:43 smithi115 conmon[90393]: 2022-05-14T01:05:43.590+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:05:44.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:05:44 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:05:44.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:05:44 smithi047 ceph-mon[77147]: pgmap v3291: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:05:44.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:05:44 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:05:44.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:05:44 smithi115 ceph-mon[78593]: pgmap v3291: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:05:45.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:05:44 smithi115 conmon[90393]: 2022-05-14T01:05:44.637+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:05:45.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:05:45 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:05:45.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:05:45 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 5875 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:05:45.604 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:05:45 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:05:45.604 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:05:45 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 5875 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:05:45.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:05:45 smithi115 conmon[90393]: 2022-05-14T01:05:45.603+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:05:46.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:05:46 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:05:46.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:05:46 smithi047 ceph-mon[77147]: pgmap v3292: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:05:46.605 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:05:46 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:05:46.606 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:05:46 smithi115 ceph-mon[78593]: pgmap v3292: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:05:46.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:05:46 smithi115 conmon[90393]: 2022-05-14T01:05:46.604+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:05:47.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:05:47 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:05:47.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:05:47 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:05:48.110 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:05:47 smithi115 conmon[90393]: 2022-05-14T01:05:47.641+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:05:48.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:05:48 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:05:48.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:05:48 smithi047 ceph-mon[77147]: pgmap v3293: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:05:48.606 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:05:48 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:05:48.606 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:05:48 smithi115 ceph-mon[78593]: pgmap v3293: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:05:48.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:05:48 smithi115 conmon[90393]: 2022-05-14T01:05:48.605+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:05:49.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:05:49 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:05:49.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:05:49 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:05:50.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:05:49 smithi115 conmon[90393]: 2022-05-14T01:05:49.622+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:05:50.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:05:50 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:05:50.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:05:50 smithi047 ceph-mon[77147]: pgmap v3294: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:05:50.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:05:50 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:05:50.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:05:50 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:05:50.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:05:50 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 5880 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:05:50.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:05:50 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:05:50.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:05:50 smithi115 ceph-mon[78593]: pgmap v3294: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:05:50.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:05:50 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:05:50.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:05:50 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:05:50.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:05:50 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 5880 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:05:50.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:05:50 smithi115 conmon[90393]: 2022-05-14T01:05:50.606+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:05:51.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:05:51 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:05:51.595 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:05:51 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:05:51.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:05:51 smithi115 conmon[90393]: 2022-05-14T01:05:51.594+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:05:52.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:05:52 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:05:52.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:05:52 smithi047 ceph-mon[77147]: pgmap v3295: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:05:52.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:05:52 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:05:52.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:05:52 smithi115 ceph-mon[78593]: pgmap v3295: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:05:52.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:05:52 smithi115 conmon[90393]: 2022-05-14T01:05:52.606+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:05:53.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:05:53 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:05:53.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:05:53 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:05:53.564 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:05:53 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:05:53.564 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:05:53 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:05:53.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:05:53 smithi115 conmon[90393]: 2022-05-14T01:05:53.563+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:05:54.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:05:54 smithi047 ceph-mon[77147]: pgmap v3296: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:05:54.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:05:54 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:05:54.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:05:54 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:05:54.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:05:54 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:05:54.568 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:05:54 smithi115 ceph-mon[78593]: pgmap v3296: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:05:54.568 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:05:54 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:05:54.568 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:05:54 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:05:54.569 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:05:54 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:05:54.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:05:54 smithi115 conmon[90393]: 2022-05-14T01:05:54.567+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:05:55.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:05:55 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 5885 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:05:55.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:05:55 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:05:55.587 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:05:55 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 5885 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:05:55.588 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:05:55 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:05:55.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:05:55 smithi115 conmon[90393]: 2022-05-14T01:05:55.587+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:05:56.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:05:56 smithi047 ceph-mon[77147]: pgmap v3297: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:05:56.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:05:56 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:05:56.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:05:56 smithi115 ceph-mon[78593]: pgmap v3297: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:05:56.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:05:56 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:05:57.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:05:56 smithi115 conmon[90393]: 2022-05-14T01:05:56.619+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:05:57.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:05:57 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:05:57.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:05:57 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:05:58.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:05:57 smithi115 conmon[90393]: 2022-05-14T01:05:57.628+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:05:58.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:05:58 smithi047 ceph-mon[77147]: pgmap v3298: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:05:58.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:05:58 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:05:58.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:05:58 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:05:58.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:05:58 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:05:58.591 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:05:58 smithi115 ceph-mon[78593]: pgmap v3298: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:05:58.591 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:05:58 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:05:58.591 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:05:58 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:05:58.592 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:05:58 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:05:58.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:05:58 smithi115 conmon[90393]: 2022-05-14T01:05:58.590+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:05:59.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:05:59 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:05:59.565 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:05:59 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:05:59.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:05:59 smithi115 conmon[90393]: 2022-05-14T01:05:59.564+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:06:00.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:06:00 smithi047 ceph-mon[77147]: pgmap v3299: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:06:00.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:06:00 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 5890 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:06:00.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:06:00 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:06:00.556 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:06:00 smithi115 ceph-mon[78593]: pgmap v3299: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:06:00.556 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:06:00 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 5890 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:06:00.557 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:06:00 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:06:00.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:06:00 smithi115 conmon[90393]: 2022-05-14T01:06:00.555+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:06:01.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:06:01 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:06:01.570 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:06:01 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:06:01.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:06:01 smithi115 conmon[90393]: 2022-05-14T01:06:01.569+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:06:02.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:06:02 smithi047 ceph-mon[77147]: pgmap v3300: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:06:02.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:06:02 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:06:02.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:06:02 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:06:02.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:06:02 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:06:02.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:06:02 smithi115 ceph-mon[78593]: pgmap v3300: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:06:02.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:06:02 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:06:02.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:06:02 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:06:02.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:06:02 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:06:03.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:06:02 smithi115 conmon[90393]: 2022-05-14T01:06:02.611+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:06:03.519 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:06:03 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:06:03.520 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:06:03 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:06:03.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:06:03 smithi115 conmon[90393]: 2022-05-14T01:06:03.582+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:06:04.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:06:04 smithi047 ceph-mon[77147]: pgmap v3301: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:06:04.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:06:04 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T01:06:04.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:06:04 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T01:06:04.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:06:04 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T01:06:04.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:06:04 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:06:04.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:06:04 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' 2022-05-14T01:06:04.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:06:04 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' 2022-05-14T01:06:04.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:06:04 smithi115 ceph-mon[78593]: pgmap v3301: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:06:04.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:06:04 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T01:06:04.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:06:04 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T01:06:04.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:06:04 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T01:06:04.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:06:04 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:06:04.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:06:04 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' 2022-05-14T01:06:04.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:06:04 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' 2022-05-14T01:06:04.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:06:04 smithi115 conmon[90393]: 2022-05-14T01:06:04.601+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:06:05.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:06:05 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 5895 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:06:05.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:06:05 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:06:05.558 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:06:05 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 5895 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:06:05.559 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:06:05 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:06:05.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:06:05 smithi115 conmon[90393]: 2022-05-14T01:06:05.558+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:06:06.488 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:06:06 smithi115 ceph-mon[78593]: pgmap v3302: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:06:06.488 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:06:06 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:06:06.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:06:06 smithi047 ceph-mon[77147]: pgmap v3302: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:06:06.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:06:06 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:06:06.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:06:06 smithi115 conmon[90393]: 2022-05-14T01:06:06.581+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:06:07.041 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:06:06 smithi047 conmon[78521]: 2022-05-14T01:06:06.576+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:06:07.041 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:06:06 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T01:06:07.041 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:06:06 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:06:07.042 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:06:06 smithi047 conmon[78521]: yield 2022-05-14T01:06:07.042 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:06:06 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:06:07.042 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:06:06 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:06:07.042 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:06:06 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:06:07.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:06:06 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T01:06:07.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:06:06 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:06:07.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:06:06 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T01:06:07.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:06:06 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:06:07.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:06:06 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T01:06:07.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:06:06 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:06:07.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:06:06 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T01:06:07.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:06:06 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:06:07.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:06:06 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T01:06:07.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:06:06 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:06:07.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:06:06 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:06:07.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:06:06 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:06:07.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:06:07 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' 2022-05-14T01:06:07.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:06:07 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' 2022-05-14T01:06:07.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:06:07 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:06:07.793 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T01:06:07.793 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:06:07.793 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T01:06:07.793 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:06:07.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:06:07.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:06:07.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T01:06:07.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:06:07.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T01:06:07.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:06:07.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T01:06:07.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:06:07.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T01:06:07.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:06:07.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T01:06:07.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:06:07.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:06:07.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:06:07.797 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:06:07 smithi047 ceph-mon[77147]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T01:06:07.797 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:06:07 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:06:07.797 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:06:07 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-05-14T01:06:07.798 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:06:07 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-05-14T01:06:07.798 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:06:07 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-05-14T01:06:07.798 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:06:07 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-05-14T01:06:07.798 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:06:07 smithi047 ceph-mon[77147]: Adjusting osd_memory_target on smithi115 to 197.0M 2022-05-14T01:06:07.798 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:06:07 smithi047 ceph-mon[77147]: Unable to set osd_memory_target on smithi115 to 206629785: error parsing value: Value '206629785' is below minimum 939524096 2022-05-14T01:06:07.799 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:06:07 smithi047 ceph-mon[77147]: pgmap v3303: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:06:07.799 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:06:07 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-05-14T01:06:07.799 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:06:07 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-05-14T01:06:07.799 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:06:07 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-05-14T01:06:07.800 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:06:07 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-05-14T01:06:07.800 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:06:07 smithi047 ceph-mon[77147]: Adjusting osd_memory_target on smithi047 to 198.1M 2022-05-14T01:06:07.800 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:06:07 smithi047 ceph-mon[77147]: Unable to set osd_memory_target on smithi047 to 207755878: error parsing value: Value '207755878' is below minimum 939524096 2022-05-14T01:06:07.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:06:07 smithi115 conmon[90393]: 2022-05-14T01:06:07.573+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:06:07.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:06:07 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' 2022-05-14T01:06:07.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:06:07 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' 2022-05-14T01:06:07.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:06:07 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:06:07.859 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T01:06:07.859 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:06:07.860 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T01:06:07.860 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:06:07.860 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:06:07.860 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:06:07.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T01:06:07.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:06:07.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T01:06:07.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:06:07.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T01:06:07.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:06:07.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T01:06:07.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:06:07.863 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T01:06:07.863 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:06:07.863 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:06:07.863 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:06:07.864 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:06:07 smithi115 ceph-mon[78593]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T01:06:07.864 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:06:07 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:06:07.864 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:06:07 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-05-14T01:06:07.864 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:06:07 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-05-14T01:06:07.864 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:06:07 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-05-14T01:06:07.865 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:06:07 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-05-14T01:06:07.865 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:06:07 smithi115 ceph-mon[78593]: Adjusting osd_memory_target on smithi115 to 197.0M 2022-05-14T01:06:07.865 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:06:07 smithi115 ceph-mon[78593]: Unable to set osd_memory_target on smithi115 to 206629785: error parsing value: Value '206629785' is below minimum 939524096 2022-05-14T01:06:07.865 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:06:07 smithi115 ceph-mon[78593]: pgmap v3303: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:06:07.866 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:06:07 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-05-14T01:06:07.866 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:06:07 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-05-14T01:06:07.866 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:06:07 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-05-14T01:06:07.866 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:06:07 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-05-14T01:06:07.867 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:06:07 smithi115 ceph-mon[78593]: Adjusting osd_memory_target on smithi047 to 198.1M 2022-05-14T01:06:07.867 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:06:07 smithi115 ceph-mon[78593]: Unable to set osd_memory_target on smithi047 to 207755878: error parsing value: Value '207755878' is below minimum 939524096 2022-05-14T01:06:08.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:06:08 smithi115 conmon[90393]: 2022-05-14T01:06:08.549+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:06:08.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:06:08 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:06:09.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:06:08 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:06:09.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:06:09 smithi115 conmon[90393]: 2022-05-14T01:06:09.554+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:06:09.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:06:09 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:06:09.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:06:09 smithi115 ceph-mon[78593]: pgmap v3304: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:06:09.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:06:09 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 5900 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:06:10.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:06:09 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:06:10.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:06:09 smithi047 ceph-mon[77147]: pgmap v3304: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:06:10.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:06:09 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 5900 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:06:10.042 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:06:09 smithi047 conmon[78521]: 2022-05-14T01:06:09.647+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:06:10.042 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:06:09 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T01:06:10.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:06:09 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:06:10.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:06:09 smithi047 conmon[78521]: yield 2022-05-14T01:06:10.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:06:09 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:06:10.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:06:09 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:06:10.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:06:09 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:06:10.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:06:09 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T01:06:10.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:06:09 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:06:10.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:06:09 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T01:06:10.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:06:09 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:06:10.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:06:09 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T01:06:10.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:06:09 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:06:10.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:06:09 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T01:06:10.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:06:09 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:06:10.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:06:09 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T01:06:10.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:06:09 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:06:10.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:06:09 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:06:10.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:06:09 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:06:10.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:06:09 smithi047 conmon[78521]: 2022-05-14T01:06:09.648+0000 7f533ff78700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T01:06:10.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:06:09 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T01:06:10.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:06:09 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:06:10.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:06:09 smithi047 conmon[78521]: yield 2022-05-14T01:06:10.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:06:09 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:06:10.048 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:06:09 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:06:10.048 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:06:09 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:06:10.048 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:06:09 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T01:06:10.048 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:06:09 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:06:10.049 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:06:09 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T01:06:10.049 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:06:09 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:06:10.049 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:06:09 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T01:06:10.049 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:06:09 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:06:10.049 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:06:09 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T01:06:10.050 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:06:09 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:06:10.050 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:06:09 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T01:06:10.050 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:06:09 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:06:10.050 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:06:09 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:06:10.051 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:06:09 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:06:10.051 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:06:09 smithi047 conmon[78521]: 2022-05-14T01:06:10.051 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:06:09 smithi047 conmon[78521]: During handling of the above exception, another exception occurred: 2022-05-14T01:06:10.051 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:06:09 smithi047 conmon[78521]: 2022-05-14T01:06:10.051 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:06:09 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T01:06:10.052 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:06:09 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T01:06:10.052 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:06:09 smithi047 conmon[78521]: return f(*arg) 2022-05-14T01:06:10.052 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:06:09 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T01:06:10.052 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:06:09 smithi047 conmon[78521]: self._write_client_files(client_files, host) 2022-05-14T01:06:10.052 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:06:09 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T01:06:10.053 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:06:09 smithi047 conmon[78521]: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T01:06:10.053 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:06:09 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T01:06:10.053 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:06:09 smithi047 conmon[78521]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T01:06:10.053 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:06:09 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T01:06:10.054 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:06:09 smithi047 conmon[78521]: return self.event_loop.get_result(coro) 2022-05-14T01:06:10.054 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:06:09 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T01:06:10.054 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:06:09 smithi047 conmon[78521]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T01:06:10.054 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:06:09 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T01:06:10.055 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:06:09 smithi047 conmon[78521]: return self.__get_result() 2022-05-14T01:06:10.055 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:06:09 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T01:06:10.055 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:06:09 smithi047 conmon[78521]: raise self._exception 2022-05-14T01:06:10.055 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:06:09 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T01:06:10.055 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:06:09 smithi047 conmon[78521]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T01:06:10.056 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:06:09 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T01:06:10.056 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:06:09 smithi047 conmon[78521]: conn = await self._remote_connection(host, addr) 2022-05-14T01:06:10.056 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:06:09 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T01:06:10.056 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:06:09 smithi047 conmon[78521]: raise 2022-05-14T01:06:10.056 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:06:09 smithi047 conmon[78521]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T01:06:10.057 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:06:09 smithi047 conmon[78521]: self.gen.throw(type, value, traceback) 2022-05-14T01:06:10.057 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:06:09 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T01:06:10.057 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:06:09 smithi047 conmon[78521]: raise OrchestratorError(msg) 2022-05-14T01:06:10.057 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:06:09 smithi047 conmon[78521]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:06:10.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:06:10 smithi115 conmon[90393]: 2022-05-14T01:06:10.521+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:06:10.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:06:10 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:06:10.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:06:10 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:06:10.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:06:10 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:06:10.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:06:10 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:06:10.859 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T01:06:10.859 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:06:10.860 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T01:06:10.860 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:06:10.860 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:06:10.860 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:06:10.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T01:06:10.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:06:10.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T01:06:10.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:06:10.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T01:06:10.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:06:10.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T01:06:10.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:06:10.863 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T01:06:10.863 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:06:10.863 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:06:10.863 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:06:10.864 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:06:10 smithi115 ceph-mon[78593]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T01:06:10.864 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T01:06:10.864 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:06:10.864 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T01:06:10.864 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:06:10.865 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:06:10.865 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:06:10.865 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T01:06:10.865 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:06:10.866 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T01:06:10.866 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:06:10.866 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T01:06:10.866 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:06:10.866 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T01:06:10.867 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:06:10.867 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T01:06:10.867 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:06:10.867 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:06:10.868 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:06:10.868 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T01:06:10.868 INFO:journalctl@ceph.mon.c.smithi115.stdout: During handling of the above exception, another exception occurred: 2022-05-14T01:06:10.868 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T01:06:10.868 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T01:06:10.869 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T01:06:10.869 INFO:journalctl@ceph.mon.c.smithi115.stdout: return f(*arg) 2022-05-14T01:06:10.869 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T01:06:10.869 INFO:journalctl@ceph.mon.c.smithi115.stdout: self._write_client_files(client_files, host) 2022-05-14T01:06:10.869 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T01:06:10.870 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T01:06:10.870 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T01:06:10.870 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T01:06:10.870 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T01:06:10.870 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.event_loop.get_result(coro) 2022-05-14T01:06:10.871 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T01:06:10.871 INFO:journalctl@ceph.mon.c.smithi115.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T01:06:10.871 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T01:06:10.871 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.__get_result() 2022-05-14T01:06:10.872 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T01:06:10.872 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise self._exception 2022-05-14T01:06:10.872 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T01:06:10.872 INFO:journalctl@ceph.mon.c.smithi115.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T01:06:10.872 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T01:06:10.873 INFO:journalctl@ceph.mon.c.smithi115.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T01:06:10.873 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T01:06:10.873 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise 2022-05-14T01:06:10.873 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T01:06:10.873 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.gen.throw(type, value, traceback) 2022-05-14T01:06:10.874 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T01:06:10.874 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OrchestratorError(msg) 2022-05-14T01:06:10.874 INFO:journalctl@ceph.mon.c.smithi115.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:06:11.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:06:10 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:06:11.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:06:10 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:06:11.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:06:10 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:06:11.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:06:10 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:06:11.042 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T01:06:11.042 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:06:11.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T01:06:11.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:06:11.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:06:11.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:06:11.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T01:06:11.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:06:11.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T01:06:11.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:06:11.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T01:06:11.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:06:11.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T01:06:11.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:06:11.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T01:06:11.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:06:11.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:06:11.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:06:11.047 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:06:10 smithi047 ceph-mon[77147]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T01:06:11.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T01:06:11.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:06:11.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T01:06:11.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:06:11.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:06:11.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:06:11.049 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T01:06:11.049 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:06:11.049 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T01:06:11.049 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:06:11.050 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T01:06:11.050 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:06:11.050 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T01:06:11.050 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:06:11.051 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T01:06:11.051 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:06:11.051 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:06:11.051 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:06:11.051 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T01:06:11.052 INFO:journalctl@ceph.mon.b.smithi047.stdout: During handling of the above exception, another exception occurred: 2022-05-14T01:06:11.052 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T01:06:11.052 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T01:06:11.052 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T01:06:11.053 INFO:journalctl@ceph.mon.b.smithi047.stdout: return f(*arg) 2022-05-14T01:06:11.053 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T01:06:11.053 INFO:journalctl@ceph.mon.b.smithi047.stdout: self._write_client_files(client_files, host) 2022-05-14T01:06:11.053 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T01:06:11.054 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T01:06:11.054 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T01:06:11.054 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T01:06:11.054 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T01:06:11.055 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.event_loop.get_result(coro) 2022-05-14T01:06:11.055 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T01:06:11.055 INFO:journalctl@ceph.mon.b.smithi047.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T01:06:11.055 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T01:06:11.055 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.__get_result() 2022-05-14T01:06:11.056 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T01:06:11.056 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise self._exception 2022-05-14T01:06:11.056 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T01:06:11.056 INFO:journalctl@ceph.mon.b.smithi047.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T01:06:11.057 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T01:06:11.057 INFO:journalctl@ceph.mon.b.smithi047.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T01:06:11.057 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T01:06:11.057 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise 2022-05-14T01:06:11.057 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T01:06:11.058 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.gen.throw(type, value, traceback) 2022-05-14T01:06:11.058 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T01:06:11.058 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OrchestratorError(msg) 2022-05-14T01:06:11.058 INFO:journalctl@ceph.mon.b.smithi047.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:06:11.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:06:11 smithi115 conmon[90393]: 2022-05-14T01:06:11.525+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:06:11.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:06:11 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:06:11.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:06:11 smithi115 ceph-mon[78593]: pgmap v3305: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:06:12.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:06:11 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:06:12.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:06:11 smithi047 ceph-mon[77147]: pgmap v3305: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:06:12.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:06:12 smithi115 conmon[90393]: 2022-05-14T01:06:12.540+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:06:12.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:06:12 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:06:13.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:06:12 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:06:13.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:06:13 smithi115 conmon[90393]: 2022-05-14T01:06:13.563+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:06:13.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:06:13 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:06:13.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:06:13 smithi115 ceph-mon[78593]: pgmap v3306: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:06:14.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:06:13 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:06:14.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:06:13 smithi047 ceph-mon[77147]: pgmap v3306: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:06:14.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:06:14 smithi115 conmon[90393]: 2022-05-14T01:06:14.594+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:06:14.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:06:14 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:06:14.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:06:14 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:06:14.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:06:14 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:06:14.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:06:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T01:06:14.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:06:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T01:06:14.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:06:14 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 5905 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:06:15.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:06:14 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:06:15.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:06:14 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:06:15.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:06:14 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:06:15.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:06:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T01:06:15.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:06:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T01:06:15.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:06:14 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 5905 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:06:15.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:06:15 smithi115 conmon[90393]: 2022-05-14T01:06:15.613+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:06:15.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:06:15 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:06:15.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:06:15 smithi115 ceph-mon[78593]: pgmap v3307: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:06:16.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:06:15 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:06:16.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:06:15 smithi047 ceph-mon[77147]: pgmap v3307: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:06:16.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:06:16 smithi115 conmon[90393]: 2022-05-14T01:06:16.640+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:06:16.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:06:16 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:06:17.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:06:16 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:06:17.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:06:17 smithi115 conmon[90393]: 2022-05-14T01:06:17.624+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:06:17.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:06:17 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:06:17.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:06:17 smithi115 ceph-mon[78593]: pgmap v3308: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:06:18.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:06:17 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:06:18.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:06:17 smithi047 ceph-mon[77147]: pgmap v3308: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:06:18.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:06:18 smithi115 conmon[90393]: 2022-05-14T01:06:18.639+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:06:18.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:06:18 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:06:18.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:06:18 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:06:18.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:06:18 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:06:19.047 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:06:18 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:06:19.048 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:06:18 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:06:19.048 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:06:18 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:06:19.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:06:19 smithi115 conmon[90393]: 2022-05-14T01:06:19.619+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:06:19.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:06:19 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:06:19.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:06:19 smithi115 ceph-mon[78593]: pgmap v3309: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:06:19.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:06:19 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 5910 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:06:20.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:06:19 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:06:20.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:06:19 smithi047 ceph-mon[77147]: pgmap v3309: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:06:20.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:06:19 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 5910 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:06:20.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:06:20 smithi115 conmon[90393]: 2022-05-14T01:06:20.610+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:06:20.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:06:20 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:06:21.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:06:20 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:06:21.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:06:21 smithi115 conmon[90393]: 2022-05-14T01:06:21.566+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:06:21.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:06:21 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:06:21.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:06:21 smithi115 ceph-mon[78593]: pgmap v3310: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:06:22.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:06:21 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:06:22.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:06:21 smithi047 ceph-mon[77147]: pgmap v3310: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:06:22.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:06:22 smithi115 conmon[90393]: 2022-05-14T01:06:22.560+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:06:22.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:06:22 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:06:22.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:06:22 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:06:22.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:06:22 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:06:23.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:06:22 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:06:23.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:06:22 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:06:23.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:06:22 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:06:23.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:06:23 smithi115 conmon[90393]: 2022-05-14T01:06:23.541+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:06:23.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:06:23 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:06:23.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:06:23 smithi115 ceph-mon[78593]: pgmap v3311: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:06:24.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:06:23 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:06:24.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:06:23 smithi047 ceph-mon[77147]: pgmap v3311: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:06:24.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:06:24 smithi115 conmon[90393]: 2022-05-14T01:06:24.518+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:06:24.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:06:24 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:06:24.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:06:24 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 5915 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:06:25.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:06:24 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:06:25.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:06:24 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 5915 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:06:25.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:06:25 smithi115 conmon[90393]: 2022-05-14T01:06:25.550+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:06:25.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:06:25 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:06:25.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:06:25 smithi115 ceph-mon[78593]: pgmap v3312: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:06:26.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:06:25 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:06:26.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:06:25 smithi047 ceph-mon[77147]: pgmap v3312: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:06:26.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:06:26 smithi115 conmon[90393]: 2022-05-14T01:06:26.567+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:06:26.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:06:26 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:06:27.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:06:26 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:06:27.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:06:27 smithi115 conmon[90393]: 2022-05-14T01:06:27.564+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:06:27.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:06:27 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:06:27.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:06:27 smithi115 ceph-mon[78593]: pgmap v3313: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:06:28.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:06:27 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:06:28.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:06:27 smithi047 ceph-mon[77147]: pgmap v3313: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:06:28.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:06:28 smithi115 conmon[90393]: 2022-05-14T01:06:28.517+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:06:28.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:06:28 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:06:29.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:06:28 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:06:29.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:06:29 smithi115 conmon[90393]: 2022-05-14T01:06:29.489+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:06:29.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:06:29 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:06:29.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:06:29 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:06:29.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:06:29 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:06:29.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:06:29 smithi115 ceph-mon[78593]: pgmap v3314: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:06:29.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:06:29 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 5920 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:06:30.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:06:29 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:06:30.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:06:29 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:06:30.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:06:29 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:06:30.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:06:29 smithi047 ceph-mon[77147]: pgmap v3314: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:06:30.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:06:29 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 5920 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:06:30.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:06:30 smithi115 conmon[90393]: 2022-05-14T01:06:30.492+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:06:30.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:06:30 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:06:31.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:06:30 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:06:31.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:06:31 smithi115 conmon[90393]: 2022-05-14T01:06:31.518+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:06:31.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:06:31 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:06:31.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:06:31 smithi115 ceph-mon[78593]: pgmap v3315: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:06:32.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:06:31 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:06:32.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:06:31 smithi047 ceph-mon[77147]: pgmap v3315: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:06:32.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:06:32 smithi115 conmon[90393]: 2022-05-14T01:06:32.491+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:06:32.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:06:32 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:06:33.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:06:32 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:06:33.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:06:33 smithi115 conmon[90393]: 2022-05-14T01:06:33.480+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:06:33.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:06:33 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:06:33.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:06:33 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:06:33.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:06:33 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:06:33.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:06:33 smithi115 ceph-mon[78593]: pgmap v3316: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:06:34.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:06:33 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:06:34.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:06:33 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:06:34.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:06:33 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:06:34.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:06:33 smithi047 ceph-mon[77147]: pgmap v3316: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:06:34.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:06:34 smithi115 conmon[90393]: 2022-05-14T01:06:34.493+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:06:34.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:06:34 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:06:34.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:06:34 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 5925 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:06:35.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:06:34 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:06:35.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:06:34 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 5925 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:06:35.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:06:35 smithi115 conmon[90393]: 2022-05-14T01:06:35.469+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:06:35.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:06:35 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:06:35.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:06:35 smithi115 ceph-mon[78593]: pgmap v3317: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:06:36.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:06:35 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:06:36.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:06:35 smithi047 ceph-mon[77147]: pgmap v3317: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:06:36.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:06:36 smithi115 conmon[90393]: 2022-05-14T01:06:36.506+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:06:36.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:06:36 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:06:37.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:06:36 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:06:37.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:06:37 smithi115 conmon[90393]: 2022-05-14T01:06:37.499+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:06:37.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:06:37 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:06:37.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:06:37 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:06:37.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:06:37 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:06:37.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:06:37 smithi115 ceph-mon[78593]: pgmap v3318: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:06:38.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:06:37 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:06:38.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:06:37 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:06:38.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:06:37 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:06:38.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:06:37 smithi047 ceph-mon[77147]: pgmap v3318: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:06:38.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:06:38 smithi115 conmon[90393]: 2022-05-14T01:06:38.465+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:06:38.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:06:38 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:06:39.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:06:38 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:06:39.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:06:39 smithi115 conmon[90393]: 2022-05-14T01:06:39.446+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:06:39.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:06:39 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:06:39.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:06:39 smithi115 ceph-mon[78593]: pgmap v3319: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:06:39.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:06:39 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 5930 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:06:40.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:06:39 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:06:40.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:06:39 smithi047 ceph-mon[77147]: pgmap v3319: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:06:40.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:06:39 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 5930 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:06:40.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:06:40 smithi115 conmon[90393]: 2022-05-14T01:06:40.434+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:06:40.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:06:40 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:06:41.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:06:40 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:06:41.688 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:06:41 smithi115 conmon[90393]: 2022-05-14T01:06:41.405+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:06:42.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:06:41 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:06:42.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:06:41 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:06:42.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:06:41 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:06:42.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:06:41 smithi047 ceph-mon[77147]: pgmap v3320: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:06:42.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:06:41 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:06:42.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:06:41 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:06:42.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:06:41 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:06:42.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:06:41 smithi115 ceph-mon[78593]: pgmap v3320: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:06:42.687 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:06:42 smithi115 conmon[90393]: 2022-05-14T01:06:42.421+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:06:43.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:06:42 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:06:43.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:06:42 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:06:43.694 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:06:43 smithi115 conmon[90393]: 2022-05-14T01:06:43.392+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:06:44.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:06:43 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:06:44.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:06:43 smithi047 ceph-mon[77147]: pgmap v3321: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:06:44.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:06:43 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:06:44.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:06:43 smithi115 ceph-mon[78593]: pgmap v3321: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:06:44.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:06:44 smithi115 conmon[90393]: 2022-05-14T01:06:44.350+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:06:45.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:06:44 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:06:45.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:06:44 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 5935 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:06:45.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:06:44 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:06:45.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:06:44 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 5935 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:06:45.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:06:45 smithi115 conmon[90393]: 2022-05-14T01:06:45.329+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:06:46.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:06:45 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:06:46.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:06:45 smithi047 ceph-mon[77147]: pgmap v3322: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:06:46.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:06:45 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:06:46.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:06:45 smithi115 ceph-mon[78593]: pgmap v3322: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:06:46.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:06:46 smithi115 conmon[90393]: 2022-05-14T01:06:46.340+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:06:47.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:06:46 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:06:47.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:06:46 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:06:47.707 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:06:47 smithi115 conmon[90393]: 2022-05-14T01:06:47.378+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:06:48.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:06:47 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:06:48.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:06:47 smithi047 ceph-mon[77147]: pgmap v3323: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:06:48.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:06:47 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:06:48.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:06:47 smithi115 ceph-mon[78593]: pgmap v3323: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:06:48.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:06:48 smithi115 conmon[90393]: 2022-05-14T01:06:48.342+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:06:49.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:06:48 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:06:49.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:06:48 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:06:49.716 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:06:49 smithi115 conmon[90393]: 2022-05-14T01:06:49.362+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:06:50.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:06:49 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:06:50.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:06:49 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:06:50.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:06:49 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:06:50.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:06:49 smithi047 ceph-mon[77147]: pgmap v3324: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-05-14T01:06:50.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:06:49 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 5940 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:06:50.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:06:49 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:06:50.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:06:49 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:06:50.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:06:49 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:06:50.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:06:49 smithi115 ceph-mon[78593]: pgmap v3324: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-05-14T01:06:50.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:06:49 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 5940 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:06:50.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:06:50 smithi115 conmon[90393]: 2022-05-14T01:06:50.353+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:06:51.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:06:50 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:06:51.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:06:50 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:06:51.724 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:06:51 smithi115 conmon[90393]: 2022-05-14T01:06:51.358+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:06:52.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:06:51 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:06:52.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:06:51 smithi047 ceph-mon[77147]: pgmap v3325: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:06:52.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:06:51 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:06:52.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:06:51 smithi115 ceph-mon[78593]: pgmap v3325: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:06:52.729 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:06:52 smithi115 conmon[90393]: 2022-05-14T01:06:52.406+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:06:53.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:06:52 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:06:53.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:06:52 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:06:53.736 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:06:53 smithi115 conmon[90393]: 2022-05-14T01:06:53.418+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:06:54.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:06:53 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:06:54.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:06:53 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:06:54.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:06:53 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:06:54.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:06:53 smithi047 ceph-mon[77147]: pgmap v3326: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:06:54.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:06:53 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:06:54.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:06:53 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:06:54.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:06:53 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:06:54.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:06:53 smithi115 ceph-mon[78593]: pgmap v3326: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:06:54.736 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:06:54 smithi115 conmon[90393]: 2022-05-14T01:06:54.467+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:06:55.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:06:54 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:06:55.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:06:54 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 5945 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:06:55.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:06:54 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:06:55.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:06:54 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 5945 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:06:55.741 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:06:55 smithi115 conmon[90393]: 2022-05-14T01:06:55.429+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:06:56.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:06:55 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:06:56.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:06:55 smithi047 ceph-mon[77147]: pgmap v3327: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:06:56.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:06:55 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:06:56.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:06:55 smithi115 ceph-mon[78593]: pgmap v3327: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:06:56.746 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:06:56 smithi115 conmon[90393]: 2022-05-14T01:06:56.477+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:06:57.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:06:56 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:06:57.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:06:56 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:06:57.751 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:06:57 smithi115 conmon[90393]: 2022-05-14T01:06:57.449+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:06:58.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:06:57 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:06:58.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:06:57 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:06:58.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:06:57 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:06:58.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:06:57 smithi047 ceph-mon[77147]: pgmap v3328: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:06:58.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:06:57 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:06:58.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:06:57 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:06:58.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:06:57 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:06:58.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:06:57 smithi115 ceph-mon[78593]: pgmap v3328: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:06:58.753 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:06:58 smithi115 conmon[90393]: 2022-05-14T01:06:58.459+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:06:59.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:06:58 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:06:59.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:06:58 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:06:59.757 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:06:59 smithi115 conmon[90393]: 2022-05-14T01:06:59.461+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:07:00.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:06:59 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:07:00.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:06:59 smithi047 ceph-mon[77147]: pgmap v3329: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-05-14T01:07:00.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:06:59 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 5950 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:07:00.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:06:59 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:07:00.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:06:59 smithi115 ceph-mon[78593]: pgmap v3329: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-05-14T01:07:00.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:06:59 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 5950 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:07:00.759 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:07:00 smithi115 conmon[90393]: 2022-05-14T01:07:00.461+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:07:01.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:07:00 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:07:01.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:07:00 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:07:01.769 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:07:01 smithi115 conmon[90393]: 2022-05-14T01:07:01.446+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:07:02.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:07:01 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:07:02.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:07:01 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:07:02.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:07:01 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:07:02.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:07:01 smithi047 ceph-mon[77147]: pgmap v3330: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2022-05-14T01:07:02.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:07:01 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:07:02.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:07:01 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:07:02.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:07:01 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:07:02.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:07:01 smithi115 ceph-mon[78593]: pgmap v3330: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2022-05-14T01:07:02.770 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:07:02 smithi115 conmon[90393]: 2022-05-14T01:07:02.470+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:07:03.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:07:02 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:07:03.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:07:02 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:07:03.774 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:07:03 smithi115 conmon[90393]: 2022-05-14T01:07:03.465+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:07:04.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:07:03 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:07:04.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:07:03 smithi047 ceph-mon[77147]: pgmap v3331: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:07:04.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:07:03 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:07:04.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:07:03 smithi115 ceph-mon[78593]: pgmap v3331: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:07:04.775 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:07:04 smithi115 conmon[90393]: 2022-05-14T01:07:04.497+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:07:05.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:07:04 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:07:05.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:07:04 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 5955 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:07:05.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:07:04 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:07:05.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:07:04 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 5955 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:07:05.780 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:07:05 smithi115 conmon[90393]: 2022-05-14T01:07:05.505+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:07:06.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:07:05 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:07:06.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:07:05 smithi047 ceph-mon[77147]: pgmap v3332: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:07:06.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:07:05 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:07:06.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:07:05 smithi115 ceph-mon[78593]: pgmap v3332: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:07:06.782 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:07:06 smithi115 conmon[90393]: 2022-05-14T01:07:06.460+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:07:07.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:07:06 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:07:07.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:07:06 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:07:07.787 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:07:07 smithi115 conmon[90393]: 2022-05-14T01:07:07.487+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:07:08.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:07:07 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:07:08.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:07:07 smithi047 ceph-mon[77147]: pgmap v3333: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:07:08.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:07:07 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:07:08.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:07:07 smithi115 ceph-mon[78593]: pgmap v3333: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:07:08.793 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:07:08 smithi115 conmon[90393]: 2022-05-14T01:07:08.492+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:07:09.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:07:08 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:07:09.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:07:08 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:07:09.778 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:07:09 smithi115 conmon[90393]: 2022-05-14T01:07:09.510+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:07:10.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:07:09 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:07:10.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:07:09 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:07:10.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:07:09 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:07:10.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:07:09 smithi047 ceph-mon[77147]: pgmap v3334: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:07:10.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:07:09 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 5960 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:07:10.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:07:09 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T01:07:10.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:07:09 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T01:07:10.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:07:09 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T01:07:10.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:07:09 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:07:10.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:07:09 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:07:10.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:07:09 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:07:10.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:07:09 smithi115 ceph-mon[78593]: pgmap v3334: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:07:10.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:07:09 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 5960 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:07:10.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:07:09 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T01:07:10.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:07:09 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T01:07:10.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:07:09 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T01:07:10.541 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:07:10 smithi047 conmon[78521]: 2022-05-14T01:07:10.256+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:07:10.541 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:07:10 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T01:07:10.542 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:07:10 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:07:10.542 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:07:10 smithi047 conmon[78521]: yield 2022-05-14T01:07:10.542 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:07:10 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:07:10.542 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:07:10 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:07:10.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:07:10 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:07:10.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:07:10 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T01:07:10.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:07:10 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:07:10.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:07:10 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T01:07:10.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:07:10 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:07:10.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:07:10 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T01:07:10.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:07:10 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:07:10.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:07:10 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T01:07:10.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:07:10 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:07:10.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:07:10 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T01:07:10.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:07:10 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:07:10.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:07:10 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:07:10.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:07:10 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:07:10.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:07:10 smithi115 conmon[90393]: 2022-05-14T01:07:10.559+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:07:10.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:07:10 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:07:10.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:07:10 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:07:10.858 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T01:07:10.859 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:07:10.859 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T01:07:10.859 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:07:10.859 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:07:10.860 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:07:10.860 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T01:07:10.860 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:07:10.860 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T01:07:10.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:07:10.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T01:07:10.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:07:10.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T01:07:10.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:07:10.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T01:07:10.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:07:10.863 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:07:10.863 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:07:10.863 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:07:10 smithi115 ceph-mon[78593]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T01:07:11.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:07:10 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:07:11.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:07:10 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:07:11.291 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T01:07:11.292 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:07:11.292 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T01:07:11.292 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:07:11.292 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:07:11.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:07:11.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T01:07:11.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:07:11.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T01:07:11.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:07:11.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T01:07:11.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:07:11.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T01:07:11.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:07:11.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T01:07:11.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:07:11.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:07:11.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:07:11.295 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:07:10 smithi047 ceph-mon[77147]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T01:07:11.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:07:11 smithi115 conmon[90393]: 2022-05-14T01:07:11.602+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:07:11.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:07:11 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:07:11.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:07:11 smithi115 ceph-mon[78593]: pgmap v3335: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2022-05-14T01:07:12.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:07:11 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:07:12.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:07:11 smithi047 ceph-mon[77147]: pgmap v3335: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2022-05-14T01:07:12.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:07:12 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:07:12.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:07:12 smithi115 conmon[90393]: 2022-05-14T01:07:12.602+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:07:13.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:07:12 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:07:13.791 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:07:13 smithi047 conmon[78521]: 2022-05-14T01:07:13.328+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:07:13.792 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:07:13 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T01:07:13.792 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:07:13 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:07:13.792 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:07:13 smithi047 conmon[78521]: yield 2022-05-14T01:07:13.793 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:07:13 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:07:13.793 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:07:13 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:07:13.793 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:07:13 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:07:13.793 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:07:13 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T01:07:13.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:07:13 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:07:13.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:07:13 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T01:07:13.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:07:13 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:07:13.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:07:13 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T01:07:13.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:07:13 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:07:13.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:07:13 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T01:07:13.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:07:13 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:07:13.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:07:13 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T01:07:13.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:07:13 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:07:13.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:07:13 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:07:13.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:07:13 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:07:13.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:07:13 smithi047 conmon[78521]: 2022-05-14T01:07:13.329+0000 7f533f777700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T01:07:13.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:07:13 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T01:07:13.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:07:13 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:07:13.797 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:07:13 smithi047 conmon[78521]: yield 2022-05-14T01:07:13.797 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:07:13 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:07:13.797 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:07:13 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:07:13.798 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:07:13 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:07:13.798 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:07:13 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T01:07:13.798 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:07:13 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:07:13.798 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:07:13 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T01:07:13.799 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:07:13 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:07:13.799 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:07:13 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T01:07:13.799 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:07:13 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:07:13.799 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:07:13 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T01:07:13.799 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:07:13 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:07:13.800 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:07:13 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T01:07:13.800 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:07:13 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:07:13.800 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:07:13 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:07:13.800 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:07:13 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:07:13.801 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:07:13 smithi047 conmon[78521]: 2022-05-14T01:07:13.801 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:07:13 smithi047 conmon[78521]: During handling of the above exception, another exception occurred: 2022-05-14T01:07:13.801 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:07:13 smithi047 conmon[78521]: 2022-05-14T01:07:13.801 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:07:13 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T01:07:13.802 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:07:13 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T01:07:13.802 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:07:13 smithi047 conmon[78521]: return f(*arg) 2022-05-14T01:07:13.802 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:07:13 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T01:07:13.802 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:07:13 smithi047 conmon[78521]: self._write_client_files(client_files, host) 2022-05-14T01:07:13.803 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:07:13 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T01:07:13.803 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:07:13 smithi047 conmon[78521]: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T01:07:13.803 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:07:13 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T01:07:13.803 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:07:13 smithi047 conmon[78521]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T01:07:13.804 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:07:13 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T01:07:13.804 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:07:13 smithi047 conmon[78521]: return self.event_loop.get_result(coro) 2022-05-14T01:07:13.804 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:07:13 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T01:07:13.804 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:07:13 smithi047 conmon[78521]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T01:07:13.805 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:07:13 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T01:07:13.805 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:07:13 smithi047 conmon[78521]: return self.__get_result() 2022-05-14T01:07:13.807 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:07:13 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T01:07:13.808 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:07:13 smithi047 conmon[78521]: raise self._exception 2022-05-14T01:07:13.808 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:07:13 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T01:07:13.808 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:07:13 smithi047 conmon[78521]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T01:07:13.808 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:07:13 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T01:07:13.809 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:07:13 smithi047 conmon[78521]: conn = await self._remote_connection(host, addr) 2022-05-14T01:07:13.809 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:07:13 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T01:07:13.809 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:07:13 smithi047 conmon[78521]: raise 2022-05-14T01:07:13.809 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:07:13 smithi047 conmon[78521]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T01:07:13.810 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:07:13 smithi047 conmon[78521]: self.gen.throw(type, value, traceback) 2022-05-14T01:07:13.810 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:07:13 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T01:07:13.810 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:07:13 smithi047 conmon[78521]: raise OrchestratorError(msg) 2022-05-14T01:07:13.810 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:07:13 smithi047 conmon[78521]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:07:14.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:07:13 smithi115 conmon[90393]: 2022-05-14T01:07:13.627+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:07:14.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:07:13 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:07:14.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:07:13 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:07:14.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:07:13 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:07:14.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:07:13 smithi115 ceph-mon[78593]: pgmap v3336: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:07:14.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:07:13 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:07:14.109 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T01:07:14.109 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:07:14.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T01:07:14.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:07:14.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:07:14.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:07:14.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T01:07:14.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:07:14.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T01:07:14.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:07:14.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T01:07:14.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:07:14.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T01:07:14.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:07:14.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T01:07:14.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:07:14.113 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:07:14.113 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:07:14.113 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:07:13 smithi115 ceph-mon[78593]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T01:07:14.113 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T01:07:14.114 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:07:14.114 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T01:07:14.114 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:07:14.114 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:07:14.114 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:07:14.115 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T01:07:14.115 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:07:14.115 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T01:07:14.115 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:07:14.116 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T01:07:14.116 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:07:14.116 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T01:07:14.116 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:07:14.116 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T01:07:14.117 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:07:14.117 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:07:14.117 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:07:14.117 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T01:07:14.118 INFO:journalctl@ceph.mon.c.smithi115.stdout: During handling of the above exception, another exception occurred: 2022-05-14T01:07:14.118 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T01:07:14.118 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T01:07:14.119 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T01:07:14.119 INFO:journalctl@ceph.mon.c.smithi115.stdout: return f(*arg) 2022-05-14T01:07:14.119 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T01:07:14.119 INFO:journalctl@ceph.mon.c.smithi115.stdout: self._write_client_files(client_files, host) 2022-05-14T01:07:14.120 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T01:07:14.120 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T01:07:14.120 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T01:07:14.120 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T01:07:14.120 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T01:07:14.121 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.event_loop.get_result(coro) 2022-05-14T01:07:14.121 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T01:07:14.121 INFO:journalctl@ceph.mon.c.smithi115.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T01:07:14.121 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T01:07:14.122 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.__get_result() 2022-05-14T01:07:14.122 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T01:07:14.122 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise self._exception 2022-05-14T01:07:14.122 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T01:07:14.122 INFO:journalctl@ceph.mon.c.smithi115.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T01:07:14.123 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T01:07:14.123 INFO:journalctl@ceph.mon.c.smithi115.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T01:07:14.124 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T01:07:14.124 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise 2022-05-14T01:07:14.124 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T01:07:14.124 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.gen.throw(type, value, traceback) 2022-05-14T01:07:14.125 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T01:07:14.125 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OrchestratorError(msg) 2022-05-14T01:07:14.125 INFO:journalctl@ceph.mon.c.smithi115.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:07:14.125 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:07:13 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T01:07:14.125 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:07:13 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T01:07:14.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:07:13 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:07:14.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:07:13 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:07:14.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:07:13 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:07:14.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:07:13 smithi047 ceph-mon[77147]: pgmap v3336: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:07:14.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:07:13 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:07:14.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T01:07:14.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:07:14.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T01:07:14.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:07:14.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:07:14.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:07:14.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T01:07:14.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:07:14.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T01:07:14.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:07:14.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T01:07:14.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:07:14.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T01:07:14.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:07:14.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T01:07:14.298 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:07:14.298 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:07:14.298 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:07:14.298 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:07:13 smithi047 ceph-mon[77147]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T01:07:14.299 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T01:07:14.299 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:07:14.299 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T01:07:14.299 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:07:14.299 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:07:14.300 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:07:14.300 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T01:07:14.300 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:07:14.300 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T01:07:14.301 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:07:14.301 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T01:07:14.301 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:07:14.301 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T01:07:14.301 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:07:14.302 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T01:07:14.302 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:07:14.302 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:07:14.302 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:07:14.303 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T01:07:14.303 INFO:journalctl@ceph.mon.b.smithi047.stdout: During handling of the above exception, another exception occurred: 2022-05-14T01:07:14.303 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T01:07:14.303 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T01:07:14.304 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T01:07:14.304 INFO:journalctl@ceph.mon.b.smithi047.stdout: return f(*arg) 2022-05-14T01:07:14.304 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T01:07:14.304 INFO:journalctl@ceph.mon.b.smithi047.stdout: self._write_client_files(client_files, host) 2022-05-14T01:07:14.304 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T01:07:14.305 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T01:07:14.305 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T01:07:14.305 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T01:07:14.305 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T01:07:14.306 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.event_loop.get_result(coro) 2022-05-14T01:07:14.306 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T01:07:14.306 INFO:journalctl@ceph.mon.b.smithi047.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T01:07:14.306 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T01:07:14.307 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.__get_result() 2022-05-14T01:07:14.307 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T01:07:14.307 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise self._exception 2022-05-14T01:07:14.307 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T01:07:14.307 INFO:journalctl@ceph.mon.b.smithi047.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T01:07:14.308 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T01:07:14.308 INFO:journalctl@ceph.mon.b.smithi047.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T01:07:14.308 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T01:07:14.308 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise 2022-05-14T01:07:14.309 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T01:07:14.309 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.gen.throw(type, value, traceback) 2022-05-14T01:07:14.309 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T01:07:14.309 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OrchestratorError(msg) 2022-05-14T01:07:14.310 INFO:journalctl@ceph.mon.b.smithi047.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:07:14.310 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:07:13 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T01:07:14.310 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:07:13 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T01:07:15.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:07:14 smithi115 conmon[90393]: 2022-05-14T01:07:14.624+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:07:15.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:07:14 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:07:15.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:07:14 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 5965 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:07:15.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:07:14 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:07:15.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:07:14 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 5965 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:07:16.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:07:15 smithi115 conmon[90393]: 2022-05-14T01:07:15.615+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:07:16.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:07:15 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:07:16.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:07:15 smithi115 ceph-mon[78593]: pgmap v3337: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:07:16.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:07:15 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:07:16.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:07:15 smithi047 ceph-mon[77147]: pgmap v3337: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:07:17.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:07:16 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:07:17.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:07:16 smithi115 conmon[90393]: 2022-05-14T01:07:16.659+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:07:17.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:07:16 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:07:18.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:07:17 smithi115 conmon[90393]: 2022-05-14T01:07:17.644+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:07:18.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:07:17 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:07:18.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:07:17 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:07:18.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:07:17 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:07:18.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:07:17 smithi115 ceph-mon[78593]: pgmap v3338: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:07:18.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:07:17 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:07:18.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:07:17 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:07:18.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:07:17 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:07:18.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:07:17 smithi047 ceph-mon[77147]: pgmap v3338: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:07:19.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:07:18 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:07:19.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:07:18 smithi115 conmon[90393]: 2022-05-14T01:07:18.614+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:07:19.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:07:18 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:07:20.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:07:19 smithi115 conmon[90393]: 2022-05-14T01:07:19.629+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:07:20.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:07:19 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:07:20.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:07:19 smithi115 ceph-mon[78593]: pgmap v3339: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:07:20.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:07:19 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 5970 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:07:20.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:07:19 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:07:20.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:07:19 smithi047 ceph-mon[77147]: pgmap v3339: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:07:20.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:07:19 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 5970 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:07:21.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:07:20 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:07:21.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:07:20 smithi115 conmon[90393]: 2022-05-14T01:07:20.670+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:07:21.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:07:20 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:07:22.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:07:21 smithi115 conmon[90393]: 2022-05-14T01:07:21.684+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:07:22.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:07:21 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:07:22.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:07:21 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:07:22.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:07:21 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:07:22.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:07:21 smithi115 ceph-mon[78593]: pgmap v3340: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-05-14T01:07:22.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:07:21 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:07:22.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:07:21 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:07:22.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:07:21 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:07:22.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:07:21 smithi047 ceph-mon[77147]: pgmap v3340: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-05-14T01:07:23.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:07:22 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:07:23.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:07:22 smithi115 conmon[90393]: 2022-05-14T01:07:22.680+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:07:23.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:07:22 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:07:24.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:07:23 smithi115 conmon[90393]: 2022-05-14T01:07:23.713+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:07:24.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:07:23 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:07:24.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:07:23 smithi115 ceph-mon[78593]: pgmap v3341: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:07:24.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:07:23 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:07:24.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:07:23 smithi047 ceph-mon[77147]: pgmap v3341: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:07:25.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:07:24 smithi115 conmon[90393]: 2022-05-14T01:07:24.715+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:07:25.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:07:24 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:07:25.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:07:24 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 5975 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:07:25.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:07:24 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:07:25.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:07:24 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 5975 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:07:26.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:07:25 smithi115 conmon[90393]: 2022-05-14T01:07:25.723+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:07:26.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:07:25 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:07:26.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:07:25 smithi115 ceph-mon[78593]: pgmap v3342: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:07:26.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:07:25 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:07:26.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:07:25 smithi047 ceph-mon[77147]: pgmap v3342: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:07:27.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:07:26 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:07:27.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:07:26 smithi115 conmon[90393]: 2022-05-14T01:07:26.699+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:07:27.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:07:26 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:07:28.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:07:27 smithi115 conmon[90393]: 2022-05-14T01:07:27.664+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:07:28.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:07:27 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:07:28.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:07:27 smithi115 ceph-mon[78593]: pgmap v3343: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:07:28.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:07:27 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:07:28.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:07:27 smithi047 ceph-mon[77147]: pgmap v3343: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:07:28.873 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:07:28 smithi115 conmon[90393]: 2022-05-14T01:07:28.619+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:07:29.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:07:28 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:07:29.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:07:28 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:07:29.881 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:07:29 smithi115 conmon[90393]: 2022-05-14T01:07:29.614+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:07:30.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:07:29 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:07:30.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:07:29 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:07:30.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:07:29 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:07:30.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:07:29 smithi047 ceph-mon[77147]: pgmap v3344: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:07:30.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:07:29 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 5980 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:07:30.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:07:29 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:07:30.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:07:29 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:07:30.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:07:29 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:07:30.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:07:29 smithi115 ceph-mon[78593]: pgmap v3344: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:07:30.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:07:29 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 5980 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:07:30.883 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:07:30 smithi115 conmon[90393]: 2022-05-14T01:07:30.621+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:07:31.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:07:30 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:07:31.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:07:30 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:07:31.887 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:07:31 smithi115 conmon[90393]: 2022-05-14T01:07:31.634+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:07:32.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:07:31 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:07:32.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:07:31 smithi047 ceph-mon[77147]: pgmap v3345: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-05-14T01:07:32.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:07:31 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:07:32.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:07:31 smithi115 ceph-mon[78593]: pgmap v3345: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-05-14T01:07:33.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:07:32 smithi115 conmon[90393]: 2022-05-14T01:07:32.640+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:07:33.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:07:32 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:07:33.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:07:32 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:07:34.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:07:33 smithi115 conmon[90393]: 2022-05-14T01:07:33.653+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:07:34.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:07:33 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:07:34.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:07:33 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:07:34.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:07:33 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:07:34.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:07:33 smithi115 ceph-mon[78593]: pgmap v3346: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:07:34.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:07:33 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:07:34.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:07:33 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:07:34.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:07:33 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:07:34.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:07:33 smithi047 ceph-mon[77147]: pgmap v3346: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:07:35.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:07:34 smithi115 conmon[90393]: 2022-05-14T01:07:34.696+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:07:35.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:07:34 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:07:35.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:07:34 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 5985 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:07:35.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:07:34 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:07:35.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:07:34 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 5985 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:07:36.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:07:35 smithi115 conmon[90393]: 2022-05-14T01:07:35.732+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:07:36.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:07:35 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:07:36.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:07:35 smithi115 ceph-mon[78593]: pgmap v3347: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:07:36.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:07:35 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:07:36.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:07:35 smithi047 ceph-mon[77147]: pgmap v3347: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:07:37.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:07:36 smithi115 conmon[90393]: 2022-05-14T01:07:36.738+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:07:37.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:07:36 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:07:37.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:07:36 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:07:38.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:07:37 smithi115 conmon[90393]: 2022-05-14T01:07:37.778+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:07:38.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:07:37 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:07:38.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:07:37 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:07:38.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:07:37 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:07:38.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:07:37 smithi115 ceph-mon[78593]: pgmap v3348: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:07:38.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:07:37 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:07:38.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:07:37 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:07:38.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:07:37 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:07:38.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:07:37 smithi047 ceph-mon[77147]: pgmap v3348: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:07:39.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:07:38 smithi115 conmon[90393]: 2022-05-14T01:07:38.743+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:07:39.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:07:38 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:07:39.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:07:38 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:07:40.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:07:39 smithi115 conmon[90393]: 2022-05-14T01:07:39.717+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:07:40.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:07:39 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:07:40.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:07:39 smithi115 ceph-mon[78593]: pgmap v3349: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:07:40.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:07:39 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 5990 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:07:40.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:07:39 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:07:40.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:07:39 smithi047 ceph-mon[77147]: pgmap v3349: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:07:40.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:07:39 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 5990 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:07:41.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:07:40 smithi115 conmon[90393]: 2022-05-14T01:07:40.683+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:07:41.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:07:40 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:07:41.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:07:40 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:07:42.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:07:41 smithi115 conmon[90393]: 2022-05-14T01:07:41.731+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:07:42.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:07:41 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:07:42.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:07:41 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:07:42.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:07:41 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:07:42.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:07:41 smithi115 ceph-mon[78593]: pgmap v3350: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-05-14T01:07:42.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:07:41 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:07:42.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:07:41 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:07:42.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:07:41 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:07:42.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:07:41 smithi047 ceph-mon[77147]: pgmap v3350: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-05-14T01:07:43.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:07:42 smithi115 conmon[90393]: 2022-05-14T01:07:42.704+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:07:43.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:07:42 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:07:43.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:07:42 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:07:43.940 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:07:43 smithi115 conmon[90393]: 2022-05-14T01:07:43.684+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:07:44.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:07:43 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:07:44.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:07:43 smithi047 ceph-mon[77147]: pgmap v3351: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:07:44.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:07:43 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:07:44.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:07:43 smithi115 ceph-mon[78593]: pgmap v3351: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:07:45.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:07:44 smithi115 conmon[90393]: 2022-05-14T01:07:44.714+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:07:45.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:07:44 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:07:45.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:07:44 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 5995 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:07:45.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:07:44 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:07:45.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:07:44 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 5995 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:07:46.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:07:45 smithi115 conmon[90393]: 2022-05-14T01:07:45.757+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:07:46.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:07:45 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:07:46.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:07:45 smithi115 ceph-mon[78593]: pgmap v3352: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:07:46.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:07:45 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:07:46.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:07:45 smithi047 ceph-mon[77147]: pgmap v3352: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:07:47.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:07:46 smithi115 conmon[90393]: 2022-05-14T01:07:46.786+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:07:47.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:07:46 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:07:47.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:07:46 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:07:48.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:07:47 smithi115 conmon[90393]: 2022-05-14T01:07:47.831+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:07:48.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:07:47 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:07:48.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:07:47 smithi115 ceph-mon[78593]: pgmap v3353: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:07:48.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:07:47 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:07:48.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:07:47 smithi047 ceph-mon[77147]: pgmap v3353: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:07:49.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:07:48 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:07:49.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:07:48 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:07:49.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:07:48 smithi115 conmon[90393]: 2022-05-14T01:07:48.878+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:07:50.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:07:49 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:07:50.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:07:49 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:07:50.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:07:49 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:07:50.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:07:49 smithi047 ceph-mon[77147]: pgmap v3354: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:07:50.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:07:49 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 6000 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:07:50.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:07:49 smithi115 conmon[90393]: 2022-05-14T01:07:49.864+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:07:50.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:07:49 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:07:50.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:07:49 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:07:50.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:07:49 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:07:50.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:07:49 smithi115 ceph-mon[78593]: pgmap v3354: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:07:50.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:07:49 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 6000 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:07:51.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:07:50 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:07:51.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:07:50 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:07:51.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:07:50 smithi115 conmon[90393]: 2022-05-14T01:07:50.901+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:07:52.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:07:51 smithi115 conmon[90393]: 2022-05-14T01:07:51.853+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:07:52.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:07:51 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:07:52.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:07:51 smithi115 ceph-mon[78593]: pgmap v3355: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-05-14T01:07:52.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:07:51 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:07:52.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:07:51 smithi047 ceph-mon[77147]: pgmap v3355: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-05-14T01:07:53.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:07:52 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:07:53.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:07:52 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:07:53.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:07:52 smithi115 conmon[90393]: 2022-05-14T01:07:52.867+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:07:54.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:07:53 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:07:54.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:07:53 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:07:54.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:07:53 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:07:54.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:07:53 smithi047 ceph-mon[77147]: pgmap v3356: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:07:54.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:07:53 smithi115 conmon[90393]: 2022-05-14T01:07:53.879+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:07:54.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:07:53 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:07:54.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:07:53 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:07:54.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:07:53 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:07:54.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:07:53 smithi115 ceph-mon[78593]: pgmap v3356: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:07:55.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:07:54 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:07:55.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:07:54 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 6005 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:07:55.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:07:54 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:07:55.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:07:54 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 6005 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:07:55.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:07:54 smithi115 conmon[90393]: 2022-05-14T01:07:54.907+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:07:56.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:07:55 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:07:56.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:07:55 smithi047 ceph-mon[77147]: pgmap v3357: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:07:56.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:07:55 smithi115 conmon[90393]: 2022-05-14T01:07:55.929+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:07:56.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:07:55 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:07:56.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:07:55 smithi115 ceph-mon[78593]: pgmap v3357: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:07:57.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:07:56 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:07:57.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:07:56 smithi115 conmon[90393]: 2022-05-14T01:07:56.940+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:07:57.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:07:56 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:07:58.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:07:57 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:07:58.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:07:57 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:07:58.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:07:57 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:07:58.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:07:57 smithi047 ceph-mon[77147]: pgmap v3358: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:07:58.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:07:57 smithi115 conmon[90393]: 2022-05-14T01:07:57.898+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:07:58.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:07:57 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:07:58.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:07:57 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:07:58.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:07:57 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:07:58.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:07:57 smithi115 ceph-mon[78593]: pgmap v3358: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:07:59.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:07:59 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:07:59.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:07:58 smithi115 conmon[90393]: 2022-05-14T01:07:58.911+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:07:59.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:07:59 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:08:00.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:08:00 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:08:00.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:08:00 smithi047 ceph-mon[77147]: pgmap v3359: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:08:00.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:08:00 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 6010 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:08:00.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:07:59 smithi115 conmon[90393]: 2022-05-14T01:07:59.904+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:08:00.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:08:00 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:08:00.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:08:00 smithi115 ceph-mon[78593]: pgmap v3359: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:08:00.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:08:00 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 6010 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:08:01.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:08:01 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:08:01.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:08:00 smithi115 conmon[90393]: 2022-05-14T01:08:00.925+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:08:01.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:08:01 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:08:02.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:08:02 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:08:02.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:08:02 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:08:02.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:08:02 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:08:02.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:08:02 smithi047 ceph-mon[77147]: pgmap v3360: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-05-14T01:08:02.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:08:01 smithi115 conmon[90393]: 2022-05-14T01:08:01.891+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:08:02.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:08:02 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:08:02.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:08:02 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:08:02.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:08:02 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:08:02.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:08:02 smithi115 ceph-mon[78593]: pgmap v3360: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-05-14T01:08:03.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:08:02 smithi115 conmon[90393]: 2022-05-14T01:08:02.845+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:08:03.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:08:03 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:08:03.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:08:03 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:08:04.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:08:04 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:08:04.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:08:04 smithi047 ceph-mon[77147]: pgmap v3361: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:08:04.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:08:03 smithi115 conmon[90393]: 2022-05-14T01:08:03.891+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:08:04.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:08:04 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:08:04.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:08:04 smithi115 ceph-mon[78593]: pgmap v3361: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:08:05.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:08:05 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:08:05.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:08:05 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 6015 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:08:05.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:08:04 smithi115 conmon[90393]: 2022-05-14T01:08:04.919+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:08:05.412 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:08:05 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:08:05.413 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:08:05 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 6015 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:08:06.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:08:06 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:08:06.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:08:06 smithi047 ceph-mon[77147]: pgmap v3362: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:08:06.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:08:05 smithi115 conmon[90393]: 2022-05-14T01:08:05.944+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:08:06.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:08:06 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:08:06.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:08:06 smithi115 ceph-mon[78593]: pgmap v3362: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:08:07.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:08:07 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:08:07.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:08:06 smithi115 conmon[90393]: 2022-05-14T01:08:06.976+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:08:07.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:08:07 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:08:08.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:08:08 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:08:08.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:08:08 smithi047 ceph-mon[77147]: pgmap v3363: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:08:08.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:08:07 smithi115 conmon[90393]: 2022-05-14T01:08:07.995+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:08:08.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:08:08 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:08:08.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:08:08 smithi115 ceph-mon[78593]: pgmap v3363: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:08:09.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:08:09 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:08:09.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:08:08 smithi115 conmon[90393]: 2022-05-14T01:08:08.962+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:08:09.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:08:09 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:08:10.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:08:09 smithi115 conmon[90393]: 2022-05-14T01:08:09.962+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:08:10.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:08:10 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:08:10.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:08:10 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:08:10.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:08:10 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:08:10.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:08:10 smithi115 ceph-mon[78593]: pgmap v3364: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:08:10.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:08:10 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 6020 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:08:10.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:08:10 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:08:10.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:08:10 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:08:10.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:08:10 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:08:10.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:08:10 smithi047 ceph-mon[77147]: pgmap v3364: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:08:10.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:08:10 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 6020 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:08:11.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:08:10 smithi115 conmon[90393]: 2022-05-14T01:08:10.970+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:08:11.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:08:11 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:08:11.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:08:11 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:08:12.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:08:11 smithi115 conmon[90393]: 2022-05-14T01:08:11.926+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:08:12.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:08:12 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:08:12.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:08:12 smithi115 ceph-mon[78593]: pgmap v3365: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-05-14T01:08:12.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:08:12 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:08:12.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:08:12 smithi047 ceph-mon[77147]: pgmap v3365: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-05-14T01:08:13.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:08:12 smithi115 conmon[90393]: 2022-05-14T01:08:12.926+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:08:13.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:08:13 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:08:13.374 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:08:13 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:08:14.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:08:13 smithi115 conmon[90393]: 2022-05-14T01:08:13.942+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:08:14.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:08:14 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:08:14.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:08:14 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:08:14.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:08:14 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:08:14.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:08:14 smithi115 ceph-mon[78593]: pgmap v3366: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:08:14.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:08:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T01:08:14.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:08:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T01:08:14.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:08:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T01:08:14.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:08:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T01:08:14.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:08:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T01:08:14.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:08:14 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:08:14.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:08:14 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:08:14.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:08:14 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:08:14.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:08:14 smithi047 ceph-mon[77147]: pgmap v3366: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:08:14.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:08:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T01:08:14.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:08:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T01:08:14.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:08:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T01:08:14.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:08:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T01:08:14.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:08:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T01:08:15.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:08:14 smithi115 conmon[90393]: 2022-05-14T01:08:14.944+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:08:15.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:08:15 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:08:15.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:08:15 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 6025 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:08:15.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:08:15 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:08:15.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:08:15 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 6025 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:08:16.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:08:15 smithi115 conmon[90393]: 2022-05-14T01:08:15.978+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:08:16.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:08:16 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:08:16.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:08:16 smithi115 ceph-mon[78593]: pgmap v3367: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:08:16.433 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:08:16 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:08:16.433 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:08:16 smithi047 ceph-mon[77147]: pgmap v3367: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:08:16.791 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:08:16 smithi047 conmon[78521]: 2022-05-14T01:08:16.431+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:08:16.791 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:08:16 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T01:08:16.792 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:08:16 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:08:16.792 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:08:16 smithi047 conmon[78521]: yield 2022-05-14T01:08:16.792 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:08:16 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:08:16.793 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:08:16 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:08:16.793 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:08:16 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:08:16.793 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:08:16 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T01:08:16.793 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:08:16 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:08:16.793 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:08:16 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T01:08:16.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:08:16 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:08:16.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:08:16 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T01:08:16.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:08:16 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:08:16.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:08:16 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T01:08:16.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:08:16 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:08:16.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:08:16 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T01:08:16.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:08:16 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:08:16.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:08:16 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:08:16.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:08:16 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:08:17.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:08:16 smithi115 conmon[90393]: 2022-05-14T01:08:16.935+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:08:17.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:08:17 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:08:17.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:08:17 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:08:17.358 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T01:08:17.358 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:08:17.359 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T01:08:17.359 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:08:17.359 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:08:17.359 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:08:17.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T01:08:17.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:08:17.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T01:08:17.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:08:17.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T01:08:17.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:08:17.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T01:08:17.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:08:17.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T01:08:17.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:08:17.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:08:17.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:08:17.362 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:08:17 smithi115 ceph-mon[78593]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T01:08:17.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:08:17 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:08:17.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:08:17 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:08:17.542 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T01:08:17.542 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:08:17.542 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T01:08:17.542 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:08:17.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:08:17.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:08:17.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T01:08:17.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:08:17.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T01:08:17.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:08:17.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T01:08:17.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:08:17.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T01:08:17.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:08:17.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T01:08:17.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:08:17.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:08:17.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:08:17.546 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:08:17 smithi047 ceph-mon[77147]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T01:08:18.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:08:17 smithi115 conmon[90393]: 2022-05-14T01:08:17.910+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:08:18.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:08:18 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:08:18.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:08:18 smithi115 ceph-mon[78593]: pgmap v3368: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:08:18.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:08:18 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:08:18.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:08:18 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:08:18.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:08:18 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:08:18.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:08:18 smithi047 ceph-mon[77147]: pgmap v3368: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:08:18.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:08:18 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:08:18.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:08:18 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:08:19.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:08:18 smithi115 conmon[90393]: 2022-05-14T01:08:18.937+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:08:19.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:08:19 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:08:19.505 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:08:19 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:08:19.792 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:08:19 smithi047 conmon[78521]: 2022-05-14T01:08:19.503+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:08:19.792 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:08:19 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T01:08:19.793 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:08:19 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:08:19.793 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:08:19 smithi047 conmon[78521]: yield 2022-05-14T01:08:19.793 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:08:19 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:08:19.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:08:19 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:08:19.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:08:19 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:08:19.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:08:19 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T01:08:19.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:08:19 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:08:19.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:08:19 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T01:08:19.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:08:19 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:08:19.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:08:19 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T01:08:19.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:08:19 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:08:19.797 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:08:19 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T01:08:19.797 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:08:19 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:08:19.797 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:08:19 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T01:08:19.797 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:08:19 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:08:19.798 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:08:19 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:08:19.798 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:08:19 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:08:19.798 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:08:19 smithi047 conmon[78521]: 2022-05-14T01:08:19.504+0000 7f5344781700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T01:08:19.798 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:08:19 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T01:08:19.798 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:08:19 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:08:19.799 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:08:19 smithi047 conmon[78521]: yield 2022-05-14T01:08:19.799 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:08:19 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:08:19.799 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:08:19 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:08:19.800 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:08:19 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:08:19.800 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:08:19 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T01:08:19.800 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:08:19 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:08:19.800 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:08:19 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T01:08:19.801 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:08:19 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:08:19.801 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:08:19 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T01:08:19.801 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:08:19 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:08:19.801 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:08:19 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T01:08:19.802 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:08:19 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:08:19.802 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:08:19 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T01:08:19.802 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:08:19 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:08:19.802 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:08:19 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:08:19.803 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:08:19 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:08:19.803 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:08:19 smithi047 conmon[78521]: 2022-05-14T01:08:19.803 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:08:19 smithi047 conmon[78521]: During handling of the above exception, another exception occurred: 2022-05-14T01:08:19.803 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:08:19 smithi047 conmon[78521]: 2022-05-14T01:08:19.803 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:08:19 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T01:08:19.804 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:08:19 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T01:08:19.804 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:08:19 smithi047 conmon[78521]: return f(*arg) 2022-05-14T01:08:19.804 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:08:19 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T01:08:19.805 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:08:19 smithi047 conmon[78521]: self._write_client_files(client_files, host) 2022-05-14T01:08:19.805 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:08:19 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T01:08:19.805 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:08:19 smithi047 conmon[78521]: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T01:08:19.805 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:08:19 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T01:08:19.806 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:08:19 smithi047 conmon[78521]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T01:08:19.806 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:08:19 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T01:08:19.806 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:08:19 smithi047 conmon[78521]: return self.event_loop.get_result(coro) 2022-05-14T01:08:19.806 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:08:19 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T01:08:19.807 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:08:19 smithi047 conmon[78521]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T01:08:19.807 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:08:19 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T01:08:19.807 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:08:19 smithi047 conmon[78521]: return self.__get_result() 2022-05-14T01:08:19.807 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:08:19 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T01:08:19.808 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:08:19 smithi047 conmon[78521]: raise self._exception 2022-05-14T01:08:19.808 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:08:19 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T01:08:19.808 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:08:19 smithi047 conmon[78521]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T01:08:19.808 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:08:19 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T01:08:19.808 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:08:19 smithi047 conmon[78521]: conn = await self._remote_connection(host, addr) 2022-05-14T01:08:19.809 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:08:19 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T01:08:19.809 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:08:19 smithi047 conmon[78521]: raise 2022-05-14T01:08:19.809 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:08:19 smithi047 conmon[78521]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T01:08:19.809 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:08:19 smithi047 conmon[78521]: self.gen.throw(type, value, traceback) 2022-05-14T01:08:19.810 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:08:19 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T01:08:19.810 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:08:19 smithi047 conmon[78521]: raise OrchestratorError(msg) 2022-05-14T01:08:19.810 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:08:19 smithi047 conmon[78521]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:08:20.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:08:19 smithi115 conmon[90393]: 2022-05-14T01:08:19.937+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:08:20.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:08:20 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:08:20.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:08:20 smithi115 ceph-mon[78593]: pgmap v3369: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:08:20.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:08:20 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 6030 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:08:20.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:08:20 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:08:20.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:08:20 smithi047 ceph-mon[77147]: pgmap v3369: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:08:20.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:08:20 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 6030 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:08:21.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:08:20 smithi115 conmon[90393]: 2022-05-14T01:08:20.918+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:08:21.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:08:21 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:08:21.358 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T01:08:21.358 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:08:21.359 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T01:08:21.359 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:08:21.359 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:08:21.359 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:08:21.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T01:08:21.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:08:21.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T01:08:21.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:08:21.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T01:08:21.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:08:21.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T01:08:21.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:08:21.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T01:08:21.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:08:21.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:08:21.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:08:21.362 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:08:21 smithi115 ceph-mon[78593]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T01:08:21.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T01:08:21.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:08:21.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T01:08:21.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:08:21.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:08:21.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:08:21.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T01:08:21.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:08:21.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T01:08:21.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:08:21.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T01:08:21.365 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:08:21.365 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T01:08:21.365 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:08:21.365 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T01:08:21.365 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:08:21.366 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:08:21.366 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:08:21.366 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T01:08:21.366 INFO:journalctl@ceph.mon.c.smithi115.stdout: During handling of the above exception, another exception occurred: 2022-05-14T01:08:21.367 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T01:08:21.367 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T01:08:21.367 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T01:08:21.367 INFO:journalctl@ceph.mon.c.smithi115.stdout: return f(*arg) 2022-05-14T01:08:21.367 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T01:08:21.368 INFO:journalctl@ceph.mon.c.smithi115.stdout: self._write_client_files(client_files, host) 2022-05-14T01:08:21.368 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T01:08:21.368 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T01:08:21.368 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T01:08:21.368 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T01:08:21.369 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T01:08:21.369 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.event_loop.get_result(coro) 2022-05-14T01:08:21.369 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T01:08:21.369 INFO:journalctl@ceph.mon.c.smithi115.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T01:08:21.369 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T01:08:21.370 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.__get_result() 2022-05-14T01:08:21.370 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T01:08:21.370 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise self._exception 2022-05-14T01:08:21.370 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T01:08:21.371 INFO:journalctl@ceph.mon.c.smithi115.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T01:08:21.371 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T01:08:21.371 INFO:journalctl@ceph.mon.c.smithi115.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T01:08:21.371 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T01:08:21.371 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise 2022-05-14T01:08:21.372 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T01:08:21.372 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.gen.throw(type, value, traceback) 2022-05-14T01:08:21.372 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T01:08:21.372 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OrchestratorError(msg) 2022-05-14T01:08:21.372 INFO:journalctl@ceph.mon.c.smithi115.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:08:21.373 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:08:21 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:08:21.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:08:21 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:08:21.541 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T01:08:21.541 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:08:21.542 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T01:08:21.542 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:08:21.542 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:08:21.542 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:08:21.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T01:08:21.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:08:21.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T01:08:21.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:08:21.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T01:08:21.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:08:21.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T01:08:21.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:08:21.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T01:08:21.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:08:21.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:08:21.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:08:21.545 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:08:21 smithi047 ceph-mon[77147]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T01:08:21.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T01:08:21.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:08:21.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T01:08:21.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:08:21.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:08:21.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:08:21.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T01:08:21.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:08:21.548 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T01:08:21.548 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:08:21.548 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T01:08:21.548 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:08:21.549 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T01:08:21.549 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:08:21.549 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T01:08:21.549 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:08:21.549 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:08:21.550 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:08:21.550 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T01:08:21.550 INFO:journalctl@ceph.mon.b.smithi047.stdout: During handling of the above exception, another exception occurred: 2022-05-14T01:08:21.550 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T01:08:21.550 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T01:08:21.551 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T01:08:21.551 INFO:journalctl@ceph.mon.b.smithi047.stdout: return f(*arg) 2022-05-14T01:08:21.551 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T01:08:21.551 INFO:journalctl@ceph.mon.b.smithi047.stdout: self._write_client_files(client_files, host) 2022-05-14T01:08:21.552 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T01:08:21.552 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T01:08:21.552 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T01:08:21.552 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T01:08:21.552 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T01:08:21.553 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.event_loop.get_result(coro) 2022-05-14T01:08:21.553 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T01:08:21.553 INFO:journalctl@ceph.mon.b.smithi047.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T01:08:21.553 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T01:08:21.553 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.__get_result() 2022-05-14T01:08:21.554 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T01:08:21.554 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise self._exception 2022-05-14T01:08:21.554 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T01:08:21.554 INFO:journalctl@ceph.mon.b.smithi047.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T01:08:21.555 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T01:08:21.555 INFO:journalctl@ceph.mon.b.smithi047.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T01:08:21.555 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T01:08:21.555 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise 2022-05-14T01:08:21.555 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T01:08:21.556 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.gen.throw(type, value, traceback) 2022-05-14T01:08:21.556 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T01:08:21.556 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OrchestratorError(msg) 2022-05-14T01:08:21.556 INFO:journalctl@ceph.mon.b.smithi047.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:08:21.556 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:08:21 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:08:22.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:08:21 smithi115 conmon[90393]: 2022-05-14T01:08:21.888+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:08:22.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:08:22 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:08:22.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:08:22 smithi115 ceph-mon[78593]: pgmap v3370: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-05-14T01:08:22.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:08:22 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:08:22.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:08:22 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:08:22.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:08:22 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:08:22.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:08:22 smithi047 ceph-mon[77147]: pgmap v3370: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-05-14T01:08:22.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:08:22 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:08:22.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:08:22 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:08:23.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:08:23 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:08:23.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:08:22 smithi115 conmon[90393]: 2022-05-14T01:08:22.916+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:08:23.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:08:23 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:08:24.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:08:23 smithi115 conmon[90393]: 2022-05-14T01:08:23.937+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:08:24.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:08:24 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:08:24.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:08:24 smithi115 ceph-mon[78593]: pgmap v3371: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:08:24.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:08:24 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:08:24.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:08:24 smithi047 ceph-mon[77147]: pgmap v3371: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:08:25.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:08:24 smithi115 conmon[90393]: 2022-05-14T01:08:24.972+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:08:25.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:08:25 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:08:25.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:08:25 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 6035 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:08:25.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:08:25 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:08:25.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:08:25 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 6035 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:08:26.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:08:26 smithi115 conmon[90393]: 2022-05-14T01:08:26.014+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:08:26.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:08:26 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:08:26.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:08:26 smithi115 ceph-mon[78593]: pgmap v3372: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:08:26.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:08:26 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:08:26.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:08:26 smithi047 ceph-mon[77147]: pgmap v3372: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:08:27.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:08:27 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:08:27.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:08:27 smithi115 conmon[90393]: 2022-05-14T01:08:27.049+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:08:27.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:08:27 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:08:28.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:08:28 smithi115 conmon[90393]: 2022-05-14T01:08:28.029+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:08:28.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:08:28 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:08:28.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:08:28 smithi115 ceph-mon[78593]: pgmap v3373: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:08:28.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:08:28 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:08:28.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:08:28 smithi047 ceph-mon[77147]: pgmap v3373: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:08:29.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:08:29 smithi115 conmon[90393]: 2022-05-14T01:08:29.010+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:08:29.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:08:29 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:08:29.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:08:29 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:08:30.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:08:29 smithi115 conmon[90393]: 2022-05-14T01:08:29.997+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:08:30.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:08:30 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:08:30.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:08:30 smithi115 ceph-mon[78593]: pgmap v3374: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:08:30.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:08:30 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:08:30.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:08:30 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:08:30.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:08:30 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 6040 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:08:30.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:08:30 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:08:30.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:08:30 smithi047 ceph-mon[77147]: pgmap v3374: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:08:30.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:08:30 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:08:30.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:08:30 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:08:30.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:08:30 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 6040 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:08:31.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:08:30 smithi115 conmon[90393]: 2022-05-14T01:08:30.957+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:08:31.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:08:31 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:08:31.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:08:31 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:08:32.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:08:31 smithi115 conmon[90393]: 2022-05-14T01:08:31.953+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:08:32.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:08:32 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:08:32.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:08:32 smithi115 ceph-mon[78593]: pgmap v3375: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-05-14T01:08:32.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:08:32 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:08:32.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:08:32 smithi047 ceph-mon[77147]: pgmap v3375: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-05-14T01:08:33.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:08:32 smithi115 conmon[90393]: 2022-05-14T01:08:32.930+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:08:33.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:08:33 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:08:33.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:08:33 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:08:34.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:08:33 smithi115 conmon[90393]: 2022-05-14T01:08:33.959+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:08:34.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:08:34 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:08:34.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:08:34 smithi115 ceph-mon[78593]: pgmap v3376: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:08:34.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:08:34 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:08:34.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:08:34 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:08:34.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:08:34 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:08:34.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:08:34 smithi047 ceph-mon[77147]: pgmap v3376: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:08:34.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:08:34 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:08:34.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:08:34 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:08:35.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:08:34 smithi115 conmon[90393]: 2022-05-14T01:08:34.945+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:08:35.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:08:35 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:08:35.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:08:35 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 6045 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:08:35.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:08:35 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:08:35.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:08:35 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 6045 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:08:36.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:08:35 smithi115 conmon[90393]: 2022-05-14T01:08:35.932+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:08:36.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:08:36 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:08:36.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:08:36 smithi115 ceph-mon[78593]: pgmap v3377: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:08:36.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:08:36 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:08:36.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:08:36 smithi047 ceph-mon[77147]: pgmap v3377: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:08:37.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:08:36 smithi115 conmon[90393]: 2022-05-14T01:08:36.898+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:08:37.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:08:37 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:08:37.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:08:37 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:08:38.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:08:37 smithi115 conmon[90393]: 2022-05-14T01:08:37.905+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:08:38.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:08:38 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:08:38.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:08:38 smithi115 ceph-mon[78593]: pgmap v3378: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:08:38.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:08:38 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:08:38.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:08:38 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:08:38.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:08:38 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:08:38.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:08:38 smithi047 ceph-mon[77147]: pgmap v3378: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:08:38.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:08:38 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:08:38.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:08:38 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:08:39.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:08:38 smithi115 conmon[90393]: 2022-05-14T01:08:38.948+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:08:39.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:08:39 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:08:39.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:08:39 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:08:40.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:08:39 smithi115 conmon[90393]: 2022-05-14T01:08:39.971+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:08:40.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:08:40 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:08:40.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:08:40 smithi115 ceph-mon[78593]: pgmap v3379: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 6 op/s 2022-05-14T01:08:40.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:08:40 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 6050 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:08:40.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:08:40 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:08:40.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:08:40 smithi047 ceph-mon[77147]: pgmap v3379: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 6 op/s 2022-05-14T01:08:40.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:08:40 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 6050 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:08:41.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:08:40 smithi115 conmon[90393]: 2022-05-14T01:08:40.945+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:08:41.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:08:41 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:08:41.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:08:41 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:08:42.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:08:41 smithi115 conmon[90393]: 2022-05-14T01:08:41.922+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:08:42.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:08:42 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:08:42.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:08:42 smithi115 ceph-mon[78593]: pgmap v3380: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 4 op/s 2022-05-14T01:08:42.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:08:42 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:08:42.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:08:42 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:08:42.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:08:42 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:08:42.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:08:42 smithi047 ceph-mon[77147]: pgmap v3380: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 4 op/s 2022-05-14T01:08:42.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:08:42 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:08:42.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:08:42 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:08:43.161 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:08:42 smithi115 conmon[90393]: 2022-05-14T01:08:42.892+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:08:43.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:08:43 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:08:43.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:08:43 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:08:44.163 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:08:43 smithi115 conmon[90393]: 2022-05-14T01:08:43.904+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:08:44.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:08:44 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:08:44.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:08:44 smithi047 ceph-mon[77147]: pgmap v3381: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 4 op/s 2022-05-14T01:08:44.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:08:44 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:08:44.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:08:44 smithi115 ceph-mon[78593]: pgmap v3381: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 4 op/s 2022-05-14T01:08:45.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:08:44 smithi115 conmon[90393]: 2022-05-14T01:08:44.934+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:08:45.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:08:45 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:08:45.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:08:45 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 6055 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:08:45.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:08:45 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:08:45.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:08:45 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 6055 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:08:46.166 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:08:45 smithi115 conmon[90393]: 2022-05-14T01:08:45.907+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:08:46.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:08:46 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:08:46.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:08:46 smithi047 ceph-mon[77147]: pgmap v3382: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 7 op/s 2022-05-14T01:08:46.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:08:46 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:08:46.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:08:46 smithi115 ceph-mon[78593]: pgmap v3382: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 7 op/s 2022-05-14T01:08:47.171 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:08:46 smithi115 conmon[90393]: 2022-05-14T01:08:46.897+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:08:47.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:08:47 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:08:47.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:08:47 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:08:48.172 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:08:47 smithi115 conmon[90393]: 2022-05-14T01:08:47.907+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:08:48.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:08:48 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:08:48.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:08:48 smithi047 ceph-mon[77147]: pgmap v3383: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 4 op/s 2022-05-14T01:08:48.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:08:48 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:08:48.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:08:48 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:08:48.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:08:48 smithi115 ceph-mon[78593]: pgmap v3383: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 4 op/s 2022-05-14T01:08:48.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:08:48 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:08:49.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:08:48 smithi115 conmon[90393]: 2022-05-14T01:08:48.951+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:08:49.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:08:49 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:08:49.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:08:49 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:08:50.179 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:08:49 smithi115 conmon[90393]: 2022-05-14T01:08:49.911+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:08:50.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:08:50 smithi047 ceph-mon[77147]: pgmap v3384: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 6 op/s 2022-05-14T01:08:50.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:08:50 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:08:50.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:08:50 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:08:50.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:08:50 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 6060 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:08:50.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:08:50 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:08:50.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:08:50 smithi115 ceph-mon[78593]: pgmap v3384: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 6 op/s 2022-05-14T01:08:50.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:08:50 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:08:50.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:08:50 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:08:50.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:08:50 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 6060 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:08:50.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:08:50 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:08:51.186 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:08:50 smithi115 conmon[90393]: 2022-05-14T01:08:50.897+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:08:51.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:08:51 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:08:51.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:08:51 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:08:52.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:08:52 smithi115 ceph-mon[78593]: pgmap v3385: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-05-14T01:08:52.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:08:52 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:08:52.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:08:51 smithi115 conmon[90393]: 2022-05-14T01:08:51.936+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:08:52.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:08:52 smithi047 ceph-mon[77147]: pgmap v3385: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-05-14T01:08:52.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:08:52 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:08:53.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:08:52 smithi115 conmon[90393]: 2022-05-14T01:08:52.952+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:08:53.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:08:53 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:08:53.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:08:53 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:08:54.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:08:53 smithi115 conmon[90393]: 2022-05-14T01:08:53.959+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:08:54.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:08:54 smithi115 ceph-mon[78593]: pgmap v3386: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:08:54.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:08:54 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:08:54.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:08:54 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:08:54.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:08:54 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:08:54.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:08:54 smithi047 ceph-mon[77147]: pgmap v3386: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:08:54.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:08:54 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:08:54.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:08:54 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:08:54.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:08:54 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:08:55.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:08:55 smithi115 conmon[90393]: 2022-05-14T01:08:55.004+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:08:55.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:08:55 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 6065 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:08:55.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:08:55 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:08:55.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:08:55 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 6065 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:08:55.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:08:55 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:08:56.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:08:56 smithi115 conmon[90393]: 2022-05-14T01:08:56.004+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:08:56.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:08:56 smithi115 ceph-mon[78593]: pgmap v3387: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:08:56.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:08:56 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:08:56.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:08:56 smithi047 ceph-mon[77147]: pgmap v3387: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:08:56.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:08:56 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:08:57.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:08:57 smithi115 conmon[90393]: 2022-05-14T01:08:57.003+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:08:57.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:08:57 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:08:57.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:08:57 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:08:58.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:08:57 smithi115 conmon[90393]: 2022-05-14T01:08:57.959+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:08:58.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:08:58 smithi115 ceph-mon[78593]: pgmap v3388: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:08:58.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:08:58 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:08:58.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:08:58 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:08:58.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:08:58 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:08:58.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:08:58 smithi047 ceph-mon[77147]: pgmap v3388: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:08:58.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:08:58 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:08:58.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:08:58 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:08:58.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:08:58 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:08:59.207 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:08:58 smithi115 conmon[90393]: 2022-05-14T01:08:58.938+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:08:59.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:08:59 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:08:59.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:08:59 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:09:00.208 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:08:59 smithi115 conmon[90393]: 2022-05-14T01:08:59.907+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:09:00.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:09:00 smithi047 ceph-mon[77147]: pgmap v3389: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:09:00.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:09:00 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 6070 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:09:00.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:09:00 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:09:00.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:09:00 smithi115 ceph-mon[78593]: pgmap v3389: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:09:00.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:09:00 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 6070 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:09:00.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:09:00 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:09:01.213 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:09:00 smithi115 conmon[90393]: 2022-05-14T01:09:00.941+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:09:01.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:09:01 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:09:01.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:09:01 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:09:02.214 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:09:01 smithi115 conmon[90393]: 2022-05-14T01:09:01.904+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:09:02.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:09:02 smithi047 ceph-mon[77147]: pgmap v3390: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-05-14T01:09:02.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:09:02 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:09:02.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:09:02 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:09:02.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:09:02 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:09:02.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:09:02 smithi115 ceph-mon[78593]: pgmap v3390: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-05-14T01:09:02.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:09:02 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:09:02.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:09:02 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:09:02.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:09:02 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:09:03.219 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:09:02 smithi115 conmon[90393]: 2022-05-14T01:09:02.931+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:09:03.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:09:03 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:09:03.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:09:03 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:09:04.220 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:09:03 smithi115 conmon[90393]: 2022-05-14T01:09:03.939+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:09:04.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:09:04 smithi047 ceph-mon[77147]: pgmap v3391: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:09:04.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:09:04 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:09:04.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:09:04 smithi115 ceph-mon[78593]: pgmap v3391: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:09:04.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:09:04 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:09:05.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:09:04 smithi115 conmon[90393]: 2022-05-14T01:09:04.988+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:09:05.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:09:05 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 6075 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:09:05.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:09:05 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:09:05.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:09:05 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 6075 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:09:05.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:09:05 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:09:06.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:09:05 smithi115 conmon[90393]: 2022-05-14T01:09:05.982+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:09:06.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:09:06 smithi115 ceph-mon[78593]: pgmap v3392: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:09:06.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:09:06 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:09:06.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:09:06 smithi047 ceph-mon[77147]: pgmap v3392: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:09:06.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:09:06 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:09:07.228 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:09:06 smithi115 conmon[90393]: 2022-05-14T01:09:06.941+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:09:07.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:09:07 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:09:07.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:09:07 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:09:08.229 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:09:07 smithi115 conmon[90393]: 2022-05-14T01:09:07.896+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:09:08.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:09:08 smithi047 ceph-mon[77147]: pgmap v3393: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:09:08.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:09:08 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:09:08.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:09:08 smithi115 ceph-mon[78593]: pgmap v3393: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:09:08.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:09:08 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:09:09.239 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:09:08 smithi115 conmon[90393]: 2022-05-14T01:09:08.943+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:09:09.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:09:09 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:09:09.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:09:09 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:09:10.240 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:09:09 smithi115 conmon[90393]: 2022-05-14T01:09:09.923+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:09:10.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:09:10 smithi047 ceph-mon[77147]: pgmap v3394: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:09:10.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:09:10 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:09:10.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:09:10 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:09:10.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:09:10 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 6080 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:09:10.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:09:10 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:09:10.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:09:10 smithi115 ceph-mon[78593]: pgmap v3394: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:09:10.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:09:10 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:09:10.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:09:10 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:09:10.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:09:10 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 6080 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:09:10.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:09:10 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:09:11.245 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:09:10 smithi115 conmon[90393]: 2022-05-14T01:09:10.932+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:09:11.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:09:11 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:09:11.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:09:11 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:09:12.247 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:09:11 smithi115 conmon[90393]: 2022-05-14T01:09:11.976+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:09:12.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:09:12 smithi047 ceph-mon[77147]: pgmap v3395: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-05-14T01:09:12.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:09:12 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:09:12.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:09:12 smithi115 ceph-mon[78593]: pgmap v3395: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-05-14T01:09:12.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:09:12 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:09:13.253 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:09:12 smithi115 conmon[90393]: 2022-05-14T01:09:12.961+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:09:13.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:09:13 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:09:13.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:09:13 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:09:14.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:09:14 smithi115 conmon[90393]: 2022-05-14T01:09:14.008+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:09:14.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:09:14 smithi115 ceph-mon[78593]: pgmap v3396: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:09:14.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:09:14 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:09:14.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:09:14 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:09:14.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:09:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T01:09:14.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:09:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T01:09:14.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:09:14 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:09:14.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:09:14 smithi047 ceph-mon[77147]: pgmap v3396: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:09:14.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:09:14 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:09:14.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:09:14 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:09:14.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:09:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T01:09:14.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:09:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T01:09:14.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:09:14 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:09:15.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:09:15 smithi115 conmon[90393]: 2022-05-14T01:09:15.015+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:09:15.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:09:15 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 6085 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:09:15.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:09:15 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:09:15.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:09:15 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 6085 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:09:15.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:09:15 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:09:16.258 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:09:15 smithi115 conmon[90393]: 2022-05-14T01:09:15.966+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:09:16.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:09:16 smithi047 ceph-mon[77147]: pgmap v3397: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:09:16.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:09:16 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:09:16.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:09:16 smithi115 ceph-mon[78593]: pgmap v3397: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:09:16.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:09:16 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:09:17.263 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:09:16 smithi115 conmon[90393]: 2022-05-14T01:09:16.963+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:09:17.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:09:17 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:09:17.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:09:17 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:09:18.265 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:09:17 smithi115 conmon[90393]: 2022-05-14T01:09:17.946+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:09:18.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:09:18 smithi047 ceph-mon[77147]: pgmap v3398: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:09:18.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:09:18 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:09:18.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:09:18 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:09:18.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:09:18 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:09:18.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:09:18 smithi115 ceph-mon[78593]: pgmap v3398: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:09:18.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:09:18 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:09:18.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:09:18 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:09:18.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:09:18 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:09:19.269 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:09:18 smithi115 conmon[90393]: 2022-05-14T01:09:18.920+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:09:19.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:09:19 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:09:19.560 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:09:19 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:09:20.270 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:09:19 smithi047 conmon[78521]: 2022-05-14T01:09:19.984+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:09:20.271 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:09:19 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T01:09:20.271 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:09:19 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:09:20.272 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:09:19 smithi047 conmon[78521]: yield 2022-05-14T01:09:20.272 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:09:19 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:09:20.272 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:09:19 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:09:20.273 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:09:19 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:09:20.273 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:09:19 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T01:09:20.273 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:09:19 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:09:20.274 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:09:19 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T01:09:20.274 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:09:19 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:09:20.274 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:09:19 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T01:09:20.275 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:09:19 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:09:20.275 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:09:19 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T01:09:20.275 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:09:19 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:09:20.276 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:09:19 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T01:09:20.276 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:09:19 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:09:20.276 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:09:19 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:09:20.277 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:09:19 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:09:20.278 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:09:19 smithi115 conmon[90393]: 2022-05-14T01:09:19.901+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:09:20.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:09:20 smithi047 ceph-mon[77147]: pgmap v3399: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:09:20.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:09:20 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T01:09:20.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:09:20 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T01:09:20.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:09:20 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T01:09:20.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:09:20 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 6090 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:09:20.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:09:20 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:09:20.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:09:20 smithi115 ceph-mon[78593]: pgmap v3399: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:09:20.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:09:20 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T01:09:20.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:09:20 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T01:09:20.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:09:20 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T01:09:20.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:09:20 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 6090 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:09:20.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:09:20 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:09:21.272 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:09:20 smithi115 conmon[90393]: 2022-05-14T01:09:20.863+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:09:21.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:09:21 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:09:21.541 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T01:09:21.541 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:09:21.542 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T01:09:21.542 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:09:21.542 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:09:21.542 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:09:21.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T01:09:21.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:09:21.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T01:09:21.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:09:21.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T01:09:21.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:09:21.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T01:09:21.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:09:21.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T01:09:21.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:09:21.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:09:21.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:09:21.546 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:09:21 smithi047 ceph-mon[77147]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T01:09:21.546 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:09:21 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:09:21.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:09:21 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:09:21.607 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T01:09:21.608 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:09:21.608 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T01:09:21.608 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:09:21.608 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:09:21.609 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:09:21.609 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T01:09:21.609 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:09:21.609 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T01:09:21.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:09:21.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T01:09:21.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:09:21.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T01:09:21.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:09:21.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T01:09:21.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:09:21.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:09:21.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:09:21.613 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:09:21 smithi115 ceph-mon[78593]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T01:09:21.613 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:09:21 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:09:22.278 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:09:21 smithi115 conmon[90393]: 2022-05-14T01:09:21.870+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:09:22.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:09:22 smithi047 ceph-mon[77147]: pgmap v3400: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-05-14T01:09:22.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:09:22 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:09:22.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:09:22 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:09:22.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:09:22 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:09:22.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:09:22 smithi115 ceph-mon[78593]: pgmap v3400: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-05-14T01:09:22.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:09:22 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:09:22.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:09:22 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:09:22.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:09:22 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:09:23.282 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:09:22 smithi115 conmon[90393]: 2022-05-14T01:09:22.919+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:09:23.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:09:23 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:09:23.541 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:09:23 smithi047 conmon[78521]: 2022-05-14T01:09:23.055+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:09:23.542 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:09:23 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T01:09:23.542 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:09:23 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:09:23.542 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:09:23 smithi047 conmon[78521]: yield 2022-05-14T01:09:23.542 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:09:23 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:09:23.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:09:23 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:09:23.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:09:23 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:09:23.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:09:23 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T01:09:23.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:09:23 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:09:23.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:09:23 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T01:09:23.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:09:23 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:09:23.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:09:23 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T01:09:23.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:09:23 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:09:23.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:09:23 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T01:09:23.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:09:23 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:09:23.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:09:23 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T01:09:23.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:09:23 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:09:23.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:09:23 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:09:23.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:09:23 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:09:23.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:09:23 smithi047 conmon[78521]: 2022-05-14T01:09:23.056+0000 7f5345f84700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T01:09:23.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:09:23 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T01:09:23.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:09:23 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:09:23.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:09:23 smithi047 conmon[78521]: yield 2022-05-14T01:09:23.547 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:09:23 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:09:23.547 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:09:23 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:09:23.547 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:09:23 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:09:23.547 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:09:23 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T01:09:23.548 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:09:23 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:09:23.548 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:09:23 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T01:09:23.548 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:09:23 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:09:23.548 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:09:23 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T01:09:23.548 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:09:23 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:09:23.549 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:09:23 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T01:09:23.549 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:09:23 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:09:23.549 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:09:23 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T01:09:23.549 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:09:23 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:09:23.550 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:09:23 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:09:23.550 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:09:23 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:09:23.550 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:09:23 smithi047 conmon[78521]: 2022-05-14T01:09:23.550 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:09:23 smithi047 conmon[78521]: During handling of the above exception, another exception occurred: 2022-05-14T01:09:23.550 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:09:23 smithi047 conmon[78521]: 2022-05-14T01:09:23.551 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:09:23 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T01:09:23.551 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:09:23 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T01:09:23.551 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:09:23 smithi047 conmon[78521]: return f(*arg) 2022-05-14T01:09:23.551 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:09:23 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T01:09:23.552 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:09:23 smithi047 conmon[78521]: self._write_client_files(client_files, host) 2022-05-14T01:09:23.552 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:09:23 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T01:09:23.552 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:09:23 smithi047 conmon[78521]: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T01:09:23.552 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:09:23 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T01:09:23.552 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:09:23 smithi047 conmon[78521]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T01:09:23.553 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:09:23 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T01:09:23.553 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:09:23 smithi047 conmon[78521]: return self.event_loop.get_result(coro) 2022-05-14T01:09:23.553 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:09:23 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T01:09:23.553 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:09:23 smithi047 conmon[78521]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T01:09:23.554 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:09:23 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T01:09:23.554 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:09:23 smithi047 conmon[78521]: return self.__get_result() 2022-05-14T01:09:23.554 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:09:23 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T01:09:23.554 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:09:23 smithi047 conmon[78521]: raise self._exception 2022-05-14T01:09:23.554 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:09:23 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T01:09:23.555 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:09:23 smithi047 conmon[78521]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T01:09:23.555 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:09:23 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T01:09:23.555 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:09:23 smithi047 conmon[78521]: conn = await self._remote_connection(host, addr) 2022-05-14T01:09:23.555 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:09:23 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T01:09:23.556 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:09:23 smithi047 conmon[78521]: raise 2022-05-14T01:09:23.556 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:09:23 smithi047 conmon[78521]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T01:09:23.556 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:09:23 smithi047 conmon[78521]: self.gen.throw(type, value, traceback) 2022-05-14T01:09:23.556 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:09:23 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T01:09:23.556 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:09:23 smithi047 conmon[78521]: raise OrchestratorError(msg) 2022-05-14T01:09:23.557 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:09:23 smithi047 conmon[78521]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:09:23.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:09:23 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:09:24.284 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:09:23 smithi115 conmon[90393]: 2022-05-14T01:09:23.911+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:09:24.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:09:24 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:09:24.541 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T01:09:24.542 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:09:24.542 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T01:09:24.542 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:09:24.542 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:09:24.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:09:24.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T01:09:24.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:09:24.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T01:09:24.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:09:24.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T01:09:24.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:09:24.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T01:09:24.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:09:24.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T01:09:24.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:09:24.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:09:24.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:09:24.547 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:09:24 smithi047 ceph-mon[77147]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T01:09:24.548 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T01:09:24.548 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:09:24.549 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T01:09:24.549 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:09:24.550 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:09:24.550 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:09:24.550 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T01:09:24.551 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:09:24.551 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T01:09:24.552 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:09:24.552 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T01:09:24.553 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:09:24.554 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T01:09:24.554 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:09:24.555 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T01:09:24.555 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:09:24.556 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:09:24.556 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:09:24.557 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T01:09:24.557 INFO:journalctl@ceph.mon.b.smithi047.stdout: During handling of the above exception, another exception occurred: 2022-05-14T01:09:24.557 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T01:09:24.557 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T01:09:24.558 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T01:09:24.558 INFO:journalctl@ceph.mon.b.smithi047.stdout: return f(*arg) 2022-05-14T01:09:24.558 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T01:09:24.558 INFO:journalctl@ceph.mon.b.smithi047.stdout: self._write_client_files(client_files, host) 2022-05-14T01:09:24.558 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T01:09:24.559 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T01:09:24.559 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T01:09:24.559 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T01:09:24.560 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T01:09:24.560 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.event_loop.get_result(coro) 2022-05-14T01:09:24.560 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T01:09:24.561 INFO:journalctl@ceph.mon.b.smithi047.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T01:09:24.561 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T01:09:24.561 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.__get_result() 2022-05-14T01:09:24.561 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T01:09:24.561 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise self._exception 2022-05-14T01:09:24.562 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T01:09:24.562 INFO:journalctl@ceph.mon.b.smithi047.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T01:09:24.562 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T01:09:24.562 INFO:journalctl@ceph.mon.b.smithi047.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T01:09:24.563 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T01:09:24.563 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise 2022-05-14T01:09:24.563 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T01:09:24.563 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.gen.throw(type, value, traceback) 2022-05-14T01:09:24.563 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T01:09:24.564 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OrchestratorError(msg) 2022-05-14T01:09:24.564 INFO:journalctl@ceph.mon.b.smithi047.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:09:24.564 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:09:24 smithi047 ceph-mon[77147]: pgmap v3401: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:09:24.564 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:09:24 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:09:24.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:09:24 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:09:24.608 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T01:09:24.608 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:09:24.609 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T01:09:24.609 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:09:24.609 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:09:24.609 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:09:24.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T01:09:24.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:09:24.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T01:09:24.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:09:24.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T01:09:24.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:09:24.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T01:09:24.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:09:24.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T01:09:24.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:09:24.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:09:24.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:09:24.612 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:09:24 smithi115 ceph-mon[78593]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T01:09:24.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T01:09:24.613 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:09:24.613 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T01:09:24.613 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:09:24.613 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:09:24.614 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:09:24.614 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T01:09:24.614 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:09:24.614 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T01:09:24.615 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:09:24.615 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T01:09:24.615 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:09:24.615 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T01:09:24.615 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:09:24.616 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T01:09:24.616 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:09:24.616 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:09:24.616 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:09:24.617 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T01:09:24.617 INFO:journalctl@ceph.mon.c.smithi115.stdout: During handling of the above exception, another exception occurred: 2022-05-14T01:09:24.617 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T01:09:24.617 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T01:09:24.618 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T01:09:24.618 INFO:journalctl@ceph.mon.c.smithi115.stdout: return f(*arg) 2022-05-14T01:09:24.618 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T01:09:24.618 INFO:journalctl@ceph.mon.c.smithi115.stdout: self._write_client_files(client_files, host) 2022-05-14T01:09:24.618 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T01:09:24.619 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T01:09:24.619 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T01:09:24.619 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T01:09:24.619 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T01:09:24.620 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.event_loop.get_result(coro) 2022-05-14T01:09:24.620 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T01:09:24.620 INFO:journalctl@ceph.mon.c.smithi115.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T01:09:24.620 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T01:09:24.620 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.__get_result() 2022-05-14T01:09:24.621 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T01:09:24.621 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise self._exception 2022-05-14T01:09:24.621 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T01:09:24.621 INFO:journalctl@ceph.mon.c.smithi115.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T01:09:24.622 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T01:09:24.622 INFO:journalctl@ceph.mon.c.smithi115.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T01:09:24.622 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T01:09:24.622 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise 2022-05-14T01:09:24.622 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T01:09:24.623 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.gen.throw(type, value, traceback) 2022-05-14T01:09:24.623 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T01:09:24.623 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OrchestratorError(msg) 2022-05-14T01:09:24.623 INFO:journalctl@ceph.mon.c.smithi115.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:09:24.624 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:09:24 smithi115 ceph-mon[78593]: pgmap v3401: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:09:24.624 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:09:24 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:09:25.286 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:09:24 smithi115 conmon[90393]: 2022-05-14T01:09:24.881+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:09:25.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:09:25 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 6095 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:09:25.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:09:25 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:09:25.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:09:25 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 6095 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:09:25.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:09:25 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:09:26.287 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:09:25 smithi115 conmon[90393]: 2022-05-14T01:09:25.913+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:09:26.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:09:26 smithi047 ceph-mon[77147]: pgmap v3402: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:09:26.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:09:26 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:09:26.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:09:26 smithi115 ceph-mon[78593]: pgmap v3402: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:09:26.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:09:26 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:09:27.292 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:09:26 smithi115 conmon[90393]: 2022-05-14T01:09:26.925+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:09:27.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:09:27 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:09:27.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:09:27 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:09:28.294 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:09:27 smithi115 conmon[90393]: 2022-05-14T01:09:27.919+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:09:28.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:09:28 smithi115 ceph-mon[78593]: pgmap v3403: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:09:28.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:09:28 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:09:28.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:09:28 smithi047 ceph-mon[77147]: pgmap v3403: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:09:28.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:09:28 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:09:29.298 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:09:28 smithi115 conmon[90393]: 2022-05-14T01:09:28.950+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:09:29.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:09:29 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:09:29.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:09:29 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:09:30.300 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:09:29 smithi115 conmon[90393]: 2022-05-14T01:09:29.963+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:09:30.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:09:30 smithi115 ceph-mon[78593]: pgmap v3404: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:09:30.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:09:30 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:09:30.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:09:30 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:09:30.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:09:30 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 6100 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:09:30.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:09:30 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:09:30.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:09:30 smithi047 ceph-mon[77147]: pgmap v3404: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:09:30.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:09:30 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:09:30.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:09:30 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:09:30.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:09:30 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 6100 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:09:30.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:09:30 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:09:31.307 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:09:30 smithi115 conmon[90393]: 2022-05-14T01:09:30.979+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:09:31.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:09:31 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:09:31.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:09:31 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:09:32.306 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:09:31 smithi115 conmon[90393]: 2022-05-14T01:09:31.962+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:09:32.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:09:32 smithi115 ceph-mon[78593]: pgmap v3405: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-05-14T01:09:32.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:09:32 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:09:32.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:09:32 smithi047 ceph-mon[77147]: pgmap v3405: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-05-14T01:09:32.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:09:32 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:09:33.313 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:09:32 smithi115 conmon[90393]: 2022-05-14T01:09:32.955+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:09:33.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:09:33 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:09:33.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:09:33 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:09:34.313 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:09:33 smithi115 conmon[90393]: 2022-05-14T01:09:33.983+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:09:34.313 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:09:34 smithi115 ceph-mon[78593]: pgmap v3406: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:09:34.314 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:09:34 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:09:34.314 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:09:34 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:09:34.314 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:09:34 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:09:34.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:09:34 smithi047 ceph-mon[77147]: pgmap v3406: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:09:34.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:09:34 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:09:34.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:09:34 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:09:34.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:09:34 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:09:35.314 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:09:35 smithi115 conmon[90393]: 2022-05-14T01:09:35.023+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:09:35.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:09:35 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 6105 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:09:35.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:09:35 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:09:35.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:09:35 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 6105 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:09:35.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:09:35 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:09:36.315 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:09:36 smithi115 conmon[90393]: 2022-05-14T01:09:36.010+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:09:36.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:09:36 smithi115 ceph-mon[78593]: pgmap v3407: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:09:36.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:09:36 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:09:36.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:09:36 smithi047 ceph-mon[77147]: pgmap v3407: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:09:36.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:09:36 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:09:37.321 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:09:36 smithi115 conmon[90393]: 2022-05-14T01:09:36.969+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:09:37.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:09:37 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:09:37.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:09:37 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:09:38.322 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:09:38 smithi115 conmon[90393]: 2022-05-14T01:09:38.002+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:09:38.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:09:38 smithi115 ceph-mon[78593]: pgmap v3408: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:09:38.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:09:38 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:09:38.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:09:38 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:09:38.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:09:38 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:09:38.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:09:38 smithi047 ceph-mon[77147]: pgmap v3408: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:09:38.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:09:38 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:09:38.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:09:38 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:09:38.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:09:38 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:09:39.327 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:09:38 smithi115 conmon[90393]: 2022-05-14T01:09:38.961+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:09:39.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:09:39 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:09:39.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:09:39 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:09:40.328 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:09:39 smithi115 conmon[90393]: 2022-05-14T01:09:39.980+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:09:40.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:09:40 smithi115 ceph-mon[78593]: pgmap v3409: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2022-05-14T01:09:40.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:09:40 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 6110 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:09:40.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:09:40 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:09:40.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:09:40 smithi047 ceph-mon[77147]: pgmap v3409: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2022-05-14T01:09:40.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:09:40 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 6110 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:09:40.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:09:40 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:09:41.334 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:09:40 smithi115 conmon[90393]: 2022-05-14T01:09:40.972+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:09:41.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:09:41 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:09:41.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:09:41 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:09:42.335 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:09:41 smithi115 conmon[90393]: 2022-05-14T01:09:41.958+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:09:42.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:09:42 smithi115 ceph-mon[78593]: pgmap v3410: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-05-14T01:09:42.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:09:42 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:09:42.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:09:42 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:09:42.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:09:42 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:09:42.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:09:42 smithi047 ceph-mon[77147]: pgmap v3410: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-05-14T01:09:42.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:09:42 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:09:42.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:09:42 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:09:42.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:09:42 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:09:43.340 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:09:42 smithi115 conmon[90393]: 2022-05-14T01:09:42.992+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:09:43.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:09:43 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:09:43.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:09:43 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:09:44.342 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:09:43 smithi115 conmon[90393]: 2022-05-14T01:09:43.946+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:09:44.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:09:44 smithi115 ceph-mon[78593]: pgmap v3411: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:09:44.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:09:44 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:09:44.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:09:44 smithi047 ceph-mon[77147]: pgmap v3411: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:09:44.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:09:44 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:09:45.343 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:09:44 smithi115 conmon[90393]: 2022-05-14T01:09:44.908+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:09:45.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:09:45 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 6115 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:09:45.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:09:45 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:09:45.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:09:45 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 6115 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:09:45.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:09:45 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:09:46.347 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:09:45 smithi115 conmon[90393]: 2022-05-14T01:09:45.915+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:09:46.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:09:46 smithi115 ceph-mon[78593]: pgmap v3412: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:09:46.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:09:46 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:09:46.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:09:46 smithi047 ceph-mon[77147]: pgmap v3412: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:09:46.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:09:46 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:09:47.346 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:09:46 smithi115 conmon[90393]: 2022-05-14T01:09:46.943+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:09:47.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:09:47 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:09:47.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:09:47 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:09:48.352 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:09:47 smithi115 conmon[90393]: 2022-05-14T01:09:47.941+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:09:48.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:09:48 smithi115 ceph-mon[78593]: pgmap v3413: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:09:48.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:09:48 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:09:48.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:09:48 smithi047 ceph-mon[77147]: pgmap v3413: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:09:48.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:09:48 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:09:49.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:09:48 smithi115 conmon[90393]: 2022-05-14T01:09:48.951+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:09:49.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:09:49 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:09:49.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:09:49 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:09:50.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:09:49 smithi115 conmon[90393]: 2022-05-14T01:09:49.983+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:09:50.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:09:50 smithi047 ceph-mon[77147]: pgmap v3414: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:09:50.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:09:50 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:09:50.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:09:50 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:09:50.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:09:50 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 6120 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:09:50.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:09:50 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:09:50.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:09:50 smithi115 ceph-mon[78593]: pgmap v3414: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:09:50.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:09:50 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:09:50.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:09:50 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:09:50.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:09:50 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 6120 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:09:50.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:09:50 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:09:51.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:09:51 smithi115 conmon[90393]: 2022-05-14T01:09:51.032+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:09:51.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:09:51 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:09:51.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:09:51 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:09:52.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:09:52 smithi115 conmon[90393]: 2022-05-14T01:09:52.070+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:09:52.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:09:52 smithi047 ceph-mon[77147]: pgmap v3415: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-05-14T01:09:52.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:09:52 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:09:52.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:09:52 smithi115 ceph-mon[78593]: pgmap v3415: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-05-14T01:09:52.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:09:52 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:09:53.370 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:09:53 smithi115 conmon[90393]: 2022-05-14T01:09:53.111+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:09:53.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:09:53 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:09:53.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:09:53 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:09:54.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:09:54 smithi115 conmon[90393]: 2022-05-14T01:09:54.088+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:09:54.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:09:54 smithi047 ceph-mon[77147]: pgmap v3416: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:09:54.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:09:54 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:09:54.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:09:54 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:09:54.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:09:54 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:09:54.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:09:54 smithi115 ceph-mon[78593]: pgmap v3416: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:09:54.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:09:54 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:09:54.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:09:54 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:09:54.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:09:54 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:09:55.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:09:55 smithi115 conmon[90393]: 2022-05-14T01:09:55.137+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:09:55.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:09:55 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 6125 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:09:55.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:09:55 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:09:55.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:09:55 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 6125 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:09:55.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:09:55 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:09:56.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:09:56 smithi115 conmon[90393]: 2022-05-14T01:09:56.163+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:09:56.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:09:56 smithi115 ceph-mon[78593]: pgmap v3417: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:09:56.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:09:56 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:09:56.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:09:56 smithi047 ceph-mon[77147]: pgmap v3417: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:09:56.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:09:56 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:09:57.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:09:57 smithi115 conmon[90393]: 2022-05-14T01:09:57.155+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:09:57.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:09:57 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:09:57.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:09:57 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:09:58.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:09:58 smithi115 conmon[90393]: 2022-05-14T01:09:58.174+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:09:58.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:09:58 smithi115 ceph-mon[78593]: pgmap v3418: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:09:58.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:09:58 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:09:58.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:09:58 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:09:58.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:09:58 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:09:58.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:09:58 smithi047 ceph-mon[77147]: pgmap v3418: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:09:58.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:09:58 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:09:58.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:09:58 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:09:58.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:09:58 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:09:59.387 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:09:59 smithi115 conmon[90393]: 2022-05-14T01:09:59.129+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:09:59.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:09:59 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:09:59.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:09:59 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:10:00.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:10:00 smithi047 conmon[77123]: 2022-05-14T01:09:59.999+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : Health detail: HEALTH_ERR 1 filesystem is degraded; 1 mds daemon damaged; 1 MDSs are read only; 1 MDSs report slow metadata IOs; 1/3 mons down, quorum b,c; Reduced data availability: 1 pg inactive; 9 slow ops, oldest one blocked for 6131 sec, osd.5 has slow ops 2022-05-14T01:10:00.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:10:00 smithi047 conmon[77123]: 2022-05-14T01:09:59.999+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : [WRN] FS_DEGRADED: 1 filesystem is degraded 2022-05-14T01:10:00.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:10:00 smithi047 conmon[77123]: 2022-05-14T01:09:59.999+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : fs cephfs is degraded 2022-05-14T01:10:00.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:10:00 smithi047 conmon[77123]: 2022-05-14T01:09:59.999+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : [ERR] MDS_DAMAGE: 1 mds daemon damaged 2022-05-14T01:10:00.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:10:00 smithi047 conmon[77123]: 2022-05-14T01:09:59.999+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : fs cephfs mds.2 is damaged 2022-05-14T01:10:00.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:10:00 smithi047 conmon[77123]: 2022-05-14T01:09:59.999+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : [WRN] MDS_READ_ONLY: 1 MDSs are read only 2022-05-14T01:10:00.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:10:00 smithi047 conmon[77123]: 2022-05-14T01:09:59.999+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : mds.l(mds.0): MDS in read-only mode 2022-05-14T01:10:00.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:10:00 smithi047 conmon[77123]: 2022-05-14T01:09:59.999+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : [WRN] MDS_SLOW_METADATA_IO: 1 MDSs report slow metadata IOs 2022-05-14T01:10:00.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:10:00 smithi047 conmon[77123]: 2022-05-14T01:09:59.999+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : mds.l(mds.0): 4 slow metadata IOs are blocked > 180 secs, oldest blocked for 6433 secs 2022-05-14T01:10:00.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:10:00 smithi047 conmon[77123]: 2022-05-14T01:09:59.999+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : [WRN] MON_DOWN: 1/3 mons down, quorum b,c 2022-05-14T01:10:00.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:10:00 smithi047 conmon[77123]: 2022-05-14T01:09:59.999+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : mon.a (rank 0) addr [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] is down (out of quorum) 2022-05-14T01:10:00.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:10:00 smithi047 conmon[77123]: 2022-05-14T01:09:59.999+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : [WRN] PG_AVAILABILITY: Reduced data availability: 1 pg inactive 2022-05-14T01:10:00.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:10:00 smithi047 conmon[77123]: 2022-05-14T01:09:59.999+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : pg 2.1 is stuck inactive for 114m, current state unknown, last acting [] 2022-05-14T01:10:00.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:10:00 smithi047 conmon[77123]: 2022-05-14T01:09:59.999+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : [WRN] SLOW_OPS: 9 slow ops, oldest one blocked for 6131 sec, osd.5 has slow ops 2022-05-14T01:10:00.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:10:00 smithi115 conmon[90393]: 2022-05-14T01:10:00.148+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:10:00.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:10:00 smithi115 ceph-mon[78593]: pgmap v3419: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:10:00.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:10:00 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 6131 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:10:00.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:10:00 smithi115 ceph-mon[78593]: Health detail: HEALTH_ERR 1 filesystem is degraded; 1 mds daemon damaged; 1 MDSs are read only; 1 MDSs report slow metadata IOs; 1/3 mons down, quorum b,c; Reduced data availability: 1 pg inactive; 9 slow ops, oldest one blocked for 6131 sec, osd.5 has slow ops 2022-05-14T01:10:00.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:10:00 smithi115 ceph-mon[78593]: [WRN] FS_DEGRADED: 1 filesystem is degraded 2022-05-14T01:10:00.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:10:00 smithi115 ceph-mon[78593]: fs cephfs is degraded 2022-05-14T01:10:00.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:10:00 smithi115 ceph-mon[78593]: [ERR] MDS_DAMAGE: 1 mds daemon damaged 2022-05-14T01:10:00.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:10:00 smithi115 ceph-mon[78593]: fs cephfs mds.2 is damaged 2022-05-14T01:10:00.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:10:00 smithi115 ceph-mon[78593]: [WRN] MDS_READ_ONLY: 1 MDSs are read only 2022-05-14T01:10:00.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:10:00 smithi115 ceph-mon[78593]: mds.l(mds.0): MDS in read-only mode 2022-05-14T01:10:00.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:10:00 smithi115 ceph-mon[78593]: [WRN] MDS_SLOW_METADATA_IO: 1 MDSs report slow metadata IOs 2022-05-14T01:10:00.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:10:00 smithi115 ceph-mon[78593]: mds.l(mds.0): 4 slow metadata IOs are blocked > 180 secs, oldest blocked for 6433 secs 2022-05-14T01:10:00.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:10:00 smithi115 ceph-mon[78593]: [WRN] MON_DOWN: 1/3 mons down, quorum b,c 2022-05-14T01:10:00.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:10:00 smithi115 ceph-mon[78593]: mon.a (rank 0) addr [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] is down (out of quorum) 2022-05-14T01:10:00.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:10:00 smithi115 ceph-mon[78593]: [WRN] PG_AVAILABILITY: Reduced data availability: 1 pg inactive 2022-05-14T01:10:00.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:10:00 smithi115 ceph-mon[78593]: pg 2.1 is stuck inactive for 114m, current state unknown, last acting [] 2022-05-14T01:10:00.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:10:00 smithi115 ceph-mon[78593]: [WRN] SLOW_OPS: 9 slow ops, oldest one blocked for 6131 sec, osd.5 has slow ops 2022-05-14T01:10:00.612 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:10:00 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:10:00.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:10:00 smithi047 ceph-mon[77147]: pgmap v3419: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:10:00.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:10:00 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 6131 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:10:00.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:10:00 smithi047 ceph-mon[77147]: Health detail: HEALTH_ERR 1 filesystem is degraded; 1 mds daemon damaged; 1 MDSs are read only; 1 MDSs report slow metadata IOs; 1/3 mons down, quorum b,c; Reduced data availability: 1 pg inactive; 9 slow ops, oldest one blocked for 6131 sec, osd.5 has slow ops 2022-05-14T01:10:00.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:10:00 smithi047 ceph-mon[77147]: [WRN] FS_DEGRADED: 1 filesystem is degraded 2022-05-14T01:10:00.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:10:00 smithi047 ceph-mon[77147]: fs cephfs is degraded 2022-05-14T01:10:00.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:10:00 smithi047 ceph-mon[77147]: [ERR] MDS_DAMAGE: 1 mds daemon damaged 2022-05-14T01:10:00.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:10:00 smithi047 ceph-mon[77147]: fs cephfs mds.2 is damaged 2022-05-14T01:10:00.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:10:00 smithi047 ceph-mon[77147]: [WRN] MDS_READ_ONLY: 1 MDSs are read only 2022-05-14T01:10:00.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:10:00 smithi047 ceph-mon[77147]: mds.l(mds.0): MDS in read-only mode 2022-05-14T01:10:00.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:10:00 smithi047 ceph-mon[77147]: [WRN] MDS_SLOW_METADATA_IO: 1 MDSs report slow metadata IOs 2022-05-14T01:10:00.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:10:00 smithi047 ceph-mon[77147]: mds.l(mds.0): 4 slow metadata IOs are blocked > 180 secs, oldest blocked for 6433 secs 2022-05-14T01:10:00.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:10:00 smithi047 ceph-mon[77147]: [WRN] MON_DOWN: 1/3 mons down, quorum b,c 2022-05-14T01:10:00.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:10:00 smithi047 ceph-mon[77147]: mon.a (rank 0) addr [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] is down (out of quorum) 2022-05-14T01:10:00.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:10:00 smithi047 ceph-mon[77147]: [WRN] PG_AVAILABILITY: Reduced data availability: 1 pg inactive 2022-05-14T01:10:00.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:10:00 smithi047 ceph-mon[77147]: pg 2.1 is stuck inactive for 114m, current state unknown, last acting [] 2022-05-14T01:10:00.795 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:10:00 smithi047 ceph-mon[77147]: [WRN] SLOW_OPS: 9 slow ops, oldest one blocked for 6131 sec, osd.5 has slow ops 2022-05-14T01:10:00.795 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:10:00 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:10:01.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:10:01 smithi115 conmon[90393]: 2022-05-14T01:10:01.194+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:10:02.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:10:02 smithi115 conmon[90393]: 2022-05-14T01:10:02.161+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:10:02.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:10:02 smithi115 ceph-mon[78593]: pgmap v3420: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-05-14T01:10:02.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:10:02 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:10:02.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:10:02 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:10:02.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:10:02 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:10:02.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:10:02 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:10:02.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:10:02 smithi047 ceph-mon[77147]: pgmap v3420: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-05-14T01:10:02.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:10:02 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:10:02.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:10:02 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:10:02.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:10:02 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:10:02.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:10:02 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:10:03.402 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:10:03 smithi115 conmon[90393]: 2022-05-14T01:10:03.114+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:10:03.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:10:03 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:10:03.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:10:03 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:10:04.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:10:04 smithi115 conmon[90393]: 2022-05-14T01:10:04.097+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:10:04.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:10:04 smithi047 ceph-mon[77147]: pgmap v3421: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:10:04.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:10:04 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:10:04.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:10:04 smithi115 ceph-mon[78593]: pgmap v3421: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:10:04.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:10:04 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:10:05.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:10:05 smithi115 conmon[90393]: 2022-05-14T01:10:05.072+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:10:05.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:10:05 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 6136 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:10:05.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:10:05 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:10:05.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:10:05 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 6136 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:10:05.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:10:05 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:10:06.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:10:06 smithi115 conmon[90393]: 2022-05-14T01:10:06.036+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:10:06.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:10:06 smithi047 ceph-mon[77147]: pgmap v3422: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:10:06.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:10:06 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:10:06.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:10:06 smithi115 ceph-mon[78593]: pgmap v3422: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:10:06.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:10:06 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:10:07.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:10:07 smithi115 conmon[90393]: 2022-05-14T01:10:07.059+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:10:07.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:10:07 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:10:07.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:10:07 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:10:08.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:10:08 smithi115 conmon[90393]: 2022-05-14T01:10:08.074+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:10:08.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:10:08 smithi047 ceph-mon[77147]: pgmap v3423: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:10:08.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:10:08 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:10:08.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:10:08 smithi115 ceph-mon[78593]: pgmap v3423: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:10:08.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:10:08 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:10:09.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:10:09 smithi115 conmon[90393]: 2022-05-14T01:10:09.072+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:10:09.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:10:09 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:10:09.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:10:09 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:10:10.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:10:10 smithi115 conmon[90393]: 2022-05-14T01:10:10.068+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:10:10.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:10:10 smithi047 ceph-mon[77147]: pgmap v3424: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:10:10.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:10:10 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:10:10.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:10:10 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:10:10.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:10:10 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 6141 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:10:10.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:10:10 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:10:10.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:10:10 smithi115 ceph-mon[78593]: pgmap v3424: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:10:10.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:10:10 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:10:10.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:10:10 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:10:10.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:10:10 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 6141 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:10:10.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:10:10 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:10:11.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:10:11 smithi115 conmon[90393]: 2022-05-14T01:10:11.053+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:10:11.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:10:11 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:10:11.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:10:11 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:10:12.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:10:12 smithi115 conmon[90393]: 2022-05-14T01:10:12.061+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:10:12.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:10:12 smithi047 ceph-mon[77147]: pgmap v3425: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-05-14T01:10:12.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:10:12 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:10:12.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:10:12 smithi115 ceph-mon[78593]: pgmap v3425: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-05-14T01:10:12.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:10:12 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:10:13.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:10:13 smithi115 conmon[90393]: 2022-05-14T01:10:13.083+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:10:13.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:10:13 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:10:13.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:10:13 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:10:14.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:10:14 smithi115 conmon[90393]: 2022-05-14T01:10:14.064+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:10:14.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:10:14 smithi047 ceph-mon[77147]: pgmap v3426: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:10:14.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:10:14 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:10:14.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:10:14 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:10:14.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:10:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T01:10:14.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:10:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T01:10:14.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:10:14 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:10:14.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:10:14 smithi115 ceph-mon[78593]: pgmap v3426: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:10:14.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:10:14 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:10:14.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:10:14 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:10:14.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:10:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T01:10:14.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:10:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T01:10:14.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:10:14 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:10:15.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:10:15 smithi115 conmon[90393]: 2022-05-14T01:10:15.023+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:10:15.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:10:15 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 6145 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:10:15.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:10:15 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:10:15.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:10:15 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 6145 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:10:15.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:10:15 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:10:16.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:10:15 smithi115 conmon[90393]: 2022-05-14T01:10:15.984+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:10:16.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:10:16 smithi047 ceph-mon[77147]: pgmap v3427: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:10:16.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:10:16 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:10:16.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:10:16 smithi115 ceph-mon[78593]: pgmap v3427: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:10:16.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:10:16 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:10:17.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:10:16 smithi115 conmon[90393]: 2022-05-14T01:10:16.988+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:10:17.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:10:17 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:10:17.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:10:17 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:10:18.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:10:17 smithi115 conmon[90393]: 2022-05-14T01:10:17.959+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:10:18.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:10:18 smithi047 ceph-mon[77147]: pgmap v3428: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:10:18.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:10:18 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:10:18.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:10:18 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:10:18.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:10:18 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:10:18.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:10:18 smithi115 ceph-mon[78593]: pgmap v3428: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:10:18.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:10:18 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:10:18.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:10:18 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:10:18.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:10:18 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:10:19.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:10:18 smithi115 conmon[90393]: 2022-05-14T01:10:18.986+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:10:19.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:10:19 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:10:19.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:10:19 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:10:20.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:10:20 smithi115 conmon[90393]: 2022-05-14T01:10:20.036+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:10:20.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:10:20 smithi047 ceph-mon[77147]: pgmap v3429: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:10:20.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:10:20 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 6150 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:10:20.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:10:20 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:10:20.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:10:20 smithi115 ceph-mon[78593]: pgmap v3429: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:10:20.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:10:20 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 6150 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:10:20.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:10:20 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:10:21.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:10:21 smithi115 conmon[90393]: 2022-05-14T01:10:21.069+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:10:21.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:10:21 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:10:21.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:10:21 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:10:22.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:10:22 smithi115 conmon[90393]: 2022-05-14T01:10:22.096+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:10:22.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:10:22 smithi047 ceph-mon[77147]: pgmap v3430: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-05-14T01:10:22.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:10:22 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:10:22.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:10:22 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:10:22.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:10:22 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:10:22.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:10:22 smithi115 ceph-mon[78593]: pgmap v3430: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-05-14T01:10:22.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:10:22 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:10:22.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:10:22 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:10:22.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:10:22 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:10:23.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:10:23 smithi115 conmon[90393]: 2022-05-14T01:10:23.056+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:10:23.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:10:23 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:10:23.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:10:23 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T01:10:23.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:10:23 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T01:10:23.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:10:23 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T01:10:23.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:10:23 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:10:23.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:10:23 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T01:10:23.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:10:23 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T01:10:23.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:10:23 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T01:10:24.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:10:24 smithi115 conmon[90393]: 2022-05-14T01:10:24.049+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:10:24.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:10:24 smithi047 ceph-mon[77147]: pgmap v3431: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:10:24.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:10:24 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:10:24.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:10:24 smithi115 ceph-mon[78593]: pgmap v3431: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:10:24.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:10:24 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:10:25.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:10:25 smithi115 conmon[90393]: 2022-05-14T01:10:25.032+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:10:25.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:10:25 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 6155 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:10:25.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:10:25 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:10:25.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:10:25 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 6155 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:10:25.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:10:25 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:10:26.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:10:26 smithi115 conmon[90393]: 2022-05-14T01:10:26.078+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:10:26.471 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:10:26 smithi047 conmon[78521]: 2022-05-14T01:10:26.160+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:10:26.472 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:10:26 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T01:10:26.472 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:10:26 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:10:26.473 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:10:26 smithi047 conmon[78521]: yield 2022-05-14T01:10:26.473 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:10:26 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:10:26.473 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:10:26 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:10:26.473 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:10:26 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:10:26.474 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:10:26 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T01:10:26.474 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:10:26 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:10:26.474 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:10:26 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T01:10:26.475 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:10:26 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:10:26.475 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:10:26 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T01:10:26.475 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:10:26 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:10:26.475 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:10:26 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T01:10:26.476 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:10:26 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:10:26.476 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:10:26 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T01:10:26.476 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:10:26 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:10:26.476 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:10:26 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:10:26.477 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:10:26 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:10:26.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:10:26 smithi047 ceph-mon[77147]: pgmap v3432: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:10:26.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:10:26 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:10:26.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:10:26 smithi115 ceph-mon[78593]: pgmap v3432: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:10:26.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:10:26 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:10:27.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:10:27 smithi115 conmon[90393]: 2022-05-14T01:10:27.032+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:10:27.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:10:27 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:10:27.791 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T01:10:27.791 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:10:27.792 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T01:10:27.792 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:10:27.792 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:10:27.793 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:10:27.793 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T01:10:27.793 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:10:27.793 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T01:10:27.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:10:27.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T01:10:27.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:10:27.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T01:10:27.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:10:27.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T01:10:27.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:10:27.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:10:27.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:10:27.796 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:10:27 smithi047 ceph-mon[77147]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T01:10:27.796 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:10:27 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:10:27.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:10:27 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:10:27.858 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T01:10:27.858 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:10:27.858 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T01:10:27.858 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:10:27.858 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:10:27.859 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:10:27.859 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T01:10:27.859 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:10:27.860 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T01:10:27.860 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:10:27.860 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T01:10:27.860 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:10:27.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T01:10:27.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:10:27.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T01:10:27.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:10:27.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:10:27.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:10:27.862 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:10:27 smithi115 ceph-mon[78593]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T01:10:27.862 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:10:27 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:10:28.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:10:28 smithi115 conmon[90393]: 2022-05-14T01:10:28.007+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:10:28.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:10:28 smithi047 ceph-mon[77147]: pgmap v3433: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:10:28.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:10:28 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:10:28.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:10:28 smithi115 ceph-mon[78593]: pgmap v3433: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:10:28.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:10:28 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:10:29.237 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:10:29 smithi047 conmon[78521]: 2022-05-14T01:10:29.232+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:10:29.238 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:10:29 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T01:10:29.238 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:10:29 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:10:29.238 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:10:29 smithi047 conmon[78521]: yield 2022-05-14T01:10:29.238 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:10:29 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:10:29.239 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:10:29 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:10:29.239 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:10:29 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:10:29.239 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:10:29 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T01:10:29.239 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:10:29 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:10:29.239 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:10:29 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T01:10:29.240 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:10:29 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:10:29.240 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:10:29 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T01:10:29.240 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:10:29 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:10:29.240 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:10:29 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T01:10:29.240 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:10:29 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:10:29.241 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:10:29 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T01:10:29.241 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:10:29 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:10:29.241 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:10:29 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:10:29.241 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:10:29 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:10:29.242 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:10:29 smithi047 conmon[78521]: 2022-05-14T01:10:29.233+0000 7f533ff78700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T01:10:29.242 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:10:29 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T01:10:29.242 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:10:29 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:10:29.242 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:10:29 smithi047 conmon[78521]: yield 2022-05-14T01:10:29.242 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:10:29 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:10:29.243 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:10:29 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:10:29.243 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:10:29 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:10:29.243 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:10:29 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T01:10:29.243 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:10:29 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:10:29.243 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:10:29 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T01:10:29.244 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:10:29 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:10:29.244 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:10:29 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T01:10:29.244 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:10:29 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:10:29.244 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:10:29 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T01:10:29.245 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:10:29 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:10:29.245 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:10:29 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T01:10:29.245 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:10:29 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:10:29.245 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:10:29 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:10:29.245 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:10:29 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:10:29.246 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:10:29 smithi047 conmon[78521]: 2022-05-14T01:10:29.246 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:10:29 smithi047 conmon[78521]: During handling of the above exception, another exception occurred: 2022-05-14T01:10:29.246 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:10:29 smithi047 conmon[78521]: 2022-05-14T01:10:29.246 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:10:29 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T01:10:29.246 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:10:29 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T01:10:29.247 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:10:29 smithi047 conmon[78521]: return f(*arg) 2022-05-14T01:10:29.247 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:10:29 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T01:10:29.247 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:10:29 smithi047 conmon[78521]: self._write_client_files(client_files, host) 2022-05-14T01:10:29.247 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:10:29 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T01:10:29.248 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:10:29 smithi047 conmon[78521]: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T01:10:29.248 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:10:29 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T01:10:29.248 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:10:29 smithi047 conmon[78521]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T01:10:29.248 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:10:29 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T01:10:29.248 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:10:29 smithi047 conmon[78521]: return self.event_loop.get_result(coro) 2022-05-14T01:10:29.249 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:10:29 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T01:10:29.249 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:10:29 smithi047 conmon[78521]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T01:10:29.249 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:10:29 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T01:10:29.249 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:10:29 smithi047 conmon[78521]: return self.__get_result() 2022-05-14T01:10:29.250 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:10:29 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T01:10:29.250 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:10:29 smithi047 conmon[78521]: raise self._exception 2022-05-14T01:10:29.250 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:10:29 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T01:10:29.250 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:10:29 smithi047 conmon[78521]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T01:10:29.250 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:10:29 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T01:10:29.251 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:10:29 smithi047 conmon[78521]: conn = await self._remote_connection(host, addr) 2022-05-14T01:10:29.251 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:10:29 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T01:10:29.251 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:10:29 smithi047 conmon[78521]: raise 2022-05-14T01:10:29.251 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:10:29 smithi047 conmon[78521]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T01:10:29.251 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:10:29 smithi047 conmon[78521]: self.gen.throw(type, value, traceback) 2022-05-14T01:10:29.252 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:10:29 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T01:10:29.252 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:10:29 smithi047 conmon[78521]: raise OrchestratorError(msg) 2022-05-14T01:10:29.252 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:10:29 smithi047 conmon[78521]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:10:29.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:10:29 smithi115 conmon[90393]: 2022-05-14T01:10:29.003+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:10:29.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:10:29 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:10:29.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:10:29 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:10:30.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:10:30 smithi115 conmon[90393]: 2022-05-14T01:10:30.007+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:10:30.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:10:30 smithi047 ceph-mon[77147]: pgmap v3434: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:10:30.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:10:30 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:10:30.792 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T01:10:30.792 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:10:30.792 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T01:10:30.792 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:10:30.793 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:10:30.793 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:10:30.793 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T01:10:30.793 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:10:30.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T01:10:30.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:10:30.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T01:10:30.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:10:30.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T01:10:30.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:10:30.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T01:10:30.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:10:30.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:10:30.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:10:30.796 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:10:30 smithi047 ceph-mon[77147]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T01:10:30.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T01:10:30.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:10:30.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T01:10:30.797 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:10:30.797 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:10:30.797 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:10:30.797 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T01:10:30.798 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:10:30.798 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T01:10:30.798 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:10:30.798 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T01:10:30.798 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:10:30.799 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T01:10:30.799 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:10:30.799 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T01:10:30.799 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:10:30.799 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:10:30.800 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:10:30.800 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T01:10:30.800 INFO:journalctl@ceph.mon.b.smithi047.stdout: During handling of the above exception, another exception occurred: 2022-05-14T01:10:30.800 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T01:10:30.801 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T01:10:30.801 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T01:10:30.801 INFO:journalctl@ceph.mon.b.smithi047.stdout: return f(*arg) 2022-05-14T01:10:30.801 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T01:10:30.801 INFO:journalctl@ceph.mon.b.smithi047.stdout: self._write_client_files(client_files, host) 2022-05-14T01:10:30.802 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T01:10:30.802 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T01:10:30.802 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T01:10:30.802 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T01:10:30.803 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T01:10:30.803 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.event_loop.get_result(coro) 2022-05-14T01:10:30.803 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T01:10:30.803 INFO:journalctl@ceph.mon.b.smithi047.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T01:10:30.803 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T01:10:30.804 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.__get_result() 2022-05-14T01:10:30.804 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T01:10:30.804 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise self._exception 2022-05-14T01:10:30.804 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T01:10:30.805 INFO:journalctl@ceph.mon.b.smithi047.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T01:10:30.805 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T01:10:30.805 INFO:journalctl@ceph.mon.b.smithi047.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T01:10:30.805 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T01:10:30.805 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise 2022-05-14T01:10:30.806 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T01:10:30.806 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.gen.throw(type, value, traceback) 2022-05-14T01:10:30.806 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T01:10:30.806 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OrchestratorError(msg) 2022-05-14T01:10:30.807 INFO:journalctl@ceph.mon.b.smithi047.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:10:30.807 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:10:30 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:10:30.807 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:10:30 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:10:30.807 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:10:30 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 6160 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:10:30.807 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:10:30 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:10:30.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:10:30 smithi115 ceph-mon[78593]: pgmap v3434: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:10:30.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:10:30 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:10:30.858 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T01:10:30.858 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:10:30.859 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T01:10:30.859 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:10:30.859 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:10:30.859 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:10:30.860 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T01:10:30.860 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:10:30.860 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T01:10:30.860 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:10:30.860 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T01:10:30.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:10:30.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T01:10:30.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:10:30.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T01:10:30.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:10:30.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:10:30.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:10:30.862 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:10:30 smithi115 ceph-mon[78593]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T01:10:30.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T01:10:30.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:10:30.863 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T01:10:30.863 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:10:30.863 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:10:30.863 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:10:30.864 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T01:10:30.864 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:10:30.864 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T01:10:30.864 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:10:30.864 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T01:10:30.865 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:10:30.865 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T01:10:30.865 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:10:30.865 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T01:10:30.865 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:10:30.866 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:10:30.866 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:10:30.866 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T01:10:30.866 INFO:journalctl@ceph.mon.c.smithi115.stdout: During handling of the above exception, another exception occurred: 2022-05-14T01:10:30.867 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T01:10:30.867 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T01:10:30.867 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T01:10:30.867 INFO:journalctl@ceph.mon.c.smithi115.stdout: return f(*arg) 2022-05-14T01:10:30.867 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T01:10:30.868 INFO:journalctl@ceph.mon.c.smithi115.stdout: self._write_client_files(client_files, host) 2022-05-14T01:10:30.868 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T01:10:30.868 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T01:10:30.868 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T01:10:30.868 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T01:10:30.869 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T01:10:30.869 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.event_loop.get_result(coro) 2022-05-14T01:10:30.869 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T01:10:30.869 INFO:journalctl@ceph.mon.c.smithi115.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T01:10:30.870 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T01:10:30.870 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.__get_result() 2022-05-14T01:10:30.870 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T01:10:30.870 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise self._exception 2022-05-14T01:10:30.870 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T01:10:30.871 INFO:journalctl@ceph.mon.c.smithi115.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T01:10:30.871 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T01:10:30.871 INFO:journalctl@ceph.mon.c.smithi115.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T01:10:30.871 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T01:10:30.871 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise 2022-05-14T01:10:30.872 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T01:10:30.872 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.gen.throw(type, value, traceback) 2022-05-14T01:10:30.878 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T01:10:30.878 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OrchestratorError(msg) 2022-05-14T01:10:30.878 INFO:journalctl@ceph.mon.c.smithi115.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:10:30.878 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:10:30 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:10:30.878 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:10:30 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:10:30.879 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:10:30 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 6160 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:10:30.879 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:10:30 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:10:31.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:10:30 smithi115 conmon[90393]: 2022-05-14T01:10:30.960+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:10:31.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:10:31 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:10:31.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:10:31 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:10:32.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:10:31 smithi115 conmon[90393]: 2022-05-14T01:10:31.957+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:10:32.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:10:32 smithi047 ceph-mon[77147]: pgmap v3435: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-05-14T01:10:32.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:10:32 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:10:32.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:10:32 smithi115 ceph-mon[78593]: pgmap v3435: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-05-14T01:10:32.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:10:32 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:10:33.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:10:32 smithi115 conmon[90393]: 2022-05-14T01:10:32.925+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:10:33.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:10:33 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:10:33.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:10:33 smithi047 ceph-mon[77147]: pgmap v3436: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:10:33.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:10:33 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:10:33.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:10:33 smithi115 ceph-mon[78593]: pgmap v3436: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:10:34.361 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:10:33 smithi115 conmon[90393]: 2022-05-14T01:10:33.897+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:10:34.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:10:34 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:10:34.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:10:34 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:10:34.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:10:34 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:10:34.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:10:34 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:10:34.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:10:34 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:10:34.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:10:34 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:10:35.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:10:34 smithi115 conmon[90393]: 2022-05-14T01:10:34.887+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:10:35.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:10:35 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 6165 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:10:35.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:10:35 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:10:35.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:10:35 smithi047 ceph-mon[77147]: pgmap v3437: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:10:35.840 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:10:35 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 6165 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:10:35.841 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:10:35 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:10:35.841 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:10:35 smithi115 ceph-mon[78593]: pgmap v3437: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:10:36.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:10:35 smithi115 conmon[90393]: 2022-05-14T01:10:35.839+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:10:36.822 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:10:36 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:10:37.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:10:36 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:10:37.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:10:36 smithi115 conmon[90393]: 2022-05-14T01:10:36.821+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:10:37.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:10:37 smithi115 conmon[90393]: 2022-05-14T01:10:37.804+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:10:37.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:10:37 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:10:37.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:10:37 smithi115 ceph-mon[78593]: pgmap v3438: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:10:38.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:10:37 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:10:38.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:10:37 smithi047 ceph-mon[77147]: pgmap v3438: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:10:38.824 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:10:38 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:10:38.824 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:10:38 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:10:38.824 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:10:38 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:10:39.045 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:10:38 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:10:39.045 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:10:38 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:10:39.046 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:10:38 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:10:39.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:10:38 smithi115 conmon[90393]: 2022-05-14T01:10:38.823+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:10:39.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:10:39 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:10:39.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:10:39 smithi115 ceph-mon[78593]: pgmap v3439: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:10:39.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:10:39 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 6170 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:10:40.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:10:39 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:10:40.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:10:39 smithi047 ceph-mon[77147]: pgmap v3439: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:10:40.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:10:39 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 6170 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:10:40.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:10:39 smithi115 conmon[90393]: 2022-05-14T01:10:39.867+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:10:40.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:10:40 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:10:41.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:10:40 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:10:41.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:10:40 smithi115 conmon[90393]: 2022-05-14T01:10:40.885+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:10:41.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:10:41 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:10:41.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:10:41 smithi115 ceph-mon[78593]: pgmap v3440: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-05-14T01:10:42.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:10:41 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:10:42.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:10:41 smithi047 ceph-mon[77147]: pgmap v3440: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-05-14T01:10:42.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:10:41 smithi115 conmon[90393]: 2022-05-14T01:10:41.873+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:10:42.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:10:42 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:10:42.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:10:42 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:10:42.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:10:42 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:10:43.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:10:42 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:10:43.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:10:42 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:10:43.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:10:42 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:10:43.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:10:42 smithi115 conmon[90393]: 2022-05-14T01:10:42.900+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:10:43.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:10:43 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:10:43.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:10:43 smithi115 ceph-mon[78593]: pgmap v3441: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:10:44.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:10:43 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:10:44.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:10:43 smithi047 ceph-mon[77147]: pgmap v3441: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:10:44.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:10:43 smithi115 conmon[90393]: 2022-05-14T01:10:43.923+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:10:44.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:10:44 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:10:44.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:10:44 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 6175 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:10:45.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:10:44 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:10:45.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:10:44 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 6175 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:10:45.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:10:44 smithi115 conmon[90393]: 2022-05-14T01:10:44.888+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:10:45.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:10:45 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:10:45.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:10:45 smithi115 ceph-mon[78593]: pgmap v3442: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:10:46.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:10:45 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:10:46.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:10:45 smithi047 ceph-mon[77147]: pgmap v3442: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:10:46.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:10:45 smithi115 conmon[90393]: 2022-05-14T01:10:45.863+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:10:46.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:10:46 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:10:47.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:10:46 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:10:47.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:10:46 smithi115 conmon[90393]: 2022-05-14T01:10:46.861+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:10:47.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:10:47 smithi115 conmon[90393]: 2022-05-14T01:10:47.819+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:10:47.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:10:47 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:10:47.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:10:47 smithi115 ceph-mon[78593]: pgmap v3443: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:10:48.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:10:47 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:10:48.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:10:47 smithi047 ceph-mon[77147]: pgmap v3443: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:10:48.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:10:48 smithi115 conmon[90393]: 2022-05-14T01:10:48.832+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:10:48.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:10:48 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:10:49.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:10:48 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:10:49.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:10:49 smithi115 conmon[90393]: 2022-05-14T01:10:49.823+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:10:49.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:10:49 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:10:49.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:10:49 smithi115 ceph-mon[78593]: pgmap v3444: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:10:49.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:10:49 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 6180 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:10:50.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:10:49 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:10:50.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:10:49 smithi047 ceph-mon[77147]: pgmap v3444: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:10:50.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:10:49 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 6180 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:10:50.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:10:50 smithi115 conmon[90393]: 2022-05-14T01:10:50.804+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:10:50.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:10:50 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:10:50.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:10:50 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:10:50.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:10:50 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:10:51.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:10:50 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:10:51.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:10:50 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:10:51.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:10:50 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:10:51.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:10:51 smithi115 conmon[90393]: 2022-05-14T01:10:51.780+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:10:51.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:10:51 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:10:51.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:10:51 smithi115 ceph-mon[78593]: pgmap v3445: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-05-14T01:10:52.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:10:51 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:10:52.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:10:51 smithi047 ceph-mon[77147]: pgmap v3445: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-05-14T01:10:52.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:10:52 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:10:52.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:10:52 smithi115 conmon[90393]: 2022-05-14T01:10:52.808+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:10:53.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:10:52 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:10:53.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:10:53 smithi115 conmon[90393]: 2022-05-14T01:10:53.776+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:10:53.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:10:53 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:10:53.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:10:53 smithi115 ceph-mon[78593]: pgmap v3446: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:10:54.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:10:53 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:10:54.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:10:53 smithi047 ceph-mon[77147]: pgmap v3446: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:10:55.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:10:54 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:10:55.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:10:54 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:10:55.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:10:54 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:10:55.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:10:54 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 6185 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:10:55.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:10:54 smithi115 conmon[90393]: 2022-05-14T01:10:54.808+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:10:55.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:10:54 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:10:55.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:10:54 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:10:55.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:10:54 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:10:55.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:10:54 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 6185 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:10:56.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:10:55 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:10:56.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:10:55 smithi047 ceph-mon[77147]: pgmap v3447: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:10:56.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:10:55 smithi115 conmon[90393]: 2022-05-14T01:10:55.779+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:10:56.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:10:55 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:10:56.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:10:55 smithi115 ceph-mon[78593]: pgmap v3447: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:10:57.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:10:56 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:10:57.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:10:56 smithi115 conmon[90393]: 2022-05-14T01:10:56.807+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:10:57.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:10:56 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:10:58.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:10:57 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:10:58.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:10:57 smithi047 ceph-mon[77147]: pgmap v3448: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:10:58.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:10:57 smithi115 conmon[90393]: 2022-05-14T01:10:57.777+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:10:58.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:10:57 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:10:58.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:10:57 smithi115 ceph-mon[78593]: pgmap v3448: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:10:59.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:10:58 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:10:59.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:10:58 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:10:59.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:10:58 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:10:59.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:10:58 smithi115 conmon[90393]: 2022-05-14T01:10:58.766+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:10:59.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:10:58 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:10:59.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:10:58 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:10:59.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:10:58 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:11:00.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:10:59 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:11:00.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:10:59 smithi047 ceph-mon[77147]: pgmap v3449: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:11:00.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:10:59 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 6190 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:11:00.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:10:59 smithi115 conmon[90393]: 2022-05-14T01:10:59.779+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:11:00.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:10:59 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:11:00.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:10:59 smithi115 ceph-mon[78593]: pgmap v3449: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:11:00.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:10:59 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 6190 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:11:01.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:11:00 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:11:01.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:11:00 smithi115 conmon[90393]: 2022-05-14T01:11:00.797+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:11:01.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:11:00 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:11:02.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:11:01 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:11:02.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:11:01 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:11:02.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:11:01 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:11:02.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:11:01 smithi047 ceph-mon[77147]: pgmap v3450: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-05-14T01:11:02.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:11:01 smithi115 conmon[90393]: 2022-05-14T01:11:01.780+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:11:02.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:11:01 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:11:02.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:11:01 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:11:02.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:11:01 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:11:02.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:11:01 smithi115 ceph-mon[78593]: pgmap v3450: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-05-14T01:11:03.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:11:02 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:11:03.113 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:11:02 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:11:03.114 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:11:02 smithi115 conmon[90393]: 2022-05-14T01:11:02.740+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:11:04.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:11:03 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:11:04.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:11:03 smithi047 ceph-mon[77147]: pgmap v3451: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:11:04.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:11:03 smithi115 conmon[90393]: 2022-05-14T01:11:03.717+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:11:04.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:11:03 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:11:04.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:11:03 smithi115 ceph-mon[78593]: pgmap v3451: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:11:05.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:11:04 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:11:05.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:11:04 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 6195 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:11:05.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:11:04 smithi115 conmon[90393]: 2022-05-14T01:11:04.748+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:11:05.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:11:04 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:11:05.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:11:04 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 6195 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:11:06.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:11:05 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:11:06.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:11:05 smithi047 ceph-mon[77147]: pgmap v3452: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:11:06.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:11:05 smithi115 conmon[90393]: 2022-05-14T01:11:05.762+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:11:06.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:11:05 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:11:06.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:11:05 smithi115 ceph-mon[78593]: pgmap v3452: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:11:07.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:11:06 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:11:07.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:11:06 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:11:07.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:11:06 smithi115 conmon[90393]: 2022-05-14T01:11:06.784+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:11:08.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:11:07 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:11:08.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:11:07 smithi047 ceph-mon[77147]: pgmap v3453: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:11:08.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:11:07 smithi115 conmon[90393]: 2022-05-14T01:11:07.827+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:11:08.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:11:07 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:11:08.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:11:07 smithi115 ceph-mon[78593]: pgmap v3453: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:11:09.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:11:08 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:11:09.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:11:08 smithi115 conmon[90393]: 2022-05-14T01:11:08.820+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:11:09.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:11:08 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:11:10.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:11:09 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:11:10.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:11:09 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:11:10.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:11:09 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:11:10.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:11:09 smithi047 ceph-mon[77147]: pgmap v3454: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:11:10.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:11:09 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 6200 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:11:10.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:11:09 smithi115 conmon[90393]: 2022-05-14T01:11:09.802+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:11:10.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:11:09 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:11:10.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:11:09 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:11:10.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:11:09 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:11:10.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:11:09 smithi115 ceph-mon[78593]: pgmap v3454: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:11:10.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:11:09 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 6200 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:11:11.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:11:10 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:11:11.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:11:10 smithi115 conmon[90393]: 2022-05-14T01:11:10.807+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:11:11.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:11:10 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:11:12.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:11:11 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:11:12.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:11:11 smithi047 ceph-mon[77147]: pgmap v3455: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-05-14T01:11:12.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:11:11 smithi115 conmon[90393]: 2022-05-14T01:11:11.816+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:11:12.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:11:11 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:11:12.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:11:11 smithi115 ceph-mon[78593]: pgmap v3455: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-05-14T01:11:13.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:11:12 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:11:13.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:11:12 smithi115 conmon[90393]: 2022-05-14T01:11:12.782+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:11:13.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:11:12 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:11:14.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:11:13 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:11:14.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:11:13 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:11:14.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:11:13 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:11:14.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:11:13 smithi047 ceph-mon[77147]: pgmap v3456: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:11:14.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:11:13 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T01:11:14.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:11:13 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T01:11:14.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:11:13 smithi115 conmon[90393]: 2022-05-14T01:11:13.796+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:11:14.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:11:13 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:11:14.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:11:13 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:11:14.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:11:13 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:11:14.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:11:13 smithi115 ceph-mon[78593]: pgmap v3456: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:11:14.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:11:13 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T01:11:14.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:11:13 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T01:11:15.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:11:14 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:11:15.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:11:14 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 6205 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:11:15.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:11:14 smithi115 conmon[90393]: 2022-05-14T01:11:14.846+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:11:15.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:11:14 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:11:15.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:11:14 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 6205 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:11:16.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:11:15 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:11:16.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:11:15 smithi047 ceph-mon[77147]: pgmap v3457: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:11:16.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:11:15 smithi115 conmon[90393]: 2022-05-14T01:11:15.817+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:11:16.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:11:15 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:11:16.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:11:15 smithi115 ceph-mon[78593]: pgmap v3457: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:11:17.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:11:16 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:11:17.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:11:16 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:11:17.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:11:16 smithi115 conmon[90393]: 2022-05-14T01:11:16.811+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:11:18.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:11:17 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:11:18.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:11:17 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:11:18.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:11:17 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:11:18.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:11:17 smithi047 ceph-mon[77147]: pgmap v3458: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:11:18.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:11:17 smithi115 conmon[90393]: 2022-05-14T01:11:17.785+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:11:18.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:11:17 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:11:18.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:11:17 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:11:18.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:11:17 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:11:18.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:11:17 smithi115 ceph-mon[78593]: pgmap v3458: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:11:19.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:11:18 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:11:19.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:11:18 smithi115 conmon[90393]: 2022-05-14T01:11:18.752+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:11:19.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:11:18 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:11:20.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:11:19 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:11:20.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:11:19 smithi047 ceph-mon[77147]: pgmap v3459: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:11:20.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:11:19 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 6210 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:11:20.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:11:19 smithi115 conmon[90393]: 2022-05-14T01:11:19.715+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:11:20.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:11:19 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:11:20.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:11:19 smithi115 ceph-mon[78593]: pgmap v3459: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:11:20.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:11:19 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 6210 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:11:21.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:11:20 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:11:21.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:11:20 smithi115 conmon[90393]: 2022-05-14T01:11:20.701+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:11:21.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:11:20 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:11:22.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:11:21 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:11:22.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:11:21 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:11:22.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:11:21 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:11:22.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:11:21 smithi047 ceph-mon[77147]: pgmap v3460: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-05-14T01:11:22.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:11:21 smithi115 conmon[90393]: 2022-05-14T01:11:21.742+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:11:22.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:11:21 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:11:22.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:11:21 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:11:22.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:11:21 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:11:22.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:11:21 smithi115 ceph-mon[78593]: pgmap v3460: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-05-14T01:11:23.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:11:22 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:11:23.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:11:22 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:11:23.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:11:22 smithi115 conmon[90393]: 2022-05-14T01:11:22.693+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:11:24.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:11:23 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:11:24.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:11:23 smithi047 ceph-mon[77147]: pgmap v3461: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:11:24.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:11:23 smithi115 conmon[90393]: 2022-05-14T01:11:23.728+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:11:24.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:11:23 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:11:24.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:11:23 smithi115 ceph-mon[78593]: pgmap v3461: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:11:25.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:11:24 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:11:25.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:11:24 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 6215 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:11:25.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:11:24 smithi115 conmon[90393]: 2022-05-14T01:11:24.682+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:11:25.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:11:24 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:11:25.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:11:24 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 6215 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:11:26.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:11:25 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:11:26.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:11:25 smithi047 ceph-mon[77147]: pgmap v3462: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:11:26.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:11:25 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:11:26.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:11:25 smithi115 ceph-mon[78593]: pgmap v3462: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:11:26.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:11:25 smithi115 conmon[90393]: 2022-05-14T01:11:25.652+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:11:27.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:11:26 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:11:27.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:11:26 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:11:27.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:11:26 smithi115 conmon[90393]: 2022-05-14T01:11:26.647+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:11:28.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:11:27 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:11:28.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:11:27 smithi047 ceph-mon[77147]: pgmap v3463: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:11:28.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:11:27 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:11:28.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:11:27 smithi115 ceph-mon[78593]: pgmap v3463: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:11:28.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:11:27 smithi115 conmon[90393]: 2022-05-14T01:11:27.639+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:11:29.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:11:28 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:11:29.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:11:28 smithi115 conmon[90393]: 2022-05-14T01:11:28.613+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:11:29.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:11:28 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:11:30.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:11:29 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:11:30.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:11:29 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:11:30.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:11:29 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:11:30.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:11:29 smithi047 ceph-mon[77147]: pgmap v3464: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:11:30.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:11:29 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T01:11:30.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:11:29 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T01:11:30.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:11:29 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T01:11:30.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:11:29 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 6220 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:11:30.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:11:29 smithi115 conmon[90393]: 2022-05-14T01:11:29.662+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:11:30.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:11:29 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:11:30.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:11:29 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:11:30.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:11:29 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:11:30.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:11:29 smithi115 ceph-mon[78593]: pgmap v3464: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:11:30.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:11:29 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T01:11:30.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:11:29 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T01:11:30.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:11:29 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T01:11:30.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:11:29 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 6220 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:11:31.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:11:30 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:11:31.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:11:30 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:11:31.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:11:30 smithi115 conmon[90393]: 2022-05-14T01:11:30.662+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:11:32.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:11:31 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:11:32.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:11:31 smithi047 ceph-mon[77147]: pgmap v3465: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-05-14T01:11:32.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:11:31 smithi115 conmon[90393]: 2022-05-14T01:11:31.652+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:11:32.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:11:31 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:11:32.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:11:31 smithi115 ceph-mon[78593]: pgmap v3465: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-05-14T01:11:32.762 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:11:32 smithi047 conmon[78521]: 2022-05-14T01:11:32.335+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:11:32.762 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:11:32 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T01:11:32.762 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:11:32 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:11:32.763 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:11:32 smithi047 conmon[78521]: yield 2022-05-14T01:11:32.763 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:11:32 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:11:32.763 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:11:32 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:11:32.763 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:11:32 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:11:32.764 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:11:32 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T01:11:32.764 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:11:32 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:11:32.764 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:11:32 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T01:11:32.764 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:11:32 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:11:32.765 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:11:32 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T01:11:32.765 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:11:32 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:11:32.765 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:11:32 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T01:11:32.765 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:11:32 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:11:32.766 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:11:32 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T01:11:32.766 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:11:32 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:11:32.766 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:11:32 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:11:32.766 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:11:32 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:11:33.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:11:32 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:11:33.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:11:32 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:11:33.041 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T01:11:33.042 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:11:33.042 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T01:11:33.042 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:11:33.042 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:11:33.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:11:33.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T01:11:33.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:11:33.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T01:11:33.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:11:33.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T01:11:33.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:11:33.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T01:11:33.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:11:33.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T01:11:33.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:11:33.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:11:33.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:11:33.045 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:11:32 smithi047 ceph-mon[77147]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T01:11:33.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:11:32 smithi115 conmon[90393]: 2022-05-14T01:11:32.610+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:11:33.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:11:32 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:11:33.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:11:32 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:11:33.108 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T01:11:33.109 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:11:33.109 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T01:11:33.109 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:11:33.109 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:11:33.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:11:33.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T01:11:33.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:11:33.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T01:11:33.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:11:33.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T01:11:33.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:11:33.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T01:11:33.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:11:33.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T01:11:33.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:11:33.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:11:33.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:11:33.113 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:11:32 smithi115 ceph-mon[78593]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T01:11:34.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:11:33 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:11:34.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:11:33 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:11:34.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:11:33 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:11:34.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:11:33 smithi047 ceph-mon[77147]: pgmap v3466: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:11:34.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:11:33 smithi115 conmon[90393]: 2022-05-14T01:11:33.618+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:11:34.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:11:33 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:11:34.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:11:33 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:11:34.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:11:33 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:11:34.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:11:33 smithi115 ceph-mon[78593]: pgmap v3466: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:11:35.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:11:34 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:11:35.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:11:34 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 6225 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:11:35.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:11:34 smithi115 conmon[90393]: 2022-05-14T01:11:34.667+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:11:35.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:11:34 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:11:35.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:11:34 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 6225 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:11:35.778 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:11:35 smithi047 conmon[78521]: 2022-05-14T01:11:35.407+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:11:35.778 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:11:35 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T01:11:35.778 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:11:35 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:11:35.779 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:11:35 smithi047 conmon[78521]: yield 2022-05-14T01:11:35.779 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:11:35 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:11:35.779 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:11:35 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:11:35.779 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:11:35 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:11:35.779 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:11:35 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T01:11:35.780 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:11:35 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:11:35.780 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:11:35 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T01:11:35.780 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:11:35 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:11:35.780 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:11:35 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T01:11:35.781 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:11:35 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:11:35.781 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:11:35 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T01:11:35.781 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:11:35 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:11:35.781 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:11:35 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T01:11:35.782 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:11:35 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:11:35.782 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:11:35 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:11:35.782 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:11:35 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:11:35.782 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:11:35 smithi047 conmon[78521]: 2022-05-14T01:11:35.408+0000 7f5340779700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T01:11:35.782 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:11:35 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T01:11:35.783 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:11:35 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:11:35.783 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:11:35 smithi047 conmon[78521]: yield 2022-05-14T01:11:35.783 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:11:35 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:11:35.784 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:11:35 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:11:35.784 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:11:35 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:11:35.784 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:11:35 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T01:11:35.784 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:11:35 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:11:35.784 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:11:35 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T01:11:35.785 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:11:35 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:11:35.785 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:11:35 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T01:11:35.786 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:11:35 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:11:35.786 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:11:35 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T01:11:35.786 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:11:35 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:11:35.786 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:11:35 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T01:11:35.787 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:11:35 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:11:35.787 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:11:35 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:11:35.787 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:11:35 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:11:35.787 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:11:35 smithi047 conmon[78521]: 2022-05-14T01:11:35.787 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:11:35 smithi047 conmon[78521]: During handling of the above exception, another exception occurred: 2022-05-14T01:11:35.788 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:11:35 smithi047 conmon[78521]: 2022-05-14T01:11:35.788 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:11:35 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T01:11:35.788 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:11:35 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T01:11:35.788 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:11:35 smithi047 conmon[78521]: return f(*arg) 2022-05-14T01:11:35.789 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:11:35 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T01:11:35.789 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:11:35 smithi047 conmon[78521]: self._write_client_files(client_files, host) 2022-05-14T01:11:35.789 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:11:35 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T01:11:35.789 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:11:35 smithi047 conmon[78521]: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T01:11:35.790 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:11:35 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T01:11:35.790 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:11:35 smithi047 conmon[78521]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T01:11:35.790 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:11:35 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T01:11:35.790 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:11:35 smithi047 conmon[78521]: return self.event_loop.get_result(coro) 2022-05-14T01:11:35.790 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:11:35 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T01:11:35.791 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:11:35 smithi047 conmon[78521]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T01:11:35.791 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:11:35 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T01:11:35.791 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:11:35 smithi047 conmon[78521]: return self.__get_result() 2022-05-14T01:11:35.791 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:11:35 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T01:11:35.792 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:11:35 smithi047 conmon[78521]: raise self._exception 2022-05-14T01:11:35.792 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:11:35 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T01:11:35.792 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:11:35 smithi047 conmon[78521]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T01:11:35.792 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:11:35 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T01:11:35.792 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:11:35 smithi047 conmon[78521]: conn = await self._remote_connection(host, addr) 2022-05-14T01:11:35.793 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:11:35 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T01:11:35.793 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:11:35 smithi047 conmon[78521]: raise 2022-05-14T01:11:35.793 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:11:35 smithi047 conmon[78521]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T01:11:35.793 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:11:35 smithi047 conmon[78521]: self.gen.throw(type, value, traceback) 2022-05-14T01:11:35.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:11:35 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T01:11:35.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:11:35 smithi047 conmon[78521]: raise OrchestratorError(msg) 2022-05-14T01:11:35.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:11:35 smithi047 conmon[78521]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:11:36.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:11:35 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:11:36.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:11:35 smithi047 ceph-mon[77147]: pgmap v3467: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:11:36.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:11:35 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:11:36.042 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T01:11:36.042 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:11:36.042 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T01:11:36.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:11:36.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:11:36.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:11:36.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T01:11:36.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:11:36.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T01:11:36.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:11:36.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T01:11:36.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:11:36.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T01:11:36.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:11:36.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T01:11:36.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:11:36.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:11:36.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:11:36.046 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:11:35 smithi047 ceph-mon[77147]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T01:11:36.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T01:11:36.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:11:36.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T01:11:36.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:11:36.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:11:36.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:11:36.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T01:11:36.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:11:36.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T01:11:36.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:11:36.049 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T01:11:36.049 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:11:36.049 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T01:11:36.049 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:11:36.050 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T01:11:36.050 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:11:36.051 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:11:36.051 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:11:36.051 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T01:11:36.052 INFO:journalctl@ceph.mon.b.smithi047.stdout: During handling of the above exception, another exception occurred: 2022-05-14T01:11:36.052 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T01:11:36.052 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T01:11:36.052 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T01:11:36.053 INFO:journalctl@ceph.mon.b.smithi047.stdout: return f(*arg) 2022-05-14T01:11:36.053 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T01:11:36.053 INFO:journalctl@ceph.mon.b.smithi047.stdout: self._write_client_files(client_files, host) 2022-05-14T01:11:36.053 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T01:11:36.053 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T01:11:36.054 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T01:11:36.054 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T01:11:36.054 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T01:11:36.054 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.event_loop.get_result(coro) 2022-05-14T01:11:36.054 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T01:11:36.055 INFO:journalctl@ceph.mon.b.smithi047.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T01:11:36.055 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T01:11:36.055 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.__get_result() 2022-05-14T01:11:36.055 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T01:11:36.055 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise self._exception 2022-05-14T01:11:36.056 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T01:11:36.056 INFO:journalctl@ceph.mon.b.smithi047.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T01:11:36.056 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T01:11:36.056 INFO:journalctl@ceph.mon.b.smithi047.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T01:11:36.057 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T01:11:36.057 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise 2022-05-14T01:11:36.057 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T01:11:36.057 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.gen.throw(type, value, traceback) 2022-05-14T01:11:36.057 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T01:11:36.058 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OrchestratorError(msg) 2022-05-14T01:11:36.058 INFO:journalctl@ceph.mon.b.smithi047.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:11:36.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:11:35 smithi115 conmon[90393]: 2022-05-14T01:11:35.680+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:11:36.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:11:35 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:11:36.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:11:35 smithi115 ceph-mon[78593]: pgmap v3467: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:11:36.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:11:35 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:11:36.109 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T01:11:36.109 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:11:36.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T01:11:36.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:11:36.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:11:36.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:11:36.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T01:11:36.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:11:36.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T01:11:36.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:11:36.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T01:11:36.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:11:36.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T01:11:36.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:11:36.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T01:11:36.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:11:36.113 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:11:36.113 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:11:36.113 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:11:35 smithi115 ceph-mon[78593]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T01:11:36.113 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T01:11:36.113 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:11:36.114 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T01:11:36.114 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:11:36.114 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:11:36.114 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:11:36.115 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T01:11:36.115 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:11:36.115 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T01:11:36.116 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:11:36.116 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T01:11:36.116 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:11:36.116 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T01:11:36.116 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:11:36.117 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T01:11:36.117 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:11:36.117 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:11:36.117 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:11:36.117 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T01:11:36.118 INFO:journalctl@ceph.mon.c.smithi115.stdout: During handling of the above exception, another exception occurred: 2022-05-14T01:11:36.118 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T01:11:36.118 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T01:11:36.118 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T01:11:36.119 INFO:journalctl@ceph.mon.c.smithi115.stdout: return f(*arg) 2022-05-14T01:11:36.119 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T01:11:36.119 INFO:journalctl@ceph.mon.c.smithi115.stdout: self._write_client_files(client_files, host) 2022-05-14T01:11:36.119 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T01:11:36.119 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T01:11:36.120 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T01:11:36.120 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T01:11:36.120 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T01:11:36.121 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.event_loop.get_result(coro) 2022-05-14T01:11:36.121 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T01:11:36.121 INFO:journalctl@ceph.mon.c.smithi115.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T01:11:36.121 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T01:11:36.122 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.__get_result() 2022-05-14T01:11:36.122 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T01:11:36.122 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise self._exception 2022-05-14T01:11:36.122 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T01:11:36.123 INFO:journalctl@ceph.mon.c.smithi115.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T01:11:36.123 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T01:11:36.123 INFO:journalctl@ceph.mon.c.smithi115.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T01:11:36.123 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T01:11:36.123 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise 2022-05-14T01:11:36.124 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T01:11:36.124 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.gen.throw(type, value, traceback) 2022-05-14T01:11:36.124 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T01:11:36.124 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OrchestratorError(msg) 2022-05-14T01:11:36.124 INFO:journalctl@ceph.mon.c.smithi115.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:11:37.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:11:36 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:11:37.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:11:36 smithi115 conmon[90393]: 2022-05-14T01:11:36.635+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:11:37.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:11:36 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:11:37.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:11:37 smithi115 conmon[90393]: 2022-05-14T01:11:37.592+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:11:37.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:11:37 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:11:37.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:11:37 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:11:37.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:11:37 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:11:37.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:11:37 smithi115 ceph-mon[78593]: pgmap v3468: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:11:38.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:11:37 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:11:38.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:11:37 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:11:38.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:11:37 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:11:38.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:11:37 smithi047 ceph-mon[77147]: pgmap v3468: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:11:38.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:11:38 smithi115 conmon[90393]: 2022-05-14T01:11:38.594+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:11:38.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:11:38 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:11:39.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:11:38 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:11:39.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:11:39 smithi115 conmon[90393]: 2022-05-14T01:11:39.597+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:11:39.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:11:39 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:11:39.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:11:39 smithi115 ceph-mon[78593]: pgmap v3469: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:11:39.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:11:39 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 6230 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:11:40.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:11:39 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:11:40.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:11:39 smithi047 ceph-mon[77147]: pgmap v3469: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:11:40.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:11:39 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 6230 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:11:41.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:11:40 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:11:41.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:11:40 smithi115 conmon[90393]: 2022-05-14T01:11:40.633+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:11:41.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:11:40 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:11:42.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:11:41 smithi115 conmon[90393]: 2022-05-14T01:11:41.617+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:11:42.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:11:41 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:11:42.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:11:41 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:11:42.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:11:41 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:11:42.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:11:41 smithi115 ceph-mon[78593]: pgmap v3470: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-05-14T01:11:42.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:11:41 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:11:42.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:11:41 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:11:42.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:11:41 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:11:42.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:11:41 smithi047 ceph-mon[77147]: pgmap v3470: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-05-14T01:11:43.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:11:42 smithi115 conmon[90393]: 2022-05-14T01:11:42.667+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:11:43.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:11:42 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:11:43.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:11:42 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:11:44.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:11:43 smithi115 conmon[90393]: 2022-05-14T01:11:43.661+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:11:44.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:11:43 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:11:44.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:11:43 smithi115 ceph-mon[78593]: pgmap v3471: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:11:44.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:11:43 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:11:44.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:11:43 smithi047 ceph-mon[77147]: pgmap v3471: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:11:45.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:11:44 smithi115 conmon[90393]: 2022-05-14T01:11:44.612+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:11:45.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:11:44 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:11:45.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:11:44 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 6235 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:11:45.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:11:44 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:11:45.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:11:44 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 6235 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:11:45.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:11:45 smithi115 conmon[90393]: 2022-05-14T01:11:45.590+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:11:45.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:11:45 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:11:45.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:11:45 smithi115 ceph-mon[78593]: pgmap v3472: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:11:46.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:11:45 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:11:46.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:11:45 smithi047 ceph-mon[77147]: pgmap v3472: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:11:47.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:11:46 smithi115 conmon[90393]: 2022-05-14T01:11:46.639+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:11:47.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:11:46 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:11:47.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:11:46 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:11:48.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:11:47 smithi115 conmon[90393]: 2022-05-14T01:11:47.645+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:11:48.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:11:47 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:11:48.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:11:47 smithi115 ceph-mon[78593]: pgmap v3473: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:11:48.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:11:47 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:11:48.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:11:47 smithi047 ceph-mon[77147]: pgmap v3473: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:11:49.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:11:48 smithi115 conmon[90393]: 2022-05-14T01:11:48.671+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:11:49.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:11:48 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:11:49.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:11:48 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:11:50.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:11:49 smithi115 conmon[90393]: 2022-05-14T01:11:49.637+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:11:50.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:11:49 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:11:50.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:11:49 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:11:50.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:11:49 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:11:50.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:11:49 smithi115 ceph-mon[78593]: pgmap v3474: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:11:50.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:11:49 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 6240 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:11:50.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:11:49 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:11:50.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:11:49 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:11:50.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:11:49 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:11:50.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:11:49 smithi047 ceph-mon[77147]: pgmap v3474: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:11:50.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:11:49 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 6240 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:11:51.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:11:50 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:11:51.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:11:50 smithi115 conmon[90393]: 2022-05-14T01:11:50.647+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:11:51.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:11:50 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:11:52.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:11:51 smithi115 conmon[90393]: 2022-05-14T01:11:51.632+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:11:52.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:11:51 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:11:52.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:11:51 smithi115 ceph-mon[78593]: pgmap v3475: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-05-14T01:11:52.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:11:51 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:11:52.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:11:51 smithi047 ceph-mon[77147]: pgmap v3475: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-05-14T01:11:53.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:11:52 smithi115 conmon[90393]: 2022-05-14T01:11:52.674+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:11:53.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:11:52 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:11:53.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:11:52 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:11:54.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:11:53 smithi115 conmon[90393]: 2022-05-14T01:11:53.664+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:11:54.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:11:53 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:11:54.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:11:53 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:11:54.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:11:53 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:11:54.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:11:53 smithi115 ceph-mon[78593]: pgmap v3476: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:11:54.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:11:53 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:11:54.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:11:53 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:11:54.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:11:53 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:11:54.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:11:53 smithi047 ceph-mon[77147]: pgmap v3476: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:11:55.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:11:54 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:11:55.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:11:54 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 6245 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:11:55.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:11:54 smithi115 conmon[90393]: 2022-05-14T01:11:54.637+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:11:55.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:11:54 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:11:55.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:11:54 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 6245 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:11:56.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:11:55 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:11:56.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:11:55 smithi115 ceph-mon[78593]: pgmap v3477: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:11:56.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:11:55 smithi115 conmon[90393]: 2022-05-14T01:11:55.646+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:11:56.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:11:55 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:11:56.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:11:55 smithi047 ceph-mon[77147]: pgmap v3477: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:11:57.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:11:56 smithi115 conmon[90393]: 2022-05-14T01:11:56.659+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:11:57.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:11:56 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:11:57.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:11:56 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:11:58.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:11:57 smithi115 conmon[90393]: 2022-05-14T01:11:57.690+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:11:58.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:11:57 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:11:58.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:11:57 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:11:58.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:11:57 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:11:58.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:11:57 smithi115 ceph-mon[78593]: pgmap v3478: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:11:58.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:11:57 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:11:58.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:11:57 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:11:58.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:11:57 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:11:58.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:11:57 smithi047 ceph-mon[77147]: pgmap v3478: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:11:59.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:11:58 smithi115 conmon[90393]: 2022-05-14T01:11:58.673+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:11:59.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:11:58 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:11:59.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:11:58 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:12:00.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:11:59 smithi115 conmon[90393]: 2022-05-14T01:11:59.630+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:12:00.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:11:59 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:12:00.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:11:59 smithi115 ceph-mon[78593]: pgmap v3479: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:12:00.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:11:59 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 6250 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:12:00.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:11:59 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:12:00.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:11:59 smithi047 ceph-mon[77147]: pgmap v3479: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:12:00.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:11:59 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 6250 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:12:01.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:12:00 smithi115 conmon[90393]: 2022-05-14T01:12:00.675+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:12:01.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:12:00 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:12:01.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:12:00 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:12:02.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:12:01 smithi115 conmon[90393]: 2022-05-14T01:12:01.706+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:12:02.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:12:01 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:12:02.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:12:01 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:12:02.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:12:01 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:12:02.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:12:01 smithi115 ceph-mon[78593]: pgmap v3480: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-05-14T01:12:02.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:12:01 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:12:02.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:12:01 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:12:02.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:12:01 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:12:02.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:12:01 smithi047 ceph-mon[77147]: pgmap v3480: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-05-14T01:12:03.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:12:02 smithi115 conmon[90393]: 2022-05-14T01:12:02.693+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:12:03.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:12:02 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:12:03.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:12:02 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:12:04.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:12:03 smithi115 conmon[90393]: 2022-05-14T01:12:03.719+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:12:04.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:12:03 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:12:04.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:12:03 smithi115 ceph-mon[78593]: pgmap v3481: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:12:04.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:12:03 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:12:04.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:12:03 smithi047 ceph-mon[77147]: pgmap v3481: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:12:05.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:12:04 smithi115 conmon[90393]: 2022-05-14T01:12:04.697+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:12:05.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:12:04 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:12:05.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:12:04 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 6255 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:12:05.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:12:04 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:12:05.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:12:04 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 6255 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:12:05.903 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:12:05 smithi115 conmon[90393]: 2022-05-14T01:12:05.653+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:12:05.904 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:12:05 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:12:05.905 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:12:05 smithi115 ceph-mon[78593]: pgmap v3482: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:12:06.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:12:05 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:12:06.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:12:05 smithi047 ceph-mon[77147]: pgmap v3482: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:12:07.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:12:06 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:12:07.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:12:06 smithi115 conmon[90393]: 2022-05-14T01:12:06.689+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:12:07.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:12:06 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:12:08.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:12:07 smithi115 conmon[90393]: 2022-05-14T01:12:07.718+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:12:08.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:12:07 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:12:08.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:12:07 smithi115 ceph-mon[78593]: pgmap v3483: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:12:08.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:12:07 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:12:08.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:12:07 smithi047 ceph-mon[77147]: pgmap v3483: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:12:09.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:12:08 smithi115 conmon[90393]: 2022-05-14T01:12:08.704+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:12:09.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:12:08 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:12:09.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:12:08 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:12:10.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:12:09 smithi115 conmon[90393]: 2022-05-14T01:12:09.724+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:12:10.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:12:09 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:12:10.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:12:09 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:12:10.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:12:09 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:12:10.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:12:09 smithi115 ceph-mon[78593]: pgmap v3484: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:12:10.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:12:09 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 6260 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:12:10.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:12:09 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:12:10.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:12:09 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:12:10.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:12:09 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:12:10.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:12:09 smithi047 ceph-mon[77147]: pgmap v3484: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:12:10.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:12:09 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 6260 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:12:11.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:12:10 smithi115 conmon[90393]: 2022-05-14T01:12:10.731+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:12:11.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:12:10 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:12:11.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:12:10 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:12:12.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:12:11 smithi115 conmon[90393]: 2022-05-14T01:12:11.760+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:12:12.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:12:11 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:12:12.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:12:11 smithi115 ceph-mon[78593]: pgmap v3485: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-05-14T01:12:12.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:12:11 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:12:12.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:12:11 smithi047 ceph-mon[77147]: pgmap v3485: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-05-14T01:12:13.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:12:12 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:12:13.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:12:12 smithi115 conmon[90393]: 2022-05-14T01:12:12.741+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:12:13.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:12:12 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:12:14.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:12:13 smithi115 conmon[90393]: 2022-05-14T01:12:13.739+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:12:14.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:12:13 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:12:14.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:12:13 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:12:14.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:12:13 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:12:14.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:12:13 smithi115 ceph-mon[78593]: pgmap v3486: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:12:14.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:12:13 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T01:12:14.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:12:13 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T01:12:14.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:12:13 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:12:14.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:12:13 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:12:14.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:12:13 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:12:14.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:12:13 smithi047 ceph-mon[77147]: pgmap v3486: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:12:14.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:12:13 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T01:12:14.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:12:13 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T01:12:15.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:12:14 smithi115 conmon[90393]: 2022-05-14T01:12:14.710+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:12:15.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:12:14 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:12:15.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:12:14 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 6265 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:12:15.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:12:14 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:12:15.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:12:14 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 6265 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:12:16.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:12:15 smithi115 conmon[90393]: 2022-05-14T01:12:15.725+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:12:16.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:12:15 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:12:16.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:12:15 smithi115 ceph-mon[78593]: pgmap v3487: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:12:16.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:12:15 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:12:16.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:12:15 smithi047 ceph-mon[77147]: pgmap v3487: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:12:17.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:12:16 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:12:17.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:12:16 smithi115 conmon[90393]: 2022-05-14T01:12:16.729+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:12:17.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:12:16 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:12:18.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:12:17 smithi115 conmon[90393]: 2022-05-14T01:12:17.702+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:12:18.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:12:17 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:12:18.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:12:17 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:12:18.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:12:17 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:12:18.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:12:17 smithi115 ceph-mon[78593]: pgmap v3488: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:12:18.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:12:17 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:12:18.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:12:17 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:12:18.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:12:17 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:12:18.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:12:17 smithi047 ceph-mon[77147]: pgmap v3488: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:12:18.958 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:12:18 smithi115 conmon[90393]: 2022-05-14T01:12:18.698+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:12:19.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:12:18 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:12:19.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:12:18 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:12:19.962 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:12:19 smithi115 conmon[90393]: 2022-05-14T01:12:19.679+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:12:20.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:12:19 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:12:20.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:12:19 smithi047 ceph-mon[77147]: pgmap v3489: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:12:20.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:12:19 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 6270 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:12:20.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:12:19 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:12:20.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:12:19 smithi115 ceph-mon[78593]: pgmap v3489: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:12:20.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:12:19 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 6270 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:12:21.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:12:20 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:12:21.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:12:20 smithi115 conmon[90393]: 2022-05-14T01:12:20.720+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:12:21.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:12:20 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:12:22.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:12:21 smithi115 conmon[90393]: 2022-05-14T01:12:21.768+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:12:22.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:12:21 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:12:22.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:12:21 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:12:22.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:12:21 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:12:22.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:12:21 smithi115 ceph-mon[78593]: pgmap v3490: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-05-14T01:12:22.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:12:21 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:12:22.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:12:21 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:12:22.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:12:21 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:12:22.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:12:21 smithi047 ceph-mon[77147]: pgmap v3490: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-05-14T01:12:23.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:12:22 smithi115 conmon[90393]: 2022-05-14T01:12:22.793+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:12:23.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:12:22 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:12:23.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:12:22 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:12:24.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:12:23 smithi115 conmon[90393]: 2022-05-14T01:12:23.774+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:12:24.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:12:23 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:12:24.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:12:23 smithi115 ceph-mon[78593]: pgmap v3491: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:12:24.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:12:23 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:12:24.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:12:23 smithi047 ceph-mon[77147]: pgmap v3491: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:12:25.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:12:24 smithi115 conmon[90393]: 2022-05-14T01:12:24.742+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:12:25.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:12:24 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:12:25.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:12:24 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 6275 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:12:25.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:12:24 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:12:25.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:12:24 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 6275 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:12:26.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:12:25 smithi115 conmon[90393]: 2022-05-14T01:12:25.773+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:12:26.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:12:25 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:12:26.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:12:25 smithi115 ceph-mon[78593]: pgmap v3492: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:12:26.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:12:25 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:12:26.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:12:25 smithi047 ceph-mon[77147]: pgmap v3492: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:12:27.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:12:26 smithi115 conmon[90393]: 2022-05-14T01:12:26.780+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:12:27.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:12:26 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:12:27.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:12:26 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:12:28.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:12:27 smithi115 conmon[90393]: 2022-05-14T01:12:27.791+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:12:28.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:12:27 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:12:28.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:12:27 smithi115 ceph-mon[78593]: pgmap v3493: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:12:28.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:12:27 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:12:28.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:12:27 smithi047 ceph-mon[77147]: pgmap v3493: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:12:29.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:12:28 smithi115 conmon[90393]: 2022-05-14T01:12:28.841+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:12:29.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:12:28 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:12:29.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:12:28 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:12:30.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:12:30 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:12:30.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:12:30 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:12:30.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:12:30 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:12:30.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:12:30 smithi047 ceph-mon[77147]: pgmap v3494: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:12:30.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:12:30 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 6280 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:12:30.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:12:29 smithi115 conmon[90393]: 2022-05-14T01:12:29.878+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:12:30.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:12:30 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:12:30.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:12:30 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:12:30.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:12:30 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:12:30.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:12:30 smithi115 ceph-mon[78593]: pgmap v3494: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:12:30.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:12:30 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 6280 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:12:31.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:12:30 smithi115 conmon[90393]: 2022-05-14T01:12:30.855+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:12:31.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:12:31 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:12:31.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:12:31 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:12:32.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:12:31 smithi115 conmon[90393]: 2022-05-14T01:12:31.832+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:12:32.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:12:32 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:12:32.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:12:32 smithi115 ceph-mon[78593]: pgmap v3495: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-05-14T01:12:32.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:12:32 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:12:32.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:12:32 smithi047 ceph-mon[77147]: pgmap v3495: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-05-14T01:12:33.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:12:32 smithi115 conmon[90393]: 2022-05-14T01:12:32.835+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:12:33.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:12:33 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:12:33.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:12:33 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:12:34.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:12:33 smithi115 conmon[90393]: 2022-05-14T01:12:33.800+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:12:34.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:12:34 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:12:34.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:12:34 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:12:34.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:12:34 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:12:34.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:12:34 smithi115 ceph-mon[78593]: pgmap v3496: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:12:34.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:12:34 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:12:34.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:12:34 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:12:34.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:12:34 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:12:34.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:12:34 smithi047 ceph-mon[77147]: pgmap v3496: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:12:35.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:12:34 smithi115 conmon[90393]: 2022-05-14T01:12:34.795+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:12:35.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:12:35 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:12:35.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:12:35 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 6285 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:12:35.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:12:35 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:12:35.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:12:35 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 6285 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:12:36.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:12:35 smithi115 conmon[90393]: 2022-05-14T01:12:35.833+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:12:36.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:12:36 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:12:36.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:12:36 smithi115 ceph-mon[78593]: pgmap v3497: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:12:36.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:12:36 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T01:12:36.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:12:36 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T01:12:36.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:12:36 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T01:12:36.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:12:36 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:12:36.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:12:36 smithi047 ceph-mon[77147]: pgmap v3497: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:12:36.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:12:36 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T01:12:36.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:12:36 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T01:12:36.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:12:36 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T01:12:36.292 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:12:36 smithi047 conmon[78521]: 2022-05-14T01:12:36.016+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:12:36.292 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:12:36 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T01:12:36.293 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:12:36 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:12:36.293 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:12:36 smithi047 conmon[78521]: yield 2022-05-14T01:12:36.293 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:12:36 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:12:36.293 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:12:36 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:12:36.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:12:36 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:12:36.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:12:36 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T01:12:36.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:12:36 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:12:36.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:12:36 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T01:12:36.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:12:36 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:12:36.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:12:36 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T01:12:36.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:12:36 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:12:36.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:12:36 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T01:12:36.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:12:36 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:12:36.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:12:36 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T01:12:36.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:12:36 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:12:36.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:12:36 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:12:36.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:12:36 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:12:37.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:12:36 smithi115 conmon[90393]: 2022-05-14T01:12:36.784+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:12:37.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:12:37 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:12:37.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:12:37 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:12:37.108 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T01:12:37.109 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:12:37.109 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T01:12:37.109 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:12:37.109 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:12:37.109 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:12:37.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T01:12:37.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:12:37.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T01:12:37.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:12:37.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T01:12:37.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:12:37.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T01:12:37.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:12:37.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T01:12:37.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:12:37.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:12:37.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:12:37.112 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:12:37 smithi115 ceph-mon[78593]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T01:12:37.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:12:37 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:12:37.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:12:37 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:12:37.292 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T01:12:37.292 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:12:37.292 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T01:12:37.292 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:12:37.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:12:37.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:12:37.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T01:12:37.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:12:37.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T01:12:37.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:12:37.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T01:12:37.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:12:37.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T01:12:37.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:12:37.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T01:12:37.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:12:37.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:12:37.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:12:37.296 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:12:37 smithi047 ceph-mon[77147]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T01:12:38.037 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:12:37 smithi115 conmon[90393]: 2022-05-14T01:12:37.762+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:12:38.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:12:38 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:12:38.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:12:38 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:12:38.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:12:38 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:12:38.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:12:38 smithi047 ceph-mon[77147]: pgmap v3498: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:12:38.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:12:38 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:12:38.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:12:38 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:12:38.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:12:38 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:12:38.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:12:38 smithi115 ceph-mon[78593]: pgmap v3498: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:12:39.039 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:12:38 smithi115 conmon[90393]: 2022-05-14T01:12:38.747+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:12:39.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:12:39 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:12:39.291 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:12:39 smithi047 conmon[78521]: 2022-05-14T01:12:39.088+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:12:39.292 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:12:39 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T01:12:39.292 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:12:39 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:12:39.292 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:12:39 smithi047 conmon[78521]: yield 2022-05-14T01:12:39.292 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:12:39 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:12:39.292 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:12:39 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:12:39.293 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:12:39 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:12:39.293 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:12:39 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T01:12:39.293 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:12:39 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:12:39.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:12:39 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T01:12:39.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:12:39 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:12:39.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:12:39 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T01:12:39.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:12:39 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:12:39.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:12:39 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T01:12:39.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:12:39 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:12:39.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:12:39 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T01:12:39.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:12:39 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:12:39.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:12:39 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:12:39.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:12:39 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:12:39.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:12:39 smithi047 conmon[78521]: 2022-05-14T01:12:39.089+0000 7f5344781700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T01:12:39.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:12:39 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T01:12:39.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:12:39 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:12:39.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:12:39 smithi047 conmon[78521]: yield 2022-05-14T01:12:39.297 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:12:39 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:12:39.297 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:12:39 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:12:39.297 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:12:39 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:12:39.297 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:12:39 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T01:12:39.298 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:12:39 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:12:39.298 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:12:39 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T01:12:39.298 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:12:39 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:12:39.299 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:12:39 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T01:12:39.299 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:12:39 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:12:39.299 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:12:39 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T01:12:39.299 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:12:39 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:12:39.299 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:12:39 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T01:12:39.300 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:12:39 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:12:39.300 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:12:39 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:12:39.300 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:12:39 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:12:39.300 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:12:39 smithi047 conmon[78521]: 2022-05-14T01:12:39.301 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:12:39 smithi047 conmon[78521]: During handling of the above exception, another exception occurred: 2022-05-14T01:12:39.301 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:12:39 smithi047 conmon[78521]: 2022-05-14T01:12:39.301 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:12:39 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T01:12:39.301 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:12:39 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T01:12:39.301 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:12:39 smithi047 conmon[78521]: return f(*arg) 2022-05-14T01:12:39.302 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:12:39 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T01:12:39.302 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:12:39 smithi047 conmon[78521]: self._write_client_files(client_files, host) 2022-05-14T01:12:39.302 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:12:39 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T01:12:39.302 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:12:39 smithi047 conmon[78521]: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T01:12:39.302 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:12:39 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T01:12:39.303 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:12:39 smithi047 conmon[78521]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T01:12:39.303 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:12:39 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T01:12:39.303 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:12:39 smithi047 conmon[78521]: return self.event_loop.get_result(coro) 2022-05-14T01:12:39.304 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:12:39 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T01:12:39.304 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:12:39 smithi047 conmon[78521]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T01:12:39.304 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:12:39 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T01:12:39.304 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:12:39 smithi047 conmon[78521]: return self.__get_result() 2022-05-14T01:12:39.304 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:12:39 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T01:12:39.305 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:12:39 smithi047 conmon[78521]: raise self._exception 2022-05-14T01:12:39.305 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:12:39 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T01:12:39.305 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:12:39 smithi047 conmon[78521]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T01:12:39.305 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:12:39 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T01:12:39.306 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:12:39 smithi047 conmon[78521]: conn = await self._remote_connection(host, addr) 2022-05-14T01:12:39.306 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:12:39 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T01:12:39.306 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:12:39 smithi047 conmon[78521]: raise 2022-05-14T01:12:39.306 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:12:39 smithi047 conmon[78521]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T01:12:39.307 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:12:39 smithi047 conmon[78521]: self.gen.throw(type, value, traceback) 2022-05-14T01:12:39.307 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:12:39 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T01:12:39.307 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:12:39 smithi047 conmon[78521]: raise OrchestratorError(msg) 2022-05-14T01:12:39.307 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:12:39 smithi047 conmon[78521]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:12:39.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:12:39 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:12:40.044 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:12:39 smithi115 conmon[90393]: 2022-05-14T01:12:39.762+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:12:40.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:12:40 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:12:40.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:12:40 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:12:40.358 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T01:12:40.359 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:12:40.359 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T01:12:40.359 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:12:40.359 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:12:40.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:12:40.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T01:12:40.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:12:40.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T01:12:40.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:12:40.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T01:12:40.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:12:40.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T01:12:40.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:12:40.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T01:12:40.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:12:40.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:12:40.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:12:40.363 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:12:40 smithi115 ceph-mon[78593]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T01:12:40.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T01:12:40.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:12:40.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T01:12:40.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:12:40.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:12:40.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:12:40.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T01:12:40.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:12:40.365 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T01:12:40.365 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:12:40.365 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T01:12:40.366 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:12:40.366 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T01:12:40.366 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:12:40.366 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T01:12:40.366 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:12:40.367 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:12:40.367 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:12:40.367 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T01:12:40.367 INFO:journalctl@ceph.mon.c.smithi115.stdout: During handling of the above exception, another exception occurred: 2022-05-14T01:12:40.368 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T01:12:40.368 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T01:12:40.368 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T01:12:40.368 INFO:journalctl@ceph.mon.c.smithi115.stdout: return f(*arg) 2022-05-14T01:12:40.368 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T01:12:40.369 INFO:journalctl@ceph.mon.c.smithi115.stdout: self._write_client_files(client_files, host) 2022-05-14T01:12:40.369 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T01:12:40.369 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T01:12:40.369 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T01:12:40.370 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T01:12:40.370 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T01:12:40.371 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.event_loop.get_result(coro) 2022-05-14T01:12:40.371 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T01:12:40.371 INFO:journalctl@ceph.mon.c.smithi115.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T01:12:40.372 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T01:12:40.372 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.__get_result() 2022-05-14T01:12:40.373 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T01:12:40.373 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise self._exception 2022-05-14T01:12:40.374 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T01:12:40.374 INFO:journalctl@ceph.mon.c.smithi115.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T01:12:40.374 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T01:12:40.375 INFO:journalctl@ceph.mon.c.smithi115.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T01:12:40.375 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T01:12:40.376 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise 2022-05-14T01:12:40.376 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T01:12:40.377 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.gen.throw(type, value, traceback) 2022-05-14T01:12:40.377 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T01:12:40.377 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OrchestratorError(msg) 2022-05-14T01:12:40.378 INFO:journalctl@ceph.mon.c.smithi115.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:12:40.378 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:12:40 smithi115 ceph-mon[78593]: pgmap v3499: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:12:40.378 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:12:40 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 6290 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:12:40.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:12:40 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:12:40.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:12:40 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:12:40.542 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T01:12:40.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:12:40.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T01:12:40.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:12:40.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:12:40.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:12:40.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T01:12:40.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:12:40.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T01:12:40.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:12:40.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T01:12:40.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:12:40.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T01:12:40.548 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:12:40.548 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T01:12:40.549 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:12:40.549 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:12:40.549 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:12:40.549 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:12:40 smithi047 ceph-mon[77147]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T01:12:40.550 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T01:12:40.550 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:12:40.550 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T01:12:40.551 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:12:40.551 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:12:40.551 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:12:40.552 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T01:12:40.552 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:12:40.552 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T01:12:40.552 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:12:40.553 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T01:12:40.553 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:12:40.553 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T01:12:40.553 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:12:40.553 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T01:12:40.554 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:12:40.554 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:12:40.554 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:12:40.554 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T01:12:40.554 INFO:journalctl@ceph.mon.b.smithi047.stdout: During handling of the above exception, another exception occurred: 2022-05-14T01:12:40.555 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T01:12:40.555 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T01:12:40.555 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T01:12:40.555 INFO:journalctl@ceph.mon.b.smithi047.stdout: return f(*arg) 2022-05-14T01:12:40.555 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T01:12:40.556 INFO:journalctl@ceph.mon.b.smithi047.stdout: self._write_client_files(client_files, host) 2022-05-14T01:12:40.556 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T01:12:40.556 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T01:12:40.557 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T01:12:40.557 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T01:12:40.557 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T01:12:40.558 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.event_loop.get_result(coro) 2022-05-14T01:12:40.558 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T01:12:40.558 INFO:journalctl@ceph.mon.b.smithi047.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T01:12:40.559 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T01:12:40.559 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.__get_result() 2022-05-14T01:12:40.560 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T01:12:40.560 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise self._exception 2022-05-14T01:12:40.560 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T01:12:40.561 INFO:journalctl@ceph.mon.b.smithi047.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T01:12:40.561 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T01:12:40.562 INFO:journalctl@ceph.mon.b.smithi047.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T01:12:40.562 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T01:12:40.562 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise 2022-05-14T01:12:40.563 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T01:12:40.563 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.gen.throw(type, value, traceback) 2022-05-14T01:12:40.563 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T01:12:40.564 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OrchestratorError(msg) 2022-05-14T01:12:40.564 INFO:journalctl@ceph.mon.b.smithi047.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:12:40.564 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:12:40 smithi047 ceph-mon[77147]: pgmap v3499: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:12:40.565 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:12:40 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 6290 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:12:41.046 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:12:40 smithi115 conmon[90393]: 2022-05-14T01:12:40.743+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:12:41.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:12:41 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:12:41.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:12:41 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:12:42.054 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:12:41 smithi115 conmon[90393]: 2022-05-14T01:12:41.702+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:12:42.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:12:42 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:12:42.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:12:42 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:12:42.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:12:42 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:12:42.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:12:42 smithi115 ceph-mon[78593]: pgmap v3500: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-05-14T01:12:42.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:12:42 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:12:42.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:12:42 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:12:42.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:12:42 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:12:42.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:12:42 smithi047 ceph-mon[77147]: pgmap v3500: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-05-14T01:12:43.056 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:12:42 smithi115 conmon[90393]: 2022-05-14T01:12:42.698+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:12:43.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:12:43 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:12:43.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:12:43 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:12:44.061 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:12:43 smithi115 conmon[90393]: 2022-05-14T01:12:43.677+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:12:44.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:12:44 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:12:44.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:12:44 smithi115 ceph-mon[78593]: pgmap v3501: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:12:44.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:12:44 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:12:44.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:12:44 smithi047 ceph-mon[77147]: pgmap v3501: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:12:45.064 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:12:44 smithi115 conmon[90393]: 2022-05-14T01:12:44.726+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:12:45.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:12:45 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:12:45.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:12:45 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 6295 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:12:45.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:12:45 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:12:45.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:12:45 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 6295 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:12:46.073 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:12:45 smithi115 conmon[90393]: 2022-05-14T01:12:45.771+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:12:46.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:12:46 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:12:46.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:12:46 smithi115 ceph-mon[78593]: pgmap v3502: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:12:46.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:12:46 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:12:46.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:12:46 smithi047 ceph-mon[77147]: pgmap v3502: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:12:47.070 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:12:46 smithi115 conmon[90393]: 2022-05-14T01:12:46.762+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:12:47.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:12:47 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:12:47.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:12:47 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:12:48.074 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:12:47 smithi115 conmon[90393]: 2022-05-14T01:12:47.790+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:12:48.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:12:48 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:12:48.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:12:48 smithi115 ceph-mon[78593]: pgmap v3503: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:12:48.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:12:48 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:12:48.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:12:48 smithi047 ceph-mon[77147]: pgmap v3503: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:12:49.076 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:12:48 smithi115 conmon[90393]: 2022-05-14T01:12:48.817+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:12:49.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:12:49 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:12:49.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:12:49 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:12:50.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:12:49 smithi115 conmon[90393]: 2022-05-14T01:12:49.840+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:12:50.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:12:50 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:12:50.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:12:50 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:12:50.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:12:50 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:12:50.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:12:50 smithi115 ceph-mon[78593]: pgmap v3504: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:12:50.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:12:50 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 6300 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:12:50.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:12:50 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:12:50.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:12:50 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:12:50.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:12:50 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:12:50.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:12:50 smithi047 ceph-mon[77147]: pgmap v3504: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:12:50.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:12:50 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 6300 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:12:51.087 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:12:50 smithi115 conmon[90393]: 2022-05-14T01:12:50.834+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:12:51.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:12:51 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:12:51.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:12:51 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:12:52.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:12:51 smithi115 conmon[90393]: 2022-05-14T01:12:51.875+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:12:52.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:12:52 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:12:52.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:12:52 smithi115 ceph-mon[78593]: pgmap v3505: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-05-14T01:12:52.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:12:52 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:12:52.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:12:52 smithi047 ceph-mon[77147]: pgmap v3505: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-05-14T01:12:53.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:12:52 smithi115 conmon[90393]: 2022-05-14T01:12:52.855+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:12:53.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:12:53 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:12:53.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:12:53 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:12:54.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:12:53 smithi115 conmon[90393]: 2022-05-14T01:12:53.883+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:12:54.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:12:54 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:12:54.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:12:54 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:12:54.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:12:54 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:12:54.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:12:54 smithi115 ceph-mon[78593]: pgmap v3506: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:12:54.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:12:54 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:12:54.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:12:54 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:12:54.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:12:54 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:12:54.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:12:54 smithi047 ceph-mon[77147]: pgmap v3506: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:12:55.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:12:54 smithi115 conmon[90393]: 2022-05-14T01:12:54.862+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:12:55.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:12:55 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:12:55.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:12:55 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 6305 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:12:55.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:12:55 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:12:55.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:12:55 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 6305 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:12:56.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:12:55 smithi115 conmon[90393]: 2022-05-14T01:12:55.888+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:12:56.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:12:56 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:12:56.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:12:56 smithi115 ceph-mon[78593]: pgmap v3507: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:12:56.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:12:56 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:12:56.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:12:56 smithi047 ceph-mon[77147]: pgmap v3507: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:12:57.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:12:56 smithi115 conmon[90393]: 2022-05-14T01:12:56.923+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:12:57.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:12:57 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:12:57.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:12:57 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:12:58.362 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:12:57 smithi115 conmon[90393]: 2022-05-14T01:12:57.898+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:12:58.362 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:12:58 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:12:58.363 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:12:58 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:12:58.363 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:12:58 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:12:58.363 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:12:58 smithi115 ceph-mon[78593]: pgmap v3508: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:12:58.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:12:58 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:12:58.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:12:58 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:12:58.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:12:58 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:12:58.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:12:58 smithi047 ceph-mon[77147]: pgmap v3508: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:12:59.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:12:58 smithi115 conmon[90393]: 2022-05-14T01:12:58.931+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:12:59.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:12:59 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:12:59.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:12:59 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:13:00.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:12:59 smithi115 conmon[90393]: 2022-05-14T01:12:59.914+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:13:00.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:13:00 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:13:00.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:13:00 smithi115 ceph-mon[78593]: pgmap v3509: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:13:00.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:13:00 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 6310 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:13:00.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:13:00 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:13:00.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:13:00 smithi047 ceph-mon[77147]: pgmap v3509: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:13:00.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:13:00 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 6310 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:13:01.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:13:00 smithi115 conmon[90393]: 2022-05-14T01:13:00.956+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:13:01.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:13:01 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:13:01.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:13:01 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:13:02.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:13:01 smithi115 conmon[90393]: 2022-05-14T01:13:01.950+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:13:02.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:13:02 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:13:02.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:13:02 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:13:02.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:13:02 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:13:02.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:13:02 smithi115 ceph-mon[78593]: pgmap v3510: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-05-14T01:13:02.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:13:02 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:13:02.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:13:02 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:13:02.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:13:02 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:13:02.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:13:02 smithi047 ceph-mon[77147]: pgmap v3510: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-05-14T01:13:03.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:13:03 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:13:03.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:13:02 smithi115 conmon[90393]: 2022-05-14T01:13:02.980+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:13:03.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:13:03 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:13:04.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:13:03 smithi115 conmon[90393]: 2022-05-14T01:13:03.957+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:13:04.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:13:04 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:13:04.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:13:04 smithi115 ceph-mon[78593]: pgmap v3511: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:13:04.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:13:04 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:13:04.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:13:04 smithi047 ceph-mon[77147]: pgmap v3511: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:13:05.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:13:05 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:13:05.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:13:05 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 6315 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:13:05.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:13:04 smithi115 conmon[90393]: 2022-05-14T01:13:04.947+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:13:05.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:13:05 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:13:05.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:13:05 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 6315 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:13:06.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:13:05 smithi115 conmon[90393]: 2022-05-14T01:13:05.944+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:13:06.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:13:06 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:13:06.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:13:06 smithi115 ceph-mon[78593]: pgmap v3512: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:13:06.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:13:06 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:13:06.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:13:06 smithi047 ceph-mon[77147]: pgmap v3512: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:13:07.152 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:13:06 smithi115 conmon[90393]: 2022-05-14T01:13:06.900+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:13:07.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:13:07 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:13:07.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:13:07 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:13:08.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:13:07 smithi115 conmon[90393]: 2022-05-14T01:13:07.927+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:13:08.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:13:08 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:13:08.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:13:08 smithi115 ceph-mon[78593]: pgmap v3513: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:13:08.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:13:08 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:13:08.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:13:08 smithi047 ceph-mon[77147]: pgmap v3513: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:13:09.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:13:08 smithi115 conmon[90393]: 2022-05-14T01:13:08.970+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:13:09.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:13:09 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:13:09.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:13:09 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:13:10.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:13:09 smithi115 conmon[90393]: 2022-05-14T01:13:09.925+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:13:10.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:13:10 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:13:10.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:13:10 smithi115 ceph-mon[78593]: pgmap v3514: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:13:10.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:13:10 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:13:10.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:13:10 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:13:10.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:13:10 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 6320 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:13:10.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:13:10 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:13:10.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:13:10 smithi047 ceph-mon[77147]: pgmap v3514: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:13:10.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:13:10 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:13:10.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:13:10 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:13:10.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:13:10 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 6320 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:13:11.175 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:13:10 smithi115 conmon[90393]: 2022-05-14T01:13:10.879+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:13:11.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:13:11 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:13:11.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:13:11 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:13:12.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:13:11 smithi115 conmon[90393]: 2022-05-14T01:13:11.854+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:13:12.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:13:12 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:13:12.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:13:12 smithi047 ceph-mon[77147]: pgmap v3515: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-05-14T01:13:12.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:13:12 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:13:12.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:13:12 smithi115 ceph-mon[78593]: pgmap v3515: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-05-14T01:13:13.182 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:13:12 smithi115 conmon[90393]: 2022-05-14T01:13:12.864+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:13:13.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:13:13 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:13:13.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:13:13 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:13:14.181 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:13:13 smithi115 conmon[90393]: 2022-05-14T01:13:13.881+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:13:14.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:13:14 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:13:14.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:13:14 smithi047 ceph-mon[77147]: pgmap v3516: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:13:14.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:13:14 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:13:14.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:13:14 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:13:14.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:13:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T01:13:14.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:13:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T01:13:14.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:13:14 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:13:14.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:13:14 smithi115 ceph-mon[78593]: pgmap v3516: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:13:14.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:13:14 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:13:14.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:13:14 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:13:14.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:13:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T01:13:14.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:13:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T01:13:15.182 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:13:14 smithi115 conmon[90393]: 2022-05-14T01:13:14.857+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:13:15.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:13:15 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:13:15.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:13:15 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 6325 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:13:15.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:13:15 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:13:15.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:13:15 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 6325 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:13:16.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:13:15 smithi115 conmon[90393]: 2022-05-14T01:13:15.855+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:13:16.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:13:16 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:13:16.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:13:16 smithi047 ceph-mon[77147]: pgmap v3517: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:13:16.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:13:16 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:13:16.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:13:16 smithi115 ceph-mon[78593]: pgmap v3517: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:13:17.189 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:13:16 smithi115 conmon[90393]: 2022-05-14T01:13:16.889+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:13:17.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:13:17 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:13:17.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:13:17 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:13:18.190 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:13:17 smithi115 conmon[90393]: 2022-05-14T01:13:17.915+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:13:18.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:13:18 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:13:18.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:13:18 smithi047 ceph-mon[77147]: pgmap v3518: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:13:18.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:13:18 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:13:18.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:13:18 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:13:18.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:13:18 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:13:18.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:13:18 smithi115 ceph-mon[78593]: pgmap v3518: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:13:18.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:13:18 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:13:18.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:13:18 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:13:19.195 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:13:18 smithi115 conmon[90393]: 2022-05-14T01:13:18.899+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:13:19.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:13:19 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:13:19.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:13:19 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:13:20.196 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:13:19 smithi115 conmon[90393]: 2022-05-14T01:13:19.877+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:13:20.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:13:20 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:13:20.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:13:20 smithi047 ceph-mon[77147]: pgmap v3519: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:13:20.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:13:20 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 6330 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:13:20.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:13:20 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:13:20.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:13:20 smithi115 ceph-mon[78593]: pgmap v3519: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:13:20.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:13:20 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 6330 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:13:21.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:13:20 smithi115 conmon[90393]: 2022-05-14T01:13:20.836+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:13:21.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:13:21 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:13:21.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:13:21 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:13:22.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:13:21 smithi115 conmon[90393]: 2022-05-14T01:13:21.788+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:13:22.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:13:22 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:13:22.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:13:22 smithi047 ceph-mon[77147]: pgmap v3520: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-05-14T01:13:22.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:13:22 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:13:22.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:13:22 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:13:22.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:13:22 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:13:22.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:13:22 smithi115 ceph-mon[78593]: pgmap v3520: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-05-14T01:13:22.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:13:22 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:13:22.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:13:22 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:13:23.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:13:22 smithi115 conmon[90393]: 2022-05-14T01:13:22.806+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:13:23.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:13:23 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:13:23.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:13:23 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:13:24.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:13:23 smithi115 conmon[90393]: 2022-05-14T01:13:23.757+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:13:24.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:13:24 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:13:24.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:13:24 smithi047 ceph-mon[77147]: pgmap v3521: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:13:24.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:13:24 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:13:24.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:13:24 smithi115 ceph-mon[78593]: pgmap v3521: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:13:25.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:13:24 smithi115 conmon[90393]: 2022-05-14T01:13:24.799+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:13:25.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:13:25 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:13:25.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:13:25 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 6335 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:13:25.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:13:25 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:13:25.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:13:25 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 6335 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:13:26.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:13:25 smithi115 conmon[90393]: 2022-05-14T01:13:25.755+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:13:26.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:13:26 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:13:26.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:13:26 smithi047 ceph-mon[77147]: pgmap v3522: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:13:26.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:13:26 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:13:26.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:13:26 smithi115 ceph-mon[78593]: pgmap v3522: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:13:27.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:13:26 smithi115 conmon[90393]: 2022-05-14T01:13:26.779+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:13:27.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:13:27 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:13:27.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:13:27 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:13:28.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:13:27 smithi115 conmon[90393]: 2022-05-14T01:13:27.752+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:13:28.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:13:28 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:13:28.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:13:28 smithi047 ceph-mon[77147]: pgmap v3523: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:13:28.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:13:28 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:13:28.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:13:28 smithi115 ceph-mon[78593]: pgmap v3523: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:13:29.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:13:28 smithi115 conmon[90393]: 2022-05-14T01:13:28.744+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:13:29.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:13:29 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:13:29.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:13:29 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:13:29.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:13:29 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:13:29.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:13:29 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:13:30.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:13:29 smithi115 conmon[90393]: 2022-05-14T01:13:29.780+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:13:30.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:13:30 smithi047 ceph-mon[77147]: pgmap v3524: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:13:30.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:13:30 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:13:30.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:13:30 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:13:30.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:13:30 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 6340 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:13:30.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:13:30 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:13:30.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:13:30 smithi115 ceph-mon[78593]: pgmap v3524: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:13:30.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:13:30 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:13:30.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:13:30 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:13:30.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:13:30 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 6340 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:13:30.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:13:30 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:13:31.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:13:30 smithi115 conmon[90393]: 2022-05-14T01:13:30.791+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:13:31.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:13:31 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:13:31.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:13:31 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:13:32.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:13:31 smithi115 conmon[90393]: 2022-05-14T01:13:31.811+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:13:32.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:13:32 smithi047 ceph-mon[77147]: pgmap v3525: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-05-14T01:13:32.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:13:32 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:13:32.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:13:32 smithi115 ceph-mon[78593]: pgmap v3525: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-05-14T01:13:32.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:13:32 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:13:33.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:13:32 smithi115 conmon[90393]: 2022-05-14T01:13:32.789+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:13:33.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:13:33 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:13:33.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:13:33 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:13:34.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:13:33 smithi115 conmon[90393]: 2022-05-14T01:13:33.786+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:13:34.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:13:34 smithi047 ceph-mon[77147]: pgmap v3526: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:13:34.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:13:34 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:13:34.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:13:34 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:13:34.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:13:34 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:13:34.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:13:34 smithi115 ceph-mon[78593]: pgmap v3526: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:13:34.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:13:34 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:13:34.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:13:34 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:13:34.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:13:34 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:13:35.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:13:34 smithi115 conmon[90393]: 2022-05-14T01:13:34.754+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:13:35.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:13:35 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 6345 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:13:35.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:13:35 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:13:35.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:13:35 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 6345 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:13:35.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:13:35 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:13:36.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:13:35 smithi115 conmon[90393]: 2022-05-14T01:13:35.706+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:13:36.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:13:36 smithi047 ceph-mon[77147]: pgmap v3527: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:13:36.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:13:36 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:13:36.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:13:36 smithi115 ceph-mon[78593]: pgmap v3527: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:13:36.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:13:36 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:13:37.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:13:36 smithi115 conmon[90393]: 2022-05-14T01:13:36.691+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:13:37.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:13:37 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:13:37.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:13:37 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:13:38.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:13:37 smithi115 conmon[90393]: 2022-05-14T01:13:37.668+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:13:38.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:13:38 smithi047 ceph-mon[77147]: pgmap v3528: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:13:38.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:13:38 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:13:38.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:13:38 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:13:38.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:13:38 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:13:38.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:13:38 smithi115 ceph-mon[78593]: pgmap v3528: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:13:38.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:13:38 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:13:38.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:13:38 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:13:38.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:13:38 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:13:39.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:13:38 smithi115 conmon[90393]: 2022-05-14T01:13:38.706+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:13:39.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:13:39 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:13:39.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:13:39 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T01:13:39.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:13:39 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T01:13:39.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:13:39 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T01:13:39.597 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:13:39 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:13:39.598 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:13:39 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T01:13:39.598 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:13:39 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T01:13:39.598 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:13:39 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T01:13:39.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:13:39 smithi115 conmon[90393]: 2022-05-14T01:13:39.660+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:13:40.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:13:40 smithi047 ceph-mon[77147]: pgmap v3529: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:13:40.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:13:40 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 6350 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:13:40.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:13:40 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:13:40.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:13:40 smithi115 ceph-mon[78593]: pgmap v3529: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:13:40.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:13:40 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 6350 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:13:40.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:13:40 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:13:41.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:13:40 smithi115 conmon[90393]: 2022-05-14T01:13:40.653+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:13:41.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:13:41 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:13:41.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:13:41 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:13:42.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:13:41 smithi115 conmon[90393]: 2022-05-14T01:13:41.701+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:13:42.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:13:42 smithi047 ceph-mon[77147]: pgmap v3530: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-05-14T01:13:42.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:13:42 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:13:42.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:13:42 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:13:42.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:13:42 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:13:42.542 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:13:42 smithi047 conmon[78521]: 2022-05-14T01:13:42.191+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:13:42.542 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:13:42 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T01:13:42.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:13:42 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:13:42.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:13:42 smithi047 conmon[78521]: yield 2022-05-14T01:13:42.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:13:42 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:13:42.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:13:42 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:13:42.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:13:42 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:13:42.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:13:42 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T01:13:42.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:13:42 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:13:42.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:13:42 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T01:13:42.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:13:42 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:13:42.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:13:42 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T01:13:42.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:13:42 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:13:42.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:13:42 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T01:13:42.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:13:42 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:13:42.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:13:42 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T01:13:42.547 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:13:42 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:13:42.547 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:13:42 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:13:42.547 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:13:42 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:13:42.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:13:42 smithi115 ceph-mon[78593]: pgmap v3530: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-05-14T01:13:42.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:13:42 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:13:42.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:13:42 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:13:42.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:13:42 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:13:43.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:13:42 smithi115 conmon[90393]: 2022-05-14T01:13:42.733+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:13:43.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:13:43 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:13:43.541 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T01:13:43.541 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:13:43.542 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T01:13:43.542 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:13:43.542 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:13:43.542 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:13:43.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T01:13:43.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:13:43.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T01:13:43.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:13:43.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T01:13:43.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:13:43.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T01:13:43.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:13:43.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T01:13:43.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:13:43.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:13:43.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:13:43.546 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:13:43 smithi047 ceph-mon[77147]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T01:13:43.546 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:13:43 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:13:43.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:13:43 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:13:43.608 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T01:13:43.608 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:13:43.608 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T01:13:43.608 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:13:43.608 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:13:43.609 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:13:43.609 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T01:13:43.609 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:13:43.609 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T01:13:43.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:13:43.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T01:13:43.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:13:43.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T01:13:43.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:13:43.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T01:13:43.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:13:43.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:13:43.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:13:43.612 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:13:43 smithi115 ceph-mon[78593]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T01:13:43.612 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:13:43 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:13:44.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:13:43 smithi115 conmon[90393]: 2022-05-14T01:13:43.783+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:13:44.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:13:44 smithi047 ceph-mon[77147]: pgmap v3531: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:13:44.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:13:44 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:13:44.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:13:44 smithi115 ceph-mon[78593]: pgmap v3531: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:13:44.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:13:44 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:13:45.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:13:44 smithi115 conmon[90393]: 2022-05-14T01:13:44.796+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:13:45.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:13:45 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 6355 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:13:45.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:13:45 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:13:45.542 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:13:45 smithi047 conmon[78521]: 2022-05-14T01:13:45.263+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:13:45.542 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:13:45 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T01:13:45.542 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:13:45 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:13:45.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:13:45 smithi047 conmon[78521]: yield 2022-05-14T01:13:45.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:13:45 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:13:45.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:13:45 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:13:45.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:13:45 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:13:45.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:13:45 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T01:13:45.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:13:45 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:13:45.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:13:45 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T01:13:45.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:13:45 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:13:45.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:13:45 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T01:13:45.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:13:45 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:13:45.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:13:45 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T01:13:45.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:13:45 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:13:45.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:13:45 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T01:13:45.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:13:45 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:13:45.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:13:45 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:13:45.547 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:13:45 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:13:45.547 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:13:45 smithi047 conmon[78521]: 2022-05-14T01:13:45.264+0000 7f5345f84700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T01:13:45.547 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:13:45 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T01:13:45.548 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:13:45 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:13:45.548 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:13:45 smithi047 conmon[78521]: yield 2022-05-14T01:13:45.548 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:13:45 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:13:45.549 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:13:45 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:13:45.549 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:13:45 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:13:45.549 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:13:45 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T01:13:45.549 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:13:45 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:13:45.550 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:13:45 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T01:13:45.550 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:13:45 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:13:45.550 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:13:45 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T01:13:45.550 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:13:45 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:13:45.550 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:13:45 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T01:13:45.551 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:13:45 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:13:45.551 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:13:45 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T01:13:45.551 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:13:45 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:13:45.551 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:13:45 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:13:45.552 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:13:45 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:13:45.552 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:13:45 smithi047 conmon[78521]: 2022-05-14T01:13:45.552 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:13:45 smithi047 conmon[78521]: During handling of the above exception, another exception occurred: 2022-05-14T01:13:45.552 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:13:45 smithi047 conmon[78521]: 2022-05-14T01:13:45.553 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:13:45 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T01:13:45.553 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:13:45 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T01:13:45.554 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:13:45 smithi047 conmon[78521]: return f(*arg) 2022-05-14T01:13:45.554 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:13:45 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T01:13:45.555 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:13:45 smithi047 conmon[78521]: self._write_client_files(client_files, host) 2022-05-14T01:13:45.555 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:13:45 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T01:13:45.555 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:13:45 smithi047 conmon[78521]: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T01:13:45.556 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:13:45 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T01:13:45.556 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:13:45 smithi047 conmon[78521]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T01:13:45.557 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:13:45 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T01:13:45.557 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:13:45 smithi047 conmon[78521]: return self.event_loop.get_result(coro) 2022-05-14T01:13:45.558 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:13:45 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T01:13:45.558 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:13:45 smithi047 conmon[78521]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T01:13:45.558 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:13:45 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T01:13:45.559 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:13:45 smithi047 conmon[78521]: return self.__get_result() 2022-05-14T01:13:45.559 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:13:45 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T01:13:45.560 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:13:45 smithi047 conmon[78521]: raise self._exception 2022-05-14T01:13:45.560 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:13:45 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T01:13:45.560 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:13:45 smithi047 conmon[78521]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T01:13:45.561 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:13:45 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T01:13:45.561 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:13:45 smithi047 conmon[78521]: conn = await self._remote_connection(host, addr) 2022-05-14T01:13:45.562 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:13:45 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T01:13:45.562 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:13:45 smithi047 conmon[78521]: raise 2022-05-14T01:13:45.562 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:13:45 smithi047 conmon[78521]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T01:13:45.562 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:13:45 smithi047 conmon[78521]: self.gen.throw(type, value, traceback) 2022-05-14T01:13:45.563 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:13:45 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T01:13:45.563 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:13:45 smithi047 conmon[78521]: raise OrchestratorError(msg) 2022-05-14T01:13:45.563 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:13:45 smithi047 conmon[78521]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:13:45.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:13:45 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 6355 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:13:45.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:13:45 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:13:46.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:13:45 smithi115 conmon[90393]: 2022-05-14T01:13:45.752+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:13:46.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:13:46 smithi047 ceph-mon[77147]: pgmap v3532: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:13:46.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:13:46 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:13:46.542 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T01:13:46.542 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:13:46.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T01:13:46.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:13:46.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:13:46.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:13:46.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T01:13:46.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:13:46.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T01:13:46.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:13:46.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T01:13:46.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:13:46.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T01:13:46.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:13:46.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T01:13:46.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:13:46.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:13:46.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:13:46.546 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:13:46 smithi047 ceph-mon[77147]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T01:13:46.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T01:13:46.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:13:46.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T01:13:46.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:13:46.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:13:46.548 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:13:46.548 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T01:13:46.548 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:13:46.548 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T01:13:46.549 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:13:46.549 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T01:13:46.549 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:13:46.549 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T01:13:46.549 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:13:46.550 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T01:13:46.550 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:13:46.551 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:13:46.552 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:13:46.552 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T01:13:46.552 INFO:journalctl@ceph.mon.b.smithi047.stdout: During handling of the above exception, another exception occurred: 2022-05-14T01:13:46.552 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T01:13:46.553 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T01:13:46.553 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T01:13:46.553 INFO:journalctl@ceph.mon.b.smithi047.stdout: return f(*arg) 2022-05-14T01:13:46.553 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T01:13:46.555 INFO:journalctl@ceph.mon.b.smithi047.stdout: self._write_client_files(client_files, host) 2022-05-14T01:13:46.555 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T01:13:46.555 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T01:13:46.556 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T01:13:46.556 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T01:13:46.556 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T01:13:46.556 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.event_loop.get_result(coro) 2022-05-14T01:13:46.557 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T01:13:46.557 INFO:journalctl@ceph.mon.b.smithi047.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T01:13:46.557 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T01:13:46.557 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.__get_result() 2022-05-14T01:13:46.558 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T01:13:46.558 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise self._exception 2022-05-14T01:13:46.558 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T01:13:46.558 INFO:journalctl@ceph.mon.b.smithi047.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T01:13:46.559 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T01:13:46.559 INFO:journalctl@ceph.mon.b.smithi047.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T01:13:46.559 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T01:13:46.559 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise 2022-05-14T01:13:46.559 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T01:13:46.560 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.gen.throw(type, value, traceback) 2022-05-14T01:13:46.560 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T01:13:46.560 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OrchestratorError(msg) 2022-05-14T01:13:46.560 INFO:journalctl@ceph.mon.b.smithi047.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:13:46.561 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:13:46 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:13:46.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:13:46 smithi115 ceph-mon[78593]: pgmap v3532: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:13:46.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:13:46 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:13:46.609 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T01:13:46.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:13:46.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T01:13:46.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:13:46.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:13:46.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:13:46.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T01:13:46.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:13:46.613 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T01:13:46.613 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:13:46.614 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T01:13:46.614 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:13:46.614 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T01:13:46.615 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:13:46.615 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T01:13:46.616 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:13:46.616 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:13:46.617 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:13:46.617 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:13:46 smithi115 ceph-mon[78593]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T01:13:46.617 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T01:13:46.618 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:13:46.618 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T01:13:46.619 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:13:46.619 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:13:46.620 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:13:46.620 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T01:13:46.620 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:13:46.621 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T01:13:46.621 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:13:46.622 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T01:13:46.622 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:13:46.622 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T01:13:46.622 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:13:46.623 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T01:13:46.623 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:13:46.623 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:13:46.623 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:13:46.623 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T01:13:46.624 INFO:journalctl@ceph.mon.c.smithi115.stdout: During handling of the above exception, another exception occurred: 2022-05-14T01:13:46.624 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T01:13:46.624 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T01:13:46.624 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T01:13:46.625 INFO:journalctl@ceph.mon.c.smithi115.stdout: return f(*arg) 2022-05-14T01:13:46.625 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T01:13:46.625 INFO:journalctl@ceph.mon.c.smithi115.stdout: self._write_client_files(client_files, host) 2022-05-14T01:13:46.625 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T01:13:46.625 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T01:13:46.626 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T01:13:46.626 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T01:13:46.626 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T01:13:46.626 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.event_loop.get_result(coro) 2022-05-14T01:13:46.627 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T01:13:46.627 INFO:journalctl@ceph.mon.c.smithi115.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T01:13:46.627 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T01:13:46.627 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.__get_result() 2022-05-14T01:13:46.627 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T01:13:46.628 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise self._exception 2022-05-14T01:13:46.628 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T01:13:46.628 INFO:journalctl@ceph.mon.c.smithi115.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T01:13:46.628 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T01:13:46.628 INFO:journalctl@ceph.mon.c.smithi115.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T01:13:46.629 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T01:13:46.629 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise 2022-05-14T01:13:46.629 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T01:13:46.629 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.gen.throw(type, value, traceback) 2022-05-14T01:13:46.630 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T01:13:46.630 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OrchestratorError(msg) 2022-05-14T01:13:46.630 INFO:journalctl@ceph.mon.c.smithi115.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:13:46.630 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:13:46 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:13:47.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:13:46 smithi115 conmon[90393]: 2022-05-14T01:13:46.732+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:13:47.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:13:47 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:13:47.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:13:47 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:13:48.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:13:47 smithi115 conmon[90393]: 2022-05-14T01:13:47.746+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:13:48.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:13:48 smithi047 ceph-mon[77147]: pgmap v3533: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:13:48.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:13:48 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:13:48.612 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:13:48 smithi115 ceph-mon[78593]: pgmap v3533: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:13:48.613 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:13:48 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:13:49.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:13:48 smithi115 conmon[90393]: 2022-05-14T01:13:48.714+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:13:49.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:13:49 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:13:49.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:13:49 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:13:50.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:13:49 smithi115 conmon[90393]: 2022-05-14T01:13:49.750+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:13:50.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:13:50 smithi047 ceph-mon[77147]: pgmap v3534: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:13:50.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:13:50 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:13:50.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:13:50 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:13:50.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:13:50 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 6365 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:13:50.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:13:50 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:13:50.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:13:50 smithi115 ceph-mon[78593]: pgmap v3534: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:13:50.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:13:50 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:13:50.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:13:50 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:13:50.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:13:50 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 6365 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:13:50.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:13:50 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:13:51.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:13:50 smithi115 conmon[90393]: 2022-05-14T01:13:50.758+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:13:51.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:13:51 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:13:51.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:13:51 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:13:52.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:13:51 smithi115 conmon[90393]: 2022-05-14T01:13:51.715+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:13:52.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:13:52 smithi115 ceph-mon[78593]: pgmap v3535: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-05-14T01:13:52.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:13:52 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:13:52.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:13:52 smithi047 ceph-mon[77147]: pgmap v3535: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-05-14T01:13:52.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:13:52 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:13:53.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:13:52 smithi115 conmon[90393]: 2022-05-14T01:13:52.672+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:13:53.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:13:53 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:13:53.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:13:53 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:13:54.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:13:53 smithi115 conmon[90393]: 2022-05-14T01:13:53.680+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:13:54.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:13:54 smithi115 ceph-mon[78593]: pgmap v3536: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:13:54.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:13:54 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:13:54.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:13:54 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:13:54.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:13:54 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:13:54.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:13:54 smithi047 ceph-mon[77147]: pgmap v3536: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:13:54.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:13:54 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:13:54.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:13:54 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:13:54.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:13:54 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:13:55.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:13:54 smithi115 conmon[90393]: 2022-05-14T01:13:54.674+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:13:55.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:13:55 smithi115 conmon[90393]: 2022-05-14T01:13:55.651+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:13:55.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:13:55 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:13:55.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:13:55 smithi115 ceph-mon[78593]: pgmap v3537: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:13:56.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:13:55 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:13:56.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:13:55 smithi047 ceph-mon[77147]: pgmap v3537: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:13:56.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:13:56 smithi115 conmon[90393]: 2022-05-14T01:13:56.645+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:13:56.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:13:56 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 6370 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:13:56.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:13:56 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:13:57.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:13:56 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 6370 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:13:57.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:13:56 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:13:57.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:13:57 smithi115 conmon[90393]: 2022-05-14T01:13:57.614+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:13:57.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:13:57 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:13:57.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:13:57 smithi115 ceph-mon[78593]: pgmap v3538: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:13:58.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:13:57 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:13:58.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:13:57 smithi047 ceph-mon[77147]: pgmap v3538: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:13:58.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:13:58 smithi115 conmon[90393]: 2022-05-14T01:13:58.570+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:13:58.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:13:58 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:13:58.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:13:58 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:13:58.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:13:58 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:13:59.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:13:58 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:13:59.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:13:58 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:13:59.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:13:58 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:13:59.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:13:59 smithi115 conmon[90393]: 2022-05-14T01:13:59.525+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:13:59.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:13:59 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:13:59.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:13:59 smithi115 ceph-mon[78593]: pgmap v3539: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:14:00.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:13:59 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:14:00.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:13:59 smithi047 ceph-mon[77147]: pgmap v3539: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:14:00.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:14:00 smithi115 conmon[90393]: 2022-05-14T01:14:00.542+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:14:00.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:14:00 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:14:01.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:14:00 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:14:01.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:14:01 smithi115 conmon[90393]: 2022-05-14T01:14:01.520+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:14:01.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:14:01 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:14:01.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:14:01 smithi115 ceph-mon[78593]: pgmap v3540: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:14:02.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:14:01 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:14:02.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:14:01 smithi047 ceph-mon[77147]: pgmap v3540: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:14:02.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:14:02 smithi115 conmon[90393]: 2022-05-14T01:14:02.515+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:14:02.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:14:02 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:14:02.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:14:02 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:14:02.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:14:02 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:14:03.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:14:02 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:14:03.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:14:02 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:14:03.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:14:02 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:14:03.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:14:03 smithi115 conmon[90393]: 2022-05-14T01:14:03.543+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:14:03.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:14:03 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:14:03.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:14:03 smithi115 ceph-mon[78593]: pgmap v3541: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:14:04.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:14:03 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:14:04.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:14:03 smithi047 ceph-mon[77147]: pgmap v3541: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:14:04.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:14:04 smithi115 conmon[90393]: 2022-05-14T01:14:04.554+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:14:04.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:14:04 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:14:04.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:14:04 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 6375 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:14:05.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:14:04 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:14:05.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:14:04 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 6375 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:14:05.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:14:05 smithi115 conmon[90393]: 2022-05-14T01:14:05.599+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:14:05.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:14:05 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:14:05.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:14:05 smithi115 ceph-mon[78593]: pgmap v3542: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:14:06.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:14:05 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:14:06.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:14:05 smithi047 ceph-mon[77147]: pgmap v3542: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:14:07.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:14:06 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:14:07.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:14:06 smithi115 conmon[90393]: 2022-05-14T01:14:06.613+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:14:07.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:14:06 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:14:08.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:14:07 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:14:08.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:14:07 smithi047 ceph-mon[77147]: pgmap v3543: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:14:08.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:14:07 smithi115 conmon[90393]: 2022-05-14T01:14:07.663+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:14:08.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:14:07 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:14:08.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:14:07 smithi115 ceph-mon[78593]: pgmap v3543: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:14:09.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:14:08 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:14:09.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:14:08 smithi115 conmon[90393]: 2022-05-14T01:14:08.686+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:14:09.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:14:08 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:14:10.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:14:09 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:14:10.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:14:09 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:14:10.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:14:09 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:14:10.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:14:09 smithi047 ceph-mon[77147]: pgmap v3544: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:14:10.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:14:09 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 6380 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:14:10.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:14:09 smithi115 conmon[90393]: 2022-05-14T01:14:09.718+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:14:10.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:14:09 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:14:10.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:14:09 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:14:10.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:14:09 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:14:10.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:14:09 smithi115 ceph-mon[78593]: pgmap v3544: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:14:10.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:14:09 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 6380 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:14:11.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:14:10 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:14:11.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:14:10 smithi115 conmon[90393]: 2022-05-14T01:14:10.748+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:14:11.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:14:10 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:14:12.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:14:11 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:14:12.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:14:11 smithi047 ceph-mon[77147]: pgmap v3545: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:14:12.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:14:11 smithi115 conmon[90393]: 2022-05-14T01:14:11.722+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:14:12.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:14:11 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:14:12.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:14:11 smithi115 ceph-mon[78593]: pgmap v3545: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:14:13.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:14:12 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:14:13.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:14:12 smithi115 conmon[90393]: 2022-05-14T01:14:12.714+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:14:13.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:14:12 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:14:14.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:14:13 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:14:14.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:14:13 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:14:14.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:14:13 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:14:14.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:14:13 smithi047 ceph-mon[77147]: pgmap v3546: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:14:14.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:14:13 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T01:14:14.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:14:13 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T01:14:14.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:14:13 smithi115 conmon[90393]: 2022-05-14T01:14:13.685+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:14:14.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:14:13 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:14:14.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:14:13 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:14:14.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:14:13 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:14:14.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:14:13 smithi115 ceph-mon[78593]: pgmap v3546: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:14:14.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:14:13 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T01:14:14.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:14:13 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T01:14:15.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:14:14 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:14:15.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:14:14 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 6385 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:14:15.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:14:14 smithi115 conmon[90393]: 2022-05-14T01:14:14.690+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:14:15.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:14:14 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:14:15.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:14:14 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 6385 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:14:16.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:14:15 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:14:16.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:14:15 smithi047 ceph-mon[77147]: pgmap v3547: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:14:16.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:14:15 smithi115 conmon[90393]: 2022-05-14T01:14:15.653+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:14:16.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:14:15 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:14:16.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:14:15 smithi115 ceph-mon[78593]: pgmap v3547: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:14:17.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:14:16 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:14:17.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:14:16 smithi115 conmon[90393]: 2022-05-14T01:14:16.643+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:14:17.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:14:16 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:14:18.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:14:17 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:14:18.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:14:17 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:14:18.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:14:17 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:14:18.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:14:17 smithi047 ceph-mon[77147]: pgmap v3548: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:14:18.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:14:17 smithi115 conmon[90393]: 2022-05-14T01:14:17.691+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:14:18.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:14:17 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:14:18.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:14:17 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:14:18.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:14:17 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:14:18.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:14:17 smithi115 ceph-mon[78593]: pgmap v3548: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:14:19.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:14:18 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:14:19.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:14:18 smithi115 conmon[90393]: 2022-05-14T01:14:18.718+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:14:19.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:14:18 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:14:20.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:14:19 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:14:20.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:14:19 smithi047 ceph-mon[77147]: pgmap v3549: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:14:20.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:14:19 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 6390 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:14:20.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:14:19 smithi115 conmon[90393]: 2022-05-14T01:14:19.758+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:14:20.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:14:19 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:14:20.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:14:19 smithi115 ceph-mon[78593]: pgmap v3549: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:14:20.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:14:19 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 6390 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:14:21.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:14:20 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:14:21.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:14:20 smithi115 conmon[90393]: 2022-05-14T01:14:20.714+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:14:21.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:14:20 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:14:22.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:14:21 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:14:22.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:14:21 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:14:22.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:14:21 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:14:22.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:14:21 smithi047 ceph-mon[77147]: pgmap v3550: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:14:22.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:14:21 smithi115 conmon[90393]: 2022-05-14T01:14:21.672+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:14:22.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:14:21 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:14:22.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:14:21 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:14:22.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:14:21 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:14:22.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:14:21 smithi115 ceph-mon[78593]: pgmap v3550: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:14:23.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:14:22 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:14:23.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:14:22 smithi115 conmon[90393]: 2022-05-14T01:14:22.715+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:14:23.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:14:22 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:14:24.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:14:23 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:14:24.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:14:23 smithi047 ceph-mon[77147]: pgmap v3551: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:14:24.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:14:23 smithi115 conmon[90393]: 2022-05-14T01:14:23.671+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:14:24.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:14:23 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:14:24.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:14:23 smithi115 ceph-mon[78593]: pgmap v3551: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:14:25.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:14:24 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:14:25.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:14:24 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 6395 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:14:25.110 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:14:24 smithi115 conmon[90393]: 2022-05-14T01:14:24.677+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:14:25.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:14:24 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:14:25.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:14:24 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 6395 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:14:26.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:14:25 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:14:26.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:14:25 smithi047 ceph-mon[77147]: pgmap v3552: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:14:26.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:14:25 smithi115 conmon[90393]: 2022-05-14T01:14:25.656+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:14:26.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:14:25 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:14:26.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:14:25 smithi115 ceph-mon[78593]: pgmap v3552: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:14:27.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:14:26 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:14:27.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:14:26 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:14:27.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:14:26 smithi115 conmon[90393]: 2022-05-14T01:14:26.637+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:14:27.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:14:27 smithi115 conmon[90393]: 2022-05-14T01:14:27.604+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:14:27.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:14:27 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:14:27.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:14:27 smithi115 ceph-mon[78593]: pgmap v3553: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:14:28.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:14:27 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:14:28.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:14:27 smithi047 ceph-mon[77147]: pgmap v3553: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:14:28.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:14:28 smithi115 conmon[90393]: 2022-05-14T01:14:28.580+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:14:28.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:14:28 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:14:29.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:14:28 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:14:29.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:14:29 smithi115 conmon[90393]: 2022-05-14T01:14:29.552+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:14:29.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:14:29 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:14:29.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:14:29 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:14:29.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:14:29 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:14:29.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:14:29 smithi115 ceph-mon[78593]: pgmap v3554: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:14:29.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:14:29 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 6400 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:14:30.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:14:29 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:14:30.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:14:29 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:14:30.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:14:29 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:14:30.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:14:29 smithi047 ceph-mon[77147]: pgmap v3554: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:14:30.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:14:29 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 6400 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:14:30.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:14:30 smithi115 conmon[90393]: 2022-05-14T01:14:30.558+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:14:30.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:14:30 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:14:31.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:14:30 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:14:31.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:14:31 smithi115 conmon[90393]: 2022-05-14T01:14:31.596+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:14:31.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:14:31 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:14:31.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:14:31 smithi115 ceph-mon[78593]: pgmap v3555: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:14:32.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:14:31 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:14:32.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:14:31 smithi047 ceph-mon[77147]: pgmap v3555: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:14:32.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:14:32 smithi115 conmon[90393]: 2022-05-14T01:14:32.579+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:14:32.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:14:32 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:14:33.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:14:32 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:14:34.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:14:33 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:14:34.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:14:33 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:14:34.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:14:33 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:14:34.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:14:33 smithi047 ceph-mon[77147]: pgmap v3556: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:14:34.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:14:33 smithi115 conmon[90393]: 2022-05-14T01:14:33.614+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:14:34.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:14:33 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:14:34.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:14:33 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:14:34.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:14:33 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:14:34.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:14:33 smithi115 ceph-mon[78593]: pgmap v3556: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:14:35.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:14:34 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:14:35.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:14:34 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 6405 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:14:35.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:14:34 smithi115 conmon[90393]: 2022-05-14T01:14:34.641+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:14:35.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:14:34 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:14:35.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:14:34 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 6405 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:14:36.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:14:35 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:14:36.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:14:35 smithi047 ceph-mon[77147]: pgmap v3557: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:14:36.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:14:35 smithi115 conmon[90393]: 2022-05-14T01:14:35.639+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:14:36.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:14:35 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:14:36.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:14:35 smithi115 ceph-mon[78593]: pgmap v3557: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:14:37.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:14:36 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:14:37.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:14:36 smithi115 conmon[90393]: 2022-05-14T01:14:36.629+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:14:37.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:14:36 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:14:37.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:14:37 smithi115 conmon[90393]: 2022-05-14T01:14:37.584+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:14:37.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:14:37 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:14:37.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:14:37 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:14:37.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:14:37 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:14:37.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:14:37 smithi115 ceph-mon[78593]: pgmap v3558: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:14:38.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:14:37 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:14:38.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:14:37 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:14:38.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:14:37 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:14:38.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:14:37 smithi047 ceph-mon[77147]: pgmap v3558: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:14:38.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:14:38 smithi115 conmon[90393]: 2022-05-14T01:14:38.564+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:14:38.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:14:38 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:14:39.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:14:38 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:14:39.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:14:39 smithi115 conmon[90393]: 2022-05-14T01:14:39.552+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:14:39.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:14:39 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:14:39.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:14:39 smithi115 ceph-mon[78593]: pgmap v3559: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:14:39.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:14:39 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 6410 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:14:40.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:14:39 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:14:40.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:14:39 smithi047 ceph-mon[77147]: pgmap v3559: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:14:40.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:14:39 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 6410 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:14:40.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:14:40 smithi115 conmon[90393]: 2022-05-14T01:14:40.541+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:14:40.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:14:40 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:14:41.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:14:40 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:14:41.793 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:14:41 smithi115 conmon[90393]: 2022-05-14T01:14:41.495+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:14:42.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:14:41 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:14:42.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:14:41 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:14:42.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:14:41 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:14:42.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:14:41 smithi047 ceph-mon[77147]: pgmap v3560: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:14:42.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:14:41 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:14:42.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:14:41 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:14:42.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:14:41 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:14:42.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:14:41 smithi115 ceph-mon[78593]: pgmap v3560: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:14:42.792 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:14:42 smithi115 conmon[90393]: 2022-05-14T01:14:42.534+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:14:43.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:14:42 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:14:43.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:14:42 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:14:43.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:14:43 smithi115 conmon[90393]: 2022-05-14T01:14:43.581+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:14:43.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:14:43 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:14:43.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:14:43 smithi115 ceph-mon[78593]: pgmap v3561: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:14:44.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:14:43 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:14:44.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:14:43 smithi047 ceph-mon[77147]: pgmap v3561: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:14:44.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:14:44 smithi115 conmon[90393]: 2022-05-14T01:14:44.592+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:14:44.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:14:44 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:14:44.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:14:44 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 6415 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:14:45.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:14:44 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:14:45.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:14:44 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 6415 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:14:45.872 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:14:45 smithi115 conmon[90393]: 2022-05-14T01:14:45.620+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:14:45.872 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:14:45 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:14:45.873 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:14:45 smithi115 ceph-mon[78593]: pgmap v3562: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:14:45.873 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:14:45 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T01:14:45.873 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:14:45 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T01:14:45.873 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:14:45 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T01:14:46.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:14:45 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:14:46.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:14:45 smithi047 ceph-mon[77147]: pgmap v3562: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:14:46.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:14:45 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T01:14:46.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:14:45 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T01:14:46.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:14:45 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T01:14:46.042 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:14:45 smithi047 conmon[78521]: 2022-05-14T01:14:45.743+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:14:46.042 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:14:45 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T01:14:46.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:14:45 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:14:46.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:14:45 smithi047 conmon[78521]: yield 2022-05-14T01:14:46.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:14:45 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:14:46.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:14:45 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:14:46.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:14:45 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:14:46.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:14:45 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T01:14:46.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:14:45 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:14:46.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:14:45 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T01:14:46.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:14:45 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:14:46.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:14:45 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T01:14:46.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:14:45 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:14:46.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:14:45 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T01:14:46.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:14:45 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:14:46.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:14:45 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T01:14:46.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:14:45 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:14:46.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:14:45 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:14:46.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:14:45 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:14:46.805 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:14:46 smithi115 conmon[90393]: 2022-05-14T01:14:46.613+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:14:47.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:14:46 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:14:47.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:14:46 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:14:47.109 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T01:14:47.109 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:14:47.109 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T01:14:47.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:14:47.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:14:47.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:14:47.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T01:14:47.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:14:47.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T01:14:47.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:14:47.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T01:14:47.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:14:47.113 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T01:14:47.113 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:14:47.114 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T01:14:47.115 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:14:47.115 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:14:47.115 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:14:47.116 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:14:46 smithi115 ceph-mon[78593]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T01:14:47.182 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:14:46 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:14:47.182 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:14:46 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:14:47.183 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T01:14:47.183 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:14:47.183 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T01:14:47.183 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:14:47.183 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:14:47.184 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:14:47.184 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T01:14:47.184 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:14:47.184 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T01:14:47.185 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:14:47.185 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T01:14:47.185 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:14:47.185 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T01:14:47.186 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:14:47.186 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T01:14:47.186 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:14:47.186 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:14:47.186 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:14:47.187 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:14:46 smithi047 ceph-mon[77147]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T01:14:47.809 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:14:47 smithi115 conmon[90393]: 2022-05-14T01:14:47.630+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:14:48.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:14:47 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:14:48.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:14:47 smithi115 ceph-mon[78593]: pgmap v3563: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:14:48.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:14:47 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:14:48.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:14:47 smithi047 ceph-mon[77147]: pgmap v3563: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:14:49.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:14:48 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:14:49.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:14:48 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' 2022-05-14T01:14:49.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:14:48 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' 2022-05-14T01:14:49.042 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:14:48 smithi047 conmon[78521]: 2022-05-14T01:14:48.815+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:14:49.042 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:14:48 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T01:14:49.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:14:48 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:14:49.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:14:48 smithi047 conmon[78521]: yield 2022-05-14T01:14:49.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:14:48 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:14:49.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:14:48 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:14:49.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:14:48 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:14:49.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:14:48 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T01:14:49.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:14:48 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:14:49.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:14:48 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T01:14:49.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:14:48 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:14:49.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:14:48 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T01:14:49.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:14:48 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:14:49.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:14:48 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T01:14:49.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:14:48 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:14:49.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:14:48 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T01:14:49.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:14:48 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:14:49.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:14:48 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:14:49.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:14:48 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:14:49.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:14:48 smithi047 conmon[78521]: 2022-05-14T01:14:48.816+0000 7f533f777700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T01:14:49.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:14:48 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T01:14:49.048 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:14:48 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:14:49.048 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:14:48 smithi047 conmon[78521]: yield 2022-05-14T01:14:49.048 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:14:48 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:14:49.048 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:14:48 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:14:49.048 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:14:48 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:14:49.049 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:14:48 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T01:14:49.049 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:14:48 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:14:49.049 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:14:48 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T01:14:49.049 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:14:48 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:14:49.050 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:14:48 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T01:14:49.050 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:14:48 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:14:49.050 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:14:48 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T01:14:49.050 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:14:48 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:14:49.051 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:14:48 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T01:14:49.051 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:14:48 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:14:49.051 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:14:48 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:14:49.051 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:14:48 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:14:49.052 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:14:48 smithi047 conmon[78521]: 2022-05-14T01:14:49.052 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:14:48 smithi047 conmon[78521]: During handling of the above exception, another exception occurred: 2022-05-14T01:14:49.052 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:14:48 smithi047 conmon[78521]: 2022-05-14T01:14:49.052 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:14:48 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T01:14:49.052 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:14:48 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T01:14:49.053 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:14:48 smithi047 conmon[78521]: return f(*arg) 2022-05-14T01:14:49.053 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:14:48 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T01:14:49.053 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:14:48 smithi047 conmon[78521]: self._write_client_files(client_files, host) 2022-05-14T01:14:49.053 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:14:48 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T01:14:49.054 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:14:48 smithi047 conmon[78521]: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T01:14:49.054 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:14:48 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T01:14:49.054 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:14:48 smithi047 conmon[78521]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T01:14:49.054 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:14:48 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T01:14:49.055 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:14:48 smithi047 conmon[78521]: return self.event_loop.get_result(coro) 2022-05-14T01:14:49.055 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:14:48 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T01:14:49.055 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:14:48 smithi047 conmon[78521]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T01:14:49.055 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:14:48 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T01:14:49.055 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:14:48 smithi047 conmon[78521]: return self.__get_result() 2022-05-14T01:14:49.056 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:14:48 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T01:14:49.056 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:14:48 smithi047 conmon[78521]: raise self._exception 2022-05-14T01:14:49.056 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:14:48 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T01:14:49.056 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:14:48 smithi047 conmon[78521]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T01:14:49.056 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:14:48 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T01:14:49.057 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:14:48 smithi047 conmon[78521]: conn = await self._remote_connection(host, addr) 2022-05-14T01:14:49.057 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:14:48 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T01:14:49.057 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:14:48 smithi047 conmon[78521]: raise 2022-05-14T01:14:49.057 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:14:48 smithi047 conmon[78521]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T01:14:49.058 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:14:48 smithi047 conmon[78521]: self.gen.throw(type, value, traceback) 2022-05-14T01:14:49.058 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:14:48 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T01:14:49.058 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:14:48 smithi047 conmon[78521]: raise OrchestratorError(msg) 2022-05-14T01:14:49.058 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:14:48 smithi047 conmon[78521]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:14:49.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:14:48 smithi115 conmon[90393]: 2022-05-14T01:14:48.650+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:14:49.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:14:48 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:14:49.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:14:48 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' 2022-05-14T01:14:49.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:14:48 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' 2022-05-14T01:14:50.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:14:49 smithi115 conmon[90393]: 2022-05-14T01:14:49.698+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:14:50.181 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:14:49 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:14:50.182 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:14:49 smithi115 ceph-mon[78593]: Detected new or changed devices on smithi115 2022-05-14T01:14:50.182 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:14:49 smithi115 ceph-mon[78593]: Detected new or changed devices on smithi047 2022-05-14T01:14:50.183 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:14:49 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:14:50.183 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:14:49 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:14:50.183 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:14:49 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:14:50.183 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T01:14:50.183 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:14:50.184 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T01:14:50.184 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:14:50.184 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:14:50.184 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:14:50.185 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T01:14:50.185 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:14:50.185 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T01:14:50.186 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:14:50.186 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T01:14:50.186 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:14:50.187 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T01:14:50.187 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:14:50.187 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T01:14:50.187 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:14:50.187 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:14:50.188 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:14:50.188 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:14:49 smithi115 ceph-mon[78593]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T01:14:50.188 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T01:14:50.188 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:14:50.188 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T01:14:50.189 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:14:50.189 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:14:50.189 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:14:50.189 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T01:14:50.190 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:14:50.190 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T01:14:50.190 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:14:50.190 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T01:14:50.190 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:14:50.191 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T01:14:50.191 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:14:50.191 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T01:14:50.191 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:14:50.191 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:14:50.192 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:14:50.192 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T01:14:50.192 INFO:journalctl@ceph.mon.c.smithi115.stdout: During handling of the above exception, another exception occurred: 2022-05-14T01:14:50.192 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T01:14:50.192 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T01:14:50.193 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T01:14:50.193 INFO:journalctl@ceph.mon.c.smithi115.stdout: return f(*arg) 2022-05-14T01:14:50.193 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T01:14:50.193 INFO:journalctl@ceph.mon.c.smithi115.stdout: self._write_client_files(client_files, host) 2022-05-14T01:14:50.194 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T01:14:50.194 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T01:14:50.194 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T01:14:50.194 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T01:14:50.194 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T01:14:50.195 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.event_loop.get_result(coro) 2022-05-14T01:14:50.195 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T01:14:50.195 INFO:journalctl@ceph.mon.c.smithi115.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T01:14:50.195 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T01:14:50.195 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.__get_result() 2022-05-14T01:14:50.196 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T01:14:50.196 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise self._exception 2022-05-14T01:14:50.196 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T01:14:50.196 INFO:journalctl@ceph.mon.c.smithi115.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T01:14:50.197 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T01:14:50.197 INFO:journalctl@ceph.mon.c.smithi115.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T01:14:50.197 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T01:14:50.197 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise 2022-05-14T01:14:50.197 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T01:14:50.198 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.gen.throw(type, value, traceback) 2022-05-14T01:14:50.198 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T01:14:50.198 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OrchestratorError(msg) 2022-05-14T01:14:50.199 INFO:journalctl@ceph.mon.c.smithi115.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:14:50.199 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:14:49 smithi115 ceph-mon[78593]: pgmap v3564: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:14:50.199 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:14:49 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 6420 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:14:50.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:14:49 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:14:50.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:14:49 smithi047 ceph-mon[77147]: Detected new or changed devices on smithi115 2022-05-14T01:14:50.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:14:49 smithi047 ceph-mon[77147]: Detected new or changed devices on smithi047 2022-05-14T01:14:50.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:14:49 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:14:50.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:14:49 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:14:50.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:14:49 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:14:50.292 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T01:14:50.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:14:50.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T01:14:50.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:14:50.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:14:50.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:14:50.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T01:14:50.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:14:50.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T01:14:50.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:14:50.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T01:14:50.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:14:50.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T01:14:50.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:14:50.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T01:14:50.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:14:50.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:14:50.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:14:50.297 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:14:49 smithi047 ceph-mon[77147]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T01:14:50.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T01:14:50.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:14:50.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T01:14:50.298 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:14:50.298 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:14:50.298 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:14:50.298 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T01:14:50.298 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:14:50.299 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T01:14:50.299 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:14:50.299 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T01:14:50.299 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:14:50.300 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T01:14:50.300 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:14:50.300 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T01:14:50.300 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:14:50.301 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:14:50.301 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:14:50.301 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T01:14:50.301 INFO:journalctl@ceph.mon.b.smithi047.stdout: During handling of the above exception, another exception occurred: 2022-05-14T01:14:50.302 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T01:14:50.302 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T01:14:50.302 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T01:14:50.302 INFO:journalctl@ceph.mon.b.smithi047.stdout: return f(*arg) 2022-05-14T01:14:50.302 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T01:14:50.303 INFO:journalctl@ceph.mon.b.smithi047.stdout: self._write_client_files(client_files, host) 2022-05-14T01:14:50.303 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T01:14:50.303 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T01:14:50.303 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T01:14:50.303 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T01:14:50.304 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T01:14:50.304 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.event_loop.get_result(coro) 2022-05-14T01:14:50.304 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T01:14:50.304 INFO:journalctl@ceph.mon.b.smithi047.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T01:14:50.305 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T01:14:50.305 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.__get_result() 2022-05-14T01:14:50.305 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T01:14:50.305 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise self._exception 2022-05-14T01:14:50.305 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T01:14:50.306 INFO:journalctl@ceph.mon.b.smithi047.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T01:14:50.306 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T01:14:50.306 INFO:journalctl@ceph.mon.b.smithi047.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T01:14:50.306 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T01:14:50.307 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise 2022-05-14T01:14:50.307 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T01:14:50.307 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.gen.throw(type, value, traceback) 2022-05-14T01:14:50.307 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T01:14:50.308 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OrchestratorError(msg) 2022-05-14T01:14:50.308 INFO:journalctl@ceph.mon.b.smithi047.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:14:50.308 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:14:49 smithi047 ceph-mon[77147]: pgmap v3564: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:14:50.308 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:14:49 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 6420 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:14:51.112 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:14:50 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:14:51.113 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:14:50 smithi115 conmon[90393]: 2022-05-14T01:14:50.714+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:14:51.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:14:50 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:14:52.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:14:51 smithi115 conmon[90393]: 2022-05-14T01:14:51.742+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:14:52.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:14:51 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:14:52.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:14:51 smithi115 ceph-mon[78593]: pgmap v3565: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:14:52.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:14:51 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:14:52.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:14:51 smithi047 ceph-mon[77147]: pgmap v3565: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:14:53.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:14:52 smithi115 conmon[90393]: 2022-05-14T01:14:52.702+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:14:53.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:14:52 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:14:53.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:14:52 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:14:54.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:14:53 smithi115 conmon[90393]: 2022-05-14T01:14:53.698+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:14:54.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:14:53 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:14:54.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:14:53 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:14:54.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:14:53 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:14:54.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:14:53 smithi115 ceph-mon[78593]: pgmap v3566: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:14:54.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:14:53 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:14:54.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:14:53 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:14:54.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:14:53 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:14:54.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:14:53 smithi047 ceph-mon[77147]: pgmap v3566: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:14:55.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:14:54 smithi115 conmon[90393]: 2022-05-14T01:14:54.739+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:14:55.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:14:54 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:14:55.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:14:54 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 6425 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:14:55.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:14:54 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:14:55.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:14:54 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 6425 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:14:56.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:14:55 smithi115 conmon[90393]: 2022-05-14T01:14:55.754+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:14:56.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:14:55 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:14:56.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:14:55 smithi115 ceph-mon[78593]: pgmap v3567: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:14:56.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:14:55 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:14:56.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:14:55 smithi047 ceph-mon[77147]: pgmap v3567: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:14:57.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:14:56 smithi115 conmon[90393]: 2022-05-14T01:14:56.770+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:14:57.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:14:56 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:14:57.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:14:56 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:14:58.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:14:57 smithi115 conmon[90393]: 2022-05-14T01:14:57.772+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:14:58.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:14:57 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:14:58.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:14:57 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:14:58.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:14:57 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:14:58.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:14:57 smithi115 ceph-mon[78593]: pgmap v3568: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:14:58.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:14:57 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:14:58.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:14:57 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:14:58.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:14:57 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:14:58.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:14:57 smithi047 ceph-mon[77147]: pgmap v3568: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:14:59.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:14:58 smithi115 conmon[90393]: 2022-05-14T01:14:58.727+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:14:59.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:14:58 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:14:59.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:14:58 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:15:00.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:14:59 smithi115 conmon[90393]: 2022-05-14T01:14:59.683+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:15:00.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:14:59 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:15:00.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:14:59 smithi115 ceph-mon[78593]: pgmap v3569: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:15:00.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:14:59 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 6430 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:15:00.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:14:59 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:15:00.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:14:59 smithi047 ceph-mon[77147]: pgmap v3569: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:15:00.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:14:59 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 6430 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:15:01.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:15:00 smithi115 conmon[90393]: 2022-05-14T01:15:00.696+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:15:01.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:15:00 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:15:01.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:15:00 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:15:02.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:15:01 smithi115 conmon[90393]: 2022-05-14T01:15:01.651+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:15:02.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:15:01 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:15:02.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:15:01 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:15:02.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:15:01 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:15:02.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:15:01 smithi115 ceph-mon[78593]: pgmap v3570: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:15:02.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:15:01 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:15:02.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:15:01 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:15:02.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:15:01 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:15:02.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:15:01 smithi047 ceph-mon[77147]: pgmap v3570: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:15:02.880 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:15:02 smithi115 conmon[90393]: 2022-05-14T01:15:02.621+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:15:03.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:15:02 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:15:03.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:15:02 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:15:03.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:15:03 smithi115 conmon[90393]: 2022-05-14T01:15:03.604+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:15:04.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:15:03 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:15:04.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:15:03 smithi047 ceph-mon[77147]: pgmap v3571: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:15:04.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:15:03 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:15:04.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:15:03 smithi115 ceph-mon[78593]: pgmap v3571: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:15:04.886 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:15:04 smithi115 conmon[90393]: 2022-05-14T01:15:04.616+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:15:05.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:15:04 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:15:05.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:15:04 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 6435 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:15:05.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:15:04 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:15:05.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:15:04 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 6435 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:15:06.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:15:05 smithi115 conmon[90393]: 2022-05-14T01:15:05.650+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:15:06.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:15:05 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:15:06.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:15:05 smithi115 ceph-mon[78593]: pgmap v3572: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:15:06.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:15:05 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:15:06.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:15:05 smithi047 ceph-mon[77147]: pgmap v3572: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:15:06.893 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:15:06 smithi115 conmon[90393]: 2022-05-14T01:15:06.627+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:15:07.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:15:06 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:15:07.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:15:06 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:15:07.898 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:15:07 smithi115 conmon[90393]: 2022-05-14T01:15:07.630+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:15:08.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:15:07 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:15:08.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:15:07 smithi047 ceph-mon[77147]: pgmap v3573: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:15:08.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:15:07 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:15:08.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:15:07 smithi115 ceph-mon[78593]: pgmap v3573: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:15:09.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:15:08 smithi115 conmon[90393]: 2022-05-14T01:15:08.665+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:15:09.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:15:08 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:15:09.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:15:08 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:15:10.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:15:09 smithi115 conmon[90393]: 2022-05-14T01:15:09.680+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:15:10.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:15:09 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:15:10.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:15:09 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:15:10.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:15:09 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:15:10.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:15:09 smithi115 ceph-mon[78593]: pgmap v3574: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:15:10.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:15:09 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 6440 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:15:10.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:15:09 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:15:10.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:15:09 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:15:10.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:15:09 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:15:10.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:15:09 smithi047 ceph-mon[77147]: pgmap v3574: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:15:10.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:15:09 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 6440 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:15:10.911 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:15:10 smithi115 conmon[90393]: 2022-05-14T01:15:10.630+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:15:11.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:15:10 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:15:11.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:15:10 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:15:11.918 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:15:11 smithi115 conmon[90393]: 2022-05-14T01:15:11.610+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:15:12.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:15:11 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:15:12.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:15:11 smithi047 ceph-mon[77147]: pgmap v3575: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:15:12.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:15:11 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:15:12.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:15:11 smithi115 ceph-mon[78593]: pgmap v3575: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:15:12.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:15:12 smithi115 conmon[90393]: 2022-05-14T01:15:12.579+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:15:13.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:15:12 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:15:13.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:15:12 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:15:13.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:15:13 smithi115 conmon[90393]: 2022-05-14T01:15:13.589+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:15:14.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:15:13 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:15:14.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:15:13 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:15:14.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:15:13 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:15:14.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:15:13 smithi047 ceph-mon[77147]: pgmap v3576: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:15:14.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:15:13 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T01:15:14.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:15:13 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T01:15:14.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:15:13 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:15:14.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:15:13 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:15:14.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:15:13 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:15:14.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:15:13 smithi115 ceph-mon[78593]: pgmap v3576: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:15:14.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:15:13 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T01:15:14.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:15:13 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T01:15:14.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:15:14 smithi115 conmon[90393]: 2022-05-14T01:15:14.590+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:15:15.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:15:14 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:15:15.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:15:14 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 6445 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:15:15.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:15:14 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:15:15.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:15:14 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 6445 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:15:15.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:15:15 smithi115 conmon[90393]: 2022-05-14T01:15:15.569+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:15:16.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:15:15 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:15:16.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:15:15 smithi047 ceph-mon[77147]: pgmap v3577: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:15:16.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:15:15 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:15:16.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:15:15 smithi115 ceph-mon[78593]: pgmap v3577: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:15:16.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:15:16 smithi115 conmon[90393]: 2022-05-14T01:15:16.581+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:15:17.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:15:16 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:15:17.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:15:16 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:15:17.943 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:15:17 smithi115 conmon[90393]: 2022-05-14T01:15:17.616+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:15:18.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:15:17 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:15:18.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:15:17 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:15:18.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:15:17 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:15:18.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:15:17 smithi047 ceph-mon[77147]: pgmap v3578: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:15:18.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:15:17 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:15:18.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:15:17 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:15:18.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:15:17 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:15:18.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:15:17 smithi115 ceph-mon[78593]: pgmap v3578: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:15:18.945 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:15:18 smithi115 conmon[90393]: 2022-05-14T01:15:18.651+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:15:19.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:15:18 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:15:19.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:15:18 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:15:19.950 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:15:19 smithi115 conmon[90393]: 2022-05-14T01:15:19.684+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:15:20.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:15:19 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:15:20.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:15:19 smithi047 ceph-mon[77147]: pgmap v3579: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:15:20.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:15:19 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 6450 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:15:20.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:15:19 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:15:20.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:15:19 smithi115 ceph-mon[78593]: pgmap v3579: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:15:20.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:15:19 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 6450 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:15:20.956 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:15:20 smithi115 conmon[90393]: 2022-05-14T01:15:20.663+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:15:21.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:15:20 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:15:21.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:15:20 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:15:21.960 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:15:21 smithi115 conmon[90393]: 2022-05-14T01:15:21.663+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:15:22.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:15:21 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:15:22.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:15:21 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:15:22.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:15:21 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:15:22.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:15:21 smithi047 ceph-mon[77147]: pgmap v3580: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:15:22.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:15:21 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:15:22.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:15:21 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:15:22.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:15:21 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:15:22.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:15:21 smithi115 ceph-mon[78593]: pgmap v3580: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:15:22.962 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:15:22 smithi115 conmon[90393]: 2022-05-14T01:15:22.650+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:15:23.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:15:22 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:15:23.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:15:22 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:15:23.966 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:15:23 smithi115 conmon[90393]: 2022-05-14T01:15:23.648+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:15:24.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:15:23 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:15:24.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:15:23 smithi047 ceph-mon[77147]: pgmap v3581: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:15:24.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:15:23 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:15:24.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:15:23 smithi115 ceph-mon[78593]: pgmap v3581: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:15:24.968 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:15:24 smithi115 conmon[90393]: 2022-05-14T01:15:24.664+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:15:25.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:15:24 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:15:25.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:15:24 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 6455 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:15:25.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:15:24 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:15:25.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:15:24 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 6455 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:15:25.973 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:15:25 smithi115 conmon[90393]: 2022-05-14T01:15:25.674+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:15:26.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:15:25 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:15:26.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:15:25 smithi047 ceph-mon[77147]: pgmap v3582: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:15:26.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:15:25 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:15:26.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:15:25 smithi115 ceph-mon[78593]: pgmap v3582: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:15:26.974 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:15:26 smithi115 conmon[90393]: 2022-05-14T01:15:26.704+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:15:27.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:15:26 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:15:27.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:15:26 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:15:27.980 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:15:27 smithi115 conmon[90393]: 2022-05-14T01:15:27.676+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:15:28.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:15:27 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:15:28.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:15:27 smithi047 ceph-mon[77147]: pgmap v3583: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:15:28.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:15:27 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:15:28.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:15:27 smithi115 ceph-mon[78593]: pgmap v3583: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:15:28.981 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:15:28 smithi115 conmon[90393]: 2022-05-14T01:15:28.669+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:15:29.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:15:28 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:15:29.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:15:28 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:15:29.991 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:15:29 smithi115 conmon[90393]: 2022-05-14T01:15:29.694+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:15:30.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:15:29 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:15:30.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:15:29 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:15:30.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:15:29 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:15:30.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:15:29 smithi047 ceph-mon[77147]: pgmap v3584: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:15:30.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:15:29 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 6460 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:15:30.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:15:29 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:15:30.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:15:29 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:15:30.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:15:29 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:15:30.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:15:29 smithi115 ceph-mon[78593]: pgmap v3584: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:15:30.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:15:29 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 6460 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:15:30.992 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:15:30 smithi115 conmon[90393]: 2022-05-14T01:15:30.707+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:15:31.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:15:30 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:15:31.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:15:30 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:15:31.997 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:15:31 smithi115 conmon[90393]: 2022-05-14T01:15:31.701+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:15:32.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:15:31 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:15:32.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:15:31 smithi047 ceph-mon[77147]: pgmap v3585: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:15:32.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:15:31 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:15:32.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:15:31 smithi115 ceph-mon[78593]: pgmap v3585: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:15:32.998 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:15:32 smithi115 conmon[90393]: 2022-05-14T01:15:32.659+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:15:33.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:15:32 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:15:33.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:15:32 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:15:34.007 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:15:33 smithi115 conmon[90393]: 2022-05-14T01:15:33.700+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:15:34.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:15:34 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:15:34.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:15:34 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:15:34.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:15:34 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:15:34.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:15:34 smithi047 ceph-mon[77147]: pgmap v3586: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:15:34.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:15:34 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:15:34.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:15:34 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:15:34.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:15:34 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:15:34.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:15:34 smithi115 ceph-mon[78593]: pgmap v3586: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:15:35.009 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:15:34 smithi115 conmon[90393]: 2022-05-14T01:15:34.694+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:15:35.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:15:35 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:15:35.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:15:35 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 6465 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:15:35.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:15:35 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:15:35.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:15:35 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 6465 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:15:36.016 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:15:35 smithi115 conmon[90393]: 2022-05-14T01:15:35.735+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:15:36.297 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:15:36 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:15:36.298 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:15:36 smithi047 ceph-mon[77147]: pgmap v3587: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:15:36.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:15:36 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:15:36.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:15:36 smithi115 ceph-mon[78593]: pgmap v3587: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:15:37.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:15:36 smithi115 conmon[90393]: 2022-05-14T01:15:36.774+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:15:37.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:15:37 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:15:37.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:15:37 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:15:38.022 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:15:37 smithi115 conmon[90393]: 2022-05-14T01:15:37.745+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:15:38.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:15:38 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:15:38.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:15:38 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:15:38.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:15:38 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:15:38.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:15:38 smithi047 ceph-mon[77147]: pgmap v3588: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:15:38.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:15:38 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:15:38.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:15:38 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:15:38.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:15:38 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:15:38.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:15:38 smithi115 ceph-mon[78593]: pgmap v3588: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:15:39.030 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:15:38 smithi115 conmon[90393]: 2022-05-14T01:15:38.729+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:15:39.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:15:39 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:15:39.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:15:39 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:15:40.037 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:15:39 smithi115 conmon[90393]: 2022-05-14T01:15:39.692+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:15:40.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:15:40 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:15:40.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:15:40 smithi047 ceph-mon[77147]: pgmap v3589: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:15:40.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:15:40 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 6470 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:15:40.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:15:40 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:15:40.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:15:40 smithi115 ceph-mon[78593]: pgmap v3589: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:15:40.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:15:40 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 6470 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:15:41.038 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:15:40 smithi115 conmon[90393]: 2022-05-14T01:15:40.728+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:15:41.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:15:41 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:15:41.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:15:41 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:15:42.047 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:15:41 smithi115 conmon[90393]: 2022-05-14T01:15:41.688+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:15:42.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:15:42 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:15:42.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:15:42 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:15:42.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:15:42 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:15:42.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:15:42 smithi115 ceph-mon[78593]: pgmap v3590: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:15:42.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:15:42 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:15:42.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:15:42 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:15:42.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:15:42 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:15:42.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:15:42 smithi047 ceph-mon[77147]: pgmap v3590: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:15:43.049 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:15:42 smithi115 conmon[90393]: 2022-05-14T01:15:42.667+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:15:43.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:15:43 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:15:43.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:15:43 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:15:44.056 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:15:43 smithi115 conmon[90393]: 2022-05-14T01:15:43.681+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:15:44.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:15:44 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:15:44.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:15:44 smithi115 ceph-mon[78593]: pgmap v3591: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:15:44.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:15:44 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:15:44.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:15:44 smithi047 ceph-mon[77147]: pgmap v3591: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:15:45.057 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:15:44 smithi115 conmon[90393]: 2022-05-14T01:15:44.664+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:15:45.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:15:45 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:15:45.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:15:45 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 6475 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:15:45.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:15:45 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:15:45.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:15:45 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 6475 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:15:46.062 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:15:45 smithi115 conmon[90393]: 2022-05-14T01:15:45.629+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:15:46.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:15:46 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:15:46.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:15:46 smithi115 ceph-mon[78593]: pgmap v3592: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:15:46.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:15:46 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:15:46.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:15:46 smithi047 ceph-mon[77147]: pgmap v3592: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:15:46.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:15:46 smithi115 conmon[90393]: 2022-05-14T01:15:46.579+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:15:47.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:15:47 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:15:47.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:15:47 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:15:47.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:15:47 smithi115 conmon[90393]: 2022-05-14T01:15:47.536+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:15:48.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:15:48 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:15:48.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:15:48 smithi115 ceph-mon[78593]: pgmap v3593: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:15:48.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:15:48 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:15:48.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:15:48 smithi047 ceph-mon[77147]: pgmap v3593: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:15:48.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:15:48 smithi115 conmon[90393]: 2022-05-14T01:15:48.571+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:15:49.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:15:49 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:15:49.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:15:49 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T01:15:49.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:15:49 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T01:15:49.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:15:49 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T01:15:49.339 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:15:49 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:15:49.340 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:15:49 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T01:15:49.340 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:15:49 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T01:15:49.340 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:15:49 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T01:15:49.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:15:49 smithi115 conmon[90393]: 2022-05-14T01:15:49.553+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:15:50.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:15:50 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:15:50.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:15:50 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:15:50.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:15:50 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:15:50.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:15:50 smithi115 ceph-mon[78593]: pgmap v3594: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:15:50.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:15:50 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 6480 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:15:50.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:15:50 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:15:50.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:15:50 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:15:50.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:15:50 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:15:50.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:15:50 smithi047 ceph-mon[77147]: pgmap v3594: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:15:50.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:15:50 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 6480 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:15:50.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:15:50 smithi115 conmon[90393]: 2022-05-14T01:15:50.552+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:15:51.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:15:51 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:15:51.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:15:51 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:15:51.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:15:51 smithi115 conmon[90393]: 2022-05-14T01:15:51.542+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:15:52.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:15:52 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:15:52.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:15:52 smithi047 ceph-mon[77147]: pgmap v3595: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:15:52.291 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:15:51 smithi047 conmon[78521]: 2022-05-14T01:15:51.920+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:15:52.292 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:15:51 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T01:15:52.292 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:15:51 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:15:52.292 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:15:51 smithi047 conmon[78521]: yield 2022-05-14T01:15:52.292 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:15:51 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:15:52.293 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:15:51 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:15:52.293 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:15:51 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:15:52.293 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:15:51 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T01:15:52.293 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:15:51 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:15:52.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:15:51 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T01:15:52.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:15:51 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:15:52.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:15:51 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T01:15:52.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:15:51 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:15:52.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:15:51 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T01:15:52.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:15:51 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:15:52.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:15:51 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T01:15:52.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:15:51 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:15:52.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:15:51 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:15:52.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:15:51 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:15:52.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:15:52 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:15:52.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:15:52 smithi115 ceph-mon[78593]: pgmap v3595: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:15:52.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:15:52 smithi115 conmon[90393]: 2022-05-14T01:15:52.574+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:15:53.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:15:53 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:15:53.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:15:53 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:15:53.358 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T01:15:53.358 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:15:53.358 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T01:15:53.359 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:15:53.359 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:15:53.359 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:15:53.359 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T01:15:53.359 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:15:53.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T01:15:53.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:15:53.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T01:15:53.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:15:53.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T01:15:53.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:15:53.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T01:15:53.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:15:53.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:15:53.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:15:53.362 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:15:53 smithi115 ceph-mon[78593]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T01:15:53.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:15:53 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:15:53.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:15:53 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:15:53.541 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T01:15:53.542 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:15:53.542 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T01:15:53.542 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:15:53.542 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:15:53.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:15:53.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T01:15:53.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:15:53.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T01:15:53.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:15:53.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T01:15:53.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:15:53.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T01:15:53.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:15:53.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T01:15:53.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:15:53.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:15:53.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:15:53.546 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:15:53 smithi047 ceph-mon[77147]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T01:15:54.092 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:15:53 smithi115 conmon[90393]: 2022-05-14T01:15:53.608+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:15:54.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:15:54 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:15:54.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:15:54 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:15:54.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:15:54 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:15:54.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:15:54 smithi115 ceph-mon[78593]: pgmap v3596: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:15:54.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:15:54 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:15:54.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:15:54 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:15:54.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:15:54 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:15:54.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:15:54 smithi047 ceph-mon[77147]: pgmap v3596: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:15:54.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:15:54 smithi115 conmon[90393]: 2022-05-14T01:15:54.564+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:15:55.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:15:55 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:15:55.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:15:55 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 6485 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:15:55.292 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:15:54 smithi047 conmon[78521]: 2022-05-14T01:15:54.992+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:15:55.292 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:15:54 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T01:15:55.292 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:15:54 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:15:55.292 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:15:54 smithi047 conmon[78521]: yield 2022-05-14T01:15:55.293 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:15:54 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:15:55.293 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:15:54 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:15:55.293 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:15:54 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:15:55.293 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:15:54 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T01:15:55.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:15:54 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:15:55.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:15:54 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T01:15:55.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:15:54 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:15:55.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:15:54 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T01:15:55.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:15:54 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:15:55.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:15:54 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T01:15:55.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:15:54 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:15:55.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:15:54 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T01:15:55.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:15:54 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:15:55.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:15:54 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:15:55.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:15:54 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:15:55.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:15:54 smithi047 conmon[78521]: 2022-05-14T01:15:54.993+0000 7f5341f7c700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T01:15:55.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:15:54 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T01:15:55.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:15:54 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:15:55.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:15:54 smithi047 conmon[78521]: yield 2022-05-14T01:15:55.297 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:15:54 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:15:55.297 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:15:54 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:15:55.297 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:15:54 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:15:55.297 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:15:54 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T01:15:55.298 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:15:54 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:15:55.298 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:15:54 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T01:15:55.298 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:15:54 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:15:55.298 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:15:54 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T01:15:55.298 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:15:54 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:15:55.299 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:15:54 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T01:15:55.299 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:15:54 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:15:55.299 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:15:54 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T01:15:55.299 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:15:54 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:15:55.299 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:15:54 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:15:55.300 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:15:54 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:15:55.300 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:15:54 smithi047 conmon[78521]: 2022-05-14T01:15:55.300 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:15:54 smithi047 conmon[78521]: During handling of the above exception, another exception occurred: 2022-05-14T01:15:55.300 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:15:54 smithi047 conmon[78521]: 2022-05-14T01:15:55.300 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:15:54 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T01:15:55.301 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:15:54 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T01:15:55.301 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:15:54 smithi047 conmon[78521]: return f(*arg) 2022-05-14T01:15:55.301 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:15:54 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T01:15:55.301 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:15:54 smithi047 conmon[78521]: self._write_client_files(client_files, host) 2022-05-14T01:15:55.301 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:15:54 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T01:15:55.302 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:15:54 smithi047 conmon[78521]: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T01:15:55.302 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:15:54 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T01:15:55.302 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:15:54 smithi047 conmon[78521]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T01:15:55.302 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:15:54 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T01:15:55.303 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:15:54 smithi047 conmon[78521]: return self.event_loop.get_result(coro) 2022-05-14T01:15:55.303 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:15:54 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T01:15:55.304 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:15:54 smithi047 conmon[78521]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T01:15:55.304 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:15:54 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T01:15:55.304 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:15:54 smithi047 conmon[78521]: return self.__get_result() 2022-05-14T01:15:55.304 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:15:54 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T01:15:55.304 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:15:54 smithi047 conmon[78521]: raise self._exception 2022-05-14T01:15:55.305 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:15:54 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T01:15:55.305 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:15:54 smithi047 conmon[78521]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T01:15:55.305 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:15:54 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T01:15:55.305 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:15:54 smithi047 conmon[78521]: conn = await self._remote_connection(host, addr) 2022-05-14T01:15:55.306 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:15:54 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T01:15:55.306 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:15:54 smithi047 conmon[78521]: raise 2022-05-14T01:15:55.306 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:15:54 smithi047 conmon[78521]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T01:15:55.306 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:15:54 smithi047 conmon[78521]: self.gen.throw(type, value, traceback) 2022-05-14T01:15:55.306 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:15:54 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T01:15:55.307 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:15:54 smithi047 conmon[78521]: raise OrchestratorError(msg) 2022-05-14T01:15:55.307 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:15:54 smithi047 conmon[78521]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:15:55.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:15:55 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:15:55.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:15:55 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 6485 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:15:55.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:15:55 smithi115 conmon[90393]: 2022-05-14T01:15:55.602+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:15:56.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:15:56 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:15:56.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:15:56 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:15:56.358 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T01:15:56.358 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:15:56.358 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T01:15:56.359 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:15:56.359 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:15:56.359 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:15:56.359 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T01:15:56.359 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:15:56.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T01:15:56.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:15:56.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T01:15:56.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:15:56.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T01:15:56.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:15:56.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T01:15:56.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:15:56.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:15:56.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:15:56.362 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:15:56 smithi115 ceph-mon[78593]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T01:15:56.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T01:15:56.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:15:56.365 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T01:15:56.365 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:15:56.366 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:15:56.366 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:15:56.366 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T01:15:56.366 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:15:56.366 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T01:15:56.367 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:15:56.367 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T01:15:56.367 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:15:56.367 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T01:15:56.368 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:15:56.368 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T01:15:56.368 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:15:56.368 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:15:56.368 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:15:56.369 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T01:15:56.369 INFO:journalctl@ceph.mon.c.smithi115.stdout: During handling of the above exception, another exception occurred: 2022-05-14T01:15:56.369 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T01:15:56.369 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T01:15:56.369 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T01:15:56.370 INFO:journalctl@ceph.mon.c.smithi115.stdout: return f(*arg) 2022-05-14T01:15:56.370 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T01:15:56.370 INFO:journalctl@ceph.mon.c.smithi115.stdout: self._write_client_files(client_files, host) 2022-05-14T01:15:56.370 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T01:15:56.370 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T01:15:56.371 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T01:15:56.371 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T01:15:56.371 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T01:15:56.372 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.event_loop.get_result(coro) 2022-05-14T01:15:56.372 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T01:15:56.372 INFO:journalctl@ceph.mon.c.smithi115.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T01:15:56.372 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T01:15:56.372 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.__get_result() 2022-05-14T01:15:56.373 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T01:15:56.373 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise self._exception 2022-05-14T01:15:56.373 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T01:15:56.373 INFO:journalctl@ceph.mon.c.smithi115.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T01:15:56.374 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T01:15:56.374 INFO:journalctl@ceph.mon.c.smithi115.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T01:15:56.374 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T01:15:56.374 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise 2022-05-14T01:15:56.374 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T01:15:56.375 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.gen.throw(type, value, traceback) 2022-05-14T01:15:56.375 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T01:15:56.375 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OrchestratorError(msg) 2022-05-14T01:15:56.375 INFO:journalctl@ceph.mon.c.smithi115.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:15:56.375 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:15:56 smithi115 ceph-mon[78593]: pgmap v3597: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:15:56.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:15:56 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:15:56.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:15:56 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:15:56.541 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T01:15:56.541 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:15:56.542 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T01:15:56.542 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:15:56.542 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:15:56.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:15:56.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T01:15:56.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:15:56.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T01:15:56.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:15:56.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T01:15:56.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:15:56.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T01:15:56.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:15:56.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T01:15:56.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:15:56.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:15:56.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:15:56.546 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:15:56 smithi047 ceph-mon[77147]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T01:15:56.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T01:15:56.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:15:56.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T01:15:56.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:15:56.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:15:56.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:15:56.548 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T01:15:56.548 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:15:56.548 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T01:15:56.548 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:15:56.548 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T01:15:56.549 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:15:56.549 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T01:15:56.549 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:15:56.549 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T01:15:56.550 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:15:56.550 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:15:56.550 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:15:56.550 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T01:15:56.550 INFO:journalctl@ceph.mon.b.smithi047.stdout: During handling of the above exception, another exception occurred: 2022-05-14T01:15:56.551 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T01:15:56.551 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T01:15:56.551 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T01:15:56.551 INFO:journalctl@ceph.mon.b.smithi047.stdout: return f(*arg) 2022-05-14T01:15:56.552 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T01:15:56.552 INFO:journalctl@ceph.mon.b.smithi047.stdout: self._write_client_files(client_files, host) 2022-05-14T01:15:56.552 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T01:15:56.552 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T01:15:56.552 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T01:15:56.553 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T01:15:56.553 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T01:15:56.553 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.event_loop.get_result(coro) 2022-05-14T01:15:56.553 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T01:15:56.554 INFO:journalctl@ceph.mon.b.smithi047.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T01:15:56.554 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T01:15:56.554 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.__get_result() 2022-05-14T01:15:56.554 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T01:15:56.555 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise self._exception 2022-05-14T01:15:56.555 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T01:15:56.555 INFO:journalctl@ceph.mon.b.smithi047.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T01:15:56.555 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T01:15:56.555 INFO:journalctl@ceph.mon.b.smithi047.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T01:15:56.556 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T01:15:56.556 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise 2022-05-14T01:15:56.556 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T01:15:56.556 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.gen.throw(type, value, traceback) 2022-05-14T01:15:56.556 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T01:15:56.557 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OrchestratorError(msg) 2022-05-14T01:15:56.557 INFO:journalctl@ceph.mon.b.smithi047.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:15:56.557 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:15:56 smithi047 ceph-mon[77147]: pgmap v3597: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:15:56.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:15:56 smithi115 conmon[90393]: 2022-05-14T01:15:56.587+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:15:57.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:15:57 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:15:57.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:15:57 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:15:57.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:15:57 smithi115 conmon[90393]: 2022-05-14T01:15:57.589+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:15:58.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:15:58 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:15:58.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:15:58 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:15:58.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:15:58 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:15:58.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:15:58 smithi047 ceph-mon[77147]: pgmap v3598: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:15:58.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:15:58 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:15:58.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:15:58 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:15:58.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:15:58 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:15:58.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:15:58 smithi115 ceph-mon[78593]: pgmap v3598: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:15:58.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:15:58 smithi115 conmon[90393]: 2022-05-14T01:15:58.606+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:15:59.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:15:59 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:15:59.570 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:15:59 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:15:59.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:15:59 smithi115 conmon[90393]: 2022-05-14T01:15:59.569+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:16:00.540 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:16:00 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:16:00.540 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:16:00 smithi115 ceph-mon[78593]: pgmap v3599: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:16:00.540 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:16:00 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 6490 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:16:00.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:16:00 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:16:00.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:16:00 smithi047 ceph-mon[77147]: pgmap v3599: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:16:00.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:16:00 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 6490 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:16:00.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:16:00 smithi115 conmon[90393]: 2022-05-14T01:16:00.539+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:16:01.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:16:01 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:16:01.550 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:16:01 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:16:01.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:16:01 smithi115 conmon[90393]: 2022-05-14T01:16:01.549+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:16:02.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:16:02 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:16:02.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:16:02 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:16:02.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:16:02 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:16:02.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:16:02 smithi047 ceph-mon[77147]: pgmap v3600: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:16:02.554 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:16:02 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:16:02.555 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:16:02 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:16:02.555 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:16:02 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:16:02.555 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:16:02 smithi115 ceph-mon[78593]: pgmap v3600: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:16:02.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:16:02 smithi115 conmon[90393]: 2022-05-14T01:16:02.553+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:16:03.533 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:16:03 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:16:03.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:16:03 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:16:03.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:16:03 smithi115 conmon[90393]: 2022-05-14T01:16:03.532+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:16:04.537 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:16:04 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:16:04.537 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:16:04 smithi115 ceph-mon[78593]: pgmap v3601: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:16:04.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:16:04 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:16:04.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:16:04 smithi047 ceph-mon[77147]: pgmap v3601: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:16:04.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:16:04 smithi115 conmon[90393]: 2022-05-14T01:16:04.536+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:16:05.527 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:16:05 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:16:05.527 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:16:05 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 6495 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:16:05.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:16:05 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:16:05.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:16:05 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 6495 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:16:05.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:16:05 smithi115 conmon[90393]: 2022-05-14T01:16:05.526+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:16:06.523 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:16:06 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:16:06.523 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:16:06 smithi115 ceph-mon[78593]: pgmap v3602: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:16:06.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:16:06 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:16:06.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:16:06 smithi047 ceph-mon[77147]: pgmap v3602: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:16:06.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:16:06 smithi115 conmon[90393]: 2022-05-14T01:16:06.522+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:16:07.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:16:07 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:16:07.560 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:16:07 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:16:07.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:16:07 smithi115 conmon[90393]: 2022-05-14T01:16:07.559+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:16:08.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:16:08 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:16:08.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:16:08 smithi047 ceph-mon[77147]: pgmap v3603: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:16:08.602 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:16:08 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:16:08.602 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:16:08 smithi115 ceph-mon[78593]: pgmap v3603: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:16:08.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:16:08 smithi115 conmon[90393]: 2022-05-14T01:16:08.601+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:16:09.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:16:09 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:16:09.566 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:16:09 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:16:09.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:16:09 smithi115 conmon[90393]: 2022-05-14T01:16:09.565+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:16:10.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:16:10 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:16:10.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:16:10 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:16:10.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:16:10 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:16:10.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:16:10 smithi047 ceph-mon[77147]: pgmap v3604: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:16:10.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:16:10 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 6500 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:16:10.582 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:16:10 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:16:10.582 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:16:10 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:16:10.582 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:16:10 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:16:10.583 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:16:10 smithi115 ceph-mon[78593]: pgmap v3604: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:16:10.583 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:16:10 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 6500 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:16:10.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:16:10 smithi115 conmon[90393]: 2022-05-14T01:16:10.581+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:16:11.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:16:11 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:16:11.585 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:16:11 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:16:11.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:16:11 smithi115 conmon[90393]: 2022-05-14T01:16:11.584+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:16:12.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:16:12 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:16:12.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:16:12 smithi047 ceph-mon[77147]: pgmap v3605: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:16:12.577 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:16:12 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:16:12.578 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:16:12 smithi115 ceph-mon[78593]: pgmap v3605: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:16:12.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:16:12 smithi115 conmon[90393]: 2022-05-14T01:16:12.576+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:16:13.546 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:16:13 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:16:13.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:16:13 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:16:13.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:16:13 smithi115 conmon[90393]: 2022-05-14T01:16:13.540+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:16:14.517 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:16:14 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:16:14.517 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:16:14 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:16:14.517 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:16:14 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:16:14.518 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:16:14 smithi115 ceph-mon[78593]: pgmap v3606: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:16:14.518 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:16:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T01:16:14.518 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:16:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T01:16:14.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:16:14 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:16:14.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:16:14 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:16:14.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:16:14 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:16:14.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:16:14 smithi047 ceph-mon[77147]: pgmap v3606: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:16:14.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:16:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T01:16:14.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:16:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T01:16:14.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:16:14 smithi115 conmon[90393]: 2022-05-14T01:16:14.516+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:16:15.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:16:15 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:16:15.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:16:15 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 6505 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:16:15.565 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:16:15 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:16:15.565 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:16:15 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 6505 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:16:15.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:16:15 smithi115 conmon[90393]: 2022-05-14T01:16:15.564+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:16:16.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:16:16 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:16:16.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:16:16 smithi047 ceph-mon[77147]: pgmap v3607: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:16:16.593 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:16:16 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:16:16.594 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:16:16 smithi115 ceph-mon[78593]: pgmap v3607: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:16:16.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:16:16 smithi115 conmon[90393]: 2022-05-14T01:16:16.592+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:16:17.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:16:17 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:16:17.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:16:17 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:16:17.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:16:17 smithi115 conmon[90393]: 2022-05-14T01:16:17.543+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:16:18.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:16:18 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:16:18.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:16:18 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:16:18.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:16:18 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:16:18.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:16:18 smithi047 ceph-mon[77147]: pgmap v3608: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:16:18.561 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:16:18 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:16:18.561 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:16:18 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:16:18.562 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:16:18 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:16:18.562 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:16:18 smithi115 ceph-mon[78593]: pgmap v3608: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:16:18.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:16:18 smithi115 conmon[90393]: 2022-05-14T01:16:18.560+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:16:19.528 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:16:19 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:16:19.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:16:19 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:16:19.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:16:19 smithi115 conmon[90393]: 2022-05-14T01:16:19.528+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:16:20.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:16:20 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:16:20.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:16:20 smithi047 ceph-mon[77147]: pgmap v3609: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:16:20.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:16:20 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 6510 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:16:20.575 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:16:20 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:16:20.576 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:16:20 smithi115 ceph-mon[78593]: pgmap v3609: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:16:20.576 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:16:20 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 6510 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:16:20.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:16:20 smithi115 conmon[90393]: 2022-05-14T01:16:20.574+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:16:21.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:16:21 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:16:21.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:16:21 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:16:22.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:16:21 smithi115 conmon[90393]: 2022-05-14T01:16:21.618+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:16:22.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:16:22 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:16:22.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:16:22 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:16:22.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:16:22 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:16:22.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:16:22 smithi047 ceph-mon[77147]: pgmap v3610: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:16:22.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:16:22 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:16:22.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:16:22 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:16:22.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:16:22 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:16:22.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:16:22 smithi115 ceph-mon[78593]: pgmap v3610: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:16:23.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:16:22 smithi115 conmon[90393]: 2022-05-14T01:16:22.649+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:16:23.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:16:23 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:16:23.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:16:23 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:16:24.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:16:23 smithi115 conmon[90393]: 2022-05-14T01:16:23.679+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:16:24.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:16:24 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:16:24.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:16:24 smithi047 ceph-mon[77147]: pgmap v3611: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:16:24.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:16:24 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:16:24.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:16:24 smithi115 ceph-mon[78593]: pgmap v3611: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:16:25.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:16:24 smithi115 conmon[90393]: 2022-05-14T01:16:24.689+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:16:25.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:16:25 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:16:25.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:16:25 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 6515 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:16:25.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:16:25 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:16:25.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:16:25 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 6515 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:16:26.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:16:25 smithi115 conmon[90393]: 2022-05-14T01:16:25.641+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:16:26.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:16:26 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:16:26.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:16:26 smithi047 ceph-mon[77147]: pgmap v3612: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:16:26.597 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:16:26 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:16:26.597 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:16:26 smithi115 ceph-mon[78593]: pgmap v3612: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:16:26.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:16:26 smithi115 conmon[90393]: 2022-05-14T01:16:26.596+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:16:27.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:16:27 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:16:27.550 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:16:27 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:16:27.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:16:27 smithi115 conmon[90393]: 2022-05-14T01:16:27.550+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:16:28.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:16:28 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:16:28.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:16:28 smithi047 ceph-mon[77147]: pgmap v3613: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:16:28.559 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:16:28 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:16:28.559 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:16:28 smithi115 ceph-mon[78593]: pgmap v3613: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:16:28.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:16:28 smithi115 conmon[90393]: 2022-05-14T01:16:28.558+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:16:29.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:16:29 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:16:29.581 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:16:29 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:16:29.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:16:29 smithi115 conmon[90393]: 2022-05-14T01:16:29.577+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:16:30.237 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:16:30 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:16:30.237 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:16:30 smithi115 ceph-mon[78593]: pgmap v3614: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:16:30.238 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:16:30 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:16:30.238 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:16:30 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:16:30.238 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:16:30 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 6525 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:16:30.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:16:30 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:16:30.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:16:30 smithi047 ceph-mon[77147]: pgmap v3614: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:16:30.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:16:30 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:16:30.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:16:30 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:16:30.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:16:30 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 6525 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:16:30.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:16:30 smithi115 conmon[90393]: 2022-05-14T01:16:30.528+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:16:31.498 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:16:31 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:16:31.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:16:31 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:16:31.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:16:31 smithi115 conmon[90393]: 2022-05-14T01:16:31.497+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:16:32.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:16:32 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:16:32.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:16:32 smithi115 ceph-mon[78593]: pgmap v3615: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:16:32.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:16:32 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:16:32.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:16:32 smithi047 ceph-mon[77147]: pgmap v3615: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:16:32.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:16:32 smithi115 conmon[90393]: 2022-05-14T01:16:32.540+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:16:33.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:16:33 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:16:33.550 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:16:33 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:16:33.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:16:33 smithi115 conmon[90393]: 2022-05-14T01:16:33.549+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:16:34.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:16:34 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:16:34.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:16:34 smithi047 ceph-mon[77147]: pgmap v3616: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:16:34.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:16:34 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:16:34.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:16:34 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:16:34.582 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:16:34 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:16:34.582 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:16:34 smithi115 ceph-mon[78593]: pgmap v3616: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:16:34.582 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:16:34 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:16:34.583 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:16:34 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:16:34.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:16:34 smithi115 conmon[90393]: 2022-05-14T01:16:34.581+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:16:35.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:16:35 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:16:35.596 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:16:35 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:16:35.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:16:35 smithi115 conmon[90393]: 2022-05-14T01:16:35.596+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:16:36.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:16:36 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:16:36.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:16:36 smithi047 ceph-mon[77147]: pgmap v3617: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:16:36.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:16:36 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 6530 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:16:36.569 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:16:36 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:16:36.570 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:16:36 smithi115 ceph-mon[78593]: pgmap v3617: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:16:36.571 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:16:36 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 6530 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:16:36.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:16:36 smithi115 conmon[90393]: 2022-05-14T01:16:36.569+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:16:37.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:16:37 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:16:37.591 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:16:37 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:16:37.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:16:37 smithi115 conmon[90393]: 2022-05-14T01:16:37.591+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:16:38.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:16:38 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:16:38.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:16:38 smithi047 ceph-mon[77147]: pgmap v3618: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:16:38.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:16:38 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:16:38.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:16:38 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:16:38.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:16:38 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:16:38.553 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:16:38 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:16:38.553 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:16:38 smithi115 ceph-mon[78593]: pgmap v3618: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:16:38.553 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:16:38 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:16:38.554 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:16:38 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:16:38.554 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:16:38 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:16:38.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:16:38 smithi115 conmon[90393]: 2022-05-14T01:16:38.552+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:16:39.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:16:39 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:16:39.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:16:39 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:16:39.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:16:39 smithi115 conmon[90393]: 2022-05-14T01:16:39.505+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:16:40.530 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:16:40 smithi115 ceph-mon[78593]: pgmap v3619: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:16:40.531 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:16:40 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:16:40.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:16:40 smithi047 ceph-mon[77147]: pgmap v3619: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:16:40.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:16:40 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:16:40.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:16:40 smithi115 conmon[90393]: 2022-05-14T01:16:40.529+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:16:41.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:16:41 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:16:41.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:16:41 smithi115 conmon[90393]: 2022-05-14T01:16:41.511+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:16:41.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:16:41 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:16:42.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:16:42 smithi047 ceph-mon[77147]: pgmap v3620: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:16:42.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:16:42 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:16:42.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:16:42 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:16:42.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:16:42 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:16:42.549 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:16:42 smithi115 ceph-mon[78593]: pgmap v3620: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:16:42.549 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:16:42 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:16:42.549 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:16:42 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:16:42.550 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:16:42 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:16:42.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:16:42 smithi115 conmon[90393]: 2022-05-14T01:16:42.548+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:16:43.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:16:43 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:16:43.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:16:43 smithi115 conmon[90393]: 2022-05-14T01:16:43.509+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:16:43.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:16:43 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:16:44.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:16:44 smithi047 ceph-mon[77147]: pgmap v3621: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:16:44.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:16:44 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:16:44.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:16:44 smithi115 conmon[90393]: 2022-05-14T01:16:44.507+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:16:44.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:16:44 smithi115 ceph-mon[78593]: pgmap v3621: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:16:44.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:16:44 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:16:45.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:16:45 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:16:45.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:16:45 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 6535 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:16:45.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:16:45 smithi115 conmon[90393]: 2022-05-14T01:16:45.500+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:16:45.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:16:45 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:16:45.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:16:45 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 6535 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:16:46.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:16:46 smithi047 ceph-mon[77147]: pgmap v3622: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:16:46.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:16:46 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:16:46.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:16:46 smithi115 ceph-mon[78593]: pgmap v3622: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:16:46.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:16:46 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:16:46.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:16:46 smithi115 conmon[90393]: 2022-05-14T01:16:46.545+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:16:47.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:16:47 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:16:47.554 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:16:47 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:16:47.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:16:47 smithi115 conmon[90393]: 2022-05-14T01:16:47.553+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:16:48.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:16:48 smithi047 ceph-mon[77147]: pgmap v3623: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:16:48.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:16:48 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:16:48.603 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:16:48 smithi115 ceph-mon[78593]: pgmap v3623: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:16:48.604 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:16:48 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:16:48.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:16:48 smithi115 conmon[90393]: 2022-05-14T01:16:48.602+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:16:49.562 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:16:49 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:16:49.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:16:49 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:16:49.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:16:49 smithi115 conmon[90393]: 2022-05-14T01:16:49.560+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:16:50.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:16:50 smithi115 conmon[90393]: 2022-05-14T01:16:50.521+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:16:50.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:16:50 smithi115 ceph-mon[78593]: pgmap v3624: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:16:50.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:16:50 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:16:50.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:16:50 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:16:50.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:16:50 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:16:50.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:16:50 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 6545 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:16:50.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:16:50 smithi047 ceph-mon[77147]: pgmap v3624: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:16:50.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:16:50 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:16:50.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:16:50 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:16:50.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:16:50 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:16:50.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:16:50 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 6545 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:16:51.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:16:51 smithi115 conmon[90393]: 2022-05-14T01:16:51.494+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:16:51.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:16:51 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:16:51.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:16:51 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:16:52.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:16:52 smithi115 conmon[90393]: 2022-05-14T01:16:52.512+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:16:52.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:16:52 smithi115 ceph-mon[78593]: pgmap v3625: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:16:52.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:16:52 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:16:52.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:16:52 smithi047 ceph-mon[77147]: pgmap v3625: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:16:52.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:16:52 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:16:53.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:16:53 smithi115 conmon[90393]: 2022-05-14T01:16:53.474+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:16:53.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:16:53 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:16:53.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:16:53 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:16:54.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:16:54 smithi115 conmon[90393]: 2022-05-14T01:16:54.463+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:16:54.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:16:54 smithi115 ceph-mon[78593]: pgmap v3626: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:16:54.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:16:54 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:16:54.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:16:54 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:16:54.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:16:54 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:16:54.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:16:54 smithi047 ceph-mon[77147]: pgmap v3626: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:16:54.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:16:54 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:16:54.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:16:54 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:16:54.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:16:54 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:16:55.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:16:55 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:16:55.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:16:55 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T01:16:55.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:16:55 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T01:16:55.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:16:55 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T01:16:55.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:16:55 smithi047 ceph-mon[77147]: pgmap v3627: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:16:55.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:16:55 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' 2022-05-14T01:16:55.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:16:55 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' 2022-05-14T01:16:55.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:16:55 smithi115 conmon[90393]: 2022-05-14T01:16:55.445+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:16:55.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:16:55 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:16:55.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:16:55 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T01:16:55.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:16:55 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T01:16:55.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:16:55 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T01:16:55.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:16:55 smithi115 ceph-mon[78593]: pgmap v3627: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:16:55.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:16:55 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' 2022-05-14T01:16:55.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:16:55 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' 2022-05-14T01:16:56.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:16:56 smithi115 conmon[90393]: 2022-05-14T01:16:56.479+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:16:56.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:16:56 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:16:56.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:16:56 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 6550 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:16:57.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:16:56 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:16:57.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:16:56 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 6550 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:16:57.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:16:57 smithi115 conmon[90393]: 2022-05-14T01:16:57.459+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:16:57.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:16:57 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:16:57.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:16:57 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:16:57.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:16:57 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:16:57.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:16:57 smithi115 ceph-mon[78593]: pgmap v3628: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:16:58.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:16:57 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:16:58.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:16:57 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:16:58.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:16:57 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:16:58.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:16:57 smithi047 ceph-mon[77147]: pgmap v3628: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:16:58.541 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:16:58 smithi047 conmon[78521]: 2022-05-14T01:16:58.095+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:16:58.542 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:16:58 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T01:16:58.542 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:16:58 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:16:58.542 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:16:58 smithi047 conmon[78521]: yield 2022-05-14T01:16:58.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:16:58 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:16:58.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:16:58 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:16:58.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:16:58 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:16:58.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:16:58 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T01:16:58.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:16:58 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:16:58.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:16:58 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T01:16:58.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:16:58 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:16:58.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:16:58 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T01:16:58.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:16:58 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:16:58.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:16:58 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T01:16:58.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:16:58 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:16:58.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:16:58 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T01:16:58.547 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:16:58 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:16:58.547 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:16:58 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:16:58.547 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:16:58 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:16:58.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:16:58 smithi115 conmon[90393]: 2022-05-14T01:16:58.456+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:16:59.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:16:59 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:16:59.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:16:59 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:16:59.542 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T01:16:59.542 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:16:59.542 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T01:16:59.542 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:16:59.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:16:59.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:16:59.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T01:16:59.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:16:59.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T01:16:59.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:16:59.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T01:16:59.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:16:59.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T01:16:59.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:16:59.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T01:16:59.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:16:59.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:16:59.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:16:59.546 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:16:59 smithi047 ceph-mon[77147]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T01:16:59.546 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:16:59 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' 2022-05-14T01:16:59.546 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:16:59 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' 2022-05-14T01:16:59.546 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:16:59 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-05-14T01:16:59.546 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:16:59 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-05-14T01:16:59.547 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:16:59 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-05-14T01:16:59.547 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:16:59 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-05-14T01:16:59.547 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:16:59 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-05-14T01:16:59.547 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:16:59 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-05-14T01:16:59.548 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:16:59 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-05-14T01:16:59.548 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:16:59 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-05-14T01:16:59.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:16:59 smithi115 conmon[90393]: 2022-05-14T01:16:59.420+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:16:59.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:16:59 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:16:59.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:16:59 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:16:59.608 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T01:16:59.609 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:16:59.609 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T01:16:59.609 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:16:59.609 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:16:59.609 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:16:59.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T01:16:59.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:16:59.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T01:16:59.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:16:59.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T01:16:59.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:16:59.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T01:16:59.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:16:59.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T01:16:59.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:16:59.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:16:59.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:16:59.612 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:16:59 smithi115 ceph-mon[78593]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T01:16:59.613 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:16:59 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' 2022-05-14T01:16:59.613 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:16:59 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' 2022-05-14T01:16:59.613 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:16:59 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-05-14T01:16:59.613 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:16:59 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-05-14T01:16:59.614 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:16:59 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-05-14T01:16:59.614 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:16:59 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-05-14T01:16:59.614 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:16:59 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-05-14T01:16:59.614 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:16:59 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-05-14T01:16:59.614 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:16:59 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-05-14T01:16:59.615 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:16:59 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-05-14T01:17:00.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:17:00 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:17:00.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:17:00 smithi047 ceph-mon[77147]: Adjusting osd_memory_target on smithi047 to 198.1M 2022-05-14T01:17:00.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:17:00 smithi047 ceph-mon[77147]: Unable to set osd_memory_target on smithi047 to 207755878: error parsing value: Value '207755878' is below minimum 939524096 2022-05-14T01:17:00.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:17:00 smithi047 ceph-mon[77147]: Adjusting osd_memory_target on smithi115 to 197.0M 2022-05-14T01:17:00.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:17:00 smithi047 ceph-mon[77147]: Unable to set osd_memory_target on smithi115 to 206629785: error parsing value: Value '206629785' is below minimum 939524096 2022-05-14T01:17:00.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:17:00 smithi047 ceph-mon[77147]: pgmap v3629: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:17:00.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:17:00 smithi115 conmon[90393]: 2022-05-14T01:17:00.386+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:17:00.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:17:00 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:17:00.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:17:00 smithi115 ceph-mon[78593]: Adjusting osd_memory_target on smithi047 to 198.1M 2022-05-14T01:17:00.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:17:00 smithi115 ceph-mon[78593]: Unable to set osd_memory_target on smithi047 to 207755878: error parsing value: Value '207755878' is below minimum 939524096 2022-05-14T01:17:00.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:17:00 smithi115 ceph-mon[78593]: Adjusting osd_memory_target on smithi115 to 197.0M 2022-05-14T01:17:00.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:17:00 smithi115 ceph-mon[78593]: Unable to set osd_memory_target on smithi115 to 206629785: error parsing value: Value '206629785' is below minimum 939524096 2022-05-14T01:17:00.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:17:00 smithi115 ceph-mon[78593]: pgmap v3629: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:17:01.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:17:01 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:17:01.542 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:17:01 smithi047 conmon[78521]: 2022-05-14T01:17:01.167+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:17:01.542 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:17:01 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T01:17:01.542 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:17:01 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:17:01.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:17:01 smithi047 conmon[78521]: yield 2022-05-14T01:17:01.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:17:01 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:17:01.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:17:01 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:17:01.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:17:01 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:17:01.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:17:01 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T01:17:01.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:17:01 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:17:01.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:17:01 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T01:17:01.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:17:01 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:17:01.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:17:01 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T01:17:01.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:17:01 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:17:01.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:17:01 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T01:17:01.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:17:01 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:17:01.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:17:01 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T01:17:01.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:17:01 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:17:01.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:17:01 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:17:01.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:17:01 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:17:01.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:17:01 smithi047 conmon[78521]: 2022-05-14T01:17:01.168+0000 7f5344781700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T01:17:01.547 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:17:01 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T01:17:01.547 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:17:01 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:17:01.547 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:17:01 smithi047 conmon[78521]: yield 2022-05-14T01:17:01.548 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:17:01 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:17:01.548 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:17:01 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:17:01.548 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:17:01 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:17:01.548 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:17:01 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T01:17:01.549 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:17:01 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:17:01.549 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:17:01 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T01:17:01.549 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:17:01 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:17:01.549 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:17:01 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T01:17:01.550 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:17:01 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:17:01.550 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:17:01 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T01:17:01.550 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:17:01 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:17:01.550 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:17:01 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T01:17:01.550 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:17:01 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:17:01.551 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:17:01 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:17:01.551 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:17:01 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:17:01.551 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:17:01 smithi047 conmon[78521]: 2022-05-14T01:17:01.551 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:17:01 smithi047 conmon[78521]: During handling of the above exception, another exception occurred: 2022-05-14T01:17:01.552 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:17:01 smithi047 conmon[78521]: 2022-05-14T01:17:01.552 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:17:01 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T01:17:01.552 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:17:01 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T01:17:01.552 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:17:01 smithi047 conmon[78521]: return f(*arg) 2022-05-14T01:17:01.552 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:17:01 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T01:17:01.553 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:17:01 smithi047 conmon[78521]: self._write_client_files(client_files, host) 2022-05-14T01:17:01.553 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:17:01 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T01:17:01.553 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:17:01 smithi047 conmon[78521]: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T01:17:01.553 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:17:01 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T01:17:01.554 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:17:01 smithi047 conmon[78521]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T01:17:01.554 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:17:01 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T01:17:01.554 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:17:01 smithi047 conmon[78521]: return self.event_loop.get_result(coro) 2022-05-14T01:17:01.554 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:17:01 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T01:17:01.554 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:17:01 smithi047 conmon[78521]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T01:17:01.555 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:17:01 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T01:17:01.555 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:17:01 smithi047 conmon[78521]: return self.__get_result() 2022-05-14T01:17:01.555 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:17:01 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T01:17:01.555 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:17:01 smithi047 conmon[78521]: raise self._exception 2022-05-14T01:17:01.555 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:17:01 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T01:17:01.556 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:17:01 smithi047 conmon[78521]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T01:17:01.556 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:17:01 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T01:17:01.556 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:17:01 smithi047 conmon[78521]: conn = await self._remote_connection(host, addr) 2022-05-14T01:17:01.556 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:17:01 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T01:17:01.557 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:17:01 smithi047 conmon[78521]: raise 2022-05-14T01:17:01.557 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:17:01 smithi047 conmon[78521]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T01:17:01.557 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:17:01 smithi047 conmon[78521]: self.gen.throw(type, value, traceback) 2022-05-14T01:17:01.557 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:17:01 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T01:17:01.557 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:17:01 smithi047 conmon[78521]: raise OrchestratorError(msg) 2022-05-14T01:17:01.558 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:17:01 smithi047 conmon[78521]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:17:01.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:17:01 smithi115 conmon[90393]: 2022-05-14T01:17:01.417+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:17:01.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:17:01 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:17:02.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:17:02 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:17:02.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:17:02 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:17:02.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T01:17:02.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:17:02.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T01:17:02.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:17:02.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:17:02.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:17:02.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T01:17:02.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:17:02.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T01:17:02.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:17:02.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T01:17:02.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:17:02.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T01:17:02.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:17:02.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T01:17:02.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:17:02.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:17:02.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:17:02.548 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:17:02 smithi047 ceph-mon[77147]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T01:17:02.548 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T01:17:02.548 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:17:02.548 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T01:17:02.548 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:17:02.549 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:17:02.549 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:17:02.549 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T01:17:02.549 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:17:02.550 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T01:17:02.550 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:17:02.550 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T01:17:02.550 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:17:02.551 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T01:17:02.551 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:17:02.551 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T01:17:02.552 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:17:02.552 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:17:02.552 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:17:02.552 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T01:17:02.553 INFO:journalctl@ceph.mon.b.smithi047.stdout: During handling of the above exception, another exception occurred: 2022-05-14T01:17:02.553 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T01:17:02.553 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T01:17:02.553 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T01:17:02.554 INFO:journalctl@ceph.mon.b.smithi047.stdout: return f(*arg) 2022-05-14T01:17:02.554 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T01:17:02.554 INFO:journalctl@ceph.mon.b.smithi047.stdout: self._write_client_files(client_files, host) 2022-05-14T01:17:02.554 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T01:17:02.554 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T01:17:02.555 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T01:17:02.555 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T01:17:02.555 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T01:17:02.555 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.event_loop.get_result(coro) 2022-05-14T01:17:02.556 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T01:17:02.556 INFO:journalctl@ceph.mon.b.smithi047.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T01:17:02.556 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T01:17:02.556 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.__get_result() 2022-05-14T01:17:02.557 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T01:17:02.557 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise self._exception 2022-05-14T01:17:02.558 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T01:17:02.558 INFO:journalctl@ceph.mon.b.smithi047.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T01:17:02.559 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T01:17:02.559 INFO:journalctl@ceph.mon.b.smithi047.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T01:17:02.559 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T01:17:02.559 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise 2022-05-14T01:17:02.560 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T01:17:02.560 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.gen.throw(type, value, traceback) 2022-05-14T01:17:02.560 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T01:17:02.561 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OrchestratorError(msg) 2022-05-14T01:17:02.561 INFO:journalctl@ceph.mon.b.smithi047.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:17:02.561 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:17:02 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:17:02.561 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:17:02 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:17:02.562 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:17:02 smithi047 ceph-mon[77147]: pgmap v3630: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:17:02.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:17:02 smithi115 conmon[90393]: 2022-05-14T01:17:02.394+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:17:02.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:17:02 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:17:02.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:17:02 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:17:02.608 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T01:17:02.609 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:17:02.609 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T01:17:02.609 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:17:02.609 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:17:02.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:17:02.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T01:17:02.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:17:02.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T01:17:02.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:17:02.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T01:17:02.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:17:02.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T01:17:02.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:17:02.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T01:17:02.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:17:02.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:17:02.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:17:02.613 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:17:02 smithi115 ceph-mon[78593]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T01:17:02.613 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T01:17:02.613 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:17:02.613 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T01:17:02.613 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:17:02.614 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:17:02.614 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:17:02.614 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T01:17:02.614 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:17:02.615 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T01:17:02.615 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:17:02.615 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T01:17:02.615 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:17:02.615 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T01:17:02.616 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:17:02.616 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T01:17:02.616 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:17:02.616 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:17:02.616 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:17:02.617 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T01:17:02.617 INFO:journalctl@ceph.mon.c.smithi115.stdout: During handling of the above exception, another exception occurred: 2022-05-14T01:17:02.617 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T01:17:02.617 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T01:17:02.618 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T01:17:02.618 INFO:journalctl@ceph.mon.c.smithi115.stdout: return f(*arg) 2022-05-14T01:17:02.618 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T01:17:02.618 INFO:journalctl@ceph.mon.c.smithi115.stdout: self._write_client_files(client_files, host) 2022-05-14T01:17:02.618 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T01:17:02.619 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T01:17:02.619 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T01:17:02.619 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T01:17:02.619 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T01:17:02.619 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.event_loop.get_result(coro) 2022-05-14T01:17:02.620 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T01:17:02.620 INFO:journalctl@ceph.mon.c.smithi115.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T01:17:02.620 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T01:17:02.620 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.__get_result() 2022-05-14T01:17:02.621 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T01:17:02.621 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise self._exception 2022-05-14T01:17:02.621 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T01:17:02.621 INFO:journalctl@ceph.mon.c.smithi115.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T01:17:02.621 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T01:17:02.622 INFO:journalctl@ceph.mon.c.smithi115.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T01:17:02.622 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T01:17:02.622 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise 2022-05-14T01:17:02.622 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T01:17:02.623 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.gen.throw(type, value, traceback) 2022-05-14T01:17:02.623 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T01:17:02.623 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OrchestratorError(msg) 2022-05-14T01:17:02.623 INFO:journalctl@ceph.mon.c.smithi115.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:17:02.624 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:17:02 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:17:02.624 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:17:02 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:17:02.624 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:17:02 smithi115 ceph-mon[78593]: pgmap v3630: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:17:03.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:17:03 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:17:03.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:17:03 smithi115 conmon[90393]: 2022-05-14T01:17:03.391+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:17:03.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:17:03 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:17:04.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:17:04 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:17:04.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:17:04 smithi047 ceph-mon[77147]: pgmap v3631: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:17:04.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:17:04 smithi115 conmon[90393]: 2022-05-14T01:17:04.426+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:17:04.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:17:04 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:17:04.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:17:04 smithi115 ceph-mon[78593]: pgmap v3631: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:17:05.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:17:05 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:17:05.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:17:05 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 6555 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:17:05.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:17:05 smithi115 conmon[90393]: 2022-05-14T01:17:05.414+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:17:05.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:17:05 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:17:05.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:17:05 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 6555 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:17:06.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:17:06 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:17:06.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:17:06 smithi047 ceph-mon[77147]: pgmap v3632: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:17:06.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:17:06 smithi115 conmon[90393]: 2022-05-14T01:17:06.404+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:17:06.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:17:06 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:17:06.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:17:06 smithi115 ceph-mon[78593]: pgmap v3632: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:17:07.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:17:07 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:17:07.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:17:07 smithi115 conmon[90393]: 2022-05-14T01:17:07.449+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:17:07.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:17:07 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:17:08.466 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:17:08 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:17:08.466 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:17:08 smithi115 ceph-mon[78593]: pgmap v3633: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:17:08.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:17:08 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:17:08.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:17:08 smithi047 ceph-mon[77147]: pgmap v3633: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:17:08.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:17:08 smithi115 conmon[90393]: 2022-05-14T01:17:08.465+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:17:09.487 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:17:09 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:17:09.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:17:09 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:17:09.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:17:09 smithi115 conmon[90393]: 2022-05-14T01:17:09.486+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:17:10.478 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:17:10 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:17:10.478 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:17:10 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:17:10.479 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:17:10 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:17:10.479 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:17:10 smithi115 ceph-mon[78593]: pgmap v3634: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:17:10.480 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:17:10 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 6560 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:17:10.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:17:10 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:17:10.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:17:10 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:17:10.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:17:10 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:17:10.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:17:10 smithi047 ceph-mon[77147]: pgmap v3634: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:17:10.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:17:10 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 6560 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:17:10.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:17:10 smithi115 conmon[90393]: 2022-05-14T01:17:10.477+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:17:11.519 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:17:11 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:17:11.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:17:11 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:17:11.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:17:11 smithi115 conmon[90393]: 2022-05-14T01:17:11.518+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:17:12.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:17:12 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:17:12.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:17:12 smithi047 ceph-mon[77147]: pgmap v3635: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:17:12.549 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:17:12 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:17:12.549 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:17:12 smithi115 ceph-mon[78593]: pgmap v3635: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:17:12.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:17:12 smithi115 conmon[90393]: 2022-05-14T01:17:12.548+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:17:13.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:17:13 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:17:13.570 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:17:13 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:17:13.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:17:13 smithi115 conmon[90393]: 2022-05-14T01:17:13.569+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:17:14.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:17:14 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:17:14.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:17:14 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:17:14.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:17:14 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:17:14.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:17:14 smithi047 ceph-mon[77147]: pgmap v3636: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:17:14.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:17:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T01:17:14.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:17:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T01:17:14.557 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:17:14 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:17:14.557 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:17:14 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:17:14.557 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:17:14 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:17:14.558 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:17:14 smithi115 ceph-mon[78593]: pgmap v3636: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:17:14.558 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:17:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T01:17:14.558 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:17:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T01:17:14.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:17:14 smithi115 conmon[90393]: 2022-05-14T01:17:14.556+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:17:15.534 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:17:15 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:17:15.534 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:17:15 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 6565 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:17:15.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:17:15 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:17:15.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:17:15 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 6565 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:17:15.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:17:15 smithi115 conmon[90393]: 2022-05-14T01:17:15.532+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:17:16.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:17:16 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:17:16.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:17:16 smithi047 ceph-mon[77147]: pgmap v3637: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:17:16.566 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:17:16 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:17:16.566 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:17:16 smithi115 ceph-mon[78593]: pgmap v3637: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:17:16.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:17:16 smithi115 conmon[90393]: 2022-05-14T01:17:16.565+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:17:17.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:17:17 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:17:17.602 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:17:17 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:17:17.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:17:17 smithi115 conmon[90393]: 2022-05-14T01:17:17.602+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:17:18.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:17:18 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:17:18.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:17:18 smithi047 ceph-mon[77147]: pgmap v3638: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:17:18.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:17:18 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:17:18.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:17:18 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:17:18.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:17:18 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:17:18.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:17:18 smithi115 ceph-mon[78593]: pgmap v3638: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:17:18.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:17:18 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:17:18.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:17:18 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:17:18.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:17:18 smithi115 conmon[90393]: 2022-05-14T01:17:18.606+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:17:19.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:17:19 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:17:19.600 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:17:19 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:17:19.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:17:19 smithi115 conmon[90393]: 2022-05-14T01:17:19.599+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:17:20.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:17:20 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:17:20.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:17:20 smithi047 ceph-mon[77147]: pgmap v3639: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:17:20.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:17:20 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 6575 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:17:20.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:17:20 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:17:20.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:17:20 smithi115 ceph-mon[78593]: pgmap v3639: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:17:20.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:17:20 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 6575 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:17:21.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:17:20 smithi115 conmon[90393]: 2022-05-14T01:17:20.617+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:17:21.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:17:21 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:17:21.582 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:17:21 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:17:21.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:17:21 smithi115 conmon[90393]: 2022-05-14T01:17:21.581+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:17:22.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:17:22 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:17:22.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:17:22 smithi047 ceph-mon[77147]: pgmap v3640: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:17:22.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:17:22 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:17:22.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:17:22 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:17:22.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:17:22 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:17:22.595 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:17:22 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:17:22.595 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:17:22 smithi115 ceph-mon[78593]: pgmap v3640: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:17:22.595 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:17:22 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:17:22.596 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:17:22 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:17:22.596 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:17:22 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:17:22.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:17:22 smithi115 conmon[90393]: 2022-05-14T01:17:22.594+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:17:23.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:17:23 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:17:23.560 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:17:23 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:17:23.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:17:23 smithi115 conmon[90393]: 2022-05-14T01:17:23.559+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:17:24.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:17:24 smithi047 ceph-mon[77147]: pgmap v3641: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:17:24.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:17:24 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:17:24.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:17:24 smithi115 ceph-mon[78593]: pgmap v3641: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:17:24.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:17:24 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:17:24.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:17:24 smithi115 conmon[90393]: 2022-05-14T01:17:24.545+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:17:25.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:17:25 smithi115 conmon[90393]: 2022-05-14T01:17:25.515+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:17:25.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:17:25 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:17:25.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:17:25 smithi115 ceph-mon[78593]: pgmap v3642: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:17:26.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:17:25 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:17:26.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:17:25 smithi047 ceph-mon[77147]: pgmap v3642: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:17:26.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:17:26 smithi115 conmon[90393]: 2022-05-14T01:17:26.521+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:17:26.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:17:26 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:17:26.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:17:26 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 6580 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:17:27.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:17:26 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:17:27.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:17:26 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 6580 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:17:27.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:17:27 smithi115 conmon[90393]: 2022-05-14T01:17:27.492+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:17:27.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:17:27 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:17:27.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:17:27 smithi115 ceph-mon[78593]: pgmap v3643: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:17:28.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:17:27 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:17:28.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:17:27 smithi047 ceph-mon[77147]: pgmap v3643: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:17:28.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:17:28 smithi115 conmon[90393]: 2022-05-14T01:17:28.504+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:17:28.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:17:28 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:17:29.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:17:28 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:17:29.860 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:17:29 smithi115 conmon[90393]: 2022-05-14T01:17:29.502+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:17:29.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:17:29 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:17:29.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:17:29 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:17:29.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:17:29 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:17:29.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:17:29 smithi115 ceph-mon[78593]: pgmap v3644: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:17:30.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:17:29 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:17:30.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:17:29 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:17:30.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:17:29 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:17:30.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:17:29 smithi047 ceph-mon[77147]: pgmap v3644: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:17:30.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:17:30 smithi115 conmon[90393]: 2022-05-14T01:17:30.534+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:17:30.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:17:30 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:17:31.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:17:30 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:17:31.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:17:31 smithi115 conmon[90393]: 2022-05-14T01:17:31.522+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:17:31.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:17:31 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:17:31.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:17:31 smithi115 ceph-mon[78593]: pgmap v3645: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:17:32.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:17:31 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:17:32.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:17:31 smithi047 ceph-mon[77147]: pgmap v3645: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:17:32.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:17:32 smithi115 conmon[90393]: 2022-05-14T01:17:32.519+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:17:32.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:17:32 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:17:33.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:17:32 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:17:33.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:17:33 smithi115 conmon[90393]: 2022-05-14T01:17:33.555+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:17:33.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:17:33 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:17:33.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:17:33 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:17:33.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:17:33 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:17:33.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:17:33 smithi115 ceph-mon[78593]: pgmap v3646: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:17:34.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:17:33 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:17:34.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:17:33 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:17:34.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:17:33 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:17:34.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:17:33 smithi047 ceph-mon[77147]: pgmap v3646: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:17:34.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:17:34 smithi115 conmon[90393]: 2022-05-14T01:17:34.535+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:17:34.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:17:34 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:17:34.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:17:34 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 6585 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:17:35.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:17:34 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:17:35.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:17:34 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 6585 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:17:35.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:17:35 smithi115 conmon[90393]: 2022-05-14T01:17:35.529+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:17:35.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:17:35 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:17:35.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:17:35 smithi115 ceph-mon[78593]: pgmap v3647: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:17:36.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:17:35 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:17:36.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:17:35 smithi047 ceph-mon[77147]: pgmap v3647: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:17:36.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:17:36 smithi115 conmon[90393]: 2022-05-14T01:17:36.562+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:17:36.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:17:36 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:17:37.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:17:36 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:17:37.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:17:37 smithi115 conmon[90393]: 2022-05-14T01:17:37.586+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:17:37.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:17:37 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:17:37.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:17:37 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:17:37.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:17:37 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:17:37.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:17:37 smithi115 ceph-mon[78593]: pgmap v3648: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:17:38.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:17:37 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:17:38.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:17:37 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:17:38.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:17:37 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:17:38.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:17:37 smithi047 ceph-mon[77147]: pgmap v3648: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:17:39.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:17:38 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:17:39.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:17:38 smithi115 conmon[90393]: 2022-05-14T01:17:38.624+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:17:39.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:17:38 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:17:39.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:17:39 smithi115 conmon[90393]: 2022-05-14T01:17:39.575+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:17:39.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:17:39 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:17:39.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:17:39 smithi115 ceph-mon[78593]: pgmap v3649: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:17:39.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:17:39 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 6590 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:17:40.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:17:39 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:17:40.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:17:39 smithi047 ceph-mon[77147]: pgmap v3649: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:17:40.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:17:39 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 6590 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:17:40.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:17:40 smithi115 conmon[90393]: 2022-05-14T01:17:40.579+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:17:40.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:17:40 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:17:41.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:17:40 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:17:41.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:17:41 smithi115 conmon[90393]: 2022-05-14T01:17:41.536+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:17:41.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:17:41 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:17:41.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:17:41 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:17:41.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:17:41 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:17:41.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:17:41 smithi115 ceph-mon[78593]: pgmap v3650: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:17:42.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:17:41 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:17:42.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:17:41 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:17:42.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:17:41 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:17:42.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:17:41 smithi047 ceph-mon[77147]: pgmap v3650: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:17:42.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:17:42 smithi115 conmon[90393]: 2022-05-14T01:17:42.548+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:17:42.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:17:42 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:17:43.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:17:42 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:17:43.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:17:43 smithi115 conmon[90393]: 2022-05-14T01:17:43.583+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:17:43.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:17:43 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:17:43.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:17:43 smithi115 ceph-mon[78593]: pgmap v3651: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:17:44.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:17:43 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:17:44.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:17:43 smithi047 ceph-mon[77147]: pgmap v3651: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:17:44.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:17:44 smithi115 conmon[90393]: 2022-05-14T01:17:44.563+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:17:44.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:17:44 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:17:44.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:17:44 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 6595 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:17:45.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:17:44 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:17:45.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:17:44 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 6595 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:17:45.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:17:45 smithi115 conmon[90393]: 2022-05-14T01:17:45.514+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:17:45.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:17:45 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:17:45.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:17:45 smithi115 ceph-mon[78593]: pgmap v3652: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:17:46.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:17:45 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:17:46.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:17:45 smithi047 ceph-mon[77147]: pgmap v3652: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:17:46.732 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:17:46 smithi115 conmon[90393]: 2022-05-14T01:17:46.473+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:17:47.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:17:46 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:17:47.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:17:46 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:17:47.738 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:17:47 smithi115 conmon[90393]: 2022-05-14T01:17:47.444+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:17:48.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:17:47 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:17:48.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:17:47 smithi047 ceph-mon[77147]: pgmap v3653: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:17:48.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:17:47 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:17:48.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:17:47 smithi115 ceph-mon[78593]: pgmap v3653: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:17:48.744 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:17:48 smithi115 conmon[90393]: 2022-05-14T01:17:48.469+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:17:49.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:17:48 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:17:49.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:17:48 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:17:49.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:17:49 smithi115 conmon[90393]: 2022-05-14T01:17:49.497+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:17:49.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:17:49 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:17:49.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:17:49 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:17:49.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:17:49 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:17:49.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:17:49 smithi115 ceph-mon[78593]: pgmap v3654: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:17:49.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:17:49 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 6600 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:17:50.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:17:49 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:17:50.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:17:49 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:17:50.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:17:49 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:17:50.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:17:49 smithi047 ceph-mon[77147]: pgmap v3654: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:17:50.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:17:49 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 6600 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:17:50.748 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:17:50 smithi115 conmon[90393]: 2022-05-14T01:17:50.494+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:17:51.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:17:50 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:17:51.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:17:50 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:17:51.753 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:17:51 smithi115 conmon[90393]: 2022-05-14T01:17:51.456+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:17:52.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:17:51 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:17:52.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:17:51 smithi047 ceph-mon[77147]: pgmap v3655: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:17:52.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:17:51 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:17:52.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:17:51 smithi115 ceph-mon[78593]: pgmap v3655: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:17:52.759 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:17:52 smithi115 conmon[90393]: 2022-05-14T01:17:52.459+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:17:53.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:17:52 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:17:53.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:17:52 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:17:53.764 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:17:53 smithi115 conmon[90393]: 2022-05-14T01:17:53.440+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:17:54.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:17:53 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:17:54.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:17:53 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:17:54.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:17:53 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:17:54.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:17:53 smithi047 ceph-mon[77147]: pgmap v3656: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:17:54.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:17:53 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:17:54.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:17:53 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:17:54.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:17:53 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:17:54.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:17:53 smithi115 ceph-mon[78593]: pgmap v3656: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:17:54.766 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:17:54 smithi115 conmon[90393]: 2022-05-14T01:17:54.465+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:17:55.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:17:54 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:17:55.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:17:54 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 6605 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:17:55.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:17:54 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:17:55.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:17:54 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 6605 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:17:55.771 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:17:55 smithi115 conmon[90393]: 2022-05-14T01:17:55.450+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:17:56.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:17:55 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:17:56.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:17:55 smithi047 ceph-mon[77147]: pgmap v3657: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:17:56.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:17:55 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:17:56.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:17:55 smithi115 ceph-mon[78593]: pgmap v3657: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:17:56.777 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:17:56 smithi115 conmon[90393]: 2022-05-14T01:17:56.439+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:17:57.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:17:56 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:17:57.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:17:56 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:17:57.781 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:17:57 smithi115 conmon[90393]: 2022-05-14T01:17:57.480+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:17:58.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:17:57 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:17:58.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:17:57 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:17:58.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:17:57 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:17:58.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:17:57 smithi047 ceph-mon[77147]: pgmap v3658: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:17:58.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:17:57 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:17:58.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:17:57 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:17:58.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:17:57 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:17:58.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:17:57 smithi115 ceph-mon[78593]: pgmap v3658: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:17:58.783 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:17:58 smithi115 conmon[90393]: 2022-05-14T01:17:58.509+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:17:59.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:17:58 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:17:59.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:17:58 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:17:59.788 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:17:59 smithi115 conmon[90393]: 2022-05-14T01:17:59.467+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:18:00.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:17:59 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:18:00.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:17:59 smithi047 ceph-mon[77147]: pgmap v3659: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:18:00.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:17:59 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 6610 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:18:00.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:17:59 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:18:00.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:17:59 smithi115 ceph-mon[78593]: pgmap v3659: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:18:00.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:17:59 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 6610 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:18:00.793 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:18:00 smithi115 conmon[90393]: 2022-05-14T01:18:00.473+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:18:01.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:18:00 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:18:01.217 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:18:00 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:18:01.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:18:01 smithi115 conmon[90393]: 2022-05-14T01:18:01.427+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:18:02.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:18:01 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:18:02.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:18:01 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:18:02.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:18:01 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:18:02.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:18:01 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T01:18:02.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:18:01 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T01:18:02.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:18:01 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T01:18:02.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:18:01 smithi047 ceph-mon[77147]: pgmap v3660: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:18:02.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:18:01 smithi047 conmon[78521]: 2022-05-14T01:18:01.776+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:18:02.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:18:01 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T01:18:02.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:18:01 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:18:02.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:18:01 smithi047 conmon[78521]: yield 2022-05-14T01:18:02.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:18:01 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:18:02.048 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:18:01 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:18:02.048 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:18:01 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:18:02.049 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:18:01 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T01:18:02.049 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:18:01 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:18:02.050 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:18:01 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T01:18:02.052 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:18:01 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:18:02.052 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:18:01 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T01:18:02.053 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:18:01 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:18:02.053 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:18:01 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T01:18:02.053 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:18:01 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:18:02.053 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:18:01 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T01:18:02.054 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:18:01 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:18:02.054 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:18:01 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:18:02.054 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:18:01 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:18:02.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:18:01 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:18:02.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:18:01 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:18:02.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:18:01 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:18:02.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:18:01 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T01:18:02.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:18:01 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T01:18:02.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:18:01 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T01:18:02.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:18:01 smithi115 ceph-mon[78593]: pgmap v3660: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:18:02.799 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:18:02 smithi115 conmon[90393]: 2022-05-14T01:18:02.457+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:18:03.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:18:02 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:18:03.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:18:02 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:18:03.108 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T01:18:03.108 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:18:03.108 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T01:18:03.109 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:18:03.109 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:18:03.109 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:18:03.109 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T01:18:03.109 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:18:03.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T01:18:03.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:18:03.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T01:18:03.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:18:03.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T01:18:03.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:18:03.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T01:18:03.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:18:03.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:18:03.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:18:03.112 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:18:02 smithi115 ceph-mon[78593]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T01:18:03.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:18:02 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:18:03.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:18:02 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:18:03.291 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T01:18:03.292 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:18:03.292 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T01:18:03.292 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:18:03.292 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:18:03.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:18:03.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T01:18:03.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:18:03.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T01:18:03.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:18:03.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T01:18:03.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:18:03.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T01:18:03.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:18:03.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T01:18:03.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:18:03.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:18:03.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:18:03.296 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:18:02 smithi047 ceph-mon[77147]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T01:18:03.804 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:18:03 smithi115 conmon[90393]: 2022-05-14T01:18:03.428+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:18:04.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:18:03 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:18:04.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:18:03 smithi115 ceph-mon[78593]: pgmap v3661: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:18:04.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:18:03 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:18:04.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:18:03 smithi047 ceph-mon[77147]: pgmap v3661: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:18:04.805 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:18:04 smithi115 conmon[90393]: 2022-05-14T01:18:04.408+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:18:05.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:18:04 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:18:05.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:18:04 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 6615 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:18:05.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:18:04 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:18:05.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:18:04 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 6615 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:18:05.292 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:18:04 smithi047 conmon[78521]: 2022-05-14T01:18:04.848+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:18:05.292 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:18:04 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T01:18:05.292 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:18:04 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:18:05.292 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:18:04 smithi047 conmon[78521]: yield 2022-05-14T01:18:05.293 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:18:04 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:18:05.293 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:18:04 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:18:05.293 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:18:04 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:18:05.293 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:18:04 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T01:18:05.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:18:04 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:18:05.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:18:04 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T01:18:05.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:18:04 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:18:05.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:18:04 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T01:18:05.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:18:04 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:18:05.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:18:04 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T01:18:05.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:18:04 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:18:05.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:18:04 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T01:18:05.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:18:04 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:18:05.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:18:04 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:18:05.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:18:04 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:18:05.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:18:04 smithi047 conmon[78521]: 2022-05-14T01:18:04.849+0000 7f5345f84700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T01:18:05.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:18:04 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T01:18:05.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:18:04 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:18:05.297 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:18:04 smithi047 conmon[78521]: yield 2022-05-14T01:18:05.297 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:18:04 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:18:05.297 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:18:04 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:18:05.297 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:18:04 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:18:05.298 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:18:04 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T01:18:05.298 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:18:04 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:18:05.298 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:18:04 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T01:18:05.298 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:18:04 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:18:05.298 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:18:04 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T01:18:05.299 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:18:04 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:18:05.299 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:18:04 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T01:18:05.299 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:18:04 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:18:05.299 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:18:04 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T01:18:05.299 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:18:04 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:18:05.300 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:18:04 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:18:05.300 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:18:04 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:18:05.300 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:18:04 smithi047 conmon[78521]: 2022-05-14T01:18:05.300 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:18:04 smithi047 conmon[78521]: During handling of the above exception, another exception occurred: 2022-05-14T01:18:05.301 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:18:04 smithi047 conmon[78521]: 2022-05-14T01:18:05.301 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:18:04 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T01:18:05.301 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:18:04 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T01:18:05.301 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:18:04 smithi047 conmon[78521]: return f(*arg) 2022-05-14T01:18:05.301 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:18:04 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T01:18:05.302 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:18:04 smithi047 conmon[78521]: self._write_client_files(client_files, host) 2022-05-14T01:18:05.302 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:18:04 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T01:18:05.302 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:18:04 smithi047 conmon[78521]: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T01:18:05.302 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:18:04 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T01:18:05.302 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:18:04 smithi047 conmon[78521]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T01:18:05.303 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:18:04 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T01:18:05.303 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:18:04 smithi047 conmon[78521]: return self.event_loop.get_result(coro) 2022-05-14T01:18:05.303 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:18:04 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T01:18:05.303 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:18:04 smithi047 conmon[78521]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T01:18:05.303 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:18:04 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T01:18:05.304 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:18:04 smithi047 conmon[78521]: return self.__get_result() 2022-05-14T01:18:05.304 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:18:04 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T01:18:05.304 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:18:04 smithi047 conmon[78521]: raise self._exception 2022-05-14T01:18:05.304 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:18:04 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T01:18:05.305 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:18:04 smithi047 conmon[78521]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T01:18:05.305 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:18:04 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T01:18:05.305 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:18:04 smithi047 conmon[78521]: conn = await self._remote_connection(host, addr) 2022-05-14T01:18:05.305 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:18:04 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T01:18:05.305 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:18:04 smithi047 conmon[78521]: raise 2022-05-14T01:18:05.306 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:18:04 smithi047 conmon[78521]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T01:18:05.306 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:18:04 smithi047 conmon[78521]: self.gen.throw(type, value, traceback) 2022-05-14T01:18:05.306 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:18:04 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T01:18:05.306 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:18:04 smithi047 conmon[78521]: raise OrchestratorError(msg) 2022-05-14T01:18:05.307 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:18:04 smithi047 conmon[78521]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:18:05.813 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:18:05 smithi115 conmon[90393]: 2022-05-14T01:18:05.393+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:18:06.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:18:05 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:18:06.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:18:05 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:18:06.108 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T01:18:06.108 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:18:06.108 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T01:18:06.109 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:18:06.109 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:18:06.109 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:18:06.109 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T01:18:06.109 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:18:06.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T01:18:06.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:18:06.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T01:18:06.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:18:06.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T01:18:06.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:18:06.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T01:18:06.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:18:06.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:18:06.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:18:06.112 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:18:05 smithi115 ceph-mon[78593]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T01:18:06.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T01:18:06.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:18:06.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T01:18:06.113 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:18:06.113 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:18:06.113 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:18:06.113 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T01:18:06.114 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:18:06.114 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T01:18:06.114 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:18:06.114 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T01:18:06.114 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:18:06.115 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T01:18:06.115 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:18:06.115 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T01:18:06.115 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:18:06.115 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:18:06.116 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:18:06.116 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T01:18:06.116 INFO:journalctl@ceph.mon.c.smithi115.stdout: During handling of the above exception, another exception occurred: 2022-05-14T01:18:06.116 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T01:18:06.116 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T01:18:06.117 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T01:18:06.117 INFO:journalctl@ceph.mon.c.smithi115.stdout: return f(*arg) 2022-05-14T01:18:06.117 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T01:18:06.117 INFO:journalctl@ceph.mon.c.smithi115.stdout: self._write_client_files(client_files, host) 2022-05-14T01:18:06.117 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T01:18:06.118 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T01:18:06.118 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T01:18:06.118 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T01:18:06.118 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T01:18:06.119 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.event_loop.get_result(coro) 2022-05-14T01:18:06.119 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T01:18:06.119 INFO:journalctl@ceph.mon.c.smithi115.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T01:18:06.119 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T01:18:06.119 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.__get_result() 2022-05-14T01:18:06.120 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T01:18:06.120 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise self._exception 2022-05-14T01:18:06.120 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T01:18:06.120 INFO:journalctl@ceph.mon.c.smithi115.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T01:18:06.120 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T01:18:06.121 INFO:journalctl@ceph.mon.c.smithi115.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T01:18:06.121 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T01:18:06.121 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise 2022-05-14T01:18:06.121 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T01:18:06.121 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.gen.throw(type, value, traceback) 2022-05-14T01:18:06.122 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T01:18:06.122 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OrchestratorError(msg) 2022-05-14T01:18:06.122 INFO:journalctl@ceph.mon.c.smithi115.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:18:06.122 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:18:05 smithi115 ceph-mon[78593]: pgmap v3662: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:18:06.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:18:05 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:18:06.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:18:05 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:18:06.292 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T01:18:06.292 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:18:06.292 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T01:18:06.292 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:18:06.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:18:06.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:18:06.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T01:18:06.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:18:06.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T01:18:06.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:18:06.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T01:18:06.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:18:06.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T01:18:06.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:18:06.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T01:18:06.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:18:06.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:18:06.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:18:06.296 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:18:05 smithi047 ceph-mon[77147]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T01:18:06.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T01:18:06.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:18:06.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T01:18:06.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:18:06.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:18:06.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:18:06.298 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T01:18:06.298 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:18:06.298 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T01:18:06.298 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:18:06.298 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T01:18:06.299 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:18:06.299 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T01:18:06.299 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:18:06.299 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T01:18:06.300 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:18:06.300 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:18:06.300 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:18:06.300 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T01:18:06.300 INFO:journalctl@ceph.mon.b.smithi047.stdout: During handling of the above exception, another exception occurred: 2022-05-14T01:18:06.301 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T01:18:06.301 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T01:18:06.301 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T01:18:06.301 INFO:journalctl@ceph.mon.b.smithi047.stdout: return f(*arg) 2022-05-14T01:18:06.302 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T01:18:06.302 INFO:journalctl@ceph.mon.b.smithi047.stdout: self._write_client_files(client_files, host) 2022-05-14T01:18:06.302 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T01:18:06.302 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T01:18:06.303 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T01:18:06.303 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T01:18:06.304 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T01:18:06.304 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.event_loop.get_result(coro) 2022-05-14T01:18:06.304 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T01:18:06.304 INFO:journalctl@ceph.mon.b.smithi047.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T01:18:06.305 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T01:18:06.305 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.__get_result() 2022-05-14T01:18:06.305 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T01:18:06.305 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise self._exception 2022-05-14T01:18:06.306 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T01:18:06.306 INFO:journalctl@ceph.mon.b.smithi047.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T01:18:06.306 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T01:18:06.306 INFO:journalctl@ceph.mon.b.smithi047.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T01:18:06.306 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T01:18:06.307 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise 2022-05-14T01:18:06.307 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T01:18:06.307 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.gen.throw(type, value, traceback) 2022-05-14T01:18:06.307 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T01:18:06.307 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OrchestratorError(msg) 2022-05-14T01:18:06.308 INFO:journalctl@ceph.mon.b.smithi047.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:18:06.308 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:18:05 smithi047 ceph-mon[77147]: pgmap v3662: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:18:06.813 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:18:06 smithi115 conmon[90393]: 2022-05-14T01:18:06.371+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:18:07.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:18:06 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:18:07.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:18:06 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:18:07.817 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:18:07 smithi115 conmon[90393]: 2022-05-14T01:18:07.410+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:18:08.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:18:07 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:18:08.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:18:07 smithi115 ceph-mon[78593]: pgmap v3663: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:18:08.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:18:07 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:18:08.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:18:07 smithi047 ceph-mon[77147]: pgmap v3663: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:18:08.819 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:18:08 smithi115 conmon[90393]: 2022-05-14T01:18:08.452+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:18:09.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:18:08 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:18:09.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:18:08 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:18:09.827 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:18:09 smithi115 conmon[90393]: 2022-05-14T01:18:09.493+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:18:10.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:18:09 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:18:10.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:18:09 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:18:10.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:18:09 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:18:10.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:18:09 smithi115 ceph-mon[78593]: pgmap v3664: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:18:10.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:18:09 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 6620 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:18:10.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:18:09 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:18:10.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:18:09 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:18:10.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:18:09 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:18:10.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:18:09 smithi047 ceph-mon[77147]: pgmap v3664: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:18:10.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:18:09 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 6620 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:18:10.829 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:18:10 smithi115 conmon[90393]: 2022-05-14T01:18:10.470+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:18:11.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:18:10 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:18:11.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:18:10 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:18:11.833 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:18:11 smithi115 conmon[90393]: 2022-05-14T01:18:11.442+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:18:12.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:18:11 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:18:12.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:18:11 smithi115 ceph-mon[78593]: pgmap v3665: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:18:12.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:18:11 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:18:12.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:18:11 smithi047 ceph-mon[77147]: pgmap v3665: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:18:12.835 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:18:12 smithi115 conmon[90393]: 2022-05-14T01:18:12.409+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:18:13.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:18:12 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:18:13.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:18:12 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:18:13.847 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:18:13 smithi115 conmon[90393]: 2022-05-14T01:18:13.406+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:18:14.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:18:13 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:18:14.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:18:13 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:18:14.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:18:13 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:18:14.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:18:13 smithi115 ceph-mon[78593]: pgmap v3666: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:18:14.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:18:13 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T01:18:14.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:18:13 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T01:18:14.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:18:13 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:18:14.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:18:13 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:18:14.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:18:13 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:18:14.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:18:13 smithi047 ceph-mon[77147]: pgmap v3666: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:18:14.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:18:13 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T01:18:14.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:18:13 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T01:18:14.848 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:18:14 smithi115 conmon[90393]: 2022-05-14T01:18:14.382+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:18:15.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:18:14 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:18:15.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:18:14 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 6625 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:18:15.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:18:14 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:18:15.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:18:14 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 6625 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:18:15.853 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:18:15 smithi115 conmon[90393]: 2022-05-14T01:18:15.372+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:18:16.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:18:15 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:18:16.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:18:15 smithi115 ceph-mon[78593]: pgmap v3667: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:18:16.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:18:15 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:18:16.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:18:15 smithi047 ceph-mon[77147]: pgmap v3667: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:18:16.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:18:16 smithi115 conmon[90393]: 2022-05-14T01:18:16.325+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:18:17.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:18:16 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:18:17.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:18:16 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:18:17.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:18:17 smithi115 conmon[90393]: 2022-05-14T01:18:17.304+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:18:18.280 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:18:17 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:18:18.280 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:18:17 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:18:18.280 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:18:17 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:18:18.281 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:18:17 smithi115 ceph-mon[78593]: pgmap v3668: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:18:18.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:18:17 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:18:18.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:18:17 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:18:18.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:18:17 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:18:18.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:18:17 smithi047 ceph-mon[77147]: pgmap v3668: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:18:18.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:18:18 smithi115 conmon[90393]: 2022-05-14T01:18:18.279+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:18:19.252 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:18:18 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:18:19.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:18:18 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:18:19.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:18:19 smithi115 conmon[90393]: 2022-05-14T01:18:19.251+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:18:20.289 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:18:19 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:18:20.289 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:18:19 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:18:20.289 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:18:19 smithi115 ceph-mon[78593]: pgmap v3669: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:18:20.290 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:18:19 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 6630 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:18:20.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:18:19 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:18:20.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:18:19 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:18:20.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:18:19 smithi047 ceph-mon[77147]: pgmap v3669: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:18:20.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:18:19 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 6630 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:18:20.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:18:20 smithi115 conmon[90393]: 2022-05-14T01:18:20.288+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:18:21.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:18:21 smithi115 conmon[90393]: 2022-05-14T01:18:21.280+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:18:22.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:18:21 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:18:22.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:18:21 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:18:22.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:18:21 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:18:22.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:18:21 smithi047 ceph-mon[77147]: pgmap v3670: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:18:22.309 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:18:21 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:18:22.310 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:18:21 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:18:22.310 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:18:21 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:18:22.310 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:18:21 smithi115 ceph-mon[78593]: pgmap v3670: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:18:22.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:18:22 smithi115 conmon[90393]: 2022-05-14T01:18:22.308+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:18:23.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:18:22 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:18:23.328 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:18:22 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:18:23.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:18:23 smithi115 conmon[90393]: 2022-05-14T01:18:23.328+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:18:24.282 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:18:23 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:18:24.282 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:18:23 smithi115 ceph-mon[78593]: pgmap v3671: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:18:24.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:18:23 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:18:24.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:18:23 smithi047 ceph-mon[77147]: pgmap v3671: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:18:24.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:18:24 smithi115 conmon[90393]: 2022-05-14T01:18:24.281+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:18:25.257 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:18:24 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:18:25.257 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:18:24 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 6635 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:18:25.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:18:24 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:18:25.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:18:24 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 6635 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:18:25.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:18:25 smithi115 conmon[90393]: 2022-05-14T01:18:25.256+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:18:26.242 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:18:25 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:18:26.243 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:18:25 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:18:26.243 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:18:25 smithi115 ceph-mon[78593]: pgmap v3672: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:18:26.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:18:25 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:18:26.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:18:25 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:18:26.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:18:25 smithi047 ceph-mon[77147]: pgmap v3672: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:18:26.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:18:26 smithi115 conmon[90393]: 2022-05-14T01:18:26.242+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:18:27.265 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:18:26 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:18:27.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:18:26 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:18:27.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:18:27 smithi115 conmon[90393]: 2022-05-14T01:18:27.265+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:18:28.259 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:18:27 smithi115 ceph-mon[78593]: pgmap v3673: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:18:28.259 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:18:27 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:18:28.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:18:27 smithi047 ceph-mon[77147]: pgmap v3673: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:18:28.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:18:27 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:18:28.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:18:28 smithi115 conmon[90393]: 2022-05-14T01:18:28.259+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:18:29.235 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:18:28 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:18:29.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:18:28 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:18:29.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:18:29 smithi115 conmon[90393]: 2022-05-14T01:18:29.234+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:18:30.211 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:18:29 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:18:30.211 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:18:29 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:18:30.211 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:18:29 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:18:30.211 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:18:29 smithi115 ceph-mon[78593]: pgmap v3674: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:18:30.212 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:18:29 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 6640 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:18:30.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:18:29 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:18:30.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:18:29 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:18:30.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:18:29 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:18:30.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:18:29 smithi047 ceph-mon[77147]: pgmap v3674: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:18:30.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:18:29 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 6640 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:18:30.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:18:30 smithi115 conmon[90393]: 2022-05-14T01:18:30.210+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:18:31.200 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:18:30 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:18:31.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:18:30 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:18:31.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:18:31 smithi115 conmon[90393]: 2022-05-14T01:18:31.200+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:18:32.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:18:31 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:18:32.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:18:31 smithi047 ceph-mon[77147]: pgmap v3675: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:18:32.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:18:32 smithi115 conmon[90393]: 2022-05-14T01:18:32.151+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:18:32.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:18:31 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:18:32.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:18:31 smithi115 ceph-mon[78593]: pgmap v3675: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:18:33.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:18:32 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:18:33.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:18:33 smithi115 conmon[90393]: 2022-05-14T01:18:33.135+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:18:33.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:18:32 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:18:34.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:18:33 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:18:34.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:18:33 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:18:34.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:18:33 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:18:34.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:18:33 smithi047 ceph-mon[77147]: pgmap v3676: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:18:34.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:18:34 smithi115 conmon[90393]: 2022-05-14T01:18:34.094+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:18:34.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:18:33 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:18:34.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:18:33 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:18:34.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:18:33 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:18:34.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:18:33 smithi115 ceph-mon[78593]: pgmap v3676: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:18:35.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:18:34 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:18:35.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:18:34 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 6645 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:18:35.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:18:34 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:18:35.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:18:34 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 6645 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:18:35.468 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:18:35 smithi115 conmon[90393]: 2022-05-14T01:18:35.100+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:18:36.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:18:35 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:18:36.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:18:35 smithi047 ceph-mon[77147]: pgmap v3677: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:18:36.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:18:36 smithi115 conmon[90393]: 2022-05-14T01:18:36.123+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:18:36.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:18:35 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:18:36.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:18:35 smithi115 ceph-mon[78593]: pgmap v3677: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:18:37.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:18:36 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:18:37.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:18:37 smithi115 conmon[90393]: 2022-05-14T01:18:37.084+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:18:37.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:18:36 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:18:38.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:18:37 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:18:38.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:18:37 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:18:38.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:18:37 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:18:38.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:18:37 smithi047 ceph-mon[77147]: pgmap v3678: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:18:38.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:18:38 smithi115 conmon[90393]: 2022-05-14T01:18:38.065+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:18:38.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:18:37 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:18:38.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:18:37 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:18:38.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:18:37 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:18:38.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:18:37 smithi115 ceph-mon[78593]: pgmap v3678: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:18:39.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:18:38 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:18:39.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:18:39 smithi115 conmon[90393]: 2022-05-14T01:18:39.031+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:18:39.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:18:38 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:18:40.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:18:39 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:18:40.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:18:39 smithi047 ceph-mon[77147]: pgmap v3679: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:18:40.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:18:39 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 6650 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:18:40.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:18:40 smithi115 conmon[90393]: 2022-05-14T01:18:40.064+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:18:40.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:18:39 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:18:40.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:18:39 smithi115 ceph-mon[78593]: pgmap v3679: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:18:40.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:18:39 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 6650 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:18:41.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:18:40 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:18:41.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:18:41 smithi115 conmon[90393]: 2022-05-14T01:18:41.055+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:18:41.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:18:40 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:18:42.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:18:41 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:18:42.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:18:41 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:18:42.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:18:41 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:18:42.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:18:41 smithi047 ceph-mon[77147]: pgmap v3680: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:18:42.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:18:42 smithi115 conmon[90393]: 2022-05-14T01:18:42.056+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:18:42.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:18:41 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:18:42.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:18:41 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:18:42.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:18:41 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:18:42.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:18:41 smithi115 ceph-mon[78593]: pgmap v3680: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:18:43.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:18:42 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:18:43.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:18:43 smithi115 conmon[90393]: 2022-05-14T01:18:43.098+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:18:43.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:18:42 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:18:44.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:18:43 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:18:44.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:18:43 smithi047 ceph-mon[77147]: pgmap v3681: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:18:44.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:18:44 smithi115 conmon[90393]: 2022-05-14T01:18:44.079+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:18:44.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:18:43 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:18:44.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:18:43 smithi115 ceph-mon[78593]: pgmap v3681: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:18:45.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:18:44 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:18:45.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:18:44 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 6654 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:18:45.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:18:45 smithi115 conmon[90393]: 2022-05-14T01:18:45.098+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:18:45.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:18:44 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:18:45.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:18:44 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 6654 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:18:46.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:18:45 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:18:46.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:18:45 smithi047 ceph-mon[77147]: pgmap v3682: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:18:46.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:18:46 smithi115 conmon[90393]: 2022-05-14T01:18:46.128+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:18:46.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:18:45 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:18:46.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:18:45 smithi115 ceph-mon[78593]: pgmap v3682: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:18:47.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:18:46 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:18:47.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:18:47 smithi115 conmon[90393]: 2022-05-14T01:18:47.114+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:18:47.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:18:46 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:18:48.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:18:47 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:18:48.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:18:47 smithi047 ceph-mon[77147]: pgmap v3683: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:18:48.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:18:48 smithi115 conmon[90393]: 2022-05-14T01:18:48.104+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:18:48.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:18:47 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:18:48.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:18:47 smithi115 ceph-mon[78593]: pgmap v3683: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:18:49.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:18:48 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:18:49.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:18:49 smithi115 conmon[90393]: 2022-05-14T01:18:49.130+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:18:49.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:18:48 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:18:50.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:18:49 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:18:50.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:18:49 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:18:50.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:18:49 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:18:50.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:18:49 smithi047 ceph-mon[77147]: pgmap v3684: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:18:50.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:18:49 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 6660 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:18:50.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:18:50 smithi115 conmon[90393]: 2022-05-14T01:18:50.139+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:18:50.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:18:49 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:18:50.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:18:49 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:18:50.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:18:49 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:18:50.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:18:49 smithi115 ceph-mon[78593]: pgmap v3684: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:18:50.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:18:49 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 6660 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:18:51.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:18:50 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:18:51.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:18:51 smithi115 conmon[90393]: 2022-05-14T01:18:51.163+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:18:51.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:18:50 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:18:52.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:18:51 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:18:52.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:18:51 smithi047 ceph-mon[77147]: pgmap v3685: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:18:52.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:18:52 smithi115 conmon[90393]: 2022-05-14T01:18:52.213+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:18:52.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:18:51 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:18:52.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:18:51 smithi115 ceph-mon[78593]: pgmap v3685: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:18:53.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:18:52 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:18:53.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:18:53 smithi115 conmon[90393]: 2022-05-14T01:18:53.221+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:18:53.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:18:52 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:18:54.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:18:54 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:18:54.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:18:54 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:18:54.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:18:54 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:18:54.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:18:54 smithi047 ceph-mon[77147]: pgmap v3686: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:18:54.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:18:54 smithi115 conmon[90393]: 2022-05-14T01:18:54.248+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:18:54.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:18:54 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:18:54.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:18:54 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:18:54.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:18:54 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:18:54.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:18:54 smithi115 ceph-mon[78593]: pgmap v3686: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:18:55.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:18:55 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:18:55.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:18:55 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 6665 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:18:55.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:18:55 smithi115 conmon[90393]: 2022-05-14T01:18:55.229+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:18:55.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:18:55 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:18:55.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:18:55 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 6665 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:18:56.281 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:18:56 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:18:56.281 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:18:56 smithi115 ceph-mon[78593]: pgmap v3687: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:18:56.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:18:56 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:18:56.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:18:56 smithi047 ceph-mon[77147]: pgmap v3687: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:18:56.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:18:56 smithi115 conmon[90393]: 2022-05-14T01:18:56.280+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:18:57.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:18:57 smithi115 conmon[90393]: 2022-05-14T01:18:57.275+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:18:58.284 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:18:58 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:18:58.284 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:18:58 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:18:58.285 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:18:58 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:18:58.285 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:18:58 smithi115 ceph-mon[78593]: pgmap v3688: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:18:58.285 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:18:58 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:18:58.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:18:58 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:18:58.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:18:58 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:18:58.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:18:58 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:18:58.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:18:58 smithi047 ceph-mon[77147]: pgmap v3688: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:18:58.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:18:58 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:18:58.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:18:58 smithi115 conmon[90393]: 2022-05-14T01:18:58.283+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:18:59.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:18:59 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:18:59.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:18:59 smithi115 conmon[90393]: 2022-05-14T01:18:59.309+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:18:59.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:18:59 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:19:00.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:19:00 smithi115 conmon[90393]: 2022-05-14T01:19:00.345+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:19:00.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:19:00 smithi115 ceph-mon[78593]: pgmap v3689: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:19:00.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:19:00 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:19:00.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:19:00 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 6675 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:19:00.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:19:00 smithi047 ceph-mon[77147]: pgmap v3689: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:19:00.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:19:00 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:19:00.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:19:00 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 6675 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:19:01.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:19:01 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:19:01.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:19:01 smithi115 conmon[90393]: 2022-05-14T01:19:01.299+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:19:01.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:19:01 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:19:02.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:19:02 smithi115 conmon[90393]: 2022-05-14T01:19:02.342+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:19:02.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:19:02 smithi115 ceph-mon[78593]: pgmap v3690: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:19:02.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:19:02 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:19:02.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:19:02 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:19:02.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:19:02 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:19:02.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:19:02 smithi047 ceph-mon[77147]: pgmap v3690: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:19:02.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:19:02 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:19:02.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:19:02 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:19:02.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:19:02 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:19:03.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:19:03 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:19:03.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:19:03 smithi115 conmon[90393]: 2022-05-14T01:19:03.385+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:19:03.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:19:03 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:19:04.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:19:04 smithi115 conmon[90393]: 2022-05-14T01:19:04.387+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:19:04.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:19:04 smithi115 ceph-mon[78593]: pgmap v3691: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:19:04.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:19:04 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:19:04.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:19:04 smithi047 ceph-mon[77147]: pgmap v3691: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:19:04.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:19:04 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:19:05.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:19:05 smithi115 conmon[90393]: 2022-05-14T01:19:05.434+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:19:06.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:19:05 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:19:06.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:19:05 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T01:19:06.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:19:05 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T01:19:06.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:19:05 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T01:19:06.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:19:05 smithi047 ceph-mon[77147]: pgmap v3692: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:19:06.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:19:05 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:19:06.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:19:05 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T01:19:06.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:19:05 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T01:19:06.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:19:05 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T01:19:06.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:19:05 smithi115 ceph-mon[78593]: pgmap v3692: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:19:06.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:19:06 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:19:06.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:19:06 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 6680 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:19:06.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:19:06 smithi115 conmon[90393]: 2022-05-14T01:19:06.450+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:19:07.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:19:06 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:19:07.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:19:06 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 6680 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:19:07.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:19:07 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:19:07.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:19:07 smithi115 ceph-mon[78593]: pgmap v3693: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:19:07.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:19:07 smithi115 conmon[90393]: 2022-05-14T01:19:07.469+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:19:07.953 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:19:07 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:19:07.953 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:19:07 smithi047 ceph-mon[77147]: pgmap v3693: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:19:08.291 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:19:07 smithi047 conmon[78521]: 2022-05-14T01:19:07.951+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:19:08.291 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:19:07 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T01:19:08.291 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:19:07 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:19:08.292 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:19:07 smithi047 conmon[78521]: yield 2022-05-14T01:19:08.292 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:19:07 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:19:08.292 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:19:07 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:19:08.293 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:19:07 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:19:08.293 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:19:07 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T01:19:08.293 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:19:07 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:19:08.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:19:07 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T01:19:08.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:19:07 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:19:08.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:19:07 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T01:19:08.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:19:07 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:19:08.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:19:07 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T01:19:08.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:19:07 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:19:08.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:19:07 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T01:19:08.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:19:07 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:19:08.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:19:07 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:19:08.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:19:07 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:19:08.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:19:08 smithi115 conmon[90393]: 2022-05-14T01:19:08.461+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:19:08.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:19:08 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:19:08.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:19:08 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:19:08.858 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T01:19:08.859 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:19:08.859 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T01:19:08.859 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:19:08.859 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:19:08.860 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:19:08.860 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T01:19:08.860 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:19:08.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T01:19:08.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:19:08.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T01:19:08.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:19:08.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T01:19:08.863 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:19:08.863 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T01:19:08.864 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:19:08.864 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:19:08.864 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:19:08.865 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:19:08 smithi115 ceph-mon[78593]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T01:19:09.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:19:08 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:19:09.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:19:08 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:19:09.041 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T01:19:09.042 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:19:09.042 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T01:19:09.042 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:19:09.042 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:19:09.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:19:09.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T01:19:09.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:19:09.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T01:19:09.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:19:09.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T01:19:09.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:19:09.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T01:19:09.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:19:09.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T01:19:09.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:19:09.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:19:09.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:19:09.046 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:19:08 smithi047 ceph-mon[77147]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T01:19:09.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:19:09 smithi115 conmon[90393]: 2022-05-14T01:19:09.510+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:19:09.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:19:09 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:19:09.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:19:09 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:19:09.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:19:09 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:19:09.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:19:09 smithi115 ceph-mon[78593]: pgmap v3694: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:19:10.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:19:09 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:19:10.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:19:09 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:19:10.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:19:09 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:19:10.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:19:09 smithi047 ceph-mon[77147]: pgmap v3694: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:19:10.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:19:10 smithi115 conmon[90393]: 2022-05-14T01:19:10.462+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:19:10.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:19:10 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:19:11.025 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:19:10 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:19:11.291 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:19:11 smithi047 conmon[78521]: 2022-05-14T01:19:11.023+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:19:11.364 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:19:11 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T01:19:11.365 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:19:11 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:19:11.365 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:19:11 smithi047 conmon[78521]: yield 2022-05-14T01:19:11.365 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:19:11 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:19:11.366 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:19:11 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:19:11.366 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:19:11 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:19:11.366 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:19:11 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T01:19:11.366 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:19:11 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:19:11.367 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:19:11 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T01:19:11.367 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:19:11 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:19:11.367 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:19:11 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T01:19:11.367 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:19:11 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:19:11.367 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:19:11 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T01:19:11.368 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:19:11 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:19:11.368 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:19:11 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T01:19:11.368 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:19:11 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:19:11.368 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:19:11 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:19:11.369 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:19:11 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:19:11.369 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:19:11 smithi047 conmon[78521]: 2022-05-14T01:19:11.024+0000 7f533f777700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T01:19:11.369 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:19:11 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T01:19:11.369 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:19:11 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:19:11.370 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:19:11 smithi047 conmon[78521]: yield 2022-05-14T01:19:11.370 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:19:11 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:19:11.370 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:19:11 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:19:11.370 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:19:11 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:19:11.371 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:19:11 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T01:19:11.371 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:19:11 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:19:11.371 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:19:11 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T01:19:11.371 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:19:11 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:19:11.372 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:19:11 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T01:19:11.372 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:19:11 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:19:11.372 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:19:11 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T01:19:11.372 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:19:11 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:19:11.372 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:19:11 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T01:19:11.373 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:19:11 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:19:11.373 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:19:11 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:19:11.373 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:19:11 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:19:11.373 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:19:11 smithi047 conmon[78521]: 2022-05-14T01:19:11.374 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:19:11 smithi047 conmon[78521]: During handling of the above exception, another exception occurred: 2022-05-14T01:19:11.374 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:19:11 smithi047 conmon[78521]: 2022-05-14T01:19:11.374 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:19:11 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T01:19:11.374 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:19:11 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T01:19:11.374 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:19:11 smithi047 conmon[78521]: return f(*arg) 2022-05-14T01:19:11.375 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:19:11 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T01:19:11.375 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:19:11 smithi047 conmon[78521]: self._write_client_files(client_files, host) 2022-05-14T01:19:11.375 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:19:11 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T01:19:11.375 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:19:11 smithi047 conmon[78521]: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T01:19:11.376 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:19:11 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T01:19:11.376 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:19:11 smithi047 conmon[78521]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T01:19:11.376 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:19:11 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T01:19:11.376 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:19:11 smithi047 conmon[78521]: return self.event_loop.get_result(coro) 2022-05-14T01:19:11.376 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:19:11 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T01:19:11.377 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:19:11 smithi047 conmon[78521]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T01:19:11.377 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:19:11 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T01:19:11.377 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:19:11 smithi047 conmon[78521]: return self.__get_result() 2022-05-14T01:19:11.377 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:19:11 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T01:19:11.378 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:19:11 smithi047 conmon[78521]: raise self._exception 2022-05-14T01:19:11.378 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:19:11 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T01:19:11.378 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:19:11 smithi047 conmon[78521]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T01:19:11.378 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:19:11 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T01:19:11.378 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:19:11 smithi047 conmon[78521]: conn = await self._remote_connection(host, addr) 2022-05-14T01:19:11.379 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:19:11 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T01:19:11.379 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:19:11 smithi047 conmon[78521]: raise 2022-05-14T01:19:11.379 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:19:11 smithi047 conmon[78521]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T01:19:11.379 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:19:11 smithi047 conmon[78521]: self.gen.throw(type, value, traceback) 2022-05-14T01:19:11.380 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:19:11 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T01:19:11.380 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:19:11 smithi047 conmon[78521]: raise OrchestratorError(msg) 2022-05-14T01:19:11.380 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:19:11 smithi047 conmon[78521]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:19:11.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:19:11 smithi115 conmon[90393]: 2022-05-14T01:19:11.487+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:19:11.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:19:11 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:19:11.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:19:11 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:19:11.859 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T01:19:11.859 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:19:11.859 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T01:19:11.859 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:19:11.859 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:19:11.860 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:19:11.860 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T01:19:11.860 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:19:11.860 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T01:19:11.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:19:11.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T01:19:11.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:19:11.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T01:19:11.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:19:11.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T01:19:11.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:19:11.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:19:11.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:19:11.863 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:19:11 smithi115 ceph-mon[78593]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T01:19:11.863 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T01:19:11.863 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:19:11.863 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T01:19:11.863 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:19:11.864 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:19:11.864 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:19:11.864 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T01:19:11.864 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:19:11.865 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T01:19:11.865 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:19:11.865 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T01:19:11.865 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:19:11.865 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T01:19:11.866 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:19:11.866 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T01:19:11.866 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:19:11.866 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:19:11.867 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:19:11.867 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T01:19:11.867 INFO:journalctl@ceph.mon.c.smithi115.stdout: During handling of the above exception, another exception occurred: 2022-05-14T01:19:11.867 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T01:19:11.867 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T01:19:11.868 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T01:19:11.868 INFO:journalctl@ceph.mon.c.smithi115.stdout: return f(*arg) 2022-05-14T01:19:11.868 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T01:19:11.868 INFO:journalctl@ceph.mon.c.smithi115.stdout: self._write_client_files(client_files, host) 2022-05-14T01:19:11.869 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T01:19:11.869 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T01:19:11.869 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T01:19:11.869 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T01:19:11.869 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T01:19:11.870 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.event_loop.get_result(coro) 2022-05-14T01:19:11.870 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T01:19:11.870 INFO:journalctl@ceph.mon.c.smithi115.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T01:19:11.870 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T01:19:11.871 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.__get_result() 2022-05-14T01:19:11.871 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T01:19:11.871 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise self._exception 2022-05-14T01:19:11.871 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T01:19:11.871 INFO:journalctl@ceph.mon.c.smithi115.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T01:19:11.872 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T01:19:11.872 INFO:journalctl@ceph.mon.c.smithi115.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T01:19:11.872 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T01:19:11.872 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise 2022-05-14T01:19:11.872 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T01:19:11.873 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.gen.throw(type, value, traceback) 2022-05-14T01:19:11.873 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T01:19:11.873 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OrchestratorError(msg) 2022-05-14T01:19:11.873 INFO:journalctl@ceph.mon.c.smithi115.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:19:11.874 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:19:11 smithi115 ceph-mon[78593]: pgmap v3695: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:19:12.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:19:11 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:19:12.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:19:11 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:19:12.041 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T01:19:12.042 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:19:12.042 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T01:19:12.042 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:19:12.042 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:19:12.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:19:12.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T01:19:12.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:19:12.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T01:19:12.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:19:12.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T01:19:12.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:19:12.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T01:19:12.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:19:12.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T01:19:12.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:19:12.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:19:12.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:19:12.047 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:19:11 smithi047 ceph-mon[77147]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T01:19:12.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T01:19:12.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:19:12.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T01:19:12.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:19:12.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:19:12.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:19:12.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T01:19:12.049 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:19:12.049 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T01:19:12.049 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:19:12.049 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T01:19:12.049 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:19:12.050 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T01:19:12.050 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:19:12.050 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T01:19:12.050 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:19:12.051 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:19:12.051 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:19:12.051 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T01:19:12.051 INFO:journalctl@ceph.mon.b.smithi047.stdout: During handling of the above exception, another exception occurred: 2022-05-14T01:19:12.051 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T01:19:12.052 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T01:19:12.052 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T01:19:12.052 INFO:journalctl@ceph.mon.b.smithi047.stdout: return f(*arg) 2022-05-14T01:19:12.052 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T01:19:12.053 INFO:journalctl@ceph.mon.b.smithi047.stdout: self._write_client_files(client_files, host) 2022-05-14T01:19:12.053 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T01:19:12.053 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T01:19:12.053 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T01:19:12.054 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T01:19:12.054 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T01:19:12.054 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.event_loop.get_result(coro) 2022-05-14T01:19:12.054 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T01:19:12.054 INFO:journalctl@ceph.mon.b.smithi047.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T01:19:12.055 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T01:19:12.055 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.__get_result() 2022-05-14T01:19:12.055 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T01:19:12.055 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise self._exception 2022-05-14T01:19:12.055 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T01:19:12.056 INFO:journalctl@ceph.mon.b.smithi047.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T01:19:12.056 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T01:19:12.056 INFO:journalctl@ceph.mon.b.smithi047.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T01:19:12.056 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T01:19:12.057 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise 2022-05-14T01:19:12.057 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T01:19:12.057 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.gen.throw(type, value, traceback) 2022-05-14T01:19:12.057 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T01:19:12.057 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OrchestratorError(msg) 2022-05-14T01:19:12.058 INFO:journalctl@ceph.mon.b.smithi047.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:19:12.058 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:19:11 smithi047 ceph-mon[77147]: pgmap v3695: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:19:12.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:19:12 smithi115 conmon[90393]: 2022-05-14T01:19:12.518+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:19:12.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:19:12 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:19:13.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:19:12 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:19:13.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:19:13 smithi115 conmon[90393]: 2022-05-14T01:19:13.503+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:19:13.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:19:13 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:19:13.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:19:13 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:19:13.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:19:13 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:19:13.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:19:13 smithi115 ceph-mon[78593]: pgmap v3696: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:19:13.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:19:13 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T01:19:13.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:19:13 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T01:19:14.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:19:13 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:19:14.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:19:13 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:19:14.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:19:13 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:19:14.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:19:13 smithi047 ceph-mon[77147]: pgmap v3696: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:19:14.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:19:13 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T01:19:14.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:19:13 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T01:19:14.710 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:19:14 smithi115 conmon[90393]: 2022-05-14T01:19:14.455+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:19:15.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:19:14 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:19:15.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:19:14 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 6685 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:19:15.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:19:14 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:19:15.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:19:14 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 6685 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:19:15.713 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:19:15 smithi115 conmon[90393]: 2022-05-14T01:19:15.461+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:19:16.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:19:15 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:19:16.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:19:15 smithi047 ceph-mon[77147]: pgmap v3697: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:19:16.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:19:15 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:19:16.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:19:15 smithi115 ceph-mon[78593]: pgmap v3697: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:19:16.715 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:19:16 smithi115 conmon[90393]: 2022-05-14T01:19:16.452+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:19:17.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:19:16 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:19:17.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:19:16 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:19:17.724 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:19:17 smithi115 conmon[90393]: 2022-05-14T01:19:17.430+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:19:18.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:19:17 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:19:18.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:19:17 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:19:18.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:19:17 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:19:18.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:19:17 smithi047 ceph-mon[77147]: pgmap v3698: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:19:18.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:19:17 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:19:18.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:19:17 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:19:18.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:19:17 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:19:18.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:19:17 smithi115 ceph-mon[78593]: pgmap v3698: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:19:18.726 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:19:18 smithi115 conmon[90393]: 2022-05-14T01:19:18.414+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:19:19.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:19:18 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:19:19.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:19:18 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:19:19.730 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:19:19 smithi115 conmon[90393]: 2022-05-14T01:19:19.436+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:19:20.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:19:19 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:19:20.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:19:19 smithi047 ceph-mon[77147]: pgmap v3699: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:19:20.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:19:19 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 6690 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:19:20.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:19:19 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:19:20.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:19:19 smithi115 ceph-mon[78593]: pgmap v3699: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:19:20.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:19:19 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 6690 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:19:20.732 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:19:20 smithi115 conmon[90393]: 2022-05-14T01:19:20.387+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:19:21.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:19:20 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:19:21.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:19:20 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:19:21.743 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:19:21 smithi115 conmon[90393]: 2022-05-14T01:19:21.373+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:19:22.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:19:21 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:19:22.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:19:21 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:19:22.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:19:21 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:19:22.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:19:21 smithi047 ceph-mon[77147]: pgmap v3700: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:19:22.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:19:21 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:19:22.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:19:21 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:19:22.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:19:21 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:19:22.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:19:21 smithi115 ceph-mon[78593]: pgmap v3700: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:19:22.743 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:19:22 smithi115 conmon[90393]: 2022-05-14T01:19:22.400+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:19:23.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:19:22 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:19:23.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:19:22 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:19:23.748 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:19:23 smithi115 conmon[90393]: 2022-05-14T01:19:23.378+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:19:24.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:19:23 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:19:24.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:19:23 smithi047 ceph-mon[77147]: pgmap v3701: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:19:24.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:19:23 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:19:24.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:19:23 smithi115 ceph-mon[78593]: pgmap v3701: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:19:24.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:19:24 smithi115 conmon[90393]: 2022-05-14T01:19:24.330+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:19:25.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:19:24 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:19:25.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:19:24 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 6695 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:19:25.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:19:24 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:19:25.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:19:24 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 6695 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:19:25.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:19:25 smithi115 conmon[90393]: 2022-05-14T01:19:25.304+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:19:26.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:19:25 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:19:26.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:19:25 smithi047 ceph-mon[77147]: pgmap v3702: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:19:26.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:19:25 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:19:26.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:19:25 smithi115 ceph-mon[78593]: pgmap v3702: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:19:26.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:19:26 smithi115 conmon[90393]: 2022-05-14T01:19:26.335+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:19:27.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:19:26 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:19:27.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:19:26 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:19:27.761 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:19:27 smithi115 conmon[90393]: 2022-05-14T01:19:27.384+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:19:28.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:19:27 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:19:28.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:19:27 smithi047 ceph-mon[77147]: pgmap v3703: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:19:28.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:19:27 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:19:28.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:19:27 smithi115 ceph-mon[78593]: pgmap v3703: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:19:28.763 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:19:28 smithi115 conmon[90393]: 2022-05-14T01:19:28.364+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:19:29.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:19:28 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:19:29.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:19:28 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:19:29.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:19:29 smithi115 conmon[90393]: 2022-05-14T01:19:29.336+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:19:30.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:19:29 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:19:30.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:19:29 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:19:30.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:19:29 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:19:30.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:19:29 smithi047 ceph-mon[77147]: pgmap v3704: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:19:30.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:19:29 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 6700 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:19:30.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:19:29 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:19:30.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:19:29 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:19:30.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:19:29 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:19:30.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:19:29 smithi115 ceph-mon[78593]: pgmap v3704: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:19:30.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:19:29 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 6700 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:19:30.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:19:30 smithi115 conmon[90393]: 2022-05-14T01:19:30.321+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:19:31.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:19:30 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:19:31.112 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:19:30 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:19:31.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:19:31 smithi115 conmon[90393]: 2022-05-14T01:19:31.330+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:19:32.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:19:31 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:19:32.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:19:31 smithi047 ceph-mon[77147]: pgmap v3705: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:19:32.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:19:31 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:19:32.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:19:31 smithi115 ceph-mon[78593]: pgmap v3705: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:19:32.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:19:32 smithi115 conmon[90393]: 2022-05-14T01:19:32.348+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:19:33.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:19:32 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:19:33.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:19:32 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:19:33.788 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:19:33 smithi115 conmon[90393]: 2022-05-14T01:19:33.385+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:19:34.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:19:33 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:19:34.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:19:33 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:19:34.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:19:33 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:19:34.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:19:33 smithi047 ceph-mon[77147]: pgmap v3706: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:19:34.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:19:33 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:19:34.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:19:33 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:19:34.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:19:33 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:19:34.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:19:33 smithi115 ceph-mon[78593]: pgmap v3706: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:19:34.790 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:19:34 smithi115 conmon[90393]: 2022-05-14T01:19:34.378+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:19:35.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:19:34 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:19:35.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:19:34 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 6705 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:19:35.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:19:34 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:19:35.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:19:34 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 6705 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:19:35.795 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:19:35 smithi115 conmon[90393]: 2022-05-14T01:19:35.379+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:19:36.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:19:35 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:19:36.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:19:35 smithi115 ceph-mon[78593]: pgmap v3707: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 7 op/s 2022-05-14T01:19:36.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:19:35 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:19:36.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:19:35 smithi047 ceph-mon[77147]: pgmap v3707: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 7 op/s 2022-05-14T01:19:36.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:19:36 smithi115 conmon[90393]: 2022-05-14T01:19:36.353+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:19:37.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:19:36 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:19:37.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:19:36 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:19:37.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:19:37 smithi115 conmon[90393]: 2022-05-14T01:19:37.328+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:19:38.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:19:37 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:19:38.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:19:37 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:19:38.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:19:37 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:19:38.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:19:37 smithi115 ceph-mon[78593]: pgmap v3708: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 4 op/s 2022-05-14T01:19:38.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:19:37 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:19:38.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:19:37 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:19:38.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:19:37 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:19:38.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:19:37 smithi047 ceph-mon[77147]: pgmap v3708: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 4 op/s 2022-05-14T01:19:38.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:19:38 smithi115 conmon[90393]: 2022-05-14T01:19:38.300+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:19:39.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:19:38 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:19:39.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:19:38 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:19:39.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:19:39 smithi115 conmon[90393]: 2022-05-14T01:19:39.283+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:19:40.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:19:39 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:19:40.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:19:39 smithi115 ceph-mon[78593]: pgmap v3709: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 7 op/s 2022-05-14T01:19:40.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:19:39 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:19:40.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:19:39 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 6710 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:19:40.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:19:39 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:19:40.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:19:39 smithi047 ceph-mon[77147]: pgmap v3709: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 7 op/s 2022-05-14T01:19:40.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:19:39 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:19:40.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:19:39 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 6710 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:19:40.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:19:40 smithi115 conmon[90393]: 2022-05-14T01:19:40.298+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:19:41.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:19:41 smithi115 conmon[90393]: 2022-05-14T01:19:41.252+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:19:42.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:19:41 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:19:42.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:19:41 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:19:42.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:19:41 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:19:42.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:19:41 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:19:42.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:19:41 smithi115 ceph-mon[78593]: pgmap v3710: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 4 op/s 2022-05-14T01:19:42.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:19:41 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:19:42.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:19:41 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:19:42.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:19:41 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:19:42.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:19:41 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:19:42.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:19:41 smithi047 ceph-mon[77147]: pgmap v3710: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 4 op/s 2022-05-14T01:19:42.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:19:42 smithi115 conmon[90393]: 2022-05-14T01:19:42.227+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:19:43.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:19:42 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:19:43.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:19:42 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:19:43.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:19:43 smithi115 conmon[90393]: 2022-05-14T01:19:43.231+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:19:44.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:19:43 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:19:44.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:19:43 smithi115 ceph-mon[78593]: pgmap v3711: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 4 op/s 2022-05-14T01:19:44.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:19:43 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:19:44.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:19:43 smithi047 ceph-mon[77147]: pgmap v3711: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 4 op/s 2022-05-14T01:19:44.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:19:44 smithi115 conmon[90393]: 2022-05-14T01:19:44.259+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:19:45.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:19:44 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:19:45.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:19:44 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 6715 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:19:45.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:19:44 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:19:45.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:19:44 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 6715 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:19:45.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:19:45 smithi115 conmon[90393]: 2022-05-14T01:19:45.252+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:19:46.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:19:45 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:19:46.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:19:45 smithi115 ceph-mon[78593]: pgmap v3712: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 7 op/s 2022-05-14T01:19:46.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:19:45 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:19:46.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:19:45 smithi047 ceph-mon[77147]: pgmap v3712: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 7 op/s 2022-05-14T01:19:46.611 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:19:46 smithi115 conmon[90393]: 2022-05-14T01:19:46.302+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:19:47.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:19:47 smithi115 conmon[90393]: 2022-05-14T01:19:47.307+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:19:47.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:19:47 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:19:47.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:19:47 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:19:48.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:19:48 smithi115 conmon[90393]: 2022-05-14T01:19:48.323+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:19:48.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:19:48 smithi115 ceph-mon[78593]: pgmap v3713: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:19:48.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:19:48 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:19:48.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:19:48 smithi047 ceph-mon[77147]: pgmap v3713: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:19:48.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:19:48 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:19:49.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:19:49 smithi115 conmon[90393]: 2022-05-14T01:19:49.275+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:19:49.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:19:49 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:19:49.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:19:49 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:19:49.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:19:49 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:19:49.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:19:49 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:19:50.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:19:50 smithi115 conmon[90393]: 2022-05-14T01:19:50.318+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:19:50.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:19:50 smithi115 ceph-mon[78593]: pgmap v3714: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:19:50.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:19:50 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:19:50.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:19:50 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:19:50.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:19:50 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 6725 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:19:50.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:19:50 smithi047 ceph-mon[77147]: pgmap v3714: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:19:50.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:19:50 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:19:50.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:19:50 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:19:50.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:19:50 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 6725 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:19:51.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:19:51 smithi115 conmon[90393]: 2022-05-14T01:19:51.275+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:19:51.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:19:51 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:19:51.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:19:51 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:19:51.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:19:51 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:19:51.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:19:51 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:19:52.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:19:52 smithi115 ceph-mon[78593]: pgmap v3715: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:19:52.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:19:52 smithi115 conmon[90393]: 2022-05-14T01:19:52.321+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:19:52.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:19:52 smithi047 ceph-mon[77147]: pgmap v3715: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:19:53.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:19:53 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:19:53.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:19:53 smithi115 conmon[90393]: 2022-05-14T01:19:53.291+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:19:53.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:19:53 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:19:54.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:19:54 smithi115 conmon[90393]: 2022-05-14T01:19:54.265+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:19:54.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:19:54 smithi115 ceph-mon[78593]: pgmap v3716: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:19:54.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:19:54 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:19:54.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:19:54 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:19:54.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:19:54 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:19:54.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:19:54 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:19:54.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:19:54 smithi047 ceph-mon[77147]: pgmap v3716: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:19:54.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:19:54 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:19:54.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:19:54 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:19:54.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:19:54 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:19:54.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:19:54 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:19:55.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:19:55 smithi115 conmon[90393]: 2022-05-14T01:19:55.243+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:19:56.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:19:55 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:19:56.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:19:55 smithi047 ceph-mon[77147]: pgmap v3717: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:19:56.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:19:55 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:19:56.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:19:55 smithi115 ceph-mon[78593]: pgmap v3717: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:19:56.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:19:56 smithi115 conmon[90393]: 2022-05-14T01:19:56.245+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:19:57.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:19:56 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 6730 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:19:57.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:19:56 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:19:57.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:19:56 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 6730 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:19:57.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:19:56 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:19:57.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:19:57 smithi115 conmon[90393]: 2022-05-14T01:19:57.287+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:19:58.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:19:57 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:19:58.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:19:57 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:19:58.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:19:57 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:19:58.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:19:57 smithi047 ceph-mon[77147]: pgmap v3718: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:19:58.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:19:57 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:19:58.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:19:57 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:19:58.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:19:57 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:19:58.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:19:57 smithi115 ceph-mon[78593]: pgmap v3718: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:19:58.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:19:58 smithi115 conmon[90393]: 2022-05-14T01:19:58.292+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:19:59.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:19:59 smithi115 conmon[90393]: 2022-05-14T01:19:59.297+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:19:59.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:19:59 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:19:59.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:19:59 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:20:00.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:20:00 smithi047 conmon[77123]: 2022-05-14T01:19:59.999+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : Health detail: HEALTH_ERR 1 filesystem is degraded; 1 mds daemon damaged; 1 MDSs are read only; 1 MDSs report slow metadata IOs; 1/3 mons down, quorum b,c; Reduced data availability: 1 pg inactive; 9 slow ops, oldest one blocked for 6735 sec, osd.5 has slow ops 2022-05-14T01:20:00.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:20:00 smithi047 conmon[77123]: 2022-05-14T01:19:59.999+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : [WRN] FS_DEGRADED: 1 filesystem is degraded 2022-05-14T01:20:00.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:20:00 smithi047 conmon[77123]: 2022-05-14T01:19:59.999+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : fs cephfs is degraded 2022-05-14T01:20:00.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:20:00 smithi047 conmon[77123]: 2022-05-14T01:19:59.999+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : [ERR] MDS_DAMAGE: 1 mds daemon damaged 2022-05-14T01:20:00.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:20:00 smithi047 conmon[77123]: 2022-05-14T01:19:59.999+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : fs cephfs mds.2 is damaged 2022-05-14T01:20:00.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:20:00 smithi047 conmon[77123]: 2022-05-14T01:19:59.999+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : [WRN] MDS_READ_ONLY: 1 MDSs are read only 2022-05-14T01:20:00.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:20:00 smithi047 conmon[77123]: 2022-05-14T01:19:59.999+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : mds.l(mds.0): MDS in read-only mode 2022-05-14T01:20:00.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:20:00 smithi047 conmon[77123]: 2022-05-14T01:19:59.999+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : [WRN] MDS_SLOW_METADATA_IO: 1 MDSs report slow metadata IOs 2022-05-14T01:20:00.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:20:00 smithi047 conmon[77123]: 2022-05-14T01:19:59.999+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : mds.l(mds.0): 4 slow metadata IOs are blocked > 180 secs, oldest blocked for 7033 secs 2022-05-14T01:20:00.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:20:00 smithi047 conmon[77123]: 2022-05-14T01:19:59.999+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : [WRN] MON_DOWN: 1/3 mons down, quorum b,c 2022-05-14T01:20:00.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:20:00 smithi047 conmon[77123]: 2022-05-14T01:19:59.999+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : mon.a (rank 0) addr [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] is down (out of quorum) 2022-05-14T01:20:00.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:20:00 smithi047 conmon[77123]: 2022-05-14T01:19:59.999+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : [WRN] PG_AVAILABILITY: Reduced data availability: 1 pg inactive 2022-05-14T01:20:00.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:20:00 smithi047 conmon[77123]: 2022-05-14T01:19:59.999+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : pg 2.1 is stuck inactive for 2h, current state unknown, last acting [] 2022-05-14T01:20:00.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:20:00 smithi047 conmon[77123]: 2022-05-14T01:19:59.999+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : [WRN] SLOW_OPS: 9 slow ops, oldest one blocked for 6735 sec, osd.5 has slow ops 2022-05-14T01:20:00.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:20:00 smithi115 conmon[90393]: 2022-05-14T01:20:00.301+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:20:00.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:20:00 smithi115 ceph-mon[78593]: pgmap v3719: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:20:00.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:20:00 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:20:00.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:20:00 smithi115 ceph-mon[78593]: Health detail: HEALTH_ERR 1 filesystem is degraded; 1 mds daemon damaged; 1 MDSs are read only; 1 MDSs report slow metadata IOs; 1/3 mons down, quorum b,c; Reduced data availability: 1 pg inactive; 9 slow ops, oldest one blocked for 6735 sec, osd.5 has slow ops 2022-05-14T01:20:00.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:20:00 smithi115 ceph-mon[78593]: [WRN] FS_DEGRADED: 1 filesystem is degraded 2022-05-14T01:20:00.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:20:00 smithi115 ceph-mon[78593]: fs cephfs is degraded 2022-05-14T01:20:00.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:20:00 smithi115 ceph-mon[78593]: [ERR] MDS_DAMAGE: 1 mds daemon damaged 2022-05-14T01:20:00.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:20:00 smithi115 ceph-mon[78593]: fs cephfs mds.2 is damaged 2022-05-14T01:20:00.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:20:00 smithi115 ceph-mon[78593]: [WRN] MDS_READ_ONLY: 1 MDSs are read only 2022-05-14T01:20:00.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:20:00 smithi115 ceph-mon[78593]: mds.l(mds.0): MDS in read-only mode 2022-05-14T01:20:00.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:20:00 smithi115 ceph-mon[78593]: [WRN] MDS_SLOW_METADATA_IO: 1 MDSs report slow metadata IOs 2022-05-14T01:20:00.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:20:00 smithi115 ceph-mon[78593]: mds.l(mds.0): 4 slow metadata IOs are blocked > 180 secs, oldest blocked for 7033 secs 2022-05-14T01:20:00.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:20:00 smithi115 ceph-mon[78593]: [WRN] MON_DOWN: 1/3 mons down, quorum b,c 2022-05-14T01:20:00.612 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:20:00 smithi115 ceph-mon[78593]: mon.a (rank 0) addr [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] is down (out of quorum) 2022-05-14T01:20:00.612 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:20:00 smithi115 ceph-mon[78593]: [WRN] PG_AVAILABILITY: Reduced data availability: 1 pg inactive 2022-05-14T01:20:00.612 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:20:00 smithi115 ceph-mon[78593]: pg 2.1 is stuck inactive for 2h, current state unknown, last acting [] 2022-05-14T01:20:00.612 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:20:00 smithi115 ceph-mon[78593]: [WRN] SLOW_OPS: 9 slow ops, oldest one blocked for 6735 sec, osd.5 has slow ops 2022-05-14T01:20:00.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:20:00 smithi047 ceph-mon[77147]: pgmap v3719: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:20:00.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:20:00 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:20:00.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:20:00 smithi047 ceph-mon[77147]: Health detail: HEALTH_ERR 1 filesystem is degraded; 1 mds daemon damaged; 1 MDSs are read only; 1 MDSs report slow metadata IOs; 1/3 mons down, quorum b,c; Reduced data availability: 1 pg inactive; 9 slow ops, oldest one blocked for 6735 sec, osd.5 has slow ops 2022-05-14T01:20:00.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:20:00 smithi047 ceph-mon[77147]: [WRN] FS_DEGRADED: 1 filesystem is degraded 2022-05-14T01:20:00.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:20:00 smithi047 ceph-mon[77147]: fs cephfs is degraded 2022-05-14T01:20:00.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:20:00 smithi047 ceph-mon[77147]: [ERR] MDS_DAMAGE: 1 mds daemon damaged 2022-05-14T01:20:00.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:20:00 smithi047 ceph-mon[77147]: fs cephfs mds.2 is damaged 2022-05-14T01:20:00.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:20:00 smithi047 ceph-mon[77147]: [WRN] MDS_READ_ONLY: 1 MDSs are read only 2022-05-14T01:20:00.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:20:00 smithi047 ceph-mon[77147]: mds.l(mds.0): MDS in read-only mode 2022-05-14T01:20:00.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:20:00 smithi047 ceph-mon[77147]: [WRN] MDS_SLOW_METADATA_IO: 1 MDSs report slow metadata IOs 2022-05-14T01:20:00.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:20:00 smithi047 ceph-mon[77147]: mds.l(mds.0): 4 slow metadata IOs are blocked > 180 secs, oldest blocked for 7033 secs 2022-05-14T01:20:00.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:20:00 smithi047 ceph-mon[77147]: [WRN] MON_DOWN: 1/3 mons down, quorum b,c 2022-05-14T01:20:00.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:20:00 smithi047 ceph-mon[77147]: mon.a (rank 0) addr [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] is down (out of quorum) 2022-05-14T01:20:00.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:20:00 smithi047 ceph-mon[77147]: [WRN] PG_AVAILABILITY: Reduced data availability: 1 pg inactive 2022-05-14T01:20:00.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:20:00 smithi047 ceph-mon[77147]: pg 2.1 is stuck inactive for 2h, current state unknown, last acting [] 2022-05-14T01:20:00.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:20:00 smithi047 ceph-mon[77147]: [WRN] SLOW_OPS: 9 slow ops, oldest one blocked for 6735 sec, osd.5 has slow ops 2022-05-14T01:20:01.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:20:01 smithi115 conmon[90393]: 2022-05-14T01:20:01.299+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:20:01.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:20:01 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:20:01.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:20:01 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:20:02.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:20:02 smithi115 conmon[90393]: 2022-05-14T01:20:02.280+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:20:02.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:20:02 smithi115 ceph-mon[78593]: pgmap v3720: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:20:02.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:20:02 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:20:02.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:20:02 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:20:02.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:20:02 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:20:02.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:20:02 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:20:02.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:20:02 smithi047 ceph-mon[77147]: pgmap v3720: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:20:02.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:20:02 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:20:02.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:20:02 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:20:02.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:20:02 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:20:02.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:20:02 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:20:03.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:20:03 smithi115 conmon[90393]: 2022-05-14T01:20:03.305+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:20:04.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:20:04 smithi115 ceph-mon[78593]: pgmap v3721: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:20:04.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:20:04 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:20:04.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:20:04 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:20:04.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:20:04 smithi115 conmon[90393]: 2022-05-14T01:20:04.257+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:20:04.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:20:04 smithi047 ceph-mon[77147]: pgmap v3721: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:20:04.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:20:04 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:20:04.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:20:04 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:20:05.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:20:05 smithi115 conmon[90393]: 2022-05-14T01:20:05.274+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:20:05.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:20:05 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 6735 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:20:05.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:20:05 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:20:05.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:20:05 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 6735 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:20:05.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:20:05 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:20:06.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:20:06 smithi115 conmon[90393]: 2022-05-14T01:20:06.317+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:20:06.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:20:06 smithi115 ceph-mon[78593]: pgmap v3722: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:20:06.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:20:06 smithi047 ceph-mon[77147]: pgmap v3722: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:20:07.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:20:07 smithi115 conmon[90393]: 2022-05-14T01:20:07.324+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:20:07.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:20:07 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:20:07.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:20:07 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:20:08.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:20:08 smithi047 ceph-mon[77147]: pgmap v3723: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:20:08.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:20:08 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:20:08.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:20:08 smithi115 conmon[90393]: 2022-05-14T01:20:08.358+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:20:08.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:20:08 smithi115 ceph-mon[78593]: pgmap v3723: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:20:08.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:20:08 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:20:09.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:20:09 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:20:09.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:20:09 smithi115 conmon[90393]: 2022-05-14T01:20:09.379+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:20:09.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:20:09 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:20:10.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:20:10 smithi115 conmon[90393]: 2022-05-14T01:20:10.347+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:20:10.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:20:10 smithi115 ceph-mon[78593]: pgmap v3724: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:20:10.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:20:10 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:20:10.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:20:10 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:20:10.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:20:10 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:20:10.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:20:10 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 6745 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:20:10.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:20:10 smithi047 ceph-mon[77147]: pgmap v3724: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:20:10.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:20:10 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:20:10.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:20:10 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:20:10.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:20:10 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:20:10.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:20:10 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 6745 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:20:11.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:20:11 smithi115 conmon[90393]: 2022-05-14T01:20:11.311+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:20:11.635 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:20:11 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:20:11.635 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:20:11 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T01:20:11.636 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:20:11 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T01:20:11.636 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:20:11 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T01:20:11.640 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:20:11 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:20:11.640 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:20:11 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T01:20:11.640 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:20:11 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T01:20:11.640 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:20:11 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T01:20:11.641 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:20:11 smithi047 conmon[78521]: 2022-05-14T01:20:11.503+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:20:11.641 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:20:11 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T01:20:11.641 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:20:11 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:20:11.642 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:20:11 smithi047 conmon[78521]: yield 2022-05-14T01:20:11.642 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:20:11 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:20:11.642 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:20:11 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:20:11.642 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:20:11 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:20:11.642 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:20:11 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T01:20:11.643 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:20:11 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:20:11.643 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:20:11 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T01:20:11.643 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:20:11 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:20:11.643 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:20:11 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T01:20:11.644 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:20:11 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:20:11.644 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:20:11 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T01:20:11.644 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:20:11 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:20:11.644 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:20:11 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T01:20:11.644 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:20:11 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:20:11.645 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:20:11 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:20:11.645 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:20:11 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:20:12.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:20:12 smithi115 conmon[90393]: 2022-05-14T01:20:12.302+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:20:12.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:20:12 smithi115 ceph-mon[78593]: pgmap v3725: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:20:12.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:20:12 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:20:12.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:20:12 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:20:12.609 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T01:20:12.609 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:20:12.609 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T01:20:12.609 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:20:12.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:20:12.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:20:12.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T01:20:12.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:20:12.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T01:20:12.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:20:12.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T01:20:12.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:20:12.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T01:20:12.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:20:12.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T01:20:12.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:20:12.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:20:12.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:20:12.612 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:20:12 smithi115 ceph-mon[78593]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T01:20:12.613 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:20:12 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' 2022-05-14T01:20:12.613 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:20:12 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' 2022-05-14T01:20:12.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:20:12 smithi047 ceph-mon[77147]: pgmap v3725: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:20:12.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:20:12 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:20:12.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:20:12 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:20:12.792 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T01:20:12.792 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:20:12.792 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T01:20:12.792 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:20:12.793 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:20:12.793 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:20:12.793 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T01:20:12.793 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:20:12.793 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T01:20:12.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:20:12.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T01:20:12.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:20:12.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T01:20:12.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:20:12.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T01:20:12.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:20:12.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:20:12.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:20:12.795 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:20:12 smithi047 ceph-mon[77147]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T01:20:12.796 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:20:12 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' 2022-05-14T01:20:12.796 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:20:12 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' 2022-05-14T01:20:13.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:20:13 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:20:13.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:20:13 smithi115 conmon[90393]: 2022-05-14T01:20:13.299+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:20:13.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:20:13 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:20:14.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:20:14 smithi115 conmon[90393]: 2022-05-14T01:20:14.324+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:20:14.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:20:14 smithi115 ceph-mon[78593]: pgmap v3726: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:20:14.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:20:14 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:20:14.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:20:14 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:20:14.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:20:14 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:20:14.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:20:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T01:20:14.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:20:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T01:20:14.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:20:14 smithi047 ceph-mon[77147]: pgmap v3726: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:20:14.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:20:14 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:20:14.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:20:14 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:20:14.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:20:14 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:20:14.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:20:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T01:20:14.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:20:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T01:20:14.793 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:20:14 smithi047 conmon[78521]: 2022-05-14T01:20:14.575+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:20:14.793 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:20:14 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T01:20:14.793 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:20:14 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:20:14.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:20:14 smithi047 conmon[78521]: yield 2022-05-14T01:20:14.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:20:14 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:20:14.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:20:14 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:20:14.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:20:14 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:20:14.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:20:14 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T01:20:14.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:20:14 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:20:14.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:20:14 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T01:20:14.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:20:14 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:20:14.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:20:14 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T01:20:14.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:20:14 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:20:14.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:20:14 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T01:20:14.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:20:14 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:20:14.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:20:14 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T01:20:14.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:20:14 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:20:14.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:20:14 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:20:14.797 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:20:14 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:20:14.797 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:20:14 smithi047 conmon[78521]: 2022-05-14T01:20:14.576+0000 7f5341f7c700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T01:20:14.797 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:20:14 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T01:20:14.797 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:20:14 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:20:14.798 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:20:14 smithi047 conmon[78521]: yield 2022-05-14T01:20:14.798 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:20:14 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:20:14.798 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:20:14 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:20:14.798 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:20:14 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:20:14.798 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:20:14 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T01:20:14.799 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:20:14 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:20:14.799 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:20:14 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T01:20:14.799 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:20:14 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:20:14.799 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:20:14 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T01:20:14.799 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:20:14 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:20:14.800 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:20:14 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T01:20:14.800 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:20:14 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:20:14.800 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:20:14 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T01:20:14.801 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:20:14 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:20:14.801 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:20:14 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:20:14.801 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:20:14 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:20:14.801 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:20:14 smithi047 conmon[78521]: 2022-05-14T01:20:14.802 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:20:14 smithi047 conmon[78521]: During handling of the above exception, another exception occurred: 2022-05-14T01:20:14.802 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:20:14 smithi047 conmon[78521]: 2022-05-14T01:20:14.802 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:20:14 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T01:20:14.802 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:20:14 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T01:20:14.802 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:20:14 smithi047 conmon[78521]: return f(*arg) 2022-05-14T01:20:14.803 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:20:14 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T01:20:14.803 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:20:14 smithi047 conmon[78521]: self._write_client_files(client_files, host) 2022-05-14T01:20:14.803 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:20:14 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T01:20:14.803 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:20:14 smithi047 conmon[78521]: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T01:20:14.803 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:20:14 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T01:20:14.804 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:20:14 smithi047 conmon[78521]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T01:20:14.804 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:20:14 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T01:20:14.804 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:20:14 smithi047 conmon[78521]: return self.event_loop.get_result(coro) 2022-05-14T01:20:14.804 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:20:14 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T01:20:14.805 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:20:14 smithi047 conmon[78521]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T01:20:14.805 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:20:14 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T01:20:14.805 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:20:14 smithi047 conmon[78521]: return self.__get_result() 2022-05-14T01:20:14.805 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:20:14 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T01:20:14.805 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:20:14 smithi047 conmon[78521]: raise self._exception 2022-05-14T01:20:14.806 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:20:14 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T01:20:14.806 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:20:14 smithi047 conmon[78521]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T01:20:14.806 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:20:14 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T01:20:14.806 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:20:14 smithi047 conmon[78521]: conn = await self._remote_connection(host, addr) 2022-05-14T01:20:14.806 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:20:14 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T01:20:14.807 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:20:14 smithi047 conmon[78521]: raise 2022-05-14T01:20:14.807 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:20:14 smithi047 conmon[78521]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T01:20:14.807 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:20:14 smithi047 conmon[78521]: self.gen.throw(type, value, traceback) 2022-05-14T01:20:14.807 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:20:14 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T01:20:14.807 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:20:14 smithi047 conmon[78521]: raise OrchestratorError(msg) 2022-05-14T01:20:14.808 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:20:14 smithi047 conmon[78521]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:20:15.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:20:15 smithi115 conmon[90393]: 2022-05-14T01:20:15.277+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:20:15.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:20:15 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:20:15.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:20:15 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:20:15.608 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T01:20:15.609 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:20:15.609 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T01:20:15.609 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:20:15.609 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:20:15.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:20:15.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T01:20:15.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:20:15.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T01:20:15.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:20:15.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T01:20:15.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:20:15.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T01:20:15.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:20:15.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T01:20:15.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:20:15.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:20:15.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:20:15.612 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:20:15 smithi115 ceph-mon[78593]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T01:20:15.613 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T01:20:15.613 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:20:15.613 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T01:20:15.613 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:20:15.614 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:20:15.614 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:20:15.614 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T01:20:15.614 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:20:15.614 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T01:20:15.615 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:20:15.615 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T01:20:15.615 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:20:15.615 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T01:20:15.616 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:20:15.616 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T01:20:15.616 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:20:15.616 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:20:15.616 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:20:15.617 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T01:20:15.617 INFO:journalctl@ceph.mon.c.smithi115.stdout: During handling of the above exception, another exception occurred: 2022-05-14T01:20:15.617 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T01:20:15.617 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T01:20:15.618 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T01:20:15.618 INFO:journalctl@ceph.mon.c.smithi115.stdout: return f(*arg) 2022-05-14T01:20:15.618 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T01:20:15.618 INFO:journalctl@ceph.mon.c.smithi115.stdout: self._write_client_files(client_files, host) 2022-05-14T01:20:15.618 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T01:20:15.619 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T01:20:15.619 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T01:20:15.619 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T01:20:15.619 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T01:20:15.619 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.event_loop.get_result(coro) 2022-05-14T01:20:15.620 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T01:20:15.620 INFO:journalctl@ceph.mon.c.smithi115.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T01:20:15.620 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T01:20:15.620 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.__get_result() 2022-05-14T01:20:15.621 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T01:20:15.621 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise self._exception 2022-05-14T01:20:15.621 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T01:20:15.621 INFO:journalctl@ceph.mon.c.smithi115.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T01:20:15.621 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T01:20:15.622 INFO:journalctl@ceph.mon.c.smithi115.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T01:20:15.622 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T01:20:15.622 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise 2022-05-14T01:20:15.622 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T01:20:15.622 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.gen.throw(type, value, traceback) 2022-05-14T01:20:15.623 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T01:20:15.623 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OrchestratorError(msg) 2022-05-14T01:20:15.623 INFO:journalctl@ceph.mon.c.smithi115.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:20:15.623 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:20:15 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:20:15.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:20:15 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:20:15.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:20:15 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:20:15.793 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T01:20:15.793 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:20:15.793 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T01:20:15.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:20:15.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:20:15.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:20:15.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T01:20:15.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:20:15.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T01:20:15.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:20:15.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T01:20:15.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:20:15.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T01:20:15.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:20:15.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T01:20:15.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:20:15.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:20:15.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:20:15.797 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:20:15 smithi047 ceph-mon[77147]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T01:20:15.797 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T01:20:15.797 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:20:15.797 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T01:20:15.798 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:20:15.798 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:20:15.798 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:20:15.798 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T01:20:15.799 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:20:15.799 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T01:20:15.799 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:20:15.799 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T01:20:15.799 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:20:15.800 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T01:20:15.800 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:20:15.800 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T01:20:15.800 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:20:15.801 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:20:15.801 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:20:15.801 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T01:20:15.801 INFO:journalctl@ceph.mon.b.smithi047.stdout: During handling of the above exception, another exception occurred: 2022-05-14T01:20:15.802 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T01:20:15.802 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T01:20:15.802 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T01:20:15.802 INFO:journalctl@ceph.mon.b.smithi047.stdout: return f(*arg) 2022-05-14T01:20:15.802 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T01:20:15.803 INFO:journalctl@ceph.mon.b.smithi047.stdout: self._write_client_files(client_files, host) 2022-05-14T01:20:15.803 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T01:20:15.803 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T01:20:15.803 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T01:20:15.803 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T01:20:15.804 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T01:20:15.804 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.event_loop.get_result(coro) 2022-05-14T01:20:15.804 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T01:20:15.804 INFO:journalctl@ceph.mon.b.smithi047.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T01:20:15.805 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T01:20:15.805 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.__get_result() 2022-05-14T01:20:15.805 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T01:20:15.805 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise self._exception 2022-05-14T01:20:15.805 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T01:20:15.806 INFO:journalctl@ceph.mon.b.smithi047.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T01:20:15.806 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T01:20:15.806 INFO:journalctl@ceph.mon.b.smithi047.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T01:20:15.806 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T01:20:15.807 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise 2022-05-14T01:20:15.807 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T01:20:15.807 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.gen.throw(type, value, traceback) 2022-05-14T01:20:15.807 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T01:20:15.807 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OrchestratorError(msg) 2022-05-14T01:20:15.808 INFO:journalctl@ceph.mon.b.smithi047.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:20:15.808 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:20:15 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:20:16.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:20:16 smithi115 conmon[90393]: 2022-05-14T01:20:16.310+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:20:16.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:20:16 smithi115 ceph-mon[78593]: pgmap v3727: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:20:16.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:20:16 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 6750 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:20:16.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:20:16 smithi047 ceph-mon[77147]: pgmap v3727: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:20:16.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:20:16 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 6750 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:20:17.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:20:17 smithi115 conmon[90393]: 2022-05-14T01:20:17.348+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:20:17.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:20:17 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:20:17.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:20:17 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:20:18.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:20:18 smithi047 ceph-mon[77147]: pgmap v3728: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:20:18.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:20:18 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:20:18.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:20:18 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:20:18.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:20:18 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:20:18.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:20:18 smithi115 conmon[90393]: 2022-05-14T01:20:18.382+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:20:18.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:20:18 smithi115 ceph-mon[78593]: pgmap v3728: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:20:18.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:20:18 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:20:18.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:20:18 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:20:18.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:20:18 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:20:19.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:20:19 smithi115 conmon[90393]: 2022-05-14T01:20:19.341+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:20:19.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:20:19 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:20:19.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:20:19 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:20:20.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:20:20 smithi115 conmon[90393]: 2022-05-14T01:20:20.295+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:20:20.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:20:20 smithi115 ceph-mon[78593]: pgmap v3729: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:20:20.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:20:20 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:20:20.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:20:20 smithi047 ceph-mon[77147]: pgmap v3729: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:20:20.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:20:20 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:20:21.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:20:21 smithi115 conmon[90393]: 2022-05-14T01:20:21.325+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:20:21.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:20:21 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:20:21.796 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:20:21 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:20:22.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:20:22 smithi115 conmon[90393]: 2022-05-14T01:20:22.332+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:20:22.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:20:22 smithi115 ceph-mon[78593]: pgmap v3730: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:20:22.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:20:22 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:20:22.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:20:22 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:20:22.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:20:22 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:20:22.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:20:22 smithi047 ceph-mon[77147]: pgmap v3730: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:20:22.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:20:22 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:20:22.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:20:22 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:20:22.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:20:22 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:20:23.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:20:23 smithi115 conmon[90393]: 2022-05-14T01:20:23.289+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:20:23.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:20:23 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:20:23.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:20:23 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:20:23.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:20:23 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:20:23.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:20:23 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:20:24.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:20:24 smithi115 conmon[90393]: 2022-05-14T01:20:24.272+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:20:24.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:20:24 smithi115 ceph-mon[78593]: pgmap v3731: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:20:24.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:20:24 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:20:24.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:20:24 smithi047 ceph-mon[77147]: pgmap v3731: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:20:24.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:20:24 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:20:25.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:20:25 smithi115 conmon[90393]: 2022-05-14T01:20:25.282+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:20:25.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:20:25 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 6755 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:20:25.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:20:25 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:20:25.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:20:25 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 6755 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:20:25.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:20:25 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:20:26.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:20:26 smithi115 conmon[90393]: 2022-05-14T01:20:26.287+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:20:26.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:20:26 smithi115 ceph-mon[78593]: pgmap v3732: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:20:26.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:20:26 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:20:26.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:20:26 smithi047 ceph-mon[77147]: pgmap v3732: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:20:26.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:20:26 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:20:27.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:20:27 smithi115 conmon[90393]: 2022-05-14T01:20:27.316+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:20:28.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:20:28 smithi115 conmon[90393]: 2022-05-14T01:20:28.306+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:20:28.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:20:28 smithi115 ceph-mon[78593]: pgmap v3733: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:20:28.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:20:28 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:20:28.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:20:28 smithi047 ceph-mon[77147]: pgmap v3733: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:20:28.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:20:28 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:20:29.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:20:29 smithi115 conmon[90393]: 2022-05-14T01:20:29.266+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:20:29.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:20:29 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:20:29.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:20:29 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:20:29.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:20:29 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:20:29.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:20:29 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:20:30.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:20:30 smithi115 conmon[90393]: 2022-05-14T01:20:30.300+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:20:30.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:20:30 smithi115 ceph-mon[78593]: pgmap v3734: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:20:30.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:20:30 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:20:30.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:20:30 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:20:30.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:20:30 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 6765 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:20:30.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:20:30 smithi047 ceph-mon[77147]: pgmap v3734: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:20:30.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:20:30 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:20:30.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:20:30 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:20:30.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:20:30 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 6765 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:20:31.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:20:31 smithi115 conmon[90393]: 2022-05-14T01:20:31.280+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:20:31.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:20:31 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:20:31.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:20:31 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:20:31.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:20:31 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:20:31.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:20:31 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:20:32.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:20:32 smithi115 conmon[90393]: 2022-05-14T01:20:32.232+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:20:32.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:20:32 smithi115 ceph-mon[78593]: pgmap v3735: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:20:32.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:20:32 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:20:32.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:20:32 smithi047 ceph-mon[77147]: pgmap v3735: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:20:32.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:20:32 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:20:33.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:20:33 smithi115 conmon[90393]: 2022-05-14T01:20:33.198+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:20:33.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:20:33 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:20:33.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:20:33 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:20:34.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:20:34 smithi115 conmon[90393]: 2022-05-14T01:20:34.243+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:20:34.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:20:34 smithi115 ceph-mon[78593]: pgmap v3736: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:20:34.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:20:34 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:20:34.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:20:34 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:20:34.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:20:34 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:20:34.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:20:34 smithi047 ceph-mon[77147]: pgmap v3736: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:20:34.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:20:34 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:20:34.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:20:34 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:20:34.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:20:34 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:20:35.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:20:35 smithi115 conmon[90393]: 2022-05-14T01:20:35.279+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:20:36.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:20:35 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:20:36.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:20:35 smithi047 ceph-mon[77147]: pgmap v3737: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:20:36.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:20:35 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:20:36.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:20:35 smithi115 ceph-mon[78593]: pgmap v3737: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:20:36.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:20:36 smithi115 conmon[90393]: 2022-05-14T01:20:36.249+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:20:37.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:20:36 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 6770 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:20:37.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:20:36 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:20:37.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:20:36 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 6770 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:20:37.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:20:36 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:20:37.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:20:37 smithi115 conmon[90393]: 2022-05-14T01:20:37.297+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:20:38.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:20:37 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:20:38.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:20:37 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:20:38.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:20:37 smithi047 ceph-mon[77147]: pgmap v3738: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:20:38.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:20:37 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:20:38.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:20:37 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:20:38.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:20:37 smithi115 ceph-mon[78593]: pgmap v3738: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:20:38.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:20:38 smithi115 conmon[90393]: 2022-05-14T01:20:38.345+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:20:39.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:20:38 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:20:39.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:20:38 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:20:39.709 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:20:39 smithi115 conmon[90393]: 2022-05-14T01:20:39.384+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:20:40.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:20:39 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:20:40.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:20:39 smithi047 ceph-mon[77147]: pgmap v3739: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:20:40.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:20:39 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:20:40.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:20:39 smithi115 ceph-mon[78593]: pgmap v3739: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:20:40.715 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:20:40 smithi115 conmon[90393]: 2022-05-14T01:20:40.357+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:20:41.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:20:40 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:20:41.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:20:40 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:20:41.720 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:20:41 smithi115 conmon[90393]: 2022-05-14T01:20:41.360+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:20:42.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:20:41 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:20:42.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:20:41 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:20:42.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:20:41 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:20:42.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:20:41 smithi047 ceph-mon[77147]: pgmap v3740: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:20:42.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:20:41 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:20:42.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:20:41 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:20:42.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:20:41 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:20:42.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:20:41 smithi115 ceph-mon[78593]: pgmap v3740: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:20:42.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:20:42 smithi115 conmon[90393]: 2022-05-14T01:20:42.321+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:20:43.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:20:42 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:20:43.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:20:42 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:20:43.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:20:43 smithi115 conmon[90393]: 2022-05-14T01:20:43.324+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:20:44.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:20:43 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:20:44.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:20:43 smithi047 ceph-mon[77147]: pgmap v3741: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:20:44.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:20:43 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:20:44.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:20:43 smithi115 ceph-mon[78593]: pgmap v3741: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:20:44.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:20:44 smithi115 conmon[90393]: 2022-05-14T01:20:44.303+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:20:45.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:20:44 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:20:45.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:20:44 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 6775 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:20:45.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:20:44 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:20:45.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:20:44 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 6775 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:20:45.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:20:45 smithi115 conmon[90393]: 2022-05-14T01:20:45.294+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:20:46.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:20:45 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:20:46.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:20:45 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:20:46.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:20:45 smithi047 ceph-mon[77147]: pgmap v3742: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:20:46.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:20:45 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:20:46.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:20:45 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:20:46.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:20:45 smithi115 ceph-mon[78593]: pgmap v3742: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:20:46.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:20:46 smithi115 conmon[90393]: 2022-05-14T01:20:46.333+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:20:47.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:20:47 smithi115 conmon[90393]: 2022-05-14T01:20:47.328+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:20:47.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:20:47 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:20:47.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:20:47 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:20:48.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:20:48 smithi115 conmon[90393]: 2022-05-14T01:20:48.328+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:20:48.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:20:48 smithi115 ceph-mon[78593]: pgmap v3743: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:20:48.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:20:48 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:20:48.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:20:48 smithi047 ceph-mon[77147]: pgmap v3743: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:20:48.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:20:48 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:20:49.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:20:49 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:20:49.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:20:49 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:20:49.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:20:49 smithi115 conmon[90393]: 2022-05-14T01:20:49.369+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:20:50.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:20:50 smithi115 conmon[90393]: 2022-05-14T01:20:50.320+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:20:50.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:20:50 smithi115 ceph-mon[78593]: pgmap v3744: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:20:50.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:20:50 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:20:50.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:20:50 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:20:50.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:20:50 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:20:50.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:20:50 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 6785 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:20:50.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:20:50 smithi047 ceph-mon[77147]: pgmap v3744: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:20:50.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:20:50 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:20:50.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:20:50 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:20:50.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:20:50 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:20:50.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:20:50 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 6785 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:20:51.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:20:51 smithi115 conmon[90393]: 2022-05-14T01:20:51.306+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:20:51.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:20:51 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:20:51.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:20:51 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:20:52.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:20:52 smithi115 conmon[90393]: 2022-05-14T01:20:52.317+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:20:52.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:20:52 smithi115 ceph-mon[78593]: pgmap v3745: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:20:52.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:20:52 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:20:52.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:20:52 smithi047 ceph-mon[77147]: pgmap v3745: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:20:52.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:20:52 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:20:53.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:20:53 smithi115 conmon[90393]: 2022-05-14T01:20:53.296+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:20:53.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:20:53 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:20:53.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:20:53 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:20:53.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:20:53 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:20:53.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:20:53 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:20:54.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:20:54 smithi115 conmon[90393]: 2022-05-14T01:20:54.342+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:20:54.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:20:54 smithi115 ceph-mon[78593]: pgmap v3746: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:20:54.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:20:54 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:20:54.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:20:54 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:20:54.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:20:54 smithi047 ceph-mon[77147]: pgmap v3746: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:20:54.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:20:54 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:20:54.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:20:54 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:20:55.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:20:55 smithi115 conmon[90393]: 2022-05-14T01:20:55.336+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:20:56.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:20:55 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:20:56.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:20:55 smithi047 ceph-mon[77147]: pgmap v3747: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:20:56.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:20:55 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:20:56.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:20:55 smithi115 ceph-mon[78593]: pgmap v3747: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:20:56.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:20:56 smithi115 conmon[90393]: 2022-05-14T01:20:56.303+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:20:57.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:20:56 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:20:57.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:20:56 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 6790 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:20:57.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:20:56 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:20:57.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:20:56 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 6790 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:20:57.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:20:57 smithi115 conmon[90393]: 2022-05-14T01:20:57.295+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:20:58.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:20:57 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:20:58.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:20:57 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:20:58.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:20:57 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:20:58.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:20:57 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:20:58.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:20:57 smithi047 ceph-mon[77147]: pgmap v3748: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:20:58.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:20:57 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:20:58.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:20:57 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:20:58.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:20:57 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:20:58.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:20:57 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:20:58.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:20:57 smithi115 ceph-mon[78593]: pgmap v3748: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:20:58.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:20:58 smithi115 conmon[90393]: 2022-05-14T01:20:58.339+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:20:59.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:20:59 smithi115 conmon[90393]: 2022-05-14T01:20:59.318+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:20:59.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:20:59 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:20:59.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:20:59 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:21:00.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:21:00 smithi115 conmon[90393]: 2022-05-14T01:21:00.322+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:21:00.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:21:00 smithi115 ceph-mon[78593]: pgmap v3749: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:21:00.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:21:00 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:21:00.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:21:00 smithi047 ceph-mon[77147]: pgmap v3749: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:21:00.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:21:00 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:21:01.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:21:01 smithi115 conmon[90393]: 2022-05-14T01:21:01.318+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:21:01.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:21:01 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:21:01.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:21:01 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:21:02.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:21:02 smithi115 conmon[90393]: 2022-05-14T01:21:02.287+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:21:02.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:21:02 smithi115 ceph-mon[78593]: pgmap v3750: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:21:02.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:21:02 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:21:02.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:21:02 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:21:02.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:21:02 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:21:02.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:21:02 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:21:02.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:21:02 smithi047 ceph-mon[77147]: pgmap v3750: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:21:02.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:21:02 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:21:02.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:21:02 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:21:02.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:21:02 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:21:02.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:21:02 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:21:03.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:21:03 smithi115 conmon[90393]: 2022-05-14T01:21:03.268+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:21:03.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:21:03 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:21:03.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:21:03 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:21:04.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:21:04 smithi115 conmon[90393]: 2022-05-14T01:21:04.306+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:21:04.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:21:04 smithi115 ceph-mon[78593]: pgmap v3751: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:21:04.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:21:04 smithi047 ceph-mon[77147]: pgmap v3751: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:21:05.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:21:05 smithi115 conmon[90393]: 2022-05-14T01:21:05.294+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:21:05.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:21:05 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:21:05.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:21:05 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 6795 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:21:05.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:21:05 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:21:05.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:21:05 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:21:05.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:21:05 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 6795 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:21:05.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:21:05 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:21:06.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:21:06 smithi115 conmon[90393]: 2022-05-14T01:21:06.319+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:21:06.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:21:06 smithi115 ceph-mon[78593]: pgmap v3752: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:21:06.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:21:06 smithi047 ceph-mon[77147]: pgmap v3752: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:21:07.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:21:07 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:21:07.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:21:07 smithi115 conmon[90393]: 2022-05-14T01:21:07.368+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:21:07.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:21:07 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:21:08.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:21:08 smithi047 ceph-mon[77147]: pgmap v3753: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:21:08.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:21:08 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:21:08.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:21:08 smithi115 conmon[90393]: 2022-05-14T01:21:08.409+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:21:08.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:21:08 smithi115 ceph-mon[78593]: pgmap v3753: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:21:08.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:21:08 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:21:09.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:21:09 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:21:09.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:21:09 smithi115 conmon[90393]: 2022-05-14T01:21:09.371+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:21:09.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:21:09 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:21:10.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:21:10 smithi047 ceph-mon[77147]: pgmap v3754: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:21:10.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:21:10 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:21:10.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:21:10 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:21:10.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:21:10 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:21:10.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:21:10 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 6805 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:21:10.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:21:10 smithi115 conmon[90393]: 2022-05-14T01:21:10.374+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:21:10.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:21:10 smithi115 ceph-mon[78593]: pgmap v3754: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:21:10.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:21:10 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:21:10.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:21:10 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:21:10.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:21:10 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:21:10.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:21:10 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 6805 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:21:11.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:21:11 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:21:11.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:21:11 smithi115 conmon[90393]: 2022-05-14T01:21:11.405+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:21:11.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:21:11 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:21:12.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:21:12 smithi047 ceph-mon[77147]: pgmap v3755: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:21:12.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:21:12 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:21:12.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:21:12 smithi115 conmon[90393]: 2022-05-14T01:21:12.365+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:21:12.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:21:12 smithi115 ceph-mon[78593]: pgmap v3755: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:21:12.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:21:12 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:21:13.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:21:13 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:21:13.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:21:13 smithi115 conmon[90393]: 2022-05-14T01:21:13.394+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:21:13.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:21:13 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:21:14.624 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:21:14 smithi115 conmon[90393]: 2022-05-14T01:21:14.361+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:21:14.625 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:21:14 smithi115 ceph-mon[78593]: pgmap v3756: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:21:14.625 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:21:14 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:21:14.625 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:21:14 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:21:14.626 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:21:14 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:21:14.626 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:21:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T01:21:14.626 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:21:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T01:21:14.654 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:21:14 smithi047 ceph-mon[77147]: pgmap v3756: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:21:14.654 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:21:14 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:21:14.654 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:21:14 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:21:14.655 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:21:14 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:21:14.655 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:21:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T01:21:14.655 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:21:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T01:21:15.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:21:15 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:21:15.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:21:15 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T01:21:15.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:21:15 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T01:21:15.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:21:15 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T01:21:15.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:21:15 smithi115 conmon[90393]: 2022-05-14T01:21:15.405+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:21:15.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:21:15 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:21:15.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:21:15 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T01:21:15.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:21:15 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T01:21:15.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:21:15 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T01:21:16.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:21:16 smithi047 ceph-mon[77147]: pgmap v3757: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:21:16.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:21:16 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:21:16.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:21:16 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 6810 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:21:16.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:21:16 smithi115 conmon[90393]: 2022-05-14T01:21:16.360+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:21:16.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:21:16 smithi115 ceph-mon[78593]: pgmap v3757: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:21:16.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:21:16 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:21:16.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:21:16 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 6810 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:21:17.682 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:21:17 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:21:17.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:21:17 smithi115 conmon[90393]: 2022-05-14T01:21:17.390+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:21:17.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:21:17 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:21:18.041 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:21:17 smithi047 conmon[78521]: 2022-05-14T01:21:17.680+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:21:18.041 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:21:17 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T01:21:18.042 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:21:17 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:21:18.042 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:21:17 smithi047 conmon[78521]: yield 2022-05-14T01:21:18.042 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:21:17 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:21:18.042 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:21:17 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:21:18.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:21:17 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:21:18.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:21:17 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T01:21:18.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:21:17 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:21:18.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:21:17 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T01:21:18.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:21:17 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:21:18.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:21:17 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T01:21:18.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:21:17 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:21:18.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:21:17 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T01:21:18.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:21:17 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:21:18.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:21:17 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T01:21:18.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:21:17 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:21:18.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:21:17 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:21:18.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:21:17 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:21:18.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:21:18 smithi047 ceph-mon[77147]: pgmap v3758: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:21:18.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:21:18 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:21:18.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:21:18 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:21:18.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:21:18 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:21:18.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:21:18 smithi115 conmon[90393]: 2022-05-14T01:21:18.409+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:21:18.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:21:18 smithi115 ceph-mon[78593]: pgmap v3758: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:21:18.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:21:18 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:21:18.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:21:18 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:21:18.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:21:18 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:21:19.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:21:19 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:21:19.791 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T01:21:19.792 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:21:19.792 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T01:21:19.792 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:21:19.793 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:21:19.793 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:21:19.793 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T01:21:19.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:21:19.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T01:21:19.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:21:19.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T01:21:19.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:21:19.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T01:21:19.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:21:19.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T01:21:19.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:21:19.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:21:19.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:21:19.796 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:21:19 smithi047 ceph-mon[77147]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T01:21:19.796 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:21:19 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:21:19.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:21:19 smithi115 conmon[90393]: 2022-05-14T01:21:19.388+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:21:19.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:21:19 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:21:19.858 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T01:21:19.858 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:21:19.859 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T01:21:19.859 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:21:19.859 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:21:19.859 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:21:19.859 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T01:21:19.860 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:21:19.860 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T01:21:19.860 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:21:19.860 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T01:21:19.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:21:19.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T01:21:19.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:21:19.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T01:21:19.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:21:19.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:21:19.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:21:19.862 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:21:19 smithi115 ceph-mon[78593]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T01:21:19.863 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:21:19 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:21:20.754 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:21:20 smithi047 ceph-mon[77147]: pgmap v3759: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:21:20.754 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:21:20 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:21:20.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:21:20 smithi115 conmon[90393]: 2022-05-14T01:21:20.357+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:21:20.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:21:20 smithi115 ceph-mon[78593]: pgmap v3759: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:21:20.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:21:20 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:21:21.041 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:21:20 smithi047 conmon[78521]: 2022-05-14T01:21:20.753+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:21:21.041 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:21:20 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T01:21:21.041 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:21:20 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:21:21.042 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:21:20 smithi047 conmon[78521]: yield 2022-05-14T01:21:21.042 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:21:20 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:21:21.042 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:21:20 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:21:21.042 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:21:20 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:21:21.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:21:20 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T01:21:21.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:21:20 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:21:21.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:21:20 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T01:21:21.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:21:20 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:21:21.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:21:20 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T01:21:21.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:21:20 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:21:21.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:21:20 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T01:21:21.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:21:20 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:21:21.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:21:20 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T01:21:21.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:21:20 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:21:21.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:21:20 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:21:21.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:21:20 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:21:21.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:21:20 smithi047 conmon[78521]: 2022-05-14T01:21:20.754+0000 7f5344781700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T01:21:21.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:21:20 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T01:21:21.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:21:20 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:21:21.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:21:20 smithi047 conmon[78521]: yield 2022-05-14T01:21:21.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:21:20 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:21:21.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:21:20 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:21:21.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:21:20 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:21:21.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:21:20 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T01:21:21.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:21:20 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:21:21.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:21:20 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T01:21:21.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:21:20 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:21:21.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:21:20 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T01:21:21.048 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:21:20 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:21:21.048 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:21:20 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T01:21:21.048 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:21:20 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:21:21.048 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:21:20 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T01:21:21.048 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:21:20 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:21:21.049 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:21:20 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:21:21.049 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:21:20 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:21:21.049 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:21:20 smithi047 conmon[78521]: 2022-05-14T01:21:21.049 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:21:20 smithi047 conmon[78521]: During handling of the above exception, another exception occurred: 2022-05-14T01:21:21.050 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:21:20 smithi047 conmon[78521]: 2022-05-14T01:21:21.050 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:21:20 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T01:21:21.050 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:21:20 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T01:21:21.050 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:21:20 smithi047 conmon[78521]: return f(*arg) 2022-05-14T01:21:21.050 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:21:20 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T01:21:21.050 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:21:20 smithi047 conmon[78521]: self._write_client_files(client_files, host) 2022-05-14T01:21:21.051 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:21:20 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T01:21:21.051 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:21:20 smithi047 conmon[78521]: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T01:21:21.051 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:21:20 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T01:21:21.051 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:21:20 smithi047 conmon[78521]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T01:21:21.051 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:21:20 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T01:21:21.052 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:21:20 smithi047 conmon[78521]: return self.event_loop.get_result(coro) 2022-05-14T01:21:21.052 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:21:20 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T01:21:21.052 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:21:20 smithi047 conmon[78521]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T01:21:21.052 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:21:20 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T01:21:21.052 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:21:20 smithi047 conmon[78521]: return self.__get_result() 2022-05-14T01:21:21.053 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:21:20 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T01:21:21.053 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:21:20 smithi047 conmon[78521]: raise self._exception 2022-05-14T01:21:21.053 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:21:20 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T01:21:21.053 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:21:20 smithi047 conmon[78521]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T01:21:21.053 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:21:20 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T01:21:21.054 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:21:20 smithi047 conmon[78521]: conn = await self._remote_connection(host, addr) 2022-05-14T01:21:21.054 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:21:20 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T01:21:21.054 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:21:20 smithi047 conmon[78521]: raise 2022-05-14T01:21:21.054 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:21:20 smithi047 conmon[78521]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T01:21:21.054 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:21:20 smithi047 conmon[78521]: self.gen.throw(type, value, traceback) 2022-05-14T01:21:21.055 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:21:20 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T01:21:21.055 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:21:20 smithi047 conmon[78521]: raise OrchestratorError(msg) 2022-05-14T01:21:21.055 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:21:20 smithi047 conmon[78521]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:21:21.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:21:21 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:21:21.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:21:21 smithi115 conmon[90393]: 2022-05-14T01:21:21.377+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:21:21.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:21:21 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:21:22.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:21:22 smithi115 conmon[90393]: 2022-05-14T01:21:22.335+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:21:22.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:21:22 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:21:22.608 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T01:21:22.608 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:21:22.609 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T01:21:22.609 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:21:22.609 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:21:22.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:21:22.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T01:21:22.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:21:22.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T01:21:22.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:21:22.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T01:21:22.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:21:22.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T01:21:22.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:21:22.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T01:21:22.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:21:22.613 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:21:22.613 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:21:22.613 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:21:22 smithi115 ceph-mon[78593]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T01:21:22.613 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T01:21:22.614 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:21:22.614 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T01:21:22.614 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:21:22.614 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:21:22.615 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:21:22.615 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T01:21:22.615 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:21:22.615 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T01:21:22.616 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:21:22.616 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T01:21:22.616 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:21:22.616 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T01:21:22.616 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:21:22.617 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T01:21:22.617 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:21:22.618 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:21:22.618 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:21:22.618 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T01:21:22.619 INFO:journalctl@ceph.mon.c.smithi115.stdout: During handling of the above exception, another exception occurred: 2022-05-14T01:21:22.619 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T01:21:22.620 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T01:21:22.620 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T01:21:22.620 INFO:journalctl@ceph.mon.c.smithi115.stdout: return f(*arg) 2022-05-14T01:21:22.620 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T01:21:22.621 INFO:journalctl@ceph.mon.c.smithi115.stdout: self._write_client_files(client_files, host) 2022-05-14T01:21:22.621 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T01:21:22.621 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T01:21:22.622 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T01:21:22.622 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T01:21:22.622 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T01:21:22.622 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.event_loop.get_result(coro) 2022-05-14T01:21:22.623 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T01:21:22.623 INFO:journalctl@ceph.mon.c.smithi115.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T01:21:22.623 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T01:21:22.623 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.__get_result() 2022-05-14T01:21:22.624 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T01:21:22.624 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise self._exception 2022-05-14T01:21:22.624 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T01:21:22.624 INFO:journalctl@ceph.mon.c.smithi115.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T01:21:22.624 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T01:21:22.625 INFO:journalctl@ceph.mon.c.smithi115.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T01:21:22.625 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T01:21:22.625 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise 2022-05-14T01:21:22.625 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T01:21:22.626 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.gen.throw(type, value, traceback) 2022-05-14T01:21:22.626 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T01:21:22.626 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OrchestratorError(msg) 2022-05-14T01:21:22.626 INFO:journalctl@ceph.mon.c.smithi115.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:21:22.626 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:21:22 smithi115 ceph-mon[78593]: pgmap v3760: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:21:22.627 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:21:22 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:21:22.627 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:21:22 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:21:22.627 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:21:22 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:21:22.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:21:22 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:21:22.791 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T01:21:22.792 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:21:22.792 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T01:21:22.792 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:21:22.792 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:21:22.792 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:21:22.793 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T01:21:22.793 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:21:22.793 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T01:21:22.793 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:21:22.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T01:21:22.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:21:22.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T01:21:22.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:21:22.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T01:21:22.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:21:22.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:21:22.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:21:22.795 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:21:22 smithi047 ceph-mon[77147]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T01:21:22.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T01:21:22.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:21:22.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T01:21:22.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:21:22.797 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:21:22.797 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:21:22.797 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T01:21:22.797 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:21:22.798 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T01:21:22.798 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:21:22.798 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T01:21:22.798 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:21:22.799 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T01:21:22.799 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:21:22.799 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T01:21:22.799 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:21:22.799 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:21:22.800 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:21:22.800 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T01:21:22.800 INFO:journalctl@ceph.mon.b.smithi047.stdout: During handling of the above exception, another exception occurred: 2022-05-14T01:21:22.800 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T01:21:22.801 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T01:21:22.801 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T01:21:22.801 INFO:journalctl@ceph.mon.b.smithi047.stdout: return f(*arg) 2022-05-14T01:21:22.801 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T01:21:22.802 INFO:journalctl@ceph.mon.b.smithi047.stdout: self._write_client_files(client_files, host) 2022-05-14T01:21:22.802 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T01:21:22.802 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T01:21:22.802 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T01:21:22.802 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T01:21:22.803 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T01:21:22.803 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.event_loop.get_result(coro) 2022-05-14T01:21:22.803 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T01:21:22.803 INFO:journalctl@ceph.mon.b.smithi047.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T01:21:22.804 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T01:21:22.804 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.__get_result() 2022-05-14T01:21:22.804 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T01:21:22.804 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise self._exception 2022-05-14T01:21:22.804 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T01:21:22.805 INFO:journalctl@ceph.mon.b.smithi047.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T01:21:22.805 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T01:21:22.805 INFO:journalctl@ceph.mon.b.smithi047.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T01:21:22.805 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T01:21:22.806 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise 2022-05-14T01:21:22.806 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T01:21:22.806 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.gen.throw(type, value, traceback) 2022-05-14T01:21:22.806 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T01:21:22.806 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OrchestratorError(msg) 2022-05-14T01:21:22.807 INFO:journalctl@ceph.mon.b.smithi047.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:21:22.807 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:21:22 smithi047 ceph-mon[77147]: pgmap v3760: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:21:22.807 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:21:22 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:21:22.807 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:21:22 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:21:22.808 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:21:22 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:21:23.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:21:23 smithi115 conmon[90393]: 2022-05-14T01:21:23.344+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:21:23.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:21:23 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:21:23.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:21:23 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:21:24.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:21:24 smithi047 ceph-mon[77147]: pgmap v3761: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:21:24.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:21:24 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:21:24.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:21:24 smithi115 conmon[90393]: 2022-05-14T01:21:24.374+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:21:24.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:21:24 smithi115 ceph-mon[78593]: pgmap v3761: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:21:24.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:21:24 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:21:25.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:21:25 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:21:25.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:21:25 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 6815 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:21:25.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:21:25 smithi115 conmon[90393]: 2022-05-14T01:21:25.359+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:21:25.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:21:25 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:21:25.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:21:25 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 6815 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:21:26.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:21:26 smithi115 conmon[90393]: 2022-05-14T01:21:26.310+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:21:26.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:21:26 smithi115 ceph-mon[78593]: pgmap v3762: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:21:26.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:21:26 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:21:26.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:21:26 smithi047 ceph-mon[77147]: pgmap v3762: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:21:26.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:21:26 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:21:27.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:21:27 smithi115 conmon[90393]: 2022-05-14T01:21:27.263+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:21:27.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:21:27 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:21:27.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:21:27 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:21:27.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:21:27 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:21:27.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:21:27 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:21:28.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:21:28 smithi115 conmon[90393]: 2022-05-14T01:21:28.232+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:21:28.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:21:28 smithi115 ceph-mon[78593]: pgmap v3763: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:21:28.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:21:28 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:21:28.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:21:28 smithi047 ceph-mon[77147]: pgmap v3763: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:21:28.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:21:28 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:21:29.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:21:29 smithi115 conmon[90393]: 2022-05-14T01:21:29.267+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:21:29.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:21:29 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:21:29.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:21:29 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:21:30.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:21:30 smithi115 conmon[90393]: 2022-05-14T01:21:30.231+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:21:30.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:21:30 smithi115 ceph-mon[78593]: pgmap v3764: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:21:30.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:21:30 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:21:30.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:21:30 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:21:30.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:21:30 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 6825 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:21:30.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:21:30 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:21:30.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:21:30 smithi047 ceph-mon[77147]: pgmap v3764: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:21:30.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:21:30 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:21:30.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:21:30 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:21:30.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:21:30 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 6825 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:21:30.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:21:30 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:21:31.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:21:31 smithi115 conmon[90393]: 2022-05-14T01:21:31.254+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:21:31.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:21:31 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:21:31.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:21:31 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:21:32.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:21:32 smithi115 conmon[90393]: 2022-05-14T01:21:32.243+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:21:32.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:21:32 smithi115 ceph-mon[78593]: pgmap v3765: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:21:32.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:21:32 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:21:32.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:21:32 smithi047 ceph-mon[77147]: pgmap v3765: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:21:32.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:21:32 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:21:33.462 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:21:33 smithi115 conmon[90393]: 2022-05-14T01:21:33.197+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:21:33.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:21:33 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:21:33.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:21:33 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:21:34.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:21:34 smithi115 conmon[90393]: 2022-05-14T01:21:34.222+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:21:34.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:21:34 smithi115 ceph-mon[78593]: pgmap v3766: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:21:34.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:21:34 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:21:34.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:21:34 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:21:34.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:21:34 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:21:34.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:21:34 smithi047 ceph-mon[77147]: pgmap v3766: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:21:34.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:21:34 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:21:34.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:21:34 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:21:34.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:21:34 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:21:35.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:21:35 smithi115 conmon[90393]: 2022-05-14T01:21:35.263+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:21:36.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:21:35 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:21:36.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:21:35 smithi047 ceph-mon[77147]: pgmap v3767: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:21:36.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:21:35 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:21:36.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:21:35 smithi115 ceph-mon[78593]: pgmap v3767: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:21:36.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:21:36 smithi115 conmon[90393]: 2022-05-14T01:21:36.297+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:21:37.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:21:36 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 6830 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:21:37.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:21:36 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:21:37.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:21:36 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 6830 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:21:37.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:21:36 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:21:37.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:21:37 smithi115 conmon[90393]: 2022-05-14T01:21:37.332+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:21:38.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:21:37 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:21:38.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:21:37 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:21:38.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:21:37 smithi047 ceph-mon[77147]: pgmap v3768: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:21:38.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:21:37 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:21:38.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:21:37 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:21:38.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:21:37 smithi115 ceph-mon[78593]: pgmap v3768: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:21:38.720 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:21:38 smithi115 conmon[90393]: 2022-05-14T01:21:38.380+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:21:39.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:21:38 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:21:39.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:21:38 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:21:39.728 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:21:39 smithi115 conmon[90393]: 2022-05-14T01:21:39.402+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:21:40.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:21:39 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:21:40.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:21:39 smithi047 ceph-mon[77147]: pgmap v3769: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:21:40.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:21:39 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:21:40.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:21:39 smithi115 ceph-mon[78593]: pgmap v3769: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:21:40.729 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:21:40 smithi115 conmon[90393]: 2022-05-14T01:21:40.412+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:21:41.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:21:40 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:21:41.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:21:40 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:21:41.741 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:21:41 smithi115 conmon[90393]: 2022-05-14T01:21:41.422+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:21:42.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:21:41 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:21:42.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:21:41 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:21:42.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:21:41 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:21:42.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:21:41 smithi047 ceph-mon[77147]: pgmap v3770: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:21:42.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:21:41 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:21:42.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:21:41 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:21:42.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:21:41 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:21:42.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:21:41 smithi115 ceph-mon[78593]: pgmap v3770: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:21:42.742 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:21:42 smithi115 conmon[90393]: 2022-05-14T01:21:42.399+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:21:43.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:21:42 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:21:43.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:21:42 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:21:43.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:21:43 smithi115 conmon[90393]: 2022-05-14T01:21:43.353+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:21:44.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:21:43 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:21:44.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:21:43 smithi047 ceph-mon[77147]: pgmap v3771: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:21:44.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:21:43 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:21:44.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:21:43 smithi115 ceph-mon[78593]: pgmap v3771: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:21:44.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:21:44 smithi115 conmon[90393]: 2022-05-14T01:21:44.355+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:21:45.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:21:44 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:21:45.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:21:44 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 6835 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:21:45.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:21:44 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:21:45.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:21:44 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 6835 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:21:45.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:21:45 smithi115 conmon[90393]: 2022-05-14T01:21:45.341+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:21:46.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:21:45 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:21:46.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:21:45 smithi047 ceph-mon[77147]: pgmap v3772: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:21:46.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:21:45 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:21:46.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:21:45 smithi115 ceph-mon[78593]: pgmap v3772: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:21:46.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:21:46 smithi115 conmon[90393]: 2022-05-14T01:21:46.318+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:21:47.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:21:46 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:21:47.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:21:46 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:21:47.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:21:47 smithi115 conmon[90393]: 2022-05-14T01:21:47.333+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:21:48.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:21:47 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:21:48.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:21:47 smithi047 ceph-mon[77147]: pgmap v3773: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:21:48.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:21:47 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:21:48.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:21:47 smithi115 ceph-mon[78593]: pgmap v3773: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:21:48.765 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:21:48 smithi115 conmon[90393]: 2022-05-14T01:21:48.376+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:21:49.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:21:48 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:21:49.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:21:48 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:21:49.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:21:49 smithi115 conmon[90393]: 2022-05-14T01:21:49.326+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:21:50.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:21:49 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:21:50.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:21:49 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:21:50.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:21:49 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:21:50.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:21:49 smithi047 ceph-mon[77147]: pgmap v3774: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:21:50.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:21:49 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 6840 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:21:50.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:21:49 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:21:50.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:21:49 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:21:50.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:21:49 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:21:50.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:21:49 smithi115 ceph-mon[78593]: pgmap v3774: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:21:50.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:21:49 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 6840 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:21:50.771 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:21:50 smithi115 conmon[90393]: 2022-05-14T01:21:50.362+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:21:51.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:21:50 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:21:51.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:21:50 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:21:51.776 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:21:51 smithi115 conmon[90393]: 2022-05-14T01:21:51.401+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:21:52.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:21:51 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:21:52.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:21:51 smithi047 ceph-mon[77147]: pgmap v3775: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:21:52.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:21:51 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:21:52.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:21:51 smithi115 ceph-mon[78593]: pgmap v3775: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:21:52.778 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:21:52 smithi115 conmon[90393]: 2022-05-14T01:21:52.410+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:21:53.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:21:52 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:21:53.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:21:52 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:21:53.788 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:21:53 smithi115 conmon[90393]: 2022-05-14T01:21:53.441+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:21:54.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:21:53 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:21:54.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:21:53 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:21:54.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:21:53 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:21:54.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:21:53 smithi047 ceph-mon[77147]: pgmap v3776: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:21:54.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:21:53 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:21:54.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:21:53 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:21:54.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:21:53 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:21:54.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:21:53 smithi115 ceph-mon[78593]: pgmap v3776: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:21:54.788 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:21:54 smithi115 conmon[90393]: 2022-05-14T01:21:54.482+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:21:55.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:21:54 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:21:55.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:21:54 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 6845 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:21:55.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:21:54 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:21:55.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:21:54 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 6845 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:21:55.793 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:21:55 smithi115 conmon[90393]: 2022-05-14T01:21:55.465+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:21:56.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:21:55 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:21:56.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:21:55 smithi115 ceph-mon[78593]: pgmap v3777: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:21:56.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:21:55 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:21:56.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:21:55 smithi047 ceph-mon[77147]: pgmap v3777: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:21:56.799 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:21:56 smithi115 conmon[90393]: 2022-05-14T01:21:56.469+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:21:57.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:21:56 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:21:57.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:21:56 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:21:57.807 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:21:57 smithi115 conmon[90393]: 2022-05-14T01:21:57.493+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:21:58.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:21:57 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:21:58.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:21:57 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:21:58.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:21:57 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:21:58.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:21:57 smithi115 ceph-mon[78593]: pgmap v3778: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:21:58.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:21:57 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:21:58.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:21:57 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:21:58.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:21:57 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:21:58.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:21:57 smithi047 ceph-mon[77147]: pgmap v3778: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:21:58.806 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:21:58 smithi115 conmon[90393]: 2022-05-14T01:21:58.473+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:21:59.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:21:58 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:21:59.814 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:21:59 smithi115 conmon[90393]: 2022-05-14T01:21:59.477+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:21:59.815 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:21:58 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:22:00.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:21:59 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:22:00.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:21:59 smithi115 ceph-mon[78593]: pgmap v3779: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:22:00.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:21:59 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 6850 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:22:00.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:21:59 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:22:00.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:21:59 smithi047 ceph-mon[77147]: pgmap v3779: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:22:00.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:21:59 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 6850 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:22:00.821 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:22:00 smithi115 conmon[90393]: 2022-05-14T01:22:00.436+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:22:01.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:22:00 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:22:01.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:22:00 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:22:01.825 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:22:01 smithi115 conmon[90393]: 2022-05-14T01:22:01.448+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:22:02.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:22:01 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:22:02.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:22:01 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:22:02.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:22:01 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:22:02.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:22:01 smithi115 ceph-mon[78593]: pgmap v3780: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:22:02.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:22:01 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:22:02.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:22:01 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:22:02.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:22:01 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:22:02.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:22:01 smithi047 ceph-mon[77147]: pgmap v3780: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:22:02.825 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:22:02 smithi115 conmon[90393]: 2022-05-14T01:22:02.420+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:22:03.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:22:02 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:22:03.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:22:02 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:22:03.830 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:22:03 smithi115 conmon[90393]: 2022-05-14T01:22:03.379+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:22:04.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:22:03 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:22:04.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:22:03 smithi115 ceph-mon[78593]: pgmap v3781: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:22:04.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:22:03 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:22:04.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:22:03 smithi047 ceph-mon[77147]: pgmap v3781: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:22:04.831 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:22:04 smithi115 conmon[90393]: 2022-05-14T01:22:04.384+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:22:05.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:22:04 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:22:05.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:22:04 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 6855 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:22:05.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:22:04 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:22:05.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:22:04 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 6855 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:22:05.837 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:22:05 smithi115 conmon[90393]: 2022-05-14T01:22:05.387+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:22:06.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:22:05 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:22:06.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:22:05 smithi115 ceph-mon[78593]: pgmap v3782: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:22:06.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:22:05 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:22:06.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:22:05 smithi047 ceph-mon[77147]: pgmap v3782: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:22:06.838 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:22:06 smithi115 conmon[90393]: 2022-05-14T01:22:06.368+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:22:07.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:22:06 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:22:07.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:22:06 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:22:07.844 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:22:07 smithi115 conmon[90393]: 2022-05-14T01:22:07.372+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:22:08.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:22:07 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:22:08.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:22:07 smithi115 ceph-mon[78593]: pgmap v3783: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:22:08.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:22:07 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:22:08.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:22:07 smithi047 ceph-mon[77147]: pgmap v3783: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:22:08.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:22:08 smithi115 conmon[90393]: 2022-05-14T01:22:08.352+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:22:09.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:22:08 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:22:09.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:22:08 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:22:09.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:22:09 smithi115 conmon[90393]: 2022-05-14T01:22:09.306+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:22:10.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:22:09 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:22:10.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:22:09 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:22:10.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:22:09 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:22:10.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:22:09 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:22:10.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:22:09 smithi115 ceph-mon[78593]: pgmap v3784: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:22:10.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:22:09 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 6860 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:22:10.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:22:09 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:22:10.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:22:09 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:22:10.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:22:09 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:22:10.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:22:09 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:22:10.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:22:09 smithi047 ceph-mon[77147]: pgmap v3784: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:22:10.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:22:09 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 6860 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:22:10.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:22:10 smithi115 conmon[90393]: 2022-05-14T01:22:10.316+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:22:11.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:22:11 smithi115 conmon[90393]: 2022-05-14T01:22:11.292+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:22:11.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:22:11 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:22:11.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:22:11 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:22:11.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:22:11 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:22:11.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:22:11 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:22:12.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:22:12 smithi115 conmon[90393]: 2022-05-14T01:22:12.270+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:22:12.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:22:12 smithi115 ceph-mon[78593]: pgmap v3785: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:22:12.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:22:12 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:22:12.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:22:12 smithi047 ceph-mon[77147]: pgmap v3785: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:22:12.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:22:12 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:22:13.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:22:13 smithi115 conmon[90393]: 2022-05-14T01:22:13.280+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:22:13.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:22:13 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:22:13.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:22:13 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:22:14.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:22:14 smithi115 conmon[90393]: 2022-05-14T01:22:14.280+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:22:14.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:22:14 smithi115 ceph-mon[78593]: pgmap v3786: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:22:14.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:22:14 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:22:14.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:22:14 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:22:14.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:22:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T01:22:14.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:22:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T01:22:14.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:22:14 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:22:14.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:22:14 smithi047 ceph-mon[77147]: pgmap v3786: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:22:14.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:22:14 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:22:14.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:22:14 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:22:14.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:22:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T01:22:14.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:22:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T01:22:14.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:22:14 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:22:15.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:22:15 smithi115 conmon[90393]: 2022-05-14T01:22:15.284+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:22:15.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:22:15 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 6865 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:22:15.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:22:15 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:22:15.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:22:15 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 6865 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:22:15.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:22:15 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:22:16.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:22:16 smithi115 conmon[90393]: 2022-05-14T01:22:16.270+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:22:16.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:22:16 smithi115 ceph-mon[78593]: pgmap v3787: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:22:16.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:22:16 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:22:16.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:22:16 smithi047 ceph-mon[77147]: pgmap v3787: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:22:16.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:22:16 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:22:17.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:22:17 smithi115 conmon[90393]: 2022-05-14T01:22:17.301+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:22:17.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:22:17 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:22:17.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:22:17 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:22:18.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:22:18 smithi115 conmon[90393]: 2022-05-14T01:22:18.261+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:22:18.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:22:18 smithi115 ceph-mon[78593]: pgmap v3788: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:22:18.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:22:18 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:22:18.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:22:18 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:22:18.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:22:18 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:22:18.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:22:18 smithi047 ceph-mon[77147]: pgmap v3788: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:22:18.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:22:18 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:22:18.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:22:18 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:22:18.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:22:18 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:22:19.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:22:19 smithi115 conmon[90393]: 2022-05-14T01:22:19.250+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:22:19.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:22:19 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:22:19.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:22:19 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:22:20.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:22:20 smithi115 conmon[90393]: 2022-05-14T01:22:20.235+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:22:20.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:22:20 smithi115 ceph-mon[78593]: pgmap v3789: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:22:20.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:22:20 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 6875 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:22:20.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:22:20 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:22:20.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:22:20 smithi047 ceph-mon[77147]: pgmap v3789: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:22:20.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:22:20 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 6875 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:22:20.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:22:20 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:22:21.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:22:21 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T01:22:21.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:22:21 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T01:22:21.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:22:21 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T01:22:21.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:22:21 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:22:21.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:22:21 smithi115 conmon[90393]: 2022-05-14T01:22:21.203+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:22:21.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:22:21 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T01:22:21.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:22:21 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T01:22:21.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:22:21 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T01:22:21.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:22:21 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:22:22.411 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:22:22 smithi115 conmon[90393]: 2022-05-14T01:22:22.157+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:22:22.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:22:22 smithi047 ceph-mon[77147]: pgmap v3790: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:22:22.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:22:22 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:22:22.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:22:22 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:22:22.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:22:22 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:22:22.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:22:22 smithi115 ceph-mon[78593]: pgmap v3790: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:22:22.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:22:22 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:22:22.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:22:22 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:22:22.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:22:22 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:22:23.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:22:23 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:22:23.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:22:23 smithi115 conmon[90393]: 2022-05-14T01:22:23.172+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:22:23.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:22:23 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:22:24.291 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:22:23 smithi047 conmon[78521]: 2022-05-14T01:22:23.855+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:22:24.291 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:22:23 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T01:22:24.292 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:22:23 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:22:24.292 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:22:23 smithi047 conmon[78521]: yield 2022-05-14T01:22:24.292 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:22:23 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:22:24.292 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:22:23 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:22:24.292 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:22:23 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:22:24.293 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:22:23 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T01:22:24.293 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:22:23 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:22:24.293 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:22:23 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T01:22:24.293 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:22:23 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:22:24.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:22:23 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T01:22:24.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:22:23 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:22:24.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:22:23 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T01:22:24.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:22:23 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:22:24.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:22:23 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T01:22:24.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:22:23 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:22:24.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:22:23 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:22:24.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:22:23 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:22:24.417 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:22:24 smithi115 conmon[90393]: 2022-05-14T01:22:24.142+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:22:24.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:22:24 smithi047 ceph-mon[77147]: pgmap v3791: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:22:24.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:22:24 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:22:24.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:22:24 smithi115 ceph-mon[78593]: pgmap v3791: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:22:24.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:22:24 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:22:25.419 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:22:25 smithi115 conmon[90393]: 2022-05-14T01:22:25.126+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:22:25.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:22:25 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:22:25.791 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T01:22:25.791 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:22:25.792 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T01:22:25.792 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:22:25.792 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:22:25.792 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:22:25.792 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T01:22:25.793 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:22:25.793 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T01:22:25.793 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:22:25.793 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T01:22:25.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:22:25.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T01:22:25.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:22:25.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T01:22:25.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:22:25.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:22:25.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:22:25.795 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:22:25 smithi047 ceph-mon[77147]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T01:22:25.795 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:22:25 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:22:25.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:22:25 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:22:25.857 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T01:22:25.858 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:22:25.858 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T01:22:25.858 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:22:25.858 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:22:25.859 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:22:25.859 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T01:22:25.859 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:22:25.859 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T01:22:25.859 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:22:25.860 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T01:22:25.860 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:22:25.860 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T01:22:25.860 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:22:25.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T01:22:25.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:22:25.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:22:25.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:22:25.862 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:22:25 smithi115 ceph-mon[78593]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T01:22:25.862 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:22:25 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:22:26.420 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:22:26 smithi115 conmon[90393]: 2022-05-14T01:22:26.146+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:22:26.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:22:26 smithi047 ceph-mon[77147]: pgmap v3792: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:22:26.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:22:26 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 6880 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:22:26.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:22:26 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:22:26.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:22:26 smithi115 ceph-mon[78593]: pgmap v3792: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:22:26.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:22:26 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 6880 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:22:26.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:22:26 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:22:27.291 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:22:26 smithi047 conmon[78521]: 2022-05-14T01:22:26.927+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:22:27.291 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:22:26 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T01:22:27.292 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:22:26 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:22:27.292 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:22:26 smithi047 conmon[78521]: yield 2022-05-14T01:22:27.292 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:22:26 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:22:27.292 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:22:26 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:22:27.293 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:22:26 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:22:27.293 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:22:26 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T01:22:27.293 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:22:26 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:22:27.293 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:22:26 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T01:22:27.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:22:26 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:22:27.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:22:26 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T01:22:27.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:22:26 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:22:27.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:22:26 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T01:22:27.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:22:26 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:22:27.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:22:26 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T01:22:27.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:22:26 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:22:27.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:22:26 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:22:27.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:22:26 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:22:27.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:22:26 smithi047 conmon[78521]: 2022-05-14T01:22:26.929+0000 7f5345f84700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T01:22:27.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:22:26 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T01:22:27.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:22:26 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:22:27.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:22:26 smithi047 conmon[78521]: yield 2022-05-14T01:22:27.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:22:26 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:22:27.297 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:22:26 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:22:27.297 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:22:26 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:22:27.297 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:22:26 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T01:22:27.297 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:22:26 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:22:27.297 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:22:26 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T01:22:27.298 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:22:26 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:22:27.298 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:22:26 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T01:22:27.298 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:22:26 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:22:27.298 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:22:26 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T01:22:27.299 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:22:26 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:22:27.299 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:22:26 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T01:22:27.299 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:22:26 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:22:27.299 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:22:26 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:22:27.299 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:22:26 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:22:27.300 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:22:26 smithi047 conmon[78521]: 2022-05-14T01:22:27.300 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:22:26 smithi047 conmon[78521]: During handling of the above exception, another exception occurred: 2022-05-14T01:22:27.300 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:22:26 smithi047 conmon[78521]: 2022-05-14T01:22:27.300 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:22:26 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T01:22:27.300 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:22:26 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T01:22:27.301 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:22:26 smithi047 conmon[78521]: return f(*arg) 2022-05-14T01:22:27.301 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:22:26 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T01:22:27.301 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:22:26 smithi047 conmon[78521]: self._write_client_files(client_files, host) 2022-05-14T01:22:27.301 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:22:26 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T01:22:27.302 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:22:26 smithi047 conmon[78521]: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T01:22:27.302 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:22:26 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T01:22:27.302 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:22:26 smithi047 conmon[78521]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T01:22:27.302 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:22:26 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T01:22:27.302 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:22:26 smithi047 conmon[78521]: return self.event_loop.get_result(coro) 2022-05-14T01:22:27.303 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:22:26 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T01:22:27.303 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:22:26 smithi047 conmon[78521]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T01:22:27.303 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:22:26 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T01:22:27.303 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:22:26 smithi047 conmon[78521]: return self.__get_result() 2022-05-14T01:22:27.304 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:22:26 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T01:22:27.304 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:22:26 smithi047 conmon[78521]: raise self._exception 2022-05-14T01:22:27.304 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:22:26 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T01:22:27.304 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:22:26 smithi047 conmon[78521]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T01:22:27.304 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:22:26 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T01:22:27.305 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:22:26 smithi047 conmon[78521]: conn = await self._remote_connection(host, addr) 2022-05-14T01:22:27.305 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:22:26 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T01:22:27.305 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:22:26 smithi047 conmon[78521]: raise 2022-05-14T01:22:27.305 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:22:26 smithi047 conmon[78521]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T01:22:27.306 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:22:26 smithi047 conmon[78521]: self.gen.throw(type, value, traceback) 2022-05-14T01:22:27.306 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:22:26 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T01:22:27.306 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:22:26 smithi047 conmon[78521]: raise OrchestratorError(msg) 2022-05-14T01:22:27.306 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:22:26 smithi047 conmon[78521]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:22:27.425 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:22:27 smithi115 conmon[90393]: 2022-05-14T01:22:27.117+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:22:27.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:22:27 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:22:27.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:22:27 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:22:28.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:22:28 smithi115 conmon[90393]: 2022-05-14T01:22:28.104+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:22:28.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:22:28 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:22:28.791 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T01:22:28.792 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:22:28.792 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T01:22:28.792 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:22:28.792 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:22:28.793 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:22:28.793 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T01:22:28.793 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:22:28.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T01:22:28.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:22:28.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T01:22:28.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:22:28.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T01:22:28.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:22:28.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T01:22:28.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:22:28.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:22:28.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:22:28.796 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:22:28 smithi047 ceph-mon[77147]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T01:22:28.797 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T01:22:28.797 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:22:28.797 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T01:22:28.797 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:22:28.798 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:22:28.798 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:22:28.798 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T01:22:28.798 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:22:28.799 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T01:22:28.799 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:22:28.799 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T01:22:28.799 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:22:28.800 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T01:22:28.800 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:22:28.800 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T01:22:28.800 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:22:28.801 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:22:28.801 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:22:28.801 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T01:22:28.801 INFO:journalctl@ceph.mon.b.smithi047.stdout: During handling of the above exception, another exception occurred: 2022-05-14T01:22:28.801 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T01:22:28.802 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T01:22:28.807 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T01:22:28.807 INFO:journalctl@ceph.mon.b.smithi047.stdout: return f(*arg) 2022-05-14T01:22:28.808 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T01:22:28.808 INFO:journalctl@ceph.mon.b.smithi047.stdout: self._write_client_files(client_files, host) 2022-05-14T01:22:28.808 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T01:22:28.809 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T01:22:28.809 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T01:22:28.810 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T01:22:28.810 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T01:22:28.810 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.event_loop.get_result(coro) 2022-05-14T01:22:28.811 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T01:22:28.811 INFO:journalctl@ceph.mon.b.smithi047.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T01:22:28.811 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T01:22:28.812 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.__get_result() 2022-05-14T01:22:28.812 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T01:22:28.812 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise self._exception 2022-05-14T01:22:28.813 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T01:22:28.813 INFO:journalctl@ceph.mon.b.smithi047.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T01:22:28.813 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T01:22:28.814 INFO:journalctl@ceph.mon.b.smithi047.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T01:22:28.814 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T01:22:28.814 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise 2022-05-14T01:22:28.815 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T01:22:28.815 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.gen.throw(type, value, traceback) 2022-05-14T01:22:28.815 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T01:22:28.816 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OrchestratorError(msg) 2022-05-14T01:22:28.816 INFO:journalctl@ceph.mon.b.smithi047.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:22:28.816 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:22:28 smithi047 ceph-mon[77147]: pgmap v3793: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:22:28.817 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:22:28 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:22:28.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:22:28 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:22:28.858 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T01:22:28.859 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:22:28.859 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T01:22:28.859 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:22:28.860 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:22:28.860 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:22:28.860 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T01:22:28.860 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:22:28.860 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T01:22:28.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:22:28.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T01:22:28.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:22:28.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T01:22:28.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:22:28.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T01:22:28.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:22:28.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:22:28.863 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:22:28.863 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:22:28 smithi115 ceph-mon[78593]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T01:22:28.863 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T01:22:28.863 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:22:28.863 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T01:22:28.864 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:22:28.864 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:22:28.864 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:22:28.864 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T01:22:28.865 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:22:28.865 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T01:22:28.865 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:22:28.865 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T01:22:28.865 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:22:28.866 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T01:22:28.866 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:22:28.866 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T01:22:28.866 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:22:28.866 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:22:28.867 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:22:28.867 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T01:22:28.867 INFO:journalctl@ceph.mon.c.smithi115.stdout: During handling of the above exception, another exception occurred: 2022-05-14T01:22:28.867 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T01:22:28.868 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T01:22:28.868 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T01:22:28.868 INFO:journalctl@ceph.mon.c.smithi115.stdout: return f(*arg) 2022-05-14T01:22:28.868 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T01:22:28.868 INFO:journalctl@ceph.mon.c.smithi115.stdout: self._write_client_files(client_files, host) 2022-05-14T01:22:28.869 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T01:22:28.869 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T01:22:28.869 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T01:22:28.869 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T01:22:28.869 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T01:22:28.870 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.event_loop.get_result(coro) 2022-05-14T01:22:28.870 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T01:22:28.870 INFO:journalctl@ceph.mon.c.smithi115.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T01:22:28.870 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T01:22:28.871 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.__get_result() 2022-05-14T01:22:28.871 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T01:22:28.871 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise self._exception 2022-05-14T01:22:28.872 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T01:22:28.872 INFO:journalctl@ceph.mon.c.smithi115.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T01:22:28.872 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T01:22:28.872 INFO:journalctl@ceph.mon.c.smithi115.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T01:22:28.873 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T01:22:28.873 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise 2022-05-14T01:22:28.873 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T01:22:28.873 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.gen.throw(type, value, traceback) 2022-05-14T01:22:28.874 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T01:22:28.874 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OrchestratorError(msg) 2022-05-14T01:22:28.874 INFO:journalctl@ceph.mon.c.smithi115.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:22:28.874 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:22:28 smithi115 ceph-mon[78593]: pgmap v3793: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:22:28.875 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:22:28 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:22:29.432 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:22:29 smithi115 conmon[90393]: 2022-05-14T01:22:29.141+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:22:29.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:22:29 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:22:29.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:22:29 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:22:30.434 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:22:30 smithi115 conmon[90393]: 2022-05-14T01:22:30.116+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:22:30.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:22:30 smithi047 ceph-mon[77147]: pgmap v3794: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:22:30.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:22:30 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:22:30.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:22:30 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:22:30.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:22:30 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:22:30.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:22:30 smithi115 ceph-mon[78593]: pgmap v3794: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:22:30.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:22:30 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:22:30.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:22:30 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:22:30.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:22:30 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:22:31.438 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:22:31 smithi115 conmon[90393]: 2022-05-14T01:22:31.149+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:22:31.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:22:31 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:22:31.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:22:31 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:22:32.440 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:22:32 smithi115 conmon[90393]: 2022-05-14T01:22:32.184+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:22:32.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:22:32 smithi047 ceph-mon[77147]: pgmap v3795: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:22:32.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:22:32 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:22:32.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:22:32 smithi115 ceph-mon[78593]: pgmap v3795: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:22:32.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:22:32 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:22:33.445 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:22:33 smithi115 conmon[90393]: 2022-05-14T01:22:33.145+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:22:33.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:22:33 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:22:33.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:22:33 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:22:34.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:22:34 smithi115 conmon[90393]: 2022-05-14T01:22:34.097+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:22:34.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:22:34 smithi047 ceph-mon[77147]: pgmap v3796: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:22:34.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:22:34 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:22:34.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:22:34 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:22:34.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:22:34 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:22:34.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:22:34 smithi115 ceph-mon[78593]: pgmap v3796: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:22:34.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:22:34 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:22:34.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:22:34 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:22:34.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:22:34 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:22:35.448 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:22:35 smithi115 conmon[90393]: 2022-05-14T01:22:35.117+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:22:35.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:22:35 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 6885 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:22:35.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:22:35 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:22:35.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:22:35 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 6885 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:22:35.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:22:35 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:22:36.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:22:36 smithi115 conmon[90393]: 2022-05-14T01:22:36.091+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:22:36.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:22:36 smithi047 ceph-mon[77147]: pgmap v3797: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:22:36.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:22:36 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:22:36.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:22:36 smithi115 ceph-mon[78593]: pgmap v3797: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:22:36.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:22:36 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:22:37.456 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:22:37 smithi115 conmon[90393]: 2022-05-14T01:22:37.139+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:22:37.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:22:37 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:22:37.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:22:37 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:22:38.456 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:22:38 smithi115 conmon[90393]: 2022-05-14T01:22:38.166+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:22:38.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:22:38 smithi047 ceph-mon[77147]: pgmap v3798: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:22:38.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:22:38 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:22:38.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:22:38 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:22:38.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:22:38 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:22:38.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:22:38 smithi115 ceph-mon[78593]: pgmap v3798: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:22:38.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:22:38 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:22:38.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:22:38 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:22:38.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:22:38 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:22:39.461 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:22:39 smithi115 conmon[90393]: 2022-05-14T01:22:39.195+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:22:39.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:22:39 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:22:39.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:22:39 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:22:40.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:22:40 smithi115 conmon[90393]: 2022-05-14T01:22:40.243+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:22:40.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:22:40 smithi115 ceph-mon[78593]: pgmap v3799: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:22:40.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:22:40 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 6895 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:22:40.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:22:40 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:22:40.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:22:40 smithi047 ceph-mon[77147]: pgmap v3799: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:22:40.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:22:40 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 6895 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:22:40.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:22:40 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:22:41.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:22:41 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:22:41.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:22:41 smithi115 conmon[90393]: 2022-05-14T01:22:41.289+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:22:41.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:22:41 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:22:42.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:22:42 smithi115 conmon[90393]: 2022-05-14T01:22:42.309+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:22:42.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:22:42 smithi115 ceph-mon[78593]: pgmap v3800: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:22:42.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:22:42 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:22:42.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:22:42 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:22:42.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:22:42 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:22:42.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:22:42 smithi047 ceph-mon[77147]: pgmap v3800: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:22:42.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:22:42 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:22:42.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:22:42 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:22:42.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:22:42 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:22:43.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:22:43 smithi115 conmon[90393]: 2022-05-14T01:22:43.327+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:22:44.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:22:44 smithi115 conmon[90393]: 2022-05-14T01:22:44.354+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:22:44.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:22:44 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:22:44.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:22:44 smithi115 ceph-mon[78593]: pgmap v3801: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:22:44.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:22:44 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:22:44.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:22:44 smithi047 ceph-mon[77147]: pgmap v3801: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:22:45.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:22:45 smithi115 conmon[90393]: 2022-05-14T01:22:45.354+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:22:46.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:22:45 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:22:46.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:22:45 smithi047 ceph-mon[77147]: pgmap v3802: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:22:46.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:22:45 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:22:46.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:22:45 smithi115 ceph-mon[78593]: pgmap v3802: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:22:46.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:22:46 smithi115 conmon[90393]: 2022-05-14T01:22:46.348+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:22:47.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:22:46 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:22:47.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:22:46 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 6900 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:22:47.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:22:46 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:22:47.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:22:46 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 6900 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:22:47.735 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:22:47 smithi115 conmon[90393]: 2022-05-14T01:22:47.372+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:22:48.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:22:47 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:22:48.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:22:47 smithi047 ceph-mon[77147]: pgmap v3803: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:22:48.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:22:47 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:22:48.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:22:47 smithi115 ceph-mon[78593]: pgmap v3803: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:22:48.734 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:22:48 smithi115 conmon[90393]: 2022-05-14T01:22:48.410+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:22:49.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:22:48 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:22:49.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:22:48 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:22:49.744 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:22:49 smithi115 conmon[90393]: 2022-05-14T01:22:49.460+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:22:50.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:22:49 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:22:50.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:22:49 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:22:50.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:22:49 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:22:50.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:22:49 smithi047 ceph-mon[77147]: pgmap v3804: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:22:50.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:22:49 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:22:50.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:22:49 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:22:50.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:22:49 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:22:50.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:22:49 smithi115 ceph-mon[78593]: pgmap v3804: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:22:50.745 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:22:50 smithi115 conmon[90393]: 2022-05-14T01:22:50.448+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:22:51.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:22:50 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:22:51.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:22:50 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:22:51.751 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:22:51 smithi115 conmon[90393]: 2022-05-14T01:22:51.454+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:22:52.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:22:51 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:22:52.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:22:51 smithi047 ceph-mon[77147]: pgmap v3805: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:22:52.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:22:51 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:22:52.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:22:51 smithi115 ceph-mon[78593]: pgmap v3805: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:22:52.756 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:22:52 smithi115 conmon[90393]: 2022-05-14T01:22:52.425+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:22:53.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:22:52 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:22:53.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:22:52 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:22:53.763 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:22:53 smithi115 conmon[90393]: 2022-05-14T01:22:53.452+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:22:54.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:22:53 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:22:54.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:22:53 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:22:54.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:22:53 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:22:54.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:22:53 smithi047 ceph-mon[77147]: pgmap v3806: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:22:54.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:22:53 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:22:54.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:22:53 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:22:54.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:22:53 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:22:54.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:22:53 smithi115 ceph-mon[78593]: pgmap v3806: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:22:54.765 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:22:54 smithi115 conmon[90393]: 2022-05-14T01:22:54.461+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:22:55.052 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:22:54 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:22:55.053 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:22:54 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 6905 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:22:55.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:22:54 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:22:55.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:22:54 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 6905 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:22:55.767 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:22:55 smithi115 conmon[90393]: 2022-05-14T01:22:55.434+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:22:56.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:22:55 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:22:56.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:22:55 smithi047 ceph-mon[77147]: pgmap v3807: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:22:56.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:22:55 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:22:56.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:22:55 smithi115 ceph-mon[78593]: pgmap v3807: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:22:56.769 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:22:56 smithi115 conmon[90393]: 2022-05-14T01:22:56.426+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:22:57.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:22:56 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:22:57.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:22:56 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:22:57.778 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:22:57 smithi115 conmon[90393]: 2022-05-14T01:22:57.471+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:22:58.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:22:57 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:22:58.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:22:57 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:22:58.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:22:57 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:22:58.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:22:57 smithi047 ceph-mon[77147]: pgmap v3808: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:22:58.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:22:57 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:22:58.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:22:57 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:22:58.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:22:57 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:22:58.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:22:57 smithi115 ceph-mon[78593]: pgmap v3808: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:22:58.779 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:22:58 smithi115 conmon[90393]: 2022-05-14T01:22:58.481+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:22:59.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:22:58 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:22:59.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:22:58 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:22:59.783 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:22:59 smithi115 conmon[90393]: 2022-05-14T01:22:59.500+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:23:00.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:22:59 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:23:00.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:22:59 smithi047 ceph-mon[77147]: pgmap v3809: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:23:00.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:22:59 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 6910 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:23:00.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:22:59 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:23:00.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:22:59 smithi115 ceph-mon[78593]: pgmap v3809: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:23:00.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:22:59 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 6910 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:23:00.785 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:23:00 smithi115 conmon[90393]: 2022-05-14T01:23:00.464+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:23:01.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:23:00 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:23:01.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:23:00 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:23:01.794 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:23:01 smithi115 conmon[90393]: 2022-05-14T01:23:01.436+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:23:02.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:23:01 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:23:02.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:23:01 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:23:02.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:23:01 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:23:02.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:23:01 smithi115 ceph-mon[78593]: pgmap v3810: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:23:02.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:23:01 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:23:02.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:23:01 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:23:02.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:23:01 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:23:02.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:23:01 smithi047 ceph-mon[77147]: pgmap v3810: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:23:02.795 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:23:02 smithi115 conmon[90393]: 2022-05-14T01:23:02.424+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:23:03.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:23:02 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:23:03.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:23:02 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:23:03.800 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:23:03 smithi115 conmon[90393]: 2022-05-14T01:23:03.424+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:23:04.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:23:03 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:23:04.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:23:03 smithi115 ceph-mon[78593]: pgmap v3811: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:23:04.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:23:03 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:23:04.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:23:03 smithi047 ceph-mon[77147]: pgmap v3811: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:23:04.804 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:23:04 smithi115 conmon[90393]: 2022-05-14T01:23:04.460+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:23:05.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:23:04 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:23:05.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:23:04 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 6915 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:23:05.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:23:04 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:23:05.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:23:04 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 6915 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:23:05.809 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:23:05 smithi115 conmon[90393]: 2022-05-14T01:23:05.417+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:23:06.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:23:05 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:23:06.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:23:05 smithi115 ceph-mon[78593]: pgmap v3812: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:23:06.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:23:05 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:23:06.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:23:05 smithi047 ceph-mon[77147]: pgmap v3812: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:23:06.809 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:23:06 smithi115 conmon[90393]: 2022-05-14T01:23:06.405+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:23:07.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:23:06 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:23:07.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:23:06 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:23:07.815 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:23:07 smithi115 conmon[90393]: 2022-05-14T01:23:07.410+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:23:08.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:23:07 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:23:08.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:23:07 smithi115 ceph-mon[78593]: pgmap v3813: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:23:08.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:23:07 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:23:08.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:23:07 smithi047 ceph-mon[77147]: pgmap v3813: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:23:08.814 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:23:08 smithi115 conmon[90393]: 2022-05-14T01:23:08.457+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:23:09.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:23:08 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:23:09.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:23:08 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:23:09.822 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:23:09 smithi115 conmon[90393]: 2022-05-14T01:23:09.428+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:23:10.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:23:09 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:23:10.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:23:09 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:23:10.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:23:09 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:23:10.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:23:09 smithi115 ceph-mon[78593]: pgmap v3814: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:23:10.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:23:09 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 6920 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:23:10.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:23:09 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:23:10.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:23:09 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:23:10.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:23:09 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:23:10.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:23:09 smithi047 ceph-mon[77147]: pgmap v3814: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:23:10.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:23:09 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 6920 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:23:10.824 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:23:10 smithi115 conmon[90393]: 2022-05-14T01:23:10.408+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:23:11.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:23:10 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:23:11.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:23:10 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:23:11.829 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:23:11 smithi115 conmon[90393]: 2022-05-14T01:23:11.399+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:23:12.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:23:11 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:23:12.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:23:11 smithi115 ceph-mon[78593]: pgmap v3815: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:23:12.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:23:11 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:23:12.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:23:11 smithi047 ceph-mon[77147]: pgmap v3815: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:23:12.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:23:12 smithi115 conmon[90393]: 2022-05-14T01:23:12.355+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:23:13.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:23:12 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:23:13.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:23:12 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:23:13.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:23:13 smithi115 conmon[90393]: 2022-05-14T01:23:13.311+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:23:14.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:23:13 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:23:14.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:23:13 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:23:14.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:23:13 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:23:14.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:23:13 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:23:14.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:23:13 smithi115 ceph-mon[78593]: pgmap v3816: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:23:14.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:23:13 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T01:23:14.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:23:13 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T01:23:14.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:23:13 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:23:14.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:23:13 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:23:14.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:23:13 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:23:14.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:23:13 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:23:14.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:23:13 smithi047 ceph-mon[77147]: pgmap v3816: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:23:14.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:23:13 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T01:23:14.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:23:13 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T01:23:14.841 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:23:14 smithi115 conmon[90393]: 2022-05-14T01:23:14.359+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:23:15.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:23:14 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 6925 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:23:15.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:23:14 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 6925 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:23:15.845 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:23:15 smithi115 conmon[90393]: 2022-05-14T01:23:15.398+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:23:16.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:23:15 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:23:16.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:23:15 smithi115 ceph-mon[78593]: pgmap v3817: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:23:16.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:23:15 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:23:16.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:23:15 smithi047 ceph-mon[77147]: pgmap v3817: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:23:16.851 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:23:16 smithi115 conmon[90393]: 2022-05-14T01:23:16.371+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:23:17.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:23:16 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:23:17.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:23:16 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:23:17.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:23:17 smithi115 conmon[90393]: 2022-05-14T01:23:17.333+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:23:18.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:23:17 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:23:18.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:23:17 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:23:18.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:23:17 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:23:18.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:23:17 smithi115 ceph-mon[78593]: pgmap v3818: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:23:18.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:23:17 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:23:18.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:23:17 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:23:18.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:23:17 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:23:18.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:23:17 smithi047 ceph-mon[77147]: pgmap v3818: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:23:18.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:23:18 smithi115 conmon[90393]: 2022-05-14T01:23:18.338+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:23:19.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:23:18 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:23:19.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:23:18 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:23:19.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:23:19 smithi115 conmon[90393]: 2022-05-14T01:23:19.306+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:23:20.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:23:19 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:23:20.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:23:19 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:23:20.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:23:19 smithi047 ceph-mon[77147]: pgmap v3819: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:23:20.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:23:19 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 6930 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:23:20.308 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:23:19 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:23:20.309 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:23:19 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:23:20.309 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:23:19 smithi115 ceph-mon[78593]: pgmap v3819: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:23:20.309 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:23:19 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 6930 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:23:20.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:23:20 smithi115 conmon[90393]: 2022-05-14T01:23:20.308+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:23:21.278 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:23:20 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:23:21.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:23:20 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:23:21.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:23:21 smithi115 conmon[90393]: 2022-05-14T01:23:21.277+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:23:22.255 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:23:21 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:23:22.255 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:23:21 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:23:22.256 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:23:21 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:23:22.256 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:23:21 smithi115 ceph-mon[78593]: pgmap v3820: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:23:22.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:23:21 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:23:22.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:23:21 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:23:22.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:23:21 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:23:22.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:23:21 smithi047 ceph-mon[77147]: pgmap v3820: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:23:22.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:23:22 smithi115 conmon[90393]: 2022-05-14T01:23:22.254+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:23:23.245 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:23:22 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:23:23.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:23:22 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:23:23.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:23:23 smithi115 conmon[90393]: 2022-05-14T01:23:23.244+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:23:24.204 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:23:23 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:23:24.205 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:23:23 smithi115 ceph-mon[78593]: pgmap v3821: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:23:24.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:23:23 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:23:24.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:23:23 smithi047 ceph-mon[77147]: pgmap v3821: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:23:24.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:23:24 smithi115 conmon[90393]: 2022-05-14T01:23:24.204+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:23:25.253 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:23:24 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:23:25.253 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:23:24 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 6935 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:23:25.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:23:24 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:23:25.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:23:24 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 6935 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:23:25.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:23:25 smithi115 conmon[90393]: 2022-05-14T01:23:25.251+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:23:26.238 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:23:25 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:23:26.238 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:23:25 smithi115 ceph-mon[78593]: pgmap v3822: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:23:26.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:23:25 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:23:26.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:23:25 smithi047 ceph-mon[77147]: pgmap v3822: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:23:26.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:23:26 smithi115 conmon[90393]: 2022-05-14T01:23:26.237+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:23:27.215 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:23:26 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:23:27.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:23:26 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:23:27.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:23:27 smithi115 conmon[90393]: 2022-05-14T01:23:27.214+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:23:27.791 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:23:27 smithi047 conmon[78521]: 2022-05-14T01:23:27.535+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:23:27.791 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:23:27 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T01:23:27.791 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:23:27 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:23:27.792 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:23:27 smithi047 conmon[78521]: yield 2022-05-14T01:23:27.792 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:23:27 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:23:27.792 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:23:27 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:23:27.792 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:23:27 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:23:27.793 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:23:27 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T01:23:27.793 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:23:27 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:23:27.793 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:23:27 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T01:23:27.793 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:23:27 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:23:27.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:23:27 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T01:23:27.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:23:27 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:23:27.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:23:27 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T01:23:27.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:23:27 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:23:27.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:23:27 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T01:23:27.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:23:27 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:23:27.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:23:27 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:23:27.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:23:27 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:23:28.206 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:23:27 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T01:23:28.207 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:23:27 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T01:23:28.207 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:23:27 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T01:23:28.207 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:23:27 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:23:28.207 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:23:27 smithi115 ceph-mon[78593]: pgmap v3823: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:23:28.207 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:23:27 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:23:28.208 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T01:23:28.208 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:23:28.208 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T01:23:28.208 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:23:28.209 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:23:28.209 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:23:28.209 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T01:23:28.209 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:23:28.210 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T01:23:28.210 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:23:28.210 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T01:23:28.210 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:23:28.210 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T01:23:28.211 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:23:28.211 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T01:23:28.211 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:23:28.211 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:23:28.211 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:23:28.212 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:23:27 smithi115 ceph-mon[78593]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T01:23:28.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:23:27 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T01:23:28.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:23:27 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T01:23:28.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:23:27 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T01:23:28.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:23:27 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:23:28.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:23:27 smithi047 ceph-mon[77147]: pgmap v3823: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:23:28.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:23:27 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:23:28.292 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T01:23:28.292 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:23:28.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T01:23:28.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:23:28.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:23:28.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:23:28.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T01:23:28.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:23:28.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T01:23:28.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:23:28.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T01:23:28.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:23:28.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T01:23:28.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:23:28.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T01:23:28.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:23:28.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:23:28.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:23:28.296 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:23:27 smithi047 ceph-mon[77147]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T01:23:28.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:23:28 smithi115 conmon[90393]: 2022-05-14T01:23:28.205+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:23:29.221 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:23:28 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:23:29.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:23:28 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:23:29.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:23:29 smithi115 conmon[90393]: 2022-05-14T01:23:29.220+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:23:30.259 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:23:29 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:23:30.259 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:23:29 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:23:30.259 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:23:29 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:23:30.260 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:23:29 smithi115 ceph-mon[78593]: pgmap v3824: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:23:30.260 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:23:29 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 6940 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:23:30.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:23:29 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:23:30.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:23:29 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:23:30.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:23:29 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:23:30.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:23:29 smithi047 ceph-mon[77147]: pgmap v3824: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:23:30.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:23:29 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 6940 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:23:30.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:23:30 smithi115 conmon[90393]: 2022-05-14T01:23:30.258+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:23:30.908 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:23:30 smithi047 conmon[78521]: 2022-05-14T01:23:30.608+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:23:30.909 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:23:30 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T01:23:30.909 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:23:30 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:23:30.909 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:23:30 smithi047 conmon[78521]: yield 2022-05-14T01:23:30.910 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:23:30 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:23:30.910 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:23:30 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:23:30.910 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:23:30 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:23:30.910 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:23:30 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T01:23:30.911 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:23:30 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:23:30.911 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:23:30 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T01:23:30.911 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:23:30 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:23:30.911 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:23:30 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T01:23:30.911 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:23:30 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:23:30.912 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:23:30 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T01:23:30.912 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:23:30 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:23:30.912 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:23:30 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T01:23:30.912 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:23:30 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:23:30.912 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:23:30 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:23:30.913 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:23:30 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:23:30.913 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:23:30 smithi047 conmon[78521]: 2022-05-14T01:23:30.609+0000 7f533f777700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T01:23:30.913 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:23:30 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T01:23:30.913 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:23:30 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:23:30.914 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:23:30 smithi047 conmon[78521]: yield 2022-05-14T01:23:30.914 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:23:30 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:23:30.914 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:23:30 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:23:30.914 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:23:30 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:23:30.914 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:23:30 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T01:23:30.915 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:23:30 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:23:30.915 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:23:30 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T01:23:30.915 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:23:30 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:23:30.915 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:23:30 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T01:23:30.915 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:23:30 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:23:30.916 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:23:30 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T01:23:30.916 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:23:30 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:23:30.916 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:23:30 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T01:23:30.916 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:23:30 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:23:30.917 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:23:30 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:23:30.917 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:23:30 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:23:30.917 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:23:30 smithi047 conmon[78521]: 2022-05-14T01:23:30.917 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:23:30 smithi047 conmon[78521]: During handling of the above exception, another exception occurred: 2022-05-14T01:23:30.918 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:23:30 smithi047 conmon[78521]: 2022-05-14T01:23:30.918 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:23:30 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T01:23:30.918 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:23:30 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T01:23:30.918 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:23:30 smithi047 conmon[78521]: return f(*arg) 2022-05-14T01:23:30.918 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:23:30 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T01:23:30.919 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:23:30 smithi047 conmon[78521]: self._write_client_files(client_files, host) 2022-05-14T01:23:30.919 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:23:30 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T01:23:30.919 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:23:30 smithi047 conmon[78521]: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T01:23:30.919 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:23:30 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T01:23:30.920 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:23:30 smithi047 conmon[78521]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T01:23:30.920 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:23:30 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T01:23:30.920 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:23:30 smithi047 conmon[78521]: return self.event_loop.get_result(coro) 2022-05-14T01:23:30.920 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:23:30 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T01:23:30.920 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:23:30 smithi047 conmon[78521]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T01:23:30.921 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:23:30 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T01:23:30.921 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:23:30 smithi047 conmon[78521]: return self.__get_result() 2022-05-14T01:23:30.921 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:23:30 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T01:23:30.921 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:23:30 smithi047 conmon[78521]: raise self._exception 2022-05-14T01:23:30.922 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:23:30 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T01:23:30.922 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:23:30 smithi047 conmon[78521]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T01:23:30.922 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:23:30 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T01:23:30.922 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:23:30 smithi047 conmon[78521]: conn = await self._remote_connection(host, addr) 2022-05-14T01:23:30.922 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:23:30 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T01:23:30.923 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:23:30 smithi047 conmon[78521]: raise 2022-05-14T01:23:30.923 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:23:30 smithi047 conmon[78521]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T01:23:30.923 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:23:30 smithi047 conmon[78521]: self.gen.throw(type, value, traceback) 2022-05-14T01:23:30.923 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:23:30 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T01:23:30.924 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:23:30 smithi047 conmon[78521]: raise OrchestratorError(msg) 2022-05-14T01:23:30.924 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:23:30 smithi047 conmon[78521]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:23:31.231 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:23:30 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:23:31.232 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:23:30 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:23:31.232 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T01:23:31.232 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:23:31.233 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T01:23:31.233 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:23:31.233 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:23:31.234 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:23:31.234 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T01:23:31.234 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:23:31.234 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T01:23:31.234 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:23:31.235 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T01:23:31.235 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:23:31.235 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T01:23:31.235 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:23:31.236 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T01:23:31.236 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:23:31.236 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:23:31.236 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:23:31.236 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:23:30 smithi115 ceph-mon[78593]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T01:23:31.237 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T01:23:31.237 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:23:31.237 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T01:23:31.237 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:23:31.237 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:23:31.238 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:23:31.238 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T01:23:31.238 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:23:31.238 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T01:23:31.238 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:23:31.239 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T01:23:31.239 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:23:31.239 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T01:23:31.239 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:23:31.239 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T01:23:31.240 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:23:31.240 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:23:31.240 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:23:31.240 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T01:23:31.240 INFO:journalctl@ceph.mon.c.smithi115.stdout: During handling of the above exception, another exception occurred: 2022-05-14T01:23:31.241 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T01:23:31.241 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T01:23:31.241 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T01:23:31.241 INFO:journalctl@ceph.mon.c.smithi115.stdout: return f(*arg) 2022-05-14T01:23:31.242 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T01:23:31.242 INFO:journalctl@ceph.mon.c.smithi115.stdout: self._write_client_files(client_files, host) 2022-05-14T01:23:31.242 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T01:23:31.242 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T01:23:31.242 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T01:23:31.243 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T01:23:31.243 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T01:23:31.243 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.event_loop.get_result(coro) 2022-05-14T01:23:31.243 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T01:23:31.243 INFO:journalctl@ceph.mon.c.smithi115.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T01:23:31.244 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T01:23:31.244 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.__get_result() 2022-05-14T01:23:31.244 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T01:23:31.244 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise self._exception 2022-05-14T01:23:31.244 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T01:23:31.245 INFO:journalctl@ceph.mon.c.smithi115.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T01:23:31.245 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T01:23:31.245 INFO:journalctl@ceph.mon.c.smithi115.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T01:23:31.245 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T01:23:31.245 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise 2022-05-14T01:23:31.246 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T01:23:31.246 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.gen.throw(type, value, traceback) 2022-05-14T01:23:31.246 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T01:23:31.246 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OrchestratorError(msg) 2022-05-14T01:23:31.246 INFO:journalctl@ceph.mon.c.smithi115.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:23:31.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:23:30 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:23:31.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:23:30 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:23:31.291 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T01:23:31.292 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:23:31.292 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T01:23:31.292 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:23:31.292 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:23:31.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:23:31.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T01:23:31.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:23:31.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T01:23:31.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:23:31.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T01:23:31.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:23:31.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T01:23:31.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:23:31.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T01:23:31.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:23:31.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:23:31.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:23:31.296 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:23:30 smithi047 ceph-mon[77147]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T01:23:31.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T01:23:31.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:23:31.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T01:23:31.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:23:31.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:23:31.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:23:31.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T01:23:31.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:23:31.298 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T01:23:31.298 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:23:31.298 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T01:23:31.298 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:23:31.299 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T01:23:31.299 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:23:31.299 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T01:23:31.299 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:23:31.299 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:23:31.300 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:23:31.300 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T01:23:31.300 INFO:journalctl@ceph.mon.b.smithi047.stdout: During handling of the above exception, another exception occurred: 2022-05-14T01:23:31.300 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T01:23:31.301 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T01:23:31.301 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T01:23:31.301 INFO:journalctl@ceph.mon.b.smithi047.stdout: return f(*arg) 2022-05-14T01:23:31.301 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T01:23:31.301 INFO:journalctl@ceph.mon.b.smithi047.stdout: self._write_client_files(client_files, host) 2022-05-14T01:23:31.302 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T01:23:31.302 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T01:23:31.302 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T01:23:31.302 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T01:23:31.303 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T01:23:31.303 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.event_loop.get_result(coro) 2022-05-14T01:23:31.303 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T01:23:31.303 INFO:journalctl@ceph.mon.b.smithi047.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T01:23:31.303 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T01:23:31.304 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.__get_result() 2022-05-14T01:23:31.304 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T01:23:31.304 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise self._exception 2022-05-14T01:23:31.304 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T01:23:31.305 INFO:journalctl@ceph.mon.b.smithi047.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T01:23:31.305 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T01:23:31.305 INFO:journalctl@ceph.mon.b.smithi047.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T01:23:31.305 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T01:23:31.305 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise 2022-05-14T01:23:31.306 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T01:23:31.306 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.gen.throw(type, value, traceback) 2022-05-14T01:23:31.306 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T01:23:31.306 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OrchestratorError(msg) 2022-05-14T01:23:31.307 INFO:journalctl@ceph.mon.b.smithi047.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:23:31.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:23:31 smithi115 conmon[90393]: 2022-05-14T01:23:31.231+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:23:32.212 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:23:31 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:23:32.213 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:23:31 smithi115 ceph-mon[78593]: pgmap v3825: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:23:32.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:23:31 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:23:32.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:23:31 smithi047 ceph-mon[77147]: pgmap v3825: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:23:32.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:23:32 smithi115 conmon[90393]: 2022-05-14T01:23:32.211+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:23:33.230 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:23:32 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:23:33.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:23:32 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:23:33.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:23:33 smithi115 conmon[90393]: 2022-05-14T01:23:33.229+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:23:34.189 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:23:33 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:23:34.189 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:23:33 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:23:34.190 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:23:33 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:23:34.190 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:23:33 smithi115 ceph-mon[78593]: pgmap v3826: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:23:34.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:23:33 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:23:34.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:23:33 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:23:34.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:23:33 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:23:34.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:23:33 smithi047 ceph-mon[77147]: pgmap v3826: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:23:34.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:23:34 smithi115 conmon[90393]: 2022-05-14T01:23:34.188+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:23:35.230 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:23:34 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:23:35.230 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:23:34 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 6945 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:23:35.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:23:34 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:23:35.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:23:34 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 6945 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:23:35.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:23:35 smithi115 conmon[90393]: 2022-05-14T01:23:35.229+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:23:36.206 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:23:35 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:23:36.206 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:23:35 smithi115 ceph-mon[78593]: pgmap v3827: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:23:36.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:23:35 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:23:36.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:23:35 smithi047 ceph-mon[77147]: pgmap v3827: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:23:36.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:23:36 smithi115 conmon[90393]: 2022-05-14T01:23:36.205+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:23:37.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:23:36 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:23:37.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:23:37 smithi115 conmon[90393]: 2022-05-14T01:23:37.190+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:23:37.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:23:36 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:23:38.213 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:23:37 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:23:38.214 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:23:37 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:23:38.214 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:23:37 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:23:38.214 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:23:37 smithi115 ceph-mon[78593]: pgmap v3828: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:23:38.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:23:37 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:23:38.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:23:37 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:23:38.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:23:37 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:23:38.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:23:37 smithi047 ceph-mon[77147]: pgmap v3828: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:23:38.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:23:38 smithi115 conmon[90393]: 2022-05-14T01:23:38.212+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:23:39.199 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:23:38 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:23:39.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:23:38 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:23:39.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:23:39 smithi115 conmon[90393]: 2022-05-14T01:23:39.198+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:23:40.222 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:23:39 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:23:40.223 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:23:39 smithi115 ceph-mon[78593]: pgmap v3829: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:23:40.223 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:23:39 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 6950 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:23:40.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:23:39 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:23:40.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:23:39 smithi047 ceph-mon[77147]: pgmap v3829: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:23:40.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:23:39 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 6950 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:23:40.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:23:40 smithi115 conmon[90393]: 2022-05-14T01:23:40.221+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:23:41.207 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:23:40 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:23:41.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:23:40 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:23:41.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:23:41 smithi115 conmon[90393]: 2022-05-14T01:23:41.205+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:23:42.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:23:41 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:23:42.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:23:41 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:23:42.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:23:41 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:23:42.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:23:41 smithi047 ceph-mon[77147]: pgmap v3830: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:23:42.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:23:42 smithi115 conmon[90393]: 2022-05-14T01:23:42.193+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:23:42.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:23:41 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:23:42.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:23:41 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:23:42.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:23:41 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:23:42.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:23:41 smithi115 ceph-mon[78593]: pgmap v3830: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:23:43.225 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:23:42 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:23:43.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:23:42 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:23:43.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:23:43 smithi115 conmon[90393]: 2022-05-14T01:23:43.224+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:23:44.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:23:43 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:23:44.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:23:43 smithi047 ceph-mon[77147]: pgmap v3831: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:23:44.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:23:43 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:23:44.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:23:43 smithi115 ceph-mon[78593]: pgmap v3831: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:23:44.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:23:44 smithi115 conmon[90393]: 2022-05-14T01:23:44.193+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:23:45.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:23:44 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:23:45.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:23:44 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 6955 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:23:45.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:23:45 smithi115 conmon[90393]: 2022-05-14T01:23:45.184+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:23:45.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:23:44 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:23:45.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:23:44 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 6955 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:23:46.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:23:45 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:23:46.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:23:45 smithi047 ceph-mon[77147]: pgmap v3832: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:23:46.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:23:46 smithi115 conmon[90393]: 2022-05-14T01:23:46.143+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:23:46.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:23:45 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:23:46.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:23:45 smithi115 ceph-mon[78593]: pgmap v3832: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:23:47.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:23:46 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:23:47.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:23:47 smithi115 conmon[90393]: 2022-05-14T01:23:47.107+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:23:47.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:23:46 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:23:48.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:23:47 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:23:48.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:23:47 smithi047 ceph-mon[77147]: pgmap v3833: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:23:48.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:23:48 smithi115 conmon[90393]: 2022-05-14T01:23:48.073+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:23:48.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:23:47 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:23:48.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:23:47 smithi115 ceph-mon[78593]: pgmap v3833: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:23:49.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:23:48 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:23:49.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:23:49 smithi115 conmon[90393]: 2022-05-14T01:23:49.104+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:23:49.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:23:48 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:23:50.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:23:49 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:23:50.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:23:49 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:23:50.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:23:49 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:23:50.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:23:49 smithi047 ceph-mon[77147]: pgmap v3834: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:23:50.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:23:49 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 6960 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:23:50.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:23:50 smithi115 conmon[90393]: 2022-05-14T01:23:50.105+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:23:50.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:23:49 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:23:50.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:23:49 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:23:50.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:23:49 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:23:50.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:23:49 smithi115 ceph-mon[78593]: pgmap v3834: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:23:50.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:23:49 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 6960 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:23:51.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:23:50 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:23:51.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:23:51 smithi115 conmon[90393]: 2022-05-14T01:23:51.106+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:23:51.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:23:50 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:23:52.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:23:51 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:23:52.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:23:51 smithi047 ceph-mon[77147]: pgmap v3835: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:23:52.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:23:52 smithi115 conmon[90393]: 2022-05-14T01:23:52.058+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:23:52.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:23:51 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:23:52.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:23:51 smithi115 ceph-mon[78593]: pgmap v3835: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:23:53.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:23:53 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:23:53.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:23:53 smithi115 conmon[90393]: 2022-05-14T01:23:53.011+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:23:53.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:23:53 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:23:54.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:23:54 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:23:54.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:23:54 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:23:54.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:23:54 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:23:54.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:23:54 smithi047 ceph-mon[77147]: pgmap v3836: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:23:54.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:23:53 smithi115 conmon[90393]: 2022-05-14T01:23:53.978+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:23:54.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:23:54 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:23:54.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:23:54 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:23:54.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:23:54 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:23:54.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:23:54 smithi115 ceph-mon[78593]: pgmap v3836: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:23:55.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:23:55 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:23:55.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:23:55 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 6964 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:23:55.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:23:54 smithi115 conmon[90393]: 2022-05-14T01:23:54.975+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:23:55.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:23:55 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:23:55.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:23:55 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 6964 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:23:56.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:23:56 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:23:56.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:23:56 smithi047 ceph-mon[77147]: pgmap v3837: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:23:56.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:23:56 smithi115 conmon[90393]: 2022-05-14T01:23:56.006+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:23:56.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:23:56 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:23:56.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:23:56 smithi115 ceph-mon[78593]: pgmap v3837: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:23:57.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:23:57 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:23:57.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:23:56 smithi115 conmon[90393]: 2022-05-14T01:23:56.987+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:23:57.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:23:57 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:23:58.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:23:58 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:23:58.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:23:58 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:23:58.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:23:58 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:23:58.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:23:58 smithi047 ceph-mon[77147]: pgmap v3838: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:23:58.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:23:57 smithi115 conmon[90393]: 2022-05-14T01:23:57.990+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:23:58.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:23:58 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:23:58.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:23:58 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:23:58.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:23:58 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:23:58.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:23:58 smithi115 ceph-mon[78593]: pgmap v3838: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:23:59.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:23:59 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:23:59.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:23:58 smithi115 conmon[90393]: 2022-05-14T01:23:58.964+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:23:59.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:23:59 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:24:00.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:24:00 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:24:00.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:24:00 smithi047 ceph-mon[77147]: pgmap v3839: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:24:00.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:24:00 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 6970 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:24:00.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:24:00 smithi115 conmon[90393]: 2022-05-14T01:23:59.999+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:24:00.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:24:00 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:24:00.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:24:00 smithi115 ceph-mon[78593]: pgmap v3839: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:24:00.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:24:00 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 6970 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:24:01.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:24:01 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:24:01.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:24:01 smithi115 conmon[90393]: 2022-05-14T01:24:01.016+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:24:01.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:24:01 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:24:02.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:24:02 smithi115 conmon[90393]: 2022-05-14T01:24:02.057+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:24:02.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:24:02 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:24:02.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:24:02 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:24:02.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:24:02 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:24:02.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:24:02 smithi115 ceph-mon[78593]: pgmap v3840: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:24:02.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:24:02 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:24:02.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:24:02 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:24:02.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:24:02 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:24:02.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:24:02 smithi047 ceph-mon[77147]: pgmap v3840: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:24:03.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:24:03 smithi115 conmon[90393]: 2022-05-14T01:24:03.037+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:24:03.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:24:03 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:24:03.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:24:03 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:24:04.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:24:04 smithi115 conmon[90393]: 2022-05-14T01:24:04.051+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:24:04.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:24:04 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:24:04.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:24:04 smithi115 ceph-mon[78593]: pgmap v3841: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:24:04.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:24:04 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:24:04.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:24:04 smithi047 ceph-mon[77147]: pgmap v3841: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:24:05.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:24:05 smithi115 conmon[90393]: 2022-05-14T01:24:05.016+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:24:05.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:24:05 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:24:05.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:24:05 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 6975 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:24:05.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:24:05 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:24:05.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:24:05 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 6975 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:24:06.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:24:06 smithi115 conmon[90393]: 2022-05-14T01:24:05.999+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:24:06.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:24:06 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:24:06.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:24:06 smithi115 ceph-mon[78593]: pgmap v3842: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:24:06.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:24:06 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:24:06.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:24:06 smithi047 ceph-mon[77147]: pgmap v3842: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:24:07.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:24:07 smithi115 conmon[90393]: 2022-05-14T01:24:07.040+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:24:07.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:24:07 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:24:07.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:24:07 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:24:08.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:24:08 smithi115 conmon[90393]: 2022-05-14T01:24:08.042+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:24:08.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:24:08 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:24:08.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:24:08 smithi115 ceph-mon[78593]: pgmap v3843: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:24:08.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:24:08 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:24:08.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:24:08 smithi047 ceph-mon[77147]: pgmap v3843: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:24:09.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:24:09 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:24:09.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:24:09 smithi115 conmon[90393]: 2022-05-14T01:24:09.080+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:24:09.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:24:09 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:24:10.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:24:10 smithi115 conmon[90393]: 2022-05-14T01:24:10.040+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:24:10.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:24:10 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:24:10.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:24:10 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:24:10.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:24:10 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:24:10.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:24:10 smithi115 ceph-mon[78593]: pgmap v3844: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:24:10.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:24:10 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 6980 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:24:10.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:24:10 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:24:10.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:24:10 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:24:10.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:24:10 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:24:10.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:24:10 smithi047 ceph-mon[77147]: pgmap v3844: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:24:10.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:24:10 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 6980 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:24:11.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:24:11 smithi115 conmon[90393]: 2022-05-14T01:24:11.082+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:24:11.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:24:11 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:24:11.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:24:11 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:24:12.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:24:12 smithi115 conmon[90393]: 2022-05-14T01:24:12.095+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:24:12.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:24:12 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:24:12.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:24:12 smithi115 ceph-mon[78593]: pgmap v3845: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:24:12.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:24:12 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:24:12.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:24:12 smithi047 ceph-mon[77147]: pgmap v3845: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:24:13.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:24:13 smithi115 conmon[90393]: 2022-05-14T01:24:13.089+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:24:13.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:24:13 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:24:13.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:24:13 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:24:14.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:24:14 smithi115 conmon[90393]: 2022-05-14T01:24:14.120+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:24:14.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:24:14 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:24:14.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:24:14 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:24:14.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:24:14 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:24:14.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:24:14 smithi115 ceph-mon[78593]: pgmap v3846: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:24:14.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:24:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T01:24:14.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:24:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T01:24:14.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:24:14 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:24:14.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:24:14 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:24:14.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:24:14 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:24:14.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:24:14 smithi047 ceph-mon[77147]: pgmap v3846: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:24:14.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:24:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T01:24:14.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:24:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T01:24:15.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:24:15 smithi115 conmon[90393]: 2022-05-14T01:24:15.092+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:24:15.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:24:15 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:24:15.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:24:15 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 6984 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:24:15.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:24:15 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:24:15.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:24:15 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 6984 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:24:16.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:24:16 smithi115 conmon[90393]: 2022-05-14T01:24:16.104+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:24:16.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:24:16 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:24:16.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:24:16 smithi115 ceph-mon[78593]: pgmap v3847: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:24:16.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:24:16 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:24:16.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:24:16 smithi047 ceph-mon[77147]: pgmap v3847: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:24:17.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:24:17 smithi115 conmon[90393]: 2022-05-14T01:24:17.088+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:24:17.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:24:17 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:24:17.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:24:17 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:24:18.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:24:18 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:24:18.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:24:18 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:24:18.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:24:18 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:24:18.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:24:18 smithi047 ceph-mon[77147]: pgmap v3848: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:24:18.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:24:18 smithi115 conmon[90393]: 2022-05-14T01:24:18.114+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:24:18.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:24:18 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:24:18.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:24:18 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:24:18.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:24:18 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:24:18.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:24:18 smithi115 ceph-mon[78593]: pgmap v3848: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:24:19.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:24:19 smithi115 conmon[90393]: 2022-05-14T01:24:19.101+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:24:19.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:24:19 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:24:19.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:24:19 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:24:20.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:24:20 smithi115 conmon[90393]: 2022-05-14T01:24:20.097+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:24:20.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:24:20 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:24:20.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:24:20 smithi115 ceph-mon[78593]: pgmap v3849: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:24:20.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:24:20 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 6990 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:24:20.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:24:20 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:24:20.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:24:20 smithi047 ceph-mon[77147]: pgmap v3849: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:24:20.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:24:20 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 6990 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:24:21.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:24:21 smithi115 conmon[90393]: 2022-05-14T01:24:21.082+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:24:21.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:24:21 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:24:21.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:24:21 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:24:22.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:24:22 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:24:22.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:24:22 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:24:22.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:24:22 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:24:22.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:24:22 smithi047 ceph-mon[77147]: pgmap v3850: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:24:22.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:24:22 smithi115 conmon[90393]: 2022-05-14T01:24:22.113+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:24:22.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:24:22 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:24:22.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:24:22 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:24:22.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:24:22 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:24:22.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:24:22 smithi115 ceph-mon[78593]: pgmap v3850: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:24:23.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:24:23 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:24:23.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:24:23 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:24:23.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:24:23 smithi115 conmon[90393]: 2022-05-14T01:24:23.123+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:24:24.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:24:24 smithi115 conmon[90393]: 2022-05-14T01:24:24.087+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:24:24.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:24:24 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:24:24.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:24:24 smithi115 ceph-mon[78593]: pgmap v3851: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:24:24.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:24:24 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:24:24.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:24:24 smithi047 ceph-mon[77147]: pgmap v3851: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:24:25.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:24:25 smithi115 conmon[90393]: 2022-05-14T01:24:25.089+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:24:25.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:24:25 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:24:25.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:24:25 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 6995 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:24:25.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:24:25 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:24:25.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:24:25 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 6995 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:24:26.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:24:26 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:24:26.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:24:26 smithi047 ceph-mon[77147]: pgmap v3852: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:24:26.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:24:26 smithi115 conmon[90393]: 2022-05-14T01:24:26.139+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:24:26.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:24:26 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:24:26.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:24:26 smithi115 ceph-mon[78593]: pgmap v3852: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:24:27.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:24:27 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:24:27.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:24:27 smithi115 conmon[90393]: 2022-05-14T01:24:27.177+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:24:27.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:24:27 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:24:28.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:24:28 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:24:28.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:24:28 smithi047 ceph-mon[77147]: pgmap v3853: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:24:28.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:24:28 smithi115 conmon[90393]: 2022-05-14T01:24:28.207+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:24:28.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:24:28 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:24:28.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:24:28 smithi115 ceph-mon[78593]: pgmap v3853: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:24:29.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:24:29 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:24:29.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:24:29 smithi115 conmon[90393]: 2022-05-14T01:24:29.175+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:24:29.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:24:29 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:24:30.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:24:30 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:24:30.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:24:30 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:24:30.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:24:30 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:24:30.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:24:30 smithi047 ceph-mon[77147]: pgmap v3854: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:24:30.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:24:30 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 7000 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:24:30.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:24:30 smithi115 conmon[90393]: 2022-05-14T01:24:30.185+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:24:30.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:24:30 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:24:30.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:24:30 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:24:30.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:24:30 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:24:30.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:24:30 smithi115 ceph-mon[78593]: pgmap v3854: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:24:30.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:24:30 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 7000 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:24:31.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:24:31 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:24:31.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:24:31 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T01:24:31.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:24:31 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T01:24:31.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:24:31 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T01:24:31.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:24:31 smithi115 conmon[90393]: 2022-05-14T01:24:31.225+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:24:31.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:24:31 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:24:31.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:24:31 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T01:24:31.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:24:31 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T01:24:31.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:24:31 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T01:24:32.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:24:32 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:24:32.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:24:32 smithi047 ceph-mon[77147]: pgmap v3855: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:24:32.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:24:32 smithi115 conmon[90393]: 2022-05-14T01:24:32.238+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:24:32.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:24:32 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:24:32.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:24:32 smithi115 ceph-mon[78593]: pgmap v3855: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:24:33.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:24:33 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:24:33.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:24:33 smithi115 conmon[90393]: 2022-05-14T01:24:33.261+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:24:33.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:24:33 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:24:34.041 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:24:33 smithi047 conmon[78521]: 2022-05-14T01:24:33.711+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:24:34.042 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:24:33 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T01:24:34.042 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:24:33 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:24:34.042 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:24:33 smithi047 conmon[78521]: yield 2022-05-14T01:24:34.042 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:24:33 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:24:34.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:24:33 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:24:34.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:24:33 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:24:34.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:24:33 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T01:24:34.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:24:33 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:24:34.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:24:33 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T01:24:34.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:24:33 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:24:34.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:24:33 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T01:24:34.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:24:33 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:24:34.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:24:33 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T01:24:34.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:24:33 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:24:34.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:24:33 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T01:24:34.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:24:33 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:24:34.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:24:33 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:24:34.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:24:33 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:24:34.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:24:34 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:24:34.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:24:34 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:24:34.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:24:34 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:24:34.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:24:34 smithi047 ceph-mon[77147]: pgmap v3856: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:24:34.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:24:34 smithi115 conmon[90393]: 2022-05-14T01:24:34.306+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:24:34.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:24:34 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:24:34.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:24:34 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:24:34.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:24:34 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:24:34.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:24:34 smithi115 ceph-mon[78593]: pgmap v3856: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:24:35.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:24:35 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:24:35.541 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T01:24:35.541 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:24:35.542 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T01:24:35.542 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:24:35.542 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:24:35.542 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:24:35.542 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T01:24:35.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:24:35.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T01:24:35.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:24:35.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T01:24:35.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:24:35.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T01:24:35.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:24:35.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T01:24:35.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:24:35.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:24:35.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:24:35.546 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:24:35 smithi047 ceph-mon[77147]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T01:24:35.546 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:24:35 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:24:35.546 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:24:35 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 7005 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:24:35.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:24:35 smithi115 conmon[90393]: 2022-05-14T01:24:35.303+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:24:35.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:24:35 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:24:35.608 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T01:24:35.609 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:24:35.609 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T01:24:35.609 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:24:35.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:24:35.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:24:35.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T01:24:35.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:24:35.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T01:24:35.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:24:35.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T01:24:35.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:24:35.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T01:24:35.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:24:35.613 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T01:24:35.613 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:24:35.613 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:24:35.613 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:24:35.614 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:24:35 smithi115 ceph-mon[78593]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T01:24:35.614 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:24:35 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:24:35.614 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:24:35 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 7005 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:24:36.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:24:36 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:24:36.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:24:36 smithi047 ceph-mon[77147]: pgmap v3857: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:24:36.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:24:36 smithi115 conmon[90393]: 2022-05-14T01:24:36.328+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:24:36.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:24:36 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:24:36.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:24:36 smithi115 ceph-mon[78593]: pgmap v3857: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:24:37.041 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:24:36 smithi047 conmon[78521]: 2022-05-14T01:24:36.783+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:24:37.041 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:24:36 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T01:24:37.042 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:24:36 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:24:37.042 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:24:36 smithi047 conmon[78521]: yield 2022-05-14T01:24:37.042 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:24:36 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:24:37.042 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:24:36 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:24:37.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:24:36 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:24:37.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:24:36 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T01:24:37.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:24:36 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:24:37.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:24:36 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T01:24:37.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:24:36 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:24:37.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:24:36 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T01:24:37.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:24:36 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:24:37.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:24:36 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T01:24:37.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:24:36 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:24:37.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:24:36 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T01:24:37.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:24:36 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:24:37.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:24:36 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:24:37.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:24:36 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:24:37.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:24:36 smithi047 conmon[78521]: 2022-05-14T01:24:36.784+0000 7f5344f82700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T01:24:37.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:24:36 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T01:24:37.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:24:36 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:24:37.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:24:36 smithi047 conmon[78521]: yield 2022-05-14T01:24:37.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:24:36 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:24:37.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:24:36 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:24:37.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:24:36 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:24:37.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:24:36 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T01:24:37.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:24:36 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:24:37.048 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:24:36 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T01:24:37.048 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:24:36 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:24:37.048 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:24:36 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T01:24:37.048 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:24:36 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:24:37.048 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:24:36 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T01:24:37.049 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:24:36 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:24:37.049 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:24:36 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T01:24:37.049 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:24:36 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:24:37.049 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:24:36 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:24:37.049 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:24:36 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:24:37.050 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:24:36 smithi047 conmon[78521]: 2022-05-14T01:24:37.050 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:24:36 smithi047 conmon[78521]: During handling of the above exception, another exception occurred: 2022-05-14T01:24:37.050 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:24:36 smithi047 conmon[78521]: 2022-05-14T01:24:37.050 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:24:36 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T01:24:37.050 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:24:36 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T01:24:37.051 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:24:36 smithi047 conmon[78521]: return f(*arg) 2022-05-14T01:24:37.051 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:24:36 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T01:24:37.051 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:24:36 smithi047 conmon[78521]: self._write_client_files(client_files, host) 2022-05-14T01:24:37.051 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:24:36 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T01:24:37.052 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:24:36 smithi047 conmon[78521]: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T01:24:37.052 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:24:36 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T01:24:37.052 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:24:36 smithi047 conmon[78521]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T01:24:37.052 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:24:36 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T01:24:37.052 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:24:36 smithi047 conmon[78521]: return self.event_loop.get_result(coro) 2022-05-14T01:24:37.053 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:24:36 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T01:24:37.053 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:24:36 smithi047 conmon[78521]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T01:24:37.053 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:24:36 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T01:24:37.053 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:24:36 smithi047 conmon[78521]: return self.__get_result() 2022-05-14T01:24:37.053 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:24:36 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T01:24:37.054 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:24:36 smithi047 conmon[78521]: raise self._exception 2022-05-14T01:24:37.054 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:24:36 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T01:24:37.054 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:24:36 smithi047 conmon[78521]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T01:24:37.054 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:24:36 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T01:24:37.054 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:24:36 smithi047 conmon[78521]: conn = await self._remote_connection(host, addr) 2022-05-14T01:24:37.055 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:24:36 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T01:24:37.055 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:24:36 smithi047 conmon[78521]: raise 2022-05-14T01:24:37.055 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:24:36 smithi047 conmon[78521]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T01:24:37.055 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:24:36 smithi047 conmon[78521]: self.gen.throw(type, value, traceback) 2022-05-14T01:24:37.055 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:24:36 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T01:24:37.056 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:24:36 smithi047 conmon[78521]: raise OrchestratorError(msg) 2022-05-14T01:24:37.056 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:24:36 smithi047 conmon[78521]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:24:37.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:24:37 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:24:37.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:24:37 smithi115 conmon[90393]: 2022-05-14T01:24:37.281+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:24:37.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:24:37 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:24:38.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:24:38 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:24:38.541 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T01:24:38.541 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:24:38.542 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T01:24:38.542 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:24:38.542 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:24:38.542 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:24:38.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T01:24:38.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:24:38.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T01:24:38.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:24:38.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T01:24:38.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:24:38.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T01:24:38.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:24:38.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T01:24:38.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:24:38.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:24:38.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:24:38.546 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:24:38 smithi047 ceph-mon[77147]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T01:24:38.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T01:24:38.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:24:38.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T01:24:38.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:24:38.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:24:38.548 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:24:38.548 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T01:24:38.548 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:24:38.548 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T01:24:38.549 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:24:38.549 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T01:24:38.549 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:24:38.549 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T01:24:38.550 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:24:38.550 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T01:24:38.550 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:24:38.550 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:24:38.551 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:24:38.551 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T01:24:38.551 INFO:journalctl@ceph.mon.b.smithi047.stdout: During handling of the above exception, another exception occurred: 2022-05-14T01:24:38.551 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T01:24:38.551 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T01:24:38.552 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T01:24:38.552 INFO:journalctl@ceph.mon.b.smithi047.stdout: return f(*arg) 2022-05-14T01:24:38.552 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T01:24:38.552 INFO:journalctl@ceph.mon.b.smithi047.stdout: self._write_client_files(client_files, host) 2022-05-14T01:24:38.552 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T01:24:38.553 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T01:24:38.553 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T01:24:38.553 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T01:24:38.553 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T01:24:38.554 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.event_loop.get_result(coro) 2022-05-14T01:24:38.554 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T01:24:38.554 INFO:journalctl@ceph.mon.b.smithi047.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T01:24:38.554 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T01:24:38.554 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.__get_result() 2022-05-14T01:24:38.555 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T01:24:38.555 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise self._exception 2022-05-14T01:24:38.555 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T01:24:38.555 INFO:journalctl@ceph.mon.b.smithi047.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T01:24:38.556 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T01:24:38.556 INFO:journalctl@ceph.mon.b.smithi047.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T01:24:38.556 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T01:24:38.556 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise 2022-05-14T01:24:38.556 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T01:24:38.557 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.gen.throw(type, value, traceback) 2022-05-14T01:24:38.557 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T01:24:38.557 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OrchestratorError(msg) 2022-05-14T01:24:38.557 INFO:journalctl@ceph.mon.b.smithi047.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:24:38.558 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:24:38 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:24:38.558 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:24:38 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:24:38.558 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:24:38 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:24:38.558 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:24:38 smithi047 ceph-mon[77147]: pgmap v3858: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:24:38.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:24:38 smithi115 conmon[90393]: 2022-05-14T01:24:38.272+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:24:38.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:24:38 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:24:38.608 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T01:24:38.608 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:24:38.609 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T01:24:38.609 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:24:38.609 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:24:38.609 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:24:38.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T01:24:38.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:24:38.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T01:24:38.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:24:38.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T01:24:38.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:24:38.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T01:24:38.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:24:38.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T01:24:38.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:24:38.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:24:38.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:24:38.613 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:24:38 smithi115 ceph-mon[78593]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T01:24:38.613 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T01:24:38.613 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:24:38.613 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T01:24:38.613 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:24:38.614 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:24:38.614 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:24:38.614 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T01:24:38.614 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:24:38.614 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T01:24:38.615 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:24:38.615 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T01:24:38.615 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:24:38.615 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T01:24:38.616 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:24:38.616 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T01:24:38.616 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:24:38.616 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:24:38.616 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:24:38.617 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T01:24:38.617 INFO:journalctl@ceph.mon.c.smithi115.stdout: During handling of the above exception, another exception occurred: 2022-05-14T01:24:38.617 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T01:24:38.617 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T01:24:38.618 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T01:24:38.618 INFO:journalctl@ceph.mon.c.smithi115.stdout: return f(*arg) 2022-05-14T01:24:38.618 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T01:24:38.618 INFO:journalctl@ceph.mon.c.smithi115.stdout: self._write_client_files(client_files, host) 2022-05-14T01:24:38.618 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T01:24:38.619 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T01:24:38.619 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T01:24:38.619 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T01:24:38.619 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T01:24:38.620 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.event_loop.get_result(coro) 2022-05-14T01:24:38.620 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T01:24:38.620 INFO:journalctl@ceph.mon.c.smithi115.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T01:24:38.620 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T01:24:38.620 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.__get_result() 2022-05-14T01:24:38.621 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T01:24:38.621 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise self._exception 2022-05-14T01:24:38.621 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T01:24:38.621 INFO:journalctl@ceph.mon.c.smithi115.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T01:24:38.621 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T01:24:38.622 INFO:journalctl@ceph.mon.c.smithi115.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T01:24:38.622 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T01:24:38.622 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise 2022-05-14T01:24:38.622 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T01:24:38.623 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.gen.throw(type, value, traceback) 2022-05-14T01:24:38.623 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T01:24:38.623 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OrchestratorError(msg) 2022-05-14T01:24:38.623 INFO:journalctl@ceph.mon.c.smithi115.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:24:38.623 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:24:38 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:24:38.624 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:24:38 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:24:38.624 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:24:38 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:24:38.624 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:24:38 smithi115 ceph-mon[78593]: pgmap v3858: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:24:39.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:24:39 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:24:39.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:24:39 smithi115 conmon[90393]: 2022-05-14T01:24:39.261+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:24:39.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:24:39 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:24:40.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:24:40 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:24:40.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:24:40 smithi047 ceph-mon[77147]: pgmap v3859: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:24:40.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:24:40 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 7010 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:24:40.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:24:40 smithi115 conmon[90393]: 2022-05-14T01:24:40.263+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:24:40.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:24:40 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:24:40.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:24:40 smithi115 ceph-mon[78593]: pgmap v3859: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:24:40.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:24:40 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 7010 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:24:41.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:24:41 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:24:41.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:24:41 smithi115 conmon[90393]: 2022-05-14T01:24:41.217+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:24:41.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:24:41 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:24:42.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:24:42 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:24:42.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:24:42 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:24:42.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:24:42 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:24:42.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:24:42 smithi047 ceph-mon[77147]: pgmap v3860: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:24:42.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:24:42 smithi115 conmon[90393]: 2022-05-14T01:24:42.208+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:24:42.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:24:42 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:24:42.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:24:42 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:24:42.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:24:42 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:24:42.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:24:42 smithi115 ceph-mon[78593]: pgmap v3860: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:24:43.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:24:43 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:24:43.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:24:43 smithi115 conmon[90393]: 2022-05-14T01:24:43.188+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:24:43.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:24:43 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:24:44.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:24:44 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:24:44.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:24:44 smithi047 ceph-mon[77147]: pgmap v3861: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:24:44.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:24:44 smithi115 conmon[90393]: 2022-05-14T01:24:44.196+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:24:44.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:24:44 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:24:44.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:24:44 smithi115 ceph-mon[78593]: pgmap v3861: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:24:45.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:24:45 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:24:45.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:24:45 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 7015 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:24:45.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:24:45 smithi115 conmon[90393]: 2022-05-14T01:24:45.245+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:24:45.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:24:45 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:24:45.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:24:45 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 7015 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:24:46.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:24:46 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:24:46.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:24:46 smithi047 ceph-mon[77147]: pgmap v3862: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:24:46.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:24:46 smithi115 conmon[90393]: 2022-05-14T01:24:46.222+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:24:46.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:24:46 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:24:46.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:24:46 smithi115 ceph-mon[78593]: pgmap v3862: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:24:47.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:24:47 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:24:47.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:24:47 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:24:47.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:24:47 smithi115 conmon[90393]: 2022-05-14T01:24:47.252+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:24:48.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:24:48 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:24:48.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:24:48 smithi047 ceph-mon[77147]: pgmap v3863: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:24:48.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:24:48 smithi115 conmon[90393]: 2022-05-14T01:24:48.255+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:24:48.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:24:48 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:24:48.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:24:48 smithi115 ceph-mon[78593]: pgmap v3863: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:24:49.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:24:49 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:24:49.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:24:49 smithi115 conmon[90393]: 2022-05-14T01:24:49.268+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:24:49.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:24:49 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:24:50.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:24:50 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:24:50.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:24:50 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:24:50.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:24:50 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:24:50.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:24:50 smithi047 ceph-mon[77147]: pgmap v3864: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:24:50.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:24:50 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 7020 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:24:50.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:24:50 smithi115 conmon[90393]: 2022-05-14T01:24:50.261+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:24:50.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:24:50 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:24:50.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:24:50 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:24:50.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:24:50 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:24:50.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:24:50 smithi115 ceph-mon[78593]: pgmap v3864: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:24:50.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:24:50 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 7020 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:24:51.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:24:51 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:24:51.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:24:51 smithi115 conmon[90393]: 2022-05-14T01:24:51.274+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:24:51.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:24:51 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:24:52.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:24:52 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:24:52.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:24:52 smithi047 ceph-mon[77147]: pgmap v3865: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:24:52.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:24:52 smithi115 conmon[90393]: 2022-05-14T01:24:52.251+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:24:52.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:24:52 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:24:52.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:24:52 smithi115 ceph-mon[78593]: pgmap v3865: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:24:53.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:24:53 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:24:53.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:24:53 smithi115 conmon[90393]: 2022-05-14T01:24:53.242+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:24:53.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:24:53 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:24:54.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:24:54 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:24:54.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:24:54 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:24:54.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:24:54 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:24:54.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:24:54 smithi047 ceph-mon[77147]: pgmap v3866: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:24:54.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:24:54 smithi115 conmon[90393]: 2022-05-14T01:24:54.288+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:24:54.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:24:54 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:24:54.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:24:54 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:24:54.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:24:54 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:24:54.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:24:54 smithi115 ceph-mon[78593]: pgmap v3866: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:24:55.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:24:55 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:24:55.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:24:55 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 7025 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:24:55.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:24:55 smithi115 conmon[90393]: 2022-05-14T01:24:55.245+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:24:55.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:24:55 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:24:55.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:24:55 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 7025 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:24:56.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:24:56 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:24:56.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:24:56 smithi047 ceph-mon[77147]: pgmap v3867: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:24:56.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:24:56 smithi115 conmon[90393]: 2022-05-14T01:24:56.265+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:24:56.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:24:56 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:24:56.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:24:56 smithi115 ceph-mon[78593]: pgmap v3867: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:24:57.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:24:57 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:24:57.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:24:57 smithi115 conmon[90393]: 2022-05-14T01:24:57.264+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:24:57.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:24:57 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:24:58.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:24:58 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:24:58.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:24:58 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:24:58.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:24:58 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:24:58.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:24:58 smithi047 ceph-mon[77147]: pgmap v3868: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:24:58.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:24:58 smithi115 conmon[90393]: 2022-05-14T01:24:58.298+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:24:58.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:24:58 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:24:58.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:24:58 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:24:58.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:24:58 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:24:58.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:24:58 smithi115 ceph-mon[78593]: pgmap v3868: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:24:59.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:24:59 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:24:59.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:24:59 smithi115 conmon[90393]: 2022-05-14T01:24:59.303+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:24:59.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:24:59 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:25:00.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:25:00 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:25:00.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:25:00 smithi047 ceph-mon[77147]: pgmap v3869: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:25:00.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:25:00 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 7030 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:25:00.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:25:00 smithi115 conmon[90393]: 2022-05-14T01:25:00.352+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:25:00.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:25:00 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:25:00.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:25:00 smithi115 ceph-mon[78593]: pgmap v3869: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:25:00.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:25:00 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 7030 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:25:01.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:25:01 smithi115 conmon[90393]: 2022-05-14T01:25:01.402+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:25:02.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:25:02 smithi115 conmon[90393]: 2022-05-14T01:25:02.381+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:25:02.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:25:02 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:25:02.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:25:02 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:25:02.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:25:02 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:25:02.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:25:02 smithi115 ceph-mon[78593]: pgmap v3870: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:25:02.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:25:02 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:25:02.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:25:02 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:25:02.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:25:02 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:25:02.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:25:02 smithi047 ceph-mon[77147]: pgmap v3870: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:25:03.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:25:03 smithi115 conmon[90393]: 2022-05-14T01:25:03.403+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:25:03.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:25:03 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:25:03.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:25:03 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:25:04.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:25:04 smithi115 conmon[90393]: 2022-05-14T01:25:04.399+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:25:04.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:25:04 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:25:04.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:25:04 smithi115 ceph-mon[78593]: pgmap v3871: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:25:04.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:25:04 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:25:04.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:25:04 smithi047 ceph-mon[77147]: pgmap v3871: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:25:05.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:25:05 smithi115 conmon[90393]: 2022-05-14T01:25:05.413+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:25:05.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:25:05 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:25:05.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:25:05 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 7035 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:25:05.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:25:05 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:25:05.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:25:05 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 7035 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:25:06.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:25:06 smithi115 conmon[90393]: 2022-05-14T01:25:06.369+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:25:06.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:25:06 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:25:06.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:25:06 smithi115 ceph-mon[78593]: pgmap v3872: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:25:06.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:25:06 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:25:06.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:25:06 smithi047 ceph-mon[77147]: pgmap v3872: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:25:07.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:25:07 smithi115 conmon[90393]: 2022-05-14T01:25:07.409+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:25:07.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:25:07 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:25:07.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:25:07 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:25:08.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:25:08 smithi115 conmon[90393]: 2022-05-14T01:25:08.415+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:25:08.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:25:08 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:25:08.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:25:08 smithi115 ceph-mon[78593]: pgmap v3873: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:25:08.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:25:08 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:25:08.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:25:08 smithi047 ceph-mon[77147]: pgmap v3873: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:25:09.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:25:09 smithi115 conmon[90393]: 2022-05-14T01:25:09.418+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:25:09.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:25:09 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:25:09.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:25:09 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:25:10.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:25:10 smithi115 conmon[90393]: 2022-05-14T01:25:10.397+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:25:10.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:25:10 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:25:10.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:25:10 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:25:10.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:25:10 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:25:10.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:25:10 smithi115 ceph-mon[78593]: pgmap v3874: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:25:10.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:25:10 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 7040 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:25:10.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:25:10 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:25:10.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:25:10 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:25:10.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:25:10 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:25:10.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:25:10 smithi047 ceph-mon[77147]: pgmap v3874: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:25:10.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:25:10 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 7040 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:25:11.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:25:11 smithi115 conmon[90393]: 2022-05-14T01:25:11.349+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:25:11.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:25:11 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:25:11.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:25:11 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:25:12.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:25:12 smithi115 conmon[90393]: 2022-05-14T01:25:12.360+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:25:12.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:25:12 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:25:12.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:25:12 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:25:12.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:25:12 smithi115 ceph-mon[78593]: pgmap v3875: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:25:12.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:25:12 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:25:12.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:25:12 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:25:12.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:25:12 smithi047 ceph-mon[77147]: pgmap v3875: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:25:13.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:25:13 smithi115 conmon[90393]: 2022-05-14T01:25:13.325+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:25:14.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:25:14 smithi115 conmon[90393]: 2022-05-14T01:25:14.291+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:25:14.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:25:14 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:25:14.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:25:14 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:25:14.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:25:14 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:25:14.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:25:14 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:25:14.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:25:14 smithi115 ceph-mon[78593]: pgmap v3876: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:25:14.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:25:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T01:25:14.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:25:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T01:25:14.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:25:14 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:25:14.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:25:14 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:25:14.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:25:14 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:25:14.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:25:14 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:25:14.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:25:14 smithi047 ceph-mon[77147]: pgmap v3876: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:25:14.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:25:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T01:25:14.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:25:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T01:25:15.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:25:15 smithi115 conmon[90393]: 2022-05-14T01:25:15.323+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:25:15.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:25:15 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:25:15.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:25:15 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 7045 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:25:15.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:25:15 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:25:15.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:25:15 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 7045 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:25:16.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:25:16 smithi115 conmon[90393]: 2022-05-14T01:25:16.287+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:25:16.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:25:16 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:25:16.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:25:16 smithi115 ceph-mon[78593]: pgmap v3877: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:25:16.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:25:16 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:25:16.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:25:16 smithi047 ceph-mon[77147]: pgmap v3877: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:25:17.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:25:17 smithi115 conmon[90393]: 2022-05-14T01:25:17.322+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:25:17.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:25:17 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:25:17.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:25:17 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:25:18.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:25:18 smithi115 conmon[90393]: 2022-05-14T01:25:18.342+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:25:18.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:25:18 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:25:18.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:25:18 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:25:18.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:25:18 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:25:18.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:25:18 smithi115 ceph-mon[78593]: pgmap v3878: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:25:18.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:25:18 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:25:18.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:25:18 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:25:18.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:25:18 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:25:18.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:25:18 smithi047 ceph-mon[77147]: pgmap v3878: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:25:19.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:25:19 smithi115 conmon[90393]: 2022-05-14T01:25:19.349+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:25:19.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:25:19 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:25:19.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:25:19 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:25:19.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:25:19 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:25:19.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:25:19 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:25:20.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:25:20 smithi115 conmon[90393]: 2022-05-14T01:25:20.324+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:25:20.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:25:20 smithi115 ceph-mon[78593]: pgmap v3879: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:25:20.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:25:20 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 7050 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:25:20.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:25:20 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:25:20.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:25:20 smithi047 ceph-mon[77147]: pgmap v3879: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:25:20.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:25:20 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 7050 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:25:20.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:25:20 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:25:21.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:25:21 smithi115 conmon[90393]: 2022-05-14T01:25:21.283+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:25:21.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:25:21 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:25:21.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:25:21 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:25:22.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:25:22 smithi115 conmon[90393]: 2022-05-14T01:25:22.312+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:25:22.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:25:22 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:25:22.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:25:22 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:25:22.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:25:22 smithi115 ceph-mon[78593]: pgmap v3880: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:25:22.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:25:22 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:25:22.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:25:22 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:25:22.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:25:22 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:25:22.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:25:22 smithi047 ceph-mon[77147]: pgmap v3880: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:25:22.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:25:22 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:25:23.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:25:23 smithi115 conmon[90393]: 2022-05-14T01:25:23.351+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:25:23.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:25:23 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:25:23.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:25:23 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:25:24.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:25:24 smithi115 conmon[90393]: 2022-05-14T01:25:24.335+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:25:24.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:25:24 smithi115 ceph-mon[78593]: pgmap v3881: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:25:24.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:25:24 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:25:24.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:25:24 smithi047 ceph-mon[77147]: pgmap v3881: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:25:24.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:25:24 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:25:25.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:25:25 smithi115 conmon[90393]: 2022-05-14T01:25:25.296+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:25:25.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:25:25 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 7055 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:25:25.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:25:25 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:25:25.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:25:25 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 7055 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:25:25.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:25:25 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:25:26.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:25:26 smithi115 conmon[90393]: 2022-05-14T01:25:26.267+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:25:26.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:25:26 smithi115 ceph-mon[78593]: pgmap v3882: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:25:26.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:25:26 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:25:26.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:25:26 smithi047 ceph-mon[77147]: pgmap v3882: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:25:26.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:25:26 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:25:27.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:25:27 smithi115 conmon[90393]: 2022-05-14T01:25:27.291+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:25:27.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:25:27 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:25:27.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:25:27 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:25:28.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:25:28 smithi115 conmon[90393]: 2022-05-14T01:25:28.328+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:25:28.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:25:28 smithi115 ceph-mon[78593]: pgmap v3883: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:25:28.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:25:28 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:25:28.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:25:28 smithi047 ceph-mon[77147]: pgmap v3883: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:25:28.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:25:28 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:25:29.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:25:29 smithi115 conmon[90393]: 2022-05-14T01:25:29.317+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:25:29.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:25:29 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:25:29.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:25:29 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:25:30.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:25:30 smithi115 conmon[90393]: 2022-05-14T01:25:30.288+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:25:30.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:25:30 smithi115 ceph-mon[78593]: pgmap v3884: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:25:30.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:25:30 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:25:30.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:25:30 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:25:30.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:25:30 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 7065 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:25:30.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:25:30 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:25:30.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:25:30 smithi047 ceph-mon[77147]: pgmap v3884: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:25:30.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:25:30 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:25:30.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:25:30 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:25:30.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:25:30 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 7065 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:25:30.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:25:30 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:25:31.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:25:31 smithi115 conmon[90393]: 2022-05-14T01:25:31.290+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:25:31.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:25:31 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:25:31.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:25:31 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:25:32.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:25:32 smithi115 conmon[90393]: 2022-05-14T01:25:32.322+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:25:32.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:25:32 smithi115 ceph-mon[78593]: pgmap v3885: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:25:32.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:25:32 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:25:32.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:25:32 smithi047 ceph-mon[77147]: pgmap v3885: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:25:32.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:25:32 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:25:33.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:25:33 smithi115 conmon[90393]: 2022-05-14T01:25:33.313+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:25:33.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:25:33 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:25:33.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:25:33 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:25:34.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:25:34 smithi115 conmon[90393]: 2022-05-14T01:25:34.275+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:25:34.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:25:34 smithi115 ceph-mon[78593]: pgmap v3886: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:25:34.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:25:34 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:25:34.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:25:34 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:25:34.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:25:34 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:25:34.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:25:34 smithi047 ceph-mon[77147]: pgmap v3886: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:25:34.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:25:34 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:25:34.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:25:34 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:25:34.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:25:34 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:25:35.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:25:35 smithi115 conmon[90393]: 2022-05-14T01:25:35.300+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:25:36.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:25:35 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:25:36.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:25:35 smithi047 ceph-mon[77147]: pgmap v3887: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:25:36.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:25:35 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:25:36.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:25:35 smithi115 ceph-mon[78593]: pgmap v3887: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:25:36.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:25:36 smithi115 conmon[90393]: 2022-05-14T01:25:36.310+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:25:37.045 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:25:36 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 7070 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:25:37.045 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:25:36 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:25:37.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:25:36 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 7070 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:25:37.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:25:36 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:25:37.541 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:25:37 smithi047 conmon[78521]: 2022-05-14T01:25:37.263+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:25:37.542 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:25:37 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T01:25:37.542 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:25:37 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:25:37.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:25:37 smithi047 conmon[78521]: yield 2022-05-14T01:25:37.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:25:37 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:25:37.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:25:37 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:25:37.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:25:37 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:25:37.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:25:37 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T01:25:37.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:25:37 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:25:37.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:25:37 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T01:25:37.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:25:37 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:25:37.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:25:37 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T01:25:37.547 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:25:37 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:25:37.547 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:25:37 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T01:25:37.547 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:25:37 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:25:37.547 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:25:37 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T01:25:37.548 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:25:37 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:25:37.548 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:25:37 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:25:37.548 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:25:37 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:25:37.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:25:37 smithi115 conmon[90393]: 2022-05-14T01:25:37.352+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:25:38.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:25:37 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:25:38.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:25:37 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:25:38.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:25:37 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T01:25:38.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:25:37 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T01:25:38.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:25:37 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T01:25:38.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:25:37 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:25:38.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T01:25:38.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:25:38.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T01:25:38.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:25:38.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:25:38.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:25:38.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T01:25:38.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:25:38.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T01:25:38.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:25:38.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T01:25:38.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:25:38.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T01:25:38.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:25:38.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T01:25:38.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:25:38.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:25:38.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:25:38.047 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:25:37 smithi047 ceph-mon[77147]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T01:25:38.047 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:25:37 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:25:38.047 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:25:37 smithi047 ceph-mon[77147]: pgmap v3888: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:25:38.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:25:37 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:25:38.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:25:37 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:25:38.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:25:37 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T01:25:38.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:25:37 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T01:25:38.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:25:37 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T01:25:38.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:25:37 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:25:38.109 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T01:25:38.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:25:38.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T01:25:38.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:25:38.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:25:38.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:25:38.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T01:25:38.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:25:38.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T01:25:38.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:25:38.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T01:25:38.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:25:38.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T01:25:38.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:25:38.113 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T01:25:38.113 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:25:38.113 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:25:38.113 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:25:38.114 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:25:37 smithi115 ceph-mon[78593]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T01:25:38.114 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:25:37 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:25:38.114 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:25:37 smithi115 ceph-mon[78593]: pgmap v3888: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:25:38.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:25:38 smithi115 conmon[90393]: 2022-05-14T01:25:38.347+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:25:39.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:25:38 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:25:39.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:25:38 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:25:39.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:25:39 smithi115 conmon[90393]: 2022-05-14T01:25:39.330+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:25:40.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:25:39 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:25:40.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:25:39 smithi047 ceph-mon[77147]: pgmap v3889: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:25:40.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:25:39 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:25:40.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:25:39 smithi115 ceph-mon[78593]: pgmap v3889: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:25:40.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:25:40 smithi115 conmon[90393]: 2022-05-14T01:25:40.352+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:25:40.785 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:25:40 smithi047 conmon[78521]: 2022-05-14T01:25:40.335+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:25:40.786 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:25:40 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T01:25:40.786 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:25:40 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:25:40.786 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:25:40 smithi047 conmon[78521]: yield 2022-05-14T01:25:40.787 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:25:40 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:25:40.787 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:25:40 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:25:40.787 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:25:40 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:25:40.787 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:25:40 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T01:25:40.787 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:25:40 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:25:40.788 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:25:40 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T01:25:40.788 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:25:40 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:25:40.788 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:25:40 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T01:25:40.788 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:25:40 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:25:40.789 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:25:40 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T01:25:40.789 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:25:40 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:25:40.789 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:25:40 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T01:25:40.789 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:25:40 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:25:40.789 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:25:40 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:25:40.790 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:25:40 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:25:40.790 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:25:40 smithi047 conmon[78521]: 2022-05-14T01:25:40.336+0000 7f5344781700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T01:25:40.790 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:25:40 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T01:25:40.790 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:25:40 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:25:40.791 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:25:40 smithi047 conmon[78521]: yield 2022-05-14T01:25:40.791 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:25:40 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:25:40.791 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:25:40 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:25:40.791 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:25:40 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:25:40.791 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:25:40 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T01:25:40.792 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:25:40 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:25:40.792 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:25:40 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T01:25:40.792 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:25:40 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:25:40.792 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:25:40 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T01:25:40.793 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:25:40 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:25:40.793 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:25:40 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T01:25:40.793 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:25:40 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:25:40.793 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:25:40 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T01:25:40.793 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:25:40 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:25:40.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:25:40 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:25:40.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:25:40 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:25:40.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:25:40 smithi047 conmon[78521]: 2022-05-14T01:25:40.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:25:40 smithi047 conmon[78521]: During handling of the above exception, another exception occurred: 2022-05-14T01:25:40.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:25:40 smithi047 conmon[78521]: 2022-05-14T01:25:40.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:25:40 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T01:25:40.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:25:40 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T01:25:40.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:25:40 smithi047 conmon[78521]: return f(*arg) 2022-05-14T01:25:40.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:25:40 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T01:25:40.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:25:40 smithi047 conmon[78521]: self._write_client_files(client_files, host) 2022-05-14T01:25:40.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:25:40 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T01:25:40.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:25:40 smithi047 conmon[78521]: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T01:25:40.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:25:40 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T01:25:40.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:25:40 smithi047 conmon[78521]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T01:25:40.797 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:25:40 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T01:25:40.797 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:25:40 smithi047 conmon[78521]: return self.event_loop.get_result(coro) 2022-05-14T01:25:40.798 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:25:40 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T01:25:40.798 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:25:40 smithi047 conmon[78521]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T01:25:40.798 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:25:40 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T01:25:40.798 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:25:40 smithi047 conmon[78521]: return self.__get_result() 2022-05-14T01:25:40.798 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:25:40 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T01:25:40.799 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:25:40 smithi047 conmon[78521]: raise self._exception 2022-05-14T01:25:40.799 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:25:40 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T01:25:40.799 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:25:40 smithi047 conmon[78521]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T01:25:40.799 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:25:40 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T01:25:40.800 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:25:40 smithi047 conmon[78521]: conn = await self._remote_connection(host, addr) 2022-05-14T01:25:40.800 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:25:40 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T01:25:40.800 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:25:40 smithi047 conmon[78521]: raise 2022-05-14T01:25:40.800 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:25:40 smithi047 conmon[78521]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T01:25:40.800 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:25:40 smithi047 conmon[78521]: self.gen.throw(type, value, traceback) 2022-05-14T01:25:40.801 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:25:40 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T01:25:40.801 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:25:40 smithi047 conmon[78521]: raise OrchestratorError(msg) 2022-05-14T01:25:40.801 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:25:40 smithi047 conmon[78521]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:25:41.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:25:40 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:25:41.042 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T01:25:41.042 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:25:41.042 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T01:25:41.042 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:25:41.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:25:41.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:25:41.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T01:25:41.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:25:41.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T01:25:41.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:25:41.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T01:25:41.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:25:41.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T01:25:41.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:25:41.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T01:25:41.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:25:41.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:25:41.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:25:41.046 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:25:40 smithi047 ceph-mon[77147]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T01:25:41.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T01:25:41.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:25:41.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T01:25:41.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:25:41.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:25:41.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:25:41.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T01:25:41.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:25:41.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T01:25:41.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:25:41.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T01:25:41.049 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:25:41.049 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T01:25:41.049 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:25:41.049 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T01:25:41.050 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:25:41.050 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:25:41.050 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:25:41.050 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T01:25:41.051 INFO:journalctl@ceph.mon.b.smithi047.stdout: During handling of the above exception, another exception occurred: 2022-05-14T01:25:41.051 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T01:25:41.051 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T01:25:41.051 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T01:25:41.051 INFO:journalctl@ceph.mon.b.smithi047.stdout: return f(*arg) 2022-05-14T01:25:41.052 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T01:25:41.052 INFO:journalctl@ceph.mon.b.smithi047.stdout: self._write_client_files(client_files, host) 2022-05-14T01:25:41.052 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T01:25:41.052 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T01:25:41.052 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T01:25:41.053 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T01:25:41.053 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T01:25:41.053 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.event_loop.get_result(coro) 2022-05-14T01:25:41.053 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T01:25:41.054 INFO:journalctl@ceph.mon.b.smithi047.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T01:25:41.054 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T01:25:41.054 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.__get_result() 2022-05-14T01:25:41.054 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T01:25:41.054 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise self._exception 2022-05-14T01:25:41.055 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T01:25:41.055 INFO:journalctl@ceph.mon.b.smithi047.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T01:25:41.055 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T01:25:41.055 INFO:journalctl@ceph.mon.b.smithi047.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T01:25:41.056 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T01:25:41.056 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise 2022-05-14T01:25:41.056 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T01:25:41.056 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.gen.throw(type, value, traceback) 2022-05-14T01:25:41.056 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T01:25:41.057 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OrchestratorError(msg) 2022-05-14T01:25:41.057 INFO:journalctl@ceph.mon.b.smithi047.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:25:41.057 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:25:40 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:25:41.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:25:40 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:25:41.108 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T01:25:41.108 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:25:41.108 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T01:25:41.108 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:25:41.109 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:25:41.109 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:25:41.109 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T01:25:41.109 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:25:41.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T01:25:41.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:25:41.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T01:25:41.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:25:41.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T01:25:41.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:25:41.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T01:25:41.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:25:41.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:25:41.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:25:41.112 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:25:40 smithi115 ceph-mon[78593]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T01:25:41.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T01:25:41.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:25:41.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T01:25:41.113 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:25:41.113 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:25:41.113 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:25:41.113 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T01:25:41.114 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:25:41.114 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T01:25:41.114 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:25:41.114 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T01:25:41.114 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:25:41.115 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T01:25:41.115 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:25:41.115 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T01:25:41.115 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:25:41.115 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:25:41.116 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:25:41.116 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T01:25:41.116 INFO:journalctl@ceph.mon.c.smithi115.stdout: During handling of the above exception, another exception occurred: 2022-05-14T01:25:41.116 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T01:25:41.117 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T01:25:41.117 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T01:25:41.117 INFO:journalctl@ceph.mon.c.smithi115.stdout: return f(*arg) 2022-05-14T01:25:41.117 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T01:25:41.118 INFO:journalctl@ceph.mon.c.smithi115.stdout: self._write_client_files(client_files, host) 2022-05-14T01:25:41.118 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T01:25:41.118 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T01:25:41.118 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T01:25:41.118 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T01:25:41.119 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T01:25:41.119 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.event_loop.get_result(coro) 2022-05-14T01:25:41.119 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T01:25:41.119 INFO:journalctl@ceph.mon.c.smithi115.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T01:25:41.119 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T01:25:41.120 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.__get_result() 2022-05-14T01:25:41.120 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T01:25:41.120 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise self._exception 2022-05-14T01:25:41.120 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T01:25:41.120 INFO:journalctl@ceph.mon.c.smithi115.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T01:25:41.121 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T01:25:41.121 INFO:journalctl@ceph.mon.c.smithi115.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T01:25:41.121 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T01:25:41.121 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise 2022-05-14T01:25:41.122 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T01:25:41.122 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.gen.throw(type, value, traceback) 2022-05-14T01:25:41.122 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T01:25:41.122 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OrchestratorError(msg) 2022-05-14T01:25:41.122 INFO:journalctl@ceph.mon.c.smithi115.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:25:41.123 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:25:40 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:25:41.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:25:41 smithi115 conmon[90393]: 2022-05-14T01:25:41.337+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:25:42.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:25:41 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:25:42.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:25:41 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:25:42.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:25:41 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:25:42.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:25:41 smithi115 ceph-mon[78593]: pgmap v3890: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:25:42.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:25:41 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:25:42.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:25:41 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:25:42.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:25:41 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:25:42.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:25:41 smithi047 ceph-mon[77147]: pgmap v3890: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:25:42.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:25:42 smithi115 conmon[90393]: 2022-05-14T01:25:42.303+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:25:43.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:25:42 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:25:43.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:25:42 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:25:43.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:25:43 smithi115 conmon[90393]: 2022-05-14T01:25:43.332+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:25:44.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:25:43 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:25:44.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:25:43 smithi115 ceph-mon[78593]: pgmap v3891: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:25:44.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:25:43 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:25:44.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:25:43 smithi047 ceph-mon[77147]: pgmap v3891: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:25:44.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:25:44 smithi115 conmon[90393]: 2022-05-14T01:25:44.317+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:25:45.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:25:44 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:25:45.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:25:44 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 7075 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:25:45.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:25:44 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:25:45.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:25:44 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 7075 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:25:45.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:25:45 smithi115 conmon[90393]: 2022-05-14T01:25:45.309+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:25:46.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:25:45 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:25:46.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:25:45 smithi115 ceph-mon[78593]: pgmap v3892: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:25:46.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:25:45 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:25:46.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:25:45 smithi047 ceph-mon[77147]: pgmap v3892: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:25:46.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:25:46 smithi115 conmon[90393]: 2022-05-14T01:25:46.306+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:25:47.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:25:46 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:25:47.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:25:46 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:25:47.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:25:47 smithi115 conmon[90393]: 2022-05-14T01:25:47.260+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:25:48.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:25:47 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:25:48.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:25:47 smithi115 ceph-mon[78593]: pgmap v3893: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:25:48.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:25:47 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:25:48.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:25:47 smithi047 ceph-mon[77147]: pgmap v3893: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:25:48.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:25:48 smithi115 conmon[90393]: 2022-05-14T01:25:48.264+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:25:49.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:25:48 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:25:49.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:25:48 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:25:49.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:25:49 smithi115 conmon[90393]: 2022-05-14T01:25:49.282+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:25:50.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:25:49 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:25:50.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:25:49 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:25:50.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:25:49 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:25:50.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:25:49 smithi115 ceph-mon[78593]: pgmap v3894: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:25:50.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:25:49 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 7080 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:25:50.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:25:49 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:25:50.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:25:49 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:25:50.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:25:49 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:25:50.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:25:49 smithi047 ceph-mon[77147]: pgmap v3894: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:25:50.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:25:49 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 7080 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:25:50.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:25:50 smithi115 conmon[90393]: 2022-05-14T01:25:50.310+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:25:51.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:25:50 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:25:51.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:25:50 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:25:51.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:25:51 smithi115 conmon[90393]: 2022-05-14T01:25:51.353+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:25:52.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:25:51 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:25:52.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:25:51 smithi115 ceph-mon[78593]: pgmap v3895: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:25:52.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:25:51 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:25:52.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:25:51 smithi047 ceph-mon[77147]: pgmap v3895: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:25:52.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:25:52 smithi115 conmon[90393]: 2022-05-14T01:25:52.345+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:25:53.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:25:52 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:25:53.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:25:52 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:25:53.841 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:25:53 smithi115 conmon[90393]: 2022-05-14T01:25:53.358+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:25:54.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:25:53 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:25:54.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:25:53 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:25:54.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:25:53 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:25:54.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:25:53 smithi115 ceph-mon[78593]: pgmap v3896: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:25:54.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:25:53 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:25:54.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:25:53 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:25:54.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:25:53 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:25:54.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:25:53 smithi047 ceph-mon[77147]: pgmap v3896: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:25:54.842 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:25:54 smithi115 conmon[90393]: 2022-05-14T01:25:54.399+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:25:55.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:25:54 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 7085 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:25:55.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:25:54 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 7085 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:25:55.847 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:25:55 smithi115 conmon[90393]: 2022-05-14T01:25:55.390+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:25:56.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:25:55 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:25:56.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:25:55 smithi115 ceph-mon[78593]: pgmap v3897: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:25:56.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:25:55 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:25:56.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:25:55 smithi047 ceph-mon[77147]: pgmap v3897: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:25:56.849 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:25:56 smithi115 conmon[90393]: 2022-05-14T01:25:56.362+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:25:57.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:25:56 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:25:57.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:25:56 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:25:57.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:25:57 smithi115 conmon[90393]: 2022-05-14T01:25:57.387+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:25:58.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:25:57 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:25:58.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:25:57 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:25:58.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:25:57 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:25:58.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:25:57 smithi047 ceph-mon[77147]: pgmap v3898: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:25:58.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:25:57 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:25:58.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:25:57 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:25:58.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:25:57 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:25:58.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:25:57 smithi115 ceph-mon[78593]: pgmap v3898: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:25:58.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:25:58 smithi115 conmon[90393]: 2022-05-14T01:25:58.396+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:25:59.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:25:58 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:25:59.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:25:58 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:25:59.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:25:59 smithi115 conmon[90393]: 2022-05-14T01:25:59.383+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:26:00.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:25:59 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:26:00.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:25:59 smithi047 ceph-mon[77147]: pgmap v3899: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:26:00.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:25:59 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 7090 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:26:00.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:25:59 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:26:00.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:25:59 smithi115 ceph-mon[78593]: pgmap v3899: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:26:00.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:25:59 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 7090 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:26:00.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:26:00 smithi115 conmon[90393]: 2022-05-14T01:26:00.394+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:26:01.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:26:00 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:26:01.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:26:00 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:26:01.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:26:01 smithi115 conmon[90393]: 2022-05-14T01:26:01.416+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:26:02.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:26:01 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:26:02.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:26:01 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:26:02.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:26:01 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:26:02.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:26:01 smithi047 ceph-mon[77147]: pgmap v3900: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:26:02.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:26:01 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:26:02.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:26:01 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:26:02.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:26:01 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:26:02.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:26:01 smithi115 ceph-mon[78593]: pgmap v3900: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:26:02.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:26:02 smithi115 conmon[90393]: 2022-05-14T01:26:02.463+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:26:03.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:26:02 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:26:03.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:26:02 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:26:03.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:26:03 smithi115 conmon[90393]: 2022-05-14T01:26:03.417+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:26:04.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:26:03 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:26:04.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:26:03 smithi047 ceph-mon[77147]: pgmap v3901: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:26:04.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:26:03 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:26:04.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:26:03 smithi115 ceph-mon[78593]: pgmap v3901: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:26:04.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:26:04 smithi115 conmon[90393]: 2022-05-14T01:26:04.430+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:26:05.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:26:04 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:26:05.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:26:04 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 7095 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:26:05.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:26:04 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:26:05.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:26:04 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 7095 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:26:05.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:26:05 smithi115 conmon[90393]: 2022-05-14T01:26:05.476+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:26:06.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:26:05 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:26:06.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:26:05 smithi047 ceph-mon[77147]: pgmap v3902: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:26:06.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:26:05 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:26:06.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:26:05 smithi115 ceph-mon[78593]: pgmap v3902: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:26:06.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:26:06 smithi115 conmon[90393]: 2022-05-14T01:26:06.430+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:26:07.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:26:06 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:26:07.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:26:06 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:26:07.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:26:07 smithi115 conmon[90393]: 2022-05-14T01:26:07.461+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:26:08.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:26:07 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:26:08.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:26:07 smithi047 ceph-mon[77147]: pgmap v3903: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:26:08.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:26:07 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:26:08.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:26:07 smithi115 ceph-mon[78593]: pgmap v3903: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:26:08.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:26:08 smithi115 conmon[90393]: 2022-05-14T01:26:08.506+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:26:09.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:26:08 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:26:09.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:26:08 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:26:09.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:26:09 smithi115 conmon[90393]: 2022-05-14T01:26:09.525+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:26:10.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:26:09 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:26:10.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:26:09 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:26:10.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:26:09 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:26:10.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:26:09 smithi047 ceph-mon[77147]: pgmap v3904: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:26:10.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:26:09 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 7100 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:26:10.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:26:09 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:26:10.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:26:09 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:26:10.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:26:09 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:26:10.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:26:09 smithi115 ceph-mon[78593]: pgmap v3904: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:26:10.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:26:09 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 7100 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:26:10.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:26:10 smithi115 conmon[90393]: 2022-05-14T01:26:10.538+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:26:11.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:26:10 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:26:11.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:26:10 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:26:11.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:26:11 smithi115 conmon[90393]: 2022-05-14T01:26:11.493+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:26:12.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:26:11 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:26:12.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:26:11 smithi047 ceph-mon[77147]: pgmap v3905: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:26:12.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:26:11 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:26:12.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:26:11 smithi115 ceph-mon[78593]: pgmap v3905: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:26:12.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:26:12 smithi115 conmon[90393]: 2022-05-14T01:26:12.540+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:26:13.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:26:12 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:26:13.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:26:12 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:26:13.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:26:13 smithi115 conmon[90393]: 2022-05-14T01:26:13.494+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:26:14.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:26:13 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:26:14.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:26:13 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:26:14.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:26:13 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:26:14.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:26:13 smithi047 ceph-mon[77147]: pgmap v3906: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:26:14.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:26:13 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T01:26:14.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:26:13 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T01:26:14.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:26:13 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:26:14.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:26:13 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:26:14.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:26:13 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:26:14.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:26:13 smithi115 ceph-mon[78593]: pgmap v3906: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:26:14.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:26:13 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T01:26:14.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:26:13 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T01:26:14.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:26:14 smithi115 conmon[90393]: 2022-05-14T01:26:14.480+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:26:15.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:26:14 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:26:15.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:26:14 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 7105 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:26:15.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:26:14 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:26:15.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:26:14 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 7105 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:26:15.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:26:15 smithi115 conmon[90393]: 2022-05-14T01:26:15.514+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:26:16.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:26:15 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:26:16.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:26:15 smithi047 ceph-mon[77147]: pgmap v3907: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:26:16.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:26:15 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:26:16.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:26:15 smithi115 ceph-mon[78593]: pgmap v3907: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:26:16.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:26:16 smithi115 conmon[90393]: 2022-05-14T01:26:16.496+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:26:17.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:26:16 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:26:17.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:26:16 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:26:17.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:26:17 smithi115 conmon[90393]: 2022-05-14T01:26:17.489+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:26:18.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:26:17 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:26:18.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:26:17 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:26:18.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:26:17 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:26:18.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:26:17 smithi047 ceph-mon[77147]: pgmap v3908: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:26:18.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:26:17 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:26:18.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:26:17 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:26:18.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:26:17 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:26:18.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:26:17 smithi115 ceph-mon[78593]: pgmap v3908: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:26:18.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:26:18 smithi115 conmon[90393]: 2022-05-14T01:26:18.452+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:26:19.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:26:18 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:26:19.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:26:18 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:26:19.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:26:19 smithi115 conmon[90393]: 2022-05-14T01:26:19.490+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:26:20.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:26:19 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:26:20.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:26:19 smithi047 ceph-mon[77147]: pgmap v3909: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:26:20.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:26:19 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 7110 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:26:20.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:26:19 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:26:20.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:26:19 smithi115 ceph-mon[78593]: pgmap v3909: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:26:20.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:26:19 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 7110 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:26:20.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:26:20 smithi115 conmon[90393]: 2022-05-14T01:26:20.529+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:26:21.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:26:20 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:26:21.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:26:20 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:26:21.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:26:21 smithi115 conmon[90393]: 2022-05-14T01:26:21.554+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:26:22.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:26:21 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:26:22.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:26:21 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:26:22.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:26:21 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:26:22.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:26:21 smithi047 ceph-mon[77147]: pgmap v3910: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:26:22.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:26:21 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:26:22.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:26:21 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:26:22.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:26:21 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:26:22.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:26:21 smithi115 ceph-mon[78593]: pgmap v3910: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:26:22.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:26:22 smithi115 conmon[90393]: 2022-05-14T01:26:22.545+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:26:23.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:26:22 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:26:23.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:26:22 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:26:23.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:26:23 smithi115 conmon[90393]: 2022-05-14T01:26:23.499+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:26:24.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:26:23 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:26:24.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:26:23 smithi047 ceph-mon[77147]: pgmap v3911: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:26:24.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:26:23 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:26:24.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:26:23 smithi115 ceph-mon[78593]: pgmap v3911: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:26:24.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:26:24 smithi115 conmon[90393]: 2022-05-14T01:26:24.464+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:26:25.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:26:24 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:26:25.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:26:24 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 7115 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:26:25.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:26:24 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:26:25.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:26:24 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 7115 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:26:25.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:26:25 smithi115 conmon[90393]: 2022-05-14T01:26:25.512+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:26:26.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:26:25 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:26:26.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:26:25 smithi047 ceph-mon[77147]: pgmap v3912: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:26:26.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:26:25 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:26:26.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:26:25 smithi115 ceph-mon[78593]: pgmap v3912: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:26:26.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:26:26 smithi115 conmon[90393]: 2022-05-14T01:26:26.523+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:26:27.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:26:26 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:26:27.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:26:26 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:26:27.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:26:27 smithi115 conmon[90393]: 2022-05-14T01:26:27.548+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:26:28.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:26:27 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:26:28.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:26:27 smithi047 ceph-mon[77147]: pgmap v3913: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:26:28.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:26:27 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:26:28.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:26:27 smithi115 ceph-mon[78593]: pgmap v3913: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:26:28.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:26:28 smithi115 conmon[90393]: 2022-05-14T01:26:28.568+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:26:29.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:26:28 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:26:29.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:26:28 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:26:29.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:26:29 smithi115 conmon[90393]: 2022-05-14T01:26:29.578+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:26:30.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:26:29 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:26:30.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:26:29 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:26:30.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:26:29 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:26:30.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:26:29 smithi047 ceph-mon[77147]: pgmap v3914: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:26:30.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:26:29 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 7120 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:26:30.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:26:29 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:26:30.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:26:29 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:26:30.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:26:29 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:26:30.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:26:29 smithi115 ceph-mon[78593]: pgmap v3914: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:26:30.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:26:29 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 7120 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:26:30.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:26:30 smithi115 conmon[90393]: 2022-05-14T01:26:30.583+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:26:31.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:26:30 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:26:31.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:26:30 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:26:31.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:26:31 smithi115 conmon[90393]: 2022-05-14T01:26:31.542+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:26:32.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:26:31 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:26:32.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:26:31 smithi047 ceph-mon[77147]: pgmap v3915: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:26:32.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:26:31 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:26:32.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:26:31 smithi115 ceph-mon[78593]: pgmap v3915: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:26:32.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:26:32 smithi115 conmon[90393]: 2022-05-14T01:26:32.510+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:26:33.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:26:32 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:26:33.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:26:33 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:26:33.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:26:33 smithi115 conmon[90393]: 2022-05-14T01:26:33.528+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:26:34.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:26:34 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:26:34.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:26:34 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:26:34.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:26:34 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:26:34.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:26:34 smithi047 ceph-mon[77147]: pgmap v3916: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:26:34.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:26:34 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:26:34.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:26:34 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:26:34.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:26:34 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:26:34.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:26:34 smithi115 ceph-mon[78593]: pgmap v3916: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:26:34.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:26:34 smithi115 conmon[90393]: 2022-05-14T01:26:34.549+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:26:35.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:26:35 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:26:35.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:26:35 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 7125 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:26:35.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:26:35 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:26:35.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:26:35 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 7125 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:26:35.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:26:35 smithi115 conmon[90393]: 2022-05-14T01:26:35.515+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:26:36.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:26:36 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:26:36.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:26:36 smithi047 ceph-mon[77147]: pgmap v3917: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:26:36.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:26:36 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:26:36.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:26:36 smithi115 ceph-mon[78593]: pgmap v3917: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:26:36.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:26:36 smithi115 conmon[90393]: 2022-05-14T01:26:36.490+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:26:37.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:26:37 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:26:37.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:26:37 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:26:37.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:26:37 smithi115 conmon[90393]: 2022-05-14T01:26:37.497+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:26:38.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:26:38 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:26:38.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:26:38 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:26:38.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:26:38 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:26:38.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:26:38 smithi047 ceph-mon[77147]: pgmap v3918: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:26:38.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:26:38 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:26:38.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:26:38 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:26:38.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:26:38 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:26:38.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:26:38 smithi115 ceph-mon[78593]: pgmap v3918: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:26:38.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:26:38 smithi115 conmon[90393]: 2022-05-14T01:26:38.484+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:26:39.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:26:39 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:26:39.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:26:39 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:26:39.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:26:39 smithi115 conmon[90393]: 2022-05-14T01:26:39.514+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:26:40.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:26:40 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:26:40.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:26:40 smithi047 ceph-mon[77147]: pgmap v3919: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:26:40.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:26:40 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 7130 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:26:40.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:26:40 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:26:40.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:26:40 smithi115 ceph-mon[78593]: pgmap v3919: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:26:40.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:26:40 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 7130 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:26:40.843 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:26:40 smithi115 conmon[90393]: 2022-05-14T01:26:40.497+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:26:41.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:26:41 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:26:41.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:26:41 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T01:26:41.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:26:41 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T01:26:41.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:26:41 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T01:26:41.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:26:41 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:26:41.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:26:41 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T01:26:41.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:26:41 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T01:26:41.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:26:41 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T01:26:41.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:26:41 smithi115 conmon[90393]: 2022-05-14T01:26:41.467+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:26:42.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:26:42 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:26:42.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:26:42 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:26:42.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:26:42 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:26:42.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:26:42 smithi115 ceph-mon[78593]: pgmap v3920: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:26:42.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:26:42 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:26:42.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:26:42 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:26:42.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:26:42 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:26:42.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:26:42 smithi047 ceph-mon[77147]: pgmap v3920: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:26:42.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:26:42 smithi115 conmon[90393]: 2022-05-14T01:26:42.447+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:26:43.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:26:43 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:26:43.441 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:26:43 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:26:43.791 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:26:43 smithi047 conmon[78521]: 2022-05-14T01:26:43.440+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:26:43.791 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:26:43 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T01:26:43.791 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:26:43 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:26:43.792 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:26:43 smithi047 conmon[78521]: yield 2022-05-14T01:26:43.792 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:26:43 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:26:43.792 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:26:43 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:26:43.792 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:26:43 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:26:43.793 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:26:43 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T01:26:43.793 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:26:43 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:26:43.793 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:26:43 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T01:26:43.793 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:26:43 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:26:43.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:26:43 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T01:26:43.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:26:43 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:26:43.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:26:43 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T01:26:43.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:26:43 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:26:43.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:26:43 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T01:26:43.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:26:43 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:26:43.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:26:43 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:26:43.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:26:43 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:26:43.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:26:43 smithi115 conmon[90393]: 2022-05-14T01:26:43.435+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:26:44.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:26:44 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:26:44.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:26:44 smithi115 ceph-mon[78593]: pgmap v3921: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:26:44.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:26:44 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:26:44.359 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T01:26:44.359 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:26:44.359 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T01:26:44.359 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:26:44.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:26:44.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:26:44.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T01:26:44.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:26:44.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T01:26:44.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:26:44.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T01:26:44.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:26:44.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T01:26:44.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:26:44.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T01:26:44.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:26:44.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:26:44.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:26:44.363 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:26:44 smithi115 ceph-mon[78593]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T01:26:44.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:26:44 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:26:44.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:26:44 smithi047 ceph-mon[77147]: pgmap v3921: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:26:44.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:26:44 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:26:44.542 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T01:26:44.542 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:26:44.542 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T01:26:44.542 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:26:44.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:26:44.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:26:44.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T01:26:44.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:26:44.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T01:26:44.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:26:44.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T01:26:44.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:26:44.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T01:26:44.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:26:44.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T01:26:44.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:26:44.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:26:44.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:26:44.545 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:26:44 smithi047 ceph-mon[77147]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T01:26:44.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:26:44 smithi115 conmon[90393]: 2022-05-14T01:26:44.483+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:26:45.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:26:45 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:26:45.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:26:45 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 7135 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:26:45.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:26:45 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:26:45.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:26:45 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 7135 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:26:45.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:26:45 smithi115 conmon[90393]: 2022-05-14T01:26:45.476+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:26:46.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:26:46 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:26:46.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:26:46 smithi115 ceph-mon[78593]: pgmap v3922: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:26:46.513 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:26:46 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:26:46.514 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:26:46 smithi047 ceph-mon[77147]: pgmap v3922: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:26:46.791 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:26:46 smithi047 conmon[78521]: 2022-05-14T01:26:46.512+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:26:46.792 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:26:46 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T01:26:46.792 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:26:46 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:26:46.792 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:26:46 smithi047 conmon[78521]: yield 2022-05-14T01:26:46.792 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:26:46 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:26:46.793 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:26:46 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:26:46.793 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:26:46 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:26:46.793 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:26:46 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T01:26:46.793 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:26:46 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:26:46.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:26:46 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T01:26:46.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:26:46 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:26:46.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:26:46 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T01:26:46.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:26:46 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:26:46.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:26:46 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T01:26:46.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:26:46 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:26:46.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:26:46 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T01:26:46.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:26:46 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:26:46.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:26:46 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:26:46.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:26:46 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:26:46.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:26:46 smithi047 conmon[78521]: 2022-05-14T01:26:46.513+0000 7f5345f84700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T01:26:46.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:26:46 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T01:26:46.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:26:46 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:26:46.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:26:46 smithi047 conmon[78521]: yield 2022-05-14T01:26:46.797 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:26:46 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:26:46.797 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:26:46 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:26:46.797 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:26:46 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:26:46.797 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:26:46 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T01:26:46.798 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:26:46 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:26:46.798 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:26:46 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T01:26:46.798 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:26:46 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:26:46.798 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:26:46 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T01:26:46.798 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:26:46 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:26:46.799 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:26:46 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T01:26:46.799 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:26:46 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:26:46.799 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:26:46 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T01:26:46.799 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:26:46 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:26:46.799 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:26:46 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:26:46.800 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:26:46 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:26:46.800 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:26:46 smithi047 conmon[78521]: 2022-05-14T01:26:46.800 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:26:46 smithi047 conmon[78521]: During handling of the above exception, another exception occurred: 2022-05-14T01:26:46.800 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:26:46 smithi047 conmon[78521]: 2022-05-14T01:26:46.800 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:26:46 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T01:26:46.801 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:26:46 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T01:26:46.801 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:26:46 smithi047 conmon[78521]: return f(*arg) 2022-05-14T01:26:46.801 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:26:46 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T01:26:46.801 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:26:46 smithi047 conmon[78521]: self._write_client_files(client_files, host) 2022-05-14T01:26:46.802 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:26:46 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T01:26:46.802 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:26:46 smithi047 conmon[78521]: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T01:26:46.802 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:26:46 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T01:26:46.802 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:26:46 smithi047 conmon[78521]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T01:26:46.802 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:26:46 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T01:26:46.803 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:26:46 smithi047 conmon[78521]: return self.event_loop.get_result(coro) 2022-05-14T01:26:46.803 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:26:46 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T01:26:46.803 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:26:46 smithi047 conmon[78521]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T01:26:46.803 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:26:46 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T01:26:46.803 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:26:46 smithi047 conmon[78521]: return self.__get_result() 2022-05-14T01:26:46.804 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:26:46 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T01:26:46.804 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:26:46 smithi047 conmon[78521]: raise self._exception 2022-05-14T01:26:46.804 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:26:46 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T01:26:46.804 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:26:46 smithi047 conmon[78521]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T01:26:46.804 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:26:46 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T01:26:46.805 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:26:46 smithi047 conmon[78521]: conn = await self._remote_connection(host, addr) 2022-05-14T01:26:46.805 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:26:46 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T01:26:46.805 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:26:46 smithi047 conmon[78521]: raise 2022-05-14T01:26:46.805 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:26:46 smithi047 conmon[78521]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T01:26:46.806 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:26:46 smithi047 conmon[78521]: self.gen.throw(type, value, traceback) 2022-05-14T01:26:46.806 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:26:46 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T01:26:46.806 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:26:46 smithi047 conmon[78521]: raise OrchestratorError(msg) 2022-05-14T01:26:46.806 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:26:46 smithi047 conmon[78521]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:26:46.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:26:46 smithi115 conmon[90393]: 2022-05-14T01:26:46.524+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:26:47.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:26:47 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:26:47.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:26:47 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:26:47.359 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T01:26:47.359 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:26:47.359 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T01:26:47.359 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:26:47.359 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:26:47.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:26:47.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T01:26:47.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:26:47.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T01:26:47.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:26:47.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T01:26:47.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:26:47.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T01:26:47.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:26:47.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T01:26:47.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:26:47.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:26:47.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:26:47.363 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:26:47 smithi115 ceph-mon[78593]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T01:26:47.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T01:26:47.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:26:47.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T01:26:47.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:26:47.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:26:47.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:26:47.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T01:26:47.365 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:26:47.365 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T01:26:47.365 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:26:47.365 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T01:26:47.366 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:26:47.366 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T01:26:47.366 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:26:47.366 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T01:26:47.366 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:26:47.367 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:26:47.367 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:26:47.367 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T01:26:47.367 INFO:journalctl@ceph.mon.c.smithi115.stdout: During handling of the above exception, another exception occurred: 2022-05-14T01:26:47.368 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T01:26:47.368 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T01:26:47.368 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T01:26:47.368 INFO:journalctl@ceph.mon.c.smithi115.stdout: return f(*arg) 2022-05-14T01:26:47.369 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T01:26:47.369 INFO:journalctl@ceph.mon.c.smithi115.stdout: self._write_client_files(client_files, host) 2022-05-14T01:26:47.369 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T01:26:47.369 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T01:26:47.370 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T01:26:47.370 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T01:26:47.370 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T01:26:47.370 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.event_loop.get_result(coro) 2022-05-14T01:26:47.371 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T01:26:47.371 INFO:journalctl@ceph.mon.c.smithi115.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T01:26:47.371 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T01:26:47.371 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.__get_result() 2022-05-14T01:26:47.372 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T01:26:47.372 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise self._exception 2022-05-14T01:26:47.372 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T01:26:47.372 INFO:journalctl@ceph.mon.c.smithi115.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T01:26:47.372 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T01:26:47.373 INFO:journalctl@ceph.mon.c.smithi115.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T01:26:47.373 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T01:26:47.373 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise 2022-05-14T01:26:47.373 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T01:26:47.374 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.gen.throw(type, value, traceback) 2022-05-14T01:26:47.374 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T01:26:47.374 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OrchestratorError(msg) 2022-05-14T01:26:47.374 INFO:journalctl@ceph.mon.c.smithi115.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:26:47.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:26:47 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:26:47.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:26:47 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:26:47.541 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T01:26:47.542 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:26:47.542 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T01:26:47.542 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:26:47.542 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:26:47.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:26:47.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T01:26:47.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:26:47.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T01:26:47.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:26:47.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T01:26:47.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:26:47.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T01:26:47.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:26:47.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T01:26:47.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:26:47.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:26:47.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:26:47.546 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:26:47 smithi047 ceph-mon[77147]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T01:26:47.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T01:26:47.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:26:47.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T01:26:47.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:26:47.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:26:47.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:26:47.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T01:26:47.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:26:47.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T01:26:47.548 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:26:47.548 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T01:26:47.548 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:26:47.549 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T01:26:47.549 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:26:47.549 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T01:26:47.549 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:26:47.549 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:26:47.550 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:26:47.550 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T01:26:47.550 INFO:journalctl@ceph.mon.b.smithi047.stdout: During handling of the above exception, another exception occurred: 2022-05-14T01:26:47.550 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T01:26:47.550 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T01:26:47.551 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T01:26:47.551 INFO:journalctl@ceph.mon.b.smithi047.stdout: return f(*arg) 2022-05-14T01:26:47.551 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T01:26:47.551 INFO:journalctl@ceph.mon.b.smithi047.stdout: self._write_client_files(client_files, host) 2022-05-14T01:26:47.551 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T01:26:47.552 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T01:26:47.552 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T01:26:47.552 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T01:26:47.552 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T01:26:47.553 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.event_loop.get_result(coro) 2022-05-14T01:26:47.553 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T01:26:47.553 INFO:journalctl@ceph.mon.b.smithi047.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T01:26:47.553 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T01:26:47.553 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.__get_result() 2022-05-14T01:26:47.554 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T01:26:47.554 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise self._exception 2022-05-14T01:26:47.554 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T01:26:47.554 INFO:journalctl@ceph.mon.b.smithi047.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T01:26:47.554 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T01:26:47.555 INFO:journalctl@ceph.mon.b.smithi047.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T01:26:47.555 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T01:26:47.555 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise 2022-05-14T01:26:47.555 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T01:26:47.555 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.gen.throw(type, value, traceback) 2022-05-14T01:26:47.556 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T01:26:47.556 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OrchestratorError(msg) 2022-05-14T01:26:47.556 INFO:journalctl@ceph.mon.b.smithi047.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:26:47.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:26:47 smithi115 conmon[90393]: 2022-05-14T01:26:47.481+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:26:48.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:26:48 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:26:48.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:26:48 smithi115 ceph-mon[78593]: pgmap v3923: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:26:48.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:26:48 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:26:48.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:26:48 smithi047 ceph-mon[77147]: pgmap v3923: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:26:48.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:26:48 smithi115 conmon[90393]: 2022-05-14T01:26:48.450+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:26:49.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:26:49 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:26:49.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:26:49 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:26:49.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:26:49 smithi115 conmon[90393]: 2022-05-14T01:26:49.426+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:26:50.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:26:50 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:26:50.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:26:50 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:26:50.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:26:50 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:26:50.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:26:50 smithi115 ceph-mon[78593]: pgmap v3924: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:26:50.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:26:50 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 7140 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:26:50.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:26:50 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:26:50.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:26:50 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:26:50.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:26:50 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:26:50.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:26:50 smithi047 ceph-mon[77147]: pgmap v3924: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:26:50.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:26:50 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 7140 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:26:50.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:26:50 smithi115 conmon[90393]: 2022-05-14T01:26:50.446+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:26:51.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:26:51 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:26:51.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:26:51 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:26:51.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:26:51 smithi115 conmon[90393]: 2022-05-14T01:26:51.400+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:26:52.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:26:52 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:26:52.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:26:52 smithi115 ceph-mon[78593]: pgmap v3925: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:26:52.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:26:52 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:26:52.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:26:52 smithi047 ceph-mon[77147]: pgmap v3925: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:26:52.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:26:52 smithi115 conmon[90393]: 2022-05-14T01:26:52.436+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:26:53.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:26:53 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:26:53.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:26:53 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:26:53.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:26:53 smithi115 conmon[90393]: 2022-05-14T01:26:53.474+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:26:54.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:26:54 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:26:54.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:26:54 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:26:54.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:26:54 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:26:54.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:26:54 smithi115 ceph-mon[78593]: pgmap v3926: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:26:54.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:26:54 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:26:54.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:26:54 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:26:54.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:26:54 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:26:54.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:26:54 smithi047 ceph-mon[77147]: pgmap v3926: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:26:54.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:26:54 smithi115 conmon[90393]: 2022-05-14T01:26:54.435+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:26:55.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:26:55 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:26:55.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:26:55 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 7145 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:26:55.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:26:55 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:26:55.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:26:55 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 7145 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:26:55.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:26:55 smithi115 conmon[90393]: 2022-05-14T01:26:55.391+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:26:56.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:26:56 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:26:56.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:26:56 smithi115 ceph-mon[78593]: pgmap v3927: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:26:56.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:26:56 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:26:56.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:26:56 smithi047 ceph-mon[77147]: pgmap v3927: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:26:56.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:26:56 smithi115 conmon[90393]: 2022-05-14T01:26:56.413+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:26:57.452 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:26:57 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:26:57.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:26:57 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:26:57.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:26:57 smithi115 conmon[90393]: 2022-05-14T01:26:57.451+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:26:58.474 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:26:58 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:26:58.475 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:26:58 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:26:58.475 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:26:58 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:26:58.475 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:26:58 smithi115 ceph-mon[78593]: pgmap v3928: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:26:58.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:26:58 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:26:58.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:26:58 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:26:58.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:26:58 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:26:58.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:26:58 smithi047 ceph-mon[77147]: pgmap v3928: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:26:58.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:26:58 smithi115 conmon[90393]: 2022-05-14T01:26:58.474+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:26:59.485 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:26:59 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:26:59.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:26:59 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:26:59.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:26:59 smithi115 conmon[90393]: 2022-05-14T01:26:59.484+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:27:00.517 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:27:00 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:27:00.517 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:27:00 smithi115 ceph-mon[78593]: pgmap v3929: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:27:00.517 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:27:00 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 7150 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:27:00.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:27:00 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:27:00.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:27:00 smithi047 ceph-mon[77147]: pgmap v3929: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:27:00.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:27:00 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 7150 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:27:00.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:27:00 smithi115 conmon[90393]: 2022-05-14T01:27:00.516+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:27:01.519 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:27:01 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:27:01.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:27:01 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:27:01.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:27:01 smithi115 conmon[90393]: 2022-05-14T01:27:01.518+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:27:02.536 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:27:02 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:27:02.537 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:27:02 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:27:02.537 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:27:02 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:27:02.537 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:27:02 smithi115 ceph-mon[78593]: pgmap v3930: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:27:02.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:27:02 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:27:02.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:27:02 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:27:02.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:27:02 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:27:02.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:27:02 smithi047 ceph-mon[77147]: pgmap v3930: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:27:02.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:27:02 smithi115 conmon[90393]: 2022-05-14T01:27:02.536+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:27:03.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:27:03 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:27:03.556 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:27:03 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:27:03.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:27:03 smithi115 conmon[90393]: 2022-05-14T01:27:03.555+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:27:04.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:27:04 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:27:04.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:27:04 smithi047 ceph-mon[77147]: pgmap v3931: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:27:04.580 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:27:04 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:27:04.580 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:27:04 smithi115 ceph-mon[78593]: pgmap v3931: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:27:04.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:27:04 smithi115 conmon[90393]: 2022-05-14T01:27:04.579+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:27:05.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:27:05 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:27:05.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:27:05 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 7155 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:27:05.565 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:27:05 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:27:05.565 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:27:05 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 7155 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:27:05.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:27:05 smithi115 conmon[90393]: 2022-05-14T01:27:05.564+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:27:06.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:27:06 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:27:06.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:27:06 smithi047 ceph-mon[77147]: pgmap v3932: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:27:06.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:27:06 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:27:06.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:27:06 smithi115 ceph-mon[78593]: pgmap v3932: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:27:07.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:27:06 smithi115 conmon[90393]: 2022-05-14T01:27:06.609+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:27:07.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:27:07 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:27:07.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:27:07 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:27:08.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:27:07 smithi115 conmon[90393]: 2022-05-14T01:27:07.620+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:27:08.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:27:08 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:27:08.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:27:08 smithi047 ceph-mon[77147]: pgmap v3933: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:27:08.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:27:08 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:27:08.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:27:08 smithi115 ceph-mon[78593]: pgmap v3933: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:27:09.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:27:08 smithi115 conmon[90393]: 2022-05-14T01:27:08.619+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:27:09.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:27:09 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:27:09.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:27:09 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:27:10.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:27:09 smithi115 conmon[90393]: 2022-05-14T01:27:09.645+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:27:10.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:27:10 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:27:10.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:27:10 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:27:10.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:27:10 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:27:10.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:27:10 smithi047 ceph-mon[77147]: pgmap v3934: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:27:10.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:27:10 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 7160 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:27:10.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:27:10 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:27:10.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:27:10 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:27:10.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:27:10 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:27:10.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:27:10 smithi115 ceph-mon[78593]: pgmap v3934: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:27:10.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:27:10 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 7160 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:27:11.117 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:27:10 smithi115 conmon[90393]: 2022-05-14T01:27:10.647+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:27:11.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:27:11 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:27:11.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:27:11 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:27:12.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:27:11 smithi115 conmon[90393]: 2022-05-14T01:27:11.679+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:27:12.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:27:12 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:27:12.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:27:12 smithi047 ceph-mon[77147]: pgmap v3935: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:27:12.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:27:12 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:27:12.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:27:12 smithi115 ceph-mon[78593]: pgmap v3935: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:27:13.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:27:12 smithi115 conmon[90393]: 2022-05-14T01:27:12.721+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:27:13.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:27:13 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:27:13.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:27:13 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:27:14.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:27:13 smithi115 conmon[90393]: 2022-05-14T01:27:13.687+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:27:14.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:27:14 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:27:14.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:27:14 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:27:14.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:27:14 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:27:14.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:27:14 smithi047 ceph-mon[77147]: pgmap v3936: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:27:14.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:27:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T01:27:14.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:27:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T01:27:14.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:27:14 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:27:14.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:27:14 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:27:14.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:27:14 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:27:14.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:27:14 smithi115 ceph-mon[78593]: pgmap v3936: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:27:14.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:27:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T01:27:14.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:27:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T01:27:15.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:27:14 smithi115 conmon[90393]: 2022-05-14T01:27:14.703+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:27:15.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:27:15 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:27:15.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:27:15 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 7165 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:27:15.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:27:15 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:27:15.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:27:15 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 7165 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:27:16.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:27:15 smithi115 conmon[90393]: 2022-05-14T01:27:15.740+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:27:16.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:27:16 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:27:16.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:27:16 smithi047 ceph-mon[77147]: pgmap v3937: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:27:16.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:27:16 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:27:16.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:27:16 smithi115 ceph-mon[78593]: pgmap v3937: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:27:17.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:27:16 smithi115 conmon[90393]: 2022-05-14T01:27:16.789+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:27:17.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:27:17 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:27:17.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:27:17 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:27:18.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:27:17 smithi115 conmon[90393]: 2022-05-14T01:27:17.766+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:27:18.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:27:18 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:27:18.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:27:18 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:27:18.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:27:18 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:27:18.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:27:18 smithi047 ceph-mon[77147]: pgmap v3938: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:27:18.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:27:18 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:27:18.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:27:18 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:27:18.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:27:18 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:27:18.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:27:18 smithi115 ceph-mon[78593]: pgmap v3938: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:27:19.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:27:18 smithi115 conmon[90393]: 2022-05-14T01:27:18.772+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:27:19.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:27:19 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:27:19.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:27:19 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:27:20.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:27:19 smithi115 conmon[90393]: 2022-05-14T01:27:19.752+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:27:20.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:27:20 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:27:20.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:27:20 smithi047 ceph-mon[77147]: pgmap v3939: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:27:20.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:27:20 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 7170 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:27:20.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:27:20 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:27:20.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:27:20 smithi115 ceph-mon[78593]: pgmap v3939: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:27:20.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:27:20 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 7170 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:27:21.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:27:20 smithi115 conmon[90393]: 2022-05-14T01:27:20.769+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:27:21.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:27:21 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:27:21.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:27:21 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:27:22.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:27:21 smithi115 conmon[90393]: 2022-05-14T01:27:21.730+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:27:22.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:27:22 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:27:22.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:27:22 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:27:22.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:27:22 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:27:22.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:27:22 smithi047 ceph-mon[77147]: pgmap v3940: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:27:22.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:27:22 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:27:22.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:27:22 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:27:22.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:27:22 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:27:22.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:27:22 smithi115 ceph-mon[78593]: pgmap v3940: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:27:23.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:27:22 smithi115 conmon[90393]: 2022-05-14T01:27:22.730+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:27:23.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:27:23 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:27:23.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:27:23 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:27:24.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:27:23 smithi115 conmon[90393]: 2022-05-14T01:27:23.736+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:27:24.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:27:24 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:27:24.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:27:24 smithi047 ceph-mon[77147]: pgmap v3941: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:27:24.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:27:24 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:27:24.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:27:24 smithi115 ceph-mon[78593]: pgmap v3941: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:27:25.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:27:24 smithi115 conmon[90393]: 2022-05-14T01:27:24.711+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:27:25.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:27:25 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:27:25.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:27:25 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 7175 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:27:25.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:27:25 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:27:25.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:27:25 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 7175 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:27:26.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:27:25 smithi115 conmon[90393]: 2022-05-14T01:27:25.703+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:27:26.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:27:26 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:27:26.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:27:26 smithi047 ceph-mon[77147]: pgmap v3942: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:27:26.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:27:26 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:27:26.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:27:26 smithi115 ceph-mon[78593]: pgmap v3942: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:27:27.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:27:26 smithi115 conmon[90393]: 2022-05-14T01:27:26.738+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:27:27.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:27:27 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:27:27.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:27:27 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:27:28.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:27:27 smithi115 conmon[90393]: 2022-05-14T01:27:27.787+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:27:28.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:27:28 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:27:28.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:27:28 smithi047 ceph-mon[77147]: pgmap v3943: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:27:28.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:27:28 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:27:28.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:27:28 smithi115 ceph-mon[78593]: pgmap v3943: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:27:29.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:27:28 smithi115 conmon[90393]: 2022-05-14T01:27:28.790+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:27:29.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:27:29 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:27:29.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:27:29 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:27:30.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:27:29 smithi115 conmon[90393]: 2022-05-14T01:27:29.742+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:27:30.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:27:30 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:27:30.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:27:30 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:27:30.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:27:30 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:27:30.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:27:30 smithi047 ceph-mon[77147]: pgmap v3944: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:27:30.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:27:30 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 7180 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:27:30.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:27:30 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:27:30.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:27:30 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:27:30.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:27:30 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:27:30.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:27:30 smithi115 ceph-mon[78593]: pgmap v3944: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:27:30.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:27:30 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 7180 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:27:31.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:27:30 smithi115 conmon[90393]: 2022-05-14T01:27:30.747+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:27:31.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:27:31 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:27:31.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:27:31 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:27:32.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:27:31 smithi115 conmon[90393]: 2022-05-14T01:27:31.736+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:27:32.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:27:32 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:27:32.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:27:32 smithi047 ceph-mon[77147]: pgmap v3945: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:27:32.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:27:32 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:27:32.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:27:32 smithi115 ceph-mon[78593]: pgmap v3945: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:27:33.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:27:32 smithi115 conmon[90393]: 2022-05-14T01:27:32.688+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:27:33.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:27:33 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:27:33.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:27:33 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:27:34.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:27:33 smithi115 conmon[90393]: 2022-05-14T01:27:33.668+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:27:34.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:27:34 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:27:34.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:27:34 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:27:34.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:27:34 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:27:34.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:27:34 smithi047 ceph-mon[77147]: pgmap v3946: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:27:34.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:27:34 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:27:34.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:27:34 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:27:34.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:27:34 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:27:34.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:27:34 smithi115 ceph-mon[78593]: pgmap v3946: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:27:35.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:27:34 smithi115 conmon[90393]: 2022-05-14T01:27:34.692+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:27:35.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:27:35 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:27:35.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:27:35 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 7185 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:27:35.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:27:35 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:27:35.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:27:35 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 7185 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:27:36.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:27:35 smithi115 conmon[90393]: 2022-05-14T01:27:35.716+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:27:36.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:27:36 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:27:36.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:27:36 smithi047 ceph-mon[77147]: pgmap v3947: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:27:36.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:27:36 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:27:36.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:27:36 smithi115 ceph-mon[78593]: pgmap v3947: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:27:37.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:27:36 smithi115 conmon[90393]: 2022-05-14T01:27:36.724+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:27:37.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:27:37 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:27:37.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:27:37 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:27:38.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:27:37 smithi115 conmon[90393]: 2022-05-14T01:27:37.731+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:27:38.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:27:38 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:27:38.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:27:38 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:27:38.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:27:38 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:27:38.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:27:38 smithi047 ceph-mon[77147]: pgmap v3948: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:27:38.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:27:38 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:27:38.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:27:38 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:27:38.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:27:38 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:27:38.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:27:38 smithi115 ceph-mon[78593]: pgmap v3948: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:27:39.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:27:38 smithi115 conmon[90393]: 2022-05-14T01:27:38.760+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:27:39.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:27:39 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:27:39.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:27:39 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:27:40.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:27:39 smithi115 conmon[90393]: 2022-05-14T01:27:39.796+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:27:40.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:27:40 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:27:40.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:27:40 smithi047 ceph-mon[77147]: pgmap v3949: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:27:40.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:27:40 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 7190 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:27:40.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:27:40 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:27:40.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:27:40 smithi115 ceph-mon[78593]: pgmap v3949: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:27:40.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:27:40 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 7190 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:27:41.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:27:40 smithi115 conmon[90393]: 2022-05-14T01:27:40.786+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:27:41.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:27:41 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:27:41.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:27:41 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:27:42.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:27:41 smithi115 conmon[90393]: 2022-05-14T01:27:41.828+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:27:42.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:27:42 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:27:42.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:27:42 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:27:42.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:27:42 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:27:42.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:27:42 smithi115 ceph-mon[78593]: pgmap v3950: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:27:42.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:27:42 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:27:42.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:27:42 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:27:42.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:27:42 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:27:42.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:27:42 smithi047 ceph-mon[77147]: pgmap v3950: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:27:43.301 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:27:42 smithi115 conmon[90393]: 2022-05-14T01:27:42.865+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:27:43.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:27:43 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:27:43.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:27:43 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:27:44.306 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:27:43 smithi115 conmon[90393]: 2022-05-14T01:27:43.887+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:27:44.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:27:44 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:27:44.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:27:44 smithi115 ceph-mon[78593]: pgmap v3951: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:27:44.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:27:44 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:27:44.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:27:44 smithi047 ceph-mon[77147]: pgmap v3951: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:27:45.308 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:27:44 smithi115 conmon[90393]: 2022-05-14T01:27:44.925+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:27:45.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:27:45 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:27:45.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:27:45 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 7195 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:27:45.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:27:45 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:27:45.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:27:45 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 7195 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:27:46.313 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:27:45 smithi115 conmon[90393]: 2022-05-14T01:27:45.968+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:27:46.587 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:27:46 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:27:46.587 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:27:46 smithi047 ceph-mon[77147]: pgmap v3952: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:27:46.587 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:27:46 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:27:46.588 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:27:46 smithi115 ceph-mon[78593]: pgmap v3952: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:27:47.317 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:27:46 smithi115 conmon[90393]: 2022-05-14T01:27:46.944+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:27:47.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:27:47 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:27:47.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:27:47 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T01:27:47.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:27:47 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T01:27:47.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:27:47 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T01:27:47.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:27:47 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' 2022-05-14T01:27:47.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:27:47 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' 2022-05-14T01:27:47.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:27:47 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:27:47.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:27:47 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T01:27:47.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:27:47 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T01:27:47.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:27:47 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T01:27:47.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:27:47 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' 2022-05-14T01:27:47.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:27:47 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' 2022-05-14T01:27:48.317 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:27:47 smithi115 conmon[90393]: 2022-05-14T01:27:47.953+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:27:48.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:27:48 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:27:48.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:27:48 smithi115 ceph-mon[78593]: pgmap v3953: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:27:48.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:27:48 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:27:48.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:27:48 smithi047 ceph-mon[77147]: pgmap v3953: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:27:49.325 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:27:48 smithi115 conmon[90393]: 2022-05-14T01:27:48.984+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:27:49.572 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:27:49 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:27:49.582 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:27:49 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:27:50.041 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:27:49 smithi047 conmon[78521]: 2022-05-14T01:27:49.615+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:27:50.041 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:27:49 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T01:27:50.041 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:27:49 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:27:50.042 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:27:49 smithi047 conmon[78521]: yield 2022-05-14T01:27:50.042 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:27:49 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:27:50.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:27:49 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:27:50.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:27:49 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:27:50.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:27:49 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T01:27:50.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:27:49 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:27:50.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:27:49 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T01:27:50.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:27:49 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:27:50.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:27:49 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T01:27:50.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:27:49 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:27:50.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:27:49 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T01:27:50.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:27:49 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:27:50.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:27:49 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T01:27:50.048 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:27:49 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:27:50.048 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:27:49 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:27:50.048 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:27:49 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:27:50.327 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:27:49 smithi115 conmon[90393]: 2022-05-14T01:27:49.978+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:27:50.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:27:50 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:27:50.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:27:50 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:27:50.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:27:50 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:27:50.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:27:50 smithi115 ceph-mon[78593]: pgmap v3954: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:27:50.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:27:50 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' 2022-05-14T01:27:50.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:27:50 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' 2022-05-14T01:27:50.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:27:50 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:27:50.609 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T01:27:50.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:27:50.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T01:27:50.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:27:50.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:27:50.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:27:50.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T01:27:50.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:27:50.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T01:27:50.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:27:50.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T01:27:50.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:27:50.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T01:27:50.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:27:50.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T01:27:50.613 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:27:50.613 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:27:50.613 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:27:50.613 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:27:50 smithi115 ceph-mon[78593]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T01:27:50.614 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:27:50 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 7200 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:27:50.614 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:27:50 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-05-14T01:27:50.614 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:27:50 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-05-14T01:27:50.614 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:27:50 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-05-14T01:27:50.615 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:27:50 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-05-14T01:27:50.615 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:27:50 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-05-14T01:27:50.615 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:27:50 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-05-14T01:27:50.615 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:27:50 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-05-14T01:27:50.615 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:27:50 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-05-14T01:27:50.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:27:50 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:27:50.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:27:50 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:27:50.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:27:50 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:27:50.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:27:50 smithi047 ceph-mon[77147]: pgmap v3954: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:27:50.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:27:50 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' 2022-05-14T01:27:50.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:27:50 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' 2022-05-14T01:27:50.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:27:50 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:27:50.793 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T01:27:50.793 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:27:50.793 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T01:27:50.793 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:27:50.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:27:50.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:27:50.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T01:27:50.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:27:50.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T01:27:50.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:27:50.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T01:27:50.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:27:50.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T01:27:50.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:27:50.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T01:27:50.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:27:50.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:27:50.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:27:50.797 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:27:50 smithi047 ceph-mon[77147]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T01:27:50.797 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:27:50 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 7200 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:27:50.797 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:27:50 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-05-14T01:27:50.797 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:27:50 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-05-14T01:27:50.798 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:27:50 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-05-14T01:27:50.798 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:27:50 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-05-14T01:27:50.798 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:27:50 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-05-14T01:27:50.798 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:27:50 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-05-14T01:27:50.799 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:27:50 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-05-14T01:27:50.799 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:27:50 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-05-14T01:27:51.329 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:27:50 smithi115 conmon[90393]: 2022-05-14T01:27:50.930+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:27:51.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:27:51 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:27:51.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:27:51 smithi115 ceph-mon[78593]: Adjusting osd_memory_target on smithi115 to 197.0M 2022-05-14T01:27:51.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:27:51 smithi115 ceph-mon[78593]: Unable to set osd_memory_target on smithi115 to 206629785: error parsing value: Value '206629785' is below minimum 939524096 2022-05-14T01:27:51.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:27:51 smithi115 ceph-mon[78593]: Adjusting osd_memory_target on smithi047 to 198.1M 2022-05-14T01:27:51.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:27:51 smithi115 ceph-mon[78593]: Unable to set osd_memory_target on smithi047 to 207755878: error parsing value: Value '207755878' is below minimum 939524096 2022-05-14T01:27:51.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:27:51 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:27:51.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:27:51 smithi047 ceph-mon[77147]: Adjusting osd_memory_target on smithi115 to 197.0M 2022-05-14T01:27:51.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:27:51 smithi047 ceph-mon[77147]: Unable to set osd_memory_target on smithi115 to 206629785: error parsing value: Value '206629785' is below minimum 939524096 2022-05-14T01:27:51.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:27:51 smithi047 ceph-mon[77147]: Adjusting osd_memory_target on smithi047 to 198.1M 2022-05-14T01:27:51.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:27:51 smithi047 ceph-mon[77147]: Unable to set osd_memory_target on smithi047 to 207755878: error parsing value: Value '207755878' is below minimum 939524096 2022-05-14T01:27:52.333 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:27:51 smithi115 conmon[90393]: 2022-05-14T01:27:51.976+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:27:52.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:27:52 smithi115 ceph-mon[78593]: 8 slow requests (by type [ 'queued for pg' : 8 ] most affected pool [ 'cephfs_metadata' : 8 ]) 2022-05-14T01:27:52.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:27:52 smithi115 ceph-mon[78593]: pgmap v3955: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:27:52.689 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:27:52 smithi047 ceph-mon[77147]: 8 slow requests (by type [ 'queued for pg' : 8 ] most affected pool [ 'cephfs_metadata' : 8 ]) 2022-05-14T01:27:52.690 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:27:52 smithi047 ceph-mon[77147]: pgmap v3955: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:27:53.041 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:27:52 smithi047 conmon[78521]: 2022-05-14T01:27:52.687+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:27:53.042 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:27:52 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T01:27:53.042 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:27:52 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:27:53.042 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:27:52 smithi047 conmon[78521]: yield 2022-05-14T01:27:53.042 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:27:52 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:27:53.042 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:27:52 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:27:53.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:27:52 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:27:53.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:27:52 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T01:27:53.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:27:52 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:27:53.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:27:52 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T01:27:53.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:27:52 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:27:53.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:27:52 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T01:27:53.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:27:52 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:27:53.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:27:52 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T01:27:53.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:27:52 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:27:53.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:27:52 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T01:27:53.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:27:52 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:27:53.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:27:52 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:27:53.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:27:52 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:27:53.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:27:52 smithi047 conmon[78521]: 2022-05-14T01:27:52.688+0000 7f533f777700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T01:27:53.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:27:52 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T01:27:53.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:27:52 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:27:53.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:27:52 smithi047 conmon[78521]: yield 2022-05-14T01:27:53.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:27:52 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:27:53.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:27:52 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:27:53.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:27:52 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:27:53.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:27:52 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T01:27:53.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:27:52 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:27:53.048 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:27:52 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T01:27:53.048 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:27:52 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:27:53.048 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:27:52 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T01:27:53.048 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:27:52 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:27:53.049 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:27:52 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T01:27:53.049 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:27:52 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:27:53.049 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:27:52 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T01:27:53.049 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:27:52 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:27:53.050 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:27:52 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:27:53.050 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:27:52 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:27:53.050 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:27:52 smithi047 conmon[78521]: 2022-05-14T01:27:53.050 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:27:52 smithi047 conmon[78521]: During handling of the above exception, another exception occurred: 2022-05-14T01:27:53.050 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:27:52 smithi047 conmon[78521]: 2022-05-14T01:27:53.051 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:27:52 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T01:27:53.051 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:27:52 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T01:27:53.051 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:27:52 smithi047 conmon[78521]: return f(*arg) 2022-05-14T01:27:53.051 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:27:52 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T01:27:53.052 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:27:52 smithi047 conmon[78521]: self._write_client_files(client_files, host) 2022-05-14T01:27:53.052 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:27:52 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T01:27:53.052 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:27:52 smithi047 conmon[78521]: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T01:27:53.052 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:27:52 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T01:27:53.052 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:27:52 smithi047 conmon[78521]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T01:27:53.053 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:27:52 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T01:27:53.053 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:27:52 smithi047 conmon[78521]: return self.event_loop.get_result(coro) 2022-05-14T01:27:53.053 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:27:52 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T01:27:53.053 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:27:52 smithi047 conmon[78521]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T01:27:53.053 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:27:52 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T01:27:53.054 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:27:52 smithi047 conmon[78521]: return self.__get_result() 2022-05-14T01:27:53.054 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:27:52 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T01:27:53.054 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:27:52 smithi047 conmon[78521]: raise self._exception 2022-05-14T01:27:53.055 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:27:52 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T01:27:53.055 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:27:52 smithi047 conmon[78521]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T01:27:53.055 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:27:52 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T01:27:53.055 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:27:52 smithi047 conmon[78521]: conn = await self._remote_connection(host, addr) 2022-05-14T01:27:53.055 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:27:52 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T01:27:53.056 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:27:52 smithi047 conmon[78521]: raise 2022-05-14T01:27:53.056 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:27:52 smithi047 conmon[78521]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T01:27:53.056 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:27:52 smithi047 conmon[78521]: self.gen.throw(type, value, traceback) 2022-05-14T01:27:53.056 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:27:52 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T01:27:53.057 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:27:52 smithi047 conmon[78521]: raise OrchestratorError(msg) 2022-05-14T01:27:53.057 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:27:52 smithi047 conmon[78521]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:27:53.335 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:27:52 smithi115 conmon[90393]: 2022-05-14T01:27:52.965+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:27:53.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:27:53 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:27:53.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:27:53 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:27:54.346 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:27:54 smithi115 conmon[90393]: 2022-05-14T01:27:54.003+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:27:54.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:27:54 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:27:54.608 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T01:27:54.608 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:27:54.609 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T01:27:54.609 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:27:54.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:27:54.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:27:54.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T01:27:54.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:27:54.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T01:27:54.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:27:54.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T01:27:54.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:27:54.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T01:27:54.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:27:54.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T01:27:54.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:27:54.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:27:54.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:27:54.613 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:27:54 smithi115 ceph-mon[78593]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T01:27:54.613 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T01:27:54.613 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:27:54.613 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T01:27:54.614 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:27:54.614 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:27:54.614 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:27:54.614 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T01:27:54.615 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:27:54.615 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T01:27:54.615 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:27:54.615 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T01:27:54.616 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:27:54.616 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T01:27:54.616 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:27:54.616 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T01:27:54.616 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:27:54.617 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:27:54.617 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:27:54.618 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T01:27:54.618 INFO:journalctl@ceph.mon.c.smithi115.stdout: During handling of the above exception, another exception occurred: 2022-05-14T01:27:54.618 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T01:27:54.619 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T01:27:54.619 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T01:27:54.619 INFO:journalctl@ceph.mon.c.smithi115.stdout: return f(*arg) 2022-05-14T01:27:54.619 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T01:27:54.619 INFO:journalctl@ceph.mon.c.smithi115.stdout: self._write_client_files(client_files, host) 2022-05-14T01:27:54.620 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T01:27:54.620 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T01:27:54.620 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T01:27:54.620 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T01:27:54.621 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T01:27:54.621 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.event_loop.get_result(coro) 2022-05-14T01:27:54.621 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T01:27:54.621 INFO:journalctl@ceph.mon.c.smithi115.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T01:27:54.622 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T01:27:54.622 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.__get_result() 2022-05-14T01:27:54.622 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T01:27:54.622 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise self._exception 2022-05-14T01:27:54.623 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T01:27:54.623 INFO:journalctl@ceph.mon.c.smithi115.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T01:27:54.623 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T01:27:54.623 INFO:journalctl@ceph.mon.c.smithi115.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T01:27:54.623 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T01:27:54.624 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise 2022-05-14T01:27:54.624 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T01:27:54.624 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.gen.throw(type, value, traceback) 2022-05-14T01:27:54.624 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T01:27:54.625 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OrchestratorError(msg) 2022-05-14T01:27:54.625 INFO:journalctl@ceph.mon.c.smithi115.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:27:54.625 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:27:54 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:27:54.625 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:27:54 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:27:54.626 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:27:54 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:27:54.626 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:27:54 smithi115 ceph-mon[78593]: pgmap v3956: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:27:54.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:27:54 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:27:54.791 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T01:27:54.792 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:27:54.792 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T01:27:54.792 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:27:54.792 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:27:54.792 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:27:54.793 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T01:27:54.793 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:27:54.793 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T01:27:54.793 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:27:54.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T01:27:54.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:27:54.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T01:27:54.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:27:54.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T01:27:54.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:27:54.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:27:54.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:27:54.795 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:27:54 smithi047 ceph-mon[77147]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T01:27:54.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T01:27:54.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:27:54.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T01:27:54.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:27:54.797 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:27:54.797 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:27:54.797 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T01:27:54.797 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:27:54.798 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T01:27:54.798 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:27:54.798 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T01:27:54.798 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:27:54.798 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T01:27:54.799 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:27:54.799 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T01:27:54.799 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:27:54.800 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:27:54.800 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:27:54.800 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T01:27:54.800 INFO:journalctl@ceph.mon.b.smithi047.stdout: During handling of the above exception, another exception occurred: 2022-05-14T01:27:54.801 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T01:27:54.801 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T01:27:54.801 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T01:27:54.802 INFO:journalctl@ceph.mon.b.smithi047.stdout: return f(*arg) 2022-05-14T01:27:54.802 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T01:27:54.802 INFO:journalctl@ceph.mon.b.smithi047.stdout: self._write_client_files(client_files, host) 2022-05-14T01:27:54.802 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T01:27:54.803 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T01:27:54.803 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T01:27:54.803 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T01:27:54.803 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T01:27:54.804 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.event_loop.get_result(coro) 2022-05-14T01:27:54.804 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T01:27:54.804 INFO:journalctl@ceph.mon.b.smithi047.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T01:27:54.804 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T01:27:54.805 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.__get_result() 2022-05-14T01:27:54.805 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T01:27:54.805 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise self._exception 2022-05-14T01:27:54.805 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T01:27:54.806 INFO:journalctl@ceph.mon.b.smithi047.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T01:27:54.806 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T01:27:54.806 INFO:journalctl@ceph.mon.b.smithi047.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T01:27:54.806 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T01:27:54.807 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise 2022-05-14T01:27:54.807 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T01:27:54.807 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.gen.throw(type, value, traceback) 2022-05-14T01:27:54.807 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T01:27:54.808 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OrchestratorError(msg) 2022-05-14T01:27:54.808 INFO:journalctl@ceph.mon.b.smithi047.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:27:54.808 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:27:54 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:27:54.808 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:27:54 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:27:54.809 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:27:54 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:27:54.809 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:27:54 smithi047 ceph-mon[77147]: pgmap v3956: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:27:55.345 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:27:55 smithi115 conmon[90393]: 2022-05-14T01:27:55.038+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:27:55.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:27:55 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:27:55.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:27:55 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 7205 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:27:55.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:27:55 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:27:55.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:27:55 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 7205 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:27:56.352 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:27:55 smithi115 conmon[90393]: 2022-05-14T01:27:55.997+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:27:56.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:27:56 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:27:56.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:27:56 smithi115 ceph-mon[78593]: pgmap v3957: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:27:56.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:27:56 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:27:56.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:27:56 smithi047 ceph-mon[77147]: pgmap v3957: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:27:57.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:27:56 smithi115 conmon[90393]: 2022-05-14T01:27:56.963+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:27:57.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:27:57 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:27:57.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:27:57 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:27:58.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:27:58 smithi115 conmon[90393]: 2022-05-14T01:27:58.010+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:27:58.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:27:58 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:27:58.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:27:58 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:27:58.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:27:58 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:27:58.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:27:58 smithi047 ceph-mon[77147]: pgmap v3958: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:27:58.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:27:58 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:27:58.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:27:58 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:27:58.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:27:58 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:27:58.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:27:58 smithi115 ceph-mon[78593]: pgmap v3958: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:27:59.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:27:58 smithi115 conmon[90393]: 2022-05-14T01:27:58.975+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:27:59.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:27:59 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:27:59.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:27:59 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:28:00.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:27:59 smithi115 conmon[90393]: 2022-05-14T01:27:59.956+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:28:00.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:28:00 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:28:00.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:28:00 smithi047 ceph-mon[77147]: pgmap v3959: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:28:00.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:28:00 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 7210 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:28:00.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:28:00 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:28:00.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:28:00 smithi115 ceph-mon[78593]: pgmap v3959: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:28:00.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:28:00 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 7210 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:28:01.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:28:00 smithi115 conmon[90393]: 2022-05-14T01:28:00.932+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:28:01.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:28:01 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:28:01.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:28:01 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:28:02.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:28:01 smithi115 conmon[90393]: 2022-05-14T01:28:01.980+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:28:02.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:28:02 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:28:02.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:28:02 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:28:02.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:28:02 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:28:02.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:28:02 smithi047 ceph-mon[77147]: pgmap v3960: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:28:02.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:28:02 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:28:02.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:28:02 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:28:02.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:28:02 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:28:02.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:28:02 smithi115 ceph-mon[78593]: pgmap v3960: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:28:03.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:28:02 smithi115 conmon[90393]: 2022-05-14T01:28:02.992+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:28:03.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:28:03 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:28:03.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:28:03 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:28:04.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:28:03 smithi115 conmon[90393]: 2022-05-14T01:28:03.964+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:28:04.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:28:04 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:28:04.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:28:04 smithi047 ceph-mon[77147]: pgmap v3961: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:28:04.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:28:04 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:28:04.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:28:04 smithi115 ceph-mon[78593]: pgmap v3961: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:28:05.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:28:04 smithi115 conmon[90393]: 2022-05-14T01:28:04.997+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:28:05.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:28:05 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:28:05.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:28:05 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 7215 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:28:05.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:28:05 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:28:05.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:28:05 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 7215 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:28:06.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:28:06 smithi115 conmon[90393]: 2022-05-14T01:28:06.015+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:28:06.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:28:06 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:28:06.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:28:06 smithi047 ceph-mon[77147]: pgmap v3962: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:28:06.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:28:06 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:28:06.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:28:06 smithi115 ceph-mon[78593]: pgmap v3962: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:28:07.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:28:06 smithi115 conmon[90393]: 2022-05-14T01:28:06.977+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:28:07.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:28:07 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:28:07.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:28:07 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:28:07.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:28:07 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:28:07.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:28:07 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:28:08.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:28:07 smithi115 conmon[90393]: 2022-05-14T01:28:07.956+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:28:08.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:28:08 smithi047 ceph-mon[77147]: pgmap v3963: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:28:08.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:28:08 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:28:08.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:28:08 smithi115 ceph-mon[78593]: pgmap v3963: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:28:08.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:28:08 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:28:09.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:28:08 smithi115 conmon[90393]: 2022-05-14T01:28:08.909+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:28:09.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:28:09 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:28:09.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:28:09 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:28:10.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:28:09 smithi115 conmon[90393]: 2022-05-14T01:28:09.910+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:28:10.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:28:10 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:28:10.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:28:10 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:28:10.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:28:10 smithi047 ceph-mon[77147]: pgmap v3964: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:28:10.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:28:10 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 7220 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:28:10.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:28:10 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:28:10.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:28:10 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:28:10.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:28:10 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:28:10.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:28:10 smithi115 ceph-mon[78593]: pgmap v3964: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:28:10.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:28:10 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 7220 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:28:10.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:28:10 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:28:11.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:28:10 smithi115 conmon[90393]: 2022-05-14T01:28:10.950+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:28:11.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:28:11 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:28:11.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:28:11 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:28:12.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:28:11 smithi115 conmon[90393]: 2022-05-14T01:28:11.980+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:28:12.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:28:12 smithi047 ceph-mon[77147]: pgmap v3965: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:28:12.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:28:12 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:28:12.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:28:12 smithi115 ceph-mon[78593]: pgmap v3965: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:28:12.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:28:12 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:28:13.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:28:12 smithi115 conmon[90393]: 2022-05-14T01:28:12.958+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:28:13.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:28:13 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:28:13.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:28:13 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:28:14.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:28:13 smithi115 conmon[90393]: 2022-05-14T01:28:13.969+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:28:14.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:28:14 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:28:14.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:28:14 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:28:14.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:28:14 smithi047 ceph-mon[77147]: pgmap v3966: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:28:14.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:28:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T01:28:14.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:28:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T01:28:14.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:28:14 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:28:14.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:28:14 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:28:14.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:28:14 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:28:14.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:28:14 smithi115 ceph-mon[78593]: pgmap v3966: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:28:14.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:28:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T01:28:14.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:28:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T01:28:14.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:28:14 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:28:15.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:28:14 smithi115 conmon[90393]: 2022-05-14T01:28:14.945+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:28:15.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:28:15 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 7225 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:28:15.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:28:15 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:28:15.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:28:15 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 7225 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:28:15.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:28:15 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:28:16.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:28:15 smithi115 conmon[90393]: 2022-05-14T01:28:15.991+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:28:16.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:28:16 smithi047 ceph-mon[77147]: pgmap v3967: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:28:16.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:28:16 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:28:16.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:28:16 smithi115 ceph-mon[78593]: pgmap v3967: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:28:16.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:28:16 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:28:17.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:28:17 smithi115 conmon[90393]: 2022-05-14T01:28:17.006+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:28:17.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:28:17 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:28:17.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:28:17 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:28:18.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:28:17 smithi115 conmon[90393]: 2022-05-14T01:28:17.957+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:28:18.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:28:18 smithi047 ceph-mon[77147]: pgmap v3968: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:28:18.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:28:18 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:28:18.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:28:18 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:28:18.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:28:18 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:28:18.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:28:18 smithi115 ceph-mon[78593]: pgmap v3968: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:28:18.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:28:18 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:28:18.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:28:18 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:28:18.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:28:18 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:28:19.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:28:18 smithi115 conmon[90393]: 2022-05-14T01:28:18.916+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:28:19.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:28:19 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:28:19.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:28:19 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:28:20.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:28:19 smithi115 conmon[90393]: 2022-05-14T01:28:19.918+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:28:20.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:28:20 smithi047 ceph-mon[77147]: pgmap v3969: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:28:20.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:28:20 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 7235 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:28:20.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:28:20 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:28:20.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:28:20 smithi115 ceph-mon[78593]: pgmap v3969: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:28:20.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:28:20 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 7235 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:28:20.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:28:20 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:28:21.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:28:20 smithi115 conmon[90393]: 2022-05-14T01:28:20.906+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:28:21.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:28:21 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:28:21.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:28:21 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:28:22.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:28:21 smithi115 conmon[90393]: 2022-05-14T01:28:21.913+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:28:22.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:28:22 smithi047 ceph-mon[77147]: pgmap v3970: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:28:22.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:28:22 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:28:22.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:28:22 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:28:22.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:28:22 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:28:22.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:28:22 smithi115 ceph-mon[78593]: pgmap v3970: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:28:22.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:28:22 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:28:22.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:28:22 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:28:22.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:28:22 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:28:23.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:28:22 smithi115 conmon[90393]: 2022-05-14T01:28:22.933+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:28:23.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:28:23 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:28:23.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:28:23 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:28:24.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:28:23 smithi115 conmon[90393]: 2022-05-14T01:28:23.979+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:28:24.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:28:24 smithi047 ceph-mon[77147]: pgmap v3971: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:28:24.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:28:24 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:28:24.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:28:24 smithi115 ceph-mon[78593]: pgmap v3971: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:28:24.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:28:24 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:28:25.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:28:24 smithi115 conmon[90393]: 2022-05-14T01:28:24.932+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:28:26.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:28:25 smithi115 conmon[90393]: 2022-05-14T01:28:25.917+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:28:26.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:28:25 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:28:26.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:28:25 smithi115 ceph-mon[78593]: pgmap v3972: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:28:26.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:28:25 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:28:26.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:28:25 smithi047 ceph-mon[77147]: pgmap v3972: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:28:27.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:28:26 smithi115 conmon[90393]: 2022-05-14T01:28:26.916+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:28:27.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:28:26 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 7240 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:28:27.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:28:26 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:28:27.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:28:26 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 7240 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:28:27.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:28:26 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:28:28.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:28:27 smithi115 conmon[90393]: 2022-05-14T01:28:27.920+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:28:28.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:28:27 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:28:28.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:28:27 smithi115 ceph-mon[78593]: pgmap v3973: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:28:28.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:28:27 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:28:28.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:28:27 smithi047 ceph-mon[77147]: pgmap v3973: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:28:29.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:28:28 smithi115 conmon[90393]: 2022-05-14T01:28:28.951+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:28:29.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:28:28 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:28:29.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:28:28 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:28:30.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:28:29 smithi115 conmon[90393]: 2022-05-14T01:28:29.911+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:28:30.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:28:29 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:28:30.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:28:29 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:28:30.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:28:29 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:28:30.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:28:29 smithi115 ceph-mon[78593]: pgmap v3974: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:28:30.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:28:29 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:28:30.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:28:29 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:28:30.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:28:29 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:28:30.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:28:29 smithi047 ceph-mon[77147]: pgmap v3974: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:28:31.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:28:30 smithi115 conmon[90393]: 2022-05-14T01:28:30.911+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:28:31.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:28:30 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:28:31.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:28:30 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:28:32.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:28:31 smithi115 conmon[90393]: 2022-05-14T01:28:31.863+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:28:32.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:28:31 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:28:32.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:28:31 smithi115 ceph-mon[78593]: pgmap v3975: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:28:32.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:28:31 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:28:32.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:28:31 smithi047 ceph-mon[77147]: pgmap v3975: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:28:33.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:28:32 smithi115 conmon[90393]: 2022-05-14T01:28:32.884+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:28:33.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:28:32 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:28:33.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:28:32 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:28:34.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:28:33 smithi115 conmon[90393]: 2022-05-14T01:28:33.864+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:28:34.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:28:33 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:28:34.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:28:33 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:28:34.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:28:33 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:28:34.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:28:33 smithi115 ceph-mon[78593]: pgmap v3976: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:28:34.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:28:33 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:28:34.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:28:33 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:28:34.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:28:33 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:28:34.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:28:33 smithi047 ceph-mon[77147]: pgmap v3976: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:28:35.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:28:34 smithi115 conmon[90393]: 2022-05-14T01:28:34.843+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:28:35.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:28:34 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:28:35.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:28:34 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 7245 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:28:35.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:28:34 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:28:35.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:28:34 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 7245 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:28:36.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:28:35 smithi115 conmon[90393]: 2022-05-14T01:28:35.873+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:28:36.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:28:35 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:28:36.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:28:35 smithi115 ceph-mon[78593]: pgmap v3977: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:28:36.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:28:35 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:28:36.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:28:35 smithi047 ceph-mon[77147]: pgmap v3977: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:28:37.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:28:36 smithi115 conmon[90393]: 2022-05-14T01:28:36.923+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:28:37.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:28:36 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:28:37.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:28:36 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:28:38.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:28:37 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:28:38.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:28:37 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:28:38.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:28:37 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:28:38.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:28:37 smithi047 ceph-mon[77147]: pgmap v3978: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:28:38.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:28:37 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:28:38.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:28:37 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:28:38.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:28:37 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:28:38.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:28:37 smithi115 ceph-mon[78593]: pgmap v3978: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:28:38.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:28:37 smithi115 conmon[90393]: 2022-05-14T01:28:37.949+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:28:39.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:28:38 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:28:39.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:28:38 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:28:39.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:28:38 smithi115 conmon[90393]: 2022-05-14T01:28:38.900+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:28:40.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:28:39 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:28:40.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:28:39 smithi047 ceph-mon[77147]: pgmap v3979: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:28:40.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:28:39 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 7250 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:28:40.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:28:39 smithi115 conmon[90393]: 2022-05-14T01:28:39.882+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:28:40.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:28:39 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:28:40.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:28:39 smithi115 ceph-mon[78593]: pgmap v3979: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:28:40.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:28:39 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 7250 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:28:41.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:28:40 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:28:41.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:28:40 smithi115 conmon[90393]: 2022-05-14T01:28:40.865+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:28:41.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:28:40 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:28:42.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:28:41 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:28:42.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:28:41 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:28:42.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:28:41 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:28:42.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:28:41 smithi047 ceph-mon[77147]: pgmap v3980: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:28:42.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:28:41 smithi115 conmon[90393]: 2022-05-14T01:28:41.881+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:28:42.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:28:41 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:28:42.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:28:41 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:28:42.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:28:41 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:28:42.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:28:41 smithi115 ceph-mon[78593]: pgmap v3980: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:28:43.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:28:42 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:28:43.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:28:42 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:28:43.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:28:42 smithi115 conmon[90393]: 2022-05-14T01:28:42.889+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:28:44.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:28:43 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:28:44.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:28:43 smithi047 ceph-mon[77147]: pgmap v3981: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:28:44.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:28:43 smithi115 conmon[90393]: 2022-05-14T01:28:43.872+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:28:44.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:28:43 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:28:44.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:28:43 smithi115 ceph-mon[78593]: pgmap v3981: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:28:45.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:28:44 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:28:45.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:28:44 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 7255 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:28:45.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:28:44 smithi115 conmon[90393]: 2022-05-14T01:28:44.866+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:28:45.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:28:44 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:28:45.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:28:44 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 7255 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:28:46.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:28:45 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:28:46.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:28:45 smithi047 ceph-mon[77147]: pgmap v3982: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:28:46.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:28:45 smithi115 conmon[90393]: 2022-05-14T01:28:45.869+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:28:46.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:28:45 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:28:46.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:28:45 smithi115 ceph-mon[78593]: pgmap v3982: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:28:47.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:28:46 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:28:47.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:28:46 smithi115 conmon[90393]: 2022-05-14T01:28:46.887+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:28:47.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:28:46 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:28:48.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:28:47 smithi115 conmon[90393]: 2022-05-14T01:28:47.844+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:28:48.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:28:47 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:28:48.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:28:47 smithi115 ceph-mon[78593]: pgmap v3983: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:28:48.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:28:47 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:28:48.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:28:47 smithi047 ceph-mon[77147]: pgmap v3983: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:28:49.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:28:48 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:28:49.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:28:48 smithi115 conmon[90393]: 2022-05-14T01:28:48.826+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:28:49.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:28:48 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:28:50.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:28:49 smithi115 conmon[90393]: 2022-05-14T01:28:49.822+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:28:50.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:28:49 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:28:50.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:28:49 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:28:50.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:28:49 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:28:50.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:28:49 smithi115 ceph-mon[78593]: pgmap v3984: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:28:50.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:28:49 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 7260 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:28:50.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:28:49 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:28:50.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:28:49 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:28:50.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:28:49 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:28:50.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:28:49 smithi047 ceph-mon[77147]: pgmap v3984: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:28:50.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:28:49 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 7260 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:28:51.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:28:50 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:28:51.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:28:50 smithi115 conmon[90393]: 2022-05-14T01:28:50.784+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:28:51.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:28:50 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:28:52.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:28:51 smithi115 conmon[90393]: 2022-05-14T01:28:51.750+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:28:52.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:28:51 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:28:52.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:28:51 smithi115 ceph-mon[78593]: pgmap v3985: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:28:52.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:28:51 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:28:52.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:28:51 smithi047 ceph-mon[77147]: pgmap v3985: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:28:53.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:28:52 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:28:53.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:28:52 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T01:28:53.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:28:52 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T01:28:53.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:28:52 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T01:28:53.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:28:52 smithi115 conmon[90393]: 2022-05-14T01:28:52.779+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:28:53.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:28:52 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:28:53.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:28:52 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T01:28:53.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:28:52 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T01:28:53.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:28:52 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T01:28:53.541 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:28:53 smithi047 conmon[78521]: 2022-05-14T01:28:53.296+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:28:53.541 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:28:53 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T01:28:53.542 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:28:53 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:28:53.542 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:28:53 smithi047 conmon[78521]: yield 2022-05-14T01:28:53.542 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:28:53 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:28:53.542 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:28:53 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:28:53.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:28:53 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:28:53.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:28:53 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T01:28:53.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:28:53 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:28:53.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:28:53 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T01:28:53.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:28:53 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:28:53.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:28:53 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T01:28:53.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:28:53 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:28:53.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:28:53 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T01:28:53.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:28:53 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:28:53.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:28:53 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T01:28:53.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:28:53 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:28:53.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:28:53 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:28:53.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:28:53 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:28:54.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:28:53 smithi115 conmon[90393]: 2022-05-14T01:28:53.759+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:28:54.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:28:53 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:28:54.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:28:53 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:28:54.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:28:53 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:28:54.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:28:53 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:28:54.109 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T01:28:54.109 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:28:54.109 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T01:28:54.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:28:54.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:28:54.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:28:54.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T01:28:54.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:28:54.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T01:28:54.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:28:54.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T01:28:54.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:28:54.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T01:28:54.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:28:54.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T01:28:54.113 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:28:54.113 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:28:54.113 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:28:54.114 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:28:53 smithi115 ceph-mon[78593]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T01:28:54.114 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:28:53 smithi115 ceph-mon[78593]: pgmap v3986: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:28:54.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:28:53 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:28:54.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:28:53 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:28:54.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:28:53 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:28:54.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:28:53 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:28:54.292 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T01:28:54.292 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:28:54.292 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T01:28:54.292 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:28:54.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:28:54.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:28:54.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T01:28:54.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:28:54.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T01:28:54.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:28:54.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T01:28:54.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:28:54.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T01:28:54.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:28:54.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T01:28:54.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:28:54.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:28:54.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:28:54.296 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:28:53 smithi047 ceph-mon[77147]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T01:28:54.296 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:28:53 smithi047 ceph-mon[77147]: pgmap v3986: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:28:55.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:28:54 smithi115 conmon[90393]: 2022-05-14T01:28:54.740+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:28:55.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:28:54 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:28:55.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:28:54 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 7265 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:28:55.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:28:54 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:28:55.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:28:54 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 7265 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:28:56.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:28:55 smithi115 conmon[90393]: 2022-05-14T01:28:55.773+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:28:56.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:28:55 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:28:56.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:28:55 smithi115 ceph-mon[78593]: pgmap v3987: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:28:56.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:28:55 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:28:56.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:28:55 smithi047 ceph-mon[77147]: pgmap v3987: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:28:56.791 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:28:56 smithi047 conmon[78521]: 2022-05-14T01:28:56.369+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:28:56.792 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:28:56 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T01:28:56.792 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:28:56 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:28:56.792 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:28:56 smithi047 conmon[78521]: yield 2022-05-14T01:28:56.792 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:28:56 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:28:56.793 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:28:56 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:28:56.793 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:28:56 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:28:56.793 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:28:56 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T01:28:56.793 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:28:56 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:28:56.793 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:28:56 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T01:28:56.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:28:56 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:28:56.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:28:56 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T01:28:56.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:28:56 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:28:56.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:28:56 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T01:28:56.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:28:56 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:28:56.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:28:56 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T01:28:56.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:28:56 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:28:56.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:28:56 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:28:56.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:28:56 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:28:56.797 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:28:56 smithi047 conmon[78521]: 2022-05-14T01:28:56.370+0000 7f5344f82700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T01:28:56.797 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:28:56 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T01:28:56.797 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:28:56 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:28:56.798 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:28:56 smithi047 conmon[78521]: yield 2022-05-14T01:28:56.798 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:28:56 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:28:56.798 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:28:56 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:28:56.799 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:28:56 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:28:56.799 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:28:56 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T01:28:56.799 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:28:56 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:28:56.799 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:28:56 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T01:28:56.800 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:28:56 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:28:56.800 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:28:56 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T01:28:56.800 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:28:56 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:28:56.801 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:28:56 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T01:28:56.801 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:28:56 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:28:56.801 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:28:56 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T01:28:56.801 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:28:56 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:28:56.801 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:28:56 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:28:56.802 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:28:56 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:28:56.802 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:28:56 smithi047 conmon[78521]: 2022-05-14T01:28:56.802 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:28:56 smithi047 conmon[78521]: During handling of the above exception, another exception occurred: 2022-05-14T01:28:56.802 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:28:56 smithi047 conmon[78521]: 2022-05-14T01:28:56.803 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:28:56 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T01:28:56.803 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:28:56 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T01:28:56.803 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:28:56 smithi047 conmon[78521]: return f(*arg) 2022-05-14T01:28:56.803 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:28:56 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T01:28:56.803 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:28:56 smithi047 conmon[78521]: self._write_client_files(client_files, host) 2022-05-14T01:28:56.804 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:28:56 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T01:28:56.804 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:28:56 smithi047 conmon[78521]: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T01:28:56.804 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:28:56 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T01:28:56.804 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:28:56 smithi047 conmon[78521]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T01:28:56.805 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:28:56 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T01:28:56.805 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:28:56 smithi047 conmon[78521]: return self.event_loop.get_result(coro) 2022-05-14T01:28:56.805 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:28:56 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T01:28:56.805 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:28:56 smithi047 conmon[78521]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T01:28:56.805 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:28:56 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T01:28:56.806 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:28:56 smithi047 conmon[78521]: return self.__get_result() 2022-05-14T01:28:56.806 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:28:56 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T01:28:56.806 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:28:56 smithi047 conmon[78521]: raise self._exception 2022-05-14T01:28:56.806 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:28:56 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T01:28:56.807 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:28:56 smithi047 conmon[78521]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T01:28:56.807 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:28:56 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T01:28:56.807 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:28:56 smithi047 conmon[78521]: conn = await self._remote_connection(host, addr) 2022-05-14T01:28:56.807 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:28:56 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T01:28:56.808 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:28:56 smithi047 conmon[78521]: raise 2022-05-14T01:28:56.808 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:28:56 smithi047 conmon[78521]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T01:28:56.808 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:28:56 smithi047 conmon[78521]: self.gen.throw(type, value, traceback) 2022-05-14T01:28:56.808 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:28:56 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T01:28:56.808 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:28:56 smithi047 conmon[78521]: raise OrchestratorError(msg) 2022-05-14T01:28:56.809 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:28:56 smithi047 conmon[78521]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:28:57.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:28:56 smithi115 conmon[90393]: 2022-05-14T01:28:56.749+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:28:57.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:28:56 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:28:57.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:28:56 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:28:57.109 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T01:28:57.109 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:28:57.109 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T01:28:57.109 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:28:57.109 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:28:57.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:28:57.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T01:28:57.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:28:57.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T01:28:57.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:28:57.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T01:28:57.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:28:57.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T01:28:57.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:28:57.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T01:28:57.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:28:57.113 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:28:57.113 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:28:57.113 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:28:56 smithi115 ceph-mon[78593]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T01:28:57.113 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T01:28:57.114 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:28:57.114 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T01:28:57.114 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:28:57.114 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:28:57.114 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:28:57.115 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T01:28:57.115 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:28:57.115 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T01:28:57.115 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:28:57.116 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T01:28:57.116 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:28:57.116 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T01:28:57.116 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:28:57.116 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T01:28:57.117 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:28:57.117 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:28:57.117 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:28:57.117 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T01:28:57.117 INFO:journalctl@ceph.mon.c.smithi115.stdout: During handling of the above exception, another exception occurred: 2022-05-14T01:28:57.118 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T01:28:57.118 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T01:28:57.118 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T01:28:57.118 INFO:journalctl@ceph.mon.c.smithi115.stdout: return f(*arg) 2022-05-14T01:28:57.119 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T01:28:57.119 INFO:journalctl@ceph.mon.c.smithi115.stdout: self._write_client_files(client_files, host) 2022-05-14T01:28:57.119 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T01:28:57.119 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T01:28:57.119 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T01:28:57.120 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T01:28:57.120 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T01:28:57.120 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.event_loop.get_result(coro) 2022-05-14T01:28:57.120 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T01:28:57.120 INFO:journalctl@ceph.mon.c.smithi115.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T01:28:57.121 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T01:28:57.121 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.__get_result() 2022-05-14T01:28:57.121 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T01:28:57.121 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise self._exception 2022-05-14T01:28:57.122 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T01:28:57.122 INFO:journalctl@ceph.mon.c.smithi115.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T01:28:57.122 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T01:28:57.122 INFO:journalctl@ceph.mon.c.smithi115.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T01:28:57.122 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T01:28:57.123 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise 2022-05-14T01:28:57.123 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T01:28:57.123 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.gen.throw(type, value, traceback) 2022-05-14T01:28:57.123 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T01:28:57.123 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OrchestratorError(msg) 2022-05-14T01:28:57.124 INFO:journalctl@ceph.mon.c.smithi115.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:28:57.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:28:56 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:28:57.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:28:56 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:28:57.291 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T01:28:57.292 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:28:57.292 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T01:28:57.292 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:28:57.292 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:28:57.292 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:28:57.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T01:28:57.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:28:57.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T01:28:57.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:28:57.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T01:28:57.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:28:57.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T01:28:57.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:28:57.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T01:28:57.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:28:57.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:28:57.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:28:57.295 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:28:56 smithi047 ceph-mon[77147]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T01:28:57.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T01:28:57.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:28:57.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T01:28:57.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:28:57.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:28:57.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:28:57.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T01:28:57.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:28:57.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T01:28:57.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:28:57.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T01:28:57.298 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:28:57.298 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T01:28:57.298 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:28:57.298 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T01:28:57.299 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:28:57.299 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:28:57.299 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:28:57.299 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T01:28:57.299 INFO:journalctl@ceph.mon.b.smithi047.stdout: During handling of the above exception, another exception occurred: 2022-05-14T01:28:57.300 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T01:28:57.300 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T01:28:57.300 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T01:28:57.300 INFO:journalctl@ceph.mon.b.smithi047.stdout: return f(*arg) 2022-05-14T01:28:57.300 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T01:28:57.301 INFO:journalctl@ceph.mon.b.smithi047.stdout: self._write_client_files(client_files, host) 2022-05-14T01:28:57.301 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T01:28:57.301 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T01:28:57.301 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T01:28:57.301 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T01:28:57.302 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T01:28:57.302 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.event_loop.get_result(coro) 2022-05-14T01:28:57.302 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T01:28:57.302 INFO:journalctl@ceph.mon.b.smithi047.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T01:28:57.302 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T01:28:57.303 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.__get_result() 2022-05-14T01:28:57.303 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T01:28:57.303 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise self._exception 2022-05-14T01:28:57.303 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T01:28:57.303 INFO:journalctl@ceph.mon.b.smithi047.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T01:28:57.304 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T01:28:57.304 INFO:journalctl@ceph.mon.b.smithi047.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T01:28:57.304 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T01:28:57.304 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise 2022-05-14T01:28:57.304 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T01:28:57.305 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.gen.throw(type, value, traceback) 2022-05-14T01:28:57.305 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T01:28:57.305 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OrchestratorError(msg) 2022-05-14T01:28:57.305 INFO:journalctl@ceph.mon.b.smithi047.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:28:58.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:28:57 smithi115 conmon[90393]: 2022-05-14T01:28:57.796+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:28:58.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:28:57 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:28:58.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:28:57 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:28:58.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:28:57 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:28:58.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:28:57 smithi115 ceph-mon[78593]: pgmap v3988: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:28:58.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:28:57 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:28:58.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:28:57 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:28:58.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:28:57 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:28:58.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:28:57 smithi047 ceph-mon[77147]: pgmap v3988: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:28:59.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:28:58 smithi115 conmon[90393]: 2022-05-14T01:28:58.794+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:28:59.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:28:58 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:28:59.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:28:58 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:29:00.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:28:59 smithi115 conmon[90393]: 2022-05-14T01:28:59.825+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:29:00.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:28:59 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:29:00.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:28:59 smithi115 ceph-mon[78593]: pgmap v3989: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:29:00.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:28:59 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 7270 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:29:00.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:28:59 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:29:00.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:28:59 smithi047 ceph-mon[77147]: pgmap v3989: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:29:00.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:28:59 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 7270 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:29:01.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:29:00 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:29:01.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:29:00 smithi115 conmon[90393]: 2022-05-14T01:29:00.779+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:29:01.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:29:00 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:29:02.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:29:01 smithi115 conmon[90393]: 2022-05-14T01:29:01.809+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:29:02.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:29:01 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:29:02.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:29:01 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:29:02.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:29:01 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:29:02.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:29:01 smithi115 ceph-mon[78593]: pgmap v3990: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:29:02.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:29:01 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:29:02.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:29:01 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:29:02.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:29:01 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:29:02.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:29:01 smithi047 ceph-mon[77147]: pgmap v3990: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:29:03.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:29:02 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:29:03.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:29:02 smithi115 conmon[90393]: 2022-05-14T01:29:02.792+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:29:03.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:29:02 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:29:04.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:29:03 smithi115 conmon[90393]: 2022-05-14T01:29:03.747+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:29:04.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:29:03 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:29:04.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:29:03 smithi115 ceph-mon[78593]: pgmap v3991: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:29:04.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:29:03 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:29:04.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:29:03 smithi047 ceph-mon[77147]: pgmap v3991: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:29:05.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:29:04 smithi115 conmon[90393]: 2022-05-14T01:29:04.763+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:29:05.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:29:04 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:29:05.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:29:04 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 7275 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:29:05.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:29:04 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:29:05.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:29:04 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 7275 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:29:06.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:29:05 smithi115 conmon[90393]: 2022-05-14T01:29:05.810+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:29:06.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:29:05 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:29:06.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:29:05 smithi115 ceph-mon[78593]: pgmap v3992: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:29:06.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:29:05 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:29:06.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:29:05 smithi047 ceph-mon[77147]: pgmap v3992: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:29:07.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:29:06 smithi115 conmon[90393]: 2022-05-14T01:29:06.769+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:29:07.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:29:06 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:29:07.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:29:06 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:29:08.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:29:07 smithi115 conmon[90393]: 2022-05-14T01:29:07.754+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:29:08.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:29:07 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:29:08.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:29:07 smithi115 ceph-mon[78593]: pgmap v3993: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:29:08.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:29:07 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:29:08.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:29:07 smithi047 ceph-mon[77147]: pgmap v3993: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:29:08.979 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:29:08 smithi115 conmon[90393]: 2022-05-14T01:29:08.723+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:29:09.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:29:08 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:29:09.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:29:08 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:29:09.988 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:29:09 smithi115 conmon[90393]: 2022-05-14T01:29:09.682+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:29:10.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:29:09 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:29:10.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:29:09 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:29:10.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:29:09 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:29:10.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:29:09 smithi047 ceph-mon[77147]: pgmap v3994: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:29:10.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:29:09 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 7280 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:29:10.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:29:09 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:29:10.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:29:09 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:29:10.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:29:09 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:29:10.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:29:09 smithi115 ceph-mon[78593]: pgmap v3994: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:29:10.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:29:09 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 7280 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:29:10.990 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:29:10 smithi115 conmon[90393]: 2022-05-14T01:29:10.649+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:29:11.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:29:10 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:29:11.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:29:10 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:29:11.994 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:29:11 smithi115 conmon[90393]: 2022-05-14T01:29:11.629+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:29:12.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:29:11 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:29:12.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:29:11 smithi047 ceph-mon[77147]: pgmap v3995: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:29:12.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:29:11 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:29:12.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:29:11 smithi115 ceph-mon[78593]: pgmap v3995: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:29:12.998 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:29:12 smithi115 conmon[90393]: 2022-05-14T01:29:12.637+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:29:13.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:29:12 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:29:13.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:29:12 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:29:14.011 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:29:13 smithi115 conmon[90393]: 2022-05-14T01:29:13.656+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:29:14.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:29:14 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:29:14.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:29:14 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:29:14.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:29:14 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:29:14.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:29:14 smithi047 ceph-mon[77147]: pgmap v3996: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:29:14.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:29:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T01:29:14.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:29:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T01:29:14.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:29:14 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:29:14.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:29:14 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:29:14.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:29:14 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:29:14.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:29:14 smithi115 ceph-mon[78593]: pgmap v3996: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:29:14.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:29:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T01:29:14.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:29:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T01:29:15.013 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:29:14 smithi115 conmon[90393]: 2022-05-14T01:29:14.659+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:29:15.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:29:15 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:29:15.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:29:15 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 7285 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:29:15.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:29:15 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:29:15.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:29:15 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 7285 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:29:16.018 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:29:15 smithi115 conmon[90393]: 2022-05-14T01:29:15.674+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:29:16.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:29:16 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:29:16.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:29:16 smithi047 ceph-mon[77147]: pgmap v3997: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:29:16.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:29:16 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:29:16.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:29:16 smithi115 ceph-mon[78593]: pgmap v3997: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:29:17.024 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:29:16 smithi115 conmon[90393]: 2022-05-14T01:29:16.637+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:29:17.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:29:17 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:29:17.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:29:17 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:29:18.023 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:29:17 smithi115 conmon[90393]: 2022-05-14T01:29:17.679+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:29:18.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:29:18 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:29:18.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:29:18 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:29:18.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:29:18 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:29:18.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:29:18 smithi047 ceph-mon[77147]: pgmap v3998: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:29:18.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:29:18 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:29:18.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:29:18 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:29:18.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:29:18 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:29:18.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:29:18 smithi115 ceph-mon[78593]: pgmap v3998: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:29:19.029 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:29:18 smithi115 conmon[90393]: 2022-05-14T01:29:18.709+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:29:19.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:29:19 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:29:19.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:29:19 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:29:20.034 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:29:19 smithi115 conmon[90393]: 2022-05-14T01:29:19.710+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:29:20.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:29:20 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:29:20.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:29:20 smithi047 ceph-mon[77147]: pgmap v3999: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:29:20.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:29:20 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 7290 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:29:20.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:29:20 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:29:20.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:29:20 smithi115 ceph-mon[78593]: pgmap v3999: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:29:20.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:29:20 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 7290 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:29:21.039 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:29:20 smithi115 conmon[90393]: 2022-05-14T01:29:20.738+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:29:21.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:29:21 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:29:21.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:29:21 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:29:22.041 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:29:21 smithi115 conmon[90393]: 2022-05-14T01:29:21.768+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:29:22.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:29:22 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:29:22.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:29:22 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:29:22.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:29:22 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:29:22.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:29:22 smithi115 ceph-mon[78593]: pgmap v4000: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:29:22.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:29:22 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:29:22.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:29:22 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:29:22.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:29:22 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:29:22.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:29:22 smithi047 ceph-mon[77147]: pgmap v4000: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:29:23.047 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:29:22 smithi115 conmon[90393]: 2022-05-14T01:29:22.780+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:29:23.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:29:23 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:29:23.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:29:23 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:29:24.052 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:29:23 smithi115 conmon[90393]: 2022-05-14T01:29:23.790+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:29:24.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:29:24 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:29:24.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:29:24 smithi115 ceph-mon[78593]: pgmap v4001: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:29:24.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:29:24 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:29:24.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:29:24 smithi047 ceph-mon[77147]: pgmap v4001: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:29:25.053 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:29:24 smithi115 conmon[90393]: 2022-05-14T01:29:24.788+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:29:25.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:29:25 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:29:25.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:29:25 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 7295 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:29:25.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:29:25 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:29:25.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:29:25 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 7295 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:29:26.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:29:25 smithi115 conmon[90393]: 2022-05-14T01:29:25.815+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:29:26.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:29:26 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:29:26.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:29:26 smithi115 ceph-mon[78593]: pgmap v4002: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:29:26.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:29:26 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:29:26.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:29:26 smithi047 ceph-mon[77147]: pgmap v4002: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:29:27.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:29:26 smithi115 conmon[90393]: 2022-05-14T01:29:26.844+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:29:27.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:29:27 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:29:27.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:29:27 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:29:28.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:29:27 smithi115 conmon[90393]: 2022-05-14T01:29:27.844+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:29:28.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:29:28 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:29:28.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:29:28 smithi115 ceph-mon[78593]: pgmap v4003: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:29:28.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:29:28 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:29:28.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:29:28 smithi047 ceph-mon[77147]: pgmap v4003: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:29:29.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:29:28 smithi115 conmon[90393]: 2022-05-14T01:29:28.828+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:29:29.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:29:29 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:29:29.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:29:29 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:29:30.072 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:29:29 smithi115 conmon[90393]: 2022-05-14T01:29:29.793+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:29:30.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:29:30 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:29:30.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:29:30 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:29:30.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:29:30 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:29:30.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:29:30 smithi115 ceph-mon[78593]: pgmap v4004: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:29:30.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:29:30 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 7300 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:29:30.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:29:30 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:29:30.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:29:30 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:29:30.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:29:30 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:29:30.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:29:30 smithi047 ceph-mon[77147]: pgmap v4004: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:29:30.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:29:30 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 7300 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:29:31.077 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:29:30 smithi115 conmon[90393]: 2022-05-14T01:29:30.812+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:29:31.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:29:31 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:29:31.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:29:31 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:29:32.083 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:29:31 smithi115 conmon[90393]: 2022-05-14T01:29:31.787+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:29:32.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:29:32 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:29:32.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:29:32 smithi115 ceph-mon[78593]: pgmap v4005: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:29:32.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:29:32 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:29:32.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:29:32 smithi047 ceph-mon[77147]: pgmap v4005: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:29:33.084 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:29:32 smithi115 conmon[90393]: 2022-05-14T01:29:32.739+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:29:33.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:29:33 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:29:33.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:29:33 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:29:34.094 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:29:33 smithi115 conmon[90393]: 2022-05-14T01:29:33.724+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:29:34.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:29:34 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:29:34.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:29:34 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:29:34.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:29:34 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:29:34.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:29:34 smithi115 ceph-mon[78593]: pgmap v4006: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:29:34.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:29:34 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:29:34.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:29:34 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:29:34.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:29:34 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:29:34.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:29:34 smithi047 ceph-mon[77147]: pgmap v4006: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:29:35.095 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:29:34 smithi115 conmon[90393]: 2022-05-14T01:29:34.695+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:29:35.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:29:35 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:29:35.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:29:35 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 7305 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:29:35.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:29:35 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:29:35.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:29:35 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 7305 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:29:36.100 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:29:35 smithi115 conmon[90393]: 2022-05-14T01:29:35.681+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:29:36.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:29:36 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:29:36.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:29:36 smithi115 ceph-mon[78593]: pgmap v4007: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:29:36.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:29:36 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:29:36.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:29:36 smithi047 ceph-mon[77147]: pgmap v4007: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:29:37.101 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:29:36 smithi115 conmon[90393]: 2022-05-14T01:29:36.698+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:29:37.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:29:37 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:29:37.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:29:37 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:29:38.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:29:37 smithi115 conmon[90393]: 2022-05-14T01:29:37.700+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:29:38.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:29:38 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:29:38.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:29:38 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:29:38.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:29:38 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:29:38.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:29:38 smithi047 ceph-mon[77147]: pgmap v4008: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:29:38.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:29:38 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:29:38.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:29:38 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:29:38.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:29:38 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:29:38.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:29:38 smithi115 ceph-mon[78593]: pgmap v4008: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:29:39.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:29:38 smithi115 conmon[90393]: 2022-05-14T01:29:38.673+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:29:39.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:29:39 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:29:39.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:29:39 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:29:40.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:29:39 smithi115 conmon[90393]: 2022-05-14T01:29:39.696+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:29:40.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:29:40 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:29:40.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:29:40 smithi047 ceph-mon[77147]: pgmap v4009: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:29:40.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:29:40 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 7310 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:29:40.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:29:40 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:29:40.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:29:40 smithi115 ceph-mon[78593]: pgmap v4009: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:29:40.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:29:40 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 7310 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:29:41.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:29:40 smithi115 conmon[90393]: 2022-05-14T01:29:40.665+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:29:41.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:29:41 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:29:41.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:29:41 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:29:42.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:29:41 smithi115 conmon[90393]: 2022-05-14T01:29:41.678+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:29:42.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:29:42 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:29:42.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:29:42 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:29:42.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:29:42 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:29:42.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:29:42 smithi047 ceph-mon[77147]: pgmap v4010: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:29:42.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:29:42 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:29:42.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:29:42 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:29:42.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:29:42 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:29:42.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:29:42 smithi115 ceph-mon[78593]: pgmap v4010: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:29:43.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:29:42 smithi115 conmon[90393]: 2022-05-14T01:29:42.679+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:29:43.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:29:43 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:29:43.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:29:43 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:29:44.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:29:43 smithi115 conmon[90393]: 2022-05-14T01:29:43.663+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:29:44.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:29:44 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:29:44.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:29:44 smithi047 ceph-mon[77147]: pgmap v4011: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:29:44.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:29:44 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:29:44.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:29:44 smithi115 ceph-mon[78593]: pgmap v4011: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:29:45.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:29:44 smithi115 conmon[90393]: 2022-05-14T01:29:44.662+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:29:45.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:29:45 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:29:45.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:29:45 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 7315 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:29:45.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:29:45 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:29:45.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:29:45 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 7315 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:29:46.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:29:45 smithi115 conmon[90393]: 2022-05-14T01:29:45.680+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:29:46.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:29:46 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:29:46.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:29:46 smithi047 ceph-mon[77147]: pgmap v4012: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:29:46.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:29:46 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:29:46.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:29:46 smithi115 ceph-mon[78593]: pgmap v4012: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:29:47.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:29:46 smithi115 conmon[90393]: 2022-05-14T01:29:46.689+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:29:47.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:29:47 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:29:47.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:29:47 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:29:48.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:29:47 smithi115 conmon[90393]: 2022-05-14T01:29:47.712+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:29:48.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:29:48 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:29:48.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:29:48 smithi047 ceph-mon[77147]: pgmap v4013: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:29:48.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:29:48 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:29:48.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:29:48 smithi115 ceph-mon[78593]: pgmap v4013: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:29:49.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:29:48 smithi115 conmon[90393]: 2022-05-14T01:29:48.760+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:29:49.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:29:49 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:29:49.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:29:49 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:29:50.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:29:49 smithi115 conmon[90393]: 2022-05-14T01:29:49.757+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:29:50.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:29:50 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:29:50.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:29:50 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:29:50.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:29:50 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:29:50.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:29:50 smithi047 ceph-mon[77147]: pgmap v4014: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:29:50.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:29:50 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 7320 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:29:50.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:29:50 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:29:50.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:29:50 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:29:50.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:29:50 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:29:50.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:29:50 smithi115 ceph-mon[78593]: pgmap v4014: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:29:50.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:29:50 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 7320 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:29:51.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:29:50 smithi115 conmon[90393]: 2022-05-14T01:29:50.724+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:29:51.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:29:51 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:29:51.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:29:51 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:29:52.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:29:51 smithi115 conmon[90393]: 2022-05-14T01:29:51.681+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:29:52.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:29:52 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:29:52.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:29:52 smithi047 ceph-mon[77147]: pgmap v4015: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:29:52.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:29:52 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:29:52.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:29:52 smithi115 ceph-mon[78593]: pgmap v4015: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:29:53.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:29:52 smithi115 conmon[90393]: 2022-05-14T01:29:52.664+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:29:53.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:29:53 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:29:53.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:29:53 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:29:54.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:29:53 smithi115 conmon[90393]: 2022-05-14T01:29:53.642+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:29:54.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:29:54 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:29:54.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:29:54 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:29:54.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:29:54 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:29:54.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:29:54 smithi047 ceph-mon[77147]: pgmap v4016: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:29:54.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:29:54 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:29:54.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:29:54 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:29:54.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:29:54 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:29:54.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:29:54 smithi115 ceph-mon[78593]: pgmap v4016: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:29:55.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:29:54 smithi115 conmon[90393]: 2022-05-14T01:29:54.658+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:29:55.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:29:55 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:29:55.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:29:55 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 7325 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:29:55.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:29:55 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:29:55.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:29:55 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 7325 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:29:56.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:29:55 smithi115 conmon[90393]: 2022-05-14T01:29:55.657+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:29:56.443 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:29:56 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:29:56.443 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:29:56 smithi047 ceph-mon[77147]: pgmap v4017: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:29:56.446 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:29:56 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:29:56.447 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:29:56 smithi115 ceph-mon[78593]: pgmap v4017: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:29:56.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:29:56 smithi115 conmon[90393]: 2022-05-14T01:29:56.691+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:29:57.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:29:57 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:29:57.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:29:57 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T01:29:57.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:29:57 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T01:29:57.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:29:57 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T01:29:57.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:29:57 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:29:57.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:29:57 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T01:29:57.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:29:57 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T01:29:57.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:29:57 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T01:29:58.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:29:57 smithi115 conmon[90393]: 2022-05-14T01:29:57.669+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:29:58.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:29:58 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:29:58.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:29:58 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:29:58.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:29:58 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:29:58.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:29:58 smithi047 ceph-mon[77147]: pgmap v4018: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:29:58.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:29:58 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:29:58.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:29:58 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:29:58.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:29:58 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:29:58.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:29:58 smithi115 ceph-mon[78593]: pgmap v4018: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:29:59.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:29:58 smithi115 conmon[90393]: 2022-05-14T01:29:58.680+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:29:59.473 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:29:59 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:29:59.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:29:59 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:29:59.791 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:29:59 smithi047 conmon[78521]: 2022-05-14T01:29:59.472+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:29:59.791 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:29:59 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T01:29:59.792 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:29:59 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:29:59.792 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:29:59 smithi047 conmon[78521]: yield 2022-05-14T01:29:59.792 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:29:59 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:29:59.792 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:29:59 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:29:59.793 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:29:59 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:29:59.793 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:29:59 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T01:29:59.793 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:29:59 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:29:59.793 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:29:59 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T01:29:59.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:29:59 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:29:59.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:29:59 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T01:29:59.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:29:59 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:29:59.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:29:59 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T01:29:59.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:29:59 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:29:59.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:29:59 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T01:29:59.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:29:59 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:29:59.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:29:59 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:29:59.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:29:59 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:30:00.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:29:59 smithi115 conmon[90393]: 2022-05-14T01:29:59.677+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:30:00.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:30:00 smithi047 conmon[77123]: 2022-05-14T01:29:59.999+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : Health detail: HEALTH_ERR 1 filesystem is degraded; 1 mds daemon damaged; 1 MDSs are read only; 1 MDSs report slow metadata IOs; 1/3 mons down, quorum b,c; Reduced data availability: 1 pg inactive; 9 slow ops, oldest one blocked for 7330 sec, osd.5 has slow ops 2022-05-14T01:30:00.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:30:00 smithi047 conmon[77123]: 2022-05-14T01:29:59.999+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : [WRN] FS_DEGRADED: 1 filesystem is degraded 2022-05-14T01:30:00.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:30:00 smithi047 conmon[77123]: 2022-05-14T01:29:59.999+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : fs cephfs is degraded 2022-05-14T01:30:00.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:30:00 smithi047 conmon[77123]: 2022-05-14T01:29:59.999+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : [ERR] MDS_DAMAGE: 1 mds daemon damaged 2022-05-14T01:30:00.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:30:00 smithi047 conmon[77123]: 2022-05-14T01:29:59.999+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : fs cephfs mds.2 is damaged 2022-05-14T01:30:00.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:30:00 smithi047 conmon[77123]: 2022-05-14T01:29:59.999+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : [WRN] MDS_READ_ONLY: 1 MDSs are read only 2022-05-14T01:30:00.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:30:00 smithi047 conmon[77123]: 2022-05-14T01:29:59.999+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : mds.l(mds.0): MDS in read-only mode 2022-05-14T01:30:00.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:30:00 smithi047 conmon[77123]: 2022-05-14T01:29:59.999+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : [WRN] MDS_SLOW_METADATA_IO: 1 MDSs report slow metadata IOs 2022-05-14T01:30:00.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:30:00 smithi047 conmon[77123]: 2022-05-14T01:29:59.999+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : mds.l(mds.0): 4 slow metadata IOs are blocked > 180 secs, oldest blocked for 7633 secs 2022-05-14T01:30:00.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:30:00 smithi047 conmon[77123]: 2022-05-14T01:29:59.999+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : [WRN] MON_DOWN: 1/3 mons down, quorum b,c 2022-05-14T01:30:00.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:30:00 smithi047 conmon[77123]: 2022-05-14T01:29:59.999+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : mon.a (rank 0) addr [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] is down (out of quorum) 2022-05-14T01:30:00.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:30:00 smithi047 conmon[77123]: 2022-05-14T01:29:59.999+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : [WRN] PG_AVAILABILITY: Reduced data availability: 1 pg inactive 2022-05-14T01:30:00.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:30:00 smithi047 conmon[77123]: 2022-05-14T01:29:59.999+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : pg 2.1 is stuck inactive for 2h, current state unknown, last acting [] 2022-05-14T01:30:00.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:30:00 smithi047 conmon[77123]: 2022-05-14T01:29:59.999+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : [WRN] SLOW_OPS: 9 slow ops, oldest one blocked for 7330 sec, osd.5 has slow ops 2022-05-14T01:30:00.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:30:00 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:30:00.295 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:30:00 smithi047 ceph-mon[77147]: pgmap v4019: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:30:00.295 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:30:00 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:30:00.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T01:30:00.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:30:00.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T01:30:00.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:30:00.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:30:00.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:30:00.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T01:30:00.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:30:00.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T01:30:00.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:30:00.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T01:30:00.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:30:00.298 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T01:30:00.298 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:30:00.298 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T01:30:00.298 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:30:00.299 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:30:00.299 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:30:00.299 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:30:00 smithi047 ceph-mon[77147]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T01:30:00.299 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:30:00 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 7330 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:30:00.300 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:30:00 smithi047 ceph-mon[77147]: Health detail: HEALTH_ERR 1 filesystem is degraded; 1 mds daemon damaged; 1 MDSs are read only; 1 MDSs report slow metadata IOs; 1/3 mons down, quorum b,c; Reduced data availability: 1 pg inactive; 9 slow ops, oldest one blocked for 7330 sec, osd.5 has slow ops 2022-05-14T01:30:00.300 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:30:00 smithi047 ceph-mon[77147]: [WRN] FS_DEGRADED: 1 filesystem is degraded 2022-05-14T01:30:00.300 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:30:00 smithi047 ceph-mon[77147]: fs cephfs is degraded 2022-05-14T01:30:00.300 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:30:00 smithi047 ceph-mon[77147]: [ERR] MDS_DAMAGE: 1 mds daemon damaged 2022-05-14T01:30:00.300 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:30:00 smithi047 ceph-mon[77147]: fs cephfs mds.2 is damaged 2022-05-14T01:30:00.301 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:30:00 smithi047 ceph-mon[77147]: [WRN] MDS_READ_ONLY: 1 MDSs are read only 2022-05-14T01:30:00.301 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:30:00 smithi047 ceph-mon[77147]: mds.l(mds.0): MDS in read-only mode 2022-05-14T01:30:00.301 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:30:00 smithi047 ceph-mon[77147]: [WRN] MDS_SLOW_METADATA_IO: 1 MDSs report slow metadata IOs 2022-05-14T01:30:00.301 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:30:00 smithi047 ceph-mon[77147]: mds.l(mds.0): 4 slow metadata IOs are blocked > 180 secs, oldest blocked for 7633 secs 2022-05-14T01:30:00.302 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:30:00 smithi047 ceph-mon[77147]: [WRN] MON_DOWN: 1/3 mons down, quorum b,c 2022-05-14T01:30:00.302 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:30:00 smithi047 ceph-mon[77147]: mon.a (rank 0) addr [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] is down (out of quorum) 2022-05-14T01:30:00.302 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:30:00 smithi047 ceph-mon[77147]: [WRN] PG_AVAILABILITY: Reduced data availability: 1 pg inactive 2022-05-14T01:30:00.302 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:30:00 smithi047 ceph-mon[77147]: pg 2.1 is stuck inactive for 2h, current state unknown, last acting [] 2022-05-14T01:30:00.302 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:30:00 smithi047 ceph-mon[77147]: [WRN] SLOW_OPS: 9 slow ops, oldest one blocked for 7330 sec, osd.5 has slow ops 2022-05-14T01:30:00.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:30:00 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:30:00.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:30:00 smithi115 ceph-mon[78593]: pgmap v4019: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:30:00.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:30:00 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:30:00.609 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T01:30:00.609 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:30:00.609 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T01:30:00.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:30:00.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:30:00.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:30:00.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T01:30:00.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:30:00.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T01:30:00.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:30:00.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T01:30:00.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:30:00.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T01:30:00.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:30:00.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T01:30:00.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:30:00.613 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:30:00.613 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:30:00.613 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:30:00 smithi115 ceph-mon[78593]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T01:30:00.613 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:30:00 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 7330 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:30:00.614 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:30:00 smithi115 ceph-mon[78593]: Health detail: HEALTH_ERR 1 filesystem is degraded; 1 mds daemon damaged; 1 MDSs are read only; 1 MDSs report slow metadata IOs; 1/3 mons down, quorum b,c; Reduced data availability: 1 pg inactive; 9 slow ops, oldest one blocked for 7330 sec, osd.5 has slow ops 2022-05-14T01:30:00.614 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:30:00 smithi115 ceph-mon[78593]: [WRN] FS_DEGRADED: 1 filesystem is degraded 2022-05-14T01:30:00.614 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:30:00 smithi115 ceph-mon[78593]: fs cephfs is degraded 2022-05-14T01:30:00.614 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:30:00 smithi115 ceph-mon[78593]: [ERR] MDS_DAMAGE: 1 mds daemon damaged 2022-05-14T01:30:00.614 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:30:00 smithi115 ceph-mon[78593]: fs cephfs mds.2 is damaged 2022-05-14T01:30:00.615 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:30:00 smithi115 ceph-mon[78593]: [WRN] MDS_READ_ONLY: 1 MDSs are read only 2022-05-14T01:30:00.615 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:30:00 smithi115 ceph-mon[78593]: mds.l(mds.0): MDS in read-only mode 2022-05-14T01:30:00.615 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:30:00 smithi115 ceph-mon[78593]: [WRN] MDS_SLOW_METADATA_IO: 1 MDSs report slow metadata IOs 2022-05-14T01:30:00.616 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:30:00 smithi115 ceph-mon[78593]: mds.l(mds.0): 4 slow metadata IOs are blocked > 180 secs, oldest blocked for 7633 secs 2022-05-14T01:30:00.616 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:30:00 smithi115 ceph-mon[78593]: [WRN] MON_DOWN: 1/3 mons down, quorum b,c 2022-05-14T01:30:00.616 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:30:00 smithi115 ceph-mon[78593]: mon.a (rank 0) addr [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] is down (out of quorum) 2022-05-14T01:30:00.616 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:30:00 smithi115 ceph-mon[78593]: [WRN] PG_AVAILABILITY: Reduced data availability: 1 pg inactive 2022-05-14T01:30:00.616 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:30:00 smithi115 ceph-mon[78593]: pg 2.1 is stuck inactive for 2h, current state unknown, last acting [] 2022-05-14T01:30:00.617 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:30:00 smithi115 ceph-mon[78593]: [WRN] SLOW_OPS: 9 slow ops, oldest one blocked for 7330 sec, osd.5 has slow ops 2022-05-14T01:30:01.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:30:00 smithi115 conmon[90393]: 2022-05-14T01:30:00.643+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:30:01.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:30:01 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:30:01.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:30:01 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:30:02.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:30:01 smithi115 conmon[90393]: 2022-05-14T01:30:01.641+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:30:02.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:30:02 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:30:02.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:30:02 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:30:02.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:30:02 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:30:02.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:30:02 smithi047 ceph-mon[77147]: pgmap v4020: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:30:02.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:30:02 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:30:02.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:30:02 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:30:02.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:30:02 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:30:02.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:30:02 smithi115 ceph-mon[78593]: pgmap v4020: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:30:03.041 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:30:02 smithi047 conmon[78521]: 2022-05-14T01:30:02.543+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:30:03.041 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:30:02 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T01:30:03.042 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:30:02 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:30:03.042 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:30:02 smithi047 conmon[78521]: yield 2022-05-14T01:30:03.042 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:30:02 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:30:03.042 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:30:02 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:30:03.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:30:02 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:30:03.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:30:02 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T01:30:03.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:30:02 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:30:03.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:30:02 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T01:30:03.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:30:02 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:30:03.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:30:02 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T01:30:03.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:30:02 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:30:03.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:30:02 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T01:30:03.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:30:02 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:30:03.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:30:02 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T01:30:03.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:30:02 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:30:03.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:30:02 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:30:03.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:30:02 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:30:03.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:30:02 smithi047 conmon[78521]: 2022-05-14T01:30:02.544+0000 7f5344781700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T01:30:03.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:30:02 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T01:30:03.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:30:02 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:30:03.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:30:02 smithi047 conmon[78521]: yield 2022-05-14T01:30:03.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:30:02 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:30:03.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:30:02 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:30:03.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:30:02 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:30:03.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:30:02 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T01:30:03.048 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:30:02 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:30:03.048 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:30:02 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T01:30:03.048 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:30:02 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:30:03.048 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:30:02 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T01:30:03.049 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:30:02 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:30:03.049 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:30:02 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T01:30:03.049 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:30:02 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:30:03.049 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:30:02 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T01:30:03.049 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:30:02 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:30:03.050 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:30:02 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:30:03.050 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:30:02 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:30:03.050 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:30:02 smithi047 conmon[78521]: 2022-05-14T01:30:03.050 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:30:02 smithi047 conmon[78521]: During handling of the above exception, another exception occurred: 2022-05-14T01:30:03.051 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:30:02 smithi047 conmon[78521]: 2022-05-14T01:30:03.051 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:30:02 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T01:30:03.051 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:30:02 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T01:30:03.051 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:30:02 smithi047 conmon[78521]: return f(*arg) 2022-05-14T01:30:03.052 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:30:02 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T01:30:03.052 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:30:02 smithi047 conmon[78521]: self._write_client_files(client_files, host) 2022-05-14T01:30:03.053 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:30:02 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T01:30:03.053 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:30:02 smithi047 conmon[78521]: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T01:30:03.053 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:30:02 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T01:30:03.053 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:30:02 smithi047 conmon[78521]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T01:30:03.053 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:30:02 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T01:30:03.054 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:30:02 smithi047 conmon[78521]: return self.event_loop.get_result(coro) 2022-05-14T01:30:03.054 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:30:02 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T01:30:03.054 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:30:02 smithi047 conmon[78521]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T01:30:03.054 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:30:02 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T01:30:03.055 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:30:02 smithi047 conmon[78521]: return self.__get_result() 2022-05-14T01:30:03.055 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:30:02 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T01:30:03.055 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:30:02 smithi047 conmon[78521]: raise self._exception 2022-05-14T01:30:03.055 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:30:02 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T01:30:03.055 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:30:02 smithi047 conmon[78521]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T01:30:03.056 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:30:02 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T01:30:03.056 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:30:02 smithi047 conmon[78521]: conn = await self._remote_connection(host, addr) 2022-05-14T01:30:03.056 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:30:02 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T01:30:03.056 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:30:02 smithi047 conmon[78521]: raise 2022-05-14T01:30:03.056 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:30:02 smithi047 conmon[78521]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T01:30:03.057 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:30:02 smithi047 conmon[78521]: self.gen.throw(type, value, traceback) 2022-05-14T01:30:03.057 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:30:02 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T01:30:03.057 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:30:02 smithi047 conmon[78521]: raise OrchestratorError(msg) 2022-05-14T01:30:03.057 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:30:02 smithi047 conmon[78521]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:30:03.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:30:02 smithi115 conmon[90393]: 2022-05-14T01:30:02.644+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:30:03.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:30:03 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:30:03.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:30:03 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:30:03.541 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T01:30:03.542 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:30:03.542 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T01:30:03.542 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:30:03.542 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:30:03.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:30:03.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T01:30:03.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:30:03.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T01:30:03.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:30:03.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T01:30:03.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:30:03.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T01:30:03.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:30:03.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T01:30:03.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:30:03.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:30:03.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:30:03.546 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:30:03 smithi047 ceph-mon[77147]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T01:30:03.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T01:30:03.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:30:03.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T01:30:03.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:30:03.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:30:03.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:30:03.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T01:30:03.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:30:03.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T01:30:03.548 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:30:03.548 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T01:30:03.548 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:30:03.548 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T01:30:03.549 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:30:03.549 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T01:30:03.549 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:30:03.549 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:30:03.549 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:30:03.550 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T01:30:03.550 INFO:journalctl@ceph.mon.b.smithi047.stdout: During handling of the above exception, another exception occurred: 2022-05-14T01:30:03.550 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T01:30:03.551 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T01:30:03.551 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T01:30:03.551 INFO:journalctl@ceph.mon.b.smithi047.stdout: return f(*arg) 2022-05-14T01:30:03.551 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T01:30:03.551 INFO:journalctl@ceph.mon.b.smithi047.stdout: self._write_client_files(client_files, host) 2022-05-14T01:30:03.552 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T01:30:03.552 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T01:30:03.552 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T01:30:03.552 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T01:30:03.553 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T01:30:03.553 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.event_loop.get_result(coro) 2022-05-14T01:30:03.553 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T01:30:03.553 INFO:journalctl@ceph.mon.b.smithi047.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T01:30:03.553 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T01:30:03.554 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.__get_result() 2022-05-14T01:30:03.554 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T01:30:03.554 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise self._exception 2022-05-14T01:30:03.554 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T01:30:03.555 INFO:journalctl@ceph.mon.b.smithi047.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T01:30:03.555 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T01:30:03.555 INFO:journalctl@ceph.mon.b.smithi047.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T01:30:03.555 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T01:30:03.556 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise 2022-05-14T01:30:03.556 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T01:30:03.556 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.gen.throw(type, value, traceback) 2022-05-14T01:30:03.556 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T01:30:03.557 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OrchestratorError(msg) 2022-05-14T01:30:03.557 INFO:journalctl@ceph.mon.b.smithi047.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:30:03.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:30:03 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:30:03.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:30:03 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:30:03.608 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T01:30:03.608 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:30:03.608 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T01:30:03.609 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:30:03.609 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:30:03.609 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:30:03.609 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T01:30:03.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:30:03.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T01:30:03.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:30:03.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T01:30:03.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:30:03.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T01:30:03.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:30:03.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T01:30:03.613 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:30:03.613 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:30:03.613 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:30:03.614 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:30:03 smithi115 ceph-mon[78593]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T01:30:03.614 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T01:30:03.614 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:30:03.615 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T01:30:03.615 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:30:03.615 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:30:03.616 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:30:03.616 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T01:30:03.616 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:30:03.617 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T01:30:03.617 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:30:03.617 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T01:30:03.618 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:30:03.618 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T01:30:03.618 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:30:03.618 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T01:30:03.619 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:30:03.619 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:30:03.619 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:30:03.619 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T01:30:03.619 INFO:journalctl@ceph.mon.c.smithi115.stdout: During handling of the above exception, another exception occurred: 2022-05-14T01:30:03.620 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T01:30:03.620 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T01:30:03.620 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T01:30:03.620 INFO:journalctl@ceph.mon.c.smithi115.stdout: return f(*arg) 2022-05-14T01:30:03.621 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T01:30:03.621 INFO:journalctl@ceph.mon.c.smithi115.stdout: self._write_client_files(client_files, host) 2022-05-14T01:30:03.621 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T01:30:03.621 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T01:30:03.622 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T01:30:03.622 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T01:30:03.622 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T01:30:03.622 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.event_loop.get_result(coro) 2022-05-14T01:30:03.622 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T01:30:03.623 INFO:journalctl@ceph.mon.c.smithi115.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T01:30:03.623 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T01:30:03.623 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.__get_result() 2022-05-14T01:30:03.623 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T01:30:03.624 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise self._exception 2022-05-14T01:30:03.624 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T01:30:03.624 INFO:journalctl@ceph.mon.c.smithi115.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T01:30:03.624 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T01:30:03.625 INFO:journalctl@ceph.mon.c.smithi115.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T01:30:03.625 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T01:30:03.626 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise 2022-05-14T01:30:03.626 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T01:30:03.626 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.gen.throw(type, value, traceback) 2022-05-14T01:30:03.627 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T01:30:03.627 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OrchestratorError(msg) 2022-05-14T01:30:03.627 INFO:journalctl@ceph.mon.c.smithi115.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:30:04.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:30:03 smithi115 conmon[90393]: 2022-05-14T01:30:03.645+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:30:04.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:30:04 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:30:04.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:30:04 smithi047 ceph-mon[77147]: pgmap v4021: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:30:04.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:30:04 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:30:04.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:30:04 smithi115 ceph-mon[78593]: pgmap v4021: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:30:05.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:30:04 smithi115 conmon[90393]: 2022-05-14T01:30:04.648+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:30:05.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:30:05 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:30:05.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:30:05 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 7335 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:30:05.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:30:05 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:30:05.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:30:05 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 7335 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:30:06.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:30:05 smithi115 conmon[90393]: 2022-05-14T01:30:05.691+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:30:06.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:30:06 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:30:06.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:30:06 smithi047 ceph-mon[77147]: pgmap v4022: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:30:06.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:30:06 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:30:06.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:30:06 smithi115 ceph-mon[78593]: pgmap v4022: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:30:07.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:30:06 smithi115 conmon[90393]: 2022-05-14T01:30:06.675+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:30:07.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:30:07 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:30:07.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:30:07 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:30:08.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:30:07 smithi115 conmon[90393]: 2022-05-14T01:30:07.687+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:30:08.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:30:08 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:30:08.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:30:08 smithi047 ceph-mon[77147]: pgmap v4023: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:30:08.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:30:08 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:30:08.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:30:08 smithi115 ceph-mon[78593]: pgmap v4023: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:30:09.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:30:08 smithi115 conmon[90393]: 2022-05-14T01:30:08.647+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:30:09.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:30:09 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:30:09.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:30:09 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:30:10.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:30:09 smithi115 conmon[90393]: 2022-05-14T01:30:09.683+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:30:10.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:30:10 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:30:10.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:30:10 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:30:10.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:30:10 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:30:10.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:30:10 smithi047 ceph-mon[77147]: pgmap v4024: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:30:10.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:30:10 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 7340 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:30:10.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:30:10 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:30:10.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:30:10 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:30:10.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:30:10 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:30:10.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:30:10 smithi115 ceph-mon[78593]: pgmap v4024: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:30:10.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:30:10 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 7340 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:30:11.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:30:10 smithi115 conmon[90393]: 2022-05-14T01:30:10.719+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:30:11.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:30:11 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:30:11.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:30:11 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:30:12.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:30:11 smithi115 conmon[90393]: 2022-05-14T01:30:11.708+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:30:12.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:30:12 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:30:12.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:30:12 smithi047 ceph-mon[77147]: pgmap v4025: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:30:12.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:30:12 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:30:12.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:30:12 smithi115 ceph-mon[78593]: pgmap v4025: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:30:13.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:30:12 smithi115 conmon[90393]: 2022-05-14T01:30:12.732+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:30:13.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:30:13 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:30:13.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:30:13 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:30:14.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:30:13 smithi115 conmon[90393]: 2022-05-14T01:30:13.759+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:30:14.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:30:14 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:30:14.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:30:14 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:30:14.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:30:14 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:30:14.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:30:14 smithi047 ceph-mon[77147]: pgmap v4026: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:30:14.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:30:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T01:30:14.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:30:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T01:30:14.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:30:14 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:30:14.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:30:14 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:30:14.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:30:14 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:30:14.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:30:14 smithi115 ceph-mon[78593]: pgmap v4026: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:30:14.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:30:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T01:30:14.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:30:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T01:30:15.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:30:14 smithi115 conmon[90393]: 2022-05-14T01:30:14.751+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:30:15.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:30:15 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:30:15.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:30:15 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 7345 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:30:15.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:30:15 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:30:15.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:30:15 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 7345 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:30:16.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:30:15 smithi115 conmon[90393]: 2022-05-14T01:30:15.733+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:30:16.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:30:16 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:30:16.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:30:16 smithi047 ceph-mon[77147]: pgmap v4027: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:30:16.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:30:16 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:30:16.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:30:16 smithi115 ceph-mon[78593]: pgmap v4027: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:30:17.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:30:16 smithi115 conmon[90393]: 2022-05-14T01:30:16.710+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:30:17.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:30:17 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:30:17.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:30:17 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:30:18.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:30:17 smithi115 conmon[90393]: 2022-05-14T01:30:17.670+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:30:18.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:30:18 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:30:18.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:30:18 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:30:18.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:30:18 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:30:18.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:30:18 smithi047 ceph-mon[77147]: pgmap v4028: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:30:18.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:30:18 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:30:18.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:30:18 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:30:18.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:30:18 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:30:18.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:30:18 smithi115 ceph-mon[78593]: pgmap v4028: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:30:19.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:30:18 smithi115 conmon[90393]: 2022-05-14T01:30:18.675+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:30:19.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:30:19 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:30:19.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:30:19 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:30:20.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:30:19 smithi115 conmon[90393]: 2022-05-14T01:30:19.669+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:30:20.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:30:20 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:30:20.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:30:20 smithi047 ceph-mon[77147]: pgmap v4029: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:30:20.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:30:20 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 7350 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:30:20.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:30:20 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:30:20.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:30:20 smithi115 ceph-mon[78593]: pgmap v4029: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:30:20.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:30:20 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 7350 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:30:21.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:30:20 smithi115 conmon[90393]: 2022-05-14T01:30:20.645+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:30:21.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:30:21 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:30:21.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:30:21 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:30:22.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:30:21 smithi115 conmon[90393]: 2022-05-14T01:30:21.638+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:30:22.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:30:22 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:30:22.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:30:22 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:30:22.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:30:22 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:30:22.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:30:22 smithi115 ceph-mon[78593]: pgmap v4030: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:30:22.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:30:22 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:30:22.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:30:22 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:30:22.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:30:22 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:30:22.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:30:22 smithi047 ceph-mon[77147]: pgmap v4030: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:30:23.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:30:22 smithi115 conmon[90393]: 2022-05-14T01:30:22.637+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:30:23.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:30:23 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:30:23.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:30:23 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:30:24.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:30:23 smithi115 conmon[90393]: 2022-05-14T01:30:23.677+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:30:24.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:30:24 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:30:24.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:30:24 smithi115 ceph-mon[78593]: pgmap v4031: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:30:24.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:30:24 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:30:24.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:30:24 smithi047 ceph-mon[77147]: pgmap v4031: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:30:25.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:30:24 smithi115 conmon[90393]: 2022-05-14T01:30:24.644+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:30:25.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:30:25 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:30:25.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:30:25 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 7355 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:30:25.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:30:25 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:30:25.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:30:25 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 7355 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:30:26.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:30:25 smithi115 conmon[90393]: 2022-05-14T01:30:25.639+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:30:26.601 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:30:26 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:30:26.602 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:30:26 smithi115 ceph-mon[78593]: pgmap v4032: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:30:26.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:30:26 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:30:26.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:30:26 smithi047 ceph-mon[77147]: pgmap v4032: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:30:26.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:30:26 smithi115 conmon[90393]: 2022-05-14T01:30:26.600+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:30:27.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:30:27 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:30:27.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:30:27 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:30:28.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:30:27 smithi115 conmon[90393]: 2022-05-14T01:30:27.637+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:30:28.597 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:30:28 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:30:28.598 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:30:28 smithi115 ceph-mon[78593]: pgmap v4033: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:30:28.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:30:28 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:30:28.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:30:28 smithi047 ceph-mon[77147]: pgmap v4033: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:30:28.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:30:28 smithi115 conmon[90393]: 2022-05-14T01:30:28.596+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:30:29.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:30:29 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:30:29.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:30:29 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:30:30.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:30:29 smithi115 conmon[90393]: 2022-05-14T01:30:29.637+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:30:30.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:30:30 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:30:30.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:30:30 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:30:30.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:30:30 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:30:30.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:30:30 smithi115 ceph-mon[78593]: pgmap v4034: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:30:30.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:30:30 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 7360 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:30:30.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:30:30 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:30:30.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:30:30 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:30:30.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:30:30 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:30:30.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:30:30 smithi047 ceph-mon[77147]: pgmap v4034: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:30:30.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:30:30 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 7360 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:30:31.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:30:30 smithi115 conmon[90393]: 2022-05-14T01:30:30.673+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:30:31.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:30:31 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:30:31.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:30:31 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:30:32.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:30:31 smithi115 conmon[90393]: 2022-05-14T01:30:31.663+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:30:32.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:30:32 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:30:32.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:30:32 smithi115 ceph-mon[78593]: pgmap v4035: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:30:32.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:30:32 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:30:32.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:30:32 smithi047 ceph-mon[77147]: pgmap v4035: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:30:33.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:30:32 smithi115 conmon[90393]: 2022-05-14T01:30:32.698+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:30:33.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:30:33 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:30:33.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:30:33 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:30:34.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:30:33 smithi115 conmon[90393]: 2022-05-14T01:30:33.736+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:30:34.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:30:34 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:30:34.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:30:34 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:30:34.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:30:34 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:30:34.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:30:34 smithi115 ceph-mon[78593]: pgmap v4036: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:30:34.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:30:34 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:30:34.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:30:34 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:30:34.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:30:34 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:30:34.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:30:34 smithi047 ceph-mon[77147]: pgmap v4036: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:30:35.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:30:34 smithi115 conmon[90393]: 2022-05-14T01:30:34.703+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:30:35.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:30:35 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:30:35.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:30:35 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 7365 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:30:35.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:30:35 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:30:35.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:30:35 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 7365 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:30:36.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:30:35 smithi115 conmon[90393]: 2022-05-14T01:30:35.691+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:30:36.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:30:36 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:30:36.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:30:36 smithi115 ceph-mon[78593]: pgmap v4037: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 7 op/s 2022-05-14T01:30:36.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:30:36 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:30:36.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:30:36 smithi047 ceph-mon[77147]: pgmap v4037: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 7 op/s 2022-05-14T01:30:37.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:30:36 smithi115 conmon[90393]: 2022-05-14T01:30:36.727+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:30:37.730 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:30:37 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:30:37.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:30:37 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:30:38.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:30:37 smithi115 conmon[90393]: 2022-05-14T01:30:37.729+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:30:38.693 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:30:38 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:30:38.693 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:30:38 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:30:38.693 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:30:38 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:30:38.694 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:30:38 smithi115 ceph-mon[78593]: pgmap v4038: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 4 op/s 2022-05-14T01:30:38.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:30:38 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:30:38.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:30:38 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:30:38.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:30:38 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:30:38.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:30:38 smithi047 ceph-mon[77147]: pgmap v4038: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 4 op/s 2022-05-14T01:30:39.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:30:38 smithi115 conmon[90393]: 2022-05-14T01:30:38.692+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:30:39.709 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:30:39 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:30:39.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:30:39 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:30:40.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:30:39 smithi115 conmon[90393]: 2022-05-14T01:30:39.708+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:30:40.727 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:30:40 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:30:40.728 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:30:40 smithi115 ceph-mon[78593]: pgmap v4039: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 7 op/s 2022-05-14T01:30:40.728 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:30:40 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 7370 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:30:40.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:30:40 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:30:40.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:30:40 smithi047 ceph-mon[77147]: pgmap v4039: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 7 op/s 2022-05-14T01:30:40.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:30:40 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 7370 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:30:41.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:30:40 smithi115 conmon[90393]: 2022-05-14T01:30:40.726+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:30:41.734 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:30:41 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:30:41.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:30:41 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:30:42.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:30:41 smithi115 conmon[90393]: 2022-05-14T01:30:41.733+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:30:42.698 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:30:42 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:30:42.698 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:30:42 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:30:42.698 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:30:42 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:30:42.699 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:30:42 smithi115 ceph-mon[78593]: pgmap v4040: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 4 op/s 2022-05-14T01:30:42.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:30:42 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:30:42.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:30:42 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:30:42.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:30:42 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:30:42.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:30:42 smithi047 ceph-mon[77147]: pgmap v4040: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 4 op/s 2022-05-14T01:30:43.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:30:42 smithi115 conmon[90393]: 2022-05-14T01:30:42.697+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:30:43.692 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:30:43 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:30:43.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:30:43 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:30:44.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:30:43 smithi115 conmon[90393]: 2022-05-14T01:30:43.691+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:30:44.676 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:30:44 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:30:44.676 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:30:44 smithi115 ceph-mon[78593]: pgmap v4041: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 4 op/s 2022-05-14T01:30:44.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:30:44 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:30:44.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:30:44 smithi047 ceph-mon[77147]: pgmap v4041: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 4 op/s 2022-05-14T01:30:45.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:30:44 smithi115 conmon[90393]: 2022-05-14T01:30:44.676+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:30:45.718 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:30:45 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:30:45.718 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:30:45 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 7375 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:30:45.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:30:45 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:30:45.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:30:45 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 7375 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:30:46.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:30:45 smithi115 conmon[90393]: 2022-05-14T01:30:45.717+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:30:46.762 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:30:46 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:30:46.763 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:30:46 smithi115 ceph-mon[78593]: pgmap v4042: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 7 op/s 2022-05-14T01:30:46.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:30:46 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:30:46.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:30:46 smithi047 ceph-mon[77147]: pgmap v4042: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 7 op/s 2022-05-14T01:30:47.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:30:46 smithi115 conmon[90393]: 2022-05-14T01:30:46.762+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:30:47.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:30:47 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:30:47.812 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:30:47 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:30:48.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:30:47 smithi115 conmon[90393]: 2022-05-14T01:30:47.812+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:30:48.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:30:48 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:30:48.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:30:48 smithi047 ceph-mon[77147]: pgmap v4043: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:30:48.834 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:30:48 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:30:48.835 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:30:48 smithi115 ceph-mon[78593]: pgmap v4043: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:30:49.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:30:48 smithi115 conmon[90393]: 2022-05-14T01:30:48.834+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:30:49.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:30:49 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:30:49.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:30:49 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:30:50.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:30:49 smithi115 conmon[90393]: 2022-05-14T01:30:49.863+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:30:50.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:30:50 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:30:50.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:30:50 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:30:50.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:30:50 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:30:50.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:30:50 smithi047 ceph-mon[77147]: pgmap v4044: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:30:50.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:30:50 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 7380 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:30:50.833 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:30:50 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:30:50.834 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:30:50 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:30:50.834 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:30:50 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:30:50.834 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:30:50 smithi115 ceph-mon[78593]: pgmap v4044: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:30:50.834 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:30:50 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 7380 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:30:51.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:30:50 smithi115 conmon[90393]: 2022-05-14T01:30:50.833+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:30:51.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:30:51 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:30:51.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:30:51 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:30:52.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:30:51 smithi115 conmon[90393]: 2022-05-14T01:30:51.861+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:30:52.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:30:52 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:30:52.918 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:30:52 smithi047 ceph-mon[77147]: pgmap v4045: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:30:52.919 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:30:52 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:30:52.919 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:30:52 smithi115 ceph-mon[78593]: pgmap v4045: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:30:53.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:30:52 smithi115 conmon[90393]: 2022-05-14T01:30:52.874+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:30:53.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:30:53 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:30:53.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:30:53 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:30:53.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:30:53 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:30:53.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:30:53 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:30:54.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:30:53 smithi115 conmon[90393]: 2022-05-14T01:30:53.872+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:30:54.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:30:54 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:30:54.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:30:54 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:30:54.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:30:54 smithi047 ceph-mon[77147]: pgmap v4046: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:30:54.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:30:54 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:30:54.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:30:54 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:30:54.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:30:54 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:30:54.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:30:54 smithi115 ceph-mon[78593]: pgmap v4046: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:30:54.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:30:54 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:30:55.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:30:54 smithi115 conmon[90393]: 2022-05-14T01:30:54.860+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:30:55.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:30:55 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 7385 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:30:55.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:30:55 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:30:55.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:30:55 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 7385 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:30:55.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:30:55 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:30:56.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:30:55 smithi115 conmon[90393]: 2022-05-14T01:30:55.904+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:30:56.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:30:56 smithi047 ceph-mon[77147]: pgmap v4047: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:30:56.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:30:56 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:30:56.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:30:56 smithi115 ceph-mon[78593]: pgmap v4047: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:30:56.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:30:56 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:30:57.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:30:56 smithi115 conmon[90393]: 2022-05-14T01:30:56.878+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:30:57.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:30:57 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:30:57.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:30:57 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:30:58.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:30:57 smithi115 conmon[90393]: 2022-05-14T01:30:57.883+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:30:58.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:30:58 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:30:58.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:30:58 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:30:58.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:30:58 smithi047 ceph-mon[77147]: pgmap v4048: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:30:58.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:30:58 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:30:58.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:30:58 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:30:58.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:30:58 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:30:58.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:30:58 smithi115 ceph-mon[78593]: pgmap v4048: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:30:58.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:30:58 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:30:59.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:30:58 smithi115 conmon[90393]: 2022-05-14T01:30:58.914+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:30:59.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:30:59 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:30:59.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:30:59 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:31:00.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:30:59 smithi115 conmon[90393]: 2022-05-14T01:30:59.912+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:31:00.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:31:00 smithi047 ceph-mon[77147]: pgmap v4049: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:31:00.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:31:00 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 7390 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:31:00.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:31:00 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:31:00.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:31:00 smithi115 ceph-mon[78593]: pgmap v4049: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:31:00.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:31:00 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 7390 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:31:00.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:31:00 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:31:01.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:31:00 smithi115 conmon[90393]: 2022-05-14T01:31:00.900+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:31:01.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:31:01 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:31:01.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:31:01 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:31:02.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:31:01 smithi115 conmon[90393]: 2022-05-14T01:31:01.901+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:31:02.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:31:02 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:31:02.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:31:02 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:31:02.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:31:02 smithi047 ceph-mon[77147]: pgmap v4050: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:31:02.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:31:02 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:31:02.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:31:02 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:31:02.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:31:02 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:31:02.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:31:02 smithi115 ceph-mon[78593]: pgmap v4050: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:31:02.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:31:02 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:31:03.291 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:31:03 smithi047 conmon[78521]: 2022-05-14T01:31:03.023+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:31:03.291 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:31:03 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T01:31:03.292 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:31:03 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:31:03.292 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:31:03 smithi047 conmon[78521]: yield 2022-05-14T01:31:03.292 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:31:03 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:31:03.292 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:31:03 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:31:03.293 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:31:03 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:31:03.293 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:31:03 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T01:31:03.293 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:31:03 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:31:03.293 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:31:03 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T01:31:03.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:31:03 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:31:03.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:31:03 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T01:31:03.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:31:03 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:31:03.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:31:03 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T01:31:03.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:31:03 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:31:03.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:31:03 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T01:31:03.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:31:03 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:31:03.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:31:03 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:31:03.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:31:03 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:31:03.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:31:02 smithi115 conmon[90393]: 2022-05-14T01:31:02.910+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:31:03.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:31:03 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T01:31:03.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:31:03 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T01:31:03.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:31:03 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T01:31:03.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:31:03 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:31:03.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:31:03 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T01:31:03.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:31:03 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T01:31:03.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:31:03 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T01:31:03.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:31:03 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:31:04.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:31:03 smithi115 conmon[90393]: 2022-05-14T01:31:03.863+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:31:04.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:31:04 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:31:04.791 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T01:31:04.791 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:31:04.791 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T01:31:04.792 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:31:04.792 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:31:04.792 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:31:04.792 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T01:31:04.793 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:31:04.793 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T01:31:04.793 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:31:04.793 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T01:31:04.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:31:04.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T01:31:04.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:31:04.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T01:31:04.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:31:04.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:31:04.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:31:04.795 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:31:04 smithi047 ceph-mon[77147]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T01:31:04.795 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:31:04 smithi047 ceph-mon[77147]: pgmap v4051: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:31:04.796 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:31:04 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:31:04.828 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:31:04 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:31:04.828 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T01:31:04.828 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:31:04.828 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T01:31:04.829 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:31:04.829 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:31:04.829 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:31:04.829 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T01:31:04.829 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:31:04.830 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T01:31:04.830 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:31:04.830 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T01:31:04.830 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:31:04.831 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T01:31:04.831 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:31:04.831 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T01:31:04.831 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:31:04.831 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:31:04.832 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:31:04.832 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:31:04 smithi115 ceph-mon[78593]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T01:31:04.832 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:31:04 smithi115 ceph-mon[78593]: pgmap v4051: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:31:04.832 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:31:04 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:31:05.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:31:04 smithi115 conmon[90393]: 2022-05-14T01:31:04.826+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:31:05.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:31:05 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:31:05.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:31:05 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 7395 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:31:05.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:31:05 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:31:05.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:31:05 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 7395 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:31:06.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:31:05 smithi115 conmon[90393]: 2022-05-14T01:31:05.858+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:31:06.480 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:31:06 smithi047 conmon[78521]: 2022-05-14T01:31:06.095+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:31:06.480 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:31:06 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T01:31:06.481 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:31:06 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:31:06.481 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:31:06 smithi047 conmon[78521]: yield 2022-05-14T01:31:06.481 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:31:06 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:31:06.482 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:31:06 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:31:06.482 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:31:06 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:31:06.483 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:31:06 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T01:31:06.483 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:31:06 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:31:06.483 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:31:06 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T01:31:06.484 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:31:06 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:31:06.484 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:31:06 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T01:31:06.484 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:31:06 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:31:06.484 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:31:06 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T01:31:06.485 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:31:06 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:31:06.485 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:31:06 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T01:31:06.485 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:31:06 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:31:06.486 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:31:06 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:31:06.486 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:31:06 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:31:06.486 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:31:06 smithi047 conmon[78521]: 2022-05-14T01:31:06.096+0000 7f5345f84700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T01:31:06.487 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:31:06 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T01:31:06.487 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:31:06 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:31:06.487 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:31:06 smithi047 conmon[78521]: yield 2022-05-14T01:31:06.487 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:31:06 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:31:06.488 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:31:06 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:31:06.488 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:31:06 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:31:06.488 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:31:06 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T01:31:06.488 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:31:06 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:31:06.488 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:31:06 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T01:31:06.489 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:31:06 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:31:06.489 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:31:06 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T01:31:06.489 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:31:06 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:31:06.489 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:31:06 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T01:31:06.490 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:31:06 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:31:06.490 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:31:06 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T01:31:06.490 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:31:06 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:31:06.490 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:31:06 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:31:06.490 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:31:06 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:31:06.491 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:31:06 smithi047 conmon[78521]: 2022-05-14T01:31:06.491 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:31:06 smithi047 conmon[78521]: During handling of the above exception, another exception occurred: 2022-05-14T01:31:06.491 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:31:06 smithi047 conmon[78521]: 2022-05-14T01:31:06.492 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:31:06 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T01:31:06.492 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:31:06 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T01:31:06.492 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:31:06 smithi047 conmon[78521]: return f(*arg) 2022-05-14T01:31:06.493 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:31:06 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T01:31:06.493 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:31:06 smithi047 conmon[78521]: self._write_client_files(client_files, host) 2022-05-14T01:31:06.493 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:31:06 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T01:31:06.493 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:31:06 smithi047 conmon[78521]: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T01:31:06.494 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:31:06 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T01:31:06.494 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:31:06 smithi047 conmon[78521]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T01:31:06.494 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:31:06 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T01:31:06.494 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:31:06 smithi047 conmon[78521]: return self.event_loop.get_result(coro) 2022-05-14T01:31:06.495 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:31:06 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T01:31:06.495 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:31:06 smithi047 conmon[78521]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T01:31:06.495 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:31:06 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T01:31:06.495 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:31:06 smithi047 conmon[78521]: return self.__get_result() 2022-05-14T01:31:06.495 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:31:06 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T01:31:06.496 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:31:06 smithi047 conmon[78521]: raise self._exception 2022-05-14T01:31:06.496 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:31:06 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T01:31:06.496 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:31:06 smithi047 conmon[78521]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T01:31:06.496 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:31:06 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T01:31:06.497 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:31:06 smithi047 conmon[78521]: conn = await self._remote_connection(host, addr) 2022-05-14T01:31:06.497 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:31:06 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T01:31:06.497 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:31:06 smithi047 conmon[78521]: raise 2022-05-14T01:31:06.497 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:31:06 smithi047 conmon[78521]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T01:31:06.498 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:31:06 smithi047 conmon[78521]: self.gen.throw(type, value, traceback) 2022-05-14T01:31:06.498 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:31:06 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T01:31:06.498 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:31:06 smithi047 conmon[78521]: raise OrchestratorError(msg) 2022-05-14T01:31:06.498 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:31:06 smithi047 conmon[78521]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:31:06.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:31:06 smithi047 ceph-mon[77147]: pgmap v4052: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:31:06.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:31:06 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:31:06.838 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:31:06 smithi115 ceph-mon[78593]: pgmap v4052: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:31:06.838 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:31:06 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:31:07.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:31:06 smithi115 conmon[90393]: 2022-05-14T01:31:06.837+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:31:07.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:31:07 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:31:07.791 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T01:31:07.792 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:31:07.792 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T01:31:07.792 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:31:07.792 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:31:07.793 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:31:07.793 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T01:31:07.793 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:31:07.793 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T01:31:07.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:31:07.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T01:31:07.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:31:07.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T01:31:07.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:31:07.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T01:31:07.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:31:07.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:31:07.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:31:07.796 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:31:07 smithi047 ceph-mon[77147]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T01:31:07.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T01:31:07.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:31:07.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T01:31:07.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:31:07.797 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:31:07.797 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:31:07.797 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T01:31:07.797 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:31:07.798 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T01:31:07.798 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:31:07.798 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T01:31:07.798 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:31:07.799 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T01:31:07.799 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:31:07.799 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T01:31:07.799 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:31:07.799 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:31:07.800 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:31:07.800 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T01:31:07.800 INFO:journalctl@ceph.mon.b.smithi047.stdout: During handling of the above exception, another exception occurred: 2022-05-14T01:31:07.800 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T01:31:07.800 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T01:31:07.801 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T01:31:07.801 INFO:journalctl@ceph.mon.b.smithi047.stdout: return f(*arg) 2022-05-14T01:31:07.801 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T01:31:07.801 INFO:journalctl@ceph.mon.b.smithi047.stdout: self._write_client_files(client_files, host) 2022-05-14T01:31:07.802 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T01:31:07.802 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T01:31:07.802 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T01:31:07.802 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T01:31:07.802 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T01:31:07.803 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.event_loop.get_result(coro) 2022-05-14T01:31:07.803 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T01:31:07.803 INFO:journalctl@ceph.mon.b.smithi047.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T01:31:07.803 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T01:31:07.803 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.__get_result() 2022-05-14T01:31:07.804 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T01:31:07.804 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise self._exception 2022-05-14T01:31:07.804 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T01:31:07.804 INFO:journalctl@ceph.mon.b.smithi047.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T01:31:07.804 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T01:31:07.805 INFO:journalctl@ceph.mon.b.smithi047.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T01:31:07.805 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T01:31:07.805 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise 2022-05-14T01:31:07.805 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T01:31:07.806 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.gen.throw(type, value, traceback) 2022-05-14T01:31:07.806 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T01:31:07.806 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OrchestratorError(msg) 2022-05-14T01:31:07.806 INFO:journalctl@ceph.mon.b.smithi047.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:31:07.806 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:31:07 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:31:07.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:31:07 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:31:07.858 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T01:31:07.858 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:31:07.859 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T01:31:07.859 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:31:07.859 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:31:07.859 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:31:07.859 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T01:31:07.860 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:31:07.860 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T01:31:07.860 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:31:07.860 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T01:31:07.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:31:07.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T01:31:07.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:31:07.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T01:31:07.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:31:07.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:31:07.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:31:07.862 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:31:07 smithi115 ceph-mon[78593]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T01:31:07.863 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T01:31:07.863 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:31:07.863 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T01:31:07.863 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:31:07.863 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:31:07.864 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:31:07.864 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T01:31:07.864 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:31:07.864 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T01:31:07.864 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:31:07.865 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T01:31:07.865 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:31:07.865 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T01:31:07.865 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:31:07.866 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T01:31:07.866 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:31:07.866 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:31:07.866 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:31:07.867 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T01:31:07.867 INFO:journalctl@ceph.mon.c.smithi115.stdout: During handling of the above exception, another exception occurred: 2022-05-14T01:31:07.867 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T01:31:07.867 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T01:31:07.867 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T01:31:07.868 INFO:journalctl@ceph.mon.c.smithi115.stdout: return f(*arg) 2022-05-14T01:31:07.868 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T01:31:07.868 INFO:journalctl@ceph.mon.c.smithi115.stdout: self._write_client_files(client_files, host) 2022-05-14T01:31:07.868 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T01:31:07.869 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T01:31:07.869 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T01:31:07.869 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T01:31:07.869 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T01:31:07.869 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.event_loop.get_result(coro) 2022-05-14T01:31:07.870 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T01:31:07.870 INFO:journalctl@ceph.mon.c.smithi115.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T01:31:07.870 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T01:31:07.870 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.__get_result() 2022-05-14T01:31:07.871 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T01:31:07.871 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise self._exception 2022-05-14T01:31:07.871 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T01:31:07.871 INFO:journalctl@ceph.mon.c.smithi115.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T01:31:07.871 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T01:31:07.872 INFO:journalctl@ceph.mon.c.smithi115.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T01:31:07.872 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T01:31:07.872 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise 2022-05-14T01:31:07.872 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T01:31:07.873 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.gen.throw(type, value, traceback) 2022-05-14T01:31:07.873 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T01:31:07.873 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OrchestratorError(msg) 2022-05-14T01:31:07.873 INFO:journalctl@ceph.mon.c.smithi115.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:31:07.874 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:31:07 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:31:08.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:31:07 smithi115 conmon[90393]: 2022-05-14T01:31:07.860+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:31:08.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:31:08 smithi047 ceph-mon[77147]: pgmap v4053: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:31:08.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:31:08 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:31:08.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:31:08 smithi115 ceph-mon[78593]: pgmap v4053: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:31:08.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:31:08 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:31:09.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:31:08 smithi115 conmon[90393]: 2022-05-14T01:31:08.865+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:31:09.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:31:09 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:31:09.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:31:09 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:31:10.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:31:09 smithi115 conmon[90393]: 2022-05-14T01:31:09.860+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:31:10.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:31:10 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:31:10.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:31:10 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:31:10.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:31:10 smithi047 ceph-mon[77147]: pgmap v4054: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:31:10.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:31:10 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 7400 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:31:10.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:31:10 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:31:10.840 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:31:10 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:31:10.840 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:31:10 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:31:10.840 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:31:10 smithi115 ceph-mon[78593]: pgmap v4054: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:31:10.841 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:31:10 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 7400 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:31:10.841 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:31:10 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:31:11.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:31:10 smithi115 conmon[90393]: 2022-05-14T01:31:10.839+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:31:11.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:31:11 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:31:11.815 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:31:11 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:31:12.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:31:11 smithi115 conmon[90393]: 2022-05-14T01:31:11.814+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:31:12.779 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:31:12 smithi115 ceph-mon[78593]: pgmap v4055: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:31:12.780 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:31:12 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:31:12.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:31:12 smithi047 ceph-mon[77147]: pgmap v4055: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:31:12.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:31:12 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:31:13.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:31:12 smithi115 conmon[90393]: 2022-05-14T01:31:12.778+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:31:13.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:31:13 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:31:13.802 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:31:13 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:31:14.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:31:13 smithi115 conmon[90393]: 2022-05-14T01:31:13.801+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:31:14.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:31:14 smithi047 ceph-mon[77147]: pgmap v4056: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:31:14.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:31:14 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:31:14.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:31:14 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:31:14.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:31:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T01:31:14.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:31:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T01:31:14.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:31:14 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:31:14.813 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:31:14 smithi115 ceph-mon[78593]: pgmap v4056: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:31:14.814 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:31:14 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:31:14.814 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:31:14 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:31:14.814 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:31:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T01:31:14.814 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:31:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T01:31:14.814 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:31:14 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:31:15.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:31:14 smithi115 conmon[90393]: 2022-05-14T01:31:14.813+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:31:15.783 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:31:15 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:31:15.783 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:31:15 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 7405 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:31:15.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:31:15 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:31:15.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:31:15 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 7405 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:31:16.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:31:15 smithi115 conmon[90393]: 2022-05-14T01:31:15.782+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:31:16.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:31:16 smithi047 ceph-mon[77147]: pgmap v4057: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:31:16.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:31:16 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:31:16.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:31:16 smithi115 ceph-mon[78593]: pgmap v4057: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:31:16.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:31:16 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:31:17.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:31:16 smithi115 conmon[90393]: 2022-05-14T01:31:16.792+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:31:17.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:31:17 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:31:17.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:31:17 smithi115 conmon[90393]: 2022-05-14T01:31:17.754+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:31:17.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:31:17 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:31:18.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:31:18 smithi047 ceph-mon[77147]: pgmap v4058: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:31:18.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:31:18 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:31:18.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:31:18 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:31:18.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:31:18 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:31:18.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:31:18 smithi115 conmon[90393]: 2022-05-14T01:31:18.712+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:31:18.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:31:18 smithi115 ceph-mon[78593]: pgmap v4058: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:31:18.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:31:18 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:31:18.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:31:18 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:31:18.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:31:18 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:31:19.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:31:19 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:31:19.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:31:19 smithi115 conmon[90393]: 2022-05-14T01:31:19.759+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:31:19.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:31:19 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:31:20.781 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:31:20 smithi115 ceph-mon[78593]: pgmap v4059: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:31:20.782 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:31:20 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:31:20.782 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:31:20 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 7415 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:31:20.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:31:20 smithi047 ceph-mon[77147]: pgmap v4059: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:31:20.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:31:20 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:31:20.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:31:20 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 7415 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:31:21.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:31:20 smithi115 conmon[90393]: 2022-05-14T01:31:20.780+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:31:21.788 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:31:21 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:31:21.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:31:21 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:31:22.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:31:21 smithi115 conmon[90393]: 2022-05-14T01:31:21.787+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:31:22.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:31:22 smithi047 ceph-mon[77147]: pgmap v4060: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:31:22.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:31:22 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:31:22.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:31:22 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:31:22.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:31:22 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:31:22.800 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:31:22 smithi115 ceph-mon[78593]: pgmap v4060: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:31:22.800 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:31:22 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:31:22.801 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:31:22 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:31:22.801 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:31:22 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:31:23.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:31:22 smithi115 conmon[90393]: 2022-05-14T01:31:22.799+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:31:23.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:31:23 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:31:23.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:31:23 smithi115 conmon[90393]: 2022-05-14T01:31:23.782+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:31:23.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:31:23 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:31:24.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:31:24 smithi047 ceph-mon[77147]: pgmap v4061: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:31:24.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:31:24 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:31:24.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:31:24 smithi115 ceph-mon[78593]: pgmap v4061: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:31:24.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:31:24 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:31:24.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:31:24 smithi115 conmon[90393]: 2022-05-14T01:31:24.769+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:31:26.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:31:25 smithi115 conmon[90393]: 2022-05-14T01:31:25.779+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:31:26.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:31:25 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:31:26.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:31:25 smithi115 ceph-mon[78593]: pgmap v4062: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:31:26.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:31:25 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:31:26.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:31:25 smithi047 ceph-mon[77147]: pgmap v4062: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:31:27.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:31:26 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:31:27.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:31:26 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 7420 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:31:27.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:31:26 smithi115 conmon[90393]: 2022-05-14T01:31:26.812+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:31:27.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:31:26 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:31:27.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:31:26 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 7420 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:31:28.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:31:27 smithi115 conmon[90393]: 2022-05-14T01:31:27.794+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:31:28.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:31:27 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:31:28.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:31:27 smithi115 ceph-mon[78593]: pgmap v4063: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:31:28.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:31:27 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:31:28.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:31:27 smithi047 ceph-mon[77147]: pgmap v4063: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:31:29.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:31:28 smithi115 conmon[90393]: 2022-05-14T01:31:28.778+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:31:29.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:31:28 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:31:29.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:31:28 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:31:30.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:31:29 smithi115 conmon[90393]: 2022-05-14T01:31:29.785+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:31:30.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:31:29 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:31:30.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:31:29 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:31:30.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:31:29 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:31:30.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:31:29 smithi115 ceph-mon[78593]: pgmap v4064: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:31:30.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:31:29 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:31:30.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:31:29 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:31:30.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:31:29 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:31:30.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:31:29 smithi047 ceph-mon[77147]: pgmap v4064: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:31:31.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:31:30 smithi115 conmon[90393]: 2022-05-14T01:31:30.786+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:31:31.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:31:30 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:31:31.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:31:30 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:31:32.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:31:31 smithi115 conmon[90393]: 2022-05-14T01:31:31.742+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:31:32.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:31:31 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:31:32.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:31:31 smithi115 ceph-mon[78593]: pgmap v4065: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:31:32.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:31:31 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:31:32.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:31:31 smithi047 ceph-mon[77147]: pgmap v4065: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:31:33.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:31:32 smithi115 conmon[90393]: 2022-05-14T01:31:32.761+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:31:33.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:31:32 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:31:33.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:31:32 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:31:34.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:31:33 smithi115 conmon[90393]: 2022-05-14T01:31:33.740+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:31:34.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:31:33 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:31:34.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:31:33 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:31:34.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:31:33 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:31:34.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:31:33 smithi115 ceph-mon[78593]: pgmap v4066: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:31:34.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:31:33 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:31:34.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:31:33 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:31:34.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:31:33 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:31:34.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:31:33 smithi047 ceph-mon[77147]: pgmap v4066: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:31:35.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:31:34 smithi115 conmon[90393]: 2022-05-14T01:31:34.696+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:31:35.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:31:34 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:31:35.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:31:34 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 7425 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:31:35.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:31:34 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:31:35.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:31:34 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 7425 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:31:36.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:31:35 smithi115 conmon[90393]: 2022-05-14T01:31:35.677+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:31:36.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:31:35 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:31:36.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:31:35 smithi115 ceph-mon[78593]: pgmap v4067: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:31:36.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:31:35 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:31:36.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:31:35 smithi047 ceph-mon[77147]: pgmap v4067: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:31:37.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:31:36 smithi115 conmon[90393]: 2022-05-14T01:31:36.650+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:31:37.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:31:36 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:31:37.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:31:36 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:31:37.907 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:31:37 smithi115 conmon[90393]: 2022-05-14T01:31:37.656+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:31:37.908 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:31:37 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:31:37.908 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:31:37 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:31:38.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:31:37 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:31:38.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:31:37 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:31:38.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:31:37 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:31:38.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:31:37 smithi047 ceph-mon[77147]: pgmap v4068: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:31:38.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:31:37 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:31:38.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:31:37 smithi115 ceph-mon[78593]: pgmap v4068: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:31:39.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:31:38 smithi115 conmon[90393]: 2022-05-14T01:31:38.671+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:31:39.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:31:38 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:31:39.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:31:38 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:31:40.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:31:39 smithi115 conmon[90393]: 2022-05-14T01:31:39.710+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:31:40.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:31:39 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:31:40.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:31:39 smithi115 ceph-mon[78593]: pgmap v4069: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:31:40.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:31:39 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 7430 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:31:40.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:31:39 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:31:40.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:31:39 smithi047 ceph-mon[77147]: pgmap v4069: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:31:40.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:31:39 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 7430 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:31:41.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:31:40 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:31:41.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:31:40 smithi115 conmon[90393]: 2022-05-14T01:31:40.757+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:31:41.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:31:40 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:31:42.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:31:41 smithi115 conmon[90393]: 2022-05-14T01:31:41.709+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:31:42.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:31:41 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:31:42.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:31:41 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:31:42.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:31:41 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:31:42.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:31:41 smithi115 ceph-mon[78593]: pgmap v4070: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:31:42.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:31:41 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:31:42.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:31:41 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:31:42.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:31:41 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:31:42.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:31:41 smithi047 ceph-mon[77147]: pgmap v4070: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:31:42.928 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:31:42 smithi115 conmon[90393]: 2022-05-14T01:31:42.667+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:31:43.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:31:42 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:31:43.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:31:42 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:31:43.932 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:31:43 smithi115 conmon[90393]: 2022-05-14T01:31:43.666+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:31:44.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:31:43 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:31:44.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:31:43 smithi047 ceph-mon[77147]: pgmap v4071: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:31:44.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:31:43 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:31:44.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:31:43 smithi115 ceph-mon[78593]: pgmap v4071: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:31:45.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:31:44 smithi115 conmon[90393]: 2022-05-14T01:31:44.694+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:31:45.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:31:44 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:31:45.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:31:44 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 7435 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:31:45.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:31:44 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:31:45.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:31:44 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 7435 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:31:46.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:31:45 smithi115 conmon[90393]: 2022-05-14T01:31:45.693+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:31:46.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:31:45 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:31:46.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:31:45 smithi115 ceph-mon[78593]: pgmap v4072: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:31:46.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:31:45 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:31:46.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:31:45 smithi047 ceph-mon[77147]: pgmap v4072: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:31:46.941 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:31:46 smithi115 conmon[90393]: 2022-05-14T01:31:46.662+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:31:47.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:31:46 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:31:47.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:31:46 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:31:47.947 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:31:47 smithi115 conmon[90393]: 2022-05-14T01:31:47.620+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:31:48.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:31:47 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:31:48.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:31:47 smithi047 ceph-mon[77147]: pgmap v4073: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:31:48.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:31:47 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:31:48.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:31:47 smithi115 ceph-mon[78593]: pgmap v4073: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:31:48.951 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:31:48 smithi115 conmon[90393]: 2022-05-14T01:31:48.656+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:31:49.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:31:48 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:31:49.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:31:48 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:31:49.955 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:31:49 smithi115 conmon[90393]: 2022-05-14T01:31:49.641+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:31:50.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:31:49 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:31:50.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:31:49 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:31:50.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:31:49 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:31:50.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:31:49 smithi047 ceph-mon[77147]: pgmap v4074: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:31:50.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:31:49 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 7440 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:31:50.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:31:49 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:31:50.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:31:49 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:31:50.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:31:49 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:31:50.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:31:49 smithi115 ceph-mon[78593]: pgmap v4074: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:31:50.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:31:49 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 7440 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:31:50.957 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:31:50 smithi115 conmon[90393]: 2022-05-14T01:31:50.635+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:31:51.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:31:50 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:31:51.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:31:50 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:31:51.961 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:31:51 smithi115 conmon[90393]: 2022-05-14T01:31:51.614+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:31:52.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:31:51 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:31:52.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:31:51 smithi047 ceph-mon[77147]: pgmap v4075: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:31:52.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:31:51 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:31:52.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:31:51 smithi115 ceph-mon[78593]: pgmap v4075: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:31:52.967 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:31:52 smithi115 conmon[90393]: 2022-05-14T01:31:52.613+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:31:53.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:31:52 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:31:53.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:31:52 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:31:53.973 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:31:53 smithi115 conmon[90393]: 2022-05-14T01:31:53.611+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:31:54.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:31:53 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:31:54.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:31:53 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:31:54.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:31:53 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:31:54.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:31:53 smithi047 ceph-mon[77147]: pgmap v4076: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:31:54.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:31:53 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:31:54.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:31:53 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:31:54.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:31:53 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:31:54.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:31:53 smithi115 ceph-mon[78593]: pgmap v4076: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:31:54.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:31:54 smithi115 conmon[90393]: 2022-05-14T01:31:54.574+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:31:55.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:31:54 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:31:55.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:31:54 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 7445 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:31:55.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:31:54 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:31:55.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:31:54 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 7445 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:31:55.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:31:55 smithi115 conmon[90393]: 2022-05-14T01:31:55.592+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:31:56.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:31:55 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:31:56.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:31:55 smithi047 ceph-mon[77147]: pgmap v4077: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:31:56.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:31:55 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:31:56.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:31:55 smithi115 ceph-mon[78593]: pgmap v4077: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:31:56.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:31:56 smithi115 conmon[90393]: 2022-05-14T01:31:56.553+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:31:57.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:31:56 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:31:57.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:31:56 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:31:57.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:31:57 smithi115 conmon[90393]: 2022-05-14T01:31:57.546+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:31:58.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:31:57 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:31:58.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:31:57 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:31:58.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:31:57 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:31:58.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:31:57 smithi047 ceph-mon[77147]: pgmap v4078: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:31:58.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:31:57 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:31:58.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:31:57 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:31:58.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:31:57 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:31:58.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:31:57 smithi115 ceph-mon[78593]: pgmap v4078: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:31:58.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:31:58 smithi115 conmon[90393]: 2022-05-14T01:31:58.560+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:31:59.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:31:58 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:31:59.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:31:58 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:31:59.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:31:59 smithi115 conmon[90393]: 2022-05-14T01:31:59.514+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:32:00.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:31:59 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:32:00.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:31:59 smithi047 ceph-mon[77147]: pgmap v4079: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:32:00.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:31:59 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 7450 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:32:00.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:31:59 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:32:00.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:31:59 smithi115 ceph-mon[78593]: pgmap v4079: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:32:00.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:31:59 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 7450 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:32:00.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:32:00 smithi115 conmon[90393]: 2022-05-14T01:32:00.501+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:32:01.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:32:01 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:32:01.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:32:01 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:32:01.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:32:01 smithi115 conmon[90393]: 2022-05-14T01:32:01.501+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:32:02.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:32:02 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:32:02.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:32:02 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:32:02.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:32:02 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:32:02.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:32:02 smithi047 ceph-mon[77147]: pgmap v4080: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:32:02.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:32:02 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:32:02.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:32:02 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:32:02.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:32:02 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:32:02.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:32:02 smithi115 ceph-mon[78593]: pgmap v4080: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:32:02.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:32:02 smithi115 conmon[90393]: 2022-05-14T01:32:02.517+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:32:03.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:32:03 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:32:03.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:32:03 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:32:03.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:32:03 smithi115 conmon[90393]: 2022-05-14T01:32:03.544+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:32:04.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:32:04 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:32:04.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:32:04 smithi047 ceph-mon[77147]: pgmap v4081: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:32:04.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:32:04 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:32:04.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:32:04 smithi115 ceph-mon[78593]: pgmap v4081: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:32:04.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:32:04 smithi115 conmon[90393]: 2022-05-14T01:32:04.516+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:32:05.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:32:05 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:32:05.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:32:05 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 7455 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:32:05.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:32:05 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:32:05.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:32:05 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 7455 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:32:05.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:32:05 smithi115 conmon[90393]: 2022-05-14T01:32:05.546+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:32:06.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:32:06 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:32:06.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:32:06 smithi047 ceph-mon[77147]: pgmap v4082: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:32:06.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:32:06 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:32:06.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:32:06 smithi115 ceph-mon[78593]: pgmap v4082: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:32:06.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:32:06 smithi115 conmon[90393]: 2022-05-14T01:32:06.551+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:32:07.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:32:07 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:32:07.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:32:07 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T01:32:07.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:32:07 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T01:32:07.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:32:07 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T01:32:07.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:32:07 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:32:07.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:32:07 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T01:32:07.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:32:07 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T01:32:07.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:32:07 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T01:32:07.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:32:07 smithi115 conmon[90393]: 2022-05-14T01:32:07.572+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:32:08.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:32:08 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:32:08.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:32:08 smithi047 ceph-mon[77147]: pgmap v4083: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:32:08.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:32:08 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:32:08.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:32:08 smithi115 ceph-mon[78593]: pgmap v4083: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:32:09.031 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:32:08 smithi115 conmon[90393]: 2022-05-14T01:32:08.618+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:32:09.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:32:09 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:32:09.291 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:32:09 smithi047 conmon[78521]: 2022-05-14T01:32:09.200+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:32:09.291 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:32:09 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T01:32:09.292 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:32:09 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:32:09.292 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:32:09 smithi047 conmon[78521]: yield 2022-05-14T01:32:09.292 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:32:09 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:32:09.292 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:32:09 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:32:09.293 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:32:09 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:32:09.293 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:32:09 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T01:32:09.293 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:32:09 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:32:09.293 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:32:09 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T01:32:09.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:32:09 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:32:09.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:32:09 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T01:32:09.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:32:09 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:32:09.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:32:09 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T01:32:09.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:32:09 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:32:09.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:32:09 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T01:32:09.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:32:09 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:32:09.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:32:09 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:32:09.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:32:09 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:32:09.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:32:09 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:32:09.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:32:09 smithi115 conmon[90393]: 2022-05-14T01:32:09.586+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:32:10.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:32:10 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:32:10.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:32:10 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:32:10.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:32:10 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:32:10.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:32:10 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:32:10.292 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T01:32:10.292 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:32:10.292 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T01:32:10.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:32:10.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:32:10.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:32:10.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T01:32:10.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:32:10.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T01:32:10.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:32:10.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T01:32:10.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:32:10.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T01:32:10.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:32:10.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T01:32:10.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:32:10.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:32:10.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:32:10.296 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:32:10 smithi047 ceph-mon[77147]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T01:32:10.296 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:32:10 smithi047 ceph-mon[77147]: pgmap v4084: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:32:10.297 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:32:10 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 7460 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:32:10.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:32:10 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:32:10.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:32:10 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:32:10.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:32:10 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:32:10.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:32:10 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:32:10.358 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T01:32:10.359 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:32:10.359 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T01:32:10.359 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:32:10.359 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:32:10.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:32:10.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T01:32:10.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:32:10.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T01:32:10.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:32:10.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T01:32:10.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:32:10.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T01:32:10.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:32:10.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T01:32:10.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:32:10.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:32:10.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:32:10.363 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:32:10 smithi115 ceph-mon[78593]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T01:32:10.363 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:32:10 smithi115 ceph-mon[78593]: pgmap v4084: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:32:10.363 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:32:10 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 7460 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:32:11.037 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:32:10 smithi115 conmon[90393]: 2022-05-14T01:32:10.615+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:32:11.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:32:11 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:32:11.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:32:11 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:32:11.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:32:11 smithi115 conmon[90393]: 2022-05-14T01:32:11.591+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:32:12.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:32:12 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:32:12.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:32:12 smithi115 ceph-mon[78593]: pgmap v4085: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:32:12.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:32:12 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:32:12.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:32:12 smithi047 ceph-mon[77147]: pgmap v4085: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:32:12.542 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:32:12 smithi047 conmon[78521]: 2022-05-14T01:32:12.272+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:32:12.542 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:32:12 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T01:32:12.542 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:32:12 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:32:12.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:32:12 smithi047 conmon[78521]: yield 2022-05-14T01:32:12.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:32:12 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:32:12.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:32:12 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:32:12.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:32:12 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:32:12.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:32:12 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T01:32:12.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:32:12 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:32:12.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:32:12 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T01:32:12.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:32:12 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:32:12.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:32:12 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T01:32:12.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:32:12 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:32:12.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:32:12 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T01:32:12.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:32:12 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:32:12.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:32:12 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T01:32:12.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:32:12 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:32:12.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:32:12 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:32:12.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:32:12 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:32:12.547 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:32:12 smithi047 conmon[78521]: 2022-05-14T01:32:12.273+0000 7f5340779700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T01:32:12.547 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:32:12 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T01:32:12.547 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:32:12 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:32:12.547 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:32:12 smithi047 conmon[78521]: yield 2022-05-14T01:32:12.548 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:32:12 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:32:12.548 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:32:12 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:32:12.548 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:32:12 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:32:12.548 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:32:12 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T01:32:12.549 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:32:12 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:32:12.549 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:32:12 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T01:32:12.549 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:32:12 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:32:12.549 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:32:12 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T01:32:12.549 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:32:12 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:32:12.550 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:32:12 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T01:32:12.550 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:32:12 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:32:12.550 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:32:12 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T01:32:12.551 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:32:12 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:32:12.551 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:32:12 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:32:12.551 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:32:12 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:32:12.551 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:32:12 smithi047 conmon[78521]: 2022-05-14T01:32:12.552 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:32:12 smithi047 conmon[78521]: During handling of the above exception, another exception occurred: 2022-05-14T01:32:12.552 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:32:12 smithi047 conmon[78521]: 2022-05-14T01:32:12.552 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:32:12 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T01:32:12.552 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:32:12 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T01:32:12.553 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:32:12 smithi047 conmon[78521]: return f(*arg) 2022-05-14T01:32:12.553 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:32:12 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T01:32:12.553 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:32:12 smithi047 conmon[78521]: self._write_client_files(client_files, host) 2022-05-14T01:32:12.553 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:32:12 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T01:32:12.554 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:32:12 smithi047 conmon[78521]: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T01:32:12.554 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:32:12 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T01:32:12.554 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:32:12 smithi047 conmon[78521]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T01:32:12.554 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:32:12 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T01:32:12.555 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:32:12 smithi047 conmon[78521]: return self.event_loop.get_result(coro) 2022-05-14T01:32:12.555 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:32:12 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T01:32:12.555 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:32:12 smithi047 conmon[78521]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T01:32:12.555 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:32:12 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T01:32:12.556 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:32:12 smithi047 conmon[78521]: return self.__get_result() 2022-05-14T01:32:12.556 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:32:12 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T01:32:12.556 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:32:12 smithi047 conmon[78521]: raise self._exception 2022-05-14T01:32:12.556 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:32:12 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T01:32:12.556 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:32:12 smithi047 conmon[78521]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T01:32:12.557 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:32:12 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T01:32:12.557 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:32:12 smithi047 conmon[78521]: conn = await self._remote_connection(host, addr) 2022-05-14T01:32:12.557 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:32:12 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T01:32:12.558 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:32:12 smithi047 conmon[78521]: raise 2022-05-14T01:32:12.558 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:32:12 smithi047 conmon[78521]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T01:32:12.558 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:32:12 smithi047 conmon[78521]: self.gen.throw(type, value, traceback) 2022-05-14T01:32:12.559 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:32:12 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T01:32:12.559 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:32:12 smithi047 conmon[78521]: raise OrchestratorError(msg) 2022-05-14T01:32:12.559 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:32:12 smithi047 conmon[78521]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:32:13.050 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:32:12 smithi115 conmon[90393]: 2022-05-14T01:32:12.616+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:32:13.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:32:13 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:32:13.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:32:13 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:32:13.358 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T01:32:13.358 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:32:13.358 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T01:32:13.359 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:32:13.359 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:32:13.359 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:32:13.359 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T01:32:13.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:32:13.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T01:32:13.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:32:13.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T01:32:13.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:32:13.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T01:32:13.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:32:13.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T01:32:13.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:32:13.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:32:13.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:32:13.363 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:32:13 smithi115 ceph-mon[78593]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T01:32:13.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T01:32:13.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:32:13.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T01:32:13.365 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:32:13.365 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:32:13.365 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:32:13.365 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T01:32:13.366 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:32:13.366 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T01:32:13.366 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:32:13.366 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T01:32:13.366 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:32:13.367 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T01:32:13.367 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:32:13.367 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T01:32:13.367 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:32:13.367 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:32:13.368 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:32:13.368 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T01:32:13.368 INFO:journalctl@ceph.mon.c.smithi115.stdout: During handling of the above exception, another exception occurred: 2022-05-14T01:32:13.368 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T01:32:13.369 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T01:32:13.369 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T01:32:13.369 INFO:journalctl@ceph.mon.c.smithi115.stdout: return f(*arg) 2022-05-14T01:32:13.369 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T01:32:13.370 INFO:journalctl@ceph.mon.c.smithi115.stdout: self._write_client_files(client_files, host) 2022-05-14T01:32:13.370 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T01:32:13.370 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T01:32:13.370 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T01:32:13.371 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T01:32:13.371 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T01:32:13.371 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.event_loop.get_result(coro) 2022-05-14T01:32:13.371 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T01:32:13.372 INFO:journalctl@ceph.mon.c.smithi115.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T01:32:13.372 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T01:32:13.372 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.__get_result() 2022-05-14T01:32:13.372 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T01:32:13.372 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise self._exception 2022-05-14T01:32:13.373 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T01:32:13.373 INFO:journalctl@ceph.mon.c.smithi115.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T01:32:13.373 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T01:32:13.373 INFO:journalctl@ceph.mon.c.smithi115.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T01:32:13.373 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T01:32:13.374 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise 2022-05-14T01:32:13.374 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T01:32:13.375 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.gen.throw(type, value, traceback) 2022-05-14T01:32:13.375 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T01:32:13.375 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OrchestratorError(msg) 2022-05-14T01:32:13.375 INFO:journalctl@ceph.mon.c.smithi115.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:32:13.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:32:13 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:32:13.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:32:13 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:32:13.541 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T01:32:13.542 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:32:13.542 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T01:32:13.542 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:32:13.542 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:32:13.542 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:32:13.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T01:32:13.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:32:13.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T01:32:13.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:32:13.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T01:32:13.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:32:13.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T01:32:13.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:32:13.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T01:32:13.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:32:13.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:32:13.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:32:13.545 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:32:13 smithi047 ceph-mon[77147]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T01:32:13.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T01:32:13.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:32:13.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T01:32:13.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:32:13.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:32:13.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:32:13.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T01:32:13.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:32:13.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T01:32:13.548 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:32:13.548 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T01:32:13.548 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:32:13.548 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T01:32:13.548 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:32:13.549 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T01:32:13.549 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:32:13.549 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:32:13.549 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:32:13.550 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T01:32:13.550 INFO:journalctl@ceph.mon.b.smithi047.stdout: During handling of the above exception, another exception occurred: 2022-05-14T01:32:13.550 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T01:32:13.550 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T01:32:13.550 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T01:32:13.551 INFO:journalctl@ceph.mon.b.smithi047.stdout: return f(*arg) 2022-05-14T01:32:13.551 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T01:32:13.551 INFO:journalctl@ceph.mon.b.smithi047.stdout: self._write_client_files(client_files, host) 2022-05-14T01:32:13.551 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T01:32:13.552 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T01:32:13.552 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T01:32:13.552 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T01:32:13.552 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T01:32:13.552 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.event_loop.get_result(coro) 2022-05-14T01:32:13.553 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T01:32:13.553 INFO:journalctl@ceph.mon.b.smithi047.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T01:32:13.553 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T01:32:13.554 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.__get_result() 2022-05-14T01:32:13.554 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T01:32:13.554 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise self._exception 2022-05-14T01:32:13.554 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T01:32:13.554 INFO:journalctl@ceph.mon.b.smithi047.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T01:32:13.555 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T01:32:13.555 INFO:journalctl@ceph.mon.b.smithi047.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T01:32:13.555 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T01:32:13.555 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise 2022-05-14T01:32:13.556 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T01:32:13.556 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.gen.throw(type, value, traceback) 2022-05-14T01:32:13.556 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T01:32:13.556 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OrchestratorError(msg) 2022-05-14T01:32:13.557 INFO:journalctl@ceph.mon.b.smithi047.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:32:14.050 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:32:13 smithi115 conmon[90393]: 2022-05-14T01:32:13.620+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:32:14.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:32:14 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:32:14.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:32:14 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:32:14.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:32:14 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:32:14.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:32:14 smithi115 ceph-mon[78593]: pgmap v4086: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:32:14.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:32:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T01:32:14.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:32:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T01:32:14.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:32:14 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:32:14.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:32:14 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:32:14.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:32:14 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:32:14.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:32:14 smithi047 ceph-mon[77147]: pgmap v4086: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:32:14.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:32:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T01:32:14.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:32:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T01:32:15.055 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:32:14 smithi115 conmon[90393]: 2022-05-14T01:32:14.642+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:32:15.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:32:15 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:32:15.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:32:15 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 7465 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:32:15.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:32:15 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:32:15.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:32:15 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 7465 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:32:16.060 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:32:15 smithi115 conmon[90393]: 2022-05-14T01:32:15.643+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:32:16.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:32:16 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:32:16.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:32:16 smithi115 ceph-mon[78593]: pgmap v4087: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:32:16.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:32:16 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:32:16.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:32:16 smithi047 ceph-mon[77147]: pgmap v4087: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:32:16.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:32:16 smithi115 conmon[90393]: 2022-05-14T01:32:16.604+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:32:17.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:32:17 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:32:17.546 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:32:17 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:32:18.070 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:32:17 smithi115 conmon[90393]: 2022-05-14T01:32:17.625+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:32:18.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:32:18 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:32:18.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:32:18 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:32:18.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:32:18 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:32:18.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:32:18 smithi115 ceph-mon[78593]: pgmap v4088: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:32:18.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:32:18 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:32:18.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:32:18 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:32:18.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:32:18 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:32:18.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:32:18 smithi047 ceph-mon[77147]: pgmap v4088: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:32:18.861 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:32:18 smithi115 conmon[90393]: 2022-05-14T01:32:18.581+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:32:19.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:32:19 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:32:19.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:32:19 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:32:20.083 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:32:19 smithi115 conmon[90393]: 2022-05-14T01:32:19.620+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:32:20.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:32:20 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:32:20.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:32:20 smithi115 ceph-mon[78593]: pgmap v4089: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:32:20.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:32:20 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 7470 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:32:20.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:32:20 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:32:20.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:32:20 smithi047 ceph-mon[77147]: pgmap v4089: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:32:20.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:32:20 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 7470 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:32:20.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:32:20 smithi115 conmon[90393]: 2022-05-14T01:32:20.592+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:32:21.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:32:21 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:32:21.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:32:21 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:32:22.087 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:32:21 smithi115 conmon[90393]: 2022-05-14T01:32:21.618+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:32:22.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:32:22 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:32:22.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:32:22 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:32:22.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:32:22 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:32:22.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:32:22 smithi115 ceph-mon[78593]: pgmap v4090: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:32:22.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:32:22 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:32:22.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:32:22 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:32:22.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:32:22 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:32:22.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:32:22 smithi047 ceph-mon[77147]: pgmap v4090: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:32:23.088 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:32:22 smithi115 conmon[90393]: 2022-05-14T01:32:22.663+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:32:23.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:32:23 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:32:23.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:32:23 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:32:24.093 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:32:23 smithi115 conmon[90393]: 2022-05-14T01:32:23.632+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:32:24.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:32:24 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:32:24.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:32:24 smithi115 ceph-mon[78593]: pgmap v4091: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:32:24.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:32:24 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:32:24.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:32:24 smithi047 ceph-mon[77147]: pgmap v4091: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:32:25.095 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:32:24 smithi115 conmon[90393]: 2022-05-14T01:32:24.650+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:32:25.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:32:25 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:32:25.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:32:25 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 7475 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:32:25.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:32:25 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:32:25.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:32:25 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 7475 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:32:26.100 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:32:25 smithi115 conmon[90393]: 2022-05-14T01:32:25.608+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:32:26.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:32:26 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:32:26.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:32:26 smithi115 ceph-mon[78593]: pgmap v4092: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:32:26.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:32:26 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:32:26.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:32:26 smithi047 ceph-mon[77147]: pgmap v4092: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:32:26.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:32:26 smithi115 conmon[90393]: 2022-05-14T01:32:26.563+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:32:27.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:32:27 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:32:27.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:32:27 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:32:27.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:32:27 smithi115 conmon[90393]: 2022-05-14T01:32:27.600+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:32:28.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:32:28 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:32:28.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:32:28 smithi115 ceph-mon[78593]: pgmap v4093: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:32:28.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:32:28 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:32:28.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:32:28 smithi047 ceph-mon[77147]: pgmap v4093: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:32:28.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:32:28 smithi115 conmon[90393]: 2022-05-14T01:32:28.601+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:32:29.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:32:29 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:32:29.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:32:29 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:32:29.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:32:29 smithi115 conmon[90393]: 2022-05-14T01:32:29.581+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:32:30.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:32:30 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:32:30.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:32:30 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:32:30.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:32:30 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:32:30.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:32:30 smithi047 ceph-mon[77147]: pgmap v4094: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:32:30.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:32:30 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 7480 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:32:30.589 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:32:30 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:32:30.589 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:32:30 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:32:30.590 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:32:30 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:32:30.590 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:32:30 smithi115 ceph-mon[78593]: pgmap v4094: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:32:30.590 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:32:30 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 7480 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:32:30.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:32:30 smithi115 conmon[90393]: 2022-05-14T01:32:30.589+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:32:31.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:32:31 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:32:31.551 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:32:31 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:32:31.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:32:31 smithi115 conmon[90393]: 2022-05-14T01:32:31.550+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:32:32.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:32:32 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:32:32.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:32:32 smithi047 ceph-mon[77147]: pgmap v4095: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:32:32.591 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:32:32 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:32:32.592 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:32:32 smithi115 ceph-mon[78593]: pgmap v4095: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:32:32.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:32:32 smithi115 conmon[90393]: 2022-05-14T01:32:32.591+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:32:33.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:32:33 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:32:33.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:32:33 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:32:34.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:32:33 smithi115 conmon[90393]: 2022-05-14T01:32:33.617+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:32:34.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:32:34 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:32:34.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:32:34 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:32:34.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:32:34 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:32:34.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:32:34 smithi047 ceph-mon[77147]: pgmap v4096: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:32:34.582 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:32:34 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:32:34.583 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:32:34 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:32:34.583 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:32:34 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:32:34.583 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:32:34 smithi115 ceph-mon[78593]: pgmap v4096: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:32:34.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:32:34 smithi115 conmon[90393]: 2022-05-14T01:32:34.582+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:32:35.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:32:35 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:32:35.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:32:35 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 7485 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:32:35.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:32:35 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:32:35.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:32:35 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 7485 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:32:36.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:32:35 smithi115 conmon[90393]: 2022-05-14T01:32:35.616+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:32:36.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:32:36 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:32:36.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:32:36 smithi047 ceph-mon[77147]: pgmap v4097: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:32:36.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:32:36 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:32:36.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:32:36 smithi115 ceph-mon[78593]: pgmap v4097: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:32:37.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:32:36 smithi115 conmon[90393]: 2022-05-14T01:32:36.641+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:32:37.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:32:37 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:32:37.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:32:37 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:32:38.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:32:37 smithi115 conmon[90393]: 2022-05-14T01:32:37.633+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:32:38.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:32:38 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:32:38.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:32:38 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:32:38.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:32:38 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:32:38.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:32:38 smithi047 ceph-mon[77147]: pgmap v4098: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:32:38.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:32:38 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:32:38.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:32:38 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:32:38.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:32:38 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:32:38.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:32:38 smithi115 ceph-mon[78593]: pgmap v4098: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:32:39.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:32:38 smithi115 conmon[90393]: 2022-05-14T01:32:38.635+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:32:39.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:32:39 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:32:39.588 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:32:39 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:32:39.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:32:39 smithi115 conmon[90393]: 2022-05-14T01:32:39.587+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:32:40.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:32:40 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:32:40.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:32:40 smithi047 ceph-mon[77147]: pgmap v4099: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:32:40.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:32:40 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 7490 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:32:40.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:32:40 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:32:40.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:32:40 smithi115 ceph-mon[78593]: pgmap v4099: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:32:40.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:32:40 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 7490 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:32:41.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:32:40 smithi115 conmon[90393]: 2022-05-14T01:32:40.613+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:32:41.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:32:41 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:32:41.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:32:41 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:32:42.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:32:41 smithi115 conmon[90393]: 2022-05-14T01:32:41.657+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:32:42.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:32:42 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:32:42.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:32:42 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:32:42.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:32:42 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:32:42.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:32:42 smithi047 ceph-mon[77147]: pgmap v4100: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:32:42.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:32:42 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:32:42.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:32:42 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:32:42.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:32:42 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:32:42.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:32:42 smithi115 ceph-mon[78593]: pgmap v4100: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:32:43.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:32:42 smithi115 conmon[90393]: 2022-05-14T01:32:42.669+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:32:43.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:32:43 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:32:43.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:32:43 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:32:44.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:32:43 smithi115 conmon[90393]: 2022-05-14T01:32:43.682+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:32:44.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:32:44 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:32:44.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:32:44 smithi047 ceph-mon[77147]: pgmap v4101: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:32:44.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:32:44 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:32:44.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:32:44 smithi115 ceph-mon[78593]: pgmap v4101: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:32:45.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:32:44 smithi115 conmon[90393]: 2022-05-14T01:32:44.713+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:32:45.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:32:45 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:32:45.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:32:45 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 7495 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:32:45.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:32:45 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:32:45.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:32:45 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 7495 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:32:46.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:32:45 smithi115 conmon[90393]: 2022-05-14T01:32:45.763+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:32:46.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:32:46 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:32:46.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:32:46 smithi047 ceph-mon[77147]: pgmap v4102: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:32:46.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:32:46 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:32:46.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:32:46 smithi115 ceph-mon[78593]: pgmap v4102: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:32:47.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:32:46 smithi115 conmon[90393]: 2022-05-14T01:32:46.806+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:32:47.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:32:47 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:32:47.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:32:47 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:32:48.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:32:47 smithi115 conmon[90393]: 2022-05-14T01:32:47.838+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:32:48.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:32:48 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:32:48.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:32:48 smithi047 ceph-mon[77147]: pgmap v4103: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:32:48.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:32:48 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:32:48.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:32:48 smithi115 ceph-mon[78593]: pgmap v4103: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:32:49.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:32:48 smithi115 conmon[90393]: 2022-05-14T01:32:48.848+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:32:49.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:32:49 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:32:49.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:32:49 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:32:50.201 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:32:49 smithi115 conmon[90393]: 2022-05-14T01:32:49.884+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:32:50.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:32:50 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:32:50.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:32:50 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:32:50.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:32:50 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:32:50.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:32:50 smithi047 ceph-mon[77147]: pgmap v4104: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:32:50.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:32:50 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 7500 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:32:50.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:32:50 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:32:50.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:32:50 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:32:50.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:32:50 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:32:50.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:32:50 smithi115 ceph-mon[78593]: pgmap v4104: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:32:50.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:32:50 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 7500 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:32:51.206 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:32:50 smithi115 conmon[90393]: 2022-05-14T01:32:50.888+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:32:51.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:32:51 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:32:51.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:32:51 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:32:52.213 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:32:51 smithi115 conmon[90393]: 2022-05-14T01:32:51.908+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:32:52.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:32:52 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:32:52.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:32:52 smithi047 ceph-mon[77147]: pgmap v4105: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:32:52.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:32:52 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:32:52.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:32:52 smithi115 ceph-mon[78593]: pgmap v4105: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:32:53.212 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:32:52 smithi115 conmon[90393]: 2022-05-14T01:32:52.862+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:32:53.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:32:53 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:32:53.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:32:53 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:32:54.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:32:53 smithi115 conmon[90393]: 2022-05-14T01:32:53.815+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:32:54.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:32:54 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:32:54.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:32:54 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:32:54.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:32:54 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:32:54.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:32:54 smithi047 ceph-mon[77147]: pgmap v4106: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:32:54.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:32:54 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:32:54.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:32:54 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:32:54.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:32:54 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:32:54.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:32:54 smithi115 ceph-mon[78593]: pgmap v4106: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:32:55.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:32:54 smithi115 conmon[90393]: 2022-05-14T01:32:54.771+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:32:55.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:32:55 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:32:55.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:32:55 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 7505 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:32:55.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:32:55 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:32:55.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:32:55 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 7505 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:32:56.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:32:55 smithi115 conmon[90393]: 2022-05-14T01:32:55.753+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:32:56.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:32:56 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:32:56.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:32:56 smithi047 ceph-mon[77147]: pgmap v4107: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:32:56.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:32:56 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:32:56.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:32:56 smithi115 ceph-mon[78593]: pgmap v4107: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:32:57.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:32:56 smithi115 conmon[90393]: 2022-05-14T01:32:56.768+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:32:57.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:32:57 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:32:57.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:32:57 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:32:58.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:32:57 smithi115 conmon[90393]: 2022-05-14T01:32:57.817+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:32:58.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:32:58 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:32:58.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:32:58 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:32:58.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:32:58 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:32:58.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:32:58 smithi047 ceph-mon[77147]: pgmap v4108: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:32:58.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:32:58 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:32:58.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:32:58 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:32:58.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:32:58 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:32:58.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:32:58 smithi115 ceph-mon[78593]: pgmap v4108: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:32:59.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:32:58 smithi115 conmon[90393]: 2022-05-14T01:32:58.775+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:32:59.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:32:59 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:32:59.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:32:59 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:33:00.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:32:59 smithi115 conmon[90393]: 2022-05-14T01:32:59.806+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:33:00.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:33:00 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:33:00.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:33:00 smithi047 ceph-mon[77147]: pgmap v4109: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:33:00.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:33:00 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 7510 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:33:00.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:33:00 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:33:00.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:33:00 smithi115 ceph-mon[78593]: pgmap v4109: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:33:00.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:33:00 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 7510 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:33:01.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:33:00 smithi115 conmon[90393]: 2022-05-14T01:33:00.783+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:33:01.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:33:01 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:33:01.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:33:01 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:33:02.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:33:01 smithi115 conmon[90393]: 2022-05-14T01:33:01.830+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:33:02.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:33:02 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:33:02.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:33:02 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:33:02.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:33:02 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:33:02.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:33:02 smithi047 ceph-mon[77147]: pgmap v4110: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:33:02.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:33:02 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:33:02.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:33:02 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:33:02.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:33:02 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:33:02.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:33:02 smithi115 ceph-mon[78593]: pgmap v4110: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:33:03.256 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:33:02 smithi115 conmon[90393]: 2022-05-14T01:33:02.878+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:33:03.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:33:03 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:33:03.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:33:03 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:33:04.261 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:33:03 smithi115 conmon[90393]: 2022-05-14T01:33:03.907+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:33:04.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:33:04 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:33:04.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:33:04 smithi047 ceph-mon[77147]: pgmap v4111: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:33:04.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:33:04 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:33:04.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:33:04 smithi115 ceph-mon[78593]: pgmap v4111: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:33:05.262 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:33:04 smithi115 conmon[90393]: 2022-05-14T01:33:04.956+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:33:05.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:33:05 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:33:05.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:33:05 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 7515 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:33:05.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:33:05 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:33:05.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:33:05 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 7515 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:33:06.263 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:33:05 smithi115 conmon[90393]: 2022-05-14T01:33:05.984+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:33:06.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:33:06 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:33:06.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:33:06 smithi047 ceph-mon[77147]: pgmap v4112: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:33:06.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:33:06 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:33:06.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:33:06 smithi115 ceph-mon[78593]: pgmap v4112: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:33:07.269 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:33:06 smithi115 conmon[90393]: 2022-05-14T01:33:06.960+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:33:07.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:33:07 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:33:07.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:33:07 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:33:08.274 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:33:07 smithi115 conmon[90393]: 2022-05-14T01:33:07.929+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:33:08.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:33:08 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:33:08.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:33:08 smithi047 ceph-mon[77147]: pgmap v4113: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:33:08.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:33:08 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:33:08.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:33:08 smithi115 ceph-mon[78593]: pgmap v4113: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:33:09.279 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:33:08 smithi115 conmon[90393]: 2022-05-14T01:33:08.926+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:33:09.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:33:09 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:33:09.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:33:09 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:33:10.284 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:33:09 smithi115 conmon[90393]: 2022-05-14T01:33:09.965+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:33:10.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:33:10 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:33:10.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:33:10 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:33:10.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:33:10 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:33:10.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:33:10 smithi047 ceph-mon[77147]: pgmap v4114: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:33:10.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:33:10 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 7520 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:33:10.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:33:10 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:33:10.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:33:10 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:33:10.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:33:10 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:33:10.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:33:10 smithi115 ceph-mon[78593]: pgmap v4114: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:33:10.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:33:10 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 7520 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:33:11.286 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:33:10 smithi115 conmon[90393]: 2022-05-14T01:33:10.934+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:33:11.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:33:11 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:33:11.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:33:11 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:33:12.293 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:33:11 smithi115 conmon[90393]: 2022-05-14T01:33:11.941+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:33:12.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:33:12 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:33:12.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:33:12 smithi047 ceph-mon[77147]: pgmap v4115: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:33:12.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:33:12 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T01:33:12.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:33:12 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T01:33:12.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:33:12 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T01:33:12.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:33:12 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:33:12.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:33:12 smithi115 ceph-mon[78593]: pgmap v4115: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:33:12.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:33:12 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T01:33:12.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:33:12 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T01:33:12.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:33:12 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T01:33:13.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:33:12 smithi115 conmon[90393]: 2022-05-14T01:33:12.938+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:33:13.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:33:13 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:33:13.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:33:13 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:33:14.298 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:33:13 smithi115 conmon[90393]: 2022-05-14T01:33:13.937+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:33:14.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:33:14 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:33:14.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:33:14 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:33:14.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:33:14 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:33:14.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:33:14 smithi115 ceph-mon[78593]: pgmap v4116: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:33:14.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:33:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T01:33:14.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:33:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T01:33:14.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:33:14 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:33:14.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:33:14 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:33:14.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:33:14 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:33:14.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:33:14 smithi047 ceph-mon[77147]: pgmap v4116: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:33:14.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:33:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T01:33:14.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:33:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T01:33:15.305 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:33:14 smithi115 conmon[90393]: 2022-05-14T01:33:14.895+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:33:15.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:33:15 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:33:15.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:33:15 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 7525 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:33:15.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:33:15 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:33:15.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:33:15 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 7525 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:33:15.791 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:33:15 smithi047 conmon[78521]: 2022-05-14T01:33:15.375+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:33:15.792 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:33:15 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T01:33:15.792 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:33:15 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:33:15.792 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:33:15 smithi047 conmon[78521]: yield 2022-05-14T01:33:15.792 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:33:15 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:33:15.793 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:33:15 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:33:15.793 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:33:15 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:33:15.793 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:33:15 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T01:33:15.793 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:33:15 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:33:15.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:33:15 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T01:33:15.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:33:15 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:33:15.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:33:15 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T01:33:15.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:33:15 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:33:15.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:33:15 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T01:33:15.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:33:15 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:33:15.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:33:15 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T01:33:15.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:33:15 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:33:15.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:33:15 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:33:15.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:33:15 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:33:16.310 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:33:15 smithi115 conmon[90393]: 2022-05-14T01:33:15.914+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:33:16.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:33:16 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:33:16.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:33:16 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:33:16.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T01:33:16.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:33:16.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T01:33:16.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:33:16.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:33:16.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:33:16.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T01:33:16.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:33:16.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T01:33:16.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:33:16.613 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T01:33:16.613 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:33:16.613 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T01:33:16.613 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:33:16.613 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T01:33:16.614 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:33:16.614 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:33:16.614 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:33:16.614 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:33:16 smithi115 ceph-mon[78593]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T01:33:16.615 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:33:16 smithi115 ceph-mon[78593]: pgmap v4117: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:33:16.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:33:16 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:33:16.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:33:16 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:33:16.792 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T01:33:16.792 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:33:16.793 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T01:33:16.793 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:33:16.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:33:16.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:33:16.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T01:33:16.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:33:16.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T01:33:16.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:33:16.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T01:33:16.797 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:33:16.797 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T01:33:16.798 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:33:16.798 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T01:33:16.798 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:33:16.799 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:33:16.799 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:33:16.800 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:33:16 smithi047 ceph-mon[77147]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T01:33:16.800 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:33:16 smithi047 ceph-mon[77147]: pgmap v4117: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:33:17.311 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:33:16 smithi115 conmon[90393]: 2022-05-14T01:33:16.890+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:33:17.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:33:17 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:33:17.802 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:33:17 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:33:18.322 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:33:17 smithi115 conmon[90393]: 2022-05-14T01:33:17.870+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:33:18.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:33:18 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:33:18.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:33:18 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:33:18.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:33:18 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:33:18.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:33:18 smithi115 ceph-mon[78593]: pgmap v4118: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:33:18.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:33:18 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:33:18.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:33:18 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:33:18.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:33:18 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:33:18.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:33:18 smithi047 ceph-mon[77147]: pgmap v4118: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:33:18.793 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:33:18 smithi047 conmon[78521]: 2022-05-14T01:33:18.447+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:33:18.793 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:33:18 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T01:33:18.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:33:18 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:33:18.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:33:18 smithi047 conmon[78521]: yield 2022-05-14T01:33:18.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:33:18 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:33:18.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:33:18 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:33:18.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:33:18 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:33:18.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:33:18 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T01:33:18.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:33:18 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:33:18.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:33:18 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T01:33:18.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:33:18 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:33:18.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:33:18 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T01:33:18.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:33:18 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:33:18.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:33:18 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T01:33:18.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:33:18 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:33:18.797 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:33:18 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T01:33:18.797 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:33:18 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:33:18.797 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:33:18 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:33:18.797 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:33:18 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:33:18.798 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:33:18 smithi047 conmon[78521]: 2022-05-14T01:33:18.448+0000 7f5344f82700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T01:33:18.798 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:33:18 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T01:33:18.798 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:33:18 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:33:18.798 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:33:18 smithi047 conmon[78521]: yield 2022-05-14T01:33:18.798 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:33:18 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:33:18.799 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:33:18 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:33:18.799 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:33:18 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:33:18.799 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:33:18 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T01:33:18.799 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:33:18 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:33:18.799 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:33:18 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T01:33:18.800 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:33:18 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:33:18.800 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:33:18 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T01:33:18.800 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:33:18 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:33:18.800 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:33:18 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T01:33:18.801 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:33:18 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:33:18.801 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:33:18 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T01:33:18.801 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:33:18 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:33:18.801 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:33:18 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:33:18.801 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:33:18 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:33:18.802 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:33:18 smithi047 conmon[78521]: 2022-05-14T01:33:18.802 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:33:18 smithi047 conmon[78521]: During handling of the above exception, another exception occurred: 2022-05-14T01:33:18.802 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:33:18 smithi047 conmon[78521]: 2022-05-14T01:33:18.802 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:33:18 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T01:33:18.802 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:33:18 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T01:33:18.803 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:33:18 smithi047 conmon[78521]: return f(*arg) 2022-05-14T01:33:18.803 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:33:18 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T01:33:18.803 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:33:18 smithi047 conmon[78521]: self._write_client_files(client_files, host) 2022-05-14T01:33:18.804 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:33:18 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T01:33:18.804 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:33:18 smithi047 conmon[78521]: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T01:33:18.804 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:33:18 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T01:33:18.804 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:33:18 smithi047 conmon[78521]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T01:33:18.804 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:33:18 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T01:33:18.805 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:33:18 smithi047 conmon[78521]: return self.event_loop.get_result(coro) 2022-05-14T01:33:18.805 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:33:18 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T01:33:18.805 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:33:18 smithi047 conmon[78521]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T01:33:18.805 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:33:18 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T01:33:18.805 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:33:18 smithi047 conmon[78521]: return self.__get_result() 2022-05-14T01:33:18.806 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:33:18 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T01:33:18.806 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:33:18 smithi047 conmon[78521]: raise self._exception 2022-05-14T01:33:18.806 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:33:18 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T01:33:18.806 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:33:18 smithi047 conmon[78521]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T01:33:18.807 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:33:18 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T01:33:18.807 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:33:18 smithi047 conmon[78521]: conn = await self._remote_connection(host, addr) 2022-05-14T01:33:18.807 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:33:18 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T01:33:18.807 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:33:18 smithi047 conmon[78521]: raise 2022-05-14T01:33:18.807 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:33:18 smithi047 conmon[78521]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T01:33:18.808 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:33:18 smithi047 conmon[78521]: self.gen.throw(type, value, traceback) 2022-05-14T01:33:18.808 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:33:18 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T01:33:18.808 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:33:18 smithi047 conmon[78521]: raise OrchestratorError(msg) 2022-05-14T01:33:18.808 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:33:18 smithi047 conmon[78521]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:33:19.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:33:18 smithi115 conmon[90393]: 2022-05-14T01:33:18.855+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:33:19.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:33:19 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:33:19.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:33:19 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:33:19.608 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T01:33:19.608 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:33:19.609 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T01:33:19.609 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:33:19.609 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:33:19.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:33:19.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T01:33:19.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:33:19.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T01:33:19.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:33:19.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T01:33:19.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:33:19.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T01:33:19.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:33:19.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T01:33:19.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:33:19.613 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:33:19.613 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:33:19.613 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:33:19 smithi115 ceph-mon[78593]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T01:33:19.613 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T01:33:19.614 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:33:19.614 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T01:33:19.614 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:33:19.614 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:33:19.614 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:33:19.615 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T01:33:19.615 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:33:19.615 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T01:33:19.615 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:33:19.616 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T01:33:19.616 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:33:19.616 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T01:33:19.616 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:33:19.616 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T01:33:19.617 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:33:19.617 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:33:19.617 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:33:19.617 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T01:33:19.618 INFO:journalctl@ceph.mon.c.smithi115.stdout: During handling of the above exception, another exception occurred: 2022-05-14T01:33:19.618 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T01:33:19.618 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T01:33:19.619 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T01:33:19.619 INFO:journalctl@ceph.mon.c.smithi115.stdout: return f(*arg) 2022-05-14T01:33:19.619 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T01:33:19.619 INFO:journalctl@ceph.mon.c.smithi115.stdout: self._write_client_files(client_files, host) 2022-05-14T01:33:19.619 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T01:33:19.620 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T01:33:19.620 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T01:33:19.620 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T01:33:19.620 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T01:33:19.621 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.event_loop.get_result(coro) 2022-05-14T01:33:19.621 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T01:33:19.621 INFO:journalctl@ceph.mon.c.smithi115.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T01:33:19.621 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T01:33:19.622 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.__get_result() 2022-05-14T01:33:19.622 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T01:33:19.622 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise self._exception 2022-05-14T01:33:19.622 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T01:33:19.622 INFO:journalctl@ceph.mon.c.smithi115.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T01:33:19.623 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T01:33:19.623 INFO:journalctl@ceph.mon.c.smithi115.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T01:33:19.623 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T01:33:19.623 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise 2022-05-14T01:33:19.623 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T01:33:19.624 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.gen.throw(type, value, traceback) 2022-05-14T01:33:19.624 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T01:33:19.624 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OrchestratorError(msg) 2022-05-14T01:33:19.624 INFO:journalctl@ceph.mon.c.smithi115.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:33:19.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:33:19 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:33:19.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:33:19 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:33:19.791 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T01:33:19.792 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:33:19.792 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T01:33:19.792 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:33:19.792 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:33:19.792 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:33:19.793 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T01:33:19.793 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:33:19.793 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T01:33:19.793 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:33:19.793 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T01:33:19.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:33:19.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T01:33:19.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:33:19.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T01:33:19.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:33:19.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:33:19.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:33:19.795 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:33:19 smithi047 ceph-mon[77147]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T01:33:19.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T01:33:19.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:33:19.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T01:33:19.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:33:19.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:33:19.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:33:19.797 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T01:33:19.797 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:33:19.797 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T01:33:19.797 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:33:19.798 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T01:33:19.798 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:33:19.798 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T01:33:19.798 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:33:19.798 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T01:33:19.799 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:33:19.799 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:33:19.799 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:33:19.799 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T01:33:19.799 INFO:journalctl@ceph.mon.b.smithi047.stdout: During handling of the above exception, another exception occurred: 2022-05-14T01:33:19.800 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T01:33:19.800 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T01:33:19.800 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T01:33:19.800 INFO:journalctl@ceph.mon.b.smithi047.stdout: return f(*arg) 2022-05-14T01:33:19.800 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T01:33:19.801 INFO:journalctl@ceph.mon.b.smithi047.stdout: self._write_client_files(client_files, host) 2022-05-14T01:33:19.801 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T01:33:19.801 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T01:33:19.801 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T01:33:19.801 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T01:33:19.802 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T01:33:19.802 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.event_loop.get_result(coro) 2022-05-14T01:33:19.802 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T01:33:19.802 INFO:journalctl@ceph.mon.b.smithi047.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T01:33:19.802 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T01:33:19.803 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.__get_result() 2022-05-14T01:33:19.803 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T01:33:19.803 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise self._exception 2022-05-14T01:33:19.803 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T01:33:19.804 INFO:journalctl@ceph.mon.b.smithi047.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T01:33:19.804 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T01:33:19.804 INFO:journalctl@ceph.mon.b.smithi047.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T01:33:19.804 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T01:33:19.804 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise 2022-05-14T01:33:19.805 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T01:33:19.805 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.gen.throw(type, value, traceback) 2022-05-14T01:33:19.805 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T01:33:19.805 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OrchestratorError(msg) 2022-05-14T01:33:19.805 INFO:journalctl@ceph.mon.b.smithi047.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:33:20.331 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:33:19 smithi115 conmon[90393]: 2022-05-14T01:33:19.899+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:33:20.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:33:20 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:33:20.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:33:20 smithi115 ceph-mon[78593]: pgmap v4119: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:33:20.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:33:20 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 7530 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:33:20.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:33:20 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:33:20.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:33:20 smithi047 ceph-mon[77147]: pgmap v4119: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:33:20.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:33:20 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 7530 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:33:21.334 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:33:20 smithi115 conmon[90393]: 2022-05-14T01:33:20.883+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:33:21.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:33:21 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:33:21.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:33:21 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:33:22.339 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:33:21 smithi115 conmon[90393]: 2022-05-14T01:33:21.896+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:33:22.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:33:22 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:33:22.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:33:22 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:33:22.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:33:22 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:33:22.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:33:22 smithi115 ceph-mon[78593]: pgmap v4120: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:33:22.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:33:22 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:33:22.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:33:22 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:33:22.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:33:22 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:33:22.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:33:22 smithi047 ceph-mon[77147]: pgmap v4120: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:33:23.341 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:33:22 smithi115 conmon[90393]: 2022-05-14T01:33:22.917+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:33:23.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:33:23 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:33:23.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:33:23 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:33:24.346 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:33:23 smithi115 conmon[90393]: 2022-05-14T01:33:23.945+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:33:24.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:33:24 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:33:24.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:33:24 smithi115 ceph-mon[78593]: pgmap v4121: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:33:24.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:33:24 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:33:24.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:33:24 smithi047 ceph-mon[77147]: pgmap v4121: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:33:25.347 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:33:24 smithi115 conmon[90393]: 2022-05-14T01:33:24.982+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:33:25.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:33:25 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:33:25.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:33:25 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 7535 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:33:25.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:33:25 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:33:25.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:33:25 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 7535 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:33:26.352 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:33:26 smithi115 conmon[90393]: 2022-05-14T01:33:26.021+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:33:26.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:33:26 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:33:26.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:33:26 smithi115 ceph-mon[78593]: pgmap v4122: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:33:26.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:33:26 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:33:26.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:33:26 smithi047 ceph-mon[77147]: pgmap v4122: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:33:27.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:33:27 smithi115 conmon[90393]: 2022-05-14T01:33:27.017+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:33:27.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:33:27 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:33:27.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:33:27 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:33:28.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:33:28 smithi115 conmon[90393]: 2022-05-14T01:33:28.012+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:33:28.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:33:28 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:33:28.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:33:28 smithi047 ceph-mon[77147]: pgmap v4123: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:33:28.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:33:28 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:33:28.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:33:28 smithi115 ceph-mon[78593]: pgmap v4123: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:33:29.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:33:29 smithi115 conmon[90393]: 2022-05-14T01:33:29.003+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:33:29.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:33:29 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:33:29.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:33:29 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:33:30.360 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:33:30 smithi115 conmon[90393]: 2022-05-14T01:33:30.006+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:33:30.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:33:30 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:33:30.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:33:30 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:33:30.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:33:30 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:33:30.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:33:30 smithi047 ceph-mon[77147]: pgmap v4124: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:33:30.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:33:30 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 7540 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:33:30.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:33:30 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:33:30.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:33:30 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:33:30.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:33:30 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:33:30.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:33:30 smithi115 ceph-mon[78593]: pgmap v4124: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:33:30.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:33:30 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 7540 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:33:31.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:33:30 smithi115 conmon[90393]: 2022-05-14T01:33:30.997+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:33:31.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:33:31 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:33:31.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:33:31 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:33:32.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:33:32 smithi115 conmon[90393]: 2022-05-14T01:33:32.038+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:33:32.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:33:32 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:33:32.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:33:32 smithi047 ceph-mon[77147]: pgmap v4125: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:33:32.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:33:32 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:33:32.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:33:32 smithi115 ceph-mon[78593]: pgmap v4125: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:33:33.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:33:33 smithi115 conmon[90393]: 2022-05-14T01:33:33.048+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:33:33.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:33:33 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:33:33.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:33:33 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:33:34.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:33:34 smithi115 conmon[90393]: 2022-05-14T01:33:34.014+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:33:34.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:33:34 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:33:34.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:33:34 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:33:34.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:33:34 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:33:34.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:33:34 smithi047 ceph-mon[77147]: pgmap v4126: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:33:34.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:33:34 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:33:34.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:33:34 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:33:34.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:33:34 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:33:34.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:33:34 smithi115 ceph-mon[78593]: pgmap v4126: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:33:35.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:33:34 smithi115 conmon[90393]: 2022-05-14T01:33:34.971+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:33:35.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:33:35 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:33:35.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:33:35 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 7545 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:33:35.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:33:35 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:33:35.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:33:35 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 7545 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:33:36.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:33:35 smithi115 conmon[90393]: 2022-05-14T01:33:35.969+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:33:36.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:33:36 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:33:36.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:33:36 smithi047 ceph-mon[77147]: pgmap v4127: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:33:36.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:33:36 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:33:36.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:33:36 smithi115 ceph-mon[78593]: pgmap v4127: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:33:37.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:33:36 smithi115 conmon[90393]: 2022-05-14T01:33:36.969+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:33:37.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:33:37 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:33:37.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:33:37 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:33:38.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:33:37 smithi115 conmon[90393]: 2022-05-14T01:33:37.969+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:33:38.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:33:38 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:33:38.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:33:38 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:33:38.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:33:38 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:33:38.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:33:38 smithi047 ceph-mon[77147]: pgmap v4128: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:33:38.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:33:38 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:33:38.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:33:38 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:33:38.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:33:38 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:33:38.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:33:38 smithi115 ceph-mon[78593]: pgmap v4128: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:33:39.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:33:38 smithi115 conmon[90393]: 2022-05-14T01:33:38.932+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:33:39.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:33:39 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:33:39.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:33:39 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:33:40.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:33:39 smithi115 conmon[90393]: 2022-05-14T01:33:39.978+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:33:40.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:33:40 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:33:40.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:33:40 smithi047 ceph-mon[77147]: pgmap v4129: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:33:40.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:33:40 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 7550 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:33:40.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:33:40 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:33:40.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:33:40 smithi115 ceph-mon[78593]: pgmap v4129: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:33:40.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:33:40 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 7550 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:33:41.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:33:41 smithi115 conmon[90393]: 2022-05-14T01:33:41.013+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:33:41.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:33:41 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:33:41.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:33:41 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:33:42.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:33:42 smithi115 conmon[90393]: 2022-05-14T01:33:42.043+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:33:42.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:33:42 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:33:42.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:33:42 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:33:42.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:33:42 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:33:42.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:33:42 smithi047 ceph-mon[77147]: pgmap v4130: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:33:42.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:33:42 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:33:42.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:33:42 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:33:42.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:33:42 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:33:42.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:33:42 smithi115 ceph-mon[78593]: pgmap v4130: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:33:43.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:33:42 smithi115 conmon[90393]: 2022-05-14T01:33:42.998+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:33:43.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:33:43 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:33:43.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:33:43 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:33:44.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:33:43 smithi115 conmon[90393]: 2022-05-14T01:33:43.988+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:33:44.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:33:44 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:33:44.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:33:44 smithi047 ceph-mon[77147]: pgmap v4131: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:33:44.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:33:44 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:33:44.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:33:44 smithi115 ceph-mon[78593]: pgmap v4131: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:33:45.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:33:44 smithi115 conmon[90393]: 2022-05-14T01:33:44.978+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:33:45.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:33:45 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:33:45.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:33:45 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 7555 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:33:45.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:33:45 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:33:45.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:33:45 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 7555 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:33:46.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:33:45 smithi115 conmon[90393]: 2022-05-14T01:33:45.970+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:33:46.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:33:46 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:33:46.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:33:46 smithi047 ceph-mon[77147]: pgmap v4132: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:33:46.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:33:46 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:33:46.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:33:46 smithi115 ceph-mon[78593]: pgmap v4132: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:33:47.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:33:47 smithi115 conmon[90393]: 2022-05-14T01:33:47.015+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:33:47.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:33:47 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:33:47.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:33:47 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:33:48.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:33:47 smithi115 conmon[90393]: 2022-05-14T01:33:47.981+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:33:48.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:33:48 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:33:48.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:33:48 smithi047 ceph-mon[77147]: pgmap v4133: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:33:48.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:33:48 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:33:48.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:33:48 smithi115 ceph-mon[78593]: pgmap v4133: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:33:49.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:33:49 smithi115 conmon[90393]: 2022-05-14T01:33:49.011+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:33:49.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:33:49 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:33:49.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:33:49 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:33:50.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:33:50 smithi115 conmon[90393]: 2022-05-14T01:33:50.058+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:33:50.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:33:50 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:33:50.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:33:50 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:33:50.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:33:50 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:33:50.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:33:50 smithi047 ceph-mon[77147]: pgmap v4134: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:33:50.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:33:50 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 7560 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:33:50.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:33:50 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:33:50.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:33:50 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:33:50.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:33:50 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:33:50.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:33:50 smithi115 ceph-mon[78593]: pgmap v4134: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:33:50.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:33:50 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 7560 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:33:51.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:33:51 smithi115 conmon[90393]: 2022-05-14T01:33:51.071+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:33:51.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:33:51 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:33:51.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:33:51 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:33:51.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:33:51 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:33:51.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:33:51 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:33:52.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:33:52 smithi115 conmon[90393]: 2022-05-14T01:33:52.101+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:33:52.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:33:52 smithi047 ceph-mon[77147]: pgmap v4135: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:33:52.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:33:52 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:33:52.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:33:52 smithi115 ceph-mon[78593]: pgmap v4135: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:33:52.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:33:52 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:33:53.463 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:33:53 smithi115 conmon[90393]: 2022-05-14T01:33:53.126+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:33:53.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:33:53 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:33:53.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:33:53 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:33:54.472 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:33:54 smithi115 conmon[90393]: 2022-05-14T01:33:54.106+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:33:54.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:33:54 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:33:54.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:33:54 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:33:54.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:33:54 smithi047 ceph-mon[77147]: pgmap v4136: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:33:54.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:33:54 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:33:54.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:33:54 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:33:54.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:33:54 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:33:54.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:33:54 smithi115 ceph-mon[78593]: pgmap v4136: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:33:54.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:33:54 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:33:55.474 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:33:55 smithi115 conmon[90393]: 2022-05-14T01:33:55.123+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:33:55.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:33:55 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 7565 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:33:55.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:33:55 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:33:55.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:33:55 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 7565 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:33:55.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:33:55 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:33:56.478 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:33:56 smithi115 conmon[90393]: 2022-05-14T01:33:56.116+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:33:56.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:33:56 smithi047 ceph-mon[77147]: pgmap v4137: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:33:56.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:33:56 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:33:56.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:33:56 smithi115 ceph-mon[78593]: pgmap v4137: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:33:56.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:33:56 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:33:57.485 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:33:57 smithi115 conmon[90393]: 2022-05-14T01:33:57.113+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:33:57.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:33:57 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:33:57.868 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:33:57 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:33:58.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:33:58 smithi115 conmon[90393]: 2022-05-14T01:33:58.097+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:33:58.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:33:58 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:33:58.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:33:58 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:33:58.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:33:58 smithi047 ceph-mon[77147]: pgmap v4138: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:33:58.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:33:58 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:33:58.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:33:58 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:33:58.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:33:58 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:33:58.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:33:58 smithi115 ceph-mon[78593]: pgmap v4138: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:33:58.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:33:58 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:33:59.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:33:59 smithi115 conmon[90393]: 2022-05-14T01:33:59.090+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:33:59.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:33:59 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:33:59.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:33:59 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:34:00.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:34:00 smithi115 conmon[90393]: 2022-05-14T01:34:00.050+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:34:00.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:34:00 smithi047 ceph-mon[77147]: pgmap v4139: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:34:00.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:34:00 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 7571 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:34:00.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:34:00 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:34:00.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:34:00 smithi115 ceph-mon[78593]: pgmap v4139: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:34:00.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:34:00 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 7571 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:34:00.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:34:00 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:34:01.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:34:01 smithi115 conmon[90393]: 2022-05-14T01:34:01.066+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:34:01.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:34:01 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:34:01.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:34:01 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:34:02.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:34:02 smithi115 conmon[90393]: 2022-05-14T01:34:02.096+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:34:02.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:34:02 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:34:02.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:34:02 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:34:02.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:34:02 smithi047 ceph-mon[77147]: pgmap v4140: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:34:02.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:34:02 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:34:02.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:34:02 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:34:02.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:34:02 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:34:02.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:34:02 smithi115 ceph-mon[78593]: pgmap v4140: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:34:02.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:34:02 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:34:03.511 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:34:03 smithi115 conmon[90393]: 2022-05-14T01:34:03.131+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:34:03.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:34:03 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:34:03.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:34:03 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:34:04.517 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:34:04 smithi115 conmon[90393]: 2022-05-14T01:34:04.164+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:34:04.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:34:04 smithi047 ceph-mon[77147]: pgmap v4141: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:34:04.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:34:04 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:34:04.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:34:04 smithi115 ceph-mon[78593]: pgmap v4141: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:34:04.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:34:04 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:34:05.517 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:34:05 smithi115 conmon[90393]: 2022-05-14T01:34:05.190+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:34:05.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:34:05 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 7576 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:34:05.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:34:05 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:34:05.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:34:05 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 7576 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:34:05.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:34:05 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:34:06.523 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:34:06 smithi115 conmon[90393]: 2022-05-14T01:34:06.160+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:34:06.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:34:06 smithi047 ceph-mon[77147]: pgmap v4142: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:34:06.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:34:06 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:34:06.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:34:06 smithi115 ceph-mon[78593]: pgmap v4142: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:34:06.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:34:06 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:34:07.524 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:34:07 smithi115 conmon[90393]: 2022-05-14T01:34:07.165+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:34:07.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:34:07 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:34:07.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:34:07 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:34:08.529 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:34:08 smithi115 conmon[90393]: 2022-05-14T01:34:08.121+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:34:08.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:34:08 smithi047 ceph-mon[77147]: pgmap v4143: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:34:08.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:34:08 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:34:08.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:34:08 smithi115 ceph-mon[78593]: pgmap v4143: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:34:08.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:34:08 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:34:09.537 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:34:09 smithi115 conmon[90393]: 2022-05-14T01:34:09.145+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:34:09.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:34:09 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:34:09.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:34:09 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:34:10.542 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:34:10 smithi115 conmon[90393]: 2022-05-14T01:34:10.141+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:34:10.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:34:10 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:34:10.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:34:10 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:34:10.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:34:10 smithi047 ceph-mon[77147]: pgmap v4144: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:34:10.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:34:10 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 7580 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:34:10.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:34:10 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:34:10.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:34:10 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:34:10.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:34:10 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:34:10.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:34:10 smithi115 ceph-mon[78593]: pgmap v4144: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:34:10.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:34:10 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 7580 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:34:10.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:34:10 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:34:11.542 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:34:11 smithi115 conmon[90393]: 2022-05-14T01:34:11.113+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:34:11.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:34:11 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:34:12.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:34:11 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:34:12.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:34:12 smithi115 conmon[90393]: 2022-05-14T01:34:12.105+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:34:12.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:34:12 smithi115 ceph-mon[78593]: pgmap v4145: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:34:12.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:34:12 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:34:13.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:34:12 smithi047 ceph-mon[77147]: pgmap v4145: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:34:13.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:34:12 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:34:13.554 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:34:13 smithi115 conmon[90393]: 2022-05-14T01:34:13.117+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:34:13.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:34:13 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:34:14.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:34:13 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:34:14.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:34:14 smithi115 conmon[90393]: 2022-05-14T01:34:14.071+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:34:14.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:34:14 smithi115 ceph-mon[78593]: pgmap v4146: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:34:14.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:34:14 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:34:14.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:34:14 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:34:14.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:34:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T01:34:14.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:34:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T01:34:14.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:34:14 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:34:15.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:34:14 smithi047 ceph-mon[77147]: pgmap v4146: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:34:15.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:34:14 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:34:15.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:34:14 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:34:15.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:34:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T01:34:15.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:34:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T01:34:15.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:34:14 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:34:15.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:34:15 smithi115 conmon[90393]: 2022-05-14T01:34:15.106+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:34:15.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:34:15 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 7585 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:34:15.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:34:15 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:34:16.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:34:15 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 7585 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:34:16.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:34:15 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:34:16.559 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:34:16 smithi115 conmon[90393]: 2022-05-14T01:34:16.154+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:34:16.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:34:16 smithi115 ceph-mon[78593]: pgmap v4147: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:34:16.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:34:16 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:34:17.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:34:16 smithi047 ceph-mon[77147]: pgmap v4147: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:34:17.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:34:16 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:34:17.564 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:34:17 smithi115 conmon[90393]: 2022-05-14T01:34:17.176+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:34:17.565 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:34:17 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:34:18.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:34:17 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:34:18.495 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:34:18 smithi115 conmon[90393]: 2022-05-14T01:34:18.139+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:34:18.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:34:18 smithi047 ceph-mon[77147]: pgmap v4148: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:34:18.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:34:18 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:34:18.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:34:18 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:34:18.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:34:18 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:34:18.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:34:18 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T01:34:18.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:34:18 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T01:34:18.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:34:18 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T01:34:18.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:34:18 smithi115 ceph-mon[78593]: pgmap v4148: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:34:18.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:34:18 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:34:18.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:34:18 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:34:18.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:34:18 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:34:18.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:34:18 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T01:34:18.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:34:18 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T01:34:18.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:34:18 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T01:34:19.291 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:34:19 smithi047 conmon[78521]: 2022-05-14T01:34:19.055+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:34:19.291 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:34:19 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T01:34:19.291 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:34:19 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:34:19.292 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:34:19 smithi047 conmon[78521]: yield 2022-05-14T01:34:19.292 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:34:19 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:34:19.292 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:34:19 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:34:19.292 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:34:19 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:34:19.293 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:34:19 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T01:34:19.293 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:34:19 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:34:19.293 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:34:19 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T01:34:19.293 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:34:19 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:34:19.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:34:19 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T01:34:19.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:34:19 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:34:19.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:34:19 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T01:34:19.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:34:19 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:34:19.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:34:19 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T01:34:19.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:34:19 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:34:19.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:34:19 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:34:19.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:34:19 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:34:19.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:34:19 smithi115 conmon[90393]: 2022-05-14T01:34:19.163+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:34:19.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:34:19 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:34:20.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:34:19 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:34:20.570 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:34:20 smithi115 conmon[90393]: 2022-05-14T01:34:20.171+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:34:20.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:34:20 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:34:20.859 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T01:34:20.859 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:34:20.859 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T01:34:20.859 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:34:20.860 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:34:20.860 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:34:20.860 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T01:34:20.860 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:34:20.860 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T01:34:20.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:34:20.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T01:34:20.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:34:20.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T01:34:20.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:34:20.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T01:34:20.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:34:20.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:34:20.863 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:34:20.863 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:34:20 smithi115 ceph-mon[78593]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T01:34:20.863 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:34:20 smithi115 ceph-mon[78593]: pgmap v4149: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:34:20.863 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:34:20 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 7595 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:34:20.863 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:34:20 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:34:21.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:34:20 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:34:21.041 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T01:34:21.041 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:34:21.042 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T01:34:21.042 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:34:21.042 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:34:21.042 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:34:21.042 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T01:34:21.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:34:21.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T01:34:21.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:34:21.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T01:34:21.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:34:21.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T01:34:21.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:34:21.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T01:34:21.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:34:21.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:34:21.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:34:21.045 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:34:20 smithi047 ceph-mon[77147]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T01:34:21.045 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:34:20 smithi047 ceph-mon[77147]: pgmap v4149: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:34:21.046 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:34:20 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 7595 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:34:21.046 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:34:20 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:34:21.576 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:34:21 smithi115 conmon[90393]: 2022-05-14T01:34:21.157+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:34:21.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:34:21 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:34:22.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:34:21 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:34:22.541 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:34:22 smithi047 conmon[78521]: 2022-05-14T01:34:22.128+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:34:22.542 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:34:22 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T01:34:22.542 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:34:22 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:34:22.542 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:34:22 smithi047 conmon[78521]: yield 2022-05-14T01:34:22.542 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:34:22 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:34:22.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:34:22 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:34:22.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:34:22 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:34:22.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:34:22 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T01:34:22.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:34:22 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:34:22.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:34:22 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T01:34:22.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:34:22 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:34:22.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:34:22 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T01:34:22.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:34:22 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:34:22.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:34:22 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T01:34:22.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:34:22 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:34:22.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:34:22 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T01:34:22.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:34:22 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:34:22.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:34:22 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:34:22.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:34:22 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:34:22.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:34:22 smithi047 conmon[78521]: 2022-05-14T01:34:22.129+0000 7f5344781700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T01:34:22.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:34:22 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T01:34:22.547 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:34:22 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:34:22.547 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:34:22 smithi047 conmon[78521]: yield 2022-05-14T01:34:22.547 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:34:22 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:34:22.547 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:34:22 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:34:22.548 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:34:22 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:34:22.548 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:34:22 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T01:34:22.548 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:34:22 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:34:22.549 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:34:22 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T01:34:22.549 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:34:22 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:34:22.549 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:34:22 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T01:34:22.549 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:34:22 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:34:22.549 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:34:22 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T01:34:22.550 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:34:22 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:34:22.550 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:34:22 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T01:34:22.550 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:34:22 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:34:22.550 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:34:22 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:34:22.551 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:34:22 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:34:22.551 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:34:22 smithi047 conmon[78521]: 2022-05-14T01:34:22.551 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:34:22 smithi047 conmon[78521]: During handling of the above exception, another exception occurred: 2022-05-14T01:34:22.551 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:34:22 smithi047 conmon[78521]: 2022-05-14T01:34:22.551 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:34:22 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T01:34:22.552 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:34:22 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T01:34:22.552 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:34:22 smithi047 conmon[78521]: return f(*arg) 2022-05-14T01:34:22.552 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:34:22 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T01:34:22.553 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:34:22 smithi047 conmon[78521]: self._write_client_files(client_files, host) 2022-05-14T01:34:22.553 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:34:22 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T01:34:22.553 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:34:22 smithi047 conmon[78521]: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T01:34:22.553 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:34:22 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T01:34:22.554 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:34:22 smithi047 conmon[78521]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T01:34:22.554 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:34:22 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T01:34:22.554 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:34:22 smithi047 conmon[78521]: return self.event_loop.get_result(coro) 2022-05-14T01:34:22.554 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:34:22 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T01:34:22.555 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:34:22 smithi047 conmon[78521]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T01:34:22.555 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:34:22 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T01:34:22.555 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:34:22 smithi047 conmon[78521]: return self.__get_result() 2022-05-14T01:34:22.555 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:34:22 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T01:34:22.555 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:34:22 smithi047 conmon[78521]: raise self._exception 2022-05-14T01:34:22.556 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:34:22 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T01:34:22.556 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:34:22 smithi047 conmon[78521]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T01:34:22.556 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:34:22 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T01:34:22.556 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:34:22 smithi047 conmon[78521]: conn = await self._remote_connection(host, addr) 2022-05-14T01:34:22.556 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:34:22 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T01:34:22.557 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:34:22 smithi047 conmon[78521]: raise 2022-05-14T01:34:22.557 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:34:22 smithi047 conmon[78521]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T01:34:22.558 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:34:22 smithi047 conmon[78521]: self.gen.throw(type, value, traceback) 2022-05-14T01:34:22.558 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:34:22 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T01:34:22.558 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:34:22 smithi047 conmon[78521]: raise OrchestratorError(msg) 2022-05-14T01:34:22.558 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:34:22 smithi047 conmon[78521]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:34:22.577 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:34:22 smithi115 conmon[90393]: 2022-05-14T01:34:22.107+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:34:22.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:34:22 smithi115 ceph-mon[78593]: pgmap v4150: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:34:22.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:34:22 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:34:22.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:34:22 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:34:22.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:34:22 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:34:23.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:34:22 smithi047 ceph-mon[77147]: pgmap v4150: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:34:23.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:34:22 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:34:23.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:34:22 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:34:23.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:34:22 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:34:23.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:34:23 smithi115 conmon[90393]: 2022-05-14T01:34:23.099+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:34:23.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:34:23 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:34:23.859 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T01:34:23.859 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:34:23.859 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T01:34:23.859 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:34:23.860 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:34:23.860 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:34:23.860 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T01:34:23.860 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:34:23.860 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T01:34:23.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:34:23.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T01:34:23.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:34:23.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T01:34:23.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:34:23.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T01:34:23.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:34:23.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:34:23.863 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:34:23.863 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:34:23 smithi115 ceph-mon[78593]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T01:34:23.863 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T01:34:23.863 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:34:23.863 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T01:34:23.864 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:34:23.864 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:34:23.864 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:34:23.864 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T01:34:23.865 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:34:23.865 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T01:34:23.865 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:34:23.865 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T01:34:23.865 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:34:23.866 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T01:34:23.866 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:34:23.866 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T01:34:23.866 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:34:23.867 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:34:23.867 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:34:23.867 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T01:34:23.867 INFO:journalctl@ceph.mon.c.smithi115.stdout: During handling of the above exception, another exception occurred: 2022-05-14T01:34:23.868 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T01:34:23.868 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T01:34:23.868 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T01:34:23.869 INFO:journalctl@ceph.mon.c.smithi115.stdout: return f(*arg) 2022-05-14T01:34:23.869 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T01:34:23.869 INFO:journalctl@ceph.mon.c.smithi115.stdout: self._write_client_files(client_files, host) 2022-05-14T01:34:23.870 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T01:34:23.870 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T01:34:23.870 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T01:34:23.870 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T01:34:23.871 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T01:34:23.871 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.event_loop.get_result(coro) 2022-05-14T01:34:23.871 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T01:34:23.871 INFO:journalctl@ceph.mon.c.smithi115.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T01:34:23.871 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T01:34:23.872 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.__get_result() 2022-05-14T01:34:23.872 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T01:34:23.872 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise self._exception 2022-05-14T01:34:23.872 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T01:34:23.872 INFO:journalctl@ceph.mon.c.smithi115.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T01:34:23.873 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T01:34:23.873 INFO:journalctl@ceph.mon.c.smithi115.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T01:34:23.873 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T01:34:23.873 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise 2022-05-14T01:34:23.874 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T01:34:23.874 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.gen.throw(type, value, traceback) 2022-05-14T01:34:23.874 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T01:34:23.874 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OrchestratorError(msg) 2022-05-14T01:34:23.874 INFO:journalctl@ceph.mon.c.smithi115.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:34:23.875 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:34:23 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:34:24.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:34:23 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:34:24.041 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T01:34:24.041 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:34:24.042 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T01:34:24.042 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:34:24.042 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:34:24.042 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:34:24.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T01:34:24.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:34:24.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T01:34:24.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:34:24.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T01:34:24.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:34:24.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T01:34:24.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:34:24.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T01:34:24.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:34:24.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:34:24.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:34:24.046 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:34:23 smithi047 ceph-mon[77147]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T01:34:24.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T01:34:24.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:34:24.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T01:34:24.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:34:24.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:34:24.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:34:24.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T01:34:24.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:34:24.049 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T01:34:24.049 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:34:24.049 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T01:34:24.049 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:34:24.049 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T01:34:24.050 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:34:24.050 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T01:34:24.051 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:34:24.051 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:34:24.051 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:34:24.051 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T01:34:24.052 INFO:journalctl@ceph.mon.b.smithi047.stdout: During handling of the above exception, another exception occurred: 2022-05-14T01:34:24.052 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T01:34:24.052 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T01:34:24.052 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T01:34:24.053 INFO:journalctl@ceph.mon.b.smithi047.stdout: return f(*arg) 2022-05-14T01:34:24.053 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T01:34:24.053 INFO:journalctl@ceph.mon.b.smithi047.stdout: self._write_client_files(client_files, host) 2022-05-14T01:34:24.053 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T01:34:24.054 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T01:34:24.054 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T01:34:24.054 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T01:34:24.054 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T01:34:24.055 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.event_loop.get_result(coro) 2022-05-14T01:34:24.055 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T01:34:24.055 INFO:journalctl@ceph.mon.b.smithi047.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T01:34:24.055 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T01:34:24.056 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.__get_result() 2022-05-14T01:34:24.056 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T01:34:24.056 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise self._exception 2022-05-14T01:34:24.057 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T01:34:24.057 INFO:journalctl@ceph.mon.b.smithi047.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T01:34:24.057 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T01:34:24.057 INFO:journalctl@ceph.mon.b.smithi047.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T01:34:24.058 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T01:34:24.058 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise 2022-05-14T01:34:24.058 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T01:34:24.058 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.gen.throw(type, value, traceback) 2022-05-14T01:34:24.059 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T01:34:24.059 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OrchestratorError(msg) 2022-05-14T01:34:24.059 INFO:journalctl@ceph.mon.b.smithi047.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:34:24.060 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:34:23 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:34:24.582 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:34:24 smithi115 conmon[90393]: 2022-05-14T01:34:24.122+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:34:24.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:34:24 smithi115 ceph-mon[78593]: pgmap v4151: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:34:24.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:34:24 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:34:25.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:34:24 smithi047 ceph-mon[77147]: pgmap v4151: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:34:25.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:34:24 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:34:25.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:34:25 smithi115 conmon[90393]: 2022-05-14T01:34:25.116+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:34:26.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:34:25 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:34:26.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:34:25 smithi047 ceph-mon[77147]: pgmap v4152: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:34:26.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:34:26 smithi115 conmon[90393]: 2022-05-14T01:34:26.069+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:34:26.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:34:25 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:34:26.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:34:25 smithi115 ceph-mon[78593]: pgmap v4152: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:34:27.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:34:26 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 7601 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:34:27.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:34:26 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:34:27.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:34:27 smithi115 conmon[90393]: 2022-05-14T01:34:27.065+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:34:27.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:34:26 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 7601 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:34:27.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:34:26 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:34:28.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:34:27 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:34:28.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:34:27 smithi047 ceph-mon[77147]: pgmap v4153: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:34:28.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:34:28 smithi115 conmon[90393]: 2022-05-14T01:34:28.029+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:34:28.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:34:27 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:34:28.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:34:27 smithi115 ceph-mon[78593]: pgmap v4153: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:34:29.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:34:28 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:34:29.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:34:28 smithi115 conmon[90393]: 2022-05-14T01:34:28.986+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:34:29.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:34:28 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:34:30.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:34:29 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:34:30.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:34:29 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:34:30.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:34:29 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:34:30.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:34:29 smithi047 ceph-mon[77147]: pgmap v4154: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:34:30.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:34:29 smithi115 conmon[90393]: 2022-05-14T01:34:29.969+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:34:30.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:34:29 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:34:30.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:34:29 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:34:30.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:34:29 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:34:30.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:34:29 smithi115 ceph-mon[78593]: pgmap v4154: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:34:31.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:34:30 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:34:31.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:34:30 smithi115 conmon[90393]: 2022-05-14T01:34:30.975+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:34:31.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:34:30 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:34:32.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:34:31 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:34:32.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:34:31 smithi047 ceph-mon[77147]: pgmap v4155: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:34:32.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:34:32 smithi115 conmon[90393]: 2022-05-14T01:34:32.017+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:34:32.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:34:31 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:34:32.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:34:31 smithi115 ceph-mon[78593]: pgmap v4155: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:34:33.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:34:32 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:34:33.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:34:32 smithi115 conmon[90393]: 2022-05-14T01:34:32.973+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:34:33.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:34:32 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:34:34.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:34:33 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:34:34.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:34:33 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:34:34.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:34:33 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:34:34.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:34:33 smithi047 ceph-mon[77147]: pgmap v4156: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:34:34.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:34:33 smithi115 conmon[90393]: 2022-05-14T01:34:33.950+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:34:34.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:34:33 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:34:34.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:34:33 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:34:34.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:34:33 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:34:34.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:34:33 smithi115 ceph-mon[78593]: pgmap v4156: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:34:35.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:34:34 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:34:35.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:34:34 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 7605 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:34:35.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:34:34 smithi115 conmon[90393]: 2022-05-14T01:34:34.959+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:34:35.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:34:34 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:34:35.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:34:34 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 7605 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:34:36.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:34:35 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:34:36.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:34:35 smithi047 ceph-mon[77147]: pgmap v4157: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:34:36.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:34:35 smithi115 conmon[90393]: 2022-05-14T01:34:35.936+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:34:36.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:34:35 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:34:36.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:34:35 smithi115 ceph-mon[78593]: pgmap v4157: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:34:37.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:34:36 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:34:37.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:34:36 smithi115 conmon[90393]: 2022-05-14T01:34:36.890+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:34:37.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:34:36 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:34:38.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:34:37 smithi115 conmon[90393]: 2022-05-14T01:34:37.847+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:34:38.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:34:37 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:34:38.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:34:37 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:34:38.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:34:37 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:34:38.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:34:37 smithi115 ceph-mon[78593]: pgmap v4158: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:34:38.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:34:37 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:34:38.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:34:37 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:34:38.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:34:37 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:34:38.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:34:37 smithi047 ceph-mon[77147]: pgmap v4158: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:34:39.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:34:38 smithi115 conmon[90393]: 2022-05-14T01:34:38.816+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:34:39.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:34:38 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:34:39.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:34:38 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:34:40.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:34:39 smithi115 conmon[90393]: 2022-05-14T01:34:39.818+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:34:40.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:34:39 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:34:40.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:34:39 smithi115 ceph-mon[78593]: pgmap v4159: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:34:40.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:34:39 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 7610 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:34:40.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:34:39 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:34:40.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:34:39 smithi047 ceph-mon[77147]: pgmap v4159: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:34:40.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:34:39 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 7610 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:34:41.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:34:40 smithi115 conmon[90393]: 2022-05-14T01:34:40.794+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:34:41.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:34:40 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:34:41.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:34:40 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:34:42.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:34:41 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:34:42.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:34:41 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:34:42.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:34:41 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:34:42.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:34:41 smithi115 ceph-mon[78593]: pgmap v4160: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:34:42.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:34:41 smithi115 conmon[90393]: 2022-05-14T01:34:41.776+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:34:42.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:34:41 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:34:42.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:34:41 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:34:42.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:34:41 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:34:42.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:34:41 smithi047 ceph-mon[77147]: pgmap v4160: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:34:43.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:34:42 smithi115 conmon[90393]: 2022-05-14T01:34:42.820+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:34:43.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:34:42 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:34:43.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:34:42 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:34:44.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:34:43 smithi115 conmon[90393]: 2022-05-14T01:34:43.825+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:34:44.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:34:43 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:34:44.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:34:43 smithi115 ceph-mon[78593]: pgmap v4161: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:34:44.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:34:43 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:34:44.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:34:43 smithi047 ceph-mon[77147]: pgmap v4161: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:34:45.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:34:44 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:34:45.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:34:44 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 7615 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:34:45.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:34:44 smithi115 conmon[90393]: 2022-05-14T01:34:44.864+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:34:45.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:34:44 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:34:45.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:34:44 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 7615 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:34:46.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:34:45 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:34:46.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:34:45 smithi047 ceph-mon[77147]: pgmap v4162: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:34:46.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:34:45 smithi115 conmon[90393]: 2022-05-14T01:34:45.871+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:34:46.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:34:45 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:34:46.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:34:45 smithi115 ceph-mon[78593]: pgmap v4162: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:34:47.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:34:46 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:34:47.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:34:46 smithi115 conmon[90393]: 2022-05-14T01:34:46.888+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:34:47.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:34:46 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:34:48.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:34:47 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:34:48.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:34:47 smithi047 ceph-mon[77147]: pgmap v4163: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:34:48.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:34:47 smithi115 conmon[90393]: 2022-05-14T01:34:47.872+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:34:48.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:34:47 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:34:48.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:34:47 smithi115 ceph-mon[78593]: pgmap v4163: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:34:49.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:34:48 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:34:49.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:34:48 smithi115 conmon[90393]: 2022-05-14T01:34:48.862+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:34:49.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:34:48 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:34:50.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:34:49 smithi115 conmon[90393]: 2022-05-14T01:34:49.843+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:34:50.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:34:50 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:34:50.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:34:50 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:34:50.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:34:50 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:34:50.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:34:50 smithi115 ceph-mon[78593]: pgmap v4164: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:34:50.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:34:50 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 7620 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:34:50.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:34:50 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:34:50.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:34:50 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:34:50.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:34:50 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:34:50.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:34:50 smithi047 ceph-mon[77147]: pgmap v4164: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:34:50.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:34:50 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 7620 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:34:51.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:34:50 smithi115 conmon[90393]: 2022-05-14T01:34:50.843+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:34:51.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:34:51 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:34:51.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:34:51 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:34:52.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:34:51 smithi115 conmon[90393]: 2022-05-14T01:34:51.853+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:34:52.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:34:52 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:34:52.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:34:52 smithi115 ceph-mon[78593]: pgmap v4165: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:34:52.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:34:52 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:34:52.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:34:52 smithi047 ceph-mon[77147]: pgmap v4165: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:34:53.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:34:53 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:34:53.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:34:52 smithi115 conmon[90393]: 2022-05-14T01:34:52.857+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:34:53.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:34:53 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:34:54.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:34:53 smithi115 conmon[90393]: 2022-05-14T01:34:53.810+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:34:54.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:34:54 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:34:54.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:34:54 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:34:54.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:34:54 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:34:54.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:34:54 smithi115 ceph-mon[78593]: pgmap v4166: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:34:54.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:34:54 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:34:54.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:34:54 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:34:54.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:34:54 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:34:54.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:34:54 smithi047 ceph-mon[77147]: pgmap v4166: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:34:55.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:34:54 smithi115 conmon[90393]: 2022-05-14T01:34:54.797+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:34:55.188 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:34:55 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:34:55.188 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:34:55 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 7625 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:34:55.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:34:55 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:34:55.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:34:55 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 7625 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:34:56.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:34:55 smithi115 conmon[90393]: 2022-05-14T01:34:55.815+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:34:56.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:34:56 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:34:56.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:34:56 smithi115 ceph-mon[78593]: pgmap v4167: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:34:56.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:34:56 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:34:56.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:34:56 smithi047 ceph-mon[77147]: pgmap v4167: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:34:57.035 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:34:56 smithi115 conmon[90393]: 2022-05-14T01:34:56.778+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:34:57.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:34:57 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:34:57.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:34:57 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:34:58.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:34:57 smithi115 conmon[90393]: 2022-05-14T01:34:57.825+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:34:58.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:34:58 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:34:58.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:34:58 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:34:58.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:34:58 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:34:58.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:34:58 smithi115 ceph-mon[78593]: pgmap v4168: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:34:58.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:34:58 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:34:58.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:34:58 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:34:58.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:34:58 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:34:58.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:34:58 smithi047 ceph-mon[77147]: pgmap v4168: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:34:59.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:34:58 smithi115 conmon[90393]: 2022-05-14T01:34:58.835+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:34:59.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:34:59 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:34:59.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:34:59 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:35:00.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:34:59 smithi115 conmon[90393]: 2022-05-14T01:34:59.819+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:35:00.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:35:00 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:35:00.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:35:00 smithi115 ceph-mon[78593]: pgmap v4169: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:35:00.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:35:00 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 7630 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:35:00.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:35:00 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:35:00.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:35:00 smithi047 ceph-mon[77147]: pgmap v4169: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:35:00.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:35:00 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 7630 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:35:01.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:35:00 smithi115 conmon[90393]: 2022-05-14T01:35:00.835+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:35:01.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:35:01 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:35:01.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:35:01 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:35:02.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:35:01 smithi115 conmon[90393]: 2022-05-14T01:35:01.858+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:35:02.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:35:02 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:35:02.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:35:02 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:35:02.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:35:02 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:35:02.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:35:02 smithi115 ceph-mon[78593]: pgmap v4170: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:35:02.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:35:02 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:35:02.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:35:02 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:35:02.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:35:02 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:35:02.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:35:02 smithi047 ceph-mon[77147]: pgmap v4170: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:35:03.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:35:02 smithi115 conmon[90393]: 2022-05-14T01:35:02.863+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:35:03.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:35:03 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:35:03.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:35:03 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:35:04.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:35:03 smithi115 conmon[90393]: 2022-05-14T01:35:03.844+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:35:04.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:35:04 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:35:04.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:35:04 smithi115 ceph-mon[78593]: pgmap v4171: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:35:04.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:35:04 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:35:04.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:35:04 smithi047 ceph-mon[77147]: pgmap v4171: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:35:05.063 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:35:04 smithi115 conmon[90393]: 2022-05-14T01:35:04.803+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:35:05.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:35:05 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:35:05.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:35:05 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 7635 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:35:05.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:35:05 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:35:05.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:35:05 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 7635 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:35:06.069 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:35:05 smithi115 conmon[90393]: 2022-05-14T01:35:05.796+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:35:06.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:35:06 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:35:06.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:35:06 smithi115 ceph-mon[78593]: pgmap v4172: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:35:06.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:35:06 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:35:06.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:35:06 smithi047 ceph-mon[77147]: pgmap v4172: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:35:07.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:35:07 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:35:07.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:35:06 smithi115 conmon[90393]: 2022-05-14T01:35:06.821+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:35:07.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:35:07 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:35:08.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:35:07 smithi115 conmon[90393]: 2022-05-14T01:35:07.839+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:35:08.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:35:08 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:35:08.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:35:08 smithi115 ceph-mon[78593]: pgmap v4173: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:35:08.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:35:08 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:35:08.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:35:08 smithi047 ceph-mon[77147]: pgmap v4173: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:35:09.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:35:09 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:35:09.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:35:08 smithi115 conmon[90393]: 2022-05-14T01:35:08.881+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:35:09.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:35:09 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:35:10.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:35:09 smithi115 conmon[90393]: 2022-05-14T01:35:09.887+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:35:10.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:35:10 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:35:10.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:35:10 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:35:10.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:35:10 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:35:10.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:35:10 smithi115 ceph-mon[78593]: pgmap v4174: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:35:10.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:35:10 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 7640 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:35:10.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:35:10 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:35:10.917 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:35:10 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:35:10.917 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:35:10 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:35:10.917 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:35:10 smithi047 ceph-mon[77147]: pgmap v4174: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:35:10.918 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:35:10 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 7640 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:35:11.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:35:10 smithi115 conmon[90393]: 2022-05-14T01:35:10.903+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:35:11.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:35:11 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:35:11.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:35:11 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:35:12.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:35:11 smithi115 conmon[90393]: 2022-05-14T01:35:11.901+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:35:12.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:35:12 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:35:12.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:35:12 smithi115 ceph-mon[78593]: pgmap v4175: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:35:12.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:35:12 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:35:12.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:35:12 smithi047 ceph-mon[77147]: pgmap v4175: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:35:13.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:35:12 smithi115 conmon[90393]: 2022-05-14T01:35:12.906+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:35:13.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:35:13 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:35:13.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:35:13 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:35:14.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:35:13 smithi115 conmon[90393]: 2022-05-14T01:35:13.956+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:35:14.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:35:14 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:35:14.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:35:14 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:35:14.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:35:14 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:35:14.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:35:14 smithi115 ceph-mon[78593]: pgmap v4176: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:35:14.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:35:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T01:35:14.362 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:35:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T01:35:14.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:35:14 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:35:14.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:35:14 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:35:14.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:35:14 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:35:14.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:35:14 smithi047 ceph-mon[77147]: pgmap v4176: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:35:14.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:35:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T01:35:14.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:35:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T01:35:15.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:35:14 smithi115 conmon[90393]: 2022-05-14T01:35:14.927+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:35:15.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:35:15 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:35:15.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:35:15 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 7645 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:35:15.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:35:15 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:35:15.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:35:15 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 7645 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:35:16.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:35:15 smithi115 conmon[90393]: 2022-05-14T01:35:15.958+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:35:16.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:35:16 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:35:16.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:35:16 smithi115 ceph-mon[78593]: pgmap v4177: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:35:16.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:35:16 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:35:16.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:35:16 smithi047 ceph-mon[77147]: pgmap v4177: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:35:17.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:35:17 smithi115 conmon[90393]: 2022-05-14T01:35:17.000+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:35:17.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:35:17 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:35:17.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:35:17 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:35:18.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:35:17 smithi115 conmon[90393]: 2022-05-14T01:35:17.987+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:35:18.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:35:18 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:35:18.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:35:18 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:35:18.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:35:18 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:35:18.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:35:18 smithi115 ceph-mon[78593]: pgmap v4178: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:35:18.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:35:18 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:35:18.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:35:18 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:35:18.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:35:18 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:35:18.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:35:18 smithi047 ceph-mon[77147]: pgmap v4178: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:35:19.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:35:19 smithi115 conmon[90393]: 2022-05-14T01:35:19.008+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:35:19.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:35:19 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:35:19.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:35:19 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:35:20.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:35:19 smithi115 conmon[90393]: 2022-05-14T01:35:19.977+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:35:20.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:35:20 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:35:20.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:35:20 smithi115 ceph-mon[78593]: pgmap v4179: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:35:20.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:35:20 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 7650 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:35:20.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:35:20 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:35:20.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:35:20 smithi047 ceph-mon[77147]: pgmap v4179: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:35:20.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:35:20 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 7650 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:35:21.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:35:20 smithi115 conmon[90393]: 2022-05-14T01:35:20.948+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:35:21.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:35:21 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:35:21.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:35:21 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:35:22.257 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:35:21 smithi115 conmon[90393]: 2022-05-14T01:35:21.983+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:35:22.258 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:35:22 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:35:22.259 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:35:22 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:35:22.259 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:35:22 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:35:22.260 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:35:22 smithi115 ceph-mon[78593]: pgmap v4180: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:35:22.260 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:35:22 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T01:35:22.260 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:35:22 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T01:35:22.261 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:35:22 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T01:35:22.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:35:22 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:35:22.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:35:22 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:35:22.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:35:22 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:35:22.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:35:22 smithi047 ceph-mon[77147]: pgmap v4180: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:35:22.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:35:22 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T01:35:22.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:35:22 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T01:35:22.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:35:22 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T01:35:23.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:35:23 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:35:23.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:35:22 smithi115 conmon[90393]: 2022-05-14T01:35:22.980+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:35:23.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:35:23 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:35:24.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:35:23 smithi115 conmon[90393]: 2022-05-14T01:35:23.947+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:35:24.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:35:24 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:35:24.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:35:24 smithi115 ceph-mon[78593]: pgmap v4181: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:35:24.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:35:24 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:35:24.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:35:24 smithi047 ceph-mon[77147]: pgmap v4181: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:35:25.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:35:24 smithi115 conmon[90393]: 2022-05-14T01:35:24.990+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:35:25.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:35:25 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:35:25.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:35:25 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 7655 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:35:25.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:35:25 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:35:25.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:35:25 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 7655 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:35:25.542 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:35:25 smithi047 conmon[78521]: 2022-05-14T01:35:25.232+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:35:25.542 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:35:25 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T01:35:25.542 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:35:25 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:35:25.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:35:25 smithi047 conmon[78521]: yield 2022-05-14T01:35:25.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:35:25 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:35:25.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:35:25 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:35:25.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:35:25 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:35:25.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:35:25 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T01:35:25.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:35:25 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:35:25.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:35:25 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T01:35:25.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:35:25 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:35:25.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:35:25 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T01:35:25.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:35:25 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:35:25.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:35:25 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T01:35:25.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:35:25 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:35:25.547 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:35:25 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T01:35:25.547 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:35:25 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:35:25.547 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:35:25 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:35:25.547 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:35:25 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:35:26.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:35:25 smithi115 conmon[90393]: 2022-05-14T01:35:25.981+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:35:26.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:35:26 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:35:26.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:35:26 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:35:26.359 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T01:35:26.359 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:35:26.359 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T01:35:26.359 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:35:26.359 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:35:26.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:35:26.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T01:35:26.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:35:26.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T01:35:26.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:35:26.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T01:35:26.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:35:26.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T01:35:26.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:35:26.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T01:35:26.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:35:26.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:35:26.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:35:26.362 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:35:26 smithi115 ceph-mon[78593]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T01:35:26.363 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:35:26 smithi115 ceph-mon[78593]: pgmap v4182: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:35:26.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:35:26 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:35:26.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:35:26 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:35:26.541 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T01:35:26.542 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:35:26.542 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T01:35:26.542 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:35:26.542 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:35:26.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:35:26.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T01:35:26.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:35:26.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T01:35:26.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:35:26.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T01:35:26.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:35:26.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T01:35:26.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:35:26.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T01:35:26.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:35:26.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:35:26.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:35:26.545 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:35:26 smithi047 ceph-mon[77147]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T01:35:26.546 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:35:26 smithi047 ceph-mon[77147]: pgmap v4182: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:35:27.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:35:26 smithi115 conmon[90393]: 2022-05-14T01:35:26.944+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:35:27.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:35:27 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:35:27.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:35:27 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:35:28.161 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:35:27 smithi115 conmon[90393]: 2022-05-14T01:35:27.906+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:35:28.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:35:28 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:35:28.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:35:28 smithi047 ceph-mon[77147]: pgmap v4183: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:35:28.542 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:35:28 smithi047 conmon[78521]: 2022-05-14T01:35:28.304+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:35:28.542 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:35:28 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T01:35:28.542 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:35:28 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:35:28.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:35:28 smithi047 conmon[78521]: yield 2022-05-14T01:35:28.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:35:28 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:35:28.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:35:28 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:35:28.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:35:28 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:35:28.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:35:28 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T01:35:28.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:35:28 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:35:28.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:35:28 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T01:35:28.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:35:28 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:35:28.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:35:28 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T01:35:28.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:35:28 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:35:28.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:35:28 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T01:35:28.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:35:28 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:35:28.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:35:28 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T01:35:28.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:35:28 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:35:28.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:35:28 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:35:28.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:35:28 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:35:28.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:35:28 smithi047 conmon[78521]: 2022-05-14T01:35:28.305+0000 7f5345f84700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T01:35:28.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:35:28 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T01:35:28.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:35:28 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:35:28.547 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:35:28 smithi047 conmon[78521]: yield 2022-05-14T01:35:28.547 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:35:28 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:35:28.547 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:35:28 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:35:28.547 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:35:28 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:35:28.548 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:35:28 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T01:35:28.548 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:35:28 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:35:28.548 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:35:28 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T01:35:28.548 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:35:28 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:35:28.548 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:35:28 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T01:35:28.549 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:35:28 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:35:28.549 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:35:28 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T01:35:28.549 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:35:28 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:35:28.550 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:35:28 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T01:35:28.550 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:35:28 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:35:28.550 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:35:28 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:35:28.550 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:35:28 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:35:28.551 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:35:28 smithi047 conmon[78521]: 2022-05-14T01:35:28.551 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:35:28 smithi047 conmon[78521]: During handling of the above exception, another exception occurred: 2022-05-14T01:35:28.551 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:35:28 smithi047 conmon[78521]: 2022-05-14T01:35:28.551 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:35:28 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T01:35:28.551 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:35:28 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T01:35:28.552 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:35:28 smithi047 conmon[78521]: return f(*arg) 2022-05-14T01:35:28.552 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:35:28 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T01:35:28.552 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:35:28 smithi047 conmon[78521]: self._write_client_files(client_files, host) 2022-05-14T01:35:28.552 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:35:28 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T01:35:28.552 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:35:28 smithi047 conmon[78521]: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T01:35:28.553 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:35:28 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T01:35:28.553 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:35:28 smithi047 conmon[78521]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T01:35:28.553 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:35:28 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T01:35:28.553 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:35:28 smithi047 conmon[78521]: return self.event_loop.get_result(coro) 2022-05-14T01:35:28.554 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:35:28 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T01:35:28.554 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:35:28 smithi047 conmon[78521]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T01:35:28.554 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:35:28 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T01:35:28.554 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:35:28 smithi047 conmon[78521]: return self.__get_result() 2022-05-14T01:35:28.555 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:35:28 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T01:35:28.555 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:35:28 smithi047 conmon[78521]: raise self._exception 2022-05-14T01:35:28.555 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:35:28 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T01:35:28.555 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:35:28 smithi047 conmon[78521]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T01:35:28.555 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:35:28 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T01:35:28.556 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:35:28 smithi047 conmon[78521]: conn = await self._remote_connection(host, addr) 2022-05-14T01:35:28.556 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:35:28 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T01:35:28.556 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:35:28 smithi047 conmon[78521]: raise 2022-05-14T01:35:28.556 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:35:28 smithi047 conmon[78521]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T01:35:28.556 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:35:28 smithi047 conmon[78521]: self.gen.throw(type, value, traceback) 2022-05-14T01:35:28.557 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:35:28 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T01:35:28.557 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:35:28 smithi047 conmon[78521]: raise OrchestratorError(msg) 2022-05-14T01:35:28.557 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:35:28 smithi047 conmon[78521]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:35:28.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:35:28 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:35:28.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:35:28 smithi115 ceph-mon[78593]: pgmap v4183: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:35:29.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:35:28 smithi115 conmon[90393]: 2022-05-14T01:35:28.946+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:35:29.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:35:29 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:35:29.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:35:29 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:35:29.358 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T01:35:29.359 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:35:29.359 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T01:35:29.359 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:35:29.359 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:35:29.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:35:29.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T01:35:29.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:35:29.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T01:35:29.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:35:29.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T01:35:29.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:35:29.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T01:35:29.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:35:29.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T01:35:29.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:35:29.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:35:29.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:35:29.362 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:35:29 smithi115 ceph-mon[78593]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T01:35:29.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T01:35:29.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:35:29.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T01:35:29.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:35:29.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:35:29.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:35:29.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T01:35:29.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:35:29.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T01:35:29.365 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:35:29.365 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T01:35:29.365 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:35:29.365 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T01:35:29.365 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:35:29.366 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T01:35:29.366 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:35:29.366 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:35:29.366 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:35:29.366 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T01:35:29.367 INFO:journalctl@ceph.mon.c.smithi115.stdout: During handling of the above exception, another exception occurred: 2022-05-14T01:35:29.367 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T01:35:29.367 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T01:35:29.367 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T01:35:29.368 INFO:journalctl@ceph.mon.c.smithi115.stdout: return f(*arg) 2022-05-14T01:35:29.368 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T01:35:29.368 INFO:journalctl@ceph.mon.c.smithi115.stdout: self._write_client_files(client_files, host) 2022-05-14T01:35:29.368 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T01:35:29.368 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T01:35:29.369 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T01:35:29.369 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T01:35:29.369 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T01:35:29.370 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.event_loop.get_result(coro) 2022-05-14T01:35:29.370 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T01:35:29.370 INFO:journalctl@ceph.mon.c.smithi115.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T01:35:29.370 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T01:35:29.371 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.__get_result() 2022-05-14T01:35:29.371 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T01:35:29.371 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise self._exception 2022-05-14T01:35:29.371 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T01:35:29.371 INFO:journalctl@ceph.mon.c.smithi115.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T01:35:29.372 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T01:35:29.372 INFO:journalctl@ceph.mon.c.smithi115.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T01:35:29.372 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T01:35:29.372 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise 2022-05-14T01:35:29.373 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T01:35:29.373 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.gen.throw(type, value, traceback) 2022-05-14T01:35:29.373 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T01:35:29.373 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OrchestratorError(msg) 2022-05-14T01:35:29.374 INFO:journalctl@ceph.mon.c.smithi115.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:35:29.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:35:29 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:35:29.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:35:29 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:35:29.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T01:35:29.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:35:29.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T01:35:29.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:35:29.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:35:29.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:35:29.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T01:35:29.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:35:29.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T01:35:29.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:35:29.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T01:35:29.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:35:29.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T01:35:29.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:35:29.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T01:35:29.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:35:29.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:35:29.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:35:29.547 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:35:29 smithi047 ceph-mon[77147]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T01:35:29.548 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T01:35:29.548 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:35:29.548 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T01:35:29.548 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:35:29.549 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:35:29.549 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:35:29.549 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T01:35:29.549 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:35:29.549 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T01:35:29.550 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:35:29.550 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T01:35:29.550 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:35:29.550 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T01:35:29.551 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:35:29.551 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T01:35:29.551 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:35:29.551 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:35:29.552 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:35:29.552 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T01:35:29.552 INFO:journalctl@ceph.mon.b.smithi047.stdout: During handling of the above exception, another exception occurred: 2022-05-14T01:35:29.553 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T01:35:29.554 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T01:35:29.554 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T01:35:29.554 INFO:journalctl@ceph.mon.b.smithi047.stdout: return f(*arg) 2022-05-14T01:35:29.555 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T01:35:29.555 INFO:journalctl@ceph.mon.b.smithi047.stdout: self._write_client_files(client_files, host) 2022-05-14T01:35:29.555 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T01:35:29.556 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T01:35:29.556 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T01:35:29.556 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T01:35:29.556 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T01:35:29.557 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.event_loop.get_result(coro) 2022-05-14T01:35:29.557 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T01:35:29.557 INFO:journalctl@ceph.mon.b.smithi047.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T01:35:29.557 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T01:35:29.558 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.__get_result() 2022-05-14T01:35:29.558 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T01:35:29.558 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise self._exception 2022-05-14T01:35:29.558 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T01:35:29.558 INFO:journalctl@ceph.mon.b.smithi047.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T01:35:29.559 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T01:35:29.559 INFO:journalctl@ceph.mon.b.smithi047.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T01:35:29.559 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T01:35:29.559 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise 2022-05-14T01:35:29.560 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T01:35:29.560 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.gen.throw(type, value, traceback) 2022-05-14T01:35:29.560 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T01:35:29.560 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OrchestratorError(msg) 2022-05-14T01:35:29.561 INFO:journalctl@ceph.mon.b.smithi047.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:35:30.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:35:29 smithi115 conmon[90393]: 2022-05-14T01:35:29.952+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:35:30.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:35:30 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:35:30.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:35:30 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:35:30.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:35:30 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:35:30.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:35:30 smithi115 ceph-mon[78593]: pgmap v4184: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:35:30.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:35:30 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 7660 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:35:30.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:35:30 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:35:30.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:35:30 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:35:30.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:35:30 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:35:30.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:35:30 smithi047 ceph-mon[77147]: pgmap v4184: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:35:30.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:35:30 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 7660 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:35:31.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:35:30 smithi115 conmon[90393]: 2022-05-14T01:35:30.936+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:35:31.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:35:31 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:35:31.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:35:31 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:35:32.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:35:31 smithi115 conmon[90393]: 2022-05-14T01:35:31.981+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:35:32.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:35:32 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:35:32.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:35:32 smithi115 ceph-mon[78593]: pgmap v4185: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:35:32.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:35:32 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:35:32.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:35:32 smithi047 ceph-mon[77147]: pgmap v4185: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:35:33.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:35:33 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:35:33.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:35:32 smithi115 conmon[90393]: 2022-05-14T01:35:32.985+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:35:33.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:35:33 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:35:34.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:35:33 smithi115 conmon[90393]: 2022-05-14T01:35:33.981+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:35:34.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:35:34 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:35:34.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:35:34 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:35:34.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:35:34 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:35:34.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:35:34 smithi115 ceph-mon[78593]: pgmap v4186: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:35:34.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:35:34 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:35:34.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:35:34 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:35:34.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:35:34 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:35:34.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:35:34 smithi047 ceph-mon[77147]: pgmap v4186: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:35:35.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:35:34 smithi115 conmon[90393]: 2022-05-14T01:35:34.941+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:35:35.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:35:35 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:35:35.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:35:35 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 7665 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:35:35.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:35:35 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:35:35.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:35:35 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 7665 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:35:36.195 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:35:35 smithi115 conmon[90393]: 2022-05-14T01:35:35.906+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:35:36.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:35:36 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:35:36.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:35:36 smithi047 ceph-mon[77147]: pgmap v4187: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:35:36.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:35:36 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:35:36.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:35:36 smithi115 ceph-mon[78593]: pgmap v4187: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:35:37.204 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:35:36 smithi115 conmon[90393]: 2022-05-14T01:35:36.948+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:35:37.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:35:37 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:35:37.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:35:37 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:35:38.209 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:35:37 smithi115 conmon[90393]: 2022-05-14T01:35:37.923+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:35:38.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:35:38 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:35:38.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:35:38 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:35:38.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:35:38 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:35:38.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:35:38 smithi047 ceph-mon[77147]: pgmap v4188: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:35:38.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:35:38 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:35:38.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:35:38 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:35:38.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:35:38 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:35:38.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:35:38 smithi115 ceph-mon[78593]: pgmap v4188: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:35:39.208 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:35:38 smithi115 conmon[90393]: 2022-05-14T01:35:38.898+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:35:39.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:35:39 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:35:39.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:35:39 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:35:40.212 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:35:39 smithi115 conmon[90393]: 2022-05-14T01:35:39.924+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:35:40.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:35:40 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:35:40.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:35:40 smithi047 ceph-mon[77147]: pgmap v4189: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:35:40.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:35:40 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 7670 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:35:40.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:35:40 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:35:40.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:35:40 smithi115 ceph-mon[78593]: pgmap v4189: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:35:40.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:35:40 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 7670 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:35:41.214 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:35:40 smithi115 conmon[90393]: 2022-05-14T01:35:40.877+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:35:41.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:35:41 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:35:41.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:35:41 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:35:42.223 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:35:41 smithi115 conmon[90393]: 2022-05-14T01:35:41.888+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:35:42.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:35:42 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:35:42.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:35:42 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:35:42.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:35:42 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:35:42.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:35:42 smithi047 ceph-mon[77147]: pgmap v4190: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:35:42.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:35:42 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:35:42.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:35:42 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:35:42.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:35:42 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:35:42.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:35:42 smithi115 ceph-mon[78593]: pgmap v4190: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:35:43.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:35:42 smithi115 conmon[90393]: 2022-05-14T01:35:42.853+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:35:43.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:35:43 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:35:43.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:35:43 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:35:44.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:35:43 smithi115 conmon[90393]: 2022-05-14T01:35:43.820+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:35:44.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:35:44 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:35:44.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:35:44 smithi047 ceph-mon[77147]: pgmap v4191: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:35:44.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:35:44 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:35:44.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:35:44 smithi115 ceph-mon[78593]: pgmap v4191: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:35:45.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:35:44 smithi115 conmon[90393]: 2022-05-14T01:35:44.798+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:35:45.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:35:45 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:35:45.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:35:45 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 7675 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:35:45.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:35:45 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:35:45.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:35:45 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 7675 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:35:46.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:35:45 smithi115 conmon[90393]: 2022-05-14T01:35:45.753+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:35:46.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:35:46 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:35:46.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:35:46 smithi047 ceph-mon[77147]: pgmap v4192: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:35:46.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:35:46 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:35:46.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:35:46 smithi115 ceph-mon[78593]: pgmap v4192: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:35:47.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:35:46 smithi115 conmon[90393]: 2022-05-14T01:35:46.720+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:35:47.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:35:47 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:35:47.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:35:47 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:35:48.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:35:47 smithi115 conmon[90393]: 2022-05-14T01:35:47.721+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:35:48.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:35:48 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:35:48.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:35:48 smithi047 ceph-mon[77147]: pgmap v4193: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:35:48.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:35:48 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:35:48.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:35:48 smithi115 ceph-mon[78593]: pgmap v4193: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:35:49.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:35:48 smithi115 conmon[90393]: 2022-05-14T01:35:48.763+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:35:49.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:35:49 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:35:49.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:35:49 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:35:50.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:35:49 smithi115 conmon[90393]: 2022-05-14T01:35:49.809+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:35:50.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:35:50 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:35:50.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:35:50 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:35:50.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:35:50 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:35:50.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:35:50 smithi047 ceph-mon[77147]: pgmap v4194: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:35:50.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:35:50 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 7680 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:35:50.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:35:50 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:35:50.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:35:50 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:35:50.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:35:50 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:35:50.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:35:50 smithi115 ceph-mon[78593]: pgmap v4194: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:35:50.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:35:50 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 7680 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:35:51.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:35:50 smithi115 conmon[90393]: 2022-05-14T01:35:50.769+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:35:51.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:35:51 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:35:51.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:35:51 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:35:52.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:35:51 smithi115 conmon[90393]: 2022-05-14T01:35:51.771+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:35:52.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:35:52 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:35:52.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:35:52 smithi047 ceph-mon[77147]: pgmap v4195: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:35:52.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:35:52 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:35:52.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:35:52 smithi115 ceph-mon[78593]: pgmap v4195: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:35:53.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:35:52 smithi115 conmon[90393]: 2022-05-14T01:35:52.795+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:35:53.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:35:53 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:35:53.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:35:53 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:35:54.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:35:53 smithi115 conmon[90393]: 2022-05-14T01:35:53.819+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:35:54.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:35:54 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:35:54.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:35:54 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:35:54.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:35:54 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:35:54.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:35:54 smithi047 ceph-mon[77147]: pgmap v4196: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:35:54.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:35:54 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:35:54.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:35:54 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:35:54.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:35:54 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:35:54.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:35:54 smithi115 ceph-mon[78593]: pgmap v4196: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:35:55.273 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:35:54 smithi115 conmon[90393]: 2022-05-14T01:35:54.860+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:35:55.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:35:55 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:35:55.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:35:55 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 7685 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:35:55.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:35:55 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:35:55.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:35:55 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 7685 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:35:56.278 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:35:55 smithi115 conmon[90393]: 2022-05-14T01:35:55.865+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:35:56.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:35:56 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:35:56.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:35:56 smithi047 ceph-mon[77147]: pgmap v4197: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:35:56.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:35:56 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:35:56.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:35:56 smithi115 ceph-mon[78593]: pgmap v4197: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:35:57.284 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:35:56 smithi115 conmon[90393]: 2022-05-14T01:35:56.870+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:35:57.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:35:57 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:35:57.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:35:57 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:35:58.289 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:35:57 smithi115 conmon[90393]: 2022-05-14T01:35:57.914+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:35:58.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:35:58 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:35:58.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:35:58 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:35:58.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:35:58 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:35:58.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:35:58 smithi047 ceph-mon[77147]: pgmap v4198: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:35:58.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:35:58 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:35:58.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:35:58 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:35:58.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:35:58 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:35:58.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:35:58 smithi115 ceph-mon[78593]: pgmap v4198: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:35:59.290 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:35:58 smithi115 conmon[90393]: 2022-05-14T01:35:58.907+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:35:59.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:35:59 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:35:59.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:35:59 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:36:00.295 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:35:59 smithi115 conmon[90393]: 2022-05-14T01:35:59.944+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:36:00.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:36:00 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:36:00.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:36:00 smithi115 ceph-mon[78593]: pgmap v4199: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:36:00.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:36:00 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 7690 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:36:00.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:36:00 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:36:00.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:36:00 smithi047 ceph-mon[77147]: pgmap v4199: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:36:00.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:36:00 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 7690 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:36:01.301 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:36:00 smithi115 conmon[90393]: 2022-05-14T01:36:00.992+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:36:01.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:36:01 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:36:01.799 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:36:01 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:36:02.306 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:36:01 smithi115 conmon[90393]: 2022-05-14T01:36:01.987+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:36:02.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:36:02 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:36:02.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:36:02 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:36:02.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:36:02 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:36:02.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:36:02 smithi115 ceph-mon[78593]: pgmap v4200: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:36:02.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:36:02 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:36:02.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:36:02 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:36:02.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:36:02 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:36:02.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:36:02 smithi047 ceph-mon[77147]: pgmap v4200: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:36:03.308 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:36:02 smithi115 conmon[90393]: 2022-05-14T01:36:02.988+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:36:03.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:36:03 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:36:03.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:36:03 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:36:04.315 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:36:04 smithi115 conmon[90393]: 2022-05-14T01:36:04.014+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:36:04.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:36:04 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:36:04.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:36:04 smithi115 ceph-mon[78593]: pgmap v4201: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:36:04.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:36:04 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:36:04.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:36:04 smithi047 ceph-mon[77147]: pgmap v4201: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:36:05.316 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:36:05 smithi115 conmon[90393]: 2022-05-14T01:36:05.026+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:36:05.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:36:05 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:36:05.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:36:05 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 7695 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:36:05.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:36:05 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:36:05.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:36:05 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 7695 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:36:06.330 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:36:05 smithi115 conmon[90393]: 2022-05-14T01:36:05.993+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:36:06.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:36:06 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:36:06.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:36:06 smithi115 ceph-mon[78593]: pgmap v4202: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:36:06.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:36:06 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:36:06.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:36:06 smithi047 ceph-mon[77147]: pgmap v4202: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:36:07.329 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:36:06 smithi115 conmon[90393]: 2022-05-14T01:36:06.953+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:36:07.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:36:07 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:36:07.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:36:07 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:36:08.334 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:36:07 smithi115 conmon[90393]: 2022-05-14T01:36:07.925+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:36:08.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:36:08 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:36:08.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:36:08 smithi115 ceph-mon[78593]: pgmap v4203: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:36:08.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:36:08 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:36:08.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:36:08 smithi047 ceph-mon[77147]: pgmap v4203: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:36:09.340 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:36:08 smithi115 conmon[90393]: 2022-05-14T01:36:08.899+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:36:09.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:36:09 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:36:09.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:36:09 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:36:10.344 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:36:09 smithi115 conmon[90393]: 2022-05-14T01:36:09.866+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:36:10.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:36:10 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:36:10.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:36:10 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:36:10.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:36:10 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:36:10.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:36:10 smithi115 ceph-mon[78593]: pgmap v4204: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:36:10.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:36:10 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 7700 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:36:10.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:36:10 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:36:10.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:36:10 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:36:10.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:36:10 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:36:10.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:36:10 smithi047 ceph-mon[77147]: pgmap v4204: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:36:10.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:36:10 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 7700 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:36:11.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:36:10 smithi115 conmon[90393]: 2022-05-14T01:36:10.835+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:36:11.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:36:11 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:36:11.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:36:11 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:36:12.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:36:11 smithi115 conmon[90393]: 2022-05-14T01:36:11.848+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:36:12.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:36:12 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:36:12.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:36:12 smithi115 ceph-mon[78593]: pgmap v4205: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:36:12.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:36:12 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:36:12.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:36:12 smithi047 ceph-mon[77147]: pgmap v4205: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:36:13.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:36:12 smithi115 conmon[90393]: 2022-05-14T01:36:12.821+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:36:13.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:36:13 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:36:13.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:36:13 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:36:14.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:36:13 smithi115 conmon[90393]: 2022-05-14T01:36:13.828+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:36:14.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:36:14 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:36:14.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:36:14 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:36:14.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:36:14 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:36:14.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:36:14 smithi047 ceph-mon[77147]: pgmap v4206: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:36:14.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:36:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T01:36:14.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:36:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T01:36:14.855 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:36:14 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:36:14.855 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:36:14 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:36:14.856 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:36:14 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:36:14.856 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:36:14 smithi115 ceph-mon[78593]: pgmap v4206: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:36:14.856 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:36:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T01:36:14.856 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:36:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T01:36:15.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:36:14 smithi115 conmon[90393]: 2022-05-14T01:36:14.854+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:36:15.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:36:15 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:36:15.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:36:15 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 7705 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:36:15.855 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:36:15 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:36:15.856 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:36:15 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 7705 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:36:16.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:36:15 smithi115 conmon[90393]: 2022-05-14T01:36:15.854+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:36:16.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:36:16 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:36:16.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:36:16 smithi047 ceph-mon[77147]: pgmap v4207: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:36:16.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:36:16 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:36:16.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:36:16 smithi115 ceph-mon[78593]: pgmap v4207: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:36:17.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:36:16 smithi115 conmon[90393]: 2022-05-14T01:36:16.886+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:36:17.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:36:17 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:36:17.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:36:17 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:36:18.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:36:17 smithi115 conmon[90393]: 2022-05-14T01:36:17.882+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:36:18.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:36:18 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:36:18.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:36:18 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:36:18.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:36:18 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:36:18.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:36:18 smithi047 ceph-mon[77147]: pgmap v4208: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:36:18.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:36:18 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:36:18.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:36:18 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:36:18.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:36:18 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:36:18.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:36:18 smithi115 ceph-mon[78593]: pgmap v4208: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:36:19.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:36:18 smithi115 conmon[90393]: 2022-05-14T01:36:18.862+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:36:19.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:36:19 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:36:19.821 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:36:19 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:36:20.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:36:19 smithi115 conmon[90393]: 2022-05-14T01:36:19.820+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:36:20.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:36:20 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:36:20.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:36:20 smithi047 ceph-mon[77147]: pgmap v4209: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:36:20.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:36:20 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 7710 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:36:20.832 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:36:20 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:36:20.833 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:36:20 smithi115 ceph-mon[78593]: pgmap v4209: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:36:20.833 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:36:20 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 7710 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:36:21.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:36:20 smithi115 conmon[90393]: 2022-05-14T01:36:20.831+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:36:21.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:36:21 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:36:21.823 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:36:21 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:36:22.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:36:21 smithi115 conmon[90393]: 2022-05-14T01:36:21.821+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:36:22.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:36:22 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:36:22.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:36:22 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:36:22.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:36:22 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:36:22.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:36:22 smithi047 ceph-mon[77147]: pgmap v4210: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:36:22.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:36:22 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:36:22.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:36:22 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:36:22.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:36:22 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:36:22.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:36:22 smithi115 ceph-mon[78593]: pgmap v4210: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:36:23.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:36:22 smithi115 conmon[90393]: 2022-05-14T01:36:22.862+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:36:23.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:36:23 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:36:23.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:36:23 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:36:24.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:36:23 smithi115 conmon[90393]: 2022-05-14T01:36:23.884+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:36:24.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:36:24 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:36:24.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:36:24 smithi047 ceph-mon[77147]: pgmap v4211: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:36:24.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:36:24 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:36:24.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:36:24 smithi115 ceph-mon[78593]: pgmap v4211: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:36:25.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:36:24 smithi115 conmon[90393]: 2022-05-14T01:36:24.860+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:36:25.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:36:25 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:36:25.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:36:25 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 7715 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:36:25.818 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:36:25 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:36:25.818 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:36:25 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 7715 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:36:26.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:36:25 smithi115 conmon[90393]: 2022-05-14T01:36:25.817+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:36:26.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:36:26 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:36:26.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:36:26 smithi047 ceph-mon[77147]: pgmap v4212: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:36:26.828 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:36:26 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:36:26.828 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:36:26 smithi115 ceph-mon[78593]: pgmap v4212: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:36:27.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:36:26 smithi115 conmon[90393]: 2022-05-14T01:36:26.826+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:36:27.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:36:27 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:36:27.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:36:27 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:36:28.117 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:36:27 smithi115 conmon[90393]: 2022-05-14T01:36:27.856+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:36:28.721 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:36:28 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:36:28.721 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:36:28 smithi047 ceph-mon[77147]: pgmap v4213: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:36:28.722 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:36:28 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T01:36:28.722 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:36:28 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T01:36:28.722 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:36:28 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T01:36:28.820 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:36:28 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:36:28.821 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:36:28 smithi115 ceph-mon[78593]: pgmap v4213: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:36:28.821 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:36:28 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T01:36:28.821 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:36:28 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T01:36:28.821 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:36:28 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T01:36:29.042 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:36:28 smithi047 conmon[78521]: 2022-05-14T01:36:28.719+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:36:29.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:36:28 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T01:36:29.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:36:28 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:36:29.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:36:28 smithi047 conmon[78521]: yield 2022-05-14T01:36:29.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:36:28 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:36:29.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:36:28 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:36:29.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:36:28 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:36:29.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:36:28 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T01:36:29.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:36:28 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:36:29.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:36:28 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T01:36:29.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:36:28 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:36:29.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:36:28 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T01:36:29.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:36:28 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:36:29.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:36:28 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T01:36:29.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:36:28 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:36:29.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:36:28 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T01:36:29.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:36:28 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:36:29.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:36:28 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:36:29.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:36:28 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:36:29.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:36:28 smithi115 conmon[90393]: 2022-05-14T01:36:28.892+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:36:29.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:36:29 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:36:29.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:36:29 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:36:29.791 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T01:36:29.792 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:36:29.792 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T01:36:29.792 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:36:29.792 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:36:29.792 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:36:29.793 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T01:36:29.793 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:36:29.793 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T01:36:29.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:36:29.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T01:36:29.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:36:29.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T01:36:29.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:36:29.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T01:36:29.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:36:29.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:36:29.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:36:29.797 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:36:29 smithi047 ceph-mon[77147]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T01:36:29.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:36:29 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:36:29.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:36:29 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:36:29.858 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T01:36:29.858 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:36:29.858 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T01:36:29.859 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:36:29.859 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:36:29.859 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:36:29.859 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T01:36:29.860 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:36:29.860 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T01:36:29.860 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:36:29.860 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T01:36:29.860 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:36:29.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T01:36:29.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:36:29.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T01:36:29.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:36:29.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:36:29.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:36:29.862 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:36:29 smithi115 ceph-mon[78593]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T01:36:30.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:36:29 smithi115 conmon[90393]: 2022-05-14T01:36:29.863+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:36:30.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:36:30 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:36:30.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:36:30 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:36:30.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:36:30 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:36:30.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:36:30 smithi047 ceph-mon[77147]: pgmap v4214: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:36:30.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:36:30 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 7720 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:36:30.825 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:36:30 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:36:30.825 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:36:30 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:36:30.825 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:36:30 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:36:30.826 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:36:30 smithi115 ceph-mon[78593]: pgmap v4214: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:36:30.826 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:36:30 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 7720 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:36:31.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:36:30 smithi115 conmon[90393]: 2022-05-14T01:36:30.823+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:36:31.777 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:36:31 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:36:31.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:36:31 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:36:32.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:36:31 smithi115 conmon[90393]: 2022-05-14T01:36:31.776+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:36:32.291 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:36:31 smithi047 conmon[78521]: 2022-05-14T01:36:31.792+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:36:32.292 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:36:31 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T01:36:32.292 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:36:31 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:36:32.292 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:36:31 smithi047 conmon[78521]: yield 2022-05-14T01:36:32.292 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:36:31 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:36:32.293 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:36:31 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:36:32.293 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:36:31 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:36:32.293 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:36:31 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T01:36:32.293 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:36:31 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:36:32.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:36:31 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T01:36:32.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:36:31 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:36:32.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:36:31 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T01:36:32.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:36:31 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:36:32.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:36:31 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T01:36:32.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:36:31 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:36:32.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:36:31 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T01:36:32.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:36:31 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:36:32.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:36:31 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:36:32.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:36:31 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:36:32.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:36:31 smithi047 conmon[78521]: 2022-05-14T01:36:31.793+0000 7f5340779700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T01:36:32.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:36:31 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T01:36:32.297 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:36:31 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:36:32.297 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:36:31 smithi047 conmon[78521]: yield 2022-05-14T01:36:32.297 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:36:31 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:36:32.297 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:36:31 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:36:32.297 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:36:31 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:36:32.298 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:36:31 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T01:36:32.298 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:36:31 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:36:32.298 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:36:31 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T01:36:32.299 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:36:31 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:36:32.299 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:36:31 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T01:36:32.299 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:36:31 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:36:32.299 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:36:31 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T01:36:32.300 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:36:31 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:36:32.300 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:36:31 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T01:36:32.300 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:36:31 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:36:32.300 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:36:31 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:36:32.300 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:36:31 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:36:32.301 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:36:31 smithi047 conmon[78521]: 2022-05-14T01:36:32.301 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:36:31 smithi047 conmon[78521]: During handling of the above exception, another exception occurred: 2022-05-14T01:36:32.301 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:36:31 smithi047 conmon[78521]: 2022-05-14T01:36:32.301 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:36:31 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T01:36:32.302 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:36:31 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T01:36:32.302 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:36:31 smithi047 conmon[78521]: return f(*arg) 2022-05-14T01:36:32.302 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:36:31 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T01:36:32.302 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:36:31 smithi047 conmon[78521]: self._write_client_files(client_files, host) 2022-05-14T01:36:32.302 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:36:31 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T01:36:32.303 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:36:31 smithi047 conmon[78521]: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T01:36:32.303 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:36:31 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T01:36:32.304 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:36:31 smithi047 conmon[78521]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T01:36:32.304 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:36:31 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T01:36:32.305 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:36:31 smithi047 conmon[78521]: return self.event_loop.get_result(coro) 2022-05-14T01:36:32.305 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:36:31 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T01:36:32.306 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:36:31 smithi047 conmon[78521]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T01:36:32.306 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:36:31 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T01:36:32.306 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:36:31 smithi047 conmon[78521]: return self.__get_result() 2022-05-14T01:36:32.307 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:36:31 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T01:36:32.307 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:36:31 smithi047 conmon[78521]: raise self._exception 2022-05-14T01:36:32.307 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:36:31 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T01:36:32.308 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:36:31 smithi047 conmon[78521]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T01:36:32.308 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:36:31 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T01:36:32.308 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:36:31 smithi047 conmon[78521]: conn = await self._remote_connection(host, addr) 2022-05-14T01:36:32.309 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:36:31 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T01:36:32.309 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:36:31 smithi047 conmon[78521]: raise 2022-05-14T01:36:32.309 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:36:31 smithi047 conmon[78521]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T01:36:32.310 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:36:31 smithi047 conmon[78521]: self.gen.throw(type, value, traceback) 2022-05-14T01:36:32.310 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:36:31 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T01:36:32.310 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:36:31 smithi047 conmon[78521]: raise OrchestratorError(msg) 2022-05-14T01:36:32.311 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:36:31 smithi047 conmon[78521]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:36:32.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:36:32 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:36:32.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:36:32 smithi047 ceph-mon[77147]: pgmap v4215: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:36:32.811 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:36:32 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:36:32.811 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:36:32 smithi115 ceph-mon[78593]: pgmap v4215: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:36:33.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:36:32 smithi115 conmon[90393]: 2022-05-14T01:36:32.810+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:36:33.776 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:36:33 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:36:33.776 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:36:33 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:36:33.776 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T01:36:33.777 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:36:33.777 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T01:36:33.777 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:36:33.777 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:36:33.778 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:36:33.778 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T01:36:33.779 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:36:33.779 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T01:36:33.780 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:36:33.780 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T01:36:33.780 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:36:33.780 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T01:36:33.781 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:36:33.781 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T01:36:33.781 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:36:33.781 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:36:33.782 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:36:33.782 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:36:33 smithi115 ceph-mon[78593]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T01:36:33.782 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T01:36:33.782 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:36:33.782 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T01:36:33.783 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:36:33.783 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:36:33.783 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:36:33.783 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T01:36:33.784 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:36:33.784 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T01:36:33.784 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:36:33.784 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T01:36:33.785 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:36:33.785 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T01:36:33.785 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:36:33.785 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T01:36:33.786 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:36:33.786 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:36:33.786 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:36:33.786 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T01:36:33.787 INFO:journalctl@ceph.mon.c.smithi115.stdout: During handling of the above exception, another exception occurred: 2022-05-14T01:36:33.787 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T01:36:33.787 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T01:36:33.787 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T01:36:33.787 INFO:journalctl@ceph.mon.c.smithi115.stdout: return f(*arg) 2022-05-14T01:36:33.788 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T01:36:33.788 INFO:journalctl@ceph.mon.c.smithi115.stdout: self._write_client_files(client_files, host) 2022-05-14T01:36:33.788 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T01:36:33.788 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T01:36:33.788 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T01:36:33.789 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T01:36:33.789 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T01:36:33.789 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.event_loop.get_result(coro) 2022-05-14T01:36:33.789 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T01:36:33.790 INFO:journalctl@ceph.mon.c.smithi115.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T01:36:33.790 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T01:36:33.790 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.__get_result() 2022-05-14T01:36:33.790 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T01:36:33.790 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise self._exception 2022-05-14T01:36:33.791 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T01:36:33.791 INFO:journalctl@ceph.mon.c.smithi115.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T01:36:33.791 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T01:36:33.791 INFO:journalctl@ceph.mon.c.smithi115.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T01:36:33.791 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T01:36:33.792 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise 2022-05-14T01:36:33.792 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T01:36:33.793 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.gen.throw(type, value, traceback) 2022-05-14T01:36:33.793 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T01:36:33.793 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OrchestratorError(msg) 2022-05-14T01:36:33.794 INFO:journalctl@ceph.mon.c.smithi115.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:36:33.795 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:36:33 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:36:33.795 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:36:33 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:36:33.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T01:36:33.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:36:33.797 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T01:36:33.797 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:36:33.797 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:36:33.798 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:36:33.798 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T01:36:33.799 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:36:33.799 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T01:36:33.799 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:36:33.800 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T01:36:33.800 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:36:33.801 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T01:36:33.801 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:36:33.802 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T01:36:33.802 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:36:33.802 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:36:33.803 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:36:33.803 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:36:33 smithi047 ceph-mon[77147]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T01:36:33.803 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T01:36:33.804 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:36:33.804 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T01:36:33.805 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:36:33.805 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:36:33.805 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:36:33.806 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T01:36:33.806 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:36:33.806 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T01:36:33.806 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:36:33.807 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T01:36:33.807 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:36:33.807 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T01:36:33.807 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:36:33.807 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T01:36:33.808 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:36:33.808 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:36:33.808 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:36:33.808 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T01:36:33.808 INFO:journalctl@ceph.mon.b.smithi047.stdout: During handling of the above exception, another exception occurred: 2022-05-14T01:36:33.809 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T01:36:33.809 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T01:36:33.809 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T01:36:33.809 INFO:journalctl@ceph.mon.b.smithi047.stdout: return f(*arg) 2022-05-14T01:36:33.810 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T01:36:33.810 INFO:journalctl@ceph.mon.b.smithi047.stdout: self._write_client_files(client_files, host) 2022-05-14T01:36:33.810 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T01:36:33.810 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T01:36:33.810 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T01:36:33.811 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T01:36:33.811 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T01:36:33.811 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.event_loop.get_result(coro) 2022-05-14T01:36:33.811 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T01:36:33.812 INFO:journalctl@ceph.mon.b.smithi047.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T01:36:33.812 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T01:36:33.812 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.__get_result() 2022-05-14T01:36:33.812 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T01:36:33.813 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise self._exception 2022-05-14T01:36:33.813 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T01:36:33.813 INFO:journalctl@ceph.mon.b.smithi047.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T01:36:33.813 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T01:36:33.814 INFO:journalctl@ceph.mon.b.smithi047.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T01:36:33.814 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T01:36:33.814 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise 2022-05-14T01:36:33.814 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T01:36:33.814 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.gen.throw(type, value, traceback) 2022-05-14T01:36:33.815 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T01:36:33.815 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OrchestratorError(msg) 2022-05-14T01:36:33.815 INFO:journalctl@ceph.mon.b.smithi047.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:36:34.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:36:33 smithi115 conmon[90393]: 2022-05-14T01:36:33.774+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:36:34.777 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:36:34 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:36:34.778 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:36:34 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:36:34.778 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:36:34 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:36:34.778 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:36:34 smithi115 ceph-mon[78593]: pgmap v4216: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:36:34.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:36:34 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:36:34.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:36:34 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:36:34.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:36:34 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:36:34.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:36:34 smithi047 ceph-mon[77147]: pgmap v4216: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:36:35.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:36:34 smithi115 conmon[90393]: 2022-05-14T01:36:34.776+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:36:35.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:36:35 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:36:35.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:36:35 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 7725 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:36:35.799 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:36:35 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:36:35.799 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:36:35 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 7725 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:36:36.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:36:35 smithi115 conmon[90393]: 2022-05-14T01:36:35.798+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:36:36.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:36:36 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:36:36.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:36:36 smithi047 ceph-mon[77147]: pgmap v4217: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:36:36.849 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:36:36 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:36:36.850 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:36:36 smithi115 ceph-mon[78593]: pgmap v4217: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:36:37.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:36:36 smithi115 conmon[90393]: 2022-05-14T01:36:36.848+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:36:37.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:36:37 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:36:37.849 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:36:37 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:36:38.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:36:37 smithi115 conmon[90393]: 2022-05-14T01:36:37.848+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:36:38.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:36:38 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:36:38.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:36:38 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:36:38.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:36:38 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:36:38.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:36:38 smithi047 ceph-mon[77147]: pgmap v4218: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:36:38.849 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:36:38 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:36:38.850 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:36:38 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:36:38.850 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:36:38 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:36:38.850 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:36:38 smithi115 ceph-mon[78593]: pgmap v4218: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:36:39.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:36:38 smithi115 conmon[90393]: 2022-05-14T01:36:38.848+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:36:39.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:36:39 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:36:39.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:36:39 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:36:40.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:36:39 smithi115 conmon[90393]: 2022-05-14T01:36:39.872+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:36:40.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:36:40 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:36:40.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:36:40 smithi047 ceph-mon[77147]: pgmap v4219: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:36:40.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:36:40 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 7730 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:36:40.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:36:40 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:36:40.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:36:40 smithi115 ceph-mon[78593]: pgmap v4219: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:36:40.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:36:40 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 7730 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:36:41.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:36:40 smithi115 conmon[90393]: 2022-05-14T01:36:40.917+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:36:41.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:36:41 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:36:41.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:36:41 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:36:42.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:36:41 smithi115 conmon[90393]: 2022-05-14T01:36:41.890+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:36:42.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:36:42 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:36:42.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:36:42 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:36:42.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:36:42 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:36:42.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:36:42 smithi047 ceph-mon[77147]: pgmap v4220: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:36:42.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:36:42 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:36:42.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:36:42 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:36:42.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:36:42 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:36:42.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:36:42 smithi115 ceph-mon[78593]: pgmap v4220: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:36:43.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:36:42 smithi115 conmon[90393]: 2022-05-14T01:36:42.861+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:36:43.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:36:43 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:36:43.856 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:36:43 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:36:44.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:36:43 smithi115 conmon[90393]: 2022-05-14T01:36:43.854+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:36:44.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:36:44 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:36:44.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:36:44 smithi047 ceph-mon[77147]: pgmap v4221: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:36:44.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:36:44 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:36:44.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:36:44 smithi115 ceph-mon[78593]: pgmap v4221: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:36:45.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:36:44 smithi115 conmon[90393]: 2022-05-14T01:36:44.889+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:36:45.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:36:45 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:36:45.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:36:45 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 7735 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:36:45.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:36:45 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:36:45.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:36:45 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 7735 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:36:46.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:36:45 smithi115 conmon[90393]: 2022-05-14T01:36:45.903+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:36:46.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:36:46 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:36:46.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:36:46 smithi047 ceph-mon[77147]: pgmap v4222: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:36:46.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:36:46 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:36:46.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:36:46 smithi115 ceph-mon[78593]: pgmap v4222: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:36:47.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:36:46 smithi115 conmon[90393]: 2022-05-14T01:36:46.951+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:36:47.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:36:47 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:36:47.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:36:47 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:36:47.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:36:47 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:36:47.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:36:47 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:36:48.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:36:47 smithi115 conmon[90393]: 2022-05-14T01:36:47.918+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:36:48.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:36:48 smithi047 ceph-mon[77147]: pgmap v4223: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:36:48.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:36:48 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:36:48.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:36:48 smithi115 ceph-mon[78593]: pgmap v4223: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:36:48.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:36:48 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:36:49.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:36:48 smithi115 conmon[90393]: 2022-05-14T01:36:48.899+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:36:49.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:36:49 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:36:49.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:36:49 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:36:50.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:36:49 smithi115 conmon[90393]: 2022-05-14T01:36:49.880+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:36:50.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:36:50 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:36:50.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:36:50 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:36:50.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:36:50 smithi047 ceph-mon[77147]: pgmap v4224: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:36:50.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:36:50 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:36:50.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:36:50 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 7740 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:36:50.843 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:36:50 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:36:50.843 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:36:50 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:36:50.844 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:36:50 smithi115 ceph-mon[78593]: pgmap v4224: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:36:50.844 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:36:50 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:36:50.844 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:36:50 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 7740 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:36:51.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:36:50 smithi115 conmon[90393]: 2022-05-14T01:36:50.842+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:36:51.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:36:51 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:36:51.816 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:36:51 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:36:52.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:36:51 smithi115 conmon[90393]: 2022-05-14T01:36:51.816+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:36:52.782 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:36:52 smithi115 ceph-mon[78593]: pgmap v4225: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:36:52.783 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:36:52 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:36:52.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:36:52 smithi047 ceph-mon[77147]: pgmap v4225: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:36:52.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:36:52 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:36:53.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:36:52 smithi115 conmon[90393]: 2022-05-14T01:36:52.782+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:36:53.782 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:36:53 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:36:53.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:36:53 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:36:54.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:36:53 smithi115 conmon[90393]: 2022-05-14T01:36:53.782+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:36:54.783 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:36:54 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:36:54.784 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:36:54 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:36:54.784 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:36:54 smithi115 ceph-mon[78593]: pgmap v4226: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:36:54.784 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:36:54 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:36:54.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:36:54 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:36:54.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:36:54 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:36:54.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:36:54 smithi047 ceph-mon[77147]: pgmap v4226: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:36:54.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:36:54 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:36:55.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:36:54 smithi115 conmon[90393]: 2022-05-14T01:36:54.783+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:36:55.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:36:55 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:36:55.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:36:55 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 7745 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:36:55.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:36:55 smithi115 conmon[90393]: 2022-05-14T01:36:55.763+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:36:55.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:36:55 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:36:55.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:36:55 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 7745 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:36:56.788 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:36:56 smithi115 ceph-mon[78593]: pgmap v4227: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:36:56.788 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:36:56 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:36:56.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:36:56 smithi047 ceph-mon[77147]: pgmap v4227: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:36:56.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:36:56 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:36:57.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:36:56 smithi115 conmon[90393]: 2022-05-14T01:36:56.786+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:36:57.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:36:57 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:36:57.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:36:57 smithi115 conmon[90393]: 2022-05-14T01:36:57.764+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:36:57.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:36:57 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:36:58.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:36:58 smithi115 conmon[90393]: 2022-05-14T01:36:58.773+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:36:58.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:36:58 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:36:58.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:36:58 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:36:58.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:36:58 smithi115 ceph-mon[78593]: pgmap v4228: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:36:58.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:36:58 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:36:59.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:36:58 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:36:59.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:36:58 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:36:59.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:36:58 smithi047 ceph-mon[77147]: pgmap v4228: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:36:59.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:36:58 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:36:59.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:36:59 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:36:59.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:36:59 smithi115 conmon[90393]: 2022-05-14T01:36:59.746+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:37:00.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:36:59 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:37:00.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:37:00 smithi115 conmon[90393]: 2022-05-14T01:37:00.705+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:37:00.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:37:00 smithi115 ceph-mon[78593]: pgmap v4229: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:37:00.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:37:00 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:37:00.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:37:00 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 7750 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:37:01.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:37:00 smithi047 ceph-mon[77147]: pgmap v4229: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:37:01.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:37:00 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:37:01.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:37:00 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 7750 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:37:01.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:37:01 smithi115 conmon[90393]: 2022-05-14T01:37:01.742+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:37:01.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:37:01 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:37:02.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:37:01 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:37:02.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:37:02 smithi115 conmon[90393]: 2022-05-14T01:37:02.735+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:37:02.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:37:02 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:37:02.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:37:02 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:37:02.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:37:02 smithi115 ceph-mon[78593]: pgmap v4230: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:37:02.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:37:02 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:37:03.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:37:02 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:37:03.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:37:02 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:37:03.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:37:02 smithi047 ceph-mon[77147]: pgmap v4230: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:37:03.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:37:02 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:37:03.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:37:03 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:37:03.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:37:03 smithi115 conmon[90393]: 2022-05-14T01:37:03.777+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:37:04.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:37:03 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:37:04.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:37:04 smithi115 conmon[90393]: 2022-05-14T01:37:04.785+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:37:04.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:37:04 smithi115 ceph-mon[78593]: pgmap v4231: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:37:04.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:37:04 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:37:05.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:37:04 smithi047 ceph-mon[77147]: pgmap v4231: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:37:05.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:37:04 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:37:05.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:37:05 smithi115 conmon[90393]: 2022-05-14T01:37:05.762+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:37:05.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:37:05 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:37:05.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:37:05 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 7755 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:37:06.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:37:05 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:37:06.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:37:05 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 7755 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:37:06.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:37:06 smithi115 conmon[90393]: 2022-05-14T01:37:06.715+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:37:06.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:37:06 smithi115 ceph-mon[78593]: pgmap v4232: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:37:06.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:37:06 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:37:07.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:37:06 smithi047 ceph-mon[77147]: pgmap v4232: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:37:07.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:37:06 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:37:07.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:37:07 smithi115 conmon[90393]: 2022-05-14T01:37:07.765+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:37:07.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:37:07 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:37:08.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:37:07 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:37:08.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:37:08 smithi115 conmon[90393]: 2022-05-14T01:37:08.726+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:37:08.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:37:08 smithi115 ceph-mon[78593]: pgmap v4233: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:37:08.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:37:08 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:37:09.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:37:08 smithi047 ceph-mon[77147]: pgmap v4233: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:37:09.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:37:08 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:37:09.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:37:09 smithi115 conmon[90393]: 2022-05-14T01:37:09.775+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:37:09.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:37:09 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:37:10.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:37:09 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:37:10.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:37:10 smithi115 conmon[90393]: 2022-05-14T01:37:10.802+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:37:10.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:37:10 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:37:10.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:37:10 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:37:10.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:37:10 smithi115 ceph-mon[78593]: pgmap v4234: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:37:10.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:37:10 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:37:10.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:37:10 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 7760 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:37:11.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:37:10 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:37:11.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:37:10 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:37:11.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:37:10 smithi047 ceph-mon[77147]: pgmap v4234: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:37:11.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:37:10 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:37:11.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:37:10 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 7760 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:37:11.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:37:11 smithi115 conmon[90393]: 2022-05-14T01:37:11.774+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:37:11.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:37:11 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:37:12.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:37:11 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:37:12.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:37:12 smithi115 conmon[90393]: 2022-05-14T01:37:12.757+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:37:12.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:37:12 smithi115 ceph-mon[78593]: pgmap v4235: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:37:12.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:37:12 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:37:13.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:37:12 smithi047 ceph-mon[77147]: pgmap v4235: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:37:13.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:37:12 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:37:13.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:37:13 smithi115 conmon[90393]: 2022-05-14T01:37:13.747+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:37:13.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:37:13 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:37:14.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:37:13 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:37:15.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:37:14 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:37:15.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:37:14 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:37:15.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:37:14 smithi047 ceph-mon[77147]: pgmap v4236: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:37:15.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:37:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T01:37:15.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:37:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T01:37:15.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:37:14 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:37:15.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:37:14 smithi115 conmon[90393]: 2022-05-14T01:37:14.705+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:37:15.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:37:14 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:37:15.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:37:14 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:37:15.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:37:14 smithi115 ceph-mon[78593]: pgmap v4236: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:37:15.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:37:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T01:37:15.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:37:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T01:37:15.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:37:14 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:37:16.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:37:15 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:37:16.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:37:15 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 7765 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:37:16.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:37:15 smithi115 conmon[90393]: 2022-05-14T01:37:15.659+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:37:16.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:37:15 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:37:16.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:37:15 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 7765 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:37:17.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:37:16 smithi047 ceph-mon[77147]: pgmap v4237: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:37:17.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:37:16 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:37:17.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:37:16 smithi115 conmon[90393]: 2022-05-14T01:37:16.692+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:37:17.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:37:16 smithi115 ceph-mon[78593]: pgmap v4237: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:37:17.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:37:16 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:37:18.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:37:17 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:37:18.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:37:17 smithi115 conmon[90393]: 2022-05-14T01:37:17.683+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:37:18.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:37:17 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:37:19.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:37:18 smithi047 ceph-mon[77147]: pgmap v4238: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:37:19.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:37:18 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:37:19.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:37:18 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:37:19.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:37:18 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:37:19.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:37:18 smithi115 conmon[90393]: 2022-05-14T01:37:18.692+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:37:19.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:37:18 smithi115 ceph-mon[78593]: pgmap v4238: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:37:19.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:37:18 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:37:19.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:37:18 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:37:19.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:37:18 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:37:20.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:37:19 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:37:20.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:37:19 smithi115 conmon[90393]: 2022-05-14T01:37:19.659+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:37:20.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:37:19 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:37:21.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:37:20 smithi047 ceph-mon[77147]: pgmap v4239: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:37:21.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:37:20 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:37:21.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:37:20 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 7775 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:37:21.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:37:20 smithi115 conmon[90393]: 2022-05-14T01:37:20.616+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:37:21.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:37:20 smithi115 ceph-mon[78593]: pgmap v4239: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:37:21.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:37:20 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:37:21.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:37:20 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 7775 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:37:22.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:37:21 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:37:22.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:37:21 smithi115 conmon[90393]: 2022-05-14T01:37:21.642+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:37:22.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:37:21 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:37:23.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:37:22 smithi047 ceph-mon[77147]: pgmap v4240: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:37:23.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:37:22 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:37:23.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:37:22 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:37:23.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:37:22 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:37:23.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:37:22 smithi115 conmon[90393]: 2022-05-14T01:37:22.669+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:37:23.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:37:22 smithi115 ceph-mon[78593]: pgmap v4240: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:37:23.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:37:22 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:37:23.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:37:22 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:37:23.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:37:22 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:37:24.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:37:23 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:37:24.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:37:23 smithi115 conmon[90393]: 2022-05-14T01:37:23.712+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:37:24.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:37:23 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:37:25.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:37:24 smithi047 ceph-mon[77147]: pgmap v4241: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:37:25.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:37:24 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:37:25.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:37:24 smithi115 conmon[90393]: 2022-05-14T01:37:24.684+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:37:25.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:37:24 smithi115 ceph-mon[78593]: pgmap v4241: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:37:25.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:37:24 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:37:25.939 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:37:25 smithi115 conmon[90393]: 2022-05-14T01:37:25.636+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:37:26.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:37:25 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:37:26.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:37:25 smithi047 ceph-mon[77147]: pgmap v4242: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:37:26.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:37:25 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:37:26.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:37:25 smithi115 ceph-mon[78593]: pgmap v4242: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:37:26.941 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:37:26 smithi115 conmon[90393]: 2022-05-14T01:37:26.641+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:37:27.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:37:26 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:37:27.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:37:26 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 7780 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:37:27.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:37:26 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:37:27.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:37:26 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 7780 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:37:27.945 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:37:27 smithi115 conmon[90393]: 2022-05-14T01:37:27.639+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:37:28.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:37:27 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:37:28.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:37:27 smithi047 ceph-mon[77147]: pgmap v4243: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:37:28.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:37:27 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:37:28.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:37:27 smithi115 ceph-mon[78593]: pgmap v4243: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:37:28.951 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:37:28 smithi115 conmon[90393]: 2022-05-14T01:37:28.665+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:37:29.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:37:28 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:37:29.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:37:28 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:37:29.955 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:37:29 smithi115 conmon[90393]: 2022-05-14T01:37:29.644+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:37:30.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:37:29 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:37:30.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:37:29 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:37:30.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:37:29 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:37:30.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:37:29 smithi047 ceph-mon[77147]: pgmap v4244: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:37:30.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:37:29 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:37:30.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:37:29 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:37:30.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:37:29 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:37:30.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:37:29 smithi115 ceph-mon[78593]: pgmap v4244: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:37:30.957 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:37:30 smithi115 conmon[90393]: 2022-05-14T01:37:30.612+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:37:31.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:37:30 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:37:31.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:37:30 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:37:31.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:37:31 smithi115 conmon[90393]: 2022-05-14T01:37:31.592+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:37:32.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:37:31 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:37:32.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:37:31 smithi047 ceph-mon[77147]: pgmap v4245: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:37:32.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:37:31 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T01:37:32.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:37:31 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T01:37:32.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:37:31 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T01:37:32.298 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:37:31 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:37:32.299 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:37:31 smithi115 ceph-mon[78593]: pgmap v4245: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:37:32.299 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:37:31 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T01:37:32.299 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:37:31 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T01:37:32.299 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:37:31 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T01:37:32.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:37:32 smithi115 conmon[90393]: 2022-05-14T01:37:32.604+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:37:33.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:37:32 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:37:33.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:37:32 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:37:33.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:37:33 smithi115 conmon[90393]: 2022-05-14T01:37:33.582+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:37:34.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:37:33 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:37:34.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:37:33 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:37:34.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:37:33 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:37:34.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:37:33 smithi047 ceph-mon[77147]: pgmap v4246: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:37:34.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:37:33 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:37:34.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:37:33 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:37:34.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:37:33 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:37:34.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:37:33 smithi115 ceph-mon[78593]: pgmap v4246: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:37:34.973 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:37:34 smithi115 conmon[90393]: 2022-05-14T01:37:34.629+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:37:35.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:37:34 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:37:35.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:37:34 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 7785 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:37:35.292 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:37:34 smithi047 conmon[78521]: 2022-05-14T01:37:34.896+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:37:35.292 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:37:34 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T01:37:35.292 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:37:34 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:37:35.292 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:37:34 smithi047 conmon[78521]: yield 2022-05-14T01:37:35.293 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:37:34 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:37:35.293 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:37:34 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:37:35.293 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:37:34 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:37:35.293 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:37:34 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T01:37:35.293 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:37:34 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:37:35.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:37:34 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T01:37:35.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:37:34 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:37:35.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:37:34 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T01:37:35.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:37:34 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:37:35.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:37:34 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T01:37:35.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:37:34 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:37:35.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:37:34 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T01:37:35.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:37:34 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:37:35.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:37:34 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:37:35.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:37:34 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:37:35.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:37:34 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:37:35.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:37:34 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 7785 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:37:35.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:37:35 smithi115 conmon[90393]: 2022-05-14T01:37:35.596+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:37:36.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:37:35 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:37:36.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:37:35 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:37:36.291 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T01:37:36.292 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:37:36.292 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T01:37:36.292 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:37:36.292 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:37:36.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:37:36.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T01:37:36.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:37:36.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T01:37:36.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:37:36.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T01:37:36.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:37:36.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T01:37:36.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:37:36.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T01:37:36.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:37:36.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:37:36.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:37:36.296 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:37:35 smithi047 ceph-mon[77147]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T01:37:36.296 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:37:35 smithi047 ceph-mon[77147]: pgmap v4247: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:37:36.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:37:35 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:37:36.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:37:35 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:37:36.358 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T01:37:36.359 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:37:36.359 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T01:37:36.359 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:37:36.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:37:36.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:37:36.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T01:37:36.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:37:36.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T01:37:36.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:37:36.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T01:37:36.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:37:36.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T01:37:36.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:37:36.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T01:37:36.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:37:36.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:37:36.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:37:36.363 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:37:35 smithi115 ceph-mon[78593]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T01:37:36.363 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:37:35 smithi115 ceph-mon[78593]: pgmap v4247: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:37:36.980 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:37:36 smithi115 conmon[90393]: 2022-05-14T01:37:36.625+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:37:37.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:37:36 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:37:37.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:37:36 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:37:37.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:37:37 smithi115 conmon[90393]: 2022-05-14T01:37:37.597+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:37:38.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:37:37 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:37:38.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:37:37 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:37:38.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:37:37 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:37:38.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:37:37 smithi047 ceph-mon[77147]: pgmap v4248: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:37:38.293 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:37:37 smithi047 conmon[78521]: 2022-05-14T01:37:37.968+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:37:38.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:37:37 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T01:37:38.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:37:37 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:37:38.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:37:37 smithi047 conmon[78521]: yield 2022-05-14T01:37:38.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:37:37 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:37:38.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:37:37 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:37:38.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:37:37 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:37:38.297 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:37:37 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T01:37:38.297 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:37:37 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:37:38.297 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:37:37 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T01:37:38.298 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:37:37 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:37:38.298 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:37:37 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T01:37:38.298 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:37:37 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:37:38.299 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:37:37 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T01:37:38.299 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:37:37 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:37:38.299 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:37:37 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T01:37:38.299 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:37:37 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:37:38.299 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:37:37 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:37:38.300 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:37:37 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:37:38.300 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:37:37 smithi047 conmon[78521]: 2022-05-14T01:37:37.969+0000 7f5344f82700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T01:37:38.300 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:37:37 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T01:37:38.300 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:37:37 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:37:38.301 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:37:37 smithi047 conmon[78521]: yield 2022-05-14T01:37:38.301 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:37:37 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:37:38.301 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:37:37 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:37:38.301 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:37:37 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:37:38.302 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:37:37 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T01:37:38.302 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:37:37 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:37:38.302 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:37:37 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T01:37:38.302 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:37:37 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:37:38.302 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:37:37 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T01:37:38.303 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:37:37 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:37:38.303 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:37:37 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T01:37:38.303 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:37:37 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:37:38.303 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:37:37 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T01:37:38.304 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:37:37 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:37:38.304 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:37:37 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:37:38.304 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:37:37 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:37:38.304 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:37:37 smithi047 conmon[78521]: 2022-05-14T01:37:38.304 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:37:37 smithi047 conmon[78521]: During handling of the above exception, another exception occurred: 2022-05-14T01:37:38.305 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:37:37 smithi047 conmon[78521]: 2022-05-14T01:37:38.305 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:37:37 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T01:37:38.305 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:37:37 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T01:37:38.306 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:37:37 smithi047 conmon[78521]: return f(*arg) 2022-05-14T01:37:38.306 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:37:37 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T01:37:38.306 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:37:37 smithi047 conmon[78521]: self._write_client_files(client_files, host) 2022-05-14T01:37:38.306 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:37:37 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T01:37:38.307 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:37:37 smithi047 conmon[78521]: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T01:37:38.307 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:37:37 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T01:37:38.307 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:37:37 smithi047 conmon[78521]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T01:37:38.307 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:37:37 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T01:37:38.308 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:37:37 smithi047 conmon[78521]: return self.event_loop.get_result(coro) 2022-05-14T01:37:38.308 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:37:37 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T01:37:38.308 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:37:37 smithi047 conmon[78521]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T01:37:38.308 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:37:37 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T01:37:38.309 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:37:37 smithi047 conmon[78521]: return self.__get_result() 2022-05-14T01:37:38.309 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:37:37 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T01:37:38.309 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:37:37 smithi047 conmon[78521]: raise self._exception 2022-05-14T01:37:38.309 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:37:37 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T01:37:38.309 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:37:37 smithi047 conmon[78521]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T01:37:38.310 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:37:37 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T01:37:38.310 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:37:37 smithi047 conmon[78521]: conn = await self._remote_connection(host, addr) 2022-05-14T01:37:38.310 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:37:37 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T01:37:38.310 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:37:37 smithi047 conmon[78521]: raise 2022-05-14T01:37:38.311 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:37:37 smithi047 conmon[78521]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T01:37:38.311 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:37:37 smithi047 conmon[78521]: self.gen.throw(type, value, traceback) 2022-05-14T01:37:38.311 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:37:37 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T01:37:38.311 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:37:37 smithi047 conmon[78521]: raise OrchestratorError(msg) 2022-05-14T01:37:38.311 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:37:37 smithi047 conmon[78521]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:37:38.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:37:37 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:37:38.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:37:37 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:37:38.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:37:37 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:37:38.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:37:37 smithi115 ceph-mon[78593]: pgmap v4248: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:37:38.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:37:38 smithi115 conmon[90393]: 2022-05-14T01:37:38.591+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:37:39.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:37:38 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:37:39.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:37:38 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:37:39.292 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T01:37:39.292 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:37:39.292 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T01:37:39.292 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:37:39.292 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:37:39.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:37:39.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T01:37:39.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:37:39.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T01:37:39.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:37:39.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T01:37:39.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:37:39.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T01:37:39.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:37:39.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T01:37:39.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:37:39.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:37:39.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:37:39.296 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:37:38 smithi047 ceph-mon[77147]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T01:37:39.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T01:37:39.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:37:39.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T01:37:39.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:37:39.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:37:39.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:37:39.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T01:37:39.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:37:39.298 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T01:37:39.298 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:37:39.298 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T01:37:39.298 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:37:39.298 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T01:37:39.299 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:37:39.299 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T01:37:39.299 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:37:39.299 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:37:39.299 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:37:39.300 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T01:37:39.300 INFO:journalctl@ceph.mon.b.smithi047.stdout: During handling of the above exception, another exception occurred: 2022-05-14T01:37:39.300 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T01:37:39.300 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T01:37:39.301 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T01:37:39.301 INFO:journalctl@ceph.mon.b.smithi047.stdout: return f(*arg) 2022-05-14T01:37:39.301 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T01:37:39.301 INFO:journalctl@ceph.mon.b.smithi047.stdout: self._write_client_files(client_files, host) 2022-05-14T01:37:39.302 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T01:37:39.302 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T01:37:39.302 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T01:37:39.302 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T01:37:39.302 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T01:37:39.303 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.event_loop.get_result(coro) 2022-05-14T01:37:39.303 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T01:37:39.303 INFO:journalctl@ceph.mon.b.smithi047.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T01:37:39.303 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T01:37:39.303 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.__get_result() 2022-05-14T01:37:39.304 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T01:37:39.304 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise self._exception 2022-05-14T01:37:39.304 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T01:37:39.304 INFO:journalctl@ceph.mon.b.smithi047.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T01:37:39.305 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T01:37:39.305 INFO:journalctl@ceph.mon.b.smithi047.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T01:37:39.305 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T01:37:39.305 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise 2022-05-14T01:37:39.305 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T01:37:39.306 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.gen.throw(type, value, traceback) 2022-05-14T01:37:39.306 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T01:37:39.306 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OrchestratorError(msg) 2022-05-14T01:37:39.306 INFO:journalctl@ceph.mon.b.smithi047.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:37:39.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:37:38 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:37:39.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:37:38 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:37:39.358 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T01:37:39.359 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:37:39.359 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T01:37:39.359 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:37:39.359 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:37:39.359 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:37:39.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T01:37:39.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:37:39.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T01:37:39.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:37:39.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T01:37:39.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:37:39.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T01:37:39.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:37:39.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T01:37:39.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:37:39.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:37:39.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:37:39.362 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:37:38 smithi115 ceph-mon[78593]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T01:37:39.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T01:37:39.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:37:39.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T01:37:39.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:37:39.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:37:39.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:37:39.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T01:37:39.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:37:39.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T01:37:39.365 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:37:39.365 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T01:37:39.365 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:37:39.365 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T01:37:39.366 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:37:39.366 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T01:37:39.366 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:37:39.367 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:37:39.367 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:37:39.367 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T01:37:39.368 INFO:journalctl@ceph.mon.c.smithi115.stdout: During handling of the above exception, another exception occurred: 2022-05-14T01:37:39.368 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T01:37:39.368 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T01:37:39.369 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T01:37:39.369 INFO:journalctl@ceph.mon.c.smithi115.stdout: return f(*arg) 2022-05-14T01:37:39.369 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T01:37:39.370 INFO:journalctl@ceph.mon.c.smithi115.stdout: self._write_client_files(client_files, host) 2022-05-14T01:37:39.370 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T01:37:39.370 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T01:37:39.370 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T01:37:39.370 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T01:37:39.371 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T01:37:39.371 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.event_loop.get_result(coro) 2022-05-14T01:37:39.371 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T01:37:39.371 INFO:journalctl@ceph.mon.c.smithi115.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T01:37:39.372 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T01:37:39.372 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.__get_result() 2022-05-14T01:37:39.372 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T01:37:39.372 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise self._exception 2022-05-14T01:37:39.372 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T01:37:39.373 INFO:journalctl@ceph.mon.c.smithi115.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T01:37:39.373 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T01:37:39.373 INFO:journalctl@ceph.mon.c.smithi115.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T01:37:39.373 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T01:37:39.374 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise 2022-05-14T01:37:39.374 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T01:37:39.374 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.gen.throw(type, value, traceback) 2022-05-14T01:37:39.374 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T01:37:39.374 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OrchestratorError(msg) 2022-05-14T01:37:39.375 INFO:journalctl@ceph.mon.c.smithi115.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:37:39.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:37:39 smithi115 conmon[90393]: 2022-05-14T01:37:39.544+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:37:40.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:37:39 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:37:40.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:37:39 smithi047 ceph-mon[77147]: pgmap v4249: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:37:40.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:37:39 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 7790 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:37:40.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:37:39 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:37:40.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:37:39 smithi115 ceph-mon[78593]: pgmap v4249: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:37:40.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:37:39 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 7790 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:37:40.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:37:40 smithi115 conmon[90393]: 2022-05-14T01:37:40.555+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:37:41.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:37:41 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:37:41.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:37:41 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:37:41.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:37:41 smithi115 conmon[90393]: 2022-05-14T01:37:41.543+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:37:42.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:37:42 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:37:42.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:37:42 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:37:42.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:37:42 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:37:42.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:37:42 smithi047 ceph-mon[77147]: pgmap v4250: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:37:42.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:37:42 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:37:42.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:37:42 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:37:42.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:37:42 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:37:42.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:37:42 smithi115 ceph-mon[78593]: pgmap v4250: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:37:42.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:37:42 smithi115 conmon[90393]: 2022-05-14T01:37:42.504+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:37:43.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:37:43 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:37:43.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:37:43 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:37:43.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:37:43 smithi115 conmon[90393]: 2022-05-14T01:37:43.490+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:37:44.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:37:44 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:37:44.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:37:44 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:37:44.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:37:44 smithi047 ceph-mon[77147]: pgmap v4251: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:37:44.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:37:44 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:37:44.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:37:44 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:37:44.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:37:44 smithi115 ceph-mon[78593]: pgmap v4251: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:37:44.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:37:44 smithi115 conmon[90393]: 2022-05-14T01:37:44.457+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:37:45.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:37:45 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:37:45.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:37:45 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 7795 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:37:45.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:37:45 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:37:45.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:37:45 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 7795 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:37:45.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:37:45 smithi115 conmon[90393]: 2022-05-14T01:37:45.500+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:37:46.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:37:46 smithi047 ceph-mon[77147]: pgmap v4252: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:37:46.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:37:46 smithi115 ceph-mon[78593]: pgmap v4252: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:37:46.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:37:46 smithi115 conmon[90393]: 2022-05-14T01:37:46.464+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:37:47.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:37:47 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:37:47.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:37:47 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:37:47.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:37:47 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:37:47.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:37:47 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:37:47.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:37:47 smithi115 conmon[90393]: 2022-05-14T01:37:47.472+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:37:48.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:37:48 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:37:48.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:37:48 smithi047 ceph-mon[77147]: pgmap v4253: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:37:48.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:37:48 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:37:48.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:37:48 smithi115 ceph-mon[78593]: pgmap v4253: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:37:48.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:37:48 smithi115 conmon[90393]: 2022-05-14T01:37:48.474+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:37:49.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:37:49 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:37:49.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:37:49 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:37:49.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:37:49 smithi115 conmon[90393]: 2022-05-14T01:37:49.513+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:37:50.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:37:50 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:37:50.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:37:50 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:37:50.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:37:50 smithi047 ceph-mon[77147]: pgmap v4254: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:37:50.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:37:50 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 7800 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:37:50.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:37:50 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:37:50.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:37:50 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:37:50.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:37:50 smithi115 ceph-mon[78593]: pgmap v4254: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:37:50.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:37:50 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 7800 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:37:50.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:37:50 smithi115 conmon[90393]: 2022-05-14T01:37:50.480+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:37:51.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:37:51 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:37:51.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:37:51 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:37:51.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:37:51 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:37:51.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:37:51 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:37:51.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:37:51 smithi115 conmon[90393]: 2022-05-14T01:37:51.485+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:37:52.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:37:52 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:37:52.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:37:52 smithi047 ceph-mon[77147]: pgmap v4255: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:37:52.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:37:52 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:37:52.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:37:52 smithi115 ceph-mon[78593]: pgmap v4255: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:37:52.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:37:52 smithi115 conmon[90393]: 2022-05-14T01:37:52.485+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:37:53.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:37:53 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:37:53.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:37:53 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:37:53.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:37:53 smithi115 conmon[90393]: 2022-05-14T01:37:53.484+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:37:54.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:37:54 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:37:54.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:37:54 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:37:54.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:37:54 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:37:54.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:37:54 smithi115 ceph-mon[78593]: pgmap v4256: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:37:54.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:37:54 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:37:54.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:37:54 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:37:54.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:37:54 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:37:54.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:37:54 smithi047 ceph-mon[77147]: pgmap v4256: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:37:54.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:37:54 smithi115 conmon[90393]: 2022-05-14T01:37:54.490+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:37:55.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:37:55 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:37:55.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:37:55 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 7805 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:37:55.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:37:55 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:37:55.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:37:55 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 7805 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:37:55.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:37:55 smithi115 conmon[90393]: 2022-05-14T01:37:55.541+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:37:56.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:37:56 smithi115 ceph-mon[78593]: pgmap v4257: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:37:56.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:37:56 smithi047 ceph-mon[77147]: pgmap v4257: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:37:56.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:37:56 smithi115 conmon[90393]: 2022-05-14T01:37:56.504+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:37:57.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:37:57 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:37:57.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:37:57 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:37:57.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:37:57 smithi115 conmon[90393]: 2022-05-14T01:37:57.482+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:37:58.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:37:58 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:37:58.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:37:58 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:37:58.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:37:58 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:37:58.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:37:58 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:37:58.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:37:58 smithi115 ceph-mon[78593]: pgmap v4258: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:37:58.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:37:58 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:37:58.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:37:58 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:37:58.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:37:58 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:37:58.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:37:58 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:37:58.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:37:58 smithi047 ceph-mon[77147]: pgmap v4258: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:37:58.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:37:58 smithi115 conmon[90393]: 2022-05-14T01:37:58.435+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:37:59.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:37:59 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:37:59.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:37:59 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:37:59.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:37:59 smithi115 conmon[90393]: 2022-05-14T01:37:59.386+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:38:00.351 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:38:00 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:38:00.351 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:38:00 smithi115 ceph-mon[78593]: pgmap v4259: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:38:00.352 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:38:00 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 7810 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:38:00.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:38:00 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:38:00.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:38:00 smithi047 ceph-mon[77147]: pgmap v4259: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:38:00.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:38:00 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 7810 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:38:00.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:38:00 smithi115 conmon[90393]: 2022-05-14T01:38:00.350+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:38:01.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:38:01 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:38:01.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:38:01 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:38:01.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:38:01 smithi115 conmon[90393]: 2022-05-14T01:38:01.364+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:38:02.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:38:02 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:38:02.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:38:02 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:38:02.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:38:02 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:38:02.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:38:02 smithi115 ceph-mon[78593]: pgmap v4260: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:38:02.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:38:02 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:38:02.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:38:02 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:38:02.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:38:02 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:38:02.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:38:02 smithi047 ceph-mon[77147]: pgmap v4260: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:38:02.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:38:02 smithi115 conmon[90393]: 2022-05-14T01:38:02.378+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:38:03.351 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:38:03 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:38:03.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:38:03 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:38:03.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:38:03 smithi115 conmon[90393]: 2022-05-14T01:38:03.350+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:38:04.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:38:04 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:38:04.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:38:04 smithi115 ceph-mon[78593]: pgmap v4261: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:38:04.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:38:04 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:38:04.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:38:04 smithi047 ceph-mon[77147]: pgmap v4261: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:38:04.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:38:04 smithi115 conmon[90393]: 2022-05-14T01:38:04.397+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:38:05.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:38:05 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:38:05.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:38:05 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 7815 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:38:05.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:38:05 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:38:05.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:38:05 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 7815 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:38:05.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:38:05 smithi115 conmon[90393]: 2022-05-14T01:38:05.427+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:38:06.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:38:06 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:38:06.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:38:06 smithi115 ceph-mon[78593]: pgmap v4262: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:38:06.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:38:06 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:38:06.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:38:06 smithi047 ceph-mon[77147]: pgmap v4262: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:38:06.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:38:06 smithi115 conmon[90393]: 2022-05-14T01:38:06.391+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:38:07.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:38:07 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:38:07.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:38:07 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:38:07.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:38:07 smithi115 conmon[90393]: 2022-05-14T01:38:07.357+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:38:08.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:38:08 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:38:08.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:38:08 smithi115 ceph-mon[78593]: pgmap v4263: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:38:08.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:38:08 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:38:08.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:38:08 smithi047 ceph-mon[77147]: pgmap v4263: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:38:08.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:38:08 smithi115 conmon[90393]: 2022-05-14T01:38:08.364+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:38:09.365 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:38:09 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:38:09.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:38:09 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:38:09.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:38:09 smithi115 conmon[90393]: 2022-05-14T01:38:09.363+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:38:10.397 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:38:10 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:38:10.397 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:38:10 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:38:10.397 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:38:10 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:38:10.398 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:38:10 smithi115 ceph-mon[78593]: pgmap v4264: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:38:10.398 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:38:10 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 7820 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:38:10.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:38:10 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:38:10.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:38:10 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:38:10.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:38:10 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:38:10.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:38:10 smithi047 ceph-mon[77147]: pgmap v4264: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:38:10.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:38:10 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 7820 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:38:10.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:38:10 smithi115 conmon[90393]: 2022-05-14T01:38:10.395+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:38:11.415 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:38:11 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:38:11.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:38:11 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:38:11.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:38:11 smithi115 conmon[90393]: 2022-05-14T01:38:11.414+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:38:12.445 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:38:12 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:38:12.445 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:38:12 smithi115 ceph-mon[78593]: pgmap v4265: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:38:12.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:38:12 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:38:12.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:38:12 smithi047 ceph-mon[77147]: pgmap v4265: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:38:12.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:38:12 smithi115 conmon[90393]: 2022-05-14T01:38:12.443+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:38:13.470 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:38:13 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:38:13.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:38:13 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:38:13.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:38:13 smithi115 conmon[90393]: 2022-05-14T01:38:13.469+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:38:14.485 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:38:14 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:38:14.485 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:38:14 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:38:14.485 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:38:14 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:38:14.486 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:38:14 smithi115 ceph-mon[78593]: pgmap v4266: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:38:14.486 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:38:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T01:38:14.486 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:38:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T01:38:14.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:38:14 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:38:14.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:38:14 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:38:14.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:38:14 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:38:14.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:38:14 smithi047 ceph-mon[77147]: pgmap v4266: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:38:14.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:38:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T01:38:14.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:38:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T01:38:14.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:38:14 smithi115 conmon[90393]: 2022-05-14T01:38:14.483+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:38:15.521 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:38:15 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:38:15.522 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:38:15 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 7825 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:38:15.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:38:15 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:38:15.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:38:15 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 7825 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:38:15.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:38:15 smithi115 conmon[90393]: 2022-05-14T01:38:15.520+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:38:16.498 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:38:16 smithi115 ceph-mon[78593]: pgmap v4267: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:38:16.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:38:16 smithi047 ceph-mon[77147]: pgmap v4267: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:38:16.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:38:16 smithi115 conmon[90393]: 2022-05-14T01:38:16.497+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:38:17.499 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:38:17 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:38:17.499 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:38:17 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:38:17.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:38:17 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:38:17.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:38:17 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:38:17.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:38:17 smithi115 conmon[90393]: 2022-05-14T01:38:17.498+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:38:18.527 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:38:18 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:38:18.528 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:38:18 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:38:18.528 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:38:18 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:38:18.528 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:38:18 smithi115 ceph-mon[78593]: pgmap v4268: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:38:18.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:38:18 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:38:18.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:38:18 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:38:18.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:38:18 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:38:18.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:38:18 smithi047 ceph-mon[77147]: pgmap v4268: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:38:18.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:38:18 smithi115 conmon[90393]: 2022-05-14T01:38:18.525+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:38:19.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:38:19 smithi115 conmon[90393]: 2022-05-14T01:38:19.491+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:38:19.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:38:19 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:38:19.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:38:19 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:38:20.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:38:19 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:38:20.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:38:19 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:38:20.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:38:20 smithi115 conmon[90393]: 2022-05-14T01:38:20.541+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:38:20.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:38:20 smithi115 ceph-mon[78593]: pgmap v4269: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:38:20.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:38:20 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 7830 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:38:21.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:38:20 smithi047 ceph-mon[77147]: pgmap v4269: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:38:21.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:38:20 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 7830 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:38:21.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:38:21 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:38:21.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:38:21 smithi115 conmon[90393]: 2022-05-14T01:38:21.582+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:38:22.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:38:21 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:38:22.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:38:22 smithi115 conmon[90393]: 2022-05-14T01:38:22.541+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:38:22.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:38:22 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:38:22.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:38:22 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:38:22.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:38:22 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:38:22.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:38:22 smithi115 ceph-mon[78593]: pgmap v4270: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:38:23.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:38:22 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:38:23.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:38:22 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:38:23.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:38:22 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:38:23.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:38:22 smithi047 ceph-mon[77147]: pgmap v4270: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:38:23.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:38:23 smithi115 conmon[90393]: 2022-05-14T01:38:23.554+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:38:23.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:38:23 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:38:24.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:38:23 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:38:24.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:38:24 smithi115 conmon[90393]: 2022-05-14T01:38:24.576+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:38:24.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:38:24 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:38:24.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:38:24 smithi115 ceph-mon[78593]: pgmap v4271: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:38:25.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:38:24 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:38:25.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:38:24 smithi047 ceph-mon[77147]: pgmap v4271: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:38:25.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:38:25 smithi115 conmon[90393]: 2022-05-14T01:38:25.627+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:38:25.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:38:25 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:38:25.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:38:25 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 7835 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:38:26.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:38:25 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:38:26.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:38:25 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 7835 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:38:26.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:38:26 smithi115 conmon[90393]: 2022-05-14T01:38:26.638+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:38:26.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:38:26 smithi115 ceph-mon[78593]: pgmap v4272: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:38:26.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:38:26 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:38:27.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:38:26 smithi047 ceph-mon[77147]: pgmap v4272: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:38:27.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:38:26 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:38:27.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:38:27 smithi115 conmon[90393]: 2022-05-14T01:38:27.657+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:38:27.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:38:27 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:38:28.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:38:27 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:38:28.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:38:28 smithi115 conmon[90393]: 2022-05-14T01:38:28.626+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:38:28.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:38:28 smithi115 ceph-mon[78593]: pgmap v4273: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:38:28.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:38:28 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:38:29.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:38:28 smithi047 ceph-mon[77147]: pgmap v4273: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:38:29.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:38:28 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:38:29.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:38:29 smithi115 conmon[90393]: 2022-05-14T01:38:29.600+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:38:29.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:38:29 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:38:30.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:38:29 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:38:30.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:38:30 smithi115 conmon[90393]: 2022-05-14T01:38:30.598+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:38:30.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:38:30 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:38:30.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:38:30 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:38:30.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:38:30 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:38:30.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:38:30 smithi115 ceph-mon[78593]: pgmap v4274: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:38:30.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:38:30 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 7840 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:38:31.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:38:30 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:38:31.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:38:30 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:38:31.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:38:30 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:38:31.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:38:30 smithi047 ceph-mon[77147]: pgmap v4274: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:38:31.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:38:30 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 7840 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:38:31.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:38:31 smithi115 conmon[90393]: 2022-05-14T01:38:31.552+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:38:31.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:38:31 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:38:32.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:38:31 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:38:32.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:38:32 smithi115 conmon[90393]: 2022-05-14T01:38:32.583+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:38:32.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:38:32 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:38:32.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:38:32 smithi115 ceph-mon[78593]: pgmap v4275: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:38:33.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:38:32 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:38:33.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:38:32 smithi047 ceph-mon[77147]: pgmap v4275: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:38:34.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:38:33 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:38:34.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:38:33 smithi115 conmon[90393]: 2022-05-14T01:38:33.625+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:38:34.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:38:33 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:38:35.047 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:38:34 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:38:35.048 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:38:34 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:38:35.048 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:38:34 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:38:35.048 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:38:34 smithi047 ceph-mon[77147]: pgmap v4276: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:38:35.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:38:34 smithi115 conmon[90393]: 2022-05-14T01:38:34.658+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:38:35.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:38:34 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:38:35.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:38:34 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:38:35.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:38:34 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:38:35.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:38:34 smithi115 ceph-mon[78593]: pgmap v4276: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:38:36.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:38:35 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:38:36.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:38:35 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 7845 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:38:36.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:38:35 smithi115 conmon[90393]: 2022-05-14T01:38:35.650+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:38:36.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:38:35 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:38:36.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:38:35 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 7845 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:38:37.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:38:36 smithi047 ceph-mon[77147]: pgmap v4277: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:38:37.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:38:36 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:38:37.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:38:36 smithi115 conmon[90393]: 2022-05-14T01:38:36.645+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:38:37.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:38:36 smithi115 ceph-mon[78593]: pgmap v4277: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:38:37.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:38:36 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:38:38.014 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:38:37 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:38:38.015 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:38:37 smithi115 conmon[90393]: 2022-05-14T01:38:37.618+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:38:38.016 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:38:37 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:38:38.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:38:38 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:38:38.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:38:38 smithi047 ceph-mon[77147]: pgmap v4278: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:38:38.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:38:38 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:38:38.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:38:38 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:38:38.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:38:38 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T01:38:38.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:38:38 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T01:38:38.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:38:38 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T01:38:38.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:38:38 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' 2022-05-14T01:38:38.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:38:38 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' 2022-05-14T01:38:38.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:38:38 smithi115 conmon[90393]: 2022-05-14T01:38:38.667+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:38:38.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:38:38 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:38:38.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:38:38 smithi115 ceph-mon[78593]: pgmap v4278: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:38:38.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:38:38 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:38:38.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:38:38 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:38:38.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:38:38 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T01:38:38.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:38:38 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T01:38:38.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:38:38 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T01:38:38.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:38:38 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' 2022-05-14T01:38:38.862 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:38:38 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' 2022-05-14T01:38:40.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:38:39 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:38:40.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:38:39 smithi115 conmon[90393]: 2022-05-14T01:38:39.659+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:38:40.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:38:39 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:38:40.987 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:38:40 smithi115 conmon[90393]: 2022-05-14T01:38:40.667+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:38:40.987 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:38:40 smithi115 ceph-mon[78593]: pgmap v4279: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:38:40.987 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:38:40 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:38:40.988 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:38:40 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 7855 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:38:41.034 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:38:40 smithi047 ceph-mon[77147]: pgmap v4279: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:38:41.034 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:38:40 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:38:41.034 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:38:40 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 7855 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:38:41.291 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:38:41 smithi047 conmon[78521]: 2022-05-14T01:38:41.071+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:38:41.291 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:38:41 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T01:38:41.292 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:38:41 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:38:41.292 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:38:41 smithi047 conmon[78521]: yield 2022-05-14T01:38:41.292 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:38:41 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:38:41.292 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:38:41 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:38:41.292 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:38:41 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:38:41.293 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:38:41 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T01:38:41.293 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:38:41 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:38:41.293 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:38:41 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T01:38:41.293 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:38:41 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:38:41.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:38:41 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T01:38:41.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:38:41 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:38:41.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:38:41 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T01:38:41.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:38:41 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:38:41.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:38:41 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T01:38:41.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:38:41 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:38:41.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:38:41 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:38:41.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:38:41 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:38:41.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:38:41 smithi115 conmon[90393]: 2022-05-14T01:38:41.626+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:38:42.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:38:41 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:38:42.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:38:41 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' 2022-05-14T01:38:42.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:38:41 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' 2022-05-14T01:38:42.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:38:41 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:38:42.292 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T01:38:42.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:38:42.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T01:38:42.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:38:42.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:38:42.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:38:42.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T01:38:42.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:38:42.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T01:38:42.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:38:42.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T01:38:42.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:38:42.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T01:38:42.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:38:42.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T01:38:42.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:38:42.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:38:42.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:38:42.297 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:38:41 smithi047 ceph-mon[77147]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T01:38:42.297 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:38:41 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-05-14T01:38:42.297 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:38:41 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-05-14T01:38:42.298 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:38:41 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-05-14T01:38:42.298 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:38:41 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-05-14T01:38:42.298 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:38:41 smithi047 ceph-mon[77147]: Adjusting osd_memory_target on smithi115 to 197.0M 2022-05-14T01:38:42.298 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:38:41 smithi047 ceph-mon[77147]: Unable to set osd_memory_target on smithi115 to 206629785: error parsing value: Value '206629785' is below minimum 939524096 2022-05-14T01:38:42.299 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:38:41 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-05-14T01:38:42.299 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:38:41 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-05-14T01:38:42.300 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:38:41 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-05-14T01:38:42.300 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:38:41 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-05-14T01:38:42.300 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:38:41 smithi047 ceph-mon[77147]: Adjusting osd_memory_target on smithi047 to 198.1M 2022-05-14T01:38:42.300 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:38:41 smithi047 ceph-mon[77147]: Unable to set osd_memory_target on smithi047 to 207755878: error parsing value: Value '207755878' is below minimum 939524096 2022-05-14T01:38:42.301 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:38:41 smithi047 ceph-mon[77147]: pgmap v4280: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:38:42.301 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:38:41 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:38:42.301 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:38:41 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:38:42.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:38:41 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:38:42.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:38:41 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' 2022-05-14T01:38:42.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:38:41 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' 2022-05-14T01:38:42.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:38:41 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:38:42.359 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T01:38:42.359 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:38:42.359 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T01:38:42.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:38:42.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:38:42.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:38:42.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T01:38:42.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:38:42.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T01:38:42.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:38:42.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T01:38:42.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:38:42.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T01:38:42.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:38:42.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T01:38:42.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:38:42.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:38:42.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:38:42.363 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:38:41 smithi115 ceph-mon[78593]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T01:38:42.363 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:38:41 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-05-14T01:38:42.364 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:38:41 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-05-14T01:38:42.364 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:38:41 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-05-14T01:38:42.364 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:38:41 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-05-14T01:38:42.364 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:38:41 smithi115 ceph-mon[78593]: Adjusting osd_memory_target on smithi115 to 197.0M 2022-05-14T01:38:42.365 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:38:41 smithi115 ceph-mon[78593]: Unable to set osd_memory_target on smithi115 to 206629785: error parsing value: Value '206629785' is below minimum 939524096 2022-05-14T01:38:42.365 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:38:41 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-05-14T01:38:42.365 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:38:41 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-05-14T01:38:42.365 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:38:41 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-05-14T01:38:42.366 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:38:41 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-05-14T01:38:42.366 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:38:41 smithi115 ceph-mon[78593]: Adjusting osd_memory_target on smithi047 to 198.1M 2022-05-14T01:38:42.366 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:38:41 smithi115 ceph-mon[78593]: Unable to set osd_memory_target on smithi047 to 207755878: error parsing value: Value '207755878' is below minimum 939524096 2022-05-14T01:38:42.366 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:38:41 smithi115 ceph-mon[78593]: pgmap v4280: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:38:42.367 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:38:41 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:38:42.367 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:38:41 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:38:43.044 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:38:42 smithi115 conmon[90393]: 2022-05-14T01:38:42.635+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:38:43.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:38:43 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:38:43.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:38:43 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:38:44.049 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:38:43 smithi115 conmon[90393]: 2022-05-14T01:38:43.651+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:38:44.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:38:44 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:38:44.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:38:44 smithi115 ceph-mon[78593]: pgmap v4281: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:38:44.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:38:44 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:38:44.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:38:44 smithi047 ceph-mon[77147]: pgmap v4281: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:38:44.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:38:44 smithi047 conmon[78521]: 2022-05-14T01:38:44.143+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:38:44.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:38:44 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T01:38:44.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:38:44 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:38:44.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:38:44 smithi047 conmon[78521]: yield 2022-05-14T01:38:44.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:38:44 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:38:44.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:38:44 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:38:44.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:38:44 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:38:44.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:38:44 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T01:38:44.547 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:38:44 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:38:44.547 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:38:44 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T01:38:44.547 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:38:44 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:38:44.548 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:38:44 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T01:38:44.548 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:38:44 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:38:44.549 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:38:44 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T01:38:44.549 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:38:44 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:38:44.550 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:38:44 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T01:38:44.551 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:38:44 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:38:44.551 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:38:44 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:38:44.551 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:38:44 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:38:44.552 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:38:44 smithi047 conmon[78521]: 2022-05-14T01:38:44.144+0000 7f5344781700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T01:38:44.552 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:38:44 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T01:38:44.553 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:38:44 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:38:44.553 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:38:44 smithi047 conmon[78521]: yield 2022-05-14T01:38:44.554 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:38:44 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:38:44.554 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:38:44 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:38:44.554 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:38:44 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:38:44.555 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:38:44 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T01:38:44.555 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:38:44 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:38:44.556 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:38:44 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T01:38:44.556 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:38:44 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:38:44.557 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:38:44 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T01:38:44.557 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:38:44 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:38:44.558 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:38:44 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T01:38:44.558 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:38:44 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:38:44.559 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:38:44 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T01:38:44.559 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:38:44 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:38:44.559 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:38:44 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:38:44.560 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:38:44 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:38:44.560 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:38:44 smithi047 conmon[78521]: 2022-05-14T01:38:44.560 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:38:44 smithi047 conmon[78521]: During handling of the above exception, another exception occurred: 2022-05-14T01:38:44.561 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:38:44 smithi047 conmon[78521]: 2022-05-14T01:38:44.561 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:38:44 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T01:38:44.562 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:38:44 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T01:38:44.562 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:38:44 smithi047 conmon[78521]: return f(*arg) 2022-05-14T01:38:44.562 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:38:44 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T01:38:44.563 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:38:44 smithi047 conmon[78521]: self._write_client_files(client_files, host) 2022-05-14T01:38:44.563 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:38:44 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T01:38:44.563 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:38:44 smithi047 conmon[78521]: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T01:38:44.564 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:38:44 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T01:38:44.564 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:38:44 smithi047 conmon[78521]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T01:38:44.564 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:38:44 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T01:38:44.565 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:38:44 smithi047 conmon[78521]: return self.event_loop.get_result(coro) 2022-05-14T01:38:44.565 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:38:44 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T01:38:44.565 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:38:44 smithi047 conmon[78521]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T01:38:44.566 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:38:44 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T01:38:44.566 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:38:44 smithi047 conmon[78521]: return self.__get_result() 2022-05-14T01:38:44.566 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:38:44 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T01:38:44.566 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:38:44 smithi047 conmon[78521]: raise self._exception 2022-05-14T01:38:44.566 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:38:44 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T01:38:44.567 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:38:44 smithi047 conmon[78521]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T01:38:44.567 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:38:44 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T01:38:44.567 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:38:44 smithi047 conmon[78521]: conn = await self._remote_connection(host, addr) 2022-05-14T01:38:44.567 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:38:44 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T01:38:44.568 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:38:44 smithi047 conmon[78521]: raise 2022-05-14T01:38:44.568 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:38:44 smithi047 conmon[78521]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T01:38:44.568 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:38:44 smithi047 conmon[78521]: self.gen.throw(type, value, traceback) 2022-05-14T01:38:44.568 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:38:44 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T01:38:44.569 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:38:44 smithi047 conmon[78521]: raise OrchestratorError(msg) 2022-05-14T01:38:44.569 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:38:44 smithi047 conmon[78521]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:38:45.050 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:38:44 smithi115 conmon[90393]: 2022-05-14T01:38:44.623+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:38:45.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:38:45 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:38:45.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:38:45 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:38:45.358 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T01:38:45.358 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:38:45.359 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T01:38:45.359 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:38:45.359 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:38:45.359 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:38:45.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T01:38:45.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:38:45.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T01:38:45.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:38:45.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T01:38:45.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:38:45.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T01:38:45.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:38:45.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T01:38:45.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:38:45.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:38:45.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:38:45.363 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:38:45 smithi115 ceph-mon[78593]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T01:38:45.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T01:38:45.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:38:45.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T01:38:45.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:38:45.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:38:45.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:38:45.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T01:38:45.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:38:45.365 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T01:38:45.365 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:38:45.365 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T01:38:45.365 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:38:45.366 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T01:38:45.366 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:38:45.366 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T01:38:45.366 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:38:45.366 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:38:45.367 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:38:45.367 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T01:38:45.367 INFO:journalctl@ceph.mon.c.smithi115.stdout: During handling of the above exception, another exception occurred: 2022-05-14T01:38:45.367 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T01:38:45.368 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T01:38:45.368 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T01:38:45.368 INFO:journalctl@ceph.mon.c.smithi115.stdout: return f(*arg) 2022-05-14T01:38:45.368 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T01:38:45.369 INFO:journalctl@ceph.mon.c.smithi115.stdout: self._write_client_files(client_files, host) 2022-05-14T01:38:45.369 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T01:38:45.369 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T01:38:45.369 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T01:38:45.370 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T01:38:45.370 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T01:38:45.370 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.event_loop.get_result(coro) 2022-05-14T01:38:45.370 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T01:38:45.370 INFO:journalctl@ceph.mon.c.smithi115.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T01:38:45.371 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T01:38:45.371 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.__get_result() 2022-05-14T01:38:45.371 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T01:38:45.371 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise self._exception 2022-05-14T01:38:45.372 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T01:38:45.372 INFO:journalctl@ceph.mon.c.smithi115.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T01:38:45.372 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T01:38:45.372 INFO:journalctl@ceph.mon.c.smithi115.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T01:38:45.372 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T01:38:45.373 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise 2022-05-14T01:38:45.373 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T01:38:45.373 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.gen.throw(type, value, traceback) 2022-05-14T01:38:45.373 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T01:38:45.373 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OrchestratorError(msg) 2022-05-14T01:38:45.374 INFO:journalctl@ceph.mon.c.smithi115.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:38:45.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:38:45 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:38:45.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:38:45 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:38:45.541 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T01:38:45.542 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:38:45.542 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T01:38:45.542 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:38:45.542 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:38:45.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:38:45.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T01:38:45.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:38:45.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T01:38:45.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:38:45.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T01:38:45.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:38:45.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T01:38:45.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:38:45.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T01:38:45.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:38:45.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:38:45.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:38:45.545 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:38:45 smithi047 ceph-mon[77147]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T01:38:45.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T01:38:45.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:38:45.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T01:38:45.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:38:45.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:38:45.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:38:45.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T01:38:45.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:38:45.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T01:38:45.548 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:38:45.548 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T01:38:45.548 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:38:45.548 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T01:38:45.548 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:38:45.549 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T01:38:45.549 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:38:45.549 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:38:45.549 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:38:45.550 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T01:38:45.550 INFO:journalctl@ceph.mon.b.smithi047.stdout: During handling of the above exception, another exception occurred: 2022-05-14T01:38:45.550 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T01:38:45.550 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T01:38:45.550 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T01:38:45.551 INFO:journalctl@ceph.mon.b.smithi047.stdout: return f(*arg) 2022-05-14T01:38:45.551 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T01:38:45.551 INFO:journalctl@ceph.mon.b.smithi047.stdout: self._write_client_files(client_files, host) 2022-05-14T01:38:45.551 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T01:38:45.551 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T01:38:45.552 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T01:38:45.552 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T01:38:45.552 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T01:38:45.552 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.event_loop.get_result(coro) 2022-05-14T01:38:45.553 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T01:38:45.553 INFO:journalctl@ceph.mon.b.smithi047.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T01:38:45.553 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T01:38:45.553 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.__get_result() 2022-05-14T01:38:45.554 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T01:38:45.554 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise self._exception 2022-05-14T01:38:45.554 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T01:38:45.554 INFO:journalctl@ceph.mon.b.smithi047.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T01:38:45.554 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T01:38:45.555 INFO:journalctl@ceph.mon.b.smithi047.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T01:38:45.555 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T01:38:45.555 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise 2022-05-14T01:38:45.555 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T01:38:45.555 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.gen.throw(type, value, traceback) 2022-05-14T01:38:45.556 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T01:38:45.556 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OrchestratorError(msg) 2022-05-14T01:38:45.556 INFO:journalctl@ceph.mon.b.smithi047.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:38:46.055 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:38:45 smithi115 conmon[90393]: 2022-05-14T01:38:45.645+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:38:46.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:38:46 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:38:46.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:38:46 smithi115 ceph-mon[78593]: pgmap v4282: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:38:46.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:38:46 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:38:46.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:38:46 smithi047 ceph-mon[77147]: pgmap v4282: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:38:47.057 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:38:46 smithi115 conmon[90393]: 2022-05-14T01:38:46.648+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:38:47.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:38:47 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:38:47.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:38:47 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 7860 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:38:47.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:38:47 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:38:47.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:38:47 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 7860 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:38:48.062 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:38:47 smithi115 conmon[90393]: 2022-05-14T01:38:47.689+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:38:48.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:38:48 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:38:48.417 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:38:48 smithi115 ceph-mon[78593]: pgmap v4283: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:38:48.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:38:48 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:38:48.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:38:48 smithi047 ceph-mon[77147]: pgmap v4283: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:38:49.067 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:38:48 smithi115 conmon[90393]: 2022-05-14T01:38:48.722+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:38:49.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:38:49 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:38:49.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:38:49 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:38:50.076 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:38:49 smithi115 conmon[90393]: 2022-05-14T01:38:49.724+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:38:50.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:38:50 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:38:50.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:38:50 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:38:50.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:38:50 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:38:50.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:38:50 smithi115 ceph-mon[78593]: pgmap v4284: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:38:50.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:38:50 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:38:50.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:38:50 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:38:50.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:38:50 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:38:50.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:38:50 smithi047 ceph-mon[77147]: pgmap v4284: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:38:51.076 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:38:50 smithi115 conmon[90393]: 2022-05-14T01:38:50.683+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:38:51.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:38:51 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:38:51.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:38:51 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:38:52.081 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:38:51 smithi115 conmon[90393]: 2022-05-14T01:38:51.670+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:38:52.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:38:52 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:38:52.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:38:52 smithi115 ceph-mon[78593]: pgmap v4285: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:38:52.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:38:52 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:38:52.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:38:52 smithi047 ceph-mon[77147]: pgmap v4285: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:38:53.082 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:38:52 smithi115 conmon[90393]: 2022-05-14T01:38:52.665+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:38:53.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:38:53 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:38:53.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:38:53 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:38:54.091 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:38:53 smithi115 conmon[90393]: 2022-05-14T01:38:53.678+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:38:54.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:38:54 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:38:54.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:38:54 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:38:54.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:38:54 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:38:54.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:38:54 smithi115 ceph-mon[78593]: pgmap v4286: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:38:54.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:38:54 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:38:54.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:38:54 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:38:54.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:38:54 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:38:54.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:38:54 smithi047 ceph-mon[77147]: pgmap v4286: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:38:55.092 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:38:54 smithi115 conmon[90393]: 2022-05-14T01:38:54.633+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:38:55.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:38:55 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:38:55.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:38:55 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 7865 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:38:55.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:38:55 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:38:55.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:38:55 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 7865 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:38:56.097 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:38:55 smithi115 conmon[90393]: 2022-05-14T01:38:55.644+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:38:56.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:38:56 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:38:56.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:38:56 smithi115 ceph-mon[78593]: pgmap v4287: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:38:56.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:38:56 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:38:56.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:38:56 smithi047 ceph-mon[77147]: pgmap v4287: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:38:57.098 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:38:56 smithi115 conmon[90393]: 2022-05-14T01:38:56.641+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:38:57.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:38:57 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:38:57.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:38:57 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:38:58.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:38:57 smithi115 conmon[90393]: 2022-05-14T01:38:57.690+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:38:58.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:38:58 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:38:58.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:38:58 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:38:58.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:38:58 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:38:58.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:38:58 smithi115 ceph-mon[78593]: pgmap v4288: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:38:58.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:38:58 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:38:58.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:38:58 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:38:58.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:38:58 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:38:58.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:38:58 smithi047 ceph-mon[77147]: pgmap v4288: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:38:59.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:38:58 smithi115 conmon[90393]: 2022-05-14T01:38:58.690+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:38:59.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:38:59 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:38:59.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:38:59 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:39:00.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:38:59 smithi115 conmon[90393]: 2022-05-14T01:38:59.692+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:39:00.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:39:00 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:39:00.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:39:00 smithi047 ceph-mon[77147]: pgmap v4289: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:39:00.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:39:00 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 7870 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:39:00.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:39:00 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:39:00.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:39:00 smithi115 ceph-mon[78593]: pgmap v4289: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:39:00.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:39:00 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 7870 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:39:01.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:39:00 smithi115 conmon[90393]: 2022-05-14T01:39:00.739+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:39:01.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:39:01 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:39:01.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:39:01 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:39:02.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:39:01 smithi115 conmon[90393]: 2022-05-14T01:39:01.690+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:39:02.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:39:02 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:39:02.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:39:02 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:39:02.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:39:02 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:39:02.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:39:02 smithi047 ceph-mon[77147]: pgmap v4290: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:39:02.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:39:02 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:39:02.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:39:02 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:39:02.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:39:02 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:39:02.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:39:02 smithi115 ceph-mon[78593]: pgmap v4290: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:39:03.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:39:02 smithi115 conmon[90393]: 2022-05-14T01:39:02.684+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:39:03.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:39:03 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:39:03.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:39:03 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:39:04.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:39:03 smithi115 conmon[90393]: 2022-05-14T01:39:03.725+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:39:04.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:39:04 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:39:04.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:39:04 smithi047 ceph-mon[77147]: pgmap v4291: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:39:04.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:39:04 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:39:04.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:39:04 smithi115 ceph-mon[78593]: pgmap v4291: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:39:05.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:39:04 smithi115 conmon[90393]: 2022-05-14T01:39:04.734+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:39:05.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:39:05 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:39:05.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:39:05 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 7875 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:39:05.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:39:05 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:39:05.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:39:05 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 7875 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:39:06.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:39:05 smithi115 conmon[90393]: 2022-05-14T01:39:05.740+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:39:06.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:39:06 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:39:06.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:39:06 smithi047 ceph-mon[77147]: pgmap v4292: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:39:06.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:39:06 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:39:06.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:39:06 smithi115 ceph-mon[78593]: pgmap v4292: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:39:07.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:39:06 smithi115 conmon[90393]: 2022-05-14T01:39:06.769+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:39:07.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:39:07 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:39:07.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:39:07 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:39:08.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:39:07 smithi115 conmon[90393]: 2022-05-14T01:39:07.781+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:39:08.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:39:08 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:39:08.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:39:08 smithi047 ceph-mon[77147]: pgmap v4293: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:39:08.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:39:08 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:39:08.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:39:08 smithi115 ceph-mon[78593]: pgmap v4293: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:39:09.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:39:08 smithi115 conmon[90393]: 2022-05-14T01:39:08.821+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:39:09.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:39:09 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:39:09.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:39:09 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:39:10.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:39:09 smithi115 conmon[90393]: 2022-05-14T01:39:09.845+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:39:10.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:39:10 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:39:10.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:39:10 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:39:10.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:39:10 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:39:10.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:39:10 smithi047 ceph-mon[77147]: pgmap v4294: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:39:10.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:39:10 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 7880 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:39:10.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:39:10 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:39:10.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:39:10 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:39:10.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:39:10 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:39:10.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:39:10 smithi115 ceph-mon[78593]: pgmap v4294: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:39:10.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:39:10 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 7880 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:39:11.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:39:10 smithi115 conmon[90393]: 2022-05-14T01:39:10.844+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:39:11.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:39:11 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:39:11.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:39:11 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:39:12.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:39:11 smithi115 conmon[90393]: 2022-05-14T01:39:11.831+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:39:12.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:39:12 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:39:12.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:39:12 smithi047 ceph-mon[77147]: pgmap v4295: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:39:12.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:39:12 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:39:12.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:39:12 smithi115 ceph-mon[78593]: pgmap v4295: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:39:13.163 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:39:12 smithi115 conmon[90393]: 2022-05-14T01:39:12.877+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:39:13.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:39:13 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:39:13.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:39:13 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:39:14.172 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:39:13 smithi115 conmon[90393]: 2022-05-14T01:39:13.873+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:39:14.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:39:14 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:39:14.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:39:14 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:39:14.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:39:14 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:39:14.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:39:14 smithi047 ceph-mon[77147]: pgmap v4296: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:39:14.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:39:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T01:39:14.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:39:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T01:39:14.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:39:14 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:39:14.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:39:14 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:39:14.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:39:14 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:39:14.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:39:14 smithi115 ceph-mon[78593]: pgmap v4296: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:39:14.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:39:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T01:39:14.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:39:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T01:39:15.173 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:39:14 smithi115 conmon[90393]: 2022-05-14T01:39:14.864+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:39:15.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:39:15 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:39:15.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:39:15 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 7885 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:39:15.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:39:15 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:39:15.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:39:15 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 7885 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:39:16.179 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:39:15 smithi115 conmon[90393]: 2022-05-14T01:39:15.867+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:39:16.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:39:16 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:39:16.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:39:16 smithi047 ceph-mon[77147]: pgmap v4297: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:39:16.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:39:16 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:39:16.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:39:16 smithi115 ceph-mon[78593]: pgmap v4297: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:39:17.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:39:16 smithi115 conmon[90393]: 2022-05-14T01:39:16.838+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:39:17.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:39:17 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:39:17.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:39:17 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:39:18.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:39:17 smithi115 conmon[90393]: 2022-05-14T01:39:17.848+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:39:18.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:39:18 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:39:18.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:39:18 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:39:18.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:39:18 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:39:18.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:39:18 smithi047 ceph-mon[77147]: pgmap v4298: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:39:18.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:39:18 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:39:18.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:39:18 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:39:18.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:39:18 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:39:18.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:39:18 smithi115 ceph-mon[78593]: pgmap v4298: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:39:19.191 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:39:18 smithi115 conmon[90393]: 2022-05-14T01:39:18.861+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:39:19.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:39:19 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:39:19.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:39:19 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:39:20.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:39:19 smithi115 conmon[90393]: 2022-05-14T01:39:19.813+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:39:20.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:39:20 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:39:20.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:39:20 smithi047 ceph-mon[77147]: pgmap v4299: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:39:20.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:39:20 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 7890 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:39:20.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:39:20 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:39:20.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:39:20 smithi115 ceph-mon[78593]: pgmap v4299: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:39:20.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:39:20 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 7890 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:39:21.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:39:20 smithi115 conmon[90393]: 2022-05-14T01:39:20.767+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:39:21.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:39:21 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:39:21.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:39:21 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:39:22.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:39:21 smithi115 conmon[90393]: 2022-05-14T01:39:21.784+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:39:22.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:39:22 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:39:22.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:39:22 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:39:22.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:39:22 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:39:22.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:39:22 smithi047 ceph-mon[77147]: pgmap v4300: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:39:22.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:39:22 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:39:22.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:39:22 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:39:22.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:39:22 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:39:22.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:39:22 smithi115 ceph-mon[78593]: pgmap v4300: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:39:23.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:39:22 smithi115 conmon[90393]: 2022-05-14T01:39:22.803+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:39:23.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:39:23 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:39:23.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:39:23 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:39:24.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:39:23 smithi115 conmon[90393]: 2022-05-14T01:39:23.811+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:39:24.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:39:24 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:39:24.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:39:24 smithi047 ceph-mon[77147]: pgmap v4301: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:39:24.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:39:24 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:39:24.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:39:24 smithi115 ceph-mon[78593]: pgmap v4301: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:39:25.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:39:24 smithi115 conmon[90393]: 2022-05-14T01:39:24.775+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:39:25.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:39:25 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:39:25.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:39:25 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 7895 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:39:25.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:39:25 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:39:25.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:39:25 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 7895 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:39:26.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:39:25 smithi115 conmon[90393]: 2022-05-14T01:39:25.782+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:39:26.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:39:26 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:39:26.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:39:26 smithi047 ceph-mon[77147]: pgmap v4302: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:39:26.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:39:26 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:39:26.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:39:26 smithi115 ceph-mon[78593]: pgmap v4302: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:39:27.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:39:26 smithi115 conmon[90393]: 2022-05-14T01:39:26.740+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:39:27.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:39:27 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:39:27.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:39:27 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:39:28.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:39:27 smithi115 conmon[90393]: 2022-05-14T01:39:27.751+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:39:28.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:39:28 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:39:28.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:39:28 smithi047 ceph-mon[77147]: pgmap v4303: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:39:28.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:39:28 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:39:28.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:39:28 smithi115 ceph-mon[78593]: pgmap v4303: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:39:29.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:39:28 smithi115 conmon[90393]: 2022-05-14T01:39:28.740+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:39:29.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:39:29 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:39:29.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:39:29 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:39:30.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:39:29 smithi115 conmon[90393]: 2022-05-14T01:39:29.788+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:39:30.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:39:30 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:39:30.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:39:30 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:39:30.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:39:30 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:39:30.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:39:30 smithi047 ceph-mon[77147]: pgmap v4304: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:39:30.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:39:30 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 7900 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:39:30.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:39:30 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:39:30.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:39:30 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:39:30.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:39:30 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:39:30.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:39:30 smithi115 ceph-mon[78593]: pgmap v4304: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:39:30.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:39:30 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 7900 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:39:31.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:39:30 smithi115 conmon[90393]: 2022-05-14T01:39:30.803+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:39:31.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:39:31 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:39:31.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:39:31 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:39:32.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:39:31 smithi115 conmon[90393]: 2022-05-14T01:39:31.833+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:39:32.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:39:32 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:39:32.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:39:32 smithi047 ceph-mon[77147]: pgmap v4305: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:39:32.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:39:32 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:39:32.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:39:32 smithi115 ceph-mon[78593]: pgmap v4305: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:39:33.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:39:32 smithi115 conmon[90393]: 2022-05-14T01:39:32.856+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:39:33.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:39:33 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:39:33.613 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:39:33 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:39:34.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:39:33 smithi115 conmon[90393]: 2022-05-14T01:39:33.807+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:39:34.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:39:34 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:39:34.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:39:34 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:39:34.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:39:34 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:39:34.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:39:34 smithi047 ceph-mon[77147]: pgmap v4306: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:39:34.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:39:34 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:39:34.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:39:34 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:39:34.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:39:34 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:39:34.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:39:34 smithi115 ceph-mon[78593]: pgmap v4306: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:39:35.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:39:34 smithi115 conmon[90393]: 2022-05-14T01:39:34.821+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:39:35.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:39:35 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:39:35.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:39:35 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 7905 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:39:35.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:39:35 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:39:35.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:39:35 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 7905 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:39:36.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:39:35 smithi115 conmon[90393]: 2022-05-14T01:39:35.815+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:39:36.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:39:36 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:39:36.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:39:36 smithi047 ceph-mon[77147]: pgmap v4307: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 7 op/s 2022-05-14T01:39:36.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:39:36 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:39:36.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:39:36 smithi115 ceph-mon[78593]: pgmap v4307: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 7 op/s 2022-05-14T01:39:37.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:39:36 smithi115 conmon[90393]: 2022-05-14T01:39:36.821+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:39:37.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:39:37 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:39:37.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:39:37 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:39:38.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:39:37 smithi115 conmon[90393]: 2022-05-14T01:39:37.837+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:39:38.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:39:38 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:39:38.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:39:38 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:39:38.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:39:38 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:39:38.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:39:38 smithi047 ceph-mon[77147]: pgmap v4308: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 4 op/s 2022-05-14T01:39:38.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:39:38 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:39:38.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:39:38 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:39:38.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:39:38 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:39:38.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:39:38 smithi115 ceph-mon[78593]: pgmap v4308: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 4 op/s 2022-05-14T01:39:39.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:39:38 smithi115 conmon[90393]: 2022-05-14T01:39:38.834+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:39:39.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:39:39 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:39:39.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:39:39 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:39:40.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:39:39 smithi115 conmon[90393]: 2022-05-14T01:39:39.833+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:39:40.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:39:40 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:39:40.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:39:40 smithi047 ceph-mon[77147]: pgmap v4309: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 7 op/s 2022-05-14T01:39:40.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:39:40 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 7910 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:39:40.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:39:40 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:39:40.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:39:40 smithi115 ceph-mon[78593]: pgmap v4309: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 7 op/s 2022-05-14T01:39:40.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:39:40 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 7910 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:39:41.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:39:40 smithi115 conmon[90393]: 2022-05-14T01:39:40.784+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:39:41.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:39:41 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:39:41.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:39:41 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:39:42.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:39:41 smithi115 conmon[90393]: 2022-05-14T01:39:41.775+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:39:42.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:39:42 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:39:42.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:39:42 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:39:42.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:39:42 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:39:42.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:39:42 smithi047 ceph-mon[77147]: pgmap v4310: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 4 op/s 2022-05-14T01:39:42.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:39:42 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:39:42.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:39:42 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:39:42.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:39:42 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:39:42.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:39:42 smithi115 ceph-mon[78593]: pgmap v4310: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 4 op/s 2022-05-14T01:39:43.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:39:42 smithi115 conmon[90393]: 2022-05-14T01:39:42.736+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:39:43.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:39:43 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:39:43.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:39:43 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:39:44.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:39:43 smithi115 conmon[90393]: 2022-05-14T01:39:43.776+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:39:44.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:39:44 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:39:44.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:39:44 smithi047 ceph-mon[77147]: pgmap v4311: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 4 op/s 2022-05-14T01:39:44.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:39:44 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T01:39:44.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:39:44 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T01:39:44.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:39:44 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T01:39:44.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:39:44 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:39:44.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:39:44 smithi115 ceph-mon[78593]: pgmap v4311: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 4 op/s 2022-05-14T01:39:44.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:39:44 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T01:39:44.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:39:44 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T01:39:44.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:39:44 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T01:39:45.041 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:39:44 smithi047 conmon[78521]: 2022-05-14T01:39:44.751+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:39:45.041 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:39:44 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T01:39:45.041 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:39:44 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:39:45.042 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:39:44 smithi047 conmon[78521]: yield 2022-05-14T01:39:45.042 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:39:44 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:39:45.042 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:39:44 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:39:45.042 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:39:44 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:39:45.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:39:44 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T01:39:45.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:39:44 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:39:45.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:39:44 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T01:39:45.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:39:44 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:39:45.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:39:44 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T01:39:45.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:39:44 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:39:45.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:39:44 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T01:39:45.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:39:44 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:39:45.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:39:44 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T01:39:45.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:39:44 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:39:45.048 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:39:44 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:39:45.048 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:39:44 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:39:45.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:39:44 smithi115 conmon[90393]: 2022-05-14T01:39:44.791+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:39:45.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:39:45 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:39:45.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:39:45 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:39:45.609 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T01:39:45.609 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:39:45.609 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T01:39:45.609 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:39:45.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:39:45.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:39:45.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T01:39:45.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:39:45.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T01:39:45.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:39:45.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T01:39:45.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:39:45.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T01:39:45.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:39:45.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T01:39:45.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:39:45.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:39:45.613 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:39:45.613 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:39:45 smithi115 ceph-mon[78593]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T01:39:45.613 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:39:45 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 7915 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:39:45.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:39:45 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:39:45.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:39:45 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:39:45.791 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T01:39:45.792 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:39:45.792 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T01:39:45.792 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:39:45.792 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:39:45.793 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:39:45.793 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T01:39:45.793 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:39:45.793 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T01:39:45.793 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:39:45.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T01:39:45.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:39:45.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T01:39:45.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:39:45.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T01:39:45.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:39:45.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:39:45.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:39:45.795 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:39:45 smithi047 ceph-mon[77147]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T01:39:45.796 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:39:45 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 7915 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:39:46.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:39:45 smithi115 conmon[90393]: 2022-05-14T01:39:45.758+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:39:46.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:39:46 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:39:46.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:39:46 smithi115 ceph-mon[78593]: pgmap v4312: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 7 op/s 2022-05-14T01:39:46.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:39:46 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:39:46.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:39:46 smithi047 ceph-mon[77147]: pgmap v4312: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 7 op/s 2022-05-14T01:39:47.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:39:46 smithi115 conmon[90393]: 2022-05-14T01:39:46.788+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:39:47.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:39:47 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:39:47.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:39:47 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:39:48.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:39:47 smithi115 conmon[90393]: 2022-05-14T01:39:47.820+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:39:48.291 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:39:47 smithi047 conmon[78521]: 2022-05-14T01:39:47.824+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:39:48.291 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:39:47 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T01:39:48.292 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:39:47 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:39:48.292 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:39:47 smithi047 conmon[78521]: yield 2022-05-14T01:39:48.292 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:39:47 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:39:48.292 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:39:47 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:39:48.293 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:39:47 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:39:48.293 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:39:47 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T01:39:48.293 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:39:47 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:39:48.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:39:47 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T01:39:48.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:39:47 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:39:48.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:39:47 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T01:39:48.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:39:47 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:39:48.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:39:47 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T01:39:48.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:39:47 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:39:48.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:39:47 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T01:39:48.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:39:47 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:39:48.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:39:47 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:39:48.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:39:47 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:39:48.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:39:47 smithi047 conmon[78521]: 2022-05-14T01:39:47.825+0000 7f5345f84700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T01:39:48.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:39:47 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T01:39:48.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:39:47 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:39:48.297 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:39:47 smithi047 conmon[78521]: yield 2022-05-14T01:39:48.297 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:39:47 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:39:48.297 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:39:47 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:39:48.298 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:39:47 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:39:48.298 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:39:47 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T01:39:48.298 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:39:47 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:39:48.298 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:39:47 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T01:39:48.299 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:39:47 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:39:48.299 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:39:47 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T01:39:48.299 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:39:47 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:39:48.299 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:39:47 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T01:39:48.299 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:39:47 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:39:48.300 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:39:47 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T01:39:48.300 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:39:47 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:39:48.300 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:39:47 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:39:48.301 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:39:47 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:39:48.301 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:39:47 smithi047 conmon[78521]: 2022-05-14T01:39:48.301 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:39:47 smithi047 conmon[78521]: During handling of the above exception, another exception occurred: 2022-05-14T01:39:48.301 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:39:47 smithi047 conmon[78521]: 2022-05-14T01:39:48.302 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:39:47 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T01:39:48.302 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:39:47 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T01:39:48.302 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:39:47 smithi047 conmon[78521]: return f(*arg) 2022-05-14T01:39:48.302 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:39:47 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T01:39:48.302 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:39:47 smithi047 conmon[78521]: self._write_client_files(client_files, host) 2022-05-14T01:39:48.303 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:39:47 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T01:39:48.303 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:39:47 smithi047 conmon[78521]: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T01:39:48.303 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:39:47 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T01:39:48.303 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:39:47 smithi047 conmon[78521]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T01:39:48.304 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:39:47 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T01:39:48.304 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:39:47 smithi047 conmon[78521]: return self.event_loop.get_result(coro) 2022-05-14T01:39:48.304 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:39:47 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T01:39:48.304 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:39:47 smithi047 conmon[78521]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T01:39:48.304 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:39:47 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T01:39:48.305 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:39:47 smithi047 conmon[78521]: return self.__get_result() 2022-05-14T01:39:48.305 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:39:47 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T01:39:48.305 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:39:47 smithi047 conmon[78521]: raise self._exception 2022-05-14T01:39:48.305 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:39:47 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T01:39:48.306 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:39:47 smithi047 conmon[78521]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T01:39:48.306 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:39:47 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T01:39:48.306 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:39:47 smithi047 conmon[78521]: conn = await self._remote_connection(host, addr) 2022-05-14T01:39:48.306 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:39:47 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T01:39:48.306 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:39:47 smithi047 conmon[78521]: raise 2022-05-14T01:39:48.307 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:39:47 smithi047 conmon[78521]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T01:39:48.307 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:39:47 smithi047 conmon[78521]: self.gen.throw(type, value, traceback) 2022-05-14T01:39:48.307 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:39:47 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T01:39:48.307 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:39:47 smithi047 conmon[78521]: raise OrchestratorError(msg) 2022-05-14T01:39:48.308 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:39:47 smithi047 conmon[78521]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:39:48.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:39:48 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:39:48.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:39:48 smithi115 ceph-mon[78593]: pgmap v4313: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:39:48.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:39:48 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:39:48.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:39:48 smithi047 ceph-mon[77147]: pgmap v4313: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:39:49.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:39:48 smithi115 conmon[90393]: 2022-05-14T01:39:48.845+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:39:49.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:39:49 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:39:49.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:39:49 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:39:49.608 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T01:39:49.608 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:39:49.609 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T01:39:49.609 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:39:49.609 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:39:49.609 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:39:49.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T01:39:49.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:39:49.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T01:39:49.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:39:49.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T01:39:49.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:39:49.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T01:39:49.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:39:49.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T01:39:49.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:39:49.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:39:49.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:39:49.612 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:39:49 smithi115 ceph-mon[78593]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T01:39:49.613 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T01:39:49.613 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:39:49.613 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T01:39:49.613 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:39:49.614 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:39:49.614 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:39:49.614 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T01:39:49.614 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:39:49.615 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T01:39:49.615 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:39:49.615 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T01:39:49.615 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:39:49.615 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T01:39:49.616 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:39:49.616 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T01:39:49.616 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:39:49.616 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:39:49.617 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:39:49.617 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T01:39:49.617 INFO:journalctl@ceph.mon.c.smithi115.stdout: During handling of the above exception, another exception occurred: 2022-05-14T01:39:49.617 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T01:39:49.618 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T01:39:49.618 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T01:39:49.618 INFO:journalctl@ceph.mon.c.smithi115.stdout: return f(*arg) 2022-05-14T01:39:49.618 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T01:39:49.618 INFO:journalctl@ceph.mon.c.smithi115.stdout: self._write_client_files(client_files, host) 2022-05-14T01:39:49.619 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T01:39:49.619 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T01:39:49.619 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T01:39:49.619 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T01:39:49.620 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T01:39:49.620 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.event_loop.get_result(coro) 2022-05-14T01:39:49.620 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T01:39:49.620 INFO:journalctl@ceph.mon.c.smithi115.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T01:39:49.621 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T01:39:49.621 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.__get_result() 2022-05-14T01:39:49.621 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T01:39:49.621 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise self._exception 2022-05-14T01:39:49.622 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T01:39:49.622 INFO:journalctl@ceph.mon.c.smithi115.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T01:39:49.622 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T01:39:49.622 INFO:journalctl@ceph.mon.c.smithi115.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T01:39:49.623 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T01:39:49.623 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise 2022-05-14T01:39:49.623 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T01:39:49.623 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.gen.throw(type, value, traceback) 2022-05-14T01:39:49.624 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T01:39:49.624 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OrchestratorError(msg) 2022-05-14T01:39:49.624 INFO:journalctl@ceph.mon.c.smithi115.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:39:49.801 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:39:49 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:39:49.802 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:39:49 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:39:49.802 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T01:39:49.802 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:39:49.803 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T01:39:49.803 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:39:49.803 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:39:49.803 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:39:49.803 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T01:39:49.804 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:39:49.804 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T01:39:49.804 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:39:49.804 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T01:39:49.805 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:39:49.805 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T01:39:49.805 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:39:49.805 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T01:39:49.805 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:39:49.806 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:39:49.806 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:39:49.806 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:39:49 smithi047 ceph-mon[77147]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T01:39:49.806 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T01:39:49.807 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:39:49.807 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T01:39:49.807 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:39:49.807 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:39:49.807 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:39:49.808 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T01:39:49.808 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:39:49.808 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T01:39:49.808 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:39:49.808 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T01:39:49.809 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:39:49.809 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T01:39:49.809 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:39:49.809 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T01:39:49.809 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:39:49.810 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:39:49.810 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:39:49.810 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T01:39:49.810 INFO:journalctl@ceph.mon.b.smithi047.stdout: During handling of the above exception, another exception occurred: 2022-05-14T01:39:49.811 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T01:39:49.811 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T01:39:49.812 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T01:39:49.812 INFO:journalctl@ceph.mon.b.smithi047.stdout: return f(*arg) 2022-05-14T01:39:49.812 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T01:39:49.812 INFO:journalctl@ceph.mon.b.smithi047.stdout: self._write_client_files(client_files, host) 2022-05-14T01:39:49.812 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T01:39:49.813 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T01:39:49.813 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T01:39:49.813 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T01:39:49.813 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T01:39:49.814 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.event_loop.get_result(coro) 2022-05-14T01:39:49.814 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T01:39:49.814 INFO:journalctl@ceph.mon.b.smithi047.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T01:39:49.814 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T01:39:49.814 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.__get_result() 2022-05-14T01:39:49.815 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T01:39:49.815 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise self._exception 2022-05-14T01:39:49.815 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T01:39:49.815 INFO:journalctl@ceph.mon.b.smithi047.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T01:39:49.815 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T01:39:49.816 INFO:journalctl@ceph.mon.b.smithi047.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T01:39:49.816 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T01:39:49.816 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise 2022-05-14T01:39:49.816 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T01:39:49.816 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.gen.throw(type, value, traceback) 2022-05-14T01:39:49.817 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T01:39:49.817 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OrchestratorError(msg) 2022-05-14T01:39:49.817 INFO:journalctl@ceph.mon.b.smithi047.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:39:50.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:39:49 smithi115 conmon[90393]: 2022-05-14T01:39:49.815+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:39:50.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:39:50 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:39:50.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:39:50 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:39:50.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:39:50 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:39:50.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:39:50 smithi115 ceph-mon[78593]: pgmap v4314: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:39:50.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:39:50 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 7920 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:39:50.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:39:50 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:39:50.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:39:50 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:39:50.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:39:50 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:39:50.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:39:50 smithi047 ceph-mon[77147]: pgmap v4314: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:39:50.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:39:50 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 7920 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:39:51.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:39:50 smithi115 conmon[90393]: 2022-05-14T01:39:50.774+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:39:51.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:39:51 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:39:51.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:39:51 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:39:52.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:39:51 smithi115 conmon[90393]: 2022-05-14T01:39:51.741+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:39:52.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:39:52 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:39:52.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:39:52 smithi115 ceph-mon[78593]: pgmap v4315: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:39:52.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:39:52 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:39:52.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:39:52 smithi047 ceph-mon[77147]: pgmap v4315: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:39:53.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:39:52 smithi115 conmon[90393]: 2022-05-14T01:39:52.719+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:39:53.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:39:53 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:39:53.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:39:53 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:39:54.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:39:53 smithi115 conmon[90393]: 2022-05-14T01:39:53.698+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:39:54.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:39:54 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:39:54.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:39:54 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:39:54.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:39:54 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:39:54.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:39:54 smithi115 ceph-mon[78593]: pgmap v4316: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:39:54.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:39:54 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:39:54.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:39:54 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:39:54.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:39:54 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:39:54.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:39:54 smithi047 ceph-mon[77147]: pgmap v4316: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:39:55.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:39:54 smithi115 conmon[90393]: 2022-05-14T01:39:54.686+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:39:55.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:39:55 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:39:55.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:39:55 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 7925 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:39:55.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:39:55 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:39:55.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:39:55 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 7925 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:39:56.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:39:55 smithi115 conmon[90393]: 2022-05-14T01:39:55.725+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:39:56.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:39:56 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:39:56.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:39:56 smithi115 ceph-mon[78593]: pgmap v4317: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:39:56.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:39:56 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:39:56.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:39:56 smithi047 ceph-mon[77147]: pgmap v4317: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:39:57.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:39:56 smithi115 conmon[90393]: 2022-05-14T01:39:56.700+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:39:57.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:39:57 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:39:57.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:39:57 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:39:58.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:39:57 smithi115 conmon[90393]: 2022-05-14T01:39:57.683+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:39:58.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:39:58 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:39:58.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:39:58 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:39:58.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:39:58 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:39:58.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:39:58 smithi115 ceph-mon[78593]: pgmap v4318: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:39:58.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:39:58 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:39:58.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:39:58 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:39:58.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:39:58 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:39:58.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:39:58 smithi047 ceph-mon[77147]: pgmap v4318: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:39:59.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:39:58 smithi115 conmon[90393]: 2022-05-14T01:39:58.674+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:39:59.682 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:39:59 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:39:59.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:39:59 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:40:00.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:39:59 smithi115 conmon[90393]: 2022-05-14T01:39:59.680+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:40:00.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:40:00 smithi047 conmon[77123]: 2022-05-14T01:39:59.999+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : Health detail: HEALTH_ERR 1 filesystem is degraded; 1 mds daemon damaged; 1 MDSs are read only; 1 MDSs report slow metadata IOs; 1/3 mons down, quorum b,c; Reduced data availability: 1 pg inactive; 9 slow ops, oldest one blocked for 7930 sec, osd.5 has slow ops 2022-05-14T01:40:00.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:40:00 smithi047 conmon[77123]: 2022-05-14T01:39:59.999+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : [WRN] FS_DEGRADED: 1 filesystem is degraded 2022-05-14T01:40:00.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:40:00 smithi047 conmon[77123]: 2022-05-14T01:39:59.999+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : fs cephfs is degraded 2022-05-14T01:40:00.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:40:00 smithi047 conmon[77123]: 2022-05-14T01:39:59.999+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : [ERR] MDS_DAMAGE: 1 mds daemon damaged 2022-05-14T01:40:00.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:40:00 smithi047 conmon[77123]: 2022-05-14T01:39:59.999+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : fs cephfs mds.2 is damaged 2022-05-14T01:40:00.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:40:00 smithi047 conmon[77123]: 2022-05-14T01:39:59.999+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : [WRN] MDS_READ_ONLY: 1 MDSs are read only 2022-05-14T01:40:00.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:40:00 smithi047 conmon[77123]: 2022-05-14T01:39:59.999+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : mds.l(mds.0): MDS in read-only mode 2022-05-14T01:40:00.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:40:00 smithi047 conmon[77123]: 2022-05-14T01:39:59.999+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : [WRN] MDS_SLOW_METADATA_IO: 1 MDSs report slow metadata IOs 2022-05-14T01:40:00.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:40:00 smithi047 conmon[77123]: 2022-05-14T01:40:00.000+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : mds.l(mds.0): 4 slow metadata IOs are blocked > 180 secs, oldest blocked for 8233 secs 2022-05-14T01:40:00.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:40:00 smithi047 conmon[77123]: 2022-05-14T01:40:00.000+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : [WRN] MON_DOWN: 1/3 mons down, quorum b,c 2022-05-14T01:40:00.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:40:00 smithi047 conmon[77123]: 2022-05-14T01:40:00.000+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : mon.a (rank 0) addr [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] is down (out of quorum) 2022-05-14T01:40:00.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:40:00 smithi047 conmon[77123]: 2022-05-14T01:40:00.000+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : [WRN] PG_AVAILABILITY: Reduced data availability: 1 pg inactive 2022-05-14T01:40:00.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:40:00 smithi047 conmon[77123]: 2022-05-14T01:40:00.000+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : pg 2.1 is stuck inactive for 2h, current state unknown, last acting [] 2022-05-14T01:40:00.295 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:40:00 smithi047 conmon[77123]: 2022-05-14T01:40:00.000+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : [WRN] SLOW_OPS: 9 slow ops, oldest one blocked for 7930 sec, osd.5 has slow ops 2022-05-14T01:40:00.635 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:40:00 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:40:00.636 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:40:00 smithi115 ceph-mon[78593]: pgmap v4319: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:40:00.636 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:40:00 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 7930 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:40:00.636 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:40:00 smithi115 ceph-mon[78593]: Health detail: HEALTH_ERR 1 filesystem is degraded; 1 mds daemon damaged; 1 MDSs are read only; 1 MDSs report slow metadata IOs; 1/3 mons down, quorum b,c; Reduced data availability: 1 pg inactive; 9 slow ops, oldest one blocked for 7930 sec, osd.5 has slow ops 2022-05-14T01:40:00.637 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:40:00 smithi115 ceph-mon[78593]: [WRN] FS_DEGRADED: 1 filesystem is degraded 2022-05-14T01:40:00.637 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:40:00 smithi115 ceph-mon[78593]: fs cephfs is degraded 2022-05-14T01:40:00.637 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:40:00 smithi115 ceph-mon[78593]: [ERR] MDS_DAMAGE: 1 mds daemon damaged 2022-05-14T01:40:00.637 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:40:00 smithi115 ceph-mon[78593]: fs cephfs mds.2 is damaged 2022-05-14T01:40:00.638 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:40:00 smithi115 ceph-mon[78593]: [WRN] MDS_READ_ONLY: 1 MDSs are read only 2022-05-14T01:40:00.638 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:40:00 smithi115 ceph-mon[78593]: mds.l(mds.0): MDS in read-only mode 2022-05-14T01:40:00.638 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:40:00 smithi115 ceph-mon[78593]: [WRN] MDS_SLOW_METADATA_IO: 1 MDSs report slow metadata IOs 2022-05-14T01:40:00.638 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:40:00 smithi115 ceph-mon[78593]: mds.l(mds.0): 4 slow metadata IOs are blocked > 180 secs, oldest blocked for 8233 secs 2022-05-14T01:40:00.638 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:40:00 smithi115 ceph-mon[78593]: [WRN] MON_DOWN: 1/3 mons down, quorum b,c 2022-05-14T01:40:00.639 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:40:00 smithi115 ceph-mon[78593]: mon.a (rank 0) addr [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] is down (out of quorum) 2022-05-14T01:40:00.639 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:40:00 smithi115 ceph-mon[78593]: [WRN] PG_AVAILABILITY: Reduced data availability: 1 pg inactive 2022-05-14T01:40:00.639 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:40:00 smithi115 ceph-mon[78593]: pg 2.1 is stuck inactive for 2h, current state unknown, last acting [] 2022-05-14T01:40:00.639 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:40:00 smithi115 ceph-mon[78593]: [WRN] SLOW_OPS: 9 slow ops, oldest one blocked for 7930 sec, osd.5 has slow ops 2022-05-14T01:40:00.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:40:00 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:40:00.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:40:00 smithi047 ceph-mon[77147]: pgmap v4319: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:40:00.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:40:00 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 7930 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:40:00.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:40:00 smithi047 ceph-mon[77147]: Health detail: HEALTH_ERR 1 filesystem is degraded; 1 mds daemon damaged; 1 MDSs are read only; 1 MDSs report slow metadata IOs; 1/3 mons down, quorum b,c; Reduced data availability: 1 pg inactive; 9 slow ops, oldest one blocked for 7930 sec, osd.5 has slow ops 2022-05-14T01:40:00.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:40:00 smithi047 ceph-mon[77147]: [WRN] FS_DEGRADED: 1 filesystem is degraded 2022-05-14T01:40:00.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:40:00 smithi047 ceph-mon[77147]: fs cephfs is degraded 2022-05-14T01:40:00.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:40:00 smithi047 ceph-mon[77147]: [ERR] MDS_DAMAGE: 1 mds daemon damaged 2022-05-14T01:40:00.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:40:00 smithi047 ceph-mon[77147]: fs cephfs mds.2 is damaged 2022-05-14T01:40:00.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:40:00 smithi047 ceph-mon[77147]: [WRN] MDS_READ_ONLY: 1 MDSs are read only 2022-05-14T01:40:00.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:40:00 smithi047 ceph-mon[77147]: mds.l(mds.0): MDS in read-only mode 2022-05-14T01:40:00.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:40:00 smithi047 ceph-mon[77147]: [WRN] MDS_SLOW_METADATA_IO: 1 MDSs report slow metadata IOs 2022-05-14T01:40:00.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:40:00 smithi047 ceph-mon[77147]: mds.l(mds.0): 4 slow metadata IOs are blocked > 180 secs, oldest blocked for 8233 secs 2022-05-14T01:40:00.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:40:00 smithi047 ceph-mon[77147]: [WRN] MON_DOWN: 1/3 mons down, quorum b,c 2022-05-14T01:40:00.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:40:00 smithi047 ceph-mon[77147]: mon.a (rank 0) addr [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] is down (out of quorum) 2022-05-14T01:40:00.795 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:40:00 smithi047 ceph-mon[77147]: [WRN] PG_AVAILABILITY: Reduced data availability: 1 pg inactive 2022-05-14T01:40:00.795 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:40:00 smithi047 ceph-mon[77147]: pg 2.1 is stuck inactive for 2h, current state unknown, last acting [] 2022-05-14T01:40:00.795 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:40:00 smithi047 ceph-mon[77147]: [WRN] SLOW_OPS: 9 slow ops, oldest one blocked for 7930 sec, osd.5 has slow ops 2022-05-14T01:40:01.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:40:00 smithi115 conmon[90393]: 2022-05-14T01:40:00.634+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:40:01.627 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:40:01 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:40:01.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:40:01 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:40:02.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:40:01 smithi115 conmon[90393]: 2022-05-14T01:40:01.626+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:40:02.641 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:40:02 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:40:02.641 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:40:02 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:40:02.642 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:40:02 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:40:02.642 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:40:02 smithi115 ceph-mon[78593]: pgmap v4320: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:40:02.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:40:02 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:40:02.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:40:02 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:40:02.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:40:02 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:40:02.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:40:02 smithi047 ceph-mon[77147]: pgmap v4320: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:40:03.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:40:02 smithi115 conmon[90393]: 2022-05-14T01:40:02.640+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:40:03.678 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:40:03 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:40:03.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:40:03 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:40:04.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:40:03 smithi115 conmon[90393]: 2022-05-14T01:40:03.677+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:40:04.684 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:40:04 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:40:04.685 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:40:04 smithi115 ceph-mon[78593]: pgmap v4321: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:40:04.790 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:40:04 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:40:04.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:40:04 smithi047 ceph-mon[77147]: pgmap v4321: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:40:05.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:40:04 smithi115 conmon[90393]: 2022-05-14T01:40:04.684+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:40:05.709 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:40:05 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:40:05.710 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:40:05 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 7935 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:40:05.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:40:05 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:40:05.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:40:05 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 7935 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:40:06.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:40:05 smithi115 conmon[90393]: 2022-05-14T01:40:05.707+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:40:06.716 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:40:06 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:40:06.717 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:40:06 smithi115 ceph-mon[78593]: pgmap v4322: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:40:06.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:40:06 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:40:06.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:40:06 smithi047 ceph-mon[77147]: pgmap v4322: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:40:07.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:40:06 smithi115 conmon[90393]: 2022-05-14T01:40:06.715+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:40:07.700 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:40:07 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:40:07.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:40:07 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:40:08.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:40:07 smithi115 conmon[90393]: 2022-05-14T01:40:07.699+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:40:08.717 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:40:08 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:40:08.717 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:40:08 smithi115 ceph-mon[78593]: pgmap v4323: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:40:08.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:40:08 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:40:08.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:40:08 smithi047 ceph-mon[77147]: pgmap v4323: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:40:09.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:40:08 smithi115 conmon[90393]: 2022-05-14T01:40:08.716+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:40:09.691 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:40:09 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:40:09.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:40:09 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:40:10.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:40:09 smithi115 conmon[90393]: 2022-05-14T01:40:09.690+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:40:10.677 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:40:10 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:40:10.678 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:40:10 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:40:10.678 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:40:10 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:40:10.678 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:40:10 smithi115 ceph-mon[78593]: pgmap v4324: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:40:10.679 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:40:10 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 7940 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:40:10.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:40:10 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:40:10.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:40:10 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:40:10.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:40:10 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:40:10.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:40:10 smithi047 ceph-mon[77147]: pgmap v4324: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:40:10.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:40:10 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 7940 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:40:11.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:40:10 smithi115 conmon[90393]: 2022-05-14T01:40:10.676+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:40:11.718 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:40:11 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:40:11.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:40:11 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:40:12.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:40:11 smithi115 conmon[90393]: 2022-05-14T01:40:11.717+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:40:12.674 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:40:12 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:40:12.674 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:40:12 smithi115 ceph-mon[78593]: pgmap v4325: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:40:12.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:40:12 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:40:12.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:40:12 smithi047 ceph-mon[77147]: pgmap v4325: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:40:13.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:40:12 smithi115 conmon[90393]: 2022-05-14T01:40:12.673+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:40:13.682 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:40:13 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:40:13.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:40:13 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:40:14.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:40:13 smithi115 conmon[90393]: 2022-05-14T01:40:13.681+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:40:14.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:40:14 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:40:14.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:40:14 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:40:14.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:40:14 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:40:14.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:40:14 smithi047 ceph-mon[77147]: pgmap v4326: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:40:14.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:40:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T01:40:14.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:40:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T01:40:14.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:40:14 smithi115 conmon[90393]: 2022-05-14T01:40:14.632+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:40:14.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:40:14 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:40:14.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:40:14 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:40:14.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:40:14 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:40:14.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:40:14 smithi115 ceph-mon[78593]: pgmap v4326: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:40:14.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:40:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T01:40:14.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:40:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T01:40:15.676 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:40:15 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:40:15.676 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:40:15 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 7945 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:40:15.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:40:15 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:40:15.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:40:15 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 7945 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:40:16.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:40:15 smithi115 conmon[90393]: 2022-05-14T01:40:15.675+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:40:16.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:40:16 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:40:16.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:40:16 smithi047 ceph-mon[77147]: pgmap v4327: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:40:16.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:40:16 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:40:16.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:40:16 smithi115 ceph-mon[78593]: pgmap v4327: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:40:16.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:40:16 smithi115 conmon[90393]: 2022-05-14T01:40:16.672+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:40:17.693 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:40:17 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:40:17.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:40:17 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:40:18.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:40:17 smithi115 conmon[90393]: 2022-05-14T01:40:17.692+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:40:18.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:40:18 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:40:18.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:40:18 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:40:18.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:40:18 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:40:18.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:40:18 smithi047 ceph-mon[77147]: pgmap v4328: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:40:18.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:40:18 smithi115 conmon[90393]: 2022-05-14T01:40:18.654+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:40:18.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:40:18 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:40:18.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:40:18 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:40:18.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:40:18 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:40:18.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:40:18 smithi115 ceph-mon[78593]: pgmap v4328: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:40:19.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:40:19 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:40:19.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:40:19 smithi115 conmon[90393]: 2022-05-14T01:40:19.684+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:40:19.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:40:19 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:40:20.735 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:40:20 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:40:20.735 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:40:20 smithi115 ceph-mon[78593]: pgmap v4329: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:40:20.736 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:40:20 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 7950 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:40:20.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:40:20 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:40:20.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:40:20 smithi047 ceph-mon[77147]: pgmap v4329: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:40:20.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:40:20 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 7950 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:40:21.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:40:20 smithi115 conmon[90393]: 2022-05-14T01:40:20.734+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:40:21.730 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:40:21 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:40:21.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:40:21 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:40:22.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:40:21 smithi115 conmon[90393]: 2022-05-14T01:40:21.729+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:40:22.779 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:40:22 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:40:22.780 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:40:22 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:40:22.780 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:40:22 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:40:22.780 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:40:22 smithi115 ceph-mon[78593]: pgmap v4330: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:40:22.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:40:22 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:40:22.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:40:22 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:40:22.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:40:22 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:40:22.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:40:22 smithi047 ceph-mon[77147]: pgmap v4330: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:40:23.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:40:22 smithi115 conmon[90393]: 2022-05-14T01:40:22.777+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:40:23.750 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:40:23 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:40:23.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:40:23 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:40:24.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:40:23 smithi115 conmon[90393]: 2022-05-14T01:40:23.748+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:40:24.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:40:24 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:40:24.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:40:24 smithi047 ceph-mon[77147]: pgmap v4331: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:40:24.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:40:24 smithi115 conmon[90393]: 2022-05-14T01:40:24.705+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:40:24.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:40:24 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:40:24.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:40:24 smithi115 ceph-mon[78593]: pgmap v4331: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:40:25.742 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:40:25 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:40:25.742 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:40:25 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 7955 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:40:25.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:40:25 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:40:25.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:40:25 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 7955 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:40:26.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:40:25 smithi115 conmon[90393]: 2022-05-14T01:40:25.742+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:40:26.781 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:40:26 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:40:26.781 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:40:26 smithi115 ceph-mon[78593]: pgmap v4332: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:40:26.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:40:26 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:40:26.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:40:26 smithi047 ceph-mon[77147]: pgmap v4332: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:40:27.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:40:26 smithi115 conmon[90393]: 2022-05-14T01:40:26.780+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:40:27.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:40:27 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:40:27.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:40:27 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:40:28.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:40:27 smithi115 conmon[90393]: 2022-05-14T01:40:27.790+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:40:28.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:40:28 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:40:28.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:40:28 smithi047 ceph-mon[77147]: pgmap v4333: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:40:28.810 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:40:28 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:40:28.810 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:40:28 smithi115 ceph-mon[78593]: pgmap v4333: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:40:29.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:40:28 smithi115 conmon[90393]: 2022-05-14T01:40:28.808+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:40:29.767 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:40:29 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:40:29.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:40:29 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:40:30.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:40:29 smithi115 conmon[90393]: 2022-05-14T01:40:29.766+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:40:30.755 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:40:30 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:40:30.756 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:40:30 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:40:30.757 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:40:30 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:40:30.757 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:40:30 smithi115 ceph-mon[78593]: pgmap v4334: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:40:30.757 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:40:30 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 7960 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:40:30.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:40:30 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:40:30.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:40:30 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:40:30.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:40:30 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:40:30.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:40:30 smithi047 ceph-mon[77147]: pgmap v4334: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:40:30.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:40:30 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 7960 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:40:31.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:40:30 smithi115 conmon[90393]: 2022-05-14T01:40:30.754+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:40:31.746 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:40:31 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:40:31.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:40:31 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:40:32.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:40:31 smithi115 conmon[90393]: 2022-05-14T01:40:31.745+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:40:32.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:40:32 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:40:32.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:40:32 smithi047 ceph-mon[77147]: pgmap v4335: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:40:32.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:40:32 smithi115 conmon[90393]: 2022-05-14T01:40:32.699+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:40:32.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:40:32 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:40:32.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:40:32 smithi115 ceph-mon[78593]: pgmap v4335: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:40:33.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:40:33 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:40:33.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:40:33 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:40:33.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:40:33 smithi115 conmon[90393]: 2022-05-14T01:40:33.708+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:40:34.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:40:34 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:40:34.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:40:34 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:40:34.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:40:34 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:40:34.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:40:34 smithi047 ceph-mon[77147]: pgmap v4336: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:40:34.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:40:34 smithi115 conmon[90393]: 2022-05-14T01:40:34.660+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:40:34.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:40:34 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:40:34.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:40:34 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:40:34.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:40:34 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:40:34.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:40:34 smithi115 ceph-mon[78593]: pgmap v4336: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:40:35.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:40:35 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:40:35.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:40:35 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 7965 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:40:35.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:40:35 smithi115 conmon[90393]: 2022-05-14T01:40:35.695+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:40:35.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:40:35 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:40:35.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:40:35 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 7965 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:40:36.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:40:36 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:40:36.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:40:36 smithi047 ceph-mon[77147]: pgmap v4337: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:40:36.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:40:36 smithi115 conmon[90393]: 2022-05-14T01:40:36.676+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:40:36.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:40:36 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:40:36.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:40:36 smithi115 ceph-mon[78593]: pgmap v4337: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:40:37.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:40:37 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:40:37.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:40:37 smithi115 conmon[90393]: 2022-05-14T01:40:37.700+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:40:37.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:40:37 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:40:38.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:40:38 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:40:38.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:40:38 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:40:38.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:40:38 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:40:38.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:40:38 smithi047 ceph-mon[77147]: pgmap v4338: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:40:38.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:40:38 smithi115 conmon[90393]: 2022-05-14T01:40:38.673+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:40:38.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:40:38 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:40:38.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:40:38 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:40:38.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:40:38 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:40:38.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:40:38 smithi115 ceph-mon[78593]: pgmap v4338: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:40:39.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:40:39 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:40:39.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:40:39 smithi115 conmon[90393]: 2022-05-14T01:40:39.697+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:40:39.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:40:39 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:40:40.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:40:40 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:40:40.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:40:40 smithi047 ceph-mon[77147]: pgmap v4339: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:40:40.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:40:40 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 7970 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:40:40.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:40:40 smithi115 conmon[90393]: 2022-05-14T01:40:40.649+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:40:40.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:40:40 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:40:40.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:40:40 smithi115 ceph-mon[78593]: pgmap v4339: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:40:40.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:40:40 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 7970 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:40:41.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:40:41 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:40:41.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:40:41 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:40:41.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:40:41 smithi115 conmon[90393]: 2022-05-14T01:40:41.694+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:40:41.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:40:41 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:40:41.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:40:41 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:40:42.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:40:42 smithi115 conmon[90393]: 2022-05-14T01:40:42.690+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:40:42.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:40:42 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:40:42.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:40:42 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:40:42.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:40:42 smithi115 ceph-mon[78593]: pgmap v4340: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:40:42.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:40:42 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:40:43.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:40:42 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:40:43.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:40:42 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:40:43.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:40:42 smithi047 ceph-mon[77147]: pgmap v4340: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:40:43.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:40:42 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:40:43.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:40:43 smithi115 conmon[90393]: 2022-05-14T01:40:43.713+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:40:43.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:40:43 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:40:44.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:40:43 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:40:44.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:40:44 smithi115 conmon[90393]: 2022-05-14T01:40:44.761+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:40:44.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:40:44 smithi115 ceph-mon[78593]: pgmap v4341: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:40:44.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:40:44 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:40:45.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:40:44 smithi047 ceph-mon[77147]: pgmap v4341: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:40:45.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:40:44 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:40:45.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:40:45 smithi115 conmon[90393]: 2022-05-14T01:40:45.760+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:40:45.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:40:45 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:40:45.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:40:45 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 7975 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:40:46.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:40:45 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:40:46.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:40:45 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 7975 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:40:46.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:40:46 smithi115 conmon[90393]: 2022-05-14T01:40:46.762+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:40:46.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:40:46 smithi115 ceph-mon[78593]: pgmap v4342: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:40:46.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:40:46 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:40:47.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:40:46 smithi047 ceph-mon[77147]: pgmap v4342: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:40:47.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:40:46 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:40:47.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:40:47 smithi115 conmon[90393]: 2022-05-14T01:40:47.734+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:40:47.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:40:47 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:40:47.872 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:40:47 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:40:48.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:40:48 smithi115 ceph-mon[78593]: pgmap v4343: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:40:48.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:40:48 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:40:48.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:40:48 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T01:40:48.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:40:48 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T01:40:48.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:40:48 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T01:40:48.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:40:48 smithi047 ceph-mon[77147]: pgmap v4343: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:40:48.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:40:48 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:40:48.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:40:48 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T01:40:48.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:40:48 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T01:40:48.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:40:48 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T01:40:49.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:40:48 smithi115 conmon[90393]: 2022-05-14T01:40:48.745+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:40:49.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:40:49 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:40:49.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:40:49 smithi115 conmon[90393]: 2022-05-14T01:40:49.744+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:40:50.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:40:49 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:40:50.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:40:50 smithi115 conmon[90393]: 2022-05-14T01:40:50.760+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:40:50.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:40:50 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:40:50.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:40:50 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:40:50.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:40:50 smithi115 ceph-mon[78593]: pgmap v4344: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:40:50.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:40:50 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:40:50.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:40:50 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 7980 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:40:50.930 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:40:50 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:40:50.930 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:40:50 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:40:50.930 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:40:50 smithi047 ceph-mon[77147]: pgmap v4344: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:40:50.931 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:40:50 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:40:50.931 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:40:50 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 7980 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:40:51.291 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:40:50 smithi047 conmon[78521]: 2022-05-14T01:40:50.928+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:40:51.292 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:40:50 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T01:40:51.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:40:50 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:40:51.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:40:50 smithi047 conmon[78521]: yield 2022-05-14T01:40:51.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:40:50 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:40:51.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:40:50 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:40:51.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:40:50 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:40:51.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:40:50 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T01:40:51.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:40:50 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:40:51.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:40:50 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T01:40:51.297 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:40:50 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:40:51.297 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:40:50 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T01:40:51.297 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:40:50 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:40:51.297 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:40:50 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T01:40:51.298 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:40:50 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:40:51.298 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:40:50 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T01:40:51.298 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:40:50 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:40:51.298 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:40:50 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:40:51.299 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:40:50 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:40:51.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:40:51 smithi115 conmon[90393]: 2022-05-14T01:40:51.712+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:40:51.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:40:51 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:40:52.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:40:51 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:40:52.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:40:52 smithi115 conmon[90393]: 2022-05-14T01:40:52.743+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:40:52.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:40:52 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:40:52.858 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T01:40:52.859 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:40:52.859 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T01:40:52.859 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:40:52.859 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:40:52.860 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:40:52.860 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T01:40:52.860 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:40:52.860 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T01:40:52.860 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:40:52.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T01:40:52.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:40:52.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T01:40:52.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:40:52.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T01:40:52.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:40:52.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:40:52.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:40:52.862 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:40:52 smithi115 ceph-mon[78593]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T01:40:52.863 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:40:52 smithi115 ceph-mon[78593]: pgmap v4345: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:40:52.863 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:40:52 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:40:53.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:40:52 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:40:53.041 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T01:40:53.041 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:40:53.042 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T01:40:53.042 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:40:53.042 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:40:53.042 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:40:53.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T01:40:53.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:40:53.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T01:40:53.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:40:53.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T01:40:53.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:40:53.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T01:40:53.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:40:53.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T01:40:53.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:40:53.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:40:53.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:40:53.045 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:40:52 smithi047 ceph-mon[77147]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T01:40:53.046 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:40:52 smithi047 ceph-mon[77147]: pgmap v4345: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:40:53.046 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:40:52 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:40:53.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:40:53 smithi115 conmon[90393]: 2022-05-14T01:40:53.787+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:40:53.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:40:53 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:40:54.002 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:40:53 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:40:54.291 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:40:54 smithi047 conmon[78521]: 2022-05-14T01:40:53.999+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:40:54.292 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:40:54 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T01:40:54.292 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:40:54 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:40:54.292 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:40:54 smithi047 conmon[78521]: yield 2022-05-14T01:40:54.292 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:40:54 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:40:54.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:40:54 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:40:54.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:40:54 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:40:54.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:40:54 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T01:40:54.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:40:54 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:40:54.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:40:54 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T01:40:54.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:40:54 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:40:54.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:40:54 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T01:40:54.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:40:54 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:40:54.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:40:54 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T01:40:54.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:40:54 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:40:54.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:40:54 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T01:40:54.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:40:54 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:40:54.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:40:54 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:40:54.297 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:40:54 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:40:54.297 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:40:54 smithi047 conmon[78521]: 2022-05-14T01:40:54.000+0000 7f5340779700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T01:40:54.297 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:40:54 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T01:40:54.297 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:40:54 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:40:54.298 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:40:54 smithi047 conmon[78521]: yield 2022-05-14T01:40:54.298 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:40:54 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:40:54.298 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:40:54 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:40:54.298 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:40:54 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:40:54.298 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:40:54 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T01:40:54.299 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:40:54 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:40:54.299 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:40:54 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T01:40:54.299 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:40:54 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:40:54.299 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:40:54 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T01:40:54.300 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:40:54 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:40:54.300 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:40:54 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T01:40:54.300 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:40:54 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:40:54.300 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:40:54 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T01:40:54.300 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:40:54 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:40:54.301 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:40:54 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:40:54.301 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:40:54 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:40:54.301 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:40:54 smithi047 conmon[78521]: 2022-05-14T01:40:54.301 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:40:54 smithi047 conmon[78521]: During handling of the above exception, another exception occurred: 2022-05-14T01:40:54.302 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:40:54 smithi047 conmon[78521]: 2022-05-14T01:40:54.302 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:40:54 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T01:40:54.302 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:40:54 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T01:40:54.302 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:40:54 smithi047 conmon[78521]: return f(*arg) 2022-05-14T01:40:54.302 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:40:54 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T01:40:54.303 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:40:54 smithi047 conmon[78521]: self._write_client_files(client_files, host) 2022-05-14T01:40:54.303 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:40:54 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T01:40:54.303 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:40:54 smithi047 conmon[78521]: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T01:40:54.303 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:40:54 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T01:40:54.304 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:40:54 smithi047 conmon[78521]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T01:40:54.304 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:40:54 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T01:40:54.304 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:40:54 smithi047 conmon[78521]: return self.event_loop.get_result(coro) 2022-05-14T01:40:54.304 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:40:54 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T01:40:54.304 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:40:54 smithi047 conmon[78521]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T01:40:54.305 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:40:54 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T01:40:54.305 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:40:54 smithi047 conmon[78521]: return self.__get_result() 2022-05-14T01:40:54.305 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:40:54 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T01:40:54.305 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:40:54 smithi047 conmon[78521]: raise self._exception 2022-05-14T01:40:54.305 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:40:54 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T01:40:54.306 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:40:54 smithi047 conmon[78521]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T01:40:54.306 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:40:54 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T01:40:54.306 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:40:54 smithi047 conmon[78521]: conn = await self._remote_connection(host, addr) 2022-05-14T01:40:54.306 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:40:54 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T01:40:54.306 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:40:54 smithi047 conmon[78521]: raise 2022-05-14T01:40:54.307 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:40:54 smithi047 conmon[78521]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T01:40:54.307 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:40:54 smithi047 conmon[78521]: self.gen.throw(type, value, traceback) 2022-05-14T01:40:54.307 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:40:54 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T01:40:54.307 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:40:54 smithi047 conmon[78521]: raise OrchestratorError(msg) 2022-05-14T01:40:54.308 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:40:54 smithi047 conmon[78521]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:40:54.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:40:54 smithi115 conmon[90393]: 2022-05-14T01:40:54.818+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:40:54.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:40:54 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:40:54.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:40:54 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:40:54.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:40:54 smithi115 ceph-mon[78593]: pgmap v4346: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:40:54.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:40:54 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:40:55.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:40:54 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:40:55.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:40:54 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:40:55.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:40:54 smithi047 ceph-mon[77147]: pgmap v4346: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:40:55.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:40:54 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:40:55.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:40:55 smithi115 conmon[90393]: 2022-05-14T01:40:55.794+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:40:55.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:40:55 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:40:55.859 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T01:40:55.859 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:40:55.859 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T01:40:55.859 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:40:55.859 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:40:55.860 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:40:55.860 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T01:40:55.860 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:40:55.860 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T01:40:55.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:40:55.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T01:40:55.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:40:55.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T01:40:55.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:40:55.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T01:40:55.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:40:55.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:40:55.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:40:55.863 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:40:55 smithi115 ceph-mon[78593]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T01:40:55.863 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T01:40:55.863 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:40:55.863 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T01:40:55.863 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:40:55.864 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:40:55.864 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:40:55.864 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T01:40:55.864 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:40:55.865 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T01:40:55.865 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:40:55.865 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T01:40:55.865 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:40:55.866 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T01:40:55.866 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:40:55.866 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T01:40:55.866 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:40:55.866 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:40:55.867 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:40:55.867 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T01:40:55.867 INFO:journalctl@ceph.mon.c.smithi115.stdout: During handling of the above exception, another exception occurred: 2022-05-14T01:40:55.867 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T01:40:55.868 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T01:40:55.868 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T01:40:55.868 INFO:journalctl@ceph.mon.c.smithi115.stdout: return f(*arg) 2022-05-14T01:40:55.868 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T01:40:55.868 INFO:journalctl@ceph.mon.c.smithi115.stdout: self._write_client_files(client_files, host) 2022-05-14T01:40:55.869 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T01:40:55.869 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T01:40:55.869 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T01:40:55.869 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T01:40:55.870 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T01:40:55.870 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.event_loop.get_result(coro) 2022-05-14T01:40:55.870 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T01:40:55.870 INFO:journalctl@ceph.mon.c.smithi115.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T01:40:55.870 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T01:40:55.871 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.__get_result() 2022-05-14T01:40:55.871 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T01:40:55.871 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise self._exception 2022-05-14T01:40:55.871 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T01:40:55.872 INFO:journalctl@ceph.mon.c.smithi115.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T01:40:55.872 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T01:40:55.872 INFO:journalctl@ceph.mon.c.smithi115.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T01:40:55.872 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T01:40:55.872 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise 2022-05-14T01:40:55.873 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T01:40:55.873 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.gen.throw(type, value, traceback) 2022-05-14T01:40:55.873 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T01:40:55.873 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OrchestratorError(msg) 2022-05-14T01:40:55.874 INFO:journalctl@ceph.mon.c.smithi115.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:40:55.874 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:40:55 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:40:55.874 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:40:55 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 7985 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:40:56.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:40:55 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:40:56.041 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T01:40:56.042 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:40:56.042 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T01:40:56.042 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:40:56.042 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:40:56.042 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:40:56.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T01:40:56.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:40:56.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T01:40:56.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:40:56.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T01:40:56.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:40:56.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T01:40:56.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:40:56.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T01:40:56.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:40:56.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:40:56.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:40:56.045 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:40:55 smithi047 ceph-mon[77147]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T01:40:56.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T01:40:56.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:40:56.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T01:40:56.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:40:56.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:40:56.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:40:56.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T01:40:56.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:40:56.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T01:40:56.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:40:56.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T01:40:56.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:40:56.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T01:40:56.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:40:56.049 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T01:40:56.049 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:40:56.049 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:40:56.049 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:40:56.049 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T01:40:56.050 INFO:journalctl@ceph.mon.b.smithi047.stdout: During handling of the above exception, another exception occurred: 2022-05-14T01:40:56.050 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T01:40:56.050 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T01:40:56.050 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T01:40:56.051 INFO:journalctl@ceph.mon.b.smithi047.stdout: return f(*arg) 2022-05-14T01:40:56.051 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T01:40:56.051 INFO:journalctl@ceph.mon.b.smithi047.stdout: self._write_client_files(client_files, host) 2022-05-14T01:40:56.051 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T01:40:56.051 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T01:40:56.052 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T01:40:56.052 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T01:40:56.052 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T01:40:56.052 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.event_loop.get_result(coro) 2022-05-14T01:40:56.053 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T01:40:56.053 INFO:journalctl@ceph.mon.b.smithi047.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T01:40:56.053 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T01:40:56.053 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.__get_result() 2022-05-14T01:40:56.053 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T01:40:56.054 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise self._exception 2022-05-14T01:40:56.054 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T01:40:56.054 INFO:journalctl@ceph.mon.b.smithi047.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T01:40:56.054 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T01:40:56.055 INFO:journalctl@ceph.mon.b.smithi047.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T01:40:56.055 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T01:40:56.055 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise 2022-05-14T01:40:56.055 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T01:40:56.056 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.gen.throw(type, value, traceback) 2022-05-14T01:40:56.056 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T01:40:56.056 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OrchestratorError(msg) 2022-05-14T01:40:56.056 INFO:journalctl@ceph.mon.b.smithi047.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:40:56.057 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:40:55 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:40:56.057 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:40:55 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 7985 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:40:56.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:40:56 smithi115 conmon[90393]: 2022-05-14T01:40:56.762+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:40:56.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:40:56 smithi115 ceph-mon[78593]: pgmap v4347: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:40:56.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:40:56 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:40:57.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:40:56 smithi047 ceph-mon[77147]: pgmap v4347: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:40:57.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:40:56 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:40:58.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:40:57 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:40:58.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:40:57 smithi115 conmon[90393]: 2022-05-14T01:40:57.762+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:40:58.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:40:57 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:40:59.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:40:58 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:40:59.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:40:58 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:40:59.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:40:58 smithi047 ceph-mon[77147]: pgmap v4348: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:40:59.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:40:58 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:40:59.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:40:58 smithi115 conmon[90393]: 2022-05-14T01:40:58.757+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:40:59.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:40:58 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:40:59.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:40:58 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:40:59.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:40:58 smithi115 ceph-mon[78593]: pgmap v4348: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:40:59.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:40:58 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:41:00.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:40:59 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:41:00.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:40:59 smithi115 conmon[90393]: 2022-05-14T01:40:59.729+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:41:00.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:40:59 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:41:01.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:41:00 smithi047 ceph-mon[77147]: pgmap v4349: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:41:01.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:41:00 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:41:01.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:41:00 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 7990 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:41:01.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:41:00 smithi115 conmon[90393]: 2022-05-14T01:41:00.761+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:41:01.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:41:00 smithi115 ceph-mon[78593]: pgmap v4349: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:41:01.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:41:00 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:41:01.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:41:00 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 7990 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:41:02.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:41:01 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:41:02.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:41:01 smithi115 conmon[90393]: 2022-05-14T01:41:01.761+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:41:02.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:41:01 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:41:03.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:41:02 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:41:03.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:41:02 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:41:03.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:41:02 smithi047 ceph-mon[77147]: pgmap v4350: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:41:03.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:41:02 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:41:03.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:41:02 smithi115 conmon[90393]: 2022-05-14T01:41:02.718+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:41:03.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:41:02 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:41:03.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:41:02 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:41:03.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:41:02 smithi115 ceph-mon[78593]: pgmap v4350: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:41:03.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:41:02 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:41:04.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:41:03 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:41:04.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:41:03 smithi115 conmon[90393]: 2022-05-14T01:41:03.669+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:41:04.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:41:03 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:41:05.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:41:04 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:41:05.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:41:04 smithi047 ceph-mon[77147]: pgmap v4351: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:41:05.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:41:04 smithi115 conmon[90393]: 2022-05-14T01:41:04.682+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:41:05.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:41:04 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:41:05.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:41:04 smithi115 ceph-mon[78593]: pgmap v4351: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:41:06.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:41:05 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:41:06.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:41:05 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 7995 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:41:06.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:41:05 smithi115 conmon[90393]: 2022-05-14T01:41:05.680+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:41:06.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:41:05 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:41:06.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:41:05 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 7995 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:41:07.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:41:06 smithi047 ceph-mon[77147]: pgmap v4352: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:41:07.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:41:06 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:41:07.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:41:06 smithi115 conmon[90393]: 2022-05-14T01:41:06.715+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:41:07.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:41:06 smithi115 ceph-mon[78593]: pgmap v4352: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:41:07.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:41:06 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:41:08.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:41:07 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:41:08.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:41:07 smithi115 conmon[90393]: 2022-05-14T01:41:07.691+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:41:08.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:41:07 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:41:09.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:41:08 smithi047 ceph-mon[77147]: pgmap v4353: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:41:09.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:41:08 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:41:09.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:41:08 smithi115 conmon[90393]: 2022-05-14T01:41:08.735+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:41:09.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:41:08 smithi115 ceph-mon[78593]: pgmap v4353: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:41:09.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:41:08 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:41:10.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:41:09 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:41:10.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:41:09 smithi115 conmon[90393]: 2022-05-14T01:41:09.763+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:41:10.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:41:09 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:41:11.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:41:10 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:41:11.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:41:10 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:41:11.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:41:10 smithi047 ceph-mon[77147]: pgmap v4354: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:41:11.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:41:10 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:41:11.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:41:10 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 8000 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:41:11.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:41:10 smithi115 conmon[90393]: 2022-05-14T01:41:10.798+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:41:11.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:41:10 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:41:11.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:41:10 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:41:11.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:41:10 smithi115 ceph-mon[78593]: pgmap v4354: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:41:11.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:41:10 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:41:11.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:41:10 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 8000 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:41:12.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:41:11 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:41:12.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:41:11 smithi115 conmon[90393]: 2022-05-14T01:41:11.749+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:41:12.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:41:11 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:41:13.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:41:12 smithi047 ceph-mon[77147]: pgmap v4355: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:41:13.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:41:12 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:41:13.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:41:12 smithi115 conmon[90393]: 2022-05-14T01:41:12.769+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:41:13.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:41:12 smithi115 ceph-mon[78593]: pgmap v4355: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:41:13.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:41:12 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:41:14.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:41:13 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:41:14.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:41:13 smithi115 conmon[90393]: 2022-05-14T01:41:13.773+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:41:14.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:41:13 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:41:15.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:41:14 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:41:15.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:41:14 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:41:15.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:41:14 smithi047 ceph-mon[77147]: pgmap v4356: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:41:15.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:41:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T01:41:15.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:41:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T01:41:15.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:41:14 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:41:15.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:41:14 smithi115 conmon[90393]: 2022-05-14T01:41:14.805+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:41:15.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:41:14 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:41:15.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:41:14 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:41:15.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:41:14 smithi115 ceph-mon[78593]: pgmap v4356: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:41:15.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:41:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T01:41:15.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:41:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T01:41:15.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:41:14 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:41:15.681 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:41:15 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:41:15.682 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:41:15 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 8005 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:41:16.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:41:15 smithi115 conmon[90393]: 2022-05-14T01:41:15.824+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:41:16.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:41:15 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:41:16.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:41:15 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 8005 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:41:17.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:41:16 smithi047 ceph-mon[77147]: pgmap v4357: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:41:17.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:41:16 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:41:17.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:41:16 smithi115 conmon[90393]: 2022-05-14T01:41:16.787+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:41:17.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:41:16 smithi115 ceph-mon[78593]: pgmap v4357: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:41:17.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:41:16 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:41:18.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:41:17 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:41:18.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:41:17 smithi115 conmon[90393]: 2022-05-14T01:41:17.749+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:41:18.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:41:17 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:41:19.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:41:18 smithi047 ceph-mon[77147]: pgmap v4358: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:41:19.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:41:18 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:41:19.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:41:18 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:41:19.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:41:18 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:41:19.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:41:18 smithi115 conmon[90393]: 2022-05-14T01:41:18.769+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:41:19.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:41:18 smithi115 ceph-mon[78593]: pgmap v4358: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:41:19.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:41:18 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:41:19.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:41:18 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:41:19.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:41:18 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:41:20.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:41:19 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:41:20.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:41:19 smithi115 conmon[90393]: 2022-05-14T01:41:19.758+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:41:20.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:41:19 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:41:21.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:41:20 smithi047 ceph-mon[77147]: pgmap v4359: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:41:21.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:41:20 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:41:21.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:41:20 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 8015 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:41:21.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:41:20 smithi115 conmon[90393]: 2022-05-14T01:41:20.771+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:41:21.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:41:20 smithi115 ceph-mon[78593]: pgmap v4359: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:41:21.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:41:20 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:41:21.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:41:20 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 8015 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:41:22.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:41:21 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:41:22.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:41:21 smithi115 conmon[90393]: 2022-05-14T01:41:21.749+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:41:22.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:41:21 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:41:23.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:41:22 smithi047 ceph-mon[77147]: pgmap v4360: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:41:23.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:41:22 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:41:23.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:41:22 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:41:23.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:41:22 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:41:23.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:41:22 smithi115 conmon[90393]: 2022-05-14T01:41:22.727+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:41:23.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:41:22 smithi115 ceph-mon[78593]: pgmap v4360: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:41:23.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:41:22 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:41:23.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:41:22 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:41:23.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:41:22 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:41:24.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:41:23 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:41:24.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:41:23 smithi115 conmon[90393]: 2022-05-14T01:41:23.732+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:41:24.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:41:23 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:41:25.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:41:24 smithi047 ceph-mon[77147]: pgmap v4361: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:41:25.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:41:24 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:41:25.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:41:24 smithi115 conmon[90393]: 2022-05-14T01:41:24.758+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:41:25.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:41:24 smithi115 ceph-mon[78593]: pgmap v4361: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:41:25.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:41:24 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:41:25.999 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:41:25 smithi115 conmon[90393]: 2022-05-14T01:41:25.745+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:41:26.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:41:25 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:41:26.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:41:25 smithi047 ceph-mon[77147]: pgmap v4362: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:41:26.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:41:25 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:41:26.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:41:25 smithi115 ceph-mon[78593]: pgmap v4362: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:41:27.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:41:26 smithi115 conmon[90393]: 2022-05-14T01:41:26.786+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:41:27.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:41:27 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:41:27.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:41:27 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 8020 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:41:27.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:41:26 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:41:27.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:41:26 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 8020 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:41:28.005 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:41:27 smithi115 conmon[90393]: 2022-05-14T01:41:27.753+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:41:28.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:41:28 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:41:28.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:41:28 smithi047 ceph-mon[77147]: pgmap v4363: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:41:28.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:41:28 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:41:28.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:41:28 smithi115 ceph-mon[78593]: pgmap v4363: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:41:29.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:41:29 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:41:29.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:41:28 smithi115 conmon[90393]: 2022-05-14T01:41:28.767+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:41:29.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:41:29 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:41:30.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:41:29 smithi115 conmon[90393]: 2022-05-14T01:41:29.781+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:41:30.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:41:30 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:41:30.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:41:30 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:41:30.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:41:30 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:41:30.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:41:30 smithi115 ceph-mon[78593]: pgmap v4364: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:41:30.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:41:30 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:41:30.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:41:30 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:41:30.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:41:30 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:41:30.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:41:30 smithi047 ceph-mon[77147]: pgmap v4364: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:41:31.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:41:30 smithi115 conmon[90393]: 2022-05-14T01:41:30.807+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:41:31.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:41:31 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:41:31.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:41:31 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:41:32.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:41:31 smithi115 conmon[90393]: 2022-05-14T01:41:31.795+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:41:32.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:41:32 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:41:32.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:41:32 smithi115 ceph-mon[78593]: pgmap v4365: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:41:32.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:41:32 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:41:32.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:41:32 smithi047 ceph-mon[77147]: pgmap v4365: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:41:33.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:41:32 smithi115 conmon[90393]: 2022-05-14T01:41:32.776+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:41:33.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:41:33 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:41:33.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:41:33 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:41:34.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:41:33 smithi115 conmon[90393]: 2022-05-14T01:41:33.812+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:41:34.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:41:34 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:41:34.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:41:34 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:41:34.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:41:34 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:41:34.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:41:34 smithi115 ceph-mon[78593]: pgmap v4366: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:41:34.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:41:34 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:41:34.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:41:34 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:41:34.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:41:34 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:41:34.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:41:34 smithi047 ceph-mon[77147]: pgmap v4366: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:41:35.033 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:41:34 smithi115 conmon[90393]: 2022-05-14T01:41:34.767+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:41:35.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:41:35 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:41:35.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:41:35 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 8025 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:41:35.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:41:35 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:41:35.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:41:35 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 8025 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:41:36.037 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:41:35 smithi115 conmon[90393]: 2022-05-14T01:41:35.725+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:41:36.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:41:36 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:41:36.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:41:36 smithi047 ceph-mon[77147]: pgmap v4367: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:41:36.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:41:36 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:41:36.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:41:36 smithi115 ceph-mon[78593]: pgmap v4367: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:41:37.039 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:41:36 smithi115 conmon[90393]: 2022-05-14T01:41:36.726+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:41:37.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:41:37 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:41:37.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:41:37 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:41:38.048 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:41:37 smithi115 conmon[90393]: 2022-05-14T01:41:37.700+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:41:38.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:41:38 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:41:38.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:41:38 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:41:38.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:41:38 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:41:38.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:41:38 smithi115 ceph-mon[78593]: pgmap v4368: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:41:38.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:41:38 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:41:38.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:41:38 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:41:38.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:41:38 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:41:38.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:41:38 smithi047 ceph-mon[77147]: pgmap v4368: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:41:39.049 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:41:38 smithi115 conmon[90393]: 2022-05-14T01:41:38.669+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:41:39.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:41:39 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:41:39.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:41:39 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:41:40.053 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:41:39 smithi115 conmon[90393]: 2022-05-14T01:41:39.645+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:41:40.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:41:40 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:41:40.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:41:40 smithi115 ceph-mon[78593]: pgmap v4369: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:41:40.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:41:40 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 8030 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:41:40.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:41:40 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:41:40.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:41:40 smithi047 ceph-mon[77147]: pgmap v4369: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:41:40.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:41:40 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 8030 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:41:41.055 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:41:40 smithi115 conmon[90393]: 2022-05-14T01:41:40.692+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:41:41.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:41:41 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:41:41.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:41:41 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:41:42.065 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:41:41 smithi115 conmon[90393]: 2022-05-14T01:41:41.706+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:41:42.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:41:42 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:41:42.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:41:42 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:41:42.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:41:42 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:41:42.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:41:42 smithi115 ceph-mon[78593]: pgmap v4370: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:41:42.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:41:42 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:41:42.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:41:42 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:41:42.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:41:42 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:41:42.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:41:42 smithi047 ceph-mon[77147]: pgmap v4370: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:41:43.066 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:41:42 smithi115 conmon[90393]: 2022-05-14T01:41:42.680+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:41:43.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:41:43 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:41:43.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:41:43 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:41:44.071 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:41:43 smithi115 conmon[90393]: 2022-05-14T01:41:43.678+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:41:44.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:41:44 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:41:44.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:41:44 smithi115 ceph-mon[78593]: pgmap v4371: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:41:44.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:41:44 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:41:44.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:41:44 smithi047 ceph-mon[77147]: pgmap v4371: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:41:45.072 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:41:44 smithi115 conmon[90393]: 2022-05-14T01:41:44.634+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:41:45.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:41:45 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:41:45.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:41:45 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 8035 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:41:45.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:41:45 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:41:45.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:41:45 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 8035 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:41:45.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:41:45 smithi115 conmon[90393]: 2022-05-14T01:41:45.591+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:41:46.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:41:46 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:41:46.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:41:46 smithi115 ceph-mon[78593]: pgmap v4372: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:41:46.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:41:46 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:41:46.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:41:46 smithi047 ceph-mon[77147]: pgmap v4372: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:41:46.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:41:46 smithi115 conmon[90393]: 2022-05-14T01:41:46.548+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:41:47.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:41:47 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:41:47.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:41:47 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:41:47.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:41:47 smithi115 conmon[90393]: 2022-05-14T01:41:47.557+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:41:48.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:41:48 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:41:48.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:41:48 smithi115 ceph-mon[78593]: pgmap v4373: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:41:48.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:41:48 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:41:48.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:41:48 smithi047 ceph-mon[77147]: pgmap v4373: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:41:48.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:41:48 smithi115 conmon[90393]: 2022-05-14T01:41:48.586+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:41:49.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:41:49 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:41:49.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:41:49 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:41:49.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:41:49 smithi115 conmon[90393]: 2022-05-14T01:41:49.557+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:41:50.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:41:50 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:41:50.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:41:50 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:41:50.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:41:50 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:41:50.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:41:50 smithi115 ceph-mon[78593]: pgmap v4374: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:41:50.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:41:50 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 8040 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:41:50.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:41:50 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:41:50.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:41:50 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:41:50.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:41:50 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:41:50.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:41:50 smithi047 ceph-mon[77147]: pgmap v4374: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:41:50.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:41:50 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 8040 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:41:50.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:41:50 smithi115 conmon[90393]: 2022-05-14T01:41:50.571+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:41:51.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:41:51 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:41:51.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:41:51 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:41:51.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:41:51 smithi115 conmon[90393]: 2022-05-14T01:41:51.571+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:41:52.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:41:52 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:41:52.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:41:52 smithi115 ceph-mon[78593]: pgmap v4375: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:41:52.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:41:52 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:41:52.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:41:52 smithi047 ceph-mon[77147]: pgmap v4375: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:41:53.102 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:41:52 smithi115 conmon[90393]: 2022-05-14T01:41:52.610+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:41:53.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:41:53 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:41:53.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:41:53 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:41:54.057 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:41:53 smithi115 conmon[90393]: 2022-05-14T01:41:53.645+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:41:54.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:41:54 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:41:54.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:41:54 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:41:54.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:41:54 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:41:54.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:41:54 smithi115 ceph-mon[78593]: pgmap v4376: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:41:54.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:41:54 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T01:41:54.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:41:54 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T01:41:54.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:41:54 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T01:41:54.481 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:41:54 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:41:54.481 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:41:54 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:41:54.481 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:41:54 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:41:54.482 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:41:54 smithi047 ceph-mon[77147]: pgmap v4376: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:41:54.482 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:41:54 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T01:41:54.482 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:41:54 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T01:41:54.482 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:41:54 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T01:41:54.791 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:41:54 smithi047 conmon[78521]: 2022-05-14T01:41:54.479+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:41:54.791 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:41:54 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T01:41:54.792 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:41:54 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:41:54.792 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:41:54 smithi047 conmon[78521]: yield 2022-05-14T01:41:54.792 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:41:54 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:41:54.792 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:41:54 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:41:54.793 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:41:54 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:41:54.793 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:41:54 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T01:41:54.793 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:41:54 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:41:54.793 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:41:54 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T01:41:54.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:41:54 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:41:54.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:41:54 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T01:41:54.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:41:54 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:41:54.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:41:54 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T01:41:54.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:41:54 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:41:54.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:41:54 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T01:41:54.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:41:54 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:41:54.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:41:54 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:41:54.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:41:54 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:41:54.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:41:54 smithi115 conmon[90393]: 2022-05-14T01:41:54.688+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:41:55.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:41:55 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:41:55.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:41:55 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:41:55.542 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T01:41:55.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:41:55.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T01:41:55.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:41:55.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:41:55.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:41:55.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T01:41:55.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:41:55.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T01:41:55.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:41:55.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T01:41:55.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:41:55.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T01:41:55.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:41:55.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T01:41:55.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:41:55.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:41:55.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:41:55.547 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:41:55 smithi047 ceph-mon[77147]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T01:41:55.547 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:41:55 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 8045 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:41:55.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:41:55 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:41:55.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:41:55 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:41:55.609 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T01:41:55.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:41:55.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T01:41:55.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:41:55.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:41:55.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:41:55.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T01:41:55.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:41:55.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T01:41:55.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:41:55.613 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T01:41:55.613 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:41:55.613 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T01:41:55.613 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:41:55.614 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T01:41:55.614 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:41:55.614 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:41:55.614 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:41:55.615 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:41:55 smithi115 ceph-mon[78593]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T01:41:55.615 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:41:55 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 8045 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:41:56.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:41:55 smithi115 conmon[90393]: 2022-05-14T01:41:55.672+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:41:56.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:41:56 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:41:56.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:41:56 smithi047 ceph-mon[77147]: pgmap v4377: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:41:56.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:41:56 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:41:56.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:41:56 smithi115 ceph-mon[78593]: pgmap v4377: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:41:57.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:41:56 smithi115 conmon[90393]: 2022-05-14T01:41:56.661+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:41:57.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:41:57 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:41:57.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:41:57 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:41:58.041 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:41:57 smithi047 conmon[78521]: 2022-05-14T01:41:57.551+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:41:58.042 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:41:57 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T01:41:58.042 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:41:57 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:41:58.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:41:57 smithi047 conmon[78521]: yield 2022-05-14T01:41:58.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:41:57 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:41:58.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:41:57 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:41:58.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:41:57 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:41:58.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:41:57 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T01:41:58.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:41:57 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:41:58.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:41:57 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T01:41:58.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:41:57 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:41:58.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:41:57 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T01:41:58.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:41:57 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:41:58.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:41:57 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T01:41:58.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:41:57 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:41:58.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:41:57 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T01:41:58.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:41:57 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:41:58.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:41:57 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:41:58.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:41:57 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:41:58.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:41:57 smithi047 conmon[78521]: 2022-05-14T01:41:57.552+0000 7f5344f82700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T01:41:58.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:41:57 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T01:41:58.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:41:57 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:41:58.048 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:41:57 smithi047 conmon[78521]: yield 2022-05-14T01:41:58.048 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:41:57 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:41:58.048 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:41:57 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:41:58.049 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:41:57 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:41:58.049 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:41:57 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T01:41:58.049 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:41:57 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:41:58.049 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:41:57 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T01:41:58.050 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:41:57 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:41:58.050 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:41:57 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T01:41:58.050 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:41:57 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:41:58.050 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:41:57 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T01:41:58.050 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:41:57 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:41:58.051 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:41:57 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T01:41:58.051 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:41:57 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:41:58.051 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:41:57 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:41:58.051 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:41:57 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:41:58.051 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:41:57 smithi047 conmon[78521]: 2022-05-14T01:41:58.052 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:41:57 smithi047 conmon[78521]: During handling of the above exception, another exception occurred: 2022-05-14T01:41:58.052 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:41:57 smithi047 conmon[78521]: 2022-05-14T01:41:58.052 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:41:57 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T01:41:58.052 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:41:57 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T01:41:58.053 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:41:57 smithi047 conmon[78521]: return f(*arg) 2022-05-14T01:41:58.053 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:41:57 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T01:41:58.053 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:41:57 smithi047 conmon[78521]: self._write_client_files(client_files, host) 2022-05-14T01:41:58.054 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:41:57 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T01:41:58.054 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:41:57 smithi047 conmon[78521]: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T01:41:58.054 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:41:57 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T01:41:58.054 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:41:57 smithi047 conmon[78521]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T01:41:58.055 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:41:57 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T01:41:58.055 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:41:57 smithi047 conmon[78521]: return self.event_loop.get_result(coro) 2022-05-14T01:41:58.055 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:41:57 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T01:41:58.055 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:41:57 smithi047 conmon[78521]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T01:41:58.055 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:41:57 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T01:41:58.056 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:41:57 smithi047 conmon[78521]: return self.__get_result() 2022-05-14T01:41:58.056 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:41:57 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T01:41:58.056 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:41:57 smithi047 conmon[78521]: raise self._exception 2022-05-14T01:41:58.056 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:41:57 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T01:41:58.057 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:41:57 smithi047 conmon[78521]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T01:41:58.057 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:41:57 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T01:41:58.057 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:41:57 smithi047 conmon[78521]: conn = await self._remote_connection(host, addr) 2022-05-14T01:41:58.057 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:41:57 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T01:41:58.058 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:41:57 smithi047 conmon[78521]: raise 2022-05-14T01:41:58.058 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:41:57 smithi047 conmon[78521]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T01:41:58.058 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:41:57 smithi047 conmon[78521]: self.gen.throw(type, value, traceback) 2022-05-14T01:41:58.058 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:41:57 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T01:41:58.059 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:41:57 smithi047 conmon[78521]: raise OrchestratorError(msg) 2022-05-14T01:41:58.059 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:41:57 smithi047 conmon[78521]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:41:58.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:41:57 smithi115 conmon[90393]: 2022-05-14T01:41:57.621+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:41:58.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:41:58 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:41:58.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:41:58 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:41:58.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:41:58 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:41:58.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:41:58 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:41:58.542 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T01:41:58.542 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:41:58.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T01:41:58.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:41:58.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:41:58.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:41:58.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T01:41:58.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:41:58.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T01:41:58.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:41:58.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T01:41:58.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:41:58.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T01:41:58.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:41:58.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T01:41:58.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:41:58.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:41:58.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:41:58.546 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:41:58 smithi047 ceph-mon[77147]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T01:41:58.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T01:41:58.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:41:58.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T01:41:58.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:41:58.548 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:41:58.548 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:41:58.548 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T01:41:58.548 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:41:58.548 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T01:41:58.549 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:41:58.549 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T01:41:58.549 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:41:58.550 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T01:41:58.550 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:41:58.550 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T01:41:58.550 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:41:58.551 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:41:58.551 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:41:58.551 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T01:41:58.551 INFO:journalctl@ceph.mon.b.smithi047.stdout: During handling of the above exception, another exception occurred: 2022-05-14T01:41:58.551 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T01:41:58.552 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T01:41:58.552 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T01:41:58.552 INFO:journalctl@ceph.mon.b.smithi047.stdout: return f(*arg) 2022-05-14T01:41:58.552 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T01:41:58.553 INFO:journalctl@ceph.mon.b.smithi047.stdout: self._write_client_files(client_files, host) 2022-05-14T01:41:58.553 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T01:41:58.553 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T01:41:58.553 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T01:41:58.554 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T01:41:58.554 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T01:41:58.554 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.event_loop.get_result(coro) 2022-05-14T01:41:58.554 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T01:41:58.554 INFO:journalctl@ceph.mon.b.smithi047.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T01:41:58.555 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T01:41:58.555 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.__get_result() 2022-05-14T01:41:58.555 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T01:41:58.555 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise self._exception 2022-05-14T01:41:58.556 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T01:41:58.556 INFO:journalctl@ceph.mon.b.smithi047.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T01:41:58.556 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T01:41:58.556 INFO:journalctl@ceph.mon.b.smithi047.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T01:41:58.557 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T01:41:58.557 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise 2022-05-14T01:41:58.557 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T01:41:58.557 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.gen.throw(type, value, traceback) 2022-05-14T01:41:58.557 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T01:41:58.558 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OrchestratorError(msg) 2022-05-14T01:41:58.558 INFO:journalctl@ceph.mon.b.smithi047.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:41:58.558 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:41:58 smithi047 ceph-mon[77147]: pgmap v4378: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:41:58.586 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:41:58 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:41:58.586 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:41:58 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:41:58.587 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:41:58 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:41:58.587 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:41:58 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:41:58.587 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T01:41:58.587 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:41:58.588 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T01:41:58.588 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:41:58.588 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:41:58.588 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:41:58.589 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T01:41:58.589 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:41:58.589 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T01:41:58.589 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:41:58.590 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T01:41:58.590 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:41:58.590 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T01:41:58.590 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:41:58.590 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T01:41:58.591 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:41:58.591 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:41:58.591 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:41:58.591 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:41:58 smithi115 ceph-mon[78593]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T01:41:58.591 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T01:41:58.592 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:41:58.592 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T01:41:58.592 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:41:58.592 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:41:58.592 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:41:58.593 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T01:41:58.593 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:41:58.593 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T01:41:58.593 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:41:58.594 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T01:41:58.594 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:41:58.594 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T01:41:58.594 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:41:58.595 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T01:41:58.595 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:41:58.595 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:41:58.595 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:41:58.596 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T01:41:58.596 INFO:journalctl@ceph.mon.c.smithi115.stdout: During handling of the above exception, another exception occurred: 2022-05-14T01:41:58.596 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T01:41:58.596 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T01:41:58.596 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T01:41:58.597 INFO:journalctl@ceph.mon.c.smithi115.stdout: return f(*arg) 2022-05-14T01:41:58.597 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T01:41:58.597 INFO:journalctl@ceph.mon.c.smithi115.stdout: self._write_client_files(client_files, host) 2022-05-14T01:41:58.597 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T01:41:58.598 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T01:41:58.598 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T01:41:58.598 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T01:41:58.598 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T01:41:58.598 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.event_loop.get_result(coro) 2022-05-14T01:41:58.599 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T01:41:58.599 INFO:journalctl@ceph.mon.c.smithi115.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T01:41:58.599 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T01:41:58.599 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.__get_result() 2022-05-14T01:41:58.600 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T01:41:58.600 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise self._exception 2022-05-14T01:41:58.600 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T01:41:58.600 INFO:journalctl@ceph.mon.c.smithi115.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T01:41:58.600 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T01:41:58.601 INFO:journalctl@ceph.mon.c.smithi115.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T01:41:58.601 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T01:41:58.601 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise 2022-05-14T01:41:58.601 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T01:41:58.602 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.gen.throw(type, value, traceback) 2022-05-14T01:41:58.602 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T01:41:58.602 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OrchestratorError(msg) 2022-05-14T01:41:58.602 INFO:journalctl@ceph.mon.c.smithi115.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:41:58.602 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:41:58 smithi115 ceph-mon[78593]: pgmap v4378: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:41:58.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:41:58 smithi115 conmon[90393]: 2022-05-14T01:41:58.584+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:41:59.538 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:41:59 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:41:59.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:41:59 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:41:59.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:41:59 smithi115 conmon[90393]: 2022-05-14T01:41:59.537+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:42:00.504 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:42:00 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:42:00.504 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:42:00 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:42:00.504 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:42:00 smithi115 ceph-mon[78593]: pgmap v4379: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:42:00.505 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:42:00 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 8050 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:42:00.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:42:00 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:42:00.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:42:00 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:42:00.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:42:00 smithi047 ceph-mon[77147]: pgmap v4379: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:42:00.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:42:00 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 8050 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:42:00.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:42:00 smithi115 conmon[90393]: 2022-05-14T01:42:00.502+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:42:01.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:42:01 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:42:01.550 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:42:01 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:42:01.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:42:01 smithi115 conmon[90393]: 2022-05-14T01:42:01.549+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:42:02.538 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:42:02 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:42:02.538 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:42:02 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:42:02.538 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:42:02 smithi115 ceph-mon[78593]: pgmap v4380: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:42:02.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:42:02 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:42:02.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:42:02 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:42:02.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:42:02 smithi047 ceph-mon[77147]: pgmap v4380: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:42:02.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:42:02 smithi115 conmon[90393]: 2022-05-14T01:42:02.536+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:42:03.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:42:03 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:42:03.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:42:03 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:42:03.588 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:42:03 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:42:03.588 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:42:03 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:42:03.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:42:03 smithi115 conmon[90393]: 2022-05-14T01:42:03.587+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:42:04.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:42:04 smithi047 ceph-mon[77147]: pgmap v4381: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:42:04.578 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:42:04 smithi115 ceph-mon[78593]: pgmap v4381: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:42:04.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:42:04 smithi115 conmon[90393]: 2022-05-14T01:42:04.577+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:42:05.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:42:05 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:42:05.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:42:05 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 8055 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:42:05.579 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:42:05 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:42:05.579 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:42:05 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 8055 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:42:05.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:42:05 smithi115 conmon[90393]: 2022-05-14T01:42:05.578+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:42:06.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:42:06 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:42:06.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:42:06 smithi047 ceph-mon[77147]: pgmap v4382: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:42:06.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:42:06 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:42:06.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:42:06 smithi115 ceph-mon[78593]: pgmap v4382: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:42:06.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:42:06 smithi115 conmon[90393]: 2022-05-14T01:42:06.545+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:42:07.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:42:07 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:42:07.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:42:07 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:42:07.564 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:42:07 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:42:07.564 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:42:07 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:42:07.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:42:07 smithi115 conmon[90393]: 2022-05-14T01:42:07.563+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:42:08.516 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:42:08 smithi115 ceph-mon[78593]: pgmap v4383: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:42:08.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:42:08 smithi047 ceph-mon[77147]: pgmap v4383: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:42:08.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:42:08 smithi115 conmon[90393]: 2022-05-14T01:42:08.515+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:42:09.499 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:42:09 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:42:09.499 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:42:09 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:42:09.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:42:09 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:42:09.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:42:09 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:42:09.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:42:09 smithi115 conmon[90393]: 2022-05-14T01:42:09.498+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:42:10.526 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:42:10 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:42:10.527 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:42:10 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:42:10.527 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:42:10 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:42:10.528 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:42:10 smithi115 ceph-mon[78593]: pgmap v4384: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:42:10.528 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:42:10 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 8060 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:42:10.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:42:10 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:42:10.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:42:10 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:42:10.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:42:10 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:42:10.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:42:10 smithi047 ceph-mon[77147]: pgmap v4384: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:42:10.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:42:10 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 8060 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:42:10.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:42:10 smithi115 conmon[90393]: 2022-05-14T01:42:10.525+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:42:11.515 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:42:11 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:42:11.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:42:11 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:42:11.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:42:11 smithi115 conmon[90393]: 2022-05-14T01:42:11.514+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:42:12.487 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:42:12 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:42:12.488 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:42:12 smithi115 ceph-mon[78593]: pgmap v4385: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:42:12.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:42:12 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:42:12.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:42:12 smithi047 ceph-mon[77147]: pgmap v4385: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:42:12.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:42:12 smithi115 conmon[90393]: 2022-05-14T01:42:12.487+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:42:13.537 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:42:13 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:42:13.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:42:13 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:42:13.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:42:13 smithi115 conmon[90393]: 2022-05-14T01:42:13.537+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:42:14.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:42:14 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:42:14.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:42:14 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:42:14.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:42:14 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:42:14.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:42:14 smithi047 ceph-mon[77147]: pgmap v4386: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:42:14.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:42:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T01:42:14.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:42:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T01:42:14.576 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:42:14 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:42:14.576 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:42:14 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:42:14.576 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:42:14 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:42:14.577 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:42:14 smithi115 ceph-mon[78593]: pgmap v4386: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:42:14.577 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:42:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T01:42:14.577 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:42:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T01:42:14.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:42:14 smithi115 conmon[90393]: 2022-05-14T01:42:14.574+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:42:15.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:42:15 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 8065 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:42:15.558 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:42:15 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 8065 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:42:15.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:42:15 smithi115 conmon[90393]: 2022-05-14T01:42:15.557+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:42:16.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:42:16 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:42:16.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:42:16 smithi047 ceph-mon[77147]: pgmap v4387: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:42:16.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:42:16 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:42:16.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:42:16 smithi115 ceph-mon[78593]: pgmap v4387: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:42:16.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:42:16 smithi115 conmon[90393]: 2022-05-14T01:42:16.546+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:42:17.521 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:42:17 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:42:17.522 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:42:17 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:42:17.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:42:17 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:42:17.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:42:17 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:42:17.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:42:17 smithi115 conmon[90393]: 2022-05-14T01:42:17.520+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:42:18.495 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:42:18 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:42:18.496 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:42:18 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:42:18.496 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:42:18 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:42:18.497 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:42:18 smithi115 ceph-mon[78593]: pgmap v4388: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:42:18.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:42:18 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:42:18.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:42:18 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:42:18.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:42:18 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:42:18.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:42:18 smithi047 ceph-mon[77147]: pgmap v4388: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:42:18.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:42:18 smithi115 conmon[90393]: 2022-05-14T01:42:18.494+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:42:19.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:42:19 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:42:19.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:42:19 smithi115 conmon[90393]: 2022-05-14T01:42:19.449+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:42:19.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:42:19 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:42:20.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:42:20 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:42:20.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:42:20 smithi047 ceph-mon[77147]: pgmap v4389: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:42:20.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:42:20 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 8070 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:42:20.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:42:20 smithi115 conmon[90393]: 2022-05-14T01:42:20.427+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:42:20.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:42:20 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:42:20.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:42:20 smithi115 ceph-mon[78593]: pgmap v4389: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:42:20.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:42:20 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 8070 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:42:21.465 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:42:21 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:42:21.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:42:21 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:42:21.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:42:21 smithi115 conmon[90393]: 2022-05-14T01:42:21.463+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:42:22.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:42:22 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:42:22.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:42:22 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:42:22.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:42:22 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:42:22.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:42:22 smithi047 ceph-mon[77147]: pgmap v4390: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:42:22.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:42:22 smithi115 conmon[90393]: 2022-05-14T01:42:22.451+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:42:22.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:42:22 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:42:22.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:42:22 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:42:22.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:42:22 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:42:22.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:42:22 smithi115 ceph-mon[78593]: pgmap v4390: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:42:23.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:42:23 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:42:23.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:42:23 smithi115 conmon[90393]: 2022-05-14T01:42:23.420+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:42:23.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:42:23 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:42:24.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:42:24 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:42:24.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:42:24 smithi047 ceph-mon[77147]: pgmap v4391: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:42:24.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:42:24 smithi115 conmon[90393]: 2022-05-14T01:42:24.465+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:42:24.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:42:24 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:42:24.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:42:24 smithi115 ceph-mon[78593]: pgmap v4391: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:42:25.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:42:25 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:42:25.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:42:25 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 8075 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:42:25.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:42:25 smithi115 conmon[90393]: 2022-05-14T01:42:25.478+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:42:25.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:42:25 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:42:25.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:42:25 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 8075 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:42:26.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:42:26 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:42:26.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:42:26 smithi047 ceph-mon[77147]: pgmap v4392: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:42:26.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:42:26 smithi115 conmon[90393]: 2022-05-14T01:42:26.445+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:42:26.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:42:26 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:42:26.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:42:26 smithi115 ceph-mon[78593]: pgmap v4392: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:42:27.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:42:27 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:42:27.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:42:27 smithi115 conmon[90393]: 2022-05-14T01:42:27.431+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:42:27.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:42:27 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:42:28.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:42:28 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:42:28.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:42:28 smithi047 ceph-mon[77147]: pgmap v4393: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:42:28.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:42:28 smithi115 conmon[90393]: 2022-05-14T01:42:28.445+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:42:28.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:42:28 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:42:28.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:42:28 smithi115 ceph-mon[78593]: pgmap v4393: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:42:29.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:42:29 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:42:29.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:42:29 smithi115 conmon[90393]: 2022-05-14T01:42:29.446+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:42:29.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:42:29 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:42:30.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:42:30 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:42:30.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:42:30 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:42:30.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:42:30 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:42:30.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:42:30 smithi047 ceph-mon[77147]: pgmap v4394: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:42:30.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:42:30 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 8080 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:42:30.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:42:30 smithi115 conmon[90393]: 2022-05-14T01:42:30.472+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:42:30.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:42:30 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:42:30.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:42:30 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:42:30.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:42:30 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:42:30.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:42:30 smithi115 ceph-mon[78593]: pgmap v4394: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:42:30.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:42:30 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 8080 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:42:31.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:42:31 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:42:31.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:42:31 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:42:31.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:42:31 smithi115 conmon[90393]: 2022-05-14T01:42:31.455+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:42:32.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:42:32 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:42:32.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:42:32 smithi047 ceph-mon[77147]: pgmap v4395: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:42:32.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:42:32 smithi115 conmon[90393]: 2022-05-14T01:42:32.409+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:42:32.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:42:32 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:42:32.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:42:32 smithi115 ceph-mon[78593]: pgmap v4395: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:42:33.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:42:33 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:42:33.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:42:33 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:42:33.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:42:33 smithi115 conmon[90393]: 2022-05-14T01:42:33.377+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:42:34.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:42:34 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:42:34.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:42:34 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:42:34.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:42:34 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:42:34.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:42:34 smithi047 ceph-mon[77147]: pgmap v4396: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:42:34.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:42:34 smithi115 conmon[90393]: 2022-05-14T01:42:34.338+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:42:34.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:42:34 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:42:34.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:42:34 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:42:34.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:42:34 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:42:34.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:42:34 smithi115 ceph-mon[78593]: pgmap v4396: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:42:35.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:42:35 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:42:35.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:42:35 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 8085 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:42:35.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:42:35 smithi115 conmon[90393]: 2022-05-14T01:42:35.327+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:42:35.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:42:35 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:42:35.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:42:35 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 8085 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:42:36.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:42:36 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:42:36.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:42:36 smithi047 ceph-mon[77147]: pgmap v4397: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:42:36.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:42:36 smithi115 conmon[90393]: 2022-05-14T01:42:36.303+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:42:36.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:42:36 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:42:36.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:42:36 smithi115 ceph-mon[78593]: pgmap v4397: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:42:37.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:42:37 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:42:37.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:42:37 smithi115 conmon[90393]: 2022-05-14T01:42:37.325+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:42:37.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:42:37 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:42:38.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:42:38 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:42:38.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:42:38 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:42:38.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:42:38 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:42:38.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:42:38 smithi047 ceph-mon[77147]: pgmap v4398: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:42:38.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:42:38 smithi115 conmon[90393]: 2022-05-14T01:42:38.327+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:42:38.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:42:38 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:42:38.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:42:38 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:42:38.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:42:38 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:42:38.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:42:38 smithi115 ceph-mon[78593]: pgmap v4398: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:42:39.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:42:39 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:42:39.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:42:39 smithi115 conmon[90393]: 2022-05-14T01:42:39.342+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:42:39.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:42:39 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:42:40.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:42:40 smithi115 conmon[90393]: 2022-05-14T01:42:40.317+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:42:40.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:42:40 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:42:40.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:42:40 smithi115 ceph-mon[78593]: pgmap v4399: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:42:40.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:42:40 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 8090 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:42:40.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:42:40 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:42:40.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:42:40 smithi047 ceph-mon[77147]: pgmap v4399: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:42:40.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:42:40 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 8090 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:42:41.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:42:41 smithi115 conmon[90393]: 2022-05-14T01:42:41.334+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:42:41.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:42:41 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:42:41.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:42:41 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:42:42.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:42:42 smithi115 conmon[90393]: 2022-05-14T01:42:42.343+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:42:42.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:42:42 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:42:42.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:42:42 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:42:42.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:42:42 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:42:42.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:42:42 smithi115 ceph-mon[78593]: pgmap v4400: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:42:42.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:42:42 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:42:42.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:42:42 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:42:42.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:42:42 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:42:42.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:42:42 smithi047 ceph-mon[77147]: pgmap v4400: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:42:43.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:42:43 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:42:43.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:42:43 smithi115 conmon[90393]: 2022-05-14T01:42:43.307+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:42:43.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:42:43 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:42:44.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:42:44 smithi115 conmon[90393]: 2022-05-14T01:42:44.357+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:42:44.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:42:44 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:42:44.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:42:44 smithi115 ceph-mon[78593]: pgmap v4401: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:42:44.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:42:44 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:42:44.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:42:44 smithi047 ceph-mon[77147]: pgmap v4401: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:42:45.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:42:45 smithi115 conmon[90393]: 2022-05-14T01:42:45.340+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:42:45.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:42:45 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:42:45.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:42:45 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 8095 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:42:45.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:42:45 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:42:45.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:42:45 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 8095 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:42:46.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:42:46 smithi115 conmon[90393]: 2022-05-14T01:42:46.310+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:42:46.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:42:46 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:42:46.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:42:46 smithi115 ceph-mon[78593]: pgmap v4402: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:42:46.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:42:46 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:42:46.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:42:46 smithi047 ceph-mon[77147]: pgmap v4402: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:42:47.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:42:47 smithi115 conmon[90393]: 2022-05-14T01:42:47.286+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:42:47.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:42:47 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:42:47.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:42:47 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:42:48.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:42:48 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:42:48.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:42:48 smithi115 ceph-mon[78593]: pgmap v4403: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:42:48.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:42:48 smithi115 conmon[90393]: 2022-05-14T01:42:48.332+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:42:48.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:42:48 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:42:48.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:42:48 smithi047 ceph-mon[77147]: pgmap v4403: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:42:49.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:42:49 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:42:49.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:42:49 smithi115 conmon[90393]: 2022-05-14T01:42:49.358+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:42:49.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:42:49 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:42:50.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:42:50 smithi115 conmon[90393]: 2022-05-14T01:42:50.380+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:42:50.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:42:50 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:42:50.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:42:50 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:42:50.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:42:50 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:42:50.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:42:50 smithi115 ceph-mon[78593]: pgmap v4404: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:42:50.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:42:50 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 8100 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:42:50.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:42:50 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:42:50.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:42:50 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:42:50.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:42:50 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:42:50.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:42:50 smithi047 ceph-mon[77147]: pgmap v4404: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:42:50.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:42:50 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 8100 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:42:51.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:42:51 smithi115 conmon[90393]: 2022-05-14T01:42:51.356+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:42:51.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:42:51 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:42:51.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:42:51 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:42:52.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:42:52 smithi115 conmon[90393]: 2022-05-14T01:42:52.314+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:42:52.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:42:52 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:42:52.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:42:52 smithi115 ceph-mon[78593]: pgmap v4405: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:42:52.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:42:52 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:42:52.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:42:52 smithi047 ceph-mon[77147]: pgmap v4405: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:42:53.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:42:53 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:42:53.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:42:53 smithi115 conmon[90393]: 2022-05-14T01:42:53.364+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:42:53.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:42:53 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:42:54.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:42:54 smithi115 conmon[90393]: 2022-05-14T01:42:54.407+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:42:54.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:42:54 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:42:54.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:42:54 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:42:54.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:42:54 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:42:54.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:42:54 smithi115 ceph-mon[78593]: pgmap v4406: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:42:54.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:42:54 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:42:54.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:42:54 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:42:54.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:42:54 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:42:54.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:42:54 smithi047 ceph-mon[77147]: pgmap v4406: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:42:55.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:42:55 smithi115 conmon[90393]: 2022-05-14T01:42:55.455+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:42:55.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:42:55 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:42:55.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:42:55 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 8105 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:42:55.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:42:55 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:42:55.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:42:55 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 8105 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:42:56.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:42:56 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:42:56.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:42:56 smithi047 ceph-mon[77147]: pgmap v4407: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:42:56.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:42:56 smithi115 conmon[90393]: 2022-05-14T01:42:56.433+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:42:56.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:42:56 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:42:56.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:42:56 smithi115 ceph-mon[78593]: pgmap v4407: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:42:57.629 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:42:57 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:42:57.629 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:42:57 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:42:57.630 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:42:57 smithi115 conmon[90393]: 2022-05-14T01:42:57.481+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:42:58.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:42:58 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:42:58.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:42:58 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:42:58.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:42:58 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:42:58.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:42:58 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T01:42:58.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:42:58 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T01:42:58.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:42:58 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T01:42:58.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:42:58 smithi047 ceph-mon[77147]: pgmap v4408: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:42:58.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:42:58 smithi115 conmon[90393]: 2022-05-14T01:42:58.486+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:42:58.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:42:58 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:42:58.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:42:58 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:42:58.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:42:58 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:42:58.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:42:58 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T01:42:58.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:42:58 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T01:42:58.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:42:58 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T01:42:58.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:42:58 smithi115 ceph-mon[78593]: pgmap v4408: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:42:59.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:42:59 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:42:59.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:42:59 smithi115 conmon[90393]: 2022-05-14T01:42:59.465+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:42:59.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:42:59 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:43:00.658 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:43:00 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:43:00.658 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:43:00 smithi047 ceph-mon[77147]: pgmap v4409: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:43:00.658 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:43:00 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 8110 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:43:00.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:43:00 smithi115 conmon[90393]: 2022-05-14T01:43:00.460+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:43:00.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:43:00 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:43:00.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:43:00 smithi115 ceph-mon[78593]: pgmap v4409: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:43:00.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:43:00 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 8110 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:43:01.041 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:43:00 smithi047 conmon[78521]: 2022-05-14T01:43:00.656+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:43:01.057 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:43:00 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T01:43:01.058 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:43:00 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:43:01.058 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:43:00 smithi047 conmon[78521]: yield 2022-05-14T01:43:01.058 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:43:00 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:43:01.058 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:43:00 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:43:01.059 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:43:00 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:43:01.059 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:43:00 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T01:43:01.059 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:43:00 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:43:01.059 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:43:00 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T01:43:01.059 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:43:00 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:43:01.060 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:43:00 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T01:43:01.060 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:43:00 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:43:01.060 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:43:00 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T01:43:01.060 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:43:00 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:43:01.060 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:43:00 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T01:43:01.061 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:43:00 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:43:01.061 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:43:00 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:43:01.061 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:43:00 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:43:01.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:43:01 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:43:01.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:43:01 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:43:01.792 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T01:43:01.792 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:43:01.792 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T01:43:01.792 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:43:01.793 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:43:01.793 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:43:01.793 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T01:43:01.793 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:43:01.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T01:43:01.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:43:01.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T01:43:01.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:43:01.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T01:43:01.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:43:01.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T01:43:01.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:43:01.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:43:01.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:43:01.796 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:43:01 smithi047 ceph-mon[77147]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T01:43:01.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:43:01 smithi115 conmon[90393]: 2022-05-14T01:43:01.504+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:43:01.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:43:01 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:43:01.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:43:01 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:43:01.859 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T01:43:01.859 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:43:01.859 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T01:43:01.860 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:43:01.860 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:43:01.860 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:43:01.860 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T01:43:01.860 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:43:01.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T01:43:01.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:43:01.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T01:43:01.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:43:01.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T01:43:01.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:43:01.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T01:43:01.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:43:01.863 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:43:01.863 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:43:01.863 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:43:01 smithi115 ceph-mon[78593]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T01:43:02.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:43:02 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:43:02.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:43:02 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:43:02.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:43:02 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:43:02.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:43:02 smithi047 ceph-mon[77147]: pgmap v4410: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:43:02.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:43:02 smithi115 conmon[90393]: 2022-05-14T01:43:02.483+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:43:02.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:43:02 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:43:02.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:43:02 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:43:02.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:43:02 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:43:02.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:43:02 smithi115 ceph-mon[78593]: pgmap v4410: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:43:03.729 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:43:03 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:43:03.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:43:03 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:43:03.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:43:03 smithi115 conmon[90393]: 2022-05-14T01:43:03.520+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:43:04.041 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:43:03 smithi047 conmon[78521]: 2022-05-14T01:43:03.728+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:43:04.042 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:43:03 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T01:43:04.042 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:43:03 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:43:04.042 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:43:03 smithi047 conmon[78521]: yield 2022-05-14T01:43:04.042 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:43:03 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:43:04.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:43:03 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:43:04.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:43:03 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:43:04.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:43:03 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T01:43:04.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:43:03 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:43:04.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:43:03 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T01:43:04.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:43:03 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:43:04.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:43:03 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T01:43:04.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:43:03 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:43:04.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:43:03 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T01:43:04.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:43:03 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:43:04.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:43:03 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T01:43:04.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:43:03 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:43:04.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:43:03 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:43:04.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:43:03 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:43:04.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:43:03 smithi047 conmon[78521]: 2022-05-14T01:43:03.729+0000 7f5344781700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T01:43:04.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:43:03 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T01:43:04.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:43:03 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:43:04.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:43:03 smithi047 conmon[78521]: yield 2022-05-14T01:43:04.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:43:03 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:43:04.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:43:03 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:43:04.048 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:43:03 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:43:04.048 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:43:03 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T01:43:04.048 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:43:03 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:43:04.048 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:43:03 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T01:43:04.049 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:43:03 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:43:04.049 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:43:03 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T01:43:04.049 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:43:03 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:43:04.049 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:43:03 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T01:43:04.050 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:43:03 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:43:04.050 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:43:03 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T01:43:04.050 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:43:03 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:43:04.050 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:43:03 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:43:04.051 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:43:03 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:43:04.051 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:43:03 smithi047 conmon[78521]: 2022-05-14T01:43:04.051 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:43:03 smithi047 conmon[78521]: During handling of the above exception, another exception occurred: 2022-05-14T01:43:04.051 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:43:03 smithi047 conmon[78521]: 2022-05-14T01:43:04.052 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:43:03 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T01:43:04.052 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:43:03 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T01:43:04.052 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:43:03 smithi047 conmon[78521]: return f(*arg) 2022-05-14T01:43:04.052 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:43:03 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T01:43:04.052 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:43:03 smithi047 conmon[78521]: self._write_client_files(client_files, host) 2022-05-14T01:43:04.053 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:43:03 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T01:43:04.053 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:43:03 smithi047 conmon[78521]: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T01:43:04.053 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:43:03 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T01:43:04.053 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:43:03 smithi047 conmon[78521]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T01:43:04.054 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:43:03 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T01:43:04.054 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:43:03 smithi047 conmon[78521]: return self.event_loop.get_result(coro) 2022-05-14T01:43:04.054 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:43:03 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T01:43:04.054 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:43:03 smithi047 conmon[78521]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T01:43:04.054 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:43:03 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T01:43:04.055 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:43:03 smithi047 conmon[78521]: return self.__get_result() 2022-05-14T01:43:04.055 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:43:03 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T01:43:04.055 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:43:03 smithi047 conmon[78521]: raise self._exception 2022-05-14T01:43:04.055 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:43:03 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T01:43:04.056 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:43:03 smithi047 conmon[78521]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T01:43:04.056 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:43:03 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T01:43:04.056 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:43:03 smithi047 conmon[78521]: conn = await self._remote_connection(host, addr) 2022-05-14T01:43:04.056 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:43:03 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T01:43:04.056 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:43:03 smithi047 conmon[78521]: raise 2022-05-14T01:43:04.057 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:43:03 smithi047 conmon[78521]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T01:43:04.057 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:43:03 smithi047 conmon[78521]: self.gen.throw(type, value, traceback) 2022-05-14T01:43:04.057 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:43:03 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T01:43:04.057 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:43:03 smithi047 conmon[78521]: raise OrchestratorError(msg) 2022-05-14T01:43:04.058 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:43:03 smithi047 conmon[78521]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:43:04.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:43:04 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:43:04.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:43:04 smithi047 ceph-mon[77147]: pgmap v4411: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:43:04.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:43:04 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:43:04.792 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T01:43:04.792 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:43:04.792 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T01:43:04.793 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:43:04.793 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:43:04.793 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:43:04.793 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T01:43:04.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:43:04.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T01:43:04.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:43:04.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T01:43:04.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:43:04.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T01:43:04.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:43:04.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T01:43:04.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:43:04.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:43:04.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:43:04.796 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:43:04 smithi047 ceph-mon[77147]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T01:43:04.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T01:43:04.797 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:43:04.797 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T01:43:04.797 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:43:04.797 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:43:04.797 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:43:04.798 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T01:43:04.798 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:43:04.798 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T01:43:04.798 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:43:04.798 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T01:43:04.799 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:43:04.799 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T01:43:04.799 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:43:04.799 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T01:43:04.799 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:43:04.800 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:43:04.800 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:43:04.800 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T01:43:04.800 INFO:journalctl@ceph.mon.b.smithi047.stdout: During handling of the above exception, another exception occurred: 2022-05-14T01:43:04.801 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T01:43:04.801 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T01:43:04.801 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T01:43:04.801 INFO:journalctl@ceph.mon.b.smithi047.stdout: return f(*arg) 2022-05-14T01:43:04.801 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T01:43:04.802 INFO:journalctl@ceph.mon.b.smithi047.stdout: self._write_client_files(client_files, host) 2022-05-14T01:43:04.802 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T01:43:04.802 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T01:43:04.802 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T01:43:04.803 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T01:43:04.803 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T01:43:04.803 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.event_loop.get_result(coro) 2022-05-14T01:43:04.803 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T01:43:04.803 INFO:journalctl@ceph.mon.b.smithi047.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T01:43:04.804 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T01:43:04.804 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.__get_result() 2022-05-14T01:43:04.804 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T01:43:04.804 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise self._exception 2022-05-14T01:43:04.804 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T01:43:04.805 INFO:journalctl@ceph.mon.b.smithi047.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T01:43:04.805 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T01:43:04.805 INFO:journalctl@ceph.mon.b.smithi047.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T01:43:04.805 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T01:43:04.806 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise 2022-05-14T01:43:04.806 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T01:43:04.806 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.gen.throw(type, value, traceback) 2022-05-14T01:43:04.806 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T01:43:04.806 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OrchestratorError(msg) 2022-05-14T01:43:04.807 INFO:journalctl@ceph.mon.b.smithi047.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:43:04.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:43:04 smithi115 conmon[90393]: 2022-05-14T01:43:04.493+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:43:04.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:43:04 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:43:04.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:43:04 smithi115 ceph-mon[78593]: pgmap v4411: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:43:04.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:43:04 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:43:04.859 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T01:43:04.859 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:43:04.860 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T01:43:04.860 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:43:04.860 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:43:04.860 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:43:04.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T01:43:04.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:43:04.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T01:43:04.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:43:04.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T01:43:04.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:43:04.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T01:43:04.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:43:04.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T01:43:04.863 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:43:04.863 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:43:04.863 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:43:04.863 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:43:04 smithi115 ceph-mon[78593]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T01:43:04.863 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T01:43:04.864 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:43:04.864 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T01:43:04.864 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:43:04.864 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:43:04.865 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:43:04.865 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T01:43:04.865 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:43:04.865 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T01:43:04.865 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:43:04.866 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T01:43:04.866 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:43:04.866 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T01:43:04.866 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:43:04.867 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T01:43:04.867 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:43:04.867 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:43:04.867 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:43:04.868 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T01:43:04.868 INFO:journalctl@ceph.mon.c.smithi115.stdout: During handling of the above exception, another exception occurred: 2022-05-14T01:43:04.868 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T01:43:04.868 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T01:43:04.869 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T01:43:04.869 INFO:journalctl@ceph.mon.c.smithi115.stdout: return f(*arg) 2022-05-14T01:43:04.869 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T01:43:04.870 INFO:journalctl@ceph.mon.c.smithi115.stdout: self._write_client_files(client_files, host) 2022-05-14T01:43:04.870 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T01:43:04.870 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T01:43:04.870 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T01:43:04.871 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T01:43:04.871 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T01:43:04.871 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.event_loop.get_result(coro) 2022-05-14T01:43:04.871 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T01:43:04.871 INFO:journalctl@ceph.mon.c.smithi115.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T01:43:04.872 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T01:43:04.872 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.__get_result() 2022-05-14T01:43:04.872 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T01:43:04.872 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise self._exception 2022-05-14T01:43:04.873 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T01:43:04.873 INFO:journalctl@ceph.mon.c.smithi115.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T01:43:04.873 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T01:43:04.873 INFO:journalctl@ceph.mon.c.smithi115.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T01:43:04.873 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T01:43:04.874 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise 2022-05-14T01:43:04.874 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T01:43:04.874 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.gen.throw(type, value, traceback) 2022-05-14T01:43:04.874 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T01:43:04.874 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OrchestratorError(msg) 2022-05-14T01:43:04.875 INFO:journalctl@ceph.mon.c.smithi115.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:43:05.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:43:05 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:43:05.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:43:05 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 8115 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:43:05.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:43:05 smithi115 conmon[90393]: 2022-05-14T01:43:05.476+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:43:05.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:43:05 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:43:05.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:43:05 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 8115 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:43:06.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:43:06 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:43:06.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:43:06 smithi047 ceph-mon[77147]: pgmap v4412: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:43:06.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:43:06 smithi115 conmon[90393]: 2022-05-14T01:43:06.484+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:43:06.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:43:06 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:43:06.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:43:06 smithi115 ceph-mon[78593]: pgmap v4412: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:43:07.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:43:07 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:43:07.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:43:07 smithi115 conmon[90393]: 2022-05-14T01:43:07.444+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:43:07.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:43:07 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:43:08.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:43:08 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:43:08.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:43:08 smithi047 ceph-mon[77147]: pgmap v4413: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:43:08.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:43:08 smithi115 conmon[90393]: 2022-05-14T01:43:08.412+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:43:08.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:43:08 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:43:08.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:43:08 smithi115 ceph-mon[78593]: pgmap v4413: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:43:09.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:43:09 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:43:09.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:43:09 smithi115 conmon[90393]: 2022-05-14T01:43:09.390+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:43:09.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:43:09 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:43:10.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:43:10 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:43:10.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:43:10 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:43:10.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:43:10 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:43:10.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:43:10 smithi047 ceph-mon[77147]: pgmap v4414: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:43:10.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:43:10 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 8120 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:43:10.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:43:10 smithi115 conmon[90393]: 2022-05-14T01:43:10.374+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:43:10.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:43:10 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:43:10.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:43:10 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:43:10.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:43:10 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:43:10.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:43:10 smithi115 ceph-mon[78593]: pgmap v4414: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:43:10.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:43:10 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 8120 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:43:11.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:43:11 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:43:11.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:43:11 smithi115 conmon[90393]: 2022-05-14T01:43:11.403+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:43:11.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:43:11 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:43:12.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:43:12 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:43:12.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:43:12 smithi047 ceph-mon[77147]: pgmap v4415: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:43:12.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:43:12 smithi115 conmon[90393]: 2022-05-14T01:43:12.447+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:43:12.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:43:12 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:43:12.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:43:12 smithi115 ceph-mon[78593]: pgmap v4415: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:43:13.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:43:13 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:43:13.863 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:43:13 smithi115 conmon[90393]: 2022-05-14T01:43:13.421+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:43:13.863 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:43:13 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:43:14.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:43:14 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:43:14.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:43:14 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:43:14.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:43:14 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:43:14.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:43:14 smithi047 ceph-mon[77147]: pgmap v4416: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:43:14.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:43:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T01:43:14.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:43:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T01:43:14.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:43:14 smithi115 conmon[90393]: 2022-05-14T01:43:14.436+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:43:14.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:43:14 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:43:14.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:43:14 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:43:14.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:43:14 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:43:14.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:43:14 smithi115 ceph-mon[78593]: pgmap v4416: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:43:14.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:43:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T01:43:14.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:43:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T01:43:15.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:43:15 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:43:15.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:43:15 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 8125 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:43:15.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:43:15 smithi115 conmon[90393]: 2022-05-14T01:43:15.483+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:43:15.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:43:15 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:43:15.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:43:15 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 8125 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:43:16.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:43:16 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:43:16.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:43:16 smithi047 ceph-mon[77147]: pgmap v4417: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:43:16.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:43:16 smithi115 conmon[90393]: 2022-05-14T01:43:16.469+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:43:16.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:43:16 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:43:16.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:43:16 smithi115 ceph-mon[78593]: pgmap v4417: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:43:17.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:43:17 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:43:17.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:43:17 smithi115 conmon[90393]: 2022-05-14T01:43:17.499+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:43:17.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:43:17 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:43:18.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:43:18 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:43:18.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:43:18 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:43:18.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:43:18 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:43:18.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:43:18 smithi047 ceph-mon[77147]: pgmap v4418: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:43:18.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:43:18 smithi115 conmon[90393]: 2022-05-14T01:43:18.506+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:43:18.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:43:18 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:43:18.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:43:18 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:43:18.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:43:18 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:43:18.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:43:18 smithi115 ceph-mon[78593]: pgmap v4418: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:43:19.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:43:19 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:43:19.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:43:19 smithi115 conmon[90393]: 2022-05-14T01:43:19.555+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:43:19.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:43:19 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:43:20.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:43:20 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:43:20.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:43:20 smithi047 ceph-mon[77147]: pgmap v4419: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:43:20.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:43:20 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 8130 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:43:20.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:43:20 smithi115 conmon[90393]: 2022-05-14T01:43:20.585+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:43:20.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:43:20 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:43:20.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:43:20 smithi115 ceph-mon[78593]: pgmap v4419: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:43:20.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:43:20 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 8130 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:43:21.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:43:21 smithi115 conmon[90393]: 2022-05-14T01:43:21.537+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:43:22.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:43:22 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:43:22.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:43:22 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:43:22.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:43:22 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:43:22.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:43:22 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:43:22.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:43:22 smithi047 ceph-mon[77147]: pgmap v4420: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:43:22.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:43:22 smithi115 conmon[90393]: 2022-05-14T01:43:22.509+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:43:22.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:43:22 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:43:22.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:43:22 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:43:22.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:43:22 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:43:22.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:43:22 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:43:22.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:43:22 smithi115 ceph-mon[78593]: pgmap v4420: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:43:23.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:43:23 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:43:23.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:43:23 smithi115 conmon[90393]: 2022-05-14T01:43:23.487+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:43:23.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:43:23 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:43:24.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:43:24 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:43:24.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:43:24 smithi047 ceph-mon[77147]: pgmap v4421: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:43:24.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:43:24 smithi115 conmon[90393]: 2022-05-14T01:43:24.500+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:43:24.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:43:24 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:43:24.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:43:24 smithi115 ceph-mon[78593]: pgmap v4421: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:43:25.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:43:25 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:43:25.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:43:25 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 8135 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:43:25.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:43:25 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:43:25.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:43:25 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 8135 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:43:25.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:43:25 smithi115 conmon[90393]: 2022-05-14T01:43:25.492+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:43:26.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:43:26 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:43:26.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:43:26 smithi047 ceph-mon[77147]: pgmap v4422: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:43:26.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:43:26 smithi115 conmon[90393]: 2022-05-14T01:43:26.511+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:43:26.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:43:26 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:43:26.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:43:26 smithi115 ceph-mon[78593]: pgmap v4422: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:43:27.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:43:27 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:43:27.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:43:27 smithi115 conmon[90393]: 2022-05-14T01:43:27.545+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:43:27.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:43:27 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:43:28.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:43:28 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:43:28.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:43:28 smithi047 ceph-mon[77147]: pgmap v4423: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:43:28.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:43:28 smithi115 conmon[90393]: 2022-05-14T01:43:28.506+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:43:28.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:43:28 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:43:28.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:43:28 smithi115 ceph-mon[78593]: pgmap v4423: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:43:29.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:43:29 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:43:29.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:43:29 smithi115 conmon[90393]: 2022-05-14T01:43:29.551+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:43:29.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:43:29 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:43:30.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:43:30 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:43:30.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:43:30 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:43:30.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:43:30 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:43:30.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:43:30 smithi047 ceph-mon[77147]: pgmap v4424: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:43:30.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:43:30 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 8140 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:43:30.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:43:30 smithi115 conmon[90393]: 2022-05-14T01:43:30.567+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:43:30.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:43:30 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:43:30.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:43:30 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:43:30.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:43:30 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:43:30.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:43:30 smithi115 ceph-mon[78593]: pgmap v4424: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:43:30.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:43:30 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 8140 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:43:32.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:43:31 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:43:32.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:43:31 smithi115 conmon[90393]: 2022-05-14T01:43:31.610+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:43:32.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:43:31 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:43:32.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:43:32 smithi115 conmon[90393]: 2022-05-14T01:43:32.582+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:43:32.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:43:32 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:43:32.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:43:32 smithi115 ceph-mon[78593]: pgmap v4425: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:43:33.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:43:32 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:43:33.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:43:32 smithi047 ceph-mon[77147]: pgmap v4425: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:43:33.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:43:33 smithi115 conmon[90393]: 2022-05-14T01:43:33.571+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:43:33.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:43:33 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:43:34.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:43:33 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:43:34.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:43:34 smithi115 conmon[90393]: 2022-05-14T01:43:34.531+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:43:34.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:43:34 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:43:34.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:43:34 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:43:34.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:43:34 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:43:34.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:43:34 smithi115 ceph-mon[78593]: pgmap v4426: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:43:34.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:43:34 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:43:35.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:43:34 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:43:35.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:43:34 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:43:35.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:43:34 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:43:35.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:43:34 smithi047 ceph-mon[77147]: pgmap v4426: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:43:35.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:43:34 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:43:35.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:43:35 smithi115 conmon[90393]: 2022-05-14T01:43:35.543+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:43:35.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:43:35 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 8145 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:43:35.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:43:35 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:43:36.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:43:35 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 8145 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:43:36.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:43:35 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:43:36.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:43:36 smithi115 conmon[90393]: 2022-05-14T01:43:36.501+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:43:36.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:43:36 smithi115 ceph-mon[78593]: pgmap v4427: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:43:36.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:43:36 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:43:37.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:43:36 smithi047 ceph-mon[77147]: pgmap v4427: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:43:37.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:43:36 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:43:37.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:43:37 smithi115 conmon[90393]: 2022-05-14T01:43:37.520+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:43:37.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:43:37 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:43:38.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:43:37 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:43:38.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:43:38 smithi115 conmon[90393]: 2022-05-14T01:43:38.543+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:43:38.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:43:38 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:43:38.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:43:38 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:43:38.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:43:38 smithi115 ceph-mon[78593]: pgmap v4428: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:43:38.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:43:38 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:43:39.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:43:38 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:43:39.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:43:38 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:43:39.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:43:38 smithi047 ceph-mon[77147]: pgmap v4428: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:43:39.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:43:38 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:43:39.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:43:39 smithi115 conmon[90393]: 2022-05-14T01:43:39.593+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:43:40.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:43:40 smithi115 conmon[90393]: 2022-05-14T01:43:40.572+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:43:40.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:43:40 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:43:40.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:43:40 smithi115 ceph-mon[78593]: pgmap v4429: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:43:40.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:43:40 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 8155 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:43:41.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:43:40 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:43:41.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:43:40 smithi047 ceph-mon[77147]: pgmap v4429: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:43:41.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:43:40 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 8155 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:43:41.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:43:41 smithi115 conmon[90393]: 2022-05-14T01:43:41.603+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:43:42.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:43:42 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:43:42.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:43:42 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:43:42.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:43:42 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:43:42.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:43:42 smithi047 ceph-mon[77147]: pgmap v4430: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:43:42.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:43:42 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:43:42.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:43:42 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:43:42.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:43:42 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:43:42.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:43:42 smithi115 ceph-mon[78593]: pgmap v4430: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:43:42.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:43:42 smithi115 conmon[90393]: 2022-05-14T01:43:42.583+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:43:43.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:43:43 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:43:43.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:43:43 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:43:43.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:43:43 smithi115 conmon[90393]: 2022-05-14T01:43:43.566+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:43:44.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:43:44 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:43:44.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:43:44 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:43:44.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:43:44 smithi047 ceph-mon[77147]: pgmap v4431: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:43:44.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:43:44 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:43:44.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:43:44 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:43:44.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:43:44 smithi115 ceph-mon[78593]: pgmap v4431: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:43:44.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:43:44 smithi115 conmon[90393]: 2022-05-14T01:43:44.590+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:43:45.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:43:45 smithi115 conmon[90393]: 2022-05-14T01:43:45.603+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:43:46.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:43:46 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:43:46.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:43:46 smithi047 ceph-mon[77147]: pgmap v4432: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:43:46.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:43:46 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:43:46.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:43:46 smithi115 ceph-mon[78593]: pgmap v4432: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:43:47.035 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:43:46 smithi115 conmon[90393]: 2022-05-14T01:43:46.650+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:43:47.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:43:47 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:43:47.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:43:47 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 8160 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:43:47.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:43:47 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:43:47.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:43:47 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 8160 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:43:48.040 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:43:47 smithi115 conmon[90393]: 2022-05-14T01:43:47.655+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:43:48.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:43:48 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:43:48.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:43:48 smithi115 ceph-mon[78593]: pgmap v4433: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:43:48.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:43:48 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:43:48.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:43:48 smithi047 ceph-mon[77147]: pgmap v4433: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:43:49.046 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:43:48 smithi115 conmon[90393]: 2022-05-14T01:43:48.698+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:43:49.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:43:49 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:43:49.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:43:49 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:43:50.047 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:43:49 smithi115 conmon[90393]: 2022-05-14T01:43:49.735+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:43:50.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:43:50 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:43:50.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:43:50 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:43:50.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:43:50 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:43:50.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:43:50 smithi115 ceph-mon[78593]: pgmap v4434: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:43:50.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:43:50 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:43:50.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:43:50 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:43:50.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:43:50 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:43:50.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:43:50 smithi047 ceph-mon[77147]: pgmap v4434: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:43:51.052 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:43:50 smithi115 conmon[90393]: 2022-05-14T01:43:50.707+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:43:51.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:43:51 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:43:51.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:43:51 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:43:52.056 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:43:51 smithi115 conmon[90393]: 2022-05-14T01:43:51.755+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:43:52.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:43:52 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:43:52.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:43:52 smithi115 ceph-mon[78593]: pgmap v4435: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:43:52.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:43:52 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:43:52.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:43:52 smithi047 ceph-mon[77147]: pgmap v4435: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:43:53.058 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:43:52 smithi115 conmon[90393]: 2022-05-14T01:43:52.777+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:43:53.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:43:53 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:43:53.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:43:53 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:43:54.069 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:43:53 smithi115 conmon[90393]: 2022-05-14T01:43:53.754+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:43:54.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:43:54 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:43:54.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:43:54 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:43:54.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:43:54 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:43:54.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:43:54 smithi115 ceph-mon[78593]: pgmap v4436: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:43:54.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:43:54 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:43:54.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:43:54 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:43:54.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:43:54 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:43:54.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:43:54 smithi047 ceph-mon[77147]: pgmap v4436: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:43:55.070 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:43:54 smithi115 conmon[90393]: 2022-05-14T01:43:54.756+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:43:55.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:43:55 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:43:55.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:43:55 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 8165 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:43:55.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:43:55 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:43:55.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:43:55 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 8165 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:43:56.073 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:43:55 smithi115 conmon[90393]: 2022-05-14T01:43:55.771+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:43:56.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:43:56 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:43:56.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:43:56 smithi115 ceph-mon[78593]: pgmap v4437: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:43:56.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:43:56 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:43:56.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:43:56 smithi047 ceph-mon[77147]: pgmap v4437: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:43:57.079 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:43:56 smithi115 conmon[90393]: 2022-05-14T01:43:56.725+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:43:57.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:43:57 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:43:57.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:43:57 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:43:58.084 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:43:57 smithi115 conmon[90393]: 2022-05-14T01:43:57.762+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:43:58.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:43:58 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:43:58.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:43:58 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:43:58.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:43:58 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:43:58.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:43:58 smithi115 ceph-mon[78593]: pgmap v4438: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:43:58.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:43:58 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:43:58.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:43:58 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:43:58.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:43:58 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:43:58.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:43:58 smithi047 ceph-mon[77147]: pgmap v4438: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:43:59.085 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:43:58 smithi115 conmon[90393]: 2022-05-14T01:43:58.761+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:43:59.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:43:59 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:43:59.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:43:59 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:44:00.090 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:43:59 smithi115 conmon[90393]: 2022-05-14T01:43:59.771+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:44:00.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:44:00 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:44:00.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:44:00 smithi115 ceph-mon[78593]: pgmap v4439: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:44:00.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:44:00 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 8170 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:44:00.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:44:00 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:44:00.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:44:00 smithi047 ceph-mon[77147]: pgmap v4439: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:44:00.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:44:00 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 8170 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:44:01.097 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:44:00 smithi115 conmon[90393]: 2022-05-14T01:44:00.738+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:44:01.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:44:01 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:44:01.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:44:01 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:44:02.098 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:44:01 smithi115 conmon[90393]: 2022-05-14T01:44:01.785+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:44:02.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:44:02 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:44:02.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:44:02 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:44:02.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:44:02 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:44:02.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:44:02 smithi115 ceph-mon[78593]: pgmap v4440: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:44:02.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:44:02 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:44:02.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:44:02 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:44:02.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:44:02 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:44:02.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:44:02 smithi047 ceph-mon[77147]: pgmap v4440: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:44:03.104 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:44:02 smithi115 conmon[90393]: 2022-05-14T01:44:02.754+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:44:03.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:44:03 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:44:03.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:44:03 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:44:04.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:44:03 smithi115 conmon[90393]: 2022-05-14T01:44:03.710+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:44:04.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:44:04 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:44:04.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:44:04 smithi047 ceph-mon[77147]: pgmap v4441: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:44:04.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:44:04 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T01:44:04.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:44:04 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T01:44:04.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:44:04 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T01:44:04.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:44:04 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:44:04.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:44:04 smithi115 ceph-mon[78593]: pgmap v4441: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:44:04.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:44:04 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T01:44:04.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:44:04 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T01:44:04.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:44:04 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T01:44:05.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:44:04 smithi115 conmon[90393]: 2022-05-14T01:44:04.687+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:44:05.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:44:05 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:44:05.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:44:05 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 8175 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:44:05.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:44:05 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:44:05.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:44:05 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 8175 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:44:06.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:44:05 smithi115 conmon[90393]: 2022-05-14T01:44:05.675+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:44:06.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:44:06 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:44:06.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:44:06 smithi047 ceph-mon[77147]: pgmap v4442: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:44:06.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:44:06 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:44:06.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:44:06 smithi115 ceph-mon[78593]: pgmap v4442: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:44:07.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:44:06 smithi115 conmon[90393]: 2022-05-14T01:44:06.651+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:44:07.115 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:44:06 smithi047 conmon[78521]: 2022-05-14T01:44:06.831+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:44:07.116 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:44:06 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T01:44:07.116 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:44:06 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:44:07.116 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:44:06 smithi047 conmon[78521]: yield 2022-05-14T01:44:07.116 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:44:06 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:44:07.117 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:44:06 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:44:07.117 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:44:06 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:44:07.117 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:44:06 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T01:44:07.117 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:44:06 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:44:07.117 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:44:06 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T01:44:07.118 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:44:06 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:44:07.118 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:44:06 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T01:44:07.118 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:44:06 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:44:07.118 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:44:06 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T01:44:07.119 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:44:06 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:44:07.119 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:44:06 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T01:44:07.119 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:44:06 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:44:07.119 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:44:06 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:44:07.119 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:44:06 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:44:07.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:44:07 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:44:07.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:44:07 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:44:07.541 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T01:44:07.542 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:44:07.542 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T01:44:07.542 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:44:07.542 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:44:07.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:44:07.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T01:44:07.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:44:07.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T01:44:07.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:44:07.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T01:44:07.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:44:07.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T01:44:07.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:44:07.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T01:44:07.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:44:07.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:44:07.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:44:07.546 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:44:07 smithi047 ceph-mon[77147]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T01:44:07.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:44:07 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:44:07.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:44:07 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:44:07.608 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T01:44:07.609 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:44:07.609 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T01:44:07.609 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:44:07.609 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:44:07.609 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:44:07.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T01:44:07.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:44:07.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T01:44:07.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:44:07.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T01:44:07.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:44:07.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T01:44:07.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:44:07.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T01:44:07.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:44:07.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:44:07.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:44:07.612 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:44:07 smithi115 ceph-mon[78593]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T01:44:08.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:44:07 smithi115 conmon[90393]: 2022-05-14T01:44:07.697+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:44:08.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:44:08 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:44:08.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:44:08 smithi047 ceph-mon[77147]: pgmap v4443: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:44:08.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:44:08 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:44:08.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:44:08 smithi115 ceph-mon[78593]: pgmap v4443: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:44:09.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:44:08 smithi115 conmon[90393]: 2022-05-14T01:44:08.742+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:44:09.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:44:09 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:44:09.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:44:09 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:44:10.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:44:09 smithi115 conmon[90393]: 2022-05-14T01:44:09.756+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:44:10.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:44:10 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:44:10.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:44:10 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:44:10.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:44:10 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:44:10.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:44:10 smithi047 ceph-mon[77147]: pgmap v4444: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:44:10.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:44:10 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 8180 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:44:10.293 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:44:09 smithi047 conmon[78521]: 2022-05-14T01:44:09.903+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:44:10.293 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:44:09 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T01:44:10.293 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:44:09 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:44:10.293 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:44:09 smithi047 conmon[78521]: yield 2022-05-14T01:44:10.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:44:09 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:44:10.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:44:09 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:44:10.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:44:09 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:44:10.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:44:09 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T01:44:10.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:44:09 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:44:10.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:44:09 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T01:44:10.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:44:09 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:44:10.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:44:09 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T01:44:10.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:44:09 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:44:10.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:44:09 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T01:44:10.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:44:09 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:44:10.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:44:09 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T01:44:10.297 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:44:09 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:44:10.297 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:44:09 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:44:10.297 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:44:09 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:44:10.297 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:44:09 smithi047 conmon[78521]: 2022-05-14T01:44:09.904+0000 7f5345f84700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T01:44:10.297 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:44:09 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T01:44:10.298 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:44:09 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:44:10.298 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:44:09 smithi047 conmon[78521]: yield 2022-05-14T01:44:10.298 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:44:09 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:44:10.298 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:44:09 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:44:10.299 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:44:09 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:44:10.299 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:44:09 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T01:44:10.299 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:44:09 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:44:10.299 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:44:09 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T01:44:10.300 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:44:09 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:44:10.300 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:44:09 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T01:44:10.300 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:44:09 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:44:10.300 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:44:09 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T01:44:10.300 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:44:09 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:44:10.301 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:44:09 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T01:44:10.301 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:44:09 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:44:10.301 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:44:09 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:44:10.301 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:44:09 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:44:10.301 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:44:09 smithi047 conmon[78521]: 2022-05-14T01:44:10.302 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:44:09 smithi047 conmon[78521]: During handling of the above exception, another exception occurred: 2022-05-14T01:44:10.302 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:44:09 smithi047 conmon[78521]: 2022-05-14T01:44:10.302 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:44:09 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T01:44:10.302 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:44:09 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T01:44:10.303 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:44:09 smithi047 conmon[78521]: return f(*arg) 2022-05-14T01:44:10.303 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:44:09 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T01:44:10.303 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:44:09 smithi047 conmon[78521]: self._write_client_files(client_files, host) 2022-05-14T01:44:10.303 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:44:09 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T01:44:10.303 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:44:09 smithi047 conmon[78521]: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T01:44:10.304 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:44:09 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T01:44:10.304 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:44:09 smithi047 conmon[78521]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T01:44:10.304 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:44:09 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T01:44:10.304 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:44:09 smithi047 conmon[78521]: return self.event_loop.get_result(coro) 2022-05-14T01:44:10.305 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:44:09 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T01:44:10.305 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:44:09 smithi047 conmon[78521]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T01:44:10.305 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:44:09 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T01:44:10.305 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:44:09 smithi047 conmon[78521]: return self.__get_result() 2022-05-14T01:44:10.306 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:44:09 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T01:44:10.306 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:44:09 smithi047 conmon[78521]: raise self._exception 2022-05-14T01:44:10.306 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:44:09 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T01:44:10.306 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:44:09 smithi047 conmon[78521]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T01:44:10.306 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:44:09 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T01:44:10.307 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:44:09 smithi047 conmon[78521]: conn = await self._remote_connection(host, addr) 2022-05-14T01:44:10.307 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:44:09 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T01:44:10.307 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:44:09 smithi047 conmon[78521]: raise 2022-05-14T01:44:10.307 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:44:09 smithi047 conmon[78521]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T01:44:10.307 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:44:09 smithi047 conmon[78521]: self.gen.throw(type, value, traceback) 2022-05-14T01:44:10.308 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:44:09 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T01:44:10.308 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:44:09 smithi047 conmon[78521]: raise OrchestratorError(msg) 2022-05-14T01:44:10.308 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:44:09 smithi047 conmon[78521]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:44:10.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:44:10 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:44:10.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:44:10 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:44:10.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:44:10 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:44:10.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:44:10 smithi115 ceph-mon[78593]: pgmap v4444: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:44:10.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:44:10 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 8180 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:44:11.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:44:10 smithi115 conmon[90393]: 2022-05-14T01:44:10.757+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:44:11.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:44:11 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:44:11.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:44:11 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:44:11.542 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T01:44:11.542 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:44:11.542 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T01:44:11.542 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:44:11.542 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:44:11.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:44:11.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T01:44:11.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:44:11.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T01:44:11.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:44:11.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T01:44:11.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:44:11.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T01:44:11.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:44:11.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T01:44:11.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:44:11.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:44:11.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:44:11.546 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:44:11 smithi047 ceph-mon[77147]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T01:44:11.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T01:44:11.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:44:11.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T01:44:11.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:44:11.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:44:11.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:44:11.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T01:44:11.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:44:11.548 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T01:44:11.548 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:44:11.548 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T01:44:11.548 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:44:11.549 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T01:44:11.549 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:44:11.549 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T01:44:11.549 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:44:11.550 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:44:11.550 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:44:11.550 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T01:44:11.550 INFO:journalctl@ceph.mon.b.smithi047.stdout: During handling of the above exception, another exception occurred: 2022-05-14T01:44:11.551 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T01:44:11.551 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T01:44:11.551 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T01:44:11.551 INFO:journalctl@ceph.mon.b.smithi047.stdout: return f(*arg) 2022-05-14T01:44:11.551 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T01:44:11.552 INFO:journalctl@ceph.mon.b.smithi047.stdout: self._write_client_files(client_files, host) 2022-05-14T01:44:11.552 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T01:44:11.552 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T01:44:11.552 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T01:44:11.552 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T01:44:11.553 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T01:44:11.553 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.event_loop.get_result(coro) 2022-05-14T01:44:11.553 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T01:44:11.553 INFO:journalctl@ceph.mon.b.smithi047.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T01:44:11.554 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T01:44:11.554 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.__get_result() 2022-05-14T01:44:11.554 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T01:44:11.554 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise self._exception 2022-05-14T01:44:11.555 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T01:44:11.555 INFO:journalctl@ceph.mon.b.smithi047.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T01:44:11.555 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T01:44:11.555 INFO:journalctl@ceph.mon.b.smithi047.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T01:44:11.555 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T01:44:11.556 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise 2022-05-14T01:44:11.556 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T01:44:11.556 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.gen.throw(type, value, traceback) 2022-05-14T01:44:11.556 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T01:44:11.556 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OrchestratorError(msg) 2022-05-14T01:44:11.557 INFO:journalctl@ceph.mon.b.smithi047.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:44:11.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:44:11 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:44:11.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:44:11 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:44:11.608 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T01:44:11.608 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:44:11.609 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T01:44:11.609 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:44:11.609 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:44:11.609 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:44:11.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T01:44:11.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:44:11.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T01:44:11.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:44:11.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T01:44:11.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:44:11.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T01:44:11.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:44:11.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T01:44:11.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:44:11.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:44:11.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:44:11.612 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:44:11 smithi115 ceph-mon[78593]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T01:44:11.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T01:44:11.613 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:44:11.613 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T01:44:11.613 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:44:11.613 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:44:11.614 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:44:11.614 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T01:44:11.614 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:44:11.614 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T01:44:11.614 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:44:11.615 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T01:44:11.615 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:44:11.615 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T01:44:11.615 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:44:11.615 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T01:44:11.616 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:44:11.616 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:44:11.616 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:44:11.616 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T01:44:11.617 INFO:journalctl@ceph.mon.c.smithi115.stdout: During handling of the above exception, another exception occurred: 2022-05-14T01:44:11.617 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T01:44:11.617 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T01:44:11.617 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T01:44:11.617 INFO:journalctl@ceph.mon.c.smithi115.stdout: return f(*arg) 2022-05-14T01:44:11.618 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T01:44:11.618 INFO:journalctl@ceph.mon.c.smithi115.stdout: self._write_client_files(client_files, host) 2022-05-14T01:44:11.618 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T01:44:11.618 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T01:44:11.619 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T01:44:11.619 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T01:44:11.619 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T01:44:11.619 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.event_loop.get_result(coro) 2022-05-14T01:44:11.619 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T01:44:11.620 INFO:journalctl@ceph.mon.c.smithi115.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T01:44:11.620 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T01:44:11.620 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.__get_result() 2022-05-14T01:44:11.620 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T01:44:11.620 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise self._exception 2022-05-14T01:44:11.621 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T01:44:11.621 INFO:journalctl@ceph.mon.c.smithi115.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T01:44:11.621 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T01:44:11.621 INFO:journalctl@ceph.mon.c.smithi115.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T01:44:11.622 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T01:44:11.622 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise 2022-05-14T01:44:11.622 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T01:44:11.622 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.gen.throw(type, value, traceback) 2022-05-14T01:44:11.622 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T01:44:11.623 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OrchestratorError(msg) 2022-05-14T01:44:11.623 INFO:journalctl@ceph.mon.c.smithi115.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:44:12.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:44:11 smithi115 conmon[90393]: 2022-05-14T01:44:11.762+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:44:12.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:44:12 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:44:12.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:44:12 smithi047 ceph-mon[77147]: pgmap v4445: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:44:12.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:44:12 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:44:12.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:44:12 smithi115 ceph-mon[78593]: pgmap v4445: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:44:13.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:44:12 smithi115 conmon[90393]: 2022-05-14T01:44:12.783+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:44:13.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:44:13 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:44:13.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:44:13 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:44:14.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:44:13 smithi115 conmon[90393]: 2022-05-14T01:44:13.767+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:44:14.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:44:14 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:44:14.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:44:14 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:44:14.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:44:14 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:44:14.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:44:14 smithi047 ceph-mon[77147]: pgmap v4446: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:44:14.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:44:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T01:44:14.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:44:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T01:44:14.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:44:14 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:44:14.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:44:14 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:44:14.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:44:14 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:44:14.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:44:14 smithi115 ceph-mon[78593]: pgmap v4446: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:44:14.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:44:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T01:44:14.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:44:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T01:44:15.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:44:14 smithi115 conmon[90393]: 2022-05-14T01:44:14.744+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:44:15.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:44:15 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:44:15.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:44:15 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 8185 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:44:15.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:44:15 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:44:15.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:44:15 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 8185 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:44:16.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:44:15 smithi115 conmon[90393]: 2022-05-14T01:44:15.752+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:44:16.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:44:16 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:44:16.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:44:16 smithi047 ceph-mon[77147]: pgmap v4447: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:44:16.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:44:16 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:44:16.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:44:16 smithi115 ceph-mon[78593]: pgmap v4447: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:44:17.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:44:16 smithi115 conmon[90393]: 2022-05-14T01:44:16.768+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:44:17.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:44:17 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:44:17.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:44:17 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:44:18.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:44:17 smithi115 conmon[90393]: 2022-05-14T01:44:17.736+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:44:18.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:44:18 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:44:18.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:44:18 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:44:18.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:44:18 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:44:18.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:44:18 smithi047 ceph-mon[77147]: pgmap v4448: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:44:18.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:44:18 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:44:18.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:44:18 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:44:18.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:44:18 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:44:18.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:44:18 smithi115 ceph-mon[78593]: pgmap v4448: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:44:19.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:44:18 smithi115 conmon[90393]: 2022-05-14T01:44:18.741+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:44:19.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:44:19 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:44:19.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:44:19 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:44:20.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:44:19 smithi115 conmon[90393]: 2022-05-14T01:44:19.742+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:44:20.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:44:20 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:44:20.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:44:20 smithi047 ceph-mon[77147]: pgmap v4449: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:44:20.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:44:20 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 8190 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:44:20.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:44:20 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:44:20.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:44:20 smithi115 ceph-mon[78593]: pgmap v4449: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:44:20.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:44:20 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 8190 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:44:21.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:44:20 smithi115 conmon[90393]: 2022-05-14T01:44:20.760+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:44:21.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:44:21 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:44:21.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:44:21 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:44:22.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:44:21 smithi115 conmon[90393]: 2022-05-14T01:44:21.739+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:44:22.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:44:22 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:44:22.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:44:22 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:44:22.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:44:22 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:44:22.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:44:22 smithi047 ceph-mon[77147]: pgmap v4450: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:44:22.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:44:22 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:44:22.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:44:22 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:44:22.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:44:22 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:44:22.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:44:22 smithi115 ceph-mon[78593]: pgmap v4450: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:44:23.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:44:22 smithi115 conmon[90393]: 2022-05-14T01:44:22.715+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:44:23.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:44:23 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:44:23.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:44:23 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:44:24.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:44:23 smithi115 conmon[90393]: 2022-05-14T01:44:23.709+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:44:24.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:44:24 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:44:24.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:44:24 smithi047 ceph-mon[77147]: pgmap v4451: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:44:24.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:44:24 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:44:24.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:44:24 smithi115 ceph-mon[78593]: pgmap v4451: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:44:25.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:44:24 smithi115 conmon[90393]: 2022-05-14T01:44:24.687+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:44:25.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:44:25 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:44:25.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:44:25 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 8195 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:44:25.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:44:25 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:44:25.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:44:25 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 8195 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:44:26.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:44:25 smithi115 conmon[90393]: 2022-05-14T01:44:25.680+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:44:26.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:44:26 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:44:26.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:44:26 smithi047 ceph-mon[77147]: pgmap v4452: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:44:26.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:44:26 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:44:26.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:44:26 smithi115 ceph-mon[78593]: pgmap v4452: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:44:27.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:44:26 smithi115 conmon[90393]: 2022-05-14T01:44:26.638+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:44:27.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:44:27 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:44:27.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:44:27 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:44:28.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:44:27 smithi115 conmon[90393]: 2022-05-14T01:44:27.643+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:44:28.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:44:28 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:44:28.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:44:28 smithi047 ceph-mon[77147]: pgmap v4453: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:44:28.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:44:28 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:44:28.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:44:28 smithi115 ceph-mon[78593]: pgmap v4453: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:44:29.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:44:28 smithi115 conmon[90393]: 2022-05-14T01:44:28.631+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:44:29.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:44:29 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:44:29.586 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:44:29 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:44:29.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:44:29 smithi115 conmon[90393]: 2022-05-14T01:44:29.585+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:44:30.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:44:30 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:44:30.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:44:30 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:44:30.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:44:30 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:44:30.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:44:30 smithi047 ceph-mon[77147]: pgmap v4454: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:44:30.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:44:30 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 8200 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:44:30.567 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:44:30 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:44:30.567 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:44:30 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:44:30.568 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:44:30 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:44:30.568 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:44:30 smithi115 ceph-mon[78593]: pgmap v4454: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:44:30.568 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:44:30 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 8200 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:44:30.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:44:30 smithi115 conmon[90393]: 2022-05-14T01:44:30.565+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:44:31.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:44:31 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:44:31.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:44:31 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:44:31.560 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:44:31 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:44:31.561 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:44:31 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:44:31.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:44:31 smithi115 conmon[90393]: 2022-05-14T01:44:31.559+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:44:32.535 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:44:32 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:44:32.536 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:44:32 smithi115 ceph-mon[78593]: pgmap v4455: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:44:32.540 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:44:32 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:44:32.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:44:32 smithi047 ceph-mon[77147]: pgmap v4455: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:44:32.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:44:32 smithi115 conmon[90393]: 2022-05-14T01:44:32.534+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:44:33.538 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:44:33 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:44:33.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:44:33 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:44:33.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:44:33 smithi115 conmon[90393]: 2022-05-14T01:44:33.537+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:44:34.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:44:34 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:44:34.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:44:34 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:44:34.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:44:34 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:44:34.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:44:34 smithi047 ceph-mon[77147]: pgmap v4456: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:44:34.584 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:44:34 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:44:34.584 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:44:34 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:44:34.585 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:44:34 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:44:34.585 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:44:34 smithi115 ceph-mon[78593]: pgmap v4456: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:44:34.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:44:34 smithi115 conmon[90393]: 2022-05-14T01:44:34.583+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:44:35.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:44:35 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 8205 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:44:35.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:44:35 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 8205 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:44:36.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:44:35 smithi115 conmon[90393]: 2022-05-14T01:44:35.614+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:44:36.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:44:36 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:44:36.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:44:36 smithi047 ceph-mon[77147]: pgmap v4457: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:44:36.586 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:44:36 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:44:36.586 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:44:36 smithi115 ceph-mon[78593]: pgmap v4457: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:44:36.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:44:36 smithi115 conmon[90393]: 2022-05-14T01:44:36.585+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:44:37.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:44:37 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:44:37.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:44:37 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:44:38.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:44:37 smithi115 conmon[90393]: 2022-05-14T01:44:37.628+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:44:38.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:44:38 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:44:38.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:44:38 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:44:38.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:44:38 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:44:38.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:44:38 smithi047 ceph-mon[77147]: pgmap v4458: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:44:38.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:44:38 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:44:38.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:44:38 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:44:38.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:44:38 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:44:38.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:44:38 smithi115 ceph-mon[78593]: pgmap v4458: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:44:39.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:44:38 smithi115 conmon[90393]: 2022-05-14T01:44:38.619+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:44:39.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:44:39 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:44:39.598 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:44:39 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:44:39.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:44:39 smithi115 conmon[90393]: 2022-05-14T01:44:39.596+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:44:40.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:44:40 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:44:40.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:44:40 smithi047 ceph-mon[77147]: pgmap v4459: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:44:40.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:44:40 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 8210 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:44:40.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:44:40 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:44:40.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:44:40 smithi115 ceph-mon[78593]: pgmap v4459: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:44:40.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:44:40 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 8210 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:44:41.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:44:40 smithi115 conmon[90393]: 2022-05-14T01:44:40.607+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:44:41.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:44:41 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:44:41.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:44:41 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:44:42.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:44:41 smithi115 conmon[90393]: 2022-05-14T01:44:41.607+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:44:42.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:44:42 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:44:42.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:44:42 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:44:42.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:44:42 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:44:42.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:44:42 smithi047 ceph-mon[77147]: pgmap v4460: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:44:42.586 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:44:42 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:44:42.586 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:44:42 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:44:42.587 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:44:42 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:44:42.587 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:44:42 smithi115 ceph-mon[78593]: pgmap v4460: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:44:42.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:44:42 smithi115 conmon[90393]: 2022-05-14T01:44:42.585+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:44:43.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:44:43 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:44:43.565 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:44:43 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:44:43.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:44:43 smithi115 conmon[90393]: 2022-05-14T01:44:43.564+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:44:44.540 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:44:44 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:44:44.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:44:44 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:44:44.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:44:44 smithi115 ceph-mon[78593]: pgmap v4461: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:44:44.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:44:44 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:44:44.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:44:44 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:44:44.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:44:44 smithi047 ceph-mon[77147]: pgmap v4461: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:44:44.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:44:44 smithi115 conmon[90393]: 2022-05-14T01:44:44.539+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:44:45.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:44:45 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:44:45.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:44:45 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 8215 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:44:45.577 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:44:45 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:44:45.578 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:44:45 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 8215 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:44:45.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:44:45 smithi115 conmon[90393]: 2022-05-14T01:44:45.577+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:44:46.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:44:46 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:44:46.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:44:46 smithi047 ceph-mon[77147]: pgmap v4462: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:44:46.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:44:46 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:44:46.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:44:46 smithi115 ceph-mon[78593]: pgmap v4462: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:44:47.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:44:46 smithi115 conmon[90393]: 2022-05-14T01:44:46.614+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:44:48.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:44:47 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:44:48.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:44:47 smithi115 conmon[90393]: 2022-05-14T01:44:47.610+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:44:48.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:44:47 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:44:48.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:44:48 smithi115 conmon[90393]: 2022-05-14T01:44:48.604+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:44:48.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:44:48 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:44:48.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:44:48 smithi115 ceph-mon[78593]: pgmap v4463: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:44:49.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:44:48 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:44:49.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:44:48 smithi047 ceph-mon[77147]: pgmap v4463: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:44:50.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:44:49 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:44:50.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:44:49 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:44:50.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:44:49 smithi115 conmon[90393]: 2022-05-14T01:44:49.616+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:44:51.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:44:50 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:44:51.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:44:50 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:44:51.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:44:50 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:44:51.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:44:50 smithi047 ceph-mon[77147]: pgmap v4464: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:44:51.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:44:50 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 8220 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:44:51.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:44:50 smithi115 conmon[90393]: 2022-05-14T01:44:50.630+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:44:51.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:44:50 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:44:51.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:44:50 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:44:51.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:44:50 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:44:51.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:44:50 smithi115 ceph-mon[78593]: pgmap v4464: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:44:51.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:44:50 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 8220 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:44:52.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:44:51 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:44:52.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:44:51 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:44:52.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:44:51 smithi115 conmon[90393]: 2022-05-14T01:44:51.666+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:44:53.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:44:52 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:44:53.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:44:52 smithi047 ceph-mon[77147]: pgmap v4465: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:44:53.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:44:52 smithi115 conmon[90393]: 2022-05-14T01:44:52.648+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:44:53.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:44:52 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:44:53.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:44:52 smithi115 ceph-mon[78593]: pgmap v4465: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:44:54.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:44:53 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:44:54.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:44:53 smithi115 conmon[90393]: 2022-05-14T01:44:53.619+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:44:54.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:44:53 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:44:55.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:44:54 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:44:55.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:44:54 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:44:55.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:44:54 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:44:55.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:44:54 smithi047 ceph-mon[77147]: pgmap v4466: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:44:55.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:44:54 smithi115 conmon[90393]: 2022-05-14T01:44:54.623+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:44:55.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:44:54 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:44:55.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:44:54 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:44:55.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:44:54 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:44:55.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:44:54 smithi115 ceph-mon[78593]: pgmap v4466: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:44:56.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:44:55 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:44:56.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:44:55 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 8225 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:44:56.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:44:55 smithi115 conmon[90393]: 2022-05-14T01:44:55.628+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:44:56.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:44:55 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:44:56.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:44:55 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 8225 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:44:57.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:44:56 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:44:57.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:44:56 smithi047 ceph-mon[77147]: pgmap v4467: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:44:57.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:44:56 smithi115 conmon[90393]: 2022-05-14T01:44:56.623+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:44:57.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:44:56 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:44:57.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:44:56 smithi115 ceph-mon[78593]: pgmap v4467: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:44:58.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:44:57 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:44:58.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:44:57 smithi115 conmon[90393]: 2022-05-14T01:44:57.612+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:44:58.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:44:57 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:44:59.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:44:58 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:44:59.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:44:58 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:44:59.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:44:58 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:44:59.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:44:58 smithi047 ceph-mon[77147]: pgmap v4468: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:44:59.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:44:58 smithi115 conmon[90393]: 2022-05-14T01:44:58.616+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:44:59.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:44:58 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:44:59.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:44:58 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:44:59.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:44:58 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:44:59.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:44:58 smithi115 ceph-mon[78593]: pgmap v4468: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:44:59.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:44:59 smithi115 conmon[90393]: 2022-05-14T01:44:59.576+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:44:59.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:44:59 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:44:59.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:44:59 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:45:00.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:44:59 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:45:00.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:44:59 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:45:00.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:45:00 smithi115 conmon[90393]: 2022-05-14T01:45:00.550+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:45:00.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:45:00 smithi115 ceph-mon[78593]: pgmap v4469: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:45:00.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:45:00 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 8230 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:45:00.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:45:00 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:45:01.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:45:00 smithi047 ceph-mon[77147]: pgmap v4469: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:45:01.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:45:00 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 8230 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:45:01.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:45:00 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:45:01.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:45:01 smithi115 conmon[90393]: 2022-05-14T01:45:01.550+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:45:01.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:45:01 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:45:02.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:45:01 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:45:02.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:45:02 smithi115 conmon[90393]: 2022-05-14T01:45:02.533+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:45:02.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:45:02 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:45:02.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:45:02 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:45:02.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:45:02 smithi115 ceph-mon[78593]: pgmap v4470: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:45:02.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:45:02 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:45:03.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:45:02 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:45:03.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:45:02 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:45:03.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:45:02 smithi047 ceph-mon[77147]: pgmap v4470: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:45:03.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:45:02 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:45:03.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:45:03 smithi115 conmon[90393]: 2022-05-14T01:45:03.539+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:45:03.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:45:03 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:45:04.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:45:03 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:45:04.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:45:04 smithi115 conmon[90393]: 2022-05-14T01:45:04.510+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:45:04.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:45:04 smithi115 ceph-mon[78593]: pgmap v4471: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:45:04.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:45:04 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:45:05.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:45:04 smithi047 ceph-mon[77147]: pgmap v4471: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:45:05.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:45:04 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:45:05.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:45:05 smithi115 conmon[90393]: 2022-05-14T01:45:05.482+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:45:05.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:45:05 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 8235 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:45:05.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:45:05 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:45:06.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:45:05 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 8235 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:45:06.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:45:05 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:45:06.718 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:45:06 smithi115 conmon[90393]: 2022-05-14T01:45:06.445+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:45:07.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:45:06 smithi047 ceph-mon[77147]: pgmap v4472: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:45:07.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:45:06 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:45:07.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:45:06 smithi115 ceph-mon[78593]: pgmap v4472: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:45:07.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:45:06 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:45:07.718 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:45:07 smithi115 conmon[90393]: 2022-05-14T01:45:07.423+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:45:08.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:45:07 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:45:08.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:45:07 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:45:08.723 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:45:08 smithi115 conmon[90393]: 2022-05-14T01:45:08.421+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:45:09.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:45:08 smithi047 ceph-mon[77147]: pgmap v4473: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:45:09.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:45:08 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:45:09.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:45:08 smithi115 ceph-mon[78593]: pgmap v4473: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:45:09.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:45:08 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:45:09.724 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:45:09 smithi115 conmon[90393]: 2022-05-14T01:45:09.455+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:45:09.978 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:45:09 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:45:09.979 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:45:09 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:45:09.979 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:45:09 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:45:09.979 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:45:09 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:45:09.980 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:45:09 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:45:09.980 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:45:09 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:45:10.728 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:45:10 smithi115 conmon[90393]: 2022-05-14T01:45:10.502+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:45:10.729 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:45:10 smithi047 conmon[78521]: 2022-05-14T01:45:10.511+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:45:10.729 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:45:10 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T01:45:10.729 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:45:10 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:45:10.730 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:45:10 smithi047 conmon[78521]: yield 2022-05-14T01:45:10.730 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:45:10 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:45:10.730 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:45:10 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:45:10.730 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:45:10 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:45:10.730 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:45:10 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T01:45:10.731 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:45:10 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:45:10.731 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:45:10 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T01:45:10.731 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:45:10 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:45:10.731 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:45:10 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T01:45:10.732 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:45:10 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:45:10.732 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:45:10 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T01:45:10.732 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:45:10 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:45:10.732 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:45:10 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T01:45:10.732 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:45:10 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:45:10.733 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:45:10 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:45:10.733 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:45:10 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:45:11.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:45:10 smithi047 ceph-mon[77147]: pgmap v4474: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:45:11.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:45:10 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T01:45:11.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:45:10 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T01:45:11.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:45:10 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T01:45:11.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:45:10 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 8240 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:45:11.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:45:10 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:45:11.102 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:45:10 smithi115 ceph-mon[78593]: pgmap v4474: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:45:11.103 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:45:10 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T01:45:11.103 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:45:10 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T01:45:11.103 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:45:10 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T01:45:11.103 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:45:10 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 8240 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:45:11.104 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:45:10 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:45:11.730 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:45:11 smithi115 conmon[90393]: 2022-05-14T01:45:11.521+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:45:11.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:45:11 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:45:11.791 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T01:45:11.792 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:45:11.792 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T01:45:11.792 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:45:11.793 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:45:11.793 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:45:11.793 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T01:45:11.793 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:45:11.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T01:45:11.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:45:11.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T01:45:11.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:45:11.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T01:45:11.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:45:11.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T01:45:11.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:45:11.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:45:11.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:45:11.796 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:45:11 smithi047 ceph-mon[77147]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T01:45:11.796 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:45:11 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:45:11.994 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:45:11 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:45:11.994 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T01:45:11.995 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:45:11.995 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T01:45:11.995 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:45:11.995 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:45:11.996 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:45:11.996 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T01:45:11.996 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:45:11.996 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T01:45:11.996 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:45:11.997 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T01:45:11.997 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:45:11.997 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T01:45:11.997 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:45:11.998 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T01:45:11.998 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:45:11.998 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:45:11.998 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:45:11.998 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:45:11 smithi115 ceph-mon[78593]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T01:45:11.999 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:45:11 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:45:12.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:45:12 smithi115 conmon[90393]: 2022-05-14T01:45:12.495+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:45:12.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:45:12 smithi115 ceph-mon[78593]: pgmap v4475: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:45:12.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:45:12 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:45:13.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:45:12 smithi047 ceph-mon[77147]: pgmap v4475: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:45:13.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:45:12 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:45:13.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:45:13 smithi115 conmon[90393]: 2022-05-14T01:45:13.455+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:45:14.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:45:13 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:45:14.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:45:13 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:45:14.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:45:13 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' 2022-05-14T01:45:14.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:45:13 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' 2022-05-14T01:45:14.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:45:13 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:45:14.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:45:13 smithi047 conmon[78521]: 2022-05-14T01:45:13.583+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:45:14.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:45:13 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T01:45:14.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:45:13 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:45:14.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:45:13 smithi047 conmon[78521]: yield 2022-05-14T01:45:14.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:45:13 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:45:14.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:45:13 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:45:14.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:45:13 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:45:14.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:45:13 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T01:45:14.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:45:13 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:45:14.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:45:13 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T01:45:14.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:45:13 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:45:14.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:45:13 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T01:45:14.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:45:13 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:45:14.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:45:13 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T01:45:14.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:45:13 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:45:14.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:45:13 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T01:45:14.048 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:45:13 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:45:14.048 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:45:13 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:45:14.048 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:45:13 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:45:14.048 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:45:13 smithi047 conmon[78521]: 2022-05-14T01:45:13.584+0000 7f5340779700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T01:45:14.048 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:45:13 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T01:45:14.049 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:45:13 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:45:14.049 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:45:13 smithi047 conmon[78521]: yield 2022-05-14T01:45:14.049 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:45:13 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:45:14.049 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:45:13 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:45:14.050 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:45:13 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:45:14.050 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:45:13 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T01:45:14.050 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:45:13 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:45:14.050 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:45:13 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T01:45:14.050 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:45:13 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:45:14.051 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:45:13 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T01:45:14.051 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:45:13 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:45:14.051 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:45:13 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T01:45:14.051 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:45:13 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:45:14.051 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:45:13 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T01:45:14.052 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:45:13 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:45:14.052 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:45:13 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:45:14.052 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:45:13 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:45:14.052 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:45:13 smithi047 conmon[78521]: 2022-05-14T01:45:14.052 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:45:13 smithi047 conmon[78521]: During handling of the above exception, another exception occurred: 2022-05-14T01:45:14.053 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:45:13 smithi047 conmon[78521]: 2022-05-14T01:45:14.053 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:45:13 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T01:45:14.053 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:45:13 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T01:45:14.053 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:45:13 smithi047 conmon[78521]: return f(*arg) 2022-05-14T01:45:14.053 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:45:13 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T01:45:14.054 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:45:13 smithi047 conmon[78521]: self._write_client_files(client_files, host) 2022-05-14T01:45:14.054 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:45:13 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T01:45:14.054 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:45:13 smithi047 conmon[78521]: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T01:45:14.054 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:45:13 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T01:45:14.055 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:45:13 smithi047 conmon[78521]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T01:45:14.055 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:45:13 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T01:45:14.055 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:45:13 smithi047 conmon[78521]: return self.event_loop.get_result(coro) 2022-05-14T01:45:14.056 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:45:13 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T01:45:14.056 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:45:13 smithi047 conmon[78521]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T01:45:14.056 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:45:13 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T01:45:14.057 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:45:13 smithi047 conmon[78521]: return self.__get_result() 2022-05-14T01:45:14.057 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:45:13 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T01:45:14.057 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:45:13 smithi047 conmon[78521]: raise self._exception 2022-05-14T01:45:14.058 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:45:13 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T01:45:14.058 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:45:13 smithi047 conmon[78521]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T01:45:14.058 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:45:13 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T01:45:14.059 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:45:13 smithi047 conmon[78521]: conn = await self._remote_connection(host, addr) 2022-05-14T01:45:14.059 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:45:13 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T01:45:14.059 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:45:13 smithi047 conmon[78521]: raise 2022-05-14T01:45:14.059 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:45:13 smithi047 conmon[78521]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T01:45:14.059 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:45:13 smithi047 conmon[78521]: self.gen.throw(type, value, traceback) 2022-05-14T01:45:14.060 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:45:13 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T01:45:14.060 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:45:13 smithi047 conmon[78521]: raise OrchestratorError(msg) 2022-05-14T01:45:14.060 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:45:13 smithi047 conmon[78521]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:45:14.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:45:13 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:45:14.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:45:13 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:45:14.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:45:13 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' 2022-05-14T01:45:14.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:45:13 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' 2022-05-14T01:45:14.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:45:13 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:45:14.745 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:45:14 smithi115 conmon[90393]: 2022-05-14T01:45:14.432+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:45:15.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:45:14 smithi047 ceph-mon[77147]: Detected new or changed devices on smithi047 2022-05-14T01:45:15.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:45:14 smithi047 ceph-mon[77147]: Detected new or changed devices on smithi115 2022-05-14T01:45:15.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:45:14 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:45:15.042 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T01:45:15.042 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:45:15.042 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T01:45:15.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:45:15.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:45:15.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:45:15.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T01:45:15.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:45:15.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T01:45:15.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:45:15.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T01:45:15.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:45:15.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T01:45:15.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:45:15.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T01:45:15.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:45:15.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:45:15.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:45:15.046 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:45:14 smithi047 ceph-mon[77147]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T01:45:15.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T01:45:15.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:45:15.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T01:45:15.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:45:15.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:45:15.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:45:15.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T01:45:15.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:45:15.049 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T01:45:15.049 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:45:15.050 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T01:45:15.050 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:45:15.050 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T01:45:15.051 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:45:15.051 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T01:45:15.051 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:45:15.051 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:45:15.052 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:45:15.052 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T01:45:15.052 INFO:journalctl@ceph.mon.b.smithi047.stdout: During handling of the above exception, another exception occurred: 2022-05-14T01:45:15.052 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T01:45:15.053 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T01:45:15.053 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T01:45:15.053 INFO:journalctl@ceph.mon.b.smithi047.stdout: return f(*arg) 2022-05-14T01:45:15.054 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T01:45:15.054 INFO:journalctl@ceph.mon.b.smithi047.stdout: self._write_client_files(client_files, host) 2022-05-14T01:45:15.054 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T01:45:15.055 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T01:45:15.055 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T01:45:15.056 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T01:45:15.056 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T01:45:15.056 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.event_loop.get_result(coro) 2022-05-14T01:45:15.057 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T01:45:15.057 INFO:journalctl@ceph.mon.b.smithi047.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T01:45:15.058 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T01:45:15.058 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.__get_result() 2022-05-14T01:45:15.059 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T01:45:15.059 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise self._exception 2022-05-14T01:45:15.059 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T01:45:15.060 INFO:journalctl@ceph.mon.b.smithi047.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T01:45:15.060 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T01:45:15.061 INFO:journalctl@ceph.mon.b.smithi047.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T01:45:15.061 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T01:45:15.062 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise 2022-05-14T01:45:15.062 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T01:45:15.062 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.gen.throw(type, value, traceback) 2022-05-14T01:45:15.063 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T01:45:15.063 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OrchestratorError(msg) 2022-05-14T01:45:15.064 INFO:journalctl@ceph.mon.b.smithi047.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:45:15.064 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:45:14 smithi047 ceph-mon[77147]: pgmap v4476: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:45:15.064 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:45:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T01:45:15.065 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:45:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T01:45:15.065 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:45:14 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:45:15.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:45:14 smithi115 ceph-mon[78593]: Detected new or changed devices on smithi047 2022-05-14T01:45:15.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:45:14 smithi115 ceph-mon[78593]: Detected new or changed devices on smithi115 2022-05-14T01:45:15.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:45:14 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:45:15.109 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T01:45:15.109 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:45:15.109 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T01:45:15.109 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:45:15.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:45:15.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:45:15.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T01:45:15.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:45:15.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T01:45:15.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:45:15.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T01:45:15.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:45:15.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T01:45:15.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:45:15.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T01:45:15.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:45:15.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:45:15.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:45:15.112 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:45:14 smithi115 ceph-mon[78593]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T01:45:15.113 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T01:45:15.113 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:45:15.113 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T01:45:15.113 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:45:15.113 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:45:15.114 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:45:15.114 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T01:45:15.114 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:45:15.114 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T01:45:15.115 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:45:15.115 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T01:45:15.115 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:45:15.115 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T01:45:15.115 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:45:15.116 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T01:45:15.116 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:45:15.116 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:45:15.116 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:45:15.116 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T01:45:15.117 INFO:journalctl@ceph.mon.c.smithi115.stdout: During handling of the above exception, another exception occurred: 2022-05-14T01:45:15.117 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T01:45:15.117 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T01:45:15.117 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T01:45:15.117 INFO:journalctl@ceph.mon.c.smithi115.stdout: return f(*arg) 2022-05-14T01:45:15.118 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T01:45:15.118 INFO:journalctl@ceph.mon.c.smithi115.stdout: self._write_client_files(client_files, host) 2022-05-14T01:45:15.118 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T01:45:15.118 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T01:45:15.118 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T01:45:15.119 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T01:45:15.119 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T01:45:15.119 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.event_loop.get_result(coro) 2022-05-14T01:45:15.119 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T01:45:15.119 INFO:journalctl@ceph.mon.c.smithi115.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T01:45:15.120 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T01:45:15.120 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.__get_result() 2022-05-14T01:45:15.120 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T01:45:15.120 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise self._exception 2022-05-14T01:45:15.120 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T01:45:15.121 INFO:journalctl@ceph.mon.c.smithi115.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T01:45:15.121 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T01:45:15.121 INFO:journalctl@ceph.mon.c.smithi115.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T01:45:15.121 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T01:45:15.122 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise 2022-05-14T01:45:15.122 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T01:45:15.122 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.gen.throw(type, value, traceback) 2022-05-14T01:45:15.122 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T01:45:15.122 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OrchestratorError(msg) 2022-05-14T01:45:15.123 INFO:journalctl@ceph.mon.c.smithi115.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:45:15.123 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:45:14 smithi115 ceph-mon[78593]: pgmap v4476: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:45:15.123 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:45:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T01:45:15.123 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:45:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T01:45:15.123 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:45:14 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:45:15.744 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:45:15 smithi115 conmon[90393]: 2022-05-14T01:45:15.461+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:45:16.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:45:15 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 8245 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:45:16.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:45:15 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:45:16.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:45:15 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 8245 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:45:16.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:45:15 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:45:16.749 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:45:16 smithi115 conmon[90393]: 2022-05-14T01:45:16.450+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:45:17.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:45:16 smithi047 ceph-mon[77147]: pgmap v4477: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:45:17.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:45:16 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:45:17.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:45:16 smithi115 ceph-mon[78593]: pgmap v4477: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:45:17.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:45:16 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:45:17.750 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:45:17 smithi115 conmon[90393]: 2022-05-14T01:45:17.405+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:45:18.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:45:17 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:45:18.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:45:17 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:45:18.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:45:17 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:45:18.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:45:17 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:45:18.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:45:17 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:45:18.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:45:17 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:45:18.755 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:45:18 smithi115 conmon[90393]: 2022-05-14T01:45:18.404+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:45:19.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:45:18 smithi047 ceph-mon[77147]: pgmap v4478: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:45:19.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:45:18 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:45:19.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:45:18 smithi115 ceph-mon[78593]: pgmap v4478: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:45:19.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:45:18 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:45:19.760 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:45:19 smithi115 conmon[90393]: 2022-05-14T01:45:19.436+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:45:20.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:45:19 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:45:20.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:45:19 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:45:20.762 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:45:20 smithi115 conmon[90393]: 2022-05-14T01:45:20.425+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:45:21.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:45:20 smithi047 ceph-mon[77147]: pgmap v4479: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:45:21.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:45:20 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 8255 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:45:21.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:45:20 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:45:21.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:45:20 smithi115 ceph-mon[78593]: pgmap v4479: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:45:21.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:45:20 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 8255 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:45:21.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:45:20 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:45:21.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:45:21 smithi115 conmon[90393]: 2022-05-14T01:45:21.396+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:45:22.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:45:22 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:45:22.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:45:22 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:45:22.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:45:22 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:45:22.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:45:22 smithi115 ceph-mon[78593]: pgmap v4480: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:45:22.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:45:22 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:45:22.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:45:22 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:45:22.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:45:22 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:45:22.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:45:22 smithi047 ceph-mon[77147]: pgmap v4480: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:45:22.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:45:22 smithi115 conmon[90393]: 2022-05-14T01:45:22.428+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:45:23.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:45:23 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:45:23.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:45:23 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:45:23.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:45:23 smithi115 conmon[90393]: 2022-05-14T01:45:23.470+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:45:24.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:45:24 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:45:24.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:45:24 smithi115 ceph-mon[78593]: pgmap v4481: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:45:24.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:45:24 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:45:24.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:45:24 smithi047 ceph-mon[77147]: pgmap v4481: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:45:24.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:45:24 smithi115 conmon[90393]: 2022-05-14T01:45:24.433+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:45:25.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:45:25 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:45:25.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:45:25 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:45:25.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:45:25 smithi115 conmon[90393]: 2022-05-14T01:45:25.416+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:45:26.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:45:26 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:45:26.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:45:26 smithi115 ceph-mon[78593]: pgmap v4482: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:45:26.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:45:26 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:45:26.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:45:26 smithi047 ceph-mon[77147]: pgmap v4482: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:45:26.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:45:26 smithi115 conmon[90393]: 2022-05-14T01:45:26.457+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:45:27.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:45:27 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 8260 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:45:27.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:45:27 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:45:27.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:45:27 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 8260 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:45:27.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:45:27 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:45:27.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:45:27 smithi115 conmon[90393]: 2022-05-14T01:45:27.425+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:45:28.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:45:28 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:45:28.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:45:28 smithi115 ceph-mon[78593]: pgmap v4483: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:45:28.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:45:28 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:45:28.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:45:28 smithi047 ceph-mon[77147]: pgmap v4483: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:45:28.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:45:28 smithi115 conmon[90393]: 2022-05-14T01:45:28.408+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:45:29.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:45:29 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:45:29.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:45:29 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:45:29.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:45:29 smithi115 conmon[90393]: 2022-05-14T01:45:29.430+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:45:30.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:45:30 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:45:30.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:45:30 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:45:30.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:45:30 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:45:30.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:45:30 smithi115 ceph-mon[78593]: pgmap v4484: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:45:30.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:45:30 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:45:30.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:45:30 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:45:30.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:45:30 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:45:30.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:45:30 smithi047 ceph-mon[77147]: pgmap v4484: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:45:30.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:45:30 smithi115 conmon[90393]: 2022-05-14T01:45:30.417+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:45:31.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:45:31 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:45:31.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:45:31 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:45:31.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:45:31 smithi115 conmon[90393]: 2022-05-14T01:45:31.448+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:45:32.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:45:32 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:45:32.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:45:32 smithi115 ceph-mon[78593]: pgmap v4485: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:45:32.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:45:32 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:45:32.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:45:32 smithi047 ceph-mon[77147]: pgmap v4485: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:45:32.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:45:32 smithi115 conmon[90393]: 2022-05-14T01:45:32.413+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:45:33.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:45:33 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:45:33.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:45:33 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:45:33.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:45:33 smithi115 conmon[90393]: 2022-05-14T01:45:33.377+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:45:34.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:45:34 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:45:34.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:45:34 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:45:34.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:45:34 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:45:34.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:45:34 smithi115 ceph-mon[78593]: pgmap v4486: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:45:34.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:45:34 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:45:34.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:45:34 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:45:34.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:45:34 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:45:34.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:45:34 smithi047 ceph-mon[77147]: pgmap v4486: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:45:34.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:45:34 smithi115 conmon[90393]: 2022-05-14T01:45:34.360+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:45:35.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:45:35 smithi115 conmon[90393]: 2022-05-14T01:45:35.316+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:45:35.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:45:35 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:45:35.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:45:35 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 8265 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:45:35.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:45:35 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:45:35.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:45:35 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 8265 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:45:36.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:45:36 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:45:36.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:45:36 smithi047 ceph-mon[77147]: pgmap v4487: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:45:36.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:45:36 smithi115 conmon[90393]: 2022-05-14T01:45:36.269+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:45:36.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:45:36 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:45:36.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:45:36 smithi115 ceph-mon[78593]: pgmap v4487: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:45:37.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:45:37 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:45:37.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:45:37 smithi115 conmon[90393]: 2022-05-14T01:45:37.258+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:45:37.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:45:37 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:45:38.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:45:38 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:45:38.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:45:38 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:45:38.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:45:38 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:45:38.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:45:38 smithi047 ceph-mon[77147]: pgmap v4488: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:45:38.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:45:38 smithi115 conmon[90393]: 2022-05-14T01:45:38.223+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:45:38.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:45:38 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:45:38.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:45:38 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:45:38.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:45:38 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:45:38.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:45:38 smithi115 ceph-mon[78593]: pgmap v4488: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:45:39.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:45:39 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:45:39.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:45:39 smithi115 conmon[90393]: 2022-05-14T01:45:39.245+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:45:39.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:45:39 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:45:40.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:45:40 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:45:40.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:45:40 smithi047 ceph-mon[77147]: pgmap v4489: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:45:40.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:45:40 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 8270 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:45:40.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:45:40 smithi115 conmon[90393]: 2022-05-14T01:45:40.230+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:45:40.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:45:40 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:45:40.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:45:40 smithi115 ceph-mon[78593]: pgmap v4489: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:45:40.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:45:40 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 8270 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:45:41.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:45:41 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:45:41.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:45:41 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:45:41.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:45:41 smithi115 conmon[90393]: 2022-05-14T01:45:41.244+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:45:42.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:45:42 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:45:42.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:45:42 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:45:42.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:45:42 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:45:42.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:45:42 smithi047 ceph-mon[77147]: pgmap v4490: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:45:42.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:45:42 smithi115 conmon[90393]: 2022-05-14T01:45:42.195+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:45:42.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:45:42 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:45:42.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:45:42 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:45:42.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:45:42 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:45:42.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:45:42 smithi115 ceph-mon[78593]: pgmap v4490: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:45:43.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:45:43 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:45:43.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:45:43 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:45:43.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:45:43 smithi115 conmon[90393]: 2022-05-14T01:45:43.227+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:45:44.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:45:44 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:45:44.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:45:44 smithi047 ceph-mon[77147]: pgmap v4491: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:45:44.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:45:44 smithi115 conmon[90393]: 2022-05-14T01:45:44.226+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:45:44.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:45:44 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:45:44.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:45:44 smithi115 ceph-mon[78593]: pgmap v4491: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:45:45.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:45:45 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:45:45.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:45:45 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 8275 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:45:45.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:45:45 smithi115 conmon[90393]: 2022-05-14T01:45:45.270+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:45:45.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:45:45 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:45:45.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:45:45 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 8275 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:45:46.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:45:46 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:45:46.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:45:46 smithi047 ceph-mon[77147]: pgmap v4492: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:45:46.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:45:46 smithi115 conmon[90393]: 2022-05-14T01:45:46.222+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:45:46.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:45:46 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:45:46.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:45:46 smithi115 ceph-mon[78593]: pgmap v4492: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:45:47.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:45:47 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:45:47.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:45:47 smithi115 conmon[90393]: 2022-05-14T01:45:47.189+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:45:47.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:45:47 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:45:48.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:45:48 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:45:48.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:45:48 smithi047 ceph-mon[77147]: pgmap v4493: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:45:48.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:45:48 smithi115 conmon[90393]: 2022-05-14T01:45:48.157+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:45:48.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:45:48 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:45:48.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:45:48 smithi115 ceph-mon[78593]: pgmap v4493: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:45:49.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:45:49 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:45:49.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:45:49 smithi115 conmon[90393]: 2022-05-14T01:45:49.116+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:45:49.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:45:49 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:45:50.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:45:50 smithi115 conmon[90393]: 2022-05-14T01:45:50.099+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:45:50.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:45:50 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:45:50.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:45:50 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:45:50.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:45:50 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:45:50.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:45:50 smithi115 ceph-mon[78593]: pgmap v4494: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:45:50.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:45:50 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 8280 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:45:50.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:45:50 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:45:50.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:45:50 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:45:50.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:45:50 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:45:50.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:45:50 smithi047 ceph-mon[77147]: pgmap v4494: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:45:50.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:45:50 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 8280 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:45:51.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:45:51 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:45:51.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:45:51 smithi115 conmon[90393]: 2022-05-14T01:45:51.119+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:45:51.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:45:51 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:45:52.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:45:52 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:45:52.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:45:52 smithi047 ceph-mon[77147]: pgmap v4495: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:45:52.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:45:52 smithi115 conmon[90393]: 2022-05-14T01:45:52.160+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:45:52.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:45:52 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:45:52.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:45:52 smithi115 ceph-mon[78593]: pgmap v4495: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:45:53.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:45:53 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:45:53.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:45:53 smithi115 conmon[90393]: 2022-05-14T01:45:53.151+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:45:53.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:45:53 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:45:54.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:45:54 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:45:54.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:45:54 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:45:54.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:45:54 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:45:54.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:45:54 smithi047 ceph-mon[77147]: pgmap v4496: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:45:54.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:45:54 smithi115 conmon[90393]: 2022-05-14T01:45:54.139+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:45:54.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:45:54 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:45:54.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:45:54 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:45:54.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:45:54 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:45:54.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:45:54 smithi115 ceph-mon[78593]: pgmap v4496: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:45:55.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:45:55 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:45:55.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:45:55 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 8286 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:45:55.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:45:55 smithi115 conmon[90393]: 2022-05-14T01:45:55.171+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:45:55.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:45:55 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:45:55.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:45:55 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 8286 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:45:56.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:45:56 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:45:56.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:45:56 smithi047 ceph-mon[77147]: pgmap v4497: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:45:56.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:45:56 smithi115 conmon[90393]: 2022-05-14T01:45:56.206+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:45:56.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:45:56 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:45:56.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:45:56 smithi115 ceph-mon[78593]: pgmap v4497: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:45:57.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:45:57 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:45:57.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:45:57 smithi115 conmon[90393]: 2022-05-14T01:45:57.227+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:45:57.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:45:57 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:45:58.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:45:58 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:45:58.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:45:58 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:45:58.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:45:58 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:45:58.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:45:58 smithi047 ceph-mon[77147]: pgmap v4498: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:45:58.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:45:58 smithi115 conmon[90393]: 2022-05-14T01:45:58.269+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:45:58.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:45:58 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:45:58.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:45:58 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:45:58.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:45:58 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:45:58.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:45:58 smithi115 ceph-mon[78593]: pgmap v4498: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:45:59.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:45:59 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:45:59.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:45:59 smithi115 conmon[90393]: 2022-05-14T01:45:59.294+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:45:59.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:45:59 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:46:00.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:46:00 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:46:00.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:46:00 smithi047 ceph-mon[77147]: pgmap v4499: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:46:00.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:46:00 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 8291 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:46:00.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:46:00 smithi115 conmon[90393]: 2022-05-14T01:46:00.316+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:46:00.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:46:00 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:46:00.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:46:00 smithi115 ceph-mon[78593]: pgmap v4499: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:46:00.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:46:00 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 8291 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:46:01.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:46:01 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:46:01.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:46:01 smithi115 conmon[90393]: 2022-05-14T01:46:01.292+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:46:01.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:46:01 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:46:02.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:46:02 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:46:02.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:46:02 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:46:02.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:46:02 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:46:02.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:46:02 smithi047 ceph-mon[77147]: pgmap v4500: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:46:02.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:46:02 smithi115 conmon[90393]: 2022-05-14T01:46:02.299+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:46:02.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:46:02 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:46:02.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:46:02 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:46:02.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:46:02 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:46:02.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:46:02 smithi115 ceph-mon[78593]: pgmap v4500: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:46:03.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:46:03 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:46:03.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:46:03 smithi115 conmon[90393]: 2022-05-14T01:46:03.316+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:46:03.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:46:03 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:46:04.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:46:04 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:46:04.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:46:04 smithi047 ceph-mon[77147]: pgmap v4501: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:46:04.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:46:04 smithi115 conmon[90393]: 2022-05-14T01:46:04.295+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:46:04.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:46:04 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:46:04.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:46:04 smithi115 ceph-mon[78593]: pgmap v4501: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:46:05.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:46:05 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:46:05.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:46:05 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 8295 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:46:05.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:46:05 smithi115 conmon[90393]: 2022-05-14T01:46:05.340+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:46:05.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:46:05 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:46:05.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:46:05 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 8295 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:46:06.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:46:06 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:46:06.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:46:06 smithi047 ceph-mon[77147]: pgmap v4502: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:46:06.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:46:06 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:46:06.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:46:06 smithi115 ceph-mon[78593]: pgmap v4502: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:46:06.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:46:06 smithi115 conmon[90393]: 2022-05-14T01:46:06.363+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:46:07.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:46:07 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:46:07.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:46:07 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:46:07.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:46:07 smithi115 conmon[90393]: 2022-05-14T01:46:07.317+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:46:08.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:46:08 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:46:08.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:46:08 smithi047 ceph-mon[77147]: pgmap v4503: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:46:08.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:46:08 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:46:08.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:46:08 smithi115 ceph-mon[78593]: pgmap v4503: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:46:08.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:46:08 smithi115 conmon[90393]: 2022-05-14T01:46:08.309+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:46:09.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:46:09 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:46:09.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:46:09 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:46:09.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:46:09 smithi115 conmon[90393]: 2022-05-14T01:46:09.338+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:46:10.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:46:10 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:46:10.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:46:10 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:46:10.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:46:10 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:46:10.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:46:10 smithi047 ceph-mon[77147]: pgmap v4504: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:46:10.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:46:10 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 8300 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:46:10.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:46:10 smithi115 conmon[90393]: 2022-05-14T01:46:10.369+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:46:10.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:46:10 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:46:10.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:46:10 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:46:10.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:46:10 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:46:10.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:46:10 smithi115 ceph-mon[78593]: pgmap v4504: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:46:10.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:46:10 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 8300 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:46:11.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:46:11 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:46:11.612 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:46:11 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:46:11.612 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:46:11 smithi115 conmon[90393]: 2022-05-14T01:46:11.383+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:46:12.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:46:12 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:46:12.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:46:12 smithi047 ceph-mon[77147]: pgmap v4505: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:46:12.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:46:12 smithi115 conmon[90393]: 2022-05-14T01:46:12.420+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:46:12.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:46:12 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:46:12.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:46:12 smithi115 ceph-mon[78593]: pgmap v4505: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:46:13.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:46:13 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:46:13.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:46:13 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:46:13.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:46:13 smithi115 conmon[90393]: 2022-05-14T01:46:13.383+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:46:14.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:46:14 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:46:14.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:46:14 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:46:14.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:46:14 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:46:14.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:46:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T01:46:14.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:46:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T01:46:14.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:46:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T01:46:14.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:46:14 smithi115 ceph-mon[78593]: pgmap v4506: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:46:14.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:46:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T01:46:14.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:46:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T01:46:14.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:46:14 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:46:14.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:46:14 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:46:14.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:46:14 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:46:14.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:46:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T01:46:14.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:46:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T01:46:14.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:46:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T01:46:14.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:46:14 smithi047 ceph-mon[77147]: pgmap v4506: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:46:14.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:46:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T01:46:14.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:46:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T01:46:14.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:46:14 smithi115 conmon[90393]: 2022-05-14T01:46:14.375+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:46:15.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:46:15 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:46:15.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:46:15 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 8305 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:46:15.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:46:15 smithi115 conmon[90393]: 2022-05-14T01:46:15.424+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:46:15.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:46:15 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:46:15.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:46:15 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 8305 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:46:16.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:46:16 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:46:16.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:46:16 smithi047 ceph-mon[77147]: pgmap v4507: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:46:16.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:46:16 smithi115 conmon[90393]: 2022-05-14T01:46:16.378+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:46:16.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:46:16 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:46:16.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:46:16 smithi115 ceph-mon[78593]: pgmap v4507: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:46:17.041 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:46:16 smithi047 conmon[78521]: 2022-05-14T01:46:16.688+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:46:17.042 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:46:16 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T01:46:17.042 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:46:16 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:46:17.042 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:46:16 smithi047 conmon[78521]: yield 2022-05-14T01:46:17.042 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:46:16 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:46:17.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:46:16 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:46:17.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:46:16 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:46:17.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:46:16 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T01:46:17.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:46:16 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:46:17.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:46:16 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T01:46:17.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:46:16 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:46:17.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:46:16 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T01:46:17.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:46:16 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:46:17.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:46:16 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T01:46:17.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:46:16 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:46:17.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:46:16 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T01:46:17.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:46:16 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:46:17.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:46:16 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:46:17.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:46:16 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:46:17.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:46:17 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:46:17.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:46:17 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:46:17.542 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T01:46:17.542 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:46:17.542 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T01:46:17.542 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:46:17.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:46:17.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:46:17.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T01:46:17.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:46:17.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T01:46:17.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:46:17.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T01:46:17.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:46:17.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T01:46:17.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:46:17.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T01:46:17.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:46:17.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:46:17.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:46:17.546 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:46:17 smithi047 ceph-mon[77147]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T01:46:17.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:46:17 smithi115 conmon[90393]: 2022-05-14T01:46:17.389+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:46:17.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:46:17 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:46:17.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:46:17 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:46:17.609 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T01:46:17.609 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:46:17.609 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T01:46:17.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:46:17.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:46:17.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:46:17.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T01:46:17.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:46:17.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T01:46:17.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:46:17.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T01:46:17.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:46:17.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T01:46:17.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:46:17.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T01:46:17.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:46:17.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:46:17.613 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:46:17.613 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:46:17 smithi115 ceph-mon[78593]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T01:46:18.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:46:18 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:46:18.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:46:18 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:46:18.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:46:18 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:46:18.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:46:18 smithi047 ceph-mon[77147]: pgmap v4508: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:46:18.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:46:18 smithi115 conmon[90393]: 2022-05-14T01:46:18.436+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:46:18.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:46:18 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:46:18.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:46:18 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:46:18.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:46:18 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:46:18.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:46:18 smithi115 ceph-mon[78593]: pgmap v4508: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:46:19.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:46:19 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:46:19.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:46:19 smithi115 conmon[90393]: 2022-05-14T01:46:19.480+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:46:19.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:46:19 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:46:20.042 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:46:19 smithi047 conmon[78521]: 2022-05-14T01:46:19.759+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:46:20.042 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:46:19 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T01:46:20.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:46:19 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:46:20.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:46:19 smithi047 conmon[78521]: yield 2022-05-14T01:46:20.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:46:19 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:46:20.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:46:19 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:46:20.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:46:19 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:46:20.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:46:19 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T01:46:20.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:46:19 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:46:20.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:46:19 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T01:46:20.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:46:19 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:46:20.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:46:19 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T01:46:20.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:46:19 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:46:20.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:46:19 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T01:46:20.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:46:19 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:46:20.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:46:19 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T01:46:20.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:46:19 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:46:20.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:46:19 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:46:20.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:46:19 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:46:20.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:46:19 smithi047 conmon[78521]: 2022-05-14T01:46:19.760+0000 7f5344f82700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T01:46:20.048 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:46:19 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T01:46:20.048 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:46:19 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:46:20.048 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:46:19 smithi047 conmon[78521]: yield 2022-05-14T01:46:20.048 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:46:19 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:46:20.049 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:46:19 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:46:20.049 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:46:19 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:46:20.049 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:46:19 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T01:46:20.050 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:46:19 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:46:20.050 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:46:19 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T01:46:20.051 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:46:19 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:46:20.051 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:46:19 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T01:46:20.052 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:46:19 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:46:20.052 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:46:19 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T01:46:20.053 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:46:19 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:46:20.053 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:46:19 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T01:46:20.054 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:46:19 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:46:20.054 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:46:19 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:46:20.054 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:46:19 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:46:20.055 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:46:19 smithi047 conmon[78521]: 2022-05-14T01:46:20.055 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:46:19 smithi047 conmon[78521]: During handling of the above exception, another exception occurred: 2022-05-14T01:46:20.056 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:46:19 smithi047 conmon[78521]: 2022-05-14T01:46:20.056 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:46:19 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T01:46:20.056 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:46:19 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T01:46:20.057 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:46:19 smithi047 conmon[78521]: return f(*arg) 2022-05-14T01:46:20.057 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:46:19 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T01:46:20.057 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:46:19 smithi047 conmon[78521]: self._write_client_files(client_files, host) 2022-05-14T01:46:20.058 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:46:19 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T01:46:20.058 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:46:19 smithi047 conmon[78521]: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T01:46:20.058 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:46:19 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T01:46:20.059 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:46:19 smithi047 conmon[78521]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T01:46:20.059 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:46:19 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T01:46:20.060 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:46:19 smithi047 conmon[78521]: return self.event_loop.get_result(coro) 2022-05-14T01:46:20.060 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:46:19 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T01:46:20.060 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:46:19 smithi047 conmon[78521]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T01:46:20.061 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:46:19 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T01:46:20.061 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:46:19 smithi047 conmon[78521]: return self.__get_result() 2022-05-14T01:46:20.061 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:46:19 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T01:46:20.062 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:46:19 smithi047 conmon[78521]: raise self._exception 2022-05-14T01:46:20.063 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:46:19 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T01:46:20.063 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:46:19 smithi047 conmon[78521]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T01:46:20.063 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:46:19 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T01:46:20.064 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:46:19 smithi047 conmon[78521]: conn = await self._remote_connection(host, addr) 2022-05-14T01:46:20.064 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:46:19 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T01:46:20.064 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:46:19 smithi047 conmon[78521]: raise 2022-05-14T01:46:20.064 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:46:19 smithi047 conmon[78521]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T01:46:20.064 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:46:19 smithi047 conmon[78521]: self.gen.throw(type, value, traceback) 2022-05-14T01:46:20.065 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:46:19 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T01:46:20.065 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:46:19 smithi047 conmon[78521]: raise OrchestratorError(msg) 2022-05-14T01:46:20.065 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:46:19 smithi047 conmon[78521]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:46:20.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:46:20 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:46:20.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:46:20 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:46:20.542 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T01:46:20.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:46:20.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T01:46:20.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:46:20.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:46:20.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:46:20.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T01:46:20.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:46:20.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T01:46:20.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:46:20.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T01:46:20.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:46:20.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T01:46:20.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:46:20.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T01:46:20.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:46:20.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:46:20.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:46:20.546 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:46:20 smithi047 ceph-mon[77147]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T01:46:20.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T01:46:20.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:46:20.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T01:46:20.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:46:20.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:46:20.548 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:46:20.548 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T01:46:20.548 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:46:20.548 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T01:46:20.548 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:46:20.549 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T01:46:20.549 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:46:20.549 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T01:46:20.549 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:46:20.549 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T01:46:20.550 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:46:20.550 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:46:20.550 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:46:20.550 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T01:46:20.550 INFO:journalctl@ceph.mon.b.smithi047.stdout: During handling of the above exception, another exception occurred: 2022-05-14T01:46:20.551 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T01:46:20.551 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T01:46:20.551 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T01:46:20.552 INFO:journalctl@ceph.mon.b.smithi047.stdout: return f(*arg) 2022-05-14T01:46:20.552 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T01:46:20.552 INFO:journalctl@ceph.mon.b.smithi047.stdout: self._write_client_files(client_files, host) 2022-05-14T01:46:20.552 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T01:46:20.553 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T01:46:20.553 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T01:46:20.553 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T01:46:20.553 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T01:46:20.554 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.event_loop.get_result(coro) 2022-05-14T01:46:20.554 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T01:46:20.554 INFO:journalctl@ceph.mon.b.smithi047.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T01:46:20.554 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T01:46:20.555 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.__get_result() 2022-05-14T01:46:20.555 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T01:46:20.555 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise self._exception 2022-05-14T01:46:20.555 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T01:46:20.555 INFO:journalctl@ceph.mon.b.smithi047.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T01:46:20.556 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T01:46:20.556 INFO:journalctl@ceph.mon.b.smithi047.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T01:46:20.556 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T01:46:20.556 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise 2022-05-14T01:46:20.556 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T01:46:20.557 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.gen.throw(type, value, traceback) 2022-05-14T01:46:20.557 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T01:46:20.557 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OrchestratorError(msg) 2022-05-14T01:46:20.557 INFO:journalctl@ceph.mon.b.smithi047.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:46:20.557 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:46:20 smithi047 ceph-mon[77147]: pgmap v4509: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:46:20.558 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:46:20 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 8310 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:46:20.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:46:20 smithi115 conmon[90393]: 2022-05-14T01:46:20.448+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:46:20.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:46:20 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:46:20.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:46:20 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:46:20.609 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T01:46:20.609 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:46:20.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T01:46:20.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:46:20.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:46:20.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:46:20.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T01:46:20.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:46:20.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T01:46:20.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:46:20.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T01:46:20.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:46:20.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T01:46:20.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:46:20.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T01:46:20.613 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:46:20.613 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:46:20.613 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:46:20.613 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:46:20 smithi115 ceph-mon[78593]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T01:46:20.613 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T01:46:20.614 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:46:20.614 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T01:46:20.614 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:46:20.614 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:46:20.615 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:46:20.615 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T01:46:20.615 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:46:20.616 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T01:46:20.616 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:46:20.616 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T01:46:20.616 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:46:20.616 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T01:46:20.617 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:46:20.617 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T01:46:20.617 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:46:20.617 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:46:20.618 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:46:20.618 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T01:46:20.618 INFO:journalctl@ceph.mon.c.smithi115.stdout: During handling of the above exception, another exception occurred: 2022-05-14T01:46:20.618 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T01:46:20.618 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T01:46:20.619 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T01:46:20.619 INFO:journalctl@ceph.mon.c.smithi115.stdout: return f(*arg) 2022-05-14T01:46:20.619 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T01:46:20.619 INFO:journalctl@ceph.mon.c.smithi115.stdout: self._write_client_files(client_files, host) 2022-05-14T01:46:20.619 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T01:46:20.620 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T01:46:20.620 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T01:46:20.620 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T01:46:20.620 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T01:46:20.620 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.event_loop.get_result(coro) 2022-05-14T01:46:20.621 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T01:46:20.621 INFO:journalctl@ceph.mon.c.smithi115.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T01:46:20.621 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T01:46:20.621 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.__get_result() 2022-05-14T01:46:20.622 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T01:46:20.622 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise self._exception 2022-05-14T01:46:20.622 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T01:46:20.622 INFO:journalctl@ceph.mon.c.smithi115.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T01:46:20.622 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T01:46:20.623 INFO:journalctl@ceph.mon.c.smithi115.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T01:46:20.623 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T01:46:20.623 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise 2022-05-14T01:46:20.623 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T01:46:20.624 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.gen.throw(type, value, traceback) 2022-05-14T01:46:20.624 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T01:46:20.624 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OrchestratorError(msg) 2022-05-14T01:46:20.624 INFO:journalctl@ceph.mon.c.smithi115.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:46:20.624 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:46:20 smithi115 ceph-mon[78593]: pgmap v4509: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:46:20.625 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:46:20 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 8310 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:46:21.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:46:21 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:46:21.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:46:21 smithi115 conmon[90393]: 2022-05-14T01:46:21.459+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:46:21.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:46:21 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:46:22.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:46:22 smithi115 conmon[90393]: 2022-05-14T01:46:22.419+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:46:22.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:46:22 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:46:22.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:46:22 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:46:22.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:46:22 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:46:22.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:46:22 smithi115 ceph-mon[78593]: pgmap v4510: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:46:22.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:46:22 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:46:22.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:46:22 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:46:22.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:46:22 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:46:22.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:46:22 smithi047 ceph-mon[77147]: pgmap v4510: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:46:23.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:46:23 smithi115 conmon[90393]: 2022-05-14T01:46:23.406+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:46:23.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:46:23 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:46:23.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:46:23 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:46:24.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:46:24 smithi115 conmon[90393]: 2022-05-14T01:46:24.395+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:46:24.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:46:24 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:46:24.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:46:24 smithi115 ceph-mon[78593]: pgmap v4511: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:46:24.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:46:24 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:46:24.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:46:24 smithi047 ceph-mon[77147]: pgmap v4511: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:46:25.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:46:25 smithi115 conmon[90393]: 2022-05-14T01:46:25.420+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:46:25.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:46:25 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:46:25.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:46:25 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 8315 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:46:25.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:46:25 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:46:25.850 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:46:25 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 8315 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:46:26.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:46:26 smithi115 conmon[90393]: 2022-05-14T01:46:26.446+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:46:26.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:46:26 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:46:26.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:46:26 smithi115 ceph-mon[78593]: pgmap v4512: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:46:26.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:46:26 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:46:26.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:46:26 smithi047 ceph-mon[77147]: pgmap v4512: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:46:27.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:46:27 smithi115 conmon[90393]: 2022-05-14T01:46:27.455+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:46:27.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:46:27 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:46:27.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:46:27 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:46:28.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:46:28 smithi115 conmon[90393]: 2022-05-14T01:46:28.418+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:46:28.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:46:28 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:46:28.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:46:28 smithi115 ceph-mon[78593]: pgmap v4513: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:46:28.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:46:28 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:46:28.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:46:28 smithi047 ceph-mon[77147]: pgmap v4513: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:46:29.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:46:29 smithi115 conmon[90393]: 2022-05-14T01:46:29.461+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:46:29.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:46:29 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:46:29.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:46:29 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:46:30.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:46:30 smithi115 conmon[90393]: 2022-05-14T01:46:30.494+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:46:30.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:46:30 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:46:30.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:46:30 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:46:30.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:46:30 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:46:30.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:46:30 smithi115 ceph-mon[78593]: pgmap v4514: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:46:30.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:46:30 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 8320 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:46:30.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:46:30 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:46:30.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:46:30 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:46:30.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:46:30 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:46:30.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:46:30 smithi047 ceph-mon[77147]: pgmap v4514: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:46:30.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:46:30 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 8320 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:46:31.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:46:31 smithi115 conmon[90393]: 2022-05-14T01:46:31.509+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:46:31.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:46:31 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:46:31.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:46:31 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:46:32.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:46:32 smithi115 conmon[90393]: 2022-05-14T01:46:32.465+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:46:32.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:46:32 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:46:32.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:46:32 smithi115 ceph-mon[78593]: pgmap v4515: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:46:32.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:46:32 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:46:32.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:46:32 smithi047 ceph-mon[77147]: pgmap v4515: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:46:33.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:46:33 smithi115 conmon[90393]: 2022-05-14T01:46:33.457+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:46:33.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:46:33 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:46:33.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:46:33 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:46:34.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:46:34 smithi115 conmon[90393]: 2022-05-14T01:46:34.482+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:46:34.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:46:34 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:46:34.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:46:34 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:46:34.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:46:34 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:46:34.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:46:34 smithi115 ceph-mon[78593]: pgmap v4516: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:46:34.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:46:34 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:46:34.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:46:34 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:46:34.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:46:34 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:46:34.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:46:34 smithi047 ceph-mon[77147]: pgmap v4516: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:46:35.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:46:35 smithi115 conmon[90393]: 2022-05-14T01:46:35.485+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:46:35.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:46:35 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:46:35.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:46:35 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 8325 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:46:35.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:46:35 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:46:35.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:46:35 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 8325 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:46:36.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:46:36 smithi115 conmon[90393]: 2022-05-14T01:46:36.483+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:46:36.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:46:36 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:46:36.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:46:36 smithi115 ceph-mon[78593]: pgmap v4517: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:46:36.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:46:36 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:46:36.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:46:36 smithi047 ceph-mon[77147]: pgmap v4517: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:46:37.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:46:37 smithi115 conmon[90393]: 2022-05-14T01:46:37.523+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:46:37.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:46:37 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:46:37.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:46:37 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:46:38.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:46:38 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:46:38.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:46:38 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:46:38.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:46:38 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:46:38.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:46:38 smithi047 ceph-mon[77147]: pgmap v4518: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:46:38.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:46:38 smithi115 conmon[90393]: 2022-05-14T01:46:38.563+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:46:38.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:46:38 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:46:38.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:46:38 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:46:38.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:46:38 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:46:38.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:46:38 smithi115 ceph-mon[78593]: pgmap v4518: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:46:39.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:46:39 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:46:39.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:46:39 smithi115 conmon[90393]: 2022-05-14T01:46:39.515+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:46:39.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:46:39 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:46:40.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:46:40 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:46:40.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:46:40 smithi047 ceph-mon[77147]: pgmap v4519: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:46:40.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:46:40 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 8330 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:46:40.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:46:40 smithi115 conmon[90393]: 2022-05-14T01:46:40.542+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:46:40.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:46:40 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:46:40.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:46:40 smithi115 ceph-mon[78593]: pgmap v4519: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:46:40.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:46:40 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 8330 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:46:41.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:46:41 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:46:41.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:46:41 smithi115 conmon[90393]: 2022-05-14T01:46:41.539+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:46:41.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:46:41 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:46:42.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:46:42 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:46:42.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:46:42 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:46:42.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:46:42 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:46:42.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:46:42 smithi047 ceph-mon[77147]: pgmap v4520: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:46:42.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:46:42 smithi115 conmon[90393]: 2022-05-14T01:46:42.516+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:46:42.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:46:42 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:46:42.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:46:42 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:46:42.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:46:42 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:46:42.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:46:42 smithi115 ceph-mon[78593]: pgmap v4520: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:46:43.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:46:43 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:46:43.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:46:43 smithi115 conmon[90393]: 2022-05-14T01:46:43.520+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:46:43.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:46:43 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:46:44.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:46:44 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:46:44.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:46:44 smithi047 ceph-mon[77147]: pgmap v4521: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:46:44.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:46:44 smithi115 conmon[90393]: 2022-05-14T01:46:44.540+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:46:44.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:46:44 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:46:44.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:46:44 smithi115 ceph-mon[78593]: pgmap v4521: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:46:45.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:46:45 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:46:45.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:46:45 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 8335 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:46:45.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:46:45 smithi115 conmon[90393]: 2022-05-14T01:46:45.548+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:46:45.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:46:45 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:46:45.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:46:45 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 8335 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:46:46.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:46:46 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:46:46.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:46:46 smithi047 ceph-mon[77147]: pgmap v4522: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:46:46.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:46:46 smithi115 conmon[90393]: 2022-05-14T01:46:46.517+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:46:46.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:46:46 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:46:46.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:46:46 smithi115 ceph-mon[78593]: pgmap v4522: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:46:47.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:46:47 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:46:47.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:46:47 smithi115 conmon[90393]: 2022-05-14T01:46:47.529+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:46:47.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:46:47 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:46:48.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:46:48 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:46:48.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:46:48 smithi047 ceph-mon[77147]: pgmap v4523: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:46:48.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:46:48 smithi115 conmon[90393]: 2022-05-14T01:46:48.523+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:46:48.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:46:48 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:46:48.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:46:48 smithi115 ceph-mon[78593]: pgmap v4523: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:46:49.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:46:49 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:46:49.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:46:49 smithi115 conmon[90393]: 2022-05-14T01:46:49.562+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:46:49.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:46:49 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:46:50.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:46:50 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:46:50.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:46:50 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:46:50.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:46:50 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:46:50.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:46:50 smithi047 ceph-mon[77147]: pgmap v4524: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:46:50.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:46:50 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 8340 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:46:50.870 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:46:50 smithi115 conmon[90393]: 2022-05-14T01:46:50.533+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:46:50.870 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:46:50 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:46:50.870 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:46:50 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:46:50.871 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:46:50 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:46:50.871 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:46:50 smithi115 ceph-mon[78593]: pgmap v4524: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:46:50.871 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:46:50 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 8340 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:46:51.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:46:51 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:46:51.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:46:51 smithi115 conmon[90393]: 2022-05-14T01:46:51.548+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:46:51.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:46:51 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:46:52.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:46:52 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:46:52.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:46:52 smithi047 ceph-mon[77147]: pgmap v4525: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:46:52.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:46:52 smithi115 conmon[90393]: 2022-05-14T01:46:52.581+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:46:52.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:46:52 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:46:52.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:46:52 smithi115 ceph-mon[78593]: pgmap v4525: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:46:53.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:46:53 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:46:53.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:46:53 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:46:53.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:46:53 smithi115 conmon[90393]: 2022-05-14T01:46:53.604+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:46:54.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:46:54 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:46:54.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:46:54 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:46:54.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:46:54 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:46:54.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:46:54 smithi047 ceph-mon[77147]: pgmap v4526: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:46:54.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:46:54 smithi115 conmon[90393]: 2022-05-14T01:46:54.609+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:46:54.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:46:54 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:46:54.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:46:54 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:46:54.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:46:54 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:46:54.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:46:54 smithi115 ceph-mon[78593]: pgmap v4526: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:46:55.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:46:55 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 8345 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:46:55.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:46:55 smithi115 conmon[90393]: 2022-05-14T01:46:55.619+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:46:55.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:46:55 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 8345 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:46:56.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:46:56 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:46:56.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:46:56 smithi047 ceph-mon[77147]: pgmap v4527: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:46:56.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:46:56 smithi115 conmon[90393]: 2022-05-14T01:46:56.619+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:46:56.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:46:56 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:46:56.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:46:56 smithi115 ceph-mon[78593]: pgmap v4527: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:46:57.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:46:57 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:46:57.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:46:57 smithi115 conmon[90393]: 2022-05-14T01:46:57.580+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:46:57.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:46:57 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:46:58.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:46:58 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:46:58.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:46:58 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:46:58.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:46:58 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:46:58.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:46:58 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:46:58.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:46:58 smithi047 ceph-mon[77147]: pgmap v4528: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:46:58.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:46:58 smithi115 conmon[90393]: 2022-05-14T01:46:58.577+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:46:58.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:46:58 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:46:58.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:46:58 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:46:58.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:46:58 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:46:58.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:46:58 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:46:58.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:46:58 smithi115 ceph-mon[78593]: pgmap v4528: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:46:59.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:46:59 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:46:59.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:46:59 smithi115 conmon[90393]: 2022-05-14T01:46:59.622+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:46:59.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:46:59 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:47:00.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:47:00 smithi047 ceph-mon[77147]: pgmap v4529: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:47:00.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:47:00 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 8350 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:47:00.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:47:00 smithi115 conmon[90393]: 2022-05-14T01:47:00.640+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:47:00.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:47:00 smithi115 ceph-mon[78593]: pgmap v4529: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:47:00.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:47:00 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 8350 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:47:01.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:47:01 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:47:01.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:47:01 smithi115 conmon[90393]: 2022-05-14T01:47:01.603+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:47:01.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:47:01 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:47:02.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:47:02 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:47:02.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:47:02 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:47:02.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:47:02 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:47:02.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:47:02 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:47:02.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:47:02 smithi047 ceph-mon[77147]: pgmap v4530: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:47:02.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:47:02 smithi115 conmon[90393]: 2022-05-14T01:47:02.625+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:47:02.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:47:02 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:47:02.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:47:02 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:47:02.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:47:02 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:47:02.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:47:02 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:47:02.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:47:02 smithi115 ceph-mon[78593]: pgmap v4530: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:47:04.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:47:03 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:47:04.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:47:03 smithi115 conmon[90393]: 2022-05-14T01:47:03.622+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:47:04.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:47:03 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:47:04.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:47:04 smithi115 conmon[90393]: 2022-05-14T01:47:04.599+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:47:04.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:47:04 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:47:04.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:47:04 smithi115 ceph-mon[78593]: pgmap v4531: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:47:04.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:47:04 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:47:05.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:47:04 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:47:05.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:47:04 smithi047 ceph-mon[77147]: pgmap v4531: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:47:05.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:47:04 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:47:06.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:47:05 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 8355 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:47:06.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:47:05 smithi115 conmon[90393]: 2022-05-14T01:47:05.624+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:47:06.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:47:05 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 8355 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:47:07.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:47:06 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:47:07.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:47:06 smithi047 ceph-mon[77147]: pgmap v4532: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:47:07.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:47:06 smithi115 conmon[90393]: 2022-05-14T01:47:06.648+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:47:07.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:47:06 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:47:07.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:47:06 smithi115 ceph-mon[78593]: pgmap v4532: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:47:08.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:47:07 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:47:08.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:47:07 smithi115 conmon[90393]: 2022-05-14T01:47:07.685+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:47:08.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:47:07 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:47:09.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:47:08 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:47:09.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:47:08 smithi047 ceph-mon[77147]: pgmap v4533: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:47:09.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:47:08 smithi115 conmon[90393]: 2022-05-14T01:47:08.656+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:47:09.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:47:08 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:47:09.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:47:08 smithi115 ceph-mon[78593]: pgmap v4533: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:47:10.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:47:09 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:47:10.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:47:09 smithi115 conmon[90393]: 2022-05-14T01:47:09.672+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:47:10.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:47:09 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:47:11.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:47:10 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:47:11.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:47:10 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:47:11.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:47:10 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:47:11.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:47:10 smithi047 ceph-mon[77147]: pgmap v4534: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:47:11.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:47:10 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 8360 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:47:11.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:47:10 smithi115 conmon[90393]: 2022-05-14T01:47:10.696+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:47:11.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:47:10 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:47:11.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:47:10 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:47:11.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:47:10 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:47:11.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:47:10 smithi115 ceph-mon[78593]: pgmap v4534: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:47:11.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:47:10 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 8360 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:47:12.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:47:11 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:47:12.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:47:11 smithi115 conmon[90393]: 2022-05-14T01:47:11.679+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:47:12.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:47:11 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:47:13.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:47:12 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:47:13.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:47:12 smithi047 ceph-mon[77147]: pgmap v4535: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:47:13.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:47:12 smithi115 conmon[90393]: 2022-05-14T01:47:12.643+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:47:13.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:47:12 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:47:13.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:47:12 smithi115 ceph-mon[78593]: pgmap v4535: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:47:14.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:47:13 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:47:14.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:47:13 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:47:14.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:47:13 smithi115 conmon[90393]: 2022-05-14T01:47:13.676+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:47:15.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:47:14 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:47:15.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:47:14 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:47:15.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:47:14 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:47:15.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:47:14 smithi047 ceph-mon[77147]: pgmap v4536: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:47:15.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:47:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T01:47:15.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:47:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T01:47:15.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:47:14 smithi115 conmon[90393]: 2022-05-14T01:47:14.642+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:47:15.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:47:14 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:47:15.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:47:14 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:47:15.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:47:14 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:47:15.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:47:14 smithi115 ceph-mon[78593]: pgmap v4536: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:47:15.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:47:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T01:47:15.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:47:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T01:47:16.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:47:15 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:47:16.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:47:15 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 8365 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:47:16.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:47:15 smithi115 conmon[90393]: 2022-05-14T01:47:15.631+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:47:16.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:47:15 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:47:16.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:47:15 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 8365 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:47:17.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:47:16 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:47:17.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:47:16 smithi047 ceph-mon[77147]: pgmap v4537: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:47:17.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:47:16 smithi115 conmon[90393]: 2022-05-14T01:47:16.677+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:47:17.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:47:16 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:47:17.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:47:16 smithi115 ceph-mon[78593]: pgmap v4537: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:47:18.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:47:17 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:47:18.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:47:17 smithi115 conmon[90393]: 2022-05-14T01:47:17.662+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:47:18.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:47:17 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:47:19.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:47:18 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:47:19.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:47:18 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:47:19.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:47:18 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:47:19.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:47:18 smithi047 ceph-mon[77147]: pgmap v4538: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:47:19.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:47:18 smithi115 conmon[90393]: 2022-05-14T01:47:18.634+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:47:19.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:47:18 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:47:19.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:47:18 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:47:19.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:47:18 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:47:19.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:47:18 smithi115 ceph-mon[78593]: pgmap v4538: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:47:20.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:47:19 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:47:20.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:47:19 smithi115 conmon[90393]: 2022-05-14T01:47:19.667+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:47:20.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:47:19 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:47:20.541 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:47:20 smithi047 conmon[78521]: 2022-05-14T01:47:20.239+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:47:20.542 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:47:20 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T01:47:20.542 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:47:20 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:47:20.542 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:47:20 smithi047 conmon[78521]: yield 2022-05-14T01:47:20.542 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:47:20 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:47:20.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:47:20 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:47:20.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:47:20 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:47:20.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:47:20 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T01:47:20.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:47:20 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:47:20.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:47:20 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T01:47:20.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:47:20 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:47:20.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:47:20 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T01:47:20.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:47:20 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:47:20.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:47:20 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T01:47:20.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:47:20 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:47:20.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:47:20 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T01:47:20.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:47:20 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:47:20.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:47:20 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:47:20.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:47:20 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:47:21.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:47:20 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:47:21.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:47:20 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T01:47:21.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:47:20 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T01:47:21.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:47:20 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T01:47:21.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:47:20 smithi047 ceph-mon[77147]: pgmap v4539: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:47:21.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:47:20 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 8370 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:47:21.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:47:20 smithi115 conmon[90393]: 2022-05-14T01:47:20.708+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:47:21.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:47:20 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:47:21.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:47:20 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T01:47:21.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:47:20 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T01:47:21.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:47:20 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T01:47:21.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:47:20 smithi115 ceph-mon[78593]: pgmap v4539: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:47:21.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:47:20 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 8370 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:47:22.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:47:21 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:47:22.042 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T01:47:22.042 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:47:22.042 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T01:47:22.042 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:47:22.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:47:22.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:47:22.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T01:47:22.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:47:22.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T01:47:22.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:47:22.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T01:47:22.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:47:22.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T01:47:22.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:47:22.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T01:47:22.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:47:22.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:47:22.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:47:22.046 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:47:21 smithi047 ceph-mon[77147]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T01:47:22.046 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:47:21 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:47:22.047 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:47:21 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:47:22.047 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:47:21 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:47:22.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:47:21 smithi115 conmon[90393]: 2022-05-14T01:47:21.681+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:47:22.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:47:21 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:47:22.108 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T01:47:22.109 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:47:22.109 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T01:47:22.109 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:47:22.109 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:47:22.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:47:22.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T01:47:22.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:47:22.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T01:47:22.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:47:22.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T01:47:22.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:47:22.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T01:47:22.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:47:22.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T01:47:22.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:47:22.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:47:22.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:47:22.112 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:47:21 smithi115 ceph-mon[78593]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T01:47:22.113 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:47:21 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:47:22.114 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:47:21 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:47:22.114 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:47:21 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:47:23.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:47:22 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:47:23.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:47:22 smithi047 ceph-mon[77147]: pgmap v4540: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:47:23.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:47:22 smithi115 conmon[90393]: 2022-05-14T01:47:22.695+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:47:23.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:47:22 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:47:23.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:47:22 smithi115 ceph-mon[78593]: pgmap v4540: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:47:23.744 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:47:23 smithi047 conmon[78521]: 2022-05-14T01:47:23.311+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:47:23.744 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:47:23 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T01:47:23.744 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:47:23 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:47:23.745 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:47:23 smithi047 conmon[78521]: yield 2022-05-14T01:47:23.745 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:47:23 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:47:23.745 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:47:23 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:47:23.745 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:47:23 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:47:23.746 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:47:23 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T01:47:23.746 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:47:23 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:47:23.746 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:47:23 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T01:47:23.746 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:47:23 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:47:23.747 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:47:23 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T01:47:23.747 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:47:23 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:47:23.747 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:47:23 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T01:47:23.747 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:47:23 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:47:23.747 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:47:23 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T01:47:23.748 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:47:23 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:47:23.748 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:47:23 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:47:23.748 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:47:23 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:47:23.748 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:47:23 smithi047 conmon[78521]: 2022-05-14T01:47:23.312+0000 7f5344781700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T01:47:23.749 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:47:23 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T01:47:23.749 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:47:23 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:47:23.749 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:47:23 smithi047 conmon[78521]: yield 2022-05-14T01:47:23.750 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:47:23 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:47:23.750 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:47:23 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:47:23.750 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:47:23 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:47:23.750 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:47:23 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T01:47:23.751 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:47:23 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:47:23.751 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:47:23 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T01:47:23.751 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:47:23 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:47:23.751 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:47:23 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T01:47:23.751 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:47:23 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:47:23.752 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:47:23 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T01:47:23.752 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:47:23 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:47:23.752 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:47:23 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T01:47:23.752 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:47:23 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:47:23.752 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:47:23 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:47:23.753 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:47:23 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:47:23.753 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:47:23 smithi047 conmon[78521]: 2022-05-14T01:47:23.753 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:47:23 smithi047 conmon[78521]: During handling of the above exception, another exception occurred: 2022-05-14T01:47:23.753 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:47:23 smithi047 conmon[78521]: 2022-05-14T01:47:23.754 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:47:23 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T01:47:23.754 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:47:23 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T01:47:23.754 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:47:23 smithi047 conmon[78521]: return f(*arg) 2022-05-14T01:47:23.754 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:47:23 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T01:47:23.755 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:47:23 smithi047 conmon[78521]: self._write_client_files(client_files, host) 2022-05-14T01:47:23.755 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:47:23 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T01:47:23.755 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:47:23 smithi047 conmon[78521]: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T01:47:23.755 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:47:23 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T01:47:23.755 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:47:23 smithi047 conmon[78521]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T01:47:23.756 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:47:23 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T01:47:23.756 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:47:23 smithi047 conmon[78521]: return self.event_loop.get_result(coro) 2022-05-14T01:47:23.756 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:47:23 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T01:47:23.756 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:47:23 smithi047 conmon[78521]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T01:47:23.756 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:47:23 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T01:47:23.757 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:47:23 smithi047 conmon[78521]: return self.__get_result() 2022-05-14T01:47:23.757 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:47:23 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T01:47:23.757 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:47:23 smithi047 conmon[78521]: raise self._exception 2022-05-14T01:47:23.757 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:47:23 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T01:47:23.758 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:47:23 smithi047 conmon[78521]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T01:47:23.758 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:47:23 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T01:47:23.758 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:47:23 smithi047 conmon[78521]: conn = await self._remote_connection(host, addr) 2022-05-14T01:47:23.758 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:47:23 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T01:47:23.758 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:47:23 smithi047 conmon[78521]: raise 2022-05-14T01:47:23.759 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:47:23 smithi047 conmon[78521]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T01:47:23.759 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:47:23 smithi047 conmon[78521]: self.gen.throw(type, value, traceback) 2022-05-14T01:47:23.759 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:47:23 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T01:47:23.760 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:47:23 smithi047 conmon[78521]: raise OrchestratorError(msg) 2022-05-14T01:47:23.760 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:47:23 smithi047 conmon[78521]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:47:24.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:47:23 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:47:24.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:47:23 smithi115 conmon[90393]: 2022-05-14T01:47:23.668+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:47:24.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:47:23 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:47:25.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:47:24 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:47:25.042 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T01:47:25.042 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:47:25.042 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T01:47:25.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:47:25.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:47:25.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:47:25.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T01:47:25.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:47:25.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T01:47:25.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:47:25.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T01:47:25.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:47:25.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T01:47:25.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:47:25.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T01:47:25.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:47:25.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:47:25.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:47:25.046 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:47:24 smithi047 ceph-mon[77147]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T01:47:25.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T01:47:25.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:47:25.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T01:47:25.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:47:25.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:47:25.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:47:25.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T01:47:25.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:47:25.049 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T01:47:25.049 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:47:25.050 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T01:47:25.050 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:47:25.051 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T01:47:25.051 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:47:25.051 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T01:47:25.052 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:47:25.052 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:47:25.052 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:47:25.053 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T01:47:25.053 INFO:journalctl@ceph.mon.b.smithi047.stdout: During handling of the above exception, another exception occurred: 2022-05-14T01:47:25.054 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T01:47:25.054 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T01:47:25.054 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T01:47:25.055 INFO:journalctl@ceph.mon.b.smithi047.stdout: return f(*arg) 2022-05-14T01:47:25.055 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T01:47:25.055 INFO:journalctl@ceph.mon.b.smithi047.stdout: self._write_client_files(client_files, host) 2022-05-14T01:47:25.056 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T01:47:25.056 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T01:47:25.056 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T01:47:25.057 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T01:47:25.057 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T01:47:25.058 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.event_loop.get_result(coro) 2022-05-14T01:47:25.058 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T01:47:25.059 INFO:journalctl@ceph.mon.b.smithi047.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T01:47:25.059 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T01:47:25.060 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.__get_result() 2022-05-14T01:47:25.060 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T01:47:25.061 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise self._exception 2022-05-14T01:47:25.061 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T01:47:25.062 INFO:journalctl@ceph.mon.b.smithi047.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T01:47:25.062 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T01:47:25.063 INFO:journalctl@ceph.mon.b.smithi047.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T01:47:25.063 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T01:47:25.064 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise 2022-05-14T01:47:25.064 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T01:47:25.064 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.gen.throw(type, value, traceback) 2022-05-14T01:47:25.065 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T01:47:25.065 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OrchestratorError(msg) 2022-05-14T01:47:25.065 INFO:journalctl@ceph.mon.b.smithi047.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:47:25.066 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:47:24 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:47:25.066 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:47:24 smithi047 ceph-mon[77147]: pgmap v4541: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:47:25.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:47:24 smithi115 conmon[90393]: 2022-05-14T01:47:24.624+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:47:25.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:47:24 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:47:25.109 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T01:47:25.109 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:47:25.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T01:47:25.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:47:25.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:47:25.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:47:25.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T01:47:25.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:47:25.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T01:47:25.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:47:25.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T01:47:25.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:47:25.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T01:47:25.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:47:25.113 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T01:47:25.113 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:47:25.113 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:47:25.113 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:47:25.113 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:47:24 smithi115 ceph-mon[78593]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T01:47:25.114 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T01:47:25.114 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:47:25.114 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T01:47:25.114 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:47:25.115 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:47:25.115 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:47:25.115 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T01:47:25.115 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:47:25.115 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T01:47:25.116 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:47:25.116 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T01:47:25.116 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:47:25.116 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T01:47:25.116 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:47:25.117 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T01:47:25.117 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:47:25.117 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:47:25.120 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:47:25.120 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T01:47:25.120 INFO:journalctl@ceph.mon.c.smithi115.stdout: During handling of the above exception, another exception occurred: 2022-05-14T01:47:25.120 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T01:47:25.121 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T01:47:25.121 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T01:47:25.121 INFO:journalctl@ceph.mon.c.smithi115.stdout: return f(*arg) 2022-05-14T01:47:25.121 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T01:47:25.122 INFO:journalctl@ceph.mon.c.smithi115.stdout: self._write_client_files(client_files, host) 2022-05-14T01:47:25.122 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T01:47:25.122 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T01:47:25.122 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T01:47:25.122 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T01:47:25.123 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T01:47:25.123 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.event_loop.get_result(coro) 2022-05-14T01:47:25.123 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T01:47:25.123 INFO:journalctl@ceph.mon.c.smithi115.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T01:47:25.123 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T01:47:25.124 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.__get_result() 2022-05-14T01:47:25.124 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T01:47:25.124 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise self._exception 2022-05-14T01:47:25.124 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T01:47:25.125 INFO:journalctl@ceph.mon.c.smithi115.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T01:47:25.125 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T01:47:25.125 INFO:journalctl@ceph.mon.c.smithi115.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T01:47:25.125 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T01:47:25.125 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise 2022-05-14T01:47:25.126 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T01:47:25.126 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.gen.throw(type, value, traceback) 2022-05-14T01:47:25.126 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T01:47:25.126 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OrchestratorError(msg) 2022-05-14T01:47:25.127 INFO:journalctl@ceph.mon.c.smithi115.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:47:25.127 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:47:24 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:47:25.127 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:47:24 smithi115 ceph-mon[78593]: pgmap v4541: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:47:26.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:47:25 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:47:26.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:47:25 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 8375 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:47:26.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:47:25 smithi115 conmon[90393]: 2022-05-14T01:47:25.644+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:47:26.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:47:25 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:47:26.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:47:25 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 8375 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:47:27.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:47:26 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:47:27.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:47:26 smithi047 ceph-mon[77147]: pgmap v4542: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:47:27.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:47:26 smithi115 conmon[90393]: 2022-05-14T01:47:26.676+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:47:27.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:47:26 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:47:27.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:47:26 smithi115 ceph-mon[78593]: pgmap v4542: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:47:28.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:47:27 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:47:28.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:47:27 smithi115 conmon[90393]: 2022-05-14T01:47:27.680+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:47:28.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:47:27 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:47:29.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:47:28 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:47:29.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:47:28 smithi047 ceph-mon[77147]: pgmap v4543: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:47:29.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:47:28 smithi115 conmon[90393]: 2022-05-14T01:47:28.705+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:47:29.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:47:28 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:47:29.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:47:28 smithi115 ceph-mon[78593]: pgmap v4543: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:47:30.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:47:29 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:47:30.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:47:29 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:47:30.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:47:29 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:47:30.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:47:29 smithi115 conmon[90393]: 2022-05-14T01:47:29.671+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:47:30.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:47:29 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:47:30.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:47:29 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:47:30.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:47:29 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:47:31.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:47:30 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:47:31.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:47:30 smithi047 ceph-mon[77147]: pgmap v4544: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:47:31.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:47:30 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 8380 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:47:31.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:47:30 smithi115 conmon[90393]: 2022-05-14T01:47:30.684+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:47:31.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:47:30 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:47:31.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:47:30 smithi115 ceph-mon[78593]: pgmap v4544: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:47:31.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:47:30 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 8380 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:47:32.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:47:31 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:47:32.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:47:31 smithi115 conmon[90393]: 2022-05-14T01:47:31.728+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:47:32.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:47:31 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:47:33.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:47:32 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:47:33.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:47:32 smithi047 ceph-mon[77147]: pgmap v4545: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:47:33.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:47:32 smithi115 conmon[90393]: 2022-05-14T01:47:32.702+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:47:33.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:47:32 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:47:33.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:47:32 smithi115 ceph-mon[78593]: pgmap v4545: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:47:34.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:47:33 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:47:34.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:47:33 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:47:34.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:47:33 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:47:34.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:47:33 smithi115 conmon[90393]: 2022-05-14T01:47:33.726+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:47:34.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:47:33 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:47:34.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:47:33 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:47:34.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:47:33 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:47:35.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:47:34 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:47:35.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:47:34 smithi047 ceph-mon[77147]: pgmap v4546: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:47:35.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:47:34 smithi115 conmon[90393]: 2022-05-14T01:47:34.756+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:47:35.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:47:34 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:47:35.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:47:34 smithi115 ceph-mon[78593]: pgmap v4546: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:47:36.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:47:35 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:47:36.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:47:35 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 8385 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:47:36.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:47:35 smithi115 conmon[90393]: 2022-05-14T01:47:35.713+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:47:36.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:47:35 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:47:36.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:47:35 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 8385 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:47:37.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:47:36 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:47:37.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:47:36 smithi047 ceph-mon[77147]: pgmap v4547: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:47:37.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:47:36 smithi115 conmon[90393]: 2022-05-14T01:47:36.685+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:47:37.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:47:36 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:47:37.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:47:36 smithi115 ceph-mon[78593]: pgmap v4547: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:47:38.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:47:37 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:47:38.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:47:37 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:47:38.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:47:37 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:47:38.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:47:37 smithi115 conmon[90393]: 2022-05-14T01:47:37.734+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:47:38.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:47:37 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:47:38.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:47:37 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:47:38.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:47:37 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:47:39.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:47:38 smithi115 conmon[90393]: 2022-05-14T01:47:38.687+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:47:39.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:47:38 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:47:39.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:47:38 smithi115 ceph-mon[78593]: pgmap v4548: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:47:39.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:47:38 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:47:39.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:47:38 smithi047 ceph-mon[77147]: pgmap v4548: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:47:40.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:47:39 smithi115 conmon[90393]: 2022-05-14T01:47:39.728+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:47:40.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:47:39 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:47:40.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:47:39 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:47:41.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:47:40 smithi115 conmon[90393]: 2022-05-14T01:47:40.778+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:47:41.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:47:40 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:47:41.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:47:40 smithi115 ceph-mon[78593]: pgmap v4549: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:47:41.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:47:40 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 8395 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:47:41.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:47:40 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:47:41.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:47:40 smithi047 ceph-mon[77147]: pgmap v4549: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:47:41.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:47:40 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 8395 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:47:42.092 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:47:41 smithi115 conmon[90393]: 2022-05-14T01:47:41.740+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:47:42.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:47:42 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:47:42.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:47:42 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:47:42.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:47:42 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:47:42.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:47:42 smithi115 ceph-mon[78593]: pgmap v4550: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:47:42.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:47:42 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:47:42.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:47:42 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:47:42.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:47:42 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:47:42.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:47:42 smithi047 ceph-mon[77147]: pgmap v4550: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:47:43.093 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:47:42 smithi115 conmon[90393]: 2022-05-14T01:47:42.701+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:47:43.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:47:43 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:47:43.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:47:43 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:47:44.101 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:47:43 smithi115 conmon[90393]: 2022-05-14T01:47:43.702+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:47:44.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:47:44 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:47:44.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:47:44 smithi115 ceph-mon[78593]: pgmap v4551: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:47:44.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:47:44 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:47:44.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:47:44 smithi047 ceph-mon[77147]: pgmap v4551: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:47:45.100 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:47:44 smithi115 conmon[90393]: 2022-05-14T01:47:44.702+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:47:45.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:47:45 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:47:45.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:47:45 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:47:46.106 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:47:45 smithi115 conmon[90393]: 2022-05-14T01:47:45.654+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:47:46.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:47:46 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:47:46.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:47:46 smithi115 ceph-mon[78593]: pgmap v4552: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:47:46.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:47:46 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:47:46.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:47:46 smithi047 ceph-mon[77147]: pgmap v4552: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:47:47.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:47:46 smithi115 conmon[90393]: 2022-05-14T01:47:46.689+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:47:47.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:47:47 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:47:47.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:47:47 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 8400 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:47:47.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:47:47 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:47:47.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:47:47 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 8400 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:47:48.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:47:47 smithi115 conmon[90393]: 2022-05-14T01:47:47.691+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:47:48.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:47:48 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:47:48.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:47:48 smithi047 ceph-mon[77147]: pgmap v4553: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:47:48.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:47:48 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:47:48.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:47:48 smithi115 ceph-mon[78593]: pgmap v4553: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:47:49.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:47:48 smithi115 conmon[90393]: 2022-05-14T01:47:48.708+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:47:49.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:47:49 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:47:49.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:47:49 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:47:50.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:47:49 smithi115 conmon[90393]: 2022-05-14T01:47:49.675+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:47:50.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:47:50 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:47:50.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:47:50 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:47:50.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:47:50 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:47:50.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:47:50 smithi047 ceph-mon[77147]: pgmap v4554: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:47:50.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:47:50 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:47:50.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:47:50 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:47:50.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:47:50 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:47:50.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:47:50 smithi115 ceph-mon[78593]: pgmap v4554: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:47:51.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:47:50 smithi115 conmon[90393]: 2022-05-14T01:47:50.656+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:47:51.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:47:51 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:47:51.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:47:51 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:47:52.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:47:51 smithi115 conmon[90393]: 2022-05-14T01:47:51.695+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:47:52.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:47:52 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:47:52.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:47:52 smithi047 ceph-mon[77147]: pgmap v4555: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:47:52.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:47:52 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:47:52.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:47:52 smithi115 ceph-mon[78593]: pgmap v4555: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:47:53.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:47:52 smithi115 conmon[90393]: 2022-05-14T01:47:52.668+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:47:53.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:47:53 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:47:53.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:47:53 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:47:54.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:47:53 smithi115 conmon[90393]: 2022-05-14T01:47:53.712+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:47:54.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:47:54 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:47:54.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:47:54 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:47:54.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:47:54 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:47:54.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:47:54 smithi047 ceph-mon[77147]: pgmap v4556: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:47:54.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:47:54 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:47:54.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:47:54 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:47:54.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:47:54 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:47:54.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:47:54 smithi115 ceph-mon[78593]: pgmap v4556: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:47:55.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:47:54 smithi115 conmon[90393]: 2022-05-14T01:47:54.734+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:47:55.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:47:55 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:47:55.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:47:55 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 8405 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:47:55.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:47:55 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:47:55.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:47:55 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 8405 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:47:56.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:47:55 smithi115 conmon[90393]: 2022-05-14T01:47:55.707+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:47:56.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:47:56 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:47:56.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:47:56 smithi047 ceph-mon[77147]: pgmap v4557: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:47:56.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:47:56 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:47:56.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:47:56 smithi115 ceph-mon[78593]: pgmap v4557: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:47:57.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:47:56 smithi115 conmon[90393]: 2022-05-14T01:47:56.688+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:47:57.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:47:57 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:47:57.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:47:57 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:47:58.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:47:57 smithi115 conmon[90393]: 2022-05-14T01:47:57.641+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:47:58.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:47:58 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:47:58.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:47:58 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:47:58.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:47:58 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:47:58.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:47:58 smithi047 ceph-mon[77147]: pgmap v4558: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:47:58.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:47:58 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:47:58.612 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:47:58 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:47:58.612 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:47:58 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:47:58.612 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:47:58 smithi115 ceph-mon[78593]: pgmap v4558: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:47:59.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:47:58 smithi115 conmon[90393]: 2022-05-14T01:47:58.666+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:47:59.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:47:59 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:47:59.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:47:59 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:48:00.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:47:59 smithi115 conmon[90393]: 2022-05-14T01:47:59.703+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:48:00.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:48:00 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:48:00.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:48:00 smithi047 ceph-mon[77147]: pgmap v4559: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:48:00.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:48:00 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 8410 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:48:00.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:48:00 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:48:00.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:48:00 smithi115 ceph-mon[78593]: pgmap v4559: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:48:00.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:48:00 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 8410 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:48:01.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:48:00 smithi115 conmon[90393]: 2022-05-14T01:48:00.682+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:48:01.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:48:01 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:48:01.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:48:01 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:48:02.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:48:01 smithi115 conmon[90393]: 2022-05-14T01:48:01.667+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:48:02.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:48:02 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:48:02.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:48:02 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:48:02.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:48:02 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:48:02.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:48:02 smithi047 ceph-mon[77147]: pgmap v4560: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:48:02.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:48:02 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:48:02.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:48:02 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:48:02.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:48:02 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:48:02.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:48:02 smithi115 ceph-mon[78593]: pgmap v4560: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:48:03.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:48:02 smithi115 conmon[90393]: 2022-05-14T01:48:02.701+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:48:03.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:48:03 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:48:03.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:48:03 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:48:04.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:48:03 smithi115 conmon[90393]: 2022-05-14T01:48:03.687+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:48:04.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:48:04 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:48:04.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:48:04 smithi047 ceph-mon[77147]: pgmap v4561: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:48:04.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:48:04 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:48:04.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:48:04 smithi115 ceph-mon[78593]: pgmap v4561: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:48:05.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:48:04 smithi115 conmon[90393]: 2022-05-14T01:48:04.644+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:48:05.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:48:05 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:48:05.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:48:05 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 8415 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:48:05.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:48:05 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:48:05.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:48:05 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 8415 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:48:06.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:48:05 smithi115 conmon[90393]: 2022-05-14T01:48:05.638+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:48:06.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:48:06 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:48:06.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:48:06 smithi047 ceph-mon[77147]: pgmap v4562: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:48:06.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:48:06 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:48:06.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:48:06 smithi115 ceph-mon[78593]: pgmap v4562: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:48:07.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:48:06 smithi115 conmon[90393]: 2022-05-14T01:48:06.678+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:48:07.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:48:07 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:48:07.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:48:07 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:48:08.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:48:07 smithi115 conmon[90393]: 2022-05-14T01:48:07.690+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:48:08.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:48:08 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:48:08.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:48:08 smithi047 ceph-mon[77147]: pgmap v4563: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:48:08.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:48:08 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:48:08.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:48:08 smithi115 ceph-mon[78593]: pgmap v4563: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:48:09.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:48:08 smithi115 conmon[90393]: 2022-05-14T01:48:08.728+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:48:09.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:48:09 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:48:09.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:48:09 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:48:10.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:48:09 smithi115 conmon[90393]: 2022-05-14T01:48:09.743+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:48:10.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:48:10 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:48:10.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:48:10 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:48:10.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:48:10 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:48:10.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:48:10 smithi047 ceph-mon[77147]: pgmap v4564: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:48:10.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:48:10 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 8420 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:48:10.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:48:10 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:48:10.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:48:10 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:48:10.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:48:10 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:48:10.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:48:10 smithi115 ceph-mon[78593]: pgmap v4564: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:48:10.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:48:10 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 8420 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:48:11.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:48:10 smithi115 conmon[90393]: 2022-05-14T01:48:10.763+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:48:11.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:48:11 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:48:11.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:48:11 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:48:12.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:48:11 smithi115 conmon[90393]: 2022-05-14T01:48:11.811+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:48:12.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:48:12 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:48:12.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:48:12 smithi047 ceph-mon[77147]: pgmap v4565: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:48:12.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:48:12 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:48:12.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:48:12 smithi115 ceph-mon[78593]: pgmap v4565: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:48:13.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:48:12 smithi115 conmon[90393]: 2022-05-14T01:48:12.774+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:48:13.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:48:13 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:48:13.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:48:13 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:48:14.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:48:13 smithi115 conmon[90393]: 2022-05-14T01:48:13.727+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:48:14.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:48:14 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:48:14.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:48:14 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:48:14.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:48:14 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:48:14.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:48:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T01:48:14.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:48:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T01:48:14.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:48:14 smithi047 ceph-mon[77147]: pgmap v4566: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:48:14.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:48:14 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:48:14.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:48:14 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:48:14.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:48:14 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:48:14.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:48:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T01:48:14.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:48:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T01:48:14.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:48:14 smithi115 ceph-mon[78593]: pgmap v4566: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:48:15.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:48:14 smithi115 conmon[90393]: 2022-05-14T01:48:14.680+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:48:15.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:48:15 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:48:15.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:48:15 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 8425 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:48:15.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:48:15 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:48:15.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:48:15 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 8425 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:48:16.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:48:15 smithi115 conmon[90393]: 2022-05-14T01:48:15.632+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:48:16.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:48:16 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:48:16.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:48:16 smithi047 ceph-mon[77147]: pgmap v4567: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:48:16.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:48:16 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:48:16.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:48:16 smithi115 ceph-mon[78593]: pgmap v4567: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:48:17.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:48:16 smithi115 conmon[90393]: 2022-05-14T01:48:16.609+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:48:17.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:48:17 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:48:17.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:48:17 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:48:17.566 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:48:17 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:48:17.567 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:48:17 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:48:17.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:48:17 smithi115 conmon[90393]: 2022-05-14T01:48:17.565+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:48:18.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:48:18 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:48:18.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:48:18 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:48:18.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:48:18 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:48:18.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:48:18 smithi047 ceph-mon[77147]: pgmap v4568: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:48:18.575 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:48:18 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:48:18.575 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:48:18 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:48:18.576 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:48:18 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:48:18.576 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:48:18 smithi115 ceph-mon[78593]: pgmap v4568: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:48:18.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:48:18 smithi115 conmon[90393]: 2022-05-14T01:48:18.573+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:48:19.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:48:19 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:48:19.556 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:48:19 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:48:19.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:48:19 smithi115 conmon[90393]: 2022-05-14T01:48:19.555+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:48:20.531 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:48:20 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:48:20.531 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:48:20 smithi115 ceph-mon[78593]: pgmap v4569: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:48:20.532 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:48:20 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 8430 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:48:20.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:48:20 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:48:20.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:48:20 smithi047 ceph-mon[77147]: pgmap v4569: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:48:20.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:48:20 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 8430 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:48:20.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:48:20 smithi115 conmon[90393]: 2022-05-14T01:48:20.530+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:48:21.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:48:21 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:48:21.578 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:48:21 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:48:21.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:48:21 smithi115 conmon[90393]: 2022-05-14T01:48:21.578+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:48:22.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:48:22 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:48:22.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:48:22 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:48:22.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:48:22 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:48:22.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:48:22 smithi047 ceph-mon[77147]: pgmap v4570: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:48:22.585 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:48:22 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:48:22.585 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:48:22 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:48:22.585 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:48:22 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:48:22.585 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:48:22 smithi115 ceph-mon[78593]: pgmap v4570: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:48:22.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:48:22 smithi115 conmon[90393]: 2022-05-14T01:48:22.584+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:48:23.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:48:23 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:48:23.565 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:48:23 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:48:23.820 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:48:23 smithi115 conmon[90393]: 2022-05-14T01:48:23.565+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:48:24.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:48:24 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T01:48:24.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:48:24 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T01:48:24.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:48:24 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T01:48:24.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:48:24 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:48:24.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:48:24 smithi047 ceph-mon[77147]: pgmap v4571: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:48:24.566 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:48:24 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T01:48:24.567 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:48:24 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T01:48:24.567 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:48:24 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T01:48:24.567 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:48:24 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:48:24.567 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:48:24 smithi115 ceph-mon[78593]: pgmap v4571: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:48:24.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:48:24 smithi115 conmon[90393]: 2022-05-14T01:48:24.565+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:48:25.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:48:25 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:48:25.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:48:25 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 8435 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:48:25.558 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:48:25 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:48:25.558 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:48:25 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 8435 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:48:25.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:48:25 smithi115 conmon[90393]: 2022-05-14T01:48:25.556+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:48:26.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:48:26 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:48:26.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:48:26 smithi047 ceph-mon[77147]: pgmap v4572: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:48:26.542 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:48:26 smithi047 conmon[78521]: 2022-05-14T01:48:26.416+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:48:26.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:48:26 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T01:48:26.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:48:26 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:48:26.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:48:26 smithi047 conmon[78521]: yield 2022-05-14T01:48:26.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:48:26 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:48:26.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:48:26 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:48:26.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:48:26 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:48:26.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:48:26 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T01:48:26.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:48:26 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:48:26.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:48:26 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T01:48:26.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:48:26 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:48:26.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:48:26 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T01:48:26.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:48:26 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:48:26.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:48:26 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T01:48:26.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:48:26 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:48:26.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:48:26 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T01:48:26.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:48:26 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:48:26.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:48:26 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:48:26.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:48:26 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:48:26.567 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:48:26 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:48:26.567 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:48:26 smithi115 ceph-mon[78593]: pgmap v4572: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:48:26.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:48:26 smithi115 conmon[90393]: 2022-05-14T01:48:26.565+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:48:27.530 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:48:27 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:48:27.531 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T01:48:27.531 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:48:27.532 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T01:48:27.532 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:48:27.532 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:48:27.533 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:48:27.533 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T01:48:27.533 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:48:27.534 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T01:48:27.534 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:48:27.534 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T01:48:27.535 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:48:27.535 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T01:48:27.536 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:48:27.536 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T01:48:27.536 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:48:27.537 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:48:27.537 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:48:27.538 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:48:27 smithi115 ceph-mon[78593]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T01:48:27.538 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:48:27 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:48:27.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:48:27 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:48:27.541 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T01:48:27.542 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:48:27.542 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T01:48:27.542 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:48:27.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:48:27.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:48:27.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T01:48:27.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:48:27.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T01:48:27.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:48:27.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T01:48:27.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:48:27.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T01:48:27.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:48:27.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T01:48:27.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:48:27.548 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:48:27.548 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:48:27.548 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:48:27 smithi047 ceph-mon[77147]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T01:48:27.549 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:48:27 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:48:27.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:48:27 smithi115 conmon[90393]: 2022-05-14T01:48:27.529+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:48:28.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:48:28 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:48:28.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:48:28 smithi047 ceph-mon[77147]: pgmap v4573: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:48:28.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:48:28 smithi115 conmon[90393]: 2022-05-14T01:48:28.497+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:48:28.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:48:28 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:48:28.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:48:28 smithi115 ceph-mon[78593]: pgmap v4573: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:48:29.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:48:29 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:48:29.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:48:29 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:48:29.542 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:48:29 smithi047 conmon[78521]: 2022-05-14T01:48:29.488+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:48:29.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:48:29 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T01:48:29.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:48:29 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:48:29.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:48:29 smithi047 conmon[78521]: yield 2022-05-14T01:48:29.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:48:29 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:48:29.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:48:29 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:48:29.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:48:29 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:48:29.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:48:29 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T01:48:29.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:48:29 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:48:29.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:48:29 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T01:48:29.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:48:29 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:48:29.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:48:29 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T01:48:29.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:48:29 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:48:29.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:48:29 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T01:48:29.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:48:29 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:48:29.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:48:29 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T01:48:29.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:48:29 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:48:29.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:48:29 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:48:29.547 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:48:29 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:48:29.547 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:48:29 smithi047 conmon[78521]: 2022-05-14T01:48:29.489+0000 7f5345f84700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T01:48:29.547 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:48:29 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T01:48:29.547 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:48:29 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:48:29.548 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:48:29 smithi047 conmon[78521]: yield 2022-05-14T01:48:29.548 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:48:29 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:48:29.548 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:48:29 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:48:29.548 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:48:29 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:48:29.548 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:48:29 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T01:48:29.549 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:48:29 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:48:29.549 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:48:29 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T01:48:29.549 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:48:29 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:48:29.549 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:48:29 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T01:48:29.550 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:48:29 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:48:29.550 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:48:29 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T01:48:29.550 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:48:29 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:48:29.550 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:48:29 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T01:48:29.551 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:48:29 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:48:29.551 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:48:29 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:48:29.551 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:48:29 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:48:29.551 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:48:29 smithi047 conmon[78521]: 2022-05-14T01:48:29.551 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:48:29 smithi047 conmon[78521]: During handling of the above exception, another exception occurred: 2022-05-14T01:48:29.552 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:48:29 smithi047 conmon[78521]: 2022-05-14T01:48:29.552 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:48:29 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T01:48:29.552 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:48:29 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T01:48:29.552 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:48:29 smithi047 conmon[78521]: return f(*arg) 2022-05-14T01:48:29.553 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:48:29 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T01:48:29.553 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:48:29 smithi047 conmon[78521]: self._write_client_files(client_files, host) 2022-05-14T01:48:29.553 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:48:29 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T01:48:29.553 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:48:29 smithi047 conmon[78521]: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T01:48:29.554 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:48:29 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T01:48:29.554 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:48:29 smithi047 conmon[78521]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T01:48:29.554 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:48:29 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T01:48:29.554 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:48:29 smithi047 conmon[78521]: return self.event_loop.get_result(coro) 2022-05-14T01:48:29.554 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:48:29 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T01:48:29.555 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:48:29 smithi047 conmon[78521]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T01:48:29.555 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:48:29 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T01:48:29.555 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:48:29 smithi047 conmon[78521]: return self.__get_result() 2022-05-14T01:48:29.555 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:48:29 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T01:48:29.556 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:48:29 smithi047 conmon[78521]: raise self._exception 2022-05-14T01:48:29.556 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:48:29 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T01:48:29.556 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:48:29 smithi047 conmon[78521]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T01:48:29.556 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:48:29 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T01:48:29.556 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:48:29 smithi047 conmon[78521]: conn = await self._remote_connection(host, addr) 2022-05-14T01:48:29.557 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:48:29 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T01:48:29.557 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:48:29 smithi047 conmon[78521]: raise 2022-05-14T01:48:29.557 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:48:29 smithi047 conmon[78521]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T01:48:29.557 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:48:29 smithi047 conmon[78521]: self.gen.throw(type, value, traceback) 2022-05-14T01:48:29.558 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:48:29 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T01:48:29.558 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:48:29 smithi047 conmon[78521]: raise OrchestratorError(msg) 2022-05-14T01:48:29.558 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:48:29 smithi047 conmon[78521]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:48:29.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:48:29 smithi115 conmon[90393]: 2022-05-14T01:48:29.539+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:48:30.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:48:30 smithi115 conmon[90393]: 2022-05-14T01:48:30.510+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:48:30.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:48:30 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:48:30.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:48:30 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:48:30.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:48:30 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:48:30.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T01:48:30.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:48:30.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T01:48:30.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:48:30.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:48:30.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:48:30.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T01:48:30.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:48:30.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T01:48:30.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:48:30.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T01:48:30.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:48:30.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T01:48:30.613 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:48:30.613 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T01:48:30.613 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:48:30.613 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:48:30.613 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:48:30.614 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:48:30 smithi115 ceph-mon[78593]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T01:48:30.614 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T01:48:30.614 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:48:30.614 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T01:48:30.614 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:48:30.615 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:48:30.616 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:48:30.617 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T01:48:30.617 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:48:30.617 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T01:48:30.617 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:48:30.617 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T01:48:30.618 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:48:30.618 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T01:48:30.618 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:48:30.618 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T01:48:30.619 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:48:30.619 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:48:30.619 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:48:30.619 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T01:48:30.620 INFO:journalctl@ceph.mon.c.smithi115.stdout: During handling of the above exception, another exception occurred: 2022-05-14T01:48:30.620 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T01:48:30.620 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T01:48:30.620 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T01:48:30.620 INFO:journalctl@ceph.mon.c.smithi115.stdout: return f(*arg) 2022-05-14T01:48:30.621 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T01:48:30.621 INFO:journalctl@ceph.mon.c.smithi115.stdout: self._write_client_files(client_files, host) 2022-05-14T01:48:30.621 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T01:48:30.621 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T01:48:30.622 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T01:48:30.622 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T01:48:30.622 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T01:48:30.622 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.event_loop.get_result(coro) 2022-05-14T01:48:30.622 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T01:48:30.623 INFO:journalctl@ceph.mon.c.smithi115.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T01:48:30.623 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T01:48:30.623 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.__get_result() 2022-05-14T01:48:30.623 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T01:48:30.623 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise self._exception 2022-05-14T01:48:30.624 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T01:48:30.624 INFO:journalctl@ceph.mon.c.smithi115.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T01:48:30.624 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T01:48:30.624 INFO:journalctl@ceph.mon.c.smithi115.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T01:48:30.625 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T01:48:30.625 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise 2022-05-14T01:48:30.625 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T01:48:30.625 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.gen.throw(type, value, traceback) 2022-05-14T01:48:30.625 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T01:48:30.626 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OrchestratorError(msg) 2022-05-14T01:48:30.626 INFO:journalctl@ceph.mon.c.smithi115.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:48:30.626 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:48:30 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:48:30.626 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:48:30 smithi115 ceph-mon[78593]: pgmap v4574: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:48:30.627 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:48:30 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 8440 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:48:30.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:48:30 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:48:30.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:48:30 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:48:30.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:48:30 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:48:30.792 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T01:48:30.792 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:48:30.793 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T01:48:30.793 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:48:30.793 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:48:30.793 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:48:30.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T01:48:30.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:48:30.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T01:48:30.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:48:30.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T01:48:30.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:48:30.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T01:48:30.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:48:30.797 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T01:48:30.797 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:48:30.797 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:48:30.797 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:48:30.798 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:48:30 smithi047 ceph-mon[77147]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T01:48:30.798 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T01:48:30.798 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:48:30.798 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T01:48:30.798 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:48:30.799 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:48:30.799 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:48:30.799 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T01:48:30.799 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:48:30.800 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T01:48:30.800 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:48:30.800 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T01:48:30.800 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:48:30.801 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T01:48:30.801 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:48:30.801 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T01:48:30.801 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:48:30.801 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:48:30.802 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:48:30.802 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T01:48:30.802 INFO:journalctl@ceph.mon.b.smithi047.stdout: During handling of the above exception, another exception occurred: 2022-05-14T01:48:30.802 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T01:48:30.803 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T01:48:30.803 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T01:48:30.803 INFO:journalctl@ceph.mon.b.smithi047.stdout: return f(*arg) 2022-05-14T01:48:30.803 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T01:48:30.804 INFO:journalctl@ceph.mon.b.smithi047.stdout: self._write_client_files(client_files, host) 2022-05-14T01:48:30.804 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T01:48:30.804 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T01:48:30.804 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T01:48:30.805 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T01:48:30.805 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T01:48:30.805 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.event_loop.get_result(coro) 2022-05-14T01:48:30.805 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T01:48:30.806 INFO:journalctl@ceph.mon.b.smithi047.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T01:48:30.806 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T01:48:30.806 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.__get_result() 2022-05-14T01:48:30.806 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T01:48:30.807 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise self._exception 2022-05-14T01:48:30.807 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T01:48:30.807 INFO:journalctl@ceph.mon.b.smithi047.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T01:48:30.807 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T01:48:30.808 INFO:journalctl@ceph.mon.b.smithi047.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T01:48:30.808 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T01:48:30.808 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise 2022-05-14T01:48:30.808 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T01:48:30.809 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.gen.throw(type, value, traceback) 2022-05-14T01:48:30.809 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T01:48:30.809 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OrchestratorError(msg) 2022-05-14T01:48:30.809 INFO:journalctl@ceph.mon.b.smithi047.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:48:30.810 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:48:30 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:48:30.810 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:48:30 smithi047 ceph-mon[77147]: pgmap v4574: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:48:30.810 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:48:30 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 8440 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:48:31.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:48:31 smithi115 conmon[90393]: 2022-05-14T01:48:31.479+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:48:31.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:48:31 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:48:31.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:48:31 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:48:32.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:48:32 smithi115 conmon[90393]: 2022-05-14T01:48:32.432+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:48:32.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:48:32 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:48:32.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:48:32 smithi115 ceph-mon[78593]: pgmap v4575: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:48:32.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:48:32 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:48:32.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:48:32 smithi047 ceph-mon[77147]: pgmap v4575: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:48:33.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:48:33 smithi115 conmon[90393]: 2022-05-14T01:48:33.429+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:48:33.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:48:33 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:48:33.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:48:33 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:48:34.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:48:34 smithi115 conmon[90393]: 2022-05-14T01:48:34.462+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:48:34.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:48:34 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:48:34.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:48:34 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:48:34.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:48:34 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:48:34.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:48:34 smithi115 ceph-mon[78593]: pgmap v4576: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:48:34.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:48:34 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:48:34.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:48:34 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:48:34.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:48:34 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:48:34.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:48:34 smithi047 ceph-mon[77147]: pgmap v4576: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:48:35.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:48:35 smithi115 conmon[90393]: 2022-05-14T01:48:35.511+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:48:35.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:48:35 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:48:35.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:48:35 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 8445 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:48:35.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:48:35 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:48:35.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:48:35 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 8445 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:48:36.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:48:36 smithi115 conmon[90393]: 2022-05-14T01:48:36.525+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:48:36.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:48:36 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:48:36.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:48:36 smithi115 ceph-mon[78593]: pgmap v4577: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:48:36.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:48:36 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:48:36.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:48:36 smithi047 ceph-mon[77147]: pgmap v4577: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:48:37.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:48:37 smithi115 conmon[90393]: 2022-05-14T01:48:37.503+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:48:37.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:48:37 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:48:37.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:48:37 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:48:38.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:48:38 smithi115 conmon[90393]: 2022-05-14T01:48:38.461+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:48:38.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:48:38 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:48:38.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:48:38 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:48:38.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:48:38 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:48:38.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:48:38 smithi115 ceph-mon[78593]: pgmap v4578: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:48:38.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:48:38 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:48:38.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:48:38 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:48:38.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:48:38 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:48:38.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:48:38 smithi047 ceph-mon[77147]: pgmap v4578: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:48:39.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:48:39 smithi115 conmon[90393]: 2022-05-14T01:48:39.418+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:48:39.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:48:39 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:48:39.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:48:39 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:48:40.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:48:40 smithi115 conmon[90393]: 2022-05-14T01:48:40.459+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:48:40.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:48:40 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:48:40.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:48:40 smithi115 ceph-mon[78593]: pgmap v4579: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:48:40.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:48:40 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 8450 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:48:40.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:48:40 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:48:40.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:48:40 smithi047 ceph-mon[77147]: pgmap v4579: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:48:40.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:48:40 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 8450 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:48:41.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:48:41 smithi115 conmon[90393]: 2022-05-14T01:48:41.470+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:48:41.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:48:41 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:48:41.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:48:41 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:48:42.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:48:42 smithi115 conmon[90393]: 2022-05-14T01:48:42.487+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:48:42.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:48:42 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:48:42.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:48:42 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:48:42.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:48:42 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:48:42.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:48:42 smithi115 ceph-mon[78593]: pgmap v4580: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:48:42.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:48:42 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:48:42.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:48:42 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:48:42.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:48:42 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:48:42.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:48:42 smithi047 ceph-mon[77147]: pgmap v4580: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:48:43.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:48:43 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:48:43.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:48:43 smithi115 conmon[90393]: 2022-05-14T01:48:43.466+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:48:43.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:48:43 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:48:44.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:48:44 smithi115 conmon[90393]: 2022-05-14T01:48:44.438+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:48:44.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:48:44 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:48:44.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:48:44 smithi115 ceph-mon[78593]: pgmap v4581: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:48:44.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:48:44 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:48:44.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:48:44 smithi047 ceph-mon[77147]: pgmap v4581: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:48:45.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:48:45 smithi115 conmon[90393]: 2022-05-14T01:48:45.454+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:48:45.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:48:45 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:48:45.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:48:45 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 8455 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:48:45.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:48:45 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:48:45.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:48:45 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 8455 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:48:46.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:48:46 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:48:46.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:48:46 smithi047 ceph-mon[77147]: pgmap v4582: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:48:46.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:48:46 smithi115 conmon[90393]: 2022-05-14T01:48:46.463+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:48:46.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:48:46 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:48:46.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:48:46 smithi115 ceph-mon[78593]: pgmap v4582: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:48:47.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:48:47 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:48:47.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:48:47 smithi115 conmon[90393]: 2022-05-14T01:48:47.427+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:48:47.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:48:47 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:48:48.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:48:48 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:48:48.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:48:48 smithi047 ceph-mon[77147]: pgmap v4583: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:48:48.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:48:48 smithi115 conmon[90393]: 2022-05-14T01:48:48.384+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:48:48.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:48:48 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:48:48.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:48:48 smithi115 ceph-mon[78593]: pgmap v4583: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:48:49.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:48:49 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:48:49.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:48:49 smithi115 conmon[90393]: 2022-05-14T01:48:49.390+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:48:49.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:48:49 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:48:50.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:48:50 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:48:50.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:48:50 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:48:50.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:48:50 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:48:50.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:48:50 smithi047 ceph-mon[77147]: pgmap v4584: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:48:50.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:48:50 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 8460 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:48:50.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:48:50 smithi115 conmon[90393]: 2022-05-14T01:48:50.423+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:48:50.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:48:50 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:48:50.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:48:50 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:48:50.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:48:50 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:48:50.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:48:50 smithi115 ceph-mon[78593]: pgmap v4584: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:48:50.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:48:50 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 8460 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:48:51.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:48:51 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:48:51.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:48:51 smithi115 conmon[90393]: 2022-05-14T01:48:51.399+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:48:51.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:48:51 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:48:52.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:48:52 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:48:52.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:48:52 smithi047 ceph-mon[77147]: pgmap v4585: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:48:52.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:48:52 smithi115 conmon[90393]: 2022-05-14T01:48:52.387+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:48:52.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:48:52 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:48:52.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:48:52 smithi115 ceph-mon[78593]: pgmap v4585: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:48:53.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:48:53 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:48:53.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:48:53 smithi115 conmon[90393]: 2022-05-14T01:48:53.385+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:48:53.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:48:53 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:48:54.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:48:54 smithi115 conmon[90393]: 2022-05-14T01:48:54.353+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:48:54.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:48:54 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:48:54.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:48:54 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:48:54.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:48:54 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:48:54.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:48:54 smithi115 ceph-mon[78593]: pgmap v4586: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:48:54.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:48:54 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:48:54.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:48:54 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:48:54.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:48:54 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:48:54.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:48:54 smithi047 ceph-mon[77147]: pgmap v4586: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:48:55.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:48:55 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:48:55.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:48:55 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 8465 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:48:55.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:48:55 smithi115 conmon[90393]: 2022-05-14T01:48:55.391+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:48:55.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:48:55 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:48:55.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:48:55 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 8465 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:48:56.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:48:56 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:48:56.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:48:56 smithi047 ceph-mon[77147]: pgmap v4587: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:48:56.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:48:56 smithi115 conmon[90393]: 2022-05-14T01:48:56.373+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:48:56.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:48:56 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:48:56.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:48:56 smithi115 ceph-mon[78593]: pgmap v4587: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:48:57.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:48:57 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:48:57.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:48:57 smithi115 conmon[90393]: 2022-05-14T01:48:57.408+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:48:57.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:48:57 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:48:58.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:48:58 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:48:58.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:48:58 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:48:58.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:48:58 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:48:58.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:48:58 smithi047 ceph-mon[77147]: pgmap v4588: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:48:58.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:48:58 smithi115 conmon[90393]: 2022-05-14T01:48:58.387+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:48:58.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:48:58 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:48:58.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:48:58 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:48:58.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:48:58 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:48:58.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:48:58 smithi115 ceph-mon[78593]: pgmap v4588: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:48:59.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:48:59 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:48:59.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:48:59 smithi115 conmon[90393]: 2022-05-14T01:48:59.383+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:48:59.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:48:59 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:49:00.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:49:00 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:49:00.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:49:00 smithi047 ceph-mon[77147]: pgmap v4589: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:49:00.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:49:00 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 8470 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:49:00.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:49:00 smithi115 conmon[90393]: 2022-05-14T01:49:00.417+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:49:00.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:49:00 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:49:00.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:49:00 smithi115 ceph-mon[78593]: pgmap v4589: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:49:00.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:49:00 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 8470 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:49:01.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:49:01 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:49:01.867 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:49:01 smithi115 conmon[90393]: 2022-05-14T01:49:01.438+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:49:01.867 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:49:01 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:49:02.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:49:02 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:49:02.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:49:02 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:49:02.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:49:02 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:49:02.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:49:02 smithi047 ceph-mon[77147]: pgmap v4590: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:49:02.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:49:02 smithi115 conmon[90393]: 2022-05-14T01:49:02.390+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:49:02.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:49:02 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:49:02.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:49:02 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:49:02.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:49:02 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:49:02.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:49:02 smithi115 ceph-mon[78593]: pgmap v4590: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:49:03.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:49:03 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:49:03.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:49:03 smithi115 conmon[90393]: 2022-05-14T01:49:03.350+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:49:03.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:49:03 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:49:04.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:49:04 smithi115 conmon[90393]: 2022-05-14T01:49:04.311+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:49:04.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:49:04 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:49:04.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:49:04 smithi115 ceph-mon[78593]: pgmap v4591: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:49:04.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:49:04 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:49:04.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:49:04 smithi047 ceph-mon[77147]: pgmap v4591: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:49:05.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:49:05 smithi115 conmon[90393]: 2022-05-14T01:49:05.285+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:49:05.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:49:05 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:49:05.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:49:05 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 8475 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:49:05.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:49:05 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:49:05.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:49:05 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 8475 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:49:06.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:49:06 smithi115 conmon[90393]: 2022-05-14T01:49:06.280+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:49:06.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:49:06 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:49:06.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:49:06 smithi115 ceph-mon[78593]: pgmap v4592: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:49:06.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:49:06 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:49:06.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:49:06 smithi047 ceph-mon[77147]: pgmap v4592: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:49:07.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:49:07 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:49:07.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:49:07 smithi115 conmon[90393]: 2022-05-14T01:49:07.319+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:49:07.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:49:07 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:49:08.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:49:08 smithi115 conmon[90393]: 2022-05-14T01:49:08.336+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:49:08.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:49:08 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:49:08.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:49:08 smithi115 ceph-mon[78593]: pgmap v4593: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:49:08.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:49:08 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:49:08.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:49:08 smithi047 ceph-mon[77147]: pgmap v4593: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:49:09.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:49:09 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:49:09.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:49:09 smithi115 conmon[90393]: 2022-05-14T01:49:09.349+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:49:09.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:49:09 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:49:10.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:49:10 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:49:10.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:49:10 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:49:10.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:49:10 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:49:10.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:49:10 smithi047 ceph-mon[77147]: pgmap v4594: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:49:10.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:49:10 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 8480 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:49:10.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:49:10 smithi115 conmon[90393]: 2022-05-14T01:49:10.378+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:49:10.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:49:10 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:49:10.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:49:10 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:49:10.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:49:10 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:49:10.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:49:10 smithi115 ceph-mon[78593]: pgmap v4594: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:49:10.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:49:10 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 8480 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:49:11.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:49:11 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:49:11.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:49:11 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:49:11.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:49:11 smithi115 conmon[90393]: 2022-05-14T01:49:11.399+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:49:12.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:49:12 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:49:12.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:49:12 smithi047 ceph-mon[77147]: pgmap v4595: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:49:12.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:49:12 smithi115 conmon[90393]: 2022-05-14T01:49:12.421+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:49:12.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:49:12 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:49:12.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:49:12 smithi115 ceph-mon[78593]: pgmap v4595: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:49:13.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:49:13 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:49:13.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:49:13 smithi115 conmon[90393]: 2022-05-14T01:49:13.417+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:49:13.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:49:13 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:49:14.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:49:14 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:49:14.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:49:14 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:49:14.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:49:14 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:49:14.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:49:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T01:49:14.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:49:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T01:49:14.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:49:14 smithi047 ceph-mon[77147]: pgmap v4596: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:49:14.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:49:14 smithi115 conmon[90393]: 2022-05-14T01:49:14.375+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:49:14.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:49:14 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:49:14.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:49:14 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:49:14.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:49:14 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:49:14.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:49:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T01:49:14.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:49:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T01:49:14.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:49:14 smithi115 ceph-mon[78593]: pgmap v4596: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:49:15.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:49:15 smithi115 conmon[90393]: 2022-05-14T01:49:15.335+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:49:15.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:49:15 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:49:15.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:49:15 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 8485 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:49:15.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:49:15 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:49:15.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:49:15 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 8485 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:49:16.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:49:16 smithi115 conmon[90393]: 2022-05-14T01:49:16.356+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:49:16.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:49:16 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:49:16.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:49:16 smithi115 ceph-mon[78593]: pgmap v4597: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:49:16.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:49:16 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:49:16.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:49:16 smithi047 ceph-mon[77147]: pgmap v4597: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:49:17.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:49:17 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:49:17.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:49:17 smithi115 conmon[90393]: 2022-05-14T01:49:17.369+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:49:17.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:49:17 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:49:18.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:49:18 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:49:18.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:49:18 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:49:18.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:49:18 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:49:18.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:49:18 smithi047 ceph-mon[77147]: pgmap v4598: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:49:18.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:49:18 smithi115 conmon[90393]: 2022-05-14T01:49:18.373+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:49:18.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:49:18 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:49:18.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:49:18 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:49:18.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:49:18 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:49:18.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:49:18 smithi115 ceph-mon[78593]: pgmap v4598: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:49:19.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:49:19 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:49:19.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:49:19 smithi115 conmon[90393]: 2022-05-14T01:49:19.393+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:49:19.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:49:19 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:49:20.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:49:20 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:49:20.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:49:20 smithi047 ceph-mon[77147]: pgmap v4599: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:49:20.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:49:20 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 8490 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:49:20.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:49:20 smithi115 conmon[90393]: 2022-05-14T01:49:20.365+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:49:20.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:49:20 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:49:20.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:49:20 smithi115 ceph-mon[78593]: pgmap v4599: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:49:20.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:49:20 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 8490 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:49:21.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:49:21 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:49:21.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:49:21 smithi115 conmon[90393]: 2022-05-14T01:49:21.394+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:49:21.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:49:21 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:49:22.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:49:22 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:49:22.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:49:22 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:49:22.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:49:22 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:49:22.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:49:22 smithi047 ceph-mon[77147]: pgmap v4600: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:49:22.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:49:22 smithi115 conmon[90393]: 2022-05-14T01:49:22.373+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:49:22.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:49:22 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:49:22.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:49:22 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:49:22.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:49:22 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:49:22.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:49:22 smithi115 ceph-mon[78593]: pgmap v4600: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:49:23.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:49:23 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:49:23.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:49:23 smithi115 conmon[90393]: 2022-05-14T01:49:23.407+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:49:23.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:49:23 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:49:24.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:49:24 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:49:24.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:49:24 smithi047 ceph-mon[77147]: pgmap v4601: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:49:24.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:49:24 smithi115 conmon[90393]: 2022-05-14T01:49:24.446+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:49:24.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:49:24 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:49:24.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:49:24 smithi115 ceph-mon[78593]: pgmap v4601: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:49:25.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:49:25 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:49:25.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:49:25 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 8495 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:49:25.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:49:25 smithi115 conmon[90393]: 2022-05-14T01:49:25.434+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:49:25.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:49:25 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:49:25.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:49:25 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 8495 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:49:26.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:49:26 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:49:26.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:49:26 smithi047 ceph-mon[77147]: pgmap v4602: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:49:26.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:49:26 smithi115 conmon[90393]: 2022-05-14T01:49:26.472+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:49:26.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:49:26 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:49:26.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:49:26 smithi115 ceph-mon[78593]: pgmap v4602: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:49:27.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:49:27 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:49:27.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:49:27 smithi115 conmon[90393]: 2022-05-14T01:49:27.508+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:49:27.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:49:27 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:49:28.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:49:28 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:49:28.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:49:28 smithi047 ceph-mon[77147]: pgmap v4603: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:49:28.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:49:28 smithi115 conmon[90393]: 2022-05-14T01:49:28.550+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:49:28.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:49:28 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:49:28.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:49:28 smithi115 ceph-mon[78593]: pgmap v4603: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:49:29.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:49:29 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:49:29.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:49:29 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T01:49:29.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:49:29 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T01:49:29.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:49:29 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T01:49:29.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:49:29 smithi115 conmon[90393]: 2022-05-14T01:49:29.556+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:49:29.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:49:29 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:49:29.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:49:29 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T01:49:29.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:49:29 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T01:49:29.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:49:29 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T01:49:30.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:49:30 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:49:30.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:49:30 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:49:30.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:49:30 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:49:30.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:49:30 smithi047 ceph-mon[77147]: pgmap v4604: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:49:30.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:49:30 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' 2022-05-14T01:49:30.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:49:30 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' 2022-05-14T01:49:30.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:49:30 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 8500 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:49:30.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:49:30 smithi115 conmon[90393]: 2022-05-14T01:49:30.594+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:49:30.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:49:30 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:49:30.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:49:30 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:49:30.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:49:30 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:49:30.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:49:30 smithi115 ceph-mon[78593]: pgmap v4604: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:49:30.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:49:30 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' 2022-05-14T01:49:30.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:49:30 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' 2022-05-14T01:49:30.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:49:30 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 8500 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:49:31.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:49:31 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:49:31.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:49:31 smithi115 conmon[90393]: 2022-05-14T01:49:31.607+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:49:32.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:49:31 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:49:32.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:49:32 smithi115 conmon[90393]: 2022-05-14T01:49:32.583+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:49:32.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:49:32 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:49:32.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:49:32 smithi115 ceph-mon[78593]: pgmap v4605: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:49:33.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:49:32 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:49:33.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:49:32 smithi047 ceph-mon[77147]: pgmap v4605: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:49:33.042 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:49:32 smithi047 conmon[78521]: 2022-05-14T01:49:32.591+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:49:33.042 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:49:32 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T01:49:33.042 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:49:32 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:49:33.042 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:49:32 smithi047 conmon[78521]: yield 2022-05-14T01:49:33.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:49:32 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:49:33.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:49:32 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:49:33.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:49:32 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:49:33.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:49:32 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T01:49:33.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:49:32 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:49:33.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:49:32 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T01:49:33.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:49:32 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:49:33.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:49:32 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T01:49:33.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:49:32 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:49:33.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:49:32 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T01:49:33.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:49:32 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:49:33.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:49:32 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T01:49:33.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:49:32 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:49:33.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:49:32 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:49:33.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:49:32 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:49:33.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:49:33 smithi115 conmon[90393]: 2022-05-14T01:49:33.569+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:49:33.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:49:33 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:49:33.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:49:33 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:49:33.859 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T01:49:33.859 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:49:33.859 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T01:49:33.859 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:49:33.859 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:49:33.860 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:49:33.860 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T01:49:33.860 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:49:33.860 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T01:49:33.860 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:49:33.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T01:49:33.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:49:33.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T01:49:33.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:49:33.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T01:49:33.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:49:33.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:49:33.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:49:33.862 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:49:33 smithi115 ceph-mon[78593]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T01:49:33.863 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:49:33 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' 2022-05-14T01:49:33.863 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:49:33 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' 2022-05-14T01:49:33.863 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:49:33 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-05-14T01:49:33.863 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:49:33 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-05-14T01:49:33.863 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:49:33 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-05-14T01:49:33.864 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:49:33 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-05-14T01:49:33.864 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:49:33 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-05-14T01:49:33.864 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:49:33 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-05-14T01:49:33.864 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:49:33 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-05-14T01:49:33.865 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:49:33 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-05-14T01:49:34.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:49:33 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:49:34.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:49:33 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:49:34.042 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T01:49:34.042 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:49:34.042 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T01:49:34.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:49:34.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:49:34.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:49:34.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T01:49:34.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:49:34.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T01:49:34.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:49:34.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T01:49:34.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:49:34.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T01:49:34.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:49:34.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T01:49:34.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:49:34.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:49:34.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:49:34.046 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:49:33 smithi047 ceph-mon[77147]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T01:49:34.046 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:49:33 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' 2022-05-14T01:49:34.046 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:49:33 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' 2022-05-14T01:49:34.047 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:49:33 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-05-14T01:49:34.047 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:49:33 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-05-14T01:49:34.047 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:49:33 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-05-14T01:49:34.047 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:49:33 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-05-14T01:49:34.047 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:49:33 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-05-14T01:49:34.048 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:49:33 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-05-14T01:49:34.048 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:49:33 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-05-14T01:49:34.048 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:49:33 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-05-14T01:49:34.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:49:34 smithi115 conmon[90393]: 2022-05-14T01:49:34.538+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:49:34.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:49:34 smithi115 ceph-mon[78593]: Adjusting osd_memory_target on smithi115 to 197.0M 2022-05-14T01:49:34.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:49:34 smithi115 ceph-mon[78593]: Unable to set osd_memory_target on smithi115 to 206629785: error parsing value: Value '206629785' is below minimum 939524096 2022-05-14T01:49:34.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:49:34 smithi115 ceph-mon[78593]: Adjusting osd_memory_target on smithi047 to 198.1M 2022-05-14T01:49:34.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:49:34 smithi115 ceph-mon[78593]: Unable to set osd_memory_target on smithi047 to 207755878: error parsing value: Value '207755878' is below minimum 939524096 2022-05-14T01:49:34.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:49:34 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:49:34.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:49:34 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:49:34.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:49:34 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:49:34.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:49:34 smithi115 ceph-mon[78593]: pgmap v4606: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:49:35.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:49:34 smithi047 ceph-mon[77147]: Adjusting osd_memory_target on smithi115 to 197.0M 2022-05-14T01:49:35.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:49:34 smithi047 ceph-mon[77147]: Unable to set osd_memory_target on smithi115 to 206629785: error parsing value: Value '206629785' is below minimum 939524096 2022-05-14T01:49:35.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:49:34 smithi047 ceph-mon[77147]: Adjusting osd_memory_target on smithi047 to 198.1M 2022-05-14T01:49:35.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:49:34 smithi047 ceph-mon[77147]: Unable to set osd_memory_target on smithi047 to 207755878: error parsing value: Value '207755878' is below minimum 939524096 2022-05-14T01:49:35.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:49:34 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:49:35.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:49:34 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:49:35.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:49:34 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:49:35.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:49:34 smithi047 ceph-mon[77147]: pgmap v4606: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:49:35.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:49:35 smithi115 conmon[90393]: 2022-05-14T01:49:35.584+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:49:35.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:49:35 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:49:35.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:49:35 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 8505 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:49:36.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:49:35 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:49:36.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:49:35 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 8505 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:49:36.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:49:35 smithi047 conmon[78521]: 2022-05-14T01:49:35.663+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:49:36.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:49:35 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T01:49:36.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:49:35 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:49:36.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:49:35 smithi047 conmon[78521]: yield 2022-05-14T01:49:36.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:49:35 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:49:36.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:49:35 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:49:36.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:49:35 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:49:36.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:49:35 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T01:49:36.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:49:35 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:49:36.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:49:35 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T01:49:36.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:49:35 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:49:36.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:49:35 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T01:49:36.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:49:35 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:49:36.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:49:35 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T01:49:36.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:49:35 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:49:36.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:49:35 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T01:49:36.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:49:35 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:49:36.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:49:35 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:49:36.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:49:35 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:49:36.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:49:35 smithi047 conmon[78521]: 2022-05-14T01:49:35.664+0000 7f5340779700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T01:49:36.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:49:35 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T01:49:36.048 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:49:35 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:49:36.048 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:49:35 smithi047 conmon[78521]: yield 2022-05-14T01:49:36.048 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:49:35 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:49:36.048 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:49:35 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:49:36.048 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:49:35 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:49:36.049 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:49:35 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T01:49:36.049 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:49:35 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:49:36.049 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:49:35 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T01:49:36.049 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:49:35 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:49:36.050 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:49:35 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T01:49:36.050 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:49:35 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:49:36.050 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:49:35 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T01:49:36.050 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:49:35 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:49:36.050 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:49:35 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T01:49:36.051 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:49:35 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:49:36.051 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:49:35 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:49:36.051 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:49:35 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:49:36.052 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:49:35 smithi047 conmon[78521]: 2022-05-14T01:49:36.052 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:49:35 smithi047 conmon[78521]: During handling of the above exception, another exception occurred: 2022-05-14T01:49:36.052 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:49:35 smithi047 conmon[78521]: 2022-05-14T01:49:36.053 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:49:35 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T01:49:36.053 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:49:35 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T01:49:36.053 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:49:35 smithi047 conmon[78521]: return f(*arg) 2022-05-14T01:49:36.053 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:49:35 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T01:49:36.053 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:49:35 smithi047 conmon[78521]: self._write_client_files(client_files, host) 2022-05-14T01:49:36.054 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:49:35 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T01:49:36.054 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:49:35 smithi047 conmon[78521]: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T01:49:36.055 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:49:35 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T01:49:36.055 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:49:35 smithi047 conmon[78521]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T01:49:36.055 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:49:35 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T01:49:36.056 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:49:35 smithi047 conmon[78521]: return self.event_loop.get_result(coro) 2022-05-14T01:49:36.056 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:49:35 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T01:49:36.057 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:49:35 smithi047 conmon[78521]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T01:49:36.057 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:49:35 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T01:49:36.057 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:49:35 smithi047 conmon[78521]: return self.__get_result() 2022-05-14T01:49:36.058 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:49:35 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T01:49:36.058 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:49:35 smithi047 conmon[78521]: raise self._exception 2022-05-14T01:49:36.058 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:49:35 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T01:49:36.059 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:49:35 smithi047 conmon[78521]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T01:49:36.059 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:49:35 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T01:49:36.060 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:49:35 smithi047 conmon[78521]: conn = await self._remote_connection(host, addr) 2022-05-14T01:49:36.060 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:49:35 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T01:49:36.060 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:49:35 smithi047 conmon[78521]: raise 2022-05-14T01:49:36.061 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:49:35 smithi047 conmon[78521]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T01:49:36.061 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:49:35 smithi047 conmon[78521]: self.gen.throw(type, value, traceback) 2022-05-14T01:49:36.062 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:49:35 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T01:49:36.062 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:49:35 smithi047 conmon[78521]: raise OrchestratorError(msg) 2022-05-14T01:49:36.062 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:49:35 smithi047 conmon[78521]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:49:36.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:49:36 smithi115 conmon[90393]: 2022-05-14T01:49:36.557+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:49:36.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:49:36 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:49:36.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:49:36 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:49:36.859 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T01:49:36.860 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:49:36.860 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T01:49:36.860 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:49:36.860 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:49:36.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:49:36.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T01:49:36.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:49:36.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T01:49:36.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:49:36.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T01:49:36.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:49:36.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T01:49:36.863 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:49:36.863 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T01:49:36.863 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:49:36.863 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:49:36.864 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:49:36.864 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:49:36 smithi115 ceph-mon[78593]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T01:49:36.864 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T01:49:36.864 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:49:36.865 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T01:49:36.865 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:49:36.865 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:49:36.865 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:49:36.866 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T01:49:36.866 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:49:36.866 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T01:49:36.866 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:49:36.866 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T01:49:36.867 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:49:36.867 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T01:49:36.867 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:49:36.867 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T01:49:36.868 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:49:36.868 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:49:36.868 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:49:36.868 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T01:49:36.869 INFO:journalctl@ceph.mon.c.smithi115.stdout: During handling of the above exception, another exception occurred: 2022-05-14T01:49:36.869 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T01:49:36.869 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T01:49:36.869 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T01:49:36.870 INFO:journalctl@ceph.mon.c.smithi115.stdout: return f(*arg) 2022-05-14T01:49:36.870 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T01:49:36.870 INFO:journalctl@ceph.mon.c.smithi115.stdout: self._write_client_files(client_files, host) 2022-05-14T01:49:36.870 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T01:49:36.871 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T01:49:36.871 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T01:49:36.871 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T01:49:36.871 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T01:49:36.872 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.event_loop.get_result(coro) 2022-05-14T01:49:36.872 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T01:49:36.872 INFO:journalctl@ceph.mon.c.smithi115.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T01:49:36.872 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T01:49:36.873 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.__get_result() 2022-05-14T01:49:36.873 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T01:49:36.873 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise self._exception 2022-05-14T01:49:36.873 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T01:49:36.874 INFO:journalctl@ceph.mon.c.smithi115.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T01:49:36.874 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T01:49:36.874 INFO:journalctl@ceph.mon.c.smithi115.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T01:49:36.874 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T01:49:36.874 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise 2022-05-14T01:49:36.875 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T01:49:36.875 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.gen.throw(type, value, traceback) 2022-05-14T01:49:36.875 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T01:49:36.875 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OrchestratorError(msg) 2022-05-14T01:49:36.876 INFO:journalctl@ceph.mon.c.smithi115.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:49:36.876 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:49:36 smithi115 ceph-mon[78593]: pgmap v4607: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:49:37.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:49:36 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:49:37.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:49:36 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:49:37.042 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T01:49:37.042 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:49:37.042 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T01:49:37.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:49:37.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:49:37.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:49:37.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T01:49:37.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:49:37.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T01:49:37.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:49:37.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T01:49:37.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:49:37.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T01:49:37.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:49:37.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T01:49:37.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:49:37.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:49:37.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:49:37.046 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:49:36 smithi047 ceph-mon[77147]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T01:49:37.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T01:49:37.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:49:37.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T01:49:37.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:49:37.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:49:37.049 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:49:37.049 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T01:49:37.049 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:49:37.049 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T01:49:37.049 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:49:37.050 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T01:49:37.050 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:49:37.050 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T01:49:37.050 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:49:37.051 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T01:49:37.051 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:49:37.051 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:49:37.051 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:49:37.051 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T01:49:37.052 INFO:journalctl@ceph.mon.b.smithi047.stdout: During handling of the above exception, another exception occurred: 2022-05-14T01:49:37.052 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T01:49:37.052 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T01:49:37.052 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T01:49:37.053 INFO:journalctl@ceph.mon.b.smithi047.stdout: return f(*arg) 2022-05-14T01:49:37.053 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T01:49:37.053 INFO:journalctl@ceph.mon.b.smithi047.stdout: self._write_client_files(client_files, host) 2022-05-14T01:49:37.053 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T01:49:37.054 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T01:49:37.054 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T01:49:37.054 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T01:49:37.055 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T01:49:37.055 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.event_loop.get_result(coro) 2022-05-14T01:49:37.055 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T01:49:37.055 INFO:journalctl@ceph.mon.b.smithi047.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T01:49:37.056 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T01:49:37.056 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.__get_result() 2022-05-14T01:49:37.056 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T01:49:37.056 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise self._exception 2022-05-14T01:49:37.057 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T01:49:37.057 INFO:journalctl@ceph.mon.b.smithi047.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T01:49:37.057 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T01:49:37.057 INFO:journalctl@ceph.mon.b.smithi047.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T01:49:37.057 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T01:49:37.058 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise 2022-05-14T01:49:37.058 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T01:49:37.058 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.gen.throw(type, value, traceback) 2022-05-14T01:49:37.058 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T01:49:37.059 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OrchestratorError(msg) 2022-05-14T01:49:37.059 INFO:journalctl@ceph.mon.b.smithi047.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:49:37.059 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:49:36 smithi047 ceph-mon[77147]: pgmap v4607: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:49:37.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:49:37 smithi115 conmon[90393]: 2022-05-14T01:49:37.580+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:49:37.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:49:37 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:49:38.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:49:37 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:49:38.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:49:38 smithi115 conmon[90393]: 2022-05-14T01:49:38.568+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:49:38.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:49:38 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:49:38.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:49:38 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:49:38.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:49:38 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:49:38.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:49:38 smithi115 ceph-mon[78593]: pgmap v4608: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:49:39.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:49:38 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:49:39.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:49:38 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:49:39.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:49:38 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:49:39.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:49:38 smithi047 ceph-mon[77147]: pgmap v4608: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:49:39.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:49:39 smithi115 conmon[90393]: 2022-05-14T01:49:39.523+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:49:39.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:49:39 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:49:40.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:49:39 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:49:40.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:49:40 smithi115 conmon[90393]: 2022-05-14T01:49:40.562+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:49:40.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:49:40 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:49:40.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:49:40 smithi115 ceph-mon[78593]: pgmap v4609: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:49:40.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:49:40 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 8510 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:49:41.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:49:40 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:49:41.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:49:40 smithi047 ceph-mon[77147]: pgmap v4609: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:49:41.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:49:40 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 8510 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:49:41.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:49:41 smithi115 conmon[90393]: 2022-05-14T01:49:41.590+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:49:41.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:49:41 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:49:41.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:49:41 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:49:42.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:49:41 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:49:42.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:49:41 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:49:42.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:49:42 smithi115 conmon[90393]: 2022-05-14T01:49:42.586+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:49:42.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:49:42 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:49:42.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:49:42 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:49:42.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:49:42 smithi115 ceph-mon[78593]: pgmap v4610: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:49:42.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:49:42 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:49:43.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:49:42 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:49:43.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:49:42 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:49:43.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:49:42 smithi047 ceph-mon[77147]: pgmap v4610: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:49:43.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:49:42 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:49:43.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:49:43 smithi115 conmon[90393]: 2022-05-14T01:49:43.597+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:49:43.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:49:43 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:49:44.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:49:43 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:49:44.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:49:44 smithi115 conmon[90393]: 2022-05-14T01:49:44.589+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:49:44.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:49:44 smithi115 ceph-mon[78593]: pgmap v4611: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:49:44.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:49:44 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:49:45.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:49:44 smithi047 ceph-mon[77147]: pgmap v4611: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:49:45.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:49:44 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:49:45.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:49:45 smithi115 conmon[90393]: 2022-05-14T01:49:45.547+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:49:45.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:49:45 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 8515 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:49:45.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:49:45 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:49:46.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:49:45 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 8515 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:49:46.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:49:45 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:49:46.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:49:46 smithi115 conmon[90393]: 2022-05-14T01:49:46.505+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:49:46.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:49:46 smithi115 ceph-mon[78593]: pgmap v4612: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:49:46.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:49:46 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:49:47.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:49:46 smithi047 ceph-mon[77147]: pgmap v4612: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:49:47.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:49:46 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:49:47.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:49:47 smithi115 conmon[90393]: 2022-05-14T01:49:47.541+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:49:47.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:49:47 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:49:48.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:49:47 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:49:48.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:49:48 smithi115 conmon[90393]: 2022-05-14T01:49:48.513+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:49:48.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:49:48 smithi115 ceph-mon[78593]: pgmap v4613: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:49:48.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:49:48 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:49:49.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:49:48 smithi047 ceph-mon[77147]: pgmap v4613: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:49:49.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:49:48 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:49:49.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:49:49 smithi115 conmon[90393]: 2022-05-14T01:49:49.551+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:49:49.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:49:49 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:49:50.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:49:49 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:49:50.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:49:50 smithi115 conmon[90393]: 2022-05-14T01:49:50.536+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:49:50.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:49:50 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:49:50.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:49:50 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:49:50.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:49:50 smithi115 ceph-mon[78593]: pgmap v4614: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:49:50.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:49:50 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 8520 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:49:50.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:49:50 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:49:51.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:49:50 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:49:51.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:49:50 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:49:51.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:49:50 smithi047 ceph-mon[77147]: pgmap v4614: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:49:51.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:49:50 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 8520 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:49:51.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:49:50 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:49:51.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:49:51 smithi115 conmon[90393]: 2022-05-14T01:49:51.526+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:49:51.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:49:51 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:49:52.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:49:51 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:49:52.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:49:52 smithi115 conmon[90393]: 2022-05-14T01:49:52.551+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:49:52.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:49:52 smithi115 ceph-mon[78593]: pgmap v4615: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:49:52.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:49:52 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:49:53.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:49:52 smithi047 ceph-mon[77147]: pgmap v4615: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:49:53.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:49:52 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:49:53.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:49:53 smithi115 conmon[90393]: 2022-05-14T01:49:53.582+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:49:53.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:49:53 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:49:54.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:49:53 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:49:55.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:49:54 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:49:55.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:49:54 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:49:55.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:49:54 smithi047 ceph-mon[77147]: pgmap v4616: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:49:55.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:49:54 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:49:55.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:49:54 smithi115 conmon[90393]: 2022-05-14T01:49:54.609+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:49:55.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:49:54 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:49:55.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:49:54 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:49:55.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:49:54 smithi115 ceph-mon[78593]: pgmap v4616: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:49:55.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:49:54 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:49:55.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:49:55 smithi115 conmon[90393]: 2022-05-14T01:49:55.570+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:49:55.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:49:55 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 8525 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:49:55.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:49:55 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:49:56.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:49:55 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 8525 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:49:56.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:49:55 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:49:56.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:49:56 smithi115 conmon[90393]: 2022-05-14T01:49:56.564+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:49:56.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:49:56 smithi115 ceph-mon[78593]: pgmap v4617: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:49:56.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:49:56 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:49:57.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:49:56 smithi047 ceph-mon[77147]: pgmap v4617: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:49:57.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:49:56 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:49:57.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:49:57 smithi115 conmon[90393]: 2022-05-14T01:49:57.530+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:49:57.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:49:57 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:49:58.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:49:57 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:49:58.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:49:58 smithi115 conmon[90393]: 2022-05-14T01:49:58.563+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:49:58.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:49:58 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:49:58.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:49:58 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:49:58.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:49:58 smithi115 ceph-mon[78593]: pgmap v4618: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:49:58.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:49:58 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:49:59.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:49:58 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:49:59.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:49:58 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:49:59.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:49:58 smithi047 ceph-mon[77147]: pgmap v4618: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:49:59.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:49:58 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:49:59.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:49:59 smithi115 conmon[90393]: 2022-05-14T01:49:59.520+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:49:59.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:49:59 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:50:00.000 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:49:59 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:50:00.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:50:00 smithi047 conmon[77123]: 2022-05-14T01:49:59.999+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : Health detail: HEALTH_ERR 1 filesystem is degraded; 1 mds daemon damaged; 1 MDSs are read only; 1 MDSs report slow metadata IOs; 1/3 mons down, quorum b,c; Reduced data availability: 1 pg inactive; 9 slow ops, oldest one blocked for 8530 sec, osd.5 has slow ops 2022-05-14T01:50:00.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:50:00 smithi047 conmon[77123]: 2022-05-14T01:49:59.999+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : [WRN] FS_DEGRADED: 1 filesystem is degraded 2022-05-14T01:50:00.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:50:00 smithi047 conmon[77123]: 2022-05-14T01:49:59.999+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : fs cephfs is degraded 2022-05-14T01:50:00.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:50:00 smithi047 conmon[77123]: 2022-05-14T01:50:00.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:50:00 smithi047 conmon[77123]: 2022-05-14T01:49:59.999+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : [ERR] MDS_DAMAGE: 1 mds daemon damaged 2022-05-14T01:50:00.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:50:00 smithi047 conmon[77123]: 2022-05-14T01:49:59.999+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : fs cephfs mds.2 is damaged 2022-05-14T01:50:00.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:50:00 smithi047 conmon[77123]: 2022-05-14T01:49:59.999+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : [WRN] MDS_READ_ONLY: 1 MDSs are read only 2022-05-14T01:50:00.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:50:00 smithi047 conmon[77123]: 2022-05-14T01:49:59.999+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : mds.l(mds.0): MDS in read-only mode 2022-05-14T01:50:00.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:50:00 smithi047 conmon[77123]: 2022-05-14T01:49:59.999+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : [WRN] MDS_SLOW_METADATA_IO: 1 MDSs report slow metadata IOs 2022-05-14T01:50:00.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:50:00 smithi047 conmon[77123]: 2022-05-14T01:49:59.999+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : mds.l(mds.0): 4 slow metadata IOs are blocked > 180 secs, oldest blocked for 8833 secs 2022-05-14T01:50:00.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:50:00 smithi047 conmon[77123]: 2022-05-14T01:49:59.999+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : [WRN] MON_DOWN: 1/3 mons down, quorum b,c 2022-05-14T01:50:00.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:50:00 smithi047 conmon[77123]: 2022-05-14T01:50:00.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:50:00 smithi047 conmon[77123]: 2022-05-14T01:49:59.999+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : mon.a (rank 0) addr [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] is down (out of quorum) 2022-05-14T01:50:00.295 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:50:00 smithi047 conmon[77123]: 2022-05-14T01:49:59.999+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : [WRN] PG_AVAILABILITY: Reduced data availability: 1 pg inactive 2022-05-14T01:50:00.295 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:50:00 smithi047 conmon[77123]: 2022-05-14T01:49:59.999+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : pg 2.1 is stuck inactive for 2h, current state unknown, last acting [] 2022-05-14T01:50:00.295 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:50:00 smithi047 conmon[77123]: 2022-05-14T01:49:59.999+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : [WRN] SLOW_OPS: 9 slow ops, oldest one blocked for 8530 sec, osd.5 has slow ops 2022-05-14T01:50:00.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:50:00 smithi115 conmon[90393]: 2022-05-14T01:50:00.480+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:50:00.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:50:00 smithi115 ceph-mon[78593]: pgmap v4619: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:50:00.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:50:00 smithi115 ceph-mon[78593]: Health detail: HEALTH_ERR 1 filesystem is degraded; 1 mds daemon damaged; 1 MDSs are read only; 1 MDSs report slow metadata IOs; 1/3 mons down, quorum b,c; Reduced data availability: 1 pg inactive; 9 slow ops, oldest one blocked for 8530 sec, osd.5 has slow ops 2022-05-14T01:50:00.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:50:00 smithi115 ceph-mon[78593]: [WRN] FS_DEGRADED: 1 filesystem is degraded 2022-05-14T01:50:00.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:50:00 smithi115 ceph-mon[78593]: fs cephfs is degraded 2022-05-14T01:50:00.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:50:00 smithi115 ceph-mon[78593]: [ERR] MDS_DAMAGE: 1 mds daemon damaged 2022-05-14T01:50:00.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:50:00 smithi115 ceph-mon[78593]: fs cephfs mds.2 is damaged 2022-05-14T01:50:00.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:50:00 smithi115 ceph-mon[78593]: [WRN] MDS_READ_ONLY: 1 MDSs are read only 2022-05-14T01:50:00.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:50:00 smithi115 ceph-mon[78593]: mds.l(mds.0): MDS in read-only mode 2022-05-14T01:50:00.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:50:00 smithi115 ceph-mon[78593]: [WRN] MDS_SLOW_METADATA_IO: 1 MDSs report slow metadata IOs 2022-05-14T01:50:00.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:50:00 smithi115 ceph-mon[78593]: mds.l(mds.0): 4 slow metadata IOs are blocked > 180 secs, oldest blocked for 8833 secs 2022-05-14T01:50:00.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:50:00 smithi115 ceph-mon[78593]: [WRN] MON_DOWN: 1/3 mons down, quorum b,c 2022-05-14T01:50:00.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:50:00 smithi115 ceph-mon[78593]: mon.a (rank 0) addr [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] is down (out of quorum) 2022-05-14T01:50:00.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:50:00 smithi115 ceph-mon[78593]: [WRN] PG_AVAILABILITY: Reduced data availability: 1 pg inactive 2022-05-14T01:50:00.862 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:50:00 smithi115 ceph-mon[78593]: pg 2.1 is stuck inactive for 2h, current state unknown, last acting [] 2022-05-14T01:50:00.862 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:50:00 smithi115 ceph-mon[78593]: [WRN] SLOW_OPS: 9 slow ops, oldest one blocked for 8530 sec, osd.5 has slow ops 2022-05-14T01:50:00.862 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:50:00 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 8530 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:50:00.862 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:50:00 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:50:01.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:50:00 smithi047 ceph-mon[77147]: pgmap v4619: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:50:01.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:50:00 smithi047 ceph-mon[77147]: Health detail: HEALTH_ERR 1 filesystem is degraded; 1 mds daemon damaged; 1 MDSs are read only; 1 MDSs report slow metadata IOs; 1/3 mons down, quorum b,c; Reduced data availability: 1 pg inactive; 9 slow ops, oldest one blocked for 8530 sec, osd.5 has slow ops 2022-05-14T01:50:01.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:50:00 smithi047 ceph-mon[77147]: [WRN] FS_DEGRADED: 1 filesystem is degraded 2022-05-14T01:50:01.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:50:00 smithi047 ceph-mon[77147]: fs cephfs is degraded 2022-05-14T01:50:01.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:50:00 smithi047 ceph-mon[77147]: [ERR] MDS_DAMAGE: 1 mds daemon damaged 2022-05-14T01:50:01.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:50:00 smithi047 ceph-mon[77147]: fs cephfs mds.2 is damaged 2022-05-14T01:50:01.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:50:00 smithi047 ceph-mon[77147]: [WRN] MDS_READ_ONLY: 1 MDSs are read only 2022-05-14T01:50:01.045 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:50:00 smithi047 ceph-mon[77147]: mds.l(mds.0): MDS in read-only mode 2022-05-14T01:50:01.045 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:50:00 smithi047 ceph-mon[77147]: [WRN] MDS_SLOW_METADATA_IO: 1 MDSs report slow metadata IOs 2022-05-14T01:50:01.046 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:50:00 smithi047 ceph-mon[77147]: mds.l(mds.0): 4 slow metadata IOs are blocked > 180 secs, oldest blocked for 8833 secs 2022-05-14T01:50:01.046 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:50:00 smithi047 ceph-mon[77147]: [WRN] MON_DOWN: 1/3 mons down, quorum b,c 2022-05-14T01:50:01.046 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:50:00 smithi047 ceph-mon[77147]: mon.a (rank 0) addr [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] is down (out of quorum) 2022-05-14T01:50:01.047 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:50:00 smithi047 ceph-mon[77147]: [WRN] PG_AVAILABILITY: Reduced data availability: 1 pg inactive 2022-05-14T01:50:01.047 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:50:00 smithi047 ceph-mon[77147]: pg 2.1 is stuck inactive for 2h, current state unknown, last acting [] 2022-05-14T01:50:01.048 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:50:00 smithi047 ceph-mon[77147]: [WRN] SLOW_OPS: 9 slow ops, oldest one blocked for 8530 sec, osd.5 has slow ops 2022-05-14T01:50:01.048 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:50:00 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 8530 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:50:01.049 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:50:00 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:50:01.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:50:01 smithi115 conmon[90393]: 2022-05-14T01:50:01.480+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:50:01.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:50:01 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:50:02.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:50:01 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:50:02.861 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:50:02 smithi115 conmon[90393]: 2022-05-14T01:50:02.503+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:50:02.862 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:50:02 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:50:02.862 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:50:02 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:50:02.862 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:50:02 smithi115 ceph-mon[78593]: pgmap v4620: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:50:02.862 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:50:02 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:50:03.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:50:02 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:50:03.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:50:02 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:50:03.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:50:02 smithi047 ceph-mon[77147]: pgmap v4620: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:50:03.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:50:02 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:50:03.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:50:03 smithi115 conmon[90393]: 2022-05-14T01:50:03.550+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:50:03.865 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:50:03 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:50:04.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:50:03 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:50:04.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:50:04 smithi115 conmon[90393]: 2022-05-14T01:50:04.589+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:50:04.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:50:04 smithi115 ceph-mon[78593]: pgmap v4621: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:50:04.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:50:04 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:50:05.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:50:04 smithi047 ceph-mon[77147]: pgmap v4621: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:50:05.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:50:04 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:50:05.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:50:05 smithi115 conmon[90393]: 2022-05-14T01:50:05.583+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:50:05.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:50:05 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 8535 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:50:05.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:50:05 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:50:06.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:50:05 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 8535 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:50:06.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:50:05 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:50:07.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:50:06 smithi047 ceph-mon[77147]: pgmap v4622: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:50:07.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:50:06 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:50:07.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:50:06 smithi115 conmon[90393]: 2022-05-14T01:50:06.621+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:50:07.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:50:06 smithi115 ceph-mon[78593]: pgmap v4622: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:50:07.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:50:06 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:50:07.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:50:07 smithi115 conmon[90393]: 2022-05-14T01:50:07.591+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:50:07.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:50:07 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:50:08.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:50:07 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:50:08.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:50:08 smithi115 conmon[90393]: 2022-05-14T01:50:08.595+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:50:08.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:50:08 smithi115 ceph-mon[78593]: pgmap v4623: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:50:08.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:50:08 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:50:09.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:50:08 smithi047 ceph-mon[77147]: pgmap v4623: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:50:09.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:50:08 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:50:09.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:50:09 smithi115 conmon[90393]: 2022-05-14T01:50:09.554+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:50:09.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:50:09 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:50:09.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:50:09 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:50:09.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:50:09 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:50:10.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:50:09 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:50:10.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:50:09 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:50:10.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:50:09 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:50:10.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:50:10 smithi115 conmon[90393]: 2022-05-14T01:50:10.599+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:50:10.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:50:10 smithi115 ceph-mon[78593]: pgmap v4624: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:50:10.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:50:10 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 8540 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:50:10.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:50:10 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:50:11.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:50:10 smithi047 ceph-mon[77147]: pgmap v4624: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:50:11.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:50:10 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 8540 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:50:11.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:50:10 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:50:11.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:50:11 smithi115 conmon[90393]: 2022-05-14T01:50:11.576+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:50:11.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:50:11 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:50:12.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:50:11 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:50:12.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:50:12 smithi115 conmon[90393]: 2022-05-14T01:50:12.589+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:50:12.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:50:12 smithi115 ceph-mon[78593]: pgmap v4625: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:50:12.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:50:12 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:50:13.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:50:12 smithi047 ceph-mon[77147]: pgmap v4625: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:50:13.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:50:12 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:50:14.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:50:13 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:50:14.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:50:13 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:50:14.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:50:13 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:50:14.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:50:13 smithi115 conmon[90393]: 2022-05-14T01:50:13.615+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:50:14.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:50:13 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:50:14.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:50:13 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:50:14.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:50:13 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:50:15.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:50:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T01:50:15.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:50:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T01:50:15.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:50:14 smithi047 ceph-mon[77147]: pgmap v4626: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:50:15.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:50:14 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:50:15.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:50:14 smithi115 conmon[90393]: 2022-05-14T01:50:14.615+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:50:15.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:50:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T01:50:15.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:50:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T01:50:15.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:50:14 smithi115 ceph-mon[78593]: pgmap v4626: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:50:15.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:50:14 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:50:15.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:50:15 smithi115 conmon[90393]: 2022-05-14T01:50:15.590+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:50:15.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:50:15 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 8545 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:50:15.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:50:15 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:50:16.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:50:15 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 8545 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:50:16.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:50:15 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:50:17.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:50:16 smithi047 ceph-mon[77147]: pgmap v4627: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:50:17.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:50:16 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:50:17.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:50:16 smithi115 conmon[90393]: 2022-05-14T01:50:16.639+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:50:17.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:50:16 smithi115 ceph-mon[78593]: pgmap v4627: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:50:17.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:50:16 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:50:18.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:50:17 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:50:18.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:50:17 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:50:18.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:50:17 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:50:18.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:50:17 smithi115 conmon[90393]: 2022-05-14T01:50:17.637+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:50:18.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:50:17 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:50:18.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:50:17 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:50:18.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:50:17 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:50:19.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:50:18 smithi047 ceph-mon[77147]: pgmap v4628: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:50:19.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:50:18 smithi115 conmon[90393]: 2022-05-14T01:50:18.685+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:50:19.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:50:18 smithi115 ceph-mon[78593]: pgmap v4628: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:50:20.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:50:19 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:50:20.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:50:19 smithi115 conmon[90393]: 2022-05-14T01:50:19.727+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:50:20.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:50:19 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:50:21.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:50:20 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:50:21.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:50:20 smithi047 ceph-mon[77147]: pgmap v4629: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:50:21.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:50:20 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 8550 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:50:21.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:50:20 smithi115 conmon[90393]: 2022-05-14T01:50:20.700+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:50:21.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:50:20 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:50:21.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:50:20 smithi115 ceph-mon[78593]: pgmap v4629: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:50:21.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:50:20 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 8550 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:50:22.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:50:21 smithi115 conmon[90393]: 2022-05-14T01:50:21.697+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:50:22.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:50:22 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:50:22.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:50:22 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:50:22.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:50:22 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:50:22.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:50:22 smithi047 ceph-mon[77147]: pgmap v4630: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:50:22.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:50:22 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:50:22.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:50:22 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:50:22.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:50:22 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:50:22.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:50:22 smithi115 ceph-mon[78593]: pgmap v4630: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:50:23.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:50:22 smithi115 conmon[90393]: 2022-05-14T01:50:22.696+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:50:23.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:50:23 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:50:23.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:50:23 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:50:24.110 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:50:23 smithi115 conmon[90393]: 2022-05-14T01:50:23.681+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:50:24.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:50:24 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:50:24.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:50:24 smithi047 ceph-mon[77147]: pgmap v4631: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:50:24.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:50:24 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:50:24.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:50:24 smithi115 ceph-mon[78593]: pgmap v4631: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:50:25.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:50:24 smithi115 conmon[90393]: 2022-05-14T01:50:24.658+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:50:25.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:50:25 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:50:25.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:50:25 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 8555 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:50:25.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:50:25 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:50:25.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:50:25 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 8555 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:50:26.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:50:25 smithi115 conmon[90393]: 2022-05-14T01:50:25.706+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:50:26.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:50:26 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:50:26.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:50:26 smithi047 ceph-mon[77147]: pgmap v4632: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:50:26.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:50:26 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:50:26.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:50:26 smithi115 ceph-mon[78593]: pgmap v4632: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:50:27.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:50:26 smithi115 conmon[90393]: 2022-05-14T01:50:26.738+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:50:27.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:50:27 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:50:27.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:50:27 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:50:28.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:50:27 smithi115 conmon[90393]: 2022-05-14T01:50:27.752+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:50:28.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:50:28 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:50:28.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:50:28 smithi047 ceph-mon[77147]: pgmap v4633: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:50:28.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:50:28 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:50:28.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:50:28 smithi115 ceph-mon[78593]: pgmap v4633: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:50:29.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:50:28 smithi115 conmon[90393]: 2022-05-14T01:50:28.760+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:50:29.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:50:29 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:50:29.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:50:29 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:50:30.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:50:29 smithi115 conmon[90393]: 2022-05-14T01:50:29.809+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:50:30.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:50:30 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:50:30.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:50:30 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:50:30.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:50:30 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:50:30.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:50:30 smithi047 ceph-mon[77147]: pgmap v4634: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:50:30.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:50:30 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 8560 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:50:30.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:50:30 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:50:30.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:50:30 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:50:30.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:50:30 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:50:30.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:50:30 smithi115 ceph-mon[78593]: pgmap v4634: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:50:30.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:50:30 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 8560 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:50:31.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:50:30 smithi115 conmon[90393]: 2022-05-14T01:50:30.832+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:50:31.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:50:31 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:50:31.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:50:31 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:50:32.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:50:31 smithi115 conmon[90393]: 2022-05-14T01:50:31.840+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:50:32.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:50:32 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:50:32.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:50:32 smithi047 ceph-mon[77147]: pgmap v4635: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:50:32.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:50:32 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:50:32.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:50:32 smithi115 ceph-mon[78593]: pgmap v4635: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:50:33.176 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:50:32 smithi115 conmon[90393]: 2022-05-14T01:50:32.882+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:50:33.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:50:33 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:50:33.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:50:33 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:50:34.181 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:50:33 smithi115 conmon[90393]: 2022-05-14T01:50:33.858+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:50:34.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:50:34 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:50:34.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:50:34 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:50:34.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:50:34 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:50:34.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:50:34 smithi047 ceph-mon[77147]: pgmap v4636: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:50:34.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:50:34 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:50:34.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:50:34 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:50:34.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:50:34 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:50:34.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:50:34 smithi115 ceph-mon[78593]: pgmap v4636: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:50:35.183 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:50:34 smithi115 conmon[90393]: 2022-05-14T01:50:34.894+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:50:35.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:50:35 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:50:35.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:50:35 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 8565 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:50:35.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:50:35 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:50:35.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:50:35 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 8565 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:50:36.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:50:35 smithi115 conmon[90393]: 2022-05-14T01:50:35.886+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:50:36.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:50:36 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:50:36.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:50:36 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T01:50:36.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:50:36 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T01:50:36.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:50:36 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T01:50:36.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:50:36 smithi047 ceph-mon[77147]: pgmap v4637: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 7 op/s 2022-05-14T01:50:36.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:50:36 smithi047 conmon[78521]: 2022-05-14T01:50:36.273+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:50:36.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:50:36 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T01:50:36.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:50:36 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:50:36.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:50:36 smithi047 conmon[78521]: yield 2022-05-14T01:50:36.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:50:36 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:50:36.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:50:36 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:50:36.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:50:36 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:50:36.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:50:36 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T01:50:36.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:50:36 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:50:36.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:50:36 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T01:50:36.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:50:36 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:50:36.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:50:36 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T01:50:36.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:50:36 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:50:36.547 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:50:36 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T01:50:36.547 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:50:36 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:50:36.547 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:50:36 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T01:50:36.547 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:50:36 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:50:36.548 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:50:36 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:50:36.548 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:50:36 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:50:36.577 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:50:36 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:50:36.578 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:50:36 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T01:50:36.578 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:50:36 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T01:50:36.578 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:50:36 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T01:50:36.578 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:50:36 smithi115 ceph-mon[78593]: pgmap v4637: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 7 op/s 2022-05-14T01:50:37.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:50:36 smithi115 conmon[90393]: 2022-05-14T01:50:36.862+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:50:37.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:50:37 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:50:37.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:50:37 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:50:37.858 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T01:50:37.859 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:50:37.859 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T01:50:37.859 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:50:37.859 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:50:37.859 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:50:37.860 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T01:50:37.860 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:50:37.860 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T01:50:37.860 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:50:37.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T01:50:37.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:50:37.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T01:50:37.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:50:37.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T01:50:37.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:50:37.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:50:37.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:50:37.862 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:50:37 smithi115 ceph-mon[78593]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T01:50:37.863 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:50:37 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' 2022-05-14T01:50:37.863 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:50:37 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' 2022-05-14T01:50:38.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:50:37 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:50:38.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:50:37 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:50:38.042 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T01:50:38.042 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:50:38.042 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T01:50:38.042 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:50:38.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:50:38.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:50:38.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T01:50:38.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:50:38.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T01:50:38.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:50:38.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T01:50:38.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:50:38.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T01:50:38.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:50:38.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T01:50:38.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:50:38.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:50:38.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:50:38.047 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:50:37 smithi047 ceph-mon[77147]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T01:50:38.047 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:50:37 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' 2022-05-14T01:50:38.047 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:50:37 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' 2022-05-14T01:50:38.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:50:37 smithi115 conmon[90393]: 2022-05-14T01:50:37.910+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:50:38.906 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:50:38 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:50:38.907 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:50:38 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:50:38.907 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:50:38 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:50:38.907 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:50:38 smithi115 ceph-mon[78593]: pgmap v4638: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 4 op/s 2022-05-14T01:50:39.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:50:38 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:50:39.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:50:38 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:50:39.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:50:38 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:50:39.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:50:38 smithi047 ceph-mon[77147]: pgmap v4638: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 4 op/s 2022-05-14T01:50:39.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:50:38 smithi115 conmon[90393]: 2022-05-14T01:50:38.905+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:50:39.624 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:50:39 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:50:39.624 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:50:39 smithi047 conmon[78521]: 2022-05-14T01:50:39.344+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:50:39.625 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:50:39 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T01:50:39.625 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:50:39 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:50:39.625 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:50:39 smithi047 conmon[78521]: yield 2022-05-14T01:50:39.626 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:50:39 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:50:39.626 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:50:39 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:50:39.626 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:50:39 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:50:39.626 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:50:39 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T01:50:39.627 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:50:39 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:50:39.627 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:50:39 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T01:50:39.627 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:50:39 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:50:39.627 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:50:39 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T01:50:39.628 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:50:39 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:50:39.628 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:50:39 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T01:50:39.628 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:50:39 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:50:39.628 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:50:39 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T01:50:39.628 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:50:39 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:50:39.629 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:50:39 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:50:39.629 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:50:39 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:50:39.629 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:50:39 smithi047 conmon[78521]: 2022-05-14T01:50:39.344+0000 7f5344f82700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T01:50:39.629 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:50:39 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T01:50:39.630 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:50:39 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:50:39.630 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:50:39 smithi047 conmon[78521]: yield 2022-05-14T01:50:39.630 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:50:39 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:50:39.630 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:50:39 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:50:39.631 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:50:39 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:50:39.631 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:50:39 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T01:50:39.631 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:50:39 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:50:39.631 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:50:39 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T01:50:39.632 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:50:39 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:50:39.632 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:50:39 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T01:50:39.632 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:50:39 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:50:39.633 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:50:39 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T01:50:39.633 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:50:39 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:50:39.633 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:50:39 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T01:50:39.633 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:50:39 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:50:39.634 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:50:39 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:50:39.635 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:50:39 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:50:39.635 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:50:39 smithi047 conmon[78521]: 2022-05-14T01:50:39.635 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:50:39 smithi047 conmon[78521]: During handling of the above exception, another exception occurred: 2022-05-14T01:50:39.635 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:50:39 smithi047 conmon[78521]: 2022-05-14T01:50:39.636 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:50:39 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T01:50:39.636 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:50:39 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T01:50:39.636 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:50:39 smithi047 conmon[78521]: return f(*arg) 2022-05-14T01:50:39.636 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:50:39 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T01:50:39.637 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:50:39 smithi047 conmon[78521]: self._write_client_files(client_files, host) 2022-05-14T01:50:39.637 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:50:39 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T01:50:39.637 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:50:39 smithi047 conmon[78521]: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T01:50:39.637 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:50:39 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T01:50:39.637 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:50:39 smithi047 conmon[78521]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T01:50:39.638 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:50:39 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T01:50:39.638 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:50:39 smithi047 conmon[78521]: return self.event_loop.get_result(coro) 2022-05-14T01:50:39.638 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:50:39 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T01:50:39.638 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:50:39 smithi047 conmon[78521]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T01:50:39.639 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:50:39 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T01:50:39.639 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:50:39 smithi047 conmon[78521]: return self.__get_result() 2022-05-14T01:50:39.639 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:50:39 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T01:50:39.639 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:50:39 smithi047 conmon[78521]: raise self._exception 2022-05-14T01:50:39.639 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:50:39 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T01:50:39.640 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:50:39 smithi047 conmon[78521]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T01:50:39.640 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:50:39 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T01:50:39.640 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:50:39 smithi047 conmon[78521]: conn = await self._remote_connection(host, addr) 2022-05-14T01:50:39.640 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:50:39 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T01:50:39.641 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:50:39 smithi047 conmon[78521]: raise 2022-05-14T01:50:39.641 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:50:39 smithi047 conmon[78521]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T01:50:39.641 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:50:39 smithi047 conmon[78521]: self.gen.throw(type, value, traceback) 2022-05-14T01:50:39.642 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:50:39 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T01:50:39.642 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:50:39 smithi047 conmon[78521]: raise OrchestratorError(msg) 2022-05-14T01:50:39.642 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:50:39 smithi047 conmon[78521]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:50:39.956 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:50:39 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:50:40.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:50:39 smithi115 conmon[90393]: 2022-05-14T01:50:39.953+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:50:40.986 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:50:40 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:50:40.986 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:50:40 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:50:40.986 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T01:50:40.987 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:50:40.987 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T01:50:40.987 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:50:40.988 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:50:40.988 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:50:40.988 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T01:50:40.988 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:50:40.989 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T01:50:40.989 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:50:40.989 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T01:50:40.989 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:50:40.990 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T01:50:40.990 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:50:40.990 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T01:50:40.990 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:50:40.991 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:50:40.991 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:50:40.991 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:50:40 smithi115 ceph-mon[78593]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T01:50:40.991 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T01:50:40.992 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:50:40.992 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T01:50:40.992 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:50:40.992 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:50:40.993 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:50:40.993 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T01:50:40.993 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:50:40.994 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T01:50:40.994 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:50:40.994 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T01:50:40.994 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:50:40.995 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T01:50:40.995 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:50:40.995 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T01:50:40.996 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:50:40.996 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:50:40.997 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:50:40.997 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T01:50:40.998 INFO:journalctl@ceph.mon.c.smithi115.stdout: During handling of the above exception, another exception occurred: 2022-05-14T01:50:40.998 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T01:50:40.998 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T01:50:40.999 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T01:50:40.999 INFO:journalctl@ceph.mon.c.smithi115.stdout: return f(*arg) 2022-05-14T01:50:41.000 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T01:50:41.000 INFO:journalctl@ceph.mon.c.smithi115.stdout: self._write_client_files(client_files, host) 2022-05-14T01:50:41.001 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T01:50:41.001 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T01:50:41.001 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T01:50:41.002 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T01:50:41.002 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T01:50:41.003 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.event_loop.get_result(coro) 2022-05-14T01:50:41.003 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T01:50:41.003 INFO:journalctl@ceph.mon.c.smithi115.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T01:50:41.004 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T01:50:41.004 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.__get_result() 2022-05-14T01:50:41.005 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T01:50:41.005 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise self._exception 2022-05-14T01:50:41.006 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T01:50:41.006 INFO:journalctl@ceph.mon.c.smithi115.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T01:50:41.006 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T01:50:41.007 INFO:journalctl@ceph.mon.c.smithi115.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T01:50:41.007 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T01:50:41.007 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise 2022-05-14T01:50:41.007 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T01:50:41.008 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.gen.throw(type, value, traceback) 2022-05-14T01:50:41.008 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T01:50:41.008 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OrchestratorError(msg) 2022-05-14T01:50:41.008 INFO:journalctl@ceph.mon.c.smithi115.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:50:41.008 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:50:40 smithi115 ceph-mon[78593]: pgmap v4639: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 7 op/s 2022-05-14T01:50:41.009 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:50:40 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 8570 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:50:41.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:50:40 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:50:41.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:50:40 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:50:41.042 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T01:50:41.042 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:50:41.042 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T01:50:41.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:50:41.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:50:41.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:50:41.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T01:50:41.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:50:41.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T01:50:41.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:50:41.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T01:50:41.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:50:41.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T01:50:41.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:50:41.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T01:50:41.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:50:41.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:50:41.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:50:41.046 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:50:40 smithi047 ceph-mon[77147]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T01:50:41.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T01:50:41.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:50:41.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T01:50:41.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:50:41.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:50:41.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:50:41.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T01:50:41.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:50:41.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T01:50:41.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:50:41.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T01:50:41.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:50:41.049 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T01:50:41.049 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:50:41.049 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T01:50:41.049 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:50:41.049 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:50:41.050 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:50:41.050 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T01:50:41.050 INFO:journalctl@ceph.mon.b.smithi047.stdout: During handling of the above exception, another exception occurred: 2022-05-14T01:50:41.050 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T01:50:41.050 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T01:50:41.051 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T01:50:41.051 INFO:journalctl@ceph.mon.b.smithi047.stdout: return f(*arg) 2022-05-14T01:50:41.051 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T01:50:41.051 INFO:journalctl@ceph.mon.b.smithi047.stdout: self._write_client_files(client_files, host) 2022-05-14T01:50:41.051 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T01:50:41.052 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T01:50:41.052 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T01:50:41.052 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T01:50:41.052 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T01:50:41.053 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.event_loop.get_result(coro) 2022-05-14T01:50:41.053 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T01:50:41.053 INFO:journalctl@ceph.mon.b.smithi047.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T01:50:41.053 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T01:50:41.053 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.__get_result() 2022-05-14T01:50:41.054 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T01:50:41.054 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise self._exception 2022-05-14T01:50:41.054 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T01:50:41.054 INFO:journalctl@ceph.mon.b.smithi047.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T01:50:41.054 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T01:50:41.055 INFO:journalctl@ceph.mon.b.smithi047.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T01:50:41.055 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T01:50:41.055 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise 2022-05-14T01:50:41.055 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T01:50:41.055 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.gen.throw(type, value, traceback) 2022-05-14T01:50:41.056 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T01:50:41.056 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OrchestratorError(msg) 2022-05-14T01:50:41.056 INFO:journalctl@ceph.mon.b.smithi047.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:50:41.056 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:50:40 smithi047 ceph-mon[77147]: pgmap v4639: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 7 op/s 2022-05-14T01:50:41.056 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:50:40 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 8570 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:50:41.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:50:40 smithi115 conmon[90393]: 2022-05-14T01:50:40.982+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:50:41.995 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:50:41 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:50:42.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:50:41 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:50:42.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:50:41 smithi115 conmon[90393]: 2022-05-14T01:50:41.993+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:50:42.969 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:50:42 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:50:42.970 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:50:42 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:50:42.970 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:50:42 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:50:42.971 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:50:42 smithi115 ceph-mon[78593]: pgmap v4640: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 4 op/s 2022-05-14T01:50:43.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:50:42 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:50:43.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:50:42 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:50:43.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:50:42 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:50:43.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:50:42 smithi047 ceph-mon[77147]: pgmap v4640: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 4 op/s 2022-05-14T01:50:43.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:50:42 smithi115 conmon[90393]: 2022-05-14T01:50:42.968+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:50:43.922 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:50:43 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:50:44.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:50:43 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:50:44.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:50:43 smithi115 conmon[90393]: 2022-05-14T01:50:43.920+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:50:44.969 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:50:44 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:50:44.970 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:50:44 smithi115 ceph-mon[78593]: pgmap v4641: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 4 op/s 2022-05-14T01:50:45.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:50:44 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:50:45.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:50:44 smithi047 ceph-mon[77147]: pgmap v4641: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 4 op/s 2022-05-14T01:50:45.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:50:44 smithi115 conmon[90393]: 2022-05-14T01:50:44.968+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:50:46.001 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:50:45 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:50:46.001 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:50:45 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 8575 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:50:46.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:50:45 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:50:46.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:50:45 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 8575 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:50:46.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:50:46 smithi115 conmon[90393]: 2022-05-14T01:50:45.999+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:50:47.025 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:50:46 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:50:47.025 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:50:46 smithi115 ceph-mon[78593]: pgmap v4642: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 7 op/s 2022-05-14T01:50:47.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:50:46 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:50:47.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:50:46 smithi047 ceph-mon[77147]: pgmap v4642: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 7 op/s 2022-05-14T01:50:47.363 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:50:47 smithi115 conmon[90393]: 2022-05-14T01:50:47.023+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:50:48.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:50:47 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:50:48.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:50:47 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:50:48.069 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:50:47 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:50:48.070 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:50:47 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:50:48.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:50:48 smithi115 conmon[90393]: 2022-05-14T01:50:48.068+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:50:49.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:50:48 smithi047 ceph-mon[77147]: pgmap v4643: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:50:49.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:50:48 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:50:49.050 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:50:48 smithi115 ceph-mon[78593]: pgmap v4643: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:50:49.051 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:50:48 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:50:49.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:50:49 smithi115 conmon[90393]: 2022-05-14T01:50:49.049+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:50:50.040 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:50:49 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:50:50.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:50:49 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:50:50.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:50:50 smithi115 conmon[90393]: 2022-05-14T01:50:50.039+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:50:51.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:50:50 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:50:51.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:50:50 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:50:51.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:50:50 smithi047 ceph-mon[77147]: pgmap v4644: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:50:51.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:50:50 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:50:51.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:50:50 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 8580 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:50:51.088 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:50:50 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:50:51.088 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:50:50 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:50:51.089 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:50:50 smithi115 ceph-mon[78593]: pgmap v4644: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:50:51.089 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:50:50 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:50:51.089 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:50:50 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 8580 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:50:51.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:50:51 smithi115 conmon[90393]: 2022-05-14T01:50:51.087+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:50:52.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:50:51 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:50:52.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:50:51 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:50:52.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:50:52 smithi115 conmon[90393]: 2022-05-14T01:50:52.131+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:50:53.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:50:52 smithi047 ceph-mon[77147]: pgmap v4645: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:50:53.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:50:52 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:50:53.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:50:52 smithi115 ceph-mon[78593]: pgmap v4645: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:50:53.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:50:52 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:50:53.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:50:53 smithi115 conmon[90393]: 2022-05-14T01:50:53.133+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:50:54.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:50:53 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:50:54.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:50:53 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:50:54.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:50:54 smithi115 conmon[90393]: 2022-05-14T01:50:54.156+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:50:55.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:50:54 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:50:55.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:50:54 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:50:55.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:50:54 smithi047 ceph-mon[77147]: pgmap v4646: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:50:55.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:50:54 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:50:55.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:50:54 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:50:55.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:50:54 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:50:55.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:50:54 smithi115 ceph-mon[78593]: pgmap v4646: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:50:55.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:50:54 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:50:55.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:50:55 smithi115 conmon[90393]: 2022-05-14T01:50:55.160+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:50:56.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:50:55 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 8584 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:50:56.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:50:55 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:50:56.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:50:55 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 8584 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:50:56.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:50:55 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:50:56.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:50:56 smithi115 conmon[90393]: 2022-05-14T01:50:56.194+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:50:57.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:50:56 smithi047 ceph-mon[77147]: pgmap v4647: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:50:57.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:50:56 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:50:57.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:50:56 smithi115 ceph-mon[78593]: pgmap v4647: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:50:57.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:50:56 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:50:57.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:50:57 smithi115 conmon[90393]: 2022-05-14T01:50:57.146+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:50:58.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:50:57 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:50:58.105 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:50:57 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:50:58.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:50:58 smithi115 conmon[90393]: 2022-05-14T01:50:58.105+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:50:59.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:50:58 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:50:59.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:50:58 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:50:59.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:50:58 smithi047 ceph-mon[77147]: pgmap v4648: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:50:59.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:50:58 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:50:59.058 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:50:58 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:50:59.058 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:50:58 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:50:59.059 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:50:58 smithi115 ceph-mon[78593]: pgmap v4648: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:50:59.059 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:50:58 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:50:59.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:50:59 smithi115 conmon[90393]: 2022-05-14T01:50:59.056+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:51:00.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:50:59 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:51:00.092 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:50:59 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:51:00.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:51:00 smithi115 conmon[90393]: 2022-05-14T01:51:00.091+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:51:01.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:51:00 smithi047 ceph-mon[77147]: pgmap v4649: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:51:01.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:51:00 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:51:01.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:51:00 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 8590 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:51:01.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:51:00 smithi115 ceph-mon[78593]: pgmap v4649: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:51:01.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:51:00 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:51:01.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:51:00 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 8590 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:51:01.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:51:01 smithi115 conmon[90393]: 2022-05-14T01:51:01.111+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:51:02.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:51:01 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:51:02.065 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:51:01 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:51:02.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:51:02 smithi115 conmon[90393]: 2022-05-14T01:51:02.065+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:51:03.036 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:51:02 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:51:03.037 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:51:02 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:51:03.037 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:51:02 smithi115 ceph-mon[78593]: pgmap v4650: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:51:03.037 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:51:02 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:51:03.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:51:02 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:51:03.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:51:02 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:51:03.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:51:02 smithi047 ceph-mon[77147]: pgmap v4650: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:51:03.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:51:02 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:51:03.361 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:51:03 smithi115 conmon[90393]: 2022-05-14T01:51:03.036+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:51:04.004 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:51:03 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:51:04.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:51:03 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:51:04.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:51:04 smithi115 conmon[90393]: 2022-05-14T01:51:04.003+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:51:05.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:51:04 smithi047 ceph-mon[77147]: pgmap v4651: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:51:05.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:51:04 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:51:05.043 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:51:04 smithi115 ceph-mon[78593]: pgmap v4651: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:51:05.043 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:51:04 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:51:05.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:51:05 smithi115 conmon[90393]: 2022-05-14T01:51:05.042+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:51:06.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:51:05 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:51:06.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:51:05 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 8595 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:51:06.072 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:51:05 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:51:06.073 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:51:05 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 8595 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:51:06.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:51:06 smithi115 conmon[90393]: 2022-05-14T01:51:06.071+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:51:07.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:51:06 smithi047 ceph-mon[77147]: pgmap v4652: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:51:07.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:51:06 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:51:07.076 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:51:06 smithi115 ceph-mon[78593]: pgmap v4652: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:51:07.076 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:51:06 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:51:07.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:51:07 smithi115 conmon[90393]: 2022-05-14T01:51:07.074+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:51:08.040 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:51:07 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:51:08.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:51:07 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:51:08.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:51:08 smithi115 conmon[90393]: 2022-05-14T01:51:08.038+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:51:09.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:51:08 smithi047 ceph-mon[77147]: pgmap v4653: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:51:09.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:51:08 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:51:09.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:51:08 smithi115 ceph-mon[78593]: pgmap v4653: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:51:09.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:51:08 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:51:09.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:51:08 smithi115 conmon[90393]: 2022-05-14T01:51:08.993+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:51:10.016 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:51:09 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:51:10.016 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:51:09 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:51:10.016 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:51:09 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:51:10.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:51:09 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:51:10.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:51:09 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:51:10.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:51:09 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:51:10.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:51:10 smithi115 conmon[90393]: 2022-05-14T01:51:10.014+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:51:11.010 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:51:10 smithi115 ceph-mon[78593]: pgmap v4654: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:51:11.010 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:51:10 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:51:11.010 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:51:10 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 8600 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:51:11.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:51:10 smithi047 ceph-mon[77147]: pgmap v4654: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:51:11.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:51:10 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:51:11.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:51:10 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 8600 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:51:11.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:51:11 smithi115 conmon[90393]: 2022-05-14T01:51:11.008+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:51:12.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:51:11 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:51:12.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:51:11 smithi115 conmon[90393]: 2022-05-14T01:51:11.981+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:51:12.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:51:11 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:51:13.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:51:12 smithi047 ceph-mon[77147]: pgmap v4655: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:51:13.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:51:12 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:51:13.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:51:12 smithi115 conmon[90393]: 2022-05-14T01:51:12.996+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:51:13.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:51:12 smithi115 ceph-mon[78593]: pgmap v4655: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:51:13.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:51:12 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:51:14.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:51:13 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:51:14.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:51:13 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:51:14.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:51:13 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:51:14.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:51:14 smithi115 conmon[90393]: 2022-05-14T01:51:14.000+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:51:14.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:51:13 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:51:14.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:51:13 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:51:14.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:51:13 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:51:15.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:51:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T01:51:15.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:51:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T01:51:15.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:51:14 smithi047 ceph-mon[77147]: pgmap v4656: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:51:15.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:51:14 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:51:15.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:51:14 smithi115 conmon[90393]: 2022-05-14T01:51:14.990+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:51:15.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:51:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T01:51:15.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:51:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T01:51:15.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:51:14 smithi115 ceph-mon[78593]: pgmap v4656: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:51:15.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:51:14 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:51:16.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:51:15 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:51:16.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:51:15 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 8605 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:51:16.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:51:15 smithi115 conmon[90393]: 2022-05-14T01:51:15.967+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:51:16.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:51:15 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:51:16.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:51:15 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 8605 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:51:17.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:51:16 smithi047 ceph-mon[77147]: pgmap v4657: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:51:17.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:51:16 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:51:17.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:51:17 smithi115 conmon[90393]: 2022-05-14T01:51:17.013+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:51:17.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:51:16 smithi115 ceph-mon[78593]: pgmap v4657: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:51:17.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:51:16 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:51:18.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:51:17 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:51:18.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:51:17 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:51:18.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:51:17 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:51:18.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:51:17 smithi115 conmon[90393]: 2022-05-14T01:51:17.975+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:51:18.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:51:17 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:51:18.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:51:17 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:51:18.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:51:17 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:51:19.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:51:18 smithi047 ceph-mon[77147]: pgmap v4658: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:51:19.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:51:18 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:51:19.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:51:18 smithi115 conmon[90393]: 2022-05-14T01:51:18.957+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:51:19.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:51:18 smithi115 ceph-mon[78593]: pgmap v4658: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:51:19.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:51:18 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:51:20.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:51:19 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:51:20.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:51:19 smithi115 conmon[90393]: 2022-05-14T01:51:19.929+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:51:20.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:51:19 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:51:21.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:51:20 smithi047 ceph-mon[77147]: pgmap v4659: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:51:21.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:51:20 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:51:21.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:51:20 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 8610 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:51:21.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:51:20 smithi115 conmon[90393]: 2022-05-14T01:51:20.949+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:51:21.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:51:20 smithi115 ceph-mon[78593]: pgmap v4659: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:51:21.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:51:20 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:51:21.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:51:20 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 8610 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:51:22.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:51:21 smithi115 conmon[90393]: 2022-05-14T01:51:21.958+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:51:22.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:51:22 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:51:22.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:51:22 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:51:22.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:51:22 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:51:22.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:51:22 smithi115 ceph-mon[78593]: pgmap v4660: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:51:22.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:51:22 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:51:22.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:51:22 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:51:22.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:51:22 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:51:22.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:51:22 smithi047 ceph-mon[77147]: pgmap v4660: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:51:23.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:51:22 smithi115 conmon[90393]: 2022-05-14T01:51:22.961+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:51:23.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:51:23 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:51:23.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:51:23 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:51:24.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:51:23 smithi115 conmon[90393]: 2022-05-14T01:51:23.920+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:51:24.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:51:24 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:51:24.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:51:24 smithi115 ceph-mon[78593]: pgmap v4661: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:51:24.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:51:24 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:51:24.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:51:24 smithi047 ceph-mon[77147]: pgmap v4661: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:51:25.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:51:24 smithi115 conmon[90393]: 2022-05-14T01:51:24.922+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:51:25.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:51:25 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:51:25.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:51:25 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 8615 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:51:25.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:51:25 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:51:25.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:51:25 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 8615 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:51:26.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:51:25 smithi115 conmon[90393]: 2022-05-14T01:51:25.928+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:51:26.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:51:26 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:51:26.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:51:26 smithi115 ceph-mon[78593]: pgmap v4662: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:51:26.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:51:26 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:51:26.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:51:26 smithi047 ceph-mon[77147]: pgmap v4662: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:51:27.161 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:51:26 smithi115 conmon[90393]: 2022-05-14T01:51:26.880+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:51:27.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:51:27 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:51:27.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:51:27 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:51:28.166 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:51:27 smithi115 conmon[90393]: 2022-05-14T01:51:27.885+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:51:28.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:51:28 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:51:28.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:51:28 smithi047 ceph-mon[77147]: pgmap v4663: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:51:28.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:51:28 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:51:28.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:51:28 smithi115 ceph-mon[78593]: pgmap v4663: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:51:29.172 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:51:28 smithi115 conmon[90393]: 2022-05-14T01:51:28.908+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:51:29.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:51:29 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:51:29.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:51:29 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:51:30.175 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:51:29 smithi115 conmon[90393]: 2022-05-14T01:51:29.903+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:51:30.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:51:30 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:51:30.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:51:30 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:51:30.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:51:30 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:51:30.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:51:30 smithi047 ceph-mon[77147]: pgmap v4664: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:51:30.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:51:30 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 8620 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:51:30.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:51:30 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:51:30.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:51:30 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:51:30.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:51:30 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:51:30.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:51:30 smithi115 ceph-mon[78593]: pgmap v4664: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:51:30.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:51:30 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 8620 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:51:31.182 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:51:30 smithi115 conmon[90393]: 2022-05-14T01:51:30.874+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:51:31.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:51:31 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:51:31.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:51:31 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:51:32.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:51:31 smithi115 conmon[90393]: 2022-05-14T01:51:31.826+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:51:32.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:51:32 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:51:32.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:51:32 smithi047 ceph-mon[77147]: pgmap v4665: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:51:32.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:51:32 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:51:32.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:51:32 smithi115 ceph-mon[78593]: pgmap v4665: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:51:33.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:51:32 smithi115 conmon[90393]: 2022-05-14T01:51:32.798+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:51:33.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:51:33 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:51:33.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:51:33 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:51:34.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:51:33 smithi115 conmon[90393]: 2022-05-14T01:51:33.790+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:51:34.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:51:34 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:51:34.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:51:34 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:51:34.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:51:34 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:51:34.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:51:34 smithi047 ceph-mon[77147]: pgmap v4666: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:51:34.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:51:34 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:51:34.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:51:34 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:51:34.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:51:34 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:51:34.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:51:34 smithi115 ceph-mon[78593]: pgmap v4666: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:51:35.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:51:34 smithi115 conmon[90393]: 2022-05-14T01:51:34.797+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:51:35.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:51:35 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:51:35.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:51:35 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 8625 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:51:35.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:51:35 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:51:35.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:51:35 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 8625 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:51:36.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:51:35 smithi115 conmon[90393]: 2022-05-14T01:51:35.783+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:51:36.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:51:36 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:51:36.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:51:36 smithi047 ceph-mon[77147]: pgmap v4667: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:51:36.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:51:36 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:51:36.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:51:36 smithi115 ceph-mon[78593]: pgmap v4667: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:51:37.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:51:36 smithi115 conmon[90393]: 2022-05-14T01:51:36.742+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:51:37.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:51:37 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:51:37.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:51:37 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:51:38.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:51:37 smithi115 conmon[90393]: 2022-05-14T01:51:37.791+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:51:38.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:51:38 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:51:38.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:51:38 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:51:38.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:51:38 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:51:38.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:51:38 smithi047 ceph-mon[77147]: pgmap v4668: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:51:38.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:51:38 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:51:38.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:51:38 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:51:38.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:51:38 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:51:38.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:51:38 smithi115 ceph-mon[78593]: pgmap v4668: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:51:39.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:51:38 smithi115 conmon[90393]: 2022-05-14T01:51:38.789+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:51:39.474 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:51:39 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:51:39.475 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:51:39 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:51:40.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:51:39 smithi115 conmon[90393]: 2022-05-14T01:51:39.768+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:51:40.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:51:40 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:51:40.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:51:40 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T01:51:40.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:51:40 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T01:51:40.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:51:40 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T01:51:40.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:51:40 smithi047 ceph-mon[77147]: pgmap v4669: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:51:40.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:51:40 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 8630 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:51:40.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:51:40 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:51:40.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:51:40 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T01:51:40.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:51:40 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T01:51:40.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:51:40 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T01:51:40.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:51:40 smithi115 ceph-mon[78593]: pgmap v4669: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:51:40.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:51:40 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 8630 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:51:41.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:51:40 smithi115 conmon[90393]: 2022-05-14T01:51:40.761+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:51:41.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:51:41 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:51:41.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:51:41 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:51:42.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:51:41 smithi115 conmon[90393]: 2022-05-14T01:51:41.792+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:51:42.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:51:42 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:51:42.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:51:42 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:51:42.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:51:42 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:51:42.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:51:42 smithi047 ceph-mon[77147]: pgmap v4670: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:51:42.542 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:51:42 smithi047 conmon[78521]: 2022-05-14T01:51:42.448+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:51:42.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:51:42 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T01:51:42.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:51:42 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:51:42.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:51:42 smithi047 conmon[78521]: yield 2022-05-14T01:51:42.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:51:42 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:51:42.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:51:42 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:51:42.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:51:42 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:51:42.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:51:42 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T01:51:42.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:51:42 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:51:42.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:51:42 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T01:51:42.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:51:42 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:51:42.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:51:42 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T01:51:42.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:51:42 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:51:42.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:51:42 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T01:51:42.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:51:42 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:51:42.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:51:42 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T01:51:42.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:51:42 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:51:42.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:51:42 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:51:42.547 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:51:42 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:51:42.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:51:42 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:51:42.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:51:42 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:51:42.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:51:42 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:51:42.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:51:42 smithi115 ceph-mon[78593]: pgmap v4670: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:51:43.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:51:42 smithi115 conmon[90393]: 2022-05-14T01:51:42.759+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:51:43.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:51:43 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:51:43.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:51:43 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:51:43.542 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T01:51:43.542 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:51:43.542 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T01:51:43.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:51:43.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:51:43.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:51:43.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T01:51:43.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:51:43.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T01:51:43.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:51:43.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T01:51:43.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:51:43.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T01:51:43.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:51:43.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T01:51:43.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:51:43.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:51:43.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:51:43.546 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:51:43 smithi047 ceph-mon[77147]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T01:51:43.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:51:43 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:51:43.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:51:43 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:51:43.608 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T01:51:43.609 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:51:43.609 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T01:51:43.609 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:51:43.609 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:51:43.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:51:43.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T01:51:43.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:51:43.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T01:51:43.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:51:43.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T01:51:43.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:51:43.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T01:51:43.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:51:43.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T01:51:43.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:51:43.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:51:43.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:51:43.612 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:51:43 smithi115 ceph-mon[78593]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T01:51:44.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:51:43 smithi115 conmon[90393]: 2022-05-14T01:51:43.803+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:51:44.546 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:51:44 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:51:44.547 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:51:44 smithi047 ceph-mon[77147]: pgmap v4671: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:51:44.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:51:44 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:51:44.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:51:44 smithi115 ceph-mon[78593]: pgmap v4671: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:51:45.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:51:44 smithi115 conmon[90393]: 2022-05-14T01:51:44.809+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:51:45.521 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:51:45 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:51:45.522 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:51:45 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 8635 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:51:45.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:51:45 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:51:45.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:51:45 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 8635 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:51:45.792 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:51:45 smithi047 conmon[78521]: 2022-05-14T01:51:45.519+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:51:45.792 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:51:45 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T01:51:45.792 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:51:45 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:51:45.793 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:51:45 smithi047 conmon[78521]: yield 2022-05-14T01:51:45.793 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:51:45 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:51:45.793 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:51:45 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:51:45.793 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:51:45 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:51:45.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:51:45 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T01:51:45.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:51:45 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:51:45.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:51:45 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T01:51:45.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:51:45 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:51:45.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:51:45 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T01:51:45.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:51:45 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:51:45.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:51:45 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T01:51:45.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:51:45 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:51:45.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:51:45 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T01:51:45.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:51:45 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:51:45.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:51:45 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:51:45.797 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:51:45 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:51:45.797 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:51:45 smithi047 conmon[78521]: 2022-05-14T01:51:45.520+0000 7f5344781700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T01:51:45.797 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:51:45 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T01:51:45.797 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:51:45 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:51:45.798 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:51:45 smithi047 conmon[78521]: yield 2022-05-14T01:51:45.798 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:51:45 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:51:45.798 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:51:45 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:51:45.798 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:51:45 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:51:45.799 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:51:45 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T01:51:45.799 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:51:45 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:51:45.799 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:51:45 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T01:51:45.799 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:51:45 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:51:45.799 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:51:45 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T01:51:45.800 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:51:45 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:51:45.800 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:51:45 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T01:51:45.800 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:51:45 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:51:45.800 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:51:45 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T01:51:45.801 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:51:45 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:51:45.801 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:51:45 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:51:45.802 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:51:45 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:51:45.802 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:51:45 smithi047 conmon[78521]: 2022-05-14T01:51:45.802 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:51:45 smithi047 conmon[78521]: During handling of the above exception, another exception occurred: 2022-05-14T01:51:45.802 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:51:45 smithi047 conmon[78521]: 2022-05-14T01:51:45.803 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:51:45 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T01:51:45.803 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:51:45 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T01:51:45.803 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:51:45 smithi047 conmon[78521]: return f(*arg) 2022-05-14T01:51:45.803 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:51:45 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T01:51:45.803 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:51:45 smithi047 conmon[78521]: self._write_client_files(client_files, host) 2022-05-14T01:51:45.804 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:51:45 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T01:51:45.804 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:51:45 smithi047 conmon[78521]: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T01:51:45.804 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:51:45 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T01:51:45.804 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:51:45 smithi047 conmon[78521]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T01:51:45.805 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:51:45 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T01:51:45.805 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:51:45 smithi047 conmon[78521]: return self.event_loop.get_result(coro) 2022-05-14T01:51:45.805 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:51:45 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T01:51:45.805 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:51:45 smithi047 conmon[78521]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T01:51:45.806 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:51:45 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T01:51:45.806 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:51:45 smithi047 conmon[78521]: return self.__get_result() 2022-05-14T01:51:45.806 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:51:45 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T01:51:45.806 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:51:45 smithi047 conmon[78521]: raise self._exception 2022-05-14T01:51:45.807 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:51:45 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T01:51:45.807 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:51:45 smithi047 conmon[78521]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T01:51:45.807 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:51:45 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T01:51:45.807 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:51:45 smithi047 conmon[78521]: conn = await self._remote_connection(host, addr) 2022-05-14T01:51:45.807 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:51:45 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T01:51:45.808 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:51:45 smithi047 conmon[78521]: raise 2022-05-14T01:51:45.808 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:51:45 smithi047 conmon[78521]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T01:51:45.808 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:51:45 smithi047 conmon[78521]: self.gen.throw(type, value, traceback) 2022-05-14T01:51:45.808 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:51:45 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T01:51:45.809 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:51:45 smithi047 conmon[78521]: raise OrchestratorError(msg) 2022-05-14T01:51:45.809 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:51:45 smithi047 conmon[78521]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:51:46.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:51:45 smithi115 conmon[90393]: 2022-05-14T01:51:45.821+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:51:46.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:51:46 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:51:46.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:51:46 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:51:46.542 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T01:51:46.542 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:51:46.542 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T01:51:46.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:51:46.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:51:46.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:51:46.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T01:51:46.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:51:46.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T01:51:46.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:51:46.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T01:51:46.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:51:46.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T01:51:46.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:51:46.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T01:51:46.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:51:46.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:51:46.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:51:46.546 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:51:46 smithi047 ceph-mon[77147]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T01:51:46.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T01:51:46.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:51:46.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T01:51:46.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:51:46.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:51:46.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:51:46.548 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T01:51:46.548 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:51:46.548 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T01:51:46.548 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:51:46.549 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T01:51:46.549 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:51:46.549 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T01:51:46.549 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:51:46.550 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T01:51:46.550 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:51:46.550 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:51:46.550 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:51:46.551 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T01:51:46.551 INFO:journalctl@ceph.mon.b.smithi047.stdout: During handling of the above exception, another exception occurred: 2022-05-14T01:51:46.551 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T01:51:46.551 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T01:51:46.551 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T01:51:46.552 INFO:journalctl@ceph.mon.b.smithi047.stdout: return f(*arg) 2022-05-14T01:51:46.552 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T01:51:46.552 INFO:journalctl@ceph.mon.b.smithi047.stdout: self._write_client_files(client_files, host) 2022-05-14T01:51:46.552 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T01:51:46.552 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T01:51:46.553 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T01:51:46.553 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T01:51:46.553 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T01:51:46.553 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.event_loop.get_result(coro) 2022-05-14T01:51:46.554 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T01:51:46.554 INFO:journalctl@ceph.mon.b.smithi047.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T01:51:46.554 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T01:51:46.554 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.__get_result() 2022-05-14T01:51:46.554 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T01:51:46.555 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise self._exception 2022-05-14T01:51:46.555 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T01:51:46.555 INFO:journalctl@ceph.mon.b.smithi047.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T01:51:46.555 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T01:51:46.555 INFO:journalctl@ceph.mon.b.smithi047.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T01:51:46.556 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T01:51:46.556 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise 2022-05-14T01:51:46.556 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T01:51:46.556 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.gen.throw(type, value, traceback) 2022-05-14T01:51:46.557 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T01:51:46.557 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OrchestratorError(msg) 2022-05-14T01:51:46.557 INFO:journalctl@ceph.mon.b.smithi047.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:51:46.557 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:51:46 smithi047 ceph-mon[77147]: pgmap v4672: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:51:46.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:51:46 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:51:46.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:51:46 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:51:46.609 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T01:51:46.609 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:51:46.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T01:51:46.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:51:46.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:51:46.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:51:46.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T01:51:46.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:51:46.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T01:51:46.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:51:46.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T01:51:46.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:51:46.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T01:51:46.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:51:46.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T01:51:46.613 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:51:46.613 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:51:46.613 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:51:46.613 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:51:46 smithi115 ceph-mon[78593]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T01:51:46.613 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T01:51:46.614 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:51:46.614 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T01:51:46.614 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:51:46.614 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:51:46.614 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:51:46.615 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T01:51:46.615 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:51:46.615 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T01:51:46.616 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:51:46.616 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T01:51:46.617 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:51:46.617 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T01:51:46.617 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:51:46.617 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T01:51:46.618 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:51:46.618 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:51:46.618 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:51:46.618 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T01:51:46.619 INFO:journalctl@ceph.mon.c.smithi115.stdout: During handling of the above exception, another exception occurred: 2022-05-14T01:51:46.619 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T01:51:46.619 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T01:51:46.619 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T01:51:46.619 INFO:journalctl@ceph.mon.c.smithi115.stdout: return f(*arg) 2022-05-14T01:51:46.620 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T01:51:46.620 INFO:journalctl@ceph.mon.c.smithi115.stdout: self._write_client_files(client_files, host) 2022-05-14T01:51:46.620 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T01:51:46.620 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T01:51:46.621 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T01:51:46.621 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T01:51:46.621 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T01:51:46.621 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.event_loop.get_result(coro) 2022-05-14T01:51:46.621 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T01:51:46.622 INFO:journalctl@ceph.mon.c.smithi115.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T01:51:46.622 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T01:51:46.622 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.__get_result() 2022-05-14T01:51:46.622 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T01:51:46.623 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise self._exception 2022-05-14T01:51:46.624 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T01:51:46.624 INFO:journalctl@ceph.mon.c.smithi115.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T01:51:46.624 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T01:51:46.624 INFO:journalctl@ceph.mon.c.smithi115.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T01:51:46.624 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T01:51:46.625 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise 2022-05-14T01:51:46.625 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T01:51:46.625 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.gen.throw(type, value, traceback) 2022-05-14T01:51:46.625 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T01:51:46.626 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OrchestratorError(msg) 2022-05-14T01:51:46.626 INFO:journalctl@ceph.mon.c.smithi115.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:51:46.626 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:51:46 smithi115 ceph-mon[78593]: pgmap v4672: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:51:47.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:51:46 smithi115 conmon[90393]: 2022-05-14T01:51:46.818+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:51:47.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:51:47 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:51:47.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:51:47 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:51:48.252 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:51:47 smithi115 conmon[90393]: 2022-05-14T01:51:47.856+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:51:48.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:51:48 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:51:48.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:51:48 smithi047 ceph-mon[77147]: pgmap v4673: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:51:48.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:51:48 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:51:48.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:51:48 smithi115 ceph-mon[78593]: pgmap v4673: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:51:49.258 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:51:48 smithi115 conmon[90393]: 2022-05-14T01:51:48.900+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:51:49.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:51:49 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:51:49.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:51:49 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:51:50.259 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:51:49 smithi115 conmon[90393]: 2022-05-14T01:51:49.889+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:51:50.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:51:50 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:51:50.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:51:50 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:51:50.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:51:50 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:51:50.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:51:50 smithi047 ceph-mon[77147]: pgmap v4674: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:51:50.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:51:50 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 8640 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:51:50.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:51:50 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:51:50.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:51:50 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:51:50.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:51:50 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:51:50.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:51:50 smithi115 ceph-mon[78593]: pgmap v4674: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:51:50.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:51:50 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 8640 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:51:51.266 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:51:50 smithi115 conmon[90393]: 2022-05-14T01:51:50.928+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:51:51.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:51:51 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:51:51.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:51:51 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:51:52.273 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:51:51 smithi115 conmon[90393]: 2022-05-14T01:51:51.890+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:51:52.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:51:52 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:51:52.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:51:52 smithi047 ceph-mon[77147]: pgmap v4675: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:51:52.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:51:52 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:51:52.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:51:52 smithi115 ceph-mon[78593]: pgmap v4675: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:51:53.279 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:51:52 smithi115 conmon[90393]: 2022-05-14T01:51:52.924+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:51:53.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:51:53 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:51:53.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:51:53 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:51:54.284 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:51:53 smithi115 conmon[90393]: 2022-05-14T01:51:53.919+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:51:54.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:51:54 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:51:54.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:51:54 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:51:54.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:51:54 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:51:54.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:51:54 smithi047 ceph-mon[77147]: pgmap v4676: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:51:54.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:51:54 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:51:54.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:51:54 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:51:54.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:51:54 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:51:54.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:51:54 smithi115 ceph-mon[78593]: pgmap v4676: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:51:55.284 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:51:54 smithi115 conmon[90393]: 2022-05-14T01:51:54.880+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:51:55.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:51:55 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:51:55.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:51:55 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 8645 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:51:55.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:51:55 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:51:55.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:51:55 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 8645 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:51:56.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:51:55 smithi115 conmon[90393]: 2022-05-14T01:51:55.847+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:51:56.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:51:56 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:51:56.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:51:56 smithi047 ceph-mon[77147]: pgmap v4677: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:51:56.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:51:56 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:51:56.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:51:56 smithi115 ceph-mon[78593]: pgmap v4677: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:51:57.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:51:56 smithi115 conmon[90393]: 2022-05-14T01:51:56.833+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:51:57.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:51:57 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:51:57.796 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:51:57 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:51:58.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:51:57 smithi115 conmon[90393]: 2022-05-14T01:51:57.784+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:51:58.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:51:58 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:51:58.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:51:58 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:51:58.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:51:58 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:51:58.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:51:58 smithi115 ceph-mon[78593]: pgmap v4678: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:51:58.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:51:58 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:51:58.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:51:58 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:51:58.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:51:58 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:51:58.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:51:58 smithi047 ceph-mon[77147]: pgmap v4678: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:51:59.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:51:58 smithi115 conmon[90393]: 2022-05-14T01:51:58.778+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:51:59.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:51:59 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:51:59.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:51:59 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:52:00.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:51:59 smithi115 conmon[90393]: 2022-05-14T01:51:59.769+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:52:00.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:52:00 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:52:00.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:52:00 smithi115 ceph-mon[78593]: pgmap v4679: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:52:00.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:52:00 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 8650 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:52:00.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:52:00 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:52:00.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:52:00 smithi047 ceph-mon[77147]: pgmap v4679: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:52:00.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:52:00 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 8650 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:52:01.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:52:00 smithi115 conmon[90393]: 2022-05-14T01:52:00.728+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:52:01.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:52:01 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:52:01.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:52:01 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:52:02.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:52:01 smithi115 conmon[90393]: 2022-05-14T01:52:01.722+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:52:02.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:52:02 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:52:02.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:52:02 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:52:02.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:52:02 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:52:02.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:52:02 smithi115 ceph-mon[78593]: pgmap v4680: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:52:02.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:52:02 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:52:02.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:52:02 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:52:02.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:52:02 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:52:02.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:52:02 smithi047 ceph-mon[77147]: pgmap v4680: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:52:03.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:52:02 smithi115 conmon[90393]: 2022-05-14T01:52:02.767+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:52:03.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:52:03 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:52:03.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:52:03 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:52:04.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:52:03 smithi115 conmon[90393]: 2022-05-14T01:52:03.817+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:52:04.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:52:04 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:52:04.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:52:04 smithi115 ceph-mon[78593]: pgmap v4681: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:52:04.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:52:04 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:52:04.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:52:04 smithi047 ceph-mon[77147]: pgmap v4681: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:52:05.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:52:04 smithi115 conmon[90393]: 2022-05-14T01:52:04.817+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:52:05.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:52:05 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:52:05.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:52:05 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 8655 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:52:05.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:52:05 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:52:05.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:52:05 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 8655 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:52:06.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:52:05 smithi115 conmon[90393]: 2022-05-14T01:52:05.840+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:52:06.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:52:06 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:52:06.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:52:06 smithi115 ceph-mon[78593]: pgmap v4682: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:52:06.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:52:06 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:52:06.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:52:06 smithi047 ceph-mon[77147]: pgmap v4682: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:52:07.331 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:52:06 smithi115 conmon[90393]: 2022-05-14T01:52:06.880+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:52:07.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:52:07 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:52:07.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:52:07 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:52:08.336 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:52:07 smithi115 conmon[90393]: 2022-05-14T01:52:07.913+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:52:08.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:52:08 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:52:08.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:52:08 smithi115 ceph-mon[78593]: pgmap v4683: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:52:08.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:52:08 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:52:08.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:52:08 smithi047 ceph-mon[77147]: pgmap v4683: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:52:09.337 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:52:08 smithi115 conmon[90393]: 2022-05-14T01:52:08.905+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:52:09.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:52:09 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:52:09.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:52:09 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:52:10.349 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:52:09 smithi115 conmon[90393]: 2022-05-14T01:52:09.922+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:52:10.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:52:10 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:52:10.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:52:10 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:52:10.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:52:10 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:52:10.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:52:10 smithi115 ceph-mon[78593]: pgmap v4684: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:52:10.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:52:10 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 8660 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:52:10.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:52:10 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:52:10.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:52:10 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:52:10.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:52:10 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:52:10.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:52:10 smithi047 ceph-mon[77147]: pgmap v4684: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:52:10.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:52:10 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 8660 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:52:11.348 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:52:10 smithi115 conmon[90393]: 2022-05-14T01:52:10.951+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:52:11.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:52:11 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:52:11.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:52:11 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:52:12.353 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:52:11 smithi115 conmon[90393]: 2022-05-14T01:52:11.911+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:52:12.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:52:12 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:52:12.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:52:12 smithi115 ceph-mon[78593]: pgmap v4685: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:52:12.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:52:12 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:52:12.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:52:12 smithi047 ceph-mon[77147]: pgmap v4685: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:52:13.355 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:52:12 smithi115 conmon[90393]: 2022-05-14T01:52:12.878+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:52:13.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:52:13 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:52:13.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:52:13 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:52:14.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:52:13 smithi115 conmon[90393]: 2022-05-14T01:52:13.844+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:52:14.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:52:14 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:52:14.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:52:14 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:52:14.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:52:14 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:52:14.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:52:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T01:52:14.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:52:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T01:52:14.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:52:14 smithi047 ceph-mon[77147]: pgmap v4686: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:52:14.837 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:52:14 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:52:14.837 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:52:14 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:52:14.837 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:52:14 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:52:14.838 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:52:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T01:52:14.838 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:52:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T01:52:14.838 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:52:14 smithi115 ceph-mon[78593]: pgmap v4686: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:52:15.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:52:14 smithi115 conmon[90393]: 2022-05-14T01:52:14.835+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:52:15.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:52:15 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:52:15.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:52:15 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 8665 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:52:15.823 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:52:15 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:52:15.823 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:52:15 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 8665 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:52:16.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:52:15 smithi115 conmon[90393]: 2022-05-14T01:52:15.822+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:52:16.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:52:16 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:52:16.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:52:16 smithi047 ceph-mon[77147]: pgmap v4687: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:52:16.836 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:52:16 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:52:16.836 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:52:16 smithi115 ceph-mon[78593]: pgmap v4687: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:52:17.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:52:16 smithi115 conmon[90393]: 2022-05-14T01:52:16.834+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:52:17.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:52:17 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:52:17.855 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:52:17 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:52:18.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:52:17 smithi115 conmon[90393]: 2022-05-14T01:52:17.853+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:52:18.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:52:18 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:52:18.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:52:18 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:52:18.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:52:18 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:52:18.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:52:18 smithi047 ceph-mon[77147]: pgmap v4688: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:52:18.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:52:18 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:52:18.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:52:18 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:52:18.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:52:18 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:52:18.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:52:18 smithi115 ceph-mon[78593]: pgmap v4688: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:52:19.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:52:18 smithi115 conmon[90393]: 2022-05-14T01:52:18.866+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:52:19.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:52:19 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:52:19.822 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:52:19 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:52:20.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:52:19 smithi115 conmon[90393]: 2022-05-14T01:52:19.821+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:52:20.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:52:20 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:52:20.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:52:20 smithi047 ceph-mon[77147]: pgmap v4689: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:52:20.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:52:20 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 8670 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:52:20.809 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:52:20 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:52:20.809 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:52:20 smithi115 ceph-mon[78593]: pgmap v4689: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:52:20.810 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:52:20 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 8670 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:52:21.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:52:20 smithi115 conmon[90393]: 2022-05-14T01:52:20.807+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:52:21.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:52:21 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:52:21.805 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:52:21 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:52:22.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:52:21 smithi115 conmon[90393]: 2022-05-14T01:52:21.804+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:52:22.764 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:52:22 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:52:22.765 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:52:22 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:52:22.765 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:52:22 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:52:22.765 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:52:22 smithi115 ceph-mon[78593]: pgmap v4690: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:52:22.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:52:22 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:52:22.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:52:22 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:52:22.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:52:22 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:52:22.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:52:22 smithi047 ceph-mon[77147]: pgmap v4690: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:52:23.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:52:22 smithi115 conmon[90393]: 2022-05-14T01:52:22.762+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:52:23.753 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:52:23 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:52:23.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:52:23 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:52:24.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:52:23 smithi115 conmon[90393]: 2022-05-14T01:52:23.752+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:52:24.775 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:52:24 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:52:24.775 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:52:24 smithi115 ceph-mon[78593]: pgmap v4691: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:52:24.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:52:24 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:52:24.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:52:24 smithi047 ceph-mon[77147]: pgmap v4691: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:52:25.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:52:24 smithi115 conmon[90393]: 2022-05-14T01:52:24.773+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:52:25.776 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:52:25 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:52:25.777 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:52:25 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 8675 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:52:25.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:52:25 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:52:25.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:52:25 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 8675 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:52:26.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:52:25 smithi115 conmon[90393]: 2022-05-14T01:52:25.774+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:52:26.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:52:26 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:52:26.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:52:26 smithi047 ceph-mon[77147]: pgmap v4692: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:52:26.822 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:52:26 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:52:26.822 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:52:26 smithi115 ceph-mon[78593]: pgmap v4692: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:52:27.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:52:26 smithi115 conmon[90393]: 2022-05-14T01:52:26.821+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:52:27.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:52:27 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:52:27.855 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:52:27 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:52:28.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:52:27 smithi115 conmon[90393]: 2022-05-14T01:52:27.854+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:52:28.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:52:28 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:52:28.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:52:28 smithi047 ceph-mon[77147]: pgmap v4693: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:52:28.815 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:52:28 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:52:28.815 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:52:28 smithi115 ceph-mon[78593]: pgmap v4693: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:52:29.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:52:28 smithi115 conmon[90393]: 2022-05-14T01:52:28.814+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:52:29.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:52:29 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:52:29.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:52:29 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:52:30.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:52:29 smithi115 conmon[90393]: 2022-05-14T01:52:29.858+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:52:30.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:52:30 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:52:30.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:52:30 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:52:30.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:52:30 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:52:30.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:52:30 smithi047 ceph-mon[77147]: pgmap v4694: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:52:30.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:52:30 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 8680 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:52:30.848 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:52:30 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:52:30.848 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:52:30 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:52:30.848 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:52:30 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:52:30.848 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:52:30 smithi115 ceph-mon[78593]: pgmap v4694: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:52:30.849 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:52:30 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 8680 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:52:31.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:52:30 smithi115 conmon[90393]: 2022-05-14T01:52:30.846+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:52:31.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:52:31 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:52:31.825 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:52:31 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:52:32.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:52:31 smithi115 conmon[90393]: 2022-05-14T01:52:31.824+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:52:32.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:52:32 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:52:32.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:52:32 smithi047 ceph-mon[77147]: pgmap v4695: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:52:32.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:52:32 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:52:32.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:52:32 smithi115 ceph-mon[78593]: pgmap v4695: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:52:33.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:52:32 smithi115 conmon[90393]: 2022-05-14T01:52:32.860+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:52:33.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:52:33 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:52:33.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:52:33 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:52:34.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:52:33 smithi115 conmon[90393]: 2022-05-14T01:52:33.879+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:52:34.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:52:34 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:52:34.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:52:34 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:52:34.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:52:34 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:52:34.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:52:34 smithi047 ceph-mon[77147]: pgmap v4696: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:52:34.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:52:34 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:52:34.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:52:34 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:52:34.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:52:34 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:52:34.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:52:34 smithi115 ceph-mon[78593]: pgmap v4696: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:52:35.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:52:34 smithi115 conmon[90393]: 2022-05-14T01:52:34.862+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:52:35.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:52:35 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:52:35.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:52:35 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 8685 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:52:35.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:52:35 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:52:35.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:52:35 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 8685 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:52:36.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:52:35 smithi115 conmon[90393]: 2022-05-14T01:52:35.884+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:52:36.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:52:36 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:52:36.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:52:36 smithi047 ceph-mon[77147]: pgmap v4697: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:52:36.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:52:36 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:52:36.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:52:36 smithi115 ceph-mon[78593]: pgmap v4697: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:52:37.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:52:36 smithi115 conmon[90393]: 2022-05-14T01:52:36.857+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:52:37.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:52:37 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:52:37.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:52:37 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:52:38.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:52:37 smithi115 conmon[90393]: 2022-05-14T01:52:37.878+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:52:38.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:52:38 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:52:38.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:52:38 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:52:38.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:52:38 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:52:38.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:52:38 smithi047 ceph-mon[77147]: pgmap v4698: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:52:38.850 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:52:38 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:52:38.851 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:52:38 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:52:38.851 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:52:38 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:52:38.851 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:52:38 smithi115 ceph-mon[78593]: pgmap v4698: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:52:39.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:52:38 smithi115 conmon[90393]: 2022-05-14T01:52:38.849+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:52:39.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:52:39 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:52:39.828 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:52:39 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:52:40.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:52:39 smithi115 conmon[90393]: 2022-05-14T01:52:39.827+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:52:40.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:52:40 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:52:40.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:52:40 smithi047 ceph-mon[77147]: pgmap v4699: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:52:40.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:52:40 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 8690 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:52:40.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:52:40 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:52:40.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:52:40 smithi115 ceph-mon[78593]: pgmap v4699: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:52:40.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:52:40 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 8690 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:52:41.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:52:40 smithi115 conmon[90393]: 2022-05-14T01:52:40.795+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:52:41.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:52:41 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:52:41.825 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:52:41 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:52:42.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:52:41 smithi115 conmon[90393]: 2022-05-14T01:52:41.824+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:52:42.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:52:42 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:52:42.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:52:42 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:52:42.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:52:42 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:52:42.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:52:42 smithi047 ceph-mon[77147]: pgmap v4700: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:52:42.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:52:42 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:52:42.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:52:42 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:52:42.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:52:42 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:52:42.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:52:42 smithi115 ceph-mon[78593]: pgmap v4700: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:52:43.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:52:42 smithi115 conmon[90393]: 2022-05-14T01:52:42.859+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:52:43.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:52:43 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:52:43.827 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:52:43 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:52:44.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:52:43 smithi115 conmon[90393]: 2022-05-14T01:52:43.826+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:52:44.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:52:44 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:52:44.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:52:44 smithi047 ceph-mon[77147]: pgmap v4701: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:52:44.804 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:52:44 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:52:44.804 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:52:44 smithi115 ceph-mon[78593]: pgmap v4701: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:52:45.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:52:44 smithi115 conmon[90393]: 2022-05-14T01:52:44.801+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:52:45.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:52:45 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:52:45.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:52:45 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 8695 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:52:45.825 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:52:45 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:52:45.826 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:52:45 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 8695 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:52:46.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:52:45 smithi115 conmon[90393]: 2022-05-14T01:52:45.823+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:52:46.291 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:52:46 smithi047 conmon[78521]: 2022-05-14T01:52:45.999+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:52:46.292 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:52:46 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T01:52:46.292 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:52:46 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:52:46.292 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:52:46 smithi047 conmon[78521]: yield 2022-05-14T01:52:46.293 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:52:46 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:52:46.293 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:52:46 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:52:46.293 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:52:46 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:52:46.293 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:52:46 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T01:52:46.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:52:46 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:52:46.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:52:46 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T01:52:46.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:52:46 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:52:46.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:52:46 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T01:52:46.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:52:46 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:52:46.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:52:46 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T01:52:46.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:52:46 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:52:46.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:52:46 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T01:52:46.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:52:46 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:52:46.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:52:46 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:52:46.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:52:46 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:52:46.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:52:46 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:52:46.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:52:46 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T01:52:46.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:52:46 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T01:52:46.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:52:46 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T01:52:46.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:52:46 smithi047 ceph-mon[77147]: pgmap v4702: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:52:46.824 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:52:46 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:52:46.824 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:52:46 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T01:52:46.825 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:52:46 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T01:52:46.825 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:52:46 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T01:52:46.826 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:52:46 smithi115 ceph-mon[78593]: pgmap v4702: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:52:47.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:52:46 smithi115 conmon[90393]: 2022-05-14T01:52:46.822+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:52:47.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:52:47 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:52:47.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:52:47 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:52:47.792 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T01:52:47.792 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:52:47.792 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T01:52:47.792 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:52:47.793 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:52:47.793 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:52:47.793 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T01:52:47.793 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:52:47.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T01:52:47.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:52:47.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T01:52:47.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:52:47.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T01:52:47.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:52:47.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T01:52:47.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:52:47.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:52:47.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:52:47.796 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:52:47 smithi047 ceph-mon[77147]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T01:52:47.834 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:52:47 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:52:47.834 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:52:47 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:52:47.834 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T01:52:47.834 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:52:47.835 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T01:52:47.835 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:52:47.835 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:52:47.835 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:52:47.835 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T01:52:47.836 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:52:47.836 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T01:52:47.836 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:52:47.836 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T01:52:47.837 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:52:47.837 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T01:52:47.837 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:52:47.837 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T01:52:47.838 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:52:47.838 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:52:47.838 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:52:47.838 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:52:47 smithi115 ceph-mon[78593]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T01:52:48.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:52:47 smithi115 conmon[90393]: 2022-05-14T01:52:47.833+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:52:48.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:52:48 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:52:48.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:52:48 smithi047 ceph-mon[77147]: pgmap v4703: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:52:48.806 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:52:48 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:52:48.806 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:52:48 smithi115 ceph-mon[78593]: pgmap v4703: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:52:49.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:52:48 smithi115 conmon[90393]: 2022-05-14T01:52:48.806+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:52:49.514 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:52:49 smithi047 conmon[78521]: 2022-05-14T01:52:49.071+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:52:49.514 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:52:49 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T01:52:49.514 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:52:49 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:52:49.515 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:52:49 smithi047 conmon[78521]: yield 2022-05-14T01:52:49.515 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:52:49 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:52:49.515 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:52:49 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:52:49.515 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:52:49 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:52:49.516 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:52:49 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T01:52:49.516 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:52:49 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:52:49.516 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:52:49 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T01:52:49.516 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:52:49 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:52:49.516 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:52:49 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T01:52:49.517 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:52:49 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:52:49.517 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:52:49 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T01:52:49.517 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:52:49 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:52:49.517 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:52:49 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T01:52:49.518 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:52:49 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:52:49.518 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:52:49 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:52:49.518 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:52:49 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:52:49.519 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:52:49 smithi047 conmon[78521]: 2022-05-14T01:52:49.072+0000 7f5345f84700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T01:52:49.519 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:52:49 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T01:52:49.519 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:52:49 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:52:49.519 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:52:49 smithi047 conmon[78521]: yield 2022-05-14T01:52:49.520 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:52:49 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:52:49.520 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:52:49 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:52:49.520 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:52:49 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:52:49.520 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:52:49 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T01:52:49.521 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:52:49 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:52:49.521 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:52:49 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T01:52:49.521 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:52:49 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:52:49.521 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:52:49 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T01:52:49.522 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:52:49 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:52:49.522 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:52:49 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T01:52:49.522 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:52:49 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:52:49.522 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:52:49 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T01:52:49.522 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:52:49 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:52:49.523 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:52:49 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:52:49.523 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:52:49 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:52:49.523 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:52:49 smithi047 conmon[78521]: 2022-05-14T01:52:49.523 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:52:49 smithi047 conmon[78521]: During handling of the above exception, another exception occurred: 2022-05-14T01:52:49.523 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:52:49 smithi047 conmon[78521]: 2022-05-14T01:52:49.524 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:52:49 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T01:52:49.524 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:52:49 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T01:52:49.524 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:52:49 smithi047 conmon[78521]: return f(*arg) 2022-05-14T01:52:49.524 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:52:49 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T01:52:49.525 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:52:49 smithi047 conmon[78521]: self._write_client_files(client_files, host) 2022-05-14T01:52:49.525 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:52:49 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T01:52:49.525 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:52:49 smithi047 conmon[78521]: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T01:52:49.525 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:52:49 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T01:52:49.525 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:52:49 smithi047 conmon[78521]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T01:52:49.526 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:52:49 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T01:52:49.526 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:52:49 smithi047 conmon[78521]: return self.event_loop.get_result(coro) 2022-05-14T01:52:49.526 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:52:49 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T01:52:49.526 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:52:49 smithi047 conmon[78521]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T01:52:49.526 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:52:49 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T01:52:49.527 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:52:49 smithi047 conmon[78521]: return self.__get_result() 2022-05-14T01:52:49.527 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:52:49 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T01:52:49.527 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:52:49 smithi047 conmon[78521]: raise self._exception 2022-05-14T01:52:49.527 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:52:49 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T01:52:49.527 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:52:49 smithi047 conmon[78521]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T01:52:49.528 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:52:49 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T01:52:49.528 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:52:49 smithi047 conmon[78521]: conn = await self._remote_connection(host, addr) 2022-05-14T01:52:49.528 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:52:49 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T01:52:49.528 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:52:49 smithi047 conmon[78521]: raise 2022-05-14T01:52:49.528 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:52:49 smithi047 conmon[78521]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T01:52:49.529 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:52:49 smithi047 conmon[78521]: self.gen.throw(type, value, traceback) 2022-05-14T01:52:49.529 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:52:49 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T01:52:49.529 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:52:49 smithi047 conmon[78521]: raise OrchestratorError(msg) 2022-05-14T01:52:49.529 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:52:49 smithi047 conmon[78521]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:52:49.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:52:49 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:52:49.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:52:49 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:52:49.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:52:49 smithi115 conmon[90393]: 2022-05-14T01:52:49.764+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:52:50.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:52:50 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:52:50.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:52:50 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:52:50.792 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T01:52:50.792 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:52:50.793 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T01:52:50.793 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:52:50.793 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:52:50.793 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:52:50.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T01:52:50.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:52:50.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T01:52:50.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:52:50.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T01:52:50.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:52:50.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T01:52:50.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:52:50.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T01:52:50.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:52:50.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:52:50.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:52:50.796 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:52:50 smithi047 ceph-mon[77147]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T01:52:50.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T01:52:50.797 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:52:50.797 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T01:52:50.797 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:52:50.797 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:52:50.797 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:52:50.798 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T01:52:50.798 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:52:50.798 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T01:52:50.798 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:52:50.798 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T01:52:50.799 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:52:50.799 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T01:52:50.799 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:52:50.799 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T01:52:50.800 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:52:50.800 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:52:50.800 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:52:50.800 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T01:52:50.800 INFO:journalctl@ceph.mon.b.smithi047.stdout: During handling of the above exception, another exception occurred: 2022-05-14T01:52:50.801 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T01:52:50.801 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T01:52:50.801 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T01:52:50.801 INFO:journalctl@ceph.mon.b.smithi047.stdout: return f(*arg) 2022-05-14T01:52:50.802 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T01:52:50.802 INFO:journalctl@ceph.mon.b.smithi047.stdout: self._write_client_files(client_files, host) 2022-05-14T01:52:50.802 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T01:52:50.802 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T01:52:50.802 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T01:52:50.803 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T01:52:50.803 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T01:52:50.803 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.event_loop.get_result(coro) 2022-05-14T01:52:50.803 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T01:52:50.804 INFO:journalctl@ceph.mon.b.smithi047.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T01:52:50.804 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T01:52:50.804 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.__get_result() 2022-05-14T01:52:50.805 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T01:52:50.805 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise self._exception 2022-05-14T01:52:50.805 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T01:52:50.805 INFO:journalctl@ceph.mon.b.smithi047.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T01:52:50.806 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T01:52:50.806 INFO:journalctl@ceph.mon.b.smithi047.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T01:52:50.806 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T01:52:50.806 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise 2022-05-14T01:52:50.806 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T01:52:50.807 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.gen.throw(type, value, traceback) 2022-05-14T01:52:50.807 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T01:52:50.807 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OrchestratorError(msg) 2022-05-14T01:52:50.807 INFO:journalctl@ceph.mon.b.smithi047.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:52:50.807 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:52:50 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:52:50.808 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:52:50 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:52:50.808 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:52:50 smithi047 ceph-mon[77147]: pgmap v4704: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:52:50.808 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:52:50 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 8700 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:52:50.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:52:50 smithi115 conmon[90393]: 2022-05-14T01:52:50.752+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:52:50.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:52:50 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:52:50.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:52:50 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:52:50.859 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T01:52:50.859 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:52:50.860 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T01:52:50.860 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:52:50.860 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:52:50.860 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:52:50.860 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T01:52:50.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:52:50.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T01:52:50.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:52:50.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T01:52:50.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:52:50.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T01:52:50.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:52:50.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T01:52:50.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:52:50.863 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:52:50.863 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:52:50.863 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:52:50 smithi115 ceph-mon[78593]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T01:52:50.863 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T01:52:50.864 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:52:50.864 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T01:52:50.864 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:52:50.865 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:52:50.865 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:52:50.865 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T01:52:50.866 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:52:50.866 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T01:52:50.866 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:52:50.866 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T01:52:50.866 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:52:50.867 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T01:52:50.867 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:52:50.867 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T01:52:50.867 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:52:50.867 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:52:50.868 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:52:50.868 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T01:52:50.868 INFO:journalctl@ceph.mon.c.smithi115.stdout: During handling of the above exception, another exception occurred: 2022-05-14T01:52:50.868 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T01:52:50.869 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T01:52:50.869 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T01:52:50.869 INFO:journalctl@ceph.mon.c.smithi115.stdout: return f(*arg) 2022-05-14T01:52:50.869 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T01:52:50.869 INFO:journalctl@ceph.mon.c.smithi115.stdout: self._write_client_files(client_files, host) 2022-05-14T01:52:50.870 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T01:52:50.870 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T01:52:50.870 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T01:52:50.870 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T01:52:50.870 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T01:52:50.871 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.event_loop.get_result(coro) 2022-05-14T01:52:50.871 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T01:52:50.871 INFO:journalctl@ceph.mon.c.smithi115.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T01:52:50.871 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T01:52:50.871 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.__get_result() 2022-05-14T01:52:50.872 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T01:52:50.872 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise self._exception 2022-05-14T01:52:50.872 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T01:52:50.872 INFO:journalctl@ceph.mon.c.smithi115.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T01:52:50.873 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T01:52:50.873 INFO:journalctl@ceph.mon.c.smithi115.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T01:52:50.873 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T01:52:50.873 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise 2022-05-14T01:52:50.873 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T01:52:50.874 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.gen.throw(type, value, traceback) 2022-05-14T01:52:50.874 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T01:52:50.874 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OrchestratorError(msg) 2022-05-14T01:52:50.874 INFO:journalctl@ceph.mon.c.smithi115.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:52:50.875 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:52:50 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:52:50.875 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:52:50 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:52:50.875 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:52:50 smithi115 ceph-mon[78593]: pgmap v4704: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:52:50.875 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:52:50 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 8700 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:52:51.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:52:51 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:52:51.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:52:51 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:52:51.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:52:51 smithi115 conmon[90393]: 2022-05-14T01:52:51.751+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:52:52.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:52:52 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:52:52.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:52:52 smithi047 ceph-mon[77147]: pgmap v4705: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:52:52.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:52:52 smithi115 conmon[90393]: 2022-05-14T01:52:52.758+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:52:52.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:52:52 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:52:52.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:52:52 smithi115 ceph-mon[78593]: pgmap v4705: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:52:53.784 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:52:53 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:52:53.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:52:53 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:52:54.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:52:53 smithi115 conmon[90393]: 2022-05-14T01:52:53.782+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:52:54.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:52:54 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:52:54.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:52:54 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:52:54.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:52:54 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:52:54.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:52:54 smithi047 ceph-mon[77147]: pgmap v4706: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:52:54.800 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:52:54 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:52:54.800 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:52:54 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:52:54.800 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:52:54 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:52:54.801 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:52:54 smithi115 ceph-mon[78593]: pgmap v4706: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:52:55.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:52:54 smithi115 conmon[90393]: 2022-05-14T01:52:54.798+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:52:55.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:52:55 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:52:55.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:52:55 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 8705 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:52:55.819 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:52:55 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:52:55.820 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:52:55 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 8705 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:52:56.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:52:55 smithi115 conmon[90393]: 2022-05-14T01:52:55.818+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:52:56.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:52:56 smithi115 conmon[90393]: 2022-05-14T01:52:56.780+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:52:56.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:52:56 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:52:56.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:52:56 smithi115 ceph-mon[78593]: pgmap v4707: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:52:57.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:52:56 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:52:57.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:52:56 smithi047 ceph-mon[77147]: pgmap v4707: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:52:57.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:52:57 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:52:57.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:52:57 smithi115 conmon[90393]: 2022-05-14T01:52:57.741+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:52:58.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:52:57 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:52:58.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:52:58 smithi115 conmon[90393]: 2022-05-14T01:52:58.700+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:52:58.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:52:58 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:52:58.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:52:58 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:52:58.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:52:58 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:52:58.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:52:58 smithi115 ceph-mon[78593]: pgmap v4708: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:52:59.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:52:58 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:52:59.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:52:58 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:52:59.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:52:58 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:52:59.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:52:58 smithi047 ceph-mon[77147]: pgmap v4708: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:52:59.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:52:59 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:52:59.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:52:59 smithi115 conmon[90393]: 2022-05-14T01:52:59.697+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:53:00.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:52:59 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:53:00.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:53:00 smithi115 conmon[90393]: 2022-05-14T01:53:00.660+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:53:00.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:53:00 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:53:00.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:53:00 smithi115 ceph-mon[78593]: pgmap v4709: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:53:00.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:53:00 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 8710 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:53:01.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:53:00 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:53:01.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:53:00 smithi047 ceph-mon[77147]: pgmap v4709: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:53:01.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:53:00 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 8710 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:53:01.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:53:01 smithi115 conmon[90393]: 2022-05-14T01:53:01.695+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:53:01.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:53:01 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:53:01.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:53:01 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:53:02.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:53:01 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:53:02.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:53:01 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:53:02.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:53:02 smithi115 conmon[90393]: 2022-05-14T01:53:02.686+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:53:02.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:53:02 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:53:02.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:53:02 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:53:02.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:53:02 smithi115 ceph-mon[78593]: pgmap v4710: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:53:03.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:53:02 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:53:03.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:53:02 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:53:03.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:53:02 smithi047 ceph-mon[77147]: pgmap v4710: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:53:03.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:53:03 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:53:03.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:53:03 smithi115 conmon[90393]: 2022-05-14T01:53:03.705+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:53:04.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:53:03 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:53:04.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:53:04 smithi115 conmon[90393]: 2022-05-14T01:53:04.711+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:53:04.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:53:04 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:53:04.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:53:04 smithi115 ceph-mon[78593]: pgmap v4711: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:53:05.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:53:04 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:53:05.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:53:04 smithi047 ceph-mon[77147]: pgmap v4711: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:53:05.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:53:05 smithi115 conmon[90393]: 2022-05-14T01:53:05.755+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:53:06.013 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:53:05 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:53:06.013 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:53:05 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 8715 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:53:06.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:53:05 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:53:06.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:53:05 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 8715 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:53:06.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:53:06 smithi115 conmon[90393]: 2022-05-14T01:53:06.775+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:53:06.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:53:06 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:53:06.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:53:06 smithi115 ceph-mon[78593]: pgmap v4712: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:53:07.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:53:06 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:53:07.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:53:06 smithi047 ceph-mon[77147]: pgmap v4712: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:53:07.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:53:07 smithi115 conmon[90393]: 2022-05-14T01:53:07.745+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:53:07.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:53:07 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:53:08.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:53:07 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:53:08.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:53:08 smithi115 conmon[90393]: 2022-05-14T01:53:08.752+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:53:08.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:53:08 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:53:08.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:53:08 smithi115 ceph-mon[78593]: pgmap v4713: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:53:09.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:53:08 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:53:09.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:53:08 smithi047 ceph-mon[77147]: pgmap v4713: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:53:09.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:53:09 smithi115 conmon[90393]: 2022-05-14T01:53:09.786+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:53:09.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:53:09 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:53:10.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:53:09 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:53:10.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:53:10 smithi115 conmon[90393]: 2022-05-14T01:53:10.806+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:53:10.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:53:10 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:53:10.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:53:10 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:53:10.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:53:10 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:53:10.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:53:10 smithi115 ceph-mon[78593]: pgmap v4714: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:53:10.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:53:10 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 8720 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:53:11.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:53:10 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:53:11.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:53:10 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:53:11.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:53:10 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:53:11.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:53:10 smithi047 ceph-mon[77147]: pgmap v4714: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:53:11.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:53:10 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 8720 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:53:11.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:53:11 smithi115 conmon[90393]: 2022-05-14T01:53:11.820+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:53:11.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:53:11 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:53:12.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:53:11 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:53:13.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:53:12 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:53:13.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:53:12 smithi047 ceph-mon[77147]: pgmap v4715: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:53:13.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:53:12 smithi115 conmon[90393]: 2022-05-14T01:53:12.847+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:53:13.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:53:12 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:53:13.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:53:12 smithi115 ceph-mon[78593]: pgmap v4715: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:53:14.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:53:13 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:53:14.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:53:13 smithi115 conmon[90393]: 2022-05-14T01:53:13.819+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:53:14.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:53:13 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:53:15.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:53:14 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:53:15.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:53:14 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:53:15.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:53:14 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:53:15.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:53:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T01:53:15.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:53:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T01:53:15.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:53:14 smithi047 ceph-mon[77147]: pgmap v4716: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:53:15.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:53:14 smithi115 conmon[90393]: 2022-05-14T01:53:14.819+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:53:15.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:53:14 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:53:15.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:53:14 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:53:15.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:53:14 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:53:15.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:53:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T01:53:15.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:53:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T01:53:15.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:53:14 smithi115 ceph-mon[78593]: pgmap v4716: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:53:16.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:53:15 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:53:16.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:53:15 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 8725 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:53:16.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:53:15 smithi115 conmon[90393]: 2022-05-14T01:53:15.813+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:53:16.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:53:15 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:53:16.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:53:15 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 8725 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:53:17.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:53:16 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:53:17.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:53:16 smithi047 ceph-mon[77147]: pgmap v4717: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:53:17.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:53:16 smithi115 conmon[90393]: 2022-05-14T01:53:16.804+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:53:17.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:53:16 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:53:17.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:53:16 smithi115 ceph-mon[78593]: pgmap v4717: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:53:18.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:53:17 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:53:18.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:53:17 smithi115 conmon[90393]: 2022-05-14T01:53:17.825+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:53:18.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:53:17 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:53:19.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:53:18 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:53:19.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:53:18 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:53:19.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:53:18 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:53:19.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:53:18 smithi047 ceph-mon[77147]: pgmap v4718: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:53:19.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:53:18 smithi115 conmon[90393]: 2022-05-14T01:53:18.781+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:53:19.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:53:18 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:53:19.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:53:18 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:53:19.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:53:18 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:53:19.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:53:18 smithi115 ceph-mon[78593]: pgmap v4718: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:53:20.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:53:19 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:53:20.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:53:19 smithi115 conmon[90393]: 2022-05-14T01:53:19.772+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:53:20.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:53:19 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:53:21.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:53:20 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:53:21.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:53:20 smithi047 ceph-mon[77147]: pgmap v4719: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:53:21.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:53:20 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 8730 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:53:21.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:53:20 smithi115 conmon[90393]: 2022-05-14T01:53:20.751+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:53:21.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:53:20 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:53:21.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:53:20 smithi115 ceph-mon[78593]: pgmap v4719: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:53:21.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:53:20 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 8730 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:53:22.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:53:21 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:53:22.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:53:21 smithi115 conmon[90393]: 2022-05-14T01:53:21.793+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:53:22.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:53:21 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:53:23.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:53:22 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:53:23.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:53:22 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:53:23.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:53:22 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:53:23.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:53:22 smithi047 ceph-mon[77147]: pgmap v4720: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:53:23.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:53:22 smithi115 conmon[90393]: 2022-05-14T01:53:22.805+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:53:23.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:53:22 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:53:23.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:53:22 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:53:23.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:53:22 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:53:23.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:53:22 smithi115 ceph-mon[78593]: pgmap v4720: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:53:24.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:53:23 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:53:24.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:53:23 smithi115 conmon[90393]: 2022-05-14T01:53:23.760+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:53:24.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:53:23 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:53:25.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:53:24 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:53:25.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:53:24 smithi047 ceph-mon[77147]: pgmap v4721: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:53:25.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:53:24 smithi115 conmon[90393]: 2022-05-14T01:53:24.724+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:53:25.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:53:24 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:53:25.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:53:24 smithi115 ceph-mon[78593]: pgmap v4721: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:53:26.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:53:25 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:53:26.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:53:25 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 8735 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:53:26.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:53:25 smithi115 conmon[90393]: 2022-05-14T01:53:25.692+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:53:26.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:53:25 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:53:26.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:53:25 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 8735 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:53:27.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:53:26 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:53:27.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:53:26 smithi047 ceph-mon[77147]: pgmap v4722: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:53:27.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:53:26 smithi115 conmon[90393]: 2022-05-14T01:53:26.737+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:53:27.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:53:26 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:53:27.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:53:26 smithi115 ceph-mon[78593]: pgmap v4722: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:53:28.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:53:27 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:53:28.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:53:27 smithi115 conmon[90393]: 2022-05-14T01:53:27.753+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:53:28.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:53:27 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:53:29.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:53:28 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:53:29.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:53:28 smithi047 ceph-mon[77147]: pgmap v4723: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:53:29.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:53:28 smithi115 conmon[90393]: 2022-05-14T01:53:28.731+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:53:29.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:53:28 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:53:29.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:53:28 smithi115 ceph-mon[78593]: pgmap v4723: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:53:30.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:53:29 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:53:30.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:53:29 smithi115 conmon[90393]: 2022-05-14T01:53:29.768+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:53:30.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:53:29 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:53:31.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:53:30 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:53:31.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:53:30 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:53:31.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:53:30 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:53:31.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:53:30 smithi047 ceph-mon[77147]: pgmap v4724: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:53:31.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:53:30 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 8740 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:53:31.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:53:30 smithi115 conmon[90393]: 2022-05-14T01:53:30.795+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:53:31.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:53:30 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:53:31.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:53:30 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:53:31.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:53:30 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:53:31.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:53:30 smithi115 ceph-mon[78593]: pgmap v4724: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:53:31.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:53:30 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 8740 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:53:32.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:53:31 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:53:32.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:53:31 smithi115 conmon[90393]: 2022-05-14T01:53:31.832+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:53:32.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:53:31 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:53:33.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:53:32 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:53:33.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:53:32 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:53:33.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:53:32 smithi047 ceph-mon[77147]: pgmap v4725: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:53:33.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:53:32 smithi115 conmon[90393]: 2022-05-14T01:53:32.845+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:53:33.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:53:32 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:53:33.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:53:32 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:53:33.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:53:32 smithi115 ceph-mon[78593]: pgmap v4725: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:53:34.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:53:33 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:53:34.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:53:33 smithi115 conmon[90393]: 2022-05-14T01:53:33.872+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:53:34.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:53:33 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:53:35.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:53:34 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:53:35.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:53:34 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:53:35.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:53:34 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:53:35.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:53:34 smithi047 ceph-mon[77147]: pgmap v4726: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:53:35.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:53:34 smithi115 conmon[90393]: 2022-05-14T01:53:34.844+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:53:35.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:53:34 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:53:35.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:53:34 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:53:35.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:53:34 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:53:35.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:53:34 smithi115 ceph-mon[78593]: pgmap v4726: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:53:36.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:53:35 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:53:36.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:53:35 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 8745 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:53:36.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:53:35 smithi115 conmon[90393]: 2022-05-14T01:53:35.868+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:53:36.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:53:35 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:53:36.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:53:35 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 8745 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:53:37.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:53:36 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:53:37.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:53:36 smithi047 ceph-mon[77147]: pgmap v4727: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:53:37.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:53:36 smithi115 conmon[90393]: 2022-05-14T01:53:36.909+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:53:37.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:53:36 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:53:37.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:53:36 smithi115 ceph-mon[78593]: pgmap v4727: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:53:38.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:53:37 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:53:38.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:53:37 smithi115 conmon[90393]: 2022-05-14T01:53:37.945+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:53:38.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:53:37 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:53:39.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:53:38 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:53:39.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:53:38 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:53:39.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:53:38 smithi047 ceph-mon[77147]: pgmap v4728: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:53:39.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:53:38 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:53:39.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:53:38 smithi115 conmon[90393]: 2022-05-14T01:53:38.897+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:53:39.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:53:38 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:53:39.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:53:38 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:53:39.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:53:38 smithi115 ceph-mon[78593]: pgmap v4728: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:53:39.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:53:38 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:53:40.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:53:39 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:53:40.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:53:39 smithi115 conmon[90393]: 2022-05-14T01:53:39.888+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:53:40.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:53:39 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:53:41.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:53:40 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:53:41.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:53:40 smithi047 ceph-mon[77147]: pgmap v4729: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:53:41.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:53:40 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 8750 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:53:41.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:53:40 smithi115 conmon[90393]: 2022-05-14T01:53:40.899+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:53:41.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:53:40 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:53:41.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:53:40 smithi115 ceph-mon[78593]: pgmap v4729: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:53:41.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:53:40 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 8750 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:53:42.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:53:41 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:53:42.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:53:41 smithi115 conmon[90393]: 2022-05-14T01:53:41.919+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:53:42.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:53:41 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:53:43.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:53:42 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:53:43.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:53:42 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:53:43.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:53:42 smithi047 ceph-mon[77147]: pgmap v4730: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:53:43.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:53:42 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:53:43.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:53:42 smithi115 conmon[90393]: 2022-05-14T01:53:42.877+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:53:43.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:53:42 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:53:43.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:53:42 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:53:43.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:53:42 smithi115 ceph-mon[78593]: pgmap v4730: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:53:43.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:53:42 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:53:44.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:53:43 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:53:44.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:53:43 smithi115 conmon[90393]: 2022-05-14T01:53:43.888+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:53:44.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:53:43 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:53:45.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:53:44 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:53:45.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:53:44 smithi047 ceph-mon[77147]: pgmap v4731: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:53:45.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:53:44 smithi115 conmon[90393]: 2022-05-14T01:53:44.920+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:53:45.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:53:44 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:53:45.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:53:44 smithi115 ceph-mon[78593]: pgmap v4731: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:53:46.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:53:45 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:53:46.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:53:45 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 8755 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:53:46.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:53:45 smithi115 conmon[90393]: 2022-05-14T01:53:45.920+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:53:46.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:53:45 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:53:46.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:53:45 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 8755 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:53:47.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:53:46 smithi047 ceph-mon[77147]: pgmap v4732: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:53:47.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:53:46 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:53:47.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:53:46 smithi115 conmon[90393]: 2022-05-14T01:53:46.900+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:53:47.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:53:46 smithi115 ceph-mon[78593]: pgmap v4732: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:53:47.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:53:46 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:53:48.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:53:47 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:53:48.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:53:47 smithi115 conmon[90393]: 2022-05-14T01:53:47.902+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:53:48.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:53:47 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:53:49.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:53:48 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:53:49.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:53:48 smithi047 ceph-mon[77147]: pgmap v4733: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:53:49.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:53:48 smithi115 conmon[90393]: 2022-05-14T01:53:48.904+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:53:49.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:53:48 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:53:49.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:53:48 smithi115 ceph-mon[78593]: pgmap v4733: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:53:49.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:53:49 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:53:49.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:53:49 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:53:49.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:53:49 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:53:49.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:53:49 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T01:53:49.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:53:49 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T01:53:49.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:53:49 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T01:53:50.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:53:49 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:53:50.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:53:49 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:53:50.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:53:49 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:53:50.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:53:49 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T01:53:50.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:53:49 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T01:53:50.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:53:49 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T01:53:50.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:53:49 smithi115 conmon[90393]: 2022-05-14T01:53:49.880+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:53:51.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:53:50 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:53:51.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:53:50 smithi047 ceph-mon[77147]: pgmap v4734: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:53:51.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:53:50 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 8760 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:53:51.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:53:50 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:53:51.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:53:50 smithi115 ceph-mon[78593]: pgmap v4734: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:53:51.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:53:50 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 8760 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:53:51.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:53:50 smithi115 conmon[90393]: 2022-05-14T01:53:50.863+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:53:52.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:53:51 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:53:52.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:53:51 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:53:52.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:53:51 smithi115 conmon[90393]: 2022-05-14T01:53:51.890+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:53:52.541 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:53:52 smithi047 conmon[78521]: 2022-05-14T01:53:52.176+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:53:52.542 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:53:52 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T01:53:52.542 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:53:52 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:53:52.542 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:53:52 smithi047 conmon[78521]: yield 2022-05-14T01:53:52.542 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:53:52 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:53:52.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:53:52 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:53:52.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:53:52 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:53:52.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:53:52 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T01:53:52.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:53:52 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:53:52.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:53:52 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T01:53:52.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:53:52 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:53:52.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:53:52 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T01:53:52.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:53:52 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:53:52.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:53:52 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T01:53:52.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:53:52 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:53:52.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:53:52 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T01:53:52.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:53:52 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:53:52.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:53:52 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:53:52.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:53:52 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:53:53.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:53:52 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:53:53.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:53:52 smithi047 ceph-mon[77147]: pgmap v4735: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:53:53.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:53:52 smithi115 conmon[90393]: 2022-05-14T01:53:52.863+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:53:53.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:53:52 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:53:53.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:53:52 smithi115 ceph-mon[78593]: pgmap v4735: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:53:54.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:53:53 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:53:54.042 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T01:53:54.042 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:53:54.042 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T01:53:54.042 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:53:54.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:53:54.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:53:54.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T01:53:54.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:53:54.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T01:53:54.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:53:54.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T01:53:54.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:53:54.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T01:53:54.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:53:54.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T01:53:54.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:53:54.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:53:54.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:53:54.046 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:53:53 smithi047 ceph-mon[77147]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T01:53:54.046 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:53:53 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:53:54.046 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:53:53 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:53:54.047 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:53:53 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:53:54.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:53:53 smithi115 conmon[90393]: 2022-05-14T01:53:53.818+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:53:54.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:53:53 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:53:54.108 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T01:53:54.108 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:53:54.109 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T01:53:54.109 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:53:54.109 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:53:54.109 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:53:54.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T01:53:54.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:53:54.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T01:53:54.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:53:54.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T01:53:54.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:53:54.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T01:53:54.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:53:54.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T01:53:54.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:53:54.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:53:54.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:53:54.112 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:53:53 smithi115 ceph-mon[78593]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T01:53:54.112 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:53:53 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:53:54.112 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:53:53 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:53:54.112 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:53:53 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:53:55.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:53:54 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:53:55.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:53:54 smithi047 ceph-mon[77147]: pgmap v4736: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:53:55.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:53:54 smithi115 conmon[90393]: 2022-05-14T01:53:54.773+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:53:55.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:53:54 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:53:55.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:53:54 smithi115 ceph-mon[78593]: pgmap v4736: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:53:55.542 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:53:55 smithi047 conmon[78521]: 2022-05-14T01:53:55.248+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:53:55.542 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:53:55 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T01:53:55.542 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:53:55 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:53:55.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:53:55 smithi047 conmon[78521]: yield 2022-05-14T01:53:55.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:53:55 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:53:55.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:53:55 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:53:55.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:53:55 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:53:55.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:53:55 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T01:53:55.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:53:55 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:53:55.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:53:55 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T01:53:55.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:53:55 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:53:55.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:53:55 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T01:53:55.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:53:55 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:53:55.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:53:55 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T01:53:55.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:53:55 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:53:55.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:53:55 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T01:53:55.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:53:55 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:53:55.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:53:55 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:53:55.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:53:55 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:53:55.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:53:55 smithi047 conmon[78521]: 2022-05-14T01:53:55.249+0000 7f5340779700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T01:53:55.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:53:55 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T01:53:55.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:53:55 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:53:55.547 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:53:55 smithi047 conmon[78521]: yield 2022-05-14T01:53:55.547 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:53:55 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:53:55.547 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:53:55 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:53:55.547 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:53:55 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:53:55.548 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:53:55 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T01:53:55.548 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:53:55 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:53:55.548 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:53:55 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T01:53:55.548 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:53:55 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:53:55.548 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:53:55 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T01:53:55.549 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:53:55 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:53:55.549 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:53:55 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T01:53:55.549 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:53:55 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:53:55.549 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:53:55 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T01:53:55.550 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:53:55 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:53:55.550 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:53:55 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:53:55.550 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:53:55 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:53:55.550 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:53:55 smithi047 conmon[78521]: 2022-05-14T01:53:55.550 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:53:55 smithi047 conmon[78521]: During handling of the above exception, another exception occurred: 2022-05-14T01:53:55.551 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:53:55 smithi047 conmon[78521]: 2022-05-14T01:53:55.551 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:53:55 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T01:53:55.551 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:53:55 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T01:53:55.551 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:53:55 smithi047 conmon[78521]: return f(*arg) 2022-05-14T01:53:55.552 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:53:55 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T01:53:55.552 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:53:55 smithi047 conmon[78521]: self._write_client_files(client_files, host) 2022-05-14T01:53:55.552 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:53:55 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T01:53:55.552 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:53:55 smithi047 conmon[78521]: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T01:53:55.552 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:53:55 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T01:53:55.553 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:53:55 smithi047 conmon[78521]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T01:53:55.553 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:53:55 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T01:53:55.553 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:53:55 smithi047 conmon[78521]: return self.event_loop.get_result(coro) 2022-05-14T01:53:55.553 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:53:55 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T01:53:55.553 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:53:55 smithi047 conmon[78521]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T01:53:55.554 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:53:55 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T01:53:55.554 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:53:55 smithi047 conmon[78521]: return self.__get_result() 2022-05-14T01:53:55.554 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:53:55 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T01:53:55.554 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:53:55 smithi047 conmon[78521]: raise self._exception 2022-05-14T01:53:55.555 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:53:55 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T01:53:55.555 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:53:55 smithi047 conmon[78521]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T01:53:55.555 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:53:55 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T01:53:55.555 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:53:55 smithi047 conmon[78521]: conn = await self._remote_connection(host, addr) 2022-05-14T01:53:55.555 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:53:55 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T01:53:55.556 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:53:55 smithi047 conmon[78521]: raise 2022-05-14T01:53:55.556 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:53:55 smithi047 conmon[78521]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T01:53:55.556 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:53:55 smithi047 conmon[78521]: self.gen.throw(type, value, traceback) 2022-05-14T01:53:55.556 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:53:55 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T01:53:55.556 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:53:55 smithi047 conmon[78521]: raise OrchestratorError(msg) 2022-05-14T01:53:55.557 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:53:55 smithi047 conmon[78521]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:53:56.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:53:55 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:53:56.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:53:55 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 8765 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:53:56.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:53:55 smithi115 conmon[90393]: 2022-05-14T01:53:55.738+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:53:56.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:53:55 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:53:56.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:53:55 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 8765 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:53:57.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:53:56 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:53:57.042 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T01:53:57.042 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:53:57.042 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T01:53:57.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:53:57.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:53:57.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:53:57.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T01:53:57.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:53:57.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T01:53:57.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:53:57.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T01:53:57.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:53:57.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T01:53:57.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:53:57.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T01:53:57.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:53:57.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:53:57.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:53:57.046 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:53:56 smithi047 ceph-mon[77147]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T01:53:57.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T01:53:57.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:53:57.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T01:53:57.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:53:57.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:53:57.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:53:57.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T01:53:57.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:53:57.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T01:53:57.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:53:57.049 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T01:53:57.049 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:53:57.049 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T01:53:57.049 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:53:57.050 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T01:53:57.050 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:53:57.050 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:53:57.050 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:53:57.051 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T01:53:57.051 INFO:journalctl@ceph.mon.b.smithi047.stdout: During handling of the above exception, another exception occurred: 2022-05-14T01:53:57.051 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T01:53:57.051 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T01:53:57.051 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T01:53:57.052 INFO:journalctl@ceph.mon.b.smithi047.stdout: return f(*arg) 2022-05-14T01:53:57.052 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T01:53:57.052 INFO:journalctl@ceph.mon.b.smithi047.stdout: self._write_client_files(client_files, host) 2022-05-14T01:53:57.052 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T01:53:57.052 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T01:53:57.053 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T01:53:57.053 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T01:53:57.053 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T01:53:57.053 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.event_loop.get_result(coro) 2022-05-14T01:53:57.054 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T01:53:57.054 INFO:journalctl@ceph.mon.b.smithi047.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T01:53:57.054 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T01:53:57.054 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.__get_result() 2022-05-14T01:53:57.055 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T01:53:57.055 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise self._exception 2022-05-14T01:53:57.055 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T01:53:57.055 INFO:journalctl@ceph.mon.b.smithi047.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T01:53:57.055 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T01:53:57.056 INFO:journalctl@ceph.mon.b.smithi047.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T01:53:57.056 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T01:53:57.056 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise 2022-05-14T01:53:57.056 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T01:53:57.057 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.gen.throw(type, value, traceback) 2022-05-14T01:53:57.057 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T01:53:57.057 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OrchestratorError(msg) 2022-05-14T01:53:57.057 INFO:journalctl@ceph.mon.b.smithi047.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:53:57.057 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:53:56 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:53:57.058 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:53:56 smithi047 ceph-mon[77147]: pgmap v4737: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:53:57.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:53:56 smithi115 conmon[90393]: 2022-05-14T01:53:56.711+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:53:57.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:53:56 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:53:57.109 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T01:53:57.109 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:53:57.109 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T01:53:57.109 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:53:57.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:53:57.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:53:57.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T01:53:57.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:53:57.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T01:53:57.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:53:57.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T01:53:57.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:53:57.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T01:53:57.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:53:57.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T01:53:57.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:53:57.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:53:57.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:53:57.113 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:53:56 smithi115 ceph-mon[78593]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T01:53:57.113 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T01:53:57.113 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:53:57.113 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T01:53:57.114 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:53:57.114 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:53:57.114 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:53:57.114 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T01:53:57.114 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:53:57.115 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T01:53:57.115 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:53:57.115 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T01:53:57.115 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:53:57.115 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T01:53:57.116 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:53:57.116 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T01:53:57.116 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:53:57.116 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:53:57.117 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:53:57.117 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T01:53:57.117 INFO:journalctl@ceph.mon.c.smithi115.stdout: During handling of the above exception, another exception occurred: 2022-05-14T01:53:57.117 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T01:53:57.117 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T01:53:57.118 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T01:53:57.118 INFO:journalctl@ceph.mon.c.smithi115.stdout: return f(*arg) 2022-05-14T01:53:57.118 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T01:53:57.118 INFO:journalctl@ceph.mon.c.smithi115.stdout: self._write_client_files(client_files, host) 2022-05-14T01:53:57.118 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T01:53:57.119 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T01:53:57.119 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T01:53:57.119 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T01:53:57.119 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T01:53:57.119 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.event_loop.get_result(coro) 2022-05-14T01:53:57.120 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T01:53:57.120 INFO:journalctl@ceph.mon.c.smithi115.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T01:53:57.120 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T01:53:57.120 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.__get_result() 2022-05-14T01:53:57.121 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T01:53:57.121 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise self._exception 2022-05-14T01:53:57.121 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T01:53:57.121 INFO:journalctl@ceph.mon.c.smithi115.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T01:53:57.121 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T01:53:57.122 INFO:journalctl@ceph.mon.c.smithi115.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T01:53:57.122 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T01:53:57.122 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise 2022-05-14T01:53:57.122 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T01:53:57.122 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.gen.throw(type, value, traceback) 2022-05-14T01:53:57.123 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T01:53:57.123 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OrchestratorError(msg) 2022-05-14T01:53:57.123 INFO:journalctl@ceph.mon.c.smithi115.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:53:57.123 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:53:56 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:53:57.124 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:53:56 smithi115 ceph-mon[78593]: pgmap v4737: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:53:58.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:53:57 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:53:58.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:53:57 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:53:58.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:53:57 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:53:58.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:53:57 smithi115 conmon[90393]: 2022-05-14T01:53:57.758+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:53:58.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:53:57 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:53:58.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:53:57 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:53:58.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:53:57 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:53:59.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:53:58 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:53:59.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:53:58 smithi047 ceph-mon[77147]: pgmap v4738: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:53:59.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:53:58 smithi115 conmon[90393]: 2022-05-14T01:53:58.733+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:53:59.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:53:58 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:53:59.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:53:58 smithi115 ceph-mon[78593]: pgmap v4738: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:54:00.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:53:59 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:54:00.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:53:59 smithi115 conmon[90393]: 2022-05-14T01:53:59.764+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:54:00.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:53:59 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:54:01.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:54:00 smithi115 conmon[90393]: 2022-05-14T01:54:00.785+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:54:01.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:54:00 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:54:01.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:54:00 smithi115 ceph-mon[78593]: pgmap v4739: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:54:01.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:54:00 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 8770 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:54:01.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:54:00 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:54:01.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:54:00 smithi047 ceph-mon[77147]: pgmap v4739: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:54:01.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:54:00 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 8770 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:54:02.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:54:01 smithi115 conmon[90393]: 2022-05-14T01:54:01.743+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:54:02.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:54:02 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:54:02.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:54:02 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:54:02.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:54:02 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:54:02.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:54:02 smithi047 ceph-mon[77147]: pgmap v4740: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:54:02.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:54:02 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:54:02.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:54:02 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:54:02.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:54:02 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:54:02.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:54:02 smithi115 ceph-mon[78593]: pgmap v4740: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:54:03.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:54:02 smithi115 conmon[90393]: 2022-05-14T01:54:02.791+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:54:03.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:54:03 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:54:03.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:54:03 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:54:04.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:54:03 smithi115 conmon[90393]: 2022-05-14T01:54:03.793+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:54:04.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:54:04 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:54:04.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:54:04 smithi047 ceph-mon[77147]: pgmap v4741: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:54:04.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:54:04 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:54:04.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:54:04 smithi115 ceph-mon[78593]: pgmap v4741: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:54:05.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:54:04 smithi115 conmon[90393]: 2022-05-14T01:54:04.821+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:54:05.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:54:05 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:54:05.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:54:05 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 8775 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:54:05.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:54:05 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:54:05.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:54:05 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 8775 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:54:06.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:54:05 smithi115 conmon[90393]: 2022-05-14T01:54:05.836+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:54:06.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:54:06 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:54:06.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:54:06 smithi047 ceph-mon[77147]: pgmap v4742: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:54:06.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:54:06 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:54:06.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:54:06 smithi115 ceph-mon[78593]: pgmap v4742: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:54:07.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:54:06 smithi115 conmon[90393]: 2022-05-14T01:54:06.828+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:54:07.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:54:07 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:54:07.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:54:07 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:54:08.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:54:07 smithi115 conmon[90393]: 2022-05-14T01:54:07.792+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:54:08.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:54:08 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:54:08.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:54:08 smithi047 ceph-mon[77147]: pgmap v4743: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:54:08.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:54:08 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:54:08.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:54:08 smithi115 ceph-mon[78593]: pgmap v4743: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:54:09.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:54:08 smithi115 conmon[90393]: 2022-05-14T01:54:08.838+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:54:09.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:54:09 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:54:09.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:54:09 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:54:10.218 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:54:09 smithi115 conmon[90393]: 2022-05-14T01:54:09.884+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:54:10.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:54:10 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:54:10.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:54:10 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:54:10.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:54:10 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:54:10.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:54:10 smithi047 ceph-mon[77147]: pgmap v4744: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:54:10.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:54:10 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 8780 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:54:10.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:54:10 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:54:10.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:54:10 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:54:10.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:54:10 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:54:10.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:54:10 smithi115 ceph-mon[78593]: pgmap v4744: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:54:10.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:54:10 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 8780 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:54:11.220 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:54:10 smithi115 conmon[90393]: 2022-05-14T01:54:10.889+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:54:11.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:54:11 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:54:11.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:54:11 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:54:12.225 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:54:11 smithi115 conmon[90393]: 2022-05-14T01:54:11.872+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:54:12.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:54:12 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:54:12.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:54:12 smithi047 ceph-mon[77147]: pgmap v4745: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:54:12.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:54:12 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:54:12.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:54:12 smithi115 ceph-mon[78593]: pgmap v4745: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:54:13.231 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:54:12 smithi115 conmon[90393]: 2022-05-14T01:54:12.880+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:54:13.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:54:13 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:54:13.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:54:13 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:54:14.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:54:13 smithi115 conmon[90393]: 2022-05-14T01:54:13.846+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:54:14.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:54:14 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:54:14.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:54:14 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:54:14.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:54:14 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:54:14.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:54:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T01:54:14.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:54:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T01:54:14.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:54:14 smithi047 ceph-mon[77147]: pgmap v4746: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:54:14.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:54:14 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:54:14.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:54:14 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:54:14.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:54:14 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:54:14.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:54:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T01:54:14.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:54:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T01:54:14.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:54:14 smithi115 ceph-mon[78593]: pgmap v4746: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:54:15.239 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:54:14 smithi115 conmon[90393]: 2022-05-14T01:54:14.870+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:54:15.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:54:15 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:54:15.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:54:15 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 8785 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:54:15.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:54:15 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:54:15.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:54:15 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 8785 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:54:16.244 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:54:15 smithi115 conmon[90393]: 2022-05-14T01:54:15.874+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:54:16.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:54:16 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:54:16.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:54:16 smithi047 ceph-mon[77147]: pgmap v4747: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:54:16.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:54:16 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:54:16.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:54:16 smithi115 ceph-mon[78593]: pgmap v4747: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:54:17.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:54:16 smithi115 conmon[90393]: 2022-05-14T01:54:16.843+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:54:17.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:54:17 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:54:17.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:54:17 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:54:18.252 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:54:17 smithi115 conmon[90393]: 2022-05-14T01:54:17.869+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:54:18.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:54:18 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:54:18.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:54:18 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:54:18.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:54:18 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:54:18.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:54:18 smithi047 ceph-mon[77147]: pgmap v4748: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:54:18.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:54:18 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:54:18.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:54:18 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:54:18.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:54:18 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:54:18.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:54:18 smithi115 ceph-mon[78593]: pgmap v4748: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:54:19.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:54:18 smithi115 conmon[90393]: 2022-05-14T01:54:18.827+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:54:19.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:54:19 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:54:19.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:54:19 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:54:20.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:54:19 smithi115 conmon[90393]: 2022-05-14T01:54:19.847+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:54:20.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:54:20 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:54:20.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:54:20 smithi047 ceph-mon[77147]: pgmap v4749: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:54:20.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:54:20 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 8790 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:54:20.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:54:20 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:54:20.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:54:20 smithi115 ceph-mon[78593]: pgmap v4749: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:54:20.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:54:20 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 8790 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:54:21.264 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:54:20 smithi115 conmon[90393]: 2022-05-14T01:54:20.857+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:54:21.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:54:21 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:54:21.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:54:21 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:54:22.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:54:21 smithi115 conmon[90393]: 2022-05-14T01:54:21.824+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:54:22.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:54:22 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:54:22.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:54:22 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:54:22.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:54:22 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:54:22.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:54:22 smithi047 ceph-mon[77147]: pgmap v4750: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:54:22.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:54:22 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:54:22.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:54:22 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:54:22.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:54:22 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:54:22.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:54:22 smithi115 ceph-mon[78593]: pgmap v4750: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:54:23.271 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:54:22 smithi115 conmon[90393]: 2022-05-14T01:54:22.860+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:54:23.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:54:23 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:54:23.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:54:23 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:54:24.275 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:54:23 smithi115 conmon[90393]: 2022-05-14T01:54:23.905+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:54:24.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:54:24 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:54:24.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:54:24 smithi047 ceph-mon[77147]: pgmap v4751: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:54:24.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:54:24 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:54:24.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:54:24 smithi115 ceph-mon[78593]: pgmap v4751: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:54:25.277 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:54:24 smithi115 conmon[90393]: 2022-05-14T01:54:24.946+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:54:25.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:54:25 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:54:25.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:54:25 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 8795 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:54:25.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:54:25 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:54:25.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:54:25 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 8795 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:54:26.282 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:54:25 smithi115 conmon[90393]: 2022-05-14T01:54:25.986+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:54:26.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:54:26 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:54:26.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:54:26 smithi047 ceph-mon[77147]: pgmap v4752: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:54:26.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:54:26 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:54:26.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:54:26 smithi115 ceph-mon[78593]: pgmap v4752: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:54:27.283 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:54:27 smithi115 conmon[90393]: 2022-05-14T01:54:27.018+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:54:27.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:54:27 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:54:27.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:54:27 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:54:28.289 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:54:27 smithi115 conmon[90393]: 2022-05-14T01:54:27.989+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:54:28.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:54:28 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:54:28.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:54:28 smithi047 ceph-mon[77147]: pgmap v4753: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:54:28.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:54:28 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:54:28.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:54:28 smithi115 ceph-mon[78593]: pgmap v4753: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:54:29.289 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:54:28 smithi115 conmon[90393]: 2022-05-14T01:54:28.949+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:54:29.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:54:29 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:54:29.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:54:29 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:54:30.297 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:54:29 smithi115 conmon[90393]: 2022-05-14T01:54:29.998+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:54:30.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:54:30 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:54:30.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:54:30 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:54:30.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:54:30 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:54:30.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:54:30 smithi115 ceph-mon[78593]: pgmap v4754: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:54:30.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:54:30 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 8800 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:54:30.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:54:30 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:54:30.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:54:30 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:54:30.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:54:30 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:54:30.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:54:30 smithi047 ceph-mon[77147]: pgmap v4754: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:54:30.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:54:30 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 8800 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:54:31.299 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:54:30 smithi115 conmon[90393]: 2022-05-14T01:54:30.966+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:54:31.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:54:31 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:54:31.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:54:31 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:54:32.304 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:54:31 smithi115 conmon[90393]: 2022-05-14T01:54:31.938+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:54:32.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:54:32 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:54:32.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:54:32 smithi115 ceph-mon[78593]: pgmap v4755: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:54:32.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:54:32 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:54:32.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:54:32 smithi047 ceph-mon[77147]: pgmap v4755: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:54:33.306 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:54:32 smithi115 conmon[90393]: 2022-05-14T01:54:32.910+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:54:33.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:54:33 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:54:33.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:54:33 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:54:34.317 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:54:33 smithi115 conmon[90393]: 2022-05-14T01:54:33.935+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:54:34.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:54:34 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:54:34.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:54:34 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:54:34.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:54:34 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:54:34.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:54:34 smithi115 ceph-mon[78593]: pgmap v4756: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:54:34.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:54:34 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:54:34.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:54:34 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:54:34.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:54:34 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:54:34.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:54:34 smithi047 ceph-mon[77147]: pgmap v4756: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:54:35.318 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:54:34 smithi115 conmon[90393]: 2022-05-14T01:54:34.947+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:54:35.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:54:35 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:54:35.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:54:35 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 8805 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:54:35.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:54:35 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:54:35.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:54:35 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 8805 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:54:36.324 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:54:35 smithi115 conmon[90393]: 2022-05-14T01:54:35.925+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:54:36.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:54:36 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:54:36.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:54:36 smithi115 ceph-mon[78593]: pgmap v4757: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:54:36.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:54:36 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:54:36.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:54:36 smithi047 ceph-mon[77147]: pgmap v4757: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:54:37.324 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:54:36 smithi115 conmon[90393]: 2022-05-14T01:54:36.930+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:54:37.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:54:37 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:54:37.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:54:37 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:54:38.335 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:54:37 smithi115 conmon[90393]: 2022-05-14T01:54:37.921+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:54:38.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:54:38 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:54:38.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:54:38 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:54:38.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:54:38 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:54:38.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:54:38 smithi115 ceph-mon[78593]: pgmap v4758: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:54:38.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:54:38 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:54:38.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:54:38 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:54:38.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:54:38 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:54:38.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:54:38 smithi047 ceph-mon[77147]: pgmap v4758: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:54:39.334 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:54:38 smithi115 conmon[90393]: 2022-05-14T01:54:38.886+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:54:39.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:54:39 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:54:39.795 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:54:39 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:54:40.338 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:54:39 smithi115 conmon[90393]: 2022-05-14T01:54:39.879+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:54:40.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:54:40 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:54:40.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:54:40 smithi115 ceph-mon[78593]: pgmap v4759: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:54:40.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:54:40 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 8810 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:54:40.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:54:40 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:54:40.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:54:40 smithi047 ceph-mon[77147]: pgmap v4759: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:54:40.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:54:40 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 8810 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:54:41.347 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:54:40 smithi115 conmon[90393]: 2022-05-14T01:54:40.919+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:54:41.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:54:41 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:54:41.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:54:41 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:54:42.346 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:54:41 smithi115 conmon[90393]: 2022-05-14T01:54:41.917+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:54:42.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:54:42 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:54:42.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:54:42 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:54:42.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:54:42 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:54:42.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:54:42 smithi115 ceph-mon[78593]: pgmap v4760: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:54:42.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:54:42 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:54:42.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:54:42 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:54:42.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:54:42 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:54:42.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:54:42 smithi047 ceph-mon[77147]: pgmap v4760: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:54:43.354 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:54:42 smithi115 conmon[90393]: 2022-05-14T01:54:42.894+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:54:43.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:54:43 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:54:43.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:54:43 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:54:44.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:54:43 smithi115 conmon[90393]: 2022-05-14T01:54:43.880+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:54:44.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:54:44 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:54:44.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:54:44 smithi047 ceph-mon[77147]: pgmap v4761: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:54:44.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:54:44 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:54:44.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:54:44 smithi115 ceph-mon[78593]: pgmap v4761: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:54:45.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:54:44 smithi115 conmon[90393]: 2022-05-14T01:54:44.883+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:54:45.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:54:45 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:54:45.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:54:45 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 8815 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:54:45.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:54:45 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:54:45.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:54:45 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 8815 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:54:46.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:54:45 smithi115 conmon[90393]: 2022-05-14T01:54:45.923+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:54:46.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:54:46 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:54:46.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:54:46 smithi047 ceph-mon[77147]: pgmap v4762: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:54:46.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:54:46 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:54:46.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:54:46 smithi115 ceph-mon[78593]: pgmap v4762: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:54:47.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:54:46 smithi115 conmon[90393]: 2022-05-14T01:54:46.942+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:54:47.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:54:47 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:54:47.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:54:47 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:54:48.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:54:47 smithi115 conmon[90393]: 2022-05-14T01:54:47.961+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:54:48.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:54:48 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:54:48.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:54:48 smithi047 ceph-mon[77147]: pgmap v4763: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:54:48.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:54:48 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:54:48.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:54:48 smithi115 ceph-mon[78593]: pgmap v4763: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:54:49.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:54:49 smithi115 conmon[90393]: 2022-05-14T01:54:49.001+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:54:49.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:54:49 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:54:49.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:54:49 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:54:50.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:54:49 smithi115 conmon[90393]: 2022-05-14T01:54:49.976+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:54:50.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:54:50 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:54:50.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:54:50 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:54:50.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:54:50 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:54:50.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:54:50 smithi047 ceph-mon[77147]: pgmap v4764: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:54:50.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:54:50 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 8820 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:54:50.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:54:50 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:54:50.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:54:50 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:54:50.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:54:50 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:54:50.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:54:50 smithi115 ceph-mon[78593]: pgmap v4764: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:54:50.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:54:50 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 8820 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:54:51.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:54:50 smithi115 conmon[90393]: 2022-05-14T01:54:50.946+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:54:51.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:54:51 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:54:51.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:54:51 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:54:52.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:54:51 smithi115 conmon[90393]: 2022-05-14T01:54:51.997+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:54:52.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:54:52 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:54:52.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:54:52 smithi047 ceph-mon[77147]: pgmap v4765: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:54:52.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:54:52 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:54:52.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:54:52 smithi115 ceph-mon[78593]: pgmap v4765: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:54:53.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:54:53 smithi115 conmon[90393]: 2022-05-14T01:54:53.027+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:54:53.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:54:53 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:54:53.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:54:53 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:54:54.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:54:54 smithi115 conmon[90393]: 2022-05-14T01:54:54.029+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:54:54.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:54:54 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:54:54.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:54:54 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:54:54.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:54:54 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:54:54.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:54:54 smithi047 ceph-mon[77147]: pgmap v4766: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:54:54.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:54:54 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:54:54.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:54:54 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:54:54.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:54:54 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:54:54.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:54:54 smithi115 ceph-mon[78593]: pgmap v4766: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:54:55.298 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:54:55 smithi115 conmon[90393]: 2022-05-14T01:54:55.039+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:54:55.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:54:55 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:54:55.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:54:55 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 8825 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:54:55.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:54:55 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T01:54:55.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:54:55 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T01:54:55.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:54:55 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T01:54:55.766 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:54:55 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:54:55.767 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:54:55 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 8825 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:54:55.767 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:54:55 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T01:54:55.767 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:54:55 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T01:54:55.767 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:54:55 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T01:54:56.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:54:56 smithi115 conmon[90393]: 2022-05-14T01:54:56.040+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:54:56.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:54:56 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:54:56.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:54:56 smithi047 ceph-mon[77147]: pgmap v4767: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:54:56.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:54:56 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:54:56.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:54:56 smithi115 ceph-mon[78593]: pgmap v4767: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:54:57.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:54:57 smithi115 conmon[90393]: 2022-05-14T01:54:57.041+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:54:57.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:54:57 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:54:57.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:54:57 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:54:58.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:54:58 smithi115 conmon[90393]: 2022-05-14T01:54:58.023+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:54:58.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:54:58 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:54:58.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:54:58 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:54:58.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:54:58 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:54:58.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:54:58 smithi047 ceph-mon[77147]: pgmap v4768: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:54:58.792 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:54:58 smithi047 conmon[78521]: 2022-05-14T01:54:58.351+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:54:58.793 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:54:58 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T01:54:58.793 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:54:58 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:54:58.793 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:54:58 smithi047 conmon[78521]: yield 2022-05-14T01:54:58.793 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:54:58 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:54:58.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:54:58 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:54:58.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:54:58 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:54:58.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:54:58 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T01:54:58.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:54:58 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:54:58.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:54:58 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T01:54:58.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:54:58 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:54:58.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:54:58 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T01:54:58.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:54:58 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:54:58.797 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:54:58 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T01:54:58.797 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:54:58 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:54:58.797 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:54:58 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T01:54:58.798 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:54:58 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:54:58.798 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:54:58 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:54:58.798 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:54:58 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:54:58.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:54:58 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:54:58.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:54:58 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:54:58.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:54:58 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:54:58.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:54:58 smithi115 ceph-mon[78593]: pgmap v4768: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:54:59.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:54:59 smithi115 conmon[90393]: 2022-05-14T01:54:59.014+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:54:59.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:54:59 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:54:59.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:54:59 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:54:59.792 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T01:54:59.792 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:54:59.792 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T01:54:59.793 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:54:59.793 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:54:59.793 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:54:59.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T01:54:59.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:54:59.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T01:54:59.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:54:59.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T01:54:59.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:54:59.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T01:54:59.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:54:59.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T01:54:59.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:54:59.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:54:59.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:54:59.797 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:54:59 smithi047 ceph-mon[77147]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T01:54:59.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:54:59 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:54:59.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:54:59 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:54:59.858 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T01:54:59.859 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:54:59.859 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T01:54:59.859 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:54:59.859 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:54:59.860 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:54:59.860 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T01:54:59.860 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:54:59.860 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T01:54:59.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:54:59.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T01:54:59.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:54:59.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T01:54:59.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:54:59.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T01:54:59.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:54:59.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:54:59.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:54:59.863 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:54:59 smithi115 ceph-mon[78593]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T01:55:00.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:55:00 smithi115 conmon[90393]: 2022-05-14T01:55:00.051+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:55:00.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:55:00 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:55:00.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:55:00 smithi047 ceph-mon[77147]: pgmap v4769: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:55:00.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:55:00 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 8830 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:55:00.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:55:00 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:55:00.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:55:00 smithi115 ceph-mon[78593]: pgmap v4769: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:55:00.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:55:00 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 8830 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:55:01.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:55:01 smithi115 conmon[90393]: 2022-05-14T01:55:01.101+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:55:01.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:55:01 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:55:01.792 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:55:01 smithi047 conmon[78521]: 2022-05-14T01:55:01.423+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:55:01.792 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:55:01 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T01:55:01.792 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:55:01 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:55:01.793 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:55:01 smithi047 conmon[78521]: yield 2022-05-14T01:55:01.793 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:55:01 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:55:01.793 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:55:01 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:55:01.793 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:55:01 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:55:01.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:55:01 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T01:55:01.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:55:01 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:55:01.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:55:01 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T01:55:01.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:55:01 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:55:01.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:55:01 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T01:55:01.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:55:01 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:55:01.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:55:01 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T01:55:01.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:55:01 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:55:01.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:55:01 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T01:55:01.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:55:01 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:55:01.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:55:01 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:55:01.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:55:01 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:55:01.797 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:55:01 smithi047 conmon[78521]: 2022-05-14T01:55:01.424+0000 7f5344f82700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T01:55:01.797 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:55:01 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T01:55:01.797 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:55:01 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:55:01.797 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:55:01 smithi047 conmon[78521]: yield 2022-05-14T01:55:01.797 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:55:01 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:55:01.798 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:55:01 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:55:01.798 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:55:01 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:55:01.798 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:55:01 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T01:55:01.798 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:55:01 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:55:01.799 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:55:01 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T01:55:01.799 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:55:01 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:55:01.799 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:55:01 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T01:55:01.799 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:55:01 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:55:01.799 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:55:01 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T01:55:01.800 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:55:01 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:55:01.800 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:55:01 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T01:55:01.800 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:55:01 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:55:01.800 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:55:01 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:55:01.801 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:55:01 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:55:01.801 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:55:01 smithi047 conmon[78521]: 2022-05-14T01:55:01.801 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:55:01 smithi047 conmon[78521]: During handling of the above exception, another exception occurred: 2022-05-14T01:55:01.801 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:55:01 smithi047 conmon[78521]: 2022-05-14T01:55:01.801 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:55:01 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T01:55:01.802 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:55:01 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T01:55:01.802 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:55:01 smithi047 conmon[78521]: return f(*arg) 2022-05-14T01:55:01.802 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:55:01 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T01:55:01.802 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:55:01 smithi047 conmon[78521]: self._write_client_files(client_files, host) 2022-05-14T01:55:01.803 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:55:01 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T01:55:01.803 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:55:01 smithi047 conmon[78521]: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T01:55:01.803 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:55:01 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T01:55:01.803 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:55:01 smithi047 conmon[78521]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T01:55:01.803 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:55:01 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T01:55:01.804 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:55:01 smithi047 conmon[78521]: return self.event_loop.get_result(coro) 2022-05-14T01:55:01.804 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:55:01 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T01:55:01.804 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:55:01 smithi047 conmon[78521]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T01:55:01.804 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:55:01 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T01:55:01.804 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:55:01 smithi047 conmon[78521]: return self.__get_result() 2022-05-14T01:55:01.805 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:55:01 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T01:55:01.805 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:55:01 smithi047 conmon[78521]: raise self._exception 2022-05-14T01:55:01.805 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:55:01 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T01:55:01.805 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:55:01 smithi047 conmon[78521]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T01:55:01.806 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:55:01 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T01:55:01.806 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:55:01 smithi047 conmon[78521]: conn = await self._remote_connection(host, addr) 2022-05-14T01:55:01.806 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:55:01 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T01:55:01.806 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:55:01 smithi047 conmon[78521]: raise 2022-05-14T01:55:01.806 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:55:01 smithi047 conmon[78521]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T01:55:01.807 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:55:01 smithi047 conmon[78521]: self.gen.throw(type, value, traceback) 2022-05-14T01:55:01.807 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:55:01 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T01:55:01.807 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:55:01 smithi047 conmon[78521]: raise OrchestratorError(msg) 2022-05-14T01:55:01.807 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:55:01 smithi047 conmon[78521]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:55:01.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:55:01 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:55:02.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:55:02 smithi115 conmon[90393]: 2022-05-14T01:55:02.059+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:55:02.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:55:02 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:55:02.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:55:02 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:55:02.792 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T01:55:02.792 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:55:02.793 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T01:55:02.793 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:55:02.793 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:55:02.793 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:55:02.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T01:55:02.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:55:02.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T01:55:02.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:55:02.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T01:55:02.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:55:02.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T01:55:02.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:55:02.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T01:55:02.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:55:02.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:55:02.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:55:02.796 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:55:02 smithi047 ceph-mon[77147]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T01:55:02.797 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T01:55:02.797 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:55:02.797 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T01:55:02.797 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:55:02.797 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:55:02.798 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:55:02.798 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T01:55:02.798 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:55:02.798 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T01:55:02.799 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:55:02.799 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T01:55:02.799 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:55:02.799 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T01:55:02.799 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:55:02.800 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T01:55:02.800 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:55:02.800 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:55:02.801 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:55:02.801 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T01:55:02.801 INFO:journalctl@ceph.mon.b.smithi047.stdout: During handling of the above exception, another exception occurred: 2022-05-14T01:55:02.801 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T01:55:02.802 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T01:55:02.802 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T01:55:02.802 INFO:journalctl@ceph.mon.b.smithi047.stdout: return f(*arg) 2022-05-14T01:55:02.802 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T01:55:02.802 INFO:journalctl@ceph.mon.b.smithi047.stdout: self._write_client_files(client_files, host) 2022-05-14T01:55:02.803 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T01:55:02.803 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T01:55:02.803 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T01:55:02.803 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T01:55:02.804 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T01:55:02.804 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.event_loop.get_result(coro) 2022-05-14T01:55:02.804 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T01:55:02.804 INFO:journalctl@ceph.mon.b.smithi047.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T01:55:02.804 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T01:55:02.805 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.__get_result() 2022-05-14T01:55:02.805 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T01:55:02.805 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise self._exception 2022-05-14T01:55:02.805 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T01:55:02.806 INFO:journalctl@ceph.mon.b.smithi047.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T01:55:02.806 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T01:55:02.806 INFO:journalctl@ceph.mon.b.smithi047.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T01:55:02.806 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T01:55:02.806 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise 2022-05-14T01:55:02.807 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T01:55:02.807 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.gen.throw(type, value, traceback) 2022-05-14T01:55:02.807 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T01:55:02.807 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OrchestratorError(msg) 2022-05-14T01:55:02.808 INFO:journalctl@ceph.mon.b.smithi047.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:55:02.808 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:55:02 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:55:02.808 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:55:02 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:55:02.808 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:55:02 smithi047 ceph-mon[77147]: pgmap v4770: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:55:02.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:55:02 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:55:02.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:55:02 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:55:02.859 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T01:55:02.859 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:55:02.859 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T01:55:02.859 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:55:02.860 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:55:02.860 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:55:02.860 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T01:55:02.860 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:55:02.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T01:55:02.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:55:02.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T01:55:02.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:55:02.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T01:55:02.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:55:02.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T01:55:02.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:55:02.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:55:02.863 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:55:02.863 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:55:02 smithi115 ceph-mon[78593]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T01:55:02.863 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T01:55:02.863 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:55:02.864 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T01:55:02.864 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:55:02.864 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:55:02.864 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:55:02.864 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T01:55:02.865 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:55:02.865 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T01:55:02.865 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:55:02.865 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T01:55:02.866 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:55:02.866 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T01:55:02.866 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:55:02.866 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T01:55:02.866 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:55:02.867 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:55:02.867 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:55:02.867 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T01:55:02.867 INFO:journalctl@ceph.mon.c.smithi115.stdout: During handling of the above exception, another exception occurred: 2022-05-14T01:55:02.868 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T01:55:02.868 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T01:55:02.868 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T01:55:02.868 INFO:journalctl@ceph.mon.c.smithi115.stdout: return f(*arg) 2022-05-14T01:55:02.868 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T01:55:02.869 INFO:journalctl@ceph.mon.c.smithi115.stdout: self._write_client_files(client_files, host) 2022-05-14T01:55:02.869 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T01:55:02.869 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T01:55:02.869 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T01:55:02.870 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T01:55:02.870 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T01:55:02.870 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.event_loop.get_result(coro) 2022-05-14T01:55:02.870 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T01:55:02.870 INFO:journalctl@ceph.mon.c.smithi115.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T01:55:02.871 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T01:55:02.871 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.__get_result() 2022-05-14T01:55:02.871 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T01:55:02.871 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise self._exception 2022-05-14T01:55:02.872 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T01:55:02.872 INFO:journalctl@ceph.mon.c.smithi115.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T01:55:02.872 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T01:55:02.872 INFO:journalctl@ceph.mon.c.smithi115.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T01:55:02.872 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T01:55:02.873 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise 2022-05-14T01:55:02.873 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T01:55:02.873 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.gen.throw(type, value, traceback) 2022-05-14T01:55:02.873 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T01:55:02.873 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OrchestratorError(msg) 2022-05-14T01:55:02.874 INFO:journalctl@ceph.mon.c.smithi115.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:55:02.874 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:55:02 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:55:02.874 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:55:02 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:55:02.874 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:55:02 smithi115 ceph-mon[78593]: pgmap v4770: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:55:03.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:55:03 smithi115 conmon[90393]: 2022-05-14T01:55:03.018+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:55:03.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:55:03 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:55:03.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:55:03 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:55:04.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:55:04 smithi115 conmon[90393]: 2022-05-14T01:55:04.021+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:55:04.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:55:04 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:55:04.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:55:04 smithi047 ceph-mon[77147]: pgmap v4771: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:55:04.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:55:04 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:55:04.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:55:04 smithi115 ceph-mon[78593]: pgmap v4771: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:55:05.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:55:05 smithi115 conmon[90393]: 2022-05-14T01:55:05.026+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:55:05.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:55:05 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:55:05.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:55:05 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 8835 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:55:05.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:55:05 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:55:05.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:55:05 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 8835 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:55:06.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:55:05 smithi115 conmon[90393]: 2022-05-14T01:55:05.994+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:55:06.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:55:06 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:55:06.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:55:06 smithi047 ceph-mon[77147]: pgmap v4772: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:55:06.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:55:06 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:55:06.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:55:06 smithi115 ceph-mon[78593]: pgmap v4772: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:55:07.365 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:55:06 smithi115 conmon[90393]: 2022-05-14T01:55:06.952+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:55:07.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:55:07 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:55:07.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:55:07 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:55:08.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:55:07 smithi115 conmon[90393]: 2022-05-14T01:55:07.903+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:55:08.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:55:08 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:55:08.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:55:08 smithi047 ceph-mon[77147]: pgmap v4773: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:55:08.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:55:08 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:55:08.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:55:08 smithi115 ceph-mon[78593]: pgmap v4773: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:55:09.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:55:08 smithi115 conmon[90393]: 2022-05-14T01:55:08.895+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:55:09.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:55:09 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:55:09.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:55:09 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:55:10.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:55:09 smithi115 conmon[90393]: 2022-05-14T01:55:09.885+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:55:10.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:55:10 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:55:10.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:55:10 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:55:10.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:55:10 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:55:10.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:55:10 smithi047 ceph-mon[77147]: pgmap v4774: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:55:10.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:55:10 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 8840 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:55:10.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:55:10 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:55:10.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:55:10 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:55:10.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:55:10 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:55:10.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:55:10 smithi115 ceph-mon[78593]: pgmap v4774: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:55:10.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:55:10 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 8840 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:55:11.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:55:10 smithi115 conmon[90393]: 2022-05-14T01:55:10.895+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:55:11.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:55:11 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:55:11.855 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:55:11 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:55:12.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:55:11 smithi115 conmon[90393]: 2022-05-14T01:55:11.853+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:55:12.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:55:12 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:55:12.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:55:12 smithi047 ceph-mon[77147]: pgmap v4775: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:55:12.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:55:12 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:55:12.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:55:12 smithi115 ceph-mon[78593]: pgmap v4775: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:55:13.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:55:12 smithi115 conmon[90393]: 2022-05-14T01:55:12.865+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:55:13.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:55:13 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:55:13.851 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:55:13 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:55:14.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:55:13 smithi115 conmon[90393]: 2022-05-14T01:55:13.850+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:55:14.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:55:14 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:55:14.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:55:14 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:55:14.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:55:14 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:55:14.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:55:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T01:55:14.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:55:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T01:55:14.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:55:14 smithi047 ceph-mon[77147]: pgmap v4776: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:55:14.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:55:14 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:55:14.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:55:14 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:55:14.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:55:14 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:55:14.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:55:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T01:55:14.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:55:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T01:55:14.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:55:14 smithi115 ceph-mon[78593]: pgmap v4776: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:55:15.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:55:14 smithi115 conmon[90393]: 2022-05-14T01:55:14.883+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:55:15.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:55:15 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:55:15.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:55:15 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 8845 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:55:15.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:55:15 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:55:15.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:55:15 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 8845 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:55:16.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:55:15 smithi115 conmon[90393]: 2022-05-14T01:55:15.905+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:55:16.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:55:16 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:55:16.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:55:16 smithi047 ceph-mon[77147]: pgmap v4777: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:55:16.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:55:16 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:55:16.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:55:16 smithi115 ceph-mon[78593]: pgmap v4777: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:55:17.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:55:16 smithi115 conmon[90393]: 2022-05-14T01:55:16.894+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:55:17.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:55:17 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:55:17.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:55:17 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:55:18.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:55:17 smithi115 conmon[90393]: 2022-05-14T01:55:17.909+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:55:18.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:55:18 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:55:18.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:55:18 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:55:18.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:55:18 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:55:18.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:55:18 smithi047 ceph-mon[77147]: pgmap v4778: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:55:18.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:55:18 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:55:18.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:55:18 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:55:18.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:55:18 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:55:18.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:55:18 smithi115 ceph-mon[78593]: pgmap v4778: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:55:19.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:55:18 smithi115 conmon[90393]: 2022-05-14T01:55:18.901+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:55:19.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:55:19 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:55:19.863 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:55:19 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:55:20.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:55:19 smithi115 conmon[90393]: 2022-05-14T01:55:19.923+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:55:20.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:55:20 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:55:20.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:55:20 smithi047 ceph-mon[77147]: pgmap v4779: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:55:20.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:55:20 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 8850 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:55:20.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:55:20 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:55:20.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:55:20 smithi115 ceph-mon[78593]: pgmap v4779: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:55:20.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:55:20 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 8850 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:55:21.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:55:20 smithi115 conmon[90393]: 2022-05-14T01:55:20.884+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:55:21.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:55:21 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:55:21.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:55:21 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:55:22.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:55:21 smithi115 conmon[90393]: 2022-05-14T01:55:21.934+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:55:22.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:55:22 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:55:22.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:55:22 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:55:22.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:55:22 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:55:22.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:55:22 smithi047 ceph-mon[77147]: pgmap v4780: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:55:22.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:55:22 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:55:22.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:55:22 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:55:22.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:55:22 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:55:22.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:55:22 smithi115 ceph-mon[78593]: pgmap v4780: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:55:23.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:55:22 smithi115 conmon[90393]: 2022-05-14T01:55:22.981+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:55:23.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:55:23 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:55:23.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:55:23 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:55:24.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:55:23 smithi115 conmon[90393]: 2022-05-14T01:55:23.989+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:55:24.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:55:24 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:55:24.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:55:24 smithi047 ceph-mon[77147]: pgmap v4781: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:55:24.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:55:24 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:55:24.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:55:24 smithi115 ceph-mon[78593]: pgmap v4781: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:55:25.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:55:24 smithi115 conmon[90393]: 2022-05-14T01:55:24.954+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:55:25.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:55:25 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:55:25.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:55:25 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 8855 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:55:25.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:55:25 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:55:25.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:55:25 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 8855 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:55:26.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:55:25 smithi115 conmon[90393]: 2022-05-14T01:55:25.993+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:55:26.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:55:26 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:55:26.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:55:26 smithi047 ceph-mon[77147]: pgmap v4782: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:55:26.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:55:26 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:55:26.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:55:26 smithi115 ceph-mon[78593]: pgmap v4782: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:55:27.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:55:27 smithi115 conmon[90393]: 2022-05-14T01:55:27.007+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:55:27.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:55:27 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:55:27.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:55:27 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:55:28.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:55:27 smithi115 conmon[90393]: 2022-05-14T01:55:27.975+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:55:28.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:55:28 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:55:28.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:55:28 smithi047 ceph-mon[77147]: pgmap v4783: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:55:28.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:55:28 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:55:28.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:55:28 smithi115 ceph-mon[78593]: pgmap v4783: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:55:29.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:55:29 smithi115 conmon[90393]: 2022-05-14T01:55:29.002+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:55:29.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:55:29 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:55:29.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:55:29 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:55:30.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:55:30 smithi115 conmon[90393]: 2022-05-14T01:55:30.040+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:55:30.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:55:30 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:55:30.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:55:30 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:55:30.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:55:30 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:55:30.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:55:30 smithi115 ceph-mon[78593]: pgmap v4784: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:55:30.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:55:30 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 8860 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:55:31.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:55:30 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:55:31.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:55:30 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:55:31.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:55:30 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:55:31.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:55:30 smithi047 ceph-mon[77147]: pgmap v4784: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:55:31.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:55:30 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 8860 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:55:31.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:55:31 smithi115 conmon[90393]: 2022-05-14T01:55:31.014+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:55:31.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:55:31 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:55:32.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:55:31 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:55:32.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:55:32 smithi115 conmon[90393]: 2022-05-14T01:55:32.045+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:55:32.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:55:32 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:55:32.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:55:32 smithi115 ceph-mon[78593]: pgmap v4785: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:55:33.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:55:32 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:55:33.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:55:32 smithi047 ceph-mon[77147]: pgmap v4785: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:55:33.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:55:33 smithi115 conmon[90393]: 2022-05-14T01:55:33.052+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:55:33.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:55:33 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:55:34.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:55:33 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:55:34.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:55:34 smithi115 conmon[90393]: 2022-05-14T01:55:34.087+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:55:34.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:55:34 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:55:34.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:55:34 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:55:34.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:55:34 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:55:34.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:55:34 smithi115 ceph-mon[78593]: pgmap v4786: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:55:35.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:55:34 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:55:35.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:55:34 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:55:35.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:55:34 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:55:35.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:55:34 smithi047 ceph-mon[77147]: pgmap v4786: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:55:35.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:55:35 smithi115 conmon[90393]: 2022-05-14T01:55:35.105+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:55:35.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:55:35 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:55:35.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:55:35 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 8865 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:55:36.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:55:35 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:55:36.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:55:35 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 8865 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:55:36.573 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:55:36 smithi115 conmon[90393]: 2022-05-14T01:55:36.108+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:55:36.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:55:36 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:55:36.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:55:36 smithi115 ceph-mon[78593]: pgmap v4787: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:55:37.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:55:36 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:55:37.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:55:36 smithi047 ceph-mon[77147]: pgmap v4787: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:55:37.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:55:37 smithi115 conmon[90393]: 2022-05-14T01:55:37.087+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:55:37.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:55:37 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:55:38.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:55:37 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:55:38.584 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:55:38 smithi115 conmon[90393]: 2022-05-14T01:55:38.137+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:55:38.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:55:38 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:55:38.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:55:38 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:55:38.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:55:38 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:55:38.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:55:38 smithi115 ceph-mon[78593]: pgmap v4788: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:55:39.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:55:38 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:55:39.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:55:38 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:55:39.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:55:38 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:55:39.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:55:38 smithi047 ceph-mon[77147]: pgmap v4788: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:55:39.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:55:39 smithi115 conmon[90393]: 2022-05-14T01:55:39.091+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:55:39.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:55:39 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:55:40.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:55:39 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:55:40.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:55:40 smithi115 conmon[90393]: 2022-05-14T01:55:40.075+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:55:40.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:55:40 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:55:40.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:55:40 smithi115 ceph-mon[78593]: pgmap v4789: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:55:40.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:55:40 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 8869 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:55:41.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:55:40 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:55:41.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:55:40 smithi047 ceph-mon[77147]: pgmap v4789: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:55:41.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:55:40 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 8869 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:55:41.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:55:41 smithi115 conmon[90393]: 2022-05-14T01:55:41.036+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:55:41.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:55:41 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:55:42.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:55:41 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:55:42.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:55:42 smithi115 conmon[90393]: 2022-05-14T01:55:42.043+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:55:42.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:55:42 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:55:42.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:55:42 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:55:42.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:55:42 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:55:42.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:55:42 smithi115 ceph-mon[78593]: pgmap v4790: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:55:43.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:55:42 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:55:43.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:55:42 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:55:43.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:55:42 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:55:43.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:55:42 smithi047 ceph-mon[77147]: pgmap v4790: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:55:43.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:55:43 smithi115 conmon[90393]: 2022-05-14T01:55:43.068+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:55:43.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:55:43 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:55:44.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:55:43 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:55:44.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:55:44 smithi115 conmon[90393]: 2022-05-14T01:55:44.042+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:55:45.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:55:44 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:55:45.060 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:55:44 smithi047 ceph-mon[77147]: pgmap v4791: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:55:45.061 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:55:44 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:55:45.061 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:55:44 smithi115 ceph-mon[78593]: pgmap v4791: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:55:45.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:55:45 smithi115 conmon[90393]: 2022-05-14T01:55:45.057+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:55:46.010 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:55:45 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:55:46.010 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:55:45 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 8875 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:55:46.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:55:45 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:55:46.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:55:45 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 8875 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:55:46.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:55:46 smithi115 conmon[90393]: 2022-05-14T01:55:46.008+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:55:47.015 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:55:46 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:55:47.015 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:55:46 smithi115 ceph-mon[78593]: pgmap v4792: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:55:47.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:55:46 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:55:47.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:55:46 smithi047 ceph-mon[77147]: pgmap v4792: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:55:47.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:55:47 smithi115 conmon[90393]: 2022-05-14T01:55:47.013+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:55:48.032 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:55:47 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:55:48.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:55:47 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:55:48.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:55:48 smithi115 conmon[90393]: 2022-05-14T01:55:48.030+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:55:49.012 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:55:48 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:55:49.012 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:55:48 smithi115 ceph-mon[78593]: pgmap v4793: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:55:49.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:55:48 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:55:49.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:55:48 smithi047 ceph-mon[77147]: pgmap v4793: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:55:49.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:55:49 smithi115 conmon[90393]: 2022-05-14T01:55:49.010+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:55:49.980 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:55:49 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:55:50.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:55:49 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:55:50.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:55:49 smithi115 conmon[90393]: 2022-05-14T01:55:49.978+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:55:51.008 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:55:50 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:55:51.008 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:55:50 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:55:51.009 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:55:50 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:55:51.009 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:55:50 smithi115 ceph-mon[78593]: pgmap v4794: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:55:51.009 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:55:50 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 8880 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:55:51.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:55:50 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:55:51.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:55:50 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:55:51.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:55:50 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:55:51.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:55:50 smithi047 ceph-mon[77147]: pgmap v4794: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:55:51.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:55:50 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 8880 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:55:51.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:55:51 smithi115 conmon[90393]: 2022-05-14T01:55:51.006+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:55:52.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:55:51 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:55:52.055 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:55:51 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:55:52.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:55:52 smithi115 conmon[90393]: 2022-05-14T01:55:52.053+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:55:53.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:55:52 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:55:53.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:55:52 smithi047 ceph-mon[77147]: pgmap v4795: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:55:53.043 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:55:52 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:55:53.043 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:55:52 smithi115 ceph-mon[78593]: pgmap v4795: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:55:53.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:55:53 smithi115 conmon[90393]: 2022-05-14T01:55:53.041+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:55:54.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:55:53 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:55:54.087 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:55:53 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:55:54.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:55:54 smithi115 conmon[90393]: 2022-05-14T01:55:54.086+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:55:55.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:55:54 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:55:55.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:55:54 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:55:55.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:55:54 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:55:55.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:55:54 smithi047 ceph-mon[77147]: pgmap v4796: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:55:55.102 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:55:54 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:55:55.102 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:55:54 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:55:55.102 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:55:54 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:55:55.103 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:55:54 smithi115 ceph-mon[78593]: pgmap v4796: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:55:55.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:55:55 smithi115 conmon[90393]: 2022-05-14T01:55:55.100+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:55:56.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:55:55 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:55:56.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:55:55 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 8885 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:55:56.066 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:55:55 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:55:56.067 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:55:55 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 8885 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:55:56.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:55:56 smithi115 conmon[90393]: 2022-05-14T01:55:56.065+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:55:57.024 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:55:56 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:55:57.024 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:55:56 smithi115 ceph-mon[78593]: pgmap v4797: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:55:57.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:55:56 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:55:57.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:55:56 smithi047 ceph-mon[77147]: pgmap v4797: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:55:57.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:55:57 smithi115 conmon[90393]: 2022-05-14T01:55:57.022+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:55:58.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:55:57 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:55:58.069 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:55:57 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:55:58.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:55:58 smithi115 conmon[90393]: 2022-05-14T01:55:58.068+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:55:59.021 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:55:58 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:55:59.022 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:55:58 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:55:59.022 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:55:58 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:55:59.022 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:55:58 smithi115 ceph-mon[78593]: pgmap v4798: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:55:59.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:55:58 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:55:59.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:55:58 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:55:59.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:55:58 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:55:59.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:55:58 smithi047 ceph-mon[77147]: pgmap v4798: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:55:59.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:55:59 smithi115 conmon[90393]: 2022-05-14T01:55:59.020+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:56:00.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:55:59 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:56:00.048 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:55:59 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:56:00.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:56:00 smithi115 conmon[90393]: 2022-05-14T01:56:00.047+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:56:01.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:56:00 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:56:01.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:56:00 smithi047 ceph-mon[77147]: pgmap v4799: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:56:01.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:56:00 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 8889 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:56:01.088 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:56:00 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:56:01.089 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:56:00 smithi115 ceph-mon[78593]: pgmap v4799: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:56:01.089 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:56:00 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 8889 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:56:01.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:56:01 smithi115 conmon[90393]: 2022-05-14T01:56:01.087+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:56:01.797 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:56:01 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:56:01.797 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:56:01 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T01:56:01.798 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:56:01 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T01:56:01.798 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:56:01 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T01:56:01.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:56:01 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:56:01.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:56:01 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T01:56:01.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:56:01 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T01:56:01.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:56:01 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T01:56:02.291 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:56:02 smithi047 conmon[78521]: 2022-05-14T01:56:02.031+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:56:02.292 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:56:02 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T01:56:02.292 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:56:02 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:56:02.292 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:56:02 smithi047 conmon[78521]: yield 2022-05-14T01:56:02.292 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:56:02 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:56:02.293 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:56:02 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:56:02.293 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:56:02 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:56:02.293 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:56:02 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T01:56:02.293 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:56:02 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:56:02.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:56:02 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T01:56:02.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:56:02 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:56:02.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:56:02 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T01:56:02.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:56:02 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:56:02.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:56:02 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T01:56:02.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:56:02 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:56:02.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:56:02 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T01:56:02.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:56:02 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:56:02.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:56:02 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:56:02.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:56:02 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:56:02.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:56:02 smithi115 conmon[90393]: 2022-05-14T01:56:02.067+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:56:03.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:56:02 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:56:03.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:56:02 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:56:03.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:56:02 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:56:03.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:56:02 smithi047 ceph-mon[77147]: pgmap v4800: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:56:03.073 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:56:02 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:56:03.074 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:56:02 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:56:03.074 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:56:02 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:56:03.074 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:56:02 smithi115 ceph-mon[78593]: pgmap v4800: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:56:03.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:56:03 smithi115 conmon[90393]: 2022-05-14T01:56:03.072+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:56:04.038 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:56:03 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:56:04.038 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T01:56:04.039 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:56:04.039 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T01:56:04.039 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:56:04.039 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:56:04.040 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:56:04.040 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T01:56:04.040 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:56:04.040 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T01:56:04.041 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:56:04.041 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T01:56:04.041 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:56:04.041 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T01:56:04.042 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:56:04.042 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T01:56:04.042 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:56:04.042 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:56:04.042 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:56:04.043 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:56:03 smithi115 ceph-mon[78593]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T01:56:04.043 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:56:03 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:56:04.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:56:03 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:56:04.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T01:56:04.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:56:04.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T01:56:04.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:56:04.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:56:04.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:56:04.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T01:56:04.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:56:04.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T01:56:04.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:56:04.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T01:56:04.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:56:04.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T01:56:04.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:56:04.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T01:56:04.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:56:04.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:56:04.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:56:04.048 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:56:03 smithi047 ceph-mon[77147]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T01:56:04.048 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:56:03 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:56:04.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:56:04 smithi115 conmon[90393]: 2022-05-14T01:56:04.036+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:56:05.022 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:56:04 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:56:05.023 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:56:04 smithi115 ceph-mon[78593]: pgmap v4801: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:56:05.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:56:04 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:56:05.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:56:04 smithi047 ceph-mon[77147]: pgmap v4801: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:56:05.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:56:05 smithi115 conmon[90393]: 2022-05-14T01:56:05.021+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:56:05.542 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:56:05 smithi047 conmon[78521]: 2022-05-14T01:56:05.104+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:56:05.542 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:56:05 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T01:56:05.542 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:56:05 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:56:05.542 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:56:05 smithi047 conmon[78521]: yield 2022-05-14T01:56:05.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:56:05 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:56:05.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:56:05 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:56:05.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:56:05 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:56:05.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:56:05 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T01:56:05.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:56:05 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:56:05.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:56:05 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T01:56:05.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:56:05 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:56:05.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:56:05 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T01:56:05.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:56:05 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:56:05.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:56:05 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T01:56:05.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:56:05 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:56:05.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:56:05 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T01:56:05.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:56:05 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:56:05.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:56:05 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:56:05.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:56:05 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:56:05.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:56:05 smithi047 conmon[78521]: 2022-05-14T01:56:05.104+0000 7f5344781700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T01:56:05.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:56:05 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T01:56:05.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:56:05 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:56:05.547 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:56:05 smithi047 conmon[78521]: yield 2022-05-14T01:56:05.547 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:56:05 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:56:05.547 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:56:05 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:56:05.547 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:56:05 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:56:05.547 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:56:05 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T01:56:05.548 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:56:05 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:56:05.548 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:56:05 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T01:56:05.548 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:56:05 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:56:05.548 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:56:05 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T01:56:05.548 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:56:05 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:56:05.549 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:56:05 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T01:56:05.549 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:56:05 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:56:05.549 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:56:05 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T01:56:05.549 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:56:05 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:56:05.549 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:56:05 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:56:05.550 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:56:05 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:56:05.550 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:56:05 smithi047 conmon[78521]: 2022-05-14T01:56:05.550 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:56:05 smithi047 conmon[78521]: During handling of the above exception, another exception occurred: 2022-05-14T01:56:05.550 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:56:05 smithi047 conmon[78521]: 2022-05-14T01:56:05.550 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:56:05 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T01:56:05.551 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:56:05 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T01:56:05.551 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:56:05 smithi047 conmon[78521]: return f(*arg) 2022-05-14T01:56:05.551 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:56:05 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T01:56:05.551 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:56:05 smithi047 conmon[78521]: self._write_client_files(client_files, host) 2022-05-14T01:56:05.551 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:56:05 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T01:56:05.552 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:56:05 smithi047 conmon[78521]: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T01:56:05.552 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:56:05 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T01:56:05.552 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:56:05 smithi047 conmon[78521]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T01:56:05.552 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:56:05 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T01:56:05.553 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:56:05 smithi047 conmon[78521]: return self.event_loop.get_result(coro) 2022-05-14T01:56:05.553 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:56:05 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T01:56:05.553 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:56:05 smithi047 conmon[78521]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T01:56:05.553 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:56:05 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T01:56:05.553 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:56:05 smithi047 conmon[78521]: return self.__get_result() 2022-05-14T01:56:05.554 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:56:05 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T01:56:05.554 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:56:05 smithi047 conmon[78521]: raise self._exception 2022-05-14T01:56:05.554 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:56:05 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T01:56:05.554 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:56:05 smithi047 conmon[78521]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T01:56:05.554 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:56:05 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T01:56:05.555 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:56:05 smithi047 conmon[78521]: conn = await self._remote_connection(host, addr) 2022-05-14T01:56:05.555 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:56:05 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T01:56:05.555 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:56:05 smithi047 conmon[78521]: raise 2022-05-14T01:56:05.555 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:56:05 smithi047 conmon[78521]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T01:56:05.555 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:56:05 smithi047 conmon[78521]: self.gen.throw(type, value, traceback) 2022-05-14T01:56:05.556 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:56:05 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T01:56:05.556 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:56:05 smithi047 conmon[78521]: raise OrchestratorError(msg) 2022-05-14T01:56:05.556 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:56:05 smithi047 conmon[78521]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:56:06.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:56:05 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:56:06.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:56:05 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 8895 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:56:06.061 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:56:05 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:56:06.062 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:56:05 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 8895 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:56:06.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:56:06 smithi115 conmon[90393]: 2022-05-14T01:56:06.060+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:56:07.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:56:06 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:56:07.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:56:06 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:56:07.042 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T01:56:07.042 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:56:07.042 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T01:56:07.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:56:07.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:56:07.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:56:07.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T01:56:07.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:56:07.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T01:56:07.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:56:07.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T01:56:07.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:56:07.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T01:56:07.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:56:07.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T01:56:07.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:56:07.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:56:07.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:56:07.046 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:56:06 smithi047 ceph-mon[77147]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T01:56:07.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T01:56:07.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:56:07.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T01:56:07.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:56:07.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:56:07.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:56:07.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T01:56:07.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:56:07.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T01:56:07.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:56:07.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T01:56:07.049 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:56:07.049 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T01:56:07.049 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:56:07.049 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T01:56:07.050 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:56:07.050 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:56:07.050 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:56:07.050 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T01:56:07.050 INFO:journalctl@ceph.mon.b.smithi047.stdout: During handling of the above exception, another exception occurred: 2022-05-14T01:56:07.051 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T01:56:07.051 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T01:56:07.051 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T01:56:07.051 INFO:journalctl@ceph.mon.b.smithi047.stdout: return f(*arg) 2022-05-14T01:56:07.052 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T01:56:07.052 INFO:journalctl@ceph.mon.b.smithi047.stdout: self._write_client_files(client_files, host) 2022-05-14T01:56:07.052 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T01:56:07.052 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T01:56:07.052 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T01:56:07.053 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T01:56:07.053 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T01:56:07.053 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.event_loop.get_result(coro) 2022-05-14T01:56:07.053 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T01:56:07.053 INFO:journalctl@ceph.mon.b.smithi047.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T01:56:07.054 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T01:56:07.054 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.__get_result() 2022-05-14T01:56:07.054 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T01:56:07.054 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise self._exception 2022-05-14T01:56:07.054 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T01:56:07.055 INFO:journalctl@ceph.mon.b.smithi047.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T01:56:07.055 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T01:56:07.055 INFO:journalctl@ceph.mon.b.smithi047.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T01:56:07.055 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T01:56:07.055 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise 2022-05-14T01:56:07.056 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T01:56:07.056 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.gen.throw(type, value, traceback) 2022-05-14T01:56:07.056 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T01:56:07.056 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OrchestratorError(msg) 2022-05-14T01:56:07.057 INFO:journalctl@ceph.mon.b.smithi047.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:56:07.057 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:56:06 smithi047 ceph-mon[77147]: pgmap v4802: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:56:07.058 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:56:06 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:56:07.058 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:56:06 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:56:07.058 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T01:56:07.058 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:56:07.059 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T01:56:07.059 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:56:07.059 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:56:07.059 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:56:07.059 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T01:56:07.060 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:56:07.060 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T01:56:07.060 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:56:07.060 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T01:56:07.061 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:56:07.061 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T01:56:07.061 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:56:07.061 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T01:56:07.061 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:56:07.062 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:56:07.062 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:56:07.062 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:56:06 smithi115 ceph-mon[78593]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T01:56:07.062 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T01:56:07.063 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:56:07.063 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T01:56:07.063 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:56:07.063 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:56:07.063 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:56:07.064 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T01:56:07.064 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:56:07.064 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T01:56:07.064 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:56:07.064 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T01:56:07.065 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:56:07.065 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T01:56:07.065 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:56:07.065 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T01:56:07.066 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:56:07.066 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:56:07.066 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:56:07.066 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T01:56:07.066 INFO:journalctl@ceph.mon.c.smithi115.stdout: During handling of the above exception, another exception occurred: 2022-05-14T01:56:07.067 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T01:56:07.067 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T01:56:07.067 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T01:56:07.067 INFO:journalctl@ceph.mon.c.smithi115.stdout: return f(*arg) 2022-05-14T01:56:07.068 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T01:56:07.068 INFO:journalctl@ceph.mon.c.smithi115.stdout: self._write_client_files(client_files, host) 2022-05-14T01:56:07.068 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T01:56:07.068 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T01:56:07.068 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T01:56:07.069 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T01:56:07.069 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T01:56:07.069 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.event_loop.get_result(coro) 2022-05-14T01:56:07.069 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T01:56:07.069 INFO:journalctl@ceph.mon.c.smithi115.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T01:56:07.070 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T01:56:07.070 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.__get_result() 2022-05-14T01:56:07.070 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T01:56:07.070 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise self._exception 2022-05-14T01:56:07.070 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T01:56:07.071 INFO:journalctl@ceph.mon.c.smithi115.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T01:56:07.071 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T01:56:07.071 INFO:journalctl@ceph.mon.c.smithi115.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T01:56:07.071 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T01:56:07.071 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise 2022-05-14T01:56:07.072 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T01:56:07.072 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.gen.throw(type, value, traceback) 2022-05-14T01:56:07.072 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T01:56:07.072 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OrchestratorError(msg) 2022-05-14T01:56:07.073 INFO:journalctl@ceph.mon.c.smithi115.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:56:07.073 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:56:06 smithi115 ceph-mon[78593]: pgmap v4802: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:56:07.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:56:07 smithi115 conmon[90393]: 2022-05-14T01:56:07.044+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:56:08.012 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:56:07 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:56:08.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:56:07 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:56:08.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:56:08 smithi115 conmon[90393]: 2022-05-14T01:56:08.011+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:56:08.966 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:56:08 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:56:08.967 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:56:08 smithi115 ceph-mon[78593]: pgmap v4803: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:56:09.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:56:08 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:56:09.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:56:08 smithi047 ceph-mon[77147]: pgmap v4803: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:56:09.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:56:08 smithi115 conmon[90393]: 2022-05-14T01:56:08.965+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:56:09.961 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:56:09 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:56:10.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:56:09 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:56:10.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:56:09 smithi115 conmon[90393]: 2022-05-14T01:56:09.960+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:56:11.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:56:10 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:56:11.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:56:10 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:56:11.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:56:10 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:56:11.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:56:10 smithi047 ceph-mon[77147]: pgmap v4804: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:56:11.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:56:10 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 8900 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:56:11.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:56:10 smithi115 conmon[90393]: 2022-05-14T01:56:10.957+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:56:11.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:56:10 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:56:11.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:56:10 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:56:11.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:56:10 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:56:11.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:56:10 smithi115 ceph-mon[78593]: pgmap v4804: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:56:11.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:56:10 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 8900 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:56:12.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:56:11 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:56:12.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:56:11 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:56:12.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:56:11 smithi115 conmon[90393]: 2022-05-14T01:56:11.955+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:56:12.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:56:11 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:56:12.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:56:11 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:56:13.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:56:12 smithi047 ceph-mon[77147]: pgmap v4805: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:56:13.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:56:12 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:56:13.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:56:12 smithi115 conmon[90393]: 2022-05-14T01:56:12.951+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:56:13.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:56:12 smithi115 ceph-mon[78593]: pgmap v4805: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:56:13.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:56:12 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:56:14.000 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:56:13 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:56:14.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:56:13 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:56:14.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:56:13 smithi115 conmon[90393]: 2022-05-14T01:56:13.998+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:56:15.033 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:56:14 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:56:15.034 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:56:14 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:56:15.034 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:56:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T01:56:15.034 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:56:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T01:56:15.035 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:56:14 smithi115 ceph-mon[78593]: pgmap v4806: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:56:15.035 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:56:14 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:56:15.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:56:14 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:56:15.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:56:14 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:56:15.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:56:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T01:56:15.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:56:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T01:56:15.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:56:14 smithi047 ceph-mon[77147]: pgmap v4806: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:56:15.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:56:14 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:56:15.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:56:15 smithi115 conmon[90393]: 2022-05-14T01:56:15.032+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:56:16.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:56:15 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:56:16.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:56:15 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 8905 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:56:16.065 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:56:15 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:56:16.065 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:56:15 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 8905 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:56:16.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:56:16 smithi115 conmon[90393]: 2022-05-14T01:56:16.064+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:56:17.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:56:16 smithi047 ceph-mon[77147]: pgmap v4807: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:56:17.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:56:16 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:56:17.101 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:56:16 smithi115 ceph-mon[78593]: pgmap v4807: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:56:17.102 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:56:16 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:56:17.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:56:17 smithi115 conmon[90393]: 2022-05-14T01:56:17.100+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:56:18.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:56:17 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:56:18.057 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:56:17 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:56:18.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:56:18 smithi115 conmon[90393]: 2022-05-14T01:56:18.056+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:56:19.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:56:18 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:56:19.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:56:18 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:56:19.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:56:18 smithi047 ceph-mon[77147]: pgmap v4808: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:56:19.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:56:18 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:56:19.064 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:56:18 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:56:19.064 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:56:18 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:56:19.065 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:56:18 smithi115 ceph-mon[78593]: pgmap v4808: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:56:19.065 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:56:18 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:56:19.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:56:19 smithi115 conmon[90393]: 2022-05-14T01:56:19.063+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:56:20.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:56:19 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:56:20.053 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:56:19 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:56:20.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:56:20 smithi115 conmon[90393]: 2022-05-14T01:56:20.052+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:56:21.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:56:20 smithi047 ceph-mon[77147]: pgmap v4809: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:56:21.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:56:20 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:56:21.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:56:20 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 8910 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:56:21.048 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:56:20 smithi115 ceph-mon[78593]: pgmap v4809: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:56:21.048 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:56:20 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:56:21.048 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:56:20 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 8910 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:56:21.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:56:21 smithi115 conmon[90393]: 2022-05-14T01:56:21.047+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:56:22.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:56:22 smithi115 conmon[90393]: 2022-05-14T01:56:22.052+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:56:22.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:56:22 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:56:22.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:56:22 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:56:22.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:56:22 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:56:22.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:56:22 smithi115 ceph-mon[78593]: pgmap v4810: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:56:22.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:56:22 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:56:22.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:56:22 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:56:22.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:56:22 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:56:22.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:56:22 smithi047 ceph-mon[77147]: pgmap v4810: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:56:23.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:56:23 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:56:23.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:56:23 smithi115 conmon[90393]: 2022-05-14T01:56:23.080+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:56:23.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:56:23 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:56:24.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:56:24 smithi115 conmon[90393]: 2022-05-14T01:56:24.041+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:56:24.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:56:24 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:56:24.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:56:24 smithi115 ceph-mon[78593]: pgmap v4811: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:56:24.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:56:24 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:56:24.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:56:24 smithi047 ceph-mon[77147]: pgmap v4811: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:56:25.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:56:24 smithi115 conmon[90393]: 2022-05-14T01:56:24.992+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:56:25.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:56:25 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:56:25.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:56:25 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 8915 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:56:25.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:56:25 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:56:25.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:56:25 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 8915 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:56:26.236 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:56:25 smithi115 conmon[90393]: 2022-05-14T01:56:25.944+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:56:26.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:56:26 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:56:26.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:56:26 smithi047 ceph-mon[77147]: pgmap v4812: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:56:26.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:56:26 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:56:26.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:56:26 smithi115 ceph-mon[78593]: pgmap v4812: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:56:27.236 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:56:26 smithi115 conmon[90393]: 2022-05-14T01:56:26.963+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:56:27.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:56:27 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:56:27.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:56:27 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:56:28.243 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:56:27 smithi115 conmon[90393]: 2022-05-14T01:56:27.975+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:56:28.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:56:28 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:56:28.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:56:28 smithi047 ceph-mon[77147]: pgmap v4813: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:56:28.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:56:28 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:56:28.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:56:28 smithi115 ceph-mon[78593]: pgmap v4813: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:56:29.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:56:29 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:56:29.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:56:29 smithi115 conmon[90393]: 2022-05-14T01:56:29.024+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:56:29.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:56:29 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:56:30.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:56:30 smithi115 conmon[90393]: 2022-05-14T01:56:30.043+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:56:30.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:56:30 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:56:30.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:56:30 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:56:30.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:56:30 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:56:30.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:56:30 smithi115 ceph-mon[78593]: pgmap v4814: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:56:30.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:56:30 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 8920 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:56:30.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:56:30 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:56:30.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:56:30 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:56:30.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:56:30 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:56:30.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:56:30 smithi047 ceph-mon[77147]: pgmap v4814: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:56:30.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:56:30 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 8920 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:56:31.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:56:31 smithi115 conmon[90393]: 2022-05-14T01:56:31.023+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:56:31.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:56:31 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:56:31.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:56:31 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:56:32.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:56:32 smithi115 conmon[90393]: 2022-05-14T01:56:32.045+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:56:32.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:56:32 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:56:32.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:56:32 smithi115 ceph-mon[78593]: pgmap v4815: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:56:32.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:56:32 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:56:32.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:56:32 smithi047 ceph-mon[77147]: pgmap v4815: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:56:33.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:56:33 smithi115 conmon[90393]: 2022-05-14T01:56:33.034+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:56:33.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:56:33 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:56:33.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:56:33 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:56:34.268 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:56:34 smithi115 conmon[90393]: 2022-05-14T01:56:34.012+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:56:34.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:56:34 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:56:34.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:56:34 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:56:34.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:56:34 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:56:34.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:56:34 smithi047 ceph-mon[77147]: pgmap v4816: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:56:34.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:56:34 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:56:34.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:56:34 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:56:34.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:56:34 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:56:34.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:56:34 smithi115 ceph-mon[78593]: pgmap v4816: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:56:35.270 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:56:34 smithi115 conmon[90393]: 2022-05-14T01:56:34.991+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:56:35.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:56:35 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:56:35.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:56:35 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 8925 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:56:35.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:56:35 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:56:35.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:56:35 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 8925 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:56:36.275 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:56:36 smithi115 conmon[90393]: 2022-05-14T01:56:36.003+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:56:36.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:56:36 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:56:36.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:56:36 smithi047 ceph-mon[77147]: pgmap v4817: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:56:36.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:56:36 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:56:36.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:56:36 smithi115 ceph-mon[78593]: pgmap v4817: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:56:37.276 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:56:37 smithi115 conmon[90393]: 2022-05-14T01:56:37.015+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:56:37.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:56:37 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:56:37.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:56:37 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:56:38.282 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:56:38 smithi115 conmon[90393]: 2022-05-14T01:56:38.023+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:56:38.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:56:38 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:56:38.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:56:38 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:56:38.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:56:38 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:56:38.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:56:38 smithi047 ceph-mon[77147]: pgmap v4818: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:56:38.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:56:38 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:56:38.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:56:38 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:56:38.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:56:38 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:56:38.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:56:38 smithi115 ceph-mon[78593]: pgmap v4818: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:56:39.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:56:39 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:56:39.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:56:39 smithi115 conmon[90393]: 2022-05-14T01:56:39.068+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:56:39.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:56:39 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:56:40.297 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:56:40 smithi115 conmon[90393]: 2022-05-14T01:56:40.034+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:56:40.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:56:40 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:56:40.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:56:40 smithi115 ceph-mon[78593]: pgmap v4819: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:56:40.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:56:40 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 8930 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:56:40.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:56:40 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:56:40.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:56:40 smithi047 ceph-mon[77147]: pgmap v4819: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:56:40.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:56:40 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 8930 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:56:41.300 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:56:41 smithi115 conmon[90393]: 2022-05-14T01:56:41.007+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:56:41.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:56:41 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:56:41.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:56:41 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:56:42.305 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:56:41 smithi115 conmon[90393]: 2022-05-14T01:56:41.993+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:56:42.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:56:42 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:56:42.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:56:42 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:56:42.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:56:42 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:56:42.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:56:42 smithi115 ceph-mon[78593]: pgmap v4820: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:56:42.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:56:42 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:56:42.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:56:42 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:56:42.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:56:42 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:56:42.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:56:42 smithi047 ceph-mon[77147]: pgmap v4820: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:56:43.307 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:56:42 smithi115 conmon[90393]: 2022-05-14T01:56:42.991+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:56:43.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:56:43 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:56:43.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:56:43 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:56:44.311 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:56:44 smithi115 conmon[90393]: 2022-05-14T01:56:44.006+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:56:44.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:56:44 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:56:44.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:56:44 smithi115 ceph-mon[78593]: pgmap v4821: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:56:44.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:56:44 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:56:44.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:56:44 smithi047 ceph-mon[77147]: pgmap v4821: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:56:45.313 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:56:44 smithi115 conmon[90393]: 2022-05-14T01:56:44.962+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:56:45.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:56:45 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:56:45.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:56:45 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 8935 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:56:45.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:56:45 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:56:45.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:56:45 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 8935 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:56:46.318 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:56:45 smithi115 conmon[90393]: 2022-05-14T01:56:45.984+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:56:46.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:56:46 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:56:46.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:56:46 smithi115 ceph-mon[78593]: pgmap v4822: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:56:46.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:56:46 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:56:46.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:56:46 smithi047 ceph-mon[77147]: pgmap v4822: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:56:47.319 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:56:46 smithi115 conmon[90393]: 2022-05-14T01:56:46.969+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:56:47.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:56:47 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:56:47.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:56:47 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:56:48.326 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:56:48 smithi115 conmon[90393]: 2022-05-14T01:56:48.000+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:56:48.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:56:48 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:56:48.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:56:48 smithi115 ceph-mon[78593]: pgmap v4823: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:56:48.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:56:48 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:56:48.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:56:48 smithi047 ceph-mon[77147]: pgmap v4823: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:56:49.326 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:56:48 smithi115 conmon[90393]: 2022-05-14T01:56:48.958+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:56:49.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:56:49 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:56:49.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:56:49 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:56:50.337 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:56:49 smithi115 conmon[90393]: 2022-05-14T01:56:49.989+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:56:50.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:56:50 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:56:50.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:56:50 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:56:50.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:56:50 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:56:50.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:56:50 smithi115 ceph-mon[78593]: pgmap v4824: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:56:50.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:56:50 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 8940 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:56:50.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:56:50 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:56:50.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:56:50 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:56:50.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:56:50 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:56:50.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:56:50 smithi047 ceph-mon[77147]: pgmap v4824: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:56:50.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:56:50 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 8940 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:56:51.339 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:56:51 smithi115 conmon[90393]: 2022-05-14T01:56:51.035+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:56:51.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:56:51 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:56:51.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:56:51 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:56:52.343 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:56:52 smithi115 conmon[90393]: 2022-05-14T01:56:52.069+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:56:52.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:56:52 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:56:52.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:56:52 smithi115 ceph-mon[78593]: pgmap v4825: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:56:52.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:56:52 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:56:52.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:56:52 smithi047 ceph-mon[77147]: pgmap v4825: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:56:53.348 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:56:53 smithi115 conmon[90393]: 2022-05-14T01:56:53.072+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:56:53.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:56:53 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:56:53.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:56:53 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:56:54.354 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:56:54 smithi115 conmon[90393]: 2022-05-14T01:56:54.031+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:56:54.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:56:54 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:56:54.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:56:54 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:56:54.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:56:54 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:56:54.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:56:54 smithi115 ceph-mon[78593]: pgmap v4826: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:56:54.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:56:54 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:56:54.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:56:54 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:56:54.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:56:54 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:56:54.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:56:54 smithi047 ceph-mon[77147]: pgmap v4826: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:56:55.355 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:56:54 smithi115 conmon[90393]: 2022-05-14T01:56:54.991+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:56:55.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:56:55 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:56:55.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:56:55 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 8945 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:56:55.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:56:55 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:56:55.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:56:55 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 8945 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:56:56.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:56:55 smithi115 conmon[90393]: 2022-05-14T01:56:55.949+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:56:56.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:56:56 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:56:56.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:56:56 smithi047 ceph-mon[77147]: pgmap v4827: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:56:56.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:56:56 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:56:56.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:56:56 smithi115 ceph-mon[78593]: pgmap v4827: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:56:57.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:56:56 smithi115 conmon[90393]: 2022-05-14T01:56:56.931+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:56:57.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:56:57 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:56:57.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:56:57 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:56:58.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:56:57 smithi115 conmon[90393]: 2022-05-14T01:56:57.893+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:56:58.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:56:58 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:56:58.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:56:58 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:56:58.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:56:58 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:56:58.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:56:58 smithi047 ceph-mon[77147]: pgmap v4828: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:56:58.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:56:58 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:56:58.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:56:58 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:56:58.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:56:58 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:56:58.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:56:58 smithi115 ceph-mon[78593]: pgmap v4828: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:56:59.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:56:58 smithi115 conmon[90393]: 2022-05-14T01:56:58.887+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:56:59.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:56:59 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:56:59.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:56:59 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:57:00.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:56:59 smithi115 conmon[90393]: 2022-05-14T01:56:59.933+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:57:00.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:57:00 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:57:00.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:57:00 smithi047 ceph-mon[77147]: pgmap v4829: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:57:00.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:57:00 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 8950 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:57:00.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:57:00 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:57:00.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:57:00 smithi115 ceph-mon[78593]: pgmap v4829: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:57:00.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:57:00 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 8950 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:57:01.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:57:00 smithi115 conmon[90393]: 2022-05-14T01:57:00.903+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:57:01.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:57:01 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:57:01.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:57:01 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:57:02.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:57:01 smithi115 conmon[90393]: 2022-05-14T01:57:01.876+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:57:02.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:57:02 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:57:02.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:57:02 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:57:02.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:57:02 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:57:02.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:57:02 smithi047 ceph-mon[77147]: pgmap v4830: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:57:02.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:57:02 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:57:02.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:57:02 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:57:02.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:57:02 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:57:02.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:57:02 smithi115 ceph-mon[78593]: pgmap v4830: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:57:03.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:57:02 smithi115 conmon[90393]: 2022-05-14T01:57:02.856+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:57:03.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:57:03 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:57:03.830 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:57:03 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:57:04.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:57:03 smithi115 conmon[90393]: 2022-05-14T01:57:03.828+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:57:04.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:57:04 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:57:04.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:57:04 smithi047 ceph-mon[77147]: pgmap v4831: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:57:04.836 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:57:04 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:57:04.836 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:57:04 smithi115 ceph-mon[78593]: pgmap v4831: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:57:05.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:57:04 smithi115 conmon[90393]: 2022-05-14T01:57:04.835+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:57:05.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:57:05 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:57:05.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:57:05 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T01:57:05.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:57:05 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T01:57:05.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:57:05 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T01:57:05.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:57:05 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 8955 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:57:05.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:57:05 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:57:05.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:57:05 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T01:57:05.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:57:05 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T01:57:05.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:57:05 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T01:57:05.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:57:05 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 8955 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:57:06.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:57:05 smithi115 conmon[90393]: 2022-05-14T01:57:05.854+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:57:06.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:57:06 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:57:06.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:57:06 smithi047 ceph-mon[77147]: pgmap v4832: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:57:06.835 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:57:06 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:57:06.835 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:57:06 smithi115 ceph-mon[78593]: pgmap v4832: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:57:07.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:57:06 smithi115 conmon[90393]: 2022-05-14T01:57:06.833+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:57:07.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:57:07 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:57:07.811 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:57:07 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:57:08.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:57:07 smithi115 conmon[90393]: 2022-05-14T01:57:07.810+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:57:08.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:57:08 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:57:08.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:57:08 smithi047 ceph-mon[77147]: pgmap v4833: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:57:08.542 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:57:08 smithi047 conmon[78521]: 2022-05-14T01:57:08.208+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:57:08.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:57:08 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T01:57:08.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:57:08 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:57:08.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:57:08 smithi047 conmon[78521]: yield 2022-05-14T01:57:08.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:57:08 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:57:08.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:57:08 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:57:08.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:57:08 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:57:08.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:57:08 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T01:57:08.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:57:08 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:57:08.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:57:08 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T01:57:08.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:57:08 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:57:08.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:57:08 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T01:57:08.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:57:08 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:57:08.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:57:08 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T01:57:08.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:57:08 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:57:08.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:57:08 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T01:57:08.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:57:08 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:57:08.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:57:08 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:57:08.547 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:57:08 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:57:08.777 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:57:08 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:57:08.778 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:57:08 smithi115 ceph-mon[78593]: pgmap v4833: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:57:09.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:57:08 smithi115 conmon[90393]: 2022-05-14T01:57:08.776+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:57:09.761 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:57:09 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:57:09.761 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:57:09 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:57:09.761 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T01:57:09.762 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:57:09.762 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T01:57:09.762 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:57:09.762 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:57:09.762 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:57:09.763 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T01:57:09.763 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:57:09.763 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T01:57:09.763 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:57:09.764 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T01:57:09.764 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:57:09.764 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T01:57:09.764 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:57:09.764 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T01:57:09.765 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:57:09.765 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:57:09.765 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:57:09.765 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:57:09 smithi115 ceph-mon[78593]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T01:57:09.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:57:09 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:57:09.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:57:09 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:57:09.792 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T01:57:09.792 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:57:09.792 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T01:57:09.793 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:57:09.793 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:57:09.793 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:57:09.793 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T01:57:09.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:57:09.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T01:57:09.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:57:09.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T01:57:09.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:57:09.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T01:57:09.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:57:09.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T01:57:09.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:57:09.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:57:09.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:57:09.796 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:57:09 smithi047 ceph-mon[77147]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T01:57:10.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:57:09 smithi115 conmon[90393]: 2022-05-14T01:57:09.760+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:57:10.758 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:57:10 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:57:10.758 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:57:10 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:57:10.759 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:57:10 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:57:10.759 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:57:10 smithi115 ceph-mon[78593]: pgmap v4834: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:57:10.759 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:57:10 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 8960 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:57:10.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:57:10 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:57:10.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:57:10 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:57:10.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:57:10 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:57:10.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:57:10 smithi047 ceph-mon[77147]: pgmap v4834: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:57:10.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:57:10 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 8960 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:57:11.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:57:10 smithi115 conmon[90393]: 2022-05-14T01:57:10.757+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:57:11.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:57:11 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:57:11.542 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:57:11 smithi047 conmon[78521]: 2022-05-14T01:57:11.280+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:57:11.542 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:57:11 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T01:57:11.542 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:57:11 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:57:11.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:57:11 smithi047 conmon[78521]: yield 2022-05-14T01:57:11.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:57:11 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:57:11.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:57:11 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:57:11.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:57:11 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:57:11.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:57:11 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T01:57:11.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:57:11 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:57:11.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:57:11 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T01:57:11.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:57:11 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:57:11.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:57:11 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T01:57:11.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:57:11 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:57:11.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:57:11 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T01:57:11.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:57:11 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:57:11.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:57:11 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T01:57:11.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:57:11 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:57:11.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:57:11 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:57:11.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:57:11 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:57:11.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:57:11 smithi047 conmon[78521]: 2022-05-14T01:57:11.281+0000 7f5345f84700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T01:57:11.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:57:11 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T01:57:11.547 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:57:11 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:57:11.547 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:57:11 smithi047 conmon[78521]: yield 2022-05-14T01:57:11.547 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:57:11 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:57:11.547 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:57:11 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:57:11.547 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:57:11 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:57:11.548 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:57:11 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T01:57:11.548 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:57:11 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:57:11.548 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:57:11 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T01:57:11.548 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:57:11 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:57:11.549 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:57:11 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T01:57:11.549 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:57:11 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:57:11.549 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:57:11 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T01:57:11.549 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:57:11 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:57:11.550 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:57:11 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T01:57:11.550 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:57:11 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:57:11.550 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:57:11 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:57:11.550 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:57:11 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:57:11.550 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:57:11 smithi047 conmon[78521]: 2022-05-14T01:57:11.551 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:57:11 smithi047 conmon[78521]: During handling of the above exception, another exception occurred: 2022-05-14T01:57:11.551 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:57:11 smithi047 conmon[78521]: 2022-05-14T01:57:11.552 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:57:11 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T01:57:11.552 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:57:11 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T01:57:11.552 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:57:11 smithi047 conmon[78521]: return f(*arg) 2022-05-14T01:57:11.552 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:57:11 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T01:57:11.552 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:57:11 smithi047 conmon[78521]: self._write_client_files(client_files, host) 2022-05-14T01:57:11.553 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:57:11 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T01:57:11.553 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:57:11 smithi047 conmon[78521]: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T01:57:11.553 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:57:11 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T01:57:11.554 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:57:11 smithi047 conmon[78521]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T01:57:11.554 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:57:11 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T01:57:11.554 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:57:11 smithi047 conmon[78521]: return self.event_loop.get_result(coro) 2022-05-14T01:57:11.554 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:57:11 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T01:57:11.554 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:57:11 smithi047 conmon[78521]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T01:57:11.555 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:57:11 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T01:57:11.555 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:57:11 smithi047 conmon[78521]: return self.__get_result() 2022-05-14T01:57:11.555 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:57:11 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T01:57:11.555 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:57:11 smithi047 conmon[78521]: raise self._exception 2022-05-14T01:57:11.556 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:57:11 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T01:57:11.556 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:57:11 smithi047 conmon[78521]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T01:57:11.556 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:57:11 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T01:57:11.556 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:57:11 smithi047 conmon[78521]: conn = await self._remote_connection(host, addr) 2022-05-14T01:57:11.556 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:57:11 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T01:57:11.557 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:57:11 smithi047 conmon[78521]: raise 2022-05-14T01:57:11.557 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:57:11 smithi047 conmon[78521]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T01:57:11.557 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:57:11 smithi047 conmon[78521]: self.gen.throw(type, value, traceback) 2022-05-14T01:57:11.557 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:57:11 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T01:57:11.558 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:57:11 smithi047 conmon[78521]: raise OrchestratorError(msg) 2022-05-14T01:57:11.558 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:57:11 smithi047 conmon[78521]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:57:11.773 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:57:11 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:57:12.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:57:11 smithi115 conmon[90393]: 2022-05-14T01:57:11.772+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:57:12.747 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:57:12 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:57:12.748 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:57:12 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:57:12.748 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T01:57:12.748 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:57:12.748 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T01:57:12.749 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:57:12.749 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:57:12.749 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:57:12.749 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T01:57:12.750 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:57:12.750 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T01:57:12.750 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:57:12.750 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T01:57:12.750 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:57:12.751 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T01:57:12.751 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:57:12.751 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T01:57:12.751 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:57:12.752 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:57:12.752 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:57:12.752 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:57:12 smithi115 ceph-mon[78593]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T01:57:12.752 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T01:57:12.752 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:57:12.753 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T01:57:12.753 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:57:12.753 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:57:12.753 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:57:12.754 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T01:57:12.754 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:57:12.754 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T01:57:12.754 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:57:12.754 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T01:57:12.755 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:57:12.755 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T01:57:12.755 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:57:12.755 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T01:57:12.756 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:57:12.756 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:57:12.756 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:57:12.756 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T01:57:12.756 INFO:journalctl@ceph.mon.c.smithi115.stdout: During handling of the above exception, another exception occurred: 2022-05-14T01:57:12.757 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T01:57:12.757 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T01:57:12.757 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T01:57:12.757 INFO:journalctl@ceph.mon.c.smithi115.stdout: return f(*arg) 2022-05-14T01:57:12.757 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T01:57:12.758 INFO:journalctl@ceph.mon.c.smithi115.stdout: self._write_client_files(client_files, host) 2022-05-14T01:57:12.758 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T01:57:12.758 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T01:57:12.759 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T01:57:12.759 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T01:57:12.759 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T01:57:12.759 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.event_loop.get_result(coro) 2022-05-14T01:57:12.759 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T01:57:12.760 INFO:journalctl@ceph.mon.c.smithi115.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T01:57:12.760 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T01:57:12.760 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.__get_result() 2022-05-14T01:57:12.760 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T01:57:12.761 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise self._exception 2022-05-14T01:57:12.761 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T01:57:12.761 INFO:journalctl@ceph.mon.c.smithi115.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T01:57:12.761 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T01:57:12.761 INFO:journalctl@ceph.mon.c.smithi115.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T01:57:12.762 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T01:57:12.762 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise 2022-05-14T01:57:12.762 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T01:57:12.762 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.gen.throw(type, value, traceback) 2022-05-14T01:57:12.762 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T01:57:12.763 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OrchestratorError(msg) 2022-05-14T01:57:12.763 INFO:journalctl@ceph.mon.c.smithi115.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:57:12.763 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:57:12 smithi115 ceph-mon[78593]: pgmap v4835: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:57:12.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:57:12 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:57:12.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:57:12 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:57:12.792 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T01:57:12.792 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:57:12.793 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T01:57:12.793 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:57:12.793 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:57:12.793 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:57:12.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T01:57:12.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:57:12.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T01:57:12.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:57:12.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T01:57:12.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:57:12.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T01:57:12.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:57:12.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T01:57:12.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:57:12.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:57:12.797 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:57:12.797 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:57:12 smithi047 ceph-mon[77147]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T01:57:12.797 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T01:57:12.797 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:57:12.798 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T01:57:12.798 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:57:12.798 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:57:12.798 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:57:12.798 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T01:57:12.799 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:57:12.799 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T01:57:12.799 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:57:12.799 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T01:57:12.800 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:57:12.800 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T01:57:12.800 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:57:12.800 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T01:57:12.800 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:57:12.801 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:57:12.801 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:57:12.801 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T01:57:12.801 INFO:journalctl@ceph.mon.b.smithi047.stdout: During handling of the above exception, another exception occurred: 2022-05-14T01:57:12.801 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T01:57:12.802 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T01:57:12.802 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T01:57:12.802 INFO:journalctl@ceph.mon.b.smithi047.stdout: return f(*arg) 2022-05-14T01:57:12.802 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T01:57:12.803 INFO:journalctl@ceph.mon.b.smithi047.stdout: self._write_client_files(client_files, host) 2022-05-14T01:57:12.803 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T01:57:12.803 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T01:57:12.803 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T01:57:12.803 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T01:57:12.804 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T01:57:12.804 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.event_loop.get_result(coro) 2022-05-14T01:57:12.804 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T01:57:12.804 INFO:journalctl@ceph.mon.b.smithi047.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T01:57:12.804 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T01:57:12.805 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.__get_result() 2022-05-14T01:57:12.805 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T01:57:12.805 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise self._exception 2022-05-14T01:57:12.805 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T01:57:12.806 INFO:journalctl@ceph.mon.b.smithi047.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T01:57:12.806 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T01:57:12.806 INFO:journalctl@ceph.mon.b.smithi047.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T01:57:12.806 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T01:57:12.806 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise 2022-05-14T01:57:12.807 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T01:57:12.807 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.gen.throw(type, value, traceback) 2022-05-14T01:57:12.807 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T01:57:12.807 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OrchestratorError(msg) 2022-05-14T01:57:12.808 INFO:journalctl@ceph.mon.b.smithi047.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:57:12.808 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:57:12 smithi047 ceph-mon[77147]: pgmap v4835: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:57:13.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:57:12 smithi115 conmon[90393]: 2022-05-14T01:57:12.746+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:57:13.773 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:57:13 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:57:13.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:57:13 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:57:14.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:57:13 smithi115 conmon[90393]: 2022-05-14T01:57:13.773+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:57:14.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:57:14 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:57:14.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:57:14 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:57:14.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:57:14 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:57:14.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:57:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T01:57:14.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:57:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T01:57:14.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:57:14 smithi047 ceph-mon[77147]: pgmap v4836: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:57:14.816 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:57:14 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:57:14.817 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:57:14 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:57:14.817 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:57:14 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:57:14.817 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:57:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T01:57:14.817 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:57:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T01:57:14.818 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:57:14 smithi115 ceph-mon[78593]: pgmap v4836: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:57:15.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:57:14 smithi115 conmon[90393]: 2022-05-14T01:57:14.815+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:57:15.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:57:15 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:57:15.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:57:15 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 8965 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:57:15.809 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:57:15 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:57:15.809 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:57:15 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 8965 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:57:16.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:57:15 smithi115 conmon[90393]: 2022-05-14T01:57:15.807+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:57:16.782 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:57:16 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:57:16.782 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:57:16 smithi115 ceph-mon[78593]: pgmap v4837: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:57:16.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:57:16 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:57:16.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:57:16 smithi047 ceph-mon[77147]: pgmap v4837: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:57:17.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:57:16 smithi115 conmon[90393]: 2022-05-14T01:57:16.780+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:57:17.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:57:17 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:57:17.804 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:57:17 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:57:18.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:57:17 smithi115 conmon[90393]: 2022-05-14T01:57:17.802+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:57:18.790 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:57:18 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:57:18.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:57:18 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:57:18.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:57:18 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:57:18.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:57:18 smithi115 ceph-mon[78593]: pgmap v4838: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:57:18.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:57:18 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:57:18.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:57:18 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:57:18.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:57:18 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:57:18.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:57:18 smithi047 ceph-mon[77147]: pgmap v4838: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:57:19.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:57:18 smithi115 conmon[90393]: 2022-05-14T01:57:18.788+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:57:19.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:57:19 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:57:19.832 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:57:19 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:57:20.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:57:19 smithi115 conmon[90393]: 2022-05-14T01:57:19.830+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:57:20.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:57:20 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:57:20.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:57:20 smithi047 ceph-mon[77147]: pgmap v4839: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:57:20.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:57:20 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 8970 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:57:20.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:57:20 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:57:20.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:57:20 smithi115 ceph-mon[78593]: pgmap v4839: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:57:20.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:57:20 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 8970 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:57:21.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:57:20 smithi115 conmon[90393]: 2022-05-14T01:57:20.795+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:57:21.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:57:21 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:57:21.800 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:57:21 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:57:22.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:57:21 smithi115 conmon[90393]: 2022-05-14T01:57:21.799+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:57:22.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:57:22 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:57:22.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:57:22 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:57:22.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:57:22 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:57:22.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:57:22 smithi047 ceph-mon[77147]: pgmap v4840: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:57:22.822 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:57:22 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:57:22.823 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:57:22 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:57:22.823 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:57:22 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:57:22.823 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:57:22 smithi115 ceph-mon[78593]: pgmap v4840: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:57:23.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:57:22 smithi115 conmon[90393]: 2022-05-14T01:57:22.820+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:57:23.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:57:23 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:57:23.846 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:57:23 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:57:24.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:57:23 smithi115 conmon[90393]: 2022-05-14T01:57:23.844+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:57:24.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:57:24 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:57:24.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:57:24 smithi047 ceph-mon[77147]: pgmap v4841: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:57:24.854 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:57:24 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:57:24.855 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:57:24 smithi115 ceph-mon[78593]: pgmap v4841: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:57:25.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:57:24 smithi115 conmon[90393]: 2022-05-14T01:57:24.852+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:57:25.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:57:25 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:57:25.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:57:25 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 8975 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:57:25.840 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:57:25 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:57:25.840 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:57:25 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 8975 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:57:26.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:57:25 smithi115 conmon[90393]: 2022-05-14T01:57:25.839+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:57:26.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:57:26 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:57:26.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:57:26 smithi047 ceph-mon[77147]: pgmap v4842: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:57:26.822 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:57:26 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:57:26.822 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:57:26 smithi115 ceph-mon[78593]: pgmap v4842: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:57:27.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:57:26 smithi115 conmon[90393]: 2022-05-14T01:57:26.820+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:57:27.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:57:27 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:57:27.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:57:27 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:57:28.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:57:27 smithi115 conmon[90393]: 2022-05-14T01:57:27.794+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:57:28.779 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:57:28 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:57:28.780 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:57:28 smithi115 ceph-mon[78593]: pgmap v4843: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:57:28.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:57:28 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:57:28.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:57:28 smithi047 ceph-mon[77147]: pgmap v4843: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:57:29.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:57:28 smithi115 conmon[90393]: 2022-05-14T01:57:28.777+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:57:29.750 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:57:29 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:57:29.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:57:29 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:57:30.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:57:29 smithi115 conmon[90393]: 2022-05-14T01:57:29.748+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:57:30.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:57:30 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:57:30.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:57:30 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:57:30.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:57:30 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:57:30.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:57:30 smithi047 ceph-mon[77147]: pgmap v4844: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:57:30.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:57:30 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 8980 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:57:30.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:57:30 smithi115 conmon[90393]: 2022-05-14T01:57:30.751+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:57:30.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:57:30 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:57:30.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:57:30 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:57:30.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:57:30 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:57:30.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:57:30 smithi115 ceph-mon[78593]: pgmap v4844: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:57:30.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:57:30 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 8980 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:57:31.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:57:31 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:57:31.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:57:31 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:57:31.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:57:31 smithi115 conmon[90393]: 2022-05-14T01:57:31.732+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:57:32.781 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:57:32 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:57:32.782 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:57:32 smithi115 ceph-mon[78593]: pgmap v4845: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:57:32.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:57:32 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:57:32.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:57:32 smithi047 ceph-mon[77147]: pgmap v4845: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:57:33.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:57:32 smithi115 conmon[90393]: 2022-05-14T01:57:32.779+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:57:33.780 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:57:33 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:57:33.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:57:33 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:57:34.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:57:33 smithi115 conmon[90393]: 2022-05-14T01:57:33.778+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:57:34.780 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:57:34 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:57:34.781 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:57:34 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:57:34.781 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:57:34 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:57:34.781 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:57:34 smithi115 ceph-mon[78593]: pgmap v4846: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:57:34.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:57:34 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:57:34.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:57:34 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:57:34.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:57:34 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:57:34.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:57:34 smithi047 ceph-mon[77147]: pgmap v4846: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:57:35.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:57:34 smithi115 conmon[90393]: 2022-05-14T01:57:34.779+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:57:35.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:57:35 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:57:35.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:57:35 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 8985 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:57:35.816 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:57:35 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:57:35.817 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:57:35 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 8985 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:57:36.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:57:35 smithi115 conmon[90393]: 2022-05-14T01:57:35.815+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:57:36.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:57:36 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:57:36.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:57:36 smithi047 ceph-mon[77147]: pgmap v4847: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:57:36.827 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:57:36 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:57:36.828 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:57:36 smithi115 ceph-mon[78593]: pgmap v4847: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:57:37.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:57:36 smithi115 conmon[90393]: 2022-05-14T01:57:36.823+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:57:37.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:57:37 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:57:37.821 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:57:37 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:57:38.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:57:37 smithi115 conmon[90393]: 2022-05-14T01:57:37.820+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:57:38.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:57:38 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:57:38.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:57:38 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:57:38.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:57:38 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:57:38.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:57:38 smithi047 ceph-mon[77147]: pgmap v4848: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:57:38.840 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:57:38 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:57:38.841 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:57:38 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:57:38.841 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:57:38 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:57:38.841 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:57:38 smithi115 ceph-mon[78593]: pgmap v4848: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:57:39.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:57:38 smithi115 conmon[90393]: 2022-05-14T01:57:38.839+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:57:39.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:57:39 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:57:40.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:57:39 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:57:40.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:57:39 smithi115 conmon[90393]: 2022-05-14T01:57:39.862+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:57:40.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:57:40 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:57:40.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:57:40 smithi115 ceph-mon[78593]: pgmap v4849: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:57:40.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:57:40 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 8990 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:57:41.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:57:40 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:57:41.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:57:40 smithi047 ceph-mon[77147]: pgmap v4849: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:57:41.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:57:40 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 8990 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:57:41.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:57:40 smithi115 conmon[90393]: 2022-05-14T01:57:40.892+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:57:41.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:57:41 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:57:42.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:57:41 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:57:42.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:57:41 smithi115 conmon[90393]: 2022-05-14T01:57:41.932+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:57:42.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:57:42 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:57:42.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:57:42 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:57:42.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:57:42 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:57:42.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:57:42 smithi115 ceph-mon[78593]: pgmap v4850: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:57:43.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:57:42 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:57:43.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:57:42 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:57:43.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:57:42 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:57:43.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:57:42 smithi047 ceph-mon[77147]: pgmap v4850: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:57:43.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:57:42 smithi115 conmon[90393]: 2022-05-14T01:57:42.895+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:57:43.850 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:57:43 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:57:44.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:57:43 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:57:44.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:57:43 smithi115 conmon[90393]: 2022-05-14T01:57:43.848+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:57:44.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:57:44 smithi115 conmon[90393]: 2022-05-14T01:57:44.809+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:57:44.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:57:44 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:57:44.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:57:44 smithi115 ceph-mon[78593]: pgmap v4851: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:57:45.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:57:44 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:57:45.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:57:44 smithi047 ceph-mon[77147]: pgmap v4851: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:57:45.825 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:57:45 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:57:45.826 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:57:45 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 8995 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:57:46.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:57:45 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:57:46.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:57:45 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 8995 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:57:46.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:57:45 smithi115 conmon[90393]: 2022-05-14T01:57:45.824+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:57:46.573 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:57:46 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:57:46.574 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:57:46 smithi115 ceph-mon[78593]: pgmap v4852: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:57:47.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:57:46 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:57:47.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:57:46 smithi047 ceph-mon[77147]: pgmap v4852: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:57:47.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:57:46 smithi115 conmon[90393]: 2022-05-14T01:57:46.821+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:57:47.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:57:47 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:57:47.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:57:47 smithi115 conmon[90393]: 2022-05-14T01:57:47.810+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:57:48.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:57:47 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:57:48.839 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:57:48 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:57:48.840 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:57:48 smithi115 ceph-mon[78593]: pgmap v4853: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:57:49.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:57:48 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:57:49.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:57:48 smithi047 ceph-mon[77147]: pgmap v4853: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:57:49.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:57:48 smithi115 conmon[90393]: 2022-05-14T01:57:48.838+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:57:49.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:57:49 smithi115 conmon[90393]: 2022-05-14T01:57:49.813+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:57:49.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:57:49 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:57:50.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:57:49 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:57:50.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:57:50 smithi115 conmon[90393]: 2022-05-14T01:57:50.797+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:57:50.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:57:50 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:57:50.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:57:50 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:57:50.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:57:50 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:57:50.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:57:50 smithi115 ceph-mon[78593]: pgmap v4854: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:57:50.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:57:50 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 9000 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:57:51.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:57:50 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:57:51.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:57:50 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:57:51.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:57:50 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:57:51.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:57:50 smithi047 ceph-mon[77147]: pgmap v4854: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:57:51.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:57:50 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 9000 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:57:51.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:57:51 smithi115 conmon[90393]: 2022-05-14T01:57:51.811+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:57:51.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:57:51 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:57:52.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:57:51 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:57:52.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:57:52 smithi115 conmon[90393]: 2022-05-14T01:57:52.782+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:57:52.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:57:52 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:57:52.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:57:52 smithi115 ceph-mon[78593]: pgmap v4855: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:57:53.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:57:52 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:57:53.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:57:52 smithi047 ceph-mon[77147]: pgmap v4855: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:57:53.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:57:53 smithi115 conmon[90393]: 2022-05-14T01:57:53.741+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:57:53.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:57:53 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:57:54.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:57:53 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:57:54.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:57:54 smithi115 conmon[90393]: 2022-05-14T01:57:54.743+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:57:54.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:57:54 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:57:54.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:57:54 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:57:54.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:57:54 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:57:54.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:57:54 smithi115 ceph-mon[78593]: pgmap v4856: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:57:55.048 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:57:54 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:57:55.049 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:57:54 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:57:55.049 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:57:54 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:57:55.049 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:57:54 smithi047 ceph-mon[77147]: pgmap v4856: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:57:55.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:57:55 smithi115 conmon[90393]: 2022-05-14T01:57:55.791+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:57:55.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:57:55 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:57:55.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:57:55 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 9005 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:57:56.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:57:55 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:57:56.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:57:55 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 9005 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:57:57.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:57:56 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:57:57.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:57:56 smithi047 ceph-mon[77147]: pgmap v4857: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:57:57.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:57:56 smithi115 conmon[90393]: 2022-05-14T01:57:56.778+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:57:57.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:57:56 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:57:57.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:57:56 smithi115 ceph-mon[78593]: pgmap v4857: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:57:58.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:57:57 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:57:58.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:57:57 smithi115 conmon[90393]: 2022-05-14T01:57:57.778+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:57:58.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:57:57 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:57:59.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:57:58 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:57:59.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:57:58 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:57:59.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:57:58 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:57:59.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:57:58 smithi047 ceph-mon[77147]: pgmap v4858: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:57:59.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:57:58 smithi115 conmon[90393]: 2022-05-14T01:57:58.802+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:57:59.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:57:58 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:57:59.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:57:58 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:57:59.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:57:58 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:57:59.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:57:58 smithi115 ceph-mon[78593]: pgmap v4858: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:58:00.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:57:59 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:58:00.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:57:59 smithi115 conmon[90393]: 2022-05-14T01:57:59.845+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:58:00.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:57:59 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:58:01.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:58:00 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:58:01.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:58:00 smithi047 ceph-mon[77147]: pgmap v4859: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:58:01.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:58:00 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 9010 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:58:01.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:58:00 smithi115 conmon[90393]: 2022-05-14T01:58:00.867+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:58:01.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:58:00 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:58:01.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:58:00 smithi115 ceph-mon[78593]: pgmap v4859: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:58:01.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:58:00 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 9010 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:58:02.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:58:01 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:58:02.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:58:01 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:58:02.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:58:01 smithi115 conmon[90393]: 2022-05-14T01:58:01.843+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:58:03.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:58:02 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:58:03.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:58:02 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:58:03.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:58:02 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:58:03.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:58:02 smithi047 ceph-mon[77147]: pgmap v4860: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:58:03.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:58:02 smithi115 conmon[90393]: 2022-05-14T01:58:02.796+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:58:03.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:58:02 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:58:03.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:58:02 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:58:03.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:58:02 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:58:03.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:58:02 smithi115 ceph-mon[78593]: pgmap v4860: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:58:04.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:58:03 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:58:04.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:58:03 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:58:04.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:58:03 smithi115 conmon[90393]: 2022-05-14T01:58:03.832+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:58:05.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:58:04 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:58:05.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:58:04 smithi047 ceph-mon[77147]: pgmap v4861: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:58:05.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:58:04 smithi115 conmon[90393]: 2022-05-14T01:58:04.838+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:58:05.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:58:04 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:58:05.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:58:04 smithi115 ceph-mon[78593]: pgmap v4861: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:58:06.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:58:05 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:58:06.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:58:05 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 9015 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:58:06.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:58:05 smithi115 conmon[90393]: 2022-05-14T01:58:05.828+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:58:06.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:58:05 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:58:06.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:58:05 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 9015 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:58:07.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:58:06 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:58:07.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:58:06 smithi047 ceph-mon[77147]: pgmap v4862: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:58:07.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:58:06 smithi115 conmon[90393]: 2022-05-14T01:58:06.812+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:58:07.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:58:06 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:58:07.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:58:06 smithi115 ceph-mon[78593]: pgmap v4862: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:58:08.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:58:07 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:58:08.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:58:07 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:58:08.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:58:07 smithi115 conmon[90393]: 2022-05-14T01:58:07.844+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:58:09.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:58:08 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:58:09.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:58:08 smithi047 ceph-mon[77147]: pgmap v4863: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:58:09.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:58:08 smithi115 conmon[90393]: 2022-05-14T01:58:08.870+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:58:09.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:58:08 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:58:09.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:58:08 smithi115 ceph-mon[78593]: pgmap v4863: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:58:10.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:58:09 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:58:10.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:58:09 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:58:10.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:58:09 smithi115 conmon[90393]: 2022-05-14T01:58:09.849+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:58:11.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:58:10 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:58:11.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:58:10 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:58:11.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:58:10 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:58:11.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:58:10 smithi047 ceph-mon[77147]: pgmap v4864: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:58:11.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:58:10 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 9020 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:58:11.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:58:10 smithi115 conmon[90393]: 2022-05-14T01:58:10.801+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:58:11.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:58:10 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:58:11.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:58:10 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:58:11.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:58:10 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:58:11.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:58:10 smithi115 ceph-mon[78593]: pgmap v4864: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:58:11.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:58:10 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 9020 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:58:12.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:58:11 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:58:12.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:58:11 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T01:58:12.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:58:11 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T01:58:12.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:58:11 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T01:58:12.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:58:11 smithi047 conmon[78521]: 2022-05-14T01:58:11.759+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:58:12.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:58:11 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T01:58:12.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:58:11 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:58:12.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:58:11 smithi047 conmon[78521]: yield 2022-05-14T01:58:12.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:58:11 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:58:12.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:58:11 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:58:12.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:58:11 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:58:12.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:58:11 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T01:58:12.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:58:11 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:58:12.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:58:11 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T01:58:12.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:58:11 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:58:12.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:58:11 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T01:58:12.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:58:11 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:58:12.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:58:11 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T01:58:12.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:58:11 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:58:12.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:58:11 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T01:58:12.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:58:11 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:58:12.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:58:11 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:58:12.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:58:11 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:58:12.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:58:11 smithi115 conmon[90393]: 2022-05-14T01:58:11.761+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:58:12.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:58:11 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:58:12.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:58:11 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T01:58:12.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:58:11 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T01:58:12.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:58:11 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T01:58:13.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:58:12 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:58:13.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:58:12 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:58:13.042 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T01:58:13.042 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:58:13.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T01:58:13.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:58:13.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:58:13.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:58:13.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T01:58:13.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:58:13.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T01:58:13.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:58:13.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T01:58:13.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:58:13.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T01:58:13.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:58:13.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T01:58:13.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:58:13.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:58:13.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:58:13.046 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:58:12 smithi047 ceph-mon[77147]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T01:58:13.047 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:58:12 smithi047 ceph-mon[77147]: pgmap v4865: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:58:13.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:58:12 smithi115 conmon[90393]: 2022-05-14T01:58:12.727+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:58:13.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:58:12 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:58:13.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:58:12 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:58:13.109 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T01:58:13.109 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:58:13.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T01:58:13.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:58:13.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:58:13.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:58:13.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T01:58:13.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:58:13.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T01:58:13.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:58:13.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T01:58:13.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:58:13.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T01:58:13.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:58:13.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T01:58:13.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:58:13.113 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:58:13.113 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:58:13.113 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:58:12 smithi115 ceph-mon[78593]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T01:58:13.113 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:58:12 smithi115 ceph-mon[78593]: pgmap v4865: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:58:14.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:58:13 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:58:14.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:58:13 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:58:14.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:58:13 smithi115 conmon[90393]: 2022-05-14T01:58:13.716+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:58:14.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:58:13 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:58:14.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:58:13 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:58:15.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:58:14 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:58:15.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:58:14 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:58:15.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:58:14 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:58:15.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:58:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T01:58:15.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:58:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T01:58:15.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:58:14 smithi047 ceph-mon[77147]: pgmap v4866: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:58:15.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:58:14 smithi047 conmon[78521]: 2022-05-14T01:58:14.895+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:58:15.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:58:14 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T01:58:15.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:58:14 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:58:15.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:58:14 smithi047 conmon[78521]: yield 2022-05-14T01:58:15.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:58:14 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:58:15.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:58:14 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:58:15.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:58:14 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:58:15.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:58:14 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T01:58:15.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:58:14 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:58:15.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:58:14 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T01:58:15.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:58:14 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:58:15.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:58:14 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T01:58:15.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:58:14 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:58:15.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:58:14 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T01:58:15.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:58:14 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:58:15.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:58:14 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T01:58:15.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:58:14 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:58:15.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:58:14 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:58:15.048 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:58:14 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:58:15.048 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:58:14 smithi047 conmon[78521]: 2022-05-14T01:58:14.896+0000 7f5340779700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T01:58:15.048 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:58:14 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T01:58:15.048 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:58:14 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:58:15.049 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:58:14 smithi047 conmon[78521]: yield 2022-05-14T01:58:15.049 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:58:14 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:58:15.049 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:58:14 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:58:15.049 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:58:14 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:58:15.050 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:58:14 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T01:58:15.050 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:58:14 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:58:15.050 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:58:14 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T01:58:15.050 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:58:14 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:58:15.051 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:58:14 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T01:58:15.051 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:58:14 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:58:15.051 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:58:14 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T01:58:15.051 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:58:14 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:58:15.051 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:58:14 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T01:58:15.052 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:58:14 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:58:15.052 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:58:14 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:58:15.052 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:58:14 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:58:15.052 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:58:14 smithi047 conmon[78521]: 2022-05-14T01:58:15.053 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:58:14 smithi047 conmon[78521]: During handling of the above exception, another exception occurred: 2022-05-14T01:58:15.053 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:58:14 smithi047 conmon[78521]: 2022-05-14T01:58:15.053 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:58:14 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T01:58:15.053 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:58:14 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T01:58:15.054 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:58:14 smithi047 conmon[78521]: return f(*arg) 2022-05-14T01:58:15.054 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:58:14 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T01:58:15.054 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:58:14 smithi047 conmon[78521]: self._write_client_files(client_files, host) 2022-05-14T01:58:15.054 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:58:14 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T01:58:15.054 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:58:14 smithi047 conmon[78521]: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T01:58:15.055 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:58:14 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T01:58:15.055 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:58:14 smithi047 conmon[78521]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T01:58:15.055 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:58:14 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T01:58:15.055 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:58:14 smithi047 conmon[78521]: return self.event_loop.get_result(coro) 2022-05-14T01:58:15.055 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:58:14 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T01:58:15.056 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:58:14 smithi047 conmon[78521]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T01:58:15.056 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:58:14 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T01:58:15.056 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:58:14 smithi047 conmon[78521]: return self.__get_result() 2022-05-14T01:58:15.056 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:58:14 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T01:58:15.057 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:58:14 smithi047 conmon[78521]: raise self._exception 2022-05-14T01:58:15.057 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:58:14 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T01:58:15.057 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:58:14 smithi047 conmon[78521]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T01:58:15.057 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:58:14 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T01:58:15.058 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:58:14 smithi047 conmon[78521]: conn = await self._remote_connection(host, addr) 2022-05-14T01:58:15.058 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:58:14 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T01:58:15.058 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:58:14 smithi047 conmon[78521]: raise 2022-05-14T01:58:15.058 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:58:14 smithi047 conmon[78521]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T01:58:15.058 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:58:14 smithi047 conmon[78521]: self.gen.throw(type, value, traceback) 2022-05-14T01:58:15.059 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:58:14 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T01:58:15.059 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:58:14 smithi047 conmon[78521]: raise OrchestratorError(msg) 2022-05-14T01:58:15.059 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:58:14 smithi047 conmon[78521]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:58:15.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:58:14 smithi115 conmon[90393]: 2022-05-14T01:58:14.722+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:58:15.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:58:14 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:58:15.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:58:14 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:58:15.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:58:14 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:58:15.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:58:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T01:58:15.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:58:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T01:58:15.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:58:14 smithi115 ceph-mon[78593]: pgmap v4866: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:58:16.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:58:15 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:58:16.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:58:15 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:58:16.042 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T01:58:16.042 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:58:16.042 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T01:58:16.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:58:16.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:58:16.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:58:16.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T01:58:16.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:58:16.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T01:58:16.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:58:16.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T01:58:16.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:58:16.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T01:58:16.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:58:16.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T01:58:16.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:58:16.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:58:16.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:58:16.046 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:58:15 smithi047 ceph-mon[77147]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T01:58:16.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T01:58:16.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:58:16.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T01:58:16.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:58:16.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:58:16.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:58:16.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T01:58:16.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:58:16.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T01:58:16.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:58:16.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T01:58:16.049 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:58:16.049 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T01:58:16.049 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:58:16.049 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T01:58:16.049 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:58:16.050 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:58:16.050 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:58:16.050 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T01:58:16.050 INFO:journalctl@ceph.mon.b.smithi047.stdout: During handling of the above exception, another exception occurred: 2022-05-14T01:58:16.050 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T01:58:16.051 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T01:58:16.051 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T01:58:16.051 INFO:journalctl@ceph.mon.b.smithi047.stdout: return f(*arg) 2022-05-14T01:58:16.051 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T01:58:16.052 INFO:journalctl@ceph.mon.b.smithi047.stdout: self._write_client_files(client_files, host) 2022-05-14T01:58:16.052 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T01:58:16.052 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T01:58:16.052 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T01:58:16.052 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T01:58:16.053 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T01:58:16.053 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.event_loop.get_result(coro) 2022-05-14T01:58:16.053 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T01:58:16.053 INFO:journalctl@ceph.mon.b.smithi047.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T01:58:16.053 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T01:58:16.054 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.__get_result() 2022-05-14T01:58:16.054 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T01:58:16.054 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise self._exception 2022-05-14T01:58:16.054 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T01:58:16.055 INFO:journalctl@ceph.mon.b.smithi047.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T01:58:16.055 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T01:58:16.055 INFO:journalctl@ceph.mon.b.smithi047.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T01:58:16.055 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T01:58:16.055 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise 2022-05-14T01:58:16.056 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T01:58:16.056 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.gen.throw(type, value, traceback) 2022-05-14T01:58:16.056 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T01:58:16.056 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OrchestratorError(msg) 2022-05-14T01:58:16.056 INFO:journalctl@ceph.mon.b.smithi047.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:58:16.057 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:58:15 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 9025 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:58:16.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:58:15 smithi115 conmon[90393]: 2022-05-14T01:58:15.678+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:58:16.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:58:15 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:58:16.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:58:15 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:58:16.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T01:58:16.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:58:16.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T01:58:16.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:58:16.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:58:16.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:58:16.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T01:58:16.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:58:16.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T01:58:16.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:58:16.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T01:58:16.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:58:16.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T01:58:16.113 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:58:16.113 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T01:58:16.113 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:58:16.113 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:58:16.113 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:58:16.114 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:58:15 smithi115 ceph-mon[78593]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T01:58:16.114 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T01:58:16.114 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:58:16.114 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T01:58:16.115 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:58:16.115 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:58:16.115 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:58:16.115 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T01:58:16.115 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:58:16.116 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T01:58:16.116 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:58:16.116 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T01:58:16.116 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:58:16.116 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T01:58:16.117 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:58:16.117 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T01:58:16.117 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:58:16.117 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:58:16.117 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:58:16.118 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T01:58:16.118 INFO:journalctl@ceph.mon.c.smithi115.stdout: During handling of the above exception, another exception occurred: 2022-05-14T01:58:16.118 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T01:58:16.118 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T01:58:16.119 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T01:58:16.119 INFO:journalctl@ceph.mon.c.smithi115.stdout: return f(*arg) 2022-05-14T01:58:16.119 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T01:58:16.119 INFO:journalctl@ceph.mon.c.smithi115.stdout: self._write_client_files(client_files, host) 2022-05-14T01:58:16.119 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T01:58:16.120 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T01:58:16.120 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T01:58:16.120 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T01:58:16.120 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T01:58:16.120 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.event_loop.get_result(coro) 2022-05-14T01:58:16.121 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T01:58:16.121 INFO:journalctl@ceph.mon.c.smithi115.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T01:58:16.121 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T01:58:16.121 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.__get_result() 2022-05-14T01:58:16.121 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T01:58:16.122 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise self._exception 2022-05-14T01:58:16.122 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T01:58:16.122 INFO:journalctl@ceph.mon.c.smithi115.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T01:58:16.122 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T01:58:16.122 INFO:journalctl@ceph.mon.c.smithi115.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T01:58:16.123 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T01:58:16.123 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise 2022-05-14T01:58:16.123 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T01:58:16.123 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.gen.throw(type, value, traceback) 2022-05-14T01:58:16.123 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T01:58:16.124 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OrchestratorError(msg) 2022-05-14T01:58:16.124 INFO:journalctl@ceph.mon.c.smithi115.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:58:16.124 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:58:15 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 9025 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:58:17.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:58:16 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:58:17.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:58:16 smithi047 ceph-mon[77147]: pgmap v4867: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:58:17.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:58:16 smithi115 conmon[90393]: 2022-05-14T01:58:16.667+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:58:17.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:58:16 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:58:17.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:58:16 smithi115 ceph-mon[78593]: pgmap v4867: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:58:18.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:58:17 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:58:18.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:58:17 smithi115 conmon[90393]: 2022-05-14T01:58:17.631+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:58:18.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:58:17 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:58:19.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:58:18 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:58:19.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:58:18 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:58:19.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:58:18 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:58:19.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:58:18 smithi047 ceph-mon[77147]: pgmap v4868: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:58:19.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:58:18 smithi115 conmon[90393]: 2022-05-14T01:58:18.624+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:58:19.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:58:18 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:58:19.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:58:18 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:58:19.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:58:18 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:58:19.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:58:18 smithi115 ceph-mon[78593]: pgmap v4868: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:58:20.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:58:19 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:58:20.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:58:19 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:58:20.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:58:19 smithi115 conmon[90393]: 2022-05-14T01:58:19.656+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:58:21.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:58:20 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:58:21.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:58:20 smithi047 ceph-mon[77147]: pgmap v4869: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:58:21.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:58:20 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 9030 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:58:21.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:58:20 smithi115 conmon[90393]: 2022-05-14T01:58:20.702+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:58:21.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:58:20 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:58:21.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:58:20 smithi115 ceph-mon[78593]: pgmap v4869: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:58:21.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:58:20 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 9030 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:58:22.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:58:21 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:58:22.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:58:21 smithi115 conmon[90393]: 2022-05-14T01:58:21.691+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:58:22.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:58:21 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:58:23.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:58:22 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:58:23.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:58:22 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:58:23.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:58:22 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:58:23.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:58:22 smithi047 ceph-mon[77147]: pgmap v4870: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:58:23.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:58:22 smithi115 conmon[90393]: 2022-05-14T01:58:22.698+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:58:23.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:58:22 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:58:23.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:58:22 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:58:23.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:58:22 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:58:23.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:58:22 smithi115 ceph-mon[78593]: pgmap v4870: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:58:24.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:58:23 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:58:24.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:58:23 smithi115 conmon[90393]: 2022-05-14T01:58:23.735+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:58:24.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:58:23 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:58:25.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:58:24 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:58:25.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:58:24 smithi047 ceph-mon[77147]: pgmap v4871: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:58:25.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:58:24 smithi115 conmon[90393]: 2022-05-14T01:58:24.772+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:58:25.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:58:24 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:58:25.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:58:24 smithi115 ceph-mon[78593]: pgmap v4871: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:58:26.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:58:25 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 9035 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:58:26.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:58:25 smithi115 conmon[90393]: 2022-05-14T01:58:25.762+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:58:26.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:58:25 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 9035 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:58:27.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:58:26 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:58:27.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:58:26 smithi047 ceph-mon[77147]: pgmap v4872: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:58:27.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:58:26 smithi115 conmon[90393]: 2022-05-14T01:58:26.717+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:58:27.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:58:26 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:58:27.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:58:26 smithi115 ceph-mon[78593]: pgmap v4872: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:58:28.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:58:27 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:58:28.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:58:27 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:58:28.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:58:27 smithi115 conmon[90393]: 2022-05-14T01:58:27.767+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:58:28.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:58:27 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:58:28.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:58:27 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:58:29.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:58:28 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:58:29.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:58:28 smithi047 ceph-mon[77147]: pgmap v4873: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:58:29.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:58:28 smithi115 conmon[90393]: 2022-05-14T01:58:28.756+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:58:29.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:58:28 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:58:29.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:58:28 smithi115 ceph-mon[78593]: pgmap v4873: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:58:30.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:58:29 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:58:30.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:58:29 smithi115 conmon[90393]: 2022-05-14T01:58:29.760+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:58:30.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:58:29 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:58:31.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:58:30 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:58:31.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:58:30 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:58:31.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:58:30 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:58:31.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:58:30 smithi047 ceph-mon[77147]: pgmap v4874: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:58:31.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:58:30 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 9040 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:58:31.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:58:30 smithi115 conmon[90393]: 2022-05-14T01:58:30.759+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:58:31.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:58:30 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:58:31.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:58:30 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:58:31.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:58:30 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:58:31.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:58:30 smithi115 ceph-mon[78593]: pgmap v4874: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:58:31.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:58:30 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 9040 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:58:32.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:58:31 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:58:32.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:58:31 smithi115 conmon[90393]: 2022-05-14T01:58:31.756+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:58:32.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:58:31 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:58:33.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:58:32 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:58:33.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:58:32 smithi047 ceph-mon[77147]: pgmap v4875: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:58:33.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:58:32 smithi115 conmon[90393]: 2022-05-14T01:58:32.762+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:58:33.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:58:32 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:58:33.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:58:32 smithi115 ceph-mon[78593]: pgmap v4875: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:58:34.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:58:33 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:58:34.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:58:33 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:58:34.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:58:33 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:58:34.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:58:33 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:58:34.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:58:33 smithi115 conmon[90393]: 2022-05-14T01:58:33.728+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:58:34.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:58:33 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:58:34.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:58:33 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:58:34.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:58:33 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:58:34.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:58:33 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:58:35.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:58:34 smithi047 ceph-mon[77147]: pgmap v4876: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:58:35.045 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:58:34 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:58:35.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:58:34 smithi115 conmon[90393]: 2022-05-14T01:58:34.696+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:58:35.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:58:34 smithi115 ceph-mon[78593]: pgmap v4876: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:58:35.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:58:34 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:58:36.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:58:35 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 9045 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:58:36.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:58:35 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:58:36.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:58:35 smithi115 conmon[90393]: 2022-05-14T01:58:35.697+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:58:36.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:58:35 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 9045 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:58:36.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:58:35 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:58:37.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:58:36 smithi047 ceph-mon[77147]: pgmap v4877: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:58:37.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:58:36 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:58:37.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:58:36 smithi115 conmon[90393]: 2022-05-14T01:58:36.657+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:58:37.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:58:36 smithi115 ceph-mon[78593]: pgmap v4877: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:58:37.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:58:36 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:58:38.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:58:37 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:58:38.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:58:37 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:58:38.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:58:37 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:58:38.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:58:37 smithi115 conmon[90393]: 2022-05-14T01:58:37.623+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:58:38.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:58:37 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:58:38.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:58:37 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:58:38.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:58:37 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:58:38.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:58:38 smithi115 conmon[90393]: 2022-05-14T01:58:38.578+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:58:38.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:58:38 smithi115 ceph-mon[78593]: pgmap v4878: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:58:38.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:58:38 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:58:39.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:58:38 smithi047 ceph-mon[77147]: pgmap v4878: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:58:39.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:58:38 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:58:39.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:58:39 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:58:39.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:58:39 smithi115 conmon[90393]: 2022-05-14T01:58:39.551+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:58:40.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:58:39 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:58:40.794 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:58:40 smithi115 conmon[90393]: 2022-05-14T01:58:40.514+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:58:41.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:58:40 smithi115 ceph-mon[78593]: pgmap v4879: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:58:41.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:58:40 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 9050 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:58:41.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:58:40 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:58:41.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:58:40 smithi047 ceph-mon[77147]: pgmap v4879: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:58:41.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:58:40 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 9050 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:58:41.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:58:40 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:58:41.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:58:41 smithi115 conmon[90393]: 2022-05-14T01:58:41.503+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:58:42.522 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:58:42 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:58:42.522 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:58:42 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:58:42.522 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:58:42 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:58:42.522 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:58:42 smithi115 ceph-mon[78593]: pgmap v4880: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:58:42.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:58:42 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:58:42.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:58:42 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:58:42.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:58:42 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:58:42.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:58:42 smithi047 ceph-mon[77147]: pgmap v4880: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:58:42.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:58:42 smithi115 conmon[90393]: 2022-05-14T01:58:42.520+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:58:43.521 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:58:43 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:58:43.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:58:43 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:58:43.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:58:43 smithi115 conmon[90393]: 2022-05-14T01:58:43.519+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:58:44.523 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:58:44 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:58:44.524 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:58:44 smithi115 ceph-mon[78593]: pgmap v4881: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:58:44.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:58:44 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:58:44.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:58:44 smithi047 ceph-mon[77147]: pgmap v4881: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:58:44.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:58:44 smithi115 conmon[90393]: 2022-05-14T01:58:44.522+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:58:45.534 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:58:45 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:58:45.534 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:58:45 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 9055 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:58:45.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:58:45 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:58:45.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:58:45 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 9055 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:58:45.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:58:45 smithi115 conmon[90393]: 2022-05-14T01:58:45.532+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:58:46.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:58:46 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:58:46.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:58:46 smithi047 ceph-mon[77147]: pgmap v4882: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:58:46.545 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:58:46 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:58:46.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:58:46 smithi115 ceph-mon[78593]: pgmap v4882: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:58:46.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:58:46 smithi115 conmon[90393]: 2022-05-14T01:58:46.544+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:58:47.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:58:47 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:58:47.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:58:47 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:58:47.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:58:47 smithi115 conmon[90393]: 2022-05-14T01:58:47.509+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:58:48.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:58:48 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:58:48.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:58:48 smithi047 ceph-mon[77147]: pgmap v4883: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:58:48.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:58:48 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:58:48.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:58:48 smithi115 ceph-mon[78593]: pgmap v4883: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:58:48.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:58:48 smithi115 conmon[90393]: 2022-05-14T01:58:48.502+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:58:49.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:58:49 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:58:49.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:58:49 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:58:49.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:58:49 smithi115 conmon[90393]: 2022-05-14T01:58:49.517+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:58:50.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:58:50 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:58:50.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:58:50 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:58:50.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:58:50 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:58:50.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:58:50 smithi047 ceph-mon[77147]: pgmap v4884: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:58:50.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:58:50 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 9060 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:58:50.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:58:50 smithi115 conmon[90393]: 2022-05-14T01:58:50.489+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:58:50.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:58:50 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:58:50.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:58:50 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:58:50.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:58:50 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:58:50.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:58:50 smithi115 ceph-mon[78593]: pgmap v4884: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:58:50.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:58:50 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 9060 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:58:51.539 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:58:51 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:58:51.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:58:51 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:58:51.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:58:51 smithi115 conmon[90393]: 2022-05-14T01:58:51.538+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:58:52.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:58:52 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:58:52.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:58:52 smithi047 ceph-mon[77147]: pgmap v4885: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:58:52.569 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:58:52 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:58:52.569 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:58:52 smithi115 ceph-mon[78593]: pgmap v4885: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:58:52.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:58:52 smithi115 conmon[90393]: 2022-05-14T01:58:52.568+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:58:53.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:58:53 smithi115 conmon[90393]: 2022-05-14T01:58:53.540+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:58:53.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:58:53 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:58:53.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:58:53 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:58:54.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:58:54 smithi115 conmon[90393]: 2022-05-14T01:58:54.493+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:58:54.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:58:54 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:58:54.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:58:54 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:58:54.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:58:54 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:58:54.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:58:54 smithi115 ceph-mon[78593]: pgmap v4886: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:58:54.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:58:54 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:58:54.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:58:54 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:58:54.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:58:54 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:58:54.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:58:54 smithi047 ceph-mon[77147]: pgmap v4886: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:58:55.610 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:58:55 smithi115 conmon[90393]: 2022-05-14T01:58:55.525+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:58:55.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:58:55 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:58:55.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:58:55 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 9065 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:58:55.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:58:55 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:58:55.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:58:55 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 9065 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:58:56.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:58:56 smithi115 conmon[90393]: 2022-05-14T01:58:56.548+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:58:56.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:58:56 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:58:56.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:58:56 smithi115 ceph-mon[78593]: pgmap v4887: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:58:56.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:58:56 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:58:56.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:58:56 smithi047 ceph-mon[77147]: pgmap v4887: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:58:57.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:58:57 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:58:57.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:58:57 smithi115 conmon[90393]: 2022-05-14T01:58:57.508+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:58:57.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:58:57 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:58:58.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:58:58 smithi115 conmon[90393]: 2022-05-14T01:58:58.508+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:58:58.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:58:58 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:58:58.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:58:58 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:58:58.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:58:58 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:58:58.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:58:58 smithi115 ceph-mon[78593]: pgmap v4888: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:58:58.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:58:58 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:58:58.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:58:58 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:58:58.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:58:58 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:58:58.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:58:58 smithi047 ceph-mon[77147]: pgmap v4888: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:58:59.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:58:59 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:58:59.610 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:58:59 smithi115 conmon[90393]: 2022-05-14T01:58:59.494+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:58:59.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:58:59 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:59:00.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:59:00 smithi115 conmon[90393]: 2022-05-14T01:59:00.495+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:59:00.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:59:00 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:59:00.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:59:00 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 9070 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:59:00.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:59:00 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:59:00.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:59:00 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 9070 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:59:01.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:59:01 smithi115 conmon[90393]: 2022-05-14T01:59:01.454+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:59:01.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:59:01 smithi115 ceph-mon[78593]: pgmap v4889: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:59:01.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:59:01 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:59:01.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:59:01 smithi047 ceph-mon[77147]: pgmap v4889: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:59:01.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:59:01 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:59:02.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:59:02 smithi115 conmon[90393]: 2022-05-14T01:59:02.434+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:59:02.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:59:02 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:59:02.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:59:02 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:59:02.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:59:02 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:59:02.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:59:02 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:59:02.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:59:02 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:59:02.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:59:02 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:59:03.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:59:03 smithi115 conmon[90393]: 2022-05-14T01:59:03.442+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:59:03.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:59:03 smithi115 ceph-mon[78593]: pgmap v4890: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:59:03.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:59:03 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:59:03.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:59:03 smithi047 ceph-mon[77147]: pgmap v4890: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:59:03.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:59:03 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:59:04.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:59:04 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:59:04.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:59:04 smithi115 conmon[90393]: 2022-05-14T01:59:04.472+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:59:04.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:59:04 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:59:05.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:59:05 smithi115 conmon[90393]: 2022-05-14T01:59:05.447+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:59:05.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:59:05 smithi115 ceph-mon[78593]: pgmap v4891: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:59:05.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:59:05 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:59:05.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:59:05 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 9075 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:59:05.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:59:05 smithi047 ceph-mon[77147]: pgmap v4891: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:59:05.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:59:05 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:59:05.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:59:05 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 9075 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:59:06.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:59:06 smithi115 conmon[90393]: 2022-05-14T01:59:06.470+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:59:06.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:59:06 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:59:06.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:59:06 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:59:07.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:59:07 smithi115 conmon[90393]: 2022-05-14T01:59:07.495+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:59:07.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:59:07 smithi115 ceph-mon[78593]: pgmap v4892: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:59:07.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:59:07 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:59:07.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:59:07 smithi047 ceph-mon[77147]: pgmap v4892: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:59:07.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:59:07 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:59:08.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:59:08 smithi115 conmon[90393]: 2022-05-14T01:59:08.453+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:59:08.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:59:08 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:59:08.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:59:08 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:59:09.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:59:09 smithi047 ceph-mon[77147]: pgmap v4893: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:59:09.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:59:09 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:59:09.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:59:09 smithi115 conmon[90393]: 2022-05-14T01:59:09.471+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:59:09.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:59:09 smithi115 ceph-mon[78593]: pgmap v4893: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:59:09.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:59:09 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:59:10.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:59:10 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:59:10.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:59:10 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:59:10.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:59:10 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:59:10.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:59:10 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 9080 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:59:10.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:59:10 smithi115 conmon[90393]: 2022-05-14T01:59:10.498+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:59:10.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:59:10 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:59:10.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:59:10 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:59:10.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:59:10 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:59:10.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:59:10 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 9080 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:59:11.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:59:11 smithi047 ceph-mon[77147]: pgmap v4894: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:59:11.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:59:11 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:59:11.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:59:11 smithi115 conmon[90393]: 2022-05-14T01:59:11.519+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:59:11.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:59:11 smithi115 ceph-mon[78593]: pgmap v4894: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:59:11.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:59:11 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:59:12.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:59:12 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:59:12.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:59:12 smithi115 conmon[90393]: 2022-05-14T01:59:12.515+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:59:12.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:59:12 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:59:13.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:59:13 smithi047 ceph-mon[77147]: pgmap v4895: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:59:13.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:59:13 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:59:13.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:59:13 smithi115 conmon[90393]: 2022-05-14T01:59:13.546+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:59:13.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:59:13 smithi115 ceph-mon[78593]: pgmap v4895: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:59:13.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:59:13 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:59:14.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:59:14 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:59:14.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:59:14 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:59:14.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:59:14 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:59:14.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:59:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T01:59:14.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:59:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T01:59:14.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:59:14 smithi115 conmon[90393]: 2022-05-14T01:59:14.542+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:59:14.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:59:14 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:59:14.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:59:14 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:59:14.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:59:14 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:59:14.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:59:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T01:59:14.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:59:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T01:59:15.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:59:15 smithi047 ceph-mon[77147]: pgmap v4896: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:59:15.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:59:15 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:59:15.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:59:15 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T01:59:15.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:59:15 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T01:59:15.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:59:15 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T01:59:15.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:59:15 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 9085 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:59:15.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:59:15 smithi115 conmon[90393]: 2022-05-14T01:59:15.581+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:59:15.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:59:15 smithi115 ceph-mon[78593]: pgmap v4896: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:59:15.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:59:15 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:59:15.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:59:15 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T01:59:15.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:59:15 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T01:59:15.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:59:15 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T01:59:15.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:59:15 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 9085 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:59:16.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:59:16 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:59:16.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:59:16 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:59:16.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:59:16 smithi115 conmon[90393]: 2022-05-14T01:59:16.570+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:59:17.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:59:17 smithi047 ceph-mon[77147]: pgmap v4897: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:59:17.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:59:17 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:59:17.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:59:17 smithi115 conmon[90393]: 2022-05-14T01:59:17.531+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:59:17.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:59:17 smithi115 ceph-mon[78593]: pgmap v4897: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:59:17.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:59:17 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:59:18.291 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:59:18 smithi047 conmon[78521]: 2022-05-14T01:59:18.000+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:59:18.292 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:59:18 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T01:59:18.292 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:59:18 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:59:18.292 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:59:18 smithi047 conmon[78521]: yield 2022-05-14T01:59:18.292 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:59:18 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:59:18.293 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:59:18 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:59:18.293 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:59:18 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:59:18.293 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:59:18 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T01:59:18.293 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:59:18 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:59:18.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:59:18 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T01:59:18.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:59:18 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:59:18.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:59:18 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T01:59:18.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:59:18 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:59:18.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:59:18 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T01:59:18.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:59:18 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:59:18.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:59:18 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T01:59:18.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:59:18 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:59:18.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:59:18 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:59:18.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:59:18 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:59:18.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:59:18 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:59:18.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:59:18 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:59:18.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:59:18 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:59:18.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:59:18 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:59:18.792 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T01:59:18.793 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:59:18.793 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T01:59:18.793 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:59:18.793 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:59:18.793 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:59:18.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T01:59:18.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:59:18.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T01:59:18.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:59:18.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T01:59:18.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:59:18.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T01:59:18.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:59:18.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T01:59:18.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:59:18.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:59:18.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:59:18.797 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:59:18 smithi047 ceph-mon[77147]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T01:59:18.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:59:18 smithi115 conmon[90393]: 2022-05-14T01:59:18.504+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:59:18.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:59:18 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:59:18.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:59:18 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:59:18.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:59:18 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:59:18.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:59:18 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:59:18.860 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T01:59:18.860 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:59:18.860 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T01:59:18.860 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:59:18.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:59:18.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:59:18.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T01:59:18.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:59:18.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T01:59:18.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:59:18.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T01:59:18.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:59:18.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T01:59:18.863 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:59:18.863 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T01:59:18.863 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:59:18.863 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:59:18.863 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:59:18.864 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:59:18 smithi115 ceph-mon[78593]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T01:59:19.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:59:19 smithi047 ceph-mon[77147]: pgmap v4898: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:59:19.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:59:19 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:59:19.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:59:19 smithi115 conmon[90393]: 2022-05-14T01:59:19.518+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:59:19.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:59:19 smithi115 ceph-mon[78593]: pgmap v4898: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:59:19.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:59:19 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:59:20.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:59:20 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:59:20.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:59:20 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 9090 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:59:20.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:59:20 smithi115 conmon[90393]: 2022-05-14T01:59:20.470+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:59:20.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:59:20 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:59:20.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:59:20 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 9090 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:59:21.418 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:59:21 smithi047 conmon[78521]: 2022-05-14T01:59:21.072+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:59:21.418 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:59:21 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T01:59:21.418 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:59:21 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:59:21.419 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:59:21 smithi047 conmon[78521]: yield 2022-05-14T01:59:21.419 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:59:21 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:59:21.419 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:59:21 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:59:21.419 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:59:21 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:59:21.420 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:59:21 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T01:59:21.420 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:59:21 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:59:21.420 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:59:21 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T01:59:21.420 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:59:21 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:59:21.421 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:59:21 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T01:59:21.421 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:59:21 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:59:21.421 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:59:21 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T01:59:21.421 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:59:21 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:59:21.422 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:59:21 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T01:59:21.422 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:59:21 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:59:21.422 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:59:21 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:59:21.422 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:59:21 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:59:21.422 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:59:21 smithi047 conmon[78521]: 2022-05-14T01:59:21.073+0000 7f5344f82700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T01:59:21.423 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:59:21 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T01:59:21.423 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:59:21 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:59:21.423 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:59:21 smithi047 conmon[78521]: yield 2022-05-14T01:59:21.423 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:59:21 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:59:21.424 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:59:21 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:59:21.424 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:59:21 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:59:21.424 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:59:21 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T01:59:21.424 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:59:21 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:59:21.425 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:59:21 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T01:59:21.425 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:59:21 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:59:21.425 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:59:21 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T01:59:21.426 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:59:21 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:59:21.426 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:59:21 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T01:59:21.426 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:59:21 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:59:21.426 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:59:21 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T01:59:21.426 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:59:21 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:59:21.427 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:59:21 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:59:21.427 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:59:21 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:59:21.427 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:59:21 smithi047 conmon[78521]: 2022-05-14T01:59:21.427 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:59:21 smithi047 conmon[78521]: During handling of the above exception, another exception occurred: 2022-05-14T01:59:21.428 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:59:21 smithi047 conmon[78521]: 2022-05-14T01:59:21.428 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:59:21 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T01:59:21.428 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:59:21 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T01:59:21.428 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:59:21 smithi047 conmon[78521]: return f(*arg) 2022-05-14T01:59:21.429 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:59:21 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T01:59:21.429 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:59:21 smithi047 conmon[78521]: self._write_client_files(client_files, host) 2022-05-14T01:59:21.429 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:59:21 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T01:59:21.429 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:59:21 smithi047 conmon[78521]: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T01:59:21.429 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:59:21 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T01:59:21.430 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:59:21 smithi047 conmon[78521]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T01:59:21.430 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:59:21 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T01:59:21.430 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:59:21 smithi047 conmon[78521]: return self.event_loop.get_result(coro) 2022-05-14T01:59:21.430 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:59:21 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T01:59:21.431 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:59:21 smithi047 conmon[78521]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T01:59:21.431 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:59:21 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T01:59:21.431 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:59:21 smithi047 conmon[78521]: return self.__get_result() 2022-05-14T01:59:21.431 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:59:21 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T01:59:21.432 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:59:21 smithi047 conmon[78521]: raise self._exception 2022-05-14T01:59:21.432 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:59:21 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T01:59:21.432 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:59:21 smithi047 conmon[78521]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T01:59:21.432 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:59:21 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T01:59:21.433 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:59:21 smithi047 conmon[78521]: conn = await self._remote_connection(host, addr) 2022-05-14T01:59:21.433 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:59:21 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T01:59:21.433 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:59:21 smithi047 conmon[78521]: raise 2022-05-14T01:59:21.433 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:59:21 smithi047 conmon[78521]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T01:59:21.434 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:59:21 smithi047 conmon[78521]: self.gen.throw(type, value, traceback) 2022-05-14T01:59:21.434 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:59:21 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T01:59:21.434 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:59:21 smithi047 conmon[78521]: raise OrchestratorError(msg) 2022-05-14T01:59:21.434 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 01:59:21 smithi047 conmon[78521]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:59:21.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:59:21 smithi047 ceph-mon[77147]: pgmap v4899: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:59:21.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:59:21 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:59:21.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:59:21 smithi115 conmon[90393]: 2022-05-14T01:59:21.471+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:59:21.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:59:21 smithi115 ceph-mon[78593]: pgmap v4899: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:59:21.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:59:21 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:59:22.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:59:22 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:59:22.792 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T01:59:22.792 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:59:22.792 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T01:59:22.793 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:59:22.793 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:59:22.793 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:59:22.793 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T01:59:22.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:59:22.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T01:59:22.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:59:22.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T01:59:22.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:59:22.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T01:59:22.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:59:22.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T01:59:22.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:59:22.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:59:22.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:59:22.796 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:59:22 smithi047 ceph-mon[77147]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T01:59:22.797 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T01:59:22.797 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:59:22.797 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T01:59:22.797 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:59:22.798 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:59:22.798 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:59:22.798 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T01:59:22.798 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:59:22.798 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T01:59:22.799 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:59:22.799 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T01:59:22.799 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:59:22.799 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T01:59:22.799 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:59:22.800 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T01:59:22.800 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:59:22.800 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:59:22.800 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:59:22.801 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T01:59:22.801 INFO:journalctl@ceph.mon.b.smithi047.stdout: During handling of the above exception, another exception occurred: 2022-05-14T01:59:22.801 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T01:59:22.801 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T01:59:22.801 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T01:59:22.802 INFO:journalctl@ceph.mon.b.smithi047.stdout: return f(*arg) 2022-05-14T01:59:22.802 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T01:59:22.802 INFO:journalctl@ceph.mon.b.smithi047.stdout: self._write_client_files(client_files, host) 2022-05-14T01:59:22.802 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T01:59:22.802 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T01:59:22.803 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T01:59:22.803 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T01:59:22.803 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T01:59:22.803 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.event_loop.get_result(coro) 2022-05-14T01:59:22.804 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T01:59:22.804 INFO:journalctl@ceph.mon.b.smithi047.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T01:59:22.804 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T01:59:22.804 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.__get_result() 2022-05-14T01:59:22.804 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T01:59:22.805 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise self._exception 2022-05-14T01:59:22.805 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T01:59:22.805 INFO:journalctl@ceph.mon.b.smithi047.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T01:59:22.805 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T01:59:22.805 INFO:journalctl@ceph.mon.b.smithi047.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T01:59:22.806 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T01:59:22.806 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise 2022-05-14T01:59:22.806 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T01:59:22.806 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.gen.throw(type, value, traceback) 2022-05-14T01:59:22.806 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T01:59:22.807 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OrchestratorError(msg) 2022-05-14T01:59:22.807 INFO:journalctl@ceph.mon.b.smithi047.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:59:22.807 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:59:22 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:59:22.807 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:59:22 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:59:22.807 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:59:22 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:59:22.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:59:22 smithi115 conmon[90393]: 2022-05-14T01:59:22.465+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:59:22.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:59:22 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:59:22.860 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T01:59:22.860 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:59:22.860 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T01:59:22.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:59:22.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:59:22.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:59:22.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T01:59:22.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:59:22.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T01:59:22.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:59:22.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T01:59:22.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:59:22.863 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T01:59:22.863 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:59:22.863 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T01:59:22.863 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:59:22.863 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:59:22.864 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:59:22.864 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:59:22 smithi115 ceph-mon[78593]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T01:59:22.864 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T01:59:22.864 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T01:59:22.864 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T01:59:22.865 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T01:59:22.865 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T01:59:22.865 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T01:59:22.865 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T01:59:22.866 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T01:59:22.866 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T01:59:22.866 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T01:59:22.866 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T01:59:22.866 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T01:59:22.867 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T01:59:22.867 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T01:59:22.867 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T01:59:22.867 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T01:59:22.867 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T01:59:22.868 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:59:22.868 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T01:59:22.868 INFO:journalctl@ceph.mon.c.smithi115.stdout: During handling of the above exception, another exception occurred: 2022-05-14T01:59:22.868 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T01:59:22.868 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T01:59:22.869 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T01:59:22.869 INFO:journalctl@ceph.mon.c.smithi115.stdout: return f(*arg) 2022-05-14T01:59:22.869 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T01:59:22.869 INFO:journalctl@ceph.mon.c.smithi115.stdout: self._write_client_files(client_files, host) 2022-05-14T01:59:22.869 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T01:59:22.870 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T01:59:22.870 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T01:59:22.870 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T01:59:22.870 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T01:59:22.870 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.event_loop.get_result(coro) 2022-05-14T01:59:22.871 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T01:59:22.871 INFO:journalctl@ceph.mon.c.smithi115.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T01:59:22.871 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T01:59:22.871 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.__get_result() 2022-05-14T01:59:22.871 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T01:59:22.872 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise self._exception 2022-05-14T01:59:22.872 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T01:59:22.872 INFO:journalctl@ceph.mon.c.smithi115.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T01:59:22.872 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T01:59:22.873 INFO:journalctl@ceph.mon.c.smithi115.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T01:59:22.873 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T01:59:22.873 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise 2022-05-14T01:59:22.873 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T01:59:22.873 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.gen.throw(type, value, traceback) 2022-05-14T01:59:22.874 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T01:59:22.874 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OrchestratorError(msg) 2022-05-14T01:59:22.874 INFO:journalctl@ceph.mon.c.smithi115.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T01:59:22.874 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:59:22 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:59:22.874 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:59:22 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:59:22.875 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:59:22 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:59:23.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:59:23 smithi047 ceph-mon[77147]: pgmap v4900: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:59:23.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:59:23 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:59:23.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:59:23 smithi115 conmon[90393]: 2022-05-14T01:59:23.447+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:59:23.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:59:23 smithi115 ceph-mon[78593]: pgmap v4900: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:59:23.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:59:23 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:59:24.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:59:24 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:59:24.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:59:24 smithi115 conmon[90393]: 2022-05-14T01:59:24.468+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:59:24.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:59:24 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:59:25.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:59:25 smithi047 ceph-mon[77147]: pgmap v4901: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:59:25.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:59:25 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:59:25.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:59:25 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 9095 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:59:25.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:59:25 smithi115 ceph-mon[78593]: pgmap v4901: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:59:25.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:59:25 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:59:25.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:59:25 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 9095 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:59:25.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:59:25 smithi115 conmon[90393]: 2022-05-14T01:59:25.455+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:59:26.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:59:26 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:59:26.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:59:26 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:59:26.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:59:26 smithi115 conmon[90393]: 2022-05-14T01:59:26.433+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:59:27.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:59:27 smithi047 ceph-mon[77147]: pgmap v4902: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:59:27.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:59:27 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:59:27.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:59:27 smithi115 conmon[90393]: 2022-05-14T01:59:27.412+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:59:27.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:59:27 smithi115 ceph-mon[78593]: pgmap v4902: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:59:27.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:59:27 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:59:28.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:59:28 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:59:28.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:59:28 smithi115 conmon[90393]: 2022-05-14T01:59:28.422+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:59:28.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:59:28 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:59:29.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:59:29 smithi047 ceph-mon[77147]: pgmap v4903: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:59:29.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:59:29 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:59:29.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:59:29 smithi115 conmon[90393]: 2022-05-14T01:59:29.435+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:59:29.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:59:29 smithi115 ceph-mon[78593]: pgmap v4903: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:59:29.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:59:29 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:59:30.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:59:30 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:59:30.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:59:30 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:59:30.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:59:30 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:59:30.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:59:30 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 9100 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:59:30.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:59:30 smithi115 conmon[90393]: 2022-05-14T01:59:30.391+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:59:30.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:59:30 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:59:30.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:59:30 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:59:30.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:59:30 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:59:30.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:59:30 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 9100 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:59:31.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:59:31 smithi047 ceph-mon[77147]: pgmap v4904: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:59:31.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:59:31 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:59:31.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:59:31 smithi115 conmon[90393]: 2022-05-14T01:59:31.439+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:59:31.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:59:31 smithi115 ceph-mon[78593]: pgmap v4904: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:59:31.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:59:31 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:59:32.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:59:32 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:59:32.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:59:32 smithi115 conmon[90393]: 2022-05-14T01:59:32.454+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:59:32.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:59:32 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:59:33.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:59:33 smithi047 ceph-mon[77147]: pgmap v4905: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:59:33.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:59:33 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:59:33.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:59:33 smithi115 conmon[90393]: 2022-05-14T01:59:33.468+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:59:33.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:59:33 smithi115 ceph-mon[78593]: pgmap v4905: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:59:33.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:59:33 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:59:34.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:59:34 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:59:34.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:59:34 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:59:34.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:59:34 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:59:34.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:59:34 smithi115 conmon[90393]: 2022-05-14T01:59:34.475+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:59:34.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:59:34 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:59:34.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:59:34 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:59:34.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:59:34 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:59:35.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:59:35 smithi047 ceph-mon[77147]: pgmap v4906: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:59:35.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:59:35 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:59:35.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:59:35 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 9105 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:59:35.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:59:35 smithi115 conmon[90393]: 2022-05-14T01:59:35.480+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:59:35.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:59:35 smithi115 ceph-mon[78593]: pgmap v4906: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:59:35.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:59:35 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:59:35.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:59:35 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 9105 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:59:36.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:59:36 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:59:36.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:59:36 smithi115 conmon[90393]: 2022-05-14T01:59:36.507+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:59:36.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:59:36 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:59:37.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:59:37 smithi047 ceph-mon[77147]: pgmap v4907: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:59:37.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:59:37 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:59:37.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:59:37 smithi115 conmon[90393]: 2022-05-14T01:59:37.503+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:59:37.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:59:37 smithi115 ceph-mon[78593]: pgmap v4907: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:59:37.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:59:37 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:59:38.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:59:38 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:59:38.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:59:38 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:59:38.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:59:38 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:59:38.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:59:38 smithi115 conmon[90393]: 2022-05-14T01:59:38.461+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:59:38.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:59:38 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:59:38.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:59:38 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:59:38.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:59:38 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:59:39.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:59:39 smithi047 ceph-mon[77147]: pgmap v4908: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:59:39.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:59:39 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:59:39.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:59:39 smithi115 conmon[90393]: 2022-05-14T01:59:39.463+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:59:39.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:59:39 smithi115 ceph-mon[78593]: pgmap v4908: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:59:39.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:59:39 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:59:40.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:59:40 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:59:40.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:59:40 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 9110 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:59:40.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:59:40 smithi115 conmon[90393]: 2022-05-14T01:59:40.509+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:59:40.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:59:40 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:59:40.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:59:40 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 9110 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:59:41.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:59:41 smithi047 ceph-mon[77147]: pgmap v4909: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:59:41.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:59:41 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:59:41.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:59:41 smithi115 conmon[90393]: 2022-05-14T01:59:41.500+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:59:41.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:59:41 smithi115 ceph-mon[78593]: pgmap v4909: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:59:41.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:59:41 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:59:42.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:59:42 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:59:42.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:59:42 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:59:42.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:59:42 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:59:42.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:59:42 smithi115 conmon[90393]: 2022-05-14T01:59:42.522+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:59:42.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:59:42 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:59:42.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:59:42 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:59:42.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:59:42 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:59:43.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:59:43 smithi047 ceph-mon[77147]: pgmap v4910: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:59:43.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:59:43 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:59:43.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:59:43 smithi115 conmon[90393]: 2022-05-14T01:59:43.510+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:59:43.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:59:43 smithi115 ceph-mon[78593]: pgmap v4910: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:59:43.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:59:43 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:59:44.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:59:44 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:59:44.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:59:44 smithi115 conmon[90393]: 2022-05-14T01:59:44.501+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:59:44.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:59:44 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:59:45.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:59:45 smithi047 ceph-mon[77147]: pgmap v4911: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:59:45.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:59:45 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:59:45.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:59:45 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 9115 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:59:45.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:59:45 smithi115 conmon[90393]: 2022-05-14T01:59:45.538+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:59:45.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:59:45 smithi115 ceph-mon[78593]: pgmap v4911: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:59:45.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:59:45 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:59:45.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:59:45 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 9115 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:59:46.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:59:46 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:59:46.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:59:46 smithi115 conmon[90393]: 2022-05-14T01:59:46.496+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:59:46.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:59:46 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:59:47.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:59:47 smithi047 ceph-mon[77147]: pgmap v4912: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:59:47.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:59:47 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:59:47.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:59:47 smithi115 conmon[90393]: 2022-05-14T01:59:47.455+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:59:47.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:59:47 smithi115 ceph-mon[78593]: pgmap v4912: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:59:47.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:59:47 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:59:48.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:59:48 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:59:48.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:59:48 smithi115 conmon[90393]: 2022-05-14T01:59:48.461+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:59:48.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:59:48 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:59:49.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:59:49 smithi047 ceph-mon[77147]: pgmap v4913: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:59:49.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:59:49 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:59:49.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:59:49 smithi115 conmon[90393]: 2022-05-14T01:59:49.456+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:59:49.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:59:49 smithi115 ceph-mon[78593]: pgmap v4913: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:59:49.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:59:49 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:59:50.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:59:50 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:59:50.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:59:50 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:59:50.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:59:50 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:59:50.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:59:50 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 9120 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:59:50.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:59:50 smithi115 conmon[90393]: 2022-05-14T01:59:50.484+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:59:50.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:59:50 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:59:50.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:59:50 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:59:50.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:59:50 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:59:50.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:59:50 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 9120 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:59:51.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:59:51 smithi047 ceph-mon[77147]: pgmap v4914: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:59:51.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:59:51 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:59:51.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:59:51 smithi115 conmon[90393]: 2022-05-14T01:59:51.465+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:59:51.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:59:51 smithi115 ceph-mon[78593]: pgmap v4914: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:59:51.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:59:51 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:59:52.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:59:52 smithi115 conmon[90393]: 2022-05-14T01:59:52.491+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:59:52.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:59:52 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:59:53.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:59:52 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:59:53.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:59:53 smithi115 conmon[90393]: 2022-05-14T01:59:53.450+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:59:53.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:59:53 smithi115 ceph-mon[78593]: pgmap v4915: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:59:53.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:59:53 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:59:54.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:59:53 smithi047 ceph-mon[77147]: pgmap v4915: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:59:54.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:59:53 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:59:54.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:59:54 smithi115 conmon[90393]: 2022-05-14T01:59:54.491+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:59:54.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:59:54 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:59:54.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:59:54 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:59:54.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:59:54 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:59:55.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:59:54 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:59:55.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:59:54 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:59:55.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:59:54 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:59:55.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:59:55 smithi115 conmon[90393]: 2022-05-14T01:59:55.450+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:59:55.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:59:55 smithi115 ceph-mon[78593]: pgmap v4916: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:59:55.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:59:55 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:59:55.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:59:55 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 9125 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:59:56.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:59:55 smithi047 ceph-mon[77147]: pgmap v4916: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:59:56.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:59:55 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:59:56.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:59:55 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 9125 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T01:59:56.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:59:56 smithi115 conmon[90393]: 2022-05-14T01:59:56.441+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:59:56.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:59:56 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:59:57.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:59:56 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:59:57.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:59:57 smithi115 conmon[90393]: 2022-05-14T01:59:57.459+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:59:57.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:59:57 smithi115 ceph-mon[78593]: pgmap v4917: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:59:57.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:59:57 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:59:58.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:59:57 smithi047 ceph-mon[77147]: pgmap v4917: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T01:59:58.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:59:57 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:59:58.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:59:58 smithi115 conmon[90393]: 2022-05-14T01:59:58.470+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:59:58.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:59:58 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:59:58.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:59:58 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:59:58.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:59:58 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:59:59.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:59:58 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T01:59:59.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:59:58 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T01:59:59.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:59:58 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T01:59:59.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 01:59:59 smithi115 conmon[90393]: 2022-05-14T01:59:59.498+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T01:59:59.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:59:59 smithi115 ceph-mon[78593]: pgmap v4918: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T01:59:59.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 01:59:59 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:00:00.001 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:59:59 smithi047 ceph-mon[77147]: pgmap v4918: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:00:00.001 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 01:59:59 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:00:00.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:00:00 smithi047 conmon[77123]: 2022-05-14T01:59:59.999+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : Health detail: HEALTH_ERR 1 filesystem is degraded; 1 mds daemon damaged; 1 MDSs are read only; 1 MDSs report slow metadata IOs; 1/3 mons down, quorum b,c; Reduced data availability: 1 pg inactive; 9 slow ops, oldest one blocked for 9130 sec, osd.5 has slow ops 2022-05-14T02:00:00.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:00:00 smithi047 conmon[77123]: 2022-05-14T01:59:59.999+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : [WRN] FS_DEGRADED: 1 filesystem is degraded 2022-05-14T02:00:00.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:00:00 smithi047 conmon[77123]: 2022-05-14T01:59:59.999+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : fs cephfs is degraded 2022-05-14T02:00:00.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:00:00 smithi047 conmon[77123]: 2022-05-14T01:59:59.999+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : [ERR] MDS_DAMAGE: 1 mds daemon damaged 2022-05-14T02:00:00.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:00:00 smithi047 conmon[77123]: 2022-05-14T01:59:59.999+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : fs cephfs mds.2 is damaged 2022-05-14T02:00:00.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:00:00 smithi047 conmon[77123]: 2022-05-14T01:59:59.999+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : [WRN] MDS_READ_ONLY: 1 MDSs are read only 2022-05-14T02:00:00.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:00:00 smithi047 conmon[77123]: 2022-05-14T01:59:59.999+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : mds.l(mds.0): MDS in read-only mode 2022-05-14T02:00:00.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:00:00 smithi047 conmon[77123]: 2022-05-14T01:59:59.999+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : [WRN] MDS_SLOW_METADATA_IO: 1 MDSs report slow metadata IOs 2022-05-14T02:00:00.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:00:00 smithi047 conmon[77123]: 2022-05-14T02:00:00.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:00:00 smithi047 conmon[77123]: 2022-05-14T01:59:59.999+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : mds.l(mds.0): 4 slow metadata IOs are blocked > 180 secs, oldest blocked for 9433 secs 2022-05-14T02:00:00.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:00:00 smithi047 conmon[77123]: 2022-05-14T01:59:59.999+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : [WRN] MON_DOWN: 1/3 mons down, quorum b,c 2022-05-14T02:00:00.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:00:00 smithi047 conmon[77123]: 2022-05-14T01:59:59.999+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : mon.a (rank 0) addr [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] is down (out of quorum) 2022-05-14T02:00:00.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:00:00 smithi047 conmon[77123]: 2022-05-14T01:59:59.999+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : [WRN] PG_AVAILABILITY: Reduced data availability: 1 pg inactive 2022-05-14T02:00:00.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:00:00 smithi047 conmon[77123]: 2022-05-14T01:59:59.999+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : pg 2.1 is stuck inactive for 2h, current state unknown, last acting [] 2022-05-14T02:00:00.295 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:00:00 smithi047 conmon[77123]: 2022-05-14T01:59:59.999+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : [WRN] SLOW_OPS: 9 slow ops, oldest one blocked for 9130 sec, osd.5 has slow ops 2022-05-14T02:00:00.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:00:00 smithi115 conmon[90393]: 2022-05-14T02:00:00.546+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:00:00.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:00:00 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:00:00.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:00:00 smithi115 ceph-mon[78593]: Health detail: HEALTH_ERR 1 filesystem is degraded; 1 mds daemon damaged; 1 MDSs are read only; 1 MDSs report slow metadata IOs; 1/3 mons down, quorum b,c; Reduced data availability: 1 pg inactive; 9 slow ops, oldest one blocked for 9130 sec, osd.5 has slow ops 2022-05-14T02:00:00.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:00:00 smithi115 ceph-mon[78593]: [WRN] FS_DEGRADED: 1 filesystem is degraded 2022-05-14T02:00:00.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:00:00 smithi115 ceph-mon[78593]: fs cephfs is degraded 2022-05-14T02:00:00.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:00:00 smithi115 ceph-mon[78593]: [ERR] MDS_DAMAGE: 1 mds daemon damaged 2022-05-14T02:00:00.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:00:00 smithi115 ceph-mon[78593]: fs cephfs mds.2 is damaged 2022-05-14T02:00:00.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:00:00 smithi115 ceph-mon[78593]: [WRN] MDS_READ_ONLY: 1 MDSs are read only 2022-05-14T02:00:00.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:00:00 smithi115 ceph-mon[78593]: mds.l(mds.0): MDS in read-only mode 2022-05-14T02:00:00.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:00:00 smithi115 ceph-mon[78593]: [WRN] MDS_SLOW_METADATA_IO: 1 MDSs report slow metadata IOs 2022-05-14T02:00:00.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:00:00 smithi115 ceph-mon[78593]: mds.l(mds.0): 4 slow metadata IOs are blocked > 180 secs, oldest blocked for 9433 secs 2022-05-14T02:00:00.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:00:00 smithi115 ceph-mon[78593]: [WRN] MON_DOWN: 1/3 mons down, quorum b,c 2022-05-14T02:00:00.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:00:00 smithi115 ceph-mon[78593]: mon.a (rank 0) addr [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] is down (out of quorum) 2022-05-14T02:00:00.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:00:00 smithi115 ceph-mon[78593]: [WRN] PG_AVAILABILITY: Reduced data availability: 1 pg inactive 2022-05-14T02:00:00.862 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:00:00 smithi115 ceph-mon[78593]: pg 2.1 is stuck inactive for 2h, current state unknown, last acting [] 2022-05-14T02:00:00.862 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:00:00 smithi115 ceph-mon[78593]: [WRN] SLOW_OPS: 9 slow ops, oldest one blocked for 9130 sec, osd.5 has slow ops 2022-05-14T02:00:00.862 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:00:00 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 9130 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:00:01.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:00:00 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:00:01.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:00:00 smithi047 ceph-mon[77147]: Health detail: HEALTH_ERR 1 filesystem is degraded; 1 mds daemon damaged; 1 MDSs are read only; 1 MDSs report slow metadata IOs; 1/3 mons down, quorum b,c; Reduced data availability: 1 pg inactive; 9 slow ops, oldest one blocked for 9130 sec, osd.5 has slow ops 2022-05-14T02:00:01.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:00:00 smithi047 ceph-mon[77147]: [WRN] FS_DEGRADED: 1 filesystem is degraded 2022-05-14T02:00:01.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:00:00 smithi047 ceph-mon[77147]: fs cephfs is degraded 2022-05-14T02:00:01.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:00:00 smithi047 ceph-mon[77147]: [ERR] MDS_DAMAGE: 1 mds daemon damaged 2022-05-14T02:00:01.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:00:00 smithi047 ceph-mon[77147]: fs cephfs mds.2 is damaged 2022-05-14T02:00:01.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:00:00 smithi047 ceph-mon[77147]: [WRN] MDS_READ_ONLY: 1 MDSs are read only 2022-05-14T02:00:01.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:00:00 smithi047 ceph-mon[77147]: mds.l(mds.0): MDS in read-only mode 2022-05-14T02:00:01.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:00:00 smithi047 ceph-mon[77147]: [WRN] MDS_SLOW_METADATA_IO: 1 MDSs report slow metadata IOs 2022-05-14T02:00:01.045 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:00:00 smithi047 ceph-mon[77147]: mds.l(mds.0): 4 slow metadata IOs are blocked > 180 secs, oldest blocked for 9433 secs 2022-05-14T02:00:01.045 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:00:00 smithi047 ceph-mon[77147]: [WRN] MON_DOWN: 1/3 mons down, quorum b,c 2022-05-14T02:00:01.045 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:00:00 smithi047 ceph-mon[77147]: mon.a (rank 0) addr [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] is down (out of quorum) 2022-05-14T02:00:01.045 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:00:00 smithi047 ceph-mon[77147]: [WRN] PG_AVAILABILITY: Reduced data availability: 1 pg inactive 2022-05-14T02:00:01.046 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:00:00 smithi047 ceph-mon[77147]: pg 2.1 is stuck inactive for 2h, current state unknown, last acting [] 2022-05-14T02:00:01.046 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:00:00 smithi047 ceph-mon[77147]: [WRN] SLOW_OPS: 9 slow ops, oldest one blocked for 9130 sec, osd.5 has slow ops 2022-05-14T02:00:01.046 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:00:00 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 9130 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:00:01.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:00:01 smithi115 conmon[90393]: 2022-05-14T02:00:01.538+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:00:01.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:00:01 smithi115 ceph-mon[78593]: pgmap v4919: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:00:01.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:00:01 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:00:02.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:00:01 smithi047 ceph-mon[77147]: pgmap v4919: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:00:02.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:00:01 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:00:02.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:00:02 smithi115 conmon[90393]: 2022-05-14T02:00:02.550+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:00:02.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:00:02 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:00:02.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:00:02 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:00:02.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:00:02 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:00:03.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:00:02 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:00:03.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:00:02 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:00:03.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:00:02 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:00:03.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:00:03 smithi115 conmon[90393]: 2022-05-14T02:00:03.576+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:00:03.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:00:03 smithi115 ceph-mon[78593]: pgmap v4920: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:00:03.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:00:03 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:00:04.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:00:03 smithi047 ceph-mon[77147]: pgmap v4920: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:00:04.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:00:03 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:00:04.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:00:04 smithi115 conmon[90393]: 2022-05-14T02:00:04.550+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:00:04.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:00:04 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:00:05.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:00:04 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:00:05.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:00:05 smithi115 conmon[90393]: 2022-05-14T02:00:05.520+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:00:05.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:00:05 smithi115 ceph-mon[78593]: pgmap v4921: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:00:05.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:00:05 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:00:05.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:00:05 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 9135 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:00:06.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:00:05 smithi047 ceph-mon[77147]: pgmap v4921: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:00:06.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:00:05 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:00:06.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:00:05 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 9135 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:00:06.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:00:06 smithi115 conmon[90393]: 2022-05-14T02:00:06.521+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:00:06.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:00:06 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:00:07.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:00:06 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:00:07.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:00:07 smithi115 conmon[90393]: 2022-05-14T02:00:07.480+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:00:07.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:00:07 smithi115 ceph-mon[78593]: pgmap v4922: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:00:07.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:00:07 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:00:08.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:00:07 smithi047 ceph-mon[77147]: pgmap v4922: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:00:08.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:00:07 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:00:08.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:00:08 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:00:08.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:00:08 smithi115 conmon[90393]: 2022-05-14T02:00:08.507+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:00:09.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:00:08 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:00:09.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:00:09 smithi115 conmon[90393]: 2022-05-14T02:00:09.498+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:00:09.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:00:09 smithi115 ceph-mon[78593]: pgmap v4923: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:00:09.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:00:09 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:00:10.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:00:09 smithi047 ceph-mon[77147]: pgmap v4923: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:00:10.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:00:09 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:00:10.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:00:10 smithi115 conmon[90393]: 2022-05-14T02:00:10.491+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:00:10.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:00:10 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:00:10.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:00:10 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:00:10.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:00:10 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:00:10.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:00:10 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 9140 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:00:11.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:00:10 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:00:11.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:00:10 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:00:11.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:00:10 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:00:11.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:00:10 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 9140 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:00:11.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:00:11 smithi115 conmon[90393]: 2022-05-14T02:00:11.537+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:00:11.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:00:11 smithi115 ceph-mon[78593]: pgmap v4924: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:00:11.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:00:11 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:00:12.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:00:11 smithi047 ceph-mon[77147]: pgmap v4924: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:00:12.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:00:11 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:00:12.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:00:12 smithi115 conmon[90393]: 2022-05-14T02:00:12.545+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:00:12.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:00:12 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:00:13.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:00:12 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:00:13.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:00:13 smithi115 conmon[90393]: 2022-05-14T02:00:13.579+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:00:13.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:00:13 smithi115 ceph-mon[78593]: pgmap v4925: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:00:13.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:00:13 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:00:14.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:00:13 smithi047 ceph-mon[77147]: pgmap v4925: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:00:14.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:00:13 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:00:15.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:00:14 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:00:15.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:00:14 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:00:15.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:00:14 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:00:15.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:00:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T02:00:15.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:00:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T02:00:15.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:00:14 smithi115 conmon[90393]: 2022-05-14T02:00:14.614+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:00:15.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:00:14 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:00:15.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:00:14 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:00:15.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:00:14 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:00:15.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:00:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T02:00:15.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:00:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T02:00:16.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:00:15 smithi047 ceph-mon[77147]: pgmap v4926: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:00:16.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:00:15 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:00:16.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:00:15 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 9145 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:00:16.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:00:15 smithi115 ceph-mon[78593]: pgmap v4926: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:00:16.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:00:15 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:00:16.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:00:15 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 9145 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:00:16.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:00:15 smithi115 conmon[90393]: 2022-05-14T02:00:15.627+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:00:17.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:00:16 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:00:17.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:00:16 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:00:17.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:00:16 smithi115 conmon[90393]: 2022-05-14T02:00:16.634+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:00:18.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:00:17 smithi047 ceph-mon[77147]: pgmap v4927: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:00:18.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:00:17 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:00:18.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:00:17 smithi115 conmon[90393]: 2022-05-14T02:00:17.669+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:00:18.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:00:17 smithi115 ceph-mon[78593]: pgmap v4927: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:00:18.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:00:17 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:00:19.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:00:18 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:00:19.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:00:18 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:00:19.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:00:18 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:00:19.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:00:18 smithi115 conmon[90393]: 2022-05-14T02:00:18.675+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:00:19.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:00:18 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:00:19.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:00:18 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:00:19.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:00:18 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:00:20.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:00:19 smithi047 ceph-mon[77147]: pgmap v4928: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:00:20.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:00:19 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:00:20.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:00:19 smithi115 conmon[90393]: 2022-05-14T02:00:19.687+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:00:20.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:00:19 smithi115 ceph-mon[78593]: pgmap v4928: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:00:20.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:00:19 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:00:21.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:00:20 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:00:21.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:00:20 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 9150 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:00:21.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:00:20 smithi115 conmon[90393]: 2022-05-14T02:00:20.729+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:00:21.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:00:20 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:00:21.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:00:20 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 9150 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:00:21.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:00:21 smithi115 conmon[90393]: 2022-05-14T02:00:21.758+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:00:21.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:00:21 smithi115 ceph-mon[78593]: pgmap v4929: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:00:21.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:00:21 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:00:21.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:00:21 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T02:00:21.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:00:21 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T02:00:21.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:00:21 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T02:00:21.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:00:21 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' 2022-05-14T02:00:21.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:00:21 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' 2022-05-14T02:00:22.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:00:21 smithi047 ceph-mon[77147]: pgmap v4929: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:00:22.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:00:21 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:00:22.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:00:21 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T02:00:22.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:00:21 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T02:00:22.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:00:21 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T02:00:22.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:00:21 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' 2022-05-14T02:00:22.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:00:21 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' 2022-05-14T02:00:23.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:00:22 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:00:23.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:00:22 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:00:23.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:00:22 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:00:23.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:00:22 smithi115 conmon[90393]: 2022-05-14T02:00:22.743+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:00:23.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:00:22 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:00:23.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:00:22 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:00:23.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:00:22 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:00:24.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:00:23 smithi047 ceph-mon[77147]: pgmap v4930: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:00:24.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:00:23 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:00:24.099 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:00:23 smithi115 conmon[90393]: 2022-05-14T02:00:23.707+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:00:24.100 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:00:23 smithi115 ceph-mon[78593]: pgmap v4930: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:00:24.100 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:00:23 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:00:24.542 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:00:24 smithi047 conmon[78521]: 2022-05-14T02:00:24.175+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:00:24.542 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:00:24 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T02:00:24.542 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:00:24 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:00:24.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:00:24 smithi047 conmon[78521]: yield 2022-05-14T02:00:24.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:00:24 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:00:24.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:00:24 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:00:24.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:00:24 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:00:24.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:00:24 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T02:00:24.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:00:24 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:00:24.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:00:24 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T02:00:24.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:00:24 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:00:24.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:00:24 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T02:00:24.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:00:24 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:00:24.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:00:24 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T02:00:24.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:00:24 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:00:24.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:00:24 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T02:00:24.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:00:24 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:00:24.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:00:24 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:00:24.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:00:24 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:00:24.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:00:24 smithi115 conmon[90393]: 2022-05-14T02:00:24.698+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:00:24.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:00:24 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:00:24.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:00:24 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' 2022-05-14T02:00:24.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:00:24 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' 2022-05-14T02:00:24.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:00:24 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-05-14T02:00:24.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:00:24 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-05-14T02:00:24.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:00:24 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-05-14T02:00:24.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:00:24 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-05-14T02:00:24.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:00:24 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-05-14T02:00:24.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:00:24 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-05-14T02:00:24.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:00:24 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-05-14T02:00:24.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:00:24 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-05-14T02:00:25.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:00:24 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:00:25.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:00:24 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' 2022-05-14T02:00:25.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:00:24 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' 2022-05-14T02:00:25.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:00:24 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-05-14T02:00:25.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:00:24 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-05-14T02:00:25.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:00:24 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-05-14T02:00:25.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:00:24 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-05-14T02:00:25.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:00:24 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-05-14T02:00:25.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:00:24 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-05-14T02:00:25.045 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:00:24 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-05-14T02:00:25.045 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:00:24 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-05-14T02:00:26.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:00:25 smithi047 ceph-mon[77147]: pgmap v4931: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:00:26.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:00:25 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:00:26.042 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T02:00:26.042 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:00:26.042 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T02:00:26.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:00:26.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:00:26.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:00:26.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T02:00:26.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:00:26.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T02:00:26.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:00:26.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T02:00:26.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:00:26.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T02:00:26.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:00:26.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T02:00:26.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:00:26.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:00:26.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:00:26.046 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:00:25 smithi047 ceph-mon[77147]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T02:00:26.046 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:00:25 smithi047 ceph-mon[77147]: Adjusting osd_memory_target on smithi115 to 197.0M 2022-05-14T02:00:26.047 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:00:25 smithi047 ceph-mon[77147]: Unable to set osd_memory_target on smithi115 to 206629785: error parsing value: Value '206629785' is below minimum 939524096 2022-05-14T02:00:26.047 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:00:25 smithi047 ceph-mon[77147]: Adjusting osd_memory_target on smithi047 to 198.1M 2022-05-14T02:00:26.047 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:00:25 smithi047 ceph-mon[77147]: Unable to set osd_memory_target on smithi047 to 207755878: error parsing value: Value '207755878' is below minimum 939524096 2022-05-14T02:00:26.047 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:00:25 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:00:26.047 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:00:25 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 9155 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:00:26.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:00:25 smithi115 conmon[90393]: 2022-05-14T02:00:25.691+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:00:26.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:00:25 smithi115 ceph-mon[78593]: pgmap v4931: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:00:26.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:00:25 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:00:26.109 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T02:00:26.109 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:00:26.109 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T02:00:26.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:00:26.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:00:26.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:00:26.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T02:00:26.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:00:26.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T02:00:26.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:00:26.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T02:00:26.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:00:26.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T02:00:26.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:00:26.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T02:00:26.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:00:26.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:00:26.113 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:00:26.113 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:00:25 smithi115 ceph-mon[78593]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T02:00:26.113 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:00:25 smithi115 ceph-mon[78593]: Adjusting osd_memory_target on smithi115 to 197.0M 2022-05-14T02:00:26.113 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:00:25 smithi115 ceph-mon[78593]: Unable to set osd_memory_target on smithi115 to 206629785: error parsing value: Value '206629785' is below minimum 939524096 2022-05-14T02:00:26.114 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:00:25 smithi115 ceph-mon[78593]: Adjusting osd_memory_target on smithi047 to 198.1M 2022-05-14T02:00:26.114 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:00:25 smithi115 ceph-mon[78593]: Unable to set osd_memory_target on smithi047 to 207755878: error parsing value: Value '207755878' is below minimum 939524096 2022-05-14T02:00:26.114 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:00:25 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:00:26.114 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:00:25 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 9155 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:00:27.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:00:26 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:00:27.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:00:26 smithi115 conmon[90393]: 2022-05-14T02:00:26.642+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:00:27.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:00:26 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:00:27.542 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:00:27 smithi047 conmon[78521]: 2022-05-14T02:00:27.247+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:00:27.542 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:00:27 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T02:00:27.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:00:27 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:00:27.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:00:27 smithi047 conmon[78521]: yield 2022-05-14T02:00:27.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:00:27 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:00:27.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:00:27 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:00:27.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:00:27 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:00:27.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:00:27 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T02:00:27.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:00:27 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:00:27.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:00:27 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T02:00:27.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:00:27 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:00:27.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:00:27 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T02:00:27.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:00:27 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:00:27.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:00:27 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T02:00:27.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:00:27 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:00:27.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:00:27 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T02:00:27.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:00:27 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:00:27.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:00:27 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:00:27.547 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:00:27 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:00:27.547 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:00:27 smithi047 conmon[78521]: 2022-05-14T02:00:27.248+0000 7f5344781700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T02:00:27.547 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:00:27 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T02:00:27.547 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:00:27 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:00:27.547 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:00:27 smithi047 conmon[78521]: yield 2022-05-14T02:00:27.548 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:00:27 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:00:27.548 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:00:27 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:00:27.548 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:00:27 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:00:27.548 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:00:27 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T02:00:27.549 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:00:27 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:00:27.549 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:00:27 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T02:00:27.549 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:00:27 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:00:27.549 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:00:27 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T02:00:27.550 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:00:27 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:00:27.550 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:00:27 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T02:00:27.550 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:00:27 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:00:27.550 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:00:27 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T02:00:27.550 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:00:27 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:00:27.551 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:00:27 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:00:27.551 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:00:27 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:00:27.551 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:00:27 smithi047 conmon[78521]: 2022-05-14T02:00:27.551 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:00:27 smithi047 conmon[78521]: During handling of the above exception, another exception occurred: 2022-05-14T02:00:27.552 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:00:27 smithi047 conmon[78521]: 2022-05-14T02:00:27.552 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:00:27 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T02:00:27.552 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:00:27 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T02:00:27.552 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:00:27 smithi047 conmon[78521]: return f(*arg) 2022-05-14T02:00:27.552 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:00:27 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T02:00:27.553 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:00:27 smithi047 conmon[78521]: self._write_client_files(client_files, host) 2022-05-14T02:00:27.553 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:00:27 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T02:00:27.553 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:00:27 smithi047 conmon[78521]: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T02:00:27.553 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:00:27 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T02:00:27.553 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:00:27 smithi047 conmon[78521]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T02:00:27.554 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:00:27 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T02:00:27.554 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:00:27 smithi047 conmon[78521]: return self.event_loop.get_result(coro) 2022-05-14T02:00:27.554 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:00:27 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T02:00:27.554 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:00:27 smithi047 conmon[78521]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T02:00:27.554 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:00:27 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T02:00:27.555 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:00:27 smithi047 conmon[78521]: return self.__get_result() 2022-05-14T02:00:27.555 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:00:27 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T02:00:27.555 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:00:27 smithi047 conmon[78521]: raise self._exception 2022-05-14T02:00:27.555 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:00:27 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T02:00:27.555 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:00:27 smithi047 conmon[78521]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T02:00:27.556 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:00:27 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T02:00:27.556 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:00:27 smithi047 conmon[78521]: conn = await self._remote_connection(host, addr) 2022-05-14T02:00:27.556 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:00:27 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T02:00:27.556 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:00:27 smithi047 conmon[78521]: raise 2022-05-14T02:00:27.556 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:00:27 smithi047 conmon[78521]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T02:00:27.557 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:00:27 smithi047 conmon[78521]: self.gen.throw(type, value, traceback) 2022-05-14T02:00:27.557 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:00:27 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T02:00:27.557 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:00:27 smithi047 conmon[78521]: raise OrchestratorError(msg) 2022-05-14T02:00:27.557 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:00:27 smithi047 conmon[78521]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:00:28.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:00:27 smithi047 ceph-mon[77147]: pgmap v4932: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:00:28.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:00:27 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:00:28.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:00:27 smithi115 conmon[90393]: 2022-05-14T02:00:27.685+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:00:28.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:00:27 smithi115 ceph-mon[78593]: pgmap v4932: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:00:28.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:00:27 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:00:29.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:00:28 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:00:29.042 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T02:00:29.042 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:00:29.042 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T02:00:29.042 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:00:29.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:00:29.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:00:29.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T02:00:29.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:00:29.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T02:00:29.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:00:29.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T02:00:29.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:00:29.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T02:00:29.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:00:29.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T02:00:29.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:00:29.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:00:29.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:00:29.047 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:00:28 smithi047 ceph-mon[77147]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T02:00:29.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T02:00:29.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:00:29.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T02:00:29.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:00:29.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:00:29.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:00:29.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T02:00:29.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:00:29.049 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T02:00:29.049 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:00:29.049 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T02:00:29.049 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:00:29.050 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T02:00:29.050 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:00:29.050 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T02:00:29.050 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:00:29.051 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:00:29.051 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:00:29.051 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T02:00:29.052 INFO:journalctl@ceph.mon.b.smithi047.stdout: During handling of the above exception, another exception occurred: 2022-05-14T02:00:29.052 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T02:00:29.052 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T02:00:29.052 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T02:00:29.053 INFO:journalctl@ceph.mon.b.smithi047.stdout: return f(*arg) 2022-05-14T02:00:29.053 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T02:00:29.053 INFO:journalctl@ceph.mon.b.smithi047.stdout: self._write_client_files(client_files, host) 2022-05-14T02:00:29.053 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T02:00:29.054 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T02:00:29.054 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T02:00:29.054 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T02:00:29.054 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T02:00:29.055 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.event_loop.get_result(coro) 2022-05-14T02:00:29.055 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T02:00:29.055 INFO:journalctl@ceph.mon.b.smithi047.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T02:00:29.055 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T02:00:29.056 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.__get_result() 2022-05-14T02:00:29.056 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T02:00:29.056 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise self._exception 2022-05-14T02:00:29.056 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T02:00:29.056 INFO:journalctl@ceph.mon.b.smithi047.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T02:00:29.057 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T02:00:29.057 INFO:journalctl@ceph.mon.b.smithi047.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T02:00:29.057 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T02:00:29.057 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise 2022-05-14T02:00:29.058 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T02:00:29.058 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.gen.throw(type, value, traceback) 2022-05-14T02:00:29.058 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T02:00:29.058 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OrchestratorError(msg) 2022-05-14T02:00:29.059 INFO:journalctl@ceph.mon.b.smithi047.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:00:29.059 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:00:28 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:00:29.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:00:28 smithi115 conmon[90393]: 2022-05-14T02:00:28.676+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:00:29.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:00:28 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:00:29.109 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T02:00:29.109 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:00:29.109 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T02:00:29.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:00:29.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:00:29.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:00:29.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T02:00:29.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:00:29.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T02:00:29.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:00:29.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T02:00:29.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:00:29.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T02:00:29.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:00:29.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T02:00:29.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:00:29.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:00:29.113 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:00:29.113 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:00:28 smithi115 ceph-mon[78593]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T02:00:29.113 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T02:00:29.113 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:00:29.113 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T02:00:29.114 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:00:29.114 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:00:29.114 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:00:29.114 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T02:00:29.114 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:00:29.115 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T02:00:29.115 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:00:29.115 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T02:00:29.115 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:00:29.115 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T02:00:29.116 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:00:29.116 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T02:00:29.116 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:00:29.116 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:00:29.116 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:00:29.117 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T02:00:29.117 INFO:journalctl@ceph.mon.c.smithi115.stdout: During handling of the above exception, another exception occurred: 2022-05-14T02:00:29.117 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T02:00:29.117 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T02:00:29.118 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T02:00:29.118 INFO:journalctl@ceph.mon.c.smithi115.stdout: return f(*arg) 2022-05-14T02:00:29.118 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T02:00:29.118 INFO:journalctl@ceph.mon.c.smithi115.stdout: self._write_client_files(client_files, host) 2022-05-14T02:00:29.118 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T02:00:29.119 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T02:00:29.119 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T02:00:29.119 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T02:00:29.119 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T02:00:29.119 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.event_loop.get_result(coro) 2022-05-14T02:00:29.120 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T02:00:29.120 INFO:journalctl@ceph.mon.c.smithi115.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T02:00:29.120 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T02:00:29.120 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.__get_result() 2022-05-14T02:00:29.120 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T02:00:29.121 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise self._exception 2022-05-14T02:00:29.121 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T02:00:29.121 INFO:journalctl@ceph.mon.c.smithi115.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T02:00:29.121 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T02:00:29.121 INFO:journalctl@ceph.mon.c.smithi115.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T02:00:29.122 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T02:00:29.122 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise 2022-05-14T02:00:29.122 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T02:00:29.122 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.gen.throw(type, value, traceback) 2022-05-14T02:00:29.123 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T02:00:29.123 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OrchestratorError(msg) 2022-05-14T02:00:29.123 INFO:journalctl@ceph.mon.c.smithi115.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:00:29.123 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:00:28 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:00:30.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:00:29 smithi047 ceph-mon[77147]: pgmap v4933: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:00:30.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:00:29 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:00:30.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:00:29 smithi115 conmon[90393]: 2022-05-14T02:00:29.707+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:00:30.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:00:29 smithi115 ceph-mon[78593]: pgmap v4933: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:00:30.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:00:29 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:00:31.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:00:30 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:00:31.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:00:30 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:00:31.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:00:30 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:00:31.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:00:30 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 9160 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:00:31.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:00:30 smithi115 conmon[90393]: 2022-05-14T02:00:30.683+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:00:31.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:00:30 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:00:31.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:00:30 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:00:31.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:00:30 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:00:31.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:00:30 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 9160 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:00:32.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:00:31 smithi047 ceph-mon[77147]: pgmap v4934: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:00:32.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:00:31 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:00:32.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:00:31 smithi115 conmon[90393]: 2022-05-14T02:00:31.715+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:00:32.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:00:31 smithi115 ceph-mon[78593]: pgmap v4934: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:00:32.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:00:31 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:00:33.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:00:32 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:00:33.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:00:32 smithi115 conmon[90393]: 2022-05-14T02:00:32.758+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:00:33.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:00:32 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:00:34.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:00:33 smithi047 ceph-mon[77147]: pgmap v4935: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:00:34.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:00:33 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:00:34.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:00:33 smithi115 conmon[90393]: 2022-05-14T02:00:33.781+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:00:34.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:00:33 smithi115 ceph-mon[78593]: pgmap v4935: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:00:34.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:00:33 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:00:35.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:00:34 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:00:35.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:00:34 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:00:35.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:00:34 smithi115 conmon[90393]: 2022-05-14T02:00:34.818+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:00:35.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:00:34 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:00:35.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:00:34 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:00:36.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:00:35 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:00:36.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:00:35 smithi047 ceph-mon[77147]: pgmap v4936: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:00:36.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:00:35 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 9165 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:00:36.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:00:35 smithi115 conmon[90393]: 2022-05-14T02:00:35.813+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:00:36.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:00:35 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:00:36.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:00:35 smithi115 ceph-mon[78593]: pgmap v4936: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:00:36.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:00:35 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 9165 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:00:37.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:00:36 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:00:37.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:00:36 smithi115 conmon[90393]: 2022-05-14T02:00:36.793+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:00:37.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:00:36 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:00:38.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:00:37 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:00:38.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:00:37 smithi047 ceph-mon[77147]: pgmap v4937: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:00:38.110 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:00:37 smithi115 conmon[90393]: 2022-05-14T02:00:37.797+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:00:38.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:00:37 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:00:38.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:00:37 smithi115 ceph-mon[78593]: pgmap v4937: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:00:39.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:00:38 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:00:39.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:00:38 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:00:39.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:00:38 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:00:39.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:00:38 smithi115 conmon[90393]: 2022-05-14T02:00:38.799+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:00:39.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:00:38 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:00:39.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:00:38 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:00:39.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:00:38 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:00:40.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:00:39 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:00:40.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:00:39 smithi047 ceph-mon[77147]: pgmap v4938: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:00:40.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:00:39 smithi115 conmon[90393]: 2022-05-14T02:00:39.831+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:00:40.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:00:39 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:00:40.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:00:39 smithi115 ceph-mon[78593]: pgmap v4938: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:00:41.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:00:40 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:00:41.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:00:40 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 9170 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:00:41.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:00:40 smithi115 conmon[90393]: 2022-05-14T02:00:40.874+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:00:41.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:00:40 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:00:41.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:00:40 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 9170 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:00:42.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:00:41 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:00:42.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:00:41 smithi047 ceph-mon[77147]: pgmap v4939: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:00:42.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:00:41 smithi115 conmon[90393]: 2022-05-14T02:00:41.877+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:00:42.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:00:41 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:00:42.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:00:41 smithi115 ceph-mon[78593]: pgmap v4939: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:00:43.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:00:42 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:00:43.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:00:42 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:00:43.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:00:42 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:00:43.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:00:42 smithi115 conmon[90393]: 2022-05-14T02:00:42.872+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:00:43.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:00:42 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:00:43.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:00:42 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:00:43.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:00:42 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:00:44.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:00:43 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:00:44.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:00:43 smithi047 ceph-mon[77147]: pgmap v4940: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:00:44.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:00:43 smithi115 conmon[90393]: 2022-05-14T02:00:43.835+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:00:44.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:00:43 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:00:44.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:00:43 smithi115 ceph-mon[78593]: pgmap v4940: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:00:45.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:00:44 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:00:45.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:00:44 smithi115 conmon[90393]: 2022-05-14T02:00:44.810+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:00:45.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:00:44 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:00:46.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:00:45 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:00:46.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:00:45 smithi047 ceph-mon[77147]: pgmap v4941: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:00:46.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:00:45 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 9175 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:00:46.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:00:45 smithi115 conmon[90393]: 2022-05-14T02:00:45.823+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:00:46.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:00:45 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:00:46.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:00:45 smithi115 ceph-mon[78593]: pgmap v4941: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:00:46.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:00:45 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 9175 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:00:47.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:00:46 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:00:47.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:00:46 smithi115 conmon[90393]: 2022-05-14T02:00:46.781+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:00:47.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:00:46 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:00:48.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:00:47 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:00:48.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:00:47 smithi047 ceph-mon[77147]: pgmap v4942: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:00:48.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:00:47 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:00:48.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:00:47 smithi115 conmon[90393]: 2022-05-14T02:00:47.829+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:00:48.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:00:47 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:00:48.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:00:47 smithi115 ceph-mon[78593]: pgmap v4942: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:00:48.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:00:47 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:00:49.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:00:48 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:00:49.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:00:48 smithi115 conmon[90393]: 2022-05-14T02:00:48.806+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:00:49.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:00:48 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:00:50.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:00:49 smithi047 ceph-mon[77147]: pgmap v4943: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:00:50.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:00:49 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:00:50.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:00:49 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:00:50.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:00:49 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:00:50.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:00:49 smithi115 conmon[90393]: 2022-05-14T02:00:49.781+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:00:50.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:00:49 smithi115 ceph-mon[78593]: pgmap v4943: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:00:50.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:00:49 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:00:50.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:00:49 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:00:50.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:00:49 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:00:51.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:00:50 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:00:51.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:00:50 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 9180 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:00:51.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:00:50 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:00:51.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:00:50 smithi115 conmon[90393]: 2022-05-14T02:00:50.745+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:00:51.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:00:50 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:00:51.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:00:50 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 9180 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:00:51.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:00:50 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:00:52.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:00:51 smithi047 ceph-mon[77147]: pgmap v4944: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:00:52.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:00:51 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:00:52.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:00:51 smithi115 conmon[90393]: 2022-05-14T02:00:51.711+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:00:52.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:00:51 smithi115 ceph-mon[78593]: pgmap v4944: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:00:52.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:00:51 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:00:53.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:00:52 smithi115 conmon[90393]: 2022-05-14T02:00:52.761+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:00:53.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:00:52 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:00:53.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:00:52 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:00:54.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:00:53 smithi115 conmon[90393]: 2022-05-14T02:00:53.763+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:00:54.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:00:53 smithi115 ceph-mon[78593]: pgmap v4945: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:00:54.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:00:53 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:00:54.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:00:53 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:00:54.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:00:53 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:00:54.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:00:53 smithi047 ceph-mon[77147]: pgmap v4945: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:00:54.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:00:53 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:00:54.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:00:53 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:00:54.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:00:53 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:00:55.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:00:54 smithi115 conmon[90393]: 2022-05-14T02:00:54.762+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:00:55.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:00:54 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:00:55.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:00:54 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:00:56.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:00:55 smithi115 conmon[90393]: 2022-05-14T02:00:55.805+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:00:56.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:00:55 smithi115 ceph-mon[78593]: pgmap v4946: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:00:56.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:00:55 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 9185 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:00:56.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:00:55 smithi047 ceph-mon[77147]: pgmap v4946: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:00:56.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:00:55 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 9185 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:00:57.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:00:56 smithi115 conmon[90393]: 2022-05-14T02:00:56.811+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:00:57.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:00:56 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:00:57.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:00:56 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:00:58.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:00:57 smithi115 conmon[90393]: 2022-05-14T02:00:57.830+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:00:58.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:00:57 smithi115 ceph-mon[78593]: pgmap v4947: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:00:58.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:00:57 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:00:58.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:00:57 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:00:58.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:00:57 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:00:58.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:00:57 smithi047 ceph-mon[77147]: pgmap v4947: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:00:58.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:00:57 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:00:58.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:00:57 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:00:58.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:00:57 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:00:59.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:00:58 smithi115 conmon[90393]: 2022-05-14T02:00:58.819+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:00:59.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:00:58 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:00:59.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:00:58 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:01:00.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:00:59 smithi115 conmon[90393]: 2022-05-14T02:00:59.777+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:01:00.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:00:59 smithi115 ceph-mon[78593]: pgmap v4948: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:01:00.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:00:59 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:01:00.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:00:59 smithi047 ceph-mon[77147]: pgmap v4948: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:01:00.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:00:59 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:01:01.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:01:00 smithi115 conmon[90393]: 2022-05-14T02:01:00.790+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:01:01.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:01:00 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:01:01.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:01:00 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 9190 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:01:01.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:01:00 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:01:01.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:01:00 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 9190 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:01:01.825 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:01:01 smithi115 conmon[90393]: 2022-05-14T02:01:01.751+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:01:02.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:01:01 smithi047 ceph-mon[77147]: pgmap v4949: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:01:02.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:01:01 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:01:02.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:01:01 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:01:02.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:01:01 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:01:02.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:01:01 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:01:02.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:01:01 smithi115 ceph-mon[78593]: pgmap v4949: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:01:02.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:01:01 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:01:02.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:01:01 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:01:02.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:01:01 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:01:02.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:01:01 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:01:03.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:01:02 smithi115 conmon[90393]: 2022-05-14T02:01:02.737+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:01:03.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:01:02 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:01:03.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:01:02 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:01:04.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:01:03 smithi115 conmon[90393]: 2022-05-14T02:01:03.753+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:01:04.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:01:03 smithi115 ceph-mon[78593]: pgmap v4950: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:01:04.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:01:03 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:01:04.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:01:03 smithi047 ceph-mon[77147]: pgmap v4950: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:01:04.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:01:03 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:01:05.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:01:04 smithi115 conmon[90393]: 2022-05-14T02:01:04.733+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:01:05.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:01:04 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:01:05.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:01:04 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:01:06.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:01:05 smithi115 conmon[90393]: 2022-05-14T02:01:05.702+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:01:06.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:01:05 smithi115 ceph-mon[78593]: pgmap v4951: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:01:06.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:01:05 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 9195 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:01:06.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:01:05 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:01:06.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:01:05 smithi047 ceph-mon[77147]: pgmap v4951: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:01:06.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:01:05 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 9195 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:01:06.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:01:05 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:01:07.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:01:06 smithi115 conmon[90393]: 2022-05-14T02:01:06.721+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:01:07.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:01:06 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:01:07.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:01:06 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:01:08.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:01:07 smithi115 conmon[90393]: 2022-05-14T02:01:07.725+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:01:08.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:01:07 smithi115 ceph-mon[78593]: pgmap v4952: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:01:08.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:01:07 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:01:08.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:01:07 smithi047 ceph-mon[77147]: pgmap v4952: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:01:08.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:01:07 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:01:09.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:01:08 smithi115 conmon[90393]: 2022-05-14T02:01:08.730+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:01:09.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:01:08 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:01:09.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:01:08 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:01:10.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:01:09 smithi115 conmon[90393]: 2022-05-14T02:01:09.740+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:01:10.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:01:09 smithi115 ceph-mon[78593]: pgmap v4953: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:01:10.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:01:09 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:01:10.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:01:09 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:01:10.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:01:09 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:01:10.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:01:09 smithi047 ceph-mon[77147]: pgmap v4953: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:01:10.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:01:09 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:01:10.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:01:09 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:01:10.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:01:09 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:01:11.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:01:10 smithi115 conmon[90393]: 2022-05-14T02:01:10.696+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:01:11.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:01:10 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 9200 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:01:11.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:01:10 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:01:11.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:01:10 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 9200 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:01:11.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:01:10 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:01:12.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:01:11 smithi115 conmon[90393]: 2022-05-14T02:01:11.653+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:01:12.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:01:11 smithi115 ceph-mon[78593]: pgmap v4954: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:01:12.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:01:11 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:01:12.296 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:01:11 smithi047 ceph-mon[77147]: pgmap v4954: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:01:12.297 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:01:11 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:01:13.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:01:12 smithi115 conmon[90393]: 2022-05-14T02:01:12.649+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:01:13.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:01:12 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:01:13.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:01:12 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:01:14.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:01:13 smithi115 conmon[90393]: 2022-05-14T02:01:13.694+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:01:14.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:01:13 smithi115 ceph-mon[78593]: pgmap v4955: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:01:14.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:01:13 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:01:14.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:01:13 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:01:14.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:01:13 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:01:14.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:01:13 smithi047 ceph-mon[77147]: pgmap v4955: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:01:14.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:01:13 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:01:14.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:01:13 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:01:14.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:01:13 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:01:15.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:01:14 smithi115 conmon[90393]: 2022-05-14T02:01:14.732+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:01:15.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:01:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T02:01:15.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:01:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T02:01:15.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:01:14 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:01:15.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:01:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T02:01:15.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:01:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T02:01:15.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:01:14 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:01:16.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:01:15 smithi115 conmon[90393]: 2022-05-14T02:01:15.737+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:01:16.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:01:15 smithi115 ceph-mon[78593]: pgmap v4956: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:01:16.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:01:15 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 9205 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:01:16.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:01:15 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:01:16.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:01:15 smithi047 ceph-mon[77147]: pgmap v4956: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:01:16.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:01:15 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 9205 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:01:16.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:01:15 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:01:17.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:01:16 smithi115 conmon[90393]: 2022-05-14T02:01:16.709+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:01:17.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:01:16 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:01:17.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:01:16 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:01:18.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:01:17 smithi115 conmon[90393]: 2022-05-14T02:01:17.752+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:01:18.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:01:17 smithi115 ceph-mon[78593]: pgmap v4957: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:01:18.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:01:17 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:01:18.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:01:17 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:01:18.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:01:17 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:01:18.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:01:17 smithi047 ceph-mon[77147]: pgmap v4957: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:01:18.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:01:17 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:01:18.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:01:17 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:01:18.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:01:17 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:01:19.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:01:18 smithi115 conmon[90393]: 2022-05-14T02:01:18.719+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:01:19.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:01:18 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:01:19.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:01:18 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:01:20.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:01:19 smithi115 conmon[90393]: 2022-05-14T02:01:19.671+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:01:20.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:01:19 smithi115 ceph-mon[78593]: pgmap v4958: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:01:20.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:01:19 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:01:20.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:01:19 smithi047 ceph-mon[77147]: pgmap v4958: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:01:20.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:01:19 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:01:21.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:01:20 smithi115 conmon[90393]: 2022-05-14T02:01:20.640+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:01:21.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:01:20 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 9210 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:01:21.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:01:20 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:01:21.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:01:20 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 9210 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:01:21.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:01:20 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:01:21.889 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:01:21 smithi115 conmon[90393]: 2022-05-14T02:01:21.637+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:01:22.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:01:21 smithi047 ceph-mon[77147]: pgmap v4959: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:01:22.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:01:21 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:01:22.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:01:21 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:01:22.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:01:21 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:01:22.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:01:21 smithi115 ceph-mon[78593]: pgmap v4959: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:01:22.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:01:21 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:01:22.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:01:21 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:01:22.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:01:21 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:01:22.894 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:01:22 smithi115 conmon[90393]: 2022-05-14T02:01:22.623+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:01:23.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:01:22 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:01:23.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:01:22 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:01:23.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:01:23 smithi115 conmon[90393]: 2022-05-14T02:01:23.594+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:01:24.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:01:23 smithi047 ceph-mon[77147]: pgmap v4960: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:01:24.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:01:23 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:01:24.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:01:23 smithi115 ceph-mon[78593]: pgmap v4960: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:01:24.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:01:23 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:01:24.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:01:24 smithi115 conmon[90393]: 2022-05-14T02:01:24.600+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:01:25.299 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:01:24 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:01:25.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:01:24 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:01:25.903 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:01:25 smithi115 conmon[90393]: 2022-05-14T02:01:25.634+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:01:26.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:01:25 smithi047 ceph-mon[77147]: pgmap v4961: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:01:26.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:01:25 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 9215 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:01:26.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:01:25 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:01:26.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:01:25 smithi115 ceph-mon[78593]: pgmap v4961: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:01:26.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:01:25 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 9215 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:01:26.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:01:25 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:01:26.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:01:26 smithi115 conmon[90393]: 2022-05-14T02:01:26.591+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:01:27.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:01:26 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:01:27.300 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:01:26 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:01:27.909 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:01:27 smithi115 conmon[90393]: 2022-05-14T02:01:27.608+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:01:28.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:01:27 smithi047 ceph-mon[77147]: pgmap v4962: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:01:28.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:01:27 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T02:01:28.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:01:27 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T02:01:28.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:01:27 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T02:01:28.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:01:27 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:01:28.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:01:27 smithi047 conmon[78521]: 2022-05-14T02:01:27.855+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:01:28.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:01:27 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T02:01:28.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:01:27 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:01:28.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:01:27 smithi047 conmon[78521]: yield 2022-05-14T02:01:28.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:01:27 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:01:28.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:01:27 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:01:28.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:01:27 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:01:28.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:01:27 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T02:01:28.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:01:27 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:01:28.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:01:27 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T02:01:28.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:01:27 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:01:28.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:01:27 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T02:01:28.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:01:27 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:01:28.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:01:27 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T02:01:28.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:01:27 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:01:28.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:01:27 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T02:01:28.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:01:27 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:01:28.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:01:27 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:01:28.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:01:27 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:01:28.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:01:27 smithi115 ceph-mon[78593]: pgmap v4962: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:01:28.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:01:27 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T02:01:28.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:01:27 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T02:01:28.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:01:27 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T02:01:28.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:01:27 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:01:28.915 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:01:28 smithi115 conmon[90393]: 2022-05-14T02:01:28.611+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:01:29.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:01:28 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:01:29.292 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T02:01:29.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:01:29.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T02:01:29.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:01:29.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:01:29.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:01:29.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T02:01:29.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:01:29.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T02:01:29.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:01:29.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T02:01:29.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:01:29.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T02:01:29.298 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:01:29.298 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T02:01:29.299 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:01:29.300 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:01:29.300 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:01:29.300 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:01:28 smithi047 ceph-mon[77147]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T02:01:29.301 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:01:28 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:01:29.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:01:28 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:01:29.358 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T02:01:29.359 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:01:29.359 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T02:01:29.359 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:01:29.359 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:01:29.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:01:29.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T02:01:29.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:01:29.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T02:01:29.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:01:29.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T02:01:29.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:01:29.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T02:01:29.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:01:29.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T02:01:29.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:01:29.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:01:29.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:01:29.363 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:01:28 smithi115 ceph-mon[78593]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T02:01:29.363 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:01:28 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:01:29.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:01:29 smithi115 conmon[90393]: 2022-05-14T02:01:29.575+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:01:30.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:01:29 smithi047 ceph-mon[77147]: pgmap v4963: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:01:30.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:01:29 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:01:30.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:01:29 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:01:30.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:01:29 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:01:30.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:01:29 smithi115 ceph-mon[78593]: pgmap v4963: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:01:30.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:01:29 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:01:30.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:01:29 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:01:30.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:01:29 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:01:30.921 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:01:30 smithi115 conmon[90393]: 2022-05-14T02:01:30.617+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:01:31.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:01:30 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 9220 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:01:31.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:01:30 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:01:31.292 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:01:30 smithi047 conmon[78521]: 2022-05-14T02:01:30.927+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:01:31.293 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:01:30 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T02:01:31.293 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:01:30 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:01:31.293 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:01:30 smithi047 conmon[78521]: yield 2022-05-14T02:01:31.293 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:01:30 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:01:31.293 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:01:30 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:01:31.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:01:30 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:01:31.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:01:30 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T02:01:31.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:01:30 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:01:31.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:01:30 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T02:01:31.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:01:30 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:01:31.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:01:30 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T02:01:31.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:01:30 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:01:31.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:01:30 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T02:01:31.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:01:30 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:01:31.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:01:30 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T02:01:31.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:01:30 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:01:31.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:01:30 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:01:31.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:01:30 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:01:31.297 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:01:30 smithi047 conmon[78521]: 2022-05-14T02:01:30.928+0000 7f5345f84700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T02:01:31.297 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:01:30 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T02:01:31.297 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:01:30 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:01:31.297 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:01:30 smithi047 conmon[78521]: yield 2022-05-14T02:01:31.297 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:01:30 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:01:31.298 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:01:30 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:01:31.298 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:01:30 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:01:31.298 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:01:30 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T02:01:31.298 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:01:30 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:01:31.299 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:01:30 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T02:01:31.299 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:01:30 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:01:31.299 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:01:30 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T02:01:31.299 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:01:30 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:01:31.299 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:01:30 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T02:01:31.300 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:01:30 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:01:31.300 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:01:30 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T02:01:31.300 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:01:30 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:01:31.300 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:01:30 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:01:31.300 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:01:30 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:01:31.301 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:01:30 smithi047 conmon[78521]: 2022-05-14T02:01:31.301 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:01:30 smithi047 conmon[78521]: During handling of the above exception, another exception occurred: 2022-05-14T02:01:31.301 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:01:30 smithi047 conmon[78521]: 2022-05-14T02:01:31.301 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:01:30 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T02:01:31.301 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:01:30 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T02:01:31.302 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:01:30 smithi047 conmon[78521]: return f(*arg) 2022-05-14T02:01:31.302 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:01:30 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T02:01:31.302 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:01:30 smithi047 conmon[78521]: self._write_client_files(client_files, host) 2022-05-14T02:01:31.302 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:01:30 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T02:01:31.302 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:01:30 smithi047 conmon[78521]: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T02:01:31.303 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:01:30 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T02:01:31.303 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:01:30 smithi047 conmon[78521]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T02:01:31.303 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:01:30 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T02:01:31.303 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:01:30 smithi047 conmon[78521]: return self.event_loop.get_result(coro) 2022-05-14T02:01:31.304 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:01:30 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T02:01:31.304 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:01:30 smithi047 conmon[78521]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T02:01:31.304 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:01:30 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T02:01:31.304 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:01:30 smithi047 conmon[78521]: return self.__get_result() 2022-05-14T02:01:31.304 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:01:30 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T02:01:31.305 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:01:30 smithi047 conmon[78521]: raise self._exception 2022-05-14T02:01:31.305 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:01:30 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T02:01:31.305 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:01:30 smithi047 conmon[78521]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T02:01:31.305 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:01:30 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T02:01:31.305 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:01:30 smithi047 conmon[78521]: conn = await self._remote_connection(host, addr) 2022-05-14T02:01:31.306 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:01:30 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T02:01:31.306 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:01:30 smithi047 conmon[78521]: raise 2022-05-14T02:01:31.306 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:01:30 smithi047 conmon[78521]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T02:01:31.306 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:01:30 smithi047 conmon[78521]: self.gen.throw(type, value, traceback) 2022-05-14T02:01:31.306 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:01:30 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T02:01:31.307 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:01:30 smithi047 conmon[78521]: raise OrchestratorError(msg) 2022-05-14T02:01:31.307 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:01:30 smithi047 conmon[78521]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:01:31.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:01:30 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 9220 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:01:31.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:01:30 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:01:31.923 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:01:31 smithi115 conmon[90393]: 2022-05-14T02:01:31.637+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:01:32.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:01:31 smithi047 ceph-mon[77147]: pgmap v4964: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 7 op/s 2022-05-14T02:01:32.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:01:31 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:01:32.292 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T02:01:32.292 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:01:32.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T02:01:32.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:01:32.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:01:32.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:01:32.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T02:01:32.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:01:32.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T02:01:32.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:01:32.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T02:01:32.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:01:32.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T02:01:32.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:01:32.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T02:01:32.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:01:32.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:01:32.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:01:32.297 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:01:31 smithi047 ceph-mon[77147]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T02:01:32.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T02:01:32.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:01:32.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T02:01:32.298 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:01:32.298 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:01:32.299 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:01:32.299 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T02:01:32.299 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:01:32.300 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T02:01:32.300 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:01:32.301 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T02:01:32.301 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:01:32.301 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T02:01:32.302 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:01:32.302 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T02:01:32.302 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:01:32.303 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:01:32.303 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:01:32.304 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T02:01:32.304 INFO:journalctl@ceph.mon.b.smithi047.stdout: During handling of the above exception, another exception occurred: 2022-05-14T02:01:32.304 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T02:01:32.305 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T02:01:32.305 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T02:01:32.305 INFO:journalctl@ceph.mon.b.smithi047.stdout: return f(*arg) 2022-05-14T02:01:32.306 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T02:01:32.306 INFO:journalctl@ceph.mon.b.smithi047.stdout: self._write_client_files(client_files, host) 2022-05-14T02:01:32.306 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T02:01:32.307 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T02:01:32.307 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T02:01:32.307 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T02:01:32.307 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T02:01:32.308 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.event_loop.get_result(coro) 2022-05-14T02:01:32.308 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T02:01:32.308 INFO:journalctl@ceph.mon.b.smithi047.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T02:01:32.309 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T02:01:32.309 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.__get_result() 2022-05-14T02:01:32.309 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T02:01:32.310 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise self._exception 2022-05-14T02:01:32.310 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T02:01:32.310 INFO:journalctl@ceph.mon.b.smithi047.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T02:01:32.311 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T02:01:32.311 INFO:journalctl@ceph.mon.b.smithi047.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T02:01:32.311 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T02:01:32.311 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise 2022-05-14T02:01:32.312 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T02:01:32.312 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.gen.throw(type, value, traceback) 2022-05-14T02:01:32.312 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T02:01:32.312 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OrchestratorError(msg) 2022-05-14T02:01:32.312 INFO:journalctl@ceph.mon.b.smithi047.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:01:32.313 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:01:31 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:01:32.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:01:31 smithi115 ceph-mon[78593]: pgmap v4964: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 7 op/s 2022-05-14T02:01:32.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:01:31 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:01:32.359 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T02:01:32.359 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:01:32.359 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T02:01:32.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:01:32.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:01:32.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:01:32.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T02:01:32.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:01:32.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T02:01:32.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:01:32.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T02:01:32.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:01:32.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T02:01:32.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:01:32.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T02:01:32.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:01:32.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:01:32.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:01:32.363 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:01:31 smithi115 ceph-mon[78593]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T02:01:32.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T02:01:32.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:01:32.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T02:01:32.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:01:32.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:01:32.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:01:32.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T02:01:32.365 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:01:32.365 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T02:01:32.365 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:01:32.365 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T02:01:32.366 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:01:32.366 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T02:01:32.366 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:01:32.366 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T02:01:32.367 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:01:32.367 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:01:32.367 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:01:32.367 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T02:01:32.367 INFO:journalctl@ceph.mon.c.smithi115.stdout: During handling of the above exception, another exception occurred: 2022-05-14T02:01:32.368 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T02:01:32.368 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T02:01:32.368 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T02:01:32.368 INFO:journalctl@ceph.mon.c.smithi115.stdout: return f(*arg) 2022-05-14T02:01:32.369 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T02:01:32.369 INFO:journalctl@ceph.mon.c.smithi115.stdout: self._write_client_files(client_files, host) 2022-05-14T02:01:32.369 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T02:01:32.369 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T02:01:32.370 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T02:01:32.370 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T02:01:32.370 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T02:01:32.370 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.event_loop.get_result(coro) 2022-05-14T02:01:32.370 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T02:01:32.371 INFO:journalctl@ceph.mon.c.smithi115.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T02:01:32.371 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T02:01:32.371 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.__get_result() 2022-05-14T02:01:32.371 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T02:01:32.372 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise self._exception 2022-05-14T02:01:32.372 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T02:01:32.372 INFO:journalctl@ceph.mon.c.smithi115.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T02:01:32.372 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T02:01:32.372 INFO:journalctl@ceph.mon.c.smithi115.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T02:01:32.373 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T02:01:32.373 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise 2022-05-14T02:01:32.373 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T02:01:32.373 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.gen.throw(type, value, traceback) 2022-05-14T02:01:32.374 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T02:01:32.374 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OrchestratorError(msg) 2022-05-14T02:01:32.374 INFO:journalctl@ceph.mon.c.smithi115.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:01:32.374 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:01:31 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:01:32.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:01:32 smithi115 conmon[90393]: 2022-05-14T02:01:32.607+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:01:33.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:01:32 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:01:33.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:01:32 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:01:33.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:01:33 smithi115 conmon[90393]: 2022-05-14T02:01:33.568+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:01:34.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:01:33 smithi047 ceph-mon[77147]: pgmap v4965: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 4 op/s 2022-05-14T02:01:34.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:01:33 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:01:34.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:01:33 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:01:34.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:01:33 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:01:34.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:01:33 smithi115 ceph-mon[78593]: pgmap v4965: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 4 op/s 2022-05-14T02:01:34.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:01:33 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:01:34.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:01:33 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:01:34.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:01:33 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:01:34.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:01:34 smithi115 conmon[90393]: 2022-05-14T02:01:34.578+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:01:35.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:01:34 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:01:35.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:01:34 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:01:35.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:01:35 smithi115 conmon[90393]: 2022-05-14T02:01:35.606+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:01:36.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:01:35 smithi047 ceph-mon[77147]: pgmap v4966: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 4 op/s 2022-05-14T02:01:36.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:01:35 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 9225 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:01:36.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:01:35 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:01:36.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:01:35 smithi115 ceph-mon[78593]: pgmap v4966: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 4 op/s 2022-05-14T02:01:36.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:01:35 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 9225 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:01:36.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:01:35 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:01:36.943 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:01:36 smithi115 conmon[90393]: 2022-05-14T02:01:36.628+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:01:37.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:01:36 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:01:37.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:01:36 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:01:37.942 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:01:37 smithi115 conmon[90393]: 2022-05-14T02:01:37.647+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:01:38.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:01:37 smithi047 ceph-mon[77147]: pgmap v4967: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 7 op/s 2022-05-14T02:01:38.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:01:37 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:01:38.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:01:37 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:01:38.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:01:37 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:01:38.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:01:37 smithi115 ceph-mon[78593]: pgmap v4967: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 7 op/s 2022-05-14T02:01:38.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:01:37 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:01:38.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:01:37 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:01:38.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:01:37 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:01:38.949 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:01:38 smithi115 conmon[90393]: 2022-05-14T02:01:38.679+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:01:39.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:01:38 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:01:39.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:01:38 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:01:39.948 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:01:39 smithi115 conmon[90393]: 2022-05-14T02:01:39.675+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:01:40.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:01:39 smithi047 ceph-mon[77147]: pgmap v4968: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 4 op/s 2022-05-14T02:01:40.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:01:39 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:01:40.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:01:39 smithi115 ceph-mon[78593]: pgmap v4968: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 4 op/s 2022-05-14T02:01:40.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:01:39 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:01:40.952 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:01:40 smithi115 conmon[90393]: 2022-05-14T02:01:40.635+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:01:41.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:01:40 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 9230 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:01:41.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:01:40 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:01:41.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:01:40 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 9230 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:01:41.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:01:40 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:01:41.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:01:41 smithi115 conmon[90393]: 2022-05-14T02:01:41.606+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:01:42.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:01:41 smithi047 ceph-mon[77147]: pgmap v4969: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 7 op/s 2022-05-14T02:01:42.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:01:41 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:01:42.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:01:41 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:01:42.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:01:41 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:01:42.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:01:41 smithi115 ceph-mon[78593]: pgmap v4969: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 7 op/s 2022-05-14T02:01:42.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:01:41 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:01:42.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:01:41 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:01:42.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:01:41 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:01:42.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:01:42 smithi115 conmon[90393]: 2022-05-14T02:01:42.569+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:01:43.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:01:42 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:01:43.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:01:42 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:01:43.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:01:43 smithi115 conmon[90393]: 2022-05-14T02:01:43.594+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:01:44.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:01:43 smithi047 ceph-mon[77147]: pgmap v4970: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:01:44.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:01:43 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:01:44.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:01:43 smithi115 ceph-mon[78593]: pgmap v4970: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:01:44.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:01:43 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:01:44.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:01:44 smithi115 conmon[90393]: 2022-05-14T02:01:44.592+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:01:45.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:01:44 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:01:45.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:01:44 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:01:45.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:01:45 smithi115 conmon[90393]: 2022-05-14T02:01:45.606+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:01:46.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:01:45 smithi047 ceph-mon[77147]: pgmap v4971: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:01:46.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:01:45 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 9235 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:01:46.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:01:45 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:01:46.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:01:45 smithi115 ceph-mon[78593]: pgmap v4971: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:01:46.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:01:45 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 9235 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:01:46.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:01:45 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:01:46.975 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:01:46 smithi115 conmon[90393]: 2022-05-14T02:01:46.623+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:01:47.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:01:46 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:01:47.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:01:46 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:01:47.975 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:01:47 smithi115 conmon[90393]: 2022-05-14T02:01:47.630+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:01:48.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:01:47 smithi047 ceph-mon[77147]: pgmap v4972: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:01:48.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:01:47 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:01:48.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:01:47 smithi115 ceph-mon[78593]: pgmap v4972: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:01:48.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:01:47 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:01:48.983 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:01:48 smithi115 conmon[90393]: 2022-05-14T02:01:48.647+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:01:49.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:01:48 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:01:49.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:01:48 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:01:49.981 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:01:49 smithi115 conmon[90393]: 2022-05-14T02:01:49.673+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:01:50.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:01:49 smithi047 ceph-mon[77147]: pgmap v4973: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:01:50.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:01:49 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:01:50.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:01:49 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:01:50.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:01:49 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:01:50.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:01:49 smithi115 ceph-mon[78593]: pgmap v4973: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:01:50.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:01:49 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:01:50.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:01:49 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:01:50.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:01:49 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:01:50.987 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:01:50 smithi115 conmon[90393]: 2022-05-14T02:01:50.682+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:01:51.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:01:50 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 9240 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:01:51.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:01:50 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:01:51.362 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:01:50 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 9240 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:01:51.362 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:01:50 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:01:51.988 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:01:51 smithi115 conmon[90393]: 2022-05-14T02:01:51.708+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:01:52.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:01:51 smithi047 ceph-mon[77147]: pgmap v4974: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:01:52.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:01:51 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:01:52.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:01:51 smithi115 ceph-mon[78593]: pgmap v4974: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:01:52.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:01:51 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:01:52.993 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:01:52 smithi115 conmon[90393]: 2022-05-14T02:01:52.724+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:01:53.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:01:52 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:01:53.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:01:52 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:01:54.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:01:53 smithi115 conmon[90393]: 2022-05-14T02:01:53.772+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:01:54.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:01:53 smithi115 ceph-mon[78593]: pgmap v4975: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:01:54.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:01:53 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:01:54.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:01:53 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:01:54.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:01:53 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:01:54.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:01:53 smithi047 ceph-mon[77147]: pgmap v4975: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:01:54.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:01:53 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:01:54.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:01:53 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:01:54.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:01:53 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:01:55.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:01:54 smithi115 conmon[90393]: 2022-05-14T02:01:54.806+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:01:56.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:01:55 smithi115 conmon[90393]: 2022-05-14T02:01:55.759+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:01:56.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:01:55 smithi115 ceph-mon[78593]: pgmap v4976: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:01:56.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:01:55 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:01:56.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:01:55 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 9245 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:01:56.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:01:55 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:01:56.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:01:55 smithi047 ceph-mon[77147]: pgmap v4976: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:01:56.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:01:55 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:01:56.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:01:55 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 9245 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:01:56.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:01:55 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:01:57.005 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:01:56 smithi115 conmon[90393]: 2022-05-14T02:01:56.741+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:01:57.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:01:57 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:01:57.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:01:57 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:01:58.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:01:57 smithi115 conmon[90393]: 2022-05-14T02:01:57.788+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:01:58.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:01:58 smithi115 ceph-mon[78593]: pgmap v4977: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:01:58.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:01:58 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:01:58.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:01:58 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:01:58.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:01:58 smithi047 ceph-mon[77147]: pgmap v4977: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:01:58.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:01:58 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:01:58.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:01:58 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:01:59.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:01:59 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:01:59.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:01:58 smithi115 conmon[90393]: 2022-05-14T02:01:58.782+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:01:59.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:01:59 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:02:00.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:01:59 smithi115 conmon[90393]: 2022-05-14T02:01:59.769+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:02:00.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:02:00 smithi115 ceph-mon[78593]: pgmap v4978: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:02:00.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:02:00 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:02:00.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:02:00 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:02:00.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:02:00 smithi047 ceph-mon[77147]: pgmap v4978: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:02:00.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:02:00 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:02:00.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:02:00 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:02:01.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:02:00 smithi115 conmon[90393]: 2022-05-14T02:02:00.813+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:02:01.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:02:01 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 9250 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:02:01.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:02:01 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 9250 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:02:02.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:02:01 smithi115 conmon[90393]: 2022-05-14T02:02:01.791+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:02:02.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:02:02 smithi115 ceph-mon[78593]: pgmap v4979: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:02:02.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:02:02 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:02:02.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:02:02 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:02:02.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:02:02 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:02:02.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:02:02 smithi047 ceph-mon[77147]: pgmap v4979: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:02:02.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:02:02 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:02:02.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:02:02 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:02:02.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:02:02 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:02:03.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:02:02 smithi115 conmon[90393]: 2022-05-14T02:02:02.834+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:02:03.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:02:03 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:02:03.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:02:03 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:02:04.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:02:03 smithi115 conmon[90393]: 2022-05-14T02:02:03.808+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:02:04.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:02:04 smithi115 ceph-mon[78593]: pgmap v4980: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:02:04.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:02:04 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:02:04.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:02:04 smithi047 ceph-mon[77147]: pgmap v4980: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:02:04.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:02:04 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:02:05.029 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:02:04 smithi115 conmon[90393]: 2022-05-14T02:02:04.761+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:02:05.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:02:05 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:02:05.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:02:05 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:02:05.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:02:05 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:02:05.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:02:05 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:02:06.031 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:02:05 smithi115 conmon[90393]: 2022-05-14T02:02:05.737+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:02:06.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:02:06 smithi047 ceph-mon[77147]: pgmap v4981: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:02:06.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:02:06 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 9255 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:02:06.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:02:06 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:02:06.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:02:06 smithi115 ceph-mon[78593]: pgmap v4981: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:02:06.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:02:06 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 9255 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:02:06.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:02:06 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:02:07.036 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:02:06 smithi115 conmon[90393]: 2022-05-14T02:02:06.755+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:02:07.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:02:07 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:02:07.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:02:07 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:02:08.038 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:02:07 smithi115 conmon[90393]: 2022-05-14T02:02:07.739+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:02:08.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:02:08 smithi047 ceph-mon[77147]: pgmap v4982: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:02:08.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:02:08 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:02:08.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:02:08 smithi115 ceph-mon[78593]: pgmap v4982: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:02:08.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:02:08 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:02:09.046 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:02:08 smithi115 conmon[90393]: 2022-05-14T02:02:08.772+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:02:09.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:02:09 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:02:09.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:02:09 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:02:10.045 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:02:09 smithi115 conmon[90393]: 2022-05-14T02:02:09.779+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:02:10.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:02:10 smithi115 ceph-mon[78593]: pgmap v4983: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:02:10.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:02:10 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:02:10.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:02:10 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:02:10.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:02:10 smithi047 ceph-mon[77147]: pgmap v4983: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:02:10.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:02:10 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:02:10.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:02:10 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:02:11.049 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:02:10 smithi115 conmon[90393]: 2022-05-14T02:02:10.741+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:02:11.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:02:11 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:02:11.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:02:11 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 9260 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:02:11.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:02:11 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:02:11.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:02:11 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:02:11.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:02:11 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 9260 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:02:11.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:02:11 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:02:12.051 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:02:11 smithi115 conmon[90393]: 2022-05-14T02:02:11.698+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:02:12.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:02:12 smithi115 ceph-mon[78593]: pgmap v4984: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:02:12.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:02:12 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:02:12.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:02:12 smithi047 ceph-mon[77147]: pgmap v4984: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:02:12.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:02:12 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:02:13.057 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:02:12 smithi115 conmon[90393]: 2022-05-14T02:02:12.729+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:02:13.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:02:13 smithi115 ceph-mon[78593]: pgmap v4985: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:02:13.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:02:13 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:02:13.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:02:13 smithi047 ceph-mon[77147]: pgmap v4985: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:02:13.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:02:13 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:02:14.058 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:02:13 smithi115 conmon[90393]: 2022-05-14T02:02:13.743+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:02:14.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:02:14 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:02:14.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:02:14 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:02:14.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:02:14 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:02:14.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:02:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T02:02:14.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:02:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T02:02:14.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:02:14 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:02:14.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:02:14 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:02:14.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:02:14 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:02:14.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:02:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T02:02:14.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:02:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T02:02:15.065 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:02:14 smithi115 conmon[90393]: 2022-05-14T02:02:14.713+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:02:15.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:02:15 smithi115 ceph-mon[78593]: pgmap v4986: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:02:15.362 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:02:15 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:02:15.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:02:15 smithi047 ceph-mon[77147]: pgmap v4986: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:02:15.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:02:15 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:02:16.064 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:02:15 smithi115 conmon[90393]: 2022-05-14T02:02:15.679+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:02:16.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:02:16 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 9265 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:02:16.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:02:16 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:02:16.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:02:16 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 9265 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:02:16.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:02:16 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:02:17.072 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:02:16 smithi115 conmon[90393]: 2022-05-14T02:02:16.691+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:02:17.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:02:17 smithi115 ceph-mon[78593]: pgmap v4987: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:02:17.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:02:17 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:02:17.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:02:17 smithi047 ceph-mon[77147]: pgmap v4987: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:02:17.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:02:17 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:02:18.071 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:02:17 smithi115 conmon[90393]: 2022-05-14T02:02:17.729+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:02:18.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:02:18 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:02:18.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:02:18 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:02:18.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:02:18 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:02:18.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:02:18 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:02:18.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:02:18 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:02:18.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:02:18 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:02:19.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:02:18 smithi115 conmon[90393]: 2022-05-14T02:02:18.756+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:02:19.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:02:19 smithi047 ceph-mon[77147]: pgmap v4988: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:02:19.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:02:19 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:02:19.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:02:19 smithi115 ceph-mon[78593]: pgmap v4988: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:02:19.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:02:19 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:02:20.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:02:19 smithi115 conmon[90393]: 2022-05-14T02:02:19.723+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:02:20.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:02:20 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:02:20.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:02:20 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:02:21.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:02:20 smithi115 conmon[90393]: 2022-05-14T02:02:20.709+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:02:21.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:02:21 smithi047 ceph-mon[77147]: pgmap v4989: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:02:21.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:02:21 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 9275 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:02:21.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:02:21 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:02:21.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:02:21 smithi115 ceph-mon[78593]: pgmap v4989: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:02:21.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:02:21 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 9275 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:02:21.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:02:21 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:02:22.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:02:21 smithi115 conmon[90393]: 2022-05-14T02:02:21.701+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:02:22.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:02:22 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:02:22.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:02:22 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:02:22.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:02:22 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:02:22.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:02:22 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:02:22.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:02:22 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:02:22.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:02:22 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:02:23.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:02:22 smithi115 conmon[90393]: 2022-05-14T02:02:22.722+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:02:23.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:02:23 smithi115 ceph-mon[78593]: pgmap v4990: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:02:23.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:02:23 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:02:23.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:02:23 smithi047 ceph-mon[77147]: pgmap v4990: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:02:23.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:02:23 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:02:24.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:02:23 smithi115 conmon[90393]: 2022-05-14T02:02:23.739+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:02:24.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:02:24 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:02:24.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:02:24 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:02:25.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:02:24 smithi115 conmon[90393]: 2022-05-14T02:02:24.787+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:02:25.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:02:25 smithi115 ceph-mon[78593]: pgmap v4991: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:02:25.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:02:25 smithi047 ceph-mon[77147]: pgmap v4991: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:02:26.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:02:25 smithi115 conmon[90393]: 2022-05-14T02:02:25.835+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:02:26.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:02:26 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:02:26.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:02:26 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:02:27.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:02:26 smithi115 conmon[90393]: 2022-05-14T02:02:26.856+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:02:27.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:02:27 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:02:27.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:02:27 smithi115 ceph-mon[78593]: pgmap v4992: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:02:27.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:02:27 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 9280 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:02:27.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:02:27 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:02:27.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:02:27 smithi047 ceph-mon[77147]: pgmap v4992: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:02:27.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:02:27 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 9280 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:02:28.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:02:27 smithi115 conmon[90393]: 2022-05-14T02:02:27.807+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:02:28.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:02:28 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:02:28.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:02:28 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:02:29.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:02:28 smithi115 conmon[90393]: 2022-05-14T02:02:28.762+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:02:29.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:02:29 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:02:29.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:02:29 smithi115 ceph-mon[78593]: pgmap v4993: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:02:29.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:02:29 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:02:29.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:02:29 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:02:29.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:02:29 smithi047 ceph-mon[77147]: pgmap v4993: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:02:29.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:02:29 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:02:30.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:02:29 smithi115 conmon[90393]: 2022-05-14T02:02:29.737+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:02:30.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:02:30 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:02:30.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:02:30 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:02:30.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:02:30 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:02:30.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:02:30 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:02:30.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:02:30 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:02:30.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:02:30 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:02:30.977 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:02:30 smithi115 conmon[90393]: 2022-05-14T02:02:30.716+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:02:31.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:02:31 smithi115 ceph-mon[78593]: pgmap v4994: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:02:31.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:02:31 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:02:31.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:02:31 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T02:02:31.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:02:31 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T02:02:31.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:02:31 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T02:02:31.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:02:31 smithi047 ceph-mon[77147]: pgmap v4994: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:02:31.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:02:31 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:02:31.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:02:31 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T02:02:31.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:02:31 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T02:02:31.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:02:31 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T02:02:32.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:02:31 smithi115 conmon[90393]: 2022-05-14T02:02:31.684+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:02:32.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:02:32 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:02:32.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:02:32 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:02:33.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:02:32 smithi115 conmon[90393]: 2022-05-14T02:02:32.646+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:02:33.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:02:33 smithi115 ceph-mon[78593]: pgmap v4995: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:02:33.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:02:33 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:02:33.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:02:33 smithi047 ceph-mon[77147]: pgmap v4995: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:02:33.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:02:33 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:02:34.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:02:33 smithi115 conmon[90393]: 2022-05-14T02:02:33.647+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:02:34.291 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:02:34 smithi047 conmon[78521]: 2022-05-14T02:02:34.032+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:02:34.292 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:02:34 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T02:02:34.292 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:02:34 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:02:34.292 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:02:34 smithi047 conmon[78521]: yield 2022-05-14T02:02:34.293 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:02:34 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:02:34.293 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:02:34 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:02:34.293 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:02:34 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:02:34.293 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:02:34 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T02:02:34.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:02:34 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:02:34.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:02:34 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T02:02:34.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:02:34 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:02:34.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:02:34 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T02:02:34.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:02:34 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:02:34.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:02:34 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T02:02:34.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:02:34 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:02:34.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:02:34 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T02:02:34.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:02:34 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:02:34.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:02:34 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:02:34.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:02:34 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:02:34.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:02:34 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:02:34.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:02:34 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:02:34.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:02:34 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:02:34.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:02:34 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:02:34.609 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T02:02:34.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:02:34.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T02:02:34.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:02:34.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:02:34.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:02:34.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T02:02:34.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:02:34.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T02:02:34.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:02:34.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T02:02:34.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:02:34.613 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T02:02:34.613 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:02:34.613 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T02:02:34.614 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:02:34.614 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:02:34.614 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:02:34.615 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:02:34 smithi115 ceph-mon[78593]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T02:02:34.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:02:34 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:02:34.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:02:34 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:02:34.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:02:34 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:02:34.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:02:34 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:02:34.793 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T02:02:34.793 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:02:34.793 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T02:02:34.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:02:34.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:02:34.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:02:34.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T02:02:34.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:02:34.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T02:02:34.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:02:34.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T02:02:34.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:02:34.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T02:02:34.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:02:34.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T02:02:34.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:02:34.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:02:34.797 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:02:34.797 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:02:34 smithi047 ceph-mon[77147]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T02:02:35.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:02:34 smithi115 conmon[90393]: 2022-05-14T02:02:34.683+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:02:35.687 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:02:35 smithi115 ceph-mon[78593]: pgmap v4996: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:02:35.687 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:02:35 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:02:35.687 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:02:35 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 9285 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:02:35.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:02:35 smithi047 ceph-mon[77147]: pgmap v4996: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:02:35.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:02:35 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:02:35.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:02:35 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 9285 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:02:36.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:02:35 smithi115 conmon[90393]: 2022-05-14T02:02:35.685+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:02:36.724 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:02:36 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:02:36.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:02:36 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:02:37.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:02:36 smithi115 conmon[90393]: 2022-05-14T02:02:36.723+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:02:37.370 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:02:37 smithi047 conmon[78521]: 2022-05-14T02:02:37.104+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:02:37.371 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:02:37 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T02:02:37.371 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:02:37 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:02:37.371 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:02:37 smithi047 conmon[78521]: yield 2022-05-14T02:02:37.372 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:02:37 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:02:37.372 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:02:37 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:02:37.372 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:02:37 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:02:37.372 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:02:37 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T02:02:37.373 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:02:37 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:02:37.373 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:02:37 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T02:02:37.373 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:02:37 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:02:37.374 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:02:37 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T02:02:37.374 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:02:37 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:02:37.374 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:02:37 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T02:02:37.374 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:02:37 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:02:37.374 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:02:37 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T02:02:37.375 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:02:37 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:02:37.375 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:02:37 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:02:37.375 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:02:37 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:02:37.376 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:02:37 smithi047 conmon[78521]: 2022-05-14T02:02:37.105+0000 7f5340779700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T02:02:37.376 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:02:37 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T02:02:37.376 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:02:37 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:02:37.376 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:02:37 smithi047 conmon[78521]: yield 2022-05-14T02:02:37.377 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:02:37 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:02:37.377 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:02:37 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:02:37.377 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:02:37 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:02:37.377 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:02:37 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T02:02:37.377 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:02:37 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:02:37.378 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:02:37 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T02:02:37.378 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:02:37 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:02:37.378 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:02:37 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T02:02:37.378 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:02:37 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:02:37.379 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:02:37 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T02:02:37.379 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:02:37 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:02:37.379 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:02:37 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T02:02:37.379 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:02:37 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:02:37.379 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:02:37 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:02:37.380 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:02:37 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:02:37.380 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:02:37 smithi047 conmon[78521]: 2022-05-14T02:02:37.380 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:02:37 smithi047 conmon[78521]: During handling of the above exception, another exception occurred: 2022-05-14T02:02:37.380 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:02:37 smithi047 conmon[78521]: 2022-05-14T02:02:37.380 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:02:37 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T02:02:37.381 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:02:37 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T02:02:37.381 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:02:37 smithi047 conmon[78521]: return f(*arg) 2022-05-14T02:02:37.381 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:02:37 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T02:02:37.381 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:02:37 smithi047 conmon[78521]: self._write_client_files(client_files, host) 2022-05-14T02:02:37.382 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:02:37 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T02:02:37.382 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:02:37 smithi047 conmon[78521]: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T02:02:37.382 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:02:37 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T02:02:37.382 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:02:37 smithi047 conmon[78521]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T02:02:37.382 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:02:37 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T02:02:37.383 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:02:37 smithi047 conmon[78521]: return self.event_loop.get_result(coro) 2022-05-14T02:02:37.383 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:02:37 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T02:02:37.383 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:02:37 smithi047 conmon[78521]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T02:02:37.383 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:02:37 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T02:02:37.383 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:02:37 smithi047 conmon[78521]: return self.__get_result() 2022-05-14T02:02:37.384 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:02:37 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T02:02:37.384 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:02:37 smithi047 conmon[78521]: raise self._exception 2022-05-14T02:02:37.384 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:02:37 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T02:02:37.384 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:02:37 smithi047 conmon[78521]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T02:02:37.385 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:02:37 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T02:02:37.385 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:02:37 smithi047 conmon[78521]: conn = await self._remote_connection(host, addr) 2022-05-14T02:02:37.385 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:02:37 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T02:02:37.385 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:02:37 smithi047 conmon[78521]: raise 2022-05-14T02:02:37.386 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:02:37 smithi047 conmon[78521]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T02:02:37.386 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:02:37 smithi047 conmon[78521]: self.gen.throw(type, value, traceback) 2022-05-14T02:02:37.386 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:02:37 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T02:02:37.386 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:02:37 smithi047 conmon[78521]: raise OrchestratorError(msg) 2022-05-14T02:02:37.387 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:02:37 smithi047 conmon[78521]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:02:37.754 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:02:37 smithi115 ceph-mon[78593]: pgmap v4997: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:02:37.754 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:02:37 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:02:37.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:02:37 smithi047 ceph-mon[77147]: pgmap v4997: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:02:37.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:02:37 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:02:38.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:02:37 smithi115 conmon[90393]: 2022-05-14T02:02:37.752+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:02:38.790 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:02:38 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:02:38.791 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T02:02:38.791 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:02:38.791 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T02:02:38.792 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:02:38.792 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:02:38.792 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:02:38.792 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T02:02:38.792 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:02:38.793 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T02:02:38.793 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:02:38.793 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T02:02:38.794 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:02:38.794 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T02:02:38.794 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:02:38.794 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T02:02:38.795 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:02:38.795 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:02:38.795 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:02:38.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:02:38 smithi115 ceph-mon[78593]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T02:02:38.796 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T02:02:38.796 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:02:38.796 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T02:02:38.797 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:02:38.797 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:02:38.797 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:02:38.797 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T02:02:38.798 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:02:38.798 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T02:02:38.798 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:02:38.798 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T02:02:38.798 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:02:38.799 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T02:02:38.799 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:02:38.799 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T02:02:38.800 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:02:38.800 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:02:38.800 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:02:38.800 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T02:02:38.800 INFO:journalctl@ceph.mon.c.smithi115.stdout: During handling of the above exception, another exception occurred: 2022-05-14T02:02:38.801 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T02:02:38.801 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T02:02:38.801 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T02:02:38.801 INFO:journalctl@ceph.mon.c.smithi115.stdout: return f(*arg) 2022-05-14T02:02:38.802 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T02:02:38.802 INFO:journalctl@ceph.mon.c.smithi115.stdout: self._write_client_files(client_files, host) 2022-05-14T02:02:38.802 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T02:02:38.802 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T02:02:38.803 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T02:02:38.803 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T02:02:38.803 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T02:02:38.803 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.event_loop.get_result(coro) 2022-05-14T02:02:38.803 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T02:02:38.804 INFO:journalctl@ceph.mon.c.smithi115.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T02:02:38.804 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T02:02:38.804 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.__get_result() 2022-05-14T02:02:38.804 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T02:02:38.804 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise self._exception 2022-05-14T02:02:38.805 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T02:02:38.805 INFO:journalctl@ceph.mon.c.smithi115.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T02:02:38.805 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T02:02:38.805 INFO:journalctl@ceph.mon.c.smithi115.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T02:02:38.806 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T02:02:38.806 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise 2022-05-14T02:02:38.806 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T02:02:38.806 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.gen.throw(type, value, traceback) 2022-05-14T02:02:38.807 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T02:02:38.807 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OrchestratorError(msg) 2022-05-14T02:02:38.807 INFO:journalctl@ceph.mon.c.smithi115.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:02:38.807 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:02:38 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:02:38.807 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:02:38 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:02:38.808 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:02:38 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:02:38.808 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:02:38 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:02:38.809 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T02:02:38.809 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:02:38.810 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T02:02:38.810 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:02:38.810 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:02:38.810 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:02:38.810 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T02:02:38.811 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:02:38.811 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T02:02:38.811 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:02:38.811 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T02:02:38.812 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:02:38.812 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T02:02:38.812 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:02:38.812 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T02:02:38.812 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:02:38.813 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:02:38.813 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:02:38.813 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:02:38 smithi047 ceph-mon[77147]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T02:02:38.813 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T02:02:38.814 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:02:38.814 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T02:02:38.814 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:02:38.814 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:02:38.815 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:02:38.815 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T02:02:38.815 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:02:38.815 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T02:02:38.815 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:02:38.816 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T02:02:38.816 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:02:38.816 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T02:02:38.817 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:02:38.817 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T02:02:38.817 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:02:38.818 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:02:38.818 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:02:38.819 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T02:02:38.819 INFO:journalctl@ceph.mon.b.smithi047.stdout: During handling of the above exception, another exception occurred: 2022-05-14T02:02:38.819 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T02:02:38.820 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T02:02:38.820 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T02:02:38.821 INFO:journalctl@ceph.mon.b.smithi047.stdout: return f(*arg) 2022-05-14T02:02:38.821 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T02:02:38.822 INFO:journalctl@ceph.mon.b.smithi047.stdout: self._write_client_files(client_files, host) 2022-05-14T02:02:38.822 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T02:02:38.822 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T02:02:38.823 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T02:02:38.823 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T02:02:38.824 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T02:02:38.824 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.event_loop.get_result(coro) 2022-05-14T02:02:38.825 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T02:02:38.825 INFO:journalctl@ceph.mon.b.smithi047.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T02:02:38.825 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T02:02:38.826 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.__get_result() 2022-05-14T02:02:38.826 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T02:02:38.826 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise self._exception 2022-05-14T02:02:38.827 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T02:02:38.827 INFO:journalctl@ceph.mon.b.smithi047.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T02:02:38.828 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T02:02:38.828 INFO:journalctl@ceph.mon.b.smithi047.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T02:02:38.828 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T02:02:38.829 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise 2022-05-14T02:02:38.829 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T02:02:38.829 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.gen.throw(type, value, traceback) 2022-05-14T02:02:38.829 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T02:02:38.830 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OrchestratorError(msg) 2022-05-14T02:02:38.830 INFO:journalctl@ceph.mon.b.smithi047.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:02:38.830 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:02:38 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:02:38.830 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:02:38 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:02:38.830 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:02:38 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:02:39.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:02:38 smithi115 conmon[90393]: 2022-05-14T02:02:38.788+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:02:39.790 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:02:39 smithi115 ceph-mon[78593]: pgmap v4998: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:02:39.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:02:39 smithi047 ceph-mon[77147]: pgmap v4998: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:02:40.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:02:39 smithi115 conmon[90393]: 2022-05-14T02:02:39.788+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:02:40.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:02:40 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:02:40.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:02:40 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 9290 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:02:40.825 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:02:40 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:02:40.826 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:02:40 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 9290 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:02:41.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:02:40 smithi115 conmon[90393]: 2022-05-14T02:02:40.823+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:02:41.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:02:41 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:02:41.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:02:41 smithi047 ceph-mon[77147]: pgmap v4999: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:02:41.806 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:02:41 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:02:41.807 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:02:41 smithi115 ceph-mon[78593]: pgmap v4999: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:02:42.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:02:41 smithi115 conmon[90393]: 2022-05-14T02:02:41.805+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:02:42.771 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:02:42 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:02:42.772 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:02:42 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:02:42.772 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:02:42 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:02:42.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:02:42 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:02:42.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:02:42 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:02:42.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:02:42 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:02:43.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:02:42 smithi115 conmon[90393]: 2022-05-14T02:02:42.769+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:02:43.783 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:02:43 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:02:43.783 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:02:43 smithi115 ceph-mon[78593]: pgmap v5000: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:02:43.784 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:02:43 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:02:43.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:02:43 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:02:43.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:02:43 smithi047 ceph-mon[77147]: pgmap v5000: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:02:43.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:02:43 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:02:44.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:02:43 smithi115 conmon[90393]: 2022-05-14T02:02:43.780+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:02:44.790 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:02:44 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:02:44.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:02:44 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:02:45.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:02:44 smithi115 conmon[90393]: 2022-05-14T02:02:44.788+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:02:45.742 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:02:45 smithi115 ceph-mon[78593]: pgmap v5001: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:02:45.743 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:02:45 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 9295 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:02:45.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:02:45 smithi047 ceph-mon[77147]: pgmap v5001: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:02:45.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:02:45 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 9295 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:02:46.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:02:45 smithi115 conmon[90393]: 2022-05-14T02:02:45.741+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:02:46.787 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:02:46 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:02:46.787 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:02:46 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:02:46.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:02:46 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:02:46.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:02:46 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:02:47.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:02:46 smithi115 conmon[90393]: 2022-05-14T02:02:46.785+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:02:47.761 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:02:47 smithi115 ceph-mon[78593]: pgmap v5002: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:02:47.761 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:02:47 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:02:47.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:02:47 smithi047 ceph-mon[77147]: pgmap v5002: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:02:47.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:02:47 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:02:48.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:02:47 smithi115 conmon[90393]: 2022-05-14T02:02:47.759+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:02:48.771 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:02:48 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:02:48.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:02:48 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:02:49.110 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:02:48 smithi115 conmon[90393]: 2022-05-14T02:02:48.770+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:02:49.734 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:02:49 smithi115 ceph-mon[78593]: pgmap v5003: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:02:49.734 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:02:49 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:02:49.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:02:49 smithi047 ceph-mon[77147]: pgmap v5003: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:02:49.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:02:49 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:02:50.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:02:49 smithi115 conmon[90393]: 2022-05-14T02:02:49.732+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:02:50.766 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:02:50 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:02:50.766 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:02:50 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:02:50.767 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:02:50 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:02:50.767 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:02:50 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 9300 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:02:50.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:02:50 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:02:50.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:02:50 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:02:50.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:02:50 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:02:50.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:02:50 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 9300 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:02:51.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:02:50 smithi115 conmon[90393]: 2022-05-14T02:02:50.763+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:02:51.741 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:02:51 smithi115 ceph-mon[78593]: pgmap v5004: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:02:51.742 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:02:51 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:02:51.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:02:51 smithi047 ceph-mon[77147]: pgmap v5004: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:02:51.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:02:51 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:02:52.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:02:51 smithi115 conmon[90393]: 2022-05-14T02:02:51.739+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:02:52.702 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:02:52 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:02:52.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:02:52 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:02:53.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:02:52 smithi115 conmon[90393]: 2022-05-14T02:02:52.700+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:02:53.696 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:02:53 smithi115 ceph-mon[78593]: pgmap v5005: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:02:53.696 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:02:53 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:02:53.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:02:53 smithi047 ceph-mon[77147]: pgmap v5005: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:02:53.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:02:53 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:02:54.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:02:53 smithi115 conmon[90393]: 2022-05-14T02:02:53.694+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:02:54.740 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:02:54 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:02:54.741 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:02:54 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:02:54.741 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:02:54 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:02:54.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:02:54 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:02:54.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:02:54 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:02:54.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:02:54 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:02:55.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:02:54 smithi115 conmon[90393]: 2022-05-14T02:02:54.738+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:02:55.705 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:02:55 smithi115 ceph-mon[78593]: pgmap v5006: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:02:55.705 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:02:55 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:02:55.706 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:02:55 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 9305 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:02:55.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:02:55 smithi047 ceph-mon[77147]: pgmap v5006: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:02:55.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:02:55 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:02:55.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:02:55 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 9305 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:02:56.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:02:55 smithi115 conmon[90393]: 2022-05-14T02:02:55.703+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:02:56.752 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:02:56 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:02:56.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:02:56 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:02:57.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:02:56 smithi115 conmon[90393]: 2022-05-14T02:02:56.750+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:02:57.726 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:02:57 smithi115 ceph-mon[78593]: pgmap v5007: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:02:57.727 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:02:57 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:02:57.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:02:57 smithi047 ceph-mon[77147]: pgmap v5007: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:02:57.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:02:57 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:02:58.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:02:57 smithi115 conmon[90393]: 2022-05-14T02:02:57.724+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:02:58.767 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:02:58 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:02:58.768 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:02:58 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:02:58.768 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:02:58 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:02:58.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:02:58 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:02:58.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:02:58 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:02:58.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:02:58 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:02:59.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:02:58 smithi115 conmon[90393]: 2022-05-14T02:02:58.765+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:02:59.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:02:59 smithi047 ceph-mon[77147]: pgmap v5008: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:02:59.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:02:59 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:02:59.808 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:02:59 smithi115 ceph-mon[78593]: pgmap v5008: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:02:59.808 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:02:59 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:03:00.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:02:59 smithi115 conmon[90393]: 2022-05-14T02:02:59.806+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:03:00.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:03:00 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 9310 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:03:00.836 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:03:00 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 9310 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:03:01.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:03:00 smithi115 conmon[90393]: 2022-05-14T02:03:00.834+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:03:01.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:03:01 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:03:01.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:03:01 smithi047 ceph-mon[77147]: pgmap v5009: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:03:01.847 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:03:01 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:03:01.848 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:03:01 smithi115 ceph-mon[78593]: pgmap v5009: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:03:02.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:03:01 smithi115 conmon[90393]: 2022-05-14T02:03:01.846+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:03:02.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:03:02 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:03:02.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:03:02 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:03:02.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:03:02 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:03:02.849 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:03:02 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:03:02.850 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:03:02 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:03:02.850 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:03:02 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:03:03.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:03:02 smithi115 conmon[90393]: 2022-05-14T02:03:02.848+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:03:03.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:03:03 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:03:03.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:03:03 smithi047 ceph-mon[77147]: pgmap v5010: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:03:03.812 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:03:03 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:03:03.812 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:03:03 smithi115 ceph-mon[78593]: pgmap v5010: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:03:04.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:03:03 smithi115 conmon[90393]: 2022-05-14T02:03:03.810+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:03:04.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:03:04 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:03:04.832 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:03:04 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:03:05.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:03:04 smithi115 conmon[90393]: 2022-05-14T02:03:04.831+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:03:05.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:03:05 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:03:05.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:03:05 smithi047 ceph-mon[77147]: pgmap v5011: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:03:05.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:03:05 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 9315 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:03:05.804 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:03:05 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:03:05.804 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:03:05 smithi115 ceph-mon[78593]: pgmap v5011: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:03:05.804 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:03:05 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 9315 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:03:06.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:03:05 smithi115 conmon[90393]: 2022-05-14T02:03:05.802+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:03:06.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:03:06 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:03:06.809 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:03:06 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:03:07.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:03:06 smithi115 conmon[90393]: 2022-05-14T02:03:06.808+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:03:07.787 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:03:07 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:03:07.787 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:03:07 smithi115 ceph-mon[78593]: pgmap v5012: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:03:07.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:03:07 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:03:07.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:03:07 smithi047 ceph-mon[77147]: pgmap v5012: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:03:08.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:03:07 smithi115 conmon[90393]: 2022-05-14T02:03:07.785+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:03:08.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:03:08 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:03:08.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:03:08 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:03:08.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:03:08 smithi115 conmon[90393]: 2022-05-14T02:03:08.738+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:03:08.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:03:08 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:03:08.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:03:08 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:03:09.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:03:09 smithi047 ceph-mon[77147]: pgmap v5013: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:03:09.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:03:09 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:03:09.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:03:09 smithi115 conmon[90393]: 2022-05-14T02:03:09.708+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:03:09.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:03:09 smithi115 ceph-mon[78593]: pgmap v5013: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:03:09.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:03:09 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:03:10.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:03:10 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:03:10.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:03:10 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:03:10.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:03:10 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:03:10.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:03:10 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 9320 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:03:10.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:03:10 smithi115 conmon[90393]: 2022-05-14T02:03:10.670+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:03:10.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:03:10 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:03:10.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:03:10 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:03:10.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:03:10 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:03:10.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:03:10 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 9320 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:03:11.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:03:11 smithi047 ceph-mon[77147]: pgmap v5014: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:03:11.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:03:11 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:03:11.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:03:11 smithi115 conmon[90393]: 2022-05-14T02:03:11.694+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:03:11.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:03:11 smithi115 ceph-mon[78593]: pgmap v5014: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:03:11.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:03:11 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:03:12.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:03:12 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:03:12.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:03:12 smithi115 conmon[90393]: 2022-05-14T02:03:12.709+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:03:12.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:03:12 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:03:13.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:03:13 smithi047 ceph-mon[77147]: pgmap v5015: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:03:13.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:03:13 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:03:13.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:03:13 smithi115 conmon[90393]: 2022-05-14T02:03:13.709+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:03:13.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:03:13 smithi115 ceph-mon[78593]: pgmap v5015: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:03:13.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:03:13 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:03:14.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:03:14 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:03:14.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:03:14 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:03:14.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:03:14 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:03:14.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:03:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T02:03:14.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:03:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T02:03:14.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:03:14 smithi115 conmon[90393]: 2022-05-14T02:03:14.742+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:03:14.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:03:14 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:03:14.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:03:14 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:03:14.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:03:14 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:03:14.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:03:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T02:03:14.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:03:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T02:03:15.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:03:15 smithi047 ceph-mon[77147]: pgmap v5016: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:03:15.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:03:15 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:03:15.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:03:15 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 9325 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:03:15.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:03:15 smithi115 ceph-mon[78593]: pgmap v5016: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:03:15.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:03:15 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:03:15.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:03:15 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 9325 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:03:15.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:03:15 smithi115 conmon[90393]: 2022-05-14T02:03:15.708+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:03:16.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:03:16 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:03:16.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:03:16 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:03:16.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:03:16 smithi115 conmon[90393]: 2022-05-14T02:03:16.685+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:03:17.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:03:17 smithi047 ceph-mon[77147]: pgmap v5017: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:03:17.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:03:17 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:03:17.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:03:17 smithi115 conmon[90393]: 2022-05-14T02:03:17.657+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:03:17.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:03:17 smithi115 ceph-mon[78593]: pgmap v5017: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:03:17.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:03:17 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:03:18.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:03:18 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:03:18.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:03:18 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:03:18.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:03:18 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:03:18.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:03:18 smithi115 conmon[90393]: 2022-05-14T02:03:18.611+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:03:18.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:03:18 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:03:18.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:03:18 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:03:18.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:03:18 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:03:19.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:03:19 smithi047 ceph-mon[77147]: pgmap v5018: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:03:19.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:03:19 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:03:19.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:03:19 smithi115 conmon[90393]: 2022-05-14T02:03:19.618+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:03:19.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:03:19 smithi115 ceph-mon[78593]: pgmap v5018: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:03:19.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:03:19 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:03:20.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:03:20 smithi115 conmon[90393]: 2022-05-14T02:03:20.620+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:03:20.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:03:20 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:03:20.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:03:20 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 9330 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:03:21.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:03:20 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:03:21.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:03:20 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 9330 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:03:21.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:03:21 smithi115 conmon[90393]: 2022-05-14T02:03:21.653+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:03:21.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:03:21 smithi115 ceph-mon[78593]: pgmap v5019: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:03:21.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:03:21 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:03:22.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:03:21 smithi047 ceph-mon[77147]: pgmap v5019: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:03:22.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:03:21 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:03:22.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:03:22 smithi115 conmon[90393]: 2022-05-14T02:03:22.635+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:03:22.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:03:22 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:03:22.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:03:22 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:03:22.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:03:22 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:03:23.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:03:22 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:03:23.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:03:22 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:03:23.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:03:22 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:03:23.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:03:23 smithi115 conmon[90393]: 2022-05-14T02:03:23.684+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:03:23.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:03:23 smithi115 ceph-mon[78593]: pgmap v5020: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:03:23.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:03:23 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:03:24.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:03:23 smithi047 ceph-mon[77147]: pgmap v5020: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:03:24.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:03:23 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:03:24.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:03:24 smithi115 conmon[90393]: 2022-05-14T02:03:24.663+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:03:24.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:03:24 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:03:25.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:03:24 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:03:25.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:03:25 smithi115 conmon[90393]: 2022-05-14T02:03:25.661+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:03:25.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:03:25 smithi115 ceph-mon[78593]: pgmap v5021: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:03:25.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:03:25 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:03:25.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:03:25 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 9335 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:03:26.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:03:25 smithi047 ceph-mon[77147]: pgmap v5021: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:03:26.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:03:25 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:03:26.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:03:25 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 9335 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:03:26.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:03:26 smithi115 conmon[90393]: 2022-05-14T02:03:26.686+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:03:26.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:03:26 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:03:27.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:03:26 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:03:27.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:03:27 smithi115 conmon[90393]: 2022-05-14T02:03:27.664+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:03:27.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:03:27 smithi115 ceph-mon[78593]: pgmap v5022: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-05-14T02:03:27.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:03:27 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:03:28.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:03:27 smithi047 ceph-mon[77147]: pgmap v5022: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-05-14T02:03:28.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:03:27 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:03:28.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:03:28 smithi115 conmon[90393]: 2022-05-14T02:03:28.653+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:03:28.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:03:28 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:03:29.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:03:28 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:03:29.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:03:29 smithi115 conmon[90393]: 2022-05-14T02:03:29.683+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:03:29.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:03:29 smithi115 ceph-mon[78593]: pgmap v5023: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:03:29.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:03:29 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:03:30.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:03:29 smithi047 ceph-mon[77147]: pgmap v5023: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:03:30.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:03:29 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:03:30.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:03:30 smithi115 conmon[90393]: 2022-05-14T02:03:30.676+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:03:30.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:03:30 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:03:30.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:03:30 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:03:30.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:03:30 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:03:30.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:03:30 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 9340 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:03:31.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:03:30 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:03:31.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:03:30 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:03:31.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:03:30 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:03:31.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:03:30 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 9340 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:03:31.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:03:31 smithi115 conmon[90393]: 2022-05-14T02:03:31.711+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:03:31.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:03:31 smithi115 ceph-mon[78593]: pgmap v5024: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:03:31.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:03:31 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:03:32.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:03:31 smithi047 ceph-mon[77147]: pgmap v5024: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:03:32.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:03:31 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:03:32.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:03:32 smithi115 conmon[90393]: 2022-05-14T02:03:32.719+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:03:32.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:03:32 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:03:33.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:03:32 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:03:33.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:03:33 smithi115 conmon[90393]: 2022-05-14T02:03:33.678+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:03:33.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:03:33 smithi115 ceph-mon[78593]: pgmap v5025: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:03:33.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:03:33 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:03:34.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:03:33 smithi047 ceph-mon[77147]: pgmap v5025: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:03:34.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:03:33 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:03:34.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:03:34 smithi115 conmon[90393]: 2022-05-14T02:03:34.682+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:03:34.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:03:34 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:03:34.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:03:34 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:03:34.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:03:34 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:03:35.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:03:34 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:03:35.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:03:34 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:03:35.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:03:34 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:03:36.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:03:35 smithi047 ceph-mon[77147]: pgmap v5026: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:03:36.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:03:35 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:03:36.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:03:35 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 9345 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:03:36.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:03:35 smithi115 conmon[90393]: 2022-05-14T02:03:35.717+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:03:36.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:03:35 smithi115 ceph-mon[78593]: pgmap v5026: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:03:36.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:03:35 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:03:36.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:03:35 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 9345 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:03:37.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:03:36 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:03:37.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:03:36 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:03:37.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:03:36 smithi115 conmon[90393]: 2022-05-14T02:03:36.733+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:03:38.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:03:37 smithi047 ceph-mon[77147]: pgmap v5027: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-05-14T02:03:38.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:03:37 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:03:38.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:03:37 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T02:03:38.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:03:37 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T02:03:38.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:03:37 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T02:03:38.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:03:37 smithi047 conmon[78521]: 2022-05-14T02:03:37.583+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:03:38.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:03:37 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T02:03:38.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:03:37 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:03:38.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:03:37 smithi047 conmon[78521]: yield 2022-05-14T02:03:38.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:03:37 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:03:38.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:03:37 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:03:38.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:03:37 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:03:38.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:03:37 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T02:03:38.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:03:37 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:03:38.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:03:37 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T02:03:38.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:03:37 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:03:38.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:03:37 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T02:03:38.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:03:37 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:03:38.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:03:37 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T02:03:38.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:03:37 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:03:38.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:03:37 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T02:03:38.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:03:37 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:03:38.048 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:03:37 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:03:38.048 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:03:37 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:03:38.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:03:37 smithi115 conmon[90393]: 2022-05-14T02:03:37.777+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:03:38.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:03:37 smithi115 ceph-mon[78593]: pgmap v5027: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-05-14T02:03:38.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:03:37 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:03:38.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:03:37 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T02:03:38.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:03:37 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T02:03:38.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:03:37 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T02:03:39.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:03:38 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:03:39.042 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T02:03:39.042 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:03:39.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T02:03:39.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:03:39.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:03:39.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:03:39.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T02:03:39.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:03:39.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T02:03:39.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:03:39.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T02:03:39.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:03:39.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T02:03:39.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:03:39.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T02:03:39.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:03:39.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:03:39.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:03:39.048 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:03:38 smithi047 ceph-mon[77147]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T02:03:39.049 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:03:38 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:03:39.049 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:03:38 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:03:39.049 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:03:38 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:03:39.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:03:38 smithi115 conmon[90393]: 2022-05-14T02:03:38.795+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:03:39.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:03:38 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:03:39.109 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T02:03:39.109 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:03:39.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T02:03:39.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:03:39.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:03:39.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:03:39.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T02:03:39.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:03:39.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T02:03:39.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:03:39.113 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T02:03:39.113 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:03:39.113 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T02:03:39.114 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:03:39.114 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T02:03:39.115 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:03:39.115 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:03:39.115 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:03:39.116 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:03:38 smithi115 ceph-mon[78593]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T02:03:39.116 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:03:38 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:03:39.116 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:03:38 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:03:39.116 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:03:38 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:03:40.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:03:39 smithi047 ceph-mon[77147]: pgmap v5028: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:03:40.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:03:39 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:03:40.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:03:39 smithi115 conmon[90393]: 2022-05-14T02:03:39.757+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:03:40.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:03:39 smithi115 ceph-mon[78593]: pgmap v5028: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:03:40.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:03:39 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:03:41.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:03:40 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:03:41.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:03:40 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 9350 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:03:41.042 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:03:40 smithi047 conmon[78521]: 2022-05-14T02:03:40.655+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:03:41.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:03:40 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T02:03:41.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:03:40 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:03:41.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:03:40 smithi047 conmon[78521]: yield 2022-05-14T02:03:41.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:03:40 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:03:41.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:03:40 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:03:41.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:03:40 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:03:41.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:03:40 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T02:03:41.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:03:40 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:03:41.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:03:40 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T02:03:41.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:03:40 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:03:41.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:03:40 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T02:03:41.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:03:40 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:03:41.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:03:40 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T02:03:41.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:03:40 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:03:41.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:03:40 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T02:03:41.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:03:40 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:03:41.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:03:40 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:03:41.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:03:40 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:03:41.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:03:40 smithi047 conmon[78521]: 2022-05-14T02:03:40.656+0000 7f5344f82700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T02:03:41.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:03:40 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T02:03:41.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:03:40 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:03:41.048 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:03:40 smithi047 conmon[78521]: yield 2022-05-14T02:03:41.048 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:03:40 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:03:41.048 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:03:40 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:03:41.048 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:03:40 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:03:41.049 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:03:40 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T02:03:41.049 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:03:40 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:03:41.049 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:03:40 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T02:03:41.049 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:03:40 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:03:41.050 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:03:40 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T02:03:41.050 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:03:40 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:03:41.050 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:03:40 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T02:03:41.050 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:03:40 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:03:41.051 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:03:40 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T02:03:41.051 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:03:40 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:03:41.051 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:03:40 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:03:41.051 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:03:40 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:03:41.051 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:03:40 smithi047 conmon[78521]: 2022-05-14T02:03:41.052 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:03:40 smithi047 conmon[78521]: During handling of the above exception, another exception occurred: 2022-05-14T02:03:41.052 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:03:40 smithi047 conmon[78521]: 2022-05-14T02:03:41.052 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:03:40 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T02:03:41.052 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:03:40 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T02:03:41.053 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:03:40 smithi047 conmon[78521]: return f(*arg) 2022-05-14T02:03:41.053 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:03:40 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T02:03:41.053 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:03:40 smithi047 conmon[78521]: self._write_client_files(client_files, host) 2022-05-14T02:03:41.053 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:03:40 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T02:03:41.053 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:03:40 smithi047 conmon[78521]: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T02:03:41.054 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:03:40 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T02:03:41.054 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:03:40 smithi047 conmon[78521]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T02:03:41.054 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:03:40 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T02:03:41.054 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:03:40 smithi047 conmon[78521]: return self.event_loop.get_result(coro) 2022-05-14T02:03:41.055 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:03:40 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T02:03:41.055 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:03:40 smithi047 conmon[78521]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T02:03:41.055 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:03:40 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T02:03:41.055 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:03:40 smithi047 conmon[78521]: return self.__get_result() 2022-05-14T02:03:41.055 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:03:40 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T02:03:41.056 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:03:40 smithi047 conmon[78521]: raise self._exception 2022-05-14T02:03:41.056 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:03:40 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T02:03:41.056 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:03:40 smithi047 conmon[78521]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T02:03:41.056 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:03:40 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T02:03:41.057 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:03:40 smithi047 conmon[78521]: conn = await self._remote_connection(host, addr) 2022-05-14T02:03:41.057 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:03:40 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T02:03:41.057 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:03:40 smithi047 conmon[78521]: raise 2022-05-14T02:03:41.057 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:03:40 smithi047 conmon[78521]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T02:03:41.058 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:03:40 smithi047 conmon[78521]: self.gen.throw(type, value, traceback) 2022-05-14T02:03:41.058 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:03:40 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T02:03:41.058 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:03:40 smithi047 conmon[78521]: raise OrchestratorError(msg) 2022-05-14T02:03:41.058 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:03:40 smithi047 conmon[78521]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:03:41.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:03:40 smithi115 conmon[90393]: 2022-05-14T02:03:40.791+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:03:41.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:03:40 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:03:41.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:03:40 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 9350 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:03:42.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:03:41 smithi047 ceph-mon[77147]: pgmap v5029: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:03:42.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:03:41 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:03:42.042 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T02:03:42.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:03:42.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T02:03:42.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:03:42.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:03:42.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:03:42.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T02:03:42.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:03:42.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T02:03:42.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:03:42.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T02:03:42.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:03:42.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T02:03:42.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:03:42.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T02:03:42.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:03:42.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:03:42.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:03:42.047 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:03:41 smithi047 ceph-mon[77147]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T02:03:42.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T02:03:42.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:03:42.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T02:03:42.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:03:42.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:03:42.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:03:42.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T02:03:42.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:03:42.049 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T02:03:42.049 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:03:42.049 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T02:03:42.049 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:03:42.050 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T02:03:42.050 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:03:42.050 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T02:03:42.050 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:03:42.050 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:03:42.051 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:03:42.051 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T02:03:42.051 INFO:journalctl@ceph.mon.b.smithi047.stdout: During handling of the above exception, another exception occurred: 2022-05-14T02:03:42.051 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T02:03:42.051 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T02:03:42.052 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T02:03:42.052 INFO:journalctl@ceph.mon.b.smithi047.stdout: return f(*arg) 2022-05-14T02:03:42.052 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T02:03:42.053 INFO:journalctl@ceph.mon.b.smithi047.stdout: self._write_client_files(client_files, host) 2022-05-14T02:03:42.053 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T02:03:42.053 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T02:03:42.053 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T02:03:42.053 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T02:03:42.054 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T02:03:42.054 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.event_loop.get_result(coro) 2022-05-14T02:03:42.054 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T02:03:42.054 INFO:journalctl@ceph.mon.b.smithi047.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T02:03:42.054 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T02:03:42.055 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.__get_result() 2022-05-14T02:03:42.055 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T02:03:42.055 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise self._exception 2022-05-14T02:03:42.055 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T02:03:42.055 INFO:journalctl@ceph.mon.b.smithi047.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T02:03:42.056 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T02:03:42.056 INFO:journalctl@ceph.mon.b.smithi047.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T02:03:42.056 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T02:03:42.056 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise 2022-05-14T02:03:42.057 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T02:03:42.057 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.gen.throw(type, value, traceback) 2022-05-14T02:03:42.057 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T02:03:42.057 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OrchestratorError(msg) 2022-05-14T02:03:42.057 INFO:journalctl@ceph.mon.b.smithi047.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:03:42.058 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:03:41 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:03:42.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:03:41 smithi115 conmon[90393]: 2022-05-14T02:03:41.790+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:03:42.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:03:41 smithi115 ceph-mon[78593]: pgmap v5029: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:03:42.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:03:41 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:03:42.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T02:03:42.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:03:42.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T02:03:42.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:03:42.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:03:42.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:03:42.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T02:03:42.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:03:42.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T02:03:42.113 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:03:42.113 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T02:03:42.114 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:03:42.114 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T02:03:42.114 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:03:42.115 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T02:03:42.115 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:03:42.116 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:03:42.116 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:03:42.116 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:03:41 smithi115 ceph-mon[78593]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T02:03:42.117 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T02:03:42.118 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:03:42.118 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T02:03:42.119 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:03:42.119 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:03:42.120 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:03:42.120 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T02:03:42.120 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:03:42.121 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T02:03:42.121 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:03:42.121 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T02:03:42.121 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:03:42.122 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T02:03:42.122 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:03:42.122 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T02:03:42.122 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:03:42.123 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:03:42.123 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:03:42.123 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T02:03:42.123 INFO:journalctl@ceph.mon.c.smithi115.stdout: During handling of the above exception, another exception occurred: 2022-05-14T02:03:42.124 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T02:03:42.124 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T02:03:42.124 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T02:03:42.124 INFO:journalctl@ceph.mon.c.smithi115.stdout: return f(*arg) 2022-05-14T02:03:42.124 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T02:03:42.125 INFO:journalctl@ceph.mon.c.smithi115.stdout: self._write_client_files(client_files, host) 2022-05-14T02:03:42.125 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T02:03:42.125 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T02:03:42.126 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T02:03:42.126 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T02:03:42.126 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T02:03:42.127 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.event_loop.get_result(coro) 2022-05-14T02:03:42.127 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T02:03:42.128 INFO:journalctl@ceph.mon.c.smithi115.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T02:03:42.128 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T02:03:42.128 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.__get_result() 2022-05-14T02:03:42.129 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T02:03:42.129 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise self._exception 2022-05-14T02:03:42.130 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T02:03:42.130 INFO:journalctl@ceph.mon.c.smithi115.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T02:03:42.130 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T02:03:42.130 INFO:journalctl@ceph.mon.c.smithi115.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T02:03:42.131 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T02:03:42.131 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise 2022-05-14T02:03:42.131 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T02:03:42.132 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.gen.throw(type, value, traceback) 2022-05-14T02:03:42.132 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T02:03:42.132 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OrchestratorError(msg) 2022-05-14T02:03:42.133 INFO:journalctl@ceph.mon.c.smithi115.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:03:42.133 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:03:41 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:03:43.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:03:42 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:03:43.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:03:42 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:03:43.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:03:42 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:03:43.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:03:42 smithi115 conmon[90393]: 2022-05-14T02:03:42.784+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:03:43.112 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:03:42 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:03:43.113 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:03:42 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:03:43.113 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:03:42 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:03:44.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:03:43 smithi047 ceph-mon[77147]: pgmap v5030: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:03:44.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:03:43 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:03:44.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:03:43 smithi115 conmon[90393]: 2022-05-14T02:03:43.806+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:03:44.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:03:43 smithi115 ceph-mon[78593]: pgmap v5030: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:03:44.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:03:43 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:03:45.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:03:44 smithi115 conmon[90393]: 2022-05-14T02:03:44.818+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:03:46.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:03:45 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:03:46.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:03:45 smithi047 ceph-mon[77147]: pgmap v5031: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:03:46.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:03:45 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 9355 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:03:46.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:03:45 smithi115 conmon[90393]: 2022-05-14T02:03:45.809+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:03:46.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:03:45 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:03:46.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:03:45 smithi115 ceph-mon[78593]: pgmap v5031: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:03:46.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:03:45 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 9355 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:03:47.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:03:46 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:03:47.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:03:46 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:03:47.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:03:46 smithi115 conmon[90393]: 2022-05-14T02:03:46.790+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:03:48.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:03:47 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:03:48.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:03:47 smithi047 ceph-mon[77147]: pgmap v5032: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-05-14T02:03:48.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:03:47 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:03:48.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:03:47 smithi115 conmon[90393]: 2022-05-14T02:03:47.789+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:03:48.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:03:47 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:03:48.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:03:47 smithi115 ceph-mon[78593]: pgmap v5032: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-05-14T02:03:48.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:03:47 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:03:49.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:03:48 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:03:49.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:03:48 smithi115 conmon[90393]: 2022-05-14T02:03:48.740+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:03:49.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:03:48 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:03:50.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:03:49 smithi047 ceph-mon[77147]: pgmap v5033: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:03:50.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:03:49 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:03:50.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:03:49 smithi115 conmon[90393]: 2022-05-14T02:03:49.721+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:03:50.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:03:49 smithi115 ceph-mon[78593]: pgmap v5033: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:03:50.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:03:49 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:03:51.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:03:50 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:03:51.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:03:50 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:03:51.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:03:50 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:03:51.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:03:50 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 9360 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:03:51.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:03:50 smithi115 conmon[90393]: 2022-05-14T02:03:50.741+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:03:51.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:03:50 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:03:51.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:03:50 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:03:51.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:03:50 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:03:51.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:03:50 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 9360 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:03:52.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:03:51 smithi047 ceph-mon[77147]: pgmap v5034: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:03:52.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:03:51 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:03:52.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:03:51 smithi115 conmon[90393]: 2022-05-14T02:03:51.781+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:03:52.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:03:51 smithi115 ceph-mon[78593]: pgmap v5034: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:03:52.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:03:51 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:03:53.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:03:52 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:03:53.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:03:52 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:03:53.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:03:52 smithi115 conmon[90393]: 2022-05-14T02:03:52.799+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:03:54.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:03:53 smithi047 ceph-mon[77147]: pgmap v5035: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:03:54.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:03:53 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:03:54.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:03:53 smithi115 conmon[90393]: 2022-05-14T02:03:53.778+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:03:54.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:03:53 smithi115 ceph-mon[78593]: pgmap v5035: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:03:54.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:03:53 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:03:55.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:03:54 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:03:55.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:03:54 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:03:55.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:03:54 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:03:55.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:03:54 smithi115 conmon[90393]: 2022-05-14T02:03:54.733+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:03:55.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:03:54 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:03:55.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:03:54 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:03:55.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:03:54 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:03:56.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:03:55 smithi047 ceph-mon[77147]: pgmap v5036: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:03:56.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:03:55 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:03:56.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:03:55 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 9365 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:03:56.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:03:55 smithi115 conmon[90393]: 2022-05-14T02:03:55.728+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:03:56.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:03:55 smithi115 ceph-mon[78593]: pgmap v5036: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:03:56.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:03:55 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:03:56.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:03:55 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 9365 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:03:57.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:03:56 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:03:57.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:03:56 smithi115 conmon[90393]: 2022-05-14T02:03:56.680+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:03:57.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:03:56 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:03:58.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:03:57 smithi047 ceph-mon[77147]: pgmap v5037: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-05-14T02:03:58.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:03:57 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:03:58.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:03:57 smithi115 conmon[90393]: 2022-05-14T02:03:57.645+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:03:58.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:03:57 smithi115 ceph-mon[78593]: pgmap v5037: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-05-14T02:03:58.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:03:57 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:03:59.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:03:58 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:03:59.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:03:58 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:03:59.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:03:58 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:03:59.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:03:58 smithi115 conmon[90393]: 2022-05-14T02:03:58.627+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:03:59.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:03:58 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:03:59.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:03:58 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:03:59.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:03:58 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:04:00.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:03:59 smithi047 ceph-mon[77147]: pgmap v5038: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:04:00.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:03:59 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:04:00.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:03:59 smithi115 conmon[90393]: 2022-05-14T02:03:59.664+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:04:00.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:03:59 smithi115 ceph-mon[78593]: pgmap v5038: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:04:00.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:03:59 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:04:01.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:04:00 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:04:01.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:04:00 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 9370 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:04:01.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:04:00 smithi115 conmon[90393]: 2022-05-14T02:04:00.672+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:04:01.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:04:00 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:04:01.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:04:00 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 9370 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:04:02.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:04:01 smithi047 ceph-mon[77147]: pgmap v5039: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:04:02.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:04:01 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:04:02.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:04:01 smithi115 conmon[90393]: 2022-05-14T02:04:01.645+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:04:02.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:04:01 smithi115 ceph-mon[78593]: pgmap v5039: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:04:02.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:04:01 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:04:03.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:04:02 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:04:03.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:04:02 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:04:03.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:04:02 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:04:03.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:04:02 smithi115 conmon[90393]: 2022-05-14T02:04:02.674+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:04:03.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:04:02 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:04:03.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:04:02 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:04:03.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:04:02 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:04:04.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:04:03 smithi047 ceph-mon[77147]: pgmap v5040: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:04:04.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:04:03 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:04:04.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:04:03 smithi115 conmon[90393]: 2022-05-14T02:04:03.715+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:04:04.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:04:03 smithi115 ceph-mon[78593]: pgmap v5040: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:04:04.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:04:03 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:04:05.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:04:04 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:04:05.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:04:04 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:04:05.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:04:04 smithi115 conmon[90393]: 2022-05-14T02:04:04.674+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:04:06.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:04:05 smithi047 ceph-mon[77147]: pgmap v5041: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:04:06.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:04:05 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:04:06.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:04:05 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 9375 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:04:06.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:04:05 smithi115 conmon[90393]: 2022-05-14T02:04:05.667+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:04:06.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:04:05 smithi115 ceph-mon[78593]: pgmap v5041: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:04:06.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:04:05 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:04:06.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:04:05 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 9375 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:04:07.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:04:06 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:04:07.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:04:06 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:04:07.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:04:06 smithi115 conmon[90393]: 2022-05-14T02:04:06.634+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:04:08.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:04:07 smithi047 ceph-mon[77147]: pgmap v5042: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-05-14T02:04:08.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:04:07 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:04:08.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:04:07 smithi115 conmon[90393]: 2022-05-14T02:04:07.639+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:04:08.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:04:07 smithi115 ceph-mon[78593]: pgmap v5042: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-05-14T02:04:08.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:04:07 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:04:09.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:04:08 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:04:09.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:04:08 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:04:09.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:04:08 smithi115 conmon[90393]: 2022-05-14T02:04:08.653+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:04:10.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:04:09 smithi047 ceph-mon[77147]: pgmap v5043: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:04:10.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:04:09 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:04:10.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:04:09 smithi115 conmon[90393]: 2022-05-14T02:04:09.691+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:04:10.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:04:09 smithi115 ceph-mon[78593]: pgmap v5043: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:04:10.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:04:09 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:04:11.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:04:10 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:04:11.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:04:10 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:04:11.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:04:10 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:04:11.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:04:10 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 9380 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:04:11.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:04:10 smithi115 conmon[90393]: 2022-05-14T02:04:10.680+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:04:11.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:04:10 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:04:11.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:04:10 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:04:11.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:04:10 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:04:11.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:04:10 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 9380 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:04:12.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:04:11 smithi047 ceph-mon[77147]: pgmap v5044: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:04:12.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:04:11 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:04:12.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:04:11 smithi115 conmon[90393]: 2022-05-14T02:04:11.706+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:04:12.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:04:11 smithi115 ceph-mon[78593]: pgmap v5044: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:04:12.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:04:11 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:04:13.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:04:12 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:04:13.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:04:12 smithi115 conmon[90393]: 2022-05-14T02:04:12.748+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:04:13.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:04:12 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:04:14.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:04:13 smithi047 ceph-mon[77147]: pgmap v5045: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:04:14.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:04:13 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:04:14.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:04:13 smithi115 conmon[90393]: 2022-05-14T02:04:13.761+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:04:14.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:04:13 smithi115 ceph-mon[78593]: pgmap v5045: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:04:14.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:04:13 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:04:15.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:04:14 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:04:15.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:04:14 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:04:15.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:04:14 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:04:15.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:04:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T02:04:15.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:04:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T02:04:15.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:04:14 smithi115 conmon[90393]: 2022-05-14T02:04:14.792+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:04:15.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:04:14 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:04:15.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:04:14 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:04:15.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:04:14 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:04:15.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:04:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T02:04:15.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:04:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T02:04:16.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:04:15 smithi047 ceph-mon[77147]: pgmap v5046: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:04:16.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:04:15 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:04:16.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:04:15 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 9385 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:04:16.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:04:15 smithi115 conmon[90393]: 2022-05-14T02:04:15.801+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:04:16.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:04:15 smithi115 ceph-mon[78593]: pgmap v5046: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:04:16.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:04:15 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:04:16.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:04:15 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 9385 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:04:17.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:04:16 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:04:17.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:04:16 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:04:17.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:04:16 smithi115 conmon[90393]: 2022-05-14T02:04:16.755+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:04:18.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:04:17 smithi047 ceph-mon[77147]: pgmap v5047: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-05-14T02:04:18.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:04:17 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:04:18.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:04:17 smithi115 ceph-mon[78593]: pgmap v5047: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-05-14T02:04:18.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:04:17 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:04:18.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:04:17 smithi115 conmon[90393]: 2022-05-14T02:04:17.782+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:04:19.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:04:18 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:04:19.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:04:18 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:04:19.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:04:18 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:04:19.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:04:18 smithi115 conmon[90393]: 2022-05-14T02:04:18.812+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:04:19.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:04:18 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:04:19.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:04:18 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:04:19.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:04:18 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:04:20.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:04:19 smithi047 ceph-mon[77147]: pgmap v5048: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:04:20.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:04:19 smithi115 ceph-mon[78593]: pgmap v5048: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:04:20.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:04:19 smithi115 conmon[90393]: 2022-05-14T02:04:19.856+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:04:21.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:04:20 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:04:21.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:04:20 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 9390 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:04:21.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:04:20 smithi115 conmon[90393]: 2022-05-14T02:04:20.882+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:04:21.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:04:20 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:04:21.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:04:20 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 9390 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:04:22.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:04:21 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:04:22.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:04:21 smithi047 ceph-mon[77147]: pgmap v5049: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:04:22.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:04:21 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:04:22.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:04:21 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:04:22.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:04:21 smithi115 conmon[90393]: 2022-05-14T02:04:21.841+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:04:22.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:04:21 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:04:22.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:04:21 smithi115 ceph-mon[78593]: pgmap v5049: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:04:22.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:04:21 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:04:22.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:04:21 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:04:23.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:04:22 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:04:23.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:04:22 smithi115 conmon[90393]: 2022-05-14T02:04:22.796+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:04:23.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:04:22 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:04:24.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:04:23 smithi115 conmon[90393]: 2022-05-14T02:04:23.815+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:04:24.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:04:23 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:04:24.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:04:23 smithi115 ceph-mon[78593]: pgmap v5050: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:04:24.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:04:23 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:04:24.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:04:23 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:04:24.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:04:23 smithi047 ceph-mon[77147]: pgmap v5050: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:04:24.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:04:23 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:04:25.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:04:24 smithi115 conmon[90393]: 2022-05-14T02:04:24.794+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:04:26.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:04:25 smithi115 conmon[90393]: 2022-05-14T02:04:25.831+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:04:26.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:04:25 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:04:26.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:04:25 smithi115 ceph-mon[78593]: pgmap v5051: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:04:26.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:04:25 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:04:26.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:04:25 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 9395 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:04:26.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:04:25 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:04:26.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:04:25 smithi047 ceph-mon[77147]: pgmap v5051: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:04:26.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:04:25 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:04:26.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:04:25 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 9395 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:04:27.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:04:26 smithi115 conmon[90393]: 2022-05-14T02:04:26.855+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:04:28.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:04:27 smithi115 conmon[90393]: 2022-05-14T02:04:27.841+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:04:28.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:04:27 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:04:28.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:04:27 smithi115 ceph-mon[78593]: pgmap v5052: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-05-14T02:04:28.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:04:27 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:04:28.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:04:27 smithi047 ceph-mon[77147]: pgmap v5052: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-05-14T02:04:29.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:04:28 smithi115 conmon[90393]: 2022-05-14T02:04:28.806+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:04:29.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:04:28 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:04:29.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:04:28 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:04:30.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:04:29 smithi115 conmon[90393]: 2022-05-14T02:04:29.765+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:04:30.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:04:29 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:04:30.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:04:29 smithi115 ceph-mon[78593]: pgmap v5053: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:04:30.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:04:29 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:04:30.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:04:29 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:04:30.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:04:29 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:04:30.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:04:29 smithi047 ceph-mon[77147]: pgmap v5053: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:04:30.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:04:29 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:04:30.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:04:29 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:04:31.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:04:30 smithi115 conmon[90393]: 2022-05-14T02:04:30.728+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:04:31.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:04:30 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:04:31.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:04:30 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:04:31.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:04:30 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 9400 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:04:31.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:04:30 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:04:31.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:04:30 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:04:31.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:04:30 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:04:31.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:04:30 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 9400 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:04:31.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:04:30 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:04:32.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:04:31 smithi115 conmon[90393]: 2022-05-14T02:04:31.723+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:04:32.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:04:31 smithi115 ceph-mon[78593]: pgmap v5054: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:04:32.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:04:31 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:04:32.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:04:31 smithi047 ceph-mon[77147]: pgmap v5054: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:04:32.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:04:31 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:04:33.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:04:32 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:04:33.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:04:32 smithi115 conmon[90393]: 2022-05-14T02:04:32.686+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:04:33.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:04:32 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:04:34.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:04:33 smithi115 conmon[90393]: 2022-05-14T02:04:33.662+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:04:34.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:04:33 smithi115 ceph-mon[78593]: pgmap v5055: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:04:34.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:04:33 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:04:34.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:04:33 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:04:34.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:04:33 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:04:34.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:04:33 smithi047 ceph-mon[77147]: pgmap v5055: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:04:34.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:04:33 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:04:34.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:04:33 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:04:34.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:04:33 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:04:35.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:04:34 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:04:35.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:04:34 smithi115 conmon[90393]: 2022-05-14T02:04:34.673+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:04:35.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:04:34 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:04:36.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:04:35 smithi115 conmon[90393]: 2022-05-14T02:04:35.656+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:04:36.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:04:35 smithi115 ceph-mon[78593]: pgmap v5056: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:04:36.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:04:35 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 9405 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:04:36.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:04:35 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:04:36.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:04:35 smithi047 ceph-mon[77147]: pgmap v5056: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:04:36.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:04:35 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 9405 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:04:36.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:04:35 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:04:37.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:04:36 smithi115 conmon[90393]: 2022-05-14T02:04:36.694+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:04:37.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:04:36 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:04:37.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:04:36 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:04:38.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:04:37 smithi115 conmon[90393]: 2022-05-14T02:04:37.673+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:04:38.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:04:37 smithi115 ceph-mon[78593]: pgmap v5057: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-05-14T02:04:38.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:04:37 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:04:38.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:04:37 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:04:38.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:04:37 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:04:38.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:04:37 smithi047 ceph-mon[77147]: pgmap v5057: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-05-14T02:04:38.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:04:37 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:04:38.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:04:37 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:04:38.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:04:37 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:04:39.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:04:38 smithi115 conmon[90393]: 2022-05-14T02:04:38.690+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:04:39.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:04:38 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:04:39.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:04:38 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:04:40.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:04:39 smithi115 conmon[90393]: 2022-05-14T02:04:39.733+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:04:40.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:04:39 smithi115 ceph-mon[78593]: pgmap v5058: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:04:40.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:04:39 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:04:40.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:04:39 smithi047 ceph-mon[77147]: pgmap v5058: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:04:40.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:04:39 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:04:41.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:04:40 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 9410 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:04:41.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:04:40 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T02:04:41.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:04:40 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T02:04:41.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:04:40 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T02:04:41.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:04:40 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:04:41.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:04:40 smithi115 conmon[90393]: 2022-05-14T02:04:40.704+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:04:41.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:04:40 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 9410 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:04:41.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:04:40 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T02:04:41.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:04:40 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T02:04:41.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:04:40 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T02:04:41.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:04:40 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:04:42.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:04:41 smithi115 conmon[90393]: 2022-05-14T02:04:41.665+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:04:42.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:04:41 smithi115 ceph-mon[78593]: pgmap v5059: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:04:42.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:04:41 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:04:42.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:04:41 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:04:42.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:04:41 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:04:42.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:04:41 smithi047 ceph-mon[77147]: pgmap v5059: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:04:42.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:04:41 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:04:42.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:04:41 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:04:42.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:04:41 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:04:43.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:04:42 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:04:43.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:04:42 smithi115 conmon[90393]: 2022-05-14T02:04:42.623+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:04:43.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:04:42 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:04:44.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:04:43 smithi047 ceph-mon[77147]: pgmap v5060: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:04:44.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:04:43 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:04:44.042 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:04:43 smithi047 conmon[78521]: 2022-05-14T02:04:43.760+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:04:44.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:04:43 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T02:04:44.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:04:43 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:04:44.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:04:43 smithi047 conmon[78521]: yield 2022-05-14T02:04:44.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:04:43 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:04:44.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:04:43 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:04:44.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:04:43 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:04:44.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:04:43 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T02:04:44.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:04:43 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:04:44.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:04:43 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T02:04:44.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:04:43 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:04:44.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:04:43 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T02:04:44.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:04:43 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:04:44.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:04:43 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T02:04:44.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:04:43 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:04:44.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:04:43 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T02:04:44.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:04:43 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:04:44.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:04:43 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:04:44.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:04:43 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:04:44.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:04:43 smithi115 conmon[90393]: 2022-05-14T02:04:43.667+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:04:44.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:04:43 smithi115 ceph-mon[78593]: pgmap v5060: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:04:44.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:04:43 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:04:45.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:04:44 smithi115 conmon[90393]: 2022-05-14T02:04:44.701+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:04:45.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:04:44 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:04:45.109 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T02:04:45.109 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:04:45.109 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T02:04:45.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:04:45.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:04:45.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:04:45.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T02:04:45.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:04:45.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T02:04:45.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:04:45.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T02:04:45.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:04:45.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T02:04:45.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:04:45.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T02:04:45.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:04:45.113 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:04:45.113 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:04:45.113 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:04:44 smithi115 ceph-mon[78593]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T02:04:45.113 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:04:44 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:04:45.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:04:44 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:04:45.292 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T02:04:45.292 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:04:45.292 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T02:04:45.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:04:45.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:04:45.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:04:45.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T02:04:45.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:04:45.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T02:04:45.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:04:45.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T02:04:45.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:04:45.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T02:04:45.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:04:45.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T02:04:45.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:04:45.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:04:45.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:04:45.296 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:04:44 smithi047 ceph-mon[77147]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T02:04:45.296 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:04:44 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:04:46.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:04:45 smithi115 conmon[90393]: 2022-05-14T02:04:45.715+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:04:46.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:04:45 smithi115 ceph-mon[78593]: pgmap v5061: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:04:46.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:04:45 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 9415 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:04:46.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:04:45 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:04:46.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:04:45 smithi047 ceph-mon[77147]: pgmap v5061: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:04:46.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:04:45 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 9415 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:04:46.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:04:45 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:04:47.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:04:46 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:04:47.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:04:46 smithi115 conmon[90393]: 2022-05-14T02:04:46.706+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:04:47.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:04:46 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:04:47.292 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:04:46 smithi047 conmon[78521]: 2022-05-14T02:04:46.832+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:04:47.292 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:04:46 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T02:04:47.293 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:04:46 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:04:47.293 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:04:46 smithi047 conmon[78521]: yield 2022-05-14T02:04:47.293 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:04:46 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:04:47.293 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:04:46 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:04:47.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:04:46 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:04:47.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:04:46 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T02:04:47.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:04:46 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:04:47.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:04:46 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T02:04:47.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:04:46 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:04:47.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:04:46 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T02:04:47.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:04:46 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:04:47.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:04:46 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T02:04:47.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:04:46 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:04:47.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:04:46 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T02:04:47.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:04:46 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:04:47.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:04:46 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:04:47.297 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:04:46 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:04:47.297 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:04:46 smithi047 conmon[78521]: 2022-05-14T02:04:46.833+0000 7f5344781700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T02:04:47.298 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:04:46 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T02:04:47.298 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:04:46 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:04:47.298 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:04:46 smithi047 conmon[78521]: yield 2022-05-14T02:04:47.299 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:04:46 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:04:47.299 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:04:46 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:04:47.299 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:04:46 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:04:47.300 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:04:46 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T02:04:47.300 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:04:46 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:04:47.301 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:04:46 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T02:04:47.301 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:04:46 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:04:47.302 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:04:46 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T02:04:47.302 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:04:46 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:04:47.302 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:04:46 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T02:04:47.303 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:04:46 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:04:47.303 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:04:46 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T02:04:47.304 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:04:46 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:04:47.304 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:04:46 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:04:47.304 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:04:46 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:04:47.305 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:04:46 smithi047 conmon[78521]: 2022-05-14T02:04:47.305 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:04:46 smithi047 conmon[78521]: During handling of the above exception, another exception occurred: 2022-05-14T02:04:47.305 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:04:46 smithi047 conmon[78521]: 2022-05-14T02:04:47.306 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:04:46 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T02:04:47.306 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:04:46 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T02:04:47.307 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:04:46 smithi047 conmon[78521]: return f(*arg) 2022-05-14T02:04:47.307 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:04:46 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T02:04:47.307 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:04:46 smithi047 conmon[78521]: self._write_client_files(client_files, host) 2022-05-14T02:04:47.307 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:04:46 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T02:04:47.308 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:04:46 smithi047 conmon[78521]: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T02:04:47.308 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:04:46 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T02:04:47.308 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:04:46 smithi047 conmon[78521]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T02:04:47.308 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:04:46 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T02:04:47.309 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:04:46 smithi047 conmon[78521]: return self.event_loop.get_result(coro) 2022-05-14T02:04:47.309 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:04:46 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T02:04:47.309 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:04:46 smithi047 conmon[78521]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T02:04:47.310 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:04:46 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T02:04:47.310 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:04:46 smithi047 conmon[78521]: return self.__get_result() 2022-05-14T02:04:47.310 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:04:46 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T02:04:47.311 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:04:46 smithi047 conmon[78521]: raise self._exception 2022-05-14T02:04:47.311 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:04:46 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T02:04:47.311 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:04:46 smithi047 conmon[78521]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T02:04:47.311 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:04:46 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T02:04:47.312 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:04:46 smithi047 conmon[78521]: conn = await self._remote_connection(host, addr) 2022-05-14T02:04:47.312 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:04:46 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T02:04:47.312 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:04:46 smithi047 conmon[78521]: raise 2022-05-14T02:04:47.312 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:04:46 smithi047 conmon[78521]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T02:04:47.313 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:04:46 smithi047 conmon[78521]: self.gen.throw(type, value, traceback) 2022-05-14T02:04:47.313 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:04:46 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T02:04:47.313 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:04:46 smithi047 conmon[78521]: raise OrchestratorError(msg) 2022-05-14T02:04:47.313 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:04:46 smithi047 conmon[78521]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:04:48.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:04:47 smithi115 conmon[90393]: 2022-05-14T02:04:47.661+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:04:48.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:04:47 smithi115 ceph-mon[78593]: pgmap v5062: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-05-14T02:04:48.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:04:47 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:04:48.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T02:04:48.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:04:48.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T02:04:48.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:04:48.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:04:48.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:04:48.113 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T02:04:48.113 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:04:48.114 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T02:04:48.114 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:04:48.114 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T02:04:48.115 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:04:48.115 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T02:04:48.115 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:04:48.116 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T02:04:48.116 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:04:48.116 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:04:48.117 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:04:48.117 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:04:47 smithi115 ceph-mon[78593]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T02:04:48.118 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T02:04:48.118 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:04:48.118 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T02:04:48.118 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:04:48.119 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:04:48.119 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:04:48.119 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T02:04:48.119 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:04:48.119 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T02:04:48.120 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:04:48.120 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T02:04:48.120 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:04:48.120 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T02:04:48.121 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:04:48.121 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T02:04:48.121 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:04:48.121 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:04:48.121 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:04:48.122 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T02:04:48.122 INFO:journalctl@ceph.mon.c.smithi115.stdout: During handling of the above exception, another exception occurred: 2022-05-14T02:04:48.122 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T02:04:48.122 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T02:04:48.123 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T02:04:48.123 INFO:journalctl@ceph.mon.c.smithi115.stdout: return f(*arg) 2022-05-14T02:04:48.123 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T02:04:48.123 INFO:journalctl@ceph.mon.c.smithi115.stdout: self._write_client_files(client_files, host) 2022-05-14T02:04:48.123 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T02:04:48.124 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T02:04:48.124 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T02:04:48.124 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T02:04:48.125 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T02:04:48.125 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.event_loop.get_result(coro) 2022-05-14T02:04:48.125 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T02:04:48.125 INFO:journalctl@ceph.mon.c.smithi115.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T02:04:48.126 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T02:04:48.126 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.__get_result() 2022-05-14T02:04:48.126 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T02:04:48.126 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise self._exception 2022-05-14T02:04:48.126 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T02:04:48.127 INFO:journalctl@ceph.mon.c.smithi115.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T02:04:48.127 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T02:04:48.127 INFO:journalctl@ceph.mon.c.smithi115.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T02:04:48.127 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T02:04:48.127 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise 2022-05-14T02:04:48.128 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T02:04:48.128 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.gen.throw(type, value, traceback) 2022-05-14T02:04:48.128 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T02:04:48.128 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OrchestratorError(msg) 2022-05-14T02:04:48.128 INFO:journalctl@ceph.mon.c.smithi115.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:04:48.129 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:04:47 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:04:48.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:04:47 smithi047 ceph-mon[77147]: pgmap v5062: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-05-14T02:04:48.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:04:47 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:04:48.292 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T02:04:48.292 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:04:48.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T02:04:48.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:04:48.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:04:48.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:04:48.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T02:04:48.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:04:48.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T02:04:48.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:04:48.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T02:04:48.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:04:48.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T02:04:48.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:04:48.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T02:04:48.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:04:48.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:04:48.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:04:48.296 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:04:47 smithi047 ceph-mon[77147]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T02:04:48.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T02:04:48.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:04:48.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T02:04:48.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:04:48.298 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:04:48.298 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:04:48.298 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T02:04:48.298 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:04:48.298 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T02:04:48.299 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:04:48.299 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T02:04:48.299 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:04:48.299 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T02:04:48.299 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:04:48.300 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T02:04:48.300 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:04:48.300 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:04:48.300 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:04:48.301 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T02:04:48.301 INFO:journalctl@ceph.mon.b.smithi047.stdout: During handling of the above exception, another exception occurred: 2022-05-14T02:04:48.301 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T02:04:48.301 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T02:04:48.301 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T02:04:48.302 INFO:journalctl@ceph.mon.b.smithi047.stdout: return f(*arg) 2022-05-14T02:04:48.302 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T02:04:48.302 INFO:journalctl@ceph.mon.b.smithi047.stdout: self._write_client_files(client_files, host) 2022-05-14T02:04:48.303 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T02:04:48.303 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T02:04:48.306 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T02:04:48.306 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T02:04:48.306 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T02:04:48.306 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.event_loop.get_result(coro) 2022-05-14T02:04:48.307 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T02:04:48.307 INFO:journalctl@ceph.mon.b.smithi047.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T02:04:48.307 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T02:04:48.307 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.__get_result() 2022-05-14T02:04:48.308 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T02:04:48.308 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise self._exception 2022-05-14T02:04:48.308 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T02:04:48.308 INFO:journalctl@ceph.mon.b.smithi047.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T02:04:48.308 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T02:04:48.309 INFO:journalctl@ceph.mon.b.smithi047.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T02:04:48.309 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T02:04:48.309 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise 2022-05-14T02:04:48.309 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T02:04:48.309 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.gen.throw(type, value, traceback) 2022-05-14T02:04:48.310 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T02:04:48.310 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OrchestratorError(msg) 2022-05-14T02:04:48.310 INFO:journalctl@ceph.mon.b.smithi047.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:04:48.310 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:04:47 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:04:49.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:04:48 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:04:49.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:04:48 smithi115 conmon[90393]: 2022-05-14T02:04:48.631+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:04:49.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:04:48 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:04:49.878 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:04:49 smithi115 conmon[90393]: 2022-05-14T02:04:49.611+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:04:50.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:04:49 smithi047 ceph-mon[77147]: pgmap v5063: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:04:50.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:04:49 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:04:50.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:04:49 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:04:50.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:04:49 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:04:50.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:04:49 smithi115 ceph-mon[78593]: pgmap v5063: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:04:50.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:04:49 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:04:50.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:04:49 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:04:50.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:04:49 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:04:50.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:04:50 smithi115 conmon[90393]: 2022-05-14T02:04:50.577+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:04:51.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:04:50 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 9420 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:04:51.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:04:50 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:04:51.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:04:50 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 9420 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:04:51.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:04:50 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:04:51.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:04:51 smithi115 conmon[90393]: 2022-05-14T02:04:51.571+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:04:52.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:04:51 smithi047 ceph-mon[77147]: pgmap v5064: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:04:52.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:04:51 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:04:52.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:04:51 smithi115 ceph-mon[78593]: pgmap v5064: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:04:52.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:04:51 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:04:52.895 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:04:52 smithi115 conmon[90393]: 2022-05-14T02:04:52.614+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:04:53.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:04:52 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:04:53.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:04:52 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:04:53.896 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:04:53 smithi115 conmon[90393]: 2022-05-14T02:04:53.635+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:04:54.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:04:53 smithi047 ceph-mon[77147]: pgmap v5065: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:04:54.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:04:53 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:04:54.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:04:53 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:04:54.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:04:53 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:04:54.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:04:53 smithi115 ceph-mon[78593]: pgmap v5065: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:04:54.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:04:53 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:04:54.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:04:53 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:04:54.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:04:53 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:04:54.901 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:04:54 smithi115 conmon[90393]: 2022-05-14T02:04:54.616+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:04:55.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:04:54 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:04:55.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:04:54 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:04:55.902 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:04:55 smithi115 conmon[90393]: 2022-05-14T02:04:55.631+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:04:56.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:04:55 smithi047 ceph-mon[77147]: pgmap v5066: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:04:56.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:04:55 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 9425 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:04:56.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:04:55 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:04:56.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:04:55 smithi115 ceph-mon[78593]: pgmap v5066: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:04:56.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:04:55 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 9425 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:04:56.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:04:55 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:04:56.907 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:04:56 smithi115 conmon[90393]: 2022-05-14T02:04:56.620+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:04:57.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:04:56 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:04:57.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:04:56 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:04:57.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:04:57 smithi115 conmon[90393]: 2022-05-14T02:04:57.600+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:04:58.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:04:57 smithi047 ceph-mon[77147]: pgmap v5067: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-05-14T02:04:58.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:04:57 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:04:58.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:04:57 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:04:58.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:04:57 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:04:58.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:04:57 smithi115 ceph-mon[78593]: pgmap v5067: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-05-14T02:04:58.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:04:57 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:04:58.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:04:57 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:04:58.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:04:57 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:04:58.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:04:58 smithi115 conmon[90393]: 2022-05-14T02:04:58.599+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:04:59.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:04:58 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:04:59.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:04:58 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:04:59.915 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:04:59 smithi115 conmon[90393]: 2022-05-14T02:04:59.613+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:05:00.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:04:59 smithi047 ceph-mon[77147]: pgmap v5068: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:05:00.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:04:59 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:05:00.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:04:59 smithi115 ceph-mon[78593]: pgmap v5068: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:05:00.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:04:59 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:05:00.920 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:05:00 smithi115 conmon[90393]: 2022-05-14T02:05:00.642+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:05:01.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:05:00 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 9430 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:05:01.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:05:00 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:05:01.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:05:00 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 9430 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:05:01.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:05:00 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:05:01.921 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:05:01 smithi115 conmon[90393]: 2022-05-14T02:05:01.623+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:05:02.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:05:01 smithi047 ceph-mon[77147]: pgmap v5069: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:05:02.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:05:01 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:05:02.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:05:01 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:05:02.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:05:01 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:05:02.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:05:01 smithi115 ceph-mon[78593]: pgmap v5069: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:05:02.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:05:01 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:05:02.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:05:01 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:05:02.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:05:01 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:05:02.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:05:02 smithi115 conmon[90393]: 2022-05-14T02:05:02.588+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:05:03.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:05:02 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:05:03.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:05:02 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:05:03.928 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:05:03 smithi115 conmon[90393]: 2022-05-14T02:05:03.635+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:05:04.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:05:03 smithi047 ceph-mon[77147]: pgmap v5070: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:05:04.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:05:03 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:05:04.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:05:03 smithi115 ceph-mon[78593]: pgmap v5070: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:05:04.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:05:03 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:05:04.935 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:05:04 smithi115 conmon[90393]: 2022-05-14T02:05:04.608+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:05:05.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:05:04 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:05:05.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:05:04 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:05:05.934 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:05:05 smithi115 conmon[90393]: 2022-05-14T02:05:05.658+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:05:06.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:05:05 smithi047 ceph-mon[77147]: pgmap v5071: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:05:06.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:05:05 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 9435 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:05:06.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:05:05 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:05:06.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:05:05 smithi115 ceph-mon[78593]: pgmap v5071: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:05:06.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:05:05 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 9435 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:05:06.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:05:05 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:05:06.939 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:05:06 smithi115 conmon[90393]: 2022-05-14T02:05:06.645+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:05:07.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:05:06 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:05:07.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:05:06 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:05:07.941 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:05:07 smithi115 conmon[90393]: 2022-05-14T02:05:07.680+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:05:08.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:05:07 smithi047 ceph-mon[77147]: pgmap v5072: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-05-14T02:05:08.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:05:07 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:05:08.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:05:07 smithi115 ceph-mon[78593]: pgmap v5072: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-05-14T02:05:08.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:05:07 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:05:08.946 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:05:08 smithi115 conmon[90393]: 2022-05-14T02:05:08.641+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:05:09.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:05:08 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:05:09.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:05:08 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:05:09.947 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:05:09 smithi115 conmon[90393]: 2022-05-14T02:05:09.624+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:05:10.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:05:09 smithi047 ceph-mon[77147]: pgmap v5073: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:05:10.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:05:09 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:05:10.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:05:09 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:05:10.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:05:09 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:05:10.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:05:09 smithi115 ceph-mon[78593]: pgmap v5073: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:05:10.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:05:09 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:05:10.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:05:09 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:05:10.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:05:09 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:05:10.954 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:05:10 smithi115 conmon[90393]: 2022-05-14T02:05:10.672+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:05:11.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:05:10 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 9440 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:05:11.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:05:10 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:05:11.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:05:10 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 9440 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:05:11.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:05:10 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:05:12.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:05:11 smithi115 ceph-mon[78593]: pgmap v5074: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:05:12.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:05:11 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:05:12.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:05:11 smithi115 conmon[90393]: 2022-05-14T02:05:11.718+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:05:12.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:05:11 smithi047 ceph-mon[77147]: pgmap v5074: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:05:12.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:05:11 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:05:13.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:05:12 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:05:13.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:05:12 smithi115 conmon[90393]: 2022-05-14T02:05:12.735+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:05:13.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:05:12 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:05:13.962 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:05:13 smithi115 conmon[90393]: 2022-05-14T02:05:13.697+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:05:14.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:05:13 smithi047 ceph-mon[77147]: pgmap v5075: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:05:14.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:05:13 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:05:14.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:05:13 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:05:14.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:05:13 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:05:14.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:05:13 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T02:05:14.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:05:13 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T02:05:14.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:05:13 smithi115 ceph-mon[78593]: pgmap v5075: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:05:14.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:05:13 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:05:14.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:05:13 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:05:14.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:05:13 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:05:14.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:05:13 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T02:05:14.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:05:13 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T02:05:14.969 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:05:14 smithi115 conmon[90393]: 2022-05-14T02:05:14.714+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:05:15.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:05:14 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:05:15.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:05:14 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:05:16.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:05:15 smithi115 conmon[90393]: 2022-05-14T02:05:15.760+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:05:16.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:05:15 smithi115 ceph-mon[78593]: pgmap v5076: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:05:16.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:05:15 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 9445 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:05:16.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:05:15 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:05:16.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:05:15 smithi047 ceph-mon[77147]: pgmap v5076: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:05:16.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:05:15 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 9445 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:05:16.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:05:15 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:05:17.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:05:16 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:05:17.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:05:16 smithi115 conmon[90393]: 2022-05-14T02:05:16.766+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:05:17.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:05:16 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:05:18.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:05:17 smithi115 ceph-mon[78593]: pgmap v5077: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-05-14T02:05:18.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:05:17 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:05:18.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:05:17 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:05:18.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:05:17 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:05:18.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:05:17 smithi115 conmon[90393]: 2022-05-14T02:05:17.773+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:05:18.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:05:17 smithi047 ceph-mon[77147]: pgmap v5077: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-05-14T02:05:18.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:05:17 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:05:18.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:05:17 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:05:18.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:05:17 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:05:19.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:05:18 smithi115 conmon[90393]: 2022-05-14T02:05:18.741+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:05:19.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:05:18 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:05:19.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:05:18 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:05:20.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:05:19 smithi115 conmon[90393]: 2022-05-14T02:05:19.745+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:05:20.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:05:19 smithi115 ceph-mon[78593]: pgmap v5078: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:05:20.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:05:19 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:05:20.296 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:05:19 smithi047 ceph-mon[77147]: pgmap v5078: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:05:20.296 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:05:19 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:05:21.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:05:20 smithi115 conmon[90393]: 2022-05-14T02:05:20.753+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:05:21.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:05:20 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 9450 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:05:21.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:05:20 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:05:21.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:05:20 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 9450 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:05:21.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:05:20 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:05:21.988 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:05:21 smithi115 conmon[90393]: 2022-05-14T02:05:21.707+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:05:22.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:05:21 smithi047 ceph-mon[77147]: pgmap v5079: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:05:22.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:05:21 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:05:22.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:05:21 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:05:22.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:05:21 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:05:22.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:05:21 smithi115 ceph-mon[78593]: pgmap v5079: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:05:22.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:05:21 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:05:22.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:05:21 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:05:22.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:05:21 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:05:22.994 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:05:22 smithi115 conmon[90393]: 2022-05-14T02:05:22.687+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:05:23.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:05:22 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:05:23.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:05:22 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:05:23.995 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:05:23 smithi115 conmon[90393]: 2022-05-14T02:05:23.716+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:05:24.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:05:23 smithi047 ceph-mon[77147]: pgmap v5080: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:05:24.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:05:23 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:05:24.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:05:23 smithi115 ceph-mon[78593]: pgmap v5080: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:05:24.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:05:23 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:05:25.001 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:05:24 smithi115 conmon[90393]: 2022-05-14T02:05:24.716+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:05:25.001 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:05:25 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:05:25.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:05:24 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:05:26.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:05:25 smithi115 conmon[90393]: 2022-05-14T02:05:25.754+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:05:26.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:05:26 smithi115 ceph-mon[78593]: pgmap v5081: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:05:26.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:05:26 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 9455 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:05:26.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:05:26 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:05:26.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:05:26 smithi047 ceph-mon[77147]: pgmap v5081: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:05:26.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:05:26 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 9455 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:05:26.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:05:26 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:05:27.007 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:05:26 smithi115 conmon[90393]: 2022-05-14T02:05:26.749+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:05:27.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:05:27 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:05:27.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:05:27 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:05:28.009 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:05:27 smithi115 conmon[90393]: 2022-05-14T02:05:27.750+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:05:28.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:05:28 smithi047 ceph-mon[77147]: pgmap v5082: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-05-14T02:05:28.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:05:28 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:05:28.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:05:28 smithi115 ceph-mon[78593]: pgmap v5082: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-05-14T02:05:28.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:05:28 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:05:29.014 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:05:28 smithi115 conmon[90393]: 2022-05-14T02:05:28.749+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:05:29.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:05:29 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:05:29.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:05:29 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:05:30.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:05:29 smithi115 conmon[90393]: 2022-05-14T02:05:29.788+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:05:30.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:05:30 smithi115 ceph-mon[78593]: pgmap v5083: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:05:30.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:05:30 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:05:30.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:05:30 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:05:30.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:05:30 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:05:30.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:05:30 smithi047 ceph-mon[77147]: pgmap v5083: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:05:30.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:05:30 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:05:30.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:05:30 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:05:30.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:05:30 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:05:31.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:05:30 smithi115 conmon[90393]: 2022-05-14T02:05:30.774+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:05:31.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:05:31 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 9460 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:05:31.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:05:31 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:05:31.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:05:31 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 9460 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:05:31.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:05:31 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:05:32.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:05:31 smithi115 conmon[90393]: 2022-05-14T02:05:31.810+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:05:32.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:05:32 smithi115 ceph-mon[78593]: pgmap v5084: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:05:32.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:05:32 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:05:32.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:05:32 smithi047 ceph-mon[77147]: pgmap v5084: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:05:32.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:05:32 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:05:33.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:05:32 smithi115 conmon[90393]: 2022-05-14T02:05:32.850+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:05:34.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:05:34 smithi047 ceph-mon[77147]: pgmap v5085: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:05:34.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:05:34 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:05:34.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:05:34 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:05:34.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:05:34 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:05:34.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:05:33 smithi115 conmon[90393]: 2022-05-14T02:05:33.875+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:05:34.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:05:34 smithi115 ceph-mon[78593]: pgmap v5085: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:05:34.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:05:34 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:05:34.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:05:34 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:05:34.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:05:34 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:05:35.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:05:35 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:05:35.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:05:35 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:05:35.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:05:34 smithi115 conmon[90393]: 2022-05-14T02:05:34.890+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:05:36.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:05:36 smithi047 ceph-mon[77147]: pgmap v5086: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:05:36.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:05:36 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:05:36.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:05:36 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 9465 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:05:36.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:05:35 smithi115 conmon[90393]: 2022-05-14T02:05:35.933+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:05:36.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:05:36 smithi115 ceph-mon[78593]: pgmap v5086: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:05:36.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:05:36 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:05:36.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:05:36 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 9465 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:05:37.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:05:37 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:05:37.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:05:36 smithi115 conmon[90393]: 2022-05-14T02:05:36.928+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:05:37.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:05:37 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:05:38.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:05:38 smithi047 ceph-mon[77147]: pgmap v5087: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-05-14T02:05:38.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:05:38 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:05:38.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:05:38 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:05:38.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:05:38 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:05:38.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:05:37 smithi115 conmon[90393]: 2022-05-14T02:05:37.918+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:05:38.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:05:38 smithi115 ceph-mon[78593]: pgmap v5087: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-05-14T02:05:38.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:05:38 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:05:38.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:05:38 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:05:38.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:05:38 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:05:39.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:05:38 smithi115 conmon[90393]: 2022-05-14T02:05:38.876+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:05:39.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:05:39 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:05:39.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:05:39 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:05:40.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:05:39 smithi115 conmon[90393]: 2022-05-14T02:05:39.891+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:05:40.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:05:40 smithi115 ceph-mon[78593]: pgmap v5088: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:05:40.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:05:40 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:05:40.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:05:40 smithi047 ceph-mon[77147]: pgmap v5088: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:05:40.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:05:40 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:05:41.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:05:40 smithi115 conmon[90393]: 2022-05-14T02:05:40.902+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:05:41.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:05:41 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:05:41.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:05:41 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 9470 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:05:41.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:05:41 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:05:41.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:05:41 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 9470 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:05:42.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:05:41 smithi115 conmon[90393]: 2022-05-14T02:05:41.915+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:05:42.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:05:42 smithi115 ceph-mon[78593]: pgmap v5089: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:05:42.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:05:42 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:05:42.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:05:42 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:05:42.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:05:42 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:05:42.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:05:42 smithi047 ceph-mon[77147]: pgmap v5089: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:05:42.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:05:42 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:05:42.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:05:42 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:05:42.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:05:42 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:05:43.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:05:42 smithi115 conmon[90393]: 2022-05-14T02:05:42.961+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:05:43.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:05:43 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:05:43.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:05:43 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:05:44.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:05:44 smithi115 conmon[90393]: 2022-05-14T02:05:43.999+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:05:44.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:05:44 smithi115 ceph-mon[78593]: pgmap v5090: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:05:44.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:05:44 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:05:44.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:05:44 smithi047 ceph-mon[77147]: pgmap v5090: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:05:44.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:05:44 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:05:45.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:05:45 smithi115 conmon[90393]: 2022-05-14T02:05:45.016+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:05:45.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:05:45 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:05:45.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:05:45 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:05:46.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:05:46 smithi115 conmon[90393]: 2022-05-14T02:05:46.001+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:05:46.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:05:46 smithi115 ceph-mon[78593]: pgmap v5091: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:05:46.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:05:46 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:05:46.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:05:46 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 9475 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:05:46.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:05:46 smithi047 ceph-mon[77147]: pgmap v5091: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:05:46.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:05:46 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:05:46.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:05:46 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 9475 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:05:47.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:05:47 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:05:47.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:05:47 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T02:05:47.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:05:47 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T02:05:47.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:05:47 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T02:05:47.342 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:05:47 smithi115 conmon[90393]: 2022-05-14T02:05:47.010+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:05:47.342 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:05:47 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:05:47.343 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:05:47 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T02:05:47.343 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:05:47 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T02:05:47.343 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:05:47 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T02:05:48.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:05:47 smithi115 conmon[90393]: 2022-05-14T02:05:47.992+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:05:48.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:05:48 smithi115 ceph-mon[78593]: pgmap v5092: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-05-14T02:05:48.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:05:48 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:05:48.553 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:05:48 smithi047 ceph-mon[77147]: pgmap v5092: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-05-14T02:05:48.554 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:05:48 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:05:49.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:05:49 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:05:49.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:05:48 smithi115 conmon[90393]: 2022-05-14T02:05:48.993+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:05:49.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:05:49 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:05:50.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:05:50 smithi047 ceph-mon[77147]: pgmap v5093: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:05:50.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:05:50 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:05:50.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:05:50 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:05:50.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:05:50 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:05:50.293 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:05:49 smithi047 conmon[78521]: 2022-05-14T02:05:49.935+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:05:50.293 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:05:49 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T02:05:50.293 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:05:49 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:05:50.293 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:05:49 smithi047 conmon[78521]: yield 2022-05-14T02:05:50.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:05:49 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:05:50.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:05:49 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:05:50.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:05:49 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:05:50.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:05:49 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T02:05:50.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:05:49 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:05:50.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:05:49 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T02:05:50.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:05:49 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:05:50.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:05:49 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T02:05:50.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:05:49 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:05:50.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:05:49 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T02:05:50.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:05:49 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:05:50.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:05:49 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T02:05:50.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:05:49 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:05:50.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:05:49 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:05:50.297 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:05:49 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:05:50.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:05:49 smithi115 conmon[90393]: 2022-05-14T02:05:49.947+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:05:50.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:05:50 smithi115 ceph-mon[78593]: pgmap v5093: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:05:50.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:05:50 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:05:50.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:05:50 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:05:50.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:05:50 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:05:51.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:05:50 smithi115 conmon[90393]: 2022-05-14T02:05:50.960+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:05:51.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:05:51 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:05:51.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T02:05:51.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:05:51.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T02:05:51.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:05:51.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:05:51.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:05:51.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T02:05:51.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:05:51.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T02:05:51.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:05:51.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T02:05:51.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:05:51.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T02:05:51.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:05:51.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T02:05:51.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:05:51.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:05:51.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:05:51.364 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:05:51 smithi115 ceph-mon[78593]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T02:05:51.364 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:05:51 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:05:51.364 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:05:51 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 9480 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:05:51.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:05:51 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:05:51.542 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T02:05:51.542 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:05:51.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T02:05:51.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:05:51.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:05:51.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:05:51.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T02:05:51.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:05:51.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T02:05:51.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:05:51.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T02:05:51.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:05:51.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T02:05:51.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:05:51.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T02:05:51.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:05:51.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:05:51.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:05:51.547 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:05:51 smithi047 ceph-mon[77147]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T02:05:51.547 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:05:51 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:05:51.547 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:05:51 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 9480 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:05:52.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:05:51 smithi115 conmon[90393]: 2022-05-14T02:05:51.952+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:05:52.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:05:52 smithi115 ceph-mon[78593]: pgmap v5094: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:05:52.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:05:52 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:05:52.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:05:52 smithi047 ceph-mon[77147]: pgmap v5094: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:05:52.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:05:52 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:05:53.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:05:53 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:05:53.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:05:53 smithi047 ceph-mon[77147]: pgmap v5095: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:05:53.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:05:53 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:05:53.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T02:05:53.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:05:53.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T02:05:53.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:05:53.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:05:53.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:05:53.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T02:05:53.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:05:53.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T02:05:53.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:05:53.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T02:05:53.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:05:53.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T02:05:53.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:05:53.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T02:05:53.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:05:53.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:05:53.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:05:53.297 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:05:53 smithi047 ceph-mon[77147]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T02:05:53.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T02:05:53.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:05:53.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T02:05:53.298 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:05:53.298 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:05:53.298 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:05:53.298 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T02:05:53.299 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:05:53.299 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T02:05:53.299 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:05:53.299 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T02:05:53.300 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:05:53.300 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T02:05:53.300 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:05:53.300 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T02:05:53.300 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:05:53.301 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:05:53.301 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:05:53.301 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T02:05:53.301 INFO:journalctl@ceph.mon.b.smithi047.stdout: During handling of the above exception, another exception occurred: 2022-05-14T02:05:53.302 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T02:05:53.302 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T02:05:53.302 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T02:05:53.302 INFO:journalctl@ceph.mon.b.smithi047.stdout: return f(*arg) 2022-05-14T02:05:53.302 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T02:05:53.303 INFO:journalctl@ceph.mon.b.smithi047.stdout: self._write_client_files(client_files, host) 2022-05-14T02:05:53.303 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T02:05:53.303 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T02:05:53.303 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T02:05:53.303 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T02:05:53.304 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T02:05:53.304 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.event_loop.get_result(coro) 2022-05-14T02:05:53.304 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T02:05:53.304 INFO:journalctl@ceph.mon.b.smithi047.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T02:05:53.304 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T02:05:53.305 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.__get_result() 2022-05-14T02:05:53.305 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T02:05:53.305 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise self._exception 2022-05-14T02:05:53.305 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T02:05:53.306 INFO:journalctl@ceph.mon.b.smithi047.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T02:05:53.306 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T02:05:53.306 INFO:journalctl@ceph.mon.b.smithi047.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T02:05:53.306 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T02:05:53.306 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise 2022-05-14T02:05:53.307 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T02:05:53.307 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.gen.throw(type, value, traceback) 2022-05-14T02:05:53.307 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T02:05:53.307 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OrchestratorError(msg) 2022-05-14T02:05:53.307 INFO:journalctl@ceph.mon.b.smithi047.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:05:53.308 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:05:53 smithi047 conmon[78521]: 2022-05-14T02:05:53.007+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:05:53.308 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:05:53 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T02:05:53.308 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:05:53 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:05:53.309 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:05:53 smithi047 conmon[78521]: yield 2022-05-14T02:05:53.309 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:05:53 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:05:53.309 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:05:53 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:05:53.309 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:05:53 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:05:53.310 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:05:53 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T02:05:53.310 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:05:53 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:05:53.310 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:05:53 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T02:05:53.310 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:05:53 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:05:53.311 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:05:53 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T02:05:53.311 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:05:53 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:05:53.311 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:05:53 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T02:05:53.311 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:05:53 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:05:53.311 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:05:53 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T02:05:53.312 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:05:53 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:05:53.312 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:05:53 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:05:53.312 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:05:53 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:05:53.312 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:05:53 smithi047 conmon[78521]: 2022-05-14T02:05:53.008+0000 7f5345f84700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T02:05:53.312 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:05:53 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T02:05:53.313 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:05:53 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:05:53.313 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:05:53 smithi047 conmon[78521]: yield 2022-05-14T02:05:53.313 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:05:53 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:05:53.313 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:05:53 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:05:53.314 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:05:53 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:05:53.314 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:05:53 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T02:05:53.314 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:05:53 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:05:53.314 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:05:53 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T02:05:53.315 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:05:53 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:05:53.315 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:05:53 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T02:05:53.315 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:05:53 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:05:53.315 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:05:53 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T02:05:53.315 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:05:53 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:05:53.316 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:05:53 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T02:05:53.316 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:05:53 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:05:53.316 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:05:53 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:05:53.316 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:05:53 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:05:53.316 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:05:53 smithi047 conmon[78521]: 2022-05-14T02:05:53.317 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:05:53 smithi047 conmon[78521]: During handling of the above exception, another exception occurred: 2022-05-14T02:05:53.317 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:05:53 smithi047 conmon[78521]: 2022-05-14T02:05:53.317 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:05:53 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T02:05:53.317 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:05:53 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T02:05:53.318 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:05:53 smithi047 conmon[78521]: return f(*arg) 2022-05-14T02:05:53.318 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:05:53 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T02:05:53.318 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:05:53 smithi047 conmon[78521]: self._write_client_files(client_files, host) 2022-05-14T02:05:53.318 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:05:53 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T02:05:53.318 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:05:53 smithi047 conmon[78521]: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T02:05:53.319 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:05:53 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T02:05:53.319 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:05:53 smithi047 conmon[78521]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T02:05:53.319 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:05:53 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T02:05:53.319 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:05:53 smithi047 conmon[78521]: return self.event_loop.get_result(coro) 2022-05-14T02:05:53.319 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:05:53 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T02:05:53.320 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:05:53 smithi047 conmon[78521]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T02:05:53.320 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:05:53 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T02:05:53.320 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:05:53 smithi047 conmon[78521]: return self.__get_result() 2022-05-14T02:05:53.320 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:05:53 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T02:05:53.320 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:05:53 smithi047 conmon[78521]: raise self._exception 2022-05-14T02:05:53.321 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:05:53 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T02:05:53.321 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:05:53 smithi047 conmon[78521]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T02:05:53.321 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:05:53 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T02:05:53.321 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:05:53 smithi047 conmon[78521]: conn = await self._remote_connection(host, addr) 2022-05-14T02:05:53.322 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:05:53 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T02:05:53.322 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:05:53 smithi047 conmon[78521]: raise 2022-05-14T02:05:53.322 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:05:53 smithi047 conmon[78521]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T02:05:53.322 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:05:53 smithi047 conmon[78521]: self.gen.throw(type, value, traceback) 2022-05-14T02:05:53.322 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:05:53 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T02:05:53.323 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:05:53 smithi047 conmon[78521]: raise OrchestratorError(msg) 2022-05-14T02:05:53.323 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:05:53 smithi047 conmon[78521]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:05:53.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:05:52 smithi115 conmon[90393]: 2022-05-14T02:05:52.959+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:05:53.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:05:53 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:05:53.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:05:53 smithi115 ceph-mon[78593]: pgmap v5095: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:05:53.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:05:53 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:05:53.359 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T02:05:53.359 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:05:53.359 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T02:05:53.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:05:53.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:05:53.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:05:53.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T02:05:53.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:05:53.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T02:05:53.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:05:53.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T02:05:53.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:05:53.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T02:05:53.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:05:53.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T02:05:53.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:05:53.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:05:53.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:05:53.363 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:05:53 smithi115 ceph-mon[78593]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T02:05:53.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T02:05:53.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:05:53.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T02:05:53.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:05:53.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:05:53.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:05:53.365 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T02:05:53.365 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:05:53.365 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T02:05:53.365 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:05:53.365 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T02:05:53.366 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:05:53.366 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T02:05:53.366 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:05:53.366 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T02:05:53.366 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:05:53.367 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:05:53.367 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:05:53.367 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T02:05:53.367 INFO:journalctl@ceph.mon.c.smithi115.stdout: During handling of the above exception, another exception occurred: 2022-05-14T02:05:53.368 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T02:05:53.368 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T02:05:53.368 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T02:05:53.368 INFO:journalctl@ceph.mon.c.smithi115.stdout: return f(*arg) 2022-05-14T02:05:53.368 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T02:05:53.369 INFO:journalctl@ceph.mon.c.smithi115.stdout: self._write_client_files(client_files, host) 2022-05-14T02:05:53.369 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T02:05:53.369 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T02:05:53.369 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T02:05:53.370 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T02:05:53.370 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T02:05:53.370 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.event_loop.get_result(coro) 2022-05-14T02:05:53.370 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T02:05:53.370 INFO:journalctl@ceph.mon.c.smithi115.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T02:05:53.371 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T02:05:53.371 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.__get_result() 2022-05-14T02:05:53.371 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T02:05:53.371 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise self._exception 2022-05-14T02:05:53.371 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T02:05:53.372 INFO:journalctl@ceph.mon.c.smithi115.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T02:05:53.372 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T02:05:53.372 INFO:journalctl@ceph.mon.c.smithi115.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T02:05:53.372 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T02:05:53.372 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise 2022-05-14T02:05:53.373 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T02:05:53.373 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.gen.throw(type, value, traceback) 2022-05-14T02:05:53.373 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T02:05:53.373 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OrchestratorError(msg) 2022-05-14T02:05:53.374 INFO:journalctl@ceph.mon.c.smithi115.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:05:54.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:05:53 smithi115 conmon[90393]: 2022-05-14T02:05:53.942+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:05:54.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:05:54 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:05:54.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:05:54 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:05:54.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:05:54 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:05:54.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:05:54 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:05:54.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:05:54 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:05:54.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:05:54 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:05:55.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:05:54 smithi115 conmon[90393]: 2022-05-14T02:05:54.913+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:05:55.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:05:55 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:05:55.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:05:55 smithi115 ceph-mon[78593]: pgmap v5096: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:05:55.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:05:55 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:05:55.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:05:55 smithi047 ceph-mon[77147]: pgmap v5096: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:05:56.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:05:55 smithi115 conmon[90393]: 2022-05-14T02:05:55.960+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:05:56.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:05:56 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:05:56.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:05:56 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 9485 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:05:56.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:05:56 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:05:56.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:05:56 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 9485 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:05:57.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:05:56 smithi115 conmon[90393]: 2022-05-14T02:05:56.944+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:05:57.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:05:57 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:05:57.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:05:57 smithi115 ceph-mon[78593]: pgmap v5097: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-05-14T02:05:57.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:05:57 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:05:57.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:05:57 smithi047 ceph-mon[77147]: pgmap v5097: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-05-14T02:05:58.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:05:57 smithi115 conmon[90393]: 2022-05-14T02:05:57.943+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:05:58.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:05:58 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:05:58.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:05:58 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:05:58.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:05:58 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:05:58.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:05:58 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:05:58.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:05:58 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:05:58.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:05:58 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:05:59.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:05:58 smithi115 conmon[90393]: 2022-05-14T02:05:58.898+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:05:59.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:05:59 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:05:59.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:05:59 smithi115 ceph-mon[78593]: pgmap v5098: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:05:59.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:05:59 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:05:59.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:05:59 smithi047 ceph-mon[77147]: pgmap v5098: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:06:00.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:05:59 smithi115 conmon[90393]: 2022-05-14T02:05:59.933+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:06:00.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:06:00 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:06:00.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:06:00 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:06:01.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:06:00 smithi115 conmon[90393]: 2022-05-14T02:06:00.957+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:06:01.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:06:01 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:06:01.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:06:01 smithi115 ceph-mon[78593]: pgmap v5099: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:06:01.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:06:01 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 9490 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:06:01.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:06:01 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:06:01.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:06:01 smithi047 ceph-mon[77147]: pgmap v5099: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:06:01.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:06:01 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 9490 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:06:02.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:06:01 smithi115 conmon[90393]: 2022-05-14T02:06:01.913+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:06:02.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:06:02 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:06:02.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:06:02 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:06:02.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:06:02 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:06:02.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:06:02 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:06:02.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:06:02 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:06:02.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:06:02 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:06:03.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:06:02 smithi115 conmon[90393]: 2022-05-14T02:06:02.951+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:06:03.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:06:03 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:06:03.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:06:03 smithi047 ceph-mon[77147]: pgmap v5100: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:06:03.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:06:03 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:06:03.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:06:03 smithi115 ceph-mon[78593]: pgmap v5100: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:06:04.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:06:03 smithi115 conmon[90393]: 2022-05-14T02:06:03.988+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:06:04.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:06:04 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:06:04.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:06:04 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:06:05.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:06:04 smithi115 conmon[90393]: 2022-05-14T02:06:04.974+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:06:05.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:06:05 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:06:05.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:06:05 smithi047 ceph-mon[77147]: pgmap v5101: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:06:05.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:06:05 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 9495 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:06:05.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:06:05 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:06:05.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:06:05 smithi115 ceph-mon[78593]: pgmap v5101: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:06:05.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:06:05 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 9495 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:06:06.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:06:05 smithi115 conmon[90393]: 2022-05-14T02:06:05.967+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:06:06.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:06:06 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:06:06.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:06:06 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:06:07.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:06:06 smithi115 conmon[90393]: 2022-05-14T02:06:06.938+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:06:07.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:06:07 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:06:07.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:06:07 smithi047 ceph-mon[77147]: pgmap v5102: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-05-14T02:06:07.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:06:07 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:06:07.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:06:07 smithi115 ceph-mon[78593]: pgmap v5102: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-05-14T02:06:08.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:06:07 smithi115 conmon[90393]: 2022-05-14T02:06:07.914+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:06:08.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:06:08 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:06:08.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:06:08 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:06:09.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:06:08 smithi115 conmon[90393]: 2022-05-14T02:06:08.946+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:06:09.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:06:09 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:06:09.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:06:09 smithi047 ceph-mon[77147]: pgmap v5103: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:06:09.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:06:09 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:06:09.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:06:09 smithi115 ceph-mon[78593]: pgmap v5103: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:06:10.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:06:09 smithi115 conmon[90393]: 2022-05-14T02:06:09.912+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:06:10.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:06:10 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:06:10.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:06:10 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:06:10.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:06:10 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:06:10.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:06:10 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 9500 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:06:10.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:06:10 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:06:10.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:06:10 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:06:10.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:06:10 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:06:10.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:06:10 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 9500 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:06:11.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:06:10 smithi115 conmon[90393]: 2022-05-14T02:06:10.936+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:06:11.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:06:11 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:06:11.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:06:11 smithi047 ceph-mon[77147]: pgmap v5104: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:06:11.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:06:11 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:06:11.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:06:11 smithi115 ceph-mon[78593]: pgmap v5104: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:06:12.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:06:11 smithi115 conmon[90393]: 2022-05-14T02:06:11.926+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:06:12.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:06:12 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:06:12.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:06:12 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:06:13.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:06:12 smithi115 conmon[90393]: 2022-05-14T02:06:12.923+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:06:13.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:06:13 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:06:13.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:06:13 smithi047 ceph-mon[77147]: pgmap v5105: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:06:13.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:06:13 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:06:13.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:06:13 smithi115 ceph-mon[78593]: pgmap v5105: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:06:14.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:06:13 smithi115 conmon[90393]: 2022-05-14T02:06:13.917+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:06:14.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:06:14 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:06:14.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:06:14 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:06:14.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:06:14 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:06:14.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:06:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T02:06:14.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:06:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T02:06:14.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:06:14 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:06:14.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:06:14 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:06:14.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:06:14 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:06:14.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:06:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T02:06:14.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:06:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T02:06:15.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:06:14 smithi115 conmon[90393]: 2022-05-14T02:06:14.868+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:06:15.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:06:15 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:06:15.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:06:15 smithi047 ceph-mon[77147]: pgmap v5106: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:06:15.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:06:15 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 9505 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:06:15.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:06:15 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:06:15.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:06:15 smithi115 ceph-mon[78593]: pgmap v5106: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:06:15.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:06:15 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 9505 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:06:16.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:06:15 smithi115 conmon[90393]: 2022-05-14T02:06:15.875+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:06:16.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:06:16 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:06:16.853 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:06:16 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:06:17.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:06:16 smithi115 conmon[90393]: 2022-05-14T02:06:16.852+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:06:17.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:06:17 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:06:17.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:06:17 smithi047 ceph-mon[77147]: pgmap v5107: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-05-14T02:06:17.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:06:17 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:06:17.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:06:17 smithi115 ceph-mon[78593]: pgmap v5107: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-05-14T02:06:18.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:06:17 smithi115 conmon[90393]: 2022-05-14T02:06:17.858+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:06:18.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:06:18 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:06:18.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:06:18 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:06:18.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:06:18 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:06:18.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:06:18 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:06:18.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:06:18 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:06:18.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:06:18 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:06:19.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:06:18 smithi115 conmon[90393]: 2022-05-14T02:06:18.866+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:06:19.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:06:19 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:06:19.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:06:19 smithi047 ceph-mon[77147]: pgmap v5108: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:06:19.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:06:19 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:06:19.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:06:19 smithi115 ceph-mon[78593]: pgmap v5108: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:06:20.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:06:19 smithi115 conmon[90393]: 2022-05-14T02:06:19.883+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:06:20.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:06:20 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:06:20.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:06:20 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 9510 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:06:20.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:06:20 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:06:20.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:06:20 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 9510 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:06:21.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:06:20 smithi115 conmon[90393]: 2022-05-14T02:06:20.881+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:06:21.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:06:21 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:06:21.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:06:21 smithi047 ceph-mon[77147]: pgmap v5109: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:06:21.840 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:06:21 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:06:21.840 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:06:21 smithi115 ceph-mon[78593]: pgmap v5109: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:06:22.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:06:21 smithi115 conmon[90393]: 2022-05-14T02:06:21.838+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:06:22.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:06:22 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:06:22.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:06:22 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:06:22.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:06:22 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:06:22.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:06:22 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:06:22.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:06:22 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:06:22.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:06:22 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:06:23.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:06:22 smithi115 conmon[90393]: 2022-05-14T02:06:22.885+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:06:23.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:06:23 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:06:23.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:06:23 smithi047 ceph-mon[77147]: pgmap v5110: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:06:23.844 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:06:23 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:06:23.844 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:06:23 smithi115 ceph-mon[78593]: pgmap v5110: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:06:24.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:06:23 smithi115 conmon[90393]: 2022-05-14T02:06:23.842+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:06:24.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:06:24 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:06:24.852 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:06:24 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:06:25.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:06:24 smithi115 conmon[90393]: 2022-05-14T02:06:24.850+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:06:25.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:06:25 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:06:25.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:06:25 smithi047 ceph-mon[77147]: pgmap v5111: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:06:25.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:06:25 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 9515 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:06:25.812 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:06:25 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:06:25.813 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:06:25 smithi115 ceph-mon[78593]: pgmap v5111: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:06:25.813 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:06:25 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 9515 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:06:26.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:06:25 smithi115 conmon[90393]: 2022-05-14T02:06:25.811+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:06:26.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:06:26 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:06:26.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:06:26 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:06:26.843 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:06:26 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:06:26.844 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:06:26 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:06:27.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:06:26 smithi115 conmon[90393]: 2022-05-14T02:06:26.842+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:06:27.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:06:27 smithi047 ceph-mon[77147]: pgmap v5112: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-05-14T02:06:27.837 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:06:27 smithi115 ceph-mon[78593]: pgmap v5112: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-05-14T02:06:28.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:06:27 smithi115 conmon[90393]: 2022-05-14T02:06:27.835+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:06:28.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:06:28 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:06:28.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:06:28 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:06:29.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:06:28 smithi115 conmon[90393]: 2022-05-14T02:06:28.866+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:06:29.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:06:29 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:06:29.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:06:29 smithi047 ceph-mon[77147]: pgmap v5113: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:06:29.843 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:06:29 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:06:29.844 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:06:29 smithi115 ceph-mon[78593]: pgmap v5113: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:06:30.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:06:29 smithi115 conmon[90393]: 2022-05-14T02:06:29.841+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:06:30.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:06:30 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:06:30.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:06:30 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:06:30.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:06:30 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:06:30.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:06:30 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 9520 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:06:30.855 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:06:30 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:06:30.856 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:06:30 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:06:30.856 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:06:30 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:06:30.856 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:06:30 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 9520 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:06:31.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:06:30 smithi115 conmon[90393]: 2022-05-14T02:06:30.853+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:06:31.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:06:31 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:06:31.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:06:31 smithi047 ceph-mon[77147]: pgmap v5114: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:06:31.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:06:31 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:06:31.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:06:31 smithi115 ceph-mon[78593]: pgmap v5114: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:06:32.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:06:31 smithi115 conmon[90393]: 2022-05-14T02:06:31.884+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:06:32.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:06:32 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:06:32.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:06:32 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:06:33.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:06:32 smithi115 conmon[90393]: 2022-05-14T02:06:32.910+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:06:33.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:06:33 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:06:33.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:06:33 smithi047 ceph-mon[77147]: pgmap v5115: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:06:33.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:06:33 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:06:33.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:06:33 smithi115 ceph-mon[78593]: pgmap v5115: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:06:34.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:06:33 smithi115 conmon[90393]: 2022-05-14T02:06:33.924+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:06:34.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:06:34 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:06:34.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:06:34 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:06:34.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:06:34 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:06:34.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:06:34 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:06:34.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:06:34 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:06:34.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:06:34 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:06:35.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:06:34 smithi115 conmon[90393]: 2022-05-14T02:06:34.955+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:06:35.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:06:35 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:06:35.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:06:35 smithi047 ceph-mon[77147]: pgmap v5116: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:06:35.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:06:35 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 9530 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:06:35.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:06:35 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:06:35.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:06:35 smithi115 ceph-mon[78593]: pgmap v5116: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:06:35.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:06:35 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 9530 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:06:36.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:06:35 smithi115 conmon[90393]: 2022-05-14T02:06:35.968+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:06:36.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:06:36 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:06:36.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:06:36 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:06:37.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:06:36 smithi115 conmon[90393]: 2022-05-14T02:06:36.923+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:06:37.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:06:37 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:06:37.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:06:37 smithi047 ceph-mon[77147]: pgmap v5117: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-05-14T02:06:37.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:06:37 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:06:37.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:06:37 smithi115 ceph-mon[78593]: pgmap v5117: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-05-14T02:06:38.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:06:37 smithi115 conmon[90393]: 2022-05-14T02:06:37.906+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:06:38.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:06:38 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:06:38.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:06:38 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:06:38.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:06:38 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:06:38.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:06:38 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:06:38.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:06:38 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:06:38.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:06:38 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:06:39.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:06:38 smithi115 conmon[90393]: 2022-05-14T02:06:38.881+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:06:39.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:06:39 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:06:39.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:06:39 smithi047 ceph-mon[77147]: pgmap v5118: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:06:39.849 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:06:39 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:06:39.850 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:06:39 smithi115 ceph-mon[78593]: pgmap v5118: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:06:40.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:06:39 smithi115 conmon[90393]: 2022-05-14T02:06:39.848+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:06:40.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:06:40 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:06:40.806 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:06:40 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:06:41.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:06:40 smithi115 conmon[90393]: 2022-05-14T02:06:40.805+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:06:41.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:06:41 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:06:41.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:06:41 smithi047 ceph-mon[77147]: pgmap v5119: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:06:41.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:06:41 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 9535 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:06:41.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:06:41 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:06:41.804 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:06:41 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:06:41.805 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:06:41 smithi115 ceph-mon[78593]: pgmap v5119: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:06:41.806 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:06:41 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 9535 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:06:41.806 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:06:41 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:06:42.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:06:41 smithi115 conmon[90393]: 2022-05-14T02:06:41.802+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:06:42.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:06:42 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:06:42.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:06:42 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:06:42.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:06:42 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:06:42.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:06:42 smithi115 conmon[90393]: 2022-05-14T02:06:42.757+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:06:42.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:06:42 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:06:42.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:06:42 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:06:42.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:06:42 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:06:43.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:06:43 smithi047 ceph-mon[77147]: pgmap v5120: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:06:43.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:06:43 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:06:43.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:06:43 smithi115 conmon[90393]: 2022-05-14T02:06:43.752+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:06:43.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:06:43 smithi115 ceph-mon[78593]: pgmap v5120: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:06:43.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:06:43 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:06:44.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:06:44 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:06:44.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:06:44 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:06:44.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:06:44 smithi115 conmon[90393]: 2022-05-14T02:06:44.724+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:06:45.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:06:45 smithi047 ceph-mon[77147]: pgmap v5121: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:06:45.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:06:45 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:06:45.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:06:45 smithi115 ceph-mon[78593]: pgmap v5121: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:06:45.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:06:45 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:06:45.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:06:45 smithi115 conmon[90393]: 2022-05-14T02:06:45.771+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:06:46.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:06:46 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:06:46.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:06:46 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:06:46.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:06:46 smithi115 conmon[90393]: 2022-05-14T02:06:46.776+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:06:47.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:06:47 smithi047 ceph-mon[77147]: pgmap v5122: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:06:47.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:06:47 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:06:47.820 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:06:47 smithi115 ceph-mon[78593]: pgmap v5122: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:06:47.820 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:06:47 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:06:48.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:06:47 smithi115 conmon[90393]: 2022-05-14T02:06:47.818+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:06:48.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:06:48 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:06:49.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:06:48 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:06:49.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:06:48 smithi115 conmon[90393]: 2022-05-14T02:06:48.860+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:06:49.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:06:49 smithi115 ceph-mon[78593]: pgmap v5123: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:06:50.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:06:49 smithi047 ceph-mon[77147]: pgmap v5123: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:06:50.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:06:49 smithi115 conmon[90393]: 2022-05-14T02:06:49.863+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:06:50.856 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:06:50 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:06:50.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:06:50 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:06:50.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:06:50 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:06:50.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:06:50 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 9540 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:06:51.047 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:06:50 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:06:51.048 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:06:50 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:06:51.048 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:06:50 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:06:51.048 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:06:50 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 9540 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:06:51.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:06:50 smithi115 conmon[90393]: 2022-05-14T02:06:50.855+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:06:51.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:06:51 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:06:51.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:06:51 smithi115 ceph-mon[78593]: pgmap v5124: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:06:52.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:06:51 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:06:52.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:06:51 smithi047 ceph-mon[77147]: pgmap v5124: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:06:52.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:06:51 smithi115 conmon[90393]: 2022-05-14T02:06:51.899+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:06:52.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:06:52 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:06:53.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:06:52 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:06:53.362 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:06:52 smithi115 conmon[90393]: 2022-05-14T02:06:52.903+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:06:53.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:06:53 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:06:53.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:06:53 smithi047 ceph-mon[77147]: pgmap v5125: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:06:53.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:06:53 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T02:06:53.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:06:53 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T02:06:53.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:06:53 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T02:06:53.793 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:06:53 smithi047 conmon[78521]: 2022-05-14T02:06:53.615+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:06:53.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:06:53 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T02:06:53.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:06:53 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:06:53.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:06:53 smithi047 conmon[78521]: yield 2022-05-14T02:06:53.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:06:53 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:06:53.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:06:53 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:06:53.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:06:53 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:06:53.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:06:53 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T02:06:53.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:06:53 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:06:53.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:06:53 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T02:06:53.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:06:53 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:06:53.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:06:53 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T02:06:53.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:06:53 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:06:53.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:06:53 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T02:06:53.797 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:06:53 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:06:53.797 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:06:53 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T02:06:53.797 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:06:53 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:06:53.797 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:06:53 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:06:53.797 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:06:53 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:06:53.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:06:53 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:06:53.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:06:53 smithi115 ceph-mon[78593]: pgmap v5125: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:06:53.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:06:53 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T02:06:53.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:06:53 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T02:06:53.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:06:53 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T02:06:54.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:06:53 smithi115 conmon[90393]: 2022-05-14T02:06:53.871+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:06:54.856 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:06:54 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:06:54.856 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:06:54 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:06:54.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:06:54 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:06:54.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:06:54 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:06:54.857 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T02:06:54.858 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:06:54.858 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T02:06:54.858 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:06:54.858 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:06:54.859 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:06:54.859 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T02:06:54.859 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:06:54.859 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T02:06:54.859 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:06:54.860 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T02:06:54.860 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:06:54.860 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T02:06:54.860 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:06:54.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T02:06:54.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:06:54.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:06:54.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:06:54.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:06:54 smithi115 ceph-mon[78593]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T02:06:55.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:06:54 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:06:55.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:06:54 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:06:55.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:06:54 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:06:55.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:06:54 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:06:55.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T02:06:55.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:06:55.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T02:06:55.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:06:55.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:06:55.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:06:55.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T02:06:55.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:06:55.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T02:06:55.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:06:55.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T02:06:55.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:06:55.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T02:06:55.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:06:55.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T02:06:55.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:06:55.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:06:55.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:06:55.048 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:06:54 smithi047 ceph-mon[77147]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T02:06:55.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:06:54 smithi115 conmon[90393]: 2022-05-14T02:06:54.854+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:06:55.838 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:06:55 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:06:55.838 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:06:55 smithi115 ceph-mon[78593]: pgmap v5126: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:06:55.839 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:06:55 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 9550 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:06:56.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:06:55 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:06:56.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:06:55 smithi047 ceph-mon[77147]: pgmap v5126: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:06:56.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:06:55 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 9550 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:06:56.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:06:55 smithi115 conmon[90393]: 2022-05-14T02:06:55.837+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:06:56.838 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:06:56 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:06:57.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:06:56 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:06:57.042 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:06:56 smithi047 conmon[78521]: 2022-05-14T02:06:56.687+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:06:57.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:06:56 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T02:06:57.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:06:56 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:06:57.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:06:56 smithi047 conmon[78521]: yield 2022-05-14T02:06:57.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:06:56 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:06:57.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:06:56 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:06:57.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:06:56 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:06:57.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:06:56 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T02:06:57.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:06:56 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:06:57.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:06:56 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T02:06:57.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:06:56 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:06:57.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:06:56 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T02:06:57.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:06:56 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:06:57.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:06:56 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T02:06:57.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:06:56 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:06:57.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:06:56 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T02:06:57.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:06:56 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:06:57.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:06:56 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:06:57.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:06:56 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:06:57.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:06:56 smithi047 conmon[78521]: 2022-05-14T02:06:56.689+0000 7f5340779700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T02:06:57.048 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:06:56 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T02:06:57.048 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:06:56 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:06:57.048 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:06:56 smithi047 conmon[78521]: yield 2022-05-14T02:06:57.048 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:06:56 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:06:57.049 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:06:56 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:06:57.049 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:06:56 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:06:57.049 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:06:56 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T02:06:57.049 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:06:56 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:06:57.050 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:06:56 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T02:06:57.050 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:06:56 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:06:57.051 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:06:56 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T02:06:57.051 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:06:56 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:06:57.051 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:06:56 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T02:06:57.052 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:06:56 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:06:57.052 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:06:56 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T02:06:57.053 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:06:56 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:06:57.053 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:06:56 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:06:57.054 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:06:56 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:06:57.054 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:06:56 smithi047 conmon[78521]: 2022-05-14T02:06:57.054 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:06:56 smithi047 conmon[78521]: During handling of the above exception, another exception occurred: 2022-05-14T02:06:57.055 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:06:56 smithi047 conmon[78521]: 2022-05-14T02:06:57.055 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:06:56 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T02:06:57.055 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:06:56 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T02:06:57.055 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:06:56 smithi047 conmon[78521]: return f(*arg) 2022-05-14T02:06:57.056 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:06:56 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T02:06:57.056 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:06:56 smithi047 conmon[78521]: self._write_client_files(client_files, host) 2022-05-14T02:06:57.056 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:06:56 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T02:06:57.056 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:06:56 smithi047 conmon[78521]: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T02:06:57.057 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:06:56 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T02:06:57.057 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:06:56 smithi047 conmon[78521]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T02:06:57.057 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:06:56 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T02:06:57.057 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:06:56 smithi047 conmon[78521]: return self.event_loop.get_result(coro) 2022-05-14T02:06:57.057 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:06:56 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T02:06:57.058 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:06:56 smithi047 conmon[78521]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T02:06:57.058 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:06:56 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T02:06:57.058 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:06:56 smithi047 conmon[78521]: return self.__get_result() 2022-05-14T02:06:57.058 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:06:56 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T02:06:57.059 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:06:56 smithi047 conmon[78521]: raise self._exception 2022-05-14T02:06:57.059 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:06:56 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T02:06:57.059 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:06:56 smithi047 conmon[78521]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T02:06:57.059 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:06:56 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T02:06:57.059 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:06:56 smithi047 conmon[78521]: conn = await self._remote_connection(host, addr) 2022-05-14T02:06:57.060 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:06:56 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T02:06:57.060 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:06:56 smithi047 conmon[78521]: raise 2022-05-14T02:06:57.060 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:06:56 smithi047 conmon[78521]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T02:06:57.060 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:06:56 smithi047 conmon[78521]: self.gen.throw(type, value, traceback) 2022-05-14T02:06:57.061 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:06:56 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T02:06:57.061 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:06:56 smithi047 conmon[78521]: raise OrchestratorError(msg) 2022-05-14T02:06:57.061 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:06:56 smithi047 conmon[78521]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:06:57.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:06:56 smithi115 conmon[90393]: 2022-05-14T02:06:56.836+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:06:57.834 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:06:57 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:06:57.834 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:06:57 smithi115 ceph-mon[78593]: pgmap v5127: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:06:57.835 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:06:57 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:06:57.835 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T02:06:57.835 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:06:57.835 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T02:06:57.836 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:06:57.836 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:06:57.836 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:06:57.836 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T02:06:57.836 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:06:57.837 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T02:06:57.837 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:06:57.837 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T02:06:57.837 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:06:57.838 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T02:06:57.838 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:06:57.839 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T02:06:57.839 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:06:57.839 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:06:57.839 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:06:57.840 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:06:57 smithi115 ceph-mon[78593]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T02:06:57.840 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T02:06:57.840 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:06:57.840 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T02:06:57.841 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:06:57.841 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:06:57.841 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:06:57.841 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T02:06:57.841 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:06:57.842 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T02:06:57.842 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:06:57.842 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T02:06:57.842 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:06:57.843 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T02:06:57.843 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:06:57.843 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T02:06:57.843 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:06:57.843 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:06:57.844 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:06:57.844 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T02:06:57.844 INFO:journalctl@ceph.mon.c.smithi115.stdout: During handling of the above exception, another exception occurred: 2022-05-14T02:06:57.844 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T02:06:57.844 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T02:06:57.845 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T02:06:57.845 INFO:journalctl@ceph.mon.c.smithi115.stdout: return f(*arg) 2022-05-14T02:06:57.845 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T02:06:57.845 INFO:journalctl@ceph.mon.c.smithi115.stdout: self._write_client_files(client_files, host) 2022-05-14T02:06:57.846 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T02:06:57.846 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T02:06:57.846 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T02:06:57.846 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T02:06:57.846 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T02:06:57.847 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.event_loop.get_result(coro) 2022-05-14T02:06:57.847 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T02:06:57.847 INFO:journalctl@ceph.mon.c.smithi115.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T02:06:57.847 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T02:06:57.848 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.__get_result() 2022-05-14T02:06:57.848 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T02:06:57.848 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise self._exception 2022-05-14T02:06:57.848 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T02:06:57.848 INFO:journalctl@ceph.mon.c.smithi115.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T02:06:57.849 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T02:06:57.849 INFO:journalctl@ceph.mon.c.smithi115.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T02:06:57.850 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T02:06:57.850 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise 2022-05-14T02:06:57.850 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T02:06:57.850 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.gen.throw(type, value, traceback) 2022-05-14T02:06:57.851 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T02:06:57.851 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OrchestratorError(msg) 2022-05-14T02:06:57.851 INFO:journalctl@ceph.mon.c.smithi115.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:06:58.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:06:57 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:06:58.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:06:57 smithi047 ceph-mon[77147]: pgmap v5127: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:06:58.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:06:57 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:06:58.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T02:06:58.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:06:58.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T02:06:58.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:06:58.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:06:58.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:06:58.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T02:06:58.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:06:58.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T02:06:58.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:06:58.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T02:06:58.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:06:58.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T02:06:58.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:06:58.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T02:06:58.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:06:58.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:06:58.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:06:58.048 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:06:57 smithi047 ceph-mon[77147]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T02:06:58.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T02:06:58.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:06:58.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T02:06:58.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:06:58.049 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:06:58.049 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:06:58.049 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T02:06:58.050 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:06:58.050 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T02:06:58.051 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:06:58.051 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T02:06:58.051 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:06:58.052 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T02:06:58.052 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:06:58.052 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T02:06:58.052 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:06:58.053 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:06:58.053 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:06:58.053 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T02:06:58.053 INFO:journalctl@ceph.mon.b.smithi047.stdout: During handling of the above exception, another exception occurred: 2022-05-14T02:06:58.053 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T02:06:58.054 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T02:06:58.054 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T02:06:58.054 INFO:journalctl@ceph.mon.b.smithi047.stdout: return f(*arg) 2022-05-14T02:06:58.054 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T02:06:58.055 INFO:journalctl@ceph.mon.b.smithi047.stdout: self._write_client_files(client_files, host) 2022-05-14T02:06:58.055 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T02:06:58.055 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T02:06:58.055 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T02:06:58.056 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T02:06:58.056 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T02:06:58.056 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.event_loop.get_result(coro) 2022-05-14T02:06:58.056 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T02:06:58.057 INFO:journalctl@ceph.mon.b.smithi047.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T02:06:58.057 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T02:06:58.057 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.__get_result() 2022-05-14T02:06:58.057 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T02:06:58.057 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise self._exception 2022-05-14T02:06:58.058 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T02:06:58.058 INFO:journalctl@ceph.mon.b.smithi047.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T02:06:58.058 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T02:06:58.058 INFO:journalctl@ceph.mon.b.smithi047.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T02:06:58.058 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T02:06:58.059 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise 2022-05-14T02:06:58.059 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T02:06:58.059 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.gen.throw(type, value, traceback) 2022-05-14T02:06:58.059 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T02:06:58.059 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OrchestratorError(msg) 2022-05-14T02:06:58.060 INFO:journalctl@ceph.mon.b.smithi047.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:06:58.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:06:57 smithi115 conmon[90393]: 2022-05-14T02:06:57.833+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:06:58.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:06:58 smithi115 conmon[90393]: 2022-05-14T02:06:58.784+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:06:58.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:06:58 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:06:58.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:06:58 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:06:58.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:06:58 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:06:58.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:06:58 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:06:59.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:06:58 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:06:59.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:06:58 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:06:59.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:06:58 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:06:59.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:06:58 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:06:59.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:06:59 smithi115 conmon[90393]: 2022-05-14T02:06:59.737+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:06:59.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:06:59 smithi115 ceph-mon[78593]: pgmap v5128: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:06:59.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:06:59 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:07:00.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:06:59 smithi047 ceph-mon[77147]: pgmap v5128: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:07:00.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:06:59 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:07:00.864 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:07:00 smithi115 conmon[90393]: 2022-05-14T02:07:00.780+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:07:00.864 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:07:00 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:07:01.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:07:00 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:07:01.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:07:01 smithi115 conmon[90393]: 2022-05-14T02:07:01.807+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:07:01.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:07:01 smithi115 ceph-mon[78593]: pgmap v5129: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:07:01.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:07:01 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 9555 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:07:01.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:07:01 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:07:02.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:07:01 smithi047 ceph-mon[77147]: pgmap v5129: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:07:02.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:07:01 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 9555 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:07:02.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:07:01 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:07:02.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:07:02 smithi115 conmon[90393]: 2022-05-14T02:07:02.840+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:07:02.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:07:02 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:07:02.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:07:02 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:07:02.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:07:02 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:07:03.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:07:02 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:07:03.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:07:02 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:07:03.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:07:02 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:07:03.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:07:03 smithi115 ceph-mon[78593]: pgmap v5130: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:07:04.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:07:03 smithi047 ceph-mon[77147]: pgmap v5130: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:07:04.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:07:03 smithi115 conmon[90393]: 2022-05-14T02:07:03.884+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:07:05.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:07:04 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:07:05.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:07:04 smithi115 conmon[90393]: 2022-05-14T02:07:04.853+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:07:05.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:07:04 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:07:06.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:07:05 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:07:06.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:07:05 smithi047 ceph-mon[77147]: pgmap v5131: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:07:06.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:07:05 smithi115 conmon[90393]: 2022-05-14T02:07:05.850+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:07:06.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:07:05 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:07:06.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:07:05 smithi115 ceph-mon[78593]: pgmap v5131: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:07:06.875 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:07:06 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:07:07.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:07:06 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:07:07.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:07:06 smithi115 conmon[90393]: 2022-05-14T02:07:06.873+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:07:08.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:07:07 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:07:08.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:07:07 smithi047 ceph-mon[77147]: pgmap v5132: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:07:08.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:07:07 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:07:08.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:07:07 smithi115 ceph-mon[78593]: pgmap v5132: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:07:08.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:07:07 smithi115 conmon[90393]: 2022-05-14T02:07:07.846+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:07:09.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:07:08 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:07:09.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:07:08 smithi115 conmon[90393]: 2022-05-14T02:07:08.814+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:07:09.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:07:08 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:07:10.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:07:09 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:07:10.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:07:09 smithi047 ceph-mon[77147]: pgmap v5133: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:07:10.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:07:09 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:07:10.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:07:09 smithi115 conmon[90393]: 2022-05-14T02:07:09.809+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:07:10.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:07:09 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:07:10.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:07:09 smithi115 ceph-mon[78593]: pgmap v5133: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:07:10.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:07:09 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:07:11.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:07:10 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:07:11.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:07:10 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:07:11.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:07:10 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:07:11.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:07:10 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 9560 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:07:11.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:07:10 smithi115 conmon[90393]: 2022-05-14T02:07:10.788+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:07:11.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:07:10 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:07:11.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:07:10 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:07:11.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:07:10 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:07:11.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:07:10 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 9560 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:07:12.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:07:11 smithi047 ceph-mon[77147]: pgmap v5134: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:07:12.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:07:11 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:07:12.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:07:11 smithi115 conmon[90393]: 2022-05-14T02:07:11.823+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:07:12.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:07:11 smithi115 ceph-mon[78593]: pgmap v5134: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:07:12.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:07:11 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:07:13.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:07:12 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:07:13.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:07:12 smithi115 conmon[90393]: 2022-05-14T02:07:12.838+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:07:13.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:07:12 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:07:14.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:07:13 smithi047 ceph-mon[77147]: pgmap v5135: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:07:14.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:07:13 smithi115 conmon[90393]: 2022-05-14T02:07:13.875+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:07:14.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:07:13 smithi115 ceph-mon[78593]: pgmap v5135: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:07:15.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:07:14 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:07:15.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:07:14 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:07:15.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:07:14 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:07:15.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:07:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T02:07:15.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:07:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T02:07:15.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:07:14 smithi115 conmon[90393]: 2022-05-14T02:07:14.830+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:07:15.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:07:14 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:07:15.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:07:14 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:07:15.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:07:14 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:07:15.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:07:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T02:07:15.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:07:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T02:07:16.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:07:15 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:07:16.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:07:15 smithi047 ceph-mon[77147]: pgmap v5136: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:07:16.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:07:15 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:07:16.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:07:15 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 9570 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:07:16.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:07:15 smithi115 conmon[90393]: 2022-05-14T02:07:15.849+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:07:16.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:07:15 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:07:16.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:07:15 smithi115 ceph-mon[78593]: pgmap v5136: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:07:16.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:07:15 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:07:16.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:07:15 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 9570 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:07:17.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:07:16 smithi115 conmon[90393]: 2022-05-14T02:07:16.893+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:07:17.944 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:07:17 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:07:17.945 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:07:17 smithi115 ceph-mon[78593]: pgmap v5137: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:07:18.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:07:17 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:07:18.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:07:17 smithi047 ceph-mon[77147]: pgmap v5137: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:07:18.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:07:17 smithi115 conmon[90393]: 2022-05-14T02:07:17.943+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:07:18.953 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:07:18 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:07:18.954 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:07:18 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:07:18.954 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:07:18 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:07:19.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:07:18 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:07:19.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:07:18 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:07:19.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:07:18 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:07:19.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:07:18 smithi115 conmon[90393]: 2022-05-14T02:07:18.952+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:07:19.974 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:07:19 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:07:19.975 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:07:19 smithi115 ceph-mon[78593]: pgmap v5138: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:07:20.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:07:19 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:07:20.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:07:19 smithi047 ceph-mon[77147]: pgmap v5138: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:07:20.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:07:19 smithi115 conmon[90393]: 2022-05-14T02:07:19.973+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:07:20.931 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:07:20 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:07:21.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:07:20 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:07:21.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:07:20 smithi115 conmon[90393]: 2022-05-14T02:07:20.929+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:07:21.945 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:07:21 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:07:21.946 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:07:21 smithi115 ceph-mon[78593]: pgmap v5139: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:07:21.946 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:07:21 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 9575 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:07:22.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:07:21 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:07:22.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:07:21 smithi047 ceph-mon[77147]: pgmap v5139: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:07:22.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:07:21 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 9575 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:07:22.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:07:21 smithi115 conmon[90393]: 2022-05-14T02:07:21.943+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:07:22.962 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:07:22 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:07:22.963 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:07:22 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:07:22.963 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:07:22 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:07:23.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:07:22 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:07:23.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:07:22 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:07:23.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:07:22 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:07:23.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:07:22 smithi115 conmon[90393]: 2022-05-14T02:07:22.961+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:07:23.993 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:07:23 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:07:23.994 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:07:23 smithi115 ceph-mon[78593]: pgmap v5140: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:07:24.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:07:23 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:07:24.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:07:23 smithi047 ceph-mon[77147]: pgmap v5140: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:07:24.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:07:23 smithi115 conmon[90393]: 2022-05-14T02:07:23.991+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:07:25.024 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:07:24 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:07:25.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:07:24 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:07:25.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:07:25 smithi115 conmon[90393]: 2022-05-14T02:07:25.022+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:07:26.019 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:07:25 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:07:26.019 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:07:25 smithi115 ceph-mon[78593]: pgmap v5141: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:07:26.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:07:25 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:07:26.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:07:25 smithi047 ceph-mon[77147]: pgmap v5141: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:07:26.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:07:26 smithi115 conmon[90393]: 2022-05-14T02:07:26.017+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:07:26.998 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:07:26 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:07:27.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:07:26 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:07:27.366 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:07:26 smithi115 conmon[90393]: 2022-05-14T02:07:26.996+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:07:27.971 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:07:27 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:07:27.972 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:07:27 smithi115 ceph-mon[78593]: pgmap v5142: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:07:28.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:07:27 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:07:28.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:07:27 smithi047 ceph-mon[77147]: pgmap v5142: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:07:28.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:07:27 smithi115 conmon[90393]: 2022-05-14T02:07:27.970+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:07:29.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:07:28 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:07:29.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:07:28 smithi115 conmon[90393]: 2022-05-14T02:07:28.929+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:07:29.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:07:28 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:07:30.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:07:29 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:07:30.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:07:29 smithi047 ceph-mon[77147]: pgmap v5143: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:07:30.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:07:29 smithi115 conmon[90393]: 2022-05-14T02:07:29.943+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:07:30.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:07:29 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:07:30.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:07:29 smithi115 ceph-mon[78593]: pgmap v5143: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:07:31.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:07:30 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:07:31.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:07:30 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:07:31.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:07:30 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:07:31.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:07:30 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 9580 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:07:31.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:07:30 smithi115 conmon[90393]: 2022-05-14T02:07:30.958+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:07:31.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:07:30 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:07:31.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:07:30 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:07:31.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:07:30 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:07:31.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:07:30 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 9580 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:07:32.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:07:31 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:07:32.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:07:31 smithi047 ceph-mon[77147]: pgmap v5144: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:07:32.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:07:31 smithi115 conmon[90393]: 2022-05-14T02:07:31.965+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:07:32.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:07:31 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:07:32.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:07:31 smithi115 ceph-mon[78593]: pgmap v5144: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:07:33.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:07:32 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:07:33.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:07:32 smithi115 conmon[90393]: 2022-05-14T02:07:32.952+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:07:33.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:07:32 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:07:34.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:07:33 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:07:34.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:07:33 smithi047 ceph-mon[77147]: pgmap v5145: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:07:34.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:07:33 smithi115 conmon[90393]: 2022-05-14T02:07:33.947+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:07:34.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:07:33 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:07:34.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:07:33 smithi115 ceph-mon[78593]: pgmap v5145: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:07:34.988 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:07:34 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:07:34.989 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:07:34 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:07:34.989 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:07:34 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:07:35.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:07:34 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:07:35.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:07:34 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:07:35.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:07:34 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:07:35.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:07:34 smithi115 conmon[90393]: 2022-05-14T02:07:34.987+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:07:36.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:07:35 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:07:36.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:07:35 smithi047 ceph-mon[77147]: pgmap v5146: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:07:36.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:07:35 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 9590 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:07:36.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:07:35 smithi115 conmon[90393]: 2022-05-14T02:07:35.947+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:07:36.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:07:35 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:07:36.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:07:35 smithi115 ceph-mon[78593]: pgmap v5146: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:07:36.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:07:35 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 9590 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:07:37.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:07:36 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:07:37.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:07:36 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:07:37.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:07:36 smithi115 conmon[90393]: 2022-05-14T02:07:36.959+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:07:38.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:07:37 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:07:38.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:07:37 smithi047 ceph-mon[77147]: pgmap v5147: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:07:38.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:07:37 smithi115 conmon[90393]: 2022-05-14T02:07:37.957+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:07:38.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:07:37 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:07:38.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:07:37 smithi115 ceph-mon[78593]: pgmap v5147: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:07:39.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:07:38 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:07:39.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:07:38 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:07:39.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:07:38 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:07:39.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:07:38 smithi115 conmon[90393]: 2022-05-14T02:07:38.930+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:07:39.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:07:38 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:07:39.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:07:38 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:07:39.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:07:38 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:07:40.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:07:39 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:07:40.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:07:39 smithi047 ceph-mon[77147]: pgmap v5148: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:07:40.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:07:39 smithi115 conmon[90393]: 2022-05-14T02:07:39.912+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:07:40.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:07:39 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:07:40.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:07:39 smithi115 ceph-mon[78593]: pgmap v5148: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:07:41.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:07:40 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:07:41.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:07:40 smithi115 conmon[90393]: 2022-05-14T02:07:40.937+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:07:41.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:07:40 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:07:42.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:07:41 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:07:42.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:07:41 smithi047 ceph-mon[77147]: pgmap v5149: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:07:42.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:07:41 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 9595 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:07:42.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:07:41 smithi115 conmon[90393]: 2022-05-14T02:07:41.982+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:07:42.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:07:41 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:07:42.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:07:41 smithi115 ceph-mon[78593]: pgmap v5149: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:07:42.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:07:41 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 9595 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:07:43.030 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:07:42 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:07:43.030 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:07:42 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:07:43.030 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:07:42 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:07:43.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:07:42 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:07:43.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:07:42 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:07:43.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:07:42 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:07:43.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:07:43 smithi115 conmon[90393]: 2022-05-14T02:07:43.028+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:07:44.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:07:43 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:07:44.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:07:43 smithi047 ceph-mon[77147]: pgmap v5150: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:07:44.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:07:44 smithi115 conmon[90393]: 2022-05-14T02:07:44.013+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:07:44.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:07:43 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:07:44.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:07:43 smithi115 ceph-mon[78593]: pgmap v5150: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:07:45.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:07:44 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:07:45.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:07:44 smithi115 conmon[90393]: 2022-05-14T02:07:44.979+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:07:45.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:07:44 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:07:46.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:07:45 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:07:46.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:07:45 smithi047 ceph-mon[77147]: pgmap v5151: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:07:46.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:07:45 smithi115 conmon[90393]: 2022-05-14T02:07:45.955+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:07:46.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:07:45 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:07:46.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:07:45 smithi115 ceph-mon[78593]: pgmap v5151: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:07:47.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:07:46 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:07:47.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:07:46 smithi115 conmon[90393]: 2022-05-14T02:07:46.996+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:07:47.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:07:46 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:07:48.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:07:47 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:07:48.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:07:47 smithi047 ceph-mon[77147]: pgmap v5152: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:07:48.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:07:47 smithi115 conmon[90393]: 2022-05-14T02:07:47.947+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:07:48.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:07:47 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:07:48.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:07:47 smithi115 ceph-mon[78593]: pgmap v5152: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:07:49.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:07:48 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:07:49.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:07:48 smithi115 conmon[90393]: 2022-05-14T02:07:48.998+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:07:49.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:07:48 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:07:50.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:07:49 smithi115 conmon[90393]: 2022-05-14T02:07:49.972+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:07:50.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:07:49 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:07:50.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:07:49 smithi115 ceph-mon[78593]: pgmap v5153: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:07:50.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:07:49 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:07:50.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:07:49 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:07:50.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:07:49 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:07:50.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:07:49 smithi047 ceph-mon[77147]: pgmap v5153: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:07:50.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:07:49 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:07:50.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:07:49 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:07:51.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:07:51 smithi115 conmon[90393]: 2022-05-14T02:07:51.008+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:07:51.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:07:50 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:07:51.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:07:50 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 9600 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:07:51.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:07:50 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:07:51.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:07:50 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 9600 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:07:52.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:07:52 smithi115 conmon[90393]: 2022-05-14T02:07:52.051+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:07:52.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:07:51 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:07:52.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:07:51 smithi115 ceph-mon[78593]: pgmap v5154: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:07:52.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:07:51 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:07:52.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:07:51 smithi047 ceph-mon[77147]: pgmap v5154: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:07:53.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:07:53 smithi115 conmon[90393]: 2022-05-14T02:07:53.058+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:07:53.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:07:52 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:07:53.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:07:52 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:07:54.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:07:54 smithi115 conmon[90393]: 2022-05-14T02:07:54.021+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:07:54.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:07:53 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:07:54.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:07:53 smithi115 ceph-mon[78593]: pgmap v5155: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:07:54.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:07:53 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:07:54.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:07:53 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:07:54.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:07:53 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:07:54.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:07:53 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:07:54.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:07:53 smithi047 ceph-mon[77147]: pgmap v5155: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:07:54.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:07:53 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:07:54.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:07:53 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:07:54.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:07:53 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:07:55.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:07:55 smithi115 conmon[90393]: 2022-05-14T02:07:55.015+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:07:55.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:07:54 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:07:55.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:07:54 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:07:56.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:07:56 smithi115 conmon[90393]: 2022-05-14T02:07:56.004+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:07:56.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:07:55 smithi115 ceph-mon[78593]: pgmap v5156: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:07:56.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:07:55 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:07:56.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:07:55 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 9610 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:07:56.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:07:55 smithi047 ceph-mon[77147]: pgmap v5156: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:07:56.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:07:55 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:07:56.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:07:55 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 9610 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:07:57.022 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:07:56 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:07:57.023 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:07:56 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T02:07:57.023 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:07:56 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T02:07:57.023 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:07:56 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T02:07:57.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:07:56 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:07:57.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:07:56 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T02:07:57.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:07:56 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T02:07:57.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:07:56 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T02:07:57.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:07:57 smithi115 conmon[90393]: 2022-05-14T02:07:57.021+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:07:58.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:07:57 smithi047 ceph-mon[77147]: pgmap v5157: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:07:58.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:07:57 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:07:58.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:07:57 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:07:58.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:07:57 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:07:58.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:07:58 smithi115 conmon[90393]: 2022-05-14T02:07:58.063+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:07:58.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:07:57 smithi115 ceph-mon[78593]: pgmap v5157: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:07:58.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:07:57 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:07:58.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:07:57 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:07:58.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:07:57 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:07:59.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:07:58 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:07:59.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:07:58 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:07:59.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:07:59 smithi115 conmon[90393]: 2022-05-14T02:07:59.070+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:08:00.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:07:59 smithi047 ceph-mon[77147]: pgmap v5158: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:08:00.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:07:59 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:08:00.292 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:07:59 smithi047 conmon[78521]: 2022-05-14T02:07:59.792+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:08:00.292 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:07:59 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T02:08:00.293 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:07:59 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:08:00.293 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:07:59 smithi047 conmon[78521]: yield 2022-05-14T02:08:00.293 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:07:59 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:08:00.293 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:07:59 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:08:00.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:07:59 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:08:00.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:07:59 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T02:08:00.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:07:59 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:08:00.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:07:59 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T02:08:00.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:07:59 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:08:00.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:07:59 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T02:08:00.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:07:59 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:08:00.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:07:59 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T02:08:00.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:07:59 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:08:00.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:07:59 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T02:08:00.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:07:59 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:08:00.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:07:59 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:08:00.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:07:59 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:08:00.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:08:00 smithi115 conmon[90393]: 2022-05-14T02:08:00.089+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:08:00.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:07:59 smithi115 ceph-mon[78593]: pgmap v5158: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:08:00.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:07:59 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:08:01.121 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:08:00 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:08:01.121 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T02:08:01.121 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:08:01.122 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T02:08:01.122 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:08:01.122 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:08:01.122 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:08:01.122 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T02:08:01.123 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:08:01.123 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T02:08:01.123 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:08:01.123 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T02:08:01.124 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:08:01.124 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T02:08:01.124 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:08:01.124 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T02:08:01.124 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:08:01.125 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:08:01.125 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:08:01.125 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:08:00 smithi115 ceph-mon[78593]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T02:08:01.125 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:08:00 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:08:01.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:08:00 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:08:01.292 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T02:08:01.292 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:08:01.292 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T02:08:01.292 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:08:01.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:08:01.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:08:01.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T02:08:01.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:08:01.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T02:08:01.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:08:01.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T02:08:01.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:08:01.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T02:08:01.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:08:01.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T02:08:01.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:08:01.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:08:01.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:08:01.296 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:08:00 smithi047 ceph-mon[77147]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T02:08:01.296 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:08:00 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:08:01.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:08:01 smithi115 conmon[90393]: 2022-05-14T02:08:01.119+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:08:02.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:08:01 smithi047 ceph-mon[77147]: pgmap v5159: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:08:02.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:08:01 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 9615 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:08:02.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:08:01 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:08:02.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:08:01 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:08:02.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:08:01 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:08:02.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:08:02 smithi115 conmon[90393]: 2022-05-14T02:08:02.096+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:08:02.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:08:01 smithi115 ceph-mon[78593]: pgmap v5159: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:08:02.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:08:01 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 9615 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:08:02.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:08:01 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:08:02.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:08:01 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:08:02.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:08:01 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:08:03.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:08:02 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:08:03.292 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:08:02 smithi047 conmon[78521]: 2022-05-14T02:08:02.864+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:08:03.292 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:08:02 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T02:08:03.293 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:08:02 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:08:03.293 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:08:02 smithi047 conmon[78521]: yield 2022-05-14T02:08:03.293 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:08:02 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:08:03.293 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:08:02 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:08:03.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:08:02 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:08:03.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:08:02 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T02:08:03.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:08:02 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:08:03.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:08:02 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T02:08:03.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:08:02 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:08:03.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:08:02 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T02:08:03.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:08:02 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:08:03.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:08:02 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T02:08:03.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:08:02 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:08:03.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:08:02 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T02:08:03.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:08:02 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:08:03.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:08:02 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:08:03.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:08:02 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:08:03.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:08:02 smithi047 conmon[78521]: 2022-05-14T02:08:02.865+0000 7f5344f82700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T02:08:03.297 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:08:02 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T02:08:03.297 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:08:02 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:08:03.297 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:08:02 smithi047 conmon[78521]: yield 2022-05-14T02:08:03.297 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:08:02 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:08:03.298 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:08:02 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:08:03.298 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:08:02 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:08:03.298 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:08:02 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T02:08:03.298 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:08:02 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:08:03.298 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:08:02 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T02:08:03.299 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:08:02 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:08:03.299 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:08:02 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T02:08:03.299 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:08:02 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:08:03.299 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:08:02 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T02:08:03.300 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:08:02 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:08:03.300 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:08:02 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T02:08:03.300 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:08:02 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:08:03.300 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:08:02 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:08:03.300 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:08:02 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:08:03.301 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:08:02 smithi047 conmon[78521]: 2022-05-14T02:08:03.301 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:08:02 smithi047 conmon[78521]: During handling of the above exception, another exception occurred: 2022-05-14T02:08:03.301 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:08:02 smithi047 conmon[78521]: 2022-05-14T02:08:03.301 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:08:02 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T02:08:03.302 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:08:02 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T02:08:03.302 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:08:02 smithi047 conmon[78521]: return f(*arg) 2022-05-14T02:08:03.302 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:08:02 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T02:08:03.302 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:08:02 smithi047 conmon[78521]: self._write_client_files(client_files, host) 2022-05-14T02:08:03.303 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:08:02 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T02:08:03.303 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:08:02 smithi047 conmon[78521]: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T02:08:03.303 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:08:02 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T02:08:03.303 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:08:02 smithi047 conmon[78521]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T02:08:03.303 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:08:02 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T02:08:03.304 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:08:02 smithi047 conmon[78521]: return self.event_loop.get_result(coro) 2022-05-14T02:08:03.304 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:08:02 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T02:08:03.304 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:08:02 smithi047 conmon[78521]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T02:08:03.304 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:08:02 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T02:08:03.305 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:08:02 smithi047 conmon[78521]: return self.__get_result() 2022-05-14T02:08:03.305 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:08:02 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T02:08:03.305 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:08:02 smithi047 conmon[78521]: raise self._exception 2022-05-14T02:08:03.305 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:08:02 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T02:08:03.306 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:08:02 smithi047 conmon[78521]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T02:08:03.306 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:08:02 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T02:08:03.306 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:08:02 smithi047 conmon[78521]: conn = await self._remote_connection(host, addr) 2022-05-14T02:08:03.306 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:08:02 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T02:08:03.306 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:08:02 smithi047 conmon[78521]: raise 2022-05-14T02:08:03.307 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:08:02 smithi047 conmon[78521]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T02:08:03.307 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:08:02 smithi047 conmon[78521]: self.gen.throw(type, value, traceback) 2022-05-14T02:08:03.307 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:08:02 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T02:08:03.307 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:08:02 smithi047 conmon[78521]: raise OrchestratorError(msg) 2022-05-14T02:08:03.308 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:08:02 smithi047 conmon[78521]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:08:03.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:08:03 smithi115 conmon[90393]: 2022-05-14T02:08:03.106+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:08:03.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:08:02 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:08:04.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:08:03 smithi047 ceph-mon[77147]: pgmap v5160: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:08:04.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:08:03 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:08:04.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T02:08:04.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:08:04.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T02:08:04.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:08:04.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:08:04.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:08:04.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T02:08:04.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:08:04.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T02:08:04.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:08:04.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T02:08:04.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:08:04.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T02:08:04.298 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:08:04.298 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T02:08:04.299 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:08:04.299 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:08:04.299 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:08:04.300 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:08:03 smithi047 ceph-mon[77147]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T02:08:04.300 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T02:08:04.301 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:08:04.301 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T02:08:04.302 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:08:04.302 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:08:04.302 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:08:04.303 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T02:08:04.303 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:08:04.304 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T02:08:04.304 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:08:04.304 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T02:08:04.305 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:08:04.305 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T02:08:04.306 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:08:04.306 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T02:08:04.307 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:08:04.307 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:08:04.307 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:08:04.308 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T02:08:04.308 INFO:journalctl@ceph.mon.b.smithi047.stdout: During handling of the above exception, another exception occurred: 2022-05-14T02:08:04.308 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T02:08:04.309 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T02:08:04.309 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T02:08:04.310 INFO:journalctl@ceph.mon.b.smithi047.stdout: return f(*arg) 2022-05-14T02:08:04.310 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T02:08:04.311 INFO:journalctl@ceph.mon.b.smithi047.stdout: self._write_client_files(client_files, host) 2022-05-14T02:08:04.311 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T02:08:04.311 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T02:08:04.312 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T02:08:04.312 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T02:08:04.312 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T02:08:04.313 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.event_loop.get_result(coro) 2022-05-14T02:08:04.313 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T02:08:04.313 INFO:journalctl@ceph.mon.b.smithi047.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T02:08:04.314 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T02:08:04.314 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.__get_result() 2022-05-14T02:08:04.314 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T02:08:04.315 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise self._exception 2022-05-14T02:08:04.315 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T02:08:04.315 INFO:journalctl@ceph.mon.b.smithi047.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T02:08:04.315 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T02:08:04.316 INFO:journalctl@ceph.mon.b.smithi047.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T02:08:04.316 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T02:08:04.317 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise 2022-05-14T02:08:04.317 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T02:08:04.317 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.gen.throw(type, value, traceback) 2022-05-14T02:08:04.318 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T02:08:04.318 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OrchestratorError(msg) 2022-05-14T02:08:04.318 INFO:journalctl@ceph.mon.b.smithi047.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:08:04.319 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:08:03 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:08:04.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:08:04 smithi115 conmon[90393]: 2022-05-14T02:08:04.079+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:08:04.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:08:03 smithi115 ceph-mon[78593]: pgmap v5160: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:08:04.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:08:03 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:08:04.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T02:08:04.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:08:04.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T02:08:04.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:08:04.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:08:04.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:08:04.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T02:08:04.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:08:04.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T02:08:04.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:08:04.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T02:08:04.365 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:08:04.365 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T02:08:04.366 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:08:04.366 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T02:08:04.366 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:08:04.367 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:08:04.367 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:08:04.368 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:08:03 smithi115 ceph-mon[78593]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T02:08:04.368 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T02:08:04.368 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:08:04.369 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T02:08:04.369 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:08:04.369 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:08:04.369 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:08:04.370 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T02:08:04.370 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:08:04.370 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T02:08:04.371 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:08:04.371 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T02:08:04.372 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:08:04.372 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T02:08:04.372 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:08:04.373 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T02:08:04.373 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:08:04.373 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:08:04.373 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:08:04.374 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T02:08:04.374 INFO:journalctl@ceph.mon.c.smithi115.stdout: During handling of the above exception, another exception occurred: 2022-05-14T02:08:04.374 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T02:08:04.374 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T02:08:04.375 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T02:08:04.375 INFO:journalctl@ceph.mon.c.smithi115.stdout: return f(*arg) 2022-05-14T02:08:04.375 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T02:08:04.375 INFO:journalctl@ceph.mon.c.smithi115.stdout: self._write_client_files(client_files, host) 2022-05-14T02:08:04.375 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T02:08:04.376 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T02:08:04.376 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T02:08:04.376 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T02:08:04.376 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T02:08:04.377 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.event_loop.get_result(coro) 2022-05-14T02:08:04.377 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T02:08:04.377 INFO:journalctl@ceph.mon.c.smithi115.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T02:08:04.377 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T02:08:04.377 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.__get_result() 2022-05-14T02:08:04.378 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T02:08:04.378 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise self._exception 2022-05-14T02:08:04.378 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T02:08:04.378 INFO:journalctl@ceph.mon.c.smithi115.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T02:08:04.379 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T02:08:04.379 INFO:journalctl@ceph.mon.c.smithi115.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T02:08:04.379 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T02:08:04.379 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise 2022-05-14T02:08:04.380 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T02:08:04.380 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.gen.throw(type, value, traceback) 2022-05-14T02:08:04.380 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T02:08:04.380 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OrchestratorError(msg) 2022-05-14T02:08:04.381 INFO:journalctl@ceph.mon.c.smithi115.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:08:04.381 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:08:03 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:08:05.296 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:08:04 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:08:05.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:08:05 smithi115 conmon[90393]: 2022-05-14T02:08:05.086+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:08:05.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:08:04 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:08:06.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:08:05 smithi047 ceph-mon[77147]: pgmap v5161: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:08:06.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:08:05 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:08:06.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:08:06 smithi115 conmon[90393]: 2022-05-14T02:08:06.062+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:08:06.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:08:05 smithi115 ceph-mon[78593]: pgmap v5161: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:08:06.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:08:05 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:08:07.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:08:06 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:08:07.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:08:07 smithi115 conmon[90393]: 2022-05-14T02:08:07.077+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:08:07.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:08:06 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:08:08.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:08:07 smithi047 ceph-mon[77147]: pgmap v5162: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:08:08.389 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:08:07 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:08:08.390 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:08:08 smithi115 conmon[90393]: 2022-05-14T02:08:08.099+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:08:08.390 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:08:07 smithi115 ceph-mon[78593]: pgmap v5162: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:08:08.390 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:08:07 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:08:09.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:08:08 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:08:09.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:08:09 smithi115 conmon[90393]: 2022-05-14T02:08:09.072+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:08:09.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:08:08 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:08:10.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:08:09 smithi047 ceph-mon[77147]: pgmap v5163: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:08:10.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:08:09 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:08:10.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:08:09 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:08:10.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:08:09 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:08:10.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:08:10 smithi115 conmon[90393]: 2022-05-14T02:08:10.043+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:08:10.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:08:09 smithi115 ceph-mon[78593]: pgmap v5163: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:08:10.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:08:09 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:08:10.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:08:09 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:08:10.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:08:09 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:08:11.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:08:10 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:08:11.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:08:10 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 9620 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:08:11.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:08:11 smithi115 conmon[90393]: 2022-05-14T02:08:11.006+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:08:11.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:08:10 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:08:11.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:08:10 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 9620 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:08:12.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:08:11 smithi047 ceph-mon[77147]: pgmap v5164: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:08:12.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:08:11 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:08:12.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:08:12 smithi115 conmon[90393]: 2022-05-14T02:08:12.054+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:08:12.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:08:11 smithi115 ceph-mon[78593]: pgmap v5164: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:08:12.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:08:11 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:08:13.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:08:12 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:08:13.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:08:12 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:08:13.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:08:13 smithi115 conmon[90393]: 2022-05-14T02:08:13.047+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:08:14.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:08:13 smithi047 ceph-mon[77147]: pgmap v5165: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:08:14.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:08:13 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:08:14.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:08:13 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:08:14.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:08:13 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:08:14.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:08:13 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T02:08:14.295 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:08:13 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T02:08:14.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:08:14 smithi115 conmon[90393]: 2022-05-14T02:08:14.039+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:08:14.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:08:13 smithi115 ceph-mon[78593]: pgmap v5165: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:08:14.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:08:13 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:08:14.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:08:13 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:08:14.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:08:13 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:08:14.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:08:13 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T02:08:14.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:08:13 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T02:08:15.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:08:14 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:08:15.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:08:15 smithi115 conmon[90393]: 2022-05-14T02:08:15.049+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:08:15.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:08:14 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:08:16.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:08:15 smithi047 ceph-mon[77147]: pgmap v5166: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:08:16.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:08:15 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:08:16.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:08:15 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 9630 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:08:16.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:08:16 smithi115 conmon[90393]: 2022-05-14T02:08:16.090+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:08:16.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:08:15 smithi115 ceph-mon[78593]: pgmap v5166: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:08:16.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:08:15 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:08:16.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:08:15 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 9630 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:08:17.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:08:16 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:08:17.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:08:17 smithi115 conmon[90393]: 2022-05-14T02:08:17.095+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:08:17.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:08:16 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:08:18.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:08:17 smithi047 ceph-mon[77147]: pgmap v5167: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:08:18.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:08:17 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:08:18.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:08:17 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:08:18.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:08:17 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:08:18.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:08:18 smithi115 conmon[90393]: 2022-05-14T02:08:18.117+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:08:18.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:08:17 smithi115 ceph-mon[78593]: pgmap v5167: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:08:18.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:08:17 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:08:18.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:08:17 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:08:18.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:08:17 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:08:19.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:08:18 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:08:19.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:08:19 smithi115 conmon[90393]: 2022-05-14T02:08:19.146+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:08:19.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:08:18 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:08:20.193 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:08:19 smithi115 ceph-mon[78593]: pgmap v5168: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:08:20.193 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:08:19 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:08:20.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:08:19 smithi047 ceph-mon[77147]: pgmap v5168: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:08:20.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:08:19 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:08:20.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:08:20 smithi115 conmon[90393]: 2022-05-14T02:08:20.191+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:08:21.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:08:21 smithi115 conmon[90393]: 2022-05-14T02:08:21.189+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:08:21.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:08:21 smithi115 ceph-mon[78593]: pgmap v5169: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:08:21.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:08:21 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:08:21.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:08:21 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 9636 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:08:21.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:08:21 smithi047 ceph-mon[77147]: pgmap v5169: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:08:21.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:08:21 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:08:21.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:08:21 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 9636 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:08:22.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:08:22 smithi115 conmon[90393]: 2022-05-14T02:08:22.166+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:08:22.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:08:22 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:08:22.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:08:22 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:08:22.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:08:22 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:08:22.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:08:22 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:08:22.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:08:22 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:08:22.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:08:22 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:08:23.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:08:23 smithi115 conmon[90393]: 2022-05-14T02:08:23.203+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:08:23.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:08:23 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:08:23.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:08:23 smithi115 ceph-mon[78593]: pgmap v5170: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:08:23.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:08:23 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:08:23.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:08:23 smithi047 ceph-mon[77147]: pgmap v5170: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:08:24.610 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:08:24 smithi115 conmon[90393]: 2022-05-14T02:08:24.253+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:08:24.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:08:24 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:08:24.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:08:24 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:08:25.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:08:25 smithi115 conmon[90393]: 2022-05-14T02:08:25.277+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:08:25.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:08:25 smithi115 ceph-mon[78593]: pgmap v5171: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:08:25.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:08:25 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:08:25.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:08:25 smithi047 ceph-mon[77147]: pgmap v5171: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:08:25.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:08:25 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:08:26.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:08:26 smithi115 conmon[90393]: 2022-05-14T02:08:26.275+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:08:26.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:08:26 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:08:26.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:08:26 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:08:27.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:08:27 smithi115 conmon[90393]: 2022-05-14T02:08:27.229+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:08:27.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:08:27 smithi115 ceph-mon[78593]: pgmap v5172: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:08:27.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:08:27 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:08:27.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:08:27 smithi047 ceph-mon[77147]: pgmap v5172: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:08:27.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:08:27 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:08:28.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:08:28 smithi115 conmon[90393]: 2022-05-14T02:08:28.245+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:08:28.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:08:28 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:08:28.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:08:28 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:08:29.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:08:29 smithi115 conmon[90393]: 2022-05-14T02:08:29.253+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:08:29.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:08:29 smithi115 ceph-mon[78593]: pgmap v5173: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:08:29.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:08:29 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:08:29.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:08:29 smithi047 ceph-mon[77147]: pgmap v5173: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:08:29.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:08:29 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:08:30.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:08:30 smithi115 conmon[90393]: 2022-05-14T02:08:30.297+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:08:30.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:08:30 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:08:30.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:08:30 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:08:30.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:08:30 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:08:30.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:08:30 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 9640 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:08:30.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:08:30 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:08:30.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:08:30 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:08:30.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:08:30 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:08:30.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:08:30 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 9640 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:08:31.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:08:31 smithi115 conmon[90393]: 2022-05-14T02:08:31.276+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:08:31.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:08:31 smithi115 ceph-mon[78593]: pgmap v5174: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:08:31.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:08:31 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:08:31.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:08:31 smithi047 ceph-mon[77147]: pgmap v5174: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:08:31.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:08:31 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:08:32.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:08:32 smithi115 conmon[90393]: 2022-05-14T02:08:32.253+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:08:32.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:08:32 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:08:32.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:08:32 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:08:33.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:08:33 smithi115 conmon[90393]: 2022-05-14T02:08:33.205+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:08:33.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:08:33 smithi115 ceph-mon[78593]: pgmap v5175: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:08:33.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:08:33 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:08:33.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:08:33 smithi047 ceph-mon[77147]: pgmap v5175: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:08:33.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:08:33 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:08:34.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:08:34 smithi115 conmon[90393]: 2022-05-14T02:08:34.218+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:08:34.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:08:34 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:08:34.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:08:34 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:08:34.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:08:34 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:08:34.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:08:34 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:08:34.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:08:34 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:08:34.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:08:34 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:08:35.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:08:35 smithi115 conmon[90393]: 2022-05-14T02:08:35.229+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:08:35.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:08:35 smithi115 ceph-mon[78593]: pgmap v5176: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:08:35.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:08:35 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:08:35.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:08:35 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 9650 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:08:35.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:08:35 smithi047 ceph-mon[77147]: pgmap v5176: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:08:35.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:08:35 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:08:35.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:08:35 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 9650 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:08:36.449 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:08:36 smithi115 conmon[90393]: 2022-05-14T02:08:36.185+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:08:36.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:08:36 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:08:36.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:08:36 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:08:37.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:08:37 smithi115 conmon[90393]: 2022-05-14T02:08:37.205+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:08:37.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:08:37 smithi115 ceph-mon[78593]: pgmap v5177: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:08:37.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:08:37 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:08:37.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:08:37 smithi047 ceph-mon[77147]: pgmap v5177: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:08:37.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:08:37 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:08:38.460 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:08:38 smithi115 conmon[90393]: 2022-05-14T02:08:38.170+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:08:38.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:08:38 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:08:38.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:08:38 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:08:38.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:08:38 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:08:38.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:08:38 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:08:38.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:08:38 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:08:38.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:08:38 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:08:39.462 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:08:39 smithi115 conmon[90393]: 2022-05-14T02:08:39.181+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:08:39.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:08:39 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:08:39.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:08:39 smithi047 ceph-mon[77147]: pgmap v5178: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:08:39.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:08:39 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:08:39.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:08:39 smithi115 ceph-mon[78593]: pgmap v5178: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:08:40.466 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:08:40 smithi115 conmon[90393]: 2022-05-14T02:08:40.186+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:08:40.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:08:40 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:08:40.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:08:40 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:08:41.467 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:08:41 smithi115 conmon[90393]: 2022-05-14T02:08:41.154+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:08:41.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:08:41 smithi047 ceph-mon[77147]: pgmap v5179: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:08:41.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:08:41 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:08:41.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:08:41 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 9655 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:08:41.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:08:41 smithi115 ceph-mon[78593]: pgmap v5179: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:08:41.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:08:41 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:08:41.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:08:41 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 9655 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:08:42.479 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:08:42 smithi115 conmon[90393]: 2022-05-14T02:08:42.144+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:08:42.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:08:42 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:08:42.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:08:42 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:08:42.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:08:42 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:08:42.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:08:42 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:08:42.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:08:42 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:08:42.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:08:42 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:08:43.478 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:08:43 smithi115 conmon[90393]: 2022-05-14T02:08:43.120+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:08:43.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:08:43 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:08:43.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:08:43 smithi047 ceph-mon[77147]: pgmap v5180: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:08:43.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:08:43 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:08:43.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:08:43 smithi115 ceph-mon[78593]: pgmap v5180: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:08:44.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:08:44 smithi115 conmon[90393]: 2022-05-14T02:08:44.089+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:08:44.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:08:44 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:08:44.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:08:44 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:08:45.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:08:45 smithi115 conmon[90393]: 2022-05-14T02:08:45.084+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:08:45.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:08:45 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:08:45.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:08:45 smithi047 ceph-mon[77147]: pgmap v5181: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:08:45.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:08:45 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:08:45.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:08:45 smithi115 ceph-mon[78593]: pgmap v5181: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:08:46.493 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:08:46 smithi115 conmon[90393]: 2022-05-14T02:08:46.106+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:08:46.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:08:46 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:08:46.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:08:46 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:08:47.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:08:47 smithi115 conmon[90393]: 2022-05-14T02:08:47.075+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:08:47.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:08:47 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:08:47.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:08:47 smithi047 ceph-mon[77147]: pgmap v5182: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:08:47.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:08:47 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:08:47.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:08:47 smithi115 ceph-mon[78593]: pgmap v5182: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:08:48.498 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:08:48 smithi115 conmon[90393]: 2022-05-14T02:08:48.107+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:08:48.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:08:48 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:08:48.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:08:48 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:08:49.504 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:08:49 smithi115 conmon[90393]: 2022-05-14T02:08:49.146+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:08:49.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:08:49 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:08:49.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:08:49 smithi047 ceph-mon[77147]: pgmap v5183: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:08:49.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:08:49 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:08:49.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:08:49 smithi115 ceph-mon[78593]: pgmap v5183: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:08:50.506 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:08:50 smithi115 conmon[90393]: 2022-05-14T02:08:50.191+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:08:50.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:08:50 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:08:50.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:08:50 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:08:50.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:08:50 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:08:50.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:08:50 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 9661 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:08:50.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:08:50 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:08:50.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:08:50 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:08:50.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:08:50 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:08:50.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:08:50 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 9661 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:08:51.513 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:08:51 smithi115 conmon[90393]: 2022-05-14T02:08:51.179+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:08:51.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:08:51 smithi047 ceph-mon[77147]: pgmap v5184: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:08:51.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:08:51 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:08:51.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:08:51 smithi115 ceph-mon[78593]: pgmap v5184: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:08:51.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:08:51 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:08:52.514 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:08:52 smithi115 conmon[90393]: 2022-05-14T02:08:52.147+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:08:52.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:08:52 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:08:52.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:08:52 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:08:53.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:08:53 smithi115 conmon[90393]: 2022-05-14T02:08:53.102+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:08:53.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:08:53 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:08:53.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:08:53 smithi047 ceph-mon[77147]: pgmap v5185: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:08:53.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:08:53 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:08:53.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:08:53 smithi115 ceph-mon[78593]: pgmap v5185: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:08:54.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:08:54 smithi115 conmon[90393]: 2022-05-14T02:08:54.079+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:08:54.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:08:54 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:08:54.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:08:54 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:08:54.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:08:54 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:08:54.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:08:54 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:08:54.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:08:54 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:08:54.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:08:54 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:08:55.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:08:55 smithi115 conmon[90393]: 2022-05-14T02:08:55.054+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:08:55.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:08:55 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:08:55.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:08:55 smithi047 ceph-mon[77147]: pgmap v5186: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:08:55.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:08:55 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 9670 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:08:55.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:08:55 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:08:55.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:08:55 smithi115 ceph-mon[78593]: pgmap v5186: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:08:55.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:08:55 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 9670 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:08:56.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:08:56 smithi115 conmon[90393]: 2022-05-14T02:08:56.018+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:08:56.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:08:56 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:08:56.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:08:56 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:08:57.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:08:57 smithi115 conmon[90393]: 2022-05-14T02:08:57.015+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:08:57.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:08:57 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:08:57.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:08:57 smithi115 ceph-mon[78593]: pgmap v5187: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:08:58.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:08:57 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:08:58.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:08:57 smithi047 ceph-mon[77147]: pgmap v5187: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:08:58.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:08:58 smithi115 conmon[90393]: 2022-05-14T02:08:58.053+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:08:58.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:08:58 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:08:58.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:08:58 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:08:58.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:08:58 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:08:59.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:08:58 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:08:59.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:08:58 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:08:59.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:08:58 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:08:59.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:08:59 smithi115 conmon[90393]: 2022-05-14T02:08:59.057+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:08:59.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:08:59 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:08:59.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:08:59 smithi115 ceph-mon[78593]: pgmap v5188: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:09:00.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:08:59 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:09:00.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:08:59 smithi047 ceph-mon[77147]: pgmap v5188: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:09:00.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:09:00 smithi115 conmon[90393]: 2022-05-14T02:09:00.041+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:09:00.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:09:00 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:09:01.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:09:00 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:09:01.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:09:01 smithi115 conmon[90393]: 2022-05-14T02:09:01.055+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:09:01.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:09:01 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:09:01.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:09:01 smithi115 ceph-mon[78593]: pgmap v5189: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:09:01.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:09:01 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 9675 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:09:02.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:09:01 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:09:02.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:09:01 smithi047 ceph-mon[77147]: pgmap v5189: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:09:02.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:09:01 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 9675 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:09:02.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:09:02 smithi115 conmon[90393]: 2022-05-14T02:09:02.031+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:09:02.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:09:02 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:09:02.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:09:02 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:09:02.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:09:02 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:09:02.920 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:09:02 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:09:02.920 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:09:02 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:09:02.921 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:09:02 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:09:03.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:09:03 smithi115 conmon[90393]: 2022-05-14T02:09:03.003+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:09:03.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:09:03 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:09:03.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:09:03 smithi047 ceph-mon[77147]: pgmap v5190: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:09:03.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:09:03 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T02:09:03.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:09:03 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T02:09:03.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:09:03 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T02:09:03.793 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:09:03 smithi047 conmon[78521]: 2022-05-14T02:09:03.343+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:09:03.793 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:09:03 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T02:09:03.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:09:03 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:09:03.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:09:03 smithi047 conmon[78521]: yield 2022-05-14T02:09:03.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:09:03 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:09:03.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:09:03 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:09:03.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:09:03 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:09:03.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:09:03 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T02:09:03.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:09:03 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:09:03.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:09:03 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T02:09:03.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:09:03 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:09:03.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:09:03 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T02:09:03.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:09:03 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:09:03.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:09:03 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T02:09:03.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:09:03 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:09:03.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:09:03 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T02:09:03.797 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:09:03 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:09:03.797 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:09:03 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:09:03.797 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:09:03 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:09:03.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:09:03 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:09:03.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:09:03 smithi115 ceph-mon[78593]: pgmap v5190: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:09:03.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:09:03 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T02:09:03.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:09:03 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T02:09:03.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:09:03 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T02:09:04.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:09:04 smithi115 conmon[90393]: 2022-05-14T02:09:04.053+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:09:04.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:09:04 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:09:04.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:09:04 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:09:04.859 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T02:09:04.859 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:09:04.859 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T02:09:04.859 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:09:04.860 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:09:04.860 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:09:04.860 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T02:09:04.860 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:09:04.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T02:09:04.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:09:04.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T02:09:04.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:09:04.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T02:09:04.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:09:04.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T02:09:04.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:09:04.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:09:04.863 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:09:04.863 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:09:04 smithi115 ceph-mon[78593]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T02:09:05.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:09:04 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:09:05.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:09:04 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:09:05.042 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T02:09:05.042 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:09:05.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T02:09:05.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:09:05.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:09:05.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:09:05.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T02:09:05.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:09:05.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T02:09:05.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:09:05.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T02:09:05.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:09:05.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T02:09:05.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:09:05.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T02:09:05.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:09:05.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:09:05.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:09:05.046 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:09:04 smithi047 ceph-mon[77147]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T02:09:05.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:09:05 smithi115 conmon[90393]: 2022-05-14T02:09:05.089+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:09:05.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:09:05 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:09:05.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:09:05 smithi115 ceph-mon[78593]: pgmap v5191: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:09:06.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:09:05 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:09:06.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:09:05 smithi047 ceph-mon[77147]: pgmap v5191: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:09:06.580 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:09:06 smithi115 conmon[90393]: 2022-05-14T02:09:06.115+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:09:06.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:09:06 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:09:06.792 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:09:06 smithi047 conmon[78521]: 2022-05-14T02:09:06.415+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:09:06.792 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:09:06 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T02:09:06.793 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:09:06 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:09:06.793 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:09:06 smithi047 conmon[78521]: yield 2022-05-14T02:09:06.793 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:09:06 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:09:06.793 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:09:06 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:09:06.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:09:06 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:09:06.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:09:06 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T02:09:06.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:09:06 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:09:06.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:09:06 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T02:09:06.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:09:06 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:09:06.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:09:06 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T02:09:06.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:09:06 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:09:06.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:09:06 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T02:09:06.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:09:06 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:09:06.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:09:06 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T02:09:06.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:09:06 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:09:06.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:09:06 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:09:06.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:09:06 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:09:06.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:09:06 smithi047 conmon[78521]: 2022-05-14T02:09:06.416+0000 7f5344781700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T02:09:06.797 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:09:06 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T02:09:06.797 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:09:06 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:09:06.797 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:09:06 smithi047 conmon[78521]: yield 2022-05-14T02:09:06.797 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:09:06 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:09:06.798 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:09:06 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:09:06.798 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:09:06 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:09:06.798 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:09:06 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T02:09:06.798 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:09:06 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:09:06.798 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:09:06 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T02:09:06.799 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:09:06 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:09:06.799 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:09:06 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T02:09:06.799 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:09:06 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:09:06.799 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:09:06 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T02:09:06.800 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:09:06 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:09:06.800 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:09:06 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T02:09:06.800 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:09:06 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:09:06.800 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:09:06 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:09:06.801 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:09:06 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:09:06.801 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:09:06 smithi047 conmon[78521]: 2022-05-14T02:09:06.801 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:09:06 smithi047 conmon[78521]: During handling of the above exception, another exception occurred: 2022-05-14T02:09:06.801 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:09:06 smithi047 conmon[78521]: 2022-05-14T02:09:06.801 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:09:06 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T02:09:06.802 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:09:06 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T02:09:06.802 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:09:06 smithi047 conmon[78521]: return f(*arg) 2022-05-14T02:09:06.802 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:09:06 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T02:09:06.802 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:09:06 smithi047 conmon[78521]: self._write_client_files(client_files, host) 2022-05-14T02:09:06.803 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:09:06 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T02:09:06.803 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:09:06 smithi047 conmon[78521]: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T02:09:06.803 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:09:06 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T02:09:06.803 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:09:06 smithi047 conmon[78521]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T02:09:06.804 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:09:06 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T02:09:06.804 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:09:06 smithi047 conmon[78521]: return self.event_loop.get_result(coro) 2022-05-14T02:09:06.804 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:09:06 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T02:09:06.804 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:09:06 smithi047 conmon[78521]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T02:09:06.804 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:09:06 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T02:09:06.805 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:09:06 smithi047 conmon[78521]: return self.__get_result() 2022-05-14T02:09:06.805 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:09:06 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T02:09:06.805 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:09:06 smithi047 conmon[78521]: raise self._exception 2022-05-14T02:09:06.805 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:09:06 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T02:09:06.806 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:09:06 smithi047 conmon[78521]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T02:09:06.806 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:09:06 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T02:09:06.806 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:09:06 smithi047 conmon[78521]: conn = await self._remote_connection(host, addr) 2022-05-14T02:09:06.806 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:09:06 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T02:09:06.806 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:09:06 smithi047 conmon[78521]: raise 2022-05-14T02:09:06.807 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:09:06 smithi047 conmon[78521]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T02:09:06.807 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:09:06 smithi047 conmon[78521]: self.gen.throw(type, value, traceback) 2022-05-14T02:09:06.807 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:09:06 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T02:09:06.807 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:09:06 smithi047 conmon[78521]: raise OrchestratorError(msg) 2022-05-14T02:09:06.808 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:09:06 smithi047 conmon[78521]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:09:06.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:09:06 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:09:07.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:09:07 smithi115 conmon[90393]: 2022-05-14T02:09:07.089+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:09:07.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:09:07 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:09:07.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:09:07 smithi115 ceph-mon[78593]: pgmap v5192: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:09:07.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:09:07 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:09:07.859 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T02:09:07.859 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:09:07.860 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T02:09:07.860 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:09:07.860 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:09:07.860 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:09:07.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T02:09:07.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:09:07.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T02:09:07.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:09:07.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T02:09:07.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:09:07.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T02:09:07.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:09:07.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T02:09:07.863 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:09:07.863 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:09:07.863 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:09:07.863 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:09:07 smithi115 ceph-mon[78593]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T02:09:07.863 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T02:09:07.864 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:09:07.864 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T02:09:07.864 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:09:07.864 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:09:07.865 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:09:07.865 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T02:09:07.865 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:09:07.865 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T02:09:07.865 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:09:07.866 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T02:09:07.866 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:09:07.866 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T02:09:07.866 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:09:07.866 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T02:09:07.867 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:09:07.867 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:09:07.867 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:09:07.867 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T02:09:07.868 INFO:journalctl@ceph.mon.c.smithi115.stdout: During handling of the above exception, another exception occurred: 2022-05-14T02:09:07.868 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T02:09:07.868 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T02:09:07.868 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T02:09:07.868 INFO:journalctl@ceph.mon.c.smithi115.stdout: return f(*arg) 2022-05-14T02:09:07.869 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T02:09:07.869 INFO:journalctl@ceph.mon.c.smithi115.stdout: self._write_client_files(client_files, host) 2022-05-14T02:09:07.869 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T02:09:07.869 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T02:09:07.870 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T02:09:07.870 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T02:09:07.870 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T02:09:07.870 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.event_loop.get_result(coro) 2022-05-14T02:09:07.870 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T02:09:07.871 INFO:journalctl@ceph.mon.c.smithi115.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T02:09:07.871 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T02:09:07.871 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.__get_result() 2022-05-14T02:09:07.871 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T02:09:07.871 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise self._exception 2022-05-14T02:09:07.872 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T02:09:07.872 INFO:journalctl@ceph.mon.c.smithi115.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T02:09:07.872 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T02:09:07.872 INFO:journalctl@ceph.mon.c.smithi115.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T02:09:07.872 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T02:09:07.873 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise 2022-05-14T02:09:07.873 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T02:09:07.873 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.gen.throw(type, value, traceback) 2022-05-14T02:09:07.873 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T02:09:07.873 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OrchestratorError(msg) 2022-05-14T02:09:07.874 INFO:journalctl@ceph.mon.c.smithi115.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:09:08.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:09:07 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:09:08.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:09:07 smithi047 ceph-mon[77147]: pgmap v5192: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:09:08.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:09:07 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:09:08.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T02:09:08.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:09:08.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T02:09:08.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:09:08.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:09:08.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:09:08.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T02:09:08.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:09:08.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T02:09:08.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:09:08.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T02:09:08.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:09:08.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T02:09:08.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:09:08.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T02:09:08.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:09:08.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:09:08.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:09:08.047 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:09:07 smithi047 ceph-mon[77147]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T02:09:08.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T02:09:08.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:09:08.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T02:09:08.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:09:08.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:09:08.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:09:08.049 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T02:09:08.049 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:09:08.049 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T02:09:08.049 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:09:08.049 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T02:09:08.050 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:09:08.050 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T02:09:08.050 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:09:08.050 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T02:09:08.051 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:09:08.051 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:09:08.051 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:09:08.051 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T02:09:08.051 INFO:journalctl@ceph.mon.b.smithi047.stdout: During handling of the above exception, another exception occurred: 2022-05-14T02:09:08.052 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T02:09:08.052 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T02:09:08.052 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T02:09:08.052 INFO:journalctl@ceph.mon.b.smithi047.stdout: return f(*arg) 2022-05-14T02:09:08.052 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T02:09:08.053 INFO:journalctl@ceph.mon.b.smithi047.stdout: self._write_client_files(client_files, host) 2022-05-14T02:09:08.053 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T02:09:08.053 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T02:09:08.053 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T02:09:08.053 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T02:09:08.054 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T02:09:08.054 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.event_loop.get_result(coro) 2022-05-14T02:09:08.054 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T02:09:08.054 INFO:journalctl@ceph.mon.b.smithi047.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T02:09:08.054 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T02:09:08.055 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.__get_result() 2022-05-14T02:09:08.055 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T02:09:08.055 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise self._exception 2022-05-14T02:09:08.055 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T02:09:08.056 INFO:journalctl@ceph.mon.b.smithi047.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T02:09:08.056 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T02:09:08.056 INFO:journalctl@ceph.mon.b.smithi047.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T02:09:08.056 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T02:09:08.057 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise 2022-05-14T02:09:08.057 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T02:09:08.057 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.gen.throw(type, value, traceback) 2022-05-14T02:09:08.057 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T02:09:08.057 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OrchestratorError(msg) 2022-05-14T02:09:08.058 INFO:journalctl@ceph.mon.b.smithi047.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:09:08.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:09:08 smithi115 conmon[90393]: 2022-05-14T02:09:08.093+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:09:08.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:09:08 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:09:09.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:09:08 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:09:09.588 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:09:09 smithi115 conmon[90393]: 2022-05-14T02:09:09.108+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:09:09.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:09:09 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:09:09.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:09:09 smithi115 ceph-mon[78593]: pgmap v5193: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:09:10.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:09:09 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:09:10.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:09:09 smithi047 ceph-mon[77147]: pgmap v5193: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:09:10.597 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:09:10 smithi115 conmon[90393]: 2022-05-14T02:09:10.147+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:09:10.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:09:10 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:09:10.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:09:10 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:09:10.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:09:10 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:09:10.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:09:10 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 9680 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:09:11.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:09:10 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:09:11.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:09:10 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:09:11.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:09:10 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:09:11.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:09:10 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 9680 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:09:11.598 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:09:11 smithi115 conmon[90393]: 2022-05-14T02:09:11.183+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:09:11.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:09:11 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:09:11.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:09:11 smithi115 ceph-mon[78593]: pgmap v5194: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:09:12.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:09:11 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:09:12.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:09:11 smithi047 ceph-mon[77147]: pgmap v5194: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:09:12.605 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:09:12 smithi115 conmon[90393]: 2022-05-14T02:09:12.221+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:09:12.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:09:12 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:09:13.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:09:12 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:09:13.604 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:09:13 smithi115 conmon[90393]: 2022-05-14T02:09:13.249+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:09:13.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:09:13 smithi115 ceph-mon[78593]: pgmap v5195: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:09:13.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:09:13 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:09:14.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:09:13 smithi047 ceph-mon[77147]: pgmap v5195: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:09:14.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:09:13 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:09:14.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:09:14 smithi115 conmon[90393]: 2022-05-14T02:09:14.267+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:09:15.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:09:14 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:09:15.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:09:14 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:09:15.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:09:14 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:09:15.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:09:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T02:09:15.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:09:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T02:09:15.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:09:14 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:09:15.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:09:14 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:09:15.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:09:14 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:09:15.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:09:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T02:09:15.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:09:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T02:09:15.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:09:15 smithi115 conmon[90393]: 2022-05-14T02:09:15.251+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:09:16.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:09:15 smithi047 ceph-mon[77147]: pgmap v5196: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:09:16.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:09:15 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:09:16.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:09:15 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 9690 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:09:16.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:09:15 smithi115 ceph-mon[78593]: pgmap v5196: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:09:16.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:09:15 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:09:16.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:09:15 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 9690 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:09:16.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:09:16 smithi115 conmon[90393]: 2022-05-14T02:09:16.288+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:09:17.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:09:16 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:09:17.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:09:16 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:09:17.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:09:17 smithi115 conmon[90393]: 2022-05-14T02:09:17.253+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:09:18.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:09:17 smithi047 ceph-mon[77147]: pgmap v5197: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:09:18.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:09:17 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:09:18.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:09:17 smithi115 ceph-mon[78593]: pgmap v5197: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:09:18.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:09:17 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:09:18.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:09:18 smithi115 conmon[90393]: 2022-05-14T02:09:18.237+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:09:19.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:09:18 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:09:19.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:09:18 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:09:19.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:09:18 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:09:19.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:09:18 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:09:19.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:09:18 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:09:19.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:09:18 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:09:19.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:09:19 smithi115 conmon[90393]: 2022-05-14T02:09:19.195+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:09:20.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:09:19 smithi047 ceph-mon[77147]: pgmap v5198: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:09:20.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:09:19 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:09:20.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:09:19 smithi115 ceph-mon[78593]: pgmap v5198: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:09:20.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:09:19 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:09:20.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:09:20 smithi115 conmon[90393]: 2022-05-14T02:09:20.223+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:09:21.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:09:20 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:09:21.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:09:20 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:09:21.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:09:21 smithi115 conmon[90393]: 2022-05-14T02:09:21.189+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:09:22.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:09:21 smithi047 ceph-mon[77147]: pgmap v5199: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:09:22.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:09:21 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:09:22.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:09:21 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 9695 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:09:22.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:09:21 smithi115 ceph-mon[78593]: pgmap v5199: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:09:22.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:09:21 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:09:22.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:09:21 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 9695 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:09:22.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:09:22 smithi115 conmon[90393]: 2022-05-14T02:09:22.164+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:09:23.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:09:22 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:09:23.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:09:22 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:09:23.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:09:22 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:09:23.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:09:22 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:09:23.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:09:22 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:09:23.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:09:22 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:09:23.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:09:23 smithi115 conmon[90393]: 2022-05-14T02:09:23.173+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:09:24.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:09:23 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:09:24.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:09:23 smithi047 ceph-mon[77147]: pgmap v5200: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:09:24.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:09:23 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:09:24.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:09:23 smithi115 ceph-mon[78593]: pgmap v5200: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:09:24.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:09:24 smithi115 conmon[90393]: 2022-05-14T02:09:24.152+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:09:25.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:09:24 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:09:25.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:09:24 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:09:25.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:09:25 smithi115 conmon[90393]: 2022-05-14T02:09:25.152+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:09:26.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:09:25 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:09:26.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:09:25 smithi047 ceph-mon[77147]: pgmap v5201: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:09:26.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:09:25 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:09:26.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:09:25 smithi115 ceph-mon[78593]: pgmap v5201: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:09:26.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:09:26 smithi115 conmon[90393]: 2022-05-14T02:09:26.181+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:09:27.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:09:26 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:09:27.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:09:26 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:09:27.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:09:27 smithi115 conmon[90393]: 2022-05-14T02:09:27.150+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:09:28.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:09:27 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:09:28.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:09:27 smithi047 ceph-mon[77147]: pgmap v5202: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:09:28.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:09:27 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:09:28.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:09:27 smithi115 ceph-mon[78593]: pgmap v5202: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:09:28.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:09:28 smithi115 conmon[90393]: 2022-05-14T02:09:28.116+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:09:29.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:09:28 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:09:29.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:09:28 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:09:29.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:09:29 smithi115 conmon[90393]: 2022-05-14T02:09:29.112+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:09:30.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:09:29 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:09:30.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:09:29 smithi047 ceph-mon[77147]: pgmap v5203: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:09:30.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:09:29 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:09:30.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:09:29 smithi115 ceph-mon[78593]: pgmap v5203: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:09:30.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:09:30 smithi115 conmon[90393]: 2022-05-14T02:09:30.148+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:09:31.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:09:30 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:09:31.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:09:30 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:09:31.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:09:30 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:09:31.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:09:30 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 9701 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:09:31.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:09:30 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:09:31.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:09:30 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:09:31.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:09:30 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:09:31.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:09:30 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 9701 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:09:31.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:09:31 smithi115 conmon[90393]: 2022-05-14T02:09:31.117+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:09:32.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:09:31 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:09:32.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:09:31 smithi047 ceph-mon[77147]: pgmap v5204: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:09:32.089 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:09:31 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:09:32.089 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:09:31 smithi115 ceph-mon[78593]: pgmap v5204: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:09:32.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:09:32 smithi115 conmon[90393]: 2022-05-14T02:09:32.088+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:09:33.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:09:32 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:09:33.051 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:09:32 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:09:33.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:09:33 smithi115 conmon[90393]: 2022-05-14T02:09:33.049+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:09:34.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:09:33 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:09:34.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:09:33 smithi047 ceph-mon[77147]: pgmap v5205: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:09:34.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:09:33 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:09:34.043 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:09:33 smithi115 ceph-mon[78593]: pgmap v5205: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:09:34.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:09:34 smithi115 conmon[90393]: 2022-05-14T02:09:34.041+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:09:35.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:09:34 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:09:35.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:09:34 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:09:35.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:09:34 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:09:35.089 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:09:34 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:09:35.089 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:09:34 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:09:35.090 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:09:34 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:09:35.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:09:35 smithi115 conmon[90393]: 2022-05-14T02:09:35.088+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:09:36.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:09:35 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:09:36.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:09:35 smithi047 ceph-mon[77147]: pgmap v5206: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:09:36.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:09:35 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 9710 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:09:36.057 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:09:35 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:09:36.057 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:09:35 smithi115 ceph-mon[78593]: pgmap v5206: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:09:36.058 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:09:35 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 9710 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:09:36.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:09:36 smithi115 conmon[90393]: 2022-05-14T02:09:36.056+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:09:37.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:09:36 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:09:37.092 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:09:36 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:09:37.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:09:37 smithi115 conmon[90393]: 2022-05-14T02:09:37.091+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:09:38.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:09:37 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:09:38.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:09:37 smithi047 ceph-mon[77147]: pgmap v5207: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:09:38.084 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:09:37 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:09:38.084 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:09:37 smithi115 ceph-mon[78593]: pgmap v5207: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:09:38.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:09:38 smithi115 conmon[90393]: 2022-05-14T02:09:38.083+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:09:39.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:09:38 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:09:39.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:09:38 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:09:39.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:09:38 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:09:39.089 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:09:38 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:09:39.089 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:09:38 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:09:39.090 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:09:38 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:09:39.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:09:39 smithi115 conmon[90393]: 2022-05-14T02:09:39.088+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:09:40.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:09:39 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:09:40.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:09:39 smithi047 ceph-mon[77147]: pgmap v5208: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:09:40.086 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:09:39 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:09:40.087 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:09:39 smithi115 ceph-mon[78593]: pgmap v5208: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:09:40.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:09:40 smithi115 conmon[90393]: 2022-05-14T02:09:40.084+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:09:41.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:09:40 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:09:41.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:09:40 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:09:41.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:09:41 smithi115 conmon[90393]: 2022-05-14T02:09:41.132+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:09:42.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:09:41 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:09:42.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:09:41 smithi047 ceph-mon[77147]: pgmap v5209: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:09:42.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:09:41 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 9716 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:09:42.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:09:41 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:09:42.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:09:41 smithi115 ceph-mon[78593]: pgmap v5209: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:09:42.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:09:41 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 9716 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:09:42.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:09:42 smithi115 conmon[90393]: 2022-05-14T02:09:42.135+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:09:43.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:09:42 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:09:43.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:09:42 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:09:43.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:09:42 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:09:43.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:09:42 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:09:43.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:09:42 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:09:43.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:09:42 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:09:43.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:09:43 smithi115 conmon[90393]: 2022-05-14T02:09:43.154+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:09:44.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:09:43 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:09:44.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:09:43 smithi047 ceph-mon[77147]: pgmap v5210: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:09:44.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:09:43 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:09:44.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:09:43 smithi115 ceph-mon[78593]: pgmap v5210: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:09:44.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:09:44 smithi115 conmon[90393]: 2022-05-14T02:09:44.173+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:09:45.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:09:44 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:09:45.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:09:44 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:09:45.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:09:45 smithi115 conmon[90393]: 2022-05-14T02:09:45.166+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:09:46.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:09:45 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:09:46.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:09:45 smithi047 ceph-mon[77147]: pgmap v5211: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:09:46.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:09:45 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:09:46.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:09:45 smithi115 ceph-mon[78593]: pgmap v5211: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:09:46.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:09:46 smithi115 conmon[90393]: 2022-05-14T02:09:46.169+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:09:47.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:09:46 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:09:47.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:09:46 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:09:47.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:09:47 smithi115 conmon[90393]: 2022-05-14T02:09:47.214+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:09:48.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:09:47 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:09:48.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:09:47 smithi047 ceph-mon[77147]: pgmap v5212: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:09:48.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:09:47 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:09:48.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:09:47 smithi115 ceph-mon[78593]: pgmap v5212: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:09:48.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:09:48 smithi115 conmon[90393]: 2022-05-14T02:09:48.240+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:09:49.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:09:48 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:09:49.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:09:48 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:09:49.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:09:49 smithi115 conmon[90393]: 2022-05-14T02:09:49.214+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:09:50.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:09:49 smithi047 ceph-mon[77147]: pgmap v5213: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:09:50.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:09:49 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:09:50.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:09:49 smithi115 ceph-mon[78593]: pgmap v5213: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:09:50.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:09:49 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:09:50.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:09:50 smithi115 conmon[90393]: 2022-05-14T02:09:50.169+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:09:51.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:09:50 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:09:51.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:09:50 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:09:51.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:09:50 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:09:51.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:09:50 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 9720 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:09:51.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:09:50 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:09:51.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:09:50 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:09:51.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:09:50 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:09:51.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:09:50 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 9720 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:09:51.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:09:51 smithi115 conmon[90393]: 2022-05-14T02:09:51.185+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:09:52.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:09:51 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:09:52.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:09:51 smithi047 ceph-mon[77147]: pgmap v5214: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:09:52.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:09:51 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:09:52.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:09:51 smithi115 ceph-mon[78593]: pgmap v5214: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:09:52.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:09:52 smithi115 conmon[90393]: 2022-05-14T02:09:52.172+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:09:53.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:09:52 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:09:53.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:09:52 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:09:53.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:09:53 smithi115 conmon[90393]: 2022-05-14T02:09:53.201+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:09:54.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:09:53 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:09:54.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:09:53 smithi047 ceph-mon[77147]: pgmap v5215: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:09:54.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:09:53 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:09:54.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:09:53 smithi115 ceph-mon[78593]: pgmap v5215: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:09:54.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:09:54 smithi115 conmon[90393]: 2022-05-14T02:09:54.226+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:09:55.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:09:54 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:09:55.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:09:54 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:09:55.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:09:54 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:09:55.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:09:54 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:09:55.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:09:54 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:09:55.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:09:54 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:09:55.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:09:55 smithi115 conmon[90393]: 2022-05-14T02:09:55.238+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:09:56.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:09:55 smithi047 ceph-mon[77147]: pgmap v5216: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:09:56.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:09:55 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:09:56.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:09:55 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 9730 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:09:56.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:09:55 smithi115 ceph-mon[78593]: pgmap v5216: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:09:56.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:09:55 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:09:56.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:09:55 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 9730 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:09:56.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:09:56 smithi115 conmon[90393]: 2022-05-14T02:09:56.245+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:09:57.047 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:09:56 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:09:57.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:09:56 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:09:57.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:09:57 smithi115 conmon[90393]: 2022-05-14T02:09:57.222+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:09:58.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:09:57 smithi115 ceph-mon[78593]: pgmap v5217: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:09:58.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:09:57 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:09:58.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:09:57 smithi047 ceph-mon[77147]: pgmap v5217: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:09:58.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:09:57 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:09:58.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:09:58 smithi115 conmon[90393]: 2022-05-14T02:09:58.192+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:09:59.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:09:58 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:09:59.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:09:58 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:09:59.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:09:58 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:09:59.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:09:58 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:09:59.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:09:58 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:09:59.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:09:58 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:09:59.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:09:59 smithi115 conmon[90393]: 2022-05-14T02:09:59.150+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:10:00.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:09:59 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:10:00.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:09:59 smithi115 ceph-mon[78593]: pgmap v5218: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:10:00.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:09:59 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:10:00.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:09:59 smithi047 ceph-mon[77147]: pgmap v5218: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:10:00.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:10:00 smithi047 conmon[77123]: 2022-05-14T02:09:59.998+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : Health detail: HEALTH_ERR 1 filesystem is degraded; 1 mds daemon damaged; 1 MDSs are read only; 1 MDSs report slow metadata IOs; 1/3 mons down, quorum b,c; Reduced data availability: 1 pg inactive; 9 slow ops, oldest one blocked for 9730 sec, osd.5 has slow ops 2022-05-14T02:10:00.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:10:00 smithi047 conmon[77123]: 2022-05-14T02:09:59.998+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : [WRN] FS_DEGRADED: 1 filesystem is degraded 2022-05-14T02:10:00.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:10:00 smithi047 conmon[77123]: 2022-05-14T02:09:59.998+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : fs cephfs is degraded 2022-05-14T02:10:00.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:10:00 smithi047 conmon[77123]: 2022-05-14T02:09:59.998+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : [ERR] MDS_DAMAGE: 1 mds daemon damaged 2022-05-14T02:10:00.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:10:00 smithi047 conmon[77123]: 2022-05-14T02:09:59.998+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : fs cephfs mds.2 is damaged 2022-05-14T02:10:00.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:10:00 smithi047 conmon[77123]: 2022-05-14T02:09:59.998+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : [WRN] MDS_READ_ONLY: 1 MDSs are read only 2022-05-14T02:10:00.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:10:00 smithi047 conmon[77123]: 2022-05-14T02:09:59.998+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : mds.l(mds.0): MDS in read-only mode 2022-05-14T02:10:00.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:10:00 smithi047 conmon[77123]: 2022-05-14T02:09:59.998+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : [WRN] MDS_SLOW_METADATA_IO: 1 MDSs report slow metadata IOs 2022-05-14T02:10:00.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:10:00 smithi047 conmon[77123]: 2022-05-14T02:09:59.998+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : mds.l(mds.0): 4 slow metadata IOs are blocked > 180 secs, oldest blocked for 10033 secs 2022-05-14T02:10:00.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:10:00 smithi047 conmon[77123]: 2022-05-14T02:09:59.998+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : [WRN] MON_DOWN: 1/3 mons down, quorum b,c 2022-05-14T02:10:00.295 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:10:00 smithi047 conmon[77123]: 2022-05-14T02:09:59.998+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : mon.a (rank 0) addr [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] is down (out of quorum) 2022-05-14T02:10:00.295 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:10:00 smithi047 conmon[77123]: 2022-05-14T02:09:59.998+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : [WRN] PG_AVAILABILITY: Reduced data availability: 1 pg inactive 2022-05-14T02:10:00.295 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:10:00 smithi047 conmon[77123]: 2022-05-14T02:09:59.998+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : pg 2.1 is stuck inactive for 2h, current state unknown, last acting [] 2022-05-14T02:10:00.295 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:10:00 smithi047 conmon[77123]: 2022-05-14T02:09:59.999+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : [WRN] SLOW_OPS: 9 slow ops, oldest one blocked for 9730 sec, osd.5 has slow ops 2022-05-14T02:10:00.613 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:10:00 smithi115 conmon[90393]: 2022-05-14T02:10:00.197+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:10:01.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:10:00 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:10:01.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:10:00 smithi115 ceph-mon[78593]: Health detail: HEALTH_ERR 1 filesystem is degraded; 1 mds daemon damaged; 1 MDSs are read only; 1 MDSs report slow metadata IOs; 1/3 mons down, quorum b,c; Reduced data availability: 1 pg inactive; 9 slow ops, oldest one blocked for 9730 sec, osd.5 has slow ops 2022-05-14T02:10:01.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:10:00 smithi115 ceph-mon[78593]: [WRN] FS_DEGRADED: 1 filesystem is degraded 2022-05-14T02:10:01.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:10:00 smithi115 ceph-mon[78593]: fs cephfs is degraded 2022-05-14T02:10:01.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:10:00 smithi115 ceph-mon[78593]: [ERR] MDS_DAMAGE: 1 mds daemon damaged 2022-05-14T02:10:01.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:10:00 smithi115 ceph-mon[78593]: fs cephfs mds.2 is damaged 2022-05-14T02:10:01.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:10:00 smithi115 ceph-mon[78593]: [WRN] MDS_READ_ONLY: 1 MDSs are read only 2022-05-14T02:10:01.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:10:00 smithi115 ceph-mon[78593]: mds.l(mds.0): MDS in read-only mode 2022-05-14T02:10:01.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:10:00 smithi115 ceph-mon[78593]: [WRN] MDS_SLOW_METADATA_IO: 1 MDSs report slow metadata IOs 2022-05-14T02:10:01.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:10:00 smithi115 ceph-mon[78593]: mds.l(mds.0): 4 slow metadata IOs are blocked > 180 secs, oldest blocked for 10033 secs 2022-05-14T02:10:01.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:10:00 smithi115 ceph-mon[78593]: [WRN] MON_DOWN: 1/3 mons down, quorum b,c 2022-05-14T02:10:01.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:10:00 smithi115 ceph-mon[78593]: mon.a (rank 0) addr [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] is down (out of quorum) 2022-05-14T02:10:01.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:10:00 smithi115 ceph-mon[78593]: [WRN] PG_AVAILABILITY: Reduced data availability: 1 pg inactive 2022-05-14T02:10:01.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:10:00 smithi115 ceph-mon[78593]: pg 2.1 is stuck inactive for 2h, current state unknown, last acting [] 2022-05-14T02:10:01.112 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:10:00 smithi115 ceph-mon[78593]: [WRN] SLOW_OPS: 9 slow ops, oldest one blocked for 9730 sec, osd.5 has slow ops 2022-05-14T02:10:01.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:10:00 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:10:01.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:10:00 smithi047 ceph-mon[77147]: Health detail: HEALTH_ERR 1 filesystem is degraded; 1 mds daemon damaged; 1 MDSs are read only; 1 MDSs report slow metadata IOs; 1/3 mons down, quorum b,c; Reduced data availability: 1 pg inactive; 9 slow ops, oldest one blocked for 9730 sec, osd.5 has slow ops 2022-05-14T02:10:01.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:10:00 smithi047 ceph-mon[77147]: [WRN] FS_DEGRADED: 1 filesystem is degraded 2022-05-14T02:10:01.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:10:00 smithi047 ceph-mon[77147]: fs cephfs is degraded 2022-05-14T02:10:01.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:10:00 smithi047 ceph-mon[77147]: [ERR] MDS_DAMAGE: 1 mds daemon damaged 2022-05-14T02:10:01.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:10:00 smithi047 ceph-mon[77147]: fs cephfs mds.2 is damaged 2022-05-14T02:10:01.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:10:00 smithi047 ceph-mon[77147]: [WRN] MDS_READ_ONLY: 1 MDSs are read only 2022-05-14T02:10:01.295 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:10:00 smithi047 ceph-mon[77147]: mds.l(mds.0): MDS in read-only mode 2022-05-14T02:10:01.295 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:10:00 smithi047 ceph-mon[77147]: [WRN] MDS_SLOW_METADATA_IO: 1 MDSs report slow metadata IOs 2022-05-14T02:10:01.296 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:10:00 smithi047 ceph-mon[77147]: mds.l(mds.0): 4 slow metadata IOs are blocked > 180 secs, oldest blocked for 10033 secs 2022-05-14T02:10:01.296 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:10:00 smithi047 ceph-mon[77147]: [WRN] MON_DOWN: 1/3 mons down, quorum b,c 2022-05-14T02:10:01.296 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:10:00 smithi047 ceph-mon[77147]: mon.a (rank 0) addr [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] is down (out of quorum) 2022-05-14T02:10:01.297 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:10:00 smithi047 ceph-mon[77147]: [WRN] PG_AVAILABILITY: Reduced data availability: 1 pg inactive 2022-05-14T02:10:01.297 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:10:00 smithi047 ceph-mon[77147]: pg 2.1 is stuck inactive for 2h, current state unknown, last acting [] 2022-05-14T02:10:01.298 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:10:00 smithi047 ceph-mon[77147]: [WRN] SLOW_OPS: 9 slow ops, oldest one blocked for 9730 sec, osd.5 has slow ops 2022-05-14T02:10:01.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:10:01 smithi115 conmon[90393]: 2022-05-14T02:10:01.214+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:10:02.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:10:01 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:10:02.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:10:01 smithi115 ceph-mon[78593]: pgmap v5219: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:10:02.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:10:01 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 9736 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:10:02.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:10:01 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:10:02.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:10:01 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:10:02.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:10:01 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:10:02.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:10:01 smithi047 ceph-mon[77147]: pgmap v5219: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:10:02.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:10:01 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 9736 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:10:02.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:10:01 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:10:02.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:10:01 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:10:02.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:10:02 smithi115 conmon[90393]: 2022-05-14T02:10:02.210+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:10:03.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:10:02 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:10:03.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:10:02 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:10:03.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:10:03 smithi115 conmon[90393]: 2022-05-14T02:10:03.213+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:10:04.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:10:03 smithi115 ceph-mon[78593]: pgmap v5220: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:10:04.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:10:03 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:10:04.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:10:03 smithi047 ceph-mon[77147]: pgmap v5220: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:10:04.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:10:03 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:10:04.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:10:04 smithi115 conmon[90393]: 2022-05-14T02:10:04.203+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:10:05.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:10:04 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:10:05.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:10:04 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:10:05.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:10:05 smithi115 conmon[90393]: 2022-05-14T02:10:05.174+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:10:06.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:10:05 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:10:06.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:10:05 smithi115 ceph-mon[78593]: pgmap v5221: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:10:06.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:10:05 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:10:06.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:10:05 smithi047 ceph-mon[77147]: pgmap v5221: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:10:06.463 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:10:06 smithi115 conmon[90393]: 2022-05-14T02:10:06.182+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:10:07.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:10:06 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:10:07.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:10:06 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T02:10:07.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:10:06 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T02:10:07.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:10:06 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T02:10:07.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:10:06 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:10:07.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:10:06 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T02:10:07.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:10:06 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T02:10:07.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:10:06 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T02:10:07.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:10:07 smithi115 conmon[90393]: 2022-05-14T02:10:07.232+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:10:08.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:10:07 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:10:08.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:10:07 smithi115 ceph-mon[78593]: pgmap v5222: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:10:08.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:10:07 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:10:08.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:10:07 smithi047 ceph-mon[77147]: pgmap v5222: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:10:08.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:10:08 smithi115 conmon[90393]: 2022-05-14T02:10:08.248+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:10:09.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:10:08 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:10:09.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:10:08 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:10:09.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:10:09 smithi115 conmon[90393]: 2022-05-14T02:10:09.248+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:10:09.791 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:10:09 smithi047 conmon[78521]: 2022-05-14T02:10:09.520+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:10:09.792 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:10:09 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T02:10:09.792 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:10:09 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:10:09.792 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:10:09 smithi047 conmon[78521]: yield 2022-05-14T02:10:09.792 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:10:09 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:10:09.793 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:10:09 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:10:09.793 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:10:09 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:10:09.793 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:10:09 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T02:10:09.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:10:09 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:10:09.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:10:09 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T02:10:09.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:10:09 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:10:09.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:10:09 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T02:10:09.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:10:09 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:10:09.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:10:09 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T02:10:09.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:10:09 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:10:09.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:10:09 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T02:10:09.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:10:09 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:10:09.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:10:09 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:10:09.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:10:09 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:10:10.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:10:09 smithi115 ceph-mon[78593]: pgmap v5223: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:10:10.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:10:09 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:10:10.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:10:09 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:10:10.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:10:09 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:10:10.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:10:09 smithi047 ceph-mon[77147]: pgmap v5223: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:10:10.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:10:09 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:10:10.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:10:09 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:10:10.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:10:09 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:10:10.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:10:10 smithi115 conmon[90393]: 2022-05-14T02:10:10.252+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:10:11.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:10:10 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:10:11.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:10:10 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:10:11.109 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T02:10:11.109 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:10:11.109 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T02:10:11.109 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:10:11.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:10:11.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:10:11.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T02:10:11.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:10:11.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T02:10:11.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:10:11.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T02:10:11.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:10:11.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T02:10:11.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:10:11.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T02:10:11.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:10:11.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:10:11.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:10:11.113 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:10:10 smithi115 ceph-mon[78593]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T02:10:11.113 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:10:10 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 9740 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:10:11.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:10:10 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:10:11.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:10:10 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:10:11.292 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T02:10:11.292 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:10:11.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T02:10:11.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:10:11.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:10:11.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:10:11.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T02:10:11.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:10:11.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T02:10:11.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:10:11.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T02:10:11.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:10:11.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T02:10:11.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:10:11.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T02:10:11.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:10:11.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:10:11.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:10:11.296 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:10:10 smithi047 ceph-mon[77147]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T02:10:11.297 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:10:10 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 9740 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:10:11.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:10:11 smithi115 conmon[90393]: 2022-05-14T02:10:11.233+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:10:12.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:10:11 smithi115 ceph-mon[78593]: pgmap v5224: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:10:12.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:10:11 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:10:12.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:10:11 smithi047 ceph-mon[77147]: pgmap v5224: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:10:12.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:10:11 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:10:12.611 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:10:12 smithi115 conmon[90393]: 2022-05-14T02:10:12.185+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:10:12.847 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:10:12 smithi047 conmon[78521]: 2022-05-14T02:10:12.592+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:10:12.847 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:10:12 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T02:10:12.848 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:10:12 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:10:12.849 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:10:12 smithi047 conmon[78521]: yield 2022-05-14T02:10:12.849 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:10:12 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:10:12.849 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:10:12 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:10:12.849 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:10:12 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:10:12.850 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:10:12 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T02:10:12.850 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:10:12 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:10:12.850 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:10:12 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T02:10:12.850 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:10:12 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:10:12.851 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:10:12 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T02:10:12.851 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:10:12 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:10:12.851 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:10:12 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T02:10:12.851 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:10:12 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:10:12.851 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:10:12 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T02:10:12.852 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:10:12 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:10:12.852 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:10:12 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:10:12.852 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:10:12 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:10:12.852 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:10:12 smithi047 conmon[78521]: 2022-05-14T02:10:12.594+0000 7f5345f84700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T02:10:12.853 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:10:12 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T02:10:12.853 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:10:12 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:10:12.853 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:10:12 smithi047 conmon[78521]: yield 2022-05-14T02:10:12.853 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:10:12 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:10:12.854 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:10:12 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:10:12.854 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:10:12 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:10:12.854 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:10:12 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T02:10:12.854 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:10:12 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:10:12.854 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:10:12 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T02:10:12.855 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:10:12 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:10:12.855 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:10:12 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T02:10:12.855 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:10:12 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:10:12.855 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:10:12 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T02:10:12.856 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:10:12 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:10:12.856 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:10:12 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T02:10:12.856 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:10:12 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:10:12.856 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:10:12 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:10:12.856 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:10:12 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:10:12.857 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:10:12 smithi047 conmon[78521]: 2022-05-14T02:10:12.857 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:10:12 smithi047 conmon[78521]: During handling of the above exception, another exception occurred: 2022-05-14T02:10:12.857 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:10:12 smithi047 conmon[78521]: 2022-05-14T02:10:12.857 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:10:12 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T02:10:12.858 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:10:12 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T02:10:12.858 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:10:12 smithi047 conmon[78521]: return f(*arg) 2022-05-14T02:10:12.859 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:10:12 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T02:10:12.859 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:10:12 smithi047 conmon[78521]: self._write_client_files(client_files, host) 2022-05-14T02:10:12.859 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:10:12 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T02:10:12.859 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:10:12 smithi047 conmon[78521]: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T02:10:12.860 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:10:12 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T02:10:12.860 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:10:12 smithi047 conmon[78521]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T02:10:12.860 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:10:12 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T02:10:12.860 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:10:12 smithi047 conmon[78521]: return self.event_loop.get_result(coro) 2022-05-14T02:10:12.860 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:10:12 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T02:10:12.861 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:10:12 smithi047 conmon[78521]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T02:10:12.861 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:10:12 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T02:10:12.861 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:10:12 smithi047 conmon[78521]: return self.__get_result() 2022-05-14T02:10:12.861 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:10:12 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T02:10:12.862 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:10:12 smithi047 conmon[78521]: raise self._exception 2022-05-14T02:10:12.862 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:10:12 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T02:10:12.862 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:10:12 smithi047 conmon[78521]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T02:10:12.862 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:10:12 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T02:10:12.862 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:10:12 smithi047 conmon[78521]: conn = await self._remote_connection(host, addr) 2022-05-14T02:10:12.863 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:10:12 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T02:10:12.863 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:10:12 smithi047 conmon[78521]: raise 2022-05-14T02:10:12.863 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:10:12 smithi047 conmon[78521]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T02:10:12.863 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:10:12 smithi047 conmon[78521]: self.gen.throw(type, value, traceback) 2022-05-14T02:10:12.864 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:10:12 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T02:10:12.864 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:10:12 smithi047 conmon[78521]: raise OrchestratorError(msg) 2022-05-14T02:10:12.864 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:10:12 smithi047 conmon[78521]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:10:13.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:10:12 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:10:13.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:10:12 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:10:13.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:10:13 smithi115 conmon[90393]: 2022-05-14T02:10:13.160+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:10:14.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:10:13 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:10:14.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:10:13 smithi115 ceph-mon[78593]: pgmap v5225: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:10:14.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:10:13 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:10:14.109 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T02:10:14.109 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:10:14.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T02:10:14.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:10:14.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:10:14.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:10:14.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T02:10:14.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:10:14.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T02:10:14.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:10:14.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T02:10:14.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:10:14.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T02:10:14.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:10:14.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T02:10:14.113 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:10:14.113 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:10:14.114 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:10:14.114 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:10:13 smithi115 ceph-mon[78593]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T02:10:14.114 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T02:10:14.114 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:10:14.114 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T02:10:14.115 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:10:14.115 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:10:14.115 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:10:14.115 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T02:10:14.116 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:10:14.116 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T02:10:14.116 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:10:14.116 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T02:10:14.116 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:10:14.117 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T02:10:14.117 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:10:14.118 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T02:10:14.118 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:10:14.118 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:10:14.119 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:10:14.119 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T02:10:14.119 INFO:journalctl@ceph.mon.c.smithi115.stdout: During handling of the above exception, another exception occurred: 2022-05-14T02:10:14.120 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T02:10:14.120 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T02:10:14.121 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T02:10:14.123 INFO:journalctl@ceph.mon.c.smithi115.stdout: return f(*arg) 2022-05-14T02:10:14.123 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T02:10:14.123 INFO:journalctl@ceph.mon.c.smithi115.stdout: self._write_client_files(client_files, host) 2022-05-14T02:10:14.124 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T02:10:14.124 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T02:10:14.125 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T02:10:14.125 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T02:10:14.125 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T02:10:14.128 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.event_loop.get_result(coro) 2022-05-14T02:10:14.129 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T02:10:14.129 INFO:journalctl@ceph.mon.c.smithi115.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T02:10:14.130 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T02:10:14.130 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.__get_result() 2022-05-14T02:10:14.130 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T02:10:14.131 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise self._exception 2022-05-14T02:10:14.131 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T02:10:14.132 INFO:journalctl@ceph.mon.c.smithi115.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T02:10:14.132 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T02:10:14.134 INFO:journalctl@ceph.mon.c.smithi115.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T02:10:14.134 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T02:10:14.134 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise 2022-05-14T02:10:14.134 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T02:10:14.135 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.gen.throw(type, value, traceback) 2022-05-14T02:10:14.135 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T02:10:14.135 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OrchestratorError(msg) 2022-05-14T02:10:14.135 INFO:journalctl@ceph.mon.c.smithi115.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:10:14.136 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:10:13 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:10:14.138 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:10:13 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:10:14.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:10:13 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:10:14.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:10:13 smithi047 ceph-mon[77147]: pgmap v5225: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:10:14.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:10:13 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:10:14.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T02:10:14.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:10:14.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T02:10:14.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:10:14.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:10:14.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:10:14.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T02:10:14.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:10:14.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T02:10:14.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:10:14.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T02:10:14.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:10:14.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T02:10:14.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:10:14.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T02:10:14.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:10:14.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:10:14.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:10:14.297 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:10:13 smithi047 ceph-mon[77147]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T02:10:14.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T02:10:14.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:10:14.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T02:10:14.298 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:10:14.298 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:10:14.298 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:10:14.298 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T02:10:14.299 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:10:14.299 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T02:10:14.299 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:10:14.299 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T02:10:14.299 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:10:14.300 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T02:10:14.300 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:10:14.300 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T02:10:14.300 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:10:14.300 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:10:14.301 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:10:14.301 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T02:10:14.301 INFO:journalctl@ceph.mon.b.smithi047.stdout: During handling of the above exception, another exception occurred: 2022-05-14T02:10:14.301 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T02:10:14.302 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T02:10:14.302 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T02:10:14.302 INFO:journalctl@ceph.mon.b.smithi047.stdout: return f(*arg) 2022-05-14T02:10:14.302 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T02:10:14.302 INFO:journalctl@ceph.mon.b.smithi047.stdout: self._write_client_files(client_files, host) 2022-05-14T02:10:14.303 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T02:10:14.303 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T02:10:14.303 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T02:10:14.303 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T02:10:14.304 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T02:10:14.304 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.event_loop.get_result(coro) 2022-05-14T02:10:14.304 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T02:10:14.304 INFO:journalctl@ceph.mon.b.smithi047.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T02:10:14.304 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T02:10:14.305 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.__get_result() 2022-05-14T02:10:14.305 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T02:10:14.305 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise self._exception 2022-05-14T02:10:14.305 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T02:10:14.305 INFO:journalctl@ceph.mon.b.smithi047.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T02:10:14.306 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T02:10:14.306 INFO:journalctl@ceph.mon.b.smithi047.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T02:10:14.306 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T02:10:14.306 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise 2022-05-14T02:10:14.306 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T02:10:14.307 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.gen.throw(type, value, traceback) 2022-05-14T02:10:14.307 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T02:10:14.307 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OrchestratorError(msg) 2022-05-14T02:10:14.307 INFO:journalctl@ceph.mon.b.smithi047.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:10:14.307 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:10:13 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:10:14.308 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:10:13 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:10:14.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:10:14 smithi115 conmon[90393]: 2022-05-14T02:10:14.165+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:10:15.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:10:14 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:10:15.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:10:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T02:10:15.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:10:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T02:10:15.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:10:14 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:10:15.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:10:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T02:10:15.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:10:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T02:10:15.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:10:15 smithi115 conmon[90393]: 2022-05-14T02:10:15.174+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:10:16.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:10:15 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:10:16.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:10:15 smithi115 ceph-mon[78593]: pgmap v5226: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:10:16.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:10:15 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 9751 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:10:16.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:10:15 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:10:16.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:10:15 smithi047 ceph-mon[77147]: pgmap v5226: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:10:16.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:10:15 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 9751 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:10:16.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:10:16 smithi115 conmon[90393]: 2022-05-14T02:10:16.224+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:10:17.220 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:10:16 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:10:17.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:10:16 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:10:17.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:10:17 smithi115 conmon[90393]: 2022-05-14T02:10:17.218+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:10:18.203 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:10:17 smithi115 ceph-mon[78593]: pgmap v5227: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:10:18.203 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:10:17 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:10:18.203 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:10:17 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:10:18.204 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:10:17 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:10:18.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:10:17 smithi047 ceph-mon[77147]: pgmap v5227: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:10:18.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:10:17 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:10:18.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:10:17 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:10:18.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:10:17 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:10:18.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:10:18 smithi115 conmon[90393]: 2022-05-14T02:10:18.201+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:10:19.173 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:10:18 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:10:19.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:10:18 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:10:19.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:10:19 smithi115 conmon[90393]: 2022-05-14T02:10:19.171+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:10:20.214 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:10:19 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:10:20.214 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:10:19 smithi115 ceph-mon[78593]: pgmap v5228: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:10:20.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:10:19 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:10:20.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:10:19 smithi047 ceph-mon[77147]: pgmap v5228: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:10:20.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:10:20 smithi115 conmon[90393]: 2022-05-14T02:10:20.212+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:10:21.173 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:10:20 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:10:21.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:10:20 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:10:21.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:10:21 smithi115 conmon[90393]: 2022-05-14T02:10:21.172+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:10:22.179 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:10:21 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:10:22.179 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:10:21 smithi115 ceph-mon[78593]: pgmap v5229: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:10:22.180 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:10:21 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 9755 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:10:22.180 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:10:21 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:10:22.180 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:10:21 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:10:22.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:10:21 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:10:22.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:10:21 smithi047 ceph-mon[77147]: pgmap v5229: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:10:22.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:10:21 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 9755 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:10:22.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:10:21 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:10:22.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:10:21 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:10:22.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:10:22 smithi115 conmon[90393]: 2022-05-14T02:10:22.178+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:10:23.183 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:10:22 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:10:23.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:10:22 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:10:23.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:10:23 smithi115 conmon[90393]: 2022-05-14T02:10:23.181+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:10:24.133 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:10:23 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:10:24.134 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:10:23 smithi115 ceph-mon[78593]: pgmap v5230: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:10:24.134 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:10:23 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:10:24.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:10:23 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:10:24.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:10:23 smithi047 ceph-mon[77147]: pgmap v5230: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:10:24.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:10:23 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:10:24.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:10:24 smithi115 conmon[90393]: 2022-05-14T02:10:24.132+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:10:25.149 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:10:24 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:10:25.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:10:24 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:10:25.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:10:25 smithi115 conmon[90393]: 2022-05-14T02:10:25.148+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:10:26.187 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:10:25 smithi115 ceph-mon[78593]: pgmap v5231: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:10:26.188 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:10:25 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:10:26.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:10:25 smithi047 ceph-mon[77147]: pgmap v5231: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:10:26.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:10:25 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:10:26.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:10:26 smithi115 conmon[90393]: 2022-05-14T02:10:26.185+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:10:27.148 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:10:26 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:10:27.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:10:26 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:10:27.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:10:27 smithi115 conmon[90393]: 2022-05-14T02:10:27.146+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:10:28.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:10:27 smithi047 ceph-mon[77147]: pgmap v5232: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:10:28.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:10:27 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:10:28.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:10:28 smithi115 conmon[90393]: 2022-05-14T02:10:28.109+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:10:28.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:10:27 smithi115 ceph-mon[78593]: pgmap v5232: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:10:28.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:10:27 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:10:29.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:10:28 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:10:29.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:10:28 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:10:29.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:10:29 smithi115 conmon[90393]: 2022-05-14T02:10:29.130+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:10:30.154 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:10:29 smithi115 ceph-mon[78593]: pgmap v5233: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:10:30.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:10:29 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:10:30.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:10:29 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:10:30.156 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:10:29 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:10:30.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:10:29 smithi047 ceph-mon[77147]: pgmap v5233: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:10:30.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:10:29 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:10:30.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:10:29 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:10:30.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:10:29 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:10:30.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:10:30 smithi115 conmon[90393]: 2022-05-14T02:10:30.153+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:10:31.200 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:10:30 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:10:31.201 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:10:30 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 9761 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:10:31.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:10:30 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:10:31.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:10:30 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 9761 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:10:31.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:10:31 smithi115 conmon[90393]: 2022-05-14T02:10:31.199+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:10:32.226 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:10:31 smithi115 ceph-mon[78593]: pgmap v5234: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 7 op/s 2022-05-14T02:10:32.226 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:10:31 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:10:32.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:10:31 smithi047 ceph-mon[77147]: pgmap v5234: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 7 op/s 2022-05-14T02:10:32.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:10:31 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:10:32.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:10:32 smithi115 conmon[90393]: 2022-05-14T02:10:32.225+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:10:33.206 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:10:32 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:10:33.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:10:32 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:10:33.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:10:33 smithi115 conmon[90393]: 2022-05-14T02:10:33.204+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:10:34.243 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:10:33 smithi115 ceph-mon[78593]: pgmap v5235: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 4 op/s 2022-05-14T02:10:34.243 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:10:33 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:10:34.243 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:10:33 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:10:34.244 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:10:33 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:10:34.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:10:33 smithi047 ceph-mon[77147]: pgmap v5235: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 4 op/s 2022-05-14T02:10:34.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:10:33 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:10:34.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:10:33 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:10:34.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:10:33 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:10:34.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:10:34 smithi115 conmon[90393]: 2022-05-14T02:10:34.241+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:10:35.267 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:10:34 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:10:35.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:10:34 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:10:35.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:10:35 smithi115 conmon[90393]: 2022-05-14T02:10:35.265+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:10:36.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:10:35 smithi047 ceph-mon[77147]: pgmap v5236: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 7 op/s 2022-05-14T02:10:36.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:10:35 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:10:36.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:10:35 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 9770 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:10:36.299 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:10:35 smithi115 ceph-mon[78593]: pgmap v5236: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 7 op/s 2022-05-14T02:10:36.299 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:10:35 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:10:36.299 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:10:35 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 9770 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:10:36.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:10:36 smithi115 conmon[90393]: 2022-05-14T02:10:36.297+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:10:37.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:10:36 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:10:37.334 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:10:36 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:10:37.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:10:37 smithi115 conmon[90393]: 2022-05-14T02:10:37.332+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:10:38.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:10:37 smithi047 ceph-mon[77147]: pgmap v5237: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 4 op/s 2022-05-14T02:10:38.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:10:37 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:10:38.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:10:37 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:10:38.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:10:37 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:10:38.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:10:37 smithi115 ceph-mon[78593]: pgmap v5237: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 4 op/s 2022-05-14T02:10:38.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:10:37 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:10:38.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:10:37 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:10:38.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:10:37 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:10:38.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:10:38 smithi115 conmon[90393]: 2022-05-14T02:10:38.356+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:10:39.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:10:38 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:10:39.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:10:38 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:10:39.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:10:39 smithi115 conmon[90393]: 2022-05-14T02:10:39.359+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:10:40.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:10:39 smithi047 ceph-mon[77147]: pgmap v5238: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 4 op/s 2022-05-14T02:10:40.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:10:39 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:10:40.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:10:39 smithi115 ceph-mon[78593]: pgmap v5238: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 4 op/s 2022-05-14T02:10:40.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:10:39 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:10:40.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:10:40 smithi115 conmon[90393]: 2022-05-14T02:10:40.396+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:10:41.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:10:41 smithi047 ceph-mon[77147]: pgmap v5239: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 7 op/s 2022-05-14T02:10:41.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:10:41 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:10:41.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:10:41 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 9775 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:10:41.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:10:41 smithi115 conmon[90393]: 2022-05-14T02:10:41.415+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:10:41.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:10:41 smithi115 ceph-mon[78593]: pgmap v5239: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 7 op/s 2022-05-14T02:10:41.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:10:41 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:10:41.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:10:41 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 9775 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:10:42.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:10:42 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:10:42.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:10:42 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:10:42.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:10:42 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:10:42.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:10:42 smithi115 conmon[90393]: 2022-05-14T02:10:42.372+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:10:42.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:10:42 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:10:42.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:10:42 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:10:42.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:10:42 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:10:43.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:10:43 smithi047 ceph-mon[77147]: pgmap v5240: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:10:43.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:10:43 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:10:43.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:10:43 smithi115 conmon[90393]: 2022-05-14T02:10:43.397+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:10:43.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:10:43 smithi115 ceph-mon[78593]: pgmap v5240: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:10:43.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:10:43 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:10:44.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:10:44 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:10:44.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:10:44 smithi115 conmon[90393]: 2022-05-14T02:10:44.423+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:10:44.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:10:44 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:10:45.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:10:45 smithi047 ceph-mon[77147]: pgmap v5241: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:10:45.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:10:45 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:10:45.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:10:45 smithi115 conmon[90393]: 2022-05-14T02:10:45.466+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:10:45.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:10:45 smithi115 ceph-mon[78593]: pgmap v5241: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:10:45.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:10:45 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:10:46.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:10:46 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:10:46.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:10:46 smithi115 conmon[90393]: 2022-05-14T02:10:46.479+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:10:46.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:10:46 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:10:47.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:10:47 smithi047 ceph-mon[77147]: pgmap v5242: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:10:47.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:10:47 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:10:47.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:10:47 smithi115 conmon[90393]: 2022-05-14T02:10:47.431+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:10:47.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:10:47 smithi115 ceph-mon[78593]: pgmap v5242: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:10:47.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:10:47 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:10:48.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:10:48 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:10:48.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:10:48 smithi115 conmon[90393]: 2022-05-14T02:10:48.414+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:10:48.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:10:48 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:10:49.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:10:49 smithi047 ceph-mon[77147]: pgmap v5243: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:10:49.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:10:49 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:10:49.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:10:49 smithi115 conmon[90393]: 2022-05-14T02:10:49.396+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:10:49.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:10:49 smithi115 ceph-mon[78593]: pgmap v5243: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:10:49.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:10:49 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:10:50.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:10:50 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:10:50.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:10:50 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:10:50.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:10:50 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:10:50.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:10:50 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 9780 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:10:50.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:10:50 smithi115 conmon[90393]: 2022-05-14T02:10:50.429+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:10:50.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:10:50 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:10:50.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:10:50 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:10:50.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:10:50 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:10:50.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:10:50 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 9780 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:10:51.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:10:51 smithi047 ceph-mon[77147]: pgmap v5244: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:10:51.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:10:51 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:10:51.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:10:51 smithi115 conmon[90393]: 2022-05-14T02:10:51.409+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:10:51.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:10:51 smithi115 ceph-mon[78593]: pgmap v5244: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:10:51.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:10:51 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:10:52.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:10:52 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:10:52.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:10:52 smithi115 conmon[90393]: 2022-05-14T02:10:52.456+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:10:52.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:10:52 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:10:53.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:10:53 smithi047 ceph-mon[77147]: pgmap v5245: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:10:53.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:10:53 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:10:53.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:10:53 smithi115 conmon[90393]: 2022-05-14T02:10:53.490+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:10:53.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:10:53 smithi115 ceph-mon[78593]: pgmap v5245: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:10:53.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:10:53 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:10:54.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:10:54 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:10:54.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:10:54 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:10:54.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:10:54 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:10:54.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:10:54 smithi115 conmon[90393]: 2022-05-14T02:10:54.453+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:10:54.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:10:54 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:10:54.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:10:54 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:10:54.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:10:54 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:10:55.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:10:55 smithi047 ceph-mon[77147]: pgmap v5246: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:10:55.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:10:55 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:10:55.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:10:55 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 9790 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:10:55.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:10:55 smithi115 conmon[90393]: 2022-05-14T02:10:55.442+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:10:55.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:10:55 smithi115 ceph-mon[78593]: pgmap v5246: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:10:55.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:10:55 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:10:55.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:10:55 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 9790 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:10:56.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:10:56 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:10:56.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:10:56 smithi115 conmon[90393]: 2022-05-14T02:10:56.400+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:10:56.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:10:56 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:10:57.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:10:57 smithi047 ceph-mon[77147]: pgmap v5247: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:10:57.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:10:57 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:10:57.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:10:57 smithi115 conmon[90393]: 2022-05-14T02:10:57.432+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:10:57.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:10:57 smithi115 ceph-mon[78593]: pgmap v5247: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:10:57.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:10:57 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:10:58.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:10:58 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:10:58.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:10:58 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:10:58.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:10:58 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:10:58.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:10:58 smithi115 conmon[90393]: 2022-05-14T02:10:58.391+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:10:58.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:10:58 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:10:58.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:10:58 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:10:58.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:10:58 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:10:59.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:10:59 smithi047 ceph-mon[77147]: pgmap v5248: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:10:59.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:10:59 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:10:59.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:10:59 smithi115 conmon[90393]: 2022-05-14T02:10:59.441+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:10:59.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:10:59 smithi115 ceph-mon[78593]: pgmap v5248: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:10:59.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:10:59 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:11:00.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:11:00 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:11:00.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:11:00 smithi115 conmon[90393]: 2022-05-14T02:11:00.392+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:11:00.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:11:00 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:11:01.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:11:01 smithi047 ceph-mon[77147]: pgmap v5249: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:11:01.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:11:01 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:11:01.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:11:01 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 9795 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:11:01.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:11:01 smithi115 conmon[90393]: 2022-05-14T02:11:01.410+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:11:01.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:11:01 smithi115 ceph-mon[78593]: pgmap v5249: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:11:01.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:11:01 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:11:01.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:11:01 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 9795 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:11:02.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:11:02 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:11:02.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:11:02 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:11:02.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:11:02 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:11:02.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:11:02 smithi115 conmon[90393]: 2022-05-14T02:11:02.445+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:11:02.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:11:02 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:11:02.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:11:02 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:11:02.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:11:02 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:11:03.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:11:03 smithi047 ceph-mon[77147]: pgmap v5250: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:11:03.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:11:03 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:11:03.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:11:03 smithi115 conmon[90393]: 2022-05-14T02:11:03.419+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:11:03.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:11:03 smithi115 ceph-mon[78593]: pgmap v5250: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:11:03.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:11:03 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:11:04.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:11:04 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:11:04.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:11:04 smithi115 conmon[90393]: 2022-05-14T02:11:04.438+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:11:04.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:11:04 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:11:05.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:11:05 smithi047 ceph-mon[77147]: pgmap v5251: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:11:05.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:11:05 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:11:05.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:11:05 smithi115 conmon[90393]: 2022-05-14T02:11:05.461+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:11:05.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:11:05 smithi115 ceph-mon[78593]: pgmap v5251: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:11:05.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:11:05 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:11:06.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:11:06 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:11:06.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:11:06 smithi115 conmon[90393]: 2022-05-14T02:11:06.488+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:11:06.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:11:06 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:11:07.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:11:07 smithi047 ceph-mon[77147]: pgmap v5252: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:11:07.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:11:07 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:11:07.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:11:07 smithi115 conmon[90393]: 2022-05-14T02:11:07.523+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:11:07.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:11:07 smithi115 ceph-mon[78593]: pgmap v5252: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:11:07.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:11:07 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:11:08.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:11:08 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:11:08.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:11:08 smithi115 conmon[90393]: 2022-05-14T02:11:08.540+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:11:08.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:11:08 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:11:09.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:11:09 smithi047 ceph-mon[77147]: pgmap v5253: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:11:09.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:11:09 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:11:09.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:11:09 smithi115 conmon[90393]: 2022-05-14T02:11:09.530+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:11:09.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:11:09 smithi115 ceph-mon[78593]: pgmap v5253: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:11:09.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:11:09 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:11:10.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:11:10 smithi115 conmon[90393]: 2022-05-14T02:11:10.542+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:11:10.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:11:10 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:11:10.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:11:10 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:11:10.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:11:10 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:11:10.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:11:10 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 9800 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:11:11.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:11:10 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:11:11.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:11:10 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:11:11.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:11:10 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:11:11.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:11:10 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 9800 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:11:11.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:11:11 smithi115 conmon[90393]: 2022-05-14T02:11:11.510+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:11:11.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:11:11 smithi115 ceph-mon[78593]: pgmap v5254: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:11:11.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:11:11 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:11:12.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:11:11 smithi047 ceph-mon[77147]: pgmap v5254: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:11:12.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:11:11 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:11:12.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:11:12 smithi115 conmon[90393]: 2022-05-14T02:11:12.511+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:11:12.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:11:12 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:11:13.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:11:12 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:11:13.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:11:13 smithi115 conmon[90393]: 2022-05-14T02:11:13.480+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:11:13.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:11:13 smithi115 ceph-mon[78593]: pgmap v5255: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:11:13.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:11:13 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:11:13.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:11:13 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T02:11:13.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:11:13 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T02:11:13.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:11:13 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T02:11:13.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:11:13 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' 2022-05-14T02:11:13.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:11:13 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' 2022-05-14T02:11:14.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:11:13 smithi047 ceph-mon[77147]: pgmap v5255: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:11:14.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:11:13 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:11:14.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:11:13 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T02:11:14.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:11:13 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T02:11:14.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:11:13 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T02:11:14.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:11:13 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' 2022-05-14T02:11:14.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:11:13 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' 2022-05-14T02:11:14.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:11:14 smithi115 conmon[90393]: 2022-05-14T02:11:14.478+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:11:14.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:11:14 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:11:14.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:11:14 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:11:14.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:11:14 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:11:14.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:11:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T02:11:14.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:11:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T02:11:15.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:11:14 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:11:15.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:11:14 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:11:15.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:11:14 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:11:15.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:11:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T02:11:15.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:11:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T02:11:15.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:11:15 smithi115 conmon[90393]: 2022-05-14T02:11:15.492+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:11:15.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:11:15 smithi115 ceph-mon[78593]: pgmap v5256: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:11:15.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:11:15 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:11:15.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:11:15 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 9810 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:11:16.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:11:15 smithi047 ceph-mon[77147]: pgmap v5256: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:11:16.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:11:15 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:11:16.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:11:15 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 9810 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:11:16.042 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:11:15 smithi047 conmon[78521]: 2022-05-14T02:11:15.695+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:11:16.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:11:15 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T02:11:16.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:11:15 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:11:16.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:11:15 smithi047 conmon[78521]: yield 2022-05-14T02:11:16.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:11:15 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:11:16.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:11:15 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:11:16.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:11:15 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:11:16.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:11:15 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T02:11:16.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:11:15 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:11:16.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:11:15 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T02:11:16.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:11:15 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:11:16.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:11:15 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T02:11:16.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:11:15 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:11:16.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:11:15 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T02:11:16.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:11:15 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:11:16.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:11:15 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T02:11:16.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:11:15 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:11:16.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:11:15 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:11:16.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:11:15 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:11:16.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:11:16 smithi115 conmon[90393]: 2022-05-14T02:11:16.467+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:11:16.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:11:16 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:11:16.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:11:16 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' 2022-05-14T02:11:16.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:11:16 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' 2022-05-14T02:11:16.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:11:16 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:11:16.860 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T02:11:16.860 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:11:16.860 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T02:11:16.860 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:11:16.860 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:11:16.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:11:16.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T02:11:16.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:11:16.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T02:11:16.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:11:16.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T02:11:16.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:11:16.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T02:11:16.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:11:16.863 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T02:11:16.863 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:11:16.863 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:11:16.863 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:11:16.863 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:11:16 smithi115 ceph-mon[78593]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T02:11:16.864 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:11:16 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-05-14T02:11:16.864 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:11:16 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-05-14T02:11:16.864 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:11:16 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-05-14T02:11:16.864 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:11:16 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-05-14T02:11:16.865 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:11:16 smithi115 ceph-mon[78593]: Adjusting osd_memory_target on smithi115 to 197.0M 2022-05-14T02:11:16.865 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:11:16 smithi115 ceph-mon[78593]: Unable to set osd_memory_target on smithi115 to 206629785: error parsing value: Value '206629785' is below minimum 939524096 2022-05-14T02:11:16.865 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:11:16 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-05-14T02:11:16.865 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:11:16 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-05-14T02:11:16.865 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:11:16 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-05-14T02:11:16.866 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:11:16 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-05-14T02:11:17.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:11:16 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:11:17.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:11:16 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' 2022-05-14T02:11:17.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:11:16 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' 2022-05-14T02:11:17.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:11:16 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:11:17.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T02:11:17.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:11:17.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T02:11:17.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:11:17.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:11:17.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:11:17.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T02:11:17.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:11:17.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T02:11:17.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:11:17.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T02:11:17.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:11:17.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T02:11:17.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:11:17.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T02:11:17.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:11:17.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:11:17.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:11:17.047 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:11:16 smithi047 ceph-mon[77147]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T02:11:17.047 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:11:16 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-05-14T02:11:17.047 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:11:16 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-05-14T02:11:17.047 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:11:16 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-05-14T02:11:17.047 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:11:16 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-05-14T02:11:17.048 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:11:16 smithi047 ceph-mon[77147]: Adjusting osd_memory_target on smithi115 to 197.0M 2022-05-14T02:11:17.048 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:11:16 smithi047 ceph-mon[77147]: Unable to set osd_memory_target on smithi115 to 206629785: error parsing value: Value '206629785' is below minimum 939524096 2022-05-14T02:11:17.048 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:11:16 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-05-14T02:11:17.048 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:11:16 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-05-14T02:11:17.049 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:11:16 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-05-14T02:11:17.049 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:11:16 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-05-14T02:11:17.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:11:17 smithi115 conmon[90393]: 2022-05-14T02:11:17.465+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:11:17.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:11:17 smithi115 ceph-mon[78593]: Adjusting osd_memory_target on smithi047 to 198.1M 2022-05-14T02:11:17.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:11:17 smithi115 ceph-mon[78593]: Unable to set osd_memory_target on smithi047 to 207755878: error parsing value: Value '207755878' is below minimum 939524096 2022-05-14T02:11:17.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:11:17 smithi115 ceph-mon[78593]: pgmap v5257: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:11:17.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:11:17 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:11:18.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:11:17 smithi047 ceph-mon[77147]: Adjusting osd_memory_target on smithi047 to 198.1M 2022-05-14T02:11:18.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:11:17 smithi047 ceph-mon[77147]: Unable to set osd_memory_target on smithi047 to 207755878: error parsing value: Value '207755878' is below minimum 939524096 2022-05-14T02:11:18.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:11:17 smithi047 ceph-mon[77147]: pgmap v5257: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:11:18.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:11:17 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:11:18.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:11:18 smithi115 conmon[90393]: 2022-05-14T02:11:18.482+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:11:18.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:11:18 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:11:18.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:11:18 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:11:18.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:11:18 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:11:19.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:11:18 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:11:19.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:11:18 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:11:19.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:11:18 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:11:19.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:11:18 smithi047 conmon[78521]: 2022-05-14T02:11:18.767+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:11:19.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:11:18 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T02:11:19.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:11:18 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:11:19.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:11:18 smithi047 conmon[78521]: yield 2022-05-14T02:11:19.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:11:18 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:11:19.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:11:18 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:11:19.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:11:18 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:11:19.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:11:18 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T02:11:19.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:11:18 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:11:19.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:11:18 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T02:11:19.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:11:18 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:11:19.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:11:18 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T02:11:19.048 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:11:18 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:11:19.048 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:11:18 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T02:11:19.048 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:11:18 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:11:19.048 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:11:18 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T02:11:19.049 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:11:18 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:11:19.049 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:11:18 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:11:19.049 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:11:18 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:11:19.049 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:11:18 smithi047 conmon[78521]: 2022-05-14T02:11:18.768+0000 7f5340779700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T02:11:19.050 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:11:18 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T02:11:19.050 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:11:18 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:11:19.050 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:11:18 smithi047 conmon[78521]: yield 2022-05-14T02:11:19.051 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:11:18 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:11:19.051 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:11:18 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:11:19.052 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:11:18 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:11:19.052 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:11:18 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T02:11:19.052 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:11:18 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:11:19.052 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:11:18 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T02:11:19.052 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:11:18 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:11:19.053 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:11:18 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T02:11:19.053 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:11:18 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:11:19.053 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:11:18 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T02:11:19.053 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:11:18 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:11:19.054 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:11:18 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T02:11:19.054 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:11:18 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:11:19.054 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:11:18 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:11:19.054 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:11:18 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:11:19.054 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:11:18 smithi047 conmon[78521]: 2022-05-14T02:11:19.055 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:11:18 smithi047 conmon[78521]: During handling of the above exception, another exception occurred: 2022-05-14T02:11:19.055 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:11:18 smithi047 conmon[78521]: 2022-05-14T02:11:19.055 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:11:18 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T02:11:19.055 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:11:18 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T02:11:19.056 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:11:18 smithi047 conmon[78521]: return f(*arg) 2022-05-14T02:11:19.056 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:11:18 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T02:11:19.056 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:11:18 smithi047 conmon[78521]: self._write_client_files(client_files, host) 2022-05-14T02:11:19.057 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:11:18 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T02:11:19.057 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:11:18 smithi047 conmon[78521]: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T02:11:19.058 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:11:18 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T02:11:19.058 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:11:18 smithi047 conmon[78521]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T02:11:19.058 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:11:18 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T02:11:19.059 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:11:18 smithi047 conmon[78521]: return self.event_loop.get_result(coro) 2022-05-14T02:11:19.059 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:11:18 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T02:11:19.060 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:11:18 smithi047 conmon[78521]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T02:11:19.060 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:11:18 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T02:11:19.060 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:11:18 smithi047 conmon[78521]: return self.__get_result() 2022-05-14T02:11:19.060 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:11:18 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T02:11:19.061 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:11:18 smithi047 conmon[78521]: raise self._exception 2022-05-14T02:11:19.061 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:11:18 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T02:11:19.061 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:11:18 smithi047 conmon[78521]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T02:11:19.062 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:11:18 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T02:11:19.062 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:11:18 smithi047 conmon[78521]: conn = await self._remote_connection(host, addr) 2022-05-14T02:11:19.062 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:11:18 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T02:11:19.062 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:11:18 smithi047 conmon[78521]: raise 2022-05-14T02:11:19.062 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:11:18 smithi047 conmon[78521]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T02:11:19.063 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:11:18 smithi047 conmon[78521]: self.gen.throw(type, value, traceback) 2022-05-14T02:11:19.063 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:11:18 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T02:11:19.063 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:11:18 smithi047 conmon[78521]: raise OrchestratorError(msg) 2022-05-14T02:11:19.063 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:11:18 smithi047 conmon[78521]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:11:19.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:11:19 smithi115 conmon[90393]: 2022-05-14T02:11:19.486+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:11:19.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:11:19 smithi115 ceph-mon[78593]: pgmap v5258: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:11:19.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:11:19 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:11:19.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:11:19 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:11:19.859 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T02:11:19.860 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:11:19.860 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T02:11:19.860 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:11:19.860 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:11:19.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:11:19.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T02:11:19.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:11:19.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T02:11:19.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:11:19.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T02:11:19.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:11:19.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T02:11:19.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:11:19.863 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T02:11:19.863 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:11:19.863 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:11:19.863 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:11:19.863 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:11:19 smithi115 ceph-mon[78593]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T02:11:19.864 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T02:11:19.864 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:11:19.864 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T02:11:19.864 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:11:19.865 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:11:19.865 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:11:19.865 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T02:11:19.865 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:11:19.865 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T02:11:19.866 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:11:19.866 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T02:11:19.866 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:11:19.866 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T02:11:19.866 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:11:19.867 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T02:11:19.867 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:11:19.867 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:11:19.867 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:11:19.868 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T02:11:19.868 INFO:journalctl@ceph.mon.c.smithi115.stdout: During handling of the above exception, another exception occurred: 2022-05-14T02:11:19.868 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T02:11:19.868 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T02:11:19.868 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T02:11:19.869 INFO:journalctl@ceph.mon.c.smithi115.stdout: return f(*arg) 2022-05-14T02:11:19.869 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T02:11:19.869 INFO:journalctl@ceph.mon.c.smithi115.stdout: self._write_client_files(client_files, host) 2022-05-14T02:11:19.869 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T02:11:19.870 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T02:11:19.870 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T02:11:19.870 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T02:11:19.870 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T02:11:19.870 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.event_loop.get_result(coro) 2022-05-14T02:11:19.871 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T02:11:19.871 INFO:journalctl@ceph.mon.c.smithi115.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T02:11:19.871 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T02:11:19.871 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.__get_result() 2022-05-14T02:11:19.871 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T02:11:19.872 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise self._exception 2022-05-14T02:11:19.872 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T02:11:19.872 INFO:journalctl@ceph.mon.c.smithi115.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T02:11:19.872 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T02:11:19.873 INFO:journalctl@ceph.mon.c.smithi115.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T02:11:19.873 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T02:11:19.873 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise 2022-05-14T02:11:19.873 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T02:11:19.873 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.gen.throw(type, value, traceback) 2022-05-14T02:11:19.874 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T02:11:19.874 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OrchestratorError(msg) 2022-05-14T02:11:19.874 INFO:journalctl@ceph.mon.c.smithi115.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:11:20.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:11:19 smithi047 ceph-mon[77147]: pgmap v5258: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:11:20.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:11:19 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:11:20.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:11:19 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:11:20.042 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T02:11:20.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:11:20.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T02:11:20.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:11:20.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:11:20.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:11:20.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T02:11:20.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:11:20.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T02:11:20.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:11:20.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T02:11:20.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:11:20.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T02:11:20.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:11:20.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T02:11:20.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:11:20.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:11:20.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:11:20.046 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:11:19 smithi047 ceph-mon[77147]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T02:11:20.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T02:11:20.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:11:20.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T02:11:20.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:11:20.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:11:20.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:11:20.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T02:11:20.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:11:20.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T02:11:20.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:11:20.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T02:11:20.049 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:11:20.049 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T02:11:20.049 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:11:20.049 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T02:11:20.049 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:11:20.050 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:11:20.050 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:11:20.050 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T02:11:20.050 INFO:journalctl@ceph.mon.b.smithi047.stdout: During handling of the above exception, another exception occurred: 2022-05-14T02:11:20.050 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T02:11:20.051 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T02:11:20.051 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T02:11:20.051 INFO:journalctl@ceph.mon.b.smithi047.stdout: return f(*arg) 2022-05-14T02:11:20.051 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T02:11:20.051 INFO:journalctl@ceph.mon.b.smithi047.stdout: self._write_client_files(client_files, host) 2022-05-14T02:11:20.052 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T02:11:20.052 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T02:11:20.052 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T02:11:20.052 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T02:11:20.052 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T02:11:20.053 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.event_loop.get_result(coro) 2022-05-14T02:11:20.053 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T02:11:20.053 INFO:journalctl@ceph.mon.b.smithi047.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T02:11:20.053 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T02:11:20.053 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.__get_result() 2022-05-14T02:11:20.054 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T02:11:20.054 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise self._exception 2022-05-14T02:11:20.054 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T02:11:20.054 INFO:journalctl@ceph.mon.b.smithi047.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T02:11:20.054 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T02:11:20.055 INFO:journalctl@ceph.mon.b.smithi047.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T02:11:20.055 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T02:11:20.055 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise 2022-05-14T02:11:20.055 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T02:11:20.055 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.gen.throw(type, value, traceback) 2022-05-14T02:11:20.056 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T02:11:20.056 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OrchestratorError(msg) 2022-05-14T02:11:20.056 INFO:journalctl@ceph.mon.b.smithi047.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:11:20.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:11:20 smithi115 conmon[90393]: 2022-05-14T02:11:20.515+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:11:20.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:11:20 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:11:21.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:11:20 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:11:21.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:11:21 smithi115 conmon[90393]: 2022-05-14T02:11:21.532+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:11:21.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:11:21 smithi115 ceph-mon[78593]: pgmap v5259: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:11:21.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:11:21 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:11:21.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:11:21 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 9815 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:11:22.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:11:21 smithi047 ceph-mon[77147]: pgmap v5259: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:11:22.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:11:21 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:11:22.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:11:21 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 9815 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:11:22.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:11:22 smithi115 conmon[90393]: 2022-05-14T02:11:22.565+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:11:22.919 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:11:22 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:11:22.920 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:11:22 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:11:22.920 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:11:22 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:11:23.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:11:22 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:11:23.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:11:22 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:11:23.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:11:22 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:11:23.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:11:23 smithi115 conmon[90393]: 2022-05-14T02:11:23.597+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:11:23.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:11:23 smithi115 ceph-mon[78593]: pgmap v5260: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:11:23.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:11:23 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:11:24.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:11:23 smithi047 ceph-mon[77147]: pgmap v5260: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:11:24.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:11:23 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:11:24.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:11:24 smithi115 conmon[90393]: 2022-05-14T02:11:24.564+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:11:24.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:11:24 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:11:25.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:11:24 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:11:25.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:11:25 smithi115 conmon[90393]: 2022-05-14T02:11:25.563+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:11:25.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:11:25 smithi115 ceph-mon[78593]: pgmap v5261: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:11:25.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:11:25 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:11:26.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:11:25 smithi047 ceph-mon[77147]: pgmap v5261: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:11:26.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:11:25 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:11:27.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:11:26 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:11:27.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:11:26 smithi115 conmon[90393]: 2022-05-14T02:11:26.610+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:11:27.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:11:26 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:11:28.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:11:27 smithi047 ceph-mon[77147]: pgmap v5262: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:11:28.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:11:27 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:11:28.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:11:27 smithi115 conmon[90393]: 2022-05-14T02:11:27.635+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:11:28.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:11:27 smithi115 ceph-mon[78593]: pgmap v5262: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:11:28.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:11:27 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:11:28.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:11:28 smithi115 conmon[90393]: 2022-05-14T02:11:28.587+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:11:28.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:11:28 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:11:29.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:11:28 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:11:30.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:11:29 smithi047 ceph-mon[77147]: pgmap v5263: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:11:30.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:11:29 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:11:30.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:11:29 smithi115 conmon[90393]: 2022-05-14T02:11:29.629+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:11:30.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:11:29 smithi115 ceph-mon[78593]: pgmap v5263: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:11:30.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:11:29 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:11:31.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:11:30 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:11:31.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:11:30 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:11:31.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:11:30 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:11:31.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:11:30 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 9820 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:11:31.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:11:30 smithi115 conmon[90393]: 2022-05-14T02:11:30.614+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:11:31.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:11:30 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:11:31.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:11:30 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:11:31.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:11:30 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:11:31.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:11:30 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 9820 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:11:32.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:11:31 smithi047 ceph-mon[77147]: pgmap v5264: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:11:32.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:11:31 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:11:32.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:11:31 smithi115 conmon[90393]: 2022-05-14T02:11:31.618+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:11:32.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:11:31 smithi115 ceph-mon[78593]: pgmap v5264: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:11:32.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:11:31 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:11:32.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:11:32 smithi115 conmon[90393]: 2022-05-14T02:11:32.598+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:11:32.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:11:32 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:11:33.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:11:32 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:11:34.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:11:33 smithi047 ceph-mon[77147]: pgmap v5265: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:11:34.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:11:33 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:11:34.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:11:33 smithi115 conmon[90393]: 2022-05-14T02:11:33.641+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:11:34.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:11:33 smithi115 ceph-mon[78593]: pgmap v5265: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:11:34.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:11:33 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:11:35.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:11:34 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:11:35.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:11:34 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:11:35.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:11:34 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:11:35.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:11:34 smithi115 conmon[90393]: 2022-05-14T02:11:34.672+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:11:35.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:11:34 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:11:35.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:11:34 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:11:35.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:11:34 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:11:36.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:11:35 smithi047 ceph-mon[77147]: pgmap v5266: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:11:36.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:11:35 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:11:36.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:11:35 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 9830 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:11:36.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:11:35 smithi115 conmon[90393]: 2022-05-14T02:11:35.635+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:11:36.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:11:35 smithi115 ceph-mon[78593]: pgmap v5266: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:11:36.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:11:35 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:11:36.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:11:35 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 9830 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:11:36.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:11:36 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:11:36.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:11:36 smithi115 conmon[90393]: 2022-05-14T02:11:36.591+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:11:37.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:11:36 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:11:38.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:11:37 smithi047 ceph-mon[77147]: pgmap v5267: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:11:38.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:11:37 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:11:38.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:11:37 smithi115 conmon[90393]: 2022-05-14T02:11:37.628+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:11:38.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:11:37 smithi115 ceph-mon[78593]: pgmap v5267: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:11:38.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:11:37 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:11:39.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:11:38 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:11:39.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:11:38 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:11:39.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:11:38 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:11:39.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:11:38 smithi115 conmon[90393]: 2022-05-14T02:11:38.613+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:11:39.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:11:38 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:11:39.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:11:38 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:11:39.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:11:38 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:11:40.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:11:39 smithi047 ceph-mon[77147]: pgmap v5268: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:11:40.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:11:39 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:11:40.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:11:39 smithi115 conmon[90393]: 2022-05-14T02:11:39.647+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:11:40.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:11:39 smithi115 ceph-mon[78593]: pgmap v5268: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:11:40.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:11:39 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:11:40.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:11:40 smithi115 conmon[90393]: 2022-05-14T02:11:40.604+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:11:40.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:11:40 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:11:41.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:11:40 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:11:41.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:11:41 smithi115 conmon[90393]: 2022-05-14T02:11:41.573+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:11:41.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:11:41 smithi115 ceph-mon[78593]: pgmap v5269: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:11:41.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:11:41 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:11:41.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:11:41 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 9835 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:11:42.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:11:41 smithi047 ceph-mon[77147]: pgmap v5269: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:11:42.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:11:41 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:11:42.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:11:41 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 9835 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:11:42.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:11:42 smithi115 conmon[90393]: 2022-05-14T02:11:42.562+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:11:42.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:11:42 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:11:42.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:11:42 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:11:42.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:11:42 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:11:43.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:11:42 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:11:43.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:11:42 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:11:43.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:11:42 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:11:43.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:11:43 smithi115 conmon[90393]: 2022-05-14T02:11:43.552+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:11:43.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:11:43 smithi115 ceph-mon[78593]: pgmap v5270: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:11:43.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:11:43 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:11:44.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:11:43 smithi047 ceph-mon[77147]: pgmap v5270: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:11:44.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:11:43 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:11:44.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:11:44 smithi115 conmon[90393]: 2022-05-14T02:11:44.571+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:11:44.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:11:44 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:11:45.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:11:44 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:11:45.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:11:45 smithi115 conmon[90393]: 2022-05-14T02:11:45.540+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:11:45.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:11:45 smithi115 ceph-mon[78593]: pgmap v5271: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:11:45.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:11:45 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:11:46.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:11:45 smithi047 ceph-mon[77147]: pgmap v5271: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:11:46.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:11:45 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:11:46.793 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:11:46 smithi115 conmon[90393]: 2022-05-14T02:11:46.537+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:11:47.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:11:46 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:11:47.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:11:46 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:11:47.792 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:11:47 smithi115 conmon[90393]: 2022-05-14T02:11:47.531+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:11:48.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:11:47 smithi115 ceph-mon[78593]: pgmap v5272: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:11:48.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:11:47 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:11:48.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:11:47 smithi047 ceph-mon[77147]: pgmap v5272: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:11:48.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:11:47 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:11:48.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:11:48 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:11:48.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:11:48 smithi115 conmon[90393]: 2022-05-14T02:11:48.551+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:11:49.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:11:48 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:11:49.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:11:49 smithi115 conmon[90393]: 2022-05-14T02:11:49.564+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:11:49.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:11:49 smithi115 ceph-mon[78593]: pgmap v5273: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:11:49.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:11:49 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:11:50.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:11:49 smithi047 ceph-mon[77147]: pgmap v5273: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:11:50.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:11:49 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:11:50.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:11:50 smithi115 conmon[90393]: 2022-05-14T02:11:50.576+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:11:50.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:11:50 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:11:50.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:11:50 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:11:50.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:11:50 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:11:50.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:11:50 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 9840 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:11:51.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:11:50 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:11:51.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:11:50 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:11:51.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:11:50 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:11:51.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:11:50 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 9840 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:11:51.807 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:11:51 smithi115 conmon[90393]: 2022-05-14T02:11:51.548+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:11:52.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:11:51 smithi115 ceph-mon[78593]: pgmap v5274: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:11:52.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:11:51 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:11:52.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:11:51 smithi047 ceph-mon[77147]: pgmap v5274: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:11:52.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:11:51 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:11:52.813 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:11:52 smithi115 conmon[90393]: 2022-05-14T02:11:52.513+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:11:53.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:11:52 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:11:53.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:11:52 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:11:53.814 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:11:53 smithi115 conmon[90393]: 2022-05-14T02:11:53.494+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:11:54.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:11:53 smithi115 ceph-mon[78593]: pgmap v5275: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:11:54.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:11:53 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:11:54.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:11:53 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:11:54.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:11:53 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:11:54.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:11:53 smithi047 ceph-mon[77147]: pgmap v5275: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:11:54.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:11:53 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:11:54.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:11:53 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:11:54.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:11:53 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:11:54.819 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:11:54 smithi115 conmon[90393]: 2022-05-14T02:11:54.457+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:11:55.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:11:54 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:11:55.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:11:54 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:11:55.821 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:11:55 smithi115 conmon[90393]: 2022-05-14T02:11:55.481+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:11:56.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:11:55 smithi115 ceph-mon[78593]: pgmap v5276: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:11:56.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:11:55 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:11:56.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:11:55 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 9850 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:11:56.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:11:55 smithi047 ceph-mon[77147]: pgmap v5276: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:11:56.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:11:55 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:11:56.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:11:55 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 9850 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:11:56.828 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:11:56 smithi115 conmon[90393]: 2022-05-14T02:11:56.523+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:11:57.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:11:56 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:11:57.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:11:56 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:11:57.827 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:11:57 smithi115 conmon[90393]: 2022-05-14T02:11:57.568+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:11:58.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:11:57 smithi115 ceph-mon[78593]: pgmap v5277: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:11:58.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:11:57 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:11:58.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:11:57 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:11:58.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:11:57 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:11:58.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:11:57 smithi047 ceph-mon[77147]: pgmap v5277: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:11:58.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:11:57 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:11:58.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:11:57 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:11:58.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:11:57 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:11:58.828 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:11:58 smithi115 conmon[90393]: 2022-05-14T02:11:58.544+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:11:59.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:11:58 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:11:59.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:11:58 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:11:59.834 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:11:59 smithi115 conmon[90393]: 2022-05-14T02:11:59.525+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:12:00.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:11:59 smithi115 ceph-mon[78593]: pgmap v5278: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:12:00.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:11:59 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:12:00.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:11:59 smithi047 ceph-mon[77147]: pgmap v5278: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:12:00.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:11:59 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:12:00.839 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:12:00 smithi115 conmon[90393]: 2022-05-14T02:12:00.554+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:12:01.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:12:00 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:12:01.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:12:00 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:12:01.840 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:12:01 smithi115 conmon[90393]: 2022-05-14T02:12:01.534+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:12:02.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:12:01 smithi115 ceph-mon[78593]: pgmap v5279: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:12:02.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:12:01 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:12:02.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:12:01 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 9855 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:12:02.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:12:01 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:12:02.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:12:01 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:12:02.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:12:01 smithi047 ceph-mon[77147]: pgmap v5279: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:12:02.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:12:01 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:12:02.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:12:01 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 9855 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:12:02.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:12:01 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:12:02.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:12:01 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:12:02.846 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:12:02 smithi115 conmon[90393]: 2022-05-14T02:12:02.569+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:12:03.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:12:02 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:12:03.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:12:02 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:12:03.847 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:12:03 smithi115 conmon[90393]: 2022-05-14T02:12:03.520+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:12:04.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:12:03 smithi115 ceph-mon[78593]: pgmap v5280: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:12:04.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:12:03 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:12:04.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:12:03 smithi047 ceph-mon[77147]: pgmap v5280: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:12:04.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:12:03 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:12:04.854 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:12:04 smithi115 conmon[90393]: 2022-05-14T02:12:04.543+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:12:05.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:12:04 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:12:05.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:12:04 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:12:05.853 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:12:05 smithi115 conmon[90393]: 2022-05-14T02:12:05.514+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:12:06.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:12:05 smithi115 ceph-mon[78593]: pgmap v5281: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:12:06.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:12:05 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:12:06.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:12:05 smithi047 ceph-mon[77147]: pgmap v5281: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:12:06.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:12:05 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:12:06.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:12:06 smithi115 conmon[90393]: 2022-05-14T02:12:06.517+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:12:07.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:12:06 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:12:07.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:12:06 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:12:07.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:12:07 smithi115 conmon[90393]: 2022-05-14T02:12:07.548+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:12:08.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:12:07 smithi047 ceph-mon[77147]: pgmap v5282: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:12:08.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:12:07 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:12:08.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:12:07 smithi115 ceph-mon[78593]: pgmap v5282: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:12:08.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:12:07 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:12:08.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:12:08 smithi115 conmon[90393]: 2022-05-14T02:12:08.554+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:12:09.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:12:08 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:12:09.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:12:08 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:12:09.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:12:09 smithi115 conmon[90393]: 2022-05-14T02:12:09.587+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:12:10.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:12:09 smithi047 ceph-mon[77147]: pgmap v5283: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:12:10.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:12:09 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:12:10.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:12:09 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:12:10.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:12:09 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:12:10.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:12:09 smithi115 ceph-mon[78593]: pgmap v5283: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:12:10.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:12:09 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:12:10.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:12:09 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:12:10.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:12:09 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:12:10.876 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:12:10 smithi115 conmon[90393]: 2022-05-14T02:12:10.609+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:12:11.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:12:10 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:12:11.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:12:10 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 9860 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:12:11.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:12:10 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:12:11.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:12:10 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 9860 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:12:12.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:12:11 smithi115 conmon[90393]: 2022-05-14T02:12:11.643+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:12:12.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:12:11 smithi115 ceph-mon[78593]: pgmap v5284: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:12:12.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:12:11 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:12:12.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:12:11 smithi047 ceph-mon[77147]: pgmap v5284: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:12:12.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:12:11 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:12:12.882 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:12:12 smithi115 conmon[90393]: 2022-05-14T02:12:12.609+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:12:13.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:12:12 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:12:13.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:12:12 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:12:14.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:12:13 smithi115 conmon[90393]: 2022-05-14T02:12:13.651+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:12:14.139 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:12:13 smithi115 ceph-mon[78593]: pgmap v5285: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:12:14.139 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:12:13 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:12:14.139 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:12:13 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:12:14.139 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:12:13 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:12:14.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:12:13 smithi047 ceph-mon[77147]: pgmap v5285: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:12:14.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:12:13 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:12:14.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:12:13 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:12:14.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:12:13 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:12:15.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:12:14 smithi115 conmon[90393]: 2022-05-14T02:12:14.668+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:12:15.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:12:14 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:12:15.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:12:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T02:12:15.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:12:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T02:12:15.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:12:14 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:12:15.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:12:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T02:12:15.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:12:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T02:12:16.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:12:15 smithi115 conmon[90393]: 2022-05-14T02:12:15.641+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:12:16.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:12:15 smithi115 ceph-mon[78593]: pgmap v5286: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:12:16.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:12:15 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:12:16.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:12:15 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 9870 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:12:16.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:12:15 smithi047 ceph-mon[77147]: pgmap v5286: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:12:16.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:12:15 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:12:16.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:12:15 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 9870 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:12:16.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:12:16 smithi115 conmon[90393]: 2022-05-14T02:12:16.599+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:12:17.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:12:16 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:12:17.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:12:16 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:12:17.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:12:17 smithi115 conmon[90393]: 2022-05-14T02:12:17.576+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:12:18.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:12:17 smithi047 ceph-mon[77147]: pgmap v5287: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:12:18.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:12:17 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:12:18.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:12:17 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:12:18.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:12:17 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:12:18.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:12:17 smithi115 ceph-mon[78593]: pgmap v5287: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:12:18.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:12:17 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:12:18.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:12:17 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:12:18.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:12:17 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:12:18.847 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:12:18 smithi115 conmon[90393]: 2022-05-14T02:12:18.571+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:12:19.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:12:18 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:12:19.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:12:18 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T02:12:19.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:12:18 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T02:12:19.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:12:18 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T02:12:19.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:12:18 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:12:19.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:12:18 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T02:12:19.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:12:18 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T02:12:19.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:12:18 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T02:12:19.791 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:12:19 smithi047 conmon[78521]: 2022-05-14T02:12:19.375+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:12:19.792 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:12:19 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T02:12:19.792 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:12:19 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:12:19.792 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:12:19 smithi047 conmon[78521]: yield 2022-05-14T02:12:19.793 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:12:19 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:12:19.793 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:12:19 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:12:19.793 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:12:19 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:12:19.793 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:12:19 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T02:12:19.793 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:12:19 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:12:19.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:12:19 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T02:12:19.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:12:19 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:12:19.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:12:19 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T02:12:19.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:12:19 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:12:19.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:12:19 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T02:12:19.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:12:19 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:12:19.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:12:19 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T02:12:19.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:12:19 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:12:19.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:12:19 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:12:19.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:12:19 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:12:19.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:12:19 smithi115 conmon[90393]: 2022-05-14T02:12:19.579+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:12:20.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:12:19 smithi047 ceph-mon[77147]: pgmap v5288: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:12:20.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:12:19 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:12:20.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:12:19 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:12:20.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:12:19 smithi115 ceph-mon[78593]: pgmap v5288: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:12:20.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:12:19 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:12:20.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:12:19 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:12:20.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:12:20 smithi115 conmon[90393]: 2022-05-14T02:12:20.575+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:12:21.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:12:20 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:12:21.292 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T02:12:21.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:12:21.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T02:12:21.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:12:21.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:12:21.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:12:21.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T02:12:21.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:12:21.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T02:12:21.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:12:21.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T02:12:21.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:12:21.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T02:12:21.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:12:21.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T02:12:21.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:12:21.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:12:21.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:12:21.297 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:12:20 smithi047 ceph-mon[77147]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T02:12:21.298 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:12:20 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:12:21.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:12:20 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:12:21.359 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T02:12:21.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:12:21.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T02:12:21.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:12:21.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:12:21.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:12:21.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T02:12:21.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:12:21.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T02:12:21.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:12:21.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T02:12:21.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:12:21.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T02:12:21.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:12:21.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T02:12:21.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:12:21.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:12:21.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:12:21.363 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:12:20 smithi115 ceph-mon[78593]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T02:12:21.364 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:12:20 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:12:21.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:12:21 smithi115 conmon[90393]: 2022-05-14T02:12:21.562+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:12:22.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:12:21 smithi047 ceph-mon[77147]: pgmap v5289: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:12:22.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:12:21 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 9875 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:12:22.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:12:21 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:12:22.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:12:21 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:12:22.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:12:21 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:12:22.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:12:21 smithi115 ceph-mon[78593]: pgmap v5289: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:12:22.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:12:21 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 9875 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:12:22.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:12:21 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:12:22.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:12:21 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:12:22.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:12:21 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:12:22.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:12:22 smithi047 conmon[78521]: 2022-05-14T02:12:22.447+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:12:22.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:12:22 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T02:12:22.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:12:22 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:12:22.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:12:22 smithi047 conmon[78521]: yield 2022-05-14T02:12:22.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:12:22 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:12:22.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:12:22 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:12:22.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:12:22 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:12:22.797 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:12:22 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T02:12:22.797 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:12:22 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:12:22.797 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:12:22 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T02:12:22.797 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:12:22 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:12:22.798 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:12:22 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T02:12:22.798 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:12:22 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:12:22.798 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:12:22 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T02:12:22.798 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:12:22 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:12:22.799 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:12:22 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T02:12:22.799 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:12:22 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:12:22.799 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:12:22 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:12:22.799 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:12:22 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:12:22.799 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:12:22 smithi047 conmon[78521]: 2022-05-14T02:12:22.448+0000 7f5344f82700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T02:12:22.800 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:12:22 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T02:12:22.800 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:12:22 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:12:22.800 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:12:22 smithi047 conmon[78521]: yield 2022-05-14T02:12:22.801 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:12:22 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:12:22.801 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:12:22 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:12:22.801 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:12:22 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:12:22.801 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:12:22 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T02:12:22.802 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:12:22 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:12:22.802 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:12:22 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T02:12:22.802 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:12:22 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:12:22.802 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:12:22 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T02:12:22.802 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:12:22 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:12:22.803 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:12:22 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T02:12:22.803 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:12:22 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:12:22.803 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:12:22 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T02:12:22.803 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:12:22 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:12:22.804 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:12:22 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:12:22.804 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:12:22 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:12:22.804 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:12:22 smithi047 conmon[78521]: 2022-05-14T02:12:22.804 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:12:22 smithi047 conmon[78521]: During handling of the above exception, another exception occurred: 2022-05-14T02:12:22.805 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:12:22 smithi047 conmon[78521]: 2022-05-14T02:12:22.805 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:12:22 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T02:12:22.805 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:12:22 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T02:12:22.805 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:12:22 smithi047 conmon[78521]: return f(*arg) 2022-05-14T02:12:22.806 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:12:22 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T02:12:22.806 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:12:22 smithi047 conmon[78521]: self._write_client_files(client_files, host) 2022-05-14T02:12:22.807 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:12:22 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T02:12:22.807 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:12:22 smithi047 conmon[78521]: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T02:12:22.807 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:12:22 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T02:12:22.808 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:12:22 smithi047 conmon[78521]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T02:12:22.808 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:12:22 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T02:12:22.808 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:12:22 smithi047 conmon[78521]: return self.event_loop.get_result(coro) 2022-05-14T02:12:22.808 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:12:22 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T02:12:22.809 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:12:22 smithi047 conmon[78521]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T02:12:22.809 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:12:22 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T02:12:22.809 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:12:22 smithi047 conmon[78521]: return self.__get_result() 2022-05-14T02:12:22.809 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:12:22 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T02:12:22.809 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:12:22 smithi047 conmon[78521]: raise self._exception 2022-05-14T02:12:22.810 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:12:22 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T02:12:22.810 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:12:22 smithi047 conmon[78521]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T02:12:22.810 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:12:22 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T02:12:22.810 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:12:22 smithi047 conmon[78521]: conn = await self._remote_connection(host, addr) 2022-05-14T02:12:22.811 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:12:22 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T02:12:22.811 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:12:22 smithi047 conmon[78521]: raise 2022-05-14T02:12:22.811 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:12:22 smithi047 conmon[78521]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T02:12:22.811 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:12:22 smithi047 conmon[78521]: self.gen.throw(type, value, traceback) 2022-05-14T02:12:22.812 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:12:22 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T02:12:22.812 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:12:22 smithi047 conmon[78521]: raise OrchestratorError(msg) 2022-05-14T02:12:22.812 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:12:22 smithi047 conmon[78521]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:12:22.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:12:22 smithi115 conmon[90393]: 2022-05-14T02:12:22.521+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:12:23.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:12:22 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:12:23.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:12:22 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:12:23.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:12:23 smithi115 conmon[90393]: 2022-05-14T02:12:23.499+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:12:24.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:12:23 smithi047 ceph-mon[77147]: pgmap v5290: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:12:24.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:12:23 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:12:24.292 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T02:12:24.292 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:12:24.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T02:12:24.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:12:24.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:12:24.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:12:24.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T02:12:24.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:12:24.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T02:12:24.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:12:24.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T02:12:24.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:12:24.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T02:12:24.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:12:24.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T02:12:24.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:12:24.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:12:24.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:12:24.297 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:12:23 smithi047 ceph-mon[77147]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T02:12:24.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T02:12:24.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:12:24.298 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T02:12:24.298 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:12:24.298 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:12:24.298 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:12:24.298 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T02:12:24.299 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:12:24.299 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T02:12:24.299 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:12:24.299 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T02:12:24.300 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:12:24.300 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T02:12:24.300 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:12:24.300 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T02:12:24.300 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:12:24.301 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:12:24.301 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:12:24.301 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T02:12:24.301 INFO:journalctl@ceph.mon.b.smithi047.stdout: During handling of the above exception, another exception occurred: 2022-05-14T02:12:24.302 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T02:12:24.302 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T02:12:24.302 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T02:12:24.302 INFO:journalctl@ceph.mon.b.smithi047.stdout: return f(*arg) 2022-05-14T02:12:24.303 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T02:12:24.303 INFO:journalctl@ceph.mon.b.smithi047.stdout: self._write_client_files(client_files, host) 2022-05-14T02:12:24.303 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T02:12:24.303 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T02:12:24.304 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T02:12:24.304 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T02:12:24.304 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T02:12:24.304 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.event_loop.get_result(coro) 2022-05-14T02:12:24.304 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T02:12:24.305 INFO:journalctl@ceph.mon.b.smithi047.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T02:12:24.305 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T02:12:24.305 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.__get_result() 2022-05-14T02:12:24.305 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T02:12:24.305 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise self._exception 2022-05-14T02:12:24.306 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T02:12:24.306 INFO:journalctl@ceph.mon.b.smithi047.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T02:12:24.306 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T02:12:24.306 INFO:journalctl@ceph.mon.b.smithi047.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T02:12:24.307 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T02:12:24.307 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise 2022-05-14T02:12:24.307 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T02:12:24.307 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.gen.throw(type, value, traceback) 2022-05-14T02:12:24.307 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T02:12:24.308 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OrchestratorError(msg) 2022-05-14T02:12:24.308 INFO:journalctl@ceph.mon.b.smithi047.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:12:24.308 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:12:23 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:12:24.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:12:23 smithi115 ceph-mon[78593]: pgmap v5290: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:12:24.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:12:23 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:12:24.359 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T02:12:24.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:12:24.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T02:12:24.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:12:24.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:12:24.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:12:24.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T02:12:24.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:12:24.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T02:12:24.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:12:24.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T02:12:24.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:12:24.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T02:12:24.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:12:24.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T02:12:24.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:12:24.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:12:24.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:12:24.363 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:12:23 smithi115 ceph-mon[78593]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T02:12:24.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T02:12:24.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:12:24.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T02:12:24.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:12:24.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:12:24.365 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:12:24.365 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T02:12:24.365 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:12:24.365 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T02:12:24.366 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:12:24.366 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T02:12:24.366 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:12:24.366 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T02:12:24.366 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:12:24.367 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T02:12:24.367 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:12:24.367 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:12:24.367 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:12:24.367 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T02:12:24.368 INFO:journalctl@ceph.mon.c.smithi115.stdout: During handling of the above exception, another exception occurred: 2022-05-14T02:12:24.368 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T02:12:24.368 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T02:12:24.368 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T02:12:24.368 INFO:journalctl@ceph.mon.c.smithi115.stdout: return f(*arg) 2022-05-14T02:12:24.369 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T02:12:24.369 INFO:journalctl@ceph.mon.c.smithi115.stdout: self._write_client_files(client_files, host) 2022-05-14T02:12:24.369 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T02:12:24.369 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T02:12:24.370 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T02:12:24.370 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T02:12:24.370 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T02:12:24.370 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.event_loop.get_result(coro) 2022-05-14T02:12:24.370 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T02:12:24.371 INFO:journalctl@ceph.mon.c.smithi115.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T02:12:24.371 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T02:12:24.371 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.__get_result() 2022-05-14T02:12:24.371 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T02:12:24.371 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise self._exception 2022-05-14T02:12:24.372 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T02:12:24.372 INFO:journalctl@ceph.mon.c.smithi115.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T02:12:24.372 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T02:12:24.372 INFO:journalctl@ceph.mon.c.smithi115.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T02:12:24.373 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T02:12:24.373 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise 2022-05-14T02:12:24.373 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T02:12:24.373 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.gen.throw(type, value, traceback) 2022-05-14T02:12:24.373 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T02:12:24.374 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OrchestratorError(msg) 2022-05-14T02:12:24.374 INFO:journalctl@ceph.mon.c.smithi115.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:12:24.374 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:12:23 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:12:24.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:12:24 smithi115 conmon[90393]: 2022-05-14T02:12:24.467+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:12:25.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:12:24 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:12:25.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:12:24 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:12:25.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:12:25 smithi115 conmon[90393]: 2022-05-14T02:12:25.500+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:12:26.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:12:25 smithi047 ceph-mon[77147]: pgmap v5291: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:12:26.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:12:25 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:12:26.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:12:25 smithi115 ceph-mon[78593]: pgmap v5291: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:12:26.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:12:25 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:12:26.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:12:26 smithi115 conmon[90393]: 2022-05-14T02:12:26.466+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:12:27.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:12:26 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:12:27.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:12:26 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:12:27.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:12:27 smithi115 conmon[90393]: 2022-05-14T02:12:27.469+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:12:28.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:12:27 smithi047 ceph-mon[77147]: pgmap v5292: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:12:28.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:12:27 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:12:28.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:12:27 smithi115 ceph-mon[78593]: pgmap v5292: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:12:28.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:12:27 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:12:28.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:12:28 smithi115 conmon[90393]: 2022-05-14T02:12:28.420+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:12:29.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:12:28 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:12:29.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:12:28 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:12:29.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:12:29 smithi115 conmon[90393]: 2022-05-14T02:12:29.465+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:12:30.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:12:29 smithi047 ceph-mon[77147]: pgmap v5293: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:12:30.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:12:29 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:12:30.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:12:29 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:12:30.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:12:29 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:12:30.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:12:29 smithi115 ceph-mon[78593]: pgmap v5293: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:12:30.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:12:29 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:12:30.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:12:29 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:12:30.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:12:29 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:12:30.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:12:30 smithi115 conmon[90393]: 2022-05-14T02:12:30.469+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:12:31.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:12:30 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 9880 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:12:31.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:12:30 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:12:31.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:12:30 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 9880 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:12:31.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:12:30 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:12:31.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:12:31 smithi115 conmon[90393]: 2022-05-14T02:12:31.450+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:12:32.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:12:31 smithi047 ceph-mon[77147]: pgmap v5294: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:12:32.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:12:31 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:12:32.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:12:31 smithi115 ceph-mon[78593]: pgmap v5294: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:12:32.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:12:31 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:12:32.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:12:32 smithi115 conmon[90393]: 2022-05-14T02:12:32.405+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:12:33.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:12:32 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:12:33.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:12:32 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:12:33.865 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:12:33 smithi115 conmon[90393]: 2022-05-14T02:12:33.367+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:12:34.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:12:33 smithi047 ceph-mon[77147]: pgmap v5295: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:12:34.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:12:33 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:12:34.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:12:33 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:12:34.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:12:33 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:12:34.353 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:12:33 smithi115 ceph-mon[78593]: pgmap v5295: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:12:34.354 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:12:33 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:12:34.354 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:12:33 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:12:34.354 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:12:33 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:12:34.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:12:34 smithi115 conmon[90393]: 2022-05-14T02:12:34.351+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:12:35.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:12:34 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:12:35.312 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:12:34 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:12:35.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:12:35 smithi115 conmon[90393]: 2022-05-14T02:12:35.310+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:12:36.288 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:12:35 smithi115 ceph-mon[78593]: pgmap v5296: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:12:36.288 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:12:35 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:12:36.288 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:12:35 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 9890 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:12:36.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:12:35 smithi047 ceph-mon[77147]: pgmap v5296: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:12:36.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:12:35 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:12:36.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:12:35 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 9890 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:12:36.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:12:36 smithi115 conmon[90393]: 2022-05-14T02:12:36.286+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:12:37.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:12:36 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:12:37.315 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:12:36 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:12:37.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:12:37 smithi115 conmon[90393]: 2022-05-14T02:12:37.314+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:12:38.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:12:37 smithi115 ceph-mon[78593]: pgmap v5297: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:12:38.293 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:12:37 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:12:38.293 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:12:37 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:12:38.293 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:12:37 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:12:38.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:12:37 smithi047 ceph-mon[77147]: pgmap v5297: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:12:38.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:12:37 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:12:38.295 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:12:37 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:12:38.295 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:12:37 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:12:38.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:12:38 smithi115 conmon[90393]: 2022-05-14T02:12:38.290+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:12:39.247 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:12:38 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:12:39.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:12:38 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:12:39.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:12:39 smithi115 conmon[90393]: 2022-05-14T02:12:39.246+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:12:40.262 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:12:39 smithi115 ceph-mon[78593]: pgmap v5298: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:12:40.263 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:12:39 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:12:40.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:12:39 smithi047 ceph-mon[77147]: pgmap v5298: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:12:40.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:12:39 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:12:40.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:12:40 smithi115 conmon[90393]: 2022-05-14T02:12:40.261+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:12:41.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:12:41 smithi115 conmon[90393]: 2022-05-14T02:12:41.217+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:12:41.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:12:41 smithi115 ceph-mon[78593]: pgmap v5299: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:12:41.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:12:41 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:12:41.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:12:41 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 9895 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:12:41.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:12:41 smithi047 ceph-mon[77147]: pgmap v5299: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:12:41.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:12:41 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:12:41.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:12:41 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 9895 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:12:42.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:12:42 smithi115 conmon[90393]: 2022-05-14T02:12:42.243+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:12:42.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:12:42 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:12:42.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:12:42 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:12:42.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:12:42 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:12:42.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:12:42 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:12:42.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:12:42 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:12:42.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:12:42 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:12:43.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:12:43 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:12:43.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:12:43 smithi115 ceph-mon[78593]: pgmap v5300: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:12:43.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:12:43 smithi115 conmon[90393]: 2022-05-14T02:12:43.237+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:12:43.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:12:43 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:12:43.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:12:43 smithi047 ceph-mon[77147]: pgmap v5300: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:12:44.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:12:44 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:12:44.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:12:44 smithi115 conmon[90393]: 2022-05-14T02:12:44.271+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:12:44.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:12:44 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:12:45.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:12:45 smithi115 conmon[90393]: 2022-05-14T02:12:45.272+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:12:45.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:12:45 smithi115 ceph-mon[78593]: pgmap v5301: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:12:45.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:12:45 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:12:45.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:12:45 smithi047 ceph-mon[77147]: pgmap v5301: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:12:45.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:12:45 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:12:46.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:12:46 smithi115 conmon[90393]: 2022-05-14T02:12:46.314+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:12:46.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:12:46 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:12:46.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:12:46 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:12:47.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:12:47 smithi115 conmon[90393]: 2022-05-14T02:12:47.351+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:12:47.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:12:47 smithi115 ceph-mon[78593]: pgmap v5302: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:12:47.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:12:47 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:12:47.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:12:47 smithi047 ceph-mon[77147]: pgmap v5302: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:12:47.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:12:47 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:12:48.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:12:48 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:12:48.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:12:48 smithi115 conmon[90393]: 2022-05-14T02:12:48.375+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:12:48.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:12:48 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:12:49.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:12:49 smithi115 conmon[90393]: 2022-05-14T02:12:49.347+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:12:49.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:12:49 smithi115 ceph-mon[78593]: pgmap v5303: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:12:49.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:12:49 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:12:49.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:12:49 smithi047 ceph-mon[77147]: pgmap v5303: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:12:49.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:12:49 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:12:50.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:12:50 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:12:50.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:12:50 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:12:50.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:12:50 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:12:50.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:12:50 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 9900 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:12:50.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:12:50 smithi115 conmon[90393]: 2022-05-14T02:12:50.372+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:12:50.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:12:50 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:12:50.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:12:50 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:12:50.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:12:50 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:12:50.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:12:50 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 9900 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:12:51.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:12:51 smithi115 conmon[90393]: 2022-05-14T02:12:51.353+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:12:51.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:12:51 smithi115 ceph-mon[78593]: pgmap v5304: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:12:51.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:12:51 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:12:51.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:12:51 smithi047 ceph-mon[77147]: pgmap v5304: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:12:51.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:12:51 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:12:52.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:12:52 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:12:52.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:12:52 smithi115 conmon[90393]: 2022-05-14T02:12:52.356+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:12:52.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:12:52 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:12:53.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:12:53 smithi115 conmon[90393]: 2022-05-14T02:12:53.337+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:12:53.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:12:53 smithi115 ceph-mon[78593]: pgmap v5305: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:12:53.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:12:53 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:12:53.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:12:53 smithi047 ceph-mon[77147]: pgmap v5305: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:12:53.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:12:53 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:12:54.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:12:54 smithi115 conmon[90393]: 2022-05-14T02:12:54.289+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:12:54.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:12:54 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:12:54.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:12:54 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:12:54.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:12:54 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:12:54.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:12:54 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:12:54.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:12:54 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:12:54.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:12:54 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:12:55.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:12:55 smithi115 conmon[90393]: 2022-05-14T02:12:55.295+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:12:55.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:12:55 smithi115 ceph-mon[78593]: pgmap v5306: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:12:55.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:12:55 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:12:55.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:12:55 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 9910 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:12:55.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:12:55 smithi047 ceph-mon[77147]: pgmap v5306: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:12:55.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:12:55 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:12:55.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:12:55 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 9910 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:12:56.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:12:56 smithi115 conmon[90393]: 2022-05-14T02:12:56.265+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:12:56.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:12:56 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:12:56.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:12:56 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:12:57.521 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:12:57 smithi115 conmon[90393]: 2022-05-14T02:12:57.266+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:12:57.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:12:57 smithi047 ceph-mon[77147]: pgmap v5307: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:12:57.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:12:57 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:12:57.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:12:57 smithi115 ceph-mon[78593]: pgmap v5307: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:12:57.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:12:57 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:12:58.526 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:12:58 smithi115 conmon[90393]: 2022-05-14T02:12:58.268+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:12:58.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:12:58 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:12:58.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:12:58 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:12:58.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:12:58 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:12:58.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:12:58 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:12:58.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:12:58 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:12:58.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:12:58 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:12:59.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:12:59 smithi115 ceph-mon[78593]: pgmap v5308: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:12:59.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:12:59 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:12:59.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:12:59 smithi115 conmon[90393]: 2022-05-14T02:12:59.305+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:12:59.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:12:59 smithi047 ceph-mon[77147]: pgmap v5308: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:12:59.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:12:59 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:13:00.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:13:00 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:13:00.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:13:00 smithi115 conmon[90393]: 2022-05-14T02:13:00.285+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:13:00.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:13:00 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:13:01.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:13:01 smithi115 conmon[90393]: 2022-05-14T02:13:01.289+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:13:01.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:13:01 smithi115 ceph-mon[78593]: pgmap v5309: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:13:01.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:13:01 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:13:01.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:13:01 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 9915 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:13:01.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:13:01 smithi047 ceph-mon[77147]: pgmap v5309: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:13:01.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:13:01 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:13:01.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:13:01 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 9915 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:13:02.546 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:13:02 smithi115 conmon[90393]: 2022-05-14T02:13:02.242+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:13:02.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:13:02 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:13:02.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:13:02 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:13:02.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:13:02 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:13:03.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:13:02 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:13:03.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:13:02 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:13:03.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:13:02 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:13:03.545 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:13:03 smithi115 conmon[90393]: 2022-05-14T02:13:03.223+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:13:03.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:13:03 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:13:03.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:13:03 smithi115 ceph-mon[78593]: pgmap v5310: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:13:04.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:13:03 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:13:04.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:13:03 smithi047 ceph-mon[77147]: pgmap v5310: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:13:04.549 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:13:04 smithi115 conmon[90393]: 2022-05-14T02:13:04.195+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:13:04.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:13:04 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:13:05.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:13:04 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:13:05.551 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:13:05 smithi115 conmon[90393]: 2022-05-14T02:13:05.162+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:13:05.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:13:05 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:13:05.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:13:05 smithi115 ceph-mon[78593]: pgmap v5311: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:13:06.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:13:05 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:13:06.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:13:05 smithi047 ceph-mon[77147]: pgmap v5311: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:13:06.555 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:13:06 smithi115 conmon[90393]: 2022-05-14T02:13:06.169+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:13:06.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:13:06 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:13:07.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:13:06 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:13:07.557 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:13:07 smithi115 conmon[90393]: 2022-05-14T02:13:07.141+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:13:07.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:13:07 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:13:07.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:13:07 smithi115 ceph-mon[78593]: pgmap v5312: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:13:08.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:13:07 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:13:08.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:13:07 smithi047 ceph-mon[77147]: pgmap v5312: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:13:08.562 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:13:08 smithi115 conmon[90393]: 2022-05-14T02:13:08.114+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:13:08.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:13:08 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:13:09.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:13:08 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:13:09.564 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:13:09 smithi115 conmon[90393]: 2022-05-14T02:13:09.153+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:13:09.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:13:09 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:13:09.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:13:09 smithi115 ceph-mon[78593]: pgmap v5313: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:13:10.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:13:09 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:13:10.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:13:09 smithi047 ceph-mon[77147]: pgmap v5313: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:13:10.573 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:13:10 smithi115 conmon[90393]: 2022-05-14T02:13:10.156+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:13:10.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:13:10 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:13:10.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:13:10 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:13:10.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:13:10 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:13:10.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:13:10 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 9920 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:13:11.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:13:10 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:13:11.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:13:10 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:13:11.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:13:10 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:13:11.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:13:10 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 9920 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:13:11.575 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:13:11 smithi115 conmon[90393]: 2022-05-14T02:13:11.119+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:13:11.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:13:11 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:13:11.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:13:11 smithi115 ceph-mon[78593]: pgmap v5314: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:13:12.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:13:11 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:13:12.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:13:11 smithi047 ceph-mon[77147]: pgmap v5314: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:13:12.580 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:13:12 smithi115 conmon[90393]: 2022-05-14T02:13:12.120+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:13:12.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:13:12 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:13:13.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:13:12 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:13:13.582 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:13:13 smithi115 conmon[90393]: 2022-05-14T02:13:13.106+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:13:13.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:13:13 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:13:13.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:13:13 smithi115 ceph-mon[78593]: pgmap v5315: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:13:14.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:13:13 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:13:14.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:13:13 smithi047 ceph-mon[77147]: pgmap v5315: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:13:14.591 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:13:14 smithi115 conmon[90393]: 2022-05-14T02:13:14.111+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:13:14.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:13:14 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:13:14.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:13:14 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:13:14.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:13:14 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:13:14.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:13:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T02:13:14.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:13:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T02:13:15.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:13:14 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:13:15.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:13:14 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:13:15.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:13:14 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:13:15.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:13:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T02:13:15.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:13:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T02:13:15.593 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:13:15 smithi115 conmon[90393]: 2022-05-14T02:13:15.112+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:13:15.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:13:15 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:13:15.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:13:15 smithi115 ceph-mon[78593]: pgmap v5316: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:13:15.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:13:15 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 9931 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:13:16.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:13:15 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:13:16.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:13:15 smithi047 ceph-mon[77147]: pgmap v5316: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:13:16.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:13:15 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 9931 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:13:16.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:13:16 smithi115 conmon[90393]: 2022-05-14T02:13:16.063+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:13:16.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:13:16 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:13:17.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:13:16 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:13:17.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:13:17 smithi115 conmon[90393]: 2022-05-14T02:13:17.022+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:13:17.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:13:17 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:13:17.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:13:17 smithi115 ceph-mon[78593]: pgmap v5317: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:13:18.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:13:17 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:13:18.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:13:17 smithi047 ceph-mon[77147]: pgmap v5317: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:13:18.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:13:18 smithi115 conmon[90393]: 2022-05-14T02:13:18.021+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:13:19.003 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:13:18 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:13:19.003 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:13:18 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:13:19.004 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:13:18 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:13:19.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:13:18 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:13:19.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:13:18 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:13:19.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:13:18 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:13:19.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:13:19 smithi115 conmon[90393]: 2022-05-14T02:13:19.001+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:13:19.983 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:13:19 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:13:19.983 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:13:19 smithi115 ceph-mon[78593]: pgmap v5318: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:13:20.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:13:19 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:13:20.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:13:19 smithi047 ceph-mon[77147]: pgmap v5318: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:13:20.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:13:19 smithi115 conmon[90393]: 2022-05-14T02:13:19.981+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:13:20.975 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:13:20 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:13:21.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:13:20 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:13:21.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:13:20 smithi115 conmon[90393]: 2022-05-14T02:13:20.973+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:13:21.944 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:13:21 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:13:21.944 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:13:21 smithi115 ceph-mon[78593]: pgmap v5319: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:13:21.945 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:13:21 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 9935 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:13:22.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:13:21 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:13:22.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:13:21 smithi047 ceph-mon[77147]: pgmap v5319: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:13:22.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:13:21 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 9935 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:13:22.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:13:21 smithi115 conmon[90393]: 2022-05-14T02:13:21.942+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:13:22.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:13:22 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:13:22.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:13:22 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:13:22.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:13:22 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:13:22.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:13:22 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T02:13:22.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:13:22 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T02:13:22.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:13:22 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T02:13:22.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:13:22 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:13:22.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:13:22 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:13:22.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:13:22 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:13:22.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:13:22 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T02:13:22.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:13:22 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T02:13:22.862 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:13:22 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T02:13:23.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:13:22 smithi115 conmon[90393]: 2022-05-14T02:13:22.904+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:13:23.880 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:13:23 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:13:23.880 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:13:23 smithi115 ceph-mon[78593]: pgmap v5320: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:13:23.881 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:13:23 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:13:24.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:13:23 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:13:24.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:13:23 smithi047 ceph-mon[77147]: pgmap v5320: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:13:24.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:13:23 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:13:24.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:13:23 smithi115 conmon[90393]: 2022-05-14T02:13:23.878+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:13:25.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:13:24 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:13:25.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:13:24 smithi115 conmon[90393]: 2022-05-14T02:13:24.875+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:13:25.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:13:24 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:13:26.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:13:25 smithi047 ceph-mon[77147]: pgmap v5321: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:13:26.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:13:25 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:13:26.042 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:13:25 smithi047 conmon[78521]: 2022-05-14T02:13:25.552+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:13:26.042 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:13:25 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T02:13:26.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:13:25 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:13:26.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:13:25 smithi047 conmon[78521]: yield 2022-05-14T02:13:26.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:13:25 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:13:26.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:13:25 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:13:26.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:13:25 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:13:26.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:13:25 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T02:13:26.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:13:25 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:13:26.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:13:25 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T02:13:26.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:13:25 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:13:26.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:13:25 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T02:13:26.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:13:25 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:13:26.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:13:25 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T02:13:26.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:13:25 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:13:26.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:13:25 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T02:13:26.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:13:25 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:13:26.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:13:25 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:13:26.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:13:25 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:13:26.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:13:25 smithi115 conmon[90393]: 2022-05-14T02:13:25.881+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:13:26.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:13:25 smithi115 ceph-mon[78593]: pgmap v5321: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:13:26.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:13:25 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:13:27.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:13:26 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:13:27.042 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T02:13:27.042 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:13:27.042 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T02:13:27.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:13:27.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:13:27.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:13:27.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T02:13:27.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:13:27.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T02:13:27.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:13:27.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T02:13:27.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:13:27.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T02:13:27.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:13:27.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T02:13:27.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:13:27.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:13:27.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:13:27.046 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:13:26 smithi047 ceph-mon[77147]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T02:13:27.046 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:13:26 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:13:27.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:13:26 smithi115 conmon[90393]: 2022-05-14T02:13:26.880+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:13:27.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:13:26 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:13:27.109 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T02:13:27.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:13:27.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T02:13:27.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:13:27.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:13:27.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:13:27.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T02:13:27.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:13:27.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T02:13:27.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:13:27.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T02:13:27.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:13:27.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T02:13:27.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:13:27.113 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T02:13:27.113 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:13:27.113 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:13:27.114 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:13:27.114 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:13:26 smithi115 ceph-mon[78593]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T02:13:27.114 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:13:26 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:13:27.913 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:13:27 smithi115 ceph-mon[78593]: pgmap v5322: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:13:27.913 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:13:27 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:13:28.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:13:27 smithi047 ceph-mon[77147]: pgmap v5322: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:13:28.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:13:27 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:13:28.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:13:27 smithi115 conmon[90393]: 2022-05-14T02:13:27.911+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:13:29.042 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:13:28 smithi047 conmon[78521]: 2022-05-14T02:13:28.624+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:13:29.042 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:13:28 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T02:13:29.042 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:13:28 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:13:29.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:13:28 smithi047 conmon[78521]: yield 2022-05-14T02:13:29.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:13:28 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:13:29.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:13:28 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:13:29.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:13:28 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:13:29.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:13:28 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T02:13:29.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:13:28 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:13:29.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:13:28 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T02:13:29.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:13:28 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:13:29.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:13:28 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T02:13:29.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:13:28 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:13:29.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:13:28 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T02:13:29.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:13:28 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:13:29.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:13:28 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T02:13:29.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:13:28 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:13:29.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:13:28 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:13:29.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:13:28 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:13:29.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:13:28 smithi047 conmon[78521]: 2022-05-14T02:13:28.625+0000 7f5341f7c700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T02:13:29.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:13:28 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T02:13:29.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:13:28 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:13:29.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:13:28 smithi047 conmon[78521]: yield 2022-05-14T02:13:29.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:13:28 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:13:29.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:13:28 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:13:29.048 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:13:28 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:13:29.048 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:13:28 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T02:13:29.048 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:13:28 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:13:29.048 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:13:28 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T02:13:29.049 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:13:28 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:13:29.049 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:13:28 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T02:13:29.049 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:13:28 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:13:29.049 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:13:28 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T02:13:29.049 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:13:28 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:13:29.050 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:13:28 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T02:13:29.050 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:13:28 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:13:29.050 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:13:28 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:13:29.050 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:13:28 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:13:29.051 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:13:28 smithi047 conmon[78521]: 2022-05-14T02:13:29.051 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:13:28 smithi047 conmon[78521]: During handling of the above exception, another exception occurred: 2022-05-14T02:13:29.051 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:13:28 smithi047 conmon[78521]: 2022-05-14T02:13:29.051 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:13:28 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T02:13:29.051 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:13:28 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T02:13:29.052 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:13:28 smithi047 conmon[78521]: return f(*arg) 2022-05-14T02:13:29.052 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:13:28 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T02:13:29.052 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:13:28 smithi047 conmon[78521]: self._write_client_files(client_files, host) 2022-05-14T02:13:29.052 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:13:28 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T02:13:29.053 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:13:28 smithi047 conmon[78521]: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T02:13:29.053 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:13:28 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T02:13:29.053 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:13:28 smithi047 conmon[78521]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T02:13:29.053 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:13:28 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T02:13:29.053 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:13:28 smithi047 conmon[78521]: return self.event_loop.get_result(coro) 2022-05-14T02:13:29.054 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:13:28 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T02:13:29.054 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:13:28 smithi047 conmon[78521]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T02:13:29.054 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:13:28 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T02:13:29.054 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:13:28 smithi047 conmon[78521]: return self.__get_result() 2022-05-14T02:13:29.054 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:13:28 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T02:13:29.055 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:13:28 smithi047 conmon[78521]: raise self._exception 2022-05-14T02:13:29.055 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:13:28 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T02:13:29.055 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:13:28 smithi047 conmon[78521]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T02:13:29.055 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:13:28 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T02:13:29.056 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:13:28 smithi047 conmon[78521]: conn = await self._remote_connection(host, addr) 2022-05-14T02:13:29.056 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:13:28 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T02:13:29.056 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:13:28 smithi047 conmon[78521]: raise 2022-05-14T02:13:29.056 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:13:28 smithi047 conmon[78521]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T02:13:29.056 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:13:28 smithi047 conmon[78521]: self.gen.throw(type, value, traceback) 2022-05-14T02:13:29.057 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:13:28 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T02:13:29.057 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:13:28 smithi047 conmon[78521]: raise OrchestratorError(msg) 2022-05-14T02:13:29.057 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:13:28 smithi047 conmon[78521]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:13:29.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:13:28 smithi115 conmon[90393]: 2022-05-14T02:13:28.896+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:13:29.912 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:13:29 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:13:29.912 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:13:29 smithi115 ceph-mon[78593]: pgmap v5323: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:13:29.912 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:13:29 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:13:29.912 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T02:13:29.913 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:13:29.913 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T02:13:29.913 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:13:29.913 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:13:29.914 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:13:29.914 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T02:13:29.914 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:13:29.914 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T02:13:29.914 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:13:29.915 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T02:13:29.915 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:13:29.915 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T02:13:29.915 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:13:29.916 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T02:13:29.916 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:13:29.916 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:13:29.916 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:13:29.917 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:13:29 smithi115 ceph-mon[78593]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T02:13:29.917 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T02:13:29.917 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:13:29.917 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T02:13:29.918 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:13:29.918 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:13:29.918 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:13:29.918 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T02:13:29.919 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:13:29.919 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T02:13:29.919 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:13:29.919 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T02:13:29.920 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:13:29.920 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T02:13:29.920 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:13:29.920 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T02:13:29.920 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:13:29.921 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:13:29.921 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:13:29.921 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T02:13:29.921 INFO:journalctl@ceph.mon.c.smithi115.stdout: During handling of the above exception, another exception occurred: 2022-05-14T02:13:29.922 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T02:13:29.922 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T02:13:29.922 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T02:13:29.922 INFO:journalctl@ceph.mon.c.smithi115.stdout: return f(*arg) 2022-05-14T02:13:29.922 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T02:13:29.923 INFO:journalctl@ceph.mon.c.smithi115.stdout: self._write_client_files(client_files, host) 2022-05-14T02:13:29.923 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T02:13:29.923 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T02:13:29.923 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T02:13:29.923 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T02:13:29.924 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T02:13:29.924 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.event_loop.get_result(coro) 2022-05-14T02:13:29.924 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T02:13:29.924 INFO:journalctl@ceph.mon.c.smithi115.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T02:13:29.924 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T02:13:29.925 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.__get_result() 2022-05-14T02:13:29.925 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T02:13:29.925 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise self._exception 2022-05-14T02:13:29.925 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T02:13:29.925 INFO:journalctl@ceph.mon.c.smithi115.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T02:13:29.926 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T02:13:29.926 INFO:journalctl@ceph.mon.c.smithi115.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T02:13:29.926 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T02:13:29.926 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise 2022-05-14T02:13:29.926 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T02:13:29.927 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.gen.throw(type, value, traceback) 2022-05-14T02:13:29.927 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T02:13:29.927 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OrchestratorError(msg) 2022-05-14T02:13:29.927 INFO:journalctl@ceph.mon.c.smithi115.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:13:29.928 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:13:29 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:13:30.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:13:29 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:13:30.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:13:29 smithi047 ceph-mon[77147]: pgmap v5323: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:13:30.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:13:29 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:13:30.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T02:13:30.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:13:30.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T02:13:30.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:13:30.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:13:30.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:13:30.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T02:13:30.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:13:30.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T02:13:30.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:13:30.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T02:13:30.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:13:30.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T02:13:30.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:13:30.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T02:13:30.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:13:30.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:13:30.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:13:30.047 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:13:29 smithi047 ceph-mon[77147]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T02:13:30.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T02:13:30.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:13:30.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T02:13:30.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:13:30.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:13:30.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:13:30.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T02:13:30.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:13:30.049 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T02:13:30.049 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:13:30.049 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T02:13:30.049 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:13:30.050 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T02:13:30.050 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:13:30.050 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T02:13:30.050 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:13:30.050 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:13:30.051 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:13:30.051 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T02:13:30.051 INFO:journalctl@ceph.mon.b.smithi047.stdout: During handling of the above exception, another exception occurred: 2022-05-14T02:13:30.051 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T02:13:30.051 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T02:13:30.052 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T02:13:30.052 INFO:journalctl@ceph.mon.b.smithi047.stdout: return f(*arg) 2022-05-14T02:13:30.052 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T02:13:30.052 INFO:journalctl@ceph.mon.b.smithi047.stdout: self._write_client_files(client_files, host) 2022-05-14T02:13:30.053 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T02:13:30.053 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T02:13:30.053 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T02:13:30.053 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T02:13:30.053 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T02:13:30.054 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.event_loop.get_result(coro) 2022-05-14T02:13:30.054 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T02:13:30.054 INFO:journalctl@ceph.mon.b.smithi047.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T02:13:30.054 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T02:13:30.054 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.__get_result() 2022-05-14T02:13:30.055 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T02:13:30.055 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise self._exception 2022-05-14T02:13:30.055 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T02:13:30.056 INFO:journalctl@ceph.mon.b.smithi047.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T02:13:30.056 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T02:13:30.056 INFO:journalctl@ceph.mon.b.smithi047.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T02:13:30.056 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T02:13:30.056 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise 2022-05-14T02:13:30.057 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T02:13:30.057 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.gen.throw(type, value, traceback) 2022-05-14T02:13:30.057 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T02:13:30.057 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OrchestratorError(msg) 2022-05-14T02:13:30.058 INFO:journalctl@ceph.mon.b.smithi047.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:13:30.058 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:13:29 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:13:30.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:13:29 smithi115 conmon[90393]: 2022-05-14T02:13:29.910+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:13:30.948 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:13:30 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:13:30.948 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:13:30 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:13:30.948 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:13:30 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:13:30.948 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:13:30 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 9940 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:13:31.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:13:30 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:13:31.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:13:30 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:13:31.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:13:30 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:13:31.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:13:30 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 9940 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:13:31.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:13:30 smithi115 conmon[90393]: 2022-05-14T02:13:30.946+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:13:31.993 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:13:31 smithi115 ceph-mon[78593]: pgmap v5324: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:13:32.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:13:31 smithi047 ceph-mon[77147]: pgmap v5324: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:13:32.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:13:31 smithi115 conmon[90393]: 2022-05-14T02:13:31.992+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:13:32.982 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:13:32 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:13:33.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:13:32 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:13:33.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:13:32 smithi115 conmon[90393]: 2022-05-14T02:13:32.980+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:13:34.003 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:13:33 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:13:34.003 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:13:33 smithi115 ceph-mon[78593]: pgmap v5325: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:13:34.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:13:33 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:13:34.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:13:33 smithi047 ceph-mon[77147]: pgmap v5325: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:13:34.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:13:34 smithi115 conmon[90393]: 2022-05-14T02:13:34.001+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:13:35.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:13:34 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:13:35.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:13:34 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:13:35.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:13:34 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:13:35.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:13:34 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:13:35.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:13:34 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:13:35.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:13:34 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:13:35.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:13:35 smithi115 conmon[90393]: 2022-05-14T02:13:35.039+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:13:36.017 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:13:35 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:13:36.017 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:13:35 smithi115 ceph-mon[78593]: pgmap v5326: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:13:36.017 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:13:35 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 9950 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:13:36.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:13:35 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:13:36.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:13:35 smithi047 ceph-mon[77147]: pgmap v5326: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:13:36.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:13:35 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 9950 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:13:36.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:13:36 smithi115 conmon[90393]: 2022-05-14T02:13:36.015+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:13:36.986 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:13:36 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:13:37.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:13:36 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:13:37.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:13:36 smithi115 conmon[90393]: 2022-05-14T02:13:36.985+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:13:38.028 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:13:37 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:13:38.028 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:13:37 smithi115 ceph-mon[78593]: pgmap v5327: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:13:38.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:13:37 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:13:38.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:13:37 smithi047 ceph-mon[77147]: pgmap v5327: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:13:38.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:13:38 smithi115 conmon[90393]: 2022-05-14T02:13:38.026+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:13:39.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:13:38 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:13:39.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:13:38 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:13:39.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:13:38 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:13:39.048 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:13:38 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:13:39.049 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:13:38 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:13:39.049 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:13:38 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:13:39.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:13:39 smithi115 conmon[90393]: 2022-05-14T02:13:39.047+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:13:40.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:13:39 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:13:40.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:13:39 smithi047 ceph-mon[77147]: pgmap v5328: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:13:40.093 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:13:39 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:13:40.094 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:13:39 smithi115 ceph-mon[78593]: pgmap v5328: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:13:40.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:13:40 smithi115 conmon[90393]: 2022-05-14T02:13:40.092+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:13:41.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:13:40 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:13:41.057 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:13:40 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:13:41.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:13:41 smithi115 conmon[90393]: 2022-05-14T02:13:41.055+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:13:42.011 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:13:41 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:13:42.011 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:13:41 smithi115 ceph-mon[78593]: pgmap v5329: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:13:42.012 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:13:41 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 9955 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:13:42.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:13:41 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:13:42.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:13:41 smithi047 ceph-mon[77147]: pgmap v5329: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:13:42.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:13:41 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 9955 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:13:42.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:13:42 smithi115 conmon[90393]: 2022-05-14T02:13:42.009+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:13:43.038 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:13:42 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:13:43.038 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:13:42 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:13:43.038 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:13:42 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:13:43.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:13:42 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:13:43.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:13:42 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:13:43.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:13:42 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:13:43.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:13:43 smithi115 conmon[90393]: 2022-05-14T02:13:43.036+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:13:44.028 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:13:43 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:13:44.029 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:13:43 smithi115 ceph-mon[78593]: pgmap v5330: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:13:44.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:13:43 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:13:44.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:13:43 smithi047 ceph-mon[77147]: pgmap v5330: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:13:44.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:13:44 smithi115 conmon[90393]: 2022-05-14T02:13:44.027+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:13:44.999 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:13:44 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:13:45.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:13:44 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:13:45.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:13:44 smithi115 conmon[90393]: 2022-05-14T02:13:44.997+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:13:46.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:13:45 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:13:46.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:13:45 smithi115 ceph-mon[78593]: pgmap v5331: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:13:46.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:13:45 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:13:46.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:13:45 smithi047 ceph-mon[77147]: pgmap v5331: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:13:46.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:13:46 smithi115 conmon[90393]: 2022-05-14T02:13:46.039+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:13:47.008 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:13:46 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:13:47.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:13:46 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:13:47.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:13:47 smithi115 conmon[90393]: 2022-05-14T02:13:47.007+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:13:48.002 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:13:47 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:13:48.003 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:13:47 smithi115 ceph-mon[78593]: pgmap v5332: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:13:48.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:13:47 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:13:48.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:13:47 smithi047 ceph-mon[77147]: pgmap v5332: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:13:48.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:13:48 smithi115 conmon[90393]: 2022-05-14T02:13:48.001+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:13:49.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:13:48 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:13:49.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:13:48 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:13:49.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:13:48 smithi115 conmon[90393]: 2022-05-14T02:13:48.963+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:13:50.011 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:13:49 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:13:50.012 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:13:49 smithi115 ceph-mon[78593]: pgmap v5333: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:13:50.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:13:49 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:13:50.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:13:49 smithi047 ceph-mon[77147]: pgmap v5333: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:13:50.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:13:50 smithi115 conmon[90393]: 2022-05-14T02:13:50.010+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:13:51.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:13:50 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:13:51.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:13:50 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:13:51.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:13:50 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:13:51.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:13:50 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 9960 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:13:51.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:13:50 smithi115 conmon[90393]: 2022-05-14T02:13:50.977+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:13:51.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:13:50 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:13:51.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:13:50 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:13:51.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:13:50 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:13:51.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:13:50 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 9960 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:13:52.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:13:51 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:13:52.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:13:51 smithi047 ceph-mon[77147]: pgmap v5334: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:13:52.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:13:51 smithi115 conmon[90393]: 2022-05-14T02:13:51.959+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:13:52.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:13:51 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:13:52.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:13:51 smithi115 ceph-mon[78593]: pgmap v5334: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:13:53.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:13:52 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:13:53.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:13:52 smithi115 conmon[90393]: 2022-05-14T02:13:52.990+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:13:53.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:13:52 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:13:54.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:13:53 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:13:54.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:13:53 smithi047 ceph-mon[77147]: pgmap v5335: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:13:54.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:13:53 smithi115 conmon[90393]: 2022-05-14T02:13:53.991+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:13:54.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:13:53 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:13:54.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:13:53 smithi115 ceph-mon[78593]: pgmap v5335: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:13:55.021 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:13:54 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:13:55.021 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:13:54 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:13:55.022 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:13:54 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:13:55.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:13:54 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:13:55.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:13:54 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:13:55.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:13:54 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:13:55.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:13:55 smithi115 conmon[90393]: 2022-05-14T02:13:55.019+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:13:56.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:13:55 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:13:56.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:13:55 smithi047 ceph-mon[77147]: pgmap v5336: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:13:56.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:13:55 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 9970 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:13:56.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:13:56 smithi115 conmon[90393]: 2022-05-14T02:13:56.001+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:13:56.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:13:55 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:13:56.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:13:55 smithi115 ceph-mon[78593]: pgmap v5336: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:13:56.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:13:55 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 9970 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:13:57.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:13:56 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:13:57.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:13:57 smithi115 conmon[90393]: 2022-05-14T02:13:57.008+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:13:57.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:13:56 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:13:58.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:13:57 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:13:58.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:13:57 smithi047 ceph-mon[77147]: pgmap v5337: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:13:58.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:13:57 smithi115 conmon[90393]: 2022-05-14T02:13:57.965+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:13:58.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:13:57 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:13:58.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:13:57 smithi115 ceph-mon[78593]: pgmap v5337: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:13:59.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:13:58 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:13:59.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:13:58 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:13:59.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:13:58 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:13:59.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:13:58 smithi115 conmon[90393]: 2022-05-14T02:13:58.981+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:13:59.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:13:58 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:13:59.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:13:58 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:13:59.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:13:58 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:14:00.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:13:59 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:14:00.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:13:59 smithi047 ceph-mon[77147]: pgmap v5338: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:14:00.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:13:59 smithi115 conmon[90393]: 2022-05-14T02:13:59.954+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:14:00.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:13:59 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:14:00.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:13:59 smithi115 ceph-mon[78593]: pgmap v5338: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:14:01.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:14:00 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:14:01.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:14:00 smithi115 conmon[90393]: 2022-05-14T02:14:00.989+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:14:01.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:14:00 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:14:02.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:14:01 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:14:02.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:14:01 smithi047 ceph-mon[77147]: pgmap v5339: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:14:02.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:14:01 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 9975 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:14:02.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:14:02 smithi115 conmon[90393]: 2022-05-14T02:14:02.002+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:14:02.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:14:01 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:14:02.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:14:01 smithi115 ceph-mon[78593]: pgmap v5339: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:14:02.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:14:01 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 9975 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:14:03.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:14:02 smithi115 conmon[90393]: 2022-05-14T02:14:02.966+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:14:03.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:14:02 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:14:03.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:14:02 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:14:03.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:14:02 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:14:03.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:14:02 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:14:03.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:14:02 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:14:03.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:14:02 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:14:04.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:14:03 smithi115 conmon[90393]: 2022-05-14T02:14:03.947+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:14:04.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:14:03 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:14:04.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:14:03 smithi115 ceph-mon[78593]: pgmap v5340: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:14:04.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:14:03 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:14:04.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:14:03 smithi047 ceph-mon[77147]: pgmap v5340: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:14:05.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:14:04 smithi115 conmon[90393]: 2022-05-14T02:14:04.942+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:14:05.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:14:04 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:14:05.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:14:04 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:14:06.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:14:05 smithi115 conmon[90393]: 2022-05-14T02:14:05.953+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:14:06.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:14:05 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:14:06.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:14:05 smithi115 ceph-mon[78593]: pgmap v5341: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:14:06.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:14:05 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:14:06.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:14:05 smithi047 ceph-mon[77147]: pgmap v5341: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:14:07.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:14:06 smithi115 conmon[90393]: 2022-05-14T02:14:06.912+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:14:07.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:14:06 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:14:07.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:14:06 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:14:08.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:14:07 smithi115 conmon[90393]: 2022-05-14T02:14:07.954+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:14:08.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:14:07 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:14:08.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:14:07 smithi115 ceph-mon[78593]: pgmap v5342: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:14:08.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:14:07 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:14:08.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:14:07 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:14:08.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:14:07 smithi047 ceph-mon[77147]: pgmap v5342: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:14:08.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:14:07 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:14:09.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:14:08 smithi115 conmon[90393]: 2022-05-14T02:14:08.988+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:14:10.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:14:09 smithi115 conmon[90393]: 2022-05-14T02:14:09.962+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:14:10.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:14:09 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:14:10.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:14:09 smithi115 ceph-mon[78593]: pgmap v5343: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:14:10.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:14:09 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:14:10.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:14:09 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:14:10.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:14:09 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:14:10.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:14:09 smithi047 ceph-mon[77147]: pgmap v5343: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:14:10.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:14:09 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:14:10.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:14:09 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:14:11.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:14:10 smithi115 conmon[90393]: 2022-05-14T02:14:10.915+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:14:11.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:14:10 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:14:11.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:14:10 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 9980 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:14:11.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:14:10 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:14:11.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:14:10 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 9980 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:14:12.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:14:11 smithi115 conmon[90393]: 2022-05-14T02:14:11.910+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:14:12.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:14:11 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:14:12.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:14:11 smithi115 ceph-mon[78593]: pgmap v5344: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:14:12.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:14:11 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:14:12.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:14:11 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:14:12.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:14:11 smithi047 ceph-mon[77147]: pgmap v5344: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:14:12.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:14:11 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:14:13.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:14:12 smithi115 conmon[90393]: 2022-05-14T02:14:12.925+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:14:13.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:14:12 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:14:13.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:14:12 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:14:14.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:14:13 smithi115 conmon[90393]: 2022-05-14T02:14:13.927+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:14:14.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:14:13 smithi115 ceph-mon[78593]: pgmap v5345: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:14:14.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:14:13 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:14:14.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:14:13 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:14:14.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:14:13 smithi047 ceph-mon[77147]: pgmap v5345: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:14:14.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:14:13 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:14:14.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:14:13 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:14:15.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:14:14 smithi115 conmon[90393]: 2022-05-14T02:14:14.931+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:14:15.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:14:14 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:14:15.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:14:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T02:14:15.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:14:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T02:14:15.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:14:14 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:14:15.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:14:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T02:14:15.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:14:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T02:14:16.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:14:15 smithi115 conmon[90393]: 2022-05-14T02:14:15.948+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:14:16.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:14:15 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:14:16.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:14:15 smithi115 ceph-mon[78593]: pgmap v5346: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:14:16.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:14:15 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 9990 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:14:16.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:14:15 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:14:16.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:14:15 smithi047 ceph-mon[77147]: pgmap v5346: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:14:16.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:14:15 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 9990 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:14:17.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:14:16 smithi115 conmon[90393]: 2022-05-14T02:14:16.918+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:14:17.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:14:16 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:14:17.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:14:16 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:14:18.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:14:17 smithi115 conmon[90393]: 2022-05-14T02:14:17.885+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:14:18.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:14:17 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:14:18.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:14:17 smithi115 ceph-mon[78593]: pgmap v5347: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:14:18.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:14:17 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:14:18.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:14:17 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:14:18.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:14:17 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:14:18.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:14:17 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:14:18.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:14:17 smithi047 ceph-mon[77147]: pgmap v5347: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:14:18.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:14:17 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:14:18.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:14:17 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:14:18.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:14:17 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:14:19.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:14:18 smithi115 conmon[90393]: 2022-05-14T02:14:18.862+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:14:19.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:14:18 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:14:19.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:14:18 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:14:20.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:14:19 smithi115 conmon[90393]: 2022-05-14T02:14:19.844+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:14:20.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:14:19 smithi115 ceph-mon[78593]: pgmap v5348: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:14:20.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:14:19 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:14:20.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:14:19 smithi047 ceph-mon[77147]: pgmap v5348: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:14:20.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:14:19 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:14:21.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:14:20 smithi115 conmon[90393]: 2022-05-14T02:14:20.813+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:14:21.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:14:20 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:14:21.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:14:20 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:14:22.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:14:21 smithi115 conmon[90393]: 2022-05-14T02:14:21.856+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:14:22.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:14:21 smithi115 ceph-mon[78593]: pgmap v5349: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:14:22.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:14:21 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:14:22.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:14:21 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 9995 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:14:22.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:14:21 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:14:22.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:14:21 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:14:22.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:14:21 smithi047 ceph-mon[77147]: pgmap v5349: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:14:22.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:14:21 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:14:22.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:14:21 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 9995 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:14:22.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:14:21 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:14:22.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:14:21 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:14:23.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:14:22 smithi115 conmon[90393]: 2022-05-14T02:14:22.820+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:14:23.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:14:22 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:14:23.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:14:22 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:14:24.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:14:23 smithi115 conmon[90393]: 2022-05-14T02:14:23.790+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:14:24.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:14:23 smithi115 ceph-mon[78593]: pgmap v5350: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:14:24.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:14:23 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:14:24.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:14:23 smithi047 ceph-mon[77147]: pgmap v5350: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:14:24.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:14:23 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:14:25.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:14:24 smithi115 conmon[90393]: 2022-05-14T02:14:24.829+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:14:25.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:14:24 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:14:25.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:14:24 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:14:26.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:14:25 smithi115 conmon[90393]: 2022-05-14T02:14:25.838+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:14:26.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:14:25 smithi115 ceph-mon[78593]: pgmap v5351: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:14:26.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:14:25 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:14:26.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:14:25 smithi047 ceph-mon[77147]: pgmap v5351: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:14:26.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:14:25 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:14:27.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:14:26 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:14:27.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:14:26 smithi115 conmon[90393]: 2022-05-14T02:14:26.872+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:14:27.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:14:26 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:14:28.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:14:27 smithi115 conmon[90393]: 2022-05-14T02:14:27.844+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:14:28.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:14:27 smithi115 ceph-mon[78593]: pgmap v5352: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:14:28.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:14:27 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:14:28.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:14:27 smithi047 ceph-mon[77147]: pgmap v5352: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:14:28.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:14:27 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:14:29.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:14:28 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:14:29.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:14:28 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T02:14:29.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:14:28 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T02:14:29.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:14:28 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T02:14:29.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:14:28 smithi115 conmon[90393]: 2022-05-14T02:14:28.894+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:14:29.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:14:28 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:14:29.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:14:28 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T02:14:29.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:14:28 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T02:14:29.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:14:28 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T02:14:29.542 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:14:29 smithi047 conmon[78521]: 2022-05-14T02:14:29.103+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:14:29.542 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:14:29 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T02:14:29.542 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:14:29 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:14:29.542 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:14:29 smithi047 conmon[78521]: yield 2022-05-14T02:14:29.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:14:29 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:14:29.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:14:29 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:14:29.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:14:29 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:14:29.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:14:29 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T02:14:29.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:14:29 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:14:29.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:14:29 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T02:14:29.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:14:29 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:14:29.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:14:29 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T02:14:29.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:14:29 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:14:29.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:14:29 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T02:14:29.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:14:29 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:14:29.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:14:29 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T02:14:29.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:14:29 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:14:29.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:14:29 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:14:29.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:14:29 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:14:30.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:14:29 smithi047 ceph-mon[77147]: pgmap v5353: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:14:30.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:14:29 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:14:30.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:14:29 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:14:30.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:14:29 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:14:30.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:14:29 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:14:30.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T02:14:30.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:14:30.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T02:14:30.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:14:30.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:14:30.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:14:30.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T02:14:30.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:14:30.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T02:14:30.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:14:30.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T02:14:30.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:14:30.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T02:14:30.298 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:14:30.298 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T02:14:30.298 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:14:30.298 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:14:30.298 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:14:30.299 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:14:29 smithi047 ceph-mon[77147]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T02:14:30.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:14:29 smithi115 conmon[90393]: 2022-05-14T02:14:29.879+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:14:30.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:14:29 smithi115 ceph-mon[78593]: pgmap v5353: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:14:30.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:14:29 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:14:30.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:14:29 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:14:30.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:14:29 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:14:30.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:14:29 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:14:30.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T02:14:30.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:14:30.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T02:14:30.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:14:30.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:14:30.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:14:30.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T02:14:30.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:14:30.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T02:14:30.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:14:30.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T02:14:30.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:14:30.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T02:14:30.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:14:30.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T02:14:30.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:14:30.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:14:30.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:14:30.364 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:14:29 smithi115 ceph-mon[78593]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T02:14:31.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:14:30 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:14:31.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:14:30 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 10000 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:14:31.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:14:30 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:14:31.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:14:30 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 10000 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:14:31.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:14:30 smithi115 conmon[90393]: 2022-05-14T02:14:30.858+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:14:32.177 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:14:31 smithi047 ceph-mon[77147]: pgmap v5354: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:14:32.178 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:14:31 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:14:32.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:14:31 smithi115 ceph-mon[78593]: pgmap v5354: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:14:32.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:14:31 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:14:32.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:14:31 smithi115 conmon[90393]: 2022-05-14T02:14:31.894+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:14:32.542 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:14:32 smithi047 conmon[78521]: 2022-05-14T02:14:32.175+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:14:32.542 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:14:32 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T02:14:32.542 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:14:32 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:14:32.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:14:32 smithi047 conmon[78521]: yield 2022-05-14T02:14:32.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:14:32 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:14:32.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:14:32 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:14:32.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:14:32 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:14:32.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:14:32 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T02:14:32.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:14:32 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:14:32.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:14:32 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T02:14:32.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:14:32 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:14:32.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:14:32 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T02:14:32.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:14:32 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:14:32.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:14:32 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T02:14:32.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:14:32 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:14:32.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:14:32 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T02:14:32.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:14:32 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:14:32.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:14:32 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:14:32.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:14:32 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:14:32.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:14:32 smithi047 conmon[78521]: 2022-05-14T02:14:32.176+0000 7f5345f84700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T02:14:32.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:14:32 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T02:14:32.547 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:14:32 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:14:32.547 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:14:32 smithi047 conmon[78521]: yield 2022-05-14T02:14:32.547 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:14:32 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:14:32.547 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:14:32 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:14:32.548 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:14:32 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:14:32.548 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:14:32 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T02:14:32.548 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:14:32 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:14:32.548 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:14:32 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T02:14:32.548 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:14:32 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:14:32.549 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:14:32 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T02:14:32.549 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:14:32 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:14:32.549 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:14:32 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T02:14:32.549 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:14:32 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:14:32.550 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:14:32 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T02:14:32.550 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:14:32 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:14:32.550 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:14:32 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:14:32.550 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:14:32 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:14:32.551 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:14:32 smithi047 conmon[78521]: 2022-05-14T02:14:32.551 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:14:32 smithi047 conmon[78521]: During handling of the above exception, another exception occurred: 2022-05-14T02:14:32.551 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:14:32 smithi047 conmon[78521]: 2022-05-14T02:14:32.551 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:14:32 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T02:14:32.552 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:14:32 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T02:14:32.552 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:14:32 smithi047 conmon[78521]: return f(*arg) 2022-05-14T02:14:32.552 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:14:32 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T02:14:32.552 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:14:32 smithi047 conmon[78521]: self._write_client_files(client_files, host) 2022-05-14T02:14:32.552 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:14:32 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T02:14:32.553 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:14:32 smithi047 conmon[78521]: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T02:14:32.553 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:14:32 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T02:14:32.553 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:14:32 smithi047 conmon[78521]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T02:14:32.553 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:14:32 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T02:14:32.553 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:14:32 smithi047 conmon[78521]: return self.event_loop.get_result(coro) 2022-05-14T02:14:32.554 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:14:32 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T02:14:32.554 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:14:32 smithi047 conmon[78521]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T02:14:32.554 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:14:32 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T02:14:32.554 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:14:32 smithi047 conmon[78521]: return self.__get_result() 2022-05-14T02:14:32.555 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:14:32 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T02:14:32.555 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:14:32 smithi047 conmon[78521]: raise self._exception 2022-05-14T02:14:32.555 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:14:32 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T02:14:32.555 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:14:32 smithi047 conmon[78521]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T02:14:32.555 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:14:32 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T02:14:32.556 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:14:32 smithi047 conmon[78521]: conn = await self._remote_connection(host, addr) 2022-05-14T02:14:32.556 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:14:32 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T02:14:32.556 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:14:32 smithi047 conmon[78521]: raise 2022-05-14T02:14:32.556 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:14:32 smithi047 conmon[78521]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T02:14:32.556 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:14:32 smithi047 conmon[78521]: self.gen.throw(type, value, traceback) 2022-05-14T02:14:32.557 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:14:32 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T02:14:32.557 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:14:32 smithi047 conmon[78521]: raise OrchestratorError(msg) 2022-05-14T02:14:32.557 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:14:32 smithi047 conmon[78521]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:14:33.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:14:32 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:14:33.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:14:32 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:14:33.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:14:32 smithi115 conmon[90393]: 2022-05-14T02:14:32.859+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:14:34.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:14:33 smithi115 conmon[90393]: 2022-05-14T02:14:33.836+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:14:34.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:14:33 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:14:34.109 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T02:14:34.109 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:14:34.109 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T02:14:34.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:14:34.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:14:34.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:14:34.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T02:14:34.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:14:34.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T02:14:34.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:14:34.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T02:14:34.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:14:34.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T02:14:34.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:14:34.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T02:14:34.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:14:34.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:14:34.113 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:14:34.113 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:14:33 smithi115 ceph-mon[78593]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T02:14:34.113 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T02:14:34.113 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:14:34.113 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T02:14:34.114 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:14:34.114 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:14:34.114 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:14:34.114 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T02:14:34.115 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:14:34.115 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T02:14:34.115 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:14:34.115 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T02:14:34.115 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:14:34.116 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T02:14:34.116 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:14:34.116 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T02:14:34.116 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:14:34.116 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:14:34.117 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:14:34.117 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T02:14:34.117 INFO:journalctl@ceph.mon.c.smithi115.stdout: During handling of the above exception, another exception occurred: 2022-05-14T02:14:34.117 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T02:14:34.118 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T02:14:34.118 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T02:14:34.118 INFO:journalctl@ceph.mon.c.smithi115.stdout: return f(*arg) 2022-05-14T02:14:34.118 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T02:14:34.118 INFO:journalctl@ceph.mon.c.smithi115.stdout: self._write_client_files(client_files, host) 2022-05-14T02:14:34.119 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T02:14:34.119 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T02:14:34.119 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T02:14:34.119 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T02:14:34.120 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T02:14:34.120 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.event_loop.get_result(coro) 2022-05-14T02:14:34.120 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T02:14:34.120 INFO:journalctl@ceph.mon.c.smithi115.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T02:14:34.120 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T02:14:34.121 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.__get_result() 2022-05-14T02:14:34.121 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T02:14:34.121 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise self._exception 2022-05-14T02:14:34.121 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T02:14:34.121 INFO:journalctl@ceph.mon.c.smithi115.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T02:14:34.122 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T02:14:34.122 INFO:journalctl@ceph.mon.c.smithi115.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T02:14:34.122 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T02:14:34.123 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise 2022-05-14T02:14:34.123 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T02:14:34.123 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.gen.throw(type, value, traceback) 2022-05-14T02:14:34.123 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T02:14:34.124 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OrchestratorError(msg) 2022-05-14T02:14:34.124 INFO:journalctl@ceph.mon.c.smithi115.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:14:34.124 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:14:33 smithi115 ceph-mon[78593]: pgmap v5355: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:14:34.124 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:14:33 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:14:34.124 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:14:33 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:14:34.125 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:14:33 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:14:34.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:14:33 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:14:34.292 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T02:14:34.292 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:14:34.292 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T02:14:34.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:14:34.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:14:34.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:14:34.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T02:14:34.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:14:34.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T02:14:34.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:14:34.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T02:14:34.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:14:34.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T02:14:34.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:14:34.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T02:14:34.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:14:34.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:14:34.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:14:34.297 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:14:33 smithi047 ceph-mon[77147]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T02:14:34.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T02:14:34.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:14:34.298 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T02:14:34.298 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:14:34.298 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:14:34.298 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:14:34.299 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T02:14:34.299 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:14:34.299 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T02:14:34.299 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:14:34.300 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T02:14:34.300 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:14:34.300 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T02:14:34.300 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:14:34.301 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T02:14:34.301 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:14:34.301 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:14:34.301 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:14:34.302 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T02:14:34.302 INFO:journalctl@ceph.mon.b.smithi047.stdout: During handling of the above exception, another exception occurred: 2022-05-14T02:14:34.302 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T02:14:34.302 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T02:14:34.302 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T02:14:34.303 INFO:journalctl@ceph.mon.b.smithi047.stdout: return f(*arg) 2022-05-14T02:14:34.303 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T02:14:34.303 INFO:journalctl@ceph.mon.b.smithi047.stdout: self._write_client_files(client_files, host) 2022-05-14T02:14:34.304 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T02:14:34.304 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T02:14:34.304 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T02:14:34.304 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T02:14:34.305 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T02:14:34.305 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.event_loop.get_result(coro) 2022-05-14T02:14:34.305 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T02:14:34.305 INFO:journalctl@ceph.mon.b.smithi047.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T02:14:34.306 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T02:14:34.306 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.__get_result() 2022-05-14T02:14:34.306 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T02:14:34.306 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise self._exception 2022-05-14T02:14:34.306 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T02:14:34.307 INFO:journalctl@ceph.mon.b.smithi047.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T02:14:34.307 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T02:14:34.307 INFO:journalctl@ceph.mon.b.smithi047.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T02:14:34.307 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T02:14:34.308 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise 2022-05-14T02:14:34.308 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T02:14:34.308 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.gen.throw(type, value, traceback) 2022-05-14T02:14:34.309 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T02:14:34.309 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OrchestratorError(msg) 2022-05-14T02:14:34.309 INFO:journalctl@ceph.mon.b.smithi047.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:14:34.309 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:14:33 smithi047 ceph-mon[77147]: pgmap v5355: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:14:34.310 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:14:33 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:14:34.310 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:14:33 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:14:34.310 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:14:33 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:14:35.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:14:34 smithi115 conmon[90393]: 2022-05-14T02:14:34.821+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:14:35.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:14:34 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:14:35.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:14:34 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:14:36.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:14:35 smithi115 conmon[90393]: 2022-05-14T02:14:35.777+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:14:36.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:14:35 smithi115 ceph-mon[78593]: pgmap v5356: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:14:36.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:14:35 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:14:36.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:14:35 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 10010 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:14:36.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:14:35 smithi047 ceph-mon[77147]: pgmap v5356: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:14:36.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:14:35 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:14:36.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:14:35 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 10010 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:14:37.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:14:36 smithi115 conmon[90393]: 2022-05-14T02:14:36.744+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:14:37.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:14:36 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:14:37.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:14:36 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:14:38.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:14:37 smithi115 conmon[90393]: 2022-05-14T02:14:37.699+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:14:38.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:14:37 smithi115 ceph-mon[78593]: pgmap v5357: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:14:38.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:14:37 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:14:38.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:14:37 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:14:38.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:14:37 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:14:38.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:14:37 smithi047 ceph-mon[77147]: pgmap v5357: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:14:38.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:14:37 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:14:38.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:14:37 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:14:38.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:14:37 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:14:39.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:14:38 smithi115 conmon[90393]: 2022-05-14T02:14:38.709+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:14:39.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:14:38 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:14:39.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:14:38 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:14:40.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:14:39 smithi115 conmon[90393]: 2022-05-14T02:14:39.753+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:14:40.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:14:39 smithi115 ceph-mon[78593]: pgmap v5358: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:14:40.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:14:39 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:14:40.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:14:39 smithi047 ceph-mon[77147]: pgmap v5358: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:14:40.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:14:39 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:14:41.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:14:40 smithi115 conmon[90393]: 2022-05-14T02:14:40.745+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:14:41.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:14:40 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:14:41.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:14:40 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:14:42.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:14:41 smithi115 conmon[90393]: 2022-05-14T02:14:41.749+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:14:42.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:14:41 smithi115 ceph-mon[78593]: pgmap v5359: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:14:42.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:14:41 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:14:42.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:14:41 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 10015 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:14:42.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:14:41 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:14:42.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:14:41 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:14:42.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:14:41 smithi047 ceph-mon[77147]: pgmap v5359: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:14:42.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:14:41 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:14:42.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:14:41 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 10015 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:14:42.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:14:41 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:14:42.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:14:41 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:14:43.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:14:42 smithi115 conmon[90393]: 2022-05-14T02:14:42.798+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:14:43.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:14:42 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:14:43.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:14:42 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:14:44.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:14:43 smithi115 conmon[90393]: 2022-05-14T02:14:43.762+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:14:44.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:14:43 smithi115 ceph-mon[78593]: pgmap v5360: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:14:44.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:14:43 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:14:44.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:14:43 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:14:44.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:14:43 smithi047 ceph-mon[77147]: pgmap v5360: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:14:44.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:14:43 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:14:44.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:14:43 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:14:45.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:14:44 smithi115 conmon[90393]: 2022-05-14T02:14:44.804+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:14:45.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:14:44 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:14:45.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:14:44 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:14:46.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:14:45 smithi115 conmon[90393]: 2022-05-14T02:14:45.834+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:14:46.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:14:45 smithi115 ceph-mon[78593]: pgmap v5361: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:14:46.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:14:45 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:14:46.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:14:45 smithi047 ceph-mon[77147]: pgmap v5361: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:14:46.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:14:45 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:14:47.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:14:46 smithi115 conmon[90393]: 2022-05-14T02:14:46.796+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:14:47.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:14:46 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:14:47.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:14:46 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:14:48.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:14:47 smithi115 conmon[90393]: 2022-05-14T02:14:47.791+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:14:48.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:14:47 smithi115 ceph-mon[78593]: pgmap v5362: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:14:48.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:14:47 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:14:48.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:14:47 smithi047 ceph-mon[77147]: pgmap v5362: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:14:48.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:14:47 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:14:49.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:14:48 smithi115 conmon[90393]: 2022-05-14T02:14:48.777+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:14:49.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:14:48 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:14:49.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:14:48 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:14:50.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:14:49 smithi115 conmon[90393]: 2022-05-14T02:14:49.818+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:14:50.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:14:49 smithi115 ceph-mon[78593]: pgmap v5363: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:14:50.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:14:49 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:14:50.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:14:49 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:14:50.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:14:49 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:14:50.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:14:49 smithi047 ceph-mon[77147]: pgmap v5363: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:14:50.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:14:49 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:14:50.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:14:49 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:14:50.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:14:49 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:14:51.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:14:50 smithi115 conmon[90393]: 2022-05-14T02:14:50.784+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:14:51.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:14:50 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 10020 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:14:51.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:14:50 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:14:51.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:14:50 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 10020 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:14:51.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:14:50 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:14:52.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:14:51 smithi115 conmon[90393]: 2022-05-14T02:14:51.740+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:14:52.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:14:51 smithi115 ceph-mon[78593]: pgmap v5364: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:14:52.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:14:51 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:14:52.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:14:51 smithi047 ceph-mon[77147]: pgmap v5364: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:14:52.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:14:51 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:14:53.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:14:52 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:14:53.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:14:52 smithi115 conmon[90393]: 2022-05-14T02:14:52.725+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:14:53.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:14:52 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:14:53.963 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:14:53 smithi115 conmon[90393]: 2022-05-14T02:14:53.697+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:14:54.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:14:53 smithi047 ceph-mon[77147]: pgmap v5365: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:14:54.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:14:53 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:14:54.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:14:53 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:14:54.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:14:53 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:14:54.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:14:53 smithi115 ceph-mon[78593]: pgmap v5365: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:14:54.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:14:53 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:14:54.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:14:53 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:14:54.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:14:53 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:14:55.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:14:54 smithi115 conmon[90393]: 2022-05-14T02:14:54.741+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:14:55.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:14:54 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:14:55.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:14:54 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:14:56.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:14:55 smithi115 conmon[90393]: 2022-05-14T02:14:55.750+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:14:56.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:14:55 smithi115 ceph-mon[78593]: pgmap v5366: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:14:56.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:14:55 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 10030 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:14:56.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:14:55 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:14:56.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:14:55 smithi047 ceph-mon[77147]: pgmap v5366: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:14:56.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:14:55 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 10030 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:14:56.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:14:55 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:14:57.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:14:56 smithi115 conmon[90393]: 2022-05-14T02:14:56.759+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:14:57.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:14:56 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:14:57.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:14:56 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:14:58.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:14:57 smithi115 conmon[90393]: 2022-05-14T02:14:57.739+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:14:58.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:14:57 smithi115 ceph-mon[78593]: pgmap v5367: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:14:58.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:14:57 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:14:58.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:14:57 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:14:58.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:14:57 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:14:58.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:14:57 smithi047 ceph-mon[77147]: pgmap v5367: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:14:58.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:14:57 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:14:58.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:14:57 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:14:58.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:14:57 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:14:59.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:14:58 smithi115 conmon[90393]: 2022-05-14T02:14:58.781+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:14:59.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:14:58 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:14:59.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:14:58 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:15:00.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:14:59 smithi115 conmon[90393]: 2022-05-14T02:14:59.789+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:15:00.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:14:59 smithi115 ceph-mon[78593]: pgmap v5368: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:15:00.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:14:59 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:15:00.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:14:59 smithi047 ceph-mon[77147]: pgmap v5368: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:15:00.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:14:59 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:15:01.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:15:00 smithi115 conmon[90393]: 2022-05-14T02:15:00.820+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:15:01.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:15:01 smithi047 ceph-mon[77147]: pgmap v5369: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:15:01.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:15:01 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:15:01.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:15:01 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 10035 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:15:01.828 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:15:01 smithi115 ceph-mon[78593]: pgmap v5369: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:15:01.829 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:15:01 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:15:01.829 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:15:01 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 10035 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:15:02.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:15:01 smithi115 conmon[90393]: 2022-05-14T02:15:01.826+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:15:02.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:15:02 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:15:02.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:15:02 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:15:02.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:15:02 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:15:02.830 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:15:02 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:15:02.831 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:15:02 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:15:02.831 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:15:02 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:15:03.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:15:02 smithi115 conmon[90393]: 2022-05-14T02:15:02.828+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:15:03.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:15:03 smithi047 ceph-mon[77147]: pgmap v5370: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:15:03.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:15:03 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:15:03.822 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:15:03 smithi115 ceph-mon[78593]: pgmap v5370: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:15:03.822 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:15:03 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:15:04.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:15:03 smithi115 conmon[90393]: 2022-05-14T02:15:03.820+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:15:04.778 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:15:04 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:15:04.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:15:04 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:15:05.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:15:04 smithi115 conmon[90393]: 2022-05-14T02:15:04.776+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:15:05.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:15:05 smithi047 ceph-mon[77147]: pgmap v5371: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:15:05.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:15:05 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:15:05.807 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:15:05 smithi115 ceph-mon[78593]: pgmap v5371: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:15:05.807 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:15:05 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:15:06.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:15:05 smithi115 conmon[90393]: 2022-05-14T02:15:05.805+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:15:06.766 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:15:06 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:15:06.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:15:06 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:15:07.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:15:06 smithi115 conmon[90393]: 2022-05-14T02:15:06.764+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:15:07.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:15:07 smithi047 ceph-mon[77147]: pgmap v5372: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:15:07.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:15:07 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:15:07.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:15:07 smithi115 conmon[90393]: 2022-05-14T02:15:07.733+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:15:07.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:15:07 smithi115 ceph-mon[78593]: pgmap v5372: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:15:07.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:15:07 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:15:08.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:15:08 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:15:08.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:15:08 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:15:08.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:15:08 smithi115 conmon[90393]: 2022-05-14T02:15:08.756+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:15:09.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:15:09 smithi047 ceph-mon[77147]: pgmap v5373: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:15:09.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:15:09 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:15:09.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:15:09 smithi115 conmon[90393]: 2022-05-14T02:15:09.759+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:15:09.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:15:09 smithi115 ceph-mon[78593]: pgmap v5373: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:15:09.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:15:09 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:15:10.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:15:10 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:15:10.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:15:10 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:15:10.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:15:10 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:15:10.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:15:10 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 10040 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:15:10.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:15:10 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:15:10.799 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:15:10 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:15:10.799 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:15:10 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:15:10.799 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:15:10 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 10040 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:15:11.115 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:15:10 smithi115 conmon[90393]: 2022-05-14T02:15:10.796+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:15:11.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:15:11 smithi047 ceph-mon[77147]: pgmap v5374: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:15:11.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:15:11 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:15:11.815 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:15:11 smithi115 ceph-mon[78593]: pgmap v5374: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:15:11.815 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:15:11 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:15:12.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:15:11 smithi115 conmon[90393]: 2022-05-14T02:15:11.813+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:15:12.789 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:15:12 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:15:12.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:15:12 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:15:13.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:15:12 smithi115 conmon[90393]: 2022-05-14T02:15:12.787+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:15:13.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:15:13 smithi047 ceph-mon[77147]: pgmap v5375: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:15:13.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:15:13 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:15:13.800 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:15:13 smithi115 ceph-mon[78593]: pgmap v5375: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:15:13.801 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:15:13 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:15:14.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:15:13 smithi115 conmon[90393]: 2022-05-14T02:15:13.799+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:15:14.820 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:15:14 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:15:14.820 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:15:14 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:15:14.820 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:15:14 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:15:14.821 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:15:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T02:15:14.821 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:15:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T02:15:15.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:15:14 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:15:15.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:15:14 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:15:15.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:15:14 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:15:15.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:15:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T02:15:15.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:15:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T02:15:15.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:15:14 smithi115 conmon[90393]: 2022-05-14T02:15:14.818+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:15:15.856 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:15:15 smithi115 ceph-mon[78593]: pgmap v5376: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:15:15.856 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:15:15 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:15:15.856 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:15:15 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 10050 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:15:16.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:15:15 smithi047 ceph-mon[77147]: pgmap v5376: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:15:16.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:15:15 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:15:16.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:15:15 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 10050 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:15:16.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:15:15 smithi115 conmon[90393]: 2022-05-14T02:15:15.854+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:15:16.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:15:16 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:15:17.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:15:16 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:15:17.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:15:16 smithi115 conmon[90393]: 2022-05-14T02:15:16.872+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:15:17.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:15:17 smithi115 ceph-mon[78593]: pgmap v5377: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:15:17.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:15:17 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:15:18.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:15:17 smithi047 ceph-mon[77147]: pgmap v5377: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:15:18.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:15:17 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:15:18.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:15:17 smithi115 conmon[90393]: 2022-05-14T02:15:17.901+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:15:18.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:15:18 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:15:18.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:15:18 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:15:18.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:15:18 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:15:19.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:15:18 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:15:19.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:15:18 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:15:19.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:15:18 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:15:19.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:15:18 smithi115 conmon[90393]: 2022-05-14T02:15:18.943+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:15:19.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:15:19 smithi115 ceph-mon[78593]: pgmap v5378: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:15:20.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:15:19 smithi047 ceph-mon[77147]: pgmap v5378: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:15:20.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:15:19 smithi115 conmon[90393]: 2022-05-14T02:15:19.927+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:15:20.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:15:20 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:15:20.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:15:20 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:15:21.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:15:20 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:15:21.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:15:20 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:15:21.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:15:20 smithi115 conmon[90393]: 2022-05-14T02:15:20.952+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:15:21.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:15:21 smithi115 ceph-mon[78593]: pgmap v5379: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:15:21.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:15:21 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 10055 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:15:22.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:15:21 smithi047 ceph-mon[77147]: pgmap v5379: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:15:22.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:15:21 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 10055 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:15:22.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:15:21 smithi115 conmon[90393]: 2022-05-14T02:15:21.932+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:15:22.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:15:22 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:15:22.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:15:22 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:15:22.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:15:22 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:15:23.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:15:22 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:15:23.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:15:22 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:15:23.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:15:22 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:15:23.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:15:22 smithi115 conmon[90393]: 2022-05-14T02:15:22.981+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:15:23.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:15:23 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:15:23.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:15:23 smithi115 ceph-mon[78593]: pgmap v5380: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:15:24.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:15:23 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:15:24.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:15:23 smithi047 ceph-mon[77147]: pgmap v5380: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:15:24.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:15:23 smithi115 conmon[90393]: 2022-05-14T02:15:23.985+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:15:24.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:15:24 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:15:25.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:15:24 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:15:25.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:15:24 smithi115 conmon[90393]: 2022-05-14T02:15:24.978+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:15:25.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:15:25 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:15:25.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:15:25 smithi115 ceph-mon[78593]: pgmap v5381: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:15:26.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:15:25 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:15:26.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:15:25 smithi047 ceph-mon[77147]: pgmap v5381: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:15:26.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:15:25 smithi115 conmon[90393]: 2022-05-14T02:15:25.947+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:15:26.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:15:26 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:15:27.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:15:26 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:15:27.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:15:26 smithi115 conmon[90393]: 2022-05-14T02:15:26.956+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:15:27.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:15:27 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:15:27.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:15:27 smithi115 ceph-mon[78593]: pgmap v5382: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:15:28.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:15:27 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:15:28.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:15:27 smithi047 ceph-mon[77147]: pgmap v5382: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:15:28.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:15:27 smithi115 conmon[90393]: 2022-05-14T02:15:27.982+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:15:28.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:15:28 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:15:29.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:15:28 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:15:29.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:15:28 smithi115 conmon[90393]: 2022-05-14T02:15:28.965+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:15:29.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:15:29 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:15:29.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:15:29 smithi115 ceph-mon[78593]: pgmap v5383: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:15:30.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:15:29 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:15:30.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:15:29 smithi047 ceph-mon[77147]: pgmap v5383: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:15:30.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:15:29 smithi115 conmon[90393]: 2022-05-14T02:15:29.932+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:15:30.968 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:15:30 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:15:30.968 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:15:30 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:15:30.968 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:15:30 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:15:30.969 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:15:30 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:15:30.969 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:15:30 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 10060 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:15:31.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:15:30 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:15:31.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:15:30 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:15:31.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:15:30 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:15:31.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:15:30 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:15:31.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:15:30 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 10060 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:15:31.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:15:30 smithi115 conmon[90393]: 2022-05-14T02:15:30.966+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:15:31.971 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:15:31 smithi115 ceph-mon[78593]: pgmap v5384: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:15:32.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:15:31 smithi047 ceph-mon[77147]: pgmap v5384: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:15:32.224 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:15:31 smithi115 conmon[90393]: 2022-05-14T02:15:31.969+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:15:32.976 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:15:32 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:15:32.977 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:15:32 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T02:15:32.977 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:15:32 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T02:15:32.977 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:15:32 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T02:15:33.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:15:32 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:15:33.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:15:32 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T02:15:33.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:15:32 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T02:15:33.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:15:32 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T02:15:33.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:15:32 smithi115 conmon[90393]: 2022-05-14T02:15:32.974+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:15:33.713 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:15:33 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:15:33.714 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:15:33 smithi047 ceph-mon[77147]: pgmap v5385: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:15:33.751 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:15:33 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:15:33.752 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:15:33 smithi115 ceph-mon[78593]: pgmap v5385: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:15:34.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:15:33 smithi115 conmon[90393]: 2022-05-14T02:15:33.949+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:15:34.937 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:15:34 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:15:34.938 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:15:34 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:15:34.938 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:15:34 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:15:34.987 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:15:34 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:15:34.987 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:15:34 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:15:34.987 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:15:34 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:15:35.268 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:15:34 smithi115 conmon[90393]: 2022-05-14T02:15:34.935+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:15:35.541 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:15:35 smithi047 conmon[78521]: 2022-05-14T02:15:35.279+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:15:35.542 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:15:35 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T02:15:35.542 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:15:35 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:15:35.542 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:15:35 smithi047 conmon[78521]: yield 2022-05-14T02:15:35.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:15:35 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:15:35.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:15:35 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:15:35.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:15:35 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:15:35.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:15:35 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T02:15:35.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:15:35 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:15:35.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:15:35 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T02:15:35.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:15:35 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:15:35.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:15:35 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T02:15:35.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:15:35 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:15:35.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:15:35 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T02:15:35.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:15:35 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:15:35.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:15:35 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T02:15:35.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:15:35 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:15:35.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:15:35 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:15:35.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:15:35 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:15:35.979 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:15:35 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:15:35.980 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:15:35 smithi115 ceph-mon[78593]: pgmap v5386: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:15:35.980 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:15:35 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' 2022-05-14T02:15:35.980 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:15:35 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 10070 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:15:35.981 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:15:35 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' 2022-05-14T02:15:36.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:15:35 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:15:36.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:15:35 smithi047 ceph-mon[77147]: pgmap v5386: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:15:36.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:15:35 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' 2022-05-14T02:15:36.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:15:35 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 10070 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:15:36.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:15:35 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' 2022-05-14T02:15:36.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:15:35 smithi115 conmon[90393]: 2022-05-14T02:15:35.978+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:15:37.017 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:15:36 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:15:37.017 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:15:36 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:15:37.018 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T02:15:37.018 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:15:37.018 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T02:15:37.018 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:15:37.018 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:15:37.019 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:15:37.019 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T02:15:37.019 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:15:37.019 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T02:15:37.020 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:15:37.020 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T02:15:37.020 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:15:37.020 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T02:15:37.020 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:15:37.021 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T02:15:37.021 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:15:37.021 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:15:37.021 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:15:37.022 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:15:36 smithi115 ceph-mon[78593]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T02:15:37.022 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:15:36 smithi115 ceph-mon[78593]: Detected new or changed devices on smithi047 2022-05-14T02:15:37.022 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:15:36 smithi115 ceph-mon[78593]: Detected new or changed devices on smithi115 2022-05-14T02:15:37.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:15:36 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:15:37.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:15:36 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:15:37.042 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T02:15:37.042 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:15:37.042 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T02:15:37.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:15:37.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:15:37.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:15:37.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T02:15:37.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:15:37.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T02:15:37.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:15:37.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T02:15:37.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:15:37.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T02:15:37.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:15:37.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T02:15:37.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:15:37.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:15:37.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:15:37.046 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:15:36 smithi047 ceph-mon[77147]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T02:15:37.046 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:15:36 smithi047 ceph-mon[77147]: Detected new or changed devices on smithi047 2022-05-14T02:15:37.047 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:15:36 smithi047 ceph-mon[77147]: Detected new or changed devices on smithi115 2022-05-14T02:15:37.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:15:37 smithi115 conmon[90393]: 2022-05-14T02:15:37.016+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:15:38.001 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:15:37 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:15:38.001 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:15:37 smithi115 ceph-mon[78593]: pgmap v5387: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:15:38.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:15:37 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:15:38.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:15:37 smithi047 ceph-mon[77147]: pgmap v5387: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:15:38.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:15:38 smithi115 conmon[90393]: 2022-05-14T02:15:37.999+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:15:38.646 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:15:38 smithi047 conmon[78521]: 2022-05-14T02:15:38.352+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:15:38.646 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:15:38 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T02:15:38.647 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:15:38 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:15:38.647 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:15:38 smithi047 conmon[78521]: yield 2022-05-14T02:15:38.647 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:15:38 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:15:38.647 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:15:38 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:15:38.648 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:15:38 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:15:38.648 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:15:38 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T02:15:38.648 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:15:38 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:15:38.648 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:15:38 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T02:15:38.649 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:15:38 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:15:38.649 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:15:38 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T02:15:38.649 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:15:38 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:15:38.649 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:15:38 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T02:15:38.650 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:15:38 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:15:38.650 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:15:38 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T02:15:38.650 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:15:38 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:15:38.651 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:15:38 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:15:38.651 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:15:38 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:15:38.651 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:15:38 smithi047 conmon[78521]: 2022-05-14T02:15:38.353+0000 7f5340779700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T02:15:38.651 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:15:38 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T02:15:38.651 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:15:38 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:15:38.652 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:15:38 smithi047 conmon[78521]: yield 2022-05-14T02:15:38.652 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:15:38 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:15:38.652 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:15:38 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:15:38.652 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:15:38 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:15:38.653 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:15:38 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T02:15:38.653 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:15:38 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:15:38.653 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:15:38 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T02:15:38.653 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:15:38 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:15:38.653 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:15:38 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T02:15:38.654 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:15:38 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:15:38.654 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:15:38 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T02:15:38.654 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:15:38 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:15:38.655 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:15:38 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T02:15:38.655 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:15:38 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:15:38.655 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:15:38 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:15:38.655 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:15:38 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:15:38.656 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:15:38 smithi047 conmon[78521]: 2022-05-14T02:15:38.656 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:15:38 smithi047 conmon[78521]: During handling of the above exception, another exception occurred: 2022-05-14T02:15:38.656 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:15:38 smithi047 conmon[78521]: 2022-05-14T02:15:38.656 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:15:38 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T02:15:38.656 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:15:38 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T02:15:38.657 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:15:38 smithi047 conmon[78521]: return f(*arg) 2022-05-14T02:15:38.657 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:15:38 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T02:15:38.657 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:15:38 smithi047 conmon[78521]: self._write_client_files(client_files, host) 2022-05-14T02:15:38.658 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:15:38 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T02:15:38.658 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:15:38 smithi047 conmon[78521]: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T02:15:38.658 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:15:38 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T02:15:38.658 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:15:38 smithi047 conmon[78521]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T02:15:38.659 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:15:38 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T02:15:38.659 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:15:38 smithi047 conmon[78521]: return self.event_loop.get_result(coro) 2022-05-14T02:15:38.659 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:15:38 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T02:15:38.659 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:15:38 smithi047 conmon[78521]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T02:15:38.660 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:15:38 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T02:15:38.660 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:15:38 smithi047 conmon[78521]: return self.__get_result() 2022-05-14T02:15:38.660 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:15:38 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T02:15:38.660 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:15:38 smithi047 conmon[78521]: raise self._exception 2022-05-14T02:15:38.660 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:15:38 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T02:15:38.661 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:15:38 smithi047 conmon[78521]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T02:15:38.661 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:15:38 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T02:15:38.661 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:15:38 smithi047 conmon[78521]: conn = await self._remote_connection(host, addr) 2022-05-14T02:15:38.661 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:15:38 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T02:15:38.662 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:15:38 smithi047 conmon[78521]: raise 2022-05-14T02:15:38.662 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:15:38 smithi047 conmon[78521]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T02:15:38.662 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:15:38 smithi047 conmon[78521]: self.gen.throw(type, value, traceback) 2022-05-14T02:15:38.662 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:15:38 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T02:15:38.663 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:15:38 smithi047 conmon[78521]: raise OrchestratorError(msg) 2022-05-14T02:15:38.663 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:15:38 smithi047 conmon[78521]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:15:38.984 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:15:38 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:15:38.984 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:15:38 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:15:38.985 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:15:38 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:15:39.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:15:38 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:15:39.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:15:38 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:15:39.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:15:38 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:15:39.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:15:38 smithi115 conmon[90393]: 2022-05-14T02:15:38.983+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:15:39.953 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:15:39 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:15:39.954 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:15:39 smithi115 ceph-mon[78593]: pgmap v5388: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:15:39.954 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:15:39 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:15:39.954 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T02:15:39.954 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:15:39.955 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T02:15:39.955 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:15:39.955 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:15:39.955 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:15:39.955 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T02:15:39.956 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:15:39.956 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T02:15:39.956 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:15:39.956 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T02:15:39.957 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:15:39.957 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T02:15:39.957 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:15:39.957 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T02:15:39.958 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:15:39.958 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:15:39.958 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:15:39.958 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:15:39 smithi115 ceph-mon[78593]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T02:15:39.958 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T02:15:39.959 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:15:39.959 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T02:15:39.959 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:15:39.959 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:15:39.960 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:15:39.960 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T02:15:39.960 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:15:39.960 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T02:15:39.961 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:15:39.961 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T02:15:39.961 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:15:39.961 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T02:15:39.961 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:15:39.962 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T02:15:39.962 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:15:39.962 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:15:39.962 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:15:39.962 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T02:15:39.963 INFO:journalctl@ceph.mon.c.smithi115.stdout: During handling of the above exception, another exception occurred: 2022-05-14T02:15:39.963 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T02:15:39.963 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T02:15:39.963 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T02:15:39.963 INFO:journalctl@ceph.mon.c.smithi115.stdout: return f(*arg) 2022-05-14T02:15:39.964 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T02:15:39.964 INFO:journalctl@ceph.mon.c.smithi115.stdout: self._write_client_files(client_files, host) 2022-05-14T02:15:39.964 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T02:15:39.964 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T02:15:39.964 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T02:15:39.965 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T02:15:39.965 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T02:15:39.965 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.event_loop.get_result(coro) 2022-05-14T02:15:39.965 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T02:15:39.965 INFO:journalctl@ceph.mon.c.smithi115.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T02:15:39.966 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T02:15:39.966 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.__get_result() 2022-05-14T02:15:39.966 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T02:15:39.966 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise self._exception 2022-05-14T02:15:39.967 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T02:15:39.967 INFO:journalctl@ceph.mon.c.smithi115.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T02:15:39.967 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T02:15:39.967 INFO:journalctl@ceph.mon.c.smithi115.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T02:15:39.967 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T02:15:39.968 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise 2022-05-14T02:15:39.968 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T02:15:39.968 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.gen.throw(type, value, traceback) 2022-05-14T02:15:39.968 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T02:15:39.969 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OrchestratorError(msg) 2022-05-14T02:15:39.969 INFO:journalctl@ceph.mon.c.smithi115.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:15:40.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:15:39 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:15:40.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:15:39 smithi047 ceph-mon[77147]: pgmap v5388: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:15:40.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:15:39 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:15:40.042 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T02:15:40.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:15:40.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T02:15:40.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:15:40.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:15:40.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:15:40.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T02:15:40.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:15:40.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T02:15:40.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:15:40.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T02:15:40.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:15:40.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T02:15:40.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:15:40.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T02:15:40.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:15:40.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:15:40.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:15:40.046 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:15:39 smithi047 ceph-mon[77147]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T02:15:40.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T02:15:40.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:15:40.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T02:15:40.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:15:40.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:15:40.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:15:40.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T02:15:40.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:15:40.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T02:15:40.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:15:40.049 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T02:15:40.049 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:15:40.049 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T02:15:40.049 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:15:40.050 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T02:15:40.050 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:15:40.050 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:15:40.050 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:15:40.050 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T02:15:40.051 INFO:journalctl@ceph.mon.b.smithi047.stdout: During handling of the above exception, another exception occurred: 2022-05-14T02:15:40.051 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T02:15:40.051 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T02:15:40.051 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T02:15:40.051 INFO:journalctl@ceph.mon.b.smithi047.stdout: return f(*arg) 2022-05-14T02:15:40.052 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T02:15:40.052 INFO:journalctl@ceph.mon.b.smithi047.stdout: self._write_client_files(client_files, host) 2022-05-14T02:15:40.052 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T02:15:40.052 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T02:15:40.053 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T02:15:40.053 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T02:15:40.053 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T02:15:40.053 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.event_loop.get_result(coro) 2022-05-14T02:15:40.054 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T02:15:40.054 INFO:journalctl@ceph.mon.b.smithi047.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T02:15:40.054 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T02:15:40.054 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.__get_result() 2022-05-14T02:15:40.055 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T02:15:40.055 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise self._exception 2022-05-14T02:15:40.055 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T02:15:40.055 INFO:journalctl@ceph.mon.b.smithi047.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T02:15:40.056 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T02:15:40.056 INFO:journalctl@ceph.mon.b.smithi047.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T02:15:40.056 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T02:15:40.056 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise 2022-05-14T02:15:40.056 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T02:15:40.057 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.gen.throw(type, value, traceback) 2022-05-14T02:15:40.057 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T02:15:40.057 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OrchestratorError(msg) 2022-05-14T02:15:40.058 INFO:journalctl@ceph.mon.b.smithi047.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:15:40.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:15:39 smithi115 conmon[90393]: 2022-05-14T02:15:39.951+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:15:40.983 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:15:40 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:15:41.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:15:40 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:15:41.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:15:40 smithi115 conmon[90393]: 2022-05-14T02:15:40.982+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:15:41.985 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:15:41 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:15:41.986 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:15:41 smithi115 ceph-mon[78593]: pgmap v5389: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:15:41.986 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:15:41 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 10075 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:15:42.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:15:41 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:15:42.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:15:41 smithi047 ceph-mon[77147]: pgmap v5389: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:15:42.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:15:41 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 10075 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:15:42.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:15:41 smithi115 conmon[90393]: 2022-05-14T02:15:41.984+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:15:43.009 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:15:42 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:15:43.009 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:15:42 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:15:43.010 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:15:42 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:15:43.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:15:42 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:15:43.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:15:42 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:15:43.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:15:42 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:15:43.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:15:43 smithi115 conmon[90393]: 2022-05-14T02:15:43.007+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:15:44.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:15:43 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:15:44.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:15:43 smithi047 ceph-mon[77147]: pgmap v5390: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:15:44.055 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:15:43 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:15:44.055 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:15:43 smithi115 ceph-mon[78593]: pgmap v5390: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:15:44.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:15:44 smithi115 conmon[90393]: 2022-05-14T02:15:44.054+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:15:45.018 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:15:44 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:15:45.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:15:44 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:15:45.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:15:45 smithi115 conmon[90393]: 2022-05-14T02:15:45.017+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:15:46.014 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:15:45 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:15:46.014 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:15:45 smithi115 ceph-mon[78593]: pgmap v5391: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:15:46.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:15:45 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:15:46.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:15:45 smithi047 ceph-mon[77147]: pgmap v5391: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:15:46.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:15:46 smithi115 conmon[90393]: 2022-05-14T02:15:46.012+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:15:47.015 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:15:46 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:15:47.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:15:46 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:15:47.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:15:47 smithi115 conmon[90393]: 2022-05-14T02:15:47.013+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:15:48.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:15:47 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:15:48.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:15:47 smithi047 ceph-mon[77147]: pgmap v5392: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:15:48.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:15:47 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:15:48.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:15:47 smithi115 ceph-mon[78593]: pgmap v5392: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:15:48.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:15:48 smithi115 conmon[90393]: 2022-05-14T02:15:48.043+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:15:49.036 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:15:48 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:15:49.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:15:48 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:15:49.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:15:49 smithi115 conmon[90393]: 2022-05-14T02:15:49.036+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:15:49.989 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:15:49 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:15:49.990 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:15:49 smithi115 ceph-mon[78593]: pgmap v5393: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:15:50.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:15:49 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:15:50.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:15:49 smithi047 ceph-mon[77147]: pgmap v5393: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:15:50.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:15:49 smithi115 conmon[90393]: 2022-05-14T02:15:49.989+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:15:51.031 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:15:50 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:15:51.032 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:15:50 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:15:51.032 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:15:50 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:15:51.032 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:15:50 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 10080 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:15:51.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:15:50 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:15:51.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:15:50 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:15:51.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:15:50 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:15:51.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:15:50 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 10080 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:15:51.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:15:51 smithi115 conmon[90393]: 2022-05-14T02:15:51.031+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:15:52.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:15:51 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:15:52.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:15:51 smithi115 ceph-mon[78593]: pgmap v5394: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:15:52.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:15:51 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:15:52.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:15:51 smithi047 ceph-mon[77147]: pgmap v5394: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:15:52.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:15:52 smithi115 conmon[90393]: 2022-05-14T02:15:52.039+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:15:53.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:15:52 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:15:53.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:15:52 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:15:53.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:15:53 smithi115 conmon[90393]: 2022-05-14T02:15:53.044+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:15:54.012 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:15:53 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:15:54.012 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:15:53 smithi115 ceph-mon[78593]: pgmap v5395: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:15:54.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:15:53 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:15:54.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:15:53 smithi047 ceph-mon[77147]: pgmap v5395: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:15:54.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:15:54 smithi115 conmon[90393]: 2022-05-14T02:15:54.010+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:15:54.990 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:15:54 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:15:54.991 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:15:54 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:15:54.991 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:15:54 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:15:55.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:15:54 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:15:55.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:15:54 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:15:55.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:15:54 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:15:55.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:15:54 smithi115 conmon[90393]: 2022-05-14T02:15:54.988+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:15:55.981 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:15:55 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:15:55.982 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:15:55 smithi115 ceph-mon[78593]: pgmap v5396: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:15:55.982 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:15:55 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 10090 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:15:56.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:15:55 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:15:56.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:15:55 smithi047 ceph-mon[77147]: pgmap v5396: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:15:56.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:15:55 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 10090 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:15:56.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:15:55 smithi115 conmon[90393]: 2022-05-14T02:15:55.979+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:15:57.030 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:15:56 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:15:57.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:15:56 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:15:57.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:15:57 smithi115 conmon[90393]: 2022-05-14T02:15:57.028+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:15:58.031 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:15:57 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:15:58.032 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:15:57 smithi115 ceph-mon[78593]: pgmap v5397: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:15:58.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:15:57 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:15:58.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:15:57 smithi047 ceph-mon[77147]: pgmap v5397: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:15:58.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:15:58 smithi115 conmon[90393]: 2022-05-14T02:15:58.029+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:15:59.035 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:15:58 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:15:59.036 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:15:58 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:15:59.036 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:15:58 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:15:59.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:15:58 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:15:59.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:15:58 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:15:59.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:15:58 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:15:59.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:15:59 smithi115 conmon[90393]: 2022-05-14T02:15:59.033+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:16:00.018 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:15:59 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:16:00.019 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:15:59 smithi115 ceph-mon[78593]: pgmap v5398: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:16:00.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:15:59 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:16:00.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:15:59 smithi047 ceph-mon[77147]: pgmap v5398: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:16:00.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:16:00 smithi115 conmon[90393]: 2022-05-14T02:16:00.017+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:16:01.002 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:16:00 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:16:01.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:16:00 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:16:01.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:16:01 smithi115 conmon[90393]: 2022-05-14T02:16:01.001+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:16:02.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:16:01 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:16:02.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:16:01 smithi047 ceph-mon[77147]: pgmap v5399: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:16:02.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:16:01 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 10095 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:16:02.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:16:01 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:16:02.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:16:01 smithi115 ceph-mon[78593]: pgmap v5399: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:16:02.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:16:01 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 10095 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:16:02.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:16:01 smithi115 conmon[90393]: 2022-05-14T02:16:01.980+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:16:03.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:16:02 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:16:03.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:16:02 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:16:03.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:16:02 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:16:03.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:16:02 smithi115 conmon[90393]: 2022-05-14T02:16:02.988+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:16:03.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:16:02 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:16:03.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:16:02 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:16:03.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:16:02 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:16:04.004 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:16:03 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:16:04.005 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:16:03 smithi115 ceph-mon[78593]: pgmap v5400: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:16:04.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:16:03 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:16:04.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:16:03 smithi047 ceph-mon[77147]: pgmap v5400: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:16:04.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:16:04 smithi115 conmon[90393]: 2022-05-14T02:16:04.003+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:16:05.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:16:04 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:16:05.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:16:04 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:16:05.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:16:04 smithi115 conmon[90393]: 2022-05-14T02:16:04.968+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:16:06.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:16:05 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:16:06.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:16:05 smithi047 ceph-mon[77147]: pgmap v5401: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:16:06.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:16:05 smithi115 conmon[90393]: 2022-05-14T02:16:05.984+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:16:06.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:16:05 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:16:06.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:16:05 smithi115 ceph-mon[78593]: pgmap v5401: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:16:07.025 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:16:06 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:16:07.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:16:06 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:16:07.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:16:07 smithi115 conmon[90393]: 2022-05-14T02:16:07.023+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:16:08.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:16:07 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:16:08.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:16:07 smithi047 ceph-mon[77147]: pgmap v5402: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:16:08.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:16:08 smithi115 conmon[90393]: 2022-05-14T02:16:08.005+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:16:08.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:16:07 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:16:08.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:16:07 smithi115 ceph-mon[78593]: pgmap v5402: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:16:09.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:16:08 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:16:09.052 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:16:08 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:16:09.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:16:09 smithi115 conmon[90393]: 2022-05-14T02:16:09.050+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:16:10.038 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:16:09 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:16:10.039 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:16:09 smithi115 ceph-mon[78593]: pgmap v5403: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:16:10.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:16:09 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:16:10.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:16:09 smithi047 ceph-mon[77147]: pgmap v5403: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:16:10.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:16:10 smithi115 conmon[90393]: 2022-05-14T02:16:10.036+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:16:11.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:16:10 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:16:11.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:16:10 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:16:11.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:16:10 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:16:11.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:16:10 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 10100 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:16:11.050 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:16:10 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:16:11.051 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:16:10 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:16:11.051 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:16:10 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:16:11.051 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:16:10 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 10100 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:16:11.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:16:11 smithi115 conmon[90393]: 2022-05-14T02:16:11.049+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:16:12.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:16:11 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:16:12.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:16:11 smithi047 ceph-mon[77147]: pgmap v5404: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:16:12.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:16:11 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:16:12.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:16:11 smithi115 ceph-mon[78593]: pgmap v5404: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:16:12.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:16:12 smithi115 conmon[90393]: 2022-05-14T02:16:12.021+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:16:13.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:16:12 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:16:13.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:16:13 smithi115 conmon[90393]: 2022-05-14T02:16:13.009+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:16:13.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:16:12 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:16:14.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:16:13 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:16:14.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:16:13 smithi047 ceph-mon[77147]: pgmap v5405: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:16:14.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:16:13 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:16:14.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:16:13 smithi115 ceph-mon[78593]: pgmap v5405: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:16:14.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:16:13 smithi115 conmon[90393]: 2022-05-14T02:16:13.975+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:16:15.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:16:14 smithi115 conmon[90393]: 2022-05-14T02:16:14.960+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:16:15.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:16:14 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:16:15.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:16:14 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:16:15.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:16:14 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:16:15.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:16:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T02:16:15.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:16:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T02:16:15.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:16:14 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:16:15.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:16:14 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:16:15.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:16:14 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:16:15.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:16:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T02:16:15.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:16:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T02:16:16.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:16:16 smithi115 conmon[90393]: 2022-05-14T02:16:16.001+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:16:16.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:16:15 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:16:16.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:16:15 smithi115 ceph-mon[78593]: pgmap v5406: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:16:16.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:16:15 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 10110 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:16:16.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:16:15 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:16:16.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:16:15 smithi047 ceph-mon[77147]: pgmap v5406: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:16:16.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:16:15 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 10110 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:16:17.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:16:17 smithi115 conmon[90393]: 2022-05-14T02:16:17.006+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:16:17.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:16:16 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:16:17.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:16:16 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:16:18.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:16:18 smithi115 conmon[90393]: 2022-05-14T02:16:18.046+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:16:18.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:16:17 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:16:18.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:16:17 smithi115 ceph-mon[78593]: pgmap v5407: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:16:18.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:16:17 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:16:18.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:16:17 smithi047 ceph-mon[77147]: pgmap v5407: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:16:19.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:16:19 smithi115 conmon[90393]: 2022-05-14T02:16:19.016+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:16:19.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:16:18 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:16:19.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:16:18 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:16:19.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:16:18 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:16:19.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:16:18 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:16:19.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:16:18 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:16:19.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:16:18 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:16:20.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:16:20 smithi115 conmon[90393]: 2022-05-14T02:16:20.000+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:16:20.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:16:19 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:16:20.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:16:19 smithi115 ceph-mon[78593]: pgmap v5408: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:16:20.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:16:19 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:16:20.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:16:19 smithi047 ceph-mon[77147]: pgmap v5408: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:16:21.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:16:20 smithi115 conmon[90393]: 2022-05-14T02:16:20.985+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:16:21.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:16:20 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:16:21.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:16:20 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:16:22.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:16:22 smithi115 conmon[90393]: 2022-05-14T02:16:22.002+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:16:22.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:16:21 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:16:22.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:16:21 smithi115 ceph-mon[78593]: pgmap v5409: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:16:22.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:16:21 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 10115 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:16:22.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:16:21 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:16:22.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:16:21 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:16:22.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:16:21 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:16:22.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:16:21 smithi047 ceph-mon[77147]: pgmap v5409: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:16:22.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:16:21 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 10115 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:16:22.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:16:21 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:16:22.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:16:21 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:16:23.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:16:22 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:16:23.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:16:22 smithi115 conmon[90393]: 2022-05-14T02:16:22.969+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:16:23.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:16:22 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:16:24.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:16:24 smithi115 conmon[90393]: 2022-05-14T02:16:24.009+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:16:24.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:16:23 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:16:24.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:16:23 smithi115 ceph-mon[78593]: pgmap v5410: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:16:24.296 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:16:23 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:16:24.297 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:16:23 smithi047 ceph-mon[77147]: pgmap v5410: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:16:25.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:16:25 smithi115 conmon[90393]: 2022-05-14T02:16:25.008+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:16:25.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:16:24 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:16:25.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:16:24 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:16:26.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:16:26 smithi115 conmon[90393]: 2022-05-14T02:16:26.030+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:16:26.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:16:25 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:16:26.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:16:25 smithi115 ceph-mon[78593]: pgmap v5411: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:16:26.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:16:25 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:16:26.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:16:25 smithi047 ceph-mon[77147]: pgmap v5411: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:16:27.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:16:27 smithi115 conmon[90393]: 2022-05-14T02:16:27.042+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:16:27.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:16:26 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:16:27.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:16:26 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:16:28.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:16:28 smithi115 conmon[90393]: 2022-05-14T02:16:28.084+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:16:28.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:16:27 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:16:28.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:16:27 smithi115 ceph-mon[78593]: pgmap v5412: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:16:28.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:16:27 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:16:28.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:16:27 smithi047 ceph-mon[77147]: pgmap v5412: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:16:29.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:16:28 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:16:29.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:16:28 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:16:29.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:16:29 smithi115 conmon[90393]: 2022-05-14T02:16:29.118+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:16:30.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:16:29 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:16:30.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:16:29 smithi115 ceph-mon[78593]: pgmap v5413: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:16:30.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:16:29 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:16:30.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:16:29 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:16:30.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:16:29 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:16:30.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:16:29 smithi047 ceph-mon[77147]: pgmap v5413: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:16:30.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:16:29 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:16:30.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:16:29 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:16:30.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:16:30 smithi115 conmon[90393]: 2022-05-14T02:16:30.161+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:16:31.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:16:30 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:16:31.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:16:30 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 10120 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:16:31.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:16:30 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:16:31.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:16:30 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 10120 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:16:31.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:16:31 smithi115 conmon[90393]: 2022-05-14T02:16:31.175+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:16:32.205 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:16:31 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:16:32.205 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:16:31 smithi115 ceph-mon[78593]: pgmap v5414: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:16:32.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:16:31 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:16:32.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:16:31 smithi047 ceph-mon[77147]: pgmap v5414: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:16:32.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:16:32 smithi115 conmon[90393]: 2022-05-14T02:16:32.203+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:16:33.179 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:16:32 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:16:33.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:16:32 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:16:33.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:16:33 smithi115 conmon[90393]: 2022-05-14T02:16:33.177+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:16:34.208 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:16:33 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:16:34.209 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:16:33 smithi115 ceph-mon[78593]: pgmap v5415: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:16:34.209 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:16:33 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:16:34.209 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:16:33 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:16:34.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:16:33 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:16:34.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:16:33 smithi047 ceph-mon[77147]: pgmap v5415: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:16:34.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:16:33 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:16:34.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:16:33 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:16:34.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:16:34 smithi115 conmon[90393]: 2022-05-14T02:16:34.207+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:16:35.185 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:16:34 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:16:35.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:16:34 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:16:35.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:16:35 smithi115 conmon[90393]: 2022-05-14T02:16:35.184+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:16:36.149 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:16:35 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:16:36.150 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:16:35 smithi115 ceph-mon[78593]: pgmap v5416: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:16:36.150 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:16:35 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 10131 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:16:36.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:16:35 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:16:36.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:16:35 smithi047 ceph-mon[77147]: pgmap v5416: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:16:36.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:16:35 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 10131 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:16:36.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:16:36 smithi115 conmon[90393]: 2022-05-14T02:16:36.147+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:16:37.145 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:16:36 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:16:37.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:16:36 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:16:37.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:16:37 smithi115 conmon[90393]: 2022-05-14T02:16:37.144+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:16:38.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:16:37 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:16:38.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:16:37 smithi047 ceph-mon[77147]: pgmap v5417: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:16:38.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:16:37 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:16:38.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:16:37 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:16:38.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:16:38 smithi115 conmon[90393]: 2022-05-14T02:16:38.111+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:16:38.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:16:37 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:16:38.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:16:37 smithi115 ceph-mon[78593]: pgmap v5417: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:16:38.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:16:37 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:16:38.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:16:37 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:16:39.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:16:38 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:16:39.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:16:38 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T02:16:39.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:16:38 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T02:16:39.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:16:38 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T02:16:39.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:16:38 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:16:39.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:16:38 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T02:16:39.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:16:38 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T02:16:39.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:16:38 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T02:16:39.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:16:39 smithi115 conmon[90393]: 2022-05-14T02:16:39.081+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:16:40.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:16:39 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:16:40.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:16:39 smithi047 ceph-mon[77147]: pgmap v5418: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:16:40.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:16:39 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:16:40.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:16:39 smithi115 ceph-mon[78593]: pgmap v5418: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:16:40.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:16:40 smithi115 conmon[90393]: 2022-05-14T02:16:40.072+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:16:41.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:16:40 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:16:41.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:16:40 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:16:41.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:16:41 smithi115 conmon[90393]: 2022-05-14T02:16:41.090+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:16:41.791 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:16:41 smithi047 conmon[78521]: 2022-05-14T02:16:41.455+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:16:41.792 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:16:41 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T02:16:41.792 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:16:41 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:16:41.792 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:16:41 smithi047 conmon[78521]: yield 2022-05-14T02:16:41.793 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:16:41 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:16:41.793 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:16:41 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:16:41.793 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:16:41 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:16:41.793 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:16:41 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T02:16:41.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:16:41 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:16:41.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:16:41 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T02:16:41.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:16:41 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:16:41.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:16:41 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T02:16:41.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:16:41 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:16:41.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:16:41 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T02:16:41.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:16:41 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:16:41.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:16:41 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T02:16:41.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:16:41 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:16:41.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:16:41 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:16:41.797 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:16:41 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:16:42.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:16:41 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:16:42.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:16:41 smithi047 ceph-mon[77147]: pgmap v5419: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:16:42.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:16:41 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 10135 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:16:42.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:16:41 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:16:42.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:16:41 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:16:42.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:16:42 smithi115 conmon[90393]: 2022-05-14T02:16:42.095+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:16:42.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:16:41 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:16:42.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:16:41 smithi115 ceph-mon[78593]: pgmap v5419: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:16:42.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:16:41 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 10135 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:16:42.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:16:41 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:16:42.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:16:41 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:16:43.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:16:42 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:16:43.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:16:42 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:16:43.292 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T02:16:43.292 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:16:43.292 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T02:16:43.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:16:43.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:16:43.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:16:43.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T02:16:43.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:16:43.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T02:16:43.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:16:43.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T02:16:43.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:16:43.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T02:16:43.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:16:43.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T02:16:43.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:16:43.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:16:43.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:16:43.297 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:16:42 smithi047 ceph-mon[77147]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T02:16:43.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:16:43 smithi115 conmon[90393]: 2022-05-14T02:16:43.047+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:16:43.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:16:42 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:16:43.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:16:42 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:16:43.359 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T02:16:43.359 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:16:43.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T02:16:43.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:16:43.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:16:43.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:16:43.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T02:16:43.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:16:43.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T02:16:43.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:16:43.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T02:16:43.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:16:43.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T02:16:43.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:16:43.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T02:16:43.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:16:43.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:16:43.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:16:43.363 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:16:42 smithi115 ceph-mon[78593]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T02:16:44.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:16:43 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:16:44.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:16:43 smithi047 ceph-mon[77147]: pgmap v5420: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:16:44.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:16:44 smithi115 conmon[90393]: 2022-05-14T02:16:44.027+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:16:44.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:16:43 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:16:44.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:16:43 smithi115 ceph-mon[78593]: pgmap v5420: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:16:44.792 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:16:44 smithi047 conmon[78521]: 2022-05-14T02:16:44.527+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:16:44.792 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:16:44 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T02:16:44.792 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:16:44 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:16:44.793 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:16:44 smithi047 conmon[78521]: yield 2022-05-14T02:16:44.793 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:16:44 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:16:44.793 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:16:44 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:16:44.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:16:44 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:16:44.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:16:44 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T02:16:44.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:16:44 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:16:44.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:16:44 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T02:16:44.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:16:44 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:16:44.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:16:44 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T02:16:44.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:16:44 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:16:44.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:16:44 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T02:16:44.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:16:44 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:16:44.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:16:44 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T02:16:44.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:16:44 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:16:44.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:16:44 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:16:44.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:16:44 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:16:44.797 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:16:44 smithi047 conmon[78521]: 2022-05-14T02:16:44.528+0000 7f5344f82700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T02:16:44.797 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:16:44 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T02:16:44.797 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:16:44 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:16:44.797 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:16:44 smithi047 conmon[78521]: yield 2022-05-14T02:16:44.798 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:16:44 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:16:44.798 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:16:44 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:16:44.798 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:16:44 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:16:44.798 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:16:44 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T02:16:44.798 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:16:44 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:16:44.799 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:16:44 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T02:16:44.799 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:16:44 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:16:44.799 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:16:44 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T02:16:44.799 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:16:44 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:16:44.800 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:16:44 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T02:16:44.800 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:16:44 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:16:44.800 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:16:44 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T02:16:44.800 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:16:44 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:16:44.800 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:16:44 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:16:44.801 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:16:44 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:16:44.801 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:16:44 smithi047 conmon[78521]: 2022-05-14T02:16:44.801 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:16:44 smithi047 conmon[78521]: During handling of the above exception, another exception occurred: 2022-05-14T02:16:44.801 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:16:44 smithi047 conmon[78521]: 2022-05-14T02:16:44.802 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:16:44 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T02:16:44.802 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:16:44 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T02:16:44.802 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:16:44 smithi047 conmon[78521]: return f(*arg) 2022-05-14T02:16:44.802 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:16:44 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T02:16:44.803 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:16:44 smithi047 conmon[78521]: self._write_client_files(client_files, host) 2022-05-14T02:16:44.803 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:16:44 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T02:16:44.803 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:16:44 smithi047 conmon[78521]: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T02:16:44.803 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:16:44 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T02:16:44.803 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:16:44 smithi047 conmon[78521]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T02:16:44.804 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:16:44 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T02:16:44.804 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:16:44 smithi047 conmon[78521]: return self.event_loop.get_result(coro) 2022-05-14T02:16:44.804 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:16:44 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T02:16:44.804 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:16:44 smithi047 conmon[78521]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T02:16:44.805 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:16:44 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T02:16:44.805 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:16:44 smithi047 conmon[78521]: return self.__get_result() 2022-05-14T02:16:44.805 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:16:44 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T02:16:44.805 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:16:44 smithi047 conmon[78521]: raise self._exception 2022-05-14T02:16:44.805 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:16:44 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T02:16:44.806 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:16:44 smithi047 conmon[78521]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T02:16:44.806 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:16:44 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T02:16:44.806 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:16:44 smithi047 conmon[78521]: conn = await self._remote_connection(host, addr) 2022-05-14T02:16:44.806 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:16:44 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T02:16:44.806 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:16:44 smithi047 conmon[78521]: raise 2022-05-14T02:16:44.807 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:16:44 smithi047 conmon[78521]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T02:16:44.807 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:16:44 smithi047 conmon[78521]: self.gen.throw(type, value, traceback) 2022-05-14T02:16:44.807 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:16:44 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T02:16:44.808 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:16:44 smithi047 conmon[78521]: raise OrchestratorError(msg) 2022-05-14T02:16:44.808 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:16:44 smithi047 conmon[78521]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:16:45.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:16:44 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:16:45.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:16:44 smithi115 conmon[90393]: 2022-05-14T02:16:44.997+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:16:45.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:16:44 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:16:46.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:16:45 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:16:46.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:16:45 smithi047 ceph-mon[77147]: pgmap v5421: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:16:46.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:16:45 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:16:46.292 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T02:16:46.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:16:46.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T02:16:46.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:16:46.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:16:46.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:16:46.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T02:16:46.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:16:46.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T02:16:46.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:16:46.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T02:16:46.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:16:46.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T02:16:46.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:16:46.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T02:16:46.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:16:46.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:16:46.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:16:46.296 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:16:45 smithi047 ceph-mon[77147]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T02:16:46.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T02:16:46.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:16:46.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T02:16:46.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:16:46.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:16:46.298 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:16:46.298 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T02:16:46.298 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:16:46.298 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T02:16:46.299 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:16:46.299 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T02:16:46.299 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:16:46.300 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T02:16:46.300 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:16:46.300 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T02:16:46.301 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:16:46.301 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:16:46.301 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:16:46.301 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T02:16:46.302 INFO:journalctl@ceph.mon.b.smithi047.stdout: During handling of the above exception, another exception occurred: 2022-05-14T02:16:46.302 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T02:16:46.302 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T02:16:46.302 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T02:16:46.303 INFO:journalctl@ceph.mon.b.smithi047.stdout: return f(*arg) 2022-05-14T02:16:46.303 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T02:16:46.303 INFO:journalctl@ceph.mon.b.smithi047.stdout: self._write_client_files(client_files, host) 2022-05-14T02:16:46.303 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T02:16:46.303 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T02:16:46.304 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T02:16:46.304 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T02:16:46.304 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T02:16:46.304 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.event_loop.get_result(coro) 2022-05-14T02:16:46.305 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T02:16:46.305 INFO:journalctl@ceph.mon.b.smithi047.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T02:16:46.305 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T02:16:46.305 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.__get_result() 2022-05-14T02:16:46.305 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T02:16:46.306 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise self._exception 2022-05-14T02:16:46.306 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T02:16:46.306 INFO:journalctl@ceph.mon.b.smithi047.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T02:16:46.306 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T02:16:46.306 INFO:journalctl@ceph.mon.b.smithi047.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T02:16:46.307 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T02:16:46.307 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise 2022-05-14T02:16:46.307 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T02:16:46.307 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.gen.throw(type, value, traceback) 2022-05-14T02:16:46.308 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T02:16:46.308 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OrchestratorError(msg) 2022-05-14T02:16:46.308 INFO:journalctl@ceph.mon.b.smithi047.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:16:46.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:16:45 smithi115 conmon[90393]: 2022-05-14T02:16:45.992+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:16:46.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:16:45 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:16:46.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:16:45 smithi115 ceph-mon[78593]: pgmap v5421: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:16:46.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:16:45 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:16:46.359 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T02:16:46.359 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:16:46.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T02:16:46.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:16:46.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:16:46.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:16:46.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T02:16:46.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:16:46.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T02:16:46.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:16:46.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T02:16:46.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:16:46.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T02:16:46.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:16:46.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T02:16:46.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:16:46.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:16:46.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:16:46.363 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:16:45 smithi115 ceph-mon[78593]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T02:16:46.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T02:16:46.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:16:46.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T02:16:46.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:16:46.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:16:46.365 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:16:46.365 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T02:16:46.365 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:16:46.365 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T02:16:46.365 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:16:46.366 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T02:16:46.366 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:16:46.366 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T02:16:46.366 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:16:46.366 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T02:16:46.367 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:16:46.367 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:16:46.367 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:16:46.367 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T02:16:46.367 INFO:journalctl@ceph.mon.c.smithi115.stdout: During handling of the above exception, another exception occurred: 2022-05-14T02:16:46.368 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T02:16:46.368 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T02:16:46.368 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T02:16:46.368 INFO:journalctl@ceph.mon.c.smithi115.stdout: return f(*arg) 2022-05-14T02:16:46.369 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T02:16:46.369 INFO:journalctl@ceph.mon.c.smithi115.stdout: self._write_client_files(client_files, host) 2022-05-14T02:16:46.369 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T02:16:46.369 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T02:16:46.370 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T02:16:46.370 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T02:16:46.370 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T02:16:46.370 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.event_loop.get_result(coro) 2022-05-14T02:16:46.370 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T02:16:46.371 INFO:journalctl@ceph.mon.c.smithi115.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T02:16:46.371 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T02:16:46.371 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.__get_result() 2022-05-14T02:16:46.371 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T02:16:46.371 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise self._exception 2022-05-14T02:16:46.372 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T02:16:46.372 INFO:journalctl@ceph.mon.c.smithi115.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T02:16:46.372 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T02:16:46.372 INFO:journalctl@ceph.mon.c.smithi115.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T02:16:46.373 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T02:16:46.373 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise 2022-05-14T02:16:46.373 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T02:16:46.373 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.gen.throw(type, value, traceback) 2022-05-14T02:16:46.373 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T02:16:46.374 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OrchestratorError(msg) 2022-05-14T02:16:46.374 INFO:journalctl@ceph.mon.c.smithi115.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:16:47.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:16:46 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:16:47.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:16:47 smithi115 conmon[90393]: 2022-05-14T02:16:47.042+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:16:47.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:16:46 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:16:48.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:16:47 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:16:48.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:16:47 smithi047 ceph-mon[77147]: pgmap v5422: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:16:48.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:16:48 smithi115 conmon[90393]: 2022-05-14T02:16:48.013+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:16:48.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:16:47 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:16:48.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:16:47 smithi115 ceph-mon[78593]: pgmap v5422: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:16:49.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:16:48 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:16:49.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:16:49 smithi115 conmon[90393]: 2022-05-14T02:16:49.050+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:16:49.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:16:48 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:16:50.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:16:49 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:16:50.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:16:49 smithi047 ceph-mon[77147]: pgmap v5423: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:16:50.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:16:49 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:16:50.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:16:49 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:16:50.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:16:50 smithi115 conmon[90393]: 2022-05-14T02:16:50.039+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:16:50.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:16:49 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:16:50.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:16:49 smithi115 ceph-mon[78593]: pgmap v5423: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:16:50.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:16:49 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:16:50.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:16:49 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:16:51.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:16:50 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:16:51.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:16:50 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 10140 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:16:51.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:16:51 smithi115 conmon[90393]: 2022-05-14T02:16:51.022+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:16:51.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:16:50 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:16:51.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:16:50 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 10140 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:16:52.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:16:51 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:16:52.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:16:51 smithi047 ceph-mon[77147]: pgmap v5424: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:16:52.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:16:52 smithi115 conmon[90393]: 2022-05-14T02:16:52.064+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:16:52.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:16:51 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:16:52.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:16:51 smithi115 ceph-mon[78593]: pgmap v5424: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:16:53.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:16:52 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:16:53.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:16:53 smithi115 conmon[90393]: 2022-05-14T02:16:53.060+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:16:53.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:16:52 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:16:54.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:16:53 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:16:54.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:16:53 smithi047 ceph-mon[77147]: pgmap v5425: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:16:54.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:16:53 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:16:54.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:16:53 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:16:54.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:16:54 smithi115 conmon[90393]: 2022-05-14T02:16:54.043+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:16:54.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:16:53 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:16:54.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:16:53 smithi115 ceph-mon[78593]: pgmap v5425: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:16:54.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:16:53 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:16:54.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:16:53 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:16:55.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:16:54 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:16:55.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:16:55 smithi115 conmon[90393]: 2022-05-14T02:16:55.039+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:16:55.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:16:54 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:16:56.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:16:55 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:16:56.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:16:55 smithi047 ceph-mon[77147]: pgmap v5426: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:16:56.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:16:55 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 10150 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:16:56.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:16:56 smithi115 conmon[90393]: 2022-05-14T02:16:56.079+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:16:56.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:16:55 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:16:56.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:16:55 smithi115 ceph-mon[78593]: pgmap v5426: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:16:56.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:16:55 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 10150 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:16:57.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:16:56 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:16:57.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:16:57 smithi115 conmon[90393]: 2022-05-14T02:16:57.071+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:16:57.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:16:56 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:16:58.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:16:57 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:16:58.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:16:57 smithi047 ceph-mon[77147]: pgmap v5427: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:16:58.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:16:57 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:16:58.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:16:57 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:16:58.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:16:58 smithi115 conmon[90393]: 2022-05-14T02:16:58.053+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:16:58.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:16:57 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:16:58.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:16:57 smithi115 ceph-mon[78593]: pgmap v5427: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:16:58.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:16:57 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:16:58.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:16:57 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:16:59.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:16:58 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:16:59.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:16:58 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:16:59.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:16:59 smithi115 conmon[90393]: 2022-05-14T02:16:59.089+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:17:00.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:16:59 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:17:00.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:16:59 smithi047 ceph-mon[77147]: pgmap v5428: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:17:00.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:17:00 smithi115 conmon[90393]: 2022-05-14T02:17:00.045+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:17:00.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:16:59 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:17:00.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:16:59 smithi115 ceph-mon[78593]: pgmap v5428: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:17:01.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:17:00 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:17:01.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:17:01 smithi115 conmon[90393]: 2022-05-14T02:17:01.045+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:17:01.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:17:00 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:17:02.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:17:01 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:17:02.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:17:01 smithi047 ceph-mon[77147]: pgmap v5429: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:17:02.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:17:01 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 10156 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:17:02.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:17:01 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:17:02.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:17:01 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:17:02.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:17:02 smithi115 conmon[90393]: 2022-05-14T02:17:02.064+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:17:02.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:17:01 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:17:02.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:17:01 smithi115 ceph-mon[78593]: pgmap v5429: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:17:02.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:17:01 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 10156 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:17:02.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:17:01 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:17:02.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:17:01 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:17:03.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:17:02 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:17:03.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:17:02 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:17:03.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:17:03 smithi115 conmon[90393]: 2022-05-14T02:17:03.052+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:17:03.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:17:02 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:17:03.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:17:02 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:17:04.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:17:03 smithi047 ceph-mon[77147]: pgmap v5430: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:17:04.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:17:03 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:17:04.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:17:04 smithi115 conmon[90393]: 2022-05-14T02:17:04.015+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:17:04.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:17:03 smithi115 ceph-mon[78593]: pgmap v5430: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:17:04.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:17:03 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:17:05.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:17:04 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:17:05.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:17:04 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:17:05.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:17:05 smithi115 conmon[90393]: 2022-05-14T02:17:05.051+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:17:06.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:17:05 smithi047 ceph-mon[77147]: pgmap v5431: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:17:06.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:17:05 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:17:06.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:17:06 smithi115 conmon[90393]: 2022-05-14T02:17:06.076+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:17:06.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:17:05 smithi115 ceph-mon[78593]: pgmap v5431: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:17:06.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:17:05 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:17:07.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:17:06 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:17:07.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:17:06 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:17:07.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:17:07 smithi115 conmon[90393]: 2022-05-14T02:17:07.045+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:17:08.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:17:07 smithi047 ceph-mon[77147]: pgmap v5432: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:17:08.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:17:07 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:17:08.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:17:08 smithi115 conmon[90393]: 2022-05-14T02:17:08.030+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:17:08.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:17:07 smithi115 ceph-mon[78593]: pgmap v5432: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:17:08.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:17:07 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:17:09.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:17:08 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:17:09.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:17:08 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:17:09.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:17:08 smithi115 conmon[90393]: 2022-05-14T02:17:08.993+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:17:10.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:17:09 smithi047 ceph-mon[77147]: pgmap v5433: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:17:10.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:17:09 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:17:10.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:17:09 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:17:10.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:17:09 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:17:10.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:17:09 smithi115 conmon[90393]: 2022-05-14T02:17:09.967+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:17:10.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:17:09 smithi115 ceph-mon[78593]: pgmap v5433: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:17:10.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:17:09 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:17:10.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:17:09 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:17:10.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:17:09 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:17:11.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:17:10 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:17:11.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:17:10 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 10160 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:17:11.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:17:10 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:17:11.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:17:10 smithi115 conmon[90393]: 2022-05-14T02:17:10.939+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:17:11.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:17:10 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:17:11.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:17:10 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 10160 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:17:11.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:17:10 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:17:12.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:17:11 smithi047 ceph-mon[77147]: pgmap v5434: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:17:12.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:17:11 smithi115 conmon[90393]: 2022-05-14T02:17:11.989+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:17:12.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:17:11 smithi115 ceph-mon[78593]: pgmap v5434: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:17:13.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:17:12 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:17:13.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:17:12 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:17:13.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:17:12 smithi115 conmon[90393]: 2022-05-14T02:17:12.976+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:17:14.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:17:13 smithi047 ceph-mon[77147]: pgmap v5435: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:17:14.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:17:13 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:17:14.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:17:13 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:17:14.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:17:13 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:17:14.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:17:13 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T02:17:14.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:17:13 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T02:17:14.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:17:13 smithi115 conmon[90393]: 2022-05-14T02:17:13.960+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:17:14.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:17:13 smithi115 ceph-mon[78593]: pgmap v5435: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:17:14.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:17:13 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:17:14.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:17:13 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:17:14.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:17:13 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:17:14.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:17:13 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T02:17:14.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:17:13 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T02:17:15.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:17:15 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:17:15.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:17:15 smithi115 conmon[90393]: 2022-05-14T02:17:15.000+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:17:15.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:17:15 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:17:16.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:17:16 smithi047 ceph-mon[77147]: pgmap v5436: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:17:16.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:17:16 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:17:16.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:17:16 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 10170 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:17:16.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:17:16 smithi115 conmon[90393]: 2022-05-14T02:17:16.017+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:17:16.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:17:16 smithi115 ceph-mon[78593]: pgmap v5436: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:17:16.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:17:16 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:17:16.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:17:16 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 10170 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:17:17.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:17:17 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:17:17.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:17:17 smithi115 conmon[90393]: 2022-05-14T02:17:17.057+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:17:17.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:17:17 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:17:18.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:17:18 smithi047 ceph-mon[77147]: pgmap v5437: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:17:18.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:17:18 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:17:18.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:17:18 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:17:18.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:17:18 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:17:18.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:17:18 smithi115 conmon[90393]: 2022-05-14T02:17:18.029+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:17:18.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:17:18 smithi115 ceph-mon[78593]: pgmap v5437: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:17:18.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:17:18 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:17:18.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:17:18 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:17:18.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:17:18 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:17:19.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:17:19 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:17:19.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:17:19 smithi115 conmon[90393]: 2022-05-14T02:17:19.018+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:17:19.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:17:19 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:17:20.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:17:20 smithi047 ceph-mon[77147]: pgmap v5438: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:17:20.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:17:20 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:17:20.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:17:20 smithi115 conmon[90393]: 2022-05-14T02:17:20.043+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:17:20.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:17:20 smithi115 ceph-mon[78593]: pgmap v5438: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:17:20.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:17:20 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:17:21.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:17:21 smithi115 conmon[90393]: 2022-05-14T02:17:21.056+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:17:21.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:17:21 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:17:21.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:17:21 smithi047 ceph-mon[77147]: pgmap v5439: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:17:21.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:17:21 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 10175 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:17:21.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:17:21 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:17:21.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:17:21 smithi115 ceph-mon[78593]: pgmap v5439: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:17:21.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:17:21 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 10175 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:17:22.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:17:22 smithi115 conmon[90393]: 2022-05-14T02:17:22.075+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:17:22.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:17:22 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:17:22.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:17:22 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:17:22.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:17:22 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:17:22.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:17:22 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:17:22.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:17:22 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:17:22.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:17:22 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:17:23.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:17:23 smithi115 conmon[90393]: 2022-05-14T02:17:23.035+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:17:23.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:17:23 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:17:23.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:17:23 smithi047 ceph-mon[77147]: pgmap v5440: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:17:23.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:17:23 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:17:23.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:17:23 smithi115 ceph-mon[78593]: pgmap v5440: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:17:24.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:17:24 smithi115 conmon[90393]: 2022-05-14T02:17:24.081+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:17:24.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:17:24 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:17:24.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:17:24 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:17:25.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:17:25 smithi115 conmon[90393]: 2022-05-14T02:17:25.106+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:17:25.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:17:25 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:17:25.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:17:25 smithi047 ceph-mon[77147]: pgmap v5441: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:17:25.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:17:25 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:17:25.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:17:25 smithi115 ceph-mon[78593]: pgmap v5441: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:17:26.542 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:17:26 smithi115 conmon[90393]: 2022-05-14T02:17:26.138+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:17:26.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:17:26 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:17:26.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:17:26 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:17:27.540 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:17:27 smithi115 conmon[90393]: 2022-05-14T02:17:27.157+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:17:27.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:17:27 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:17:27.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:17:27 smithi047 ceph-mon[77147]: pgmap v5442: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:17:27.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:17:27 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:17:27.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:17:27 smithi115 ceph-mon[78593]: pgmap v5442: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:17:28.545 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:17:28 smithi115 conmon[90393]: 2022-05-14T02:17:28.116+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:17:28.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:17:28 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:17:29.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:17:28 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:17:29.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:17:29 smithi115 conmon[90393]: 2022-05-14T02:17:29.080+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:17:29.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:17:29 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:17:29.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:17:29 smithi115 ceph-mon[78593]: pgmap v5443: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:17:30.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:17:29 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:17:30.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:17:29 smithi047 ceph-mon[77147]: pgmap v5443: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:17:30.555 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:17:30 smithi115 conmon[90393]: 2022-05-14T02:17:30.130+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:17:30.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:17:30 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:17:30.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:17:30 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:17:30.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:17:30 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:17:30.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:17:30 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 10180 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:17:31.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:17:30 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:17:31.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:17:30 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:17:31.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:17:30 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:17:31.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:17:30 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 10180 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:17:31.556 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:17:31 smithi115 conmon[90393]: 2022-05-14T02:17:31.153+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:17:31.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:17:31 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:17:31.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:17:31 smithi115 ceph-mon[78593]: pgmap v5444: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:17:32.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:17:31 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:17:32.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:17:31 smithi047 ceph-mon[77147]: pgmap v5444: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:17:32.561 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:17:32 smithi115 conmon[90393]: 2022-05-14T02:17:32.191+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:17:32.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:17:32 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:17:33.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:17:32 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:17:33.562 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:17:33 smithi115 conmon[90393]: 2022-05-14T02:17:33.206+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:17:33.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:17:33 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:17:33.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:17:33 smithi115 ceph-mon[78593]: pgmap v5445: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:17:34.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:17:33 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:17:34.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:17:33 smithi047 ceph-mon[77147]: pgmap v5445: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:17:34.572 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:17:34 smithi115 conmon[90393]: 2022-05-14T02:17:34.217+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:17:34.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:17:34 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:17:34.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:17:34 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:17:34.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:17:34 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:17:35.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:17:34 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:17:35.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:17:34 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:17:35.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:17:34 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:17:35.576 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:17:35 smithi115 conmon[90393]: 2022-05-14T02:17:35.179+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:17:35.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:17:35 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:17:35.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:17:35 smithi115 ceph-mon[78593]: pgmap v5446: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:17:35.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:17:35 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 10190 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:17:36.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:17:35 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:17:36.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:17:35 smithi047 ceph-mon[77147]: pgmap v5446: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:17:36.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:17:35 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 10190 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:17:36.578 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:17:36 smithi115 conmon[90393]: 2022-05-14T02:17:36.186+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:17:36.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:17:36 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:17:37.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:17:36 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:17:37.579 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:17:37 smithi115 conmon[90393]: 2022-05-14T02:17:37.217+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:17:37.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:17:37 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:17:37.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:17:37 smithi115 ceph-mon[78593]: pgmap v5447: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:17:38.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:17:37 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:17:38.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:17:37 smithi047 ceph-mon[77147]: pgmap v5447: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:17:38.588 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:17:38 smithi115 conmon[90393]: 2022-05-14T02:17:38.231+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:17:38.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:17:38 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:17:38.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:17:38 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:17:38.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:17:38 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:17:39.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:17:38 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:17:39.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:17:38 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:17:39.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:17:38 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:17:39.590 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:17:39 smithi115 conmon[90393]: 2022-05-14T02:17:39.212+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:17:39.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:17:39 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:17:39.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:17:39 smithi115 ceph-mon[78593]: pgmap v5448: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:17:40.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:17:39 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:17:40.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:17:39 smithi047 ceph-mon[77147]: pgmap v5448: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:17:40.594 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:17:40 smithi115 conmon[90393]: 2022-05-14T02:17:40.257+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:17:40.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:17:40 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:17:41.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:17:40 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:17:41.596 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:17:41 smithi115 conmon[90393]: 2022-05-14T02:17:41.217+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:17:41.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:17:41 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:17:41.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:17:41 smithi115 ceph-mon[78593]: pgmap v5449: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:17:41.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:17:41 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 10196 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:17:42.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:17:41 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:17:42.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:17:41 smithi047 ceph-mon[77147]: pgmap v5449: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:17:42.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:17:41 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 10196 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:17:42.605 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:17:42 smithi115 conmon[90393]: 2022-05-14T02:17:42.256+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:17:42.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:17:42 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:17:42.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:17:42 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:17:42.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:17:42 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:17:43.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:17:42 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:17:43.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:17:42 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:17:43.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:17:42 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:17:43.606 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:17:43 smithi115 conmon[90393]: 2022-05-14T02:17:43.211+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:17:43.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:17:43 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:17:43.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:17:43 smithi115 ceph-mon[78593]: pgmap v5450: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:17:44.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:17:43 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:17:44.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:17:43 smithi047 ceph-mon[77147]: pgmap v5450: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:17:44.577 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:17:44 smithi115 conmon[90393]: 2022-05-14T02:17:44.202+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:17:44.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:17:44 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:17:44.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:17:44 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T02:17:44.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:17:44 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T02:17:44.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:17:44 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T02:17:45.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:17:44 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:17:45.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:17:44 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T02:17:45.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:17:44 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T02:17:45.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:17:44 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T02:17:45.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:17:45 smithi115 conmon[90393]: 2022-05-14T02:17:45.208+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:17:45.542 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:17:45 smithi047 conmon[78521]: 2022-05-14T02:17:45.135+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:17:45.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:17:45 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T02:17:45.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:17:45 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:17:45.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:17:45 smithi047 conmon[78521]: yield 2022-05-14T02:17:45.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:17:45 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:17:45.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:17:45 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:17:45.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:17:45 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:17:45.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:17:45 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T02:17:45.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:17:45 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:17:45.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:17:45 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T02:17:45.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:17:45 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:17:45.547 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:17:45 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T02:17:45.547 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:17:45 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:17:45.548 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:17:45 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T02:17:45.548 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:17:45 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:17:45.549 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:17:45 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T02:17:45.549 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:17:45 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:17:45.549 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:17:45 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:17:45.550 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:17:45 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:17:46.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:17:45 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:17:46.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:17:45 smithi047 ceph-mon[77147]: pgmap v5451: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:17:46.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:17:45 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:17:46.042 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T02:17:46.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:17:46.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T02:17:46.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:17:46.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:17:46.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:17:46.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T02:17:46.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:17:46.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T02:17:46.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:17:46.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T02:17:46.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:17:46.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T02:17:46.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:17:46.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T02:17:46.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:17:46.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:17:46.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:17:46.047 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:17:45 smithi047 ceph-mon[77147]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T02:17:46.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:17:45 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:17:46.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:17:45 smithi115 ceph-mon[78593]: pgmap v5451: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:17:46.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:17:45 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:17:46.109 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T02:17:46.109 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:17:46.109 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T02:17:46.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:17:46.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:17:46.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:17:46.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T02:17:46.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:17:46.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T02:17:46.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:17:46.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T02:17:46.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:17:46.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T02:17:46.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:17:46.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T02:17:46.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:17:46.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:17:46.113 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:17:46.113 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:17:45 smithi115 ceph-mon[78593]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T02:17:46.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:17:46 smithi115 conmon[90393]: 2022-05-14T02:17:46.158+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:17:47.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:17:46 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:17:47.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:17:46 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:17:47.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:17:47 smithi115 conmon[90393]: 2022-05-14T02:17:47.195+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:17:48.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:17:47 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:17:48.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:17:47 smithi047 ceph-mon[77147]: pgmap v5452: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:17:48.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:17:47 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:17:48.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:17:47 smithi115 ceph-mon[78593]: pgmap v5452: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:17:48.542 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:17:48 smithi047 conmon[78521]: 2022-05-14T02:17:48.207+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:17:48.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:17:48 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T02:17:48.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:17:48 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:17:48.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:17:48 smithi047 conmon[78521]: yield 2022-05-14T02:17:48.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:17:48 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:17:48.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:17:48 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:17:48.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:17:48 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:17:48.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:17:48 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T02:17:48.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:17:48 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:17:48.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:17:48 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T02:17:48.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:17:48 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:17:48.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:17:48 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T02:17:48.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:17:48 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:17:48.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:17:48 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T02:17:48.547 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:17:48 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:17:48.547 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:17:48 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T02:17:48.547 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:17:48 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:17:48.548 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:17:48 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:17:48.548 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:17:48 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:17:48.548 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:17:48 smithi047 conmon[78521]: 2022-05-14T02:17:48.208+0000 7f5341f7c700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T02:17:48.548 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:17:48 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T02:17:48.549 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:17:48 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:17:48.549 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:17:48 smithi047 conmon[78521]: yield 2022-05-14T02:17:48.549 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:17:48 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:17:48.549 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:17:48 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:17:48.550 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:17:48 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:17:48.550 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:17:48 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T02:17:48.550 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:17:48 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:17:48.550 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:17:48 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T02:17:48.550 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:17:48 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:17:48.551 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:17:48 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T02:17:48.551 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:17:48 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:17:48.551 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:17:48 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T02:17:48.551 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:17:48 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:17:48.552 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:17:48 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T02:17:48.552 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:17:48 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:17:48.552 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:17:48 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:17:48.552 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:17:48 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:17:48.553 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:17:48 smithi047 conmon[78521]: 2022-05-14T02:17:48.553 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:17:48 smithi047 conmon[78521]: During handling of the above exception, another exception occurred: 2022-05-14T02:17:48.553 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:17:48 smithi047 conmon[78521]: 2022-05-14T02:17:48.554 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:17:48 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T02:17:48.554 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:17:48 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T02:17:48.554 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:17:48 smithi047 conmon[78521]: return f(*arg) 2022-05-14T02:17:48.555 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:17:48 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T02:17:48.555 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:17:48 smithi047 conmon[78521]: self._write_client_files(client_files, host) 2022-05-14T02:17:48.555 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:17:48 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T02:17:48.555 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:17:48 smithi047 conmon[78521]: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T02:17:48.555 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:17:48 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T02:17:48.556 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:17:48 smithi047 conmon[78521]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T02:17:48.556 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:17:48 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T02:17:48.556 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:17:48 smithi047 conmon[78521]: return self.event_loop.get_result(coro) 2022-05-14T02:17:48.556 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:17:48 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T02:17:48.556 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:17:48 smithi047 conmon[78521]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T02:17:48.557 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:17:48 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T02:17:48.557 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:17:48 smithi047 conmon[78521]: return self.__get_result() 2022-05-14T02:17:48.557 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:17:48 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T02:17:48.557 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:17:48 smithi047 conmon[78521]: raise self._exception 2022-05-14T02:17:48.558 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:17:48 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T02:17:48.558 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:17:48 smithi047 conmon[78521]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T02:17:48.558 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:17:48 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T02:17:48.558 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:17:48 smithi047 conmon[78521]: conn = await self._remote_connection(host, addr) 2022-05-14T02:17:48.558 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:17:48 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T02:17:48.559 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:17:48 smithi047 conmon[78521]: raise 2022-05-14T02:17:48.559 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:17:48 smithi047 conmon[78521]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T02:17:48.559 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:17:48 smithi047 conmon[78521]: self.gen.throw(type, value, traceback) 2022-05-14T02:17:48.559 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:17:48 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T02:17:48.560 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:17:48 smithi047 conmon[78521]: raise OrchestratorError(msg) 2022-05-14T02:17:48.560 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:17:48 smithi047 conmon[78521]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:17:48.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:17:48 smithi115 conmon[90393]: 2022-05-14T02:17:48.185+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:17:49.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:17:48 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:17:49.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:17:48 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:17:49.042 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T02:17:49.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:17:49.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T02:17:49.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:17:49.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:17:49.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:17:49.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T02:17:49.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:17:49.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T02:17:49.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:17:49.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T02:17:49.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:17:49.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T02:17:49.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:17:49.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T02:17:49.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:17:49.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:17:49.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:17:49.048 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:17:48 smithi047 ceph-mon[77147]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T02:17:49.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T02:17:49.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:17:49.049 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T02:17:49.049 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:17:49.049 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:17:49.049 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:17:49.050 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T02:17:49.050 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:17:49.051 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T02:17:49.051 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:17:49.051 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T02:17:49.052 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:17:49.052 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T02:17:49.053 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:17:49.053 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T02:17:49.053 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:17:49.054 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:17:49.054 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:17:49.055 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T02:17:49.055 INFO:journalctl@ceph.mon.b.smithi047.stdout: During handling of the above exception, another exception occurred: 2022-05-14T02:17:49.056 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T02:17:49.056 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T02:17:49.057 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T02:17:49.057 INFO:journalctl@ceph.mon.b.smithi047.stdout: return f(*arg) 2022-05-14T02:17:49.057 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T02:17:49.058 INFO:journalctl@ceph.mon.b.smithi047.stdout: self._write_client_files(client_files, host) 2022-05-14T02:17:49.058 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T02:17:49.059 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T02:17:49.059 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T02:17:49.059 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T02:17:49.060 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T02:17:49.061 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.event_loop.get_result(coro) 2022-05-14T02:17:49.062 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T02:17:49.062 INFO:journalctl@ceph.mon.b.smithi047.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T02:17:49.062 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T02:17:49.062 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.__get_result() 2022-05-14T02:17:49.063 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T02:17:49.063 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise self._exception 2022-05-14T02:17:49.063 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T02:17:49.063 INFO:journalctl@ceph.mon.b.smithi047.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T02:17:49.064 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T02:17:49.064 INFO:journalctl@ceph.mon.b.smithi047.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T02:17:49.064 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T02:17:49.064 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise 2022-05-14T02:17:49.064 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T02:17:49.065 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.gen.throw(type, value, traceback) 2022-05-14T02:17:49.065 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T02:17:49.065 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OrchestratorError(msg) 2022-05-14T02:17:49.066 INFO:journalctl@ceph.mon.b.smithi047.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:17:49.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:17:48 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:17:49.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:17:48 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:17:49.109 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T02:17:49.109 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:17:49.109 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T02:17:49.109 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:17:49.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:17:49.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:17:49.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T02:17:49.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:17:49.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T02:17:49.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:17:49.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T02:17:49.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:17:49.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T02:17:49.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:17:49.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T02:17:49.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:17:49.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:17:49.113 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:17:49.113 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:17:48 smithi115 ceph-mon[78593]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T02:17:49.113 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T02:17:49.113 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:17:49.114 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T02:17:49.114 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:17:49.114 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:17:49.114 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:17:49.115 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T02:17:49.115 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:17:49.115 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T02:17:49.115 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:17:49.115 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T02:17:49.116 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:17:49.116 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T02:17:49.116 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:17:49.116 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T02:17:49.116 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:17:49.117 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:17:49.117 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:17:49.117 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T02:17:49.117 INFO:journalctl@ceph.mon.c.smithi115.stdout: During handling of the above exception, another exception occurred: 2022-05-14T02:17:49.118 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T02:17:49.118 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T02:17:49.118 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T02:17:49.118 INFO:journalctl@ceph.mon.c.smithi115.stdout: return f(*arg) 2022-05-14T02:17:49.119 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T02:17:49.119 INFO:journalctl@ceph.mon.c.smithi115.stdout: self._write_client_files(client_files, host) 2022-05-14T02:17:49.119 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T02:17:49.119 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T02:17:49.119 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T02:17:49.120 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T02:17:49.120 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T02:17:49.120 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.event_loop.get_result(coro) 2022-05-14T02:17:49.120 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T02:17:49.121 INFO:journalctl@ceph.mon.c.smithi115.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T02:17:49.121 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T02:17:49.121 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.__get_result() 2022-05-14T02:17:49.121 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T02:17:49.121 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise self._exception 2022-05-14T02:17:49.122 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T02:17:49.122 INFO:journalctl@ceph.mon.c.smithi115.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T02:17:49.122 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T02:17:49.122 INFO:journalctl@ceph.mon.c.smithi115.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T02:17:49.123 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T02:17:49.123 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise 2022-05-14T02:17:49.123 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T02:17:49.123 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.gen.throw(type, value, traceback) 2022-05-14T02:17:49.123 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T02:17:49.124 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OrchestratorError(msg) 2022-05-14T02:17:49.124 INFO:journalctl@ceph.mon.c.smithi115.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:17:49.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:17:49 smithi115 conmon[90393]: 2022-05-14T02:17:49.177+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:17:50.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:17:49 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:17:50.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:17:49 smithi047 ceph-mon[77147]: pgmap v5453: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:17:50.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:17:49 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:17:50.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:17:49 smithi115 ceph-mon[78593]: pgmap v5453: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:17:50.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:17:50 smithi115 conmon[90393]: 2022-05-14T02:17:50.183+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:17:51.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:17:50 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:17:51.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:17:50 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:17:51.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:17:50 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:17:51.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:17:50 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 10201 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:17:51.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:17:50 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:17:51.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:17:50 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:17:51.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:17:50 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:17:51.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:17:50 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 10201 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:17:51.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:17:51 smithi115 conmon[90393]: 2022-05-14T02:17:51.157+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:17:52.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:17:51 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:17:52.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:17:51 smithi047 ceph-mon[77147]: pgmap v5454: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:17:52.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:17:51 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:17:52.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:17:51 smithi115 ceph-mon[78593]: pgmap v5454: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:17:52.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:17:52 smithi115 conmon[90393]: 2022-05-14T02:17:52.114+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:17:53.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:17:52 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:17:53.088 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:17:52 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:17:53.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:17:53 smithi115 conmon[90393]: 2022-05-14T02:17:53.086+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:17:54.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:17:53 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:17:54.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:17:53 smithi047 ceph-mon[77147]: pgmap v5455: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:17:54.089 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:17:53 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:17:54.089 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:17:53 smithi115 ceph-mon[78593]: pgmap v5455: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:17:54.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:17:54 smithi115 conmon[90393]: 2022-05-14T02:17:54.087+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:17:55.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:17:54 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:17:55.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:17:54 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:17:55.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:17:54 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:17:55.068 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:17:54 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:17:55.069 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:17:54 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:17:55.069 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:17:54 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:17:55.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:17:55 smithi115 conmon[90393]: 2022-05-14T02:17:55.066+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:17:56.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:17:55 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:17:56.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:17:55 smithi047 ceph-mon[77147]: pgmap v5456: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:17:56.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:17:55 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 10211 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:17:56.106 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:17:55 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:17:56.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:17:55 smithi115 ceph-mon[78593]: pgmap v5456: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:17:56.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:17:55 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 10211 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:17:56.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:17:56 smithi115 conmon[90393]: 2022-05-14T02:17:56.104+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:17:57.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:17:56 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:17:57.096 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:17:56 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:17:57.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:17:57 smithi115 conmon[90393]: 2022-05-14T02:17:57.094+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:17:58.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:17:57 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:17:58.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:17:57 smithi047 ceph-mon[77147]: pgmap v5457: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:17:58.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:17:57 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:17:58.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:17:57 smithi115 ceph-mon[78593]: pgmap v5457: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:17:58.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:17:58 smithi115 conmon[90393]: 2022-05-14T02:17:58.112+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:17:59.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:17:58 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:17:59.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:17:58 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:17:59.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:17:58 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:17:59.067 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:17:58 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:17:59.068 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:17:58 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:17:59.068 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:17:58 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:17:59.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:17:59 smithi115 conmon[90393]: 2022-05-14T02:17:59.066+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:18:00.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:17:59 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:18:00.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:17:59 smithi047 ceph-mon[77147]: pgmap v5458: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:18:00.052 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:17:59 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:18:00.053 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:17:59 smithi115 ceph-mon[78593]: pgmap v5458: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:18:00.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:18:00 smithi115 conmon[90393]: 2022-05-14T02:18:00.051+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:18:01.036 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:18:00 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:18:01.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:18:00 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:18:01.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:18:01 smithi115 conmon[90393]: 2022-05-14T02:18:01.034+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:18:02.038 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:18:01 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:18:02.038 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:18:01 smithi115 ceph-mon[78593]: pgmap v5459: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:18:02.039 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:18:01 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 10215 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:18:02.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:18:01 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:18:02.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:18:01 smithi047 ceph-mon[77147]: pgmap v5459: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:18:02.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:18:01 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 10215 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:18:02.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:18:02 smithi115 conmon[90393]: 2022-05-14T02:18:02.036+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:18:02.997 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:18:02 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:18:02.997 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:18:02 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:18:02.997 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:18:02 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:18:03.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:18:02 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:18:03.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:18:02 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:18:03.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:18:02 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:18:03.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:18:02 smithi115 conmon[90393]: 2022-05-14T02:18:02.995+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:18:04.011 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:18:03 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:18:04.012 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:18:03 smithi115 ceph-mon[78593]: pgmap v5460: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:18:04.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:18:03 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:18:04.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:18:03 smithi047 ceph-mon[77147]: pgmap v5460: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:18:04.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:18:04 smithi115 conmon[90393]: 2022-05-14T02:18:04.010+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:18:04.991 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:18:04 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:18:05.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:18:04 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:18:05.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:18:04 smithi115 conmon[90393]: 2022-05-14T02:18:04.990+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:18:05.996 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:18:05 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:18:05.996 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:18:05 smithi115 ceph-mon[78593]: pgmap v5461: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:18:06.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:18:05 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:18:06.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:18:05 smithi047 ceph-mon[77147]: pgmap v5461: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:18:06.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:18:05 smithi115 conmon[90393]: 2022-05-14T02:18:05.994+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:18:06.982 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:18:06 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:18:07.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:18:06 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:18:07.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:18:06 smithi115 conmon[90393]: 2022-05-14T02:18:06.980+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:18:08.007 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:18:07 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:18:08.008 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:18:07 smithi115 ceph-mon[78593]: pgmap v5462: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:18:08.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:18:07 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:18:08.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:18:07 smithi047 ceph-mon[77147]: pgmap v5462: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:18:08.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:18:08 smithi115 conmon[90393]: 2022-05-14T02:18:08.005+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:18:09.027 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:18:08 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:18:09.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:18:08 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:18:09.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:18:09 smithi115 conmon[90393]: 2022-05-14T02:18:09.025+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:18:10.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:18:09 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:18:10.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:18:09 smithi047 ceph-mon[77147]: pgmap v5463: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:18:10.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:18:09 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:18:10.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:18:09 smithi115 ceph-mon[78593]: pgmap v5463: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:18:10.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:18:10 smithi115 conmon[90393]: 2022-05-14T02:18:10.045+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:18:11.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:18:10 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:18:11.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:18:10 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:18:11.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:18:10 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:18:11.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:18:10 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 10220 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:18:11.093 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:18:10 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:18:11.093 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:18:10 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:18:11.093 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:18:10 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:18:11.094 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:18:10 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 10220 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:18:11.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:18:11 smithi115 conmon[90393]: 2022-05-14T02:18:11.091+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:18:12.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:18:11 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:18:12.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:18:11 smithi047 ceph-mon[77147]: pgmap v5464: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:18:12.088 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:18:11 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:18:12.089 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:18:11 smithi115 ceph-mon[78593]: pgmap v5464: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:18:12.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:18:12 smithi115 conmon[90393]: 2022-05-14T02:18:12.087+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:18:13.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:18:12 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:18:13.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:18:12 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:18:13.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:18:13 smithi115 conmon[90393]: 2022-05-14T02:18:13.133+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:18:14.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:18:13 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:18:14.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:18:13 smithi047 ceph-mon[77147]: pgmap v5465: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:18:14.103 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:18:13 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:18:14.104 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:18:13 smithi115 ceph-mon[78593]: pgmap v5465: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:18:14.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:18:14 smithi115 conmon[90393]: 2022-05-14T02:18:14.101+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:18:15.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:18:14 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:18:15.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:18:14 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:18:15.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:18:14 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:18:15.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:18:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T02:18:15.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:18:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T02:18:15.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:18:14 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:18:15.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:18:14 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:18:15.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:18:14 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:18:15.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:18:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T02:18:15.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:18:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T02:18:15.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:18:15 smithi115 conmon[90393]: 2022-05-14T02:18:15.108+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:18:16.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:18:15 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:18:16.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:18:15 smithi047 ceph-mon[77147]: pgmap v5466: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:18:16.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:18:15 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 10230 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:18:16.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:18:15 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:18:16.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:18:15 smithi115 ceph-mon[78593]: pgmap v5466: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:18:16.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:18:15 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 10230 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:18:16.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:18:16 smithi115 conmon[90393]: 2022-05-14T02:18:16.138+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:18:17.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:18:16 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:18:17.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:18:16 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:18:17.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:18:17 smithi115 conmon[90393]: 2022-05-14T02:18:17.175+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:18:18.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:18:17 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:18:18.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:18:17 smithi047 ceph-mon[77147]: pgmap v5467: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:18:18.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:18:17 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:18:18.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:18:17 smithi115 ceph-mon[78593]: pgmap v5467: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:18:18.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:18:18 smithi115 conmon[90393]: 2022-05-14T02:18:18.178+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:18:19.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:18:18 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:18:19.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:18:18 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:18:19.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:18:18 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:18:19.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:18:18 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:18:19.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:18:18 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:18:19.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:18:18 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:18:19.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:18:19 smithi115 conmon[90393]: 2022-05-14T02:18:19.131+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:18:20.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:18:19 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:18:20.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:18:19 smithi047 ceph-mon[77147]: pgmap v5468: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:18:20.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:18:19 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:18:20.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:18:19 smithi115 ceph-mon[78593]: pgmap v5468: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:18:20.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:18:20 smithi115 conmon[90393]: 2022-05-14T02:18:20.117+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:18:21.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:18:20 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:18:21.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:18:20 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:18:21.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:18:21 smithi115 conmon[90393]: 2022-05-14T02:18:21.146+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:18:22.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:18:21 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:18:22.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:18:21 smithi047 ceph-mon[77147]: pgmap v5469: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:18:22.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:18:21 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 10235 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:18:22.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:18:21 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:18:22.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:18:21 smithi115 ceph-mon[78593]: pgmap v5469: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:18:22.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:18:21 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 10235 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:18:22.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:18:22 smithi115 conmon[90393]: 2022-05-14T02:18:22.195+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:18:23.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:18:22 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:18:23.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:18:22 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:18:23.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:18:22 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:18:23.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:18:22 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:18:23.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:18:22 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:18:23.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:18:22 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:18:23.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:18:23 smithi115 conmon[90393]: 2022-05-14T02:18:23.155+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:18:24.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:18:23 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:18:24.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:18:23 smithi047 ceph-mon[77147]: pgmap v5470: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:18:24.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:18:23 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:18:24.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:18:23 smithi115 ceph-mon[78593]: pgmap v5470: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:18:24.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:18:24 smithi115 conmon[90393]: 2022-05-14T02:18:24.121+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:18:25.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:18:24 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:18:25.077 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:18:24 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:18:25.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:18:25 smithi115 conmon[90393]: 2022-05-14T02:18:25.076+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:18:26.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:18:25 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:18:26.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:18:25 smithi047 ceph-mon[77147]: pgmap v5471: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:18:26.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:18:25 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:18:26.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:18:25 smithi115 ceph-mon[78593]: pgmap v5471: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:18:26.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:18:26 smithi115 conmon[90393]: 2022-05-14T02:18:26.119+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:18:27.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:18:26 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:18:27.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:18:26 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:18:27.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:18:27 smithi115 conmon[90393]: 2022-05-14T02:18:27.140+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:18:28.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:18:27 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:18:28.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:18:27 smithi047 ceph-mon[77147]: pgmap v5472: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:18:28.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:18:27 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:18:28.129 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:18:27 smithi115 ceph-mon[78593]: pgmap v5472: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:18:28.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:18:28 smithi115 conmon[90393]: 2022-05-14T02:18:28.164+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:18:29.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:18:28 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:18:29.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:18:28 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:18:29.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:18:29 smithi115 conmon[90393]: 2022-05-14T02:18:29.200+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:18:30.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:18:29 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:18:30.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:18:29 smithi115 ceph-mon[78593]: pgmap v5473: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:18:30.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:18:29 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:18:30.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:18:29 smithi047 ceph-mon[77147]: pgmap v5473: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:18:30.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:18:30 smithi115 conmon[90393]: 2022-05-14T02:18:30.184+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:18:31.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:18:30 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:18:31.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:18:30 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:18:31.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:18:30 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:18:31.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:18:30 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 10240 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:18:31.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:18:30 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:18:31.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:18:30 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:18:31.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:18:30 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:18:31.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:18:30 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 10240 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:18:31.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:18:31 smithi115 conmon[90393]: 2022-05-14T02:18:31.222+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:18:32.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:18:31 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:18:32.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:18:31 smithi115 ceph-mon[78593]: pgmap v5474: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:18:32.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:18:31 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:18:32.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:18:31 smithi047 ceph-mon[77147]: pgmap v5474: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:18:32.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:18:32 smithi115 conmon[90393]: 2022-05-14T02:18:32.243+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:18:33.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:18:32 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:18:33.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:18:32 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:18:33.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:18:33 smithi115 conmon[90393]: 2022-05-14T02:18:33.281+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:18:34.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:18:33 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:18:34.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:18:33 smithi115 ceph-mon[78593]: pgmap v5475: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:18:34.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:18:33 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:18:34.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:18:33 smithi047 ceph-mon[77147]: pgmap v5475: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:18:34.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:18:34 smithi115 conmon[90393]: 2022-05-14T02:18:34.288+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:18:35.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:18:34 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:18:35.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:18:34 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:18:35.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:18:34 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:18:35.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:18:34 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:18:35.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:18:34 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:18:35.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:18:34 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:18:35.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:18:35 smithi115 conmon[90393]: 2022-05-14T02:18:35.275+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:18:36.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:18:35 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:18:36.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:18:35 smithi115 ceph-mon[78593]: pgmap v5476: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:18:36.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:18:35 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 10250 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:18:36.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:18:35 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:18:36.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:18:35 smithi047 ceph-mon[77147]: pgmap v5476: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:18:36.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:18:35 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 10250 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:18:36.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:18:36 smithi115 conmon[90393]: 2022-05-14T02:18:36.306+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:18:37.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:18:36 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:18:37.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:18:36 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:18:37.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:18:37 smithi115 conmon[90393]: 2022-05-14T02:18:37.346+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:18:38.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:18:37 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:18:38.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:18:37 smithi115 ceph-mon[78593]: pgmap v5477: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:18:38.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:18:37 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:18:38.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:18:37 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:18:38.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:18:37 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:18:38.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:18:37 smithi047 ceph-mon[77147]: pgmap v5477: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:18:38.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:18:37 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:18:38.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:18:37 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:18:38.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:18:38 smithi115 conmon[90393]: 2022-05-14T02:18:38.331+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:18:39.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:18:38 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:18:39.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:18:38 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:18:39.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:18:39 smithi115 conmon[90393]: 2022-05-14T02:18:39.300+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:18:40.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:18:39 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:18:40.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:18:39 smithi115 ceph-mon[78593]: pgmap v5478: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:18:40.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:18:39 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:18:40.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:18:39 smithi047 ceph-mon[77147]: pgmap v5478: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:18:40.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:18:40 smithi115 conmon[90393]: 2022-05-14T02:18:40.251+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:18:41.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:18:40 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:18:41.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:18:40 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:18:41.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:18:41 smithi115 conmon[90393]: 2022-05-14T02:18:41.227+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:18:42.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:18:41 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:18:42.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:18:41 smithi115 ceph-mon[78593]: pgmap v5479: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:18:42.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:18:41 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 10255 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:18:42.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:18:41 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:18:42.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:18:41 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:18:42.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:18:41 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:18:42.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:18:41 smithi047 ceph-mon[77147]: pgmap v5479: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:18:42.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:18:41 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 10255 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:18:42.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:18:41 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:18:42.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:18:41 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:18:42.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:18:42 smithi115 conmon[90393]: 2022-05-14T02:18:42.256+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:18:43.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:18:42 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:18:43.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:18:42 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:18:43.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:18:43 smithi115 conmon[90393]: 2022-05-14T02:18:43.228+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:18:44.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:18:43 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:18:44.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:18:43 smithi115 ceph-mon[78593]: pgmap v5480: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:18:44.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:18:43 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:18:44.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:18:43 smithi047 ceph-mon[77147]: pgmap v5480: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:18:44.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:18:44 smithi115 conmon[90393]: 2022-05-14T02:18:44.217+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:18:45.198 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:18:44 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:18:45.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:18:44 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:18:45.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:18:45 smithi115 conmon[90393]: 2022-05-14T02:18:45.196+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:18:46.244 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:18:45 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:18:46.245 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:18:45 smithi115 ceph-mon[78593]: pgmap v5481: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:18:46.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:18:45 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:18:46.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:18:45 smithi047 ceph-mon[77147]: pgmap v5481: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:18:46.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:18:46 smithi115 conmon[90393]: 2022-05-14T02:18:46.242+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:18:47.214 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:18:46 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:18:47.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:18:46 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:18:47.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:18:47 smithi115 conmon[90393]: 2022-05-14T02:18:47.212+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:18:48.184 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:18:47 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:18:48.184 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:18:47 smithi115 ceph-mon[78593]: pgmap v5482: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:18:48.254 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:18:47 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:18:48.254 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:18:47 smithi047 ceph-mon[77147]: pgmap v5482: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:18:48.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:18:48 smithi115 conmon[90393]: 2022-05-14T02:18:48.183+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:18:49.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:18:48 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:18:49.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:18:48 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T02:18:49.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:18:48 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T02:18:49.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:18:48 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T02:18:49.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:18:48 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:18:49.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:18:48 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T02:18:49.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:18:48 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T02:18:49.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:18:48 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T02:18:49.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:18:49 smithi115 conmon[90393]: 2022-05-14T02:18:49.154+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:18:50.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:18:49 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:18:50.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:18:49 smithi047 ceph-mon[77147]: pgmap v5483: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:18:50.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:18:49 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:18:50.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:18:49 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:18:50.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:18:50 smithi115 conmon[90393]: 2022-05-14T02:18:50.105+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:18:50.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:18:49 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:18:50.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:18:49 smithi115 ceph-mon[78593]: pgmap v5483: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:18:50.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:18:49 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:18:50.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:18:49 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:18:51.145 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:18:50 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:18:51.145 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:18:50 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 10260 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:18:51.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:18:50 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:18:51.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:18:50 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 10260 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:18:51.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:18:51 smithi115 conmon[90393]: 2022-05-14T02:18:51.143+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:18:51.791 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:18:51 smithi047 conmon[78521]: 2022-05-14T02:18:51.312+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:18:51.792 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:18:51 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T02:18:51.792 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:18:51 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:18:51.792 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:18:51 smithi047 conmon[78521]: yield 2022-05-14T02:18:51.792 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:18:51 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:18:51.793 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:18:51 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:18:51.793 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:18:51 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:18:51.793 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:18:51 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T02:18:51.793 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:18:51 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:18:51.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:18:51 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T02:18:51.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:18:51 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:18:51.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:18:51 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T02:18:51.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:18:51 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:18:51.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:18:51 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T02:18:51.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:18:51 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:18:51.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:18:51 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T02:18:51.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:18:51 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:18:51.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:18:51 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:18:51.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:18:51 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:18:52.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:18:51 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:18:52.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:18:51 smithi047 ceph-mon[77147]: pgmap v5484: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:18:52.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:18:52 smithi115 conmon[90393]: 2022-05-14T02:18:52.100+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:18:52.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:18:51 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:18:52.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:18:51 smithi115 ceph-mon[78593]: pgmap v5484: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:18:53.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:18:52 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:18:53.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:18:52 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:18:53.292 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T02:18:53.292 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:18:53.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T02:18:53.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:18:53.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:18:53.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:18:53.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T02:18:53.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:18:53.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T02:18:53.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:18:53.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T02:18:53.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:18:53.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T02:18:53.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:18:53.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T02:18:53.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:18:53.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:18:53.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:18:53.298 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:18:52 smithi047 ceph-mon[77147]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T02:18:53.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:18:53 smithi115 conmon[90393]: 2022-05-14T02:18:53.120+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:18:53.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:18:52 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:18:53.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:18:52 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:18:53.359 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T02:18:53.359 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:18:53.359 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T02:18:53.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:18:53.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:18:53.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:18:53.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T02:18:53.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:18:53.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T02:18:53.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:18:53.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T02:18:53.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:18:53.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T02:18:53.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:18:53.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T02:18:53.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:18:53.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:18:53.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:18:53.363 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:18:52 smithi115 ceph-mon[78593]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T02:18:54.168 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:18:53 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:18:54.169 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:18:53 smithi115 ceph-mon[78593]: pgmap v5485: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:18:54.169 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:18:53 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:18:54.169 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:18:53 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:18:54.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:18:53 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:18:54.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:18:53 smithi047 ceph-mon[77147]: pgmap v5485: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:18:54.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:18:53 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:18:54.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:18:53 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:18:54.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:18:54 smithi115 conmon[90393]: 2022-05-14T02:18:54.167+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:18:54.792 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:18:54 smithi047 conmon[78521]: 2022-05-14T02:18:54.384+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:18:54.792 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:18:54 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T02:18:54.792 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:18:54 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:18:54.793 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:18:54 smithi047 conmon[78521]: yield 2022-05-14T02:18:54.793 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:18:54 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:18:54.793 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:18:54 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:18:54.793 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:18:54 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:18:54.793 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:18:54 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T02:18:54.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:18:54 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:18:54.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:18:54 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T02:18:54.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:18:54 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:18:54.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:18:54 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T02:18:54.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:18:54 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:18:54.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:18:54 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T02:18:54.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:18:54 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:18:54.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:18:54 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T02:18:54.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:18:54 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:18:54.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:18:54 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:18:54.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:18:54 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:18:54.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:18:54 smithi047 conmon[78521]: 2022-05-14T02:18:54.386+0000 7f5345f84700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T02:18:54.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:18:54 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T02:18:54.797 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:18:54 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:18:54.797 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:18:54 smithi047 conmon[78521]: yield 2022-05-14T02:18:54.797 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:18:54 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:18:54.797 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:18:54 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:18:54.798 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:18:54 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:18:54.798 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:18:54 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T02:18:54.798 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:18:54 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:18:54.798 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:18:54 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T02:18:54.798 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:18:54 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:18:54.799 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:18:54 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T02:18:54.799 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:18:54 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:18:54.799 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:18:54 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T02:18:54.799 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:18:54 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:18:54.799 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:18:54 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T02:18:54.800 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:18:54 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:18:54.800 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:18:54 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:18:54.800 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:18:54 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:18:54.800 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:18:54 smithi047 conmon[78521]: 2022-05-14T02:18:54.801 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:18:54 smithi047 conmon[78521]: During handling of the above exception, another exception occurred: 2022-05-14T02:18:54.801 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:18:54 smithi047 conmon[78521]: 2022-05-14T02:18:54.801 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:18:54 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T02:18:54.801 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:18:54 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T02:18:54.801 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:18:54 smithi047 conmon[78521]: return f(*arg) 2022-05-14T02:18:54.802 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:18:54 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T02:18:54.802 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:18:54 smithi047 conmon[78521]: self._write_client_files(client_files, host) 2022-05-14T02:18:54.802 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:18:54 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T02:18:54.802 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:18:54 smithi047 conmon[78521]: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T02:18:54.803 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:18:54 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T02:18:54.803 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:18:54 smithi047 conmon[78521]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T02:18:54.803 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:18:54 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T02:18:54.803 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:18:54 smithi047 conmon[78521]: return self.event_loop.get_result(coro) 2022-05-14T02:18:54.803 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:18:54 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T02:18:54.804 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:18:54 smithi047 conmon[78521]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T02:18:54.804 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:18:54 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T02:18:54.804 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:18:54 smithi047 conmon[78521]: return self.__get_result() 2022-05-14T02:18:54.804 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:18:54 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T02:18:54.805 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:18:54 smithi047 conmon[78521]: raise self._exception 2022-05-14T02:18:54.805 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:18:54 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T02:18:54.805 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:18:54 smithi047 conmon[78521]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T02:18:54.805 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:18:54 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T02:18:54.806 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:18:54 smithi047 conmon[78521]: conn = await self._remote_connection(host, addr) 2022-05-14T02:18:54.806 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:18:54 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T02:18:54.806 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:18:54 smithi047 conmon[78521]: raise 2022-05-14T02:18:54.806 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:18:54 smithi047 conmon[78521]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T02:18:54.806 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:18:54 smithi047 conmon[78521]: self.gen.throw(type, value, traceback) 2022-05-14T02:18:54.807 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:18:54 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T02:18:54.807 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:18:54 smithi047 conmon[78521]: raise OrchestratorError(msg) 2022-05-14T02:18:54.807 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:18:54 smithi047 conmon[78521]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:18:55.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:18:54 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:18:55.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:18:55 smithi115 conmon[90393]: 2022-05-14T02:18:55.125+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:18:55.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:18:54 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:18:56.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:18:55 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:18:56.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:18:55 smithi047 ceph-mon[77147]: pgmap v5486: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:18:56.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:18:55 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:18:56.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T02:18:56.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:18:56.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T02:18:56.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:18:56.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:18:56.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:18:56.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T02:18:56.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:18:56.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T02:18:56.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:18:56.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T02:18:56.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:18:56.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T02:18:56.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:18:56.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T02:18:56.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:18:56.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:18:56.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:18:56.297 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:18:55 smithi047 ceph-mon[77147]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T02:18:56.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T02:18:56.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:18:56.298 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T02:18:56.298 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:18:56.298 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:18:56.298 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:18:56.298 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T02:18:56.299 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:18:56.299 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T02:18:56.299 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:18:56.299 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T02:18:56.300 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:18:56.300 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T02:18:56.300 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:18:56.300 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T02:18:56.301 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:18:56.301 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:18:56.301 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:18:56.301 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T02:18:56.302 INFO:journalctl@ceph.mon.b.smithi047.stdout: During handling of the above exception, another exception occurred: 2022-05-14T02:18:56.302 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T02:18:56.302 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T02:18:56.302 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T02:18:56.302 INFO:journalctl@ceph.mon.b.smithi047.stdout: return f(*arg) 2022-05-14T02:18:56.303 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T02:18:56.303 INFO:journalctl@ceph.mon.b.smithi047.stdout: self._write_client_files(client_files, host) 2022-05-14T02:18:56.303 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T02:18:56.303 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T02:18:56.304 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T02:18:56.304 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T02:18:56.304 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T02:18:56.304 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.event_loop.get_result(coro) 2022-05-14T02:18:56.305 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T02:18:56.305 INFO:journalctl@ceph.mon.b.smithi047.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T02:18:56.305 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T02:18:56.305 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.__get_result() 2022-05-14T02:18:56.306 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T02:18:56.306 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise self._exception 2022-05-14T02:18:56.306 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T02:18:56.306 INFO:journalctl@ceph.mon.b.smithi047.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T02:18:56.306 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T02:18:56.307 INFO:journalctl@ceph.mon.b.smithi047.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T02:18:56.307 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T02:18:56.307 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise 2022-05-14T02:18:56.307 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T02:18:56.307 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.gen.throw(type, value, traceback) 2022-05-14T02:18:56.308 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T02:18:56.308 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OrchestratorError(msg) 2022-05-14T02:18:56.308 INFO:journalctl@ceph.mon.b.smithi047.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:18:56.308 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:18:55 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 10270 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:18:56.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:18:56 smithi115 conmon[90393]: 2022-05-14T02:18:56.102+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:18:56.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:18:55 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:18:56.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:18:55 smithi115 ceph-mon[78593]: pgmap v5486: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:18:56.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:18:55 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:18:56.359 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T02:18:56.359 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:18:56.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T02:18:56.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:18:56.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:18:56.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:18:56.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T02:18:56.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:18:56.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T02:18:56.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:18:56.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T02:18:56.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:18:56.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T02:18:56.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:18:56.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T02:18:56.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:18:56.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:18:56.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:18:56.363 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:18:55 smithi115 ceph-mon[78593]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T02:18:56.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T02:18:56.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:18:56.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T02:18:56.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:18:56.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:18:56.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:18:56.365 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T02:18:56.365 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:18:56.365 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T02:18:56.365 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:18:56.365 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T02:18:56.366 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:18:56.366 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T02:18:56.366 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:18:56.366 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T02:18:56.367 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:18:56.367 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:18:56.367 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:18:56.367 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T02:18:56.367 INFO:journalctl@ceph.mon.c.smithi115.stdout: During handling of the above exception, another exception occurred: 2022-05-14T02:18:56.368 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T02:18:56.368 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T02:18:56.368 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T02:18:56.368 INFO:journalctl@ceph.mon.c.smithi115.stdout: return f(*arg) 2022-05-14T02:18:56.369 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T02:18:56.369 INFO:journalctl@ceph.mon.c.smithi115.stdout: self._write_client_files(client_files, host) 2022-05-14T02:18:56.369 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T02:18:56.369 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T02:18:56.369 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T02:18:56.370 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T02:18:56.370 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T02:18:56.370 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.event_loop.get_result(coro) 2022-05-14T02:18:56.370 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T02:18:56.370 INFO:journalctl@ceph.mon.c.smithi115.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T02:18:56.371 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T02:18:56.371 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.__get_result() 2022-05-14T02:18:56.371 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T02:18:56.371 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise self._exception 2022-05-14T02:18:56.372 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T02:18:56.372 INFO:journalctl@ceph.mon.c.smithi115.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T02:18:56.372 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T02:18:56.372 INFO:journalctl@ceph.mon.c.smithi115.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T02:18:56.372 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T02:18:56.373 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise 2022-05-14T02:18:56.373 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T02:18:56.373 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.gen.throw(type, value, traceback) 2022-05-14T02:18:56.373 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T02:18:56.373 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OrchestratorError(msg) 2022-05-14T02:18:56.374 INFO:journalctl@ceph.mon.c.smithi115.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:18:56.374 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:18:55 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 10270 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:18:57.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:18:56 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:18:57.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:18:57 smithi115 conmon[90393]: 2022-05-14T02:18:57.087+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:18:57.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:18:56 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:18:58.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:18:57 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:18:58.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:18:57 smithi047 ceph-mon[77147]: pgmap v5487: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:18:58.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:18:57 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:18:58.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:18:57 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:18:58.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:18:58 smithi115 conmon[90393]: 2022-05-14T02:18:58.065+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:18:58.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:18:57 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:18:58.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:18:57 smithi115 ceph-mon[78593]: pgmap v5487: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:18:58.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:18:57 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:18:58.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:18:57 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:18:59.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:18:58 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:18:59.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:18:59 smithi115 conmon[90393]: 2022-05-14T02:18:59.044+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:18:59.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:18:58 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:19:00.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:18:59 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:19:00.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:18:59 smithi047 ceph-mon[77147]: pgmap v5488: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:19:00.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:19:00 smithi115 conmon[90393]: 2022-05-14T02:19:00.024+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:19:00.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:18:59 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:19:00.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:18:59 smithi115 ceph-mon[78593]: pgmap v5488: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:19:01.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:19:00 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:19:01.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:19:01 smithi115 conmon[90393]: 2022-05-14T02:19:01.072+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:19:01.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:19:00 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:19:02.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:19:01 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:19:02.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:19:01 smithi047 ceph-mon[77147]: pgmap v5489: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:19:02.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:19:01 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 10274 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:19:02.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:19:01 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:19:02.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:19:01 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:19:02.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:19:02 smithi115 conmon[90393]: 2022-05-14T02:19:02.033+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:19:02.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:19:01 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:19:02.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:19:01 smithi115 ceph-mon[78593]: pgmap v5489: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:19:02.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:19:01 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 10274 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:19:02.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:19:01 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:19:02.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:19:01 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:19:03.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:19:02 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:19:03.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:19:03 smithi115 conmon[90393]: 2022-05-14T02:19:03.033+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:19:03.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:19:02 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:19:04.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:19:03 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:19:04.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:19:03 smithi047 ceph-mon[77147]: pgmap v5490: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:19:04.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:19:04 smithi115 conmon[90393]: 2022-05-14T02:19:04.020+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:19:04.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:19:03 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:19:04.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:19:03 smithi115 ceph-mon[78593]: pgmap v5490: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:19:05.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:19:04 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:19:05.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:19:05 smithi115 conmon[90393]: 2022-05-14T02:19:05.004+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:19:05.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:19:04 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:19:06.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:19:05 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:19:06.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:19:05 smithi047 ceph-mon[77147]: pgmap v5491: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:19:06.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:19:05 smithi115 conmon[90393]: 2022-05-14T02:19:05.987+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:19:06.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:19:05 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:19:06.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:19:05 smithi115 ceph-mon[78593]: pgmap v5491: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:19:07.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:19:06 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:19:07.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:19:07 smithi115 conmon[90393]: 2022-05-14T02:19:07.032+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:19:07.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:19:06 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:19:08.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:19:07 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:19:08.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:19:07 smithi047 ceph-mon[77147]: pgmap v5492: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:19:08.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:19:08 smithi115 conmon[90393]: 2022-05-14T02:19:08.044+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:19:08.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:19:07 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:19:08.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:19:07 smithi115 ceph-mon[78593]: pgmap v5492: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:19:09.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:19:08 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:19:09.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:19:09 smithi115 conmon[90393]: 2022-05-14T02:19:09.088+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:19:09.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:19:08 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:19:10.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:19:09 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:19:10.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:19:09 smithi047 ceph-mon[77147]: pgmap v5493: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:19:10.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:19:09 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:19:10.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:19:09 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:19:10.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:19:10 smithi115 conmon[90393]: 2022-05-14T02:19:10.053+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:19:10.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:19:09 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:19:10.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:19:09 smithi115 ceph-mon[78593]: pgmap v5493: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:19:10.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:19:09 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:19:10.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:19:09 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:19:11.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:19:10 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:19:11.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:19:10 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 10280 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:19:11.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:19:11 smithi115 conmon[90393]: 2022-05-14T02:19:11.011+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:19:11.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:19:10 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:19:11.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:19:10 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 10280 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:19:12.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:19:11 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:19:12.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:19:11 smithi047 ceph-mon[77147]: pgmap v5494: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:19:12.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:19:11 smithi115 conmon[90393]: 2022-05-14T02:19:11.976+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:19:12.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:19:11 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:19:12.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:19:11 smithi115 ceph-mon[78593]: pgmap v5494: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:19:13.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:19:12 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:19:13.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:19:12 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:19:13.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:19:13 smithi115 conmon[90393]: 2022-05-14T02:19:13.024+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:19:14.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:19:13 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:19:14.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:19:13 smithi047 ceph-mon[77147]: pgmap v5495: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:19:14.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:19:13 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:19:14.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:19:13 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:19:14.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:19:14 smithi115 conmon[90393]: 2022-05-14T02:19:14.059+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:19:14.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:19:13 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:19:14.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:19:13 smithi115 ceph-mon[78593]: pgmap v5495: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:19:14.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:19:13 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:19:14.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:19:13 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:19:15.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:19:14 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:19:15.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:19:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T02:19:15.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:19:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T02:19:15.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:19:15 smithi115 conmon[90393]: 2022-05-14T02:19:15.081+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:19:15.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:19:14 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:19:15.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:19:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T02:19:15.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:19:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T02:19:16.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:19:15 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:19:16.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:19:15 smithi047 ceph-mon[77147]: pgmap v5496: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:19:16.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:19:15 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 10289 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:19:16.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:19:16 smithi115 conmon[90393]: 2022-05-14T02:19:16.109+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:19:16.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:19:15 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:19:16.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:19:15 smithi115 ceph-mon[78593]: pgmap v5496: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:19:16.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:19:15 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 10289 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:19:17.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:19:16 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:19:17.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:19:17 smithi115 conmon[90393]: 2022-05-14T02:19:17.138+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:19:17.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:19:16 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:19:18.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:19:17 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:19:18.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:19:17 smithi047 ceph-mon[77147]: pgmap v5497: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:19:18.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:19:17 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:19:18.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:19:17 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:19:18.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:19:18 smithi115 conmon[90393]: 2022-05-14T02:19:18.180+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:19:18.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:19:17 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:19:18.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:19:17 smithi115 ceph-mon[78593]: pgmap v5497: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:19:18.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:19:17 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:19:18.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:19:17 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:19:19.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:19:18 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:19:19.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:19:19 smithi115 conmon[90393]: 2022-05-14T02:19:19.154+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:19:19.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:19:18 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:19:20.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:19:19 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:19:20.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:19:19 smithi047 ceph-mon[77147]: pgmap v5498: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:19:20.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:19:20 smithi115 conmon[90393]: 2022-05-14T02:19:20.162+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:19:20.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:19:19 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:19:20.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:19:19 smithi115 ceph-mon[78593]: pgmap v5498: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:19:21.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:19:20 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:19:21.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:19:21 smithi115 conmon[90393]: 2022-05-14T02:19:21.174+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:19:21.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:19:20 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:19:22.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:19:21 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:19:22.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:19:21 smithi047 ceph-mon[77147]: pgmap v5499: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:19:22.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:19:21 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 10295 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:19:22.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:19:21 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:19:22.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:19:21 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:19:22.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:19:22 smithi115 conmon[90393]: 2022-05-14T02:19:22.142+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:19:22.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:19:21 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:19:22.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:19:21 smithi115 ceph-mon[78593]: pgmap v5499: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:19:22.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:19:21 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 10295 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:19:22.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:19:21 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:19:22.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:19:21 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:19:23.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:19:22 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:19:23.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:19:23 smithi115 conmon[90393]: 2022-05-14T02:19:23.146+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:19:23.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:19:22 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:19:24.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:19:23 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:19:24.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:19:23 smithi047 ceph-mon[77147]: pgmap v5500: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:19:24.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:19:23 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:19:24.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:19:24 smithi115 conmon[90393]: 2022-05-14T02:19:24.165+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:19:24.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:19:23 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:19:24.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:19:23 smithi115 ceph-mon[78593]: pgmap v5500: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:19:24.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:19:23 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:19:25.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:19:24 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:19:25.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:19:25 smithi115 conmon[90393]: 2022-05-14T02:19:25.200+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:19:25.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:19:24 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:19:26.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:19:25 smithi047 ceph-mon[77147]: pgmap v5501: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:19:26.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:19:25 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:19:26.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:19:26 smithi115 conmon[90393]: 2022-05-14T02:19:26.240+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:19:26.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:19:25 smithi115 ceph-mon[78593]: pgmap v5501: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:19:26.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:19:25 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:19:27.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:19:26 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:19:27.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:19:27 smithi115 conmon[90393]: 2022-05-14T02:19:27.223+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:19:27.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:19:26 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:19:28.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:19:27 smithi047 ceph-mon[77147]: pgmap v5502: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:19:28.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:19:27 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:19:28.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:19:28 smithi115 conmon[90393]: 2022-05-14T02:19:28.241+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:19:28.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:19:27 smithi115 ceph-mon[78593]: pgmap v5502: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:19:28.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:19:27 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:19:29.290 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:19:29 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:19:29.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:19:29 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:19:29.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:19:29 smithi115 conmon[90393]: 2022-05-14T02:19:29.288+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:19:30.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:19:30 smithi047 ceph-mon[77147]: pgmap v5503: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:19:30.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:19:30 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:19:30.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:19:30 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:19:30.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:19:30 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:19:30.326 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:19:30 smithi115 ceph-mon[78593]: pgmap v5503: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:19:30.326 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:19:30 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:19:30.326 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:19:30 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:19:30.326 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:19:30 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:19:30.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:19:30 smithi115 conmon[90393]: 2022-05-14T02:19:30.324+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:19:31.288 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:19:31 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:19:31.289 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:19:31 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 10300 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:19:31.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:19:31 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:19:31.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:19:31 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 10300 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:19:31.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:19:31 smithi115 conmon[90393]: 2022-05-14T02:19:31.287+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:19:32.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:19:32 smithi047 ceph-mon[77147]: pgmap v5504: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:19:32.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:19:32 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:19:32.301 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:19:32 smithi115 ceph-mon[78593]: pgmap v5504: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:19:32.302 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:19:32 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:19:32.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:19:32 smithi115 conmon[90393]: 2022-05-14T02:19:32.300+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:19:33.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:19:33 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:19:33.318 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:19:33 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:19:33.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:19:33 smithi115 conmon[90393]: 2022-05-14T02:19:33.317+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:19:34.285 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:19:34 smithi115 ceph-mon[78593]: pgmap v5505: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:19:34.285 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:19:34 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:19:34.286 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:19:34 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:19:34.286 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:19:34 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:19:34.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:19:34 smithi047 ceph-mon[77147]: pgmap v5505: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:19:34.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:19:34 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:19:34.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:19:34 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:19:34.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:19:34 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:19:34.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:19:34 smithi115 conmon[90393]: 2022-05-14T02:19:34.283+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:19:35.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:19:35 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:19:35.312 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:19:35 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:19:35.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:19:35 smithi115 conmon[90393]: 2022-05-14T02:19:35.310+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:19:36.278 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:19:36 smithi115 ceph-mon[78593]: pgmap v5506: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:19:36.279 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:19:36 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:19:36.279 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:19:36 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 10310 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:19:36.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:19:36 smithi047 ceph-mon[77147]: pgmap v5506: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:19:36.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:19:36 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:19:36.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:19:36 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 10310 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:19:36.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:19:36 smithi115 conmon[90393]: 2022-05-14T02:19:36.276+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:19:37.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:19:37 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:19:37.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:19:37 smithi115 conmon[90393]: 2022-05-14T02:19:37.265+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:19:37.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:19:37 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:19:38.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:19:38 smithi047 ceph-mon[77147]: pgmap v5507: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:19:38.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:19:38 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:19:38.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:19:38 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:19:38.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:19:38 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:19:38.294 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:19:38 smithi115 ceph-mon[78593]: pgmap v5507: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:19:38.294 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:19:38 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:19:38.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:19:38 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:19:38.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:19:38 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:19:38.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:19:38 smithi115 conmon[90393]: 2022-05-14T02:19:38.292+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:19:39.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:19:39 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:19:39.316 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:19:39 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:19:39.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:19:39 smithi115 conmon[90393]: 2022-05-14T02:19:39.315+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:19:40.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:19:40 smithi047 ceph-mon[77147]: pgmap v5508: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:19:40.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:19:40 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:19:40.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:19:40 smithi115 ceph-mon[78593]: pgmap v5508: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:19:40.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:19:40 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:19:40.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:19:40 smithi115 conmon[90393]: 2022-05-14T02:19:40.271+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:19:41.555 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:19:41 smithi115 conmon[90393]: 2022-05-14T02:19:41.226+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:19:41.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:19:41 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:19:41.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:19:41 smithi115 ceph-mon[78593]: pgmap v5509: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:19:41.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:19:41 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 10315 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:19:42.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:19:41 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:19:42.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:19:41 smithi047 ceph-mon[77147]: pgmap v5509: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:19:42.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:19:41 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 10315 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:19:42.560 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:19:42 smithi115 conmon[90393]: 2022-05-14T02:19:42.193+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:19:42.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:19:42 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:19:42.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:19:42 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:19:42.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:19:42 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:19:43.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:19:42 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:19:43.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:19:42 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:19:43.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:19:42 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:19:43.562 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:19:43 smithi115 conmon[90393]: 2022-05-14T02:19:43.186+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:19:43.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:19:43 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:19:43.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:19:43 smithi115 ceph-mon[78593]: pgmap v5510: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:19:44.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:19:43 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:19:44.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:19:43 smithi047 ceph-mon[77147]: pgmap v5510: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:19:44.567 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:19:44 smithi115 conmon[90393]: 2022-05-14T02:19:44.217+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:19:44.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:19:44 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:19:45.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:19:44 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:19:45.568 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:19:45 smithi115 conmon[90393]: 2022-05-14T02:19:45.261+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:19:45.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:19:45 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:19:45.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:19:45 smithi115 ceph-mon[78593]: pgmap v5511: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:19:46.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:19:45 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:19:46.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:19:45 smithi047 ceph-mon[77147]: pgmap v5511: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:19:46.573 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:19:46 smithi115 conmon[90393]: 2022-05-14T02:19:46.290+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:19:46.863 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:19:46 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:19:47.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:19:46 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:19:47.575 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:19:47 smithi115 conmon[90393]: 2022-05-14T02:19:47.283+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:19:47.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:19:47 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:19:47.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:19:47 smithi115 ceph-mon[78593]: pgmap v5512: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:19:48.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:19:47 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:19:48.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:19:47 smithi047 ceph-mon[77147]: pgmap v5512: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:19:48.579 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:19:48 smithi115 conmon[90393]: 2022-05-14T02:19:48.290+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:19:48.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:19:48 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:19:49.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:19:48 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:19:49.581 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:19:49 smithi115 conmon[90393]: 2022-05-14T02:19:49.283+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:19:49.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:19:49 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:19:49.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:19:49 smithi115 ceph-mon[78593]: pgmap v5513: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:19:50.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:19:49 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:19:50.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:19:49 smithi047 ceph-mon[77147]: pgmap v5513: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:19:50.589 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:19:50 smithi115 conmon[90393]: 2022-05-14T02:19:50.321+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:19:50.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:19:50 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:19:50.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:19:50 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:19:50.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:19:50 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:19:50.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:19:50 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 10320 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:19:51.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:19:50 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:19:51.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:19:50 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:19:51.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:19:50 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:19:51.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:19:50 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 10320 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:19:51.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:19:51 smithi115 conmon[90393]: 2022-05-14T02:19:51.343+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:19:51.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:19:51 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:19:51.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:19:51 smithi115 ceph-mon[78593]: pgmap v5514: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:19:52.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:19:51 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:19:52.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:19:51 smithi047 ceph-mon[77147]: pgmap v5514: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:19:52.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:19:52 smithi115 conmon[90393]: 2022-05-14T02:19:52.346+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:19:52.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:19:52 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:19:53.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:19:52 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:19:53.598 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:19:53 smithi115 conmon[90393]: 2022-05-14T02:19:53.322+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:19:53.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:19:53 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:19:53.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:19:53 smithi115 ceph-mon[78593]: pgmap v5515: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:19:54.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:19:53 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:19:54.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:19:53 smithi047 ceph-mon[77147]: pgmap v5515: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:19:54.605 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:19:54 smithi115 conmon[90393]: 2022-05-14T02:19:54.353+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:19:54.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:19:54 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:19:54.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:19:54 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:19:54.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:19:54 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:19:54.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:19:54 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T02:19:54.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:19:54 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T02:19:54.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:19:54 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T02:19:54.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:19:54 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:19:54.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:19:54 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:19:54.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:19:54 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:19:54.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:19:54 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T02:19:54.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:19:54 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T02:19:54.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:19:54 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T02:19:55.291 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:19:54 smithi047 conmon[78521]: 2022-05-14T02:19:54.863+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:19:55.292 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:19:54 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T02:19:55.292 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:19:54 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:19:55.292 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:19:54 smithi047 conmon[78521]: yield 2022-05-14T02:19:55.292 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:19:54 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:19:55.293 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:19:54 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:19:55.293 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:19:54 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:19:55.293 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:19:54 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T02:19:55.293 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:19:54 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:19:55.293 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:19:54 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T02:19:55.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:19:54 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:19:55.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:19:54 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T02:19:55.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:19:54 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:19:55.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:19:54 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T02:19:55.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:19:54 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:19:55.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:19:54 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T02:19:55.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:19:54 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:19:55.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:19:54 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:19:55.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:19:54 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:19:55.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:19:55 smithi115 conmon[90393]: 2022-05-14T02:19:55.320+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:19:55.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:19:55 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:19:55.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:19:55 smithi115 ceph-mon[78593]: pgmap v5516: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:19:55.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:19:55 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:19:55.859 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T02:19:55.859 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:19:55.859 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T02:19:55.860 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:19:55.860 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:19:55.860 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:19:55.860 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T02:19:55.860 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:19:55.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T02:19:55.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:19:55.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T02:19:55.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:19:55.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T02:19:55.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:19:55.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T02:19:55.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:19:55.863 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:19:55.863 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:19:55.863 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:19:55 smithi115 ceph-mon[78593]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T02:19:55.863 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:19:55 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 10330 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:19:56.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:19:55 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:19:56.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:19:55 smithi047 ceph-mon[77147]: pgmap v5516: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:19:56.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:19:55 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:19:56.042 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T02:19:56.042 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:19:56.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T02:19:56.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:19:56.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:19:56.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:19:56.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T02:19:56.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:19:56.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T02:19:56.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:19:56.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T02:19:56.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:19:56.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T02:19:56.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:19:56.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T02:19:56.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:19:56.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:19:56.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:19:56.046 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:19:55 smithi047 ceph-mon[77147]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T02:19:56.046 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:19:55 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 10330 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:19:56.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:19:56 smithi115 conmon[90393]: 2022-05-14T02:19:56.301+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:19:57.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:19:56 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:19:57.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:19:56 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:19:57.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:19:57 smithi115 conmon[90393]: 2022-05-14T02:19:57.266+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:19:57.937 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:19:57 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:19:57.938 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:19:57 smithi047 ceph-mon[77147]: pgmap v5517: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:19:58.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:19:57 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:19:58.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:19:57 smithi115 ceph-mon[78593]: pgmap v5517: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:19:58.292 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:19:57 smithi047 conmon[78521]: 2022-05-14T02:19:57.935+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:19:58.292 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:19:57 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T02:19:58.293 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:19:57 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:19:58.293 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:19:57 smithi047 conmon[78521]: yield 2022-05-14T02:19:58.293 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:19:57 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:19:58.293 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:19:57 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:19:58.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:19:57 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:19:58.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:19:57 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T02:19:58.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:19:57 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:19:58.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:19:57 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T02:19:58.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:19:57 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:19:58.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:19:57 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T02:19:58.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:19:57 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:19:58.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:19:57 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T02:19:58.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:19:57 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:19:58.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:19:57 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T02:19:58.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:19:57 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:19:58.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:19:57 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:19:58.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:19:57 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:19:58.297 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:19:57 smithi047 conmon[78521]: 2022-05-14T02:19:57.936+0000 7f5340779700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T02:19:58.297 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:19:57 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T02:19:58.297 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:19:57 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:19:58.297 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:19:57 smithi047 conmon[78521]: yield 2022-05-14T02:19:58.298 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:19:57 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:19:58.298 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:19:57 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:19:58.298 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:19:57 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:19:58.298 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:19:57 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T02:19:58.299 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:19:57 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:19:58.299 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:19:57 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T02:19:58.299 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:19:57 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:19:58.299 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:19:57 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T02:19:58.299 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:19:57 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:19:58.300 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:19:57 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T02:19:58.300 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:19:57 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:19:58.300 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:19:57 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T02:19:58.300 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:19:57 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:19:58.301 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:19:57 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:19:58.301 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:19:57 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:19:58.301 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:19:57 smithi047 conmon[78521]: 2022-05-14T02:19:58.301 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:19:57 smithi047 conmon[78521]: During handling of the above exception, another exception occurred: 2022-05-14T02:19:58.302 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:19:57 smithi047 conmon[78521]: 2022-05-14T02:19:58.302 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:19:57 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T02:19:58.302 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:19:57 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T02:19:58.302 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:19:57 smithi047 conmon[78521]: return f(*arg) 2022-05-14T02:19:58.303 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:19:57 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T02:19:58.303 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:19:57 smithi047 conmon[78521]: self._write_client_files(client_files, host) 2022-05-14T02:19:58.303 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:19:57 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T02:19:58.303 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:19:57 smithi047 conmon[78521]: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T02:19:58.303 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:19:57 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T02:19:58.304 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:19:57 smithi047 conmon[78521]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T02:19:58.304 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:19:57 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T02:19:58.304 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:19:57 smithi047 conmon[78521]: return self.event_loop.get_result(coro) 2022-05-14T02:19:58.304 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:19:57 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T02:19:58.305 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:19:57 smithi047 conmon[78521]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T02:19:58.305 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:19:57 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T02:19:58.305 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:19:57 smithi047 conmon[78521]: return self.__get_result() 2022-05-14T02:19:58.305 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:19:57 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T02:19:58.306 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:19:57 smithi047 conmon[78521]: raise self._exception 2022-05-14T02:19:58.306 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:19:57 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T02:19:58.306 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:19:57 smithi047 conmon[78521]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T02:19:58.306 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:19:57 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T02:19:58.306 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:19:57 smithi047 conmon[78521]: conn = await self._remote_connection(host, addr) 2022-05-14T02:19:58.307 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:19:57 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T02:19:58.307 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:19:57 smithi047 conmon[78521]: raise 2022-05-14T02:19:58.307 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:19:57 smithi047 conmon[78521]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T02:19:58.307 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:19:57 smithi047 conmon[78521]: self.gen.throw(type, value, traceback) 2022-05-14T02:19:58.308 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:19:57 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T02:19:58.308 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:19:57 smithi047 conmon[78521]: raise OrchestratorError(msg) 2022-05-14T02:19:58.308 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:19:57 smithi047 conmon[78521]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:19:58.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:19:58 smithi115 conmon[90393]: 2022-05-14T02:19:58.313+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:19:59.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:19:58 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:19:59.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:19:58 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:19:59.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:19:58 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:19:59.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:19:58 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:19:59.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T02:19:59.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:19:59.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T02:19:59.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:19:59.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:19:59.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:19:59.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T02:19:59.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:19:59.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T02:19:59.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:19:59.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T02:19:59.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:19:59.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T02:19:59.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:19:59.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T02:19:59.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:19:59.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:19:59.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:19:59.047 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:19:58 smithi047 ceph-mon[77147]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T02:19:59.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T02:19:59.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:19:59.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T02:19:59.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:19:59.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:19:59.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:19:59.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T02:19:59.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:19:59.049 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T02:19:59.049 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:19:59.049 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T02:19:59.049 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:19:59.050 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T02:19:59.050 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:19:59.050 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T02:19:59.050 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:19:59.050 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:19:59.051 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:19:59.051 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T02:19:59.051 INFO:journalctl@ceph.mon.b.smithi047.stdout: During handling of the above exception, another exception occurred: 2022-05-14T02:19:59.051 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T02:19:59.051 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T02:19:59.052 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T02:19:59.052 INFO:journalctl@ceph.mon.b.smithi047.stdout: return f(*arg) 2022-05-14T02:19:59.052 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T02:19:59.052 INFO:journalctl@ceph.mon.b.smithi047.stdout: self._write_client_files(client_files, host) 2022-05-14T02:19:59.053 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T02:19:59.053 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T02:19:59.053 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T02:19:59.053 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T02:19:59.054 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T02:19:59.054 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.event_loop.get_result(coro) 2022-05-14T02:19:59.054 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T02:19:59.054 INFO:journalctl@ceph.mon.b.smithi047.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T02:19:59.054 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T02:19:59.055 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.__get_result() 2022-05-14T02:19:59.055 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T02:19:59.055 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise self._exception 2022-05-14T02:19:59.056 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T02:19:59.056 INFO:journalctl@ceph.mon.b.smithi047.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T02:19:59.056 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T02:19:59.056 INFO:journalctl@ceph.mon.b.smithi047.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T02:19:59.057 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T02:19:59.057 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise 2022-05-14T02:19:59.057 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T02:19:59.057 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.gen.throw(type, value, traceback) 2022-05-14T02:19:59.057 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T02:19:59.058 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OrchestratorError(msg) 2022-05-14T02:19:59.058 INFO:journalctl@ceph.mon.b.smithi047.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:19:59.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:19:58 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:19:59.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:19:58 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:19:59.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:19:58 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:19:59.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:19:58 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:19:59.109 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T02:19:59.109 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:19:59.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T02:19:59.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:19:59.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:19:59.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:19:59.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T02:19:59.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:19:59.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T02:19:59.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:19:59.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T02:19:59.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:19:59.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T02:19:59.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:19:59.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T02:19:59.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:19:59.113 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:19:59.115 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:19:59.115 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:19:58 smithi115 ceph-mon[78593]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T02:19:59.115 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T02:19:59.116 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:19:59.116 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T02:19:59.116 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:19:59.116 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:19:59.117 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:19:59.117 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T02:19:59.117 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:19:59.117 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T02:19:59.118 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:19:59.118 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T02:19:59.118 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:19:59.118 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T02:19:59.118 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:19:59.119 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T02:19:59.119 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:19:59.119 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:19:59.119 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:19:59.120 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T02:19:59.120 INFO:journalctl@ceph.mon.c.smithi115.stdout: During handling of the above exception, another exception occurred: 2022-05-14T02:19:59.120 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T02:19:59.120 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T02:19:59.120 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T02:19:59.121 INFO:journalctl@ceph.mon.c.smithi115.stdout: return f(*arg) 2022-05-14T02:19:59.121 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T02:19:59.121 INFO:journalctl@ceph.mon.c.smithi115.stdout: self._write_client_files(client_files, host) 2022-05-14T02:19:59.122 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T02:19:59.122 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T02:19:59.122 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T02:19:59.122 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T02:19:59.122 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T02:19:59.123 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.event_loop.get_result(coro) 2022-05-14T02:19:59.123 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T02:19:59.123 INFO:journalctl@ceph.mon.c.smithi115.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T02:19:59.123 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T02:19:59.124 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.__get_result() 2022-05-14T02:19:59.124 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T02:19:59.124 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise self._exception 2022-05-14T02:19:59.124 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T02:19:59.125 INFO:journalctl@ceph.mon.c.smithi115.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T02:19:59.125 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T02:19:59.125 INFO:journalctl@ceph.mon.c.smithi115.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T02:19:59.125 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T02:19:59.126 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise 2022-05-14T02:19:59.126 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T02:19:59.126 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.gen.throw(type, value, traceback) 2022-05-14T02:19:59.126 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T02:19:59.127 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OrchestratorError(msg) 2022-05-14T02:19:59.127 INFO:journalctl@ceph.mon.c.smithi115.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:19:59.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:19:59 smithi115 conmon[90393]: 2022-05-14T02:19:59.263+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:20:00.000 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:19:59 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:20:00.001 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:19:59 smithi047 ceph-mon[77147]: pgmap v5518: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:20:00.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:19:59 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:20:00.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:19:59 smithi115 ceph-mon[78593]: pgmap v5518: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:20:00.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:20:00 smithi047 conmon[77123]: 2022-05-14T02:19:59.999+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : Health detail: HEALTH_ERR 1 filesystem is degraded; 1 mds daemon damaged; 1 MDSs are read only; 1 MDSs report slow metadata IOs; 1/3 mons down, quorum b,c; Reduced data availability: 1 pg inactive; 9 slow ops, oldest one blocked for 10330 sec, osd.5 has slow ops 2022-05-14T02:20:00.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:20:00 smithi047 conmon[77123]: 2022-05-14T02:19:59.999+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : [WRN] FS_DEGRADED: 1 filesystem is degraded 2022-05-14T02:20:00.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:20:00 smithi047 conmon[77123]: 2022-05-14T02:19:59.999+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : fs cephfs is degraded 2022-05-14T02:20:00.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:20:00 smithi047 conmon[77123]: 2022-05-14T02:19:59.999+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : [ERR] MDS_DAMAGE: 1 mds daemon damaged 2022-05-14T02:20:00.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:20:00 smithi047 conmon[77123]: 2022-05-14T02:19:59.999+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : fs cephfs mds.2 is damaged 2022-05-14T02:20:00.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:20:00 smithi047 conmon[77123]: 2022-05-14T02:19:59.999+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : [WRN] MDS_READ_ONLY: 1 MDSs are read only 2022-05-14T02:20:00.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:20:00 smithi047 conmon[77123]: 2022-05-14T02:19:59.999+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : mds.l(mds.0): MDS in read-only mode 2022-05-14T02:20:00.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:20:00 smithi047 conmon[77123]: 2022-05-14T02:19:59.999+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : [WRN] MDS_SLOW_METADATA_IO: 1 MDSs report slow metadata IOs 2022-05-14T02:20:00.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:20:00 smithi047 conmon[77123]: 2022-05-14T02:19:59.999+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : mds.l(mds.0): 4 slow metadata IOs are blocked > 180 secs, oldest blocked for 10633 secs 2022-05-14T02:20:00.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:20:00 smithi047 conmon[77123]: 2022-05-14T02:19:59.999+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : [WRN] MON_DOWN: 1/3 mons down, quorum b,c 2022-05-14T02:20:00.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:20:00 smithi047 conmon[77123]: 2022-05-14T02:19:59.999+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : mon.a (rank 0) addr [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] is down (out of quorum) 2022-05-14T02:20:00.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:20:00 smithi047 conmon[77123]: 2022-05-14T02:19:59.999+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : [WRN] PG_AVAILABILITY: Reduced data availability: 1 pg inactive 2022-05-14T02:20:00.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:20:00 smithi047 conmon[77123]: 2022-05-14T02:19:59.999+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : pg 2.1 is stuck inactive for 3h, current state unknown, last acting [] 2022-05-14T02:20:00.295 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:20:00 smithi047 conmon[77123]: 2022-05-14T02:19:59.999+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : [WRN] SLOW_OPS: 9 slow ops, oldest one blocked for 10330 sec, osd.5 has slow ops 2022-05-14T02:20:00.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:20:00 smithi115 conmon[90393]: 2022-05-14T02:20:00.308+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:20:01.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:20:00 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:20:01.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:20:00 smithi047 ceph-mon[77147]: Health detail: HEALTH_ERR 1 filesystem is degraded; 1 mds daemon damaged; 1 MDSs are read only; 1 MDSs report slow metadata IOs; 1/3 mons down, quorum b,c; Reduced data availability: 1 pg inactive; 9 slow ops, oldest one blocked for 10330 sec, osd.5 has slow ops 2022-05-14T02:20:01.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:20:00 smithi047 ceph-mon[77147]: [WRN] FS_DEGRADED: 1 filesystem is degraded 2022-05-14T02:20:01.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:20:00 smithi047 ceph-mon[77147]: fs cephfs is degraded 2022-05-14T02:20:01.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:20:00 smithi047 ceph-mon[77147]: [ERR] MDS_DAMAGE: 1 mds daemon damaged 2022-05-14T02:20:01.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:20:00 smithi047 ceph-mon[77147]: fs cephfs mds.2 is damaged 2022-05-14T02:20:01.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:20:00 smithi047 ceph-mon[77147]: [WRN] MDS_READ_ONLY: 1 MDSs are read only 2022-05-14T02:20:01.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:20:00 smithi047 ceph-mon[77147]: mds.l(mds.0): MDS in read-only mode 2022-05-14T02:20:01.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:20:00 smithi047 ceph-mon[77147]: [WRN] MDS_SLOW_METADATA_IO: 1 MDSs report slow metadata IOs 2022-05-14T02:20:01.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:20:00 smithi047 ceph-mon[77147]: mds.l(mds.0): 4 slow metadata IOs are blocked > 180 secs, oldest blocked for 10633 secs 2022-05-14T02:20:01.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:20:00 smithi047 ceph-mon[77147]: [WRN] MON_DOWN: 1/3 mons down, quorum b,c 2022-05-14T02:20:01.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:20:00 smithi047 ceph-mon[77147]: mon.a (rank 0) addr [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] is down (out of quorum) 2022-05-14T02:20:01.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:20:00 smithi047 ceph-mon[77147]: [WRN] PG_AVAILABILITY: Reduced data availability: 1 pg inactive 2022-05-14T02:20:01.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:20:00 smithi047 ceph-mon[77147]: pg 2.1 is stuck inactive for 3h, current state unknown, last acting [] 2022-05-14T02:20:01.045 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:20:00 smithi047 ceph-mon[77147]: [WRN] SLOW_OPS: 9 slow ops, oldest one blocked for 10330 sec, osd.5 has slow ops 2022-05-14T02:20:01.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:20:00 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:20:01.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:20:00 smithi115 ceph-mon[78593]: Health detail: HEALTH_ERR 1 filesystem is degraded; 1 mds daemon damaged; 1 MDSs are read only; 1 MDSs report slow metadata IOs; 1/3 mons down, quorum b,c; Reduced data availability: 1 pg inactive; 9 slow ops, oldest one blocked for 10330 sec, osd.5 has slow ops 2022-05-14T02:20:01.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:20:00 smithi115 ceph-mon[78593]: [WRN] FS_DEGRADED: 1 filesystem is degraded 2022-05-14T02:20:01.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:20:00 smithi115 ceph-mon[78593]: fs cephfs is degraded 2022-05-14T02:20:01.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:20:00 smithi115 ceph-mon[78593]: [ERR] MDS_DAMAGE: 1 mds daemon damaged 2022-05-14T02:20:01.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:20:00 smithi115 ceph-mon[78593]: fs cephfs mds.2 is damaged 2022-05-14T02:20:01.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:20:00 smithi115 ceph-mon[78593]: [WRN] MDS_READ_ONLY: 1 MDSs are read only 2022-05-14T02:20:01.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:20:00 smithi115 ceph-mon[78593]: mds.l(mds.0): MDS in read-only mode 2022-05-14T02:20:01.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:20:00 smithi115 ceph-mon[78593]: [WRN] MDS_SLOW_METADATA_IO: 1 MDSs report slow metadata IOs 2022-05-14T02:20:01.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:20:00 smithi115 ceph-mon[78593]: mds.l(mds.0): 4 slow metadata IOs are blocked > 180 secs, oldest blocked for 10633 secs 2022-05-14T02:20:01.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:20:00 smithi115 ceph-mon[78593]: [WRN] MON_DOWN: 1/3 mons down, quorum b,c 2022-05-14T02:20:01.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:20:00 smithi115 ceph-mon[78593]: mon.a (rank 0) addr [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] is down (out of quorum) 2022-05-14T02:20:01.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:20:00 smithi115 ceph-mon[78593]: [WRN] PG_AVAILABILITY: Reduced data availability: 1 pg inactive 2022-05-14T02:20:01.112 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:20:00 smithi115 ceph-mon[78593]: pg 2.1 is stuck inactive for 3h, current state unknown, last acting [] 2022-05-14T02:20:01.112 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:20:00 smithi115 ceph-mon[78593]: [WRN] SLOW_OPS: 9 slow ops, oldest one blocked for 10330 sec, osd.5 has slow ops 2022-05-14T02:20:01.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:20:01 smithi115 conmon[90393]: 2022-05-14T02:20:01.262+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:20:02.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:20:01 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:20:02.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:20:01 smithi047 ceph-mon[77147]: pgmap v5519: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:20:02.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:20:01 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 10335 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:20:02.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:20:01 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:20:02.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:20:01 smithi115 ceph-mon[78593]: pgmap v5519: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:20:02.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:20:01 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 10335 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:20:02.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:20:02 smithi115 conmon[90393]: 2022-05-14T02:20:02.218+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:20:03.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:20:02 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:20:03.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:20:02 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:20:03.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:20:02 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:20:03.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:20:02 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:20:03.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:20:02 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:20:03.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:20:02 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:20:03.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:20:03 smithi115 conmon[90393]: 2022-05-14T02:20:03.189+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:20:04.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:20:03 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:20:04.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:20:03 smithi047 ceph-mon[77147]: pgmap v5520: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:20:04.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:20:03 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:20:04.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:20:03 smithi115 ceph-mon[78593]: pgmap v5520: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:20:04.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:20:04 smithi115 conmon[90393]: 2022-05-14T02:20:04.219+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:20:05.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:20:04 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:20:05.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:20:04 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:20:05.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:20:05 smithi115 conmon[90393]: 2022-05-14T02:20:05.202+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:20:06.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:20:05 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:20:06.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:20:05 smithi047 ceph-mon[77147]: pgmap v5521: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:20:06.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:20:05 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:20:06.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:20:05 smithi115 ceph-mon[78593]: pgmap v5521: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:20:06.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:20:06 smithi115 conmon[90393]: 2022-05-14T02:20:06.252+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:20:07.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:20:06 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:20:07.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:20:06 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:20:07.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:20:07 smithi115 conmon[90393]: 2022-05-14T02:20:07.280+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:20:08.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:20:07 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:20:08.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:20:07 smithi047 ceph-mon[77147]: pgmap v5522: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:20:08.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:20:07 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:20:08.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:20:07 smithi115 ceph-mon[78593]: pgmap v5522: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:20:08.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:20:08 smithi115 conmon[90393]: 2022-05-14T02:20:08.288+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:20:09.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:20:08 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:20:09.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:20:08 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:20:09.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:20:09 smithi115 conmon[90393]: 2022-05-14T02:20:09.285+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:20:10.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:20:09 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:20:10.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:20:09 smithi047 ceph-mon[77147]: pgmap v5523: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:20:10.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:20:09 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:20:10.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:20:09 smithi115 ceph-mon[78593]: pgmap v5523: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:20:10.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:20:10 smithi115 conmon[90393]: 2022-05-14T02:20:10.307+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:20:11.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:20:10 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:20:11.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:20:10 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:20:11.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:20:10 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:20:11.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:20:10 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 10340 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:20:11.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:20:10 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:20:11.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:20:10 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:20:11.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:20:10 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:20:11.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:20:10 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 10340 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:20:11.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:20:11 smithi115 conmon[90393]: 2022-05-14T02:20:11.350+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:20:12.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:20:11 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:20:12.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:20:11 smithi047 ceph-mon[77147]: pgmap v5524: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:20:12.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:20:11 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:20:12.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:20:11 smithi115 ceph-mon[78593]: pgmap v5524: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:20:12.680 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:20:12 smithi115 conmon[90393]: 2022-05-14T02:20:12.365+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:20:13.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:20:12 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:20:13.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:20:12 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:20:13.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:20:13 smithi115 conmon[90393]: 2022-05-14T02:20:13.339+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:20:14.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:20:13 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:20:14.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:20:13 smithi047 ceph-mon[77147]: pgmap v5525: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:20:14.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:20:13 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:20:14.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:20:13 smithi115 ceph-mon[78593]: pgmap v5525: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:20:14.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:20:14 smithi115 conmon[90393]: 2022-05-14T02:20:14.353+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:20:15.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:20:14 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:20:15.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:20:14 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:20:15.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:20:14 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:20:15.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:20:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T02:20:15.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:20:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T02:20:15.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:20:14 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:20:15.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:20:14 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:20:15.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:20:14 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:20:15.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:20:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T02:20:15.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:20:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T02:20:15.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:20:15 smithi115 conmon[90393]: 2022-05-14T02:20:15.315+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:20:16.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:20:15 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:20:16.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:20:15 smithi047 ceph-mon[77147]: pgmap v5526: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:20:16.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:20:15 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 10350 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:20:16.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:20:15 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:20:16.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:20:15 smithi115 ceph-mon[78593]: pgmap v5526: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:20:16.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:20:15 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 10350 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:20:16.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:20:16 smithi115 conmon[90393]: 2022-05-14T02:20:16.272+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:20:17.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:20:16 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:20:17.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:20:16 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:20:17.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:20:17 smithi115 conmon[90393]: 2022-05-14T02:20:17.310+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:20:18.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:20:17 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:20:18.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:20:17 smithi047 ceph-mon[77147]: pgmap v5527: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:20:18.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:20:17 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:20:18.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:20:17 smithi115 ceph-mon[78593]: pgmap v5527: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:20:18.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:20:18 smithi115 conmon[90393]: 2022-05-14T02:20:18.334+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:20:19.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:20:18 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:20:19.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:20:18 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:20:19.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:20:18 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:20:19.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:20:18 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:20:19.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:20:18 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:20:19.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:20:18 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:20:19.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:20:19 smithi115 conmon[90393]: 2022-05-14T02:20:19.313+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:20:20.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:20:19 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:20:20.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:20:19 smithi047 ceph-mon[77147]: pgmap v5528: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:20:20.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:20:19 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:20:20.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:20:19 smithi115 ceph-mon[78593]: pgmap v5528: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:20:20.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:20:20 smithi115 conmon[90393]: 2022-05-14T02:20:20.335+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:20:21.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:20:20 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:20:21.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:20:20 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:20:21.612 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:20:21 smithi115 conmon[90393]: 2022-05-14T02:20:21.352+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:20:22.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:20:21 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:20:22.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:20:21 smithi047 ceph-mon[77147]: pgmap v5529: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:20:22.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:20:21 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 10355 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:20:22.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:20:21 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:20:22.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:20:21 smithi115 ceph-mon[78593]: pgmap v5529: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:20:22.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:20:21 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 10355 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:20:22.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:20:22 smithi115 conmon[90393]: 2022-05-14T02:20:22.331+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:20:23.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:20:22 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:20:23.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:20:22 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:20:23.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:20:22 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:20:23.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:20:22 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:20:23.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:20:22 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:20:23.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:20:22 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:20:23.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:20:23 smithi115 conmon[90393]: 2022-05-14T02:20:23.328+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:20:24.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:20:23 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:20:24.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:20:23 smithi047 ceph-mon[77147]: pgmap v5530: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:20:24.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:20:23 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:20:24.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:20:23 smithi115 ceph-mon[78593]: pgmap v5530: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:20:24.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:20:24 smithi115 conmon[90393]: 2022-05-14T02:20:24.332+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:20:25.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:20:24 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:20:25.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:20:24 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:20:25.733 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:20:25 smithi115 conmon[90393]: 2022-05-14T02:20:25.358+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:20:26.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:20:25 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:20:26.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:20:25 smithi047 ceph-mon[77147]: pgmap v5531: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:20:26.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:20:25 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:20:26.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:20:25 smithi115 ceph-mon[78593]: pgmap v5531: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:20:26.738 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:20:26 smithi115 conmon[90393]: 2022-05-14T02:20:26.394+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:20:27.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:20:26 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:20:27.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:20:26 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:20:27.739 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:20:27 smithi115 conmon[90393]: 2022-05-14T02:20:27.365+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:20:28.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:20:27 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:20:28.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:20:27 smithi047 ceph-mon[77147]: pgmap v5532: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:20:28.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:20:27 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:20:28.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:20:27 smithi115 ceph-mon[78593]: pgmap v5532: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:20:28.744 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:20:28 smithi115 conmon[90393]: 2022-05-14T02:20:28.357+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:20:29.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:20:28 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:20:29.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:20:28 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:20:29.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:20:29 smithi115 conmon[90393]: 2022-05-14T02:20:29.340+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:20:30.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:20:29 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:20:30.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:20:29 smithi047 ceph-mon[77147]: pgmap v5533: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:20:30.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:20:29 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:20:30.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:20:29 smithi115 ceph-mon[78593]: pgmap v5533: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:20:30.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:20:30 smithi115 conmon[90393]: 2022-05-14T02:20:30.300+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:20:31.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:20:30 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:20:31.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:20:30 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:20:31.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:20:30 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:20:31.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:20:30 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 10360 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:20:31.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:20:30 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:20:31.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:20:30 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:20:31.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:20:30 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:20:31.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:20:30 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 10360 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:20:31.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:20:31 smithi115 conmon[90393]: 2022-05-14T02:20:31.307+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:20:32.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:20:31 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:20:32.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:20:31 smithi047 ceph-mon[77147]: pgmap v5534: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:20:32.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:20:31 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:20:32.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:20:31 smithi115 ceph-mon[78593]: pgmap v5534: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:20:32.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:20:32 smithi115 conmon[90393]: 2022-05-14T02:20:32.277+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:20:33.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:20:32 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:20:33.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:20:32 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:20:33.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:20:33 smithi115 conmon[90393]: 2022-05-14T02:20:33.231+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:20:34.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:20:33 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:20:34.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:20:33 smithi047 ceph-mon[77147]: pgmap v5535: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:20:34.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:20:33 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:20:34.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:20:33 smithi115 ceph-mon[78593]: pgmap v5535: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:20:34.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:20:34 smithi115 conmon[90393]: 2022-05-14T02:20:34.210+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:20:35.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:20:34 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:20:35.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:20:34 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:20:35.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:20:34 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:20:35.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:20:34 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:20:35.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:20:34 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:20:35.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:20:34 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:20:35.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:20:35 smithi115 conmon[90393]: 2022-05-14T02:20:35.245+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:20:36.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:20:35 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:20:36.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:20:35 smithi047 ceph-mon[77147]: pgmap v5536: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:20:36.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:20:35 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 10370 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:20:36.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:20:35 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:20:36.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:20:35 smithi115 ceph-mon[78593]: pgmap v5536: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:20:36.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:20:35 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 10370 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:20:36.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:20:36 smithi115 conmon[90393]: 2022-05-14T02:20:36.275+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:20:37.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:20:36 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:20:37.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:20:36 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:20:37.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:20:37 smithi115 conmon[90393]: 2022-05-14T02:20:37.264+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:20:38.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:20:37 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:20:38.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:20:37 smithi047 ceph-mon[77147]: pgmap v5537: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:20:38.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:20:37 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:20:38.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:20:37 smithi115 ceph-mon[78593]: pgmap v5537: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:20:38.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:20:38 smithi115 conmon[90393]: 2022-05-14T02:20:38.240+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:20:39.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:20:38 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:20:39.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:20:38 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:20:39.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:20:38 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:20:39.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:20:38 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:20:39.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:20:38 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:20:39.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:20:38 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:20:39.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:20:39 smithi115 conmon[90393]: 2022-05-14T02:20:39.290+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:20:40.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:20:39 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:20:40.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:20:39 smithi047 ceph-mon[77147]: pgmap v5538: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:20:40.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:20:39 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:20:40.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:20:39 smithi115 ceph-mon[78593]: pgmap v5538: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:20:40.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:20:40 smithi115 conmon[90393]: 2022-05-14T02:20:40.304+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:20:41.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:20:40 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:20:41.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:20:40 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:20:41.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:20:41 smithi115 conmon[90393]: 2022-05-14T02:20:41.352+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:20:42.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:20:41 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:20:42.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:20:41 smithi115 ceph-mon[78593]: pgmap v5539: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:20:42.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:20:41 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 10375 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:20:42.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:20:41 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:20:42.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:20:41 smithi047 ceph-mon[77147]: pgmap v5539: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:20:42.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:20:41 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 10375 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:20:42.806 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:20:42 smithi115 conmon[90393]: 2022-05-14T02:20:42.394+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:20:43.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:20:42 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:20:43.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:20:42 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:20:43.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:20:42 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:20:43.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:20:42 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:20:43.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:20:42 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:20:43.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:20:42 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:20:43.805 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:20:43 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:20:43.806 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:20:43 smithi115 ceph-mon[78593]: pgmap v5540: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:20:43.812 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:20:43 smithi115 conmon[90393]: 2022-05-14T02:20:43.441+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:20:44.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:20:43 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:20:44.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:20:43 smithi047 ceph-mon[77147]: pgmap v5540: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:20:44.810 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:20:44 smithi115 conmon[90393]: 2022-05-14T02:20:44.422+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:20:45.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:20:44 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:20:45.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:20:44 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:20:45.811 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:20:45 smithi115 conmon[90393]: 2022-05-14T02:20:45.457+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:20:46.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:20:45 smithi115 ceph-mon[78593]: pgmap v5541: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:20:46.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:20:45 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:20:46.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:20:45 smithi047 ceph-mon[77147]: pgmap v5541: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:20:46.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:20:45 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:20:46.816 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:20:46 smithi115 conmon[90393]: 2022-05-14T02:20:46.478+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:20:47.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:20:46 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:20:47.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:20:46 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:20:47.818 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:20:47 smithi115 conmon[90393]: 2022-05-14T02:20:47.472+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:20:48.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:20:47 smithi115 ceph-mon[78593]: pgmap v5542: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:20:48.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:20:47 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:20:48.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:20:47 smithi047 ceph-mon[77147]: pgmap v5542: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:20:48.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:20:47 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:20:48.822 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:20:48 smithi115 conmon[90393]: 2022-05-14T02:20:48.512+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:20:49.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:20:48 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:20:49.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:20:48 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:20:49.828 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:20:49 smithi115 conmon[90393]: 2022-05-14T02:20:49.551+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:20:50.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:20:49 smithi115 ceph-mon[78593]: pgmap v5543: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:20:50.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:20:49 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:20:50.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:20:49 smithi047 ceph-mon[77147]: pgmap v5543: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:20:50.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:20:49 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:20:50.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:20:50 smithi115 conmon[90393]: 2022-05-14T02:20:50.590+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:20:50.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:20:50 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:20:50.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:20:50 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:20:50.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:20:50 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:20:50.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:20:50 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 10380 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:20:51.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:20:50 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:20:51.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:20:50 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:20:51.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:20:50 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:20:51.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:20:50 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 10380 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:20:51.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:20:51 smithi115 conmon[90393]: 2022-05-14T02:20:51.606+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:20:51.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:20:51 smithi115 ceph-mon[78593]: pgmap v5544: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:20:51.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:20:51 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:20:52.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:20:51 smithi047 ceph-mon[77147]: pgmap v5544: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:20:52.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:20:51 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:20:53.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:20:52 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:20:53.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:20:52 smithi115 conmon[90393]: 2022-05-14T02:20:52.607+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:20:53.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:20:52 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:20:54.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:20:53 smithi115 conmon[90393]: 2022-05-14T02:20:53.631+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:20:54.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:20:53 smithi115 ceph-mon[78593]: pgmap v5545: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:20:54.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:20:53 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:20:54.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:20:53 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:20:54.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:20:53 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:20:54.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:20:53 smithi047 ceph-mon[77147]: pgmap v5545: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:20:54.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:20:53 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:20:54.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:20:53 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:20:54.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:20:53 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:20:54.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:20:54 smithi115 conmon[90393]: 2022-05-14T02:20:54.603+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:20:54.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:20:54 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:20:55.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:20:54 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:20:55.848 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:20:55 smithi115 conmon[90393]: 2022-05-14T02:20:55.559+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:20:56.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:20:55 smithi115 ceph-mon[78593]: pgmap v5546: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:20:56.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:20:55 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:20:56.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:20:55 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 10390 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:20:56.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:20:55 smithi047 ceph-mon[77147]: pgmap v5546: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:20:56.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:20:55 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:20:56.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:20:55 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 10390 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:20:56.854 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:20:56 smithi115 conmon[90393]: 2022-05-14T02:20:56.526+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:20:57.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:20:56 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:20:57.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:20:56 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:20:57.855 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:20:57 smithi115 conmon[90393]: 2022-05-14T02:20:57.508+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:20:58.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:20:57 smithi115 ceph-mon[78593]: pgmap v5547: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:20:58.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:20:57 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:20:58.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:20:57 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:20:58.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:20:57 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:20:58.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:20:57 smithi047 ceph-mon[77147]: pgmap v5547: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:20:58.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:20:57 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:20:58.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:20:57 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:20:58.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:20:57 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:20:58.837 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:20:58 smithi115 conmon[90393]: 2022-05-14T02:20:58.528+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:20:59.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:20:58 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:20:59.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:20:58 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T02:20:59.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:20:58 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T02:20:59.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:20:58 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T02:20:59.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:20:58 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' 2022-05-14T02:20:59.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:20:58 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:20:59.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:20:58 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T02:20:59.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:20:58 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T02:20:59.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:20:58 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T02:20:59.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:20:58 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' 2022-05-14T02:20:59.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:20:59 smithi115 conmon[90393]: 2022-05-14T02:20:59.536+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:21:00.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:20:59 smithi047 ceph-mon[77147]: pgmap v5548: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:21:00.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:20:59 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:21:00.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:20:59 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' 2022-05-14T02:21:00.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:20:59 smithi115 ceph-mon[78593]: pgmap v5548: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:21:00.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:20:59 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:21:00.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:20:59 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' 2022-05-14T02:21:00.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:21:00 smithi115 conmon[90393]: 2022-05-14T02:21:00.509+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:21:01.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:21:00 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:21:01.292 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:21:01 smithi047 conmon[78521]: 2022-05-14T02:21:01.039+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:21:01.292 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:21:01 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T02:21:01.292 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:21:01 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:21:01.293 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:21:01 smithi047 conmon[78521]: yield 2022-05-14T02:21:01.293 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:21:01 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:21:01.293 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:21:01 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:21:01.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:21:01 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:21:01.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:21:01 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T02:21:01.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:21:01 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:21:01.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:21:01 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T02:21:01.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:21:01 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:21:01.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:21:01 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T02:21:01.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:21:01 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:21:01.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:21:01 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T02:21:01.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:21:01 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:21:01.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:21:01 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T02:21:01.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:21:01 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:21:01.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:21:01 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:21:01.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:21:01 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:21:01.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:21:00 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:21:01.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:21:01 smithi115 conmon[90393]: 2022-05-14T02:21:01.558+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:21:02.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:21:01 smithi047 ceph-mon[77147]: pgmap v5549: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:21:02.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:21:01 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:21:02.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:21:01 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 10395 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:21:02.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:21:01 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:21:02.292 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T02:21:02.292 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:21:02.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T02:21:02.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:21:02.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:21:02.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:21:02.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T02:21:02.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:21:02.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T02:21:02.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:21:02.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T02:21:02.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:21:02.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T02:21:02.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:21:02.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T02:21:02.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:21:02.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:21:02.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:21:02.296 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:21:01 smithi047 ceph-mon[77147]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T02:21:02.297 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:21:01 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:21:02.297 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:21:01 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:21:02.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:21:01 smithi115 ceph-mon[78593]: pgmap v5549: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:21:02.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:21:01 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:21:02.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:21:01 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 10395 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:21:02.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:21:01 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:21:02.359 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T02:21:02.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:21:02.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T02:21:02.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:21:02.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:21:02.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:21:02.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T02:21:02.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:21:02.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T02:21:02.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:21:02.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T02:21:02.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:21:02.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T02:21:02.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:21:02.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T02:21:02.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:21:02.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:21:02.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:21:02.363 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:21:01 smithi115 ceph-mon[78593]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T02:21:02.364 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:21:01 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:21:02.364 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:21:01 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:21:02.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:21:02 smithi115 conmon[90393]: 2022-05-14T02:21:02.530+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:21:03.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:21:02 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:21:03.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:21:02 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:21:03.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:21:03 smithi115 conmon[90393]: 2022-05-14T02:21:03.539+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:21:04.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:21:03 smithi047 ceph-mon[77147]: pgmap v5550: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:21:04.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:21:03 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:21:04.292 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:21:04 smithi047 conmon[78521]: 2022-05-14T02:21:04.112+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:21:04.293 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:21:04 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T02:21:04.293 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:21:04 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:21:04.293 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:21:04 smithi047 conmon[78521]: yield 2022-05-14T02:21:04.293 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:21:04 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:21:04.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:21:04 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:21:04.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:21:04 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:21:04.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:21:04 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T02:21:04.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:21:04 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:21:04.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:21:04 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T02:21:04.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:21:04 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:21:04.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:21:04 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T02:21:04.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:21:04 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:21:04.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:21:04 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T02:21:04.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:21:04 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:21:04.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:21:04 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T02:21:04.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:21:04 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:21:04.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:21:04 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:21:04.297 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:21:04 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:21:04.297 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:21:04 smithi047 conmon[78521]: 2022-05-14T02:21:04.113+0000 7f534177b700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T02:21:04.297 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:21:04 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T02:21:04.297 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:21:04 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:21:04.298 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:21:04 smithi047 conmon[78521]: yield 2022-05-14T02:21:04.298 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:21:04 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:21:04.298 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:21:04 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:21:04.298 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:21:04 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:21:04.298 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:21:04 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T02:21:04.299 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:21:04 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:21:04.299 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:21:04 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T02:21:04.299 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:21:04 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:21:04.300 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:21:04 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T02:21:04.300 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:21:04 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:21:04.300 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:21:04 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T02:21:04.301 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:21:04 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:21:04.301 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:21:04 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T02:21:04.301 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:21:04 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:21:04.302 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:21:04 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:21:04.302 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:21:04 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:21:04.302 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:21:04 smithi047 conmon[78521]: 2022-05-14T02:21:04.303 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:21:04 smithi047 conmon[78521]: During handling of the above exception, another exception occurred: 2022-05-14T02:21:04.303 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:21:04 smithi047 conmon[78521]: 2022-05-14T02:21:04.304 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:21:04 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T02:21:04.304 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:21:04 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T02:21:04.304 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:21:04 smithi047 conmon[78521]: return f(*arg) 2022-05-14T02:21:04.305 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:21:04 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T02:21:04.305 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:21:04 smithi047 conmon[78521]: self._write_client_files(client_files, host) 2022-05-14T02:21:04.305 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:21:04 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T02:21:04.306 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:21:04 smithi047 conmon[78521]: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T02:21:04.306 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:21:04 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T02:21:04.306 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:21:04 smithi047 conmon[78521]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T02:21:04.307 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:21:04 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T02:21:04.307 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:21:04 smithi047 conmon[78521]: return self.event_loop.get_result(coro) 2022-05-14T02:21:04.307 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:21:04 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T02:21:04.308 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:21:04 smithi047 conmon[78521]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T02:21:04.308 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:21:04 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T02:21:04.308 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:21:04 smithi047 conmon[78521]: return self.__get_result() 2022-05-14T02:21:04.308 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:21:04 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T02:21:04.308 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:21:04 smithi047 conmon[78521]: raise self._exception 2022-05-14T02:21:04.309 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:21:04 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T02:21:04.309 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:21:04 smithi047 conmon[78521]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T02:21:04.309 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:21:04 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T02:21:04.309 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:21:04 smithi047 conmon[78521]: conn = await self._remote_connection(host, addr) 2022-05-14T02:21:04.310 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:21:04 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T02:21:04.310 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:21:04 smithi047 conmon[78521]: raise 2022-05-14T02:21:04.310 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:21:04 smithi047 conmon[78521]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T02:21:04.311 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:21:04 smithi047 conmon[78521]: self.gen.throw(type, value, traceback) 2022-05-14T02:21:04.311 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:21:04 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T02:21:04.311 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:21:04 smithi047 conmon[78521]: raise OrchestratorError(msg) 2022-05-14T02:21:04.311 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:21:04 smithi047 conmon[78521]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:21:04.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:21:03 smithi115 ceph-mon[78593]: pgmap v5550: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:21:04.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:21:03 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:21:04.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:21:04 smithi115 conmon[90393]: 2022-05-14T02:21:04.547+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:21:05.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:21:04 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:21:05.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:21:04 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:21:05.292 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T02:21:05.292 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:21:05.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T02:21:05.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:21:05.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:21:05.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:21:05.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T02:21:05.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:21:05.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T02:21:05.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:21:05.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T02:21:05.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:21:05.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T02:21:05.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:21:05.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T02:21:05.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:21:05.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:21:05.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:21:05.296 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:21:04 smithi047 ceph-mon[77147]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T02:21:05.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T02:21:05.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:21:05.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T02:21:05.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:21:05.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:21:05.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:21:05.298 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T02:21:05.298 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:21:05.299 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T02:21:05.299 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:21:05.299 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T02:21:05.299 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:21:05.299 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T02:21:05.300 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:21:05.300 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T02:21:05.300 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:21:05.300 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:21:05.300 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:21:05.301 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T02:21:05.301 INFO:journalctl@ceph.mon.b.smithi047.stdout: During handling of the above exception, another exception occurred: 2022-05-14T02:21:05.301 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T02:21:05.301 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T02:21:05.302 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T02:21:05.302 INFO:journalctl@ceph.mon.b.smithi047.stdout: return f(*arg) 2022-05-14T02:21:05.302 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T02:21:05.302 INFO:journalctl@ceph.mon.b.smithi047.stdout: self._write_client_files(client_files, host) 2022-05-14T02:21:05.302 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T02:21:05.303 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T02:21:05.303 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T02:21:05.303 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T02:21:05.303 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T02:21:05.303 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.event_loop.get_result(coro) 2022-05-14T02:21:05.304 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T02:21:05.304 INFO:journalctl@ceph.mon.b.smithi047.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T02:21:05.304 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T02:21:05.304 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.__get_result() 2022-05-14T02:21:05.305 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T02:21:05.305 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise self._exception 2022-05-14T02:21:05.305 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T02:21:05.305 INFO:journalctl@ceph.mon.b.smithi047.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T02:21:05.306 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T02:21:05.306 INFO:journalctl@ceph.mon.b.smithi047.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T02:21:05.306 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T02:21:05.306 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise 2022-05-14T02:21:05.307 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T02:21:05.307 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.gen.throw(type, value, traceback) 2022-05-14T02:21:05.307 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T02:21:05.307 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OrchestratorError(msg) 2022-05-14T02:21:05.307 INFO:journalctl@ceph.mon.b.smithi047.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:21:05.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:21:04 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:21:05.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:21:04 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:21:05.359 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T02:21:05.359 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:21:05.359 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T02:21:05.359 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:21:05.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:21:05.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:21:05.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T02:21:05.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:21:05.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T02:21:05.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:21:05.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T02:21:05.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:21:05.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T02:21:05.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:21:05.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T02:21:05.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:21:05.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:21:05.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:21:05.363 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:21:04 smithi115 ceph-mon[78593]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T02:21:05.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T02:21:05.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:21:05.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T02:21:05.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:21:05.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:21:05.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:21:05.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T02:21:05.365 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:21:05.365 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T02:21:05.365 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:21:05.365 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T02:21:05.365 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:21:05.366 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T02:21:05.366 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:21:05.366 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T02:21:05.366 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:21:05.367 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:21:05.367 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:21:05.367 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T02:21:05.367 INFO:journalctl@ceph.mon.c.smithi115.stdout: During handling of the above exception, another exception occurred: 2022-05-14T02:21:05.367 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T02:21:05.368 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T02:21:05.368 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T02:21:05.368 INFO:journalctl@ceph.mon.c.smithi115.stdout: return f(*arg) 2022-05-14T02:21:05.368 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T02:21:05.368 INFO:journalctl@ceph.mon.c.smithi115.stdout: self._write_client_files(client_files, host) 2022-05-14T02:21:05.369 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T02:21:05.369 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T02:21:05.369 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T02:21:05.369 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T02:21:05.369 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T02:21:05.370 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.event_loop.get_result(coro) 2022-05-14T02:21:05.370 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T02:21:05.370 INFO:journalctl@ceph.mon.c.smithi115.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T02:21:05.370 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T02:21:05.371 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.__get_result() 2022-05-14T02:21:05.371 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T02:21:05.371 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise self._exception 2022-05-14T02:21:05.371 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T02:21:05.372 INFO:journalctl@ceph.mon.c.smithi115.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T02:21:05.372 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T02:21:05.372 INFO:journalctl@ceph.mon.c.smithi115.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T02:21:05.372 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T02:21:05.373 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise 2022-05-14T02:21:05.373 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T02:21:05.373 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.gen.throw(type, value, traceback) 2022-05-14T02:21:05.373 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T02:21:05.374 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OrchestratorError(msg) 2022-05-14T02:21:05.374 INFO:journalctl@ceph.mon.c.smithi115.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:21:05.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:21:05 smithi115 conmon[90393]: 2022-05-14T02:21:05.527+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:21:06.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:21:05 smithi047 ceph-mon[77147]: pgmap v5551: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:21:06.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:21:05 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:21:06.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:21:05 smithi115 ceph-mon[78593]: pgmap v5551: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:21:06.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:21:05 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:21:06.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:21:06 smithi115 conmon[90393]: 2022-05-14T02:21:06.519+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:21:07.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:21:06 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:21:07.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:21:06 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:21:07.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:21:07 smithi115 conmon[90393]: 2022-05-14T02:21:07.546+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:21:08.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:21:07 smithi047 ceph-mon[77147]: pgmap v5552: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:21:08.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:21:07 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:21:08.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:21:07 smithi115 ceph-mon[78593]: pgmap v5552: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:21:08.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:21:07 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:21:08.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:21:08 smithi115 conmon[90393]: 2022-05-14T02:21:08.523+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:21:09.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:21:08 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:21:09.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:21:08 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:21:09.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:21:09 smithi115 conmon[90393]: 2022-05-14T02:21:09.516+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:21:10.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:21:09 smithi047 ceph-mon[77147]: pgmap v5553: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:21:10.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:21:09 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:21:10.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:21:09 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:21:10.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:21:09 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:21:10.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:21:09 smithi115 ceph-mon[78593]: pgmap v5553: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:21:10.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:21:09 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:21:10.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:21:09 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:21:10.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:21:09 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:21:10.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:21:10 smithi115 conmon[90393]: 2022-05-14T02:21:10.517+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:21:11.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:21:10 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:21:11.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:21:10 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 10400 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:21:11.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:21:10 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:21:11.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:21:10 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 10400 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:21:11.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:21:11 smithi115 conmon[90393]: 2022-05-14T02:21:11.532+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:21:12.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:21:11 smithi047 ceph-mon[77147]: pgmap v5554: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:21:12.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:21:11 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:21:12.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:21:11 smithi115 ceph-mon[78593]: pgmap v5554: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:21:12.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:21:11 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:21:12.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:21:12 smithi115 conmon[90393]: 2022-05-14T02:21:12.532+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:21:13.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:21:12 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:21:13.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:21:12 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:21:13.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:21:13 smithi115 conmon[90393]: 2022-05-14T02:21:13.527+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:21:14.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:21:13 smithi047 ceph-mon[77147]: pgmap v5555: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:21:14.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:21:13 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:21:14.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:21:13 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:21:14.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:21:13 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:21:14.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:21:13 smithi115 ceph-mon[78593]: pgmap v5555: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:21:14.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:21:13 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:21:14.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:21:13 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:21:14.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:21:13 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:21:14.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:21:14 smithi115 conmon[90393]: 2022-05-14T02:21:14.553+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:21:15.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:21:14 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:21:15.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:21:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T02:21:15.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:21:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T02:21:15.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:21:14 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:21:15.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:21:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T02:21:15.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:21:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T02:21:15.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:21:15 smithi115 conmon[90393]: 2022-05-14T02:21:15.512+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:21:16.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:21:15 smithi047 ceph-mon[77147]: pgmap v5556: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:21:16.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:21:15 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:21:16.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:21:15 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 10410 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:21:16.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:21:15 smithi115 ceph-mon[78593]: pgmap v5556: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:21:16.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:21:15 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:21:16.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:21:15 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 10410 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:21:16.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:21:16 smithi115 conmon[90393]: 2022-05-14T02:21:16.490+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:21:17.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:21:16 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:21:17.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:21:16 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:21:17.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:21:17 smithi115 conmon[90393]: 2022-05-14T02:21:17.521+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:21:18.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:21:17 smithi047 ceph-mon[77147]: pgmap v5557: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:21:18.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:21:17 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:21:18.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:21:17 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:21:18.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:21:17 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:21:18.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:21:17 smithi115 ceph-mon[78593]: pgmap v5557: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:21:18.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:21:17 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:21:18.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:21:17 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:21:18.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:21:17 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:21:18.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:21:18 smithi115 conmon[90393]: 2022-05-14T02:21:18.540+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:21:19.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:21:18 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:21:19.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:21:18 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:21:19.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:21:19 smithi115 conmon[90393]: 2022-05-14T02:21:19.528+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:21:20.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:21:19 smithi047 ceph-mon[77147]: pgmap v5558: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:21:20.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:21:19 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:21:20.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:21:19 smithi115 ceph-mon[78593]: pgmap v5558: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:21:20.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:21:19 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:21:20.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:21:20 smithi115 conmon[90393]: 2022-05-14T02:21:20.484+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:21:21.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:21:20 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:21:21.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:21:20 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:21:21.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:21:21 smithi115 conmon[90393]: 2022-05-14T02:21:21.474+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:21:22.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:21:21 smithi047 ceph-mon[77147]: pgmap v5559: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:21:22.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:21:21 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:21:22.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:21:21 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 10415 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:21:22.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:21:21 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:21:22.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:21:21 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:21:22.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:21:21 smithi115 ceph-mon[78593]: pgmap v5559: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:21:22.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:21:21 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:21:22.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:21:21 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 10415 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:21:22.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:21:21 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:21:22.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:21:21 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:21:22.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:21:22 smithi115 conmon[90393]: 2022-05-14T02:21:22.493+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:21:23.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:21:22 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:21:23.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:21:22 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:21:23.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:21:23 smithi115 conmon[90393]: 2022-05-14T02:21:23.530+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:21:24.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:21:23 smithi047 ceph-mon[77147]: pgmap v5560: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:21:24.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:21:23 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:21:24.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:21:23 smithi115 ceph-mon[78593]: pgmap v5560: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:21:24.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:21:23 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:21:24.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:21:24 smithi115 conmon[90393]: 2022-05-14T02:21:24.577+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:21:25.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:21:24 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:21:25.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:21:24 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:21:25.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:21:25 smithi115 conmon[90393]: 2022-05-14T02:21:25.556+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:21:26.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:21:25 smithi047 ceph-mon[77147]: pgmap v5561: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 7 op/s 2022-05-14T02:21:26.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:21:25 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:21:26.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:21:25 smithi115 ceph-mon[78593]: pgmap v5561: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 7 op/s 2022-05-14T02:21:26.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:21:25 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:21:26.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:21:26 smithi115 conmon[90393]: 2022-05-14T02:21:26.560+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:21:27.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:21:26 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:21:27.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:21:26 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:21:27.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:21:27 smithi115 conmon[90393]: 2022-05-14T02:21:27.533+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:21:28.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:21:27 smithi047 ceph-mon[77147]: pgmap v5562: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 4 op/s 2022-05-14T02:21:28.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:21:27 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:21:28.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:21:27 smithi115 ceph-mon[78593]: pgmap v5562: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 4 op/s 2022-05-14T02:21:28.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:21:27 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:21:28.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:21:28 smithi115 conmon[90393]: 2022-05-14T02:21:28.520+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:21:29.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:21:28 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:21:29.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:21:28 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:21:29.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:21:29 smithi115 conmon[90393]: 2022-05-14T02:21:29.491+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:21:30.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:21:29 smithi047 ceph-mon[77147]: pgmap v5563: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 4 op/s 2022-05-14T02:21:30.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:21:29 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:21:30.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:21:29 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:21:30.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:21:29 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:21:30.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:21:29 smithi115 ceph-mon[78593]: pgmap v5563: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 4 op/s 2022-05-14T02:21:30.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:21:29 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:21:30.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:21:29 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:21:30.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:21:29 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:21:30.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:21:30 smithi115 conmon[90393]: 2022-05-14T02:21:30.459+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:21:31.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:21:30 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:21:31.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:21:30 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 10420 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:21:31.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:21:30 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:21:31.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:21:30 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 10420 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:21:31.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:21:31 smithi115 conmon[90393]: 2022-05-14T02:21:31.423+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:21:32.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:21:31 smithi047 ceph-mon[77147]: pgmap v5564: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 7 op/s 2022-05-14T02:21:32.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:21:31 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:21:32.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:21:31 smithi115 ceph-mon[78593]: pgmap v5564: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 7 op/s 2022-05-14T02:21:32.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:21:31 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:21:32.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:21:32 smithi115 conmon[90393]: 2022-05-14T02:21:32.411+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:21:33.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:21:32 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:21:33.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:21:32 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:21:33.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:21:33 smithi115 conmon[90393]: 2022-05-14T02:21:33.374+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:21:34.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:21:33 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:21:34.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:21:33 smithi047 ceph-mon[77147]: pgmap v5565: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 4 op/s 2022-05-14T02:21:34.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:21:33 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:21:34.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:21:33 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:21:34.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:21:33 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:21:34.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:21:33 smithi115 ceph-mon[78593]: pgmap v5565: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 4 op/s 2022-05-14T02:21:34.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:21:33 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:21:34.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:21:33 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:21:34.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:21:34 smithi115 conmon[90393]: 2022-05-14T02:21:34.388+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:21:35.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:21:34 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:21:35.343 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:21:34 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:21:35.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:21:35 smithi115 conmon[90393]: 2022-05-14T02:21:35.341+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:21:36.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:21:35 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:21:36.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:21:35 smithi047 ceph-mon[77147]: pgmap v5566: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 7 op/s 2022-05-14T02:21:36.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:21:35 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 10430 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:21:36.304 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:21:35 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:21:36.304 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:21:35 smithi115 ceph-mon[78593]: pgmap v5566: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 7 op/s 2022-05-14T02:21:36.304 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:21:35 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 10430 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:21:36.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:21:36 smithi115 conmon[90393]: 2022-05-14T02:21:36.302+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:21:37.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:21:36 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:21:37.340 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:21:36 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:21:37.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:21:37 smithi115 conmon[90393]: 2022-05-14T02:21:37.338+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:21:38.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:21:37 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:21:38.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:21:37 smithi047 ceph-mon[77147]: pgmap v5567: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:21:38.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:21:37 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:21:38.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:21:37 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:21:38.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:21:37 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:21:38.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:21:37 smithi115 ceph-mon[78593]: pgmap v5567: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:21:38.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:21:37 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:21:38.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:21:37 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:21:38.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:21:38 smithi115 conmon[90393]: 2022-05-14T02:21:38.373+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:21:39.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:21:39 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:21:39.350 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:21:39 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:21:39.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:21:39 smithi115 conmon[90393]: 2022-05-14T02:21:39.349+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:21:40.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:21:40 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:21:40.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:21:40 smithi047 ceph-mon[77147]: pgmap v5568: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:21:40.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:21:40 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:21:40.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:21:40 smithi115 ceph-mon[78593]: pgmap v5568: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:21:40.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:21:40 smithi115 conmon[90393]: 2022-05-14T02:21:40.386+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:21:41.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:21:41 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:21:41.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:21:41 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:21:41.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:21:41 smithi115 conmon[90393]: 2022-05-14T02:21:41.369+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:21:42.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:21:42 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:21:42.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:21:42 smithi047 ceph-mon[77147]: pgmap v5569: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:21:42.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:21:42 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 10435 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:21:42.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:21:42 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:21:42.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:21:42 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:21:42.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:21:42 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:21:42.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:21:42 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:21:42.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:21:42 smithi115 ceph-mon[78593]: pgmap v5569: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:21:42.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:21:42 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 10435 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:21:42.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:21:42 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:21:42.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:21:42 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:21:42.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:21:42 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:21:42.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:21:42 smithi115 conmon[90393]: 2022-05-14T02:21:42.388+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:21:43.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:21:43 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:21:43.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:21:43 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:21:43.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:21:43 smithi115 conmon[90393]: 2022-05-14T02:21:43.437+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:21:44.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:21:44 smithi047 ceph-mon[77147]: pgmap v5570: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:21:44.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:21:44 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:21:44.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:21:44 smithi115 ceph-mon[78593]: pgmap v5570: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:21:44.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:21:44 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:21:44.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:21:44 smithi115 conmon[90393]: 2022-05-14T02:21:44.474+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:21:45.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:21:45 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:21:45.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:21:45 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:21:45.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:21:45 smithi115 conmon[90393]: 2022-05-14T02:21:45.511+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:21:46.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:21:46 smithi047 ceph-mon[77147]: pgmap v5571: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:21:46.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:21:46 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:21:46.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:21:46 smithi115 ceph-mon[78593]: pgmap v5571: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:21:46.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:21:46 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:21:46.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:21:46 smithi115 conmon[90393]: 2022-05-14T02:21:46.463+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:21:47.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:21:47 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:21:47.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:21:47 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:21:47.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:21:47 smithi115 conmon[90393]: 2022-05-14T02:21:47.421+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:21:48.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:21:48 smithi047 ceph-mon[77147]: pgmap v5572: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:21:48.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:21:48 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:21:48.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:21:48 smithi115 ceph-mon[78593]: pgmap v5572: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:21:48.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:21:48 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:21:48.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:21:48 smithi115 conmon[90393]: 2022-05-14T02:21:48.421+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:21:49.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:21:49 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:21:49.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:21:49 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:21:49.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:21:49 smithi115 conmon[90393]: 2022-05-14T02:21:49.404+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:21:50.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:21:50 smithi047 ceph-mon[77147]: pgmap v5573: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:21:50.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:21:50 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:21:50.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:21:50 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:21:50.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:21:50 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:21:50.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:21:50 smithi115 ceph-mon[78593]: pgmap v5573: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:21:50.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:21:50 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:21:50.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:21:50 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:21:50.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:21:50 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:21:50.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:21:50 smithi115 conmon[90393]: 2022-05-14T02:21:50.414+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:21:51.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:21:51 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:21:51.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:21:51 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 10440 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:21:51.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:21:51 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:21:51.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:21:51 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 10440 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:21:51.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:21:51 smithi115 conmon[90393]: 2022-05-14T02:21:51.440+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:21:52.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:21:52 smithi047 ceph-mon[77147]: pgmap v5574: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:21:52.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:21:52 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:21:52.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:21:52 smithi115 ceph-mon[78593]: pgmap v5574: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:21:52.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:21:52 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:21:52.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:21:52 smithi115 conmon[90393]: 2022-05-14T02:21:52.447+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:21:53.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:21:53 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:21:53.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:21:53 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:21:53.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:21:53 smithi115 conmon[90393]: 2022-05-14T02:21:53.462+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:21:54.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:21:54 smithi115 ceph-mon[78593]: pgmap v5575: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:21:54.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:21:54 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:21:54.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:21:54 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:21:54.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:21:54 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:21:54.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:21:54 smithi047 ceph-mon[77147]: pgmap v5575: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:21:54.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:21:54 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:21:54.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:21:54 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:21:54.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:21:54 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:21:54.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:21:54 smithi115 conmon[90393]: 2022-05-14T02:21:54.454+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:21:55.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:21:55 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:21:55.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:21:55 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:21:55.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:21:55 smithi115 conmon[90393]: 2022-05-14T02:21:55.423+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:21:56.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:21:56 smithi115 ceph-mon[78593]: pgmap v5576: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:21:56.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:21:56 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:21:56.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:21:56 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 10450 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:21:56.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:21:56 smithi047 ceph-mon[77147]: pgmap v5576: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:21:56.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:21:56 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:21:56.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:21:56 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 10450 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:21:56.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:21:56 smithi115 conmon[90393]: 2022-05-14T02:21:56.418+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:21:57.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:21:57 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:21:57.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:21:57 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:21:57.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:21:57 smithi115 conmon[90393]: 2022-05-14T02:21:57.381+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:21:58.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:21:58 smithi115 ceph-mon[78593]: pgmap v5577: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:21:58.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:21:58 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:21:58.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:21:58 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:21:58.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:21:58 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:21:58.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:21:58 smithi047 ceph-mon[77147]: pgmap v5577: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:21:58.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:21:58 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:21:58.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:21:58 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:21:58.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:21:58 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:21:58.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:21:58 smithi115 conmon[90393]: 2022-05-14T02:21:58.396+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:21:59.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:21:59 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:21:59.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:21:59 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:21:59.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:21:59 smithi115 conmon[90393]: 2022-05-14T02:21:59.363+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:22:00.349 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:22:00 smithi115 ceph-mon[78593]: pgmap v5578: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:22:00.349 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:22:00 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:22:00.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:22:00 smithi047 ceph-mon[77147]: pgmap v5578: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:22:00.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:22:00 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:22:00.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:22:00 smithi115 conmon[90393]: 2022-05-14T02:22:00.348+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:22:01.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:22:01 smithi115 conmon[90393]: 2022-05-14T02:22:01.372+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:22:01.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:22:01 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:22:01.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:22:01 smithi115 ceph-mon[78593]: pgmap v5579: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:22:01.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:22:01 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 10455 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:22:02.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:22:01 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:22:02.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:22:01 smithi047 ceph-mon[77147]: pgmap v5579: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:22:02.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:22:01 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 10455 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:22:02.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:22:02 smithi115 conmon[90393]: 2022-05-14T02:22:02.369+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:22:02.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:22:02 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:22:02.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:22:02 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:22:02.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:22:02 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:22:03.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:22:02 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:22:03.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:22:02 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:22:03.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:22:02 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:22:03.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:22:03 smithi115 conmon[90393]: 2022-05-14T02:22:03.345+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:22:03.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:22:03 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:22:03.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:22:03 smithi115 ceph-mon[78593]: pgmap v5580: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:22:04.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:22:03 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:22:04.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:22:03 smithi047 ceph-mon[77147]: pgmap v5580: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:22:04.571 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:22:04 smithi115 conmon[90393]: 2022-05-14T02:22:04.361+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:22:04.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:22:04 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:22:04.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:22:04 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T02:22:04.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:22:04 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T02:22:04.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:22:04 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T02:22:04.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:22:04 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' 2022-05-14T02:22:04.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:22:04 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' 2022-05-14T02:22:05.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:22:04 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:22:05.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:22:04 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T02:22:05.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:22:04 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T02:22:05.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:22:04 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T02:22:05.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:22:04 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' 2022-05-14T02:22:05.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:22:04 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' 2022-05-14T02:22:05.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:22:05 smithi115 conmon[90393]: 2022-05-14T02:22:05.391+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:22:05.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:22:05 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:22:05.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:22:05 smithi115 ceph-mon[78593]: pgmap v5581: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:22:06.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:22:05 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:22:06.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:22:05 smithi047 ceph-mon[77147]: pgmap v5581: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:22:06.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:22:06 smithi115 conmon[90393]: 2022-05-14T02:22:06.424+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:22:06.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:22:06 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:22:07.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:22:06 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:22:07.541 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:22:07 smithi047 conmon[78521]: 2022-05-14T02:22:07.216+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:22:07.542 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:22:07 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T02:22:07.542 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:22:07 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:22:07.542 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:22:07 smithi047 conmon[78521]: yield 2022-05-14T02:22:07.542 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:22:07 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:22:07.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:22:07 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:22:07.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:22:07 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:22:07.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:22:07 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T02:22:07.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:22:07 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:22:07.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:22:07 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T02:22:07.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:22:07 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:22:07.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:22:07 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T02:22:07.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:22:07 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:22:07.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:22:07 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T02:22:07.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:22:07 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:22:07.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:22:07 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T02:22:07.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:22:07 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:22:07.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:22:07 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:22:07.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:22:07 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:22:07.605 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:22:07 smithi115 conmon[90393]: 2022-05-14T02:22:07.421+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:22:07.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:22:07 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:22:07.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:22:07 smithi115 ceph-mon[78593]: pgmap v5582: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:22:07.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:22:07 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' 2022-05-14T02:22:07.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:22:07 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:22:07.859 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T02:22:07.859 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:22:07.859 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T02:22:07.860 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:22:07.860 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:22:07.860 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:22:07.860 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T02:22:07.860 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:22:07.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T02:22:07.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:22:07.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T02:22:07.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:22:07.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T02:22:07.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:22:07.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T02:22:07.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:22:07.863 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:22:07.863 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:22:07.863 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:22:07 smithi115 ceph-mon[78593]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T02:22:07.863 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:22:07 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' 2022-05-14T02:22:08.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:22:07 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:22:08.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:22:07 smithi047 ceph-mon[77147]: pgmap v5582: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:22:08.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:22:07 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' 2022-05-14T02:22:08.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:22:07 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:22:08.042 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T02:22:08.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:22:08.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T02:22:08.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:22:08.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:22:08.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:22:08.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T02:22:08.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:22:08.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T02:22:08.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:22:08.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T02:22:08.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:22:08.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T02:22:08.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:22:08.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T02:22:08.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:22:08.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:22:08.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:22:08.046 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:22:07 smithi047 ceph-mon[77147]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T02:22:08.046 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:22:07 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' 2022-05-14T02:22:08.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:22:08 smithi115 conmon[90393]: 2022-05-14T02:22:08.392+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:22:08.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:22:08 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:22:08.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:22:08 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-05-14T02:22:08.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:22:08 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-05-14T02:22:08.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:22:08 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-05-14T02:22:08.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:22:08 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-05-14T02:22:08.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:22:08 smithi115 ceph-mon[78593]: Adjusting osd_memory_target on smithi047 to 198.1M 2022-05-14T02:22:08.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:22:08 smithi115 ceph-mon[78593]: Unable to set osd_memory_target on smithi047 to 207755878: error parsing value: Value '207755878' is below minimum 939524096 2022-05-14T02:22:08.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:22:08 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-05-14T02:22:08.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:22:08 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-05-14T02:22:08.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:22:08 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-05-14T02:22:08.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:22:08 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-05-14T02:22:08.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:22:08 smithi115 ceph-mon[78593]: Adjusting osd_memory_target on smithi115 to 197.0M 2022-05-14T02:22:08.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:22:08 smithi115 ceph-mon[78593]: Unable to set osd_memory_target on smithi115 to 206629785: error parsing value: Value '206629785' is below minimum 939524096 2022-05-14T02:22:09.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:22:08 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:22:09.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:22:08 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-05-14T02:22:09.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:22:08 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-05-14T02:22:09.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:22:08 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-05-14T02:22:09.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:22:08 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-05-14T02:22:09.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:22:08 smithi047 ceph-mon[77147]: Adjusting osd_memory_target on smithi047 to 198.1M 2022-05-14T02:22:09.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:22:08 smithi047 ceph-mon[77147]: Unable to set osd_memory_target on smithi047 to 207755878: error parsing value: Value '207755878' is below minimum 939524096 2022-05-14T02:22:09.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:22:08 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-05-14T02:22:09.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:22:08 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-05-14T02:22:09.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:22:08 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-05-14T02:22:09.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:22:08 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-05-14T02:22:09.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:22:08 smithi047 ceph-mon[77147]: Adjusting osd_memory_target on smithi115 to 197.0M 2022-05-14T02:22:09.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:22:08 smithi047 ceph-mon[77147]: Unable to set osd_memory_target on smithi115 to 206629785: error parsing value: Value '206629785' is below minimum 939524096 2022-05-14T02:22:09.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:22:09 smithi115 conmon[90393]: 2022-05-14T02:22:09.388+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:22:09.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:22:09 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:22:09.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:22:09 smithi115 ceph-mon[78593]: pgmap v5583: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:22:10.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:22:09 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:22:10.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:22:09 smithi047 ceph-mon[77147]: pgmap v5583: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:22:10.542 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:22:10 smithi047 conmon[78521]: 2022-05-14T02:22:10.287+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:22:10.542 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:22:10 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T02:22:10.542 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:22:10 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:22:10.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:22:10 smithi047 conmon[78521]: yield 2022-05-14T02:22:10.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:22:10 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:22:10.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:22:10 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:22:10.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:22:10 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:22:10.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:22:10 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T02:22:10.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:22:10 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:22:10.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:22:10 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T02:22:10.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:22:10 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:22:10.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:22:10 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T02:22:10.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:22:10 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:22:10.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:22:10 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T02:22:10.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:22:10 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:22:10.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:22:10 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T02:22:10.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:22:10 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:22:10.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:22:10 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:22:10.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:22:10 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:22:10.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:22:10 smithi047 conmon[78521]: 2022-05-14T02:22:10.288+0000 7f5341f7c700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T02:22:10.547 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:22:10 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T02:22:10.547 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:22:10 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:22:10.547 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:22:10 smithi047 conmon[78521]: yield 2022-05-14T02:22:10.547 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:22:10 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:22:10.547 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:22:10 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:22:10.548 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:22:10 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:22:10.548 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:22:10 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T02:22:10.548 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:22:10 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:22:10.548 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:22:10 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T02:22:10.549 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:22:10 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:22:10.549 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:22:10 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T02:22:10.549 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:22:10 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:22:10.549 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:22:10 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T02:22:10.549 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:22:10 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:22:10.550 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:22:10 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T02:22:10.550 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:22:10 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:22:10.550 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:22:10 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:22:10.550 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:22:10 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:22:10.551 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:22:10 smithi047 conmon[78521]: 2022-05-14T02:22:10.551 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:22:10 smithi047 conmon[78521]: During handling of the above exception, another exception occurred: 2022-05-14T02:22:10.551 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:22:10 smithi047 conmon[78521]: 2022-05-14T02:22:10.551 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:22:10 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T02:22:10.551 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:22:10 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T02:22:10.552 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:22:10 smithi047 conmon[78521]: return f(*arg) 2022-05-14T02:22:10.552 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:22:10 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T02:22:10.552 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:22:10 smithi047 conmon[78521]: self._write_client_files(client_files, host) 2022-05-14T02:22:10.552 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:22:10 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T02:22:10.553 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:22:10 smithi047 conmon[78521]: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T02:22:10.553 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:22:10 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T02:22:10.553 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:22:10 smithi047 conmon[78521]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T02:22:10.553 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:22:10 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T02:22:10.554 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:22:10 smithi047 conmon[78521]: return self.event_loop.get_result(coro) 2022-05-14T02:22:10.554 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:22:10 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T02:22:10.554 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:22:10 smithi047 conmon[78521]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T02:22:10.554 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:22:10 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T02:22:10.554 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:22:10 smithi047 conmon[78521]: return self.__get_result() 2022-05-14T02:22:10.555 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:22:10 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T02:22:10.555 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:22:10 smithi047 conmon[78521]: raise self._exception 2022-05-14T02:22:10.555 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:22:10 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T02:22:10.555 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:22:10 smithi047 conmon[78521]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T02:22:10.555 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:22:10 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T02:22:10.556 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:22:10 smithi047 conmon[78521]: conn = await self._remote_connection(host, addr) 2022-05-14T02:22:10.556 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:22:10 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T02:22:10.556 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:22:10 smithi047 conmon[78521]: raise 2022-05-14T02:22:10.556 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:22:10 smithi047 conmon[78521]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T02:22:10.556 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:22:10 smithi047 conmon[78521]: self.gen.throw(type, value, traceback) 2022-05-14T02:22:10.557 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:22:10 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T02:22:10.557 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:22:10 smithi047 conmon[78521]: raise OrchestratorError(msg) 2022-05-14T02:22:10.557 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:22:10 smithi047 conmon[78521]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:22:10.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:22:10 smithi115 conmon[90393]: 2022-05-14T02:22:10.421+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:22:10.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:22:10 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:22:10.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:22:10 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:22:10.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:22:10 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:22:10.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:22:10 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 10460 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:22:11.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:22:10 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:22:11.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:22:10 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:22:11.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:22:10 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:22:11.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:22:10 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 10460 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:22:11.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:22:11 smithi115 conmon[90393]: 2022-05-14T02:22:11.403+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:22:11.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:22:11 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:22:11.859 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T02:22:11.859 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:22:11.859 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T02:22:11.859 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:22:11.860 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:22:11.860 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:22:11.860 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T02:22:11.860 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:22:11.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T02:22:11.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:22:11.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T02:22:11.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:22:11.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T02:22:11.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:22:11.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T02:22:11.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:22:11.863 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:22:11.863 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:22:11.863 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:22:11 smithi115 ceph-mon[78593]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T02:22:11.864 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T02:22:11.864 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:22:11.864 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T02:22:11.864 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:22:11.864 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:22:11.865 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:22:11.865 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T02:22:11.865 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:22:11.865 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T02:22:11.866 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:22:11.866 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T02:22:11.866 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:22:11.866 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T02:22:11.867 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:22:11.867 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T02:22:11.867 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:22:11.867 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:22:11.868 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:22:11.868 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T02:22:11.868 INFO:journalctl@ceph.mon.c.smithi115.stdout: During handling of the above exception, another exception occurred: 2022-05-14T02:22:11.868 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T02:22:11.869 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T02:22:11.869 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T02:22:11.869 INFO:journalctl@ceph.mon.c.smithi115.stdout: return f(*arg) 2022-05-14T02:22:11.869 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T02:22:11.870 INFO:journalctl@ceph.mon.c.smithi115.stdout: self._write_client_files(client_files, host) 2022-05-14T02:22:11.870 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T02:22:11.870 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T02:22:11.870 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T02:22:11.871 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T02:22:11.871 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T02:22:11.871 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.event_loop.get_result(coro) 2022-05-14T02:22:11.871 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T02:22:11.871 INFO:journalctl@ceph.mon.c.smithi115.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T02:22:11.872 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T02:22:11.872 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.__get_result() 2022-05-14T02:22:11.872 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T02:22:11.872 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise self._exception 2022-05-14T02:22:11.873 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T02:22:11.873 INFO:journalctl@ceph.mon.c.smithi115.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T02:22:11.873 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T02:22:11.873 INFO:journalctl@ceph.mon.c.smithi115.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T02:22:11.874 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T02:22:11.874 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise 2022-05-14T02:22:11.874 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T02:22:11.874 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.gen.throw(type, value, traceback) 2022-05-14T02:22:11.874 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T02:22:11.875 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OrchestratorError(msg) 2022-05-14T02:22:11.875 INFO:journalctl@ceph.mon.c.smithi115.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:22:11.875 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:22:11 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:22:11.875 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:22:11 smithi115 ceph-mon[78593]: pgmap v5584: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:22:12.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:22:11 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:22:12.042 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T02:22:12.042 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:22:12.042 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T02:22:12.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:22:12.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:22:12.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:22:12.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T02:22:12.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:22:12.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T02:22:12.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:22:12.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T02:22:12.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:22:12.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T02:22:12.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:22:12.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T02:22:12.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:22:12.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:22:12.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:22:12.046 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:22:11 smithi047 ceph-mon[77147]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T02:22:12.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T02:22:12.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:22:12.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T02:22:12.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:22:12.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:22:12.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:22:12.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T02:22:12.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:22:12.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T02:22:12.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:22:12.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T02:22:12.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:22:12.049 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T02:22:12.049 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:22:12.049 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T02:22:12.049 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:22:12.049 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:22:12.050 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:22:12.050 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T02:22:12.050 INFO:journalctl@ceph.mon.b.smithi047.stdout: During handling of the above exception, another exception occurred: 2022-05-14T02:22:12.050 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T02:22:12.050 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T02:22:12.051 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T02:22:12.051 INFO:journalctl@ceph.mon.b.smithi047.stdout: return f(*arg) 2022-05-14T02:22:12.051 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T02:22:12.051 INFO:journalctl@ceph.mon.b.smithi047.stdout: self._write_client_files(client_files, host) 2022-05-14T02:22:12.052 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T02:22:12.052 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T02:22:12.052 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T02:22:12.052 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T02:22:12.052 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T02:22:12.053 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.event_loop.get_result(coro) 2022-05-14T02:22:12.053 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T02:22:12.053 INFO:journalctl@ceph.mon.b.smithi047.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T02:22:12.053 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T02:22:12.054 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.__get_result() 2022-05-14T02:22:12.054 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T02:22:12.054 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise self._exception 2022-05-14T02:22:12.054 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T02:22:12.054 INFO:journalctl@ceph.mon.b.smithi047.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T02:22:12.055 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T02:22:12.055 INFO:journalctl@ceph.mon.b.smithi047.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T02:22:12.055 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T02:22:12.055 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise 2022-05-14T02:22:12.055 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T02:22:12.056 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.gen.throw(type, value, traceback) 2022-05-14T02:22:12.056 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T02:22:12.056 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OrchestratorError(msg) 2022-05-14T02:22:12.056 INFO:journalctl@ceph.mon.b.smithi047.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:22:12.057 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:22:11 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:22:12.057 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:22:11 smithi047 ceph-mon[77147]: pgmap v5584: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:22:12.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:22:12 smithi115 conmon[90393]: 2022-05-14T02:22:12.422+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:22:12.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:22:12 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:22:13.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:22:12 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:22:13.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:22:13 smithi115 conmon[90393]: 2022-05-14T02:22:13.457+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:22:13.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:22:13 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:22:13.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:22:13 smithi115 ceph-mon[78593]: pgmap v5585: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:22:14.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:22:13 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:22:14.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:22:13 smithi047 ceph-mon[77147]: pgmap v5585: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:22:14.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:22:14 smithi115 conmon[90393]: 2022-05-14T02:22:14.436+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:22:14.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:22:14 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:22:14.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:22:14 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:22:14.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:22:14 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:22:14.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:22:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T02:22:14.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:22:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T02:22:15.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:22:14 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:22:15.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:22:14 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:22:15.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:22:14 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:22:15.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:22:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T02:22:15.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:22:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T02:22:15.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:22:15 smithi115 conmon[90393]: 2022-05-14T02:22:15.472+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:22:15.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:22:15 smithi115 ceph-mon[78593]: pgmap v5586: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:22:15.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:22:15 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:22:15.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:22:15 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 10470 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:22:16.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:22:15 smithi047 ceph-mon[77147]: pgmap v5586: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:22:16.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:22:15 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:22:16.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:22:15 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 10470 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:22:16.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:22:16 smithi115 conmon[90393]: 2022-05-14T02:22:16.492+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:22:16.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:22:16 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:22:17.046 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:22:16 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:22:17.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:22:17 smithi115 conmon[90393]: 2022-05-14T02:22:17.536+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:22:17.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:22:17 smithi115 ceph-mon[78593]: pgmap v5587: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:22:17.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:22:17 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:22:18.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:22:17 smithi047 ceph-mon[77147]: pgmap v5587: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:22:18.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:22:17 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:22:18.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:22:18 smithi115 conmon[90393]: 2022-05-14T02:22:18.571+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:22:18.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:22:18 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:22:18.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:22:18 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:22:18.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:22:18 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:22:19.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:22:18 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:22:19.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:22:18 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:22:19.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:22:18 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:22:19.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:22:19 smithi115 conmon[90393]: 2022-05-14T02:22:19.593+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:22:19.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:22:19 smithi115 ceph-mon[78593]: pgmap v5588: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:22:19.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:22:19 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:22:20.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:22:19 smithi047 ceph-mon[77147]: pgmap v5588: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:22:20.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:22:19 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:22:20.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:22:20 smithi115 conmon[90393]: 2022-05-14T02:22:20.561+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:22:20.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:22:20 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:22:21.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:22:20 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:22:21.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:22:21 smithi115 conmon[90393]: 2022-05-14T02:22:21.597+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:22:21.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:22:21 smithi115 ceph-mon[78593]: pgmap v5589: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:22:21.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:22:21 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:22:21.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:22:21 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 10475 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:22:22.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:22:21 smithi047 ceph-mon[77147]: pgmap v5589: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:22:22.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:22:21 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:22:22.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:22:21 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 10475 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:22:22.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:22:22 smithi115 conmon[90393]: 2022-05-14T02:22:22.604+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:22:22.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:22:22 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:22:22.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:22:22 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:22:22.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:22:22 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:22:23.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:22:22 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:22:23.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:22:22 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:22:23.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:22:22 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:22:24.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:22:23 smithi047 ceph-mon[77147]: pgmap v5590: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:22:24.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:22:23 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:22:24.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:22:23 smithi115 conmon[90393]: 2022-05-14T02:22:23.643+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:22:24.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:22:23 smithi115 ceph-mon[78593]: pgmap v5590: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:22:24.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:22:23 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:22:25.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:22:24 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:22:25.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:22:24 smithi115 conmon[90393]: 2022-05-14T02:22:24.656+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:22:25.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:22:24 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:22:26.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:22:25 smithi047 ceph-mon[77147]: pgmap v5591: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:22:26.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:22:25 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:22:26.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:22:25 smithi115 conmon[90393]: 2022-05-14T02:22:25.625+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:22:26.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:22:25 smithi115 ceph-mon[78593]: pgmap v5591: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:22:26.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:22:25 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:22:27.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:22:26 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:22:27.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:22:26 smithi115 conmon[90393]: 2022-05-14T02:22:26.632+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:22:27.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:22:26 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:22:27.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:22:27 smithi115 conmon[90393]: 2022-05-14T02:22:27.597+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:22:27.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:22:27 smithi115 ceph-mon[78593]: pgmap v5592: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:22:27.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:22:27 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:22:28.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:22:27 smithi047 ceph-mon[77147]: pgmap v5592: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:22:28.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:22:27 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:22:28.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:22:28 smithi115 conmon[90393]: 2022-05-14T02:22:28.576+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:22:28.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:22:28 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:22:29.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:22:28 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:22:29.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:22:29 smithi115 conmon[90393]: 2022-05-14T02:22:29.527+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:22:29.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:22:29 smithi115 ceph-mon[78593]: pgmap v5593: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:22:29.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:22:29 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:22:30.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:22:29 smithi047 ceph-mon[77147]: pgmap v5593: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:22:30.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:22:29 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:22:30.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:22:30 smithi115 conmon[90393]: 2022-05-14T02:22:30.510+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:22:30.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:22:30 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:22:30.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:22:30 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:22:30.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:22:30 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:22:30.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:22:30 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 10480 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:22:31.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:22:30 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:22:31.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:22:30 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:22:31.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:22:30 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:22:31.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:22:30 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 10480 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:22:31.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:22:31 smithi115 conmon[90393]: 2022-05-14T02:22:31.543+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:22:31.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:22:31 smithi115 ceph-mon[78593]: pgmap v5594: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:22:31.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:22:31 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:22:32.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:22:31 smithi047 ceph-mon[77147]: pgmap v5594: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:22:32.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:22:31 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:22:32.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:22:32 smithi115 conmon[90393]: 2022-05-14T02:22:32.548+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:22:32.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:22:32 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:22:33.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:22:32 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:22:33.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:22:33 smithi115 conmon[90393]: 2022-05-14T02:22:33.539+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:22:33.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:22:33 smithi115 ceph-mon[78593]: pgmap v5595: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:22:33.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:22:33 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:22:34.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:22:33 smithi047 ceph-mon[77147]: pgmap v5595: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:22:34.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:22:33 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:22:34.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:22:34 smithi115 conmon[90393]: 2022-05-14T02:22:34.585+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:22:34.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:22:34 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:22:34.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:22:34 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:22:34.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:22:34 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:22:35.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:22:34 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:22:35.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:22:34 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:22:35.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:22:34 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:22:35.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:22:35 smithi115 conmon[90393]: 2022-05-14T02:22:35.582+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:22:35.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:22:35 smithi115 ceph-mon[78593]: pgmap v5596: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:22:35.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:22:35 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:22:35.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:22:35 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 10490 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:22:36.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:22:35 smithi047 ceph-mon[77147]: pgmap v5596: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:22:36.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:22:35 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:22:36.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:22:35 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 10490 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:22:36.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:22:36 smithi115 conmon[90393]: 2022-05-14T02:22:36.590+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:22:36.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:22:36 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:22:37.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:22:36 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:22:37.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:22:37 smithi115 conmon[90393]: 2022-05-14T02:22:37.541+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:22:37.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:22:37 smithi115 ceph-mon[78593]: pgmap v5597: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:22:37.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:22:37 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:22:38.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:22:37 smithi047 ceph-mon[77147]: pgmap v5597: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:22:38.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:22:37 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:22:38.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:22:38 smithi115 conmon[90393]: 2022-05-14T02:22:38.517+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:22:38.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:22:38 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:22:38.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:22:38 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:22:38.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:22:38 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:22:39.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:22:38 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:22:39.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:22:38 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:22:39.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:22:38 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:22:39.735 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:22:39 smithi115 conmon[90393]: 2022-05-14T02:22:39.477+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:22:40.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:22:39 smithi047 ceph-mon[77147]: pgmap v5598: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:22:40.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:22:39 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:22:40.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:22:39 smithi115 ceph-mon[78593]: pgmap v5598: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:22:40.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:22:39 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:22:40.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:22:40 smithi115 conmon[90393]: 2022-05-14T02:22:40.496+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:22:40.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:22:40 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:22:41.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:22:40 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:22:41.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:22:41 smithi115 conmon[90393]: 2022-05-14T02:22:41.492+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:22:41.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:22:41 smithi115 ceph-mon[78593]: pgmap v5599: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:22:41.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:22:41 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:22:41.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:22:41 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 10495 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:22:42.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:22:41 smithi047 ceph-mon[77147]: pgmap v5599: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:22:42.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:22:41 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:22:42.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:22:41 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 10495 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:22:42.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:22:42 smithi115 conmon[90393]: 2022-05-14T02:22:42.527+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:22:42.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:22:42 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:22:42.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:22:42 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:22:42.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:22:42 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:22:43.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:22:42 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:22:43.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:22:42 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:22:43.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:22:42 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:22:43.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:22:43 smithi115 conmon[90393]: 2022-05-14T02:22:43.547+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:22:43.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:22:43 smithi115 ceph-mon[78593]: pgmap v5600: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:22:43.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:22:43 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:22:44.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:22:43 smithi047 ceph-mon[77147]: pgmap v5600: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:22:44.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:22:43 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:22:44.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:22:44 smithi115 conmon[90393]: 2022-05-14T02:22:44.530+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:22:44.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:22:44 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:22:45.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:22:44 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:22:45.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:22:45 smithi115 conmon[90393]: 2022-05-14T02:22:45.535+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:22:45.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:22:45 smithi115 ceph-mon[78593]: pgmap v5601: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:22:45.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:22:45 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:22:46.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:22:45 smithi047 ceph-mon[77147]: pgmap v5601: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:22:46.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:22:45 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:22:46.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:22:46 smithi115 conmon[90393]: 2022-05-14T02:22:46.516+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:22:46.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:22:46 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:22:47.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:22:46 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:22:47.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:22:47 smithi115 conmon[90393]: 2022-05-14T02:22:47.543+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:22:47.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:22:47 smithi115 ceph-mon[78593]: pgmap v5602: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:22:47.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:22:47 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:22:48.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:22:47 smithi047 ceph-mon[77147]: pgmap v5602: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:22:48.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:22:47 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:22:48.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:22:48 smithi115 conmon[90393]: 2022-05-14T02:22:48.523+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:22:48.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:22:48 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:22:49.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:22:48 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:22:49.772 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:22:49 smithi115 conmon[90393]: 2022-05-14T02:22:49.508+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:22:50.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:22:49 smithi047 ceph-mon[77147]: pgmap v5603: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:22:50.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:22:49 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:22:50.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:22:49 smithi115 ceph-mon[78593]: pgmap v5603: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:22:50.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:22:49 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:22:50.782 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:22:50 smithi115 conmon[90393]: 2022-05-14T02:22:50.511+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:22:51.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:22:50 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:22:51.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:22:50 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:22:51.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:22:50 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:22:51.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:22:50 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 10500 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:22:51.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:22:50 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:22:51.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:22:50 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:22:51.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:22:50 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:22:51.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:22:50 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 10500 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:22:51.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:22:51 smithi115 conmon[90393]: 2022-05-14T02:22:51.542+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:22:51.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:22:51 smithi115 ceph-mon[78593]: pgmap v5604: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:22:51.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:22:51 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:22:52.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:22:51 smithi047 ceph-mon[77147]: pgmap v5604: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:22:52.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:22:51 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:22:52.788 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:22:52 smithi115 conmon[90393]: 2022-05-14T02:22:52.536+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:22:53.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:22:52 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:22:53.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:22:52 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:22:53.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:22:53 smithi115 conmon[90393]: 2022-05-14T02:22:53.564+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:22:53.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:22:53 smithi115 ceph-mon[78593]: pgmap v5605: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:22:53.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:22:53 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:22:54.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:22:53 smithi047 ceph-mon[77147]: pgmap v5605: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:22:54.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:22:53 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:22:54.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:22:54 smithi115 conmon[90393]: 2022-05-14T02:22:54.600+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:22:54.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:22:54 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:22:54.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:22:54 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:22:54.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:22:54 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:22:55.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:22:54 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:22:55.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:22:54 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:22:55.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:22:54 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:22:55.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:22:55 smithi115 conmon[90393]: 2022-05-14T02:22:55.592+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:22:55.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:22:55 smithi115 ceph-mon[78593]: pgmap v5606: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:22:55.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:22:55 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:22:55.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:22:55 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 10510 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:22:56.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:22:55 smithi047 ceph-mon[77147]: pgmap v5606: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:22:56.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:22:55 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:22:56.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:22:55 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 10510 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:22:57.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:22:56 smithi115 conmon[90393]: 2022-05-14T02:22:56.637+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:22:57.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:22:56 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:22:57.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:22:56 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:22:58.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:22:57 smithi115 conmon[90393]: 2022-05-14T02:22:57.628+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:22:58.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:22:57 smithi115 ceph-mon[78593]: pgmap v5607: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:22:58.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:22:57 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:22:58.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:22:57 smithi047 ceph-mon[77147]: pgmap v5607: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:22:58.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:22:57 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:22:59.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:22:58 smithi115 conmon[90393]: 2022-05-14T02:22:58.678+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:22:59.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:22:58 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:22:59.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:22:58 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:22:59.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:22:58 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:22:59.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:22:58 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:22:59.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:22:58 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:22:59.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:22:58 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:23:00.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:22:59 smithi115 conmon[90393]: 2022-05-14T02:22:59.637+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:23:00.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:22:59 smithi115 ceph-mon[78593]: pgmap v5608: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:23:00.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:22:59 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:23:00.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:22:59 smithi047 ceph-mon[77147]: pgmap v5608: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:23:00.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:22:59 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:23:01.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:23:00 smithi115 conmon[90393]: 2022-05-14T02:23:00.676+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:23:01.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:23:00 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:23:01.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:23:00 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:23:02.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:23:01 smithi115 conmon[90393]: 2022-05-14T02:23:01.660+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:23:02.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:23:01 smithi115 ceph-mon[78593]: pgmap v5609: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:23:02.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:23:01 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:23:02.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:23:01 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 10515 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:23:02.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:23:01 smithi047 ceph-mon[77147]: pgmap v5609: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:23:02.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:23:01 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:23:02.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:23:01 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 10515 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:23:03.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:23:02 smithi115 conmon[90393]: 2022-05-14T02:23:02.664+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:23:03.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:23:02 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:23:03.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:23:02 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:23:03.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:23:02 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:23:03.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:23:02 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:23:03.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:23:02 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:23:03.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:23:02 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:23:04.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:23:03 smithi115 conmon[90393]: 2022-05-14T02:23:03.710+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:23:04.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:23:03 smithi115 ceph-mon[78593]: pgmap v5610: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:23:04.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:23:03 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:23:04.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:23:03 smithi047 ceph-mon[77147]: pgmap v5610: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:23:04.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:23:03 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:23:05.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:23:04 smithi115 conmon[90393]: 2022-05-14T02:23:04.677+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:23:05.126 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:23:04 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:23:05.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:23:04 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:23:06.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:23:05 smithi115 conmon[90393]: 2022-05-14T02:23:05.648+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:23:06.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:23:05 smithi115 ceph-mon[78593]: pgmap v5611: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:23:06.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:23:05 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:23:06.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:23:05 smithi047 ceph-mon[77147]: pgmap v5611: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:23:06.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:23:05 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:23:07.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:23:06 smithi115 conmon[90393]: 2022-05-14T02:23:06.621+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:23:07.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:23:06 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:23:07.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:23:06 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:23:07.850 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:23:07 smithi115 conmon[90393]: 2022-05-14T02:23:07.578+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:23:08.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:23:07 smithi115 ceph-mon[78593]: pgmap v5612: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:23:08.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:23:07 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:23:08.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:23:07 smithi047 ceph-mon[77147]: pgmap v5612: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:23:08.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:23:07 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:23:08.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:23:08 smithi115 conmon[90393]: 2022-05-14T02:23:08.552+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:23:09.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:23:08 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:23:09.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:23:08 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:23:09.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:23:09 smithi115 conmon[90393]: 2022-05-14T02:23:09.554+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:23:10.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:23:09 smithi047 ceph-mon[77147]: pgmap v5613: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:23:10.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:23:09 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:23:10.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:23:09 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:23:10.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:23:09 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:23:10.338 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:23:09 smithi115 ceph-mon[78593]: pgmap v5613: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:23:10.338 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:23:09 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:23:10.339 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:23:09 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:23:10.339 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:23:09 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:23:10.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:23:10 smithi115 conmon[90393]: 2022-05-14T02:23:10.562+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:23:11.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:23:10 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:23:11.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:23:10 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T02:23:11.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:23:10 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T02:23:11.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:23:10 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T02:23:11.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:23:10 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 10520 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:23:11.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:23:10 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:23:11.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:23:10 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T02:23:11.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:23:10 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T02:23:11.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:23:10 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T02:23:11.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:23:10 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 10520 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:23:11.293 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:23:10 smithi047 conmon[78521]: 2022-05-14T02:23:10.895+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:23:11.293 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:23:10 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T02:23:11.293 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:23:10 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:23:11.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:23:10 smithi047 conmon[78521]: yield 2022-05-14T02:23:11.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:23:10 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:23:11.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:23:10 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:23:11.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:23:10 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:23:11.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:23:10 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T02:23:11.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:23:10 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:23:11.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:23:10 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T02:23:11.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:23:10 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:23:11.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:23:10 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T02:23:11.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:23:10 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:23:11.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:23:10 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T02:23:11.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:23:10 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:23:11.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:23:10 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T02:23:11.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:23:10 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:23:11.297 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:23:10 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:23:11.297 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:23:10 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:23:11.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:23:11 smithi115 conmon[90393]: 2022-05-14T02:23:11.573+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:23:12.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:23:11 smithi047 ceph-mon[77147]: pgmap v5614: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:23:12.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:23:11 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:23:12.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:23:11 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:23:12.292 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T02:23:12.292 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:23:12.298 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T02:23:12.298 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:23:12.298 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:23:12.299 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:23:12.299 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T02:23:12.299 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:23:12.299 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T02:23:12.300 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:23:12.300 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T02:23:12.300 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:23:12.300 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T02:23:12.301 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:23:12.301 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T02:23:12.301 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:23:12.301 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:23:12.301 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:23:12.302 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:23:11 smithi047 ceph-mon[77147]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T02:23:12.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:23:11 smithi115 ceph-mon[78593]: pgmap v5614: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:23:12.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:23:11 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:23:12.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:23:11 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:23:12.359 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T02:23:12.359 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:23:12.359 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T02:23:12.359 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:23:12.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:23:12.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:23:12.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T02:23:12.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:23:12.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T02:23:12.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:23:12.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T02:23:12.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:23:12.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T02:23:12.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:23:12.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T02:23:12.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:23:12.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:23:12.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:23:12.363 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:23:11 smithi115 ceph-mon[78593]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T02:23:12.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:23:12 smithi115 conmon[90393]: 2022-05-14T02:23:12.572+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:23:13.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:23:12 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:23:13.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:23:12 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:23:13.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:23:13 smithi115 conmon[90393]: 2022-05-14T02:23:13.594+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:23:14.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:23:13 smithi047 ceph-mon[77147]: pgmap v5615: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:23:14.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:23:13 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:23:14.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:23:13 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:23:14.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:23:13 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:23:14.293 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:23:13 smithi047 conmon[78521]: 2022-05-14T02:23:13.967+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:23:14.293 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:23:13 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T02:23:14.293 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:23:13 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:23:14.293 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:23:13 smithi047 conmon[78521]: yield 2022-05-14T02:23:14.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:23:13 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:23:14.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:23:13 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:23:14.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:23:13 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:23:14.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:23:13 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T02:23:14.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:23:13 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:23:14.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:23:13 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T02:23:14.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:23:13 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:23:14.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:23:13 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T02:23:14.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:23:13 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:23:14.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:23:13 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T02:23:14.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:23:13 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:23:14.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:23:13 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T02:23:14.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:23:13 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:23:14.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:23:13 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:23:14.297 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:23:13 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:23:14.297 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:23:13 smithi047 conmon[78521]: 2022-05-14T02:23:13.968+0000 7f5345f84700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T02:23:14.297 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:23:13 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T02:23:14.297 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:23:13 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:23:14.297 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:23:13 smithi047 conmon[78521]: yield 2022-05-14T02:23:14.298 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:23:13 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:23:14.298 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:23:13 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:23:14.298 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:23:13 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:23:14.298 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:23:13 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T02:23:14.299 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:23:13 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:23:14.299 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:23:13 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T02:23:14.299 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:23:13 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:23:14.299 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:23:13 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T02:23:14.299 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:23:13 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:23:14.300 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:23:13 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T02:23:14.300 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:23:13 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:23:14.300 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:23:13 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T02:23:14.300 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:23:13 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:23:14.300 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:23:13 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:23:14.301 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:23:13 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:23:14.301 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:23:13 smithi047 conmon[78521]: 2022-05-14T02:23:14.301 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:23:13 smithi047 conmon[78521]: During handling of the above exception, another exception occurred: 2022-05-14T02:23:14.301 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:23:13 smithi047 conmon[78521]: 2022-05-14T02:23:14.301 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:23:13 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T02:23:14.302 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:23:13 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T02:23:14.302 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:23:13 smithi047 conmon[78521]: return f(*arg) 2022-05-14T02:23:14.302 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:23:13 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T02:23:14.302 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:23:13 smithi047 conmon[78521]: self._write_client_files(client_files, host) 2022-05-14T02:23:14.302 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:23:13 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T02:23:14.303 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:23:13 smithi047 conmon[78521]: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T02:23:14.303 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:23:13 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T02:23:14.303 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:23:13 smithi047 conmon[78521]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T02:23:14.303 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:23:13 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T02:23:14.303 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:23:13 smithi047 conmon[78521]: return self.event_loop.get_result(coro) 2022-05-14T02:23:14.304 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:23:13 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T02:23:14.304 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:23:13 smithi047 conmon[78521]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T02:23:14.304 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:23:13 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T02:23:14.304 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:23:13 smithi047 conmon[78521]: return self.__get_result() 2022-05-14T02:23:14.304 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:23:13 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T02:23:14.305 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:23:13 smithi047 conmon[78521]: raise self._exception 2022-05-14T02:23:14.305 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:23:13 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T02:23:14.305 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:23:13 smithi047 conmon[78521]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T02:23:14.305 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:23:13 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T02:23:14.306 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:23:13 smithi047 conmon[78521]: conn = await self._remote_connection(host, addr) 2022-05-14T02:23:14.306 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:23:13 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T02:23:14.306 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:23:13 smithi047 conmon[78521]: raise 2022-05-14T02:23:14.306 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:23:13 smithi047 conmon[78521]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T02:23:14.306 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:23:13 smithi047 conmon[78521]: self.gen.throw(type, value, traceback) 2022-05-14T02:23:14.307 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:23:13 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T02:23:14.307 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:23:13 smithi047 conmon[78521]: raise OrchestratorError(msg) 2022-05-14T02:23:14.307 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:23:13 smithi047 conmon[78521]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:23:14.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:23:13 smithi115 ceph-mon[78593]: pgmap v5615: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:23:14.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:23:13 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:23:14.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:23:13 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:23:14.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:23:13 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:23:14.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:23:14 smithi115 conmon[90393]: 2022-05-14T02:23:14.573+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:23:15.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:23:14 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:23:15.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:23:14 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:23:15.292 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T02:23:15.292 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:23:15.292 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T02:23:15.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:23:15.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:23:15.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:23:15.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T02:23:15.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:23:15.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T02:23:15.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:23:15.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T02:23:15.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:23:15.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T02:23:15.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:23:15.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T02:23:15.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:23:15.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:23:15.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:23:15.296 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:23:14 smithi047 ceph-mon[77147]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T02:23:15.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T02:23:15.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:23:15.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T02:23:15.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:23:15.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:23:15.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:23:15.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T02:23:15.298 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:23:15.298 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T02:23:15.298 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:23:15.298 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T02:23:15.298 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:23:15.299 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T02:23:15.299 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:23:15.299 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T02:23:15.299 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:23:15.299 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:23:15.300 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:23:15.300 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T02:23:15.300 INFO:journalctl@ceph.mon.b.smithi047.stdout: During handling of the above exception, another exception occurred: 2022-05-14T02:23:15.300 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T02:23:15.300 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T02:23:15.301 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T02:23:15.301 INFO:journalctl@ceph.mon.b.smithi047.stdout: return f(*arg) 2022-05-14T02:23:15.301 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T02:23:15.301 INFO:journalctl@ceph.mon.b.smithi047.stdout: self._write_client_files(client_files, host) 2022-05-14T02:23:15.301 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T02:23:15.302 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T02:23:15.302 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T02:23:15.302 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T02:23:15.302 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T02:23:15.303 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.event_loop.get_result(coro) 2022-05-14T02:23:15.303 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T02:23:15.303 INFO:journalctl@ceph.mon.b.smithi047.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T02:23:15.303 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T02:23:15.303 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.__get_result() 2022-05-14T02:23:15.304 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T02:23:15.304 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise self._exception 2022-05-14T02:23:15.304 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T02:23:15.304 INFO:journalctl@ceph.mon.b.smithi047.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T02:23:15.304 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T02:23:15.305 INFO:journalctl@ceph.mon.b.smithi047.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T02:23:15.305 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T02:23:15.305 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise 2022-05-14T02:23:15.305 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T02:23:15.306 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.gen.throw(type, value, traceback) 2022-05-14T02:23:15.306 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T02:23:15.306 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OrchestratorError(msg) 2022-05-14T02:23:15.306 INFO:journalctl@ceph.mon.b.smithi047.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:23:15.306 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:23:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T02:23:15.307 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:23:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T02:23:15.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:23:14 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:23:15.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:23:14 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:23:15.359 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T02:23:15.359 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:23:15.359 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T02:23:15.359 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:23:15.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:23:15.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:23:15.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T02:23:15.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:23:15.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T02:23:15.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:23:15.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T02:23:15.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:23:15.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T02:23:15.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:23:15.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T02:23:15.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:23:15.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:23:15.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:23:15.363 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:23:14 smithi115 ceph-mon[78593]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T02:23:15.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T02:23:15.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:23:15.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T02:23:15.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:23:15.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:23:15.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:23:15.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T02:23:15.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:23:15.365 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T02:23:15.365 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:23:15.365 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T02:23:15.365 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:23:15.366 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T02:23:15.366 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:23:15.366 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T02:23:15.366 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:23:15.366 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:23:15.367 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:23:15.367 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T02:23:15.367 INFO:journalctl@ceph.mon.c.smithi115.stdout: During handling of the above exception, another exception occurred: 2022-05-14T02:23:15.367 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T02:23:15.367 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T02:23:15.368 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T02:23:15.368 INFO:journalctl@ceph.mon.c.smithi115.stdout: return f(*arg) 2022-05-14T02:23:15.368 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T02:23:15.368 INFO:journalctl@ceph.mon.c.smithi115.stdout: self._write_client_files(client_files, host) 2022-05-14T02:23:15.369 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T02:23:15.369 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T02:23:15.369 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T02:23:15.369 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T02:23:15.370 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T02:23:15.370 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.event_loop.get_result(coro) 2022-05-14T02:23:15.370 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T02:23:15.370 INFO:journalctl@ceph.mon.c.smithi115.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T02:23:15.370 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T02:23:15.371 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.__get_result() 2022-05-14T02:23:15.371 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T02:23:15.371 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise self._exception 2022-05-14T02:23:15.371 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T02:23:15.371 INFO:journalctl@ceph.mon.c.smithi115.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T02:23:15.372 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T02:23:15.372 INFO:journalctl@ceph.mon.c.smithi115.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T02:23:15.372 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T02:23:15.372 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise 2022-05-14T02:23:15.373 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T02:23:15.373 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.gen.throw(type, value, traceback) 2022-05-14T02:23:15.373 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T02:23:15.373 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OrchestratorError(msg) 2022-05-14T02:23:15.373 INFO:journalctl@ceph.mon.c.smithi115.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:23:15.374 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:23:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T02:23:15.374 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:23:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T02:23:15.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:23:15 smithi115 conmon[90393]: 2022-05-14T02:23:15.534+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:23:16.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:23:15 smithi047 ceph-mon[77147]: pgmap v5616: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:23:16.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:23:15 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:23:16.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:23:15 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 10530 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:23:16.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:23:15 smithi115 ceph-mon[78593]: pgmap v5616: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:23:16.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:23:15 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:23:16.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:23:15 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 10530 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:23:16.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:23:16 smithi115 conmon[90393]: 2022-05-14T02:23:16.566+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:23:17.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:23:16 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:23:17.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:23:16 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:23:17.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:23:17 smithi115 conmon[90393]: 2022-05-14T02:23:17.577+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:23:18.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:23:17 smithi047 ceph-mon[77147]: pgmap v5617: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:23:18.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:23:17 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:23:18.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:23:17 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:23:18.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:23:17 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:23:18.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:23:17 smithi115 ceph-mon[78593]: pgmap v5617: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:23:18.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:23:17 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:23:18.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:23:17 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:23:18.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:23:17 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:23:18.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:23:18 smithi115 conmon[90393]: 2022-05-14T02:23:18.550+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:23:19.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:23:18 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:23:19.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:23:18 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:23:19.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:23:19 smithi115 conmon[90393]: 2022-05-14T02:23:19.506+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:23:20.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:23:19 smithi047 ceph-mon[77147]: pgmap v5618: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:23:20.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:23:19 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:23:20.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:23:19 smithi115 ceph-mon[78593]: pgmap v5618: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:23:20.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:23:19 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:23:20.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:23:20 smithi115 conmon[90393]: 2022-05-14T02:23:20.550+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:23:21.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:23:20 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:23:21.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:23:20 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:23:21.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:23:21 smithi115 conmon[90393]: 2022-05-14T02:23:21.523+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:23:22.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:23:21 smithi047 ceph-mon[77147]: pgmap v5619: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:23:22.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:23:21 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:23:22.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:23:21 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 10535 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:23:22.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:23:21 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:23:22.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:23:21 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:23:22.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:23:21 smithi115 ceph-mon[78593]: pgmap v5619: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:23:22.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:23:21 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:23:22.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:23:21 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 10535 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:23:22.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:23:21 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:23:22.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:23:21 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:23:22.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:23:22 smithi115 conmon[90393]: 2022-05-14T02:23:22.551+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:23:23.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:23:22 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:23:23.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:23:22 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:23:23.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:23:23 smithi115 conmon[90393]: 2022-05-14T02:23:23.558+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:23:24.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:23:23 smithi047 ceph-mon[77147]: pgmap v5620: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:23:24.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:23:23 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:23:24.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:23:23 smithi115 ceph-mon[78593]: pgmap v5620: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:23:24.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:23:23 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:23:24.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:23:24 smithi115 conmon[90393]: 2022-05-14T02:23:24.587+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:23:25.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:23:24 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:23:25.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:23:24 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:23:25.912 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:23:25 smithi115 conmon[90393]: 2022-05-14T02:23:25.628+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:23:26.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:23:25 smithi047 ceph-mon[77147]: pgmap v5621: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:23:26.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:23:25 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:23:26.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:23:25 smithi115 ceph-mon[78593]: pgmap v5621: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:23:26.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:23:25 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:23:27.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:23:26 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:23:27.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:23:26 smithi115 conmon[90393]: 2022-05-14T02:23:26.671+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:23:27.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:23:26 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:23:28.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:23:27 smithi115 conmon[90393]: 2022-05-14T02:23:27.684+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:23:28.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:23:27 smithi115 ceph-mon[78593]: pgmap v5622: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:23:28.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:23:27 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:23:28.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:23:27 smithi047 ceph-mon[77147]: pgmap v5622: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:23:28.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:23:27 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:23:28.924 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:23:28 smithi115 conmon[90393]: 2022-05-14T02:23:28.652+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:23:29.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:23:28 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:23:29.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:23:28 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:23:30.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:23:29 smithi115 conmon[90393]: 2022-05-14T02:23:29.681+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:23:30.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:23:29 smithi115 ceph-mon[78593]: pgmap v5623: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:23:30.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:23:29 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:23:30.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:23:29 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:23:30.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:23:29 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:23:30.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:23:29 smithi047 ceph-mon[77147]: pgmap v5623: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:23:30.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:23:29 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:23:30.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:23:29 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:23:30.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:23:29 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:23:31.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:23:30 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:23:31.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:23:30 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 10540 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:23:31.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:23:30 smithi115 conmon[90393]: 2022-05-14T02:23:30.689+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:23:31.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:23:30 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:23:31.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:23:30 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 10540 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:23:32.113 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:23:31 smithi115 ceph-mon[78593]: pgmap v5624: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:23:32.114 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:23:31 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:23:32.114 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:23:31 smithi115 conmon[90393]: 2022-05-14T02:23:31.704+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:23:32.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:23:31 smithi047 ceph-mon[77147]: pgmap v5624: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:23:32.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:23:31 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:23:33.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:23:32 smithi115 conmon[90393]: 2022-05-14T02:23:32.747+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:23:33.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:23:32 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:23:33.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:23:32 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:23:34.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:23:33 smithi115 conmon[90393]: 2022-05-14T02:23:33.727+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:23:34.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:23:33 smithi115 ceph-mon[78593]: pgmap v5625: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:23:34.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:23:33 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:23:34.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:23:33 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:23:34.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:23:33 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:23:34.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:23:33 smithi047 ceph-mon[77147]: pgmap v5625: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:23:34.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:23:33 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:23:34.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:23:33 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:23:34.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:23:33 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:23:35.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:23:34 smithi115 conmon[90393]: 2022-05-14T02:23:34.719+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:23:35.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:23:34 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:23:35.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:23:34 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:23:36.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:23:35 smithi115 conmon[90393]: 2022-05-14T02:23:35.751+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:23:36.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:23:35 smithi115 ceph-mon[78593]: pgmap v5626: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:23:36.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:23:35 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:23:36.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:23:35 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 10550 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:23:36.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:23:35 smithi047 ceph-mon[77147]: pgmap v5626: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:23:36.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:23:35 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:23:36.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:23:35 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 10550 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:23:37.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:23:36 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:23:37.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:23:36 smithi115 conmon[90393]: 2022-05-14T02:23:36.728+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:23:37.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:23:36 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:23:37.952 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:23:37 smithi115 conmon[90393]: 2022-05-14T02:23:37.697+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:23:38.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:23:37 smithi047 ceph-mon[77147]: pgmap v5627: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:23:38.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:23:37 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:23:38.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:23:37 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:23:38.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:23:37 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:23:38.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:23:37 smithi115 ceph-mon[78593]: pgmap v5627: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:23:38.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:23:37 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:23:38.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:23:37 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:23:38.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:23:37 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:23:38.958 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:23:38 smithi115 conmon[90393]: 2022-05-14T02:23:38.658+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:23:39.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:23:38 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:23:39.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:23:38 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:23:39.958 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:23:39 smithi115 conmon[90393]: 2022-05-14T02:23:39.625+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:23:40.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:23:39 smithi047 ceph-mon[77147]: pgmap v5628: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:23:40.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:23:39 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:23:40.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:23:39 smithi115 ceph-mon[78593]: pgmap v5628: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:23:40.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:23:39 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:23:40.963 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:23:40 smithi115 conmon[90393]: 2022-05-14T02:23:40.639+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:23:41.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:23:40 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:23:41.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:23:40 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:23:41.964 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:23:41 smithi115 conmon[90393]: 2022-05-14T02:23:41.643+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:23:42.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:23:41 smithi047 ceph-mon[77147]: pgmap v5629: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:23:42.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:23:41 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:23:42.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:23:41 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 10555 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:23:42.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:23:41 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:23:42.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:23:41 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:23:42.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:23:41 smithi115 ceph-mon[78593]: pgmap v5629: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:23:42.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:23:41 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:23:42.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:23:41 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 10555 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:23:42.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:23:41 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:23:42.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:23:41 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:23:42.970 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:23:42 smithi115 conmon[90393]: 2022-05-14T02:23:42.622+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:23:43.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:23:42 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:23:43.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:23:42 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:23:43.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:23:43 smithi115 conmon[90393]: 2022-05-14T02:23:43.606+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:23:44.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:23:43 smithi047 ceph-mon[77147]: pgmap v5630: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:23:44.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:23:43 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:23:44.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:23:43 smithi115 ceph-mon[78593]: pgmap v5630: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:23:44.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:23:43 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:23:44.976 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:23:44 smithi115 conmon[90393]: 2022-05-14T02:23:44.627+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:23:45.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:23:44 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:23:45.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:23:44 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:23:45.978 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:23:45 smithi115 conmon[90393]: 2022-05-14T02:23:45.628+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:23:46.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:23:45 smithi047 ceph-mon[77147]: pgmap v5631: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:23:46.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:23:45 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:23:46.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:23:45 smithi115 ceph-mon[78593]: pgmap v5631: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:23:46.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:23:45 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:23:46.982 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:23:46 smithi115 conmon[90393]: 2022-05-14T02:23:46.646+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:23:47.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:23:46 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:23:47.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:23:46 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:23:47.984 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:23:47 smithi115 conmon[90393]: 2022-05-14T02:23:47.684+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:23:48.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:23:47 smithi047 ceph-mon[77147]: pgmap v5632: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:23:48.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:23:47 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:23:48.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:23:47 smithi115 ceph-mon[78593]: pgmap v5632: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:23:48.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:23:47 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:23:48.989 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:23:48 smithi115 conmon[90393]: 2022-05-14T02:23:48.660+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:23:49.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:23:48 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:23:49.991 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:23:49 smithi115 conmon[90393]: 2022-05-14T02:23:49.630+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:23:49.991 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:23:48 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:23:50.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:23:49 smithi047 ceph-mon[77147]: pgmap v5633: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:23:50.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:23:49 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:23:50.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:23:49 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:23:50.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:23:49 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:23:50.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:23:49 smithi115 ceph-mon[78593]: pgmap v5633: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:23:50.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:23:49 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:23:50.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:23:49 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:23:50.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:23:49 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:23:50.995 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:23:50 smithi115 conmon[90393]: 2022-05-14T02:23:50.617+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:23:51.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:23:50 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:23:51.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:23:50 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 10560 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:23:51.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:23:50 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:23:51.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:23:50 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 10560 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:23:51.997 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:23:51 smithi115 conmon[90393]: 2022-05-14T02:23:51.632+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:23:52.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:23:51 smithi047 ceph-mon[77147]: pgmap v5634: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:23:52.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:23:51 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:23:52.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:23:51 smithi115 ceph-mon[78593]: pgmap v5634: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:23:52.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:23:51 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:23:52.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:23:52 smithi115 conmon[90393]: 2022-05-14T02:23:52.595+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:23:53.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:23:53 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:23:53.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:23:53 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:23:53.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:23:53 smithi115 conmon[90393]: 2022-05-14T02:23:53.568+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:23:54.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:23:54 smithi047 ceph-mon[77147]: pgmap v5635: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:23:54.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:23:54 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:23:54.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:23:54 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:23:54.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:23:54 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:23:54.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:23:54 smithi115 ceph-mon[78593]: pgmap v5635: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:23:54.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:23:54 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:23:54.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:23:54 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:23:54.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:23:54 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:23:54.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:23:54 smithi115 conmon[90393]: 2022-05-14T02:23:54.591+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:23:55.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:23:55 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:23:55.364 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:23:55 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:23:56.010 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:23:55 smithi115 conmon[90393]: 2022-05-14T02:23:55.632+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:23:56.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:23:56 smithi047 ceph-mon[77147]: pgmap v5636: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:23:56.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:23:56 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:23:56.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:23:56 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 10570 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:23:56.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:23:56 smithi115 ceph-mon[78593]: pgmap v5636: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:23:56.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:23:56 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:23:56.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:23:56 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 10570 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:23:57.016 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:23:56 smithi115 conmon[90393]: 2022-05-14T02:23:56.663+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:23:57.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:23:57 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:23:57.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:23:57 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:23:58.016 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:23:57 smithi115 conmon[90393]: 2022-05-14T02:23:57.623+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:23:58.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:23:58 smithi047 ceph-mon[77147]: pgmap v5637: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:23:58.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:23:58 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:23:58.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:23:58 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:23:58.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:23:58 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:23:58.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:23:58 smithi115 ceph-mon[78593]: pgmap v5637: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:23:58.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:23:58 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:23:58.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:23:58 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:23:58.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:23:58 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:23:58.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:23:58 smithi115 conmon[90393]: 2022-05-14T02:23:58.580+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:23:59.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:23:59 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:23:59.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:23:59 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:23:59.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:23:59 smithi115 conmon[90393]: 2022-05-14T02:23:59.595+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:24:00.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:24:00 smithi047 ceph-mon[77147]: pgmap v5638: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:24:00.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:24:00 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:24:00.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:24:00 smithi115 ceph-mon[78593]: pgmap v5638: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:24:00.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:24:00 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:24:00.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:24:00 smithi115 conmon[90393]: 2022-05-14T02:24:00.584+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:24:01.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:24:01 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:24:01.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:24:01 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:24:02.028 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:24:01 smithi115 conmon[90393]: 2022-05-14T02:24:01.629+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:24:02.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:24:02 smithi047 ceph-mon[77147]: pgmap v5639: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:24:02.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:24:02 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:24:02.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:24:02 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 10575 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:24:02.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:24:02 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:24:02.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:24:02 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:24:02.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:24:02 smithi115 ceph-mon[78593]: pgmap v5639: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:24:02.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:24:02 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:24:02.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:24:02 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 10575 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:24:02.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:24:02 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:24:02.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:24:02 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:24:02.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:24:02 smithi115 conmon[90393]: 2022-05-14T02:24:02.594+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:24:03.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:24:03 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:24:03.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:24:03 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:24:03.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:24:03 smithi115 conmon[90393]: 2022-05-14T02:24:03.574+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:24:04.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:24:04 smithi047 ceph-mon[77147]: pgmap v5640: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:24:04.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:24:04 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:24:04.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:24:04 smithi115 ceph-mon[78593]: pgmap v5640: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:24:04.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:24:04 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:24:04.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:24:04 smithi115 conmon[90393]: 2022-05-14T02:24:04.569+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:24:05.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:24:05 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:24:05.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:24:05 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:24:05.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:24:05 smithi115 conmon[90393]: 2022-05-14T02:24:05.586+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:24:06.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:24:06 smithi047 ceph-mon[77147]: pgmap v5641: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:24:06.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:24:06 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:24:06.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:24:06 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:24:06.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:24:06 smithi115 ceph-mon[78593]: pgmap v5641: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:24:06.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:24:06 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:24:06.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:24:06 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:24:06.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:24:06 smithi115 conmon[90393]: 2022-05-14T02:24:06.559+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:24:07.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:24:07 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:24:07.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:24:07 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:24:07.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:24:07 smithi115 conmon[90393]: 2022-05-14T02:24:07.523+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:24:08.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:24:08 smithi115 ceph-mon[78593]: pgmap v5642: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:24:08.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:24:08 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:24:08.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:24:08 smithi047 ceph-mon[77147]: pgmap v5642: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:24:08.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:24:08 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:24:08.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:24:08 smithi115 conmon[90393]: 2022-05-14T02:24:08.533+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:24:09.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:24:09 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:24:09.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:24:09 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:24:09.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:24:09 smithi115 conmon[90393]: 2022-05-14T02:24:09.539+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:24:10.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:24:10 smithi115 ceph-mon[78593]: pgmap v5643: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:24:10.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:24:10 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:24:10.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:24:10 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:24:10.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:24:10 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:24:10.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:24:10 smithi047 ceph-mon[77147]: pgmap v5643: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:24:10.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:24:10 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:24:10.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:24:10 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:24:10.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:24:10 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:24:10.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:24:10 smithi115 conmon[90393]: 2022-05-14T02:24:10.553+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:24:11.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:24:11 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:24:11.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:24:11 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 10580 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:24:11.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:24:11 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:24:11.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:24:11 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 10580 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:24:11.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:24:11 smithi115 conmon[90393]: 2022-05-14T02:24:11.556+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:24:12.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:24:12 smithi115 ceph-mon[78593]: pgmap v5644: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:24:12.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:24:12 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:24:12.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:24:12 smithi047 ceph-mon[77147]: pgmap v5644: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:24:12.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:24:12 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:24:12.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:24:12 smithi115 conmon[90393]: 2022-05-14T02:24:12.585+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:24:13.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:24:13 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:24:13.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:24:13 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:24:13.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:24:13 smithi115 conmon[90393]: 2022-05-14T02:24:13.602+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:24:14.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:24:14 smithi047 ceph-mon[77147]: pgmap v5645: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:24:14.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:24:14 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:24:14.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:24:14 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:24:14.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:24:14 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:24:14.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:24:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T02:24:14.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:24:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T02:24:14.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:24:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T02:24:14.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:24:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T02:24:14.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:24:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T02:24:14.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:24:14 smithi115 ceph-mon[78593]: pgmap v5645: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:24:14.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:24:14 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:24:14.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:24:14 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:24:14.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:24:14 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:24:14.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:24:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T02:24:14.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:24:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T02:24:14.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:24:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T02:24:14.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:24:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T02:24:14.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:24:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T02:24:14.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:24:14 smithi115 conmon[90393]: 2022-05-14T02:24:14.637+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:24:15.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:24:15 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:24:15.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:24:15 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:24:15.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:24:15 smithi115 conmon[90393]: 2022-05-14T02:24:15.602+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:24:16.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:24:16 smithi115 ceph-mon[78593]: pgmap v5646: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:24:16.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:24:16 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:24:16.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:24:16 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 10590 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:24:16.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:24:16 smithi047 ceph-mon[77147]: pgmap v5646: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:24:16.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:24:16 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:24:16.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:24:16 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 10590 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:24:17.081 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:24:16 smithi115 conmon[90393]: 2022-05-14T02:24:16.628+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:24:17.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:24:17 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:24:17.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:24:17 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:24:17.542 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:24:17 smithi047 conmon[78521]: 2022-05-14T02:24:17.072+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:24:17.542 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:24:17 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T02:24:17.542 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:24:17 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:24:17.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:24:17 smithi047 conmon[78521]: yield 2022-05-14T02:24:17.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:24:17 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:24:17.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:24:17 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:24:17.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:24:17 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:24:17.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:24:17 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T02:24:17.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:24:17 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:24:17.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:24:17 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T02:24:17.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:24:17 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:24:17.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:24:17 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T02:24:17.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:24:17 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:24:17.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:24:17 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T02:24:17.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:24:17 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:24:17.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:24:17 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T02:24:17.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:24:17 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:24:17.547 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:24:17 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:24:17.547 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:24:17 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:24:18.080 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:24:17 smithi115 conmon[90393]: 2022-05-14T02:24:17.642+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:24:18.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:24:18 smithi115 ceph-mon[78593]: pgmap v5647: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:24:18.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:24:18 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:24:18.358 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T02:24:18.359 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:24:18.359 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T02:24:18.359 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:24:18.359 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:24:18.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:24:18.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T02:24:18.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:24:18.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T02:24:18.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:24:18.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T02:24:18.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:24:18.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T02:24:18.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:24:18.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T02:24:18.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:24:18.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:24:18.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:24:18.363 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:24:18 smithi115 ceph-mon[78593]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T02:24:18.363 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:24:18 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:24:18.363 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:24:18 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:24:18.363 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:24:18 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:24:18.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:24:18 smithi047 ceph-mon[77147]: pgmap v5647: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:24:18.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:24:18 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:24:18.542 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T02:24:18.542 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:24:18.542 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T02:24:18.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:24:18.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:24:18.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:24:18.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T02:24:18.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:24:18.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T02:24:18.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:24:18.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T02:24:18.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:24:18.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T02:24:18.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:24:18.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T02:24:18.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:24:18.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:24:18.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:24:18.546 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:24:18 smithi047 ceph-mon[77147]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T02:24:18.546 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:24:18 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:24:18.546 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:24:18 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:24:18.546 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:24:18 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:24:19.085 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:24:18 smithi115 conmon[90393]: 2022-05-14T02:24:18.627+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:24:19.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:24:19 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:24:19.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:24:19 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:24:19.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:24:19 smithi115 conmon[90393]: 2022-05-14T02:24:19.586+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:24:20.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:24:20 smithi115 ceph-mon[78593]: pgmap v5648: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:24:20.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:24:20 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:24:20.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:24:20 smithi047 ceph-mon[77147]: pgmap v5648: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:24:20.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:24:20 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:24:20.542 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:24:20 smithi047 conmon[78521]: 2022-05-14T02:24:20.144+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:24:20.542 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:24:20 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T02:24:20.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:24:20 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:24:20.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:24:20 smithi047 conmon[78521]: yield 2022-05-14T02:24:20.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:24:20 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:24:20.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:24:20 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:24:20.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:24:20 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:24:20.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:24:20 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T02:24:20.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:24:20 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:24:20.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:24:20 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T02:24:20.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:24:20 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:24:20.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:24:20 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T02:24:20.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:24:20 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:24:20.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:24:20 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T02:24:20.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:24:20 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:24:20.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:24:20 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T02:24:20.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:24:20 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:24:20.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:24:20 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:24:20.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:24:20 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:24:20.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:24:20 smithi047 conmon[78521]: 2022-05-14T02:24:20.145+0000 7f5340779700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T02:24:20.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:24:20 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T02:24:20.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:24:20 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:24:20.547 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:24:20 smithi047 conmon[78521]: yield 2022-05-14T02:24:20.547 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:24:20 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:24:20.547 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:24:20 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:24:20.547 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:24:20 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:24:20.547 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:24:20 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T02:24:20.548 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:24:20 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:24:20.548 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:24:20 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T02:24:20.548 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:24:20 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:24:20.548 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:24:20 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T02:24:20.548 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:24:20 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:24:20.549 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:24:20 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T02:24:20.549 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:24:20 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:24:20.549 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:24:20 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T02:24:20.549 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:24:20 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:24:20.550 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:24:20 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:24:20.550 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:24:20 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:24:20.550 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:24:20 smithi047 conmon[78521]: 2022-05-14T02:24:20.550 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:24:20 smithi047 conmon[78521]: During handling of the above exception, another exception occurred: 2022-05-14T02:24:20.550 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:24:20 smithi047 conmon[78521]: 2022-05-14T02:24:20.551 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:24:20 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T02:24:20.551 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:24:20 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T02:24:20.551 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:24:20 smithi047 conmon[78521]: return f(*arg) 2022-05-14T02:24:20.551 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:24:20 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T02:24:20.551 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:24:20 smithi047 conmon[78521]: self._write_client_files(client_files, host) 2022-05-14T02:24:20.552 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:24:20 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T02:24:20.552 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:24:20 smithi047 conmon[78521]: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T02:24:20.552 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:24:20 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T02:24:20.552 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:24:20 smithi047 conmon[78521]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T02:24:20.552 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:24:20 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T02:24:20.553 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:24:20 smithi047 conmon[78521]: return self.event_loop.get_result(coro) 2022-05-14T02:24:20.553 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:24:20 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T02:24:20.553 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:24:20 smithi047 conmon[78521]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T02:24:20.553 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:24:20 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T02:24:20.554 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:24:20 smithi047 conmon[78521]: return self.__get_result() 2022-05-14T02:24:20.554 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:24:20 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T02:24:20.554 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:24:20 smithi047 conmon[78521]: raise self._exception 2022-05-14T02:24:20.554 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:24:20 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T02:24:20.554 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:24:20 smithi047 conmon[78521]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T02:24:20.555 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:24:20 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T02:24:20.555 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:24:20 smithi047 conmon[78521]: conn = await self._remote_connection(host, addr) 2022-05-14T02:24:20.555 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:24:20 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T02:24:20.555 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:24:20 smithi047 conmon[78521]: raise 2022-05-14T02:24:20.556 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:24:20 smithi047 conmon[78521]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T02:24:20.556 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:24:20 smithi047 conmon[78521]: self.gen.throw(type, value, traceback) 2022-05-14T02:24:20.556 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:24:20 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T02:24:20.556 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:24:20 smithi047 conmon[78521]: raise OrchestratorError(msg) 2022-05-14T02:24:20.556 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:24:20 smithi047 conmon[78521]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:24:20.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:24:20 smithi115 conmon[90393]: 2022-05-14T02:24:20.602+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:24:21.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:24:21 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:24:21.358 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T02:24:21.358 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:24:21.359 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T02:24:21.359 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:24:21.359 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:24:21.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:24:21.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T02:24:21.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:24:21.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T02:24:21.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:24:21.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T02:24:21.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:24:21.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T02:24:21.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:24:21.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T02:24:21.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:24:21.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:24:21.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:24:21.363 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:24:21 smithi115 ceph-mon[78593]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T02:24:21.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T02:24:21.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:24:21.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T02:24:21.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:24:21.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:24:21.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:24:21.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T02:24:21.365 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:24:21.365 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T02:24:21.365 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:24:21.365 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T02:24:21.366 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:24:21.366 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T02:24:21.366 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:24:21.366 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T02:24:21.367 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:24:21.367 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:24:21.367 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:24:21.367 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T02:24:21.368 INFO:journalctl@ceph.mon.c.smithi115.stdout: During handling of the above exception, another exception occurred: 2022-05-14T02:24:21.368 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T02:24:21.368 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T02:24:21.368 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T02:24:21.369 INFO:journalctl@ceph.mon.c.smithi115.stdout: return f(*arg) 2022-05-14T02:24:21.369 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T02:24:21.369 INFO:journalctl@ceph.mon.c.smithi115.stdout: self._write_client_files(client_files, host) 2022-05-14T02:24:21.369 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T02:24:21.370 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T02:24:21.370 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T02:24:21.370 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T02:24:21.370 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T02:24:21.371 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.event_loop.get_result(coro) 2022-05-14T02:24:21.371 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T02:24:21.371 INFO:journalctl@ceph.mon.c.smithi115.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T02:24:21.371 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T02:24:21.372 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.__get_result() 2022-05-14T02:24:21.372 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T02:24:21.372 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise self._exception 2022-05-14T02:24:21.372 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T02:24:21.372 INFO:journalctl@ceph.mon.c.smithi115.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T02:24:21.373 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T02:24:21.373 INFO:journalctl@ceph.mon.c.smithi115.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T02:24:21.373 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T02:24:21.373 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise 2022-05-14T02:24:21.374 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T02:24:21.374 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.gen.throw(type, value, traceback) 2022-05-14T02:24:21.374 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T02:24:21.374 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OrchestratorError(msg) 2022-05-14T02:24:21.374 INFO:journalctl@ceph.mon.c.smithi115.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:24:21.375 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:24:21 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:24:21.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:24:21 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:24:21.542 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T02:24:21.542 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:24:21.542 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T02:24:21.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:24:21.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:24:21.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:24:21.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T02:24:21.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:24:21.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T02:24:21.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:24:21.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T02:24:21.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:24:21.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T02:24:21.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:24:21.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T02:24:21.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:24:21.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:24:21.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:24:21.546 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:24:21 smithi047 ceph-mon[77147]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T02:24:21.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T02:24:21.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:24:21.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T02:24:21.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:24:21.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:24:21.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:24:21.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T02:24:21.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:24:21.548 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T02:24:21.548 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:24:21.548 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T02:24:21.548 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:24:21.548 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T02:24:21.549 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:24:21.549 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T02:24:21.549 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:24:21.549 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:24:21.549 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:24:21.550 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T02:24:21.550 INFO:journalctl@ceph.mon.b.smithi047.stdout: During handling of the above exception, another exception occurred: 2022-05-14T02:24:21.550 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T02:24:21.550 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T02:24:21.550 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T02:24:21.551 INFO:journalctl@ceph.mon.b.smithi047.stdout: return f(*arg) 2022-05-14T02:24:21.551 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T02:24:21.551 INFO:journalctl@ceph.mon.b.smithi047.stdout: self._write_client_files(client_files, host) 2022-05-14T02:24:21.551 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T02:24:21.552 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T02:24:21.552 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T02:24:21.552 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T02:24:21.552 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T02:24:21.552 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.event_loop.get_result(coro) 2022-05-14T02:24:21.553 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T02:24:21.553 INFO:journalctl@ceph.mon.b.smithi047.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T02:24:21.553 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T02:24:21.553 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.__get_result() 2022-05-14T02:24:21.553 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T02:24:21.554 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise self._exception 2022-05-14T02:24:21.554 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T02:24:21.554 INFO:journalctl@ceph.mon.b.smithi047.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T02:24:21.554 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T02:24:21.555 INFO:journalctl@ceph.mon.b.smithi047.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T02:24:21.555 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T02:24:21.555 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise 2022-05-14T02:24:21.555 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T02:24:21.555 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.gen.throw(type, value, traceback) 2022-05-14T02:24:21.556 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T02:24:21.556 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OrchestratorError(msg) 2022-05-14T02:24:21.556 INFO:journalctl@ceph.mon.b.smithi047.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:24:21.556 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:24:21 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:24:22.093 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:24:21 smithi115 conmon[90393]: 2022-05-14T02:24:21.616+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:24:22.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:24:22 smithi115 ceph-mon[78593]: pgmap v5649: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:24:22.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:24:22 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 10595 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:24:22.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:24:22 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:24:22.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:24:22 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:24:22.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:24:22 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:24:22.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:24:22 smithi047 ceph-mon[77147]: pgmap v5649: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:24:22.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:24:22 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 10595 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:24:22.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:24:22 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:24:22.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:24:22 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:24:22.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:24:22 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:24:23.098 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:24:22 smithi115 conmon[90393]: 2022-05-14T02:24:22.621+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:24:23.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:24:23 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:24:23.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:24:23 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:24:24.099 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:24:23 smithi115 conmon[90393]: 2022-05-14T02:24:23.650+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:24:24.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:24:24 smithi115 ceph-mon[78593]: pgmap v5650: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:24:24.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:24:24 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:24:24.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:24:24 smithi047 ceph-mon[77147]: pgmap v5650: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:24:24.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:24:24 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:24:25.104 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:24:24 smithi115 conmon[90393]: 2022-05-14T02:24:24.642+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:24:25.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:24:25 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:24:25.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:24:25 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:24:26.106 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:24:25 smithi115 conmon[90393]: 2022-05-14T02:24:25.614+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:24:26.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:24:26 smithi115 ceph-mon[78593]: pgmap v5651: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:24:26.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:24:26 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:24:26.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:24:26 smithi047 ceph-mon[77147]: pgmap v5651: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:24:26.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:24:26 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:24:26.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:24:26 smithi115 conmon[90393]: 2022-05-14T02:24:26.568+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:24:27.525 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:24:27 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:24:27.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:24:27 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:24:27.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:24:27 smithi115 conmon[90393]: 2022-05-14T02:24:27.524+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:24:28.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:24:28 smithi047 ceph-mon[77147]: pgmap v5652: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:24:28.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:24:28 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:24:28.557 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:24:28 smithi115 ceph-mon[78593]: pgmap v5652: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:24:28.558 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:24:28 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:24:28.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:24:28 smithi115 conmon[90393]: 2022-05-14T02:24:28.555+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:24:29.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:24:29 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:24:29.606 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:24:29 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:24:29.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:24:29 smithi115 conmon[90393]: 2022-05-14T02:24:29.604+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:24:30.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:24:30 smithi047 ceph-mon[77147]: pgmap v5653: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:24:30.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:24:30 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:24:30.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:24:30 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:24:30.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:24:30 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:24:30.571 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:24:30 smithi115 ceph-mon[78593]: pgmap v5653: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:24:30.572 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:24:30 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:24:30.572 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:24:30 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:24:30.572 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:24:30 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:24:30.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:24:30 smithi115 conmon[90393]: 2022-05-14T02:24:30.570+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:24:31.530 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:24:31 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:24:31.531 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:24:31 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 10600 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:24:31.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:24:31 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:24:31.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:24:31 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 10600 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:24:31.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:24:31 smithi115 conmon[90393]: 2022-05-14T02:24:31.528+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:24:32.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:24:32 smithi047 ceph-mon[77147]: pgmap v5654: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:24:32.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:24:32 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:24:32.566 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:24:32 smithi115 ceph-mon[78593]: pgmap v5654: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:24:32.566 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:24:32 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:24:32.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:24:32 smithi115 conmon[90393]: 2022-05-14T02:24:32.565+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:24:33.529 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:24:33 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:24:33.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:24:33 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:24:33.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:24:33 smithi115 conmon[90393]: 2022-05-14T02:24:33.528+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:24:34.485 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:24:34 smithi115 ceph-mon[78593]: pgmap v5655: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:24:34.486 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:24:34 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:24:34.486 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:24:34 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:24:34.486 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:24:34 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:24:34.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:24:34 smithi047 ceph-mon[77147]: pgmap v5655: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:24:34.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:24:34 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:24:34.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:24:34 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:24:34.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:24:34 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:24:34.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:24:34 smithi115 conmon[90393]: 2022-05-14T02:24:34.484+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:24:35.453 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:24:35 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:24:35.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:24:35 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:24:35.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:24:35 smithi115 conmon[90393]: 2022-05-14T02:24:35.452+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:24:36.457 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:24:36 smithi115 ceph-mon[78593]: pgmap v5656: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:24:36.458 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:24:36 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:24:36.458 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:24:36 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 10610 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:24:36.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:24:36 smithi047 ceph-mon[77147]: pgmap v5656: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:24:36.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:24:36 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:24:36.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:24:36 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 10610 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:24:36.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:24:36 smithi115 conmon[90393]: 2022-05-14T02:24:36.456+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:24:37.481 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:24:37 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:24:37.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:24:37 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:24:37.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:24:37 smithi115 conmon[90393]: 2022-05-14T02:24:37.480+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:24:38.527 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:24:38 smithi115 ceph-mon[78593]: pgmap v5657: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:24:38.527 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:24:38 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:24:38.527 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:24:38 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:24:38.528 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:24:38 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:24:38.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:24:38 smithi047 ceph-mon[77147]: pgmap v5657: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:24:38.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:24:38 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:24:38.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:24:38 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:24:38.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:24:38 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:24:38.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:24:38 smithi115 conmon[90393]: 2022-05-14T02:24:38.526+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:24:39.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:24:39 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:24:39.552 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:24:39 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:24:39.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:24:39 smithi115 conmon[90393]: 2022-05-14T02:24:39.551+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:24:40.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:24:40 smithi047 ceph-mon[77147]: pgmap v5658: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:24:40.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:24:40 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:24:40.583 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:24:40 smithi115 ceph-mon[78593]: pgmap v5658: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:24:40.583 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:24:40 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:24:40.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:24:40 smithi115 conmon[90393]: 2022-05-14T02:24:40.581+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:24:41.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:24:41 smithi115 conmon[90393]: 2022-05-14T02:24:41.593+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:24:41.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:24:41 smithi115 ceph-mon[78593]: pgmap v5659: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:24:41.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:24:41 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:24:41.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:24:41 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 10615 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:24:42.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:24:41 smithi047 ceph-mon[77147]: pgmap v5659: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:24:42.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:24:41 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:24:42.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:24:41 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 10615 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:24:43.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:24:42 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:24:43.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:24:42 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:24:43.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:24:42 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:24:43.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:24:42 smithi115 conmon[90393]: 2022-05-14T02:24:42.626+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:24:43.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:24:42 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:24:43.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:24:42 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:24:43.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:24:42 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:24:44.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:24:43 smithi047 ceph-mon[77147]: pgmap v5660: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:24:44.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:24:43 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:24:44.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:24:43 smithi115 conmon[90393]: 2022-05-14T02:24:43.634+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:24:44.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:24:43 smithi115 ceph-mon[78593]: pgmap v5660: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:24:44.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:24:43 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:24:45.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:24:44 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:24:45.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:24:44 smithi115 conmon[90393]: 2022-05-14T02:24:44.622+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:24:45.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:24:44 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:24:46.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:24:45 smithi047 ceph-mon[77147]: pgmap v5661: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:24:46.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:24:45 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:24:46.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:24:45 smithi115 conmon[90393]: 2022-05-14T02:24:45.641+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:24:46.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:24:45 smithi115 ceph-mon[78593]: pgmap v5661: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:24:46.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:24:45 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:24:47.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:24:46 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:24:47.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:24:46 smithi115 conmon[90393]: 2022-05-14T02:24:46.691+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:24:47.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:24:46 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:24:48.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:24:47 smithi047 ceph-mon[77147]: pgmap v5662: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:24:48.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:24:47 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:24:48.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:24:47 smithi115 conmon[90393]: 2022-05-14T02:24:47.720+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:24:48.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:24:47 smithi115 ceph-mon[78593]: pgmap v5662: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:24:48.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:24:47 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:24:49.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:24:48 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:24:49.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:24:48 smithi115 conmon[90393]: 2022-05-14T02:24:48.690+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:24:49.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:24:48 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:24:50.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:24:49 smithi047 ceph-mon[77147]: pgmap v5663: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:24:50.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:24:49 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:24:50.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:24:49 smithi115 conmon[90393]: 2022-05-14T02:24:49.690+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:24:50.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:24:49 smithi115 ceph-mon[78593]: pgmap v5663: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:24:50.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:24:49 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:24:51.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:24:50 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:24:51.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:24:50 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:24:51.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:24:50 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:24:51.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:24:50 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 10620 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:24:51.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:24:50 smithi115 conmon[90393]: 2022-05-14T02:24:50.740+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:24:51.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:24:50 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:24:51.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:24:50 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:24:51.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:24:50 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:24:51.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:24:50 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 10620 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:24:52.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:24:51 smithi047 ceph-mon[77147]: pgmap v5664: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:24:52.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:24:51 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:24:52.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:24:51 smithi115 conmon[90393]: 2022-05-14T02:24:51.764+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:24:52.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:24:51 smithi115 ceph-mon[78593]: pgmap v5664: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:24:52.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:24:51 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:24:53.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:24:52 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:24:53.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:24:52 smithi115 conmon[90393]: 2022-05-14T02:24:52.764+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:24:53.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:24:52 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:24:54.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:24:53 smithi047 ceph-mon[77147]: pgmap v5665: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:24:54.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:24:53 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:24:54.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:24:53 smithi115 conmon[90393]: 2022-05-14T02:24:53.774+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:24:54.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:24:53 smithi115 ceph-mon[78593]: pgmap v5665: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:24:54.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:24:53 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:24:55.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:24:54 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:24:55.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:24:54 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:24:55.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:24:54 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:24:55.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:24:54 smithi115 conmon[90393]: 2022-05-14T02:24:54.773+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:24:55.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:24:54 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:24:55.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:24:54 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:24:55.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:24:54 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:24:56.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:24:55 smithi047 ceph-mon[77147]: pgmap v5666: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:24:56.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:24:55 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:24:56.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:24:55 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 10630 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:24:56.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:24:55 smithi115 conmon[90393]: 2022-05-14T02:24:55.807+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:24:56.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:24:55 smithi115 ceph-mon[78593]: pgmap v5666: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:24:56.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:24:55 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:24:56.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:24:55 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 10630 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:24:57.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:24:56 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:24:57.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:24:56 smithi115 conmon[90393]: 2022-05-14T02:24:56.778+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:24:57.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:24:56 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:24:58.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:24:57 smithi047 ceph-mon[77147]: pgmap v5667: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:24:58.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:24:57 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:24:58.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:24:57 smithi115 conmon[90393]: 2022-05-14T02:24:57.760+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:24:58.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:24:57 smithi115 ceph-mon[78593]: pgmap v5667: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:24:58.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:24:57 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:24:59.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:24:58 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:24:59.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:24:58 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:24:59.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:24:58 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:24:59.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:24:58 smithi115 conmon[90393]: 2022-05-14T02:24:58.790+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:24:59.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:24:58 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:24:59.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:24:58 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:24:59.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:24:58 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:25:00.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:24:59 smithi047 ceph-mon[77147]: pgmap v5668: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:25:00.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:24:59 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:25:00.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:24:59 smithi115 conmon[90393]: 2022-05-14T02:24:59.823+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:25:00.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:24:59 smithi115 ceph-mon[78593]: pgmap v5668: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:25:00.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:24:59 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:25:01.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:25:00 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:25:01.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:25:00 smithi115 conmon[90393]: 2022-05-14T02:25:00.822+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:25:01.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:25:00 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:25:02.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:25:01 smithi047 ceph-mon[77147]: pgmap v5669: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:25:02.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:25:01 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 10635 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:25:02.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:25:01 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:25:02.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:25:01 smithi115 conmon[90393]: 2022-05-14T02:25:01.842+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:25:02.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:25:01 smithi115 ceph-mon[78593]: pgmap v5669: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:25:02.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:25:01 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 10635 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:25:02.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:25:01 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:25:03.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:25:02 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:25:03.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:25:02 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:25:03.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:25:02 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:25:03.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:25:02 smithi115 conmon[90393]: 2022-05-14T02:25:02.842+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:25:03.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:25:02 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:25:03.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:25:02 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:25:03.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:25:02 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:25:04.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:25:03 smithi047 ceph-mon[77147]: pgmap v5670: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:25:04.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:25:03 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:25:04.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:25:03 smithi115 conmon[90393]: 2022-05-14T02:25:03.875+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:25:04.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:25:03 smithi115 ceph-mon[78593]: pgmap v5670: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:25:04.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:25:03 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:25:05.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:25:04 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:25:05.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:25:04 smithi115 conmon[90393]: 2022-05-14T02:25:04.885+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:25:05.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:25:04 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:25:06.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:25:05 smithi047 ceph-mon[77147]: pgmap v5671: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:25:06.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:25:05 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:25:06.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:25:05 smithi115 conmon[90393]: 2022-05-14T02:25:05.848+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:25:06.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:25:05 smithi115 ceph-mon[78593]: pgmap v5671: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:25:06.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:25:05 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:25:07.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:25:06 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:25:07.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:25:06 smithi115 conmon[90393]: 2022-05-14T02:25:06.806+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:25:07.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:25:06 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:25:08.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:25:07 smithi047 ceph-mon[77147]: pgmap v5672: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:25:08.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:25:07 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:25:08.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:25:07 smithi115 conmon[90393]: 2022-05-14T02:25:07.845+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:25:08.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:25:07 smithi115 ceph-mon[78593]: pgmap v5672: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:25:08.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:25:07 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:25:09.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:25:08 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:25:09.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:25:08 smithi115 conmon[90393]: 2022-05-14T02:25:08.823+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:25:09.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:25:08 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:25:10.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:25:09 smithi047 ceph-mon[77147]: pgmap v5673: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:25:10.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:25:09 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:25:10.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:25:09 smithi115 conmon[90393]: 2022-05-14T02:25:09.822+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:25:10.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:25:09 smithi115 ceph-mon[78593]: pgmap v5673: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:25:10.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:25:09 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:25:11.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:25:10 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:25:11.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:25:10 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:25:11.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:25:10 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:25:11.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:25:10 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 10640 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:25:11.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:25:10 smithi115 conmon[90393]: 2022-05-14T02:25:10.836+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:25:11.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:25:10 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:25:11.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:25:10 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:25:11.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:25:10 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:25:11.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:25:10 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 10640 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:25:12.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:25:11 smithi047 ceph-mon[77147]: pgmap v5674: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:25:12.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:25:11 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:25:12.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:25:11 smithi115 conmon[90393]: 2022-05-14T02:25:11.835+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:25:12.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:25:11 smithi115 ceph-mon[78593]: pgmap v5674: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:25:12.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:25:11 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:25:13.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:25:12 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:25:13.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:25:12 smithi115 conmon[90393]: 2022-05-14T02:25:12.787+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:25:13.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:25:12 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:25:14.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:25:13 smithi047 ceph-mon[77147]: pgmap v5675: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:25:14.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:25:13 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:25:14.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:25:13 smithi115 conmon[90393]: 2022-05-14T02:25:13.830+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:25:14.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:25:13 smithi115 ceph-mon[78593]: pgmap v5675: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:25:14.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:25:13 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:25:15.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:25:14 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:25:15.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:25:14 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:25:15.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:25:14 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:25:15.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:25:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T02:25:15.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:25:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T02:25:15.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:25:14 smithi115 conmon[90393]: 2022-05-14T02:25:14.796+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:25:15.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:25:14 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:25:15.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:25:14 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:25:15.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:25:14 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:25:15.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:25:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T02:25:15.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:25:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T02:25:16.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:25:15 smithi047 ceph-mon[77147]: pgmap v5676: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:25:16.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:25:15 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:25:16.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:25:15 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 10650 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:25:16.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:25:15 smithi115 conmon[90393]: 2022-05-14T02:25:15.814+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:25:16.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:25:15 smithi115 ceph-mon[78593]: pgmap v5676: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:25:16.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:25:15 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:25:16.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:25:15 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 10650 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:25:17.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:25:16 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:25:17.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:25:16 smithi115 conmon[90393]: 2022-05-14T02:25:16.847+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:25:17.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:25:16 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:25:18.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:25:17 smithi047 ceph-mon[77147]: pgmap v5677: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:25:18.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:25:17 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:25:18.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:25:17 smithi115 conmon[90393]: 2022-05-14T02:25:17.821+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:25:18.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:25:17 smithi115 ceph-mon[78593]: pgmap v5677: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:25:18.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:25:17 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:25:19.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:25:18 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:25:19.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:25:18 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:25:19.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:25:18 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:25:19.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:25:18 smithi115 conmon[90393]: 2022-05-14T02:25:18.849+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:25:19.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:25:18 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:25:19.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:25:18 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:25:19.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:25:18 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:25:20.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:25:19 smithi047 ceph-mon[77147]: pgmap v5678: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:25:20.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:25:19 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:25:20.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:25:19 smithi115 conmon[90393]: 2022-05-14T02:25:19.842+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:25:20.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:25:19 smithi115 ceph-mon[78593]: pgmap v5678: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:25:20.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:25:19 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:25:21.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:25:20 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:25:21.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:25:20 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T02:25:21.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:25:20 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T02:25:21.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:25:20 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T02:25:21.042 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:25:20 smithi047 conmon[78521]: 2022-05-14T02:25:20.623+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:25:21.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:25:20 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T02:25:21.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:25:20 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:25:21.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:25:20 smithi047 conmon[78521]: yield 2022-05-14T02:25:21.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:25:20 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:25:21.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:25:20 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:25:21.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:25:20 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:25:21.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:25:20 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T02:25:21.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:25:20 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:25:21.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:25:20 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T02:25:21.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:25:20 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:25:21.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:25:20 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T02:25:21.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:25:20 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:25:21.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:25:20 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T02:25:21.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:25:20 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:25:21.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:25:20 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T02:25:21.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:25:20 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:25:21.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:25:20 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:25:21.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:25:20 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:25:21.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:25:20 smithi115 conmon[90393]: 2022-05-14T02:25:20.801+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:25:21.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:25:20 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:25:21.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:25:20 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T02:25:21.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:25:20 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T02:25:21.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:25:20 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T02:25:22.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:25:21 smithi115 conmon[90393]: 2022-05-14T02:25:21.761+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:25:22.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:25:21 smithi115 ceph-mon[78593]: pgmap v5679: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:25:22.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:25:21 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:25:22.109 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T02:25:22.109 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:25:22.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T02:25:22.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:25:22.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:25:22.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:25:22.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T02:25:22.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:25:22.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T02:25:22.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:25:22.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T02:25:22.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:25:22.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T02:25:22.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:25:22.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T02:25:22.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:25:22.113 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:25:22.113 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:25:22.113 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:25:21 smithi115 ceph-mon[78593]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T02:25:22.113 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:25:21 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 10655 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:25:22.114 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:25:21 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:25:22.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:25:21 smithi047 ceph-mon[77147]: pgmap v5679: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:25:22.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:25:21 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:25:22.292 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T02:25:22.292 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:25:22.292 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T02:25:22.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:25:22.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:25:22.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:25:22.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T02:25:22.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:25:22.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T02:25:22.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:25:22.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T02:25:22.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:25:22.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T02:25:22.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:25:22.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T02:25:22.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:25:22.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:25:22.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:25:22.296 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:25:21 smithi047 ceph-mon[77147]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T02:25:22.296 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:25:21 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 10655 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:25:22.296 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:25:21 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:25:23.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:25:22 smithi115 conmon[90393]: 2022-05-14T02:25:22.716+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:25:23.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:25:22 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:25:23.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:25:22 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:25:23.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:25:22 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:25:23.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:25:22 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:25:23.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:25:22 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:25:23.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:25:22 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:25:24.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:25:23 smithi047 ceph-mon[77147]: pgmap v5680: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:25:24.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:25:23 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:25:24.042 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:25:23 smithi047 conmon[78521]: 2022-05-14T02:25:23.695+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:25:24.042 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:25:23 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T02:25:24.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:25:23 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:25:24.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:25:23 smithi047 conmon[78521]: yield 2022-05-14T02:25:24.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:25:23 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:25:24.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:25:23 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:25:24.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:25:23 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:25:24.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:25:23 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T02:25:24.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:25:23 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:25:24.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:25:23 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T02:25:24.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:25:23 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:25:24.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:25:23 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T02:25:24.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:25:23 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:25:24.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:25:23 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T02:25:24.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:25:23 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:25:24.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:25:23 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T02:25:24.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:25:23 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:25:24.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:25:23 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:25:24.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:25:23 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:25:24.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:25:23 smithi047 conmon[78521]: 2022-05-14T02:25:23.696+0000 7f534177b700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T02:25:24.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:25:23 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T02:25:24.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:25:23 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:25:24.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:25:23 smithi047 conmon[78521]: yield 2022-05-14T02:25:24.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:25:23 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:25:24.048 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:25:23 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:25:24.048 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:25:23 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:25:24.048 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:25:23 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T02:25:24.048 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:25:23 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:25:24.048 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:25:23 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T02:25:24.049 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:25:23 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:25:24.049 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:25:23 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T02:25:24.049 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:25:23 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:25:24.049 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:25:23 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T02:25:24.049 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:25:23 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:25:24.050 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:25:23 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T02:25:24.050 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:25:23 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:25:24.050 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:25:23 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:25:24.050 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:25:23 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:25:24.050 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:25:23 smithi047 conmon[78521]: 2022-05-14T02:25:24.051 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:25:23 smithi047 conmon[78521]: During handling of the above exception, another exception occurred: 2022-05-14T02:25:24.051 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:25:23 smithi047 conmon[78521]: 2022-05-14T02:25:24.051 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:25:23 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T02:25:24.051 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:25:23 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T02:25:24.051 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:25:23 smithi047 conmon[78521]: return f(*arg) 2022-05-14T02:25:24.052 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:25:23 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T02:25:24.052 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:25:23 smithi047 conmon[78521]: self._write_client_files(client_files, host) 2022-05-14T02:25:24.052 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:25:23 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T02:25:24.052 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:25:23 smithi047 conmon[78521]: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T02:25:24.053 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:25:23 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T02:25:24.053 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:25:23 smithi047 conmon[78521]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T02:25:24.053 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:25:23 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T02:25:24.053 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:25:23 smithi047 conmon[78521]: return self.event_loop.get_result(coro) 2022-05-14T02:25:24.053 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:25:23 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T02:25:24.054 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:25:23 smithi047 conmon[78521]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T02:25:24.054 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:25:23 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T02:25:24.054 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:25:23 smithi047 conmon[78521]: return self.__get_result() 2022-05-14T02:25:24.054 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:25:23 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T02:25:24.054 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:25:23 smithi047 conmon[78521]: raise self._exception 2022-05-14T02:25:24.055 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:25:23 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T02:25:24.055 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:25:23 smithi047 conmon[78521]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T02:25:24.055 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:25:23 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T02:25:24.055 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:25:23 smithi047 conmon[78521]: conn = await self._remote_connection(host, addr) 2022-05-14T02:25:24.056 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:25:23 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T02:25:24.056 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:25:23 smithi047 conmon[78521]: raise 2022-05-14T02:25:24.056 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:25:23 smithi047 conmon[78521]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T02:25:24.056 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:25:23 smithi047 conmon[78521]: self.gen.throw(type, value, traceback) 2022-05-14T02:25:24.056 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:25:23 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T02:25:24.057 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:25:23 smithi047 conmon[78521]: raise OrchestratorError(msg) 2022-05-14T02:25:24.057 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:25:23 smithi047 conmon[78521]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:25:24.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:25:23 smithi115 conmon[90393]: 2022-05-14T02:25:23.734+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:25:24.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:25:23 smithi115 ceph-mon[78593]: pgmap v5680: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:25:24.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:25:23 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:25:25.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:25:24 smithi115 conmon[90393]: 2022-05-14T02:25:24.739+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:25:25.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:25:24 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:25:25.109 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T02:25:25.109 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:25:25.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T02:25:25.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:25:25.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:25:25.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:25:25.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T02:25:25.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:25:25.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T02:25:25.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:25:25.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T02:25:25.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:25:25.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T02:25:25.113 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:25:25.113 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T02:25:25.113 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:25:25.113 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:25:25.113 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:25:25.114 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:25:24 smithi115 ceph-mon[78593]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T02:25:25.114 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T02:25:25.114 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:25:25.114 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T02:25:25.115 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:25:25.115 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:25:25.115 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:25:25.115 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T02:25:25.115 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:25:25.116 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T02:25:25.116 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:25:25.116 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T02:25:25.116 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:25:25.117 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T02:25:25.117 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:25:25.117 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T02:25:25.117 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:25:25.118 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:25:25.118 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:25:25.118 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T02:25:25.118 INFO:journalctl@ceph.mon.c.smithi115.stdout: During handling of the above exception, another exception occurred: 2022-05-14T02:25:25.119 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T02:25:25.119 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T02:25:25.119 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T02:25:25.119 INFO:journalctl@ceph.mon.c.smithi115.stdout: return f(*arg) 2022-05-14T02:25:25.119 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T02:25:25.120 INFO:journalctl@ceph.mon.c.smithi115.stdout: self._write_client_files(client_files, host) 2022-05-14T02:25:25.120 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T02:25:25.120 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T02:25:25.120 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T02:25:25.121 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T02:25:25.121 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T02:25:25.121 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.event_loop.get_result(coro) 2022-05-14T02:25:25.121 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T02:25:25.122 INFO:journalctl@ceph.mon.c.smithi115.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T02:25:25.122 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T02:25:25.122 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.__get_result() 2022-05-14T02:25:25.122 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T02:25:25.123 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise self._exception 2022-05-14T02:25:25.123 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T02:25:25.123 INFO:journalctl@ceph.mon.c.smithi115.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T02:25:25.123 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T02:25:25.123 INFO:journalctl@ceph.mon.c.smithi115.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T02:25:25.124 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T02:25:25.124 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise 2022-05-14T02:25:25.124 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T02:25:25.124 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.gen.throw(type, value, traceback) 2022-05-14T02:25:25.125 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T02:25:25.125 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OrchestratorError(msg) 2022-05-14T02:25:25.125 INFO:journalctl@ceph.mon.c.smithi115.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:25:25.125 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:25:24 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:25:25.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:25:24 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:25:25.292 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T02:25:25.292 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:25:25.292 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T02:25:25.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:25:25.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:25:25.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:25:25.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T02:25:25.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:25:25.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T02:25:25.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:25:25.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T02:25:25.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:25:25.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T02:25:25.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:25:25.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T02:25:25.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:25:25.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:25:25.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:25:25.296 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:25:24 smithi047 ceph-mon[77147]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T02:25:25.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T02:25:25.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:25:25.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T02:25:25.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:25:25.298 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:25:25.298 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:25:25.298 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T02:25:25.298 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:25:25.299 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T02:25:25.299 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:25:25.299 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T02:25:25.299 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:25:25.300 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T02:25:25.300 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:25:25.300 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T02:25:25.300 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:25:25.301 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:25:25.301 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:25:25.301 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T02:25:25.301 INFO:journalctl@ceph.mon.b.smithi047.stdout: During handling of the above exception, another exception occurred: 2022-05-14T02:25:25.302 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T02:25:25.302 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T02:25:25.302 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T02:25:25.303 INFO:journalctl@ceph.mon.b.smithi047.stdout: return f(*arg) 2022-05-14T02:25:25.303 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T02:25:25.303 INFO:journalctl@ceph.mon.b.smithi047.stdout: self._write_client_files(client_files, host) 2022-05-14T02:25:25.304 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T02:25:25.304 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T02:25:25.304 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T02:25:25.304 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T02:25:25.304 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T02:25:25.305 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.event_loop.get_result(coro) 2022-05-14T02:25:25.305 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T02:25:25.305 INFO:journalctl@ceph.mon.b.smithi047.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T02:25:25.305 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T02:25:25.306 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.__get_result() 2022-05-14T02:25:25.306 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T02:25:25.306 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise self._exception 2022-05-14T02:25:25.306 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T02:25:25.307 INFO:journalctl@ceph.mon.b.smithi047.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T02:25:25.307 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T02:25:25.307 INFO:journalctl@ceph.mon.b.smithi047.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T02:25:25.307 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T02:25:25.308 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise 2022-05-14T02:25:25.308 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T02:25:25.308 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.gen.throw(type, value, traceback) 2022-05-14T02:25:25.308 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T02:25:25.308 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OrchestratorError(msg) 2022-05-14T02:25:25.309 INFO:journalctl@ceph.mon.b.smithi047.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:25:25.309 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:25:24 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:25:26.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:25:25 smithi115 conmon[90393]: 2022-05-14T02:25:25.758+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:25:26.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:25:25 smithi115 ceph-mon[78593]: pgmap v5681: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:25:26.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:25:25 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:25:26.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:25:25 smithi047 ceph-mon[77147]: pgmap v5681: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:25:26.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:25:25 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:25:27.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:25:26 smithi115 conmon[90393]: 2022-05-14T02:25:26.745+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:25:27.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:25:26 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:25:27.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:25:26 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:25:28.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:25:27 smithi115 conmon[90393]: 2022-05-14T02:25:27.701+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:25:28.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:25:27 smithi115 ceph-mon[78593]: pgmap v5682: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:25:28.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:25:27 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:25:28.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:25:27 smithi047 ceph-mon[77147]: pgmap v5682: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:25:28.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:25:27 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:25:29.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:25:28 smithi115 conmon[90393]: 2022-05-14T02:25:28.713+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:25:29.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:25:28 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:25:29.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:25:28 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:25:30.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:25:29 smithi115 conmon[90393]: 2022-05-14T02:25:29.753+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:25:30.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:25:29 smithi115 ceph-mon[78593]: pgmap v5683: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:25:30.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:25:29 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:25:30.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:25:29 smithi047 ceph-mon[77147]: pgmap v5683: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:25:30.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:25:29 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:25:31.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:25:30 smithi115 conmon[90393]: 2022-05-14T02:25:30.780+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:25:31.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:25:30 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:25:31.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:25:30 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:25:31.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:25:30 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:25:31.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:25:30 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 10660 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:25:31.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:25:30 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:25:31.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:25:30 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:25:31.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:25:30 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:25:31.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:25:30 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 10660 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:25:32.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:25:31 smithi115 conmon[90393]: 2022-05-14T02:25:31.734+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:25:32.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:25:31 smithi115 ceph-mon[78593]: pgmap v5684: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:25:32.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:25:31 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:25:32.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:25:31 smithi047 ceph-mon[77147]: pgmap v5684: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:25:32.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:25:31 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:25:33.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:25:32 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:25:33.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:25:32 smithi115 conmon[90393]: 2022-05-14T02:25:32.747+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:25:33.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:25:32 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:25:34.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:25:33 smithi115 conmon[90393]: 2022-05-14T02:25:33.698+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:25:34.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:25:33 smithi115 ceph-mon[78593]: pgmap v5685: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:25:34.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:25:33 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:25:34.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:25:33 smithi047 ceph-mon[77147]: pgmap v5685: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:25:34.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:25:33 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:25:35.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:25:34 smithi115 conmon[90393]: 2022-05-14T02:25:34.657+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:25:35.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:25:34 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:25:35.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:25:34 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:25:35.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:25:34 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:25:35.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:25:34 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:25:35.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:25:34 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:25:35.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:25:34 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:25:36.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:25:35 smithi115 conmon[90393]: 2022-05-14T02:25:35.635+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:25:36.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:25:35 smithi115 ceph-mon[78593]: pgmap v5686: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:25:36.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:25:35 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:25:36.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:25:35 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 10670 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:25:36.134 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:25:35 smithi047 ceph-mon[77147]: pgmap v5686: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:25:36.134 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:25:35 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:25:36.134 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:25:35 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 10670 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:25:37.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:25:36 smithi115 conmon[90393]: 2022-05-14T02:25:36.616+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:25:37.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:25:36 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:25:37.116 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:25:36 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:25:38.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:25:37 smithi047 ceph-mon[77147]: pgmap v5687: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:25:38.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:25:37 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:25:38.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:25:37 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:25:38.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:25:37 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:25:38.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:25:37 smithi115 conmon[90393]: 2022-05-14T02:25:37.636+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:25:38.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:25:37 smithi115 ceph-mon[78593]: pgmap v5687: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:25:38.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:25:37 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:25:38.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:25:37 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:25:38.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:25:37 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:25:39.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:25:38 smithi115 conmon[90393]: 2022-05-14T02:25:38.674+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:25:39.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:25:38 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:25:39.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:25:38 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:25:40.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:25:39 smithi115 conmon[90393]: 2022-05-14T02:25:39.707+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:25:40.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:25:39 smithi115 ceph-mon[78593]: pgmap v5688: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:25:40.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:25:39 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:25:40.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:25:39 smithi047 ceph-mon[77147]: pgmap v5688: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:25:40.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:25:39 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:25:41.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:25:40 smithi115 conmon[90393]: 2022-05-14T02:25:40.712+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:25:41.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:25:40 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:25:41.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:25:40 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:25:42.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:25:41 smithi115 conmon[90393]: 2022-05-14T02:25:41.676+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:25:42.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:25:41 smithi115 ceph-mon[78593]: pgmap v5689: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:25:42.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:25:41 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:25:42.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:25:41 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 10675 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:25:42.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:25:41 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:25:42.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:25:41 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:25:42.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:25:41 smithi047 ceph-mon[77147]: pgmap v5689: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:25:42.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:25:41 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:25:42.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:25:41 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 10675 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:25:42.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:25:41 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:25:42.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:25:41 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:25:43.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:25:42 smithi115 conmon[90393]: 2022-05-14T02:25:42.660+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:25:43.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:25:42 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:25:43.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:25:42 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:25:44.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:25:43 smithi115 conmon[90393]: 2022-05-14T02:25:43.642+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:25:44.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:25:43 smithi115 ceph-mon[78593]: pgmap v5690: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:25:44.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:25:43 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:25:44.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:25:43 smithi047 ceph-mon[77147]: pgmap v5690: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:25:44.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:25:43 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:25:45.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:25:44 smithi115 conmon[90393]: 2022-05-14T02:25:44.632+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:25:45.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:25:44 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:25:45.264 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:25:44 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:25:45.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:25:45 smithi115 conmon[90393]: 2022-05-14T02:25:45.606+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:25:46.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:25:45 smithi047 ceph-mon[77147]: pgmap v5691: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:25:46.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:25:45 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:25:46.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:25:45 smithi115 ceph-mon[78593]: pgmap v5691: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:25:46.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:25:45 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:25:46.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:25:46 smithi115 conmon[90393]: 2022-05-14T02:25:46.573+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:25:47.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:25:46 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:25:47.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:25:46 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:25:47.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:25:47 smithi115 conmon[90393]: 2022-05-14T02:25:47.544+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:25:48.266 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:25:47 smithi047 ceph-mon[77147]: pgmap v5692: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:25:48.267 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:25:47 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:25:48.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:25:47 smithi115 ceph-mon[78593]: pgmap v5692: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:25:48.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:25:47 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:25:48.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:25:48 smithi115 conmon[90393]: 2022-05-14T02:25:48.548+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:25:49.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:25:48 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:25:49.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:25:48 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:25:49.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:25:49 smithi115 conmon[90393]: 2022-05-14T02:25:49.585+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:25:50.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:25:49 smithi047 ceph-mon[77147]: pgmap v5693: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:25:50.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:25:49 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:25:50.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:25:49 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:25:50.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:25:49 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:25:50.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:25:49 smithi115 ceph-mon[78593]: pgmap v5693: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:25:50.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:25:49 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:25:50.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:25:49 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:25:50.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:25:49 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:25:50.903 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:25:50 smithi115 conmon[90393]: 2022-05-14T02:25:50.626+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:25:51.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:25:50 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:25:51.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:25:50 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 10680 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:25:51.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:25:50 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:25:51.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:25:50 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 10680 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:25:51.903 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:25:51 smithi115 conmon[90393]: 2022-05-14T02:25:51.633+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:25:52.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:25:51 smithi047 ceph-mon[77147]: pgmap v5694: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:25:52.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:25:51 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:25:52.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:25:51 smithi115 ceph-mon[78593]: pgmap v5694: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:25:52.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:25:51 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:25:52.908 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:25:52 smithi115 conmon[90393]: 2022-05-14T02:25:52.648+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:25:53.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:25:52 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:25:53.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:25:52 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:25:53.910 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:25:53 smithi115 conmon[90393]: 2022-05-14T02:25:53.629+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:25:54.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:25:53 smithi047 ceph-mon[77147]: pgmap v5695: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:25:54.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:25:53 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:25:54.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:25:53 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:25:54.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:25:53 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:25:54.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:25:53 smithi115 ceph-mon[78593]: pgmap v5695: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:25:54.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:25:53 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:25:54.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:25:53 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:25:54.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:25:53 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:25:54.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:25:54 smithi115 conmon[90393]: 2022-05-14T02:25:54.588+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:25:55.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:25:54 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:25:55.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:25:54 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:25:55.917 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:25:55 smithi115 conmon[90393]: 2022-05-14T02:25:55.614+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:25:56.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:25:55 smithi047 ceph-mon[77147]: pgmap v5696: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:25:56.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:25:55 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:25:56.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:25:55 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 10690 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:25:56.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:25:55 smithi115 ceph-mon[78593]: pgmap v5696: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:25:56.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:25:55 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:25:56.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:25:55 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 10690 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:25:56.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:25:56 smithi115 conmon[90393]: 2022-05-14T02:25:56.605+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:25:57.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:25:56 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:25:57.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:25:56 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:25:57.922 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:25:57 smithi115 conmon[90393]: 2022-05-14T02:25:57.650+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:25:58.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:25:57 smithi047 ceph-mon[77147]: pgmap v5697: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:25:58.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:25:57 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:25:58.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:25:57 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:25:58.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:25:57 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:25:58.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:25:57 smithi115 ceph-mon[78593]: pgmap v5697: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:25:58.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:25:57 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:25:58.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:25:57 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:25:58.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:25:57 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:25:58.927 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:25:58 smithi115 conmon[90393]: 2022-05-14T02:25:58.657+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:25:59.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:25:58 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:25:59.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:25:58 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:25:59.928 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:25:59 smithi115 conmon[90393]: 2022-05-14T02:25:59.624+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:26:00.279 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:25:59 smithi047 ceph-mon[77147]: pgmap v5698: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:26:00.279 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:25:59 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:26:00.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:25:59 smithi115 ceph-mon[78593]: pgmap v5698: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:26:00.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:25:59 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:26:00.933 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:26:00 smithi115 conmon[90393]: 2022-05-14T02:26:00.664+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:26:01.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:26:00 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:26:01.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:26:00 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:26:02.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:26:01 smithi115 conmon[90393]: 2022-05-14T02:26:01.706+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:26:02.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:26:01 smithi115 ceph-mon[78593]: pgmap v5699: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:26:02.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:26:01 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:26:02.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:26:01 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 10695 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:26:02.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:26:01 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:26:02.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:26:01 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:26:02.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:26:01 smithi047 ceph-mon[77147]: pgmap v5699: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:26:02.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:26:01 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:26:02.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:26:01 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 10695 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:26:02.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:26:01 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:26:02.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:26:01 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:26:02.939 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:26:02 smithi115 conmon[90393]: 2022-05-14T02:26:02.668+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:26:03.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:26:02 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:26:03.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:26:02 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:26:04.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:26:03 smithi115 conmon[90393]: 2022-05-14T02:26:03.709+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:26:04.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:26:03 smithi115 ceph-mon[78593]: pgmap v5700: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:26:04.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:26:03 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:26:04.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:26:03 smithi047 ceph-mon[77147]: pgmap v5700: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:26:04.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:26:03 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:26:05.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:26:04 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:26:05.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:26:04 smithi115 conmon[90393]: 2022-05-14T02:26:04.715+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:26:05.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:26:04 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:26:06.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:26:05 smithi115 conmon[90393]: 2022-05-14T02:26:05.707+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:26:06.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:26:05 smithi115 ceph-mon[78593]: pgmap v5701: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:26:06.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:26:05 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:26:06.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:26:05 smithi047 ceph-mon[77147]: pgmap v5701: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:26:06.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:26:05 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:26:06.952 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:26:06 smithi115 conmon[90393]: 2022-05-14T02:26:06.690+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:26:07.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:26:06 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:26:07.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:26:06 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:26:07.954 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:26:07 smithi115 conmon[90393]: 2022-05-14T02:26:07.675+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:26:08.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:26:07 smithi047 ceph-mon[77147]: pgmap v5702: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:26:08.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:26:07 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:26:08.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:26:07 smithi115 ceph-mon[78593]: pgmap v5702: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:26:08.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:26:07 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:26:08.959 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:26:08 smithi115 conmon[90393]: 2022-05-14T02:26:08.657+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:26:09.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:26:08 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:26:09.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:26:08 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:26:09.961 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:26:09 smithi115 conmon[90393]: 2022-05-14T02:26:09.691+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:26:10.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:26:09 smithi047 ceph-mon[77147]: pgmap v5703: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:26:10.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:26:09 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:26:10.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:26:09 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:26:10.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:26:09 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:26:10.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:26:09 smithi115 ceph-mon[78593]: pgmap v5703: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:26:10.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:26:09 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:26:10.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:26:09 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:26:10.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:26:09 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:26:10.961 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:26:10 smithi115 conmon[90393]: 2022-05-14T02:26:10.698+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:26:11.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:26:10 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:26:11.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:26:10 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 10700 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:26:11.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:26:10 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:26:11.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:26:10 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 10700 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:26:12.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:26:11 smithi115 conmon[90393]: 2022-05-14T02:26:11.716+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:26:12.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:26:11 smithi115 ceph-mon[78593]: pgmap v5704: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:26:12.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:26:11 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:26:12.297 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:26:11 smithi047 ceph-mon[77147]: pgmap v5704: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:26:12.297 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:26:11 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:26:13.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:26:12 smithi115 conmon[90393]: 2022-05-14T02:26:12.732+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:26:13.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:26:12 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:26:13.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:26:12 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:26:14.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:26:13 smithi115 conmon[90393]: 2022-05-14T02:26:13.765+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:26:14.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:26:13 smithi115 ceph-mon[78593]: pgmap v5705: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:26:14.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:26:13 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:26:14.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:26:13 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:26:14.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:26:13 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:26:14.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:26:13 smithi047 ceph-mon[77147]: pgmap v5705: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:26:14.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:26:13 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:26:14.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:26:13 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:26:14.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:26:13 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:26:14.978 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:26:14 smithi115 conmon[90393]: 2022-05-14T02:26:14.719+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:26:15.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:26:14 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:26:15.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:26:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T02:26:15.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:26:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T02:26:15.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:26:14 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:26:15.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:26:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T02:26:15.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:26:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T02:26:15.982 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:26:15 smithi115 conmon[90393]: 2022-05-14T02:26:15.698+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:26:16.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:26:15 smithi047 ceph-mon[77147]: pgmap v5706: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:26:16.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:26:15 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:26:16.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:26:15 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 10710 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:26:16.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:26:15 smithi115 ceph-mon[78593]: pgmap v5706: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:26:16.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:26:15 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:26:16.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:26:15 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 10710 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:26:16.988 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:26:16 smithi115 conmon[90393]: 2022-05-14T02:26:16.726+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:26:17.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:26:16 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:26:17.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:26:16 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:26:18.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:26:17 smithi115 conmon[90393]: 2022-05-14T02:26:17.769+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:26:18.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:26:17 smithi115 ceph-mon[78593]: pgmap v5707: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:26:18.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:26:17 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:26:18.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:26:17 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:26:18.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:26:17 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:26:18.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:26:17 smithi047 ceph-mon[77147]: pgmap v5707: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:26:18.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:26:17 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:26:18.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:26:17 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:26:18.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:26:17 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:26:19.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:26:18 smithi115 conmon[90393]: 2022-05-14T02:26:18.750+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:26:19.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:26:18 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:26:19.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:26:18 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:26:20.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:26:19 smithi115 conmon[90393]: 2022-05-14T02:26:19.760+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:26:20.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:26:19 smithi115 ceph-mon[78593]: pgmap v5708: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:26:20.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:26:19 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:26:20.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:26:19 smithi047 ceph-mon[77147]: pgmap v5708: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:26:20.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:26:19 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:26:21.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:26:20 smithi115 conmon[90393]: 2022-05-14T02:26:20.765+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:26:21.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:26:20 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:26:21.999 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:26:21 smithi115 conmon[90393]: 2022-05-14T02:26:21.742+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:26:22.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:26:20 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:26:22.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:26:21 smithi047 ceph-mon[77147]: pgmap v5709: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:26:22.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:26:21 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:26:22.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:26:21 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 10715 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:26:22.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:26:21 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:26:22.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:26:21 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:26:22.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:26:21 smithi115 ceph-mon[78593]: pgmap v5709: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:26:22.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:26:21 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:26:22.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:26:21 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 10715 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:26:22.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:26:21 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:26:22.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:26:21 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:26:23.004 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:26:22 smithi115 conmon[90393]: 2022-05-14T02:26:22.702+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:26:23.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:26:23 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:26:23.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:26:23 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:26:24.005 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:26:23 smithi115 conmon[90393]: 2022-05-14T02:26:23.731+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:26:24.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:26:24 smithi047 ceph-mon[77147]: pgmap v5710: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:26:24.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:26:24 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:26:24.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:26:24 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T02:26:24.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:26:24 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T02:26:24.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:26:24 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T02:26:24.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:26:24 smithi115 ceph-mon[78593]: pgmap v5710: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:26:24.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:26:24 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:26:24.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:26:24 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T02:26:24.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:26:24 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T02:26:24.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:26:24 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T02:26:25.010 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:26:24 smithi115 conmon[90393]: 2022-05-14T02:26:24.758+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:26:25.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:26:25 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:26:25.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:26:25 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:26:26.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:26:25 smithi115 conmon[90393]: 2022-05-14T02:26:25.804+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:26:26.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:26:26 smithi115 ceph-mon[78593]: pgmap v5711: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:26:26.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:26:26 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:26:26.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:26:26 smithi047 ceph-mon[77147]: pgmap v5711: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:26:26.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:26:26 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:26:27.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:26:27 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:26:27.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:26:26 smithi115 conmon[90393]: 2022-05-14T02:26:26.825+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:26:27.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:26:27 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:26:27.292 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:26:26 smithi047 conmon[78521]: 2022-05-14T02:26:26.799+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:26:27.292 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:26:26 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T02:26:27.292 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:26:26 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:26:27.292 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:26:26 smithi047 conmon[78521]: yield 2022-05-14T02:26:27.293 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:26:26 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:26:27.293 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:26:26 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:26:27.293 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:26:26 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:26:27.293 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:26:26 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T02:26:27.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:26:26 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:26:27.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:26:26 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T02:26:27.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:26:26 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:26:27.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:26:26 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T02:26:27.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:26:26 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:26:27.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:26:26 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T02:26:27.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:26:26 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:26:27.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:26:26 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T02:26:27.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:26:26 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:26:27.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:26:26 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:26:27.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:26:26 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:26:28.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:26:27 smithi115 conmon[90393]: 2022-05-14T02:26:27.800+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:26:28.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:26:28 smithi115 ceph-mon[78593]: pgmap v5712: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:26:28.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:26:28 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:26:28.109 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T02:26:28.109 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:26:28.109 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T02:26:28.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:26:28.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:26:28.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:26:28.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T02:26:28.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:26:28.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T02:26:28.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:26:28.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T02:26:28.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:26:28.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T02:26:28.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:26:28.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T02:26:28.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:26:28.113 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:26:28.113 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:26:28.113 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:26:28 smithi115 ceph-mon[78593]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T02:26:28.113 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:26:28 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:26:28.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:26:28 smithi047 ceph-mon[77147]: pgmap v5712: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:26:28.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:26:28 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:26:28.292 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T02:26:28.292 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:26:28.292 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T02:26:28.292 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:26:28.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:26:28.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:26:28.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T02:26:28.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:26:28.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T02:26:28.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:26:28.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T02:26:28.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:26:28.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T02:26:28.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:26:28.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T02:26:28.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:26:28.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:26:28.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:26:28.296 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:26:28 smithi047 ceph-mon[77147]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T02:26:28.296 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:26:28 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:26:29.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:26:28 smithi115 conmon[90393]: 2022-05-14T02:26:28.809+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:26:29.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:26:29 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:26:29.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:26:29 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:26:30.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:26:29 smithi115 conmon[90393]: 2022-05-14T02:26:29.798+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:26:30.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:26:30 smithi115 ceph-mon[78593]: pgmap v5713: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:26:30.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:26:30 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:26:30.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:26:30 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:26:30.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:26:30 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:26:30.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:26:30 smithi047 ceph-mon[77147]: pgmap v5713: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:26:30.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:26:30 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:26:30.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:26:30 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:26:30.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:26:30 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:26:30.292 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:26:29 smithi047 conmon[78521]: 2022-05-14T02:26:29.872+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:26:30.292 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:26:29 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T02:26:30.293 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:26:29 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:26:30.293 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:26:29 smithi047 conmon[78521]: yield 2022-05-14T02:26:30.293 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:26:29 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:26:30.293 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:26:29 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:26:30.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:26:29 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:26:30.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:26:29 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T02:26:30.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:26:29 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:26:30.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:26:29 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T02:26:30.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:26:29 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:26:30.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:26:29 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T02:26:30.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:26:29 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:26:30.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:26:29 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T02:26:30.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:26:29 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:26:30.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:26:29 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T02:26:30.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:26:29 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:26:30.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:26:29 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:26:30.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:26:29 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:26:30.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:26:29 smithi047 conmon[78521]: 2022-05-14T02:26:29.874+0000 7f5341f7c700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T02:26:30.297 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:26:29 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T02:26:30.297 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:26:29 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:26:30.297 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:26:29 smithi047 conmon[78521]: yield 2022-05-14T02:26:30.297 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:26:29 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:26:30.297 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:26:29 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:26:30.298 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:26:29 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:26:30.298 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:26:29 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T02:26:30.298 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:26:29 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:26:30.298 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:26:29 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T02:26:30.299 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:26:29 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:26:30.299 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:26:29 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T02:26:30.299 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:26:29 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:26:30.299 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:26:29 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T02:26:30.299 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:26:29 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:26:30.300 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:26:29 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T02:26:30.300 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:26:29 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:26:30.300 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:26:29 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:26:30.300 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:26:29 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:26:30.300 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:26:29 smithi047 conmon[78521]: 2022-05-14T02:26:30.301 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:26:29 smithi047 conmon[78521]: During handling of the above exception, another exception occurred: 2022-05-14T02:26:30.301 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:26:29 smithi047 conmon[78521]: 2022-05-14T02:26:30.301 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:26:29 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T02:26:30.301 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:26:29 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T02:26:30.302 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:26:29 smithi047 conmon[78521]: return f(*arg) 2022-05-14T02:26:30.302 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:26:29 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T02:26:30.302 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:26:29 smithi047 conmon[78521]: self._write_client_files(client_files, host) 2022-05-14T02:26:30.302 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:26:29 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T02:26:30.302 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:26:29 smithi047 conmon[78521]: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T02:26:30.303 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:26:29 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T02:26:30.303 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:26:29 smithi047 conmon[78521]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T02:26:30.303 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:26:29 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T02:26:30.303 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:26:29 smithi047 conmon[78521]: return self.event_loop.get_result(coro) 2022-05-14T02:26:30.304 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:26:29 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T02:26:30.304 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:26:29 smithi047 conmon[78521]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T02:26:30.304 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:26:29 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T02:26:30.304 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:26:29 smithi047 conmon[78521]: return self.__get_result() 2022-05-14T02:26:30.305 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:26:29 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T02:26:30.305 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:26:29 smithi047 conmon[78521]: raise self._exception 2022-05-14T02:26:30.305 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:26:29 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T02:26:30.305 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:26:29 smithi047 conmon[78521]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T02:26:30.305 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:26:29 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T02:26:30.306 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:26:29 smithi047 conmon[78521]: conn = await self._remote_connection(host, addr) 2022-05-14T02:26:30.306 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:26:29 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T02:26:30.306 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:26:29 smithi047 conmon[78521]: raise 2022-05-14T02:26:30.306 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:26:29 smithi047 conmon[78521]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T02:26:30.306 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:26:29 smithi047 conmon[78521]: self.gen.throw(type, value, traceback) 2022-05-14T02:26:30.307 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:26:29 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T02:26:30.307 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:26:29 smithi047 conmon[78521]: raise OrchestratorError(msg) 2022-05-14T02:26:30.307 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:26:29 smithi047 conmon[78521]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:26:31.033 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:26:30 smithi115 conmon[90393]: 2022-05-14T02:26:30.774+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:26:31.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:26:31 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:26:31.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:26:31 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:26:31.292 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T02:26:31.292 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:26:31.292 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T02:26:31.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:26:31.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:26:31.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:26:31.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T02:26:31.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:26:31.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T02:26:31.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:26:31.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T02:26:31.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:26:31.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T02:26:31.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:26:31.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T02:26:31.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:26:31.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:26:31.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:26:31.296 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:26:31 smithi047 ceph-mon[77147]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T02:26:31.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T02:26:31.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:26:31.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T02:26:31.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:26:31.298 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:26:31.298 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:26:31.298 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T02:26:31.298 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:26:31.298 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T02:26:31.299 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:26:31.299 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T02:26:31.299 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:26:31.299 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T02:26:31.300 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:26:31.300 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T02:26:31.300 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:26:31.300 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:26:31.300 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:26:31.301 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T02:26:31.301 INFO:journalctl@ceph.mon.b.smithi047.stdout: During handling of the above exception, another exception occurred: 2022-05-14T02:26:31.301 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T02:26:31.301 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T02:26:31.302 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T02:26:31.302 INFO:journalctl@ceph.mon.b.smithi047.stdout: return f(*arg) 2022-05-14T02:26:31.302 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T02:26:31.302 INFO:journalctl@ceph.mon.b.smithi047.stdout: self._write_client_files(client_files, host) 2022-05-14T02:26:31.303 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T02:26:31.303 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T02:26:31.303 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T02:26:31.303 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T02:26:31.304 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T02:26:31.304 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.event_loop.get_result(coro) 2022-05-14T02:26:31.304 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T02:26:31.304 INFO:journalctl@ceph.mon.b.smithi047.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T02:26:31.304 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T02:26:31.305 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.__get_result() 2022-05-14T02:26:31.305 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T02:26:31.305 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise self._exception 2022-05-14T02:26:31.305 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T02:26:31.306 INFO:journalctl@ceph.mon.b.smithi047.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T02:26:31.306 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T02:26:31.306 INFO:journalctl@ceph.mon.b.smithi047.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T02:26:31.306 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T02:26:31.307 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise 2022-05-14T02:26:31.307 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T02:26:31.307 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.gen.throw(type, value, traceback) 2022-05-14T02:26:31.307 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T02:26:31.308 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OrchestratorError(msg) 2022-05-14T02:26:31.308 INFO:journalctl@ceph.mon.b.smithi047.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:26:31.308 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:26:31 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 10720 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:26:31.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:26:31 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:26:31.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:26:31 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:26:31.359 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T02:26:31.359 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:26:31.359 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T02:26:31.359 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:26:31.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:26:31.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:26:31.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T02:26:31.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:26:31.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T02:26:31.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:26:31.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T02:26:31.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:26:31.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T02:26:31.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:26:31.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T02:26:31.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:26:31.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:26:31.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:26:31.363 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:26:31 smithi115 ceph-mon[78593]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T02:26:31.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T02:26:31.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:26:31.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T02:26:31.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:26:31.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:26:31.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:26:31.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T02:26:31.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:26:31.365 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T02:26:31.365 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:26:31.365 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T02:26:31.365 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:26:31.365 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T02:26:31.366 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:26:31.366 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T02:26:31.366 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:26:31.366 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:26:31.366 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:26:31.367 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T02:26:31.367 INFO:journalctl@ceph.mon.c.smithi115.stdout: During handling of the above exception, another exception occurred: 2022-05-14T02:26:31.367 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T02:26:31.367 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T02:26:31.367 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T02:26:31.368 INFO:journalctl@ceph.mon.c.smithi115.stdout: return f(*arg) 2022-05-14T02:26:31.368 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T02:26:31.368 INFO:journalctl@ceph.mon.c.smithi115.stdout: self._write_client_files(client_files, host) 2022-05-14T02:26:31.368 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T02:26:31.369 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T02:26:31.369 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T02:26:31.369 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T02:26:31.369 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T02:26:31.369 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.event_loop.get_result(coro) 2022-05-14T02:26:31.370 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T02:26:31.370 INFO:journalctl@ceph.mon.c.smithi115.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T02:26:31.370 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T02:26:31.370 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.__get_result() 2022-05-14T02:26:31.370 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T02:26:31.371 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise self._exception 2022-05-14T02:26:31.371 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T02:26:31.371 INFO:journalctl@ceph.mon.c.smithi115.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T02:26:31.371 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T02:26:31.371 INFO:journalctl@ceph.mon.c.smithi115.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T02:26:31.372 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T02:26:31.372 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise 2022-05-14T02:26:31.372 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T02:26:31.372 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.gen.throw(type, value, traceback) 2022-05-14T02:26:31.373 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T02:26:31.373 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OrchestratorError(msg) 2022-05-14T02:26:31.373 INFO:journalctl@ceph.mon.c.smithi115.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:26:31.373 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:26:31 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 10720 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:26:32.035 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:26:31 smithi115 conmon[90393]: 2022-05-14T02:26:31.724+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:26:32.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:26:32 smithi047 ceph-mon[77147]: pgmap v5714: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:26:32.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:26:32 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:26:32.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:26:32 smithi115 ceph-mon[78593]: pgmap v5714: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:26:32.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:26:32 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:26:33.041 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:26:32 smithi115 conmon[90393]: 2022-05-14T02:26:32.757+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:26:33.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:26:33 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:26:33.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:26:33 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:26:34.042 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:26:33 smithi115 conmon[90393]: 2022-05-14T02:26:33.775+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:26:34.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:26:34 smithi115 ceph-mon[78593]: pgmap v5715: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:26:34.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:26:34 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:26:34.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:26:34 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:26:34.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:26:34 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:26:34.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:26:34 smithi047 ceph-mon[77147]: pgmap v5715: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:26:34.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:26:34 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:26:34.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:26:34 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:26:34.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:26:34 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:26:35.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:26:35 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:26:35.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:26:34 smithi115 conmon[90393]: 2022-05-14T02:26:34.805+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:26:35.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:26:35 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:26:36.050 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:26:35 smithi115 conmon[90393]: 2022-05-14T02:26:35.758+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:26:36.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:26:36 smithi115 ceph-mon[78593]: pgmap v5716: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:26:36.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:26:36 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:26:36.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:26:36 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 10730 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:26:36.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:26:36 smithi047 ceph-mon[77147]: pgmap v5716: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:26:36.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:26:36 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:26:36.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:26:36 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 10730 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:26:37.055 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:26:36 smithi115 conmon[90393]: 2022-05-14T02:26:36.753+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:26:37.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:26:37 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:26:37.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:26:37 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:26:38.056 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:26:37 smithi115 conmon[90393]: 2022-05-14T02:26:37.727+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:26:38.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:26:38 smithi115 ceph-mon[78593]: pgmap v5717: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:26:38.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:26:38 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:26:38.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:26:38 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:26:38.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:26:38 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:26:38.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:26:38 smithi047 ceph-mon[77147]: pgmap v5717: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:26:38.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:26:38 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:26:38.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:26:38 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:26:38.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:26:38 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:26:39.061 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:26:38 smithi115 conmon[90393]: 2022-05-14T02:26:38.736+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:26:39.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:26:39 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:26:39.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:26:39 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:26:40.063 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:26:39 smithi115 conmon[90393]: 2022-05-14T02:26:39.760+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:26:40.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:26:40 smithi115 ceph-mon[78593]: pgmap v5718: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:26:40.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:26:40 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:26:40.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:26:40 smithi047 ceph-mon[77147]: pgmap v5718: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:26:40.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:26:40 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:26:41.067 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:26:40 smithi115 conmon[90393]: 2022-05-14T02:26:40.754+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:26:41.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:26:41 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:26:41.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:26:41 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:26:42.069 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:26:41 smithi115 conmon[90393]: 2022-05-14T02:26:41.769+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:26:42.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:26:42 smithi115 ceph-mon[78593]: pgmap v5719: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:26:42.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:26:42 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:26:42.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:26:42 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 10735 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:26:42.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:26:42 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:26:42.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:26:42 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:26:42.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:26:42 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:26:42.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:26:42 smithi047 ceph-mon[77147]: pgmap v5719: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:26:42.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:26:42 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:26:42.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:26:42 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 10735 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:26:42.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:26:42 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:26:42.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:26:42 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:26:42.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:26:42 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:26:43.074 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:26:42 smithi115 conmon[90393]: 2022-05-14T02:26:42.737+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:26:43.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:26:43 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:26:43.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:26:43 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:26:44.076 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:26:43 smithi115 conmon[90393]: 2022-05-14T02:26:43.768+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:26:44.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:26:44 smithi115 ceph-mon[78593]: pgmap v5720: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:26:44.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:26:44 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:26:44.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:26:44 smithi047 ceph-mon[77147]: pgmap v5720: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:26:44.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:26:44 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:26:45.080 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:26:44 smithi115 conmon[90393]: 2022-05-14T02:26:44.791+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:26:45.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:26:45 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:26:45.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:26:45 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:26:46.082 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:26:45 smithi115 conmon[90393]: 2022-05-14T02:26:45.781+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:26:46.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:26:46 smithi115 ceph-mon[78593]: pgmap v5721: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:26:46.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:26:46 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:26:46.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:26:46 smithi047 ceph-mon[77147]: pgmap v5721: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:26:46.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:26:46 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:26:47.086 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:26:46 smithi115 conmon[90393]: 2022-05-14T02:26:46.815+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:26:47.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:26:47 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:26:47.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:26:47 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:26:48.088 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:26:47 smithi115 conmon[90393]: 2022-05-14T02:26:47.819+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:26:48.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:26:48 smithi115 ceph-mon[78593]: pgmap v5722: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:26:48.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:26:48 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:26:48.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:26:48 smithi047 ceph-mon[77147]: pgmap v5722: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:26:48.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:26:48 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:26:49.093 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:26:48 smithi115 conmon[90393]: 2022-05-14T02:26:48.785+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:26:49.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:26:49 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:26:49.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:26:49 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:26:50.094 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:26:49 smithi115 conmon[90393]: 2022-05-14T02:26:49.740+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:26:50.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:26:50 smithi115 ceph-mon[78593]: pgmap v5723: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:26:50.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:26:50 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:26:50.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:26:50 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:26:50.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:26:50 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:26:50.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:26:50 smithi047 ceph-mon[77147]: pgmap v5723: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:26:50.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:26:50 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:26:50.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:26:50 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:26:50.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:26:50 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:26:51.099 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:26:50 smithi115 conmon[90393]: 2022-05-14T02:26:50.714+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:26:51.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:26:51 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 10740 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:26:51.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:26:51 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:26:51.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:26:51 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 10740 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:26:51.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:26:51 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:26:52.101 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:26:51 smithi115 conmon[90393]: 2022-05-14T02:26:51.733+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:26:52.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:26:52 smithi115 ceph-mon[78593]: pgmap v5724: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:26:52.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:26:52 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:26:52.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:26:52 smithi047 ceph-mon[77147]: pgmap v5724: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:26:52.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:26:52 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:26:53.106 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:26:52 smithi115 conmon[90393]: 2022-05-14T02:26:52.772+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:26:53.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:26:53 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:26:53.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:26:53 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:26:54.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:26:53 smithi115 conmon[90393]: 2022-05-14T02:26:53.789+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:26:54.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:26:54 smithi115 ceph-mon[78593]: pgmap v5725: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:26:54.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:26:54 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:26:54.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:26:54 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:26:54.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:26:54 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:26:54.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:26:54 smithi047 ceph-mon[77147]: pgmap v5725: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:26:54.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:26:54 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:26:54.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:26:54 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:26:54.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:26:54 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:26:55.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:26:54 smithi115 conmon[90393]: 2022-05-14T02:26:54.827+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:26:55.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:26:55 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:26:55.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:26:55 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:26:56.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:26:55 smithi115 conmon[90393]: 2022-05-14T02:26:55.821+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:26:56.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:26:56 smithi047 ceph-mon[77147]: pgmap v5726: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:26:56.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:26:56 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 10750 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:26:56.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:26:56 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:26:56.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:26:56 smithi115 ceph-mon[78593]: pgmap v5726: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:26:56.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:26:56 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 10750 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:26:56.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:26:56 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:26:57.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:26:56 smithi115 conmon[90393]: 2022-05-14T02:26:56.853+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:26:57.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:26:57 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:26:57.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:26:57 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:26:58.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:26:57 smithi115 conmon[90393]: 2022-05-14T02:26:57.875+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:26:58.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:26:58 smithi115 ceph-mon[78593]: pgmap v5727: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:26:58.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:26:58 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:26:58.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:26:58 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:26:58.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:26:58 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:26:58.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:26:58 smithi047 ceph-mon[77147]: pgmap v5727: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:26:58.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:26:58 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:26:58.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:26:58 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:26:58.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:26:58 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:26:59.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:26:58 smithi115 conmon[90393]: 2022-05-14T02:26:58.848+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:26:59.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:26:59 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:26:59.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:26:59 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:27:00.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:26:59 smithi115 conmon[90393]: 2022-05-14T02:26:59.828+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:27:00.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:27:00 smithi047 ceph-mon[77147]: pgmap v5728: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:27:00.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:27:00 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:27:00.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:27:00 smithi115 ceph-mon[78593]: pgmap v5728: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:27:00.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:27:00 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:27:01.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:27:00 smithi115 conmon[90393]: 2022-05-14T02:27:00.848+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:27:02.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:27:01 smithi047 ceph-mon[77147]: pgmap v5729: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:27:02.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:27:01 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:27:02.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:27:01 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 10755 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:27:02.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:27:01 smithi115 conmon[90393]: 2022-05-14T02:27:01.828+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:27:02.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:27:01 smithi115 ceph-mon[78593]: pgmap v5729: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:27:02.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:27:01 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:27:02.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:27:01 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 10755 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:27:03.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:27:02 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:27:03.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:27:02 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:27:03.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:27:02 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:27:03.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:27:02 smithi115 conmon[90393]: 2022-05-14T02:27:02.783+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:27:03.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:27:02 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:27:03.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:27:02 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:27:03.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:27:02 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:27:04.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:27:03 smithi047 ceph-mon[77147]: pgmap v5730: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:27:04.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:27:03 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:27:04.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:27:03 smithi115 conmon[90393]: 2022-05-14T02:27:03.782+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:27:04.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:27:03 smithi115 ceph-mon[78593]: pgmap v5730: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:27:04.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:27:03 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:27:05.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:27:04 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:27:05.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:27:04 smithi115 conmon[90393]: 2022-05-14T02:27:04.832+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:27:05.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:27:04 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:27:06.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:27:05 smithi047 ceph-mon[77147]: pgmap v5731: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:27:06.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:27:05 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:27:06.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:27:05 smithi115 conmon[90393]: 2022-05-14T02:27:05.829+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:27:06.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:27:05 smithi115 ceph-mon[78593]: pgmap v5731: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:27:06.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:27:05 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:27:07.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:27:06 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:27:07.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:27:06 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:27:07.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:27:06 smithi115 conmon[90393]: 2022-05-14T02:27:06.834+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:27:08.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:27:07 smithi047 ceph-mon[77147]: pgmap v5732: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:27:08.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:27:07 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:27:08.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:27:07 smithi115 conmon[90393]: 2022-05-14T02:27:07.831+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:27:08.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:27:07 smithi115 ceph-mon[78593]: pgmap v5732: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:27:08.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:27:07 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:27:09.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:27:08 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:27:09.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:27:08 smithi115 conmon[90393]: 2022-05-14T02:27:08.804+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:27:09.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:27:08 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:27:10.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:27:09 smithi047 ceph-mon[77147]: pgmap v5733: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:27:10.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:27:09 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:27:10.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:27:09 smithi115 conmon[90393]: 2022-05-14T02:27:09.770+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:27:10.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:27:09 smithi115 ceph-mon[78593]: pgmap v5733: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:27:10.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:27:09 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:27:11.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:27:10 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:27:11.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:27:10 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:27:11.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:27:10 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:27:11.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:27:10 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 10760 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:27:11.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:27:10 smithi115 conmon[90393]: 2022-05-14T02:27:10.727+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:27:11.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:27:10 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:27:11.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:27:10 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:27:11.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:27:10 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:27:11.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:27:10 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 10760 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:27:12.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:27:11 smithi047 ceph-mon[77147]: pgmap v5734: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:27:12.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:27:11 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:27:12.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:27:11 smithi115 conmon[90393]: 2022-05-14T02:27:11.761+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:27:12.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:27:11 smithi115 ceph-mon[78593]: pgmap v5734: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:27:12.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:27:11 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:27:13.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:27:12 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:27:13.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:27:12 smithi115 conmon[90393]: 2022-05-14T02:27:12.768+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:27:13.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:27:12 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:27:14.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:27:13 smithi047 ceph-mon[77147]: pgmap v5735: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:27:14.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:27:13 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:27:14.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:27:13 smithi115 conmon[90393]: 2022-05-14T02:27:13.747+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:27:14.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:27:13 smithi115 ceph-mon[78593]: pgmap v5735: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:27:14.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:27:13 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:27:15.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:27:14 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:27:15.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:27:14 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:27:15.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:27:14 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:27:15.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:27:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T02:27:15.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:27:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T02:27:15.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:27:14 smithi115 conmon[90393]: 2022-05-14T02:27:14.759+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:27:15.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:27:14 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:27:15.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:27:14 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:27:15.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:27:14 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:27:15.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:27:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T02:27:15.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:27:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T02:27:16.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:27:15 smithi047 ceph-mon[77147]: pgmap v5736: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:27:16.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:27:15 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:27:16.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:27:15 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 10770 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:27:16.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:27:15 smithi115 conmon[90393]: 2022-05-14T02:27:15.792+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:27:16.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:27:15 smithi115 ceph-mon[78593]: pgmap v5736: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:27:16.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:27:15 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:27:16.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:27:15 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 10770 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:27:17.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:27:16 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:27:17.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:27:16 smithi115 conmon[90393]: 2022-05-14T02:27:16.839+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:27:17.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:27:16 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:27:18.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:27:17 smithi047 ceph-mon[77147]: pgmap v5737: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:27:18.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:27:17 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:27:18.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:27:17 smithi115 conmon[90393]: 2022-05-14T02:27:17.868+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:27:18.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:27:17 smithi115 ceph-mon[78593]: pgmap v5737: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:27:18.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:27:17 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:27:19.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:27:18 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:27:19.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:27:18 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:27:19.048 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:27:18 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:27:19.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:27:18 smithi115 conmon[90393]: 2022-05-14T02:27:18.877+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:27:19.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:27:18 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:27:19.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:27:18 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:27:19.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:27:18 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:27:20.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:27:19 smithi047 ceph-mon[77147]: pgmap v5738: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:27:20.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:27:19 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:27:20.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:27:19 smithi115 conmon[90393]: 2022-05-14T02:27:19.831+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:27:20.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:27:19 smithi115 ceph-mon[78593]: pgmap v5738: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:27:20.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:27:19 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:27:21.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:27:20 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:27:21.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:27:20 smithi115 conmon[90393]: 2022-05-14T02:27:20.783+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:27:21.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:27:20 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:27:22.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:27:21 smithi047 ceph-mon[77147]: pgmap v5739: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:27:22.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:27:21 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 10775 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:27:22.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:27:21 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:27:22.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:27:21 smithi115 conmon[90393]: 2022-05-14T02:27:21.766+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:27:22.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:27:21 smithi115 ceph-mon[78593]: pgmap v5739: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:27:22.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:27:21 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 10775 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:27:22.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:27:21 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:27:23.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:27:22 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:27:23.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:27:22 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:27:23.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:27:22 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:27:23.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:27:22 smithi115 conmon[90393]: 2022-05-14T02:27:22.781+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:27:23.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:27:22 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:27:23.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:27:22 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:27:23.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:27:22 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:27:24.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:27:23 smithi047 ceph-mon[77147]: pgmap v5740: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:27:24.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:27:23 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:27:24.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:27:23 smithi115 conmon[90393]: 2022-05-14T02:27:23.829+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:27:24.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:27:23 smithi115 ceph-mon[78593]: pgmap v5740: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:27:24.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:27:23 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:27:25.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:27:24 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:27:25.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:27:24 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:27:25.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:27:24 smithi115 conmon[90393]: 2022-05-14T02:27:24.809+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:27:26.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:27:25 smithi047 ceph-mon[77147]: pgmap v5741: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:27:26.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:27:25 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:27:26.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:27:25 smithi115 conmon[90393]: 2022-05-14T02:27:25.800+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:27:26.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:27:25 smithi115 ceph-mon[78593]: pgmap v5741: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:27:26.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:27:25 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:27:27.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:27:26 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:27:27.114 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:27:26 smithi115 conmon[90393]: 2022-05-14T02:27:26.793+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:27:27.114 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:27:26 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:27:28.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:27:27 smithi047 ceph-mon[77147]: pgmap v5742: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:27:28.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:27:27 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:27:28.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:27:27 smithi115 conmon[90393]: 2022-05-14T02:27:27.760+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:27:28.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:27:27 smithi115 ceph-mon[78593]: pgmap v5742: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:27:28.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:27:27 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:27:29.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:27:28 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:27:29.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:27:28 smithi115 conmon[90393]: 2022-05-14T02:27:28.735+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:27:29.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:27:28 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:27:29.948 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:27:29 smithi115 ceph-mon[78593]: pgmap v5743: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:27:29.948 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:27:29 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:27:29.948 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:27:29 smithi115 conmon[90393]: 2022-05-14T02:27:29.695+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:27:30.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:27:29 smithi047 ceph-mon[77147]: pgmap v5743: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:27:30.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:27:29 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:27:31.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:27:30 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:27:31.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:27:30 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:27:31.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:27:30 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:27:31.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:27:30 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T02:27:31.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:27:30 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T02:27:31.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:27:30 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T02:27:31.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:27:30 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 10780 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:27:31.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:27:30 smithi115 conmon[90393]: 2022-05-14T02:27:30.697+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:27:31.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:27:30 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:27:31.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:27:30 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:27:31.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:27:30 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:27:31.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:27:30 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T02:27:31.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:27:30 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T02:27:31.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:27:30 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T02:27:31.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:27:30 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 10780 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:27:32.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:27:31 smithi047 ceph-mon[77147]: pgmap v5744: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:27:32.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:27:31 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:27:32.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:27:31 smithi115 conmon[90393]: 2022-05-14T02:27:31.727+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:27:32.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:27:31 smithi115 ceph-mon[78593]: pgmap v5744: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:27:32.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:27:31 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:27:33.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:27:32 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:27:33.042 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:27:32 smithi047 conmon[78521]: 2022-05-14T02:27:32.976+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:27:33.042 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:27:32 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T02:27:33.042 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:27:32 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:27:33.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:27:32 smithi047 conmon[78521]: yield 2022-05-14T02:27:33.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:27:32 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:27:33.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:27:32 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:27:33.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:27:32 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:27:33.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:27:32 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T02:27:33.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:27:32 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:27:33.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:27:32 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T02:27:33.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:27:32 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:27:33.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:27:32 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T02:27:33.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:27:32 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:27:33.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:27:32 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T02:27:33.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:27:32 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:27:33.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:27:32 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T02:27:33.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:27:32 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:27:33.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:27:32 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:27:33.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:27:32 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:27:33.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:27:32 smithi115 conmon[90393]: 2022-05-14T02:27:32.699+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:27:33.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:27:32 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:27:34.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:27:33 smithi115 conmon[90393]: 2022-05-14T02:27:33.739+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:27:34.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:27:33 smithi115 ceph-mon[78593]: pgmap v5745: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:27:34.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:27:33 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:27:34.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:27:33 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:27:34.109 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T02:27:34.109 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:27:34.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T02:27:34.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:27:34.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:27:34.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:27:34.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T02:27:34.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:27:34.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T02:27:34.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:27:34.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T02:27:34.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:27:34.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T02:27:34.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:27:34.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T02:27:34.113 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:27:34.113 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:27:34.113 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:27:34.113 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:27:33 smithi115 ceph-mon[78593]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T02:27:34.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:27:33 smithi047 ceph-mon[77147]: pgmap v5745: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:27:34.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:27:33 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:27:34.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:27:33 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:27:34.292 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T02:27:34.292 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:27:34.292 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T02:27:34.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:27:34.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:27:34.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:27:34.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T02:27:34.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:27:34.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T02:27:34.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:27:34.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T02:27:34.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:27:34.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T02:27:34.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:27:34.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T02:27:34.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:27:34.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:27:34.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:27:34.296 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:27:33 smithi047 ceph-mon[77147]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T02:27:35.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:27:34 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:27:35.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:27:34 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:27:35.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:27:34 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:27:35.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:27:34 smithi115 conmon[90393]: 2022-05-14T02:27:34.759+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:27:35.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:27:34 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:27:35.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:27:34 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:27:35.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:27:34 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:27:36.049 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:27:35 smithi047 ceph-mon[77147]: pgmap v5746: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:27:36.050 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:27:35 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:27:36.050 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:27:35 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 10790 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:27:36.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:27:35 smithi115 conmon[90393]: 2022-05-14T02:27:35.753+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:27:36.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:27:35 smithi115 ceph-mon[78593]: pgmap v5746: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:27:36.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:27:35 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:27:36.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:27:35 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 10790 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:27:36.542 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:27:36 smithi047 conmon[78521]: 2022-05-14T02:27:36.047+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:27:36.542 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:27:36 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T02:27:36.542 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:27:36 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:27:36.542 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:27:36 smithi047 conmon[78521]: yield 2022-05-14T02:27:36.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:27:36 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:27:36.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:27:36 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:27:36.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:27:36 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:27:36.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:27:36 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T02:27:36.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:27:36 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:27:36.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:27:36 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T02:27:36.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:27:36 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:27:36.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:27:36 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T02:27:36.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:27:36 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:27:36.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:27:36 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T02:27:36.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:27:36 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:27:36.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:27:36 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T02:27:36.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:27:36 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:27:36.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:27:36 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:27:36.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:27:36 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:27:36.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:27:36 smithi047 conmon[78521]: 2022-05-14T02:27:36.048+0000 7f5345f84700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T02:27:36.547 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:27:36 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T02:27:36.547 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:27:36 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:27:36.547 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:27:36 smithi047 conmon[78521]: yield 2022-05-14T02:27:36.547 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:27:36 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:27:36.548 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:27:36 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:27:36.548 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:27:36 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:27:36.548 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:27:36 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T02:27:36.548 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:27:36 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:27:36.549 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:27:36 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T02:27:36.549 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:27:36 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:27:36.549 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:27:36 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T02:27:36.549 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:27:36 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:27:36.550 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:27:36 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T02:27:36.550 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:27:36 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:27:36.550 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:27:36 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T02:27:36.550 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:27:36 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:27:36.551 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:27:36 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:27:36.551 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:27:36 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:27:36.551 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:27:36 smithi047 conmon[78521]: 2022-05-14T02:27:36.551 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:27:36 smithi047 conmon[78521]: During handling of the above exception, another exception occurred: 2022-05-14T02:27:36.551 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:27:36 smithi047 conmon[78521]: 2022-05-14T02:27:36.552 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:27:36 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T02:27:36.552 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:27:36 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T02:27:36.552 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:27:36 smithi047 conmon[78521]: return f(*arg) 2022-05-14T02:27:36.552 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:27:36 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T02:27:36.553 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:27:36 smithi047 conmon[78521]: self._write_client_files(client_files, host) 2022-05-14T02:27:36.553 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:27:36 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T02:27:36.553 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:27:36 smithi047 conmon[78521]: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T02:27:36.553 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:27:36 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T02:27:36.554 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:27:36 smithi047 conmon[78521]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T02:27:36.554 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:27:36 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T02:27:36.554 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:27:36 smithi047 conmon[78521]: return self.event_loop.get_result(coro) 2022-05-14T02:27:36.554 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:27:36 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T02:27:36.554 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:27:36 smithi047 conmon[78521]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T02:27:36.555 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:27:36 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T02:27:36.555 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:27:36 smithi047 conmon[78521]: return self.__get_result() 2022-05-14T02:27:36.555 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:27:36 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T02:27:36.555 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:27:36 smithi047 conmon[78521]: raise self._exception 2022-05-14T02:27:36.555 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:27:36 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T02:27:36.556 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:27:36 smithi047 conmon[78521]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T02:27:36.556 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:27:36 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T02:27:36.556 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:27:36 smithi047 conmon[78521]: conn = await self._remote_connection(host, addr) 2022-05-14T02:27:36.556 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:27:36 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T02:27:36.557 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:27:36 smithi047 conmon[78521]: raise 2022-05-14T02:27:36.557 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:27:36 smithi047 conmon[78521]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T02:27:36.557 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:27:36 smithi047 conmon[78521]: self.gen.throw(type, value, traceback) 2022-05-14T02:27:36.558 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:27:36 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T02:27:36.558 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:27:36 smithi047 conmon[78521]: raise OrchestratorError(msg) 2022-05-14T02:27:36.558 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:27:36 smithi047 conmon[78521]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:27:37.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:27:36 smithi115 conmon[90393]: 2022-05-14T02:27:36.770+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:27:37.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:27:36 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:27:37.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:27:36 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:27:37.109 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T02:27:37.109 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:27:37.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T02:27:37.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:27:37.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:27:37.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:27:37.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T02:27:37.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:27:37.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T02:27:37.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:27:37.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T02:27:37.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:27:37.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T02:27:37.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:27:37.113 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T02:27:37.113 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:27:37.113 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:27:37.113 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:27:37.114 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:27:36 smithi115 ceph-mon[78593]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T02:27:37.114 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T02:27:37.114 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:27:37.114 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T02:27:37.114 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:27:37.115 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:27:37.115 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:27:37.115 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T02:27:37.115 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:27:37.116 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T02:27:37.116 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:27:37.116 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T02:27:37.116 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:27:37.117 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T02:27:37.117 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:27:37.117 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T02:27:37.117 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:27:37.117 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:27:37.118 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:27:37.118 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T02:27:37.118 INFO:journalctl@ceph.mon.c.smithi115.stdout: During handling of the above exception, another exception occurred: 2022-05-14T02:27:37.118 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T02:27:37.119 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T02:27:37.119 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T02:27:37.119 INFO:journalctl@ceph.mon.c.smithi115.stdout: return f(*arg) 2022-05-14T02:27:37.119 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T02:27:37.119 INFO:journalctl@ceph.mon.c.smithi115.stdout: self._write_client_files(client_files, host) 2022-05-14T02:27:37.120 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T02:27:37.120 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T02:27:37.120 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T02:27:37.120 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T02:27:37.120 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T02:27:37.121 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.event_loop.get_result(coro) 2022-05-14T02:27:37.121 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T02:27:37.121 INFO:journalctl@ceph.mon.c.smithi115.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T02:27:37.121 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T02:27:37.122 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.__get_result() 2022-05-14T02:27:37.122 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T02:27:37.122 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise self._exception 2022-05-14T02:27:37.122 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T02:27:37.123 INFO:journalctl@ceph.mon.c.smithi115.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T02:27:37.123 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T02:27:37.123 INFO:journalctl@ceph.mon.c.smithi115.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T02:27:37.123 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T02:27:37.123 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise 2022-05-14T02:27:37.124 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T02:27:37.124 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.gen.throw(type, value, traceback) 2022-05-14T02:27:37.124 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T02:27:37.124 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OrchestratorError(msg) 2022-05-14T02:27:37.125 INFO:journalctl@ceph.mon.c.smithi115.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:27:37.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:27:36 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:27:37.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:27:36 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:27:37.292 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T02:27:37.292 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:27:37.292 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T02:27:37.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:27:37.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:27:37.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:27:37.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T02:27:37.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:27:37.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T02:27:37.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:27:37.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T02:27:37.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:27:37.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T02:27:37.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:27:37.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T02:27:37.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:27:37.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:27:37.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:27:37.296 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:27:36 smithi047 ceph-mon[77147]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T02:27:37.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T02:27:37.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:27:37.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T02:27:37.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:27:37.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:27:37.298 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:27:37.298 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T02:27:37.298 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:27:37.298 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T02:27:37.298 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:27:37.299 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T02:27:37.299 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:27:37.299 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T02:27:37.299 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:27:37.300 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T02:27:37.300 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:27:37.300 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:27:37.300 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:27:37.300 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T02:27:37.301 INFO:journalctl@ceph.mon.b.smithi047.stdout: During handling of the above exception, another exception occurred: 2022-05-14T02:27:37.301 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T02:27:37.301 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T02:27:37.301 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T02:27:37.301 INFO:journalctl@ceph.mon.b.smithi047.stdout: return f(*arg) 2022-05-14T02:27:37.302 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T02:27:37.302 INFO:journalctl@ceph.mon.b.smithi047.stdout: self._write_client_files(client_files, host) 2022-05-14T02:27:37.302 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T02:27:37.302 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T02:27:37.303 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T02:27:37.303 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T02:27:37.303 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T02:27:37.303 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.event_loop.get_result(coro) 2022-05-14T02:27:37.303 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T02:27:37.304 INFO:journalctl@ceph.mon.b.smithi047.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T02:27:37.304 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T02:27:37.304 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.__get_result() 2022-05-14T02:27:37.305 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T02:27:37.305 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise self._exception 2022-05-14T02:27:37.305 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T02:27:37.305 INFO:journalctl@ceph.mon.b.smithi047.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T02:27:37.306 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T02:27:37.306 INFO:journalctl@ceph.mon.b.smithi047.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T02:27:37.306 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T02:27:37.306 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise 2022-05-14T02:27:37.307 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T02:27:37.307 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.gen.throw(type, value, traceback) 2022-05-14T02:27:37.307 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T02:27:37.307 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OrchestratorError(msg) 2022-05-14T02:27:37.307 INFO:journalctl@ceph.mon.b.smithi047.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:27:38.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:27:37 smithi115 conmon[90393]: 2022-05-14T02:27:37.739+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:27:38.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:27:37 smithi115 ceph-mon[78593]: pgmap v5747: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:27:38.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:27:37 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:27:38.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:27:37 smithi047 ceph-mon[77147]: pgmap v5747: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:27:38.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:27:37 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:27:39.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:27:38 smithi115 conmon[90393]: 2022-05-14T02:27:38.760+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:27:39.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:27:38 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:27:39.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:27:38 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:27:39.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:27:38 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:27:39.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:27:38 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:27:39.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:27:38 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:27:39.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:27:38 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:27:40.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:27:39 smithi115 conmon[90393]: 2022-05-14T02:27:39.712+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:27:40.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:27:39 smithi115 ceph-mon[78593]: pgmap v5748: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:27:40.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:27:39 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:27:40.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:27:39 smithi047 ceph-mon[77147]: pgmap v5748: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:27:40.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:27:39 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:27:41.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:27:40 smithi115 conmon[90393]: 2022-05-14T02:27:40.684+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:27:41.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:27:40 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:27:41.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:27:40 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:27:42.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:27:41 smithi115 conmon[90393]: 2022-05-14T02:27:41.702+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:27:42.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:27:41 smithi115 ceph-mon[78593]: pgmap v5749: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:27:42.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:27:41 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:27:42.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:27:41 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 10795 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:27:42.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:27:41 smithi047 ceph-mon[77147]: pgmap v5749: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:27:42.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:27:41 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:27:42.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:27:41 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 10795 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:27:43.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:27:42 smithi115 conmon[90393]: 2022-05-14T02:27:42.693+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:27:43.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:27:42 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:27:43.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:27:42 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:27:43.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:27:42 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:27:43.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:27:42 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:27:43.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:27:42 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:27:43.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:27:42 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:27:44.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:27:43 smithi115 conmon[90393]: 2022-05-14T02:27:43.710+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:27:44.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:27:43 smithi115 ceph-mon[78593]: pgmap v5750: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:27:44.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:27:43 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:27:44.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:27:43 smithi047 ceph-mon[77147]: pgmap v5750: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:27:44.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:27:43 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:27:45.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:27:44 smithi115 conmon[90393]: 2022-05-14T02:27:44.670+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:27:45.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:27:44 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:27:45.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:27:44 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:27:46.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:27:45 smithi115 conmon[90393]: 2022-05-14T02:27:45.686+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:27:46.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:27:45 smithi115 ceph-mon[78593]: pgmap v5751: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:27:46.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:27:45 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:27:46.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:27:45 smithi047 ceph-mon[77147]: pgmap v5751: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:27:46.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:27:45 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:27:47.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:27:46 smithi115 conmon[90393]: 2022-05-14T02:27:46.715+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:27:47.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:27:46 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:27:47.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:27:46 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:27:48.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:27:47 smithi115 conmon[90393]: 2022-05-14T02:27:47.669+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:27:48.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:27:47 smithi115 ceph-mon[78593]: pgmap v5752: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:27:48.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:27:47 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:27:48.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:27:47 smithi047 ceph-mon[77147]: pgmap v5752: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:27:48.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:27:47 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:27:49.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:27:48 smithi115 conmon[90393]: 2022-05-14T02:27:48.666+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:27:49.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:27:48 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:27:49.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:27:48 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:27:50.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:27:49 smithi115 conmon[90393]: 2022-05-14T02:27:49.636+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:27:50.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:27:49 smithi115 ceph-mon[78593]: pgmap v5753: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:27:50.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:27:49 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:27:50.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:27:49 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:27:50.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:27:49 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:27:50.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:27:49 smithi047 ceph-mon[77147]: pgmap v5753: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:27:50.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:27:49 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:27:50.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:27:49 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:27:50.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:27:49 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:27:51.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:27:50 smithi115 conmon[90393]: 2022-05-14T02:27:50.632+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:27:51.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:27:50 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:27:51.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:27:50 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 10800 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:27:51.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:27:50 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:27:51.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:27:50 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 10800 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:27:52.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:27:51 smithi115 conmon[90393]: 2022-05-14T02:27:51.653+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:27:52.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:27:51 smithi115 ceph-mon[78593]: pgmap v5754: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:27:52.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:27:51 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:27:52.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:27:51 smithi047 ceph-mon[77147]: pgmap v5754: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:27:52.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:27:51 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:27:53.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:27:52 smithi115 conmon[90393]: 2022-05-14T02:27:52.665+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:27:53.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:27:52 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:27:53.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:27:52 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:27:54.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:27:53 smithi115 conmon[90393]: 2022-05-14T02:27:53.646+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:27:54.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:27:53 smithi115 ceph-mon[78593]: pgmap v5755: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:27:54.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:27:53 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:27:54.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:27:53 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:27:54.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:27:53 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:27:54.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:27:53 smithi047 ceph-mon[77147]: pgmap v5755: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:27:54.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:27:53 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:27:54.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:27:53 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:27:54.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:27:53 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:27:55.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:27:54 smithi115 conmon[90393]: 2022-05-14T02:27:54.659+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:27:55.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:27:54 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:27:55.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:27:54 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:27:56.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:27:55 smithi115 conmon[90393]: 2022-05-14T02:27:55.688+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:27:56.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:27:55 smithi115 ceph-mon[78593]: pgmap v5756: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:27:56.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:27:55 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:27:56.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:27:55 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 10810 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:27:56.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:27:55 smithi047 ceph-mon[77147]: pgmap v5756: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:27:56.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:27:55 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:27:56.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:27:55 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 10810 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:27:57.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:27:56 smithi115 conmon[90393]: 2022-05-14T02:27:56.671+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:27:57.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:27:56 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:27:57.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:27:56 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:27:58.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:27:57 smithi115 conmon[90393]: 2022-05-14T02:27:57.663+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:27:58.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:27:57 smithi115 ceph-mon[78593]: pgmap v5757: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:27:58.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:27:57 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:27:58.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:27:57 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:27:58.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:27:57 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:27:58.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:27:57 smithi047 ceph-mon[77147]: pgmap v5757: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:27:58.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:27:57 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:27:58.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:27:57 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:27:58.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:27:57 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:27:58.887 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:27:58 smithi115 conmon[90393]: 2022-05-14T02:27:58.632+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:27:59.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:27:58 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:27:59.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:27:58 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:27:59.888 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:27:59 smithi115 conmon[90393]: 2022-05-14T02:27:59.607+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:28:00.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:27:59 smithi047 ceph-mon[77147]: pgmap v5758: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:28:00.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:27:59 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:28:00.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:27:59 smithi115 ceph-mon[78593]: pgmap v5758: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:28:00.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:27:59 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:28:00.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:28:00 smithi115 conmon[90393]: 2022-05-14T02:28:00.596+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:28:01.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:28:00 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:28:01.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:28:00 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:28:01.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:28:01 smithi115 conmon[90393]: 2022-05-14T02:28:01.555+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:28:02.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:28:01 smithi047 ceph-mon[77147]: pgmap v5759: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:28:02.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:28:01 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:28:02.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:28:01 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 10815 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:28:02.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:28:01 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:28:02.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:28:01 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:28:02.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:28:01 smithi115 ceph-mon[78593]: pgmap v5759: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:28:02.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:28:01 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:28:02.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:28:01 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 10815 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:28:02.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:28:01 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:28:02.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:28:01 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:28:02.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:28:02 smithi115 conmon[90393]: 2022-05-14T02:28:02.536+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:28:03.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:28:02 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:28:03.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:28:02 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:28:03.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:28:03 smithi115 conmon[90393]: 2022-05-14T02:28:03.518+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:28:04.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:28:03 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:28:04.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:28:03 smithi047 ceph-mon[77147]: pgmap v5760: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:28:04.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:28:03 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:28:04.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:28:03 smithi115 ceph-mon[78593]: pgmap v5760: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:28:04.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:28:04 smithi115 conmon[90393]: 2022-05-14T02:28:04.475+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:28:05.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:28:04 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:28:05.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:28:04 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:28:05.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:28:05 smithi115 conmon[90393]: 2022-05-14T02:28:05.517+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:28:06.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:28:05 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:28:06.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:28:05 smithi047 ceph-mon[77147]: pgmap v5761: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:28:06.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:28:05 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:28:06.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:28:05 smithi115 ceph-mon[78593]: pgmap v5761: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:28:06.860 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:28:06 smithi115 conmon[90393]: 2022-05-14T02:28:06.525+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:28:07.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:28:06 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:28:07.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:28:06 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:28:07.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:28:07 smithi115 conmon[90393]: 2022-05-14T02:28:07.571+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:28:08.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:28:07 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:28:08.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:28:07 smithi047 ceph-mon[77147]: pgmap v5762: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:28:08.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:28:07 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:28:08.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:28:07 smithi115 ceph-mon[78593]: pgmap v5762: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:28:08.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:28:08 smithi115 conmon[90393]: 2022-05-14T02:28:08.526+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:28:09.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:28:08 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:28:09.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:28:08 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:28:09.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:28:09 smithi115 conmon[90393]: 2022-05-14T02:28:09.531+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:28:10.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:28:09 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:28:10.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:28:09 smithi047 ceph-mon[77147]: pgmap v5763: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:28:10.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:28:09 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:28:10.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:28:09 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:28:10.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:28:09 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:28:10.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:28:09 smithi115 ceph-mon[78593]: pgmap v5763: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:28:10.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:28:09 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:28:10.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:28:09 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:28:10.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:28:10 smithi115 conmon[90393]: 2022-05-14T02:28:10.523+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:28:11.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:28:10 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:28:11.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:28:10 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 10820 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:28:11.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:28:10 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:28:11.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:28:10 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 10820 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:28:11.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:28:11 smithi115 conmon[90393]: 2022-05-14T02:28:11.503+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:28:12.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:28:11 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:28:12.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:28:11 smithi047 ceph-mon[77147]: pgmap v5764: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:28:12.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:28:11 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:28:12.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:28:11 smithi115 ceph-mon[78593]: pgmap v5764: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:28:12.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:28:12 smithi115 conmon[90393]: 2022-05-14T02:28:12.510+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:28:13.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:28:12 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:28:13.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:28:12 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:28:13.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:28:13 smithi115 conmon[90393]: 2022-05-14T02:28:13.547+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:28:14.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:28:13 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:28:14.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:28:13 smithi047 ceph-mon[77147]: pgmap v5765: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:28:14.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:28:13 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:28:14.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:28:13 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:28:14.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:28:13 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:28:14.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:28:13 smithi115 ceph-mon[78593]: pgmap v5765: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:28:14.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:28:13 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:28:14.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:28:13 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:28:14.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:28:14 smithi115 conmon[90393]: 2022-05-14T02:28:14.596+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:28:15.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:28:14 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:28:15.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:28:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T02:28:15.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:28:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T02:28:15.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:28:14 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:28:15.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:28:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T02:28:15.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:28:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T02:28:15.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:28:15 smithi115 conmon[90393]: 2022-05-14T02:28:15.559+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:28:16.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:28:15 smithi047 ceph-mon[77147]: pgmap v5766: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:28:16.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:28:15 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:28:16.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:28:15 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 10830 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:28:16.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:28:15 smithi115 ceph-mon[78593]: pgmap v5766: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:28:16.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:28:15 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:28:16.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:28:15 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 10830 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:28:16.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:28:16 smithi115 conmon[90393]: 2022-05-14T02:28:16.534+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:28:17.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:28:16 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:28:17.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:28:16 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:28:17.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:28:17 smithi115 conmon[90393]: 2022-05-14T02:28:17.569+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:28:18.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:28:17 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:28:18.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:28:17 smithi047 ceph-mon[77147]: pgmap v5767: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:28:18.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:28:17 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:28:18.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:28:17 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:28:18.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:28:17 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:28:18.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:28:17 smithi115 ceph-mon[78593]: pgmap v5767: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:28:18.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:28:17 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:28:18.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:28:17 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:28:18.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:28:18 smithi115 conmon[90393]: 2022-05-14T02:28:18.537+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:28:19.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:28:18 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:28:19.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:28:18 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:28:19.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:28:19 smithi115 conmon[90393]: 2022-05-14T02:28:19.539+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:28:20.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:28:19 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:28:20.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:28:19 smithi047 ceph-mon[77147]: pgmap v5768: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:28:20.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:28:19 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:28:20.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:28:19 smithi115 ceph-mon[78593]: pgmap v5768: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:28:20.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:28:20 smithi115 conmon[90393]: 2022-05-14T02:28:20.580+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:28:21.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:28:20 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:28:21.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:28:20 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:28:21.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:28:21 smithi115 conmon[90393]: 2022-05-14T02:28:21.533+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:28:22.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:28:21 smithi047 ceph-mon[77147]: pgmap v5769: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:28:22.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:28:21 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:28:22.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:28:21 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 10835 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:28:22.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:28:21 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:28:22.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:28:21 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:28:22.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:28:21 smithi115 ceph-mon[78593]: pgmap v5769: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:28:22.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:28:21 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:28:22.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:28:21 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 10835 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:28:22.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:28:21 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:28:22.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:28:21 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:28:22.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:28:22 smithi115 conmon[90393]: 2022-05-14T02:28:22.568+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:28:23.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:28:22 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:28:23.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:28:22 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:28:23.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:28:23 smithi115 conmon[90393]: 2022-05-14T02:28:23.535+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:28:24.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:28:23 smithi047 ceph-mon[77147]: pgmap v5770: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:28:24.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:28:23 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:28:24.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:28:23 smithi115 ceph-mon[78593]: pgmap v5770: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:28:24.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:28:23 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:28:24.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:28:24 smithi115 conmon[90393]: 2022-05-14T02:28:24.534+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:28:25.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:28:24 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:28:25.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:28:24 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:28:25.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:28:25 smithi115 conmon[90393]: 2022-05-14T02:28:25.490+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:28:26.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:28:25 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:28:26.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:28:25 smithi047 ceph-mon[77147]: pgmap v5771: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:28:26.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:28:25 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:28:26.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:28:25 smithi115 ceph-mon[78593]: pgmap v5771: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:28:26.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:28:26 smithi115 conmon[90393]: 2022-05-14T02:28:26.457+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:28:27.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:28:26 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:28:27.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:28:26 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:28:27.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:28:27 smithi115 conmon[90393]: 2022-05-14T02:28:27.435+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:28:28.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:28:27 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:28:28.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:28:27 smithi047 ceph-mon[77147]: pgmap v5772: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:28:28.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:28:27 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:28:28.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:28:27 smithi115 ceph-mon[78593]: pgmap v5772: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:28:28.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:28:28 smithi115 conmon[90393]: 2022-05-14T02:28:28.422+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:28:29.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:28:28 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:28:29.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:28:28 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:28:29.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:28:29 smithi115 conmon[90393]: 2022-05-14T02:28:29.414+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:28:30.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:28:29 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:28:30.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:28:29 smithi047 ceph-mon[77147]: pgmap v5773: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:28:30.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:28:29 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:28:30.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:28:29 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:28:30.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:28:29 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:28:30.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:28:29 smithi115 ceph-mon[78593]: pgmap v5773: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:28:30.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:28:29 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:28:30.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:28:29 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:28:30.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:28:30 smithi115 conmon[90393]: 2022-05-14T02:28:30.395+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:28:31.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:28:30 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:28:31.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:28:30 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 10840 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:28:31.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:28:30 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:28:31.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:28:30 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 10840 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:28:31.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:28:31 smithi115 conmon[90393]: 2022-05-14T02:28:31.439+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:28:32.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:28:31 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:28:32.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:28:31 smithi047 ceph-mon[77147]: pgmap v5774: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:28:32.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:28:31 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:28:32.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:28:31 smithi115 ceph-mon[78593]: pgmap v5774: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:28:32.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:28:32 smithi115 conmon[90393]: 2022-05-14T02:28:32.422+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:28:33.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:28:32 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:28:33.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:28:33 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:28:33.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:28:33 smithi115 conmon[90393]: 2022-05-14T02:28:33.416+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:28:34.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:28:34 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:28:34.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:28:34 smithi047 ceph-mon[77147]: pgmap v5775: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:28:34.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:28:34 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:28:34.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:28:34 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:28:34.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:28:34 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:28:34.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:28:34 smithi115 ceph-mon[78593]: pgmap v5775: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:28:34.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:28:34 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:28:34.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:28:34 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:28:34.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:28:34 smithi115 conmon[90393]: 2022-05-14T02:28:34.388+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:28:35.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:28:35 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:28:35.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:28:35 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:28:35.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:28:35 smithi115 conmon[90393]: 2022-05-14T02:28:35.437+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:28:36.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:28:36 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:28:36.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:28:36 smithi047 ceph-mon[77147]: pgmap v5776: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:28:36.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:28:36 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 10850 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:28:36.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:28:36 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:28:36.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:28:36 smithi115 ceph-mon[78593]: pgmap v5776: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:28:36.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:28:36 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 10850 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:28:36.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:28:36 smithi115 conmon[90393]: 2022-05-14T02:28:36.387+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:28:37.013 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:28:36 smithi047 conmon[78521]: 2022-05-14T02:28:36.591+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:28:37.014 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:28:36 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T02:28:37.014 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:28:36 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:28:37.014 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:28:36 smithi047 conmon[78521]: yield 2022-05-14T02:28:37.014 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:28:36 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:28:37.015 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:28:36 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:28:37.015 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:28:36 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:28:37.015 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:28:36 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T02:28:37.015 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:28:36 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:28:37.015 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:28:36 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T02:28:37.016 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:28:36 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:28:37.016 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:28:36 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T02:28:37.016 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:28:36 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:28:37.016 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:28:36 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T02:28:37.017 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:28:36 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:28:37.017 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:28:36 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T02:28:37.017 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:28:36 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:28:37.017 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:28:36 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:28:37.018 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:28:36 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:28:37.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:28:37 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:28:37.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:28:37 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T02:28:37.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:28:37 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T02:28:37.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:28:37 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T02:28:37.347 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:28:37 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:28:37.347 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:28:37 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T02:28:37.347 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:28:37 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T02:28:37.347 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:28:37 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T02:28:37.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:28:37 smithi115 conmon[90393]: 2022-05-14T02:28:37.345+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:28:38.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:28:38 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:28:38.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:28:38 smithi047 ceph-mon[77147]: pgmap v5777: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:28:38.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:28:38 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:28:38.292 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T02:28:38.292 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:28:38.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T02:28:38.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:28:38.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:28:38.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:28:38.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T02:28:38.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:28:38.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T02:28:38.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:28:38.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T02:28:38.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:28:38.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T02:28:38.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:28:38.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T02:28:38.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:28:38.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:28:38.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:28:38.296 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:28:38 smithi047 ceph-mon[77147]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T02:28:38.296 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:28:38 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:28:38.297 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:28:38 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:28:38.330 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:28:38 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:28:38.330 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:28:38 smithi115 ceph-mon[78593]: pgmap v5777: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:28:38.330 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:28:38 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:28:38.330 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T02:28:38.331 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:28:38.331 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T02:28:38.331 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:28:38.331 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:28:38.332 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:28:38.332 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T02:28:38.332 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:28:38.332 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T02:28:38.332 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:28:38.333 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T02:28:38.333 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:28:38.333 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T02:28:38.333 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:28:38.334 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T02:28:38.334 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:28:38.334 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:28:38.334 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:28:38.334 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:28:38 smithi115 ceph-mon[78593]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T02:28:38.335 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:28:38 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:28:38.335 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:28:38 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:28:38.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:28:38 smithi115 conmon[90393]: 2022-05-14T02:28:38.328+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:28:39.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:28:39 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:28:39.347 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:28:39 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:28:39.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:28:39 smithi115 conmon[90393]: 2022-05-14T02:28:39.346+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:28:40.021 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:28:39 smithi047 conmon[78521]: 2022-05-14T02:28:39.663+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:28:40.022 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:28:39 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T02:28:40.022 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:28:39 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:28:40.022 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:28:39 smithi047 conmon[78521]: yield 2022-05-14T02:28:40.022 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:28:39 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:28:40.023 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:28:39 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:28:40.023 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:28:39 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:28:40.023 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:28:39 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T02:28:40.023 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:28:39 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:28:40.024 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:28:39 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T02:28:40.024 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:28:39 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:28:40.024 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:28:39 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T02:28:40.024 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:28:39 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:28:40.024 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:28:39 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T02:28:40.025 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:28:39 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:28:40.025 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:28:39 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T02:28:40.025 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:28:39 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:28:40.025 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:28:39 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:28:40.026 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:28:39 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:28:40.026 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:28:39 smithi047 conmon[78521]: 2022-05-14T02:28:39.664+0000 7f5340779700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T02:28:40.026 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:28:39 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T02:28:40.026 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:28:39 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:28:40.026 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:28:39 smithi047 conmon[78521]: yield 2022-05-14T02:28:40.027 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:28:39 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:28:40.027 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:28:39 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:28:40.027 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:28:39 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:28:40.027 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:28:39 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T02:28:40.028 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:28:39 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:28:40.028 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:28:39 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T02:28:40.028 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:28:39 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:28:40.028 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:28:39 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T02:28:40.028 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:28:39 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:28:40.029 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:28:39 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T02:28:40.029 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:28:39 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:28:40.029 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:28:39 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T02:28:40.029 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:28:39 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:28:40.029 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:28:39 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:28:40.030 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:28:39 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:28:40.030 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:28:39 smithi047 conmon[78521]: 2022-05-14T02:28:40.030 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:28:39 smithi047 conmon[78521]: During handling of the above exception, another exception occurred: 2022-05-14T02:28:40.030 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:28:39 smithi047 conmon[78521]: 2022-05-14T02:28:40.031 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:28:39 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T02:28:40.031 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:28:39 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T02:28:40.031 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:28:39 smithi047 conmon[78521]: return f(*arg) 2022-05-14T02:28:40.031 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:28:39 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T02:28:40.031 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:28:39 smithi047 conmon[78521]: self._write_client_files(client_files, host) 2022-05-14T02:28:40.032 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:28:39 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T02:28:40.032 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:28:39 smithi047 conmon[78521]: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T02:28:40.032 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:28:39 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T02:28:40.032 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:28:39 smithi047 conmon[78521]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T02:28:40.033 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:28:39 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T02:28:40.033 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:28:39 smithi047 conmon[78521]: return self.event_loop.get_result(coro) 2022-05-14T02:28:40.033 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:28:39 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T02:28:40.033 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:28:39 smithi047 conmon[78521]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T02:28:40.033 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:28:39 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T02:28:40.034 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:28:39 smithi047 conmon[78521]: return self.__get_result() 2022-05-14T02:28:40.034 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:28:39 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T02:28:40.034 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:28:39 smithi047 conmon[78521]: raise self._exception 2022-05-14T02:28:40.034 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:28:39 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T02:28:40.034 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:28:39 smithi047 conmon[78521]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T02:28:40.035 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:28:39 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T02:28:40.035 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:28:39 smithi047 conmon[78521]: conn = await self._remote_connection(host, addr) 2022-05-14T02:28:40.035 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:28:39 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T02:28:40.035 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:28:39 smithi047 conmon[78521]: raise 2022-05-14T02:28:40.036 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:28:39 smithi047 conmon[78521]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T02:28:40.036 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:28:39 smithi047 conmon[78521]: self.gen.throw(type, value, traceback) 2022-05-14T02:28:40.036 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:28:39 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T02:28:40.036 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:28:39 smithi047 conmon[78521]: raise OrchestratorError(msg) 2022-05-14T02:28:40.036 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:28:39 smithi047 conmon[78521]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:28:40.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:28:40 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:28:40.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:28:40 smithi047 ceph-mon[77147]: pgmap v5778: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:28:40.302 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:28:40 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:28:40.302 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:28:40 smithi115 ceph-mon[78593]: pgmap v5778: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:28:40.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:28:40 smithi115 conmon[90393]: 2022-05-14T02:28:40.301+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:28:41.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:28:41 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:28:41.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:28:41 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:28:41.292 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T02:28:41.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:28:41.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T02:28:41.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:28:41.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:28:41.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:28:41.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T02:28:41.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:28:41.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T02:28:41.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:28:41.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T02:28:41.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:28:41.298 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T02:28:41.298 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:28:41.299 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T02:28:41.299 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:28:41.299 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:28:41.300 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:28:41.300 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:28:41 smithi047 ceph-mon[77147]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T02:28:41.300 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T02:28:41.301 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:28:41.301 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T02:28:41.302 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:28:41.302 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:28:41.302 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:28:41.302 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T02:28:41.303 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:28:41.303 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T02:28:41.303 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:28:41.304 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T02:28:41.304 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:28:41.305 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T02:28:41.305 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:28:41.305 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T02:28:41.305 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:28:41.306 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:28:41.306 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:28:41.306 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T02:28:41.306 INFO:journalctl@ceph.mon.b.smithi047.stdout: During handling of the above exception, another exception occurred: 2022-05-14T02:28:41.307 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T02:28:41.307 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T02:28:41.307 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T02:28:41.307 INFO:journalctl@ceph.mon.b.smithi047.stdout: return f(*arg) 2022-05-14T02:28:41.307 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T02:28:41.308 INFO:journalctl@ceph.mon.b.smithi047.stdout: self._write_client_files(client_files, host) 2022-05-14T02:28:41.308 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T02:28:41.308 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T02:28:41.309 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T02:28:41.309 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T02:28:41.309 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T02:28:41.310 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.event_loop.get_result(coro) 2022-05-14T02:28:41.310 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T02:28:41.310 INFO:journalctl@ceph.mon.b.smithi047.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T02:28:41.310 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T02:28:41.311 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.__get_result() 2022-05-14T02:28:41.311 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T02:28:41.312 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise self._exception 2022-05-14T02:28:41.312 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T02:28:41.312 INFO:journalctl@ceph.mon.b.smithi047.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T02:28:41.312 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T02:28:41.312 INFO:journalctl@ceph.mon.b.smithi047.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T02:28:41.313 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T02:28:41.313 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise 2022-05-14T02:28:41.313 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T02:28:41.313 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.gen.throw(type, value, traceback) 2022-05-14T02:28:41.314 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T02:28:41.314 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OrchestratorError(msg) 2022-05-14T02:28:41.314 INFO:journalctl@ceph.mon.b.smithi047.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:28:41.315 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:28:41 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:28:41.315 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:28:41 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:28:41.315 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T02:28:41.316 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:28:41.316 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T02:28:41.316 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:28:41.316 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:28:41.317 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:28:41.317 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T02:28:41.317 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:28:41.317 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T02:28:41.317 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:28:41.318 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T02:28:41.318 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:28:41.318 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T02:28:41.318 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:28:41.319 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T02:28:41.319 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:28:41.319 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:28:41.319 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:28:41.319 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:28:41 smithi115 ceph-mon[78593]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T02:28:41.320 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T02:28:41.320 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:28:41.320 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T02:28:41.320 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:28:41.321 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:28:41.321 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:28:41.321 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T02:28:41.321 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:28:41.322 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T02:28:41.322 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:28:41.322 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T02:28:41.322 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:28:41.322 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T02:28:41.323 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:28:41.323 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T02:28:41.323 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:28:41.323 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:28:41.323 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:28:41.324 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T02:28:41.324 INFO:journalctl@ceph.mon.c.smithi115.stdout: During handling of the above exception, another exception occurred: 2022-05-14T02:28:41.324 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T02:28:41.324 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T02:28:41.325 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T02:28:41.325 INFO:journalctl@ceph.mon.c.smithi115.stdout: return f(*arg) 2022-05-14T02:28:41.325 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T02:28:41.325 INFO:journalctl@ceph.mon.c.smithi115.stdout: self._write_client_files(client_files, host) 2022-05-14T02:28:41.326 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T02:28:41.326 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T02:28:41.326 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T02:28:41.326 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T02:28:41.326 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T02:28:41.327 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.event_loop.get_result(coro) 2022-05-14T02:28:41.327 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T02:28:41.327 INFO:journalctl@ceph.mon.c.smithi115.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T02:28:41.327 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T02:28:41.328 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.__get_result() 2022-05-14T02:28:41.328 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T02:28:41.328 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise self._exception 2022-05-14T02:28:41.328 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T02:28:41.328 INFO:journalctl@ceph.mon.c.smithi115.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T02:28:41.329 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T02:28:41.329 INFO:journalctl@ceph.mon.c.smithi115.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T02:28:41.329 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T02:28:41.329 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise 2022-05-14T02:28:41.330 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T02:28:41.330 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.gen.throw(type, value, traceback) 2022-05-14T02:28:41.330 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T02:28:41.330 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OrchestratorError(msg) 2022-05-14T02:28:41.331 INFO:journalctl@ceph.mon.c.smithi115.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:28:41.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:28:41 smithi115 conmon[90393]: 2022-05-14T02:28:41.290+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:28:42.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:28:42 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:28:42.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:28:42 smithi047 ceph-mon[77147]: pgmap v5779: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:28:42.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:28:42 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 10855 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:28:42.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:28:42 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:28:42.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:28:42 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:28:42.314 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:28:42 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:28:42.314 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:28:42 smithi115 ceph-mon[78593]: pgmap v5779: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:28:42.315 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:28:42 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 10855 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:28:42.315 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:28:42 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:28:42.315 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:28:42 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:28:42.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:28:42 smithi115 conmon[90393]: 2022-05-14T02:28:42.313+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:28:43.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:28:43 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:28:43.303 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:28:43 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:28:43.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:28:43 smithi115 conmon[90393]: 2022-05-14T02:28:43.302+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:28:44.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:28:44 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:28:44.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:28:44 smithi047 ceph-mon[77147]: pgmap v5780: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:28:44.329 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:28:44 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:28:44.329 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:28:44 smithi115 ceph-mon[78593]: pgmap v5780: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:28:44.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:28:44 smithi115 conmon[90393]: 2022-05-14T02:28:44.327+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:28:45.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:28:45 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:28:45.297 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:28:45 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:28:45.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:28:45 smithi115 conmon[90393]: 2022-05-14T02:28:45.296+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:28:46.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:28:46 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:28:46.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:28:46 smithi047 ceph-mon[77147]: pgmap v5781: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:28:46.326 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:28:46 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:28:46.327 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:28:46 smithi115 ceph-mon[78593]: pgmap v5781: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:28:46.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:28:46 smithi115 conmon[90393]: 2022-05-14T02:28:46.324+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:28:47.318 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:28:47 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:28:47.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:28:47 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:28:47.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:28:47 smithi115 conmon[90393]: 2022-05-14T02:28:47.317+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:28:48.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:28:48 smithi115 conmon[90393]: 2022-05-14T02:28:48.292+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:28:48.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:28:48 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:28:48.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:28:48 smithi115 ceph-mon[78593]: pgmap v5782: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:28:48.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:28:48 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:28:48.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:28:48 smithi047 ceph-mon[77147]: pgmap v5782: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:28:49.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:28:49 smithi115 conmon[90393]: 2022-05-14T02:28:49.276+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:28:49.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:28:49 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:28:49.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:28:49 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:28:50.315 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:28:50 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:28:50.315 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:28:50 smithi115 ceph-mon[78593]: pgmap v5783: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:28:50.315 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:28:50 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:28:50.316 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:28:50 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:28:50.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:28:50 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:28:50.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:28:50 smithi047 ceph-mon[77147]: pgmap v5783: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:28:50.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:28:50 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:28:50.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:28:50 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:28:50.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:28:50 smithi115 conmon[90393]: 2022-05-14T02:28:50.314+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:28:51.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:28:51 smithi115 conmon[90393]: 2022-05-14T02:28:51.285+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:28:51.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:28:51 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:28:51.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:28:51 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 10860 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:28:51.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:28:51 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:28:51.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:28:51 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 10860 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:28:52.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:28:52 smithi115 conmon[90393]: 2022-05-14T02:28:52.246+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:28:52.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:28:52 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:28:52.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:28:52 smithi115 ceph-mon[78593]: pgmap v5784: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:28:52.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:28:52 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:28:52.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:28:52 smithi047 ceph-mon[77147]: pgmap v5784: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:28:53.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:28:53 smithi115 conmon[90393]: 2022-05-14T02:28:53.287+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:28:53.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:28:53 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:28:53.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:28:53 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:28:54.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:28:54 smithi115 conmon[90393]: 2022-05-14T02:28:54.300+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:28:54.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:28:54 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:28:54.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:28:54 smithi115 ceph-mon[78593]: pgmap v5785: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:28:54.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:28:54 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:28:54.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:28:54 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:28:54.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:28:54 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:28:54.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:28:54 smithi047 ceph-mon[77147]: pgmap v5785: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:28:54.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:28:54 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:28:54.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:28:54 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:28:55.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:28:55 smithi115 conmon[90393]: 2022-05-14T02:28:55.265+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:28:55.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:28:55 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:28:55.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:28:55 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:28:56.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:28:56 smithi115 conmon[90393]: 2022-05-14T02:28:56.260+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:28:56.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:28:56 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:28:56.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:28:56 smithi115 ceph-mon[78593]: pgmap v5786: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:28:56.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:28:56 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 10870 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:28:56.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:28:56 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:28:56.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:28:56 smithi047 ceph-mon[77147]: pgmap v5786: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:28:56.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:28:56 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 10870 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:28:57.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:28:57 smithi115 conmon[90393]: 2022-05-14T02:28:57.264+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:28:57.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:28:57 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:28:57.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:28:57 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:28:58.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:28:58 smithi115 conmon[90393]: 2022-05-14T02:28:58.227+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:28:58.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:28:58 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:28:58.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:28:58 smithi115 ceph-mon[78593]: pgmap v5787: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:28:58.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:28:58 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:28:58.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:28:58 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:28:58.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:28:58 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:28:58.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:28:58 smithi047 ceph-mon[77147]: pgmap v5787: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:28:58.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:28:58 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:28:58.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:28:58 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:28:59.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:28:59 smithi115 conmon[90393]: 2022-05-14T02:28:59.220+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:28:59.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:28:59 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:28:59.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:28:59 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:29:00.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:29:00 smithi115 conmon[90393]: 2022-05-14T02:29:00.210+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:29:00.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:29:00 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:29:00.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:29:00 smithi115 ceph-mon[78593]: pgmap v5788: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:29:00.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:29:00 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:29:00.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:29:00 smithi047 ceph-mon[77147]: pgmap v5788: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:29:01.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:29:01 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:29:01.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:29:01 smithi115 conmon[90393]: 2022-05-14T02:29:01.236+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:29:01.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:29:01 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:29:02.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:29:02 smithi115 conmon[90393]: 2022-05-14T02:29:02.211+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:29:02.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:29:02 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:29:02.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:29:02 smithi115 ceph-mon[78593]: pgmap v5789: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:29:02.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:29:02 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 10875 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:29:02.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:29:02 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:29:02.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:29:02 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:29:02.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:29:02 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:29:02.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:29:02 smithi047 ceph-mon[77147]: pgmap v5789: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:29:02.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:29:02 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 10875 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:29:02.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:29:02 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:29:02.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:29:02 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:29:03.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:29:03 smithi115 conmon[90393]: 2022-05-14T02:29:03.175+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:29:03.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:29:03 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:29:03.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:29:03 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:29:04.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:29:04 smithi115 conmon[90393]: 2022-05-14T02:29:04.180+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:29:04.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:29:04 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:29:04.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:29:04 smithi115 ceph-mon[78593]: pgmap v5790: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:29:04.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:29:04 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:29:04.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:29:04 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:29:04.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:29:04 smithi047 ceph-mon[77147]: pgmap v5790: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:29:04.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:29:04 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:29:05.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:29:05 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:29:05.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:29:05 smithi115 conmon[90393]: 2022-05-14T02:29:05.162+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:29:05.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:29:05 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:29:06.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:29:06 smithi115 conmon[90393]: 2022-05-14T02:29:06.140+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:29:06.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:29:06 smithi115 ceph-mon[78593]: pgmap v5791: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:29:06.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:29:06 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:29:06.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:29:06 smithi047 ceph-mon[77147]: pgmap v5791: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:29:06.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:29:06 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:29:07.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:29:07 smithi115 conmon[90393]: 2022-05-14T02:29:07.094+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:29:07.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:29:07 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:29:07.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:29:07 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:29:08.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:29:08 smithi115 conmon[90393]: 2022-05-14T02:29:08.049+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:29:08.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:29:08 smithi115 ceph-mon[78593]: pgmap v5792: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:29:08.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:29:08 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:29:08.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:29:08 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:29:08.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:29:08 smithi047 ceph-mon[77147]: pgmap v5792: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:29:08.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:29:08 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:29:08.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:29:08 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:29:09.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:29:09 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:29:09.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:29:09 smithi115 conmon[90393]: 2022-05-14T02:29:09.074+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:29:09.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:29:09 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:29:10.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:29:10 smithi115 conmon[90393]: 2022-05-14T02:29:10.094+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:29:10.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:29:10 smithi115 ceph-mon[78593]: pgmap v5793: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:29:10.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:29:10 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:29:10.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:29:10 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:29:10.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:29:10 smithi047 ceph-mon[77147]: pgmap v5793: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:29:10.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:29:10 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:29:10.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:29:10 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:29:11.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:29:11 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:29:11.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:29:11 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 10880 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:29:11.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:29:11 smithi115 conmon[90393]: 2022-05-14T02:29:11.121+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:29:11.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:29:11 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:29:11.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:29:11 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 10880 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:29:12.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:29:12 smithi115 conmon[90393]: 2022-05-14T02:29:12.097+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:29:12.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:29:12 smithi115 ceph-mon[78593]: pgmap v5794: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:29:12.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:29:12 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:29:12.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:29:12 smithi047 ceph-mon[77147]: pgmap v5794: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:29:12.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:29:12 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:29:13.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:29:13 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:29:13.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:29:13 smithi115 conmon[90393]: 2022-05-14T02:29:13.102+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:29:13.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:29:13 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:29:14.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:29:14 smithi047 ceph-mon[77147]: pgmap v5795: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:29:14.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:29:14 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:29:14.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:29:14 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:29:14.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:29:14 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:29:14.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:29:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T02:29:14.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:29:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T02:29:14.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:29:14 smithi115 conmon[90393]: 2022-05-14T02:29:14.133+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:29:14.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:29:14 smithi115 ceph-mon[78593]: pgmap v5795: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:29:14.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:29:14 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:29:14.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:29:14 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:29:14.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:29:14 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:29:14.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:29:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T02:29:14.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:29:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T02:29:15.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:29:15 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:29:15.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:29:15 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:29:15.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:29:15 smithi115 conmon[90393]: 2022-05-14T02:29:15.161+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:29:16.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:29:16 smithi047 ceph-mon[77147]: pgmap v5796: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:29:16.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:29:16 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:29:16.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:29:16 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 10890 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:29:16.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:29:16 smithi115 conmon[90393]: 2022-05-14T02:29:16.136+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:29:16.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:29:16 smithi115 ceph-mon[78593]: pgmap v5796: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:29:16.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:29:16 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:29:16.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:29:16 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 10890 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:29:17.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:29:17 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:29:17.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:29:17 smithi115 conmon[90393]: 2022-05-14T02:29:17.103+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:29:17.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:29:17 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:29:18.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:29:18 smithi047 ceph-mon[77147]: pgmap v5797: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:29:18.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:29:18 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:29:18.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:29:18 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:29:18.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:29:18 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:29:18.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:29:18 smithi115 conmon[90393]: 2022-05-14T02:29:18.112+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:29:18.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:29:18 smithi115 ceph-mon[78593]: pgmap v5797: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:29:18.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:29:18 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:29:18.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:29:18 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:29:18.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:29:18 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:29:19.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:29:19 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:29:19.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:29:19 smithi115 conmon[90393]: 2022-05-14T02:29:19.148+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:29:19.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:29:19 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:29:20.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:29:20 smithi047 ceph-mon[77147]: pgmap v5798: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:29:20.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:29:20 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:29:20.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:29:20 smithi115 conmon[90393]: 2022-05-14T02:29:20.144+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:29:20.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:29:20 smithi115 ceph-mon[78593]: pgmap v5798: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:29:20.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:29:20 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:29:21.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:29:21 smithi115 conmon[90393]: 2022-05-14T02:29:21.106+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:29:22.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:29:21 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:29:22.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:29:21 smithi047 ceph-mon[77147]: pgmap v5799: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:29:22.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:29:21 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 10895 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:29:22.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:29:21 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:29:22.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:29:21 smithi115 ceph-mon[78593]: pgmap v5799: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:29:22.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:29:21 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 10895 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:29:22.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:29:22 smithi115 conmon[90393]: 2022-05-14T02:29:22.117+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:29:23.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:29:22 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:29:23.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:29:22 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:29:23.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:29:22 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:29:23.106 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:29:22 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:29:23.106 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:29:22 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:29:23.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:29:22 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:29:23.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:29:23 smithi115 conmon[90393]: 2022-05-14T02:29:23.105+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:29:24.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:29:23 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:29:24.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:29:23 smithi047 ceph-mon[77147]: pgmap v5800: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:29:24.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:29:23 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:29:24.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:29:23 smithi115 ceph-mon[78593]: pgmap v5800: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:29:24.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:29:24 smithi115 conmon[90393]: 2022-05-14T02:29:24.111+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:29:25.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:29:24 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:29:25.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:29:24 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:29:25.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:29:25 smithi115 conmon[90393]: 2022-05-14T02:29:25.109+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:29:26.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:29:25 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:29:26.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:29:25 smithi047 ceph-mon[77147]: pgmap v5801: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:29:26.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:29:25 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:29:26.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:29:25 smithi115 ceph-mon[78593]: pgmap v5801: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:29:26.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:29:26 smithi115 conmon[90393]: 2022-05-14T02:29:26.155+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:29:27.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:29:26 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:29:27.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:29:26 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:29:27.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:29:27 smithi115 conmon[90393]: 2022-05-14T02:29:27.161+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:29:28.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:29:27 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:29:28.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:29:27 smithi047 ceph-mon[77147]: pgmap v5802: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:29:28.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:29:27 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:29:28.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:29:27 smithi115 ceph-mon[78593]: pgmap v5802: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:29:28.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:29:28 smithi115 conmon[90393]: 2022-05-14T02:29:28.158+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:29:29.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:29:28 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:29:29.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:29:28 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:29:29.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:29:29 smithi115 conmon[90393]: 2022-05-14T02:29:29.184+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:29:30.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:29:29 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:29:30.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:29:29 smithi047 ceph-mon[77147]: pgmap v5803: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:29:30.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:29:29 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:29:30.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:29:29 smithi115 ceph-mon[78593]: pgmap v5803: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:29:30.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:29:30 smithi115 conmon[90393]: 2022-05-14T02:29:30.166+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:29:31.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:29:30 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:29:31.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:29:30 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:29:31.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:29:30 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:29:31.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:29:30 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 10900 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:29:31.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:29:30 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:29:31.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:29:30 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:29:31.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:29:30 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:29:31.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:29:30 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 10900 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:29:31.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:29:31 smithi115 conmon[90393]: 2022-05-14T02:29:31.163+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:29:32.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:29:31 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:29:32.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:29:31 smithi047 ceph-mon[77147]: pgmap v5804: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:29:32.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:29:31 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:29:32.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:29:31 smithi115 ceph-mon[78593]: pgmap v5804: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:29:32.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:29:32 smithi115 conmon[90393]: 2022-05-14T02:29:32.142+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:29:33.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:29:32 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:29:33.094 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:29:32 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:29:33.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:29:33 smithi115 conmon[90393]: 2022-05-14T02:29:33.093+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:29:34.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:29:33 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:29:34.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:29:33 smithi047 ceph-mon[77147]: pgmap v5805: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:29:34.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:29:33 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:29:34.100 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:29:33 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:29:34.100 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:29:33 smithi115 ceph-mon[78593]: pgmap v5805: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:29:34.101 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:29:33 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:29:34.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:29:34 smithi115 conmon[90393]: 2022-05-14T02:29:34.098+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:29:35.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:29:34 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:29:35.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:29:34 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:29:35.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:29:34 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:29:35.079 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:29:34 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:29:35.080 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:29:34 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:29:35.080 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:29:34 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:29:35.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:29:35 smithi115 conmon[90393]: 2022-05-14T02:29:35.078+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:29:36.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:29:35 smithi047 ceph-mon[77147]: pgmap v5806: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:29:36.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:29:35 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:29:36.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:29:35 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 10911 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:29:36.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:29:35 smithi115 ceph-mon[78593]: pgmap v5806: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:29:36.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:29:35 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:29:36.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:29:35 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 10911 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:29:36.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:29:36 smithi115 conmon[90393]: 2022-05-14T02:29:36.112+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:29:37.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:29:37 smithi115 conmon[90393]: 2022-05-14T02:29:37.065+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:29:38.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:29:37 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:29:38.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:29:37 smithi047 ceph-mon[77147]: pgmap v5807: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:29:38.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:29:37 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:29:38.090 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:29:37 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:29:38.091 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:29:37 smithi115 ceph-mon[78593]: pgmap v5807: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:29:38.091 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:29:37 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:29:38.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:29:38 smithi115 conmon[90393]: 2022-05-14T02:29:38.089+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:29:39.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:29:38 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:29:39.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:29:38 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:29:39.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:29:38 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:29:39.098 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:29:38 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:29:39.099 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:29:38 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:29:39.099 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:29:38 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:29:39.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:29:39 smithi115 conmon[90393]: 2022-05-14T02:29:39.097+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:29:40.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:29:39 smithi047 ceph-mon[77147]: pgmap v5808: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:29:40.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:29:39 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:29:40.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:29:39 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T02:29:40.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:29:39 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T02:29:40.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:29:39 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T02:29:40.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:29:39 smithi115 ceph-mon[78593]: pgmap v5808: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:29:40.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:29:39 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:29:40.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:29:39 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T02:29:40.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:29:39 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T02:29:40.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:29:39 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T02:29:40.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:29:40 smithi115 conmon[90393]: 2022-05-14T02:29:40.119+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:29:41.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:29:41 smithi115 conmon[90393]: 2022-05-14T02:29:41.165+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:29:42.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:29:41 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:29:42.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:29:41 smithi047 ceph-mon[77147]: pgmap v5809: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:29:42.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:29:41 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 10916 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:29:42.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:29:41 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:29:42.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:29:41 smithi115 ceph-mon[78593]: pgmap v5809: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:29:42.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:29:41 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 10916 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:29:42.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:29:42 smithi115 conmon[90393]: 2022-05-14T02:29:42.134+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:29:43.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:29:42 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:29:43.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:29:42 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:29:43.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:29:42 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:29:43.042 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:29:42 smithi047 conmon[78521]: 2022-05-14T02:29:42.768+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:29:43.042 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:29:42 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T02:29:43.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:29:42 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:29:43.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:29:42 smithi047 conmon[78521]: yield 2022-05-14T02:29:43.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:29:42 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:29:43.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:29:42 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:29:43.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:29:42 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:29:43.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:29:42 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T02:29:43.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:29:42 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:29:43.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:29:42 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T02:29:43.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:29:42 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:29:43.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:29:42 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T02:29:43.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:29:42 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:29:43.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:29:42 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T02:29:43.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:29:42 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:29:43.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:29:42 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T02:29:43.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:29:42 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:29:43.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:29:42 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:29:43.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:29:42 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:29:43.092 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:29:42 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:29:43.093 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:29:42 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:29:43.093 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:29:42 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:29:43.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:29:43 smithi115 conmon[90393]: 2022-05-14T02:29:43.091+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:29:44.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:29:43 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:29:44.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:29:43 smithi047 ceph-mon[77147]: pgmap v5810: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:29:44.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:29:43 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:29:44.042 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T02:29:44.042 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:29:44.042 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T02:29:44.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:29:44.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:29:44.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:29:44.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T02:29:44.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:29:44.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T02:29:44.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:29:44.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T02:29:44.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:29:44.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T02:29:44.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:29:44.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T02:29:44.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:29:44.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:29:44.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:29:44.047 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:29:43 smithi047 ceph-mon[77147]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T02:29:44.047 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:29:43 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:29:44.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:29:43 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:29:44.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:29:43 smithi115 ceph-mon[78593]: pgmap v5810: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:29:44.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:29:43 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:29:44.109 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T02:29:44.109 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:29:44.109 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T02:29:44.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:29:44.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:29:44.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:29:44.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T02:29:44.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:29:44.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T02:29:44.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:29:44.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T02:29:44.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:29:44.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T02:29:44.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:29:44.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T02:29:44.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:29:44.113 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:29:44.113 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:29:44.113 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:29:43 smithi115 ceph-mon[78593]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T02:29:44.113 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:29:43 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:29:44.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:29:44 smithi115 conmon[90393]: 2022-05-14T02:29:44.135+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:29:45.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:29:45 smithi115 conmon[90393]: 2022-05-14T02:29:45.175+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:29:46.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:29:45 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:29:46.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:29:45 smithi047 ceph-mon[77147]: pgmap v5811: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:29:46.042 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:29:45 smithi047 conmon[78521]: 2022-05-14T02:29:45.840+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:29:46.042 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:29:45 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T02:29:46.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:29:45 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:29:46.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:29:45 smithi047 conmon[78521]: yield 2022-05-14T02:29:46.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:29:45 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:29:46.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:29:45 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:29:46.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:29:45 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:29:46.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:29:45 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T02:29:46.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:29:45 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:29:46.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:29:45 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T02:29:46.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:29:45 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:29:46.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:29:45 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T02:29:46.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:29:45 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:29:46.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:29:45 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T02:29:46.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:29:45 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:29:46.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:29:45 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T02:29:46.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:29:45 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:29:46.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:29:45 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:29:46.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:29:45 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:29:46.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:29:45 smithi047 conmon[78521]: 2022-05-14T02:29:45.841+0000 7f534177b700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T02:29:46.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:29:45 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T02:29:46.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:29:45 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:29:46.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:29:45 smithi047 conmon[78521]: yield 2022-05-14T02:29:46.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:29:45 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:29:46.048 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:29:45 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:29:46.048 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:29:45 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:29:46.048 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:29:45 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T02:29:46.048 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:29:45 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:29:46.049 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:29:45 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T02:29:46.049 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:29:45 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:29:46.049 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:29:45 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T02:29:46.049 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:29:45 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:29:46.050 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:29:45 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T02:29:46.050 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:29:45 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:29:46.050 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:29:45 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T02:29:46.050 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:29:45 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:29:46.051 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:29:45 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:29:46.051 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:29:45 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:29:46.051 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:29:45 smithi047 conmon[78521]: 2022-05-14T02:29:46.051 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:29:45 smithi047 conmon[78521]: During handling of the above exception, another exception occurred: 2022-05-14T02:29:46.051 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:29:45 smithi047 conmon[78521]: 2022-05-14T02:29:46.052 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:29:45 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T02:29:46.052 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:29:45 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T02:29:46.052 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:29:45 smithi047 conmon[78521]: return f(*arg) 2022-05-14T02:29:46.052 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:29:45 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T02:29:46.053 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:29:45 smithi047 conmon[78521]: self._write_client_files(client_files, host) 2022-05-14T02:29:46.053 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:29:45 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T02:29:46.053 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:29:45 smithi047 conmon[78521]: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T02:29:46.053 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:29:45 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T02:29:46.053 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:29:45 smithi047 conmon[78521]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T02:29:46.054 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:29:45 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T02:29:46.054 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:29:45 smithi047 conmon[78521]: return self.event_loop.get_result(coro) 2022-05-14T02:29:46.054 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:29:45 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T02:29:46.054 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:29:45 smithi047 conmon[78521]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T02:29:46.054 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:29:45 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T02:29:46.055 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:29:45 smithi047 conmon[78521]: return self.__get_result() 2022-05-14T02:29:46.055 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:29:45 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T02:29:46.055 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:29:45 smithi047 conmon[78521]: raise self._exception 2022-05-14T02:29:46.055 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:29:45 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T02:29:46.056 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:29:45 smithi047 conmon[78521]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T02:29:46.056 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:29:45 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T02:29:46.056 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:29:45 smithi047 conmon[78521]: conn = await self._remote_connection(host, addr) 2022-05-14T02:29:46.056 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:29:45 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T02:29:46.056 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:29:45 smithi047 conmon[78521]: raise 2022-05-14T02:29:46.057 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:29:45 smithi047 conmon[78521]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T02:29:46.057 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:29:45 smithi047 conmon[78521]: self.gen.throw(type, value, traceback) 2022-05-14T02:29:46.057 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:29:45 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T02:29:46.057 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:29:45 smithi047 conmon[78521]: raise OrchestratorError(msg) 2022-05-14T02:29:46.058 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:29:45 smithi047 conmon[78521]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:29:46.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:29:45 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:29:46.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:29:45 smithi115 ceph-mon[78593]: pgmap v5811: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:29:46.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:29:46 smithi115 conmon[90393]: 2022-05-14T02:29:46.152+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:29:47.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:29:46 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:29:47.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:29:46 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:29:47.042 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T02:29:47.042 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:29:47.042 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T02:29:47.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:29:47.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:29:47.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:29:47.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T02:29:47.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:29:47.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T02:29:47.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:29:47.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T02:29:47.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:29:47.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T02:29:47.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:29:47.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T02:29:47.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:29:47.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:29:47.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:29:47.046 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:29:46 smithi047 ceph-mon[77147]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T02:29:47.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T02:29:47.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:29:47.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T02:29:47.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:29:47.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:29:47.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:29:47.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T02:29:47.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:29:47.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T02:29:47.049 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:29:47.049 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T02:29:47.049 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:29:47.049 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T02:29:47.049 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:29:47.050 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T02:29:47.050 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:29:47.050 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:29:47.050 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:29:47.050 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T02:29:47.051 INFO:journalctl@ceph.mon.b.smithi047.stdout: During handling of the above exception, another exception occurred: 2022-05-14T02:29:47.051 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T02:29:47.051 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T02:29:47.051 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T02:29:47.051 INFO:journalctl@ceph.mon.b.smithi047.stdout: return f(*arg) 2022-05-14T02:29:47.052 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T02:29:47.052 INFO:journalctl@ceph.mon.b.smithi047.stdout: self._write_client_files(client_files, host) 2022-05-14T02:29:47.052 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T02:29:47.052 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T02:29:47.053 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T02:29:47.053 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T02:29:47.053 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T02:29:47.053 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.event_loop.get_result(coro) 2022-05-14T02:29:47.053 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T02:29:47.054 INFO:journalctl@ceph.mon.b.smithi047.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T02:29:47.054 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T02:29:47.054 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.__get_result() 2022-05-14T02:29:47.054 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T02:29:47.054 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise self._exception 2022-05-14T02:29:47.055 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T02:29:47.055 INFO:journalctl@ceph.mon.b.smithi047.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T02:29:47.055 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T02:29:47.055 INFO:journalctl@ceph.mon.b.smithi047.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T02:29:47.056 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T02:29:47.056 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise 2022-05-14T02:29:47.056 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T02:29:47.056 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.gen.throw(type, value, traceback) 2022-05-14T02:29:47.056 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T02:29:47.057 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OrchestratorError(msg) 2022-05-14T02:29:47.057 INFO:journalctl@ceph.mon.b.smithi047.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:29:47.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:29:46 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:29:47.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:29:46 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:29:47.108 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T02:29:47.109 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:29:47.109 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T02:29:47.109 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:29:47.109 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:29:47.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:29:47.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T02:29:47.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:29:47.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T02:29:47.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:29:47.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T02:29:47.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:29:47.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T02:29:47.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:29:47.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T02:29:47.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:29:47.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:29:47.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:29:47.113 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:29:46 smithi115 ceph-mon[78593]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T02:29:47.113 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T02:29:47.113 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:29:47.113 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T02:29:47.113 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:29:47.114 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:29:47.114 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:29:47.114 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T02:29:47.114 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:29:47.114 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T02:29:47.115 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:29:47.115 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T02:29:47.115 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:29:47.115 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T02:29:47.116 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:29:47.116 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T02:29:47.116 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:29:47.116 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:29:47.116 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:29:47.117 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T02:29:47.117 INFO:journalctl@ceph.mon.c.smithi115.stdout: During handling of the above exception, another exception occurred: 2022-05-14T02:29:47.117 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T02:29:47.117 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T02:29:47.118 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T02:29:47.118 INFO:journalctl@ceph.mon.c.smithi115.stdout: return f(*arg) 2022-05-14T02:29:47.118 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T02:29:47.118 INFO:journalctl@ceph.mon.c.smithi115.stdout: self._write_client_files(client_files, host) 2022-05-14T02:29:47.118 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T02:29:47.119 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T02:29:47.119 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T02:29:47.119 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T02:29:47.119 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T02:29:47.120 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.event_loop.get_result(coro) 2022-05-14T02:29:47.120 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T02:29:47.120 INFO:journalctl@ceph.mon.c.smithi115.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T02:29:47.120 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T02:29:47.121 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.__get_result() 2022-05-14T02:29:47.121 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T02:29:47.121 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise self._exception 2022-05-14T02:29:47.121 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T02:29:47.121 INFO:journalctl@ceph.mon.c.smithi115.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T02:29:47.122 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T02:29:47.122 INFO:journalctl@ceph.mon.c.smithi115.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T02:29:47.122 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T02:29:47.122 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise 2022-05-14T02:29:47.123 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T02:29:47.123 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.gen.throw(type, value, traceback) 2022-05-14T02:29:47.123 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T02:29:47.123 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OrchestratorError(msg) 2022-05-14T02:29:47.124 INFO:journalctl@ceph.mon.c.smithi115.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:29:47.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:29:47 smithi115 conmon[90393]: 2022-05-14T02:29:47.191+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:29:48.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:29:47 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:29:48.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:29:47 smithi047 ceph-mon[77147]: pgmap v5812: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:29:48.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:29:47 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:29:48.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:29:47 smithi115 ceph-mon[78593]: pgmap v5812: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:29:48.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:29:48 smithi115 conmon[90393]: 2022-05-14T02:29:48.165+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:29:49.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:29:48 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:29:49.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:29:48 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:29:49.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:29:49 smithi115 conmon[90393]: 2022-05-14T02:29:49.196+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:29:50.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:29:49 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:29:50.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:29:49 smithi115 ceph-mon[78593]: pgmap v5813: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:29:50.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:29:49 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:29:50.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:29:49 smithi047 ceph-mon[77147]: pgmap v5813: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:29:50.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:29:50 smithi115 conmon[90393]: 2022-05-14T02:29:50.222+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:29:51.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:29:50 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:29:51.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:29:50 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:29:51.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:29:50 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:29:51.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:29:50 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 10920 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:29:51.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:29:50 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:29:51.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:29:50 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:29:51.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:29:50 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:29:51.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:29:50 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 10920 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:29:51.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:29:51 smithi115 conmon[90393]: 2022-05-14T02:29:51.180+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:29:52.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:29:51 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:29:52.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:29:51 smithi115 ceph-mon[78593]: pgmap v5814: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:29:52.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:29:51 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:29:52.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:29:51 smithi047 ceph-mon[77147]: pgmap v5814: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:29:52.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:29:52 smithi115 conmon[90393]: 2022-05-14T02:29:52.171+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:29:53.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:29:52 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:29:53.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:29:52 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:29:53.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:29:53 smithi115 conmon[90393]: 2022-05-14T02:29:53.174+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:29:54.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:29:53 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:29:54.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:29:53 smithi115 ceph-mon[78593]: pgmap v5815: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:29:54.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:29:53 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:29:54.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:29:53 smithi047 ceph-mon[77147]: pgmap v5815: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:29:54.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:29:54 smithi115 conmon[90393]: 2022-05-14T02:29:54.146+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:29:55.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:29:54 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:29:55.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:29:54 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:29:55.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:29:54 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:29:55.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:29:54 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:29:55.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:29:54 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:29:55.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:29:54 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:29:55.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:29:55 smithi115 conmon[90393]: 2022-05-14T02:29:55.111+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:29:56.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:29:55 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:29:56.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:29:55 smithi115 ceph-mon[78593]: pgmap v5816: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:29:56.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:29:55 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 10930 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:29:56.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:29:55 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:29:56.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:29:55 smithi047 ceph-mon[77147]: pgmap v5816: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:29:56.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:29:55 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 10930 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:29:56.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:29:56 smithi115 conmon[90393]: 2022-05-14T02:29:56.143+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:29:57.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:29:56 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:29:57.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:29:56 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:29:57.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:29:57 smithi115 conmon[90393]: 2022-05-14T02:29:57.132+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:29:58.103 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:29:57 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:29:58.103 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:29:57 smithi115 ceph-mon[78593]: pgmap v5817: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:29:58.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:29:57 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:29:58.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:29:57 smithi047 ceph-mon[77147]: pgmap v5817: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:29:58.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:29:58 smithi115 conmon[90393]: 2022-05-14T02:29:58.101+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:29:59.095 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:29:58 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:29:59.095 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:29:58 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:29:59.095 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:29:58 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:29:59.095 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:29:58 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:29:59.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:29:58 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:29:59.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:29:58 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:29:59.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:29:58 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:29:59.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:29:58 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:29:59.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:29:59 smithi115 conmon[90393]: 2022-05-14T02:29:59.093+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:30:00.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:29:59 smithi115 ceph-mon[78593]: pgmap v5818: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:30:00.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:29:59 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:30:00.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:29:59 smithi047 ceph-mon[77147]: pgmap v5818: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:30:00.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:29:59 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:30:00.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:30:00 smithi047 conmon[77123]: 2022-05-14T02:29:59.999+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : Health detail: HEALTH_ERR 1 filesystem is degraded; 1 mds daemon damaged; 1 MDSs are read only; 1 MDSs report slow metadata IOs; 1/3 mons down, quorum b,c; Reduced data availability: 1 pg inactive; 9 slow ops, oldest one blocked for 10930 sec, osd.5 has slow ops 2022-05-14T02:30:00.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:30:00 smithi047 conmon[77123]: 2022-05-14T02:29:59.999+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : [WRN] FS_DEGRADED: 1 filesystem is degraded 2022-05-14T02:30:00.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:30:00 smithi047 conmon[77123]: 2022-05-14T02:29:59.999+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : fs cephfs is degraded 2022-05-14T02:30:00.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:30:00 smithi047 conmon[77123]: 2022-05-14T02:29:59.999+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : [ERR] MDS_DAMAGE: 1 mds daemon damaged 2022-05-14T02:30:00.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:30:00 smithi047 conmon[77123]: 2022-05-14T02:29:59.999+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : fs cephfs mds.2 is damaged 2022-05-14T02:30:00.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:30:00 smithi047 conmon[77123]: 2022-05-14T02:29:59.999+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : [WRN] MDS_READ_ONLY: 1 MDSs are read only 2022-05-14T02:30:00.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:30:00 smithi047 conmon[77123]: 2022-05-14T02:29:59.999+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : mds.l(mds.0): MDS in read-only mode 2022-05-14T02:30:00.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:30:00 smithi047 conmon[77123]: 2022-05-14T02:29:59.999+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : [WRN] MDS_SLOW_METADATA_IO: 1 MDSs report slow metadata IOs 2022-05-14T02:30:00.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:30:00 smithi047 conmon[77123]: 2022-05-14T02:29:59.999+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : mds.l(mds.0): 4 slow metadata IOs are blocked > 180 secs, oldest blocked for 11233 secs 2022-05-14T02:30:00.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:30:00 smithi047 conmon[77123]: 2022-05-14T02:29:59.999+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : [WRN] MON_DOWN: 1/3 mons down, quorum b,c 2022-05-14T02:30:00.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:30:00 smithi047 conmon[77123]: 2022-05-14T02:29:59.999+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : mon.a (rank 0) addr [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] is down (out of quorum) 2022-05-14T02:30:00.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:30:00 smithi047 conmon[77123]: 2022-05-14T02:29:59.999+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : [WRN] PG_AVAILABILITY: Reduced data availability: 1 pg inactive 2022-05-14T02:30:00.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:30:00 smithi047 conmon[77123]: 2022-05-14T02:29:59.999+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : pg 2.1 is stuck inactive for 3h, current state unknown, last acting [] 2022-05-14T02:30:00.295 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:30:00 smithi047 conmon[77123]: 2022-05-14T02:29:59.999+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : [WRN] SLOW_OPS: 9 slow ops, oldest one blocked for 10930 sec, osd.5 has slow ops 2022-05-14T02:30:00.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:30:00 smithi115 conmon[90393]: 2022-05-14T02:30:00.140+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:30:01.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:30:00 smithi115 ceph-mon[78593]: Health detail: HEALTH_ERR 1 filesystem is degraded; 1 mds daemon damaged; 1 MDSs are read only; 1 MDSs report slow metadata IOs; 1/3 mons down, quorum b,c; Reduced data availability: 1 pg inactive; 9 slow ops, oldest one blocked for 10930 sec, osd.5 has slow ops 2022-05-14T02:30:01.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:30:00 smithi115 ceph-mon[78593]: [WRN] FS_DEGRADED: 1 filesystem is degraded 2022-05-14T02:30:01.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:30:00 smithi115 ceph-mon[78593]: fs cephfs is degraded 2022-05-14T02:30:01.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:30:00 smithi115 ceph-mon[78593]: [ERR] MDS_DAMAGE: 1 mds daemon damaged 2022-05-14T02:30:01.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:30:00 smithi115 ceph-mon[78593]: fs cephfs mds.2 is damaged 2022-05-14T02:30:01.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:30:00 smithi115 ceph-mon[78593]: [WRN] MDS_READ_ONLY: 1 MDSs are read only 2022-05-14T02:30:01.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:30:00 smithi115 ceph-mon[78593]: mds.l(mds.0): MDS in read-only mode 2022-05-14T02:30:01.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:30:00 smithi115 ceph-mon[78593]: [WRN] MDS_SLOW_METADATA_IO: 1 MDSs report slow metadata IOs 2022-05-14T02:30:01.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:30:00 smithi115 ceph-mon[78593]: mds.l(mds.0): 4 slow metadata IOs are blocked > 180 secs, oldest blocked for 11233 secs 2022-05-14T02:30:01.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:30:00 smithi115 ceph-mon[78593]: [WRN] MON_DOWN: 1/3 mons down, quorum b,c 2022-05-14T02:30:01.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:30:00 smithi115 ceph-mon[78593]: mon.a (rank 0) addr [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] is down (out of quorum) 2022-05-14T02:30:01.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:30:00 smithi115 ceph-mon[78593]: [WRN] PG_AVAILABILITY: Reduced data availability: 1 pg inactive 2022-05-14T02:30:01.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:30:00 smithi115 ceph-mon[78593]: pg 2.1 is stuck inactive for 3h, current state unknown, last acting [] 2022-05-14T02:30:01.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:30:00 smithi115 ceph-mon[78593]: [WRN] SLOW_OPS: 9 slow ops, oldest one blocked for 10930 sec, osd.5 has slow ops 2022-05-14T02:30:01.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:30:00 smithi047 ceph-mon[77147]: Health detail: HEALTH_ERR 1 filesystem is degraded; 1 mds daemon damaged; 1 MDSs are read only; 1 MDSs report slow metadata IOs; 1/3 mons down, quorum b,c; Reduced data availability: 1 pg inactive; 9 slow ops, oldest one blocked for 10930 sec, osd.5 has slow ops 2022-05-14T02:30:01.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:30:00 smithi047 ceph-mon[77147]: [WRN] FS_DEGRADED: 1 filesystem is degraded 2022-05-14T02:30:01.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:30:00 smithi047 ceph-mon[77147]: fs cephfs is degraded 2022-05-14T02:30:01.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:30:00 smithi047 ceph-mon[77147]: [ERR] MDS_DAMAGE: 1 mds daemon damaged 2022-05-14T02:30:01.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:30:00 smithi047 ceph-mon[77147]: fs cephfs mds.2 is damaged 2022-05-14T02:30:01.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:30:00 smithi047 ceph-mon[77147]: [WRN] MDS_READ_ONLY: 1 MDSs are read only 2022-05-14T02:30:01.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:30:00 smithi047 ceph-mon[77147]: mds.l(mds.0): MDS in read-only mode 2022-05-14T02:30:01.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:30:00 smithi047 ceph-mon[77147]: [WRN] MDS_SLOW_METADATA_IO: 1 MDSs report slow metadata IOs 2022-05-14T02:30:01.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:30:00 smithi047 ceph-mon[77147]: mds.l(mds.0): 4 slow metadata IOs are blocked > 180 secs, oldest blocked for 11233 secs 2022-05-14T02:30:01.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:30:00 smithi047 ceph-mon[77147]: [WRN] MON_DOWN: 1/3 mons down, quorum b,c 2022-05-14T02:30:01.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:30:00 smithi047 ceph-mon[77147]: mon.a (rank 0) addr [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] is down (out of quorum) 2022-05-14T02:30:01.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:30:00 smithi047 ceph-mon[77147]: [WRN] PG_AVAILABILITY: Reduced data availability: 1 pg inactive 2022-05-14T02:30:01.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:30:00 smithi047 ceph-mon[77147]: pg 2.1 is stuck inactive for 3h, current state unknown, last acting [] 2022-05-14T02:30:01.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:30:00 smithi047 ceph-mon[77147]: [WRN] SLOW_OPS: 9 slow ops, oldest one blocked for 10930 sec, osd.5 has slow ops 2022-05-14T02:30:01.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:30:01 smithi115 conmon[90393]: 2022-05-14T02:30:01.152+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:30:02.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:30:01 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:30:02.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:30:01 smithi115 ceph-mon[78593]: pgmap v5819: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:30:02.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:30:01 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 10936 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:30:02.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:30:01 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:30:02.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:30:01 smithi047 ceph-mon[77147]: pgmap v5819: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:30:02.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:30:01 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 10936 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:30:02.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:30:02 smithi115 conmon[90393]: 2022-05-14T02:30:02.134+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:30:03.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:30:02 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:30:03.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:30:02 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:30:03.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:30:02 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:30:03.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:30:02 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:30:03.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:30:02 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:30:03.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:30:02 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:30:03.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:30:03 smithi115 conmon[90393]: 2022-05-14T02:30:03.154+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:30:04.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:30:03 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:30:04.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:30:03 smithi115 ceph-mon[78593]: pgmap v5820: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:30:04.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:30:03 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:30:04.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:30:03 smithi047 ceph-mon[77147]: pgmap v5820: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:30:04.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:30:04 smithi115 conmon[90393]: 2022-05-14T02:30:04.113+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:30:05.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:30:05 smithi115 conmon[90393]: 2022-05-14T02:30:05.069+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:30:05.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:30:04 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:30:05.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:30:04 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:30:06.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:30:05 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:30:06.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:30:05 smithi047 ceph-mon[77147]: pgmap v5821: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:30:06.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:30:05 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:30:06.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:30:06 smithi115 conmon[90393]: 2022-05-14T02:30:06.056+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:30:06.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:30:05 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:30:06.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:30:05 smithi115 ceph-mon[78593]: pgmap v5821: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:30:06.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:30:05 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:30:07.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:30:06 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:30:07.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:30:07 smithi115 conmon[90393]: 2022-05-14T02:30:07.037+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:30:07.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:30:06 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:30:08.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:30:07 smithi047 ceph-mon[77147]: pgmap v5822: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:30:08.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:30:07 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:30:08.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:30:08 smithi115 conmon[90393]: 2022-05-14T02:30:08.080+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:30:08.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:30:07 smithi115 ceph-mon[78593]: pgmap v5822: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:30:08.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:30:07 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:30:08.871 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:30:08 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:30:09.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:30:08 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:30:09.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:30:09 smithi115 conmon[90393]: 2022-05-14T02:30:09.048+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:30:10.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:30:09 smithi047 ceph-mon[77147]: pgmap v5823: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:30:10.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:30:09 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:30:10.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:30:09 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:30:10.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:30:09 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:30:10.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:30:10 smithi115 conmon[90393]: 2022-05-14T02:30:10.022+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:30:10.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:30:09 smithi115 ceph-mon[78593]: pgmap v5823: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:30:10.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:30:09 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:30:10.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:30:09 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:30:10.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:30:09 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:30:11.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:30:10 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:30:11.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:30:10 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 10940 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:30:11.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:30:10 smithi115 conmon[90393]: 2022-05-14T02:30:10.985+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:30:11.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:30:10 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:30:11.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:30:10 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 10940 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:30:12.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:30:11 smithi047 ceph-mon[77147]: pgmap v5824: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:30:12.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:30:11 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:30:12.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:30:11 smithi115 conmon[90393]: 2022-05-14T02:30:11.986+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:30:12.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:30:11 smithi115 ceph-mon[78593]: pgmap v5824: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:30:12.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:30:11 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:30:13.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:30:12 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:30:13.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:30:12 smithi115 conmon[90393]: 2022-05-14T02:30:12.982+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:30:13.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:30:12 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:30:14.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:30:13 smithi047 ceph-mon[77147]: pgmap v5825: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:30:14.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:30:13 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:30:14.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:30:13 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:30:14.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:30:13 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:30:14.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:30:13 smithi115 conmon[90393]: 2022-05-14T02:30:13.954+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:30:14.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:30:13 smithi115 ceph-mon[78593]: pgmap v5825: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:30:14.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:30:13 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:30:14.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:30:13 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:30:14.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:30:13 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:30:15.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:30:14 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:30:15.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:30:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T02:30:15.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:30:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T02:30:15.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:30:14 smithi115 conmon[90393]: 2022-05-14T02:30:14.973+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:30:15.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:30:14 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:30:15.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:30:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T02:30:15.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:30:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T02:30:16.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:30:15 smithi047 ceph-mon[77147]: pgmap v5826: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:30:16.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:30:15 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:30:16.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:30:15 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 10950 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:30:16.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:30:16 smithi115 conmon[90393]: 2022-05-14T02:30:16.010+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:30:16.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:30:15 smithi115 ceph-mon[78593]: pgmap v5826: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:30:16.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:30:15 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:30:16.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:30:15 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 10950 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:30:17.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:30:16 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:30:17.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:30:17 smithi115 conmon[90393]: 2022-05-14T02:30:17.023+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:30:17.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:30:16 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:30:18.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:30:17 smithi047 ceph-mon[77147]: pgmap v5827: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:30:18.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:30:17 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:30:18.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:30:17 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:30:18.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:30:17 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:30:18.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:30:18 smithi115 conmon[90393]: 2022-05-14T02:30:18.057+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:30:18.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:30:17 smithi115 ceph-mon[78593]: pgmap v5827: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:30:18.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:30:17 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:30:18.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:30:17 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:30:18.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:30:17 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:30:19.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:30:18 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:30:19.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:30:19 smithi115 conmon[90393]: 2022-05-14T02:30:19.064+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:30:19.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:30:18 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:30:20.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:30:19 smithi047 ceph-mon[77147]: pgmap v5828: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:30:20.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:30:19 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:30:20.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:30:20 smithi115 conmon[90393]: 2022-05-14T02:30:20.111+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:30:20.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:30:19 smithi115 ceph-mon[78593]: pgmap v5828: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:30:20.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:30:19 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:30:21.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:30:21 smithi115 conmon[90393]: 2022-05-14T02:30:21.084+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:30:22.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:30:21 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:30:22.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:30:21 smithi047 ceph-mon[77147]: pgmap v5829: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:30:22.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:30:21 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 10955 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:30:22.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:30:21 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:30:22.089 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:30:21 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:30:22.089 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:30:21 smithi115 ceph-mon[78593]: pgmap v5829: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:30:22.089 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:30:21 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 10955 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:30:22.090 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:30:21 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:30:22.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:30:22 smithi115 conmon[90393]: 2022-05-14T02:30:22.087+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:30:23.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:30:22 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:30:23.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:30:22 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:30:23.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:30:22 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:30:23.079 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:30:22 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:30:23.079 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:30:22 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:30:23.079 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:30:22 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:30:23.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:30:23 smithi115 conmon[90393]: 2022-05-14T02:30:23.077+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:30:24.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:30:23 smithi047 ceph-mon[77147]: pgmap v5830: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:30:24.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:30:23 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:30:24.058 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:30:23 smithi115 ceph-mon[78593]: pgmap v5830: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:30:24.059 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:30:23 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:30:24.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:30:24 smithi115 conmon[90393]: 2022-05-14T02:30:24.057+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:30:25.015 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:30:24 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:30:25.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:30:24 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:30:25.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:30:25 smithi115 conmon[90393]: 2022-05-14T02:30:25.013+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:30:25.978 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:30:25 smithi115 ceph-mon[78593]: pgmap v5831: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:30:25.978 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:30:25 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:30:26.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:30:25 smithi047 ceph-mon[77147]: pgmap v5831: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:30:26.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:30:25 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:30:26.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:30:25 smithi115 conmon[90393]: 2022-05-14T02:30:25.976+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:30:27.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:30:26 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:30:27.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:30:26 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:30:27.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:30:26 smithi115 conmon[90393]: 2022-05-14T02:30:26.972+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:30:27.984 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:30:27 smithi115 ceph-mon[78593]: pgmap v5832: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:30:27.984 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:30:27 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:30:28.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:30:27 smithi047 ceph-mon[77147]: pgmap v5832: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:30:28.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:30:27 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:30:28.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:30:27 smithi115 conmon[90393]: 2022-05-14T02:30:27.982+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:30:28.995 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:30:28 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:30:29.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:30:28 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:30:29.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:30:28 smithi115 conmon[90393]: 2022-05-14T02:30:28.993+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:30:30.003 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:30:29 smithi115 ceph-mon[78593]: pgmap v5833: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:30:30.003 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:30:29 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:30:30.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:30:29 smithi047 ceph-mon[77147]: pgmap v5833: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:30:30.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:30:29 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:30:30.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:30:30 smithi115 conmon[90393]: 2022-05-14T02:30:30.002+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:30:31.011 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:30:30 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:30:31.012 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:30:30 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:30:31.012 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:30:30 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:30:31.012 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:30:30 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 10960 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:30:31.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:30:30 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:30:31.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:30:30 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:30:31.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:30:30 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:30:31.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:30:30 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 10960 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:30:31.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:30:31 smithi115 conmon[90393]: 2022-05-14T02:30:31.010+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:30:31.994 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:30:31 smithi115 ceph-mon[78593]: pgmap v5834: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:30:31.995 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:30:31 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:30:32.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:30:31 smithi047 ceph-mon[77147]: pgmap v5834: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:30:32.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:30:31 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:30:32.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:30:31 smithi115 conmon[90393]: 2022-05-14T02:30:31.993+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:30:33.027 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:30:32 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:30:33.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:30:32 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:30:33.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:30:33 smithi115 conmon[90393]: 2022-05-14T02:30:33.025+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:30:34.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:30:33 smithi047 ceph-mon[77147]: pgmap v5835: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:30:34.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:30:33 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:30:34.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:30:33 smithi115 conmon[90393]: 2022-05-14T02:30:33.991+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:30:34.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:30:33 smithi115 ceph-mon[78593]: pgmap v5835: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:30:34.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:30:33 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:30:35.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:30:34 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:30:35.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:30:34 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:30:35.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:30:34 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:30:35.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:30:35 smithi115 conmon[90393]: 2022-05-14T02:30:35.004+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:30:35.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:30:34 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:30:35.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:30:34 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:30:35.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:30:34 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:30:36.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:30:35 smithi047 ceph-mon[77147]: pgmap v5836: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:30:36.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:30:35 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:30:36.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:30:35 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 10970 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:30:36.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:30:35 smithi115 conmon[90393]: 2022-05-14T02:30:35.960+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:30:36.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:30:35 smithi115 ceph-mon[78593]: pgmap v5836: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:30:36.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:30:35 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:30:36.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:30:35 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 10970 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:30:37.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:30:36 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:30:37.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:30:36 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:30:37.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:30:36 smithi115 conmon[90393]: 2022-05-14T02:30:36.992+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:30:38.039 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:30:37 smithi115 ceph-mon[78593]: pgmap v5837: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:30:38.040 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:30:37 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:30:38.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:30:37 smithi047 ceph-mon[77147]: pgmap v5837: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:30:38.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:30:37 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:30:38.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:30:38 smithi115 conmon[90393]: 2022-05-14T02:30:38.038+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:30:39.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:30:38 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:30:39.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:30:38 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:30:39.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:30:38 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:30:39.072 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:30:38 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:30:39.073 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:30:38 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:30:39.073 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:30:38 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:30:39.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:30:39 smithi115 conmon[90393]: 2022-05-14T02:30:39.071+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:30:40.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:30:39 smithi047 ceph-mon[77147]: pgmap v5838: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:30:40.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:30:39 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:30:40.067 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:30:39 smithi115 ceph-mon[78593]: pgmap v5838: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:30:40.067 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:30:39 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:30:40.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:30:40 smithi115 conmon[90393]: 2022-05-14T02:30:40.065+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:30:41.034 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:30:40 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:30:41.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:30:40 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:30:41.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:30:41 smithi115 conmon[90393]: 2022-05-14T02:30:41.033+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:30:42.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:30:41 smithi047 ceph-mon[77147]: pgmap v5839: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:30:42.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:30:41 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 10975 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:30:42.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:30:41 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:30:42.071 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:30:41 smithi115 ceph-mon[78593]: pgmap v5839: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:30:42.072 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:30:41 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 10975 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:30:42.072 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:30:41 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:30:42.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:30:42 smithi115 conmon[90393]: 2022-05-14T02:30:42.070+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:30:43.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:30:42 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:30:43.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:30:42 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:30:43.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:30:42 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:30:43.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:30:42 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:30:43.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:30:42 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:30:43.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:30:42 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:30:43.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:30:43 smithi115 conmon[90393]: 2022-05-14T02:30:43.119+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:30:44.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:30:43 smithi115 ceph-mon[78593]: pgmap v5840: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:30:44.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:30:43 smithi047 ceph-mon[77147]: pgmap v5840: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:30:44.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:30:44 smithi115 conmon[90393]: 2022-05-14T02:30:44.158+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:30:45.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:30:44 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:30:45.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:30:44 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:30:45.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:30:45 smithi115 conmon[90393]: 2022-05-14T02:30:45.112+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:30:46.070 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:30:45 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:30:46.070 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:30:45 smithi115 ceph-mon[78593]: pgmap v5841: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:30:46.070 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:30:45 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:30:46.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:30:45 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:30:46.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:30:45 smithi047 ceph-mon[77147]: pgmap v5841: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:30:46.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:30:45 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:30:46.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:30:46 smithi115 conmon[90393]: 2022-05-14T02:30:46.068+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:30:46.791 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:30:46 smithi047 conmon[78521]: 2022-05-14T02:30:46.319+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:30:46.792 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:30:46 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T02:30:46.792 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:30:46 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:30:46.792 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:30:46 smithi047 conmon[78521]: yield 2022-05-14T02:30:46.793 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:30:46 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:30:46.793 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:30:46 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:30:46.793 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:30:46 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:30:46.793 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:30:46 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T02:30:46.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:30:46 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:30:46.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:30:46 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T02:30:46.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:30:46 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:30:46.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:30:46 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T02:30:46.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:30:46 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:30:46.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:30:46 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T02:30:46.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:30:46 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:30:46.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:30:46 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T02:30:46.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:30:46 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:30:46.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:30:46 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:30:46.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:30:46 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:30:47.076 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:30:46 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T02:30:47.077 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:30:46 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T02:30:47.077 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:30:46 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T02:30:47.078 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:30:46 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:30:47.078 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:30:46 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:30:47.079 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T02:30:47.079 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:30:47.080 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T02:30:47.080 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:30:47.080 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:30:47.081 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:30:47.081 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T02:30:47.082 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:30:47.082 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T02:30:47.083 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:30:47.083 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T02:30:47.083 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:30:47.084 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T02:30:47.084 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:30:47.084 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T02:30:47.085 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:30:47.085 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:30:47.086 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:30:47.086 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:30:46 smithi115 ceph-mon[78593]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T02:30:47.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:30:46 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T02:30:47.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:30:46 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T02:30:47.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:30:46 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T02:30:47.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:30:46 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:30:47.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:30:46 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:30:47.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T02:30:47.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:30:47.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T02:30:47.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:30:47.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:30:47.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:30:47.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T02:30:47.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:30:47.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T02:30:47.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:30:47.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T02:30:47.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:30:47.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T02:30:47.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:30:47.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T02:30:47.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:30:47.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:30:47.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:30:47.297 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:30:46 smithi047 ceph-mon[77147]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T02:30:47.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:30:47 smithi115 conmon[90393]: 2022-05-14T02:30:47.076+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:30:48.095 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:30:47 smithi115 ceph-mon[78593]: pgmap v5842: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:30:48.096 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:30:47 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:30:48.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:30:47 smithi047 ceph-mon[77147]: pgmap v5842: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:30:48.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:30:47 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:30:48.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:30:48 smithi115 conmon[90393]: 2022-05-14T02:30:48.095+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:30:49.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:30:48 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:30:49.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:30:48 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:30:49.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:30:49 smithi115 conmon[90393]: 2022-05-14T02:30:49.142+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:30:49.792 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:30:49 smithi047 conmon[78521]: 2022-05-14T02:30:49.392+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:30:49.792 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:30:49 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T02:30:49.793 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:30:49 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:30:49.793 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:30:49 smithi047 conmon[78521]: yield 2022-05-14T02:30:49.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:30:49 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:30:49.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:30:49 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:30:49.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:30:49 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:30:49.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:30:49 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T02:30:49.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:30:49 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:30:49.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:30:49 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T02:30:49.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:30:49 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:30:49.797 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:30:49 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T02:30:49.797 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:30:49 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:30:49.797 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:30:49 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T02:30:49.798 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:30:49 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:30:49.798 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:30:49 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T02:30:49.798 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:30:49 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:30:49.798 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:30:49 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:30:49.798 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:30:49 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:30:49.799 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:30:49 smithi047 conmon[78521]: 2022-05-14T02:30:49.392+0000 7f5341f7c700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T02:30:49.799 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:30:49 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T02:30:49.799 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:30:49 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:30:49.799 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:30:49 smithi047 conmon[78521]: yield 2022-05-14T02:30:49.800 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:30:49 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:30:49.800 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:30:49 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:30:49.800 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:30:49 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:30:49.800 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:30:49 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T02:30:49.801 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:30:49 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:30:49.801 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:30:49 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T02:30:49.801 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:30:49 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:30:49.801 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:30:49 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T02:30:49.802 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:30:49 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:30:49.802 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:30:49 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T02:30:49.802 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:30:49 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:30:49.802 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:30:49 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T02:30:49.803 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:30:49 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:30:49.803 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:30:49 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:30:49.803 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:30:49 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:30:49.803 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:30:49 smithi047 conmon[78521]: 2022-05-14T02:30:49.804 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:30:49 smithi047 conmon[78521]: During handling of the above exception, another exception occurred: 2022-05-14T02:30:49.804 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:30:49 smithi047 conmon[78521]: 2022-05-14T02:30:49.804 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:30:49 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T02:30:49.804 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:30:49 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T02:30:49.804 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:30:49 smithi047 conmon[78521]: return f(*arg) 2022-05-14T02:30:49.805 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:30:49 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T02:30:49.805 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:30:49 smithi047 conmon[78521]: self._write_client_files(client_files, host) 2022-05-14T02:30:49.805 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:30:49 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T02:30:49.806 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:30:49 smithi047 conmon[78521]: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T02:30:49.806 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:30:49 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T02:30:49.806 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:30:49 smithi047 conmon[78521]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T02:30:49.806 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:30:49 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T02:30:49.807 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:30:49 smithi047 conmon[78521]: return self.event_loop.get_result(coro) 2022-05-14T02:30:49.807 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:30:49 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T02:30:49.807 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:30:49 smithi047 conmon[78521]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T02:30:49.807 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:30:49 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T02:30:49.808 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:30:49 smithi047 conmon[78521]: return self.__get_result() 2022-05-14T02:30:49.808 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:30:49 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T02:30:49.808 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:30:49 smithi047 conmon[78521]: raise self._exception 2022-05-14T02:30:49.808 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:30:49 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T02:30:49.809 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:30:49 smithi047 conmon[78521]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T02:30:49.809 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:30:49 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T02:30:49.809 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:30:49 smithi047 conmon[78521]: conn = await self._remote_connection(host, addr) 2022-05-14T02:30:49.810 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:30:49 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T02:30:49.810 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:30:49 smithi047 conmon[78521]: raise 2022-05-14T02:30:49.811 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:30:49 smithi047 conmon[78521]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T02:30:49.811 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:30:49 smithi047 conmon[78521]: self.gen.throw(type, value, traceback) 2022-05-14T02:30:49.812 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:30:49 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T02:30:49.812 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:30:49 smithi047 conmon[78521]: raise OrchestratorError(msg) 2022-05-14T02:30:49.813 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:30:49 smithi047 conmon[78521]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:30:50.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:30:49 smithi115 ceph-mon[78593]: pgmap v5843: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:30:50.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:30:49 smithi047 ceph-mon[77147]: pgmap v5843: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:30:50.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:30:50 smithi115 conmon[90393]: 2022-05-14T02:30:50.133+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:30:51.102 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:30:50 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:30:51.102 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:30:50 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:30:51.102 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T02:30:51.103 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:30:51.103 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T02:30:51.103 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:30:51.103 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:30:51.104 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:30:51.104 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T02:30:51.104 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:30:51.104 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T02:30:51.105 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:30:51.105 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T02:30:51.105 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:30:51.105 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T02:30:51.105 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:30:51.106 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T02:30:51.106 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:30:51.106 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:30:51.106 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:30:51.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:30:50 smithi115 ceph-mon[78593]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T02:30:51.107 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T02:30:51.107 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:30:51.107 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T02:30:51.107 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:30:51.108 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:30:51.108 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:30:51.108 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T02:30:51.108 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:30:51.109 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T02:30:51.109 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:30:51.109 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T02:30:51.109 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:30:51.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T02:30:51.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:30:51.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T02:30:51.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:30:51.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:30:51.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:30:51.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T02:30:51.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: During handling of the above exception, another exception occurred: 2022-05-14T02:30:51.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T02:30:51.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T02:30:51.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T02:30:51.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: return f(*arg) 2022-05-14T02:30:51.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T02:30:51.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: self._write_client_files(client_files, host) 2022-05-14T02:30:51.113 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T02:30:51.113 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T02:30:51.113 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T02:30:51.113 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T02:30:51.114 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T02:30:51.114 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.event_loop.get_result(coro) 2022-05-14T02:30:51.114 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T02:30:51.114 INFO:journalctl@ceph.mon.c.smithi115.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T02:30:51.114 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T02:30:51.115 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.__get_result() 2022-05-14T02:30:51.115 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T02:30:51.115 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise self._exception 2022-05-14T02:30:51.115 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T02:30:51.116 INFO:journalctl@ceph.mon.c.smithi115.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T02:30:51.116 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T02:30:51.116 INFO:journalctl@ceph.mon.c.smithi115.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T02:30:51.116 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T02:30:51.116 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise 2022-05-14T02:30:51.117 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T02:30:51.117 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.gen.throw(type, value, traceback) 2022-05-14T02:30:51.117 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T02:30:51.117 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OrchestratorError(msg) 2022-05-14T02:30:51.118 INFO:journalctl@ceph.mon.c.smithi115.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:30:51.118 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:30:50 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:30:51.118 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:30:50 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:30:51.118 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:30:50 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 10980 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:30:51.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:30:50 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:30:51.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:30:50 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:30:51.292 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T02:30:51.292 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:30:51.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T02:30:51.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:30:51.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:30:51.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:30:51.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T02:30:51.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:30:51.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T02:30:51.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:30:51.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T02:30:51.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:30:51.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T02:30:51.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:30:51.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T02:30:51.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:30:51.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:30:51.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:30:51.296 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:30:50 smithi047 ceph-mon[77147]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T02:30:51.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T02:30:51.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:30:51.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T02:30:51.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:30:51.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:30:51.298 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:30:51.298 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T02:30:51.298 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:30:51.298 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T02:30:51.298 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:30:51.299 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T02:30:51.299 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:30:51.299 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T02:30:51.299 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:30:51.299 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T02:30:51.300 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:30:51.300 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:30:51.300 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:30:51.300 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T02:30:51.300 INFO:journalctl@ceph.mon.b.smithi047.stdout: During handling of the above exception, another exception occurred: 2022-05-14T02:30:51.301 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T02:30:51.301 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T02:30:51.301 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T02:30:51.301 INFO:journalctl@ceph.mon.b.smithi047.stdout: return f(*arg) 2022-05-14T02:30:51.302 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T02:30:51.302 INFO:journalctl@ceph.mon.b.smithi047.stdout: self._write_client_files(client_files, host) 2022-05-14T02:30:51.302 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T02:30:51.302 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T02:30:51.302 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T02:30:51.303 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T02:30:51.303 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T02:30:51.303 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.event_loop.get_result(coro) 2022-05-14T02:30:51.303 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T02:30:51.303 INFO:journalctl@ceph.mon.b.smithi047.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T02:30:51.304 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T02:30:51.304 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.__get_result() 2022-05-14T02:30:51.304 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T02:30:51.304 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise self._exception 2022-05-14T02:30:51.304 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T02:30:51.305 INFO:journalctl@ceph.mon.b.smithi047.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T02:30:51.305 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T02:30:51.305 INFO:journalctl@ceph.mon.b.smithi047.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T02:30:51.305 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T02:30:51.306 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise 2022-05-14T02:30:51.306 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T02:30:51.306 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.gen.throw(type, value, traceback) 2022-05-14T02:30:51.306 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T02:30:51.306 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OrchestratorError(msg) 2022-05-14T02:30:51.307 INFO:journalctl@ceph.mon.b.smithi047.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:30:51.307 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:30:50 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:30:51.307 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:30:50 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:30:51.307 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:30:50 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 10980 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:30:51.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:30:51 smithi115 conmon[90393]: 2022-05-14T02:30:51.100+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:30:52.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:30:51 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:30:52.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:30:51 smithi115 ceph-mon[78593]: pgmap v5844: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:30:52.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:30:51 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:30:52.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:30:51 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:30:52.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:30:51 smithi047 ceph-mon[77147]: pgmap v5844: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:30:52.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:30:51 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:30:52.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:30:52 smithi115 conmon[90393]: 2022-05-14T02:30:52.145+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:30:53.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:30:53 smithi115 conmon[90393]: 2022-05-14T02:30:53.192+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:30:54.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:30:53 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:30:54.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:30:53 smithi115 ceph-mon[78593]: pgmap v5845: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:30:54.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:30:53 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:30:54.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:30:53 smithi047 ceph-mon[77147]: pgmap v5845: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:30:54.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:30:54 smithi115 conmon[90393]: 2022-05-14T02:30:54.212+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:30:55.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:30:54 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:30:55.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:30:54 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:30:55.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:30:54 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:30:55.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:30:54 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:30:55.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:30:54 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:30:55.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:30:54 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:30:55.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:30:55 smithi115 conmon[90393]: 2022-05-14T02:30:55.242+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:30:56.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:30:55 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:30:56.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:30:55 smithi115 ceph-mon[78593]: pgmap v5846: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:30:56.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:30:55 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 10990 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:30:56.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:30:55 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:30:56.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:30:55 smithi047 ceph-mon[77147]: pgmap v5846: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:30:56.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:30:55 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 10990 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:30:56.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:30:56 smithi115 conmon[90393]: 2022-05-14T02:30:56.256+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:30:57.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:30:56 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:30:57.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:30:56 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:30:57.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:30:57 smithi115 conmon[90393]: 2022-05-14T02:30:57.229+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:30:58.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:30:57 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:30:58.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:30:57 smithi115 ceph-mon[78593]: pgmap v5847: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:30:58.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:30:57 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:30:58.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:30:57 smithi047 ceph-mon[77147]: pgmap v5847: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:30:58.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:30:58 smithi115 conmon[90393]: 2022-05-14T02:30:58.248+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:30:59.268 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:30:58 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:30:59.268 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:30:58 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:30:59.269 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:30:58 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:30:59.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:30:58 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:30:59.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:30:58 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:30:59.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:30:58 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:30:59.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:30:59 smithi115 conmon[90393]: 2022-05-14T02:30:59.266+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:31:00.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:30:59 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:31:00.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:30:59 smithi047 ceph-mon[77147]: pgmap v5848: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:31:00.312 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:30:59 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:31:00.312 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:30:59 smithi115 ceph-mon[78593]: pgmap v5848: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:31:00.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:31:00 smithi115 conmon[90393]: 2022-05-14T02:31:00.310+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:31:01.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:31:00 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:31:01.306 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:31:00 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:31:01.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:31:01 smithi115 conmon[90393]: 2022-05-14T02:31:01.305+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:31:02.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:31:01 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:31:02.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:31:01 smithi047 ceph-mon[77147]: pgmap v5849: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:31:02.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:31:01 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 10995 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:31:02.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:31:01 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:31:02.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:31:01 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:31:02.316 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:31:01 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:31:02.316 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:31:01 smithi115 ceph-mon[78593]: pgmap v5849: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:31:02.316 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:31:01 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 10995 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:31:02.317 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:31:01 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:31:02.317 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:31:01 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:31:02.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:31:02 smithi115 conmon[90393]: 2022-05-14T02:31:02.314+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:31:03.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:31:02 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:31:03.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:31:02 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:31:03.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:31:03 smithi115 conmon[90393]: 2022-05-14T02:31:03.361+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:31:04.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:31:03 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:31:04.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:31:03 smithi047 ceph-mon[77147]: pgmap v5850: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:31:04.352 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:31:03 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:31:04.353 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:31:03 smithi115 ceph-mon[78593]: pgmap v5850: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:31:04.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:31:04 smithi115 conmon[90393]: 2022-05-14T02:31:04.351+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:31:05.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:31:04 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:31:05.346 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:31:04 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:31:05.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:31:05 smithi115 conmon[90393]: 2022-05-14T02:31:05.344+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:31:06.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:31:05 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:31:06.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:31:05 smithi047 ceph-mon[77147]: pgmap v5851: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:31:06.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:31:05 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:31:06.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:31:05 smithi115 ceph-mon[78593]: pgmap v5851: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:31:06.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:31:06 smithi115 conmon[90393]: 2022-05-14T02:31:06.373+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:31:07.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:31:06 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:31:07.355 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:31:06 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:31:07.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:31:07 smithi115 conmon[90393]: 2022-05-14T02:31:07.354+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:31:08.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:31:07 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:31:08.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:31:07 smithi047 ceph-mon[77147]: pgmap v5852: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:31:08.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:31:07 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:31:08.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:31:07 smithi115 ceph-mon[78593]: pgmap v5852: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:31:08.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:31:08 smithi115 conmon[90393]: 2022-05-14T02:31:08.363+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:31:09.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:31:08 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:31:09.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:31:08 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:31:09.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:31:09 smithi115 conmon[90393]: 2022-05-14T02:31:09.384+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:31:10.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:31:09 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:31:10.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:31:09 smithi047 ceph-mon[77147]: pgmap v5853: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:31:10.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:31:09 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:31:10.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:31:09 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:31:10.344 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:31:09 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:31:10.345 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:31:09 smithi115 ceph-mon[78593]: pgmap v5853: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:31:10.345 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:31:09 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:31:10.345 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:31:09 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:31:10.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:31:10 smithi115 conmon[90393]: 2022-05-14T02:31:10.343+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:31:11.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:31:10 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:31:11.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:31:10 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 11000 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:31:11.346 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:31:10 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:31:11.347 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:31:10 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 11000 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:31:11.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:31:11 smithi115 conmon[90393]: 2022-05-14T02:31:11.344+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:31:12.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:31:11 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:31:12.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:31:11 smithi047 ceph-mon[77147]: pgmap v5854: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:31:12.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:31:11 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:31:12.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:31:11 smithi115 ceph-mon[78593]: pgmap v5854: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:31:12.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:31:12 smithi115 conmon[90393]: 2022-05-14T02:31:12.360+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:31:13.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:31:12 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:31:13.356 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:31:12 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:31:13.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:31:13 smithi115 conmon[90393]: 2022-05-14T02:31:13.355+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:31:14.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:31:13 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:31:14.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:31:13 smithi047 ceph-mon[77147]: pgmap v5855: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:31:14.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:31:13 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:31:14.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:31:13 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:31:14.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:31:13 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:31:14.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:31:13 smithi115 ceph-mon[78593]: pgmap v5855: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:31:14.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:31:13 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:31:14.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:31:13 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:31:14.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:31:14 smithi115 conmon[90393]: 2022-05-14T02:31:14.384+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:31:15.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:31:14 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:31:15.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:31:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T02:31:15.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:31:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T02:31:15.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:31:14 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:31:15.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:31:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T02:31:15.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:31:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T02:31:15.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:31:15 smithi115 conmon[90393]: 2022-05-14T02:31:15.379+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:31:16.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:31:15 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:31:16.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:31:15 smithi047 ceph-mon[77147]: pgmap v5856: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:31:16.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:31:15 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 11010 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:31:16.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:31:15 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:31:16.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:31:15 smithi115 ceph-mon[78593]: pgmap v5856: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:31:16.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:31:15 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 11010 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:31:16.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:31:16 smithi115 conmon[90393]: 2022-05-14T02:31:16.410+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:31:17.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:31:16 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:31:17.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:31:16 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:31:17.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:31:17 smithi115 conmon[90393]: 2022-05-14T02:31:17.448+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:31:18.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:31:17 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:31:18.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:31:17 smithi047 ceph-mon[77147]: pgmap v5857: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:31:18.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:31:17 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:31:18.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:31:17 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:31:18.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:31:17 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:31:18.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:31:17 smithi115 ceph-mon[78593]: pgmap v5857: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:31:18.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:31:17 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:31:18.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:31:17 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:31:18.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:31:18 smithi115 conmon[90393]: 2022-05-14T02:31:18.451+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:31:19.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:31:18 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:31:19.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:31:18 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:31:19.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:31:19 smithi115 conmon[90393]: 2022-05-14T02:31:19.480+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:31:20.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:31:19 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:31:20.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:31:19 smithi047 ceph-mon[77147]: pgmap v5858: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:31:20.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:31:19 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:31:20.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:31:19 smithi115 ceph-mon[78593]: pgmap v5858: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:31:20.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:31:20 smithi115 conmon[90393]: 2022-05-14T02:31:20.489+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:31:21.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:31:20 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:31:21.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:31:20 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:31:21.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:31:21 smithi115 conmon[90393]: 2022-05-14T02:31:21.464+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:31:22.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:31:21 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:31:22.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:31:21 smithi047 ceph-mon[77147]: pgmap v5859: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:31:22.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:31:21 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 11015 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:31:22.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:31:21 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:31:22.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:31:21 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:31:22.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:31:21 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:31:22.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:31:21 smithi115 ceph-mon[78593]: pgmap v5859: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:31:22.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:31:21 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 11015 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:31:22.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:31:21 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:31:22.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:31:21 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:31:22.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:31:22 smithi115 conmon[90393]: 2022-05-14T02:31:22.429+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:31:23.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:31:22 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:31:23.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:31:22 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:31:23.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:31:23 smithi115 conmon[90393]: 2022-05-14T02:31:23.440+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:31:24.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:31:23 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:31:24.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:31:23 smithi047 ceph-mon[77147]: pgmap v5860: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:31:24.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:31:23 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:31:24.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:31:23 smithi115 ceph-mon[78593]: pgmap v5860: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:31:24.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:31:24 smithi115 conmon[90393]: 2022-05-14T02:31:24.433+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:31:25.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:31:24 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:31:25.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:31:24 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:31:25.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:31:25 smithi115 conmon[90393]: 2022-05-14T02:31:25.481+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:31:26.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:31:25 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:31:26.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:31:25 smithi047 ceph-mon[77147]: pgmap v5861: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:31:26.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:31:25 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:31:26.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:31:25 smithi115 ceph-mon[78593]: pgmap v5861: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:31:26.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:31:26 smithi115 conmon[90393]: 2022-05-14T02:31:26.443+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:31:27.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:31:26 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:31:27.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:31:26 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:31:27.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:31:27 smithi115 conmon[90393]: 2022-05-14T02:31:27.469+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:31:28.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:31:27 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:31:28.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:31:27 smithi047 ceph-mon[77147]: pgmap v5862: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:31:28.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:31:27 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:31:28.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:31:27 smithi115 ceph-mon[78593]: pgmap v5862: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:31:28.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:31:28 smithi115 conmon[90393]: 2022-05-14T02:31:28.446+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:31:29.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:31:28 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:31:29.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:31:28 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:31:29.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:31:29 smithi115 conmon[90393]: 2022-05-14T02:31:29.440+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:31:30.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:31:29 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:31:30.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:31:29 smithi047 ceph-mon[77147]: pgmap v5863: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:31:30.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:31:29 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:31:30.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:31:29 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:31:30.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:31:29 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:31:30.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:31:29 smithi115 ceph-mon[78593]: pgmap v5863: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:31:30.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:31:29 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:31:30.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:31:29 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:31:30.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:31:30 smithi115 conmon[90393]: 2022-05-14T02:31:30.396+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:31:31.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:31:30 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:31:31.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:31:30 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 11020 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:31:31.356 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:31:30 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:31:31.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:31:30 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 11020 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:31:31.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:31:31 smithi115 conmon[90393]: 2022-05-14T02:31:31.355+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:31:32.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:31:31 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:31:32.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:31:31 smithi047 ceph-mon[77147]: pgmap v5864: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:31:32.333 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:31:31 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:31:32.333 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:31:31 smithi115 ceph-mon[78593]: pgmap v5864: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:31:32.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:31:32 smithi115 conmon[90393]: 2022-05-14T02:31:32.332+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:31:33.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:31:32 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:31:33.339 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:31:32 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:31:33.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:31:33 smithi115 conmon[90393]: 2022-05-14T02:31:33.338+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:31:34.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:31:33 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:31:34.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:31:33 smithi047 ceph-mon[77147]: pgmap v5865: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:31:34.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:31:33 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:31:34.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:31:33 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:31:34.330 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:31:33 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:31:34.331 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:31:33 smithi115 ceph-mon[78593]: pgmap v5865: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:31:34.331 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:31:33 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:31:34.331 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:31:33 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:31:34.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:31:34 smithi115 conmon[90393]: 2022-05-14T02:31:34.329+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:31:35.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:31:34 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:31:35.332 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:31:34 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:31:35.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:31:35 smithi115 conmon[90393]: 2022-05-14T02:31:35.331+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:31:36.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:31:35 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:31:36.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:31:35 smithi047 ceph-mon[77147]: pgmap v5866: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:31:36.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:31:35 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 11030 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:31:36.315 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:31:35 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:31:36.316 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:31:35 smithi115 ceph-mon[78593]: pgmap v5866: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:31:36.316 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:31:35 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 11030 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:31:36.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:31:36 smithi115 conmon[90393]: 2022-05-14T02:31:36.314+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:31:37.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:31:36 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:31:37.318 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:31:36 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:31:37.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:31:37 smithi115 conmon[90393]: 2022-05-14T02:31:37.317+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:31:38.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:31:37 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:31:38.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:31:37 smithi047 ceph-mon[77147]: pgmap v5867: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:31:38.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:31:37 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:31:38.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:31:37 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:31:38.299 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:31:37 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:31:38.299 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:31:37 smithi115 ceph-mon[78593]: pgmap v5867: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:31:38.300 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:31:37 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:31:38.300 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:31:37 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:31:38.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:31:38 smithi115 conmon[90393]: 2022-05-14T02:31:38.298+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:31:39.260 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:31:38 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:31:39.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:31:38 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:31:39.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:31:39 smithi115 conmon[90393]: 2022-05-14T02:31:39.259+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:31:40.248 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:31:39 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:31:40.249 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:31:39 smithi115 ceph-mon[78593]: pgmap v5868: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:31:40.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:31:39 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:31:40.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:31:39 smithi047 ceph-mon[77147]: pgmap v5868: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:31:40.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:31:40 smithi115 conmon[90393]: 2022-05-14T02:31:40.248+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:31:41.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:31:40 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:31:41.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:31:41 smithi115 conmon[90393]: 2022-05-14T02:31:41.248+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:31:41.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:31:40 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:31:42.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:31:41 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:31:42.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:31:41 smithi047 ceph-mon[77147]: pgmap v5869: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:31:42.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:31:41 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 11035 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:31:42.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:31:41 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:31:42.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:31:41 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:31:42.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:31:42 smithi115 conmon[90393]: 2022-05-14T02:31:42.208+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:31:42.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:31:41 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:31:42.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:31:41 smithi115 ceph-mon[78593]: pgmap v5869: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:31:42.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:31:41 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 11035 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:31:42.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:31:41 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:31:42.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:31:42 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:31:43.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:31:43 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:31:43.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:31:43 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:31:43.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:31:43 smithi115 conmon[90393]: 2022-05-14T02:31:43.241+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:31:44.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:31:44 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:31:44.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:31:44 smithi047 ceph-mon[77147]: pgmap v5870: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:31:44.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:31:44 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:31:44.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:31:44 smithi115 ceph-mon[78593]: pgmap v5870: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:31:44.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:31:44 smithi115 conmon[90393]: 2022-05-14T02:31:44.220+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:31:45.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:31:45 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:31:45.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:31:45 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:31:45.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:31:45 smithi115 conmon[90393]: 2022-05-14T02:31:45.238+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:31:46.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:31:46 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:31:46.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:31:46 smithi047 ceph-mon[77147]: pgmap v5871: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:31:46.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:31:46 smithi115 conmon[90393]: 2022-05-14T02:31:46.241+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:31:46.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:31:46 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:31:46.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:31:46 smithi115 ceph-mon[78593]: pgmap v5871: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:31:47.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:31:47 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:31:47.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:31:47 smithi115 conmon[90393]: 2022-05-14T02:31:47.257+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:31:47.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:31:47 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:31:48.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:31:48 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:31:48.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:31:48 smithi047 ceph-mon[77147]: pgmap v5872: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:31:48.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:31:48 smithi115 conmon[90393]: 2022-05-14T02:31:48.259+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:31:48.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:31:48 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:31:48.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:31:48 smithi115 ceph-mon[78593]: pgmap v5872: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:31:49.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:31:49 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:31:49.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:31:49 smithi115 conmon[90393]: 2022-05-14T02:31:49.212+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:31:49.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:31:49 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:31:50.164 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:31:50 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:31:50.165 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:31:50 smithi115 ceph-mon[78593]: pgmap v5873: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:31:50.165 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:31:50 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:31:50.165 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:31:50 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:31:50.165 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:31:50 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T02:31:50.165 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:31:50 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T02:31:50.166 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:31:50 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T02:31:50.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:31:50 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:31:50.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:31:50 smithi047 ceph-mon[77147]: pgmap v5873: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:31:50.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:31:50 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:31:50.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:31:50 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:31:50.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:31:50 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T02:31:50.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:31:50 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T02:31:50.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:31:50 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T02:31:50.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:31:50 smithi115 conmon[90393]: 2022-05-14T02:31:50.162+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:31:51.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:31:51 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:31:51.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:31:51 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 11040 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:31:51.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:31:51 smithi115 conmon[90393]: 2022-05-14T02:31:51.206+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:31:51.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:31:51 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:31:51.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:31:51 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 11040 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:31:52.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:31:52 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:31:52.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:31:52 smithi047 ceph-mon[77147]: pgmap v5874: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:31:52.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:31:52 smithi115 conmon[90393]: 2022-05-14T02:31:52.211+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:31:52.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:31:52 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:31:52.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:31:52 smithi115 ceph-mon[78593]: pgmap v5874: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:31:52.791 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:31:52 smithi047 conmon[78521]: 2022-05-14T02:31:52.495+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:31:52.792 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:31:52 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T02:31:52.792 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:31:52 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:31:52.792 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:31:52 smithi047 conmon[78521]: yield 2022-05-14T02:31:52.793 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:31:52 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:31:52.793 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:31:52 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:31:52.793 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:31:52 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:31:52.793 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:31:52 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T02:31:52.793 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:31:52 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:31:52.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:31:52 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T02:31:52.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:31:52 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:31:52.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:31:52 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T02:31:52.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:31:52 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:31:52.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:31:52 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T02:31:52.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:31:52 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:31:52.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:31:52 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T02:31:52.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:31:52 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:31:52.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:31:52 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:31:52.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:31:52 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:31:53.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:31:53 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:31:53.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:31:53 smithi115 conmon[90393]: 2022-05-14T02:31:53.177+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:31:53.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:31:53 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:31:54.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:31:54 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:31:54.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:31:54 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:31:54.292 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T02:31:54.292 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:31:54.292 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T02:31:54.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:31:54.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:31:54.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:31:54.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T02:31:54.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:31:54.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T02:31:54.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:31:54.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T02:31:54.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:31:54.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T02:31:54.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:31:54.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T02:31:54.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:31:54.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:31:54.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:31:54.296 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:31:54 smithi047 ceph-mon[77147]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T02:31:54.296 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:31:54 smithi047 ceph-mon[77147]: pgmap v5875: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:31:54.297 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:31:54 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:31:54.297 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:31:54 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:31:54.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:31:54 smithi115 conmon[90393]: 2022-05-14T02:31:54.146+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:31:54.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:31:54 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:31:54.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:31:54 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:31:54.359 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T02:31:54.359 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:31:54.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T02:31:54.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:31:54.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:31:54.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:31:54.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T02:31:54.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:31:54.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T02:31:54.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:31:54.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T02:31:54.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:31:54.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T02:31:54.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:31:54.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T02:31:54.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:31:54.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:31:54.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:31:54.363 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:31:54 smithi115 ceph-mon[78593]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T02:31:54.363 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:31:54 smithi115 ceph-mon[78593]: pgmap v5875: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:31:54.363 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:31:54 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:31:54.364 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:31:54 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:31:55.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:31:55 smithi115 conmon[90393]: 2022-05-14T02:31:55.166+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:31:55.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:31:55 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:31:55.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:31:55 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:31:56.042 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:31:55 smithi047 conmon[78521]: 2022-05-14T02:31:55.568+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:31:56.042 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:31:55 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T02:31:56.042 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:31:55 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:31:56.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:31:55 smithi047 conmon[78521]: yield 2022-05-14T02:31:56.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:31:55 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:31:56.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:31:55 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:31:56.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:31:55 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:31:56.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:31:55 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T02:31:56.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:31:55 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:31:56.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:31:55 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T02:31:56.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:31:55 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:31:56.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:31:55 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T02:31:56.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:31:55 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:31:56.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:31:55 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T02:31:56.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:31:55 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:31:56.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:31:55 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T02:31:56.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:31:55 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:31:56.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:31:55 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:31:56.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:31:55 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:31:56.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:31:55 smithi047 conmon[78521]: 2022-05-14T02:31:55.568+0000 7f5345f84700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T02:31:56.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:31:55 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T02:31:56.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:31:55 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:31:56.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:31:55 smithi047 conmon[78521]: yield 2022-05-14T02:31:56.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:31:55 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:31:56.048 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:31:55 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:31:56.048 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:31:55 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:31:56.048 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:31:55 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T02:31:56.048 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:31:55 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:31:56.049 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:31:55 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T02:31:56.049 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:31:55 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:31:56.049 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:31:55 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T02:31:56.049 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:31:55 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:31:56.049 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:31:55 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T02:31:56.050 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:31:55 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:31:56.050 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:31:55 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T02:31:56.050 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:31:55 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:31:56.050 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:31:55 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:31:56.051 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:31:55 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:31:56.051 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:31:55 smithi047 conmon[78521]: 2022-05-14T02:31:56.051 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:31:55 smithi047 conmon[78521]: During handling of the above exception, another exception occurred: 2022-05-14T02:31:56.051 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:31:55 smithi047 conmon[78521]: 2022-05-14T02:31:56.052 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:31:55 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T02:31:56.052 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:31:55 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T02:31:56.052 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:31:55 smithi047 conmon[78521]: return f(*arg) 2022-05-14T02:31:56.052 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:31:55 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T02:31:56.053 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:31:55 smithi047 conmon[78521]: self._write_client_files(client_files, host) 2022-05-14T02:31:56.053 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:31:55 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T02:31:56.053 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:31:55 smithi047 conmon[78521]: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T02:31:56.053 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:31:55 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T02:31:56.054 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:31:55 smithi047 conmon[78521]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T02:31:56.054 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:31:55 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T02:31:56.054 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:31:55 smithi047 conmon[78521]: return self.event_loop.get_result(coro) 2022-05-14T02:31:56.054 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:31:55 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T02:31:56.054 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:31:55 smithi047 conmon[78521]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T02:31:56.055 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:31:55 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T02:31:56.055 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:31:55 smithi047 conmon[78521]: return self.__get_result() 2022-05-14T02:31:56.055 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:31:55 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T02:31:56.055 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:31:55 smithi047 conmon[78521]: raise self._exception 2022-05-14T02:31:56.056 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:31:55 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T02:31:56.056 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:31:55 smithi047 conmon[78521]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T02:31:56.056 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:31:55 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T02:31:56.056 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:31:55 smithi047 conmon[78521]: conn = await self._remote_connection(host, addr) 2022-05-14T02:31:56.057 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:31:55 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T02:31:56.057 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:31:55 smithi047 conmon[78521]: raise 2022-05-14T02:31:56.057 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:31:55 smithi047 conmon[78521]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T02:31:56.057 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:31:55 smithi047 conmon[78521]: self.gen.throw(type, value, traceback) 2022-05-14T02:31:56.058 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:31:55 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T02:31:56.058 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:31:55 smithi047 conmon[78521]: raise OrchestratorError(msg) 2022-05-14T02:31:56.058 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:31:55 smithi047 conmon[78521]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:31:56.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:31:56 smithi115 conmon[90393]: 2022-05-14T02:31:56.148+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:31:56.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:31:56 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:31:56.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:31:56 smithi115 ceph-mon[78593]: pgmap v5876: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:31:56.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:31:56 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 11050 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:31:56.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:31:56 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:31:56.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:31:56 smithi047 ceph-mon[77147]: pgmap v5876: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:31:56.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:31:56 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 11050 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:31:57.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:31:57 smithi115 conmon[90393]: 2022-05-14T02:31:57.151+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:31:57.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:31:57 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:31:57.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:31:57 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:31:57.359 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T02:31:57.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:31:57.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T02:31:57.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:31:57.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:31:57.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:31:57.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T02:31:57.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:31:57.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T02:31:57.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:31:57.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T02:31:57.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:31:57.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T02:31:57.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:31:57.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T02:31:57.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:31:57.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:31:57.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:31:57.364 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:31:57 smithi115 ceph-mon[78593]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T02:31:57.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T02:31:57.365 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:31:57.365 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T02:31:57.365 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:31:57.365 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:31:57.366 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:31:57.366 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T02:31:57.366 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:31:57.367 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T02:31:57.367 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:31:57.367 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T02:31:57.367 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:31:57.368 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T02:31:57.368 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:31:57.368 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T02:31:57.368 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:31:57.368 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:31:57.369 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:31:57.369 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T02:31:57.369 INFO:journalctl@ceph.mon.c.smithi115.stdout: During handling of the above exception, another exception occurred: 2022-05-14T02:31:57.370 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T02:31:57.370 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T02:31:57.371 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T02:31:57.371 INFO:journalctl@ceph.mon.c.smithi115.stdout: return f(*arg) 2022-05-14T02:31:57.371 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T02:31:57.371 INFO:journalctl@ceph.mon.c.smithi115.stdout: self._write_client_files(client_files, host) 2022-05-14T02:31:57.372 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T02:31:57.372 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T02:31:57.372 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T02:31:57.372 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T02:31:57.372 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T02:31:57.373 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.event_loop.get_result(coro) 2022-05-14T02:31:57.373 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T02:31:57.373 INFO:journalctl@ceph.mon.c.smithi115.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T02:31:57.373 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T02:31:57.374 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.__get_result() 2022-05-14T02:31:57.374 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T02:31:57.374 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise self._exception 2022-05-14T02:31:57.374 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T02:31:57.375 INFO:journalctl@ceph.mon.c.smithi115.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T02:31:57.375 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T02:31:57.375 INFO:journalctl@ceph.mon.c.smithi115.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T02:31:57.375 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T02:31:57.375 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise 2022-05-14T02:31:57.376 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T02:31:57.376 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.gen.throw(type, value, traceback) 2022-05-14T02:31:57.376 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T02:31:57.376 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OrchestratorError(msg) 2022-05-14T02:31:57.377 INFO:journalctl@ceph.mon.c.smithi115.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:31:57.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:31:57 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:31:57.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:31:57 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:31:57.542 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T02:31:57.542 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:31:57.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T02:31:57.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:31:57.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:31:57.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:31:57.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T02:31:57.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:31:57.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T02:31:57.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:31:57.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T02:31:57.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:31:57.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T02:31:57.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:31:57.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T02:31:57.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:31:57.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:31:57.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:31:57.546 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:31:57 smithi047 ceph-mon[77147]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T02:31:57.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T02:31:57.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:31:57.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T02:31:57.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:31:57.548 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:31:57.548 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:31:57.548 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T02:31:57.548 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:31:57.548 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T02:31:57.549 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:31:57.549 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T02:31:57.549 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:31:57.549 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T02:31:57.549 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:31:57.550 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T02:31:57.550 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:31:57.550 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:31:57.550 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:31:57.550 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T02:31:57.551 INFO:journalctl@ceph.mon.b.smithi047.stdout: During handling of the above exception, another exception occurred: 2022-05-14T02:31:57.551 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T02:31:57.551 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T02:31:57.551 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T02:31:57.551 INFO:journalctl@ceph.mon.b.smithi047.stdout: return f(*arg) 2022-05-14T02:31:57.552 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T02:31:57.552 INFO:journalctl@ceph.mon.b.smithi047.stdout: self._write_client_files(client_files, host) 2022-05-14T02:31:57.552 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T02:31:57.552 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T02:31:57.553 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T02:31:57.553 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T02:31:57.553 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T02:31:57.553 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.event_loop.get_result(coro) 2022-05-14T02:31:57.554 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T02:31:57.554 INFO:journalctl@ceph.mon.b.smithi047.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T02:31:57.554 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T02:31:57.554 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.__get_result() 2022-05-14T02:31:57.554 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T02:31:57.555 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise self._exception 2022-05-14T02:31:57.555 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T02:31:57.555 INFO:journalctl@ceph.mon.b.smithi047.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T02:31:57.555 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T02:31:57.555 INFO:journalctl@ceph.mon.b.smithi047.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T02:31:57.556 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T02:31:57.556 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise 2022-05-14T02:31:57.556 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T02:31:57.556 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.gen.throw(type, value, traceback) 2022-05-14T02:31:57.557 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T02:31:57.557 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OrchestratorError(msg) 2022-05-14T02:31:57.557 INFO:journalctl@ceph.mon.b.smithi047.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:31:58.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:31:58 smithi115 conmon[90393]: 2022-05-14T02:31:58.184+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:31:58.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:31:58 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:31:58.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:31:58 smithi115 ceph-mon[78593]: pgmap v5877: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:31:58.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:31:58 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:31:58.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:31:58 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:31:58.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:31:58 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:31:58.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:31:58 smithi047 ceph-mon[77147]: pgmap v5877: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:31:58.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:31:58 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:31:58.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:31:58 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:31:59.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:31:59 smithi115 conmon[90393]: 2022-05-14T02:31:59.151+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:31:59.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:31:59 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:31:59.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:31:59 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:32:00.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:32:00 smithi115 conmon[90393]: 2022-05-14T02:32:00.167+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:32:00.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:32:00 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:32:00.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:32:00 smithi115 ceph-mon[78593]: pgmap v5878: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:32:00.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:32:00 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:32:00.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:32:00 smithi047 ceph-mon[77147]: pgmap v5878: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:32:01.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:32:01 smithi115 conmon[90393]: 2022-05-14T02:32:01.124+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:32:01.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:32:01 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:32:01.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:32:01 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:32:02.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:32:02 smithi115 conmon[90393]: 2022-05-14T02:32:02.167+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:32:02.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:32:02 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:32:02.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:32:02 smithi115 ceph-mon[78593]: pgmap v5879: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:32:02.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:32:02 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 11055 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:32:02.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:32:02 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:32:02.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:32:02 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:32:02.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:32:02 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:32:02.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:32:02 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:32:02.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:32:02 smithi047 ceph-mon[77147]: pgmap v5879: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:32:02.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:32:02 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 11055 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:32:02.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:32:02 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:32:02.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:32:02 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:32:02.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:32:02 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:32:03.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:32:03 smithi115 conmon[90393]: 2022-05-14T02:32:03.216+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:32:04.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:32:04 smithi115 conmon[90393]: 2022-05-14T02:32:04.176+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:32:04.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:32:04 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:32:04.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:32:04 smithi115 ceph-mon[78593]: pgmap v5880: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:32:04.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:32:04 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:32:04.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:32:04 smithi047 ceph-mon[77147]: pgmap v5880: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:32:05.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:32:05 smithi115 conmon[90393]: 2022-05-14T02:32:05.179+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:32:05.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:32:05 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:32:05.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:32:05 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:32:06.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:32:06 smithi115 conmon[90393]: 2022-05-14T02:32:06.132+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:32:06.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:32:06 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:32:06.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:32:06 smithi115 ceph-mon[78593]: pgmap v5881: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:32:06.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:32:06 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:32:06.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:32:06 smithi047 ceph-mon[77147]: pgmap v5881: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:32:07.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:32:07 smithi115 conmon[90393]: 2022-05-14T02:32:07.165+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:32:07.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:32:07 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:32:07.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:32:07 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:32:08.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:32:08 smithi115 conmon[90393]: 2022-05-14T02:32:08.135+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:32:08.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:32:08 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:32:08.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:32:08 smithi115 ceph-mon[78593]: pgmap v5882: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:32:08.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:32:08 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:32:08.589 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:32:08 smithi047 ceph-mon[77147]: pgmap v5882: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:32:09.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:32:09 smithi115 conmon[90393]: 2022-05-14T02:32:09.127+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:32:09.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:32:09 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:32:09.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:32:09 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:32:10.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:32:10 smithi115 conmon[90393]: 2022-05-14T02:32:10.082+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:32:10.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:32:10 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:32:10.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:32:10 smithi115 ceph-mon[78593]: pgmap v5883: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:32:10.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:32:10 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:32:10.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:32:10 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:32:10.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:32:10 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:32:10.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:32:10 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:32:10.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:32:10 smithi047 ceph-mon[77147]: pgmap v5883: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:32:10.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:32:10 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:32:10.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:32:10 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:32:10.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:32:10 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:32:11.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:32:11 smithi115 conmon[90393]: 2022-05-14T02:32:11.061+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:32:11.363 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:32:11 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:32:11.363 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:32:11 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 11060 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:32:11.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:32:11 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:32:11.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:32:11 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 11060 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:32:12.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:32:12 smithi115 conmon[90393]: 2022-05-14T02:32:12.076+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:32:12.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:32:12 smithi115 ceph-mon[78593]: pgmap v5884: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:32:12.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:32:12 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:32:12.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:32:12 smithi047 ceph-mon[77147]: pgmap v5884: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:32:12.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:32:12 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:32:13.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:32:13 smithi115 conmon[90393]: 2022-05-14T02:32:13.052+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:32:13.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:32:13 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:32:13.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:32:13 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:32:14.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:32:14 smithi115 conmon[90393]: 2022-05-14T02:32:14.098+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:32:14.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:32:14 smithi115 ceph-mon[78593]: pgmap v5885: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:32:14.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:32:14 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:32:14.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:32:14 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:32:14.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:32:14 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:32:14.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:32:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T02:32:14.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:32:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T02:32:14.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:32:14 smithi047 ceph-mon[77147]: pgmap v5885: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:32:14.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:32:14 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:32:14.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:32:14 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:32:14.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:32:14 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:32:14.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:32:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T02:32:14.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:32:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T02:32:15.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:32:15 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:32:15.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:32:15 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:32:15.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:32:15 smithi115 conmon[90393]: 2022-05-14T02:32:15.130+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:32:16.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:32:16 smithi047 ceph-mon[77147]: pgmap v5886: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:32:16.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:32:16 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 11071 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:32:16.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:32:16 smithi115 conmon[90393]: 2022-05-14T02:32:16.109+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:32:16.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:32:16 smithi115 ceph-mon[78593]: pgmap v5886: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:32:16.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:32:16 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 11071 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:32:17.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:32:17 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:32:17.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:32:17 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:32:17.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:32:17 smithi115 conmon[90393]: 2022-05-14T02:32:17.154+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:32:17.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:32:17 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:32:17.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:32:17 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:32:18.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:32:18 smithi047 ceph-mon[77147]: pgmap v5887: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:32:18.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:32:18 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:32:18.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:32:18 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:32:18.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:32:18 smithi115 conmon[90393]: 2022-05-14T02:32:18.198+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:32:18.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:32:18 smithi115 ceph-mon[78593]: pgmap v5887: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:32:18.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:32:18 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:32:18.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:32:18 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:32:19.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:32:19 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:32:19.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:32:19 smithi115 conmon[90393]: 2022-05-14T02:32:19.150+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:32:19.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:32:19 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:32:20.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:32:20 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:32:20.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:32:20 smithi047 ceph-mon[77147]: pgmap v5888: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:32:20.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:32:20 smithi115 conmon[90393]: 2022-05-14T02:32:20.166+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:32:20.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:32:20 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:32:20.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:32:20 smithi115 ceph-mon[78593]: pgmap v5888: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:32:21.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:32:21 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:32:21.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:32:21 smithi115 conmon[90393]: 2022-05-14T02:32:21.141+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:32:21.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:32:21 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:32:22.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:32:22 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:32:22.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:32:22 smithi047 ceph-mon[77147]: pgmap v5889: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 7 op/s 2022-05-14T02:32:22.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:32:22 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 11075 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:32:22.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:32:22 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:32:22.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:32:22 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:32:22.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:32:22 smithi115 conmon[90393]: 2022-05-14T02:32:22.116+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:32:22.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:32:22 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:32:22.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:32:22 smithi115 ceph-mon[78593]: pgmap v5889: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 7 op/s 2022-05-14T02:32:22.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:32:22 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 11075 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:32:22.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:32:22 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:32:22.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:32:22 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:32:23.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:32:23 smithi115 conmon[90393]: 2022-05-14T02:32:23.086+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:32:23.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:32:23 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:32:23.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:32:23 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:32:23.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:32:23 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:32:23.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:32:23 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:32:24.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:32:24 smithi047 ceph-mon[77147]: pgmap v5890: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 4 op/s 2022-05-14T02:32:24.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:32:24 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:32:24.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:32:24 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:32:24.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:32:24 smithi115 conmon[90393]: 2022-05-14T02:32:24.120+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:32:24.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:32:24 smithi115 ceph-mon[78593]: pgmap v5890: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 4 op/s 2022-05-14T02:32:24.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:32:24 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:32:24.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:32:24 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:32:25.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:32:25 smithi115 conmon[90393]: 2022-05-14T02:32:25.091+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:32:25.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:32:25 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:32:25.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:32:25 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:32:26.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:32:26 smithi115 conmon[90393]: 2022-05-14T02:32:26.063+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:32:26.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:32:26 smithi115 ceph-mon[78593]: pgmap v5891: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 7 op/s 2022-05-14T02:32:26.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:32:26 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:32:26.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:32:26 smithi047 ceph-mon[77147]: pgmap v5891: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 7 op/s 2022-05-14T02:32:26.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:32:26 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:32:27.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:32:27 smithi115 conmon[90393]: 2022-05-14T02:32:27.072+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:32:27.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:32:27 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:32:27.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:32:27 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:32:28.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:32:28 smithi115 conmon[90393]: 2022-05-14T02:32:28.071+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:32:28.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:32:28 smithi115 ceph-mon[78593]: pgmap v5892: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 4 op/s 2022-05-14T02:32:28.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:32:28 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:32:28.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:32:28 smithi047 ceph-mon[77147]: pgmap v5892: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 4 op/s 2022-05-14T02:32:28.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:32:28 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:32:29.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:32:29 smithi115 conmon[90393]: 2022-05-14T02:32:29.087+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:32:29.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:32:29 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:32:29.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:32:29 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:32:30.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:32:30 smithi047 ceph-mon[77147]: pgmap v5893: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 4 op/s 2022-05-14T02:32:30.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:32:30 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:32:30.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:32:30 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:32:30.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:32:30 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:32:30.611 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:32:30 smithi115 conmon[90393]: 2022-05-14T02:32:30.132+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:32:30.612 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:32:30 smithi115 ceph-mon[78593]: pgmap v5893: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 4 op/s 2022-05-14T02:32:30.613 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:32:30 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:32:30.613 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:32:30 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:32:30.613 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:32:30 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:32:31.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:32:31 smithi115 conmon[90393]: 2022-05-14T02:32:31.086+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:32:31.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:32:31 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 11079 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:32:31.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:32:31 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:32:31.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:32:31 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 11079 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:32:31.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:32:31 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:32:32.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:32:32 smithi115 conmon[90393]: 2022-05-14T02:32:32.060+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:32:32.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:32:32 smithi115 ceph-mon[78593]: pgmap v5894: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 7 op/s 2022-05-14T02:32:32.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:32:32 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:32:32.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:32:32 smithi047 ceph-mon[77147]: pgmap v5894: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 7 op/s 2022-05-14T02:32:32.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:32:32 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:32:33.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:32:33 smithi115 conmon[90393]: 2022-05-14T02:32:33.082+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:32:33.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:32:33 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:32:33.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:32:33 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:32:34.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:32:34 smithi115 conmon[90393]: 2022-05-14T02:32:34.050+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:32:34.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:32:34 smithi115 ceph-mon[78593]: pgmap v5895: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:32:34.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:32:34 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:32:34.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:32:34 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:32:34.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:32:34 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:32:34.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:32:34 smithi047 ceph-mon[77147]: pgmap v5895: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:32:34.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:32:34 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:32:34.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:32:34 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:32:34.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:32:34 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:32:35.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:32:35 smithi115 conmon[90393]: 2022-05-14T02:32:35.016+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:32:35.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:32:35 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:32:35.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:32:35 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:32:36.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:32:35 smithi115 conmon[90393]: 2022-05-14T02:32:35.994+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:32:36.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:32:36 smithi115 ceph-mon[78593]: pgmap v5896: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:32:36.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:32:36 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 11090 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:32:36.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:32:36 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:32:36.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:32:36 smithi047 ceph-mon[77147]: pgmap v5896: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:32:36.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:32:36 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 11090 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:32:36.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:32:36 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:32:37.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:32:37 smithi115 conmon[90393]: 2022-05-14T02:32:37.003+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:32:37.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:32:37 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:32:37.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:32:37 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:32:38.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:32:38 smithi115 conmon[90393]: 2022-05-14T02:32:37.999+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:32:38.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:32:38 smithi115 ceph-mon[78593]: pgmap v5897: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:32:38.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:32:38 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:32:38.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:32:38 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:32:38.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:32:38 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:32:38.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:32:38 smithi047 ceph-mon[77147]: pgmap v5897: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:32:38.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:32:38 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:32:38.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:32:38 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:32:38.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:32:38 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:32:39.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:32:38 smithi115 conmon[90393]: 2022-05-14T02:32:38.970+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:32:39.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:32:39 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:32:39.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:32:39 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:32:40.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:32:39 smithi115 conmon[90393]: 2022-05-14T02:32:39.995+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:32:40.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:32:40 smithi115 ceph-mon[78593]: pgmap v5898: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:32:40.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:32:40 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:32:40.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:32:40 smithi047 ceph-mon[77147]: pgmap v5898: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:32:40.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:32:40 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:32:41.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:32:40 smithi115 conmon[90393]: 2022-05-14T02:32:40.961+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:32:42.003 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:32:41 smithi115 ceph-mon[78593]: pgmap v5899: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:32:42.003 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:32:41 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:32:42.004 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:32:41 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 11095 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:32:42.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:32:41 smithi047 ceph-mon[77147]: pgmap v5899: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:32:42.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:32:41 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:32:42.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:32:41 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 11095 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:32:42.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:32:42 smithi115 conmon[90393]: 2022-05-14T02:32:42.001+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:32:43.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:32:42 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:32:43.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:32:42 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:32:43.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:32:42 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:32:43.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:32:42 smithi115 conmon[90393]: 2022-05-14T02:32:42.955+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:32:43.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:32:42 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:32:43.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:32:42 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:32:43.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:32:42 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:32:44.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:32:43 smithi047 ceph-mon[77147]: pgmap v5900: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:32:44.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:32:43 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:32:44.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:32:43 smithi115 conmon[90393]: 2022-05-14T02:32:43.947+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:32:44.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:32:43 smithi115 ceph-mon[78593]: pgmap v5900: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:32:44.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:32:43 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:32:45.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:32:44 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:32:45.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:32:44 smithi115 conmon[90393]: 2022-05-14T02:32:44.943+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:32:45.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:32:44 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:32:46.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:32:45 smithi047 ceph-mon[77147]: pgmap v5901: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:32:46.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:32:45 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:32:46.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:32:45 smithi115 conmon[90393]: 2022-05-14T02:32:45.955+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:32:46.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:32:45 smithi115 ceph-mon[78593]: pgmap v5901: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:32:46.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:32:45 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:32:47.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:32:46 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:32:47.114 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:32:46 smithi115 conmon[90393]: 2022-05-14T02:32:46.976+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:32:47.114 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:32:46 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:32:48.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:32:47 smithi047 ceph-mon[77147]: pgmap v5902: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:32:48.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:32:47 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:32:48.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:32:47 smithi115 conmon[90393]: 2022-05-14T02:32:47.964+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:32:48.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:32:47 smithi115 ceph-mon[78593]: pgmap v5902: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:32:48.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:32:47 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:32:49.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:32:48 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:32:49.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:32:48 smithi115 conmon[90393]: 2022-05-14T02:32:48.970+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:32:49.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:32:48 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:32:50.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:32:49 smithi047 ceph-mon[77147]: pgmap v5903: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:32:50.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:32:49 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:32:50.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:32:49 smithi115 conmon[90393]: 2022-05-14T02:32:49.995+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:32:50.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:32:49 smithi115 ceph-mon[78593]: pgmap v5903: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:32:50.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:32:49 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:32:51.034 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:32:50 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:32:51.034 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:32:50 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:32:51.034 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:32:50 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:32:51.035 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:32:50 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 11100 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:32:51.047 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:32:50 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:32:51.047 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:32:50 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:32:51.047 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:32:50 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:32:51.048 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:32:50 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 11100 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:32:51.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:32:51 smithi115 conmon[90393]: 2022-05-14T02:32:51.032+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:32:52.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:32:51 smithi047 ceph-mon[77147]: pgmap v5904: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:32:52.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:32:51 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:32:52.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:32:52 smithi115 conmon[90393]: 2022-05-14T02:32:52.014+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:32:52.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:32:51 smithi115 ceph-mon[78593]: pgmap v5904: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:32:52.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:32:51 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:32:53.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:32:52 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:32:53.053 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:32:52 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:32:53.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:32:53 smithi115 conmon[90393]: 2022-05-14T02:32:53.051+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:32:54.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:32:53 smithi047 ceph-mon[77147]: pgmap v5905: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:32:54.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:32:53 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:32:54.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:32:54 smithi115 conmon[90393]: 2022-05-14T02:32:54.015+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:32:54.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:32:53 smithi115 ceph-mon[78593]: pgmap v5905: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:32:54.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:32:53 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:32:55.064 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:32:54 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:32:55.065 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:32:54 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:32:55.065 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:32:54 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:32:55.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:32:54 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:32:55.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:32:54 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:32:55.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:32:54 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:32:55.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:32:55 smithi115 conmon[90393]: 2022-05-14T02:32:55.063+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:32:56.027 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:32:55 smithi115 ceph-mon[78593]: pgmap v5906: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:32:56.027 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:32:55 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:32:56.028 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:32:55 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T02:32:56.028 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:32:55 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T02:32:56.028 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:32:55 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T02:32:56.029 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:32:55 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 11110 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:32:56.038 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:32:55 smithi047 ceph-mon[77147]: pgmap v5906: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:32:56.039 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:32:55 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:32:56.039 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:32:55 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T02:32:56.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:32:55 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T02:32:56.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:32:55 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T02:32:56.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:32:55 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 11110 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:32:56.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:32:56 smithi115 conmon[90393]: 2022-05-14T02:32:56.055+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:32:57.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:32:57 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' 2022-05-14T02:32:57.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:32:57 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' 2022-05-14T02:32:57.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:32:57 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:32:57.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:32:57 smithi115 conmon[90393]: 2022-05-14T02:32:57.022+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:32:57.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:32:57 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' 2022-05-14T02:32:57.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:32:57 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' 2022-05-14T02:32:57.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:32:57 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:32:58.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:32:57 smithi115 conmon[90393]: 2022-05-14T02:32:57.978+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:32:58.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:32:58 smithi115 ceph-mon[78593]: pgmap v5907: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:32:58.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:32:58 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:32:58.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:32:58 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:32:58.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:32:58 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:32:58.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:32:58 smithi047 ceph-mon[77147]: pgmap v5907: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:32:58.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:32:58 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:32:58.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:32:58 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:32:58.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:32:58 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:32:59.041 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:32:58 smithi047 conmon[78521]: 2022-05-14T02:32:58.673+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:32:59.042 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:32:58 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T02:32:59.042 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:32:58 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:32:59.042 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:32:58 smithi047 conmon[78521]: yield 2022-05-14T02:32:59.042 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:32:58 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:32:59.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:32:58 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:32:59.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:32:58 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:32:59.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:32:58 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T02:32:59.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:32:58 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:32:59.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:32:58 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T02:32:59.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:32:58 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:32:59.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:32:58 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T02:32:59.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:32:58 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:32:59.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:32:58 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T02:32:59.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:32:58 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:32:59.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:32:58 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T02:32:59.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:32:58 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:32:59.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:32:58 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:32:59.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:32:58 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:32:59.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:32:58 smithi115 conmon[90393]: 2022-05-14T02:32:58.948+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:32:59.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:32:59 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:32:59.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:32:59 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' 2022-05-14T02:32:59.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:32:59 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' 2022-05-14T02:32:59.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:32:59 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:32:59.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:32:59 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' 2022-05-14T02:32:59.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:32:59 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' 2022-05-14T02:33:00.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:32:59 smithi115 conmon[90393]: 2022-05-14T02:32:59.933+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:33:00.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:33:00 smithi115 ceph-mon[78593]: pgmap v5908: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:33:00.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:33:00 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:33:00.359 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T02:33:00.359 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:33:00.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T02:33:00.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:33:00.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:33:00.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:33:00.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T02:33:00.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:33:00.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T02:33:00.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:33:00.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T02:33:00.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:33:00.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T02:33:00.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:33:00.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T02:33:00.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:33:00.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:33:00.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:33:00.363 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:33:00 smithi115 ceph-mon[78593]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T02:33:00.364 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:33:00 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:33:00.364 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:33:00 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-05-14T02:33:00.364 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:33:00 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-05-14T02:33:00.364 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:33:00 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-05-14T02:33:00.364 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:33:00 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-05-14T02:33:00.365 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:33:00 smithi115 ceph-mon[78593]: Adjusting osd_memory_target on smithi115 to 197.0M 2022-05-14T02:33:00.365 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:33:00 smithi115 ceph-mon[78593]: Unable to set osd_memory_target on smithi115 to 206629785: error parsing value: Value '206629785' is below minimum 939524096 2022-05-14T02:33:00.365 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:33:00 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-05-14T02:33:00.365 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:33:00 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-05-14T02:33:00.366 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:33:00 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-05-14T02:33:00.366 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:33:00 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-05-14T02:33:00.366 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:33:00 smithi115 ceph-mon[78593]: Adjusting osd_memory_target on smithi047 to 198.1M 2022-05-14T02:33:00.366 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:33:00 smithi115 ceph-mon[78593]: Unable to set osd_memory_target on smithi047 to 207755878: error parsing value: Value '207755878' is below minimum 939524096 2022-05-14T02:33:00.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:33:00 smithi047 ceph-mon[77147]: pgmap v5908: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:33:00.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:33:00 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:33:00.542 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T02:33:00.542 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:33:00.542 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T02:33:00.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:33:00.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:33:00.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:33:00.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T02:33:00.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:33:00.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T02:33:00.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:33:00.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T02:33:00.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:33:00.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T02:33:00.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:33:00.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T02:33:00.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:33:00.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:33:00.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:33:00.546 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:33:00 smithi047 ceph-mon[77147]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T02:33:00.546 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:33:00 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:33:00.546 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:33:00 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-05-14T02:33:00.547 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:33:00 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-05-14T02:33:00.547 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:33:00 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-05-14T02:33:00.547 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:33:00 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-05-14T02:33:00.547 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:33:00 smithi047 ceph-mon[77147]: Adjusting osd_memory_target on smithi115 to 197.0M 2022-05-14T02:33:00.548 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:33:00 smithi047 ceph-mon[77147]: Unable to set osd_memory_target on smithi115 to 206629785: error parsing value: Value '206629785' is below minimum 939524096 2022-05-14T02:33:00.548 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:33:00 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-05-14T02:33:00.548 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:33:00 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-05-14T02:33:00.548 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:33:00 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-05-14T02:33:00.548 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:33:00 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-05-14T02:33:00.549 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:33:00 smithi047 ceph-mon[77147]: Adjusting osd_memory_target on smithi047 to 198.1M 2022-05-14T02:33:00.549 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:33:00 smithi047 ceph-mon[77147]: Unable to set osd_memory_target on smithi047 to 207755878: error parsing value: Value '207755878' is below minimum 939524096 2022-05-14T02:33:01.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:33:00 smithi115 conmon[90393]: 2022-05-14T02:33:00.934+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:33:01.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:33:01 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:33:01.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:33:01 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:33:02.042 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:33:01 smithi047 conmon[78521]: 2022-05-14T02:33:01.744+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:33:02.042 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:33:01 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T02:33:02.042 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:33:01 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:33:02.042 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:33:01 smithi047 conmon[78521]: yield 2022-05-14T02:33:02.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:33:01 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:33:02.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:33:01 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:33:02.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:33:01 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:33:02.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:33:01 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T02:33:02.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:33:01 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:33:02.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:33:01 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T02:33:02.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:33:01 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:33:02.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:33:01 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T02:33:02.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:33:01 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:33:02.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:33:01 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T02:33:02.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:33:01 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:33:02.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:33:01 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T02:33:02.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:33:01 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:33:02.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:33:01 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:33:02.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:33:01 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:33:02.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:33:01 smithi047 conmon[78521]: 2022-05-14T02:33:01.745+0000 7f5340779700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T02:33:02.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:33:01 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T02:33:02.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:33:01 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:33:02.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:33:01 smithi047 conmon[78521]: yield 2022-05-14T02:33:02.048 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:33:01 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:33:02.048 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:33:01 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:33:02.048 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:33:01 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:33:02.048 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:33:01 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T02:33:02.049 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:33:01 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:33:02.049 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:33:01 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T02:33:02.049 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:33:01 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:33:02.049 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:33:01 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T02:33:02.050 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:33:01 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:33:02.050 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:33:01 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T02:33:02.050 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:33:01 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:33:02.050 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:33:01 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T02:33:02.050 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:33:01 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:33:02.051 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:33:01 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:33:02.051 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:33:01 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:33:02.051 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:33:01 smithi047 conmon[78521]: 2022-05-14T02:33:02.051 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:33:01 smithi047 conmon[78521]: During handling of the above exception, another exception occurred: 2022-05-14T02:33:02.052 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:33:01 smithi047 conmon[78521]: 2022-05-14T02:33:02.052 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:33:01 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T02:33:02.052 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:33:01 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T02:33:02.052 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:33:01 smithi047 conmon[78521]: return f(*arg) 2022-05-14T02:33:02.052 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:33:01 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T02:33:02.053 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:33:01 smithi047 conmon[78521]: self._write_client_files(client_files, host) 2022-05-14T02:33:02.053 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:33:01 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T02:33:02.053 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:33:01 smithi047 conmon[78521]: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T02:33:02.053 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:33:01 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T02:33:02.054 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:33:01 smithi047 conmon[78521]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T02:33:02.054 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:33:01 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T02:33:02.054 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:33:01 smithi047 conmon[78521]: return self.event_loop.get_result(coro) 2022-05-14T02:33:02.054 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:33:01 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T02:33:02.055 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:33:01 smithi047 conmon[78521]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T02:33:02.055 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:33:01 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T02:33:02.055 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:33:01 smithi047 conmon[78521]: return self.__get_result() 2022-05-14T02:33:02.055 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:33:01 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T02:33:02.055 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:33:01 smithi047 conmon[78521]: raise self._exception 2022-05-14T02:33:02.056 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:33:01 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T02:33:02.056 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:33:01 smithi047 conmon[78521]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T02:33:02.056 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:33:01 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T02:33:02.056 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:33:01 smithi047 conmon[78521]: conn = await self._remote_connection(host, addr) 2022-05-14T02:33:02.057 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:33:01 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T02:33:02.057 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:33:01 smithi047 conmon[78521]: raise 2022-05-14T02:33:02.057 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:33:01 smithi047 conmon[78521]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T02:33:02.057 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:33:01 smithi047 conmon[78521]: self.gen.throw(type, value, traceback) 2022-05-14T02:33:02.057 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:33:01 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T02:33:02.058 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:33:01 smithi047 conmon[78521]: raise OrchestratorError(msg) 2022-05-14T02:33:02.058 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:33:01 smithi047 conmon[78521]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:33:02.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:33:01 smithi115 conmon[90393]: 2022-05-14T02:33:01.927+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:33:02.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:33:02 smithi115 ceph-mon[78593]: pgmap v5909: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:33:02.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:33:02 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:33:02.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:33:02 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 11115 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:33:02.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:33:02 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:33:02.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:33:02 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:33:02.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:33:02 smithi047 ceph-mon[77147]: pgmap v5909: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:33:02.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:33:02 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:33:02.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:33:02 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 11115 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:33:02.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:33:02 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:33:02.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:33:02 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:33:03.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:33:02 smithi115 conmon[90393]: 2022-05-14T02:33:02.897+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:33:03.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:33:03 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:33:03.359 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T02:33:03.359 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:33:03.359 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T02:33:03.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:33:03.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:33:03.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:33:03.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T02:33:03.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:33:03.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T02:33:03.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:33:03.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T02:33:03.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:33:03.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T02:33:03.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:33:03.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T02:33:03.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:33:03.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:33:03.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:33:03.363 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:33:03 smithi115 ceph-mon[78593]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T02:33:03.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T02:33:03.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:33:03.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T02:33:03.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:33:03.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:33:03.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:33:03.365 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T02:33:03.365 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:33:03.365 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T02:33:03.365 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:33:03.366 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T02:33:03.366 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:33:03.366 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T02:33:03.366 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:33:03.366 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T02:33:03.367 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:33:03.367 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:33:03.367 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:33:03.367 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T02:33:03.367 INFO:journalctl@ceph.mon.c.smithi115.stdout: During handling of the above exception, another exception occurred: 2022-05-14T02:33:03.368 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T02:33:03.368 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T02:33:03.368 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T02:33:03.368 INFO:journalctl@ceph.mon.c.smithi115.stdout: return f(*arg) 2022-05-14T02:33:03.369 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T02:33:03.369 INFO:journalctl@ceph.mon.c.smithi115.stdout: self._write_client_files(client_files, host) 2022-05-14T02:33:03.369 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T02:33:03.369 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T02:33:03.369 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T02:33:03.370 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T02:33:03.370 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T02:33:03.370 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.event_loop.get_result(coro) 2022-05-14T02:33:03.370 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T02:33:03.371 INFO:journalctl@ceph.mon.c.smithi115.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T02:33:03.371 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T02:33:03.371 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.__get_result() 2022-05-14T02:33:03.371 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T02:33:03.371 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise self._exception 2022-05-14T02:33:03.372 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T02:33:03.372 INFO:journalctl@ceph.mon.c.smithi115.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T02:33:03.372 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T02:33:03.372 INFO:journalctl@ceph.mon.c.smithi115.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T02:33:03.372 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T02:33:03.373 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise 2022-05-14T02:33:03.373 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T02:33:03.373 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.gen.throw(type, value, traceback) 2022-05-14T02:33:03.373 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T02:33:03.374 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OrchestratorError(msg) 2022-05-14T02:33:03.374 INFO:journalctl@ceph.mon.c.smithi115.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:33:03.374 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:33:03 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:33:03.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:33:03 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:33:03.542 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T02:33:03.542 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:33:03.542 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T02:33:03.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:33:03.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:33:03.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:33:03.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T02:33:03.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:33:03.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T02:33:03.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:33:03.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T02:33:03.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:33:03.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T02:33:03.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:33:03.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T02:33:03.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:33:03.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:33:03.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:33:03.546 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:33:03 smithi047 ceph-mon[77147]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T02:33:03.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T02:33:03.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:33:03.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T02:33:03.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:33:03.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:33:03.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:33:03.548 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T02:33:03.548 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:33:03.548 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T02:33:03.548 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:33:03.548 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T02:33:03.549 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:33:03.549 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T02:33:03.549 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:33:03.549 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T02:33:03.549 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:33:03.550 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:33:03.550 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:33:03.550 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T02:33:03.550 INFO:journalctl@ceph.mon.b.smithi047.stdout: During handling of the above exception, another exception occurred: 2022-05-14T02:33:03.551 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T02:33:03.551 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T02:33:03.551 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T02:33:03.551 INFO:journalctl@ceph.mon.b.smithi047.stdout: return f(*arg) 2022-05-14T02:33:03.551 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T02:33:03.552 INFO:journalctl@ceph.mon.b.smithi047.stdout: self._write_client_files(client_files, host) 2022-05-14T02:33:03.552 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T02:33:03.552 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T02:33:03.552 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T02:33:03.552 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T02:33:03.553 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T02:33:03.553 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.event_loop.get_result(coro) 2022-05-14T02:33:03.553 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T02:33:03.553 INFO:journalctl@ceph.mon.b.smithi047.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T02:33:03.554 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T02:33:03.554 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.__get_result() 2022-05-14T02:33:03.554 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T02:33:03.554 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise self._exception 2022-05-14T02:33:03.554 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T02:33:03.555 INFO:journalctl@ceph.mon.b.smithi047.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T02:33:03.555 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T02:33:03.555 INFO:journalctl@ceph.mon.b.smithi047.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T02:33:03.555 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T02:33:03.555 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise 2022-05-14T02:33:03.556 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T02:33:03.556 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.gen.throw(type, value, traceback) 2022-05-14T02:33:03.556 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T02:33:03.556 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OrchestratorError(msg) 2022-05-14T02:33:03.556 INFO:journalctl@ceph.mon.b.smithi047.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:33:03.557 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:33:03 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:33:04.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:33:03 smithi115 conmon[90393]: 2022-05-14T02:33:03.910+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:33:04.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:33:04 smithi115 ceph-mon[78593]: pgmap v5910: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:33:04.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:33:04 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:33:04.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:33:04 smithi047 ceph-mon[77147]: pgmap v5910: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:33:04.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:33:04 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:33:05.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:33:04 smithi115 conmon[90393]: 2022-05-14T02:33:04.947+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:33:05.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:33:05 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:33:05.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:33:05 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:33:06.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:33:05 smithi115 conmon[90393]: 2022-05-14T02:33:05.948+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:33:06.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:33:06 smithi115 ceph-mon[78593]: pgmap v5911: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:33:06.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:33:06 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:33:06.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:33:06 smithi047 ceph-mon[77147]: pgmap v5911: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:33:06.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:33:06 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:33:07.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:33:06 smithi115 conmon[90393]: 2022-05-14T02:33:06.982+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:33:07.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:33:07 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:33:07.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:33:07 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:33:08.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:33:07 smithi115 conmon[90393]: 2022-05-14T02:33:07.940+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:33:08.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:33:08 smithi115 ceph-mon[78593]: pgmap v5912: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:33:08.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:33:08 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:33:08.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:33:08 smithi047 ceph-mon[77147]: pgmap v5912: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:33:08.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:33:08 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:33:09.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:33:08 smithi115 conmon[90393]: 2022-05-14T02:33:08.926+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:33:09.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:33:09 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:33:09.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:33:09 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:33:10.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:33:09 smithi115 conmon[90393]: 2022-05-14T02:33:09.903+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:33:10.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:33:10 smithi115 ceph-mon[78593]: pgmap v5913: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:33:10.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:33:10 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:33:10.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:33:10 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:33:10.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:33:10 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:33:10.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:33:10 smithi047 ceph-mon[77147]: pgmap v5913: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:33:10.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:33:10 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:33:10.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:33:10 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:33:10.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:33:10 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:33:11.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:33:10 smithi115 conmon[90393]: 2022-05-14T02:33:10.894+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:33:11.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:33:11 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:33:11.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:33:11 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 11120 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:33:11.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:33:11 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:33:11.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:33:11 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 11120 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:33:12.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:33:11 smithi115 conmon[90393]: 2022-05-14T02:33:11.871+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:33:12.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:33:12 smithi115 ceph-mon[78593]: pgmap v5914: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:33:12.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:33:12 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:33:12.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:33:12 smithi047 ceph-mon[77147]: pgmap v5914: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:33:12.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:33:12 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:33:13.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:33:12 smithi115 conmon[90393]: 2022-05-14T02:33:12.874+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:33:13.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:33:13 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:33:13.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:33:13 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:33:14.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:33:13 smithi115 conmon[90393]: 2022-05-14T02:33:13.895+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:33:14.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:33:14 smithi115 ceph-mon[78593]: pgmap v5915: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:33:14.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:33:14 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:33:14.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:33:14 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:33:14.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:33:14 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:33:14.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:33:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T02:33:14.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:33:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T02:33:14.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:33:14 smithi047 ceph-mon[77147]: pgmap v5915: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:33:14.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:33:14 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:33:14.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:33:14 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:33:14.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:33:14 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:33:14.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:33:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T02:33:14.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:33:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T02:33:15.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:33:14 smithi115 conmon[90393]: 2022-05-14T02:33:14.934+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:33:15.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:33:15 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:33:15.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:33:15 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:33:16.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:33:15 smithi115 conmon[90393]: 2022-05-14T02:33:15.971+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:33:16.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:33:16 smithi115 ceph-mon[78593]: pgmap v5916: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:33:16.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:33:16 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:33:16.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:33:16 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 11130 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:33:16.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:33:16 smithi047 ceph-mon[77147]: pgmap v5916: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:33:16.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:33:16 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:33:16.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:33:16 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 11130 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:33:17.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:33:16 smithi115 conmon[90393]: 2022-05-14T02:33:16.992+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:33:17.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:33:17 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:33:17.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:33:17 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:33:18.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:33:18 smithi115 conmon[90393]: 2022-05-14T02:33:18.003+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:33:18.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:33:18 smithi115 ceph-mon[78593]: pgmap v5917: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:33:18.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:33:18 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:33:18.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:33:18 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:33:18.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:33:18 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:33:18.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:33:18 smithi047 ceph-mon[77147]: pgmap v5917: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:33:18.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:33:18 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:33:18.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:33:18 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:33:18.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:33:18 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:33:19.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:33:19 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:33:19.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:33:19 smithi115 conmon[90393]: 2022-05-14T02:33:19.010+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:33:19.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:33:19 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:33:20.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:33:19 smithi115 conmon[90393]: 2022-05-14T02:33:19.985+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:33:20.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:33:20 smithi115 ceph-mon[78593]: pgmap v5918: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:33:20.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:33:20 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:33:20.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:33:20 smithi047 ceph-mon[77147]: pgmap v5918: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:33:20.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:33:20 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:33:21.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:33:20 smithi115 conmon[90393]: 2022-05-14T02:33:20.949+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:33:21.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:33:21 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:33:21.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:33:21 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:33:22.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:33:21 smithi115 conmon[90393]: 2022-05-14T02:33:21.902+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:33:22.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:33:22 smithi115 ceph-mon[78593]: pgmap v5919: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:33:22.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:33:22 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:33:22.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:33:22 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 11135 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:33:22.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:33:22 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:33:22.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:33:22 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:33:22.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:33:22 smithi047 ceph-mon[77147]: pgmap v5919: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:33:22.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:33:22 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:33:22.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:33:22 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 11135 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:33:22.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:33:22 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:33:22.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:33:22 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:33:23.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:33:22 smithi115 conmon[90393]: 2022-05-14T02:33:22.924+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:33:23.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:33:23 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:33:23.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:33:23 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:33:24.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:33:23 smithi115 conmon[90393]: 2022-05-14T02:33:23.971+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:33:24.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:33:24 smithi115 ceph-mon[78593]: pgmap v5920: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:33:24.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:33:24 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:33:24.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:33:24 smithi047 ceph-mon[77147]: pgmap v5920: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:33:24.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:33:24 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:33:25.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:33:24 smithi115 conmon[90393]: 2022-05-14T02:33:24.979+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:33:25.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:33:25 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:33:25.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:33:25 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:33:26.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:33:25 smithi115 conmon[90393]: 2022-05-14T02:33:25.976+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:33:26.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:33:26 smithi115 ceph-mon[78593]: pgmap v5921: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:33:26.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:33:26 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:33:26.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:33:26 smithi047 ceph-mon[77147]: pgmap v5921: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:33:26.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:33:26 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:33:27.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:33:27 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:33:27.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:33:27 smithi115 conmon[90393]: 2022-05-14T02:33:27.002+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:33:27.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:33:27 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:33:28.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:33:28 smithi115 conmon[90393]: 2022-05-14T02:33:28.048+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:33:28.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:33:28 smithi115 ceph-mon[78593]: pgmap v5922: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:33:28.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:33:28 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:33:28.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:33:28 smithi047 ceph-mon[77147]: pgmap v5922: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:33:28.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:33:28 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:33:29.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:33:29 smithi115 conmon[90393]: 2022-05-14T02:33:29.038+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:33:29.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:33:29 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:33:29.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:33:29 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:33:30.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:33:29 smithi115 conmon[90393]: 2022-05-14T02:33:29.991+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:33:30.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:33:30 smithi115 ceph-mon[78593]: pgmap v5923: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:33:30.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:33:30 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:33:30.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:33:30 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:33:30.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:33:30 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:33:30.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:33:30 smithi047 ceph-mon[77147]: pgmap v5923: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:33:30.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:33:30 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:33:30.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:33:30 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:33:30.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:33:30 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:33:31.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:33:31 smithi115 conmon[90393]: 2022-05-14T02:33:31.005+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:33:31.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:33:31 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:33:31.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:33:31 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 11140 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:33:31.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:33:31 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:33:31.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:33:31 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 11140 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:33:32.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:33:31 smithi115 conmon[90393]: 2022-05-14T02:33:31.956+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:33:32.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:33:32 smithi115 ceph-mon[78593]: pgmap v5924: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:33:32.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:33:32 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:33:32.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:33:32 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:33:32.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:33:32 smithi047 ceph-mon[77147]: pgmap v5924: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:33:32.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:33:32 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:33:32.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:33:32 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:33:33.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:33:32 smithi115 conmon[90393]: 2022-05-14T02:33:32.942+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:33:33.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:33:33 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:33:33.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:33:33 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:33:34.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:33:33 smithi115 conmon[90393]: 2022-05-14T02:33:33.918+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:33:34.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:33:34 smithi115 ceph-mon[78593]: pgmap v5925: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:33:34.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:33:34 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:33:34.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:33:34 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:33:34.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:33:34 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:33:34.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:33:34 smithi047 ceph-mon[77147]: pgmap v5925: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:33:34.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:33:34 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:33:34.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:33:34 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:33:34.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:33:34 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:33:35.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:33:34 smithi115 conmon[90393]: 2022-05-14T02:33:34.932+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:33:35.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:33:35 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:33:35.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:33:35 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:33:36.169 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:33:35 smithi115 conmon[90393]: 2022-05-14T02:33:35.901+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:33:36.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:33:36 smithi047 ceph-mon[77147]: pgmap v5926: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:33:36.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:33:36 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 11150 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:33:36.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:33:36 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:33:36.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:33:36 smithi115 ceph-mon[78593]: pgmap v5926: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:33:36.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:33:36 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 11150 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:33:36.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:33:36 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:33:37.175 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:33:36 smithi115 conmon[90393]: 2022-05-14T02:33:36.868+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:33:37.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:33:37 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:33:37.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:33:37 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:33:38.176 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:33:37 smithi115 conmon[90393]: 2022-05-14T02:33:37.884+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:33:38.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:33:38 smithi047 ceph-mon[77147]: pgmap v5927: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:33:38.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:33:38 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:33:38.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:33:38 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:33:38.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:33:38 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:33:38.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:33:38 smithi115 ceph-mon[78593]: pgmap v5927: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:33:38.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:33:38 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:33:38.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:33:38 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:33:38.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:33:38 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:33:39.181 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:33:38 smithi115 conmon[90393]: 2022-05-14T02:33:38.882+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:33:39.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:33:39 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:33:39.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:33:39 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:33:40.183 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:33:39 smithi115 conmon[90393]: 2022-05-14T02:33:39.925+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:33:40.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:33:40 smithi047 ceph-mon[77147]: pgmap v5928: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:33:40.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:33:40 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:33:40.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:33:40 smithi115 ceph-mon[78593]: pgmap v5928: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:33:40.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:33:40 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:33:41.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:33:40 smithi115 conmon[90393]: 2022-05-14T02:33:40.886+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:33:42.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:33:41 smithi047 ceph-mon[77147]: pgmap v5929: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:33:42.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:33:41 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:33:42.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:33:41 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 11155 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:33:42.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:33:41 smithi115 conmon[90393]: 2022-05-14T02:33:41.913+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:33:42.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:33:41 smithi115 ceph-mon[78593]: pgmap v5929: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:33:42.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:33:41 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:33:42.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:33:41 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 11155 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:33:43.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:33:42 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:33:43.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:33:42 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:33:43.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:33:42 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:33:43.110 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:33:42 smithi115 conmon[90393]: 2022-05-14T02:33:42.937+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:33:43.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:33:42 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:33:43.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:33:42 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:33:43.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:33:42 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:33:44.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:33:43 smithi047 ceph-mon[77147]: pgmap v5930: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:33:44.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:33:43 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:33:44.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:33:43 smithi115 conmon[90393]: 2022-05-14T02:33:43.979+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:33:44.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:33:43 smithi115 ceph-mon[78593]: pgmap v5930: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:33:44.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:33:43 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:33:45.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:33:44 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:33:45.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:33:44 smithi115 conmon[90393]: 2022-05-14T02:33:44.950+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:33:45.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:33:44 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:33:46.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:33:45 smithi047 ceph-mon[77147]: pgmap v5931: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:33:46.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:33:45 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:33:46.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:33:45 smithi115 conmon[90393]: 2022-05-14T02:33:45.903+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:33:46.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:33:45 smithi115 ceph-mon[78593]: pgmap v5931: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:33:46.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:33:45 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:33:47.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:33:46 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:33:47.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:33:46 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:33:47.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:33:46 smithi115 conmon[90393]: 2022-05-14T02:33:46.896+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:33:48.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:33:47 smithi047 ceph-mon[77147]: pgmap v5932: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:33:48.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:33:47 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:33:48.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:33:47 smithi115 conmon[90393]: 2022-05-14T02:33:47.913+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:33:48.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:33:47 smithi115 ceph-mon[78593]: pgmap v5932: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:33:48.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:33:47 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:33:49.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:33:48 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:33:49.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:33:48 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:33:49.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:33:48 smithi115 conmon[90393]: 2022-05-14T02:33:48.914+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:33:50.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:33:49 smithi047 ceph-mon[77147]: pgmap v5933: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:33:50.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:33:49 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:33:50.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:33:49 smithi115 conmon[90393]: 2022-05-14T02:33:49.899+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:33:50.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:33:49 smithi115 ceph-mon[78593]: pgmap v5933: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:33:50.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:33:49 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:33:51.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:33:50 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:33:51.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:33:50 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:33:51.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:33:50 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:33:51.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:33:50 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 11160 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:33:51.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:33:50 smithi115 conmon[90393]: 2022-05-14T02:33:50.899+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:33:51.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:33:50 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:33:51.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:33:50 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:33:51.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:33:50 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:33:51.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:33:50 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 11160 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:33:52.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:33:51 smithi115 conmon[90393]: 2022-05-14T02:33:51.868+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:33:52.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:33:51 smithi115 ceph-mon[78593]: pgmap v5934: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:33:52.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:33:51 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:33:52.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:33:51 smithi047 ceph-mon[77147]: pgmap v5934: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:33:52.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:33:51 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:33:53.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:33:52 smithi115 conmon[90393]: 2022-05-14T02:33:52.896+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:33:53.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:33:52 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:33:53.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:33:52 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:33:54.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:33:53 smithi115 ceph-mon[78593]: pgmap v5935: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:33:54.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:33:53 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:33:54.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:33:53 smithi115 conmon[90393]: 2022-05-14T02:33:53.914+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:33:54.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:33:53 smithi047 ceph-mon[77147]: pgmap v5935: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:33:54.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:33:53 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:33:55.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:33:54 smithi115 conmon[90393]: 2022-05-14T02:33:54.929+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:33:55.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:33:54 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:33:55.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:33:54 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:33:55.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:33:54 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:33:55.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:33:54 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:33:55.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:33:54 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:33:55.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:33:54 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:33:56.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:33:55 smithi115 conmon[90393]: 2022-05-14T02:33:55.962+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:33:56.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:33:55 smithi115 ceph-mon[78593]: pgmap v5936: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:33:56.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:33:55 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:33:56.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:33:55 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 11170 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:33:56.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:33:55 smithi047 ceph-mon[77147]: pgmap v5936: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:33:56.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:33:55 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:33:56.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:33:55 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 11170 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:33:57.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:33:56 smithi115 conmon[90393]: 2022-05-14T02:33:56.945+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:33:57.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:33:56 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:33:57.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:33:56 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:33:58.110 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:33:57 smithi115 conmon[90393]: 2022-05-14T02:33:57.944+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:33:58.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:33:57 smithi115 ceph-mon[78593]: pgmap v5937: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:33:58.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:33:57 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:33:58.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:33:57 smithi047 ceph-mon[77147]: pgmap v5937: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:33:58.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:33:57 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:33:59.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:33:58 smithi115 conmon[90393]: 2022-05-14T02:33:58.951+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:33:59.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:33:58 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:33:59.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:33:58 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:33:59.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:33:58 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:33:59.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:33:58 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:33:59.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:33:58 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:33:59.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:33:58 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:34:00.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:33:59 smithi115 conmon[90393]: 2022-05-14T02:33:59.956+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:34:00.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:33:59 smithi115 ceph-mon[78593]: pgmap v5938: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:34:00.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:33:59 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:34:00.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:33:59 smithi047 ceph-mon[77147]: pgmap v5938: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:34:00.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:33:59 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:34:01.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:34:00 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:34:01.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:34:00 smithi115 conmon[90393]: 2022-05-14T02:34:00.996+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:34:01.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:34:00 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:34:02.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:34:01 smithi115 conmon[90393]: 2022-05-14T02:34:01.977+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:34:02.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:34:01 smithi115 ceph-mon[78593]: pgmap v5939: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:34:02.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:34:01 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 11175 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:34:02.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:34:01 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:34:02.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:34:01 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T02:34:02.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:34:01 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T02:34:02.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:34:01 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T02:34:02.264 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:34:01 smithi047 ceph-mon[77147]: pgmap v5939: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:34:02.264 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:34:01 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 11175 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:34:02.264 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:34:01 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:34:02.264 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:34:01 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T02:34:02.265 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:34:01 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T02:34:02.265 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:34:01 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T02:34:02.541 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:34:02 smithi047 conmon[78521]: 2022-05-14T02:34:02.351+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:34:02.542 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:34:02 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T02:34:02.542 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:34:02 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:34:02.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:34:02 smithi047 conmon[78521]: yield 2022-05-14T02:34:02.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:34:02 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:34:02.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:34:02 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:34:02.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:34:02 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:34:02.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:34:02 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T02:34:02.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:34:02 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:34:02.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:34:02 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T02:34:02.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:34:02 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:34:02.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:34:02 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T02:34:02.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:34:02 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:34:02.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:34:02 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T02:34:02.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:34:02 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:34:02.547 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:34:02 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T02:34:02.547 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:34:02 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:34:02.547 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:34:02 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:34:02.547 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:34:02 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:34:03.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:34:02 smithi115 conmon[90393]: 2022-05-14T02:34:02.966+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:34:03.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:34:02 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:34:03.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:34:02 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:34:03.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:34:02 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:34:03.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:34:02 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:34:03.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T02:34:03.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:34:03.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T02:34:03.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:34:03.113 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:34:03.113 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:34:03.114 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T02:34:03.115 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:34:03.115 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T02:34:03.116 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:34:03.116 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T02:34:03.116 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:34:03.116 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T02:34:03.117 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:34:03.117 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T02:34:03.117 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:34:03.117 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:34:03.118 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:34:03.118 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:34:02 smithi115 ceph-mon[78593]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T02:34:03.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:34:02 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:34:03.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:34:02 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:34:03.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:34:02 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:34:03.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:34:02 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:34:03.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T02:34:03.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:34:03.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T02:34:03.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:34:03.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:34:03.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:34:03.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T02:34:03.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:34:03.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T02:34:03.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:34:03.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T02:34:03.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:34:03.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T02:34:03.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:34:03.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T02:34:03.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:34:03.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:34:03.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:34:03.297 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:34:02 smithi047 ceph-mon[77147]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T02:34:04.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:34:03 smithi115 conmon[90393]: 2022-05-14T02:34:03.955+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:34:04.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:34:03 smithi115 ceph-mon[78593]: pgmap v5940: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:34:04.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:34:03 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:34:04.296 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:34:03 smithi047 ceph-mon[77147]: pgmap v5940: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:34:04.297 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:34:03 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:34:05.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:34:04 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:34:05.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:34:04 smithi115 conmon[90393]: 2022-05-14T02:34:04.981+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:34:05.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:34:04 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:34:05.792 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:34:05 smithi047 conmon[78521]: 2022-05-14T02:34:05.423+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:34:05.793 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:34:05 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T02:34:05.793 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:34:05 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:34:05.793 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:34:05 smithi047 conmon[78521]: yield 2022-05-14T02:34:05.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:34:05 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:34:05.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:34:05 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:34:05.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:34:05 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:34:05.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:34:05 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T02:34:05.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:34:05 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:34:05.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:34:05 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T02:34:05.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:34:05 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:34:05.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:34:05 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T02:34:05.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:34:05 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:34:05.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:34:05 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T02:34:05.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:34:05 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:34:05.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:34:05 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T02:34:05.797 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:34:05 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:34:05.797 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:34:05 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:34:05.797 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:34:05 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:34:05.797 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:34:05 smithi047 conmon[78521]: 2022-05-14T02:34:05.424+0000 7f534177b700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T02:34:05.798 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:34:05 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T02:34:05.798 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:34:05 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:34:05.798 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:34:05 smithi047 conmon[78521]: yield 2022-05-14T02:34:05.798 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:34:05 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:34:05.799 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:34:05 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:34:05.799 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:34:05 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:34:05.799 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:34:05 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T02:34:05.799 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:34:05 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:34:05.800 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:34:05 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T02:34:05.800 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:34:05 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:34:05.800 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:34:05 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T02:34:05.800 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:34:05 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:34:05.801 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:34:05 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T02:34:05.801 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:34:05 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:34:05.801 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:34:05 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T02:34:05.801 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:34:05 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:34:05.801 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:34:05 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:34:05.802 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:34:05 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:34:05.802 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:34:05 smithi047 conmon[78521]: 2022-05-14T02:34:05.802 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:34:05 smithi047 conmon[78521]: During handling of the above exception, another exception occurred: 2022-05-14T02:34:05.802 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:34:05 smithi047 conmon[78521]: 2022-05-14T02:34:05.803 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:34:05 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T02:34:05.803 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:34:05 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T02:34:05.803 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:34:05 smithi047 conmon[78521]: return f(*arg) 2022-05-14T02:34:05.803 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:34:05 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T02:34:05.803 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:34:05 smithi047 conmon[78521]: self._write_client_files(client_files, host) 2022-05-14T02:34:05.804 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:34:05 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T02:34:05.804 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:34:05 smithi047 conmon[78521]: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T02:34:05.804 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:34:05 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T02:34:05.804 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:34:05 smithi047 conmon[78521]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T02:34:05.805 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:34:05 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T02:34:05.805 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:34:05 smithi047 conmon[78521]: return self.event_loop.get_result(coro) 2022-05-14T02:34:05.805 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:34:05 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T02:34:05.805 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:34:05 smithi047 conmon[78521]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T02:34:05.806 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:34:05 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T02:34:05.806 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:34:05 smithi047 conmon[78521]: return self.__get_result() 2022-05-14T02:34:05.807 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:34:05 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T02:34:05.807 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:34:05 smithi047 conmon[78521]: raise self._exception 2022-05-14T02:34:05.808 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:34:05 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T02:34:05.808 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:34:05 smithi047 conmon[78521]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T02:34:05.808 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:34:05 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T02:34:05.808 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:34:05 smithi047 conmon[78521]: conn = await self._remote_connection(host, addr) 2022-05-14T02:34:05.809 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:34:05 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T02:34:05.809 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:34:05 smithi047 conmon[78521]: raise 2022-05-14T02:34:05.809 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:34:05 smithi047 conmon[78521]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T02:34:05.809 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:34:05 smithi047 conmon[78521]: self.gen.throw(type, value, traceback) 2022-05-14T02:34:05.810 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:34:05 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T02:34:05.810 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:34:05 smithi047 conmon[78521]: raise OrchestratorError(msg) 2022-05-14T02:34:05.810 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:34:05 smithi047 conmon[78521]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:34:06.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:34:05 smithi115 ceph-mon[78593]: pgmap v5941: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:34:06.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:34:05 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:34:06.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:34:05 smithi115 conmon[90393]: 2022-05-14T02:34:05.994+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:34:06.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:34:05 smithi047 ceph-mon[77147]: pgmap v5941: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:34:06.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:34:05 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:34:07.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:34:06 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:34:07.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T02:34:07.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:34:07.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T02:34:07.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:34:07.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:34:07.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:34:07.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T02:34:07.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:34:07.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T02:34:07.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:34:07.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T02:34:07.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:34:07.298 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T02:34:07.298 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:34:07.299 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T02:34:07.299 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:34:07.300 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:34:07.300 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:34:07.301 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:34:06 smithi047 ceph-mon[77147]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T02:34:07.301 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T02:34:07.302 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:34:07.302 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T02:34:07.303 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:34:07.303 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:34:07.303 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:34:07.304 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T02:34:07.304 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:34:07.305 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T02:34:07.305 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:34:07.306 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T02:34:07.306 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:34:07.307 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T02:34:07.307 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:34:07.307 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T02:34:07.308 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:34:07.308 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:34:07.308 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:34:07.309 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T02:34:07.309 INFO:journalctl@ceph.mon.b.smithi047.stdout: During handling of the above exception, another exception occurred: 2022-05-14T02:34:07.309 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T02:34:07.310 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T02:34:07.310 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T02:34:07.310 INFO:journalctl@ceph.mon.b.smithi047.stdout: return f(*arg) 2022-05-14T02:34:07.311 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T02:34:07.311 INFO:journalctl@ceph.mon.b.smithi047.stdout: self._write_client_files(client_files, host) 2022-05-14T02:34:07.311 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T02:34:07.312 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T02:34:07.312 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T02:34:07.312 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T02:34:07.313 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T02:34:07.313 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.event_loop.get_result(coro) 2022-05-14T02:34:07.313 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T02:34:07.314 INFO:journalctl@ceph.mon.b.smithi047.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T02:34:07.314 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T02:34:07.314 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.__get_result() 2022-05-14T02:34:07.314 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T02:34:07.314 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise self._exception 2022-05-14T02:34:07.315 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T02:34:07.315 INFO:journalctl@ceph.mon.b.smithi047.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T02:34:07.315 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T02:34:07.315 INFO:journalctl@ceph.mon.b.smithi047.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T02:34:07.315 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T02:34:07.316 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise 2022-05-14T02:34:07.316 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T02:34:07.316 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.gen.throw(type, value, traceback) 2022-05-14T02:34:07.316 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T02:34:07.317 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OrchestratorError(msg) 2022-05-14T02:34:07.317 INFO:journalctl@ceph.mon.b.smithi047.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:34:07.317 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:34:06 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:34:07.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:34:06 smithi115 conmon[90393]: 2022-05-14T02:34:06.992+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:34:07.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:34:06 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:34:07.359 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T02:34:07.359 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:34:07.359 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T02:34:07.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:34:07.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:34:07.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:34:07.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T02:34:07.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:34:07.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T02:34:07.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:34:07.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T02:34:07.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:34:07.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T02:34:07.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:34:07.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T02:34:07.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:34:07.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:34:07.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:34:07.363 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:34:06 smithi115 ceph-mon[78593]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T02:34:07.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T02:34:07.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:34:07.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T02:34:07.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:34:07.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:34:07.365 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:34:07.365 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T02:34:07.366 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:34:07.367 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T02:34:07.367 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:34:07.367 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T02:34:07.367 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:34:07.368 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T02:34:07.368 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:34:07.368 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T02:34:07.368 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:34:07.369 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:34:07.369 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:34:07.369 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T02:34:07.369 INFO:journalctl@ceph.mon.c.smithi115.stdout: During handling of the above exception, another exception occurred: 2022-05-14T02:34:07.370 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T02:34:07.370 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T02:34:07.370 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T02:34:07.370 INFO:journalctl@ceph.mon.c.smithi115.stdout: return f(*arg) 2022-05-14T02:34:07.370 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T02:34:07.371 INFO:journalctl@ceph.mon.c.smithi115.stdout: self._write_client_files(client_files, host) 2022-05-14T02:34:07.371 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T02:34:07.371 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T02:34:07.371 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T02:34:07.372 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T02:34:07.372 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T02:34:07.372 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.event_loop.get_result(coro) 2022-05-14T02:34:07.372 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T02:34:07.373 INFO:journalctl@ceph.mon.c.smithi115.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T02:34:07.373 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T02:34:07.373 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.__get_result() 2022-05-14T02:34:07.373 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T02:34:07.373 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise self._exception 2022-05-14T02:34:07.374 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T02:34:07.374 INFO:journalctl@ceph.mon.c.smithi115.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T02:34:07.374 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T02:34:07.374 INFO:journalctl@ceph.mon.c.smithi115.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T02:34:07.375 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T02:34:07.375 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise 2022-05-14T02:34:07.375 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T02:34:07.375 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.gen.throw(type, value, traceback) 2022-05-14T02:34:07.375 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T02:34:07.376 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OrchestratorError(msg) 2022-05-14T02:34:07.376 INFO:journalctl@ceph.mon.c.smithi115.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:34:07.376 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:34:06 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:34:08.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:34:07 smithi047 ceph-mon[77147]: pgmap v5942: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:34:08.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:34:07 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:34:08.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:34:07 smithi115 conmon[90393]: 2022-05-14T02:34:07.948+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:34:08.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:34:07 smithi115 ceph-mon[78593]: pgmap v5942: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:34:08.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:34:07 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:34:09.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:34:08 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:34:09.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:34:08 smithi115 conmon[90393]: 2022-05-14T02:34:08.932+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:34:09.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:34:08 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:34:10.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:34:09 smithi047 ceph-mon[77147]: pgmap v5943: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:34:10.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:34:09 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:34:10.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:34:09 smithi115 conmon[90393]: 2022-05-14T02:34:09.893+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:34:10.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:34:09 smithi115 ceph-mon[78593]: pgmap v5943: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:34:10.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:34:09 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:34:11.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:34:10 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:34:11.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:34:10 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:34:11.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:34:10 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:34:11.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:34:10 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 11180 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:34:11.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:34:10 smithi115 conmon[90393]: 2022-05-14T02:34:10.923+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:34:11.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:34:10 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:34:11.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:34:10 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:34:11.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:34:10 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:34:11.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:34:10 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 11180 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:34:12.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:34:11 smithi047 ceph-mon[77147]: pgmap v5944: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:34:12.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:34:11 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:34:12.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:34:11 smithi115 conmon[90393]: 2022-05-14T02:34:11.911+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:34:12.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:34:11 smithi115 ceph-mon[78593]: pgmap v5944: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:34:12.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:34:11 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:34:13.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:34:12 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:34:13.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:34:12 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:34:13.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:34:12 smithi115 conmon[90393]: 2022-05-14T02:34:12.922+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:34:14.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:34:13 smithi047 ceph-mon[77147]: pgmap v5945: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:34:14.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:34:13 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:34:14.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:34:13 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:34:14.295 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:34:13 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:34:14.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:34:13 smithi115 conmon[90393]: 2022-05-14T02:34:13.949+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:34:14.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:34:13 smithi115 ceph-mon[78593]: pgmap v5945: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:34:14.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:34:13 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:34:14.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:34:13 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:34:14.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:34:13 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:34:15.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:34:14 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:34:15.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:34:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T02:34:15.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:34:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T02:34:15.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:34:14 smithi115 conmon[90393]: 2022-05-14T02:34:14.979+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:34:15.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:34:14 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:34:15.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:34:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T02:34:15.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:34:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T02:34:16.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:34:15 smithi047 ceph-mon[77147]: pgmap v5946: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:34:16.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:34:15 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:34:16.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:34:15 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 11190 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:34:16.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:34:16 smithi115 conmon[90393]: 2022-05-14T02:34:16.017+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:34:16.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:34:15 smithi115 ceph-mon[78593]: pgmap v5946: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:34:16.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:34:15 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:34:16.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:34:15 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 11190 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:34:17.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:34:16 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:34:17.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:34:16 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:34:17.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:34:17 smithi115 conmon[90393]: 2022-05-14T02:34:17.043+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:34:18.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:34:17 smithi047 ceph-mon[77147]: pgmap v5947: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:34:18.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:34:17 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:34:18.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:34:17 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:34:18.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:34:17 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:34:18.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:34:18 smithi115 conmon[90393]: 2022-05-14T02:34:18.058+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:34:18.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:34:17 smithi115 ceph-mon[78593]: pgmap v5947: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:34:18.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:34:17 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:34:18.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:34:17 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:34:18.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:34:17 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:34:19.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:34:18 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:34:19.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:34:18 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:34:19.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:34:19 smithi115 conmon[90393]: 2022-05-14T02:34:19.020+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:34:20.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:34:19 smithi047 ceph-mon[77147]: pgmap v5948: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:34:20.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:34:19 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:34:20.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:34:19 smithi115 conmon[90393]: 2022-05-14T02:34:19.977+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:34:20.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:34:19 smithi115 ceph-mon[78593]: pgmap v5948: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:34:20.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:34:19 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:34:21.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:34:20 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:34:21.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:34:20 smithi115 conmon[90393]: 2022-05-14T02:34:20.956+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:34:21.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:34:20 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:34:22.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:34:21 smithi047 ceph-mon[77147]: pgmap v5949: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:34:22.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:34:21 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 11195 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:34:22.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:34:21 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:34:22.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:34:21 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:34:22.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:34:21 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:34:22.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:34:21 smithi115 conmon[90393]: 2022-05-14T02:34:21.937+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:34:22.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:34:21 smithi115 ceph-mon[78593]: pgmap v5949: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:34:22.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:34:21 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 11195 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:34:22.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:34:21 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:34:22.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:34:21 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:34:22.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:34:21 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:34:23.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:34:22 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:34:23.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:34:22 smithi115 conmon[90393]: 2022-05-14T02:34:22.944+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:34:23.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:34:22 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:34:24.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:34:23 smithi047 ceph-mon[77147]: pgmap v5950: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:34:24.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:34:23 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:34:24.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:34:23 smithi115 conmon[90393]: 2022-05-14T02:34:23.917+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:34:24.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:34:23 smithi115 ceph-mon[78593]: pgmap v5950: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:34:24.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:34:23 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:34:25.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:34:24 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:34:25.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:34:24 smithi115 conmon[90393]: 2022-05-14T02:34:24.930+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:34:25.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:34:24 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:34:26.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:34:25 smithi047 ceph-mon[77147]: pgmap v5951: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:34:26.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:34:25 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:34:26.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:34:25 smithi115 conmon[90393]: 2022-05-14T02:34:25.974+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:34:26.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:34:25 smithi115 ceph-mon[78593]: pgmap v5951: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:34:26.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:34:25 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:34:27.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:34:26 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:34:27.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:34:26 smithi115 conmon[90393]: 2022-05-14T02:34:26.936+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:34:27.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:34:26 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:34:28.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:34:27 smithi047 ceph-mon[77147]: pgmap v5952: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:34:28.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:34:27 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:34:28.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:34:27 smithi115 conmon[90393]: 2022-05-14T02:34:27.958+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:34:28.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:34:27 smithi115 ceph-mon[78593]: pgmap v5952: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:34:28.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:34:27 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:34:29.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:34:28 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:34:29.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:34:28 smithi115 conmon[90393]: 2022-05-14T02:34:28.956+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:34:29.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:34:28 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:34:30.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:34:29 smithi047 ceph-mon[77147]: pgmap v5953: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:34:30.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:34:29 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:34:30.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:34:29 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:34:30.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:34:29 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:34:30.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:34:29 smithi115 conmon[90393]: 2022-05-14T02:34:29.973+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:34:30.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:34:29 smithi115 ceph-mon[78593]: pgmap v5953: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:34:30.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:34:29 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:34:30.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:34:29 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:34:30.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:34:29 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:34:31.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:34:30 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:34:31.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:34:30 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 11200 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:34:31.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:34:30 smithi115 conmon[90393]: 2022-05-14T02:34:30.977+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:34:31.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:34:30 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:34:31.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:34:30 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 11200 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:34:32.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:34:31 smithi047 ceph-mon[77147]: pgmap v5954: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:34:32.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:34:31 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:34:32.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:34:31 smithi115 conmon[90393]: 2022-05-14T02:34:31.993+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:34:32.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:34:31 smithi115 ceph-mon[78593]: pgmap v5954: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:34:32.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:34:31 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:34:33.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:34:32 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:34:33.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:34:33 smithi115 conmon[90393]: 2022-05-14T02:34:33.037+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:34:33.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:34:32 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:34:34.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:34:33 smithi047 ceph-mon[77147]: pgmap v5955: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:34:34.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:34:33 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:34:34.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:34:33 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:34:34.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:34:33 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:34:34.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:34:34 smithi115 conmon[90393]: 2022-05-14T02:34:34.020+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:34:34.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:34:33 smithi115 ceph-mon[78593]: pgmap v5955: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:34:34.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:34:33 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:34:34.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:34:33 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:34:34.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:34:33 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:34:35.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:34:34 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:34:35.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:34:34 smithi115 conmon[90393]: 2022-05-14T02:34:34.991+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:34:35.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:34:34 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:34:36.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:34:35 smithi047 ceph-mon[77147]: pgmap v5956: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:34:36.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:34:35 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:34:36.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:34:35 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 11210 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:34:36.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:34:36 smithi115 conmon[90393]: 2022-05-14T02:34:36.009+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:34:36.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:34:35 smithi115 ceph-mon[78593]: pgmap v5956: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:34:36.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:34:35 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:34:36.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:34:35 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 11210 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:34:37.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:34:36 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:34:37.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:34:37 smithi115 conmon[90393]: 2022-05-14T02:34:37.044+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:34:37.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:34:36 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:34:38.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:34:37 smithi047 ceph-mon[77147]: pgmap v5957: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:34:38.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:34:37 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:34:38.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:34:37 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:34:38.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:34:37 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:34:38.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:34:38 smithi115 conmon[90393]: 2022-05-14T02:34:38.087+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:34:38.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:34:37 smithi115 ceph-mon[78593]: pgmap v5957: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:34:38.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:34:37 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:34:38.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:34:37 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:34:38.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:34:37 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:34:39.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:34:38 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:34:39.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:34:39 smithi115 conmon[90393]: 2022-05-14T02:34:39.052+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:34:39.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:34:38 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:34:40.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:34:39 smithi047 ceph-mon[77147]: pgmap v5958: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:34:40.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:34:39 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:34:40.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:34:40 smithi115 conmon[90393]: 2022-05-14T02:34:40.091+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:34:40.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:34:39 smithi115 ceph-mon[78593]: pgmap v5958: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:34:40.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:34:39 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:34:41.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:34:40 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:34:41.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:34:41 smithi115 conmon[90393]: 2022-05-14T02:34:41.110+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:34:41.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:34:40 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:34:42.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:34:41 smithi047 ceph-mon[77147]: pgmap v5959: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:34:42.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:34:41 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 11215 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:34:42.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:34:41 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:34:42.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:34:41 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:34:42.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:34:41 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:34:42.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:34:42 smithi115 conmon[90393]: 2022-05-14T02:34:42.124+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:34:42.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:34:41 smithi115 ceph-mon[78593]: pgmap v5959: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:34:42.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:34:41 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 11215 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:34:42.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:34:41 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:34:42.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:34:41 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:34:42.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:34:41 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:34:43.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:34:42 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:34:43.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:34:43 smithi115 conmon[90393]: 2022-05-14T02:34:43.130+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:34:43.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:34:42 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:34:44.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:34:43 smithi047 ceph-mon[77147]: pgmap v5960: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:34:44.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:34:43 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:34:44.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:34:44 smithi115 conmon[90393]: 2022-05-14T02:34:44.178+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:34:44.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:34:43 smithi115 ceph-mon[78593]: pgmap v5960: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:34:44.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:34:43 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:34:45.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:34:45 smithi115 conmon[90393]: 2022-05-14T02:34:45.212+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:34:46.263 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:34:45 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:34:46.264 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:34:45 smithi115 ceph-mon[78593]: pgmap v5961: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:34:46.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:34:45 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:34:46.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:34:45 smithi047 ceph-mon[77147]: pgmap v5961: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:34:46.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:34:46 smithi115 conmon[90393]: 2022-05-14T02:34:46.262+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:34:47.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:34:46 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:34:47.309 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:34:46 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:34:47.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:34:47 smithi115 conmon[90393]: 2022-05-14T02:34:47.308+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:34:48.286 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:34:47 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:34:48.287 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:34:47 smithi115 ceph-mon[78593]: pgmap v5962: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:34:48.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:34:47 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:34:48.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:34:47 smithi047 ceph-mon[77147]: pgmap v5962: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:34:48.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:34:48 smithi115 conmon[90393]: 2022-05-14T02:34:48.285+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:34:49.284 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:34:48 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:34:49.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:34:48 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:34:49.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:34:49 smithi115 conmon[90393]: 2022-05-14T02:34:49.283+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:34:50.253 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:34:49 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:34:50.253 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:34:49 smithi115 ceph-mon[78593]: pgmap v5963: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:34:50.254 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:34:49 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:34:50.254 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:34:49 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:34:50.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:34:49 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:34:50.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:34:49 smithi047 ceph-mon[77147]: pgmap v5963: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:34:50.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:34:49 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:34:50.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:34:49 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:34:50.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:34:50 smithi115 conmon[90393]: 2022-05-14T02:34:50.251+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:34:51.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:34:51 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:34:51.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:34:51 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 11220 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:34:51.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:34:51 smithi115 conmon[90393]: 2022-05-14T02:34:51.227+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:34:51.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:34:51 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:34:51.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:34:51 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 11220 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:34:52.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:34:52 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:34:52.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:34:52 smithi047 ceph-mon[77147]: pgmap v5964: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:34:52.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:34:52 smithi115 conmon[90393]: 2022-05-14T02:34:52.195+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:34:52.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:34:52 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:34:52.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:34:52 smithi115 ceph-mon[78593]: pgmap v5964: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:34:53.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:34:53 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:34:53.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:34:53 smithi115 conmon[90393]: 2022-05-14T02:34:53.218+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:34:53.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:34:53 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:34:54.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:34:54 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:34:54.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:34:54 smithi047 ceph-mon[77147]: pgmap v5965: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:34:54.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:34:54 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:34:54.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:34:54 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:34:54.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:34:54 smithi115 conmon[90393]: 2022-05-14T02:34:54.183+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:34:54.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:34:54 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:34:54.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:34:54 smithi115 ceph-mon[78593]: pgmap v5965: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:34:54.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:34:54 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:34:54.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:34:54 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:34:55.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:34:55 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:34:55.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:34:55 smithi115 conmon[90393]: 2022-05-14T02:34:55.134+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:34:55.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:34:55 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:34:56.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:34:56 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:34:56.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:34:56 smithi047 ceph-mon[77147]: pgmap v5966: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:34:56.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:34:56 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:34:56.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:34:56 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 11230 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:34:56.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:34:56 smithi115 conmon[90393]: 2022-05-14T02:34:56.149+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:34:56.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:34:56 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:34:56.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:34:56 smithi115 ceph-mon[78593]: pgmap v5966: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:34:56.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:34:56 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:34:56.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:34:56 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 11230 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:34:57.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:34:57 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:34:57.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:34:57 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:34:57.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:34:57 smithi115 conmon[90393]: 2022-05-14T02:34:57.165+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:34:58.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:34:58 smithi047 ceph-mon[77147]: pgmap v5967: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:34:58.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:34:58 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:34:58.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:34:58 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:34:58.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:34:58 smithi115 conmon[90393]: 2022-05-14T02:34:58.120+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:34:58.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:34:58 smithi115 ceph-mon[78593]: pgmap v5967: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:34:58.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:34:58 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:34:58.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:34:58 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:34:59.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:34:59 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:34:59.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:34:59 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:34:59.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:34:59 smithi115 conmon[90393]: 2022-05-14T02:34:59.075+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:34:59.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:34:59 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:34:59.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:34:59 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:35:00.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:35:00 smithi047 ceph-mon[77147]: pgmap v5968: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:35:00.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:35:00 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:35:00.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:35:00 smithi115 conmon[90393]: 2022-05-14T02:35:00.080+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:35:00.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:35:00 smithi115 ceph-mon[78593]: pgmap v5968: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:35:00.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:35:00 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:35:01.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:35:01 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:35:01.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:35:01 smithi115 conmon[90393]: 2022-05-14T02:35:01.114+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:35:01.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:35:01 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:35:02.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:35:02 smithi047 ceph-mon[77147]: pgmap v5969: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:35:02.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:35:02 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 11235 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:35:02.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:35:02 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:35:02.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:35:02 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:35:02.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:35:02 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:35:02.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:35:02 smithi115 conmon[90393]: 2022-05-14T02:35:02.133+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:35:02.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:35:02 smithi115 ceph-mon[78593]: pgmap v5969: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:35:02.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:35:02 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 11235 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:35:02.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:35:02 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:35:02.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:35:02 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:35:02.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:35:02 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:35:03.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:35:03 smithi115 conmon[90393]: 2022-05-14T02:35:03.085+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:35:03.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:35:03 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:35:03.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:35:03 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:35:04.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:35:04 smithi115 conmon[90393]: 2022-05-14T02:35:04.046+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:35:04.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:35:04 smithi115 ceph-mon[78593]: pgmap v5970: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:35:04.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:35:04 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:35:04.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:35:04 smithi047 ceph-mon[77147]: pgmap v5970: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:35:04.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:35:04 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:35:05.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:35:05 smithi115 conmon[90393]: 2022-05-14T02:35:05.017+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:35:05.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:35:05 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:35:05.472 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:35:05 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:35:06.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:35:06 smithi047 ceph-mon[77147]: pgmap v5971: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:35:06.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:35:06 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:35:06.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:35:06 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T02:35:06.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:35:06 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T02:35:06.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:35:06 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T02:35:06.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:35:06 smithi115 conmon[90393]: 2022-05-14T02:35:06.025+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:35:06.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:35:06 smithi115 ceph-mon[78593]: pgmap v5971: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:35:06.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:35:06 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:35:06.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:35:06 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T02:35:06.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:35:06 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T02:35:06.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:35:06 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T02:35:07.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:35:06 smithi115 conmon[90393]: 2022-05-14T02:35:06.987+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:35:07.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:35:07 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:35:07.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:35:07 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:35:08.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:35:07 smithi115 conmon[90393]: 2022-05-14T02:35:07.938+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:35:08.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:35:08 smithi115 ceph-mon[78593]: pgmap v5972: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:35:08.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:35:08 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:35:08.530 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:35:08 smithi047 ceph-mon[77147]: pgmap v5972: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:35:08.530 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:35:08 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:35:08.791 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:35:08 smithi047 conmon[78521]: 2022-05-14T02:35:08.528+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:35:08.792 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:35:08 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T02:35:08.792 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:35:08 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:35:08.792 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:35:08 smithi047 conmon[78521]: yield 2022-05-14T02:35:08.793 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:35:08 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:35:08.793 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:35:08 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:35:08.793 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:35:08 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:35:08.793 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:35:08 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T02:35:08.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:35:08 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:35:08.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:35:08 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T02:35:08.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:35:08 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:35:08.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:35:08 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T02:35:08.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:35:08 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:35:08.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:35:08 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T02:35:08.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:35:08 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:35:08.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:35:08 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T02:35:08.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:35:08 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:35:08.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:35:08 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:35:08.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:35:08 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:35:09.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:35:08 smithi115 conmon[90393]: 2022-05-14T02:35:08.970+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:35:09.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:35:09 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:35:09.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:35:09 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:35:10.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:35:09 smithi115 conmon[90393]: 2022-05-14T02:35:09.993+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:35:10.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:35:10 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:35:10.359 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T02:35:10.359 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:35:10.359 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T02:35:10.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:35:10.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:35:10.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:35:10.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T02:35:10.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:35:10.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T02:35:10.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:35:10.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T02:35:10.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:35:10.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T02:35:10.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:35:10.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T02:35:10.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:35:10.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:35:10.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:35:10.363 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:35:10 smithi115 ceph-mon[78593]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T02:35:10.363 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:35:10 smithi115 ceph-mon[78593]: pgmap v5973: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:35:10.363 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:35:10 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:35:10.364 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:35:10 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:35:10.364 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:35:10 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:35:10.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:35:10 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:35:10.542 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T02:35:10.542 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:35:10.542 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T02:35:10.542 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:35:10.542 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:35:10.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:35:10.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T02:35:10.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:35:10.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T02:35:10.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:35:10.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T02:35:10.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:35:10.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T02:35:10.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:35:10.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T02:35:10.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:35:10.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:35:10.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:35:10.546 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:35:10 smithi047 ceph-mon[77147]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T02:35:10.546 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:35:10 smithi047 ceph-mon[77147]: pgmap v5973: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:35:10.546 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:35:10 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:35:10.546 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:35:10 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:35:10.546 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:35:10 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:35:11.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:35:10 smithi115 conmon[90393]: 2022-05-14T02:35:10.982+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:35:11.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:35:11 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:35:11.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:35:11 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 11240 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:35:11.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:35:11 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:35:11.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:35:11 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 11240 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:35:12.042 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:35:11 smithi047 conmon[78521]: 2022-05-14T02:35:11.601+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:35:12.042 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:35:11 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T02:35:12.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:35:11 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:35:12.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:35:11 smithi047 conmon[78521]: yield 2022-05-14T02:35:12.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:35:11 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:35:12.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:35:11 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:35:12.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:35:11 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:35:12.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:35:11 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T02:35:12.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:35:11 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:35:12.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:35:11 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T02:35:12.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:35:11 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:35:12.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:35:11 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T02:35:12.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:35:11 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:35:12.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:35:11 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T02:35:12.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:35:11 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:35:12.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:35:11 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T02:35:12.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:35:11 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:35:12.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:35:11 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:35:12.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:35:11 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:35:12.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:35:11 smithi047 conmon[78521]: 2022-05-14T02:35:11.601+0000 7f5341f7c700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T02:35:12.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:35:11 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T02:35:12.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:35:11 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:35:12.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:35:11 smithi047 conmon[78521]: yield 2022-05-14T02:35:12.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:35:11 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:35:12.048 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:35:11 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:35:12.048 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:35:11 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:35:12.048 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:35:11 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T02:35:12.048 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:35:11 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:35:12.049 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:35:11 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T02:35:12.049 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:35:11 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:35:12.049 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:35:11 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T02:35:12.049 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:35:11 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:35:12.049 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:35:11 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T02:35:12.050 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:35:11 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:35:12.050 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:35:11 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T02:35:12.050 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:35:11 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:35:12.050 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:35:11 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:35:12.051 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:35:11 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:35:12.051 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:35:11 smithi047 conmon[78521]: 2022-05-14T02:35:12.051 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:35:11 smithi047 conmon[78521]: During handling of the above exception, another exception occurred: 2022-05-14T02:35:12.051 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:35:11 smithi047 conmon[78521]: 2022-05-14T02:35:12.052 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:35:11 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T02:35:12.052 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:35:11 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T02:35:12.052 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:35:11 smithi047 conmon[78521]: return f(*arg) 2022-05-14T02:35:12.052 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:35:11 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T02:35:12.052 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:35:11 smithi047 conmon[78521]: self._write_client_files(client_files, host) 2022-05-14T02:35:12.053 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:35:11 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T02:35:12.053 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:35:11 smithi047 conmon[78521]: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T02:35:12.053 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:35:11 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T02:35:12.053 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:35:11 smithi047 conmon[78521]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T02:35:12.054 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:35:11 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T02:35:12.054 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:35:11 smithi047 conmon[78521]: return self.event_loop.get_result(coro) 2022-05-14T02:35:12.054 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:35:11 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T02:35:12.054 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:35:11 smithi047 conmon[78521]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T02:35:12.054 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:35:11 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T02:35:12.055 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:35:11 smithi047 conmon[78521]: return self.__get_result() 2022-05-14T02:35:12.055 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:35:11 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T02:35:12.055 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:35:11 smithi047 conmon[78521]: raise self._exception 2022-05-14T02:35:12.055 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:35:11 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T02:35:12.056 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:35:11 smithi047 conmon[78521]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T02:35:12.056 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:35:11 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T02:35:12.056 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:35:11 smithi047 conmon[78521]: conn = await self._remote_connection(host, addr) 2022-05-14T02:35:12.056 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:35:11 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T02:35:12.056 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:35:11 smithi047 conmon[78521]: raise 2022-05-14T02:35:12.057 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:35:11 smithi047 conmon[78521]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T02:35:12.057 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:35:11 smithi047 conmon[78521]: self.gen.throw(type, value, traceback) 2022-05-14T02:35:12.057 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:35:11 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T02:35:12.057 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:35:11 smithi047 conmon[78521]: raise OrchestratorError(msg) 2022-05-14T02:35:12.058 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:35:11 smithi047 conmon[78521]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:35:12.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:35:11 smithi115 conmon[90393]: 2022-05-14T02:35:11.949+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:35:12.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:35:12 smithi115 ceph-mon[78593]: pgmap v5974: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:35:12.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:35:12 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:35:12.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:35:12 smithi047 ceph-mon[77147]: pgmap v5974: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:35:12.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:35:12 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:35:13.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:35:12 smithi115 conmon[90393]: 2022-05-14T02:35:12.952+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:35:13.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:35:13 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:35:13.359 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T02:35:13.359 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:35:13.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T02:35:13.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:35:13.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:35:13.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:35:13.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T02:35:13.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:35:13.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T02:35:13.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:35:13.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T02:35:13.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:35:13.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T02:35:13.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:35:13.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T02:35:13.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:35:13.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:35:13.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:35:13.365 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:35:13 smithi115 ceph-mon[78593]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T02:35:13.365 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T02:35:13.366 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:35:13.366 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T02:35:13.366 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:35:13.367 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:35:13.367 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:35:13.368 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T02:35:13.368 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:35:13.369 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T02:35:13.369 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:35:13.369 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T02:35:13.370 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:35:13.370 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T02:35:13.370 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:35:13.371 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T02:35:13.372 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:35:13.372 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:35:13.372 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:35:13.373 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T02:35:13.373 INFO:journalctl@ceph.mon.c.smithi115.stdout: During handling of the above exception, another exception occurred: 2022-05-14T02:35:13.374 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T02:35:13.374 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T02:35:13.375 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T02:35:13.375 INFO:journalctl@ceph.mon.c.smithi115.stdout: return f(*arg) 2022-05-14T02:35:13.429 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T02:35:13.430 INFO:journalctl@ceph.mon.c.smithi115.stdout: self._write_client_files(client_files, host) 2022-05-14T02:35:13.430 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T02:35:13.430 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T02:35:13.430 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T02:35:13.431 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T02:35:13.431 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T02:35:13.431 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.event_loop.get_result(coro) 2022-05-14T02:35:13.431 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T02:35:13.431 INFO:journalctl@ceph.mon.c.smithi115.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T02:35:13.432 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T02:35:13.432 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.__get_result() 2022-05-14T02:35:13.432 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T02:35:13.432 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise self._exception 2022-05-14T02:35:13.433 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T02:35:13.433 INFO:journalctl@ceph.mon.c.smithi115.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T02:35:13.433 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T02:35:13.433 INFO:journalctl@ceph.mon.c.smithi115.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T02:35:13.433 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T02:35:13.434 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise 2022-05-14T02:35:13.434 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T02:35:13.434 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.gen.throw(type, value, traceback) 2022-05-14T02:35:13.434 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T02:35:13.435 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OrchestratorError(msg) 2022-05-14T02:35:13.435 INFO:journalctl@ceph.mon.c.smithi115.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:35:13.435 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:35:13 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:35:13.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:35:13 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:35:13.542 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T02:35:13.542 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:35:13.542 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T02:35:13.542 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:35:13.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:35:13.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:35:13.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T02:35:13.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:35:13.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T02:35:13.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:35:13.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T02:35:13.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:35:13.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T02:35:13.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:35:13.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T02:35:13.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:35:13.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:35:13.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:35:13.546 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:35:13 smithi047 ceph-mon[77147]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T02:35:13.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T02:35:13.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:35:13.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T02:35:13.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:35:13.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:35:13.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:35:13.548 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T02:35:13.548 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:35:13.548 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T02:35:13.548 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:35:13.548 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T02:35:13.549 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:35:13.549 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T02:35:13.549 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:35:13.549 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T02:35:13.550 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:35:13.550 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:35:13.550 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:35:13.550 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T02:35:13.551 INFO:journalctl@ceph.mon.b.smithi047.stdout: During handling of the above exception, another exception occurred: 2022-05-14T02:35:13.551 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T02:35:13.551 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T02:35:13.551 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T02:35:13.551 INFO:journalctl@ceph.mon.b.smithi047.stdout: return f(*arg) 2022-05-14T02:35:13.552 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T02:35:13.552 INFO:journalctl@ceph.mon.b.smithi047.stdout: self._write_client_files(client_files, host) 2022-05-14T02:35:13.552 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T02:35:13.552 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T02:35:13.553 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T02:35:13.553 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T02:35:13.553 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T02:35:13.553 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.event_loop.get_result(coro) 2022-05-14T02:35:13.554 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T02:35:13.554 INFO:journalctl@ceph.mon.b.smithi047.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T02:35:13.554 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T02:35:13.554 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.__get_result() 2022-05-14T02:35:13.554 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T02:35:13.555 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise self._exception 2022-05-14T02:35:13.555 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T02:35:13.555 INFO:journalctl@ceph.mon.b.smithi047.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T02:35:13.555 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T02:35:13.556 INFO:journalctl@ceph.mon.b.smithi047.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T02:35:13.556 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T02:35:13.557 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise 2022-05-14T02:35:13.558 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T02:35:13.558 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.gen.throw(type, value, traceback) 2022-05-14T02:35:13.558 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T02:35:13.558 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OrchestratorError(msg) 2022-05-14T02:35:13.559 INFO:journalctl@ceph.mon.b.smithi047.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:35:13.559 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:35:13 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:35:14.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:35:13 smithi115 conmon[90393]: 2022-05-14T02:35:13.976+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:35:14.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:35:14 smithi115 ceph-mon[78593]: pgmap v5975: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:35:14.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:35:14 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:35:14.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:35:14 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:35:14.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:35:14 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:35:14.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:35:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T02:35:14.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:35:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T02:35:14.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:35:14 smithi047 ceph-mon[77147]: pgmap v5975: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:35:14.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:35:14 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:35:14.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:35:14 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:35:14.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:35:14 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:35:14.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:35:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T02:35:14.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:35:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T02:35:15.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:35:14 smithi115 conmon[90393]: 2022-05-14T02:35:14.989+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:35:15.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:35:15 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:35:15.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:35:15 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:35:16.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:35:16 smithi115 conmon[90393]: 2022-05-14T02:35:16.016+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:35:16.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:35:16 smithi115 ceph-mon[78593]: pgmap v5976: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:35:16.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:35:16 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:35:16.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:35:16 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 11250 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:35:16.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:35:16 smithi047 ceph-mon[77147]: pgmap v5976: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:35:16.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:35:16 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:35:16.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:35:16 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 11250 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:35:17.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:35:17 smithi115 conmon[90393]: 2022-05-14T02:35:17.014+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:35:17.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:35:17 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:35:17.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:35:17 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:35:18.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:35:17 smithi115 conmon[90393]: 2022-05-14T02:35:17.967+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:35:18.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:35:18 smithi115 ceph-mon[78593]: pgmap v5977: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:35:18.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:35:18 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:35:18.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:35:18 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:35:18.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:35:18 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:35:18.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:35:18 smithi047 ceph-mon[77147]: pgmap v5977: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:35:18.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:35:18 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:35:18.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:35:18 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:35:18.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:35:18 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:35:19.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:35:18 smithi115 conmon[90393]: 2022-05-14T02:35:18.997+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:35:19.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:35:19 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:35:19.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:35:19 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:35:20.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:35:20 smithi115 conmon[90393]: 2022-05-14T02:35:20.033+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:35:20.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:35:20 smithi115 ceph-mon[78593]: pgmap v5978: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:35:20.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:35:20 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:35:20.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:35:20 smithi047 ceph-mon[77147]: pgmap v5978: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:35:20.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:35:20 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:35:21.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:35:21 smithi115 conmon[90393]: 2022-05-14T02:35:21.010+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:35:21.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:35:21 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:35:21.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:35:21 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:35:22.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:35:22 smithi115 conmon[90393]: 2022-05-14T02:35:22.028+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:35:22.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:35:22 smithi115 ceph-mon[78593]: pgmap v5979: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:35:22.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:35:22 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:35:22.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:35:22 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 11255 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:35:22.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:35:22 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:35:22.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:35:22 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:35:22.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:35:22 smithi047 ceph-mon[77147]: pgmap v5979: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:35:22.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:35:22 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:35:22.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:35:22 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 11255 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:35:22.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:35:22 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:35:22.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:35:22 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:35:23.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:35:22 smithi115 conmon[90393]: 2022-05-14T02:35:22.980+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:35:23.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:35:23 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:35:23.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:35:23 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:35:24.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:35:23 smithi115 conmon[90393]: 2022-05-14T02:35:23.935+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:35:24.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:35:24 smithi115 ceph-mon[78593]: pgmap v5980: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:35:24.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:35:24 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:35:24.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:35:24 smithi047 ceph-mon[77147]: pgmap v5980: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:35:24.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:35:24 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:35:25.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:35:24 smithi115 conmon[90393]: 2022-05-14T02:35:24.963+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:35:25.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:35:25 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:35:25.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:35:25 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:35:26.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:35:25 smithi115 conmon[90393]: 2022-05-14T02:35:25.963+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:35:26.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:35:26 smithi115 ceph-mon[78593]: pgmap v5981: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:35:26.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:35:26 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:35:26.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:35:26 smithi047 ceph-mon[77147]: pgmap v5981: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:35:26.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:35:26 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:35:27.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:35:27 smithi115 conmon[90393]: 2022-05-14T02:35:27.006+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:35:27.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:35:27 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:35:27.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:35:27 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:35:28.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:35:28 smithi115 conmon[90393]: 2022-05-14T02:35:28.032+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:35:28.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:35:28 smithi115 ceph-mon[78593]: pgmap v5982: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:35:28.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:35:28 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:35:28.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:35:28 smithi047 ceph-mon[77147]: pgmap v5982: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:35:28.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:35:28 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:35:29.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:35:29 smithi115 conmon[90393]: 2022-05-14T02:35:29.058+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:35:29.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:35:29 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:35:29.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:35:29 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:35:30.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:35:30 smithi115 conmon[90393]: 2022-05-14T02:35:30.098+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:35:30.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:35:30 smithi115 ceph-mon[78593]: pgmap v5983: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:35:30.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:35:30 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:35:30.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:35:30 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:35:30.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:35:30 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:35:30.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:35:30 smithi047 ceph-mon[77147]: pgmap v5983: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:35:30.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:35:30 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:35:30.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:35:30 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:35:30.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:35:30 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:35:31.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:35:31 smithi115 conmon[90393]: 2022-05-14T02:35:31.055+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:35:31.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:35:31 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:35:31.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:35:31 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 11260 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:35:31.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:35:31 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:35:31.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:35:31 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 11260 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:35:32.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:35:32 smithi115 conmon[90393]: 2022-05-14T02:35:32.075+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:35:32.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:35:32 smithi115 ceph-mon[78593]: pgmap v5984: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:35:32.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:35:32 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:35:32.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:35:32 smithi047 ceph-mon[77147]: pgmap v5984: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:35:32.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:35:32 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:35:33.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:35:33 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:35:33.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:35:33 smithi115 conmon[90393]: 2022-05-14T02:35:33.045+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:35:33.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:35:33 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:35:34.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:35:34 smithi115 conmon[90393]: 2022-05-14T02:35:34.018+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:35:34.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:35:34 smithi115 ceph-mon[78593]: pgmap v5985: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:35:34.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:35:34 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:35:34.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:35:34 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:35:34.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:35:34 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:35:34.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:35:34 smithi047 ceph-mon[77147]: pgmap v5985: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:35:34.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:35:34 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:35:34.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:35:34 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:35:34.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:35:34 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:35:35.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:35:34 smithi115 conmon[90393]: 2022-05-14T02:35:34.978+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:35:35.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:35:35 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:35:35.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:35:35 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:35:36.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:35:36 smithi115 conmon[90393]: 2022-05-14T02:35:36.019+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:35:36.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:35:36 smithi115 ceph-mon[78593]: pgmap v5986: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:35:36.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:35:36 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:35:36.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:35:36 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 11270 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:35:36.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:35:36 smithi047 ceph-mon[77147]: pgmap v5986: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:35:36.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:35:36 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:35:36.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:35:36 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 11270 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:35:37.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:35:37 smithi115 conmon[90393]: 2022-05-14T02:35:37.032+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:35:37.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:35:37 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:35:37.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:35:37 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:35:38.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:35:38 smithi115 conmon[90393]: 2022-05-14T02:35:38.060+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:35:38.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:35:38 smithi115 ceph-mon[78593]: pgmap v5987: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:35:38.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:35:38 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:35:38.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:35:38 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:35:38.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:35:38 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:35:38.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:35:38 smithi047 ceph-mon[77147]: pgmap v5987: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:35:38.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:35:38 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:35:38.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:35:38 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:35:38.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:35:38 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:35:39.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:35:39 smithi115 conmon[90393]: 2022-05-14T02:35:39.088+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:35:39.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:35:39 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:35:39.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:35:39 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:35:40.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:35:40 smithi115 conmon[90393]: 2022-05-14T02:35:40.045+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:35:40.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:35:40 smithi115 ceph-mon[78593]: pgmap v5988: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:35:40.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:35:40 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:35:40.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:35:40 smithi047 ceph-mon[77147]: pgmap v5988: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:35:40.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:35:40 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:35:41.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:35:41 smithi115 conmon[90393]: 2022-05-14T02:35:41.072+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:35:41.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:35:41 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:35:41.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:35:41 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:35:42.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:35:42 smithi115 conmon[90393]: 2022-05-14T02:35:42.099+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:35:42.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:35:42 smithi115 ceph-mon[78593]: pgmap v5989: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:35:42.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:35:42 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:35:42.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:35:42 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 11276 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:35:42.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:35:42 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:35:42.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:35:42 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:35:42.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:35:42 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:35:42.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:35:42 smithi047 ceph-mon[77147]: pgmap v5989: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:35:42.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:35:42 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:35:42.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:35:42 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 11276 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:35:42.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:35:42 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:35:42.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:35:42 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:35:42.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:35:42 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:35:43.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:35:43 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:35:43.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:35:43 smithi115 conmon[90393]: 2022-05-14T02:35:43.108+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:35:43.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:35:43 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:35:44.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:35:44 smithi047 ceph-mon[77147]: pgmap v5990: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:35:44.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:35:44 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:35:44.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:35:44 smithi115 conmon[90393]: 2022-05-14T02:35:44.114+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:35:44.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:35:44 smithi115 ceph-mon[78593]: pgmap v5990: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:35:44.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:35:44 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:35:45.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:35:45 smithi115 conmon[90393]: 2022-05-14T02:35:45.102+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:35:45.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:35:45 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:35:45.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:35:45 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:35:46.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:35:46 smithi047 ceph-mon[77147]: pgmap v5991: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:35:46.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:35:46 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:35:46.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:35:46 smithi115 conmon[90393]: 2022-05-14T02:35:46.133+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:35:46.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:35:46 smithi115 ceph-mon[78593]: pgmap v5991: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:35:46.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:35:46 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:35:47.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:35:47 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:35:47.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:35:47 smithi115 conmon[90393]: 2022-05-14T02:35:47.150+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:35:47.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:35:47 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:35:48.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:35:48 smithi047 ceph-mon[77147]: pgmap v5992: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:35:48.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:35:48 smithi115 conmon[90393]: 2022-05-14T02:35:48.178+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:35:48.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:35:48 smithi115 ceph-mon[78593]: pgmap v5992: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:35:49.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:35:49 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:35:49.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:35:49 smithi115 conmon[90393]: 2022-05-14T02:35:49.198+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:35:49.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:35:49 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:35:50.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:35:50 smithi047 ceph-mon[77147]: pgmap v5993: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:35:50.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:35:50 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:35:50.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:35:50 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:35:50.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:35:50 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:35:50.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:35:50 smithi115 conmon[90393]: 2022-05-14T02:35:50.216+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:35:50.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:35:50 smithi115 ceph-mon[78593]: pgmap v5993: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:35:50.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:35:50 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:35:50.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:35:50 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:35:50.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:35:50 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:35:51.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:35:51 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:35:51.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:35:51 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 11281 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:35:51.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:35:51 smithi115 conmon[90393]: 2022-05-14T02:35:51.197+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:35:51.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:35:51 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:35:51.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:35:51 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 11281 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:35:52.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:35:52 smithi047 ceph-mon[77147]: pgmap v5994: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:35:52.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:35:52 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:35:52.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:35:52 smithi115 conmon[90393]: 2022-05-14T02:35:52.182+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:35:52.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:35:52 smithi115 ceph-mon[78593]: pgmap v5994: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:35:52.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:35:52 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:35:53.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:35:53 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:35:53.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:35:53 smithi115 conmon[90393]: 2022-05-14T02:35:53.195+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:35:53.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:35:53 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:35:54.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:35:54 smithi047 ceph-mon[77147]: pgmap v5995: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:35:54.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:35:54 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:35:54.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:35:54 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:35:54.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:35:54 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:35:54.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:35:54 smithi115 conmon[90393]: 2022-05-14T02:35:54.204+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:35:54.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:35:54 smithi115 ceph-mon[78593]: pgmap v5995: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:35:54.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:35:54 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:35:54.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:35:54 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:35:54.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:35:54 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:35:55.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:35:55 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:35:55.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:35:55 smithi115 conmon[90393]: 2022-05-14T02:35:55.191+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:35:55.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:35:55 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:35:56.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:35:56 smithi047 ceph-mon[77147]: pgmap v5996: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:35:56.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:35:56 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:35:56.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:35:56 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 11290 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:35:56.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:35:56 smithi115 conmon[90393]: 2022-05-14T02:35:56.196+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:35:56.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:35:56 smithi115 ceph-mon[78593]: pgmap v5996: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:35:56.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:35:56 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:35:56.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:35:56 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 11290 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:35:57.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:35:57 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:35:57.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:35:57 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:35:57.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:35:57 smithi115 conmon[90393]: 2022-05-14T02:35:57.198+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:35:58.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:35:58 smithi047 ceph-mon[77147]: pgmap v5997: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:35:58.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:35:58 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:35:58.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:35:58 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:35:58.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:35:58 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:35:58.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:35:58 smithi115 conmon[90393]: 2022-05-14T02:35:58.248+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:35:58.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:35:58 smithi115 ceph-mon[78593]: pgmap v5997: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:35:58.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:35:58 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:35:58.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:35:58 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:35:58.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:35:58 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:35:59.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:35:59 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:35:59.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:35:59 smithi115 conmon[90393]: 2022-05-14T02:35:59.240+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:35:59.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:35:59 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:36:00.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:36:00 smithi047 ceph-mon[77147]: pgmap v5998: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:36:00.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:36:00 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:36:00.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:36:00 smithi115 conmon[90393]: 2022-05-14T02:36:00.205+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:36:00.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:36:00 smithi115 ceph-mon[78593]: pgmap v5998: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:36:00.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:36:00 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:36:01.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:36:01 smithi115 conmon[90393]: 2022-05-14T02:36:01.156+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:36:02.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:36:01 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:36:02.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:36:01 smithi115 ceph-mon[78593]: pgmap v5999: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:36:02.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:36:01 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:36:02.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:36:01 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 11295 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:36:02.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:36:01 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:36:02.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:36:01 smithi047 ceph-mon[77147]: pgmap v5999: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:36:02.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:36:01 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:36:02.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:36:01 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 11295 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:36:02.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:36:02 smithi115 conmon[90393]: 2022-05-14T02:36:02.122+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:36:03.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:36:02 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:36:03.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:36:02 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:36:03.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:36:02 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:36:03.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:36:02 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:36:03.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:36:02 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:36:03.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:36:02 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:36:03.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:36:03 smithi115 conmon[90393]: 2022-05-14T02:36:03.118+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:36:04.071 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:36:03 smithi115 ceph-mon[78593]: pgmap v6000: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:36:04.072 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:36:03 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:36:04.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:36:03 smithi047 ceph-mon[77147]: pgmap v6000: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:36:04.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:36:03 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:36:04.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:36:04 smithi115 conmon[90393]: 2022-05-14T02:36:04.070+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:36:05.088 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:36:04 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:36:05.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:36:04 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:36:05.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:36:05 smithi115 conmon[90393]: 2022-05-14T02:36:05.087+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:36:06.088 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:36:05 smithi115 ceph-mon[78593]: pgmap v6001: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:36:06.088 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:36:05 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:36:06.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:36:05 smithi047 ceph-mon[77147]: pgmap v6001: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:36:06.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:36:05 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:36:06.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:36:06 smithi115 conmon[90393]: 2022-05-14T02:36:06.087+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:36:07.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:36:06 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:36:07.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:36:06 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:36:07.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:36:07 smithi115 conmon[90393]: 2022-05-14T02:36:07.119+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:36:08.071 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:36:07 smithi115 ceph-mon[78593]: pgmap v6002: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:36:08.071 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:36:07 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:36:08.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:36:07 smithi047 ceph-mon[77147]: pgmap v6002: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:36:08.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:36:07 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:36:08.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:36:08 smithi115 conmon[90393]: 2022-05-14T02:36:08.069+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:36:09.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:36:09 smithi115 conmon[90393]: 2022-05-14T02:36:09.047+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:36:09.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:36:08 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:36:09.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:36:08 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:36:10.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:36:10 smithi115 conmon[90393]: 2022-05-14T02:36:10.055+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:36:10.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:36:09 smithi115 ceph-mon[78593]: pgmap v6003: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:36:10.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:36:09 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:36:10.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:36:09 smithi047 ceph-mon[77147]: pgmap v6003: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:36:10.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:36:09 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:36:11.110 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:36:11 smithi115 conmon[90393]: 2022-05-14T02:36:11.048+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:36:11.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:36:10 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:36:11.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:36:10 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:36:11.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:36:10 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:36:11.112 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:36:10 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 11300 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:36:11.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:36:10 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:36:11.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:36:10 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:36:11.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:36:10 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:36:11.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:36:10 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 11300 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:36:12.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:36:11 smithi047 ceph-mon[77147]: pgmap v6004: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:36:12.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:36:11 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:36:12.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:36:11 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T02:36:12.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:36:11 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T02:36:12.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:36:11 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T02:36:12.066 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:36:11 smithi115 ceph-mon[78593]: pgmap v6004: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:36:12.066 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:36:11 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:36:12.067 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:36:11 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T02:36:12.067 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:36:11 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T02:36:12.067 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:36:11 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T02:36:12.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:36:12 smithi115 conmon[90393]: 2022-05-14T02:36:12.064+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:36:12.541 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:36:12 smithi047 conmon[78521]: 2022-05-14T02:36:12.079+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:36:12.542 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:36:12 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T02:36:12.542 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:36:12 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:36:12.542 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:36:12 smithi047 conmon[78521]: yield 2022-05-14T02:36:12.542 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:36:12 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:36:12.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:36:12 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:36:12.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:36:12 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:36:12.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:36:12 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T02:36:12.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:36:12 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:36:12.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:36:12 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T02:36:12.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:36:12 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:36:12.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:36:12 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T02:36:12.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:36:12 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:36:12.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:36:12 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T02:36:12.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:36:12 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:36:12.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:36:12 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T02:36:12.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:36:12 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:36:12.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:36:12 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:36:12.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:36:12 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:36:13.091 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:36:12 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:36:13.091 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:36:12 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:36:13.091 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T02:36:13.092 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:36:13.092 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T02:36:13.092 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:36:13.092 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:36:13.092 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:36:13.093 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T02:36:13.093 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:36:13.093 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T02:36:13.093 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:36:13.094 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T02:36:13.094 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:36:13.094 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T02:36:13.094 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:36:13.095 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T02:36:13.095 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:36:13.095 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:36:13.102 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:36:13.102 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:36:12 smithi115 ceph-mon[78593]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T02:36:13.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:36:12 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:36:13.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:36:12 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:36:13.292 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T02:36:13.292 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:36:13.292 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T02:36:13.292 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:36:13.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:36:13.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:36:13.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T02:36:13.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:36:13.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T02:36:13.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:36:13.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T02:36:13.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:36:13.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T02:36:13.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:36:13.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T02:36:13.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:36:13.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:36:13.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:36:13.296 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:36:12 smithi047 ceph-mon[77147]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T02:36:13.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:36:13 smithi115 conmon[90393]: 2022-05-14T02:36:13.089+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:36:14.102 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:36:13 smithi115 ceph-mon[78593]: pgmap v6005: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:36:14.103 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:36:13 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:36:14.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:36:13 smithi047 ceph-mon[77147]: pgmap v6005: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:36:14.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:36:13 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:36:14.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:36:14 smithi115 conmon[90393]: 2022-05-14T02:36:14.100+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:36:15.097 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:36:14 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:36:15.098 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:36:14 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:36:15.098 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:36:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T02:36:15.098 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:36:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T02:36:15.098 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:36:14 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:36:15.154 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:36:14 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:36:15.154 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:36:14 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:36:15.155 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:36:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T02:36:15.155 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:36:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T02:36:15.155 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:36:14 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:36:15.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:36:15 smithi115 conmon[90393]: 2022-05-14T02:36:15.095+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:36:15.541 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:36:15 smithi047 conmon[78521]: 2022-05-14T02:36:15.151+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:36:15.542 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:36:15 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T02:36:15.542 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:36:15 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:36:15.542 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:36:15 smithi047 conmon[78521]: yield 2022-05-14T02:36:15.542 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:36:15 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:36:15.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:36:15 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:36:15.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:36:15 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:36:15.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:36:15 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T02:36:15.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:36:15 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:36:15.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:36:15 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T02:36:15.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:36:15 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:36:15.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:36:15 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T02:36:15.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:36:15 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:36:15.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:36:15 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T02:36:15.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:36:15 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:36:15.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:36:15 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T02:36:15.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:36:15 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:36:15.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:36:15 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:36:15.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:36:15 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:36:15.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:36:15 smithi047 conmon[78521]: 2022-05-14T02:36:15.152+0000 7f5345f84700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T02:36:15.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:36:15 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T02:36:15.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:36:15 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:36:15.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:36:15 smithi047 conmon[78521]: yield 2022-05-14T02:36:15.547 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:36:15 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:36:15.547 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:36:15 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:36:15.547 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:36:15 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:36:15.547 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:36:15 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T02:36:15.547 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:36:15 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:36:15.548 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:36:15 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T02:36:15.548 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:36:15 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:36:15.548 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:36:15 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T02:36:15.548 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:36:15 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:36:15.549 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:36:15 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T02:36:15.549 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:36:15 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:36:15.549 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:36:15 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T02:36:15.549 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:36:15 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:36:15.549 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:36:15 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:36:15.550 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:36:15 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:36:15.550 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:36:15 smithi047 conmon[78521]: 2022-05-14T02:36:15.550 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:36:15 smithi047 conmon[78521]: During handling of the above exception, another exception occurred: 2022-05-14T02:36:15.550 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:36:15 smithi047 conmon[78521]: 2022-05-14T02:36:15.551 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:36:15 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T02:36:15.551 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:36:15 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T02:36:15.551 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:36:15 smithi047 conmon[78521]: return f(*arg) 2022-05-14T02:36:15.551 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:36:15 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T02:36:15.551 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:36:15 smithi047 conmon[78521]: self._write_client_files(client_files, host) 2022-05-14T02:36:15.552 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:36:15 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T02:36:15.552 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:36:15 smithi047 conmon[78521]: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T02:36:15.552 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:36:15 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T02:36:15.552 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:36:15 smithi047 conmon[78521]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T02:36:15.552 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:36:15 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T02:36:15.553 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:36:15 smithi047 conmon[78521]: return self.event_loop.get_result(coro) 2022-05-14T02:36:15.553 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:36:15 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T02:36:15.553 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:36:15 smithi047 conmon[78521]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T02:36:15.553 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:36:15 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T02:36:15.554 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:36:15 smithi047 conmon[78521]: return self.__get_result() 2022-05-14T02:36:15.554 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:36:15 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T02:36:15.554 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:36:15 smithi047 conmon[78521]: raise self._exception 2022-05-14T02:36:15.554 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:36:15 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T02:36:15.554 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:36:15 smithi047 conmon[78521]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T02:36:15.555 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:36:15 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T02:36:15.555 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:36:15 smithi047 conmon[78521]: conn = await self._remote_connection(host, addr) 2022-05-14T02:36:15.555 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:36:15 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T02:36:15.555 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:36:15 smithi047 conmon[78521]: raise 2022-05-14T02:36:15.556 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:36:15 smithi047 conmon[78521]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T02:36:15.556 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:36:15 smithi047 conmon[78521]: self.gen.throw(type, value, traceback) 2022-05-14T02:36:15.556 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:36:15 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T02:36:15.556 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:36:15 smithi047 conmon[78521]: raise OrchestratorError(msg) 2022-05-14T02:36:15.556 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:36:15 smithi047 conmon[78521]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:36:16.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:36:15 smithi115 ceph-mon[78593]: pgmap v6006: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:36:16.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:36:15 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:36:16.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:36:15 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:36:16.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T02:36:16.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:36:16.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T02:36:16.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:36:16.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:36:16.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:36:16.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T02:36:16.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:36:16.113 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T02:36:16.113 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:36:16.114 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T02:36:16.114 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:36:16.115 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T02:36:16.115 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:36:16.116 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T02:36:16.116 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:36:16.117 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:36:16.117 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:36:16.117 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:36:15 smithi115 ceph-mon[78593]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T02:36:16.118 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T02:36:16.118 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:36:16.119 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T02:36:16.119 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:36:16.120 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:36:16.120 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:36:16.120 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T02:36:16.121 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:36:16.122 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T02:36:16.123 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:36:16.124 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T02:36:16.124 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:36:16.124 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T02:36:16.124 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:36:16.125 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T02:36:16.125 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:36:16.125 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:36:16.126 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:36:16.126 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T02:36:16.126 INFO:journalctl@ceph.mon.c.smithi115.stdout: During handling of the above exception, another exception occurred: 2022-05-14T02:36:16.126 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T02:36:16.127 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T02:36:16.127 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T02:36:16.128 INFO:journalctl@ceph.mon.c.smithi115.stdout: return f(*arg) 2022-05-14T02:36:16.128 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T02:36:16.129 INFO:journalctl@ceph.mon.c.smithi115.stdout: self._write_client_files(client_files, host) 2022-05-14T02:36:16.129 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T02:36:16.130 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T02:36:16.130 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T02:36:16.131 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T02:36:16.131 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T02:36:16.132 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.event_loop.get_result(coro) 2022-05-14T02:36:16.132 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T02:36:16.133 INFO:journalctl@ceph.mon.c.smithi115.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T02:36:16.133 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T02:36:16.134 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.__get_result() 2022-05-14T02:36:16.134 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T02:36:16.134 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise self._exception 2022-05-14T02:36:16.135 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T02:36:16.135 INFO:journalctl@ceph.mon.c.smithi115.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T02:36:16.141 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T02:36:16.141 INFO:journalctl@ceph.mon.c.smithi115.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T02:36:16.144 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T02:36:16.144 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise 2022-05-14T02:36:16.144 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T02:36:16.145 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.gen.throw(type, value, traceback) 2022-05-14T02:36:16.145 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T02:36:16.145 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OrchestratorError(msg) 2022-05-14T02:36:16.145 INFO:journalctl@ceph.mon.c.smithi115.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:36:16.146 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:36:15 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 11311 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:36:16.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:36:15 smithi047 ceph-mon[77147]: pgmap v6006: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:36:16.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:36:15 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:36:16.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:36:15 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:36:16.292 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T02:36:16.292 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:36:16.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T02:36:16.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:36:16.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:36:16.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:36:16.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T02:36:16.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:36:16.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T02:36:16.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:36:16.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T02:36:16.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:36:16.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T02:36:16.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:36:16.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T02:36:16.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:36:16.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:36:16.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:36:16.296 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:36:15 smithi047 ceph-mon[77147]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T02:36:16.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T02:36:16.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:36:16.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T02:36:16.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:36:16.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:36:16.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:36:16.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T02:36:16.298 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:36:16.298 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T02:36:16.298 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:36:16.299 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T02:36:16.299 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:36:16.299 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T02:36:16.300 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:36:16.300 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T02:36:16.301 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:36:16.302 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:36:16.302 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:36:16.302 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T02:36:16.302 INFO:journalctl@ceph.mon.b.smithi047.stdout: During handling of the above exception, another exception occurred: 2022-05-14T02:36:16.303 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T02:36:16.303 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T02:36:16.303 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T02:36:16.303 INFO:journalctl@ceph.mon.b.smithi047.stdout: return f(*arg) 2022-05-14T02:36:16.303 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T02:36:16.304 INFO:journalctl@ceph.mon.b.smithi047.stdout: self._write_client_files(client_files, host) 2022-05-14T02:36:16.304 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T02:36:16.304 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T02:36:16.304 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T02:36:16.305 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T02:36:16.305 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T02:36:16.305 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.event_loop.get_result(coro) 2022-05-14T02:36:16.305 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T02:36:16.305 INFO:journalctl@ceph.mon.b.smithi047.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T02:36:16.306 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T02:36:16.306 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.__get_result() 2022-05-14T02:36:16.306 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T02:36:16.306 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise self._exception 2022-05-14T02:36:16.307 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T02:36:16.307 INFO:journalctl@ceph.mon.b.smithi047.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T02:36:16.307 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T02:36:16.307 INFO:journalctl@ceph.mon.b.smithi047.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T02:36:16.307 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T02:36:16.308 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise 2022-05-14T02:36:16.308 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T02:36:16.308 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.gen.throw(type, value, traceback) 2022-05-14T02:36:16.308 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T02:36:16.309 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OrchestratorError(msg) 2022-05-14T02:36:16.309 INFO:journalctl@ceph.mon.b.smithi047.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:36:16.310 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:36:15 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 11311 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:36:16.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:36:16 smithi115 conmon[90393]: 2022-05-14T02:36:16.116+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:36:17.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:36:16 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:36:17.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:36:16 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:36:17.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:36:17 smithi115 conmon[90393]: 2022-05-14T02:36:17.152+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:36:18.113 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:36:17 smithi115 ceph-mon[78593]: pgmap v6007: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:36:18.113 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:36:17 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:36:18.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:36:17 smithi047 ceph-mon[77147]: pgmap v6007: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:36:18.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:36:17 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:36:18.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:36:18 smithi115 conmon[90393]: 2022-05-14T02:36:18.119+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:36:19.121 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:36:18 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:36:19.121 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:36:18 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:36:19.122 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:36:18 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:36:19.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:36:18 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:36:19.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:36:18 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:36:19.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:36:18 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:36:19.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:36:19 smithi115 conmon[90393]: 2022-05-14T02:36:19.119+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:36:20.134 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:36:19 smithi115 ceph-mon[78593]: pgmap v6008: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:36:20.137 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:36:19 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:36:20.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:36:19 smithi047 ceph-mon[77147]: pgmap v6008: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:36:20.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:36:19 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:36:20.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:36:20 smithi115 conmon[90393]: 2022-05-14T02:36:20.132+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:36:21.126 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:36:20 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:36:21.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:36:20 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:36:21.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:36:21 smithi115 conmon[90393]: 2022-05-14T02:36:21.124+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:36:22.152 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:36:21 smithi115 ceph-mon[78593]: pgmap v6009: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:36:22.152 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:36:21 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 11316 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:36:22.152 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:36:21 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:36:22.153 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:36:21 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:36:22.153 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:36:21 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:36:22.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:36:21 smithi047 ceph-mon[77147]: pgmap v6009: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:36:22.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:36:21 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 11316 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:36:22.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:36:21 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:36:22.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:36:21 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:36:22.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:36:21 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:36:22.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:36:22 smithi115 conmon[90393]: 2022-05-14T02:36:22.150+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:36:23.137 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:36:22 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:36:23.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:36:22 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:36:23.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:36:23 smithi115 conmon[90393]: 2022-05-14T02:36:23.136+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:36:24.165 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:36:23 smithi115 ceph-mon[78593]: pgmap v6010: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:36:24.166 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:36:23 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:36:24.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:36:23 smithi047 ceph-mon[77147]: pgmap v6010: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:36:24.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:36:23 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:36:24.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:36:24 smithi115 conmon[90393]: 2022-05-14T02:36:24.164+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:36:25.210 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:36:24 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:36:25.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:36:24 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:36:25.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:36:25 smithi115 conmon[90393]: 2022-05-14T02:36:25.209+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:36:26.193 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:36:25 smithi115 ceph-mon[78593]: pgmap v6011: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:36:26.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:36:25 smithi047 ceph-mon[77147]: pgmap v6011: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:36:26.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:36:26 smithi115 conmon[90393]: 2022-05-14T02:36:26.191+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:36:27.173 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:36:26 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:36:27.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:36:26 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:36:27.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:36:27 smithi115 conmon[90393]: 2022-05-14T02:36:27.171+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:36:28.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:36:27 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:36:28.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:36:27 smithi047 ceph-mon[77147]: pgmap v6012: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:36:28.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:36:27 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:36:28.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:36:28 smithi115 conmon[90393]: 2022-05-14T02:36:28.124+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:36:28.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:36:27 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:36:28.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:36:27 smithi115 ceph-mon[78593]: pgmap v6012: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:36:28.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:36:27 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:36:29.152 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:36:28 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:36:29.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:36:28 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:36:29.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:36:29 smithi115 conmon[90393]: 2022-05-14T02:36:29.150+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:36:30.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:36:29 smithi047 ceph-mon[77147]: pgmap v6013: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:36:30.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:36:29 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:36:30.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:36:29 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:36:30.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:36:29 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:36:30.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:36:30 smithi115 conmon[90393]: 2022-05-14T02:36:30.132+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:36:30.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:36:29 smithi115 ceph-mon[78593]: pgmap v6013: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:36:30.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:36:29 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:36:30.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:36:29 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:36:30.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:36:29 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:36:31.178 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:36:30 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:36:31.178 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:36:30 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 11320 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:36:31.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:36:30 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:36:31.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:36:30 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 11320 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:36:31.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:36:31 smithi115 conmon[90393]: 2022-05-14T02:36:31.176+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:36:32.167 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:36:31 smithi115 ceph-mon[78593]: pgmap v6014: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:36:32.168 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:36:31 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:36:32.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:36:31 smithi047 ceph-mon[77147]: pgmap v6014: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:36:32.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:36:31 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:36:32.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:36:32 smithi115 conmon[90393]: 2022-05-14T02:36:32.166+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:36:33.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:36:32 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:36:33.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:36:33 smithi115 conmon[90393]: 2022-05-14T02:36:33.127+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:36:33.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:36:32 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:36:34.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:36:33 smithi047 ceph-mon[77147]: pgmap v6015: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:36:34.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:36:33 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:36:34.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:36:33 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:36:34.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:36:33 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:36:34.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:36:34 smithi115 conmon[90393]: 2022-05-14T02:36:34.121+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:36:34.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:36:33 smithi115 ceph-mon[78593]: pgmap v6015: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:36:34.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:36:33 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:36:34.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:36:33 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:36:34.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:36:33 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:36:35.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:36:34 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:36:35.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:36:35 smithi115 conmon[90393]: 2022-05-14T02:36:35.155+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:36:35.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:36:34 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:36:36.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:36:35 smithi047 ceph-mon[77147]: pgmap v6016: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:36:36.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:36:35 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:36:36.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:36:35 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 11331 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:36:36.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:36:36 smithi115 conmon[90393]: 2022-05-14T02:36:36.135+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:36:36.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:36:35 smithi115 ceph-mon[78593]: pgmap v6016: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:36:36.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:36:35 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:36:36.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:36:35 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 11331 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:36:37.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:36:36 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:36:37.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:36:37 smithi115 conmon[90393]: 2022-05-14T02:36:37.096+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:36:37.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:36:36 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:36:38.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:36:37 smithi047 ceph-mon[77147]: pgmap v6017: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:36:38.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:36:37 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:36:38.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:36:37 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:36:38.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:36:37 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:36:38.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:36:38 smithi115 conmon[90393]: 2022-05-14T02:36:38.133+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:36:38.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:36:37 smithi115 ceph-mon[78593]: pgmap v6017: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:36:38.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:36:37 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:36:38.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:36:37 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:36:38.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:36:37 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:36:39.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:36:38 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:36:39.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:36:38 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:36:39.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:36:39 smithi115 conmon[90393]: 2022-05-14T02:36:39.084+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:36:40.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:36:39 smithi047 ceph-mon[77147]: pgmap v6018: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:36:40.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:36:39 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:36:40.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:36:40 smithi115 conmon[90393]: 2022-05-14T02:36:40.113+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:36:40.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:36:39 smithi115 ceph-mon[78593]: pgmap v6018: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:36:40.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:36:39 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:36:41.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:36:40 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:36:41.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:36:41 smithi115 conmon[90393]: 2022-05-14T02:36:41.075+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:36:41.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:36:40 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:36:42.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:36:41 smithi047 ceph-mon[77147]: pgmap v6019: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:36:42.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:36:41 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 11335 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:36:42.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:36:41 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:36:42.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:36:41 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:36:42.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:36:41 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:36:42.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:36:42 smithi115 conmon[90393]: 2022-05-14T02:36:42.073+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:36:42.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:36:41 smithi115 ceph-mon[78593]: pgmap v6019: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:36:42.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:36:41 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 11335 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:36:42.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:36:41 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:36:42.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:36:41 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:36:42.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:36:41 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:36:43.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:36:42 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:36:43.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:36:43 smithi115 conmon[90393]: 2022-05-14T02:36:43.045+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:36:43.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:36:42 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:36:44.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:36:43 smithi047 ceph-mon[77147]: pgmap v6020: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:36:44.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:36:43 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:36:44.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:36:44 smithi115 conmon[90393]: 2022-05-14T02:36:44.053+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:36:44.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:36:43 smithi115 ceph-mon[78593]: pgmap v6020: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:36:44.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:36:43 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:36:45.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:36:44 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:36:45.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:36:44 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:36:45.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:36:45 smithi115 conmon[90393]: 2022-05-14T02:36:45.054+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:36:46.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:36:45 smithi047 ceph-mon[77147]: pgmap v6021: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:36:46.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:36:45 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:36:46.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:36:46 smithi115 conmon[90393]: 2022-05-14T02:36:46.100+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:36:46.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:36:45 smithi115 ceph-mon[78593]: pgmap v6021: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:36:46.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:36:45 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:36:47.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:36:46 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:36:47.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:36:47 smithi115 conmon[90393]: 2022-05-14T02:36:47.085+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:36:47.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:36:46 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:36:48.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:36:47 smithi047 ceph-mon[77147]: pgmap v6022: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:36:48.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:36:47 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:36:48.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:36:48 smithi115 conmon[90393]: 2022-05-14T02:36:48.128+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:36:48.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:36:47 smithi115 ceph-mon[78593]: pgmap v6022: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:36:48.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:36:47 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:36:49.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:36:48 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:36:49.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:36:49 smithi115 conmon[90393]: 2022-05-14T02:36:49.085+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:36:49.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:36:48 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:36:50.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:36:49 smithi047 ceph-mon[77147]: pgmap v6023: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:36:50.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:36:49 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:36:50.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:36:49 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:36:50.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:36:49 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:36:50.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:36:50 smithi115 conmon[90393]: 2022-05-14T02:36:50.086+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:36:50.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:36:49 smithi115 ceph-mon[78593]: pgmap v6023: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:36:50.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:36:49 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:36:50.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:36:49 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:36:50.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:36:49 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:36:51.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:36:50 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:36:51.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:36:50 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 11340 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:36:51.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:36:51 smithi115 conmon[90393]: 2022-05-14T02:36:51.109+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:36:51.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:36:50 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:36:51.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:36:50 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 11340 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:36:52.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:36:51 smithi047 ceph-mon[77147]: pgmap v6024: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:36:52.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:36:51 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:36:52.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:36:52 smithi115 conmon[90393]: 2022-05-14T02:36:52.142+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:36:52.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:36:51 smithi115 ceph-mon[78593]: pgmap v6024: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:36:52.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:36:51 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:36:53.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:36:52 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:36:53.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:36:53 smithi115 conmon[90393]: 2022-05-14T02:36:53.125+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:36:53.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:36:52 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:36:54.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:36:53 smithi047 ceph-mon[77147]: pgmap v6025: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:36:54.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:36:53 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:36:54.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:36:53 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:36:54.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:36:53 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:36:54.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:36:54 smithi115 conmon[90393]: 2022-05-14T02:36:54.129+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:36:54.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:36:53 smithi115 ceph-mon[78593]: pgmap v6025: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:36:54.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:36:53 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:36:54.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:36:53 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:36:54.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:36:53 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:36:55.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:36:54 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:36:55.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:36:55 smithi115 conmon[90393]: 2022-05-14T02:36:55.133+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:36:55.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:36:54 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:36:56.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:36:55 smithi047 ceph-mon[77147]: pgmap v6026: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:36:56.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:36:55 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:36:56.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:36:55 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 11351 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:36:56.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:36:56 smithi115 conmon[90393]: 2022-05-14T02:36:56.151+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:36:56.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:36:55 smithi115 ceph-mon[78593]: pgmap v6026: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:36:56.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:36:55 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:36:56.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:36:55 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 11351 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:36:57.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:36:56 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:36:57.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:36:57 smithi115 conmon[90393]: 2022-05-14T02:36:57.182+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:36:57.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:36:56 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:36:58.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:36:57 smithi047 ceph-mon[77147]: pgmap v6027: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:36:58.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:36:57 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:36:58.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:36:57 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:36:58.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:36:57 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:36:58.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:36:58 smithi115 conmon[90393]: 2022-05-14T02:36:58.209+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:36:58.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:36:57 smithi115 ceph-mon[78593]: pgmap v6027: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:36:58.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:36:57 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:36:58.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:36:57 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:36:58.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:36:57 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:36:59.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:36:59 smithi115 conmon[90393]: 2022-05-14T02:36:59.248+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:37:00.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:36:59 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:37:00.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:36:59 smithi047 ceph-mon[77147]: pgmap v6028: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:37:00.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:37:00 smithi115 conmon[90393]: 2022-05-14T02:37:00.212+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:37:00.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:36:59 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:37:00.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:36:59 smithi115 ceph-mon[78593]: pgmap v6028: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:37:01.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:37:00 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:37:01.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:37:01 smithi115 conmon[90393]: 2022-05-14T02:37:01.189+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:37:01.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:37:00 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:37:02.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:37:01 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:37:02.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:37:01 smithi047 ceph-mon[77147]: pgmap v6029: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:37:02.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:37:01 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 11355 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:37:02.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:37:01 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:37:02.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:37:01 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:37:02.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:37:02 smithi115 conmon[90393]: 2022-05-14T02:37:02.188+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:37:02.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:37:01 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:37:02.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:37:01 smithi115 ceph-mon[78593]: pgmap v6029: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:37:02.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:37:01 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 11355 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:37:02.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:37:01 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:37:02.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:37:01 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:37:03.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:37:03 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:37:03.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:37:03 smithi115 conmon[90393]: 2022-05-14T02:37:03.171+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:37:03.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:37:03 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:37:04.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:37:04 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:37:04.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:37:04 smithi047 ceph-mon[77147]: pgmap v6030: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:37:04.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:37:04 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:37:04.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:37:04 smithi115 conmon[90393]: 2022-05-14T02:37:04.156+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:37:04.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:37:04 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:37:04.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:37:04 smithi115 ceph-mon[78593]: pgmap v6030: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:37:04.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:37:04 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:37:05.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:37:05 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:37:05.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:37:05 smithi115 conmon[90393]: 2022-05-14T02:37:05.136+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:37:05.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:37:05 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:37:06.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:37:06 smithi047 ceph-mon[77147]: pgmap v6031: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:37:06.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:37:06 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:37:06.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:37:06 smithi115 conmon[90393]: 2022-05-14T02:37:06.175+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:37:06.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:37:06 smithi115 ceph-mon[78593]: pgmap v6031: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:37:06.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:37:06 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:37:07.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:37:07 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:37:07.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:37:07 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:37:07.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:37:07 smithi115 conmon[90393]: 2022-05-14T02:37:07.202+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:37:08.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:37:08 smithi047 ceph-mon[77147]: pgmap v6032: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:37:08.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:37:08 smithi115 conmon[90393]: 2022-05-14T02:37:08.185+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:37:08.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:37:08 smithi115 ceph-mon[78593]: pgmap v6032: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:37:09.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:37:09 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:37:09.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:37:09 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:37:09.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:37:09 smithi115 conmon[90393]: 2022-05-14T02:37:09.218+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:37:09.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:37:09 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:37:09.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:37:09 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:37:10.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:37:10 smithi047 ceph-mon[77147]: pgmap v6033: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:37:10.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:37:10 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:37:10.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:37:10 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:37:10.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:37:10 smithi115 conmon[90393]: 2022-05-14T02:37:10.193+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:37:10.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:37:10 smithi115 ceph-mon[78593]: pgmap v6033: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:37:10.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:37:10 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:37:10.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:37:10 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:37:11.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:37:11 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:37:11.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:37:11 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 11360 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:37:11.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:37:11 smithi115 conmon[90393]: 2022-05-14T02:37:11.230+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:37:11.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:37:11 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:37:11.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:37:11 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 11360 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:37:12.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:37:12 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:37:12.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:37:12 smithi047 ceph-mon[77147]: pgmap v6034: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:37:12.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:37:12 smithi115 conmon[90393]: 2022-05-14T02:37:12.225+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:37:12.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:37:12 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:37:12.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:37:12 smithi115 ceph-mon[78593]: pgmap v6034: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:37:13.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:37:13 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:37:13.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:37:13 smithi115 conmon[90393]: 2022-05-14T02:37:13.259+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:37:13.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:37:13 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:37:14.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:37:14 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:37:14.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:37:14 smithi047 ceph-mon[77147]: pgmap v6035: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:37:14.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:37:14 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:37:14.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:37:14 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:37:14.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:37:14 smithi115 conmon[90393]: 2022-05-14T02:37:14.249+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:37:14.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:37:14 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:37:14.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:37:14 smithi115 ceph-mon[78593]: pgmap v6035: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:37:14.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:37:14 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:37:14.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:37:14 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:37:15.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:37:15 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:37:15.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:37:15 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T02:37:15.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:37:15 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T02:37:15.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:37:15 smithi115 conmon[90393]: 2022-05-14T02:37:15.278+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:37:15.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:37:15 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:37:15.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:37:15 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T02:37:15.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:37:15 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T02:37:16.301 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:37:16 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:37:16.420 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:37:16 smithi115 ceph-mon[78593]: pgmap v6036: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:37:16.421 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:37:16 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T02:37:16.421 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:37:16 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T02:37:16.421 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:37:16 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T02:37:16.422 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:37:16 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 11370 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:37:16.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:37:16 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:37:16.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:37:16 smithi047 ceph-mon[77147]: pgmap v6036: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:37:16.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:37:16 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T02:37:16.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:37:16 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T02:37:16.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:37:16 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T02:37:16.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:37:16 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 11370 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:37:16.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:37:16 smithi115 conmon[90393]: 2022-05-14T02:37:16.300+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:37:17.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:37:17 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:37:17.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:37:17 smithi115 conmon[90393]: 2022-05-14T02:37:17.259+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:37:17.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:37:17 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:37:18.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:37:18 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:37:18.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:37:18 smithi115 ceph-mon[78593]: pgmap v6037: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:37:18.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:37:18 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:37:18.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:37:18 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:37:18.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:37:18 smithi115 conmon[90393]: 2022-05-14T02:37:18.228+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:37:18.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:37:18 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:37:18.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:37:18 smithi047 ceph-mon[77147]: pgmap v6037: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:37:18.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:37:18 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:37:18.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:37:18 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:37:18.542 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:37:18 smithi047 conmon[78521]: 2022-05-14T02:37:18.255+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:37:18.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:37:18 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T02:37:18.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:37:18 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:37:18.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:37:18 smithi047 conmon[78521]: yield 2022-05-14T02:37:18.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:37:18 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:37:18.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:37:18 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:37:18.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:37:18 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:37:18.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:37:18 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T02:37:18.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:37:18 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:37:18.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:37:18 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T02:37:18.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:37:18 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:37:18.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:37:18 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T02:37:18.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:37:18 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:37:18.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:37:18 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T02:37:18.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:37:18 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:37:18.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:37:18 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T02:37:18.547 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:37:18 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:37:18.547 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:37:18 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:37:18.547 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:37:18 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:37:19.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:37:19 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:37:19.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:37:19 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:37:19.358 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T02:37:19.359 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:37:19.359 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T02:37:19.359 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:37:19.359 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:37:19.359 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:37:19.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T02:37:19.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:37:19.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T02:37:19.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:37:19.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T02:37:19.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:37:19.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T02:37:19.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:37:19.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T02:37:19.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:37:19.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:37:19.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:37:19.362 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:37:19 smithi115 ceph-mon[78593]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T02:37:19.363 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:37:19 smithi115 conmon[90393]: 2022-05-14T02:37:19.249+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:37:19.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:37:19 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:37:19.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:37:19 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:37:19.542 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T02:37:19.542 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:37:19.542 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T02:37:19.542 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:37:19.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:37:19.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:37:19.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T02:37:19.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:37:19.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T02:37:19.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:37:19.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T02:37:19.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:37:19.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T02:37:19.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:37:19.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T02:37:19.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:37:19.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:37:19.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:37:19.546 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:37:19 smithi047 ceph-mon[77147]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T02:37:20.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:37:20 smithi115 conmon[90393]: 2022-05-14T02:37:20.211+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:37:20.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:37:20 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:37:20.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:37:20 smithi115 ceph-mon[78593]: pgmap v6038: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:37:20.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:37:20 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:37:20.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:37:20 smithi047 ceph-mon[77147]: pgmap v6038: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:37:21.329 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:37:21 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:37:21.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:37:21 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:37:21.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:37:21 smithi115 conmon[90393]: 2022-05-14T02:37:21.198+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:37:21.792 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:37:21 smithi047 conmon[78521]: 2022-05-14T02:37:21.327+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:37:21.792 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:37:21 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T02:37:21.792 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:37:21 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:37:21.793 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:37:21 smithi047 conmon[78521]: yield 2022-05-14T02:37:21.793 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:37:21 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:37:21.793 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:37:21 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:37:21.793 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:37:21 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:37:21.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:37:21 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T02:37:21.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:37:21 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:37:21.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:37:21 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T02:37:21.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:37:21 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:37:21.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:37:21 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T02:37:21.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:37:21 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:37:21.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:37:21 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T02:37:21.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:37:21 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:37:21.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:37:21 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T02:37:21.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:37:21 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:37:21.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:37:21 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:37:21.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:37:21 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:37:21.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:37:21 smithi047 conmon[78521]: 2022-05-14T02:37:21.328+0000 7f5340779700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T02:37:21.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:37:21 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T02:37:21.797 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:37:21 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:37:21.797 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:37:21 smithi047 conmon[78521]: yield 2022-05-14T02:37:21.797 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:37:21 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:37:21.797 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:37:21 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:37:21.798 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:37:21 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:37:21.798 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:37:21 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T02:37:21.798 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:37:21 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:37:21.798 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:37:21 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T02:37:21.798 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:37:21 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:37:21.799 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:37:21 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T02:37:21.799 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:37:21 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:37:21.799 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:37:21 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T02:37:21.799 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:37:21 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:37:21.800 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:37:21 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T02:37:21.800 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:37:21 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:37:21.800 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:37:21 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:37:21.800 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:37:21 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:37:21.800 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:37:21 smithi047 conmon[78521]: 2022-05-14T02:37:21.801 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:37:21 smithi047 conmon[78521]: During handling of the above exception, another exception occurred: 2022-05-14T02:37:21.801 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:37:21 smithi047 conmon[78521]: 2022-05-14T02:37:21.801 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:37:21 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T02:37:21.801 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:37:21 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T02:37:21.802 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:37:21 smithi047 conmon[78521]: return f(*arg) 2022-05-14T02:37:21.802 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:37:21 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T02:37:21.802 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:37:21 smithi047 conmon[78521]: self._write_client_files(client_files, host) 2022-05-14T02:37:21.802 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:37:21 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T02:37:21.803 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:37:21 smithi047 conmon[78521]: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T02:37:21.803 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:37:21 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T02:37:21.803 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:37:21 smithi047 conmon[78521]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T02:37:21.803 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:37:21 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T02:37:21.803 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:37:21 smithi047 conmon[78521]: return self.event_loop.get_result(coro) 2022-05-14T02:37:21.804 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:37:21 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T02:37:21.804 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:37:21 smithi047 conmon[78521]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T02:37:21.804 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:37:21 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T02:37:21.804 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:37:21 smithi047 conmon[78521]: return self.__get_result() 2022-05-14T02:37:21.804 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:37:21 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T02:37:21.805 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:37:21 smithi047 conmon[78521]: raise self._exception 2022-05-14T02:37:21.805 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:37:21 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T02:37:21.805 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:37:21 smithi047 conmon[78521]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T02:37:21.805 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:37:21 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T02:37:21.806 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:37:21 smithi047 conmon[78521]: conn = await self._remote_connection(host, addr) 2022-05-14T02:37:21.806 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:37:21 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T02:37:21.806 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:37:21 smithi047 conmon[78521]: raise 2022-05-14T02:37:21.806 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:37:21 smithi047 conmon[78521]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T02:37:21.806 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:37:21 smithi047 conmon[78521]: self.gen.throw(type, value, traceback) 2022-05-14T02:37:21.807 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:37:21 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T02:37:21.807 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:37:21 smithi047 conmon[78521]: raise OrchestratorError(msg) 2022-05-14T02:37:21.807 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:37:21 smithi047 conmon[78521]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:37:22.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:37:22 smithi115 conmon[90393]: 2022-05-14T02:37:22.213+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:37:22.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:37:22 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:37:22.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:37:22 smithi115 ceph-mon[78593]: pgmap v6039: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:37:22.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:37:22 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 11375 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:37:22.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:37:22 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:37:22.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T02:37:22.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:37:22.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T02:37:22.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:37:22.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:37:22.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:37:22.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T02:37:22.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:37:22.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T02:37:22.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:37:22.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T02:37:22.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:37:22.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T02:37:22.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:37:22.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T02:37:22.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:37:22.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:37:22.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:37:22.364 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:37:22 smithi115 ceph-mon[78593]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T02:37:22.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T02:37:22.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:37:22.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T02:37:22.365 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:37:22.365 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:37:22.365 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:37:22.365 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T02:37:22.366 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:37:22.366 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T02:37:22.366 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:37:22.366 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T02:37:22.366 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:37:22.367 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T02:37:22.367 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:37:22.367 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T02:37:22.367 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:37:22.367 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:37:22.368 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:37:22.368 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T02:37:22.368 INFO:journalctl@ceph.mon.c.smithi115.stdout: During handling of the above exception, another exception occurred: 2022-05-14T02:37:22.368 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T02:37:22.369 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T02:37:22.369 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T02:37:22.369 INFO:journalctl@ceph.mon.c.smithi115.stdout: return f(*arg) 2022-05-14T02:37:22.369 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T02:37:22.369 INFO:journalctl@ceph.mon.c.smithi115.stdout: self._write_client_files(client_files, host) 2022-05-14T02:37:22.370 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T02:37:22.370 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T02:37:22.370 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T02:37:22.370 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T02:37:22.371 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T02:37:22.371 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.event_loop.get_result(coro) 2022-05-14T02:37:22.371 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T02:37:22.371 INFO:journalctl@ceph.mon.c.smithi115.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T02:37:22.371 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T02:37:22.372 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.__get_result() 2022-05-14T02:37:22.372 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T02:37:22.372 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise self._exception 2022-05-14T02:37:22.372 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T02:37:22.372 INFO:journalctl@ceph.mon.c.smithi115.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T02:37:22.373 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T02:37:22.373 INFO:journalctl@ceph.mon.c.smithi115.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T02:37:22.373 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T02:37:22.373 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise 2022-05-14T02:37:22.373 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T02:37:22.374 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.gen.throw(type, value, traceback) 2022-05-14T02:37:22.374 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T02:37:22.374 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OrchestratorError(msg) 2022-05-14T02:37:22.374 INFO:journalctl@ceph.mon.c.smithi115.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:37:22.375 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:37:22 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:37:22.375 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:37:22 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:37:22.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:37:22 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:37:22.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:37:22 smithi047 ceph-mon[77147]: pgmap v6039: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:37:22.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:37:22 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 11375 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:37:22.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:37:22 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:37:22.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T02:37:22.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:37:22.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T02:37:22.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:37:22.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:37:22.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:37:22.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T02:37:22.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:37:22.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T02:37:22.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:37:22.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T02:37:22.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:37:22.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T02:37:22.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:37:22.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T02:37:22.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:37:22.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:37:22.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:37:22.547 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:37:22 smithi047 ceph-mon[77147]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T02:37:22.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T02:37:22.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:37:22.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T02:37:22.548 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:37:22.548 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:37:22.548 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:37:22.548 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T02:37:22.548 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:37:22.549 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T02:37:22.549 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:37:22.549 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T02:37:22.549 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:37:22.550 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T02:37:22.550 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:37:22.550 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T02:37:22.550 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:37:22.550 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:37:22.551 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:37:22.551 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T02:37:22.551 INFO:journalctl@ceph.mon.b.smithi047.stdout: During handling of the above exception, another exception occurred: 2022-05-14T02:37:22.551 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T02:37:22.551 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T02:37:22.552 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T02:37:22.552 INFO:journalctl@ceph.mon.b.smithi047.stdout: return f(*arg) 2022-05-14T02:37:22.552 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T02:37:22.552 INFO:journalctl@ceph.mon.b.smithi047.stdout: self._write_client_files(client_files, host) 2022-05-14T02:37:22.553 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T02:37:22.553 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T02:37:22.553 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T02:37:22.553 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T02:37:22.553 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T02:37:22.554 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.event_loop.get_result(coro) 2022-05-14T02:37:22.554 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T02:37:22.554 INFO:journalctl@ceph.mon.b.smithi047.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T02:37:22.554 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T02:37:22.555 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.__get_result() 2022-05-14T02:37:22.555 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T02:37:22.555 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise self._exception 2022-05-14T02:37:22.555 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T02:37:22.555 INFO:journalctl@ceph.mon.b.smithi047.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T02:37:22.556 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T02:37:22.556 INFO:journalctl@ceph.mon.b.smithi047.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T02:37:22.556 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T02:37:22.556 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise 2022-05-14T02:37:22.556 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T02:37:22.557 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.gen.throw(type, value, traceback) 2022-05-14T02:37:22.557 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T02:37:22.557 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OrchestratorError(msg) 2022-05-14T02:37:22.557 INFO:journalctl@ceph.mon.b.smithi047.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:37:22.558 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:37:22 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:37:22.558 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:37:22 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:37:23.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:37:23 smithi115 conmon[90393]: 2022-05-14T02:37:23.194+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:37:23.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:37:23 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:37:23.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:37:23 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:37:24.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:37:24 smithi115 conmon[90393]: 2022-05-14T02:37:24.228+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:37:24.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:37:24 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:37:24.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:37:24 smithi115 ceph-mon[78593]: pgmap v6040: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:37:24.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:37:24 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:37:24.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:37:24 smithi047 ceph-mon[77147]: pgmap v6040: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:37:25.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:37:25 smithi115 conmon[90393]: 2022-05-14T02:37:25.276+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:37:25.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:37:25 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:37:25.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:37:25 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:37:26.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:37:26 smithi115 conmon[90393]: 2022-05-14T02:37:26.287+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:37:26.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:37:26 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:37:26.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:37:26 smithi115 ceph-mon[78593]: pgmap v6041: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:37:26.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:37:26 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:37:26.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:37:26 smithi047 ceph-mon[77147]: pgmap v6041: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:37:27.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:37:27 smithi115 conmon[90393]: 2022-05-14T02:37:27.302+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:37:27.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:37:27 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:37:27.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:37:27 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:37:28.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:37:28 smithi115 conmon[90393]: 2022-05-14T02:37:28.298+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:37:28.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:37:28 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:37:28.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:37:28 smithi115 ceph-mon[78593]: pgmap v6042: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:37:28.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:37:28 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:37:28.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:37:28 smithi047 ceph-mon[77147]: pgmap v6042: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:37:29.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:37:29 smithi115 conmon[90393]: 2022-05-14T02:37:29.280+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:37:29.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:37:29 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:37:29.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:37:29 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:37:30.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:37:30 smithi115 conmon[90393]: 2022-05-14T02:37:30.239+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:37:30.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:37:30 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:37:30.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:37:30 smithi115 ceph-mon[78593]: pgmap v6043: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:37:30.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:37:30 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:37:30.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:37:30 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:37:30.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:37:30 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:37:30.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:37:30 smithi047 ceph-mon[77147]: pgmap v6043: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:37:30.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:37:30 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:37:30.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:37:30 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:37:31.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:37:31 smithi115 conmon[90393]: 2022-05-14T02:37:31.263+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:37:31.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:37:31 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:37:31.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:37:31 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 11380 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:37:31.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:37:31 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:37:31.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:37:31 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 11380 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:37:32.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:37:32 smithi115 conmon[90393]: 2022-05-14T02:37:32.269+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:37:32.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:37:32 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:37:32.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:37:32 smithi115 ceph-mon[78593]: pgmap v6044: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:37:32.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:37:32 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:37:32.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:37:32 smithi047 ceph-mon[77147]: pgmap v6044: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:37:33.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:37:33 smithi115 conmon[90393]: 2022-05-14T02:37:33.289+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:37:33.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:37:33 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:37:33.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:37:33 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:37:34.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:37:34 smithi115 conmon[90393]: 2022-05-14T02:37:34.243+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:37:34.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:37:34 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:37:34.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:37:34 smithi115 ceph-mon[78593]: pgmap v6045: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:37:34.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:37:34 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:37:34.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:37:34 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:37:34.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:37:34 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:37:34.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:37:34 smithi047 ceph-mon[77147]: pgmap v6045: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:37:34.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:37:34 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:37:34.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:37:34 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:37:35.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:37:35 smithi115 conmon[90393]: 2022-05-14T02:37:35.277+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:37:35.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:37:35 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:37:35.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:37:35 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:37:36.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:37:36 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:37:36.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:37:36 smithi047 ceph-mon[77147]: pgmap v6046: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:37:36.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:37:36 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 11390 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:37:36.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:37:36 smithi115 conmon[90393]: 2022-05-14T02:37:36.287+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:37:36.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:37:36 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:37:36.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:37:36 smithi115 ceph-mon[78593]: pgmap v6046: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:37:36.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:37:36 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 11390 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:37:37.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:37:37 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:37:37.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:37:37 smithi115 conmon[90393]: 2022-05-14T02:37:37.261+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:37:37.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:37:37 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:37:38.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:37:38 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:37:38.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:37:38 smithi047 ceph-mon[77147]: pgmap v6047: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:37:38.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:37:38 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:37:38.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:37:38 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:37:38.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:37:38 smithi115 conmon[90393]: 2022-05-14T02:37:38.214+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:37:38.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:37:38 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:37:38.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:37:38 smithi115 ceph-mon[78593]: pgmap v6047: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:37:38.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:37:38 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:37:38.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:37:38 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:37:39.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:37:39 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:37:39.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:37:39 smithi115 conmon[90393]: 2022-05-14T02:37:39.226+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:37:39.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:37:39 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:37:40.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:37:40 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:37:40.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:37:40 smithi047 ceph-mon[77147]: pgmap v6048: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:37:40.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:37:40 smithi115 conmon[90393]: 2022-05-14T02:37:40.192+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:37:40.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:37:40 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:37:40.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:37:40 smithi115 ceph-mon[78593]: pgmap v6048: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:37:41.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:37:41 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:37:41.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:37:41 smithi115 conmon[90393]: 2022-05-14T02:37:41.210+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:37:41.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:37:41 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:37:42.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:37:42 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:37:42.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:37:42 smithi047 ceph-mon[77147]: pgmap v6049: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:37:42.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:37:42 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 11395 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:37:42.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:37:42 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:37:42.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:37:42 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:37:42.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:37:42 smithi115 conmon[90393]: 2022-05-14T02:37:42.216+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:37:42.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:37:42 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:37:42.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:37:42 smithi115 ceph-mon[78593]: pgmap v6049: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:37:42.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:37:42 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 11395 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:37:42.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:37:42 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:37:42.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:37:42 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:37:43.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:37:43 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:37:43.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:37:43 smithi115 conmon[90393]: 2022-05-14T02:37:43.192+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:37:43.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:37:43 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:37:44.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:37:44 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:37:44.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:37:44 smithi047 ceph-mon[77147]: pgmap v6050: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:37:44.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:37:44 smithi115 conmon[90393]: 2022-05-14T02:37:44.199+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:37:44.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:37:44 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:37:44.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:37:44 smithi115 ceph-mon[78593]: pgmap v6050: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:37:45.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:37:45 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:37:45.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:37:45 smithi115 conmon[90393]: 2022-05-14T02:37:45.209+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:37:45.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:37:45 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:37:46.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:37:46 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:37:46.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:37:46 smithi047 ceph-mon[77147]: pgmap v6051: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:37:46.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:37:46 smithi115 conmon[90393]: 2022-05-14T02:37:46.171+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:37:46.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:37:46 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:37:46.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:37:46 smithi115 ceph-mon[78593]: pgmap v6051: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:37:47.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:37:47 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:37:47.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:37:47 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:37:47.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:37:47 smithi115 conmon[90393]: 2022-05-14T02:37:47.193+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:37:47.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:37:47 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:37:47.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:37:47 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:37:48.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:37:48 smithi047 ceph-mon[77147]: pgmap v6052: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:37:48.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:37:48 smithi115 conmon[90393]: 2022-05-14T02:37:48.196+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:37:48.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:37:48 smithi115 ceph-mon[78593]: pgmap v6052: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:37:49.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:37:49 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:37:49.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:37:49 smithi115 conmon[90393]: 2022-05-14T02:37:49.228+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:37:49.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:37:49 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:37:50.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:37:50 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:37:50.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:37:50 smithi047 ceph-mon[77147]: pgmap v6053: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:37:50.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:37:50 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:37:50.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:37:50 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:37:50.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:37:50 smithi115 conmon[90393]: 2022-05-14T02:37:50.259+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:37:50.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:37:50 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:37:50.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:37:50 smithi115 ceph-mon[78593]: pgmap v6053: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:37:50.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:37:50 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:37:50.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:37:50 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:37:51.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:37:51 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:37:51.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:37:51 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 11400 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:37:51.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:37:51 smithi115 conmon[90393]: 2022-05-14T02:37:51.223+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:37:51.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:37:51 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:37:51.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:37:51 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 11400 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:37:52.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:37:52 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:37:52.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:37:52 smithi047 ceph-mon[77147]: pgmap v6054: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:37:52.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:37:52 smithi115 conmon[90393]: 2022-05-14T02:37:52.258+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:37:52.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:37:52 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:37:52.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:37:52 smithi115 ceph-mon[78593]: pgmap v6054: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:37:53.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:37:53 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:37:53.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:37:53 smithi115 conmon[90393]: 2022-05-14T02:37:53.301+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:37:53.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:37:53 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:37:54.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:37:54 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:37:54.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:37:54 smithi047 ceph-mon[77147]: pgmap v6055: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:37:54.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:37:54 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:37:54.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:37:54 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:37:54.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:37:54 smithi115 conmon[90393]: 2022-05-14T02:37:54.301+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:37:54.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:37:54 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:37:54.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:37:54 smithi115 ceph-mon[78593]: pgmap v6055: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:37:54.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:37:54 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:37:54.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:37:54 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:37:55.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:37:55 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:37:55.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:37:55 smithi115 conmon[90393]: 2022-05-14T02:37:55.298+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:37:55.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:37:55 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:37:56.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:37:56 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:37:56.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:37:56 smithi047 ceph-mon[77147]: pgmap v6056: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:37:56.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:37:56 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 11410 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:37:56.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:37:56 smithi115 conmon[90393]: 2022-05-14T02:37:56.296+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:37:56.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:37:56 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:37:56.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:37:56 smithi115 ceph-mon[78593]: pgmap v6056: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:37:56.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:37:56 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 11410 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:37:57.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:37:57 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:37:57.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:37:57 smithi115 conmon[90393]: 2022-05-14T02:37:57.269+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:37:57.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:37:57 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:37:58.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:37:58 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:37:58.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:37:58 smithi047 ceph-mon[77147]: pgmap v6057: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:37:58.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:37:58 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:37:58.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:37:58 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:37:58.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:37:58 smithi115 conmon[90393]: 2022-05-14T02:37:58.253+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:37:58.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:37:58 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:37:58.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:37:58 smithi115 ceph-mon[78593]: pgmap v6057: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:37:58.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:37:58 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:37:58.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:37:58 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:37:59.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:37:59 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:37:59.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:37:59 smithi115 conmon[90393]: 2022-05-14T02:37:59.256+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:37:59.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:37:59 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:38:00.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:38:00 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:38:00.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:38:00 smithi047 ceph-mon[77147]: pgmap v6058: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:38:00.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:38:00 smithi115 conmon[90393]: 2022-05-14T02:38:00.212+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:38:00.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:38:00 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:38:00.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:38:00 smithi115 ceph-mon[78593]: pgmap v6058: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:38:01.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:38:01 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:38:01.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:38:01 smithi115 conmon[90393]: 2022-05-14T02:38:01.252+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:38:01.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:38:01 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:38:02.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:38:02 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:38:02.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:38:02 smithi047 ceph-mon[77147]: pgmap v6059: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:38:02.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:38:02 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 11415 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:38:02.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:38:02 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:38:02.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:38:02 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:38:02.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:38:02 smithi115 conmon[90393]: 2022-05-14T02:38:02.271+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:38:02.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:38:02 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:38:02.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:38:02 smithi115 ceph-mon[78593]: pgmap v6059: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:38:02.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:38:02 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 11415 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:38:02.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:38:02 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:38:02.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:38:02 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:38:03.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:38:03 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:38:03.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:38:03 smithi115 conmon[90393]: 2022-05-14T02:38:03.256+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:38:03.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:38:03 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:38:04.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:38:04 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:38:04.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:38:04 smithi047 ceph-mon[77147]: pgmap v6060: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:38:04.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:38:04 smithi115 conmon[90393]: 2022-05-14T02:38:04.240+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:38:04.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:38:04 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:38:04.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:38:04 smithi115 ceph-mon[78593]: pgmap v6060: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:38:05.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:38:05 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:38:05.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:38:05 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:38:05.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:38:05 smithi115 conmon[90393]: 2022-05-14T02:38:05.239+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:38:05.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:38:05 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:38:05.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:38:05 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:38:06.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:38:06 smithi047 ceph-mon[77147]: pgmap v6061: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:38:06.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:38:06 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:38:06.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:38:06 smithi115 conmon[90393]: 2022-05-14T02:38:06.254+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:38:06.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:38:06 smithi115 ceph-mon[78593]: pgmap v6061: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:38:06.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:38:06 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:38:07.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:38:07 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:38:07.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:38:07 smithi115 conmon[90393]: 2022-05-14T02:38:07.249+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:38:07.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:38:07 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:38:08.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:38:08 smithi047 ceph-mon[77147]: pgmap v6062: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:38:08.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:38:08 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:38:08.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:38:08 smithi115 conmon[90393]: 2022-05-14T02:38:08.288+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:38:08.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:38:08 smithi115 ceph-mon[78593]: pgmap v6062: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:38:08.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:38:08 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:38:09.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:38:09 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:38:09.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:38:09 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:38:09.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:38:09 smithi115 conmon[90393]: 2022-05-14T02:38:09.256+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:38:10.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:38:10 smithi047 ceph-mon[77147]: pgmap v6063: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:38:10.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:38:10 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:38:10.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:38:10 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:38:10.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:38:10 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:38:10.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:38:10 smithi115 conmon[90393]: 2022-05-14T02:38:10.303+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:38:10.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:38:10 smithi115 ceph-mon[78593]: pgmap v6063: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:38:10.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:38:10 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:38:10.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:38:10 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:38:10.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:38:10 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:38:11.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:38:11 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:38:11.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:38:11 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 11420 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:38:11.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:38:11 smithi115 conmon[90393]: 2022-05-14T02:38:11.336+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:38:11.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:38:11 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:38:11.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:38:11 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 11420 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:38:12.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:38:12 smithi047 ceph-mon[77147]: pgmap v6064: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:38:12.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:38:12 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:38:12.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:38:12 smithi115 conmon[90393]: 2022-05-14T02:38:12.317+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:38:12.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:38:12 smithi115 ceph-mon[78593]: pgmap v6064: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:38:12.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:38:12 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:38:13.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:38:13 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:38:13.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:38:13 smithi115 conmon[90393]: 2022-05-14T02:38:13.343+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:38:13.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:38:13 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:38:14.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:38:14 smithi047 ceph-mon[77147]: pgmap v6065: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:38:14.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:38:14 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:38:14.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:38:14 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:38:14.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:38:14 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:38:14.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:38:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T02:38:14.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:38:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T02:38:14.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:38:14 smithi115 conmon[90393]: 2022-05-14T02:38:14.317+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:38:14.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:38:14 smithi115 ceph-mon[78593]: pgmap v6065: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:38:14.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:38:14 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:38:14.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:38:14 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:38:14.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:38:14 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:38:14.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:38:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T02:38:14.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:38:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T02:38:15.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:38:15 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:38:15.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:38:15 smithi115 conmon[90393]: 2022-05-14T02:38:15.335+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:38:15.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:38:15 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:38:16.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:38:16 smithi047 ceph-mon[77147]: pgmap v6066: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:38:16.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:38:16 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:38:16.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:38:16 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 11430 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:38:16.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:38:16 smithi115 conmon[90393]: 2022-05-14T02:38:16.288+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:38:16.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:38:16 smithi115 ceph-mon[78593]: pgmap v6066: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:38:16.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:38:16 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:38:16.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:38:16 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 11430 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:38:17.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:38:17 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:38:17.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:38:17 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:38:17.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:38:17 smithi115 conmon[90393]: 2022-05-14T02:38:17.275+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:38:18.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:38:18 smithi047 ceph-mon[77147]: pgmap v6067: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:38:18.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:38:18 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:38:18.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:38:18 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:38:18.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:38:18 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:38:18.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:38:18 smithi115 conmon[90393]: 2022-05-14T02:38:18.252+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:38:18.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:38:18 smithi115 ceph-mon[78593]: pgmap v6067: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:38:18.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:38:18 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:38:18.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:38:18 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:38:18.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:38:18 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:38:19.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:38:19 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:38:19.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:38:19 smithi115 conmon[90393]: 2022-05-14T02:38:19.286+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:38:19.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:38:19 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:38:20.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:38:20 smithi047 ceph-mon[77147]: pgmap v6068: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:38:20.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:38:20 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:38:20.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:38:20 smithi115 conmon[90393]: 2022-05-14T02:38:20.249+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:38:20.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:38:20 smithi115 ceph-mon[78593]: pgmap v6068: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:38:20.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:38:20 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:38:21.459 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:38:21 smithi115 conmon[90393]: 2022-05-14T02:38:21.201+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:38:22.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:38:21 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:38:22.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:38:21 smithi115 ceph-mon[78593]: pgmap v6069: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:38:22.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:38:21 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 11435 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:38:22.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:38:21 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T02:38:22.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:38:21 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T02:38:22.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:38:21 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T02:38:22.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:38:21 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:38:22.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:38:21 smithi047 ceph-mon[77147]: pgmap v6069: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:38:22.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:38:21 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 11435 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:38:22.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:38:21 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T02:38:22.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:38:21 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T02:38:22.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:38:21 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T02:38:22.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:38:22 smithi115 conmon[90393]: 2022-05-14T02:38:22.245+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:38:23.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:38:22 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:38:23.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:38:22 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:38:23.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:38:22 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:38:23.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:38:22 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:38:23.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:38:22 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:38:23.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:38:22 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:38:23.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:38:23 smithi115 conmon[90393]: 2022-05-14T02:38:23.196+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:38:24.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:38:23 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:38:24.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:38:23 smithi115 ceph-mon[78593]: pgmap v6070: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:38:24.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:38:23 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:38:24.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:38:23 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:38:24.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:38:23 smithi047 ceph-mon[77147]: pgmap v6070: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:38:24.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:38:23 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:38:24.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:38:24 smithi115 conmon[90393]: 2022-05-14T02:38:24.233+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:38:24.791 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:38:24 smithi047 conmon[78521]: 2022-05-14T02:38:24.432+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:38:24.792 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:38:24 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T02:38:24.792 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:38:24 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:38:24.793 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:38:24 smithi047 conmon[78521]: yield 2022-05-14T02:38:24.793 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:38:24 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:38:24.793 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:38:24 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:38:24.793 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:38:24 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:38:24.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:38:24 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T02:38:24.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:38:24 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:38:24.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:38:24 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T02:38:24.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:38:24 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:38:24.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:38:24 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T02:38:24.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:38:24 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:38:24.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:38:24 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T02:38:24.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:38:24 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:38:24.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:38:24 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T02:38:24.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:38:24 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:38:24.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:38:24 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:38:24.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:38:24 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:38:25.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:38:24 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:38:25.108 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T02:38:25.108 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:38:25.109 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T02:38:25.109 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:38:25.109 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:38:25.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:38:25.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T02:38:25.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:38:25.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T02:38:25.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:38:25.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T02:38:25.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:38:25.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T02:38:25.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:38:25.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T02:38:25.113 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:38:25.113 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:38:25.113 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:38:25.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:38:24 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:38:25.291 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T02:38:25.292 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:38:25.292 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T02:38:25.292 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:38:25.292 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:38:25.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:38:25.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T02:38:25.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:38:25.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T02:38:25.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:38:25.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T02:38:25.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:38:25.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T02:38:25.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:38:25.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T02:38:25.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:38:25.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:38:25.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:38:25.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:38:25 smithi115 conmon[90393]: 2022-05-14T02:38:25.202+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:38:26.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:38:25 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:38:26.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:38:25 smithi115 ceph-mon[78593]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T02:38:26.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:38:25 smithi115 ceph-mon[78593]: pgmap v6071: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:38:26.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:38:25 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:38:26.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:38:25 smithi047 ceph-mon[77147]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T02:38:26.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:38:25 smithi047 ceph-mon[77147]: pgmap v6071: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:38:26.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:38:26 smithi115 conmon[90393]: 2022-05-14T02:38:26.168+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:38:27.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:38:26 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:38:27.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:38:26 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:38:27.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:38:26 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:38:27.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:38:26 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:38:27.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:38:27 smithi115 conmon[90393]: 2022-05-14T02:38:27.159+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:38:27.792 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:38:27 smithi047 conmon[78521]: 2022-05-14T02:38:27.504+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:38:27.792 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:38:27 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T02:38:27.792 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:38:27 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:38:27.793 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:38:27 smithi047 conmon[78521]: yield 2022-05-14T02:38:27.793 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:38:27 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:38:27.793 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:38:27 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:38:27.793 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:38:27 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:38:27.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:38:27 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T02:38:27.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:38:27 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:38:27.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:38:27 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T02:38:27.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:38:27 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:38:27.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:38:27 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T02:38:27.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:38:27 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:38:27.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:38:27 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T02:38:27.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:38:27 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:38:27.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:38:27 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T02:38:27.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:38:27 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:38:27.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:38:27 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:38:27.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:38:27 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:38:27.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:38:27 smithi047 conmon[78521]: 2022-05-14T02:38:27.505+0000 7f534177b700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T02:38:27.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:38:27 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T02:38:27.797 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:38:27 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:38:27.797 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:38:27 smithi047 conmon[78521]: yield 2022-05-14T02:38:27.797 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:38:27 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:38:27.797 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:38:27 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:38:27.797 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:38:27 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:38:27.798 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:38:27 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T02:38:27.798 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:38:27 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:38:27.798 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:38:27 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T02:38:27.798 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:38:27 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:38:27.799 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:38:27 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T02:38:27.799 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:38:27 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:38:27.799 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:38:27 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T02:38:27.799 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:38:27 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:38:27.799 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:38:27 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T02:38:27.800 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:38:27 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:38:27.800 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:38:27 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:38:27.800 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:38:27 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:38:27.800 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:38:27 smithi047 conmon[78521]: 2022-05-14T02:38:27.801 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:38:27 smithi047 conmon[78521]: During handling of the above exception, another exception occurred: 2022-05-14T02:38:27.801 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:38:27 smithi047 conmon[78521]: 2022-05-14T02:38:27.801 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:38:27 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T02:38:27.801 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:38:27 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T02:38:27.801 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:38:27 smithi047 conmon[78521]: return f(*arg) 2022-05-14T02:38:27.802 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:38:27 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T02:38:27.802 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:38:27 smithi047 conmon[78521]: self._write_client_files(client_files, host) 2022-05-14T02:38:27.802 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:38:27 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T02:38:27.802 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:38:27 smithi047 conmon[78521]: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T02:38:27.802 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:38:27 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T02:38:27.803 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:38:27 smithi047 conmon[78521]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T02:38:27.803 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:38:27 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T02:38:27.803 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:38:27 smithi047 conmon[78521]: return self.event_loop.get_result(coro) 2022-05-14T02:38:27.803 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:38:27 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T02:38:27.804 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:38:27 smithi047 conmon[78521]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T02:38:27.804 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:38:27 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T02:38:27.804 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:38:27 smithi047 conmon[78521]: return self.__get_result() 2022-05-14T02:38:27.804 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:38:27 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T02:38:27.805 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:38:27 smithi047 conmon[78521]: raise self._exception 2022-05-14T02:38:27.805 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:38:27 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T02:38:27.805 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:38:27 smithi047 conmon[78521]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T02:38:27.805 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:38:27 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T02:38:27.805 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:38:27 smithi047 conmon[78521]: conn = await self._remote_connection(host, addr) 2022-05-14T02:38:27.806 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:38:27 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T02:38:27.806 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:38:27 smithi047 conmon[78521]: raise 2022-05-14T02:38:27.806 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:38:27 smithi047 conmon[78521]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T02:38:27.806 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:38:27 smithi047 conmon[78521]: self.gen.throw(type, value, traceback) 2022-05-14T02:38:27.807 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:38:27 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T02:38:27.807 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:38:27 smithi047 conmon[78521]: raise OrchestratorError(msg) 2022-05-14T02:38:27.807 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:38:27 smithi047 conmon[78521]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:38:28.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:38:27 smithi115 ceph-mon[78593]: pgmap v6072: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:38:28.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:38:27 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:38:28.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:38:27 smithi047 ceph-mon[77147]: pgmap v6072: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:38:28.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:38:27 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:38:28.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:38:28 smithi115 conmon[90393]: 2022-05-14T02:38:28.166+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:38:29.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:38:28 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:38:29.108 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T02:38:29.109 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:38:29.109 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T02:38:29.109 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:38:29.109 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:38:29.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:38:29.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T02:38:29.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:38:29.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T02:38:29.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:38:29.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T02:38:29.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:38:29.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T02:38:29.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:38:29.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T02:38:29.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:38:29.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:38:29.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:38:29.112 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:38:28 smithi115 ceph-mon[78593]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T02:38:29.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T02:38:29.113 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:38:29.113 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T02:38:29.113 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:38:29.113 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:38:29.114 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:38:29.114 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T02:38:29.114 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:38:29.114 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T02:38:29.114 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:38:29.115 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T02:38:29.115 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:38:29.115 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T02:38:29.115 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:38:29.116 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T02:38:29.116 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:38:29.116 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:38:29.116 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:38:29.116 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T02:38:29.117 INFO:journalctl@ceph.mon.c.smithi115.stdout: During handling of the above exception, another exception occurred: 2022-05-14T02:38:29.117 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T02:38:29.117 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T02:38:29.117 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T02:38:29.118 INFO:journalctl@ceph.mon.c.smithi115.stdout: return f(*arg) 2022-05-14T02:38:29.118 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T02:38:29.118 INFO:journalctl@ceph.mon.c.smithi115.stdout: self._write_client_files(client_files, host) 2022-05-14T02:38:29.118 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T02:38:29.118 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T02:38:29.119 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T02:38:29.119 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T02:38:29.119 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T02:38:29.119 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.event_loop.get_result(coro) 2022-05-14T02:38:29.119 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T02:38:29.120 INFO:journalctl@ceph.mon.c.smithi115.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T02:38:29.120 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T02:38:29.120 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.__get_result() 2022-05-14T02:38:29.120 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T02:38:29.120 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise self._exception 2022-05-14T02:38:29.121 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T02:38:29.121 INFO:journalctl@ceph.mon.c.smithi115.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T02:38:29.121 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T02:38:29.121 INFO:journalctl@ceph.mon.c.smithi115.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T02:38:29.122 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T02:38:29.122 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise 2022-05-14T02:38:29.122 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T02:38:29.122 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.gen.throw(type, value, traceback) 2022-05-14T02:38:29.122 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T02:38:29.123 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OrchestratorError(msg) 2022-05-14T02:38:29.123 INFO:journalctl@ceph.mon.c.smithi115.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:38:29.123 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:38:28 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:38:29.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:38:28 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:38:29.292 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T02:38:29.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:38:29.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T02:38:29.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:38:29.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:38:29.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:38:29.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T02:38:29.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:38:29.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T02:38:29.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:38:29.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T02:38:29.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:38:29.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T02:38:29.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:38:29.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T02:38:29.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:38:29.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:38:29.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:38:29.296 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:38:28 smithi047 ceph-mon[77147]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T02:38:29.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T02:38:29.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:38:29.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T02:38:29.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:38:29.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:38:29.298 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:38:29.298 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T02:38:29.298 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:38:29.298 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T02:38:29.299 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:38:29.299 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T02:38:29.299 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:38:29.299 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T02:38:29.299 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:38:29.300 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T02:38:29.300 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:38:29.300 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:38:29.300 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:38:29.301 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T02:38:29.301 INFO:journalctl@ceph.mon.b.smithi047.stdout: During handling of the above exception, another exception occurred: 2022-05-14T02:38:29.301 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T02:38:29.301 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T02:38:29.301 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T02:38:29.302 INFO:journalctl@ceph.mon.b.smithi047.stdout: return f(*arg) 2022-05-14T02:38:29.302 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T02:38:29.302 INFO:journalctl@ceph.mon.b.smithi047.stdout: self._write_client_files(client_files, host) 2022-05-14T02:38:29.302 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T02:38:29.303 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T02:38:29.303 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T02:38:29.303 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T02:38:29.303 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T02:38:29.303 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.event_loop.get_result(coro) 2022-05-14T02:38:29.304 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T02:38:29.304 INFO:journalctl@ceph.mon.b.smithi047.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T02:38:29.304 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T02:38:29.304 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.__get_result() 2022-05-14T02:38:29.305 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T02:38:29.305 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise self._exception 2022-05-14T02:38:29.305 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T02:38:29.305 INFO:journalctl@ceph.mon.b.smithi047.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T02:38:29.305 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T02:38:29.306 INFO:journalctl@ceph.mon.b.smithi047.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T02:38:29.306 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T02:38:29.306 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise 2022-05-14T02:38:29.306 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T02:38:29.306 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.gen.throw(type, value, traceback) 2022-05-14T02:38:29.307 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T02:38:29.307 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OrchestratorError(msg) 2022-05-14T02:38:29.307 INFO:journalctl@ceph.mon.b.smithi047.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:38:29.307 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:38:28 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:38:29.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:38:29 smithi115 conmon[90393]: 2022-05-14T02:38:29.181+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:38:30.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:38:29 smithi115 ceph-mon[78593]: pgmap v6073: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:38:30.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:38:29 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:38:30.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:38:29 smithi047 ceph-mon[77147]: pgmap v6073: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:38:30.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:38:29 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:38:30.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:38:30 smithi115 conmon[90393]: 2022-05-14T02:38:30.228+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:38:31.247 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:38:30 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:38:31.248 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:38:30 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:38:31.248 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:38:30 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 11440 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:38:31.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:38:30 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:38:31.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:38:30 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:38:31.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:38:30 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 11440 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:38:31.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:38:31 smithi115 conmon[90393]: 2022-05-14T02:38:31.246+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:38:32.290 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:38:31 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:38:32.290 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:38:31 smithi115 ceph-mon[78593]: pgmap v6074: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:38:32.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:38:31 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:38:32.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:38:31 smithi047 ceph-mon[77147]: pgmap v6074: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:38:32.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:38:32 smithi115 conmon[90393]: 2022-05-14T02:38:32.289+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:38:33.270 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:38:32 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:38:33.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:38:32 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:38:33.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:38:33 smithi115 conmon[90393]: 2022-05-14T02:38:33.269+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:38:34.282 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:38:33 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:38:34.282 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:38:33 smithi115 ceph-mon[78593]: pgmap v6075: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:38:34.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:38:33 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:38:34.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:38:33 smithi047 ceph-mon[77147]: pgmap v6075: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:38:34.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:38:34 smithi115 conmon[90393]: 2022-05-14T02:38:34.280+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:38:35.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:38:34 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:38:35.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:38:34 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:38:35.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:38:34 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:38:35.314 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:38:34 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:38:35.315 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:38:34 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:38:35.315 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:38:34 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:38:35.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:38:35 smithi115 conmon[90393]: 2022-05-14T02:38:35.313+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:38:36.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:38:35 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:38:36.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:38:35 smithi047 ceph-mon[77147]: pgmap v6076: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:38:36.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:38:35 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 11450 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:38:36.352 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:38:35 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:38:36.353 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:38:35 smithi115 ceph-mon[78593]: pgmap v6076: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:38:36.353 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:38:35 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 11450 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:38:36.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:38:36 smithi115 conmon[90393]: 2022-05-14T02:38:36.352+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:38:37.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:38:36 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:38:37.338 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:38:36 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:38:37.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:38:37 smithi115 conmon[90393]: 2022-05-14T02:38:37.337+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:38:38.300 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:38:37 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:38:38.300 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:38:37 smithi047 ceph-mon[77147]: pgmap v6077: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:38:38.300 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:38:37 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:38:38.300 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:38:37 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:38:38.325 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:38:37 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:38:38.326 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:38:37 smithi115 ceph-mon[78593]: pgmap v6077: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:38:38.326 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:38:37 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:38:38.326 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:38:37 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:38:38.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:38:38 smithi115 conmon[90393]: 2022-05-14T02:38:38.324+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:38:39.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:38:38 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:38:39.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:38:38 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:38:39.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:38:39 smithi115 conmon[90393]: 2022-05-14T02:38:39.358+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:38:40.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:38:39 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:38:40.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:38:39 smithi047 ceph-mon[77147]: pgmap v6078: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:38:40.313 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:38:39 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:38:40.313 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:38:39 smithi115 ceph-mon[78593]: pgmap v6078: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:38:40.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:38:40 smithi115 conmon[90393]: 2022-05-14T02:38:40.312+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:38:41.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:38:40 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:38:41.355 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:38:40 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:38:41.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:38:41 smithi115 conmon[90393]: 2022-05-14T02:38:41.354+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:38:42.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:38:41 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:38:42.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:38:41 smithi047 ceph-mon[77147]: pgmap v6079: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:38:42.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:38:41 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 11455 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:38:42.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:38:41 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:38:42.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:38:41 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:38:42.336 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:38:41 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:38:42.337 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:38:41 smithi115 ceph-mon[78593]: pgmap v6079: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:38:42.337 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:38:41 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 11455 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:38:42.338 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:38:41 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:38:42.338 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:38:41 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:38:42.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:38:42 smithi115 conmon[90393]: 2022-05-14T02:38:42.335+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:38:43.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:38:42 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:38:43.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:38:42 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:38:43.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:38:43 smithi115 conmon[90393]: 2022-05-14T02:38:43.295+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:38:44.269 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:38:43 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:38:44.269 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:38:43 smithi115 ceph-mon[78593]: pgmap v6080: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:38:44.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:38:43 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:38:44.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:38:43 smithi047 ceph-mon[77147]: pgmap v6080: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:38:44.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:38:44 smithi115 conmon[90393]: 2022-05-14T02:38:44.268+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:38:45.233 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:38:44 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:38:45.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:38:44 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:38:45.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:38:45 smithi115 conmon[90393]: 2022-05-14T02:38:45.232+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:38:46.260 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:38:45 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:38:46.260 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:38:45 smithi115 ceph-mon[78593]: pgmap v6081: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:38:46.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:38:45 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:38:46.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:38:45 smithi047 ceph-mon[77147]: pgmap v6081: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:38:46.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:38:46 smithi115 conmon[90393]: 2022-05-14T02:38:46.259+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:38:47.279 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:38:46 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:38:47.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:38:46 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:38:47.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:38:47 smithi115 conmon[90393]: 2022-05-14T02:38:47.277+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:38:48.283 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:38:47 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:38:48.284 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:38:47 smithi115 ceph-mon[78593]: pgmap v6082: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:38:48.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:38:47 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:38:48.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:38:47 smithi047 ceph-mon[77147]: pgmap v6082: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:38:48.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:38:48 smithi115 conmon[90393]: 2022-05-14T02:38:48.282+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:38:49.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:38:48 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:38:49.323 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:38:48 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:38:49.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:38:49 smithi115 conmon[90393]: 2022-05-14T02:38:49.321+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:38:50.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:38:49 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:38:50.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:38:49 smithi047 ceph-mon[77147]: pgmap v6083: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:38:50.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:38:49 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:38:50.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:38:49 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:38:50.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:38:49 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:38:50.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:38:49 smithi115 ceph-mon[78593]: pgmap v6083: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:38:50.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:38:49 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:38:50.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:38:49 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:38:50.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:38:50 smithi115 conmon[90393]: 2022-05-14T02:38:50.358+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:38:51.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:38:50 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:38:51.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:38:50 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 11460 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:38:51.333 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:38:50 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:38:51.334 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:38:50 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 11460 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:38:51.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:38:51 smithi115 conmon[90393]: 2022-05-14T02:38:51.332+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:38:52.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:38:51 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:38:52.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:38:51 smithi047 ceph-mon[77147]: pgmap v6084: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:38:52.302 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:38:51 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:38:52.302 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:38:51 smithi115 ceph-mon[78593]: pgmap v6084: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:38:52.613 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:38:52 smithi115 conmon[90393]: 2022-05-14T02:38:52.301+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:38:53.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:38:52 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:38:53.344 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:38:52 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:38:53.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:38:53 smithi115 conmon[90393]: 2022-05-14T02:38:53.342+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:38:54.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:38:53 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:38:54.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:38:53 smithi047 ceph-mon[77147]: pgmap v6085: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:38:54.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:38:53 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:38:54.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:38:53 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:38:54.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:38:53 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:38:54.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:38:53 smithi115 ceph-mon[78593]: pgmap v6085: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:38:54.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:38:53 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:38:54.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:38:53 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:38:54.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:38:54 smithi115 conmon[90393]: 2022-05-14T02:38:54.384+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:38:55.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:38:54 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:38:55.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:38:54 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:38:55.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:38:55 smithi115 conmon[90393]: 2022-05-14T02:38:55.418+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:38:56.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:38:55 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:38:56.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:38:55 smithi047 ceph-mon[77147]: pgmap v6086: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:38:56.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:38:55 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 11470 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:38:56.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:38:55 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:38:56.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:38:55 smithi115 ceph-mon[78593]: pgmap v6086: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:38:56.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:38:55 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 11470 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:38:56.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:38:56 smithi115 conmon[90393]: 2022-05-14T02:38:56.419+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:38:57.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:38:56 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:38:57.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:38:56 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:38:57.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:38:57 smithi115 conmon[90393]: 2022-05-14T02:38:57.430+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:38:58.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:38:57 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:38:58.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:38:57 smithi047 ceph-mon[77147]: pgmap v6087: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:38:58.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:38:57 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:38:58.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:38:57 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:38:58.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:38:57 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:38:58.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:38:57 smithi115 ceph-mon[78593]: pgmap v6087: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:38:58.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:38:57 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:38:58.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:38:57 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:38:58.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:38:58 smithi115 conmon[90393]: 2022-05-14T02:38:58.439+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:38:59.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:38:58 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:38:59.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:38:58 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:38:59.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:38:59 smithi115 conmon[90393]: 2022-05-14T02:38:59.444+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:39:00.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:38:59 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:39:00.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:38:59 smithi047 ceph-mon[77147]: pgmap v6088: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:39:00.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:38:59 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:39:00.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:38:59 smithi115 ceph-mon[78593]: pgmap v6088: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:39:00.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:39:00 smithi115 conmon[90393]: 2022-05-14T02:39:00.452+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:39:01.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:39:00 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:39:01.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:39:00 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:39:01.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:39:01 smithi115 conmon[90393]: 2022-05-14T02:39:01.479+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:39:02.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:39:01 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:39:02.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:39:01 smithi047 ceph-mon[77147]: pgmap v6089: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:39:02.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:39:01 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 11475 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:39:02.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:39:01 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:39:02.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:39:01 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:39:02.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:39:01 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:39:02.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:39:01 smithi115 ceph-mon[78593]: pgmap v6089: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:39:02.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:39:01 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 11475 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:39:02.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:39:01 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:39:02.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:39:01 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:39:02.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:39:02 smithi115 conmon[90393]: 2022-05-14T02:39:02.452+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:39:03.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:39:02 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:39:03.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:39:02 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:39:03.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:39:03 smithi115 conmon[90393]: 2022-05-14T02:39:03.487+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:39:04.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:39:03 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:39:04.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:39:03 smithi047 ceph-mon[77147]: pgmap v6090: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:39:04.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:39:03 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:39:04.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:39:03 smithi115 ceph-mon[78593]: pgmap v6090: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:39:04.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:39:04 smithi115 conmon[90393]: 2022-05-14T02:39:04.445+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:39:05.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:39:04 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:39:05.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:39:04 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:39:05.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:39:05 smithi115 conmon[90393]: 2022-05-14T02:39:05.482+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:39:06.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:39:05 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:39:06.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:39:05 smithi047 ceph-mon[77147]: pgmap v6091: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:39:06.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:39:05 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:39:06.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:39:05 smithi115 ceph-mon[78593]: pgmap v6091: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:39:06.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:39:06 smithi115 conmon[90393]: 2022-05-14T02:39:06.515+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:39:07.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:39:06 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:39:07.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:39:06 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:39:07.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:39:07 smithi115 conmon[90393]: 2022-05-14T02:39:07.494+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:39:08.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:39:07 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:39:08.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:39:07 smithi047 ceph-mon[77147]: pgmap v6092: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:39:08.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:39:07 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:39:08.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:39:07 smithi115 ceph-mon[78593]: pgmap v6092: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:39:08.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:39:08 smithi115 conmon[90393]: 2022-05-14T02:39:08.540+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:39:09.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:39:08 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:39:09.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:39:08 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:39:09.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:39:09 smithi115 conmon[90393]: 2022-05-14T02:39:09.521+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:39:10.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:39:09 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:39:10.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:39:09 smithi047 ceph-mon[77147]: pgmap v6093: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:39:10.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:39:09 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:39:10.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:39:09 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:39:10.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:39:09 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:39:10.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:39:09 smithi115 ceph-mon[78593]: pgmap v6093: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:39:10.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:39:09 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:39:10.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:39:09 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:39:10.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:39:10 smithi115 conmon[90393]: 2022-05-14T02:39:10.515+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:39:11.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:39:10 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:39:11.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:39:10 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 11480 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:39:11.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:39:10 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:39:11.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:39:10 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 11480 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:39:11.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:39:11 smithi115 conmon[90393]: 2022-05-14T02:39:11.537+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:39:12.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:39:11 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:39:12.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:39:11 smithi047 ceph-mon[77147]: pgmap v6094: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:39:12.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:39:11 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:39:12.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:39:11 smithi115 ceph-mon[78593]: pgmap v6094: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:39:12.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:39:12 smithi115 conmon[90393]: 2022-05-14T02:39:12.554+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:39:13.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:39:13 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:39:13.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:39:13 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:39:13.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:39:13 smithi115 conmon[90393]: 2022-05-14T02:39:13.506+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:39:14.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:39:14 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:39:14.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:39:14 smithi047 ceph-mon[77147]: pgmap v6095: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:39:14.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:39:14 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:39:14.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:39:14 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:39:14.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:39:14 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:39:14.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:39:14 smithi115 ceph-mon[78593]: pgmap v6095: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:39:14.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:39:14 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:39:14.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:39:14 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:39:14.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:39:14 smithi115 conmon[90393]: 2022-05-14T02:39:14.543+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:39:15.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:39:15 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:39:15.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:39:15 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T02:39:15.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:39:15 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T02:39:15.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:39:15 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:39:15.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:39:15 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T02:39:15.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:39:15 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T02:39:15.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:39:15 smithi115 conmon[90393]: 2022-05-14T02:39:15.494+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:39:16.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:39:16 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:39:16.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:39:16 smithi047 ceph-mon[77147]: pgmap v6096: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:39:16.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:39:16 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 11490 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:39:16.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:39:16 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:39:16.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:39:16 smithi115 ceph-mon[78593]: pgmap v6096: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:39:16.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:39:16 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 11490 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:39:16.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:39:16 smithi115 conmon[90393]: 2022-05-14T02:39:16.497+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:39:17.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:39:17 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:39:17.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:39:17 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:39:17.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:39:17 smithi115 conmon[90393]: 2022-05-14T02:39:17.454+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:39:18.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:39:18 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:39:18.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:39:18 smithi047 ceph-mon[77147]: pgmap v6097: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:39:18.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:39:18 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:39:18.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:39:18 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:39:18.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:39:18 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:39:18.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:39:18 smithi115 ceph-mon[78593]: pgmap v6097: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:39:18.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:39:18 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:39:18.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:39:18 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:39:18.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:39:18 smithi115 conmon[90393]: 2022-05-14T02:39:18.480+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:39:19.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:39:19 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:39:19.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:39:19 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:39:19.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:39:19 smithi115 conmon[90393]: 2022-05-14T02:39:19.446+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:39:20.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:39:20 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:39:20.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:39:20 smithi047 ceph-mon[77147]: pgmap v6098: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:39:20.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:39:20 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:39:20.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:39:20 smithi115 ceph-mon[78593]: pgmap v6098: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:39:20.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:39:20 smithi115 conmon[90393]: 2022-05-14T02:39:20.408+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:39:21.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:39:21 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:39:21.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:39:21 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:39:21.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:39:21 smithi115 conmon[90393]: 2022-05-14T02:39:21.378+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:39:22.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:39:22 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:39:22.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:39:22 smithi047 ceph-mon[77147]: pgmap v6099: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:39:22.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:39:22 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 11495 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:39:22.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:39:22 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:39:22.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:39:22 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:39:22.343 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:39:22 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:39:22.344 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:39:22 smithi115 ceph-mon[78593]: pgmap v6099: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:39:22.344 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:39:22 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 11495 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:39:22.344 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:39:22 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:39:22.345 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:39:22 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:39:22.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:39:22 smithi115 conmon[90393]: 2022-05-14T02:39:22.342+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:39:23.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:39:23 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:39:23.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:39:23 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:39:23.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:39:23 smithi115 conmon[90393]: 2022-05-14T02:39:23.359+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:39:24.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:39:24 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:39:24.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:39:24 smithi115 ceph-mon[78593]: pgmap v6100: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:39:24.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:39:24 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:39:24.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:39:24 smithi047 ceph-mon[77147]: pgmap v6100: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:39:24.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:39:24 smithi115 conmon[90393]: 2022-05-14T02:39:24.401+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:39:25.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:39:25 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:39:25.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:39:25 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:39:25.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:39:25 smithi115 conmon[90393]: 2022-05-14T02:39:25.401+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:39:26.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:39:26 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:39:26.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:39:26 smithi115 ceph-mon[78593]: pgmap v6101: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:39:26.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:39:26 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:39:26.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:39:26 smithi047 ceph-mon[77147]: pgmap v6101: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:39:26.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:39:26 smithi115 conmon[90393]: 2022-05-14T02:39:26.400+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:39:27.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:39:27 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:39:27.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:39:27 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:39:27.634 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:39:27 smithi115 conmon[90393]: 2022-05-14T02:39:27.364+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:39:28.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:39:28 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:39:28.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:39:28 smithi047 ceph-mon[77147]: pgmap v6102: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:39:28.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:39:28 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T02:39:28.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:39:28 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T02:39:28.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:39:28 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T02:39:28.293 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:39:28 smithi047 conmon[78521]: 2022-05-14T02:39:28.111+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:39:28.293 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:39:28 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T02:39:28.293 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:39:28 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:39:28.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:39:28 smithi047 conmon[78521]: yield 2022-05-14T02:39:28.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:39:28 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:39:28.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:39:28 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:39:28.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:39:28 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:39:28.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:39:28 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T02:39:28.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:39:28 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:39:28.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:39:28 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T02:39:28.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:39:28 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:39:28.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:39:28 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T02:39:28.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:39:28 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:39:28.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:39:28 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T02:39:28.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:39:28 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:39:28.297 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:39:28 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T02:39:28.297 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:39:28 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:39:28.297 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:39:28 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:39:28.297 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:39:28 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:39:28.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:39:28 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:39:28.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:39:28 smithi115 ceph-mon[78593]: pgmap v6102: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:39:28.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:39:28 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T02:39:28.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:39:28 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T02:39:28.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:39:28 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T02:39:28.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:39:28 smithi115 conmon[90393]: 2022-05-14T02:39:28.388+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:39:29.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:39:29 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:39:29.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:39:29 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:39:29.359 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T02:39:29.359 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:39:29.359 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T02:39:29.359 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:39:29.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:39:29.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:39:29.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T02:39:29.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:39:29.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T02:39:29.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:39:29.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T02:39:29.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:39:29.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T02:39:29.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:39:29.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T02:39:29.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:39:29.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:39:29.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:39:29.364 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:39:29 smithi115 ceph-mon[78593]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T02:39:29.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:39:29 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:39:29.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:39:29 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:39:29.542 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T02:39:29.542 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:39:29.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T02:39:29.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:39:29.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:39:29.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:39:29.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T02:39:29.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:39:29.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T02:39:29.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:39:29.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T02:39:29.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:39:29.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T02:39:29.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:39:29.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T02:39:29.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:39:29.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:39:29.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:39:29.546 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:39:29 smithi047 ceph-mon[77147]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T02:39:29.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:39:29 smithi115 conmon[90393]: 2022-05-14T02:39:29.382+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:39:30.365 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:39:30 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:39:30.365 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:39:30 smithi115 ceph-mon[78593]: pgmap v6103: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:39:30.366 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:39:30 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:39:30.366 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:39:30 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:39:30.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:39:30 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:39:30.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:39:30 smithi047 ceph-mon[77147]: pgmap v6103: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:39:30.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:39:30 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:39:30.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:39:30 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:39:30.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:39:30 smithi115 conmon[90393]: 2022-05-14T02:39:30.397+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:39:31.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:39:31 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:39:31.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:39:31 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 11500 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:39:31.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:39:31 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:39:31.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:39:31 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 11500 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:39:31.542 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:39:31 smithi047 conmon[78521]: 2022-05-14T02:39:31.183+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:39:31.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:39:31 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T02:39:31.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:39:31 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:39:31.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:39:31 smithi047 conmon[78521]: yield 2022-05-14T02:39:31.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:39:31 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:39:31.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:39:31 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:39:31.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:39:31 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:39:31.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:39:31 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T02:39:31.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:39:31 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:39:31.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:39:31 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T02:39:31.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:39:31 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:39:31.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:39:31 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T02:39:31.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:39:31 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:39:31.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:39:31 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T02:39:31.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:39:31 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:39:31.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:39:31 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T02:39:31.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:39:31 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:39:31.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:39:31 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:39:31.547 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:39:31 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:39:31.547 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:39:31 smithi047 conmon[78521]: 2022-05-14T02:39:31.184+0000 7f5341f7c700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T02:39:31.547 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:39:31 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T02:39:31.547 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:39:31 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:39:31.547 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:39:31 smithi047 conmon[78521]: yield 2022-05-14T02:39:31.548 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:39:31 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:39:31.548 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:39:31 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:39:31.548 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:39:31 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:39:31.548 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:39:31 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T02:39:31.549 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:39:31 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:39:31.549 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:39:31 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T02:39:31.549 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:39:31 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:39:31.549 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:39:31 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T02:39:31.549 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:39:31 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:39:31.550 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:39:31 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T02:39:31.550 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:39:31 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:39:31.550 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:39:31 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T02:39:31.550 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:39:31 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:39:31.550 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:39:31 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:39:31.551 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:39:31 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:39:31.551 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:39:31 smithi047 conmon[78521]: 2022-05-14T02:39:31.551 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:39:31 smithi047 conmon[78521]: During handling of the above exception, another exception occurred: 2022-05-14T02:39:31.551 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:39:31 smithi047 conmon[78521]: 2022-05-14T02:39:31.552 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:39:31 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T02:39:31.552 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:39:31 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T02:39:31.552 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:39:31 smithi047 conmon[78521]: return f(*arg) 2022-05-14T02:39:31.552 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:39:31 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T02:39:31.552 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:39:31 smithi047 conmon[78521]: self._write_client_files(client_files, host) 2022-05-14T02:39:31.553 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:39:31 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T02:39:31.553 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:39:31 smithi047 conmon[78521]: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T02:39:31.553 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:39:31 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T02:39:31.553 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:39:31 smithi047 conmon[78521]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T02:39:31.554 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:39:31 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T02:39:31.554 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:39:31 smithi047 conmon[78521]: return self.event_loop.get_result(coro) 2022-05-14T02:39:31.554 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:39:31 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T02:39:31.554 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:39:31 smithi047 conmon[78521]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T02:39:31.554 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:39:31 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T02:39:31.555 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:39:31 smithi047 conmon[78521]: return self.__get_result() 2022-05-14T02:39:31.555 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:39:31 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T02:39:31.555 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:39:31 smithi047 conmon[78521]: raise self._exception 2022-05-14T02:39:31.555 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:39:31 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T02:39:31.556 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:39:31 smithi047 conmon[78521]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T02:39:31.556 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:39:31 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T02:39:31.556 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:39:31 smithi047 conmon[78521]: conn = await self._remote_connection(host, addr) 2022-05-14T02:39:31.556 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:39:31 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T02:39:31.556 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:39:31 smithi047 conmon[78521]: raise 2022-05-14T02:39:31.557 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:39:31 smithi047 conmon[78521]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T02:39:31.557 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:39:31 smithi047 conmon[78521]: self.gen.throw(type, value, traceback) 2022-05-14T02:39:31.557 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:39:31 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T02:39:31.557 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:39:31 smithi047 conmon[78521]: raise OrchestratorError(msg) 2022-05-14T02:39:31.558 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:39:31 smithi047 conmon[78521]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:39:31.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:39:31 smithi115 conmon[90393]: 2022-05-14T02:39:31.420+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:39:32.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:39:32 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:39:32.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:39:32 smithi115 ceph-mon[78593]: pgmap v6104: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:39:32.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:39:32 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:39:32.359 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T02:39:32.359 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:39:32.359 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T02:39:32.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:39:32.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:39:32.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:39:32.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T02:39:32.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:39:32.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T02:39:32.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:39:32.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T02:39:32.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:39:32.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T02:39:32.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:39:32.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T02:39:32.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:39:32.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:39:32.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:39:32.363 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:39:32 smithi115 ceph-mon[78593]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T02:39:32.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T02:39:32.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:39:32.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T02:39:32.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:39:32.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:39:32.365 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:39:32.365 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T02:39:32.365 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:39:32.365 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T02:39:32.365 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:39:32.366 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T02:39:32.366 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:39:32.366 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T02:39:32.366 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:39:32.367 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T02:39:32.367 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:39:32.367 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:39:32.367 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:39:32.368 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T02:39:32.368 INFO:journalctl@ceph.mon.c.smithi115.stdout: During handling of the above exception, another exception occurred: 2022-05-14T02:39:32.368 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T02:39:32.368 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T02:39:32.369 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T02:39:32.369 INFO:journalctl@ceph.mon.c.smithi115.stdout: return f(*arg) 2022-05-14T02:39:32.369 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T02:39:32.369 INFO:journalctl@ceph.mon.c.smithi115.stdout: self._write_client_files(client_files, host) 2022-05-14T02:39:32.369 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T02:39:32.370 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T02:39:32.370 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T02:39:32.370 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T02:39:32.370 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T02:39:32.371 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.event_loop.get_result(coro) 2022-05-14T02:39:32.371 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T02:39:32.371 INFO:journalctl@ceph.mon.c.smithi115.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T02:39:32.372 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T02:39:32.372 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.__get_result() 2022-05-14T02:39:32.372 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T02:39:32.372 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise self._exception 2022-05-14T02:39:32.372 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T02:39:32.373 INFO:journalctl@ceph.mon.c.smithi115.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T02:39:32.373 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T02:39:32.373 INFO:journalctl@ceph.mon.c.smithi115.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T02:39:32.373 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T02:39:32.374 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise 2022-05-14T02:39:32.374 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T02:39:32.374 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.gen.throw(type, value, traceback) 2022-05-14T02:39:32.374 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T02:39:32.374 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OrchestratorError(msg) 2022-05-14T02:39:32.375 INFO:journalctl@ceph.mon.c.smithi115.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:39:32.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:39:32 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:39:32.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:39:32 smithi047 ceph-mon[77147]: pgmap v6104: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:39:32.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:39:32 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:39:32.542 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T02:39:32.542 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:39:32.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T02:39:32.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:39:32.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:39:32.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:39:32.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T02:39:32.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:39:32.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T02:39:32.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:39:32.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T02:39:32.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:39:32.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T02:39:32.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:39:32.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T02:39:32.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:39:32.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:39:32.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:39:32.547 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:39:32 smithi047 ceph-mon[77147]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T02:39:32.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T02:39:32.548 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:39:32.548 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T02:39:32.548 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:39:32.548 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:39:32.549 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:39:32.549 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T02:39:32.549 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:39:32.549 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T02:39:32.549 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:39:32.550 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T02:39:32.550 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:39:32.550 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T02:39:32.550 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:39:32.551 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T02:39:32.551 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:39:32.551 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:39:32.551 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:39:32.551 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T02:39:32.552 INFO:journalctl@ceph.mon.b.smithi047.stdout: During handling of the above exception, another exception occurred: 2022-05-14T02:39:32.552 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T02:39:32.552 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T02:39:32.552 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T02:39:32.553 INFO:journalctl@ceph.mon.b.smithi047.stdout: return f(*arg) 2022-05-14T02:39:32.553 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T02:39:32.553 INFO:journalctl@ceph.mon.b.smithi047.stdout: self._write_client_files(client_files, host) 2022-05-14T02:39:32.553 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T02:39:32.554 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T02:39:32.554 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T02:39:32.554 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T02:39:32.554 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T02:39:32.555 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.event_loop.get_result(coro) 2022-05-14T02:39:32.555 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T02:39:32.555 INFO:journalctl@ceph.mon.b.smithi047.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T02:39:32.555 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T02:39:32.556 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.__get_result() 2022-05-14T02:39:32.556 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T02:39:32.556 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise self._exception 2022-05-14T02:39:32.556 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T02:39:32.556 INFO:journalctl@ceph.mon.b.smithi047.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T02:39:32.557 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T02:39:32.557 INFO:journalctl@ceph.mon.b.smithi047.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T02:39:32.557 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T02:39:32.557 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise 2022-05-14T02:39:32.558 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T02:39:32.558 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.gen.throw(type, value, traceback) 2022-05-14T02:39:32.558 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T02:39:32.558 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OrchestratorError(msg) 2022-05-14T02:39:32.558 INFO:journalctl@ceph.mon.b.smithi047.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:39:32.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:39:32 smithi115 conmon[90393]: 2022-05-14T02:39:32.424+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:39:33.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:39:33 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:39:33.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:39:33 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:39:33.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:39:33 smithi115 conmon[90393]: 2022-05-14T02:39:33.396+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:39:34.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:39:34 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:39:34.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:39:34 smithi115 ceph-mon[78593]: pgmap v6105: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:39:34.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:39:34 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:39:34.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:39:34 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:39:34.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:39:34 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:39:34.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:39:34 smithi047 ceph-mon[77147]: pgmap v6105: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:39:34.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:39:34 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:39:34.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:39:34 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:39:34.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:39:34 smithi115 conmon[90393]: 2022-05-14T02:39:34.368+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:39:35.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:39:35 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:39:35.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:39:35 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:39:35.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:39:35 smithi115 conmon[90393]: 2022-05-14T02:39:35.383+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:39:36.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:39:36 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:39:36.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:39:36 smithi115 ceph-mon[78593]: pgmap v6106: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:39:36.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:39:36 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 11510 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:39:36.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:39:36 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:39:36.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:39:36 smithi047 ceph-mon[77147]: pgmap v6106: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:39:36.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:39:36 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 11510 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:39:36.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:39:36 smithi115 conmon[90393]: 2022-05-14T02:39:36.394+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:39:37.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:39:37 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:39:37.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:39:37 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:39:37.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:39:37 smithi115 conmon[90393]: 2022-05-14T02:39:37.419+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:39:38.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:39:38 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:39:38.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:39:38 smithi115 ceph-mon[78593]: pgmap v6107: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:39:38.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:39:38 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:39:38.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:39:38 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:39:38.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:39:38 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:39:38.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:39:38 smithi047 ceph-mon[77147]: pgmap v6107: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:39:38.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:39:38 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:39:38.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:39:38 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:39:38.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:39:38 smithi115 conmon[90393]: 2022-05-14T02:39:38.373+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:39:39.353 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:39:39 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:39:39.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:39:39 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:39:39.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:39:39 smithi115 conmon[90393]: 2022-05-14T02:39:39.353+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:39:40.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:39:40 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:39:40.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:39:40 smithi115 ceph-mon[78593]: pgmap v6108: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:39:40.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:39:40 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:39:40.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:39:40 smithi047 ceph-mon[77147]: pgmap v6108: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:39:40.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:39:40 smithi115 conmon[90393]: 2022-05-14T02:39:40.390+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:39:41.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:39:41 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:39:41.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:39:41 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:39:41.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:39:41 smithi115 conmon[90393]: 2022-05-14T02:39:41.415+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:39:42.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:39:42 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:39:42.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:39:42 smithi115 ceph-mon[78593]: pgmap v6109: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:39:42.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:39:42 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 11515 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:39:42.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:39:42 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:39:42.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:39:42 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:39:42.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:39:42 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:39:42.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:39:42 smithi047 ceph-mon[77147]: pgmap v6109: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:39:42.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:39:42 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 11515 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:39:42.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:39:42 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:39:42.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:39:42 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:39:42.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:39:42 smithi115 conmon[90393]: 2022-05-14T02:39:42.408+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:39:43.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:39:43 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:39:43.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:39:43 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:39:43.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:39:43 smithi115 conmon[90393]: 2022-05-14T02:39:43.422+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:39:44.380 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:39:44 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:39:44.381 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:39:44 smithi115 ceph-mon[78593]: pgmap v6110: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:39:44.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:39:44 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:39:44.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:39:44 smithi047 ceph-mon[77147]: pgmap v6110: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:39:44.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:39:44 smithi115 conmon[90393]: 2022-05-14T02:39:44.379+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:39:45.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:39:45 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:39:45.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:39:45 smithi115 conmon[90393]: 2022-05-14T02:39:45.356+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:39:45.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:39:45 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:39:46.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:39:46 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:39:46.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:39:46 smithi047 ceph-mon[77147]: pgmap v6111: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:39:46.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:39:46 smithi115 conmon[90393]: 2022-05-14T02:39:46.308+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:39:46.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:39:46 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:39:46.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:39:46 smithi115 ceph-mon[78593]: pgmap v6111: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:39:47.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:39:47 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:39:47.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:39:47 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:39:47.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:39:47 smithi115 conmon[90393]: 2022-05-14T02:39:47.356+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:39:48.373 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:39:48 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:39:48.374 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:39:48 smithi115 ceph-mon[78593]: pgmap v6112: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:39:48.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:39:48 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:39:48.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:39:48 smithi047 ceph-mon[77147]: pgmap v6112: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:39:48.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:39:48 smithi115 conmon[90393]: 2022-05-14T02:39:48.372+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:39:49.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:39:49 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:39:49.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:39:49 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:39:49.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:39:49 smithi115 conmon[90393]: 2022-05-14T02:39:49.326+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:39:50.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:39:50 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:39:50.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:39:50 smithi047 ceph-mon[77147]: pgmap v6113: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:39:50.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:39:50 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:39:50.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:39:50 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:39:50.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:39:50 smithi115 conmon[90393]: 2022-05-14T02:39:50.341+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:39:50.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:39:50 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:39:50.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:39:50 smithi115 ceph-mon[78593]: pgmap v6113: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:39:50.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:39:50 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:39:50.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:39:50 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:39:51.387 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:39:51 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:39:51.388 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:39:51 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 11520 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:39:51.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:39:51 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:39:51.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:39:51 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 11520 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:39:51.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:39:51 smithi115 conmon[90393]: 2022-05-14T02:39:51.385+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:39:52.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:39:52 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:39:52.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:39:52 smithi047 ceph-mon[77147]: pgmap v6114: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:39:52.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:39:52 smithi115 conmon[90393]: 2022-05-14T02:39:52.347+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:39:52.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:39:52 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:39:52.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:39:52 smithi115 ceph-mon[78593]: pgmap v6114: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:39:53.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:39:53 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:39:53.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:39:53 smithi115 conmon[90393]: 2022-05-14T02:39:53.310+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:39:53.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:39:53 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:39:54.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:39:54 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:39:54.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:39:54 smithi047 ceph-mon[77147]: pgmap v6115: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:39:54.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:39:54 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:39:54.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:39:54 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:39:54.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:39:54 smithi115 conmon[90393]: 2022-05-14T02:39:54.335+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:39:54.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:39:54 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:39:54.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:39:54 smithi115 ceph-mon[78593]: pgmap v6115: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:39:54.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:39:54 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:39:54.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:39:54 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:39:55.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:39:55 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:39:55.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:39:55 smithi115 conmon[90393]: 2022-05-14T02:39:55.311+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:39:55.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:39:55 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:39:56.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:39:56 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:39:56.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:39:56 smithi047 ceph-mon[77147]: pgmap v6116: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:39:56.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:39:56 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 11530 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:39:56.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:39:56 smithi115 conmon[90393]: 2022-05-14T02:39:56.322+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:39:56.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:39:56 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:39:56.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:39:56 smithi115 ceph-mon[78593]: pgmap v6116: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:39:56.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:39:56 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 11530 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:39:57.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:39:57 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:39:57.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:39:57 smithi115 conmon[90393]: 2022-05-14T02:39:57.293+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:39:57.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:39:57 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:39:58.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:39:58 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:39:58.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:39:58 smithi047 ceph-mon[77147]: pgmap v6117: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:39:58.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:39:58 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:39:58.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:39:58 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:39:58.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:39:58 smithi115 conmon[90393]: 2022-05-14T02:39:58.337+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:39:58.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:39:58 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:39:58.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:39:58 smithi115 ceph-mon[78593]: pgmap v6117: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:39:58.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:39:58 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:39:58.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:39:58 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:39:59.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:39:59 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:39:59.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:39:59 smithi115 conmon[90393]: 2022-05-14T02:39:59.321+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:39:59.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:39:59 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:40:00.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:40:00 smithi047 conmon[77123]: 2022-05-14T02:39:59.999+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : Health detail: HEALTH_ERR 1 filesystem is degraded; 1 mds daemon damaged; 1 MDSs are read only; 1 MDSs report slow metadata IOs; 1/3 mons down, quorum b,c; Reduced data availability: 1 pg inactive; 9 slow ops, oldest one blocked for 11530 sec, osd.5 has slow ops 2022-05-14T02:40:00.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:40:00 smithi047 conmon[77123]: 2022-05-14T02:39:59.999+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : [WRN] FS_DEGRADED: 1 filesystem is degraded 2022-05-14T02:40:00.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:40:00 smithi047 conmon[77123]: 2022-05-14T02:39:59.999+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : fs cephfs is degraded 2022-05-14T02:40:00.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:40:00 smithi047 conmon[77123]: 2022-05-14T02:39:59.999+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : [ERR] MDS_DAMAGE: 1 mds daemon damaged 2022-05-14T02:40:00.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:40:00 smithi047 conmon[77123]: 2022-05-14T02:39:59.999+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : fs cephfs mds.2 is damaged 2022-05-14T02:40:00.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:40:00 smithi047 conmon[77123]: 2022-05-14T02:39:59.999+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : [WRN] MDS_READ_ONLY: 1 MDSs are read only 2022-05-14T02:40:00.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:40:00 smithi047 conmon[77123]: 2022-05-14T02:40:00.000+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : mds.l(mds.0): MDS in read-only mode 2022-05-14T02:40:00.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:40:00 smithi047 conmon[77123]: 2022-05-14T02:40:00.000+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : [WRN] MDS_SLOW_METADATA_IO: 1 MDSs report slow metadata IOs 2022-05-14T02:40:00.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:40:00 smithi047 conmon[77123]: 2022-05-14T02:40:00.000+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : mds.l(mds.0): 4 slow metadata IOs are blocked > 180 secs, oldest blocked for 11833 secs 2022-05-14T02:40:00.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:40:00 smithi047 conmon[77123]: 2022-05-14T02:40:00.000+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : [WRN] MON_DOWN: 1/3 mons down, quorum b,c 2022-05-14T02:40:00.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:40:00 smithi047 conmon[77123]: 2022-05-14T02:40:00.000+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : mon.a (rank 0) addr [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] is down (out of quorum) 2022-05-14T02:40:00.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:40:00 smithi047 conmon[77123]: 2022-05-14T02:40:00.000+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : [WRN] PG_AVAILABILITY: Reduced data availability: 1 pg inactive 2022-05-14T02:40:00.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:40:00 smithi047 conmon[77123]: 2022-05-14T02:40:00.000+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : pg 2.1 is stuck inactive for 3h, current state unknown, last acting [] 2022-05-14T02:40:00.295 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:40:00 smithi047 conmon[77123]: 2022-05-14T02:40:00.000+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : [WRN] SLOW_OPS: 9 slow ops, oldest one blocked for 11530 sec, osd.5 has slow ops 2022-05-14T02:40:00.295 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:40:00 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:40:00.295 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:40:00 smithi047 ceph-mon[77147]: pgmap v6118: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:40:00.295 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:40:00 smithi047 ceph-mon[77147]: Health detail: HEALTH_ERR 1 filesystem is degraded; 1 mds daemon damaged; 1 MDSs are read only; 1 MDSs report slow metadata IOs; 1/3 mons down, quorum b,c; Reduced data availability: 1 pg inactive; 9 slow ops, oldest one blocked for 11530 sec, osd.5 has slow ops 2022-05-14T02:40:00.295 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:40:00 smithi047 ceph-mon[77147]: [WRN] FS_DEGRADED: 1 filesystem is degraded 2022-05-14T02:40:00.296 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:40:00 smithi047 ceph-mon[77147]: fs cephfs is degraded 2022-05-14T02:40:00.296 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:40:00 smithi047 ceph-mon[77147]: [ERR] MDS_DAMAGE: 1 mds daemon damaged 2022-05-14T02:40:00.296 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:40:00 smithi047 ceph-mon[77147]: fs cephfs mds.2 is damaged 2022-05-14T02:40:00.296 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:40:00 smithi047 ceph-mon[77147]: [WRN] MDS_READ_ONLY: 1 MDSs are read only 2022-05-14T02:40:00.296 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:40:00 smithi047 ceph-mon[77147]: mds.l(mds.0): MDS in read-only mode 2022-05-14T02:40:00.297 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:40:00 smithi047 ceph-mon[77147]: [WRN] MDS_SLOW_METADATA_IO: 1 MDSs report slow metadata IOs 2022-05-14T02:40:00.297 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:40:00 smithi047 ceph-mon[77147]: mds.l(mds.0): 4 slow metadata IOs are blocked > 180 secs, oldest blocked for 11833 secs 2022-05-14T02:40:00.297 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:40:00 smithi047 ceph-mon[77147]: [WRN] MON_DOWN: 1/3 mons down, quorum b,c 2022-05-14T02:40:00.297 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:40:00 smithi047 ceph-mon[77147]: mon.a (rank 0) addr [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] is down (out of quorum) 2022-05-14T02:40:00.298 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:40:00 smithi047 ceph-mon[77147]: [WRN] PG_AVAILABILITY: Reduced data availability: 1 pg inactive 2022-05-14T02:40:00.298 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:40:00 smithi047 ceph-mon[77147]: pg 2.1 is stuck inactive for 3h, current state unknown, last acting [] 2022-05-14T02:40:00.298 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:40:00 smithi047 ceph-mon[77147]: [WRN] SLOW_OPS: 9 slow ops, oldest one blocked for 11530 sec, osd.5 has slow ops 2022-05-14T02:40:00.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:40:00 smithi115 conmon[90393]: 2022-05-14T02:40:00.348+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:40:00.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:40:00 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:40:00.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:40:00 smithi115 ceph-mon[78593]: pgmap v6118: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:40:00.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:40:00 smithi115 ceph-mon[78593]: Health detail: HEALTH_ERR 1 filesystem is degraded; 1 mds daemon damaged; 1 MDSs are read only; 1 MDSs report slow metadata IOs; 1/3 mons down, quorum b,c; Reduced data availability: 1 pg inactive; 9 slow ops, oldest one blocked for 11530 sec, osd.5 has slow ops 2022-05-14T02:40:00.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:40:00 smithi115 ceph-mon[78593]: [WRN] FS_DEGRADED: 1 filesystem is degraded 2022-05-14T02:40:00.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:40:00 smithi115 ceph-mon[78593]: fs cephfs is degraded 2022-05-14T02:40:00.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:40:00 smithi115 ceph-mon[78593]: [ERR] MDS_DAMAGE: 1 mds daemon damaged 2022-05-14T02:40:00.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:40:00 smithi115 ceph-mon[78593]: fs cephfs mds.2 is damaged 2022-05-14T02:40:00.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:40:00 smithi115 ceph-mon[78593]: [WRN] MDS_READ_ONLY: 1 MDSs are read only 2022-05-14T02:40:00.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:40:00 smithi115 ceph-mon[78593]: mds.l(mds.0): MDS in read-only mode 2022-05-14T02:40:00.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:40:00 smithi115 ceph-mon[78593]: [WRN] MDS_SLOW_METADATA_IO: 1 MDSs report slow metadata IOs 2022-05-14T02:40:00.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:40:00 smithi115 ceph-mon[78593]: mds.l(mds.0): 4 slow metadata IOs are blocked > 180 secs, oldest blocked for 11833 secs 2022-05-14T02:40:00.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:40:00 smithi115 ceph-mon[78593]: [WRN] MON_DOWN: 1/3 mons down, quorum b,c 2022-05-14T02:40:00.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:40:00 smithi115 ceph-mon[78593]: mon.a (rank 0) addr [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] is down (out of quorum) 2022-05-14T02:40:00.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:40:00 smithi115 ceph-mon[78593]: [WRN] PG_AVAILABILITY: Reduced data availability: 1 pg inactive 2022-05-14T02:40:00.612 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:40:00 smithi115 ceph-mon[78593]: pg 2.1 is stuck inactive for 3h, current state unknown, last acting [] 2022-05-14T02:40:00.612 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:40:00 smithi115 ceph-mon[78593]: [WRN] SLOW_OPS: 9 slow ops, oldest one blocked for 11530 sec, osd.5 has slow ops 2022-05-14T02:40:01.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:40:01 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:40:01.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:40:01 smithi115 conmon[90393]: 2022-05-14T02:40:01.337+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:40:01.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:40:01 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:40:02.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:40:02 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:40:02.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:40:02 smithi047 ceph-mon[77147]: pgmap v6119: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:40:02.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:40:02 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 11535 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:40:02.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:40:02 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:40:02.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:40:02 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:40:02.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:40:02 smithi115 conmon[90393]: 2022-05-14T02:40:02.297+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:40:02.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:40:02 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:40:02.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:40:02 smithi115 ceph-mon[78593]: pgmap v6119: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:40:02.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:40:02 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 11535 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:40:02.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:40:02 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:40:02.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:40:02 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:40:03.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:40:03 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:40:03.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:40:03 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:40:03.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:40:03 smithi115 conmon[90393]: 2022-05-14T02:40:03.304+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:40:04.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:40:04 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:40:04.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:40:04 smithi047 ceph-mon[77147]: pgmap v6120: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:40:04.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:40:04 smithi115 conmon[90393]: 2022-05-14T02:40:04.267+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:40:04.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:40:04 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:40:04.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:40:04 smithi115 ceph-mon[78593]: pgmap v6120: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:40:05.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:40:05 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:40:05.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:40:05 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:40:05.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:40:05 smithi115 conmon[90393]: 2022-05-14T02:40:05.271+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:40:06.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:40:06 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:40:06.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:40:06 smithi047 ceph-mon[77147]: pgmap v6121: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:40:06.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:40:06 smithi115 conmon[90393]: 2022-05-14T02:40:06.249+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:40:06.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:40:06 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:40:06.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:40:06 smithi115 ceph-mon[78593]: pgmap v6121: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:40:07.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:40:07 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:40:07.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:40:07 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:40:07.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:40:07 smithi115 conmon[90393]: 2022-05-14T02:40:07.287+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:40:08.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:40:08 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:40:08.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:40:08 smithi047 ceph-mon[77147]: pgmap v6122: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:40:08.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:40:08 smithi115 conmon[90393]: 2022-05-14T02:40:08.318+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:40:08.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:40:08 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:40:08.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:40:08 smithi115 ceph-mon[78593]: pgmap v6122: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:40:09.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:40:09 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:40:09.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:40:09 smithi115 conmon[90393]: 2022-05-14T02:40:09.357+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:40:09.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:40:09 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:40:10.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:40:10 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:40:10.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:40:10 smithi047 ceph-mon[77147]: pgmap v6123: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:40:10.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:40:10 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:40:10.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:40:10 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:40:10.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:40:10 smithi115 conmon[90393]: 2022-05-14T02:40:10.326+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:40:10.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:40:10 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:40:10.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:40:10 smithi115 ceph-mon[78593]: pgmap v6123: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:40:10.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:40:10 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:40:10.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:40:10 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:40:11.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:40:11 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:40:11.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:40:11 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 11540 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:40:11.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:40:11 smithi115 conmon[90393]: 2022-05-14T02:40:11.349+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:40:11.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:40:11 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:40:11.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:40:11 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 11540 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:40:12.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:40:12 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:40:12.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:40:12 smithi047 ceph-mon[77147]: pgmap v6124: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:40:12.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:40:12 smithi115 conmon[90393]: 2022-05-14T02:40:12.304+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:40:12.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:40:12 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:40:12.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:40:12 smithi115 ceph-mon[78593]: pgmap v6124: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:40:13.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:40:13 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:40:13.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:40:13 smithi115 conmon[90393]: 2022-05-14T02:40:13.283+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:40:13.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:40:13 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:40:14.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:40:14 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:40:14.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:40:14 smithi047 ceph-mon[77147]: pgmap v6125: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:40:14.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:40:14 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:40:14.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:40:14 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:40:14.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:40:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T02:40:14.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:40:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T02:40:14.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:40:14 smithi115 conmon[90393]: 2022-05-14T02:40:14.245+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:40:14.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:40:14 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:40:14.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:40:14 smithi115 ceph-mon[78593]: pgmap v6125: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:40:14.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:40:14 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:40:14.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:40:14 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:40:14.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:40:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T02:40:14.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:40:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T02:40:15.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:40:15 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:40:15.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:40:15 smithi115 conmon[90393]: 2022-05-14T02:40:15.261+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:40:15.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:40:15 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:40:16.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:40:16 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:40:16.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:40:16 smithi047 ceph-mon[77147]: pgmap v6126: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:40:16.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:40:16 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 11550 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:40:16.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:40:16 smithi115 conmon[90393]: 2022-05-14T02:40:16.238+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:40:16.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:40:16 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:40:16.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:40:16 smithi115 ceph-mon[78593]: pgmap v6126: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:40:16.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:40:16 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 11550 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:40:17.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:40:17 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:40:17.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:40:17 smithi115 conmon[90393]: 2022-05-14T02:40:17.260+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:40:17.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:40:17 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:40:18.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:40:18 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:40:18.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:40:18 smithi047 ceph-mon[77147]: pgmap v6127: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:40:18.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:40:18 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:40:18.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:40:18 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:40:18.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:40:18 smithi115 conmon[90393]: 2022-05-14T02:40:18.260+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:40:18.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:40:18 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:40:18.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:40:18 smithi115 ceph-mon[78593]: pgmap v6127: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:40:18.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:40:18 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:40:18.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:40:18 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:40:19.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:40:19 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:40:19.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:40:19 smithi115 conmon[90393]: 2022-05-14T02:40:19.246+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:40:19.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:40:19 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:40:20.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:40:20 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:40:20.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:40:20 smithi047 ceph-mon[77147]: pgmap v6128: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:40:20.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:40:20 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:40:20.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:40:20 smithi115 conmon[90393]: 2022-05-14T02:40:20.247+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:40:20.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:40:20 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:40:20.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:40:20 smithi115 ceph-mon[78593]: pgmap v6128: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:40:20.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:40:20 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:40:21.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:40:21 smithi115 conmon[90393]: 2022-05-14T02:40:21.271+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:40:22.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:40:21 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:40:22.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:40:21 smithi115 ceph-mon[78593]: pgmap v6129: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:40:22.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:40:21 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 11555 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:40:22.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:40:21 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:40:22.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:40:21 smithi047 ceph-mon[77147]: pgmap v6129: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:40:22.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:40:21 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 11555 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:40:22.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:40:22 smithi115 conmon[90393]: 2022-05-14T02:40:22.311+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:40:23.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:40:22 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:40:23.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:40:22 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:40:23.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:40:22 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:40:23.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:40:22 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:40:23.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:40:22 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:40:23.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:40:22 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:40:23.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:40:23 smithi115 conmon[90393]: 2022-05-14T02:40:23.349+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:40:24.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:40:23 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:40:24.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:40:23 smithi047 ceph-mon[77147]: pgmap v6130: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:40:24.304 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:40:23 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:40:24.304 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:40:23 smithi115 ceph-mon[78593]: pgmap v6130: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:40:24.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:40:24 smithi115 conmon[90393]: 2022-05-14T02:40:24.303+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:40:25.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:40:24 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:40:25.326 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:40:24 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:40:25.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:40:25 smithi115 conmon[90393]: 2022-05-14T02:40:25.325+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:40:26.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:40:25 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:40:26.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:40:25 smithi047 ceph-mon[77147]: pgmap v6131: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:40:26.345 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:40:25 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:40:26.345 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:40:25 smithi115 ceph-mon[78593]: pgmap v6131: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:40:26.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:40:26 smithi115 conmon[90393]: 2022-05-14T02:40:26.344+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:40:27.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:40:26 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:40:27.344 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:40:26 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:40:27.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:40:27 smithi115 conmon[90393]: 2022-05-14T02:40:27.343+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:40:28.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:40:27 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:40:28.295 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:40:27 smithi047 ceph-mon[77147]: pgmap v6132: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:40:28.344 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:40:27 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:40:28.345 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:40:27 smithi115 ceph-mon[78593]: pgmap v6132: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:40:28.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:40:28 smithi115 conmon[90393]: 2022-05-14T02:40:28.343+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:40:29.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:40:28 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:40:29.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:40:28 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:40:29.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:40:29 smithi115 conmon[90393]: 2022-05-14T02:40:29.381+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:40:30.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:40:29 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:40:30.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:40:29 smithi047 ceph-mon[77147]: pgmap v6133: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:40:30.332 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:40:29 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:40:30.333 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:40:29 smithi115 ceph-mon[78593]: pgmap v6133: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:40:30.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:40:30 smithi115 conmon[90393]: 2022-05-14T02:40:30.330+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:40:31.230 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:40:30 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:40:31.230 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:40:30 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:40:31.231 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:40:30 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:40:31.231 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:40:30 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 11560 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:40:31.233 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:40:30 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:40:31.233 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:40:30 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:40:31.234 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:40:30 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:40:31.234 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:40:30 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 11560 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:40:31.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:40:31 smithi115 conmon[90393]: 2022-05-14T02:40:31.361+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:40:32.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:40:31 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:40:32.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:40:31 smithi047 ceph-mon[77147]: pgmap v6134: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:40:32.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:40:31 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T02:40:32.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:40:31 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T02:40:32.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:40:31 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T02:40:32.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:40:31 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:40:32.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:40:31 smithi115 ceph-mon[78593]: pgmap v6134: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:40:32.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:40:31 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T02:40:32.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:40:31 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T02:40:32.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:40:31 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T02:40:32.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:40:32 smithi115 conmon[90393]: 2022-05-14T02:40:32.383+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:40:33.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:40:32 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:40:33.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:40:32 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:40:33.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:40:33 smithi115 conmon[90393]: 2022-05-14T02:40:33.382+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:40:34.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:40:33 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:40:34.290 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:40:33 smithi047 ceph-mon[77147]: pgmap v6135: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:40:34.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:40:33 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:40:34.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:40:33 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:40:34.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:40:33 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:40:34.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:40:33 smithi115 ceph-mon[78593]: pgmap v6135: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:40:34.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:40:33 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:40:34.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:40:33 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:40:34.541 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:40:34 smithi047 conmon[78521]: 2022-05-14T02:40:34.288+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:40:34.542 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:40:34 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T02:40:34.542 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:40:34 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:40:34.542 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:40:34 smithi047 conmon[78521]: yield 2022-05-14T02:40:34.542 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:40:34 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:40:34.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:40:34 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:40:34.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:40:34 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:40:34.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:40:34 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T02:40:34.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:40:34 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:40:34.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:40:34 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T02:40:34.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:40:34 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:40:34.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:40:34 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T02:40:34.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:40:34 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:40:34.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:40:34 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T02:40:34.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:40:34 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:40:34.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:40:34 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T02:40:34.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:40:34 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:40:34.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:40:34 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:40:34.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:40:34 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:40:34.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:40:34 smithi115 conmon[90393]: 2022-05-14T02:40:34.433+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:40:35.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:40:34 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:40:35.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:40:34 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:40:35.292 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T02:40:35.292 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:40:35.292 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T02:40:35.292 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:40:35.292 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:40:35.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:40:35.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T02:40:35.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:40:35.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T02:40:35.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:40:35.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T02:40:35.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:40:35.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T02:40:35.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:40:35.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T02:40:35.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:40:35.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:40:35.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:40:35.296 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:40:34 smithi047 ceph-mon[77147]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T02:40:35.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:40:34 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:40:35.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:40:34 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:40:35.358 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T02:40:35.359 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:40:35.359 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T02:40:35.359 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:40:35.359 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:40:35.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:40:35.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T02:40:35.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:40:35.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T02:40:35.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:40:35.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T02:40:35.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:40:35.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T02:40:35.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:40:35.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T02:40:35.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:40:35.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:40:35.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:40:35.363 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:40:34 smithi115 ceph-mon[78593]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T02:40:35.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:40:35 smithi115 conmon[90393]: 2022-05-14T02:40:35.430+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:40:36.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:40:35 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:40:36.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:40:35 smithi047 ceph-mon[77147]: pgmap v6136: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:40:36.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:40:35 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 11570 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:40:36.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:40:35 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:40:36.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:40:35 smithi115 ceph-mon[78593]: pgmap v6136: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:40:36.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:40:35 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 11570 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:40:36.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:40:36 smithi115 conmon[90393]: 2022-05-14T02:40:36.425+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:40:37.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:40:36 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:40:37.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:40:36 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:40:37.792 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:40:37 smithi047 conmon[78521]: 2022-05-14T02:40:37.360+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:40:37.792 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:40:37 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T02:40:37.792 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:40:37 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:40:37.792 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:40:37 smithi047 conmon[78521]: yield 2022-05-14T02:40:37.793 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:40:37 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:40:37.793 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:40:37 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:40:37.793 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:40:37 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:40:37.793 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:40:37 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T02:40:37.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:40:37 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:40:37.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:40:37 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T02:40:37.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:40:37 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:40:37.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:40:37 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T02:40:37.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:40:37 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:40:37.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:40:37 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T02:40:37.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:40:37 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:40:37.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:40:37 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T02:40:37.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:40:37 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:40:37.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:40:37 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:40:37.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:40:37 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:40:37.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:40:37 smithi047 conmon[78521]: 2022-05-14T02:40:37.361+0000 7f5345f84700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T02:40:37.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:40:37 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T02:40:37.797 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:40:37 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:40:37.797 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:40:37 smithi047 conmon[78521]: yield 2022-05-14T02:40:37.797 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:40:37 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:40:37.797 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:40:37 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:40:37.797 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:40:37 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:40:37.798 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:40:37 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T02:40:37.798 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:40:37 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:40:37.798 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:40:37 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T02:40:37.798 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:40:37 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:40:37.799 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:40:37 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T02:40:37.799 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:40:37 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:40:37.799 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:40:37 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T02:40:37.799 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:40:37 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:40:37.799 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:40:37 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T02:40:37.800 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:40:37 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:40:37.800 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:40:37 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:40:37.800 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:40:37 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:40:37.800 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:40:37 smithi047 conmon[78521]: 2022-05-14T02:40:37.801 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:40:37 smithi047 conmon[78521]: During handling of the above exception, another exception occurred: 2022-05-14T02:40:37.801 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:40:37 smithi047 conmon[78521]: 2022-05-14T02:40:37.801 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:40:37 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T02:40:37.801 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:40:37 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T02:40:37.802 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:40:37 smithi047 conmon[78521]: return f(*arg) 2022-05-14T02:40:37.802 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:40:37 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T02:40:37.802 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:40:37 smithi047 conmon[78521]: self._write_client_files(client_files, host) 2022-05-14T02:40:37.802 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:40:37 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T02:40:37.802 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:40:37 smithi047 conmon[78521]: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T02:40:37.803 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:40:37 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T02:40:37.803 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:40:37 smithi047 conmon[78521]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T02:40:37.803 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:40:37 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T02:40:37.803 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:40:37 smithi047 conmon[78521]: return self.event_loop.get_result(coro) 2022-05-14T02:40:37.804 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:40:37 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T02:40:37.804 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:40:37 smithi047 conmon[78521]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T02:40:37.804 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:40:37 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T02:40:37.804 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:40:37 smithi047 conmon[78521]: return self.__get_result() 2022-05-14T02:40:37.804 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:40:37 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T02:40:37.805 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:40:37 smithi047 conmon[78521]: raise self._exception 2022-05-14T02:40:37.805 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:40:37 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T02:40:37.805 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:40:37 smithi047 conmon[78521]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T02:40:37.805 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:40:37 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T02:40:37.806 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:40:37 smithi047 conmon[78521]: conn = await self._remote_connection(host, addr) 2022-05-14T02:40:37.806 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:40:37 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T02:40:37.806 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:40:37 smithi047 conmon[78521]: raise 2022-05-14T02:40:37.806 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:40:37 smithi047 conmon[78521]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T02:40:37.806 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:40:37 smithi047 conmon[78521]: self.gen.throw(type, value, traceback) 2022-05-14T02:40:37.807 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:40:37 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T02:40:37.807 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:40:37 smithi047 conmon[78521]: raise OrchestratorError(msg) 2022-05-14T02:40:37.807 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:40:37 smithi047 conmon[78521]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:40:37.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:40:37 smithi115 conmon[90393]: 2022-05-14T02:40:37.461+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:40:38.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:40:37 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:40:38.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:40:37 smithi047 ceph-mon[77147]: pgmap v6137: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:40:38.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:40:37 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:40:38.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:40:37 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:40:38.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:40:37 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:40:38.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T02:40:38.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:40:38.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T02:40:38.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:40:38.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:40:38.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:40:38.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T02:40:38.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:40:38.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T02:40:38.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:40:38.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T02:40:38.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:40:38.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T02:40:38.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:40:38.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T02:40:38.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:40:38.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:40:38.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:40:38.297 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:40:37 smithi047 ceph-mon[77147]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T02:40:38.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T02:40:38.298 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:40:38.298 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T02:40:38.298 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:40:38.298 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:40:38.299 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:40:38.299 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T02:40:38.299 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:40:38.299 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T02:40:38.299 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:40:38.300 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T02:40:38.300 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:40:38.300 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T02:40:38.300 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:40:38.300 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T02:40:38.301 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:40:38.301 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:40:38.301 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:40:38.301 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T02:40:38.302 INFO:journalctl@ceph.mon.b.smithi047.stdout: During handling of the above exception, another exception occurred: 2022-05-14T02:40:38.302 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T02:40:38.302 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T02:40:38.302 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T02:40:38.302 INFO:journalctl@ceph.mon.b.smithi047.stdout: return f(*arg) 2022-05-14T02:40:38.303 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T02:40:38.303 INFO:journalctl@ceph.mon.b.smithi047.stdout: self._write_client_files(client_files, host) 2022-05-14T02:40:38.303 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T02:40:38.303 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T02:40:38.303 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T02:40:38.304 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T02:40:38.304 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T02:40:38.304 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.event_loop.get_result(coro) 2022-05-14T02:40:38.304 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T02:40:38.305 INFO:journalctl@ceph.mon.b.smithi047.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T02:40:38.305 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T02:40:38.305 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.__get_result() 2022-05-14T02:40:38.305 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T02:40:38.305 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise self._exception 2022-05-14T02:40:38.306 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T02:40:38.306 INFO:journalctl@ceph.mon.b.smithi047.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T02:40:38.306 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T02:40:38.306 INFO:journalctl@ceph.mon.b.smithi047.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T02:40:38.306 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T02:40:38.307 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise 2022-05-14T02:40:38.307 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T02:40:38.307 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.gen.throw(type, value, traceback) 2022-05-14T02:40:38.307 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T02:40:38.308 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OrchestratorError(msg) 2022-05-14T02:40:38.308 INFO:journalctl@ceph.mon.b.smithi047.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:40:38.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:40:37 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:40:38.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:40:37 smithi115 ceph-mon[78593]: pgmap v6137: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:40:38.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:40:37 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:40:38.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:40:37 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:40:38.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:40:37 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:40:38.359 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T02:40:38.359 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:40:38.359 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T02:40:38.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:40:38.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:40:38.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:40:38.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T02:40:38.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:40:38.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T02:40:38.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:40:38.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T02:40:38.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:40:38.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T02:40:38.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:40:38.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T02:40:38.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:40:38.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:40:38.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:40:38.363 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:40:37 smithi115 ceph-mon[78593]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T02:40:38.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T02:40:38.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:40:38.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T02:40:38.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:40:38.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:40:38.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:40:38.365 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T02:40:38.365 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:40:38.365 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T02:40:38.365 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:40:38.365 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T02:40:38.366 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:40:38.366 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T02:40:38.366 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:40:38.366 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T02:40:38.366 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:40:38.367 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:40:38.367 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:40:38.367 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T02:40:38.367 INFO:journalctl@ceph.mon.c.smithi115.stdout: During handling of the above exception, another exception occurred: 2022-05-14T02:40:38.368 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T02:40:38.368 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T02:40:38.368 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T02:40:38.368 INFO:journalctl@ceph.mon.c.smithi115.stdout: return f(*arg) 2022-05-14T02:40:38.368 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T02:40:38.369 INFO:journalctl@ceph.mon.c.smithi115.stdout: self._write_client_files(client_files, host) 2022-05-14T02:40:38.369 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T02:40:38.369 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T02:40:38.369 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T02:40:38.370 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T02:40:38.370 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T02:40:38.370 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.event_loop.get_result(coro) 2022-05-14T02:40:38.370 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T02:40:38.370 INFO:journalctl@ceph.mon.c.smithi115.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T02:40:38.371 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T02:40:38.371 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.__get_result() 2022-05-14T02:40:38.371 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T02:40:38.371 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise self._exception 2022-05-14T02:40:38.371 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T02:40:38.372 INFO:journalctl@ceph.mon.c.smithi115.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T02:40:38.372 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T02:40:38.372 INFO:journalctl@ceph.mon.c.smithi115.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T02:40:38.372 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T02:40:38.373 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise 2022-05-14T02:40:38.373 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T02:40:38.373 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.gen.throw(type, value, traceback) 2022-05-14T02:40:38.373 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T02:40:38.373 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OrchestratorError(msg) 2022-05-14T02:40:38.374 INFO:journalctl@ceph.mon.c.smithi115.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:40:38.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:40:38 smithi115 conmon[90393]: 2022-05-14T02:40:38.487+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:40:39.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:40:38 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:40:39.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:40:38 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:40:39.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:40:39 smithi115 conmon[90393]: 2022-05-14T02:40:39.488+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:40:40.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:40:39 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:40:40.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:40:39 smithi047 ceph-mon[77147]: pgmap v6138: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:40:40.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:40:39 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:40:40.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:40:39 smithi115 ceph-mon[78593]: pgmap v6138: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:40:40.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:40:40 smithi115 conmon[90393]: 2022-05-14T02:40:40.466+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:40:41.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:40:40 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:40:41.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:40:40 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:40:41.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:40:41 smithi115 conmon[90393]: 2022-05-14T02:40:41.440+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:40:42.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:40:41 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:40:42.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:40:41 smithi047 ceph-mon[77147]: pgmap v6139: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:40:42.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:40:41 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 11575 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:40:42.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:40:41 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:40:42.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:40:41 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:40:42.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:40:41 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:40:42.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:40:41 smithi115 ceph-mon[78593]: pgmap v6139: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:40:42.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:40:41 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 11575 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:40:42.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:40:41 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:40:42.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:40:41 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:40:42.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:40:42 smithi115 conmon[90393]: 2022-05-14T02:40:42.425+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:40:43.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:40:42 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:40:43.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:40:42 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:40:43.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:40:43 smithi115 conmon[90393]: 2022-05-14T02:40:43.422+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:40:44.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:40:43 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:40:44.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:40:43 smithi047 ceph-mon[77147]: pgmap v6140: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:40:44.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:40:43 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:40:44.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:40:43 smithi115 ceph-mon[78593]: pgmap v6140: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:40:44.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:40:44 smithi115 conmon[90393]: 2022-05-14T02:40:44.416+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:40:45.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:40:44 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:40:45.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:40:44 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:40:45.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:40:45 smithi115 conmon[90393]: 2022-05-14T02:40:45.448+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:40:46.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:40:45 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:40:46.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:40:45 smithi047 ceph-mon[77147]: pgmap v6141: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:40:46.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:40:45 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:40:46.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:40:45 smithi115 ceph-mon[78593]: pgmap v6141: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:40:46.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:40:46 smithi115 conmon[90393]: 2022-05-14T02:40:46.455+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:40:47.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:40:46 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:40:47.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:40:46 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:40:47.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:40:47 smithi115 conmon[90393]: 2022-05-14T02:40:47.452+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:40:48.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:40:47 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:40:48.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:40:47 smithi047 ceph-mon[77147]: pgmap v6142: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:40:48.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:40:47 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:40:48.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:40:47 smithi115 ceph-mon[78593]: pgmap v6142: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:40:48.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:40:48 smithi115 conmon[90393]: 2022-05-14T02:40:48.462+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:40:49.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:40:48 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:40:49.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:40:48 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:40:49.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:40:49 smithi115 conmon[90393]: 2022-05-14T02:40:49.433+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:40:50.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:40:49 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:40:50.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:40:49 smithi047 ceph-mon[77147]: pgmap v6143: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:40:50.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:40:49 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:40:50.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:40:49 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:40:50.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:40:49 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:40:50.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:40:49 smithi115 ceph-mon[78593]: pgmap v6143: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:40:50.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:40:49 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:40:50.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:40:49 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:40:50.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:40:50 smithi115 conmon[90393]: 2022-05-14T02:40:50.403+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:40:51.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:40:50 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:40:51.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:40:50 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 11580 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:40:51.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:40:50 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:40:51.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:40:50 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 11580 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:40:51.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:40:51 smithi115 conmon[90393]: 2022-05-14T02:40:51.426+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:40:52.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:40:51 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:40:52.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:40:51 smithi047 ceph-mon[77147]: pgmap v6144: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:40:52.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:40:51 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:40:52.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:40:51 smithi115 ceph-mon[78593]: pgmap v6144: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:40:52.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:40:52 smithi115 conmon[90393]: 2022-05-14T02:40:52.399+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:40:53.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:40:52 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:40:53.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:40:52 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:40:53.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:40:53 smithi115 conmon[90393]: 2022-05-14T02:40:53.375+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:40:54.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:40:53 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:40:54.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:40:53 smithi047 ceph-mon[77147]: pgmap v6145: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:40:54.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:40:53 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:40:54.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:40:53 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:40:54.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:40:53 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:40:54.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:40:53 smithi115 ceph-mon[78593]: pgmap v6145: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:40:54.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:40:53 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:40:54.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:40:53 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:40:54.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:40:54 smithi115 conmon[90393]: 2022-05-14T02:40:54.363+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:40:55.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:40:54 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:40:55.342 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:40:54 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:40:55.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:40:55 smithi115 conmon[90393]: 2022-05-14T02:40:55.340+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:40:56.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:40:55 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:40:56.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:40:55 smithi047 ceph-mon[77147]: pgmap v6146: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:40:56.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:40:55 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 11590 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:40:56.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:40:55 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:40:56.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:40:55 smithi115 ceph-mon[78593]: pgmap v6146: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:40:56.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:40:55 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 11590 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:40:56.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:40:56 smithi115 conmon[90393]: 2022-05-14T02:40:56.375+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:40:57.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:40:56 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:40:57.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:40:56 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:40:57.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:40:57 smithi115 conmon[90393]: 2022-05-14T02:40:57.400+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:40:58.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:40:57 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:40:58.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:40:57 smithi047 ceph-mon[77147]: pgmap v6147: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:40:58.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:40:57 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:40:58.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:40:57 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:40:58.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:40:57 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:40:58.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:40:57 smithi115 ceph-mon[78593]: pgmap v6147: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:40:58.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:40:57 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:40:58.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:40:57 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:40:58.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:40:58 smithi115 conmon[90393]: 2022-05-14T02:40:58.417+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:40:59.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:40:58 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:40:59.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:40:58 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:40:59.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:40:59 smithi115 conmon[90393]: 2022-05-14T02:40:59.436+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:41:00.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:40:59 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:41:00.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:40:59 smithi047 ceph-mon[77147]: pgmap v6148: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:41:00.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:40:59 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:41:00.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:40:59 smithi115 ceph-mon[78593]: pgmap v6148: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:41:00.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:41:00 smithi115 conmon[90393]: 2022-05-14T02:41:00.475+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:41:01.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:41:00 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:41:01.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:41:00 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:41:01.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:41:01 smithi115 conmon[90393]: 2022-05-14T02:41:01.520+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:41:02.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:41:01 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:41:02.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:41:01 smithi047 ceph-mon[77147]: pgmap v6149: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:41:02.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:41:01 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 11595 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:41:02.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:41:01 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:41:02.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:41:01 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:41:02.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:41:01 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:41:02.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:41:01 smithi115 ceph-mon[78593]: pgmap v6149: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:41:02.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:41:01 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 11595 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:41:02.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:41:01 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:41:02.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:41:01 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:41:02.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:41:02 smithi115 conmon[90393]: 2022-05-14T02:41:02.540+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:41:03.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:41:02 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:41:03.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:41:02 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:41:03.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:41:03 smithi115 conmon[90393]: 2022-05-14T02:41:03.490+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:41:04.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:41:03 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:41:04.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:41:03 smithi047 ceph-mon[77147]: pgmap v6150: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:41:04.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:41:03 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:41:04.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:41:03 smithi115 ceph-mon[78593]: pgmap v6150: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:41:04.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:41:04 smithi115 conmon[90393]: 2022-05-14T02:41:04.467+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:41:05.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:41:04 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:41:05.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:41:04 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:41:05.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:41:05 smithi115 conmon[90393]: 2022-05-14T02:41:05.423+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:41:06.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:41:06 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:41:06.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:41:06 smithi047 ceph-mon[77147]: pgmap v6151: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:41:06.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:41:06 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:41:06.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:41:06 smithi115 ceph-mon[78593]: pgmap v6151: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:41:06.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:41:06 smithi115 conmon[90393]: 2022-05-14T02:41:06.467+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:41:07.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:41:07 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:41:07.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:41:07 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:41:07.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:41:07 smithi115 conmon[90393]: 2022-05-14T02:41:07.419+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:41:08.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:41:08 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:41:08.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:41:08 smithi047 ceph-mon[77147]: pgmap v6152: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:41:08.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:41:08 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:41:08.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:41:08 smithi115 ceph-mon[78593]: pgmap v6152: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:41:08.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:41:08 smithi115 conmon[90393]: 2022-05-14T02:41:08.406+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:41:09.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:41:09 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:41:09.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:41:09 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:41:09.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:41:09 smithi115 conmon[90393]: 2022-05-14T02:41:09.411+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:41:10.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:41:10 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:41:10.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:41:10 smithi047 ceph-mon[77147]: pgmap v6153: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:41:10.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:41:10 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:41:10.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:41:10 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:41:10.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:41:10 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:41:10.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:41:10 smithi115 ceph-mon[78593]: pgmap v6153: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:41:10.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:41:10 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:41:10.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:41:10 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:41:10.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:41:10 smithi115 conmon[90393]: 2022-05-14T02:41:10.435+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:41:11.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:41:11 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:41:11.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:41:11 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 11600 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:41:11.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:41:11 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:41:11.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:41:11 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 11600 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:41:11.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:41:11 smithi115 conmon[90393]: 2022-05-14T02:41:11.459+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:41:12.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:41:12 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:41:12.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:41:12 smithi047 ceph-mon[77147]: pgmap v6154: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:41:12.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:41:12 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:41:12.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:41:12 smithi115 ceph-mon[78593]: pgmap v6154: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:41:12.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:41:12 smithi115 conmon[90393]: 2022-05-14T02:41:12.496+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:41:13.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:41:13 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:41:13.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:41:13 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:41:13.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:41:13 smithi115 conmon[90393]: 2022-05-14T02:41:13.481+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:41:14.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:41:14 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:41:14.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:41:14 smithi047 ceph-mon[77147]: pgmap v6155: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:41:14.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:41:14 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:41:14.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:41:14 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:41:14.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:41:14 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:41:14.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:41:14 smithi115 ceph-mon[78593]: pgmap v6155: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:41:14.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:41:14 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:41:14.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:41:14 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:41:14.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:41:14 smithi115 conmon[90393]: 2022-05-14T02:41:14.441+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:41:15.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:41:15 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:41:15.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:41:15 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T02:41:15.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:41:15 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T02:41:15.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:41:15 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:41:15.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:41:15 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T02:41:15.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:41:15 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T02:41:15.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:41:15 smithi115 conmon[90393]: 2022-05-14T02:41:15.416+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:41:16.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:41:16 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:41:16.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:41:16 smithi047 ceph-mon[77147]: pgmap v6156: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:41:16.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:41:16 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 11610 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:41:16.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:41:16 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:41:16.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:41:16 smithi115 ceph-mon[78593]: pgmap v6156: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:41:16.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:41:16 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 11610 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:41:16.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:41:16 smithi115 conmon[90393]: 2022-05-14T02:41:16.390+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:41:17.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:41:17 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:41:17.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:41:17 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:41:17.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:41:17 smithi115 conmon[90393]: 2022-05-14T02:41:17.438+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:41:18.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:41:18 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:41:18.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:41:18 smithi047 ceph-mon[77147]: pgmap v6157: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:41:18.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:41:18 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:41:18.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:41:18 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:41:18.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:41:18 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:41:18.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:41:18 smithi115 ceph-mon[78593]: pgmap v6157: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:41:18.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:41:18 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:41:18.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:41:18 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:41:18.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:41:18 smithi115 conmon[90393]: 2022-05-14T02:41:18.445+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:41:19.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:41:19 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:41:19.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:41:19 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:41:19.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:41:19 smithi115 conmon[90393]: 2022-05-14T02:41:19.396+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:41:20.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:41:20 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:41:20.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:41:20 smithi115 ceph-mon[78593]: pgmap v6158: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:41:20.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:41:20 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:41:20.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:41:20 smithi047 ceph-mon[77147]: pgmap v6158: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:41:20.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:41:20 smithi115 conmon[90393]: 2022-05-14T02:41:20.440+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:41:21.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:41:21 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:41:21.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:41:21 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:41:21.863 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:41:21 smithi115 conmon[90393]: 2022-05-14T02:41:21.428+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:41:22.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:41:22 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:41:22.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:41:22 smithi115 ceph-mon[78593]: pgmap v6159: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 7 op/s 2022-05-14T02:41:22.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:41:22 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 11615 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:41:22.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:41:22 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:41:22.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:41:22 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:41:22.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:41:22 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:41:22.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:41:22 smithi047 ceph-mon[77147]: pgmap v6159: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 7 op/s 2022-05-14T02:41:22.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:41:22 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 11615 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:41:22.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:41:22 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:41:22.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:41:22 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:41:22.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:41:22 smithi115 conmon[90393]: 2022-05-14T02:41:22.403+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:41:23.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:41:23 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:41:23.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:41:23 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:41:23.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:41:23 smithi115 conmon[90393]: 2022-05-14T02:41:23.397+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:41:24.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:41:24 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:41:24.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:41:24 smithi115 ceph-mon[78593]: pgmap v6160: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 4 op/s 2022-05-14T02:41:24.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:41:24 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:41:24.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:41:24 smithi047 ceph-mon[77147]: pgmap v6160: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 4 op/s 2022-05-14T02:41:24.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:41:24 smithi115 conmon[90393]: 2022-05-14T02:41:24.400+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:41:25.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:41:25 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:41:25.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:41:25 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:41:25.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:41:25 smithi115 conmon[90393]: 2022-05-14T02:41:25.417+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:41:26.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:41:26 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:41:26.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:41:26 smithi115 ceph-mon[78593]: pgmap v6161: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 7 op/s 2022-05-14T02:41:26.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:41:26 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:41:26.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:41:26 smithi047 ceph-mon[77147]: pgmap v6161: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 7 op/s 2022-05-14T02:41:26.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:41:26 smithi115 conmon[90393]: 2022-05-14T02:41:26.385+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:41:27.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:41:27 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:41:27.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:41:27 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:41:27.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:41:27 smithi115 conmon[90393]: 2022-05-14T02:41:27.371+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:41:28.335 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:41:28 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:41:28.336 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:41:28 smithi115 ceph-mon[78593]: pgmap v6162: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 4 op/s 2022-05-14T02:41:28.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:41:28 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:41:28.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:41:28 smithi047 ceph-mon[77147]: pgmap v6162: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 4 op/s 2022-05-14T02:41:28.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:41:28 smithi115 conmon[90393]: 2022-05-14T02:41:28.333+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:41:29.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:41:29 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:41:29.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:41:29 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:41:29.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:41:29 smithi115 conmon[90393]: 2022-05-14T02:41:29.369+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:41:30.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:41:30 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:41:30.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:41:30 smithi115 ceph-mon[78593]: pgmap v6163: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 4 op/s 2022-05-14T02:41:30.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:41:30 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:41:30.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:41:30 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:41:30.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:41:30 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:41:30.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:41:30 smithi047 ceph-mon[77147]: pgmap v6163: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 4 op/s 2022-05-14T02:41:30.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:41:30 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:41:30.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:41:30 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:41:30.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:41:30 smithi115 conmon[90393]: 2022-05-14T02:41:30.387+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:41:31.352 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:41:31 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:41:31.353 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:41:31 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 11620 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:41:31.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:41:31 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:41:31.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:41:31 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 11620 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:41:31.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:41:31 smithi115 conmon[90393]: 2022-05-14T02:41:31.351+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:41:32.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:41:32 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:41:32.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:41:32 smithi115 ceph-mon[78593]: pgmap v6164: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 7 op/s 2022-05-14T02:41:32.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:41:32 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:41:32.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:41:32 smithi047 ceph-mon[77147]: pgmap v6164: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 7 op/s 2022-05-14T02:41:32.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:41:32 smithi115 conmon[90393]: 2022-05-14T02:41:32.381+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:41:33.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:41:33 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:41:33.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:41:33 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:41:33.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:41:33 smithi115 conmon[90393]: 2022-05-14T02:41:33.418+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:41:34.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:41:34 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:41:34.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:41:34 smithi115 ceph-mon[78593]: pgmap v6165: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:41:34.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:41:34 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:41:34.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:41:34 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:41:34.545 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:41:34 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:41:34.546 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:41:34 smithi047 ceph-mon[77147]: pgmap v6165: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:41:34.546 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:41:34 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:41:34.546 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:41:34 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:41:34.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:41:34 smithi115 conmon[90393]: 2022-05-14T02:41:34.405+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:41:35.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:41:35 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:41:35.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:41:35 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:41:35.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:41:35 smithi115 conmon[90393]: 2022-05-14T02:41:35.367+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:41:36.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:41:36 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:41:36.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:41:36 smithi115 ceph-mon[78593]: pgmap v6166: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:41:36.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:41:36 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 11630 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:41:36.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:41:36 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:41:36.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:41:36 smithi047 ceph-mon[77147]: pgmap v6166: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:41:36.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:41:36 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 11630 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:41:36.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:41:36 smithi115 conmon[90393]: 2022-05-14T02:41:36.372+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:41:37.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:41:37 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:41:37.417 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:41:37 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:41:37.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:41:37 smithi115 conmon[90393]: 2022-05-14T02:41:37.392+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:41:38.107 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:41:37 smithi047 conmon[78521]: 2022-05-14T02:41:37.839+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:41:38.108 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:41:37 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T02:41:38.108 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:41:37 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:41:38.108 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:41:37 smithi047 conmon[78521]: yield 2022-05-14T02:41:38.109 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:41:37 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:41:38.109 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:41:37 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:41:38.109 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:41:37 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:41:38.109 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:41:37 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T02:41:38.110 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:41:37 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:41:38.110 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:41:37 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T02:41:38.110 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:41:37 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:41:38.110 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:41:37 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T02:41:38.110 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:41:37 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:41:38.111 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:41:37 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T02:41:38.111 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:41:37 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:41:38.111 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:41:37 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T02:41:38.111 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:41:37 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:41:38.111 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:41:37 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:41:38.112 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:41:37 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:41:38.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:41:38 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:41:38.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:41:38 smithi115 ceph-mon[78593]: pgmap v6167: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:41:38.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:41:38 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:41:38.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:41:38 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:41:38.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:41:38 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T02:41:38.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:41:38 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T02:41:38.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:41:38 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T02:41:38.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:41:38 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:41:38.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:41:38 smithi047 ceph-mon[77147]: pgmap v6167: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:41:38.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:41:38 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:41:38.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:41:38 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:41:38.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:41:38 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T02:41:38.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:41:38 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T02:41:38.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:41:38 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T02:41:38.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:41:38 smithi115 conmon[90393]: 2022-05-14T02:41:38.374+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:41:39.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:41:39 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:41:39.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:41:39 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:41:39.542 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T02:41:39.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:41:39.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T02:41:39.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:41:39.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:41:39.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:41:39.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T02:41:39.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:41:39.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T02:41:39.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:41:39.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T02:41:39.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:41:39.551 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T02:41:39.551 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:41:39.552 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T02:41:39.552 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:41:39.552 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:41:39.553 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:41:39.553 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:41:39 smithi047 ceph-mon[77147]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T02:41:39.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:41:39 smithi115 conmon[90393]: 2022-05-14T02:41:39.329+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:41:39.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:41:39 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:41:39.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:41:39 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:41:39.609 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T02:41:39.609 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:41:39.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T02:41:39.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:41:39.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:41:39.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:41:39.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T02:41:39.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:41:39.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T02:41:39.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:41:39.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T02:41:39.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:41:39.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T02:41:39.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:41:39.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T02:41:39.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:41:39.613 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:41:39.613 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:41:39.613 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:41:39 smithi115 ceph-mon[78593]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T02:41:40.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:41:40 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:41:40.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:41:40 smithi047 ceph-mon[77147]: pgmap v6168: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:41:40.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:41:40 smithi115 conmon[90393]: 2022-05-14T02:41:40.329+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:41:40.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:41:40 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:41:40.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:41:40 smithi115 ceph-mon[78593]: pgmap v6168: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:41:41.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:41:41 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:41:41.293 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:41:40 smithi047 conmon[78521]: 2022-05-14T02:41:40.911+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:41:41.293 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:41:40 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T02:41:41.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:41:40 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:41:41.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:41:40 smithi047 conmon[78521]: yield 2022-05-14T02:41:41.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:41:40 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:41:41.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:41:40 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:41:41.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:41:40 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:41:41.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:41:40 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T02:41:41.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:41:40 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:41:41.297 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:41:40 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T02:41:41.297 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:41:40 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:41:41.298 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:41:40 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T02:41:41.298 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:41:40 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:41:41.299 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:41:40 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T02:41:41.299 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:41:40 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:41:41.299 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:41:40 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T02:41:41.300 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:41:40 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:41:41.300 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:41:40 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:41:41.301 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:41:40 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:41:41.301 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:41:40 smithi047 conmon[78521]: 2022-05-14T02:41:40.912+0000 7f5340779700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T02:41:41.302 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:41:40 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T02:41:41.302 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:41:40 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:41:41.302 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:41:40 smithi047 conmon[78521]: yield 2022-05-14T02:41:41.303 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:41:40 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:41:41.303 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:41:40 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:41:41.304 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:41:40 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:41:41.304 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:41:40 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T02:41:41.305 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:41:40 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:41:41.305 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:41:40 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T02:41:41.306 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:41:40 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:41:41.306 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:41:40 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T02:41:41.306 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:41:40 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:41:41.307 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:41:40 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T02:41:41.307 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:41:40 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:41:41.308 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:41:40 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T02:41:41.308 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:41:40 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:41:41.309 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:41:40 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:41:41.309 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:41:40 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:41:41.310 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:41:40 smithi047 conmon[78521]: 2022-05-14T02:41:41.310 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:41:40 smithi047 conmon[78521]: During handling of the above exception, another exception occurred: 2022-05-14T02:41:41.310 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:41:40 smithi047 conmon[78521]: 2022-05-14T02:41:41.310 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:41:40 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T02:41:41.311 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:41:40 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T02:41:41.311 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:41:40 smithi047 conmon[78521]: return f(*arg) 2022-05-14T02:41:41.311 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:41:40 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T02:41:41.311 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:41:40 smithi047 conmon[78521]: self._write_client_files(client_files, host) 2022-05-14T02:41:41.311 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:41:40 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T02:41:41.312 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:41:40 smithi047 conmon[78521]: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T02:41:41.312 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:41:40 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T02:41:41.312 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:41:40 smithi047 conmon[78521]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T02:41:41.312 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:41:40 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T02:41:41.313 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:41:40 smithi047 conmon[78521]: return self.event_loop.get_result(coro) 2022-05-14T02:41:41.313 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:41:40 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T02:41:41.313 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:41:40 smithi047 conmon[78521]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T02:41:41.313 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:41:40 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T02:41:41.314 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:41:40 smithi047 conmon[78521]: return self.__get_result() 2022-05-14T02:41:41.314 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:41:40 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T02:41:41.314 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:41:40 smithi047 conmon[78521]: raise self._exception 2022-05-14T02:41:41.314 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:41:40 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T02:41:41.315 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:41:40 smithi047 conmon[78521]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T02:41:41.315 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:41:40 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T02:41:41.315 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:41:40 smithi047 conmon[78521]: conn = await self._remote_connection(host, addr) 2022-05-14T02:41:41.315 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:41:40 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T02:41:41.316 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:41:40 smithi047 conmon[78521]: raise 2022-05-14T02:41:41.316 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:41:40 smithi047 conmon[78521]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T02:41:41.316 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:41:40 smithi047 conmon[78521]: self.gen.throw(type, value, traceback) 2022-05-14T02:41:41.316 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:41:40 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T02:41:41.316 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:41:40 smithi047 conmon[78521]: raise OrchestratorError(msg) 2022-05-14T02:41:41.317 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:41:40 smithi047 conmon[78521]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:41:41.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:41:41 smithi115 conmon[90393]: 2022-05-14T02:41:41.312+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:41:41.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:41:41 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:41:42.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:41:42 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:41:42.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:41:42 smithi047 ceph-mon[77147]: pgmap v6169: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:41:42.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:41:42 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:41:42.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T02:41:42.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:41:42.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T02:41:42.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:41:42.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:41:42.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:41:42.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T02:41:42.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:41:42.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T02:41:42.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:41:42.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T02:41:42.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:41:42.548 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T02:41:42.548 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:41:42.548 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T02:41:42.549 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:41:42.549 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:41:42.549 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:41:42.550 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:41:42 smithi047 ceph-mon[77147]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T02:41:42.550 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T02:41:42.550 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:41:42.551 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T02:41:42.551 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:41:42.552 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:41:42.552 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:41:42.552 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T02:41:42.553 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:41:42.553 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T02:41:42.553 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:41:42.554 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T02:41:42.554 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:41:42.554 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T02:41:42.555 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:41:42.555 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T02:41:42.555 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:41:42.556 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:41:42.556 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:41:42.556 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T02:41:42.557 INFO:journalctl@ceph.mon.b.smithi047.stdout: During handling of the above exception, another exception occurred: 2022-05-14T02:41:42.557 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T02:41:42.557 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T02:41:42.558 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T02:41:42.558 INFO:journalctl@ceph.mon.b.smithi047.stdout: return f(*arg) 2022-05-14T02:41:42.558 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T02:41:42.558 INFO:journalctl@ceph.mon.b.smithi047.stdout: self._write_client_files(client_files, host) 2022-05-14T02:41:42.558 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T02:41:42.559 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T02:41:42.559 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T02:41:42.559 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T02:41:42.559 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T02:41:42.560 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.event_loop.get_result(coro) 2022-05-14T02:41:42.560 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T02:41:42.560 INFO:journalctl@ceph.mon.b.smithi047.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T02:41:42.560 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T02:41:42.560 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.__get_result() 2022-05-14T02:41:42.561 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T02:41:42.561 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise self._exception 2022-05-14T02:41:42.561 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T02:41:42.561 INFO:journalctl@ceph.mon.b.smithi047.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T02:41:42.562 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T02:41:42.562 INFO:journalctl@ceph.mon.b.smithi047.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T02:41:42.562 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T02:41:42.563 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise 2022-05-14T02:41:42.563 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T02:41:42.563 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.gen.throw(type, value, traceback) 2022-05-14T02:41:42.563 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T02:41:42.564 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OrchestratorError(msg) 2022-05-14T02:41:42.564 INFO:journalctl@ceph.mon.b.smithi047.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:41:42.564 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:41:42 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 11635 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:41:42.564 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:41:42 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:41:42.565 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:41:42 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:41:42.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:41:42 smithi115 conmon[90393]: 2022-05-14T02:41:42.312+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:41:42.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:41:42 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:41:42.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:41:42 smithi115 ceph-mon[78593]: pgmap v6169: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:41:42.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:41:42 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:41:42.609 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T02:41:42.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:41:42.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T02:41:42.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:41:42.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:41:42.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:41:42.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T02:41:42.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:41:42.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T02:41:42.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:41:42.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T02:41:42.613 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:41:42.613 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T02:41:42.613 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:41:42.613 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T02:41:42.614 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:41:42.614 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:41:42.614 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:41:42.614 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:41:42 smithi115 ceph-mon[78593]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T02:41:42.615 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T02:41:42.615 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:41:42.615 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T02:41:42.615 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:41:42.616 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:41:42.616 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:41:42.616 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T02:41:42.616 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:41:42.617 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T02:41:42.617 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:41:42.617 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T02:41:42.617 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:41:42.618 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T02:41:42.618 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:41:42.618 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T02:41:42.618 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:41:42.618 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:41:42.619 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:41:42.619 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T02:41:42.619 INFO:journalctl@ceph.mon.c.smithi115.stdout: During handling of the above exception, another exception occurred: 2022-05-14T02:41:42.619 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T02:41:42.620 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T02:41:42.620 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T02:41:42.620 INFO:journalctl@ceph.mon.c.smithi115.stdout: return f(*arg) 2022-05-14T02:41:42.620 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T02:41:42.620 INFO:journalctl@ceph.mon.c.smithi115.stdout: self._write_client_files(client_files, host) 2022-05-14T02:41:42.621 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T02:41:42.621 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T02:41:42.621 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T02:41:42.621 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T02:41:42.621 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T02:41:42.622 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.event_loop.get_result(coro) 2022-05-14T02:41:42.622 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T02:41:42.622 INFO:journalctl@ceph.mon.c.smithi115.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T02:41:42.622 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T02:41:42.622 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.__get_result() 2022-05-14T02:41:42.623 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T02:41:42.623 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise self._exception 2022-05-14T02:41:42.623 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T02:41:42.623 INFO:journalctl@ceph.mon.c.smithi115.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T02:41:42.624 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T02:41:42.624 INFO:journalctl@ceph.mon.c.smithi115.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T02:41:42.624 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T02:41:42.624 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise 2022-05-14T02:41:42.624 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T02:41:42.625 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.gen.throw(type, value, traceback) 2022-05-14T02:41:42.625 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T02:41:42.625 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OrchestratorError(msg) 2022-05-14T02:41:42.625 INFO:journalctl@ceph.mon.c.smithi115.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:41:42.626 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:41:42 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 11635 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:41:42.626 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:41:42 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:41:42.626 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:41:42 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:41:43.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:41:43 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:41:43.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:41:43 smithi115 conmon[90393]: 2022-05-14T02:41:43.341+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:41:43.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:41:43 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:41:44.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:41:44 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:41:44.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:41:44 smithi047 ceph-mon[77147]: pgmap v6170: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:41:44.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:41:44 smithi115 conmon[90393]: 2022-05-14T02:41:44.376+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:41:44.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:41:44 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:41:44.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:41:44 smithi115 ceph-mon[78593]: pgmap v6170: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:41:45.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:41:45 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:41:45.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:41:45 smithi115 conmon[90393]: 2022-05-14T02:41:45.351+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:41:45.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:41:45 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:41:46.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:41:46 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:41:46.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:41:46 smithi047 ceph-mon[77147]: pgmap v6171: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:41:46.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:41:46 smithi115 conmon[90393]: 2022-05-14T02:41:46.332+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:41:46.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:41:46 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:41:46.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:41:46 smithi115 ceph-mon[78593]: pgmap v6171: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:41:47.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:41:47 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:41:47.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:41:47 smithi115 conmon[90393]: 2022-05-14T02:41:47.359+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:41:47.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:41:47 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:41:48.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:41:48 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:41:48.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:41:48 smithi047 ceph-mon[77147]: pgmap v6172: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:41:48.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:41:48 smithi115 conmon[90393]: 2022-05-14T02:41:48.342+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:41:48.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:41:48 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:41:48.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:41:48 smithi115 ceph-mon[78593]: pgmap v6172: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:41:49.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:41:49 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:41:49.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:41:49 smithi115 conmon[90393]: 2022-05-14T02:41:49.354+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:41:49.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:41:49 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:41:50.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:41:50 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:41:50.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:41:50 smithi047 ceph-mon[77147]: pgmap v6173: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:41:50.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:41:50 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:41:50.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:41:50 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:41:50.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:41:50 smithi115 conmon[90393]: 2022-05-14T02:41:50.359+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:41:50.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:41:50 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:41:50.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:41:50 smithi115 ceph-mon[78593]: pgmap v6173: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:41:50.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:41:50 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:41:50.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:41:50 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:41:51.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:41:51 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:41:51.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:41:51 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 11640 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:41:51.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:41:51 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:41:51.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:41:51 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 11640 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:41:51.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:41:51 smithi115 conmon[90393]: 2022-05-14T02:41:51.337+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:41:52.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:41:52 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:41:52.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:41:52 smithi047 ceph-mon[77147]: pgmap v6174: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:41:52.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:41:52 smithi115 conmon[90393]: 2022-05-14T02:41:52.291+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:41:52.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:41:52 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:41:52.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:41:52 smithi115 ceph-mon[78593]: pgmap v6174: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:41:53.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:41:53 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:41:53.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:41:53 smithi115 conmon[90393]: 2022-05-14T02:41:53.249+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:41:53.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:41:53 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:41:54.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:41:54 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:41:54.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:41:54 smithi047 ceph-mon[77147]: pgmap v6175: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:41:54.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:41:54 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:41:54.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:41:54 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:41:54.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:41:54 smithi115 conmon[90393]: 2022-05-14T02:41:54.286+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:41:54.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:41:54 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:41:54.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:41:54 smithi115 ceph-mon[78593]: pgmap v6175: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:41:54.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:41:54 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:41:54.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:41:54 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:41:55.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:41:55 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:41:55.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:41:55 smithi115 conmon[90393]: 2022-05-14T02:41:55.320+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:41:55.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:41:55 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:41:56.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:41:56 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:41:56.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:41:56 smithi047 ceph-mon[77147]: pgmap v6176: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:41:56.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:41:56 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 11650 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:41:56.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:41:56 smithi115 conmon[90393]: 2022-05-14T02:41:56.366+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:41:56.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:41:56 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:41:56.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:41:56 smithi115 ceph-mon[78593]: pgmap v6176: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:41:56.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:41:56 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 11650 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:41:57.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:41:57 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:41:57.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:41:57 smithi115 conmon[90393]: 2022-05-14T02:41:57.337+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:41:57.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:41:57 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:41:58.546 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:41:58 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:41:58.546 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:41:58 smithi047 ceph-mon[77147]: pgmap v6177: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:41:58.547 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:41:58 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:41:58.547 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:41:58 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:41:58.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:41:58 smithi115 conmon[90393]: 2022-05-14T02:41:58.365+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:41:58.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:41:58 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:41:58.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:41:58 smithi115 ceph-mon[78593]: pgmap v6177: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:41:58.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:41:58 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:41:58.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:41:58 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:41:59.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:41:59 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:41:59.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:41:59 smithi115 conmon[90393]: 2022-05-14T02:41:59.336+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:41:59.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:41:59 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:42:00.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:42:00 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:42:00.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:42:00 smithi047 ceph-mon[77147]: pgmap v6178: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:42:00.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:42:00 smithi115 conmon[90393]: 2022-05-14T02:42:00.382+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:42:00.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:42:00 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:42:00.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:42:00 smithi115 ceph-mon[78593]: pgmap v6178: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:42:01.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:42:01 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:42:01.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:42:01 smithi115 conmon[90393]: 2022-05-14T02:42:01.394+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:42:01.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:42:01 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:42:02.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:42:02 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:42:02.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:42:02 smithi047 ceph-mon[77147]: pgmap v6179: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:42:02.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:42:02 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 11655 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:42:02.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:42:02 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:42:02.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:42:02 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:42:02.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:42:02 smithi115 conmon[90393]: 2022-05-14T02:42:02.401+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:42:02.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:42:02 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:42:02.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:42:02 smithi115 ceph-mon[78593]: pgmap v6179: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:42:02.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:42:02 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 11655 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:42:02.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:42:02 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:42:02.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:42:02 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:42:03.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:42:03 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:42:03.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:42:03 smithi115 conmon[90393]: 2022-05-14T02:42:03.416+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:42:03.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:42:03 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:42:04.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:42:04 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:42:04.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:42:04 smithi047 ceph-mon[77147]: pgmap v6180: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:42:04.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:42:04 smithi115 conmon[90393]: 2022-05-14T02:42:04.447+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:42:04.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:42:04 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:42:04.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:42:04 smithi115 ceph-mon[78593]: pgmap v6180: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:42:05.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:42:05 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:42:05.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:42:05 smithi115 conmon[90393]: 2022-05-14T02:42:05.419+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:42:05.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:42:05 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:42:06.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:42:06 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:42:06.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:42:06 smithi047 ceph-mon[77147]: pgmap v6181: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:42:06.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:42:06 smithi115 conmon[90393]: 2022-05-14T02:42:06.451+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:42:06.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:42:06 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:42:06.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:42:06 smithi115 ceph-mon[78593]: pgmap v6181: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:42:07.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:42:07 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:42:07.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:42:07 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:42:07.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:42:07 smithi115 conmon[90393]: 2022-05-14T02:42:07.448+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:42:08.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:42:08 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:42:08.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:42:08 smithi047 ceph-mon[77147]: pgmap v6182: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:42:08.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:42:08 smithi115 conmon[90393]: 2022-05-14T02:42:08.413+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:42:08.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:42:08 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:42:08.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:42:08 smithi115 ceph-mon[78593]: pgmap v6182: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:42:09.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:42:09 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:42:09.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:42:09 smithi115 conmon[90393]: 2022-05-14T02:42:09.368+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:42:09.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:42:09 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:42:10.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:42:10 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:42:10.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:42:10 smithi047 ceph-mon[77147]: pgmap v6183: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:42:10.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:42:10 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:42:10.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:42:10 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:42:10.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:42:10 smithi115 conmon[90393]: 2022-05-14T02:42:10.362+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:42:10.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:42:10 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:42:10.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:42:10 smithi115 ceph-mon[78593]: pgmap v6183: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:42:10.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:42:10 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:42:10.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:42:10 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:42:11.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:42:11 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:42:11.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:42:11 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 11660 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:42:11.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:42:11 smithi115 conmon[90393]: 2022-05-14T02:42:11.373+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:42:11.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:42:11 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:42:11.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:42:11 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 11660 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:42:12.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:42:12 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:42:12.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:42:12 smithi047 ceph-mon[77147]: pgmap v6184: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:42:12.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:42:12 smithi115 conmon[90393]: 2022-05-14T02:42:12.392+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:42:12.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:42:12 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:42:12.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:42:12 smithi115 ceph-mon[78593]: pgmap v6184: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:42:13.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:42:13 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:42:13.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:42:13 smithi115 conmon[90393]: 2022-05-14T02:42:13.383+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:42:13.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:42:13 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:42:14.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:42:14 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:42:14.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:42:14 smithi047 ceph-mon[77147]: pgmap v6185: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:42:14.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:42:14 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:42:14.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:42:14 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:42:14.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:42:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T02:42:14.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:42:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T02:42:14.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:42:14 smithi115 conmon[90393]: 2022-05-14T02:42:14.383+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:42:14.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:42:14 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:42:14.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:42:14 smithi115 ceph-mon[78593]: pgmap v6185: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:42:14.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:42:14 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:42:14.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:42:14 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:42:14.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:42:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T02:42:14.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:42:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T02:42:15.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:42:15 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:42:15.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:42:15 smithi115 conmon[90393]: 2022-05-14T02:42:15.360+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:42:15.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:42:15 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:42:16.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:42:16 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:42:16.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:42:16 smithi047 ceph-mon[77147]: pgmap v6186: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:42:16.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:42:16 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 11670 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:42:16.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:42:16 smithi115 conmon[90393]: 2022-05-14T02:42:16.394+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:42:16.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:42:16 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:42:16.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:42:16 smithi115 ceph-mon[78593]: pgmap v6186: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:42:16.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:42:16 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 11670 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:42:17.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:42:17 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:42:17.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:42:17 smithi115 conmon[90393]: 2022-05-14T02:42:17.417+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:42:17.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:42:17 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:42:18.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:42:18 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:42:18.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:42:18 smithi047 ceph-mon[77147]: pgmap v6187: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:42:18.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:42:18 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:42:18.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:42:18 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:42:18.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:42:18 smithi115 conmon[90393]: 2022-05-14T02:42:18.438+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:42:18.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:42:18 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:42:18.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:42:18 smithi115 ceph-mon[78593]: pgmap v6187: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:42:18.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:42:18 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:42:18.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:42:18 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:42:19.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:42:19 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:42:19.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:42:19 smithi115 conmon[90393]: 2022-05-14T02:42:19.391+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:42:19.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:42:19 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:42:20.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:42:20 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:42:20.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:42:20 smithi047 ceph-mon[77147]: pgmap v6188: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:42:20.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:42:20 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:42:20.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:42:20 smithi115 conmon[90393]: 2022-05-14T02:42:20.413+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:42:20.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:42:20 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:42:20.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:42:20 smithi115 ceph-mon[78593]: pgmap v6188: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:42:20.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:42:20 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:42:21.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:42:21 smithi115 conmon[90393]: 2022-05-14T02:42:21.374+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:42:22.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:42:21 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:42:22.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:42:21 smithi047 ceph-mon[77147]: pgmap v6189: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:42:22.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:42:21 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 11675 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:42:22.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:42:21 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:42:22.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:42:21 smithi115 ceph-mon[78593]: pgmap v6189: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:42:22.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:42:21 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 11675 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:42:22.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:42:22 smithi115 conmon[90393]: 2022-05-14T02:42:22.380+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:42:23.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:42:22 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:42:23.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:42:22 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:42:23.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:42:22 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:42:23.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:42:22 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:42:23.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:42:22 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:42:23.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:42:22 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:42:23.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:42:23 smithi115 conmon[90393]: 2022-05-14T02:42:23.371+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:42:24.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:42:23 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:42:24.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:42:23 smithi047 ceph-mon[77147]: pgmap v6190: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:42:24.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:42:23 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:42:24.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:42:23 smithi115 ceph-mon[78593]: pgmap v6190: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:42:24.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:42:24 smithi115 conmon[90393]: 2022-05-14T02:42:24.415+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:42:25.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:42:24 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:42:25.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:42:24 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:42:25.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:42:25 smithi115 conmon[90393]: 2022-05-14T02:42:25.454+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:42:26.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:42:25 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:42:26.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:42:25 smithi047 ceph-mon[77147]: pgmap v6191: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:42:26.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:42:25 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:42:26.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:42:25 smithi115 ceph-mon[78593]: pgmap v6191: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:42:26.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:42:26 smithi115 conmon[90393]: 2022-05-14T02:42:26.418+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:42:27.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:42:26 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:42:27.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:42:26 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:42:27.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:42:27 smithi115 conmon[90393]: 2022-05-14T02:42:27.404+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:42:28.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:42:27 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:42:28.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:42:27 smithi047 ceph-mon[77147]: pgmap v6192: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:42:28.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:42:27 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:42:28.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:42:27 smithi115 ceph-mon[78593]: pgmap v6192: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:42:28.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:42:28 smithi115 conmon[90393]: 2022-05-14T02:42:28.448+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:42:29.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:42:28 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:42:29.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:42:28 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:42:29.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:42:29 smithi115 conmon[90393]: 2022-05-14T02:42:29.413+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:42:30.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:42:29 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:42:30.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:42:29 smithi047 ceph-mon[77147]: pgmap v6193: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:42:30.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:42:29 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:42:30.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:42:29 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:42:30.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:42:29 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:42:30.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:42:29 smithi115 ceph-mon[78593]: pgmap v6193: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:42:30.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:42:29 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:42:30.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:42:29 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:42:30.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:42:30 smithi115 conmon[90393]: 2022-05-14T02:42:30.434+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:42:31.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:42:30 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:42:31.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:42:30 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 11680 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:42:31.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:42:30 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:42:31.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:42:30 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 11680 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:42:31.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:42:31 smithi115 conmon[90393]: 2022-05-14T02:42:31.401+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:42:32.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:42:31 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:42:32.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:42:31 smithi047 ceph-mon[77147]: pgmap v6194: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:42:32.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:42:31 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:42:32.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:42:31 smithi115 ceph-mon[78593]: pgmap v6194: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:42:32.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:42:32 smithi115 conmon[90393]: 2022-05-14T02:42:32.420+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:42:33.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:42:32 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:42:33.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:42:32 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:42:33.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:42:33 smithi115 conmon[90393]: 2022-05-14T02:42:33.409+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:42:34.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:42:33 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:42:34.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:42:33 smithi047 ceph-mon[77147]: pgmap v6195: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:42:34.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:42:33 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:42:34.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:42:33 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:42:34.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:42:33 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:42:34.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:42:33 smithi115 ceph-mon[78593]: pgmap v6195: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:42:34.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:42:33 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:42:34.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:42:33 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:42:34.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:42:34 smithi115 conmon[90393]: 2022-05-14T02:42:34.381+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:42:35.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:42:34 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:42:35.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:42:34 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:42:35.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:42:35 smithi115 conmon[90393]: 2022-05-14T02:42:35.393+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:42:36.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:42:35 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:42:36.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:42:35 smithi047 ceph-mon[77147]: pgmap v6196: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:42:36.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:42:35 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 11690 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:42:36.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:42:35 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:42:36.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:42:35 smithi115 ceph-mon[78593]: pgmap v6196: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:42:36.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:42:35 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 11690 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:42:36.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:42:36 smithi115 conmon[90393]: 2022-05-14T02:42:36.433+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:42:37.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:42:36 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:42:37.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:42:36 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:42:37.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:42:37 smithi115 conmon[90393]: 2022-05-14T02:42:37.427+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:42:38.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:42:37 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:42:38.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:42:37 smithi047 ceph-mon[77147]: pgmap v6197: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:42:38.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:42:37 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:42:38.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:42:37 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:42:38.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:42:37 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:42:38.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:42:37 smithi115 ceph-mon[78593]: pgmap v6197: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:42:38.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:42:37 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:42:38.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:42:37 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:42:38.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:42:38 smithi115 conmon[90393]: 2022-05-14T02:42:38.445+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:42:39.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:42:38 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:42:39.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:42:38 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:42:39.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:42:39 smithi115 conmon[90393]: 2022-05-14T02:42:39.456+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:42:40.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:42:39 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:42:40.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:42:39 smithi047 ceph-mon[77147]: pgmap v6198: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:42:40.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:42:39 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:42:40.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:42:39 smithi115 ceph-mon[78593]: pgmap v6198: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:42:40.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:42:40 smithi115 conmon[90393]: 2022-05-14T02:42:40.498+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:42:41.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:42:40 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:42:41.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:42:40 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T02:42:41.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:42:40 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T02:42:41.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:42:40 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T02:42:41.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:42:40 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:42:41.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:42:40 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T02:42:41.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:42:40 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T02:42:41.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:42:40 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T02:42:41.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:42:41 smithi115 conmon[90393]: 2022-05-14T02:42:41.544+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:42:42.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:42:41 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:42:42.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:42:41 smithi047 ceph-mon[77147]: pgmap v6199: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:42:42.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:42:41 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 11695 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:42:42.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:42:41 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:42:42.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:42:41 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:42:42.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:42:41 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:42:42.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:42:41 smithi115 ceph-mon[78593]: pgmap v6199: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:42:42.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:42:41 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 11695 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:42:42.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:42:41 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:42:42.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:42:41 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:42:42.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:42:42 smithi115 conmon[90393]: 2022-05-14T02:42:42.581+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:42:43.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:42:42 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:42:43.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:42:42 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:42:43.957 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:42:43 smithi115 conmon[90393]: 2022-05-14T02:42:43.609+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:42:44.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:42:43 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:42:44.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:42:43 smithi047 ceph-mon[77147]: pgmap v6200: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:42:44.292 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:42:44 smithi047 conmon[78521]: 2022-05-14T02:42:44.015+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:42:44.292 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:42:44 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T02:42:44.292 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:42:44 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:42:44.293 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:42:44 smithi047 conmon[78521]: yield 2022-05-14T02:42:44.293 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:42:44 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:42:44.293 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:42:44 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:42:44.293 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:42:44 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:42:44.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:42:44 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T02:42:44.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:42:44 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:42:44.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:42:44 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T02:42:44.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:42:44 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:42:44.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:42:44 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T02:42:44.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:42:44 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:42:44.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:42:44 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T02:42:44.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:42:44 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:42:44.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:42:44 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T02:42:44.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:42:44 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:42:44.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:42:44 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:42:44.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:42:44 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:42:44.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:42:43 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:42:44.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:42:43 smithi115 ceph-mon[78593]: pgmap v6200: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:42:44.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:42:44 smithi115 conmon[90393]: 2022-05-14T02:42:44.560+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:42:45.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:42:44 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:42:45.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:42:44 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:42:45.292 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T02:42:45.292 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:42:45.292 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T02:42:45.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:42:45.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:42:45.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:42:45.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T02:42:45.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:42:45.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T02:42:45.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:42:45.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T02:42:45.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:42:45.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T02:42:45.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:42:45.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T02:42:45.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:42:45.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:42:45.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:42:45.298 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:42:44 smithi047 ceph-mon[77147]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T02:42:45.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:42:44 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:42:45.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:42:44 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:42:45.358 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T02:42:45.359 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:42:45.359 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T02:42:45.359 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:42:45.359 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:42:45.359 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:42:45.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T02:42:45.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:42:45.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T02:42:45.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:42:45.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T02:42:45.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:42:45.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T02:42:45.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:42:45.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T02:42:45.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:42:45.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:42:45.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:42:45.362 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:42:44 smithi115 ceph-mon[78593]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T02:42:45.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:42:45 smithi115 conmon[90393]: 2022-05-14T02:42:45.591+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:42:46.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:42:45 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:42:46.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:42:45 smithi047 ceph-mon[77147]: pgmap v6201: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:42:46.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:42:45 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:42:46.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:42:45 smithi115 ceph-mon[78593]: pgmap v6201: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:42:46.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:42:46 smithi115 conmon[90393]: 2022-05-14T02:42:46.575+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:42:47.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:42:46 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:42:47.292 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:42:47 smithi047 conmon[78521]: 2022-05-14T02:42:47.087+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:42:47.292 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:42:47 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T02:42:47.292 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:42:47 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:42:47.293 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:42:47 smithi047 conmon[78521]: yield 2022-05-14T02:42:47.293 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:42:47 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:42:47.293 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:42:47 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:42:47.293 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:42:47 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:42:47.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:42:47 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T02:42:47.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:42:47 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:42:47.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:42:47 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T02:42:47.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:42:47 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:42:47.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:42:47 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T02:42:47.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:42:47 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:42:47.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:42:47 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T02:42:47.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:42:47 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:42:47.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:42:47 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T02:42:47.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:42:47 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:42:47.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:42:47 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:42:47.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:42:47 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:42:47.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:42:47 smithi047 conmon[78521]: 2022-05-14T02:42:47.088+0000 7f5344781700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T02:42:47.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:42:47 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T02:42:47.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:42:47 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:42:47.297 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:42:47 smithi047 conmon[78521]: yield 2022-05-14T02:42:47.297 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:42:47 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:42:47.297 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:42:47 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:42:47.297 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:42:47 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:42:47.298 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:42:47 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T02:42:47.298 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:42:47 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:42:47.298 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:42:47 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T02:42:47.298 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:42:47 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:42:47.298 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:42:47 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T02:42:47.299 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:42:47 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:42:47.299 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:42:47 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T02:42:47.299 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:42:47 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:42:47.299 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:42:47 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T02:42:47.300 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:42:47 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:42:47.300 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:42:47 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:42:47.300 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:42:47 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:42:47.300 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:42:47 smithi047 conmon[78521]: 2022-05-14T02:42:47.300 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:42:47 smithi047 conmon[78521]: During handling of the above exception, another exception occurred: 2022-05-14T02:42:47.301 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:42:47 smithi047 conmon[78521]: 2022-05-14T02:42:47.301 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:42:47 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T02:42:47.301 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:42:47 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T02:42:47.301 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:42:47 smithi047 conmon[78521]: return f(*arg) 2022-05-14T02:42:47.302 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:42:47 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T02:42:47.302 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:42:47 smithi047 conmon[78521]: self._write_client_files(client_files, host) 2022-05-14T02:42:47.302 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:42:47 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T02:42:47.302 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:42:47 smithi047 conmon[78521]: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T02:42:47.302 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:42:47 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T02:42:47.303 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:42:47 smithi047 conmon[78521]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T02:42:47.303 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:42:47 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T02:42:47.303 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:42:47 smithi047 conmon[78521]: return self.event_loop.get_result(coro) 2022-05-14T02:42:47.303 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:42:47 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T02:42:47.303 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:42:47 smithi047 conmon[78521]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T02:42:47.304 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:42:47 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T02:42:47.304 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:42:47 smithi047 conmon[78521]: return self.__get_result() 2022-05-14T02:42:47.304 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:42:47 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T02:42:47.304 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:42:47 smithi047 conmon[78521]: raise self._exception 2022-05-14T02:42:47.304 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:42:47 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T02:42:47.305 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:42:47 smithi047 conmon[78521]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T02:42:47.305 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:42:47 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T02:42:47.305 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:42:47 smithi047 conmon[78521]: conn = await self._remote_connection(host, addr) 2022-05-14T02:42:47.305 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:42:47 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T02:42:47.305 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:42:47 smithi047 conmon[78521]: raise 2022-05-14T02:42:47.306 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:42:47 smithi047 conmon[78521]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T02:42:47.306 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:42:47 smithi047 conmon[78521]: self.gen.throw(type, value, traceback) 2022-05-14T02:42:47.306 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:42:47 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T02:42:47.306 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:42:47 smithi047 conmon[78521]: raise OrchestratorError(msg) 2022-05-14T02:42:47.306 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:42:47 smithi047 conmon[78521]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:42:47.357 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:42:46 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:42:47.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:42:47 smithi115 conmon[90393]: 2022-05-14T02:42:47.558+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:42:48.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:42:47 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:42:48.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:42:47 smithi047 ceph-mon[77147]: pgmap v6202: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:42:48.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:42:47 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:42:48.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T02:42:48.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:42:48.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T02:42:48.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:42:48.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:42:48.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:42:48.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T02:42:48.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:42:48.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T02:42:48.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:42:48.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T02:42:48.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:42:48.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T02:42:48.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:42:48.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T02:42:48.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:42:48.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:42:48.298 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:42:48.298 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:42:47 smithi047 ceph-mon[77147]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T02:42:48.298 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T02:42:48.298 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:42:48.299 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T02:42:48.299 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:42:48.299 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:42:48.299 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:42:48.300 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T02:42:48.300 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:42:48.300 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T02:42:48.300 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:42:48.301 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T02:42:48.301 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:42:48.301 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T02:42:48.301 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:42:48.302 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T02:42:48.302 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:42:48.302 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:42:48.302 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:42:48.302 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T02:42:48.303 INFO:journalctl@ceph.mon.b.smithi047.stdout: During handling of the above exception, another exception occurred: 2022-05-14T02:42:48.303 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T02:42:48.303 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T02:42:48.304 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T02:42:48.304 INFO:journalctl@ceph.mon.b.smithi047.stdout: return f(*arg) 2022-05-14T02:42:48.304 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T02:42:48.304 INFO:journalctl@ceph.mon.b.smithi047.stdout: self._write_client_files(client_files, host) 2022-05-14T02:42:48.304 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T02:42:48.305 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T02:42:48.305 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T02:42:48.305 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T02:42:48.306 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T02:42:48.306 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.event_loop.get_result(coro) 2022-05-14T02:42:48.306 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T02:42:48.306 INFO:journalctl@ceph.mon.b.smithi047.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T02:42:48.306 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T02:42:48.307 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.__get_result() 2022-05-14T02:42:48.307 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T02:42:48.307 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise self._exception 2022-05-14T02:42:48.307 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T02:42:48.308 INFO:journalctl@ceph.mon.b.smithi047.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T02:42:48.308 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T02:42:48.308 INFO:journalctl@ceph.mon.b.smithi047.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T02:42:48.308 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T02:42:48.308 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise 2022-05-14T02:42:48.309 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T02:42:48.309 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.gen.throw(type, value, traceback) 2022-05-14T02:42:48.309 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T02:42:48.309 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OrchestratorError(msg) 2022-05-14T02:42:48.309 INFO:journalctl@ceph.mon.b.smithi047.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:42:48.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:42:47 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:42:48.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:42:47 smithi115 ceph-mon[78593]: pgmap v6202: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:42:48.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:42:47 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:42:48.359 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T02:42:48.359 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:42:48.359 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T02:42:48.359 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:42:48.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:42:48.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:42:48.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T02:42:48.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:42:48.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T02:42:48.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:42:48.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T02:42:48.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:42:48.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T02:42:48.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:42:48.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T02:42:48.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:42:48.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:42:48.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:42:48.362 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:42:47 smithi115 ceph-mon[78593]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T02:42:48.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T02:42:48.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:42:48.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T02:42:48.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:42:48.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:42:48.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:42:48.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T02:42:48.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:42:48.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T02:42:48.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:42:48.365 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T02:42:48.365 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:42:48.365 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T02:42:48.365 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:42:48.365 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T02:42:48.366 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:42:48.366 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:42:48.366 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:42:48.366 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T02:42:48.366 INFO:journalctl@ceph.mon.c.smithi115.stdout: During handling of the above exception, another exception occurred: 2022-05-14T02:42:48.367 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T02:42:48.367 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T02:42:48.367 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T02:42:48.367 INFO:journalctl@ceph.mon.c.smithi115.stdout: return f(*arg) 2022-05-14T02:42:48.367 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T02:42:48.368 INFO:journalctl@ceph.mon.c.smithi115.stdout: self._write_client_files(client_files, host) 2022-05-14T02:42:48.368 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T02:42:48.368 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T02:42:48.368 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T02:42:48.368 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T02:42:48.369 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T02:42:48.369 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.event_loop.get_result(coro) 2022-05-14T02:42:48.369 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T02:42:48.369 INFO:journalctl@ceph.mon.c.smithi115.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T02:42:48.370 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T02:42:48.370 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.__get_result() 2022-05-14T02:42:48.370 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T02:42:48.370 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise self._exception 2022-05-14T02:42:48.370 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T02:42:48.371 INFO:journalctl@ceph.mon.c.smithi115.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T02:42:48.371 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T02:42:48.371 INFO:journalctl@ceph.mon.c.smithi115.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T02:42:48.371 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T02:42:48.371 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise 2022-05-14T02:42:48.371 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T02:42:48.372 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.gen.throw(type, value, traceback) 2022-05-14T02:42:48.372 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T02:42:48.372 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OrchestratorError(msg) 2022-05-14T02:42:48.372 INFO:journalctl@ceph.mon.c.smithi115.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:42:48.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:42:48 smithi115 conmon[90393]: 2022-05-14T02:42:48.535+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:42:49.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:42:48 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:42:49.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:42:48 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:42:49.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:42:49 smithi115 conmon[90393]: 2022-05-14T02:42:49.518+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:42:50.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:42:49 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:42:50.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:42:49 smithi047 ceph-mon[77147]: pgmap v6203: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:42:50.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:42:49 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:42:50.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:42:49 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:42:50.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:42:49 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:42:50.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:42:49 smithi115 ceph-mon[78593]: pgmap v6203: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:42:50.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:42:49 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:42:50.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:42:49 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:42:50.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:42:50 smithi115 conmon[90393]: 2022-05-14T02:42:50.522+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:42:51.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:42:50 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:42:51.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:42:50 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 11700 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:42:51.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:42:50 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:42:51.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:42:50 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 11700 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:42:51.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:42:51 smithi115 conmon[90393]: 2022-05-14T02:42:51.554+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:42:52.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:42:51 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:42:52.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:42:51 smithi047 ceph-mon[77147]: pgmap v6204: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:42:52.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:42:51 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:42:52.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:42:51 smithi115 ceph-mon[78593]: pgmap v6204: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:42:52.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:42:52 smithi115 conmon[90393]: 2022-05-14T02:42:52.560+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:42:53.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:42:52 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:42:53.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:42:52 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:42:53.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:42:53 smithi115 conmon[90393]: 2022-05-14T02:42:53.559+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:42:54.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:42:53 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:42:54.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:42:53 smithi047 ceph-mon[77147]: pgmap v6205: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:42:54.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:42:53 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:42:54.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:42:53 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:42:54.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:42:53 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:42:54.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:42:53 smithi115 ceph-mon[78593]: pgmap v6205: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:42:54.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:42:53 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:42:54.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:42:53 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:42:54.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:42:54 smithi115 conmon[90393]: 2022-05-14T02:42:54.549+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:42:55.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:42:54 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:42:55.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:42:54 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:42:55.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:42:55 smithi115 conmon[90393]: 2022-05-14T02:42:55.593+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:42:56.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:42:55 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:42:56.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:42:55 smithi047 ceph-mon[77147]: pgmap v6206: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:42:56.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:42:55 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 11710 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:42:56.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:42:55 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:42:56.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:42:55 smithi115 ceph-mon[78593]: pgmap v6206: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:42:56.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:42:55 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 11710 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:42:56.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:42:56 smithi115 conmon[90393]: 2022-05-14T02:42:56.588+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:42:57.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:42:57 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:42:57.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:42:57 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:42:57.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:42:57 smithi115 conmon[90393]: 2022-05-14T02:42:57.562+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:42:58.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:42:58 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:42:58.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:42:58 smithi047 ceph-mon[77147]: pgmap v6207: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:42:58.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:42:58 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:42:58.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:42:58 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:42:58.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:42:58 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:42:58.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:42:58 smithi115 ceph-mon[78593]: pgmap v6207: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:42:58.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:42:58 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:42:58.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:42:58 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:42:58.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:42:58 smithi115 conmon[90393]: 2022-05-14T02:42:58.559+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:42:59.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:42:59 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:42:59.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:42:59 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:42:59.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:42:59 smithi115 conmon[90393]: 2022-05-14T02:42:59.555+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:43:00.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:43:00 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:43:00.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:43:00 smithi047 ceph-mon[77147]: pgmap v6208: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:43:00.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:43:00 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:43:00.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:43:00 smithi115 ceph-mon[78593]: pgmap v6208: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:43:00.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:43:00 smithi115 conmon[90393]: 2022-05-14T02:43:00.579+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:43:01.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:43:01 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:43:01.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:43:01 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:43:01.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:43:01 smithi115 conmon[90393]: 2022-05-14T02:43:01.591+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:43:02.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:43:02 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:43:02.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:43:02 smithi047 ceph-mon[77147]: pgmap v6209: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:43:02.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:43:02 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 11715 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:43:02.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:43:02 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:43:02.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:43:02 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:43:02.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:43:02 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:43:02.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:43:02 smithi115 ceph-mon[78593]: pgmap v6209: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:43:02.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:43:02 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 11715 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:43:02.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:43:02 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:43:02.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:43:02 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:43:02.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:43:02 smithi115 conmon[90393]: 2022-05-14T02:43:02.560+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:43:03.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:43:03 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:43:03.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:43:03 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:43:03.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:43:03 smithi115 conmon[90393]: 2022-05-14T02:43:03.606+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:43:04.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:43:04 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:43:04.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:43:04 smithi047 ceph-mon[77147]: pgmap v6210: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:43:04.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:43:04 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:43:04.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:43:04 smithi115 ceph-mon[78593]: pgmap v6210: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:43:04.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:43:04 smithi115 conmon[90393]: 2022-05-14T02:43:04.559+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:43:05.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:43:05 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:43:05.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:43:05 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:43:05.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:43:05 smithi115 conmon[90393]: 2022-05-14T02:43:05.520+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:43:06.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:43:06 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:43:06.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:43:06 smithi047 ceph-mon[77147]: pgmap v6211: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:43:06.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:43:06 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:43:06.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:43:06 smithi115 ceph-mon[78593]: pgmap v6211: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:43:06.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:43:06 smithi115 conmon[90393]: 2022-05-14T02:43:06.533+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:43:07.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:43:07 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:43:07.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:43:07 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:43:07.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:43:07 smithi115 conmon[90393]: 2022-05-14T02:43:07.511+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:43:08.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:43:08 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:43:08.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:43:08 smithi047 ceph-mon[77147]: pgmap v6212: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:43:08.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:43:08 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:43:08.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:43:08 smithi115 ceph-mon[78593]: pgmap v6212: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:43:08.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:43:08 smithi115 conmon[90393]: 2022-05-14T02:43:08.492+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:43:09.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:43:09 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:43:09.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:43:09 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:43:09.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:43:09 smithi115 conmon[90393]: 2022-05-14T02:43:09.513+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:43:10.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:43:10 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:43:10.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:43:10 smithi115 ceph-mon[78593]: pgmap v6213: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:43:10.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:43:10 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:43:10.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:43:10 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:43:10.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:43:10 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:43:10.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:43:10 smithi047 ceph-mon[77147]: pgmap v6213: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:43:10.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:43:10 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:43:10.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:43:10 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:43:10.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:43:10 smithi115 conmon[90393]: 2022-05-14T02:43:10.495+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:43:11.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:43:11 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:43:11.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:43:11 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 11720 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:43:11.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:43:11 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:43:11.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:43:11 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 11720 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:43:11.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:43:11 smithi115 conmon[90393]: 2022-05-14T02:43:11.501+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:43:12.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:43:12 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:43:12.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:43:12 smithi115 ceph-mon[78593]: pgmap v6214: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:43:12.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:43:12 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:43:12.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:43:12 smithi047 ceph-mon[77147]: pgmap v6214: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:43:12.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:43:12 smithi115 conmon[90393]: 2022-05-14T02:43:12.467+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:43:13.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:43:13 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:43:13.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:43:13 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:43:13.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:43:13 smithi115 conmon[90393]: 2022-05-14T02:43:13.417+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:43:14.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:43:14 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:43:14.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:43:14 smithi115 ceph-mon[78593]: pgmap v6215: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:43:14.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:43:14 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:43:14.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:43:14 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:43:14.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:43:14 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:43:14.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:43:14 smithi047 ceph-mon[77147]: pgmap v6215: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:43:14.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:43:14 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:43:14.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:43:14 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:43:14.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:43:14 smithi115 conmon[90393]: 2022-05-14T02:43:14.466+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:43:15.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:43:15 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:43:15.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:43:15 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T02:43:15.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:43:15 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T02:43:15.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:43:15 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:43:15.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:43:15 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T02:43:15.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:43:15 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T02:43:15.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:43:15 smithi115 conmon[90393]: 2022-05-14T02:43:15.490+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:43:16.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:43:16 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:43:16.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:43:16 smithi115 ceph-mon[78593]: pgmap v6216: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:43:16.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:43:16 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 11730 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:43:16.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:43:16 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:43:16.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:43:16 smithi047 ceph-mon[77147]: pgmap v6216: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:43:16.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:43:16 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 11730 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:43:16.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:43:16 smithi115 conmon[90393]: 2022-05-14T02:43:16.484+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:43:17.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:43:17 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:43:17.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:43:17 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:43:17.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:43:17 smithi115 conmon[90393]: 2022-05-14T02:43:17.480+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:43:18.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:43:18 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:43:18.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:43:18 smithi115 ceph-mon[78593]: pgmap v6217: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:43:18.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:43:18 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:43:18.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:43:18 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:43:18.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:43:18 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:43:18.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:43:18 smithi047 ceph-mon[77147]: pgmap v6217: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:43:18.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:43:18 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:43:18.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:43:18 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:43:18.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:43:18 smithi115 conmon[90393]: 2022-05-14T02:43:18.458+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:43:19.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:43:19 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:43:19.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:43:19 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:43:19.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:43:19 smithi115 conmon[90393]: 2022-05-14T02:43:19.428+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:43:20.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:43:20 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:43:20.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:43:20 smithi115 ceph-mon[78593]: pgmap v6218: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:43:20.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:43:20 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:43:20.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:43:20 smithi047 ceph-mon[77147]: pgmap v6218: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:43:20.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:43:20 smithi115 conmon[90393]: 2022-05-14T02:43:20.384+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:43:21.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:43:21 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:43:21.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:43:21 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:43:21.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:43:21 smithi115 conmon[90393]: 2022-05-14T02:43:21.366+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:43:22.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:43:22 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:43:22.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:43:22 smithi115 ceph-mon[78593]: pgmap v6219: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:43:22.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:43:22 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 11735 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:43:22.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:43:22 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:43:22.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:43:22 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:43:22.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:43:22 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:43:22.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:43:22 smithi047 ceph-mon[77147]: pgmap v6219: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:43:22.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:43:22 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 11735 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:43:22.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:43:22 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:43:22.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:43:22 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:43:22.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:43:22 smithi115 conmon[90393]: 2022-05-14T02:43:22.374+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:43:23.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:43:23 smithi115 conmon[90393]: 2022-05-14T02:43:23.328+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:43:23.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:43:23 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:43:23.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:43:23 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:43:24.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:43:24 smithi115 conmon[90393]: 2022-05-14T02:43:24.323+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:43:24.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:43:24 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:43:24.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:43:24 smithi115 ceph-mon[78593]: pgmap v6220: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:43:24.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:43:24 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:43:24.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:43:24 smithi047 ceph-mon[77147]: pgmap v6220: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:43:25.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:43:25 smithi115 conmon[90393]: 2022-05-14T02:43:25.280+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:43:25.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:43:25 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:43:25.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:43:25 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:43:26.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:43:26 smithi115 conmon[90393]: 2022-05-14T02:43:26.282+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:43:26.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:43:26 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:43:26.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:43:26 smithi115 ceph-mon[78593]: pgmap v6221: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:43:26.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:43:26 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:43:26.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:43:26 smithi047 ceph-mon[77147]: pgmap v6221: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:43:27.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:43:27 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:43:27.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:43:27 smithi115 conmon[90393]: 2022-05-14T02:43:27.236+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:43:27.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:43:27 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:43:28.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:43:28 smithi115 conmon[90393]: 2022-05-14T02:43:28.236+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:43:28.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:43:28 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:43:28.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:43:28 smithi115 ceph-mon[78593]: pgmap v6222: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:43:28.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:43:28 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:43:28.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:43:28 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:43:28.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:43:28 smithi047 ceph-mon[77147]: pgmap v6222: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:43:28.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:43:28 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:43:29.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:43:29 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:43:29.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:43:29 smithi115 conmon[90393]: 2022-05-14T02:43:29.188+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:43:29.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:43:29 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:43:30.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:43:30 smithi047 ceph-mon[77147]: pgmap v6223: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:43:30.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:43:30 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:43:30.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:43:30 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:43:30.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:43:30 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:43:30.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:43:30 smithi115 conmon[90393]: 2022-05-14T02:43:30.181+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:43:30.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:43:30 smithi115 ceph-mon[78593]: pgmap v6223: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:43:30.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:43:30 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:43:30.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:43:30 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:43:30.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:43:30 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:43:31.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:43:31 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:43:31.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:43:31 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 11740 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:43:31.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:43:31 smithi115 conmon[90393]: 2022-05-14T02:43:31.207+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:43:31.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:43:31 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:43:31.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:43:31 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 11740 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:43:32.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:43:32 smithi047 ceph-mon[77147]: pgmap v6224: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:43:32.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:43:32 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:43:32.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:43:32 smithi115 conmon[90393]: 2022-05-14T02:43:32.167+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:43:32.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:43:32 smithi115 ceph-mon[78593]: pgmap v6224: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:43:32.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:43:32 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:43:33.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:43:33 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:43:33.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:43:33 smithi115 conmon[90393]: 2022-05-14T02:43:33.191+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:43:33.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:43:33 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:43:34.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:43:34 smithi047 ceph-mon[77147]: pgmap v6225: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:43:34.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:43:34 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:43:34.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:43:34 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:43:34.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:43:34 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:43:34.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:43:34 smithi115 conmon[90393]: 2022-05-14T02:43:34.200+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:43:34.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:43:34 smithi115 ceph-mon[78593]: pgmap v6225: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:43:34.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:43:34 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:43:34.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:43:34 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:43:34.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:43:34 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:43:35.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:43:35 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:43:35.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:43:35 smithi115 conmon[90393]: 2022-05-14T02:43:35.181+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:43:35.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:43:35 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:43:36.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:43:36 smithi047 ceph-mon[77147]: pgmap v6226: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:43:36.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:43:36 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:43:36.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:43:36 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 11750 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:43:36.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:43:36 smithi115 conmon[90393]: 2022-05-14T02:43:36.209+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:43:36.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:43:36 smithi115 ceph-mon[78593]: pgmap v6226: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:43:36.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:43:36 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:43:36.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:43:36 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 11750 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:43:37.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:43:37 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:43:37.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:43:37 smithi115 conmon[90393]: 2022-05-14T02:43:37.180+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:43:37.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:43:37 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:43:38.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:43:38 smithi047 ceph-mon[77147]: pgmap v6227: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:43:38.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:43:38 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:43:38.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:43:38 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:43:38.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:43:38 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:43:38.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:43:38 smithi115 conmon[90393]: 2022-05-14T02:43:38.141+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:43:38.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:43:38 smithi115 ceph-mon[78593]: pgmap v6227: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:43:38.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:43:38 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:43:38.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:43:38 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:43:38.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:43:38 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:43:39.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:43:39 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:43:39.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:43:39 smithi115 conmon[90393]: 2022-05-14T02:43:39.107+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:43:39.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:43:39 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:43:40.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:43:40 smithi047 ceph-mon[77147]: pgmap v6228: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:43:40.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:43:40 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:43:40.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:43:40 smithi115 conmon[90393]: 2022-05-14T02:43:40.138+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:43:40.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:43:40 smithi115 ceph-mon[78593]: pgmap v6228: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:43:40.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:43:40 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:43:41.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:43:41 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:43:41.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:43:41 smithi115 conmon[90393]: 2022-05-14T02:43:41.139+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:43:41.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:43:41 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:43:42.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:43:42 smithi047 ceph-mon[77147]: pgmap v6229: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:43:42.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:43:42 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:43:42.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:43:42 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 11756 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:43:42.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:43:42 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:43:42.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:43:42 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:43:42.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:43:42 smithi115 conmon[90393]: 2022-05-14T02:43:42.120+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:43:42.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:43:42 smithi115 ceph-mon[78593]: pgmap v6229: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:43:42.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:43:42 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:43:42.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:43:42 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 11756 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:43:42.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:43:42 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:43:42.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:43:42 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:43:43.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:43:43 smithi115 conmon[90393]: 2022-05-14T02:43:43.077+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:43:43.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:43:43 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:43:43.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:43:43 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:43:44.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:43:44 smithi115 conmon[90393]: 2022-05-14T02:43:44.086+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:43:44.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:43:44 smithi115 ceph-mon[78593]: pgmap v6230: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:43:44.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:43:44 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:43:44.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:43:44 smithi047 ceph-mon[77147]: pgmap v6230: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:43:44.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:43:44 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:43:45.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:43:45 smithi115 conmon[90393]: 2022-05-14T02:43:45.037+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:43:45.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:43:45 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:43:45.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:43:45 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:43:46.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:43:46 smithi115 conmon[90393]: 2022-05-14T02:43:46.028+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:43:46.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:43:46 smithi115 ceph-mon[78593]: pgmap v6231: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:43:46.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:43:46 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:43:46.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:43:46 smithi047 ceph-mon[77147]: pgmap v6231: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:43:46.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:43:46 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:43:47.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:43:47 smithi115 conmon[90393]: 2022-05-14T02:43:47.036+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:43:47.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:43:47 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:43:47.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:43:47 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T02:43:47.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:43:47 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T02:43:47.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:43:47 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T02:43:47.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:43:47 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:43:47.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:43:47 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T02:43:47.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:43:47 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T02:43:47.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:43:47 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T02:43:48.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:43:48 smithi115 conmon[90393]: 2022-05-14T02:43:48.010+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:43:48.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:43:48 smithi115 ceph-mon[78593]: pgmap v6232: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:43:48.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:43:48 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:43:48.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:43:48 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' 2022-05-14T02:43:48.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:43:48 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' 2022-05-14T02:43:48.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:43:48 smithi047 ceph-mon[77147]: pgmap v6232: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:43:48.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:43:48 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:43:48.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:43:48 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' 2022-05-14T02:43:48.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:43:48 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' 2022-05-14T02:43:49.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:43:49 smithi115 conmon[90393]: 2022-05-14T02:43:49.016+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:43:49.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:43:49 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:43:49.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:43:49 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:43:50.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:43:50 smithi047 ceph-mon[77147]: pgmap v6233: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:43:50.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:43:50 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:43:50.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:43:50 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:43:50.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:43:50 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:43:50.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:43:50 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' 2022-05-14T02:43:50.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:43:50 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' 2022-05-14T02:43:50.198 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:43:50 smithi047 conmon[78521]: 2022-05-14T02:43:50.192+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:43:50.198 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:43:50 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T02:43:50.198 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:43:50 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:43:50.199 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:43:50 smithi047 conmon[78521]: yield 2022-05-14T02:43:50.199 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:43:50 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:43:50.199 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:43:50 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:43:50.199 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:43:50 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:43:50.199 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:43:50 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T02:43:50.200 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:43:50 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:43:50.200 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:43:50 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T02:43:50.200 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:43:50 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:43:50.201 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:43:50 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T02:43:50.201 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:43:50 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:43:50.201 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:43:50 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T02:43:50.201 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:43:50 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:43:50.201 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:43:50 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T02:43:50.202 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:43:50 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:43:50.202 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:43:50 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:43:50.202 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:43:50 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:43:50.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:43:49 smithi115 conmon[90393]: 2022-05-14T02:43:49.971+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:43:50.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:43:50 smithi115 ceph-mon[78593]: pgmap v6233: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:43:50.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:43:50 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:43:50.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:43:50 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:43:50.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:43:50 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:43:50.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:43:50 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' 2022-05-14T02:43:50.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:43:50 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' 2022-05-14T02:43:51.183 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:43:50 smithi115 conmon[90393]: 2022-05-14T02:43:50.927+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:43:51.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:43:51 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:43:51.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:43:51 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:43:51.542 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T02:43:51.542 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:43:51.542 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T02:43:51.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:43:51.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:43:51.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:43:51.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T02:43:51.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:43:51.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T02:43:51.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:43:51.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T02:43:51.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:43:51.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T02:43:51.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:43:51.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T02:43:51.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:43:51.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:43:51.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:43:51.547 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:43:51 smithi047 ceph-mon[77147]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T02:43:51.547 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:43:51 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-05-14T02:43:51.547 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:43:51 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-05-14T02:43:51.547 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:43:51 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-05-14T02:43:51.547 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:43:51 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-05-14T02:43:51.548 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:43:51 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-05-14T02:43:51.548 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:43:51 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-05-14T02:43:51.548 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:43:51 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-05-14T02:43:51.548 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:43:51 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-05-14T02:43:51.549 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:43:51 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 11760 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:43:51.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:43:51 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:43:51.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:43:51 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:43:51.609 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T02:43:51.609 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:43:51.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T02:43:51.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:43:51.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:43:51.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:43:51.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T02:43:51.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:43:51.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T02:43:51.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:43:51.613 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T02:43:51.613 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:43:51.613 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T02:43:51.614 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:43:51.614 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T02:43:51.614 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:43:51.615 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:43:51.615 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:43:51.615 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:43:51 smithi115 ceph-mon[78593]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T02:43:51.615 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:43:51 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-05-14T02:43:51.616 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:43:51 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-05-14T02:43:51.616 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:43:51 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-05-14T02:43:51.616 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:43:51 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-05-14T02:43:51.617 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:43:51 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-05-14T02:43:51.617 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:43:51 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-05-14T02:43:51.617 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:43:51 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-05-14T02:43:51.618 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:43:51 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-05-14T02:43:51.618 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:43:51 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 11760 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:43:52.182 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:43:51 smithi115 conmon[90393]: 2022-05-14T02:43:51.927+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:43:52.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:43:52 smithi047 ceph-mon[77147]: Adjusting osd_memory_target on smithi115 to 197.0M 2022-05-14T02:43:52.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:43:52 smithi047 ceph-mon[77147]: Unable to set osd_memory_target on smithi115 to 206629785: error parsing value: Value '206629785' is below minimum 939524096 2022-05-14T02:43:52.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:43:52 smithi047 ceph-mon[77147]: Adjusting osd_memory_target on smithi047 to 197.1M 2022-05-14T02:43:52.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:43:52 smithi047 ceph-mon[77147]: Unable to set osd_memory_target on smithi047 to 206682136: error parsing value: Value '206682136' is below minimum 939524096 2022-05-14T02:43:52.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:43:52 smithi047 ceph-mon[77147]: pgmap v6234: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:43:52.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:43:52 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:43:52.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:43:52 smithi115 ceph-mon[78593]: Adjusting osd_memory_target on smithi115 to 197.0M 2022-05-14T02:43:52.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:43:52 smithi115 ceph-mon[78593]: Unable to set osd_memory_target on smithi115 to 206629785: error parsing value: Value '206629785' is below minimum 939524096 2022-05-14T02:43:52.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:43:52 smithi115 ceph-mon[78593]: Adjusting osd_memory_target on smithi047 to 197.1M 2022-05-14T02:43:52.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:43:52 smithi115 ceph-mon[78593]: Unable to set osd_memory_target on smithi047 to 206682136: error parsing value: Value '206682136' is below minimum 939524096 2022-05-14T02:43:52.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:43:52 smithi115 ceph-mon[78593]: pgmap v6234: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:43:52.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:43:52 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:43:53.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:43:52 smithi115 conmon[90393]: 2022-05-14T02:43:52.951+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:43:53.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:43:53 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:43:53.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:43:53 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:43:53.542 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:43:53 smithi047 conmon[78521]: 2022-05-14T02:43:53.264+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:43:53.542 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:43:53 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T02:43:53.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:43:53 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:43:53.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:43:53 smithi047 conmon[78521]: yield 2022-05-14T02:43:53.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:43:53 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:43:53.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:43:53 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:43:53.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:43:53 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:43:53.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:43:53 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T02:43:53.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:43:53 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:43:53.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:43:53 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T02:43:53.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:43:53 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:43:53.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:43:53 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T02:43:53.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:43:53 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:43:53.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:43:53 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T02:43:53.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:43:53 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:43:53.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:43:53 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T02:43:53.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:43:53 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:43:53.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:43:53 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:43:53.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:43:53 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:43:53.547 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:43:53 smithi047 conmon[78521]: 2022-05-14T02:43:53.265+0000 7f5341f7c700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T02:43:53.547 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:43:53 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T02:43:53.547 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:43:53 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:43:53.547 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:43:53 smithi047 conmon[78521]: yield 2022-05-14T02:43:53.547 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:43:53 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:43:53.548 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:43:53 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:43:53.548 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:43:53 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:43:53.548 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:43:53 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T02:43:53.548 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:43:53 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:43:53.548 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:43:53 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T02:43:53.549 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:43:53 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:43:53.549 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:43:53 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T02:43:53.549 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:43:53 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:43:53.549 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:43:53 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T02:43:53.549 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:43:53 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:43:53.550 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:43:53 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T02:43:53.550 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:43:53 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:43:53.550 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:43:53 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:43:53.550 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:43:53 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:43:53.551 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:43:53 smithi047 conmon[78521]: 2022-05-14T02:43:53.551 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:43:53 smithi047 conmon[78521]: During handling of the above exception, another exception occurred: 2022-05-14T02:43:53.551 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:43:53 smithi047 conmon[78521]: 2022-05-14T02:43:53.551 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:43:53 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T02:43:53.552 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:43:53 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T02:43:53.552 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:43:53 smithi047 conmon[78521]: return f(*arg) 2022-05-14T02:43:53.552 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:43:53 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T02:43:53.552 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:43:53 smithi047 conmon[78521]: self._write_client_files(client_files, host) 2022-05-14T02:43:53.552 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:43:53 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T02:43:53.553 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:43:53 smithi047 conmon[78521]: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T02:43:53.553 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:43:53 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T02:43:53.553 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:43:53 smithi047 conmon[78521]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T02:43:53.553 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:43:53 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T02:43:53.553 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:43:53 smithi047 conmon[78521]: return self.event_loop.get_result(coro) 2022-05-14T02:43:53.554 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:43:53 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T02:43:53.554 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:43:53 smithi047 conmon[78521]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T02:43:53.554 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:43:53 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T02:43:53.554 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:43:53 smithi047 conmon[78521]: return self.__get_result() 2022-05-14T02:43:53.554 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:43:53 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T02:43:53.555 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:43:53 smithi047 conmon[78521]: raise self._exception 2022-05-14T02:43:53.555 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:43:53 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T02:43:53.555 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:43:53 smithi047 conmon[78521]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T02:43:53.555 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:43:53 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T02:43:53.555 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:43:53 smithi047 conmon[78521]: conn = await self._remote_connection(host, addr) 2022-05-14T02:43:53.556 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:43:53 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T02:43:53.556 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:43:53 smithi047 conmon[78521]: raise 2022-05-14T02:43:53.556 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:43:53 smithi047 conmon[78521]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T02:43:53.556 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:43:53 smithi047 conmon[78521]: self.gen.throw(type, value, traceback) 2022-05-14T02:43:53.557 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:43:53 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T02:43:53.557 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:43:53 smithi047 conmon[78521]: raise OrchestratorError(msg) 2022-05-14T02:43:53.557 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:43:53 smithi047 conmon[78521]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:43:54.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:43:53 smithi115 conmon[90393]: 2022-05-14T02:43:53.940+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:43:54.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:43:54 smithi115 ceph-mon[78593]: pgmap v6235: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:43:54.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:43:54 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:43:54.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:43:54 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:43:54.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:43:54 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:43:54.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:43:54 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:43:54.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T02:43:54.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:43:54.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T02:43:54.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:43:54.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:43:54.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:43:54.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T02:43:54.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:43:54.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T02:43:54.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:43:54.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T02:43:54.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:43:54.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T02:43:54.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:43:54.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T02:43:54.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:43:54.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:43:54.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:43:54.364 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:43:54 smithi115 ceph-mon[78593]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T02:43:54.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T02:43:54.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:43:54.365 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T02:43:54.365 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:43:54.365 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:43:54.365 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:43:54.366 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T02:43:54.366 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:43:54.366 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T02:43:54.366 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:43:54.367 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T02:43:54.367 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:43:54.368 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T02:43:54.368 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:43:54.368 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T02:43:54.368 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:43:54.368 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:43:54.369 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:43:54.369 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T02:43:54.369 INFO:journalctl@ceph.mon.c.smithi115.stdout: During handling of the above exception, another exception occurred: 2022-05-14T02:43:54.369 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T02:43:54.370 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T02:43:54.370 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T02:43:54.370 INFO:journalctl@ceph.mon.c.smithi115.stdout: return f(*arg) 2022-05-14T02:43:54.370 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T02:43:54.370 INFO:journalctl@ceph.mon.c.smithi115.stdout: self._write_client_files(client_files, host) 2022-05-14T02:43:54.371 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T02:43:54.371 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T02:43:54.371 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T02:43:54.371 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T02:43:54.372 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T02:43:54.372 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.event_loop.get_result(coro) 2022-05-14T02:43:54.372 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T02:43:54.372 INFO:journalctl@ceph.mon.c.smithi115.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T02:43:54.372 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T02:43:54.373 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.__get_result() 2022-05-14T02:43:54.373 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T02:43:54.373 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise self._exception 2022-05-14T02:43:54.373 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T02:43:54.374 INFO:journalctl@ceph.mon.c.smithi115.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T02:43:54.374 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T02:43:54.374 INFO:journalctl@ceph.mon.c.smithi115.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T02:43:54.374 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T02:43:54.374 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise 2022-05-14T02:43:54.375 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T02:43:54.375 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.gen.throw(type, value, traceback) 2022-05-14T02:43:54.375 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T02:43:54.375 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OrchestratorError(msg) 2022-05-14T02:43:54.375 INFO:journalctl@ceph.mon.c.smithi115.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:43:54.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:43:54 smithi047 ceph-mon[77147]: pgmap v6235: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:43:54.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:43:54 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:43:54.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:43:54 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:43:54.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:43:54 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:43:54.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:43:54 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:43:54.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T02:43:54.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:43:54.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T02:43:54.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:43:54.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:43:54.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:43:54.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T02:43:54.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:43:54.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T02:43:54.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:43:54.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T02:43:54.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:43:54.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T02:43:54.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:43:54.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T02:43:54.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:43:54.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:43:54.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:43:54.546 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:43:54 smithi047 ceph-mon[77147]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T02:43:54.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T02:43:54.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:43:54.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T02:43:54.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:43:54.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:43:54.548 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:43:54.548 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T02:43:54.548 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:43:54.548 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T02:43:54.549 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:43:54.549 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T02:43:54.549 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:43:54.549 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T02:43:54.549 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:43:54.550 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T02:43:54.550 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:43:54.550 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:43:54.550 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:43:54.550 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T02:43:54.551 INFO:journalctl@ceph.mon.b.smithi047.stdout: During handling of the above exception, another exception occurred: 2022-05-14T02:43:54.551 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T02:43:54.551 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T02:43:54.551 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T02:43:54.552 INFO:journalctl@ceph.mon.b.smithi047.stdout: return f(*arg) 2022-05-14T02:43:54.552 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T02:43:54.552 INFO:journalctl@ceph.mon.b.smithi047.stdout: self._write_client_files(client_files, host) 2022-05-14T02:43:54.552 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T02:43:54.552 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T02:43:54.553 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T02:43:54.553 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T02:43:54.553 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T02:43:54.553 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.event_loop.get_result(coro) 2022-05-14T02:43:54.553 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T02:43:54.554 INFO:journalctl@ceph.mon.b.smithi047.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T02:43:54.554 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T02:43:54.554 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.__get_result() 2022-05-14T02:43:54.554 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T02:43:54.555 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise self._exception 2022-05-14T02:43:54.555 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T02:43:54.555 INFO:journalctl@ceph.mon.b.smithi047.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T02:43:54.555 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T02:43:54.555 INFO:journalctl@ceph.mon.b.smithi047.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T02:43:54.556 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T02:43:54.556 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise 2022-05-14T02:43:54.556 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T02:43:54.556 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.gen.throw(type, value, traceback) 2022-05-14T02:43:54.556 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T02:43:54.557 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OrchestratorError(msg) 2022-05-14T02:43:54.557 INFO:journalctl@ceph.mon.b.smithi047.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:43:55.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:43:54 smithi115 conmon[90393]: 2022-05-14T02:43:54.964+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:43:55.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:43:55 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:43:55.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:43:55 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:43:56.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:43:55 smithi115 conmon[90393]: 2022-05-14T02:43:55.996+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:43:56.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:43:56 smithi115 ceph-mon[78593]: pgmap v6236: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:43:56.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:43:56 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:43:56.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:43:56 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 11770 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:43:56.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:43:56 smithi047 ceph-mon[77147]: pgmap v6236: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:43:56.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:43:56 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:43:56.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:43:56 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 11770 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:43:57.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:43:56 smithi115 conmon[90393]: 2022-05-14T02:43:56.997+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:43:57.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:43:57 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:43:57.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:43:57 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:43:58.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:43:58 smithi115 conmon[90393]: 2022-05-14T02:43:58.008+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:43:58.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:43:58 smithi115 ceph-mon[78593]: pgmap v6237: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:43:58.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:43:58 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:43:58.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:43:58 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:43:58.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:43:58 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:43:58.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:43:58 smithi047 ceph-mon[77147]: pgmap v6237: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:43:58.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:43:58 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:43:58.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:43:58 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:43:58.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:43:58 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:43:59.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:43:58 smithi115 conmon[90393]: 2022-05-14T02:43:58.977+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:43:59.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:43:59 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:43:59.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:43:59 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:44:00.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:43:59 smithi115 conmon[90393]: 2022-05-14T02:43:59.979+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:44:00.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:44:00 smithi115 ceph-mon[78593]: pgmap v6238: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:44:00.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:44:00 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:44:00.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:44:00 smithi047 ceph-mon[77147]: pgmap v6238: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:44:00.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:44:00 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:44:01.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:44:00 smithi115 conmon[90393]: 2022-05-14T02:44:00.966+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:44:01.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:44:01 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:44:01.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:44:01 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:44:02.215 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:44:01 smithi115 conmon[90393]: 2022-05-14T02:44:01.920+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:44:02.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:44:02 smithi047 ceph-mon[77147]: pgmap v6239: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:44:02.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:44:02 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:44:02.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:44:02 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 11775 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:44:02.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:44:02 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:44:02.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:44:02 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:44:02.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:44:02 smithi115 ceph-mon[78593]: pgmap v6239: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:44:02.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:44:02 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:44:02.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:44:02 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 11775 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:44:02.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:44:02 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:44:02.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:44:02 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:44:03.219 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:44:02 smithi115 conmon[90393]: 2022-05-14T02:44:02.966+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:44:03.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:44:03 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:44:03.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:44:03 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:44:04.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:44:04 smithi115 ceph-mon[78593]: pgmap v6240: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:44:04.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:44:04 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:44:04.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:44:03 smithi115 conmon[90393]: 2022-05-14T02:44:03.979+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:44:04.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:44:04 smithi047 ceph-mon[77147]: pgmap v6240: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:44:04.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:44:04 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:44:05.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:44:05 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:44:05.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:44:05 smithi115 conmon[90393]: 2022-05-14T02:44:05.010+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:44:05.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:44:05 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:44:06.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:44:06 smithi115 conmon[90393]: 2022-05-14T02:44:06.007+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:44:06.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:44:06 smithi115 ceph-mon[78593]: pgmap v6241: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:44:06.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:44:06 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:44:06.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:44:06 smithi047 ceph-mon[77147]: pgmap v6241: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:44:06.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:44:06 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:44:07.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:44:07 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:44:07.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:44:07 smithi115 conmon[90393]: 2022-05-14T02:44:07.006+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:44:07.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:44:07 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:44:08.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:44:08 smithi115 conmon[90393]: 2022-05-14T02:44:08.006+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:44:08.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:44:08 smithi115 ceph-mon[78593]: pgmap v6242: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:44:08.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:44:08 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:44:08.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:44:08 smithi047 ceph-mon[77147]: pgmap v6242: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:44:08.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:44:08 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:44:09.241 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:44:08 smithi115 conmon[90393]: 2022-05-14T02:44:08.971+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:44:09.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:44:09 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:44:09.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:44:09 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:44:10.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:44:10 smithi115 conmon[90393]: 2022-05-14T02:44:10.008+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:44:10.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:44:10 smithi115 ceph-mon[78593]: pgmap v6243: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:44:10.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:44:10 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:44:10.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:44:10 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:44:10.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:44:10 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:44:10.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:44:10 smithi047 ceph-mon[77147]: pgmap v6243: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:44:10.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:44:10 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:44:10.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:44:10 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:44:10.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:44:10 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:44:11.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:44:11 smithi115 conmon[90393]: 2022-05-14T02:44:11.017+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:44:11.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:44:11 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:44:11.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:44:11 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 11780 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:44:11.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:44:11 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:44:11.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:44:11 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 11780 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:44:12.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:44:12 smithi115 conmon[90393]: 2022-05-14T02:44:12.037+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:44:12.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:44:12 smithi115 ceph-mon[78593]: pgmap v6244: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:44:12.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:44:12 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:44:12.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:44:12 smithi047 ceph-mon[77147]: pgmap v6244: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:44:12.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:44:12 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:44:13.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:44:13 smithi115 conmon[90393]: 2022-05-14T02:44:13.009+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:44:13.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:44:13 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:44:13.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:44:13 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:44:14.255 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:44:13 smithi115 conmon[90393]: 2022-05-14T02:44:13.973+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:44:14.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:44:14 smithi047 ceph-mon[77147]: pgmap v6245: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:44:14.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:44:14 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:44:14.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:44:14 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:44:14.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:44:14 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:44:14.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:44:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T02:44:14.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:44:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T02:44:14.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:44:14 smithi115 ceph-mon[78593]: pgmap v6245: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:44:14.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:44:14 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:44:14.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:44:14 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:44:14.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:44:14 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:44:14.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:44:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T02:44:14.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:44:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T02:44:15.263 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:44:14 smithi115 conmon[90393]: 2022-05-14T02:44:14.941+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:44:15.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:44:15 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:44:15.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:44:15 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:44:16.262 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:44:15 smithi115 conmon[90393]: 2022-05-14T02:44:15.961+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:44:16.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:44:16 smithi047 ceph-mon[77147]: pgmap v6246: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:44:16.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:44:16 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:44:16.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:44:16 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 11790 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:44:16.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:44:16 smithi115 ceph-mon[78593]: pgmap v6246: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:44:16.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:44:16 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:44:16.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:44:16 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 11790 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:44:17.268 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:44:16 smithi115 conmon[90393]: 2022-05-14T02:44:16.958+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:44:17.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:44:17 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:44:17.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:44:17 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:44:18.268 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:44:17 smithi115 conmon[90393]: 2022-05-14T02:44:17.927+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:44:18.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:44:18 smithi047 ceph-mon[77147]: pgmap v6247: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:44:18.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:44:18 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:44:18.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:44:18 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:44:18.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:44:18 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:44:18.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:44:18 smithi115 ceph-mon[78593]: pgmap v6247: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:44:18.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:44:18 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:44:18.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:44:18 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:44:18.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:44:18 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:44:19.273 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:44:18 smithi115 conmon[90393]: 2022-05-14T02:44:18.881+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:44:19.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:44:19 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:44:19.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:44:19 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:44:20.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:44:19 smithi115 conmon[90393]: 2022-05-14T02:44:19.839+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:44:20.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:44:20 smithi047 ceph-mon[77147]: pgmap v6248: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:44:20.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:44:20 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:44:20.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:44:20 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:44:20.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:44:20 smithi115 ceph-mon[78593]: pgmap v6248: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:44:20.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:44:20 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:44:20.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:44:20 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:44:21.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:44:20 smithi115 conmon[90393]: 2022-05-14T02:44:20.796+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:44:22.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:44:21 smithi115 conmon[90393]: 2022-05-14T02:44:21.778+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:44:22.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:44:21 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:44:22.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:44:21 smithi115 ceph-mon[78593]: pgmap v6249: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:44:22.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:44:21 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 11795 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:44:22.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:44:21 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:44:22.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:44:21 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:44:22.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:44:21 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:44:22.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:44:21 smithi047 ceph-mon[77147]: pgmap v6249: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:44:22.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:44:21 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 11795 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:44:22.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:44:21 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:44:22.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:44:21 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:44:23.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:44:22 smithi115 conmon[90393]: 2022-05-14T02:44:22.754+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:44:23.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:44:22 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:44:23.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:44:22 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:44:24.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:44:23 smithi115 conmon[90393]: 2022-05-14T02:44:23.752+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:44:24.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:44:23 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:44:24.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:44:23 smithi115 ceph-mon[78593]: pgmap v6250: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:44:24.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:44:23 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:44:24.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:44:23 smithi047 ceph-mon[77147]: pgmap v6250: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:44:25.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:44:24 smithi115 conmon[90393]: 2022-05-14T02:44:24.709+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:44:25.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:44:24 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:44:25.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:44:24 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:44:25.918 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:44:25 smithi115 conmon[90393]: 2022-05-14T02:44:25.664+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:44:26.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:44:25 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:44:26.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:44:25 smithi047 ceph-mon[77147]: pgmap v6251: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:44:26.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:44:25 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:44:26.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:44:25 smithi115 ceph-mon[78593]: pgmap v6251: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:44:27.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:44:26 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:44:27.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:44:26 smithi115 conmon[90393]: 2022-05-14T02:44:26.689+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:44:27.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:44:26 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:44:27.924 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:44:27 smithi115 conmon[90393]: 2022-05-14T02:44:27.651+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:44:28.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:44:27 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:44:28.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:44:27 smithi047 ceph-mon[77147]: pgmap v6252: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:44:28.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:44:27 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:44:28.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:44:27 smithi115 ceph-mon[78593]: pgmap v6252: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:44:28.930 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:44:28 smithi115 conmon[90393]: 2022-05-14T02:44:28.663+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:44:29.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:44:28 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:44:29.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:44:28 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:44:29.931 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:44:29 smithi115 conmon[90393]: 2022-05-14T02:44:29.657+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:44:30.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:44:29 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:44:30.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:44:29 smithi047 ceph-mon[77147]: pgmap v6253: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:44:30.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:44:29 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:44:30.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:44:29 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:44:30.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:44:29 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:44:30.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:44:29 smithi115 ceph-mon[78593]: pgmap v6253: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:44:30.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:44:29 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:44:30.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:44:29 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:44:30.935 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:44:30 smithi115 conmon[90393]: 2022-05-14T02:44:30.632+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:44:31.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:44:30 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:44:31.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:44:30 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 11800 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:44:31.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:44:30 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:44:31.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:44:30 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 11800 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:44:31.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:44:31 smithi115 conmon[90393]: 2022-05-14T02:44:31.596+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:44:32.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:44:31 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:44:32.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:44:31 smithi047 ceph-mon[77147]: pgmap v6254: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:44:32.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:44:31 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:44:32.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:44:31 smithi115 ceph-mon[78593]: pgmap v6254: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:44:32.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:44:32 smithi115 conmon[90393]: 2022-05-14T02:44:32.597+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:44:33.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:44:32 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:44:33.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:44:32 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:44:33.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:44:33 smithi115 conmon[90393]: 2022-05-14T02:44:33.588+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:44:34.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:44:33 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:44:34.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:44:33 smithi047 ceph-mon[77147]: pgmap v6255: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:44:34.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:44:33 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:44:34.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:44:33 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:44:34.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:44:33 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:44:34.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:44:33 smithi115 ceph-mon[78593]: pgmap v6255: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:44:34.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:44:33 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:44:34.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:44:33 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:44:34.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:44:34 smithi115 conmon[90393]: 2022-05-14T02:44:34.573+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:44:35.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:44:34 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:44:35.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:44:34 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:44:35.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:44:35 smithi115 conmon[90393]: 2022-05-14T02:44:35.594+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:44:36.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:44:35 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:44:36.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:44:35 smithi047 ceph-mon[77147]: pgmap v6256: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:44:36.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:44:35 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 11810 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:44:36.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:44:35 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:44:36.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:44:35 smithi115 ceph-mon[78593]: pgmap v6256: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:44:36.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:44:35 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 11810 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:44:36.956 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:44:36 smithi115 conmon[90393]: 2022-05-14T02:44:36.616+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:44:37.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:44:36 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:44:37.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:44:36 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:44:37.957 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:44:37 smithi115 conmon[90393]: 2022-05-14T02:44:37.661+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:44:38.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:44:37 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:44:38.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:44:37 smithi047 ceph-mon[77147]: pgmap v6257: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:44:38.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:44:37 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:44:38.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:44:37 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:44:38.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:44:37 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:44:38.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:44:37 smithi115 ceph-mon[78593]: pgmap v6257: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:44:38.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:44:37 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:44:38.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:44:37 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:44:38.961 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:44:38 smithi115 conmon[90393]: 2022-05-14T02:44:38.653+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:44:39.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:44:38 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:44:39.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:44:38 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:44:39.963 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:44:39 smithi115 conmon[90393]: 2022-05-14T02:44:39.667+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:44:40.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:44:39 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:44:40.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:44:39 smithi047 ceph-mon[77147]: pgmap v6258: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:44:40.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:44:39 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:44:40.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:44:39 smithi115 ceph-mon[78593]: pgmap v6258: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:44:40.968 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:44:40 smithi115 conmon[90393]: 2022-05-14T02:44:40.635+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:44:41.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:44:40 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:44:41.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:44:40 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:44:41.969 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:44:41 smithi115 conmon[90393]: 2022-05-14T02:44:41.658+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:44:42.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:44:41 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:44:42.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:44:41 smithi047 ceph-mon[77147]: pgmap v6259: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:44:42.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:44:41 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 11815 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:44:42.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:44:41 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:44:42.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:44:41 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:44:42.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:44:41 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:44:42.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:44:41 smithi115 ceph-mon[78593]: pgmap v6259: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:44:42.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:44:41 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 11815 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:44:42.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:44:41 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:44:42.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:44:41 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:44:42.974 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:44:42 smithi115 conmon[90393]: 2022-05-14T02:44:42.683+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:44:43.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:44:42 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:44:43.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:44:42 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:44:43.976 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:44:43 smithi115 conmon[90393]: 2022-05-14T02:44:43.723+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:44:44.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:44:43 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:44:44.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:44:43 smithi047 ceph-mon[77147]: pgmap v6260: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:44:44.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:44:43 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:44:44.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:44:43 smithi115 ceph-mon[78593]: pgmap v6260: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:44:45.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:44:44 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:44:45.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:44:44 smithi115 conmon[90393]: 2022-05-14T02:44:44.749+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:44:45.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:44:44 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:44:46.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:44:45 smithi115 conmon[90393]: 2022-05-14T02:44:45.779+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:44:46.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:44:45 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:44:46.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:44:45 smithi115 ceph-mon[78593]: pgmap v6261: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:44:46.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:44:45 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:44:46.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:44:45 smithi047 ceph-mon[77147]: pgmap v6261: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:44:47.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:44:46 smithi115 conmon[90393]: 2022-05-14T02:44:46.795+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:44:47.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:44:46 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:44:47.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:44:46 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:44:48.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:44:47 smithi115 conmon[90393]: 2022-05-14T02:44:47.803+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:44:48.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:44:47 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:44:48.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:44:47 smithi115 ceph-mon[78593]: pgmap v6262: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:44:48.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:44:47 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:44:48.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:44:47 smithi047 ceph-mon[77147]: pgmap v6262: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:44:49.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:44:48 smithi115 conmon[90393]: 2022-05-14T02:44:48.821+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:44:49.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:44:48 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:44:49.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:44:48 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:44:50.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:44:49 smithi115 conmon[90393]: 2022-05-14T02:44:49.810+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:44:50.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:44:49 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:44:50.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:44:49 smithi115 ceph-mon[78593]: pgmap v6263: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:44:50.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:44:49 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:44:50.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:44:49 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:44:50.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:44:49 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:44:50.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:44:49 smithi047 ceph-mon[77147]: pgmap v6263: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:44:50.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:44:49 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:44:50.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:44:49 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:44:51.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:44:50 smithi115 conmon[90393]: 2022-05-14T02:44:50.853+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:44:51.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:44:50 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:44:51.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:44:50 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 11820 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:44:51.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:44:50 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:44:51.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:44:50 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 11820 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:44:52.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:44:52 smithi047 ceph-mon[77147]: pgmap v6264: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:44:52.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:44:52 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:44:52.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:44:51 smithi115 conmon[90393]: 2022-05-14T02:44:51.887+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:44:52.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:44:52 smithi115 ceph-mon[78593]: pgmap v6264: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:44:52.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:44:52 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:44:53.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:44:53 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:44:53.314 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:44:53 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:44:53.314 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:44:52 smithi115 conmon[90393]: 2022-05-14T02:44:52.872+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:44:54.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:44:54 smithi047 ceph-mon[77147]: pgmap v6265: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:44:54.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:44:54 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:44:54.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:44:54 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:44:54.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:44:54 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:44:54.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:44:54 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T02:44:54.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:44:54 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T02:44:54.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:44:54 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T02:44:54.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:44:53 smithi047 conmon[78521]: 2022-05-14T02:44:53.871+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:44:54.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:44:53 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T02:44:54.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:44:53 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:44:54.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:44:53 smithi047 conmon[78521]: yield 2022-05-14T02:44:54.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:44:53 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:44:54.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:44:53 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:44:54.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:44:53 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:44:54.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:44:53 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T02:44:54.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:44:53 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:44:54.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:44:53 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T02:44:54.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:44:53 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:44:54.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:44:53 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T02:44:54.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:44:53 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:44:54.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:44:53 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T02:44:54.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:44:53 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:44:54.048 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:44:53 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T02:44:54.049 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:44:53 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:44:54.049 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:44:53 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:44:54.049 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:44:53 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:44:54.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:44:53 smithi115 conmon[90393]: 2022-05-14T02:44:53.842+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:44:54.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:44:54 smithi115 ceph-mon[78593]: pgmap v6265: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:44:54.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:44:54 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:44:54.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:44:54 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:44:54.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:44:54 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:44:54.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:44:54 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T02:44:54.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:44:54 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T02:44:54.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:44:54 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T02:44:55.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:44:55 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:44:55.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:44:55 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:44:55.292 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T02:44:55.292 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:44:55.292 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T02:44:55.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:44:55.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:44:55.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:44:55.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T02:44:55.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:44:55.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T02:44:55.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:44:55.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T02:44:55.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:44:55.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T02:44:55.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:44:55.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T02:44:55.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:44:55.298 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:44:55.298 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:44:55.298 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:44:55 smithi047 ceph-mon[77147]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T02:44:55.360 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:44:54 smithi115 conmon[90393]: 2022-05-14T02:44:54.873+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:44:55.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:44:55 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:44:55.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:44:55 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:44:55.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T02:44:55.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:44:55.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T02:44:55.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:44:55.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:44:55.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:44:55.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T02:44:55.365 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:44:55.365 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T02:44:55.365 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:44:55.366 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T02:44:55.366 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:44:55.367 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T02:44:55.367 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:44:55.368 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T02:44:55.368 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:44:55.369 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:44:55.369 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:44:55.370 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:44:55 smithi115 ceph-mon[78593]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T02:44:56.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:44:56 smithi047 ceph-mon[77147]: pgmap v6266: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:44:56.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:44:56 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:44:56.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:44:56 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 11830 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:44:56.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:44:55 smithi115 conmon[90393]: 2022-05-14T02:44:55.886+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:44:56.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:44:56 smithi115 ceph-mon[78593]: pgmap v6266: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:44:56.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:44:56 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:44:56.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:44:56 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 11830 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:44:57.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:44:57 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:44:57.293 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:44:56 smithi047 conmon[78521]: 2022-05-14T02:44:56.943+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:44:57.293 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:44:56 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T02:44:57.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:44:56 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:44:57.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:44:56 smithi047 conmon[78521]: yield 2022-05-14T02:44:57.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:44:56 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:44:57.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:44:56 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:44:57.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:44:56 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:44:57.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:44:56 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T02:44:57.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:44:56 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:44:57.297 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:44:56 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T02:44:57.297 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:44:56 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:44:57.298 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:44:56 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T02:44:57.298 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:44:56 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:44:57.299 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:44:56 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T02:44:57.299 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:44:56 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:44:57.299 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:44:56 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T02:44:57.300 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:44:56 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:44:57.300 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:44:56 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:44:57.301 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:44:56 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:44:57.301 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:44:56 smithi047 conmon[78521]: 2022-05-14T02:44:56.944+0000 7f5345f84700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T02:44:57.302 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:44:56 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T02:44:57.302 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:44:56 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:44:57.302 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:44:56 smithi047 conmon[78521]: yield 2022-05-14T02:44:57.303 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:44:56 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:44:57.303 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:44:56 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:44:57.304 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:44:56 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:44:57.304 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:44:56 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T02:44:57.304 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:44:56 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:44:57.305 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:44:56 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T02:44:57.305 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:44:56 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:44:57.306 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:44:56 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T02:44:57.306 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:44:56 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:44:57.306 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:44:56 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T02:44:57.307 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:44:56 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:44:57.307 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:44:56 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T02:44:57.308 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:44:56 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:44:57.308 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:44:56 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:44:57.308 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:44:56 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:44:57.309 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:44:56 smithi047 conmon[78521]: 2022-05-14T02:44:57.309 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:44:56 smithi047 conmon[78521]: During handling of the above exception, another exception occurred: 2022-05-14T02:44:57.309 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:44:56 smithi047 conmon[78521]: 2022-05-14T02:44:57.310 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:44:56 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T02:44:57.310 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:44:56 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T02:44:57.311 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:44:56 smithi047 conmon[78521]: return f(*arg) 2022-05-14T02:44:57.311 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:44:56 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T02:44:57.311 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:44:56 smithi047 conmon[78521]: self._write_client_files(client_files, host) 2022-05-14T02:44:57.312 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:44:56 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T02:44:57.312 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:44:56 smithi047 conmon[78521]: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T02:44:57.313 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:44:56 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T02:44:57.313 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:44:56 smithi047 conmon[78521]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T02:44:57.313 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:44:56 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T02:44:57.314 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:44:56 smithi047 conmon[78521]: return self.event_loop.get_result(coro) 2022-05-14T02:44:57.314 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:44:56 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T02:44:57.314 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:44:56 smithi047 conmon[78521]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T02:44:57.315 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:44:56 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T02:44:57.315 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:44:56 smithi047 conmon[78521]: return self.__get_result() 2022-05-14T02:44:57.316 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:44:56 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T02:44:57.316 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:44:56 smithi047 conmon[78521]: raise self._exception 2022-05-14T02:44:57.317 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:44:56 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T02:44:57.317 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:44:56 smithi047 conmon[78521]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T02:44:57.317 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:44:56 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T02:44:57.318 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:44:56 smithi047 conmon[78521]: conn = await self._remote_connection(host, addr) 2022-05-14T02:44:57.318 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:44:56 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T02:44:57.319 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:44:56 smithi047 conmon[78521]: raise 2022-05-14T02:44:57.319 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:44:56 smithi047 conmon[78521]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T02:44:57.319 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:44:56 smithi047 conmon[78521]: self.gen.throw(type, value, traceback) 2022-05-14T02:44:57.320 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:44:56 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T02:44:57.320 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:44:56 smithi047 conmon[78521]: raise OrchestratorError(msg) 2022-05-14T02:44:57.320 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:44:56 smithi047 conmon[78521]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:44:57.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:44:56 smithi115 conmon[90393]: 2022-05-14T02:44:56.877+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:44:57.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:44:57 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:44:58.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:44:57 smithi115 conmon[90393]: 2022-05-14T02:44:57.840+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:44:58.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:44:58 smithi115 ceph-mon[78593]: pgmap v6267: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:44:58.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:44:58 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:44:58.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:44:58 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:44:58.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T02:44:58.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:44:58.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T02:44:58.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:44:58.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:44:58.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:44:58.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T02:44:58.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:44:58.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T02:44:58.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:44:58.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T02:44:58.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:44:58.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T02:44:58.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:44:58.113 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T02:44:58.113 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:44:58.113 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:44:58.113 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:44:58.114 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:44:58 smithi115 ceph-mon[78593]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T02:44:58.114 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T02:44:58.114 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:44:58.114 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T02:44:58.115 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:44:58.115 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:44:58.115 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:44:58.115 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T02:44:58.116 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:44:58.116 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T02:44:58.116 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:44:58.116 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T02:44:58.116 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:44:58.117 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T02:44:58.117 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:44:58.117 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T02:44:58.117 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:44:58.118 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:44:58.118 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:44:58.118 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T02:44:58.118 INFO:journalctl@ceph.mon.c.smithi115.stdout: During handling of the above exception, another exception occurred: 2022-05-14T02:44:58.119 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T02:44:58.119 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T02:44:58.119 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T02:44:58.119 INFO:journalctl@ceph.mon.c.smithi115.stdout: return f(*arg) 2022-05-14T02:44:58.119 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T02:44:58.120 INFO:journalctl@ceph.mon.c.smithi115.stdout: self._write_client_files(client_files, host) 2022-05-14T02:44:58.120 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T02:44:58.120 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T02:44:58.120 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T02:44:58.121 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T02:44:58.121 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T02:44:58.121 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.event_loop.get_result(coro) 2022-05-14T02:44:58.121 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T02:44:58.121 INFO:journalctl@ceph.mon.c.smithi115.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T02:44:58.122 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T02:44:58.122 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.__get_result() 2022-05-14T02:44:58.122 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T02:44:58.122 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise self._exception 2022-05-14T02:44:58.123 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T02:44:58.123 INFO:journalctl@ceph.mon.c.smithi115.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T02:44:58.123 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T02:44:58.123 INFO:journalctl@ceph.mon.c.smithi115.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T02:44:58.124 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T02:44:58.124 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise 2022-05-14T02:44:58.124 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T02:44:58.124 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.gen.throw(type, value, traceback) 2022-05-14T02:44:58.124 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T02:44:58.125 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OrchestratorError(msg) 2022-05-14T02:44:58.125 INFO:journalctl@ceph.mon.c.smithi115.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:44:58.125 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:44:58 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:44:58.125 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:44:58 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:44:58.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:44:58 smithi047 ceph-mon[77147]: pgmap v6267: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:44:58.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:44:58 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:44:58.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:44:58 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:44:58.292 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T02:44:58.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:44:58.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T02:44:58.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:44:58.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:44:58.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:44:58.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T02:44:58.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:44:58.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T02:44:58.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:44:58.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T02:44:58.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:44:58.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T02:44:58.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:44:58.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T02:44:58.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:44:58.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:44:58.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:44:58.296 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:44:58 smithi047 ceph-mon[77147]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T02:44:58.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T02:44:58.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:44:58.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T02:44:58.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:44:58.298 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:44:58.298 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:44:58.298 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T02:44:58.298 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:44:58.298 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T02:44:58.299 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:44:58.299 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T02:44:58.299 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:44:58.299 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T02:44:58.299 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:44:58.300 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T02:44:58.300 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:44:58.300 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:44:58.300 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:44:58.301 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T02:44:58.301 INFO:journalctl@ceph.mon.b.smithi047.stdout: During handling of the above exception, another exception occurred: 2022-05-14T02:44:58.301 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T02:44:58.301 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T02:44:58.301 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T02:44:58.302 INFO:journalctl@ceph.mon.b.smithi047.stdout: return f(*arg) 2022-05-14T02:44:58.302 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T02:44:58.302 INFO:journalctl@ceph.mon.b.smithi047.stdout: self._write_client_files(client_files, host) 2022-05-14T02:44:58.302 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T02:44:58.303 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T02:44:58.303 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T02:44:58.303 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T02:44:58.303 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T02:44:58.303 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.event_loop.get_result(coro) 2022-05-14T02:44:58.304 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T02:44:58.304 INFO:journalctl@ceph.mon.b.smithi047.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T02:44:58.304 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T02:44:58.304 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.__get_result() 2022-05-14T02:44:58.305 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T02:44:58.305 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise self._exception 2022-05-14T02:44:58.305 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T02:44:58.306 INFO:journalctl@ceph.mon.b.smithi047.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T02:44:58.306 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T02:44:58.307 INFO:journalctl@ceph.mon.b.smithi047.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T02:44:58.307 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T02:44:58.308 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise 2022-05-14T02:44:58.308 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T02:44:58.308 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.gen.throw(type, value, traceback) 2022-05-14T02:44:58.309 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T02:44:58.309 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OrchestratorError(msg) 2022-05-14T02:44:58.309 INFO:journalctl@ceph.mon.b.smithi047.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:44:58.310 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:44:58 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:44:58.310 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:44:58 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:44:59.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:44:58 smithi115 conmon[90393]: 2022-05-14T02:44:58.816+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:44:59.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:44:59 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:44:59.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:44:59 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:45:00.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:44:59 smithi115 conmon[90393]: 2022-05-14T02:44:59.823+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:45:00.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:45:00 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:45:00.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:45:00 smithi115 ceph-mon[78593]: pgmap v6268: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:45:00.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:45:00 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:45:00.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:45:00 smithi047 ceph-mon[77147]: pgmap v6268: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:45:01.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:45:00 smithi115 conmon[90393]: 2022-05-14T02:45:00.822+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:45:01.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:45:01 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:45:01.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:45:01 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:45:02.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:45:02 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:45:02.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:45:02 smithi047 ceph-mon[77147]: pgmap v6269: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:45:02.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:45:02 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 11835 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:45:02.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:45:02 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:45:02.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:45:02 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:45:02.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:45:01 smithi115 conmon[90393]: 2022-05-14T02:45:01.867+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:45:02.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:45:02 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:45:02.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:45:02 smithi115 ceph-mon[78593]: pgmap v6269: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:45:02.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:45:02 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 11835 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:45:02.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:45:02 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:45:02.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:45:02 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:45:03.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:45:02 smithi115 conmon[90393]: 2022-05-14T02:45:02.831+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:45:03.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:45:03 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:45:03.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:45:03 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:45:04.040 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:45:03 smithi115 conmon[90393]: 2022-05-14T02:45:03.783+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:45:04.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:45:04 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:45:04.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:45:04 smithi047 ceph-mon[77147]: pgmap v6270: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:45:04.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:45:04 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:45:04.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:45:04 smithi115 ceph-mon[78593]: pgmap v6270: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:45:05.045 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:45:04 smithi115 conmon[90393]: 2022-05-14T02:45:04.759+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:45:05.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:45:05 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:45:05.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:45:05 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:45:06.046 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:45:05 smithi115 conmon[90393]: 2022-05-14T02:45:05.775+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:45:06.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:45:06 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:45:06.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:45:06 smithi115 ceph-mon[78593]: pgmap v6271: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:45:06.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:45:06 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:45:06.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:45:06 smithi047 ceph-mon[77147]: pgmap v6271: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:45:07.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:45:07 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:45:07.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:45:06 smithi115 conmon[90393]: 2022-05-14T02:45:06.807+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:45:07.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:45:07 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:45:08.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:45:07 smithi115 conmon[90393]: 2022-05-14T02:45:07.844+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:45:08.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:45:08 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:45:08.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:45:08 smithi115 ceph-mon[78593]: pgmap v6272: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:45:08.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:45:08 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:45:08.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:45:08 smithi047 ceph-mon[77147]: pgmap v6272: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:45:09.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:45:09 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:45:09.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:45:08 smithi115 conmon[90393]: 2022-05-14T02:45:08.818+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:45:09.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:45:09 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:45:10.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:45:09 smithi115 conmon[90393]: 2022-05-14T02:45:09.831+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:45:10.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:45:10 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:45:10.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:45:10 smithi115 ceph-mon[78593]: pgmap v6273: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:45:10.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:45:10 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:45:10.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:45:10 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:45:10.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:45:10 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:45:10.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:45:10 smithi047 ceph-mon[77147]: pgmap v6273: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:45:10.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:45:10 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:45:10.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:45:10 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:45:11.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:45:10 smithi115 conmon[90393]: 2022-05-14T02:45:10.879+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:45:11.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:45:11 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:45:11.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:45:11 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 11840 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:45:11.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:45:11 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:45:11.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:45:11 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 11840 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:45:12.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:45:11 smithi115 conmon[90393]: 2022-05-14T02:45:11.923+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:45:12.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:45:12 smithi115 ceph-mon[78593]: pgmap v6274: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:45:12.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:45:12 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:45:12.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:45:12 smithi047 ceph-mon[77147]: pgmap v6274: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:45:12.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:45:12 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:45:13.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:45:13 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:45:13.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:45:12 smithi115 conmon[90393]: 2022-05-14T02:45:12.900+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:45:13.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:45:13 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:45:14.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:45:13 smithi115 conmon[90393]: 2022-05-14T02:45:13.860+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:45:14.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:45:14 smithi115 ceph-mon[78593]: pgmap v6275: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:45:14.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:45:14 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:45:14.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:45:14 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:45:14.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:45:14 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:45:14.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:45:14 smithi047 ceph-mon[77147]: pgmap v6275: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:45:14.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:45:14 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:45:14.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:45:14 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:45:14.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:45:14 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:45:15.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:45:14 smithi115 conmon[90393]: 2022-05-14T02:45:14.841+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:45:15.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:45:15 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:45:15.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:45:15 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T02:45:15.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:45:15 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T02:45:15.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:45:15 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:45:15.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:45:15 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T02:45:15.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:45:15 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T02:45:16.079 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:45:15 smithi115 conmon[90393]: 2022-05-14T02:45:15.811+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:45:16.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:45:16 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:45:16.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:45:16 smithi115 ceph-mon[78593]: pgmap v6276: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:45:16.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:45:16 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 11850 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:45:16.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:45:16 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:45:16.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:45:16 smithi047 ceph-mon[77147]: pgmap v6276: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:45:16.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:45:16 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 11850 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:45:17.084 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:45:16 smithi115 conmon[90393]: 2022-05-14T02:45:16.819+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:45:17.085 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:45:17 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:45:17.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:45:17 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:45:18.085 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:45:17 smithi115 conmon[90393]: 2022-05-14T02:45:17.798+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:45:18.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:45:18 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:45:18.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:45:18 smithi115 ceph-mon[78593]: pgmap v6277: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:45:18.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:45:18 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:45:18.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:45:18 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:45:18.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:45:18 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:45:18.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:45:18 smithi047 ceph-mon[77147]: pgmap v6277: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:45:18.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:45:18 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:45:18.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:45:18 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:45:19.090 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:45:18 smithi115 conmon[90393]: 2022-05-14T02:45:18.826+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:45:19.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:45:19 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:45:19.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:45:19 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:45:20.092 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:45:19 smithi115 conmon[90393]: 2022-05-14T02:45:19.820+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:45:20.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:45:20 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:45:20.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:45:20 smithi115 ceph-mon[78593]: pgmap v6278: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:45:20.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:45:20 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:45:20.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:45:20 smithi047 ceph-mon[77147]: pgmap v6278: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:45:21.097 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:45:20 smithi115 conmon[90393]: 2022-05-14T02:45:20.844+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:45:21.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:45:21 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:45:21.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:45:21 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:45:22.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:45:21 smithi115 conmon[90393]: 2022-05-14T02:45:21.868+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:45:22.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:45:22 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:45:22.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:45:22 smithi115 ceph-mon[78593]: pgmap v6279: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:45:22.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:45:22 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 11855 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:45:22.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:45:22 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:45:22.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:45:22 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:45:22.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:45:22 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:45:22.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:45:22 smithi047 ceph-mon[77147]: pgmap v6279: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:45:22.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:45:22 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 11855 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:45:22.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:45:22 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:45:22.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:45:22 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:45:23.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:45:23 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:45:23.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:45:22 smithi115 conmon[90393]: 2022-05-14T02:45:22.910+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:45:23.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:45:23 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:45:24.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:45:23 smithi115 conmon[90393]: 2022-05-14T02:45:23.880+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:45:24.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:45:24 smithi115 ceph-mon[78593]: pgmap v6280: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:45:24.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:45:24 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:45:24.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:45:24 smithi047 ceph-mon[77147]: pgmap v6280: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:45:24.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:45:24 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:45:25.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:45:25 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:45:25.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:45:24 smithi115 conmon[90393]: 2022-05-14T02:45:24.903+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:45:25.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:45:25 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:45:26.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:45:25 smithi115 conmon[90393]: 2022-05-14T02:45:25.945+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:45:26.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:45:26 smithi115 ceph-mon[78593]: pgmap v6281: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:45:26.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:45:26 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:45:26.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:45:26 smithi047 ceph-mon[77147]: pgmap v6281: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:45:26.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:45:26 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:45:27.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:45:27 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:45:27.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:45:26 smithi115 conmon[90393]: 2022-05-14T02:45:26.982+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:45:27.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:45:27 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:45:28.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:45:28 smithi115 conmon[90393]: 2022-05-14T02:45:28.009+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:45:28.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:45:28 smithi115 ceph-mon[78593]: pgmap v6282: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:45:28.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:45:28 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:45:28.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:45:28 smithi047 ceph-mon[77147]: pgmap v6282: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:45:28.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:45:28 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:45:29.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:45:29 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:45:29.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:45:29 smithi115 conmon[90393]: 2022-05-14T02:45:29.058+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:45:29.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:45:29 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:45:30.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:45:30 smithi115 conmon[90393]: 2022-05-14T02:45:30.072+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:45:30.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:45:30 smithi115 ceph-mon[78593]: pgmap v6283: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:45:30.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:45:30 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:45:30.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:45:30 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:45:30.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:45:30 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:45:30.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:45:30 smithi047 ceph-mon[77147]: pgmap v6283: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:45:30.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:45:30 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:45:30.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:45:30 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:45:30.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:45:30 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:45:31.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:45:31 smithi115 conmon[90393]: 2022-05-14T02:45:31.028+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:45:31.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:45:31 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:45:31.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:45:31 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 11860 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:45:31.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:45:31 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:45:31.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:45:31 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 11860 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:45:32.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:45:31 smithi115 conmon[90393]: 2022-05-14T02:45:31.988+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:45:32.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:45:32 smithi115 ceph-mon[78593]: pgmap v6284: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:45:32.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:45:32 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:45:32.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:45:32 smithi047 ceph-mon[77147]: pgmap v6284: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:45:32.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:45:32 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:45:33.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:45:33 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:45:33.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:45:32 smithi115 conmon[90393]: 2022-05-14T02:45:32.998+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:45:33.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:45:33 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:45:34.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:45:34 smithi115 conmon[90393]: 2022-05-14T02:45:34.039+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:45:34.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:45:34 smithi115 ceph-mon[78593]: pgmap v6285: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:45:34.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:45:34 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:45:34.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:45:34 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:45:34.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:45:34 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:45:34.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:45:34 smithi047 ceph-mon[77147]: pgmap v6285: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:45:34.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:45:34 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:45:34.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:45:34 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:45:34.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:45:34 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:45:35.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:45:35 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:45:35.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:45:35 smithi115 conmon[90393]: 2022-05-14T02:45:35.037+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:45:35.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:45:35 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:45:36.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:45:36 smithi115 conmon[90393]: 2022-05-14T02:45:36.069+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:45:36.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:45:36 smithi115 ceph-mon[78593]: pgmap v6286: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:45:36.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:45:36 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:45:36.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:45:36 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 11870 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:45:36.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:45:36 smithi047 ceph-mon[77147]: pgmap v6286: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:45:36.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:45:36 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:45:36.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:45:36 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 11870 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:45:37.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:45:37 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:45:37.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:45:37 smithi115 conmon[90393]: 2022-05-14T02:45:37.095+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:45:37.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:45:37 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:45:38.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:45:38 smithi115 conmon[90393]: 2022-05-14T02:45:38.098+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:45:38.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:45:38 smithi115 ceph-mon[78593]: pgmap v6287: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:45:38.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:45:38 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:45:38.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:45:38 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:45:38.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:45:38 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:45:38.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:45:38 smithi047 ceph-mon[77147]: pgmap v6287: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:45:38.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:45:38 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:45:38.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:45:38 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:45:38.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:45:38 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:45:39.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:45:39 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:45:39.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:45:39 smithi115 conmon[90393]: 2022-05-14T02:45:39.060+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:45:39.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:45:39 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:45:40.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:45:40 smithi115 conmon[90393]: 2022-05-14T02:45:40.035+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:45:40.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:45:40 smithi115 ceph-mon[78593]: pgmap v6288: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:45:40.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:45:40 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:45:40.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:45:40 smithi047 ceph-mon[77147]: pgmap v6288: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:45:40.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:45:40 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:45:41.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:45:41 smithi115 conmon[90393]: 2022-05-14T02:45:41.013+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:45:41.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:45:41 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:45:41.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:45:41 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:45:42.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:45:42 smithi115 conmon[90393]: 2022-05-14T02:45:42.003+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:45:42.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:45:42 smithi115 ceph-mon[78593]: pgmap v6289: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:45:42.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:45:42 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:45:42.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:45:42 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 11875 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:45:42.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:45:42 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:45:42.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:45:42 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:45:42.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:45:42 smithi047 ceph-mon[77147]: pgmap v6289: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:45:42.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:45:42 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:45:42.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:45:42 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 11875 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:45:42.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:45:42 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:45:42.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:45:42 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:45:43.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:45:42 smithi115 conmon[90393]: 2022-05-14T02:45:42.962+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:45:43.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:45:43 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:45:43.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:45:43 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:45:44.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:45:44 smithi115 conmon[90393]: 2022-05-14T02:45:44.008+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:45:44.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:45:44 smithi115 ceph-mon[78593]: pgmap v6290: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:45:44.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:45:44 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:45:44.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:45:44 smithi047 ceph-mon[77147]: pgmap v6290: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:45:44.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:45:44 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:45:45.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:45:44 smithi115 conmon[90393]: 2022-05-14T02:45:44.976+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:45:45.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:45:45 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:45:45.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:45:45 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:45:46.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:45:46 smithi115 conmon[90393]: 2022-05-14T02:45:46.013+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:45:46.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:45:46 smithi115 ceph-mon[78593]: pgmap v6291: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:45:46.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:45:46 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:45:46.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:45:46 smithi047 ceph-mon[77147]: pgmap v6291: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:45:46.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:45:46 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:45:47.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:45:47 smithi115 conmon[90393]: 2022-05-14T02:45:47.038+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:45:47.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:45:47 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:45:47.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:45:47 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:45:48.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:45:48 smithi115 conmon[90393]: 2022-05-14T02:45:48.059+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:45:48.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:45:48 smithi115 ceph-mon[78593]: pgmap v6292: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:45:48.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:45:48 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:45:48.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:45:48 smithi047 ceph-mon[77147]: pgmap v6292: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:45:48.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:45:48 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:45:49.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:45:49 smithi115 conmon[90393]: 2022-05-14T02:45:49.071+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:45:49.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:45:49 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:45:49.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:45:49 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:45:50.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:45:50 smithi115 conmon[90393]: 2022-05-14T02:45:50.089+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:45:50.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:45:50 smithi115 ceph-mon[78593]: pgmap v6293: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:45:50.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:45:50 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:45:50.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:45:50 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:45:50.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:45:50 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:45:50.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:45:50 smithi047 ceph-mon[77147]: pgmap v6293: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:45:50.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:45:50 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:45:50.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:45:50 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:45:50.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:45:50 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:45:51.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:45:51 smithi115 conmon[90393]: 2022-05-14T02:45:51.055+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:45:51.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:45:51 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:45:51.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:45:51 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 11880 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:45:51.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:45:51 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:45:51.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:45:51 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 11880 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:45:52.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:45:52 smithi115 conmon[90393]: 2022-05-14T02:45:52.018+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:45:52.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:45:52 smithi115 ceph-mon[78593]: pgmap v6294: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:45:52.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:45:52 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:45:52.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:45:52 smithi047 ceph-mon[77147]: pgmap v6294: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:45:52.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:45:52 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:45:53.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:45:53 smithi115 conmon[90393]: 2022-05-14T02:45:53.027+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:45:53.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:45:53 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:45:53.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:45:53 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:45:54.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:45:54 smithi115 conmon[90393]: 2022-05-14T02:45:54.060+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:45:54.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:45:54 smithi115 ceph-mon[78593]: pgmap v6295: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:45:54.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:45:54 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:45:54.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:45:54 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:45:54.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:45:54 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:45:54.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:45:54 smithi047 ceph-mon[77147]: pgmap v6295: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:45:54.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:45:54 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:45:54.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:45:54 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:45:54.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:45:54 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:45:55.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:45:55 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:45:55.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:45:55 smithi115 conmon[90393]: 2022-05-14T02:45:55.076+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:45:55.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:45:55 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:45:56.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:45:56 smithi047 ceph-mon[77147]: pgmap v6296: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:45:56.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:45:56 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:45:56.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:45:56 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 11890 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:45:56.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:45:56 smithi115 conmon[90393]: 2022-05-14T02:45:56.125+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:45:56.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:45:56 smithi115 ceph-mon[78593]: pgmap v6296: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:45:56.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:45:56 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:45:56.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:45:56 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 11890 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:45:57.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:45:57 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:45:57.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:45:57 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T02:45:57.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:45:57 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T02:45:57.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:45:57 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T02:45:57.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:45:57 smithi115 conmon[90393]: 2022-05-14T02:45:57.122+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:45:57.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:45:57 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:45:57.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:45:57 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T02:45:57.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:45:57 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T02:45:57.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:45:57 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T02:45:58.475 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:45:58 smithi115 conmon[90393]: 2022-05-14T02:45:58.146+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:45:58.475 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:45:58 smithi115 ceph-mon[78593]: pgmap v6297: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:45:58.476 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:45:58 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:45:58.476 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:45:58 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:45:58.476 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:45:58 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:45:58.521 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:45:58 smithi047 ceph-mon[77147]: pgmap v6297: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:45:58.522 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:45:58 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:45:58.522 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:45:58 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:45:58.522 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:45:58 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:45:59.220 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:45:59 smithi115 conmon[90393]: 2022-05-14T02:45:59.110+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:45:59.222 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:45:59 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:45:59.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:45:59 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:46:00.195 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:46:00 smithi047 conmon[78521]: 2022-05-14T02:46:00.048+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:46:00.195 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:46:00 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T02:46:00.196 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:46:00 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:46:00.196 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:46:00 smithi047 conmon[78521]: yield 2022-05-14T02:46:00.196 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:46:00 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:46:00.196 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:46:00 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:46:00.197 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:46:00 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:46:00.197 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:46:00 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T02:46:00.197 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:46:00 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:46:00.197 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:46:00 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T02:46:00.197 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:46:00 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:46:00.198 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:46:00 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T02:46:00.198 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:46:00 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:46:00.198 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:46:00 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T02:46:00.198 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:46:00 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:46:00.199 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:46:00 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T02:46:00.199 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:46:00 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:46:00.199 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:46:00 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:46:00.199 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:46:00 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:46:00.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:46:00 smithi115 conmon[90393]: 2022-05-14T02:46:00.115+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:46:00.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:46:00 smithi115 ceph-mon[78593]: pgmap v6298: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:46:00.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:46:00 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:46:00.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:46:00 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' 2022-05-14T02:46:00.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:46:00 smithi047 ceph-mon[77147]: pgmap v6298: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:46:00.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:46:00 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:46:00.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:46:00 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' 2022-05-14T02:46:01.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:46:01 smithi115 conmon[90393]: 2022-05-14T02:46:01.099+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:46:01.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:46:01 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:46:01.359 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T02:46:01.359 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:46:01.359 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T02:46:01.359 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:46:01.359 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:46:01.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:46:01.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T02:46:01.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:46:01.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T02:46:01.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:46:01.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T02:46:01.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:46:01.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T02:46:01.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:46:01.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T02:46:01.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:46:01.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:46:01.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:46:01.363 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:46:01 smithi115 ceph-mon[78593]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T02:46:01.363 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:46:01 smithi115 ceph-mon[78593]: Detected new or changed devices on smithi115 2022-05-14T02:46:01.363 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:46:01 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:46:01.363 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:46:01 smithi115 ceph-mon[78593]: Detected new or changed devices on smithi047 2022-05-14T02:46:01.364 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:46:01 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' 2022-05-14T02:46:01.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:46:01 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:46:01.542 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T02:46:01.542 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:46:01.542 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T02:46:01.542 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:46:01.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:46:01.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:46:01.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T02:46:01.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:46:01.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T02:46:01.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:46:01.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T02:46:01.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:46:01.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T02:46:01.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:46:01.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T02:46:01.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:46:01.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:46:01.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:46:01.546 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:46:01 smithi047 ceph-mon[77147]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T02:46:01.546 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:46:01 smithi047 ceph-mon[77147]: Detected new or changed devices on smithi115 2022-05-14T02:46:01.547 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:46:01 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:46:01.547 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:46:01 smithi047 ceph-mon[77147]: Detected new or changed devices on smithi047 2022-05-14T02:46:01.547 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:46:01 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' 2022-05-14T02:46:02.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:46:02 smithi047 ceph-mon[77147]: pgmap v6299: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:46:02.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:46:02 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:46:02.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:46:02 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 11896 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:46:02.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:46:02 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:46:02.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:46:02 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:46:02.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:46:02 smithi115 conmon[90393]: 2022-05-14T02:46:02.125+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:46:02.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:46:02 smithi115 ceph-mon[78593]: pgmap v6299: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:46:02.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:46:02 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:46:02.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:46:02 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 11896 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:46:02.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:46:02 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:46:02.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:46:02 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:46:03.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:46:03 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:46:03.542 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:46:03 smithi047 conmon[78521]: 2022-05-14T02:46:03.120+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:46:03.542 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:46:03 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T02:46:03.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:46:03 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:46:03.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:46:03 smithi047 conmon[78521]: yield 2022-05-14T02:46:03.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:46:03 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:46:03.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:46:03 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:46:03.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:46:03 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:46:03.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:46:03 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T02:46:03.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:46:03 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:46:03.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:46:03 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T02:46:03.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:46:03 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:46:03.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:46:03 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T02:46:03.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:46:03 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:46:03.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:46:03 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T02:46:03.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:46:03 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:46:03.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:46:03 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T02:46:03.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:46:03 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:46:03.547 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:46:03 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:46:03.547 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:46:03 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:46:03.547 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:46:03 smithi047 conmon[78521]: 2022-05-14T02:46:03.121+0000 7f5340779700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T02:46:03.547 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:46:03 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T02:46:03.548 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:46:03 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:46:03.548 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:46:03 smithi047 conmon[78521]: yield 2022-05-14T02:46:03.548 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:46:03 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:46:03.548 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:46:03 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:46:03.548 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:46:03 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:46:03.549 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:46:03 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T02:46:03.549 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:46:03 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:46:03.549 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:46:03 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T02:46:03.549 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:46:03 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:46:03.550 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:46:03 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T02:46:03.550 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:46:03 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:46:03.550 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:46:03 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T02:46:03.550 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:46:03 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:46:03.551 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:46:03 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T02:46:03.551 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:46:03 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:46:03.551 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:46:03 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:46:03.551 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:46:03 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:46:03.551 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:46:03 smithi047 conmon[78521]: 2022-05-14T02:46:03.552 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:46:03 smithi047 conmon[78521]: During handling of the above exception, another exception occurred: 2022-05-14T02:46:03.552 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:46:03 smithi047 conmon[78521]: 2022-05-14T02:46:03.552 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:46:03 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T02:46:03.552 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:46:03 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T02:46:03.553 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:46:03 smithi047 conmon[78521]: return f(*arg) 2022-05-14T02:46:03.553 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:46:03 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T02:46:03.553 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:46:03 smithi047 conmon[78521]: self._write_client_files(client_files, host) 2022-05-14T02:46:03.553 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:46:03 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T02:46:03.554 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:46:03 smithi047 conmon[78521]: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T02:46:03.554 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:46:03 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T02:46:03.554 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:46:03 smithi047 conmon[78521]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T02:46:03.555 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:46:03 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T02:46:03.555 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:46:03 smithi047 conmon[78521]: return self.event_loop.get_result(coro) 2022-05-14T02:46:03.555 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:46:03 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T02:46:03.555 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:46:03 smithi047 conmon[78521]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T02:46:03.556 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:46:03 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T02:46:03.556 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:46:03 smithi047 conmon[78521]: return self.__get_result() 2022-05-14T02:46:03.556 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:46:03 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T02:46:03.557 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:46:03 smithi047 conmon[78521]: raise self._exception 2022-05-14T02:46:03.557 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:46:03 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T02:46:03.558 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:46:03 smithi047 conmon[78521]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T02:46:03.558 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:46:03 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T02:46:03.558 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:46:03 smithi047 conmon[78521]: conn = await self._remote_connection(host, addr) 2022-05-14T02:46:03.558 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:46:03 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T02:46:03.559 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:46:03 smithi047 conmon[78521]: raise 2022-05-14T02:46:03.559 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:46:03 smithi047 conmon[78521]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T02:46:03.559 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:46:03 smithi047 conmon[78521]: self.gen.throw(type, value, traceback) 2022-05-14T02:46:03.559 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:46:03 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T02:46:03.560 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:46:03 smithi047 conmon[78521]: raise OrchestratorError(msg) 2022-05-14T02:46:03.560 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:46:03 smithi047 conmon[78521]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:46:03.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:46:03 smithi115 conmon[90393]: 2022-05-14T02:46:03.108+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:46:03.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:46:03 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:46:04.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:46:04 smithi047 ceph-mon[77147]: pgmap v6300: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:46:04.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:46:04 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:46:04.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:46:04 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:46:04.542 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T02:46:04.542 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:46:04.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T02:46:04.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:46:04.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:46:04.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:46:04.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T02:46:04.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:46:04.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T02:46:04.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:46:04.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T02:46:04.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:46:04.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T02:46:04.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:46:04.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T02:46:04.548 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:46:04.548 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:46:04.548 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:46:04.548 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:46:04 smithi047 ceph-mon[77147]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T02:46:04.549 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T02:46:04.549 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:46:04.549 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T02:46:04.549 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:46:04.550 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:46:04.550 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:46:04.550 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T02:46:04.550 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:46:04.551 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T02:46:04.551 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:46:04.551 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T02:46:04.552 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:46:04.552 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T02:46:04.552 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:46:04.552 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T02:46:04.553 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:46:04.553 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:46:04.553 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:46:04.553 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T02:46:04.554 INFO:journalctl@ceph.mon.b.smithi047.stdout: During handling of the above exception, another exception occurred: 2022-05-14T02:46:04.554 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T02:46:04.554 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T02:46:04.554 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T02:46:04.555 INFO:journalctl@ceph.mon.b.smithi047.stdout: return f(*arg) 2022-05-14T02:46:04.555 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T02:46:04.555 INFO:journalctl@ceph.mon.b.smithi047.stdout: self._write_client_files(client_files, host) 2022-05-14T02:46:04.556 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T02:46:04.556 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T02:46:04.557 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T02:46:04.557 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T02:46:04.558 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T02:46:04.558 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.event_loop.get_result(coro) 2022-05-14T02:46:04.558 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T02:46:04.559 INFO:journalctl@ceph.mon.b.smithi047.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T02:46:04.559 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T02:46:04.560 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.__get_result() 2022-05-14T02:46:04.560 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T02:46:04.561 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise self._exception 2022-05-14T02:46:04.561 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T02:46:04.561 INFO:journalctl@ceph.mon.b.smithi047.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T02:46:04.561 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T02:46:04.562 INFO:journalctl@ceph.mon.b.smithi047.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T02:46:04.562 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T02:46:04.562 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise 2022-05-14T02:46:04.563 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T02:46:04.563 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.gen.throw(type, value, traceback) 2022-05-14T02:46:04.563 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T02:46:04.563 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OrchestratorError(msg) 2022-05-14T02:46:04.564 INFO:journalctl@ceph.mon.b.smithi047.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:46:04.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:46:04 smithi115 conmon[90393]: 2022-05-14T02:46:04.124+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:46:04.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:46:04 smithi115 ceph-mon[78593]: pgmap v6300: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:46:04.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:46:04 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:46:04.612 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:46:04 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:46:04.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T02:46:04.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:46:04.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T02:46:04.613 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:46:04.613 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:46:04.613 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:46:04.614 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T02:46:04.614 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:46:04.614 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T02:46:04.615 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:46:04.615 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T02:46:04.615 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:46:04.615 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T02:46:04.616 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:46:04.616 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T02:46:04.616 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:46:04.616 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:46:04.616 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:46:04.617 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:46:04 smithi115 ceph-mon[78593]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T02:46:04.617 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T02:46:04.617 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:46:04.617 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T02:46:04.618 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:46:04.618 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:46:04.619 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:46:04.619 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T02:46:04.619 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:46:04.620 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T02:46:04.620 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:46:04.621 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T02:46:04.621 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:46:04.621 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T02:46:04.622 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:46:04.622 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T02:46:04.623 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:46:04.623 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:46:04.623 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:46:04.624 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T02:46:04.624 INFO:journalctl@ceph.mon.c.smithi115.stdout: During handling of the above exception, another exception occurred: 2022-05-14T02:46:04.625 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T02:46:04.625 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T02:46:04.626 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T02:46:04.626 INFO:journalctl@ceph.mon.c.smithi115.stdout: return f(*arg) 2022-05-14T02:46:04.626 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T02:46:04.627 INFO:journalctl@ceph.mon.c.smithi115.stdout: self._write_client_files(client_files, host) 2022-05-14T02:46:04.627 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T02:46:04.628 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T02:46:04.628 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T02:46:04.629 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T02:46:04.629 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T02:46:04.629 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.event_loop.get_result(coro) 2022-05-14T02:46:04.630 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T02:46:04.631 INFO:journalctl@ceph.mon.c.smithi115.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T02:46:04.631 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T02:46:04.632 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.__get_result() 2022-05-14T02:46:04.632 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T02:46:04.633 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise self._exception 2022-05-14T02:46:04.634 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T02:46:04.634 INFO:journalctl@ceph.mon.c.smithi115.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T02:46:04.635 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T02:46:04.635 INFO:journalctl@ceph.mon.c.smithi115.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T02:46:04.636 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T02:46:04.636 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise 2022-05-14T02:46:04.636 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T02:46:04.637 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.gen.throw(type, value, traceback) 2022-05-14T02:46:04.637 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T02:46:04.638 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OrchestratorError(msg) 2022-05-14T02:46:04.638 INFO:journalctl@ceph.mon.c.smithi115.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:46:05.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:46:05 smithi115 conmon[90393]: 2022-05-14T02:46:05.074+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:46:05.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:46:05 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:46:05.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:46:05 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:46:06.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:46:06 smithi115 conmon[90393]: 2022-05-14T02:46:06.092+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:46:06.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:46:06 smithi115 ceph-mon[78593]: pgmap v6301: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:46:06.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:46:06 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:46:06.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:46:06 smithi047 ceph-mon[77147]: pgmap v6301: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:46:06.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:46:06 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:46:07.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:46:07 smithi115 conmon[90393]: 2022-05-14T02:46:07.048+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:46:07.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:46:07 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:46:07.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:46:07 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:46:08.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:46:08 smithi115 conmon[90393]: 2022-05-14T02:46:08.062+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:46:08.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:46:08 smithi115 ceph-mon[78593]: pgmap v6302: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:46:08.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:46:08 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:46:08.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:46:08 smithi047 ceph-mon[77147]: pgmap v6302: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:46:08.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:46:08 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:46:09.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:46:09 smithi115 conmon[90393]: 2022-05-14T02:46:09.094+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:46:09.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:46:09 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:46:09.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:46:09 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:46:10.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:46:10 smithi115 conmon[90393]: 2022-05-14T02:46:10.090+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:46:10.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:46:10 smithi115 ceph-mon[78593]: pgmap v6303: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:46:10.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:46:10 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:46:10.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:46:10 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:46:10.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:46:10 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:46:10.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:46:10 smithi047 ceph-mon[77147]: pgmap v6303: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:46:10.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:46:10 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:46:10.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:46:10 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:46:10.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:46:10 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:46:11.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:46:11 smithi115 conmon[90393]: 2022-05-14T02:46:11.064+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:46:11.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:46:11 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:46:11.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:46:11 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 11901 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:46:11.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:46:11 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:46:11.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:46:11 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 11901 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:46:12.362 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:46:12 smithi115 conmon[90393]: 2022-05-14T02:46:12.053+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:46:12.363 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:46:12 smithi115 ceph-mon[78593]: pgmap v6304: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:46:12.363 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:46:12 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:46:12.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:46:12 smithi047 ceph-mon[77147]: pgmap v6304: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:46:12.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:46:12 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:46:13.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:46:13 smithi115 conmon[90393]: 2022-05-14T02:46:13.036+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:46:13.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:46:13 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:46:13.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:46:13 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:46:14.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:46:14 smithi115 conmon[90393]: 2022-05-14T02:46:14.027+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:46:14.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:46:14 smithi115 ceph-mon[78593]: pgmap v6305: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:46:14.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:46:14 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:46:14.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:46:14 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:46:14.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:46:14 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:46:14.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:46:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T02:46:14.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:46:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T02:46:14.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:46:14 smithi047 ceph-mon[77147]: pgmap v6305: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:46:14.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:46:14 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:46:14.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:46:14 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:46:14.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:46:14 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:46:14.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:46:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T02:46:14.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:46:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T02:46:15.269 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:46:14 smithi115 conmon[90393]: 2022-05-14T02:46:14.996+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:46:15.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:46:15 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:46:15.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:46:15 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:46:16.271 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:46:15 smithi115 conmon[90393]: 2022-05-14T02:46:15.989+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:46:16.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:46:16 smithi047 ceph-mon[77147]: pgmap v6306: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:46:16.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:46:16 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:46:16.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:46:16 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 11910 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:46:16.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:46:16 smithi115 ceph-mon[78593]: pgmap v6306: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:46:16.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:46:16 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:46:16.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:46:16 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 11910 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:46:17.276 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:46:16 smithi115 conmon[90393]: 2022-05-14T02:46:16.940+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:46:17.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:46:17 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:46:17.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:46:17 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:46:18.277 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:46:17 smithi115 conmon[90393]: 2022-05-14T02:46:17.900+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:46:18.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:46:18 smithi047 ceph-mon[77147]: pgmap v6307: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:46:18.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:46:18 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:46:18.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:46:18 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:46:18.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:46:18 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:46:18.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:46:18 smithi115 ceph-mon[78593]: pgmap v6307: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:46:18.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:46:18 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:46:18.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:46:18 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:46:18.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:46:18 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:46:19.282 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:46:18 smithi115 conmon[90393]: 2022-05-14T02:46:18.921+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:46:19.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:46:19 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:46:19.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:46:19 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:46:20.284 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:46:19 smithi115 conmon[90393]: 2022-05-14T02:46:19.920+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:46:20.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:46:20 smithi047 ceph-mon[77147]: pgmap v6308: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:46:20.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:46:20 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:46:20.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:46:20 smithi115 ceph-mon[78593]: pgmap v6308: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:46:20.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:46:20 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:46:21.289 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:46:20 smithi115 conmon[90393]: 2022-05-14T02:46:20.900+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:46:21.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:46:21 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:46:21.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:46:21 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:46:22.290 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:46:21 smithi115 conmon[90393]: 2022-05-14T02:46:21.899+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:46:22.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:46:22 smithi047 ceph-mon[77147]: pgmap v6309: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:46:22.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:46:22 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:46:22.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:46:22 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 11915 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:46:22.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:46:22 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:46:22.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:46:22 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:46:22.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:46:22 smithi115 ceph-mon[78593]: pgmap v6309: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:46:22.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:46:22 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:46:22.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:46:22 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 11915 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:46:22.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:46:22 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:46:22.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:46:22 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:46:23.295 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:46:22 smithi115 conmon[90393]: 2022-05-14T02:46:22.920+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:46:23.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:46:23 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:46:23.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:46:23 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:46:24.297 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:46:23 smithi115 conmon[90393]: 2022-05-14T02:46:23.912+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:46:24.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:46:24 smithi115 ceph-mon[78593]: pgmap v6310: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:46:24.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:46:24 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:46:24.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:46:24 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:46:24.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:46:24 smithi047 ceph-mon[77147]: pgmap v6310: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:46:24.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:46:24 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:46:24.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:46:24 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:46:25.304 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:46:24 smithi115 conmon[90393]: 2022-05-14T02:46:24.872+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:46:25.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:46:25 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:46:25.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:46:25 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:46:26.304 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:46:25 smithi115 conmon[90393]: 2022-05-14T02:46:25.887+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:46:26.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:46:26 smithi115 ceph-mon[78593]: pgmap v6311: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:46:26.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:46:26 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:46:26.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:46:26 smithi047 ceph-mon[77147]: pgmap v6311: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:46:26.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:46:26 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:46:27.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:46:26 smithi115 conmon[90393]: 2022-05-14T02:46:26.843+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:46:27.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:46:27 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:46:27.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:46:27 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:46:28.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:46:27 smithi115 conmon[90393]: 2022-05-14T02:46:27.849+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:46:28.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:46:28 smithi115 ceph-mon[78593]: pgmap v6312: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:46:28.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:46:28 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:46:28.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:46:28 smithi047 ceph-mon[77147]: pgmap v6312: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:46:28.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:46:28 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:46:29.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:46:28 smithi115 conmon[90393]: 2022-05-14T02:46:28.834+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:46:29.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:46:29 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:46:29.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:46:29 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:46:30.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:46:29 smithi115 conmon[90393]: 2022-05-14T02:46:29.789+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:46:30.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:46:30 smithi115 ceph-mon[78593]: pgmap v6313: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:46:30.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:46:30 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:46:30.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:46:30 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:46:30.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:46:30 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:46:30.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:46:30 smithi047 ceph-mon[77147]: pgmap v6313: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:46:30.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:46:30 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:46:30.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:46:30 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:46:30.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:46:30 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:46:31.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:46:30 smithi115 conmon[90393]: 2022-05-14T02:46:30.761+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:46:31.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:46:31 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:46:31.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:46:31 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 11920 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:46:31.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:46:31 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:46:31.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:46:31 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 11920 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:46:32.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:46:31 smithi115 conmon[90393]: 2022-05-14T02:46:31.738+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:46:32.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:46:32 smithi115 ceph-mon[78593]: pgmap v6314: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:46:32.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:46:32 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:46:32.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:46:32 smithi047 ceph-mon[77147]: pgmap v6314: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:46:32.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:46:32 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:46:33.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:46:32 smithi115 conmon[90393]: 2022-05-14T02:46:32.714+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:46:33.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:46:33 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:46:33.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:46:33 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:46:34.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:46:33 smithi115 conmon[90393]: 2022-05-14T02:46:33.694+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:46:34.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:46:34 smithi115 ceph-mon[78593]: pgmap v6315: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:46:34.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:46:34 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:46:34.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:46:34 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:46:34.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:46:34 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:46:34.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:46:34 smithi047 ceph-mon[77147]: pgmap v6315: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:46:34.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:46:34 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:46:34.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:46:34 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:46:34.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:46:34 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:46:35.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:46:34 smithi115 conmon[90393]: 2022-05-14T02:46:34.722+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:46:35.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:46:35 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:46:35.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:46:35 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:46:36.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:46:35 smithi115 conmon[90393]: 2022-05-14T02:46:35.708+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:46:36.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:46:36 smithi115 ceph-mon[78593]: pgmap v6316: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:46:36.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:46:36 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:46:36.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:46:36 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 11930 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:46:36.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:46:36 smithi047 ceph-mon[77147]: pgmap v6316: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:46:36.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:46:36 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:46:36.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:46:36 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 11930 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:46:37.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:46:36 smithi115 conmon[90393]: 2022-05-14T02:46:36.679+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:46:37.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:46:37 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:46:37.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:46:37 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:46:38.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:46:37 smithi115 conmon[90393]: 2022-05-14T02:46:37.655+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:46:38.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:46:38 smithi115 ceph-mon[78593]: pgmap v6317: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:46:38.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:46:38 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:46:38.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:46:38 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:46:38.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:46:38 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:46:38.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:46:38 smithi047 ceph-mon[77147]: pgmap v6317: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:46:38.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:46:38 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:46:38.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:46:38 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:46:38.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:46:38 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:46:39.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:46:38 smithi115 conmon[90393]: 2022-05-14T02:46:38.617+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:46:39.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:46:39 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:46:39.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:46:39 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:46:40.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:46:39 smithi115 conmon[90393]: 2022-05-14T02:46:39.612+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:46:40.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:46:40 smithi115 ceph-mon[78593]: pgmap v6318: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:46:40.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:46:40 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:46:40.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:46:40 smithi047 ceph-mon[77147]: pgmap v6318: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:46:40.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:46:40 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:46:41.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:46:40 smithi115 conmon[90393]: 2022-05-14T02:46:40.653+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:46:41.941 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:46:41 smithi115 conmon[90393]: 2022-05-14T02:46:41.622+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:46:42.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:46:41 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:46:42.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:46:41 smithi047 ceph-mon[77147]: pgmap v6319: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:46:42.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:46:41 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 11935 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:46:42.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:46:41 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:46:42.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:46:41 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:46:42.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:46:41 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:46:42.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:46:41 smithi115 ceph-mon[78593]: pgmap v6319: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:46:42.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:46:41 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 11935 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:46:42.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:46:41 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:46:42.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:46:41 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:46:42.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:46:42 smithi115 conmon[90393]: 2022-05-14T02:46:42.579+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:46:43.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:46:42 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:46:43.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:46:42 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:46:43.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:46:43 smithi115 conmon[90393]: 2022-05-14T02:46:43.531+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:46:44.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:46:43 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:46:44.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:46:43 smithi047 ceph-mon[77147]: pgmap v6320: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:46:44.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:46:43 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:46:44.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:46:43 smithi115 ceph-mon[78593]: pgmap v6320: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:46:44.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:46:44 smithi115 conmon[90393]: 2022-05-14T02:46:44.546+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:46:45.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:46:44 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:46:45.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:46:44 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:46:45.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:46:45 smithi115 conmon[90393]: 2022-05-14T02:46:45.502+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:46:46.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:46:45 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:46:46.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:46:45 smithi047 ceph-mon[77147]: pgmap v6321: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:46:46.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:46:45 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:46:46.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:46:45 smithi115 ceph-mon[78593]: pgmap v6321: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:46:46.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:46:46 smithi115 conmon[90393]: 2022-05-14T02:46:46.520+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:46:47.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:46:46 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:46:47.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:46:46 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:46:47.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:46:47 smithi115 conmon[90393]: 2022-05-14T02:46:47.546+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:46:48.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:46:47 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:46:48.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:46:47 smithi047 ceph-mon[77147]: pgmap v6322: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:46:48.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:46:47 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:46:48.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:46:47 smithi115 ceph-mon[78593]: pgmap v6322: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:46:48.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:46:48 smithi115 conmon[90393]: 2022-05-14T02:46:48.562+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:46:49.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:46:48 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:46:49.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:46:48 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:46:49.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:46:49 smithi115 conmon[90393]: 2022-05-14T02:46:49.519+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:46:50.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:46:49 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:46:50.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:46:49 smithi047 ceph-mon[77147]: pgmap v6323: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:46:50.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:46:49 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:46:50.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:46:49 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:46:50.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:46:49 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:46:50.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:46:49 smithi115 ceph-mon[78593]: pgmap v6323: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:46:50.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:46:49 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:46:50.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:46:49 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:46:50.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:46:50 smithi115 conmon[90393]: 2022-05-14T02:46:50.526+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:46:51.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:46:50 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:46:51.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:46:50 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 11940 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:46:51.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:46:50 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:46:51.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:46:50 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 11940 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:46:51.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:46:51 smithi115 conmon[90393]: 2022-05-14T02:46:51.492+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:46:52.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:46:51 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:46:52.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:46:51 smithi047 ceph-mon[77147]: pgmap v6324: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:46:52.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:46:51 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:46:52.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:46:51 smithi115 ceph-mon[78593]: pgmap v6324: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:46:52.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:46:52 smithi115 conmon[90393]: 2022-05-14T02:46:52.521+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:46:53.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:46:52 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:46:53.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:46:52 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:46:53.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:46:53 smithi115 conmon[90393]: 2022-05-14T02:46:53.561+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:46:54.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:46:53 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:46:54.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:46:53 smithi047 ceph-mon[77147]: pgmap v6325: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:46:54.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:46:53 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:46:54.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:46:53 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:46:54.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:46:53 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:46:54.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:46:53 smithi115 ceph-mon[78593]: pgmap v6325: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:46:54.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:46:53 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:46:54.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:46:53 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:46:54.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:46:54 smithi115 conmon[90393]: 2022-05-14T02:46:54.604+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:46:55.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:46:54 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:46:55.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:46:54 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:46:55.990 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:46:55 smithi115 conmon[90393]: 2022-05-14T02:46:55.624+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:46:56.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:46:55 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:46:56.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:46:55 smithi047 ceph-mon[77147]: pgmap v6326: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:46:56.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:46:55 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 11950 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:46:56.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:46:55 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:46:56.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:46:55 smithi115 ceph-mon[78593]: pgmap v6326: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:46:56.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:46:55 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 11950 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:46:56.996 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:46:56 smithi115 conmon[90393]: 2022-05-14T02:46:56.661+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:46:57.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:46:56 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:46:57.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:46:56 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:46:57.998 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:46:57 smithi115 conmon[90393]: 2022-05-14T02:46:57.663+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:46:58.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:46:57 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:46:58.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:46:57 smithi047 ceph-mon[77147]: pgmap v6327: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:46:58.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:46:57 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:46:58.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:46:57 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:46:58.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:46:57 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:46:58.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:46:57 smithi115 ceph-mon[78593]: pgmap v6327: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:46:58.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:46:57 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:46:58.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:46:57 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:46:59.003 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:46:58 smithi115 conmon[90393]: 2022-05-14T02:46:58.695+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:46:59.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:46:59 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:46:59.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:46:59 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:47:00.004 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:46:59 smithi115 conmon[90393]: 2022-05-14T02:46:59.673+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:47:00.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:47:00 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:47:00.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:47:00 smithi047 ceph-mon[77147]: pgmap v6328: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:47:00.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:47:00 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:47:00.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:47:00 smithi115 ceph-mon[78593]: pgmap v6328: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:47:01.009 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:47:00 smithi115 conmon[90393]: 2022-05-14T02:47:00.653+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:47:01.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:47:01 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:47:01.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:47:01 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:47:02.010 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:47:01 smithi115 conmon[90393]: 2022-05-14T02:47:01.618+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:47:02.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:47:02 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:47:02.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:47:02 smithi047 ceph-mon[77147]: pgmap v6329: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:47:02.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:47:02 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 11955 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:47:02.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:47:02 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:47:02.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:47:02 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:47:02.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:47:02 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:47:02.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:47:02 smithi115 ceph-mon[78593]: pgmap v6329: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:47:02.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:47:02 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 11955 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:47:02.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:47:02 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:47:02.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:47:02 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:47:02.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:47:02 smithi115 conmon[90393]: 2022-05-14T02:47:02.584+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:47:03.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:47:03 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:47:03.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:47:03 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:47:03.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:47:03 smithi115 conmon[90393]: 2022-05-14T02:47:03.589+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:47:04.017 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:47:03 smithi047 conmon[78521]: 2022-05-14T02:47:03.600+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:47:04.017 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:47:03 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T02:47:04.017 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:47:03 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:47:04.018 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:47:03 smithi047 conmon[78521]: yield 2022-05-14T02:47:04.018 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:47:03 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:47:04.018 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:47:03 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:47:04.019 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:47:03 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:47:04.019 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:47:03 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T02:47:04.019 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:47:03 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:47:04.019 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:47:03 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T02:47:04.020 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:47:03 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:47:04.020 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:47:03 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T02:47:04.020 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:47:03 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:47:04.020 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:47:03 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T02:47:04.021 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:47:03 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:47:04.021 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:47:03 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T02:47:04.022 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:47:03 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:47:04.022 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:47:03 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:47:04.022 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:47:03 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:47:04.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:47:04 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:47:04.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:47:04 smithi047 ceph-mon[77147]: pgmap v6330: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:47:04.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:47:04 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T02:47:04.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:47:04 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T02:47:04.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:47:04 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T02:47:04.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:47:04 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:47:04.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:47:04 smithi115 ceph-mon[78593]: pgmap v6330: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:47:04.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:47:04 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T02:47:04.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:47:04 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T02:47:04.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:47:04 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T02:47:04.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:47:04 smithi115 conmon[90393]: 2022-05-14T02:47:04.576+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:47:05.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:47:05 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:47:05.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:47:05 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:47:05.292 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T02:47:05.292 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:47:05.292 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T02:47:05.292 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:47:05.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:47:05.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:47:05.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T02:47:05.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:47:05.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T02:47:05.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:47:05.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T02:47:05.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:47:05.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T02:47:05.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:47:05.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T02:47:05.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:47:05.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:47:05.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:47:05.296 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:47:05 smithi047 ceph-mon[77147]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T02:47:05.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:47:05 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:47:05.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:47:05 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:47:05.359 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T02:47:05.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:47:05.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T02:47:05.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:47:05.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:47:05.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:47:05.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T02:47:05.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:47:05.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T02:47:05.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:47:05.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T02:47:05.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:47:05.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T02:47:05.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:47:05.365 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T02:47:05.365 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:47:05.366 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:47:05.366 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:47:05.366 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:47:05 smithi115 ceph-mon[78593]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T02:47:05.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:47:05 smithi115 conmon[90393]: 2022-05-14T02:47:05.601+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:47:06.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:47:06 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:47:06.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:47:06 smithi047 ceph-mon[77147]: pgmap v6331: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:47:06.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:47:06 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:47:06.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:47:06 smithi115 ceph-mon[78593]: pgmap v6331: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:47:07.030 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:47:06 smithi115 conmon[90393]: 2022-05-14T02:47:06.620+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:47:07.031 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:47:06 smithi047 conmon[78521]: 2022-05-14T02:47:06.671+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:47:07.031 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:47:06 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T02:47:07.032 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:47:06 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:47:07.032 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:47:06 smithi047 conmon[78521]: yield 2022-05-14T02:47:07.032 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:47:06 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:47:07.032 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:47:06 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:47:07.033 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:47:06 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:47:07.033 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:47:06 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T02:47:07.033 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:47:06 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:47:07.033 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:47:06 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T02:47:07.033 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:47:06 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:47:07.034 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:47:06 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T02:47:07.034 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:47:06 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:47:07.034 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:47:06 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T02:47:07.035 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:47:06 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:47:07.035 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:47:06 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T02:47:07.035 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:47:06 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:47:07.035 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:47:06 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:47:07.036 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:47:06 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:47:07.036 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:47:06 smithi047 conmon[78521]: 2022-05-14T02:47:06.672+0000 7f5344781700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T02:47:07.036 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:47:06 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T02:47:07.036 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:47:06 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:47:07.036 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:47:06 smithi047 conmon[78521]: yield 2022-05-14T02:47:07.037 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:47:06 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:47:07.037 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:47:06 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:47:07.037 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:47:06 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:47:07.037 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:47:06 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T02:47:07.037 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:47:06 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:47:07.038 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:47:06 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T02:47:07.038 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:47:06 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:47:07.038 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:47:06 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T02:47:07.038 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:47:06 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:47:07.038 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:47:06 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T02:47:07.039 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:47:06 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:47:07.039 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:47:06 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T02:47:07.039 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:47:06 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:47:07.039 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:47:06 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:47:07.039 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:47:06 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:47:07.040 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:47:06 smithi047 conmon[78521]: 2022-05-14T02:47:07.040 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:47:06 smithi047 conmon[78521]: During handling of the above exception, another exception occurred: 2022-05-14T02:47:07.040 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:47:06 smithi047 conmon[78521]: 2022-05-14T02:47:07.040 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:47:06 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T02:47:07.040 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:47:06 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T02:47:07.041 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:47:06 smithi047 conmon[78521]: return f(*arg) 2022-05-14T02:47:07.041 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:47:06 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T02:47:07.041 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:47:06 smithi047 conmon[78521]: self._write_client_files(client_files, host) 2022-05-14T02:47:07.041 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:47:06 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T02:47:07.042 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:47:06 smithi047 conmon[78521]: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T02:47:07.042 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:47:06 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T02:47:07.042 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:47:06 smithi047 conmon[78521]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T02:47:07.042 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:47:06 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T02:47:07.042 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:47:06 smithi047 conmon[78521]: return self.event_loop.get_result(coro) 2022-05-14T02:47:07.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:47:06 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T02:47:07.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:47:06 smithi047 conmon[78521]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T02:47:07.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:47:06 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T02:47:07.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:47:06 smithi047 conmon[78521]: return self.__get_result() 2022-05-14T02:47:07.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:47:06 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T02:47:07.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:47:06 smithi047 conmon[78521]: raise self._exception 2022-05-14T02:47:07.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:47:06 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T02:47:07.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:47:06 smithi047 conmon[78521]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T02:47:07.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:47:06 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T02:47:07.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:47:06 smithi047 conmon[78521]: conn = await self._remote_connection(host, addr) 2022-05-14T02:47:07.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:47:06 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T02:47:07.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:47:06 smithi047 conmon[78521]: raise 2022-05-14T02:47:07.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:47:06 smithi047 conmon[78521]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T02:47:07.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:47:06 smithi047 conmon[78521]: self.gen.throw(type, value, traceback) 2022-05-14T02:47:07.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:47:06 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T02:47:07.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:47:06 smithi047 conmon[78521]: raise OrchestratorError(msg) 2022-05-14T02:47:07.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:47:06 smithi047 conmon[78521]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:47:07.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:47:07 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:47:07.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:47:07 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:47:08.033 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:47:07 smithi115 conmon[90393]: 2022-05-14T02:47:07.660+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:47:08.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:47:08 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:47:08.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:47:08 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:47:08.292 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T02:47:08.292 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:47:08.292 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T02:47:08.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:47:08.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:47:08.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:47:08.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T02:47:08.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:47:08.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T02:47:08.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:47:08.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T02:47:08.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:47:08.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T02:47:08.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:47:08.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T02:47:08.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:47:08.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:47:08.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:47:08.296 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:47:08 smithi047 ceph-mon[77147]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T02:47:08.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T02:47:08.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:47:08.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T02:47:08.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:47:08.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:47:08.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:47:08.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T02:47:08.298 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:47:08.298 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T02:47:08.298 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:47:08.298 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T02:47:08.298 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:47:08.299 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T02:47:08.299 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:47:08.299 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T02:47:08.299 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:47:08.299 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:47:08.300 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:47:08.300 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T02:47:08.300 INFO:journalctl@ceph.mon.b.smithi047.stdout: During handling of the above exception, another exception occurred: 2022-05-14T02:47:08.301 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T02:47:08.301 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T02:47:08.301 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T02:47:08.301 INFO:journalctl@ceph.mon.b.smithi047.stdout: return f(*arg) 2022-05-14T02:47:08.301 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T02:47:08.302 INFO:journalctl@ceph.mon.b.smithi047.stdout: self._write_client_files(client_files, host) 2022-05-14T02:47:08.302 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T02:47:08.302 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T02:47:08.302 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T02:47:08.303 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T02:47:08.303 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T02:47:08.303 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.event_loop.get_result(coro) 2022-05-14T02:47:08.303 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T02:47:08.304 INFO:journalctl@ceph.mon.b.smithi047.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T02:47:08.304 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T02:47:08.304 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.__get_result() 2022-05-14T02:47:08.304 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T02:47:08.304 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise self._exception 2022-05-14T02:47:08.305 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T02:47:08.305 INFO:journalctl@ceph.mon.b.smithi047.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T02:47:08.305 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T02:47:08.305 INFO:journalctl@ceph.mon.b.smithi047.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T02:47:08.305 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T02:47:08.306 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise 2022-05-14T02:47:08.306 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T02:47:08.306 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.gen.throw(type, value, traceback) 2022-05-14T02:47:08.306 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T02:47:08.306 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OrchestratorError(msg) 2022-05-14T02:47:08.307 INFO:journalctl@ceph.mon.b.smithi047.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:47:08.307 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:47:08 smithi047 ceph-mon[77147]: pgmap v6332: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:47:08.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:47:08 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:47:08.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:47:08 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:47:08.359 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T02:47:08.359 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:47:08.359 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T02:47:08.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:47:08.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:47:08.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:47:08.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T02:47:08.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:47:08.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T02:47:08.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:47:08.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T02:47:08.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:47:08.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T02:47:08.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:47:08.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T02:47:08.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:47:08.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:47:08.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:47:08.363 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:47:08 smithi115 ceph-mon[78593]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T02:47:08.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T02:47:08.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:47:08.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T02:47:08.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:47:08.365 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:47:08.365 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:47:08.365 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T02:47:08.366 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:47:08.366 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T02:47:08.366 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:47:08.366 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T02:47:08.367 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:47:08.367 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T02:47:08.367 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:47:08.367 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T02:47:08.368 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:47:08.368 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:47:08.368 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:47:08.369 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T02:47:08.369 INFO:journalctl@ceph.mon.c.smithi115.stdout: During handling of the above exception, another exception occurred: 2022-05-14T02:47:08.369 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T02:47:08.369 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T02:47:08.370 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T02:47:08.370 INFO:journalctl@ceph.mon.c.smithi115.stdout: return f(*arg) 2022-05-14T02:47:08.370 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T02:47:08.370 INFO:journalctl@ceph.mon.c.smithi115.stdout: self._write_client_files(client_files, host) 2022-05-14T02:47:08.371 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T02:47:08.371 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T02:47:08.371 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T02:47:08.371 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T02:47:08.372 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T02:47:08.372 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.event_loop.get_result(coro) 2022-05-14T02:47:08.372 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T02:47:08.373 INFO:journalctl@ceph.mon.c.smithi115.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T02:47:08.373 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T02:47:08.373 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.__get_result() 2022-05-14T02:47:08.373 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T02:47:08.374 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise self._exception 2022-05-14T02:47:08.374 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T02:47:08.374 INFO:journalctl@ceph.mon.c.smithi115.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T02:47:08.374 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T02:47:08.374 INFO:journalctl@ceph.mon.c.smithi115.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T02:47:08.375 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T02:47:08.375 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise 2022-05-14T02:47:08.375 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T02:47:08.375 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.gen.throw(type, value, traceback) 2022-05-14T02:47:08.375 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T02:47:08.376 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OrchestratorError(msg) 2022-05-14T02:47:08.376 INFO:journalctl@ceph.mon.c.smithi115.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:47:08.376 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:47:08 smithi115 ceph-mon[78593]: pgmap v6332: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:47:09.040 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:47:08 smithi115 conmon[90393]: 2022-05-14T02:47:08.621+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:47:09.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:47:09 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:47:09.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:47:09 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:47:09.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:47:09 smithi115 conmon[90393]: 2022-05-14T02:47:09.589+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:47:10.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:47:10 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:47:10.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:47:10 smithi047 ceph-mon[77147]: pgmap v6333: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:47:10.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:47:10 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:47:10.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:47:10 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:47:10.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:47:10 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:47:10.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:47:10 smithi115 ceph-mon[78593]: pgmap v6333: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:47:10.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:47:10 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:47:10.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:47:10 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:47:10.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:47:10 smithi115 conmon[90393]: 2022-05-14T02:47:10.601+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:47:11.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:47:11 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:47:11.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:47:11 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 11960 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:47:11.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:47:11 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:47:11.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:47:11 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 11960 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:47:12.044 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:47:11 smithi115 conmon[90393]: 2022-05-14T02:47:11.628+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:47:12.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:47:12 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:47:12.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:47:12 smithi115 ceph-mon[78593]: pgmap v6334: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:47:12.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:47:12 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:47:12.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:47:12 smithi047 ceph-mon[77147]: pgmap v6334: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:47:13.050 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:47:12 smithi115 conmon[90393]: 2022-05-14T02:47:12.614+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:47:13.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:47:13 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:47:13.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:47:13 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:47:14.051 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:47:13 smithi115 conmon[90393]: 2022-05-14T02:47:13.614+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:47:14.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:47:14 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:47:14.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:47:14 smithi115 ceph-mon[78593]: pgmap v6335: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:47:14.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:47:14 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:47:14.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:47:14 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:47:14.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:47:14 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:47:14.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:47:14 smithi047 ceph-mon[77147]: pgmap v6335: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:47:14.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:47:14 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:47:14.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:47:14 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:47:14.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:47:14 smithi115 conmon[90393]: 2022-05-14T02:47:14.568+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:47:15.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:47:15 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:47:15.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:47:15 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T02:47:15.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:47:15 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T02:47:15.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:47:15 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:47:15.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:47:15 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T02:47:15.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:47:15 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T02:47:15.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:47:15 smithi115 conmon[90393]: 2022-05-14T02:47:15.544+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:47:16.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:47:16 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:47:16.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:47:16 smithi115 ceph-mon[78593]: pgmap v6336: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:47:16.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:47:16 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 11970 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:47:16.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:47:16 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:47:16.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:47:16 smithi047 ceph-mon[77147]: pgmap v6336: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:47:16.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:47:16 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 11970 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:47:16.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:47:16 smithi115 conmon[90393]: 2022-05-14T02:47:16.576+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:47:17.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:47:17 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:47:17.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:47:17 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:47:17.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:47:17 smithi115 conmon[90393]: 2022-05-14T02:47:17.558+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:47:18.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:47:18 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:47:18.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:47:18 smithi115 ceph-mon[78593]: pgmap v6337: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:47:18.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:47:18 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:47:18.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:47:18 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:47:18.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:47:18 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:47:18.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:47:18 smithi047 ceph-mon[77147]: pgmap v6337: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:47:18.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:47:18 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:47:18.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:47:18 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:47:18.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:47:18 smithi115 conmon[90393]: 2022-05-14T02:47:18.517+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:47:19.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:47:19 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:47:19.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:47:19 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:47:19.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:47:19 smithi115 conmon[90393]: 2022-05-14T02:47:19.515+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:47:20.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:47:20 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:47:20.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:47:20 smithi115 ceph-mon[78593]: pgmap v6338: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:47:20.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:47:20 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:47:20.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:47:20 smithi047 ceph-mon[77147]: pgmap v6338: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:47:20.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:47:20 smithi115 conmon[90393]: 2022-05-14T02:47:20.486+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:47:21.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:47:21 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:47:21.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:47:21 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:47:21.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:47:21 smithi115 conmon[90393]: 2022-05-14T02:47:21.442+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:47:22.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:47:22 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:47:22.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:47:22 smithi115 ceph-mon[78593]: pgmap v6339: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:47:22.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:47:22 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 11975 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:47:22.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:47:22 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:47:22.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:47:22 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:47:22.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:47:22 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:47:22.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:47:22 smithi047 ceph-mon[77147]: pgmap v6339: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:47:22.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:47:22 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 11975 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:47:22.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:47:22 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:47:22.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:47:22 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:47:22.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:47:22 smithi115 conmon[90393]: 2022-05-14T02:47:22.471+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:47:23.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:47:23 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:47:23.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:47:23 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:47:23.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:47:23 smithi115 conmon[90393]: 2022-05-14T02:47:23.517+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:47:24.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:47:24 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:47:24.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:47:24 smithi115 ceph-mon[78593]: pgmap v6340: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:47:24.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:47:24 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:47:24.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:47:24 smithi047 ceph-mon[77147]: pgmap v6340: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:47:24.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:47:24 smithi115 conmon[90393]: 2022-05-14T02:47:24.515+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:47:25.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:47:25 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:47:25.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:47:25 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:47:25.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:47:25 smithi115 conmon[90393]: 2022-05-14T02:47:25.493+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:47:26.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:47:26 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:47:26.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:47:26 smithi115 ceph-mon[78593]: pgmap v6341: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:47:26.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:47:26 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:47:26.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:47:26 smithi047 ceph-mon[77147]: pgmap v6341: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:47:26.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:47:26 smithi115 conmon[90393]: 2022-05-14T02:47:26.541+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:47:27.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:47:27 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:47:27.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:47:27 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:47:27.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:47:27 smithi115 conmon[90393]: 2022-05-14T02:47:27.540+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:47:28.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:47:28 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:47:28.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:47:28 smithi115 ceph-mon[78593]: pgmap v6342: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:47:28.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:47:28 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:47:28.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:47:28 smithi047 ceph-mon[77147]: pgmap v6342: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:47:28.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:47:28 smithi115 conmon[90393]: 2022-05-14T02:47:28.519+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:47:29.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:47:29 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:47:29.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:47:29 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:47:29.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:47:29 smithi115 conmon[90393]: 2022-05-14T02:47:29.525+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:47:30.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:47:30 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:47:30.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:47:30 smithi115 ceph-mon[78593]: pgmap v6343: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:47:30.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:47:30 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:47:30.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:47:30 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:47:30.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:47:30 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:47:30.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:47:30 smithi047 ceph-mon[77147]: pgmap v6343: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:47:30.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:47:30 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:47:30.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:47:30 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:47:30.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:47:30 smithi115 conmon[90393]: 2022-05-14T02:47:30.525+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:47:31.498 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:47:31 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:47:31.498 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:47:31 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 11980 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:47:31.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:47:31 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:47:31.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:47:31 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 11980 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:47:31.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:47:31 smithi115 conmon[90393]: 2022-05-14T02:47:31.497+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:47:32.501 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:47:32 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:47:32.502 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:47:32 smithi115 ceph-mon[78593]: pgmap v6344: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:47:32.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:47:32 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:47:32.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:47:32 smithi047 ceph-mon[77147]: pgmap v6344: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:47:32.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:47:32 smithi115 conmon[90393]: 2022-05-14T02:47:32.500+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:47:33.520 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:47:33 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:47:33.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:47:33 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:47:33.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:47:33 smithi115 conmon[90393]: 2022-05-14T02:47:33.519+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:47:34.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:47:34 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:47:34.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:47:34 smithi047 ceph-mon[77147]: pgmap v6345: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:47:34.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:47:34 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:47:34.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:47:34 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:47:34.563 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:47:34 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:47:34.563 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:47:34 smithi115 ceph-mon[78593]: pgmap v6345: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:47:34.563 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:47:34 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:47:34.564 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:47:34 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:47:34.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:47:34 smithi115 conmon[90393]: 2022-05-14T02:47:34.562+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:47:35.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:47:35 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:47:35.605 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:47:35 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:47:35.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:47:35 smithi115 conmon[90393]: 2022-05-14T02:47:35.604+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:47:36.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:47:36 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:47:36.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:47:36 smithi047 ceph-mon[77147]: pgmap v6346: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:47:36.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:47:36 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 11990 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:47:36.574 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:47:36 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:47:36.575 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:47:36 smithi115 ceph-mon[78593]: pgmap v6346: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:47:36.575 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:47:36 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 11990 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:47:36.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:47:36 smithi115 conmon[90393]: 2022-05-14T02:47:36.573+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:47:37.531 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:47:37 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:47:37.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:47:37 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:47:37.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:47:37 smithi115 conmon[90393]: 2022-05-14T02:47:37.531+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:47:38.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:47:38 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:47:38.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:47:38 smithi047 ceph-mon[77147]: pgmap v6347: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:47:38.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:47:38 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:47:38.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:47:38 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:47:38.551 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:47:38 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:47:38.551 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:47:38 smithi115 ceph-mon[78593]: pgmap v6347: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:47:38.552 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:47:38 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:47:38.552 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:47:38 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:47:38.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:47:38 smithi115 conmon[90393]: 2022-05-14T02:47:38.550+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:47:39.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:47:39 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:47:39.574 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:47:39 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:47:39.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:47:39 smithi115 conmon[90393]: 2022-05-14T02:47:39.573+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:47:40.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:47:40 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:47:40.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:47:40 smithi047 ceph-mon[77147]: pgmap v6348: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:47:40.555 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:47:40 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:47:40.555 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:47:40 smithi115 ceph-mon[78593]: pgmap v6348: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:47:40.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:47:40 smithi115 conmon[90393]: 2022-05-14T02:47:40.553+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:47:41.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:47:41 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:47:41.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:47:41 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:47:41.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:47:41 smithi115 conmon[90393]: 2022-05-14T02:47:41.540+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:47:42.535 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:47:42 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:47:42.535 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:47:42 smithi115 ceph-mon[78593]: pgmap v6349: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:47:42.535 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:47:42 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 11995 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:47:42.536 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:47:42 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:47:42.536 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:47:42 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:47:42.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:47:42 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:47:42.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:47:42 smithi047 ceph-mon[77147]: pgmap v6349: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:47:42.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:47:42 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 11995 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:47:42.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:47:42 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:47:42.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:47:42 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:47:42.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:47:42 smithi115 conmon[90393]: 2022-05-14T02:47:42.533+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:47:43.506 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:47:43 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:47:43.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:47:43 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:47:43.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:47:43 smithi115 conmon[90393]: 2022-05-14T02:47:43.505+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:47:44.529 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:47:44 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:47:44.529 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:47:44 smithi115 ceph-mon[78593]: pgmap v6350: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:47:44.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:47:44 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:47:44.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:47:44 smithi047 ceph-mon[77147]: pgmap v6350: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:47:44.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:47:44 smithi115 conmon[90393]: 2022-05-14T02:47:44.527+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:47:45.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:47:45 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:47:45.563 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:47:45 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:47:45.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:47:45 smithi115 conmon[90393]: 2022-05-14T02:47:45.562+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:47:46.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:47:46 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:47:46.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:47:46 smithi047 ceph-mon[77147]: pgmap v6351: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:47:46.549 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:47:46 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:47:46.550 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:47:46 smithi115 ceph-mon[78593]: pgmap v6351: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:47:46.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:47:46 smithi115 conmon[90393]: 2022-05-14T02:47:46.548+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:47:47.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:47:47 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:47:47.553 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:47:47 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:47:47.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:47:47 smithi115 conmon[90393]: 2022-05-14T02:47:47.552+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:47:48.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:47:48 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:47:48.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:47:48 smithi047 ceph-mon[77147]: pgmap v6352: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:47:48.573 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:47:48 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:47:48.573 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:47:48 smithi115 ceph-mon[78593]: pgmap v6352: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:47:48.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:47:48 smithi115 conmon[90393]: 2022-05-14T02:47:48.571+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:47:49.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:47:49 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:47:49.597 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:47:49 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:47:49.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:47:49 smithi115 conmon[90393]: 2022-05-14T02:47:49.595+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:47:50.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:47:50 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:47:50.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:47:50 smithi047 ceph-mon[77147]: pgmap v6353: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:47:50.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:47:50 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:47:50.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:47:50 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:47:50.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:47:50 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:47:50.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:47:50 smithi115 ceph-mon[78593]: pgmap v6353: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:47:50.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:47:50 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:47:50.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:47:50 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:47:51.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:47:50 smithi115 conmon[90393]: 2022-05-14T02:47:50.635+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:47:51.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:47:51 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:47:51.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:47:51 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 12000 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:47:51.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:47:51 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:47:51.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:47:51 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 12000 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:47:52.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:47:51 smithi115 conmon[90393]: 2022-05-14T02:47:51.607+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:47:52.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:47:52 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:47:52.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:47:52 smithi047 ceph-mon[77147]: pgmap v6354: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:47:52.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:47:52 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:47:52.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:47:52 smithi115 ceph-mon[78593]: pgmap v6354: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:47:53.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:47:52 smithi115 conmon[90393]: 2022-05-14T02:47:52.616+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:47:53.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:47:53 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:47:53.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:47:53 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:47:54.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:47:53 smithi115 conmon[90393]: 2022-05-14T02:47:53.626+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:47:54.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:47:54 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:47:54.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:47:54 smithi047 ceph-mon[77147]: pgmap v6355: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:47:54.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:47:54 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:47:54.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:47:54 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:47:54.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:47:54 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:47:54.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:47:54 smithi115 ceph-mon[78593]: pgmap v6355: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:47:54.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:47:54 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:47:54.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:47:54 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:47:55.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:47:54 smithi115 conmon[90393]: 2022-05-14T02:47:54.632+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:47:55.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:47:55 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:47:55.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:47:55 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:47:56.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:47:55 smithi115 conmon[90393]: 2022-05-14T02:47:55.629+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:47:56.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:47:56 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:47:56.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:47:56 smithi047 ceph-mon[77147]: pgmap v6356: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:47:56.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:47:56 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 12010 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:47:56.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:47:56 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:47:56.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:47:56 smithi115 ceph-mon[78593]: pgmap v6356: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:47:56.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:47:56 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 12010 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:47:57.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:47:56 smithi115 conmon[90393]: 2022-05-14T02:47:56.630+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:47:57.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:47:57 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:47:57.604 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:47:57 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:47:57.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:47:57 smithi115 conmon[90393]: 2022-05-14T02:47:57.603+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:47:58.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:47:58 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:47:58.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:47:58 smithi047 ceph-mon[77147]: pgmap v6357: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:47:58.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:47:58 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:47:58.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:47:58 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:47:58.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:47:58 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:47:58.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:47:58 smithi115 ceph-mon[78593]: pgmap v6357: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:47:58.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:47:58 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:47:58.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:47:58 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:47:59.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:47:58 smithi115 conmon[90393]: 2022-05-14T02:47:58.636+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:47:59.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:47:59 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:47:59.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:47:59 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:48:00.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:47:59 smithi115 conmon[90393]: 2022-05-14T02:47:59.667+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:48:00.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:48:00 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:48:00.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:48:00 smithi047 ceph-mon[77147]: pgmap v6358: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:48:00.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:48:00 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:48:00.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:48:00 smithi115 ceph-mon[78593]: pgmap v6358: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:48:01.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:48:00 smithi115 conmon[90393]: 2022-05-14T02:48:00.665+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:48:01.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:48:01 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:48:01.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:48:01 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:48:02.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:48:01 smithi115 conmon[90393]: 2022-05-14T02:48:01.686+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:48:02.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:48:02 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:48:02.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:48:02 smithi047 ceph-mon[77147]: pgmap v6359: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:48:02.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:48:02 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 12015 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:48:02.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:48:02 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:48:02.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:48:02 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:48:02.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:48:02 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:48:02.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:48:02 smithi115 ceph-mon[78593]: pgmap v6359: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:48:02.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:48:02 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 12015 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:48:02.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:48:02 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:48:02.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:48:02 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:48:03.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:48:02 smithi115 conmon[90393]: 2022-05-14T02:48:02.686+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:48:03.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:48:03 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:48:03.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:48:03 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:48:04.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:48:03 smithi115 conmon[90393]: 2022-05-14T02:48:03.708+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:48:04.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:48:04 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:48:04.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:48:04 smithi047 ceph-mon[77147]: pgmap v6360: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:48:04.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:48:04 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:48:04.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:48:04 smithi115 ceph-mon[78593]: pgmap v6360: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:48:05.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:48:04 smithi115 conmon[90393]: 2022-05-14T02:48:04.732+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:48:05.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:48:05 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:48:05.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:48:05 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:48:06.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:48:05 smithi115 conmon[90393]: 2022-05-14T02:48:05.755+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:48:06.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:48:06 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:48:06.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:48:06 smithi047 ceph-mon[77147]: pgmap v6361: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:48:06.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:48:06 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:48:06.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:48:06 smithi115 ceph-mon[78593]: pgmap v6361: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:48:07.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:48:06 smithi115 conmon[90393]: 2022-05-14T02:48:06.782+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:48:07.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:48:07 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:48:07.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:48:07 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T02:48:07.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:48:07 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T02:48:07.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:48:07 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T02:48:07.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:48:07 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:48:07.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:48:07 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T02:48:07.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:48:07 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T02:48:07.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:48:07 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T02:48:08.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:48:07 smithi115 conmon[90393]: 2022-05-14T02:48:07.803+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:48:08.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:48:08 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:48:08.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:48:08 smithi047 ceph-mon[77147]: pgmap v6362: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:48:08.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:48:08 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:48:08.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:48:08 smithi115 ceph-mon[78593]: pgmap v6362: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:48:09.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:48:08 smithi115 conmon[90393]: 2022-05-14T02:48:08.763+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:48:09.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:48:09 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:48:09.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:48:09 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:48:10.041 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:48:09 smithi047 conmon[78521]: 2022-05-14T02:48:09.776+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:48:10.042 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:48:09 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T02:48:10.042 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:48:09 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:48:10.042 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:48:09 smithi047 conmon[78521]: yield 2022-05-14T02:48:10.042 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:48:09 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:48:10.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:48:09 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:48:10.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:48:09 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:48:10.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:48:09 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T02:48:10.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:48:09 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:48:10.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:48:09 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T02:48:10.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:48:09 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:48:10.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:48:09 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T02:48:10.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:48:09 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:48:10.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:48:09 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T02:48:10.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:48:09 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:48:10.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:48:09 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T02:48:10.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:48:09 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:48:10.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:48:09 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:48:10.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:48:09 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:48:10.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:48:09 smithi115 conmon[90393]: 2022-05-14T02:48:09.749+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:48:10.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:48:10 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:48:10.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:48:10 smithi047 ceph-mon[77147]: pgmap v6363: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:48:10.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:48:10 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:48:10.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:48:10 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:48:10.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:48:10 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:48:10.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:48:10 smithi115 ceph-mon[78593]: pgmap v6363: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:48:10.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:48:10 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:48:10.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:48:10 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:48:11.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:48:10 smithi115 conmon[90393]: 2022-05-14T02:48:10.760+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:48:11.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:48:11 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:48:11.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:48:11 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:48:11.542 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T02:48:11.542 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:48:11.542 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T02:48:11.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:48:11.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:48:11.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:48:11.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T02:48:11.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:48:11.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T02:48:11.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:48:11.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T02:48:11.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:48:11.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T02:48:11.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:48:11.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T02:48:11.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:48:11.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:48:11.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:48:11.546 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:48:11 smithi047 ceph-mon[77147]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T02:48:11.546 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:48:11 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 12020 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:48:11.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:48:11 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:48:11.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:48:11 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:48:11.609 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T02:48:11.609 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:48:11.609 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T02:48:11.609 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:48:11.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:48:11.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:48:11.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T02:48:11.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:48:11.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T02:48:11.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:48:11.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T02:48:11.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:48:11.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T02:48:11.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:48:11.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T02:48:11.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:48:11.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:48:11.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:48:11.613 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:48:11 smithi115 ceph-mon[78593]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T02:48:11.613 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:48:11 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 12020 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:48:12.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:48:11 smithi115 conmon[90393]: 2022-05-14T02:48:11.793+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:48:12.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:48:12 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:48:12.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:48:12 smithi047 ceph-mon[77147]: pgmap v6364: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:48:12.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:48:12 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:48:12.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:48:12 smithi115 ceph-mon[78593]: pgmap v6364: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:48:13.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:48:12 smithi115 conmon[90393]: 2022-05-14T02:48:12.827+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:48:13.250 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:48:12 smithi047 conmon[78521]: 2022-05-14T02:48:12.847+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:48:13.250 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:48:12 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T02:48:13.250 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:48:12 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:48:13.250 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:48:12 smithi047 conmon[78521]: yield 2022-05-14T02:48:13.251 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:48:12 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:48:13.251 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:48:12 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:48:13.251 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:48:12 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:48:13.251 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:48:12 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T02:48:13.252 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:48:12 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:48:13.252 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:48:12 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T02:48:13.252 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:48:12 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:48:13.252 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:48:12 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T02:48:13.253 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:48:12 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:48:13.253 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:48:12 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T02:48:13.253 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:48:12 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:48:13.253 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:48:12 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T02:48:13.253 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:48:12 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:48:13.254 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:48:12 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:48:13.254 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:48:12 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:48:13.254 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:48:12 smithi047 conmon[78521]: 2022-05-14T02:48:12.848+0000 7f5340f7a700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T02:48:13.254 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:48:12 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T02:48:13.255 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:48:12 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:48:13.255 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:48:12 smithi047 conmon[78521]: yield 2022-05-14T02:48:13.255 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:48:12 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:48:13.255 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:48:12 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:48:13.256 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:48:12 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:48:13.256 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:48:12 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T02:48:13.256 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:48:12 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:48:13.256 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:48:12 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T02:48:13.256 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:48:12 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:48:13.257 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:48:12 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T02:48:13.257 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:48:12 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:48:13.257 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:48:12 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T02:48:13.257 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:48:12 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:48:13.258 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:48:12 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T02:48:13.258 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:48:12 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:48:13.258 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:48:12 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:48:13.258 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:48:12 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:48:13.259 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:48:12 smithi047 conmon[78521]: 2022-05-14T02:48:13.259 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:48:12 smithi047 conmon[78521]: During handling of the above exception, another exception occurred: 2022-05-14T02:48:13.259 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:48:12 smithi047 conmon[78521]: 2022-05-14T02:48:13.259 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:48:12 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T02:48:13.259 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:48:12 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T02:48:13.260 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:48:12 smithi047 conmon[78521]: return f(*arg) 2022-05-14T02:48:13.260 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:48:12 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T02:48:13.260 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:48:12 smithi047 conmon[78521]: self._write_client_files(client_files, host) 2022-05-14T02:48:13.260 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:48:12 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T02:48:13.261 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:48:12 smithi047 conmon[78521]: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T02:48:13.261 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:48:12 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T02:48:13.261 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:48:12 smithi047 conmon[78521]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T02:48:13.261 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:48:12 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T02:48:13.262 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:48:12 smithi047 conmon[78521]: return self.event_loop.get_result(coro) 2022-05-14T02:48:13.262 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:48:12 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T02:48:13.262 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:48:12 smithi047 conmon[78521]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T02:48:13.262 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:48:12 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T02:48:13.263 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:48:12 smithi047 conmon[78521]: return self.__get_result() 2022-05-14T02:48:13.263 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:48:12 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T02:48:13.263 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:48:12 smithi047 conmon[78521]: raise self._exception 2022-05-14T02:48:13.263 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:48:12 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T02:48:13.263 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:48:12 smithi047 conmon[78521]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T02:48:13.264 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:48:12 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T02:48:13.264 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:48:12 smithi047 conmon[78521]: conn = await self._remote_connection(host, addr) 2022-05-14T02:48:13.264 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:48:12 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T02:48:13.264 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:48:12 smithi047 conmon[78521]: raise 2022-05-14T02:48:13.264 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:48:12 smithi047 conmon[78521]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T02:48:13.265 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:48:12 smithi047 conmon[78521]: self.gen.throw(type, value, traceback) 2022-05-14T02:48:13.265 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:48:12 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T02:48:13.265 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:48:12 smithi047 conmon[78521]: raise OrchestratorError(msg) 2022-05-14T02:48:13.265 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:48:12 smithi047 conmon[78521]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:48:13.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:48:13 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:48:13.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:48:13 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:48:14.251 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:48:13 smithi115 conmon[90393]: 2022-05-14T02:48:13.875+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:48:14.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:48:14 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:48:14.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:48:14 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:48:14.542 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T02:48:14.542 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:48:14.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T02:48:14.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:48:14.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:48:14.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:48:14.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T02:48:14.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:48:14.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T02:48:14.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:48:14.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T02:48:14.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:48:14.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T02:48:14.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:48:14.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T02:48:14.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:48:14.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:48:14.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:48:14.546 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:48:14 smithi047 ceph-mon[77147]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T02:48:14.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T02:48:14.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:48:14.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T02:48:14.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:48:14.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:48:14.548 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:48:14.548 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T02:48:14.548 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:48:14.548 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T02:48:14.549 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:48:14.549 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T02:48:14.549 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:48:14.549 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T02:48:14.550 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:48:14.550 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T02:48:14.550 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:48:14.550 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:48:14.550 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:48:14.551 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T02:48:14.551 INFO:journalctl@ceph.mon.b.smithi047.stdout: During handling of the above exception, another exception occurred: 2022-05-14T02:48:14.551 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T02:48:14.551 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T02:48:14.552 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T02:48:14.552 INFO:journalctl@ceph.mon.b.smithi047.stdout: return f(*arg) 2022-05-14T02:48:14.552 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T02:48:14.552 INFO:journalctl@ceph.mon.b.smithi047.stdout: self._write_client_files(client_files, host) 2022-05-14T02:48:14.552 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T02:48:14.553 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T02:48:14.553 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T02:48:14.553 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T02:48:14.553 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T02:48:14.553 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.event_loop.get_result(coro) 2022-05-14T02:48:14.554 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T02:48:14.554 INFO:journalctl@ceph.mon.b.smithi047.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T02:48:14.554 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T02:48:14.554 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.__get_result() 2022-05-14T02:48:14.555 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T02:48:14.555 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise self._exception 2022-05-14T02:48:14.555 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T02:48:14.555 INFO:journalctl@ceph.mon.b.smithi047.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T02:48:14.555 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T02:48:14.556 INFO:journalctl@ceph.mon.b.smithi047.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T02:48:14.556 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T02:48:14.556 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise 2022-05-14T02:48:14.556 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T02:48:14.557 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.gen.throw(type, value, traceback) 2022-05-14T02:48:14.557 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T02:48:14.557 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OrchestratorError(msg) 2022-05-14T02:48:14.557 INFO:journalctl@ceph.mon.b.smithi047.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:48:14.557 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:48:14 smithi047 ceph-mon[77147]: pgmap v6365: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:48:14.558 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:48:14 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:48:14.558 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:48:14 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:48:14.558 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:48:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T02:48:14.558 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:48:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T02:48:14.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:48:14 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:48:14.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:48:14 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:48:14.609 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T02:48:14.609 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:48:14.609 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T02:48:14.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:48:14.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:48:14.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:48:14.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T02:48:14.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:48:14.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T02:48:14.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:48:14.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T02:48:14.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:48:14.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T02:48:14.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:48:14.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T02:48:14.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:48:14.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:48:14.613 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:48:14.613 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:48:14 smithi115 ceph-mon[78593]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T02:48:14.613 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T02:48:14.613 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:48:14.613 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T02:48:14.614 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:48:14.614 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:48:14.614 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:48:14.614 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T02:48:14.615 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:48:14.615 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T02:48:14.615 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:48:14.615 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T02:48:14.615 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:48:14.616 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T02:48:14.616 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:48:14.616 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T02:48:14.616 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:48:14.616 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:48:14.617 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:48:14.617 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T02:48:14.617 INFO:journalctl@ceph.mon.c.smithi115.stdout: During handling of the above exception, another exception occurred: 2022-05-14T02:48:14.617 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T02:48:14.618 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T02:48:14.618 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T02:48:14.618 INFO:journalctl@ceph.mon.c.smithi115.stdout: return f(*arg) 2022-05-14T02:48:14.618 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T02:48:14.618 INFO:journalctl@ceph.mon.c.smithi115.stdout: self._write_client_files(client_files, host) 2022-05-14T02:48:14.619 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T02:48:14.619 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T02:48:14.619 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T02:48:14.619 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T02:48:14.620 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T02:48:14.620 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.event_loop.get_result(coro) 2022-05-14T02:48:14.620 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T02:48:14.620 INFO:journalctl@ceph.mon.c.smithi115.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T02:48:14.620 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T02:48:14.621 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.__get_result() 2022-05-14T02:48:14.621 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T02:48:14.621 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise self._exception 2022-05-14T02:48:14.621 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T02:48:14.622 INFO:journalctl@ceph.mon.c.smithi115.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T02:48:14.622 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T02:48:14.622 INFO:journalctl@ceph.mon.c.smithi115.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T02:48:14.622 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T02:48:14.622 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise 2022-05-14T02:48:14.623 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T02:48:14.623 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.gen.throw(type, value, traceback) 2022-05-14T02:48:14.623 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T02:48:14.623 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OrchestratorError(msg) 2022-05-14T02:48:14.624 INFO:journalctl@ceph.mon.c.smithi115.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:48:14.624 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:48:14 smithi115 ceph-mon[78593]: pgmap v6365: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:48:14.624 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:48:14 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:48:14.624 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:48:14 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:48:14.625 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:48:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T02:48:14.625 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:48:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T02:48:15.256 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:48:14 smithi115 conmon[90393]: 2022-05-14T02:48:14.866+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:48:15.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:48:15 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:48:15.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:48:15 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:48:16.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:48:15 smithi115 conmon[90393]: 2022-05-14T02:48:15.825+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:48:16.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:48:16 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:48:16.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:48:16 smithi047 ceph-mon[77147]: pgmap v6366: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:48:16.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:48:16 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 12030 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:48:16.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:48:16 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:48:16.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:48:16 smithi115 ceph-mon[78593]: pgmap v6366: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:48:16.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:48:16 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 12030 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:48:17.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:48:16 smithi115 conmon[90393]: 2022-05-14T02:48:16.801+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:48:17.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:48:17 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:48:17.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:48:17 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:48:18.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:48:17 smithi115 conmon[90393]: 2022-05-14T02:48:17.809+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:48:18.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:48:18 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:48:18.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:48:18 smithi047 ceph-mon[77147]: pgmap v6367: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:48:18.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:48:18 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:48:18.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:48:18 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:48:18.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:48:18 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:48:18.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:48:18 smithi115 ceph-mon[78593]: pgmap v6367: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:48:18.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:48:18 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:48:18.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:48:18 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:48:19.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:48:18 smithi115 conmon[90393]: 2022-05-14T02:48:18.805+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:48:19.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:48:19 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:48:19.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:48:19 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:48:20.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:48:19 smithi115 conmon[90393]: 2022-05-14T02:48:19.802+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:48:20.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:48:20 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:48:20.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:48:20 smithi047 ceph-mon[77147]: pgmap v6368: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:48:20.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:48:20 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:48:20.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:48:20 smithi115 ceph-mon[78593]: pgmap v6368: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:48:21.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:48:20 smithi115 conmon[90393]: 2022-05-14T02:48:20.778+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:48:21.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:48:21 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:48:21.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:48:21 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:48:22.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:48:21 smithi115 conmon[90393]: 2022-05-14T02:48:21.808+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:48:22.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:48:22 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:48:22.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:48:22 smithi047 ceph-mon[77147]: pgmap v6369: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:48:22.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:48:22 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 12035 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:48:22.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:48:22 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:48:22.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:48:22 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:48:22.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:48:22 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:48:22.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:48:22 smithi115 ceph-mon[78593]: pgmap v6369: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:48:22.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:48:22 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 12035 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:48:22.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:48:22 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:48:22.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:48:22 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:48:23.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:48:22 smithi115 conmon[90393]: 2022-05-14T02:48:22.768+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:48:23.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:48:23 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:48:23.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:48:23 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:48:24.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:48:23 smithi115 conmon[90393]: 2022-05-14T02:48:23.734+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:48:24.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:48:24 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:48:24.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:48:24 smithi047 ceph-mon[77147]: pgmap v6370: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:48:24.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:48:24 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:48:24.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:48:24 smithi115 ceph-mon[78593]: pgmap v6370: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:48:25.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:48:24 smithi115 conmon[90393]: 2022-05-14T02:48:24.765+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:48:25.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:48:25 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:48:25.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:48:25 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:48:26.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:48:25 smithi115 conmon[90393]: 2022-05-14T02:48:25.759+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:48:26.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:48:26 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:48:26.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:48:26 smithi047 ceph-mon[77147]: pgmap v6371: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:48:26.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:48:26 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:48:26.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:48:26 smithi115 ceph-mon[78593]: pgmap v6371: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:48:27.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:48:27 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:48:27.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:48:27 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:48:28.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:48:27 smithi115 conmon[90393]: 2022-05-14T02:48:27.745+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:48:28.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:48:28 smithi115 ceph-mon[78593]: pgmap v6372: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:48:28.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:48:28 smithi047 ceph-mon[77147]: pgmap v6372: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:48:29.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:48:28 smithi115 conmon[90393]: 2022-05-14T02:48:28.795+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:48:29.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:48:29 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:48:29.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:48:29 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:48:30.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:48:29 smithi115 conmon[90393]: 2022-05-14T02:48:29.756+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:48:30.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:48:30 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:48:30.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:48:30 smithi115 ceph-mon[78593]: pgmap v6373: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:48:30.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:48:30 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:48:30.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:48:30 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:48:30.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:48:30 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:48:30.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:48:30 smithi047 ceph-mon[77147]: pgmap v6373: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:48:30.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:48:30 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:48:30.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:48:30 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:48:31.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:48:30 smithi115 conmon[90393]: 2022-05-14T02:48:30.746+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:48:31.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:48:31 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:48:31.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:48:31 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 12040 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:48:31.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:48:31 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:48:31.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:48:31 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 12040 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:48:32.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:48:31 smithi115 conmon[90393]: 2022-05-14T02:48:31.748+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:48:32.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:48:32 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:48:32.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:48:32 smithi115 ceph-mon[78593]: pgmap v6374: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:48:32.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:48:32 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:48:32.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:48:32 smithi047 ceph-mon[77147]: pgmap v6374: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:48:33.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:48:32 smithi115 conmon[90393]: 2022-05-14T02:48:32.769+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:48:33.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:48:33 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:48:33.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:48:33 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:48:34.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:48:33 smithi115 conmon[90393]: 2022-05-14T02:48:33.778+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:48:34.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:48:34 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:48:34.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:48:34 smithi115 ceph-mon[78593]: pgmap v6375: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:48:34.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:48:34 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:48:34.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:48:34 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:48:34.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:48:34 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:48:34.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:48:34 smithi047 ceph-mon[77147]: pgmap v6375: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:48:34.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:48:34 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:48:34.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:48:34 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:48:35.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:48:34 smithi115 conmon[90393]: 2022-05-14T02:48:34.825+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:48:35.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:48:35 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:48:35.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:48:35 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:48:36.323 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:48:35 smithi115 conmon[90393]: 2022-05-14T02:48:35.866+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:48:36.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:48:36 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:48:36.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:48:36 smithi115 ceph-mon[78593]: pgmap v6376: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:48:36.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:48:36 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 12050 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:48:36.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:48:36 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:48:36.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:48:36 smithi047 ceph-mon[77147]: pgmap v6376: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:48:36.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:48:36 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 12050 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:48:37.327 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:48:36 smithi115 conmon[90393]: 2022-05-14T02:48:36.913+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:48:37.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:48:37 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:48:37.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:48:37 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:48:37.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:48:37 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:48:37.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:48:37 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:48:38.327 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:48:37 smithi115 conmon[90393]: 2022-05-14T02:48:37.892+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:48:38.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:48:38 smithi115 ceph-mon[78593]: pgmap v6377: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:48:38.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:48:38 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:48:38.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:48:38 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:48:38.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:48:38 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:48:38.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:48:38 smithi047 ceph-mon[77147]: pgmap v6377: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:48:38.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:48:38 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:48:38.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:48:38 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:48:38.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:48:38 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:48:39.333 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:48:38 smithi115 conmon[90393]: 2022-05-14T02:48:38.899+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:48:39.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:48:39 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:48:39.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:48:39 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:48:40.334 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:48:39 smithi115 conmon[90393]: 2022-05-14T02:48:39.867+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:48:40.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:48:40 smithi115 ceph-mon[78593]: pgmap v6378: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:48:40.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:48:40 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:48:40.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:48:40 smithi047 ceph-mon[77147]: pgmap v6378: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:48:40.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:48:40 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:48:41.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:48:40 smithi115 conmon[90393]: 2022-05-14T02:48:40.826+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:48:42.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:48:41 smithi115 conmon[90393]: 2022-05-14T02:48:41.851+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:48:42.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:48:41 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:48:42.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:48:41 smithi115 ceph-mon[78593]: pgmap v6379: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:48:42.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:48:41 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 12055 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:48:42.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:48:41 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:48:42.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:48:41 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:48:42.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:48:41 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:48:42.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:48:41 smithi047 ceph-mon[77147]: pgmap v6379: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:48:42.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:48:41 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 12055 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:48:42.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:48:41 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:48:42.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:48:41 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:48:43.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:48:42 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:48:43.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:48:42 smithi115 conmon[90393]: 2022-05-14T02:48:42.857+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:48:43.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:48:42 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:48:44.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:48:43 smithi115 conmon[90393]: 2022-05-14T02:48:43.808+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:48:44.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:48:43 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:48:44.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:48:43 smithi115 ceph-mon[78593]: pgmap v6380: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:48:44.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:48:43 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:48:44.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:48:43 smithi047 ceph-mon[77147]: pgmap v6380: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:48:45.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:48:44 smithi115 conmon[90393]: 2022-05-14T02:48:44.841+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:48:45.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:48:44 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:48:45.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:48:44 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:48:46.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:48:45 smithi115 conmon[90393]: 2022-05-14T02:48:45.828+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:48:46.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:48:45 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:48:46.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:48:45 smithi115 ceph-mon[78593]: pgmap v6381: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:48:46.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:48:45 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:48:46.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:48:45 smithi047 ceph-mon[77147]: pgmap v6381: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:48:47.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:48:46 smithi115 conmon[90393]: 2022-05-14T02:48:46.850+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:48:47.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:48:46 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:48:47.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:48:46 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:48:48.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:48:47 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:48:48.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:48:47 smithi047 ceph-mon[77147]: pgmap v6382: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:48:48.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:48:47 smithi115 conmon[90393]: 2022-05-14T02:48:47.858+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:48:48.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:48:47 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:48:48.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:48:47 smithi115 ceph-mon[78593]: pgmap v6382: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:48:49.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:48:48 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:48:49.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:48:48 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:48:49.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:48:48 smithi115 conmon[90393]: 2022-05-14T02:48:48.886+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:48:50.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:48:49 smithi115 conmon[90393]: 2022-05-14T02:48:49.842+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:48:50.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:48:49 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:48:50.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:48:49 smithi115 ceph-mon[78593]: pgmap v6383: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:48:50.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:48:49 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:48:50.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:48:49 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:48:50.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:48:49 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:48:50.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:48:49 smithi047 ceph-mon[77147]: pgmap v6383: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:48:50.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:48:49 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:48:50.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:48:49 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:48:51.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:48:50 smithi115 conmon[90393]: 2022-05-14T02:48:50.817+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:48:51.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:48:51 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:48:51.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:48:51 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 12060 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:48:51.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:48:51 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:48:51.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:48:51 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 12060 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:48:52.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:48:51 smithi115 conmon[90393]: 2022-05-14T02:48:51.774+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:48:52.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:48:52 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:48:52.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:48:52 smithi115 ceph-mon[78593]: pgmap v6384: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:48:52.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:48:52 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:48:52.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:48:52 smithi047 ceph-mon[77147]: pgmap v6384: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:48:53.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:48:52 smithi115 conmon[90393]: 2022-05-14T02:48:52.808+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:48:53.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:48:53 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:48:53.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:48:53 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:48:54.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:48:53 smithi115 conmon[90393]: 2022-05-14T02:48:53.778+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:48:54.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:48:54 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:48:54.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:48:54 smithi115 ceph-mon[78593]: pgmap v6385: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:48:54.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:48:54 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:48:54.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:48:54 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:48:54.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:48:54 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:48:54.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:48:54 smithi047 ceph-mon[77147]: pgmap v6385: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:48:54.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:48:54 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:48:54.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:48:54 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:48:55.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:48:54 smithi115 conmon[90393]: 2022-05-14T02:48:54.786+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:48:55.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:48:55 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:48:55.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:48:55 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:48:56.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:48:55 smithi115 conmon[90393]: 2022-05-14T02:48:55.772+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:48:56.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:48:56 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:48:56.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:48:56 smithi115 ceph-mon[78593]: pgmap v6386: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:48:56.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:48:56 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 12070 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:48:56.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:48:56 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:48:56.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:48:56 smithi047 ceph-mon[77147]: pgmap v6386: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:48:56.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:48:56 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 12070 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:48:57.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:48:56 smithi115 conmon[90393]: 2022-05-14T02:48:56.774+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:48:57.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:48:57 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:48:57.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:48:57 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:48:58.023 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:48:57 smithi115 conmon[90393]: 2022-05-14T02:48:57.736+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:48:58.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:48:58 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:48:58.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:48:58 smithi047 ceph-mon[77147]: pgmap v6387: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:48:58.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:48:58 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:48:58.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:48:58 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:48:58.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:48:58 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:48:58.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:48:58 smithi115 ceph-mon[78593]: pgmap v6387: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:48:58.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:48:58 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:48:58.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:48:58 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:48:59.028 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:48:58 smithi115 conmon[90393]: 2022-05-14T02:48:58.776+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:48:59.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:48:59 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:48:59.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:48:59 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:49:00.030 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:48:59 smithi115 conmon[90393]: 2022-05-14T02:48:59.752+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:49:00.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:49:00 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:49:00.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:49:00 smithi047 ceph-mon[77147]: pgmap v6388: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:49:00.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:49:00 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:49:00.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:49:00 smithi115 ceph-mon[78593]: pgmap v6388: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:49:01.035 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:49:00 smithi115 conmon[90393]: 2022-05-14T02:49:00.721+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:49:01.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:49:01 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:49:01.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:49:01 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:49:02.036 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:49:01 smithi115 conmon[90393]: 2022-05-14T02:49:01.720+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:49:02.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:49:02 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:49:02.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:49:02 smithi047 ceph-mon[77147]: pgmap v6389: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:49:02.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:49:02 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 12075 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:49:02.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:49:02 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:49:02.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:49:02 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:49:02.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:49:02 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:49:02.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:49:02 smithi115 ceph-mon[78593]: pgmap v6389: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:49:02.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:49:02 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 12075 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:49:02.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:49:02 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:49:02.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:49:02 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:49:03.041 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:49:02 smithi115 conmon[90393]: 2022-05-14T02:49:02.738+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:49:03.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:49:03 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:49:03.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:49:03 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:49:04.042 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:49:03 smithi115 conmon[90393]: 2022-05-14T02:49:03.724+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:49:04.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:49:04 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:49:04.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:49:04 smithi115 ceph-mon[78593]: pgmap v6390: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:49:04.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:49:04 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:49:04.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:49:04 smithi047 ceph-mon[77147]: pgmap v6390: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:49:05.048 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:49:04 smithi115 conmon[90393]: 2022-05-14T02:49:04.766+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:49:05.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:49:05 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:49:05.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:49:05 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:49:06.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:49:06 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:49:06.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:49:06 smithi115 ceph-mon[78593]: pgmap v6391: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:49:06.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:49:05 smithi115 conmon[90393]: 2022-05-14T02:49:05.814+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:49:06.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:49:06 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:49:06.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:49:06 smithi047 ceph-mon[77147]: pgmap v6391: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:49:07.058 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:49:06 smithi115 conmon[90393]: 2022-05-14T02:49:06.805+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:49:07.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:49:07 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:49:07.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:49:07 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:49:08.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:49:07 smithi115 conmon[90393]: 2022-05-14T02:49:07.835+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:49:08.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:49:08 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:49:08.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:49:08 smithi115 ceph-mon[78593]: pgmap v6392: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:49:08.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:49:08 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:49:08.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:49:08 smithi047 ceph-mon[77147]: pgmap v6392: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:49:09.066 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:49:08 smithi115 conmon[90393]: 2022-05-14T02:49:08.789+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:49:09.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:49:09 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:49:09.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:49:09 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:49:10.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:49:09 smithi115 conmon[90393]: 2022-05-14T02:49:09.827+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:49:10.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:49:10 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:49:10.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:49:10 smithi115 ceph-mon[78593]: pgmap v6393: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:49:10.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:49:10 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:49:10.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:49:10 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:49:10.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:49:10 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:49:10.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:49:10 smithi047 ceph-mon[77147]: pgmap v6393: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:49:10.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:49:10 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:49:10.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:49:10 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:49:11.073 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:49:10 smithi115 conmon[90393]: 2022-05-14T02:49:10.821+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:49:11.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:49:11 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:49:11.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:49:11 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 12080 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:49:11.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:49:11 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:49:11.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:49:11 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 12080 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:49:12.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:49:11 smithi115 conmon[90393]: 2022-05-14T02:49:11.843+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:49:12.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:49:12 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:49:12.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:49:12 smithi115 ceph-mon[78593]: pgmap v6394: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:49:12.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:49:12 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:49:12.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:49:12 smithi047 ceph-mon[77147]: pgmap v6394: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:49:13.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:49:13 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:49:13.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:49:13 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T02:49:13.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:49:13 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T02:49:13.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:49:13 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T02:49:13.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:49:12 smithi115 conmon[90393]: 2022-05-14T02:49:12.860+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:49:13.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:49:13 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:49:13.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:49:13 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T02:49:13.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:49:13 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T02:49:13.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:49:13 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T02:49:14.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:49:13 smithi115 conmon[90393]: 2022-05-14T02:49:13.882+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:49:14.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:49:14 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:49:14.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:49:14 smithi115 ceph-mon[78593]: pgmap v6395: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:49:14.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:49:14 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:49:14.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:49:14 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:49:14.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:49:14 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:49:14.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:49:14 smithi047 ceph-mon[77147]: pgmap v6395: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:49:14.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:49:14 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:49:14.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:49:14 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:49:15.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:49:14 smithi115 conmon[90393]: 2022-05-14T02:49:14.886+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:49:15.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:49:15 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:49:15.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:49:15 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T02:49:15.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:49:15 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T02:49:15.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:49:15 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:49:15.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:49:15 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T02:49:15.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:49:15 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T02:49:16.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:49:16 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:49:16.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:49:16 smithi047 ceph-mon[77147]: pgmap v6396: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:49:16.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:49:16 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 12090 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:49:16.292 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:49:15 smithi047 conmon[78521]: 2022-05-14T02:49:15.952+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:49:16.293 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:49:15 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T02:49:16.293 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:49:15 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:49:16.293 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:49:15 smithi047 conmon[78521]: yield 2022-05-14T02:49:16.293 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:49:15 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:49:16.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:49:15 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:49:16.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:49:15 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:49:16.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:49:15 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T02:49:16.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:49:15 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:49:16.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:49:15 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T02:49:16.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:49:15 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:49:16.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:49:15 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T02:49:16.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:49:15 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:49:16.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:49:15 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T02:49:16.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:49:15 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:49:16.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:49:15 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T02:49:16.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:49:15 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:49:16.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:49:15 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:49:16.297 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:49:15 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:49:16.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:49:15 smithi115 conmon[90393]: 2022-05-14T02:49:15.872+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:49:16.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:49:16 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:49:16.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:49:16 smithi115 ceph-mon[78593]: pgmap v6396: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:49:16.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:49:16 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 12090 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:49:17.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:49:16 smithi115 conmon[90393]: 2022-05-14T02:49:16.875+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:49:17.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:49:17 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:49:17.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:49:17 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:49:17.359 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T02:49:17.359 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:49:17.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T02:49:17.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:49:17.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:49:17.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:49:17.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T02:49:17.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:49:17.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T02:49:17.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:49:17.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T02:49:17.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:49:17.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T02:49:17.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:49:17.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T02:49:17.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:49:17.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:49:17.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:49:17.363 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:49:17 smithi115 ceph-mon[78593]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T02:49:17.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:49:17 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:49:17.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:49:17 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:49:17.542 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T02:49:17.542 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:49:17.542 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T02:49:17.542 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:49:17.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:49:17.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:49:17.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T02:49:17.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:49:17.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T02:49:17.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:49:17.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T02:49:17.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:49:17.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T02:49:17.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:49:17.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T02:49:17.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:49:17.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:49:17.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:49:17.546 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:49:17 smithi047 ceph-mon[77147]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T02:49:18.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:49:17 smithi115 conmon[90393]: 2022-05-14T02:49:17.891+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:49:18.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:49:18 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:49:18.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:49:18 smithi115 ceph-mon[78593]: pgmap v6397: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:49:18.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:49:18 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:49:18.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:49:18 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:49:18.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:49:18 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:49:18.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:49:18 smithi047 ceph-mon[77147]: pgmap v6397: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:49:18.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:49:18 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:49:18.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:49:18 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:49:19.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:49:19 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:49:19.292 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:49:19 smithi047 conmon[78521]: 2022-05-14T02:49:19.024+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:49:19.292 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:49:19 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T02:49:19.292 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:49:19 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:49:19.293 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:49:19 smithi047 conmon[78521]: yield 2022-05-14T02:49:19.293 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:49:19 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:49:19.293 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:49:19 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:49:19.293 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:49:19 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:49:19.293 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:49:19 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T02:49:19.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:49:19 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:49:19.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:49:19 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T02:49:19.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:49:19 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:49:19.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:49:19 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T02:49:19.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:49:19 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:49:19.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:49:19 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T02:49:19.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:49:19 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:49:19.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:49:19 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T02:49:19.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:49:19 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:49:19.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:49:19 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:49:19.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:49:19 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:49:19.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:49:19 smithi047 conmon[78521]: 2022-05-14T02:49:19.025+0000 7f5345f84700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T02:49:19.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:49:19 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T02:49:19.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:49:19 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:49:19.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:49:19 smithi047 conmon[78521]: yield 2022-05-14T02:49:19.297 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:49:19 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:49:19.297 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:49:19 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:49:19.297 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:49:19 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:49:19.297 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:49:19 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T02:49:19.297 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:49:19 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:49:19.298 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:49:19 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T02:49:19.298 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:49:19 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:49:19.298 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:49:19 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T02:49:19.298 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:49:19 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:49:19.298 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:49:19 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T02:49:19.299 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:49:19 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:49:19.299 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:49:19 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T02:49:19.299 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:49:19 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:49:19.299 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:49:19 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:49:19.299 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:49:19 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:49:19.300 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:49:19 smithi047 conmon[78521]: 2022-05-14T02:49:19.300 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:49:19 smithi047 conmon[78521]: During handling of the above exception, another exception occurred: 2022-05-14T02:49:19.300 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:49:19 smithi047 conmon[78521]: 2022-05-14T02:49:19.300 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:49:19 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T02:49:19.300 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:49:19 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T02:49:19.301 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:49:19 smithi047 conmon[78521]: return f(*arg) 2022-05-14T02:49:19.301 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:49:19 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T02:49:19.301 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:49:19 smithi047 conmon[78521]: self._write_client_files(client_files, host) 2022-05-14T02:49:19.301 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:49:19 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T02:49:19.301 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:49:19 smithi047 conmon[78521]: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T02:49:19.302 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:49:19 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T02:49:19.302 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:49:19 smithi047 conmon[78521]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T02:49:19.302 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:49:19 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T02:49:19.302 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:49:19 smithi047 conmon[78521]: return self.event_loop.get_result(coro) 2022-05-14T02:49:19.303 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:49:19 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T02:49:19.303 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:49:19 smithi047 conmon[78521]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T02:49:19.303 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:49:19 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T02:49:19.303 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:49:19 smithi047 conmon[78521]: return self.__get_result() 2022-05-14T02:49:19.303 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:49:19 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T02:49:19.303 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:49:19 smithi047 conmon[78521]: raise self._exception 2022-05-14T02:49:19.304 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:49:19 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T02:49:19.304 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:49:19 smithi047 conmon[78521]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T02:49:19.304 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:49:19 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T02:49:19.304 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:49:19 smithi047 conmon[78521]: conn = await self._remote_connection(host, addr) 2022-05-14T02:49:19.304 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:49:19 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T02:49:19.305 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:49:19 smithi047 conmon[78521]: raise 2022-05-14T02:49:19.305 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:49:19 smithi047 conmon[78521]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T02:49:19.305 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:49:19 smithi047 conmon[78521]: self.gen.throw(type, value, traceback) 2022-05-14T02:49:19.305 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:49:19 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T02:49:19.305 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:49:19 smithi047 conmon[78521]: raise OrchestratorError(msg) 2022-05-14T02:49:19.306 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:49:19 smithi047 conmon[78521]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:49:19.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:49:18 smithi115 conmon[90393]: 2022-05-14T02:49:18.929+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:49:19.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:49:19 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:49:20.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:49:19 smithi115 conmon[90393]: 2022-05-14T02:49:19.943+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:49:20.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:49:20 smithi115 ceph-mon[78593]: pgmap v6398: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:49:20.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:49:20 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:49:20.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:49:20 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:49:20.359 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T02:49:20.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:49:20.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T02:49:20.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:49:20.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:49:20.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:49:20.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T02:49:20.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:49:20.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T02:49:20.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:49:20.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T02:49:20.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:49:20.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T02:49:20.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:49:20.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T02:49:20.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:49:20.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:49:20.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:49:20.363 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:49:20 smithi115 ceph-mon[78593]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T02:49:20.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T02:49:20.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:49:20.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T02:49:20.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:49:20.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:49:20.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:49:20.365 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T02:49:20.365 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:49:20.365 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T02:49:20.365 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:49:20.366 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T02:49:20.366 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:49:20.366 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T02:49:20.366 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:49:20.366 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T02:49:20.367 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:49:20.367 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:49:20.367 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:49:20.367 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T02:49:20.367 INFO:journalctl@ceph.mon.c.smithi115.stdout: During handling of the above exception, another exception occurred: 2022-05-14T02:49:20.368 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T02:49:20.368 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T02:49:20.368 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T02:49:20.368 INFO:journalctl@ceph.mon.c.smithi115.stdout: return f(*arg) 2022-05-14T02:49:20.369 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T02:49:20.369 INFO:journalctl@ceph.mon.c.smithi115.stdout: self._write_client_files(client_files, host) 2022-05-14T02:49:20.369 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T02:49:20.369 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T02:49:20.369 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T02:49:20.370 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T02:49:20.370 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T02:49:20.370 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.event_loop.get_result(coro) 2022-05-14T02:49:20.370 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T02:49:20.371 INFO:journalctl@ceph.mon.c.smithi115.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T02:49:20.371 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T02:49:20.371 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.__get_result() 2022-05-14T02:49:20.371 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T02:49:20.371 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise self._exception 2022-05-14T02:49:20.372 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T02:49:20.372 INFO:journalctl@ceph.mon.c.smithi115.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T02:49:20.372 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T02:49:20.372 INFO:journalctl@ceph.mon.c.smithi115.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T02:49:20.372 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T02:49:20.373 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise 2022-05-14T02:49:20.373 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T02:49:20.373 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.gen.throw(type, value, traceback) 2022-05-14T02:49:20.373 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T02:49:20.374 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OrchestratorError(msg) 2022-05-14T02:49:20.374 INFO:journalctl@ceph.mon.c.smithi115.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:49:20.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:49:20 smithi047 ceph-mon[77147]: pgmap v6398: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:49:20.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:49:20 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:49:20.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:49:20 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:49:20.542 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T02:49:20.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:49:20.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T02:49:20.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:49:20.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:49:20.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:49:20.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T02:49:20.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:49:20.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T02:49:20.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:49:20.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T02:49:20.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:49:20.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T02:49:20.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:49:20.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T02:49:20.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:49:20.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:49:20.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:49:20.546 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:49:20 smithi047 ceph-mon[77147]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T02:49:20.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T02:49:20.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:49:20.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T02:49:20.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:49:20.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:49:20.548 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:49:20.548 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T02:49:20.548 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:49:20.548 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T02:49:20.549 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:49:20.549 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T02:49:20.549 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:49:20.549 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T02:49:20.549 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:49:20.550 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T02:49:20.550 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:49:20.550 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:49:20.550 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:49:20.551 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T02:49:20.551 INFO:journalctl@ceph.mon.b.smithi047.stdout: During handling of the above exception, another exception occurred: 2022-05-14T02:49:20.551 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T02:49:20.551 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T02:49:20.551 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T02:49:20.552 INFO:journalctl@ceph.mon.b.smithi047.stdout: return f(*arg) 2022-05-14T02:49:20.552 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T02:49:20.552 INFO:journalctl@ceph.mon.b.smithi047.stdout: self._write_client_files(client_files, host) 2022-05-14T02:49:20.552 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T02:49:20.552 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T02:49:20.553 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T02:49:20.553 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T02:49:20.553 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T02:49:20.553 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.event_loop.get_result(coro) 2022-05-14T02:49:20.554 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T02:49:20.554 INFO:journalctl@ceph.mon.b.smithi047.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T02:49:20.554 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T02:49:20.554 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.__get_result() 2022-05-14T02:49:20.554 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T02:49:20.555 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise self._exception 2022-05-14T02:49:20.555 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T02:49:20.555 INFO:journalctl@ceph.mon.b.smithi047.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T02:49:20.555 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T02:49:20.556 INFO:journalctl@ceph.mon.b.smithi047.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T02:49:20.556 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T02:49:20.556 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise 2022-05-14T02:49:20.556 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T02:49:20.556 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.gen.throw(type, value, traceback) 2022-05-14T02:49:20.557 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T02:49:20.557 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OrchestratorError(msg) 2022-05-14T02:49:20.557 INFO:journalctl@ceph.mon.b.smithi047.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:49:21.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:49:20 smithi115 conmon[90393]: 2022-05-14T02:49:20.922+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:49:21.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:49:21 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:49:21.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:49:21 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:49:22.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:49:21 smithi115 conmon[90393]: 2022-05-14T02:49:21.944+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:49:22.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:49:22 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:49:22.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:49:22 smithi115 ceph-mon[78593]: pgmap v6399: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:49:22.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:49:22 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 12095 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:49:22.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:49:22 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:49:22.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:49:22 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:49:22.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:49:22 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:49:22.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:49:22 smithi047 ceph-mon[77147]: pgmap v6399: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:49:22.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:49:22 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 12095 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:49:22.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:49:22 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:49:22.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:49:22 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:49:23.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:49:22 smithi115 conmon[90393]: 2022-05-14T02:49:22.965+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:49:23.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:49:23 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:49:23.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:49:23 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:49:24.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:49:24 smithi115 conmon[90393]: 2022-05-14T02:49:24.003+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:49:24.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:49:24 smithi115 ceph-mon[78593]: pgmap v6400: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:49:24.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:49:24 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:49:24.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:49:24 smithi047 ceph-mon[77147]: pgmap v6400: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:49:24.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:49:24 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:49:25.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:49:24 smithi115 conmon[90393]: 2022-05-14T02:49:24.981+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:49:25.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:49:25 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:49:25.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:49:25 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:49:26.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:49:25 smithi115 conmon[90393]: 2022-05-14T02:49:25.978+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:49:26.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:49:26 smithi115 ceph-mon[78593]: pgmap v6401: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:49:26.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:49:26 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:49:26.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:49:26 smithi047 ceph-mon[77147]: pgmap v6401: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:49:26.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:49:26 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:49:27.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:49:26 smithi115 conmon[90393]: 2022-05-14T02:49:26.950+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:49:27.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:49:27 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:49:27.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:49:27 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:49:28.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:49:27 smithi115 conmon[90393]: 2022-05-14T02:49:27.954+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:49:28.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:49:28 smithi115 ceph-mon[78593]: pgmap v6402: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:49:28.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:49:28 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:49:28.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:49:28 smithi047 ceph-mon[77147]: pgmap v6402: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:49:28.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:49:28 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:49:29.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:49:28 smithi115 conmon[90393]: 2022-05-14T02:49:28.943+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:49:29.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:49:29 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:49:29.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:49:29 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:49:30.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:49:29 smithi115 conmon[90393]: 2022-05-14T02:49:29.963+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:49:30.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:49:30 smithi115 ceph-mon[78593]: pgmap v6403: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:49:30.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:49:30 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:49:30.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:49:30 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:49:30.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:49:30 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:49:30.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:49:30 smithi047 ceph-mon[77147]: pgmap v6403: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:49:30.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:49:30 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:49:30.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:49:30 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:49:30.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:49:30 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:49:31.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:49:30 smithi115 conmon[90393]: 2022-05-14T02:49:30.946+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:49:31.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:49:31 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:49:31.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:49:31 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 12100 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:49:31.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:49:31 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:49:31.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:49:31 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 12100 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:49:32.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:49:31 smithi115 conmon[90393]: 2022-05-14T02:49:31.945+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:49:32.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:49:32 smithi115 ceph-mon[78593]: pgmap v6404: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:49:32.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:49:32 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:49:32.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:49:32 smithi047 ceph-mon[77147]: pgmap v6404: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:49:32.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:49:32 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:49:33.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:49:32 smithi115 conmon[90393]: 2022-05-14T02:49:32.933+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:49:33.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:49:33 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:49:33.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:49:33 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:49:34.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:49:33 smithi115 conmon[90393]: 2022-05-14T02:49:33.912+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:49:34.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:49:34 smithi115 ceph-mon[78593]: pgmap v6405: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:49:34.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:49:34 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:49:34.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:49:34 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:49:34.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:49:34 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:49:34.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:49:34 smithi047 ceph-mon[77147]: pgmap v6405: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:49:34.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:49:34 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:49:34.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:49:34 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:49:34.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:49:34 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:49:35.147 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:49:34 smithi115 conmon[90393]: 2022-05-14T02:49:34.868+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:49:35.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:49:35 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:49:35.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:49:35 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:49:36.149 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:49:35 smithi115 conmon[90393]: 2022-05-14T02:49:35.892+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:49:36.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:49:36 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:49:36.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:49:36 smithi047 ceph-mon[77147]: pgmap v6406: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:49:36.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:49:36 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 12110 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:49:36.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:49:36 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:49:36.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:49:36 smithi115 ceph-mon[78593]: pgmap v6406: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:49:36.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:49:36 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 12110 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:49:37.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:49:37 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:49:37.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:49:36 smithi115 conmon[90393]: 2022-05-14T02:49:36.909+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:49:37.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:49:37 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:49:38.155 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:49:37 smithi115 conmon[90393]: 2022-05-14T02:49:37.864+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:49:38.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:49:38 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:49:38.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:49:38 smithi047 ceph-mon[77147]: pgmap v6407: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:49:38.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:49:38 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:49:38.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:49:38 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:49:38.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:49:38 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:49:38.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:49:38 smithi115 ceph-mon[78593]: pgmap v6407: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:49:38.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:49:38 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:49:38.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:49:38 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:49:39.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:49:38 smithi115 conmon[90393]: 2022-05-14T02:49:38.855+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:49:39.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:49:39 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:49:39.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:49:39 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:49:40.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:49:39 smithi115 conmon[90393]: 2022-05-14T02:49:39.839+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:49:40.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:49:40 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:49:40.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:49:40 smithi047 ceph-mon[77147]: pgmap v6408: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:49:40.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:49:40 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:49:40.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:49:40 smithi115 ceph-mon[78593]: pgmap v6408: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:49:41.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:49:40 smithi115 conmon[90393]: 2022-05-14T02:49:40.829+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:49:41.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:49:41 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:49:41.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:49:41 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:49:42.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:49:41 smithi115 conmon[90393]: 2022-05-14T02:49:41.841+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:49:42.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:49:42 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:49:42.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:49:42 smithi047 ceph-mon[77147]: pgmap v6409: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:49:42.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:49:42 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 12115 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:49:42.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:49:42 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:49:42.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:49:42 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:49:42.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:49:42 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:49:42.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:49:42 smithi115 ceph-mon[78593]: pgmap v6409: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:49:42.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:49:42 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 12115 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:49:42.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:49:42 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:49:42.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:49:42 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:49:43.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:49:42 smithi115 conmon[90393]: 2022-05-14T02:49:42.850+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:49:43.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:49:43 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:49:43.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:49:43 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:49:44.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:49:43 smithi115 conmon[90393]: 2022-05-14T02:49:43.850+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:49:44.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:49:44 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:49:44.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:49:44 smithi047 ceph-mon[77147]: pgmap v6410: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:49:44.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:49:44 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:49:44.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:49:44 smithi115 ceph-mon[78593]: pgmap v6410: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:49:45.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:49:44 smithi115 conmon[90393]: 2022-05-14T02:49:44.842+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:49:45.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:49:45 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:49:45.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:49:45 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:49:46.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:49:45 smithi115 conmon[90393]: 2022-05-14T02:49:45.837+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:49:46.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:49:46 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:49:46.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:49:46 smithi047 ceph-mon[77147]: pgmap v6411: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:49:46.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:49:46 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:49:46.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:49:46 smithi115 ceph-mon[78593]: pgmap v6411: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:49:47.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:49:46 smithi115 conmon[90393]: 2022-05-14T02:49:46.787+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:49:47.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:49:47 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:49:47.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:49:47 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:49:48.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:49:47 smithi115 conmon[90393]: 2022-05-14T02:49:47.834+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:49:48.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:49:48 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:49:48.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:49:48 smithi047 ceph-mon[77147]: pgmap v6412: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:49:48.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:49:48 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:49:48.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:49:48 smithi115 ceph-mon[78593]: pgmap v6412: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:49:49.193 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:49:48 smithi115 conmon[90393]: 2022-05-14T02:49:48.874+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:49:49.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:49:49 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:49:49.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:49:49 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:49:50.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:49:49 smithi115 conmon[90393]: 2022-05-14T02:49:49.855+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:49:50.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:49:50 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:49:50.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:49:50 smithi047 ceph-mon[77147]: pgmap v6413: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:49:50.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:49:50 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:49:50.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:49:50 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:49:50.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:49:50 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:49:50.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:49:50 smithi115 ceph-mon[78593]: pgmap v6413: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:49:50.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:49:50 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:49:50.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:49:50 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:49:51.203 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:49:50 smithi115 conmon[90393]: 2022-05-14T02:49:50.869+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:49:51.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:49:51 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:49:51.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:49:51 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 12120 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:49:51.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:49:51 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:49:51.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:49:51 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 12120 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:49:52.202 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:49:51 smithi115 conmon[90393]: 2022-05-14T02:49:51.895+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:49:52.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:49:52 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:49:52.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:49:52 smithi047 ceph-mon[77147]: pgmap v6414: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:49:52.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:49:52 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:49:52.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:49:52 smithi115 ceph-mon[78593]: pgmap v6414: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:49:53.210 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:49:52 smithi115 conmon[90393]: 2022-05-14T02:49:52.915+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:49:53.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:49:53 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:49:53.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:49:53 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:49:54.209 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:49:53 smithi115 conmon[90393]: 2022-05-14T02:49:53.876+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:49:54.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:49:54 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:49:54.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:49:54 smithi047 ceph-mon[77147]: pgmap v6415: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:49:54.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:49:54 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:49:54.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:49:54 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:49:54.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:49:54 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:49:54.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:49:54 smithi115 ceph-mon[78593]: pgmap v6415: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:49:54.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:49:54 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:49:54.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:49:54 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:49:55.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:49:54 smithi115 conmon[90393]: 2022-05-14T02:49:54.838+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:49:55.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:49:55 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:49:55.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:49:55 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:49:56.215 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:49:55 smithi115 conmon[90393]: 2022-05-14T02:49:55.857+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:49:56.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:49:56 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:49:56.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:49:56 smithi047 ceph-mon[77147]: pgmap v6416: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:49:56.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:49:56 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 12130 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:49:56.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:49:56 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:49:56.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:49:56 smithi115 ceph-mon[78593]: pgmap v6416: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:49:56.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:49:56 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 12130 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:49:57.221 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:49:56 smithi115 conmon[90393]: 2022-05-14T02:49:56.885+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:49:57.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:49:57 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:49:57.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:49:57 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:49:58.222 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:49:57 smithi115 conmon[90393]: 2022-05-14T02:49:57.934+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:49:58.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:49:58 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:49:58.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:49:58 smithi047 ceph-mon[77147]: pgmap v6417: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:49:58.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:49:58 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:49:58.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:49:58 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:49:58.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:49:58 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:49:58.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:49:58 smithi115 ceph-mon[78593]: pgmap v6417: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:49:58.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:49:58 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:49:58.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:49:58 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:49:59.227 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:49:58 smithi115 conmon[90393]: 2022-05-14T02:49:58.941+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:49:59.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:49:59 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:49:59.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:49:59 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:50:00.229 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:49:59 smithi115 conmon[90393]: 2022-05-14T02:49:59.929+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:50:00.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:50:00 smithi047 conmon[77123]: 2022-05-14T02:49:59.999+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : Health detail: HEALTH_ERR 1 filesystem is degraded; 1 mds daemon damaged; 1 MDSs are read only; 1 MDSs report slow metadata IOs; 1/3 mons down, quorum b,c; Reduced data availability: 1 pg inactive; 9 slow ops, oldest one blocked for 12130 sec, osd.5 has slow ops 2022-05-14T02:50:00.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:50:00 smithi047 conmon[77123]: 2022-05-14T02:49:59.999+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : [WRN] FS_DEGRADED: 1 filesystem is degraded 2022-05-14T02:50:00.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:50:00 smithi047 conmon[77123]: 2022-05-14T02:49:59.999+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : fs cephfs is degraded 2022-05-14T02:50:00.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:50:00 smithi047 conmon[77123]: 2022-05-14T02:49:59.999+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : [ERR] MDS_DAMAGE: 1 mds daemon damaged 2022-05-14T02:50:00.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:50:00 smithi047 conmon[77123]: 2022-05-14T02:49:59.999+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : fs cephfs mds.2 is damaged 2022-05-14T02:50:00.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:50:00 smithi047 conmon[77123]: 2022-05-14T02:49:59.999+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : [WRN] MDS_READ_ONLY: 1 MDSs are read only 2022-05-14T02:50:00.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:50:00 smithi047 conmon[77123]: 2022-05-14T02:49:59.999+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : mds.l(mds.0): MDS in read-only mode 2022-05-14T02:50:00.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:50:00 smithi047 conmon[77123]: 2022-05-14T02:50:00.000+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : [WRN] MDS_SLOW_METADATA_IO: 1 MDSs report slow metadata IOs 2022-05-14T02:50:00.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:50:00 smithi047 conmon[77123]: 2022-05-14T02:50:00.000+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : mds.l(mds.0): 4 slow metadata IOs are blocked > 180 secs, oldest blocked for 12433 secs 2022-05-14T02:50:00.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:50:00 smithi047 conmon[77123]: 2022-05-14T02:50:00.000+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : [WRN] MON_DOWN: 1/3 mons down, quorum b,c 2022-05-14T02:50:00.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:50:00 smithi047 conmon[77123]: 2022-05-14T02:50:00.000+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : mon.a (rank 0) addr [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] is down (out of quorum) 2022-05-14T02:50:00.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:50:00 smithi047 conmon[77123]: 2022-05-14T02:50:00.000+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : [WRN] PG_AVAILABILITY: Reduced data availability: 1 pg inactive 2022-05-14T02:50:00.295 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:50:00 smithi047 conmon[77123]: 2022-05-14T02:50:00.000+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : pg 2.1 is stuck inactive for 3h, current state unknown, last acting [] 2022-05-14T02:50:00.295 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:50:00 smithi047 conmon[77123]: 2022-05-14T02:50:00.000+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : [WRN] SLOW_OPS: 9 slow ops, oldest one blocked for 12130 sec, osd.5 has slow ops 2022-05-14T02:50:00.295 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:50:00 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:50:00.295 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:50:00 smithi047 ceph-mon[77147]: pgmap v6418: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:50:00.296 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:50:00 smithi047 ceph-mon[77147]: Health detail: HEALTH_ERR 1 filesystem is degraded; 1 mds daemon damaged; 1 MDSs are read only; 1 MDSs report slow metadata IOs; 1/3 mons down, quorum b,c; Reduced data availability: 1 pg inactive; 9 slow ops, oldest one blocked for 12130 sec, osd.5 has slow ops 2022-05-14T02:50:00.296 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:50:00 smithi047 ceph-mon[77147]: [WRN] FS_DEGRADED: 1 filesystem is degraded 2022-05-14T02:50:00.296 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:50:00 smithi047 ceph-mon[77147]: fs cephfs is degraded 2022-05-14T02:50:00.296 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:50:00 smithi047 ceph-mon[77147]: [ERR] MDS_DAMAGE: 1 mds daemon damaged 2022-05-14T02:50:00.297 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:50:00 smithi047 ceph-mon[77147]: fs cephfs mds.2 is damaged 2022-05-14T02:50:00.297 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:50:00 smithi047 ceph-mon[77147]: [WRN] MDS_READ_ONLY: 1 MDSs are read only 2022-05-14T02:50:00.297 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:50:00 smithi047 ceph-mon[77147]: mds.l(mds.0): MDS in read-only mode 2022-05-14T02:50:00.297 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:50:00 smithi047 ceph-mon[77147]: [WRN] MDS_SLOW_METADATA_IO: 1 MDSs report slow metadata IOs 2022-05-14T02:50:00.298 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:50:00 smithi047 ceph-mon[77147]: mds.l(mds.0): 4 slow metadata IOs are blocked > 180 secs, oldest blocked for 12433 secs 2022-05-14T02:50:00.298 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:50:00 smithi047 ceph-mon[77147]: [WRN] MON_DOWN: 1/3 mons down, quorum b,c 2022-05-14T02:50:00.298 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:50:00 smithi047 ceph-mon[77147]: mon.a (rank 0) addr [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] is down (out of quorum) 2022-05-14T02:50:00.298 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:50:00 smithi047 ceph-mon[77147]: [WRN] PG_AVAILABILITY: Reduced data availability: 1 pg inactive 2022-05-14T02:50:00.298 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:50:00 smithi047 ceph-mon[77147]: pg 2.1 is stuck inactive for 3h, current state unknown, last acting [] 2022-05-14T02:50:00.299 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:50:00 smithi047 ceph-mon[77147]: [WRN] SLOW_OPS: 9 slow ops, oldest one blocked for 12130 sec, osd.5 has slow ops 2022-05-14T02:50:00.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:50:00 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:50:00.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:50:00 smithi115 ceph-mon[78593]: pgmap v6418: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:50:00.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:50:00 smithi115 ceph-mon[78593]: Health detail: HEALTH_ERR 1 filesystem is degraded; 1 mds daemon damaged; 1 MDSs are read only; 1 MDSs report slow metadata IOs; 1/3 mons down, quorum b,c; Reduced data availability: 1 pg inactive; 9 slow ops, oldest one blocked for 12130 sec, osd.5 has slow ops 2022-05-14T02:50:00.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:50:00 smithi115 ceph-mon[78593]: [WRN] FS_DEGRADED: 1 filesystem is degraded 2022-05-14T02:50:00.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:50:00 smithi115 ceph-mon[78593]: fs cephfs is degraded 2022-05-14T02:50:00.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:50:00 smithi115 ceph-mon[78593]: [ERR] MDS_DAMAGE: 1 mds daemon damaged 2022-05-14T02:50:00.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:50:00 smithi115 ceph-mon[78593]: fs cephfs mds.2 is damaged 2022-05-14T02:50:00.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:50:00 smithi115 ceph-mon[78593]: [WRN] MDS_READ_ONLY: 1 MDSs are read only 2022-05-14T02:50:00.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:50:00 smithi115 ceph-mon[78593]: mds.l(mds.0): MDS in read-only mode 2022-05-14T02:50:00.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:50:00 smithi115 ceph-mon[78593]: [WRN] MDS_SLOW_METADATA_IO: 1 MDSs report slow metadata IOs 2022-05-14T02:50:00.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:50:00 smithi115 ceph-mon[78593]: mds.l(mds.0): 4 slow metadata IOs are blocked > 180 secs, oldest blocked for 12433 secs 2022-05-14T02:50:00.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:50:00 smithi115 ceph-mon[78593]: [WRN] MON_DOWN: 1/3 mons down, quorum b,c 2022-05-14T02:50:00.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:50:00 smithi115 ceph-mon[78593]: mon.a (rank 0) addr [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] is down (out of quorum) 2022-05-14T02:50:00.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:50:00 smithi115 ceph-mon[78593]: [WRN] PG_AVAILABILITY: Reduced data availability: 1 pg inactive 2022-05-14T02:50:00.612 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:50:00 smithi115 ceph-mon[78593]: pg 2.1 is stuck inactive for 3h, current state unknown, last acting [] 2022-05-14T02:50:00.612 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:50:00 smithi115 ceph-mon[78593]: [WRN] SLOW_OPS: 9 slow ops, oldest one blocked for 12130 sec, osd.5 has slow ops 2022-05-14T02:50:01.234 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:50:00 smithi115 conmon[90393]: 2022-05-14T02:50:00.977+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:50:01.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:50:01 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:50:01.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:50:01 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:50:02.235 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:50:01 smithi115 conmon[90393]: 2022-05-14T02:50:01.951+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:50:02.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:50:02 smithi047 ceph-mon[77147]: pgmap v6419: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:50:02.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:50:02 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:50:02.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:50:02 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 12135 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:50:02.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:50:02 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:50:02.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:50:02 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:50:02.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:50:02 smithi115 ceph-mon[78593]: pgmap v6419: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:50:02.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:50:02 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:50:02.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:50:02 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 12135 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:50:02.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:50:02 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:50:02.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:50:02 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:50:03.240 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:50:02 smithi115 conmon[90393]: 2022-05-14T02:50:02.902+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:50:03.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:50:03 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:50:03.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:50:03 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:50:04.242 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:50:03 smithi115 conmon[90393]: 2022-05-14T02:50:03.916+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:50:04.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:50:04 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:50:04.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:50:04 smithi047 ceph-mon[77147]: pgmap v6420: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:50:04.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:50:04 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:50:04.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:50:04 smithi115 ceph-mon[78593]: pgmap v6420: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:50:05.249 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:50:04 smithi115 conmon[90393]: 2022-05-14T02:50:04.914+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:50:05.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:50:05 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:50:05.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:50:05 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:50:06.248 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:50:05 smithi115 conmon[90393]: 2022-05-14T02:50:05.893+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:50:06.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:50:06 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:50:06.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:50:06 smithi047 ceph-mon[77147]: pgmap v6421: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:50:06.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:50:06 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:50:06.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:50:06 smithi115 ceph-mon[78593]: pgmap v6421: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:50:07.253 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:50:06 smithi115 conmon[90393]: 2022-05-14T02:50:06.890+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:50:07.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:50:07 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:50:07.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:50:07 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:50:08.261 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:50:07 smithi115 conmon[90393]: 2022-05-14T02:50:07.905+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:50:08.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:50:08 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:50:08.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:50:08 smithi047 ceph-mon[77147]: pgmap v6422: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:50:08.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:50:08 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:50:08.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:50:08 smithi115 ceph-mon[78593]: pgmap v6422: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:50:09.260 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:50:08 smithi115 conmon[90393]: 2022-05-14T02:50:08.896+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:50:09.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:50:09 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:50:09.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:50:09 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:50:10.261 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:50:09 smithi115 conmon[90393]: 2022-05-14T02:50:09.915+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:50:10.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:50:10 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:50:10.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:50:10 smithi047 ceph-mon[77147]: pgmap v6423: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:50:10.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:50:10 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:50:10.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:50:10 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:50:10.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:50:10 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:50:10.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:50:10 smithi115 ceph-mon[78593]: pgmap v6423: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:50:10.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:50:10 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:50:10.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:50:10 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:50:11.266 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:50:10 smithi115 conmon[90393]: 2022-05-14T02:50:10.923+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:50:11.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:50:11 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:50:11.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:50:11 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 12140 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:50:11.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:50:11 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:50:11.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:50:11 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 12140 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:50:12.267 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:50:11 smithi115 conmon[90393]: 2022-05-14T02:50:11.942+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:50:12.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:50:12 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:50:12.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:50:12 smithi047 ceph-mon[77147]: pgmap v6424: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:50:12.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:50:12 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:50:12.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:50:12 smithi115 ceph-mon[78593]: pgmap v6424: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:50:13.272 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:50:12 smithi115 conmon[90393]: 2022-05-14T02:50:12.962+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:50:13.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:50:13 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:50:13.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:50:13 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:50:14.274 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:50:13 smithi115 conmon[90393]: 2022-05-14T02:50:13.946+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:50:14.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:50:14 smithi047 ceph-mon[77147]: pgmap v6425: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:50:14.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:50:14 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:50:14.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:50:14 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:50:14.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:50:14 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:50:14.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:50:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T02:50:14.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:50:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T02:50:14.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:50:14 smithi115 ceph-mon[78593]: pgmap v6425: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:50:14.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:50:14 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:50:14.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:50:14 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:50:14.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:50:14 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:50:14.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:50:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T02:50:14.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:50:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T02:50:15.278 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:50:14 smithi115 conmon[90393]: 2022-05-14T02:50:14.976+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:50:15.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:50:15 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:50:15.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:50:15 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:50:16.280 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:50:15 smithi115 conmon[90393]: 2022-05-14T02:50:15.973+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:50:16.646 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:50:16 smithi047 ceph-mon[77147]: pgmap v6426: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:50:16.646 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:50:16 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:50:16.647 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:50:16 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 12150 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:50:16.647 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:50:16 smithi115 ceph-mon[78593]: pgmap v6426: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:50:16.648 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:50:16 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:50:16.648 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:50:16 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 12150 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:50:17.285 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:50:16 smithi115 conmon[90393]: 2022-05-14T02:50:16.953+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:50:17.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:50:17 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:50:17.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:50:17 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:50:18.287 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:50:17 smithi115 conmon[90393]: 2022-05-14T02:50:17.937+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:50:18.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:50:18 smithi047 ceph-mon[77147]: pgmap v6427: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:50:18.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:50:18 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:50:18.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:50:18 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:50:18.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:50:18 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:50:18.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:50:18 smithi115 ceph-mon[78593]: pgmap v6427: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:50:18.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:50:18 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:50:18.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:50:18 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:50:18.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:50:18 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:50:19.292 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:50:18 smithi115 conmon[90393]: 2022-05-14T02:50:18.932+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:50:19.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:50:19 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:50:19.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:50:19 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T02:50:19.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:50:19 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T02:50:19.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:50:19 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T02:50:19.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:50:19 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:50:19.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:50:19 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T02:50:19.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:50:19 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T02:50:19.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:50:19 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T02:50:20.041 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:50:19 smithi047 conmon[78521]: 2022-05-14T02:50:19.631+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:50:20.042 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:50:19 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T02:50:20.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:50:19 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:50:20.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:50:19 smithi047 conmon[78521]: yield 2022-05-14T02:50:20.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:50:19 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:50:20.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:50:19 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:50:20.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:50:19 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:50:20.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:50:19 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T02:50:20.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:50:19 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:50:20.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:50:19 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T02:50:20.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:50:19 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:50:20.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:50:19 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T02:50:20.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:50:19 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:50:20.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:50:19 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T02:50:20.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:50:19 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:50:20.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:50:19 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T02:50:20.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:50:19 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:50:20.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:50:19 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:50:20.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:50:19 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:50:20.292 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:50:19 smithi115 conmon[90393]: 2022-05-14T02:50:19.910+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:50:20.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:50:20 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:50:20.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:50:20 smithi115 ceph-mon[78593]: pgmap v6428: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:50:20.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:50:20 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:50:20.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:50:20 smithi047 ceph-mon[77147]: pgmap v6428: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:50:21.301 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:50:20 smithi115 conmon[90393]: 2022-05-14T02:50:20.927+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:50:21.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:50:21 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:50:21.609 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T02:50:21.609 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:50:21.609 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T02:50:21.609 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:50:21.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:50:21.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:50:21.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T02:50:21.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:50:21.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T02:50:21.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:50:21.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T02:50:21.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:50:21.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T02:50:21.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:50:21.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T02:50:21.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:50:21.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:50:21.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:50:21.613 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:50:21 smithi115 ceph-mon[78593]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T02:50:21.613 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:50:21 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:50:21.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:50:21 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:50:21.792 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T02:50:21.792 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:50:21.793 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T02:50:21.793 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:50:21.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:50:21.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:50:21.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T02:50:21.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:50:21.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T02:50:21.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:50:21.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T02:50:21.797 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:50:21.797 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T02:50:21.798 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:50:21.798 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T02:50:21.798 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:50:21.799 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:50:21.799 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:50:21.800 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:50:21 smithi047 ceph-mon[77147]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T02:50:21.800 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:50:21 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:50:22.300 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:50:21 smithi115 conmon[90393]: 2022-05-14T02:50:21.884+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:50:22.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:50:22 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:50:22.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:50:22 smithi115 ceph-mon[78593]: pgmap v6429: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:50:22.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:50:22 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 12155 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:50:22.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:50:22 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:50:22.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:50:22 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:50:22.705 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:50:22 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:50:22.706 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:50:22 smithi047 ceph-mon[77147]: pgmap v6429: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:50:22.706 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:50:22 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 12155 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:50:22.706 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:50:22 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:50:22.707 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:50:22 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:50:23.042 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:50:22 smithi047 conmon[78521]: 2022-05-14T02:50:22.703+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:50:23.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:50:22 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T02:50:23.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:50:22 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:50:23.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:50:22 smithi047 conmon[78521]: yield 2022-05-14T02:50:23.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:50:22 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:50:23.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:50:22 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:50:23.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:50:22 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:50:23.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:50:22 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T02:50:23.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:50:22 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:50:23.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:50:22 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T02:50:23.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:50:22 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:50:23.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:50:22 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T02:50:23.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:50:22 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:50:23.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:50:22 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T02:50:23.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:50:22 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:50:23.048 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:50:22 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T02:50:23.048 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:50:22 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:50:23.048 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:50:22 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:50:23.049 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:50:22 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:50:23.049 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:50:22 smithi047 conmon[78521]: 2022-05-14T02:50:22.704+0000 7f5340779700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T02:50:23.049 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:50:22 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T02:50:23.050 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:50:22 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:50:23.050 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:50:22 smithi047 conmon[78521]: yield 2022-05-14T02:50:23.050 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:50:22 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:50:23.051 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:50:22 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:50:23.051 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:50:22 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:50:23.051 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:50:22 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T02:50:23.052 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:50:22 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:50:23.052 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:50:22 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T02:50:23.052 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:50:22 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:50:23.052 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:50:22 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T02:50:23.053 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:50:22 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:50:23.053 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:50:22 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T02:50:23.053 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:50:22 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:50:23.054 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:50:22 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T02:50:23.054 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:50:22 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:50:23.054 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:50:22 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:50:23.055 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:50:22 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:50:23.055 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:50:22 smithi047 conmon[78521]: 2022-05-14T02:50:23.055 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:50:22 smithi047 conmon[78521]: During handling of the above exception, another exception occurred: 2022-05-14T02:50:23.056 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:50:22 smithi047 conmon[78521]: 2022-05-14T02:50:23.056 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:50:22 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T02:50:23.056 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:50:22 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T02:50:23.057 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:50:22 smithi047 conmon[78521]: return f(*arg) 2022-05-14T02:50:23.057 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:50:22 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T02:50:23.057 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:50:22 smithi047 conmon[78521]: self._write_client_files(client_files, host) 2022-05-14T02:50:23.058 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:50:22 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T02:50:23.058 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:50:22 smithi047 conmon[78521]: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T02:50:23.058 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:50:22 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T02:50:23.059 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:50:22 smithi047 conmon[78521]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T02:50:23.059 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:50:22 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T02:50:23.059 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:50:22 smithi047 conmon[78521]: return self.event_loop.get_result(coro) 2022-05-14T02:50:23.060 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:50:22 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T02:50:23.060 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:50:22 smithi047 conmon[78521]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T02:50:23.060 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:50:22 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T02:50:23.060 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:50:22 smithi047 conmon[78521]: return self.__get_result() 2022-05-14T02:50:23.061 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:50:22 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T02:50:23.061 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:50:22 smithi047 conmon[78521]: raise self._exception 2022-05-14T02:50:23.062 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:50:22 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T02:50:23.062 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:50:22 smithi047 conmon[78521]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T02:50:23.062 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:50:22 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T02:50:23.062 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:50:22 smithi047 conmon[78521]: conn = await self._remote_connection(host, addr) 2022-05-14T02:50:23.063 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:50:22 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T02:50:23.063 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:50:22 smithi047 conmon[78521]: raise 2022-05-14T02:50:23.063 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:50:22 smithi047 conmon[78521]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T02:50:23.064 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:50:22 smithi047 conmon[78521]: self.gen.throw(type, value, traceback) 2022-05-14T02:50:23.064 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:50:22 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T02:50:23.064 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:50:22 smithi047 conmon[78521]: raise OrchestratorError(msg) 2022-05-14T02:50:23.065 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:50:22 smithi047 conmon[78521]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:50:23.304 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:50:22 smithi115 conmon[90393]: 2022-05-14T02:50:22.895+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:50:23.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:50:23 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:50:23.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:50:23 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:50:24.307 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:50:23 smithi115 conmon[90393]: 2022-05-14T02:50:23.879+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:50:24.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:50:24 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:50:24.609 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T02:50:24.609 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:50:24.609 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T02:50:24.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:50:24.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:50:24.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:50:24.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T02:50:24.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:50:24.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T02:50:24.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:50:24.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T02:50:24.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:50:24.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T02:50:24.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:50:24.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T02:50:24.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:50:24.613 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:50:24.613 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:50:24.613 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:50:24 smithi115 ceph-mon[78593]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T02:50:24.613 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T02:50:24.613 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:50:24.614 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T02:50:24.614 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:50:24.614 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:50:24.615 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:50:24.615 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T02:50:24.615 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:50:24.615 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T02:50:24.615 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:50:24.616 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T02:50:24.616 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:50:24.616 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T02:50:24.616 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:50:24.617 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T02:50:24.617 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:50:24.617 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:50:24.617 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:50:24.618 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T02:50:24.618 INFO:journalctl@ceph.mon.c.smithi115.stdout: During handling of the above exception, another exception occurred: 2022-05-14T02:50:24.618 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T02:50:24.618 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T02:50:24.619 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T02:50:24.619 INFO:journalctl@ceph.mon.c.smithi115.stdout: return f(*arg) 2022-05-14T02:50:24.619 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T02:50:24.619 INFO:journalctl@ceph.mon.c.smithi115.stdout: self._write_client_files(client_files, host) 2022-05-14T02:50:24.619 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T02:50:24.620 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T02:50:24.620 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T02:50:24.620 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T02:50:24.620 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T02:50:24.620 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.event_loop.get_result(coro) 2022-05-14T02:50:24.621 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T02:50:24.621 INFO:journalctl@ceph.mon.c.smithi115.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T02:50:24.621 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T02:50:24.621 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.__get_result() 2022-05-14T02:50:24.622 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T02:50:24.622 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise self._exception 2022-05-14T02:50:24.622 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T02:50:24.622 INFO:journalctl@ceph.mon.c.smithi115.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T02:50:24.623 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T02:50:24.623 INFO:journalctl@ceph.mon.c.smithi115.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T02:50:24.623 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T02:50:24.623 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise 2022-05-14T02:50:24.624 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T02:50:24.624 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.gen.throw(type, value, traceback) 2022-05-14T02:50:24.624 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T02:50:24.624 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OrchestratorError(msg) 2022-05-14T02:50:24.625 INFO:journalctl@ceph.mon.c.smithi115.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:50:24.625 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:50:24 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:50:24.625 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:50:24 smithi115 ceph-mon[78593]: pgmap v6430: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:50:24.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:50:24 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:50:24.792 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T02:50:24.792 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:50:24.792 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T02:50:24.792 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:50:24.793 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:50:24.793 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:50:24.793 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T02:50:24.793 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:50:24.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T02:50:24.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:50:24.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T02:50:24.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:50:24.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T02:50:24.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:50:24.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T02:50:24.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:50:24.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:50:24.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:50:24.796 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:50:24 smithi047 ceph-mon[77147]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T02:50:24.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T02:50:24.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:50:24.797 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T02:50:24.797 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:50:24.797 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:50:24.797 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:50:24.798 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T02:50:24.798 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:50:24.798 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T02:50:24.798 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:50:24.798 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T02:50:24.799 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:50:24.799 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T02:50:24.799 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:50:24.799 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T02:50:24.800 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:50:24.800 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:50:24.800 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:50:24.800 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T02:50:24.800 INFO:journalctl@ceph.mon.b.smithi047.stdout: During handling of the above exception, another exception occurred: 2022-05-14T02:50:24.801 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T02:50:24.801 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T02:50:24.801 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T02:50:24.801 INFO:journalctl@ceph.mon.b.smithi047.stdout: return f(*arg) 2022-05-14T02:50:24.802 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T02:50:24.802 INFO:journalctl@ceph.mon.b.smithi047.stdout: self._write_client_files(client_files, host) 2022-05-14T02:50:24.802 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T02:50:24.802 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T02:50:24.802 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T02:50:24.803 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T02:50:24.803 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T02:50:24.803 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.event_loop.get_result(coro) 2022-05-14T02:50:24.803 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T02:50:24.803 INFO:journalctl@ceph.mon.b.smithi047.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T02:50:24.804 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T02:50:24.804 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.__get_result() 2022-05-14T02:50:24.804 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T02:50:24.804 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise self._exception 2022-05-14T02:50:24.805 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T02:50:24.805 INFO:journalctl@ceph.mon.b.smithi047.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T02:50:24.805 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T02:50:24.806 INFO:journalctl@ceph.mon.b.smithi047.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T02:50:24.806 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T02:50:24.806 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise 2022-05-14T02:50:24.806 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T02:50:24.806 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.gen.throw(type, value, traceback) 2022-05-14T02:50:24.807 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T02:50:24.807 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OrchestratorError(msg) 2022-05-14T02:50:24.807 INFO:journalctl@ceph.mon.b.smithi047.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:50:24.808 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:50:24 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:50:24.808 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:50:24 smithi047 ceph-mon[77147]: pgmap v6430: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:50:25.312 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:50:24 smithi115 conmon[90393]: 2022-05-14T02:50:24.907+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:50:25.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:50:25 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:50:25.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:50:25 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:50:26.314 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:50:25 smithi115 conmon[90393]: 2022-05-14T02:50:25.927+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:50:26.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:50:26 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:50:26.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:50:26 smithi115 ceph-mon[78593]: pgmap v6431: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:50:26.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:50:26 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:50:26.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:50:26 smithi047 ceph-mon[77147]: pgmap v6431: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:50:27.318 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:50:26 smithi115 conmon[90393]: 2022-05-14T02:50:26.931+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:50:27.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:50:27 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:50:27.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:50:27 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:50:28.319 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:50:27 smithi115 conmon[90393]: 2022-05-14T02:50:27.944+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:50:28.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:50:28 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:50:28.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:50:28 smithi115 ceph-mon[78593]: pgmap v6432: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:50:28.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:50:28 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:50:28.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:50:28 smithi047 ceph-mon[77147]: pgmap v6432: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:50:29.325 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:50:28 smithi115 conmon[90393]: 2022-05-14T02:50:28.986+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:50:29.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:50:29 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:50:29.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:50:29 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:50:30.326 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:50:30 smithi115 conmon[90393]: 2022-05-14T02:50:29.999+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:50:30.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:50:30 smithi115 ceph-mon[78593]: pgmap v6433: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:50:30.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:50:30 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:50:30.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:50:30 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:50:30.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:50:30 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:50:30.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:50:30 smithi047 ceph-mon[77147]: pgmap v6433: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:50:30.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:50:30 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:50:30.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:50:30 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:50:30.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:50:30 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:50:31.335 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:50:31 smithi115 conmon[90393]: 2022-05-14T02:50:31.040+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:50:31.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:50:31 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:50:31.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:50:31 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 12160 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:50:31.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:50:31 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:50:31.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:50:31 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 12160 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:50:32.334 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:50:32 smithi115 conmon[90393]: 2022-05-14T02:50:32.061+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:50:32.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:50:32 smithi115 ceph-mon[78593]: pgmap v6434: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:50:32.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:50:32 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:50:32.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:50:32 smithi047 ceph-mon[77147]: pgmap v6434: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:50:32.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:50:32 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:50:33.343 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:50:33 smithi115 conmon[90393]: 2022-05-14T02:50:33.082+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:50:33.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:50:33 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:50:33.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:50:33 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:50:34.343 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:50:34 smithi115 conmon[90393]: 2022-05-14T02:50:34.066+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:50:34.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:50:34 smithi115 ceph-mon[78593]: pgmap v6435: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:50:34.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:50:34 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:50:34.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:50:34 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:50:34.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:50:34 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:50:34.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:50:34 smithi047 ceph-mon[77147]: pgmap v6435: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:50:34.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:50:34 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:50:34.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:50:34 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:50:34.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:50:34 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:50:35.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:50:35 smithi115 conmon[90393]: 2022-05-14T02:50:35.114+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:50:35.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:50:35 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:50:35.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:50:35 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:50:36.349 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:50:36 smithi115 conmon[90393]: 2022-05-14T02:50:36.081+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:50:36.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:50:36 smithi115 ceph-mon[78593]: pgmap v6436: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:50:36.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:50:36 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:50:36.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:50:36 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 12169 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:50:36.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:50:36 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:50:36.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:50:36 smithi047 ceph-mon[77147]: pgmap v6436: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:50:36.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:50:36 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:50:36.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:50:36 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 12169 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:50:36.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:50:36 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:50:37.356 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:50:37 smithi115 conmon[90393]: 2022-05-14T02:50:37.047+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:50:37.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:50:37 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:50:37.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:50:37 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:50:38.356 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:50:38 smithi115 conmon[90393]: 2022-05-14T02:50:38.041+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:50:38.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:50:38 smithi115 ceph-mon[78593]: pgmap v6437: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:50:38.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:50:38 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:50:38.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:50:38 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:50:38.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:50:38 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:50:38.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:50:38 smithi047 ceph-mon[77147]: pgmap v6437: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:50:38.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:50:38 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:50:38.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:50:38 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:50:38.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:50:38 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:50:39.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:50:39 smithi115 conmon[90393]: 2022-05-14T02:50:39.072+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:50:39.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:50:39 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:50:39.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:50:39 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:50:40.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:50:40 smithi115 conmon[90393]: 2022-05-14T02:50:40.077+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:50:40.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:50:40 smithi047 ceph-mon[77147]: pgmap v6438: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:50:40.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:50:40 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:50:40.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:50:40 smithi115 ceph-mon[78593]: pgmap v6438: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:50:40.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:50:40 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:50:41.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:50:41 smithi115 conmon[90393]: 2022-05-14T02:50:41.031+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:50:42.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:50:41 smithi047 ceph-mon[77147]: pgmap v6439: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:50:42.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:50:41 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:50:42.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:50:41 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 12174 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:50:42.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:50:41 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:50:42.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:50:41 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:50:42.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:50:41 smithi115 conmon[90393]: 2022-05-14T02:50:41.997+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:50:42.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:50:41 smithi115 ceph-mon[78593]: pgmap v6439: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:50:42.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:50:41 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:50:42.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:50:41 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 12174 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:50:42.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:50:41 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:50:42.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:50:41 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:50:43.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:50:43 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:50:43.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:50:43 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:50:43.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:50:43 smithi115 conmon[90393]: 2022-05-14T02:50:43.038+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:50:44.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:50:44 smithi047 ceph-mon[77147]: pgmap v6440: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:50:44.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:50:44 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:50:44.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:50:44 smithi115 conmon[90393]: 2022-05-14T02:50:44.079+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:50:44.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:50:44 smithi115 ceph-mon[78593]: pgmap v6440: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:50:44.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:50:44 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:50:45.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:50:45 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:50:45.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:50:45 smithi115 conmon[90393]: 2022-05-14T02:50:45.037+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:50:45.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:50:45 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:50:46.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:50:46 smithi047 ceph-mon[77147]: pgmap v6441: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:50:46.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:50:46 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:50:46.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:50:45 smithi115 conmon[90393]: 2022-05-14T02:50:45.998+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:50:46.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:50:46 smithi115 ceph-mon[78593]: pgmap v6441: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:50:46.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:50:46 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:50:47.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:50:47 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:50:47.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:50:46 smithi115 conmon[90393]: 2022-05-14T02:50:46.965+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:50:47.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:50:47 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:50:48.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:50:48 smithi047 ceph-mon[77147]: pgmap v6442: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:50:48.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:50:48 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:50:48.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:50:48 smithi115 conmon[90393]: 2022-05-14T02:50:48.003+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:50:48.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:50:48 smithi115 ceph-mon[78593]: pgmap v6442: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:50:48.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:50:48 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:50:49.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:50:49 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:50:49.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:50:48 smithi115 conmon[90393]: 2022-05-14T02:50:48.977+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:50:49.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:50:49 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:50:50.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:50:50 smithi047 ceph-mon[77147]: pgmap v6443: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:50:50.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:50:50 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:50:50.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:50:50 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:50:50.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:50:50 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:50:50.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:50:49 smithi115 conmon[90393]: 2022-05-14T02:50:49.966+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:50:50.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:50:50 smithi115 ceph-mon[78593]: pgmap v6443: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:50:50.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:50:50 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:50:50.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:50:50 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:50:50.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:50:50 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:50:51.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:50:51 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:50:51.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:50:51 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 12179 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:50:51.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:50:51 smithi115 conmon[90393]: 2022-05-14T02:50:51.007+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:50:51.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:50:51 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:50:51.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:50:51 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 12179 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:50:52.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:50:52 smithi047 ceph-mon[77147]: pgmap v6444: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:50:52.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:50:52 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:50:52.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:50:51 smithi115 conmon[90393]: 2022-05-14T02:50:51.997+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:50:52.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:50:52 smithi115 ceph-mon[78593]: pgmap v6444: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:50:52.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:50:52 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:50:53.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:50:53 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:50:53.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:50:53 smithi115 conmon[90393]: 2022-05-14T02:50:53.011+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:50:53.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:50:53 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:50:54.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:50:54 smithi047 ceph-mon[77147]: pgmap v6445: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:50:54.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:50:54 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:50:54.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:50:54 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:50:54.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:50:54 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:50:54.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:50:54 smithi115 conmon[90393]: 2022-05-14T02:50:54.058+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:50:54.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:50:54 smithi115 ceph-mon[78593]: pgmap v6445: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:50:54.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:50:54 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:50:54.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:50:54 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:50:54.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:50:54 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:50:55.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:50:55 smithi115 conmon[90393]: 2022-05-14T02:50:55.015+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:50:55.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:50:55 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:50:55.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:50:55 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:50:56.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:50:56 smithi115 conmon[90393]: 2022-05-14T02:50:56.063+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:50:56.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:50:56 smithi115 ceph-mon[78593]: pgmap v6446: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:50:56.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:50:56 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:50:56.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:50:56 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 12189 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:50:56.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:50:56 smithi047 ceph-mon[77147]: pgmap v6446: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:50:56.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:50:56 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:50:56.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:50:56 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 12189 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:50:57.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:50:57 smithi115 conmon[90393]: 2022-05-14T02:50:57.058+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:50:57.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:50:57 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:50:57.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:50:57 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:50:58.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:50:58 smithi115 conmon[90393]: 2022-05-14T02:50:58.034+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:50:58.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:50:58 smithi115 ceph-mon[78593]: pgmap v6447: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:50:58.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:50:58 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:50:58.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:50:58 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:50:58.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:50:58 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:50:58.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:50:58 smithi047 ceph-mon[77147]: pgmap v6447: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:50:58.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:50:58 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:50:58.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:50:58 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:50:58.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:50:58 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:50:59.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:50:59 smithi115 conmon[90393]: 2022-05-14T02:50:59.008+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:50:59.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:50:59 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:50:59.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:50:59 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:51:00.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:51:00 smithi115 conmon[90393]: 2022-05-14T02:51:00.045+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:51:00.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:51:00 smithi115 ceph-mon[78593]: pgmap v6448: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:51:00.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:51:00 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:51:00.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:51:00 smithi047 ceph-mon[77147]: pgmap v6448: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:51:00.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:51:00 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:51:01.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:51:01 smithi115 conmon[90393]: 2022-05-14T02:51:01.063+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:51:01.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:51:01 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:51:01.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:51:01 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:51:02.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:51:02 smithi115 conmon[90393]: 2022-05-14T02:51:02.088+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:51:02.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:51:02 smithi115 ceph-mon[78593]: pgmap v6449: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:51:02.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:51:02 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 12195 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:51:02.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:51:02 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:51:02.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:51:02 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:51:02.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:51:02 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:51:02.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:51:02 smithi047 ceph-mon[77147]: pgmap v6449: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:51:02.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:51:02 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 12195 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:51:02.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:51:02 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:51:02.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:51:02 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:51:02.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:51:02 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:51:03.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:51:03 smithi115 conmon[90393]: 2022-05-14T02:51:03.065+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:51:03.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:51:03 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:51:03.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:51:03 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:51:04.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:51:04 smithi115 conmon[90393]: 2022-05-14T02:51:04.073+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:51:04.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:51:04 smithi115 ceph-mon[78593]: pgmap v6450: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:51:04.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:51:04 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:51:04.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:51:04 smithi047 ceph-mon[77147]: pgmap v6450: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:51:04.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:51:04 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:51:05.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:51:05 smithi115 conmon[90393]: 2022-05-14T02:51:05.113+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:51:05.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:51:05 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:51:05.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:51:05 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:51:06.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:51:06 smithi115 conmon[90393]: 2022-05-14T02:51:06.158+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:51:06.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:51:06 smithi115 ceph-mon[78593]: pgmap v6451: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:51:06.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:51:06 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:51:06.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:51:06 smithi047 ceph-mon[77147]: pgmap v6451: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:51:06.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:51:06 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:51:07.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:51:07 smithi115 conmon[90393]: 2022-05-14T02:51:07.150+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:51:07.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:51:07 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:51:07.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:51:07 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:51:08.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:51:08 smithi115 conmon[90393]: 2022-05-14T02:51:08.192+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:51:08.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:51:08 smithi115 ceph-mon[78593]: pgmap v6452: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:51:08.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:51:08 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:51:08.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:51:08 smithi047 ceph-mon[77147]: pgmap v6452: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:51:08.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:51:08 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:51:09.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:51:09 smithi115 conmon[90393]: 2022-05-14T02:51:09.236+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:51:09.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:51:09 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:51:09.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:51:09 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:51:10.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:51:10 smithi115 conmon[90393]: 2022-05-14T02:51:10.234+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:51:10.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:51:10 smithi115 ceph-mon[78593]: pgmap v6453: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:51:10.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:51:10 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:51:10.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:51:10 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:51:10.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:51:10 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:51:10.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:51:10 smithi047 ceph-mon[77147]: pgmap v6453: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:51:10.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:51:10 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:51:10.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:51:10 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:51:10.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:51:10 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:51:11.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:51:11 smithi115 conmon[90393]: 2022-05-14T02:51:11.208+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:51:11.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:51:11 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:51:11.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:51:11 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 12199 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:51:11.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:51:11 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:51:11.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:51:11 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 12199 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:51:12.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:51:12 smithi115 conmon[90393]: 2022-05-14T02:51:12.189+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:51:12.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:51:12 smithi115 ceph-mon[78593]: pgmap v6454: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:51:12.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:51:12 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:51:12.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:51:12 smithi047 ceph-mon[77147]: pgmap v6454: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:51:12.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:51:12 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:51:13.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:51:13 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:51:13.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:51:13 smithi115 conmon[90393]: 2022-05-14T02:51:13.178+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:51:13.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:51:13 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:51:14.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:51:14 smithi115 conmon[90393]: 2022-05-14T02:51:14.200+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:51:14.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:51:14 smithi115 ceph-mon[78593]: pgmap v6455: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:51:14.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:51:14 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:51:14.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:51:14 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:51:14.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:51:14 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:51:14.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:51:14 smithi047 ceph-mon[77147]: pgmap v6455: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:51:14.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:51:14 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:51:14.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:51:14 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:51:14.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:51:14 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:51:15.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:51:15 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T02:51:15.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:51:15 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T02:51:15.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:51:15 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:51:15.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:51:15 smithi115 conmon[90393]: 2022-05-14T02:51:15.238+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:51:15.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:51:15 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T02:51:15.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:51:15 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T02:51:15.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:51:15 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:51:16.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:51:16 smithi047 ceph-mon[77147]: pgmap v6456: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:51:16.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:51:16 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:51:16.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:51:16 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 12210 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:51:16.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:51:16 smithi115 conmon[90393]: 2022-05-14T02:51:16.245+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:51:16.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:51:16 smithi115 ceph-mon[78593]: pgmap v6456: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:51:16.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:51:16 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:51:16.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:51:16 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 12210 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:51:17.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:51:17 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:51:17.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:51:17 smithi115 conmon[90393]: 2022-05-14T02:51:17.275+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:51:17.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:51:17 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:51:18.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:51:18 smithi047 ceph-mon[77147]: pgmap v6457: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:51:19.039 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:51:18 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:51:19.039 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:51:18 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:51:19.040 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:51:18 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:51:19.041 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:51:18 smithi115 conmon[90393]: 2022-05-14T02:51:18.309+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:51:19.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:51:18 smithi115 ceph-mon[78593]: pgmap v6457: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:51:19.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:51:18 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:51:19.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:51:18 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:51:19.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:51:18 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:51:19.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:51:19 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:51:19.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:51:19 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:51:19.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:51:19 smithi115 conmon[90393]: 2022-05-14T02:51:19.334+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:51:20.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:51:20 smithi047 ceph-mon[77147]: pgmap v6458: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:51:20.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:51:20 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:51:20.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:51:20 smithi115 conmon[90393]: 2022-05-14T02:51:20.345+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:51:20.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:51:20 smithi115 ceph-mon[78593]: pgmap v6458: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:51:20.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:51:20 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:51:21.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:51:21 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:51:21.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:51:21 smithi115 conmon[90393]: 2022-05-14T02:51:21.343+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:51:21.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:51:21 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:51:22.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:51:22 smithi047 ceph-mon[77147]: pgmap v6459: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:51:22.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:51:22 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 12215 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:51:22.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:51:22 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:51:22.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:51:22 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:51:22.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:51:22 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:51:22.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:51:22 smithi115 conmon[90393]: 2022-05-14T02:51:22.320+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:51:22.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:51:22 smithi115 ceph-mon[78593]: pgmap v6459: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:51:22.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:51:22 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 12215 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:51:22.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:51:22 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:51:22.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:51:22 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:51:22.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:51:22 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:51:23.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:51:23 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:51:23.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:51:23 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T02:51:23.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:51:23 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T02:51:23.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:51:23 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T02:51:23.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:51:23 smithi115 conmon[90393]: 2022-05-14T02:51:23.359+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:51:23.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:51:23 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:51:23.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:51:23 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T02:51:23.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:51:23 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T02:51:23.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:51:23 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T02:51:24.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:51:24 smithi047 ceph-mon[77147]: pgmap v6460: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:51:24.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:51:24 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' 2022-05-14T02:51:24.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:51:24 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' 2022-05-14T02:51:24.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:51:24 smithi115 conmon[90393]: 2022-05-14T02:51:24.354+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:51:24.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:51:24 smithi115 ceph-mon[78593]: pgmap v6460: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:51:24.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:51:24 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' 2022-05-14T02:51:24.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:51:24 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' 2022-05-14T02:51:25.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:51:25 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:51:25.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:51:25 smithi115 conmon[90393]: 2022-05-14T02:51:25.384+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:51:25.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:51:25 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:51:26.154 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:51:25 smithi047 conmon[78521]: 2022-05-14T02:51:25.808+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:51:26.155 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:51:25 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T02:51:26.155 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:51:25 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:51:26.155 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:51:25 smithi047 conmon[78521]: yield 2022-05-14T02:51:26.155 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:51:25 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:51:26.156 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:51:25 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:51:26.156 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:51:25 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:51:26.156 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:51:25 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T02:51:26.156 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:51:25 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:51:26.157 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:51:25 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T02:51:26.157 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:51:25 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:51:26.157 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:51:25 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T02:51:26.157 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:51:25 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:51:26.158 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:51:25 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T02:51:26.158 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:51:25 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:51:26.158 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:51:25 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T02:51:26.159 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:51:25 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:51:26.159 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:51:25 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:51:26.159 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:51:25 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:51:26.431 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:51:26 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:51:26.431 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:51:26 smithi115 ceph-mon[78593]: pgmap v6461: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:51:26.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:51:26 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:51:26.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:51:26 smithi047 ceph-mon[77147]: pgmap v6461: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:51:26.860 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:51:26 smithi115 conmon[90393]: 2022-05-14T02:51:26.429+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:51:27.417 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:51:27 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:51:27.417 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:51:27 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:51:27.418 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T02:51:27.418 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:51:27.418 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T02:51:27.419 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:51:27.419 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:51:27.419 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:51:27.419 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T02:51:27.419 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:51:27.420 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T02:51:27.420 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:51:27.420 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T02:51:27.420 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:51:27.421 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T02:51:27.421 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:51:27.421 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T02:51:27.421 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:51:27.422 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:51:27.422 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:51:27.422 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:51:27 smithi115 ceph-mon[78593]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T02:51:27.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:51:27 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:51:27.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:51:27 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:51:27.542 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T02:51:27.542 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:51:27.542 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T02:51:27.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:51:27.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:51:27.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:51:27.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T02:51:27.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:51:27.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T02:51:27.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:51:27.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T02:51:27.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:51:27.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T02:51:27.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:51:27.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T02:51:27.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:51:27.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:51:27.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:51:27.546 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:51:27 smithi047 ceph-mon[77147]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T02:51:27.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:51:27 smithi115 conmon[90393]: 2022-05-14T02:51:27.415+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:51:28.426 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:51:28 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:51:28.426 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:51:28 smithi115 ceph-mon[78593]: pgmap v6462: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:51:28.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:51:28 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:51:28.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:51:28 smithi047 ceph-mon[77147]: pgmap v6462: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:51:28.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:51:28 smithi115 conmon[90393]: 2022-05-14T02:51:28.424+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:51:29.166 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:51:28 smithi047 conmon[78521]: 2022-05-14T02:51:28.880+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:51:29.166 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:51:28 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T02:51:29.166 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:51:28 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:51:29.167 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:51:28 smithi047 conmon[78521]: yield 2022-05-14T02:51:29.167 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:51:28 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:51:29.167 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:51:28 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:51:29.167 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:51:28 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:51:29.168 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:51:28 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T02:51:29.168 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:51:28 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:51:29.168 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:51:28 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T02:51:29.168 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:51:28 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:51:29.169 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:51:28 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T02:51:29.169 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:51:28 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:51:29.169 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:51:28 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T02:51:29.169 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:51:28 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:51:29.169 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:51:28 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T02:51:29.170 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:51:28 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:51:29.170 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:51:28 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:51:29.170 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:51:28 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:51:29.170 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:51:28 smithi047 conmon[78521]: 2022-05-14T02:51:28.881+0000 7f5344781700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T02:51:29.171 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:51:28 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T02:51:29.171 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:51:28 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:51:29.171 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:51:28 smithi047 conmon[78521]: yield 2022-05-14T02:51:29.171 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:51:28 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:51:29.171 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:51:28 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:51:29.172 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:51:28 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:51:29.172 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:51:28 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T02:51:29.172 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:51:28 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:51:29.172 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:51:28 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T02:51:29.172 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:51:28 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:51:29.173 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:51:28 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T02:51:29.173 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:51:28 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:51:29.173 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:51:28 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T02:51:29.173 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:51:28 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:51:29.174 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:51:28 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T02:51:29.174 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:51:28 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:51:29.174 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:51:28 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:51:29.174 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:51:28 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:51:29.174 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:51:28 smithi047 conmon[78521]: 2022-05-14T02:51:29.175 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:51:28 smithi047 conmon[78521]: During handling of the above exception, another exception occurred: 2022-05-14T02:51:29.175 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:51:28 smithi047 conmon[78521]: 2022-05-14T02:51:29.175 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:51:28 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T02:51:29.175 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:51:28 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T02:51:29.176 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:51:28 smithi047 conmon[78521]: return f(*arg) 2022-05-14T02:51:29.176 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:51:28 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T02:51:29.176 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:51:28 smithi047 conmon[78521]: self._write_client_files(client_files, host) 2022-05-14T02:51:29.176 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:51:28 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T02:51:29.176 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:51:28 smithi047 conmon[78521]: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T02:51:29.177 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:51:28 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T02:51:29.177 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:51:28 smithi047 conmon[78521]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T02:51:29.177 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:51:28 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T02:51:29.177 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:51:28 smithi047 conmon[78521]: return self.event_loop.get_result(coro) 2022-05-14T02:51:29.177 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:51:28 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T02:51:29.178 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:51:28 smithi047 conmon[78521]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T02:51:29.178 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:51:28 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T02:51:29.178 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:51:28 smithi047 conmon[78521]: return self.__get_result() 2022-05-14T02:51:29.178 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:51:28 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T02:51:29.179 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:51:28 smithi047 conmon[78521]: raise self._exception 2022-05-14T02:51:29.179 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:51:28 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T02:51:29.179 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:51:28 smithi047 conmon[78521]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T02:51:29.179 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:51:28 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T02:51:29.179 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:51:28 smithi047 conmon[78521]: conn = await self._remote_connection(host, addr) 2022-05-14T02:51:29.180 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:51:28 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T02:51:29.180 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:51:28 smithi047 conmon[78521]: raise 2022-05-14T02:51:29.180 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:51:28 smithi047 conmon[78521]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T02:51:29.180 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:51:28 smithi047 conmon[78521]: self.gen.throw(type, value, traceback) 2022-05-14T02:51:29.181 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:51:28 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T02:51:29.181 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:51:28 smithi047 conmon[78521]: raise OrchestratorError(msg) 2022-05-14T02:51:29.181 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:51:28 smithi047 conmon[78521]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:51:29.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:51:29 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:51:29.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:51:29 smithi115 conmon[90393]: 2022-05-14T02:51:29.396+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:51:29.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:51:29 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:51:30.424 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:51:30 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:51:30.424 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:51:30 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:51:30.424 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T02:51:30.425 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:51:30.425 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T02:51:30.425 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:51:30.425 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:51:30.426 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:51:30.426 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T02:51:30.426 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:51:30.426 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T02:51:30.426 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:51:30.427 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T02:51:30.427 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:51:30.427 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T02:51:30.427 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:51:30.427 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T02:51:30.428 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:51:30.428 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:51:30.428 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:51:30.428 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:51:30 smithi115 ceph-mon[78593]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T02:51:30.429 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T02:51:30.429 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:51:30.429 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T02:51:30.429 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:51:30.429 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:51:30.430 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:51:30.430 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T02:51:30.430 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:51:30.430 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T02:51:30.431 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:51:30.431 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T02:51:30.431 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:51:30.431 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T02:51:30.431 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:51:30.432 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T02:51:30.432 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:51:30.432 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:51:30.432 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:51:30.432 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T02:51:30.433 INFO:journalctl@ceph.mon.c.smithi115.stdout: During handling of the above exception, another exception occurred: 2022-05-14T02:51:30.433 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T02:51:30.433 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T02:51:30.433 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T02:51:30.434 INFO:journalctl@ceph.mon.c.smithi115.stdout: return f(*arg) 2022-05-14T02:51:30.434 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T02:51:30.434 INFO:journalctl@ceph.mon.c.smithi115.stdout: self._write_client_files(client_files, host) 2022-05-14T02:51:30.435 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T02:51:30.435 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T02:51:30.435 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T02:51:30.435 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T02:51:30.435 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T02:51:30.436 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.event_loop.get_result(coro) 2022-05-14T02:51:30.436 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T02:51:30.436 INFO:journalctl@ceph.mon.c.smithi115.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T02:51:30.436 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T02:51:30.436 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.__get_result() 2022-05-14T02:51:30.437 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T02:51:30.437 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise self._exception 2022-05-14T02:51:30.437 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T02:51:30.437 INFO:journalctl@ceph.mon.c.smithi115.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T02:51:30.438 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T02:51:30.438 INFO:journalctl@ceph.mon.c.smithi115.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T02:51:30.438 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T02:51:30.438 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise 2022-05-14T02:51:30.439 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T02:51:30.439 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.gen.throw(type, value, traceback) 2022-05-14T02:51:30.439 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T02:51:30.439 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OrchestratorError(msg) 2022-05-14T02:51:30.439 INFO:journalctl@ceph.mon.c.smithi115.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:51:30.440 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:51:30 smithi115 ceph-mon[78593]: pgmap v6463: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:51:30.440 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:51:30 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:51:30.440 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:51:30 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:51:30.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:51:30 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:51:30.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:51:30 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:51:30.542 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T02:51:30.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:51:30.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T02:51:30.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:51:30.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:51:30.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:51:30.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T02:51:30.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:51:30.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T02:51:30.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:51:30.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T02:51:30.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:51:30.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T02:51:30.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:51:30.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T02:51:30.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:51:30.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:51:30.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:51:30.547 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:51:30 smithi047 ceph-mon[77147]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T02:51:30.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T02:51:30.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:51:30.548 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T02:51:30.548 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:51:30.548 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:51:30.548 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:51:30.549 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T02:51:30.549 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:51:30.549 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T02:51:30.552 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:51:30.552 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T02:51:30.552 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:51:30.553 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T02:51:30.553 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:51:30.553 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T02:51:30.553 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:51:30.554 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:51:30.554 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:51:30.554 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T02:51:30.554 INFO:journalctl@ceph.mon.b.smithi047.stdout: During handling of the above exception, another exception occurred: 2022-05-14T02:51:30.554 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T02:51:30.555 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T02:51:30.555 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T02:51:30.555 INFO:journalctl@ceph.mon.b.smithi047.stdout: return f(*arg) 2022-05-14T02:51:30.555 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T02:51:30.555 INFO:journalctl@ceph.mon.b.smithi047.stdout: self._write_client_files(client_files, host) 2022-05-14T02:51:30.556 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T02:51:30.556 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T02:51:30.556 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T02:51:30.556 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T02:51:30.556 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T02:51:30.557 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.event_loop.get_result(coro) 2022-05-14T02:51:30.557 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T02:51:30.557 INFO:journalctl@ceph.mon.b.smithi047.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T02:51:30.557 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T02:51:30.557 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.__get_result() 2022-05-14T02:51:30.558 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T02:51:30.558 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise self._exception 2022-05-14T02:51:30.558 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T02:51:30.558 INFO:journalctl@ceph.mon.b.smithi047.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T02:51:30.559 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T02:51:30.559 INFO:journalctl@ceph.mon.b.smithi047.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T02:51:30.559 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T02:51:30.559 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise 2022-05-14T02:51:30.559 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T02:51:30.560 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.gen.throw(type, value, traceback) 2022-05-14T02:51:30.560 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T02:51:30.560 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OrchestratorError(msg) 2022-05-14T02:51:30.560 INFO:journalctl@ceph.mon.b.smithi047.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:51:30.560 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:51:30 smithi047 ceph-mon[77147]: pgmap v6463: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:51:30.561 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:51:30 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:51:30.561 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:51:30 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:51:30.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:51:30 smithi115 conmon[90393]: 2022-05-14T02:51:30.422+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:51:31.450 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:51:31 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:51:31.451 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:51:31 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 12220 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:51:31.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:51:31 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:51:31.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:51:31 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 12220 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:51:31.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:51:31 smithi115 conmon[90393]: 2022-05-14T02:51:31.449+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:51:32.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:51:32 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:51:32.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:51:32 smithi047 ceph-mon[77147]: pgmap v6464: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:51:32.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:51:32 smithi115 conmon[90393]: 2022-05-14T02:51:32.415+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:51:32.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:51:32 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:51:32.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:51:32 smithi115 ceph-mon[78593]: pgmap v6464: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:51:33.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:51:33 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:51:33.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:51:33 smithi115 conmon[90393]: 2022-05-14T02:51:33.378+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:51:33.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:51:33 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:51:34.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:51:34 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:51:34.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:51:34 smithi047 ceph-mon[77147]: pgmap v6465: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:51:34.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:51:34 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:51:34.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:51:34 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:51:34.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:51:34 smithi115 conmon[90393]: 2022-05-14T02:51:34.392+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:51:34.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:51:34 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:51:34.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:51:34 smithi115 ceph-mon[78593]: pgmap v6465: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:51:34.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:51:34 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:51:34.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:51:34 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:51:35.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:51:35 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:51:35.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:51:35 smithi115 conmon[90393]: 2022-05-14T02:51:35.352+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:51:35.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:51:35 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:51:36.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:51:36 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:51:36.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:51:36 smithi047 ceph-mon[77147]: pgmap v6466: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:51:36.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:51:36 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:51:36.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:51:36 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 12230 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:51:36.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:51:36 smithi115 conmon[90393]: 2022-05-14T02:51:36.396+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:51:36.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:51:36 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:51:36.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:51:36 smithi115 ceph-mon[78593]: pgmap v6466: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:51:36.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:51:36 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:51:36.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:51:36 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 12230 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:51:37.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:51:37 smithi115 conmon[90393]: 2022-05-14T02:51:37.379+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:51:38.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:51:38 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:51:38.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:51:38 smithi047 ceph-mon[77147]: pgmap v6467: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:51:38.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:51:38 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:51:38.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:51:38 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:51:38.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:51:38 smithi115 conmon[90393]: 2022-05-14T02:51:38.347+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:51:38.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:51:38 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:51:38.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:51:38 smithi115 ceph-mon[78593]: pgmap v6467: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:51:38.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:51:38 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:51:38.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:51:38 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:51:39.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:51:39 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:51:39.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:51:39 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:51:39.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:51:39 smithi115 conmon[90393]: 2022-05-14T02:51:39.393+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:51:39.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:51:39 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:51:39.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:51:39 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:51:40.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:51:40 smithi047 ceph-mon[77147]: pgmap v6468: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:51:40.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:51:40 smithi115 conmon[90393]: 2022-05-14T02:51:40.415+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:51:40.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:51:40 smithi115 ceph-mon[78593]: pgmap v6468: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:51:41.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:51:41 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:51:41.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:51:41 smithi115 conmon[90393]: 2022-05-14T02:51:41.376+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:51:41.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:51:41 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:51:42.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:51:42 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:51:42.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:51:42 smithi047 ceph-mon[77147]: pgmap v6469: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:51:42.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:51:42 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 12235 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:51:42.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:51:42 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:51:42.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:51:42 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:51:42.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:51:42 smithi115 conmon[90393]: 2022-05-14T02:51:42.352+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:51:42.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:51:42 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:51:42.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:51:42 smithi115 ceph-mon[78593]: pgmap v6469: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:51:42.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:51:42 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 12235 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:51:42.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:51:42 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:51:42.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:51:42 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:51:43.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:51:43 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:51:43.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:51:43 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:51:43.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:51:43 smithi115 conmon[90393]: 2022-05-14T02:51:43.321+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:51:43.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:51:43 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:51:43.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:51:43 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:51:44.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:51:44 smithi047 ceph-mon[77147]: pgmap v6470: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:51:44.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:51:44 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:51:44.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:51:44 smithi115 conmon[90393]: 2022-05-14T02:51:44.317+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:51:44.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:51:44 smithi115 ceph-mon[78593]: pgmap v6470: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:51:44.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:51:44 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:51:45.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:51:45 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:51:45.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:51:45 smithi115 conmon[90393]: 2022-05-14T02:51:45.330+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:51:45.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:51:45 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:51:46.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:51:46 smithi047 ceph-mon[77147]: pgmap v6471: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:51:46.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:51:46 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:51:46.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:51:46 smithi115 conmon[90393]: 2022-05-14T02:51:46.354+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:51:46.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:51:46 smithi115 ceph-mon[78593]: pgmap v6471: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:51:46.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:51:46 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:51:47.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:51:47 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:51:47.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:51:47 smithi115 conmon[90393]: 2022-05-14T02:51:47.382+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:51:47.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:51:47 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:51:48.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:51:48 smithi047 ceph-mon[77147]: pgmap v6472: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:51:48.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:51:48 smithi115 conmon[90393]: 2022-05-14T02:51:48.407+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:51:48.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:51:48 smithi115 ceph-mon[78593]: pgmap v6472: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:51:49.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:51:49 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:51:49.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:51:49 smithi115 conmon[90393]: 2022-05-14T02:51:49.432+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:51:49.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:51:49 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:51:50.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:51:50 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:51:50.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:51:50 smithi047 ceph-mon[77147]: pgmap v6473: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:51:50.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:51:50 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:51:50.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:51:50 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:51:50.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:51:50 smithi115 conmon[90393]: 2022-05-14T02:51:50.456+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:51:50.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:51:50 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:51:50.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:51:50 smithi115 ceph-mon[78593]: pgmap v6473: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:51:50.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:51:50 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:51:50.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:51:50 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:51:51.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:51:51 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:51:51.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:51:51 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 12240 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:51:51.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:51:51 smithi115 conmon[90393]: 2022-05-14T02:51:51.438+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:51:51.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:51:51 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:51:51.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:51:51 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 12240 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:51:52.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:51:52 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:51:52.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:51:52 smithi047 ceph-mon[77147]: pgmap v6474: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:51:52.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:51:52 smithi115 conmon[90393]: 2022-05-14T02:51:52.456+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:51:52.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:51:52 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:51:52.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:51:52 smithi115 ceph-mon[78593]: pgmap v6474: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:51:53.500 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:51:53 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:51:53.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:51:53 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:51:53.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:51:53 smithi115 conmon[90393]: 2022-05-14T02:51:53.498+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:51:54.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:51:54 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:51:54.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:51:54 smithi047 ceph-mon[77147]: pgmap v6475: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:51:54.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:51:54 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:51:54.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:51:54 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:51:54.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:51:54 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:51:54.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:51:54 smithi115 ceph-mon[78593]: pgmap v6475: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:51:54.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:51:54 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:51:54.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:51:54 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:51:54.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:51:54 smithi115 conmon[90393]: 2022-05-14T02:51:54.544+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:51:55.524 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:51:55 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:51:55.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:51:55 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:51:55.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:51:55 smithi115 conmon[90393]: 2022-05-14T02:51:55.523+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:51:56.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:51:56 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:51:56.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:51:56 smithi047 ceph-mon[77147]: pgmap v6476: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:51:56.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:51:56 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 12250 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:51:56.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:51:56 smithi115 conmon[90393]: 2022-05-14T02:51:56.492+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:51:56.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:51:56 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:51:56.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:51:56 smithi115 ceph-mon[78593]: pgmap v6476: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:51:56.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:51:56 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 12250 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:51:57.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:51:57 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:51:57.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:51:57 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:51:57.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:51:57 smithi115 conmon[90393]: 2022-05-14T02:51:57.478+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:51:58.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:51:58 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:51:58.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:51:58 smithi047 ceph-mon[77147]: pgmap v6477: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:51:58.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:51:58 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:51:58.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:51:58 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:51:58.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:51:58 smithi115 conmon[90393]: 2022-05-14T02:51:58.486+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:51:58.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:51:58 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:51:58.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:51:58 smithi115 ceph-mon[78593]: pgmap v6477: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:51:58.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:51:58 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:51:58.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:51:58 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:51:59.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:51:59 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:51:59.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:51:59 smithi115 conmon[90393]: 2022-05-14T02:51:59.479+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:51:59.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:51:59 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:52:00.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:52:00 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:52:00.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:52:00 smithi047 ceph-mon[77147]: pgmap v6478: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:52:00.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:52:00 smithi115 conmon[90393]: 2022-05-14T02:52:00.461+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:52:00.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:52:00 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:52:00.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:52:00 smithi115 ceph-mon[78593]: pgmap v6478: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:52:01.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:52:01 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:52:01.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:52:01 smithi115 conmon[90393]: 2022-05-14T02:52:01.505+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:52:01.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:52:01 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:52:02.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:52:02 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:52:02.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:52:02 smithi047 ceph-mon[77147]: pgmap v6479: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:52:02.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:52:02 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 12255 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:52:02.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:52:02 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:52:02.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:52:02 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:52:02.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:52:02 smithi115 conmon[90393]: 2022-05-14T02:52:02.506+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:52:02.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:52:02 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:52:02.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:52:02 smithi115 ceph-mon[78593]: pgmap v6479: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:52:02.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:52:02 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 12255 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:52:02.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:52:02 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:52:02.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:52:02 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:52:03.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:52:03 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:52:03.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:52:03 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:52:03.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:52:03 smithi115 conmon[90393]: 2022-05-14T02:52:03.472+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:52:04.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:52:04 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:52:04.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:52:04 smithi047 ceph-mon[77147]: pgmap v6480: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:52:04.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:52:04 smithi115 conmon[90393]: 2022-05-14T02:52:04.439+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:52:04.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:52:04 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:52:04.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:52:04 smithi115 ceph-mon[78593]: pgmap v6480: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:52:05.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:52:05 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:52:05.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:52:05 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:52:05.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:52:05 smithi115 conmon[90393]: 2022-05-14T02:52:05.393+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:52:06.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:52:06 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:52:06.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:52:06 smithi047 ceph-mon[77147]: pgmap v6481: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:52:06.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:52:06 smithi115 conmon[90393]: 2022-05-14T02:52:06.368+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:52:06.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:52:06 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:52:06.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:52:06 smithi115 ceph-mon[78593]: pgmap v6481: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:52:07.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:52:07 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:52:07.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:52:07 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:52:07.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:52:07 smithi115 conmon[90393]: 2022-05-14T02:52:07.361+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:52:08.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:52:08 smithi115 conmon[90393]: 2022-05-14T02:52:08.409+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:52:08.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:52:08 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:52:08.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:52:08 smithi115 ceph-mon[78593]: pgmap v6482: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:52:08.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:52:08 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:52:08.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:52:08 smithi047 ceph-mon[77147]: pgmap v6482: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:52:09.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:52:09 smithi115 conmon[90393]: 2022-05-14T02:52:09.434+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:52:09.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:52:09 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:52:09.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:52:09 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:52:10.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:52:10 smithi115 conmon[90393]: 2022-05-14T02:52:10.440+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:52:10.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:52:10 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:52:10.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:52:10 smithi115 ceph-mon[78593]: pgmap v6483: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:52:10.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:52:10 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:52:10.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:52:10 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:52:10.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:52:10 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:52:10.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:52:10 smithi047 ceph-mon[77147]: pgmap v6483: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:52:10.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:52:10 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:52:10.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:52:10 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:52:11.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:52:11 smithi115 conmon[90393]: 2022-05-14T02:52:11.478+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:52:11.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:52:11 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:52:11.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:52:11 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 12260 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:52:11.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:52:11 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:52:11.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:52:11 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 12260 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:52:12.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:52:12 smithi115 conmon[90393]: 2022-05-14T02:52:12.490+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:52:12.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:52:12 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:52:12.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:52:12 smithi115 ceph-mon[78593]: pgmap v6484: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:52:12.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:52:12 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:52:12.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:52:12 smithi047 ceph-mon[77147]: pgmap v6484: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:52:13.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:52:13 smithi115 conmon[90393]: 2022-05-14T02:52:13.497+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:52:13.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:52:13 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:52:13.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:52:13 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:52:14.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:52:14 smithi115 conmon[90393]: 2022-05-14T02:52:14.489+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:52:14.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:52:14 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:52:14.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:52:14 smithi115 ceph-mon[78593]: pgmap v6485: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:52:14.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:52:14 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:52:14.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:52:14 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:52:14.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:52:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T02:52:14.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:52:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T02:52:14.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:52:14 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:52:14.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:52:14 smithi047 ceph-mon[77147]: pgmap v6485: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:52:14.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:52:14 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:52:14.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:52:14 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:52:14.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:52:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T02:52:14.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:52:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T02:52:15.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:52:15 smithi115 conmon[90393]: 2022-05-14T02:52:15.461+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:52:15.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:52:15 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:52:15.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:52:15 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:52:16.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:52:16 smithi115 conmon[90393]: 2022-05-14T02:52:16.508+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:52:16.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:52:16 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:52:16.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:52:16 smithi115 ceph-mon[78593]: pgmap v6486: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 7 op/s 2022-05-14T02:52:16.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:52:16 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 12270 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:52:16.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:52:16 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:52:16.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:52:16 smithi047 ceph-mon[77147]: pgmap v6486: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 7 op/s 2022-05-14T02:52:16.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:52:16 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 12270 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:52:17.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:52:17 smithi115 conmon[90393]: 2022-05-14T02:52:17.494+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:52:17.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:52:17 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:52:17.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:52:17 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:52:18.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:52:18 smithi115 conmon[90393]: 2022-05-14T02:52:18.516+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:52:18.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:52:18 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:52:18.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:52:18 smithi115 ceph-mon[78593]: pgmap v6487: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 4 op/s 2022-05-14T02:52:18.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:52:18 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:52:18.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:52:18 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:52:18.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:52:18 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:52:18.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:52:18 smithi047 ceph-mon[77147]: pgmap v6487: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 4 op/s 2022-05-14T02:52:18.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:52:18 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:52:18.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:52:18 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:52:19.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:52:19 smithi115 conmon[90393]: 2022-05-14T02:52:19.477+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:52:19.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:52:19 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:52:19.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:52:19 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:52:20.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:52:20 smithi115 conmon[90393]: 2022-05-14T02:52:20.493+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:52:20.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:52:20 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:52:20.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:52:20 smithi115 ceph-mon[78593]: pgmap v6488: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 4 op/s 2022-05-14T02:52:20.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:52:20 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:52:20.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:52:20 smithi047 ceph-mon[77147]: pgmap v6488: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 4 op/s 2022-05-14T02:52:21.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:52:21 smithi115 conmon[90393]: 2022-05-14T02:52:21.541+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:52:21.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:52:21 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:52:21.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:52:21 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:52:22.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:52:22 smithi115 conmon[90393]: 2022-05-14T02:52:22.562+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:52:22.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:52:22 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:52:22.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:52:22 smithi115 ceph-mon[78593]: pgmap v6489: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 7 op/s 2022-05-14T02:52:22.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:52:22 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 12275 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:52:22.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:52:22 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:52:22.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:52:22 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:52:22.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:52:22 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:52:22.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:52:22 smithi047 ceph-mon[77147]: pgmap v6489: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 7 op/s 2022-05-14T02:52:22.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:52:22 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 12275 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:52:22.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:52:22 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:52:22.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:52:22 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:52:23.599 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:52:23 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:52:23.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:52:23 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:52:23.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:52:23 smithi115 conmon[90393]: 2022-05-14T02:52:23.597+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:52:24.600 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:52:24 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:52:24.601 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:52:24 smithi115 ceph-mon[78593]: pgmap v6490: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 4 op/s 2022-05-14T02:52:24.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:52:24 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:52:24.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:52:24 smithi047 ceph-mon[77147]: pgmap v6490: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 4 op/s 2022-05-14T02:52:24.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:52:24 smithi115 conmon[90393]: 2022-05-14T02:52:24.599+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:52:25.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:52:25 smithi115 conmon[90393]: 2022-05-14T02:52:25.582+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:52:25.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:52:25 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:52:25.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:52:25 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:52:26.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:52:26 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:52:26.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:52:26 smithi115 ceph-mon[78593]: pgmap v6491: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 7 op/s 2022-05-14T02:52:26.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:52:26 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:52:26.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:52:26 smithi047 ceph-mon[77147]: pgmap v6491: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 7 op/s 2022-05-14T02:52:27.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:52:26 smithi115 conmon[90393]: 2022-05-14T02:52:26.622+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:52:27.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:52:27 smithi115 conmon[90393]: 2022-05-14T02:52:27.602+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:52:27.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:52:27 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:52:27.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:52:27 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:52:28.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:52:28 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:52:28.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:52:28 smithi047 ceph-mon[77147]: pgmap v6492: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:52:28.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:52:28 smithi115 conmon[90393]: 2022-05-14T02:52:28.580+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:52:28.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:52:28 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:52:28.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:52:28 smithi115 ceph-mon[78593]: pgmap v6492: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:52:29.631 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:52:29 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:52:29.632 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:52:29 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T02:52:29.632 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:52:29 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T02:52:29.632 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:52:29 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T02:52:29.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:52:29 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:52:29.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:52:29 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T02:52:29.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:52:29 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T02:52:29.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:52:29 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T02:52:29.793 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:52:29 smithi047 conmon[78521]: 2022-05-14T02:52:29.360+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:52:29.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:52:29 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T02:52:29.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:52:29 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:52:29.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:52:29 smithi047 conmon[78521]: yield 2022-05-14T02:52:29.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:52:29 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:52:29.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:52:29 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:52:29.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:52:29 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:52:29.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:52:29 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T02:52:29.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:52:29 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:52:29.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:52:29 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T02:52:29.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:52:29 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:52:29.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:52:29 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T02:52:29.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:52:29 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:52:29.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:52:29 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T02:52:29.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:52:29 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:52:29.797 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:52:29 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T02:52:29.797 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:52:29 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:52:29.797 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:52:29 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:52:29.798 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:52:29 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:52:30.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:52:29 smithi115 conmon[90393]: 2022-05-14T02:52:29.630+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:52:30.672 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:52:30 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:52:30.672 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:52:30 smithi115 ceph-mon[78593]: pgmap v6493: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:52:30.672 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:52:30 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:52:30.672 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T02:52:30.673 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:52:30.673 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T02:52:30.673 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:52:30.673 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:52:30.674 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:52:30.674 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T02:52:30.674 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:52:30.674 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T02:52:30.674 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:52:30.675 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T02:52:30.675 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:52:30.675 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T02:52:30.675 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:52:30.675 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T02:52:30.676 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:52:30.676 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:52:30.676 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:52:30.676 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:52:30 smithi115 ceph-mon[78593]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T02:52:30.677 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:52:30 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:52:30.677 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:52:30 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:52:30.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:52:30 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:52:30.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:52:30 smithi047 ceph-mon[77147]: pgmap v6493: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:52:30.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:52:30 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:52:30.792 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T02:52:30.793 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:52:30.793 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T02:52:30.793 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:52:30.793 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:52:30.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:52:30.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T02:52:30.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:52:30.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T02:52:30.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:52:30.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T02:52:30.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:52:30.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T02:52:30.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:52:30.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T02:52:30.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:52:30.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:52:30.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:52:30.796 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:52:30 smithi047 ceph-mon[77147]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T02:52:30.797 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:52:30 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:52:30.797 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:52:30 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:52:31.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:52:30 smithi115 conmon[90393]: 2022-05-14T02:52:30.670+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:52:31.692 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:52:31 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:52:31.693 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:52:31 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 12280 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:52:31.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:52:31 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:52:31.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:52:31 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 12280 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:52:32.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:52:31 smithi115 conmon[90393]: 2022-05-14T02:52:31.690+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:52:32.692 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:52:32 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:52:32.693 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:52:32 smithi115 ceph-mon[78593]: pgmap v6494: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:52:32.693 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:52:32 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:52:32.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:52:32 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:52:32.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:52:32 smithi047 ceph-mon[77147]: pgmap v6494: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:52:32.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:52:32 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:52:32.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:52:32 smithi047 conmon[78521]: 2022-05-14T02:52:32.431+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:52:32.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:52:32 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T02:52:32.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:52:32 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:52:32.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:52:32 smithi047 conmon[78521]: yield 2022-05-14T02:52:32.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:52:32 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:52:32.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:52:32 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:52:32.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:52:32 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:52:32.797 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:52:32 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T02:52:32.797 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:52:32 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:52:32.797 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:52:32 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T02:52:32.798 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:52:32 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:52:32.798 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:52:32 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T02:52:32.798 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:52:32 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:52:32.798 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:52:32 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T02:52:32.799 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:52:32 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:52:32.799 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:52:32 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T02:52:32.799 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:52:32 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:52:32.799 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:52:32 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:52:32.800 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:52:32 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:52:32.800 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:52:32 smithi047 conmon[78521]: 2022-05-14T02:52:32.432+0000 7f534177b700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T02:52:32.800 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:52:32 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T02:52:32.800 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:52:32 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:52:32.801 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:52:32 smithi047 conmon[78521]: yield 2022-05-14T02:52:32.801 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:52:32 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:52:32.801 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:52:32 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:52:32.803 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:52:32 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:52:32.803 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:52:32 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T02:52:32.803 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:52:32 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:52:32.803 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:52:32 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T02:52:32.804 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:52:32 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:52:32.804 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:52:32 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T02:52:32.804 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:52:32 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:52:32.804 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:52:32 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T02:52:32.805 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:52:32 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:52:32.805 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:52:32 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T02:52:32.805 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:52:32 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:52:32.805 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:52:32 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:52:32.805 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:52:32 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:52:32.806 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:52:32 smithi047 conmon[78521]: 2022-05-14T02:52:32.806 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:52:32 smithi047 conmon[78521]: During handling of the above exception, another exception occurred: 2022-05-14T02:52:32.806 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:52:32 smithi047 conmon[78521]: 2022-05-14T02:52:32.806 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:52:32 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T02:52:32.807 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:52:32 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T02:52:32.807 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:52:32 smithi047 conmon[78521]: return f(*arg) 2022-05-14T02:52:32.807 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:52:32 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T02:52:32.807 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:52:32 smithi047 conmon[78521]: self._write_client_files(client_files, host) 2022-05-14T02:52:32.807 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:52:32 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T02:52:32.808 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:52:32 smithi047 conmon[78521]: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T02:52:32.808 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:52:32 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T02:52:32.808 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:52:32 smithi047 conmon[78521]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T02:52:32.808 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:52:32 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T02:52:32.808 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:52:32 smithi047 conmon[78521]: return self.event_loop.get_result(coro) 2022-05-14T02:52:32.809 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:52:32 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T02:52:32.809 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:52:32 smithi047 conmon[78521]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T02:52:32.809 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:52:32 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T02:52:32.809 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:52:32 smithi047 conmon[78521]: return self.__get_result() 2022-05-14T02:52:32.809 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:52:32 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T02:52:32.810 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:52:32 smithi047 conmon[78521]: raise self._exception 2022-05-14T02:52:32.810 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:52:32 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T02:52:32.810 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:52:32 smithi047 conmon[78521]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T02:52:32.810 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:52:32 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T02:52:32.811 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:52:32 smithi047 conmon[78521]: conn = await self._remote_connection(host, addr) 2022-05-14T02:52:32.811 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:52:32 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T02:52:32.811 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:52:32 smithi047 conmon[78521]: raise 2022-05-14T02:52:32.811 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:52:32 smithi047 conmon[78521]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T02:52:32.811 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:52:32 smithi047 conmon[78521]: self.gen.throw(type, value, traceback) 2022-05-14T02:52:32.812 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:52:32 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T02:52:32.812 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:52:32 smithi047 conmon[78521]: raise OrchestratorError(msg) 2022-05-14T02:52:32.812 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:52:32 smithi047 conmon[78521]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:52:33.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:52:32 smithi115 conmon[90393]: 2022-05-14T02:52:32.691+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:52:33.693 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:52:33 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:52:33.693 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T02:52:33.694 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:52:33.694 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T02:52:33.694 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:52:33.694 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:52:33.695 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:52:33.695 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T02:52:33.695 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:52:33.695 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T02:52:33.695 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:52:33.696 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T02:52:33.696 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:52:33.696 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T02:52:33.696 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:52:33.696 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T02:52:33.697 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:52:33.697 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:52:33.697 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:52:33.697 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:52:33 smithi115 ceph-mon[78593]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T02:52:33.698 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T02:52:33.698 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:52:33.698 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T02:52:33.698 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:52:33.698 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:52:33.699 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:52:33.699 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T02:52:33.699 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:52:33.699 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T02:52:33.699 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:52:33.700 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T02:52:33.700 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:52:33.700 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T02:52:33.700 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:52:33.700 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T02:52:33.701 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:52:33.701 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:52:33.701 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:52:33.701 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T02:52:33.702 INFO:journalctl@ceph.mon.c.smithi115.stdout: During handling of the above exception, another exception occurred: 2022-05-14T02:52:33.702 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T02:52:33.702 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T02:52:33.702 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T02:52:33.702 INFO:journalctl@ceph.mon.c.smithi115.stdout: return f(*arg) 2022-05-14T02:52:33.703 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T02:52:33.703 INFO:journalctl@ceph.mon.c.smithi115.stdout: self._write_client_files(client_files, host) 2022-05-14T02:52:33.703 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T02:52:33.703 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T02:52:33.703 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T02:52:33.704 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T02:52:33.704 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T02:52:33.704 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.event_loop.get_result(coro) 2022-05-14T02:52:33.704 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T02:52:33.704 INFO:journalctl@ceph.mon.c.smithi115.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T02:52:33.705 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T02:52:33.705 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.__get_result() 2022-05-14T02:52:33.705 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T02:52:33.705 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise self._exception 2022-05-14T02:52:33.705 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T02:52:33.706 INFO:journalctl@ceph.mon.c.smithi115.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T02:52:33.706 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T02:52:33.706 INFO:journalctl@ceph.mon.c.smithi115.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T02:52:33.706 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T02:52:33.707 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise 2022-05-14T02:52:33.707 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T02:52:33.707 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.gen.throw(type, value, traceback) 2022-05-14T02:52:33.707 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T02:52:33.707 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OrchestratorError(msg) 2022-05-14T02:52:33.708 INFO:journalctl@ceph.mon.c.smithi115.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:52:33.708 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:52:33 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:52:33.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:52:33 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:52:33.792 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T02:52:33.792 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:52:33.792 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T02:52:33.792 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:52:33.793 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:52:33.793 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:52:33.793 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T02:52:33.793 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:52:33.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T02:52:33.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:52:33.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T02:52:33.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:52:33.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T02:52:33.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:52:33.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T02:52:33.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:52:33.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:52:33.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:52:33.796 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:52:33 smithi047 ceph-mon[77147]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T02:52:33.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T02:52:33.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:52:33.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T02:52:33.797 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:52:33.797 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:52:33.797 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:52:33.797 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T02:52:33.798 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:52:33.798 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T02:52:33.798 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:52:33.798 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T02:52:33.798 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:52:33.799 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T02:52:33.799 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:52:33.799 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T02:52:33.799 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:52:33.799 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:52:33.800 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:52:33.800 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T02:52:33.800 INFO:journalctl@ceph.mon.b.smithi047.stdout: During handling of the above exception, another exception occurred: 2022-05-14T02:52:33.800 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T02:52:33.800 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T02:52:33.801 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T02:52:33.801 INFO:journalctl@ceph.mon.b.smithi047.stdout: return f(*arg) 2022-05-14T02:52:33.801 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T02:52:33.801 INFO:journalctl@ceph.mon.b.smithi047.stdout: self._write_client_files(client_files, host) 2022-05-14T02:52:33.801 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T02:52:33.802 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T02:52:33.802 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T02:52:33.802 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T02:52:33.802 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T02:52:33.803 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.event_loop.get_result(coro) 2022-05-14T02:52:33.803 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T02:52:33.803 INFO:journalctl@ceph.mon.b.smithi047.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T02:52:33.803 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T02:52:33.803 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.__get_result() 2022-05-14T02:52:33.804 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T02:52:33.804 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise self._exception 2022-05-14T02:52:33.804 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T02:52:33.804 INFO:journalctl@ceph.mon.b.smithi047.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T02:52:33.804 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T02:52:33.805 INFO:journalctl@ceph.mon.b.smithi047.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T02:52:33.805 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T02:52:33.805 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise 2022-05-14T02:52:33.805 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T02:52:33.805 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.gen.throw(type, value, traceback) 2022-05-14T02:52:33.806 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T02:52:33.806 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OrchestratorError(msg) 2022-05-14T02:52:33.806 INFO:journalctl@ceph.mon.b.smithi047.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:52:33.806 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:52:33 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:52:34.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:52:33 smithi115 conmon[90393]: 2022-05-14T02:52:33.691+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:52:34.716 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:52:34 smithi115 ceph-mon[78593]: pgmap v6495: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:52:34.716 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:52:34 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:52:34.716 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:52:34 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:52:34.717 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:52:34 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:52:34.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:52:34 smithi047 ceph-mon[77147]: pgmap v6495: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:52:34.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:52:34 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:52:34.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:52:34 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:52:34.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:52:34 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:52:35.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:52:34 smithi115 conmon[90393]: 2022-05-14T02:52:34.714+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:52:35.757 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:52:35 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:52:35.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:52:35 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:52:36.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:52:35 smithi115 conmon[90393]: 2022-05-14T02:52:35.755+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:52:36.767 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:52:36 smithi115 ceph-mon[78593]: pgmap v6496: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:52:36.768 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:52:36 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:52:36.768 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:52:36 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 12290 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:52:36.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:52:36 smithi047 ceph-mon[77147]: pgmap v6496: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:52:36.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:52:36 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:52:36.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:52:36 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 12290 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:52:37.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:52:36 smithi115 conmon[90393]: 2022-05-14T02:52:36.765+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:52:37.759 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:52:37 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:52:37.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:52:37 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:52:38.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:52:37 smithi115 conmon[90393]: 2022-05-14T02:52:37.757+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:52:38.722 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:52:38 smithi115 ceph-mon[78593]: pgmap v6497: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:52:38.722 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:52:38 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:52:38.722 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:52:38 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:52:38.723 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:52:38 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:52:38.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:52:38 smithi047 ceph-mon[77147]: pgmap v6497: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:52:38.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:52:38 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:52:38.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:52:38 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:52:38.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:52:38 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:52:39.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:52:38 smithi115 conmon[90393]: 2022-05-14T02:52:38.720+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:52:39.729 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:52:39 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:52:39.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:52:39 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:52:40.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:52:39 smithi115 conmon[90393]: 2022-05-14T02:52:39.727+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:52:40.699 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:52:40 smithi115 ceph-mon[78593]: pgmap v6498: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:52:40.724 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:52:40 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:52:40.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:52:40 smithi047 ceph-mon[77147]: pgmap v6498: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:52:40.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:52:40 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:52:41.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:52:40 smithi115 conmon[90393]: 2022-05-14T02:52:40.698+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:52:42.032 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:52:41 smithi115 conmon[90393]: 2022-05-14T02:52:41.729+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:52:42.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:52:42 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:52:42.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:52:42 smithi047 ceph-mon[77147]: pgmap v6499: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:52:42.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:52:42 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 12295 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:52:42.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:52:42 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:52:42.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:52:42 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:52:42.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:52:42 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:52:42.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:52:42 smithi115 ceph-mon[78593]: pgmap v6499: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:52:42.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:52:42 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 12295 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:52:42.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:52:42 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:52:42.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:52:42 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:52:43.036 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:52:42 smithi115 conmon[90393]: 2022-05-14T02:52:42.696+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:52:43.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:52:43 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:52:43.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:52:43 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:52:44.038 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:52:43 smithi115 conmon[90393]: 2022-05-14T02:52:43.655+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:52:44.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:52:44 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:52:44.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:52:44 smithi047 ceph-mon[77147]: pgmap v6500: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:52:44.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:52:44 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:52:44.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:52:44 smithi115 ceph-mon[78593]: pgmap v6500: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:52:45.043 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:52:44 smithi115 conmon[90393]: 2022-05-14T02:52:44.612+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:52:45.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:52:45 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:52:45.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:52:45 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:52:46.045 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:52:45 smithi115 conmon[90393]: 2022-05-14T02:52:45.625+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:52:46.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:52:46 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:52:46.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:52:46 smithi115 ceph-mon[78593]: pgmap v6501: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:52:46.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:52:46 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:52:46.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:52:46 smithi047 ceph-mon[77147]: pgmap v6501: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:52:47.049 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:52:46 smithi115 conmon[90393]: 2022-05-14T02:52:46.612+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:52:47.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:52:47 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:52:47.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:52:47 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:52:48.051 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:52:47 smithi115 conmon[90393]: 2022-05-14T02:52:47.611+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:52:48.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:52:48 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:52:48.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:52:48 smithi115 ceph-mon[78593]: pgmap v6502: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:52:48.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:52:48 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:52:48.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:52:48 smithi047 ceph-mon[77147]: pgmap v6502: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:52:49.057 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:52:48 smithi115 conmon[90393]: 2022-05-14T02:52:48.645+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:52:49.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:52:49 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:52:49.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:52:49 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:52:50.058 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:52:49 smithi115 conmon[90393]: 2022-05-14T02:52:49.642+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:52:50.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:52:50 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:52:50.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:52:50 smithi115 ceph-mon[78593]: pgmap v6503: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:52:50.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:52:50 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:52:50.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:52:50 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:52:50.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:52:50 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:52:50.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:52:50 smithi047 ceph-mon[77147]: pgmap v6503: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:52:50.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:52:50 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:52:50.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:52:50 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:52:50.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:52:50 smithi115 conmon[90393]: 2022-05-14T02:52:50.604+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:52:51.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:52:51 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:52:51.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:52:51 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 12300 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:52:51.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:52:51 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:52:51.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:52:51 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 12300 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:52:51.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:52:51 smithi115 conmon[90393]: 2022-05-14T02:52:51.588+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:52:52.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:52:52 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:52:52.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:52:52 smithi115 ceph-mon[78593]: pgmap v6504: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:52:52.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:52:52 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:52:52.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:52:52 smithi047 ceph-mon[77147]: pgmap v6504: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:52:52.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:52:52 smithi115 conmon[90393]: 2022-05-14T02:52:52.556+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:52:53.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:52:53 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:52:53.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:52:53 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:52:53.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:52:53 smithi115 conmon[90393]: 2022-05-14T02:52:53.606+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:52:54.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:52:54 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:52:54.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:52:54 smithi115 ceph-mon[78593]: pgmap v6505: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:52:54.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:52:54 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:52:54.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:52:54 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:52:54.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:52:54 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:52:54.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:52:54 smithi047 ceph-mon[77147]: pgmap v6505: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:52:54.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:52:54 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:52:54.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:52:54 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:52:55.074 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:52:54 smithi115 conmon[90393]: 2022-05-14T02:52:54.621+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:52:55.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:52:55 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:52:55.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:52:55 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:52:56.075 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:52:55 smithi115 conmon[90393]: 2022-05-14T02:52:55.671+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:52:56.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:52:56 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:52:56.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:52:56 smithi115 ceph-mon[78593]: pgmap v6506: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:52:56.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:52:56 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 12310 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:52:56.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:52:56 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:52:56.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:52:56 smithi047 ceph-mon[77147]: pgmap v6506: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:52:56.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:52:56 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 12310 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:52:57.082 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:52:56 smithi115 conmon[90393]: 2022-05-14T02:52:56.662+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:52:57.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:52:57 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:52:57.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:52:57 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:52:58.082 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:52:57 smithi115 conmon[90393]: 2022-05-14T02:52:57.699+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:52:58.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:52:58 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:52:58.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:52:58 smithi115 ceph-mon[78593]: pgmap v6507: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:52:58.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:52:58 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:52:58.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:52:58 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:52:58.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:52:58 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:52:58.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:52:58 smithi047 ceph-mon[77147]: pgmap v6507: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:52:58.545 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:52:58 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:52:58.545 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:52:58 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:52:59.086 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:52:58 smithi115 conmon[90393]: 2022-05-14T02:52:58.650+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:52:59.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:52:59 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:52:59.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:52:59 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:53:00.089 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:52:59 smithi115 conmon[90393]: 2022-05-14T02:52:59.685+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:53:00.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:53:00 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:53:00.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:53:00 smithi115 ceph-mon[78593]: pgmap v6508: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:53:00.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:53:00 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:53:00.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:53:00 smithi047 ceph-mon[77147]: pgmap v6508: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:53:01.095 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:53:00 smithi115 conmon[90393]: 2022-05-14T02:53:00.706+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:53:01.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:53:01 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:53:01.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:53:01 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:53:02.097 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:53:01 smithi115 conmon[90393]: 2022-05-14T02:53:01.732+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:53:02.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:53:02 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:53:02.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:53:02 smithi115 ceph-mon[78593]: pgmap v6509: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:53:02.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:53:02 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 12315 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:53:02.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:53:02 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:53:02.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:53:02 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:53:02.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:53:02 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:53:02.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:53:02 smithi047 ceph-mon[77147]: pgmap v6509: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:53:02.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:53:02 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 12315 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:53:02.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:53:02 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:53:02.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:53:02 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:53:03.102 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:53:02 smithi115 conmon[90393]: 2022-05-14T02:53:02.765+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:53:03.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:53:03 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:53:03.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:53:03 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:53:04.103 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:53:03 smithi115 conmon[90393]: 2022-05-14T02:53:03.720+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:53:04.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:53:04 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:53:04.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:53:04 smithi115 ceph-mon[78593]: pgmap v6510: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:53:04.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:53:04 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:53:04.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:53:04 smithi047 ceph-mon[77147]: pgmap v6510: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:53:05.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:53:04 smithi115 conmon[90393]: 2022-05-14T02:53:04.697+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:53:05.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:53:05 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:53:05.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:53:05 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:53:06.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:53:05 smithi115 conmon[90393]: 2022-05-14T02:53:05.688+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:53:06.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:53:06 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:53:06.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:53:06 smithi047 ceph-mon[77147]: pgmap v6511: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:53:06.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:53:06 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:53:06.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:53:06 smithi115 ceph-mon[78593]: pgmap v6511: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:53:07.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:53:06 smithi115 conmon[90393]: 2022-05-14T02:53:06.732+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:53:07.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:53:07 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:53:07.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:53:07 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:53:08.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:53:07 smithi115 conmon[90393]: 2022-05-14T02:53:07.756+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:53:08.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:53:08 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:53:08.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:53:08 smithi047 ceph-mon[77147]: pgmap v6512: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:53:08.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:53:08 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:53:08.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:53:08 smithi115 ceph-mon[78593]: pgmap v6512: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:53:09.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:53:08 smithi115 conmon[90393]: 2022-05-14T02:53:08.762+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:53:09.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:53:09 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:53:09.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:53:09 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:53:10.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:53:09 smithi115 conmon[90393]: 2022-05-14T02:53:09.722+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:53:10.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:53:10 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:53:10.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:53:10 smithi047 ceph-mon[77147]: pgmap v6513: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:53:10.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:53:10 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:53:10.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:53:10 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:53:10.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:53:10 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:53:10.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:53:10 smithi115 ceph-mon[78593]: pgmap v6513: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:53:10.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:53:10 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:53:10.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:53:10 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:53:11.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:53:10 smithi115 conmon[90393]: 2022-05-14T02:53:10.755+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:53:11.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:53:11 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:53:11.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:53:11 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 12320 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:53:11.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:53:11 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:53:11.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:53:11 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 12320 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:53:12.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:53:11 smithi115 conmon[90393]: 2022-05-14T02:53:11.760+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:53:12.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:53:12 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:53:12.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:53:12 smithi047 ceph-mon[77147]: pgmap v6514: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:53:12.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:53:12 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:53:12.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:53:12 smithi115 ceph-mon[78593]: pgmap v6514: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:53:13.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:53:12 smithi115 conmon[90393]: 2022-05-14T02:53:12.772+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:53:13.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:53:13 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:53:13.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:53:13 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:53:14.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:53:13 smithi115 conmon[90393]: 2022-05-14T02:53:13.767+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:53:14.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:53:14 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:53:14.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:53:14 smithi047 ceph-mon[77147]: pgmap v6515: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:53:14.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:53:14 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:53:14.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:53:14 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:53:14.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:53:14 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:53:14.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:53:14 smithi115 ceph-mon[78593]: pgmap v6515: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:53:14.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:53:14 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:53:14.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:53:14 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:53:15.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:53:14 smithi115 conmon[90393]: 2022-05-14T02:53:14.810+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:53:15.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:53:15 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:53:15.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:53:15 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T02:53:15.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:53:15 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T02:53:15.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:53:15 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:53:15.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:53:15 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T02:53:15.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:53:15 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T02:53:16.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:53:15 smithi115 conmon[90393]: 2022-05-14T02:53:15.840+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:53:16.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:53:16 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:53:16.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:53:16 smithi047 ceph-mon[77147]: pgmap v6516: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:53:16.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:53:16 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 12330 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:53:16.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:53:16 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:53:16.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:53:16 smithi115 ceph-mon[78593]: pgmap v6516: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:53:16.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:53:16 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 12330 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:53:17.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:53:16 smithi115 conmon[90393]: 2022-05-14T02:53:16.838+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:53:17.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:53:17 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:53:17.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:53:17 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:53:18.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:53:17 smithi115 conmon[90393]: 2022-05-14T02:53:17.833+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:53:18.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:53:18 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:53:18.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:53:18 smithi047 ceph-mon[77147]: pgmap v6517: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:53:18.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:53:18 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:53:18.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:53:18 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:53:18.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:53:18 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:53:18.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:53:18 smithi115 ceph-mon[78593]: pgmap v6517: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:53:18.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:53:18 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:53:18.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:53:18 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:53:19.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:53:18 smithi115 conmon[90393]: 2022-05-14T02:53:18.826+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:53:19.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:53:19 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:53:19.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:53:19 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:53:20.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:53:19 smithi115 conmon[90393]: 2022-05-14T02:53:19.821+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:53:20.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:53:20 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:53:20.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:53:20 smithi047 ceph-mon[77147]: pgmap v6518: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:53:20.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:53:20 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:53:20.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:53:20 smithi115 ceph-mon[78593]: pgmap v6518: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:53:21.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:53:20 smithi115 conmon[90393]: 2022-05-14T02:53:20.797+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:53:21.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:53:21 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:53:21.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:53:21 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:53:22.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:53:21 smithi115 conmon[90393]: 2022-05-14T02:53:21.841+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:53:22.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:53:22 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:53:22.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:53:22 smithi047 ceph-mon[77147]: pgmap v6519: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:53:22.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:53:22 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 12335 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:53:22.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:53:22 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:53:22.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:53:22 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:53:22.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:53:22 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:53:22.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:53:22 smithi115 ceph-mon[78593]: pgmap v6519: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:53:22.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:53:22 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 12335 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:53:22.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:53:22 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:53:22.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:53:22 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:53:23.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:53:22 smithi115 conmon[90393]: 2022-05-14T02:53:22.846+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:53:23.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:53:23 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:53:23.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:53:23 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:53:24.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:53:23 smithi115 conmon[90393]: 2022-05-14T02:53:23.854+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:53:24.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:53:24 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:53:24.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:53:24 smithi047 ceph-mon[77147]: pgmap v6520: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:53:24.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:53:24 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:53:24.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:53:24 smithi115 ceph-mon[78593]: pgmap v6520: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:53:25.172 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:53:24 smithi115 conmon[90393]: 2022-05-14T02:53:24.876+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:53:25.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:53:25 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:53:25.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:53:25 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:53:26.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:53:25 smithi115 conmon[90393]: 2022-05-14T02:53:25.845+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:53:26.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:53:26 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:53:26.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:53:26 smithi047 ceph-mon[77147]: pgmap v6521: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:53:26.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:53:26 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:53:26.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:53:26 smithi115 ceph-mon[78593]: pgmap v6521: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:53:27.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:53:26 smithi115 conmon[90393]: 2022-05-14T02:53:26.844+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:53:27.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:53:27 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:53:27.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:53:27 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:53:28.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:53:27 smithi115 conmon[90393]: 2022-05-14T02:53:27.815+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:53:28.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:53:28 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:53:28.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:53:28 smithi047 ceph-mon[77147]: pgmap v6522: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:53:28.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:53:28 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:53:28.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:53:28 smithi115 ceph-mon[78593]: pgmap v6522: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:53:29.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:53:28 smithi115 conmon[90393]: 2022-05-14T02:53:28.845+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:53:29.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:53:29 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:53:29.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:53:29 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:53:30.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:53:29 smithi115 conmon[90393]: 2022-05-14T02:53:29.850+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:53:30.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:53:30 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:53:30.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:53:30 smithi047 ceph-mon[77147]: pgmap v6523: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:53:30.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:53:30 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:53:30.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:53:30 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:53:30.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:53:30 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:53:30.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:53:30 smithi115 ceph-mon[78593]: pgmap v6523: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:53:30.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:53:30 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:53:30.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:53:30 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:53:31.191 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:53:30 smithi115 conmon[90393]: 2022-05-14T02:53:30.857+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:53:31.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:53:31 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:53:31.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:53:31 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 12340 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:53:31.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:53:31 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:53:31.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:53:31 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 12340 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:53:32.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:53:31 smithi115 conmon[90393]: 2022-05-14T02:53:31.823+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:53:32.478 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:53:32 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:53:32.479 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:53:32 smithi047 ceph-mon[77147]: pgmap v6524: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:53:32.479 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:53:32 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:53:32.480 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:53:32 smithi115 ceph-mon[78593]: pgmap v6524: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:53:33.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:53:32 smithi115 conmon[90393]: 2022-05-14T02:53:32.811+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:53:33.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:53:33 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:53:33.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:53:33 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T02:53:33.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:53:33 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T02:53:33.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:53:33 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T02:53:33.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:53:33 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:53:33.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:53:33 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T02:53:33.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:53:33 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T02:53:33.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:53:33 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T02:53:34.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:53:33 smithi115 conmon[90393]: 2022-05-14T02:53:33.850+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:53:34.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:53:34 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:53:34.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:53:34 smithi047 ceph-mon[77147]: pgmap v6525: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:53:34.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:53:34 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:53:34.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:53:34 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:53:34.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:53:34 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:53:34.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:53:34 smithi115 ceph-mon[78593]: pgmap v6525: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:53:34.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:53:34 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:53:34.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:53:34 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:53:35.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:53:34 smithi115 conmon[90393]: 2022-05-14T02:53:34.856+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:53:35.538 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:53:35 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:53:35.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:53:35 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:53:35.792 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:53:35 smithi047 conmon[78521]: 2022-05-14T02:53:35.535+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:53:35.792 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:53:35 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T02:53:35.792 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:53:35 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:53:35.793 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:53:35 smithi047 conmon[78521]: yield 2022-05-14T02:53:35.793 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:53:35 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:53:35.793 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:53:35 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:53:35.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:53:35 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:53:35.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:53:35 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T02:53:35.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:53:35 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:53:35.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:53:35 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T02:53:35.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:53:35 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:53:35.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:53:35 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T02:53:35.797 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:53:35 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:53:35.797 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:53:35 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T02:53:35.798 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:53:35 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:53:35.798 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:53:35 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T02:53:35.799 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:53:35 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:53:35.799 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:53:35 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:53:35.799 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:53:35 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:53:36.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:53:35 smithi115 conmon[90393]: 2022-05-14T02:53:35.815+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:53:36.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:53:36 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:53:36.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:53:36 smithi047 ceph-mon[77147]: pgmap v6526: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:53:36.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:53:36 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:53:36.542 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T02:53:36.542 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:53:36.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T02:53:36.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:53:36.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:53:36.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:53:36.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T02:53:36.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:53:36.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T02:53:36.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:53:36.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T02:53:36.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:53:36.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T02:53:36.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:53:36.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T02:53:36.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:53:36.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:53:36.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:53:36.546 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:53:36 smithi047 ceph-mon[77147]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T02:53:36.546 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:53:36 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 12350 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:53:36.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:53:36 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:53:36.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:53:36 smithi115 ceph-mon[78593]: pgmap v6526: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:53:36.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:53:36 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:53:36.609 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T02:53:36.609 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:53:36.609 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T02:53:36.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:53:36.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:53:36.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:53:36.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T02:53:36.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:53:36.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T02:53:36.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:53:36.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T02:53:36.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:53:36.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T02:53:36.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:53:36.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T02:53:36.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:53:36.613 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:53:36.613 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:53:36.613 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:53:36 smithi115 ceph-mon[78593]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T02:53:36.613 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:53:36 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 12350 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:53:37.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:53:36 smithi115 conmon[90393]: 2022-05-14T02:53:36.804+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:53:37.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:53:37 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:53:37.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:53:37 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:53:38.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:53:37 smithi115 conmon[90393]: 2022-05-14T02:53:37.816+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:53:38.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:53:38 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:53:38.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:53:38 smithi047 ceph-mon[77147]: pgmap v6527: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:53:38.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:53:38 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:53:38.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:53:38 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:53:38.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:53:38 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:53:38.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:53:38 smithi115 ceph-mon[78593]: pgmap v6527: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:53:38.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:53:38 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:53:38.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:53:38 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:53:39.042 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:53:38 smithi047 conmon[78521]: 2022-05-14T02:53:38.607+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:53:39.042 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:53:38 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T02:53:39.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:53:38 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:53:39.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:53:38 smithi047 conmon[78521]: yield 2022-05-14T02:53:39.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:53:38 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:53:39.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:53:38 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:53:39.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:53:38 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:53:39.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:53:38 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T02:53:39.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:53:38 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:53:39.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:53:38 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T02:53:39.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:53:38 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:53:39.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:53:38 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T02:53:39.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:53:38 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:53:39.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:53:38 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T02:53:39.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:53:38 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:53:39.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:53:38 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T02:53:39.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:53:38 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:53:39.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:53:38 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:53:39.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:53:38 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:53:39.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:53:38 smithi047 conmon[78521]: 2022-05-14T02:53:38.608+0000 7f5345f84700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T02:53:39.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:53:38 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T02:53:39.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:53:38 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:53:39.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:53:38 smithi047 conmon[78521]: yield 2022-05-14T02:53:39.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:53:38 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:53:39.048 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:53:38 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:53:39.048 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:53:38 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:53:39.048 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:53:38 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T02:53:39.048 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:53:38 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:53:39.048 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:53:38 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T02:53:39.049 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:53:38 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:53:39.049 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:53:38 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T02:53:39.049 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:53:38 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:53:39.049 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:53:38 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T02:53:39.049 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:53:38 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:53:39.050 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:53:38 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T02:53:39.050 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:53:38 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:53:39.050 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:53:38 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:53:39.050 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:53:38 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:53:39.051 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:53:38 smithi047 conmon[78521]: 2022-05-14T02:53:39.051 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:53:38 smithi047 conmon[78521]: During handling of the above exception, another exception occurred: 2022-05-14T02:53:39.051 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:53:38 smithi047 conmon[78521]: 2022-05-14T02:53:39.052 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:53:38 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T02:53:39.052 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:53:38 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T02:53:39.052 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:53:38 smithi047 conmon[78521]: return f(*arg) 2022-05-14T02:53:39.052 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:53:38 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T02:53:39.052 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:53:38 smithi047 conmon[78521]: self._write_client_files(client_files, host) 2022-05-14T02:53:39.053 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:53:38 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T02:53:39.053 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:53:38 smithi047 conmon[78521]: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T02:53:39.053 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:53:38 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T02:53:39.053 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:53:38 smithi047 conmon[78521]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T02:53:39.054 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:53:38 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T02:53:39.054 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:53:38 smithi047 conmon[78521]: return self.event_loop.get_result(coro) 2022-05-14T02:53:39.054 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:53:38 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T02:53:39.054 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:53:38 smithi047 conmon[78521]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T02:53:39.054 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:53:38 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T02:53:39.055 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:53:38 smithi047 conmon[78521]: return self.__get_result() 2022-05-14T02:53:39.055 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:53:38 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T02:53:39.055 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:53:38 smithi047 conmon[78521]: raise self._exception 2022-05-14T02:53:39.055 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:53:38 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T02:53:39.055 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:53:38 smithi047 conmon[78521]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T02:53:39.056 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:53:38 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T02:53:39.056 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:53:38 smithi047 conmon[78521]: conn = await self._remote_connection(host, addr) 2022-05-14T02:53:39.056 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:53:38 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T02:53:39.056 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:53:38 smithi047 conmon[78521]: raise 2022-05-14T02:53:39.056 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:53:38 smithi047 conmon[78521]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T02:53:39.057 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:53:38 smithi047 conmon[78521]: self.gen.throw(type, value, traceback) 2022-05-14T02:53:39.057 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:53:38 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T02:53:39.057 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:53:38 smithi047 conmon[78521]: raise OrchestratorError(msg) 2022-05-14T02:53:39.057 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:53:38 smithi047 conmon[78521]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:53:39.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:53:38 smithi115 conmon[90393]: 2022-05-14T02:53:38.816+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:53:39.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:53:39 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:53:39.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:53:39 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:53:40.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:53:39 smithi115 conmon[90393]: 2022-05-14T02:53:39.789+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:53:40.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:53:40 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:53:40.542 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T02:53:40.542 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:53:40.542 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T02:53:40.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:53:40.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:53:40.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:53:40.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T02:53:40.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:53:40.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T02:53:40.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:53:40.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T02:53:40.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:53:40.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T02:53:40.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:53:40.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T02:53:40.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:53:40.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:53:40.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:53:40.546 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:53:40 smithi047 ceph-mon[77147]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T02:53:40.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T02:53:40.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:53:40.552 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T02:53:40.552 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:53:40.552 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:53:40.552 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:53:40.553 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T02:53:40.553 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:53:40.553 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T02:53:40.553 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:53:40.554 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T02:53:40.554 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:53:40.554 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T02:53:40.554 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:53:40.555 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T02:53:40.555 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:53:40.555 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:53:40.555 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:53:40.556 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T02:53:40.556 INFO:journalctl@ceph.mon.b.smithi047.stdout: During handling of the above exception, another exception occurred: 2022-05-14T02:53:40.556 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T02:53:40.556 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T02:53:40.557 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T02:53:40.558 INFO:journalctl@ceph.mon.b.smithi047.stdout: return f(*arg) 2022-05-14T02:53:40.558 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T02:53:40.558 INFO:journalctl@ceph.mon.b.smithi047.stdout: self._write_client_files(client_files, host) 2022-05-14T02:53:40.558 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T02:53:40.558 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T02:53:40.559 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T02:53:40.559 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T02:53:40.559 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T02:53:40.559 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.event_loop.get_result(coro) 2022-05-14T02:53:40.560 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T02:53:40.560 INFO:journalctl@ceph.mon.b.smithi047.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T02:53:40.560 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T02:53:40.560 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.__get_result() 2022-05-14T02:53:40.560 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T02:53:40.561 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise self._exception 2022-05-14T02:53:40.561 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T02:53:40.561 INFO:journalctl@ceph.mon.b.smithi047.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T02:53:40.561 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T02:53:40.562 INFO:journalctl@ceph.mon.b.smithi047.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T02:53:40.562 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T02:53:40.562 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise 2022-05-14T02:53:40.562 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T02:53:40.562 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.gen.throw(type, value, traceback) 2022-05-14T02:53:40.563 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T02:53:40.563 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OrchestratorError(msg) 2022-05-14T02:53:40.563 INFO:journalctl@ceph.mon.b.smithi047.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:53:40.563 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:53:40 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:53:40.564 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:53:40 smithi047 ceph-mon[77147]: pgmap v6528: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:53:40.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:53:40 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:53:40.609 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T02:53:40.609 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:53:40.609 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T02:53:40.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:53:40.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:53:40.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:53:40.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T02:53:40.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:53:40.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T02:53:40.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:53:40.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T02:53:40.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:53:40.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T02:53:40.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:53:40.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T02:53:40.613 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:53:40.613 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:53:40.613 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:53:40.613 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:53:40 smithi115 ceph-mon[78593]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T02:53:40.613 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T02:53:40.614 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:53:40.614 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T02:53:40.614 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:53:40.614 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:53:40.614 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:53:40.615 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T02:53:40.615 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:53:40.615 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T02:53:40.615 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:53:40.616 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T02:53:40.616 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:53:40.616 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T02:53:40.616 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:53:40.616 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T02:53:40.617 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:53:40.617 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:53:40.617 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:53:40.617 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T02:53:40.618 INFO:journalctl@ceph.mon.c.smithi115.stdout: During handling of the above exception, another exception occurred: 2022-05-14T02:53:40.618 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T02:53:40.619 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T02:53:40.619 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T02:53:40.620 INFO:journalctl@ceph.mon.c.smithi115.stdout: return f(*arg) 2022-05-14T02:53:40.620 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T02:53:40.620 INFO:journalctl@ceph.mon.c.smithi115.stdout: self._write_client_files(client_files, host) 2022-05-14T02:53:40.621 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T02:53:40.621 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T02:53:40.621 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T02:53:40.621 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T02:53:40.622 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T02:53:40.622 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.event_loop.get_result(coro) 2022-05-14T02:53:40.622 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T02:53:40.622 INFO:journalctl@ceph.mon.c.smithi115.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T02:53:40.623 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T02:53:40.623 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.__get_result() 2022-05-14T02:53:40.623 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T02:53:40.623 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise self._exception 2022-05-14T02:53:40.624 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T02:53:40.624 INFO:journalctl@ceph.mon.c.smithi115.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T02:53:40.624 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T02:53:40.624 INFO:journalctl@ceph.mon.c.smithi115.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T02:53:40.625 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T02:53:40.625 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise 2022-05-14T02:53:40.625 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T02:53:40.625 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.gen.throw(type, value, traceback) 2022-05-14T02:53:40.625 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T02:53:40.626 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OrchestratorError(msg) 2022-05-14T02:53:40.626 INFO:journalctl@ceph.mon.c.smithi115.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:53:40.626 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:53:40 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:53:40.626 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:53:40 smithi115 ceph-mon[78593]: pgmap v6528: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:53:41.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:53:40 smithi115 conmon[90393]: 2022-05-14T02:53:40.813+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:53:41.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:53:41 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:53:41.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:53:41 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:53:42.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:53:41 smithi115 conmon[90393]: 2022-05-14T02:53:41.854+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:53:42.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:53:42 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:53:42.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:53:42 smithi047 ceph-mon[77147]: pgmap v6529: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:53:42.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:53:42 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 12355 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:53:42.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:53:42 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:53:42.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:53:42 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:53:42.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:53:42 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:53:42.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:53:42 smithi115 ceph-mon[78593]: pgmap v6529: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:53:42.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:53:42 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 12355 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:53:42.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:53:42 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:53:42.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:53:42 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:53:43.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:53:42 smithi115 conmon[90393]: 2022-05-14T02:53:42.855+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:53:43.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:53:43 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:53:43.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:53:43 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:53:44.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:53:43 smithi115 conmon[90393]: 2022-05-14T02:53:43.832+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:53:44.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:53:44 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:53:44.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:53:44 smithi047 ceph-mon[77147]: pgmap v6530: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:53:44.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:53:44 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:53:44.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:53:44 smithi115 ceph-mon[78593]: pgmap v6530: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:53:45.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:53:44 smithi115 conmon[90393]: 2022-05-14T02:53:44.793+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:53:45.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:53:45 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:53:45.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:53:45 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:53:46.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:53:45 smithi115 conmon[90393]: 2022-05-14T02:53:45.806+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:53:46.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:53:46 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:53:46.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:53:46 smithi047 ceph-mon[77147]: pgmap v6531: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:53:46.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:53:46 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:53:46.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:53:46 smithi115 ceph-mon[78593]: pgmap v6531: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:53:47.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:53:46 smithi115 conmon[90393]: 2022-05-14T02:53:46.849+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:53:47.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:53:47 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:53:47.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:53:47 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:53:48.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:53:47 smithi115 conmon[90393]: 2022-05-14T02:53:47.848+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:53:48.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:53:48 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:53:48.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:53:48 smithi047 ceph-mon[77147]: pgmap v6532: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:53:48.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:53:48 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:53:48.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:53:48 smithi115 ceph-mon[78593]: pgmap v6532: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:53:49.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:53:48 smithi115 conmon[90393]: 2022-05-14T02:53:48.818+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:53:49.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:53:49 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:53:49.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:53:49 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:53:50.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:53:49 smithi115 conmon[90393]: 2022-05-14T02:53:49.776+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:53:50.504 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:53:50 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:53:50.504 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:53:50 smithi047 ceph-mon[77147]: pgmap v6533: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:53:50.505 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:53:50 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:53:50.505 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:53:50 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:53:50.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:53:50 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:53:50.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:53:50 smithi115 ceph-mon[78593]: pgmap v6533: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:53:50.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:53:50 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:53:50.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:53:50 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:53:51.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:53:50 smithi115 conmon[90393]: 2022-05-14T02:53:50.794+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:53:51.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:53:51 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:53:51.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:53:51 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 12360 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:53:51.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:53:51 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:53:51.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:53:51 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 12360 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:53:52.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:53:51 smithi115 conmon[90393]: 2022-05-14T02:53:51.754+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:53:52.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:53:52 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:53:52.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:53:52 smithi047 ceph-mon[77147]: pgmap v6534: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:53:52.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:53:52 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:53:52.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:53:52 smithi115 ceph-mon[78593]: pgmap v6534: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:53:53.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:53:52 smithi115 conmon[90393]: 2022-05-14T02:53:52.731+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:53:53.412 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:53:53 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:53:53.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:53:53 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:53:54.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:53:53 smithi115 conmon[90393]: 2022-05-14T02:53:53.759+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:53:54.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:53:54 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:53:54.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:53:54 smithi047 ceph-mon[77147]: pgmap v6535: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:53:54.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:53:54 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:53:54.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:53:54 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:53:54.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:53:54 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:53:54.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:53:54 smithi115 ceph-mon[78593]: pgmap v6535: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:53:54.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:53:54 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:53:54.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:53:54 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:53:55.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:53:54 smithi115 conmon[90393]: 2022-05-14T02:53:54.728+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:53:55.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:53:55 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:53:55.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:53:55 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:53:56.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:53:55 smithi115 conmon[90393]: 2022-05-14T02:53:55.727+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:53:56.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:53:56 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:53:56.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:53:56 smithi047 ceph-mon[77147]: pgmap v6536: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:53:56.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:53:56 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 12370 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:53:56.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:53:56 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:53:56.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:53:56 smithi115 ceph-mon[78593]: pgmap v6536: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:53:56.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:53:56 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 12370 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:53:57.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:53:56 smithi115 conmon[90393]: 2022-05-14T02:53:56.712+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:53:57.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:53:57 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:53:57.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:53:57 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:53:58.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:53:57 smithi115 conmon[90393]: 2022-05-14T02:53:57.674+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:53:58.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:53:58 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:53:58.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:53:58 smithi047 ceph-mon[77147]: pgmap v6537: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:53:58.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:53:58 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:53:58.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:53:58 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:53:58.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:53:58 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:53:58.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:53:58 smithi115 ceph-mon[78593]: pgmap v6537: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:53:58.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:53:58 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:53:58.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:53:58 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:53:59.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:53:58 smithi115 conmon[90393]: 2022-05-14T02:53:58.719+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:53:59.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:53:59 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:53:59.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:53:59 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:54:00.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:53:59 smithi115 conmon[90393]: 2022-05-14T02:53:59.724+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:54:00.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:54:00 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:54:00.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:54:00 smithi047 ceph-mon[77147]: pgmap v6538: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:54:00.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:54:00 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:54:00.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:54:00 smithi115 ceph-mon[78593]: pgmap v6538: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:54:01.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:54:00 smithi115 conmon[90393]: 2022-05-14T02:54:00.689+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:54:01.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:54:01 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:54:01.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:54:01 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:54:02.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:54:01 smithi115 conmon[90393]: 2022-05-14T02:54:01.688+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:54:02.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:54:02 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:54:02.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:54:02 smithi047 ceph-mon[77147]: pgmap v6539: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:54:02.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:54:02 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 12375 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:54:02.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:54:02 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:54:02.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:54:02 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:54:02.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:54:02 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:54:02.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:54:02 smithi115 ceph-mon[78593]: pgmap v6539: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:54:02.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:54:02 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 12375 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:54:02.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:54:02 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:54:02.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:54:02 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:54:03.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:54:02 smithi115 conmon[90393]: 2022-05-14T02:54:02.711+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:54:03.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:54:03 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:54:03.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:54:03 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:54:04.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:54:03 smithi115 conmon[90393]: 2022-05-14T02:54:03.718+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:54:04.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:54:04 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:54:04.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:54:04 smithi115 ceph-mon[78593]: pgmap v6540: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:54:04.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:54:04 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:54:04.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:54:04 smithi047 ceph-mon[77147]: pgmap v6540: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:54:05.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:54:04 smithi115 conmon[90393]: 2022-05-14T02:54:04.676+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:54:05.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:54:05 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:54:05.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:54:05 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:54:06.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:54:05 smithi115 conmon[90393]: 2022-05-14T02:54:05.710+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:54:06.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:54:06 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:54:06.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:54:06 smithi115 ceph-mon[78593]: pgmap v6541: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:54:06.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:54:06 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:54:06.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:54:06 smithi047 ceph-mon[77147]: pgmap v6541: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:54:07.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:54:06 smithi115 conmon[90393]: 2022-05-14T02:54:06.704+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:54:07.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:54:07 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:54:07.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:54:07 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:54:08.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:54:07 smithi115 conmon[90393]: 2022-05-14T02:54:07.661+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:54:08.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:54:08 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:54:08.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:54:08 smithi115 ceph-mon[78593]: pgmap v6542: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:54:08.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:54:08 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:54:08.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:54:08 smithi047 ceph-mon[77147]: pgmap v6542: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:54:09.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:54:08 smithi115 conmon[90393]: 2022-05-14T02:54:08.710+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:54:09.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:54:09 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:54:09.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:54:09 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:54:10.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:54:09 smithi115 conmon[90393]: 2022-05-14T02:54:09.679+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:54:10.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:54:10 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:54:10.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:54:10 smithi115 ceph-mon[78593]: pgmap v6543: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:54:10.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:54:10 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:54:10.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:54:10 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:54:10.677 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:54:10 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:54:10.677 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:54:10 smithi047 ceph-mon[77147]: pgmap v6543: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:54:10.678 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:54:10 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:54:10.678 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:54:10 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:54:11.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:54:10 smithi115 conmon[90393]: 2022-05-14T02:54:10.717+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:54:11.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:54:11 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:54:11.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:54:11 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 12380 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:54:11.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:54:11 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:54:11.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:54:11 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 12380 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:54:12.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:54:11 smithi115 conmon[90393]: 2022-05-14T02:54:11.719+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:54:12.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:54:12 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:54:12.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:54:12 smithi115 ceph-mon[78593]: pgmap v6544: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:54:12.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:54:12 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:54:12.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:54:12 smithi047 ceph-mon[77147]: pgmap v6544: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:54:13.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:54:12 smithi115 conmon[90393]: 2022-05-14T02:54:12.750+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:54:13.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:54:13 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:54:13.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:54:13 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:54:14.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:54:13 smithi115 conmon[90393]: 2022-05-14T02:54:13.750+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:54:14.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:54:14 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:54:14.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:54:14 smithi115 ceph-mon[78593]: pgmap v6545: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:54:14.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:54:14 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:54:14.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:54:14 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:54:14.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:54:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T02:54:14.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:54:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T02:54:14.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:54:14 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:54:14.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:54:14 smithi047 ceph-mon[77147]: pgmap v6545: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:54:14.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:54:14 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:54:14.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:54:14 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:54:14.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:54:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T02:54:14.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:54:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T02:54:15.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:54:14 smithi115 conmon[90393]: 2022-05-14T02:54:14.779+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:54:15.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:54:15 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:54:15.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:54:15 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:54:16.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:54:15 smithi115 conmon[90393]: 2022-05-14T02:54:15.760+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:54:16.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:54:16 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:54:16.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:54:16 smithi115 ceph-mon[78593]: pgmap v6546: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:54:16.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:54:16 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 12390 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:54:16.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:54:16 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:54:16.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:54:16 smithi047 ceph-mon[77147]: pgmap v6546: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:54:16.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:54:16 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 12390 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:54:17.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:54:16 smithi115 conmon[90393]: 2022-05-14T02:54:16.723+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:54:17.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:54:17 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:54:17.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:54:17 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:54:18.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:54:17 smithi115 conmon[90393]: 2022-05-14T02:54:17.698+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:54:18.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:54:18 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:54:18.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:54:18 smithi115 ceph-mon[78593]: pgmap v6547: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:54:18.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:54:18 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:54:18.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:54:18 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:54:18.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:54:18 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:54:18.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:54:18 smithi047 ceph-mon[77147]: pgmap v6547: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:54:18.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:54:18 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:54:18.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:54:18 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:54:19.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:54:18 smithi115 conmon[90393]: 2022-05-14T02:54:18.652+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:54:19.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:54:19 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:54:19.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:54:19 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:54:20.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:54:19 smithi115 conmon[90393]: 2022-05-14T02:54:19.666+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:54:20.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:54:20 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:54:20.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:54:20 smithi115 ceph-mon[78593]: pgmap v6548: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:54:20.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:54:20 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:54:20.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:54:20 smithi047 ceph-mon[77147]: pgmap v6548: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:54:21.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:54:20 smithi115 conmon[90393]: 2022-05-14T02:54:20.676+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:54:21.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:54:21 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:54:21.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:54:21 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:54:22.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:54:21 smithi115 conmon[90393]: 2022-05-14T02:54:21.633+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:54:22.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:54:22 smithi115 conmon[90393]: 2022-05-14T02:54:22.589+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:54:22.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:54:22 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:54:22.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:54:22 smithi115 ceph-mon[78593]: pgmap v6549: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:54:22.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:54:22 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 12395 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:54:22.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:54:22 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:54:22.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:54:22 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:54:22.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:54:22 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:54:22.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:54:22 smithi047 ceph-mon[77147]: pgmap v6549: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:54:22.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:54:22 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 12395 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:54:22.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:54:22 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:54:22.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:54:22 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:54:23.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:54:23 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:54:23.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:54:23 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:54:24.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:54:23 smithi115 conmon[90393]: 2022-05-14T02:54:23.615+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:54:24.615 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:54:24 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:54:24.615 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:54:24 smithi115 ceph-mon[78593]: pgmap v6550: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:54:24.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:54:24 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:54:24.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:54:24 smithi047 ceph-mon[77147]: pgmap v6550: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:54:25.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:54:24 smithi115 conmon[90393]: 2022-05-14T02:54:24.613+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:54:25.642 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:54:25 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:54:25.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:54:25 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:54:26.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:54:25 smithi115 conmon[90393]: 2022-05-14T02:54:25.641+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:54:26.643 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:54:26 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:54:26.644 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:54:26 smithi115 ceph-mon[78593]: pgmap v6551: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:54:26.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:54:26 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:54:26.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:54:26 smithi047 ceph-mon[77147]: pgmap v6551: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:54:27.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:54:26 smithi115 conmon[90393]: 2022-05-14T02:54:26.641+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:54:27.634 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:54:27 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:54:27.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:54:27 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:54:28.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:54:27 smithi115 conmon[90393]: 2022-05-14T02:54:27.633+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:54:28.636 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:54:28 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:54:28.636 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:54:28 smithi115 ceph-mon[78593]: pgmap v6552: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:54:28.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:54:28 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:54:28.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:54:28 smithi047 ceph-mon[77147]: pgmap v6552: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:54:29.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:54:28 smithi115 conmon[90393]: 2022-05-14T02:54:28.634+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:54:29.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:54:29 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:54:29.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:54:29 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:54:29.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:54:29 smithi115 conmon[90393]: 2022-05-14T02:54:29.597+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:54:30.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:54:30 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:54:30.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:54:30 smithi047 ceph-mon[77147]: pgmap v6553: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:54:30.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:54:30 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:54:30.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:54:30 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:54:30.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:54:30 smithi115 conmon[90393]: 2022-05-14T02:54:30.608+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:54:30.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:54:30 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:54:30.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:54:30 smithi115 ceph-mon[78593]: pgmap v6553: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:54:30.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:54:30 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:54:30.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:54:30 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:54:31.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:54:31 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:54:31.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:54:31 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 12400 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:54:31.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:54:31 smithi115 conmon[90393]: 2022-05-14T02:54:31.628+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:54:31.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:54:31 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:54:31.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:54:31 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 12400 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:54:32.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:54:32 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:54:32.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:54:32 smithi047 ceph-mon[77147]: pgmap v6554: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:54:32.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:54:32 smithi115 conmon[90393]: 2022-05-14T02:54:32.631+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:54:32.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:54:32 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:54:32.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:54:32 smithi115 ceph-mon[78593]: pgmap v6554: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:54:33.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:54:33 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:54:33.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:54:33 smithi115 conmon[90393]: 2022-05-14T02:54:33.630+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:54:33.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:54:33 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:54:34.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:54:34 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:54:34.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:54:34 smithi047 ceph-mon[77147]: pgmap v6555: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:54:34.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:54:34 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:54:34.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:54:34 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:54:34.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:54:34 smithi115 conmon[90393]: 2022-05-14T02:54:34.612+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:54:34.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:54:34 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:54:34.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:54:34 smithi115 ceph-mon[78593]: pgmap v6555: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:54:34.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:54:34 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:54:34.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:54:34 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:54:35.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:54:35 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:54:35.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:54:35 smithi115 conmon[90393]: 2022-05-14T02:54:35.632+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:54:35.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:54:35 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:54:36.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:54:36 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:54:36.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:54:36 smithi047 ceph-mon[77147]: pgmap v6556: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:54:36.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:54:36 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:54:36.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:54:36 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 12410 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:54:36.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:54:36 smithi115 conmon[90393]: 2022-05-14T02:54:36.641+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:54:36.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:54:36 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:54:36.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:54:36 smithi115 ceph-mon[78593]: pgmap v6556: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:54:36.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:54:36 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:54:36.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:54:36 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 12410 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:54:37.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:54:37 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:54:37.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:54:37 smithi115 conmon[90393]: 2022-05-14T02:54:37.643+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:54:37.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:54:37 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:54:38.657 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:54:38 smithi115 conmon[90393]: 2022-05-14T02:54:38.606+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:54:38.657 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:54:38 smithi115 ceph-mon[78593]: pgmap v6557: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:54:38.658 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:54:38 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:54:38.658 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:54:38 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:54:38.658 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:54:38 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:54:38.661 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:54:38 smithi047 ceph-mon[77147]: pgmap v6557: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:54:38.661 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:54:38 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:54:38.661 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:54:38 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:54:38.662 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:54:38 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:54:39.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:54:39 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:54:39.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:54:39 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T02:54:39.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:54:39 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T02:54:39.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:54:39 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T02:54:39.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:54:39 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' 2022-05-14T02:54:39.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:54:39 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' 2022-05-14T02:54:39.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:54:39 smithi115 conmon[90393]: 2022-05-14T02:54:39.580+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:54:39.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:54:39 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:54:39.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:54:39 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T02:54:39.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:54:39 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T02:54:39.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:54:39 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T02:54:39.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:54:39 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' 2022-05-14T02:54:39.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:54:39 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' 2022-05-14T02:54:40.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:54:40 smithi047 ceph-mon[77147]: pgmap v6558: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:54:40.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:54:40 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:54:40.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:54:40 smithi115 conmon[90393]: 2022-05-14T02:54:40.594+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:54:40.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:54:40 smithi115 ceph-mon[78593]: pgmap v6558: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:54:40.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:54:40 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:54:42.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:54:41 smithi047 conmon[78521]: 2022-05-14T02:54:41.714+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:54:42.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:54:41 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T02:54:42.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:54:41 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:54:42.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:54:41 smithi047 conmon[78521]: yield 2022-05-14T02:54:42.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:54:41 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:54:42.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:54:41 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:54:42.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:54:41 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:54:42.048 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:54:41 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T02:54:42.048 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:54:41 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:54:42.048 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:54:41 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T02:54:42.048 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:54:41 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:54:42.049 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:54:41 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T02:54:42.049 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:54:41 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:54:42.049 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:54:41 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T02:54:42.049 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:54:41 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:54:42.050 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:54:41 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T02:54:42.050 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:54:41 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:54:42.050 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:54:41 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:54:42.050 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:54:41 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:54:42.055 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:54:41 smithi115 conmon[90393]: 2022-05-14T02:54:41.639+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:54:42.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:54:42 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:54:42.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:54:42 smithi115 ceph-mon[78593]: pgmap v6559: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:54:42.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:54:42 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 12415 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:54:42.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:54:42 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:54:42.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:54:42 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:54:42.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:54:42 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' 2022-05-14T02:54:42.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:54:42 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' 2022-05-14T02:54:42.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:54:42 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:54:42.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:54:42 smithi047 ceph-mon[77147]: pgmap v6559: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:54:42.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:54:42 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 12415 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:54:42.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:54:42 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:54:42.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:54:42 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:54:42.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:54:42 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' 2022-05-14T02:54:42.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:54:42 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' 2022-05-14T02:54:43.063 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:54:42 smithi115 conmon[90393]: 2022-05-14T02:54:42.628+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:54:43.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:54:43 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:54:43.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:54:43 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:54:43.359 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T02:54:43.359 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:54:43.359 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T02:54:43.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:54:43.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:54:43.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:54:43.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T02:54:43.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:54:43.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T02:54:43.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:54:43.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T02:54:43.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:54:43.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T02:54:43.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:54:43.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T02:54:43.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:54:43.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:54:43.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:54:43.363 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:54:43 smithi115 ceph-mon[78593]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T02:54:43.363 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:54:43 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-05-14T02:54:43.364 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:54:43 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-05-14T02:54:43.364 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:54:43 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-05-14T02:54:43.364 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:54:43 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-05-14T02:54:43.364 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:54:43 smithi115 ceph-mon[78593]: Adjusting osd_memory_target on smithi047 to 185.5M 2022-05-14T02:54:43.364 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:54:43 smithi115 ceph-mon[78593]: Unable to set osd_memory_target on smithi047 to 194602541: error parsing value: Value '194602541' is below minimum 939524096 2022-05-14T02:54:43.365 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:54:43 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-05-14T02:54:43.365 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:54:43 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-05-14T02:54:43.365 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:54:43 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-05-14T02:54:43.365 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:54:43 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-05-14T02:54:43.366 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:54:43 smithi115 ceph-mon[78593]: Adjusting osd_memory_target on smithi115 to 197.0M 2022-05-14T02:54:43.366 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:54:43 smithi115 ceph-mon[78593]: Unable to set osd_memory_target on smithi115 to 206629785: error parsing value: Value '206629785' is below minimum 939524096 2022-05-14T02:54:43.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:54:43 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:54:43.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:54:43 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:54:43.542 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T02:54:43.542 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:54:43.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T02:54:43.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:54:43.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:54:43.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:54:43.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T02:54:43.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:54:43.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T02:54:43.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:54:43.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T02:54:43.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:54:43.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T02:54:43.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:54:43.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T02:54:43.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:54:43.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:54:43.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:54:43.546 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:54:43 smithi047 ceph-mon[77147]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T02:54:43.546 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:54:43 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-05-14T02:54:43.547 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:54:43 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-05-14T02:54:43.547 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:54:43 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-05-14T02:54:43.547 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:54:43 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-05-14T02:54:43.547 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:54:43 smithi047 ceph-mon[77147]: Adjusting osd_memory_target on smithi047 to 185.5M 2022-05-14T02:54:43.548 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:54:43 smithi047 ceph-mon[77147]: Unable to set osd_memory_target on smithi047 to 194602541: error parsing value: Value '194602541' is below minimum 939524096 2022-05-14T02:54:43.548 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:54:43 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-05-14T02:54:43.548 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:54:43 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-05-14T02:54:43.548 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:54:43 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-05-14T02:54:43.548 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:54:43 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-05-14T02:54:43.549 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:54:43 smithi047 ceph-mon[77147]: Adjusting osd_memory_target on smithi115 to 197.0M 2022-05-14T02:54:43.549 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:54:43 smithi047 ceph-mon[77147]: Unable to set osd_memory_target on smithi115 to 206629785: error parsing value: Value '206629785' is below minimum 939524096 2022-05-14T02:54:44.062 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:54:43 smithi115 conmon[90393]: 2022-05-14T02:54:43.635+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:54:44.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:54:44 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:54:44.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:54:44 smithi115 ceph-mon[78593]: pgmap v6560: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:54:44.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:54:44 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:54:44.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:54:44 smithi047 ceph-mon[77147]: pgmap v6560: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:54:45.042 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:54:44 smithi047 conmon[78521]: 2022-05-14T02:54:44.784+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:54:45.042 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:54:44 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T02:54:45.042 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:54:44 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:54:45.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:54:44 smithi047 conmon[78521]: yield 2022-05-14T02:54:45.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:54:44 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:54:45.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:54:44 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:54:45.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:54:44 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:54:45.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:54:44 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T02:54:45.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:54:44 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:54:45.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:54:44 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T02:54:45.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:54:44 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:54:45.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:54:44 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T02:54:45.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:54:44 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:54:45.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:54:44 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T02:54:45.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:54:44 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:54:45.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:54:44 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T02:54:45.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:54:44 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:54:45.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:54:44 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:54:45.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:54:44 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:54:45.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:54:44 smithi047 conmon[78521]: 2022-05-14T02:54:44.785+0000 7f5340779700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T02:54:45.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:54:44 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T02:54:45.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:54:44 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:54:45.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:54:44 smithi047 conmon[78521]: yield 2022-05-14T02:54:45.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:54:44 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:54:45.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:54:44 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:54:45.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:54:44 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:54:45.048 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:54:44 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T02:54:45.048 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:54:44 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:54:45.048 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:54:44 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T02:54:45.048 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:54:44 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:54:45.048 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:54:44 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T02:54:45.049 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:54:44 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:54:45.049 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:54:44 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T02:54:45.049 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:54:44 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:54:45.049 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:54:44 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T02:54:45.049 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:54:44 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:54:45.050 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:54:44 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:54:45.050 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:54:44 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:54:45.050 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:54:44 smithi047 conmon[78521]: 2022-05-14T02:54:45.050 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:54:44 smithi047 conmon[78521]: During handling of the above exception, another exception occurred: 2022-05-14T02:54:45.050 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:54:44 smithi047 conmon[78521]: 2022-05-14T02:54:45.051 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:54:44 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T02:54:45.051 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:54:44 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T02:54:45.051 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:54:44 smithi047 conmon[78521]: return f(*arg) 2022-05-14T02:54:45.051 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:54:44 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T02:54:45.052 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:54:44 smithi047 conmon[78521]: self._write_client_files(client_files, host) 2022-05-14T02:54:45.052 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:54:44 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T02:54:45.052 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:54:44 smithi047 conmon[78521]: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T02:54:45.053 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:54:44 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T02:54:45.053 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:54:44 smithi047 conmon[78521]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T02:54:45.053 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:54:44 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T02:54:45.054 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:54:44 smithi047 conmon[78521]: return self.event_loop.get_result(coro) 2022-05-14T02:54:45.054 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:54:44 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T02:54:45.054 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:54:44 smithi047 conmon[78521]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T02:54:45.054 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:54:44 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T02:54:45.055 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:54:44 smithi047 conmon[78521]: return self.__get_result() 2022-05-14T02:54:45.055 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:54:44 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T02:54:45.055 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:54:44 smithi047 conmon[78521]: raise self._exception 2022-05-14T02:54:45.055 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:54:44 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T02:54:45.056 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:54:44 smithi047 conmon[78521]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T02:54:45.056 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:54:44 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T02:54:45.056 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:54:44 smithi047 conmon[78521]: conn = await self._remote_connection(host, addr) 2022-05-14T02:54:45.056 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:54:44 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T02:54:45.056 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:54:44 smithi047 conmon[78521]: raise 2022-05-14T02:54:45.057 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:54:44 smithi047 conmon[78521]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T02:54:45.057 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:54:44 smithi047 conmon[78521]: self.gen.throw(type, value, traceback) 2022-05-14T02:54:45.057 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:54:44 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T02:54:45.057 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:54:44 smithi047 conmon[78521]: raise OrchestratorError(msg) 2022-05-14T02:54:45.057 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:54:44 smithi047 conmon[78521]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:54:45.069 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:54:44 smithi115 conmon[90393]: 2022-05-14T02:54:44.648+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:54:45.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:54:45 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:54:45.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:54:45 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:54:46.071 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:54:45 smithi115 conmon[90393]: 2022-05-14T02:54:45.683+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:54:46.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:54:46 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:54:46.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:54:46 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:54:46.359 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T02:54:46.359 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:54:46.359 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T02:54:46.359 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:54:46.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:54:46.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:54:46.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T02:54:46.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:54:46.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T02:54:46.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:54:46.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T02:54:46.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:54:46.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T02:54:46.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:54:46.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T02:54:46.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:54:46.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:54:46.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:54:46.363 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:54:46 smithi115 ceph-mon[78593]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T02:54:46.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T02:54:46.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:54:46.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T02:54:46.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:54:46.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:54:46.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:54:46.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T02:54:46.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:54:46.365 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T02:54:46.365 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:54:46.365 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T02:54:46.365 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:54:46.366 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T02:54:46.366 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:54:46.366 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T02:54:46.366 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:54:46.366 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:54:46.367 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:54:46.367 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T02:54:46.367 INFO:journalctl@ceph.mon.c.smithi115.stdout: During handling of the above exception, another exception occurred: 2022-05-14T02:54:46.367 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T02:54:46.367 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T02:54:46.368 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T02:54:46.368 INFO:journalctl@ceph.mon.c.smithi115.stdout: return f(*arg) 2022-05-14T02:54:46.368 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T02:54:46.368 INFO:journalctl@ceph.mon.c.smithi115.stdout: self._write_client_files(client_files, host) 2022-05-14T02:54:46.369 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T02:54:46.369 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T02:54:46.369 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T02:54:46.369 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T02:54:46.369 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T02:54:46.370 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.event_loop.get_result(coro) 2022-05-14T02:54:46.370 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T02:54:46.370 INFO:journalctl@ceph.mon.c.smithi115.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T02:54:46.370 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T02:54:46.370 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.__get_result() 2022-05-14T02:54:46.371 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T02:54:46.371 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise self._exception 2022-05-14T02:54:46.371 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T02:54:46.371 INFO:journalctl@ceph.mon.c.smithi115.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T02:54:46.371 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T02:54:46.372 INFO:journalctl@ceph.mon.c.smithi115.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T02:54:46.372 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T02:54:46.372 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise 2022-05-14T02:54:46.372 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T02:54:46.373 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.gen.throw(type, value, traceback) 2022-05-14T02:54:46.373 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T02:54:46.373 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OrchestratorError(msg) 2022-05-14T02:54:46.373 INFO:journalctl@ceph.mon.c.smithi115.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:54:46.373 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:54:46 smithi115 ceph-mon[78593]: pgmap v6561: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:54:46.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:54:46 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:54:46.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:54:46 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:54:46.542 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T02:54:46.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:54:46.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T02:54:46.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:54:46.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:54:46.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:54:46.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T02:54:46.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:54:46.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T02:54:46.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:54:46.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T02:54:46.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:54:46.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T02:54:46.548 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:54:46.548 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T02:54:46.549 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:54:46.549 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:54:46.549 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:54:46.550 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:54:46 smithi047 ceph-mon[77147]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T02:54:46.550 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T02:54:46.551 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:54:46.551 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T02:54:46.551 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:54:46.552 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:54:46.552 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:54:46.552 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T02:54:46.556 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:54:46.556 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T02:54:46.557 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:54:46.557 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T02:54:46.558 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:54:46.558 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T02:54:46.559 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:54:46.559 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T02:54:46.559 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:54:46.559 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:54:46.560 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:54:46.560 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T02:54:46.560 INFO:journalctl@ceph.mon.b.smithi047.stdout: During handling of the above exception, another exception occurred: 2022-05-14T02:54:46.561 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T02:54:46.569 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T02:54:46.570 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T02:54:46.570 INFO:journalctl@ceph.mon.b.smithi047.stdout: return f(*arg) 2022-05-14T02:54:46.571 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T02:54:46.571 INFO:journalctl@ceph.mon.b.smithi047.stdout: self._write_client_files(client_files, host) 2022-05-14T02:54:46.572 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T02:54:46.572 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T02:54:46.572 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T02:54:46.573 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T02:54:46.573 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T02:54:46.573 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.event_loop.get_result(coro) 2022-05-14T02:54:46.574 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T02:54:46.574 INFO:journalctl@ceph.mon.b.smithi047.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T02:54:46.574 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T02:54:46.574 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.__get_result() 2022-05-14T02:54:46.575 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T02:54:46.575 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise self._exception 2022-05-14T02:54:46.575 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T02:54:46.575 INFO:journalctl@ceph.mon.b.smithi047.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T02:54:46.575 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T02:54:46.576 INFO:journalctl@ceph.mon.b.smithi047.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T02:54:46.576 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T02:54:46.576 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise 2022-05-14T02:54:46.576 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T02:54:46.577 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.gen.throw(type, value, traceback) 2022-05-14T02:54:46.577 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T02:54:46.577 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OrchestratorError(msg) 2022-05-14T02:54:46.577 INFO:journalctl@ceph.mon.b.smithi047.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:54:46.577 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:54:46 smithi047 ceph-mon[77147]: pgmap v6561: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:54:47.073 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:54:46 smithi115 conmon[90393]: 2022-05-14T02:54:46.634+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:54:47.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:54:47 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:54:47.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:54:47 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:54:48.075 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:54:47 smithi115 conmon[90393]: 2022-05-14T02:54:47.636+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:54:48.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:54:48 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:54:48.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:54:48 smithi115 ceph-mon[78593]: pgmap v6562: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:54:48.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:54:48 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:54:48.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:54:48 smithi047 ceph-mon[77147]: pgmap v6562: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:54:49.080 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:54:48 smithi115 conmon[90393]: 2022-05-14T02:54:48.633+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:54:49.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:54:49 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:54:49.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:54:49 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:54:49.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:54:49 smithi115 conmon[90393]: 2022-05-14T02:54:49.605+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:54:50.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:54:50 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:54:50.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:54:50 smithi115 ceph-mon[78593]: pgmap v6563: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:54:50.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:54:50 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:54:50.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:54:50 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:54:50.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:54:50 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:54:50.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:54:50 smithi047 ceph-mon[77147]: pgmap v6563: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:54:50.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:54:50 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:54:50.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:54:50 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:54:50.860 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:54:50 smithi115 conmon[90393]: 2022-05-14T02:54:50.588+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:54:51.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:54:51 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:54:51.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:54:51 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 12420 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:54:51.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:54:51 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:54:51.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:54:51 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 12420 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:54:52.086 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:54:51 smithi115 conmon[90393]: 2022-05-14T02:54:51.624+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:54:52.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:54:52 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:54:52.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:54:52 smithi115 ceph-mon[78593]: pgmap v6564: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:54:52.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:54:52 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:54:52.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:54:52 smithi047 ceph-mon[77147]: pgmap v6564: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:54:53.094 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:54:52 smithi115 conmon[90393]: 2022-05-14T02:54:52.620+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:54:53.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:54:53 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:54:53.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:54:53 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:54:54.093 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:54:53 smithi115 conmon[90393]: 2022-05-14T02:54:53.663+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:54:54.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:54:54 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:54:54.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:54:54 smithi115 ceph-mon[78593]: pgmap v6565: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:54:54.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:54:54 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:54:54.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:54:54 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:54:54.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:54:54 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:54:54.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:54:54 smithi047 ceph-mon[77147]: pgmap v6565: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:54:54.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:54:54 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:54:54.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:54:54 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:54:55.098 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:54:54 smithi115 conmon[90393]: 2022-05-14T02:54:54.702+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:54:55.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:54:55 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:54:55.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:54:55 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:54:56.100 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:54:55 smithi115 conmon[90393]: 2022-05-14T02:54:55.716+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:54:56.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:54:56 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:54:56.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:54:56 smithi115 ceph-mon[78593]: pgmap v6566: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:54:56.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:54:56 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 12430 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:54:56.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:54:56 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:54:56.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:54:56 smithi047 ceph-mon[77147]: pgmap v6566: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:54:56.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:54:56 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 12430 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:54:57.105 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:54:56 smithi115 conmon[90393]: 2022-05-14T02:54:56.717+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:54:57.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:54:57 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:54:57.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:54:57 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:54:58.106 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:54:57 smithi115 conmon[90393]: 2022-05-14T02:54:57.756+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:54:58.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:54:58 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:54:58.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:54:58 smithi115 ceph-mon[78593]: pgmap v6567: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:54:58.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:54:58 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:54:58.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:54:58 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:54:58.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:54:58 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:54:58.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:54:58 smithi047 ceph-mon[77147]: pgmap v6567: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:54:58.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:54:58 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:54:58.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:54:58 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:54:59.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:54:58 smithi115 conmon[90393]: 2022-05-14T02:54:58.806+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:54:59.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:54:59 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:54:59.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:54:59 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:55:00.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:54:59 smithi115 conmon[90393]: 2022-05-14T02:54:59.826+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:55:00.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:55:00 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:55:00.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:55:00 smithi047 ceph-mon[77147]: pgmap v6568: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:55:00.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:55:00 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:55:00.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:55:00 smithi115 ceph-mon[78593]: pgmap v6568: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:55:01.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:55:00 smithi115 conmon[90393]: 2022-05-14T02:55:00.796+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:55:01.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:55:01 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:55:01.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:55:01 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:55:02.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:55:01 smithi115 conmon[90393]: 2022-05-14T02:55:01.751+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:55:02.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:55:02 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:55:02.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:55:02 smithi047 ceph-mon[77147]: pgmap v6569: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:55:02.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:55:02 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 12435 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:55:02.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:55:02 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:55:02.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:55:02 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:55:02.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:55:02 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:55:02.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:55:02 smithi115 ceph-mon[78593]: pgmap v6569: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:55:02.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:55:02 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 12435 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:55:02.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:55:02 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:55:02.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:55:02 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:55:03.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:55:02 smithi115 conmon[90393]: 2022-05-14T02:55:02.708+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:55:03.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:55:03 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:55:03.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:55:03 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:55:04.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:55:03 smithi115 conmon[90393]: 2022-05-14T02:55:03.684+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:55:04.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:55:04 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:55:04.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:55:04 smithi047 ceph-mon[77147]: pgmap v6570: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:55:04.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:55:04 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:55:04.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:55:04 smithi115 ceph-mon[78593]: pgmap v6570: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:55:05.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:55:04 smithi115 conmon[90393]: 2022-05-14T02:55:04.732+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:55:05.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:55:05 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:55:05.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:55:05 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:55:06.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:55:05 smithi115 conmon[90393]: 2022-05-14T02:55:05.752+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:55:06.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:55:06 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:55:06.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:55:06 smithi047 ceph-mon[77147]: pgmap v6571: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:55:06.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:55:06 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:55:06.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:55:06 smithi115 ceph-mon[78593]: pgmap v6571: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:55:07.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:55:06 smithi115 conmon[90393]: 2022-05-14T02:55:06.728+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:55:07.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:55:07 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:55:07.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:55:07 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:55:08.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:55:07 smithi115 conmon[90393]: 2022-05-14T02:55:07.698+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:55:08.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:55:08 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:55:08.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:55:08 smithi047 ceph-mon[77147]: pgmap v6572: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:55:08.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:55:08 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:55:08.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:55:08 smithi115 ceph-mon[78593]: pgmap v6572: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:55:09.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:55:08 smithi115 conmon[90393]: 2022-05-14T02:55:08.660+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:55:09.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:55:09 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:55:09.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:55:09 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:55:10.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:55:09 smithi115 conmon[90393]: 2022-05-14T02:55:09.700+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:55:10.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:55:10 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:55:10.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:55:10 smithi047 ceph-mon[77147]: pgmap v6573: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:55:10.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:55:10 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:55:10.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:55:10 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:55:10.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:55:10 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:55:10.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:55:10 smithi115 ceph-mon[78593]: pgmap v6573: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:55:10.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:55:10 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:55:10.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:55:10 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:55:11.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:55:10 smithi115 conmon[90393]: 2022-05-14T02:55:10.657+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:55:11.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:55:11 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:55:11.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:55:11 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 12440 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:55:11.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:55:11 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:55:11.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:55:11 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 12440 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:55:12.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:55:11 smithi115 conmon[90393]: 2022-05-14T02:55:11.621+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:55:12.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:55:12 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:55:12.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:55:12 smithi047 ceph-mon[77147]: pgmap v6574: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:55:12.596 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:55:12 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:55:12.596 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:55:12 smithi115 ceph-mon[78593]: pgmap v6574: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:55:12.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:55:12 smithi115 conmon[90393]: 2022-05-14T02:55:12.594+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:55:13.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:55:13 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:55:13.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:55:13 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:55:14.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:55:13 smithi115 conmon[90393]: 2022-05-14T02:55:13.608+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:55:14.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:55:14 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:55:14.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:55:14 smithi047 ceph-mon[77147]: pgmap v6575: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:55:14.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:55:14 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:55:14.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:55:14 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:55:14.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:55:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T02:55:14.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:55:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T02:55:14.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:55:14 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:55:14.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:55:14 smithi115 ceph-mon[78593]: pgmap v6575: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:55:14.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:55:14 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:55:14.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:55:14 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:55:14.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:55:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T02:55:14.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:55:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T02:55:15.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:55:14 smithi115 conmon[90393]: 2022-05-14T02:55:14.652+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:55:15.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:55:15 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:55:15.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:55:15 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:55:16.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:55:15 smithi115 conmon[90393]: 2022-05-14T02:55:15.676+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:55:16.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:55:16 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:55:16.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:55:16 smithi047 ceph-mon[77147]: pgmap v6576: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:55:16.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:55:16 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 12450 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:55:16.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:55:16 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:55:16.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:55:16 smithi115 ceph-mon[78593]: pgmap v6576: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:55:16.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:55:16 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 12450 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:55:17.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:55:16 smithi115 conmon[90393]: 2022-05-14T02:55:16.691+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:55:17.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:55:17 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:55:17.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:55:17 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:55:18.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:55:17 smithi115 conmon[90393]: 2022-05-14T02:55:17.681+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:55:18.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:55:18 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:55:18.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:55:18 smithi047 ceph-mon[77147]: pgmap v6577: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:55:18.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:55:18 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:55:18.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:55:18 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:55:18.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:55:18 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:55:18.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:55:18 smithi115 ceph-mon[78593]: pgmap v6577: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:55:18.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:55:18 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:55:18.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:55:18 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:55:19.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:55:18 smithi115 conmon[90393]: 2022-05-14T02:55:18.729+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:55:19.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:55:19 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:55:19.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:55:19 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:55:20.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:55:19 smithi115 conmon[90393]: 2022-05-14T02:55:19.767+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:55:20.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:55:20 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:55:20.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:55:20 smithi047 ceph-mon[77147]: pgmap v6578: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:55:20.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:55:20 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:55:20.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:55:20 smithi115 ceph-mon[78593]: pgmap v6578: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:55:21.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:55:20 smithi115 conmon[90393]: 2022-05-14T02:55:20.806+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:55:21.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:55:21 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:55:21.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:55:21 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:55:22.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:55:21 smithi115 conmon[90393]: 2022-05-14T02:55:21.818+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:55:22.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:55:22 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:55:22.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:55:22 smithi047 ceph-mon[77147]: pgmap v6579: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:55:22.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:55:22 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 12455 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:55:22.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:55:22 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:55:22.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:55:22 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:55:22.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:55:22 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:55:22.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:55:22 smithi115 ceph-mon[78593]: pgmap v6579: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:55:22.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:55:22 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 12455 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:55:22.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:55:22 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:55:22.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:55:22 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:55:23.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:55:22 smithi115 conmon[90393]: 2022-05-14T02:55:22.791+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:55:23.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:55:23 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:55:23.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:55:23 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:55:24.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:55:23 smithi115 conmon[90393]: 2022-05-14T02:55:23.814+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:55:24.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:55:24 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:55:24.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:55:24 smithi047 ceph-mon[77147]: pgmap v6580: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:55:24.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:55:24 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:55:24.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:55:24 smithi115 ceph-mon[78593]: pgmap v6580: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:55:25.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:55:24 smithi115 conmon[90393]: 2022-05-14T02:55:24.820+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:55:25.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:55:25 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:55:25.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:55:25 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:55:26.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:55:25 smithi115 conmon[90393]: 2022-05-14T02:55:25.845+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:55:26.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:55:26 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:55:26.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:55:26 smithi047 ceph-mon[77147]: pgmap v6581: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:55:26.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:55:26 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:55:26.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:55:26 smithi115 ceph-mon[78593]: pgmap v6581: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:55:27.208 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:55:26 smithi115 conmon[90393]: 2022-05-14T02:55:26.858+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:55:27.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:55:27 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:55:27.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:55:27 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:55:28.210 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:55:27 smithi115 conmon[90393]: 2022-05-14T02:55:27.881+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:55:28.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:55:28 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:55:28.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:55:28 smithi047 ceph-mon[77147]: pgmap v6582: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:55:28.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:55:28 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:55:28.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:55:28 smithi115 ceph-mon[78593]: pgmap v6582: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:55:29.211 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:55:28 smithi115 conmon[90393]: 2022-05-14T02:55:28.875+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:55:29.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:55:29 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:55:29.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:55:29 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:55:30.216 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:55:29 smithi115 conmon[90393]: 2022-05-14T02:55:29.882+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:55:30.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:55:30 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:55:30.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:55:30 smithi047 ceph-mon[77147]: pgmap v6583: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:55:30.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:55:30 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:55:30.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:55:30 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:55:30.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:55:30 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:55:30.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:55:30 smithi115 ceph-mon[78593]: pgmap v6583: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:55:30.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:55:30 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:55:30.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:55:30 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:55:31.221 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:55:30 smithi115 conmon[90393]: 2022-05-14T02:55:30.887+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:55:31.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:55:31 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:55:31.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:55:31 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 12460 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:55:31.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:55:31 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:55:31.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:55:31 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 12460 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:55:32.223 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:55:31 smithi115 conmon[90393]: 2022-05-14T02:55:31.883+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:55:32.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:55:32 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:55:32.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:55:32 smithi047 ceph-mon[77147]: pgmap v6584: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:55:32.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:55:32 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:55:32.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:55:32 smithi115 ceph-mon[78593]: pgmap v6584: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:55:33.228 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:55:32 smithi115 conmon[90393]: 2022-05-14T02:55:32.888+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:55:33.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:55:33 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:55:33.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:55:33 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:55:34.229 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:55:33 smithi115 conmon[90393]: 2022-05-14T02:55:33.897+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:55:34.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:55:34 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:55:34.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:55:34 smithi047 ceph-mon[77147]: pgmap v6585: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:55:34.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:55:34 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:55:34.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:55:34 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:55:34.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:55:34 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:55:34.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:55:34 smithi115 ceph-mon[78593]: pgmap v6585: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:55:34.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:55:34 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:55:34.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:55:34 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:55:35.234 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:55:34 smithi115 conmon[90393]: 2022-05-14T02:55:34.931+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:55:35.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:55:35 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:55:35.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:55:35 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:55:36.236 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:55:35 smithi115 conmon[90393]: 2022-05-14T02:55:35.958+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:55:36.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:55:36 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:55:36.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:55:36 smithi047 ceph-mon[77147]: pgmap v6586: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:55:36.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:55:36 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 12470 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:55:36.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:55:36 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:55:36.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:55:36 smithi115 ceph-mon[78593]: pgmap v6586: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:55:36.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:55:36 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 12470 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:55:37.242 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:55:36 smithi115 conmon[90393]: 2022-05-14T02:55:36.920+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:55:37.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:55:37 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:55:37.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:55:37 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:55:38.242 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:55:37 smithi115 conmon[90393]: 2022-05-14T02:55:37.923+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:55:38.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:55:38 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:55:38.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:55:38 smithi047 ceph-mon[77147]: pgmap v6587: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:55:38.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:55:38 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:55:38.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:55:38 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:55:38.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:55:38 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:55:38.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:55:38 smithi115 ceph-mon[78593]: pgmap v6587: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:55:38.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:55:38 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:55:38.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:55:38 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:55:39.247 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:55:38 smithi115 conmon[90393]: 2022-05-14T02:55:38.887+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:55:39.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:55:39 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:55:39.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:55:39 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:55:40.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:55:39 smithi115 conmon[90393]: 2022-05-14T02:55:39.848+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:55:40.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:55:40 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:55:40.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:55:40 smithi047 ceph-mon[77147]: pgmap v6588: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:55:40.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:55:40 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:55:40.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:55:40 smithi115 ceph-mon[78593]: pgmap v6588: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:55:41.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:55:40 smithi115 conmon[90393]: 2022-05-14T02:55:40.848+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:55:41.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:55:41 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:55:41.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:55:41 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:55:42.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:55:41 smithi115 conmon[90393]: 2022-05-14T02:55:41.808+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:55:42.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:55:42 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:55:42.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:55:42 smithi047 ceph-mon[77147]: pgmap v6589: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:55:42.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:55:42 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 12475 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:55:42.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:55:42 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:55:42.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:55:42 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:55:42.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:55:42 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:55:42.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:55:42 smithi115 ceph-mon[78593]: pgmap v6589: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:55:42.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:55:42 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 12475 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:55:42.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:55:42 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:55:42.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:55:42 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:55:43.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:55:42 smithi115 conmon[90393]: 2022-05-14T02:55:42.828+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:55:43.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:55:43 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:55:43.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:55:43 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:55:44.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:55:43 smithi115 conmon[90393]: 2022-05-14T02:55:43.843+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:55:44.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:55:44 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:55:44.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:55:44 smithi047 ceph-mon[77147]: pgmap v6590: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:55:44.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:55:44 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:55:44.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:55:44 smithi115 ceph-mon[78593]: pgmap v6590: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:55:45.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:55:44 smithi115 conmon[90393]: 2022-05-14T02:55:44.837+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:55:45.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:55:45 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:55:45.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:55:45 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T02:55:45.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:55:45 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T02:55:45.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:55:45 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T02:55:45.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:55:45 smithi047 conmon[78521]: 2022-05-14T02:55:45.391+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:55:45.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:55:45 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T02:55:45.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:55:45 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:55:45.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:55:45 smithi047 conmon[78521]: yield 2022-05-14T02:55:45.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:55:45 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:55:45.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:55:45 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:55:45.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:55:45 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:55:45.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:55:45 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T02:55:45.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:55:45 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:55:45.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:55:45 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T02:55:45.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:55:45 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:55:45.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:55:45 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T02:55:45.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:55:45 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:55:45.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:55:45 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T02:55:45.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:55:45 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:55:45.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:55:45 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T02:55:45.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:55:45 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:55:45.547 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:55:45 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:55:45.547 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:55:45 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:55:45.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:55:45 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:55:45.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:55:45 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T02:55:45.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:55:45 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T02:55:45.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:55:45 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T02:55:46.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:55:45 smithi115 conmon[90393]: 2022-05-14T02:55:45.835+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:55:46.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:55:46 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:55:46.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:55:46 smithi047 ceph-mon[77147]: pgmap v6591: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:55:46.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:55:46 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:55:46.542 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T02:55:46.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:55:46.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T02:55:46.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:55:46.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:55:46.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:55:46.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T02:55:46.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:55:46.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T02:55:46.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:55:46.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T02:55:46.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:55:46.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T02:55:46.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:55:46.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T02:55:46.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:55:46.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:55:46.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:55:46.546 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:55:46 smithi047 ceph-mon[77147]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T02:55:46.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:55:46 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:55:46.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:55:46 smithi115 ceph-mon[78593]: pgmap v6591: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:55:46.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:55:46 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:55:46.609 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T02:55:46.609 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:55:46.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T02:55:46.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:55:46.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:55:46.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:55:46.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T02:55:46.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:55:46.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T02:55:46.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:55:46.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T02:55:46.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:55:46.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T02:55:46.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:55:46.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T02:55:46.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:55:46.613 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:55:46.613 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:55:46.613 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:55:46 smithi115 ceph-mon[78593]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T02:55:47.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:55:46 smithi115 conmon[90393]: 2022-05-14T02:55:46.789+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:55:47.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:55:47 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:55:47.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:55:47 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:55:48.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:55:47 smithi115 conmon[90393]: 2022-05-14T02:55:47.833+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:55:48.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:55:48 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:55:48.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:55:48 smithi047 ceph-mon[77147]: pgmap v6592: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:55:48.542 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:55:48 smithi047 conmon[78521]: 2022-05-14T02:55:48.464+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:55:48.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:55:48 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T02:55:48.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:55:48 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:55:48.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:55:48 smithi047 conmon[78521]: yield 2022-05-14T02:55:48.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:55:48 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:55:48.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:55:48 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:55:48.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:55:48 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:55:48.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:55:48 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T02:55:48.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:55:48 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:55:48.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:55:48 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T02:55:48.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:55:48 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:55:48.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:55:48 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T02:55:48.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:55:48 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:55:48.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:55:48 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T02:55:48.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:55:48 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:55:48.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:55:48 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T02:55:48.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:55:48 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:55:48.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:55:48 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:55:48.547 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:55:48 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:55:48.547 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:55:48 smithi047 conmon[78521]: 2022-05-14T02:55:48.465+0000 7f5344781700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T02:55:48.547 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:55:48 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T02:55:48.547 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:55:48 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:55:48.547 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:55:48 smithi047 conmon[78521]: yield 2022-05-14T02:55:48.548 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:55:48 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:55:48.548 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:55:48 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:55:48.548 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:55:48 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:55:48.548 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:55:48 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T02:55:48.549 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:55:48 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:55:48.549 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:55:48 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T02:55:48.549 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:55:48 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:55:48.549 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:55:48 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T02:55:48.550 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:55:48 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:55:48.550 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:55:48 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T02:55:48.550 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:55:48 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:55:48.550 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:55:48 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T02:55:48.550 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:55:48 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:55:48.551 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:55:48 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:55:48.551 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:55:48 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:55:48.551 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:55:48 smithi047 conmon[78521]: 2022-05-14T02:55:48.551 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:55:48 smithi047 conmon[78521]: During handling of the above exception, another exception occurred: 2022-05-14T02:55:48.552 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:55:48 smithi047 conmon[78521]: 2022-05-14T02:55:48.552 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:55:48 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T02:55:48.552 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:55:48 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T02:55:48.552 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:55:48 smithi047 conmon[78521]: return f(*arg) 2022-05-14T02:55:48.552 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:55:48 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T02:55:48.553 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:55:48 smithi047 conmon[78521]: self._write_client_files(client_files, host) 2022-05-14T02:55:48.553 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:55:48 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T02:55:48.553 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:55:48 smithi047 conmon[78521]: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T02:55:48.553 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:55:48 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T02:55:48.553 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:55:48 smithi047 conmon[78521]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T02:55:48.554 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:55:48 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T02:55:48.554 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:55:48 smithi047 conmon[78521]: return self.event_loop.get_result(coro) 2022-05-14T02:55:48.554 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:55:48 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T02:55:48.554 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:55:48 smithi047 conmon[78521]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T02:55:48.555 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:55:48 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T02:55:48.555 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:55:48 smithi047 conmon[78521]: return self.__get_result() 2022-05-14T02:55:48.555 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:55:48 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T02:55:48.555 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:55:48 smithi047 conmon[78521]: raise self._exception 2022-05-14T02:55:48.555 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:55:48 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T02:55:48.556 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:55:48 smithi047 conmon[78521]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T02:55:48.556 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:55:48 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T02:55:48.556 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:55:48 smithi047 conmon[78521]: conn = await self._remote_connection(host, addr) 2022-05-14T02:55:48.556 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:55:48 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T02:55:48.557 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:55:48 smithi047 conmon[78521]: raise 2022-05-14T02:55:48.557 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:55:48 smithi047 conmon[78521]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T02:55:48.557 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:55:48 smithi047 conmon[78521]: self.gen.throw(type, value, traceback) 2022-05-14T02:55:48.557 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:55:48 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T02:55:48.557 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:55:48 smithi047 conmon[78521]: raise OrchestratorError(msg) 2022-05-14T02:55:48.558 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:55:48 smithi047 conmon[78521]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:55:48.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:55:48 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:55:48.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:55:48 smithi115 ceph-mon[78593]: pgmap v6592: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:55:49.281 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:55:48 smithi115 conmon[90393]: 2022-05-14T02:55:48.883+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:55:49.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:55:49 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:55:49.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:55:49 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:55:49.542 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T02:55:49.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:55:49.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T02:55:49.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:55:49.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:55:49.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:55:49.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T02:55:49.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:55:49.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T02:55:49.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:55:49.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T02:55:49.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:55:49.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T02:55:49.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:55:49.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T02:55:49.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:55:49.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:55:49.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:55:49.547 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:55:49 smithi047 ceph-mon[77147]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T02:55:49.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T02:55:49.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:55:49.548 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T02:55:49.548 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:55:49.548 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:55:49.548 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:55:49.548 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T02:55:49.549 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:55:49.549 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T02:55:49.549 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:55:49.549 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T02:55:49.550 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:55:49.550 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T02:55:49.551 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:55:49.551 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T02:55:49.551 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:55:49.551 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:55:49.551 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:55:49.552 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T02:55:49.552 INFO:journalctl@ceph.mon.b.smithi047.stdout: During handling of the above exception, another exception occurred: 2022-05-14T02:55:49.552 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T02:55:49.552 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T02:55:49.553 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T02:55:49.553 INFO:journalctl@ceph.mon.b.smithi047.stdout: return f(*arg) 2022-05-14T02:55:49.553 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T02:55:49.553 INFO:journalctl@ceph.mon.b.smithi047.stdout: self._write_client_files(client_files, host) 2022-05-14T02:55:49.554 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T02:55:49.554 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T02:55:49.554 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T02:55:49.554 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T02:55:49.554 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T02:55:49.555 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.event_loop.get_result(coro) 2022-05-14T02:55:49.555 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T02:55:49.555 INFO:journalctl@ceph.mon.b.smithi047.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T02:55:49.555 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T02:55:49.556 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.__get_result() 2022-05-14T02:55:49.556 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T02:55:49.556 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise self._exception 2022-05-14T02:55:49.556 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T02:55:49.556 INFO:journalctl@ceph.mon.b.smithi047.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T02:55:49.557 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T02:55:49.557 INFO:journalctl@ceph.mon.b.smithi047.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T02:55:49.557 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T02:55:49.557 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise 2022-05-14T02:55:49.558 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T02:55:49.558 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.gen.throw(type, value, traceback) 2022-05-14T02:55:49.558 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T02:55:49.558 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OrchestratorError(msg) 2022-05-14T02:55:49.558 INFO:journalctl@ceph.mon.b.smithi047.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:55:49.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:55:49 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:55:49.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:55:49 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:55:49.609 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T02:55:49.609 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:55:49.609 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T02:55:49.609 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:55:49.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:55:49.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:55:49.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T02:55:49.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:55:49.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T02:55:49.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:55:49.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T02:55:49.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:55:49.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T02:55:49.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:55:49.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T02:55:49.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:55:49.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:55:49.613 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:55:49.613 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:55:49 smithi115 ceph-mon[78593]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T02:55:49.613 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T02:55:49.613 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:55:49.614 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T02:55:49.614 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:55:49.614 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:55:49.615 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:55:49.615 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T02:55:49.615 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:55:49.615 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T02:55:49.615 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:55:49.616 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T02:55:49.616 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:55:49.616 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T02:55:49.616 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:55:49.617 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T02:55:49.617 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:55:49.617 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:55:49.617 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:55:49.618 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T02:55:49.618 INFO:journalctl@ceph.mon.c.smithi115.stdout: During handling of the above exception, another exception occurred: 2022-05-14T02:55:49.618 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T02:55:49.619 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T02:55:49.619 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T02:55:49.619 INFO:journalctl@ceph.mon.c.smithi115.stdout: return f(*arg) 2022-05-14T02:55:49.619 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T02:55:49.619 INFO:journalctl@ceph.mon.c.smithi115.stdout: self._write_client_files(client_files, host) 2022-05-14T02:55:49.620 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T02:55:49.620 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T02:55:49.620 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T02:55:49.620 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T02:55:49.621 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T02:55:49.621 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.event_loop.get_result(coro) 2022-05-14T02:55:49.621 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T02:55:49.621 INFO:journalctl@ceph.mon.c.smithi115.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T02:55:49.621 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T02:55:49.622 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.__get_result() 2022-05-14T02:55:49.622 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T02:55:49.622 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise self._exception 2022-05-14T02:55:49.622 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T02:55:49.622 INFO:journalctl@ceph.mon.c.smithi115.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T02:55:49.623 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T02:55:49.623 INFO:journalctl@ceph.mon.c.smithi115.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T02:55:49.623 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T02:55:49.623 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise 2022-05-14T02:55:49.624 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T02:55:49.624 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.gen.throw(type, value, traceback) 2022-05-14T02:55:49.624 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T02:55:49.624 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OrchestratorError(msg) 2022-05-14T02:55:49.624 INFO:journalctl@ceph.mon.c.smithi115.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:55:50.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:55:49 smithi115 conmon[90393]: 2022-05-14T02:55:49.850+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:55:50.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:55:50 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:55:50.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:55:50 smithi047 ceph-mon[77147]: pgmap v6593: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:55:50.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:55:50 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:55:50.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:55:50 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:55:50.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:55:50 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:55:50.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:55:50 smithi115 ceph-mon[78593]: pgmap v6593: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:55:50.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:55:50 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:55:50.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:55:50 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:55:51.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:55:50 smithi115 conmon[90393]: 2022-05-14T02:55:50.846+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:55:51.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:55:51 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:55:51.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:55:51 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 12480 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:55:51.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:55:51 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:55:51.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:55:51 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 12480 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:55:52.290 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:55:51 smithi115 conmon[90393]: 2022-05-14T02:55:51.879+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:55:52.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:55:52 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:55:52.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:55:52 smithi047 ceph-mon[77147]: pgmap v6594: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:55:52.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:55:52 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:55:52.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:55:52 smithi115 ceph-mon[78593]: pgmap v6594: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:55:53.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:55:52 smithi115 conmon[90393]: 2022-05-14T02:55:52.848+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:55:53.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:55:53 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:55:53.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:55:53 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:55:54.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:55:53 smithi115 conmon[90393]: 2022-05-14T02:55:53.822+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:55:54.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:55:54 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:55:54.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:55:54 smithi115 ceph-mon[78593]: pgmap v6595: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:55:54.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:55:54 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:55:54.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:55:54 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:55:54.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:55:54 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:55:54.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:55:54 smithi047 ceph-mon[77147]: pgmap v6595: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:55:54.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:55:54 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:55:54.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:55:54 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:55:55.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:55:54 smithi115 conmon[90393]: 2022-05-14T02:55:54.776+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:55:55.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:55:55 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:55:55.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:55:55 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:55:56.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:55:55 smithi115 conmon[90393]: 2022-05-14T02:55:55.755+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:55:56.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:55:56 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:55:56.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:55:56 smithi115 ceph-mon[78593]: pgmap v6596: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:55:56.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:55:56 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 12490 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:55:56.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:55:56 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:55:56.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:55:56 smithi047 ceph-mon[77147]: pgmap v6596: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:55:56.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:55:56 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 12490 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:55:57.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:55:56 smithi115 conmon[90393]: 2022-05-14T02:55:56.784+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:55:57.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:55:57 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:55:57.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:55:57 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:55:58.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:55:57 smithi115 conmon[90393]: 2022-05-14T02:55:57.744+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:55:58.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:55:58 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:55:58.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:55:58 smithi115 ceph-mon[78593]: pgmap v6597: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:55:58.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:55:58 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:55:58.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:55:58 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:55:58.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:55:58 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:55:58.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:55:58 smithi047 ceph-mon[77147]: pgmap v6597: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:55:58.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:55:58 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:55:58.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:55:58 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:55:59.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:55:58 smithi115 conmon[90393]: 2022-05-14T02:55:58.777+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:55:59.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:55:59 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:55:59.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:55:59 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:56:00.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:55:59 smithi115 conmon[90393]: 2022-05-14T02:55:59.803+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:56:00.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:56:00 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:56:00.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:56:00 smithi115 ceph-mon[78593]: pgmap v6598: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:56:00.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:56:00 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:56:00.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:56:00 smithi047 ceph-mon[77147]: pgmap v6598: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:56:01.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:56:00 smithi115 conmon[90393]: 2022-05-14T02:56:00.813+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:56:01.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:56:01 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:56:01.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:56:01 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:56:02.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:56:01 smithi115 conmon[90393]: 2022-05-14T02:56:01.847+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:56:02.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:56:02 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:56:02.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:56:02 smithi115 ceph-mon[78593]: pgmap v6599: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:56:02.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:56:02 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 12495 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:56:02.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:56:02 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:56:02.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:56:02 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:56:02.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:56:02 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:56:02.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:56:02 smithi047 ceph-mon[77147]: pgmap v6599: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:56:02.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:56:02 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 12495 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:56:02.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:56:02 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:56:02.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:56:02 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:56:03.330 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:56:02 smithi115 conmon[90393]: 2022-05-14T02:56:02.896+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:56:03.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:56:03 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:56:03.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:56:03 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:56:04.331 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:56:03 smithi115 conmon[90393]: 2022-05-14T02:56:03.864+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:56:04.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:56:04 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:56:04.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:56:04 smithi115 ceph-mon[78593]: pgmap v6600: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:56:04.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:56:04 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:56:04.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:56:04 smithi047 ceph-mon[77147]: pgmap v6600: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:56:05.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:56:04 smithi115 conmon[90393]: 2022-05-14T02:56:04.827+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:56:05.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:56:05 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:56:05.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:56:05 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:56:06.337 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:56:05 smithi115 conmon[90393]: 2022-05-14T02:56:05.872+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:56:06.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:56:06 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:56:06.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:56:06 smithi115 ceph-mon[78593]: pgmap v6601: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:56:06.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:56:06 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:56:06.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:56:06 smithi047 ceph-mon[77147]: pgmap v6601: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:56:07.343 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:56:06 smithi115 conmon[90393]: 2022-05-14T02:56:06.907+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:56:07.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:56:07 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:56:07.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:56:07 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:56:08.344 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:56:07 smithi115 conmon[90393]: 2022-05-14T02:56:07.863+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:56:08.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:56:08 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:56:08.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:56:08 smithi115 ceph-mon[78593]: pgmap v6602: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:56:08.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:56:08 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:56:08.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:56:08 smithi047 ceph-mon[77147]: pgmap v6602: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:56:09.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:56:08 smithi115 conmon[90393]: 2022-05-14T02:56:08.828+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:56:09.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:56:09 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:56:09.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:56:09 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:56:10.351 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:56:09 smithi115 conmon[90393]: 2022-05-14T02:56:09.871+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:56:10.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:56:10 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:56:10.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:56:10 smithi115 ceph-mon[78593]: pgmap v6603: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:56:10.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:56:10 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:56:10.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:56:10 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:56:10.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:56:10 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:56:10.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:56:10 smithi047 ceph-mon[77147]: pgmap v6603: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:56:10.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:56:10 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:56:10.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:56:10 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:56:11.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:56:10 smithi115 conmon[90393]: 2022-05-14T02:56:10.907+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:56:11.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:56:11 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:56:11.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:56:11 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 12500 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:56:11.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:56:11 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:56:11.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:56:11 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 12500 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:56:12.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:56:11 smithi115 conmon[90393]: 2022-05-14T02:56:11.874+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:56:12.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:56:12 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:56:12.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:56:12 smithi047 ceph-mon[77147]: pgmap v6604: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:56:12.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:56:12 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:56:12.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:56:12 smithi115 ceph-mon[78593]: pgmap v6604: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:56:13.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:56:12 smithi115 conmon[90393]: 2022-05-14T02:56:12.858+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:56:13.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:56:13 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:56:13.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:56:13 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:56:14.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:56:13 smithi115 conmon[90393]: 2022-05-14T02:56:13.868+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:56:14.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:56:14 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:56:14.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:56:14 smithi047 ceph-mon[77147]: pgmap v6605: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:56:14.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:56:14 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:56:14.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:56:14 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:56:14.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:56:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T02:56:14.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:56:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T02:56:14.856 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:56:14 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:56:14.856 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:56:14 smithi115 ceph-mon[78593]: pgmap v6605: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:56:14.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:56:14 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:56:14.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:56:14 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:56:14.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:56:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T02:56:14.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:56:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T02:56:15.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:56:14 smithi115 conmon[90393]: 2022-05-14T02:56:14.854+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:56:15.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:56:15 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:56:15.855 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:56:15 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:56:16.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:56:15 smithi115 conmon[90393]: 2022-05-14T02:56:15.853+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:56:16.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:56:16 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:56:16.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:56:16 smithi047 ceph-mon[77147]: pgmap v6606: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:56:16.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:56:16 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 12510 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:56:16.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:56:16 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:56:16.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:56:16 smithi115 ceph-mon[78593]: pgmap v6606: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:56:16.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:56:16 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 12510 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:56:17.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:56:16 smithi115 conmon[90393]: 2022-05-14T02:56:16.898+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:56:17.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:56:17 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:56:17.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:56:17 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:56:18.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:56:17 smithi115 conmon[90393]: 2022-05-14T02:56:17.905+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:56:18.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:56:18 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:56:18.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:56:18 smithi047 ceph-mon[77147]: pgmap v6607: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:56:18.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:56:18 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:56:18.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:56:18 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:56:18.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:56:18 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:56:18.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:56:18 smithi115 ceph-mon[78593]: pgmap v6607: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:56:18.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:56:18 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:56:18.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:56:18 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:56:19.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:56:18 smithi115 conmon[90393]: 2022-05-14T02:56:18.905+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:56:19.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:56:19 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:56:19.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:56:19 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:56:20.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:56:19 smithi115 conmon[90393]: 2022-05-14T02:56:19.895+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:56:20.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:56:20 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:56:20.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:56:20 smithi047 ceph-mon[77147]: pgmap v6608: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:56:20.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:56:20 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:56:20.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:56:20 smithi115 ceph-mon[78593]: pgmap v6608: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:56:21.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:56:20 smithi115 conmon[90393]: 2022-05-14T02:56:20.874+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:56:21.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:56:21 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:56:21.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:56:21 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:56:22.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:56:21 smithi115 conmon[90393]: 2022-05-14T02:56:21.907+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:56:22.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:56:22 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:56:22.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:56:22 smithi047 ceph-mon[77147]: pgmap v6609: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:56:22.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:56:22 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 12515 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:56:22.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:56:22 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:56:22.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:56:22 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:56:22.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:56:22 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:56:22.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:56:22 smithi115 ceph-mon[78593]: pgmap v6609: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:56:22.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:56:22 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 12515 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:56:22.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:56:22 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:56:22.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:56:22 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:56:23.365 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:56:22 smithi115 conmon[90393]: 2022-05-14T02:56:22.926+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:56:23.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:56:23 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:56:23.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:56:23 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:56:24.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:56:23 smithi115 conmon[90393]: 2022-05-14T02:56:23.914+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:56:24.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:56:24 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:56:24.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:56:24 smithi047 ceph-mon[77147]: pgmap v6610: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:56:24.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:56:24 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:56:24.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:56:24 smithi115 ceph-mon[78593]: pgmap v6610: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:56:25.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:56:24 smithi115 conmon[90393]: 2022-05-14T02:56:24.963+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:56:25.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:56:25 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:56:25.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:56:25 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:56:26.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:56:25 smithi115 conmon[90393]: 2022-05-14T02:56:25.914+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:56:26.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:56:26 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:56:26.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:56:26 smithi047 ceph-mon[77147]: pgmap v6611: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:56:26.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:56:26 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:56:26.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:56:26 smithi115 ceph-mon[78593]: pgmap v6611: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:56:27.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:56:26 smithi115 conmon[90393]: 2022-05-14T02:56:26.927+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:56:27.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:56:27 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:56:27.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:56:27 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:56:28.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:56:27 smithi115 conmon[90393]: 2022-05-14T02:56:27.915+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:56:28.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:56:28 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:56:28.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:56:28 smithi047 ceph-mon[77147]: pgmap v6612: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:56:28.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:56:28 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:56:28.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:56:28 smithi115 ceph-mon[78593]: pgmap v6612: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:56:29.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:56:28 smithi115 conmon[90393]: 2022-05-14T02:56:28.960+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:56:29.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:56:29 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:56:29.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:56:29 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:56:30.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:56:29 smithi115 conmon[90393]: 2022-05-14T02:56:29.970+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:56:30.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:56:30 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:56:30.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:56:30 smithi047 ceph-mon[77147]: pgmap v6613: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:56:30.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:56:30 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:56:30.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:56:30 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:56:30.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:56:30 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:56:30.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:56:30 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:56:30.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:56:30 smithi115 ceph-mon[78593]: pgmap v6613: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:56:30.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:56:30 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:56:30.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:56:30 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:56:30.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:56:30 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:56:31.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:56:30 smithi115 conmon[90393]: 2022-05-14T02:56:30.992+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:56:31.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:56:31 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:56:31.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:56:31 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 12520 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:56:31.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:56:31 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:56:31.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:56:31 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 12520 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:56:32.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:56:31 smithi115 conmon[90393]: 2022-05-14T02:56:31.958+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:56:32.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:56:32 smithi047 ceph-mon[77147]: pgmap v6614: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:56:32.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:56:32 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:56:32.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:56:32 smithi115 ceph-mon[78593]: pgmap v6614: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:56:32.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:56:32 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:56:33.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:56:32 smithi115 conmon[90393]: 2022-05-14T02:56:32.937+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:56:33.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:56:33 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:56:33.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:56:33 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:56:34.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:56:33 smithi115 conmon[90393]: 2022-05-14T02:56:33.952+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:56:34.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:56:34 smithi047 ceph-mon[77147]: pgmap v6615: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:56:34.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:56:34 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:56:34.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:56:34 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:56:34.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:56:34 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:56:34.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:56:34 smithi115 ceph-mon[78593]: pgmap v6615: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:56:34.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:56:34 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:56:34.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:56:34 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:56:34.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:56:34 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:56:35.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:56:34 smithi115 conmon[90393]: 2022-05-14T02:56:34.929+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:56:35.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:56:35 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:56:35.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:56:35 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:56:36.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:56:35 smithi115 conmon[90393]: 2022-05-14T02:56:35.940+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:56:36.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:56:36 smithi047 ceph-mon[77147]: pgmap v6616: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:56:36.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:56:36 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:56:36.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:56:36 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 12530 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:56:36.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:56:36 smithi115 ceph-mon[78593]: pgmap v6616: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:56:36.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:56:36 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:56:36.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:56:36 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 12530 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:56:37.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:56:36 smithi115 conmon[90393]: 2022-05-14T02:56:36.924+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:56:37.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:56:37 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:56:37.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:56:37 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:56:38.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:56:37 smithi115 conmon[90393]: 2022-05-14T02:56:37.926+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:56:38.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:56:38 smithi047 ceph-mon[77147]: pgmap v6617: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:56:38.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:56:38 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:56:38.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:56:38 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:56:38.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:56:38 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:56:38.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:56:38 smithi115 ceph-mon[78593]: pgmap v6617: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:56:38.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:56:38 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:56:38.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:56:38 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:56:38.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:56:38 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:56:39.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:56:38 smithi115 conmon[90393]: 2022-05-14T02:56:38.904+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:56:39.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:56:39 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:56:39.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:56:39 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:56:40.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:56:39 smithi115 conmon[90393]: 2022-05-14T02:56:39.949+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:56:40.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:56:40 smithi047 ceph-mon[77147]: pgmap v6618: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:56:40.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:56:40 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:56:40.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:56:40 smithi115 ceph-mon[78593]: pgmap v6618: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:56:40.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:56:40 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:56:41.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:56:40 smithi115 conmon[90393]: 2022-05-14T02:56:40.935+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:56:42.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:56:41 smithi115 conmon[90393]: 2022-05-14T02:56:41.939+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:56:42.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:56:42 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:56:42.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:56:42 smithi115 ceph-mon[78593]: pgmap v6619: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:56:42.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:56:42 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 12535 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:56:42.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:56:42 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:56:42.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:56:42 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:56:42.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:56:42 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:56:42.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:56:42 smithi047 ceph-mon[77147]: pgmap v6619: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:56:42.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:56:42 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 12535 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:56:42.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:56:42 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:56:42.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:56:42 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:56:43.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:56:42 smithi115 conmon[90393]: 2022-05-14T02:56:42.950+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:56:43.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:56:43 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:56:43.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:56:43 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:56:44.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:56:43 smithi115 conmon[90393]: 2022-05-14T02:56:43.978+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:56:44.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:56:44 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:56:44.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:56:44 smithi115 ceph-mon[78593]: pgmap v6620: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:56:44.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:56:44 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:56:44.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:56:44 smithi047 ceph-mon[77147]: pgmap v6620: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:56:45.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:56:44 smithi115 conmon[90393]: 2022-05-14T02:56:44.984+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:56:45.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:56:45 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:56:45.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:56:45 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:56:46.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:56:45 smithi115 conmon[90393]: 2022-05-14T02:56:45.994+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:56:46.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:56:46 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:56:46.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:56:46 smithi115 ceph-mon[78593]: pgmap v6621: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:56:46.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:56:46 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:56:46.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:56:46 smithi047 ceph-mon[77147]: pgmap v6621: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:56:47.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:56:46 smithi115 conmon[90393]: 2022-05-14T02:56:46.945+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:56:47.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:56:47 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:56:47.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:56:47 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:56:48.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:56:47 smithi115 conmon[90393]: 2022-05-14T02:56:47.960+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:56:48.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:56:48 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:56:48.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:56:48 smithi115 ceph-mon[78593]: pgmap v6622: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:56:48.512 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:56:48 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:56:48.512 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:56:48 smithi047 ceph-mon[77147]: pgmap v6622: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:56:49.267 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:56:49 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:56:49.267 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:56:49 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T02:56:49.267 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:56:49 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T02:56:49.268 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:56:49 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T02:56:49.268 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:56:48 smithi115 conmon[90393]: 2022-05-14T02:56:48.994+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:56:49.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:56:49 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:56:49.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:56:49 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T02:56:49.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:56:49 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T02:56:49.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:56:49 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T02:56:50.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:56:49 smithi115 conmon[90393]: 2022-05-14T02:56:49.958+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:56:50.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:56:50 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:56:50.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:56:50 smithi115 ceph-mon[78593]: pgmap v6623: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:56:50.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:56:50 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:56:50.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:56:50 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:56:50.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:56:50 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:56:50.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:56:50 smithi047 ceph-mon[77147]: pgmap v6623: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:56:50.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:56:50 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:56:50.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:56:50 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:56:51.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:56:50 smithi115 conmon[90393]: 2022-05-14T02:56:50.983+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:56:51.380 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:56:51 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:56:51.381 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:56:51 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 12540 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:56:51.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:56:51 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:56:51.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:56:51 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 12540 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:56:52.042 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:56:51 smithi047 conmon[78521]: 2022-05-14T02:56:51.568+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:56:52.042 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:56:51 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T02:56:52.042 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:56:51 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:56:52.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:56:51 smithi047 conmon[78521]: yield 2022-05-14T02:56:52.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:56:51 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:56:52.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:56:51 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:56:52.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:56:51 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:56:52.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:56:51 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T02:56:52.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:56:51 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:56:52.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:56:51 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T02:56:52.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:56:51 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:56:52.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:56:51 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T02:56:52.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:56:51 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:56:52.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:56:51 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T02:56:52.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:56:51 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:56:52.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:56:51 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T02:56:52.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:56:51 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:56:52.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:56:51 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:56:52.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:56:51 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:56:52.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:56:51 smithi115 conmon[90393]: 2022-05-14T02:56:51.937+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:56:52.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:56:52 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:56:52.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:56:52 smithi115 ceph-mon[78593]: pgmap v6624: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:56:52.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:56:52 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:56:52.359 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T02:56:52.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:56:52.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T02:56:52.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:56:52.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:56:52.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:56:52.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T02:56:52.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:56:52.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T02:56:52.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:56:52.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T02:56:52.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:56:52.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T02:56:52.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:56:52.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T02:56:52.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:56:52.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:56:52.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:56:52.364 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:56:52 smithi115 ceph-mon[78593]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T02:56:52.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:56:52 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:56:52.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:56:52 smithi047 ceph-mon[77147]: pgmap v6624: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:56:52.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:56:52 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:56:52.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T02:56:52.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:56:52.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T02:56:52.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:56:52.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:56:52.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:56:52.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T02:56:52.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:56:52.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T02:56:52.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:56:52.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T02:56:52.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:56:52.548 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T02:56:52.548 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:56:52.549 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T02:56:52.549 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:56:52.550 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:56:52.550 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:56:52.550 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:56:52 smithi047 ceph-mon[77147]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T02:56:53.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:56:52 smithi115 conmon[90393]: 2022-05-14T02:56:52.929+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:56:53.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:56:53 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:56:53.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:56:53 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:56:54.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:56:53 smithi115 conmon[90393]: 2022-05-14T02:56:53.926+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:56:54.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:56:54 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:56:54.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:56:54 smithi115 ceph-mon[78593]: pgmap v6625: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:56:54.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:56:54 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:56:54.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:56:54 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:56:54.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:56:54 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:56:54.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:56:54 smithi047 ceph-mon[77147]: pgmap v6625: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:56:54.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:56:54 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:56:54.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:56:54 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:56:55.042 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:56:54 smithi047 conmon[78521]: 2022-05-14T02:56:54.640+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:56:55.042 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:56:54 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T02:56:55.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:56:54 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:56:55.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:56:54 smithi047 conmon[78521]: yield 2022-05-14T02:56:55.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:56:54 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:56:55.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:56:54 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:56:55.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:56:54 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:56:55.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:56:54 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T02:56:55.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:56:54 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:56:55.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:56:54 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T02:56:55.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:56:54 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:56:55.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:56:54 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T02:56:55.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:56:54 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:56:55.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:56:54 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T02:56:55.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:56:54 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:56:55.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:56:54 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T02:56:55.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:56:54 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:56:55.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:56:54 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:56:55.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:56:54 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:56:55.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:56:54 smithi047 conmon[78521]: 2022-05-14T02:56:54.641+0000 7f534177b700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T02:56:55.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:56:54 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T02:56:55.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:56:54 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:56:55.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:56:54 smithi047 conmon[78521]: yield 2022-05-14T02:56:55.048 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:56:54 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:56:55.048 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:56:54 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:56:55.048 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:56:54 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:56:55.048 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:56:54 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T02:56:55.048 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:56:54 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:56:55.049 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:56:54 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T02:56:55.049 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:56:54 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:56:55.049 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:56:54 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T02:56:55.049 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:56:54 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:56:55.050 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:56:54 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T02:56:55.050 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:56:54 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:56:55.050 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:56:54 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T02:56:55.050 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:56:54 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:56:55.050 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:56:54 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:56:55.051 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:56:54 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:56:55.051 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:56:54 smithi047 conmon[78521]: 2022-05-14T02:56:55.051 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:56:54 smithi047 conmon[78521]: During handling of the above exception, another exception occurred: 2022-05-14T02:56:55.051 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:56:54 smithi047 conmon[78521]: 2022-05-14T02:56:55.052 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:56:54 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T02:56:55.052 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:56:54 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T02:56:55.052 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:56:54 smithi047 conmon[78521]: return f(*arg) 2022-05-14T02:56:55.052 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:56:54 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T02:56:55.052 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:56:54 smithi047 conmon[78521]: self._write_client_files(client_files, host) 2022-05-14T02:56:55.053 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:56:54 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T02:56:55.053 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:56:54 smithi047 conmon[78521]: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T02:56:55.053 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:56:54 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T02:56:55.053 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:56:54 smithi047 conmon[78521]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T02:56:55.054 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:56:54 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T02:56:55.054 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:56:54 smithi047 conmon[78521]: return self.event_loop.get_result(coro) 2022-05-14T02:56:55.054 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:56:54 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T02:56:55.054 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:56:54 smithi047 conmon[78521]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T02:56:55.055 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:56:54 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T02:56:55.055 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:56:54 smithi047 conmon[78521]: return self.__get_result() 2022-05-14T02:56:55.055 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:56:54 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T02:56:55.055 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:56:54 smithi047 conmon[78521]: raise self._exception 2022-05-14T02:56:55.055 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:56:54 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T02:56:55.056 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:56:54 smithi047 conmon[78521]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T02:56:55.056 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:56:54 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T02:56:55.056 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:56:54 smithi047 conmon[78521]: conn = await self._remote_connection(host, addr) 2022-05-14T02:56:55.056 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:56:54 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T02:56:55.056 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:56:54 smithi047 conmon[78521]: raise 2022-05-14T02:56:55.057 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:56:54 smithi047 conmon[78521]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T02:56:55.057 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:56:54 smithi047 conmon[78521]: self.gen.throw(type, value, traceback) 2022-05-14T02:56:55.057 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:56:54 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T02:56:55.057 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:56:54 smithi047 conmon[78521]: raise OrchestratorError(msg) 2022-05-14T02:56:55.058 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:56:54 smithi047 conmon[78521]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:56:55.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:56:54 smithi115 conmon[90393]: 2022-05-14T02:56:54.939+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:56:55.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:56:55 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:56:55.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:56:55 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:56:56.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:56:55 smithi115 conmon[90393]: 2022-05-14T02:56:55.894+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:56:56.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:56:56 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:56:56.359 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T02:56:56.359 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:56:56.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T02:56:56.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:56:56.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:56:56.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:56:56.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T02:56:56.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:56:56.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T02:56:56.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:56:56.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T02:56:56.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:56:56.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T02:56:56.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:56:56.365 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T02:56:56.365 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:56:56.365 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:56:56.366 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:56:56.366 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:56:56 smithi115 ceph-mon[78593]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T02:56:56.366 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T02:56:56.366 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:56:56.367 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T02:56:56.367 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:56:56.367 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:56:56.367 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:56:56.368 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T02:56:56.368 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:56:56.368 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T02:56:56.368 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:56:56.369 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T02:56:56.369 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:56:56.370 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T02:56:56.370 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:56:56.370 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T02:56:56.370 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:56:56.371 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:56:56.371 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:56:56.371 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T02:56:56.371 INFO:journalctl@ceph.mon.c.smithi115.stdout: During handling of the above exception, another exception occurred: 2022-05-14T02:56:56.372 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T02:56:56.372 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T02:56:56.372 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T02:56:56.372 INFO:journalctl@ceph.mon.c.smithi115.stdout: return f(*arg) 2022-05-14T02:56:56.373 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T02:56:56.373 INFO:journalctl@ceph.mon.c.smithi115.stdout: self._write_client_files(client_files, host) 2022-05-14T02:56:56.374 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T02:56:56.374 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T02:56:56.374 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T02:56:56.374 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T02:56:56.375 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T02:56:56.375 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.event_loop.get_result(coro) 2022-05-14T02:56:56.375 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T02:56:56.375 INFO:journalctl@ceph.mon.c.smithi115.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T02:56:56.375 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T02:56:56.376 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.__get_result() 2022-05-14T02:56:56.376 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T02:56:56.376 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise self._exception 2022-05-14T02:56:56.376 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T02:56:56.377 INFO:journalctl@ceph.mon.c.smithi115.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T02:56:56.377 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T02:56:56.377 INFO:journalctl@ceph.mon.c.smithi115.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T02:56:56.377 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T02:56:56.377 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise 2022-05-14T02:56:56.378 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T02:56:56.378 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.gen.throw(type, value, traceback) 2022-05-14T02:56:56.378 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T02:56:56.378 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OrchestratorError(msg) 2022-05-14T02:56:56.378 INFO:journalctl@ceph.mon.c.smithi115.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:56:56.379 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:56:56 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:56:56.379 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:56:56 smithi115 ceph-mon[78593]: pgmap v6626: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:56:56.379 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:56:56 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 12550 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:56:56.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:56:56 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:56:56.542 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T02:56:56.542 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:56:56.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T02:56:56.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:56:56.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:56:56.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:56:56.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T02:56:56.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:56:56.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T02:56:56.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:56:56.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T02:56:56.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:56:56.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T02:56:56.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:56:56.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T02:56:56.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:56:56.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:56:56.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:56:56.546 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:56:56 smithi047 ceph-mon[77147]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T02:56:56.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T02:56:56.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:56:56.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T02:56:56.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:56:56.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:56:56.548 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:56:56.548 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T02:56:56.548 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:56:56.548 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T02:56:56.548 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:56:56.549 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T02:56:56.549 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:56:56.549 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T02:56:56.549 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:56:56.549 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T02:56:56.550 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:56:56.550 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:56:56.550 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:56:56.550 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T02:56:56.550 INFO:journalctl@ceph.mon.b.smithi047.stdout: During handling of the above exception, another exception occurred: 2022-05-14T02:56:56.551 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T02:56:56.551 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T02:56:56.551 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T02:56:56.551 INFO:journalctl@ceph.mon.b.smithi047.stdout: return f(*arg) 2022-05-14T02:56:56.552 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T02:56:56.552 INFO:journalctl@ceph.mon.b.smithi047.stdout: self._write_client_files(client_files, host) 2022-05-14T02:56:56.552 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T02:56:56.552 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T02:56:56.552 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T02:56:56.553 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T02:56:56.553 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T02:56:56.553 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.event_loop.get_result(coro) 2022-05-14T02:56:56.553 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T02:56:56.553 INFO:journalctl@ceph.mon.b.smithi047.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T02:56:56.554 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T02:56:56.554 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.__get_result() 2022-05-14T02:56:56.554 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T02:56:56.554 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise self._exception 2022-05-14T02:56:56.554 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T02:56:56.555 INFO:journalctl@ceph.mon.b.smithi047.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T02:56:56.555 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T02:56:56.555 INFO:journalctl@ceph.mon.b.smithi047.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T02:56:56.555 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T02:56:56.556 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise 2022-05-14T02:56:56.556 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T02:56:56.556 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.gen.throw(type, value, traceback) 2022-05-14T02:56:56.556 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T02:56:56.556 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OrchestratorError(msg) 2022-05-14T02:56:56.557 INFO:journalctl@ceph.mon.b.smithi047.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:56:56.557 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:56:56 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:56:56.557 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:56:56 smithi047 ceph-mon[77147]: pgmap v6626: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:56:56.557 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:56:56 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 12550 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:56:57.132 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:56:56 smithi115 conmon[90393]: 2022-05-14T02:56:56.880+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:56:57.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:56:57 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:56:57.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:56:57 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:56:58.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:56:57 smithi115 conmon[90393]: 2022-05-14T02:56:57.883+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:56:58.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:56:58 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:56:58.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:56:58 smithi115 ceph-mon[78593]: pgmap v6627: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:56:58.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:56:58 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:56:58.362 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:56:58 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:56:58.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:56:58 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:56:58.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:56:58 smithi047 ceph-mon[77147]: pgmap v6627: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:56:58.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:56:58 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:56:58.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:56:58 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:56:59.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:56:59 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:56:59.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:56:58 smithi115 conmon[90393]: 2022-05-14T02:56:58.920+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:56:59.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:56:59 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:57:00.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:56:59 smithi115 conmon[90393]: 2022-05-14T02:56:59.962+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:57:00.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:57:00 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:57:00.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:57:00 smithi115 ceph-mon[78593]: pgmap v6628: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:57:00.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:57:00 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:57:00.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:57:00 smithi047 ceph-mon[77147]: pgmap v6628: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:57:01.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:57:01 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:57:01.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:57:00 smithi115 conmon[90393]: 2022-05-14T02:57:00.920+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:57:01.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:57:01 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:57:02.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:57:01 smithi115 conmon[90393]: 2022-05-14T02:57:01.905+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:57:02.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:57:02 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:57:02.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:57:02 smithi115 ceph-mon[78593]: pgmap v6629: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:57:02.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:57:02 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 12555 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:57:02.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:57:02 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:57:02.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:57:02 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:57:02.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:57:02 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:57:02.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:57:02 smithi047 ceph-mon[77147]: pgmap v6629: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:57:02.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:57:02 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 12555 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:57:02.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:57:02 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:57:02.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:57:02 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:57:03.154 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:57:02 smithi115 conmon[90393]: 2022-05-14T02:57:02.895+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:57:03.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:57:03 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:57:03.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:57:03 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:57:04.154 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:57:03 smithi115 conmon[90393]: 2022-05-14T02:57:03.863+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:57:04.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:57:04 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:57:04.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:57:04 smithi047 ceph-mon[77147]: pgmap v6630: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:57:04.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:57:04 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:57:04.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:57:04 smithi115 ceph-mon[78593]: pgmap v6630: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:57:05.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:57:04 smithi115 conmon[90393]: 2022-05-14T02:57:04.909+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:57:05.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:57:05 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:57:05.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:57:05 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:57:06.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:57:05 smithi115 conmon[90393]: 2022-05-14T02:57:05.943+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:57:06.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:57:06 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:57:06.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:57:06 smithi115 ceph-mon[78593]: pgmap v6631: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:57:06.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:57:06 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:57:06.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:57:06 smithi047 ceph-mon[77147]: pgmap v6631: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:57:07.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:57:06 smithi115 conmon[90393]: 2022-05-14T02:57:06.918+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:57:07.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:57:07 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:57:07.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:57:07 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:57:08.168 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:57:07 smithi115 conmon[90393]: 2022-05-14T02:57:07.913+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:57:08.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:57:08 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:57:08.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:57:08 smithi047 ceph-mon[77147]: pgmap v6632: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:57:08.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:57:08 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:57:08.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:57:08 smithi115 ceph-mon[78593]: pgmap v6632: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:57:09.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:57:08 smithi115 conmon[90393]: 2022-05-14T02:57:08.949+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:57:09.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:57:09 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:57:09.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:57:09 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:57:10.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:57:09 smithi115 conmon[90393]: 2022-05-14T02:57:09.935+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:57:10.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:57:10 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:57:10.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:57:10 smithi115 ceph-mon[78593]: pgmap v6633: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:57:10.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:57:10 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:57:10.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:57:10 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:57:10.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:57:10 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:57:10.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:57:10 smithi047 ceph-mon[77147]: pgmap v6633: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:57:10.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:57:10 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:57:10.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:57:10 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:57:11.183 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:57:10 smithi115 conmon[90393]: 2022-05-14T02:57:10.926+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:57:11.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:57:11 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:57:11.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:57:11 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 12560 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:57:11.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:57:11 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:57:11.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:57:11 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 12560 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:57:12.183 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:57:11 smithi115 conmon[90393]: 2022-05-14T02:57:11.894+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:57:12.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:57:12 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:57:12.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:57:12 smithi047 ceph-mon[77147]: pgmap v6634: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:57:12.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:57:12 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:57:12.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:57:12 smithi115 ceph-mon[78593]: pgmap v6634: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:57:13.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:57:12 smithi115 conmon[90393]: 2022-05-14T02:57:12.845+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:57:13.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:57:13 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:57:13.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:57:13 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:57:14.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:57:13 smithi115 conmon[90393]: 2022-05-14T02:57:13.818+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:57:14.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:57:14 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:57:14.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:57:14 smithi047 ceph-mon[77147]: pgmap v6635: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:57:14.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:57:14 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:57:14.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:57:14 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:57:14.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:57:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T02:57:14.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:57:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T02:57:14.583 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:57:14 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:57:14.584 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:57:14 smithi115 ceph-mon[78593]: pgmap v6635: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:57:14.584 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:57:14 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:57:14.584 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:57:14 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:57:14.584 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:57:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T02:57:14.585 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:57:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T02:57:15.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:57:14 smithi115 conmon[90393]: 2022-05-14T02:57:14.837+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:57:15.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:57:15 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:57:15.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:57:15 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:57:16.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:57:15 smithi115 conmon[90393]: 2022-05-14T02:57:15.795+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:57:16.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:57:16 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:57:16.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:57:16 smithi047 ceph-mon[77147]: pgmap v6636: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:57:16.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:57:16 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 12570 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:57:16.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:57:16 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:57:16.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:57:16 smithi115 ceph-mon[78593]: pgmap v6636: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:57:16.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:57:16 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 12570 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:57:17.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:57:16 smithi115 conmon[90393]: 2022-05-14T02:57:16.827+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:57:17.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:57:17 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:57:17.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:57:17 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:57:18.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:57:17 smithi115 conmon[90393]: 2022-05-14T02:57:17.856+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:57:18.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:57:18 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:57:18.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:57:18 smithi047 ceph-mon[77147]: pgmap v6637: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:57:18.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:57:18 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:57:18.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:57:18 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:57:18.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:57:18 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:57:18.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:57:18 smithi115 ceph-mon[78593]: pgmap v6637: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:57:18.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:57:18 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:57:18.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:57:18 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:57:19.208 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:57:18 smithi115 conmon[90393]: 2022-05-14T02:57:18.880+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:57:19.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:57:19 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:57:19.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:57:19 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:57:20.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:57:19 smithi115 conmon[90393]: 2022-05-14T02:57:19.838+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:57:20.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:57:20 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:57:20.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:57:20 smithi047 ceph-mon[77147]: pgmap v6638: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:57:20.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:57:20 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:57:20.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:57:20 smithi115 ceph-mon[78593]: pgmap v6638: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:57:21.214 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:57:20 smithi115 conmon[90393]: 2022-05-14T02:57:20.867+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:57:21.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:57:21 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:57:21.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:57:21 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:57:22.216 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:57:21 smithi115 conmon[90393]: 2022-05-14T02:57:21.888+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:57:22.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:57:22 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:57:22.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:57:22 smithi047 ceph-mon[77147]: pgmap v6639: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:57:22.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:57:22 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 12575 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:57:22.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:57:22 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:57:22.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:57:22 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:57:22.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:57:22 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:57:22.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:57:22 smithi115 ceph-mon[78593]: pgmap v6639: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:57:22.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:57:22 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 12575 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:57:22.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:57:22 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:57:22.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:57:22 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:57:23.221 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:57:22 smithi115 conmon[90393]: 2022-05-14T02:57:22.936+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:57:23.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:57:23 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:57:23.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:57:23 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:57:24.222 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:57:23 smithi115 conmon[90393]: 2022-05-14T02:57:23.962+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:57:24.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:57:24 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:57:24.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:57:24 smithi047 ceph-mon[77147]: pgmap v6640: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:57:24.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:57:24 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:57:24.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:57:24 smithi115 ceph-mon[78593]: pgmap v6640: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:57:25.227 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:57:24 smithi115 conmon[90393]: 2022-05-14T02:57:24.969+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:57:25.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:57:25 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:57:25.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:57:25 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:57:26.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:57:26 smithi115 conmon[90393]: 2022-05-14T02:57:26.013+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:57:26.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:57:26 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:57:26.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:57:26 smithi115 ceph-mon[78593]: pgmap v6641: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:57:26.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:57:26 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:57:26.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:57:26 smithi047 ceph-mon[77147]: pgmap v6641: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:57:27.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:57:27 smithi115 conmon[90393]: 2022-05-14T02:57:27.034+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:57:27.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:57:27 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:57:27.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:57:27 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:57:28.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:57:28 smithi115 conmon[90393]: 2022-05-14T02:57:28.077+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:57:28.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:57:28 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:57:28.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:57:28 smithi115 ceph-mon[78593]: pgmap v6642: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:57:28.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:57:28 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:57:28.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:57:28 smithi047 ceph-mon[77147]: pgmap v6642: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:57:29.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:57:29 smithi115 conmon[90393]: 2022-05-14T02:57:29.075+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:57:29.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:57:29 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:57:29.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:57:29 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:57:30.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:57:30 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:57:30.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:57:30 smithi047 ceph-mon[77147]: pgmap v6643: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:57:30.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:57:30 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:57:30.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:57:30 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:57:30.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:57:30 smithi115 conmon[90393]: 2022-05-14T02:57:30.107+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:57:30.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:57:30 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:57:30.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:57:30 smithi115 ceph-mon[78593]: pgmap v6643: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:57:30.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:57:30 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:57:30.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:57:30 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:57:31.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:57:31 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:57:31.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:57:31 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 12580 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:57:31.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:57:31 smithi115 conmon[90393]: 2022-05-14T02:57:31.157+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:57:31.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:57:31 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:57:31.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:57:31 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 12580 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:57:32.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:57:32 smithi047 ceph-mon[77147]: pgmap v6644: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:57:32.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:57:32 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:57:32.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:57:32 smithi115 conmon[90393]: 2022-05-14T02:57:32.160+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:57:32.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:57:32 smithi115 ceph-mon[78593]: pgmap v6644: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:57:32.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:57:32 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:57:33.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:57:33 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:57:33.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:57:33 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:57:33.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:57:33 smithi115 conmon[90393]: 2022-05-14T02:57:33.127+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:57:34.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:57:34 smithi115 conmon[90393]: 2022-05-14T02:57:34.097+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:57:34.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:57:34 smithi115 ceph-mon[78593]: pgmap v6645: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:57:34.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:57:34 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:57:34.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:57:34 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:57:34.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:57:34 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:57:34.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:57:34 smithi047 ceph-mon[77147]: pgmap v6645: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:57:34.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:57:34 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:57:34.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:57:34 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:57:34.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:57:34 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:57:35.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:57:35 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:57:35.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:57:35 smithi115 conmon[90393]: 2022-05-14T02:57:35.113+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:57:35.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:57:35 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:57:36.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:57:36 smithi115 conmon[90393]: 2022-05-14T02:57:36.083+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:57:36.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:57:36 smithi115 ceph-mon[78593]: pgmap v6646: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:57:36.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:57:36 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:57:36.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:57:36 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 12590 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:57:36.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:57:36 smithi047 ceph-mon[77147]: pgmap v6646: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:57:36.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:57:36 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:57:36.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:57:36 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 12590 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:57:37.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:57:37 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:57:37.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:57:37 smithi115 conmon[90393]: 2022-05-14T02:57:37.129+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:57:37.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:57:37 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:57:38.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:57:38 smithi047 ceph-mon[77147]: pgmap v6647: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:57:38.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:57:38 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:57:38.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:57:38 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:57:38.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:57:38 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:57:38.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:57:38 smithi115 conmon[90393]: 2022-05-14T02:57:38.175+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:57:38.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:57:38 smithi115 ceph-mon[78593]: pgmap v6647: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:57:38.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:57:38 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:57:38.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:57:38 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:57:38.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:57:38 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:57:39.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:57:39 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:57:39.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:57:39 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:57:39.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:57:39 smithi115 conmon[90393]: 2022-05-14T02:57:39.172+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:57:40.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:57:40 smithi047 ceph-mon[77147]: pgmap v6648: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:57:40.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:57:40 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:57:40.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:57:40 smithi115 conmon[90393]: 2022-05-14T02:57:40.153+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:57:40.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:57:40 smithi115 ceph-mon[78593]: pgmap v6648: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:57:40.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:57:40 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:57:41.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:57:41 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:57:41.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:57:41 smithi115 conmon[90393]: 2022-05-14T02:57:41.178+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:57:41.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:57:41 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:57:42.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:57:42 smithi047 ceph-mon[77147]: pgmap v6649: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:57:42.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:57:42 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:57:42.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:57:42 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 12595 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:57:42.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:57:42 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:57:42.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:57:42 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:57:42.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:57:42 smithi115 conmon[90393]: 2022-05-14T02:57:42.142+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:57:42.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:57:42 smithi115 ceph-mon[78593]: pgmap v6649: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:57:42.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:57:42 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:57:42.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:57:42 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 12595 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:57:42.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:57:42 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:57:42.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:57:42 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:57:43.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:57:43 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:57:43.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:57:43 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:57:43.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:57:43 smithi115 conmon[90393]: 2022-05-14T02:57:43.124+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:57:44.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:57:44 smithi047 ceph-mon[77147]: pgmap v6650: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:57:44.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:57:44 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:57:44.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:57:44 smithi115 conmon[90393]: 2022-05-14T02:57:44.140+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:57:44.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:57:44 smithi115 ceph-mon[78593]: pgmap v6650: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:57:44.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:57:44 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:57:45.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:57:45 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:57:45.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:57:45 smithi115 conmon[90393]: 2022-05-14T02:57:45.129+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:57:45.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:57:45 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:57:46.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:57:46 smithi115 conmon[90393]: 2022-05-14T02:57:46.152+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:57:46.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:57:46 smithi115 ceph-mon[78593]: pgmap v6651: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:57:46.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:57:46 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:57:46.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:57:46 smithi047 ceph-mon[77147]: pgmap v6651: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:57:46.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:57:46 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:57:47.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:57:47 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:57:47.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:57:47 smithi115 conmon[90393]: 2022-05-14T02:57:47.151+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:57:47.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:57:47 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:57:48.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:57:48 smithi115 conmon[90393]: 2022-05-14T02:57:48.113+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:57:48.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:57:48 smithi115 ceph-mon[78593]: pgmap v6652: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:57:48.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:57:48 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:57:48.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:57:48 smithi047 ceph-mon[77147]: pgmap v6652: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:57:48.810 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:57:48 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:57:49.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:57:49 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:57:49.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:57:49 smithi115 conmon[90393]: 2022-05-14T02:57:49.137+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:57:49.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:57:49 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:57:50.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:57:50 smithi115 conmon[90393]: 2022-05-14T02:57:50.147+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:57:50.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:57:50 smithi115 ceph-mon[78593]: pgmap v6653: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:57:50.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:57:50 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:57:50.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:57:50 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:57:50.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:57:50 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:57:50.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:57:50 smithi047 ceph-mon[77147]: pgmap v6653: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:57:50.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:57:50 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:57:50.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:57:50 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:57:50.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:57:50 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:57:51.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:57:51 smithi115 conmon[90393]: 2022-05-14T02:57:51.101+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:57:51.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:57:51 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:57:51.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:57:51 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 12600 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:57:51.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:57:51 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:57:51.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:57:51 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 12600 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:57:52.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:57:52 smithi115 conmon[90393]: 2022-05-14T02:57:52.142+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:57:52.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:57:52 smithi115 ceph-mon[78593]: pgmap v6654: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:57:52.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:57:52 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:57:52.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:57:52 smithi047 ceph-mon[77147]: pgmap v6654: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:57:52.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:57:52 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:57:53.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:57:53 smithi115 conmon[90393]: 2022-05-14T02:57:53.162+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:57:53.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:57:53 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:57:53.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:57:53 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:57:54.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:57:54 smithi115 conmon[90393]: 2022-05-14T02:57:54.124+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:57:54.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:57:54 smithi115 ceph-mon[78593]: pgmap v6655: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:57:54.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:57:54 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:57:54.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:57:54 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:57:54.612 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:57:54 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:57:54.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:57:54 smithi047 ceph-mon[77147]: pgmap v6655: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:57:54.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:57:54 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:57:54.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:57:54 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:57:54.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:57:54 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:57:55.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:57:55 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:57:55.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:57:55 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T02:57:55.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:57:55 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T02:57:55.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:57:55 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T02:57:55.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:57:55 smithi047 conmon[78521]: 2022-05-14T02:57:55.119+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:57:55.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:57:55 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T02:57:55.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:57:55 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:57:55.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:57:55 smithi047 conmon[78521]: yield 2022-05-14T02:57:55.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:57:55 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:57:55.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:57:55 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:57:55.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:57:55 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:57:55.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:57:55 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T02:57:55.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:57:55 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:57:55.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:57:55 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T02:57:55.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:57:55 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:57:55.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:57:55 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T02:57:55.547 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:57:55 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:57:55.547 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:57:55 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T02:57:55.547 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:57:55 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:57:55.548 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:57:55 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T02:57:55.548 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:57:55 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:57:55.548 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:57:55 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:57:55.548 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:57:55 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:57:55.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:57:55 smithi115 conmon[90393]: 2022-05-14T02:57:55.157+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:57:55.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:57:55 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:57:55.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:57:55 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T02:57:55.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:57:55 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T02:57:55.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:57:55 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T02:57:56.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:57:56 smithi115 conmon[90393]: 2022-05-14T02:57:56.187+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:57:56.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:57:56 smithi115 ceph-mon[78593]: pgmap v6656: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:57:56.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:57:56 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:57:56.609 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T02:57:56.609 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:57:56.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T02:57:56.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:57:56.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:57:56.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:57:56.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T02:57:56.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:57:56.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T02:57:56.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:57:56.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T02:57:56.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:57:56.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T02:57:56.613 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:57:56.613 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T02:57:56.613 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:57:56.613 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:57:56.613 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:57:56.614 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:57:56 smithi115 ceph-mon[78593]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T02:57:56.614 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:57:56 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:57:56.614 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:57:56 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 12610 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:57:56.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:57:56 smithi047 ceph-mon[77147]: pgmap v6656: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:57:56.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:57:56 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:57:56.792 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T02:57:56.793 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:57:56.793 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T02:57:56.793 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:57:56.793 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:57:56.793 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:57:56.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T02:57:56.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:57:56.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T02:57:56.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:57:56.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T02:57:56.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:57:56.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T02:57:56.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:57:56.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T02:57:56.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:57:56.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:57:56.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:57:56.796 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:57:56 smithi047 ceph-mon[77147]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T02:57:56.797 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:57:56 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:57:56.797 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:57:56 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 12610 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:57:57.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:57:57 smithi115 conmon[90393]: 2022-05-14T02:57:57.195+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:57:57.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:57:57 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:57:57.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:57:57 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:57:58.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:57:58 smithi047 ceph-mon[77147]: pgmap v6657: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:57:58.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:57:58 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:57:58.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:57:58 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:57:58.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:57:58 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:57:58.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:57:58 smithi047 conmon[78521]: 2022-05-14T02:57:58.191+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:57:58.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:57:58 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T02:57:58.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:57:58 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:57:58.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:57:58 smithi047 conmon[78521]: yield 2022-05-14T02:57:58.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:57:58 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:57:58.547 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:57:58 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:57:58.547 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:57:58 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:57:58.547 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:57:58 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T02:57:58.547 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:57:58 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:57:58.548 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:57:58 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T02:57:58.548 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:57:58 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:57:58.549 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:57:58 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T02:57:58.549 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:57:58 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:57:58.549 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:57:58 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T02:57:58.550 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:57:58 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:57:58.550 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:57:58 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T02:57:58.551 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:57:58 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:57:58.551 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:57:58 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:57:58.552 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:57:58 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:57:58.552 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:57:58 smithi047 conmon[78521]: 2022-05-14T02:57:58.192+0000 7f5345f84700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T02:57:58.552 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:57:58 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T02:57:58.553 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:57:58 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:57:58.553 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:57:58 smithi047 conmon[78521]: yield 2022-05-14T02:57:58.554 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:57:58 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:57:58.554 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:57:58 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:57:58.554 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:57:58 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:57:58.555 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:57:58 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T02:57:58.555 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:57:58 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:57:58.556 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:57:58 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T02:57:58.556 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:57:58 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:57:58.557 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:57:58 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T02:57:58.557 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:57:58 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:57:58.557 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:57:58 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T02:57:58.558 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:57:58 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:57:58.558 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:57:58 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T02:57:58.559 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:57:58 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:57:58.559 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:57:58 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:57:58.560 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:57:58 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:57:58.560 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:57:58 smithi047 conmon[78521]: 2022-05-14T02:57:58.561 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:57:58 smithi047 conmon[78521]: During handling of the above exception, another exception occurred: 2022-05-14T02:57:58.561 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:57:58 smithi047 conmon[78521]: 2022-05-14T02:57:58.561 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:57:58 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T02:57:58.562 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:57:58 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T02:57:58.562 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:57:58 smithi047 conmon[78521]: return f(*arg) 2022-05-14T02:57:58.563 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:57:58 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T02:57:58.563 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:57:58 smithi047 conmon[78521]: self._write_client_files(client_files, host) 2022-05-14T02:57:58.564 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:57:58 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T02:57:58.564 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:57:58 smithi047 conmon[78521]: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T02:57:58.564 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:57:58 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T02:57:58.565 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:57:58 smithi047 conmon[78521]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T02:57:58.565 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:57:58 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T02:57:58.566 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:57:58 smithi047 conmon[78521]: return self.event_loop.get_result(coro) 2022-05-14T02:57:58.566 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:57:58 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T02:57:58.567 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:57:58 smithi047 conmon[78521]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T02:57:58.567 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:57:58 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T02:57:58.567 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:57:58 smithi047 conmon[78521]: return self.__get_result() 2022-05-14T02:57:58.568 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:57:58 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T02:57:58.568 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:57:58 smithi047 conmon[78521]: raise self._exception 2022-05-14T02:57:58.569 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:57:58 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T02:57:58.569 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:57:58 smithi047 conmon[78521]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T02:57:58.570 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:57:58 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T02:57:58.570 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:57:58 smithi047 conmon[78521]: conn = await self._remote_connection(host, addr) 2022-05-14T02:57:58.571 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:57:58 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T02:57:58.571 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:57:58 smithi047 conmon[78521]: raise 2022-05-14T02:57:58.571 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:57:58 smithi047 conmon[78521]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T02:57:58.572 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:57:58 smithi047 conmon[78521]: self.gen.throw(type, value, traceback) 2022-05-14T02:57:58.572 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:57:58 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T02:57:58.574 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:57:58 smithi047 conmon[78521]: raise OrchestratorError(msg) 2022-05-14T02:57:58.574 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:57:58 smithi047 conmon[78521]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:57:58.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:57:58 smithi115 conmon[90393]: 2022-05-14T02:57:58.209+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:57:58.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:57:58 smithi115 ceph-mon[78593]: pgmap v6657: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:57:58.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:57:58 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:57:58.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:57:58 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:57:58.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:57:58 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:57:59.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:57:59 smithi115 conmon[90393]: 2022-05-14T02:57:59.236+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:57:59.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:57:59 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:57:59.609 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T02:57:59.609 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:57:59.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T02:57:59.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:57:59.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:57:59.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:57:59.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T02:57:59.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:57:59.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T02:57:59.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:57:59.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T02:57:59.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:57:59.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T02:57:59.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:57:59.613 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T02:57:59.613 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:57:59.613 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:57:59.613 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:57:59.614 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:57:59 smithi115 ceph-mon[78593]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T02:57:59.614 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T02:57:59.614 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:57:59.614 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T02:57:59.614 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:57:59.615 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:57:59.615 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:57:59.615 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T02:57:59.615 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:57:59.616 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T02:57:59.616 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:57:59.616 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T02:57:59.616 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:57:59.616 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T02:57:59.617 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:57:59.617 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T02:57:59.617 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:57:59.617 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:57:59.618 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:57:59.618 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T02:57:59.618 INFO:journalctl@ceph.mon.c.smithi115.stdout: During handling of the above exception, another exception occurred: 2022-05-14T02:57:59.618 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T02:57:59.618 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T02:57:59.619 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T02:57:59.619 INFO:journalctl@ceph.mon.c.smithi115.stdout: return f(*arg) 2022-05-14T02:57:59.619 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T02:57:59.619 INFO:journalctl@ceph.mon.c.smithi115.stdout: self._write_client_files(client_files, host) 2022-05-14T02:57:59.620 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T02:57:59.620 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T02:57:59.620 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T02:57:59.621 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T02:57:59.621 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T02:57:59.621 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.event_loop.get_result(coro) 2022-05-14T02:57:59.621 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T02:57:59.622 INFO:journalctl@ceph.mon.c.smithi115.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T02:57:59.622 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T02:57:59.622 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.__get_result() 2022-05-14T02:57:59.622 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T02:57:59.622 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise self._exception 2022-05-14T02:57:59.623 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T02:57:59.623 INFO:journalctl@ceph.mon.c.smithi115.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T02:57:59.623 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T02:57:59.623 INFO:journalctl@ceph.mon.c.smithi115.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T02:57:59.624 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T02:57:59.624 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise 2022-05-14T02:57:59.624 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T02:57:59.624 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.gen.throw(type, value, traceback) 2022-05-14T02:57:59.624 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T02:57:59.625 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OrchestratorError(msg) 2022-05-14T02:57:59.625 INFO:journalctl@ceph.mon.c.smithi115.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:57:59.625 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:57:59 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:57:59.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:57:59 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:57:59.792 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T02:57:59.792 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:57:59.793 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T02:57:59.793 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:57:59.793 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:57:59.793 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:57:59.793 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T02:57:59.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:57:59.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T02:57:59.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:57:59.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T02:57:59.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:57:59.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T02:57:59.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:57:59.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T02:57:59.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:57:59.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:57:59.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:57:59.796 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:57:59 smithi047 ceph-mon[77147]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T02:57:59.797 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T02:57:59.797 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:57:59.797 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T02:57:59.797 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:57:59.797 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:57:59.798 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:57:59.798 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T02:57:59.798 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:57:59.798 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T02:57:59.798 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:57:59.799 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T02:57:59.799 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:57:59.799 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T02:57:59.800 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:57:59.800 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T02:57:59.800 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:57:59.800 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:57:59.800 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:57:59.801 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T02:57:59.801 INFO:journalctl@ceph.mon.b.smithi047.stdout: During handling of the above exception, another exception occurred: 2022-05-14T02:57:59.801 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T02:57:59.801 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T02:57:59.802 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T02:57:59.802 INFO:journalctl@ceph.mon.b.smithi047.stdout: return f(*arg) 2022-05-14T02:57:59.802 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T02:57:59.802 INFO:journalctl@ceph.mon.b.smithi047.stdout: self._write_client_files(client_files, host) 2022-05-14T02:57:59.802 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T02:57:59.803 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T02:57:59.803 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T02:57:59.803 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T02:57:59.803 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T02:57:59.803 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.event_loop.get_result(coro) 2022-05-14T02:57:59.804 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T02:57:59.804 INFO:journalctl@ceph.mon.b.smithi047.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T02:57:59.804 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T02:57:59.804 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.__get_result() 2022-05-14T02:57:59.805 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T02:57:59.805 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise self._exception 2022-05-14T02:57:59.805 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T02:57:59.806 INFO:journalctl@ceph.mon.b.smithi047.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T02:57:59.806 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T02:57:59.806 INFO:journalctl@ceph.mon.b.smithi047.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T02:57:59.806 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T02:57:59.807 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise 2022-05-14T02:57:59.807 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T02:57:59.807 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.gen.throw(type, value, traceback) 2022-05-14T02:57:59.807 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T02:57:59.808 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OrchestratorError(msg) 2022-05-14T02:57:59.808 INFO:journalctl@ceph.mon.b.smithi047.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:57:59.808 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:57:59 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:58:00.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:58:00 smithi115 conmon[90393]: 2022-05-14T02:58:00.249+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:58:00.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:58:00 smithi115 ceph-mon[78593]: pgmap v6658: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:58:00.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:58:00 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:58:00.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:58:00 smithi047 ceph-mon[77147]: pgmap v6658: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:58:00.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:58:00 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:58:01.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:58:01 smithi115 conmon[90393]: 2022-05-14T02:58:01.296+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:58:01.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:58:01 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:58:01.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:58:01 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:58:02.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:58:02 smithi115 conmon[90393]: 2022-05-14T02:58:02.317+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:58:02.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:58:02 smithi115 ceph-mon[78593]: pgmap v6659: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:58:02.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:58:02 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:58:02.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:58:02 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 12615 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:58:02.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:58:02 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:58:02.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:58:02 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:58:02.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:58:02 smithi047 ceph-mon[77147]: pgmap v6659: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:58:02.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:58:02 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:58:02.795 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:58:02 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 12615 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:58:02.795 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:58:02 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:58:02.796 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:58:02 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:58:03.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:58:03 smithi115 conmon[90393]: 2022-05-14T02:58:03.328+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:58:03.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:58:03 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:58:03.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:58:03 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:58:04.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:58:04 smithi115 conmon[90393]: 2022-05-14T02:58:04.369+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:58:04.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:58:04 smithi115 ceph-mon[78593]: pgmap v6660: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:58:04.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:58:04 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:58:04.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:58:04 smithi047 ceph-mon[77147]: pgmap v6660: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:58:04.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:58:04 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:58:05.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:58:05 smithi115 conmon[90393]: 2022-05-14T02:58:05.352+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:58:05.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:58:05 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:58:05.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:58:05 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:58:06.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:58:06 smithi047 ceph-mon[77147]: pgmap v6661: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:58:06.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:58:06 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:58:06.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:58:06 smithi115 conmon[90393]: 2022-05-14T02:58:06.373+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:58:06.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:58:06 smithi115 ceph-mon[78593]: pgmap v6661: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:58:06.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:58:06 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:58:07.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:58:07 smithi115 conmon[90393]: 2022-05-14T02:58:07.351+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:58:07.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:58:07 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:58:07.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:58:07 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:58:08.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:58:08 smithi115 conmon[90393]: 2022-05-14T02:58:08.354+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:58:08.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:58:08 smithi115 ceph-mon[78593]: pgmap v6662: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:58:08.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:58:08 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:58:08.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:58:08 smithi047 ceph-mon[77147]: pgmap v6662: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:58:08.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:58:08 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:58:09.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:58:09 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:58:09.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:58:09 smithi115 conmon[90393]: 2022-05-14T02:58:09.367+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:58:09.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:58:09 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:58:10.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:58:10 smithi115 conmon[90393]: 2022-05-14T02:58:10.346+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:58:10.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:58:10 smithi115 ceph-mon[78593]: pgmap v6663: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:58:10.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:58:10 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:58:10.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:58:10 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 12625 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:58:10.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:58:10 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:58:10.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:58:10 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:58:10.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:58:10 smithi047 ceph-mon[77147]: pgmap v6663: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:58:10.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:58:10 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:58:10.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:58:10 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 12625 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:58:10.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:58:10 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:58:10.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:58:10 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:58:11.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:58:11 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:58:11.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:58:11 smithi115 conmon[90393]: 2022-05-14T02:58:11.369+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:58:11.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:58:11 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:58:12.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:58:12 smithi047 ceph-mon[77147]: pgmap v6664: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:58:12.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:58:12 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:58:12.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:58:12 smithi115 conmon[90393]: 2022-05-14T02:58:12.371+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:58:12.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:58:12 smithi115 ceph-mon[78593]: pgmap v6664: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:58:12.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:58:12 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:58:13.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:58:13 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:58:13.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:58:13 smithi115 conmon[90393]: 2022-05-14T02:58:13.371+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:58:13.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:58:13 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:58:14.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:58:14 smithi047 ceph-mon[77147]: pgmap v6665: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:58:14.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:58:14 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:58:14.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:58:14 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:58:14.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:58:14 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:58:14.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:58:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T02:58:14.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:58:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T02:58:14.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:58:14 smithi115 conmon[90393]: 2022-05-14T02:58:14.421+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:58:14.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:58:14 smithi115 ceph-mon[78593]: pgmap v6665: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:58:14.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:58:14 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:58:14.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:58:14 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:58:14.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:58:14 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:58:14.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:58:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T02:58:14.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:58:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T02:58:15.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:58:15 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:58:15.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:58:15 smithi115 conmon[90393]: 2022-05-14T02:58:15.421+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:58:15.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:58:15 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:58:16.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:58:16 smithi047 ceph-mon[77147]: pgmap v6666: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:58:16.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:58:16 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 12630 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:58:16.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:58:16 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:58:16.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:58:16 smithi115 conmon[90393]: 2022-05-14T02:58:16.444+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:58:16.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:58:16 smithi115 ceph-mon[78593]: pgmap v6666: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:58:16.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:58:16 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 12630 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:58:16.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:58:16 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:58:17.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:58:17 smithi115 conmon[90393]: 2022-05-14T02:58:17.475+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:58:18.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:58:18 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:58:18.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:58:18 smithi047 ceph-mon[77147]: pgmap v6667: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:58:18.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:58:18 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:58:18.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:58:18 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:58:18.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:58:18 smithi115 conmon[90393]: 2022-05-14T02:58:18.505+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:58:18.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:58:18 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:58:18.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:58:18 smithi115 ceph-mon[78593]: pgmap v6667: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:58:18.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:58:18 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:58:18.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:58:18 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:58:19.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:58:19 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:58:19.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:58:19 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:58:19.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:58:19 smithi115 conmon[90393]: 2022-05-14T02:58:19.496+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:58:20.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:58:20 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:58:20.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:58:20 smithi047 ceph-mon[77147]: pgmap v6668: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-05-14T02:58:20.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:58:20 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:58:20.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:58:20 smithi115 ceph-mon[78593]: pgmap v6668: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-05-14T02:58:20.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:58:20 smithi115 conmon[90393]: 2022-05-14T02:58:20.519+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:58:21.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:58:21 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:58:21.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:58:21 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 12635 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:58:21.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:58:21 smithi115 conmon[90393]: 2022-05-14T02:58:21.473+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:58:21.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:58:21 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:58:21.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:58:21 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 12635 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:58:22.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:58:22 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:58:22.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:58:22 smithi047 ceph-mon[77147]: pgmap v6669: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:58:22.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:58:22 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:58:22.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:58:22 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:58:22.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:58:22 smithi115 conmon[90393]: 2022-05-14T02:58:22.502+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:58:22.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:58:22 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:58:22.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:58:22 smithi115 ceph-mon[78593]: pgmap v6669: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:58:22.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:58:22 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:58:22.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:58:22 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:58:23.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:58:23 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:58:23.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:58:23 smithi115 conmon[90393]: 2022-05-14T02:58:23.543+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:58:23.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:58:23 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:58:24.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:58:24 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:58:24.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:58:24 smithi047 ceph-mon[77147]: pgmap v6670: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:58:24.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:58:24 smithi115 conmon[90393]: 2022-05-14T02:58:24.544+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:58:24.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:58:24 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:58:24.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:58:24 smithi115 ceph-mon[78593]: pgmap v6670: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:58:25.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:58:25 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:58:25.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:58:25 smithi115 conmon[90393]: 2022-05-14T02:58:25.546+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:58:25.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:58:25 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:58:26.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:58:26 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:58:26.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:58:26 smithi047 ceph-mon[77147]: pgmap v6671: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:58:26.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:58:26 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 12640 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:58:26.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:58:26 smithi115 conmon[90393]: 2022-05-14T02:58:26.516+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:58:26.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:58:26 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:58:26.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:58:26 smithi115 ceph-mon[78593]: pgmap v6671: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:58:26.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:58:26 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 12640 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:58:27.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:58:27 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:58:27.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:58:27 smithi115 conmon[90393]: 2022-05-14T02:58:27.482+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:58:27.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:58:27 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:58:28.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:58:28 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:58:28.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:58:28 smithi047 ceph-mon[77147]: pgmap v6672: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:58:28.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:58:28 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:58:28.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:58:28 smithi115 conmon[90393]: 2022-05-14T02:58:28.436+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:58:28.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:58:28 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:58:28.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:58:28 smithi115 ceph-mon[78593]: pgmap v6672: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:58:28.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:58:28 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:58:29.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:58:29 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:58:29.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:58:29 smithi115 conmon[90393]: 2022-05-14T02:58:29.455+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:58:29.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:58:29 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:58:30.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:58:30 smithi047 ceph-mon[77147]: pgmap v6673: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-05-14T02:58:30.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:58:30 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:58:30.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:58:30 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:58:30.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:58:30 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:58:30.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:58:30 smithi115 conmon[90393]: 2022-05-14T02:58:30.464+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:58:30.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:58:30 smithi115 ceph-mon[78593]: pgmap v6673: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-05-14T02:58:30.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:58:30 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:58:30.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:58:30 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:58:30.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:58:30 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:58:31.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:58:31 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:58:31.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:58:31 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 12645 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:58:31.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:58:31 smithi115 conmon[90393]: 2022-05-14T02:58:31.466+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:58:31.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:58:31 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:58:31.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:58:31 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 12645 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:58:32.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:58:32 smithi047 ceph-mon[77147]: pgmap v6674: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:58:32.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:58:32 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:58:32.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:58:32 smithi115 conmon[90393]: 2022-05-14T02:58:32.459+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:58:32.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:58:32 smithi115 ceph-mon[78593]: pgmap v6674: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:58:32.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:58:32 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:58:33.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:58:33 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:58:33.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:58:33 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:58:33.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:58:33 smithi115 conmon[90393]: 2022-05-14T02:58:33.413+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:58:33.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:58:33 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:58:33.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:58:33 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:58:34.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:58:34 smithi047 ceph-mon[77147]: pgmap v6675: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:58:34.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:58:34 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:58:34.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:58:34 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:58:34.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:58:34 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:58:34.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:58:34 smithi115 conmon[90393]: 2022-05-14T02:58:34.378+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:58:34.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:58:34 smithi115 ceph-mon[78593]: pgmap v6675: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:58:34.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:58:34 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:58:34.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:58:34 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:58:34.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:58:34 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:58:35.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:58:35 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:58:35.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:58:35 smithi115 conmon[90393]: 2022-05-14T02:58:35.407+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:58:35.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:58:35 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:58:36.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:58:36 smithi047 ceph-mon[77147]: pgmap v6676: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:58:36.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:58:36 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 12650 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:58:36.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:58:36 smithi115 conmon[90393]: 2022-05-14T02:58:36.457+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:58:36.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:58:36 smithi115 ceph-mon[78593]: pgmap v6676: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:58:36.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:58:36 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 12650 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:58:37.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:58:37 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:58:37.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:58:37 smithi115 conmon[90393]: 2022-05-14T02:58:37.439+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:58:37.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:58:37 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:58:38.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:58:38 smithi047 ceph-mon[77147]: pgmap v6677: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:58:38.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:58:38 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:58:38.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:58:38 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:58:38.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:58:38 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:58:38.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:58:38 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:58:38.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:58:38 smithi115 conmon[90393]: 2022-05-14T02:58:38.407+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:58:38.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:58:38 smithi115 ceph-mon[78593]: pgmap v6677: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:58:38.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:58:38 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:58:38.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:58:38 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:58:38.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:58:38 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:58:38.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:58:38 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:58:39.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:58:39 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:58:39.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:58:39 smithi115 conmon[90393]: 2022-05-14T02:58:39.387+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:58:39.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:58:39 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:58:40.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:58:40 smithi047 ceph-mon[77147]: pgmap v6678: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-05-14T02:58:40.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:58:40 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:58:40.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:58:40 smithi115 conmon[90393]: 2022-05-14T02:58:40.434+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:58:40.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:58:40 smithi115 ceph-mon[78593]: pgmap v6678: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-05-14T02:58:40.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:58:40 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:58:41.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:58:41 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 12655 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:58:41.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:58:41 smithi115 conmon[90393]: 2022-05-14T02:58:41.474+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:58:41.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:58:41 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 12655 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:58:42.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:58:42 smithi047 ceph-mon[77147]: pgmap v6679: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:58:42.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:58:42 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:58:42.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:58:42 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:58:42.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:58:42 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:58:42.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:58:42 smithi115 conmon[90393]: 2022-05-14T02:58:42.478+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:58:42.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:58:42 smithi115 ceph-mon[78593]: pgmap v6679: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:58:42.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:58:42 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:58:42.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:58:42 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:58:42.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:58:42 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:58:43.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:58:43 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:58:43.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:58:43 smithi115 conmon[90393]: 2022-05-14T02:58:43.514+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:58:43.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:58:43 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:58:44.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:58:44 smithi047 ceph-mon[77147]: pgmap v6680: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:58:44.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:58:44 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:58:44.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:58:44 smithi115 conmon[90393]: 2022-05-14T02:58:44.534+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:58:44.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:58:44 smithi115 ceph-mon[78593]: pgmap v6680: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:58:44.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:58:44 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:58:45.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:58:45 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:58:45.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:58:45 smithi115 conmon[90393]: 2022-05-14T02:58:45.506+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:58:45.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:58:45 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:58:46.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:58:46 smithi047 ceph-mon[77147]: pgmap v6681: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:58:46.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:58:46 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:58:46.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:58:46 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 12660 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:58:46.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:58:46 smithi115 conmon[90393]: 2022-05-14T02:58:46.470+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:58:46.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:58:46 smithi115 ceph-mon[78593]: pgmap v6681: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:58:46.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:58:46 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:58:46.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:58:46 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 12660 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:58:47.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:58:47 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:58:47.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:58:47 smithi115 conmon[90393]: 2022-05-14T02:58:47.479+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:58:47.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:58:47 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:58:48.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:58:48 smithi047 ceph-mon[77147]: pgmap v6682: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:58:48.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:58:48 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:58:48.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:58:48 smithi115 conmon[90393]: 2022-05-14T02:58:48.479+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:58:48.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:58:48 smithi115 ceph-mon[78593]: pgmap v6682: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:58:48.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:58:48 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:58:49.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:58:49 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:58:49.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:58:49 smithi115 conmon[90393]: 2022-05-14T02:58:49.467+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:58:49.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:58:49 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:58:50.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:58:50 smithi047 ceph-mon[77147]: pgmap v6683: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-05-14T02:58:50.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:58:50 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:58:50.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:58:50 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:58:50.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:58:50 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:58:50.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:58:50 smithi115 conmon[90393]: 2022-05-14T02:58:50.442+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:58:50.911 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:58:50 smithi115 ceph-mon[78593]: pgmap v6683: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-05-14T02:58:50.911 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:58:50 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:58:50.911 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:58:50 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:58:50.912 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:58:50 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:58:51.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:58:51 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:58:51.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:58:51 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 12665 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:58:51.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:58:51 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:58:51.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:58:51 smithi115 conmon[90393]: 2022-05-14T02:58:51.409+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:58:51.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:58:51 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:58:51.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:58:51 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 12665 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:58:51.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:58:51 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:58:52.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:58:52 smithi047 ceph-mon[77147]: pgmap v6684: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:58:52.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:58:52 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:58:52.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:58:52 smithi115 conmon[90393]: 2022-05-14T02:58:52.422+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:58:52.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:58:52 smithi115 ceph-mon[78593]: pgmap v6684: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:58:52.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:58:52 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:58:53.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:58:53 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:58:53.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:58:53 smithi115 conmon[90393]: 2022-05-14T02:58:53.374+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:58:53.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:58:53 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:58:54.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:58:54 smithi047 ceph-mon[77147]: pgmap v6685: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:58:54.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:58:54 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:58:54.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:58:54 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:58:54.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:58:54 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:58:54.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:58:54 smithi115 conmon[90393]: 2022-05-14T02:58:54.407+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:58:54.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:58:54 smithi115 ceph-mon[78593]: pgmap v6685: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:58:54.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:58:54 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:58:54.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:58:54 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:58:54.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:58:54 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:58:55.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:58:55 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:58:55.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:58:55 smithi115 conmon[90393]: 2022-05-14T02:58:55.361+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:58:55.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:58:55 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:58:56.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:58:56 smithi047 ceph-mon[77147]: pgmap v6686: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:58:56.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:58:56 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 12670 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:58:56.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:58:56 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:58:56.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:58:56 smithi115 conmon[90393]: 2022-05-14T02:58:56.393+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:58:56.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:58:56 smithi115 ceph-mon[78593]: pgmap v6686: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:58:56.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:58:56 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 12670 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:58:56.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:58:56 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:58:57.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:58:57 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:58:57.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:58:57 smithi115 conmon[90393]: 2022-05-14T02:58:57.422+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:58:57.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:58:57 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:58:58.536 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:58:58 smithi115 conmon[90393]: 2022-05-14T02:58:58.405+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:58:58.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:58:58 smithi047 ceph-mon[77147]: pgmap v6687: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:58:58.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:58:58 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:58:58.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:58:58 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:58:58.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:58:58 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T02:58:58.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:58:58 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T02:58:58.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:58:58 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T02:58:58.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:58:58 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:58:58.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:58:58 smithi115 ceph-mon[78593]: pgmap v6687: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:58:58.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:58:58 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:58:58.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:58:58 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:58:58.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:58:58 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T02:58:58.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:58:58 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T02:58:58.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:58:58 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T02:58:58.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:58:58 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:58:59.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:58:59 smithi115 conmon[90393]: 2022-05-14T02:58:59.399+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:58:59.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:58:59 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:59:00.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:58:59 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:59:00.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:59:00 smithi115 conmon[90393]: 2022-05-14T02:59:00.381+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:59:00.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:59:00 smithi115 ceph-mon[78593]: pgmap v6688: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-05-14T02:59:00.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:59:00 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:59:01.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:59:00 smithi047 ceph-mon[77147]: pgmap v6688: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-05-14T02:59:01.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:59:00 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:59:01.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:59:01 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 12675 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:59:01.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:59:01 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:59:01.792 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:59:01 smithi047 conmon[78521]: 2022-05-14T02:59:01.296+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:59:01.793 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:59:01 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T02:59:01.793 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:59:01 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:59:01.793 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:59:01 smithi047 conmon[78521]: yield 2022-05-14T02:59:01.793 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:59:01 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:59:01.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:59:01 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:59:01.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:59:01 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:59:01.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:59:01 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T02:59:01.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:59:01 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:59:01.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:59:01 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T02:59:01.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:59:01 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:59:01.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:59:01 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T02:59:01.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:59:01 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:59:01.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:59:01 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T02:59:01.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:59:01 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:59:01.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:59:01 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T02:59:01.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:59:01 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:59:01.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:59:01 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:59:01.797 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:59:01 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:59:01.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:59:01 smithi115 conmon[90393]: 2022-05-14T02:59:01.371+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:59:01.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:59:01 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 12675 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:59:01.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:59:01 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:59:02.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:59:02 smithi115 conmon[90393]: 2022-05-14T02:59:02.366+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:59:02.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:59:02 smithi115 ceph-mon[78593]: pgmap v6689: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:59:02.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:59:02 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:59:02.860 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T02:59:02.860 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:59:02.860 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T02:59:02.860 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:59:02.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:59:02.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:59:02.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T02:59:02.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:59:02.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T02:59:02.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:59:02.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T02:59:02.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:59:02.863 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T02:59:02.863 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:59:02.863 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T02:59:02.863 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:59:02.863 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:59:02.864 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:59:02.864 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:59:02 smithi115 ceph-mon[78593]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T02:59:02.864 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:59:02 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:59:02.864 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:59:02 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:59:02.865 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:59:02 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:59:03.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:59:02 smithi047 ceph-mon[77147]: pgmap v6689: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:59:03.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:59:02 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:59:03.042 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T02:59:03.042 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:59:03.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T02:59:03.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:59:03.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:59:03.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:59:03.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T02:59:03.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:59:03.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T02:59:03.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:59:03.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T02:59:03.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:59:03.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T02:59:03.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:59:03.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T02:59:03.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:59:03.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:59:03.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:59:03.047 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:59:02 smithi047 ceph-mon[77147]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T02:59:03.047 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:59:02 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:59:03.047 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:59:02 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:59:03.047 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:59:02 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:59:03.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:59:03 smithi115 conmon[90393]: 2022-05-14T02:59:03.347+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:59:03.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:59:03 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:59:04.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:59:03 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:59:04.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:59:04 smithi047 ceph-mon[77147]: pgmap v6690: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:59:04.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:59:04 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:59:04.792 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:59:04 smithi047 conmon[78521]: 2022-05-14T02:59:04.367+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:59:04.793 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:59:04 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T02:59:04.793 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:59:04 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:59:04.793 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:59:04 smithi047 conmon[78521]: yield 2022-05-14T02:59:04.793 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:59:04 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:59:04.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:59:04 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:59:04.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:59:04 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:59:04.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:59:04 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T02:59:04.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:59:04 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:59:04.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:59:04 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T02:59:04.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:59:04 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:59:04.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:59:04 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T02:59:04.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:59:04 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:59:04.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:59:04 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T02:59:04.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:59:04 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:59:04.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:59:04 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T02:59:04.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:59:04 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:59:04.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:59:04 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:59:04.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:59:04 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:59:04.797 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:59:04 smithi047 conmon[78521]: 2022-05-14T02:59:04.368+0000 7f5340779700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T02:59:04.797 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:59:04 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T02:59:04.797 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:59:04 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:59:04.797 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:59:04 smithi047 conmon[78521]: yield 2022-05-14T02:59:04.797 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:59:04 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:59:04.798 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:59:04 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:59:04.798 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:59:04 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:59:04.798 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:59:04 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T02:59:04.798 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:59:04 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:59:04.798 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:59:04 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T02:59:04.799 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:59:04 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:59:04.799 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:59:04 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T02:59:04.799 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:59:04 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:59:04.799 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:59:04 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T02:59:04.800 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:59:04 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:59:04.800 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:59:04 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T02:59:04.800 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:59:04 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:59:04.800 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:59:04 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:59:04.800 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:59:04 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:59:04.801 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:59:04 smithi047 conmon[78521]: 2022-05-14T02:59:04.801 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:59:04 smithi047 conmon[78521]: During handling of the above exception, another exception occurred: 2022-05-14T02:59:04.801 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:59:04 smithi047 conmon[78521]: 2022-05-14T02:59:04.801 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:59:04 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T02:59:04.801 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:59:04 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T02:59:04.802 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:59:04 smithi047 conmon[78521]: return f(*arg) 2022-05-14T02:59:04.802 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:59:04 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T02:59:04.802 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:59:04 smithi047 conmon[78521]: self._write_client_files(client_files, host) 2022-05-14T02:59:04.802 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:59:04 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T02:59:04.803 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:59:04 smithi047 conmon[78521]: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T02:59:04.803 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:59:04 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T02:59:04.803 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:59:04 smithi047 conmon[78521]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T02:59:04.803 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:59:04 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T02:59:04.803 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:59:04 smithi047 conmon[78521]: return self.event_loop.get_result(coro) 2022-05-14T02:59:04.804 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:59:04 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T02:59:04.804 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:59:04 smithi047 conmon[78521]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T02:59:04.804 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:59:04 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T02:59:04.804 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:59:04 smithi047 conmon[78521]: return self.__get_result() 2022-05-14T02:59:04.804 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:59:04 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T02:59:04.805 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:59:04 smithi047 conmon[78521]: raise self._exception 2022-05-14T02:59:04.805 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:59:04 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T02:59:04.805 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:59:04 smithi047 conmon[78521]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T02:59:04.805 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:59:04 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T02:59:04.806 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:59:04 smithi047 conmon[78521]: conn = await self._remote_connection(host, addr) 2022-05-14T02:59:04.806 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:59:04 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T02:59:04.806 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:59:04 smithi047 conmon[78521]: raise 2022-05-14T02:59:04.806 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:59:04 smithi047 conmon[78521]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T02:59:04.806 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:59:04 smithi047 conmon[78521]: self.gen.throw(type, value, traceback) 2022-05-14T02:59:04.807 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:59:04 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T02:59:04.807 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:59:04 smithi047 conmon[78521]: raise OrchestratorError(msg) 2022-05-14T02:59:04.807 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 02:59:04 smithi047 conmon[78521]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:59:04.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:59:04 smithi115 ceph-mon[78593]: pgmap v6690: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:59:04.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:59:04 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:59:04.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:59:04 smithi115 conmon[90393]: 2022-05-14T02:59:04.386+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:59:05.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:59:05 smithi115 conmon[90393]: 2022-05-14T02:59:05.350+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:59:05.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:59:05 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:59:05.609 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T02:59:05.609 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:59:05.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T02:59:05.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:59:05.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:59:05.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:59:05.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T02:59:05.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:59:05.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T02:59:05.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:59:05.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T02:59:05.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:59:05.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T02:59:05.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:59:05.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T02:59:05.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:59:05.613 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:59:05.613 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:59:05.613 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:59:05 smithi115 ceph-mon[78593]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T02:59:05.613 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T02:59:05.614 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:59:05.614 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T02:59:05.614 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:59:05.614 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:59:05.615 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:59:05.615 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T02:59:05.615 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:59:05.615 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T02:59:05.615 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:59:05.616 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T02:59:05.616 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:59:05.616 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T02:59:05.616 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:59:05.616 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T02:59:05.617 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:59:05.617 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:59:05.617 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:59:05.617 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T02:59:05.618 INFO:journalctl@ceph.mon.c.smithi115.stdout: During handling of the above exception, another exception occurred: 2022-05-14T02:59:05.618 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T02:59:05.618 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T02:59:05.618 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T02:59:05.618 INFO:journalctl@ceph.mon.c.smithi115.stdout: return f(*arg) 2022-05-14T02:59:05.619 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T02:59:05.619 INFO:journalctl@ceph.mon.c.smithi115.stdout: self._write_client_files(client_files, host) 2022-05-14T02:59:05.619 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T02:59:05.619 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T02:59:05.619 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T02:59:05.620 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T02:59:05.620 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T02:59:05.620 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.event_loop.get_result(coro) 2022-05-14T02:59:05.620 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T02:59:05.620 INFO:journalctl@ceph.mon.c.smithi115.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T02:59:05.621 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T02:59:05.621 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.__get_result() 2022-05-14T02:59:05.621 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T02:59:05.621 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise self._exception 2022-05-14T02:59:05.622 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T02:59:05.622 INFO:journalctl@ceph.mon.c.smithi115.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T02:59:05.622 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T02:59:05.622 INFO:journalctl@ceph.mon.c.smithi115.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T02:59:05.622 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T02:59:05.623 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise 2022-05-14T02:59:05.623 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T02:59:05.623 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.gen.throw(type, value, traceback) 2022-05-14T02:59:05.623 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T02:59:05.623 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OrchestratorError(msg) 2022-05-14T02:59:05.624 INFO:journalctl@ceph.mon.c.smithi115.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:59:05.624 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:59:05 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:59:06.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:59:05 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:59:06.042 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T02:59:06.042 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:59:06.042 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T02:59:06.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:59:06.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:59:06.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:59:06.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T02:59:06.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:59:06.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T02:59:06.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:59:06.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T02:59:06.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:59:06.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T02:59:06.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:59:06.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T02:59:06.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:59:06.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:59:06.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:59:06.046 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:59:05 smithi047 ceph-mon[77147]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T02:59:06.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T02:59:06.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T02:59:06.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T02:59:06.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T02:59:06.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T02:59:06.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T02:59:06.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T02:59:06.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T02:59:06.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T02:59:06.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T02:59:06.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T02:59:06.049 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T02:59:06.049 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T02:59:06.049 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T02:59:06.049 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T02:59:06.050 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T02:59:06.050 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T02:59:06.050 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:59:06.050 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T02:59:06.050 INFO:journalctl@ceph.mon.b.smithi047.stdout: During handling of the above exception, another exception occurred: 2022-05-14T02:59:06.051 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T02:59:06.051 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T02:59:06.051 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T02:59:06.051 INFO:journalctl@ceph.mon.b.smithi047.stdout: return f(*arg) 2022-05-14T02:59:06.051 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T02:59:06.052 INFO:journalctl@ceph.mon.b.smithi047.stdout: self._write_client_files(client_files, host) 2022-05-14T02:59:06.052 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T02:59:06.052 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T02:59:06.052 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T02:59:06.053 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T02:59:06.053 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T02:59:06.053 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.event_loop.get_result(coro) 2022-05-14T02:59:06.053 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T02:59:06.053 INFO:journalctl@ceph.mon.b.smithi047.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T02:59:06.054 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T02:59:06.054 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.__get_result() 2022-05-14T02:59:06.054 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T02:59:06.054 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise self._exception 2022-05-14T02:59:06.054 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T02:59:06.055 INFO:journalctl@ceph.mon.b.smithi047.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T02:59:06.055 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T02:59:06.055 INFO:journalctl@ceph.mon.b.smithi047.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T02:59:06.055 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T02:59:06.056 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise 2022-05-14T02:59:06.056 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T02:59:06.056 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.gen.throw(type, value, traceback) 2022-05-14T02:59:06.056 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T02:59:06.057 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OrchestratorError(msg) 2022-05-14T02:59:06.057 INFO:journalctl@ceph.mon.b.smithi047.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T02:59:06.057 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:59:05 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:59:06.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:59:06 smithi115 conmon[90393]: 2022-05-14T02:59:06.366+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:59:06.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:59:06 smithi115 ceph-mon[78593]: pgmap v6691: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:59:06.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:59:06 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 12680 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:59:06.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:59:06 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:59:07.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:59:06 smithi047 ceph-mon[77147]: pgmap v6691: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:59:07.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:59:06 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 12680 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:59:07.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:59:06 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:59:07.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:59:07 smithi115 conmon[90393]: 2022-05-14T02:59:07.363+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:59:07.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:59:07 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:59:08.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:59:07 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:59:08.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:59:08 smithi115 conmon[90393]: 2022-05-14T02:59:08.339+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:59:08.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:59:08 smithi115 ceph-mon[78593]: pgmap v6692: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:59:08.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:59:08 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:59:09.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:59:08 smithi047 ceph-mon[77147]: pgmap v6692: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:59:09.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:59:08 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:59:09.578 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:59:09 smithi115 conmon[90393]: 2022-05-14T02:59:09.297+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:59:09.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:59:09 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:59:10.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:59:09 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:59:10.577 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:59:10 smithi115 conmon[90393]: 2022-05-14T02:59:10.261+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:59:10.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:59:10 smithi115 ceph-mon[78593]: pgmap v6693: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-05-14T02:59:10.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:59:10 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:59:10.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:59:10 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:59:10.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:59:10 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:59:11.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:59:10 smithi047 ceph-mon[77147]: pgmap v6693: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-05-14T02:59:11.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:59:10 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:59:11.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:59:10 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:59:11.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:59:10 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:59:11.581 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:59:11 smithi115 conmon[90393]: 2022-05-14T02:59:11.293+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:59:11.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:59:11 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 12685 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:59:11.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:59:11 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:59:12.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:59:11 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 12685 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:59:12.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:59:11 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:59:12.582 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:59:12 smithi115 conmon[90393]: 2022-05-14T02:59:12.292+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:59:12.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:59:12 smithi115 ceph-mon[78593]: pgmap v6694: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:59:12.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:59:12 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:59:13.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:59:12 smithi047 ceph-mon[77147]: pgmap v6694: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:59:13.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:59:12 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:59:13.587 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:59:13 smithi115 conmon[90393]: 2022-05-14T02:59:13.263+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:59:13.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:59:13 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:59:14.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:59:13 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:59:14.588 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:59:14 smithi115 conmon[90393]: 2022-05-14T02:59:14.257+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:59:14.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:59:14 smithi115 ceph-mon[78593]: pgmap v6695: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:59:14.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:59:14 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:59:14.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:59:14 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:59:14.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:59:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T02:59:14.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:59:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T02:59:14.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:59:14 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:59:15.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:59:14 smithi047 ceph-mon[77147]: pgmap v6695: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:59:15.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:59:14 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:59:15.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:59:14 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:59:15.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:59:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T02:59:15.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:59:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T02:59:15.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:59:14 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:59:15.593 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:59:15 smithi115 conmon[90393]: 2022-05-14T02:59:15.263+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:59:15.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:59:15 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:59:16.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:59:15 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:59:16.595 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:59:16 smithi115 conmon[90393]: 2022-05-14T02:59:16.234+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:59:16.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:59:16 smithi115 ceph-mon[78593]: pgmap v6696: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:59:16.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:59:16 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 12690 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:59:16.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:59:16 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:59:17.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:59:16 smithi047 ceph-mon[77147]: pgmap v6696: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:59:17.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:59:16 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 12690 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:59:17.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:59:16 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:59:17.600 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:59:17 smithi115 conmon[90393]: 2022-05-14T02:59:17.235+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:59:17.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:59:17 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:59:18.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:59:17 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:59:18.601 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:59:18 smithi115 conmon[90393]: 2022-05-14T02:59:18.242+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:59:18.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:59:18 smithi115 ceph-mon[78593]: pgmap v6697: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:59:18.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:59:18 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:59:18.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:59:18 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:59:18.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:59:18 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:59:19.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:59:18 smithi047 ceph-mon[77147]: pgmap v6697: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:59:19.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:59:18 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:59:19.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:59:18 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:59:19.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:59:18 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:59:19.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:59:19 smithi115 conmon[90393]: 2022-05-14T02:59:19.205+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:59:19.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:59:19 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:59:20.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:59:19 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:59:20.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:59:20 smithi115 conmon[90393]: 2022-05-14T02:59:20.216+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:59:20.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:59:20 smithi115 ceph-mon[78593]: pgmap v6698: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-05-14T02:59:20.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:59:20 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:59:21.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:59:20 smithi047 ceph-mon[77147]: pgmap v6698: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-05-14T02:59:21.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:59:20 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:59:21.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:59:21 smithi115 conmon[90393]: 2022-05-14T02:59:21.243+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:59:22.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:59:21 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 12695 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:59:22.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:59:21 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:59:22.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:59:21 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 12695 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:59:22.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:59:21 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:59:22.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:59:22 smithi115 conmon[90393]: 2022-05-14T02:59:22.261+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:59:23.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:59:22 smithi047 ceph-mon[77147]: pgmap v6699: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:59:23.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:59:22 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:59:23.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:59:22 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:59:23.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:59:22 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:59:23.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:59:22 smithi115 ceph-mon[78593]: pgmap v6699: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:59:23.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:59:22 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:59:23.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:59:22 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:59:23.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:59:22 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:59:23.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:59:23 smithi115 conmon[90393]: 2022-05-14T02:59:23.263+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:59:24.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:59:23 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:59:24.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:59:23 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:59:24.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:59:24 smithi115 conmon[90393]: 2022-05-14T02:59:24.297+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:59:25.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:59:24 smithi047 ceph-mon[77147]: pgmap v6700: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:59:25.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:59:24 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:59:25.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:59:24 smithi115 ceph-mon[78593]: pgmap v6700: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:59:25.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:59:24 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:59:25.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:59:25 smithi115 conmon[90393]: 2022-05-14T02:59:25.270+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:59:26.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:59:25 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:59:26.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:59:25 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:59:26.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:59:26 smithi115 conmon[90393]: 2022-05-14T02:59:26.272+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:59:27.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:59:26 smithi047 ceph-mon[77147]: pgmap v6701: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:59:27.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:59:26 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 12700 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:59:27.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:59:26 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:59:27.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:59:26 smithi115 ceph-mon[78593]: pgmap v6701: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:59:27.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:59:26 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 12700 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:59:27.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:59:26 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:59:27.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:59:27 smithi115 conmon[90393]: 2022-05-14T02:59:27.225+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:59:28.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:59:27 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:59:28.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:59:27 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:59:28.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:59:28 smithi115 conmon[90393]: 2022-05-14T02:59:28.275+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:59:29.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:59:28 smithi047 ceph-mon[77147]: pgmap v6702: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:59:29.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:59:28 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:59:29.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:59:28 smithi115 ceph-mon[78593]: pgmap v6702: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:59:29.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:59:28 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:59:29.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:59:29 smithi115 conmon[90393]: 2022-05-14T02:59:29.298+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:59:30.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:59:29 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:59:30.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:59:29 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:59:30.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:59:30 smithi115 conmon[90393]: 2022-05-14T02:59:30.261+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:59:31.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:59:30 smithi047 ceph-mon[77147]: pgmap v6703: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-05-14T02:59:31.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:59:30 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:59:31.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:59:30 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:59:31.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:59:30 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:59:31.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:59:30 smithi115 ceph-mon[78593]: pgmap v6703: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-05-14T02:59:31.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:59:30 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:59:31.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:59:30 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:59:31.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:59:30 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:59:31.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:59:31 smithi115 conmon[90393]: 2022-05-14T02:59:31.246+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:59:32.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:59:31 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 12705 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:59:32.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:59:31 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:59:32.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:59:31 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 12705 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:59:32.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:59:31 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:59:32.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:59:32 smithi115 conmon[90393]: 2022-05-14T02:59:32.201+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:59:33.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:59:32 smithi047 ceph-mon[77147]: pgmap v6704: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:59:33.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:59:32 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:59:33.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:59:32 smithi115 ceph-mon[78593]: pgmap v6704: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:59:33.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:59:32 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:59:33.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:59:33 smithi115 conmon[90393]: 2022-05-14T02:59:33.246+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:59:34.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:59:33 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:59:34.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:59:33 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:59:34.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:59:34 smithi115 conmon[90393]: 2022-05-14T02:59:34.250+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:59:35.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:59:34 smithi047 ceph-mon[77147]: pgmap v6705: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:59:35.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:59:34 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:59:35.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:59:34 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:59:35.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:59:34 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:59:35.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:59:34 smithi115 ceph-mon[78593]: pgmap v6705: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:59:35.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:59:34 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:59:35.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:59:34 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:59:35.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:59:34 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:59:35.611 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:59:35 smithi115 conmon[90393]: 2022-05-14T02:59:35.294+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:59:36.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:59:35 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:59:36.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:59:35 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:59:36.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:59:36 smithi115 conmon[90393]: 2022-05-14T02:59:36.331+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:59:37.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:59:36 smithi047 ceph-mon[77147]: pgmap v6706: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:59:37.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:59:36 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 12710 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:59:37.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:59:36 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:59:37.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:59:36 smithi115 ceph-mon[78593]: pgmap v6706: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:59:37.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:59:36 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 12710 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:59:37.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:59:36 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:59:37.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:59:37 smithi115 conmon[90393]: 2022-05-14T02:59:37.321+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:59:38.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:59:37 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:59:38.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:59:37 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:59:38.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:59:38 smithi115 conmon[90393]: 2022-05-14T02:59:38.341+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:59:39.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:59:38 smithi047 ceph-mon[77147]: pgmap v6707: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:59:39.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:59:38 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:59:39.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:59:38 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:59:39.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:59:38 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:59:39.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:59:38 smithi115 ceph-mon[78593]: pgmap v6707: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:59:39.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:59:38 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:59:39.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:59:38 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:59:39.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:59:38 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:59:39.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:59:39 smithi115 conmon[90393]: 2022-05-14T02:59:39.311+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:59:40.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:59:39 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:59:40.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:59:39 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:59:40.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:59:40 smithi115 conmon[90393]: 2022-05-14T02:59:40.347+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:59:41.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:59:40 smithi047 ceph-mon[77147]: pgmap v6708: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-05-14T02:59:41.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:59:40 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:59:41.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:59:40 smithi115 ceph-mon[78593]: pgmap v6708: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-05-14T02:59:41.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:59:40 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:59:41.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:59:41 smithi115 conmon[90393]: 2022-05-14T02:59:41.350+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:59:42.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:59:41 smithi047 ceph-mon[77147]: pgmap v6709: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:59:42.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:59:41 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 12715 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:59:42.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:59:41 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:59:42.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:59:41 smithi115 ceph-mon[78593]: pgmap v6709: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:59:42.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:59:41 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 12715 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:59:42.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:59:41 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:59:42.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:59:42 smithi115 conmon[90393]: 2022-05-14T02:59:42.356+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:59:43.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:59:42 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:59:43.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:59:42 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:59:43.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:59:42 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:59:43.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:59:42 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:59:43.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:59:42 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:59:43.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:59:42 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:59:43.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:59:43 smithi115 conmon[90393]: 2022-05-14T02:59:43.324+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:59:44.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:59:43 smithi047 ceph-mon[77147]: pgmap v6710: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:59:44.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:59:43 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:59:44.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:59:43 smithi115 ceph-mon[78593]: pgmap v6710: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:59:44.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:59:43 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:59:44.610 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:59:44 smithi115 conmon[90393]: 2022-05-14T02:59:44.325+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:59:45.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:59:44 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:59:45.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:59:44 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:59:45.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:59:45 smithi115 conmon[90393]: 2022-05-14T02:59:45.295+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:59:46.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:59:45 smithi047 ceph-mon[77147]: pgmap v6711: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:59:46.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:59:45 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:59:46.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:59:45 smithi115 ceph-mon[78593]: pgmap v6711: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:59:46.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:59:45 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:59:46.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:59:46 smithi115 conmon[90393]: 2022-05-14T02:59:46.294+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:59:47.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:59:46 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 12720 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:59:47.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:59:46 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:59:47.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:59:46 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 12720 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:59:47.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:59:46 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:59:47.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:59:47 smithi115 conmon[90393]: 2022-05-14T02:59:47.258+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:59:48.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:59:47 smithi047 ceph-mon[77147]: pgmap v6712: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:59:48.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:59:47 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:59:48.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:59:47 smithi115 ceph-mon[78593]: pgmap v6712: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:59:48.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:59:47 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:59:48.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:59:48 smithi115 conmon[90393]: 2022-05-14T02:59:48.290+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:59:49.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:59:48 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:59:49.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:59:48 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:59:49.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:59:49 smithi115 conmon[90393]: 2022-05-14T02:59:49.280+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:59:50.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:59:49 smithi047 ceph-mon[77147]: pgmap v6713: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-05-14T02:59:50.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:59:49 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:59:50.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:59:49 smithi115 ceph-mon[78593]: pgmap v6713: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-05-14T02:59:50.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:59:49 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:59:50.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:59:50 smithi115 conmon[90393]: 2022-05-14T02:59:50.316+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:59:51.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:59:50 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:59:51.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:59:50 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:59:51.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:59:50 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:59:51.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:59:50 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:59:51.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:59:50 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:59:51.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:59:50 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:59:51.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:59:51 smithi115 conmon[90393]: 2022-05-14T02:59:51.279+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:59:52.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:59:51 smithi047 ceph-mon[77147]: pgmap v6714: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:59:52.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:59:51 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 12725 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:59:52.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:59:51 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:59:52.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:59:51 smithi115 ceph-mon[78593]: pgmap v6714: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:59:52.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:59:51 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 12725 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:59:52.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:59:51 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:59:52.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:59:52 smithi115 conmon[90393]: 2022-05-14T02:59:52.317+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:59:53.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:59:52 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:59:53.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:59:52 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:59:53.714 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:59:53 smithi115 conmon[90393]: 2022-05-14T02:59:53.364+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:59:54.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:59:53 smithi047 ceph-mon[77147]: pgmap v6715: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:59:54.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:59:53 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:59:54.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:59:53 smithi115 ceph-mon[78593]: pgmap v6715: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:59:54.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:59:53 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:59:54.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:59:54 smithi115 conmon[90393]: 2022-05-14T02:59:54.352+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:59:55.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:59:54 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:59:55.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:59:54 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:59:55.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:59:54 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:59:55.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:59:54 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:59:55.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:59:54 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:59:55.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:59:54 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:59:55.721 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:59:55 smithi115 conmon[90393]: 2022-05-14T02:59:55.359+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:59:56.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:59:55 smithi047 ceph-mon[77147]: pgmap v6716: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:59:56.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:59:55 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:59:56.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:59:55 smithi115 ceph-mon[78593]: pgmap v6716: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T02:59:56.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:59:55 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:59:56.722 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:59:56 smithi115 conmon[90393]: 2022-05-14T02:59:56.404+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:59:57.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:59:56 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 12730 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:59:57.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:59:56 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:59:57.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:59:56 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 12730 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T02:59:57.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:59:56 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:59:57.728 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:59:57 smithi115 conmon[90393]: 2022-05-14T02:59:57.431+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:59:58.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:59:57 smithi047 ceph-mon[77147]: pgmap v6717: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:59:58.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:59:57 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:59:58.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:59:57 smithi115 ceph-mon[78593]: pgmap v6717: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T02:59:58.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:59:57 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:59:58.729 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:59:58 smithi115 conmon[90393]: 2022-05-14T02:59:58.449+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T02:59:59.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:59:58 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:59:59.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:59:58 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:59:59.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:59:58 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:59:59.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:59:58 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T02:59:59.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:59:58 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T02:59:59.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:59:58 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T02:59:59.734 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 02:59:59 smithi115 conmon[90393]: 2022-05-14T02:59:59.434+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:00:00.001 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:59:59 smithi047 ceph-mon[77147]: pgmap v6718: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-05-14T03:00:00.001 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 02:59:59 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:00:00.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:59:59 smithi115 ceph-mon[78593]: pgmap v6718: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-05-14T03:00:00.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 02:59:59 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:00:00.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:00:00 smithi047 conmon[77123]: 2022-05-14T02:59:59.999+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : Health detail: HEALTH_ERR 1 filesystem is degraded; 1 mds daemon damaged; 1 MDSs are read only; 1 MDSs report slow metadata IOs; 1/3 mons down, quorum b,c; Reduced data availability: 1 pg inactive; 9 slow ops, oldest one blocked for 12735 sec, osd.5 has slow ops 2022-05-14T03:00:00.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:00:00 smithi047 conmon[77123]: 2022-05-14T02:59:59.999+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : [WRN] FS_DEGRADED: 1 filesystem is degraded 2022-05-14T03:00:00.295 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:00:00 smithi047 conmon[77123]: 2022-05-14T02:59:59.999+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : fs cephfs is degraded 2022-05-14T03:00:00.296 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:00:00 smithi047 conmon[77123]: 2022-05-14T02:59:59.999+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : [ERR] MDS_DAMAGE: 1 mds daemon damaged 2022-05-14T03:00:00.296 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:00:00 smithi047 conmon[77123]: 2022-05-14T02:59:59.999+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : fs cephfs mds.2 is damaged 2022-05-14T03:00:00.296 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:00:00 smithi047 conmon[77123]: 2022-05-14T02:59:59.999+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : [WRN] MDS_READ_ONLY: 1 MDSs are read only 2022-05-14T03:00:00.297 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:00:00 smithi047 conmon[77123]: 2022-05-14T02:59:59.999+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : mds.l(mds.0): MDS in read-only mode 2022-05-14T03:00:00.297 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:00:00 smithi047 conmon[77123]: 2022-05-14T02:59:59.999+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : [WRN] MDS_SLOW_METADATA_IO: 1 MDSs report slow metadata IOs 2022-05-14T03:00:00.298 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:00:00 smithi047 conmon[77123]: 2022-05-14T03:00:00.000+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : mds.l(mds.0): 4 slow metadata IOs are blocked > 180 secs, oldest blocked for 13033 secs 2022-05-14T03:00:00.298 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:00:00 smithi047 conmon[77123]: 2022-05-14T03:00:00.000+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : [WRN] MON_DOWN: 1/3 mons down, quorum b,c 2022-05-14T03:00:00.298 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:00:00 smithi047 conmon[77123]: 2022-05-14T03:00:00.000+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : mon.a (rank 0) addr [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] is down (out of quorum) 2022-05-14T03:00:00.299 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:00:00 smithi047 conmon[77123]: 2022-05-14T03:00:00.000+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : [WRN] PG_AVAILABILITY: Reduced data availability: 1 pg inactive 2022-05-14T03:00:00.299 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:00:00 smithi047 conmon[77123]: 2022-05-14T03:00:00.000+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : pg 2.1 is stuck inactive for 3h, current state unknown, last acting [] 2022-05-14T03:00:00.299 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:00:00 smithi047 conmon[77123]: 2022-05-14T03:00:00.000+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : [WRN] SLOW_OPS: 9 slow ops, oldest one blocked for 12735 sec, osd.5 has slow ops 2022-05-14T03:00:00.738 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:00:00 smithi115 conmon[90393]: 2022-05-14T03:00:00.415+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:00:01.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:00:00 smithi047 ceph-mon[77147]: Health detail: HEALTH_ERR 1 filesystem is degraded; 1 mds daemon damaged; 1 MDSs are read only; 1 MDSs report slow metadata IOs; 1/3 mons down, quorum b,c; Reduced data availability: 1 pg inactive; 9 slow ops, oldest one blocked for 12735 sec, osd.5 has slow ops 2022-05-14T03:00:01.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:00:00 smithi047 ceph-mon[77147]: [WRN] FS_DEGRADED: 1 filesystem is degraded 2022-05-14T03:00:01.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:00:00 smithi047 ceph-mon[77147]: fs cephfs is degraded 2022-05-14T03:00:01.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:00:00 smithi047 ceph-mon[77147]: [ERR] MDS_DAMAGE: 1 mds daemon damaged 2022-05-14T03:00:01.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:00:00 smithi047 ceph-mon[77147]: fs cephfs mds.2 is damaged 2022-05-14T03:00:01.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:00:00 smithi047 ceph-mon[77147]: [WRN] MDS_READ_ONLY: 1 MDSs are read only 2022-05-14T03:00:01.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:00:00 smithi047 ceph-mon[77147]: mds.l(mds.0): MDS in read-only mode 2022-05-14T03:00:01.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:00:00 smithi047 ceph-mon[77147]: [WRN] MDS_SLOW_METADATA_IO: 1 MDSs report slow metadata IOs 2022-05-14T03:00:01.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:00:00 smithi047 ceph-mon[77147]: mds.l(mds.0): 4 slow metadata IOs are blocked > 180 secs, oldest blocked for 13033 secs 2022-05-14T03:00:01.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:00:00 smithi047 ceph-mon[77147]: [WRN] MON_DOWN: 1/3 mons down, quorum b,c 2022-05-14T03:00:01.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:00:00 smithi047 ceph-mon[77147]: mon.a (rank 0) addr [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] is down (out of quorum) 2022-05-14T03:00:01.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:00:00 smithi047 ceph-mon[77147]: [WRN] PG_AVAILABILITY: Reduced data availability: 1 pg inactive 2022-05-14T03:00:01.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:00:00 smithi047 ceph-mon[77147]: pg 2.1 is stuck inactive for 3h, current state unknown, last acting [] 2022-05-14T03:00:01.045 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:00:00 smithi047 ceph-mon[77147]: [WRN] SLOW_OPS: 9 slow ops, oldest one blocked for 12735 sec, osd.5 has slow ops 2022-05-14T03:00:01.045 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:00:00 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:00:01.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:00:00 smithi115 ceph-mon[78593]: Health detail: HEALTH_ERR 1 filesystem is degraded; 1 mds daemon damaged; 1 MDSs are read only; 1 MDSs report slow metadata IOs; 1/3 mons down, quorum b,c; Reduced data availability: 1 pg inactive; 9 slow ops, oldest one blocked for 12735 sec, osd.5 has slow ops 2022-05-14T03:00:01.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:00:00 smithi115 ceph-mon[78593]: [WRN] FS_DEGRADED: 1 filesystem is degraded 2022-05-14T03:00:01.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:00:00 smithi115 ceph-mon[78593]: fs cephfs is degraded 2022-05-14T03:00:01.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:00:00 smithi115 ceph-mon[78593]: [ERR] MDS_DAMAGE: 1 mds daemon damaged 2022-05-14T03:00:01.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:00:00 smithi115 ceph-mon[78593]: fs cephfs mds.2 is damaged 2022-05-14T03:00:01.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:00:00 smithi115 ceph-mon[78593]: [WRN] MDS_READ_ONLY: 1 MDSs are read only 2022-05-14T03:00:01.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:00:00 smithi115 ceph-mon[78593]: mds.l(mds.0): MDS in read-only mode 2022-05-14T03:00:01.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:00:00 smithi115 ceph-mon[78593]: [WRN] MDS_SLOW_METADATA_IO: 1 MDSs report slow metadata IOs 2022-05-14T03:00:01.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:00:00 smithi115 ceph-mon[78593]: mds.l(mds.0): 4 slow metadata IOs are blocked > 180 secs, oldest blocked for 13033 secs 2022-05-14T03:00:01.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:00:00 smithi115 ceph-mon[78593]: [WRN] MON_DOWN: 1/3 mons down, quorum b,c 2022-05-14T03:00:01.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:00:00 smithi115 ceph-mon[78593]: mon.a (rank 0) addr [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] is down (out of quorum) 2022-05-14T03:00:01.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:00:00 smithi115 ceph-mon[78593]: [WRN] PG_AVAILABILITY: Reduced data availability: 1 pg inactive 2022-05-14T03:00:01.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:00:00 smithi115 ceph-mon[78593]: pg 2.1 is stuck inactive for 3h, current state unknown, last acting [] 2022-05-14T03:00:01.112 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:00:00 smithi115 ceph-mon[78593]: [WRN] SLOW_OPS: 9 slow ops, oldest one blocked for 12735 sec, osd.5 has slow ops 2022-05-14T03:00:01.112 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:00:00 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:00:01.741 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:00:01 smithi115 conmon[90393]: 2022-05-14T03:00:01.460+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:00:02.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:00:01 smithi047 ceph-mon[77147]: pgmap v6719: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:00:02.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:00:01 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 12735 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:00:02.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:00:01 smithi115 ceph-mon[78593]: pgmap v6719: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:00:02.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:00:01 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 12735 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:00:02.742 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:00:02 smithi115 conmon[90393]: 2022-05-14T03:00:02.480+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:00:03.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:00:02 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:00:03.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:00:02 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:00:03.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:00:02 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:00:03.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:00:02 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:00:03.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:00:02 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:00:03.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:00:02 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:00:03.747 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:00:03 smithi115 conmon[90393]: 2022-05-14T03:00:03.492+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:00:04.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:00:03 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:00:04.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:00:03 smithi047 ceph-mon[77147]: pgmap v6720: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:00:04.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:00:03 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:00:04.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:00:03 smithi115 ceph-mon[78593]: pgmap v6720: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:00:04.749 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:00:04 smithi115 conmon[90393]: 2022-05-14T03:00:04.522+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:00:05.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:00:04 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:00:05.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:00:04 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T03:00:05.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:00:04 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T03:00:05.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:00:04 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T03:00:05.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:00:04 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:00:05.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:00:04 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T03:00:05.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:00:04 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T03:00:05.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:00:04 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T03:00:05.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:00:05 smithi115 conmon[90393]: 2022-05-14T03:00:05.554+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:00:05.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:00:05 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:00:05.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:00:05 smithi115 ceph-mon[78593]: pgmap v6721: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:00:06.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:00:05 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:00:06.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:00:05 smithi047 ceph-mon[77147]: pgmap v6721: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:00:06.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:00:06 smithi115 conmon[90393]: 2022-05-14T03:00:06.533+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:00:06.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:00:06 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:00:06.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:00:06 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 12740 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:00:07.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:00:06 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:00:07.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:00:06 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 12740 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:00:07.761 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:00:07 smithi047 conmon[78521]: 2022-05-14T03:00:07.472+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:00:07.808 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:00:07 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T03:00:07.808 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:00:07 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:00:07.808 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:00:07 smithi047 conmon[78521]: yield 2022-05-14T03:00:07.809 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:00:07 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:00:07.809 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:00:07 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:00:07.809 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:00:07 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:00:07.809 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:00:07 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T03:00:07.809 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:00:07 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:00:07.810 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:00:07 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T03:00:07.810 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:00:07 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:00:07.810 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:00:07 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T03:00:07.810 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:00:07 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:00:07.811 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:00:07 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T03:00:07.811 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:00:07 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:00:07.811 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:00:07 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T03:00:07.812 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:00:07 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:00:07.812 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:00:07 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:00:07.812 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:00:07 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:00:07.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:00:07 smithi115 conmon[90393]: 2022-05-14T03:00:07.548+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:00:07.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:00:07 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:00:07.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:00:07 smithi115 ceph-mon[78593]: pgmap v6722: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:00:07.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:00:07 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:00:07.860 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T03:00:07.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:00:07.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T03:00:07.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:00:07.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:00:07.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:00:07.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T03:00:07.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:00:07.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T03:00:07.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:00:07.863 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T03:00:07.863 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:00:07.863 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T03:00:07.863 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:00:07.864 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T03:00:07.864 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:00:07.864 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:00:07.864 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:00:07.864 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:00:07 smithi115 ceph-mon[78593]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T03:00:08.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:00:07 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:00:08.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:00:07 smithi047 ceph-mon[77147]: pgmap v6722: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:00:08.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:00:07 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:00:08.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T03:00:08.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:00:08.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T03:00:08.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:00:08.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:00:08.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:00:08.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T03:00:08.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:00:08.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T03:00:08.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:00:08.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T03:00:08.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:00:08.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T03:00:08.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:00:08.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T03:00:08.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:00:08.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:00:08.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:00:08.047 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:00:07 smithi047 ceph-mon[77147]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T03:00:08.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:00:08 smithi115 conmon[90393]: 2022-05-14T03:00:08.571+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:00:08.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:00:08 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:00:09.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:00:08 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:00:09.861 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:00:09 smithi115 conmon[90393]: 2022-05-14T03:00:09.572+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:00:09.862 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:00:09 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:00:09.862 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:00:09 smithi115 ceph-mon[78593]: pgmap v6723: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-05-14T03:00:10.047 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:00:09 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:00:10.047 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:00:09 smithi047 ceph-mon[77147]: pgmap v6723: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-05-14T03:00:10.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:00:10 smithi115 conmon[90393]: 2022-05-14T03:00:10.525+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:00:10.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:00:10 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:00:10.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:00:10 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:00:10.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:00:10 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:00:10.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:00:10 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:00:10.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T03:00:10.865 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:00:10.866 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T03:00:10.866 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:00:10.866 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:00:10.867 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:00:10.867 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T03:00:10.868 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:00:10.868 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T03:00:10.869 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:00:10.870 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T03:00:10.871 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:00:10.872 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T03:00:10.872 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:00:10.872 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T03:00:10.873 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:00:10.873 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:00:10.874 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:00:10.874 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:00:10 smithi115 ceph-mon[78593]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T03:00:10.875 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T03:00:10.875 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:00:10.876 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T03:00:10.879 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:00:10.880 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:00:10.880 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:00:10.881 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T03:00:10.881 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:00:10.881 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T03:00:10.882 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:00:10.882 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T03:00:10.883 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:00:10.883 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T03:00:10.884 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:00:10.884 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T03:00:10.884 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:00:10.885 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:00:10.885 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:00:10.886 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T03:00:10.886 INFO:journalctl@ceph.mon.c.smithi115.stdout: During handling of the above exception, another exception occurred: 2022-05-14T03:00:10.887 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T03:00:10.887 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T03:00:10.888 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T03:00:10.890 INFO:journalctl@ceph.mon.c.smithi115.stdout: return f(*arg) 2022-05-14T03:00:10.890 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T03:00:10.890 INFO:journalctl@ceph.mon.c.smithi115.stdout: self._write_client_files(client_files, host) 2022-05-14T03:00:10.891 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T03:00:10.891 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T03:00:10.891 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T03:00:10.891 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T03:00:10.892 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T03:00:10.892 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.event_loop.get_result(coro) 2022-05-14T03:00:10.892 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T03:00:10.892 INFO:journalctl@ceph.mon.c.smithi115.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T03:00:10.892 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T03:00:10.893 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.__get_result() 2022-05-14T03:00:10.893 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T03:00:10.893 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise self._exception 2022-05-14T03:00:10.893 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T03:00:10.893 INFO:journalctl@ceph.mon.c.smithi115.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T03:00:10.894 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T03:00:10.894 INFO:journalctl@ceph.mon.c.smithi115.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T03:00:10.894 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T03:00:10.894 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise 2022-05-14T03:00:10.895 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T03:00:10.895 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.gen.throw(type, value, traceback) 2022-05-14T03:00:10.895 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T03:00:10.895 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OrchestratorError(msg) 2022-05-14T03:00:10.895 INFO:journalctl@ceph.mon.c.smithi115.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:00:11.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:00:10 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:00:11.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:00:10 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:00:11.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:00:10 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:00:11.045 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:00:10 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:00:11.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T03:00:11.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:00:11.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T03:00:11.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:00:11.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:00:11.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:00:11.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T03:00:11.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:00:11.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T03:00:11.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:00:11.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T03:00:11.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:00:11.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T03:00:11.056 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:00:11.057 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T03:00:11.058 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:00:11.058 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:00:11.059 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:00:11.059 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:00:10 smithi047 ceph-mon[77147]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T03:00:11.060 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T03:00:11.060 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:00:11.060 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T03:00:11.061 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:00:11.061 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:00:11.062 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:00:11.062 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T03:00:11.062 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:00:11.063 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T03:00:11.063 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:00:11.069 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T03:00:11.070 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:00:11.070 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T03:00:11.071 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:00:11.071 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T03:00:11.071 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:00:11.072 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:00:11.072 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:00:11.073 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T03:00:11.073 INFO:journalctl@ceph.mon.b.smithi047.stdout: During handling of the above exception, another exception occurred: 2022-05-14T03:00:11.074 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T03:00:11.074 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T03:00:11.075 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T03:00:11.075 INFO:journalctl@ceph.mon.b.smithi047.stdout: return f(*arg) 2022-05-14T03:00:11.075 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T03:00:11.076 INFO:journalctl@ceph.mon.b.smithi047.stdout: self._write_client_files(client_files, host) 2022-05-14T03:00:11.076 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T03:00:11.077 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T03:00:11.077 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T03:00:11.077 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T03:00:11.078 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T03:00:11.078 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.event_loop.get_result(coro) 2022-05-14T03:00:11.079 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T03:00:11.079 INFO:journalctl@ceph.mon.b.smithi047.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T03:00:11.079 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T03:00:11.080 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.__get_result() 2022-05-14T03:00:11.080 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T03:00:11.081 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise self._exception 2022-05-14T03:00:11.081 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T03:00:11.081 INFO:journalctl@ceph.mon.b.smithi047.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T03:00:11.081 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T03:00:11.082 INFO:journalctl@ceph.mon.b.smithi047.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T03:00:11.082 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T03:00:11.083 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise 2022-05-14T03:00:11.083 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T03:00:11.083 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.gen.throw(type, value, traceback) 2022-05-14T03:00:11.084 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T03:00:11.084 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OrchestratorError(msg) 2022-05-14T03:00:11.085 INFO:journalctl@ceph.mon.b.smithi047.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:00:11.086 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:00:10 smithi047 conmon[78521]: 2022-05-14T03:00:10.544+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:00:11.087 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:00:10 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T03:00:11.087 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:00:10 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:00:11.087 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:00:10 smithi047 conmon[78521]: yield 2022-05-14T03:00:11.088 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:00:10 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:00:11.088 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:00:10 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:00:11.089 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:00:10 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:00:11.089 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:00:10 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T03:00:11.089 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:00:10 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:00:11.090 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:00:10 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T03:00:11.090 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:00:10 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:00:11.091 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:00:10 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T03:00:11.091 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:00:10 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:00:11.091 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:00:10 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T03:00:11.092 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:00:10 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:00:11.092 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:00:10 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T03:00:11.092 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:00:10 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:00:11.092 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:00:10 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:00:11.093 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:00:10 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:00:11.093 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:00:10 smithi047 conmon[78521]: 2022-05-14T03:00:10.545+0000 7f5344781700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T03:00:11.093 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:00:10 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T03:00:11.093 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:00:10 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:00:11.094 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:00:10 smithi047 conmon[78521]: yield 2022-05-14T03:00:11.094 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:00:10 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:00:11.094 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:00:10 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:00:11.094 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:00:10 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:00:11.095 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:00:10 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T03:00:11.095 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:00:10 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:00:11.095 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:00:10 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T03:00:11.095 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:00:10 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:00:11.096 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:00:10 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T03:00:11.096 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:00:10 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:00:11.096 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:00:10 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T03:00:11.097 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:00:10 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:00:11.097 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:00:10 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T03:00:11.097 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:00:10 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:00:11.097 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:00:10 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:00:11.098 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:00:10 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:00:11.098 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:00:10 smithi047 conmon[78521]: 2022-05-14T03:00:11.098 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:00:10 smithi047 conmon[78521]: During handling of the above exception, another exception occurred: 2022-05-14T03:00:11.099 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:00:10 smithi047 conmon[78521]: 2022-05-14T03:00:11.099 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:00:10 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T03:00:11.099 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:00:10 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T03:00:11.099 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:00:10 smithi047 conmon[78521]: return f(*arg) 2022-05-14T03:00:11.100 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:00:10 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T03:00:11.100 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:00:10 smithi047 conmon[78521]: self._write_client_files(client_files, host) 2022-05-14T03:00:11.100 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:00:10 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T03:00:11.100 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:00:10 smithi047 conmon[78521]: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T03:00:11.101 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:00:10 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T03:00:11.101 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:00:10 smithi047 conmon[78521]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T03:00:11.102 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:00:10 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T03:00:11.102 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:00:10 smithi047 conmon[78521]: return self.event_loop.get_result(coro) 2022-05-14T03:00:11.102 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:00:10 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T03:00:11.102 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:00:10 smithi047 conmon[78521]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T03:00:11.103 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:00:10 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T03:00:11.103 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:00:10 smithi047 conmon[78521]: return self.__get_result() 2022-05-14T03:00:11.103 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:00:10 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T03:00:11.104 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:00:10 smithi047 conmon[78521]: raise self._exception 2022-05-14T03:00:11.104 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:00:10 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T03:00:11.104 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:00:10 smithi047 conmon[78521]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T03:00:11.105 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:00:10 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T03:00:11.105 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:00:10 smithi047 conmon[78521]: conn = await self._remote_connection(host, addr) 2022-05-14T03:00:11.106 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:00:10 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T03:00:11.106 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:00:10 smithi047 conmon[78521]: raise 2022-05-14T03:00:11.106 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:00:10 smithi047 conmon[78521]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T03:00:11.106 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:00:10 smithi047 conmon[78521]: self.gen.throw(type, value, traceback) 2022-05-14T03:00:11.107 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:00:10 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T03:00:11.107 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:00:10 smithi047 conmon[78521]: raise OrchestratorError(msg) 2022-05-14T03:00:11.107 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:00:10 smithi047 conmon[78521]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:00:11.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:00:11 smithi115 conmon[90393]: 2022-05-14T03:00:11.528+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:00:11.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:00:11 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:00:11.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:00:11 smithi115 ceph-mon[78593]: pgmap v6724: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:00:11.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:00:11 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 12745 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:00:12.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:00:11 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:00:12.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:00:11 smithi047 ceph-mon[77147]: pgmap v6724: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:00:12.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:00:11 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 12745 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:00:12.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:00:12 smithi115 conmon[90393]: 2022-05-14T03:00:12.548+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:00:12.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:00:12 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:00:13.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:00:12 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:00:13.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:00:13 smithi115 conmon[90393]: 2022-05-14T03:00:13.561+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:00:13.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:00:13 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:00:13.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:00:13 smithi115 ceph-mon[78593]: pgmap v6725: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:00:14.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:00:13 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:00:14.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:00:13 smithi047 ceph-mon[77147]: pgmap v6725: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:00:14.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:00:14 smithi115 conmon[90393]: 2022-05-14T03:00:14.582+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:00:14.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:00:14 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:00:14.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:00:14 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:00:14.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:00:14 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:00:14.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:00:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T03:00:14.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:00:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T03:00:15.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:00:14 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:00:15.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:00:14 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:00:15.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:00:14 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:00:15.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:00:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T03:00:15.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:00:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T03:00:15.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:00:15 smithi115 conmon[90393]: 2022-05-14T03:00:15.539+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:00:15.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:00:15 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:00:15.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:00:15 smithi115 ceph-mon[78593]: pgmap v6726: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:00:16.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:00:15 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:00:16.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:00:15 smithi047 ceph-mon[77147]: pgmap v6726: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:00:16.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:00:16 smithi115 conmon[90393]: 2022-05-14T03:00:16.580+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:00:16.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:00:16 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:00:16.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:00:16 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 12750 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:00:17.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:00:16 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:00:17.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:00:16 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 12750 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:00:17.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:00:17 smithi115 conmon[90393]: 2022-05-14T03:00:17.579+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:00:17.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:00:17 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:00:17.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:00:17 smithi115 ceph-mon[78593]: pgmap v6727: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:00:18.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:00:17 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:00:18.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:00:17 smithi047 ceph-mon[77147]: pgmap v6727: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:00:19.111 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:00:18 smithi115 conmon[90393]: 2022-05-14T03:00:18.619+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:00:19.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:00:18 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:00:19.112 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:00:18 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:00:19.112 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:00:18 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:00:19.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:00:18 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:00:19.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:00:18 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:00:19.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:00:18 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:00:19.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:00:19 smithi115 conmon[90393]: 2022-05-14T03:00:19.595+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:00:19.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:00:19 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:00:19.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:00:19 smithi115 ceph-mon[78593]: pgmap v6728: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-05-14T03:00:20.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:00:19 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:00:20.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:00:19 smithi047 ceph-mon[77147]: pgmap v6728: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-05-14T03:00:21.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:00:20 smithi115 conmon[90393]: 2022-05-14T03:00:20.634+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:00:21.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:00:20 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:00:21.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:00:20 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:00:22.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:00:21 smithi115 conmon[90393]: 2022-05-14T03:00:21.640+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:00:22.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:00:21 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:00:22.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:00:21 smithi115 ceph-mon[78593]: pgmap v6729: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:00:22.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:00:21 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 12755 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:00:22.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:00:21 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:00:22.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:00:21 smithi047 ceph-mon[77147]: pgmap v6729: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:00:22.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:00:21 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 12755 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:00:23.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:00:22 smithi115 conmon[90393]: 2022-05-14T03:00:22.617+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:00:23.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:00:22 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:00:23.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:00:22 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:00:23.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:00:22 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:00:23.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:00:22 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:00:23.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:00:22 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:00:23.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:00:22 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:00:24.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:00:23 smithi115 conmon[90393]: 2022-05-14T03:00:23.643+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:00:24.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:00:23 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:00:24.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:00:23 smithi115 ceph-mon[78593]: pgmap v6730: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:00:24.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:00:23 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:00:24.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:00:23 smithi047 ceph-mon[77147]: pgmap v6730: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:00:25.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:00:24 smithi115 conmon[90393]: 2022-05-14T03:00:24.628+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:00:25.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:00:24 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:00:25.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:00:24 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:00:26.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:00:25 smithi115 conmon[90393]: 2022-05-14T03:00:25.670+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:00:26.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:00:25 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:00:26.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:00:25 smithi115 ceph-mon[78593]: pgmap v6731: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:00:26.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:00:25 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:00:26.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:00:25 smithi047 ceph-mon[77147]: pgmap v6731: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:00:27.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:00:26 smithi115 conmon[90393]: 2022-05-14T03:00:26.703+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:00:27.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:00:26 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:00:27.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:00:26 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 12760 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:00:27.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:00:26 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:00:27.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:00:26 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 12760 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:00:28.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:00:27 smithi115 conmon[90393]: 2022-05-14T03:00:27.655+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:00:28.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:00:27 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:00:28.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:00:27 smithi115 ceph-mon[78593]: pgmap v6732: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:00:28.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:00:27 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:00:28.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:00:27 smithi047 ceph-mon[77147]: pgmap v6732: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:00:29.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:00:28 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:00:29.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:00:28 smithi115 conmon[90393]: 2022-05-14T03:00:28.681+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:00:29.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:00:28 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:00:30.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:00:29 smithi115 conmon[90393]: 2022-05-14T03:00:29.686+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:00:30.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:00:29 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:00:30.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:00:29 smithi115 ceph-mon[78593]: pgmap v6733: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-05-14T03:00:30.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:00:29 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:00:30.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:00:29 smithi047 ceph-mon[77147]: pgmap v6733: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-05-14T03:00:31.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:00:30 smithi115 conmon[90393]: 2022-05-14T03:00:30.671+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:00:31.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:00:30 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:00:31.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:00:30 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:00:31.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:00:30 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:00:31.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:00:30 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:00:31.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:00:30 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:00:31.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:00:30 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:00:32.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:00:31 smithi115 conmon[90393]: 2022-05-14T03:00:31.689+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:00:32.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:00:31 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:00:32.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:00:31 smithi115 ceph-mon[78593]: pgmap v6734: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:00:32.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:00:31 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 12765 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:00:32.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:00:31 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:00:32.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:00:31 smithi047 ceph-mon[77147]: pgmap v6734: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:00:32.295 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:00:31 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 12765 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:00:33.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:00:32 smithi115 conmon[90393]: 2022-05-14T03:00:32.683+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:00:33.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:00:32 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:00:33.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:00:32 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:00:34.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:00:33 smithi115 conmon[90393]: 2022-05-14T03:00:33.656+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:00:34.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:00:33 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:00:34.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:00:33 smithi115 ceph-mon[78593]: pgmap v6735: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:00:34.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:00:33 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:00:34.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:00:33 smithi047 ceph-mon[77147]: pgmap v6735: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:00:35.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:00:34 smithi115 conmon[90393]: 2022-05-14T03:00:34.641+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:00:35.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:00:34 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:00:35.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:00:34 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:00:35.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:00:34 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:00:35.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:00:34 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:00:35.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:00:34 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:00:35.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:00:34 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:00:35.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:00:35 smithi115 conmon[90393]: 2022-05-14T03:00:35.603+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:00:35.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:00:35 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:00:35.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:00:35 smithi115 ceph-mon[78593]: pgmap v6736: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:00:36.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:00:35 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:00:36.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:00:35 smithi047 ceph-mon[77147]: pgmap v6736: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:00:37.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:00:36 smithi115 conmon[90393]: 2022-05-14T03:00:36.621+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:00:37.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:00:36 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:00:37.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:00:36 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 12770 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:00:37.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:00:36 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:00:37.295 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:00:36 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 12770 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:00:38.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:00:37 smithi115 conmon[90393]: 2022-05-14T03:00:37.617+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:00:38.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:00:37 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:00:38.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:00:37 smithi115 ceph-mon[78593]: pgmap v6737: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:00:38.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:00:37 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:00:38.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:00:37 smithi047 ceph-mon[77147]: pgmap v6737: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:00:38.860 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:00:38 smithi115 conmon[90393]: 2022-05-14T03:00:38.589+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:00:39.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:00:38 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:00:39.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:00:38 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:00:39.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:00:38 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:00:39.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:00:38 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:00:39.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:00:38 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:00:39.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:00:38 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:00:39.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:00:39 smithi115 conmon[90393]: 2022-05-14T03:00:39.589+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:00:40.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:00:39 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:00:40.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:00:39 smithi047 ceph-mon[77147]: pgmap v6738: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-05-14T03:00:40.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:00:39 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:00:40.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:00:39 smithi115 ceph-mon[78593]: pgmap v6738: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-05-14T03:00:40.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:00:40 smithi115 conmon[90393]: 2022-05-14T03:00:40.553+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:00:41.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:00:40 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:00:41.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:00:40 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:00:41.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:00:41 smithi115 conmon[90393]: 2022-05-14T03:00:41.535+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:00:42.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:00:41 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:00:42.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:00:41 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 12775 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:00:42.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:00:41 smithi047 ceph-mon[77147]: pgmap v6739: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:00:42.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:00:41 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:00:42.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:00:41 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 12775 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:00:42.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:00:41 smithi115 ceph-mon[78593]: pgmap v6739: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:00:42.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:00:42 smithi115 conmon[90393]: 2022-05-14T03:00:42.579+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:00:43.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:00:42 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:00:43.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:00:42 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:00:43.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:00:42 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:00:43.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:00:42 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:00:43.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:00:42 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:00:43.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:00:42 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:00:43.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:00:43 smithi115 conmon[90393]: 2022-05-14T03:00:43.601+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:00:44.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:00:43 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:00:44.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:00:43 smithi047 ceph-mon[77147]: pgmap v6740: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:00:44.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:00:43 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:00:44.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:00:43 smithi115 ceph-mon[78593]: pgmap v6740: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:00:45.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:00:44 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:00:45.111 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:00:44 smithi115 conmon[90393]: 2022-05-14T03:00:44.649+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:00:45.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:00:44 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:00:46.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:00:45 smithi115 conmon[90393]: 2022-05-14T03:00:45.634+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:00:46.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:00:45 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:00:46.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:00:45 smithi115 ceph-mon[78593]: pgmap v6741: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:00:46.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:00:45 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:00:46.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:00:45 smithi047 ceph-mon[77147]: pgmap v6741: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:00:46.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:00:46 smithi115 conmon[90393]: 2022-05-14T03:00:46.590+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:00:47.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:00:46 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:00:47.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:00:46 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 12780 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:00:47.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:00:46 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:00:47.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:00:46 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 12780 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:00:47.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:00:47 smithi115 conmon[90393]: 2022-05-14T03:00:47.581+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:00:48.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:00:47 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:00:48.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:00:47 smithi047 ceph-mon[77147]: pgmap v6742: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:00:48.362 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:00:47 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:00:48.362 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:00:47 smithi115 ceph-mon[78593]: pgmap v6742: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:00:48.861 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:00:48 smithi115 conmon[90393]: 2022-05-14T03:00:48.543+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:00:49.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:00:48 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:00:49.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:00:48 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:00:49.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:00:49 smithi115 conmon[90393]: 2022-05-14T03:00:49.590+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:00:50.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:00:49 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:00:50.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:00:49 smithi047 ceph-mon[77147]: pgmap v6743: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-05-14T03:00:50.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:00:49 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:00:50.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:00:49 smithi115 ceph-mon[78593]: pgmap v6743: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-05-14T03:00:50.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:00:50 smithi115 conmon[90393]: 2022-05-14T03:00:50.571+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:00:51.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:00:50 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:00:51.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:00:50 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:00:51.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:00:50 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:00:51.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:00:50 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:00:51.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:00:50 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:00:51.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:00:50 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:00:51.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:00:51 smithi115 conmon[90393]: 2022-05-14T03:00:51.569+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:00:52.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:00:51 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:00:52.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:00:51 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 12785 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:00:52.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:00:51 smithi047 ceph-mon[77147]: pgmap v6744: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:00:52.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:00:51 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:00:52.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:00:51 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 12785 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:00:52.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:00:51 smithi115 ceph-mon[78593]: pgmap v6744: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:00:52.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:00:52 smithi115 conmon[90393]: 2022-05-14T03:00:52.590+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:00:53.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:00:52 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:00:53.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:00:52 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:00:53.910 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:00:53 smithi115 conmon[90393]: 2022-05-14T03:00:53.621+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:00:54.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:00:53 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:00:54.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:00:53 smithi047 ceph-mon[77147]: pgmap v6745: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:00:54.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:00:53 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:00:54.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:00:53 smithi115 ceph-mon[78593]: pgmap v6745: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:00:54.911 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:00:54 smithi115 conmon[90393]: 2022-05-14T03:00:54.659+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:00:54.912 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:00:54 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:00:54.912 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:00:54 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:00:54.912 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:00:54 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:00:55.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:00:54 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:00:55.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:00:54 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:00:55.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:00:54 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:00:56.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:00:55 smithi115 conmon[90393]: 2022-05-14T03:00:55.694+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:00:56.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:00:55 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:00:56.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:00:55 smithi115 ceph-mon[78593]: pgmap v6746: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:00:56.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:00:55 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:00:56.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:00:55 smithi047 ceph-mon[77147]: pgmap v6746: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:00:57.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:00:56 smithi115 conmon[90393]: 2022-05-14T03:00:56.741+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:00:57.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:00:56 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:00:57.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:00:56 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 12790 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:00:57.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:00:56 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:00:57.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:00:56 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 12790 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:00:58.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:00:57 smithi115 conmon[90393]: 2022-05-14T03:00:57.773+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:00:58.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:00:57 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:00:58.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:00:57 smithi115 ceph-mon[78593]: pgmap v6747: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:00:58.296 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:00:57 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:00:58.297 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:00:57 smithi047 ceph-mon[77147]: pgmap v6747: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:00:59.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:00:58 smithi115 conmon[90393]: 2022-05-14T03:00:58.790+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:00:59.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:00:58 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:00:59.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:00:58 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:00:59.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:00:58 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:00:59.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:00:58 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:00:59.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:00:58 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:00:59.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:00:58 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:01:00.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:00:59 smithi115 conmon[90393]: 2022-05-14T03:00:59.824+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:01:00.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:00:59 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:01:00.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:00:59 smithi115 ceph-mon[78593]: pgmap v6748: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-05-14T03:01:00.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:00:59 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:01:00.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:00:59 smithi047 ceph-mon[77147]: pgmap v6748: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-05-14T03:01:01.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:01:00 smithi115 conmon[90393]: 2022-05-14T03:01:00.788+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:01:01.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:01:00 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:01:01.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:01:00 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:01:01.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:01:01 smithi115 conmon[90393]: 2022-05-14T03:01:01.775+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:01:02.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:01:01 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:01:02.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:01:01 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:01:02.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:01:01 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:01:02.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:01:01 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 12795 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:01:02.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:01:01 smithi047 ceph-mon[77147]: pgmap v6749: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:01:02.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:01:01 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:01:02.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:01:01 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:01:02.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:01:01 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:01:02.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:01:01 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 12795 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:01:02.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:01:01 smithi115 ceph-mon[78593]: pgmap v6749: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:01:03.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:01:02 smithi115 conmon[90393]: 2022-05-14T03:01:02.812+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:01:03.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:01:02 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:01:03.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:01:02 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:01:04.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:01:03 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:01:04.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:01:03 smithi047 ceph-mon[77147]: pgmap v6750: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:01:04.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:01:03 smithi115 conmon[90393]: 2022-05-14T03:01:03.857+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:01:04.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:01:03 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:01:04.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:01:03 smithi115 ceph-mon[78593]: pgmap v6750: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:01:05.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:01:04 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:01:05.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:01:04 smithi115 conmon[90393]: 2022-05-14T03:01:04.891+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:01:05.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:01:04 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:01:06.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:01:06 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:01:06.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:01:06 smithi047 ceph-mon[77147]: pgmap v6751: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:01:06.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:01:05 smithi115 conmon[90393]: 2022-05-14T03:01:05.930+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:01:06.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:01:06 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:01:06.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:01:06 smithi115 ceph-mon[78593]: pgmap v6751: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:01:07.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:01:07 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:01:07.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:01:07 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 12800 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:01:07.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:01:06 smithi115 conmon[90393]: 2022-05-14T03:01:06.934+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:01:07.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:01:07 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:01:07.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:01:07 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 12800 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:01:08.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:01:08 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:01:08.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:01:08 smithi047 ceph-mon[77147]: pgmap v6752: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:01:08.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:01:07 smithi115 conmon[90393]: 2022-05-14T03:01:07.923+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:01:08.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:01:08 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:01:08.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:01:08 smithi115 ceph-mon[78593]: pgmap v6752: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:01:09.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:01:09 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:01:09.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:01:09 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:01:09.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:01:08 smithi115 conmon[90393]: 2022-05-14T03:01:08.944+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:01:10.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:01:10 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:01:10.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:01:10 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:01:10.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:01:10 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:01:10.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:01:10 smithi047 ceph-mon[77147]: pgmap v6753: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-05-14T03:01:10.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:01:09 smithi115 conmon[90393]: 2022-05-14T03:01:09.909+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:01:10.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:01:10 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:01:10.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:01:10 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:01:10.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:01:10 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:01:10.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:01:10 smithi115 ceph-mon[78593]: pgmap v6753: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-05-14T03:01:11.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:01:11 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:01:11.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:01:11 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T03:01:11.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:01:11 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T03:01:11.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:01:11 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T03:01:11.293 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:01:11 smithi047 conmon[78521]: 2022-05-14T03:01:11.151+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:01:11.293 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:01:11 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T03:01:11.293 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:01:11 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:01:11.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:01:11 smithi047 conmon[78521]: yield 2022-05-14T03:01:11.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:01:11 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:01:11.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:01:11 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:01:11.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:01:11 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:01:11.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:01:11 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T03:01:11.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:01:11 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:01:11.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:01:11 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T03:01:11.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:01:11 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:01:11.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:01:11 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T03:01:11.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:01:11 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:01:11.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:01:11 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T03:01:11.297 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:01:11 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:01:11.297 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:01:11 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T03:01:11.297 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:01:11 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:01:11.297 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:01:11 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:01:11.298 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:01:11 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:01:11.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:01:10 smithi115 conmon[90393]: 2022-05-14T03:01:10.933+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:01:11.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:01:11 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:01:11.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:01:11 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T03:01:11.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:01:11 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T03:01:11.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:01:11 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T03:01:12.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:01:12 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:01:12.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:01:12 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 12805 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:01:12.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:01:12 smithi047 ceph-mon[77147]: pgmap v6754: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:01:12.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:01:12 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:01:12.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T03:01:12.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:01:12.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T03:01:12.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:01:12.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:01:12.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:01:12.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T03:01:12.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:01:12.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T03:01:12.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:01:12.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T03:01:12.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:01:12.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T03:01:12.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:01:12.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T03:01:12.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:01:12.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:01:12.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:01:12.296 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:01:12 smithi047 ceph-mon[77147]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T03:01:12.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:01:11 smithi115 conmon[90393]: 2022-05-14T03:01:11.957+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:01:12.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:01:12 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:01:12.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:01:12 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 12805 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:01:12.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:01:12 smithi115 ceph-mon[78593]: pgmap v6754: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:01:12.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:01:12 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:01:12.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T03:01:12.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:01:12.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T03:01:12.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:01:12.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:01:12.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:01:12.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T03:01:12.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:01:12.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T03:01:12.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:01:12.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T03:01:12.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:01:12.365 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T03:01:12.365 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:01:12.365 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T03:01:12.365 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:01:12.366 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:01:12.366 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:01:12.366 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:01:12 smithi115 ceph-mon[78593]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T03:01:13.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:01:13 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:01:13.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:01:13 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:01:13.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:01:12 smithi115 conmon[90393]: 2022-05-14T03:01:12.950+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:01:14.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:01:14 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:01:14.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:01:14 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:01:14.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:01:14 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:01:14.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:01:14 smithi047 ceph-mon[77147]: pgmap v6755: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:01:14.293 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:01:14 smithi047 conmon[78521]: 2022-05-14T03:01:14.223+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:01:14.293 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:01:14 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T03:01:14.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:01:14 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:01:14.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:01:14 smithi047 conmon[78521]: yield 2022-05-14T03:01:14.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:01:14 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:01:14.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:01:14 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:01:14.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:01:14 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:01:14.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:01:14 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T03:01:14.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:01:14 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:01:14.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:01:14 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T03:01:14.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:01:14 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:01:14.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:01:14 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T03:01:14.297 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:01:14 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:01:14.301 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:01:14 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T03:01:14.302 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:01:14 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:01:14.302 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:01:14 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T03:01:14.303 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:01:14 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:01:14.303 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:01:14 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:01:14.303 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:01:14 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:01:14.303 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:01:14 smithi047 conmon[78521]: 2022-05-14T03:01:14.224+0000 7f534177b700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T03:01:14.303 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:01:14 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T03:01:14.304 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:01:14 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:01:14.304 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:01:14 smithi047 conmon[78521]: yield 2022-05-14T03:01:14.304 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:01:14 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:01:14.304 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:01:14 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:01:14.305 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:01:14 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:01:14.305 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:01:14 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T03:01:14.305 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:01:14 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:01:14.305 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:01:14 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T03:01:14.305 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:01:14 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:01:14.306 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:01:14 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T03:01:14.306 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:01:14 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:01:14.306 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:01:14 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T03:01:14.306 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:01:14 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:01:14.306 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:01:14 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T03:01:14.307 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:01:14 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:01:14.307 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:01:14 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:01:14.307 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:01:14 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:01:14.307 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:01:14 smithi047 conmon[78521]: 2022-05-14T03:01:14.308 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:01:14 smithi047 conmon[78521]: During handling of the above exception, another exception occurred: 2022-05-14T03:01:14.308 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:01:14 smithi047 conmon[78521]: 2022-05-14T03:01:14.308 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:01:14 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T03:01:14.308 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:01:14 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T03:01:14.309 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:01:14 smithi047 conmon[78521]: return f(*arg) 2022-05-14T03:01:14.309 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:01:14 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T03:01:14.309 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:01:14 smithi047 conmon[78521]: self._write_client_files(client_files, host) 2022-05-14T03:01:14.309 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:01:14 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T03:01:14.310 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:01:14 smithi047 conmon[78521]: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T03:01:14.310 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:01:14 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T03:01:14.310 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:01:14 smithi047 conmon[78521]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T03:01:14.310 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:01:14 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T03:01:14.310 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:01:14 smithi047 conmon[78521]: return self.event_loop.get_result(coro) 2022-05-14T03:01:14.311 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:01:14 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T03:01:14.311 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:01:14 smithi047 conmon[78521]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T03:01:14.311 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:01:14 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T03:01:14.311 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:01:14 smithi047 conmon[78521]: return self.__get_result() 2022-05-14T03:01:14.312 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:01:14 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T03:01:14.312 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:01:14 smithi047 conmon[78521]: raise self._exception 2022-05-14T03:01:14.312 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:01:14 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T03:01:14.312 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:01:14 smithi047 conmon[78521]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T03:01:14.312 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:01:14 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T03:01:14.313 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:01:14 smithi047 conmon[78521]: conn = await self._remote_connection(host, addr) 2022-05-14T03:01:14.313 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:01:14 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T03:01:14.313 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:01:14 smithi047 conmon[78521]: raise 2022-05-14T03:01:14.313 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:01:14 smithi047 conmon[78521]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T03:01:14.313 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:01:14 smithi047 conmon[78521]: self.gen.throw(type, value, traceback) 2022-05-14T03:01:14.314 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:01:14 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T03:01:14.314 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:01:14 smithi047 conmon[78521]: raise OrchestratorError(msg) 2022-05-14T03:01:14.314 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:01:14 smithi047 conmon[78521]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:01:14.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:01:13 smithi115 conmon[90393]: 2022-05-14T03:01:13.969+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:01:14.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:01:14 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:01:14.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:01:14 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:01:14.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:01:14 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:01:14.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:01:14 smithi115 ceph-mon[78593]: pgmap v6755: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:01:15.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:01:15 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:01:15.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:01:15 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T03:01:15.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:01:15 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T03:01:15.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:01:15 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:01:15.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T03:01:15.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:01:15.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T03:01:15.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:01:15.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:01:15.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:01:15.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T03:01:15.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:01:15.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T03:01:15.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:01:15.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T03:01:15.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:01:15.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T03:01:15.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:01:15.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T03:01:15.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:01:15.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:01:15.298 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:01:15.298 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:01:15 smithi047 ceph-mon[77147]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T03:01:15.298 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T03:01:15.299 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:01:15.299 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T03:01:15.299 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:01:15.300 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:01:15.300 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:01:15.301 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T03:01:15.301 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:01:15.302 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T03:01:15.302 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:01:15.302 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T03:01:15.302 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:01:15.303 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T03:01:15.303 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:01:15.303 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T03:01:15.304 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:01:15.304 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:01:15.304 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:01:15.304 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T03:01:15.304 INFO:journalctl@ceph.mon.b.smithi047.stdout: During handling of the above exception, another exception occurred: 2022-05-14T03:01:15.305 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T03:01:15.305 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T03:01:15.305 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T03:01:15.306 INFO:journalctl@ceph.mon.b.smithi047.stdout: return f(*arg) 2022-05-14T03:01:15.306 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T03:01:15.306 INFO:journalctl@ceph.mon.b.smithi047.stdout: self._write_client_files(client_files, host) 2022-05-14T03:01:15.306 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T03:01:15.307 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T03:01:15.307 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T03:01:15.307 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T03:01:15.307 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T03:01:15.308 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.event_loop.get_result(coro) 2022-05-14T03:01:15.308 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T03:01:15.308 INFO:journalctl@ceph.mon.b.smithi047.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T03:01:15.308 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T03:01:15.309 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.__get_result() 2022-05-14T03:01:15.309 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T03:01:15.309 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise self._exception 2022-05-14T03:01:15.309 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T03:01:15.310 INFO:journalctl@ceph.mon.b.smithi047.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T03:01:15.310 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T03:01:15.311 INFO:journalctl@ceph.mon.b.smithi047.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T03:01:15.311 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T03:01:15.311 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise 2022-05-14T03:01:15.311 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T03:01:15.312 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.gen.throw(type, value, traceback) 2022-05-14T03:01:15.312 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T03:01:15.312 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OrchestratorError(msg) 2022-05-14T03:01:15.312 INFO:journalctl@ceph.mon.b.smithi047.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:01:15.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:01:14 smithi115 conmon[90393]: 2022-05-14T03:01:14.957+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:01:15.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:01:15 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:01:15.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:01:15 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T03:01:15.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:01:15 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T03:01:15.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:01:15 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:01:15.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T03:01:15.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:01:15.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T03:01:15.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:01:15.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:01:15.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:01:15.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T03:01:15.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:01:15.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T03:01:15.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:01:15.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T03:01:15.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:01:15.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T03:01:15.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:01:15.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T03:01:15.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:01:15.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:01:15.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:01:15.364 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:01:15 smithi115 ceph-mon[78593]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T03:01:15.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T03:01:15.365 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:01:15.365 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T03:01:15.365 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:01:15.365 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:01:15.366 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:01:15.366 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T03:01:15.366 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:01:15.366 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T03:01:15.367 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:01:15.367 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T03:01:15.367 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:01:15.367 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T03:01:15.367 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:01:15.368 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T03:01:15.368 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:01:15.368 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:01:15.368 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:01:15.369 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T03:01:15.369 INFO:journalctl@ceph.mon.c.smithi115.stdout: During handling of the above exception, another exception occurred: 2022-05-14T03:01:15.369 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T03:01:15.369 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T03:01:15.370 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T03:01:15.370 INFO:journalctl@ceph.mon.c.smithi115.stdout: return f(*arg) 2022-05-14T03:01:15.370 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T03:01:15.370 INFO:journalctl@ceph.mon.c.smithi115.stdout: self._write_client_files(client_files, host) 2022-05-14T03:01:15.371 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T03:01:15.371 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T03:01:15.371 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T03:01:15.371 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T03:01:15.372 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T03:01:15.372 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.event_loop.get_result(coro) 2022-05-14T03:01:15.372 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T03:01:15.372 INFO:journalctl@ceph.mon.c.smithi115.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T03:01:15.372 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T03:01:15.373 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.__get_result() 2022-05-14T03:01:15.373 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T03:01:15.373 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise self._exception 2022-05-14T03:01:15.373 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T03:01:15.374 INFO:journalctl@ceph.mon.c.smithi115.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T03:01:15.374 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T03:01:15.374 INFO:journalctl@ceph.mon.c.smithi115.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T03:01:15.374 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T03:01:15.375 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise 2022-05-14T03:01:15.375 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T03:01:15.375 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.gen.throw(type, value, traceback) 2022-05-14T03:01:15.375 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T03:01:15.376 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OrchestratorError(msg) 2022-05-14T03:01:15.376 INFO:journalctl@ceph.mon.c.smithi115.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:01:16.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:01:15 smithi115 conmon[90393]: 2022-05-14T03:01:15.964+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:01:16.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:01:16 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:01:16.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:01:16 smithi115 ceph-mon[78593]: pgmap v6756: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:01:16.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:01:16 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:01:16.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:01:16 smithi047 ceph-mon[77147]: pgmap v6756: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:01:17.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:01:16 smithi115 conmon[90393]: 2022-05-14T03:01:16.969+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:01:17.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:01:17 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:01:17.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:01:17 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 12810 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:01:17.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:01:17 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:01:17.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:01:17 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 12810 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:01:18.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:01:17 smithi115 conmon[90393]: 2022-05-14T03:01:17.979+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:01:18.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:01:18 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:01:18.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:01:18 smithi115 ceph-mon[78593]: pgmap v6757: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:01:18.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:01:18 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:01:18.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:01:18 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:01:18.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:01:18 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:01:18.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:01:18 smithi047 ceph-mon[77147]: pgmap v6757: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:01:18.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:01:18 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:01:18.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:01:18 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:01:19.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:01:18 smithi115 conmon[90393]: 2022-05-14T03:01:18.967+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:01:19.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:01:19 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:01:19.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:01:19 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:01:20.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:01:20 smithi115 conmon[90393]: 2022-05-14T03:01:20.012+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:01:20.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:01:20 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:01:20.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:01:20 smithi115 ceph-mon[78593]: pgmap v6758: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-05-14T03:01:20.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:01:20 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:01:20.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:01:20 smithi047 ceph-mon[77147]: pgmap v6758: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-05-14T03:01:21.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:01:20 smithi115 conmon[90393]: 2022-05-14T03:01:20.998+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:01:21.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:01:21 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:01:21.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:01:21 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 12815 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:01:21.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:01:21 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:01:21.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:01:21 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 12815 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:01:22.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:01:22 smithi115 conmon[90393]: 2022-05-14T03:01:22.011+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:01:22.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:01:22 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:01:22.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:01:22 smithi115 ceph-mon[78593]: pgmap v6759: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:01:22.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:01:22 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:01:22.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:01:22 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:01:22.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:01:22 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:01:22.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:01:22 smithi047 ceph-mon[77147]: pgmap v6759: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:01:22.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:01:22 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:01:22.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:01:22 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:01:23.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:01:22 smithi115 conmon[90393]: 2022-05-14T03:01:22.993+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:01:23.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:01:23 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:01:23.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:01:23 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:01:24.362 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:01:24 smithi115 conmon[90393]: 2022-05-14T03:01:23.999+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:01:24.362 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:01:24 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:01:24.362 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:01:24 smithi115 ceph-mon[78593]: pgmap v6760: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:01:24.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:01:24 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:01:24.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:01:24 smithi047 ceph-mon[77147]: pgmap v6760: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:01:25.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:01:24 smithi115 conmon[90393]: 2022-05-14T03:01:24.966+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:01:25.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:01:25 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:01:25.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:01:25 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:01:26.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:01:25 smithi115 conmon[90393]: 2022-05-14T03:01:25.981+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:01:26.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:01:26 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:01:26.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:01:26 smithi115 ceph-mon[78593]: pgmap v6761: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:01:26.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:01:26 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 12820 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:01:26.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:01:26 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:01:26.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:01:26 smithi047 ceph-mon[77147]: pgmap v6761: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:01:26.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:01:26 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 12820 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:01:27.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:01:26 smithi115 conmon[90393]: 2022-05-14T03:01:26.942+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:01:27.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:01:27 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:01:27.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:01:27 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:01:28.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:01:27 smithi115 conmon[90393]: 2022-05-14T03:01:27.968+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:01:28.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:01:28 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:01:28.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:01:28 smithi115 ceph-mon[78593]: pgmap v6762: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:01:28.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:01:28 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:01:28.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:01:28 smithi047 ceph-mon[77147]: pgmap v6762: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:01:29.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:01:28 smithi115 conmon[90393]: 2022-05-14T03:01:28.967+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:01:29.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:01:29 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:01:29.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:01:29 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:01:30.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:01:29 smithi115 conmon[90393]: 2022-05-14T03:01:29.942+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:01:30.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:01:30 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:01:30.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:01:30 smithi115 ceph-mon[78593]: pgmap v6763: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-05-14T03:01:30.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:01:30 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:01:30.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:01:30 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:01:30.549 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:01:30 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:01:30.549 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:01:30 smithi047 ceph-mon[77147]: pgmap v6763: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-05-14T03:01:30.549 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:01:30 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:01:30.550 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:01:30 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:01:31.196 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:01:30 smithi115 conmon[90393]: 2022-05-14T03:01:30.942+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:01:31.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:01:31 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:01:31.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:01:31 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 12825 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:01:31.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:01:31 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:01:31.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:01:31 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 12825 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:01:32.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:01:31 smithi115 conmon[90393]: 2022-05-14T03:01:31.992+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:01:32.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:01:32 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:01:32.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:01:32 smithi115 ceph-mon[78593]: pgmap v6764: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:01:32.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:01:32 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:01:32.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:01:32 smithi047 ceph-mon[77147]: pgmap v6764: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:01:33.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:01:32 smithi115 conmon[90393]: 2022-05-14T03:01:32.962+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:01:33.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:01:33 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:01:33.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:01:33 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:01:34.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:01:33 smithi115 conmon[90393]: 2022-05-14T03:01:33.979+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:01:34.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:01:34 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:01:34.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:01:34 smithi115 ceph-mon[78593]: pgmap v6765: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:01:34.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:01:34 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:01:34.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:01:34 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:01:34.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:01:34 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:01:34.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:01:34 smithi047 ceph-mon[77147]: pgmap v6765: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:01:34.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:01:34 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:01:34.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:01:34 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:01:35.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:01:35 smithi115 conmon[90393]: 2022-05-14T03:01:35.013+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:01:35.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:01:35 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:01:35.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:01:35 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:01:36.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:01:36 smithi115 conmon[90393]: 2022-05-14T03:01:36.024+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:01:36.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:01:36 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:01:36.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:01:36 smithi115 ceph-mon[78593]: pgmap v6766: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:01:36.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:01:36 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 12830 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:01:36.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:01:36 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:01:36.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:01:36 smithi047 ceph-mon[77147]: pgmap v6766: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:01:36.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:01:36 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 12830 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:01:37.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:01:37 smithi115 conmon[90393]: 2022-05-14T03:01:37.017+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:01:37.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:01:37 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:01:37.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:01:37 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:01:38.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:01:38 smithi115 conmon[90393]: 2022-05-14T03:01:38.040+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:01:38.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:01:38 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:01:38.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:01:38 smithi115 ceph-mon[78593]: pgmap v6767: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:01:38.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:01:38 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:01:38.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:01:38 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:01:38.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:01:38 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:01:38.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:01:38 smithi047 ceph-mon[77147]: pgmap v6767: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:01:38.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:01:38 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:01:38.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:01:38 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:01:39.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:01:38 smithi115 conmon[90393]: 2022-05-14T03:01:38.991+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:01:39.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:01:39 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:01:39.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:01:39 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:01:40.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:01:40 smithi115 conmon[90393]: 2022-05-14T03:01:40.030+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:01:40.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:01:40 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:01:40.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:01:40 smithi115 ceph-mon[78593]: pgmap v6768: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-05-14T03:01:40.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:01:40 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:01:40.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:01:40 smithi047 ceph-mon[77147]: pgmap v6768: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-05-14T03:01:41.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:01:40 smithi115 conmon[90393]: 2022-05-14T03:01:40.985+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:01:41.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:01:41 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:01:41.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:01:41 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 12835 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:01:41.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:01:41 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:01:41.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:01:41 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 12835 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:01:42.229 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:01:41 smithi115 conmon[90393]: 2022-05-14T03:01:41.949+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:01:42.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:01:42 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:01:42.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:01:42 smithi047 ceph-mon[77147]: pgmap v6769: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:01:42.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:01:42 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:01:42.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:01:42 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:01:42.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:01:42 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:01:42.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:01:42 smithi115 ceph-mon[78593]: pgmap v6769: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:01:42.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:01:42 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:01:42.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:01:42 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:01:43.235 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:01:42 smithi115 conmon[90393]: 2022-05-14T03:01:42.906+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:01:43.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:01:43 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:01:43.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:01:43 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:01:44.233 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:01:43 smithi115 conmon[90393]: 2022-05-14T03:01:43.869+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:01:44.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:01:44 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:01:44.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:01:44 smithi047 ceph-mon[77147]: pgmap v6770: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:01:44.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:01:44 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:01:44.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:01:44 smithi115 ceph-mon[78593]: pgmap v6770: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:01:45.241 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:01:44 smithi115 conmon[90393]: 2022-05-14T03:01:44.911+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:01:45.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:01:45 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:01:45.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:01:45 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:01:46.239 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:01:45 smithi115 conmon[90393]: 2022-05-14T03:01:45.870+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:01:46.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:01:46 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:01:46.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:01:46 smithi047 ceph-mon[77147]: pgmap v6771: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:01:46.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:01:46 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 12840 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:01:46.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:01:46 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:01:46.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:01:46 smithi115 ceph-mon[78593]: pgmap v6771: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:01:46.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:01:46 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 12840 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:01:47.244 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:01:46 smithi115 conmon[90393]: 2022-05-14T03:01:46.905+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:01:47.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:01:47 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:01:47.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:01:47 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:01:48.246 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:01:47 smithi115 conmon[90393]: 2022-05-14T03:01:47.879+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:01:48.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:01:48 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:01:48.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:01:48 smithi047 ceph-mon[77147]: pgmap v6772: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:01:48.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:01:48 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:01:48.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:01:48 smithi115 ceph-mon[78593]: pgmap v6772: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:01:49.251 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:01:48 smithi115 conmon[90393]: 2022-05-14T03:01:48.884+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:01:49.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:01:49 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:01:49.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:01:49 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:01:50.252 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:01:49 smithi115 conmon[90393]: 2022-05-14T03:01:49.885+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:01:50.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:01:50 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:01:50.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:01:50 smithi047 ceph-mon[77147]: pgmap v6773: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-05-14T03:01:50.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:01:50 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:01:50.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:01:50 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:01:50.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:01:50 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:01:50.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:01:50 smithi115 ceph-mon[78593]: pgmap v6773: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-05-14T03:01:50.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:01:50 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:01:50.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:01:50 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:01:51.254 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:01:50 smithi115 conmon[90393]: 2022-05-14T03:01:50.916+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:01:51.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:01:51 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:01:51.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:01:51 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 12845 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:01:51.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:01:51 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:01:51.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:01:51 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 12845 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:01:52.254 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:01:51 smithi115 conmon[90393]: 2022-05-14T03:01:51.925+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:01:52.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:01:52 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:01:52.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:01:52 smithi047 ceph-mon[77147]: pgmap v6774: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:01:52.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:01:52 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:01:52.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:01:52 smithi115 ceph-mon[78593]: pgmap v6774: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:01:53.260 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:01:52 smithi115 conmon[90393]: 2022-05-14T03:01:52.897+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:01:53.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:01:53 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:01:53.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:01:53 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:01:54.261 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:01:53 smithi115 conmon[90393]: 2022-05-14T03:01:53.934+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:01:54.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:01:54 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:01:54.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:01:54 smithi047 ceph-mon[77147]: pgmap v6775: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:01:54.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:01:54 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:01:54.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:01:54 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:01:54.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:01:54 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:01:54.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:01:54 smithi115 ceph-mon[78593]: pgmap v6775: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:01:54.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:01:54 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:01:54.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:01:54 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:01:55.266 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:01:54 smithi115 conmon[90393]: 2022-05-14T03:01:54.886+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:01:55.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:01:55 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:01:55.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:01:55 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:01:56.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:01:55 smithi115 conmon[90393]: 2022-05-14T03:01:55.855+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:01:56.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:01:56 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:01:56.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:01:56 smithi047 ceph-mon[77147]: pgmap v6776: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:01:56.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:01:56 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 12850 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:01:56.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:01:56 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:01:56.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:01:56 smithi115 ceph-mon[78593]: pgmap v6776: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:01:56.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:01:56 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 12850 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:01:57.279 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:01:56 smithi115 conmon[90393]: 2022-05-14T03:01:56.887+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:01:57.546 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:01:57 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:01:57.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:01:57 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:01:58.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:01:57 smithi115 conmon[90393]: 2022-05-14T03:01:57.852+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:01:58.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:01:58 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:01:58.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:01:58 smithi047 ceph-mon[77147]: pgmap v6777: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:01:58.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:01:58 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:01:58.549 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:01:58 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:01:58.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:01:58 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:01:58.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:01:58 smithi115 ceph-mon[78593]: pgmap v6777: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:01:58.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:01:58 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:01:58.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:01:58 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:01:59.278 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:01:58 smithi115 conmon[90393]: 2022-05-14T03:01:58.873+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:01:59.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:01:59 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:01:59.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:01:59 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:02:00.280 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:01:59 smithi115 conmon[90393]: 2022-05-14T03:01:59.878+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:02:00.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:02:00 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:02:00.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:02:00 smithi047 ceph-mon[77147]: pgmap v6778: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-05-14T03:02:00.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:02:00 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:02:00.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:02:00 smithi115 ceph-mon[78593]: pgmap v6778: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-05-14T03:02:01.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:02:00 smithi115 conmon[90393]: 2022-05-14T03:02:00.855+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:02:01.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:02:01 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:02:01.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:02:01 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 12855 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:02:01.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:02:01 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:02:01.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:02:01 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 12855 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:02:02.285 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:02:01 smithi115 conmon[90393]: 2022-05-14T03:02:01.892+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:02:02.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:02:02 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:02:02.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:02:02 smithi047 ceph-mon[77147]: pgmap v6779: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:02:02.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:02:02 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:02:02.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:02:02 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:02:02.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:02:02 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:02:02.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:02:02 smithi115 ceph-mon[78593]: pgmap v6779: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:02:02.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:02:02 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:02:02.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:02:02 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:02:03.293 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:02:02 smithi115 conmon[90393]: 2022-05-14T03:02:02.866+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:02:03.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:02:03 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:02:03.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:02:03 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:02:04.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:02:03 smithi115 conmon[90393]: 2022-05-14T03:02:03.855+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:02:04.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:02:04 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:02:04.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:02:04 smithi115 ceph-mon[78593]: pgmap v6780: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:02:04.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:02:04 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:02:04.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:02:04 smithi047 ceph-mon[77147]: pgmap v6780: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:02:05.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:02:04 smithi115 conmon[90393]: 2022-05-14T03:02:04.852+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:02:05.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:02:05 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:02:05.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:02:05 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:02:06.302 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:02:05 smithi115 conmon[90393]: 2022-05-14T03:02:05.858+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:02:06.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:02:06 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:02:06.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:02:06 smithi115 ceph-mon[78593]: pgmap v6781: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:02:06.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:02:06 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 12860 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:02:06.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:02:06 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:02:06.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:02:06 smithi047 ceph-mon[77147]: pgmap v6781: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:02:06.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:02:06 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 12860 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:02:07.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:02:06 smithi115 conmon[90393]: 2022-05-14T03:02:06.824+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:02:07.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:02:07 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:02:07.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:02:07 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:02:08.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:02:07 smithi115 conmon[90393]: 2022-05-14T03:02:07.821+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:02:08.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:02:08 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:02:08.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:02:08 smithi115 ceph-mon[78593]: pgmap v6782: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:02:08.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:02:08 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:02:08.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:02:08 smithi047 ceph-mon[77147]: pgmap v6782: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:02:09.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:02:08 smithi115 conmon[90393]: 2022-05-14T03:02:08.779+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:02:09.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:02:09 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:02:09.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:02:09 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:02:10.110 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:02:09 smithi115 conmon[90393]: 2022-05-14T03:02:09.751+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:02:10.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:02:10 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:02:10.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:02:10 smithi115 ceph-mon[78593]: pgmap v6783: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-05-14T03:02:10.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:02:10 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:02:10.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:02:10 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:02:10.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:02:10 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:02:10.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:02:10 smithi047 ceph-mon[77147]: pgmap v6783: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-05-14T03:02:10.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:02:10 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:02:10.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:02:10 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:02:11.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:02:10 smithi115 conmon[90393]: 2022-05-14T03:02:10.738+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:02:11.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:02:11 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:02:11.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:02:11 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 12865 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:02:11.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:02:11 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:02:11.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:02:11 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 12865 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:02:12.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:02:11 smithi115 conmon[90393]: 2022-05-14T03:02:11.751+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:02:12.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:02:12 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:02:12.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:02:12 smithi115 ceph-mon[78593]: pgmap v6784: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:02:12.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:02:12 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:02:12.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:02:12 smithi047 ceph-mon[77147]: pgmap v6784: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:02:13.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:02:12 smithi115 conmon[90393]: 2022-05-14T03:02:12.721+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:02:13.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:02:13 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:02:13.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:02:13 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:02:14.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:02:13 smithi115 conmon[90393]: 2022-05-14T03:02:13.678+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:02:14.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:02:14 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:02:14.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:02:14 smithi047 ceph-mon[77147]: pgmap v6785: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:02:14.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:02:14 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:02:14.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:02:14 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:02:14.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:02:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T03:02:14.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:02:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T03:02:14.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:02:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T03:02:14.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:02:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T03:02:14.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:02:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T03:02:14.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:02:14 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:02:14.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:02:14 smithi115 ceph-mon[78593]: pgmap v6785: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:02:14.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:02:14 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:02:14.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:02:14 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:02:14.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:02:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T03:02:14.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:02:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T03:02:14.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:02:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T03:02:14.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:02:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T03:02:14.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:02:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T03:02:15.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:02:14 smithi115 conmon[90393]: 2022-05-14T03:02:14.629+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:02:15.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:02:15 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:02:15.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:02:15 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:02:16.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:02:15 smithi115 conmon[90393]: 2022-05-14T03:02:15.663+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:02:16.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:02:16 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:02:16.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:02:16 smithi115 ceph-mon[78593]: pgmap v6786: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:02:16.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:02:16 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 12870 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:02:16.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:02:16 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:02:16.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:02:16 smithi047 ceph-mon[77147]: pgmap v6786: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:02:16.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:02:16 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 12870 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:02:17.110 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:02:16 smithi115 conmon[90393]: 2022-05-14T03:02:16.663+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:02:17.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:02:17 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:02:17.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:02:17 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:02:17.793 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:02:17 smithi047 conmon[78521]: 2022-05-14T03:02:17.328+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:02:17.793 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:02:17 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T03:02:17.793 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:02:17 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:02:17.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:02:17 smithi047 conmon[78521]: yield 2022-05-14T03:02:17.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:02:17 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:02:17.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:02:17 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:02:17.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:02:17 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:02:17.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:02:17 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T03:02:17.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:02:17 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:02:17.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:02:17 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T03:02:17.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:02:17 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:02:17.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:02:17 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T03:02:17.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:02:17 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:02:17.797 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:02:17 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T03:02:17.797 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:02:17 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:02:17.797 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:02:17 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T03:02:17.798 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:02:17 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:02:17.798 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:02:17 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:02:17.799 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:02:17 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:02:18.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:02:17 smithi115 conmon[90393]: 2022-05-14T03:02:17.704+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:02:18.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:02:18 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:02:18.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:02:18 smithi115 ceph-mon[78593]: pgmap v6787: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:02:18.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:02:18 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:02:18.609 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T03:02:18.609 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:02:18.609 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T03:02:18.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:02:18.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:02:18.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:02:18.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T03:02:18.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:02:18.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T03:02:18.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:02:18.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T03:02:18.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:02:18.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T03:02:18.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:02:18.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T03:02:18.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:02:18.613 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:02:18.613 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:02:18.613 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:02:18 smithi115 ceph-mon[78593]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T03:02:18.613 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:02:18 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:02:18.614 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:02:18 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:02:18.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:02:18 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:02:18.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:02:18 smithi047 ceph-mon[77147]: pgmap v6787: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:02:18.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:02:18 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:02:18.793 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T03:02:18.793 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:02:18.793 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T03:02:18.793 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:02:18.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:02:18.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:02:18.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T03:02:18.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:02:18.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T03:02:18.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:02:18.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T03:02:18.798 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:02:18.799 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T03:02:18.799 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:02:18.800 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T03:02:18.800 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:02:18.801 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:02:18.802 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:02:18.802 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:02:18 smithi047 ceph-mon[77147]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T03:02:18.803 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:02:18 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:02:18.803 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:02:18 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:02:19.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:02:18 smithi115 conmon[90393]: 2022-05-14T03:02:18.688+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:02:19.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:02:19 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:02:19.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:02:19 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:02:20.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:02:19 smithi115 conmon[90393]: 2022-05-14T03:02:19.715+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:02:20.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:02:20 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:02:20.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:02:20 smithi115 ceph-mon[78593]: pgmap v6788: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-05-14T03:02:20.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:02:20 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:02:20.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:02:20 smithi047 ceph-mon[77147]: pgmap v6788: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-05-14T03:02:20.792 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:02:20 smithi047 conmon[78521]: 2022-05-14T03:02:20.400+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:02:20.793 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:02:20 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T03:02:20.793 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:02:20 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:02:20.793 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:02:20 smithi047 conmon[78521]: yield 2022-05-14T03:02:20.793 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:02:20 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:02:20.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:02:20 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:02:20.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:02:20 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:02:20.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:02:20 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T03:02:20.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:02:20 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:02:20.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:02:20 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T03:02:20.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:02:20 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:02:20.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:02:20 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T03:02:20.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:02:20 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:02:20.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:02:20 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T03:02:20.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:02:20 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:02:20.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:02:20 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T03:02:20.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:02:20 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:02:20.797 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:02:20 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:02:20.797 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:02:20 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:02:20.797 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:02:20 smithi047 conmon[78521]: 2022-05-14T03:02:20.401+0000 7f5345f84700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T03:02:20.797 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:02:20 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T03:02:20.797 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:02:20 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:02:20.798 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:02:20 smithi047 conmon[78521]: yield 2022-05-14T03:02:20.798 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:02:20 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:02:20.798 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:02:20 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:02:20.798 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:02:20 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:02:20.799 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:02:20 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T03:02:20.799 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:02:20 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:02:20.799 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:02:20 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T03:02:20.799 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:02:20 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:02:20.800 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:02:20 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T03:02:20.800 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:02:20 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:02:20.800 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:02:20 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T03:02:20.800 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:02:20 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:02:20.800 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:02:20 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T03:02:20.801 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:02:20 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:02:20.801 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:02:20 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:02:20.801 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:02:20 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:02:20.801 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:02:20 smithi047 conmon[78521]: 2022-05-14T03:02:20.802 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:02:20 smithi047 conmon[78521]: During handling of the above exception, another exception occurred: 2022-05-14T03:02:20.802 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:02:20 smithi047 conmon[78521]: 2022-05-14T03:02:20.802 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:02:20 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T03:02:20.802 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:02:20 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T03:02:20.802 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:02:20 smithi047 conmon[78521]: return f(*arg) 2022-05-14T03:02:20.803 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:02:20 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T03:02:20.803 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:02:20 smithi047 conmon[78521]: self._write_client_files(client_files, host) 2022-05-14T03:02:20.803 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:02:20 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T03:02:20.803 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:02:20 smithi047 conmon[78521]: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T03:02:20.803 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:02:20 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T03:02:20.804 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:02:20 smithi047 conmon[78521]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T03:02:20.804 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:02:20 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T03:02:20.804 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:02:20 smithi047 conmon[78521]: return self.event_loop.get_result(coro) 2022-05-14T03:02:20.804 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:02:20 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T03:02:20.805 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:02:20 smithi047 conmon[78521]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T03:02:20.805 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:02:20 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T03:02:20.805 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:02:20 smithi047 conmon[78521]: return self.__get_result() 2022-05-14T03:02:20.805 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:02:20 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T03:02:20.806 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:02:20 smithi047 conmon[78521]: raise self._exception 2022-05-14T03:02:20.806 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:02:20 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T03:02:20.806 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:02:20 smithi047 conmon[78521]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T03:02:20.807 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:02:20 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T03:02:20.807 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:02:20 smithi047 conmon[78521]: conn = await self._remote_connection(host, addr) 2022-05-14T03:02:20.807 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:02:20 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T03:02:20.808 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:02:20 smithi047 conmon[78521]: raise 2022-05-14T03:02:20.808 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:02:20 smithi047 conmon[78521]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T03:02:20.808 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:02:20 smithi047 conmon[78521]: self.gen.throw(type, value, traceback) 2022-05-14T03:02:20.808 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:02:20 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T03:02:20.808 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:02:20 smithi047 conmon[78521]: raise OrchestratorError(msg) 2022-05-14T03:02:20.809 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:02:20 smithi047 conmon[78521]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:02:21.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:02:20 smithi115 conmon[90393]: 2022-05-14T03:02:20.758+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:02:21.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:02:21 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:02:21.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:02:21 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:02:21.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T03:02:21.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:02:21.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T03:02:21.613 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:02:21.613 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:02:21.613 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:02:21.614 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T03:02:21.614 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:02:21.614 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T03:02:21.614 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:02:21.614 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T03:02:21.615 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:02:21.615 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T03:02:21.615 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:02:21.615 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T03:02:21.616 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:02:21.616 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:02:21.616 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:02:21.616 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:02:21 smithi115 ceph-mon[78593]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T03:02:21.616 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T03:02:21.617 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:02:21.617 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T03:02:21.617 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:02:21.617 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:02:21.618 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:02:21.618 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T03:02:21.618 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:02:21.618 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T03:02:21.618 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:02:21.619 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T03:02:21.619 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:02:21.619 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T03:02:21.619 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:02:21.620 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T03:02:21.620 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:02:21.620 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:02:21.620 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:02:21.621 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T03:02:21.621 INFO:journalctl@ceph.mon.c.smithi115.stdout: During handling of the above exception, another exception occurred: 2022-05-14T03:02:21.621 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T03:02:21.621 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T03:02:21.621 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T03:02:21.622 INFO:journalctl@ceph.mon.c.smithi115.stdout: return f(*arg) 2022-05-14T03:02:21.622 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T03:02:21.622 INFO:journalctl@ceph.mon.c.smithi115.stdout: self._write_client_files(client_files, host) 2022-05-14T03:02:21.622 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T03:02:21.623 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T03:02:21.623 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T03:02:21.623 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T03:02:21.623 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T03:02:21.624 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.event_loop.get_result(coro) 2022-05-14T03:02:21.624 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T03:02:21.624 INFO:journalctl@ceph.mon.c.smithi115.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T03:02:21.624 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T03:02:21.624 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.__get_result() 2022-05-14T03:02:21.625 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T03:02:21.625 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise self._exception 2022-05-14T03:02:21.625 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T03:02:21.625 INFO:journalctl@ceph.mon.c.smithi115.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T03:02:21.626 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T03:02:21.626 INFO:journalctl@ceph.mon.c.smithi115.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T03:02:21.626 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T03:02:21.626 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise 2022-05-14T03:02:21.626 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T03:02:21.627 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.gen.throw(type, value, traceback) 2022-05-14T03:02:21.627 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T03:02:21.627 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OrchestratorError(msg) 2022-05-14T03:02:21.627 INFO:journalctl@ceph.mon.c.smithi115.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:02:21.628 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:02:21 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 12875 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:02:21.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:02:21 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:02:21.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:02:21 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:02:21.792 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T03:02:21.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:02:21.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T03:02:21.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:02:21.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:02:21.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:02:21.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T03:02:21.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:02:21.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T03:02:21.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:02:21.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T03:02:21.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:02:21.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T03:02:21.797 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:02:21.798 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T03:02:21.798 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:02:21.798 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:02:21.799 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:02:21.799 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:02:21 smithi047 ceph-mon[77147]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T03:02:21.799 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T03:02:21.800 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:02:21.801 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T03:02:21.802 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:02:21.802 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:02:21.802 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:02:21.802 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T03:02:21.803 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:02:21.803 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T03:02:21.803 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:02:21.803 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T03:02:21.804 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:02:21.804 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T03:02:21.804 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:02:21.804 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T03:02:21.805 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:02:21.805 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:02:21.805 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:02:21.805 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T03:02:21.806 INFO:journalctl@ceph.mon.b.smithi047.stdout: During handling of the above exception, another exception occurred: 2022-05-14T03:02:21.806 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T03:02:21.806 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T03:02:21.806 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T03:02:21.807 INFO:journalctl@ceph.mon.b.smithi047.stdout: return f(*arg) 2022-05-14T03:02:21.807 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T03:02:21.807 INFO:journalctl@ceph.mon.b.smithi047.stdout: self._write_client_files(client_files, host) 2022-05-14T03:02:21.807 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T03:02:21.808 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T03:02:21.808 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T03:02:21.808 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T03:02:21.808 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T03:02:21.809 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.event_loop.get_result(coro) 2022-05-14T03:02:21.809 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T03:02:21.809 INFO:journalctl@ceph.mon.b.smithi047.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T03:02:21.809 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T03:02:21.809 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.__get_result() 2022-05-14T03:02:21.810 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T03:02:21.810 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise self._exception 2022-05-14T03:02:21.810 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T03:02:21.810 INFO:journalctl@ceph.mon.b.smithi047.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T03:02:21.811 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T03:02:21.811 INFO:journalctl@ceph.mon.b.smithi047.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T03:02:21.811 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T03:02:21.811 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise 2022-05-14T03:02:21.812 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T03:02:21.812 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.gen.throw(type, value, traceback) 2022-05-14T03:02:21.812 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T03:02:21.812 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OrchestratorError(msg) 2022-05-14T03:02:21.813 INFO:journalctl@ceph.mon.b.smithi047.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:02:21.813 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:02:21 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 12875 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:02:22.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:02:21 smithi115 conmon[90393]: 2022-05-14T03:02:21.765+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:02:22.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:02:22 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:02:22.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:02:22 smithi115 ceph-mon[78593]: pgmap v6789: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2022-05-14T03:02:22.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:02:22 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:02:22.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:02:22 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:02:22.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:02:22 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:02:22.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:02:22 smithi047 ceph-mon[77147]: pgmap v6789: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2022-05-14T03:02:22.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:02:22 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:02:22.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:02:22 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:02:23.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:02:22 smithi115 conmon[90393]: 2022-05-14T03:02:22.739+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:02:23.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:02:23 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:02:23.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:02:23 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:02:24.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:02:23 smithi115 conmon[90393]: 2022-05-14T03:02:23.765+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:02:24.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:02:24 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:02:24.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:02:24 smithi115 ceph-mon[78593]: pgmap v6790: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:02:24.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:02:24 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:02:24.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:02:24 smithi047 ceph-mon[77147]: pgmap v6790: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:02:25.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:02:24 smithi115 conmon[90393]: 2022-05-14T03:02:24.723+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:02:25.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:02:25 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:02:25.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:02:25 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:02:26.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:02:25 smithi115 conmon[90393]: 2022-05-14T03:02:25.697+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:02:26.722 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:02:26 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:02:26.723 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:02:26 smithi115 ceph-mon[78593]: pgmap v6791: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:02:26.723 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:02:26 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 12880 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:02:26.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:02:26 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:02:26.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:02:26 smithi047 ceph-mon[77147]: pgmap v6791: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:02:26.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:02:26 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 12880 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:02:27.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:02:26 smithi115 conmon[90393]: 2022-05-14T03:02:26.721+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:02:27.729 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:02:27 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:02:27.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:02:27 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:02:28.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:02:27 smithi115 conmon[90393]: 2022-05-14T03:02:27.727+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:02:28.727 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:02:28 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:02:28.727 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:02:28 smithi115 ceph-mon[78593]: pgmap v6792: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:02:28.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:02:28 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:02:28.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:02:28 smithi047 ceph-mon[77147]: pgmap v6792: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:02:29.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:02:28 smithi115 conmon[90393]: 2022-05-14T03:02:28.725+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:02:29.683 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:02:29 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:02:29.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:02:29 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:02:30.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:02:29 smithi115 conmon[90393]: 2022-05-14T03:02:29.681+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:02:30.639 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:02:30 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:02:30.639 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:02:30 smithi115 ceph-mon[78593]: pgmap v6793: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-05-14T03:02:30.639 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:02:30 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:02:30.640 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:02:30 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:02:30.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:02:30 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:02:30.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:02:30 smithi047 ceph-mon[77147]: pgmap v6793: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-05-14T03:02:30.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:02:30 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:02:30.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:02:30 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:02:31.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:02:30 smithi115 conmon[90393]: 2022-05-14T03:02:30.637+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:02:31.665 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:02:31 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:02:31.666 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:02:31 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 12885 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:02:31.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:02:31 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:02:31.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:02:31 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 12885 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:02:32.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:02:31 smithi115 conmon[90393]: 2022-05-14T03:02:31.663+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:02:32.624 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:02:32 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:02:32.624 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:02:32 smithi115 ceph-mon[78593]: pgmap v6794: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:02:32.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:02:32 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:02:32.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:02:32 smithi047 ceph-mon[77147]: pgmap v6794: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:02:33.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:02:32 smithi115 conmon[90393]: 2022-05-14T03:02:32.623+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:02:33.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:02:33 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:02:33.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:02:33 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:02:33.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:02:33 smithi115 conmon[90393]: 2022-05-14T03:02:33.626+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:02:34.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:02:34 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:02:34.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:02:34 smithi047 ceph-mon[77147]: pgmap v6795: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:02:34.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:02:34 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:02:34.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:02:34 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:02:34.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:02:34 smithi115 conmon[90393]: 2022-05-14T03:02:34.589+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:02:34.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:02:34 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:02:34.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:02:34 smithi115 ceph-mon[78593]: pgmap v6795: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:02:34.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:02:34 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:02:34.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:02:34 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:02:35.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:02:35 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:02:35.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:02:35 smithi115 conmon[90393]: 2022-05-14T03:02:35.565+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:02:35.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:02:35 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:02:36.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:02:36 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:02:36.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:02:36 smithi047 ceph-mon[77147]: pgmap v6796: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:02:36.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:02:36 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 12890 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:02:36.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:02:36 smithi115 conmon[90393]: 2022-05-14T03:02:36.596+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:02:36.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:02:36 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:02:36.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:02:36 smithi115 ceph-mon[78593]: pgmap v6796: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:02:36.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:02:36 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 12890 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:02:37.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:02:37 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:02:37.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:02:37 smithi115 conmon[90393]: 2022-05-14T03:02:37.581+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:02:37.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:02:37 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:02:38.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:02:38 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:02:38.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:02:38 smithi047 ceph-mon[77147]: pgmap v6797: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:02:38.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:02:38 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:02:38.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:02:38 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:02:38.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:02:38 smithi115 conmon[90393]: 2022-05-14T03:02:38.623+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:02:38.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:02:38 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:02:38.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:02:38 smithi115 ceph-mon[78593]: pgmap v6797: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:02:38.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:02:38 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:02:38.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:02:38 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:02:39.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:02:39 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:02:39.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:02:39 smithi115 conmon[90393]: 2022-05-14T03:02:39.641+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:02:39.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:02:39 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:02:40.656 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:02:40 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:02:40.656 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:02:40 smithi115 ceph-mon[78593]: pgmap v6798: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-05-14T03:02:40.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:02:40 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:02:40.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:02:40 smithi047 ceph-mon[77147]: pgmap v6798: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-05-14T03:02:41.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:02:40 smithi115 conmon[90393]: 2022-05-14T03:02:40.653+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:02:41.700 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:02:41 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:02:41.700 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:02:41 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 12895 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:02:41.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:02:41 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:02:41.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:02:41 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 12895 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:02:42.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:02:41 smithi115 conmon[90393]: 2022-05-14T03:02:41.698+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:02:42.720 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:02:42 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:02:42.720 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:02:42 smithi115 ceph-mon[78593]: pgmap v6799: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:02:42.720 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:02:42 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:02:42.721 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:02:42 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:02:42.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:02:42 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:02:42.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:02:42 smithi047 ceph-mon[77147]: pgmap v6799: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:02:42.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:02:42 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:02:42.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:02:42 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:02:43.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:02:42 smithi115 conmon[90393]: 2022-05-14T03:02:42.718+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:02:43.677 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:02:43 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:02:43.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:02:43 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:02:44.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:02:43 smithi115 conmon[90393]: 2022-05-14T03:02:43.675+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:02:44.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:02:44 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:02:44.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:02:44 smithi047 ceph-mon[77147]: pgmap v6800: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:02:44.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:02:44 smithi115 conmon[90393]: 2022-05-14T03:02:44.645+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:02:44.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:02:44 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:02:44.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:02:44 smithi115 ceph-mon[78593]: pgmap v6800: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:02:45.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:02:45 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:02:45.860 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:02:45 smithi115 conmon[90393]: 2022-05-14T03:02:45.616+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:02:45.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:02:45 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:02:46.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:02:46 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:02:46.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:02:46 smithi047 ceph-mon[77147]: pgmap v6801: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:02:46.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:02:46 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 12900 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:02:46.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:02:46 smithi115 conmon[90393]: 2022-05-14T03:02:46.612+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:02:46.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:02:46 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:02:46.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:02:46 smithi115 ceph-mon[78593]: pgmap v6801: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:02:46.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:02:46 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 12900 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:02:47.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:02:47 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:02:47.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:02:47 smithi115 conmon[90393]: 2022-05-14T03:02:47.610+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:02:47.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:02:47 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:02:48.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:02:48 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:02:48.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:02:48 smithi047 ceph-mon[77147]: pgmap v6802: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:02:48.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:02:48 smithi115 conmon[90393]: 2022-05-14T03:02:48.657+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:02:48.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:02:48 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:02:48.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:02:48 smithi115 ceph-mon[78593]: pgmap v6802: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:02:49.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:02:49 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:02:49.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:02:49 smithi115 conmon[90393]: 2022-05-14T03:02:49.673+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:02:49.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:02:49 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:02:50.700 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:02:50 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:02:50.700 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:02:50 smithi115 ceph-mon[78593]: pgmap v6803: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-05-14T03:02:50.701 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:02:50 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:02:50.701 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:02:50 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:02:50.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:02:50 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:02:50.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:02:50 smithi047 ceph-mon[77147]: pgmap v6803: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-05-14T03:02:50.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:02:50 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:02:50.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:02:50 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:02:51.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:02:50 smithi115 conmon[90393]: 2022-05-14T03:02:50.697+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:02:51.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:02:51 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:02:51.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:02:51 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 12905 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:02:51.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:02:51 smithi115 conmon[90393]: 2022-05-14T03:02:51.650+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:02:51.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:02:51 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:02:51.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:02:51 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 12905 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:02:52.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:02:52 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:02:52.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:02:52 smithi047 ceph-mon[77147]: pgmap v6804: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:02:52.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:02:52 smithi115 conmon[90393]: 2022-05-14T03:02:52.658+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:02:52.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:02:52 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:02:52.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:02:52 smithi115 ceph-mon[78593]: pgmap v6804: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:02:53.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:02:53 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:02:53.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:02:53 smithi115 conmon[90393]: 2022-05-14T03:02:53.679+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:02:53.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:02:53 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:02:54.725 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:02:54 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:02:54.725 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:02:54 smithi115 ceph-mon[78593]: pgmap v6805: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:02:54.725 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:02:54 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:02:54.726 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:02:54 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:02:54.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:02:54 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:02:54.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:02:54 smithi047 ceph-mon[77147]: pgmap v6805: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:02:54.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:02:54 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:02:54.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:02:54 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:02:55.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:02:54 smithi115 conmon[90393]: 2022-05-14T03:02:54.721+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:02:55.756 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:02:55 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:02:55.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:02:55 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:02:56.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:02:55 smithi115 conmon[90393]: 2022-05-14T03:02:55.754+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:02:56.752 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:02:56 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:02:56.753 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:02:56 smithi115 ceph-mon[78593]: pgmap v6806: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:02:56.753 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:02:56 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 12910 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:02:56.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:02:56 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:02:56.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:02:56 smithi047 ceph-mon[77147]: pgmap v6806: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:02:56.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:02:56 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 12910 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:02:57.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:02:56 smithi115 conmon[90393]: 2022-05-14T03:02:56.750+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:02:57.747 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:02:57 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:02:57.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:02:57 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:02:58.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:02:57 smithi115 conmon[90393]: 2022-05-14T03:02:57.745+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:02:58.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:02:58 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:02:58.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:02:58 smithi047 ceph-mon[77147]: pgmap v6807: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:02:58.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:02:58 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:02:58.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:02:58 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:02:58.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:02:58 smithi115 conmon[90393]: 2022-05-14T03:02:58.704+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:02:58.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:02:58 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:02:58.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:02:58 smithi115 ceph-mon[78593]: pgmap v6807: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:02:58.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:02:58 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:02:58.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:02:58 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:02:59.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:02:59 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:02:59.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:02:59 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:02:59.861 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:02:59 smithi115 conmon[90393]: 2022-05-14T03:02:59.661+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:03:00.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:03:00 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:03:00.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:03:00 smithi047 ceph-mon[77147]: pgmap v6808: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-05-14T03:03:00.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:03:00 smithi115 conmon[90393]: 2022-05-14T03:03:00.623+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:03:00.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:03:00 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:03:00.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:03:00 smithi115 ceph-mon[78593]: pgmap v6808: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-05-14T03:03:01.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:03:01 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:03:01.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:03:01 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 12915 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:03:01.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:03:01 smithi115 conmon[90393]: 2022-05-14T03:03:01.664+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:03:01.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:03:01 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:03:01.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:03:01 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 12915 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:03:02.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:03:02 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:03:02.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:03:02 smithi047 ceph-mon[77147]: pgmap v6809: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:03:02.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:03:02 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:03:02.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:03:02 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:03:02.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:03:02 smithi115 conmon[90393]: 2022-05-14T03:03:02.616+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:03:02.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:03:02 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:03:02.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:03:02 smithi115 ceph-mon[78593]: pgmap v6809: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:03:02.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:03:02 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:03:02.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:03:02 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:03:03.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:03:03 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:03:03.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:03:03 smithi115 conmon[90393]: 2022-05-14T03:03:03.613+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:03:03.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:03:03 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:03:04.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:03:04 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:03:04.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:03:04 smithi047 ceph-mon[77147]: pgmap v6810: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:03:04.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:03:04 smithi115 conmon[90393]: 2022-05-14T03:03:04.627+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:03:04.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:03:04 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:03:04.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:03:04 smithi115 ceph-mon[78593]: pgmap v6810: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:03:05.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:03:05 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:03:05.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:03:05 smithi115 conmon[90393]: 2022-05-14T03:03:05.650+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:03:05.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:03:05 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:03:06.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:03:06 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:03:06.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:03:06 smithi047 ceph-mon[77147]: pgmap v6811: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:03:06.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:03:06 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 12920 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:03:06.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:03:06 smithi115 conmon[90393]: 2022-05-14T03:03:06.640+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:03:06.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:03:06 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:03:06.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:03:06 smithi115 ceph-mon[78593]: pgmap v6811: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:03:06.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:03:06 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 12920 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:03:07.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:03:07 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:03:07.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:03:07 smithi115 conmon[90393]: 2022-05-14T03:03:07.622+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:03:07.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:03:07 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:03:08.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:03:08 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:03:08.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:03:08 smithi047 ceph-mon[77147]: pgmap v6812: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:03:08.861 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:03:08 smithi115 conmon[90393]: 2022-05-14T03:03:08.630+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:03:08.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:03:08 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:03:08.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:03:08 smithi115 ceph-mon[78593]: pgmap v6812: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:03:09.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:03:09 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:03:09.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:03:09 smithi115 conmon[90393]: 2022-05-14T03:03:09.593+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:03:09.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:03:09 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:03:10.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:03:10 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:03:10.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:03:10 smithi047 ceph-mon[77147]: pgmap v6813: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-05-14T03:03:10.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:03:10 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:03:10.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:03:10 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:03:10.795 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:03:10 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:03:10.863 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:03:10 smithi115 conmon[90393]: 2022-05-14T03:03:10.609+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:03:10.864 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:03:10 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:03:10.864 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:03:10 smithi115 ceph-mon[78593]: pgmap v6813: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-05-14T03:03:10.864 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:03:10 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:03:10.865 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:03:10 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:03:10.865 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:03:10 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:03:11.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:03:11 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:03:11.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:03:11 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 12925 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:03:11.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:03:11 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:03:11.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:03:11 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 12925 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:03:11.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:03:11 smithi115 conmon[90393]: 2022-05-14T03:03:11.635+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:03:12.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:03:12 smithi047 ceph-mon[77147]: pgmap v6814: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2022-05-14T03:03:12.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:03:12 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:03:12.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:03:12 smithi115 conmon[90393]: 2022-05-14T03:03:12.642+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:03:12.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:03:12 smithi115 ceph-mon[78593]: pgmap v6814: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2022-05-14T03:03:12.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:03:12 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:03:13.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:03:13 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:03:13.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:03:13 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:03:13.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:03:13 smithi115 conmon[90393]: 2022-05-14T03:03:13.631+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:03:14.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:03:14 smithi047 ceph-mon[77147]: pgmap v6815: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:03:14.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:03:14 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:03:14.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:03:14 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:03:14.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:03:14 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:03:14.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:03:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T03:03:14.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:03:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T03:03:14.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:03:14 smithi115 conmon[90393]: 2022-05-14T03:03:14.615+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:03:14.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:03:14 smithi115 ceph-mon[78593]: pgmap v6815: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:03:14.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:03:14 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:03:14.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:03:14 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:03:14.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:03:14 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:03:14.862 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:03:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T03:03:14.862 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:03:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T03:03:15.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:03:15 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:03:15.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:03:15 smithi115 conmon[90393]: 2022-05-14T03:03:15.658+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:03:15.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:03:15 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:03:16.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:03:16 smithi047 ceph-mon[77147]: pgmap v6816: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 7 op/s 2022-05-14T03:03:16.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:03:16 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:03:16.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:03:16 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 12930 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:03:16.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:03:16 smithi115 conmon[90393]: 2022-05-14T03:03:16.621+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:03:16.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:03:16 smithi115 ceph-mon[78593]: pgmap v6816: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 7 op/s 2022-05-14T03:03:16.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:03:16 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:03:16.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:03:16 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 12930 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:03:17.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:03:17 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:03:17.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:03:17 smithi115 conmon[90393]: 2022-05-14T03:03:17.656+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:03:17.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:03:17 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:03:18.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:03:18 smithi047 ceph-mon[77147]: pgmap v6817: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 4 op/s 2022-05-14T03:03:18.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:03:18 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:03:18.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:03:18 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:03:18.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:03:18 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:03:18.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:03:18 smithi115 conmon[90393]: 2022-05-14T03:03:18.684+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:03:18.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:03:18 smithi115 ceph-mon[78593]: pgmap v6817: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 4 op/s 2022-05-14T03:03:18.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:03:18 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:03:18.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:03:18 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:03:18.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:03:18 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:03:19.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:03:19 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:03:19.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:03:19 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:03:19.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:03:19 smithi115 conmon[90393]: 2022-05-14T03:03:19.718+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:03:20.715 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:03:20 smithi115 ceph-mon[78593]: pgmap v6818: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 7 op/s 2022-05-14T03:03:20.716 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:03:20 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:03:20.716 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:03:20 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T03:03:20.716 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:03:20 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T03:03:20.716 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:03:20 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T03:03:20.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:03:20 smithi047 ceph-mon[77147]: pgmap v6818: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 7 op/s 2022-05-14T03:03:20.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:03:20 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:03:20.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:03:20 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T03:03:20.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:03:20 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T03:03:20.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:03:20 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T03:03:21.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:03:20 smithi115 conmon[90393]: 2022-05-14T03:03:20.713+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:03:21.292 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:03:20 smithi047 conmon[78521]: 2022-05-14T03:03:20.880+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:03:21.292 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:03:20 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T03:03:21.292 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:03:20 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:03:21.292 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:03:20 smithi047 conmon[78521]: yield 2022-05-14T03:03:21.293 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:03:20 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:03:21.293 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:03:20 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:03:21.293 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:03:20 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:03:21.293 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:03:20 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T03:03:21.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:03:20 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:03:21.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:03:20 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T03:03:21.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:03:20 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:03:21.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:03:20 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T03:03:21.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:03:20 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:03:21.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:03:20 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T03:03:21.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:03:20 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:03:21.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:03:20 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T03:03:21.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:03:20 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:03:21.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:03:20 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:03:21.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:03:20 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:03:21.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:03:21 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:03:21.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:03:21 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 12935 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:03:21.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:03:21 smithi115 conmon[90393]: 2022-05-14T03:03:21.701+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:03:21.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:03:21 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:03:21.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:03:21 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 12935 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:03:22.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:03:22 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:03:22.792 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T03:03:22.793 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:03:22.793 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T03:03:22.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:03:22.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:03:22.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:03:22.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T03:03:22.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:03:22.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T03:03:22.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:03:22.797 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T03:03:22.797 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:03:22.797 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T03:03:22.797 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:03:22.798 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T03:03:22.798 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:03:22.798 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:03:22.798 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:03:22.799 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:03:22 smithi047 ceph-mon[77147]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T03:03:22.799 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:03:22 smithi047 ceph-mon[77147]: pgmap v6819: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 6 op/s 2022-05-14T03:03:22.799 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:03:22 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:03:22.799 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:03:22 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:03:22.800 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:03:22 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:03:22.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:03:22 smithi115 conmon[90393]: 2022-05-14T03:03:22.714+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:03:22.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:03:22 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:03:22.859 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T03:03:22.860 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:03:22.860 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T03:03:22.860 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:03:22.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:03:22.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:03:22.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T03:03:22.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:03:22.863 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T03:03:22.863 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:03:22.863 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T03:03:22.864 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:03:22.864 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T03:03:22.864 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:03:22.864 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T03:03:22.864 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:03:22.865 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:03:22.865 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:03:22.865 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:03:22 smithi115 ceph-mon[78593]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T03:03:22.865 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:03:22 smithi115 ceph-mon[78593]: pgmap v6819: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 6 op/s 2022-05-14T03:03:22.865 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:03:22 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:03:22.866 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:03:22 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:03:22.866 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:03:22 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:03:23.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:03:23 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:03:23.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:03:23 smithi115 conmon[90393]: 2022-05-14T03:03:23.731+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:03:23.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:03:23 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:03:24.292 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:03:23 smithi047 conmon[78521]: 2022-05-14T03:03:23.951+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:03:24.292 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:03:23 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T03:03:24.293 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:03:23 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:03:24.293 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:03:23 smithi047 conmon[78521]: yield 2022-05-14T03:03:24.293 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:03:23 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:03:24.293 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:03:23 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:03:24.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:03:23 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:03:24.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:03:23 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T03:03:24.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:03:23 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:03:24.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:03:23 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T03:03:24.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:03:23 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:03:24.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:03:23 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T03:03:24.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:03:23 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:03:24.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:03:23 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T03:03:24.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:03:23 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:03:24.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:03:23 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T03:03:24.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:03:23 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:03:24.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:03:23 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:03:24.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:03:23 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:03:24.297 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:03:23 smithi047 conmon[78521]: 2022-05-14T03:03:23.952+0000 7f5340779700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T03:03:24.297 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:03:23 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T03:03:24.297 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:03:23 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:03:24.297 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:03:23 smithi047 conmon[78521]: yield 2022-05-14T03:03:24.297 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:03:23 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:03:24.298 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:03:23 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:03:24.298 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:03:23 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:03:24.298 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:03:23 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T03:03:24.298 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:03:23 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:03:24.298 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:03:23 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T03:03:24.299 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:03:23 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:03:24.299 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:03:23 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T03:03:24.299 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:03:23 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:03:24.299 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:03:23 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T03:03:24.300 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:03:23 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:03:24.300 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:03:23 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T03:03:24.300 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:03:23 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:03:24.300 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:03:23 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:03:24.301 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:03:23 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:03:24.301 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:03:23 smithi047 conmon[78521]: 2022-05-14T03:03:24.301 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:03:23 smithi047 conmon[78521]: During handling of the above exception, another exception occurred: 2022-05-14T03:03:24.301 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:03:23 smithi047 conmon[78521]: 2022-05-14T03:03:24.302 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:03:23 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T03:03:24.302 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:03:23 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T03:03:24.302 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:03:23 smithi047 conmon[78521]: return f(*arg) 2022-05-14T03:03:24.302 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:03:23 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T03:03:24.302 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:03:23 smithi047 conmon[78521]: self._write_client_files(client_files, host) 2022-05-14T03:03:24.303 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:03:23 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T03:03:24.303 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:03:23 smithi047 conmon[78521]: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T03:03:24.303 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:03:23 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T03:03:24.303 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:03:23 smithi047 conmon[78521]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T03:03:24.304 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:03:23 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T03:03:24.304 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:03:23 smithi047 conmon[78521]: return self.event_loop.get_result(coro) 2022-05-14T03:03:24.304 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:03:23 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T03:03:24.304 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:03:23 smithi047 conmon[78521]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T03:03:24.305 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:03:23 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T03:03:24.305 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:03:23 smithi047 conmon[78521]: return self.__get_result() 2022-05-14T03:03:24.305 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:03:23 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T03:03:24.306 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:03:23 smithi047 conmon[78521]: raise self._exception 2022-05-14T03:03:24.306 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:03:23 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T03:03:24.306 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:03:23 smithi047 conmon[78521]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T03:03:24.306 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:03:23 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T03:03:24.307 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:03:23 smithi047 conmon[78521]: conn = await self._remote_connection(host, addr) 2022-05-14T03:03:24.307 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:03:23 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T03:03:24.307 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:03:23 smithi047 conmon[78521]: raise 2022-05-14T03:03:24.307 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:03:23 smithi047 conmon[78521]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T03:03:24.308 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:03:23 smithi047 conmon[78521]: self.gen.throw(type, value, traceback) 2022-05-14T03:03:24.308 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:03:23 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T03:03:24.308 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:03:23 smithi047 conmon[78521]: raise OrchestratorError(msg) 2022-05-14T03:03:24.308 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:03:23 smithi047 conmon[78521]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:03:24.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:03:24 smithi047 ceph-mon[77147]: pgmap v6820: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 4 op/s 2022-05-14T03:03:24.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:03:24 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:03:24.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:03:24 smithi115 conmon[90393]: 2022-05-14T03:03:24.735+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:03:24.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:03:24 smithi115 ceph-mon[78593]: pgmap v6820: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 4 op/s 2022-05-14T03:03:24.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:03:24 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:03:25.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:03:25 smithi115 conmon[90393]: 2022-05-14T03:03:25.748+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:03:25.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:03:25 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:03:25.859 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T03:03:25.859 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:03:25.860 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T03:03:25.860 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:03:25.860 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:03:25.860 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:03:25.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T03:03:25.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:03:25.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T03:03:25.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:03:25.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T03:03:25.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:03:25.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T03:03:25.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:03:25.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T03:03:25.863 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:03:25.863 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:03:25.863 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:03:25.864 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:03:25 smithi115 ceph-mon[78593]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T03:03:25.864 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T03:03:25.864 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:03:25.864 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T03:03:25.864 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:03:25.865 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:03:25.865 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:03:25.865 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T03:03:25.865 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:03:25.866 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T03:03:25.866 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:03:25.866 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T03:03:25.866 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:03:25.867 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T03:03:25.868 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:03:25.868 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T03:03:25.868 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:03:25.869 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:03:25.869 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:03:25.869 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T03:03:25.869 INFO:journalctl@ceph.mon.c.smithi115.stdout: During handling of the above exception, another exception occurred: 2022-05-14T03:03:25.870 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T03:03:25.870 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T03:03:25.870 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T03:03:25.870 INFO:journalctl@ceph.mon.c.smithi115.stdout: return f(*arg) 2022-05-14T03:03:25.871 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T03:03:25.871 INFO:journalctl@ceph.mon.c.smithi115.stdout: self._write_client_files(client_files, host) 2022-05-14T03:03:25.871 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T03:03:25.871 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T03:03:25.871 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T03:03:25.872 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T03:03:25.872 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T03:03:25.872 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.event_loop.get_result(coro) 2022-05-14T03:03:25.872 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T03:03:25.873 INFO:journalctl@ceph.mon.c.smithi115.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T03:03:25.873 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T03:03:25.873 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.__get_result() 2022-05-14T03:03:25.874 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T03:03:25.874 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise self._exception 2022-05-14T03:03:25.874 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T03:03:25.874 INFO:journalctl@ceph.mon.c.smithi115.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T03:03:25.875 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T03:03:25.875 INFO:journalctl@ceph.mon.c.smithi115.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T03:03:25.875 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T03:03:25.875 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise 2022-05-14T03:03:25.876 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T03:03:25.876 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.gen.throw(type, value, traceback) 2022-05-14T03:03:25.876 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T03:03:25.876 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OrchestratorError(msg) 2022-05-14T03:03:25.876 INFO:journalctl@ceph.mon.c.smithi115.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:03:25.877 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:03:25 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:03:26.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:03:25 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:03:26.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T03:03:26.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:03:26.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T03:03:26.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:03:26.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:03:26.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:03:26.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T03:03:26.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:03:26.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T03:03:26.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:03:26.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T03:03:26.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:03:26.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T03:03:26.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:03:26.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T03:03:26.049 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:03:26.049 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:03:26.050 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:03:26.050 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:03:25 smithi047 ceph-mon[77147]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T03:03:26.050 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T03:03:26.050 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:03:26.051 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T03:03:26.051 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:03:26.051 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:03:26.051 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:03:26.051 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T03:03:26.052 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:03:26.052 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T03:03:26.052 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:03:26.052 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T03:03:26.052 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:03:26.053 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T03:03:26.053 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:03:26.053 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T03:03:26.054 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:03:26.054 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:03:26.054 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:03:26.054 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T03:03:26.055 INFO:journalctl@ceph.mon.b.smithi047.stdout: During handling of the above exception, another exception occurred: 2022-05-14T03:03:26.055 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T03:03:26.055 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T03:03:26.055 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T03:03:26.056 INFO:journalctl@ceph.mon.b.smithi047.stdout: return f(*arg) 2022-05-14T03:03:26.056 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T03:03:26.056 INFO:journalctl@ceph.mon.b.smithi047.stdout: self._write_client_files(client_files, host) 2022-05-14T03:03:26.056 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T03:03:26.057 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T03:03:26.057 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T03:03:26.057 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T03:03:26.058 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T03:03:26.058 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.event_loop.get_result(coro) 2022-05-14T03:03:26.058 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T03:03:26.059 INFO:journalctl@ceph.mon.b.smithi047.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T03:03:26.059 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T03:03:26.060 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.__get_result() 2022-05-14T03:03:26.060 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T03:03:26.060 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise self._exception 2022-05-14T03:03:26.060 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T03:03:26.061 INFO:journalctl@ceph.mon.b.smithi047.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T03:03:26.061 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T03:03:26.061 INFO:journalctl@ceph.mon.b.smithi047.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T03:03:26.061 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T03:03:26.062 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise 2022-05-14T03:03:26.062 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T03:03:26.062 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.gen.throw(type, value, traceback) 2022-05-14T03:03:26.062 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T03:03:26.063 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OrchestratorError(msg) 2022-05-14T03:03:26.063 INFO:journalctl@ceph.mon.b.smithi047.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:03:26.063 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:03:25 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:03:26.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:03:26 smithi115 conmon[90393]: 2022-05-14T03:03:26.754+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:03:26.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:03:26 smithi115 ceph-mon[78593]: pgmap v6821: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 7 op/s 2022-05-14T03:03:26.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:03:26 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:03:26.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:03:26 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 12940 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:03:27.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:03:26 smithi047 ceph-mon[77147]: pgmap v6821: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 7 op/s 2022-05-14T03:03:27.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:03:26 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:03:27.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:03:26 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 12940 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:03:27.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:03:27 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:03:27.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:03:27 smithi115 conmon[90393]: 2022-05-14T03:03:27.751+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:03:28.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:03:27 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:03:28.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:03:28 smithi115 conmon[90393]: 2022-05-14T03:03:28.746+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:03:28.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:03:28 smithi115 ceph-mon[78593]: pgmap v6822: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:03:28.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:03:28 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:03:29.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:03:28 smithi047 ceph-mon[77147]: pgmap v6822: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:03:29.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:03:28 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:03:29.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:03:29 smithi115 conmon[90393]: 2022-05-14T03:03:29.768+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:03:29.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:03:29 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:03:30.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:03:29 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:03:30.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:03:30 smithi115 conmon[90393]: 2022-05-14T03:03:30.744+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:03:30.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:03:30 smithi115 ceph-mon[78593]: pgmap v6823: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:03:30.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:03:30 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:03:30.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:03:30 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:03:30.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:03:30 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:03:31.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:03:30 smithi047 ceph-mon[77147]: pgmap v6823: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:03:31.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:03:30 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:03:31.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:03:30 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:03:31.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:03:30 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:03:31.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:03:31 smithi115 conmon[90393]: 2022-05-14T03:03:31.777+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:03:31.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:03:31 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:03:31.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:03:31 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 12945 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:03:32.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:03:31 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:03:32.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:03:31 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 12945 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:03:32.817 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:03:32 smithi115 ceph-mon[78593]: pgmap v6824: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:03:32.818 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:03:32 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:03:33.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:03:32 smithi047 ceph-mon[77147]: pgmap v6824: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:03:33.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:03:32 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:03:33.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:03:32 smithi115 conmon[90393]: 2022-05-14T03:03:32.816+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:03:33.849 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:03:33 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:03:34.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:03:33 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:03:34.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:03:33 smithi115 conmon[90393]: 2022-05-14T03:03:33.848+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:03:34.844 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:03:34 smithi115 ceph-mon[78593]: pgmap v6825: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:03:34.844 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:03:34 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:03:34.844 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:03:34 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:03:34.845 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:03:34 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:03:35.046 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:03:34 smithi047 ceph-mon[77147]: pgmap v6825: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:03:35.046 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:03:34 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:03:35.046 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:03:34 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:03:35.047 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:03:34 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:03:35.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:03:34 smithi115 conmon[90393]: 2022-05-14T03:03:34.842+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:03:35.855 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:03:35 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:03:36.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:03:35 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:03:36.110 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:03:35 smithi115 conmon[90393]: 2022-05-14T03:03:35.852+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:03:36.865 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:03:36 smithi115 conmon[90393]: 2022-05-14T03:03:36.810+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:03:36.866 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:03:36 smithi115 ceph-mon[78593]: pgmap v6826: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:03:36.867 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:03:36 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:03:36.867 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:03:36 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 12950 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:03:37.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:03:36 smithi047 ceph-mon[77147]: pgmap v6826: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:03:37.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:03:36 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:03:37.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:03:36 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 12950 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:03:37.855 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:03:37 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:03:38.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:03:37 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:03:38.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:03:37 smithi115 conmon[90393]: 2022-05-14T03:03:37.853+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:03:38.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:03:38 smithi115 ceph-mon[78593]: pgmap v6827: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:03:38.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:03:38 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:03:38.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:03:38 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:03:38.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:03:38 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:03:39.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:03:38 smithi047 ceph-mon[77147]: pgmap v6827: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:03:39.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:03:38 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:03:39.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:03:38 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:03:39.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:03:38 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:03:39.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:03:38 smithi115 conmon[90393]: 2022-05-14T03:03:38.903+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:03:39.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:03:39 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:03:40.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:03:39 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:03:40.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:03:39 smithi115 conmon[90393]: 2022-05-14T03:03:39.871+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:03:40.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:03:40 smithi115 ceph-mon[78593]: pgmap v6828: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:03:40.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:03:40 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:03:41.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:03:40 smithi047 ceph-mon[77147]: pgmap v6828: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:03:41.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:03:40 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:03:41.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:03:40 smithi115 conmon[90393]: 2022-05-14T03:03:40.867+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:03:41.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:03:41 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:03:41.862 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:03:41 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 12955 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:03:42.046 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:03:41 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:03:42.046 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:03:41 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 12955 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:03:42.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:03:41 smithi115 conmon[90393]: 2022-05-14T03:03:41.869+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:03:42.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:03:42 smithi115 ceph-mon[78593]: pgmap v6829: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:03:42.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:03:42 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:03:42.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:03:42 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:03:42.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:03:42 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:03:43.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:03:42 smithi047 ceph-mon[77147]: pgmap v6829: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:03:43.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:03:42 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:03:43.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:03:42 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:03:43.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:03:42 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:03:43.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:03:42 smithi115 conmon[90393]: 2022-05-14T03:03:42.915+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:03:43.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:03:43 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:03:44.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:03:43 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:03:44.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:03:43 smithi115 conmon[90393]: 2022-05-14T03:03:43.951+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:03:44.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:03:44 smithi115 ceph-mon[78593]: pgmap v6830: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:03:44.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:03:44 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:03:45.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:03:44 smithi047 ceph-mon[77147]: pgmap v6830: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:03:45.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:03:44 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:03:45.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:03:44 smithi115 conmon[90393]: 2022-05-14T03:03:44.943+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:03:45.939 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:03:45 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:03:46.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:03:45 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:03:46.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:03:45 smithi115 conmon[90393]: 2022-05-14T03:03:45.937+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:03:46.906 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:03:46 smithi115 ceph-mon[78593]: pgmap v6831: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:03:46.906 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:03:46 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:03:46.907 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:03:46 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 12960 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:03:47.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:03:46 smithi047 ceph-mon[77147]: pgmap v6831: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:03:47.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:03:46 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:03:47.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:03:46 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 12960 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:03:47.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:03:46 smithi115 conmon[90393]: 2022-05-14T03:03:46.904+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:03:47.925 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:03:47 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:03:48.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:03:47 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:03:48.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:03:47 smithi115 conmon[90393]: 2022-05-14T03:03:47.923+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:03:48.910 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:03:48 smithi115 ceph-mon[78593]: pgmap v6832: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:03:48.966 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:03:48 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:03:49.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:03:48 smithi047 ceph-mon[77147]: pgmap v6832: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:03:49.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:03:48 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:03:49.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:03:48 smithi115 conmon[90393]: 2022-05-14T03:03:48.908+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:03:49.878 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:03:49 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:03:50.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:03:49 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:03:50.361 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:03:49 smithi115 conmon[90393]: 2022-05-14T03:03:49.876+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:03:51.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:03:50 smithi047 ceph-mon[77147]: pgmap v6833: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:03:51.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:03:50 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:03:51.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:03:50 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:03:51.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:03:50 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:03:51.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:03:50 smithi115 conmon[90393]: 2022-05-14T03:03:50.831+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:03:51.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:03:50 smithi115 ceph-mon[78593]: pgmap v6833: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:03:51.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:03:50 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:03:51.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:03:50 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:03:51.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:03:50 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:03:52.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:03:51 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:03:52.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:03:51 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 12965 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:03:52.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:03:51 smithi115 conmon[90393]: 2022-05-14T03:03:51.849+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:03:52.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:03:51 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:03:52.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:03:51 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 12965 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:03:53.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:03:52 smithi047 ceph-mon[77147]: pgmap v6834: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:03:53.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:03:52 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:03:53.113 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:03:52 smithi115 conmon[90393]: 2022-05-14T03:03:52.861+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:03:53.114 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:03:52 smithi115 ceph-mon[78593]: pgmap v6834: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:03:53.114 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:03:52 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:03:54.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:03:53 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:03:54.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:03:53 smithi115 conmon[90393]: 2022-05-14T03:03:53.821+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:03:54.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:03:53 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:03:55.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:03:54 smithi047 ceph-mon[77147]: pgmap v6835: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:03:55.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:03:54 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:03:55.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:03:54 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:03:55.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:03:54 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:03:55.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:03:54 smithi115 conmon[90393]: 2022-05-14T03:03:54.808+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:03:55.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:03:54 smithi115 ceph-mon[78593]: pgmap v6835: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:03:55.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:03:54 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:03:55.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:03:54 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:03:55.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:03:54 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:03:56.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:03:55 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:03:56.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:03:55 smithi115 conmon[90393]: 2022-05-14T03:03:55.815+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:03:56.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:03:55 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:03:57.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:03:56 smithi047 ceph-mon[77147]: pgmap v6836: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:03:57.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:03:56 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:03:57.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:03:56 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 12970 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:03:57.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:03:56 smithi115 conmon[90393]: 2022-05-14T03:03:56.769+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:03:57.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:03:56 smithi115 ceph-mon[78593]: pgmap v6836: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:03:57.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:03:56 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:03:57.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:03:56 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 12970 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:03:58.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:03:57 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:03:58.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:03:57 smithi115 conmon[90393]: 2022-05-14T03:03:57.794+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:03:58.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:03:57 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:03:59.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:03:58 smithi047 ceph-mon[77147]: pgmap v6837: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:03:59.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:03:58 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:03:59.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:03:58 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:03:59.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:03:58 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:03:59.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:03:58 smithi115 conmon[90393]: 2022-05-14T03:03:58.831+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:03:59.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:03:58 smithi115 ceph-mon[78593]: pgmap v6837: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:03:59.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:03:58 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:03:59.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:03:58 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:03:59.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:03:58 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:04:00.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:03:59 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:04:00.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:03:59 smithi115 conmon[90393]: 2022-05-14T03:03:59.782+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:04:00.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:03:59 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:04:01.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:04:00 smithi047 ceph-mon[77147]: pgmap v6838: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:04:01.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:04:00 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:04:01.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:04:00 smithi115 conmon[90393]: 2022-05-14T03:04:00.780+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:04:01.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:04:00 smithi115 ceph-mon[78593]: pgmap v6838: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:04:01.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:04:00 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:04:02.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:04:01 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:04:02.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:04:01 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 12975 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:04:02.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:04:01 smithi115 conmon[90393]: 2022-05-14T03:04:01.814+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:04:02.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:04:01 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:04:02.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:04:01 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 12975 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:04:03.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:04:02 smithi047 ceph-mon[77147]: pgmap v6839: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:04:03.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:04:02 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:04:03.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:04:02 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:04:03.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:04:02 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:04:03.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:04:02 smithi115 conmon[90393]: 2022-05-14T03:04:02.857+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:04:03.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:04:02 smithi115 ceph-mon[78593]: pgmap v6839: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:04:03.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:04:02 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:04:03.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:04:02 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:04:03.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:04:02 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:04:04.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:04:03 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:04:04.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:04:03 smithi115 conmon[90393]: 2022-05-14T03:04:03.905+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:04:04.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:04:03 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:04:04.957 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:04:04 smithi115 ceph-mon[78593]: pgmap v6840: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:04:04.958 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:04:04 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:04:05.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:04:04 smithi047 ceph-mon[77147]: pgmap v6840: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:04:05.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:04:04 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:04:05.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:04:04 smithi115 conmon[90393]: 2022-05-14T03:04:04.955+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:04:06.002 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:04:05 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:04:06.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:04:05 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:04:06.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:04:06 smithi115 conmon[90393]: 2022-05-14T03:04:06.000+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:04:07.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:04:06 smithi047 ceph-mon[77147]: pgmap v6841: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:04:07.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:04:06 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:04:07.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:04:06 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 12980 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:04:07.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:04:06 smithi115 ceph-mon[78593]: pgmap v6841: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:04:07.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:04:06 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:04:07.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:04:06 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 12980 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:04:07.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:04:07 smithi115 conmon[90393]: 2022-05-14T03:04:07.045+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:04:08.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:04:07 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:04:08.096 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:04:07 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:04:08.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:04:08 smithi115 conmon[90393]: 2022-05-14T03:04:08.095+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:04:09.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:04:08 smithi047 ceph-mon[77147]: pgmap v6842: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:04:09.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:04:08 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:04:09.062 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:04:08 smithi115 ceph-mon[78593]: pgmap v6842: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:04:09.063 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:04:08 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:04:09.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:04:09 smithi115 conmon[90393]: 2022-05-14T03:04:09.060+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:04:10.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:04:09 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:04:10.096 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:04:09 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:04:10.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:04:10 smithi115 conmon[90393]: 2022-05-14T03:04:10.094+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:04:11.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:04:10 smithi047 ceph-mon[77147]: pgmap v6843: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:04:11.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:04:10 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:04:11.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:04:10 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:04:11.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:04:10 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:04:11.080 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:04:10 smithi115 ceph-mon[78593]: pgmap v6843: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:04:11.080 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:04:10 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:04:11.080 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:04:10 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:04:11.081 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:04:10 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:04:11.369 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:04:11 smithi115 conmon[90393]: 2022-05-14T03:04:11.078+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:04:12.036 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:04:11 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:04:12.036 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:04:11 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 12985 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:04:12.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:04:11 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:04:12.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:04:11 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 12985 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:04:12.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:04:12 smithi115 conmon[90393]: 2022-05-14T03:04:12.034+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:04:13.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:04:12 smithi047 ceph-mon[77147]: pgmap v6844: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:04:13.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:04:12 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:04:13.051 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:04:12 smithi115 ceph-mon[78593]: pgmap v6844: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:04:13.051 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:04:12 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:04:13.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:04:13 smithi115 conmon[90393]: 2022-05-14T03:04:13.050+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:04:14.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:04:13 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:04:14.065 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:04:13 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:04:14.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:04:14 smithi115 conmon[90393]: 2022-05-14T03:04:14.063+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:04:15.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:04:14 smithi047 ceph-mon[77147]: pgmap v6845: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:04:15.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:04:14 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:04:15.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:04:14 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:04:15.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:04:14 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:04:15.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:04:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T03:04:15.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:04:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T03:04:15.070 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:04:14 smithi115 ceph-mon[78593]: pgmap v6845: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:04:15.071 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:04:14 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:04:15.071 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:04:14 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:04:15.071 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:04:14 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:04:15.071 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:04:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T03:04:15.072 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:04:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T03:04:15.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:04:15 smithi115 conmon[90393]: 2022-05-14T03:04:15.069+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:04:16.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:04:15 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:04:16.086 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:04:15 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:04:16.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:04:16 smithi115 conmon[90393]: 2022-05-14T03:04:16.085+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:04:17.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:04:16 smithi047 ceph-mon[77147]: pgmap v6846: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:04:17.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:04:16 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:04:17.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:04:16 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 12989 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:04:17.052 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:04:16 smithi115 ceph-mon[78593]: pgmap v6846: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:04:17.052 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:04:16 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:04:17.052 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:04:16 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 12989 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:04:17.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:04:17 smithi115 conmon[90393]: 2022-05-14T03:04:17.051+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:04:18.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:04:17 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:04:18.089 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:04:17 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:04:18.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:04:18 smithi115 conmon[90393]: 2022-05-14T03:04:18.087+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:04:19.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:04:18 smithi047 ceph-mon[77147]: pgmap v6847: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:04:19.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:04:18 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:04:19.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:04:18 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:04:19.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:04:18 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:04:19.049 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:04:18 smithi115 ceph-mon[78593]: pgmap v6847: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:04:19.050 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:04:18 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:04:19.050 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:04:18 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:04:19.050 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:04:18 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:04:19.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:04:19 smithi115 conmon[90393]: 2022-05-14T03:04:19.048+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:04:20.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:04:19 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:04:20.088 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:04:19 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:04:20.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:04:20 smithi115 conmon[90393]: 2022-05-14T03:04:20.086+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:04:21.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:04:20 smithi047 ceph-mon[77147]: pgmap v6848: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:04:21.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:04:20 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:04:21.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:04:20 smithi115 ceph-mon[78593]: pgmap v6848: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:04:21.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:04:20 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:04:21.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:04:21 smithi115 conmon[90393]: 2022-05-14T03:04:21.118+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:04:22.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:04:21 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:04:22.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:04:21 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 12994 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:04:22.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:04:21 smithi047 ceph-mon[77147]: pgmap v6849: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:04:22.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:04:21 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:04:22.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:04:21 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 12994 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:04:22.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:04:21 smithi115 ceph-mon[78593]: pgmap v6849: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:04:22.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:04:22 smithi115 conmon[90393]: 2022-05-14T03:04:22.150+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:04:23.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:04:22 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:04:23.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:04:22 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:04:23.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:04:22 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:04:23.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:04:22 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:04:23.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:04:22 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:04:23.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:04:22 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:04:23.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:04:23 smithi115 conmon[90393]: 2022-05-14T03:04:23.159+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:04:23.999 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:04:23 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:04:23.999 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:04:23 smithi047 ceph-mon[77147]: pgmap v6850: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:04:23.999 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:04:23 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:04:24.000 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:04:23 smithi115 ceph-mon[78593]: pgmap v6850: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:04:24.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:04:24 smithi115 conmon[90393]: 2022-05-14T03:04:24.183+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:04:25.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:04:24 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T03:04:25.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:04:24 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T03:04:25.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:04:24 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T03:04:25.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:04:24 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:04:25.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:04:24 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T03:04:25.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:04:24 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T03:04:25.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:04:24 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T03:04:25.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:04:24 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:04:25.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:04:25 smithi115 conmon[90393]: 2022-05-14T03:04:25.219+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:04:26.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:04:25 smithi047 ceph-mon[77147]: pgmap v6851: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:04:26.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:04:25 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:04:26.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:04:25 smithi115 ceph-mon[78593]: pgmap v6851: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:04:26.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:04:25 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:04:26.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:04:26 smithi115 conmon[90393]: 2022-05-14T03:04:26.232+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:04:27.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:04:26 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 13000 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:04:27.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:04:26 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:04:27.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:04:26 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 13000 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:04:27.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:04:26 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:04:27.542 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:04:27 smithi047 conmon[78521]: 2022-05-14T03:04:27.055+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:04:27.542 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:04:27 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T03:04:27.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:04:27 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:04:27.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:04:27 smithi047 conmon[78521]: yield 2022-05-14T03:04:27.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:04:27 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:04:27.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:04:27 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:04:27.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:04:27 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:04:27.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:04:27 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T03:04:27.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:04:27 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:04:27.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:04:27 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T03:04:27.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:04:27 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:04:27.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:04:27 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T03:04:27.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:04:27 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:04:27.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:04:27 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T03:04:27.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:04:27 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:04:27.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:04:27 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T03:04:27.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:04:27 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:04:27.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:04:27 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:04:27.547 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:04:27 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:04:27.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:04:27 smithi115 conmon[90393]: 2022-05-14T03:04:27.266+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:04:28.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:04:27 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:04:28.042 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T03:04:28.042 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:04:28.042 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T03:04:28.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:04:28.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:04:28.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:04:28.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T03:04:28.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:04:28.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T03:04:28.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:04:28.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T03:04:28.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:04:28.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T03:04:28.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:04:28.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T03:04:28.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:04:28.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:04:28.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:04:28.046 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:04:27 smithi047 ceph-mon[77147]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T03:04:28.047 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:04:27 smithi047 ceph-mon[77147]: pgmap v6852: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:04:28.047 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:04:27 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:04:28.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:04:27 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:04:28.108 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T03:04:28.109 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:04:28.109 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T03:04:28.109 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:04:28.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:04:28.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:04:28.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T03:04:28.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:04:28.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T03:04:28.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:04:28.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T03:04:28.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:04:28.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T03:04:28.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:04:28.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T03:04:28.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:04:28.113 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:04:28.113 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:04:28.113 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:04:27 smithi115 ceph-mon[78593]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T03:04:28.113 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:04:27 smithi115 ceph-mon[78593]: pgmap v6852: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:04:28.114 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:04:27 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:04:28.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:04:28 smithi115 conmon[90393]: 2022-05-14T03:04:28.222+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:04:29.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:04:28 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:04:29.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:04:28 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:04:29.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:04:29 smithi115 conmon[90393]: 2022-05-14T03:04:29.240+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:04:30.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:04:29 smithi047 ceph-mon[77147]: pgmap v6853: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:04:30.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:04:29 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:04:30.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:04:29 smithi115 ceph-mon[78593]: pgmap v6853: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:04:30.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:04:29 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:04:30.542 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:04:30 smithi047 conmon[78521]: 2022-05-14T03:04:30.127+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:04:30.542 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:04:30 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T03:04:30.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:04:30 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:04:30.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:04:30 smithi047 conmon[78521]: yield 2022-05-14T03:04:30.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:04:30 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:04:30.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:04:30 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:04:30.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:04:30 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:04:30.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:04:30 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T03:04:30.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:04:30 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:04:30.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:04:30 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T03:04:30.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:04:30 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:04:30.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:04:30 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T03:04:30.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:04:30 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:04:30.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:04:30 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T03:04:30.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:04:30 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:04:30.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:04:30 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T03:04:30.547 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:04:30 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:04:30.547 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:04:30 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:04:30.547 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:04:30 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:04:30.547 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:04:30 smithi047 conmon[78521]: 2022-05-14T03:04:30.128+0000 7f5344781700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T03:04:30.547 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:04:30 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T03:04:30.548 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:04:30 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:04:30.548 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:04:30 smithi047 conmon[78521]: yield 2022-05-14T03:04:30.548 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:04:30 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:04:30.548 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:04:30 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:04:30.548 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:04:30 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:04:30.549 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:04:30 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T03:04:30.549 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:04:30 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:04:30.549 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:04:30 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T03:04:30.549 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:04:30 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:04:30.550 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:04:30 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T03:04:30.550 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:04:30 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:04:30.550 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:04:30 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T03:04:30.550 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:04:30 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:04:30.550 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:04:30 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T03:04:30.551 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:04:30 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:04:30.551 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:04:30 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:04:30.552 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:04:30 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:04:30.552 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:04:30 smithi047 conmon[78521]: 2022-05-14T03:04:30.552 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:04:30 smithi047 conmon[78521]: During handling of the above exception, another exception occurred: 2022-05-14T03:04:30.553 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:04:30 smithi047 conmon[78521]: 2022-05-14T03:04:30.553 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:04:30 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T03:04:30.554 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:04:30 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T03:04:30.554 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:04:30 smithi047 conmon[78521]: return f(*arg) 2022-05-14T03:04:30.555 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:04:30 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T03:04:30.555 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:04:30 smithi047 conmon[78521]: self._write_client_files(client_files, host) 2022-05-14T03:04:30.555 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:04:30 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T03:04:30.556 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:04:30 smithi047 conmon[78521]: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T03:04:30.556 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:04:30 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T03:04:30.556 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:04:30 smithi047 conmon[78521]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T03:04:30.557 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:04:30 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T03:04:30.557 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:04:30 smithi047 conmon[78521]: return self.event_loop.get_result(coro) 2022-05-14T03:04:30.557 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:04:30 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T03:04:30.558 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:04:30 smithi047 conmon[78521]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T03:04:30.558 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:04:30 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T03:04:30.559 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:04:30 smithi047 conmon[78521]: return self.__get_result() 2022-05-14T03:04:30.559 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:04:30 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T03:04:30.559 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:04:30 smithi047 conmon[78521]: raise self._exception 2022-05-14T03:04:30.560 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:04:30 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T03:04:30.560 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:04:30 smithi047 conmon[78521]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T03:04:30.560 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:04:30 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T03:04:30.560 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:04:30 smithi047 conmon[78521]: conn = await self._remote_connection(host, addr) 2022-05-14T03:04:30.561 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:04:30 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T03:04:30.561 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:04:30 smithi047 conmon[78521]: raise 2022-05-14T03:04:30.561 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:04:30 smithi047 conmon[78521]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T03:04:30.561 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:04:30 smithi047 conmon[78521]: self.gen.throw(type, value, traceback) 2022-05-14T03:04:30.561 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:04:30 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T03:04:30.562 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:04:30 smithi047 conmon[78521]: raise OrchestratorError(msg) 2022-05-14T03:04:30.562 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:04:30 smithi047 conmon[78521]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:04:30.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:04:30 smithi115 conmon[90393]: 2022-05-14T03:04:30.224+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:04:31.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:04:30 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:04:31.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:04:30 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:04:31.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:04:30 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:04:31.042 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T03:04:31.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:04:31.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T03:04:31.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:04:31.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:04:31.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:04:31.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T03:04:31.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:04:31.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T03:04:31.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:04:31.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T03:04:31.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:04:31.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T03:04:31.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:04:31.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T03:04:31.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:04:31.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:04:31.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:04:31.047 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:04:30 smithi047 ceph-mon[77147]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T03:04:31.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T03:04:31.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:04:31.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T03:04:31.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:04:31.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:04:31.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:04:31.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T03:04:31.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:04:31.049 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T03:04:31.049 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:04:31.049 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T03:04:31.049 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:04:31.049 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T03:04:31.050 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:04:31.050 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T03:04:31.050 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:04:31.050 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:04:31.051 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:04:31.051 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T03:04:31.051 INFO:journalctl@ceph.mon.b.smithi047.stdout: During handling of the above exception, another exception occurred: 2022-05-14T03:04:31.051 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T03:04:31.052 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T03:04:31.052 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T03:04:31.052 INFO:journalctl@ceph.mon.b.smithi047.stdout: return f(*arg) 2022-05-14T03:04:31.052 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T03:04:31.052 INFO:journalctl@ceph.mon.b.smithi047.stdout: self._write_client_files(client_files, host) 2022-05-14T03:04:31.053 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T03:04:31.053 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T03:04:31.053 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T03:04:31.053 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T03:04:31.054 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T03:04:31.054 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.event_loop.get_result(coro) 2022-05-14T03:04:31.054 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T03:04:31.054 INFO:journalctl@ceph.mon.b.smithi047.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T03:04:31.054 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T03:04:31.055 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.__get_result() 2022-05-14T03:04:31.055 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T03:04:31.055 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise self._exception 2022-05-14T03:04:31.056 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T03:04:31.056 INFO:journalctl@ceph.mon.b.smithi047.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T03:04:31.056 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T03:04:31.056 INFO:journalctl@ceph.mon.b.smithi047.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T03:04:31.057 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T03:04:31.057 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise 2022-05-14T03:04:31.057 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T03:04:31.057 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.gen.throw(type, value, traceback) 2022-05-14T03:04:31.057 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T03:04:31.058 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OrchestratorError(msg) 2022-05-14T03:04:31.058 INFO:journalctl@ceph.mon.b.smithi047.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:04:31.058 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:04:30 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:04:31.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:04:30 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:04:31.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:04:30 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:04:31.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:04:30 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:04:31.109 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T03:04:31.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:04:31.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T03:04:31.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:04:31.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:04:31.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:04:31.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T03:04:31.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:04:31.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T03:04:31.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:04:31.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T03:04:31.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:04:31.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T03:04:31.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:04:31.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T03:04:31.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:04:31.113 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:04:31.113 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:04:31.113 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:04:30 smithi115 ceph-mon[78593]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T03:04:31.113 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T03:04:31.114 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:04:31.114 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T03:04:31.114 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:04:31.114 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:04:31.114 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:04:31.115 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T03:04:31.115 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:04:31.115 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T03:04:31.115 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:04:31.115 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T03:04:31.116 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:04:31.116 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T03:04:31.116 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:04:31.116 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T03:04:31.116 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:04:31.117 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:04:31.117 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:04:31.117 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T03:04:31.117 INFO:journalctl@ceph.mon.c.smithi115.stdout: During handling of the above exception, another exception occurred: 2022-05-14T03:04:31.118 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T03:04:31.118 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T03:04:31.118 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T03:04:31.118 INFO:journalctl@ceph.mon.c.smithi115.stdout: return f(*arg) 2022-05-14T03:04:31.119 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T03:04:31.119 INFO:journalctl@ceph.mon.c.smithi115.stdout: self._write_client_files(client_files, host) 2022-05-14T03:04:31.119 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T03:04:31.119 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T03:04:31.119 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T03:04:31.120 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T03:04:31.120 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T03:04:31.120 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.event_loop.get_result(coro) 2022-05-14T03:04:31.120 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T03:04:31.120 INFO:journalctl@ceph.mon.c.smithi115.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T03:04:31.121 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T03:04:31.121 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.__get_result() 2022-05-14T03:04:31.121 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T03:04:31.121 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise self._exception 2022-05-14T03:04:31.122 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T03:04:31.122 INFO:journalctl@ceph.mon.c.smithi115.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T03:04:31.122 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T03:04:31.122 INFO:journalctl@ceph.mon.c.smithi115.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T03:04:31.122 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T03:04:31.123 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise 2022-05-14T03:04:31.123 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T03:04:31.123 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.gen.throw(type, value, traceback) 2022-05-14T03:04:31.123 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T03:04:31.123 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OrchestratorError(msg) 2022-05-14T03:04:31.124 INFO:journalctl@ceph.mon.c.smithi115.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:04:31.124 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:04:30 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:04:31.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:04:31 smithi115 conmon[90393]: 2022-05-14T03:04:31.243+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:04:32.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:04:31 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 13005 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:04:32.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:04:31 smithi047 ceph-mon[77147]: pgmap v6854: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:04:32.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:04:31 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:04:32.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:04:31 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 13005 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:04:32.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:04:31 smithi115 ceph-mon[78593]: pgmap v6854: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:04:32.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:04:31 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:04:32.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:04:32 smithi115 conmon[90393]: 2022-05-14T03:04:32.238+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:04:33.547 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:04:33 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:04:33.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:04:33 smithi115 conmon[90393]: 2022-05-14T03:04:33.284+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:04:33.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:04:33 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:04:34.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:04:34 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:04:34.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:04:34 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:04:34.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:04:34 smithi047 ceph-mon[77147]: pgmap v6855: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:04:34.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:04:34 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:04:34.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:04:34 smithi115 conmon[90393]: 2022-05-14T03:04:34.321+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:04:34.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:04:34 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:04:34.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:04:34 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:04:34.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:04:34 smithi115 ceph-mon[78593]: pgmap v6855: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:04:34.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:04:34 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:04:35.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:04:35 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:04:35.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:04:35 smithi115 conmon[90393]: 2022-05-14T03:04:35.316+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:04:35.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:04:35 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:04:36.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:04:36 smithi047 ceph-mon[77147]: pgmap v6856: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:04:36.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:04:36 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:04:36.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:04:36 smithi115 conmon[90393]: 2022-05-14T03:04:36.362+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:04:36.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:04:36 smithi115 ceph-mon[78593]: pgmap v6856: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:04:36.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:04:36 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:04:37.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:04:37 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 13010 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:04:37.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:04:37 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:04:37.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:04:37 smithi115 conmon[90393]: 2022-05-14T03:04:37.337+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:04:37.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:04:37 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 13010 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:04:37.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:04:37 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:04:38.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:04:38 smithi047 ceph-mon[77147]: pgmap v6857: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:04:38.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:04:38 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:04:38.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:04:38 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:04:38.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:04:38 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:04:38.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:04:38 smithi115 conmon[90393]: 2022-05-14T03:04:38.319+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:04:38.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:04:38 smithi115 ceph-mon[78593]: pgmap v6857: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:04:38.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:04:38 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:04:38.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:04:38 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:04:38.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:04:38 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:04:39.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:04:39 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:04:39.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:04:39 smithi115 conmon[90393]: 2022-05-14T03:04:39.303+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:04:39.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:04:39 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:04:40.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:04:40 smithi047 ceph-mon[77147]: pgmap v6858: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:04:40.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:04:40 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:04:40.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:04:40 smithi115 conmon[90393]: 2022-05-14T03:04:40.277+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:04:40.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:04:40 smithi115 ceph-mon[78593]: pgmap v6858: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:04:40.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:04:40 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:04:41.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:04:41 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:04:41.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:04:41 smithi115 conmon[90393]: 2022-05-14T03:04:41.271+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:04:41.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:04:41 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:04:42.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:04:42 smithi047 ceph-mon[77147]: pgmap v6859: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:04:42.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:04:42 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:04:42.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:04:42 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:04:42.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:04:42 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:04:42.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:04:42 smithi115 conmon[90393]: 2022-05-14T03:04:42.255+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:04:42.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:04:42 smithi115 ceph-mon[78593]: pgmap v6859: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:04:42.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:04:42 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:04:42.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:04:42 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:04:42.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:04:42 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:04:43.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:04:43 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:04:43.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:04:43 smithi115 conmon[90393]: 2022-05-14T03:04:43.299+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:04:43.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:04:43 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:04:44.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:04:44 smithi047 ceph-mon[77147]: pgmap v6860: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:04:44.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:04:44 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:04:44.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:04:44 smithi115 conmon[90393]: 2022-05-14T03:04:44.264+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:04:44.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:04:44 smithi115 ceph-mon[78593]: pgmap v6860: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:04:44.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:04:44 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:04:45.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:04:45 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:04:45.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:04:45 smithi115 conmon[90393]: 2022-05-14T03:04:45.246+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:04:45.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:04:45 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:04:46.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:04:46 smithi047 ceph-mon[77147]: pgmap v6861: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:04:46.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:04:46 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 13020 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:04:46.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:04:46 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:04:46.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:04:46 smithi115 conmon[90393]: 2022-05-14T03:04:46.274+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:04:46.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:04:46 smithi115 ceph-mon[78593]: pgmap v6861: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:04:46.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:04:46 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 13020 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:04:46.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:04:46 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:04:47.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:04:47 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:04:47.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:04:47 smithi115 conmon[90393]: 2022-05-14T03:04:47.277+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:04:47.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:04:47 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:04:48.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:04:48 smithi047 ceph-mon[77147]: pgmap v6862: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:04:48.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:04:48 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:04:48.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:04:48 smithi115 conmon[90393]: 2022-05-14T03:04:48.252+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:04:48.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:04:48 smithi115 ceph-mon[78593]: pgmap v6862: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:04:48.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:04:48 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:04:49.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:04:49 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:04:49.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:04:49 smithi115 conmon[90393]: 2022-05-14T03:04:49.233+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:04:49.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:04:49 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:04:50.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:04:50 smithi047 ceph-mon[77147]: pgmap v6863: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:04:50.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:04:50 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:04:50.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:04:50 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:04:50.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:04:50 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:04:50.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:04:50 smithi115 conmon[90393]: 2022-05-14T03:04:50.217+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:04:50.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:04:50 smithi115 ceph-mon[78593]: pgmap v6863: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:04:50.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:04:50 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:04:50.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:04:50 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:04:50.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:04:50 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:04:51.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:04:51 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:04:51.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:04:51 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 13025 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:04:51.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:04:51 smithi115 conmon[90393]: 2022-05-14T03:04:51.232+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:04:51.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:04:51 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:04:51.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:04:51 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 13025 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:04:52.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:04:52 smithi047 ceph-mon[77147]: pgmap v6864: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:04:52.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:04:52 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:04:52.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:04:52 smithi115 conmon[90393]: 2022-05-14T03:04:52.269+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:04:52.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:04:52 smithi115 ceph-mon[78593]: pgmap v6864: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:04:52.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:04:52 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:04:53.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:04:53 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:04:53.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:04:53 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:04:53.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:04:53 smithi115 conmon[90393]: 2022-05-14T03:04:53.311+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:04:54.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:04:54 smithi047 ceph-mon[77147]: pgmap v6865: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:04:54.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:04:54 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:04:54.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:04:54 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:04:54.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:04:54 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:04:54.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:04:54 smithi115 conmon[90393]: 2022-05-14T03:04:54.303+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:04:54.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:04:54 smithi115 ceph-mon[78593]: pgmap v6865: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:04:54.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:04:54 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:04:54.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:04:54 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:04:54.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:04:54 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:04:55.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:04:55 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:04:55.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:04:55 smithi115 conmon[90393]: 2022-05-14T03:04:55.282+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:04:55.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:04:55 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:04:56.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:04:56 smithi047 ceph-mon[77147]: pgmap v6866: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:04:56.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:04:56 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:04:56.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:04:56 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 13030 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:04:56.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:04:56 smithi115 conmon[90393]: 2022-05-14T03:04:56.293+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:04:56.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:04:56 smithi115 ceph-mon[78593]: pgmap v6866: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:04:56.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:04:56 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:04:56.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:04:56 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 13030 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:04:57.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:04:57 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:04:57.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:04:57 smithi115 conmon[90393]: 2022-05-14T03:04:57.306+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:04:57.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:04:57 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:04:58.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:04:58 smithi047 ceph-mon[77147]: pgmap v6867: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:04:58.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:04:58 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:04:58.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:04:58 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:04:58.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:04:58 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:04:58.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:04:58 smithi115 conmon[90393]: 2022-05-14T03:04:58.311+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:04:58.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:04:58 smithi115 ceph-mon[78593]: pgmap v6867: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:04:58.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:04:58 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:04:58.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:04:58 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:04:58.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:04:58 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:04:59.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:04:59 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:04:59.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:04:59 smithi115 conmon[90393]: 2022-05-14T03:04:59.313+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:04:59.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:04:59 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:05:00.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:05:00 smithi047 ceph-mon[77147]: pgmap v6868: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:05:00.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:05:00 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:05:00.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:05:00 smithi115 conmon[90393]: 2022-05-14T03:05:00.311+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:05:00.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:05:00 smithi115 ceph-mon[78593]: pgmap v6868: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:05:00.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:05:00 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:05:01.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:05:01 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:05:01.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:05:01 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 13035 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:05:01.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:05:01 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:05:01.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:05:01 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 13035 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:05:01.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:05:01 smithi115 conmon[90393]: 2022-05-14T03:05:01.298+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:05:02.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:05:02 smithi115 ceph-mon[78593]: pgmap v6869: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:05:02.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:05:02 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:05:02.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:05:02 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:05:02.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:05:02 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:05:02.610 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:05:02 smithi115 conmon[90393]: 2022-05-14T03:05:02.278+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:05:02.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:05:02 smithi047 ceph-mon[77147]: pgmap v6869: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:05:02.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:05:02 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:05:02.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:05:02 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:05:02.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:05:02 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:05:03.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:05:03 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:05:03.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:05:03 smithi115 conmon[90393]: 2022-05-14T03:05:03.233+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:05:03.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:05:03 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:05:04.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:05:04 smithi115 conmon[90393]: 2022-05-14T03:05:04.202+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:05:04.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:05:04 smithi115 ceph-mon[78593]: pgmap v6870: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:05:04.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:05:04 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:05:04.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:05:04 smithi047 ceph-mon[77147]: pgmap v6870: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:05:04.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:05:04 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:05:05.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:05:05 smithi115 conmon[90393]: 2022-05-14T03:05:05.177+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:05:05.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:05:05 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:05:05.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:05:05 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:05:06.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:05:06 smithi115 conmon[90393]: 2022-05-14T03:05:06.197+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:05:06.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:05:06 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:05:06.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:05:06 smithi115 ceph-mon[78593]: pgmap v6871: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:05:06.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:05:06 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 13040 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:05:06.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:05:06 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:05:06.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:05:06 smithi047 ceph-mon[77147]: pgmap v6871: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:05:06.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:05:06 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 13040 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:05:07.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:05:07 smithi115 conmon[90393]: 2022-05-14T03:05:07.197+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:05:07.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:05:07 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:05:07.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:05:07 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:05:08.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:05:08 smithi115 conmon[90393]: 2022-05-14T03:05:08.151+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:05:08.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:05:08 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:05:08.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:05:08 smithi115 ceph-mon[78593]: pgmap v6872: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:05:08.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:05:08 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:05:08.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:05:08 smithi047 ceph-mon[77147]: pgmap v6872: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:05:09.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:05:09 smithi115 conmon[90393]: 2022-05-14T03:05:09.172+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:05:09.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:05:09 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:05:09.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:05:09 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:05:10.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:05:10 smithi115 conmon[90393]: 2022-05-14T03:05:10.167+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:05:10.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:05:10 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:05:10.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:05:10 smithi115 ceph-mon[78593]: pgmap v6873: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:05:10.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:05:10 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:05:10.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:05:10 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:05:10.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:05:10 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:05:10.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:05:10 smithi047 ceph-mon[77147]: pgmap v6873: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:05:10.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:05:10 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:05:10.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:05:10 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:05:11.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:05:11 smithi115 conmon[90393]: 2022-05-14T03:05:11.121+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:05:11.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:05:11 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:05:11.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:05:11 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 13045 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:05:11.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:05:11 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:05:11.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:05:11 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 13045 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:05:12.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:05:12 smithi115 conmon[90393]: 2022-05-14T03:05:12.161+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:05:12.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:05:12 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:05:12.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:05:12 smithi115 ceph-mon[78593]: pgmap v6874: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:05:12.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:05:12 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:05:12.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:05:12 smithi047 ceph-mon[77147]: pgmap v6874: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:05:13.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:05:13 smithi115 conmon[90393]: 2022-05-14T03:05:13.207+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:05:13.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:05:13 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:05:13.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:05:13 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:05:14.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:05:14 smithi115 conmon[90393]: 2022-05-14T03:05:14.169+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:05:14.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:05:14 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:05:14.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:05:14 smithi115 ceph-mon[78593]: pgmap v6875: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:05:14.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:05:14 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:05:14.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:05:14 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:05:14.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:05:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T03:05:14.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:05:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T03:05:14.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:05:14 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:05:14.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:05:14 smithi047 ceph-mon[77147]: pgmap v6875: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:05:14.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:05:14 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:05:14.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:05:14 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:05:14.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:05:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T03:05:14.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:05:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T03:05:15.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:05:15 smithi115 conmon[90393]: 2022-05-14T03:05:15.123+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:05:15.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:05:15 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:05:15.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:05:15 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:05:16.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:05:16 smithi115 conmon[90393]: 2022-05-14T03:05:16.167+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:05:16.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:05:16 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:05:16.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:05:16 smithi115 ceph-mon[78593]: pgmap v6876: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:05:16.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:05:16 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 13050 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:05:16.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:05:16 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:05:16.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:05:16 smithi047 ceph-mon[77147]: pgmap v6876: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:05:16.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:05:16 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 13050 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:05:17.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:05:17 smithi115 conmon[90393]: 2022-05-14T03:05:17.200+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:05:17.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:05:17 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:05:17.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:05:17 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:05:18.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:05:18 smithi115 conmon[90393]: 2022-05-14T03:05:18.226+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:05:18.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:05:18 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:05:18.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:05:18 smithi115 ceph-mon[78593]: pgmap v6877: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:05:18.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:05:18 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:05:18.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:05:18 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:05:18.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:05:18 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:05:18.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:05:18 smithi047 ceph-mon[77147]: pgmap v6877: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:05:18.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:05:18 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:05:18.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:05:18 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:05:19.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:05:19 smithi115 conmon[90393]: 2022-05-14T03:05:19.208+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:05:19.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:05:19 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:05:19.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:05:19 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:05:20.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:05:20 smithi115 conmon[90393]: 2022-05-14T03:05:20.251+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:05:20.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:05:20 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:05:20.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:05:20 smithi115 ceph-mon[78593]: pgmap v6878: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:05:20.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:05:20 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:05:20.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:05:20 smithi047 ceph-mon[77147]: pgmap v6878: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:05:21.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:05:21 smithi115 conmon[90393]: 2022-05-14T03:05:21.206+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:05:21.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:05:21 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:05:21.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:05:21 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 13055 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:05:21.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:05:21 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:05:21.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:05:21 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 13055 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:05:22.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:05:22 smithi115 conmon[90393]: 2022-05-14T03:05:22.204+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:05:22.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:05:22 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:05:22.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:05:22 smithi115 ceph-mon[78593]: pgmap v6879: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:05:22.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:05:22 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:05:22.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:05:22 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:05:22.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:05:22 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:05:22.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:05:22 smithi047 ceph-mon[77147]: pgmap v6879: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:05:22.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:05:22 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:05:22.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:05:22 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:05:23.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:05:23 smithi115 conmon[90393]: 2022-05-14T03:05:23.158+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:05:23.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:05:23 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:05:23.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:05:23 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:05:24.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:05:24 smithi115 conmon[90393]: 2022-05-14T03:05:24.132+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:05:24.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:05:24 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:05:24.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:05:24 smithi115 ceph-mon[78593]: pgmap v6880: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:05:24.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:05:24 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:05:24.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:05:24 smithi047 ceph-mon[77147]: pgmap v6880: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:05:25.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:05:25 smithi115 conmon[90393]: 2022-05-14T03:05:25.154+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:05:25.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:05:25 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:05:25.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:05:25 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:05:26.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:05:26 smithi115 conmon[90393]: 2022-05-14T03:05:26.197+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:05:26.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:05:26 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:05:26.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:05:26 smithi115 ceph-mon[78593]: pgmap v6881: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:05:26.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:05:26 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 13060 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:05:26.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:05:26 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:05:26.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:05:26 smithi047 ceph-mon[77147]: pgmap v6881: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:05:26.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:05:26 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 13060 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:05:27.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:05:27 smithi115 conmon[90393]: 2022-05-14T03:05:27.238+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:05:27.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:05:27 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:05:27.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:05:27 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:05:28.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:05:28 smithi115 conmon[90393]: 2022-05-14T03:05:28.285+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:05:28.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:05:28 smithi115 ceph-mon[78593]: pgmap v6882: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:05:28.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:05:28 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:05:28.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:05:28 smithi047 ceph-mon[77147]: pgmap v6882: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:05:28.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:05:28 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:05:29.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:05:29 smithi115 conmon[90393]: 2022-05-14T03:05:29.258+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:05:29.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:05:29 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:05:29.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:05:29 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:05:30.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:05:30 smithi047 ceph-mon[77147]: pgmap v6883: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:05:30.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:05:30 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:05:30.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:05:30 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:05:30.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:05:30 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:05:30.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:05:30 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T03:05:30.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:05:30 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T03:05:30.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:05:30 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T03:05:30.588 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:05:30 smithi115 ceph-mon[78593]: pgmap v6883: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:05:30.589 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:05:30 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:05:30.589 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:05:30 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:05:30.589 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:05:30 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:05:30.589 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:05:30 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T03:05:30.590 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:05:30 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T03:05:30.590 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:05:30 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T03:05:30.590 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:05:30 smithi115 conmon[90393]: 2022-05-14T03:05:30.209+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:05:31.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:05:31 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:05:31.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:05:31 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' 2022-05-14T03:05:31.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:05:31 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' 2022-05-14T03:05:31.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:05:31 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 13065 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:05:31.610 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:05:31 smithi115 conmon[90393]: 2022-05-14T03:05:31.170+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:05:31.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:05:31 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:05:31.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:05:31 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' 2022-05-14T03:05:31.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:05:31 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' 2022-05-14T03:05:31.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:05:31 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 13065 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:05:32.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:05:32 smithi115 conmon[90393]: 2022-05-14T03:05:32.202+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:05:32.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:05:32 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:05:32.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:05:32 smithi115 ceph-mon[78593]: pgmap v6884: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:05:32.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:05:32 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:05:32.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:05:32 smithi047 ceph-mon[77147]: pgmap v6884: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:05:33.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:05:33 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:05:33.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:05:33 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' 2022-05-14T03:05:33.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:05:33 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' 2022-05-14T03:05:33.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:05:33 smithi047 conmon[78521]: 2022-05-14T03:05:33.233+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:05:33.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:05:33 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T03:05:33.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:05:33 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:05:33.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:05:33 smithi047 conmon[78521]: yield 2022-05-14T03:05:33.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:05:33 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:05:33.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:05:33 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:05:33.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:05:33 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:05:33.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:05:33 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T03:05:33.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:05:33 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:05:33.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:05:33 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T03:05:33.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:05:33 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:05:33.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:05:33 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T03:05:33.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:05:33 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:05:33.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:05:33 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T03:05:33.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:05:33 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:05:33.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:05:33 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T03:05:33.547 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:05:33 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:05:33.547 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:05:33 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:05:33.547 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:05:33 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:05:33.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:05:33 smithi115 conmon[90393]: 2022-05-14T03:05:33.237+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:05:33.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:05:33 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:05:33.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:05:33 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' 2022-05-14T03:05:33.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:05:33 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' 2022-05-14T03:05:34.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:05:34 smithi115 conmon[90393]: 2022-05-14T03:05:34.246+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:05:34.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:05:34 smithi115 ceph-mon[78593]: pgmap v6885: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:05:34.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:05:34 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:05:34.609 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T03:05:34.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:05:34.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T03:05:34.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:05:34.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:05:34.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:05:34.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T03:05:34.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:05:34.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T03:05:34.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:05:34.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T03:05:34.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:05:34.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T03:05:34.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:05:34.613 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T03:05:34.613 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:05:34.613 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:05:34.613 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:05:34.614 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:05:34 smithi115 ceph-mon[78593]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T03:05:34.614 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:05:34 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:05:34.614 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:05:34 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:05:34.614 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:05:34 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:05:34.614 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:05:34 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-05-14T03:05:34.615 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:05:34 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-05-14T03:05:34.615 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:05:34 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-05-14T03:05:34.615 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:05:34 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-05-14T03:05:34.615 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:05:34 smithi115 ceph-mon[78593]: Adjusting osd_memory_target on smithi115 to 197.0M 2022-05-14T03:05:34.615 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:05:34 smithi115 ceph-mon[78593]: Unable to set osd_memory_target on smithi115 to 206629785: error parsing value: Value '206629785' is below minimum 939524096 2022-05-14T03:05:34.616 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:05:34 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-05-14T03:05:34.616 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:05:34 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-05-14T03:05:34.616 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:05:34 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-05-14T03:05:34.616 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:05:34 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-05-14T03:05:34.617 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:05:34 smithi115 ceph-mon[78593]: Adjusting osd_memory_target on smithi047 to 172.7M 2022-05-14T03:05:34.617 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:05:34 smithi115 ceph-mon[78593]: Unable to set osd_memory_target on smithi047 to 181180768: error parsing value: Value '181180768' is below minimum 939524096 2022-05-14T03:05:34.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:05:34 smithi047 ceph-mon[77147]: pgmap v6885: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:05:34.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:05:34 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:05:34.792 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T03:05:34.793 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:05:34.793 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T03:05:34.793 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:05:34.793 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:05:34.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:05:34.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T03:05:34.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:05:34.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T03:05:34.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:05:34.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T03:05:34.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:05:34.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T03:05:34.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:05:34.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T03:05:34.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:05:34.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:05:34.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:05:34.796 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:05:34 smithi047 ceph-mon[77147]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T03:05:34.796 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:05:34 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:05:34.797 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:05:34 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:05:34.797 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:05:34 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:05:34.797 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:05:34 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-05-14T03:05:34.797 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:05:34 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-05-14T03:05:34.798 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:05:34 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-05-14T03:05:34.798 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:05:34 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-05-14T03:05:34.798 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:05:34 smithi047 ceph-mon[77147]: Adjusting osd_memory_target on smithi115 to 197.0M 2022-05-14T03:05:34.798 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:05:34 smithi047 ceph-mon[77147]: Unable to set osd_memory_target on smithi115 to 206629785: error parsing value: Value '206629785' is below minimum 939524096 2022-05-14T03:05:34.798 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:05:34 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-05-14T03:05:34.799 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:05:34 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-05-14T03:05:34.799 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:05:34 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-05-14T03:05:34.799 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:05:34 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-05-14T03:05:34.800 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:05:34 smithi047 ceph-mon[77147]: Adjusting osd_memory_target on smithi047 to 172.7M 2022-05-14T03:05:34.800 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:05:34 smithi047 ceph-mon[77147]: Unable to set osd_memory_target on smithi047 to 181180768: error parsing value: Value '181180768' is below minimum 939524096 2022-05-14T03:05:35.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:05:35 smithi115 conmon[90393]: 2022-05-14T03:05:35.294+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:05:35.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:05:35 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:05:35.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:05:35 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:05:36.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:05:36 smithi115 conmon[90393]: 2022-05-14T03:05:36.303+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:05:36.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:05:36 smithi115 ceph-mon[78593]: pgmap v6886: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:05:36.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:05:36 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:05:36.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:05:36 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 13070 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:05:36.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:05:36 smithi047 ceph-mon[77147]: pgmap v6886: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:05:36.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:05:36 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:05:36.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:05:36 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 13070 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:05:36.793 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:05:36 smithi047 conmon[78521]: 2022-05-14T03:05:36.305+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:05:36.793 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:05:36 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T03:05:36.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:05:36 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:05:36.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:05:36 smithi047 conmon[78521]: yield 2022-05-14T03:05:36.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:05:36 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:05:36.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:05:36 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:05:36.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:05:36 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:05:36.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:05:36 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T03:05:36.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:05:36 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:05:36.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:05:36 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T03:05:36.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:05:36 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:05:36.797 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:05:36 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T03:05:36.797 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:05:36 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:05:36.797 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:05:36 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T03:05:36.798 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:05:36 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:05:36.798 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:05:36 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T03:05:36.799 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:05:36 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:05:36.799 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:05:36 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:05:36.799 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:05:36 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:05:36.800 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:05:36 smithi047 conmon[78521]: 2022-05-14T03:05:36.305+0000 7f534177b700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T03:05:36.801 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:05:36 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T03:05:36.801 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:05:36 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:05:36.802 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:05:36 smithi047 conmon[78521]: yield 2022-05-14T03:05:36.802 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:05:36 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:05:36.802 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:05:36 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:05:36.803 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:05:36 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:05:36.804 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:05:36 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T03:05:36.804 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:05:36 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:05:36.804 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:05:36 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T03:05:36.805 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:05:36 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:05:36.805 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:05:36 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T03:05:36.805 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:05:36 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:05:36.806 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:05:36 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T03:05:36.806 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:05:36 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:05:36.807 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:05:36 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T03:05:36.807 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:05:36 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:05:36.807 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:05:36 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:05:36.808 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:05:36 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:05:36.808 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:05:36 smithi047 conmon[78521]: 2022-05-14T03:05:36.808 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:05:36 smithi047 conmon[78521]: During handling of the above exception, another exception occurred: 2022-05-14T03:05:36.809 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:05:36 smithi047 conmon[78521]: 2022-05-14T03:05:36.809 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:05:36 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T03:05:36.809 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:05:36 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T03:05:36.810 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:05:36 smithi047 conmon[78521]: return f(*arg) 2022-05-14T03:05:36.810 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:05:36 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T03:05:36.810 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:05:36 smithi047 conmon[78521]: self._write_client_files(client_files, host) 2022-05-14T03:05:36.811 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:05:36 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T03:05:36.811 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:05:36 smithi047 conmon[78521]: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T03:05:36.811 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:05:36 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T03:05:36.812 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:05:36 smithi047 conmon[78521]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T03:05:36.812 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:05:36 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T03:05:36.812 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:05:36 smithi047 conmon[78521]: return self.event_loop.get_result(coro) 2022-05-14T03:05:36.812 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:05:36 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T03:05:36.813 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:05:36 smithi047 conmon[78521]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T03:05:36.813 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:05:36 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T03:05:36.813 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:05:36 smithi047 conmon[78521]: return self.__get_result() 2022-05-14T03:05:36.814 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:05:36 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T03:05:36.814 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:05:36 smithi047 conmon[78521]: raise self._exception 2022-05-14T03:05:36.814 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:05:36 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T03:05:36.815 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:05:36 smithi047 conmon[78521]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T03:05:36.815 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:05:36 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T03:05:36.815 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:05:36 smithi047 conmon[78521]: conn = await self._remote_connection(host, addr) 2022-05-14T03:05:36.815 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:05:36 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T03:05:36.816 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:05:36 smithi047 conmon[78521]: raise 2022-05-14T03:05:36.816 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:05:36 smithi047 conmon[78521]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T03:05:36.816 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:05:36 smithi047 conmon[78521]: self.gen.throw(type, value, traceback) 2022-05-14T03:05:36.816 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:05:36 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T03:05:36.817 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:05:36 smithi047 conmon[78521]: raise OrchestratorError(msg) 2022-05-14T03:05:36.817 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:05:36 smithi047 conmon[78521]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:05:37.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:05:37 smithi115 conmon[90393]: 2022-05-14T03:05:37.309+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:05:37.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:05:37 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:05:37.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:05:37 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:05:37.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T03:05:37.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:05:37.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T03:05:37.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:05:37.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:05:37.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:05:37.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T03:05:37.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:05:37.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T03:05:37.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:05:37.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T03:05:37.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:05:37.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T03:05:37.613 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:05:37.613 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T03:05:37.613 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:05:37.613 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:05:37.614 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:05:37.614 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:05:37 smithi115 ceph-mon[78593]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T03:05:37.614 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T03:05:37.614 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:05:37.614 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T03:05:37.615 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:05:37.615 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:05:37.615 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:05:37.615 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T03:05:37.616 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:05:37.616 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T03:05:37.616 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:05:37.616 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T03:05:37.616 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:05:37.617 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T03:05:37.617 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:05:37.617 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T03:05:37.617 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:05:37.618 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:05:37.618 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:05:37.618 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T03:05:37.618 INFO:journalctl@ceph.mon.c.smithi115.stdout: During handling of the above exception, another exception occurred: 2022-05-14T03:05:37.618 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T03:05:37.619 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T03:05:37.619 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T03:05:37.619 INFO:journalctl@ceph.mon.c.smithi115.stdout: return f(*arg) 2022-05-14T03:05:37.619 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T03:05:37.620 INFO:journalctl@ceph.mon.c.smithi115.stdout: self._write_client_files(client_files, host) 2022-05-14T03:05:37.620 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T03:05:37.620 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T03:05:37.620 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T03:05:37.620 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T03:05:37.621 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T03:05:37.621 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.event_loop.get_result(coro) 2022-05-14T03:05:37.621 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T03:05:37.621 INFO:journalctl@ceph.mon.c.smithi115.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T03:05:37.621 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T03:05:37.622 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.__get_result() 2022-05-14T03:05:37.622 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T03:05:37.622 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise self._exception 2022-05-14T03:05:37.622 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T03:05:37.623 INFO:journalctl@ceph.mon.c.smithi115.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T03:05:37.623 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T03:05:37.623 INFO:journalctl@ceph.mon.c.smithi115.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T03:05:37.623 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T03:05:37.624 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise 2022-05-14T03:05:37.624 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T03:05:37.624 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.gen.throw(type, value, traceback) 2022-05-14T03:05:37.624 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T03:05:37.624 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OrchestratorError(msg) 2022-05-14T03:05:37.625 INFO:journalctl@ceph.mon.c.smithi115.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:05:37.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:05:37 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:05:37.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:05:37 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:05:37.792 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T03:05:37.793 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:05:37.793 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T03:05:37.793 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:05:37.793 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:05:37.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:05:37.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T03:05:37.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:05:37.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T03:05:37.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:05:37.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T03:05:37.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:05:37.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T03:05:37.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:05:37.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T03:05:37.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:05:37.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:05:37.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:05:37.796 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:05:37 smithi047 ceph-mon[77147]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T03:05:37.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T03:05:37.797 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:05:37.797 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T03:05:37.797 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:05:37.797 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:05:37.798 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:05:37.798 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T03:05:37.798 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:05:37.798 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T03:05:37.798 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:05:37.799 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T03:05:37.799 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:05:37.799 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T03:05:37.799 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:05:37.800 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T03:05:37.800 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:05:37.800 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:05:37.800 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:05:37.800 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T03:05:37.801 INFO:journalctl@ceph.mon.b.smithi047.stdout: During handling of the above exception, another exception occurred: 2022-05-14T03:05:37.801 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T03:05:37.801 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T03:05:37.801 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T03:05:37.802 INFO:journalctl@ceph.mon.b.smithi047.stdout: return f(*arg) 2022-05-14T03:05:37.802 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T03:05:37.802 INFO:journalctl@ceph.mon.b.smithi047.stdout: self._write_client_files(client_files, host) 2022-05-14T03:05:37.802 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T03:05:37.802 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T03:05:37.803 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T03:05:37.803 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T03:05:37.803 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T03:05:37.803 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.event_loop.get_result(coro) 2022-05-14T03:05:37.804 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T03:05:37.804 INFO:journalctl@ceph.mon.b.smithi047.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T03:05:37.804 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T03:05:37.804 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.__get_result() 2022-05-14T03:05:37.804 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T03:05:37.805 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise self._exception 2022-05-14T03:05:37.805 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T03:05:37.805 INFO:journalctl@ceph.mon.b.smithi047.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T03:05:37.805 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T03:05:37.805 INFO:journalctl@ceph.mon.b.smithi047.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T03:05:37.806 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T03:05:37.806 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise 2022-05-14T03:05:37.806 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T03:05:37.806 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.gen.throw(type, value, traceback) 2022-05-14T03:05:37.806 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T03:05:37.807 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OrchestratorError(msg) 2022-05-14T03:05:37.807 INFO:journalctl@ceph.mon.b.smithi047.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:05:38.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:05:38 smithi115 conmon[90393]: 2022-05-14T03:05:38.336+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:05:38.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:05:38 smithi115 ceph-mon[78593]: pgmap v6887: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:05:38.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:05:38 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:05:38.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:05:38 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:05:38.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:05:38 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:05:38.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:05:38 smithi047 ceph-mon[77147]: pgmap v6887: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:05:38.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:05:38 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:05:38.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:05:38 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:05:38.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:05:38 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:05:39.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:05:39 smithi115 conmon[90393]: 2022-05-14T03:05:39.304+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:05:39.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:05:39 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:05:39.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:05:39 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:05:40.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:05:40 smithi115 conmon[90393]: 2022-05-14T03:05:40.299+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:05:40.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:05:40 smithi115 ceph-mon[78593]: pgmap v6888: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:05:40.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:05:40 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:05:40.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:05:40 smithi047 ceph-mon[77147]: pgmap v6888: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:05:40.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:05:40 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:05:41.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:05:41 smithi115 conmon[90393]: 2022-05-14T03:05:41.286+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:05:41.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:05:41 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:05:41.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:05:41 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 13075 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:05:41.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:05:41 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:05:41.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:05:41 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 13075 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:05:42.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:05:42 smithi115 conmon[90393]: 2022-05-14T03:05:42.259+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:05:42.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:05:42 smithi115 ceph-mon[78593]: pgmap v6889: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:05:42.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:05:42 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:05:42.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:05:42 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:05:42.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:05:42 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:05:42.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:05:42 smithi047 ceph-mon[77147]: pgmap v6889: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:05:42.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:05:42 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:05:42.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:05:42 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:05:42.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:05:42 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:05:43.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:05:43 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:05:43.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:05:43 smithi115 conmon[90393]: 2022-05-14T03:05:43.244+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:05:43.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:05:43 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:05:44.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:05:44 smithi115 conmon[90393]: 2022-05-14T03:05:44.266+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:05:44.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:05:44 smithi115 ceph-mon[78593]: pgmap v6890: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:05:44.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:05:44 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:05:44.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:05:44 smithi047 ceph-mon[77147]: pgmap v6890: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:05:44.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:05:44 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:05:45.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:05:45 smithi115 conmon[90393]: 2022-05-14T03:05:45.274+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:05:45.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:05:45 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:05:45.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:05:45 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:05:46.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:05:46 smithi115 conmon[90393]: 2022-05-14T03:05:46.278+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:05:46.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:05:46 smithi115 ceph-mon[78593]: pgmap v6891: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:05:46.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:05:46 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:05:46.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:05:46 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 13080 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:05:46.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:05:46 smithi047 ceph-mon[77147]: pgmap v6891: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:05:46.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:05:46 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:05:46.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:05:46 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 13080 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:05:47.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:05:47 smithi115 conmon[90393]: 2022-05-14T03:05:47.235+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:05:47.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:05:47 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:05:47.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:05:47 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:05:48.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:05:48 smithi115 conmon[90393]: 2022-05-14T03:05:48.261+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:05:48.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:05:48 smithi115 ceph-mon[78593]: pgmap v6892: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:05:48.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:05:48 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:05:48.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:05:48 smithi047 ceph-mon[77147]: pgmap v6892: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:05:48.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:05:48 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:05:49.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:05:49 smithi115 conmon[90393]: 2022-05-14T03:05:49.217+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:05:49.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:05:49 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:05:49.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:05:49 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:05:50.456 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:05:50 smithi115 conmon[90393]: 2022-05-14T03:05:50.186+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:05:50.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:05:50 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:05:50.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:05:50 smithi047 ceph-mon[77147]: pgmap v6893: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:05:50.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:05:50 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:05:50.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:05:50 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:05:50.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:05:50 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:05:50.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:05:50 smithi115 ceph-mon[78593]: pgmap v6893: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:05:50.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:05:50 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:05:50.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:05:50 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:05:51.460 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:05:51 smithi115 conmon[90393]: 2022-05-14T03:05:51.193+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:05:51.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:05:51 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:05:51.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:05:51 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 13085 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:05:51.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:05:51 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:05:51.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:05:51 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 13085 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:05:52.461 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:05:52 smithi115 conmon[90393]: 2022-05-14T03:05:52.160+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:05:52.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:05:52 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:05:52.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:05:52 smithi047 ceph-mon[77147]: pgmap v6894: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:05:52.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:05:52 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:05:52.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:05:52 smithi115 ceph-mon[78593]: pgmap v6894: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:05:53.467 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:05:53 smithi115 conmon[90393]: 2022-05-14T03:05:53.187+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:05:53.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:05:53 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:05:54.466 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:05:54 smithi115 conmon[90393]: 2022-05-14T03:05:54.183+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:05:54.466 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:05:53 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:05:54.467 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:05:54 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:05:54.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:05:54 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:05:54.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:05:54 smithi047 ceph-mon[77147]: pgmap v6895: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:05:54.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:05:54 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:05:54.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:05:54 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:05:54.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:05:54 smithi115 ceph-mon[78593]: pgmap v6895: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:05:54.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:05:54 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:05:54.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:05:54 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:05:55.470 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:05:55 smithi115 conmon[90393]: 2022-05-14T03:05:55.198+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:05:55.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:05:55 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:05:55.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:05:55 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:05:56.472 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:05:56 smithi115 conmon[90393]: 2022-05-14T03:05:56.211+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:05:56.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:05:56 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:05:56.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:05:56 smithi047 ceph-mon[77147]: pgmap v6896: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:05:56.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:05:56 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 13090 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:05:56.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:05:56 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:05:56.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:05:56 smithi115 ceph-mon[78593]: pgmap v6896: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:05:56.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:05:56 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 13090 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:05:57.478 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:05:57 smithi115 conmon[90393]: 2022-05-14T03:05:57.195+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:05:57.478 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:05:57 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:05:57.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:05:57 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:05:58.479 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:05:58 smithi115 conmon[90393]: 2022-05-14T03:05:58.201+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:05:58.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:05:58 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:05:58.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:05:58 smithi047 ceph-mon[77147]: pgmap v6897: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:05:58.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:05:58 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:05:58.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:05:58 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:05:58.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:05:58 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:05:58.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:05:58 smithi115 ceph-mon[78593]: pgmap v6897: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:05:58.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:05:58 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:05:58.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:05:58 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:05:59.484 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:05:59 smithi115 conmon[90393]: 2022-05-14T03:05:59.225+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:05:59.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:05:59 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:05:59.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:05:59 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:06:00.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:06:00 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:06:00.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:06:00 smithi115 ceph-mon[78593]: pgmap v6898: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:06:00.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:06:00 smithi115 conmon[90393]: 2022-05-14T03:06:00.242+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:06:00.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:06:00 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:06:00.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:06:00 smithi047 ceph-mon[77147]: pgmap v6898: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:06:01.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:06:01 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:06:01.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:06:01 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 13095 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:06:01.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:06:01 smithi115 conmon[90393]: 2022-05-14T03:06:01.253+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:06:01.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:06:01 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:06:01.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:06:01 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 13095 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:06:02.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:06:02 smithi115 conmon[90393]: 2022-05-14T03:06:02.303+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:06:02.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:06:02 smithi115 ceph-mon[78593]: pgmap v6899: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:06:02.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:06:02 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:06:02.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:06:02 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:06:02.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:06:02 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:06:02.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:06:02 smithi047 ceph-mon[77147]: pgmap v6899: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:06:02.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:06:02 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:06:02.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:06:02 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:06:02.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:06:02 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:06:03.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:06:03 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:06:03.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:06:03 smithi115 conmon[90393]: 2022-05-14T03:06:03.282+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:06:03.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:06:03 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:06:04.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:06:04 smithi115 conmon[90393]: 2022-05-14T03:06:04.281+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:06:04.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:06:04 smithi115 ceph-mon[78593]: pgmap v6900: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:06:04.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:06:04 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:06:04.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:06:04 smithi047 ceph-mon[77147]: pgmap v6900: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:06:04.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:06:04 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:06:05.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:06:05 smithi115 conmon[90393]: 2022-05-14T03:06:05.308+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:06:05.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:06:05 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:06:05.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:06:05 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:06:06.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:06:06 smithi115 conmon[90393]: 2022-05-14T03:06:06.317+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:06:06.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:06:06 smithi115 ceph-mon[78593]: pgmap v6901: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:06:06.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:06:06 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:06:06.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:06:06 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 13100 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:06:06.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:06:06 smithi047 ceph-mon[77147]: pgmap v6901: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:06:06.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:06:06 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:06:06.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:06:06 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 13100 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:06:07.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:06:07 smithi115 conmon[90393]: 2022-05-14T03:06:07.299+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:06:07.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:06:07 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:06:07.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:06:07 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:06:08.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:06:08 smithi115 conmon[90393]: 2022-05-14T03:06:08.323+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:06:08.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:06:08 smithi115 ceph-mon[78593]: pgmap v6902: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:06:08.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:06:08 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:06:08.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:06:08 smithi047 ceph-mon[77147]: pgmap v6902: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:06:08.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:06:08 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:06:09.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:06:09 smithi115 conmon[90393]: 2022-05-14T03:06:09.304+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:06:09.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:06:09 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:06:09.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:06:09 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:06:10.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:06:10 smithi115 conmon[90393]: 2022-05-14T03:06:10.309+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:06:10.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:06:10 smithi115 ceph-mon[78593]: pgmap v6903: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:06:10.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:06:10 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:06:10.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:06:10 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:06:10.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:06:10 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:06:10.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:06:10 smithi047 ceph-mon[77147]: pgmap v6903: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:06:10.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:06:10 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:06:10.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:06:10 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:06:10.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:06:10 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:06:11.517 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:06:11 smithi115 conmon[90393]: 2022-05-14T03:06:11.265+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:06:11.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:06:11 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:06:11.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:06:11 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 13105 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:06:11.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:06:11 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:06:11.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:06:11 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 13105 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:06:12.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:06:12 smithi115 conmon[90393]: 2022-05-14T03:06:12.287+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:06:12.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:06:12 smithi115 ceph-mon[78593]: pgmap v6904: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:06:12.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:06:12 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:06:12.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:06:12 smithi047 ceph-mon[77147]: pgmap v6904: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:06:12.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:06:12 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:06:13.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:06:13 smithi115 conmon[90393]: 2022-05-14T03:06:13.295+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:06:13.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:06:13 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:06:13.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:06:13 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:06:14.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:06:14 smithi115 conmon[90393]: 2022-05-14T03:06:14.333+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:06:14.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:06:14 smithi115 ceph-mon[78593]: pgmap v6905: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:06:14.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:06:14 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:06:14.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:06:14 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:06:14.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:06:14 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:06:14.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:06:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T03:06:14.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:06:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T03:06:14.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:06:14 smithi047 ceph-mon[77147]: pgmap v6905: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:06:14.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:06:14 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:06:14.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:06:14 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:06:14.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:06:14 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:06:14.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:06:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T03:06:14.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:06:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T03:06:15.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:06:15 smithi115 conmon[90393]: 2022-05-14T03:06:15.308+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:06:15.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:06:15 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:06:15.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:06:15 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:06:16.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:06:16 smithi115 conmon[90393]: 2022-05-14T03:06:16.334+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:06:16.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:06:16 smithi115 ceph-mon[78593]: pgmap v6906: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:06:16.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:06:16 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:06:16.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:06:16 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 13110 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:06:16.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:06:16 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:06:16.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:06:16 smithi047 ceph-mon[77147]: pgmap v6906: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:06:16.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:06:16 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:06:16.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:06:16 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 13110 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:06:16.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:06:16 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:06:17.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:06:17 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:06:17.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:06:17 smithi115 conmon[90393]: 2022-05-14T03:06:17.375+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:06:17.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:06:17 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:06:18.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:06:18 smithi047 ceph-mon[77147]: pgmap v6907: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:06:18.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:06:18 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:06:18.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:06:18 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:06:18.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:06:18 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:06:18.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:06:18 smithi115 conmon[90393]: 2022-05-14T03:06:18.422+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:06:18.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:06:18 smithi115 ceph-mon[78593]: pgmap v6907: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:06:18.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:06:18 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:06:18.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:06:18 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:06:18.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:06:18 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:06:19.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:06:19 smithi115 conmon[90393]: 2022-05-14T03:06:19.381+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:06:19.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:06:19 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:06:20.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:06:19 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:06:20.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:06:20 smithi115 conmon[90393]: 2022-05-14T03:06:20.410+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:06:20.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:06:20 smithi115 ceph-mon[78593]: pgmap v6908: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:06:20.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:06:20 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:06:21.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:06:20 smithi047 ceph-mon[77147]: pgmap v6908: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:06:21.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:06:20 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:06:21.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:06:21 smithi115 conmon[90393]: 2022-05-14T03:06:21.459+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:06:21.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:06:21 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 13115 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:06:21.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:06:21 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:06:22.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:06:21 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 13115 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:06:22.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:06:21 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:06:22.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:06:22 smithi115 conmon[90393]: 2022-05-14T03:06:22.449+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:06:22.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:06:22 smithi115 ceph-mon[78593]: pgmap v6909: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:06:22.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:06:22 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:06:22.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:06:22 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:06:22.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:06:22 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:06:23.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:06:22 smithi047 ceph-mon[77147]: pgmap v6909: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:06:23.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:06:22 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:06:23.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:06:22 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:06:23.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:06:22 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:06:23.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:06:23 smithi115 conmon[90393]: 2022-05-14T03:06:23.452+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:06:23.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:06:23 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:06:24.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:06:23 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:06:24.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:06:24 smithi115 conmon[90393]: 2022-05-14T03:06:24.427+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:06:24.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:06:24 smithi115 ceph-mon[78593]: pgmap v6910: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:06:24.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:06:24 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:06:25.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:06:24 smithi047 ceph-mon[77147]: pgmap v6910: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:06:25.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:06:24 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:06:25.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:06:25 smithi115 conmon[90393]: 2022-05-14T03:06:25.443+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:06:25.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:06:25 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:06:26.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:06:25 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:06:26.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:06:26 smithi115 conmon[90393]: 2022-05-14T03:06:26.473+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:06:26.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:06:26 smithi115 ceph-mon[78593]: pgmap v6911: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:06:26.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:06:26 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 13120 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:06:26.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:06:26 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:06:27.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:06:26 smithi047 ceph-mon[77147]: pgmap v6911: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:06:27.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:06:26 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 13120 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:06:27.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:06:26 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:06:27.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:06:27 smithi115 conmon[90393]: 2022-05-14T03:06:27.489+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:06:27.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:06:27 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:06:28.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:06:27 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:06:28.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:06:28 smithi115 conmon[90393]: 2022-05-14T03:06:28.458+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:06:28.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:06:28 smithi115 ceph-mon[78593]: pgmap v6912: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:06:28.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:06:28 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:06:29.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:06:28 smithi047 ceph-mon[77147]: pgmap v6912: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:06:29.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:06:28 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:06:29.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:06:29 smithi115 conmon[90393]: 2022-05-14T03:06:29.498+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:06:29.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:06:29 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:06:30.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:06:29 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:06:30.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:06:30 smithi115 conmon[90393]: 2022-05-14T03:06:30.485+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:06:30.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:06:30 smithi115 ceph-mon[78593]: pgmap v6913: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:06:30.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:06:30 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:06:30.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:06:30 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:06:30.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:06:30 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:06:31.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:06:30 smithi047 ceph-mon[77147]: pgmap v6913: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:06:31.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:06:30 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:06:31.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:06:30 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:06:31.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:06:30 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:06:31.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:06:31 smithi115 conmon[90393]: 2022-05-14T03:06:31.465+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:06:31.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:06:31 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 13125 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:06:31.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:06:31 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:06:32.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:06:31 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 13125 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:06:32.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:06:31 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:06:32.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:06:32 smithi115 conmon[90393]: 2022-05-14T03:06:32.418+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:06:32.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:06:32 smithi115 ceph-mon[78593]: pgmap v6914: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:06:32.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:06:32 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:06:33.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:06:32 smithi047 ceph-mon[77147]: pgmap v6914: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:06:33.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:06:32 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:06:33.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:06:33 smithi115 conmon[90393]: 2022-05-14T03:06:33.401+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:06:33.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:06:33 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:06:34.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:06:33 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:06:34.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:06:34 smithi115 conmon[90393]: 2022-05-14T03:06:34.409+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:06:34.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:06:34 smithi115 ceph-mon[78593]: pgmap v6915: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:06:34.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:06:34 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:06:34.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:06:34 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:06:34.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:06:34 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:06:35.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:06:34 smithi047 ceph-mon[77147]: pgmap v6915: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:06:35.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:06:34 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:06:35.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:06:34 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:06:35.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:06:34 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:06:35.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:06:35 smithi115 conmon[90393]: 2022-05-14T03:06:35.384+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:06:35.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:06:35 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:06:36.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:06:35 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:06:36.594 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:06:36 smithi115 conmon[90393]: 2022-05-14T03:06:36.340+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:06:36.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:06:36 smithi047 ceph-mon[77147]: pgmap v6916: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:06:36.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:06:36 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 13130 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:06:36.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:06:36 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T03:06:36.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:06:36 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T03:06:36.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:06:36 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T03:06:36.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:06:36 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:06:36.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:06:36 smithi115 ceph-mon[78593]: pgmap v6916: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:06:36.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:06:36 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 13130 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:06:36.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:06:36 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T03:06:36.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:06:36 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T03:06:36.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:06:36 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T03:06:36.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:06:36 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:06:37.292 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:06:36 smithi047 conmon[78521]: 2022-05-14T03:06:36.911+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:06:37.292 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:06:36 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T03:06:37.292 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:06:36 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:06:37.292 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:06:36 smithi047 conmon[78521]: yield 2022-05-14T03:06:37.293 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:06:36 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:06:37.293 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:06:36 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:06:37.293 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:06:36 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:06:37.293 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:06:36 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T03:06:37.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:06:36 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:06:37.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:06:36 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T03:06:37.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:06:36 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:06:37.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:06:36 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T03:06:37.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:06:36 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:06:37.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:06:36 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T03:06:37.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:06:36 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:06:37.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:06:36 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T03:06:37.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:06:36 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:06:37.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:06:36 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:06:37.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:06:36 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:06:37.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:06:37 smithi115 conmon[90393]: 2022-05-14T03:06:37.359+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:06:37.893 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:06:37 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:06:38.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:06:37 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:06:38.600 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:06:38 smithi115 conmon[90393]: 2022-05-14T03:06:38.325+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:06:38.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:06:38 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:06:38.859 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T03:06:38.859 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:06:38.859 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T03:06:38.859 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:06:38.860 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:06:38.860 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:06:38.860 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T03:06:38.860 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:06:38.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T03:06:38.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:06:38.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T03:06:38.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:06:38.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T03:06:38.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:06:38.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T03:06:38.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:06:38.863 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:06:38.863 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:06:38.863 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:06:38 smithi115 ceph-mon[78593]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T03:06:38.864 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:06:38 smithi115 ceph-mon[78593]: pgmap v6917: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:06:38.864 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:06:38 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:06:38.864 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:06:38 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:06:38.864 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:06:38 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:06:39.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:06:38 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:06:39.042 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T03:06:39.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:06:39.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T03:06:39.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:06:39.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:06:39.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:06:39.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T03:06:39.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:06:39.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T03:06:39.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:06:39.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T03:06:39.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:06:39.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T03:06:39.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:06:39.049 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T03:06:39.049 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:06:39.050 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:06:39.050 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:06:39.051 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:06:38 smithi047 ceph-mon[77147]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T03:06:39.051 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:06:38 smithi047 ceph-mon[77147]: pgmap v6917: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:06:39.051 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:06:38 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:06:39.052 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:06:38 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:06:39.052 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:06:38 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:06:39.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:06:39 smithi115 conmon[90393]: 2022-05-14T03:06:39.360+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:06:39.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:06:39 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:06:39.986 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:06:39 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:06:40.292 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:06:39 smithi047 conmon[78521]: 2022-05-14T03:06:39.983+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:06:40.292 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:06:39 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T03:06:40.292 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:06:39 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:06:40.293 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:06:39 smithi047 conmon[78521]: yield 2022-05-14T03:06:40.293 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:06:39 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:06:40.293 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:06:39 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:06:40.293 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:06:39 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:06:40.293 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:06:39 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T03:06:40.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:06:39 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:06:40.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:06:39 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T03:06:40.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:06:39 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:06:40.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:06:39 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T03:06:40.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:06:39 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:06:40.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:06:39 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T03:06:40.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:06:39 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:06:40.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:06:39 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T03:06:40.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:06:39 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:06:40.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:06:39 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:06:40.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:06:39 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:06:40.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:06:39 smithi047 conmon[78521]: 2022-05-14T03:06:39.984+0000 7f5345f84700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T03:06:40.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:06:39 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T03:06:40.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:06:39 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:06:40.297 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:06:39 smithi047 conmon[78521]: yield 2022-05-14T03:06:40.297 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:06:39 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:06:40.297 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:06:39 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:06:40.297 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:06:39 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:06:40.298 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:06:39 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T03:06:40.298 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:06:39 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:06:40.298 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:06:39 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T03:06:40.298 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:06:39 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:06:40.299 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:06:39 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T03:06:40.299 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:06:39 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:06:40.299 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:06:39 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T03:06:40.299 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:06:39 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:06:40.300 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:06:39 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T03:06:40.300 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:06:39 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:06:40.300 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:06:39 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:06:40.300 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:06:39 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:06:40.300 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:06:39 smithi047 conmon[78521]: 2022-05-14T03:06:40.301 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:06:39 smithi047 conmon[78521]: During handling of the above exception, another exception occurred: 2022-05-14T03:06:40.301 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:06:39 smithi047 conmon[78521]: 2022-05-14T03:06:40.301 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:06:39 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T03:06:40.301 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:06:39 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T03:06:40.302 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:06:39 smithi047 conmon[78521]: return f(*arg) 2022-05-14T03:06:40.302 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:06:39 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T03:06:40.302 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:06:39 smithi047 conmon[78521]: self._write_client_files(client_files, host) 2022-05-14T03:06:40.302 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:06:39 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T03:06:40.302 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:06:39 smithi047 conmon[78521]: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T03:06:40.303 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:06:39 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T03:06:40.303 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:06:39 smithi047 conmon[78521]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T03:06:40.303 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:06:39 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T03:06:40.303 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:06:39 smithi047 conmon[78521]: return self.event_loop.get_result(coro) 2022-05-14T03:06:40.304 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:06:39 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T03:06:40.304 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:06:39 smithi047 conmon[78521]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T03:06:40.304 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:06:39 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T03:06:40.304 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:06:39 smithi047 conmon[78521]: return self.__get_result() 2022-05-14T03:06:40.304 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:06:39 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T03:06:40.305 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:06:39 smithi047 conmon[78521]: raise self._exception 2022-05-14T03:06:40.305 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:06:39 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T03:06:40.305 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:06:39 smithi047 conmon[78521]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T03:06:40.306 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:06:39 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T03:06:40.306 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:06:39 smithi047 conmon[78521]: conn = await self._remote_connection(host, addr) 2022-05-14T03:06:40.306 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:06:39 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T03:06:40.306 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:06:39 smithi047 conmon[78521]: raise 2022-05-14T03:06:40.306 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:06:39 smithi047 conmon[78521]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T03:06:40.307 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:06:39 smithi047 conmon[78521]: self.gen.throw(type, value, traceback) 2022-05-14T03:06:40.307 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:06:39 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T03:06:40.307 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:06:39 smithi047 conmon[78521]: raise OrchestratorError(msg) 2022-05-14T03:06:40.307 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:06:39 smithi047 conmon[78521]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:06:40.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:06:40 smithi115 conmon[90393]: 2022-05-14T03:06:40.360+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:06:40.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:06:40 smithi115 ceph-mon[78593]: pgmap v6918: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:06:40.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:06:40 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:06:41.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:06:40 smithi047 ceph-mon[77147]: pgmap v6918: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:06:41.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:06:40 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:06:41.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:06:41 smithi115 conmon[90393]: 2022-05-14T03:06:41.316+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:06:42.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:06:41 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:06:42.042 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T03:06:42.042 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:06:42.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T03:06:42.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:06:42.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:06:42.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:06:42.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T03:06:42.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:06:42.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T03:06:42.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:06:42.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T03:06:42.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:06:42.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T03:06:42.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:06:42.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T03:06:42.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:06:42.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:06:42.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:06:42.047 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:06:41 smithi047 ceph-mon[77147]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T03:06:42.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T03:06:42.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:06:42.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T03:06:42.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:06:42.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:06:42.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:06:42.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T03:06:42.049 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:06:42.049 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T03:06:42.049 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:06:42.049 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T03:06:42.049 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:06:42.050 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T03:06:42.050 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:06:42.050 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T03:06:42.050 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:06:42.050 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:06:42.051 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:06:42.051 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T03:06:42.051 INFO:journalctl@ceph.mon.b.smithi047.stdout: During handling of the above exception, another exception occurred: 2022-05-14T03:06:42.051 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T03:06:42.052 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T03:06:42.052 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T03:06:42.052 INFO:journalctl@ceph.mon.b.smithi047.stdout: return f(*arg) 2022-05-14T03:06:42.052 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T03:06:42.052 INFO:journalctl@ceph.mon.b.smithi047.stdout: self._write_client_files(client_files, host) 2022-05-14T03:06:42.053 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T03:06:42.053 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T03:06:42.053 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T03:06:42.053 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T03:06:42.054 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T03:06:42.054 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.event_loop.get_result(coro) 2022-05-14T03:06:42.054 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T03:06:42.055 INFO:journalctl@ceph.mon.b.smithi047.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T03:06:42.055 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T03:06:42.055 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.__get_result() 2022-05-14T03:06:42.055 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T03:06:42.056 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise self._exception 2022-05-14T03:06:42.056 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T03:06:42.056 INFO:journalctl@ceph.mon.b.smithi047.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T03:06:42.056 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T03:06:42.056 INFO:journalctl@ceph.mon.b.smithi047.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T03:06:42.057 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T03:06:42.057 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise 2022-05-14T03:06:42.057 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T03:06:42.057 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.gen.throw(type, value, traceback) 2022-05-14T03:06:42.058 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T03:06:42.058 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OrchestratorError(msg) 2022-05-14T03:06:42.058 INFO:journalctl@ceph.mon.b.smithi047.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:06:42.058 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:06:41 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 13135 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:06:42.059 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:06:41 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:06:42.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:06:41 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:06:42.109 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T03:06:42.109 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:06:42.109 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T03:06:42.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:06:42.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:06:42.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:06:42.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T03:06:42.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:06:42.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T03:06:42.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:06:42.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T03:06:42.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:06:42.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T03:06:42.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:06:42.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T03:06:42.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:06:42.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:06:42.113 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:06:42.113 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:06:41 smithi115 ceph-mon[78593]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T03:06:42.113 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T03:06:42.113 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:06:42.114 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T03:06:42.114 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:06:42.114 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:06:42.114 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:06:42.114 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T03:06:42.115 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:06:42.115 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T03:06:42.115 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:06:42.115 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T03:06:42.115 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:06:42.116 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T03:06:42.116 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:06:42.116 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T03:06:42.117 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:06:42.117 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:06:42.117 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:06:42.118 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T03:06:42.118 INFO:journalctl@ceph.mon.c.smithi115.stdout: During handling of the above exception, another exception occurred: 2022-05-14T03:06:42.118 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T03:06:42.118 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T03:06:42.118 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T03:06:42.119 INFO:journalctl@ceph.mon.c.smithi115.stdout: return f(*arg) 2022-05-14T03:06:42.119 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T03:06:42.119 INFO:journalctl@ceph.mon.c.smithi115.stdout: self._write_client_files(client_files, host) 2022-05-14T03:06:42.119 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T03:06:42.119 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T03:06:42.120 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T03:06:42.120 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T03:06:42.120 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T03:06:42.120 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.event_loop.get_result(coro) 2022-05-14T03:06:42.121 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T03:06:42.121 INFO:journalctl@ceph.mon.c.smithi115.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T03:06:42.121 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T03:06:42.121 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.__get_result() 2022-05-14T03:06:42.121 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T03:06:42.122 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise self._exception 2022-05-14T03:06:42.122 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T03:06:42.122 INFO:journalctl@ceph.mon.c.smithi115.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T03:06:42.123 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T03:06:42.123 INFO:journalctl@ceph.mon.c.smithi115.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T03:06:42.123 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T03:06:42.123 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise 2022-05-14T03:06:42.123 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T03:06:42.124 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.gen.throw(type, value, traceback) 2022-05-14T03:06:42.124 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T03:06:42.124 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OrchestratorError(msg) 2022-05-14T03:06:42.124 INFO:journalctl@ceph.mon.c.smithi115.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:06:42.125 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:06:41 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 13135 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:06:42.125 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:06:41 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:06:42.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:06:42 smithi115 conmon[90393]: 2022-05-14T03:06:42.294+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:06:43.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:06:42 smithi047 ceph-mon[77147]: pgmap v6919: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:06:43.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:06:42 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:06:43.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:06:42 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:06:43.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:06:42 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:06:43.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:06:42 smithi115 ceph-mon[78593]: pgmap v6919: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:06:43.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:06:42 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:06:43.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:06:42 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:06:43.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:06:42 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:06:43.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:06:43 smithi115 conmon[90393]: 2022-05-14T03:06:43.301+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:06:44.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:06:43 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:06:44.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:06:43 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:06:44.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:06:44 smithi115 conmon[90393]: 2022-05-14T03:06:44.254+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:06:45.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:06:44 smithi047 ceph-mon[77147]: pgmap v6920: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:06:45.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:06:44 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:06:45.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:06:44 smithi115 ceph-mon[78593]: pgmap v6920: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:06:45.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:06:44 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:06:45.612 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:06:45 smithi115 conmon[90393]: 2022-05-14T03:06:45.294+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:06:46.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:06:45 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:06:46.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:06:45 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:06:46.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:06:46 smithi115 conmon[90393]: 2022-05-14T03:06:46.285+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:06:47.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:06:46 smithi047 ceph-mon[77147]: pgmap v6921: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:06:47.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:06:46 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 13140 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:06:47.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:06:46 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:06:47.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:06:46 smithi115 ceph-mon[78593]: pgmap v6921: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:06:47.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:06:46 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 13140 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:06:47.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:06:46 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:06:47.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:06:47 smithi115 conmon[90393]: 2022-05-14T03:06:47.241+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:06:48.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:06:47 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:06:48.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:06:47 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:06:48.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:06:48 smithi115 conmon[90393]: 2022-05-14T03:06:48.229+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:06:49.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:06:48 smithi047 ceph-mon[77147]: pgmap v6922: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:06:49.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:06:48 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:06:49.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:06:48 smithi115 ceph-mon[78593]: pgmap v6922: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:06:49.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:06:48 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:06:49.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:06:49 smithi115 conmon[90393]: 2022-05-14T03:06:49.274+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:06:50.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:06:49 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:06:50.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:06:49 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:06:50.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:06:50 smithi115 conmon[90393]: 2022-05-14T03:06:50.321+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:06:51.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:06:50 smithi047 ceph-mon[77147]: pgmap v6923: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:06:51.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:06:50 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:06:51.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:06:50 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:06:51.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:06:50 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:06:51.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:06:50 smithi115 ceph-mon[78593]: pgmap v6923: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:06:51.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:06:50 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:06:51.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:06:50 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:06:51.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:06:50 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:06:51.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:06:51 smithi115 conmon[90393]: 2022-05-14T03:06:51.281+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:06:52.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:06:51 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 13145 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:06:52.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:06:51 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:06:52.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:06:51 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 13145 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:06:52.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:06:51 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:06:52.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:06:52 smithi115 conmon[90393]: 2022-05-14T03:06:52.322+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:06:53.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:06:52 smithi047 ceph-mon[77147]: pgmap v6924: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:06:53.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:06:52 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:06:53.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:06:52 smithi115 ceph-mon[78593]: pgmap v6924: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:06:53.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:06:52 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:06:53.649 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:06:53 smithi115 conmon[90393]: 2022-05-14T03:06:53.360+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:06:54.049 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:06:53 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:06:54.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:06:53 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:06:54.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:06:54 smithi115 conmon[90393]: 2022-05-14T03:06:54.338+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:06:55.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:06:54 smithi047 ceph-mon[77147]: pgmap v6925: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:06:55.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:06:54 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:06:55.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:06:54 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:06:55.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:06:54 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:06:55.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:06:54 smithi115 ceph-mon[78593]: pgmap v6925: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:06:55.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:06:54 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:06:55.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:06:54 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:06:55.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:06:54 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:06:55.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:06:55 smithi115 conmon[90393]: 2022-05-14T03:06:55.295+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:06:56.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:06:55 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:06:56.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:06:55 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:06:56.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:06:56 smithi115 conmon[90393]: 2022-05-14T03:06:56.337+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:06:57.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:06:56 smithi047 ceph-mon[77147]: pgmap v6926: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:06:57.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:06:56 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 13150 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:06:57.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:06:56 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:06:57.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:06:56 smithi115 ceph-mon[78593]: pgmap v6926: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:06:57.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:06:56 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 13150 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:06:57.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:06:56 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:06:57.663 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:06:57 smithi115 conmon[90393]: 2022-05-14T03:06:57.364+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:06:58.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:06:57 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:06:58.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:06:57 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:06:58.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:06:58 smithi115 conmon[90393]: 2022-05-14T03:06:58.356+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:06:59.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:06:58 smithi047 ceph-mon[77147]: pgmap v6927: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:06:59.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:06:58 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:06:59.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:06:58 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:06:59.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:06:58 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:06:59.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:06:58 smithi115 ceph-mon[78593]: pgmap v6927: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:06:59.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:06:58 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:06:59.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:06:58 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:06:59.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:06:58 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:06:59.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:06:59 smithi115 conmon[90393]: 2022-05-14T03:06:59.320+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:07:00.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:06:59 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:07:00.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:06:59 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:07:00.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:07:00 smithi115 conmon[90393]: 2022-05-14T03:07:00.349+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:07:01.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:07:00 smithi047 ceph-mon[77147]: pgmap v6928: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:07:01.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:07:00 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:07:01.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:07:00 smithi115 ceph-mon[78593]: pgmap v6928: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:07:01.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:07:00 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:07:01.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:07:01 smithi115 conmon[90393]: 2022-05-14T03:07:01.339+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:07:02.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:07:01 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 13155 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:07:02.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:07:01 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:07:02.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:07:01 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 13155 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:07:02.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:07:01 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:07:02.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:07:02 smithi115 conmon[90393]: 2022-05-14T03:07:02.349+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:07:03.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:07:02 smithi047 ceph-mon[77147]: pgmap v6929: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:07:03.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:07:02 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:07:03.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:07:02 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:07:03.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:07:02 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:07:03.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:07:02 smithi115 ceph-mon[78593]: pgmap v6929: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:07:03.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:07:02 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:07:03.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:07:02 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:07:03.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:07:02 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:07:03.682 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:07:03 smithi115 conmon[90393]: 2022-05-14T03:07:03.398+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:07:04.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:07:03 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:07:04.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:07:03 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:07:04.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:07:04 smithi115 conmon[90393]: 2022-05-14T03:07:04.442+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:07:04.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:07:04 smithi115 ceph-mon[78593]: pgmap v6930: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:07:04.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:07:04 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:07:05.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:07:04 smithi047 ceph-mon[77147]: pgmap v6930: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:07:05.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:07:04 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:07:05.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:07:05 smithi115 conmon[90393]: 2022-05-14T03:07:05.473+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:07:05.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:07:05 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:07:06.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:07:05 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:07:06.860 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:07:06 smithi115 conmon[90393]: 2022-05-14T03:07:06.504+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:07:06.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:07:06 smithi115 ceph-mon[78593]: pgmap v6931: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:07:06.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:07:06 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 13160 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:07:06.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:07:06 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:07:07.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:07:06 smithi047 ceph-mon[77147]: pgmap v6931: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:07:07.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:07:06 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 13160 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:07:07.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:07:06 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:07:07.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:07:07 smithi115 conmon[90393]: 2022-05-14T03:07:07.502+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:07:07.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:07:07 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:07:08.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:07:07 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:07:08.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:07:08 smithi115 conmon[90393]: 2022-05-14T03:07:08.483+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:07:08.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:07:08 smithi115 ceph-mon[78593]: pgmap v6932: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:07:08.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:07:08 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:07:09.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:07:08 smithi047 ceph-mon[77147]: pgmap v6932: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:07:09.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:07:08 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:07:09.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:07:09 smithi115 conmon[90393]: 2022-05-14T03:07:09.485+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:07:09.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:07:09 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:07:10.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:07:09 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:07:10.865 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:07:10 smithi115 conmon[90393]: 2022-05-14T03:07:10.468+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:07:10.870 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:07:10 smithi115 ceph-mon[78593]: pgmap v6933: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:07:10.871 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:07:10 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:07:10.872 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:07:10 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:07:10.872 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:07:10 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:07:11.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:07:10 smithi047 ceph-mon[77147]: pgmap v6933: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:07:11.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:07:10 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:07:11.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:07:10 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:07:11.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:07:10 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:07:11.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:07:11 smithi115 conmon[90393]: 2022-05-14T03:07:11.464+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:07:11.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:07:11 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 13165 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:07:11.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:07:11 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:07:12.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:07:11 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 13165 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:07:12.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:07:11 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:07:12.708 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:07:12 smithi115 conmon[90393]: 2022-05-14T03:07:12.433+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:07:13.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:07:12 smithi047 ceph-mon[77147]: pgmap v6934: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:07:13.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:07:12 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:07:13.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:07:12 smithi115 ceph-mon[78593]: pgmap v6934: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:07:13.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:07:12 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:07:13.713 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:07:13 smithi115 conmon[90393]: 2022-05-14T03:07:13.408+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:07:14.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:07:13 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:07:14.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:07:13 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:07:14.714 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:07:14 smithi115 conmon[90393]: 2022-05-14T03:07:14.360+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:07:15.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:07:14 smithi047 ceph-mon[77147]: pgmap v6935: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:07:15.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:07:14 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:07:15.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:07:14 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:07:15.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:07:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T03:07:15.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:07:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T03:07:15.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:07:14 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:07:15.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:07:14 smithi115 ceph-mon[78593]: pgmap v6935: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:07:15.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:07:14 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:07:15.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:07:14 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:07:15.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:07:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T03:07:15.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:07:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T03:07:15.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:07:14 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:07:15.718 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:07:15 smithi115 conmon[90393]: 2022-05-14T03:07:15.370+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:07:16.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:07:15 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:07:16.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:07:15 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:07:16.721 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:07:16 smithi115 conmon[90393]: 2022-05-14T03:07:16.418+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:07:17.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:07:16 smithi047 ceph-mon[77147]: pgmap v6936: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:07:17.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:07:16 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 13170 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:07:17.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:07:16 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:07:17.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:07:16 smithi115 ceph-mon[78593]: pgmap v6936: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:07:17.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:07:16 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 13170 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:07:17.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:07:16 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:07:17.726 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:07:17 smithi115 conmon[90393]: 2022-05-14T03:07:17.437+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:07:18.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:07:17 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:07:18.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:07:17 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:07:18.727 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:07:18 smithi115 conmon[90393]: 2022-05-14T03:07:18.456+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:07:19.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:07:18 smithi047 ceph-mon[77147]: pgmap v6937: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:07:19.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:07:18 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:07:19.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:07:18 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:07:19.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:07:18 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:07:19.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:07:18 smithi115 ceph-mon[78593]: pgmap v6937: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:07:19.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:07:18 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:07:19.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:07:18 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:07:19.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:07:18 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:07:19.732 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:07:19 smithi115 conmon[90393]: 2022-05-14T03:07:19.451+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:07:20.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:07:19 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:07:20.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:07:19 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:07:20.735 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:07:20 smithi115 conmon[90393]: 2022-05-14T03:07:20.441+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:07:21.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:07:20 smithi047 ceph-mon[77147]: pgmap v6938: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:07:21.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:07:20 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:07:21.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:07:20 smithi115 ceph-mon[78593]: pgmap v6938: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:07:21.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:07:20 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:07:21.739 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:07:21 smithi115 conmon[90393]: 2022-05-14T03:07:21.472+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:07:22.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:07:21 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 13175 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:07:22.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:07:21 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:07:22.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:07:21 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 13175 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:07:22.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:07:21 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:07:22.741 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:07:22 smithi115 conmon[90393]: 2022-05-14T03:07:22.466+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:07:23.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:07:22 smithi047 ceph-mon[77147]: pgmap v6939: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:07:23.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:07:22 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:07:23.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:07:22 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:07:23.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:07:22 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:07:23.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:07:22 smithi115 ceph-mon[78593]: pgmap v6939: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:07:23.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:07:22 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:07:23.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:07:22 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:07:23.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:07:22 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:07:23.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:07:23 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:07:23.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:07:23 smithi115 conmon[90393]: 2022-05-14T03:07:23.506+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:07:24.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:07:23 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:07:24.860 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:07:24 smithi115 conmon[90393]: 2022-05-14T03:07:24.541+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:07:24.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:07:24 smithi115 ceph-mon[78593]: pgmap v6940: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:07:25.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:07:24 smithi047 ceph-mon[77147]: pgmap v6940: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:07:25.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:07:25 smithi115 conmon[90393]: 2022-05-14T03:07:25.580+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:07:25.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:07:25 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:07:25.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:07:25 smithi115 ceph-mon[78593]: pgmap v6941: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:07:26.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:07:25 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:07:26.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:07:25 smithi047 ceph-mon[77147]: pgmap v6941: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:07:26.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:07:26 smithi115 conmon[90393]: 2022-05-14T03:07:26.586+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:07:26.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:07:26 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:07:26.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:07:26 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 13180 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:07:27.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:07:26 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:07:27.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:07:26 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 13180 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:07:27.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:07:27 smithi115 conmon[90393]: 2022-05-14T03:07:27.548+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:07:27.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:07:27 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:07:27.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:07:27 smithi115 ceph-mon[78593]: pgmap v6942: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:07:28.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:07:27 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:07:28.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:07:27 smithi047 ceph-mon[77147]: pgmap v6942: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:07:28.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:07:28 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:07:28.860 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:07:28 smithi115 conmon[90393]: 2022-05-14T03:07:28.565+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:07:29.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:07:28 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:07:30.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:07:29 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:07:30.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:07:29 smithi047 ceph-mon[77147]: pgmap v6943: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:07:30.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:07:29 smithi115 conmon[90393]: 2022-05-14T03:07:29.611+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:07:30.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:07:29 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:07:30.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:07:29 smithi115 ceph-mon[78593]: pgmap v6943: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:07:31.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:07:30 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:07:31.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:07:30 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:07:31.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:07:30 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:07:31.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:07:30 smithi115 conmon[90393]: 2022-05-14T03:07:30.609+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:07:31.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:07:30 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:07:31.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:07:30 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:07:31.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:07:30 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:07:31.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:07:31 smithi115 conmon[90393]: 2022-05-14T03:07:31.594+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:07:31.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:07:31 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:07:31.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:07:31 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 13185 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:07:31.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:07:31 smithi115 ceph-mon[78593]: pgmap v6944: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:07:32.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:07:31 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:07:32.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:07:31 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 13185 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:07:32.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:07:31 smithi047 ceph-mon[77147]: pgmap v6944: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:07:32.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:07:32 smithi115 conmon[90393]: 2022-05-14T03:07:32.598+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:07:33.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:07:33 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:07:33.576 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:07:33 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:07:33.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:07:33 smithi115 conmon[90393]: 2022-05-14T03:07:33.574+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:07:34.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:07:34 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:07:34.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:07:34 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:07:34.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:07:34 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:07:34.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:07:34 smithi047 ceph-mon[77147]: pgmap v6945: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:07:34.568 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:07:34 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:07:34.568 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:07:34 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:07:34.568 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:07:34 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:07:34.569 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:07:34 smithi115 ceph-mon[78593]: pgmap v6945: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:07:34.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:07:34 smithi115 conmon[90393]: 2022-05-14T03:07:34.566+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:07:35.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:07:35 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:07:35.549 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:07:35 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:07:35.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:07:35 smithi115 conmon[90393]: 2022-05-14T03:07:35.548+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:07:36.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:07:36 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:07:36.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:07:36 smithi047 ceph-mon[77147]: pgmap v6946: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:07:36.595 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:07:36 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:07:36.596 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:07:36 smithi115 ceph-mon[78593]: pgmap v6946: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:07:36.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:07:36 smithi115 conmon[90393]: 2022-05-14T03:07:36.594+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:07:37.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:07:37 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:07:37.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:07:37 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 13190 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:07:37.597 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:07:37 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:07:37.598 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:07:37 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 13190 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:07:37.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:07:37 smithi115 conmon[90393]: 2022-05-14T03:07:37.595+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:07:38.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:07:38 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:07:38.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:07:38 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:07:38.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:07:38 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:07:38.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:07:38 smithi047 ceph-mon[77147]: pgmap v6947: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:07:38.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:07:38 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:07:38.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:07:38 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:07:38.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:07:38 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:07:38.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:07:38 smithi115 ceph-mon[78593]: pgmap v6947: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:07:39.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:07:38 smithi115 conmon[90393]: 2022-05-14T03:07:38.628+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:07:39.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:07:39 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:07:39.592 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:07:39 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:07:39.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:07:39 smithi115 conmon[90393]: 2022-05-14T03:07:39.591+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:07:40.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:07:40 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:07:40.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:07:40 smithi047 ceph-mon[77147]: pgmap v6948: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:07:40.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:07:40 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T03:07:40.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:07:40 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T03:07:40.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:07:40 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T03:07:40.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:07:40 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:07:40.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:07:40 smithi115 ceph-mon[78593]: pgmap v6948: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:07:40.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:07:40 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T03:07:40.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:07:40 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T03:07:40.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:07:40 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T03:07:40.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:07:40 smithi115 conmon[90393]: 2022-05-14T03:07:40.565+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:07:41.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:07:41 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:07:41.581 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:07:41 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:07:41.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:07:41 smithi115 conmon[90393]: 2022-05-14T03:07:41.579+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:07:42.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:07:42 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:07:42.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:07:42 smithi047 ceph-mon[77147]: pgmap v6949: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:07:42.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:07:42 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:07:42.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:07:42 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:07:42.584 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:07:42 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:07:42.585 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:07:42 smithi115 ceph-mon[78593]: pgmap v6949: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:07:42.585 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:07:42 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:07:42.586 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:07:42 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:07:42.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:07:42 smithi115 conmon[90393]: 2022-05-14T03:07:42.583+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:07:43.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:07:43 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:07:43.542 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:07:43 smithi047 conmon[78521]: 2022-05-14T03:07:43.088+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:07:43.542 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:07:43 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T03:07:43.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:07:43 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:07:43.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:07:43 smithi047 conmon[78521]: yield 2022-05-14T03:07:43.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:07:43 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:07:43.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:07:43 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:07:43.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:07:43 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:07:43.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:07:43 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T03:07:43.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:07:43 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:07:43.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:07:43 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T03:07:43.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:07:43 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:07:43.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:07:43 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T03:07:43.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:07:43 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:07:43.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:07:43 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T03:07:43.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:07:43 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:07:43.547 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:07:43 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T03:07:43.547 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:07:43 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:07:43.547 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:07:43 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:07:43.547 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:07:43 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:07:43.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:07:43 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:07:43.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:07:43 smithi115 conmon[90393]: 2022-05-14T03:07:43.545+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:07:44.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:07:44 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:07:44.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:07:44 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:07:44.542 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T03:07:44.542 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:07:44.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T03:07:44.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:07:44.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:07:44.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:07:44.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T03:07:44.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:07:44.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T03:07:44.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:07:44.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T03:07:44.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:07:44.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T03:07:44.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:07:44.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T03:07:44.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:07:44.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:07:44.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:07:44.546 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:07:44 smithi047 ceph-mon[77147]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T03:07:44.546 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:07:44 smithi047 ceph-mon[77147]: pgmap v6950: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:07:44.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:07:44 smithi115 conmon[90393]: 2022-05-14T03:07:44.512+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:07:44.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:07:44 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:07:44.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:07:44 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:07:44.609 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T03:07:44.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:07:44.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T03:07:44.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:07:44.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:07:44.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:07:44.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T03:07:44.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:07:44.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T03:07:44.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:07:44.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T03:07:44.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:07:44.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T03:07:44.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:07:44.613 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T03:07:44.613 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:07:44.614 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:07:44.614 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:07:44.614 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:07:44 smithi115 ceph-mon[78593]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T03:07:44.615 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:07:44 smithi115 ceph-mon[78593]: pgmap v6950: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:07:45.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:07:45 smithi115 conmon[90393]: 2022-05-14T03:07:45.497+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:07:45.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:07:45 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:07:45.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:07:45 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:07:45.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:07:45 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:07:45.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:07:45 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:07:46.541 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:07:46 smithi047 ceph-mon[77147]: pgmap v6951: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:07:46.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:07:46 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 13200 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:07:46.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:07:46 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:07:46.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:07:46 smithi047 conmon[78521]: 2022-05-14T03:07:46.160+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:07:46.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:07:46 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T03:07:46.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:07:46 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:07:46.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:07:46 smithi047 conmon[78521]: yield 2022-05-14T03:07:46.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:07:46 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:07:46.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:07:46 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:07:46.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:07:46 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:07:46.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:07:46 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T03:07:46.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:07:46 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:07:46.547 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:07:46 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T03:07:46.547 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:07:46 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:07:46.547 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:07:46 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T03:07:46.547 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:07:46 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:07:46.548 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:07:46 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T03:07:46.548 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:07:46 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:07:46.548 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:07:46 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T03:07:46.549 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:07:46 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:07:46.549 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:07:46 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:07:46.549 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:07:46 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:07:46.549 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:07:46 smithi047 conmon[78521]: 2022-05-14T03:07:46.161+0000 7f5340779700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T03:07:46.550 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:07:46 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T03:07:46.550 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:07:46 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:07:46.550 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:07:46 smithi047 conmon[78521]: yield 2022-05-14T03:07:46.551 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:07:46 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:07:46.551 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:07:46 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:07:46.551 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:07:46 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:07:46.551 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:07:46 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T03:07:46.552 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:07:46 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:07:46.552 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:07:46 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T03:07:46.552 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:07:46 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:07:46.552 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:07:46 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T03:07:46.552 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:07:46 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:07:46.553 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:07:46 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T03:07:46.553 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:07:46 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:07:46.553 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:07:46 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T03:07:46.553 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:07:46 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:07:46.554 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:07:46 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:07:46.554 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:07:46 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:07:46.554 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:07:46 smithi047 conmon[78521]: 2022-05-14T03:07:46.554 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:07:46 smithi047 conmon[78521]: During handling of the above exception, another exception occurred: 2022-05-14T03:07:46.555 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:07:46 smithi047 conmon[78521]: 2022-05-14T03:07:46.555 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:07:46 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T03:07:46.555 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:07:46 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T03:07:46.555 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:07:46 smithi047 conmon[78521]: return f(*arg) 2022-05-14T03:07:46.555 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:07:46 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T03:07:46.556 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:07:46 smithi047 conmon[78521]: self._write_client_files(client_files, host) 2022-05-14T03:07:46.556 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:07:46 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T03:07:46.556 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:07:46 smithi047 conmon[78521]: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T03:07:46.557 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:07:46 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T03:07:46.557 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:07:46 smithi047 conmon[78521]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T03:07:46.557 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:07:46 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T03:07:46.558 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:07:46 smithi047 conmon[78521]: return self.event_loop.get_result(coro) 2022-05-14T03:07:46.558 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:07:46 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T03:07:46.558 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:07:46 smithi047 conmon[78521]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T03:07:46.558 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:07:46 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T03:07:46.558 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:07:46 smithi047 conmon[78521]: return self.__get_result() 2022-05-14T03:07:46.559 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:07:46 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T03:07:46.559 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:07:46 smithi047 conmon[78521]: raise self._exception 2022-05-14T03:07:46.559 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:07:46 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T03:07:46.559 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:07:46 smithi047 conmon[78521]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T03:07:46.560 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:07:46 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T03:07:46.560 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:07:46 smithi047 conmon[78521]: conn = await self._remote_connection(host, addr) 2022-05-14T03:07:46.560 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:07:46 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T03:07:46.560 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:07:46 smithi047 conmon[78521]: raise 2022-05-14T03:07:46.561 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:07:46 smithi047 conmon[78521]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T03:07:46.561 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:07:46 smithi047 conmon[78521]: self.gen.throw(type, value, traceback) 2022-05-14T03:07:46.561 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:07:46 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T03:07:46.561 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:07:46 smithi047 conmon[78521]: raise OrchestratorError(msg) 2022-05-14T03:07:46.561 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:07:46 smithi047 conmon[78521]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:07:46.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:07:46 smithi115 conmon[90393]: 2022-05-14T03:07:46.488+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:07:46.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:07:46 smithi115 ceph-mon[78593]: pgmap v6951: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:07:46.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:07:46 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 13200 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:07:46.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:07:46 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:07:47.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:07:47 smithi115 conmon[90393]: 2022-05-14T03:07:47.519+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:07:47.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:07:47 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:07:47.609 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T03:07:47.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:07:47.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T03:07:47.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:07:47.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:07:47.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:07:47.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T03:07:47.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:07:47.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T03:07:47.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:07:47.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T03:07:47.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:07:47.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T03:07:47.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:07:47.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T03:07:47.613 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:07:47.613 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:07:47.613 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:07:47.613 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:07:47 smithi115 ceph-mon[78593]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T03:07:47.614 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T03:07:47.614 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:07:47.614 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T03:07:47.614 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:07:47.614 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:07:47.615 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:07:47.615 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T03:07:47.615 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:07:47.615 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T03:07:47.616 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:07:47.616 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T03:07:47.616 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:07:47.616 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T03:07:47.616 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:07:47.617 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T03:07:47.617 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:07:47.617 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:07:47.617 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:07:47.618 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T03:07:47.618 INFO:journalctl@ceph.mon.c.smithi115.stdout: During handling of the above exception, another exception occurred: 2022-05-14T03:07:47.618 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T03:07:47.618 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T03:07:47.618 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T03:07:47.619 INFO:journalctl@ceph.mon.c.smithi115.stdout: return f(*arg) 2022-05-14T03:07:47.619 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T03:07:47.619 INFO:journalctl@ceph.mon.c.smithi115.stdout: self._write_client_files(client_files, host) 2022-05-14T03:07:47.619 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T03:07:47.620 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T03:07:47.620 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T03:07:47.620 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T03:07:47.620 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T03:07:47.620 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.event_loop.get_result(coro) 2022-05-14T03:07:47.621 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T03:07:47.621 INFO:journalctl@ceph.mon.c.smithi115.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T03:07:47.621 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T03:07:47.621 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.__get_result() 2022-05-14T03:07:47.622 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T03:07:47.622 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise self._exception 2022-05-14T03:07:47.622 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T03:07:47.622 INFO:journalctl@ceph.mon.c.smithi115.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T03:07:47.622 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T03:07:47.623 INFO:journalctl@ceph.mon.c.smithi115.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T03:07:47.623 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T03:07:47.623 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise 2022-05-14T03:07:47.623 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T03:07:47.623 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.gen.throw(type, value, traceback) 2022-05-14T03:07:47.624 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T03:07:47.624 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OrchestratorError(msg) 2022-05-14T03:07:47.624 INFO:journalctl@ceph.mon.c.smithi115.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:07:47.624 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:07:47 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:07:47.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:07:47 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:07:47.792 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T03:07:47.792 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:07:47.792 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T03:07:47.793 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:07:47.793 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:07:47.793 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:07:47.793 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T03:07:47.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:07:47.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T03:07:47.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:07:47.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T03:07:47.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:07:47.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T03:07:47.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:07:47.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T03:07:47.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:07:47.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:07:47.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:07:47.796 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:07:47 smithi047 ceph-mon[77147]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T03:07:47.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T03:07:47.797 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:07:47.797 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T03:07:47.797 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:07:47.797 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:07:47.798 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:07:47.798 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T03:07:47.798 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:07:47.798 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T03:07:47.798 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:07:47.799 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T03:07:47.799 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:07:47.799 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T03:07:47.799 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:07:47.799 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T03:07:47.800 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:07:47.800 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:07:47.800 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:07:47.800 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T03:07:47.800 INFO:journalctl@ceph.mon.b.smithi047.stdout: During handling of the above exception, another exception occurred: 2022-05-14T03:07:47.801 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T03:07:47.801 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T03:07:47.801 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T03:07:47.801 INFO:journalctl@ceph.mon.b.smithi047.stdout: return f(*arg) 2022-05-14T03:07:47.802 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T03:07:47.802 INFO:journalctl@ceph.mon.b.smithi047.stdout: self._write_client_files(client_files, host) 2022-05-14T03:07:47.802 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T03:07:47.802 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T03:07:47.802 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T03:07:47.803 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T03:07:47.803 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T03:07:47.803 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.event_loop.get_result(coro) 2022-05-14T03:07:47.803 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T03:07:47.803 INFO:journalctl@ceph.mon.b.smithi047.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T03:07:47.804 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T03:07:47.804 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.__get_result() 2022-05-14T03:07:47.804 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T03:07:47.804 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise self._exception 2022-05-14T03:07:47.804 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T03:07:47.805 INFO:journalctl@ceph.mon.b.smithi047.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T03:07:47.805 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T03:07:47.805 INFO:journalctl@ceph.mon.b.smithi047.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T03:07:47.805 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T03:07:47.806 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise 2022-05-14T03:07:47.806 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T03:07:47.806 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.gen.throw(type, value, traceback) 2022-05-14T03:07:47.806 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T03:07:47.806 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OrchestratorError(msg) 2022-05-14T03:07:47.807 INFO:journalctl@ceph.mon.b.smithi047.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:07:47.807 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:07:47 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:07:48.559 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:07:48 smithi115 ceph-mon[78593]: pgmap v6952: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:07:48.559 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:07:48 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:07:48.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:07:48 smithi047 ceph-mon[77147]: pgmap v6952: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:07:48.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:07:48 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:07:48.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:07:48 smithi115 conmon[90393]: 2022-05-14T03:07:48.557+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:07:49.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:07:49 smithi115 conmon[90393]: 2022-05-14T03:07:49.543+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:07:50.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:07:50 smithi115 conmon[90393]: 2022-05-14T03:07:50.513+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:07:50.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:07:50 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:07:50.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:07:50 smithi115 ceph-mon[78593]: pgmap v6953: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:07:50.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:07:50 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:07:50.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:07:50 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:07:50.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:07:50 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:07:50.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:07:50 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:07:50.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:07:50 smithi047 ceph-mon[77147]: pgmap v6953: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:07:50.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:07:50 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:07:50.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:07:50 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:07:50.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:07:50 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:07:51.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:07:51 smithi115 conmon[90393]: 2022-05-14T03:07:51.464+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:07:51.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:07:51 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:07:51.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:07:51 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 13205 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:07:51.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:07:51 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:07:51.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:07:51 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 13205 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:07:52.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:07:52 smithi115 conmon[90393]: 2022-05-14T03:07:52.513+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:07:52.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:07:52 smithi115 ceph-mon[78593]: pgmap v6954: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:07:52.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:07:52 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:07:52.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:07:52 smithi047 ceph-mon[77147]: pgmap v6954: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:07:52.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:07:52 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:07:53.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:07:53 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:07:53.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:07:53 smithi115 conmon[90393]: 2022-05-14T03:07:53.476+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:07:53.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:07:53 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:07:54.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:07:54 smithi115 conmon[90393]: 2022-05-14T03:07:54.450+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:07:54.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:07:54 smithi115 ceph-mon[78593]: pgmap v6955: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:07:54.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:07:54 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:07:54.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:07:54 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:07:54.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:07:54 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:07:54.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:07:54 smithi047 ceph-mon[77147]: pgmap v6955: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:07:54.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:07:54 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:07:54.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:07:54 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:07:54.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:07:54 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:07:55.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:07:55 smithi115 conmon[90393]: 2022-05-14T03:07:55.427+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:07:55.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:07:55 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:07:55.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:07:55 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:07:56.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:07:56 smithi115 conmon[90393]: 2022-05-14T03:07:56.400+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:07:56.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:07:56 smithi115 ceph-mon[78593]: pgmap v6956: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:07:56.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:07:56 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:07:56.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:07:56 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 13210 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:07:56.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:07:56 smithi047 ceph-mon[77147]: pgmap v6956: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:07:56.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:07:56 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:07:56.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:07:56 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 13210 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:07:57.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:07:57 smithi115 conmon[90393]: 2022-05-14T03:07:57.442+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:07:57.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:07:57 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:07:57.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:07:57 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:07:58.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:07:58 smithi115 conmon[90393]: 2022-05-14T03:07:58.407+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:07:58.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:07:58 smithi115 ceph-mon[78593]: pgmap v6957: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:07:58.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:07:58 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:07:58.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:07:58 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:07:58.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:07:58 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:07:58.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:07:58 smithi047 ceph-mon[77147]: pgmap v6957: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:07:58.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:07:58 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:07:58.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:07:58 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:07:58.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:07:58 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:07:59.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:07:59 smithi115 conmon[90393]: 2022-05-14T03:07:59.375+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:07:59.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:07:59 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:07:59.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:07:59 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:08:00.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:08:00 smithi115 conmon[90393]: 2022-05-14T03:08:00.391+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:08:00.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:08:00 smithi115 ceph-mon[78593]: pgmap v6958: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:08:00.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:08:00 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:08:00.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:08:00 smithi047 ceph-mon[77147]: pgmap v6958: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:08:00.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:08:00 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:08:01.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:08:01 smithi115 conmon[90393]: 2022-05-14T03:08:01.421+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:08:01.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:08:01 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:08:01.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:08:01 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 13215 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:08:01.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:08:01 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:08:01.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:08:01 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 13215 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:08:02.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:08:02 smithi115 conmon[90393]: 2022-05-14T03:08:02.429+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:08:02.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:08:02 smithi115 ceph-mon[78593]: pgmap v6959: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:08:02.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:08:02 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:08:02.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:08:02 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:08:02.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:08:02 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:08:02.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:08:02 smithi047 ceph-mon[77147]: pgmap v6959: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:08:02.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:08:02 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:08:02.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:08:02 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:08:02.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:08:02 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:08:03.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:08:03 smithi115 conmon[90393]: 2022-05-14T03:08:03.397+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:08:03.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:08:03 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:08:03.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:08:03 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:08:04.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:08:04 smithi047 ceph-mon[77147]: pgmap v6960: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:08:04.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:08:04 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:08:04.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:08:04 smithi115 conmon[90393]: 2022-05-14T03:08:04.359+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:08:04.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:08:04 smithi115 ceph-mon[78593]: pgmap v6960: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:08:04.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:08:04 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:08:05.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:08:05 smithi115 conmon[90393]: 2022-05-14T03:08:05.351+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:08:05.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:08:05 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:08:05.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:08:05 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:08:06.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:08:06 smithi115 conmon[90393]: 2022-05-14T03:08:06.335+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:08:06.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:08:06 smithi115 ceph-mon[78593]: pgmap v6961: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:08:06.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:08:06 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:08:06.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:08:06 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 13220 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:08:06.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:08:06 smithi047 ceph-mon[77147]: pgmap v6961: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:08:06.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:08:06 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:08:06.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:08:06 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 13220 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:08:07.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:08:07 smithi115 conmon[90393]: 2022-05-14T03:08:07.286+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:08:07.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:08:07 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:08:07.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:08:07 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:08:08.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:08:08 smithi115 conmon[90393]: 2022-05-14T03:08:08.249+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:08:08.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:08:08 smithi115 ceph-mon[78593]: pgmap v6962: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:08:08.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:08:08 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:08:08.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:08:08 smithi047 ceph-mon[77147]: pgmap v6962: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:08:08.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:08:08 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:08:09.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:08:09 smithi115 conmon[90393]: 2022-05-14T03:08:09.242+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:08:09.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:08:09 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:08:09.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:08:09 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:08:10.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:08:10 smithi115 conmon[90393]: 2022-05-14T03:08:10.234+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:08:10.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:08:10 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:08:10.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:08:10 smithi115 ceph-mon[78593]: pgmap v6963: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:08:10.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:08:10 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:08:10.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:08:10 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:08:10.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:08:10 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:08:10.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:08:10 smithi047 ceph-mon[77147]: pgmap v6963: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:08:10.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:08:10 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:08:10.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:08:10 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:08:11.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:08:11 smithi115 conmon[90393]: 2022-05-14T03:08:11.246+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:08:11.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:08:11 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:08:11.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:08:11 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 13225 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:08:11.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:08:11 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:08:11.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:08:11 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 13225 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:08:12.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:08:12 smithi115 conmon[90393]: 2022-05-14T03:08:12.260+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:08:12.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:08:12 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:08:12.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:08:12 smithi115 ceph-mon[78593]: pgmap v6964: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:08:12.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:08:12 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:08:12.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:08:12 smithi047 ceph-mon[77147]: pgmap v6964: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:08:13.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:08:13 smithi115 conmon[90393]: 2022-05-14T03:08:13.289+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:08:13.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:08:13 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:08:13.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:08:13 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:08:14.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:08:14 smithi115 conmon[90393]: 2022-05-14T03:08:14.282+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:08:14.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:08:14 smithi115 ceph-mon[78593]: pgmap v6965: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:08:14.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:08:14 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:08:14.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:08:14 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:08:14.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:08:14 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:08:14.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:08:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T03:08:14.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:08:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T03:08:14.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:08:14 smithi047 ceph-mon[77147]: pgmap v6965: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:08:14.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:08:14 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:08:14.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:08:14 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:08:14.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:08:14 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:08:14.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:08:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T03:08:14.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:08:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T03:08:15.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:08:15 smithi115 conmon[90393]: 2022-05-14T03:08:15.263+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:08:15.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:08:15 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:08:15.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:08:15 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:08:16.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:08:16 smithi115 conmon[90393]: 2022-05-14T03:08:16.261+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:08:16.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:08:16 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:08:16.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:08:16 smithi115 ceph-mon[78593]: pgmap v6966: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:08:16.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:08:16 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 13230 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:08:16.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:08:16 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:08:16.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:08:16 smithi047 ceph-mon[77147]: pgmap v6966: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:08:16.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:08:16 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 13230 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:08:17.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:08:17 smithi115 conmon[90393]: 2022-05-14T03:08:17.214+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:08:17.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:08:17 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:08:17.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:08:17 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:08:18.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:08:18 smithi115 conmon[90393]: 2022-05-14T03:08:18.250+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:08:18.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:08:18 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:08:18.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:08:18 smithi115 ceph-mon[78593]: pgmap v6967: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:08:18.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:08:18 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:08:18.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:08:18 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:08:18.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:08:18 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:08:18.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:08:18 smithi047 ceph-mon[77147]: pgmap v6967: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:08:18.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:08:18 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:08:18.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:08:18 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:08:19.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:08:19 smithi115 conmon[90393]: 2022-05-14T03:08:19.266+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:08:19.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:08:19 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:08:19.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:08:19 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:08:20.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:08:20 smithi115 conmon[90393]: 2022-05-14T03:08:20.307+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:08:20.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:08:20 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:08:20.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:08:20 smithi115 ceph-mon[78593]: pgmap v6968: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:08:20.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:08:20 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:08:20.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:08:20 smithi047 ceph-mon[77147]: pgmap v6968: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:08:21.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:08:21 smithi115 conmon[90393]: 2022-05-14T03:08:21.316+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:08:21.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:08:21 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:08:21.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:08:21 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 13235 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:08:21.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:08:21 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:08:21.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:08:21 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 13235 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:08:22.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:08:22 smithi115 conmon[90393]: 2022-05-14T03:08:22.342+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:08:22.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:08:22 smithi115 ceph-mon[78593]: pgmap v6969: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:08:22.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:08:22 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:08:22.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:08:22 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:08:22.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:08:22 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:08:22.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:08:22 smithi047 ceph-mon[77147]: pgmap v6969: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:08:22.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:08:22 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:08:22.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:08:22 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:08:22.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:08:22 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:08:23.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:08:23 smithi115 conmon[90393]: 2022-05-14T03:08:23.325+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:08:23.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:08:23 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:08:23.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:08:23 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:08:24.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:08:24 smithi115 conmon[90393]: 2022-05-14T03:08:24.309+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:08:24.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:08:24 smithi115 ceph-mon[78593]: pgmap v6970: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:08:24.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:08:24 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:08:24.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:08:24 smithi047 ceph-mon[77147]: pgmap v6970: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:08:24.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:08:24 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:08:25.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:08:25 smithi115 conmon[90393]: 2022-05-14T03:08:25.299+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:08:25.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:08:25 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:08:25.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:08:25 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:08:26.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:08:26 smithi115 conmon[90393]: 2022-05-14T03:08:26.330+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:08:26.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:08:26 smithi115 ceph-mon[78593]: pgmap v6971: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:08:26.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:08:26 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:08:26.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:08:26 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 13240 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:08:26.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:08:26 smithi047 ceph-mon[77147]: pgmap v6971: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:08:26.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:08:26 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:08:26.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:08:26 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 13240 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:08:27.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:08:27 smithi115 conmon[90393]: 2022-05-14T03:08:27.347+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:08:27.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:08:27 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:08:27.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:08:27 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:08:28.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:08:28 smithi115 conmon[90393]: 2022-05-14T03:08:28.328+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:08:28.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:08:28 smithi115 ceph-mon[78593]: pgmap v6972: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:08:28.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:08:28 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:08:28.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:08:28 smithi047 ceph-mon[77147]: pgmap v6972: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:08:28.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:08:28 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:08:29.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:08:29 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:08:29.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:08:29 smithi115 conmon[90393]: 2022-05-14T03:08:29.300+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:08:29.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:08:29 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:08:30.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:08:30 smithi115 conmon[90393]: 2022-05-14T03:08:30.297+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:08:30.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:08:30 smithi115 ceph-mon[78593]: pgmap v6973: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:08:30.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:08:30 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:08:30.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:08:30 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:08:30.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:08:30 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:08:30.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:08:30 smithi047 ceph-mon[77147]: pgmap v6973: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:08:30.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:08:30 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:08:30.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:08:30 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:08:30.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:08:30 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:08:31.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:08:31 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:08:31.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:08:31 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 13245 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:08:31.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:08:31 smithi115 conmon[90393]: 2022-05-14T03:08:31.283+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:08:31.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:08:31 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:08:31.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:08:31 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 13245 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:08:32.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:08:32 smithi115 conmon[90393]: 2022-05-14T03:08:32.293+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:08:32.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:08:32 smithi115 ceph-mon[78593]: pgmap v6974: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:08:32.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:08:32 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:08:32.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:08:32 smithi047 ceph-mon[77147]: pgmap v6974: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:08:32.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:08:32 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:08:33.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:08:33 smithi115 conmon[90393]: 2022-05-14T03:08:33.297+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:08:33.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:08:33 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:08:33.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:08:33 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:08:34.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:08:34 smithi115 conmon[90393]: 2022-05-14T03:08:34.269+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:08:34.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:08:34 smithi115 ceph-mon[78593]: pgmap v6975: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:08:34.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:08:34 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:08:34.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:08:34 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:08:34.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:08:34 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:08:34.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:08:34 smithi047 ceph-mon[77147]: pgmap v6975: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:08:34.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:08:34 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:08:34.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:08:34 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:08:34.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:08:34 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:08:35.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:08:35 smithi115 conmon[90393]: 2022-05-14T03:08:35.255+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:08:35.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:08:35 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:08:35.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:08:35 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:08:36.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:08:36 smithi115 conmon[90393]: 2022-05-14T03:08:36.227+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:08:36.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:08:36 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:08:36.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:08:36 smithi115 ceph-mon[78593]: pgmap v6976: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:08:36.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:08:36 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 13250 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:08:36.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:08:36 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:08:36.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:08:36 smithi047 ceph-mon[77147]: pgmap v6976: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:08:36.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:08:36 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 13250 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:08:37.462 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:08:37 smithi115 conmon[90393]: 2022-05-14T03:08:37.181+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:08:37.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:08:37 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:08:37.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:08:37 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:08:38.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:08:38 smithi115 conmon[90393]: 2022-05-14T03:08:38.213+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:08:38.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:08:38 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:08:38.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:08:38 smithi115 ceph-mon[78593]: pgmap v6977: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:08:38.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:08:38 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:08:38.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:08:38 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:08:38.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:08:38 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:08:38.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:08:38 smithi047 ceph-mon[77147]: pgmap v6977: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:08:38.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:08:38 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:08:38.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:08:38 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:08:39.468 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:08:39 smithi115 conmon[90393]: 2022-05-14T03:08:39.207+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:08:39.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:08:39 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:08:39.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:08:39 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:08:40.470 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:08:40 smithi115 conmon[90393]: 2022-05-14T03:08:40.162+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:08:40.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:08:40 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:08:40.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:08:40 smithi047 ceph-mon[77147]: pgmap v6978: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:08:40.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:08:40 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:08:40.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:08:40 smithi115 ceph-mon[78593]: pgmap v6978: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:08:41.471 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:08:41 smithi115 conmon[90393]: 2022-05-14T03:08:41.126+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:08:41.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:08:41 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:08:41.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:08:41 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 13255 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:08:41.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:08:41 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:08:41.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:08:41 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 13255 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:08:42.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:08:42 smithi115 conmon[90393]: 2022-05-14T03:08:42.100+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:08:42.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:08:42 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:08:42.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:08:42 smithi047 ceph-mon[77147]: pgmap v6979: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:08:42.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:08:42 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:08:42.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:08:42 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:08:42.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:08:42 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:08:42.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:08:42 smithi115 ceph-mon[78593]: pgmap v6979: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:08:42.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:08:42 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:08:42.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:08:42 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:08:43.482 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:08:43 smithi115 conmon[90393]: 2022-05-14T03:08:43.150+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:08:43.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:08:43 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:08:43.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:08:43 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:08:44.482 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:08:44 smithi115 conmon[90393]: 2022-05-14T03:08:44.136+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:08:44.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:08:44 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:08:44.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:08:44 smithi047 ceph-mon[77147]: pgmap v6980: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:08:44.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:08:44 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:08:44.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:08:44 smithi115 ceph-mon[78593]: pgmap v6980: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:08:45.488 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:08:45 smithi115 conmon[90393]: 2022-05-14T03:08:45.118+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:08:45.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:08:45 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:08:45.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:08:45 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:08:46.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:08:46 smithi115 conmon[90393]: 2022-05-14T03:08:46.101+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:08:46.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:08:46 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:08:46.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:08:46 smithi047 ceph-mon[77147]: pgmap v6981: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:08:46.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:08:46 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T03:08:46.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:08:46 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T03:08:46.795 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:08:46 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T03:08:46.796 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:08:46 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 13260 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:08:46.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:08:46 smithi047 conmon[78521]: 2022-05-14T03:08:46.576+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:08:46.797 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:08:46 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T03:08:46.797 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:08:46 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:08:46.798 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:08:46 smithi047 conmon[78521]: yield 2022-05-14T03:08:46.798 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:08:46 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:08:46.799 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:08:46 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:08:46.799 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:08:46 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:08:46.800 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:08:46 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T03:08:46.801 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:08:46 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:08:46.802 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:08:46 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T03:08:46.802 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:08:46 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:08:46.803 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:08:46 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T03:08:46.805 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:08:46 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:08:46.806 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:08:46 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T03:08:46.806 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:08:46 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:08:46.807 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:08:46 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T03:08:46.807 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:08:46 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:08:46.808 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:08:46 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:08:46.808 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:08:46 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:08:46.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:08:46 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:08:46.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:08:46 smithi115 ceph-mon[78593]: pgmap v6981: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:08:46.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:08:46 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T03:08:46.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:08:46 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T03:08:46.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:08:46 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T03:08:46.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:08:46 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 13260 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:08:47.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:08:47 smithi115 conmon[90393]: 2022-05-14T03:08:47.069+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:08:47.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:08:47 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:08:47.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:08:47 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:08:47.792 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T03:08:47.792 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:08:47.793 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T03:08:47.793 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:08:47.793 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:08:47.793 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:08:47.793 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T03:08:47.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:08:47.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T03:08:47.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:08:47.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T03:08:47.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:08:47.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T03:08:47.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:08:47.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T03:08:47.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:08:47.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:08:47.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:08:47.796 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:08:47 smithi047 ceph-mon[77147]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T03:08:47.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:08:47 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:08:47.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:08:47 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:08:47.859 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T03:08:47.859 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:08:47.859 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T03:08:47.860 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:08:47.860 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:08:47.860 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:08:47.860 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T03:08:47.860 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:08:47.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T03:08:47.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:08:47.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T03:08:47.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:08:47.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T03:08:47.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:08:47.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T03:08:47.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:08:47.863 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:08:47.863 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:08:47.863 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:08:47 smithi115 ceph-mon[78593]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T03:08:48.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:08:48 smithi115 conmon[90393]: 2022-05-14T03:08:48.039+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:08:48.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:08:48 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:08:48.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:08:48 smithi047 ceph-mon[77147]: pgmap v6982: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:08:48.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:08:48 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:08:48.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:08:48 smithi115 ceph-mon[78593]: pgmap v6982: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:08:49.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:08:49 smithi115 conmon[90393]: 2022-05-14T03:08:49.056+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:08:49.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:08:49 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:08:49.792 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:08:49 smithi047 conmon[78521]: 2022-05-14T03:08:49.647+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:08:49.793 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:08:49 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T03:08:49.793 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:08:49 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:08:49.793 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:08:49 smithi047 conmon[78521]: yield 2022-05-14T03:08:49.793 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:08:49 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:08:49.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:08:49 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:08:49.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:08:49 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:08:49.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:08:49 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T03:08:49.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:08:49 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:08:49.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:08:49 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T03:08:49.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:08:49 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:08:49.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:08:49 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T03:08:49.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:08:49 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:08:49.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:08:49 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T03:08:49.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:08:49 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:08:49.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:08:49 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T03:08:49.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:08:49 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:08:49.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:08:49 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:08:49.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:08:49 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:08:49.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:08:49 smithi047 conmon[78521]: 2022-05-14T03:08:49.648+0000 7f5344781700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T03:08:49.797 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:08:49 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T03:08:49.797 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:08:49 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:08:49.797 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:08:49 smithi047 conmon[78521]: yield 2022-05-14T03:08:49.797 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:08:49 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:08:49.798 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:08:49 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:08:49.798 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:08:49 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:08:49.798 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:08:49 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T03:08:49.798 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:08:49 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:08:49.799 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:08:49 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T03:08:49.799 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:08:49 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:08:49.799 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:08:49 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T03:08:49.799 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:08:49 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:08:49.800 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:08:49 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T03:08:49.800 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:08:49 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:08:49.800 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:08:49 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T03:08:49.800 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:08:49 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:08:49.800 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:08:49 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:08:49.801 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:08:49 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:08:49.801 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:08:49 smithi047 conmon[78521]: 2022-05-14T03:08:49.801 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:08:49 smithi047 conmon[78521]: During handling of the above exception, another exception occurred: 2022-05-14T03:08:49.801 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:08:49 smithi047 conmon[78521]: 2022-05-14T03:08:49.801 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:08:49 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T03:08:49.802 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:08:49 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T03:08:49.802 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:08:49 smithi047 conmon[78521]: return f(*arg) 2022-05-14T03:08:49.802 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:08:49 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T03:08:49.802 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:08:49 smithi047 conmon[78521]: self._write_client_files(client_files, host) 2022-05-14T03:08:49.803 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:08:49 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T03:08:49.803 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:08:49 smithi047 conmon[78521]: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T03:08:49.803 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:08:49 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T03:08:49.803 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:08:49 smithi047 conmon[78521]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T03:08:49.803 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:08:49 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T03:08:49.804 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:08:49 smithi047 conmon[78521]: return self.event_loop.get_result(coro) 2022-05-14T03:08:49.804 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:08:49 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T03:08:49.804 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:08:49 smithi047 conmon[78521]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T03:08:49.805 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:08:49 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T03:08:49.805 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:08:49 smithi047 conmon[78521]: return self.__get_result() 2022-05-14T03:08:49.806 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:08:49 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T03:08:49.806 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:08:49 smithi047 conmon[78521]: raise self._exception 2022-05-14T03:08:49.806 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:08:49 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T03:08:49.807 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:08:49 smithi047 conmon[78521]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T03:08:49.807 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:08:49 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T03:08:49.807 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:08:49 smithi047 conmon[78521]: conn = await self._remote_connection(host, addr) 2022-05-14T03:08:49.808 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:08:49 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T03:08:49.808 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:08:49 smithi047 conmon[78521]: raise 2022-05-14T03:08:49.808 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:08:49 smithi047 conmon[78521]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T03:08:49.809 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:08:49 smithi047 conmon[78521]: self.gen.throw(type, value, traceback) 2022-05-14T03:08:49.809 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:08:49 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T03:08:49.810 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:08:49 smithi047 conmon[78521]: raise OrchestratorError(msg) 2022-05-14T03:08:49.810 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:08:49 smithi047 conmon[78521]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:08:49.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:08:49 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:08:50.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:08:50 smithi115 conmon[90393]: 2022-05-14T03:08:50.018+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:08:50.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:08:50 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:08:50.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:08:50 smithi047 ceph-mon[77147]: pgmap v6983: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:08:50.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:08:50 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:08:50.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:08:50 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:08:50.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:08:50 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:08:50.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T03:08:50.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:08:50.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T03:08:50.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:08:50.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:08:50.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:08:50.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T03:08:50.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:08:50.797 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T03:08:50.797 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:08:50.797 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T03:08:50.797 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:08:50.797 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T03:08:50.798 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:08:50.798 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T03:08:50.799 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:08:50.799 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:08:50.799 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:08:50.799 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:08:50 smithi047 ceph-mon[77147]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T03:08:50.799 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T03:08:50.800 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:08:50.800 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T03:08:50.801 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:08:50.801 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:08:50.801 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:08:50.801 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T03:08:50.802 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:08:50.802 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T03:08:50.802 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:08:50.802 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T03:08:50.802 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:08:50.803 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T03:08:50.803 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:08:50.803 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T03:08:50.803 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:08:50.803 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:08:50.804 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:08:50.804 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T03:08:50.804 INFO:journalctl@ceph.mon.b.smithi047.stdout: During handling of the above exception, another exception occurred: 2022-05-14T03:08:50.804 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T03:08:50.804 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T03:08:50.805 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T03:08:50.805 INFO:journalctl@ceph.mon.b.smithi047.stdout: return f(*arg) 2022-05-14T03:08:50.805 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T03:08:50.806 INFO:journalctl@ceph.mon.b.smithi047.stdout: self._write_client_files(client_files, host) 2022-05-14T03:08:50.806 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T03:08:50.806 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T03:08:50.806 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T03:08:50.806 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T03:08:50.807 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T03:08:50.807 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.event_loop.get_result(coro) 2022-05-14T03:08:50.807 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T03:08:50.807 INFO:journalctl@ceph.mon.b.smithi047.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T03:08:50.808 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T03:08:50.808 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.__get_result() 2022-05-14T03:08:50.808 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T03:08:50.809 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise self._exception 2022-05-14T03:08:50.809 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T03:08:50.809 INFO:journalctl@ceph.mon.b.smithi047.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T03:08:50.809 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T03:08:50.810 INFO:journalctl@ceph.mon.b.smithi047.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T03:08:50.810 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T03:08:50.810 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise 2022-05-14T03:08:50.810 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T03:08:50.810 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.gen.throw(type, value, traceback) 2022-05-14T03:08:50.811 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T03:08:50.811 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OrchestratorError(msg) 2022-05-14T03:08:50.811 INFO:journalctl@ceph.mon.b.smithi047.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:08:50.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:08:50 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:08:50.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:08:50 smithi115 ceph-mon[78593]: pgmap v6983: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:08:50.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:08:50 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:08:50.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:08:50 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:08:50.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:08:50 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:08:50.860 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T03:08:50.860 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:08:50.860 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T03:08:50.860 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:08:50.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:08:50.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:08:50.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T03:08:50.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:08:50.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T03:08:50.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:08:50.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T03:08:50.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:08:50.863 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T03:08:50.863 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:08:50.863 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T03:08:50.863 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:08:50.864 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:08:50.864 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:08:50.864 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:08:50 smithi115 ceph-mon[78593]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T03:08:50.864 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T03:08:50.865 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:08:50.865 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T03:08:50.865 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:08:50.866 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:08:50.866 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:08:50.866 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T03:08:50.866 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:08:50.866 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T03:08:50.867 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:08:50.867 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T03:08:50.867 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:08:50.867 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T03:08:50.868 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:08:50.868 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T03:08:50.868 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:08:50.868 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:08:50.868 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:08:50.869 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T03:08:50.869 INFO:journalctl@ceph.mon.c.smithi115.stdout: During handling of the above exception, another exception occurred: 2022-05-14T03:08:50.869 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T03:08:50.869 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T03:08:50.870 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T03:08:50.870 INFO:journalctl@ceph.mon.c.smithi115.stdout: return f(*arg) 2022-05-14T03:08:50.870 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T03:08:50.870 INFO:journalctl@ceph.mon.c.smithi115.stdout: self._write_client_files(client_files, host) 2022-05-14T03:08:50.870 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T03:08:50.871 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T03:08:50.871 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T03:08:50.871 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T03:08:50.871 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T03:08:50.871 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.event_loop.get_result(coro) 2022-05-14T03:08:50.872 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T03:08:50.872 INFO:journalctl@ceph.mon.c.smithi115.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T03:08:50.872 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T03:08:50.872 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.__get_result() 2022-05-14T03:08:50.873 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T03:08:50.873 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise self._exception 2022-05-14T03:08:50.873 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T03:08:50.873 INFO:journalctl@ceph.mon.c.smithi115.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T03:08:50.873 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T03:08:50.874 INFO:journalctl@ceph.mon.c.smithi115.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T03:08:50.874 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T03:08:50.874 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise 2022-05-14T03:08:50.874 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T03:08:50.875 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.gen.throw(type, value, traceback) 2022-05-14T03:08:50.875 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T03:08:50.875 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OrchestratorError(msg) 2022-05-14T03:08:50.875 INFO:journalctl@ceph.mon.c.smithi115.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:08:51.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:08:50 smithi115 conmon[90393]: 2022-05-14T03:08:50.995+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:08:51.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:08:51 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:08:51.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:08:51 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 13265 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:08:51.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:08:51 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:08:51.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:08:51 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 13265 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:08:52.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:08:52 smithi115 conmon[90393]: 2022-05-14T03:08:52.033+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:08:52.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:08:52 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:08:52.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:08:52 smithi047 ceph-mon[77147]: pgmap v6984: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:08:52.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:08:52 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:08:52.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:08:52 smithi115 ceph-mon[78593]: pgmap v6984: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:08:53.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:08:53 smithi115 conmon[90393]: 2022-05-14T03:08:53.045+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:08:53.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:08:53 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:08:53.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:08:53 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:08:54.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:08:54 smithi115 conmon[90393]: 2022-05-14T03:08:54.026+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:08:54.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:08:54 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:08:54.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:08:54 smithi047 ceph-mon[77147]: pgmap v6985: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:08:54.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:08:54 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:08:54.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:08:54 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:08:54.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:08:54 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:08:54.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:08:54 smithi115 ceph-mon[78593]: pgmap v6985: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:08:54.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:08:54 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:08:54.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:08:54 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:08:55.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:08:54 smithi115 conmon[90393]: 2022-05-14T03:08:54.990+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:08:55.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:08:55 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:08:55.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:08:55 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:08:56.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:08:55 smithi115 conmon[90393]: 2022-05-14T03:08:55.987+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:08:56.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:08:56 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:08:56.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:08:56 smithi047 ceph-mon[77147]: pgmap v6986: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:08:56.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:08:56 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 13270 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:08:56.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:08:56 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:08:56.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:08:56 smithi115 ceph-mon[78593]: pgmap v6986: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:08:56.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:08:56 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 13270 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:08:57.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:08:56 smithi115 conmon[90393]: 2022-05-14T03:08:56.949+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:08:57.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:08:57 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:08:57.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:08:57 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:08:58.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:08:57 smithi115 conmon[90393]: 2022-05-14T03:08:57.948+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:08:58.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:08:58 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:08:58.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:08:58 smithi047 ceph-mon[77147]: pgmap v6987: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:08:58.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:08:58 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:08:58.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:08:58 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:08:58.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:08:58 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:08:58.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:08:58 smithi115 ceph-mon[78593]: pgmap v6987: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:08:58.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:08:58 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:08:58.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:08:58 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:08:59.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:08:58 smithi115 conmon[90393]: 2022-05-14T03:08:58.925+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:08:59.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:08:59 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:08:59.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:08:59 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:09:00.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:08:59 smithi115 conmon[90393]: 2022-05-14T03:08:59.966+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:09:00.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:09:00 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:09:00.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:09:00 smithi047 ceph-mon[77147]: pgmap v6988: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:09:00.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:09:00 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:09:00.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:09:00 smithi115 ceph-mon[78593]: pgmap v6988: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:09:01.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:09:00 smithi115 conmon[90393]: 2022-05-14T03:09:00.970+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:09:01.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:09:01 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:09:01.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:09:01 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 13275 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:09:01.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:09:01 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:09:01.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:09:01 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 13275 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:09:02.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:09:01 smithi115 conmon[90393]: 2022-05-14T03:09:01.988+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:09:02.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:09:02 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:09:02.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:09:02 smithi047 ceph-mon[77147]: pgmap v6989: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:09:02.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:09:02 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:09:02.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:09:02 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:09:02.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:09:02 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:09:02.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:09:02 smithi115 ceph-mon[78593]: pgmap v6989: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:09:02.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:09:02 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:09:02.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:09:02 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:09:03.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:09:02 smithi115 conmon[90393]: 2022-05-14T03:09:02.982+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:09:03.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:09:03 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:09:04.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:09:03 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:09:04.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:09:03 smithi115 conmon[90393]: 2022-05-14T03:09:03.968+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:09:04.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:09:04 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:09:04.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:09:04 smithi115 ceph-mon[78593]: pgmap v6990: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:09:05.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:09:04 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:09:05.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:09:04 smithi047 ceph-mon[77147]: pgmap v6990: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:09:05.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:09:04 smithi115 conmon[90393]: 2022-05-14T03:09:04.983+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:09:05.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:09:05 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:09:06.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:09:05 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:09:06.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:09:05 smithi115 conmon[90393]: 2022-05-14T03:09:05.967+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:09:06.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:09:06 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:09:06.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:09:06 smithi115 ceph-mon[78593]: pgmap v6991: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:09:06.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:09:06 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 13280 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:09:07.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:09:06 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:09:07.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:09:06 smithi047 ceph-mon[77147]: pgmap v6991: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:09:07.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:09:06 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 13280 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:09:07.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:09:06 smithi115 conmon[90393]: 2022-05-14T03:09:06.945+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:09:07.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:09:07 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:09:08.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:09:07 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:09:08.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:09:07 smithi115 conmon[90393]: 2022-05-14T03:09:07.945+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:09:08.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:09:08 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:09:08.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:09:08 smithi115 ceph-mon[78593]: pgmap v6992: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:09:09.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:09:08 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:09:09.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:09:08 smithi047 ceph-mon[77147]: pgmap v6992: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:09:09.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:09:08 smithi115 conmon[90393]: 2022-05-14T03:09:08.911+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:09:09.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:09:09 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:09:10.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:09:09 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:09:10.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:09:09 smithi115 conmon[90393]: 2022-05-14T03:09:09.866+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:09:10.841 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:09:10 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:09:10.842 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:09:10 smithi115 ceph-mon[78593]: pgmap v6993: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:09:10.842 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:09:10 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:09:10.842 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:09:10 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:09:10.843 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:09:10 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:09:11.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:09:10 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:09:11.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:09:10 smithi047 ceph-mon[77147]: pgmap v6993: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:09:11.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:09:10 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:09:11.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:09:10 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:09:11.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:09:10 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:09:11.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:09:10 smithi115 conmon[90393]: 2022-05-14T03:09:10.839+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:09:11.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:09:11 smithi115 conmon[90393]: 2022-05-14T03:09:11.799+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:09:11.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:09:11 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:09:11.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:09:11 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 13285 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:09:12.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:09:11 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:09:12.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:09:11 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 13285 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:09:12.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:09:12 smithi115 conmon[90393]: 2022-05-14T03:09:12.773+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:09:12.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:09:12 smithi115 ceph-mon[78593]: pgmap v6994: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:09:12.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:09:12 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:09:13.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:09:12 smithi047 ceph-mon[77147]: pgmap v6994: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:09:13.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:09:12 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:09:13.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:09:13 smithi115 conmon[90393]: 2022-05-14T03:09:13.728+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:09:13.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:09:13 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:09:14.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:09:13 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:09:14.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:09:14 smithi115 conmon[90393]: 2022-05-14T03:09:14.729+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:09:14.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:09:14 smithi115 ceph-mon[78593]: pgmap v6995: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:09:14.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:09:14 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:09:14.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:09:14 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:09:14.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:09:14 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:09:14.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:09:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T03:09:14.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:09:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T03:09:15.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:09:14 smithi047 ceph-mon[77147]: pgmap v6995: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:09:15.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:09:14 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:09:15.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:09:14 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:09:15.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:09:14 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:09:15.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:09:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T03:09:15.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:09:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T03:09:15.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:09:15 smithi115 conmon[90393]: 2022-05-14T03:09:15.751+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:09:15.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:09:15 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:09:16.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:09:15 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:09:16.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:09:16 smithi115 conmon[90393]: 2022-05-14T03:09:16.742+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:09:16.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:09:16 smithi115 ceph-mon[78593]: pgmap v6996: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:09:16.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:09:16 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:09:16.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:09:16 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 13290 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:09:17.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:09:16 smithi047 ceph-mon[77147]: pgmap v6996: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:09:17.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:09:16 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:09:17.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:09:16 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 13290 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:09:17.864 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:09:17 smithi115 conmon[90393]: 2022-05-14T03:09:17.784+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:09:17.865 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:09:17 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:09:18.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:09:17 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:09:18.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:09:18 smithi115 conmon[90393]: 2022-05-14T03:09:18.798+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:09:18.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:09:18 smithi115 ceph-mon[78593]: pgmap v6997: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:09:18.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:09:18 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:09:18.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:09:18 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:09:18.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:09:18 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:09:19.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:09:18 smithi047 ceph-mon[77147]: pgmap v6997: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:09:19.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:09:18 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:09:19.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:09:18 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:09:19.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:09:18 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:09:19.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:09:19 smithi115 conmon[90393]: 2022-05-14T03:09:19.837+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:09:19.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:09:19 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:09:20.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:09:19 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:09:20.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:09:20 smithi115 conmon[90393]: 2022-05-14T03:09:20.801+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:09:20.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:09:20 smithi115 ceph-mon[78593]: pgmap v6998: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:09:20.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:09:20 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:09:21.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:09:20 smithi047 ceph-mon[77147]: pgmap v6998: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:09:21.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:09:20 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:09:21.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:09:21 smithi115 conmon[90393]: 2022-05-14T03:09:21.806+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:09:21.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:09:21 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:09:21.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:09:21 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 13295 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:09:22.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:09:21 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:09:22.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:09:21 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 13295 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:09:22.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:09:22 smithi115 conmon[90393]: 2022-05-14T03:09:22.843+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:09:22.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:09:22 smithi115 ceph-mon[78593]: pgmap v6999: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:09:22.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:09:22 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:09:22.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:09:22 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:09:22.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:09:22 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:09:23.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:09:22 smithi047 ceph-mon[77147]: pgmap v6999: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:09:23.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:09:22 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:09:23.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:09:22 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:09:23.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:09:22 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:09:23.866 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:09:23 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:09:24.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:09:23 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:09:24.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:09:23 smithi115 conmon[90393]: 2022-05-14T03:09:23.864+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:09:25.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:09:24 smithi047 ceph-mon[77147]: pgmap v7000: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:09:25.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:09:24 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:09:25.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:09:24 smithi115 conmon[90393]: 2022-05-14T03:09:24.853+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:09:25.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:09:24 smithi115 ceph-mon[78593]: pgmap v7000: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:09:25.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:09:24 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:09:25.870 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:09:25 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:09:26.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:09:25 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:09:26.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:09:25 smithi115 conmon[90393]: 2022-05-14T03:09:25.868+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:09:27.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:09:26 smithi047 ceph-mon[77147]: pgmap v7001: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:09:27.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:09:26 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:09:27.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:09:26 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 13300 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:09:27.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:09:26 smithi115 conmon[90393]: 2022-05-14T03:09:26.829+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:09:27.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:09:26 smithi115 ceph-mon[78593]: pgmap v7001: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:09:27.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:09:26 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:09:27.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:09:26 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 13300 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:09:28.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:09:27 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:09:28.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:09:27 smithi115 conmon[90393]: 2022-05-14T03:09:27.796+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:09:28.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:09:27 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:09:29.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:09:28 smithi047 ceph-mon[77147]: pgmap v7002: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:09:29.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:09:28 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:09:29.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:09:28 smithi115 conmon[90393]: 2022-05-14T03:09:28.828+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:09:29.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:09:28 smithi115 ceph-mon[78593]: pgmap v7002: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:09:29.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:09:28 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:09:30.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:09:29 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:09:30.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:09:29 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:09:30.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:09:29 smithi115 conmon[90393]: 2022-05-14T03:09:29.814+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:09:31.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:09:30 smithi047 ceph-mon[77147]: pgmap v7003: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:09:31.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:09:30 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:09:31.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:09:30 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:09:31.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:09:30 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:09:31.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:09:30 smithi115 conmon[90393]: 2022-05-14T03:09:30.853+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:09:31.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:09:30 smithi115 ceph-mon[78593]: pgmap v7003: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:09:31.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:09:30 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:09:31.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:09:30 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:09:31.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:09:30 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:09:32.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:09:31 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:09:32.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:09:31 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 13305 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:09:32.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:09:31 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:09:32.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:09:31 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 13305 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:09:32.110 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:09:31 smithi115 conmon[90393]: 2022-05-14T03:09:31.805+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:09:33.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:09:32 smithi047 ceph-mon[77147]: pgmap v7004: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:09:33.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:09:32 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:09:33.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:09:32 smithi115 ceph-mon[78593]: pgmap v7004: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:09:33.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:09:32 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:09:33.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:09:32 smithi115 conmon[90393]: 2022-05-14T03:09:32.780+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:09:34.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:09:33 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:09:34.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:09:33 smithi115 conmon[90393]: 2022-05-14T03:09:33.821+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:09:34.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:09:33 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:09:35.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:09:34 smithi047 ceph-mon[77147]: pgmap v7005: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:09:35.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:09:34 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:09:35.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:09:34 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:09:35.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:09:34 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:09:35.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:09:34 smithi115 conmon[90393]: 2022-05-14T03:09:34.772+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:09:35.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:09:34 smithi115 ceph-mon[78593]: pgmap v7005: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:09:35.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:09:34 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:09:35.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:09:34 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:09:35.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:09:34 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:09:36.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:09:35 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:09:36.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:09:35 smithi115 conmon[90393]: 2022-05-14T03:09:35.756+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:09:36.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:09:35 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:09:37.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:09:36 smithi047 ceph-mon[77147]: pgmap v7006: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:09:37.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:09:36 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:09:37.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:09:36 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 13310 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:09:37.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:09:36 smithi115 conmon[90393]: 2022-05-14T03:09:36.774+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:09:37.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:09:36 smithi115 ceph-mon[78593]: pgmap v7006: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:09:37.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:09:36 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:09:37.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:09:36 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 13310 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:09:38.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:09:37 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:09:38.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:09:37 smithi115 conmon[90393]: 2022-05-14T03:09:37.785+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:09:38.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:09:37 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:09:39.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:09:38 smithi047 ceph-mon[77147]: pgmap v7007: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:09:39.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:09:38 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:09:39.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:09:38 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:09:39.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:09:38 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:09:39.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:09:38 smithi115 conmon[90393]: 2022-05-14T03:09:38.738+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:09:39.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:09:38 smithi115 ceph-mon[78593]: pgmap v7007: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:09:39.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:09:38 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:09:39.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:09:38 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:09:39.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:09:38 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:09:40.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:09:39 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:09:40.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:09:39 smithi115 conmon[90393]: 2022-05-14T03:09:39.745+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:09:40.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:09:39 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:09:41.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:09:40 smithi047 ceph-mon[77147]: pgmap v7008: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:09:41.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:09:40 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:09:41.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:09:40 smithi115 conmon[90393]: 2022-05-14T03:09:40.749+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:09:41.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:09:40 smithi115 ceph-mon[78593]: pgmap v7008: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:09:41.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:09:40 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:09:42.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:09:41 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:09:42.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:09:41 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 13315 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:09:42.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:09:41 smithi115 conmon[90393]: 2022-05-14T03:09:41.736+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:09:42.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:09:41 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:09:42.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:09:41 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 13315 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:09:43.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:09:42 smithi047 ceph-mon[77147]: pgmap v7009: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:09:43.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:09:42 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:09:43.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:09:42 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:09:43.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:09:42 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:09:43.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:09:42 smithi115 conmon[90393]: 2022-05-14T03:09:42.772+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:09:43.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:09:42 smithi115 ceph-mon[78593]: pgmap v7009: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:09:43.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:09:42 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:09:43.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:09:42 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:09:43.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:09:42 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:09:44.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:09:43 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:09:44.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:09:43 smithi115 conmon[90393]: 2022-05-14T03:09:43.756+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:09:44.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:09:43 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:09:45.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:09:44 smithi047 ceph-mon[77147]: pgmap v7010: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:09:45.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:09:44 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:09:45.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:09:44 smithi115 conmon[90393]: 2022-05-14T03:09:44.790+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:09:45.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:09:44 smithi115 ceph-mon[78593]: pgmap v7010: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:09:45.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:09:44 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:09:46.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:09:45 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:09:46.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:09:45 smithi115 conmon[90393]: 2022-05-14T03:09:45.778+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:09:46.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:09:45 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:09:47.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:09:46 smithi047 ceph-mon[77147]: pgmap v7011: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:09:47.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:09:46 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:09:47.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:09:46 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 13320 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:09:47.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:09:46 smithi115 conmon[90393]: 2022-05-14T03:09:46.734+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:09:47.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:09:46 smithi115 ceph-mon[78593]: pgmap v7011: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:09:47.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:09:46 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:09:47.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:09:46 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 13320 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:09:48.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:09:47 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:09:48.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:09:47 smithi115 conmon[90393]: 2022-05-14T03:09:47.766+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:09:48.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:09:47 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:09:49.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:09:48 smithi047 ceph-mon[77147]: pgmap v7012: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:09:49.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:09:48 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:09:49.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:09:48 smithi115 conmon[90393]: 2022-05-14T03:09:48.749+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:09:49.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:09:48 smithi115 ceph-mon[78593]: pgmap v7012: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:09:49.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:09:48 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:09:50.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:09:49 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:09:50.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:09:49 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T03:09:50.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:09:49 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T03:09:50.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:09:49 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T03:09:50.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:09:49 smithi115 conmon[90393]: 2022-05-14T03:09:49.714+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:09:50.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:09:49 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:09:50.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:09:49 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T03:09:50.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:09:49 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T03:09:50.116 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:09:49 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T03:09:51.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:09:50 smithi047 ceph-mon[77147]: pgmap v7013: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:09:51.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:09:50 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:09:51.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:09:50 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:09:51.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:09:50 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:09:51.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:09:50 smithi115 conmon[90393]: 2022-05-14T03:09:50.740+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:09:51.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:09:50 smithi115 ceph-mon[78593]: pgmap v7013: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:09:51.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:09:50 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:09:51.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:09:50 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:09:51.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:09:50 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:09:52.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:09:51 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:09:52.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:09:51 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 13325 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:09:52.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:09:51 smithi115 conmon[90393]: 2022-05-14T03:09:51.691+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:09:52.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:09:51 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:09:52.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:09:51 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 13325 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:09:53.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:09:52 smithi047 ceph-mon[77147]: pgmap v7014: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:09:53.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:09:52 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:09:53.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:09:52 smithi047 conmon[78521]: 2022-05-14T03:09:52.751+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:09:53.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:09:52 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T03:09:53.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:09:52 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:09:53.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:09:52 smithi047 conmon[78521]: yield 2022-05-14T03:09:53.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:09:52 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:09:53.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:09:52 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:09:53.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:09:52 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:09:53.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:09:52 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T03:09:53.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:09:52 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:09:53.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:09:52 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T03:09:53.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:09:52 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:09:53.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:09:52 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T03:09:53.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:09:52 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:09:53.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:09:52 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T03:09:53.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:09:52 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:09:53.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:09:52 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T03:09:53.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:09:52 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:09:53.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:09:52 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:09:53.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:09:52 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:09:53.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:09:52 smithi115 conmon[90393]: 2022-05-14T03:09:52.648+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:09:53.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:09:52 smithi115 ceph-mon[78593]: pgmap v7014: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:09:53.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:09:52 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:09:54.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:09:53 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:09:54.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:09:53 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:09:54.042 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T03:09:54.042 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:09:54.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T03:09:54.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:09:54.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:09:54.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:09:54.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T03:09:54.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:09:54.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T03:09:54.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:09:54.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T03:09:54.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:09:54.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T03:09:54.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:09:54.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T03:09:54.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:09:54.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:09:54.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:09:54.046 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:09:53 smithi047 ceph-mon[77147]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T03:09:54.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:09:53 smithi115 conmon[90393]: 2022-05-14T03:09:53.642+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:09:54.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:09:53 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:09:54.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:09:53 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:09:54.109 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T03:09:54.109 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:09:54.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T03:09:54.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:09:54.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:09:54.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:09:54.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T03:09:54.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:09:54.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T03:09:54.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:09:54.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T03:09:54.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:09:54.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T03:09:54.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:09:54.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T03:09:54.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:09:54.113 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:09:54.113 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:09:54.113 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:09:53 smithi115 ceph-mon[78593]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T03:09:55.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:09:54 smithi047 ceph-mon[77147]: pgmap v7015: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:09:55.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:09:54 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:09:55.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:09:54 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:09:55.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:09:54 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:09:55.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:09:54 smithi115 conmon[90393]: 2022-05-14T03:09:54.679+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:09:55.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:09:54 smithi115 ceph-mon[78593]: pgmap v7015: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:09:55.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:09:54 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:09:55.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:09:54 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:09:55.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:09:54 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:09:56.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:09:55 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:09:56.042 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:09:55 smithi047 conmon[78521]: 2022-05-14T03:09:55.823+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:09:56.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:09:55 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T03:09:56.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:09:55 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:09:56.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:09:55 smithi047 conmon[78521]: yield 2022-05-14T03:09:56.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:09:55 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:09:56.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:09:55 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:09:56.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:09:55 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:09:56.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:09:55 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T03:09:56.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:09:55 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:09:56.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:09:55 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T03:09:56.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:09:55 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:09:56.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:09:55 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T03:09:56.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:09:55 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:09:56.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:09:55 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T03:09:56.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:09:55 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:09:56.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:09:55 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T03:09:56.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:09:55 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:09:56.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:09:55 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:09:56.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:09:55 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:09:56.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:09:55 smithi047 conmon[78521]: 2022-05-14T03:09:55.824+0000 7f534177b700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T03:09:56.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:09:55 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T03:09:56.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:09:55 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:09:56.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:09:55 smithi047 conmon[78521]: yield 2022-05-14T03:09:56.048 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:09:55 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:09:56.048 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:09:55 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:09:56.048 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:09:55 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:09:56.048 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:09:55 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T03:09:56.048 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:09:55 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:09:56.048 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:09:55 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T03:09:56.049 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:09:55 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:09:56.049 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:09:55 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T03:09:56.049 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:09:55 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:09:56.049 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:09:55 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T03:09:56.050 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:09:55 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:09:56.050 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:09:55 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T03:09:56.050 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:09:55 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:09:56.050 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:09:55 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:09:56.050 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:09:55 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:09:56.051 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:09:55 smithi047 conmon[78521]: 2022-05-14T03:09:56.051 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:09:55 smithi047 conmon[78521]: During handling of the above exception, another exception occurred: 2022-05-14T03:09:56.051 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:09:55 smithi047 conmon[78521]: 2022-05-14T03:09:56.051 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:09:55 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T03:09:56.052 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:09:55 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T03:09:56.052 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:09:55 smithi047 conmon[78521]: return f(*arg) 2022-05-14T03:09:56.052 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:09:55 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T03:09:56.052 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:09:55 smithi047 conmon[78521]: self._write_client_files(client_files, host) 2022-05-14T03:09:56.052 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:09:55 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T03:09:56.053 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:09:55 smithi047 conmon[78521]: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T03:09:56.053 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:09:55 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T03:09:56.053 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:09:55 smithi047 conmon[78521]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T03:09:56.053 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:09:55 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T03:09:56.053 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:09:55 smithi047 conmon[78521]: return self.event_loop.get_result(coro) 2022-05-14T03:09:56.054 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:09:55 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T03:09:56.054 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:09:55 smithi047 conmon[78521]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T03:09:56.054 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:09:55 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T03:09:56.054 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:09:55 smithi047 conmon[78521]: return self.__get_result() 2022-05-14T03:09:56.054 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:09:55 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T03:09:56.055 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:09:55 smithi047 conmon[78521]: raise self._exception 2022-05-14T03:09:56.055 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:09:55 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T03:09:56.055 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:09:55 smithi047 conmon[78521]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T03:09:56.055 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:09:55 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T03:09:56.056 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:09:55 smithi047 conmon[78521]: conn = await self._remote_connection(host, addr) 2022-05-14T03:09:56.056 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:09:55 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T03:09:56.056 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:09:55 smithi047 conmon[78521]: raise 2022-05-14T03:09:56.056 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:09:55 smithi047 conmon[78521]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T03:09:56.056 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:09:55 smithi047 conmon[78521]: self.gen.throw(type, value, traceback) 2022-05-14T03:09:56.057 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:09:55 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T03:09:56.057 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:09:55 smithi047 conmon[78521]: raise OrchestratorError(msg) 2022-05-14T03:09:56.057 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:09:55 smithi047 conmon[78521]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:09:56.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:09:55 smithi115 conmon[90393]: 2022-05-14T03:09:55.641+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:09:56.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:09:55 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:09:57.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:09:56 smithi047 ceph-mon[77147]: pgmap v7016: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:09:57.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:09:56 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:09:57.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:09:56 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 13330 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:09:57.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:09:56 smithi115 conmon[90393]: 2022-05-14T03:09:56.620+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:09:57.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:09:56 smithi115 ceph-mon[78593]: pgmap v7016: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:09:57.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:09:56 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:09:57.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:09:56 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 13330 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:09:58.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:09:57 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:09:58.042 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T03:09:58.042 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:09:58.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T03:09:58.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:09:58.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:09:58.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:09:58.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T03:09:58.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:09:58.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T03:09:58.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:09:58.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T03:09:58.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:09:58.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T03:09:58.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:09:58.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T03:09:58.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:09:58.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:09:58.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:09:58.046 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:09:57 smithi047 ceph-mon[77147]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T03:09:58.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T03:09:58.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:09:58.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T03:09:58.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:09:58.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:09:58.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:09:58.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T03:09:58.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:09:58.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T03:09:58.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:09:58.049 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T03:09:58.049 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:09:58.049 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T03:09:58.049 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:09:58.050 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T03:09:58.050 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:09:58.050 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:09:58.050 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:09:58.051 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T03:09:58.051 INFO:journalctl@ceph.mon.b.smithi047.stdout: During handling of the above exception, another exception occurred: 2022-05-14T03:09:58.051 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T03:09:58.051 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T03:09:58.051 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T03:09:58.052 INFO:journalctl@ceph.mon.b.smithi047.stdout: return f(*arg) 2022-05-14T03:09:58.052 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T03:09:58.052 INFO:journalctl@ceph.mon.b.smithi047.stdout: self._write_client_files(client_files, host) 2022-05-14T03:09:58.052 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T03:09:58.053 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T03:09:58.053 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T03:09:58.053 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T03:09:58.053 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T03:09:58.054 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.event_loop.get_result(coro) 2022-05-14T03:09:58.054 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T03:09:58.054 INFO:journalctl@ceph.mon.b.smithi047.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T03:09:58.054 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T03:09:58.054 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.__get_result() 2022-05-14T03:09:58.055 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T03:09:58.055 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise self._exception 2022-05-14T03:09:58.055 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T03:09:58.055 INFO:journalctl@ceph.mon.b.smithi047.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T03:09:58.055 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T03:09:58.056 INFO:journalctl@ceph.mon.b.smithi047.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T03:09:58.056 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T03:09:58.057 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise 2022-05-14T03:09:58.057 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T03:09:58.057 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.gen.throw(type, value, traceback) 2022-05-14T03:09:58.057 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T03:09:58.058 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OrchestratorError(msg) 2022-05-14T03:09:58.058 INFO:journalctl@ceph.mon.b.smithi047.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:09:58.058 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:09:57 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:09:58.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:09:57 smithi115 conmon[90393]: 2022-05-14T03:09:57.614+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:09:58.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:09:57 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:09:58.109 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T03:09:58.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:09:58.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T03:09:58.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:09:58.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:09:58.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:09:58.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T03:09:58.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:09:58.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T03:09:58.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:09:58.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T03:09:58.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:09:58.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T03:09:58.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:09:58.113 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T03:09:58.113 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:09:58.113 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:09:58.113 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:09:58.114 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:09:57 smithi115 ceph-mon[78593]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T03:09:58.114 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T03:09:58.114 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:09:58.114 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T03:09:58.115 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:09:58.115 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:09:58.115 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:09:58.115 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T03:09:58.115 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:09:58.116 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T03:09:58.116 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:09:58.116 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T03:09:58.116 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:09:58.116 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T03:09:58.117 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:09:58.117 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T03:09:58.117 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:09:58.117 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:09:58.118 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:09:58.118 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T03:09:58.118 INFO:journalctl@ceph.mon.c.smithi115.stdout: During handling of the above exception, another exception occurred: 2022-05-14T03:09:58.118 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T03:09:58.119 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T03:09:58.119 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T03:09:58.119 INFO:journalctl@ceph.mon.c.smithi115.stdout: return f(*arg) 2022-05-14T03:09:58.120 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T03:09:58.120 INFO:journalctl@ceph.mon.c.smithi115.stdout: self._write_client_files(client_files, host) 2022-05-14T03:09:58.120 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T03:09:58.120 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T03:09:58.121 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T03:09:58.121 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T03:09:58.121 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T03:09:58.121 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.event_loop.get_result(coro) 2022-05-14T03:09:58.122 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T03:09:58.122 INFO:journalctl@ceph.mon.c.smithi115.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T03:09:58.122 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T03:09:58.122 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.__get_result() 2022-05-14T03:09:58.122 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T03:09:58.123 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise self._exception 2022-05-14T03:09:58.123 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T03:09:58.123 INFO:journalctl@ceph.mon.c.smithi115.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T03:09:58.123 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T03:09:58.123 INFO:journalctl@ceph.mon.c.smithi115.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T03:09:58.124 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T03:09:58.124 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise 2022-05-14T03:09:58.124 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T03:09:58.124 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.gen.throw(type, value, traceback) 2022-05-14T03:09:58.124 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T03:09:58.125 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OrchestratorError(msg) 2022-05-14T03:09:58.125 INFO:journalctl@ceph.mon.c.smithi115.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:09:58.125 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:09:57 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:09:59.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:09:58 smithi047 ceph-mon[77147]: pgmap v7017: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:09:59.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:09:58 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:09:59.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:09:58 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:09:59.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:09:58 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:09:59.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:09:58 smithi115 conmon[90393]: 2022-05-14T03:09:58.648+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:09:59.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:09:58 smithi115 ceph-mon[78593]: pgmap v7017: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:09:59.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:09:58 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:09:59.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:09:58 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:09:59.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:09:58 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:10:00.001 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:09:59 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:10:00.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:09:59 smithi115 conmon[90393]: 2022-05-14T03:09:59.672+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:10:00.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:09:59 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:10:00.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:10:00 smithi047 conmon[77123]: 2022-05-14T03:09:59.999+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : Health detail: HEALTH_ERR 1 filesystem is degraded; 1 mds daemon damaged; 1 MDSs are read only; 1 MDSs report slow metadata IOs; 1/3 mons down, quorum b,c; Reduced data availability: 1 pg inactive; 9 slow ops, oldest one blocked for 13335 sec, osd.5 has slow ops 2022-05-14T03:10:00.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:10:00 smithi047 conmon[77123]: 2022-05-14T03:09:59.999+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : [WRN] FS_DEGRADED: 1 filesystem is degraded 2022-05-14T03:10:00.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:10:00 smithi047 conmon[77123]: 2022-05-14T03:09:59.999+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : fs cephfs is degraded 2022-05-14T03:10:00.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:10:00 smithi047 conmon[77123]: 2022-05-14T03:09:59.999+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : [ERR] MDS_DAMAGE: 1 mds daemon damaged 2022-05-14T03:10:00.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:10:00 smithi047 conmon[77123]: 2022-05-14T03:09:59.999+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : fs cephfs mds.2 is damaged 2022-05-14T03:10:00.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:10:00 smithi047 conmon[77123]: 2022-05-14T03:09:59.999+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : [WRN] MDS_READ_ONLY: 1 MDSs are read only 2022-05-14T03:10:00.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:10:00 smithi047 conmon[77123]: 2022-05-14T03:09:59.999+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : mds.l(mds.0): MDS in read-only mode 2022-05-14T03:10:00.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:10:00 smithi047 conmon[77123]: 2022-05-14T03:09:59.999+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : [WRN] MDS_SLOW_METADATA_IO: 1 MDSs report slow metadata IOs 2022-05-14T03:10:00.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:10:00 smithi047 conmon[77123]: 2022-05-14T03:09:59.999+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : mds.l(mds.0): 4 slow metadata IOs are blocked > 180 secs, oldest blocked for 13633 secs 2022-05-14T03:10:00.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:10:00 smithi047 conmon[77123]: 2022-05-14T03:09:59.999+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : [WRN] MON_DOWN: 1/3 mons down, quorum b,c 2022-05-14T03:10:00.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:10:00 smithi047 conmon[77123]: 2022-05-14T03:10:00.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:10:00 smithi047 conmon[77123]: 2022-05-14T03:09:59.999+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : mon.a (rank 0) addr [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] is down (out of quorum) 2022-05-14T03:10:00.295 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:10:00 smithi047 conmon[77123]: 2022-05-14T03:09:59.999+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : [WRN] PG_AVAILABILITY: Reduced data availability: 1 pg inactive 2022-05-14T03:10:00.295 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:10:00 smithi047 conmon[77123]: 2022-05-14T03:09:59.999+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : pg 2.1 is stuck inactive for 3h, current state unknown, last acting [] 2022-05-14T03:10:00.295 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:10:00 smithi047 conmon[77123]: 2022-05-14T03:09:59.999+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : [WRN] SLOW_OPS: 9 slow ops, oldest one blocked for 13335 sec, osd.5 has slow ops 2022-05-14T03:10:01.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:10:00 smithi047 ceph-mon[77147]: pgmap v7018: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:10:01.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:10:00 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:10:01.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:10:00 smithi047 ceph-mon[77147]: Health detail: HEALTH_ERR 1 filesystem is degraded; 1 mds daemon damaged; 1 MDSs are read only; 1 MDSs report slow metadata IOs; 1/3 mons down, quorum b,c; Reduced data availability: 1 pg inactive; 9 slow ops, oldest one blocked for 13335 sec, osd.5 has slow ops 2022-05-14T03:10:01.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:10:00 smithi047 ceph-mon[77147]: [WRN] FS_DEGRADED: 1 filesystem is degraded 2022-05-14T03:10:01.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:10:00 smithi047 ceph-mon[77147]: fs cephfs is degraded 2022-05-14T03:10:01.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:10:00 smithi047 ceph-mon[77147]: [ERR] MDS_DAMAGE: 1 mds daemon damaged 2022-05-14T03:10:01.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:10:00 smithi047 ceph-mon[77147]: fs cephfs mds.2 is damaged 2022-05-14T03:10:01.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:10:00 smithi047 ceph-mon[77147]: [WRN] MDS_READ_ONLY: 1 MDSs are read only 2022-05-14T03:10:01.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:10:00 smithi047 ceph-mon[77147]: mds.l(mds.0): MDS in read-only mode 2022-05-14T03:10:01.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:10:00 smithi047 ceph-mon[77147]: [WRN] MDS_SLOW_METADATA_IO: 1 MDSs report slow metadata IOs 2022-05-14T03:10:01.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:10:00 smithi047 ceph-mon[77147]: mds.l(mds.0): 4 slow metadata IOs are blocked > 180 secs, oldest blocked for 13633 secs 2022-05-14T03:10:01.045 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:10:00 smithi047 ceph-mon[77147]: [WRN] MON_DOWN: 1/3 mons down, quorum b,c 2022-05-14T03:10:01.045 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:10:00 smithi047 ceph-mon[77147]: mon.a (rank 0) addr [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] is down (out of quorum) 2022-05-14T03:10:01.045 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:10:00 smithi047 ceph-mon[77147]: [WRN] PG_AVAILABILITY: Reduced data availability: 1 pg inactive 2022-05-14T03:10:01.045 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:10:00 smithi047 ceph-mon[77147]: pg 2.1 is stuck inactive for 3h, current state unknown, last acting [] 2022-05-14T03:10:01.046 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:10:00 smithi047 ceph-mon[77147]: [WRN] SLOW_OPS: 9 slow ops, oldest one blocked for 13335 sec, osd.5 has slow ops 2022-05-14T03:10:01.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:10:00 smithi115 conmon[90393]: 2022-05-14T03:10:00.662+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:10:01.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:10:00 smithi115 ceph-mon[78593]: pgmap v7018: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:10:01.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:10:00 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:10:01.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:10:00 smithi115 ceph-mon[78593]: Health detail: HEALTH_ERR 1 filesystem is degraded; 1 mds daemon damaged; 1 MDSs are read only; 1 MDSs report slow metadata IOs; 1/3 mons down, quorum b,c; Reduced data availability: 1 pg inactive; 9 slow ops, oldest one blocked for 13335 sec, osd.5 has slow ops 2022-05-14T03:10:01.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:10:00 smithi115 ceph-mon[78593]: [WRN] FS_DEGRADED: 1 filesystem is degraded 2022-05-14T03:10:01.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:10:00 smithi115 ceph-mon[78593]: fs cephfs is degraded 2022-05-14T03:10:01.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:10:00 smithi115 ceph-mon[78593]: [ERR] MDS_DAMAGE: 1 mds daemon damaged 2022-05-14T03:10:01.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:10:00 smithi115 ceph-mon[78593]: fs cephfs mds.2 is damaged 2022-05-14T03:10:01.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:10:00 smithi115 ceph-mon[78593]: [WRN] MDS_READ_ONLY: 1 MDSs are read only 2022-05-14T03:10:01.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:10:00 smithi115 ceph-mon[78593]: mds.l(mds.0): MDS in read-only mode 2022-05-14T03:10:01.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:10:00 smithi115 ceph-mon[78593]: [WRN] MDS_SLOW_METADATA_IO: 1 MDSs report slow metadata IOs 2022-05-14T03:10:01.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:10:00 smithi115 ceph-mon[78593]: mds.l(mds.0): 4 slow metadata IOs are blocked > 180 secs, oldest blocked for 13633 secs 2022-05-14T03:10:01.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:10:00 smithi115 ceph-mon[78593]: [WRN] MON_DOWN: 1/3 mons down, quorum b,c 2022-05-14T03:10:01.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:10:00 smithi115 ceph-mon[78593]: mon.a (rank 0) addr [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] is down (out of quorum) 2022-05-14T03:10:01.112 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:10:00 smithi115 ceph-mon[78593]: [WRN] PG_AVAILABILITY: Reduced data availability: 1 pg inactive 2022-05-14T03:10:01.112 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:10:00 smithi115 ceph-mon[78593]: pg 2.1 is stuck inactive for 3h, current state unknown, last acting [] 2022-05-14T03:10:01.112 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:10:00 smithi115 ceph-mon[78593]: [WRN] SLOW_OPS: 9 slow ops, oldest one blocked for 13335 sec, osd.5 has slow ops 2022-05-14T03:10:02.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:10:01 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:10:02.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:10:01 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 13335 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:10:02.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:10:01 smithi115 conmon[90393]: 2022-05-14T03:10:01.622+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:10:02.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:10:01 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:10:02.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:10:01 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 13335 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:10:03.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:10:02 smithi047 ceph-mon[77147]: pgmap v7019: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:10:03.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:10:02 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:10:03.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:10:02 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:10:03.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:10:02 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:10:03.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:10:02 smithi115 conmon[90393]: 2022-05-14T03:10:02.662+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:10:03.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:10:02 smithi115 ceph-mon[78593]: pgmap v7019: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:10:03.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:10:02 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:10:03.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:10:02 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:10:03.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:10:02 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:10:04.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:10:03 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:10:04.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:10:03 smithi115 conmon[90393]: 2022-05-14T03:10:03.650+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:10:04.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:10:03 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:10:05.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:10:04 smithi047 ceph-mon[77147]: pgmap v7020: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:10:05.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:10:04 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:10:05.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:10:04 smithi115 conmon[90393]: 2022-05-14T03:10:04.690+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:10:05.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:10:04 smithi115 ceph-mon[78593]: pgmap v7020: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:10:05.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:10:04 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:10:06.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:10:05 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:10:06.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:10:05 smithi115 conmon[90393]: 2022-05-14T03:10:05.714+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:10:06.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:10:05 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:10:07.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:10:06 smithi047 ceph-mon[77147]: pgmap v7021: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:10:07.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:10:06 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:10:07.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:10:06 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 13340 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:10:07.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:10:06 smithi115 conmon[90393]: 2022-05-14T03:10:06.722+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:10:07.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:10:06 smithi115 ceph-mon[78593]: pgmap v7021: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:10:07.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:10:06 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:10:07.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:10:06 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 13340 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:10:08.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:10:07 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:10:08.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:10:07 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:10:08.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:10:07 smithi115 conmon[90393]: 2022-05-14T03:10:07.731+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:10:09.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:10:08 smithi047 ceph-mon[77147]: pgmap v7022: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:10:09.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:10:08 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:10:09.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:10:08 smithi115 conmon[90393]: 2022-05-14T03:10:08.769+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:10:09.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:10:08 smithi115 ceph-mon[78593]: pgmap v7022: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:10:09.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:10:08 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:10:10.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:10:09 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:10:10.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:10:09 smithi115 conmon[90393]: 2022-05-14T03:10:09.781+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:10:10.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:10:09 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:10:11.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:10:10 smithi047 ceph-mon[77147]: pgmap v7023: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:10:11.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:10:10 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:10:11.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:10:10 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:10:11.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:10:10 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:10:11.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:10:10 smithi115 conmon[90393]: 2022-05-14T03:10:10.805+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:10:11.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:10:10 smithi115 ceph-mon[78593]: pgmap v7023: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:10:11.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:10:10 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:10:11.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:10:10 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:10:11.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:10:10 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:10:12.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:10:11 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:10:12.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:10:11 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 13345 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:10:12.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:10:11 smithi115 conmon[90393]: 2022-05-14T03:10:11.810+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:10:12.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:10:11 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:10:12.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:10:11 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 13345 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:10:13.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:10:12 smithi047 ceph-mon[77147]: pgmap v7024: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:10:13.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:10:12 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:10:13.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:10:12 smithi115 conmon[90393]: 2022-05-14T03:10:12.822+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:10:13.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:10:12 smithi115 ceph-mon[78593]: pgmap v7024: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:10:13.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:10:12 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:10:14.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:10:13 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:10:14.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:10:13 smithi115 conmon[90393]: 2022-05-14T03:10:13.832+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:10:14.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:10:13 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:10:15.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:10:14 smithi047 ceph-mon[77147]: pgmap v7025: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:10:15.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:10:14 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:10:15.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:10:14 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:10:15.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:10:14 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:10:15.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:10:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T03:10:15.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:10:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T03:10:15.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:10:14 smithi115 conmon[90393]: 2022-05-14T03:10:14.851+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:10:15.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:10:14 smithi115 ceph-mon[78593]: pgmap v7025: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:10:15.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:10:14 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:10:15.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:10:14 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:10:15.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:10:14 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:10:15.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:10:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T03:10:15.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:10:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T03:10:16.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:10:15 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:10:16.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:10:15 smithi115 conmon[90393]: 2022-05-14T03:10:15.885+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:10:16.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:10:15 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:10:17.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:10:16 smithi047 ceph-mon[77147]: pgmap v7026: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:10:17.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:10:16 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:10:17.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:10:16 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 13350 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:10:17.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:10:16 smithi115 conmon[90393]: 2022-05-14T03:10:16.877+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:10:17.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:10:16 smithi115 ceph-mon[78593]: pgmap v7026: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:10:17.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:10:16 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:10:17.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:10:16 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 13350 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:10:18.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:10:17 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:10:18.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:10:17 smithi115 conmon[90393]: 2022-05-14T03:10:17.913+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:10:18.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:10:17 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:10:19.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:10:18 smithi047 ceph-mon[77147]: pgmap v7027: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:10:19.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:10:18 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:10:19.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:10:18 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:10:19.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:10:18 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:10:19.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:10:18 smithi115 conmon[90393]: 2022-05-14T03:10:18.951+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:10:19.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:10:18 smithi115 ceph-mon[78593]: pgmap v7027: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:10:19.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:10:18 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:10:19.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:10:18 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:10:19.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:10:18 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:10:20.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:10:19 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:10:20.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:10:19 smithi047 ceph-mon[77147]: pgmap v7028: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:10:20.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:10:19 smithi115 conmon[90393]: 2022-05-14T03:10:19.960+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:10:20.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:10:19 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:10:20.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:10:19 smithi115 ceph-mon[78593]: pgmap v7028: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:10:21.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:10:20 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:10:21.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:10:20 smithi115 conmon[90393]: 2022-05-14T03:10:20.927+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:10:21.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:10:20 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:10:22.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:10:21 smithi115 conmon[90393]: 2022-05-14T03:10:21.954+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:10:22.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:10:21 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:10:22.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:10:21 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 13355 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:10:22.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:10:21 smithi115 ceph-mon[78593]: pgmap v7029: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:10:22.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:10:21 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:10:22.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:10:21 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 13355 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:10:22.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:10:21 smithi047 ceph-mon[77147]: pgmap v7029: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:10:23.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:10:22 smithi115 conmon[90393]: 2022-05-14T03:10:22.930+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:10:23.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:10:22 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:10:23.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:10:22 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:10:23.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:10:22 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:10:23.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:10:22 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:10:23.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:10:22 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:10:23.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:10:22 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:10:24.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:10:23 smithi115 conmon[90393]: 2022-05-14T03:10:23.917+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:10:24.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:10:23 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:10:24.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:10:23 smithi115 ceph-mon[78593]: pgmap v7030: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:10:24.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:10:23 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:10:24.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:10:23 smithi047 ceph-mon[77147]: pgmap v7030: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:10:25.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:10:24 smithi115 conmon[90393]: 2022-05-14T03:10:24.951+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:10:25.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:10:24 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:10:25.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:10:24 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:10:26.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:10:25 smithi115 conmon[90393]: 2022-05-14T03:10:25.923+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:10:26.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:10:25 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:10:26.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:10:25 smithi115 ceph-mon[78593]: pgmap v7031: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:10:26.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:10:25 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:10:26.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:10:25 smithi047 ceph-mon[77147]: pgmap v7031: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:10:27.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:10:26 smithi115 conmon[90393]: 2022-05-14T03:10:26.911+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:10:27.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:10:26 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:10:27.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:10:26 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 13360 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:10:27.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:10:26 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:10:27.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:10:26 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 13360 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:10:28.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:10:27 smithi115 conmon[90393]: 2022-05-14T03:10:27.882+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:10:28.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:10:27 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:10:28.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:10:27 smithi115 ceph-mon[78593]: pgmap v7032: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:10:28.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:10:27 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:10:28.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:10:27 smithi047 ceph-mon[77147]: pgmap v7032: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:10:29.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:10:28 smithi115 conmon[90393]: 2022-05-14T03:10:28.875+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:10:29.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:10:28 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:10:29.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:10:28 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:10:30.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:10:29 smithi115 conmon[90393]: 2022-05-14T03:10:29.912+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:10:30.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:10:29 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:10:30.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:10:29 smithi115 ceph-mon[78593]: pgmap v7033: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:10:30.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:10:29 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:10:30.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:10:29 smithi047 ceph-mon[77147]: pgmap v7033: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:10:31.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:10:30 smithi115 conmon[90393]: 2022-05-14T03:10:30.963+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:10:31.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:10:30 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:10:31.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:10:30 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:10:31.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:10:30 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:10:31.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:10:30 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:10:31.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:10:30 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:10:31.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:10:30 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:10:32.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:10:31 smithi115 conmon[90393]: 2022-05-14T03:10:31.987+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:10:32.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:10:31 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:10:32.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:10:31 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 13365 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:10:32.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:10:31 smithi115 ceph-mon[78593]: pgmap v7034: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:10:32.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:10:31 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:10:32.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:10:31 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 13365 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:10:32.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:10:31 smithi047 ceph-mon[77147]: pgmap v7034: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:10:33.298 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:10:32 smithi115 conmon[90393]: 2022-05-14T03:10:32.951+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:10:33.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:10:33 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:10:33.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:10:33 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:10:34.303 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:10:33 smithi115 conmon[90393]: 2022-05-14T03:10:33.908+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:10:34.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:10:34 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:10:34.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:10:34 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:10:34.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:10:34 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:10:34.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:10:34 smithi115 ceph-mon[78593]: pgmap v7035: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:10:34.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:10:34 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:10:34.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:10:34 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:10:34.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:10:34 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:10:34.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:10:34 smithi047 ceph-mon[77147]: pgmap v7035: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:10:35.305 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:10:34 smithi115 conmon[90393]: 2022-05-14T03:10:34.957+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:10:35.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:10:35 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:10:35.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:10:35 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:10:36.309 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:10:35 smithi115 conmon[90393]: 2022-05-14T03:10:35.956+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:10:36.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:10:36 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:10:36.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:10:36 smithi115 ceph-mon[78593]: pgmap v7036: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:10:36.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:10:36 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:10:36.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:10:36 smithi047 ceph-mon[77147]: pgmap v7036: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:10:37.311 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:10:36 smithi115 conmon[90393]: 2022-05-14T03:10:36.968+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:10:37.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:10:37 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:10:37.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:10:37 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 13370 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:10:37.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:10:37 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:10:37.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:10:37 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 13370 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:10:38.320 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:10:37 smithi115 conmon[90393]: 2022-05-14T03:10:37.958+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:10:38.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:10:38 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:10:38.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:10:38 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:10:38.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:10:38 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:10:38.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:10:38 smithi115 ceph-mon[78593]: pgmap v7037: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:10:38.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:10:38 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:10:38.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:10:38 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:10:38.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:10:38 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:10:38.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:10:38 smithi047 ceph-mon[77147]: pgmap v7037: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:10:39.321 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:10:38 smithi115 conmon[90393]: 2022-05-14T03:10:38.940+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:10:39.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:10:39 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:10:39.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:10:39 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:10:40.326 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:10:39 smithi115 conmon[90393]: 2022-05-14T03:10:39.972+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:10:40.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:10:40 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:10:40.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:10:40 smithi115 ceph-mon[78593]: pgmap v7038: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:10:40.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:10:40 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:10:40.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:10:40 smithi047 ceph-mon[77147]: pgmap v7038: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:10:41.331 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:10:40 smithi115 conmon[90393]: 2022-05-14T03:10:40.998+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:10:41.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:10:41 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:10:41.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:10:41 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:10:42.334 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:10:42 smithi115 conmon[90393]: 2022-05-14T03:10:42.033+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:10:42.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:10:42 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:10:42.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:10:42 smithi115 ceph-mon[78593]: pgmap v7039: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:10:42.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:10:42 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:10:42.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:10:42 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:10:42.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:10:42 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:10:42.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:10:42 smithi047 ceph-mon[77147]: pgmap v7039: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:10:42.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:10:42 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:10:42.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:10:42 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:10:43.340 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:10:43 smithi115 conmon[90393]: 2022-05-14T03:10:43.044+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:10:43.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:10:43 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:10:43.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:10:43 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:10:44.339 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:10:44 smithi115 conmon[90393]: 2022-05-14T03:10:44.024+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:10:44.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:10:44 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:10:44.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:10:44 smithi115 ceph-mon[78593]: pgmap v7040: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:10:44.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:10:44 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:10:44.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:10:44 smithi047 ceph-mon[77147]: pgmap v7040: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:10:45.344 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:10:45 smithi115 conmon[90393]: 2022-05-14T03:10:45.005+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:10:45.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:10:45 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:10:45.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:10:45 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:10:46.346 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:10:46 smithi115 conmon[90393]: 2022-05-14T03:10:46.041+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:10:46.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:10:46 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:10:46.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:10:46 smithi115 ceph-mon[78593]: pgmap v7041: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:10:46.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:10:46 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 13380 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:10:46.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:10:46 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:10:46.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:10:46 smithi047 ceph-mon[77147]: pgmap v7041: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:10:46.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:10:46 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 13380 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:10:47.351 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:10:47 smithi115 conmon[90393]: 2022-05-14T03:10:47.004+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:10:47.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:10:47 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:10:47.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:10:47 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:10:48.352 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:10:48 smithi115 conmon[90393]: 2022-05-14T03:10:48.028+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:10:48.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:10:48 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:10:48.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:10:48 smithi115 ceph-mon[78593]: pgmap v7042: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:10:48.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:10:48 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:10:48.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:10:48 smithi047 ceph-mon[77147]: pgmap v7042: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:10:49.357 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:10:49 smithi115 conmon[90393]: 2022-05-14T03:10:49.069+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:10:49.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:10:49 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:10:49.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:10:49 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:10:50.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:10:50 smithi115 conmon[90393]: 2022-05-14T03:10:50.031+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:10:50.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:10:50 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:10:50.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:10:50 smithi047 ceph-mon[77147]: pgmap v7043: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:10:50.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:10:50 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:10:50.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:10:50 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:10:50.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:10:50 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:10:50.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:10:50 smithi115 ceph-mon[78593]: pgmap v7043: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:10:50.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:10:50 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:10:50.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:10:50 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:10:51.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:10:51 smithi115 conmon[90393]: 2022-05-14T03:10:51.081+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:10:51.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:10:51 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:10:51.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:10:51 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 13385 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:10:51.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:10:51 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:10:51.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:10:51 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 13385 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:10:52.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:10:52 smithi115 conmon[90393]: 2022-05-14T03:10:52.093+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:10:52.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:10:52 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:10:52.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:10:52 smithi047 ceph-mon[77147]: pgmap v7044: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:10:52.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:10:52 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:10:52.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:10:52 smithi115 ceph-mon[78593]: pgmap v7044: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:10:53.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:10:53 smithi115 conmon[90393]: 2022-05-14T03:10:53.083+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:10:53.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:10:53 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:10:53.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:10:53 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:10:54.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:10:54 smithi115 conmon[90393]: 2022-05-14T03:10:54.073+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:10:54.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:10:54 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:10:54.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:10:54 smithi047 ceph-mon[77147]: pgmap v7045: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:10:54.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:10:54 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:10:54.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:10:54 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:10:54.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:10:54 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:10:54.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:10:54 smithi115 ceph-mon[78593]: pgmap v7045: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:10:54.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:10:54 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:10:54.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:10:54 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:10:55.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:10:55 smithi115 conmon[90393]: 2022-05-14T03:10:55.043+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:10:55.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:10:55 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:10:55.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:10:55 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:10:56.329 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:10:56 smithi115 conmon[90393]: 2022-05-14T03:10:56.010+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:10:56.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:10:56 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:10:56.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:10:56 smithi115 ceph-mon[78593]: pgmap v7046: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:10:56.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:10:56 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T03:10:56.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:10:56 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T03:10:56.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:10:56 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T03:10:56.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:10:56 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 13390 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:10:56.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:10:56 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:10:56.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:10:56 smithi047 ceph-mon[77147]: pgmap v7046: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:10:56.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:10:56 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T03:10:56.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:10:56 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T03:10:56.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:10:56 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T03:10:56.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:10:56 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 13390 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:10:57.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:10:56 smithi115 conmon[90393]: 2022-05-14T03:10:56.990+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:10:57.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:10:57 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:10:57.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:10:57 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:10:58.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:10:57 smithi115 conmon[90393]: 2022-05-14T03:10:57.942+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:10:58.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:10:58 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:10:58.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:10:58 smithi047 ceph-mon[77147]: pgmap v7047: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:10:58.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:10:58 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:10:58.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:10:58 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:10:58.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:10:58 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:10:58.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:10:58 smithi115 ceph-mon[78593]: pgmap v7047: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:10:58.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:10:58 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:10:58.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:10:58 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:10:59.292 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:10:58 smithi047 conmon[78521]: 2022-05-14T03:10:58.928+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:10:59.292 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:10:58 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T03:10:59.293 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:10:58 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:10:59.293 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:10:58 smithi047 conmon[78521]: yield 2022-05-14T03:10:59.293 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:10:58 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:10:59.293 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:10:58 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:10:59.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:10:58 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:10:59.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:10:58 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T03:10:59.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:10:58 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:10:59.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:10:58 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T03:10:59.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:10:58 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:10:59.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:10:58 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T03:10:59.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:10:58 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:10:59.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:10:58 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T03:10:59.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:10:58 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:10:59.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:10:58 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T03:10:59.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:10:58 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:10:59.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:10:58 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:10:59.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:10:58 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:10:59.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:10:58 smithi115 conmon[90393]: 2022-05-14T03:10:58.955+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:10:59.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:10:59 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:10:59.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:10:59 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:11:00.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:10:59 smithi115 conmon[90393]: 2022-05-14T03:10:59.923+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:11:00.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:11:00 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:11:00.792 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T03:11:00.792 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:11:00.793 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T03:11:00.793 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:11:00.793 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:11:00.793 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:11:00.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T03:11:00.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:11:00.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T03:11:00.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:11:00.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T03:11:00.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:11:00.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T03:11:00.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:11:00.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T03:11:00.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:11:00.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:11:00.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:11:00.796 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:11:00 smithi047 ceph-mon[77147]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T03:11:00.797 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:11:00 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:11:00.797 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:11:00 smithi047 ceph-mon[77147]: pgmap v7048: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:11:00.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:11:00 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:11:00.859 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T03:11:00.859 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:11:00.859 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T03:11:00.859 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:11:00.860 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:11:00.860 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:11:00.860 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T03:11:00.860 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:11:00.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T03:11:00.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:11:00.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T03:11:00.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:11:00.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T03:11:00.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:11:00.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T03:11:00.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:11:00.863 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:11:00.863 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:11:00.863 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:11:00 smithi115 ceph-mon[78593]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T03:11:00.863 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:11:00 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:11:00.864 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:11:00 smithi115 ceph-mon[78593]: pgmap v7048: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:11:01.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:11:00 smithi115 conmon[90393]: 2022-05-14T03:11:00.965+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:11:01.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:11:01 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:11:01.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:11:01 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 13395 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:11:01.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:11:01 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:11:01.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:11:01 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 13395 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:11:02.292 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:11:02 smithi047 conmon[78521]: 2022-05-14T03:11:02.000+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:11:02.293 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:11:02 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T03:11:02.293 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:11:02 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:11:02.293 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:11:02 smithi047 conmon[78521]: yield 2022-05-14T03:11:02.293 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:11:02 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:11:02.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:11:02 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:11:02.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:11:02 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:11:02.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:11:02 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T03:11:02.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:11:02 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:11:02.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:11:02 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T03:11:02.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:11:02 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:11:02.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:11:02 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T03:11:02.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:11:02 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:11:02.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:11:02 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T03:11:02.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:11:02 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:11:02.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:11:02 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T03:11:02.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:11:02 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:11:02.297 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:11:02 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:11:02.297 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:11:02 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:11:02.297 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:11:02 smithi047 conmon[78521]: 2022-05-14T03:11:02.001+0000 7f5341f7c700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T03:11:02.297 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:11:02 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T03:11:02.297 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:11:02 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:11:02.298 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:11:02 smithi047 conmon[78521]: yield 2022-05-14T03:11:02.298 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:11:02 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:11:02.298 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:11:02 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:11:02.298 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:11:02 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:11:02.299 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:11:02 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T03:11:02.299 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:11:02 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:11:02.299 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:11:02 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T03:11:02.300 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:11:02 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:11:02.300 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:11:02 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T03:11:02.300 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:11:02 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:11:02.300 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:11:02 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T03:11:02.300 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:11:02 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:11:02.301 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:11:02 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T03:11:02.301 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:11:02 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:11:02.301 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:11:02 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:11:02.301 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:11:02 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:11:02.302 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:11:02 smithi047 conmon[78521]: 2022-05-14T03:11:02.302 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:11:02 smithi047 conmon[78521]: During handling of the above exception, another exception occurred: 2022-05-14T03:11:02.302 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:11:02 smithi047 conmon[78521]: 2022-05-14T03:11:02.302 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:11:02 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T03:11:02.303 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:11:02 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T03:11:02.303 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:11:02 smithi047 conmon[78521]: return f(*arg) 2022-05-14T03:11:02.303 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:11:02 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T03:11:02.303 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:11:02 smithi047 conmon[78521]: self._write_client_files(client_files, host) 2022-05-14T03:11:02.304 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:11:02 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T03:11:02.304 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:11:02 smithi047 conmon[78521]: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T03:11:02.304 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:11:02 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T03:11:02.304 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:11:02 smithi047 conmon[78521]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T03:11:02.304 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:11:02 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T03:11:02.305 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:11:02 smithi047 conmon[78521]: return self.event_loop.get_result(coro) 2022-05-14T03:11:02.305 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:11:02 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T03:11:02.305 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:11:02 smithi047 conmon[78521]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T03:11:02.305 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:11:02 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T03:11:02.305 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:11:02 smithi047 conmon[78521]: return self.__get_result() 2022-05-14T03:11:02.306 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:11:02 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T03:11:02.306 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:11:02 smithi047 conmon[78521]: raise self._exception 2022-05-14T03:11:02.306 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:11:02 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T03:11:02.306 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:11:02 smithi047 conmon[78521]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T03:11:02.307 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:11:02 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T03:11:02.307 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:11:02 smithi047 conmon[78521]: conn = await self._remote_connection(host, addr) 2022-05-14T03:11:02.308 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:11:02 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T03:11:02.308 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:11:02 smithi047 conmon[78521]: raise 2022-05-14T03:11:02.308 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:11:02 smithi047 conmon[78521]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T03:11:02.309 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:11:02 smithi047 conmon[78521]: self.gen.throw(type, value, traceback) 2022-05-14T03:11:02.309 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:11:02 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T03:11:02.309 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:11:02 smithi047 conmon[78521]: raise OrchestratorError(msg) 2022-05-14T03:11:02.309 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:11:02 smithi047 conmon[78521]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:11:02.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:11:01 smithi115 conmon[90393]: 2022-05-14T03:11:01.954+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:11:02.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:11:02 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:11:02.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:11:02 smithi047 ceph-mon[77147]: pgmap v7049: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:11:02.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:11:02 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:11:02.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:11:02 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:11:02.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:11:02 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:11:02.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:11:02 smithi115 ceph-mon[78593]: pgmap v7049: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:11:02.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:11:02 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:11:02.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:11:02 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:11:03.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:11:02 smithi115 conmon[90393]: 2022-05-14T03:11:02.977+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:11:03.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:11:03 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:11:03.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:11:03 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:11:03.792 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T03:11:03.792 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:11:03.793 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T03:11:03.793 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:11:03.793 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:11:03.793 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:11:03.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T03:11:03.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:11:03.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T03:11:03.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:11:03.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T03:11:03.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:11:03.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T03:11:03.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:11:03.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T03:11:03.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:11:03.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:11:03.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:11:03.796 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:11:03 smithi047 ceph-mon[77147]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T03:11:03.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T03:11:03.797 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:11:03.797 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T03:11:03.797 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:11:03.797 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:11:03.798 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:11:03.798 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T03:11:03.798 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:11:03.798 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T03:11:03.799 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:11:03.799 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T03:11:03.799 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:11:03.799 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T03:11:03.799 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:11:03.800 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T03:11:03.800 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:11:03.801 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:11:03.801 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:11:03.801 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T03:11:03.802 INFO:journalctl@ceph.mon.b.smithi047.stdout: During handling of the above exception, another exception occurred: 2022-05-14T03:11:03.802 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T03:11:03.802 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T03:11:03.803 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T03:11:03.803 INFO:journalctl@ceph.mon.b.smithi047.stdout: return f(*arg) 2022-05-14T03:11:03.803 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T03:11:03.804 INFO:journalctl@ceph.mon.b.smithi047.stdout: self._write_client_files(client_files, host) 2022-05-14T03:11:03.804 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T03:11:03.804 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T03:11:03.804 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T03:11:03.805 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T03:11:03.805 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T03:11:03.805 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.event_loop.get_result(coro) 2022-05-14T03:11:03.805 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T03:11:03.806 INFO:journalctl@ceph.mon.b.smithi047.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T03:11:03.806 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T03:11:03.806 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.__get_result() 2022-05-14T03:11:03.806 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T03:11:03.806 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise self._exception 2022-05-14T03:11:03.807 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T03:11:03.807 INFO:journalctl@ceph.mon.b.smithi047.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T03:11:03.807 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T03:11:03.807 INFO:journalctl@ceph.mon.b.smithi047.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T03:11:03.808 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T03:11:03.808 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise 2022-05-14T03:11:03.808 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T03:11:03.808 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.gen.throw(type, value, traceback) 2022-05-14T03:11:03.808 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T03:11:03.809 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OrchestratorError(msg) 2022-05-14T03:11:03.809 INFO:journalctl@ceph.mon.b.smithi047.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:11:03.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:11:03 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:11:03.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:11:03 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:11:03.859 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T03:11:03.859 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:11:03.860 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T03:11:03.860 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:11:03.860 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:11:03.860 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:11:03.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T03:11:03.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:11:03.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T03:11:03.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:11:03.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T03:11:03.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:11:03.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T03:11:03.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:11:03.863 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T03:11:03.863 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:11:03.863 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:11:03.863 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:11:03.863 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:11:03 smithi115 ceph-mon[78593]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T03:11:03.864 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T03:11:03.864 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:11:03.864 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T03:11:03.864 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:11:03.865 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:11:03.865 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:11:03.865 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T03:11:03.865 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:11:03.866 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T03:11:03.866 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:11:03.866 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T03:11:03.866 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:11:03.866 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T03:11:03.867 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:11:03.867 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T03:11:03.867 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:11:03.867 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:11:03.868 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:11:03.868 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T03:11:03.868 INFO:journalctl@ceph.mon.c.smithi115.stdout: During handling of the above exception, another exception occurred: 2022-05-14T03:11:03.868 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T03:11:03.869 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T03:11:03.869 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T03:11:03.869 INFO:journalctl@ceph.mon.c.smithi115.stdout: return f(*arg) 2022-05-14T03:11:03.869 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T03:11:03.869 INFO:journalctl@ceph.mon.c.smithi115.stdout: self._write_client_files(client_files, host) 2022-05-14T03:11:03.870 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T03:11:03.870 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T03:11:03.870 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T03:11:03.870 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T03:11:03.870 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T03:11:03.871 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.event_loop.get_result(coro) 2022-05-14T03:11:03.871 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T03:11:03.871 INFO:journalctl@ceph.mon.c.smithi115.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T03:11:03.871 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T03:11:03.872 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.__get_result() 2022-05-14T03:11:03.872 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T03:11:03.872 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise self._exception 2022-05-14T03:11:03.872 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T03:11:03.872 INFO:journalctl@ceph.mon.c.smithi115.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T03:11:03.873 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T03:11:03.873 INFO:journalctl@ceph.mon.c.smithi115.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T03:11:03.873 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T03:11:03.873 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise 2022-05-14T03:11:03.874 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T03:11:03.874 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.gen.throw(type, value, traceback) 2022-05-14T03:11:03.874 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T03:11:03.874 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OrchestratorError(msg) 2022-05-14T03:11:03.874 INFO:journalctl@ceph.mon.c.smithi115.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:11:04.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:11:03 smithi115 conmon[90393]: 2022-05-14T03:11:03.949+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:11:04.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:11:04 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:11:04.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:11:04 smithi047 ceph-mon[77147]: pgmap v7050: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:11:04.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:11:04 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:11:04.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:11:04 smithi115 ceph-mon[78593]: pgmap v7050: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:11:05.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:11:04 smithi115 conmon[90393]: 2022-05-14T03:11:04.945+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:11:05.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:11:05 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:11:05.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:11:05 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:11:06.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:11:05 smithi115 conmon[90393]: 2022-05-14T03:11:05.990+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:11:06.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:11:06 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:11:06.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:11:06 smithi047 ceph-mon[77147]: pgmap v7051: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:11:06.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:11:06 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 13400 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:11:06.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:11:06 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:11:06.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:11:06 smithi115 ceph-mon[78593]: pgmap v7051: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:11:06.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:11:06 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 13400 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:11:07.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:11:07 smithi115 conmon[90393]: 2022-05-14T03:11:07.003+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:11:07.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:11:07 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:11:07.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:11:07 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:11:08.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:11:08 smithi115 conmon[90393]: 2022-05-14T03:11:08.049+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:11:08.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:11:08 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:11:08.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:11:08 smithi047 ceph-mon[77147]: pgmap v7052: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:11:08.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:11:08 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:11:08.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:11:08 smithi115 ceph-mon[78593]: pgmap v7052: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:11:09.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:11:09 smithi115 conmon[90393]: 2022-05-14T03:11:09.053+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:11:09.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:11:09 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:11:09.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:11:09 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:11:10.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:11:10 smithi115 conmon[90393]: 2022-05-14T03:11:10.043+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:11:10.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:11:10 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:11:10.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:11:10 smithi047 ceph-mon[77147]: pgmap v7053: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:11:10.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:11:10 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:11:10.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:11:10 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:11:10.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:11:10 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:11:10.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:11:10 smithi115 ceph-mon[78593]: pgmap v7053: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:11:10.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:11:10 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:11:10.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:11:10 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:11:11.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:11:11 smithi115 conmon[90393]: 2022-05-14T03:11:11.070+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:11:11.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:11:11 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:11:11.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:11:11 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 13405 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:11:11.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:11:11 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:11:11.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:11:11 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 13405 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:11:12.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:11:12 smithi115 conmon[90393]: 2022-05-14T03:11:12.046+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:11:12.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:11:12 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:11:12.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:11:12 smithi047 ceph-mon[77147]: pgmap v7054: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:11:12.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:11:12 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:11:12.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:11:12 smithi115 ceph-mon[78593]: pgmap v7054: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:11:13.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:11:13 smithi115 conmon[90393]: 2022-05-14T03:11:13.062+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:11:13.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:11:13 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:11:13.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:11:13 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:11:14.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:11:14 smithi115 conmon[90393]: 2022-05-14T03:11:14.030+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:11:14.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:11:14 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:11:14.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:11:14 smithi047 ceph-mon[77147]: pgmap v7055: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:11:14.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:11:14 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:11:14.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:11:14 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:11:14.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:11:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T03:11:14.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:11:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T03:11:14.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:11:14 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:11:14.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:11:14 smithi115 ceph-mon[78593]: pgmap v7055: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:11:14.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:11:14 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:11:14.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:11:14 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:11:14.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:11:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T03:11:14.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:11:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T03:11:15.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:11:15 smithi115 conmon[90393]: 2022-05-14T03:11:15.029+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:11:15.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:11:15 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:11:15.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:11:15 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:11:16.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:11:16 smithi115 conmon[90393]: 2022-05-14T03:11:16.045+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:11:16.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:11:16 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:11:16.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:11:16 smithi047 ceph-mon[77147]: pgmap v7056: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:11:16.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:11:16 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 13410 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:11:16.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:11:16 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:11:16.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:11:16 smithi115 ceph-mon[78593]: pgmap v7056: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:11:16.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:11:16 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 13410 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:11:17.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:11:17 smithi115 conmon[90393]: 2022-05-14T03:11:17.068+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:11:17.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:11:17 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:11:17.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:11:17 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:11:18.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:11:18 smithi115 conmon[90393]: 2022-05-14T03:11:18.051+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:11:18.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:11:18 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:11:18.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:11:18 smithi047 ceph-mon[77147]: pgmap v7057: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:11:18.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:11:18 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:11:18.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:11:18 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:11:18.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:11:18 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:11:18.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:11:18 smithi115 ceph-mon[78593]: pgmap v7057: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:11:18.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:11:18 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:11:18.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:11:18 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:11:19.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:11:19 smithi115 conmon[90393]: 2022-05-14T03:11:19.076+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:11:19.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:11:19 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:11:19.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:11:19 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:11:20.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:11:20 smithi115 conmon[90393]: 2022-05-14T03:11:20.030+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:11:20.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:11:20 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:11:20.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:11:20 smithi047 ceph-mon[77147]: pgmap v7058: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:11:20.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:11:20 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:11:20.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:11:20 smithi115 ceph-mon[78593]: pgmap v7058: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:11:21.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:11:21 smithi115 conmon[90393]: 2022-05-14T03:11:21.034+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:11:21.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:11:21 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:11:21.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:11:21 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 13415 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:11:21.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:11:21 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:11:21.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:11:21 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 13415 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:11:22.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:11:22 smithi115 conmon[90393]: 2022-05-14T03:11:22.044+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:11:22.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:11:22 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:11:22.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:11:22 smithi047 ceph-mon[77147]: pgmap v7059: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:11:22.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:11:22 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:11:22.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:11:22 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:11:22.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:11:22 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:11:22.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:11:22 smithi115 ceph-mon[78593]: pgmap v7059: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:11:22.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:11:22 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:11:22.862 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:11:22 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:11:23.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:11:23 smithi115 conmon[90393]: 2022-05-14T03:11:23.053+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:11:23.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:11:23 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:11:23.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:11:23 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:11:24.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:11:24 smithi115 conmon[90393]: 2022-05-14T03:11:24.087+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:11:24.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:11:24 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:11:24.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:11:24 smithi047 ceph-mon[77147]: pgmap v7060: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:11:24.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:11:24 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:11:24.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:11:24 smithi115 ceph-mon[78593]: pgmap v7060: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:11:25.467 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:11:25 smithi115 conmon[90393]: 2022-05-14T03:11:25.106+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:11:25.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:11:25 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:11:25.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:11:25 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:11:26.471 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:11:26 smithi115 conmon[90393]: 2022-05-14T03:11:26.122+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:11:26.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:11:26 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:11:26.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:11:26 smithi047 ceph-mon[77147]: pgmap v7061: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:11:26.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:11:26 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 13419 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:11:26.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:11:26 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:11:26.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:11:26 smithi115 ceph-mon[78593]: pgmap v7061: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:11:26.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:11:26 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 13419 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:11:27.476 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:11:27 smithi115 conmon[90393]: 2022-05-14T03:11:27.106+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:11:27.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:11:27 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:11:27.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:11:27 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:11:28.477 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:11:28 smithi115 conmon[90393]: 2022-05-14T03:11:28.139+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:11:28.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:11:28 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:11:28.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:11:28 smithi047 ceph-mon[77147]: pgmap v7062: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:11:28.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:11:28 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:11:28.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:11:28 smithi115 ceph-mon[78593]: pgmap v7062: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:11:29.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:11:29 smithi115 conmon[90393]: 2022-05-14T03:11:29.101+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:11:29.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:11:29 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:11:29.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:11:29 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:11:30.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:11:30 smithi115 conmon[90393]: 2022-05-14T03:11:30.072+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:11:30.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:11:30 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:11:30.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:11:30 smithi047 ceph-mon[77147]: pgmap v7063: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:11:30.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:11:30 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:11:30.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:11:30 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:11:30.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:11:30 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:11:30.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:11:30 smithi115 ceph-mon[78593]: pgmap v7063: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:11:30.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:11:30 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:11:30.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:11:30 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:11:31.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:11:31 smithi115 conmon[90393]: 2022-05-14T03:11:31.036+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:11:31.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:11:31 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:11:31.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:11:31 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 13425 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:11:31.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:11:31 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:11:31.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:11:31 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 13425 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:11:32.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:11:32 smithi115 conmon[90393]: 2022-05-14T03:11:32.071+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:11:32.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:11:32 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:11:32.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:11:32 smithi047 ceph-mon[77147]: pgmap v7064: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:11:32.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:11:32 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:11:32.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:11:32 smithi115 ceph-mon[78593]: pgmap v7064: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:11:33.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:11:33 smithi115 conmon[90393]: 2022-05-14T03:11:33.090+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:11:33.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:11:33 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:11:33.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:11:33 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:11:34.493 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:11:34 smithi115 conmon[90393]: 2022-05-14T03:11:34.123+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:11:34.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:11:34 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:11:34.826 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:11:34 smithi047 ceph-mon[77147]: pgmap v7065: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:11:34.826 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:11:34 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:11:34.826 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:11:34 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:11:34.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:11:34 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:11:34.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:11:34 smithi115 ceph-mon[78593]: pgmap v7065: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:11:34.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:11:34 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:11:34.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:11:34 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:11:35.498 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:11:35 smithi115 conmon[90393]: 2022-05-14T03:11:35.163+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:11:35.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:11:35 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:11:35.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:11:35 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:11:36.499 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:11:36 smithi115 conmon[90393]: 2022-05-14T03:11:36.189+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:11:36.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:11:36 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:11:36.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:11:36 smithi047 ceph-mon[77147]: pgmap v7066: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:11:36.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:11:36 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 13430 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:11:36.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:11:36 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:11:36.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:11:36 smithi115 ceph-mon[78593]: pgmap v7066: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:11:36.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:11:36 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 13430 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:11:37.505 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:11:37 smithi115 conmon[90393]: 2022-05-14T03:11:37.234+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:11:37.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:11:37 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:11:37.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:11:37 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:11:38.506 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:11:38 smithi115 conmon[90393]: 2022-05-14T03:11:38.225+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:11:38.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:11:38 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:11:38.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:11:38 smithi047 ceph-mon[77147]: pgmap v7067: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:11:38.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:11:38 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:11:38.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:11:38 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:11:38.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:11:38 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:11:38.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:11:38 smithi115 ceph-mon[78593]: pgmap v7067: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:11:38.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:11:38 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:11:38.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:11:38 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:11:39.513 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:11:39 smithi115 conmon[90393]: 2022-05-14T03:11:39.212+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:11:39.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:11:39 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:11:39.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:11:39 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:11:40.513 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:11:40 smithi115 conmon[90393]: 2022-05-14T03:11:40.172+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:11:40.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:11:40 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:11:40.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:11:40 smithi047 ceph-mon[77147]: pgmap v7068: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:11:40.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:11:40 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:11:40.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:11:40 smithi115 ceph-mon[78593]: pgmap v7068: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:11:41.514 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:11:41 smithi115 conmon[90393]: 2022-05-14T03:11:41.218+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:11:41.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:11:41 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:11:41.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:11:41 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 13435 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:11:41.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:11:41 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:11:41.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:11:41 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 13435 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:11:42.519 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:11:42 smithi115 conmon[90393]: 2022-05-14T03:11:42.239+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:11:42.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:11:42 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:11:42.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:11:42 smithi047 ceph-mon[77147]: pgmap v7069: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:11:42.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:11:42 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:11:42.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:11:42 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:11:42.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:11:42 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:11:42.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:11:42 smithi115 ceph-mon[78593]: pgmap v7069: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:11:42.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:11:42 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:11:42.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:11:42 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:11:43.524 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:11:43 smithi115 conmon[90393]: 2022-05-14T03:11:43.202+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:11:43.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:11:43 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:11:43.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:11:43 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:11:44.525 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:11:44 smithi115 conmon[90393]: 2022-05-14T03:11:44.198+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:11:44.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:11:44 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:11:44.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:11:44 smithi047 ceph-mon[77147]: pgmap v7070: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:11:44.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:11:44 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:11:44.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:11:44 smithi115 ceph-mon[78593]: pgmap v7070: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:11:45.530 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:11:45 smithi115 conmon[90393]: 2022-05-14T03:11:45.206+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:11:45.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:11:45 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:11:45.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:11:45 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:11:46.532 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:11:46 smithi115 conmon[90393]: 2022-05-14T03:11:46.166+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:11:46.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:11:46 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:11:46.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:11:46 smithi047 ceph-mon[77147]: pgmap v7071: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:11:46.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:11:46 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 13440 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:11:46.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:11:46 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:11:46.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:11:46 smithi115 ceph-mon[78593]: pgmap v7071: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:11:46.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:11:46 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 13440 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:11:47.537 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:11:47 smithi115 conmon[90393]: 2022-05-14T03:11:47.137+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:11:47.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:11:47 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:11:47.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:11:47 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:11:48.539 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:11:48 smithi115 conmon[90393]: 2022-05-14T03:11:48.127+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:11:48.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:11:48 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:11:48.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:11:48 smithi047 ceph-mon[77147]: pgmap v7072: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:11:48.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:11:48 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:11:48.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:11:48 smithi115 ceph-mon[78593]: pgmap v7072: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:11:49.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:11:49 smithi115 conmon[90393]: 2022-05-14T03:11:49.089+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:11:49.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:11:49 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:11:50.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:11:49 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:11:50.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:11:50 smithi115 conmon[90393]: 2022-05-14T03:11:50.089+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:11:50.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:11:50 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:11:50.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:11:50 smithi115 ceph-mon[78593]: pgmap v7073: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:11:50.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:11:50 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:11:50.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:11:50 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:11:51.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:11:50 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:11:51.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:11:50 smithi047 ceph-mon[77147]: pgmap v7073: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:11:51.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:11:50 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:11:51.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:11:50 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:11:51.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:11:51 smithi115 conmon[90393]: 2022-05-14T03:11:51.057+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:11:51.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:11:51 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:11:51.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:11:51 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 13445 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:11:52.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:11:51 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:11:52.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:11:51 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 13445 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:11:52.548 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:11:52 smithi115 conmon[90393]: 2022-05-14T03:11:52.107+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:11:52.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:11:52 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:11:52.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:11:52 smithi115 ceph-mon[78593]: pgmap v7074: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:11:53.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:11:52 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:11:53.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:11:52 smithi047 ceph-mon[77147]: pgmap v7074: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:11:53.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:11:53 smithi115 conmon[90393]: 2022-05-14T03:11:53.088+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:11:53.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:11:53 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:11:54.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:11:53 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:11:54.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:11:54 smithi115 conmon[90393]: 2022-05-14T03:11:54.059+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:11:54.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:11:54 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:11:54.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:11:54 smithi115 ceph-mon[78593]: pgmap v7075: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:11:54.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:11:54 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:11:54.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:11:54 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:11:55.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:11:54 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:11:55.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:11:54 smithi047 ceph-mon[77147]: pgmap v7075: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:11:55.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:11:54 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:11:55.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:11:54 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:11:55.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:11:55 smithi115 conmon[90393]: 2022-05-14T03:11:55.029+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:11:55.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:11:55 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:11:56.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:11:55 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:11:56.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:11:56 smithi115 conmon[90393]: 2022-05-14T03:11:56.006+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:11:56.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:11:56 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:11:56.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:11:56 smithi115 ceph-mon[78593]: pgmap v7076: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:11:56.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:11:56 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 13450 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:11:57.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:11:56 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:11:57.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:11:56 smithi047 ceph-mon[77147]: pgmap v7076: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:11:57.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:11:56 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 13450 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:11:57.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:11:56 smithi115 conmon[90393]: 2022-05-14T03:11:56.998+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:11:57.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:11:57 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:11:58.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:11:57 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:11:58.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:11:58 smithi115 conmon[90393]: 2022-05-14T03:11:58.030+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:11:58.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:11:58 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:11:58.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:11:58 smithi115 ceph-mon[78593]: pgmap v7077: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:11:58.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:11:58 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:11:58.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:11:58 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:11:59.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:11:58 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:11:59.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:11:58 smithi047 ceph-mon[77147]: pgmap v7077: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:11:59.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:11:58 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:11:59.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:11:58 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:11:59.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:11:59 smithi115 conmon[90393]: 2022-05-14T03:11:59.020+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:11:59.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:11:59 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:12:00.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:11:59 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:12:00.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:12:00 smithi115 conmon[90393]: 2022-05-14T03:12:00.041+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:12:00.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:12:00 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:12:00.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:12:00 smithi115 ceph-mon[78593]: pgmap v7078: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:12:01.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:12:00 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:12:01.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:12:00 smithi047 ceph-mon[77147]: pgmap v7078: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:12:01.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:12:00 smithi115 conmon[90393]: 2022-05-14T03:12:00.998+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:12:01.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:12:01 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:12:01.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:12:01 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 13455 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:12:02.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:12:01 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:12:02.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:12:01 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 13455 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:12:02.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:12:01 smithi115 conmon[90393]: 2022-05-14T03:12:01.996+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:12:02.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:12:02 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:12:02.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:12:02 smithi047 ceph-mon[77147]: pgmap v7079: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:12:02.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:12:02 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:12:02.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:12:02 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:12:02.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:12:02 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T03:12:02.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:12:02 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T03:12:02.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:12:02 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T03:12:02.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:12:02 smithi047 conmon[78521]: 2022-05-14T03:12:02.607+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:12:02.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:12:02 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T03:12:02.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:12:02 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:12:02.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:12:02 smithi047 conmon[78521]: yield 2022-05-14T03:12:02.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:12:02 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:12:02.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:12:02 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:12:02.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:12:02 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:12:02.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:12:02 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T03:12:02.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:12:02 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:12:02.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:12:02 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T03:12:02.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:12:02 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:12:02.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:12:02 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T03:12:02.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:12:02 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:12:02.797 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:12:02 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T03:12:02.797 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:12:02 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:12:02.797 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:12:02 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T03:12:02.797 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:12:02 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:12:02.798 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:12:02 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:12:02.798 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:12:02 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:12:02.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:12:02 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:12:02.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:12:02 smithi115 ceph-mon[78593]: pgmap v7079: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:12:02.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:12:02 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:12:02.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:12:02 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:12:02.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:12:02 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T03:12:02.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:12:02 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T03:12:02.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:12:02 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T03:12:03.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:12:02 smithi115 conmon[90393]: 2022-05-14T03:12:02.956+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:12:03.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:12:03 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:12:03.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:12:03 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:12:03.859 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T03:12:03.859 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:12:03.859 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T03:12:03.859 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:12:03.860 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:12:03.860 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:12:03.860 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T03:12:03.860 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:12:03.860 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T03:12:03.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:12:03.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T03:12:03.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:12:03.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T03:12:03.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:12:03.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T03:12:03.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:12:03.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:12:03.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:12:03.863 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:12:03 smithi115 ceph-mon[78593]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T03:12:04.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:12:03 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:12:04.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:12:03 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:12:04.042 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T03:12:04.042 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:12:04.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T03:12:04.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:12:04.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:12:04.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:12:04.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T03:12:04.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:12:04.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T03:12:04.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:12:04.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T03:12:04.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:12:04.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T03:12:04.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:12:04.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T03:12:04.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:12:04.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:12:04.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:12:04.046 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:12:03 smithi047 ceph-mon[77147]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T03:12:04.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:12:03 smithi115 conmon[90393]: 2022-05-14T03:12:03.952+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:12:04.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:12:04 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:12:04.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:12:04 smithi115 ceph-mon[78593]: pgmap v7080: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:12:05.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:12:04 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:12:05.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:12:04 smithi047 ceph-mon[77147]: pgmap v7080: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:12:05.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:12:04 smithi115 conmon[90393]: 2022-05-14T03:12:04.976+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:12:05.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:12:05 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:12:06.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:12:05 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:12:06.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:12:05 smithi047 conmon[78521]: 2022-05-14T03:12:05.679+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:12:06.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:12:05 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T03:12:06.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:12:05 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:12:06.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:12:05 smithi047 conmon[78521]: yield 2022-05-14T03:12:06.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:12:05 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:12:06.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:12:05 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:12:06.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:12:05 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:12:06.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:12:05 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T03:12:06.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:12:05 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:12:06.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:12:05 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T03:12:06.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:12:05 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:12:06.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:12:05 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T03:12:06.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:12:05 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:12:06.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:12:05 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T03:12:06.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:12:05 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:12:06.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:12:05 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T03:12:06.048 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:12:05 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:12:06.048 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:12:05 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:12:06.048 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:12:05 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:12:06.048 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:12:05 smithi047 conmon[78521]: 2022-05-14T03:12:05.680+0000 7f5340779700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T03:12:06.049 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:12:05 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T03:12:06.049 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:12:05 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:12:06.049 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:12:05 smithi047 conmon[78521]: yield 2022-05-14T03:12:06.050 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:12:05 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:12:06.050 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:12:05 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:12:06.050 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:12:05 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:12:06.050 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:12:05 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T03:12:06.051 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:12:05 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:12:06.051 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:12:05 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T03:12:06.051 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:12:05 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:12:06.052 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:12:05 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T03:12:06.052 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:12:05 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:12:06.052 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:12:05 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T03:12:06.052 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:12:05 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:12:06.053 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:12:05 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T03:12:06.053 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:12:05 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:12:06.053 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:12:05 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:12:06.054 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:12:05 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:12:06.054 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:12:05 smithi047 conmon[78521]: 2022-05-14T03:12:06.054 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:12:05 smithi047 conmon[78521]: During handling of the above exception, another exception occurred: 2022-05-14T03:12:06.055 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:12:05 smithi047 conmon[78521]: 2022-05-14T03:12:06.055 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:12:05 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T03:12:06.055 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:12:05 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T03:12:06.056 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:12:05 smithi047 conmon[78521]: return f(*arg) 2022-05-14T03:12:06.056 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:12:05 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T03:12:06.056 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:12:05 smithi047 conmon[78521]: self._write_client_files(client_files, host) 2022-05-14T03:12:06.057 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:12:05 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T03:12:06.057 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:12:05 smithi047 conmon[78521]: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T03:12:06.057 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:12:05 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T03:12:06.057 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:12:05 smithi047 conmon[78521]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T03:12:06.058 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:12:05 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T03:12:06.058 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:12:05 smithi047 conmon[78521]: return self.event_loop.get_result(coro) 2022-05-14T03:12:06.058 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:12:05 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T03:12:06.059 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:12:05 smithi047 conmon[78521]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T03:12:06.059 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:12:05 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T03:12:06.059 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:12:05 smithi047 conmon[78521]: return self.__get_result() 2022-05-14T03:12:06.060 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:12:05 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T03:12:06.060 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:12:05 smithi047 conmon[78521]: raise self._exception 2022-05-14T03:12:06.060 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:12:05 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T03:12:06.060 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:12:05 smithi047 conmon[78521]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T03:12:06.060 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:12:05 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T03:12:06.061 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:12:05 smithi047 conmon[78521]: conn = await self._remote_connection(host, addr) 2022-05-14T03:12:06.061 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:12:05 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T03:12:06.061 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:12:05 smithi047 conmon[78521]: raise 2022-05-14T03:12:06.061 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:12:05 smithi047 conmon[78521]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T03:12:06.061 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:12:05 smithi047 conmon[78521]: self.gen.throw(type, value, traceback) 2022-05-14T03:12:06.062 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:12:05 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T03:12:06.062 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:12:05 smithi047 conmon[78521]: raise OrchestratorError(msg) 2022-05-14T03:12:06.062 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:12:05 smithi047 conmon[78521]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:12:06.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:12:05 smithi115 conmon[90393]: 2022-05-14T03:12:05.965+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:12:06.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:12:06 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:12:06.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:12:06 smithi115 ceph-mon[78593]: pgmap v7081: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:12:06.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:12:06 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:12:06.859 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T03:12:06.859 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:12:06.860 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T03:12:06.860 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:12:06.860 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:12:06.860 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:12:06.860 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T03:12:06.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:12:06.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T03:12:06.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:12:06.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T03:12:06.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:12:06.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T03:12:06.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:12:06.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T03:12:06.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:12:06.863 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:12:06.863 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:12:06.863 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:12:06 smithi115 ceph-mon[78593]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T03:12:06.863 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T03:12:06.863 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:12:06.864 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T03:12:06.864 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:12:06.864 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:12:06.864 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:12:06.865 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T03:12:06.865 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:12:06.865 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T03:12:06.865 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:12:06.865 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T03:12:06.866 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:12:06.866 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T03:12:06.866 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:12:06.866 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T03:12:06.867 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:12:06.867 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:12:06.867 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:12:06.867 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T03:12:06.867 INFO:journalctl@ceph.mon.c.smithi115.stdout: During handling of the above exception, another exception occurred: 2022-05-14T03:12:06.868 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T03:12:06.868 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T03:12:06.868 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T03:12:06.868 INFO:journalctl@ceph.mon.c.smithi115.stdout: return f(*arg) 2022-05-14T03:12:06.869 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T03:12:06.869 INFO:journalctl@ceph.mon.c.smithi115.stdout: self._write_client_files(client_files, host) 2022-05-14T03:12:06.869 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T03:12:06.869 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T03:12:06.869 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T03:12:06.870 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T03:12:06.870 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T03:12:06.870 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.event_loop.get_result(coro) 2022-05-14T03:12:06.870 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T03:12:06.870 INFO:journalctl@ceph.mon.c.smithi115.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T03:12:06.871 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T03:12:06.871 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.__get_result() 2022-05-14T03:12:06.871 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T03:12:06.871 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise self._exception 2022-05-14T03:12:06.872 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T03:12:06.872 INFO:journalctl@ceph.mon.c.smithi115.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T03:12:06.872 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T03:12:06.872 INFO:journalctl@ceph.mon.c.smithi115.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T03:12:06.872 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T03:12:06.873 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise 2022-05-14T03:12:06.873 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T03:12:06.873 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.gen.throw(type, value, traceback) 2022-05-14T03:12:06.873 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T03:12:06.874 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OrchestratorError(msg) 2022-05-14T03:12:06.874 INFO:journalctl@ceph.mon.c.smithi115.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:12:06.874 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:12:06 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 13460 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:12:07.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:12:06 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:12:07.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:12:06 smithi047 ceph-mon[77147]: pgmap v7081: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:12:07.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:12:06 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:12:07.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T03:12:07.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:12:07.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T03:12:07.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:12:07.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:12:07.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:12:07.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T03:12:07.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:12:07.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T03:12:07.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:12:07.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T03:12:07.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:12:07.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T03:12:07.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:12:07.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T03:12:07.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:12:07.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:12:07.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:12:07.046 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:12:06 smithi047 ceph-mon[77147]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T03:12:07.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T03:12:07.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:12:07.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T03:12:07.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:12:07.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:12:07.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:12:07.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T03:12:07.049 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:12:07.049 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T03:12:07.049 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:12:07.049 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T03:12:07.049 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:12:07.050 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T03:12:07.050 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:12:07.050 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T03:12:07.050 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:12:07.050 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:12:07.051 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:12:07.051 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T03:12:07.051 INFO:journalctl@ceph.mon.b.smithi047.stdout: During handling of the above exception, another exception occurred: 2022-05-14T03:12:07.051 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T03:12:07.051 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T03:12:07.052 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T03:12:07.052 INFO:journalctl@ceph.mon.b.smithi047.stdout: return f(*arg) 2022-05-14T03:12:07.052 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T03:12:07.052 INFO:journalctl@ceph.mon.b.smithi047.stdout: self._write_client_files(client_files, host) 2022-05-14T03:12:07.053 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T03:12:07.053 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T03:12:07.053 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T03:12:07.053 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T03:12:07.053 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T03:12:07.054 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.event_loop.get_result(coro) 2022-05-14T03:12:07.054 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T03:12:07.054 INFO:journalctl@ceph.mon.b.smithi047.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T03:12:07.054 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T03:12:07.054 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.__get_result() 2022-05-14T03:12:07.055 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T03:12:07.055 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise self._exception 2022-05-14T03:12:07.055 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T03:12:07.055 INFO:journalctl@ceph.mon.b.smithi047.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T03:12:07.055 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T03:12:07.056 INFO:journalctl@ceph.mon.b.smithi047.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T03:12:07.056 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T03:12:07.056 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise 2022-05-14T03:12:07.056 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T03:12:07.056 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.gen.throw(type, value, traceback) 2022-05-14T03:12:07.057 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T03:12:07.057 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OrchestratorError(msg) 2022-05-14T03:12:07.057 INFO:journalctl@ceph.mon.b.smithi047.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:12:07.057 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:12:06 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 13460 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:12:07.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:12:06 smithi115 conmon[90393]: 2022-05-14T03:12:06.936+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:12:07.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:12:07 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:12:08.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:12:07 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:12:08.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:12:07 smithi115 conmon[90393]: 2022-05-14T03:12:07.938+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:12:08.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:12:08 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:12:08.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:12:08 smithi115 ceph-mon[78593]: pgmap v7082: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:12:08.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:12:08 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:12:09.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:12:08 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:12:09.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:12:08 smithi047 ceph-mon[77147]: pgmap v7082: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:12:09.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:12:08 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:12:09.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:12:08 smithi115 conmon[90393]: 2022-05-14T03:12:08.932+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:12:09.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:12:09 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:12:10.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:12:09 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:12:10.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:12:09 smithi115 conmon[90393]: 2022-05-14T03:12:09.899+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:12:10.863 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:12:10 smithi115 ceph-mon[78593]: pgmap v7083: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:12:10.864 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:12:10 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:12:10.864 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:12:10 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:12:10.864 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:12:10 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:12:11.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:12:10 smithi047 ceph-mon[77147]: pgmap v7083: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:12:11.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:12:10 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:12:11.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:12:10 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:12:11.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:12:10 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:12:11.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:12:10 smithi115 conmon[90393]: 2022-05-14T03:12:10.861+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:12:11.863 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:12:11 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:12:11.863 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:12:11 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 13465 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:12:12.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:12:11 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:12:12.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:12:11 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 13465 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:12:12.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:12:11 smithi115 conmon[90393]: 2022-05-14T03:12:11.861+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:12:12.864 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:12:12 smithi115 ceph-mon[78593]: pgmap v7084: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:12:12.864 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:12:12 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:12:13.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:12:12 smithi047 ceph-mon[77147]: pgmap v7084: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:12:13.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:12:12 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:12:13.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:12:12 smithi115 conmon[90393]: 2022-05-14T03:12:12.862+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:12:14.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:12:13 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:12:14.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:12:13 smithi115 conmon[90393]: 2022-05-14T03:12:13.847+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:12:14.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:12:13 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:12:14.899 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:12:14 smithi115 ceph-mon[78593]: pgmap v7085: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:12:14.899 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:12:14 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:12:14.900 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:12:14 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:12:14.900 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:12:14 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:12:14.900 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:12:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T03:12:14.900 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:12:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T03:12:15.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:12:14 smithi047 ceph-mon[77147]: pgmap v7085: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:12:15.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:12:14 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:12:15.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:12:14 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:12:15.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:12:14 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:12:15.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:12:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T03:12:15.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:12:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T03:12:15.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:12:14 smithi115 conmon[90393]: 2022-05-14T03:12:14.896+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:12:15.945 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:12:15 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:12:16.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:12:15 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:12:16.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:12:15 smithi115 conmon[90393]: 2022-05-14T03:12:15.943+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:12:16.944 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:12:16 smithi115 ceph-mon[78593]: pgmap v7086: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 6 op/s 2022-05-14T03:12:16.945 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:12:16 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:12:16.945 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:12:16 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 13470 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:12:17.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:12:16 smithi047 ceph-mon[77147]: pgmap v7086: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 6 op/s 2022-05-14T03:12:17.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:12:16 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:12:17.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:12:16 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 13470 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:12:17.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:12:16 smithi115 conmon[90393]: 2022-05-14T03:12:16.943+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:12:17.962 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:12:17 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:12:18.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:12:17 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:12:18.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:12:17 smithi115 conmon[90393]: 2022-05-14T03:12:17.960+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:12:18.965 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:12:18 smithi115 ceph-mon[78593]: pgmap v7087: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 4 op/s 2022-05-14T03:12:18.965 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:12:18 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:12:18.965 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:12:18 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:12:18.965 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:12:18 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:12:19.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:12:18 smithi047 ceph-mon[77147]: pgmap v7087: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 4 op/s 2022-05-14T03:12:19.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:12:18 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:12:19.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:12:18 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:12:19.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:12:18 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:12:19.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:12:18 smithi115 conmon[90393]: 2022-05-14T03:12:18.963+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:12:19.967 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:12:19 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:12:20.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:12:19 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:12:20.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:12:19 smithi115 conmon[90393]: 2022-05-14T03:12:19.966+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:12:20.987 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:12:20 smithi115 ceph-mon[78593]: pgmap v7088: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 6 op/s 2022-05-14T03:12:20.987 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:12:20 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:12:21.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:12:20 smithi047 ceph-mon[77147]: pgmap v7088: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 6 op/s 2022-05-14T03:12:21.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:12:20 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:12:21.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:12:20 smithi115 conmon[90393]: 2022-05-14T03:12:20.986+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:12:22.016 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:12:21 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:12:22.017 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:12:21 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 13475 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:12:22.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:12:21 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:12:22.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:12:21 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 13475 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:12:22.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:12:22 smithi115 conmon[90393]: 2022-05-14T03:12:22.015+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:12:23.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:12:22 smithi047 ceph-mon[77147]: pgmap v7089: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 4 op/s 2022-05-14T03:12:23.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:12:22 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:12:23.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:12:22 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:12:23.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:12:22 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:12:23.063 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:12:22 smithi115 ceph-mon[78593]: pgmap v7089: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 4 op/s 2022-05-14T03:12:23.063 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:12:22 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:12:23.064 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:12:22 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:12:23.064 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:12:22 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:12:23.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:12:23 smithi115 conmon[90393]: 2022-05-14T03:12:23.062+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:12:24.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:12:23 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:12:24.085 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:12:23 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:12:24.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:12:24 smithi115 conmon[90393]: 2022-05-14T03:12:24.083+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:12:25.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:12:24 smithi047 ceph-mon[77147]: pgmap v7090: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 4 op/s 2022-05-14T03:12:25.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:12:24 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:12:25.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:12:24 smithi115 ceph-mon[78593]: pgmap v7090: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 4 op/s 2022-05-14T03:12:25.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:12:24 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:12:25.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:12:25 smithi115 conmon[90393]: 2022-05-14T03:12:25.127+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:12:26.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:12:25 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:12:26.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:12:25 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:12:26.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:12:26 smithi115 conmon[90393]: 2022-05-14T03:12:26.113+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:12:27.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:12:26 smithi047 ceph-mon[77147]: pgmap v7091: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 7 op/s 2022-05-14T03:12:27.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:12:26 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:12:27.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:12:26 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 13480 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:12:27.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:12:26 smithi115 ceph-mon[78593]: pgmap v7091: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 7 op/s 2022-05-14T03:12:27.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:12:26 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:12:27.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:12:26 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 13480 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:12:27.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:12:27 smithi115 conmon[90393]: 2022-05-14T03:12:27.144+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:12:28.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:12:27 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:12:28.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:12:27 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:12:28.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:12:28 smithi115 conmon[90393]: 2022-05-14T03:12:28.186+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:12:29.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:12:28 smithi047 ceph-mon[77147]: pgmap v7092: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:12:29.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:12:28 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:12:29.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:12:28 smithi115 ceph-mon[78593]: pgmap v7092: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:12:29.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:12:28 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:12:29.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:12:29 smithi115 conmon[90393]: 2022-05-14T03:12:29.191+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:12:30.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:12:29 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:12:30.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:12:29 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:12:30.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:12:30 smithi115 conmon[90393]: 2022-05-14T03:12:30.181+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:12:31.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:12:30 smithi047 ceph-mon[77147]: pgmap v7093: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:12:31.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:12:30 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:12:31.045 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:12:30 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:12:31.045 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:12:30 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:12:31.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:12:30 smithi115 ceph-mon[78593]: pgmap v7093: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:12:31.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:12:30 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:12:31.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:12:30 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:12:31.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:12:30 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:12:31.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:12:31 smithi115 conmon[90393]: 2022-05-14T03:12:31.193+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:12:32.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:12:31 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:12:32.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:12:31 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 13485 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:12:32.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:12:31 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:12:32.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:12:31 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 13485 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:12:32.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:12:32 smithi115 conmon[90393]: 2022-05-14T03:12:32.176+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:12:33.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:12:32 smithi047 ceph-mon[77147]: pgmap v7094: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:12:33.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:12:32 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:12:33.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:12:32 smithi115 ceph-mon[78593]: pgmap v7094: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:12:33.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:12:32 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:12:33.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:12:33 smithi115 conmon[90393]: 2022-05-14T03:12:33.139+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:12:34.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:12:33 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:12:34.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:12:33 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:12:34.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:12:34 smithi115 conmon[90393]: 2022-05-14T03:12:34.155+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:12:35.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:12:34 smithi047 ceph-mon[77147]: pgmap v7095: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:12:35.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:12:34 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:12:35.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:12:34 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:12:35.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:12:34 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:12:35.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:12:34 smithi115 ceph-mon[78593]: pgmap v7095: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:12:35.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:12:34 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:12:35.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:12:34 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:12:35.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:12:34 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:12:35.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:12:35 smithi115 conmon[90393]: 2022-05-14T03:12:35.176+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:12:36.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:12:35 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:12:36.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:12:35 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:12:36.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:12:36 smithi115 conmon[90393]: 2022-05-14T03:12:36.176+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:12:37.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:12:36 smithi047 ceph-mon[77147]: pgmap v7096: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:12:37.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:12:36 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:12:37.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:12:36 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 13490 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:12:37.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:12:36 smithi115 ceph-mon[78593]: pgmap v7096: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:12:37.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:12:36 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:12:37.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:12:36 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 13490 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:12:37.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:12:37 smithi115 conmon[90393]: 2022-05-14T03:12:37.153+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:12:38.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:12:37 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:12:38.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:12:37 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:12:38.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:12:38 smithi115 conmon[90393]: 2022-05-14T03:12:38.172+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:12:39.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:12:38 smithi047 ceph-mon[77147]: pgmap v7097: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:12:39.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:12:38 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:12:39.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:12:38 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:12:39.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:12:38 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:12:39.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:12:38 smithi115 ceph-mon[78593]: pgmap v7097: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:12:39.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:12:38 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:12:39.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:12:38 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:12:39.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:12:38 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:12:39.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:12:39 smithi115 conmon[90393]: 2022-05-14T03:12:39.183+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:12:40.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:12:39 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:12:40.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:12:39 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:12:40.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:12:40 smithi115 conmon[90393]: 2022-05-14T03:12:40.210+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:12:41.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:12:40 smithi047 ceph-mon[77147]: pgmap v7098: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:12:41.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:12:40 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:12:41.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:12:40 smithi115 ceph-mon[78593]: pgmap v7098: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:12:41.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:12:40 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:12:41.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:12:41 smithi115 conmon[90393]: 2022-05-14T03:12:41.164+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:12:42.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:12:41 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:12:42.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:12:41 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 13495 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:12:42.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:12:41 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:12:42.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:12:41 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 13495 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:12:42.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:12:42 smithi115 conmon[90393]: 2022-05-14T03:12:42.119+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:12:43.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:12:42 smithi047 ceph-mon[77147]: pgmap v7099: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:12:43.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:12:42 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:12:43.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:12:42 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:12:43.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:12:42 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:12:43.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:12:42 smithi115 ceph-mon[78593]: pgmap v7099: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:12:43.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:12:42 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:12:43.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:12:42 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:12:43.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:12:42 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:12:43.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:12:43 smithi115 conmon[90393]: 2022-05-14T03:12:43.115+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:12:44.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:12:43 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:12:44.099 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:12:43 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:12:44.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:12:44 smithi115 conmon[90393]: 2022-05-14T03:12:44.097+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:12:45.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:12:44 smithi047 ceph-mon[77147]: pgmap v7100: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:12:45.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:12:44 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:12:45.062 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:12:44 smithi115 ceph-mon[78593]: pgmap v7100: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:12:45.062 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:12:44 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:12:45.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:12:45 smithi115 conmon[90393]: 2022-05-14T03:12:45.060+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:12:46.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:12:45 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:12:46.088 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:12:45 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:12:46.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:12:46 smithi115 conmon[90393]: 2022-05-14T03:12:46.086+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:12:47.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:12:46 smithi047 ceph-mon[77147]: pgmap v7101: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:12:47.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:12:46 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:12:47.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:12:46 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 13500 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:12:47.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:12:46 smithi115 ceph-mon[78593]: pgmap v7101: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:12:47.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:12:46 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:12:47.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:12:46 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 13500 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:12:47.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:12:47 smithi115 conmon[90393]: 2022-05-14T03:12:47.121+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:12:48.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:12:47 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:12:48.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:12:47 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:12:48.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:12:48 smithi115 conmon[90393]: 2022-05-14T03:12:48.125+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:12:49.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:12:48 smithi047 ceph-mon[77147]: pgmap v7102: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:12:49.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:12:48 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:12:49.080 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:12:48 smithi115 ceph-mon[78593]: pgmap v7102: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:12:49.080 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:12:48 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:12:49.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:12:49 smithi115 conmon[90393]: 2022-05-14T03:12:49.077+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:12:50.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:12:49 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:12:50.093 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:12:49 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:12:50.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:12:50 smithi115 conmon[90393]: 2022-05-14T03:12:50.091+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:12:51.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:12:50 smithi047 ceph-mon[77147]: pgmap v7103: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:12:51.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:12:50 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:12:51.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:12:50 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:12:51.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:12:50 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:12:51.101 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:12:50 smithi115 ceph-mon[78593]: pgmap v7103: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:12:51.102 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:12:50 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:12:51.102 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:12:50 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:12:51.102 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:12:50 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:12:51.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:12:51 smithi115 conmon[90393]: 2022-05-14T03:12:51.100+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:12:52.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:12:51 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:12:52.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:12:51 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 13505 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:12:52.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:12:51 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:12:52.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:12:51 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 13505 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:12:52.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:12:52 smithi115 conmon[90393]: 2022-05-14T03:12:52.126+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:12:53.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:12:52 smithi047 ceph-mon[77147]: pgmap v7104: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:12:53.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:12:52 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:12:53.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:12:52 smithi115 ceph-mon[78593]: pgmap v7104: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:12:53.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:12:52 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:12:53.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:12:53 smithi115 conmon[90393]: 2022-05-14T03:12:53.118+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:12:54.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:12:53 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:12:54.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:12:53 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:12:54.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:12:54 smithi115 conmon[90393]: 2022-05-14T03:12:54.133+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:12:55.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:12:54 smithi047 ceph-mon[77147]: pgmap v7105: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:12:55.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:12:54 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:12:55.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:12:54 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:12:55.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:12:54 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:12:55.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:12:54 smithi115 ceph-mon[78593]: pgmap v7105: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:12:55.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:12:54 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:12:55.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:12:54 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:12:55.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:12:54 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:12:55.610 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:12:55 smithi115 conmon[90393]: 2022-05-14T03:12:55.161+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:12:56.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:12:55 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:12:56.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:12:55 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:12:56.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:12:56 smithi115 conmon[90393]: 2022-05-14T03:12:56.146+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:12:57.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:12:56 smithi047 ceph-mon[77147]: pgmap v7106: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:12:57.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:12:56 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:12:57.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:12:56 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 13510 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:12:57.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:12:56 smithi115 ceph-mon[78593]: pgmap v7106: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:12:57.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:12:56 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:12:57.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:12:56 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 13510 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:12:57.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:12:57 smithi115 conmon[90393]: 2022-05-14T03:12:57.109+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:12:58.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:12:57 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:12:58.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:12:57 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:12:58.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:12:58 smithi115 conmon[90393]: 2022-05-14T03:12:58.109+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:12:59.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:12:58 smithi047 ceph-mon[77147]: pgmap v7107: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:12:59.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:12:58 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:12:59.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:12:58 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:12:59.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:12:58 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:12:59.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:12:58 smithi115 ceph-mon[78593]: pgmap v7107: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:12:59.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:12:58 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:12:59.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:12:58 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:12:59.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:12:58 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:12:59.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:12:59 smithi115 conmon[90393]: 2022-05-14T03:12:59.131+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:13:00.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:12:59 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:13:00.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:12:59 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:13:00.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:13:00 smithi115 conmon[90393]: 2022-05-14T03:13:00.137+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:13:01.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:13:00 smithi047 ceph-mon[77147]: pgmap v7108: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:13:01.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:13:00 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:13:01.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:13:00 smithi115 ceph-mon[78593]: pgmap v7108: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:13:01.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:13:00 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:13:01.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:13:01 smithi115 conmon[90393]: 2022-05-14T03:13:01.119+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:13:02.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:13:01 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:13:02.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:13:01 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 13515 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:13:02.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:13:01 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:13:02.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:13:01 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 13515 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:13:02.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:13:02 smithi115 conmon[90393]: 2022-05-14T03:13:02.113+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:13:03.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:13:02 smithi047 ceph-mon[77147]: pgmap v7109: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:13:03.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:13:02 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:13:03.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:13:02 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:13:03.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:13:02 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:13:03.099 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:13:02 smithi115 ceph-mon[78593]: pgmap v7109: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:13:03.099 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:13:02 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:13:03.099 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:13:02 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:13:03.099 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:13:02 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:13:03.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:13:03 smithi115 conmon[90393]: 2022-05-14T03:13:03.097+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:13:04.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:13:03 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:13:04.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:13:03 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:13:04.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:13:04 smithi115 conmon[90393]: 2022-05-14T03:13:04.107+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:13:05.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:13:04 smithi047 ceph-mon[77147]: pgmap v7110: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:13:05.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:13:04 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:13:05.087 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:13:04 smithi115 ceph-mon[78593]: pgmap v7110: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:13:05.087 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:13:04 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:13:05.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:13:05 smithi115 conmon[90393]: 2022-05-14T03:13:05.086+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:13:06.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:13:05 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:13:06.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:13:05 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T03:13:06.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:13:05 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T03:13:06.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:13:05 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T03:13:06.075 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:13:05 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:13:06.075 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:13:05 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T03:13:06.075 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:13:05 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T03:13:06.076 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:13:05 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T03:13:06.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:13:06 smithi115 conmon[90393]: 2022-05-14T03:13:06.073+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:13:07.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:13:06 smithi047 ceph-mon[77147]: pgmap v7111: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:13:07.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:13:06 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:13:07.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:13:06 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 13520 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:13:07.060 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:13:06 smithi115 ceph-mon[78593]: pgmap v7111: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:13:07.061 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:13:06 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:13:07.061 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:13:06 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 13520 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:13:07.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:13:07 smithi115 conmon[90393]: 2022-05-14T03:13:07.059+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:13:08.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:13:07 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:13:08.060 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:13:07 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:13:08.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:13:08 smithi115 conmon[90393]: 2022-05-14T03:13:08.059+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:13:09.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:13:08 smithi047 ceph-mon[77147]: pgmap v7112: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:13:09.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:13:08 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:13:09.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:13:08 smithi047 conmon[78521]: 2022-05-14T03:13:08.784+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:13:09.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:13:08 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T03:13:09.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:13:08 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:13:09.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:13:08 smithi047 conmon[78521]: yield 2022-05-14T03:13:09.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:13:08 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:13:09.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:13:08 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:13:09.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:13:08 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:13:09.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:13:08 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T03:13:09.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:13:08 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:13:09.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:13:08 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T03:13:09.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:13:08 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:13:09.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:13:08 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T03:13:09.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:13:08 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:13:09.048 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:13:08 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T03:13:09.048 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:13:08 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:13:09.049 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:13:08 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T03:13:09.049 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:13:08 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:13:09.050 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:13:08 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:13:09.050 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:13:08 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:13:09.084 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:13:08 smithi115 ceph-mon[78593]: pgmap v7112: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:13:09.084 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:13:08 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:13:09.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:13:09 smithi115 conmon[90393]: 2022-05-14T03:13:09.082+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:13:10.051 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:13:09 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:13:10.052 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T03:13:10.052 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:13:10.053 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T03:13:10.053 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:13:10.053 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:13:10.054 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:13:10.054 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T03:13:10.054 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:13:10.054 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T03:13:10.055 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:13:10.055 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T03:13:10.055 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:13:10.055 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T03:13:10.056 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:13:10.056 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T03:13:10.056 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:13:10.057 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:13:10.057 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:13:10.058 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:13:09 smithi115 ceph-mon[78593]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T03:13:10.058 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:13:09 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:13:10.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:13:09 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:13:10.292 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T03:13:10.292 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:13:10.292 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T03:13:10.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:13:10.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:13:10.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:13:10.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T03:13:10.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:13:10.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T03:13:10.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:13:10.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T03:13:10.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:13:10.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T03:13:10.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:13:10.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T03:13:10.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:13:10.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:13:10.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:13:10.296 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:13:09 smithi047 ceph-mon[77147]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T03:13:10.296 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:13:09 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:13:10.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:13:10 smithi115 conmon[90393]: 2022-05-14T03:13:10.049+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:13:11.048 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:13:10 smithi115 ceph-mon[78593]: pgmap v7113: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:13:11.049 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:13:10 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:13:11.049 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:13:10 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:13:11.049 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:13:10 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:13:11.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:13:10 smithi047 ceph-mon[77147]: pgmap v7113: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:13:11.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:13:10 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:13:11.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:13:10 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:13:11.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:13:10 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:13:11.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:13:11 smithi115 conmon[90393]: 2022-05-14T03:13:11.047+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:13:12.062 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:13:11 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:13:12.063 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:13:11 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 13525 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:13:12.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:13:11 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:13:12.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:13:11 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 13525 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:13:12.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:13:11 smithi047 conmon[78521]: 2022-05-14T03:13:11.856+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:13:12.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:13:11 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T03:13:12.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:13:11 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:13:12.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:13:11 smithi047 conmon[78521]: yield 2022-05-14T03:13:12.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:13:11 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:13:12.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:13:11 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:13:12.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:13:11 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:13:12.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:13:11 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T03:13:12.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:13:11 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:13:12.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:13:11 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T03:13:12.297 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:13:11 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:13:12.297 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:13:11 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T03:13:12.297 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:13:11 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:13:12.297 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:13:11 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T03:13:12.297 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:13:11 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:13:12.298 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:13:11 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T03:13:12.298 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:13:11 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:13:12.298 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:13:11 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:13:12.298 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:13:11 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:13:12.298 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:13:11 smithi047 conmon[78521]: 2022-05-14T03:13:11.857+0000 7f5344781700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T03:13:12.299 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:13:11 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T03:13:12.299 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:13:11 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:13:12.299 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:13:11 smithi047 conmon[78521]: yield 2022-05-14T03:13:12.299 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:13:11 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:13:12.299 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:13:11 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:13:12.300 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:13:11 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:13:12.300 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:13:11 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T03:13:12.300 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:13:11 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:13:12.300 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:13:11 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T03:13:12.300 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:13:11 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:13:12.301 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:13:11 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T03:13:12.301 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:13:11 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:13:12.301 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:13:11 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T03:13:12.301 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:13:11 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:13:12.302 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:13:11 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T03:13:12.302 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:13:11 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:13:12.302 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:13:11 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:13:12.302 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:13:11 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:13:12.302 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:13:11 smithi047 conmon[78521]: 2022-05-14T03:13:12.303 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:13:11 smithi047 conmon[78521]: During handling of the above exception, another exception occurred: 2022-05-14T03:13:12.303 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:13:11 smithi047 conmon[78521]: 2022-05-14T03:13:12.303 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:13:11 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T03:13:12.303 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:13:11 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T03:13:12.303 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:13:11 smithi047 conmon[78521]: return f(*arg) 2022-05-14T03:13:12.304 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:13:11 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T03:13:12.304 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:13:11 smithi047 conmon[78521]: self._write_client_files(client_files, host) 2022-05-14T03:13:12.304 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:13:11 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T03:13:12.304 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:13:11 smithi047 conmon[78521]: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T03:13:12.305 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:13:11 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T03:13:12.305 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:13:11 smithi047 conmon[78521]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T03:13:12.305 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:13:11 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T03:13:12.305 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:13:11 smithi047 conmon[78521]: return self.event_loop.get_result(coro) 2022-05-14T03:13:12.305 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:13:11 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T03:13:12.306 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:13:11 smithi047 conmon[78521]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T03:13:12.306 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:13:11 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T03:13:12.306 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:13:11 smithi047 conmon[78521]: return self.__get_result() 2022-05-14T03:13:12.306 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:13:11 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T03:13:12.306 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:13:11 smithi047 conmon[78521]: raise self._exception 2022-05-14T03:13:12.307 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:13:11 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T03:13:12.307 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:13:11 smithi047 conmon[78521]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T03:13:12.307 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:13:11 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T03:13:12.307 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:13:11 smithi047 conmon[78521]: conn = await self._remote_connection(host, addr) 2022-05-14T03:13:12.308 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:13:11 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T03:13:12.308 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:13:11 smithi047 conmon[78521]: raise 2022-05-14T03:13:12.308 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:13:11 smithi047 conmon[78521]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T03:13:12.308 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:13:11 smithi047 conmon[78521]: self.gen.throw(type, value, traceback) 2022-05-14T03:13:12.308 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:13:11 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T03:13:12.309 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:13:11 smithi047 conmon[78521]: raise OrchestratorError(msg) 2022-05-14T03:13:12.309 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:13:11 smithi047 conmon[78521]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:13:12.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:13:12 smithi115 conmon[90393]: 2022-05-14T03:13:12.061+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:13:13.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:13:13 smithi115 conmon[90393]: 2022-05-14T03:13:13.044+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:13:13.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:13:12 smithi115 ceph-mon[78593]: pgmap v7114: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:13:13.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:13:12 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:13:13.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:13:12 smithi047 ceph-mon[77147]: pgmap v7114: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:13:13.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:13:12 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:13:14.095 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:13:13 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:13:14.096 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T03:13:14.096 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:13:14.096 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T03:13:14.096 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:13:14.097 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:13:14.097 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:13:14.097 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T03:13:14.097 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:13:14.098 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T03:13:14.098 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:13:14.098 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T03:13:14.098 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:13:14.099 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T03:13:14.099 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:13:14.099 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T03:13:14.099 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:13:14.099 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:13:14.100 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:13:14.100 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:13:13 smithi115 ceph-mon[78593]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T03:13:14.100 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T03:13:14.100 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:13:14.101 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T03:13:14.101 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:13:14.101 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:13:14.101 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:13:14.101 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T03:13:14.102 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:13:14.102 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T03:13:14.102 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:13:14.102 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T03:13:14.103 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:13:14.103 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T03:13:14.103 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:13:14.104 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T03:13:14.104 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:13:14.104 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:13:14.104 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:13:14.105 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T03:13:14.105 INFO:journalctl@ceph.mon.c.smithi115.stdout: During handling of the above exception, another exception occurred: 2022-05-14T03:13:14.105 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T03:13:14.105 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T03:13:14.106 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T03:13:14.106 INFO:journalctl@ceph.mon.c.smithi115.stdout: return f(*arg) 2022-05-14T03:13:14.106 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T03:13:14.107 INFO:journalctl@ceph.mon.c.smithi115.stdout: self._write_client_files(client_files, host) 2022-05-14T03:13:14.107 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T03:13:14.107 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T03:13:14.107 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T03:13:14.107 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T03:13:14.108 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T03:13:14.108 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.event_loop.get_result(coro) 2022-05-14T03:13:14.108 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T03:13:14.108 INFO:journalctl@ceph.mon.c.smithi115.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T03:13:14.108 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T03:13:14.109 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.__get_result() 2022-05-14T03:13:14.109 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T03:13:14.109 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise self._exception 2022-05-14T03:13:14.109 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T03:13:14.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T03:13:14.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T03:13:14.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T03:13:14.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T03:13:14.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise 2022-05-14T03:13:14.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T03:13:14.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.gen.throw(type, value, traceback) 2022-05-14T03:13:14.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T03:13:14.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OrchestratorError(msg) 2022-05-14T03:13:14.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:13:14.112 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:13:13 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:13:14.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:13:13 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:13:14.292 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T03:13:14.292 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:13:14.292 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T03:13:14.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:13:14.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:13:14.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:13:14.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T03:13:14.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:13:14.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T03:13:14.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:13:14.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T03:13:14.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:13:14.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T03:13:14.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:13:14.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T03:13:14.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:13:14.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:13:14.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:13:14.296 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:13:13 smithi047 ceph-mon[77147]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T03:13:14.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T03:13:14.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:13:14.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T03:13:14.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:13:14.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:13:14.298 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:13:14.298 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T03:13:14.298 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:13:14.298 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T03:13:14.299 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:13:14.299 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T03:13:14.299 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:13:14.299 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T03:13:14.299 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:13:14.300 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T03:13:14.300 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:13:14.300 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:13:14.300 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:13:14.301 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T03:13:14.301 INFO:journalctl@ceph.mon.b.smithi047.stdout: During handling of the above exception, another exception occurred: 2022-05-14T03:13:14.301 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T03:13:14.301 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T03:13:14.301 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T03:13:14.302 INFO:journalctl@ceph.mon.b.smithi047.stdout: return f(*arg) 2022-05-14T03:13:14.302 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T03:13:14.302 INFO:journalctl@ceph.mon.b.smithi047.stdout: self._write_client_files(client_files, host) 2022-05-14T03:13:14.302 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T03:13:14.302 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T03:13:14.303 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T03:13:14.303 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T03:13:14.303 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T03:13:14.303 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.event_loop.get_result(coro) 2022-05-14T03:13:14.303 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T03:13:14.304 INFO:journalctl@ceph.mon.b.smithi047.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T03:13:14.304 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T03:13:14.304 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.__get_result() 2022-05-14T03:13:14.304 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T03:13:14.304 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise self._exception 2022-05-14T03:13:14.305 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T03:13:14.305 INFO:journalctl@ceph.mon.b.smithi047.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T03:13:14.305 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T03:13:14.305 INFO:journalctl@ceph.mon.b.smithi047.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T03:13:14.305 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T03:13:14.306 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise 2022-05-14T03:13:14.306 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T03:13:14.306 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.gen.throw(type, value, traceback) 2022-05-14T03:13:14.306 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T03:13:14.307 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OrchestratorError(msg) 2022-05-14T03:13:14.307 INFO:journalctl@ceph.mon.b.smithi047.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:13:14.307 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:13:13 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:13:14.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:13:14 smithi115 conmon[90393]: 2022-05-14T03:13:14.093+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:13:15.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:13:14 smithi115 ceph-mon[78593]: pgmap v7115: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:13:15.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:13:14 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:13:15.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:13:14 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:13:15.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:13:14 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:13:15.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:13:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T03:13:15.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:13:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T03:13:15.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:13:14 smithi047 ceph-mon[77147]: pgmap v7115: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:13:15.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:13:14 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:13:15.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:13:14 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:13:15.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:13:14 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:13:15.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:13:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T03:13:15.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:13:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T03:13:15.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:13:15 smithi115 conmon[90393]: 2022-05-14T03:13:15.112+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:13:16.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:13:15 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:13:16.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:13:15 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:13:16.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:13:16 smithi115 conmon[90393]: 2022-05-14T03:13:16.124+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:13:17.076 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:13:16 smithi115 ceph-mon[78593]: pgmap v7116: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:13:17.076 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:13:16 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:13:17.076 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:13:16 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 13529 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:13:17.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:13:16 smithi047 ceph-mon[77147]: pgmap v7116: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:13:17.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:13:16 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:13:17.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:13:16 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 13529 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:13:17.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:13:17 smithi115 conmon[90393]: 2022-05-14T03:13:17.074+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:13:18.106 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:13:17 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:13:18.106 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:13:17 smithi115 ceph-mon[78593]: pgmap v7117: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:13:18.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:13:17 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:13:18.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:13:17 smithi047 ceph-mon[77147]: pgmap v7117: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:13:18.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:13:18 smithi115 conmon[90393]: 2022-05-14T03:13:18.104+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:13:19.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:13:18 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:13:19.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:13:18 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:13:19.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:13:18 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:13:19.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:13:18 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:13:19.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:13:18 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:13:19.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:13:18 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:13:19.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:13:19 smithi115 conmon[90393]: 2022-05-14T03:13:19.153+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:13:20.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:13:19 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:13:20.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:13:19 smithi115 ceph-mon[78593]: pgmap v7118: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:13:20.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:13:19 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:13:20.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:13:19 smithi047 ceph-mon[77147]: pgmap v7118: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:13:20.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:13:20 smithi115 conmon[90393]: 2022-05-14T03:13:20.130+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:13:21.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:13:20 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:13:21.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:13:20 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:13:21.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:13:21 smithi115 conmon[90393]: 2022-05-14T03:13:21.117+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:13:22.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:13:21 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:13:22.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:13:21 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 13535 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:13:22.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:13:21 smithi115 ceph-mon[78593]: pgmap v7119: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:13:22.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:13:21 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:13:22.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:13:21 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 13535 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:13:22.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:13:21 smithi047 ceph-mon[77147]: pgmap v7119: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:13:22.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:13:22 smithi115 conmon[90393]: 2022-05-14T03:13:22.148+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:13:23.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:13:22 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:13:23.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:13:22 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:13:23.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:13:22 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:13:23.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:13:22 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:13:23.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:13:22 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:13:23.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:13:22 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:13:23.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:13:23 smithi115 conmon[90393]: 2022-05-14T03:13:23.159+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:13:24.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:13:23 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:13:24.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:13:23 smithi115 ceph-mon[78593]: pgmap v7120: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:13:24.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:13:23 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:13:24.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:13:23 smithi047 ceph-mon[77147]: pgmap v7120: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:13:24.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:13:24 smithi115 conmon[90393]: 2022-05-14T03:13:24.160+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:13:25.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:13:24 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:13:25.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:13:24 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:13:25.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:13:25 smithi115 conmon[90393]: 2022-05-14T03:13:25.130+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:13:26.114 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:13:25 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:13:26.115 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:13:25 smithi115 ceph-mon[78593]: pgmap v7121: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:13:26.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:13:25 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:13:26.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:13:25 smithi047 ceph-mon[77147]: pgmap v7121: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:13:26.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:13:26 smithi115 conmon[90393]: 2022-05-14T03:13:26.130+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:13:27.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:13:26 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:13:27.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:13:26 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 13540 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:13:27.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:13:26 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:13:27.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:13:26 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 13540 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:13:27.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:13:27 smithi115 conmon[90393]: 2022-05-14T03:13:27.114+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:13:28.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:13:28 smithi115 conmon[90393]: 2022-05-14T03:13:28.086+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:13:28.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:13:27 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:13:28.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:13:27 smithi115 ceph-mon[78593]: pgmap v7122: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:13:28.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:13:27 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:13:28.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:13:27 smithi047 ceph-mon[77147]: pgmap v7122: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:13:29.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:13:29 smithi115 conmon[90393]: 2022-05-14T03:13:29.095+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:13:29.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:13:28 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:13:29.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:13:28 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:13:30.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:13:30 smithi115 conmon[90393]: 2022-05-14T03:13:30.084+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:13:30.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:13:29 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:13:30.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:13:29 smithi115 ceph-mon[78593]: pgmap v7123: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:13:30.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:13:29 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:13:30.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:13:29 smithi047 ceph-mon[77147]: pgmap v7123: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:13:31.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:13:30 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:13:31.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:13:30 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:13:31.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:13:30 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:13:31.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:13:31 smithi115 conmon[90393]: 2022-05-14T03:13:31.086+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:13:31.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:13:30 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:13:31.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:13:30 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:13:31.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:13:30 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:13:32.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:13:31 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:13:32.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:13:31 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 13544 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:13:32.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:13:31 smithi047 ceph-mon[77147]: pgmap v7124: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:13:32.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:13:32 smithi115 conmon[90393]: 2022-05-14T03:13:32.074+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:13:32.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:13:31 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:13:32.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:13:31 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 13544 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:13:32.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:13:31 smithi115 ceph-mon[78593]: pgmap v7124: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:13:33.336 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:13:33 smithi115 conmon[90393]: 2022-05-14T03:13:33.043+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:13:33.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:13:33 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:13:33.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:13:33 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:13:34.345 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:13:34 smithi115 conmon[90393]: 2022-05-14T03:13:34.014+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:13:34.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:13:34 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:13:34.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:13:34 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:13:34.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:13:34 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:13:34.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:13:34 smithi115 ceph-mon[78593]: pgmap v7125: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:13:34.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:13:34 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:13:34.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:13:34 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:13:34.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:13:34 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:13:34.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:13:34 smithi047 ceph-mon[77147]: pgmap v7125: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:13:35.347 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:13:34 smithi115 conmon[90393]: 2022-05-14T03:13:34.973+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:13:35.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:13:35 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:13:35.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:13:35 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:13:36.351 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:13:36 smithi115 conmon[90393]: 2022-05-14T03:13:36.011+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:13:36.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:13:36 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:13:36.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:13:36 smithi115 ceph-mon[78593]: pgmap v7126: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:13:36.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:13:36 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:13:36.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:13:36 smithi047 ceph-mon[77147]: pgmap v7126: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:13:37.352 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:13:36 smithi115 conmon[90393]: 2022-05-14T03:13:36.965+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:13:37.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:13:37 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:13:37.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:13:37 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 13550 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:13:37.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:13:37 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:13:37.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:13:37 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 13550 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:13:38.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:13:37 smithi115 conmon[90393]: 2022-05-14T03:13:37.960+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:13:38.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:13:38 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:13:38.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:13:38 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:13:38.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:13:38 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:13:38.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:13:38 smithi047 ceph-mon[77147]: pgmap v7127: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:13:38.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:13:38 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:13:38.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:13:38 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:13:38.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:13:38 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:13:38.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:13:38 smithi115 ceph-mon[78593]: pgmap v7127: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:13:39.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:13:38 smithi115 conmon[90393]: 2022-05-14T03:13:38.984+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:13:39.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:13:39 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:13:39.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:13:39 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:13:40.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:13:40 smithi115 conmon[90393]: 2022-05-14T03:13:40.001+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:13:40.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:13:40 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:13:40.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:13:40 smithi047 ceph-mon[77147]: pgmap v7128: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:13:40.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:13:40 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:13:40.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:13:40 smithi115 ceph-mon[78593]: pgmap v7128: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:13:41.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:13:41 smithi115 conmon[90393]: 2022-05-14T03:13:41.033+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:13:41.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:13:41 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:13:41.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:13:41 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:13:42.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:13:42 smithi115 conmon[90393]: 2022-05-14T03:13:42.041+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:13:42.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:13:42 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:13:42.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:13:42 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:13:42.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:13:42 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:13:42.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:13:42 smithi047 ceph-mon[77147]: pgmap v7129: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:13:42.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:13:42 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:13:42.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:13:42 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:13:42.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:13:42 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:13:42.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:13:42 smithi115 ceph-mon[78593]: pgmap v7129: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:13:43.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:13:43 smithi115 conmon[90393]: 2022-05-14T03:13:43.064+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:13:43.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:13:43 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:13:43.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:13:43 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:13:44.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:13:44 smithi115 conmon[90393]: 2022-05-14T03:13:44.017+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:13:44.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:13:44 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:13:44.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:13:44 smithi047 ceph-mon[77147]: pgmap v7130: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:13:44.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:13:44 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:13:44.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:13:44 smithi115 ceph-mon[78593]: pgmap v7130: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:13:45.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:13:44 smithi115 conmon[90393]: 2022-05-14T03:13:44.982+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:13:45.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:13:45 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:13:45.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:13:45 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:13:46.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:13:45 smithi115 conmon[90393]: 2022-05-14T03:13:45.935+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:13:46.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:13:46 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:13:46.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:13:46 smithi047 ceph-mon[77147]: pgmap v7131: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:13:46.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:13:46 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 13560 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:13:46.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:13:46 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:13:46.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:13:46 smithi115 ceph-mon[78593]: pgmap v7131: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:13:46.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:13:46 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 13560 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:13:47.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:13:46 smithi115 conmon[90393]: 2022-05-14T03:13:46.957+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:13:47.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:13:47 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:13:47.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:13:47 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:13:48.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:13:47 smithi115 conmon[90393]: 2022-05-14T03:13:47.996+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:13:48.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:13:48 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:13:48.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:13:48 smithi047 ceph-mon[77147]: pgmap v7132: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:13:48.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:13:48 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:13:48.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:13:48 smithi115 ceph-mon[78593]: pgmap v7132: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:13:49.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:13:49 smithi115 conmon[90393]: 2022-05-14T03:13:49.004+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:13:49.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:13:49 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:13:49.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:13:49 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:13:50.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:13:49 smithi115 conmon[90393]: 2022-05-14T03:13:49.992+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:13:50.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:13:50 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:13:50.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:13:50 smithi047 ceph-mon[77147]: pgmap v7133: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:13:50.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:13:50 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:13:50.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:13:50 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:13:50.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:13:50 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:13:50.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:13:50 smithi115 ceph-mon[78593]: pgmap v7133: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:13:50.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:13:50 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:13:50.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:13:50 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:13:51.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:13:50 smithi115 conmon[90393]: 2022-05-14T03:13:50.959+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:13:51.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:13:51 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:13:51.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:13:51 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 13565 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:13:51.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:13:51 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:13:51.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:13:51 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 13565 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:13:52.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:13:51 smithi115 conmon[90393]: 2022-05-14T03:13:51.917+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:13:52.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:13:52 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:13:52.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:13:52 smithi047 ceph-mon[77147]: pgmap v7134: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:13:52.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:13:52 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:13:52.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:13:52 smithi115 ceph-mon[78593]: pgmap v7134: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:13:53.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:13:52 smithi115 conmon[90393]: 2022-05-14T03:13:52.892+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:13:53.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:13:53 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:13:53.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:13:53 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:13:54.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:13:53 smithi115 conmon[90393]: 2022-05-14T03:13:53.871+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:13:54.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:13:54 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:13:54.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:13:54 smithi047 ceph-mon[77147]: pgmap v7135: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:13:54.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:13:54 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:13:54.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:13:54 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:13:54.835 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:13:54 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:13:54.835 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:13:54 smithi115 ceph-mon[78593]: pgmap v7135: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:13:54.836 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:13:54 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:13:54.836 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:13:54 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:13:55.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:13:54 smithi115 conmon[90393]: 2022-05-14T03:13:54.833+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:13:55.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:13:55 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:13:55.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:13:55 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:13:56.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:13:55 smithi115 conmon[90393]: 2022-05-14T03:13:55.791+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:13:56.744 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:13:56 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:13:56.744 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:13:56 smithi115 ceph-mon[78593]: pgmap v7136: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:13:56.744 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:13:56 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 13570 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:13:56.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:13:56 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:13:56.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:13:56 smithi047 ceph-mon[77147]: pgmap v7136: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:13:56.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:13:56 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 13570 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:13:57.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:13:56 smithi115 conmon[90393]: 2022-05-14T03:13:56.741+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:13:57.760 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:13:57 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:13:57.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:13:57 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:13:58.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:13:57 smithi115 conmon[90393]: 2022-05-14T03:13:57.758+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:13:58.736 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:13:58 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:13:58.736 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:13:58 smithi115 ceph-mon[78593]: pgmap v7137: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:13:58.737 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:13:58 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:13:58.737 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:13:58 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:13:58.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:13:58 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:13:58.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:13:58 smithi047 ceph-mon[77147]: pgmap v7137: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:13:58.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:13:58 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:13:58.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:13:58 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:13:59.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:13:58 smithi115 conmon[90393]: 2022-05-14T03:13:58.734+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:13:59.786 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:13:59 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:13:59.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:13:59 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:14:00.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:13:59 smithi115 conmon[90393]: 2022-05-14T03:13:59.785+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:14:00.779 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:14:00 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:14:00.780 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:14:00 smithi115 ceph-mon[78593]: pgmap v7138: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:14:00.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:14:00 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:14:00.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:14:00 smithi047 ceph-mon[77147]: pgmap v7138: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:14:01.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:14:00 smithi115 conmon[90393]: 2022-05-14T03:14:00.777+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:14:01.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:14:01 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:14:01.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:14:01 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 13575 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:14:01.817 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:14:01 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:14:01.817 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:14:01 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 13575 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:14:02.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:14:01 smithi115 conmon[90393]: 2022-05-14T03:14:01.816+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:14:02.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:14:02 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:14:02.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:14:02 smithi047 ceph-mon[77147]: pgmap v7139: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:14:02.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:14:02 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:14:02.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:14:02 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:14:02.827 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:14:02 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:14:02.828 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:14:02 smithi115 ceph-mon[78593]: pgmap v7139: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:14:02.828 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:14:02 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:14:02.828 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:14:02 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:14:03.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:14:02 smithi115 conmon[90393]: 2022-05-14T03:14:02.826+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:14:03.790 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:14:03 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:14:03.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:14:03 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:14:04.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:14:03 smithi115 conmon[90393]: 2022-05-14T03:14:03.788+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:14:04.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:14:04 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:14:04.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:14:04 smithi047 ceph-mon[77147]: pgmap v7140: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:14:04.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:14:04 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:14:04.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:14:04 smithi115 ceph-mon[78593]: pgmap v7140: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:14:05.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:14:04 smithi115 conmon[90393]: 2022-05-14T03:14:04.791+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:14:05.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:14:05 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:14:05.822 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:14:05 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:14:06.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:14:05 smithi115 conmon[90393]: 2022-05-14T03:14:05.821+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:14:06.790 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:14:06 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:14:06.790 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:14:06 smithi115 ceph-mon[78593]: pgmap v7141: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:14:06.790 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:14:06 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 13580 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:14:06.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:14:06 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:14:06.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:14:06 smithi047 ceph-mon[77147]: pgmap v7141: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:14:06.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:14:06 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 13580 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:14:07.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:14:06 smithi115 conmon[90393]: 2022-05-14T03:14:06.788+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:14:07.760 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:14:07 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:14:07.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:14:07 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:14:08.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:14:07 smithi115 conmon[90393]: 2022-05-14T03:14:07.759+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:14:08.717 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:14:08 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:14:08.717 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:14:08 smithi115 ceph-mon[78593]: pgmap v7142: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:14:08.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:14:08 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:14:08.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:14:08 smithi047 ceph-mon[77147]: pgmap v7142: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:14:09.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:14:08 smithi115 conmon[90393]: 2022-05-14T03:14:08.715+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:14:09.740 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:14:09 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:14:09.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:14:09 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:14:10.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:14:09 smithi115 conmon[90393]: 2022-05-14T03:14:09.739+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:14:10.779 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:14:10 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:14:10.779 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:14:10 smithi115 ceph-mon[78593]: pgmap v7143: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:14:10.779 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:14:10 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:14:10.780 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:14:10 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:14:10.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:14:10 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:14:10.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:14:10 smithi047 ceph-mon[77147]: pgmap v7143: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:14:10.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:14:10 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:14:10.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:14:10 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:14:11.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:14:10 smithi115 conmon[90393]: 2022-05-14T03:14:10.777+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:14:11.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:14:11 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:14:11.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:14:11 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 13585 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:14:11.822 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:14:11 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:14:11.822 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:14:11 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 13585 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:14:12.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:14:11 smithi115 conmon[90393]: 2022-05-14T03:14:11.820+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:14:12.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:14:12 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:14:12.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:14:12 smithi047 ceph-mon[77147]: pgmap v7144: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:14:12.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:14:12 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T03:14:12.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:14:12 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T03:14:12.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:14:12 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T03:14:12.793 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:14:12 smithi047 conmon[78521]: 2022-05-14T03:14:12.337+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:14:12.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:14:12 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T03:14:12.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:14:12 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:14:12.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:14:12 smithi047 conmon[78521]: yield 2022-05-14T03:14:12.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:14:12 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:14:12.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:14:12 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:14:12.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:14:12 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:14:12.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:14:12 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T03:14:12.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:14:12 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:14:12.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:14:12 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T03:14:12.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:14:12 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:14:12.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:14:12 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T03:14:12.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:14:12 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:14:12.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:14:12 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T03:14:12.797 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:14:12 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:14:12.797 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:14:12 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T03:14:12.797 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:14:12 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:14:12.797 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:14:12 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:14:12.798 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:14:12 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:14:12.813 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:14:12 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:14:12.813 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:14:12 smithi115 ceph-mon[78593]: pgmap v7144: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:14:12.814 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:14:12 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T03:14:12.814 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:14:12 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T03:14:12.814 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:14:12 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T03:14:13.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:14:12 smithi115 conmon[90393]: 2022-05-14T03:14:12.811+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:14:13.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:14:13 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:14:13.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:14:13 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:14:13.792 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T03:14:13.792 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:14:13.793 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T03:14:13.793 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:14:13.793 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:14:13.793 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:14:13.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T03:14:13.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:14:13.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T03:14:13.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:14:13.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T03:14:13.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:14:13.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T03:14:13.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:14:13.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T03:14:13.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:14:13.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:14:13.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:14:13.796 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:14:13 smithi047 ceph-mon[77147]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T03:14:13.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:14:13 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:14:13.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:14:13 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:14:13.859 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T03:14:13.859 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:14:13.859 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T03:14:13.860 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:14:13.860 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:14:13.860 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:14:13.860 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T03:14:13.860 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:14:13.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T03:14:13.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:14:13.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T03:14:13.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:14:13.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T03:14:13.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:14:13.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T03:14:13.863 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:14:13.863 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:14:13.863 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:14:13.863 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:14:13 smithi115 ceph-mon[78593]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T03:14:14.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:14:13 smithi115 conmon[90393]: 2022-05-14T03:14:13.857+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:14:14.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:14:14 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:14:14.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:14:14 smithi047 ceph-mon[77147]: pgmap v7145: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:14:14.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:14:14 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:14:14.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:14:14 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:14:14.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:14:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T03:14:14.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:14:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T03:14:14.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:14:14 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:14:14.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:14:14 smithi115 ceph-mon[78593]: pgmap v7145: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:14:14.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:14:14 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:14:14.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:14:14 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:14:14.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:14:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T03:14:14.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:14:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T03:14:15.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:14:14 smithi115 conmon[90393]: 2022-05-14T03:14:14.884+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:14:15.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:14:15 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:14:15.792 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:14:15 smithi047 conmon[78521]: 2022-05-14T03:14:15.407+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:14:15.793 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:14:15 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T03:14:15.793 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:14:15 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:14:15.793 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:14:15 smithi047 conmon[78521]: yield 2022-05-14T03:14:15.793 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:14:15 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:14:15.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:14:15 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:14:15.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:14:15 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:14:15.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:14:15 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T03:14:15.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:14:15 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:14:15.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:14:15 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T03:14:15.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:14:15 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:14:15.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:14:15 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T03:14:15.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:14:15 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:14:15.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:14:15 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T03:14:15.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:14:15 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:14:15.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:14:15 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T03:14:15.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:14:15 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:14:15.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:14:15 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:14:15.797 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:14:15 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:14:15.797 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:14:15 smithi047 conmon[78521]: 2022-05-14T03:14:15.408+0000 7f5345f84700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T03:14:15.797 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:14:15 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T03:14:15.797 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:14:15 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:14:15.798 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:14:15 smithi047 conmon[78521]: yield 2022-05-14T03:14:15.798 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:14:15 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:14:15.798 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:14:15 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:14:15.798 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:14:15 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:14:15.798 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:14:15 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T03:14:15.799 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:14:15 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:14:15.799 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:14:15 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T03:14:15.799 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:14:15 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:14:15.799 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:14:15 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T03:14:15.800 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:14:15 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:14:15.800 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:14:15 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T03:14:15.800 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:14:15 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:14:15.800 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:14:15 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T03:14:15.800 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:14:15 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:14:15.801 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:14:15 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:14:15.801 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:14:15 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:14:15.801 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:14:15 smithi047 conmon[78521]: 2022-05-14T03:14:15.801 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:14:15 smithi047 conmon[78521]: During handling of the above exception, another exception occurred: 2022-05-14T03:14:15.802 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:14:15 smithi047 conmon[78521]: 2022-05-14T03:14:15.802 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:14:15 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T03:14:15.802 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:14:15 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T03:14:15.802 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:14:15 smithi047 conmon[78521]: return f(*arg) 2022-05-14T03:14:15.802 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:14:15 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T03:14:15.803 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:14:15 smithi047 conmon[78521]: self._write_client_files(client_files, host) 2022-05-14T03:14:15.803 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:14:15 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T03:14:15.803 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:14:15 smithi047 conmon[78521]: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T03:14:15.803 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:14:15 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T03:14:15.803 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:14:15 smithi047 conmon[78521]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T03:14:15.804 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:14:15 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T03:14:15.804 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:14:15 smithi047 conmon[78521]: return self.event_loop.get_result(coro) 2022-05-14T03:14:15.804 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:14:15 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T03:14:15.804 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:14:15 smithi047 conmon[78521]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T03:14:15.805 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:14:15 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T03:14:15.805 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:14:15 smithi047 conmon[78521]: return self.__get_result() 2022-05-14T03:14:15.805 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:14:15 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T03:14:15.805 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:14:15 smithi047 conmon[78521]: raise self._exception 2022-05-14T03:14:15.805 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:14:15 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T03:14:15.806 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:14:15 smithi047 conmon[78521]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T03:14:15.806 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:14:15 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T03:14:15.806 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:14:15 smithi047 conmon[78521]: conn = await self._remote_connection(host, addr) 2022-05-14T03:14:15.806 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:14:15 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T03:14:15.807 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:14:15 smithi047 conmon[78521]: raise 2022-05-14T03:14:15.807 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:14:15 smithi047 conmon[78521]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T03:14:15.807 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:14:15 smithi047 conmon[78521]: self.gen.throw(type, value, traceback) 2022-05-14T03:14:15.807 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:14:15 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T03:14:15.807 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:14:15 smithi047 conmon[78521]: raise OrchestratorError(msg) 2022-05-14T03:14:15.808 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:14:15 smithi047 conmon[78521]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:14:15.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:14:15 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:14:16.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:14:15 smithi115 conmon[90393]: 2022-05-14T03:14:15.873+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:14:16.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:14:16 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:14:16.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:14:16 smithi047 ceph-mon[77147]: pgmap v7146: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:14:16.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:14:16 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:14:16.793 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T03:14:16.793 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:14:16.793 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T03:14:16.793 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:14:16.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:14:16.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:14:16.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T03:14:16.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:14:16.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T03:14:16.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:14:16.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T03:14:16.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:14:16.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T03:14:16.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:14:16.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T03:14:16.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:14:16.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:14:16.797 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:14:16.797 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:14:16 smithi047 ceph-mon[77147]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T03:14:16.797 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T03:14:16.797 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:14:16.798 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T03:14:16.798 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:14:16.798 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:14:16.798 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:14:16.798 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T03:14:16.799 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:14:16.799 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T03:14:16.799 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:14:16.799 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T03:14:16.799 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:14:16.800 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T03:14:16.800 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:14:16.800 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T03:14:16.800 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:14:16.800 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:14:16.801 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:14:16.801 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T03:14:16.801 INFO:journalctl@ceph.mon.b.smithi047.stdout: During handling of the above exception, another exception occurred: 2022-05-14T03:14:16.801 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T03:14:16.801 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T03:14:16.802 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T03:14:16.802 INFO:journalctl@ceph.mon.b.smithi047.stdout: return f(*arg) 2022-05-14T03:14:16.802 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T03:14:16.802 INFO:journalctl@ceph.mon.b.smithi047.stdout: self._write_client_files(client_files, host) 2022-05-14T03:14:16.803 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T03:14:16.803 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T03:14:16.803 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T03:14:16.803 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T03:14:16.803 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T03:14:16.804 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.event_loop.get_result(coro) 2022-05-14T03:14:16.804 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T03:14:16.804 INFO:journalctl@ceph.mon.b.smithi047.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T03:14:16.804 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T03:14:16.804 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.__get_result() 2022-05-14T03:14:16.805 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T03:14:16.805 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise self._exception 2022-05-14T03:14:16.805 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T03:14:16.805 INFO:journalctl@ceph.mon.b.smithi047.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T03:14:16.806 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T03:14:16.806 INFO:journalctl@ceph.mon.b.smithi047.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T03:14:16.806 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T03:14:16.806 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise 2022-05-14T03:14:16.806 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T03:14:16.807 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.gen.throw(type, value, traceback) 2022-05-14T03:14:16.807 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T03:14:16.807 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OrchestratorError(msg) 2022-05-14T03:14:16.807 INFO:journalctl@ceph.mon.b.smithi047.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:14:16.807 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:14:16 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 13590 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:14:16.851 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:14:16 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:14:16.851 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:14:16 smithi115 ceph-mon[78593]: pgmap v7146: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:14:16.851 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:14:16 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:14:16.852 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T03:14:16.852 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:14:16.852 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T03:14:16.852 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:14:16.853 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:14:16.853 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:14:16.853 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T03:14:16.853 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:14:16.853 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T03:14:16.854 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:14:16.854 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T03:14:16.854 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:14:16.854 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T03:14:16.855 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:14:16.855 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T03:14:16.855 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:14:16.855 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:14:16.855 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:14:16.856 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:14:16 smithi115 ceph-mon[78593]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T03:14:16.856 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T03:14:16.856 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:14:16.856 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T03:14:16.857 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:14:16.857 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:14:16.857 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:14:16.857 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T03:14:16.857 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:14:16.858 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T03:14:16.858 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:14:16.858 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T03:14:16.858 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:14:16.858 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T03:14:16.859 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:14:16.859 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T03:14:16.859 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:14:16.859 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:14:16.860 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:14:16.860 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T03:14:16.860 INFO:journalctl@ceph.mon.c.smithi115.stdout: During handling of the above exception, another exception occurred: 2022-05-14T03:14:16.860 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T03:14:16.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T03:14:16.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T03:14:16.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: return f(*arg) 2022-05-14T03:14:16.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T03:14:16.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: self._write_client_files(client_files, host) 2022-05-14T03:14:16.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T03:14:16.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T03:14:16.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T03:14:16.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T03:14:16.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T03:14:16.863 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.event_loop.get_result(coro) 2022-05-14T03:14:16.863 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T03:14:16.863 INFO:journalctl@ceph.mon.c.smithi115.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T03:14:16.863 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T03:14:16.864 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.__get_result() 2022-05-14T03:14:16.864 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T03:14:16.864 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise self._exception 2022-05-14T03:14:16.864 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T03:14:16.864 INFO:journalctl@ceph.mon.c.smithi115.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T03:14:16.865 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T03:14:16.865 INFO:journalctl@ceph.mon.c.smithi115.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T03:14:16.865 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T03:14:16.865 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise 2022-05-14T03:14:16.866 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T03:14:16.866 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.gen.throw(type, value, traceback) 2022-05-14T03:14:16.866 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T03:14:16.866 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OrchestratorError(msg) 2022-05-14T03:14:16.866 INFO:journalctl@ceph.mon.c.smithi115.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:14:16.867 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:14:16 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 13590 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:14:17.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:14:16 smithi115 conmon[90393]: 2022-05-14T03:14:16.848+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:14:17.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:14:17 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:14:17.801 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:14:17 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:14:18.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:14:17 smithi115 conmon[90393]: 2022-05-14T03:14:17.799+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:14:18.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:14:18 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:14:18.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:14:18 smithi047 ceph-mon[77147]: pgmap v7147: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:14:18.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:14:18 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:14:18.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:14:18 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:14:18.810 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:14:18 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:14:18.810 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:14:18 smithi115 ceph-mon[78593]: pgmap v7147: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:14:18.810 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:14:18 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:14:18.810 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:14:18 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:14:19.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:14:18 smithi115 conmon[90393]: 2022-05-14T03:14:18.808+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:14:19.779 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:14:19 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:14:19.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:14:19 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:14:20.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:14:19 smithi115 conmon[90393]: 2022-05-14T03:14:19.776+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:14:20.790 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:14:20 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:14:20.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:14:20 smithi115 ceph-mon[78593]: pgmap v7148: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:14:20.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:14:20 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:14:20.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:14:20 smithi047 ceph-mon[77147]: pgmap v7148: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:14:21.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:14:20 smithi115 conmon[90393]: 2022-05-14T03:14:20.788+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:14:21.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:14:21 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:14:21.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:14:21 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 13595 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:14:21.803 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:14:21 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:14:21.804 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:14:21 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 13595 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:14:22.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:14:21 smithi115 conmon[90393]: 2022-05-14T03:14:21.801+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:14:22.765 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:14:22 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:14:22.765 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:14:22 smithi115 ceph-mon[78593]: pgmap v7149: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:14:22.765 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:14:22 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:14:22.766 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:14:22 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:14:22.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:14:22 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:14:22.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:14:22 smithi047 ceph-mon[77147]: pgmap v7149: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:14:22.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:14:22 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:14:22.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:14:22 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:14:23.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:14:22 smithi115 conmon[90393]: 2022-05-14T03:14:22.763+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:14:23.747 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:14:23 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:14:23.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:14:23 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:14:24.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:14:23 smithi115 conmon[90393]: 2022-05-14T03:14:23.745+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:14:24.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:14:24 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:14:24.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:14:24 smithi047 ceph-mon[77147]: pgmap v7150: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:14:24.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:14:24 smithi115 conmon[90393]: 2022-05-14T03:14:24.706+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:14:24.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:14:24 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:14:24.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:14:24 smithi115 ceph-mon[78593]: pgmap v7150: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:14:25.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:14:25 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:14:25.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:14:25 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:14:25.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:14:25 smithi115 conmon[90393]: 2022-05-14T03:14:25.746+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:14:26.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:14:26 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:14:26.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:14:26 smithi047 ceph-mon[77147]: pgmap v7151: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:14:26.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:14:26 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 13600 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:14:26.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:14:26 smithi115 conmon[90393]: 2022-05-14T03:14:26.733+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:14:26.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:14:26 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:14:26.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:14:26 smithi115 ceph-mon[78593]: pgmap v7151: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:14:26.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:14:26 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 13600 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:14:27.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:14:27 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:14:27.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:14:27 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:14:27.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:14:27 smithi115 conmon[90393]: 2022-05-14T03:14:27.689+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:14:28.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:14:28 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:14:28.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:14:28 smithi047 ceph-mon[77147]: pgmap v7152: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:14:28.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:14:28 smithi115 conmon[90393]: 2022-05-14T03:14:28.672+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:14:28.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:14:28 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:14:28.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:14:28 smithi115 ceph-mon[78593]: pgmap v7152: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:14:29.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:14:29 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:14:29.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:14:29 smithi115 conmon[90393]: 2022-05-14T03:14:29.664+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:14:29.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:14:29 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:14:30.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:14:30 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:14:30.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:14:30 smithi047 ceph-mon[77147]: pgmap v7153: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:14:30.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:14:30 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:14:30.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:14:30 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:14:30.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:14:30 smithi115 conmon[90393]: 2022-05-14T03:14:30.695+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:14:30.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:14:30 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:14:30.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:14:30 smithi115 ceph-mon[78593]: pgmap v7153: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:14:30.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:14:30 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:14:30.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:14:30 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:14:31.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:14:31 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:14:31.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:14:31 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 13605 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:14:31.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:14:31 smithi115 conmon[90393]: 2022-05-14T03:14:31.648+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:14:31.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:14:31 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:14:31.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:14:31 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 13605 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:14:32.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:14:32 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:14:32.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:14:32 smithi047 ceph-mon[77147]: pgmap v7154: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:14:32.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:14:32 smithi115 conmon[90393]: 2022-05-14T03:14:32.662+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:14:32.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:14:32 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:14:32.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:14:32 smithi115 ceph-mon[78593]: pgmap v7154: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:14:33.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:14:33 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:14:33.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:14:33 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:14:33.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:14:33 smithi115 conmon[90393]: 2022-05-14T03:14:33.648+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:14:34.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:14:34 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:14:34.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:14:34 smithi047 ceph-mon[77147]: pgmap v7155: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:14:34.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:14:34 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:14:34.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:14:34 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:14:34.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:14:34 smithi115 conmon[90393]: 2022-05-14T03:14:34.664+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:14:34.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:14:34 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:14:34.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:14:34 smithi115 ceph-mon[78593]: pgmap v7155: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:14:34.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:14:34 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:14:34.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:14:34 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:14:35.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:14:35 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:14:35.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:14:35 smithi115 conmon[90393]: 2022-05-14T03:14:35.618+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:14:35.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:14:35 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:14:36.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:14:36 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:14:36.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:14:36 smithi047 ceph-mon[77147]: pgmap v7156: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:14:36.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:14:36 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:14:36.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:14:36 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 13610 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:14:36.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:14:36 smithi115 conmon[90393]: 2022-05-14T03:14:36.651+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:14:36.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:14:36 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:14:36.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:14:36 smithi115 ceph-mon[78593]: pgmap v7156: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:14:36.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:14:36 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:14:36.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:14:36 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 13610 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:14:38.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:14:37 smithi115 conmon[90393]: 2022-05-14T03:14:37.664+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:14:38.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:14:38 smithi115 conmon[90393]: 2022-05-14T03:14:38.632+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:14:38.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:14:38 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:14:38.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:14:38 smithi115 ceph-mon[78593]: pgmap v7157: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:14:38.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:14:38 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:14:38.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:14:38 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:14:39.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:14:38 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:14:39.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:14:38 smithi047 ceph-mon[77147]: pgmap v7157: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:14:39.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:14:38 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:14:39.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:14:38 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:14:39.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:14:39 smithi115 conmon[90393]: 2022-05-14T03:14:39.664+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:14:39.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:14:39 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:14:40.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:14:39 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:14:40.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:14:40 smithi115 conmon[90393]: 2022-05-14T03:14:40.702+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:14:40.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:14:40 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:14:40.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:14:40 smithi115 ceph-mon[78593]: pgmap v7158: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:14:41.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:14:40 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:14:41.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:14:40 smithi047 ceph-mon[77147]: pgmap v7158: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:14:41.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:14:41 smithi115 conmon[90393]: 2022-05-14T03:14:41.698+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:14:41.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:14:41 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:14:41.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:14:41 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 13615 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:14:42.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:14:41 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:14:42.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:14:41 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 13615 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:14:42.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:14:42 smithi115 conmon[90393]: 2022-05-14T03:14:42.688+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:14:42.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:14:42 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:14:42.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:14:42 smithi115 ceph-mon[78593]: pgmap v7159: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:14:42.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:14:42 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:14:42.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:14:42 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:14:43.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:14:42 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:14:43.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:14:42 smithi047 ceph-mon[77147]: pgmap v7159: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:14:43.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:14:42 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:14:43.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:14:42 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:14:43.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:14:43 smithi115 conmon[90393]: 2022-05-14T03:14:43.701+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:14:43.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:14:43 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:14:44.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:14:43 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:14:44.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:14:44 smithi115 conmon[90393]: 2022-05-14T03:14:44.705+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:14:44.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:14:44 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:14:44.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:14:44 smithi115 ceph-mon[78593]: pgmap v7160: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:14:45.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:14:44 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:14:45.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:14:44 smithi047 ceph-mon[77147]: pgmap v7160: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:14:45.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:14:45 smithi115 conmon[90393]: 2022-05-14T03:14:45.738+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:14:45.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:14:45 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:14:46.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:14:45 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:14:46.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:14:46 smithi115 conmon[90393]: 2022-05-14T03:14:46.774+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:14:46.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:14:46 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:14:46.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:14:46 smithi115 ceph-mon[78593]: pgmap v7161: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:14:46.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:14:46 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 13620 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:14:47.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:14:46 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:14:47.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:14:46 smithi047 ceph-mon[77147]: pgmap v7161: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:14:47.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:14:46 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 13620 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:14:47.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:14:47 smithi115 conmon[90393]: 2022-05-14T03:14:47.808+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:14:47.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:14:47 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:14:48.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:14:47 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:14:48.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:14:48 smithi115 conmon[90393]: 2022-05-14T03:14:48.777+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:14:48.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:14:48 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:14:48.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:14:48 smithi115 ceph-mon[78593]: pgmap v7162: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:14:49.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:14:48 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:14:49.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:14:48 smithi047 ceph-mon[77147]: pgmap v7162: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:14:49.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:14:49 smithi115 conmon[90393]: 2022-05-14T03:14:49.732+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:14:49.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:14:49 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:14:50.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:14:49 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:14:50.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:14:50 smithi115 conmon[90393]: 2022-05-14T03:14:50.751+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:14:50.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:14:50 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:14:50.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:14:50 smithi115 ceph-mon[78593]: pgmap v7163: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:14:50.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:14:50 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:14:50.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:14:50 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:14:51.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:14:50 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:14:51.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:14:50 smithi047 ceph-mon[77147]: pgmap v7163: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:14:51.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:14:50 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:14:51.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:14:50 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:14:51.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:14:51 smithi115 conmon[90393]: 2022-05-14T03:14:51.721+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:14:51.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:14:51 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:14:51.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:14:51 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 13625 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:14:52.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:14:51 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:14:52.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:14:51 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 13625 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:14:52.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:14:52 smithi115 conmon[90393]: 2022-05-14T03:14:52.705+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:14:52.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:14:52 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:14:52.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:14:52 smithi115 ceph-mon[78593]: pgmap v7164: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:14:53.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:14:52 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:14:53.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:14:52 smithi047 ceph-mon[77147]: pgmap v7164: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:14:53.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:14:53 smithi115 conmon[90393]: 2022-05-14T03:14:53.688+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:14:53.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:14:53 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:14:54.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:14:53 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:14:54.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:14:54 smithi115 conmon[90393]: 2022-05-14T03:14:54.656+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:14:54.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:14:54 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:14:54.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:14:54 smithi115 ceph-mon[78593]: pgmap v7165: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:14:54.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:14:54 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:14:54.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:14:54 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:14:55.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:14:54 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:14:55.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:14:54 smithi047 ceph-mon[77147]: pgmap v7165: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:14:55.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:14:54 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:14:55.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:14:54 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:14:55.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:14:55 smithi115 conmon[90393]: 2022-05-14T03:14:55.620+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:14:55.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:14:55 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:14:56.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:14:55 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:14:56.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:14:56 smithi115 conmon[90393]: 2022-05-14T03:14:56.593+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:14:56.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:14:56 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:14:56.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:14:56 smithi115 ceph-mon[78593]: pgmap v7166: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:14:56.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:14:56 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:14:56.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:14:56 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 13630 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:14:57.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:14:56 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:14:57.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:14:56 smithi047 ceph-mon[77147]: pgmap v7166: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:14:57.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:14:56 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:14:57.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:14:56 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 13630 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:14:57.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:14:57 smithi115 conmon[90393]: 2022-05-14T03:14:57.598+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:14:57.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:14:57 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:14:58.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:14:57 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:14:58.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:14:58 smithi115 conmon[90393]: 2022-05-14T03:14:58.637+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:14:58.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:14:58 smithi115 ceph-mon[78593]: pgmap v7167: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:14:58.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:14:58 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:14:58.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:14:58 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:14:58.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:14:58 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:14:59.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:14:58 smithi047 ceph-mon[77147]: pgmap v7167: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:14:59.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:14:58 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:14:59.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:14:58 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:14:59.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:14:58 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:15:00.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:14:59 smithi115 conmon[90393]: 2022-05-14T03:14:59.653+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:15:01.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:15:00 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:15:01.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:15:00 smithi047 ceph-mon[77147]: pgmap v7168: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:15:01.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:15:00 smithi115 conmon[90393]: 2022-05-14T03:15:00.627+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:15:01.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:15:00 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:15:01.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:15:00 smithi115 ceph-mon[78593]: pgmap v7168: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:15:01.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:15:01 smithi115 conmon[90393]: 2022-05-14T03:15:01.599+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:15:01.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:15:01 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:15:01.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:15:01 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:15:01.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:15:01 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 13635 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:15:02.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:15:01 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:15:02.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:15:01 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:15:02.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:15:01 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 13635 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:15:02.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:15:02 smithi115 conmon[90393]: 2022-05-14T03:15:02.567+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:15:02.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:15:02 smithi115 ceph-mon[78593]: pgmap v7169: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:15:02.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:15:02 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:15:02.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:15:02 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:15:02.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:15:02 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:15:03.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:15:02 smithi047 ceph-mon[77147]: pgmap v7169: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:15:03.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:15:02 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:15:03.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:15:02 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:15:03.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:15:02 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:15:03.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:15:03 smithi115 conmon[90393]: 2022-05-14T03:15:03.521+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:15:03.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:15:03 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:15:04.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:15:03 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:15:04.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:15:04 smithi115 conmon[90393]: 2022-05-14T03:15:04.525+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:15:04.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:15:04 smithi115 ceph-mon[78593]: pgmap v7170: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:15:04.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:15:04 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:15:05.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:15:04 smithi047 ceph-mon[77147]: pgmap v7170: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:15:05.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:15:04 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:15:05.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:15:05 smithi115 conmon[90393]: 2022-05-14T03:15:05.569+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:15:05.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:15:05 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:15:06.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:15:05 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:15:06.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:15:06 smithi115 conmon[90393]: 2022-05-14T03:15:06.555+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:15:06.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:15:06 smithi115 ceph-mon[78593]: pgmap v7171: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:15:06.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:15:06 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:15:06.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:15:06 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 13640 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:15:07.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:15:06 smithi047 ceph-mon[77147]: pgmap v7171: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:15:07.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:15:06 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:15:07.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:15:06 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 13640 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:15:07.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:15:07 smithi115 conmon[90393]: 2022-05-14T03:15:07.583+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:15:07.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:15:07 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:15:07.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:15:07 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:15:08.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:15:07 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:15:08.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:15:07 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:15:09.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:15:08 smithi047 ceph-mon[77147]: pgmap v7172: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:15:09.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:15:08 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:15:09.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:15:08 smithi115 conmon[90393]: 2022-05-14T03:15:08.621+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:15:09.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:15:08 smithi115 ceph-mon[78593]: pgmap v7172: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:15:09.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:15:08 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:15:10.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:15:09 smithi115 conmon[90393]: 2022-05-14T03:15:09.659+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:15:11.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:15:10 smithi047 ceph-mon[77147]: pgmap v7173: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:15:11.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:15:10 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:15:11.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:15:10 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:15:11.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:15:10 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:15:11.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:15:10 smithi115 conmon[90393]: 2022-05-14T03:15:10.693+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:15:11.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:15:10 smithi115 ceph-mon[78593]: pgmap v7173: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:15:11.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:15:10 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:15:11.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:15:10 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:15:11.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:15:10 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:15:12.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:15:11 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:15:12.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:15:11 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 13645 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:15:12.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:15:11 smithi115 conmon[90393]: 2022-05-14T03:15:11.717+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:15:12.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:15:11 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:15:12.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:15:11 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 13645 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:15:13.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:15:12 smithi047 ceph-mon[77147]: pgmap v7174: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:15:13.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:15:12 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:15:13.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:15:12 smithi115 conmon[90393]: 2022-05-14T03:15:12.723+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:15:13.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:15:12 smithi115 ceph-mon[78593]: pgmap v7174: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:15:13.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:15:12 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:15:14.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:15:13 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:15:14.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:15:13 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:15:14.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:15:13 smithi115 conmon[90393]: 2022-05-14T03:15:13.724+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:15:15.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:15:14 smithi047 ceph-mon[77147]: pgmap v7175: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:15:15.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:15:14 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:15:15.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:15:14 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:15:15.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:15:14 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:15:15.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:15:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T03:15:15.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:15:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T03:15:15.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:15:14 smithi115 conmon[90393]: 2022-05-14T03:15:14.706+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:15:15.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:15:14 smithi115 ceph-mon[78593]: pgmap v7175: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:15:15.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:15:14 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:15:15.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:15:14 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:15:15.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:15:14 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:15:15.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:15:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T03:15:15.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:15:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T03:15:15.725 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:15:15 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:15:15.725 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:15:15 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T03:15:15.725 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:15:15 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T03:15:15.726 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:15:15 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T03:15:15.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:15:15 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:15:15.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:15:15 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T03:15:15.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:15:15 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T03:15:15.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:15:15 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T03:15:16.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:15:15 smithi115 conmon[90393]: 2022-05-14T03:15:15.723+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:15:17.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:15:16 smithi047 ceph-mon[77147]: pgmap v7176: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:15:17.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:15:16 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:15:17.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:15:16 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 13650 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:15:17.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:15:16 smithi115 conmon[90393]: 2022-05-14T03:15:16.722+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:15:17.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:15:16 smithi115 ceph-mon[78593]: pgmap v7176: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:15:17.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:15:16 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:15:17.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:15:16 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 13650 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:15:18.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:15:17 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:15:18.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:15:17 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:15:18.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:15:17 smithi115 conmon[90393]: 2022-05-14T03:15:17.710+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:15:18.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:15:18 smithi047 ceph-mon[77147]: pgmap v7177: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:15:18.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:15:18 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:15:18.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:15:18 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:15:18.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:15:18 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:15:18.793 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:15:18 smithi047 conmon[78521]: 2022-05-14T03:15:18.511+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:15:18.793 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:15:18 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T03:15:18.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:15:18 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:15:18.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:15:18 smithi047 conmon[78521]: yield 2022-05-14T03:15:18.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:15:18 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:15:18.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:15:18 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:15:18.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:15:18 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:15:18.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:15:18 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T03:15:18.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:15:18 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:15:18.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:15:18 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T03:15:18.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:15:18 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:15:18.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:15:18 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T03:15:18.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:15:18 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:15:18.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:15:18 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T03:15:18.797 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:15:18 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:15:18.797 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:15:18 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T03:15:18.797 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:15:18 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:15:18.797 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:15:18 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:15:18.797 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:15:18 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:15:19.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:15:18 smithi115 conmon[90393]: 2022-05-14T03:15:18.683+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:15:19.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:15:18 smithi115 ceph-mon[78593]: pgmap v7177: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:15:19.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:15:18 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:15:19.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:15:18 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:15:19.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:15:18 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:15:20.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:15:19 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:15:20.042 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T03:15:20.042 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:15:20.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T03:15:20.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:15:20.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:15:20.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:15:20.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T03:15:20.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:15:20.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T03:15:20.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:15:20.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T03:15:20.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:15:20.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T03:15:20.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:15:20.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T03:15:20.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:15:20.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:15:20.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:15:20.046 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:15:19 smithi047 ceph-mon[77147]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T03:15:20.046 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:15:19 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:15:20.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:15:19 smithi115 conmon[90393]: 2022-05-14T03:15:19.657+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:15:20.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:15:19 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:15:20.109 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T03:15:20.109 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:15:20.109 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T03:15:20.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:15:20.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:15:20.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:15:20.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T03:15:20.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:15:20.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T03:15:20.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:15:20.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T03:15:20.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:15:20.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T03:15:20.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:15:20.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T03:15:20.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:15:20.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:15:20.113 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:15:20.113 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:15:19 smithi115 ceph-mon[78593]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T03:15:20.113 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:15:19 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:15:21.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:15:20 smithi047 ceph-mon[77147]: pgmap v7178: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:15:21.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:15:20 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:15:21.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:15:20 smithi115 conmon[90393]: 2022-05-14T03:15:20.635+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:15:21.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:15:20 smithi115 ceph-mon[78593]: pgmap v7178: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:15:21.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:15:20 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:15:22.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:15:21 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:15:22.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:15:21 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 13655 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:15:22.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:15:21 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:15:22.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:15:21 smithi047 conmon[78521]: 2022-05-14T03:15:21.583+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:15:22.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:15:21 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T03:15:22.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:15:21 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:15:22.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:15:21 smithi047 conmon[78521]: yield 2022-05-14T03:15:22.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:15:21 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:15:22.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:15:21 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:15:22.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:15:21 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:15:22.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:15:21 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T03:15:22.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:15:21 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:15:22.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:15:21 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T03:15:22.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:15:21 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:15:22.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:15:21 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T03:15:22.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:15:21 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:15:22.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:15:21 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T03:15:22.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:15:21 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:15:22.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:15:21 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T03:15:22.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:15:21 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:15:22.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:15:21 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:15:22.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:15:21 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:15:22.048 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:15:21 smithi047 conmon[78521]: 2022-05-14T03:15:21.584+0000 7f5341f7c700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T03:15:22.048 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:15:21 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T03:15:22.048 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:15:21 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:15:22.048 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:15:21 smithi047 conmon[78521]: yield 2022-05-14T03:15:22.048 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:15:21 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:15:22.049 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:15:21 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:15:22.049 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:15:21 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:15:22.049 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:15:21 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T03:15:22.049 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:15:21 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:15:22.050 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:15:21 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T03:15:22.050 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:15:21 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:15:22.050 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:15:21 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T03:15:22.050 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:15:21 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:15:22.050 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:15:21 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T03:15:22.051 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:15:21 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:15:22.051 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:15:21 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T03:15:22.051 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:15:21 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:15:22.051 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:15:21 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:15:22.052 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:15:21 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:15:22.052 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:15:21 smithi047 conmon[78521]: 2022-05-14T03:15:22.052 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:15:21 smithi047 conmon[78521]: During handling of the above exception, another exception occurred: 2022-05-14T03:15:22.052 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:15:21 smithi047 conmon[78521]: 2022-05-14T03:15:22.052 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:15:21 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T03:15:22.053 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:15:21 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T03:15:22.053 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:15:21 smithi047 conmon[78521]: return f(*arg) 2022-05-14T03:15:22.053 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:15:21 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T03:15:22.053 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:15:21 smithi047 conmon[78521]: self._write_client_files(client_files, host) 2022-05-14T03:15:22.054 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:15:21 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T03:15:22.054 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:15:21 smithi047 conmon[78521]: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T03:15:22.054 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:15:21 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T03:15:22.054 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:15:21 smithi047 conmon[78521]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T03:15:22.054 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:15:21 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T03:15:22.055 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:15:21 smithi047 conmon[78521]: return self.event_loop.get_result(coro) 2022-05-14T03:15:22.055 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:15:21 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T03:15:22.055 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:15:21 smithi047 conmon[78521]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T03:15:22.055 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:15:21 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T03:15:22.056 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:15:21 smithi047 conmon[78521]: return self.__get_result() 2022-05-14T03:15:22.056 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:15:21 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T03:15:22.056 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:15:21 smithi047 conmon[78521]: raise self._exception 2022-05-14T03:15:22.056 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:15:21 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T03:15:22.056 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:15:21 smithi047 conmon[78521]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T03:15:22.057 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:15:21 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T03:15:22.057 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:15:21 smithi047 conmon[78521]: conn = await self._remote_connection(host, addr) 2022-05-14T03:15:22.057 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:15:21 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T03:15:22.057 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:15:21 smithi047 conmon[78521]: raise 2022-05-14T03:15:22.058 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:15:21 smithi047 conmon[78521]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T03:15:22.058 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:15:21 smithi047 conmon[78521]: self.gen.throw(type, value, traceback) 2022-05-14T03:15:22.058 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:15:21 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T03:15:22.058 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:15:21 smithi047 conmon[78521]: raise OrchestratorError(msg) 2022-05-14T03:15:22.058 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:15:21 smithi047 conmon[78521]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:15:22.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:15:21 smithi115 conmon[90393]: 2022-05-14T03:15:21.607+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:15:22.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:15:21 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:15:22.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:15:21 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 13655 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:15:22.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:15:21 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:15:22.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:15:22 smithi115 conmon[90393]: 2022-05-14T03:15:22.583+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:15:22.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:15:22 smithi115 ceph-mon[78593]: pgmap v7179: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:15:22.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:15:22 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:15:22.859 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T03:15:22.860 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:15:22.860 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T03:15:22.860 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:15:22.860 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:15:22.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:15:22.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T03:15:22.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:15:22.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T03:15:22.863 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:15:22.863 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T03:15:22.864 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:15:22.864 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T03:15:22.865 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:15:22.865 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T03:15:22.866 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:15:22.866 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:15:22.866 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:15:22.866 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:15:22 smithi115 ceph-mon[78593]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T03:15:22.867 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T03:15:22.867 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:15:22.867 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T03:15:22.867 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:15:22.867 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:15:22.868 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:15:22.868 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T03:15:22.868 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:15:22.868 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T03:15:22.869 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:15:22.869 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T03:15:22.869 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:15:22.869 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T03:15:22.869 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:15:22.870 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T03:15:22.870 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:15:22.870 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:15:22.870 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:15:22.870 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T03:15:22.871 INFO:journalctl@ceph.mon.c.smithi115.stdout: During handling of the above exception, another exception occurred: 2022-05-14T03:15:22.871 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T03:15:22.871 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T03:15:22.871 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T03:15:22.871 INFO:journalctl@ceph.mon.c.smithi115.stdout: return f(*arg) 2022-05-14T03:15:22.872 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T03:15:22.872 INFO:journalctl@ceph.mon.c.smithi115.stdout: self._write_client_files(client_files, host) 2022-05-14T03:15:22.872 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T03:15:22.872 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T03:15:22.872 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T03:15:22.873 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T03:15:22.873 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T03:15:22.873 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.event_loop.get_result(coro) 2022-05-14T03:15:22.874 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T03:15:22.874 INFO:journalctl@ceph.mon.c.smithi115.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T03:15:22.874 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T03:15:22.874 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.__get_result() 2022-05-14T03:15:22.875 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T03:15:22.875 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise self._exception 2022-05-14T03:15:22.875 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T03:15:22.876 INFO:journalctl@ceph.mon.c.smithi115.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T03:15:22.876 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T03:15:22.876 INFO:journalctl@ceph.mon.c.smithi115.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T03:15:22.876 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T03:15:22.877 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise 2022-05-14T03:15:22.877 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T03:15:22.877 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.gen.throw(type, value, traceback) 2022-05-14T03:15:22.877 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T03:15:22.877 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OrchestratorError(msg) 2022-05-14T03:15:22.878 INFO:journalctl@ceph.mon.c.smithi115.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:15:22.878 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:15:22 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:15:22.878 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:15:22 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:15:22.878 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:15:22 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:15:23.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:15:22 smithi047 ceph-mon[77147]: pgmap v7179: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:15:23.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:15:22 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:15:23.042 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T03:15:23.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:15:23.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T03:15:23.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:15:23.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:15:23.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:15:23.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T03:15:23.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:15:23.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T03:15:23.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:15:23.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T03:15:23.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:15:23.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T03:15:23.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:15:23.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T03:15:23.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:15:23.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:15:23.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:15:23.046 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:15:22 smithi047 ceph-mon[77147]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T03:15:23.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T03:15:23.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:15:23.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T03:15:23.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:15:23.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:15:23.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:15:23.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T03:15:23.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:15:23.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T03:15:23.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:15:23.049 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T03:15:23.049 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:15:23.049 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T03:15:23.049 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:15:23.049 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T03:15:23.050 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:15:23.050 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:15:23.050 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:15:23.050 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T03:15:23.050 INFO:journalctl@ceph.mon.b.smithi047.stdout: During handling of the above exception, another exception occurred: 2022-05-14T03:15:23.051 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T03:15:23.051 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T03:15:23.051 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T03:15:23.051 INFO:journalctl@ceph.mon.b.smithi047.stdout: return f(*arg) 2022-05-14T03:15:23.052 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T03:15:23.052 INFO:journalctl@ceph.mon.b.smithi047.stdout: self._write_client_files(client_files, host) 2022-05-14T03:15:23.052 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T03:15:23.052 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T03:15:23.052 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T03:15:23.053 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T03:15:23.053 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T03:15:23.053 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.event_loop.get_result(coro) 2022-05-14T03:15:23.053 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T03:15:23.053 INFO:journalctl@ceph.mon.b.smithi047.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T03:15:23.054 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T03:15:23.054 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.__get_result() 2022-05-14T03:15:23.054 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T03:15:23.054 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise self._exception 2022-05-14T03:15:23.054 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T03:15:23.055 INFO:journalctl@ceph.mon.b.smithi047.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T03:15:23.055 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T03:15:23.055 INFO:journalctl@ceph.mon.b.smithi047.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T03:15:23.055 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T03:15:23.055 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise 2022-05-14T03:15:23.056 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T03:15:23.056 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.gen.throw(type, value, traceback) 2022-05-14T03:15:23.056 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T03:15:23.056 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OrchestratorError(msg) 2022-05-14T03:15:23.057 INFO:journalctl@ceph.mon.b.smithi047.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:15:23.057 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:15:22 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:15:23.057 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:15:22 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:15:23.057 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:15:22 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:15:24.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:15:23 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:15:24.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:15:23 smithi115 conmon[90393]: 2022-05-14T03:15:23.613+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:15:24.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:15:23 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:15:25.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:15:24 smithi047 ceph-mon[77147]: pgmap v7180: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:15:25.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:15:24 smithi115 conmon[90393]: 2022-05-14T03:15:24.638+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:15:25.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:15:24 smithi115 ceph-mon[78593]: pgmap v7180: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:15:26.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:15:25 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:15:26.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:15:25 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:15:26.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:15:25 smithi115 conmon[90393]: 2022-05-14T03:15:25.624+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:15:26.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:15:25 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:15:26.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:15:25 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:15:26.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:15:26 smithi115 conmon[90393]: 2022-05-14T03:15:26.582+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:15:26.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:15:26 smithi115 ceph-mon[78593]: pgmap v7181: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:15:26.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:15:26 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 13660 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:15:26.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:15:26 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:15:27.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:15:26 smithi047 ceph-mon[77147]: pgmap v7181: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:15:27.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:15:26 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 13660 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:15:27.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:15:26 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:15:28.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:15:27 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:15:28.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:15:27 smithi115 conmon[90393]: 2022-05-14T03:15:27.615+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:15:28.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:15:27 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:15:29.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:15:28 smithi047 ceph-mon[77147]: pgmap v7182: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:15:29.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:15:28 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:15:29.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:15:28 smithi115 conmon[90393]: 2022-05-14T03:15:28.630+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:15:29.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:15:28 smithi115 ceph-mon[78593]: pgmap v7182: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:15:29.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:15:28 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:15:30.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:15:29 smithi115 conmon[90393]: 2022-05-14T03:15:29.647+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:15:31.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:15:30 smithi047 ceph-mon[77147]: pgmap v7183: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:15:31.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:15:30 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:15:31.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:15:30 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:15:31.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:15:30 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:15:31.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:15:30 smithi115 conmon[90393]: 2022-05-14T03:15:30.672+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:15:31.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:15:30 smithi115 ceph-mon[78593]: pgmap v7183: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:15:31.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:15:30 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:15:31.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:15:30 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:15:31.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:15:30 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:15:32.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:15:31 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:15:32.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:15:31 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 13665 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:15:32.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:15:31 smithi115 conmon[90393]: 2022-05-14T03:15:31.707+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:15:32.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:15:31 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:15:32.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:15:31 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 13665 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:15:33.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:15:32 smithi047 ceph-mon[77147]: pgmap v7184: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:15:33.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:15:32 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:15:33.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:15:32 smithi115 conmon[90393]: 2022-05-14T03:15:32.752+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:15:33.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:15:32 smithi115 ceph-mon[78593]: pgmap v7184: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:15:33.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:15:32 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:15:34.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:15:33 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:15:34.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:15:33 smithi115 conmon[90393]: 2022-05-14T03:15:33.774+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:15:34.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:15:33 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:15:35.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:15:34 smithi047 ceph-mon[77147]: pgmap v7185: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:15:35.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:15:34 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:15:35.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:15:34 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:15:35.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:15:34 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:15:35.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:15:34 smithi115 conmon[90393]: 2022-05-14T03:15:34.810+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:15:35.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:15:34 smithi115 ceph-mon[78593]: pgmap v7185: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:15:35.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:15:34 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:15:35.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:15:34 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:15:35.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:15:34 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:15:36.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:15:35 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:15:36.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:15:35 smithi115 conmon[90393]: 2022-05-14T03:15:35.859+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:15:36.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:15:35 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:15:37.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:15:36 smithi047 ceph-mon[77147]: pgmap v7186: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:15:37.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:15:36 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:15:37.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:15:36 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 13670 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:15:37.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:15:36 smithi115 conmon[90393]: 2022-05-14T03:15:36.845+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:15:37.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:15:36 smithi115 ceph-mon[78593]: pgmap v7186: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:15:37.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:15:36 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:15:37.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:15:36 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 13670 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:15:38.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:15:37 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:15:38.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:15:37 smithi115 conmon[90393]: 2022-05-14T03:15:37.806+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:15:38.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:15:37 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:15:39.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:15:38 smithi047 ceph-mon[77147]: pgmap v7187: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:15:39.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:15:38 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:15:39.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:15:38 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:15:39.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:15:38 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:15:39.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:15:38 smithi115 conmon[90393]: 2022-05-14T03:15:38.797+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:15:39.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:15:38 smithi115 ceph-mon[78593]: pgmap v7187: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:15:39.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:15:38 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:15:39.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:15:38 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:15:39.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:15:38 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:15:40.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:15:39 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:15:40.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:15:39 smithi115 conmon[90393]: 2022-05-14T03:15:39.844+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:15:40.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:15:39 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:15:41.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:15:40 smithi047 ceph-mon[77147]: pgmap v7188: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:15:41.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:15:40 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:15:41.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:15:40 smithi115 conmon[90393]: 2022-05-14T03:15:40.883+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:15:41.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:15:40 smithi115 ceph-mon[78593]: pgmap v7188: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:15:41.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:15:40 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:15:42.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:15:41 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:15:42.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:15:41 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 13675 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:15:42.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:15:41 smithi115 conmon[90393]: 2022-05-14T03:15:41.919+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:15:42.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:15:41 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:15:42.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:15:41 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 13675 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:15:43.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:15:42 smithi047 ceph-mon[77147]: pgmap v7189: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:15:43.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:15:42 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:15:43.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:15:42 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:15:43.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:15:42 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:15:43.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:15:42 smithi115 conmon[90393]: 2022-05-14T03:15:42.936+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:15:43.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:15:42 smithi115 ceph-mon[78593]: pgmap v7189: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:15:43.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:15:42 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:15:43.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:15:42 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:15:43.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:15:42 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:15:44.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:15:43 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:15:44.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:15:43 smithi115 conmon[90393]: 2022-05-14T03:15:43.967+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:15:44.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:15:43 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:15:45.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:15:44 smithi047 ceph-mon[77147]: pgmap v7190: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:15:45.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:15:44 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:15:45.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:15:45 smithi115 conmon[90393]: 2022-05-14T03:15:45.000+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:15:45.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:15:44 smithi115 ceph-mon[78593]: pgmap v7190: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:15:45.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:15:44 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:15:46.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:15:45 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:15:46.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:15:45 smithi115 conmon[90393]: 2022-05-14T03:15:45.967+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:15:46.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:15:45 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:15:47.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:15:46 smithi047 ceph-mon[77147]: pgmap v7191: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:15:47.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:15:46 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:15:47.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:15:46 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 13680 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:15:47.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:15:47 smithi115 conmon[90393]: 2022-05-14T03:15:47.005+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:15:47.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:15:46 smithi115 ceph-mon[78593]: pgmap v7191: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:15:47.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:15:46 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:15:47.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:15:46 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 13680 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:15:48.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:15:47 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:15:48.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:15:47 smithi115 conmon[90393]: 2022-05-14T03:15:47.976+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:15:48.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:15:47 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:15:49.022 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:15:48 smithi115 ceph-mon[78593]: pgmap v7192: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:15:49.023 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:15:48 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:15:49.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:15:48 smithi047 ceph-mon[77147]: pgmap v7192: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:15:49.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:15:48 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:15:49.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:15:49 smithi115 conmon[90393]: 2022-05-14T03:15:49.021+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:15:50.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:15:49 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:15:50.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:15:50 smithi115 conmon[90393]: 2022-05-14T03:15:50.009+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:15:50.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:15:49 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:15:51.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:15:50 smithi047 ceph-mon[77147]: pgmap v7193: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:15:51.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:15:50 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:15:51.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:15:50 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:15:51.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:15:50 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:15:51.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:15:51 smithi115 conmon[90393]: 2022-05-14T03:15:51.010+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:15:51.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:15:50 smithi115 ceph-mon[78593]: pgmap v7193: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:15:51.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:15:50 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:15:51.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:15:50 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:15:51.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:15:50 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:15:52.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:15:51 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:15:52.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:15:51 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 13685 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:15:52.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:15:51 smithi115 conmon[90393]: 2022-05-14T03:15:51.996+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:15:52.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:15:51 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:15:52.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:15:51 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 13685 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:15:53.032 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:15:52 smithi115 ceph-mon[78593]: pgmap v7194: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:15:53.033 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:15:52 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:15:53.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:15:52 smithi047 ceph-mon[77147]: pgmap v7194: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:15:53.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:15:52 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:15:53.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:15:53 smithi115 conmon[90393]: 2022-05-14T03:15:53.031+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:15:54.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:15:53 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:15:54.070 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:15:53 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:15:54.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:15:54 smithi115 conmon[90393]: 2022-05-14T03:15:54.069+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:15:55.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:15:54 smithi047 ceph-mon[77147]: pgmap v7195: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:15:55.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:15:54 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:15:55.045 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:15:54 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:15:55.045 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:15:54 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:15:55.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:15:55 smithi115 conmon[90393]: 2022-05-14T03:15:55.032+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:15:55.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:15:54 smithi115 ceph-mon[78593]: pgmap v7195: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:15:55.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:15:54 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:15:55.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:15:54 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:15:55.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:15:54 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:15:56.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:15:55 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:15:56.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:15:56 smithi115 conmon[90393]: 2022-05-14T03:15:56.024+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:15:56.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:15:55 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:15:57.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:15:56 smithi047 ceph-mon[77147]: pgmap v7196: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:15:57.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:15:56 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:15:57.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:15:56 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 13690 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:15:57.073 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:15:56 smithi115 ceph-mon[78593]: pgmap v7196: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:15:57.073 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:15:56 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:15:57.074 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:15:56 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 13690 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:15:57.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:15:57 smithi115 conmon[90393]: 2022-05-14T03:15:57.071+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:15:58.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:15:58 smithi115 conmon[90393]: 2022-05-14T03:15:58.024+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:15:58.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:15:57 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:15:58.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:15:57 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:15:59.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:15:59 smithi115 conmon[90393]: 2022-05-14T03:15:59.002+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:15:59.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:15:58 smithi115 ceph-mon[78593]: pgmap v7197: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:15:59.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:15:58 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:15:59.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:15:58 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:15:59.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:15:58 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:15:59.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:15:58 smithi047 ceph-mon[77147]: pgmap v7197: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:15:59.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:15:58 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:15:59.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:15:58 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:15:59.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:15:58 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:16:00.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:16:00 smithi115 conmon[90393]: 2022-05-14T03:16:00.023+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:16:00.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:15:59 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:16:00.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:15:59 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:16:01.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:16:01 smithi115 conmon[90393]: 2022-05-14T03:16:01.040+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:16:01.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:16:00 smithi115 ceph-mon[78593]: pgmap v7198: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:16:01.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:16:00 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:16:01.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:16:00 smithi047 ceph-mon[77147]: pgmap v7198: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:16:01.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:16:00 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:16:02.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:16:02 smithi115 conmon[90393]: 2022-05-14T03:16:02.007+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:16:02.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:16:01 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:16:02.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:16:01 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 13695 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:16:02.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:16:01 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:16:02.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:16:01 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 13695 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:16:03.112 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:16:03 smithi115 conmon[90393]: 2022-05-14T03:16:03.025+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:16:03.112 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:16:02 smithi115 ceph-mon[78593]: pgmap v7199: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:16:03.113 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:16:02 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:16:03.113 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:16:02 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:16:03.113 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:16:02 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:16:03.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:16:02 smithi047 ceph-mon[77147]: pgmap v7199: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:16:03.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:16:02 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:16:03.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:16:02 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:16:03.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:16:02 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:16:04.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:16:03 smithi115 conmon[90393]: 2022-05-14T03:16:03.991+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:16:04.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:16:03 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:16:04.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:16:03 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:16:05.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:16:04 smithi115 conmon[90393]: 2022-05-14T03:16:04.978+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:16:05.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:16:04 smithi115 ceph-mon[78593]: pgmap v7200: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:16:05.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:16:04 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:16:05.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:16:04 smithi047 ceph-mon[77147]: pgmap v7200: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:16:05.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:16:04 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:16:06.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:16:05 smithi115 conmon[90393]: 2022-05-14T03:16:05.981+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:16:06.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:16:05 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:16:06.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:16:05 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:16:07.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:16:07 smithi115 conmon[90393]: 2022-05-14T03:16:06.999+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:16:07.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:16:06 smithi115 ceph-mon[78593]: pgmap v7201: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:16:07.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:16:06 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:16:07.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:16:06 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 13700 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:16:07.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:16:06 smithi047 ceph-mon[77147]: pgmap v7201: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:16:07.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:16:06 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:16:07.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:16:06 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 13700 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:16:08.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:16:08 smithi115 conmon[90393]: 2022-05-14T03:16:08.036+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:16:08.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:16:07 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:16:08.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:16:07 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:16:09.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:16:09 smithi115 conmon[90393]: 2022-05-14T03:16:09.005+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:16:09.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:16:08 smithi115 ceph-mon[78593]: pgmap v7202: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:16:09.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:16:08 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:16:09.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:16:08 smithi047 ceph-mon[77147]: pgmap v7202: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:16:09.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:16:08 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:16:10.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:16:10 smithi115 conmon[90393]: 2022-05-14T03:16:10.049+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:16:10.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:16:09 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:16:10.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:16:09 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:16:11.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:16:11 smithi115 conmon[90393]: 2022-05-14T03:16:11.045+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:16:11.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:16:10 smithi115 ceph-mon[78593]: pgmap v7203: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:16:11.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:16:10 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:16:11.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:16:10 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:16:11.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:16:10 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:16:11.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:16:10 smithi047 ceph-mon[77147]: pgmap v7203: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:16:11.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:16:10 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:16:11.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:16:10 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:16:11.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:16:10 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:16:12.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:16:12 smithi115 conmon[90393]: 2022-05-14T03:16:12.081+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:16:12.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:16:11 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:16:12.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:16:11 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 13705 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:16:12.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:16:11 smithi115 ceph-mon[78593]: pgmap v7204: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:16:12.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:16:11 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:16:12.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:16:11 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 13705 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:16:12.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:16:11 smithi047 ceph-mon[77147]: pgmap v7204: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:16:13.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:16:13 smithi115 conmon[90393]: 2022-05-14T03:16:13.064+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:16:13.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:16:13 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:16:13.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:16:13 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:16:14.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:16:14 smithi115 conmon[90393]: 2022-05-14T03:16:14.027+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:16:14.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:16:14 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:16:14.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:16:14 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:16:14.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:16:14 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:16:14.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:16:14 smithi047 ceph-mon[77147]: pgmap v7205: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:16:14.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:16:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T03:16:14.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:16:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T03:16:14.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:16:14 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:16:14.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:16:14 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:16:14.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:16:14 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:16:14.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:16:14 smithi115 ceph-mon[78593]: pgmap v7205: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:16:14.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:16:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T03:16:14.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:16:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T03:16:15.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:16:14 smithi115 conmon[90393]: 2022-05-14T03:16:14.980+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:16:15.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:16:15 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:16:15.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:16:15 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:16:16.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:16:15 smithi115 conmon[90393]: 2022-05-14T03:16:15.968+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:16:16.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:16:16 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:16:16.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:16:16 smithi047 ceph-mon[77147]: pgmap v7206: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:16:16.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:16:16 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:16:16.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:16:16 smithi115 ceph-mon[78593]: pgmap v7206: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:16:17.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:16:16 smithi115 conmon[90393]: 2022-05-14T03:16:16.944+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:16:17.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:16:17 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:16:17.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:16:17 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 13710 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:16:17.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:16:17 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:16:17.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:16:17 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 13710 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:16:18.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:16:17 smithi115 conmon[90393]: 2022-05-14T03:16:17.950+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:16:18.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:16:18 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:16:18.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:16:18 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:16:18.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:16:18 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:16:18.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:16:18 smithi047 ceph-mon[77147]: pgmap v7207: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:16:18.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:16:18 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:16:18.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:16:18 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:16:18.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:16:18 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:16:18.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:16:18 smithi115 ceph-mon[78593]: pgmap v7207: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:16:19.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:16:18 smithi115 conmon[90393]: 2022-05-14T03:16:18.986+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:16:19.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:16:19 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:16:19.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:16:19 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:16:20.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:16:19 smithi115 conmon[90393]: 2022-05-14T03:16:19.955+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:16:20.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:16:20 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:16:20.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:16:20 smithi047 ceph-mon[77147]: pgmap v7208: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:16:20.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:16:20 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:16:20.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:16:20 smithi115 ceph-mon[78593]: pgmap v7208: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:16:21.361 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:16:20 smithi115 conmon[90393]: 2022-05-14T03:16:20.943+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:16:21.684 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:16:21 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:16:21.687 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:16:21 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:16:22.041 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:16:21 smithi115 conmon[90393]: 2022-05-14T03:16:21.925+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:16:22.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:16:22 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:16:22.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:16:22 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:16:22.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:16:22 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:16:22.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:16:22 smithi047 ceph-mon[77147]: pgmap v7209: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:16:22.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:16:22 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T03:16:22.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:16:22 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T03:16:22.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:16:22 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T03:16:22.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:16:22 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' 2022-05-14T03:16:22.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:16:22 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' 2022-05-14T03:16:22.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:16:22 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:16:22.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:16:22 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:16:22.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:16:22 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:16:22.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:16:22 smithi115 ceph-mon[78593]: pgmap v7209: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:16:22.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:16:22 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T03:16:22.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:16:22 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T03:16:22.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:16:22 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T03:16:22.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:16:22 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' 2022-05-14T03:16:22.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:16:22 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' 2022-05-14T03:16:23.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:16:22 smithi115 conmon[90393]: 2022-05-14T03:16:22.943+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:16:23.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:16:23 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:16:23.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:16:23 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:16:24.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:16:23 smithi115 conmon[90393]: 2022-05-14T03:16:23.920+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:16:24.681 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:16:24 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:16:24.681 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:16:24 smithi047 ceph-mon[77147]: pgmap v7210: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:16:24.687 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:16:24 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:16:24.687 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:16:24 smithi115 ceph-mon[78593]: pgmap v7210: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:16:25.042 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:16:24 smithi047 conmon[78521]: 2022-05-14T03:16:24.690+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:16:25.042 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:16:24 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T03:16:25.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:16:24 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:16:25.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:16:24 smithi047 conmon[78521]: yield 2022-05-14T03:16:25.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:16:24 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:16:25.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:16:24 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:16:25.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:16:24 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:16:25.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:16:24 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T03:16:25.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:16:24 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:16:25.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:16:24 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T03:16:25.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:16:24 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:16:25.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:16:24 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T03:16:25.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:16:24 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:16:25.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:16:24 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T03:16:25.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:16:24 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:16:25.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:16:24 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T03:16:25.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:16:24 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:16:25.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:16:24 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:16:25.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:16:24 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:16:25.104 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:16:24 smithi115 conmon[90393]: 2022-05-14T03:16:24.889+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:16:25.874 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:16:25 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:16:25.875 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:16:25 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' 2022-05-14T03:16:25.875 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:16:25 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' 2022-05-14T03:16:25.875 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:16:25 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:16:25.876 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T03:16:25.876 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:16:25.876 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T03:16:25.876 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:16:25.876 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:16:25.877 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:16:25.877 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T03:16:25.877 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:16:25.877 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T03:16:25.878 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:16:25.878 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T03:16:25.878 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:16:25.878 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T03:16:25.878 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:16:25.879 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T03:16:25.879 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:16:25.879 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:16:25.879 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:16:25.879 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:16:25 smithi115 ceph-mon[78593]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T03:16:25.883 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:16:25 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:16:25.883 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:16:25 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' 2022-05-14T03:16:25.883 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:16:25 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' 2022-05-14T03:16:25.884 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:16:25 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:16:25.884 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T03:16:25.885 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:16:25.885 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T03:16:25.885 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:16:25.886 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:16:25.886 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:16:25.887 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T03:16:25.887 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:16:25.887 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T03:16:25.888 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:16:25.888 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T03:16:25.888 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:16:25.889 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T03:16:25.889 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:16:25.890 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T03:16:25.890 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:16:25.890 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:16:25.891 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:16:25.892 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:16:25 smithi047 ceph-mon[77147]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T03:16:26.165 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:16:25 smithi115 conmon[90393]: 2022-05-14T03:16:25.931+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:16:26.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:16:26 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:16:26.915 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:16:26 smithi115 ceph-mon[78593]: pgmap v7211: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:16:26.916 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:16:26 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 13720 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:16:27.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:16:26 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:16:27.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:16:26 smithi047 ceph-mon[77147]: pgmap v7211: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:16:27.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:16:26 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 13720 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:16:27.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:16:26 smithi115 conmon[90393]: 2022-05-14T03:16:26.981+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:16:27.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:16:27 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:16:27.793 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:16:27 smithi047 conmon[78521]: 2022-05-14T03:16:27.760+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:16:27.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:16:27 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T03:16:27.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:16:27 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:16:27.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:16:27 smithi047 conmon[78521]: yield 2022-05-14T03:16:27.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:16:27 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:16:27.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:16:27 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:16:27.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:16:27 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:16:27.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:16:27 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T03:16:27.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:16:27 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:16:27.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:16:27 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T03:16:27.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:16:27 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:16:27.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:16:27 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T03:16:27.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:16:27 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:16:27.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:16:27 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T03:16:27.797 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:16:27 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:16:27.797 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:16:27 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T03:16:27.797 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:16:27 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:16:27.797 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:16:27 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:16:27.797 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:16:27 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:16:27.798 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:16:27 smithi047 conmon[78521]: 2022-05-14T03:16:27.761+0000 7f5340779700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T03:16:27.798 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:16:27 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T03:16:27.798 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:16:27 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:16:27.798 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:16:27 smithi047 conmon[78521]: yield 2022-05-14T03:16:27.799 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:16:27 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:16:27.799 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:16:27 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:16:27.799 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:16:27 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:16:27.799 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:16:27 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T03:16:27.799 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:16:27 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:16:27.800 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:16:27 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T03:16:27.800 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:16:27 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:16:27.800 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:16:27 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T03:16:27.800 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:16:27 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:16:27.800 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:16:27 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T03:16:27.801 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:16:27 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:16:27.801 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:16:27 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T03:16:27.801 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:16:27 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:16:27.802 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:16:27 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:16:27.802 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:16:27 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:16:27.802 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:16:27 smithi047 conmon[78521]: 2022-05-14T03:16:27.802 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:16:27 smithi047 conmon[78521]: During handling of the above exception, another exception occurred: 2022-05-14T03:16:27.803 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:16:27 smithi047 conmon[78521]: 2022-05-14T03:16:27.803 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:16:27 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T03:16:27.803 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:16:27 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T03:16:27.803 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:16:27 smithi047 conmon[78521]: return f(*arg) 2022-05-14T03:16:27.803 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:16:27 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T03:16:27.804 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:16:27 smithi047 conmon[78521]: self._write_client_files(client_files, host) 2022-05-14T03:16:27.804 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:16:27 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T03:16:27.804 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:16:27 smithi047 conmon[78521]: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T03:16:27.804 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:16:27 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T03:16:27.805 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:16:27 smithi047 conmon[78521]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T03:16:27.805 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:16:27 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T03:16:27.805 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:16:27 smithi047 conmon[78521]: return self.event_loop.get_result(coro) 2022-05-14T03:16:27.805 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:16:27 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T03:16:27.806 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:16:27 smithi047 conmon[78521]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T03:16:27.806 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:16:27 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T03:16:27.806 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:16:27 smithi047 conmon[78521]: return self.__get_result() 2022-05-14T03:16:27.806 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:16:27 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T03:16:27.807 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:16:27 smithi047 conmon[78521]: raise self._exception 2022-05-14T03:16:27.807 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:16:27 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T03:16:27.807 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:16:27 smithi047 conmon[78521]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T03:16:27.807 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:16:27 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T03:16:27.808 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:16:27 smithi047 conmon[78521]: conn = await self._remote_connection(host, addr) 2022-05-14T03:16:27.808 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:16:27 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T03:16:27.808 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:16:27 smithi047 conmon[78521]: raise 2022-05-14T03:16:27.808 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:16:27 smithi047 conmon[78521]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T03:16:27.808 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:16:27 smithi047 conmon[78521]: self.gen.throw(type, value, traceback) 2022-05-14T03:16:27.809 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:16:27 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T03:16:27.809 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:16:27 smithi047 conmon[78521]: raise OrchestratorError(msg) 2022-05-14T03:16:27.809 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:16:27 smithi047 conmon[78521]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:16:27.817 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:16:27 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:16:28.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:16:27 smithi115 conmon[90393]: 2022-05-14T03:16:27.985+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:16:28.976 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:16:28 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:16:28.977 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:16:28 smithi115 ceph-mon[78593]: pgmap v7212: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:16:28.977 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:16:28 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:16:28.977 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T03:16:28.977 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:16:28.978 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T03:16:28.978 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:16:28.978 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:16:28.978 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:16:28.978 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T03:16:28.979 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:16:28.979 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T03:16:28.979 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:16:28.979 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T03:16:28.980 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:16:28.980 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T03:16:28.981 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:16:28.981 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T03:16:28.982 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:16:28.982 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:16:28.982 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:16:28.983 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:16:28 smithi115 ceph-mon[78593]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T03:16:28.983 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T03:16:28.983 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:16:28.984 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T03:16:28.984 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:16:28.984 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:16:28.985 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:16:28.985 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T03:16:28.985 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:16:28.986 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T03:16:28.986 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:16:28.986 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T03:16:28.987 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:16:28.987 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T03:16:28.987 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:16:28.988 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T03:16:28.988 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:16:28.989 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:16:28.989 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:16:28.989 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T03:16:28.989 INFO:journalctl@ceph.mon.c.smithi115.stdout: During handling of the above exception, another exception occurred: 2022-05-14T03:16:28.990 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T03:16:28.990 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T03:16:28.990 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T03:16:28.990 INFO:journalctl@ceph.mon.c.smithi115.stdout: return f(*arg) 2022-05-14T03:16:28.991 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T03:16:28.991 INFO:journalctl@ceph.mon.c.smithi115.stdout: self._write_client_files(client_files, host) 2022-05-14T03:16:28.991 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T03:16:28.992 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T03:16:28.992 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T03:16:28.992 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T03:16:28.993 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T03:16:28.993 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.event_loop.get_result(coro) 2022-05-14T03:16:28.993 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T03:16:28.994 INFO:journalctl@ceph.mon.c.smithi115.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T03:16:28.994 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T03:16:28.994 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.__get_result() 2022-05-14T03:16:28.995 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T03:16:28.995 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise self._exception 2022-05-14T03:16:28.995 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T03:16:28.995 INFO:journalctl@ceph.mon.c.smithi115.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T03:16:28.995 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T03:16:28.996 INFO:journalctl@ceph.mon.c.smithi115.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T03:16:28.996 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T03:16:28.996 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise 2022-05-14T03:16:28.996 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T03:16:28.996 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.gen.throw(type, value, traceback) 2022-05-14T03:16:28.997 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T03:16:28.997 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OrchestratorError(msg) 2022-05-14T03:16:28.997 INFO:journalctl@ceph.mon.c.smithi115.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:16:28.997 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:16:28 smithi115 ceph-mon[78593]: Detected new or changed devices on smithi115 2022-05-14T03:16:28.998 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:16:28 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' 2022-05-14T03:16:28.998 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:16:28 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-05-14T03:16:28.998 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:16:28 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-05-14T03:16:28.998 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:16:28 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-05-14T03:16:28.998 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:16:28 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-05-14T03:16:28.999 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:16:28 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' 2022-05-14T03:16:28.999 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:16:28 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-05-14T03:16:28.999 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:16:28 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-05-14T03:16:28.999 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:16:28 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-05-14T03:16:28.999 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:16:28 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-05-14T03:16:29.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:16:28 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:16:29.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:16:28 smithi047 ceph-mon[77147]: pgmap v7212: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:16:29.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:16:28 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:16:29.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T03:16:29.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:16:29.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T03:16:29.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:16:29.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:16:29.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:16:29.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T03:16:29.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:16:29.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T03:16:29.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:16:29.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T03:16:29.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:16:29.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T03:16:29.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:16:29.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T03:16:29.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:16:29.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:16:29.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:16:29.047 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:16:28 smithi047 ceph-mon[77147]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T03:16:29.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T03:16:29.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:16:29.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T03:16:29.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:16:29.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:16:29.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:16:29.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T03:16:29.049 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:16:29.049 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T03:16:29.049 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:16:29.049 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T03:16:29.049 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:16:29.050 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T03:16:29.050 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:16:29.050 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T03:16:29.050 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:16:29.050 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:16:29.051 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:16:29.051 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T03:16:29.051 INFO:journalctl@ceph.mon.b.smithi047.stdout: During handling of the above exception, another exception occurred: 2022-05-14T03:16:29.051 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T03:16:29.051 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T03:16:29.052 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T03:16:29.052 INFO:journalctl@ceph.mon.b.smithi047.stdout: return f(*arg) 2022-05-14T03:16:29.052 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T03:16:29.052 INFO:journalctl@ceph.mon.b.smithi047.stdout: self._write_client_files(client_files, host) 2022-05-14T03:16:29.053 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T03:16:29.053 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T03:16:29.053 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T03:16:29.053 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T03:16:29.053 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T03:16:29.054 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.event_loop.get_result(coro) 2022-05-14T03:16:29.054 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T03:16:29.054 INFO:journalctl@ceph.mon.b.smithi047.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T03:16:29.054 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T03:16:29.054 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.__get_result() 2022-05-14T03:16:29.055 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T03:16:29.055 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise self._exception 2022-05-14T03:16:29.055 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T03:16:29.055 INFO:journalctl@ceph.mon.b.smithi047.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T03:16:29.056 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T03:16:29.056 INFO:journalctl@ceph.mon.b.smithi047.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T03:16:29.056 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T03:16:29.056 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise 2022-05-14T03:16:29.057 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T03:16:29.057 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.gen.throw(type, value, traceback) 2022-05-14T03:16:29.058 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T03:16:29.058 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OrchestratorError(msg) 2022-05-14T03:16:29.058 INFO:journalctl@ceph.mon.b.smithi047.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:16:29.059 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:16:28 smithi047 ceph-mon[77147]: Detected new or changed devices on smithi115 2022-05-14T03:16:29.059 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:16:28 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' 2022-05-14T03:16:29.060 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:16:28 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-05-14T03:16:29.060 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:16:28 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-05-14T03:16:29.060 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:16:28 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-05-14T03:16:29.061 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:16:28 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-05-14T03:16:29.061 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:16:28 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' 2022-05-14T03:16:29.062 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:16:28 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-05-14T03:16:29.062 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:16:28 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-05-14T03:16:29.062 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:16:28 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-05-14T03:16:29.062 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:16:28 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-05-14T03:16:29.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:16:28 smithi115 conmon[90393]: 2022-05-14T03:16:28.974+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:16:30.017 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:16:29 smithi115 ceph-mon[78593]: Adjusting osd_memory_target on smithi115 to 197.0M 2022-05-14T03:16:30.018 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:16:29 smithi115 ceph-mon[78593]: Unable to set osd_memory_target on smithi115 to 206629785: error parsing value: Value '206629785' is below minimum 939524096 2022-05-14T03:16:30.018 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:16:29 smithi115 ceph-mon[78593]: Detected new or changed devices on smithi047 2022-05-14T03:16:30.018 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:16:29 smithi115 ceph-mon[78593]: Adjusting osd_memory_target on smithi047 to 161.2M 2022-05-14T03:16:30.018 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:16:29 smithi115 ceph-mon[78593]: Unable to set osd_memory_target on smithi047 to 169101172: error parsing value: Value '169101172' is below minimum 939524096 2022-05-14T03:16:30.018 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:16:29 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:16:30.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:16:29 smithi047 ceph-mon[77147]: Adjusting osd_memory_target on smithi115 to 197.0M 2022-05-14T03:16:30.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:16:29 smithi047 ceph-mon[77147]: Unable to set osd_memory_target on smithi115 to 206629785: error parsing value: Value '206629785' is below minimum 939524096 2022-05-14T03:16:30.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:16:29 smithi047 ceph-mon[77147]: Detected new or changed devices on smithi047 2022-05-14T03:16:30.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:16:29 smithi047 ceph-mon[77147]: Adjusting osd_memory_target on smithi047 to 161.2M 2022-05-14T03:16:30.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:16:29 smithi047 ceph-mon[77147]: Unable to set osd_memory_target on smithi047 to 169101172: error parsing value: Value '169101172' is below minimum 939524096 2022-05-14T03:16:30.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:16:29 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:16:30.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:16:30 smithi115 conmon[90393]: 2022-05-14T03:16:30.015+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:16:30.974 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:16:30 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:16:30.975 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:16:30 smithi115 ceph-mon[78593]: pgmap v7213: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:16:30.975 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:16:30 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:16:30.975 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:16:30 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:16:31.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:16:30 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:16:31.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:16:30 smithi047 ceph-mon[77147]: pgmap v7213: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:16:31.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:16:30 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:16:31.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:16:30 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:16:31.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:16:30 smithi115 conmon[90393]: 2022-05-14T03:16:30.972+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:16:31.973 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:16:31 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:16:31.974 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:16:31 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 13725 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:16:32.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:16:31 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:16:32.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:16:31 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 13725 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:16:32.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:16:31 smithi115 conmon[90393]: 2022-05-14T03:16:31.972+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:16:33.003 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:16:32 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:16:33.003 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:16:32 smithi115 ceph-mon[78593]: pgmap v7214: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:16:33.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:16:32 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:16:33.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:16:32 smithi047 ceph-mon[77147]: pgmap v7214: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:16:33.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:16:33 smithi115 conmon[90393]: 2022-05-14T03:16:33.001+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:16:33.956 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:16:33 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:16:34.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:16:33 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:16:34.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:16:33 smithi115 conmon[90393]: 2022-05-14T03:16:33.955+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:16:34.938 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:16:34 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:16:34.938 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:16:34 smithi115 ceph-mon[78593]: pgmap v7215: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:16:34.939 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:16:34 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:16:34.939 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:16:34 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:16:35.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:16:34 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:16:35.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:16:34 smithi047 ceph-mon[77147]: pgmap v7215: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:16:35.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:16:34 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:16:35.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:16:34 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:16:35.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:16:34 smithi115 conmon[90393]: 2022-05-14T03:16:34.936+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:16:35.914 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:16:35 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:16:36.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:16:35 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:16:36.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:16:35 smithi115 conmon[90393]: 2022-05-14T03:16:35.912+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:16:36.960 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:16:36 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:16:36.961 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:16:36 smithi115 ceph-mon[78593]: pgmap v7216: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:16:36.961 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:16:36 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 13730 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:16:37.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:16:36 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:16:37.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:16:36 smithi047 ceph-mon[77147]: pgmap v7216: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:16:37.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:16:36 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 13730 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:16:37.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:16:36 smithi115 conmon[90393]: 2022-05-14T03:16:36.959+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:16:37.962 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:16:37 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:16:38.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:16:37 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:16:38.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:16:37 smithi115 conmon[90393]: 2022-05-14T03:16:37.960+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:16:38.992 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:16:38 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:16:38.992 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:16:38 smithi115 ceph-mon[78593]: pgmap v7217: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:16:38.992 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:16:38 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:16:38.993 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:16:38 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:16:39.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:16:38 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:16:39.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:16:38 smithi047 ceph-mon[77147]: pgmap v7217: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:16:39.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:16:38 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:16:39.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:16:38 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:16:39.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:16:38 smithi115 conmon[90393]: 2022-05-14T03:16:38.990+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:16:39.981 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:16:39 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:16:40.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:16:39 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:16:40.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:16:39 smithi115 conmon[90393]: 2022-05-14T03:16:39.979+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:16:41.009 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:16:40 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:16:41.010 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:16:40 smithi115 ceph-mon[78593]: pgmap v7218: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:16:41.010 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:16:40 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:16:41.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:16:40 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:16:41.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:16:40 smithi047 ceph-mon[77147]: pgmap v7218: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:16:41.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:16:40 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:16:41.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:16:41 smithi115 conmon[90393]: 2022-05-14T03:16:41.008+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:16:41.997 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:16:41 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:16:41.998 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:16:41 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 13735 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:16:42.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:16:41 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:16:42.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:16:41 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 13735 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:16:42.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:16:41 smithi115 conmon[90393]: 2022-05-14T03:16:41.996+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:16:42.980 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:16:42 smithi115 ceph-mon[78593]: pgmap v7219: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:16:42.981 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:16:42 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:16:42.981 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:16:42 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:16:42.981 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:16:42 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:16:43.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:16:42 smithi047 ceph-mon[77147]: pgmap v7219: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:16:43.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:16:42 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:16:43.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:16:42 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:16:43.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:16:42 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:16:43.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:16:42 smithi115 conmon[90393]: 2022-05-14T03:16:42.979+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:16:43.962 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:16:43 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:16:44.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:16:43 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:16:44.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:16:43 smithi115 conmon[90393]: 2022-05-14T03:16:43.959+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:16:45.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:16:44 smithi047 ceph-mon[77147]: pgmap v7220: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:16:45.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:16:44 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:16:45.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:16:44 smithi115 conmon[90393]: 2022-05-14T03:16:44.916+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:16:45.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:16:44 smithi115 ceph-mon[78593]: pgmap v7220: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:16:45.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:16:44 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:16:46.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:16:45 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:16:46.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:16:45 smithi115 conmon[90393]: 2022-05-14T03:16:45.880+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:16:46.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:16:45 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:16:47.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:16:46 smithi047 ceph-mon[77147]: pgmap v7221: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:16:47.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:16:46 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:16:47.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:16:46 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 13740 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:16:47.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:16:46 smithi115 conmon[90393]: 2022-05-14T03:16:46.916+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:16:47.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:16:46 smithi115 ceph-mon[78593]: pgmap v7221: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:16:47.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:16:46 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:16:47.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:16:46 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 13740 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:16:48.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:16:47 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:16:48.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:16:47 smithi115 conmon[90393]: 2022-05-14T03:16:47.883+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:16:48.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:16:47 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:16:49.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:16:48 smithi047 ceph-mon[77147]: pgmap v7222: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:16:49.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:16:48 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:16:49.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:16:48 smithi115 conmon[90393]: 2022-05-14T03:16:48.895+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:16:49.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:16:48 smithi115 ceph-mon[78593]: pgmap v7222: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:16:49.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:16:48 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:16:50.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:16:49 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:16:50.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:16:49 smithi115 conmon[90393]: 2022-05-14T03:16:49.891+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:16:50.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:16:49 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:16:51.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:16:50 smithi047 ceph-mon[77147]: pgmap v7223: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:16:51.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:16:50 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:16:51.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:16:50 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:16:51.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:16:50 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:16:51.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:16:50 smithi115 conmon[90393]: 2022-05-14T03:16:50.884+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:16:51.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:16:50 smithi115 ceph-mon[78593]: pgmap v7223: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:16:51.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:16:50 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:16:51.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:16:50 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:16:51.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:16:50 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:16:52.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:16:51 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:16:52.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:16:51 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 13745 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:16:52.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:16:51 smithi115 conmon[90393]: 2022-05-14T03:16:51.890+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:16:52.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:16:51 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:16:52.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:16:51 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 13745 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:16:53.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:16:52 smithi047 ceph-mon[77147]: pgmap v7224: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:16:53.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:16:52 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:16:53.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:16:52 smithi115 conmon[90393]: 2022-05-14T03:16:52.855+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:16:53.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:16:52 smithi115 ceph-mon[78593]: pgmap v7224: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:16:53.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:16:52 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:16:54.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:16:53 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:16:54.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:16:53 smithi115 conmon[90393]: 2022-05-14T03:16:53.891+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:16:54.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:16:53 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:16:55.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:16:54 smithi047 ceph-mon[77147]: pgmap v7225: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:16:55.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:16:54 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:16:55.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:16:54 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:16:55.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:16:54 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:16:55.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:16:54 smithi115 conmon[90393]: 2022-05-14T03:16:54.862+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:16:55.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:16:54 smithi115 ceph-mon[78593]: pgmap v7225: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:16:55.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:16:54 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:16:55.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:16:54 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:16:55.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:16:54 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:16:56.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:16:55 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:16:56.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:16:55 smithi115 conmon[90393]: 2022-05-14T03:16:55.865+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:16:56.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:16:55 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:16:57.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:16:56 smithi047 ceph-mon[77147]: pgmap v7226: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:16:57.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:16:56 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:16:57.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:16:56 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 13750 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:16:57.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:16:56 smithi115 conmon[90393]: 2022-05-14T03:16:56.883+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:16:57.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:16:56 smithi115 ceph-mon[78593]: pgmap v7226: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:16:57.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:16:56 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:16:57.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:16:56 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 13750 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:16:58.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:16:57 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:16:58.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:16:57 smithi115 conmon[90393]: 2022-05-14T03:16:57.862+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:16:58.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:16:57 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:16:59.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:16:58 smithi047 ceph-mon[77147]: pgmap v7227: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:16:59.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:16:58 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:16:59.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:16:58 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:16:59.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:16:58 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:16:59.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:16:58 smithi115 conmon[90393]: 2022-05-14T03:16:58.853+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:16:59.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:16:58 smithi115 ceph-mon[78593]: pgmap v7227: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:16:59.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:16:58 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:16:59.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:16:58 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:16:59.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:16:58 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:17:00.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:16:59 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:17:00.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:16:59 smithi115 conmon[90393]: 2022-05-14T03:16:59.812+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:17:00.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:16:59 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:17:01.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:17:00 smithi047 ceph-mon[77147]: pgmap v7228: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:17:01.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:17:00 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:17:01.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:17:00 smithi115 conmon[90393]: 2022-05-14T03:17:00.821+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:17:01.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:17:00 smithi115 ceph-mon[78593]: pgmap v7228: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:17:01.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:17:00 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:17:02.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:17:01 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:17:02.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:17:01 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 13755 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:17:02.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:17:01 smithi115 conmon[90393]: 2022-05-14T03:17:01.864+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:17:02.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:17:01 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:17:02.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:17:01 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 13755 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:17:03.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:17:02 smithi047 ceph-mon[77147]: pgmap v7229: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:17:03.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:17:02 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:17:03.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:17:02 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:17:03.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:17:02 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:17:03.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:17:02 smithi115 conmon[90393]: 2022-05-14T03:17:02.900+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:17:03.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:17:02 smithi115 ceph-mon[78593]: pgmap v7229: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:17:03.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:17:02 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:17:03.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:17:02 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:17:03.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:17:02 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:17:04.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:17:03 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:17:04.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:17:03 smithi115 conmon[90393]: 2022-05-14T03:17:03.937+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:17:04.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:17:03 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:17:04.989 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:17:04 smithi115 ceph-mon[78593]: pgmap v7230: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:17:04.989 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:17:04 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:17:05.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:17:04 smithi047 ceph-mon[77147]: pgmap v7230: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:17:05.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:17:04 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:17:05.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:17:04 smithi115 conmon[90393]: 2022-05-14T03:17:04.987+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:17:06.007 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:17:05 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:17:06.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:17:05 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:17:06.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:17:06 smithi115 conmon[90393]: 2022-05-14T03:17:06.004+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:17:07.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:17:06 smithi047 ceph-mon[77147]: pgmap v7231: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:17:07.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:17:06 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:17:07.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:17:06 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 13760 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:17:07.049 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:17:06 smithi115 ceph-mon[78593]: pgmap v7231: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:17:07.050 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:17:06 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:17:07.050 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:17:06 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 13760 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:17:07.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:17:07 smithi115 conmon[90393]: 2022-05-14T03:17:07.047+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:17:08.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:17:07 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:17:08.043 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:17:07 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:17:08.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:17:08 smithi115 conmon[90393]: 2022-05-14T03:17:08.041+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:17:09.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:17:08 smithi047 ceph-mon[77147]: pgmap v7232: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:17:09.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:17:08 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:17:09.072 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:17:08 smithi115 ceph-mon[78593]: pgmap v7232: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:17:09.073 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:17:08 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:17:09.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:17:09 smithi115 conmon[90393]: 2022-05-14T03:17:09.070+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:17:10.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:17:09 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:17:10.087 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:17:09 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:17:10.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:17:10 smithi115 conmon[90393]: 2022-05-14T03:17:10.086+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:17:11.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:17:10 smithi047 ceph-mon[77147]: pgmap v7233: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:17:11.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:17:10 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:17:11.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:17:10 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:17:11.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:17:10 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:17:11.055 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:17:10 smithi115 ceph-mon[78593]: pgmap v7233: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:17:11.056 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:17:10 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:17:11.056 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:17:10 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:17:11.057 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:17:10 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:17:11.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:17:11 smithi115 conmon[90393]: 2022-05-14T03:17:11.053+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:17:12.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:17:11 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:17:12.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:17:11 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 13765 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:17:12.084 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:17:11 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:17:12.085 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:17:11 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 13765 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:17:12.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:17:12 smithi115 conmon[90393]: 2022-05-14T03:17:12.083+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:17:13.040 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:17:12 smithi115 ceph-mon[78593]: pgmap v7234: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:17:13.041 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:17:12 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:17:13.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:17:12 smithi047 ceph-mon[77147]: pgmap v7234: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:17:13.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:17:12 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:17:13.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:17:13 smithi115 conmon[90393]: 2022-05-14T03:17:13.038+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:17:14.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:17:13 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:17:14.066 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:17:13 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:17:14.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:17:14 smithi115 conmon[90393]: 2022-05-14T03:17:14.065+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:17:15.024 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:17:14 smithi115 ceph-mon[78593]: pgmap v7235: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:17:15.025 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:17:14 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:17:15.025 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:17:14 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:17:15.025 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:17:14 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:17:15.025 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:17:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T03:17:15.026 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:17:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T03:17:15.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:17:14 smithi047 ceph-mon[77147]: pgmap v7235: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:17:15.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:17:14 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:17:15.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:17:14 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:17:15.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:17:14 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:17:15.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:17:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T03:17:15.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:17:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T03:17:15.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:17:15 smithi115 conmon[90393]: 2022-05-14T03:17:15.022+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:17:16.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:17:15 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:17:16.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:17:16 smithi115 conmon[90393]: 2022-05-14T03:17:16.004+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:17:16.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:17:15 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:17:17.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:17:16 smithi047 ceph-mon[77147]: pgmap v7236: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:17:17.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:17:16 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:17:17.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:17:16 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 13770 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:17:17.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:17:16 smithi115 conmon[90393]: 2022-05-14T03:17:16.984+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:17:17.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:17:16 smithi115 ceph-mon[78593]: pgmap v7236: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:17:17.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:17:16 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:17:17.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:17:16 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 13770 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:17:18.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:17:17 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:17:18.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:17:17 smithi115 conmon[90393]: 2022-05-14T03:17:17.985+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:17:18.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:17:17 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:17:19.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:17:18 smithi047 ceph-mon[77147]: pgmap v7237: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:17:19.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:17:18 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:17:19.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:17:18 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:17:19.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:17:18 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:17:19.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:17:18 smithi115 conmon[90393]: 2022-05-14T03:17:18.966+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:17:19.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:17:18 smithi115 ceph-mon[78593]: pgmap v7237: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:17:19.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:17:18 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:17:19.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:17:18 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:17:19.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:17:18 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:17:20.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:17:19 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:17:20.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:17:19 smithi115 conmon[90393]: 2022-05-14T03:17:19.952+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:17:20.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:17:19 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:17:21.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:17:20 smithi047 ceph-mon[77147]: pgmap v7238: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:17:21.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:17:20 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:17:21.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:17:20 smithi115 conmon[90393]: 2022-05-14T03:17:20.933+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:17:21.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:17:20 smithi115 ceph-mon[78593]: pgmap v7238: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:17:21.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:17:20 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:17:22.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:17:21 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:17:22.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:17:21 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 13775 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:17:22.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:17:21 smithi115 conmon[90393]: 2022-05-14T03:17:21.977+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:17:22.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:17:21 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:17:22.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:17:21 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 13775 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:17:23.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:17:22 smithi047 ceph-mon[77147]: pgmap v7239: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:17:23.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:17:22 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:17:23.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:17:22 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:17:23.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:17:22 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:17:23.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:17:23 smithi115 conmon[90393]: 2022-05-14T03:17:23.005+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:17:23.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:17:22 smithi115 ceph-mon[78593]: pgmap v7239: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:17:23.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:17:22 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:17:23.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:17:22 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:17:23.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:17:22 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:17:24.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:17:24 smithi115 conmon[90393]: 2022-05-14T03:17:24.046+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:17:24.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:17:23 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:17:24.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:17:23 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:17:25.080 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:17:24 smithi115 ceph-mon[78593]: pgmap v7240: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:17:25.080 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:17:24 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:17:25.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:17:24 smithi047 ceph-mon[77147]: pgmap v7240: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:17:25.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:17:24 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:17:25.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:17:25 smithi115 conmon[90393]: 2022-05-14T03:17:25.078+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:17:26.082 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:17:25 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:17:26.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:17:25 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:17:26.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:17:26 smithi115 conmon[90393]: 2022-05-14T03:17:26.080+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:17:27.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:17:27 smithi115 conmon[90393]: 2022-05-14T03:17:27.035+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:17:27.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:17:26 smithi115 ceph-mon[78593]: pgmap v7241: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:17:27.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:17:26 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:17:27.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:17:26 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 13780 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:17:27.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:17:26 smithi047 ceph-mon[77147]: pgmap v7241: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:17:27.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:17:26 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:17:27.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:17:26 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 13780 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:17:28.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:17:27 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:17:28.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:17:28 smithi115 conmon[90393]: 2022-05-14T03:17:28.047+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:17:28.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:17:27 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:17:28.792 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:17:28 smithi047 conmon[78521]: 2022-05-14T03:17:28.368+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:17:28.792 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:17:28 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T03:17:28.792 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:17:28 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:17:28.793 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:17:28 smithi047 conmon[78521]: yield 2022-05-14T03:17:28.793 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:17:28 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:17:28.793 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:17:28 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:17:28.793 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:17:28 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:17:28.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:17:28 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T03:17:28.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:17:28 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:17:28.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:17:28 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T03:17:28.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:17:28 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:17:28.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:17:28 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T03:17:28.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:17:28 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:17:28.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:17:28 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T03:17:28.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:17:28 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:17:28.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:17:28 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T03:17:28.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:17:28 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:17:28.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:17:28 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:17:28.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:17:28 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:17:29.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:17:29 smithi115 conmon[90393]: 2022-05-14T03:17:29.062+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:17:29.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:17:28 smithi115 ceph-mon[78593]: pgmap v7242: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:17:29.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:17:28 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T03:17:29.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:17:28 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T03:17:29.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:17:28 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T03:17:29.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:17:28 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:17:29.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:17:28 smithi047 ceph-mon[77147]: pgmap v7242: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:17:29.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:17:28 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T03:17:29.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:17:28 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T03:17:29.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:17:28 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T03:17:29.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:17:28 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:17:30.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:17:30 smithi115 conmon[90393]: 2022-05-14T03:17:30.023+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:17:30.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:17:29 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:17:30.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T03:17:30.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:17:30.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T03:17:30.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:17:30.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:17:30.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:17:30.113 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T03:17:30.113 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:17:30.113 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T03:17:30.114 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:17:30.114 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T03:17:30.115 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:17:30.115 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T03:17:30.116 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:17:30.116 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T03:17:30.117 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:17:30.117 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:17:30.118 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:17:30.118 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:17:29 smithi115 ceph-mon[78593]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T03:17:30.118 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:17:29 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:17:30.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:17:29 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:17:30.292 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T03:17:30.292 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:17:30.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T03:17:30.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:17:30.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:17:30.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:17:30.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T03:17:30.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:17:30.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T03:17:30.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:17:30.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T03:17:30.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:17:30.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T03:17:30.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:17:30.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T03:17:30.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:17:30.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:17:30.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:17:30.296 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:17:29 smithi047 ceph-mon[77147]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T03:17:30.296 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:17:29 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:17:31.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:17:30 smithi115 ceph-mon[78593]: pgmap v7243: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:17:31.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:17:30 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:17:31.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:17:30 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:17:31.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:17:30 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:17:31.110 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:17:30 smithi115 conmon[90393]: 2022-05-14T03:17:30.994+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:17:31.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:17:30 smithi047 ceph-mon[77147]: pgmap v7243: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:17:31.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:17:30 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:17:31.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:17:30 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:17:31.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:17:30 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:17:31.792 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:17:31 smithi047 conmon[78521]: 2022-05-14T03:17:31.439+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:17:31.793 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:17:31 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T03:17:31.793 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:17:31 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:17:31.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:17:31 smithi047 conmon[78521]: yield 2022-05-14T03:17:31.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:17:31 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:17:31.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:17:31 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:17:31.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:17:31 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:17:31.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:17:31 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T03:17:31.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:17:31 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:17:31.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:17:31 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T03:17:31.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:17:31 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:17:31.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:17:31 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T03:17:31.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:17:31 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:17:31.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:17:31 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T03:17:31.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:17:31 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:17:31.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:17:31 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T03:17:31.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:17:31 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:17:31.797 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:17:31 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:17:31.797 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:17:31 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:17:31.797 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:17:31 smithi047 conmon[78521]: 2022-05-14T03:17:31.440+0000 7f5344781700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T03:17:31.797 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:17:31 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T03:17:31.798 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:17:31 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:17:31.798 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:17:31 smithi047 conmon[78521]: yield 2022-05-14T03:17:31.798 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:17:31 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:17:31.798 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:17:31 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:17:31.799 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:17:31 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:17:31.799 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:17:31 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T03:17:31.799 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:17:31 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:17:31.799 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:17:31 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T03:17:31.799 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:17:31 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:17:31.800 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:17:31 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T03:17:31.800 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:17:31 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:17:31.800 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:17:31 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T03:17:31.800 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:17:31 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:17:31.800 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:17:31 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T03:17:31.801 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:17:31 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:17:31.801 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:17:31 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:17:31.801 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:17:31 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:17:31.801 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:17:31 smithi047 conmon[78521]: 2022-05-14T03:17:31.802 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:17:31 smithi047 conmon[78521]: During handling of the above exception, another exception occurred: 2022-05-14T03:17:31.802 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:17:31 smithi047 conmon[78521]: 2022-05-14T03:17:31.802 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:17:31 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T03:17:31.802 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:17:31 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T03:17:31.802 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:17:31 smithi047 conmon[78521]: return f(*arg) 2022-05-14T03:17:31.803 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:17:31 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T03:17:31.803 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:17:31 smithi047 conmon[78521]: self._write_client_files(client_files, host) 2022-05-14T03:17:31.803 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:17:31 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T03:17:31.803 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:17:31 smithi047 conmon[78521]: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T03:17:31.803 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:17:31 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T03:17:31.804 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:17:31 smithi047 conmon[78521]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T03:17:31.804 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:17:31 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T03:17:31.804 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:17:31 smithi047 conmon[78521]: return self.event_loop.get_result(coro) 2022-05-14T03:17:31.804 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:17:31 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T03:17:31.804 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:17:31 smithi047 conmon[78521]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T03:17:31.805 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:17:31 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T03:17:31.805 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:17:31 smithi047 conmon[78521]: return self.__get_result() 2022-05-14T03:17:31.805 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:17:31 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T03:17:31.806 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:17:31 smithi047 conmon[78521]: raise self._exception 2022-05-14T03:17:31.806 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:17:31 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T03:17:31.806 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:17:31 smithi047 conmon[78521]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T03:17:31.811 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:17:31 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T03:17:31.812 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:17:31 smithi047 conmon[78521]: conn = await self._remote_connection(host, addr) 2022-05-14T03:17:31.812 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:17:31 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T03:17:31.812 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:17:31 smithi047 conmon[78521]: raise 2022-05-14T03:17:31.813 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:17:31 smithi047 conmon[78521]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T03:17:31.813 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:17:31 smithi047 conmon[78521]: self.gen.throw(type, value, traceback) 2022-05-14T03:17:31.813 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:17:31 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T03:17:31.813 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:17:31 smithi047 conmon[78521]: raise OrchestratorError(msg) 2022-05-14T03:17:31.814 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:17:31 smithi047 conmon[78521]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:17:32.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:17:31 smithi115 conmon[90393]: 2022-05-14T03:17:31.998+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:17:32.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:17:31 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:17:32.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:17:31 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 13785 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:17:32.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:17:31 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:17:32.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:17:31 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 13785 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:17:33.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:17:33 smithi115 conmon[90393]: 2022-05-14T03:17:33.013+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:17:33.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:17:32 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:17:33.109 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T03:17:33.109 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:17:33.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T03:17:33.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:17:33.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:17:33.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:17:33.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T03:17:33.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:17:33.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T03:17:33.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:17:33.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T03:17:33.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:17:33.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T03:17:33.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:17:33.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T03:17:33.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:17:33.113 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:17:33.113 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:17:33.113 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:17:32 smithi115 ceph-mon[78593]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T03:17:33.114 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T03:17:33.114 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:17:33.114 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T03:17:33.114 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:17:33.114 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:17:33.115 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:17:33.115 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T03:17:33.115 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:17:33.115 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T03:17:33.115 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:17:33.116 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T03:17:33.116 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:17:33.116 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T03:17:33.116 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:17:33.117 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T03:17:33.117 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:17:33.117 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:17:33.117 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:17:33.118 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T03:17:33.118 INFO:journalctl@ceph.mon.c.smithi115.stdout: During handling of the above exception, another exception occurred: 2022-05-14T03:17:33.118 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T03:17:33.118 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T03:17:33.118 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T03:17:33.119 INFO:journalctl@ceph.mon.c.smithi115.stdout: return f(*arg) 2022-05-14T03:17:33.119 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T03:17:33.119 INFO:journalctl@ceph.mon.c.smithi115.stdout: self._write_client_files(client_files, host) 2022-05-14T03:17:33.119 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T03:17:33.120 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T03:17:33.120 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T03:17:33.120 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T03:17:33.120 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T03:17:33.120 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.event_loop.get_result(coro) 2022-05-14T03:17:33.121 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T03:17:33.121 INFO:journalctl@ceph.mon.c.smithi115.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T03:17:33.121 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T03:17:33.121 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.__get_result() 2022-05-14T03:17:33.122 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T03:17:33.122 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise self._exception 2022-05-14T03:17:33.122 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T03:17:33.122 INFO:journalctl@ceph.mon.c.smithi115.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T03:17:33.122 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T03:17:33.123 INFO:journalctl@ceph.mon.c.smithi115.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T03:17:33.123 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T03:17:33.123 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise 2022-05-14T03:17:33.124 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T03:17:33.124 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.gen.throw(type, value, traceback) 2022-05-14T03:17:33.125 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T03:17:33.125 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OrchestratorError(msg) 2022-05-14T03:17:33.125 INFO:journalctl@ceph.mon.c.smithi115.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:17:33.126 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:17:32 smithi115 ceph-mon[78593]: pgmap v7244: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:17:33.126 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:17:32 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:17:33.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:17:32 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:17:33.292 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T03:17:33.292 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:17:33.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T03:17:33.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:17:33.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:17:33.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:17:33.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T03:17:33.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:17:33.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T03:17:33.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:17:33.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T03:17:33.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:17:33.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T03:17:33.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:17:33.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T03:17:33.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:17:33.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:17:33.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:17:33.296 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:17:32 smithi047 ceph-mon[77147]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T03:17:33.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T03:17:33.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:17:33.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T03:17:33.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:17:33.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:17:33.298 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:17:33.298 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T03:17:33.298 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:17:33.298 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T03:17:33.299 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:17:33.299 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T03:17:33.299 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:17:33.299 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T03:17:33.299 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:17:33.300 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T03:17:33.300 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:17:33.300 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:17:33.300 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:17:33.301 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T03:17:33.301 INFO:journalctl@ceph.mon.b.smithi047.stdout: During handling of the above exception, another exception occurred: 2022-05-14T03:17:33.301 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T03:17:33.301 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T03:17:33.301 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T03:17:33.302 INFO:journalctl@ceph.mon.b.smithi047.stdout: return f(*arg) 2022-05-14T03:17:33.302 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T03:17:33.302 INFO:journalctl@ceph.mon.b.smithi047.stdout: self._write_client_files(client_files, host) 2022-05-14T03:17:33.302 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T03:17:33.302 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T03:17:33.303 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T03:17:33.303 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T03:17:33.303 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T03:17:33.303 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.event_loop.get_result(coro) 2022-05-14T03:17:33.304 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T03:17:33.304 INFO:journalctl@ceph.mon.b.smithi047.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T03:17:33.304 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T03:17:33.304 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.__get_result() 2022-05-14T03:17:33.304 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T03:17:33.305 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise self._exception 2022-05-14T03:17:33.305 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T03:17:33.305 INFO:journalctl@ceph.mon.b.smithi047.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T03:17:33.305 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T03:17:33.306 INFO:journalctl@ceph.mon.b.smithi047.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T03:17:33.306 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T03:17:33.306 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise 2022-05-14T03:17:33.306 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T03:17:33.306 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.gen.throw(type, value, traceback) 2022-05-14T03:17:33.307 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T03:17:33.307 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OrchestratorError(msg) 2022-05-14T03:17:33.307 INFO:journalctl@ceph.mon.b.smithi047.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:17:33.307 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:17:32 smithi047 ceph-mon[77147]: pgmap v7244: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:17:33.307 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:17:32 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:17:34.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:17:34 smithi115 conmon[90393]: 2022-05-14T03:17:34.051+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:17:34.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:17:33 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:17:34.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:17:33 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:17:35.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:17:35 smithi115 conmon[90393]: 2022-05-14T03:17:35.051+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:17:35.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:17:34 smithi115 ceph-mon[78593]: pgmap v7245: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:17:35.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:17:34 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:17:35.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:17:34 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:17:35.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:17:34 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:17:35.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:17:34 smithi047 ceph-mon[77147]: pgmap v7245: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:17:35.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:17:34 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:17:35.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:17:34 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:17:35.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:17:34 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:17:36.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:17:35 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:17:36.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:17:36 smithi115 conmon[90393]: 2022-05-14T03:17:36.053+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:17:36.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:17:35 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:17:37.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:17:37 smithi115 conmon[90393]: 2022-05-14T03:17:37.087+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:17:37.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:17:36 smithi115 ceph-mon[78593]: pgmap v7246: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:17:37.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:17:36 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:17:37.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:17:36 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 13790 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:17:37.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:17:36 smithi047 ceph-mon[77147]: pgmap v7246: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:17:37.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:17:36 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:17:37.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:17:36 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 13790 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:17:38.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:17:37 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:17:38.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:17:38 smithi115 conmon[90393]: 2022-05-14T03:17:38.081+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:17:38.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:17:37 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:17:39.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:17:39 smithi115 conmon[90393]: 2022-05-14T03:17:39.060+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:17:39.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:17:38 smithi115 ceph-mon[78593]: pgmap v7247: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:17:39.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:17:38 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:17:39.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:17:38 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:17:39.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:17:38 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:17:39.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:17:38 smithi047 ceph-mon[77147]: pgmap v7247: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:17:39.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:17:38 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:17:39.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:17:38 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:17:39.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:17:38 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:17:40.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:17:39 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:17:40.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:17:40 smithi115 conmon[90393]: 2022-05-14T03:17:40.090+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:17:40.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:17:39 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:17:41.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:17:41 smithi115 conmon[90393]: 2022-05-14T03:17:41.062+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:17:41.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:17:40 smithi115 ceph-mon[78593]: pgmap v7248: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:17:41.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:17:40 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:17:41.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:17:40 smithi047 ceph-mon[77147]: pgmap v7248: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:17:41.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:17:40 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:17:42.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:17:41 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:17:42.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:17:41 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 13795 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:17:42.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:17:41 smithi047 ceph-mon[77147]: pgmap v7249: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:17:42.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:17:42 smithi115 conmon[90393]: 2022-05-14T03:17:42.044+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:17:42.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:17:41 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:17:42.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:17:41 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 13795 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:17:42.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:17:41 smithi115 ceph-mon[78593]: pgmap v7249: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:17:43.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:17:42 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:17:43.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:17:42 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:17:43.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:17:42 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:17:43.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:17:43 smithi115 conmon[90393]: 2022-05-14T03:17:43.004+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:17:43.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:17:42 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:17:43.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:17:42 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:17:43.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:17:42 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:17:44.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:17:43 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:17:44.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:17:43 smithi047 ceph-mon[77147]: pgmap v7250: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:17:44.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:17:43 smithi115 conmon[90393]: 2022-05-14T03:17:43.985+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:17:44.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:17:43 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:17:44.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:17:43 smithi115 ceph-mon[78593]: pgmap v7250: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:17:45.291 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:17:44 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:17:45.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:17:44 smithi115 conmon[90393]: 2022-05-14T03:17:44.990+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:17:45.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:17:44 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:17:46.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:17:45 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:17:46.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:17:45 smithi047 ceph-mon[77147]: pgmap v7251: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:17:46.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:17:45 smithi115 conmon[90393]: 2022-05-14T03:17:45.964+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:17:46.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:17:45 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:17:46.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:17:45 smithi115 ceph-mon[78593]: pgmap v7251: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:17:47.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:17:46 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:17:47.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:17:46 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 13800 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:17:47.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:17:46 smithi115 conmon[90393]: 2022-05-14T03:17:46.977+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:17:47.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:17:46 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:17:47.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:17:46 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 13800 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:17:48.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:17:47 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:17:48.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:17:47 smithi047 ceph-mon[77147]: pgmap v7252: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:17:48.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:17:47 smithi115 conmon[90393]: 2022-05-14T03:17:47.961+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:17:48.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:17:47 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:17:48.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:17:47 smithi115 ceph-mon[78593]: pgmap v7252: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:17:49.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:17:48 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:17:49.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:17:48 smithi115 conmon[90393]: 2022-05-14T03:17:48.968+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:17:49.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:17:48 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:17:50.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:17:49 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:17:50.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:17:49 smithi047 ceph-mon[77147]: pgmap v7253: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:17:50.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:17:50 smithi115 conmon[90393]: 2022-05-14T03:17:50.013+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:17:50.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:17:49 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:17:50.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:17:49 smithi115 ceph-mon[78593]: pgmap v7253: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:17:51.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:17:50 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:17:51.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:17:50 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:17:51.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:17:50 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:17:51.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:17:51 smithi115 conmon[90393]: 2022-05-14T03:17:51.010+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:17:51.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:17:50 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:17:51.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:17:50 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:17:51.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:17:50 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:17:52.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:17:51 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:17:52.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:17:51 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 13805 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:17:52.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:17:51 smithi047 ceph-mon[77147]: pgmap v7254: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:17:52.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:17:51 smithi115 conmon[90393]: 2022-05-14T03:17:51.979+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:17:52.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:17:51 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:17:52.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:17:51 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 13805 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:17:52.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:17:51 smithi115 ceph-mon[78593]: pgmap v7254: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:17:53.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:17:52 smithi115 conmon[90393]: 2022-05-14T03:17:52.960+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:17:53.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:17:53 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:17:53.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:17:53 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:17:54.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:17:53 smithi115 conmon[90393]: 2022-05-14T03:17:53.937+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:17:54.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:17:54 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:17:54.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:17:54 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:17:54.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:17:54 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:17:54.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:17:54 smithi047 ceph-mon[77147]: pgmap v7255: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:17:54.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:17:54 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:17:54.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:17:54 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:17:54.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:17:54 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:17:54.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:17:54 smithi115 ceph-mon[78593]: pgmap v7255: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:17:55.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:17:54 smithi115 conmon[90393]: 2022-05-14T03:17:54.975+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:17:55.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:17:55 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:17:55.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:17:55 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:17:56.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:17:55 smithi115 conmon[90393]: 2022-05-14T03:17:55.982+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:17:56.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:17:56 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:17:56.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:17:56 smithi047 ceph-mon[77147]: pgmap v7256: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:17:56.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:17:56 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:17:56.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:17:56 smithi115 ceph-mon[78593]: pgmap v7256: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:17:57.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:17:56 smithi115 conmon[90393]: 2022-05-14T03:17:56.976+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:17:57.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:17:57 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:17:57.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:17:57 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 13810 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:17:57.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:17:57 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:17:57.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:17:57 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 13810 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:17:58.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:17:57 smithi115 conmon[90393]: 2022-05-14T03:17:57.954+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:17:58.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:17:58 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:17:58.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:17:58 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:17:58.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:17:58 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:17:58.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:17:58 smithi047 ceph-mon[77147]: pgmap v7257: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:17:58.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:17:58 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:17:58.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:17:58 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:17:58.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:17:58 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:17:58.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:17:58 smithi115 ceph-mon[78593]: pgmap v7257: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:17:59.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:17:58 smithi115 conmon[90393]: 2022-05-14T03:17:58.984+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:17:59.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:17:59 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:17:59.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:17:59 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:18:00.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:18:00 smithi115 conmon[90393]: 2022-05-14T03:18:00.005+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:18:00.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:18:00 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:18:00.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:18:00 smithi047 ceph-mon[77147]: pgmap v7258: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:18:00.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:18:00 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:18:00.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:18:00 smithi115 ceph-mon[78593]: pgmap v7258: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:18:01.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:18:00 smithi115 conmon[90393]: 2022-05-14T03:18:00.996+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:18:01.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:18:01 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:18:01.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:18:01 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:18:02.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:18:02 smithi115 conmon[90393]: 2022-05-14T03:18:02.025+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:18:02.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:18:02 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:18:02.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:18:02 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:18:02.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:18:02 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:18:02.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:18:02 smithi047 ceph-mon[77147]: pgmap v7259: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:18:02.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:18:02 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:18:02.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:18:02 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:18:02.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:18:02 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:18:02.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:18:02 smithi115 ceph-mon[78593]: pgmap v7259: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:18:03.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:18:03 smithi115 conmon[90393]: 2022-05-14T03:18:03.055+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:18:03.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:18:03 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:18:03.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:18:03 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:18:04.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:18:04 smithi115 conmon[90393]: 2022-05-14T03:18:04.039+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:18:04.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:18:04 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:18:04.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:18:04 smithi047 ceph-mon[77147]: pgmap v7260: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:18:04.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:18:04 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:18:04.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:18:04 smithi115 ceph-mon[78593]: pgmap v7260: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:18:05.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:18:05 smithi115 conmon[90393]: 2022-05-14T03:18:05.013+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:18:05.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:18:05 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:18:05.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:18:05 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:18:06.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:18:06 smithi115 conmon[90393]: 2022-05-14T03:18:06.030+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:18:06.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:18:06 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:18:06.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:18:06 smithi047 ceph-mon[77147]: pgmap v7261: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:18:06.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:18:06 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 13815 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:18:06.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:18:06 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:18:06.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:18:06 smithi115 ceph-mon[78593]: pgmap v7261: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:18:06.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:18:06 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 13815 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:18:07.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:18:07 smithi115 conmon[90393]: 2022-05-14T03:18:07.043+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:18:07.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:18:07 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:18:07.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:18:07 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:18:08.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:18:08 smithi115 conmon[90393]: 2022-05-14T03:18:08.010+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:18:08.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:18:08 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:18:08.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:18:08 smithi047 ceph-mon[77147]: pgmap v7262: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:18:08.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:18:08 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:18:08.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:18:08 smithi115 ceph-mon[78593]: pgmap v7262: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:18:09.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:18:09 smithi115 conmon[90393]: 2022-05-14T03:18:09.053+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:18:09.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:18:09 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:18:09.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:18:09 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:18:10.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:18:10 smithi115 conmon[90393]: 2022-05-14T03:18:10.048+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:18:10.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:18:10 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:18:10.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:18:10 smithi047 ceph-mon[77147]: pgmap v7263: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:18:10.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:18:10 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:18:10.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:18:10 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:18:10.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:18:10 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:18:10.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:18:10 smithi115 ceph-mon[78593]: pgmap v7263: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:18:10.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:18:10 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:18:10.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:18:10 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:18:11.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:18:11 smithi115 conmon[90393]: 2022-05-14T03:18:11.051+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:18:11.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:18:11 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:18:11.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:18:11 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 13825 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:18:11.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:18:11 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:18:11.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:18:11 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 13825 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:18:12.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:18:12 smithi115 conmon[90393]: 2022-05-14T03:18:12.085+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:18:12.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:18:12 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:18:12.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:18:12 smithi047 ceph-mon[77147]: pgmap v7264: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:18:12.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:18:12 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:18:12.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:18:12 smithi115 ceph-mon[78593]: pgmap v7264: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:18:13.476 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:18:13 smithi115 conmon[90393]: 2022-05-14T03:18:13.112+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:18:13.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:18:13 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:18:13.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:18:13 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:18:14.478 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:18:14 smithi115 conmon[90393]: 2022-05-14T03:18:14.130+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:18:14.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:18:14 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:18:14.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:18:14 smithi047 ceph-mon[77147]: pgmap v7265: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:18:14.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:18:14 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:18:14.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:18:14 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:18:14.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:18:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T03:18:14.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:18:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T03:18:14.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:18:14 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:18:14.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:18:14 smithi115 ceph-mon[78593]: pgmap v7265: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:18:14.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:18:14 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:18:14.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:18:14 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:18:14.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:18:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T03:18:14.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:18:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T03:18:15.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:18:15 smithi115 conmon[90393]: 2022-05-14T03:18:15.097+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:18:15.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:18:15 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:18:15.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:18:15 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:18:16.484 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:18:16 smithi115 conmon[90393]: 2022-05-14T03:18:16.107+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:18:16.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:18:16 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:18:16.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:18:16 smithi047 ceph-mon[77147]: pgmap v7266: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:18:16.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:18:16 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 13830 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:18:16.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:18:16 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:18:16.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:18:16 smithi115 ceph-mon[78593]: pgmap v7266: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:18:16.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:18:16 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 13830 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:18:17.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:18:17 smithi115 conmon[90393]: 2022-05-14T03:18:17.092+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:18:17.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:18:17 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:18:17.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:18:17 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:18:18.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:18:18 smithi115 conmon[90393]: 2022-05-14T03:18:18.093+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:18:18.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:18:18 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:18:18.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:18:18 smithi047 ceph-mon[77147]: pgmap v7267: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:18:18.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:18:18 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:18:18.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:18:18 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:18:18.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:18:18 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:18:18.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:18:18 smithi115 ceph-mon[78593]: pgmap v7267: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:18:18.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:18:18 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:18:18.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:18:18 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:18:19.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:18:19 smithi115 conmon[90393]: 2022-05-14T03:18:19.100+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:18:19.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:18:19 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:18:19.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:18:19 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:18:20.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:18:20 smithi115 conmon[90393]: 2022-05-14T03:18:20.053+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:18:20.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:18:20 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:18:20.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:18:20 smithi047 ceph-mon[77147]: pgmap v7268: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:18:20.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:18:20 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:18:20.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:18:20 smithi115 ceph-mon[78593]: pgmap v7268: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:18:21.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:18:21 smithi115 conmon[90393]: 2022-05-14T03:18:21.044+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:18:21.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:18:21 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:18:21.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:18:21 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 13835 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:18:21.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:18:21 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:18:21.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:18:21 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 13835 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:18:22.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:18:22 smithi115 conmon[90393]: 2022-05-14T03:18:22.084+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:18:22.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:18:22 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:18:22.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:18:22 smithi047 ceph-mon[77147]: pgmap v7269: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:18:22.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:18:22 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:18:22.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:18:22 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:18:22.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:18:22 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:18:22.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:18:22 smithi115 ceph-mon[78593]: pgmap v7269: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:18:22.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:18:22 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:18:22.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:18:22 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:18:23.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:18:23 smithi115 conmon[90393]: 2022-05-14T03:18:23.068+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:18:23.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:18:23 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:18:23.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:18:23 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:18:24.510 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:18:24 smithi115 conmon[90393]: 2022-05-14T03:18:24.116+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:18:24.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:18:24 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:18:24.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:18:24 smithi047 ceph-mon[77147]: pgmap v7270: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:18:24.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:18:24 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:18:24.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:18:24 smithi115 ceph-mon[78593]: pgmap v7270: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:18:25.515 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:18:25 smithi115 conmon[90393]: 2022-05-14T03:18:25.136+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:18:25.791 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:18:25 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:18:25.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:18:25 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:18:26.518 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:18:26 smithi115 conmon[90393]: 2022-05-14T03:18:26.116+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:18:26.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:18:26 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:18:26.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:18:26 smithi047 ceph-mon[77147]: pgmap v7271: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:18:26.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:18:26 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 13839 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:18:26.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:18:26 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:18:26.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:18:26 smithi115 ceph-mon[78593]: pgmap v7271: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:18:26.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:18:26 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 13839 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:18:27.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:18:27 smithi115 conmon[90393]: 2022-05-14T03:18:27.099+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:18:27.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:18:27 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:18:27.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:18:27 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:18:28.523 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:18:28 smithi115 conmon[90393]: 2022-05-14T03:18:28.117+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:18:28.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:18:28 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:18:28.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:18:28 smithi047 ceph-mon[77147]: pgmap v7272: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:18:28.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:18:28 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:18:28.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:18:28 smithi115 ceph-mon[78593]: pgmap v7272: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:18:29.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:18:29 smithi115 conmon[90393]: 2022-05-14T03:18:29.098+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:18:29.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:18:29 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:18:29.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:18:29 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:18:30.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:18:30 smithi115 conmon[90393]: 2022-05-14T03:18:30.060+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:18:30.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:18:30 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:18:30.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:18:30 smithi047 ceph-mon[77147]: pgmap v7273: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:18:30.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:18:30 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:18:30.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:18:30 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:18:30.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:18:30 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:18:30.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:18:30 smithi115 ceph-mon[78593]: pgmap v7273: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:18:30.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:18:30 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:18:30.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:18:30 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:18:31.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:18:31 smithi115 conmon[90393]: 2022-05-14T03:18:31.070+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:18:31.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:18:31 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:18:31.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:18:31 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 13845 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:18:31.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:18:31 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T03:18:31.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:18:31 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T03:18:31.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:18:31 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T03:18:31.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:18:31 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:18:31.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:18:31 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 13845 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:18:31.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:18:31 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T03:18:31.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:18:31 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T03:18:31.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:18:31 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T03:18:32.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:18:32 smithi115 conmon[90393]: 2022-05-14T03:18:32.054+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:18:32.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:18:32 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:18:32.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:18:32 smithi047 ceph-mon[77147]: pgmap v7274: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:18:32.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:18:32 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:18:32.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:18:32 smithi115 ceph-mon[78593]: pgmap v7274: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:18:33.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:18:33 smithi115 conmon[90393]: 2022-05-14T03:18:33.066+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:18:33.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:18:33 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:18:33.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:18:33 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:18:34.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:18:34 smithi115 conmon[90393]: 2022-05-14T03:18:34.080+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:18:34.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:18:34 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:18:34.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:18:34 smithi047 ceph-mon[77147]: pgmap v7275: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:18:34.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:18:34 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:18:34.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:18:34 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:18:34.793 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:18:34 smithi047 conmon[78521]: 2022-05-14T03:18:34.544+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:18:34.793 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:18:34 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T03:18:34.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:18:34 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:18:34.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:18:34 smithi047 conmon[78521]: yield 2022-05-14T03:18:34.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:18:34 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:18:34.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:18:34 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:18:34.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:18:34 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:18:34.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:18:34 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T03:18:34.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:18:34 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:18:34.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:18:34 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T03:18:34.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:18:34 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:18:34.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:18:34 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T03:18:34.797 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:18:34 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:18:34.797 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:18:34 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T03:18:34.797 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:18:34 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:18:34.797 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:18:34 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T03:18:34.798 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:18:34 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:18:34.798 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:18:34 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:18:34.798 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:18:34 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:18:34.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:18:34 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:18:34.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:18:34 smithi115 ceph-mon[78593]: pgmap v7275: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:18:34.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:18:34 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:18:34.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:18:34 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:18:35.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:18:35 smithi115 conmon[90393]: 2022-05-14T03:18:35.060+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:18:35.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:18:35 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:18:35.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:18:35 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:18:35.859 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T03:18:35.860 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:18:35.860 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T03:18:35.860 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:18:35.860 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:18:35.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:18:35.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T03:18:35.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:18:35.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T03:18:35.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:18:35.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T03:18:35.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:18:35.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T03:18:35.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:18:35.863 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T03:18:35.863 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:18:35.863 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:18:35.863 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:18:35.863 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:18:35 smithi115 ceph-mon[78593]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T03:18:36.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:18:35 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:18:36.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:18:35 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:18:36.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T03:18:36.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:18:36.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T03:18:36.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:18:36.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:18:36.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:18:36.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T03:18:36.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:18:36.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T03:18:36.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:18:36.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T03:18:36.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:18:36.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T03:18:36.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:18:36.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T03:18:36.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:18:36.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:18:36.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:18:36.047 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:18:35 smithi047 ceph-mon[77147]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T03:18:36.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:18:36 smithi115 conmon[90393]: 2022-05-14T03:18:36.059+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:18:36.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:18:36 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:18:36.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:18:36 smithi115 ceph-mon[78593]: pgmap v7276: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:18:36.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:18:36 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 13849 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:18:37.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:18:36 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:18:37.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:18:36 smithi047 ceph-mon[77147]: pgmap v7276: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:18:37.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:18:36 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 13849 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:18:37.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:18:37 smithi115 conmon[90393]: 2022-05-14T03:18:37.049+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:18:37.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:18:37 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:18:38.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:18:37 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:18:38.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:18:37 smithi047 conmon[78521]: 2022-05-14T03:18:37.617+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:18:38.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:18:37 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T03:18:38.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:18:37 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:18:38.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:18:37 smithi047 conmon[78521]: yield 2022-05-14T03:18:38.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:18:37 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:18:38.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:18:37 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:18:38.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:18:37 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:18:38.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:18:37 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T03:18:38.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:18:37 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:18:38.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:18:37 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T03:18:38.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:18:37 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:18:38.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:18:37 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T03:18:38.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:18:37 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:18:38.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:18:37 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T03:18:38.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:18:37 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:18:38.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:18:37 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T03:18:38.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:18:37 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:18:38.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:18:37 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:18:38.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:18:37 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:18:38.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:18:37 smithi047 conmon[78521]: 2022-05-14T03:18:37.617+0000 7f5345f84700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T03:18:38.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:18:37 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T03:18:38.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:18:37 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:18:38.048 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:18:37 smithi047 conmon[78521]: yield 2022-05-14T03:18:38.048 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:18:37 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:18:38.048 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:18:37 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:18:38.049 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:18:37 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:18:38.049 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:18:37 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T03:18:38.049 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:18:37 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:18:38.049 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:18:37 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T03:18:38.050 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:18:37 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:18:38.050 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:18:37 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T03:18:38.050 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:18:37 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:18:38.050 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:18:37 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T03:18:38.050 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:18:37 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:18:38.051 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:18:37 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T03:18:38.051 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:18:37 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:18:38.051 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:18:37 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:18:38.051 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:18:37 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:18:38.051 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:18:37 smithi047 conmon[78521]: 2022-05-14T03:18:38.052 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:18:37 smithi047 conmon[78521]: During handling of the above exception, another exception occurred: 2022-05-14T03:18:38.052 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:18:37 smithi047 conmon[78521]: 2022-05-14T03:18:38.052 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:18:37 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T03:18:38.052 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:18:37 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T03:18:38.053 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:18:37 smithi047 conmon[78521]: return f(*arg) 2022-05-14T03:18:38.053 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:18:37 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T03:18:38.053 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:18:37 smithi047 conmon[78521]: self._write_client_files(client_files, host) 2022-05-14T03:18:38.053 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:18:37 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T03:18:38.053 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:18:37 smithi047 conmon[78521]: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T03:18:38.054 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:18:37 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T03:18:38.054 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:18:37 smithi047 conmon[78521]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T03:18:38.054 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:18:37 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T03:18:38.054 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:18:37 smithi047 conmon[78521]: return self.event_loop.get_result(coro) 2022-05-14T03:18:38.054 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:18:37 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T03:18:38.055 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:18:37 smithi047 conmon[78521]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T03:18:38.055 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:18:37 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T03:18:38.055 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:18:37 smithi047 conmon[78521]: return self.__get_result() 2022-05-14T03:18:38.055 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:18:37 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T03:18:38.056 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:18:37 smithi047 conmon[78521]: raise self._exception 2022-05-14T03:18:38.056 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:18:37 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T03:18:38.056 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:18:37 smithi047 conmon[78521]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T03:18:38.056 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:18:37 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T03:18:38.056 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:18:37 smithi047 conmon[78521]: conn = await self._remote_connection(host, addr) 2022-05-14T03:18:38.057 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:18:37 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T03:18:38.057 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:18:37 smithi047 conmon[78521]: raise 2022-05-14T03:18:38.057 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:18:37 smithi047 conmon[78521]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T03:18:38.057 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:18:37 smithi047 conmon[78521]: self.gen.throw(type, value, traceback) 2022-05-14T03:18:38.057 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:18:37 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T03:18:38.058 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:18:37 smithi047 conmon[78521]: raise OrchestratorError(msg) 2022-05-14T03:18:38.058 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:18:37 smithi047 conmon[78521]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:18:38.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:18:38 smithi115 conmon[90393]: 2022-05-14T03:18:38.022+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:18:38.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:18:38 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:18:38.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:18:38 smithi115 ceph-mon[78593]: pgmap v7277: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:18:38.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:18:38 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:18:38.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:18:38 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:18:38.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:18:38 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:18:38.859 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T03:18:38.860 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:18:38.860 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T03:18:38.860 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:18:38.860 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:18:38.860 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:18:38.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T03:18:38.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:18:38.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T03:18:38.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:18:38.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T03:18:38.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:18:38.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T03:18:38.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:18:38.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T03:18:38.863 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:18:38.863 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:18:38.863 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:18:38.863 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:18:38 smithi115 ceph-mon[78593]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T03:18:38.863 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T03:18:38.864 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:18:38.864 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T03:18:38.864 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:18:38.864 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:18:38.864 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:18:38.865 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T03:18:38.865 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:18:38.865 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T03:18:38.865 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:18:38.865 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T03:18:38.866 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:18:38.866 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T03:18:38.866 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:18:38.866 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T03:18:38.866 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:18:38.867 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:18:38.867 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:18:38.867 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T03:18:38.867 INFO:journalctl@ceph.mon.c.smithi115.stdout: During handling of the above exception, another exception occurred: 2022-05-14T03:18:38.868 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T03:18:38.868 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T03:18:38.868 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T03:18:38.868 INFO:journalctl@ceph.mon.c.smithi115.stdout: return f(*arg) 2022-05-14T03:18:38.868 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T03:18:38.869 INFO:journalctl@ceph.mon.c.smithi115.stdout: self._write_client_files(client_files, host) 2022-05-14T03:18:38.869 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T03:18:38.869 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T03:18:38.869 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T03:18:38.870 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T03:18:38.870 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T03:18:38.870 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.event_loop.get_result(coro) 2022-05-14T03:18:38.870 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T03:18:38.870 INFO:journalctl@ceph.mon.c.smithi115.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T03:18:38.871 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T03:18:38.871 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.__get_result() 2022-05-14T03:18:38.871 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T03:18:38.871 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise self._exception 2022-05-14T03:18:38.871 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T03:18:38.872 INFO:journalctl@ceph.mon.c.smithi115.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T03:18:38.872 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T03:18:38.872 INFO:journalctl@ceph.mon.c.smithi115.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T03:18:38.872 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T03:18:38.872 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise 2022-05-14T03:18:38.873 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T03:18:38.873 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.gen.throw(type, value, traceback) 2022-05-14T03:18:38.873 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T03:18:38.873 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OrchestratorError(msg) 2022-05-14T03:18:38.873 INFO:journalctl@ceph.mon.c.smithi115.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:18:39.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:18:38 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:18:39.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:18:38 smithi047 ceph-mon[77147]: pgmap v7277: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:18:39.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:18:38 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:18:39.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:18:38 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:18:39.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:18:38 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:18:39.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T03:18:39.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:18:39.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T03:18:39.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:18:39.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:18:39.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:18:39.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T03:18:39.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:18:39.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T03:18:39.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:18:39.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T03:18:39.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:18:39.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T03:18:39.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:18:39.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T03:18:39.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:18:39.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:18:39.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:18:39.049 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:18:38 smithi047 ceph-mon[77147]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T03:18:39.049 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T03:18:39.049 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:18:39.050 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T03:18:39.050 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:18:39.050 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:18:39.051 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:18:39.051 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T03:18:39.052 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:18:39.052 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T03:18:39.053 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:18:39.053 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T03:18:39.053 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:18:39.054 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T03:18:39.054 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:18:39.055 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T03:18:39.055 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:18:39.056 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:18:39.056 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:18:39.056 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T03:18:39.057 INFO:journalctl@ceph.mon.b.smithi047.stdout: During handling of the above exception, another exception occurred: 2022-05-14T03:18:39.057 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T03:18:39.058 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T03:18:39.058 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T03:18:39.059 INFO:journalctl@ceph.mon.b.smithi047.stdout: return f(*arg) 2022-05-14T03:18:39.059 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T03:18:39.060 INFO:journalctl@ceph.mon.b.smithi047.stdout: self._write_client_files(client_files, host) 2022-05-14T03:18:39.060 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T03:18:39.060 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T03:18:39.061 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T03:18:39.061 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T03:18:39.061 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T03:18:39.062 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.event_loop.get_result(coro) 2022-05-14T03:18:39.062 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T03:18:39.063 INFO:journalctl@ceph.mon.b.smithi047.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T03:18:39.063 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T03:18:39.063 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.__get_result() 2022-05-14T03:18:39.064 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T03:18:39.064 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise self._exception 2022-05-14T03:18:39.064 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T03:18:39.065 INFO:journalctl@ceph.mon.b.smithi047.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T03:18:39.065 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T03:18:39.065 INFO:journalctl@ceph.mon.b.smithi047.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T03:18:39.066 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T03:18:39.066 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise 2022-05-14T03:18:39.066 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T03:18:39.066 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.gen.throw(type, value, traceback) 2022-05-14T03:18:39.067 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T03:18:39.067 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OrchestratorError(msg) 2022-05-14T03:18:39.067 INFO:journalctl@ceph.mon.b.smithi047.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:18:39.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:18:39 smithi115 conmon[90393]: 2022-05-14T03:18:39.046+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:18:39.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:18:39 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:18:40.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:18:39 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:18:40.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:18:40 smithi115 conmon[90393]: 2022-05-14T03:18:40.069+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:18:40.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:18:40 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:18:40.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:18:40 smithi115 ceph-mon[78593]: pgmap v7278: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:18:41.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:18:40 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:18:41.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:18:40 smithi047 ceph-mon[77147]: pgmap v7278: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:18:41.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:18:41 smithi115 conmon[90393]: 2022-05-14T03:18:41.032+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:18:41.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:18:41 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:18:41.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:18:41 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 13854 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:18:42.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:18:41 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:18:42.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:18:41 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 13854 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:18:42.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:18:42 smithi115 conmon[90393]: 2022-05-14T03:18:42.063+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:18:42.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:18:42 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:18:42.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:18:42 smithi115 ceph-mon[78593]: pgmap v7279: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:18:42.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:18:42 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:18:42.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:18:42 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:18:43.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:18:42 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:18:43.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:18:42 smithi047 ceph-mon[77147]: pgmap v7279: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:18:43.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:18:42 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:18:43.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:18:42 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:18:43.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:18:43 smithi115 conmon[90393]: 2022-05-14T03:18:43.094+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:18:43.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:18:43 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:18:44.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:18:43 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:18:44.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:18:44 smithi115 conmon[90393]: 2022-05-14T03:18:44.065+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:18:44.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:18:44 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:18:44.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:18:44 smithi115 ceph-mon[78593]: pgmap v7280: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:18:45.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:18:44 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:18:45.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:18:44 smithi047 ceph-mon[77147]: pgmap v7280: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:18:45.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:18:45 smithi115 conmon[90393]: 2022-05-14T03:18:45.043+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:18:45.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:18:45 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:18:46.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:18:45 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:18:46.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:18:46 smithi115 conmon[90393]: 2022-05-14T03:18:46.017+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:18:46.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:18:46 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:18:46.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:18:46 smithi115 ceph-mon[78593]: pgmap v7281: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:18:46.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:18:46 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 13860 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:18:47.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:18:46 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:18:47.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:18:46 smithi047 ceph-mon[77147]: pgmap v7281: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:18:47.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:18:46 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 13860 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:18:47.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:18:47 smithi115 conmon[90393]: 2022-05-14T03:18:47.052+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:18:47.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:18:47 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:18:48.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:18:47 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:18:48.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:18:48 smithi115 conmon[90393]: 2022-05-14T03:18:48.058+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:18:48.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:18:48 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:18:48.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:18:48 smithi115 ceph-mon[78593]: pgmap v7282: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:18:49.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:18:48 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:18:49.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:18:48 smithi047 ceph-mon[77147]: pgmap v7282: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:18:49.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:18:49 smithi115 conmon[90393]: 2022-05-14T03:18:49.019+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:18:49.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:18:49 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:18:50.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:18:49 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:18:50.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:18:50 smithi115 conmon[90393]: 2022-05-14T03:18:50.060+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:18:50.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:18:50 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:18:50.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:18:50 smithi115 ceph-mon[78593]: pgmap v7283: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:18:50.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:18:50 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:18:50.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:18:50 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:18:51.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:18:50 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:18:51.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:18:50 smithi047 ceph-mon[77147]: pgmap v7283: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:18:51.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:18:50 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:18:51.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:18:50 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:18:51.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:18:51 smithi115 conmon[90393]: 2022-05-14T03:18:51.024+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:18:51.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:18:51 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:18:51.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:18:51 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 13865 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:18:52.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:18:51 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:18:52.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:18:51 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 13865 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:18:52.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:18:52 smithi115 conmon[90393]: 2022-05-14T03:18:52.072+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:18:52.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:18:52 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:18:52.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:18:52 smithi115 ceph-mon[78593]: pgmap v7284: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:18:53.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:18:52 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:18:53.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:18:52 smithi047 ceph-mon[77147]: pgmap v7284: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:18:53.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:18:53 smithi115 conmon[90393]: 2022-05-14T03:18:53.027+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:18:53.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:18:53 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:18:54.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:18:53 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:18:54.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:18:54 smithi115 conmon[90393]: 2022-05-14T03:18:54.065+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:18:54.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:18:54 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:18:54.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:18:54 smithi115 ceph-mon[78593]: pgmap v7285: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:18:54.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:18:54 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:18:54.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:18:54 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:18:55.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:18:54 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:18:55.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:18:54 smithi047 ceph-mon[77147]: pgmap v7285: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:18:55.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:18:54 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:18:55.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:18:54 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:18:55.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:18:55 smithi115 conmon[90393]: 2022-05-14T03:18:55.057+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:18:56.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:18:55 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:18:56.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:18:55 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:18:56.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:18:56 smithi115 conmon[90393]: 2022-05-14T03:18:56.040+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:18:56.993 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:18:56 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:18:56.993 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:18:56 smithi115 ceph-mon[78593]: pgmap v7286: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:18:56.994 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:18:56 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 13869 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:18:57.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:18:56 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:18:57.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:18:56 smithi047 ceph-mon[77147]: pgmap v7286: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:18:57.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:18:56 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 13869 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:18:57.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:18:56 smithi115 conmon[90393]: 2022-05-14T03:18:56.991+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:18:57.985 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:18:57 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:18:58.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:18:57 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:18:58.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:18:57 smithi115 conmon[90393]: 2022-05-14T03:18:57.983+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:18:58.968 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:18:58 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:18:58.968 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:18:58 smithi115 ceph-mon[78593]: pgmap v7287: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:18:58.969 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:18:58 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:18:58.969 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:18:58 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:18:59.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:18:58 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:18:59.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:18:58 smithi047 ceph-mon[77147]: pgmap v7287: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:18:59.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:18:58 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:18:59.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:18:58 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:18:59.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:18:58 smithi115 conmon[90393]: 2022-05-14T03:18:58.966+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:18:59.933 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:18:59 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:19:00.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:18:59 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:19:00.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:18:59 smithi115 conmon[90393]: 2022-05-14T03:18:59.931+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:19:00.973 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:19:00 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:19:00.974 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:19:00 smithi115 ceph-mon[78593]: pgmap v7288: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:19:01.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:19:00 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:19:01.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:19:00 smithi047 ceph-mon[77147]: pgmap v7288: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:19:01.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:19:00 smithi115 conmon[90393]: 2022-05-14T03:19:00.972+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:19:01.996 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:19:01 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:19:01.997 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:19:01 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 13875 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:19:02.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:19:01 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:19:02.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:19:01 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 13875 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:19:02.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:19:01 smithi115 conmon[90393]: 2022-05-14T03:19:01.994+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:19:03.006 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:19:02 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:19:03.006 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:19:02 smithi115 ceph-mon[78593]: pgmap v7289: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:19:03.006 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:19:02 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:19:03.007 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:19:02 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:19:03.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:19:02 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:19:03.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:19:02 smithi047 ceph-mon[77147]: pgmap v7289: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:19:03.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:19:02 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:19:03.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:19:02 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:19:03.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:19:03 smithi115 conmon[90393]: 2022-05-14T03:19:03.004+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:19:04.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:19:03 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:19:04.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:19:03 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:19:04.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:19:04 smithi115 conmon[90393]: 2022-05-14T03:19:04.045+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:19:05.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:19:04 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:19:05.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:19:04 smithi047 ceph-mon[77147]: pgmap v7290: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:19:05.070 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:19:04 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:19:05.071 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:19:04 smithi115 ceph-mon[78593]: pgmap v7290: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:19:05.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:19:05 smithi115 conmon[90393]: 2022-05-14T03:19:05.069+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:19:06.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:19:05 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:19:06.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:19:05 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:19:06.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:19:06 smithi115 conmon[90393]: 2022-05-14T03:19:06.044+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:19:07.034 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:19:06 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:19:07.034 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:19:06 smithi115 ceph-mon[78593]: pgmap v7291: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:19:07.034 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:19:06 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 13879 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:19:07.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:19:06 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:19:07.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:19:06 smithi047 ceph-mon[77147]: pgmap v7291: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:19:07.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:19:06 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 13879 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:19:07.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:19:07 smithi115 conmon[90393]: 2022-05-14T03:19:07.031+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:19:08.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:19:07 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:19:08.060 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:19:07 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:19:08.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:19:08 smithi115 conmon[90393]: 2022-05-14T03:19:08.058+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:19:09.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:19:08 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:19:09.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:19:08 smithi047 ceph-mon[77147]: pgmap v7292: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:19:09.084 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:19:08 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:19:09.084 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:19:08 smithi115 ceph-mon[78593]: pgmap v7292: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:19:09.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:19:09 smithi115 conmon[90393]: 2022-05-14T03:19:09.082+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:19:10.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:19:09 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:19:10.084 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:19:09 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:19:10.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:19:10 smithi115 conmon[90393]: 2022-05-14T03:19:10.083+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:19:11.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:19:10 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:19:11.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:19:10 smithi047 ceph-mon[77147]: pgmap v7293: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:19:11.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:19:10 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:19:11.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:19:10 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:19:11.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:19:10 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:19:11.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:19:10 smithi115 ceph-mon[78593]: pgmap v7293: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:19:11.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:19:10 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:19:11.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:19:10 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:19:11.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:19:11 smithi115 conmon[90393]: 2022-05-14T03:19:11.125+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:19:12.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:19:11 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:19:12.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:19:11 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 13884 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:19:12.101 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:19:11 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:19:12.102 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:19:11 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 13884 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:19:12.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:19:12 smithi115 conmon[90393]: 2022-05-14T03:19:12.099+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:19:13.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:19:12 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:19:13.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:19:12 smithi047 ceph-mon[77147]: pgmap v7294: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:19:13.085 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:19:12 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:19:13.085 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:19:12 smithi115 ceph-mon[78593]: pgmap v7294: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:19:13.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:19:13 smithi115 conmon[90393]: 2022-05-14T03:19:13.083+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:19:14.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:19:13 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:19:14.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:19:13 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:19:14.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:19:14 smithi115 conmon[90393]: 2022-05-14T03:19:14.113+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:19:15.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:19:14 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:19:15.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:19:14 smithi047 ceph-mon[77147]: pgmap v7295: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:19:15.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:19:14 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:19:15.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:19:14 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:19:15.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:19:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T03:19:15.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:19:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T03:19:15.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:19:14 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:19:15.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:19:14 smithi115 ceph-mon[78593]: pgmap v7295: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:19:15.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:19:14 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:19:15.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:19:14 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:19:15.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:19:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T03:19:15.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:19:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T03:19:15.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:19:15 smithi115 conmon[90393]: 2022-05-14T03:19:15.111+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:19:16.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:19:15 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:19:16.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:19:15 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:19:16.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:19:16 smithi115 conmon[90393]: 2022-05-14T03:19:16.157+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:19:17.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:19:16 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:19:17.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:19:16 smithi047 ceph-mon[77147]: pgmap v7296: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:19:17.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:19:16 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 13889 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:19:17.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:19:16 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:19:17.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:19:16 smithi115 ceph-mon[78593]: pgmap v7296: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:19:17.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:19:16 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 13889 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:19:17.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:19:17 smithi115 conmon[90393]: 2022-05-14T03:19:17.109+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:19:18.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:19:17 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:19:18.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:19:17 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:19:18.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:19:18 smithi115 conmon[90393]: 2022-05-14T03:19:18.142+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:19:19.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:19:18 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:19:19.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:19:18 smithi047 ceph-mon[77147]: pgmap v7297: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:19:19.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:19:18 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:19:19.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:19:18 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:19:19.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:19:18 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:19:19.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:19:18 smithi115 ceph-mon[78593]: pgmap v7297: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:19:19.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:19:18 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:19:19.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:19:18 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:19:19.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:19:19 smithi115 conmon[90393]: 2022-05-14T03:19:19.112+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:19:20.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:19:19 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:19:20.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:19:19 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:19:20.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:19:20 smithi115 conmon[90393]: 2022-05-14T03:19:20.138+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:19:21.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:19:20 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:19:21.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:19:20 smithi047 ceph-mon[77147]: pgmap v7298: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:19:21.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:19:20 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:19:21.101 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:19:20 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:19:21.101 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:19:20 smithi115 ceph-mon[78593]: pgmap v7298: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:19:21.101 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:19:20 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:19:21.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:19:21 smithi115 conmon[90393]: 2022-05-14T03:19:21.099+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:19:22.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:19:21 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:19:22.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:19:21 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 13895 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:19:22.092 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:19:21 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:19:22.092 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:19:21 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 13895 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:19:22.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:19:22 smithi115 conmon[90393]: 2022-05-14T03:19:22.091+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:19:23.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:19:22 smithi047 ceph-mon[77147]: pgmap v7299: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:19:23.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:19:22 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:19:23.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:19:22 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:19:23.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:19:22 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:19:23.067 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:19:22 smithi115 ceph-mon[78593]: pgmap v7299: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:19:23.068 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:19:22 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:19:23.068 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:19:22 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:19:23.068 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:19:22 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:19:23.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:19:23 smithi115 conmon[90393]: 2022-05-14T03:19:23.066+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:19:24.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:19:23 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:19:24.060 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:19:23 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:19:24.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:19:24 smithi115 conmon[90393]: 2022-05-14T03:19:24.058+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:19:25.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:19:24 smithi047 ceph-mon[77147]: pgmap v7300: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:19:25.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:19:24 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:19:25.076 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:19:24 smithi115 ceph-mon[78593]: pgmap v7300: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:19:25.076 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:19:24 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:19:25.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:19:25 smithi115 conmon[90393]: 2022-05-14T03:19:25.075+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:19:26.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:19:25 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:19:26.071 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:19:25 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:19:27.036 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:19:26 smithi115 conmon[90393]: 2022-05-14T03:19:26.070+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:19:27.036 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:19:26 smithi115 ceph-mon[78593]: pgmap v7301: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:19:27.036 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:19:26 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:19:27.037 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:19:26 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 13899 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:19:27.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:19:26 smithi047 ceph-mon[77147]: pgmap v7301: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:19:27.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:19:26 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:19:27.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:19:26 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 13899 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:19:27.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:19:27 smithi115 conmon[90393]: 2022-05-14T03:19:27.034+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:19:28.040 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:19:27 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:19:28.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:19:27 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:19:28.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:19:28 smithi115 conmon[90393]: 2022-05-14T03:19:28.038+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:19:29.035 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:19:28 smithi115 ceph-mon[78593]: pgmap v7302: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:19:29.035 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:19:28 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:19:29.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:19:28 smithi047 ceph-mon[77147]: pgmap v7302: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:19:29.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:19:28 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:19:29.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:19:29 smithi115 conmon[90393]: 2022-05-14T03:19:29.032+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:19:30.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:19:29 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:19:30.068 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:19:29 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:19:30.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:19:30 smithi115 conmon[90393]: 2022-05-14T03:19:30.066+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:19:31.028 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:19:30 smithi115 ceph-mon[78593]: pgmap v7303: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:19:31.028 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:19:30 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:19:31.029 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:19:30 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:19:31.029 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:19:30 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:19:31.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:19:30 smithi047 ceph-mon[77147]: pgmap v7303: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:19:31.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:19:30 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:19:31.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:19:30 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:19:31.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:19:30 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:19:31.365 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:19:31 smithi115 conmon[90393]: 2022-05-14T03:19:31.026+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:19:32.010 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:19:31 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:19:32.011 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:19:31 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 13905 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:19:32.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:19:31 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:19:32.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:19:31 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 13905 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:19:32.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:19:32 smithi115 conmon[90393]: 2022-05-14T03:19:32.008+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:19:33.039 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:19:32 smithi115 ceph-mon[78593]: pgmap v7304: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:19:33.039 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:19:32 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:19:33.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:19:32 smithi047 ceph-mon[77147]: pgmap v7304: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:19:33.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:19:32 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:19:33.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:19:33 smithi115 conmon[90393]: 2022-05-14T03:19:33.037+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:19:34.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:19:33 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:19:34.074 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:19:33 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:19:34.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:19:34 smithi115 conmon[90393]: 2022-05-14T03:19:34.072+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:19:35.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:19:34 smithi047 ceph-mon[77147]: pgmap v7305: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:19:35.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:19:34 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:19:35.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:19:34 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:19:35.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:19:34 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:19:35.089 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:19:34 smithi115 ceph-mon[78593]: pgmap v7305: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:19:35.090 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:19:34 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:19:35.090 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:19:34 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:19:35.090 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:19:34 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:19:35.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:19:35 smithi115 conmon[90393]: 2022-05-14T03:19:35.087+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:19:36.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:19:35 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:19:36.105 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:19:35 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:19:36.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:19:36 smithi115 conmon[90393]: 2022-05-14T03:19:36.102+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:19:37.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:19:36 smithi047 ceph-mon[77147]: pgmap v7306: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:19:37.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:19:36 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:19:37.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:19:36 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 13909 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:19:37.099 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:19:36 smithi115 ceph-mon[78593]: pgmap v7306: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:19:37.099 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:19:36 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:19:37.100 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:19:36 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 13909 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:19:37.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:19:37 smithi115 conmon[90393]: 2022-05-14T03:19:37.097+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:19:38.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:19:37 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:19:38.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:19:37 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T03:19:38.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:19:37 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T03:19:38.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:19:37 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T03:19:38.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:19:37 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:19:38.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:19:37 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T03:19:38.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:19:37 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T03:19:38.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:19:37 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T03:19:38.542 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:19:38 smithi047 conmon[78521]: 2022-05-14T03:19:38.095+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:19:38.542 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:19:38 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T03:19:38.542 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:19:38 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:19:38.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:19:38 smithi047 conmon[78521]: yield 2022-05-14T03:19:38.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:19:38 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:19:38.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:19:38 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:19:38.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:19:38 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:19:38.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:19:38 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T03:19:38.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:19:38 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:19:38.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:19:38 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T03:19:38.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:19:38 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:19:38.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:19:38 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T03:19:38.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:19:38 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:19:38.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:19:38 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T03:19:38.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:19:38 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:19:38.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:19:38 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T03:19:38.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:19:38 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:19:38.548 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:19:38 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:19:38.548 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:19:38 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:19:38.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:19:38 smithi115 conmon[90393]: 2022-05-14T03:19:38.141+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:19:39.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:19:38 smithi047 ceph-mon[77147]: pgmap v7307: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:19:39.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:19:38 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:19:39.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:19:38 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:19:39.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:19:38 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:19:39.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:19:38 smithi115 ceph-mon[78593]: pgmap v7307: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:19:39.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:19:38 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:19:39.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:19:38 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:19:39.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:19:38 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:19:39.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:19:39 smithi115 conmon[90393]: 2022-05-14T03:19:39.119+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:19:40.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:19:39 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:19:40.042 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T03:19:40.042 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:19:40.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T03:19:40.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:19:40.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:19:40.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:19:40.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T03:19:40.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:19:40.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T03:19:40.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:19:40.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T03:19:40.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:19:40.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T03:19:40.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:19:40.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T03:19:40.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:19:40.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:19:40.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:19:40.046 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:19:39 smithi047 ceph-mon[77147]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T03:19:40.047 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:19:39 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:19:40.098 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:19:39 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:19:40.098 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T03:19:40.099 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:19:40.099 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T03:19:40.099 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:19:40.099 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:19:40.100 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:19:40.100 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T03:19:40.100 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:19:40.100 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T03:19:40.100 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:19:40.101 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T03:19:40.101 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:19:40.101 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T03:19:40.102 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:19:40.102 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T03:19:40.102 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:19:40.102 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:19:40.103 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:19:40.103 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:19:39 smithi115 ceph-mon[78593]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T03:19:40.104 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:19:39 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:19:40.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:19:40 smithi115 conmon[90393]: 2022-05-14T03:19:40.096+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:19:41.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:19:40 smithi047 ceph-mon[77147]: pgmap v7308: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:19:41.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:19:40 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:19:41.074 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:19:40 smithi115 ceph-mon[78593]: pgmap v7308: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:19:41.074 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:19:40 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:19:41.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:19:41 smithi115 conmon[90393]: 2022-05-14T03:19:41.072+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:19:41.542 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:19:41 smithi047 conmon[78521]: 2022-05-14T03:19:41.167+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:19:41.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:19:41 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T03:19:41.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:19:41 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:19:41.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:19:41 smithi047 conmon[78521]: yield 2022-05-14T03:19:41.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:19:41 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:19:41.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:19:41 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:19:41.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:19:41 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:19:41.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:19:41 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T03:19:41.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:19:41 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:19:41.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:19:41 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T03:19:41.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:19:41 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:19:41.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:19:41 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T03:19:41.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:19:41 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:19:41.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:19:41 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T03:19:41.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:19:41 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:19:41.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:19:41 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T03:19:41.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:19:41 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:19:41.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:19:41 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:19:41.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:19:41 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:19:41.547 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:19:41 smithi047 conmon[78521]: 2022-05-14T03:19:41.168+0000 7f5341f7c700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T03:19:41.547 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:19:41 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T03:19:41.547 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:19:41 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:19:41.547 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:19:41 smithi047 conmon[78521]: yield 2022-05-14T03:19:41.548 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:19:41 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:19:41.548 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:19:41 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:19:41.548 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:19:41 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:19:41.548 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:19:41 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T03:19:41.548 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:19:41 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:19:41.549 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:19:41 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T03:19:41.549 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:19:41 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:19:41.549 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:19:41 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T03:19:41.549 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:19:41 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:19:41.550 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:19:41 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T03:19:41.550 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:19:41 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:19:41.551 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:19:41 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T03:19:41.551 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:19:41 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:19:41.551 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:19:41 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:19:41.551 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:19:41 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:19:41.552 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:19:41 smithi047 conmon[78521]: 2022-05-14T03:19:41.552 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:19:41 smithi047 conmon[78521]: During handling of the above exception, another exception occurred: 2022-05-14T03:19:41.552 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:19:41 smithi047 conmon[78521]: 2022-05-14T03:19:41.552 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:19:41 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T03:19:41.553 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:19:41 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T03:19:41.553 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:19:41 smithi047 conmon[78521]: return f(*arg) 2022-05-14T03:19:41.553 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:19:41 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T03:19:41.553 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:19:41 smithi047 conmon[78521]: self._write_client_files(client_files, host) 2022-05-14T03:19:41.554 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:19:41 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T03:19:41.554 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:19:41 smithi047 conmon[78521]: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T03:19:41.554 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:19:41 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T03:19:41.554 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:19:41 smithi047 conmon[78521]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T03:19:41.554 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:19:41 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T03:19:41.555 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:19:41 smithi047 conmon[78521]: return self.event_loop.get_result(coro) 2022-05-14T03:19:41.555 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:19:41 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T03:19:41.555 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:19:41 smithi047 conmon[78521]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T03:19:41.555 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:19:41 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T03:19:41.556 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:19:41 smithi047 conmon[78521]: return self.__get_result() 2022-05-14T03:19:41.556 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:19:41 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T03:19:41.556 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:19:41 smithi047 conmon[78521]: raise self._exception 2022-05-14T03:19:41.556 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:19:41 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T03:19:41.556 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:19:41 smithi047 conmon[78521]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T03:19:41.557 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:19:41 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T03:19:41.557 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:19:41 smithi047 conmon[78521]: conn = await self._remote_connection(host, addr) 2022-05-14T03:19:41.557 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:19:41 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T03:19:41.557 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:19:41 smithi047 conmon[78521]: raise 2022-05-14T03:19:41.558 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:19:41 smithi047 conmon[78521]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T03:19:41.558 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:19:41 smithi047 conmon[78521]: self.gen.throw(type, value, traceback) 2022-05-14T03:19:41.558 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:19:41 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T03:19:41.558 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:19:41 smithi047 conmon[78521]: raise OrchestratorError(msg) 2022-05-14T03:19:41.558 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:19:41 smithi047 conmon[78521]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:19:42.027 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:19:41 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:19:42.028 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:19:41 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 13915 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:19:42.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:19:41 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:19:42.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:19:41 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 13915 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:19:42.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:19:42 smithi115 conmon[90393]: 2022-05-14T03:19:42.025+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:19:43.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:19:42 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:19:43.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T03:19:43.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:19:43.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T03:19:43.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:19:43.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:19:43.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:19:43.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T03:19:43.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:19:43.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T03:19:43.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:19:43.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T03:19:43.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:19:43.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T03:19:43.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:19:43.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T03:19:43.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:19:43.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:19:43.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:19:43.048 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:19:42 smithi047 ceph-mon[77147]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T03:19:43.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T03:19:43.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:19:43.049 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T03:19:43.049 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:19:43.049 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:19:43.049 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:19:43.049 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T03:19:43.050 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:19:43.050 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T03:19:43.050 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:19:43.050 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T03:19:43.050 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:19:43.051 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T03:19:43.051 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:19:43.051 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T03:19:43.051 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:19:43.051 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:19:43.052 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:19:43.052 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T03:19:43.052 INFO:journalctl@ceph.mon.b.smithi047.stdout: During handling of the above exception, another exception occurred: 2022-05-14T03:19:43.053 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T03:19:43.053 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T03:19:43.053 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T03:19:43.053 INFO:journalctl@ceph.mon.b.smithi047.stdout: return f(*arg) 2022-05-14T03:19:43.054 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T03:19:43.054 INFO:journalctl@ceph.mon.b.smithi047.stdout: self._write_client_files(client_files, host) 2022-05-14T03:19:43.054 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T03:19:43.054 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T03:19:43.054 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T03:19:43.055 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T03:19:43.055 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T03:19:43.055 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.event_loop.get_result(coro) 2022-05-14T03:19:43.056 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T03:19:43.056 INFO:journalctl@ceph.mon.b.smithi047.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T03:19:43.056 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T03:19:43.056 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.__get_result() 2022-05-14T03:19:43.057 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T03:19:43.057 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise self._exception 2022-05-14T03:19:43.057 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T03:19:43.057 INFO:journalctl@ceph.mon.b.smithi047.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T03:19:43.057 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T03:19:43.058 INFO:journalctl@ceph.mon.b.smithi047.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T03:19:43.058 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T03:19:43.058 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise 2022-05-14T03:19:43.058 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T03:19:43.058 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.gen.throw(type, value, traceback) 2022-05-14T03:19:43.059 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T03:19:43.059 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OrchestratorError(msg) 2022-05-14T03:19:43.059 INFO:journalctl@ceph.mon.b.smithi047.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:19:43.059 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:19:42 smithi047 ceph-mon[77147]: pgmap v7309: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:19:43.060 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:19:42 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:19:43.060 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:19:42 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:19:43.060 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:19:42 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:19:43.074 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:19:42 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:19:43.075 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T03:19:43.075 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:19:43.075 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T03:19:43.075 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:19:43.076 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:19:43.076 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:19:43.076 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T03:19:43.077 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:19:43.077 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T03:19:43.077 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:19:43.077 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T03:19:43.077 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:19:43.078 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T03:19:43.078 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:19:43.078 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T03:19:43.078 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:19:43.079 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:19:43.079 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:19:43.079 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:19:42 smithi115 ceph-mon[78593]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T03:19:43.079 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T03:19:43.079 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:19:43.080 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T03:19:43.080 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:19:43.080 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:19:43.080 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:19:43.081 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T03:19:43.081 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:19:43.081 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T03:19:43.082 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:19:43.082 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T03:19:43.082 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:19:43.083 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T03:19:43.083 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:19:43.083 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T03:19:43.084 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:19:43.084 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:19:43.084 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:19:43.084 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T03:19:43.085 INFO:journalctl@ceph.mon.c.smithi115.stdout: During handling of the above exception, another exception occurred: 2022-05-14T03:19:43.085 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T03:19:43.085 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T03:19:43.085 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T03:19:43.085 INFO:journalctl@ceph.mon.c.smithi115.stdout: return f(*arg) 2022-05-14T03:19:43.086 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T03:19:43.086 INFO:journalctl@ceph.mon.c.smithi115.stdout: self._write_client_files(client_files, host) 2022-05-14T03:19:43.087 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T03:19:43.087 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T03:19:43.087 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T03:19:43.087 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T03:19:43.087 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T03:19:43.088 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.event_loop.get_result(coro) 2022-05-14T03:19:43.088 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T03:19:43.088 INFO:journalctl@ceph.mon.c.smithi115.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T03:19:43.088 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T03:19:43.089 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.__get_result() 2022-05-14T03:19:43.089 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T03:19:43.089 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise self._exception 2022-05-14T03:19:43.090 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T03:19:43.090 INFO:journalctl@ceph.mon.c.smithi115.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T03:19:43.090 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T03:19:43.090 INFO:journalctl@ceph.mon.c.smithi115.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T03:19:43.090 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T03:19:43.091 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise 2022-05-14T03:19:43.091 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T03:19:43.091 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.gen.throw(type, value, traceback) 2022-05-14T03:19:43.091 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T03:19:43.091 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OrchestratorError(msg) 2022-05-14T03:19:43.092 INFO:journalctl@ceph.mon.c.smithi115.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:19:43.092 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:19:42 smithi115 ceph-mon[78593]: pgmap v7309: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:19:43.092 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:19:42 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:19:43.093 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:19:42 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:19:43.093 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:19:42 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:19:43.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:19:43 smithi115 conmon[90393]: 2022-05-14T03:19:43.072+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:19:44.041 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:19:43 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:19:44.102 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:19:43 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:19:44.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:19:44 smithi115 conmon[90393]: 2022-05-14T03:19:44.100+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:19:45.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:19:44 smithi047 ceph-mon[77147]: pgmap v7310: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:19:45.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:19:44 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:19:45.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:19:44 smithi115 ceph-mon[78593]: pgmap v7310: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:19:45.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:19:44 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:19:45.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:19:45 smithi115 conmon[90393]: 2022-05-14T03:19:45.108+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:19:46.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:19:45 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:19:46.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:19:45 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:19:46.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:19:46 smithi115 conmon[90393]: 2022-05-14T03:19:46.146+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:19:47.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:19:46 smithi047 ceph-mon[77147]: pgmap v7311: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:19:47.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:19:46 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:19:47.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:19:46 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 13919 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:19:47.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:19:46 smithi115 ceph-mon[78593]: pgmap v7311: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:19:47.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:19:46 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:19:47.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:19:46 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 13919 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:19:47.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:19:47 smithi115 conmon[90393]: 2022-05-14T03:19:47.124+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:19:48.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:19:47 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:19:48.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:19:47 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:19:48.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:19:48 smithi115 conmon[90393]: 2022-05-14T03:19:48.154+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:19:49.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:19:48 smithi047 ceph-mon[77147]: pgmap v7312: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:19:49.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:19:48 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:19:49.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:19:48 smithi115 ceph-mon[78593]: pgmap v7312: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:19:49.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:19:48 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:19:49.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:19:49 smithi115 conmon[90393]: 2022-05-14T03:19:49.201+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:19:50.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:19:49 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:19:50.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:19:49 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:19:50.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:19:50 smithi115 conmon[90393]: 2022-05-14T03:19:50.173+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:19:51.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:19:50 smithi047 ceph-mon[77147]: pgmap v7313: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:19:51.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:19:50 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:19:51.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:19:50 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:19:51.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:19:50 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:19:51.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:19:50 smithi115 ceph-mon[78593]: pgmap v7313: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:19:51.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:19:50 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:19:51.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:19:50 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:19:51.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:19:50 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:19:51.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:19:51 smithi115 conmon[90393]: 2022-05-14T03:19:51.203+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:19:52.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:19:51 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:19:52.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:19:51 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 13925 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:19:52.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:19:51 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:19:52.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:19:51 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 13925 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:19:52.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:19:52 smithi115 conmon[90393]: 2022-05-14T03:19:52.220+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:19:53.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:19:52 smithi047 ceph-mon[77147]: pgmap v7314: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:19:53.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:19:52 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:19:53.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:19:52 smithi115 ceph-mon[78593]: pgmap v7314: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:19:53.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:19:52 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:19:53.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:19:53 smithi115 conmon[90393]: 2022-05-14T03:19:53.175+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:19:54.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:19:53 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:19:54.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:19:53 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:19:54.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:19:54 smithi115 conmon[90393]: 2022-05-14T03:19:54.128+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:19:55.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:19:54 smithi047 ceph-mon[77147]: pgmap v7315: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:19:55.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:19:54 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:19:55.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:19:54 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:19:55.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:19:54 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:19:55.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:19:54 smithi115 ceph-mon[78593]: pgmap v7315: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:19:55.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:19:54 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:19:55.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:19:54 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:19:55.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:19:54 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:19:55.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:19:55 smithi115 conmon[90393]: 2022-05-14T03:19:55.157+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:19:56.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:19:55 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:19:56.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:19:55 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:19:56.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:19:56 smithi115 conmon[90393]: 2022-05-14T03:19:56.150+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:19:57.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:19:56 smithi115 ceph-mon[78593]: pgmap v7316: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:19:57.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:19:56 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:19:57.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:19:56 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 13930 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:19:57.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:19:56 smithi047 ceph-mon[77147]: pgmap v7316: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:19:57.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:19:56 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:19:57.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:19:56 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 13930 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:19:57.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:19:57 smithi115 conmon[90393]: 2022-05-14T03:19:57.161+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:19:58.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:19:57 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:19:58.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:19:57 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:19:58.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:19:58 smithi115 conmon[90393]: 2022-05-14T03:19:58.194+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:19:59.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:19:58 smithi115 ceph-mon[78593]: pgmap v7317: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:19:59.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:19:58 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:19:59.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:19:58 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:19:59.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:19:58 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:19:59.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:19:58 smithi047 ceph-mon[77147]: pgmap v7317: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:19:59.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:19:58 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:19:59.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:19:58 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:19:59.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:19:58 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:19:59.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:19:59 smithi115 conmon[90393]: 2022-05-14T03:19:59.159+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:20:00.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:19:59 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:20:00.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:19:59 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:20:00.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:20:00 smithi047 conmon[77123]: 2022-05-14T03:19:59.998+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : Health detail: HEALTH_ERR 1 filesystem is degraded; 1 mds daemon damaged; 1 MDSs are read only; 1 MDSs report slow metadata IOs; 1/3 mons down, quorum b,c; Reduced data availability: 1 pg inactive; 9 slow ops, oldest one blocked for 13935 sec, osd.5 has slow ops 2022-05-14T03:20:00.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:20:00 smithi047 conmon[77123]: 2022-05-14T03:19:59.998+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : [WRN] FS_DEGRADED: 1 filesystem is degraded 2022-05-14T03:20:00.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:20:00 smithi047 conmon[77123]: 2022-05-14T03:19:59.998+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : fs cephfs is degraded 2022-05-14T03:20:00.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:20:00 smithi047 conmon[77123]: 2022-05-14T03:19:59.998+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : [ERR] MDS_DAMAGE: 1 mds daemon damaged 2022-05-14T03:20:00.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:20:00 smithi047 conmon[77123]: 2022-05-14T03:19:59.998+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : fs cephfs mds.2 is damaged 2022-05-14T03:20:00.295 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:20:00 smithi047 conmon[77123]: 2022-05-14T03:19:59.998+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : [WRN] MDS_READ_ONLY: 1 MDSs are read only 2022-05-14T03:20:00.295 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:20:00 smithi047 conmon[77123]: 2022-05-14T03:19:59.998+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : mds.l(mds.0): MDS in read-only mode 2022-05-14T03:20:00.295 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:20:00 smithi047 conmon[77123]: 2022-05-14T03:19:59.998+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : [WRN] MDS_SLOW_METADATA_IO: 1 MDSs report slow metadata IOs 2022-05-14T03:20:00.296 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:20:00 smithi047 conmon[77123]: 2022-05-14T03:19:59.998+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : mds.l(mds.0): 4 slow metadata IOs are blocked > 180 secs, oldest blocked for 14233 secs 2022-05-14T03:20:00.296 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:20:00 smithi047 conmon[77123]: 2022-05-14T03:19:59.998+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : [WRN] MON_DOWN: 1/3 mons down, quorum b,c 2022-05-14T03:20:00.296 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:20:00 smithi047 conmon[77123]: 2022-05-14T03:19:59.998+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : mon.a (rank 0) addr [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] is down (out of quorum) 2022-05-14T03:20:00.296 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:20:00 smithi047 conmon[77123]: 2022-05-14T03:19:59.998+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : [WRN] PG_AVAILABILITY: Reduced data availability: 1 pg inactive 2022-05-14T03:20:00.297 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:20:00 smithi047 conmon[77123]: 2022-05-14T03:20:00.297 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:20:00 smithi047 conmon[77123]: 2022-05-14T03:19:59.998+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : pg 2.1 is stuck inactive for 4h, current state unknown, last acting [] 2022-05-14T03:20:00.297 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:20:00 smithi047 conmon[77123]: 2022-05-14T03:19:59.998+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : [WRN] SLOW_OPS: 9 slow ops, oldest one blocked for 13935 sec, osd.5 has slow ops 2022-05-14T03:20:00.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:20:00 smithi115 conmon[90393]: 2022-05-14T03:20:00.147+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:20:01.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:20:00 smithi115 ceph-mon[78593]: pgmap v7318: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:20:01.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:20:00 smithi115 ceph-mon[78593]: Health detail: HEALTH_ERR 1 filesystem is degraded; 1 mds daemon damaged; 1 MDSs are read only; 1 MDSs report slow metadata IOs; 1/3 mons down, quorum b,c; Reduced data availability: 1 pg inactive; 9 slow ops, oldest one blocked for 13935 sec, osd.5 has slow ops 2022-05-14T03:20:01.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:20:00 smithi115 ceph-mon[78593]: [WRN] FS_DEGRADED: 1 filesystem is degraded 2022-05-14T03:20:01.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:20:00 smithi115 ceph-mon[78593]: fs cephfs is degraded 2022-05-14T03:20:01.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:20:00 smithi115 ceph-mon[78593]: [ERR] MDS_DAMAGE: 1 mds daemon damaged 2022-05-14T03:20:01.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:20:00 smithi115 ceph-mon[78593]: fs cephfs mds.2 is damaged 2022-05-14T03:20:01.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:20:00 smithi115 ceph-mon[78593]: [WRN] MDS_READ_ONLY: 1 MDSs are read only 2022-05-14T03:20:01.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:20:00 smithi115 ceph-mon[78593]: mds.l(mds.0): MDS in read-only mode 2022-05-14T03:20:01.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:20:00 smithi115 ceph-mon[78593]: [WRN] MDS_SLOW_METADATA_IO: 1 MDSs report slow metadata IOs 2022-05-14T03:20:01.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:20:00 smithi115 ceph-mon[78593]: mds.l(mds.0): 4 slow metadata IOs are blocked > 180 secs, oldest blocked for 14233 secs 2022-05-14T03:20:01.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:20:00 smithi115 ceph-mon[78593]: [WRN] MON_DOWN: 1/3 mons down, quorum b,c 2022-05-14T03:20:01.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:20:00 smithi115 ceph-mon[78593]: mon.a (rank 0) addr [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] is down (out of quorum) 2022-05-14T03:20:01.112 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:20:00 smithi115 ceph-mon[78593]: [WRN] PG_AVAILABILITY: Reduced data availability: 1 pg inactive 2022-05-14T03:20:01.112 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:20:00 smithi115 ceph-mon[78593]: pg 2.1 is stuck inactive for 4h, current state unknown, last acting [] 2022-05-14T03:20:01.112 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:20:00 smithi115 ceph-mon[78593]: [WRN] SLOW_OPS: 9 slow ops, oldest one blocked for 13935 sec, osd.5 has slow ops 2022-05-14T03:20:01.114 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:20:00 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:20:01.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:20:00 smithi047 ceph-mon[77147]: pgmap v7318: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:20:01.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:20:00 smithi047 ceph-mon[77147]: Health detail: HEALTH_ERR 1 filesystem is degraded; 1 mds daemon damaged; 1 MDSs are read only; 1 MDSs report slow metadata IOs; 1/3 mons down, quorum b,c; Reduced data availability: 1 pg inactive; 9 slow ops, oldest one blocked for 13935 sec, osd.5 has slow ops 2022-05-14T03:20:01.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:20:00 smithi047 ceph-mon[77147]: [WRN] FS_DEGRADED: 1 filesystem is degraded 2022-05-14T03:20:01.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:20:00 smithi047 ceph-mon[77147]: fs cephfs is degraded 2022-05-14T03:20:01.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:20:00 smithi047 ceph-mon[77147]: [ERR] MDS_DAMAGE: 1 mds daemon damaged 2022-05-14T03:20:01.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:20:00 smithi047 ceph-mon[77147]: fs cephfs mds.2 is damaged 2022-05-14T03:20:01.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:20:00 smithi047 ceph-mon[77147]: [WRN] MDS_READ_ONLY: 1 MDSs are read only 2022-05-14T03:20:01.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:20:00 smithi047 ceph-mon[77147]: mds.l(mds.0): MDS in read-only mode 2022-05-14T03:20:01.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:20:00 smithi047 ceph-mon[77147]: [WRN] MDS_SLOW_METADATA_IO: 1 MDSs report slow metadata IOs 2022-05-14T03:20:01.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:20:00 smithi047 ceph-mon[77147]: mds.l(mds.0): 4 slow metadata IOs are blocked > 180 secs, oldest blocked for 14233 secs 2022-05-14T03:20:01.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:20:00 smithi047 ceph-mon[77147]: [WRN] MON_DOWN: 1/3 mons down, quorum b,c 2022-05-14T03:20:01.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:20:00 smithi047 ceph-mon[77147]: mon.a (rank 0) addr [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] is down (out of quorum) 2022-05-14T03:20:01.295 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:20:00 smithi047 ceph-mon[77147]: [WRN] PG_AVAILABILITY: Reduced data availability: 1 pg inactive 2022-05-14T03:20:01.295 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:20:00 smithi047 ceph-mon[77147]: pg 2.1 is stuck inactive for 4h, current state unknown, last acting [] 2022-05-14T03:20:01.295 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:20:00 smithi047 ceph-mon[77147]: [WRN] SLOW_OPS: 9 slow ops, oldest one blocked for 13935 sec, osd.5 has slow ops 2022-05-14T03:20:01.295 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:20:00 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:20:01.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:20:01 smithi115 conmon[90393]: 2022-05-14T03:20:01.192+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:20:02.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:20:01 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:20:02.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:20:01 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 13935 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:20:02.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:20:01 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:20:02.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:20:01 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 13935 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:20:02.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:20:02 smithi115 conmon[90393]: 2022-05-14T03:20:02.216+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:20:03.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:20:02 smithi115 ceph-mon[78593]: pgmap v7319: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:20:03.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:20:02 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:20:03.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:20:02 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:20:03.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:20:02 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:20:03.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:20:02 smithi047 ceph-mon[77147]: pgmap v7319: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:20:03.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:20:02 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:20:03.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:20:02 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:20:03.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:20:02 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:20:03.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:20:03 smithi115 conmon[90393]: 2022-05-14T03:20:03.202+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:20:04.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:20:03 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:20:04.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:20:03 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:20:04.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:20:04 smithi115 conmon[90393]: 2022-05-14T03:20:04.193+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:20:05.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:20:04 smithi115 ceph-mon[78593]: pgmap v7320: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:20:05.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:20:04 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:20:05.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:20:04 smithi047 ceph-mon[77147]: pgmap v7320: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:20:05.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:20:04 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:20:05.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:20:05 smithi115 conmon[90393]: 2022-05-14T03:20:05.170+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:20:06.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:20:05 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:20:06.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:20:05 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:20:06.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:20:06 smithi115 conmon[90393]: 2022-05-14T03:20:06.129+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:20:07.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:20:06 smithi115 ceph-mon[78593]: pgmap v7321: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:20:07.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:20:06 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:20:07.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:20:06 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 13940 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:20:07.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:20:06 smithi047 ceph-mon[77147]: pgmap v7321: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:20:07.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:20:06 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:20:07.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:20:06 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 13940 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:20:07.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:20:07 smithi115 conmon[90393]: 2022-05-14T03:20:07.155+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:20:08.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:20:07 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:20:08.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:20:07 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:20:08.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:20:08 smithi115 conmon[90393]: 2022-05-14T03:20:08.134+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:20:09.100 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:20:08 smithi115 ceph-mon[78593]: pgmap v7322: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:20:09.101 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:20:08 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:20:09.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:20:08 smithi047 ceph-mon[77147]: pgmap v7322: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:20:09.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:20:08 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:20:09.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:20:09 smithi115 conmon[90393]: 2022-05-14T03:20:09.096+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:20:10.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:20:09 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:20:10.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:20:10 smithi115 conmon[90393]: 2022-05-14T03:20:10.079+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:20:10.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:20:09 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:20:11.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:20:11 smithi115 conmon[90393]: 2022-05-14T03:20:11.032+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:20:11.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:20:10 smithi115 ceph-mon[78593]: pgmap v7323: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:20:11.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:20:10 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:20:11.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:20:10 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:20:11.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:20:10 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:20:11.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:20:10 smithi047 ceph-mon[77147]: pgmap v7323: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:20:11.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:20:10 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:20:11.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:20:10 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:20:11.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:20:10 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:20:12.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:20:12 smithi115 conmon[90393]: 2022-05-14T03:20:12.066+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:20:12.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:20:11 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:20:12.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:20:11 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 13945 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:20:12.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:20:11 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:20:12.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:20:11 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 13945 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:20:13.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:20:13 smithi115 conmon[90393]: 2022-05-14T03:20:13.074+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:20:13.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:20:12 smithi115 ceph-mon[78593]: pgmap v7324: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:20:13.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:20:12 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:20:13.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:20:12 smithi047 ceph-mon[77147]: pgmap v7324: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:20:13.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:20:12 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:20:14.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:20:13 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:20:14.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:20:13 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:20:14.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:20:14 smithi115 conmon[90393]: 2022-05-14T03:20:14.108+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:20:15.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:20:14 smithi115 ceph-mon[78593]: pgmap v7325: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:20:15.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:20:14 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:20:15.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:20:14 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:20:15.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:20:14 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:20:15.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:20:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T03:20:15.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:20:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T03:20:15.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:20:14 smithi047 ceph-mon[77147]: pgmap v7325: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:20:15.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:20:14 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:20:15.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:20:14 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:20:15.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:20:14 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:20:15.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:20:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T03:20:15.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:20:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T03:20:15.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:20:15 smithi115 conmon[90393]: 2022-05-14T03:20:15.114+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:20:16.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:20:16 smithi115 conmon[90393]: 2022-05-14T03:20:16.103+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:20:16.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:20:15 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:20:16.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:20:15 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:20:17.121 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:20:16 smithi115 ceph-mon[78593]: pgmap v7326: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:20:17.122 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:20:16 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:20:17.122 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:20:16 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 13949 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:20:17.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:20:16 smithi047 ceph-mon[77147]: pgmap v7326: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:20:17.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:20:16 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:20:17.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:20:16 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 13949 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:20:17.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:20:17 smithi115 conmon[90393]: 2022-05-14T03:20:17.120+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:20:18.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:20:17 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:20:18.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:20:18 smithi115 conmon[90393]: 2022-05-14T03:20:18.074+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:20:18.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:20:17 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:20:19.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:20:18 smithi047 ceph-mon[77147]: pgmap v7327: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:20:19.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:20:18 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:20:19.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:20:18 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:20:19.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:20:18 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:20:19.360 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:20:19 smithi115 conmon[90393]: 2022-05-14T03:20:19.080+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:20:19.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:20:18 smithi115 ceph-mon[78593]: pgmap v7327: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:20:19.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:20:18 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:20:19.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:20:18 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:20:19.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:20:18 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:20:20.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:20:19 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:20:20.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:20:20 smithi115 conmon[90393]: 2022-05-14T03:20:20.064+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:20:20.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:20:19 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:20:21.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:20:20 smithi047 ceph-mon[77147]: pgmap v7328: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:20:21.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:20:20 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:20:21.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:20:21 smithi115 conmon[90393]: 2022-05-14T03:20:21.036+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:20:21.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:20:20 smithi115 ceph-mon[78593]: pgmap v7328: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:20:21.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:20:20 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:20:22.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:20:21 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:20:22.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:20:21 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 13954 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:20:22.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:20:21 smithi115 conmon[90393]: 2022-05-14T03:20:21.994+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:20:22.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:20:21 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:20:22.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:20:21 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 13954 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:20:23.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:20:22 smithi047 ceph-mon[77147]: pgmap v7329: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:20:23.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:20:22 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:20:23.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:20:22 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:20:23.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:20:22 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:20:23.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:20:23 smithi115 conmon[90393]: 2022-05-14T03:20:23.001+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:20:23.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:20:22 smithi115 ceph-mon[78593]: pgmap v7329: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:20:23.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:20:22 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:20:23.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:20:22 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:20:23.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:20:22 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:20:24.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:20:23 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:20:24.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:20:23 smithi047 ceph-mon[77147]: pgmap v7330: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:20:24.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:20:23 smithi115 conmon[90393]: 2022-05-14T03:20:23.995+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:20:24.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:20:23 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:20:24.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:20:23 smithi115 ceph-mon[78593]: pgmap v7330: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:20:25.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:20:24 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:20:25.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:20:24 smithi115 conmon[90393]: 2022-05-14T03:20:24.958+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:20:25.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:20:24 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:20:26.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:20:25 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:20:26.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:20:25 smithi047 ceph-mon[77147]: pgmap v7331: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:20:26.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:20:25 smithi115 conmon[90393]: 2022-05-14T03:20:25.962+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:20:26.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:20:25 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:20:26.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:20:25 smithi115 ceph-mon[78593]: pgmap v7331: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:20:27.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:20:26 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:20:27.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:20:26 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 13960 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:20:27.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:20:26 smithi115 conmon[90393]: 2022-05-14T03:20:26.949+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:20:27.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:20:26 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:20:27.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:20:26 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 13960 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:20:28.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:20:27 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:20:28.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:20:27 smithi047 ceph-mon[77147]: pgmap v7332: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:20:28.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:20:27 smithi115 conmon[90393]: 2022-05-14T03:20:27.974+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:20:28.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:20:27 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:20:28.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:20:27 smithi115 ceph-mon[78593]: pgmap v7332: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:20:29.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:20:28 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:20:29.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:20:29 smithi115 conmon[90393]: 2022-05-14T03:20:29.022+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:20:29.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:20:28 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:20:30.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:20:29 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:20:30.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:20:29 smithi047 ceph-mon[77147]: pgmap v7333: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:20:30.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:20:30 smithi115 conmon[90393]: 2022-05-14T03:20:30.062+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:20:30.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:20:29 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:20:30.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:20:29 smithi115 ceph-mon[78593]: pgmap v7333: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:20:31.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:20:30 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:20:31.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:20:30 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:20:31.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:20:30 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:20:31.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:20:31 smithi115 conmon[90393]: 2022-05-14T03:20:31.057+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:20:31.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:20:30 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:20:31.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:20:30 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:20:31.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:20:30 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:20:32.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:20:31 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:20:32.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:20:31 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 13965 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:20:32.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:20:31 smithi047 ceph-mon[77147]: pgmap v7334: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:20:32.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:20:32 smithi115 conmon[90393]: 2022-05-14T03:20:32.016+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:20:32.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:20:31 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:20:32.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:20:31 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 13965 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:20:32.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:20:31 smithi115 ceph-mon[78593]: pgmap v7334: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:20:33.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:20:33 smithi115 conmon[90393]: 2022-05-14T03:20:33.019+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:20:33.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:20:33 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:20:33.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:20:33 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:20:34.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:20:34 smithi115 conmon[90393]: 2022-05-14T03:20:34.056+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:20:34.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:20:34 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:20:34.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:20:34 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:20:34.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:20:34 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:20:34.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:20:34 smithi047 ceph-mon[77147]: pgmap v7335: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:20:34.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:20:34 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:20:34.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:20:34 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:20:34.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:20:34 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:20:34.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:20:34 smithi115 ceph-mon[78593]: pgmap v7335: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:20:35.360 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:20:35 smithi115 conmon[90393]: 2022-05-14T03:20:35.093+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:20:35.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:20:35 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:20:35.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:20:35 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:20:36.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:20:36 smithi115 conmon[90393]: 2022-05-14T03:20:36.065+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:20:36.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:20:36 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:20:36.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:20:36 smithi047 ceph-mon[77147]: pgmap v7336: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:20:36.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:20:36 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:20:36.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:20:36 smithi115 ceph-mon[78593]: pgmap v7336: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:20:37.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:20:37 smithi115 conmon[90393]: 2022-05-14T03:20:37.088+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:20:37.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:20:37 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:20:37.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:20:37 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 13969 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:20:37.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:20:37 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:20:37.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:20:37 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 13969 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:20:38.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:20:38 smithi115 conmon[90393]: 2022-05-14T03:20:38.073+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:20:38.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:20:38 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:20:38.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:20:38 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:20:38.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:20:38 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:20:38.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:20:38 smithi047 ceph-mon[77147]: pgmap v7337: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:20:38.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:20:38 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:20:38.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:20:38 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:20:38.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:20:38 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:20:38.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:20:38 smithi115 ceph-mon[78593]: pgmap v7337: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:20:39.360 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:20:39 smithi115 conmon[90393]: 2022-05-14T03:20:39.064+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:20:39.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:20:39 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:20:39.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:20:39 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:20:40.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:20:40 smithi115 conmon[90393]: 2022-05-14T03:20:40.068+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:20:40.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:20:40 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:20:40.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:20:40 smithi047 ceph-mon[77147]: pgmap v7338: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:20:40.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:20:40 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:20:40.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:20:40 smithi115 ceph-mon[78593]: pgmap v7338: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:20:41.276 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:20:41 smithi115 conmon[90393]: 2022-05-14T03:20:41.023+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:20:41.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:20:41 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:20:41.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:20:41 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T03:20:41.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:20:41 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T03:20:41.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:20:41 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T03:20:41.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:20:41 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:20:41.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:20:41 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T03:20:41.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:20:41 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T03:20:41.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:20:41 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T03:20:42.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:20:42 smithi115 conmon[90393]: 2022-05-14T03:20:42.024+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:20:42.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:20:42 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:20:42.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:20:42 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:20:42.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:20:42 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:20:42.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:20:42 smithi047 ceph-mon[77147]: pgmap v7339: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:20:42.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:20:42 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:20:42.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:20:42 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:20:42.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:20:42 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:20:42.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:20:42 smithi115 ceph-mon[78593]: pgmap v7339: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:20:43.365 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:20:43 smithi115 conmon[90393]: 2022-05-14T03:20:43.014+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:20:43.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:20:43 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:20:43.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:20:43 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:20:44.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:20:44 smithi115 conmon[90393]: 2022-05-14T03:20:44.020+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:20:44.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:20:44 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:20:44.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:20:44 smithi047 ceph-mon[77147]: pgmap v7340: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:20:44.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:20:44 smithi047 conmon[78521]: 2022-05-14T03:20:44.271+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:20:44.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:20:44 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T03:20:44.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:20:44 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:20:44.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:20:44 smithi047 conmon[78521]: yield 2022-05-14T03:20:44.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:20:44 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:20:44.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:20:44 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:20:44.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:20:44 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:20:44.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:20:44 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T03:20:44.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:20:44 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:20:44.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:20:44 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T03:20:44.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:20:44 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:20:44.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:20:44 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T03:20:44.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:20:44 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:20:44.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:20:44 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T03:20:44.547 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:20:44 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:20:44.547 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:20:44 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T03:20:44.547 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:20:44 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:20:44.547 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:20:44 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:20:44.548 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:20:44 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:20:44.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:20:44 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:20:44.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:20:44 smithi115 ceph-mon[78593]: pgmap v7340: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:20:45.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:20:45 smithi115 conmon[90393]: 2022-05-14T03:20:45.070+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:20:45.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:20:45 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:20:45.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:20:45 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:20:45.793 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T03:20:45.793 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:20:45.793 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T03:20:45.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:20:45.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:20:45.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:20:45.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T03:20:45.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:20:45.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T03:20:45.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:20:45.797 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T03:20:45.797 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:20:45.797 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T03:20:45.798 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:20:45.798 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T03:20:45.798 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:20:45.798 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:20:45.798 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:20:45.799 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:20:45 smithi047 ceph-mon[77147]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T03:20:45.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:20:45 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:20:45.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:20:45 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:20:45.860 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T03:20:45.860 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:20:45.860 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T03:20:45.860 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:20:45.860 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:20:45.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:20:45.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T03:20:45.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:20:45.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T03:20:45.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:20:45.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T03:20:45.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:20:45.863 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T03:20:45.863 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:20:45.863 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T03:20:45.863 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:20:45.864 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:20:45.864 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:20:45.864 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:20:45 smithi115 ceph-mon[78593]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T03:20:46.493 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:20:46 smithi115 conmon[90393]: 2022-05-14T03:20:46.119+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:20:46.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:20:46 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:20:46.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:20:46 smithi047 ceph-mon[77147]: pgmap v7341: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:20:46.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:20:46 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 13974 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:20:46.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:20:46 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:20:46.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:20:46 smithi115 ceph-mon[78593]: pgmap v7341: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:20:46.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:20:46 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 13974 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:20:47.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:20:47 smithi115 conmon[90393]: 2022-05-14T03:20:47.105+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:20:47.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:20:47 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:20:47.792 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:20:47 smithi047 conmon[78521]: 2022-05-14T03:20:47.344+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:20:47.793 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:20:47 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T03:20:47.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:20:47 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:20:47.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:20:47 smithi047 conmon[78521]: yield 2022-05-14T03:20:47.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:20:47 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:20:47.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:20:47 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:20:47.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:20:47 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:20:47.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:20:47 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T03:20:47.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:20:47 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:20:47.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:20:47 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T03:20:47.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:20:47 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:20:47.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:20:47 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T03:20:47.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:20:47 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:20:47.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:20:47 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T03:20:47.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:20:47 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:20:47.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:20:47 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T03:20:47.797 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:20:47 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:20:47.797 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:20:47 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:20:47.797 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:20:47 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:20:47.797 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:20:47 smithi047 conmon[78521]: 2022-05-14T03:20:47.344+0000 7f5344f82700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T03:20:47.798 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:20:47 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T03:20:47.798 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:20:47 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:20:47.798 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:20:47 smithi047 conmon[78521]: yield 2022-05-14T03:20:47.798 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:20:47 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:20:47.798 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:20:47 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:20:47.799 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:20:47 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:20:47.799 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:20:47 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T03:20:47.799 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:20:47 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:20:47.799 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:20:47 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T03:20:47.800 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:20:47 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:20:47.800 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:20:47 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T03:20:47.800 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:20:47 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:20:47.800 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:20:47 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T03:20:47.800 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:20:47 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:20:47.801 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:20:47 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T03:20:47.801 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:20:47 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:20:47.802 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:20:47 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:20:47.802 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:20:47 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:20:47.802 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:20:47 smithi047 conmon[78521]: 2022-05-14T03:20:47.802 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:20:47 smithi047 conmon[78521]: During handling of the above exception, another exception occurred: 2022-05-14T03:20:47.803 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:20:47 smithi047 conmon[78521]: 2022-05-14T03:20:47.803 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:20:47 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T03:20:47.803 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:20:47 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T03:20:47.803 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:20:47 smithi047 conmon[78521]: return f(*arg) 2022-05-14T03:20:47.803 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:20:47 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T03:20:47.804 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:20:47 smithi047 conmon[78521]: self._write_client_files(client_files, host) 2022-05-14T03:20:47.804 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:20:47 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T03:20:47.804 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:20:47 smithi047 conmon[78521]: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T03:20:47.804 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:20:47 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T03:20:47.804 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:20:47 smithi047 conmon[78521]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T03:20:47.805 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:20:47 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T03:20:47.805 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:20:47 smithi047 conmon[78521]: return self.event_loop.get_result(coro) 2022-05-14T03:20:47.805 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:20:47 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T03:20:47.806 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:20:47 smithi047 conmon[78521]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T03:20:47.806 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:20:47 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T03:20:47.806 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:20:47 smithi047 conmon[78521]: return self.__get_result() 2022-05-14T03:20:47.806 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:20:47 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T03:20:47.807 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:20:47 smithi047 conmon[78521]: raise self._exception 2022-05-14T03:20:47.807 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:20:47 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T03:20:47.807 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:20:47 smithi047 conmon[78521]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T03:20:47.807 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:20:47 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T03:20:47.807 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:20:47 smithi047 conmon[78521]: conn = await self._remote_connection(host, addr) 2022-05-14T03:20:47.808 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:20:47 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T03:20:47.808 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:20:47 smithi047 conmon[78521]: raise 2022-05-14T03:20:47.808 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:20:47 smithi047 conmon[78521]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T03:20:47.808 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:20:47 smithi047 conmon[78521]: self.gen.throw(type, value, traceback) 2022-05-14T03:20:47.808 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:20:47 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T03:20:47.809 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:20:47 smithi047 conmon[78521]: raise OrchestratorError(msg) 2022-05-14T03:20:47.809 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:20:47 smithi047 conmon[78521]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:20:47.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:20:47 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:20:48.499 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:20:48 smithi115 conmon[90393]: 2022-05-14T03:20:48.132+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:20:48.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:20:48 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:20:48.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:20:48 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:20:48.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T03:20:48.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:20:48.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T03:20:48.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:20:48.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:20:48.797 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:20:48.797 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T03:20:48.798 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:20:48.798 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T03:20:48.798 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:20:48.800 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T03:20:48.800 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:20:48.800 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T03:20:48.801 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:20:48.801 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T03:20:48.802 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:20:48.802 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:20:48.802 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:20:48.802 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:20:48 smithi047 ceph-mon[77147]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T03:20:48.803 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T03:20:48.803 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:20:48.803 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T03:20:48.803 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:20:48.804 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:20:48.804 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:20:48.804 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T03:20:48.804 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:20:48.805 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T03:20:48.805 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:20:48.806 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T03:20:48.807 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:20:48.807 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T03:20:48.807 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:20:48.809 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T03:20:48.809 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:20:48.809 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:20:48.809 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:20:48.810 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T03:20:48.810 INFO:journalctl@ceph.mon.b.smithi047.stdout: During handling of the above exception, another exception occurred: 2022-05-14T03:20:48.810 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T03:20:48.810 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T03:20:48.811 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T03:20:48.811 INFO:journalctl@ceph.mon.b.smithi047.stdout: return f(*arg) 2022-05-14T03:20:48.811 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T03:20:48.812 INFO:journalctl@ceph.mon.b.smithi047.stdout: self._write_client_files(client_files, host) 2022-05-14T03:20:48.812 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T03:20:48.812 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T03:20:48.812 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T03:20:48.814 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T03:20:48.814 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T03:20:48.814 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.event_loop.get_result(coro) 2022-05-14T03:20:48.814 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T03:20:48.814 INFO:journalctl@ceph.mon.b.smithi047.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T03:20:48.815 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T03:20:48.815 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.__get_result() 2022-05-14T03:20:48.815 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T03:20:48.815 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise self._exception 2022-05-14T03:20:48.816 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T03:20:48.816 INFO:journalctl@ceph.mon.b.smithi047.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T03:20:48.816 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T03:20:48.816 INFO:journalctl@ceph.mon.b.smithi047.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T03:20:48.817 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T03:20:48.817 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise 2022-05-14T03:20:48.817 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T03:20:48.817 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.gen.throw(type, value, traceback) 2022-05-14T03:20:48.817 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T03:20:48.818 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OrchestratorError(msg) 2022-05-14T03:20:48.818 INFO:journalctl@ceph.mon.b.smithi047.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:20:48.818 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:20:48 smithi047 ceph-mon[77147]: pgmap v7342: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:20:48.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:20:48 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:20:48.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:20:48 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:20:48.860 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T03:20:48.860 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:20:48.860 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T03:20:48.860 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:20:48.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:20:48.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:20:48.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T03:20:48.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:20:48.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T03:20:48.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:20:48.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T03:20:48.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:20:48.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T03:20:48.863 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:20:48.863 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T03:20:48.863 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:20:48.863 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:20:48.863 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:20:48.864 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:20:48 smithi115 ceph-mon[78593]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T03:20:48.865 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T03:20:48.866 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:20:48.866 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T03:20:48.866 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:20:48.866 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:20:48.866 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:20:48.867 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T03:20:48.867 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:20:48.867 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T03:20:48.867 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:20:48.868 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T03:20:48.868 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:20:48.868 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T03:20:48.868 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:20:48.869 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T03:20:48.869 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:20:48.869 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:20:48.869 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:20:48.869 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T03:20:48.870 INFO:journalctl@ceph.mon.c.smithi115.stdout: During handling of the above exception, another exception occurred: 2022-05-14T03:20:48.870 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T03:20:48.870 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T03:20:48.870 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T03:20:48.871 INFO:journalctl@ceph.mon.c.smithi115.stdout: return f(*arg) 2022-05-14T03:20:48.871 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T03:20:48.871 INFO:journalctl@ceph.mon.c.smithi115.stdout: self._write_client_files(client_files, host) 2022-05-14T03:20:48.871 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T03:20:48.872 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T03:20:48.872 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T03:20:48.872 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T03:20:48.872 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T03:20:48.873 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.event_loop.get_result(coro) 2022-05-14T03:20:48.873 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T03:20:48.873 INFO:journalctl@ceph.mon.c.smithi115.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T03:20:48.873 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T03:20:48.874 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.__get_result() 2022-05-14T03:20:48.874 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T03:20:48.874 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise self._exception 2022-05-14T03:20:48.874 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T03:20:48.874 INFO:journalctl@ceph.mon.c.smithi115.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T03:20:48.875 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T03:20:48.875 INFO:journalctl@ceph.mon.c.smithi115.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T03:20:48.875 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T03:20:48.875 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise 2022-05-14T03:20:48.876 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T03:20:48.876 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.gen.throw(type, value, traceback) 2022-05-14T03:20:48.876 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T03:20:48.876 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OrchestratorError(msg) 2022-05-14T03:20:48.876 INFO:journalctl@ceph.mon.c.smithi115.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:20:48.877 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:20:48 smithi115 ceph-mon[78593]: pgmap v7342: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:20:49.500 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:20:49 smithi115 conmon[90393]: 2022-05-14T03:20:49.119+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:20:49.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:20:49 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:20:49.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:20:49 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:20:50.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:20:50 smithi115 conmon[90393]: 2022-05-14T03:20:50.083+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:20:50.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:20:50 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:20:50.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:20:50 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:20:50.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:20:50 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:20:50.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:20:50 smithi047 ceph-mon[77147]: pgmap v7343: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:20:50.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:20:50 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:20:50.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:20:50 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:20:50.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:20:50 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:20:50.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:20:50 smithi115 ceph-mon[78593]: pgmap v7343: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:20:51.512 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:20:51 smithi115 conmon[90393]: 2022-05-14T03:20:51.123+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:20:51.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:20:51 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:20:51.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:20:51 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 13985 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:20:51.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:20:51 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:20:51.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:20:51 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 13985 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:20:52.516 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:20:52 smithi115 conmon[90393]: 2022-05-14T03:20:52.135+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:20:52.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:20:52 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:20:52.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:20:52 smithi047 ceph-mon[77147]: pgmap v7344: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:20:52.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:20:52 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:20:52.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:20:52 smithi115 ceph-mon[78593]: pgmap v7344: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:20:53.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:20:53 smithi115 conmon[90393]: 2022-05-14T03:20:53.096+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:20:53.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:20:53 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:20:53.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:20:53 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:20:54.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:20:54 smithi115 conmon[90393]: 2022-05-14T03:20:54.046+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:20:54.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:20:54 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:20:54.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:20:54 smithi047 ceph-mon[77147]: pgmap v7345: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:20:54.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:20:54 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:20:54.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:20:54 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:20:54.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:20:54 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:20:54.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:20:54 smithi115 ceph-mon[78593]: pgmap v7345: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:20:54.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:20:54 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:20:54.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:20:54 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:20:55.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:20:55 smithi115 conmon[90393]: 2022-05-14T03:20:55.087+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:20:55.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:20:55 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:20:55.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:20:55 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:20:56.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:20:56 smithi115 conmon[90393]: 2022-05-14T03:20:56.052+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:20:56.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:20:56 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:20:56.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:20:56 smithi047 ceph-mon[77147]: pgmap v7346: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:20:56.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:20:56 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 13990 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:20:56.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:20:56 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:20:56.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:20:56 smithi115 ceph-mon[78593]: pgmap v7346: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:20:56.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:20:56 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 13990 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:20:57.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:20:57 smithi115 conmon[90393]: 2022-05-14T03:20:57.009+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:20:57.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:20:57 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:20:57.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:20:57 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:20:58.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:20:58 smithi115 conmon[90393]: 2022-05-14T03:20:58.028+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:20:58.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:20:58 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:20:58.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:20:58 smithi047 ceph-mon[77147]: pgmap v7347: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:20:58.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:20:58 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:20:58.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:20:58 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:20:58.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:20:58 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:20:58.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:20:58 smithi115 ceph-mon[78593]: pgmap v7347: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:20:58.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:20:58 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:20:58.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:20:58 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:20:59.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:20:59 smithi115 conmon[90393]: 2022-05-14T03:20:59.064+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:20:59.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:20:59 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:20:59.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:20:59 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:21:00.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:21:00 smithi115 conmon[90393]: 2022-05-14T03:21:00.049+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:21:00.863 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:21:00 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:21:00.863 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:21:00 smithi115 ceph-mon[78593]: pgmap v7348: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:21:01.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:21:00 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:21:01.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:21:00 smithi047 ceph-mon[77147]: pgmap v7348: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:21:01.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:21:01 smithi115 conmon[90393]: 2022-05-14T03:21:01.079+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:21:01.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:21:01 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:21:01.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:21:01 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 13994 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:21:02.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:21:01 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:21:02.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:21:01 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 13994 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:21:02.552 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:21:02 smithi115 conmon[90393]: 2022-05-14T03:21:02.108+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:21:02.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:21:02 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:21:02.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:21:02 smithi115 ceph-mon[78593]: pgmap v7349: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:21:02.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:21:02 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:21:02.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:21:02 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:21:03.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:21:02 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:21:03.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:21:02 smithi047 ceph-mon[77147]: pgmap v7349: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:21:03.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:21:02 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:21:03.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:21:02 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:21:03.559 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:21:03 smithi115 conmon[90393]: 2022-05-14T03:21:03.143+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:21:03.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:21:03 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:21:04.045 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:21:03 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:21:04.562 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:21:04 smithi115 conmon[90393]: 2022-05-14T03:21:04.110+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:21:04.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:21:04 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:21:04.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:21:04 smithi115 ceph-mon[78593]: pgmap v7350: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:21:05.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:21:04 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:21:05.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:21:04 smithi047 ceph-mon[77147]: pgmap v7350: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:21:05.563 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:21:05 smithi115 conmon[90393]: 2022-05-14T03:21:05.158+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:21:05.866 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:21:05 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:21:06.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:21:05 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:21:06.565 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:21:06 smithi115 conmon[90393]: 2022-05-14T03:21:06.159+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:21:06.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:21:06 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:21:06.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:21:06 smithi115 ceph-mon[78593]: pgmap v7351: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:21:06.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:21:06 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 14000 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:21:07.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:21:06 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:21:07.045 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:21:06 smithi047 ceph-mon[77147]: pgmap v7351: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:21:07.045 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:21:06 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 14000 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:21:07.569 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:21:07 smithi115 conmon[90393]: 2022-05-14T03:21:07.201+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:21:07.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:21:07 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:21:08.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:21:07 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:21:08.571 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:21:08 smithi115 conmon[90393]: 2022-05-14T03:21:08.244+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:21:08.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:21:08 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:21:08.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:21:08 smithi115 ceph-mon[78593]: pgmap v7352: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:21:09.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:21:08 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:21:09.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:21:08 smithi047 ceph-mon[77147]: pgmap v7352: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:21:09.576 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:21:09 smithi115 conmon[90393]: 2022-05-14T03:21:09.236+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:21:09.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:21:09 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:21:10.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:21:09 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:21:10.578 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:21:10 smithi115 conmon[90393]: 2022-05-14T03:21:10.192+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:21:10.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:21:10 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:21:10.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:21:10 smithi115 ceph-mon[78593]: pgmap v7353: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:21:10.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:21:10 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:21:10.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:21:10 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:21:11.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:21:10 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:21:11.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:21:10 smithi047 ceph-mon[77147]: pgmap v7353: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:21:11.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:21:10 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:21:11.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:21:10 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:21:11.579 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:21:11 smithi115 conmon[90393]: 2022-05-14T03:21:11.223+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:21:11.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:21:11 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:21:11.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:21:11 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 14005 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:21:12.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:21:11 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:21:12.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:21:11 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 14005 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:21:12.580 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:21:12 smithi115 conmon[90393]: 2022-05-14T03:21:12.196+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:21:12.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:21:12 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:21:12.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:21:12 smithi115 ceph-mon[78593]: pgmap v7354: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:21:13.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:21:12 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:21:13.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:21:12 smithi047 ceph-mon[77147]: pgmap v7354: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:21:13.586 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:21:13 smithi115 conmon[90393]: 2022-05-14T03:21:13.191+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:21:13.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:21:13 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:21:14.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:21:13 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:21:14.587 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:21:14 smithi115 conmon[90393]: 2022-05-14T03:21:14.216+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:21:14.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:21:14 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:21:14.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:21:14 smithi115 ceph-mon[78593]: pgmap v7355: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:21:14.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:21:14 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:21:14.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:21:14 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:21:14.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:21:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T03:21:14.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:21:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T03:21:15.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:21:14 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:21:15.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:21:14 smithi047 ceph-mon[77147]: pgmap v7355: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:21:15.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:21:14 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:21:15.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:21:14 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:21:15.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:21:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T03:21:15.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:21:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T03:21:15.592 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:21:15 smithi115 conmon[90393]: 2022-05-14T03:21:15.201+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:21:15.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:21:15 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:21:16.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:21:15 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:21:16.594 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:21:16 smithi115 conmon[90393]: 2022-05-14T03:21:16.217+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:21:16.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:21:16 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:21:16.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:21:16 smithi115 ceph-mon[78593]: pgmap v7356: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:21:16.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:21:16 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 14010 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:21:17.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:21:16 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:21:17.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:21:16 smithi047 ceph-mon[77147]: pgmap v7356: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:21:17.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:21:16 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 14010 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:21:17.600 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:21:17 smithi115 conmon[90393]: 2022-05-14T03:21:17.259+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:21:17.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:21:17 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:21:18.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:21:17 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:21:18.601 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:21:18 smithi115 conmon[90393]: 2022-05-14T03:21:18.226+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:21:18.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:21:18 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:21:18.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:21:18 smithi115 ceph-mon[78593]: pgmap v7357: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:21:18.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:21:18 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:21:18.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:21:18 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:21:19.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:21:18 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:21:19.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:21:18 smithi047 ceph-mon[77147]: pgmap v7357: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:21:19.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:21:18 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:21:19.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:21:18 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:21:19.606 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:21:19 smithi115 conmon[90393]: 2022-05-14T03:21:19.231+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:21:19.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:21:19 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:21:20.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:21:19 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:21:20.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:21:20 smithi115 conmon[90393]: 2022-05-14T03:21:20.231+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:21:20.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:21:20 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:21:20.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:21:20 smithi115 ceph-mon[78593]: pgmap v7358: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:21:21.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:21:20 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:21:21.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:21:20 smithi047 ceph-mon[77147]: pgmap v7358: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:21:21.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:21:21 smithi115 conmon[90393]: 2022-05-14T03:21:21.255+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:21:22.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:21:21 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:21:22.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:21:21 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 14015 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:21:22.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:21:21 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:21:22.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:21:21 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 14015 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:21:22.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:21:22 smithi115 conmon[90393]: 2022-05-14T03:21:22.255+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:21:23.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:21:22 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:21:23.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:21:22 smithi047 ceph-mon[77147]: pgmap v7359: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:21:23.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:21:22 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:21:23.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:21:22 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:21:23.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:21:22 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:21:23.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:21:22 smithi115 ceph-mon[78593]: pgmap v7359: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:21:23.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:21:22 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:21:23.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:21:22 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:21:23.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:21:23 smithi115 conmon[90393]: 2022-05-14T03:21:23.233+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:21:24.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:21:23 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:21:24.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:21:23 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:21:24.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:21:24 smithi115 conmon[90393]: 2022-05-14T03:21:24.198+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:21:25.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:21:24 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:21:25.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:21:24 smithi047 ceph-mon[77147]: pgmap v7360: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:21:25.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:21:24 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:21:25.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:21:24 smithi115 ceph-mon[78593]: pgmap v7360: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:21:25.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:21:25 smithi115 conmon[90393]: 2022-05-14T03:21:25.169+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:21:26.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:21:25 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:21:26.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:21:25 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:21:26.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:21:26 smithi115 conmon[90393]: 2022-05-14T03:21:26.143+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:21:27.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:21:26 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:21:27.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:21:26 smithi047 ceph-mon[77147]: pgmap v7361: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:21:27.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:21:26 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 14020 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:21:27.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:21:26 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:21:27.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:21:26 smithi115 ceph-mon[78593]: pgmap v7361: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:21:27.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:21:26 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 14020 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:21:27.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:21:27 smithi115 conmon[90393]: 2022-05-14T03:21:27.151+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:21:28.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:21:27 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:21:28.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:21:27 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:21:28.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:21:28 smithi115 conmon[90393]: 2022-05-14T03:21:28.176+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:21:29.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:21:28 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:21:29.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:21:28 smithi047 ceph-mon[77147]: pgmap v7362: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:21:29.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:21:28 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:21:29.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:21:28 smithi115 ceph-mon[78593]: pgmap v7362: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:21:29.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:21:29 smithi115 conmon[90393]: 2022-05-14T03:21:29.133+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:21:30.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:21:29 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:21:30.106 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:21:29 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:21:30.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:21:30 smithi115 conmon[90393]: 2022-05-14T03:21:30.104+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:21:31.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:21:30 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:21:31.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:21:30 smithi047 ceph-mon[77147]: pgmap v7363: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:21:31.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:21:30 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:21:31.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:21:30 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:21:31.081 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:21:30 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:21:31.081 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:21:30 smithi115 ceph-mon[78593]: pgmap v7363: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:21:31.082 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:21:30 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:21:31.082 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:21:30 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:21:31.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:21:31 smithi115 conmon[90393]: 2022-05-14T03:21:31.079+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:21:32.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:21:31 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:21:32.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:21:31 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 14025 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:21:32.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:21:31 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:21:32.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:21:31 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 14025 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:21:32.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:21:32 smithi115 conmon[90393]: 2022-05-14T03:21:32.114+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:21:33.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:21:32 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:21:33.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:21:32 smithi047 ceph-mon[77147]: pgmap v7364: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:21:33.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:21:32 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:21:33.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:21:32 smithi115 ceph-mon[78593]: pgmap v7364: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:21:33.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:21:33 smithi115 conmon[90393]: 2022-05-14T03:21:33.125+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:21:34.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:21:33 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:21:34.093 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:21:33 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:21:34.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:21:34 smithi115 conmon[90393]: 2022-05-14T03:21:34.091+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:21:35.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:21:34 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:21:35.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:21:34 smithi047 ceph-mon[77147]: pgmap v7365: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:21:35.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:21:34 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:21:35.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:21:34 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:21:35.079 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:21:34 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:21:35.080 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:21:34 smithi115 ceph-mon[78593]: pgmap v7365: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:21:35.080 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:21:34 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:21:35.080 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:21:34 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:21:35.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:21:35 smithi115 conmon[90393]: 2022-05-14T03:21:35.077+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:21:36.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:21:35 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:21:36.106 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:21:35 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:21:36.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:21:36 smithi115 conmon[90393]: 2022-05-14T03:21:36.104+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:21:37.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:21:36 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:21:37.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:21:36 smithi047 ceph-mon[77147]: pgmap v7366: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:21:37.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:21:36 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 14030 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:21:37.069 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:21:36 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:21:37.069 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:21:36 smithi115 ceph-mon[78593]: pgmap v7366: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:21:37.070 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:21:36 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 14030 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:21:37.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:21:37 smithi115 conmon[90393]: 2022-05-14T03:21:37.067+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:21:38.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:21:37 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:21:38.085 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:21:37 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:21:38.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:21:38 smithi115 conmon[90393]: 2022-05-14T03:21:38.083+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:21:39.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:21:38 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:21:39.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:21:38 smithi047 ceph-mon[77147]: pgmap v7367: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:21:39.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:21:38 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:21:39.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:21:38 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:21:39.081 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:21:38 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:21:39.082 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:21:38 smithi115 ceph-mon[78593]: pgmap v7367: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:21:39.082 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:21:38 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:21:39.082 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:21:38 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:21:39.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:21:39 smithi115 conmon[90393]: 2022-05-14T03:21:39.079+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:21:40.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:21:39 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:21:40.053 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:21:39 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:21:40.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:21:40 smithi115 conmon[90393]: 2022-05-14T03:21:40.051+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:21:41.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:21:40 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:21:41.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:21:40 smithi047 ceph-mon[77147]: pgmap v7368: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:21:41.094 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:21:40 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:21:41.094 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:21:40 smithi115 ceph-mon[78593]: pgmap v7368: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:21:41.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:21:41 smithi115 conmon[90393]: 2022-05-14T03:21:41.092+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:21:42.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:21:41 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:21:42.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:21:41 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 14034 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:21:42.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:21:41 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:21:42.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:21:41 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 14034 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:21:42.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:21:42 smithi115 conmon[90393]: 2022-05-14T03:21:42.135+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:21:43.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:21:42 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:21:43.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:21:42 smithi047 ceph-mon[77147]: pgmap v7369: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:21:43.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:21:42 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:21:43.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:21:42 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:21:43.103 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:21:42 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:21:43.103 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:21:42 smithi115 ceph-mon[78593]: pgmap v7369: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:21:43.103 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:21:42 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:21:43.104 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:21:42 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:21:43.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:21:43 smithi115 conmon[90393]: 2022-05-14T03:21:43.101+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:21:44.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:21:43 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:21:44.083 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:21:43 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:21:44.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:21:44 smithi115 conmon[90393]: 2022-05-14T03:21:44.081+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:21:45.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:21:44 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:21:45.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:21:44 smithi047 ceph-mon[77147]: pgmap v7370: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:21:45.082 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:21:44 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:21:45.082 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:21:44 smithi115 ceph-mon[78593]: pgmap v7370: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:21:45.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:21:45 smithi115 conmon[90393]: 2022-05-14T03:21:45.080+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:21:46.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:21:45 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:21:46.087 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:21:45 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:21:46.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:21:46 smithi115 conmon[90393]: 2022-05-14T03:21:46.086+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:21:47.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:21:46 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:21:47.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:21:46 smithi047 ceph-mon[77147]: pgmap v7371: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:21:47.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:21:46 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 14040 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:21:47.078 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:21:46 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:21:47.078 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:21:46 smithi115 ceph-mon[78593]: pgmap v7371: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:21:47.079 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:21:46 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 14040 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:21:47.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:21:47 smithi115 conmon[90393]: 2022-05-14T03:21:47.076+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:21:47.946 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:21:47 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:21:47.947 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:21:47 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T03:21:47.947 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:21:47 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T03:21:47.947 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:21:47 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T03:21:47.961 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:21:47 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:21:47.962 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:21:47 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T03:21:47.962 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:21:47 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T03:21:47.962 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:21:47 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T03:21:48.253 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:21:48 smithi115 conmon[90393]: 2022-05-14T03:21:48.072+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:21:49.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:21:48 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:21:49.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:21:48 smithi047 ceph-mon[77147]: pgmap v7372: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:21:49.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:21:48 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' 2022-05-14T03:21:49.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:21:48 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' 2022-05-14T03:21:49.089 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:21:48 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:21:49.090 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:21:48 smithi115 ceph-mon[78593]: pgmap v7372: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:21:49.090 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:21:48 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' 2022-05-14T03:21:49.090 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:21:48 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' 2022-05-14T03:21:49.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:21:49 smithi115 conmon[90393]: 2022-05-14T03:21:49.087+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:21:50.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:21:49 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:21:50.101 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:21:49 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:21:50.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:21:50 smithi115 conmon[90393]: 2022-05-14T03:21:50.099+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:21:50.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:21:50 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:21:50.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:21:50 smithi047 ceph-mon[77147]: pgmap v7373: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:21:50.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:21:50 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:21:50.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:21:50 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:21:50.793 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:21:50 smithi047 conmon[78521]: 2022-05-14T03:21:50.448+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:21:50.793 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:21:50 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T03:21:50.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:21:50 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:21:50.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:21:50 smithi047 conmon[78521]: yield 2022-05-14T03:21:50.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:21:50 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:21:50.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:21:50 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:21:50.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:21:50 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:21:50.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:21:50 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T03:21:50.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:21:50 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:21:50.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:21:50 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T03:21:50.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:21:50 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:21:50.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:21:50 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T03:21:50.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:21:50 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:21:50.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:21:50 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T03:21:50.797 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:21:50 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:21:50.797 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:21:50 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T03:21:50.797 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:21:50 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:21:50.797 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:21:50 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:21:50.797 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:21:50 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:21:51.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:21:50 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:21:51.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:21:50 smithi115 ceph-mon[78593]: pgmap v7373: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:21:51.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:21:50 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:21:51.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:21:50 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:21:51.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:21:51 smithi115 conmon[90393]: 2022-05-14T03:21:51.144+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:21:52.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:21:51 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:21:52.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:21:51 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:21:52.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T03:21:52.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:21:52.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T03:21:52.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:21:52.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:21:52.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:21:52.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T03:21:52.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:21:52.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T03:21:52.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:21:52.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T03:21:52.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:21:52.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T03:21:52.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:21:52.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T03:21:52.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:21:52.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:21:52.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:21:52.048 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:21:51 smithi047 ceph-mon[77147]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T03:21:52.048 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:21:51 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 14044 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:21:52.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:21:51 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:21:52.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:21:51 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:21:52.109 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T03:21:52.109 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:21:52.109 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T03:21:52.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:21:52.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:21:52.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:21:52.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T03:21:52.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:21:52.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T03:21:52.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:21:52.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T03:21:52.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:21:52.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T03:21:52.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:21:52.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T03:21:52.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:21:52.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:21:52.113 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:21:52.113 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:21:51 smithi115 ceph-mon[78593]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T03:21:52.113 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:21:51 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 14044 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:21:52.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:21:52 smithi115 conmon[90393]: 2022-05-14T03:21:52.190+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:21:53.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:21:52 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:21:53.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:21:52 smithi047 ceph-mon[77147]: pgmap v7374: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:21:53.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:21:52 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:21:53.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:21:52 smithi115 ceph-mon[78593]: pgmap v7374: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:21:53.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:21:53 smithi115 conmon[90393]: 2022-05-14T03:21:53.150+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:21:53.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:21:53 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:21:53.792 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:21:53 smithi047 conmon[78521]: 2022-05-14T03:21:53.520+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:21:53.793 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:21:53 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T03:21:53.793 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:21:53 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:21:53.793 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:21:53 smithi047 conmon[78521]: yield 2022-05-14T03:21:53.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:21:53 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:21:53.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:21:53 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:21:53.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:21:53 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:21:53.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:21:53 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T03:21:53.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:21:53 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:21:53.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:21:53 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T03:21:53.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:21:53 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:21:53.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:21:53 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T03:21:53.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:21:53 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:21:53.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:21:53 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T03:21:53.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:21:53 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:21:53.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:21:53 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T03:21:53.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:21:53 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:21:53.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:21:53 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:21:53.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:21:53 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:21:53.797 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:21:53 smithi047 conmon[78521]: 2022-05-14T03:21:53.521+0000 7f5344781700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T03:21:53.797 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:21:53 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T03:21:53.797 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:21:53 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:21:53.797 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:21:53 smithi047 conmon[78521]: yield 2022-05-14T03:21:53.798 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:21:53 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:21:53.798 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:21:53 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:21:53.798 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:21:53 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:21:53.798 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:21:53 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T03:21:53.798 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:21:53 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:21:53.799 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:21:53 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T03:21:53.799 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:21:53 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:21:53.799 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:21:53 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T03:21:53.799 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:21:53 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:21:53.799 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:21:53 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T03:21:53.800 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:21:53 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:21:53.800 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:21:53 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T03:21:53.800 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:21:53 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:21:53.800 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:21:53 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:21:53.800 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:21:53 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:21:53.801 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:21:53 smithi047 conmon[78521]: 2022-05-14T03:21:53.801 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:21:53 smithi047 conmon[78521]: During handling of the above exception, another exception occurred: 2022-05-14T03:21:53.801 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:21:53 smithi047 conmon[78521]: 2022-05-14T03:21:53.801 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:21:53 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T03:21:53.802 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:21:53 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T03:21:53.802 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:21:53 smithi047 conmon[78521]: return f(*arg) 2022-05-14T03:21:53.802 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:21:53 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T03:21:53.802 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:21:53 smithi047 conmon[78521]: self._write_client_files(client_files, host) 2022-05-14T03:21:53.802 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:21:53 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T03:21:53.803 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:21:53 smithi047 conmon[78521]: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T03:21:53.803 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:21:53 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T03:21:53.803 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:21:53 smithi047 conmon[78521]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T03:21:53.803 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:21:53 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T03:21:53.803 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:21:53 smithi047 conmon[78521]: return self.event_loop.get_result(coro) 2022-05-14T03:21:53.804 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:21:53 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T03:21:53.804 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:21:53 smithi047 conmon[78521]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T03:21:53.804 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:21:53 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T03:21:53.804 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:21:53 smithi047 conmon[78521]: return self.__get_result() 2022-05-14T03:21:53.804 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:21:53 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T03:21:53.805 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:21:53 smithi047 conmon[78521]: raise self._exception 2022-05-14T03:21:53.805 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:21:53 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T03:21:53.805 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:21:53 smithi047 conmon[78521]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T03:21:53.805 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:21:53 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T03:21:53.806 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:21:53 smithi047 conmon[78521]: conn = await self._remote_connection(host, addr) 2022-05-14T03:21:53.806 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:21:53 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T03:21:53.806 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:21:53 smithi047 conmon[78521]: raise 2022-05-14T03:21:53.806 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:21:53 smithi047 conmon[78521]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T03:21:53.806 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:21:53 smithi047 conmon[78521]: self.gen.throw(type, value, traceback) 2022-05-14T03:21:53.807 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:21:53 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T03:21:53.807 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:21:53 smithi047 conmon[78521]: raise OrchestratorError(msg) 2022-05-14T03:21:53.807 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:21:53 smithi047 conmon[78521]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:21:54.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:21:53 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:21:54.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:21:54 smithi115 conmon[90393]: 2022-05-14T03:21:54.155+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:21:55.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:21:54 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:21:55.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:21:54 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:21:55.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T03:21:55.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:21:55.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T03:21:55.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:21:55.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:21:55.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:21:55.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T03:21:55.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:21:55.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T03:21:55.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:21:55.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T03:21:55.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:21:55.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T03:21:55.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:21:55.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T03:21:55.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:21:55.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:21:55.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:21:55.048 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:21:54 smithi047 ceph-mon[77147]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T03:21:55.049 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T03:21:55.049 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:21:55.049 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T03:21:55.049 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:21:55.051 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:21:55.051 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:21:55.051 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T03:21:55.051 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:21:55.052 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T03:21:55.052 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:21:55.052 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T03:21:55.053 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:21:55.053 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T03:21:55.054 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:21:55.054 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T03:21:55.054 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:21:55.054 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:21:55.055 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:21:55.055 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T03:21:55.055 INFO:journalctl@ceph.mon.b.smithi047.stdout: During handling of the above exception, another exception occurred: 2022-05-14T03:21:55.055 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T03:21:55.055 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T03:21:55.056 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T03:21:55.056 INFO:journalctl@ceph.mon.b.smithi047.stdout: return f(*arg) 2022-05-14T03:21:55.056 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T03:21:55.056 INFO:journalctl@ceph.mon.b.smithi047.stdout: self._write_client_files(client_files, host) 2022-05-14T03:21:55.057 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T03:21:55.057 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T03:21:55.059 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T03:21:55.060 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T03:21:55.060 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T03:21:55.061 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.event_loop.get_result(coro) 2022-05-14T03:21:55.061 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T03:21:55.061 INFO:journalctl@ceph.mon.b.smithi047.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T03:21:55.061 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T03:21:55.062 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.__get_result() 2022-05-14T03:21:55.062 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T03:21:55.062 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise self._exception 2022-05-14T03:21:55.063 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T03:21:55.063 INFO:journalctl@ceph.mon.b.smithi047.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T03:21:55.063 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T03:21:55.063 INFO:journalctl@ceph.mon.b.smithi047.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T03:21:55.063 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T03:21:55.064 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise 2022-05-14T03:21:55.064 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T03:21:55.064 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.gen.throw(type, value, traceback) 2022-05-14T03:21:55.064 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T03:21:55.065 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OrchestratorError(msg) 2022-05-14T03:21:55.065 INFO:journalctl@ceph.mon.b.smithi047.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:21:55.065 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:21:54 smithi047 ceph-mon[77147]: pgmap v7375: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:21:55.065 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:21:54 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:21:55.065 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:21:54 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:21:55.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:21:54 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:21:55.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:21:54 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:21:55.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T03:21:55.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:21:55.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T03:21:55.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:21:55.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:21:55.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:21:55.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T03:21:55.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:21:55.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T03:21:55.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:21:55.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T03:21:55.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:21:55.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T03:21:55.114 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:21:55.114 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T03:21:55.114 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:21:55.114 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:21:55.115 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:21:55.115 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:21:54 smithi115 ceph-mon[78593]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T03:21:55.115 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T03:21:55.115 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:21:55.116 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T03:21:55.116 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:21:55.116 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:21:55.116 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:21:55.116 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T03:21:55.117 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:21:55.117 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T03:21:55.117 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:21:55.118 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T03:21:55.118 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:21:55.118 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T03:21:55.118 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:21:55.118 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T03:21:55.119 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:21:55.119 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:21:55.119 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:21:55.119 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T03:21:55.120 INFO:journalctl@ceph.mon.c.smithi115.stdout: During handling of the above exception, another exception occurred: 2022-05-14T03:21:55.120 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T03:21:55.120 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T03:21:55.120 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T03:21:55.120 INFO:journalctl@ceph.mon.c.smithi115.stdout: return f(*arg) 2022-05-14T03:21:55.121 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T03:21:55.121 INFO:journalctl@ceph.mon.c.smithi115.stdout: self._write_client_files(client_files, host) 2022-05-14T03:21:55.121 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T03:21:55.121 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T03:21:55.121 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T03:21:55.122 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T03:21:55.122 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T03:21:55.122 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.event_loop.get_result(coro) 2022-05-14T03:21:55.122 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T03:21:55.123 INFO:journalctl@ceph.mon.c.smithi115.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T03:21:55.123 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T03:21:55.123 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.__get_result() 2022-05-14T03:21:55.123 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T03:21:55.123 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise self._exception 2022-05-14T03:21:55.124 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T03:21:55.124 INFO:journalctl@ceph.mon.c.smithi115.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T03:21:55.124 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T03:21:55.124 INFO:journalctl@ceph.mon.c.smithi115.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T03:21:55.125 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T03:21:55.126 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise 2022-05-14T03:21:55.126 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T03:21:55.126 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.gen.throw(type, value, traceback) 2022-05-14T03:21:55.126 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T03:21:55.127 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OrchestratorError(msg) 2022-05-14T03:21:55.127 INFO:journalctl@ceph.mon.c.smithi115.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:21:55.127 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:21:54 smithi115 ceph-mon[78593]: pgmap v7375: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:21:55.127 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:21:54 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:21:55.127 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:21:54 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:21:55.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:21:55 smithi115 conmon[90393]: 2022-05-14T03:21:55.184+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:21:56.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:21:55 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:21:56.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:21:55 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:21:56.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:21:56 smithi115 conmon[90393]: 2022-05-14T03:21:56.145+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:21:57.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:21:56 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:21:57.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:21:56 smithi047 ceph-mon[77147]: pgmap v7376: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:21:57.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:21:56 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:21:57.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:21:56 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 14050 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:21:57.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:21:56 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:21:57.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:21:56 smithi115 ceph-mon[78593]: pgmap v7376: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:21:57.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:21:56 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:21:57.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:21:56 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 14050 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:21:57.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:21:57 smithi115 conmon[90393]: 2022-05-14T03:21:57.127+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:21:58.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:21:57 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:21:58.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:21:57 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:21:58.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:21:58 smithi115 conmon[90393]: 2022-05-14T03:21:58.105+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:21:59.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:21:58 smithi047 ceph-mon[77147]: pgmap v7377: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:21:59.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:21:58 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:21:59.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:21:58 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:21:59.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:21:58 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:21:59.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:21:58 smithi115 ceph-mon[78593]: pgmap v7377: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:21:59.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:21:58 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:21:59.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:21:58 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:21:59.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:21:58 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:21:59.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:21:59 smithi115 conmon[90393]: 2022-05-14T03:21:59.136+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:22:00.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:21:59 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:22:00.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:21:59 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:22:00.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:22:00 smithi115 conmon[90393]: 2022-05-14T03:22:00.169+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:22:01.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:22:00 smithi047 ceph-mon[77147]: pgmap v7378: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:22:01.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:22:00 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:22:01.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:22:00 smithi115 ceph-mon[78593]: pgmap v7378: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:22:01.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:22:00 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:22:01.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:22:01 smithi115 conmon[90393]: 2022-05-14T03:22:01.201+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:22:02.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:22:01 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:22:02.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:22:01 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 14055 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:22:02.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:22:01 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:22:02.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:22:01 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 14055 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:22:02.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:22:02 smithi115 conmon[90393]: 2022-05-14T03:22:02.205+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:22:03.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:22:02 smithi047 ceph-mon[77147]: pgmap v7379: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:22:03.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:22:02 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:22:03.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:22:02 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:22:03.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:22:02 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:22:03.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:22:02 smithi115 ceph-mon[78593]: pgmap v7379: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:22:03.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:22:02 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:22:03.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:22:02 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:22:03.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:22:02 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:22:03.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:22:03 smithi115 conmon[90393]: 2022-05-14T03:22:03.191+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:22:04.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:22:03 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:22:04.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:22:03 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:22:04.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:22:04 smithi115 conmon[90393]: 2022-05-14T03:22:04.207+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:22:05.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:22:04 smithi047 ceph-mon[77147]: pgmap v7380: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:22:05.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:22:04 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:22:05.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:22:04 smithi115 ceph-mon[78593]: pgmap v7380: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:22:05.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:22:04 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:22:05.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:22:05 smithi115 conmon[90393]: 2022-05-14T03:22:05.253+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:22:06.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:22:05 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:22:06.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:22:05 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:22:06.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:22:06 smithi115 conmon[90393]: 2022-05-14T03:22:06.256+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:22:07.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:22:06 smithi047 ceph-mon[77147]: pgmap v7381: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:22:07.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:22:06 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:22:07.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:22:06 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 14060 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:22:07.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:22:06 smithi115 ceph-mon[78593]: pgmap v7381: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:22:07.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:22:06 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:22:07.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:22:06 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 14060 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:22:07.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:22:07 smithi115 conmon[90393]: 2022-05-14T03:22:07.260+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:22:08.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:22:07 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:22:08.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:22:07 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:22:08.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:22:08 smithi115 conmon[90393]: 2022-05-14T03:22:08.301+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:22:09.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:22:08 smithi047 ceph-mon[77147]: pgmap v7382: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:22:09.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:22:08 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:22:09.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:22:08 smithi115 ceph-mon[78593]: pgmap v7382: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:22:09.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:22:08 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:22:09.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:22:09 smithi115 conmon[90393]: 2022-05-14T03:22:09.337+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:22:10.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:22:09 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:22:10.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:22:09 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:22:10.766 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:22:10 smithi115 conmon[90393]: 2022-05-14T03:22:10.357+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:22:11.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:22:10 smithi047 ceph-mon[77147]: pgmap v7383: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:22:11.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:22:10 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:22:11.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:22:10 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:22:11.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:22:10 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:22:11.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:22:10 smithi115 ceph-mon[78593]: pgmap v7383: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:22:11.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:22:10 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:22:11.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:22:10 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:22:11.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:22:10 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:22:11.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:22:11 smithi115 conmon[90393]: 2022-05-14T03:22:11.331+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:22:12.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:22:11 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:22:12.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:22:11 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 14065 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:22:12.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:22:11 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:22:12.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:22:11 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 14065 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:22:12.770 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:22:12 smithi115 conmon[90393]: 2022-05-14T03:22:12.358+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:22:13.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:22:12 smithi047 ceph-mon[77147]: pgmap v7384: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:22:13.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:22:12 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:22:13.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:22:12 smithi115 ceph-mon[78593]: pgmap v7384: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:22:13.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:22:12 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:22:13.775 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:22:13 smithi115 conmon[90393]: 2022-05-14T03:22:13.368+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:22:14.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:22:13 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:22:14.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:22:13 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:22:14.777 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:22:14 smithi115 conmon[90393]: 2022-05-14T03:22:14.393+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:22:15.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:22:14 smithi047 ceph-mon[77147]: pgmap v7385: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:22:15.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:22:14 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:22:15.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:22:14 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:22:15.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:22:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T03:22:15.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:22:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T03:22:15.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:22:14 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:22:15.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:22:14 smithi115 ceph-mon[78593]: pgmap v7385: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:22:15.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:22:14 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:22:15.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:22:14 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:22:15.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:22:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T03:22:15.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:22:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T03:22:15.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:22:14 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:22:15.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:22:15 smithi115 conmon[90393]: 2022-05-14T03:22:15.348+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:22:16.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:22:15 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:22:16.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:22:15 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:22:16.783 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:22:16 smithi115 conmon[90393]: 2022-05-14T03:22:16.380+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:22:17.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:22:16 smithi047 ceph-mon[77147]: pgmap v7386: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:22:17.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:22:16 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:22:17.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:22:16 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 14070 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:22:17.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:22:16 smithi115 ceph-mon[78593]: pgmap v7386: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:22:17.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:22:16 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:22:17.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:22:16 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 14070 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:22:17.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:22:17 smithi115 conmon[90393]: 2022-05-14T03:22:17.347+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:22:18.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:22:17 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:22:18.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:22:17 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:22:18.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:22:18 smithi115 conmon[90393]: 2022-05-14T03:22:18.300+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:22:19.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:22:18 smithi047 ceph-mon[77147]: pgmap v7387: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:22:19.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:22:18 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:22:19.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:22:18 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:22:19.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:22:18 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:22:19.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:22:18 smithi115 ceph-mon[78593]: pgmap v7387: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:22:19.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:22:18 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:22:19.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:22:18 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:22:19.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:22:18 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:22:19.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:22:19 smithi115 conmon[90393]: 2022-05-14T03:22:19.328+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:22:20.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:22:19 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:22:20.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:22:19 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:22:20.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:22:20 smithi115 conmon[90393]: 2022-05-14T03:22:20.289+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:22:21.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:22:20 smithi115 ceph-mon[78593]: pgmap v7388: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:22:21.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:22:20 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:22:21.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:22:20 smithi047 ceph-mon[77147]: pgmap v7388: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:22:21.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:22:20 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:22:21.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:22:21 smithi115 conmon[90393]: 2022-05-14T03:22:21.277+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:22:22.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:22:21 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:22:22.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:22:21 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 14075 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:22:22.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:22:21 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:22:22.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:22:21 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 14075 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:22:22.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:22:22 smithi115 conmon[90393]: 2022-05-14T03:22:22.307+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:22:23.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:22:22 smithi115 ceph-mon[78593]: pgmap v7389: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:22:23.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:22:22 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:22:23.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:22:22 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:22:23.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:22:22 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:22:23.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:22:22 smithi047 ceph-mon[77147]: pgmap v7389: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:22:23.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:22:22 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:22:23.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:22:22 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:22:23.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:22:22 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:22:23.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:22:23 smithi115 conmon[90393]: 2022-05-14T03:22:23.342+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:22:24.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:22:23 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:22:24.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:22:23 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:22:24.810 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:22:24 smithi115 conmon[90393]: 2022-05-14T03:22:24.362+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:22:25.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:22:24 smithi115 ceph-mon[78593]: pgmap v7390: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:22:25.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:22:24 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:22:25.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:22:24 smithi047 ceph-mon[77147]: pgmap v7390: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:22:25.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:22:24 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:22:25.815 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:22:25 smithi115 conmon[90393]: 2022-05-14T03:22:25.383+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:22:26.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:22:25 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:22:26.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:22:25 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:22:26.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:22:26 smithi115 conmon[90393]: 2022-05-14T03:22:26.347+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:22:27.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:22:26 smithi115 ceph-mon[78593]: pgmap v7391: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:22:27.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:22:26 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:22:27.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:22:26 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 14080 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:22:27.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:22:26 smithi047 ceph-mon[77147]: pgmap v7391: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:22:27.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:22:26 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:22:27.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:22:26 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 14080 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:22:27.822 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:22:27 smithi115 conmon[90393]: 2022-05-14T03:22:27.374+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:22:28.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:22:27 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:22:28.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:22:27 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:22:28.824 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:22:28 smithi115 conmon[90393]: 2022-05-14T03:22:28.396+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:22:29.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:22:28 smithi115 ceph-mon[78593]: pgmap v7392: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:22:29.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:22:28 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:22:29.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:22:28 smithi047 ceph-mon[77147]: pgmap v7392: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:22:29.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:22:28 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:22:29.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:22:29 smithi115 conmon[90393]: 2022-05-14T03:22:29.350+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:22:30.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:22:29 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:22:30.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:22:29 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:22:30.831 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:22:30 smithi115 conmon[90393]: 2022-05-14T03:22:30.367+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:22:31.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:22:30 smithi115 ceph-mon[78593]: pgmap v7393: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:22:31.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:22:30 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:22:31.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:22:30 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:22:31.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:22:30 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:22:31.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:22:30 smithi047 ceph-mon[77147]: pgmap v7393: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:22:31.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:22:30 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:22:31.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:22:30 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:22:31.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:22:30 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:22:31.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:22:31 smithi115 conmon[90393]: 2022-05-14T03:22:31.348+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:22:32.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:22:31 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:22:32.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:22:31 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 14085 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:22:32.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:22:31 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:22:32.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:22:31 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 14085 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:22:32.834 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:22:32 smithi115 conmon[90393]: 2022-05-14T03:22:32.398+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:22:33.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:22:32 smithi115 ceph-mon[78593]: pgmap v7394: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:22:33.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:22:32 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:22:33.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:22:32 smithi047 ceph-mon[77147]: pgmap v7394: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:22:33.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:22:32 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:22:33.842 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:22:33 smithi115 conmon[90393]: 2022-05-14T03:22:33.396+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:22:34.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:22:33 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:22:34.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:22:33 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:22:34.840 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:22:34 smithi115 conmon[90393]: 2022-05-14T03:22:34.416+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:22:35.116 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:22:34 smithi115 ceph-mon[78593]: pgmap v7395: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:22:35.117 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:22:34 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:22:35.117 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:22:34 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:22:35.118 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:22:34 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:22:35.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:22:34 smithi047 ceph-mon[77147]: pgmap v7395: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:22:35.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:22:34 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:22:35.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:22:34 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:22:35.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:22:34 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:22:35.845 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:22:35 smithi115 conmon[90393]: 2022-05-14T03:22:35.461+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:22:36.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:22:35 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:22:36.296 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:22:35 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:22:36.849 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:22:36 smithi115 conmon[90393]: 2022-05-14T03:22:36.510+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:22:37.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:22:36 smithi115 ceph-mon[78593]: pgmap v7396: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:22:37.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:22:36 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 14090 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:22:37.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:22:36 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:22:37.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:22:36 smithi047 ceph-mon[77147]: pgmap v7396: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:22:37.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:22:36 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 14090 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:22:37.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:22:36 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:22:37.853 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:22:37 smithi115 conmon[90393]: 2022-05-14T03:22:37.554+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:22:38.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:22:37 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:22:38.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:22:37 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:22:38.853 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:22:38 smithi115 conmon[90393]: 2022-05-14T03:22:38.570+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:22:39.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:22:38 smithi115 ceph-mon[78593]: pgmap v7397: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:22:39.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:22:38 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:22:39.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:22:38 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:22:39.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:22:38 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:22:39.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:22:38 smithi047 ceph-mon[77147]: pgmap v7397: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:22:39.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:22:38 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:22:39.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:22:38 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:22:39.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:22:38 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:22:39.861 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:22:39 smithi115 conmon[90393]: 2022-05-14T03:22:39.605+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:22:40.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:22:39 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:22:40.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:22:39 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:22:40.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:22:40 smithi115 conmon[90393]: 2022-05-14T03:22:40.605+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:22:41.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:22:40 smithi047 ceph-mon[77147]: pgmap v7398: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:22:41.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:22:40 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:22:41.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:22:40 smithi115 ceph-mon[78593]: pgmap v7398: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:22:41.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:22:40 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:22:42.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:22:41 smithi115 conmon[90393]: 2022-05-14T03:22:41.640+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:22:42.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:22:41 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 14095 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:22:42.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:22:41 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:22:42.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:22:41 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 14095 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:22:42.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:22:41 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:22:42.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:22:42 smithi115 conmon[90393]: 2022-05-14T03:22:42.604+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:22:43.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:22:42 smithi047 ceph-mon[77147]: pgmap v7399: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:22:43.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:22:42 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:22:43.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:22:42 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:22:43.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:22:42 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:22:43.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:22:42 smithi115 ceph-mon[78593]: pgmap v7399: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:22:43.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:22:42 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:22:43.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:22:42 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:22:43.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:22:42 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:22:44.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:22:43 smithi115 conmon[90393]: 2022-05-14T03:22:43.636+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:22:44.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:22:43 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:22:44.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:22:43 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:22:44.874 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:22:44 smithi115 conmon[90393]: 2022-05-14T03:22:44.613+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:22:45.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:22:44 smithi047 ceph-mon[77147]: pgmap v7400: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:22:45.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:22:44 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:22:45.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:22:44 smithi115 ceph-mon[78593]: pgmap v7400: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:22:45.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:22:44 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:22:45.881 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:22:45 smithi115 conmon[90393]: 2022-05-14T03:22:45.625+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:22:46.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:22:45 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:22:46.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:22:45 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:22:47.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:22:46 smithi115 conmon[90393]: 2022-05-14T03:22:46.651+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:22:47.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:22:46 smithi115 ceph-mon[78593]: pgmap v7401: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:22:47.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:22:46 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 14100 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:22:47.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:22:46 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:22:47.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:22:46 smithi047 ceph-mon[77147]: pgmap v7401: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:22:47.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:22:46 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 14100 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:22:47.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:22:46 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:22:48.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:22:47 smithi115 conmon[90393]: 2022-05-14T03:22:47.641+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:22:48.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:22:47 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:22:48.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:22:47 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:22:49.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:22:48 smithi115 conmon[90393]: 2022-05-14T03:22:48.671+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:22:49.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:22:48 smithi115 ceph-mon[78593]: pgmap v7402: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:22:49.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:22:48 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:22:49.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:22:48 smithi047 ceph-mon[77147]: pgmap v7402: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:22:49.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:22:48 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:22:50.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:22:49 smithi115 conmon[90393]: 2022-05-14T03:22:49.673+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:22:50.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:22:49 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:22:50.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:22:49 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:22:51.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:22:50 smithi115 conmon[90393]: 2022-05-14T03:22:50.709+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:22:51.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:22:50 smithi115 ceph-mon[78593]: pgmap v7403: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:22:51.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:22:50 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:22:51.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:22:50 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:22:51.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:22:50 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:22:51.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:22:50 smithi047 ceph-mon[77147]: pgmap v7403: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:22:51.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:22:50 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:22:51.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:22:50 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:22:51.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:22:50 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:22:52.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:22:51 smithi115 conmon[90393]: 2022-05-14T03:22:51.698+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:22:52.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:22:51 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 14105 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:22:52.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:22:51 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:22:52.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:22:51 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 14105 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:22:52.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:22:51 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:22:53.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:22:52 smithi115 conmon[90393]: 2022-05-14T03:22:52.746+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:22:53.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:22:52 smithi115 ceph-mon[78593]: pgmap v7404: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:22:53.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:22:52 smithi047 ceph-mon[77147]: pgmap v7404: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:22:53.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:22:53 smithi115 conmon[90393]: 2022-05-14T03:22:53.709+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:22:54.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:22:53 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:22:54.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:22:53 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T03:22:54.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:22:53 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T03:22:54.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:22:53 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T03:22:54.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:22:53 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:22:54.293 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:22:54 smithi047 conmon[78521]: 2022-05-14T03:22:54.127+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:22:54.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:22:54 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T03:22:54.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:22:54 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:22:54.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:22:54 smithi047 conmon[78521]: yield 2022-05-14T03:22:54.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:22:54 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:22:54.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:22:54 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:22:54.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:22:54 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:22:54.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:22:54 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T03:22:54.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:22:54 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:22:54.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:22:54 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T03:22:54.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:22:54 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:22:54.297 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:22:54 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T03:22:54.297 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:22:54 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:22:54.297 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:22:54 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T03:22:54.297 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:22:54 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:22:54.297 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:22:54 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T03:22:54.298 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:22:54 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:22:54.298 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:22:54 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:22:54.298 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:22:54 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:22:54.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:22:53 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:22:54.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:22:53 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T03:22:54.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:22:53 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T03:22:54.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:22:53 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T03:22:54.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:22:53 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:22:55.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:22:54 smithi115 conmon[90393]: 2022-05-14T03:22:54.707+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:22:55.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:22:54 smithi115 ceph-mon[78593]: pgmap v7405: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:22:55.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:22:54 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:22:55.113 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:22:54 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:22:55.114 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:22:54 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:22:55.114 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T03:22:55.114 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:22:55.114 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T03:22:55.115 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:22:55.115 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:22:55.115 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:22:55.115 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T03:22:55.115 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:22:55.116 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T03:22:55.116 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:22:55.116 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T03:22:55.116 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:22:55.116 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T03:22:55.117 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:22:55.117 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T03:22:55.117 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:22:55.117 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:22:55.118 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:22:55.118 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:22:54 smithi115 ceph-mon[78593]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T03:22:55.118 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:22:54 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:22:55.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:22:54 smithi047 ceph-mon[77147]: pgmap v7405: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:22:55.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:22:54 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:22:55.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:22:54 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:22:55.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:22:54 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:22:55.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T03:22:55.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:22:55.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T03:22:55.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:22:55.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:22:55.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:22:55.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T03:22:55.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:22:55.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T03:22:55.300 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:22:55.300 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T03:22:55.300 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:22:55.301 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T03:22:55.301 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:22:55.301 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T03:22:55.301 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:22:55.302 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:22:55.302 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:22:55.302 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:22:54 smithi047 ceph-mon[77147]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T03:22:55.302 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:22:54 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:22:56.110 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:22:55 smithi115 conmon[90393]: 2022-05-14T03:22:55.658+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:22:56.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:22:55 smithi115 ceph-mon[78593]: pgmap v7406: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:22:56.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:22:55 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:22:56.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:22:55 smithi047 ceph-mon[77147]: pgmap v7406: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:22:56.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:22:55 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:22:57.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:22:56 smithi115 conmon[90393]: 2022-05-14T03:22:56.657+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:22:57.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:22:56 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 14110 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:22:57.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:22:56 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:22:57.202 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:22:56 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 14110 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:22:57.203 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:22:56 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:22:57.542 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:22:57 smithi047 conmon[78521]: 2022-05-14T03:22:57.199+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:22:57.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:22:57 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T03:22:57.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:22:57 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:22:57.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:22:57 smithi047 conmon[78521]: yield 2022-05-14T03:22:57.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:22:57 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:22:57.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:22:57 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:22:57.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:22:57 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:22:57.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:22:57 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T03:22:57.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:22:57 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:22:57.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:22:57 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T03:22:57.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:22:57 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:22:57.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:22:57 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T03:22:57.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:22:57 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:22:57.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:22:57 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T03:22:57.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:22:57 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:22:57.547 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:22:57 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T03:22:57.547 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:22:57 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:22:57.547 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:22:57 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:22:57.547 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:22:57 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:22:57.548 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:22:57 smithi047 conmon[78521]: 2022-05-14T03:22:57.200+0000 7f533f777700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T03:22:57.548 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:22:57 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T03:22:57.548 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:22:57 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:22:57.548 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:22:57 smithi047 conmon[78521]: yield 2022-05-14T03:22:57.548 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:22:57 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:22:57.549 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:22:57 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:22:57.549 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:22:57 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:22:57.549 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:22:57 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T03:22:57.549 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:22:57 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:22:57.549 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:22:57 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T03:22:57.550 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:22:57 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:22:57.550 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:22:57 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T03:22:57.550 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:22:57 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:22:57.550 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:22:57 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T03:22:57.551 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:22:57 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:22:57.551 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:22:57 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T03:22:57.551 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:22:57 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:22:57.551 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:22:57 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:22:57.551 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:22:57 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:22:57.552 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:22:57 smithi047 conmon[78521]: 2022-05-14T03:22:57.552 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:22:57 smithi047 conmon[78521]: During handling of the above exception, another exception occurred: 2022-05-14T03:22:57.552 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:22:57 smithi047 conmon[78521]: 2022-05-14T03:22:57.552 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:22:57 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T03:22:57.553 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:22:57 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T03:22:57.553 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:22:57 smithi047 conmon[78521]: return f(*arg) 2022-05-14T03:22:57.553 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:22:57 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T03:22:57.553 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:22:57 smithi047 conmon[78521]: self._write_client_files(client_files, host) 2022-05-14T03:22:57.553 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:22:57 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T03:22:57.554 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:22:57 smithi047 conmon[78521]: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T03:22:57.554 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:22:57 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T03:22:57.554 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:22:57 smithi047 conmon[78521]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T03:22:57.554 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:22:57 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T03:22:57.555 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:22:57 smithi047 conmon[78521]: return self.event_loop.get_result(coro) 2022-05-14T03:22:57.555 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:22:57 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T03:22:57.555 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:22:57 smithi047 conmon[78521]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T03:22:57.556 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:22:57 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T03:22:57.556 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:22:57 smithi047 conmon[78521]: return self.__get_result() 2022-05-14T03:22:57.556 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:22:57 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T03:22:57.556 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:22:57 smithi047 conmon[78521]: raise self._exception 2022-05-14T03:22:57.557 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:22:57 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T03:22:57.557 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:22:57 smithi047 conmon[78521]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T03:22:57.557 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:22:57 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T03:22:57.557 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:22:57 smithi047 conmon[78521]: conn = await self._remote_connection(host, addr) 2022-05-14T03:22:57.557 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:22:57 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T03:22:57.558 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:22:57 smithi047 conmon[78521]: raise 2022-05-14T03:22:57.558 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:22:57 smithi047 conmon[78521]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T03:22:57.558 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:22:57 smithi047 conmon[78521]: self.gen.throw(type, value, traceback) 2022-05-14T03:22:57.558 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:22:57 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T03:22:57.558 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:22:57 smithi047 conmon[78521]: raise OrchestratorError(msg) 2022-05-14T03:22:57.559 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:22:57 smithi047 conmon[78521]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:22:57.915 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:22:57 smithi115 conmon[90393]: 2022-05-14T03:22:57.636+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:22:58.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:22:57 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:22:58.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T03:22:58.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:22:58.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T03:22:58.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:22:58.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:22:58.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:22:58.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T03:22:58.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:22:58.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T03:22:58.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:22:58.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T03:22:58.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:22:58.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T03:22:58.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:22:58.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T03:22:58.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:22:58.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:22:58.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:22:58.297 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:22:57 smithi047 ceph-mon[77147]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T03:22:58.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T03:22:58.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:22:58.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T03:22:58.298 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:22:58.298 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:22:58.298 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:22:58.298 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T03:22:58.299 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:22:58.299 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T03:22:58.299 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:22:58.300 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T03:22:58.300 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:22:58.300 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T03:22:58.300 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:22:58.300 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T03:22:58.301 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:22:58.301 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:22:58.301 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:22:58.302 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T03:22:58.302 INFO:journalctl@ceph.mon.b.smithi047.stdout: During handling of the above exception, another exception occurred: 2022-05-14T03:22:58.302 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T03:22:58.302 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T03:22:58.303 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T03:22:58.303 INFO:journalctl@ceph.mon.b.smithi047.stdout: return f(*arg) 2022-05-14T03:22:58.303 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T03:22:58.303 INFO:journalctl@ceph.mon.b.smithi047.stdout: self._write_client_files(client_files, host) 2022-05-14T03:22:58.304 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T03:22:58.304 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T03:22:58.304 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T03:22:58.304 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T03:22:58.304 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T03:22:58.305 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.event_loop.get_result(coro) 2022-05-14T03:22:58.305 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T03:22:58.305 INFO:journalctl@ceph.mon.b.smithi047.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T03:22:58.305 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T03:22:58.306 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.__get_result() 2022-05-14T03:22:58.306 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T03:22:58.306 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise self._exception 2022-05-14T03:22:58.306 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T03:22:58.306 INFO:journalctl@ceph.mon.b.smithi047.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T03:22:58.307 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T03:22:58.307 INFO:journalctl@ceph.mon.b.smithi047.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T03:22:58.307 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T03:22:58.307 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise 2022-05-14T03:22:58.308 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T03:22:58.308 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.gen.throw(type, value, traceback) 2022-05-14T03:22:58.308 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T03:22:58.308 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OrchestratorError(msg) 2022-05-14T03:22:58.308 INFO:journalctl@ceph.mon.b.smithi047.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:22:58.309 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:22:57 smithi047 ceph-mon[77147]: pgmap v7407: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:22:58.309 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:22:57 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:22:58.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:22:57 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:22:58.359 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T03:22:58.359 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:22:58.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T03:22:58.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:22:58.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:22:58.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:22:58.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T03:22:58.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:22:58.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T03:22:58.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:22:58.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T03:22:58.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:22:58.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T03:22:58.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:22:58.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T03:22:58.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:22:58.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:22:58.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:22:58.364 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:22:57 smithi115 ceph-mon[78593]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T03:22:58.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T03:22:58.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:22:58.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T03:22:58.365 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:22:58.365 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:22:58.366 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:22:58.366 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T03:22:58.366 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:22:58.366 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T03:22:58.366 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:22:58.367 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T03:22:58.367 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:22:58.367 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T03:22:58.368 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:22:58.368 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T03:22:58.368 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:22:58.368 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:22:58.369 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:22:58.369 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T03:22:58.369 INFO:journalctl@ceph.mon.c.smithi115.stdout: During handling of the above exception, another exception occurred: 2022-05-14T03:22:58.369 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T03:22:58.370 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T03:22:58.370 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T03:22:58.370 INFO:journalctl@ceph.mon.c.smithi115.stdout: return f(*arg) 2022-05-14T03:22:58.370 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T03:22:58.371 INFO:journalctl@ceph.mon.c.smithi115.stdout: self._write_client_files(client_files, host) 2022-05-14T03:22:58.371 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T03:22:58.371 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T03:22:58.371 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T03:22:58.372 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T03:22:58.372 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T03:22:58.372 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.event_loop.get_result(coro) 2022-05-14T03:22:58.372 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T03:22:58.372 INFO:journalctl@ceph.mon.c.smithi115.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T03:22:58.373 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T03:22:58.373 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.__get_result() 2022-05-14T03:22:58.373 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T03:22:58.373 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise self._exception 2022-05-14T03:22:58.374 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T03:22:58.374 INFO:journalctl@ceph.mon.c.smithi115.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T03:22:58.374 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T03:22:58.374 INFO:journalctl@ceph.mon.c.smithi115.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T03:22:58.375 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T03:22:58.375 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise 2022-05-14T03:22:58.375 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T03:22:58.376 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.gen.throw(type, value, traceback) 2022-05-14T03:22:58.376 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T03:22:58.376 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OrchestratorError(msg) 2022-05-14T03:22:58.376 INFO:journalctl@ceph.mon.c.smithi115.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:22:58.377 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:22:57 smithi115 ceph-mon[78593]: pgmap v7407: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:22:58.377 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:22:57 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:22:59.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:22:58 smithi115 conmon[90393]: 2022-05-14T03:22:58.680+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:22:59.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:22:58 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:22:59.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:22:58 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:22:59.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:22:58 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:22:59.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:22:58 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:22:59.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:22:58 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:22:59.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:22:58 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:23:00.110 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:22:59 smithi115 conmon[90393]: 2022-05-14T03:22:59.691+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:23:00.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:22:59 smithi115 ceph-mon[78593]: pgmap v7408: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:23:00.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:22:59 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:23:00.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:22:59 smithi047 ceph-mon[77147]: pgmap v7408: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:23:00.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:22:59 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:23:01.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:23:00 smithi115 conmon[90393]: 2022-05-14T03:23:00.699+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:23:01.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:23:00 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:23:01.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:23:00 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:23:02.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:23:01 smithi115 conmon[90393]: 2022-05-14T03:23:01.700+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:23:02.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:23:01 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 14115 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:23:02.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:23:01 smithi115 ceph-mon[78593]: pgmap v7409: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:23:02.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:23:01 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:23:02.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:23:01 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 14115 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:23:02.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:23:01 smithi047 ceph-mon[77147]: pgmap v7409: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:23:02.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:23:01 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:23:02.931 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:23:02 smithi115 conmon[90393]: 2022-05-14T03:23:02.655+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:23:03.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:23:02 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:23:03.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:23:02 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:23:03.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:23:02 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:23:03.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:23:02 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:23:03.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:23:02 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:23:03.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:23:02 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:23:04.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:23:03 smithi115 conmon[90393]: 2022-05-14T03:23:03.699+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:23:04.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:23:03 smithi115 ceph-mon[78593]: pgmap v7410: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:23:04.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:23:03 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:23:04.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:23:03 smithi047 ceph-mon[77147]: pgmap v7410: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:23:04.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:23:03 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:23:05.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:23:04 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:23:05.110 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:23:04 smithi115 conmon[90393]: 2022-05-14T03:23:04.694+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:23:05.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:23:04 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:23:06.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:23:05 smithi115 ceph-mon[78593]: pgmap v7411: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:23:06.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:23:05 smithi115 conmon[90393]: 2022-05-14T03:23:05.734+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:23:06.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:23:05 smithi047 ceph-mon[77147]: pgmap v7411: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:23:07.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:23:06 smithi115 conmon[90393]: 2022-05-14T03:23:06.776+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:23:07.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:23:06 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:23:07.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:23:06 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 14120 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:23:07.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:23:06 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:23:07.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:23:06 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 14120 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:23:08.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:23:07 smithi115 conmon[90393]: 2022-05-14T03:23:07.756+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:23:08.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:23:07 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:23:08.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:23:07 smithi115 ceph-mon[78593]: pgmap v7412: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:23:08.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:23:07 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:23:08.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:23:07 smithi047 ceph-mon[77147]: pgmap v7412: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:23:09.113 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:23:08 smithi115 conmon[90393]: 2022-05-14T03:23:08.777+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:23:09.113 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:23:08 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:23:09.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:23:08 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:23:10.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:23:09 smithi115 conmon[90393]: 2022-05-14T03:23:09.777+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:23:10.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:23:09 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:23:10.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:23:09 smithi115 ceph-mon[78593]: pgmap v7413: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 7 op/s 2022-05-14T03:23:10.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:23:09 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:23:10.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:23:09 smithi047 ceph-mon[77147]: pgmap v7413: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 7 op/s 2022-05-14T03:23:11.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:23:10 smithi115 conmon[90393]: 2022-05-14T03:23:10.795+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:23:11.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:23:10 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:23:11.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:23:10 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:23:11.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:23:10 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:23:11.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:23:10 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:23:11.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:23:10 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:23:11.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:23:10 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:23:12.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:23:11 smithi115 conmon[90393]: 2022-05-14T03:23:11.833+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:23:12.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:23:11 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:23:12.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:23:11 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 14125 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:23:12.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:23:11 smithi115 ceph-mon[78593]: pgmap v7414: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 4 op/s 2022-05-14T03:23:12.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:23:11 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:23:12.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:23:11 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 14125 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:23:12.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:23:11 smithi047 ceph-mon[77147]: pgmap v7414: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 4 op/s 2022-05-14T03:23:13.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:23:12 smithi115 conmon[90393]: 2022-05-14T03:23:12.808+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:23:13.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:23:13 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:23:13.830 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:23:13 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:23:14.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:23:13 smithi115 conmon[90393]: 2022-05-14T03:23:13.828+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:23:14.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:23:14 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:23:14.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:23:14 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:23:14.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:23:14 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:23:14.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:23:14 smithi047 ceph-mon[77147]: pgmap v7415: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 4 op/s 2022-05-14T03:23:14.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:23:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T03:23:14.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:23:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T03:23:14.805 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:23:14 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:23:14.806 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:23:14 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:23:14.806 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:23:14 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:23:14.806 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:23:14 smithi115 ceph-mon[78593]: pgmap v7415: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 4 op/s 2022-05-14T03:23:14.806 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:23:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T03:23:14.807 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:23:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T03:23:15.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:23:14 smithi115 conmon[90393]: 2022-05-14T03:23:14.803+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:23:15.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:23:15 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:23:15.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:23:15 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:23:16.110 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:23:15 smithi115 conmon[90393]: 2022-05-14T03:23:15.794+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:23:16.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:23:16 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:23:16.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:23:16 smithi047 ceph-mon[77147]: pgmap v7416: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 7 op/s 2022-05-14T03:23:16.828 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:23:16 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:23:16.828 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:23:16 smithi115 ceph-mon[78593]: pgmap v7416: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 7 op/s 2022-05-14T03:23:17.110 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:23:16 smithi115 conmon[90393]: 2022-05-14T03:23:16.826+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:23:17.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:23:17 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:23:17.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:23:17 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 14130 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:23:17.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:23:17 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:23:17.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:23:17 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 14130 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:23:18.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:23:17 smithi115 conmon[90393]: 2022-05-14T03:23:17.860+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:23:18.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:23:18 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:23:18.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:23:18 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:23:18.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:23:18 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:23:18.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:23:18 smithi047 ceph-mon[77147]: pgmap v7417: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 4 op/s 2022-05-14T03:23:18.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:23:18 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:23:18.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:23:18 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:23:18.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:23:18 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:23:18.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:23:18 smithi115 ceph-mon[78593]: pgmap v7417: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 4 op/s 2022-05-14T03:23:19.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:23:18 smithi115 conmon[90393]: 2022-05-14T03:23:18.856+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:23:19.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:23:19 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:23:19.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:23:19 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:23:20.360 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:23:19 smithi115 conmon[90393]: 2022-05-14T03:23:19.864+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:23:20.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:23:20 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:23:20.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:23:20 smithi047 ceph-mon[77147]: pgmap v7418: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 7 op/s 2022-05-14T03:23:20.830 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:23:20 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:23:20.830 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:23:20 smithi115 ceph-mon[78593]: pgmap v7418: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 7 op/s 2022-05-14T03:23:21.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:23:20 smithi115 conmon[90393]: 2022-05-14T03:23:20.828+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:23:21.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:23:21 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:23:21.844 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:23:21 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:23:22.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:23:21 smithi115 conmon[90393]: 2022-05-14T03:23:21.842+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:23:22.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:23:22 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:23:22.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:23:22 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:23:22.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:23:22 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:23:22.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:23:22 smithi047 ceph-mon[77147]: pgmap v7419: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:23:22.853 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:23:22 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:23:22.853 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:23:22 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:23:22.853 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:23:22 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:23:22.854 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:23:22 smithi115 ceph-mon[78593]: pgmap v7419: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:23:23.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:23:22 smithi115 conmon[90393]: 2022-05-14T03:23:22.850+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:23:23.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:23:23 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:23:23.849 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:23:23 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:23:24.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:23:23 smithi115 conmon[90393]: 2022-05-14T03:23:23.847+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:23:24.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:23:24 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:23:24.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:23:24 smithi047 ceph-mon[77147]: pgmap v7420: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:23:24.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:23:24 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:23:24.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:23:24 smithi115 ceph-mon[78593]: pgmap v7420: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:23:25.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:23:24 smithi115 conmon[90393]: 2022-05-14T03:23:24.888+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:23:25.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:23:25 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:23:25.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:23:25 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:23:26.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:23:25 smithi115 conmon[90393]: 2022-05-14T03:23:25.857+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:23:26.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:23:26 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:23:26.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:23:26 smithi047 ceph-mon[77147]: pgmap v7421: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:23:26.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:23:26 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 14135 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:23:26.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:23:26 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:23:26.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:23:26 smithi115 ceph-mon[78593]: pgmap v7421: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:23:26.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:23:26 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 14135 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:23:27.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:23:26 smithi115 conmon[90393]: 2022-05-14T03:23:26.888+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:23:27.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:23:27 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:23:27.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:23:27 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:23:28.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:23:27 smithi115 conmon[90393]: 2022-05-14T03:23:27.930+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:23:28.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:23:28 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:23:28.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:23:28 smithi047 ceph-mon[77147]: pgmap v7422: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:23:28.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:23:28 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:23:28.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:23:28 smithi115 ceph-mon[78593]: pgmap v7422: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:23:29.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:23:28 smithi115 conmon[90393]: 2022-05-14T03:23:28.956+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:23:29.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:23:29 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:23:29.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:23:29 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:23:30.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:23:29 smithi115 conmon[90393]: 2022-05-14T03:23:29.956+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:23:30.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:23:30 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:23:30.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:23:30 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:23:30.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:23:30 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:23:30.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:23:30 smithi115 ceph-mon[78593]: pgmap v7423: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:23:31.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:23:30 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:23:31.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:23:30 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:23:31.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:23:30 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:23:31.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:23:30 smithi047 ceph-mon[77147]: pgmap v7423: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:23:31.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:23:30 smithi115 conmon[90393]: 2022-05-14T03:23:30.911+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:23:31.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:23:31 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:23:31.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:23:31 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 14145 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:23:32.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:23:31 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:23:32.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:23:31 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 14145 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:23:32.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:23:31 smithi115 conmon[90393]: 2022-05-14T03:23:31.925+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:23:32.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:23:32 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:23:32.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:23:32 smithi115 ceph-mon[78593]: pgmap v7424: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:23:33.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:23:32 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:23:33.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:23:32 smithi047 ceph-mon[77147]: pgmap v7424: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:23:33.360 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:23:32 smithi115 conmon[90393]: 2022-05-14T03:23:32.923+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:23:33.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:23:33 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:23:34.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:23:33 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:23:34.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:23:33 smithi115 conmon[90393]: 2022-05-14T03:23:33.900+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:23:34.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:23:34 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:23:34.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:23:34 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:23:34.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:23:34 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:23:34.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:23:34 smithi115 ceph-mon[78593]: pgmap v7425: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:23:35.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:23:34 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:23:35.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:23:34 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:23:35.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:23:34 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:23:35.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:23:34 smithi047 ceph-mon[77147]: pgmap v7425: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:23:35.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:23:34 smithi115 conmon[90393]: 2022-05-14T03:23:34.923+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:23:35.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:23:35 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:23:36.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:23:35 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:23:36.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:23:35 smithi115 conmon[90393]: 2022-05-14T03:23:35.952+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:23:36.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:23:36 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:23:36.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:23:36 smithi115 ceph-mon[78593]: pgmap v7426: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:23:36.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:23:36 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 14150 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:23:37.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:23:36 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:23:37.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:23:36 smithi047 ceph-mon[77147]: pgmap v7426: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:23:37.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:23:36 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 14150 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:23:37.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:23:36 smithi115 conmon[90393]: 2022-05-14T03:23:36.948+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:23:37.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:23:37 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:23:38.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:23:37 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:23:38.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:23:37 smithi115 conmon[90393]: 2022-05-14T03:23:37.981+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:23:38.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:23:38 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:23:38.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:23:38 smithi115 ceph-mon[78593]: pgmap v7427: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:23:38.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:23:38 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:23:38.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:23:38 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:23:39.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:23:38 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:23:39.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:23:38 smithi047 ceph-mon[77147]: pgmap v7427: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:23:39.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:23:38 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:23:39.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:23:38 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:23:39.360 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:23:38 smithi115 conmon[90393]: 2022-05-14T03:23:38.962+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:23:39.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:23:39 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:23:40.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:23:39 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:23:40.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:23:39 smithi115 conmon[90393]: 2022-05-14T03:23:39.937+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:23:40.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:23:40 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:23:40.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:23:40 smithi115 ceph-mon[78593]: pgmap v7428: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:23:41.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:23:40 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:23:41.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:23:40 smithi047 ceph-mon[77147]: pgmap v7428: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:23:41.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:23:40 smithi115 conmon[90393]: 2022-05-14T03:23:40.959+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:23:41.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:23:41 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:23:41.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:23:41 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 14155 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:23:42.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:23:41 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:23:42.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:23:41 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 14155 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:23:42.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:23:41 smithi115 conmon[90393]: 2022-05-14T03:23:41.974+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:23:42.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:23:42 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:23:42.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:23:42 smithi115 ceph-mon[78593]: pgmap v7429: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:23:42.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:23:42 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:23:42.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:23:42 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:23:43.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:23:42 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:23:43.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:23:42 smithi047 ceph-mon[77147]: pgmap v7429: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:23:43.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:23:42 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:23:43.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:23:42 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:23:43.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:23:42 smithi115 conmon[90393]: 2022-05-14T03:23:42.940+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:23:43.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:23:43 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:23:44.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:23:43 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:23:44.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:23:43 smithi115 conmon[90393]: 2022-05-14T03:23:43.972+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:23:44.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:23:44 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:23:44.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:23:44 smithi115 ceph-mon[78593]: pgmap v7430: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:23:45.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:23:44 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:23:45.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:23:44 smithi047 ceph-mon[77147]: pgmap v7430: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:23:45.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:23:45 smithi115 conmon[90393]: 2022-05-14T03:23:45.018+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:23:45.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:23:45 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:23:46.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:23:45 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:23:46.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:23:45 smithi115 conmon[90393]: 2022-05-14T03:23:45.989+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:23:46.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:23:46 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:23:46.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:23:46 smithi115 ceph-mon[78593]: pgmap v7431: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:23:46.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:23:46 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 14160 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:23:47.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:23:46 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:23:47.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:23:46 smithi047 ceph-mon[77147]: pgmap v7431: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:23:47.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:23:46 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 14160 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:23:47.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:23:46 smithi115 conmon[90393]: 2022-05-14T03:23:46.961+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:23:47.926 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:23:47 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:23:48.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:23:47 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:23:48.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:23:47 smithi115 conmon[90393]: 2022-05-14T03:23:47.924+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:23:48.931 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:23:48 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:23:48.932 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:23:48 smithi115 ceph-mon[78593]: pgmap v7432: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:23:49.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:23:48 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:23:49.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:23:48 smithi047 ceph-mon[77147]: pgmap v7432: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:23:49.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:23:48 smithi115 conmon[90393]: 2022-05-14T03:23:48.928+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:23:49.923 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:23:49 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:23:50.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:23:49 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:23:50.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:23:49 smithi115 conmon[90393]: 2022-05-14T03:23:49.921+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:23:50.951 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:23:50 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:23:50.952 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:23:50 smithi115 ceph-mon[78593]: pgmap v7433: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:23:50.952 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:23:50 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:23:50.952 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:23:50 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:23:51.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:23:50 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:23:51.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:23:50 smithi047 ceph-mon[77147]: pgmap v7433: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:23:51.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:23:50 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:23:51.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:23:50 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:23:51.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:23:50 smithi115 conmon[90393]: 2022-05-14T03:23:50.948+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:23:51.988 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:23:51 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:23:51.989 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:23:51 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 14165 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:23:52.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:23:51 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:23:52.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:23:51 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 14165 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:23:52.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:23:51 smithi115 conmon[90393]: 2022-05-14T03:23:51.986+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:23:52.963 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:23:52 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:23:52.963 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:23:52 smithi115 ceph-mon[78593]: pgmap v7434: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:23:53.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:23:52 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:23:53.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:23:52 smithi047 ceph-mon[77147]: pgmap v7434: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:23:53.362 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:23:52 smithi115 conmon[90393]: 2022-05-14T03:23:52.960+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:23:53.983 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:23:53 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:23:54.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:23:53 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:23:54.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:23:53 smithi115 conmon[90393]: 2022-05-14T03:23:53.981+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:23:54.976 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:23:54 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:23:54.976 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:23:54 smithi115 ceph-mon[78593]: pgmap v7435: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:23:54.976 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:23:54 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:23:54.977 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:23:54 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:23:55.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:23:54 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:23:55.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:23:54 smithi047 ceph-mon[77147]: pgmap v7435: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:23:55.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:23:54 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:23:55.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:23:54 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:23:55.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:23:54 smithi115 conmon[90393]: 2022-05-14T03:23:54.973+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:23:55.950 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:23:55 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:23:56.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:23:55 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:23:56.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:23:55 smithi115 conmon[90393]: 2022-05-14T03:23:55.948+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:23:56.968 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:23:56 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:23:56.968 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:23:56 smithi115 ceph-mon[78593]: pgmap v7436: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:23:56.969 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:23:56 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 14170 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:23:57.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:23:56 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:23:57.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:23:56 smithi047 ceph-mon[77147]: pgmap v7436: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:23:57.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:23:56 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 14170 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:23:57.249 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:23:56 smithi115 conmon[90393]: 2022-05-14T03:23:56.965+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:23:58.002 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:23:57 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:23:58.002 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:23:57 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T03:23:58.003 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:23:57 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T03:23:58.003 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:23:57 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T03:23:58.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:23:57 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:23:58.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:23:57 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T03:23:58.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:23:57 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T03:23:58.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:23:57 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T03:23:58.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:23:58 smithi115 conmon[90393]: 2022-05-14T03:23:57.999+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:23:58.994 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:23:58 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:23:58.994 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:23:58 smithi115 ceph-mon[78593]: pgmap v7437: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:23:58.994 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:23:58 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:23:58.995 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:23:58 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:23:59.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:23:58 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:23:59.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:23:58 smithi047 ceph-mon[77147]: pgmap v7437: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:23:59.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:23:58 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:23:59.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:23:58 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:23:59.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:23:58 smithi115 conmon[90393]: 2022-05-14T03:23:58.991+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:24:00.007 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:23:59 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:24:00.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:23:59 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:24:00.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:24:00 smithi115 conmon[90393]: 2022-05-14T03:24:00.005+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:24:00.651 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:24:00 smithi047 conmon[78521]: 2022-05-14T03:24:00.304+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:24:00.651 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:24:00 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T03:24:00.651 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:24:00 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:24:00.652 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:24:00 smithi047 conmon[78521]: yield 2022-05-14T03:24:00.652 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:24:00 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:24:00.652 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:24:00 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:24:00.652 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:24:00 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:24:00.653 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:24:00 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T03:24:00.653 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:24:00 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:24:00.653 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:24:00 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T03:24:00.653 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:24:00 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:24:00.653 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:24:00 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T03:24:00.654 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:24:00 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:24:00.654 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:24:00 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T03:24:00.654 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:24:00 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:24:00.654 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:24:00 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T03:24:00.655 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:24:00 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:24:00.655 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:24:00 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:24:00.655 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:24:00 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:24:01.032 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:24:00 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:24:01.033 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:24:00 smithi115 ceph-mon[78593]: pgmap v7438: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:24:01.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:24:00 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:24:01.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:24:00 smithi047 ceph-mon[77147]: pgmap v7438: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:24:01.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:24:01 smithi115 conmon[90393]: 2022-05-14T03:24:01.030+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:24:02.024 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:24:01 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:24:02.024 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:24:01 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:24:02.024 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T03:24:02.025 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:24:02.025 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T03:24:02.025 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:24:02.025 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:24:02.025 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:24:02.026 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T03:24:02.026 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:24:02.027 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T03:24:02.027 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:24:02.027 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T03:24:02.027 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:24:02.028 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T03:24:02.028 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:24:02.028 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T03:24:02.029 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:24:02.029 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:24:02.030 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:24:02.030 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:24:01 smithi115 ceph-mon[78593]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T03:24:02.031 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:24:01 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 14175 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:24:02.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:24:01 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:24:02.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:24:01 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:24:02.042 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T03:24:02.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:24:02.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T03:24:02.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:24:02.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:24:02.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:24:02.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T03:24:02.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:24:02.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T03:24:02.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:24:02.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T03:24:02.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:24:02.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T03:24:02.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:24:02.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T03:24:02.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:24:02.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:24:02.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:24:02.047 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:24:01 smithi047 ceph-mon[77147]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T03:24:02.047 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:24:01 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 14175 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:24:02.360 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:24:02 smithi115 conmon[90393]: 2022-05-14T03:24:02.021+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:24:03.024 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:24:02 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:24:03.024 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:24:02 smithi115 ceph-mon[78593]: pgmap v7439: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:24:03.025 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:24:02 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:24:03.025 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:24:02 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:24:03.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:24:02 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:24:03.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:24:02 smithi047 ceph-mon[77147]: pgmap v7439: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:24:03.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:24:02 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:24:03.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:24:02 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:24:03.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:24:03 smithi115 conmon[90393]: 2022-05-14T03:24:03.021+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:24:03.662 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:24:03 smithi047 conmon[78521]: 2022-05-14T03:24:03.376+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:24:03.663 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:24:03 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T03:24:03.663 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:24:03 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:24:03.663 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:24:03 smithi047 conmon[78521]: yield 2022-05-14T03:24:03.663 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:24:03 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:24:03.663 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:24:03 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:24:03.664 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:24:03 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:24:03.664 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:24:03 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T03:24:03.664 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:24:03 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:24:03.664 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:24:03 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T03:24:03.665 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:24:03 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:24:03.665 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:24:03 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T03:24:03.665 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:24:03 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:24:03.665 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:24:03 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T03:24:03.666 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:24:03 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:24:03.666 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:24:03 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T03:24:03.666 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:24:03 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:24:03.667 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:24:03 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:24:03.667 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:24:03 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:24:03.667 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:24:03 smithi047 conmon[78521]: 2022-05-14T03:24:03.377+0000 7f5341f7c700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T03:24:03.668 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:24:03 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T03:24:03.668 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:24:03 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:24:03.668 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:24:03 smithi047 conmon[78521]: yield 2022-05-14T03:24:03.668 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:24:03 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:24:03.669 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:24:03 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:24:03.669 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:24:03 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:24:03.669 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:24:03 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T03:24:03.669 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:24:03 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:24:03.670 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:24:03 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T03:24:03.670 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:24:03 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:24:03.670 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:24:03 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T03:24:03.670 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:24:03 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:24:03.671 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:24:03 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T03:24:03.671 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:24:03 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:24:03.671 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:24:03 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T03:24:03.671 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:24:03 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:24:03.672 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:24:03 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:24:03.672 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:24:03 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:24:03.672 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:24:03 smithi047 conmon[78521]: 2022-05-14T03:24:03.672 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:24:03 smithi047 conmon[78521]: During handling of the above exception, another exception occurred: 2022-05-14T03:24:03.673 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:24:03 smithi047 conmon[78521]: 2022-05-14T03:24:03.673 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:24:03 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T03:24:03.673 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:24:03 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T03:24:03.673 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:24:03 smithi047 conmon[78521]: return f(*arg) 2022-05-14T03:24:03.674 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:24:03 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T03:24:03.674 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:24:03 smithi047 conmon[78521]: self._write_client_files(client_files, host) 2022-05-14T03:24:03.674 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:24:03 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T03:24:03.674 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:24:03 smithi047 conmon[78521]: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T03:24:03.675 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:24:03 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T03:24:03.675 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:24:03 smithi047 conmon[78521]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T03:24:03.675 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:24:03 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T03:24:03.675 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:24:03 smithi047 conmon[78521]: return self.event_loop.get_result(coro) 2022-05-14T03:24:03.675 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:24:03 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T03:24:03.676 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:24:03 smithi047 conmon[78521]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T03:24:03.676 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:24:03 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T03:24:03.676 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:24:03 smithi047 conmon[78521]: return self.__get_result() 2022-05-14T03:24:03.676 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:24:03 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T03:24:03.677 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:24:03 smithi047 conmon[78521]: raise self._exception 2022-05-14T03:24:03.677 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:24:03 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T03:24:03.677 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:24:03 smithi047 conmon[78521]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T03:24:03.677 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:24:03 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T03:24:03.678 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:24:03 smithi047 conmon[78521]: conn = await self._remote_connection(host, addr) 2022-05-14T03:24:03.678 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:24:03 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T03:24:03.678 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:24:03 smithi047 conmon[78521]: raise 2022-05-14T03:24:03.678 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:24:03 smithi047 conmon[78521]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T03:24:03.678 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:24:03 smithi047 conmon[78521]: self.gen.throw(type, value, traceback) 2022-05-14T03:24:03.679 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:24:03 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T03:24:03.679 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:24:03 smithi047 conmon[78521]: raise OrchestratorError(msg) 2022-05-14T03:24:03.679 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:24:03 smithi047 conmon[78521]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:24:03.978 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:24:03 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:24:04.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:24:03 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:24:04.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:24:03 smithi115 conmon[90393]: 2022-05-14T03:24:03.976+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:24:04.929 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:24:04 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:24:04.930 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:24:04 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:24:04.930 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T03:24:04.931 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:24:04.931 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T03:24:04.931 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:24:04.931 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:24:04.932 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:24:04.932 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T03:24:04.932 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:24:04.932 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T03:24:04.933 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:24:04.933 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T03:24:04.933 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:24:04.933 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T03:24:04.934 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:24:04.934 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T03:24:04.934 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:24:04.934 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:24:04.934 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:24:04.935 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:24:04 smithi115 ceph-mon[78593]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T03:24:04.935 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T03:24:04.935 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:24:04.935 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T03:24:04.936 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:24:04.936 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:24:04.936 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:24:04.936 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T03:24:04.937 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:24:04.937 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T03:24:04.937 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:24:04.937 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T03:24:04.938 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:24:04.938 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T03:24:04.938 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:24:04.938 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T03:24:04.938 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:24:04.939 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:24:04.939 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:24:04.939 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T03:24:04.939 INFO:journalctl@ceph.mon.c.smithi115.stdout: During handling of the above exception, another exception occurred: 2022-05-14T03:24:04.940 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T03:24:04.940 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T03:24:04.940 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T03:24:04.940 INFO:journalctl@ceph.mon.c.smithi115.stdout: return f(*arg) 2022-05-14T03:24:04.941 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T03:24:04.941 INFO:journalctl@ceph.mon.c.smithi115.stdout: self._write_client_files(client_files, host) 2022-05-14T03:24:04.941 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T03:24:04.942 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T03:24:04.942 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T03:24:04.943 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T03:24:04.943 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T03:24:04.944 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.event_loop.get_result(coro) 2022-05-14T03:24:04.944 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T03:24:04.944 INFO:journalctl@ceph.mon.c.smithi115.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T03:24:04.945 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T03:24:04.945 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.__get_result() 2022-05-14T03:24:04.946 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T03:24:04.946 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise self._exception 2022-05-14T03:24:04.947 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T03:24:04.947 INFO:journalctl@ceph.mon.c.smithi115.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T03:24:04.947 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T03:24:04.948 INFO:journalctl@ceph.mon.c.smithi115.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T03:24:04.948 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T03:24:04.949 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise 2022-05-14T03:24:04.949 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T03:24:04.949 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.gen.throw(type, value, traceback) 2022-05-14T03:24:04.950 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T03:24:04.950 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OrchestratorError(msg) 2022-05-14T03:24:04.951 INFO:journalctl@ceph.mon.c.smithi115.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:24:04.951 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:24:04 smithi115 ceph-mon[78593]: pgmap v7440: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:24:05.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:24:04 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:24:05.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:24:04 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:24:05.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T03:24:05.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:24:05.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T03:24:05.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:24:05.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:24:05.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:24:05.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T03:24:05.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:24:05.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T03:24:05.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:24:05.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T03:24:05.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:24:05.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T03:24:05.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:24:05.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T03:24:05.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:24:05.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:24:05.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:24:05.047 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:24:04 smithi047 ceph-mon[77147]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T03:24:05.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T03:24:05.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:24:05.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T03:24:05.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:24:05.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:24:05.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:24:05.049 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T03:24:05.049 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:24:05.049 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T03:24:05.049 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:24:05.050 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T03:24:05.050 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:24:05.050 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T03:24:05.050 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:24:05.050 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T03:24:05.051 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:24:05.051 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:24:05.051 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:24:05.051 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T03:24:05.051 INFO:journalctl@ceph.mon.b.smithi047.stdout: During handling of the above exception, another exception occurred: 2022-05-14T03:24:05.052 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T03:24:05.052 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T03:24:05.052 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T03:24:05.052 INFO:journalctl@ceph.mon.b.smithi047.stdout: return f(*arg) 2022-05-14T03:24:05.052 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T03:24:05.053 INFO:journalctl@ceph.mon.b.smithi047.stdout: self._write_client_files(client_files, host) 2022-05-14T03:24:05.053 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T03:24:05.053 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T03:24:05.053 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T03:24:05.054 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T03:24:05.054 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T03:24:05.054 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.event_loop.get_result(coro) 2022-05-14T03:24:05.054 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T03:24:05.054 INFO:journalctl@ceph.mon.b.smithi047.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T03:24:05.055 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T03:24:05.055 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.__get_result() 2022-05-14T03:24:05.055 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T03:24:05.055 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise self._exception 2022-05-14T03:24:05.055 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T03:24:05.056 INFO:journalctl@ceph.mon.b.smithi047.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T03:24:05.056 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T03:24:05.056 INFO:journalctl@ceph.mon.b.smithi047.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T03:24:05.056 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T03:24:05.057 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise 2022-05-14T03:24:05.057 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T03:24:05.057 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.gen.throw(type, value, traceback) 2022-05-14T03:24:05.057 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T03:24:05.057 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OrchestratorError(msg) 2022-05-14T03:24:05.058 INFO:journalctl@ceph.mon.b.smithi047.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:24:05.058 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:24:04 smithi047 ceph-mon[77147]: pgmap v7440: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:24:05.360 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:24:04 smithi115 conmon[90393]: 2022-05-14T03:24:04.927+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:24:05.944 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:24:05 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:24:06.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:24:05 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:24:06.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:24:05 smithi115 conmon[90393]: 2022-05-14T03:24:05.942+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:24:07.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:24:06 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:24:07.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:24:06 smithi047 ceph-mon[77147]: pgmap v7441: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:24:07.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:24:06 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 14180 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:24:07.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:24:06 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:24:07.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:24:06 smithi115 ceph-mon[78593]: pgmap v7441: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:24:07.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:24:06 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 14180 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:24:07.111 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:24:06 smithi115 conmon[90393]: 2022-05-14T03:24:06.904+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:24:08.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:24:07 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:24:08.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:24:07 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:24:08.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:24:07 smithi115 conmon[90393]: 2022-05-14T03:24:07.857+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:24:09.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:24:08 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:24:09.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:24:08 smithi047 ceph-mon[77147]: pgmap v7442: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:24:09.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:24:08 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:24:09.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:24:08 smithi115 ceph-mon[78593]: pgmap v7442: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:24:09.110 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:24:08 smithi115 conmon[90393]: 2022-05-14T03:24:08.905+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:24:10.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:24:09 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:24:10.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:24:09 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:24:10.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:24:09 smithi115 conmon[90393]: 2022-05-14T03:24:09.888+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:24:11.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:24:10 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:24:11.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:24:10 smithi047 ceph-mon[77147]: pgmap v7443: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:24:11.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:24:10 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:24:11.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:24:10 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:24:11.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:24:10 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:24:11.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:24:10 smithi115 ceph-mon[78593]: pgmap v7443: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:24:11.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:24:10 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:24:11.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:24:10 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:24:11.110 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:24:10 smithi115 conmon[90393]: 2022-05-14T03:24:10.884+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:24:12.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:24:11 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:24:12.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:24:11 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 14185 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:24:12.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:24:11 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:24:12.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:24:11 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 14185 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:24:12.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:24:11 smithi115 conmon[90393]: 2022-05-14T03:24:11.929+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:24:13.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:24:12 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:24:13.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:24:12 smithi047 ceph-mon[77147]: pgmap v7444: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:24:13.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:24:12 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:24:13.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:24:12 smithi115 ceph-mon[78593]: pgmap v7444: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:24:13.110 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:24:12 smithi115 conmon[90393]: 2022-05-14T03:24:12.896+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:24:14.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:24:13 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:24:14.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:24:13 smithi115 conmon[90393]: 2022-05-14T03:24:13.922+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:24:14.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:24:13 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:24:15.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:24:14 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:24:15.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:24:14 smithi047 ceph-mon[77147]: pgmap v7445: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:24:15.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:24:14 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:24:15.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:24:14 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:24:15.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:24:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T03:24:15.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:24:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T03:24:15.113 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:24:14 smithi115 conmon[90393]: 2022-05-14T03:24:14.905+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:24:15.114 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:24:14 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:24:15.114 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:24:14 smithi115 ceph-mon[78593]: pgmap v7445: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:24:15.114 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:24:14 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:24:15.115 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:24:14 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:24:15.115 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:24:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T03:24:15.115 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:24:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T03:24:15.954 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:24:15 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:24:16.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:24:15 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:24:16.362 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:24:15 smithi115 conmon[90393]: 2022-05-14T03:24:15.951+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:24:17.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:24:16 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:24:17.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:24:16 smithi047 ceph-mon[77147]: pgmap v7446: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:24:17.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:24:16 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 14190 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:24:17.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:24:16 smithi115 conmon[90393]: 2022-05-14T03:24:16.909+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:24:17.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:24:16 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:24:17.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:24:16 smithi115 ceph-mon[78593]: pgmap v7446: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:24:17.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:24:16 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 14190 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:24:18.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:24:17 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:24:18.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:24:17 smithi115 conmon[90393]: 2022-05-14T03:24:17.894+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:24:18.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:24:17 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:24:19.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:24:18 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:24:19.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:24:18 smithi047 ceph-mon[77147]: pgmap v7447: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:24:19.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:24:18 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:24:19.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:24:18 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:24:19.110 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:24:18 smithi115 conmon[90393]: 2022-05-14T03:24:18.938+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:24:19.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:24:18 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:24:19.112 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:24:18 smithi115 ceph-mon[78593]: pgmap v7447: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:24:19.112 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:24:18 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:24:19.112 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:24:18 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:24:20.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:24:19 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:24:20.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:24:19 smithi115 conmon[90393]: 2022-05-14T03:24:19.915+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:24:20.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:24:19 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:24:21.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:24:20 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:24:21.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:24:20 smithi047 ceph-mon[77147]: pgmap v7448: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:24:21.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:24:20 smithi115 conmon[90393]: 2022-05-14T03:24:20.883+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:24:21.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:24:20 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:24:21.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:24:20 smithi115 ceph-mon[78593]: pgmap v7448: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:24:22.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:24:21 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:24:22.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:24:21 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 14195 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:24:22.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:24:21 smithi115 conmon[90393]: 2022-05-14T03:24:21.902+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:24:22.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:24:21 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:24:22.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:24:21 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 14195 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:24:23.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:24:22 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:24:23.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:24:22 smithi047 ceph-mon[77147]: pgmap v7449: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:24:23.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:24:22 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:24:23.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:24:22 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:24:23.112 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:24:22 smithi115 conmon[90393]: 2022-05-14T03:24:22.903+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:24:23.113 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:24:22 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:24:23.113 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:24:22 smithi115 ceph-mon[78593]: pgmap v7449: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:24:23.113 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:24:22 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:24:23.114 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:24:22 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:24:24.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:24:23 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:24:24.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:24:23 smithi115 conmon[90393]: 2022-05-14T03:24:23.920+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:24:24.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:24:23 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:24:25.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:24:24 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:24:25.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:24:24 smithi047 ceph-mon[77147]: pgmap v7450: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:24:25.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:24:24 smithi115 conmon[90393]: 2022-05-14T03:24:24.936+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:24:25.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:24:24 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:24:25.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:24:24 smithi115 ceph-mon[78593]: pgmap v7450: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:24:26.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:24:25 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:24:26.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:24:25 smithi115 conmon[90393]: 2022-05-14T03:24:25.927+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:24:26.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:24:25 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:24:27.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:24:26 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:24:27.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:24:26 smithi047 ceph-mon[77147]: pgmap v7451: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:24:27.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:24:26 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 14200 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:24:27.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:24:26 smithi115 conmon[90393]: 2022-05-14T03:24:26.971+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:24:27.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:24:26 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:24:27.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:24:26 smithi115 ceph-mon[78593]: pgmap v7451: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:24:27.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:24:26 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 14200 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:24:28.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:24:27 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:24:28.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:24:27 smithi115 conmon[90393]: 2022-05-14T03:24:27.952+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:24:28.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:24:27 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:24:29.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:24:28 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:24:29.045 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:24:28 smithi047 ceph-mon[77147]: pgmap v7452: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:24:29.045 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:24:28 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:24:29.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:24:28 smithi115 conmon[90393]: 2022-05-14T03:24:28.933+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:24:29.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:24:28 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:24:29.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:24:28 smithi115 ceph-mon[78593]: pgmap v7452: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:24:29.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:24:28 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:24:30.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:24:29 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:24:30.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:24:29 smithi115 conmon[90393]: 2022-05-14T03:24:29.904+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:24:30.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:24:29 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:24:31.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:24:30 smithi047 ceph-mon[77147]: pgmap v7453: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:24:31.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:24:30 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:24:31.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:24:30 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:24:31.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:24:30 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:24:31.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:24:30 smithi115 conmon[90393]: 2022-05-14T03:24:30.914+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:24:31.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:24:30 smithi115 ceph-mon[78593]: pgmap v7453: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:24:31.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:24:30 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:24:31.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:24:30 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:24:31.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:24:30 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:24:32.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:24:31 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:24:32.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:24:31 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 14205 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:24:32.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:24:31 smithi115 conmon[90393]: 2022-05-14T03:24:31.936+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:24:32.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:24:31 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:24:32.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:24:31 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 14205 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:24:33.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:24:32 smithi047 ceph-mon[77147]: pgmap v7454: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:24:33.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:24:32 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:24:33.110 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:24:32 smithi115 conmon[90393]: 2022-05-14T03:24:32.910+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:24:33.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:24:32 smithi115 ceph-mon[78593]: pgmap v7454: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:24:33.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:24:32 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:24:34.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:24:33 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:24:34.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:24:33 smithi115 conmon[90393]: 2022-05-14T03:24:33.886+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:24:34.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:24:33 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:24:35.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:24:34 smithi047 ceph-mon[77147]: pgmap v7455: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:24:35.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:24:34 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:24:35.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:24:34 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:24:35.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:24:34 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:24:35.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:24:34 smithi115 conmon[90393]: 2022-05-14T03:24:34.845+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:24:35.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:24:34 smithi115 ceph-mon[78593]: pgmap v7455: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:24:35.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:24:34 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:24:35.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:24:34 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:24:35.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:24:34 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:24:36.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:24:35 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:24:36.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:24:35 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:24:36.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:24:35 smithi115 conmon[90393]: 2022-05-14T03:24:35.871+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:24:37.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:24:36 smithi047 ceph-mon[77147]: pgmap v7456: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:24:37.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:24:36 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:24:37.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:24:36 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 14210 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:24:37.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:24:36 smithi115 conmon[90393]: 2022-05-14T03:24:36.905+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:24:37.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:24:36 smithi115 ceph-mon[78593]: pgmap v7456: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:24:37.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:24:36 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:24:37.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:24:36 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 14210 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:24:38.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:24:37 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:24:38.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:24:37 smithi115 conmon[90393]: 2022-05-14T03:24:37.915+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:24:38.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:24:37 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:24:39.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:24:38 smithi047 ceph-mon[77147]: pgmap v7457: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:24:39.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:24:38 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:24:39.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:24:38 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:24:39.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:24:38 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:24:39.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:24:38 smithi115 conmon[90393]: 2022-05-14T03:24:38.923+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:24:39.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:24:38 smithi115 ceph-mon[78593]: pgmap v7457: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:24:39.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:24:38 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:24:39.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:24:38 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:24:39.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:24:38 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:24:40.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:24:39 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:24:40.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:24:39 smithi115 conmon[90393]: 2022-05-14T03:24:39.939+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:24:40.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:24:39 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:24:41.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:24:40 smithi047 ceph-mon[77147]: pgmap v7458: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:24:41.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:24:40 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:24:41.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:24:40 smithi115 conmon[90393]: 2022-05-14T03:24:40.900+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:24:41.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:24:40 smithi115 ceph-mon[78593]: pgmap v7458: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:24:41.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:24:40 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:24:42.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:24:41 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:24:42.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:24:41 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 14215 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:24:42.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:24:41 smithi115 conmon[90393]: 2022-05-14T03:24:41.897+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:24:42.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:24:41 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:24:42.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:24:41 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 14215 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:24:43.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:24:42 smithi047 ceph-mon[77147]: pgmap v7459: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:24:43.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:24:42 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:24:43.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:24:42 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:24:43.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:24:42 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:24:43.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:24:42 smithi115 conmon[90393]: 2022-05-14T03:24:42.925+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:24:43.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:24:42 smithi115 ceph-mon[78593]: pgmap v7459: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:24:43.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:24:42 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:24:43.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:24:42 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:24:43.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:24:42 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:24:44.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:24:43 smithi115 conmon[90393]: 2022-05-14T03:24:43.884+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:24:44.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:24:43 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:24:44.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:24:43 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:24:45.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:24:44 smithi115 conmon[90393]: 2022-05-14T03:24:44.844+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:24:45.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:24:44 smithi115 ceph-mon[78593]: pgmap v7460: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:24:45.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:24:44 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:24:45.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:24:44 smithi047 ceph-mon[77147]: pgmap v7460: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:24:45.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:24:44 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:24:46.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:24:45 smithi115 conmon[90393]: 2022-05-14T03:24:45.826+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:24:46.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:24:45 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:24:46.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:24:45 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:24:47.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:24:46 smithi115 conmon[90393]: 2022-05-14T03:24:46.870+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:24:47.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:24:46 smithi115 ceph-mon[78593]: pgmap v7461: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:24:47.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:24:46 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:24:47.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:24:46 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 14220 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:24:47.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:24:46 smithi047 ceph-mon[77147]: pgmap v7461: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:24:47.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:24:46 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:24:47.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:24:46 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 14220 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:24:48.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:24:47 smithi115 conmon[90393]: 2022-05-14T03:24:47.915+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:24:48.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:24:47 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:24:48.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:24:47 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:24:49.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:24:48 smithi115 conmon[90393]: 2022-05-14T03:24:48.959+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:24:49.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:24:48 smithi115 ceph-mon[78593]: pgmap v7462: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:24:49.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:24:48 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:24:49.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:24:48 smithi047 ceph-mon[77147]: pgmap v7462: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:24:49.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:24:48 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:24:50.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:24:49 smithi115 conmon[90393]: 2022-05-14T03:24:49.982+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:24:50.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:24:49 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:24:50.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:24:49 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:24:51.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:24:51 smithi115 conmon[90393]: 2022-05-14T03:24:51.021+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:24:51.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:24:50 smithi115 ceph-mon[78593]: pgmap v7463: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:24:51.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:24:50 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:24:51.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:24:50 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:24:51.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:24:50 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:24:51.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:24:50 smithi047 ceph-mon[77147]: pgmap v7463: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:24:51.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:24:50 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:24:51.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:24:50 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:24:51.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:24:50 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:24:52.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:24:52 smithi115 conmon[90393]: 2022-05-14T03:24:52.042+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:24:52.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:24:51 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:24:52.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:24:51 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 14225 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:24:52.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:24:51 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:24:52.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:24:51 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 14225 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:24:53.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:24:53 smithi115 conmon[90393]: 2022-05-14T03:24:53.032+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:24:53.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:24:52 smithi115 ceph-mon[78593]: pgmap v7464: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:24:53.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:24:52 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:24:53.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:24:52 smithi047 ceph-mon[77147]: pgmap v7464: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:24:53.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:24:52 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:24:54.075 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:24:53 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:24:54.299 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:24:53 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:24:54.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:24:54 smithi115 conmon[90393]: 2022-05-14T03:24:54.073+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:24:55.085 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:24:54 smithi115 ceph-mon[78593]: pgmap v7465: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:24:55.085 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:24:54 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:24:55.085 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:24:54 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:24:55.086 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:24:54 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:24:55.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:24:54 smithi047 ceph-mon[77147]: pgmap v7465: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:24:55.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:24:54 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:24:55.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:24:54 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:24:55.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:24:54 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:24:55.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:24:55 smithi115 conmon[90393]: 2022-05-14T03:24:55.083+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:24:56.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:24:55 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:24:56.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:24:56 smithi115 conmon[90393]: 2022-05-14T03:24:56.039+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:24:56.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:24:55 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:24:57.090 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:24:56 smithi115 ceph-mon[78593]: pgmap v7466: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:24:57.090 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:24:56 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:24:57.090 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:24:56 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 14229 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:24:57.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:24:56 smithi047 ceph-mon[77147]: pgmap v7466: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:24:57.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:24:56 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:24:57.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:24:56 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 14229 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:24:57.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:24:57 smithi115 conmon[90393]: 2022-05-14T03:24:57.088+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:24:58.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:24:57 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:24:58.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:24:57 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:24:58.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:24:58 smithi115 conmon[90393]: 2022-05-14T03:24:58.120+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:24:59.089 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:24:58 smithi115 ceph-mon[78593]: pgmap v7467: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:24:59.089 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:24:58 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:24:59.090 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:24:58 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:24:59.090 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:24:58 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:24:59.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:24:58 smithi047 ceph-mon[77147]: pgmap v7467: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:24:59.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:24:58 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:24:59.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:24:58 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:24:59.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:24:58 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:24:59.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:24:59 smithi115 conmon[90393]: 2022-05-14T03:24:59.087+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:25:00.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:25:00 smithi115 conmon[90393]: 2022-05-14T03:25:00.052+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:25:00.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:24:59 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:25:00.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:24:59 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:25:01.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:25:01 smithi115 conmon[90393]: 2022-05-14T03:25:01.007+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:25:01.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:25:00 smithi115 ceph-mon[78593]: pgmap v7468: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:25:01.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:25:00 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:25:01.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:25:00 smithi047 ceph-mon[77147]: pgmap v7468: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:25:01.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:25:00 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:25:02.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:25:01 smithi115 conmon[90393]: 2022-05-14T03:25:01.994+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:25:02.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:25:01 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:25:02.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:25:01 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 14235 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:25:02.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:25:01 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:25:02.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:25:01 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 14235 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:25:03.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:25:02 smithi115 conmon[90393]: 2022-05-14T03:25:02.949+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:25:03.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:25:02 smithi115 ceph-mon[78593]: pgmap v7469: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:25:03.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:25:02 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:25:03.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:25:02 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:25:03.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:25:02 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:25:03.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:25:02 smithi047 ceph-mon[77147]: pgmap v7469: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:25:03.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:25:02 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:25:03.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:25:02 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:25:03.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:25:02 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:25:04.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:25:03 smithi115 conmon[90393]: 2022-05-14T03:25:03.949+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:25:04.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:25:03 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:25:04.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:25:03 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T03:25:04.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:25:03 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T03:25:04.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:25:03 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T03:25:04.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:25:03 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:25:04.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:25:03 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T03:25:04.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:25:03 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T03:25:04.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:25:03 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T03:25:04.293 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:25:03 smithi047 conmon[78521]: 2022-05-14T03:25:03.856+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:25:04.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:25:03 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T03:25:04.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:25:03 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:25:04.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:25:03 smithi047 conmon[78521]: yield 2022-05-14T03:25:04.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:25:03 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:25:04.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:25:03 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:25:04.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:25:03 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:25:04.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:25:03 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T03:25:04.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:25:03 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:25:04.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:25:03 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T03:25:04.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:25:03 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:25:04.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:25:03 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T03:25:04.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:25:03 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:25:04.297 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:25:03 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T03:25:04.297 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:25:03 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:25:04.297 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:25:03 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T03:25:04.297 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:25:03 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:25:04.297 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:25:03 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:25:04.298 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:25:03 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:25:05.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:25:04 smithi115 conmon[90393]: 2022-05-14T03:25:04.911+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:25:05.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:25:04 smithi115 ceph-mon[78593]: pgmap v7470: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:25:05.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:25:04 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:25:05.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T03:25:05.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:25:05.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T03:25:05.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:25:05.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:25:05.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:25:05.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T03:25:05.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:25:05.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T03:25:05.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:25:05.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T03:25:05.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:25:05.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T03:25:05.113 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:25:05.113 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T03:25:05.113 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:25:05.113 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:25:05.114 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:25:05.114 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:25:04 smithi115 ceph-mon[78593]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T03:25:05.114 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:25:04 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:25:05.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:25:04 smithi047 ceph-mon[77147]: pgmap v7470: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:25:05.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:25:04 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:25:05.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T03:25:05.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:25:05.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T03:25:05.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:25:05.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:25:05.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:25:05.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T03:25:05.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:25:05.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T03:25:05.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:25:05.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T03:25:05.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:25:05.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T03:25:05.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:25:05.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T03:25:05.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:25:05.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:25:05.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:25:05.297 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:25:04 smithi047 ceph-mon[77147]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T03:25:05.297 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:25:04 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:25:06.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:25:05 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:25:06.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:25:05 smithi115 conmon[90393]: 2022-05-14T03:25:05.962+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:25:06.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:25:05 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:25:07.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:25:06 smithi047 ceph-mon[77147]: pgmap v7471: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:25:07.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:25:06 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:25:07.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:25:06 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 14240 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:25:07.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:25:06 smithi047 conmon[78521]: 2022-05-14T03:25:06.927+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:25:07.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:25:06 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T03:25:07.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:25:06 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:25:07.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:25:06 smithi047 conmon[78521]: yield 2022-05-14T03:25:07.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:25:06 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:25:07.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:25:06 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:25:07.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:25:06 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:25:07.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:25:06 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T03:25:07.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:25:06 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:25:07.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:25:06 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T03:25:07.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:25:06 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:25:07.297 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:25:06 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T03:25:07.297 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:25:06 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:25:07.297 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:25:06 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T03:25:07.297 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:25:06 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:25:07.298 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:25:06 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T03:25:07.298 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:25:06 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:25:07.298 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:25:06 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:25:07.299 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:25:06 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:25:07.299 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:25:06 smithi047 conmon[78521]: 2022-05-14T03:25:06.928+0000 7f5344f82700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T03:25:07.299 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:25:06 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T03:25:07.299 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:25:06 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:25:07.300 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:25:06 smithi047 conmon[78521]: yield 2022-05-14T03:25:07.300 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:25:06 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:25:07.300 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:25:06 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:25:07.300 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:25:06 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:25:07.300 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:25:06 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T03:25:07.301 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:25:06 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:25:07.301 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:25:06 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T03:25:07.301 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:25:06 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:25:07.301 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:25:06 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T03:25:07.302 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:25:06 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:25:07.302 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:25:06 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T03:25:07.302 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:25:06 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:25:07.302 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:25:06 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T03:25:07.302 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:25:06 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:25:07.303 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:25:06 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:25:07.303 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:25:06 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:25:07.303 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:25:06 smithi047 conmon[78521]: 2022-05-14T03:25:07.303 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:25:06 smithi047 conmon[78521]: During handling of the above exception, another exception occurred: 2022-05-14T03:25:07.303 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:25:06 smithi047 conmon[78521]: 2022-05-14T03:25:07.304 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:25:06 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T03:25:07.304 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:25:06 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T03:25:07.304 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:25:06 smithi047 conmon[78521]: return f(*arg) 2022-05-14T03:25:07.304 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:25:06 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T03:25:07.305 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:25:06 smithi047 conmon[78521]: self._write_client_files(client_files, host) 2022-05-14T03:25:07.305 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:25:06 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T03:25:07.305 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:25:06 smithi047 conmon[78521]: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T03:25:07.305 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:25:06 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T03:25:07.306 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:25:06 smithi047 conmon[78521]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T03:25:07.306 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:25:06 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T03:25:07.306 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:25:06 smithi047 conmon[78521]: return self.event_loop.get_result(coro) 2022-05-14T03:25:07.306 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:25:06 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T03:25:07.306 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:25:06 smithi047 conmon[78521]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T03:25:07.307 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:25:06 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T03:25:07.307 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:25:06 smithi047 conmon[78521]: return self.__get_result() 2022-05-14T03:25:07.307 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:25:06 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T03:25:07.308 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:25:06 smithi047 conmon[78521]: raise self._exception 2022-05-14T03:25:07.308 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:25:06 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T03:25:07.308 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:25:06 smithi047 conmon[78521]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T03:25:07.309 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:25:06 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T03:25:07.309 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:25:06 smithi047 conmon[78521]: conn = await self._remote_connection(host, addr) 2022-05-14T03:25:07.309 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:25:06 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T03:25:07.309 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:25:06 smithi047 conmon[78521]: raise 2022-05-14T03:25:07.310 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:25:06 smithi047 conmon[78521]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T03:25:07.310 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:25:06 smithi047 conmon[78521]: self.gen.throw(type, value, traceback) 2022-05-14T03:25:07.310 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:25:06 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T03:25:07.310 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:25:06 smithi047 conmon[78521]: raise OrchestratorError(msg) 2022-05-14T03:25:07.311 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:25:06 smithi047 conmon[78521]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:25:07.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:25:06 smithi115 conmon[90393]: 2022-05-14T03:25:06.996+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:25:07.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:25:06 smithi115 ceph-mon[78593]: pgmap v7471: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:25:07.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:25:06 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:25:07.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:25:06 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 14240 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:25:08.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:25:07 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:25:08.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:25:08 smithi115 conmon[90393]: 2022-05-14T03:25:08.001+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:25:08.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:25:07 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:25:09.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:25:08 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:25:09.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T03:25:09.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:25:09.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T03:25:09.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:25:09.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:25:09.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:25:09.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T03:25:09.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:25:09.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T03:25:09.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:25:09.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T03:25:09.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:25:09.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T03:25:09.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:25:09.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T03:25:09.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:25:09.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:25:09.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:25:09.297 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:25:08 smithi047 ceph-mon[77147]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T03:25:09.298 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T03:25:09.298 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:25:09.298 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T03:25:09.298 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:25:09.299 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:25:09.299 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:25:09.299 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T03:25:09.299 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:25:09.300 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T03:25:09.300 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:25:09.300 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T03:25:09.300 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:25:09.300 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T03:25:09.301 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:25:09.301 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T03:25:09.301 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:25:09.301 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:25:09.302 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:25:09.302 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T03:25:09.302 INFO:journalctl@ceph.mon.b.smithi047.stdout: During handling of the above exception, another exception occurred: 2022-05-14T03:25:09.302 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T03:25:09.303 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T03:25:09.303 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T03:25:09.303 INFO:journalctl@ceph.mon.b.smithi047.stdout: return f(*arg) 2022-05-14T03:25:09.303 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T03:25:09.304 INFO:journalctl@ceph.mon.b.smithi047.stdout: self._write_client_files(client_files, host) 2022-05-14T03:25:09.304 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T03:25:09.304 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T03:25:09.304 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T03:25:09.304 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T03:25:09.305 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T03:25:09.305 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.event_loop.get_result(coro) 2022-05-14T03:25:09.305 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T03:25:09.305 INFO:journalctl@ceph.mon.b.smithi047.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T03:25:09.306 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T03:25:09.306 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.__get_result() 2022-05-14T03:25:09.306 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T03:25:09.306 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise self._exception 2022-05-14T03:25:09.307 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T03:25:09.307 INFO:journalctl@ceph.mon.b.smithi047.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T03:25:09.307 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T03:25:09.307 INFO:journalctl@ceph.mon.b.smithi047.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T03:25:09.307 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T03:25:09.308 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise 2022-05-14T03:25:09.308 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T03:25:09.308 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.gen.throw(type, value, traceback) 2022-05-14T03:25:09.308 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T03:25:09.309 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OrchestratorError(msg) 2022-05-14T03:25:09.309 INFO:journalctl@ceph.mon.b.smithi047.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:25:09.309 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:25:08 smithi047 ceph-mon[77147]: pgmap v7472: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:25:09.309 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:25:08 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:25:09.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:25:09 smithi115 conmon[90393]: 2022-05-14T03:25:09.014+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:25:09.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:25:08 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:25:09.359 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T03:25:09.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:25:09.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T03:25:09.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:25:09.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:25:09.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:25:09.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T03:25:09.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:25:09.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T03:25:09.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:25:09.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T03:25:09.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:25:09.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T03:25:09.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:25:09.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T03:25:09.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:25:09.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:25:09.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:25:09.363 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:25:08 smithi115 ceph-mon[78593]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T03:25:09.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T03:25:09.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:25:09.365 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T03:25:09.365 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:25:09.365 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:25:09.365 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:25:09.365 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T03:25:09.366 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:25:09.366 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T03:25:09.366 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:25:09.367 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T03:25:09.367 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:25:09.367 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T03:25:09.367 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:25:09.367 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T03:25:09.368 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:25:09.368 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:25:09.368 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:25:09.368 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T03:25:09.369 INFO:journalctl@ceph.mon.c.smithi115.stdout: During handling of the above exception, another exception occurred: 2022-05-14T03:25:09.369 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T03:25:09.369 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T03:25:09.369 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T03:25:09.369 INFO:journalctl@ceph.mon.c.smithi115.stdout: return f(*arg) 2022-05-14T03:25:09.370 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T03:25:09.370 INFO:journalctl@ceph.mon.c.smithi115.stdout: self._write_client_files(client_files, host) 2022-05-14T03:25:09.370 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T03:25:09.370 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T03:25:09.370 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T03:25:09.371 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T03:25:09.371 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T03:25:09.371 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.event_loop.get_result(coro) 2022-05-14T03:25:09.371 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T03:25:09.371 INFO:journalctl@ceph.mon.c.smithi115.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T03:25:09.372 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T03:25:09.372 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.__get_result() 2022-05-14T03:25:09.372 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T03:25:09.372 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise self._exception 2022-05-14T03:25:09.373 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T03:25:09.373 INFO:journalctl@ceph.mon.c.smithi115.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T03:25:09.373 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T03:25:09.373 INFO:journalctl@ceph.mon.c.smithi115.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T03:25:09.373 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T03:25:09.374 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise 2022-05-14T03:25:09.374 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T03:25:09.374 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.gen.throw(type, value, traceback) 2022-05-14T03:25:09.374 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T03:25:09.374 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OrchestratorError(msg) 2022-05-14T03:25:09.375 INFO:journalctl@ceph.mon.c.smithi115.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:25:09.375 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:25:08 smithi115 ceph-mon[78593]: pgmap v7472: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:25:09.375 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:25:08 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:25:10.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:25:09 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:25:10.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:25:10 smithi115 conmon[90393]: 2022-05-14T03:25:10.020+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:25:10.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:25:09 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:25:11.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:25:10 smithi047 ceph-mon[77147]: pgmap v7473: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:25:11.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:25:10 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:25:11.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:25:10 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:25:11.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:25:10 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:25:11.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:25:11 smithi115 conmon[90393]: 2022-05-14T03:25:11.034+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:25:11.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:25:10 smithi115 ceph-mon[78593]: pgmap v7473: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:25:11.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:25:10 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:25:11.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:25:10 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:25:11.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:25:10 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:25:12.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:25:11 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:25:12.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:25:11 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 14245 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:25:12.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:25:12 smithi115 conmon[90393]: 2022-05-14T03:25:12.075+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:25:12.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:25:11 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:25:12.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:25:11 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 14245 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:25:13.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:25:12 smithi047 ceph-mon[77147]: pgmap v7474: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:25:13.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:25:12 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:25:13.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:25:13 smithi115 conmon[90393]: 2022-05-14T03:25:13.050+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:25:13.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:25:12 smithi115 ceph-mon[78593]: pgmap v7474: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:25:13.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:25:12 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:25:14.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:25:13 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:25:14.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:25:14 smithi115 conmon[90393]: 2022-05-14T03:25:14.019+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:25:14.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:25:13 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:25:15.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:25:14 smithi047 ceph-mon[77147]: pgmap v7475: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:25:15.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:25:14 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:25:15.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:25:14 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:25:15.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:25:14 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:25:15.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:25:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T03:25:15.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:25:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T03:25:15.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:25:15 smithi115 conmon[90393]: 2022-05-14T03:25:15.024+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:25:15.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:25:14 smithi115 ceph-mon[78593]: pgmap v7475: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:25:15.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:25:14 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:25:15.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:25:14 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:25:15.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:25:14 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:25:15.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:25:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T03:25:15.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:25:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T03:25:16.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:25:15 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:25:16.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:25:15 smithi115 conmon[90393]: 2022-05-14T03:25:15.976+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:25:16.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:25:15 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:25:17.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:25:16 smithi047 ceph-mon[77147]: pgmap v7476: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:25:17.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:25:16 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:25:17.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:25:16 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 14250 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:25:17.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:25:17 smithi115 conmon[90393]: 2022-05-14T03:25:17.002+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:25:17.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:25:16 smithi115 ceph-mon[78593]: pgmap v7476: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:25:17.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:25:16 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:25:17.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:25:16 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 14250 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:25:18.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:25:17 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:25:18.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:25:17 smithi115 conmon[90393]: 2022-05-14T03:25:17.983+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:25:18.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:25:17 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:25:19.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:25:18 smithi047 ceph-mon[77147]: pgmap v7477: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:25:19.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:25:18 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:25:19.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:25:18 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:25:19.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:25:18 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:25:19.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:25:18 smithi115 conmon[90393]: 2022-05-14T03:25:18.994+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:25:19.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:25:18 smithi115 ceph-mon[78593]: pgmap v7477: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:25:19.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:25:18 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:25:19.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:25:18 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:25:19.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:25:18 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:25:20.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:25:19 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:25:20.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:25:19 smithi115 conmon[90393]: 2022-05-14T03:25:19.980+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:25:20.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:25:19 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:25:21.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:25:20 smithi047 ceph-mon[77147]: pgmap v7478: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:25:21.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:25:20 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:25:21.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:25:20 smithi115 conmon[90393]: 2022-05-14T03:25:20.996+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:25:21.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:25:20 smithi115 ceph-mon[78593]: pgmap v7478: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:25:21.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:25:20 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:25:22.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:25:21 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:25:22.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:25:21 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 14255 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:25:22.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:25:21 smithi115 conmon[90393]: 2022-05-14T03:25:21.949+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:25:22.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:25:21 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:25:22.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:25:21 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 14255 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:25:23.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:25:22 smithi047 ceph-mon[77147]: pgmap v7479: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:25:23.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:25:22 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:25:23.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:25:22 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:25:23.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:25:22 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:25:23.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:25:22 smithi115 conmon[90393]: 2022-05-14T03:25:22.918+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:25:23.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:25:22 smithi115 ceph-mon[78593]: pgmap v7479: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:25:23.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:25:22 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:25:23.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:25:22 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:25:23.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:25:22 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:25:24.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:25:23 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:25:24.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:25:23 smithi115 conmon[90393]: 2022-05-14T03:25:23.891+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:25:24.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:25:23 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:25:25.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:25:24 smithi047 ceph-mon[77147]: pgmap v7480: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:25:25.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:25:24 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:25:25.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:25:24 smithi115 conmon[90393]: 2022-05-14T03:25:24.905+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:25:25.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:25:24 smithi115 ceph-mon[78593]: pgmap v7480: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:25:25.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:25:24 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:25:26.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:25:25 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:25:26.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:25:25 smithi115 conmon[90393]: 2022-05-14T03:25:25.863+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:25:26.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:25:25 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:25:27.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:25:26 smithi047 ceph-mon[77147]: pgmap v7481: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:25:27.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:25:26 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:25:27.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:25:26 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 14260 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:25:27.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:25:26 smithi115 conmon[90393]: 2022-05-14T03:25:26.871+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:25:27.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:25:26 smithi115 ceph-mon[78593]: pgmap v7481: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:25:27.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:25:26 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:25:27.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:25:26 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 14260 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:25:28.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:25:27 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:25:28.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:25:27 smithi047 ceph-mon[77147]: pgmap v7482: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:25:28.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:25:27 smithi115 conmon[90393]: 2022-05-14T03:25:27.871+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:25:28.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:25:27 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:25:28.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:25:27 smithi115 ceph-mon[78593]: pgmap v7482: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:25:29.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:25:28 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:25:29.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:25:28 smithi115 conmon[90393]: 2022-05-14T03:25:28.905+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:25:29.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:25:28 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:25:30.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:25:29 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:25:30.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:25:29 smithi047 ceph-mon[77147]: pgmap v7483: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:25:30.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:25:29 smithi115 conmon[90393]: 2022-05-14T03:25:29.907+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:25:30.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:25:29 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:25:30.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:25:29 smithi115 ceph-mon[78593]: pgmap v7483: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:25:31.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:25:30 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:25:31.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:25:30 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:25:31.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:25:30 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:25:31.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:25:30 smithi115 conmon[90393]: 2022-05-14T03:25:30.873+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:25:31.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:25:30 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:25:31.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:25:30 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:25:31.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:25:30 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:25:32.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:25:31 smithi115 conmon[90393]: 2022-05-14T03:25:31.834+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:25:32.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:25:31 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:25:32.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:25:31 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 14265 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:25:32.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:25:31 smithi115 ceph-mon[78593]: pgmap v7484: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:25:32.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:25:31 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:25:32.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:25:31 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 14265 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:25:32.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:25:31 smithi047 ceph-mon[77147]: pgmap v7484: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:25:33.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:25:32 smithi115 conmon[90393]: 2022-05-14T03:25:32.821+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:25:33.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:25:33 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:25:33.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:25:33 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:25:34.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:25:33 smithi115 conmon[90393]: 2022-05-14T03:25:33.868+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:25:34.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:25:34 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:25:34.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:25:34 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:25:34.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:25:34 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:25:34.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:25:34 smithi047 ceph-mon[77147]: pgmap v7485: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:25:34.836 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:25:34 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:25:34.837 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:25:34 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:25:34.837 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:25:34 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:25:34.837 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:25:34 smithi115 ceph-mon[78593]: pgmap v7485: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:25:35.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:25:34 smithi115 conmon[90393]: 2022-05-14T03:25:34.834+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:25:35.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:25:35 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:25:35.813 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:25:35 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:25:36.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:25:35 smithi115 conmon[90393]: 2022-05-14T03:25:35.811+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:25:36.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:25:36 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:25:36.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:25:36 smithi047 ceph-mon[77147]: pgmap v7486: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:25:36.824 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:25:36 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:25:36.824 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:25:36 smithi115 ceph-mon[78593]: pgmap v7486: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:25:37.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:25:36 smithi115 conmon[90393]: 2022-05-14T03:25:36.821+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:25:37.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:25:37 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:25:37.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:25:37 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 14270 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:25:37.854 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:25:37 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:25:37.855 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:25:37 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 14270 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:25:38.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:25:37 smithi115 conmon[90393]: 2022-05-14T03:25:37.852+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:25:38.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:25:38 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:25:38.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:25:38 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:25:38.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:25:38 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:25:38.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:25:38 smithi047 ceph-mon[77147]: pgmap v7487: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:25:38.850 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:25:38 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:25:38.851 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:25:38 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:25:38.851 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:25:38 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:25:38.851 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:25:38 smithi115 ceph-mon[78593]: pgmap v7487: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:25:39.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:25:38 smithi115 conmon[90393]: 2022-05-14T03:25:38.848+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:25:39.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:25:39 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:25:39.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:25:39 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:25:40.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:25:39 smithi115 conmon[90393]: 2022-05-14T03:25:39.895+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:25:40.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:25:40 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:25:40.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:25:40 smithi047 ceph-mon[77147]: pgmap v7488: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:25:40.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:25:40 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:25:40.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:25:40 smithi115 ceph-mon[78593]: pgmap v7488: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:25:41.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:25:40 smithi115 conmon[90393]: 2022-05-14T03:25:40.898+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:25:41.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:25:41 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:25:42.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:25:41 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:25:42.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:25:41 smithi115 conmon[90393]: 2022-05-14T03:25:41.897+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:25:42.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:25:42 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:25:42.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:25:42 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:25:42.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:25:42 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:25:42.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:25:42 smithi115 ceph-mon[78593]: pgmap v7489: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:25:43.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:25:42 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:25:43.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:25:42 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:25:43.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:25:42 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:25:43.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:25:42 smithi047 ceph-mon[77147]: pgmap v7489: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:25:43.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:25:42 smithi115 conmon[90393]: 2022-05-14T03:25:42.922+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:25:43.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:25:43 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:25:44.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:25:43 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:25:44.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:25:43 smithi115 conmon[90393]: 2022-05-14T03:25:43.887+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:25:44.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:25:44 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:25:44.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:25:44 smithi115 ceph-mon[78593]: pgmap v7490: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:25:45.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:25:44 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:25:45.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:25:44 smithi047 ceph-mon[77147]: pgmap v7490: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:25:45.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:25:44 smithi115 conmon[90393]: 2022-05-14T03:25:44.895+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:25:45.850 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:25:45 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:25:46.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:25:45 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:25:46.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:25:45 smithi115 conmon[90393]: 2022-05-14T03:25:45.848+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:25:46.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:25:46 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:25:46.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:25:46 smithi115 ceph-mon[78593]: pgmap v7491: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:25:46.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:25:46 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 14275 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:25:47.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:25:46 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:25:47.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:25:46 smithi047 ceph-mon[77147]: pgmap v7491: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:25:47.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:25:46 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 14275 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:25:47.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:25:46 smithi115 conmon[90393]: 2022-05-14T03:25:46.858+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:25:47.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:25:47 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:25:48.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:25:47 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:25:48.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:25:47 smithi115 conmon[90393]: 2022-05-14T03:25:47.894+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:25:48.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:25:48 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:25:48.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:25:48 smithi115 ceph-mon[78593]: pgmap v7492: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:25:49.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:25:48 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:25:49.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:25:48 smithi047 ceph-mon[77147]: pgmap v7492: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:25:49.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:25:48 smithi115 conmon[90393]: 2022-05-14T03:25:48.871+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:25:49.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:25:49 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:25:50.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:25:49 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:25:50.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:25:49 smithi115 conmon[90393]: 2022-05-14T03:25:49.912+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:25:50.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:25:50 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:25:50.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:25:50 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:25:50.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:25:50 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:25:50.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:25:50 smithi115 ceph-mon[78593]: pgmap v7493: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:25:51.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:25:50 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:25:51.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:25:50 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:25:51.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:25:50 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:25:51.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:25:50 smithi047 ceph-mon[77147]: pgmap v7493: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:25:51.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:25:50 smithi115 conmon[90393]: 2022-05-14T03:25:50.895+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:25:51.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:25:51 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:25:51.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:25:51 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 14285 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:25:52.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:25:51 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:25:52.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:25:51 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 14285 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:25:52.361 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:25:51 smithi115 conmon[90393]: 2022-05-14T03:25:51.940+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:25:52.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:25:52 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:25:52.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:25:52 smithi115 ceph-mon[78593]: pgmap v7494: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:25:53.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:25:52 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:25:53.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:25:52 smithi047 ceph-mon[77147]: pgmap v7494: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:25:53.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:25:52 smithi115 conmon[90393]: 2022-05-14T03:25:52.923+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:25:53.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:25:53 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:25:54.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:25:53 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:25:54.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:25:53 smithi115 conmon[90393]: 2022-05-14T03:25:53.951+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:25:54.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:25:54 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:25:54.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:25:54 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:25:54.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:25:54 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:25:54.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:25:54 smithi115 ceph-mon[78593]: pgmap v7495: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:25:55.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:25:54 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:25:55.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:25:54 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:25:55.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:25:54 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:25:55.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:25:54 smithi047 ceph-mon[77147]: pgmap v7495: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:25:55.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:25:54 smithi115 conmon[90393]: 2022-05-14T03:25:54.930+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:25:55.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:25:55 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:25:56.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:25:55 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:25:56.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:25:55 smithi115 conmon[90393]: 2022-05-14T03:25:55.919+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:25:56.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:25:56 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:25:56.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:25:56 smithi115 ceph-mon[78593]: pgmap v7496: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:25:57.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:25:56 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:25:57.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:25:56 smithi047 ceph-mon[77147]: pgmap v7496: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:25:57.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:25:56 smithi115 conmon[90393]: 2022-05-14T03:25:56.963+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:25:57.934 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:25:57 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:25:57.934 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:25:57 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 14290 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:25:58.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:25:57 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:25:58.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:25:57 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 14290 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:25:58.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:25:57 smithi115 conmon[90393]: 2022-05-14T03:25:57.932+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:25:58.978 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:25:58 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:25:58.978 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:25:58 smithi115 ceph-mon[78593]: pgmap v7497: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:25:58.979 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:25:58 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:25:58.979 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:25:58 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:25:59.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:25:58 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:25:59.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:25:58 smithi047 ceph-mon[77147]: pgmap v7497: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:25:59.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:25:58 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:25:59.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:25:58 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:25:59.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:25:58 smithi115 conmon[90393]: 2022-05-14T03:25:58.976+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:25:59.950 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:25:59 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:26:00.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:25:59 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:26:00.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:25:59 smithi115 conmon[90393]: 2022-05-14T03:25:59.948+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:26:00.906 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:26:00 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:26:00.906 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:26:00 smithi115 ceph-mon[78593]: pgmap v7498: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:26:01.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:26:00 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:26:01.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:26:00 smithi047 ceph-mon[77147]: pgmap v7498: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:26:01.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:26:00 smithi115 conmon[90393]: 2022-05-14T03:26:00.903+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:26:01.956 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:26:01 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:26:02.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:26:01 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:26:02.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:26:01 smithi115 conmon[90393]: 2022-05-14T03:26:01.949+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:26:02.998 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:26:02 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:26:02.999 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:26:02 smithi115 ceph-mon[78593]: pgmap v7499: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:26:02.999 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:26:02 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:26:02.999 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:26:02 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:26:03.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:26:02 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:26:03.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:26:02 smithi047 ceph-mon[77147]: pgmap v7499: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:26:03.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:26:02 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:26:03.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:26:02 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:26:03.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:26:02 smithi115 conmon[90393]: 2022-05-14T03:26:02.996+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:26:03.988 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:26:03 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:26:04.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:26:03 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:26:04.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:26:03 smithi115 conmon[90393]: 2022-05-14T03:26:03.986+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:26:05.012 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:26:04 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:26:05.012 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:26:04 smithi115 ceph-mon[78593]: pgmap v7500: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:26:05.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:26:04 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:26:05.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:26:04 smithi047 ceph-mon[77147]: pgmap v7500: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:26:05.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:26:05 smithi115 conmon[90393]: 2022-05-14T03:26:05.010+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:26:06.007 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:26:05 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:26:06.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:26:05 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:26:06.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:26:06 smithi115 conmon[90393]: 2022-05-14T03:26:06.004+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:26:06.975 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:26:06 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:26:06.976 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:26:06 smithi047 ceph-mon[77147]: pgmap v7501: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:26:06.976 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:26:06 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 14300 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:26:06.976 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:26:06 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:26:06.977 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:26:06 smithi115 ceph-mon[78593]: pgmap v7501: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:26:06.977 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:26:06 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 14300 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:26:07.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:26:06 smithi115 conmon[90393]: 2022-05-14T03:26:06.982+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:26:07.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:26:07 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:26:07.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:26:07 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T03:26:07.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:26:07 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T03:26:07.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:26:07 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T03:26:07.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:26:07 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:26:07.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:26:07 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T03:26:07.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:26:07 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T03:26:07.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:26:07 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T03:26:08.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:26:07 smithi115 conmon[90393]: 2022-05-14T03:26:07.970+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:26:08.986 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:26:08 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:26:08.987 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:26:08 smithi115 ceph-mon[78593]: pgmap v7502: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:26:09.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:26:08 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:26:09.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:26:08 smithi047 ceph-mon[77147]: pgmap v7502: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:26:09.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:26:08 smithi115 conmon[90393]: 2022-05-14T03:26:08.984+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:26:09.940 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:26:09 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:26:10.034 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:26:09 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:26:10.292 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:26:10 smithi047 conmon[78521]: 2022-05-14T03:26:10.031+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:26:10.292 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:26:10 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T03:26:10.293 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:26:10 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:26:10.293 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:26:10 smithi047 conmon[78521]: yield 2022-05-14T03:26:10.293 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:26:10 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:26:10.293 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:26:10 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:26:10.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:26:10 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:26:10.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:26:10 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T03:26:10.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:26:10 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:26:10.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:26:10 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T03:26:10.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:26:10 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:26:10.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:26:10 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T03:26:10.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:26:10 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:26:10.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:26:10 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T03:26:10.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:26:10 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:26:10.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:26:10 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T03:26:10.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:26:10 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:26:10.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:26:10 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:26:10.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:26:10 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:26:10.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:26:09 smithi115 conmon[90393]: 2022-05-14T03:26:09.938+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:26:10.930 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:26:10 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:26:10.930 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:26:10 smithi115 ceph-mon[78593]: pgmap v7503: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:26:10.930 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:26:10 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:26:10.931 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:26:10 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:26:11.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:26:10 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:26:11.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:26:10 smithi047 ceph-mon[77147]: pgmap v7503: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:26:11.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:26:10 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:26:11.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:26:10 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:26:11.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:26:10 smithi115 conmon[90393]: 2022-05-14T03:26:10.927+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:26:11.969 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:26:11 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:26:11.970 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:26:11 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:26:11.970 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T03:26:11.971 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:26:11.971 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T03:26:11.971 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:26:11.972 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:26:11.973 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:26:11.973 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T03:26:11.973 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:26:11.974 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T03:26:11.974 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:26:11.975 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T03:26:11.975 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:26:11.975 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T03:26:11.975 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:26:11.976 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T03:26:11.976 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:26:11.976 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:26:11.976 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:26:11.976 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:26:11 smithi115 ceph-mon[78593]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T03:26:11.977 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:26:11 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 14305 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:26:12.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:26:11 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:26:12.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:26:11 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:26:12.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T03:26:12.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:26:12.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T03:26:12.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:26:12.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:26:12.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:26:12.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T03:26:12.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:26:12.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T03:26:12.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:26:12.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T03:26:12.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:26:12.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T03:26:12.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:26:12.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T03:26:12.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:26:12.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:26:12.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:26:12.047 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:26:11 smithi047 ceph-mon[77147]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T03:26:12.047 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:26:11 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 14305 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:26:12.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:26:11 smithi115 conmon[90393]: 2022-05-14T03:26:11.968+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:26:12.935 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:26:12 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:26:12.935 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:26:12 smithi115 ceph-mon[78593]: pgmap v7504: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:26:13.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:26:12 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:26:13.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:26:12 smithi047 ceph-mon[77147]: pgmap v7504: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:26:13.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:26:12 smithi115 conmon[90393]: 2022-05-14T03:26:12.932+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:26:13.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:26:13 smithi047 conmon[78521]: 2022-05-14T03:26:13.104+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:26:13.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:26:13 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T03:26:13.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:26:13 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:26:13.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:26:13 smithi047 conmon[78521]: yield 2022-05-14T03:26:13.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:26:13 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:26:13.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:26:13 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:26:13.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:26:13 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:26:13.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:26:13 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T03:26:13.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:26:13 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:26:13.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:26:13 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T03:26:13.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:26:13 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:26:13.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:26:13 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T03:26:13.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:26:13 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:26:13.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:26:13 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T03:26:13.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:26:13 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:26:13.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:26:13 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T03:26:13.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:26:13 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:26:13.547 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:26:13 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:26:13.547 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:26:13 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:26:13.547 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:26:13 smithi047 conmon[78521]: 2022-05-14T03:26:13.104+0000 7f5344781700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T03:26:13.547 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:26:13 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T03:26:13.548 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:26:13 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:26:13.548 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:26:13 smithi047 conmon[78521]: yield 2022-05-14T03:26:13.548 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:26:13 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:26:13.548 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:26:13 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:26:13.548 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:26:13 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:26:13.549 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:26:13 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T03:26:13.549 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:26:13 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:26:13.549 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:26:13 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T03:26:13.549 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:26:13 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:26:13.550 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:26:13 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T03:26:13.550 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:26:13 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:26:13.550 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:26:13 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T03:26:13.550 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:26:13 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:26:13.550 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:26:13 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T03:26:13.551 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:26:13 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:26:13.551 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:26:13 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:26:13.552 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:26:13 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:26:13.552 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:26:13 smithi047 conmon[78521]: 2022-05-14T03:26:13.552 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:26:13 smithi047 conmon[78521]: During handling of the above exception, another exception occurred: 2022-05-14T03:26:13.552 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:26:13 smithi047 conmon[78521]: 2022-05-14T03:26:13.552 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:26:13 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T03:26:13.553 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:26:13 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T03:26:13.553 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:26:13 smithi047 conmon[78521]: return f(*arg) 2022-05-14T03:26:13.553 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:26:13 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T03:26:13.553 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:26:13 smithi047 conmon[78521]: self._write_client_files(client_files, host) 2022-05-14T03:26:13.554 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:26:13 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T03:26:13.554 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:26:13 smithi047 conmon[78521]: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T03:26:13.554 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:26:13 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T03:26:13.554 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:26:13 smithi047 conmon[78521]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T03:26:13.554 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:26:13 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T03:26:13.555 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:26:13 smithi047 conmon[78521]: return self.event_loop.get_result(coro) 2022-05-14T03:26:13.555 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:26:13 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T03:26:13.555 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:26:13 smithi047 conmon[78521]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T03:26:13.555 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:26:13 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T03:26:13.555 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:26:13 smithi047 conmon[78521]: return self.__get_result() 2022-05-14T03:26:13.556 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:26:13 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T03:26:13.556 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:26:13 smithi047 conmon[78521]: raise self._exception 2022-05-14T03:26:13.556 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:26:13 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T03:26:13.556 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:26:13 smithi047 conmon[78521]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T03:26:13.557 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:26:13 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T03:26:13.557 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:26:13 smithi047 conmon[78521]: conn = await self._remote_connection(host, addr) 2022-05-14T03:26:13.557 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:26:13 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T03:26:13.557 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:26:13 smithi047 conmon[78521]: raise 2022-05-14T03:26:13.558 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:26:13 smithi047 conmon[78521]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T03:26:13.558 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:26:13 smithi047 conmon[78521]: self.gen.throw(type, value, traceback) 2022-05-14T03:26:13.558 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:26:13 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T03:26:13.558 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:26:13 smithi047 conmon[78521]: raise OrchestratorError(msg) 2022-05-14T03:26:13.558 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:26:13 smithi047 conmon[78521]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:26:14.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:26:13 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:26:14.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:26:13 smithi115 conmon[90393]: 2022-05-14T03:26:13.902+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:26:14.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:26:13 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:26:14.952 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:26:14 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:26:14.953 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:26:14 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:26:14.953 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T03:26:14.953 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:26:14.953 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T03:26:14.954 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:26:14.954 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:26:14.954 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:26:14.954 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T03:26:14.955 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:26:14.955 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T03:26:14.955 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:26:14.955 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T03:26:14.955 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:26:14.956 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T03:26:14.956 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:26:14.957 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T03:26:14.957 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:26:14.957 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:26:14.957 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:26:14.957 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:26:14 smithi115 ceph-mon[78593]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T03:26:14.958 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T03:26:14.958 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:26:14.958 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T03:26:14.958 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:26:14.959 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:26:14.959 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:26:14.960 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T03:26:14.960 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:26:14.961 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T03:26:14.961 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:26:14.962 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T03:26:14.962 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:26:14.963 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T03:26:14.963 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:26:14.963 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T03:26:14.964 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:26:14.964 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:26:14.964 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:26:14.965 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T03:26:14.965 INFO:journalctl@ceph.mon.c.smithi115.stdout: During handling of the above exception, another exception occurred: 2022-05-14T03:26:14.965 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T03:26:14.965 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T03:26:14.966 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T03:26:14.966 INFO:journalctl@ceph.mon.c.smithi115.stdout: return f(*arg) 2022-05-14T03:26:14.967 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T03:26:14.968 INFO:journalctl@ceph.mon.c.smithi115.stdout: self._write_client_files(client_files, host) 2022-05-14T03:26:14.968 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T03:26:14.968 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T03:26:14.969 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T03:26:14.969 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T03:26:14.970 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T03:26:14.970 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.event_loop.get_result(coro) 2022-05-14T03:26:14.971 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T03:26:14.971 INFO:journalctl@ceph.mon.c.smithi115.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T03:26:14.971 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T03:26:14.972 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.__get_result() 2022-05-14T03:26:14.972 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T03:26:14.972 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise self._exception 2022-05-14T03:26:14.973 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T03:26:14.974 INFO:journalctl@ceph.mon.c.smithi115.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T03:26:14.974 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T03:26:14.975 INFO:journalctl@ceph.mon.c.smithi115.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T03:26:14.975 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T03:26:14.975 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise 2022-05-14T03:26:14.976 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T03:26:14.976 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.gen.throw(type, value, traceback) 2022-05-14T03:26:14.976 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T03:26:14.977 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OrchestratorError(msg) 2022-05-14T03:26:14.977 INFO:journalctl@ceph.mon.c.smithi115.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:26:14.977 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:26:14 smithi115 ceph-mon[78593]: pgmap v7505: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:26:14.977 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:26:14 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:26:14.978 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:26:14 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:26:14.978 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:26:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T03:26:14.978 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:26:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T03:26:15.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:26:14 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:26:15.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:26:14 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:26:15.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T03:26:15.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:26:15.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T03:26:15.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:26:15.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:26:15.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:26:15.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T03:26:15.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:26:15.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T03:26:15.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:26:15.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T03:26:15.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:26:15.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T03:26:15.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:26:15.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T03:26:15.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:26:15.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:26:15.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:26:15.049 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:26:14 smithi047 ceph-mon[77147]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T03:26:15.049 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T03:26:15.049 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:26:15.050 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T03:26:15.050 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:26:15.050 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:26:15.051 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:26:15.051 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T03:26:15.051 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:26:15.052 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T03:26:15.052 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:26:15.052 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T03:26:15.052 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:26:15.053 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T03:26:15.053 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:26:15.054 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T03:26:15.054 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:26:15.054 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:26:15.055 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:26:15.055 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T03:26:15.055 INFO:journalctl@ceph.mon.b.smithi047.stdout: During handling of the above exception, another exception occurred: 2022-05-14T03:26:15.055 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T03:26:15.056 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T03:26:15.056 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T03:26:15.056 INFO:journalctl@ceph.mon.b.smithi047.stdout: return f(*arg) 2022-05-14T03:26:15.057 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T03:26:15.057 INFO:journalctl@ceph.mon.b.smithi047.stdout: self._write_client_files(client_files, host) 2022-05-14T03:26:15.057 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T03:26:15.057 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T03:26:15.058 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T03:26:15.058 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T03:26:15.058 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T03:26:15.058 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.event_loop.get_result(coro) 2022-05-14T03:26:15.058 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T03:26:15.059 INFO:journalctl@ceph.mon.b.smithi047.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T03:26:15.059 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T03:26:15.059 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.__get_result() 2022-05-14T03:26:15.059 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T03:26:15.060 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise self._exception 2022-05-14T03:26:15.060 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T03:26:15.060 INFO:journalctl@ceph.mon.b.smithi047.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T03:26:15.061 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T03:26:15.061 INFO:journalctl@ceph.mon.b.smithi047.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T03:26:15.062 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T03:26:15.062 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise 2022-05-14T03:26:15.062 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T03:26:15.063 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.gen.throw(type, value, traceback) 2022-05-14T03:26:15.063 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T03:26:15.065 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OrchestratorError(msg) 2022-05-14T03:26:15.066 INFO:journalctl@ceph.mon.b.smithi047.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:26:15.066 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:26:14 smithi047 ceph-mon[77147]: pgmap v7505: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:26:15.066 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:26:14 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:26:15.066 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:26:14 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:26:15.067 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:26:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T03:26:15.067 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:26:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T03:26:15.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:26:14 smithi115 conmon[90393]: 2022-05-14T03:26:14.949+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:26:15.932 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:26:15 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:26:16.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:26:15 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:26:16.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:26:15 smithi115 conmon[90393]: 2022-05-14T03:26:15.930+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:26:16.981 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:26:16 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:26:16.981 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:26:16 smithi115 ceph-mon[78593]: pgmap v7506: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:26:16.981 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:26:16 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 14310 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:26:17.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:26:16 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:26:17.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:26:16 smithi047 ceph-mon[77147]: pgmap v7506: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:26:17.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:26:16 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 14310 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:26:17.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:26:16 smithi115 conmon[90393]: 2022-05-14T03:26:16.979+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:26:17.940 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:26:17 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:26:18.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:26:17 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:26:18.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:26:17 smithi115 conmon[90393]: 2022-05-14T03:26:17.938+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:26:18.964 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:26:18 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:26:18.965 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:26:18 smithi115 ceph-mon[78593]: pgmap v7507: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:26:18.965 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:26:18 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:26:18.965 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:26:18 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:26:19.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:26:18 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:26:19.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:26:18 smithi047 ceph-mon[77147]: pgmap v7507: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:26:19.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:26:18 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:26:19.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:26:18 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:26:19.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:26:18 smithi115 conmon[90393]: 2022-05-14T03:26:18.962+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:26:19.972 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:26:19 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:26:20.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:26:19 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:26:20.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:26:19 smithi115 conmon[90393]: 2022-05-14T03:26:19.970+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:26:20.955 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:26:20 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:26:20.956 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:26:20 smithi115 ceph-mon[78593]: pgmap v7508: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:26:21.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:26:20 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:26:21.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:26:20 smithi047 ceph-mon[77147]: pgmap v7508: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:26:21.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:26:20 smithi115 conmon[90393]: 2022-05-14T03:26:20.954+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:26:21.998 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:26:21 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:26:21.998 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:26:21 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 14315 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:26:22.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:26:21 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:26:22.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:26:21 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 14315 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:26:22.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:26:21 smithi115 conmon[90393]: 2022-05-14T03:26:21.996+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:26:22.963 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:26:22 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:26:22.963 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:26:22 smithi115 ceph-mon[78593]: pgmap v7509: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:26:22.963 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:26:22 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:26:22.963 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:26:22 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:26:23.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:26:22 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:26:23.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:26:22 smithi047 ceph-mon[77147]: pgmap v7509: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:26:23.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:26:22 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:26:23.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:26:22 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:26:23.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:26:22 smithi115 conmon[90393]: 2022-05-14T03:26:22.960+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:26:24.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:26:23 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:26:24.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:26:23 smithi115 conmon[90393]: 2022-05-14T03:26:23.931+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:26:24.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:26:23 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:26:25.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:26:24 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:26:25.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:26:24 smithi047 ceph-mon[77147]: pgmap v7510: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:26:25.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:26:24 smithi115 conmon[90393]: 2022-05-14T03:26:24.917+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:26:25.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:26:24 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:26:25.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:26:24 smithi115 ceph-mon[78593]: pgmap v7510: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:26:26.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:26:25 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:26:26.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:26:25 smithi115 conmon[90393]: 2022-05-14T03:26:25.941+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:26:26.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:26:25 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:26:27.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:26:26 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:26:27.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:26:26 smithi047 ceph-mon[77147]: pgmap v7511: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:26:27.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:26:26 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 14320 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:26:27.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:26:26 smithi115 conmon[90393]: 2022-05-14T03:26:26.936+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:26:27.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:26:26 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:26:27.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:26:26 smithi115 ceph-mon[78593]: pgmap v7511: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:26:27.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:26:26 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 14320 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:26:28.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:26:27 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:26:28.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:26:27 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:26:28.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:26:27 smithi115 conmon[90393]: 2022-05-14T03:26:27.933+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:26:29.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:26:28 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:26:29.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:26:28 smithi047 ceph-mon[77147]: pgmap v7512: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:26:29.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:26:28 smithi115 conmon[90393]: 2022-05-14T03:26:28.897+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:26:29.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:26:28 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:26:29.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:26:28 smithi115 ceph-mon[78593]: pgmap v7512: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:26:30.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:26:29 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:26:30.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:26:29 smithi115 conmon[90393]: 2022-05-14T03:26:29.869+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:26:30.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:26:29 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:26:31.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:26:30 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:26:31.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:26:30 smithi047 ceph-mon[77147]: pgmap v7513: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:26:31.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:26:30 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:26:31.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:26:30 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:26:31.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:26:30 smithi115 conmon[90393]: 2022-05-14T03:26:30.853+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:26:31.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:26:30 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:26:31.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:26:30 smithi115 ceph-mon[78593]: pgmap v7513: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:26:31.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:26:30 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:26:31.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:26:30 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:26:32.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:26:31 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:26:32.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:26:31 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 14325 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:26:32.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:26:31 smithi115 conmon[90393]: 2022-05-14T03:26:31.831+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:26:32.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:26:31 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:26:32.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:26:31 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 14325 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:26:33.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:26:32 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:26:33.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:26:32 smithi047 ceph-mon[77147]: pgmap v7514: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:26:33.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:26:32 smithi115 conmon[90393]: 2022-05-14T03:26:32.836+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:26:33.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:26:32 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:26:33.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:26:32 smithi115 ceph-mon[78593]: pgmap v7514: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:26:34.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:26:33 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:26:34.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:26:33 smithi115 conmon[90393]: 2022-05-14T03:26:33.792+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:26:34.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:26:33 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:26:35.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:26:34 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:26:35.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:26:34 smithi047 ceph-mon[77147]: pgmap v7515: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:26:35.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:26:34 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:26:35.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:26:34 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:26:35.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:26:34 smithi115 conmon[90393]: 2022-05-14T03:26:34.828+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:26:35.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:26:34 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:26:35.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:26:34 smithi115 ceph-mon[78593]: pgmap v7515: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:26:35.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:26:34 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:26:35.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:26:34 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:26:36.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:26:35 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:26:36.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:26:35 smithi115 conmon[90393]: 2022-05-14T03:26:35.852+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:26:36.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:26:35 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:26:37.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:26:36 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:26:37.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:26:36 smithi047 ceph-mon[77147]: pgmap v7516: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:26:37.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:26:36 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 14330 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:26:37.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:26:36 smithi115 conmon[90393]: 2022-05-14T03:26:36.818+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:26:37.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:26:36 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:26:37.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:26:36 smithi115 ceph-mon[78593]: pgmap v7516: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:26:37.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:26:36 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 14330 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:26:38.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:26:37 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:26:38.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:26:37 smithi115 conmon[90393]: 2022-05-14T03:26:37.778+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:26:38.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:26:37 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:26:39.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:26:38 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:26:39.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:26:38 smithi047 ceph-mon[77147]: pgmap v7517: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:26:39.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:26:38 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:26:39.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:26:38 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:26:39.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:26:38 smithi115 conmon[90393]: 2022-05-14T03:26:38.813+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:26:39.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:26:38 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:26:39.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:26:38 smithi115 ceph-mon[78593]: pgmap v7517: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:26:39.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:26:38 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:26:39.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:26:38 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:26:40.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:26:39 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:26:40.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:26:39 smithi115 conmon[90393]: 2022-05-14T03:26:39.780+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:26:40.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:26:39 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:26:41.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:26:40 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:26:41.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:26:40 smithi047 ceph-mon[77147]: pgmap v7518: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:26:41.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:26:40 smithi115 conmon[90393]: 2022-05-14T03:26:40.790+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:26:41.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:26:40 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:26:41.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:26:40 smithi115 ceph-mon[78593]: pgmap v7518: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:26:42.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:26:41 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:26:42.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:26:41 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 14335 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:26:42.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:26:41 smithi115 conmon[90393]: 2022-05-14T03:26:41.835+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:26:42.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:26:41 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:26:42.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:26:41 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 14335 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:26:43.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:26:42 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:26:43.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:26:42 smithi047 ceph-mon[77147]: pgmap v7519: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:26:43.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:26:42 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:26:43.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:26:42 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:26:43.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:26:42 smithi115 conmon[90393]: 2022-05-14T03:26:42.828+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:26:43.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:26:42 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:26:43.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:26:42 smithi115 ceph-mon[78593]: pgmap v7519: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:26:43.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:26:42 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:26:43.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:26:42 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:26:44.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:26:43 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:26:44.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:26:43 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:26:44.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:26:43 smithi115 conmon[90393]: 2022-05-14T03:26:43.844+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:26:45.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:26:44 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:26:45.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:26:44 smithi047 ceph-mon[77147]: pgmap v7520: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:26:45.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:26:44 smithi115 conmon[90393]: 2022-05-14T03:26:44.820+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:26:45.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:26:44 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:26:45.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:26:44 smithi115 ceph-mon[78593]: pgmap v7520: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:26:46.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:26:45 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:26:46.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:26:45 smithi115 conmon[90393]: 2022-05-14T03:26:45.797+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:26:46.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:26:45 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:26:47.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:26:46 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:26:47.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:26:46 smithi047 ceph-mon[77147]: pgmap v7521: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:26:47.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:26:46 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 14340 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:26:47.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:26:46 smithi115 conmon[90393]: 2022-05-14T03:26:46.801+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:26:47.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:26:46 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:26:47.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:26:46 smithi115 ceph-mon[78593]: pgmap v7521: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:26:47.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:26:46 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 14340 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:26:48.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:26:47 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:26:48.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:26:47 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:26:48.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:26:47 smithi115 conmon[90393]: 2022-05-14T03:26:47.811+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:26:48.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:26:47 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:26:48.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:26:47 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:26:49.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:26:48 smithi047 ceph-mon[77147]: pgmap v7522: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:26:49.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:26:48 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:26:49.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:26:48 smithi115 conmon[90393]: 2022-05-14T03:26:48.780+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:26:49.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:26:48 smithi115 ceph-mon[78593]: pgmap v7522: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:26:49.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:26:48 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:26:50.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:26:49 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:26:50.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:26:49 smithi115 conmon[90393]: 2022-05-14T03:26:49.828+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:26:50.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:26:49 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:26:51.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:26:50 smithi047 ceph-mon[77147]: pgmap v7523: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:26:51.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:26:50 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:26:51.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:26:50 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:26:51.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:26:50 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:26:51.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:26:50 smithi115 conmon[90393]: 2022-05-14T03:26:50.875+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:26:51.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:26:50 smithi115 ceph-mon[78593]: pgmap v7523: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:26:51.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:26:50 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:26:51.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:26:50 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:26:51.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:26:50 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:26:52.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:26:51 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:26:52.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:26:51 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 14345 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:26:52.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:26:51 smithi115 conmon[90393]: 2022-05-14T03:26:51.872+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:26:52.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:26:51 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:26:52.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:26:51 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 14345 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:26:53.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:26:52 smithi047 ceph-mon[77147]: pgmap v7524: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:26:53.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:26:52 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:26:53.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:26:52 smithi115 conmon[90393]: 2022-05-14T03:26:52.845+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:26:53.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:26:52 smithi115 ceph-mon[78593]: pgmap v7524: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:26:53.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:26:52 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:26:54.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:26:53 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:26:54.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:26:53 smithi115 conmon[90393]: 2022-05-14T03:26:53.801+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:26:54.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:26:53 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:26:55.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:26:54 smithi047 ceph-mon[77147]: pgmap v7525: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:26:55.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:26:54 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:26:55.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:26:54 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:26:55.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:26:54 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:26:55.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:26:54 smithi115 conmon[90393]: 2022-05-14T03:26:54.796+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:26:55.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:26:54 smithi115 ceph-mon[78593]: pgmap v7525: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:26:55.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:26:54 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:26:55.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:26:54 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:26:55.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:26:54 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:26:56.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:26:55 smithi115 conmon[90393]: 2022-05-14T03:26:55.844+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:26:56.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:26:55 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:26:56.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:26:55 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:26:57.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:26:56 smithi115 conmon[90393]: 2022-05-14T03:26:56.832+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:26:57.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:26:56 smithi115 ceph-mon[78593]: pgmap v7526: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:26:57.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:26:56 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:26:57.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:26:56 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 14350 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:26:57.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:26:56 smithi047 ceph-mon[77147]: pgmap v7526: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:26:57.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:26:56 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:26:57.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:26:56 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 14350 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:26:58.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:26:57 smithi115 conmon[90393]: 2022-05-14T03:26:57.820+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:26:58.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:26:57 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:26:58.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:26:57 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:26:59.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:26:58 smithi115 conmon[90393]: 2022-05-14T03:26:58.776+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:26:59.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:26:58 smithi115 ceph-mon[78593]: pgmap v7527: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:26:59.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:26:58 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:26:59.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:26:58 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:26:59.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:26:58 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:26:59.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:26:58 smithi047 ceph-mon[77147]: pgmap v7527: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:26:59.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:26:58 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:26:59.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:26:58 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:26:59.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:26:58 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:27:00.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:26:59 smithi115 conmon[90393]: 2022-05-14T03:26:59.801+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:27:00.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:26:59 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:27:00.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:26:59 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:27:01.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:27:00 smithi115 conmon[90393]: 2022-05-14T03:27:00.846+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:27:01.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:27:00 smithi115 ceph-mon[78593]: pgmap v7528: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:27:01.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:27:00 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:27:01.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:27:00 smithi047 ceph-mon[77147]: pgmap v7528: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:27:01.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:27:00 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:27:02.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:27:01 smithi115 conmon[90393]: 2022-05-14T03:27:01.877+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:27:02.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:27:01 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:27:02.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:27:01 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 14355 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:27:02.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:27:01 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:27:02.329 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:27:01 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 14355 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:27:03.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:27:02 smithi115 conmon[90393]: 2022-05-14T03:27:02.918+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:27:03.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:27:02 smithi115 ceph-mon[78593]: pgmap v7529: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:27:03.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:27:02 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:27:03.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:27:02 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:27:03.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:27:02 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:27:03.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:27:02 smithi047 ceph-mon[77147]: pgmap v7529: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:27:03.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:27:02 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:27:03.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:27:02 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:27:03.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:27:02 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:27:04.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:27:03 smithi115 conmon[90393]: 2022-05-14T03:27:03.913+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:27:04.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:27:03 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:27:04.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:27:03 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:27:05.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:27:04 smithi115 conmon[90393]: 2022-05-14T03:27:04.948+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:27:05.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:27:04 smithi115 ceph-mon[78593]: pgmap v7530: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:27:05.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:27:04 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:27:05.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:27:04 smithi047 ceph-mon[77147]: pgmap v7530: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:27:05.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:27:04 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:27:06.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:27:05 smithi115 conmon[90393]: 2022-05-14T03:27:05.933+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:27:06.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:27:05 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:27:06.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:27:05 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:27:07.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:27:06 smithi115 conmon[90393]: 2022-05-14T03:27:06.920+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:27:07.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:27:06 smithi115 ceph-mon[78593]: pgmap v7531: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:27:07.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:27:06 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:27:07.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:27:06 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 14360 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:27:07.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:27:06 smithi047 ceph-mon[77147]: pgmap v7531: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:27:07.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:27:06 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:27:07.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:27:06 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 14360 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:27:08.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:27:07 smithi115 conmon[90393]: 2022-05-14T03:27:07.940+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:27:08.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:27:07 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:27:08.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:27:07 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:27:09.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:27:08 smithi115 conmon[90393]: 2022-05-14T03:27:08.977+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:27:09.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:27:08 smithi115 ceph-mon[78593]: pgmap v7532: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:27:09.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:27:08 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:27:09.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:27:08 smithi047 ceph-mon[77147]: pgmap v7532: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:27:09.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:27:08 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:27:10.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:27:09 smithi115 conmon[90393]: 2022-05-14T03:27:09.935+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:27:10.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:27:09 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:27:10.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:27:09 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:27:11.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:27:10 smithi115 conmon[90393]: 2022-05-14T03:27:10.958+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:27:11.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:27:10 smithi115 ceph-mon[78593]: pgmap v7533: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:27:11.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:27:10 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:27:11.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:27:10 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:27:11.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:27:10 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:27:11.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:27:10 smithi047 ceph-mon[77147]: pgmap v7533: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:27:11.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:27:10 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:27:11.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:27:10 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:27:11.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:27:10 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:27:12.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:27:12 smithi115 conmon[90393]: 2022-05-14T03:27:12.001+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:27:12.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:27:11 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:27:12.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:27:11 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 14365 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:27:12.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:27:11 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:27:12.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:27:11 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 14365 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:27:13.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:27:13 smithi115 conmon[90393]: 2022-05-14T03:27:13.005+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:27:13.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:27:12 smithi115 ceph-mon[78593]: pgmap v7534: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:27:13.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:27:12 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:27:13.151 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:27:12 smithi047 ceph-mon[77147]: pgmap v7534: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:27:13.151 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:27:12 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:27:14.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:27:13 smithi115 conmon[90393]: 2022-05-14T03:27:13.989+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:27:14.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:27:13 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:27:14.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:27:13 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T03:27:14.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:27:13 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T03:27:14.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:27:13 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T03:27:14.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:27:13 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' 2022-05-14T03:27:14.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:27:13 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' 2022-05-14T03:27:14.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:27:13 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:27:14.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:27:13 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T03:27:14.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:27:13 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T03:27:14.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:27:13 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T03:27:14.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:27:13 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' 2022-05-14T03:27:14.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:27:13 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' 2022-05-14T03:27:15.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:27:14 smithi115 conmon[90393]: 2022-05-14T03:27:14.945+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:27:15.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:27:14 smithi115 ceph-mon[78593]: pgmap v7535: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:27:15.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:27:14 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:27:15.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:27:14 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:27:15.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:27:14 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:27:15.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:27:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T03:27:15.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:27:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T03:27:15.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:27:14 smithi047 ceph-mon[77147]: pgmap v7535: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:27:15.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:27:14 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:27:15.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:27:14 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:27:15.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:27:14 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:27:15.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:27:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T03:27:15.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:27:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T03:27:16.119 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:27:15 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:27:16.119 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:27:15 smithi115 conmon[90393]: 2022-05-14T03:27:15.983+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:27:16.172 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:27:15 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:27:16.542 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:27:16 smithi047 conmon[78521]: 2022-05-14T03:27:16.209+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:27:16.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:27:16 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T03:27:16.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:27:16 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:27:16.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:27:16 smithi047 conmon[78521]: yield 2022-05-14T03:27:16.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:27:16 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:27:16.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:27:16 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:27:16.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:27:16 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:27:16.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:27:16 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T03:27:16.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:27:16 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:27:16.547 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:27:16 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T03:27:16.547 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:27:16 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:27:16.547 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:27:16 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T03:27:16.547 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:27:16 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:27:16.548 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:27:16 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T03:27:16.548 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:27:16 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:27:16.548 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:27:16 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T03:27:16.548 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:27:16 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:27:16.548 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:27:16 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:27:16.549 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:27:16 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:27:17.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:27:16 smithi047 ceph-mon[77147]: pgmap v7536: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:27:17.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:27:16 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:27:17.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:27:16 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' 2022-05-14T03:27:17.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:27:16 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' 2022-05-14T03:27:17.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:27:16 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 14370 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:27:17.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:27:16 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-05-14T03:27:17.045 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:27:16 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-05-14T03:27:17.045 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:27:16 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-05-14T03:27:17.045 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:27:16 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-05-14T03:27:17.046 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:27:16 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-05-14T03:27:17.046 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:27:16 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-05-14T03:27:17.046 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:27:16 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-05-14T03:27:17.046 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:27:16 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-05-14T03:27:17.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:27:16 smithi115 conmon[90393]: 2022-05-14T03:27:16.997+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:27:17.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:27:16 smithi115 ceph-mon[78593]: pgmap v7536: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:27:17.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:27:16 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:27:17.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:27:16 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' 2022-05-14T03:27:17.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:27:16 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' 2022-05-14T03:27:17.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:27:16 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 14370 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:27:17.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:27:16 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-05-14T03:27:17.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:27:16 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-05-14T03:27:17.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:27:16 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-05-14T03:27:17.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:27:16 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-05-14T03:27:17.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:27:16 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-05-14T03:27:17.362 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:27:16 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-05-14T03:27:17.362 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:27:16 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-05-14T03:27:17.362 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:27:16 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-05-14T03:27:18.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:27:17 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:27:18.292 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T03:27:18.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:27:18.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T03:27:18.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:27:18.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:27:18.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:27:18.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T03:27:18.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:27:18.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T03:27:18.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:27:18.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T03:27:18.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:27:18.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T03:27:18.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:27:18.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T03:27:18.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:27:18.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:27:18.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:27:18.297 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:27:17 smithi047 ceph-mon[77147]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T03:27:18.297 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:27:17 smithi047 ceph-mon[77147]: Adjusting osd_memory_target on smithi115 to 197.0M 2022-05-14T03:27:18.298 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:27:17 smithi047 ceph-mon[77147]: Unable to set osd_memory_target on smithi115 to 206629785: error parsing value: Value '206629785' is below minimum 939524096 2022-05-14T03:27:18.298 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:27:17 smithi047 ceph-mon[77147]: Adjusting osd_memory_target on smithi047 to 149.4M 2022-05-14T03:27:18.298 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:27:17 smithi047 ceph-mon[77147]: Unable to set osd_memory_target on smithi047 to 156753141: error parsing value: Value '156753141' is below minimum 939524096 2022-05-14T03:27:18.298 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:27:17 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:27:18.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:27:18 smithi115 conmon[90393]: 2022-05-14T03:27:18.011+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:27:18.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:27:17 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:27:18.359 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T03:27:18.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:27:18.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T03:27:18.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:27:18.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:27:18.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:27:18.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T03:27:18.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:27:18.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T03:27:18.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:27:18.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T03:27:18.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:27:18.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T03:27:18.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:27:18.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T03:27:18.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:27:18.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:27:18.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:27:18.364 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:27:17 smithi115 ceph-mon[78593]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T03:27:18.364 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:27:17 smithi115 ceph-mon[78593]: Adjusting osd_memory_target on smithi115 to 197.0M 2022-05-14T03:27:18.364 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:27:17 smithi115 ceph-mon[78593]: Unable to set osd_memory_target on smithi115 to 206629785: error parsing value: Value '206629785' is below minimum 939524096 2022-05-14T03:27:18.364 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:27:17 smithi115 ceph-mon[78593]: Adjusting osd_memory_target on smithi047 to 149.4M 2022-05-14T03:27:18.364 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:27:17 smithi115 ceph-mon[78593]: Unable to set osd_memory_target on smithi047 to 156753141: error parsing value: Value '156753141' is below minimum 939524096 2022-05-14T03:27:18.365 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:27:17 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:27:19.283 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:27:18 smithi047 ceph-mon[77147]: pgmap v7537: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:27:19.283 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:27:18 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:27:19.283 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:27:18 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:27:19.284 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:27:18 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:27:19.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:27:19 smithi115 conmon[90393]: 2022-05-14T03:27:19.000+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:27:19.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:27:18 smithi115 ceph-mon[78593]: pgmap v7537: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:27:19.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:27:18 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:27:19.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:27:18 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:27:19.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:27:18 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:27:19.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:27:19 smithi047 conmon[78521]: 2022-05-14T03:27:19.279+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:27:19.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:27:19 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T03:27:19.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:27:19 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:27:19.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:27:19 smithi047 conmon[78521]: yield 2022-05-14T03:27:19.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:27:19 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:27:19.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:27:19 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:27:19.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:27:19 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:27:19.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:27:19 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T03:27:19.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:27:19 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:27:19.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:27:19 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T03:27:19.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:27:19 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:27:19.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:27:19 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T03:27:19.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:27:19 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:27:19.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:27:19 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T03:27:19.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:27:19 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:27:19.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:27:19 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T03:27:19.547 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:27:19 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:27:19.547 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:27:19 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:27:19.547 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:27:19 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:27:19.547 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:27:19 smithi047 conmon[78521]: 2022-05-14T03:27:19.280+0000 7f533f777700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T03:27:19.548 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:27:19 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T03:27:19.548 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:27:19 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:27:19.548 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:27:19 smithi047 conmon[78521]: yield 2022-05-14T03:27:19.549 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:27:19 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:27:19.549 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:27:19 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:27:19.549 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:27:19 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:27:19.550 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:27:19 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T03:27:19.551 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:27:19 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:27:19.551 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:27:19 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T03:27:19.551 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:27:19 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:27:19.552 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:27:19 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T03:27:19.552 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:27:19 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:27:19.553 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:27:19 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T03:27:19.553 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:27:19 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:27:19.554 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:27:19 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T03:27:19.554 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:27:19 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:27:19.554 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:27:19 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:27:19.555 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:27:19 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:27:19.555 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:27:19 smithi047 conmon[78521]: 2022-05-14T03:27:19.556 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:27:19 smithi047 conmon[78521]: During handling of the above exception, another exception occurred: 2022-05-14T03:27:19.556 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:27:19 smithi047 conmon[78521]: 2022-05-14T03:27:19.556 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:27:19 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T03:27:19.557 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:27:19 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T03:27:19.558 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:27:19 smithi047 conmon[78521]: return f(*arg) 2022-05-14T03:27:19.558 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:27:19 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T03:27:19.558 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:27:19 smithi047 conmon[78521]: self._write_client_files(client_files, host) 2022-05-14T03:27:19.559 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:27:19 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T03:27:19.559 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:27:19 smithi047 conmon[78521]: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T03:27:19.559 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:27:19 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T03:27:19.560 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:27:19 smithi047 conmon[78521]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T03:27:19.560 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:27:19 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T03:27:19.560 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:27:19 smithi047 conmon[78521]: return self.event_loop.get_result(coro) 2022-05-14T03:27:19.561 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:27:19 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T03:27:19.561 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:27:19 smithi047 conmon[78521]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T03:27:19.562 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:27:19 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T03:27:19.562 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:27:19 smithi047 conmon[78521]: return self.__get_result() 2022-05-14T03:27:19.562 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:27:19 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T03:27:19.563 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:27:19 smithi047 conmon[78521]: raise self._exception 2022-05-14T03:27:19.563 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:27:19 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T03:27:19.563 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:27:19 smithi047 conmon[78521]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T03:27:19.563 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:27:19 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T03:27:19.563 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:27:19 smithi047 conmon[78521]: conn = await self._remote_connection(host, addr) 2022-05-14T03:27:19.564 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:27:19 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T03:27:19.564 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:27:19 smithi047 conmon[78521]: raise 2022-05-14T03:27:19.564 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:27:19 smithi047 conmon[78521]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T03:27:19.564 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:27:19 smithi047 conmon[78521]: self.gen.throw(type, value, traceback) 2022-05-14T03:27:19.565 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:27:19 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T03:27:19.565 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:27:19 smithi047 conmon[78521]: raise OrchestratorError(msg) 2022-05-14T03:27:19.565 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:27:19 smithi047 conmon[78521]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:27:20.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:27:19 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:27:20.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:27:20 smithi115 conmon[90393]: 2022-05-14T03:27:20.014+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:27:20.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:27:19 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:27:21.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:27:20 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:27:21.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T03:27:21.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:27:21.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T03:27:21.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:27:21.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:27:21.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:27:21.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T03:27:21.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:27:21.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T03:27:21.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:27:21.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T03:27:21.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:27:21.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T03:27:21.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:27:21.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T03:27:21.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:27:21.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:27:21.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:27:21.297 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:27:20 smithi047 ceph-mon[77147]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T03:27:21.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T03:27:21.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:27:21.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T03:27:21.298 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:27:21.298 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:27:21.298 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:27:21.298 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T03:27:21.299 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:27:21.299 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T03:27:21.299 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:27:21.299 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T03:27:21.299 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:27:21.300 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T03:27:21.300 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:27:21.300 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T03:27:21.300 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:27:21.301 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:27:21.301 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:27:21.301 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T03:27:21.301 INFO:journalctl@ceph.mon.b.smithi047.stdout: During handling of the above exception, another exception occurred: 2022-05-14T03:27:21.301 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T03:27:21.302 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T03:27:21.302 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T03:27:21.302 INFO:journalctl@ceph.mon.b.smithi047.stdout: return f(*arg) 2022-05-14T03:27:21.302 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T03:27:21.303 INFO:journalctl@ceph.mon.b.smithi047.stdout: self._write_client_files(client_files, host) 2022-05-14T03:27:21.303 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T03:27:21.303 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T03:27:21.303 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T03:27:21.303 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T03:27:21.304 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T03:27:21.304 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.event_loop.get_result(coro) 2022-05-14T03:27:21.304 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T03:27:21.304 INFO:journalctl@ceph.mon.b.smithi047.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T03:27:21.305 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T03:27:21.305 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.__get_result() 2022-05-14T03:27:21.305 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T03:27:21.305 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise self._exception 2022-05-14T03:27:21.306 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T03:27:21.306 INFO:journalctl@ceph.mon.b.smithi047.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T03:27:21.306 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T03:27:21.306 INFO:journalctl@ceph.mon.b.smithi047.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T03:27:21.307 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T03:27:21.307 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise 2022-05-14T03:27:21.307 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T03:27:21.307 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.gen.throw(type, value, traceback) 2022-05-14T03:27:21.308 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T03:27:21.308 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OrchestratorError(msg) 2022-05-14T03:27:21.308 INFO:journalctl@ceph.mon.b.smithi047.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:27:21.308 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:27:20 smithi047 ceph-mon[77147]: pgmap v7538: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:27:21.308 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:27:20 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:27:21.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:27:21 smithi115 conmon[90393]: 2022-05-14T03:27:21.026+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:27:21.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:27:20 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:27:21.359 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T03:27:21.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:27:21.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T03:27:21.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:27:21.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:27:21.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:27:21.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T03:27:21.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:27:21.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T03:27:21.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:27:21.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T03:27:21.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:27:21.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T03:27:21.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:27:21.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T03:27:21.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:27:21.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:27:21.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:27:21.364 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:27:20 smithi115 ceph-mon[78593]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T03:27:21.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T03:27:21.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:27:21.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T03:27:21.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:27:21.365 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:27:21.365 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:27:21.365 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T03:27:21.365 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:27:21.366 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T03:27:21.366 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:27:21.366 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T03:27:21.367 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:27:21.367 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T03:27:21.367 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:27:21.367 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T03:27:21.368 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:27:21.368 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:27:21.368 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:27:21.369 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T03:27:21.369 INFO:journalctl@ceph.mon.c.smithi115.stdout: During handling of the above exception, another exception occurred: 2022-05-14T03:27:21.369 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T03:27:21.369 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T03:27:21.369 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T03:27:21.370 INFO:journalctl@ceph.mon.c.smithi115.stdout: return f(*arg) 2022-05-14T03:27:21.370 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T03:27:21.370 INFO:journalctl@ceph.mon.c.smithi115.stdout: self._write_client_files(client_files, host) 2022-05-14T03:27:21.370 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T03:27:21.371 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T03:27:21.371 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T03:27:21.371 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T03:27:21.371 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T03:27:21.372 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.event_loop.get_result(coro) 2022-05-14T03:27:21.372 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T03:27:21.372 INFO:journalctl@ceph.mon.c.smithi115.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T03:27:21.372 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T03:27:21.372 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.__get_result() 2022-05-14T03:27:21.373 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T03:27:21.373 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise self._exception 2022-05-14T03:27:21.373 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T03:27:21.373 INFO:journalctl@ceph.mon.c.smithi115.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T03:27:21.374 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T03:27:21.374 INFO:journalctl@ceph.mon.c.smithi115.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T03:27:21.374 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T03:27:21.374 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise 2022-05-14T03:27:21.375 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T03:27:21.375 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.gen.throw(type, value, traceback) 2022-05-14T03:27:21.375 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T03:27:21.375 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OrchestratorError(msg) 2022-05-14T03:27:21.375 INFO:journalctl@ceph.mon.c.smithi115.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:27:21.376 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:27:20 smithi115 ceph-mon[78593]: pgmap v7538: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:27:21.376 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:27:20 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:27:22.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:27:21 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:27:22.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:27:21 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 14375 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:27:22.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:27:22 smithi115 conmon[90393]: 2022-05-14T03:27:22.070+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:27:22.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:27:21 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:27:22.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:27:21 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 14375 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:27:23.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:27:22 smithi047 ceph-mon[77147]: pgmap v7539: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:27:23.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:27:22 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:27:23.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:27:22 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:27:23.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:27:22 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:27:23.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:27:23 smithi115 conmon[90393]: 2022-05-14T03:27:23.118+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:27:23.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:27:22 smithi115 ceph-mon[78593]: pgmap v7539: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:27:23.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:27:22 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:27:23.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:27:22 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:27:23.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:27:22 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:27:24.169 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:27:23 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:27:24.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:27:23 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:27:24.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:27:24 smithi115 conmon[90393]: 2022-05-14T03:27:24.165+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:27:25.211 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:27:24 smithi115 ceph-mon[78593]: pgmap v7540: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:27:25.211 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:27:24 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:27:25.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:27:24 smithi047 ceph-mon[77147]: pgmap v7540: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:27:25.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:27:24 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:27:25.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:27:25 smithi115 conmon[90393]: 2022-05-14T03:27:25.208+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:27:26.231 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:27:25 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:27:26.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:27:25 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:27:26.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:27:26 smithi115 conmon[90393]: 2022-05-14T03:27:26.229+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:27:27.200 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:27:26 smithi115 ceph-mon[78593]: pgmap v7541: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:27:27.201 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:27:26 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:27:27.201 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:27:26 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 14380 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:27:27.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:27:26 smithi047 ceph-mon[77147]: pgmap v7541: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:27:27.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:27:26 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:27:27.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:27:26 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 14380 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:27:27.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:27:27 smithi115 conmon[90393]: 2022-05-14T03:27:27.198+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:27:28.154 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:27:27 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:27:28.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:27:27 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:27:28.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:27:28 smithi115 conmon[90393]: 2022-05-14T03:27:28.151+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:27:29.180 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:27:28 smithi115 ceph-mon[78593]: pgmap v7542: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:27:29.181 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:27:28 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:27:29.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:27:28 smithi047 ceph-mon[77147]: pgmap v7542: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:27:29.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:27:28 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:27:29.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:27:29 smithi115 conmon[90393]: 2022-05-14T03:27:29.178+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:27:30.193 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:27:29 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:27:30.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:27:29 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:27:30.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:27:30 smithi115 conmon[90393]: 2022-05-14T03:27:30.190+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:27:31.203 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:27:30 smithi115 ceph-mon[78593]: pgmap v7543: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:27:31.203 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:27:30 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:27:31.203 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:27:30 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:27:31.204 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:27:30 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:27:31.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:27:30 smithi047 ceph-mon[77147]: pgmap v7543: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:27:31.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:27:30 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:27:31.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:27:30 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:27:31.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:27:30 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:27:31.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:27:31 smithi115 conmon[90393]: 2022-05-14T03:27:31.200+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:27:32.162 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:27:31 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:27:32.162 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:27:31 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 14385 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:27:32.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:27:31 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:27:32.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:27:31 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 14385 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:27:32.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:27:32 smithi115 conmon[90393]: 2022-05-14T03:27:32.159+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:27:33.196 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:27:32 smithi115 ceph-mon[78593]: pgmap v7544: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:27:33.196 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:27:32 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:27:33.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:27:32 smithi047 ceph-mon[77147]: pgmap v7544: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:27:33.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:27:32 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:27:33.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:27:33 smithi115 conmon[90393]: 2022-05-14T03:27:33.193+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:27:34.200 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:27:33 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:27:34.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:27:33 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:27:34.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:27:34 smithi115 conmon[90393]: 2022-05-14T03:27:34.198+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:27:35.177 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:27:34 smithi115 ceph-mon[78593]: pgmap v7545: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:27:35.178 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:27:34 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:27:35.178 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:27:34 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:27:35.179 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:27:34 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:27:35.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:27:34 smithi047 ceph-mon[77147]: pgmap v7545: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:27:35.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:27:34 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:27:35.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:27:34 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:27:35.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:27:34 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:27:35.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:27:35 smithi115 conmon[90393]: 2022-05-14T03:27:35.174+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:27:36.206 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:27:35 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:27:36.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:27:35 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:27:36.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:27:36 smithi115 conmon[90393]: 2022-05-14T03:27:36.203+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:27:37.213 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:27:36 smithi115 ceph-mon[78593]: pgmap v7546: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:27:37.214 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:27:36 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:27:37.214 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:27:36 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 14390 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:27:37.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:27:36 smithi047 ceph-mon[77147]: pgmap v7546: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:27:37.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:27:36 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:27:37.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:27:36 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 14390 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:27:37.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:27:37 smithi115 conmon[90393]: 2022-05-14T03:27:37.211+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:27:38.230 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:27:37 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:27:38.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:27:37 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:27:38.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:27:38 smithi115 conmon[90393]: 2022-05-14T03:27:38.228+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:27:39.225 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:27:38 smithi115 ceph-mon[78593]: pgmap v7547: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:27:39.226 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:27:38 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:27:39.226 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:27:38 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:27:39.226 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:27:38 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:27:39.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:27:38 smithi047 ceph-mon[77147]: pgmap v7547: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:27:39.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:27:38 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:27:39.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:27:38 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:27:39.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:27:38 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:27:39.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:27:39 smithi115 conmon[90393]: 2022-05-14T03:27:39.222+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:27:40.247 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:27:39 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:27:40.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:27:39 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:27:40.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:27:40 smithi115 conmon[90393]: 2022-05-14T03:27:40.245+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:27:41.238 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:27:40 smithi115 ceph-mon[78593]: pgmap v7548: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:27:41.239 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:27:40 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:27:41.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:27:40 smithi047 ceph-mon[77147]: pgmap v7548: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:27:41.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:27:40 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:27:41.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:27:41 smithi115 conmon[90393]: 2022-05-14T03:27:41.236+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:27:42.208 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:27:41 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:27:42.209 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:27:41 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 14395 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:27:42.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:27:41 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:27:42.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:27:41 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 14395 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:27:42.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:27:42 smithi115 conmon[90393]: 2022-05-14T03:27:42.206+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:27:43.215 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:27:42 smithi115 ceph-mon[78593]: pgmap v7549: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:27:43.215 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:27:42 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:27:43.215 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:27:42 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:27:43.216 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:27:42 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:27:43.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:27:42 smithi047 ceph-mon[77147]: pgmap v7549: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:27:43.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:27:42 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:27:43.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:27:42 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:27:43.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:27:42 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:27:43.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:27:43 smithi115 conmon[90393]: 2022-05-14T03:27:43.212+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:27:44.252 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:27:43 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:27:44.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:27:43 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:27:44.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:27:44 smithi115 conmon[90393]: 2022-05-14T03:27:44.250+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:27:45.247 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:27:44 smithi115 ceph-mon[78593]: pgmap v7550: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:27:45.247 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:27:44 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:27:45.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:27:44 smithi047 ceph-mon[77147]: pgmap v7550: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:27:45.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:27:44 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:27:45.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:27:45 smithi115 conmon[90393]: 2022-05-14T03:27:45.245+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:27:46.211 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:27:45 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:27:46.212 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:27:45 smithi115 ceph-mon[78593]: pgmap v7551: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:27:46.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:27:45 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:27:46.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:27:45 smithi047 ceph-mon[77147]: pgmap v7551: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:27:46.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:27:46 smithi115 conmon[90393]: 2022-05-14T03:27:46.209+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:27:47.243 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:27:46 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:27:47.243 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:27:46 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 14400 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:27:47.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:27:46 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:27:47.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:27:46 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 14400 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:27:47.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:27:47 smithi115 conmon[90393]: 2022-05-14T03:27:47.241+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:27:48.217 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:27:47 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:27:48.217 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:27:47 smithi115 ceph-mon[78593]: pgmap v7552: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:27:48.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:27:47 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:27:48.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:27:47 smithi047 ceph-mon[77147]: pgmap v7552: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:27:48.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:27:48 smithi115 conmon[90393]: 2022-05-14T03:27:48.214+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:27:49.235 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:27:48 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:27:49.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:27:48 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:27:49.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:27:49 smithi115 conmon[90393]: 2022-05-14T03:27:49.232+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:27:50.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:27:49 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:27:50.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:27:49 smithi047 ceph-mon[77147]: pgmap v7553: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:27:50.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:27:50 smithi115 conmon[90393]: 2022-05-14T03:27:50.186+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:27:50.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:27:49 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:27:50.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:27:49 smithi115 ceph-mon[78593]: pgmap v7553: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:27:51.231 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:27:50 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:27:51.232 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:27:50 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:27:51.232 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:27:50 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:27:51.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:27:50 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:27:51.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:27:50 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:27:51.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:27:50 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:27:51.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:27:51 smithi115 conmon[90393]: 2022-05-14T03:27:51.229+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:27:52.220 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:27:51 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:27:52.221 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:27:51 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 14405 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:27:52.221 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:27:51 smithi115 ceph-mon[78593]: pgmap v7554: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:27:52.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:27:51 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:27:52.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:27:51 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 14405 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:27:52.295 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:27:51 smithi047 ceph-mon[77147]: pgmap v7554: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:27:52.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:27:52 smithi115 conmon[90393]: 2022-05-14T03:27:52.218+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:27:53.551 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:27:53 smithi115 conmon[90393]: 2022-05-14T03:27:53.171+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:27:53.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:27:53 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:27:54.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:27:53 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:27:54.556 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:27:54 smithi115 conmon[90393]: 2022-05-14T03:27:54.172+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:27:54.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:27:54 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:27:54.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:27:54 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:27:54.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:27:54 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:27:54.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:27:54 smithi115 ceph-mon[78593]: pgmap v7555: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:27:55.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:27:54 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:27:55.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:27:54 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:27:55.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:27:54 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:27:55.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:27:54 smithi047 ceph-mon[77147]: pgmap v7555: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:27:55.555 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:27:55 smithi115 conmon[90393]: 2022-05-14T03:27:55.215+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:27:55.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:27:55 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:27:56.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:27:55 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:27:56.563 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:27:56 smithi115 conmon[90393]: 2022-05-14T03:27:56.193+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:27:56.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:27:56 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:27:56.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:27:56 smithi115 ceph-mon[78593]: pgmap v7556: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:27:57.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:27:56 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:27:57.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:27:56 smithi047 ceph-mon[77147]: pgmap v7556: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:27:57.563 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:27:57 smithi115 conmon[90393]: 2022-05-14T03:27:57.145+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:27:57.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:27:57 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:27:57.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:27:57 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 14410 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:27:58.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:27:57 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:27:58.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:27:57 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 14410 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:27:58.574 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:27:58 smithi115 conmon[90393]: 2022-05-14T03:27:58.174+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:27:58.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:27:58 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:27:58.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:27:58 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:27:58.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:27:58 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:27:58.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:27:58 smithi115 ceph-mon[78593]: pgmap v7557: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:27:59.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:27:58 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:27:59.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:27:58 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:27:59.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:27:58 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:27:59.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:27:58 smithi047 ceph-mon[77147]: pgmap v7557: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:27:59.573 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:27:59 smithi115 conmon[90393]: 2022-05-14T03:27:59.176+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:27:59.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:27:59 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:28:00.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:27:59 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:28:00.578 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:28:00 smithi115 conmon[90393]: 2022-05-14T03:28:00.153+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:28:00.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:28:00 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:28:00.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:28:00 smithi115 ceph-mon[78593]: pgmap v7558: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:28:01.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:28:00 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:28:01.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:28:00 smithi047 ceph-mon[77147]: pgmap v7558: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:28:01.580 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:28:01 smithi115 conmon[90393]: 2022-05-14T03:28:01.191+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:28:01.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:28:01 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:28:02.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:28:01 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:28:02.587 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:28:02 smithi115 conmon[90393]: 2022-05-14T03:28:02.233+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:28:02.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:28:02 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:28:02.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:28:02 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:28:02.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:28:02 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:28:02.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:28:02 smithi115 ceph-mon[78593]: pgmap v7559: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:28:03.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:28:02 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:28:03.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:28:02 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:28:03.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:28:02 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:28:03.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:28:02 smithi047 ceph-mon[77147]: pgmap v7559: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:28:03.589 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:28:03 smithi115 conmon[90393]: 2022-05-14T03:28:03.259+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:28:03.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:28:03 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:28:04.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:28:03 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:28:04.594 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:28:04 smithi115 conmon[90393]: 2022-05-14T03:28:04.230+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:28:04.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:28:04 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:28:04.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:28:04 smithi115 ceph-mon[78593]: pgmap v7560: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:28:05.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:28:04 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:28:05.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:28:04 smithi047 ceph-mon[77147]: pgmap v7560: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:28:05.596 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:28:05 smithi115 conmon[90393]: 2022-05-14T03:28:05.233+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:28:05.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:28:05 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:28:06.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:28:05 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:28:06.599 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:28:06 smithi115 conmon[90393]: 2022-05-14T03:28:06.272+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:28:06.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:28:06 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:28:06.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:28:06 smithi115 ceph-mon[78593]: pgmap v7561: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:28:06.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:28:06 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 14415 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:28:07.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:28:06 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:28:07.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:28:06 smithi047 ceph-mon[77147]: pgmap v7561: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:28:07.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:28:06 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 14415 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:28:07.601 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:28:07 smithi115 conmon[90393]: 2022-05-14T03:28:07.232+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:28:07.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:28:07 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:28:08.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:28:07 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:28:08.606 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:28:08 smithi115 conmon[90393]: 2022-05-14T03:28:08.185+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:28:08.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:28:08 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:28:08.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:28:08 smithi115 ceph-mon[78593]: pgmap v7562: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:28:09.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:28:08 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:28:09.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:28:08 smithi047 ceph-mon[77147]: pgmap v7562: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:28:09.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:28:09 smithi115 conmon[90393]: 2022-05-14T03:28:09.157+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:28:09.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:28:09 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:28:10.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:28:09 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:28:10.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:28:10 smithi115 conmon[90393]: 2022-05-14T03:28:10.143+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:28:11.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:28:10 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:28:11.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:28:10 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:28:11.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:28:10 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:28:11.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:28:10 smithi047 ceph-mon[77147]: pgmap v7563: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:28:11.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:28:10 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:28:11.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:28:10 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:28:11.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:28:10 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:28:11.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:28:10 smithi115 ceph-mon[78593]: pgmap v7563: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:28:11.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:28:11 smithi115 conmon[90393]: 2022-05-14T03:28:11.115+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:28:12.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:28:11 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:28:12.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:28:11 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 14425 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:28:12.079 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:28:11 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:28:12.080 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:28:11 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 14425 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:28:12.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:28:12 smithi115 conmon[90393]: 2022-05-14T03:28:12.077+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:28:13.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:28:12 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:28:13.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:28:12 smithi047 ceph-mon[77147]: pgmap v7564: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:28:13.084 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:28:12 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:28:13.084 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:28:12 smithi115 ceph-mon[78593]: pgmap v7564: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:28:13.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:28:13 smithi115 conmon[90393]: 2022-05-14T03:28:13.081+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:28:14.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:28:13 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:28:14.102 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:28:13 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:28:14.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:28:14 smithi115 conmon[90393]: 2022-05-14T03:28:14.099+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:28:15.045 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:28:14 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:28:15.045 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:28:14 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:28:15.046 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:28:14 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:28:15.046 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:28:14 smithi047 ceph-mon[77147]: pgmap v7565: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:28:15.046 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:28:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T03:28:15.046 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:28:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T03:28:15.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:28:14 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:28:15.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:28:14 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:28:15.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:28:14 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:28:15.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:28:14 smithi115 ceph-mon[78593]: pgmap v7565: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:28:15.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:28:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T03:28:15.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:28:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T03:28:15.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:28:15 smithi115 conmon[90393]: 2022-05-14T03:28:15.134+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:28:16.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:28:15 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:28:16.098 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:28:15 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:28:16.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:28:16 smithi115 conmon[90393]: 2022-05-14T03:28:16.095+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:28:17.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:28:16 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:28:17.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:28:16 smithi047 ceph-mon[77147]: pgmap v7566: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:28:17.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:28:16 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:28:17.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:28:16 smithi115 ceph-mon[78593]: pgmap v7566: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:28:17.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:28:17 smithi115 conmon[90393]: 2022-05-14T03:28:17.138+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:28:18.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:28:17 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:28:18.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:28:17 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 14429 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:28:18.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:28:17 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:28:18.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:28:17 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 14429 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:28:18.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:28:18 smithi115 conmon[90393]: 2022-05-14T03:28:18.160+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:28:19.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:28:18 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:28:19.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:28:18 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:28:19.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:28:18 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:28:19.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:28:18 smithi047 ceph-mon[77147]: pgmap v7567: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:28:19.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:28:18 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:28:19.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:28:18 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:28:19.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:28:18 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:28:19.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:28:18 smithi115 ceph-mon[78593]: pgmap v7567: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:28:19.412 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:28:19 smithi115 conmon[90393]: 2022-05-14T03:28:19.138+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:28:19.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:28:19 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:28:19.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:28:19 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T03:28:19.795 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:28:19 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T03:28:19.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:28:19 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T03:28:20.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:28:19 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:28:20.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:28:19 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T03:28:20.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:28:19 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T03:28:20.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:28:19 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T03:28:20.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:28:19 smithi047 conmon[78521]: 2022-05-14T03:28:19.887+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:28:20.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:28:19 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T03:28:20.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:28:19 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:28:20.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:28:19 smithi047 conmon[78521]: yield 2022-05-14T03:28:20.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:28:19 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:28:20.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:28:19 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:28:20.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:28:19 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:28:20.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:28:19 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T03:28:20.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:28:19 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:28:20.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:28:19 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T03:28:20.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:28:19 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:28:20.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:28:19 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T03:28:20.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:28:19 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:28:20.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:28:19 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T03:28:20.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:28:19 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:28:20.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:28:19 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T03:28:20.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:28:19 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:28:20.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:28:19 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:28:20.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:28:19 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:28:20.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:28:20 smithi115 conmon[90393]: 2022-05-14T03:28:20.156+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:28:21.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:28:20 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:28:21.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:28:20 smithi047 ceph-mon[77147]: pgmap v7568: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:28:21.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:28:20 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:28:21.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T03:28:21.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:28:21.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T03:28:21.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:28:21.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:28:21.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:28:21.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T03:28:21.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:28:21.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T03:28:21.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:28:21.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T03:28:21.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:28:21.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T03:28:21.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:28:21.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T03:28:21.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:28:21.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:28:21.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:28:21.047 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:28:20 smithi047 ceph-mon[77147]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T03:28:21.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:28:20 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:28:21.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:28:20 smithi115 ceph-mon[78593]: pgmap v7568: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:28:21.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:28:20 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:28:21.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T03:28:21.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:28:21.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T03:28:21.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:28:21.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:28:21.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:28:21.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T03:28:21.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:28:21.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T03:28:21.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:28:21.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T03:28:21.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:28:21.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T03:28:21.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:28:21.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T03:28:21.113 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:28:21.113 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:28:21.113 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:28:21.113 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:28:20 smithi115 ceph-mon[78593]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T03:28:21.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:28:21 smithi115 conmon[90393]: 2022-05-14T03:28:21.117+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:28:22.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:28:21 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:28:22.090 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:28:21 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:28:22.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:28:22 smithi115 conmon[90393]: 2022-05-14T03:28:22.087+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:28:22.962 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:28:22 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:28:22.963 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:28:22 smithi047 ceph-mon[77147]: pgmap v7569: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:28:22.963 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:28:22 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:28:22.963 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:28:22 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:28:23.112 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:28:22 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:28:23.112 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:28:22 smithi115 ceph-mon[78593]: pgmap v7569: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:28:23.113 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:28:22 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:28:23.113 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:28:22 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:28:23.293 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:28:22 smithi047 conmon[78521]: 2022-05-14T03:28:22.959+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:28:23.293 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:28:22 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T03:28:23.293 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:28:22 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:28:23.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:28:22 smithi047 conmon[78521]: yield 2022-05-14T03:28:23.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:28:22 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:28:23.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:28:22 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:28:23.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:28:22 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:28:23.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:28:22 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T03:28:23.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:28:22 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:28:23.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:28:22 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T03:28:23.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:28:22 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:28:23.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:28:22 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T03:28:23.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:28:22 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:28:23.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:28:22 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T03:28:23.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:28:22 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:28:23.297 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:28:22 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T03:28:23.297 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:28:22 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:28:23.297 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:28:22 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:28:23.298 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:28:22 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:28:23.298 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:28:22 smithi047 conmon[78521]: 2022-05-14T03:28:22.960+0000 7f5341f7c700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T03:28:23.298 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:28:22 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T03:28:23.298 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:28:22 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:28:23.299 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:28:22 smithi047 conmon[78521]: yield 2022-05-14T03:28:23.299 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:28:22 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:28:23.299 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:28:22 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:28:23.300 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:28:22 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:28:23.300 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:28:22 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T03:28:23.303 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:28:22 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:28:23.303 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:28:22 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T03:28:23.303 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:28:22 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:28:23.304 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:28:22 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T03:28:23.304 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:28:22 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:28:23.304 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:28:22 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T03:28:23.304 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:28:22 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:28:23.305 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:28:22 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T03:28:23.305 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:28:22 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:28:23.305 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:28:22 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:28:23.305 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:28:22 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:28:23.306 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:28:22 smithi047 conmon[78521]: 2022-05-14T03:28:23.306 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:28:22 smithi047 conmon[78521]: During handling of the above exception, another exception occurred: 2022-05-14T03:28:23.306 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:28:22 smithi047 conmon[78521]: 2022-05-14T03:28:23.306 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:28:22 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T03:28:23.306 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:28:22 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T03:28:23.307 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:28:22 smithi047 conmon[78521]: return f(*arg) 2022-05-14T03:28:23.307 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:28:22 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T03:28:23.307 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:28:22 smithi047 conmon[78521]: self._write_client_files(client_files, host) 2022-05-14T03:28:23.307 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:28:22 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T03:28:23.308 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:28:22 smithi047 conmon[78521]: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T03:28:23.308 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:28:22 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T03:28:23.308 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:28:22 smithi047 conmon[78521]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T03:28:23.308 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:28:22 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T03:28:23.309 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:28:22 smithi047 conmon[78521]: return self.event_loop.get_result(coro) 2022-05-14T03:28:23.309 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:28:22 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T03:28:23.309 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:28:22 smithi047 conmon[78521]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T03:28:23.310 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:28:22 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T03:28:23.310 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:28:22 smithi047 conmon[78521]: return self.__get_result() 2022-05-14T03:28:23.310 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:28:22 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T03:28:23.310 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:28:22 smithi047 conmon[78521]: raise self._exception 2022-05-14T03:28:23.310 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:28:22 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T03:28:23.311 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:28:22 smithi047 conmon[78521]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T03:28:23.311 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:28:22 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T03:28:23.311 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:28:22 smithi047 conmon[78521]: conn = await self._remote_connection(host, addr) 2022-05-14T03:28:23.311 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:28:22 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T03:28:23.311 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:28:22 smithi047 conmon[78521]: raise 2022-05-14T03:28:23.312 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:28:22 smithi047 conmon[78521]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T03:28:23.312 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:28:22 smithi047 conmon[78521]: self.gen.throw(type, value, traceback) 2022-05-14T03:28:23.312 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:28:22 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T03:28:23.312 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:28:22 smithi047 conmon[78521]: raise OrchestratorError(msg) 2022-05-14T03:28:23.313 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:28:22 smithi047 conmon[78521]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:28:23.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:28:23 smithi115 conmon[90393]: 2022-05-14T03:28:23.130+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:28:24.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:28:23 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:28:24.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:28:23 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:28:24.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:28:24 smithi115 conmon[90393]: 2022-05-14T03:28:24.141+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:28:25.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:28:24 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:28:25.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T03:28:25.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:28:25.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T03:28:25.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:28:25.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:28:25.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:28:25.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T03:28:25.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:28:25.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T03:28:25.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:28:25.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T03:28:25.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:28:25.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T03:28:25.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:28:25.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T03:28:25.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:28:25.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:28:25.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:28:25.047 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:28:24 smithi047 ceph-mon[77147]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T03:28:25.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T03:28:25.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:28:25.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T03:28:25.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:28:25.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:28:25.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:28:25.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T03:28:25.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:28:25.049 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T03:28:25.049 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:28:25.049 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T03:28:25.049 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:28:25.049 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T03:28:25.050 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:28:25.050 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T03:28:25.050 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:28:25.050 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:28:25.051 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:28:25.051 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T03:28:25.051 INFO:journalctl@ceph.mon.b.smithi047.stdout: During handling of the above exception, another exception occurred: 2022-05-14T03:28:25.051 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T03:28:25.051 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T03:28:25.052 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T03:28:25.052 INFO:journalctl@ceph.mon.b.smithi047.stdout: return f(*arg) 2022-05-14T03:28:25.052 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T03:28:25.052 INFO:journalctl@ceph.mon.b.smithi047.stdout: self._write_client_files(client_files, host) 2022-05-14T03:28:25.052 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T03:28:25.053 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T03:28:25.053 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T03:28:25.053 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T03:28:25.053 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T03:28:25.054 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.event_loop.get_result(coro) 2022-05-14T03:28:25.054 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T03:28:25.054 INFO:journalctl@ceph.mon.b.smithi047.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T03:28:25.054 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T03:28:25.055 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.__get_result() 2022-05-14T03:28:25.055 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T03:28:25.055 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise self._exception 2022-05-14T03:28:25.055 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T03:28:25.056 INFO:journalctl@ceph.mon.b.smithi047.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T03:28:25.056 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T03:28:25.056 INFO:journalctl@ceph.mon.b.smithi047.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T03:28:25.056 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T03:28:25.057 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise 2022-05-14T03:28:25.057 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T03:28:25.057 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.gen.throw(type, value, traceback) 2022-05-14T03:28:25.057 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T03:28:25.057 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OrchestratorError(msg) 2022-05-14T03:28:25.058 INFO:journalctl@ceph.mon.b.smithi047.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:28:25.058 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:28:24 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:28:25.058 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:28:24 smithi047 ceph-mon[77147]: pgmap v7570: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:28:25.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:28:24 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:28:25.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T03:28:25.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:28:25.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T03:28:25.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:28:25.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:28:25.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:28:25.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T03:28:25.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:28:25.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T03:28:25.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:28:25.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T03:28:25.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:28:25.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T03:28:25.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:28:25.113 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T03:28:25.113 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:28:25.113 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:28:25.113 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:28:25.114 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:28:24 smithi115 ceph-mon[78593]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T03:28:25.114 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T03:28:25.114 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:28:25.114 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T03:28:25.114 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:28:25.115 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:28:25.115 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:28:25.115 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T03:28:25.115 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:28:25.115 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T03:28:25.116 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:28:25.116 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T03:28:25.116 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:28:25.116 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T03:28:25.117 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:28:25.117 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T03:28:25.117 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:28:25.117 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:28:25.117 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:28:25.118 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T03:28:25.124 INFO:journalctl@ceph.mon.c.smithi115.stdout: During handling of the above exception, another exception occurred: 2022-05-14T03:28:25.124 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T03:28:25.124 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T03:28:25.124 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T03:28:25.124 INFO:journalctl@ceph.mon.c.smithi115.stdout: return f(*arg) 2022-05-14T03:28:25.125 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T03:28:25.125 INFO:journalctl@ceph.mon.c.smithi115.stdout: self._write_client_files(client_files, host) 2022-05-14T03:28:25.125 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T03:28:25.125 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T03:28:25.126 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T03:28:25.126 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T03:28:25.126 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T03:28:25.127 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.event_loop.get_result(coro) 2022-05-14T03:28:25.127 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T03:28:25.127 INFO:journalctl@ceph.mon.c.smithi115.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T03:28:25.127 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T03:28:25.128 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.__get_result() 2022-05-14T03:28:25.128 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T03:28:25.128 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise self._exception 2022-05-14T03:28:25.128 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T03:28:25.128 INFO:journalctl@ceph.mon.c.smithi115.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T03:28:25.129 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T03:28:25.129 INFO:journalctl@ceph.mon.c.smithi115.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T03:28:25.130 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T03:28:25.130 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise 2022-05-14T03:28:25.130 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T03:28:25.131 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.gen.throw(type, value, traceback) 2022-05-14T03:28:25.131 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T03:28:25.131 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OrchestratorError(msg) 2022-05-14T03:28:25.131 INFO:journalctl@ceph.mon.c.smithi115.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:28:25.132 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:28:24 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:28:25.132 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:28:24 smithi115 ceph-mon[78593]: pgmap v7570: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:28:25.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:28:25 smithi115 conmon[90393]: 2022-05-14T03:28:25.148+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:28:26.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:28:25 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:28:26.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:28:25 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:28:26.610 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:28:26 smithi115 conmon[90393]: 2022-05-14T03:28:26.190+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:28:27.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:28:26 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:28:27.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:28:26 smithi047 ceph-mon[77147]: pgmap v7571: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:28:27.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:28:26 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 14440 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:28:27.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:28:26 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:28:27.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:28:26 smithi115 ceph-mon[78593]: pgmap v7571: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:28:27.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:28:26 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 14440 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:28:27.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:28:27 smithi115 conmon[90393]: 2022-05-14T03:28:27.233+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:28:28.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:28:27 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:28:28.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:28:27 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:28:28.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:28:28 smithi115 conmon[90393]: 2022-05-14T03:28:28.195+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:28:29.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:28:28 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:28:29.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:28:28 smithi047 ceph-mon[77147]: pgmap v7572: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:28:29.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:28:28 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:28:29.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:28:28 smithi115 ceph-mon[78593]: pgmap v7572: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:28:29.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:28:29 smithi115 conmon[90393]: 2022-05-14T03:28:29.230+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:28:30.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:28:29 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:28:30.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:28:29 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:28:30.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:28:30 smithi115 conmon[90393]: 2022-05-14T03:28:30.267+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:28:31.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:28:30 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:28:31.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:28:30 smithi047 ceph-mon[77147]: pgmap v7573: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:28:31.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:28:30 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:28:31.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:28:30 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:28:31.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:28:30 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:28:31.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:28:30 smithi115 ceph-mon[78593]: pgmap v7573: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:28:31.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:28:30 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:28:31.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:28:30 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:28:31.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:28:31 smithi115 conmon[90393]: 2022-05-14T03:28:31.306+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:28:32.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:28:31 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:28:32.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:28:31 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 14445 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:28:32.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:28:31 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:28:32.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:28:31 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 14445 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:28:32.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:28:32 smithi115 conmon[90393]: 2022-05-14T03:28:32.291+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:28:33.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:28:32 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:28:33.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:28:32 smithi047 ceph-mon[77147]: pgmap v7574: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:28:33.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:28:32 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:28:33.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:28:32 smithi115 ceph-mon[78593]: pgmap v7574: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:28:33.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:28:33 smithi115 conmon[90393]: 2022-05-14T03:28:33.264+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:28:34.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:28:33 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:28:34.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:28:33 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:28:34.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:28:34 smithi115 conmon[90393]: 2022-05-14T03:28:34.278+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:28:35.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:28:34 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:28:35.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:28:34 smithi047 ceph-mon[77147]: pgmap v7575: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:28:35.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:28:34 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:28:35.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:28:34 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:28:35.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:28:34 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:28:35.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:28:34 smithi115 ceph-mon[78593]: pgmap v7575: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:28:35.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:28:34 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:28:35.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:28:34 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:28:35.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:28:35 smithi115 conmon[90393]: 2022-05-14T03:28:35.242+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:28:36.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:28:35 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:28:36.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:28:35 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:28:36.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:28:36 smithi115 conmon[90393]: 2022-05-14T03:28:36.224+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:28:37.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:28:36 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:28:37.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:28:36 smithi047 ceph-mon[77147]: pgmap v7576: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:28:37.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:28:36 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 14450 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:28:37.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:28:36 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:28:37.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:28:36 smithi115 ceph-mon[78593]: pgmap v7576: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:28:37.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:28:36 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 14450 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:28:37.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:28:37 smithi115 conmon[90393]: 2022-05-14T03:28:37.269+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:28:38.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:28:37 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:28:38.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:28:37 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:28:38.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:28:38 smithi115 conmon[90393]: 2022-05-14T03:28:38.249+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:28:39.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:28:38 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:28:39.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:28:38 smithi047 ceph-mon[77147]: pgmap v7577: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:28:39.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:28:38 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:28:39.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:28:38 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:28:39.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:28:38 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:28:39.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:28:38 smithi115 ceph-mon[78593]: pgmap v7577: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:28:39.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:28:38 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:28:39.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:28:38 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:28:39.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:28:39 smithi115 conmon[90393]: 2022-05-14T03:28:39.252+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:28:40.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:28:39 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:28:40.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:28:39 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:28:40.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:28:40 smithi115 conmon[90393]: 2022-05-14T03:28:40.290+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:28:41.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:28:40 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:28:41.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:28:40 smithi047 ceph-mon[77147]: pgmap v7578: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:28:41.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:28:40 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:28:41.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:28:40 smithi115 ceph-mon[78593]: pgmap v7578: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:28:41.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:28:41 smithi115 conmon[90393]: 2022-05-14T03:28:41.278+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:28:42.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:28:41 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:28:42.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:28:41 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 14455 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:28:42.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:28:41 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:28:42.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:28:41 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 14455 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:28:42.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:28:42 smithi115 conmon[90393]: 2022-05-14T03:28:42.255+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:28:43.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:28:42 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:28:43.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:28:42 smithi047 ceph-mon[77147]: pgmap v7579: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:28:43.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:28:42 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:28:43.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:28:42 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:28:43.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:28:42 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:28:43.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:28:42 smithi115 ceph-mon[78593]: pgmap v7579: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:28:43.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:28:42 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:28:43.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:28:42 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:28:43.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:28:43 smithi115 conmon[90393]: 2022-05-14T03:28:43.272+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:28:44.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:28:43 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:28:44.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:28:43 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:28:44.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:28:44 smithi115 conmon[90393]: 2022-05-14T03:28:44.294+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:28:45.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:28:44 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:28:45.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:28:44 smithi047 ceph-mon[77147]: pgmap v7580: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:28:45.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:28:44 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:28:45.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:28:44 smithi115 ceph-mon[78593]: pgmap v7580: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:28:45.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:28:45 smithi115 conmon[90393]: 2022-05-14T03:28:45.342+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:28:46.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:28:45 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:28:46.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:28:45 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:28:46.741 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:28:46 smithi115 conmon[90393]: 2022-05-14T03:28:46.386+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:28:47.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:28:46 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:28:47.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:28:46 smithi047 ceph-mon[77147]: pgmap v7581: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:28:47.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:28:46 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 14460 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:28:47.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:28:46 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:28:47.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:28:46 smithi115 ceph-mon[78593]: pgmap v7581: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:28:47.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:28:46 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 14460 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:28:47.744 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:28:47 smithi115 conmon[90393]: 2022-05-14T03:28:47.409+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:28:48.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:28:47 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:28:48.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:28:47 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:28:48.745 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:28:48 smithi115 conmon[90393]: 2022-05-14T03:28:48.454+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:28:49.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:28:48 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:28:49.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:28:48 smithi047 ceph-mon[77147]: pgmap v7582: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:28:49.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:28:48 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:28:49.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:28:48 smithi115 ceph-mon[78593]: pgmap v7582: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:28:49.751 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:28:49 smithi115 conmon[90393]: 2022-05-14T03:28:49.489+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:28:50.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:28:49 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:28:50.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:28:49 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:28:50.752 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:28:50 smithi115 conmon[90393]: 2022-05-14T03:28:50.459+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:28:51.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:28:50 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:28:51.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:28:50 smithi047 ceph-mon[77147]: pgmap v7583: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:28:51.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:28:50 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:28:51.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:28:50 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:28:51.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:28:50 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:28:51.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:28:50 smithi115 ceph-mon[78593]: pgmap v7583: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:28:51.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:28:50 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:28:51.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:28:50 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:28:51.754 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:28:51 smithi115 conmon[90393]: 2022-05-14T03:28:51.437+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:28:52.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:28:51 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:28:52.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:28:51 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 14465 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:28:52.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:28:51 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:28:52.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:28:51 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 14465 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:28:52.755 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:28:52 smithi115 conmon[90393]: 2022-05-14T03:28:52.416+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:28:53.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:28:52 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:28:53.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:28:52 smithi047 ceph-mon[77147]: pgmap v7584: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:28:53.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:28:52 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:28:53.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:28:52 smithi115 ceph-mon[78593]: pgmap v7584: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:28:53.760 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:28:53 smithi115 conmon[90393]: 2022-05-14T03:28:53.410+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:28:54.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:28:53 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:28:54.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:28:53 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:28:54.761 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:28:54 smithi115 conmon[90393]: 2022-05-14T03:28:54.424+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:28:55.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:28:54 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:28:55.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:28:54 smithi047 ceph-mon[77147]: pgmap v7585: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:28:55.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:28:54 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:28:55.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:28:54 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:28:55.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:28:54 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:28:55.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:28:54 smithi115 ceph-mon[78593]: pgmap v7585: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:28:55.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:28:54 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:28:55.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:28:54 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:28:55.768 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:28:55 smithi115 conmon[90393]: 2022-05-14T03:28:55.395+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:28:56.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:28:55 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:28:56.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:28:55 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:28:56.769 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:28:56 smithi115 conmon[90393]: 2022-05-14T03:28:56.412+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:28:57.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:28:56 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:28:57.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:28:56 smithi047 ceph-mon[77147]: pgmap v7586: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:28:57.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:28:56 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 14470 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:28:57.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:28:56 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:28:57.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:28:56 smithi115 ceph-mon[78593]: pgmap v7586: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:28:57.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:28:56 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 14470 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:28:57.773 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:28:57 smithi115 conmon[90393]: 2022-05-14T03:28:57.461+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:28:58.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:28:57 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:28:58.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:28:57 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:28:58.775 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:28:58 smithi115 conmon[90393]: 2022-05-14T03:28:58.510+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:28:59.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:28:58 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:28:59.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:28:58 smithi047 ceph-mon[77147]: pgmap v7587: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:28:59.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:28:58 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:28:59.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:28:58 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:28:59.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:28:58 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:28:59.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:28:58 smithi115 ceph-mon[78593]: pgmap v7587: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:28:59.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:28:58 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:28:59.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:28:58 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:28:59.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:28:59 smithi115 conmon[90393]: 2022-05-14T03:28:59.540+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:28:59.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:28:59 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:29:00.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:28:59 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:29:00.784 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:29:00 smithi115 conmon[90393]: 2022-05-14T03:29:00.523+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:29:01.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:29:00 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:29:01.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:29:00 smithi047 ceph-mon[77147]: pgmap v7588: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:29:01.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:29:00 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:29:01.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:29:00 smithi115 ceph-mon[78593]: pgmap v7588: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:29:01.787 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:29:01 smithi115 conmon[90393]: 2022-05-14T03:29:01.532+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:29:02.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:29:01 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:29:02.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:29:01 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 14475 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:29:02.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:29:01 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:29:02.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:29:01 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 14475 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:29:02.789 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:29:02 smithi115 conmon[90393]: 2022-05-14T03:29:02.533+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:29:03.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:29:02 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:29:03.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:29:02 smithi047 ceph-mon[77147]: pgmap v7589: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:29:03.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:29:02 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:29:03.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:29:02 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:29:03.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:29:02 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:29:03.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:29:02 smithi115 ceph-mon[78593]: pgmap v7589: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:29:03.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:29:02 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:29:03.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:29:02 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:29:03.794 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:29:03 smithi115 conmon[90393]: 2022-05-14T03:29:03.503+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:29:04.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:29:03 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:29:04.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:29:03 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:29:04.796 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:29:04 smithi115 conmon[90393]: 2022-05-14T03:29:04.535+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:29:05.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:29:04 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:29:05.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:29:04 smithi115 ceph-mon[78593]: pgmap v7590: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:29:05.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:29:04 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:29:05.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:29:04 smithi047 ceph-mon[77147]: pgmap v7590: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:29:05.800 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:29:05 smithi115 conmon[90393]: 2022-05-14T03:29:05.511+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:29:06.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:29:05 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:29:06.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:29:05 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:29:06.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:29:05 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:29:06.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:29:05 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:29:06.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:29:06 smithi115 conmon[90393]: 2022-05-14T03:29:06.559+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:29:06.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:29:06 smithi115 ceph-mon[78593]: pgmap v7591: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:29:06.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:29:06 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 14480 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:29:06.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:29:06 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:29:07.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:29:06 smithi047 ceph-mon[77147]: pgmap v7591: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:29:07.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:29:06 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 14480 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:29:07.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:29:06 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:29:07.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:29:07 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:29:07.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:29:07 smithi115 conmon[90393]: 2022-05-14T03:29:07.599+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:29:08.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:29:07 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:29:09.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:29:08 smithi115 conmon[90393]: 2022-05-14T03:29:08.621+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:29:09.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:29:08 smithi115 ceph-mon[78593]: pgmap v7592: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:29:09.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:29:08 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:29:09.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:29:08 smithi047 ceph-mon[77147]: pgmap v7592: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:29:09.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:29:08 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:29:10.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:29:09 smithi115 conmon[90393]: 2022-05-14T03:29:09.668+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:29:10.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:29:09 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:29:10.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:29:09 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:29:11.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:29:10 smithi115 conmon[90393]: 2022-05-14T03:29:10.690+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:29:11.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:29:10 smithi115 ceph-mon[78593]: pgmap v7593: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:29:11.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:29:10 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:29:11.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:29:10 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:29:11.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:29:10 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:29:11.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:29:10 smithi047 ceph-mon[77147]: pgmap v7593: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:29:11.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:29:10 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:29:11.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:29:10 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:29:11.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:29:10 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:29:12.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:29:11 smithi115 conmon[90393]: 2022-05-14T03:29:11.731+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:29:12.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:29:11 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 14485 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:29:12.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:29:11 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:29:12.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:29:11 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 14485 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:29:12.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:29:11 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:29:13.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:29:12 smithi115 conmon[90393]: 2022-05-14T03:29:12.682+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:29:13.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:29:12 smithi115 ceph-mon[78593]: pgmap v7594: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:29:13.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:29:12 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:29:13.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:29:12 smithi047 ceph-mon[77147]: pgmap v7594: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:29:13.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:29:12 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:29:14.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:29:13 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:29:14.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:29:13 smithi115 conmon[90393]: 2022-05-14T03:29:13.666+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:29:14.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:29:13 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:29:15.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:29:14 smithi115 conmon[90393]: 2022-05-14T03:29:14.625+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:29:15.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:29:14 smithi115 ceph-mon[78593]: pgmap v7595: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:29:15.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:29:14 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:29:15.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:29:14 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:29:15.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:29:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T03:29:15.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:29:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T03:29:15.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:29:14 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:29:15.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:29:14 smithi047 ceph-mon[77147]: pgmap v7595: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:29:15.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:29:14 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:29:15.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:29:14 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:29:15.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:29:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T03:29:15.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:29:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T03:29:15.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:29:14 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:29:15.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:29:15 smithi115 conmon[90393]: 2022-05-14T03:29:15.598+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:29:15.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:29:15 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:29:16.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:29:15 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:29:16.830 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:29:16 smithi115 conmon[90393]: 2022-05-14T03:29:16.574+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:29:17.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:29:16 smithi115 ceph-mon[78593]: pgmap v7596: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:29:17.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:29:16 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 14490 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:29:17.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:29:16 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:29:17.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:29:16 smithi047 ceph-mon[77147]: pgmap v7596: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:29:17.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:29:16 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 14490 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:29:17.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:29:16 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:29:17.835 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:29:17 smithi115 conmon[90393]: 2022-05-14T03:29:17.562+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:29:18.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:29:17 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:29:18.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:29:17 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:29:18.837 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:29:18 smithi115 conmon[90393]: 2022-05-14T03:29:18.573+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:29:19.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:29:18 smithi115 ceph-mon[78593]: pgmap v7597: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:29:19.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:29:18 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:29:19.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:29:18 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:29:19.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:29:18 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:29:19.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:29:18 smithi047 ceph-mon[77147]: pgmap v7597: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:29:19.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:29:18 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:29:19.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:29:18 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:29:19.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:29:18 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:29:19.842 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:29:19 smithi115 conmon[90393]: 2022-05-14T03:29:19.526+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:29:20.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:29:19 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:29:20.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:29:19 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:29:20.843 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:29:20 smithi115 conmon[90393]: 2022-05-14T03:29:20.506+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:29:21.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:29:20 smithi115 ceph-mon[78593]: pgmap v7598: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:29:21.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:29:20 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:29:21.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:29:20 smithi047 ceph-mon[77147]: pgmap v7598: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:29:21.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:29:20 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:29:21.848 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:29:21 smithi115 conmon[90393]: 2022-05-14T03:29:21.513+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:29:22.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:29:21 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:29:22.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:29:21 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 14495 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:29:22.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:29:21 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:29:22.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:29:21 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 14495 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:29:22.852 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:29:22 smithi115 conmon[90393]: 2022-05-14T03:29:22.530+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:29:23.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:29:22 smithi115 ceph-mon[78593]: pgmap v7599: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:29:23.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:29:22 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:29:23.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:29:22 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:29:23.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:29:22 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:29:23.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:29:22 smithi047 ceph-mon[77147]: pgmap v7599: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:29:23.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:29:22 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:29:23.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:29:22 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:29:23.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:29:22 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:29:23.854 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:29:23 smithi115 conmon[90393]: 2022-05-14T03:29:23.488+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:29:24.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:29:23 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T03:29:24.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:29:23 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T03:29:24.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:29:23 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T03:29:24.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:29:23 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:29:24.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:29:23 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T03:29:24.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:29:23 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T03:29:24.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:29:23 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T03:29:24.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:29:23 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:29:24.856 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:29:24 smithi115 conmon[90393]: 2022-05-14T03:29:24.512+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:29:25.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:29:24 smithi115 ceph-mon[78593]: pgmap v7600: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:29:25.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:29:24 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:29:25.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:29:24 smithi047 ceph-mon[77147]: pgmap v7600: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:29:25.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:29:24 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:29:25.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:29:25 smithi115 conmon[90393]: 2022-05-14T03:29:25.490+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:29:26.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:29:25 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:29:26.293 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:29:26 smithi047 conmon[78521]: 2022-05-14T03:29:26.064+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:29:26.293 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:29:26 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T03:29:26.293 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:29:26 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:29:26.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:29:26 smithi047 conmon[78521]: yield 2022-05-14T03:29:26.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:29:26 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:29:26.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:29:26 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:29:26.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:29:26 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:29:26.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:29:26 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T03:29:26.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:29:26 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:29:26.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:29:26 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T03:29:26.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:29:26 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:29:26.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:29:26 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T03:29:26.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:29:26 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:29:26.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:29:26 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T03:29:26.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:29:26 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:29:26.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:29:26 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T03:29:26.297 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:29:26 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:29:26.297 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:29:26 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:29:26.297 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:29:26 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:29:26.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:29:25 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:29:26.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:29:26 smithi115 conmon[90393]: 2022-05-14T03:29:26.455+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:29:27.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:29:26 smithi047 ceph-mon[77147]: pgmap v7601: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:29:27.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:29:26 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:29:27.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:29:26 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 14500 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:29:27.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:29:26 smithi115 ceph-mon[78593]: pgmap v7601: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:29:27.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:29:26 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:29:27.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:29:26 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 14500 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:29:27.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:29:27 smithi115 conmon[90393]: 2022-05-14T03:29:27.463+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:29:28.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:29:27 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:29:28.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T03:29:28.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:29:28.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T03:29:28.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:29:28.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:29:28.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:29:28.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T03:29:28.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:29:28.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T03:29:28.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:29:28.298 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T03:29:28.298 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:29:28.298 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T03:29:28.299 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:29:28.299 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T03:29:28.299 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:29:28.299 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:29:28.300 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:29:28.300 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:29:27 smithi047 ceph-mon[77147]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T03:29:28.300 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:29:27 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:29:28.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:29:27 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:29:28.359 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T03:29:28.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:29:28.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T03:29:28.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:29:28.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:29:28.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:29:28.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T03:29:28.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:29:28.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T03:29:28.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:29:28.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T03:29:28.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:29:28.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T03:29:28.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:29:28.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T03:29:28.365 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:29:28.365 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:29:28.365 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:29:28.366 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:29:27 smithi115 ceph-mon[78593]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T03:29:28.366 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:29:27 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:29:28.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:29:28 smithi115 conmon[90393]: 2022-05-14T03:29:28.490+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:29:29.138 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:29:28 smithi047 ceph-mon[77147]: pgmap v7602: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:29:29.139 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:29:28 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:29:29.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:29:28 smithi115 ceph-mon[78593]: pgmap v7602: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:29:29.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:29:28 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:29:29.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:29:29 smithi047 conmon[78521]: 2022-05-14T03:29:29.136+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:29:29.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:29:29 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T03:29:29.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:29:29 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:29:29.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:29:29 smithi047 conmon[78521]: yield 2022-05-14T03:29:29.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:29:29 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:29:29.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:29:29 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:29:29.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:29:29 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:29:29.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:29:29 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T03:29:29.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:29:29 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:29:29.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:29:29 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T03:29:29.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:29:29 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:29:29.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:29:29 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T03:29:29.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:29:29 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:29:29.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:29:29 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T03:29:29.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:29:29 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:29:29.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:29:29 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T03:29:29.547 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:29:29 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:29:29.547 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:29:29 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:29:29.547 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:29:29 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:29:29.547 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:29:29 smithi047 conmon[78521]: 2022-05-14T03:29:29.137+0000 7f5344f82700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T03:29:29.548 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:29:29 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T03:29:29.548 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:29:29 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:29:29.548 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:29:29 smithi047 conmon[78521]: yield 2022-05-14T03:29:29.548 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:29:29 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:29:29.549 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:29:29 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:29:29.549 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:29:29 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:29:29.549 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:29:29 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T03:29:29.549 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:29:29 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:29:29.550 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:29:29 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T03:29:29.550 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:29:29 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:29:29.550 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:29:29 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T03:29:29.550 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:29:29 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:29:29.551 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:29:29 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T03:29:29.551 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:29:29 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:29:29.551 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:29:29 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T03:29:29.551 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:29:29 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:29:29.552 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:29:29 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:29:29.552 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:29:29 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:29:29.552 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:29:29 smithi047 conmon[78521]: 2022-05-14T03:29:29.552 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:29:29 smithi047 conmon[78521]: During handling of the above exception, another exception occurred: 2022-05-14T03:29:29.553 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:29:29 smithi047 conmon[78521]: 2022-05-14T03:29:29.553 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:29:29 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T03:29:29.553 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:29:29 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T03:29:29.553 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:29:29 smithi047 conmon[78521]: return f(*arg) 2022-05-14T03:29:29.554 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:29:29 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T03:29:29.554 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:29:29 smithi047 conmon[78521]: self._write_client_files(client_files, host) 2022-05-14T03:29:29.554 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:29:29 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T03:29:29.554 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:29:29 smithi047 conmon[78521]: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T03:29:29.555 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:29:29 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T03:29:29.555 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:29:29 smithi047 conmon[78521]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T03:29:29.555 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:29:29 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T03:29:29.555 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:29:29 smithi047 conmon[78521]: return self.event_loop.get_result(coro) 2022-05-14T03:29:29.556 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:29:29 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T03:29:29.556 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:29:29 smithi047 conmon[78521]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T03:29:29.556 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:29:29 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T03:29:29.556 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:29:29 smithi047 conmon[78521]: return self.__get_result() 2022-05-14T03:29:29.557 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:29:29 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T03:29:29.557 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:29:29 smithi047 conmon[78521]: raise self._exception 2022-05-14T03:29:29.557 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:29:29 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T03:29:29.557 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:29:29 smithi047 conmon[78521]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T03:29:29.558 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:29:29 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T03:29:29.558 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:29:29 smithi047 conmon[78521]: conn = await self._remote_connection(host, addr) 2022-05-14T03:29:29.558 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:29:29 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T03:29:29.558 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:29:29 smithi047 conmon[78521]: raise 2022-05-14T03:29:29.558 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:29:29 smithi047 conmon[78521]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T03:29:29.559 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:29:29 smithi047 conmon[78521]: self.gen.throw(type, value, traceback) 2022-05-14T03:29:29.559 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:29:29 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T03:29:29.559 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:29:29 smithi047 conmon[78521]: raise OrchestratorError(msg) 2022-05-14T03:29:29.559 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:29:29 smithi047 conmon[78521]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:29:29.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:29:29 smithi115 conmon[90393]: 2022-05-14T03:29:29.497+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:29:30.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:29:29 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:29:30.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:29:29 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:29:30.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:29:30 smithi115 conmon[90393]: 2022-05-14T03:29:30.498+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:29:31.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:29:30 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:29:31.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T03:29:31.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:29:31.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T03:29:31.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:29:31.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:29:31.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:29:31.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T03:29:31.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:29:31.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T03:29:31.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:29:31.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T03:29:31.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:29:31.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T03:29:31.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:29:31.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T03:29:31.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:29:31.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:29:31.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:29:31.298 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:29:30 smithi047 ceph-mon[77147]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T03:29:31.298 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T03:29:31.299 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:29:31.299 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T03:29:31.299 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:29:31.300 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:29:31.300 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:29:31.300 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T03:29:31.300 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:29:31.301 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T03:29:31.301 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:29:31.301 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T03:29:31.302 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:29:31.302 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T03:29:31.302 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:29:31.302 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T03:29:31.302 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:29:31.303 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:29:31.303 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:29:31.303 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T03:29:31.303 INFO:journalctl@ceph.mon.b.smithi047.stdout: During handling of the above exception, another exception occurred: 2022-05-14T03:29:31.304 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T03:29:31.304 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T03:29:31.304 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T03:29:31.304 INFO:journalctl@ceph.mon.b.smithi047.stdout: return f(*arg) 2022-05-14T03:29:31.304 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T03:29:31.305 INFO:journalctl@ceph.mon.b.smithi047.stdout: self._write_client_files(client_files, host) 2022-05-14T03:29:31.305 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T03:29:31.305 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T03:29:31.305 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T03:29:31.305 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T03:29:31.306 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T03:29:31.306 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.event_loop.get_result(coro) 2022-05-14T03:29:31.306 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T03:29:31.306 INFO:journalctl@ceph.mon.b.smithi047.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T03:29:31.307 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T03:29:31.307 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.__get_result() 2022-05-14T03:29:31.307 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T03:29:31.307 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise self._exception 2022-05-14T03:29:31.308 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T03:29:31.308 INFO:journalctl@ceph.mon.b.smithi047.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T03:29:31.308 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T03:29:31.308 INFO:journalctl@ceph.mon.b.smithi047.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T03:29:31.309 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T03:29:31.309 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise 2022-05-14T03:29:31.309 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T03:29:31.309 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.gen.throw(type, value, traceback) 2022-05-14T03:29:31.309 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T03:29:31.310 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OrchestratorError(msg) 2022-05-14T03:29:31.310 INFO:journalctl@ceph.mon.b.smithi047.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:29:31.310 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:29:30 smithi047 ceph-mon[77147]: pgmap v7603: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:29:31.310 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:29:30 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:29:31.311 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:29:30 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:29:31.311 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:29:30 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:29:31.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:29:30 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:29:31.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T03:29:31.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:29:31.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T03:29:31.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:29:31.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:29:31.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:29:31.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T03:29:31.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:29:31.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T03:29:31.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:29:31.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T03:29:31.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:29:31.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T03:29:31.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:29:31.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T03:29:31.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:29:31.365 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:29:31.365 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:29:31.365 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:29:30 smithi115 ceph-mon[78593]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T03:29:31.365 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T03:29:31.366 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:29:31.366 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T03:29:31.366 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:29:31.366 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:29:31.366 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:29:31.367 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T03:29:31.367 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:29:31.367 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T03:29:31.367 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:29:31.367 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T03:29:31.368 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:29:31.368 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T03:29:31.368 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:29:31.368 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T03:29:31.368 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:29:31.369 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:29:31.369 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:29:31.369 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T03:29:31.369 INFO:journalctl@ceph.mon.c.smithi115.stdout: During handling of the above exception, another exception occurred: 2022-05-14T03:29:31.370 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T03:29:31.370 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T03:29:31.370 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T03:29:31.370 INFO:journalctl@ceph.mon.c.smithi115.stdout: return f(*arg) 2022-05-14T03:29:31.370 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T03:29:31.371 INFO:journalctl@ceph.mon.c.smithi115.stdout: self._write_client_files(client_files, host) 2022-05-14T03:29:31.371 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T03:29:31.371 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T03:29:31.371 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T03:29:31.372 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T03:29:31.372 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T03:29:31.372 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.event_loop.get_result(coro) 2022-05-14T03:29:31.372 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T03:29:31.372 INFO:journalctl@ceph.mon.c.smithi115.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T03:29:31.373 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T03:29:31.373 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.__get_result() 2022-05-14T03:29:31.373 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T03:29:31.373 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise self._exception 2022-05-14T03:29:31.374 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T03:29:31.374 INFO:journalctl@ceph.mon.c.smithi115.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T03:29:31.374 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T03:29:31.374 INFO:journalctl@ceph.mon.c.smithi115.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T03:29:31.374 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T03:29:31.375 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise 2022-05-14T03:29:31.375 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T03:29:31.375 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.gen.throw(type, value, traceback) 2022-05-14T03:29:31.375 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T03:29:31.375 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OrchestratorError(msg) 2022-05-14T03:29:31.376 INFO:journalctl@ceph.mon.c.smithi115.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:29:31.376 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:29:30 smithi115 ceph-mon[78593]: pgmap v7603: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:29:31.376 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:29:30 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:29:31.376 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:29:30 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:29:31.377 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:29:30 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:29:31.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:29:31 smithi115 conmon[90393]: 2022-05-14T03:29:31.462+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:29:32.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:29:31 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:29:32.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:29:31 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 14505 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:29:32.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:29:31 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:29:32.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:29:31 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 14505 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:29:32.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:29:32 smithi115 conmon[90393]: 2022-05-14T03:29:32.442+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:29:33.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:29:32 smithi047 ceph-mon[77147]: pgmap v7604: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:29:33.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:29:32 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:29:33.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:29:32 smithi115 ceph-mon[78593]: pgmap v7604: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:29:33.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:29:32 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:29:33.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:29:33 smithi115 conmon[90393]: 2022-05-14T03:29:33.469+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:29:34.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:29:33 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:29:34.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:29:33 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:29:34.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:29:34 smithi115 conmon[90393]: 2022-05-14T03:29:34.467+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:29:35.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:29:34 smithi047 ceph-mon[77147]: pgmap v7605: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:29:35.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:29:34 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:29:35.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:29:34 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:29:35.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:29:34 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:29:35.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:29:34 smithi115 ceph-mon[78593]: pgmap v7605: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:29:35.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:29:34 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:29:35.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:29:34 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:29:35.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:29:34 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:29:35.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:29:35 smithi115 conmon[90393]: 2022-05-14T03:29:35.487+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:29:36.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:29:35 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:29:36.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:29:35 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:29:36.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:29:36 smithi115 conmon[90393]: 2022-05-14T03:29:36.467+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:29:37.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:29:36 smithi047 ceph-mon[77147]: pgmap v7606: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:29:37.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:29:36 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:29:37.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:29:36 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 14510 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:29:37.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:29:36 smithi115 ceph-mon[78593]: pgmap v7606: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:29:37.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:29:36 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:29:37.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:29:36 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 14510 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:29:37.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:29:37 smithi115 conmon[90393]: 2022-05-14T03:29:37.442+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:29:38.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:29:37 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:29:38.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:29:37 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:29:38.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:29:38 smithi115 conmon[90393]: 2022-05-14T03:29:38.441+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:29:39.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:29:38 smithi047 ceph-mon[77147]: pgmap v7607: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:29:39.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:29:38 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:29:39.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:29:38 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:29:39.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:29:38 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:29:39.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:29:38 smithi115 ceph-mon[78593]: pgmap v7607: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:29:39.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:29:38 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:29:39.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:29:38 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:29:39.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:29:38 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:29:39.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:29:39 smithi115 conmon[90393]: 2022-05-14T03:29:39.445+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:29:40.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:29:39 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:29:40.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:29:39 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:29:40.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:29:40 smithi115 conmon[90393]: 2022-05-14T03:29:40.489+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:29:41.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:29:40 smithi047 ceph-mon[77147]: pgmap v7608: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:29:41.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:29:40 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:29:41.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:29:40 smithi115 ceph-mon[78593]: pgmap v7608: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:29:41.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:29:40 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:29:41.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:29:41 smithi115 conmon[90393]: 2022-05-14T03:29:41.512+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:29:42.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:29:41 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:29:42.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:29:41 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 14515 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:29:42.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:29:41 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:29:42.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:29:41 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 14515 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:29:42.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:29:42 smithi115 conmon[90393]: 2022-05-14T03:29:42.554+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:29:43.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:29:42 smithi047 ceph-mon[77147]: pgmap v7609: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:29:43.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:29:42 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:29:43.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:29:42 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:29:43.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:29:42 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:29:43.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:29:42 smithi115 ceph-mon[78593]: pgmap v7609: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:29:43.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:29:42 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:29:43.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:29:42 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:29:43.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:29:42 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:29:43.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:29:43 smithi115 conmon[90393]: 2022-05-14T03:29:43.577+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:29:44.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:29:43 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:29:44.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:29:43 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:29:44.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:29:44 smithi115 conmon[90393]: 2022-05-14T03:29:44.557+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:29:45.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:29:44 smithi047 ceph-mon[77147]: pgmap v7610: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:29:45.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:29:44 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:29:45.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:29:44 smithi115 ceph-mon[78593]: pgmap v7610: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:29:45.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:29:44 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:29:45.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:29:45 smithi115 conmon[90393]: 2022-05-14T03:29:45.526+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:29:46.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:29:45 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:29:46.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:29:45 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:29:46.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:29:46 smithi115 conmon[90393]: 2022-05-14T03:29:46.498+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:29:47.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:29:46 smithi047 ceph-mon[77147]: pgmap v7611: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:29:47.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:29:46 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:29:47.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:29:46 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 14520 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:29:47.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:29:46 smithi115 ceph-mon[78593]: pgmap v7611: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:29:47.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:29:46 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:29:47.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:29:46 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 14520 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:29:47.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:29:47 smithi115 conmon[90393]: 2022-05-14T03:29:47.474+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:29:48.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:29:47 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:29:48.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:29:47 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:29:48.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:29:48 smithi115 conmon[90393]: 2022-05-14T03:29:48.435+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:29:49.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:29:48 smithi047 ceph-mon[77147]: pgmap v7612: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:29:49.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:29:48 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:29:49.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:29:48 smithi115 ceph-mon[78593]: pgmap v7612: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:29:49.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:29:48 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:29:49.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:29:49 smithi115 conmon[90393]: 2022-05-14T03:29:49.387+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:29:50.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:29:49 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:29:50.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:29:49 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:29:50.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:29:50 smithi115 conmon[90393]: 2022-05-14T03:29:50.406+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:29:51.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:29:50 smithi047 ceph-mon[77147]: pgmap v7613: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:29:51.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:29:50 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:29:51.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:29:50 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:29:51.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:29:50 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:29:51.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:29:50 smithi115 ceph-mon[78593]: pgmap v7613: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:29:51.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:29:50 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:29:51.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:29:50 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:29:51.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:29:50 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:29:51.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:29:51 smithi115 conmon[90393]: 2022-05-14T03:29:51.402+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:29:52.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:29:51 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:29:52.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:29:51 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 14525 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:29:52.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:29:51 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:29:52.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:29:51 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 14525 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:29:52.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:29:52 smithi115 conmon[90393]: 2022-05-14T03:29:52.396+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:29:53.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:29:52 smithi047 ceph-mon[77147]: pgmap v7614: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:29:53.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:29:52 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:29:53.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:29:52 smithi115 ceph-mon[78593]: pgmap v7614: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:29:53.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:29:52 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:29:53.864 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:29:53 smithi115 conmon[90393]: 2022-05-14T03:29:53.364+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:29:54.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:29:53 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:29:54.324 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:29:53 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:29:54.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:29:54 smithi115 conmon[90393]: 2022-05-14T03:29:54.321+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:29:55.282 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:29:54 smithi115 ceph-mon[78593]: pgmap v7615: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:29:55.283 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:29:54 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:29:55.283 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:29:54 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:29:55.283 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:29:54 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:29:55.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:29:54 smithi047 ceph-mon[77147]: pgmap v7615: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:29:55.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:29:54 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:29:55.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:29:54 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:29:55.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:29:54 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:29:55.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:29:55 smithi115 conmon[90393]: 2022-05-14T03:29:55.279+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:29:56.285 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:29:55 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:29:56.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:29:55 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:29:56.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:29:56 smithi115 conmon[90393]: 2022-05-14T03:29:56.282+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:29:57.259 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:29:56 smithi115 ceph-mon[78593]: pgmap v7616: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:29:57.259 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:29:56 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:29:57.259 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:29:56 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 14530 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:29:57.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:29:56 smithi047 ceph-mon[77147]: pgmap v7616: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:29:57.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:29:56 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:29:57.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:29:56 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 14530 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:29:57.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:29:57 smithi115 conmon[90393]: 2022-05-14T03:29:57.256+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:29:58.263 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:29:57 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:29:58.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:29:57 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:29:58.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:29:58 smithi115 conmon[90393]: 2022-05-14T03:29:58.260+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:29:59.287 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:29:58 smithi115 ceph-mon[78593]: pgmap v7617: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:29:59.287 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:29:58 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:29:59.287 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:29:58 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:29:59.287 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:29:58 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:29:59.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:29:58 smithi047 ceph-mon[77147]: pgmap v7617: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:29:59.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:29:58 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:29:59.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:29:58 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:29:59.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:29:58 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:29:59.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:29:59 smithi115 conmon[90393]: 2022-05-14T03:29:59.284+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:30:00.275 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:29:59 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:30:00.276 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:29:59 smithi115 ceph-mon[78593]: pgmap v7618: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:30:00.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:29:59 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:30:00.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:29:59 smithi047 ceph-mon[77147]: pgmap v7618: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:30:00.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:30:00 smithi047 conmon[77123]: 2022-05-14T03:29:59.998+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : Health detail: HEALTH_ERR 1 filesystem is degraded; 1 mds daemon damaged; 1 MDSs are read only; 1 MDSs report slow metadata IOs; 1/3 mons down, quorum b,c; Reduced data availability: 1 pg inactive; 9 slow ops, oldest one blocked for 14535 sec, osd.5 has slow ops 2022-05-14T03:30:00.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:30:00 smithi047 conmon[77123]: 2022-05-14T03:29:59.998+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : [WRN] FS_DEGRADED: 1 filesystem is degraded 2022-05-14T03:30:00.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:30:00 smithi047 conmon[77123]: 2022-05-14T03:29:59.998+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : fs cephfs is degraded 2022-05-14T03:30:00.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:30:00 smithi047 conmon[77123]: 2022-05-14T03:30:00.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:30:00 smithi047 conmon[77123]: 2022-05-14T03:29:59.998+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : [ERR] MDS_DAMAGE: 1 mds daemon damaged 2022-05-14T03:30:00.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:30:00 smithi047 conmon[77123]: 2022-05-14T03:29:59.998+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : fs cephfs mds.2 is damaged 2022-05-14T03:30:00.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:30:00 smithi047 conmon[77123]: 2022-05-14T03:29:59.998+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : [WRN] MDS_READ_ONLY: 1 MDSs are read only 2022-05-14T03:30:00.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:30:00 smithi047 conmon[77123]: 2022-05-14T03:29:59.998+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : mds.l(mds.0): MDS in read-only mode 2022-05-14T03:30:00.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:30:00 smithi047 conmon[77123]: 2022-05-14T03:29:59.998+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : [WRN] MDS_SLOW_METADATA_IO: 1 MDSs report slow metadata IOs 2022-05-14T03:30:00.295 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:30:00 smithi047 conmon[77123]: 2022-05-14T03:29:59.998+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : mds.l(mds.0): 4 slow metadata IOs are blocked > 180 secs, oldest blocked for 14833 secs 2022-05-14T03:30:00.295 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:30:00 smithi047 conmon[77123]: 2022-05-14T03:29:59.998+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : [WRN] MON_DOWN: 1/3 mons down, quorum b,c 2022-05-14T03:30:00.295 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:30:00 smithi047 conmon[77123]: 2022-05-14T03:29:59.998+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : mon.a (rank 0) addr [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] is down (out of quorum) 2022-05-14T03:30:00.295 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:30:00 smithi047 conmon[77123]: 2022-05-14T03:29:59.998+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : [WRN] PG_AVAILABILITY: Reduced data availability: 1 pg inactive 2022-05-14T03:30:00.296 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:30:00 smithi047 conmon[77123]: 2022-05-14T03:29:59.998+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : pg 2.1 is stuck inactive for 4h, current state unknown, last acting [] 2022-05-14T03:30:00.296 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:30:00 smithi047 conmon[77123]: 2022-05-14T03:29:59.998+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : [WRN] SLOW_OPS: 9 slow ops, oldest one blocked for 14535 sec, osd.5 has slow ops 2022-05-14T03:30:00.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:30:00 smithi115 conmon[90393]: 2022-05-14T03:30:00.272+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:30:01.276 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:30:00 smithi115 ceph-mon[78593]: Health detail: HEALTH_ERR 1 filesystem is degraded; 1 mds daemon damaged; 1 MDSs are read only; 1 MDSs report slow metadata IOs; 1/3 mons down, quorum b,c; Reduced data availability: 1 pg inactive; 9 slow ops, oldest one blocked for 14535 sec, osd.5 has slow ops 2022-05-14T03:30:01.277 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:30:00 smithi115 ceph-mon[78593]: [WRN] FS_DEGRADED: 1 filesystem is degraded 2022-05-14T03:30:01.277 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:30:00 smithi115 ceph-mon[78593]: fs cephfs is degraded 2022-05-14T03:30:01.277 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:30:00 smithi115 ceph-mon[78593]: [ERR] MDS_DAMAGE: 1 mds daemon damaged 2022-05-14T03:30:01.277 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:30:00 smithi115 ceph-mon[78593]: fs cephfs mds.2 is damaged 2022-05-14T03:30:01.278 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:30:00 smithi115 ceph-mon[78593]: [WRN] MDS_READ_ONLY: 1 MDSs are read only 2022-05-14T03:30:01.278 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:30:00 smithi115 ceph-mon[78593]: mds.l(mds.0): MDS in read-only mode 2022-05-14T03:30:01.278 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:30:00 smithi115 ceph-mon[78593]: [WRN] MDS_SLOW_METADATA_IO: 1 MDSs report slow metadata IOs 2022-05-14T03:30:01.278 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:30:00 smithi115 ceph-mon[78593]: mds.l(mds.0): 4 slow metadata IOs are blocked > 180 secs, oldest blocked for 14833 secs 2022-05-14T03:30:01.279 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:30:00 smithi115 ceph-mon[78593]: [WRN] MON_DOWN: 1/3 mons down, quorum b,c 2022-05-14T03:30:01.279 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:30:00 smithi115 ceph-mon[78593]: mon.a (rank 0) addr [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] is down (out of quorum) 2022-05-14T03:30:01.279 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:30:00 smithi115 ceph-mon[78593]: [WRN] PG_AVAILABILITY: Reduced data availability: 1 pg inactive 2022-05-14T03:30:01.279 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:30:00 smithi115 ceph-mon[78593]: pg 2.1 is stuck inactive for 4h, current state unknown, last acting [] 2022-05-14T03:30:01.279 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:30:00 smithi115 ceph-mon[78593]: [WRN] SLOW_OPS: 9 slow ops, oldest one blocked for 14535 sec, osd.5 has slow ops 2022-05-14T03:30:01.280 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:30:00 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:30:01.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:30:00 smithi047 ceph-mon[77147]: Health detail: HEALTH_ERR 1 filesystem is degraded; 1 mds daemon damaged; 1 MDSs are read only; 1 MDSs report slow metadata IOs; 1/3 mons down, quorum b,c; Reduced data availability: 1 pg inactive; 9 slow ops, oldest one blocked for 14535 sec, osd.5 has slow ops 2022-05-14T03:30:01.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:30:00 smithi047 ceph-mon[77147]: [WRN] FS_DEGRADED: 1 filesystem is degraded 2022-05-14T03:30:01.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:30:00 smithi047 ceph-mon[77147]: fs cephfs is degraded 2022-05-14T03:30:01.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:30:00 smithi047 ceph-mon[77147]: [ERR] MDS_DAMAGE: 1 mds daemon damaged 2022-05-14T03:30:01.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:30:00 smithi047 ceph-mon[77147]: fs cephfs mds.2 is damaged 2022-05-14T03:30:01.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:30:00 smithi047 ceph-mon[77147]: [WRN] MDS_READ_ONLY: 1 MDSs are read only 2022-05-14T03:30:01.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:30:00 smithi047 ceph-mon[77147]: mds.l(mds.0): MDS in read-only mode 2022-05-14T03:30:01.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:30:00 smithi047 ceph-mon[77147]: [WRN] MDS_SLOW_METADATA_IO: 1 MDSs report slow metadata IOs 2022-05-14T03:30:01.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:30:00 smithi047 ceph-mon[77147]: mds.l(mds.0): 4 slow metadata IOs are blocked > 180 secs, oldest blocked for 14833 secs 2022-05-14T03:30:01.295 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:30:00 smithi047 ceph-mon[77147]: [WRN] MON_DOWN: 1/3 mons down, quorum b,c 2022-05-14T03:30:01.295 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:30:00 smithi047 ceph-mon[77147]: mon.a (rank 0) addr [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] is down (out of quorum) 2022-05-14T03:30:01.295 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:30:00 smithi047 ceph-mon[77147]: [WRN] PG_AVAILABILITY: Reduced data availability: 1 pg inactive 2022-05-14T03:30:01.295 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:30:00 smithi047 ceph-mon[77147]: pg 2.1 is stuck inactive for 4h, current state unknown, last acting [] 2022-05-14T03:30:01.296 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:30:00 smithi047 ceph-mon[77147]: [WRN] SLOW_OPS: 9 slow ops, oldest one blocked for 14535 sec, osd.5 has slow ops 2022-05-14T03:30:01.296 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:30:00 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:30:01.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:30:01 smithi115 conmon[90393]: 2022-05-14T03:30:01.274+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:30:02.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:30:01 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:30:02.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:30:01 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 14535 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:30:02.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:30:01 smithi047 ceph-mon[77147]: pgmap v7619: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:30:02.298 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:30:01 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:30:02.299 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:30:01 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 14535 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:30:02.299 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:30:01 smithi115 ceph-mon[78593]: pgmap v7619: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:30:02.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:30:02 smithi115 conmon[90393]: 2022-05-14T03:30:02.296+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:30:03.274 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:30:02 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:30:03.274 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:30:02 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:30:03.275 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:30:02 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:30:03.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:30:02 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:30:03.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:30:02 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:30:03.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:30:02 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:30:03.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:30:03 smithi115 conmon[90393]: 2022-05-14T03:30:03.271+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:30:04.279 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:30:03 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:30:04.280 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:30:03 smithi115 ceph-mon[78593]: pgmap v7620: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:30:04.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:30:03 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:30:04.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:30:03 smithi047 ceph-mon[77147]: pgmap v7620: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:30:04.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:30:04 smithi115 conmon[90393]: 2022-05-14T03:30:04.276+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:30:05.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:30:04 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:30:05.320 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:30:04 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:30:05.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:30:05 smithi115 conmon[90393]: 2022-05-14T03:30:05.318+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:30:06.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:30:05 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:30:06.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:30:05 smithi047 ceph-mon[77147]: pgmap v7621: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:30:06.351 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:30:05 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:30:06.352 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:30:05 smithi115 ceph-mon[78593]: pgmap v7621: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:30:06.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:30:06 smithi115 conmon[90393]: 2022-05-14T03:30:06.349+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:30:07.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:30:06 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:30:07.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:30:06 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 14540 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:30:07.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:30:06 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:30:07.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:30:06 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 14540 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:30:07.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:30:07 smithi115 conmon[90393]: 2022-05-14T03:30:07.393+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:30:08.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:30:07 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:30:08.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:30:07 smithi047 ceph-mon[77147]: pgmap v7622: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:30:08.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:30:07 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:30:08.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:30:07 smithi115 ceph-mon[78593]: pgmap v7622: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:30:08.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:30:08 smithi115 conmon[90393]: 2022-05-14T03:30:08.382+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:30:09.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:30:08 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:30:09.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:30:08 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:30:09.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:30:09 smithi115 conmon[90393]: 2022-05-14T03:30:09.382+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:30:10.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:30:09 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:30:10.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:30:09 smithi047 ceph-mon[77147]: pgmap v7623: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:30:10.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:30:10 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:30:10.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:30:10 smithi115 ceph-mon[78593]: pgmap v7623: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:30:10.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:30:10 smithi115 conmon[90393]: 2022-05-14T03:30:10.425+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:30:11.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:30:10 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:30:11.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:30:10 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:30:11.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:30:11 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:30:11.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:30:11 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:30:11.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:30:11 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:30:11.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:30:11 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:30:11.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:30:11 smithi115 conmon[90393]: 2022-05-14T03:30:11.392+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:30:12.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:30:12 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:30:12.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:30:12 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 14545 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:30:12.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:30:12 smithi047 ceph-mon[77147]: pgmap v7624: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:30:12.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:30:12 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:30:12.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:30:12 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 14545 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:30:12.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:30:12 smithi115 ceph-mon[78593]: pgmap v7624: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:30:12.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:30:12 smithi115 conmon[90393]: 2022-05-14T03:30:12.430+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:30:13.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:30:13 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:30:13.860 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:30:13 smithi115 conmon[90393]: 2022-05-14T03:30:13.447+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:30:14.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:30:13 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:30:14.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:30:14 smithi115 conmon[90393]: 2022-05-14T03:30:14.481+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:30:14.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:30:14 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:30:14.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:30:14 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:30:14.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:30:14 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:30:14.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:30:14 smithi115 ceph-mon[78593]: pgmap v7625: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:30:14.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:30:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T03:30:14.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:30:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T03:30:15.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:30:14 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:30:15.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:30:14 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:30:15.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:30:14 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:30:15.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:30:14 smithi047 ceph-mon[77147]: pgmap v7625: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:30:15.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:30:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T03:30:15.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:30:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T03:30:15.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:30:15 smithi115 conmon[90393]: 2022-05-14T03:30:15.466+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:30:15.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:30:15 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:30:16.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:30:15 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:30:16.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:30:16 smithi115 conmon[90393]: 2022-05-14T03:30:16.500+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:30:16.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:30:16 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:30:16.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:30:16 smithi115 ceph-mon[78593]: pgmap v7626: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:30:17.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:30:16 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:30:17.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:30:16 smithi047 ceph-mon[77147]: pgmap v7626: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:30:17.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:30:17 smithi115 conmon[90393]: 2022-05-14T03:30:17.512+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:30:17.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:30:17 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:30:17.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:30:17 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 14550 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:30:18.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:30:17 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:30:18.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:30:17 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 14550 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:30:18.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:30:18 smithi115 conmon[90393]: 2022-05-14T03:30:18.482+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:30:18.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:30:18 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:30:18.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:30:18 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:30:18.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:30:18 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:30:18.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:30:18 smithi115 ceph-mon[78593]: pgmap v7627: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:30:19.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:30:18 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:30:19.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:30:18 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:30:19.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:30:18 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:30:19.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:30:18 smithi047 ceph-mon[77147]: pgmap v7627: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:30:19.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:30:19 smithi115 conmon[90393]: 2022-05-14T03:30:19.484+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:30:19.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:30:19 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:30:20.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:30:19 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:30:20.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:30:20 smithi115 conmon[90393]: 2022-05-14T03:30:20.504+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:30:20.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:30:20 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:30:20.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:30:20 smithi115 ceph-mon[78593]: pgmap v7628: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:30:21.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:30:20 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:30:21.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:30:20 smithi047 ceph-mon[77147]: pgmap v7628: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:30:21.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:30:21 smithi115 conmon[90393]: 2022-05-14T03:30:21.506+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:30:21.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:30:21 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:30:22.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:30:21 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:30:22.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:30:22 smithi115 conmon[90393]: 2022-05-14T03:30:22.536+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:30:22.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:30:22 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:30:22.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:30:22 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:30:22.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:30:22 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:30:22.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:30:22 smithi115 ceph-mon[78593]: pgmap v7629: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:30:23.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:30:22 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:30:23.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:30:22 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:30:23.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:30:22 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:30:23.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:30:22 smithi047 ceph-mon[77147]: pgmap v7629: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:30:23.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:30:23 smithi115 conmon[90393]: 2022-05-14T03:30:23.579+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:30:23.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:30:23 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:30:24.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:30:23 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:30:24.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:30:24 smithi115 conmon[90393]: 2022-05-14T03:30:24.594+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:30:24.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:30:24 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:30:24.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:30:24 smithi115 ceph-mon[78593]: pgmap v7630: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:30:25.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:30:24 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:30:25.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:30:24 smithi047 ceph-mon[77147]: pgmap v7630: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:30:25.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:30:25 smithi115 conmon[90393]: 2022-05-14T03:30:25.577+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:30:25.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:30:25 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:30:26.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:30:25 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:30:26.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:30:26 smithi115 conmon[90393]: 2022-05-14T03:30:26.567+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:30:26.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:30:26 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:30:26.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:30:26 smithi115 ceph-mon[78593]: pgmap v7631: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:30:26.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:30:26 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 14555 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:30:27.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:30:26 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:30:27.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:30:26 smithi047 ceph-mon[77147]: pgmap v7631: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:30:27.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:30:26 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 14555 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:30:27.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:30:27 smithi115 conmon[90393]: 2022-05-14T03:30:27.564+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:30:27.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:30:27 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:30:28.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:30:27 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:30:28.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:30:28 smithi115 conmon[90393]: 2022-05-14T03:30:28.522+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:30:28.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:30:28 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:30:28.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:30:28 smithi115 ceph-mon[78593]: pgmap v7632: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:30:29.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:30:28 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:30:29.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:30:28 smithi047 ceph-mon[77147]: pgmap v7632: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:30:29.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:30:29 smithi115 conmon[90393]: 2022-05-14T03:30:29.515+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:30:29.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:30:29 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:30:29.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:30:29 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T03:30:29.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:30:29 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T03:30:29.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:30:29 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T03:30:30.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:30:29 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:30:30.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:30:29 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T03:30:30.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:30:29 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T03:30:30.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:30:29 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T03:30:30.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:30:29 smithi047 conmon[78521]: 2022-05-14T03:30:29.615+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:30:30.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:30:29 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T03:30:30.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:30:29 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:30:30.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:30:29 smithi047 conmon[78521]: yield 2022-05-14T03:30:30.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:30:29 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:30:30.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:30:29 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:30:30.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:30:29 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:30:30.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:30:29 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T03:30:30.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:30:29 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:30:30.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:30:29 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T03:30:30.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:30:29 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:30:30.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:30:29 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T03:30:30.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:30:29 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:30:30.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:30:29 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T03:30:30.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:30:29 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:30:30.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:30:29 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T03:30:30.048 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:30:29 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:30:30.048 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:30:29 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:30:30.048 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:30:29 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:30:30.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:30:30 smithi115 conmon[90393]: 2022-05-14T03:30:30.529+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:30:30.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:30:30 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:30:30.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:30:30 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:30:30.860 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T03:30:30.860 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:30:30.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T03:30:30.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:30:30.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:30:30.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:30:30.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T03:30:30.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:30:30.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T03:30:30.863 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:30:30.863 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T03:30:30.863 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:30:30.864 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T03:30:30.864 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:30:30.865 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T03:30:30.865 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:30:30.865 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:30:30.865 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:30:30.866 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:30:30 smithi115 ceph-mon[78593]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T03:30:30.866 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:30:30 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:30:30.866 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:30:30 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:30:30.867 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:30:30 smithi115 ceph-mon[78593]: pgmap v7633: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:30:31.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:30:30 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:30:31.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:30:30 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:30:31.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T03:30:31.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:30:31.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T03:30:31.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:30:31.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:30:31.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:30:31.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T03:30:31.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:30:31.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T03:30:31.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:30:31.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T03:30:31.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:30:31.049 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T03:30:31.049 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:30:31.050 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T03:30:31.050 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:30:31.051 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:30:31.051 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:30:31.051 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:30:30 smithi047 ceph-mon[77147]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T03:30:31.052 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:30:30 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:30:31.052 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:30:30 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:30:31.052 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:30:30 smithi047 ceph-mon[77147]: pgmap v7633: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:30:31.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:30:31 smithi115 conmon[90393]: 2022-05-14T03:30:31.575+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:30:31.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:30:31 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:30:31.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:30:31 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 14565 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:30:32.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:30:31 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:30:32.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:30:31 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 14565 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:30:32.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:30:32 smithi115 conmon[90393]: 2022-05-14T03:30:32.600+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:30:32.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:30:32 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:30:32.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:30:32 smithi115 ceph-mon[78593]: pgmap v7634: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:30:33.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:30:32 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:30:33.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:30:32 smithi047 ceph-mon[77147]: pgmap v7634: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:30:33.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:30:32 smithi047 conmon[78521]: 2022-05-14T03:30:32.687+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:30:33.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:30:32 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T03:30:33.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:30:32 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:30:33.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:30:32 smithi047 conmon[78521]: yield 2022-05-14T03:30:33.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:30:32 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:30:33.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:30:32 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:30:33.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:30:32 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:30:33.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:30:32 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T03:30:33.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:30:32 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:30:33.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:30:32 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T03:30:33.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:30:32 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:30:33.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:30:32 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T03:30:33.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:30:32 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:30:33.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:30:32 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T03:30:33.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:30:32 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:30:33.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:30:32 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T03:30:33.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:30:32 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:30:33.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:30:32 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:30:33.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:30:32 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:30:33.048 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:30:32 smithi047 conmon[78521]: 2022-05-14T03:30:32.688+0000 7f5344781700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T03:30:33.048 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:30:32 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T03:30:33.048 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:30:32 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:30:33.048 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:30:32 smithi047 conmon[78521]: yield 2022-05-14T03:30:33.048 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:30:32 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:30:33.049 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:30:32 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:30:33.049 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:30:32 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:30:33.049 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:30:32 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T03:30:33.049 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:30:32 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:30:33.049 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:30:32 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T03:30:33.050 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:30:32 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:30:33.050 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:30:32 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T03:30:33.050 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:30:32 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:30:33.050 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:30:32 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T03:30:33.050 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:30:32 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:30:33.051 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:30:32 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T03:30:33.051 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:30:32 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:30:33.051 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:30:32 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:30:33.051 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:30:32 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:30:33.051 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:30:32 smithi047 conmon[78521]: 2022-05-14T03:30:33.052 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:30:32 smithi047 conmon[78521]: During handling of the above exception, another exception occurred: 2022-05-14T03:30:33.052 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:30:32 smithi047 conmon[78521]: 2022-05-14T03:30:33.052 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:30:32 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T03:30:33.052 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:30:32 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T03:30:33.053 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:30:32 smithi047 conmon[78521]: return f(*arg) 2022-05-14T03:30:33.053 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:30:32 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T03:30:33.053 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:30:32 smithi047 conmon[78521]: self._write_client_files(client_files, host) 2022-05-14T03:30:33.053 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:30:32 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T03:30:33.053 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:30:32 smithi047 conmon[78521]: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T03:30:33.054 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:30:32 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T03:30:33.054 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:30:32 smithi047 conmon[78521]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T03:30:33.054 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:30:32 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T03:30:33.054 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:30:32 smithi047 conmon[78521]: return self.event_loop.get_result(coro) 2022-05-14T03:30:33.054 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:30:32 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T03:30:33.055 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:30:32 smithi047 conmon[78521]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T03:30:33.055 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:30:32 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T03:30:33.055 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:30:32 smithi047 conmon[78521]: return self.__get_result() 2022-05-14T03:30:33.055 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:30:32 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T03:30:33.055 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:30:32 smithi047 conmon[78521]: raise self._exception 2022-05-14T03:30:33.056 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:30:32 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T03:30:33.056 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:30:32 smithi047 conmon[78521]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T03:30:33.056 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:30:32 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T03:30:33.056 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:30:32 smithi047 conmon[78521]: conn = await self._remote_connection(host, addr) 2022-05-14T03:30:33.056 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:30:32 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T03:30:33.057 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:30:32 smithi047 conmon[78521]: raise 2022-05-14T03:30:33.057 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:30:32 smithi047 conmon[78521]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T03:30:33.057 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:30:32 smithi047 conmon[78521]: self.gen.throw(type, value, traceback) 2022-05-14T03:30:33.057 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:30:32 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T03:30:33.058 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:30:32 smithi047 conmon[78521]: raise OrchestratorError(msg) 2022-05-14T03:30:33.058 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:30:32 smithi047 conmon[78521]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:30:33.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:30:33 smithi115 conmon[90393]: 2022-05-14T03:30:33.572+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:30:33.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:30:33 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:30:33.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:30:33 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:30:33.860 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T03:30:33.860 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:30:33.860 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T03:30:33.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:30:33.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:30:33.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:30:33.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T03:30:33.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:30:33.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T03:30:33.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:30:33.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T03:30:33.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:30:33.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T03:30:33.863 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:30:33.863 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T03:30:33.863 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:30:33.863 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:30:33.864 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:30:33.864 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:30:33 smithi115 ceph-mon[78593]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T03:30:33.864 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T03:30:33.864 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:30:33.864 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T03:30:33.865 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:30:33.865 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:30:33.865 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:30:33.865 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T03:30:33.866 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:30:33.866 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T03:30:33.866 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:30:33.866 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T03:30:33.866 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:30:33.867 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T03:30:33.867 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:30:33.867 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T03:30:33.867 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:30:33.867 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:30:33.868 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:30:33.868 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T03:30:33.868 INFO:journalctl@ceph.mon.c.smithi115.stdout: During handling of the above exception, another exception occurred: 2022-05-14T03:30:33.868 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T03:30:33.868 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T03:30:33.869 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T03:30:33.869 INFO:journalctl@ceph.mon.c.smithi115.stdout: return f(*arg) 2022-05-14T03:30:33.869 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T03:30:33.869 INFO:journalctl@ceph.mon.c.smithi115.stdout: self._write_client_files(client_files, host) 2022-05-14T03:30:33.870 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T03:30:33.870 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T03:30:33.870 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T03:30:33.870 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T03:30:33.870 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T03:30:33.871 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.event_loop.get_result(coro) 2022-05-14T03:30:33.871 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T03:30:33.871 INFO:journalctl@ceph.mon.c.smithi115.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T03:30:33.871 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T03:30:33.872 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.__get_result() 2022-05-14T03:30:33.872 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T03:30:33.872 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise self._exception 2022-05-14T03:30:33.872 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T03:30:33.872 INFO:journalctl@ceph.mon.c.smithi115.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T03:30:33.873 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T03:30:33.873 INFO:journalctl@ceph.mon.c.smithi115.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T03:30:33.873 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T03:30:33.873 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise 2022-05-14T03:30:33.873 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T03:30:33.874 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.gen.throw(type, value, traceback) 2022-05-14T03:30:33.874 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T03:30:33.874 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OrchestratorError(msg) 2022-05-14T03:30:33.874 INFO:journalctl@ceph.mon.c.smithi115.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:30:34.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:30:33 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:30:34.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:30:33 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:30:34.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T03:30:34.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:30:34.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T03:30:34.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:30:34.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:30:34.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:30:34.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T03:30:34.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:30:34.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T03:30:34.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:30:34.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T03:30:34.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:30:34.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T03:30:34.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:30:34.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T03:30:34.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:30:34.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:30:34.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:30:34.047 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:30:33 smithi047 ceph-mon[77147]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T03:30:34.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T03:30:34.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:30:34.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T03:30:34.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:30:34.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:30:34.049 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:30:34.049 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T03:30:34.050 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:30:34.050 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T03:30:34.050 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:30:34.051 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T03:30:34.051 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:30:34.051 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T03:30:34.051 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:30:34.052 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T03:30:34.052 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:30:34.052 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:30:34.052 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:30:34.053 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T03:30:34.053 INFO:journalctl@ceph.mon.b.smithi047.stdout: During handling of the above exception, another exception occurred: 2022-05-14T03:30:34.053 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T03:30:34.054 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T03:30:34.054 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T03:30:34.054 INFO:journalctl@ceph.mon.b.smithi047.stdout: return f(*arg) 2022-05-14T03:30:34.055 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T03:30:34.055 INFO:journalctl@ceph.mon.b.smithi047.stdout: self._write_client_files(client_files, host) 2022-05-14T03:30:34.055 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T03:30:34.056 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T03:30:34.056 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T03:30:34.057 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T03:30:34.057 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T03:30:34.057 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.event_loop.get_result(coro) 2022-05-14T03:30:34.058 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T03:30:34.058 INFO:journalctl@ceph.mon.b.smithi047.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T03:30:34.059 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T03:30:34.059 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.__get_result() 2022-05-14T03:30:34.059 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T03:30:34.060 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise self._exception 2022-05-14T03:30:34.060 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T03:30:34.061 INFO:journalctl@ceph.mon.b.smithi047.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T03:30:34.061 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T03:30:34.062 INFO:journalctl@ceph.mon.b.smithi047.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T03:30:34.062 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T03:30:34.062 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise 2022-05-14T03:30:34.063 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T03:30:34.063 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.gen.throw(type, value, traceback) 2022-05-14T03:30:34.064 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T03:30:34.064 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OrchestratorError(msg) 2022-05-14T03:30:34.064 INFO:journalctl@ceph.mon.b.smithi047.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:30:34.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:30:34 smithi115 conmon[90393]: 2022-05-14T03:30:34.529+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:30:34.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:30:34 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:30:34.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:30:34 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:30:34.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:30:34 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:30:34.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:30:34 smithi115 ceph-mon[78593]: pgmap v7635: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:30:35.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:30:34 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:30:35.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:30:34 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:30:35.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:30:34 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:30:35.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:30:34 smithi047 ceph-mon[77147]: pgmap v7635: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:30:35.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:30:35 smithi115 conmon[90393]: 2022-05-14T03:30:35.556+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:30:35.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:30:35 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:30:36.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:30:35 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:30:36.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:30:36 smithi115 conmon[90393]: 2022-05-14T03:30:36.526+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:30:36.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:30:36 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:30:36.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:30:36 smithi115 ceph-mon[78593]: pgmap v7636: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:30:37.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:30:36 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:30:37.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:30:36 smithi047 ceph-mon[77147]: pgmap v7636: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:30:37.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:30:37 smithi115 conmon[90393]: 2022-05-14T03:30:37.519+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:30:37.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:30:37 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:30:37.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:30:37 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 14570 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:30:38.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:30:37 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:30:38.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:30:37 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 14570 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:30:38.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:30:38 smithi115 conmon[90393]: 2022-05-14T03:30:38.526+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:30:38.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:30:38 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:30:38.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:30:38 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:30:38.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:30:38 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:30:38.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:30:38 smithi115 ceph-mon[78593]: pgmap v7637: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:30:39.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:30:38 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:30:39.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:30:38 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:30:39.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:30:38 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:30:39.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:30:38 smithi047 ceph-mon[77147]: pgmap v7637: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:30:39.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:30:39 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:30:39.860 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:30:39 smithi115 conmon[90393]: 2022-05-14T03:30:39.574+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:30:40.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:30:39 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:30:40.861 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:30:40 smithi115 conmon[90393]: 2022-05-14T03:30:40.537+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:30:40.862 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:30:40 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:30:40.862 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:30:40 smithi115 ceph-mon[78593]: pgmap v7638: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:30:41.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:30:40 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:30:41.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:30:40 smithi047 ceph-mon[77147]: pgmap v7638: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:30:41.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:30:41 smithi115 conmon[90393]: 2022-05-14T03:30:41.497+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:30:41.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:30:41 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:30:42.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:30:41 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:30:42.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:30:42 smithi115 conmon[90393]: 2022-05-14T03:30:42.479+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:30:42.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:30:42 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:30:42.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:30:42 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:30:42.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:30:42 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:30:42.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:30:42 smithi115 ceph-mon[78593]: pgmap v7639: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:30:43.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:30:42 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:30:43.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:30:42 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:30:43.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:30:42 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:30:43.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:30:42 smithi047 ceph-mon[77147]: pgmap v7639: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:30:43.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:30:43 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:30:43.860 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:30:43 smithi115 conmon[90393]: 2022-05-14T03:30:43.526+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:30:44.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:30:43 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:30:44.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:30:44 smithi115 conmon[90393]: 2022-05-14T03:30:44.498+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:30:44.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:30:44 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:30:44.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:30:44 smithi115 ceph-mon[78593]: pgmap v7640: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:30:45.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:30:44 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:30:45.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:30:44 smithi047 ceph-mon[77147]: pgmap v7640: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:30:45.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:30:45 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:30:45.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:30:45 smithi115 conmon[90393]: 2022-05-14T03:30:45.501+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:30:46.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:30:45 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:30:46.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:30:46 smithi115 conmon[90393]: 2022-05-14T03:30:46.544+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:30:46.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:30:46 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:30:46.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:30:46 smithi115 ceph-mon[78593]: pgmap v7641: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:30:46.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:30:46 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 14580 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:30:47.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:30:46 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:30:47.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:30:46 smithi047 ceph-mon[77147]: pgmap v7641: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:30:47.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:30:46 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 14580 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:30:47.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:30:47 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:30:47.860 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:30:47 smithi115 conmon[90393]: 2022-05-14T03:30:47.583+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:30:48.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:30:47 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:30:48.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:30:48 smithi115 conmon[90393]: 2022-05-14T03:30:48.565+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:30:48.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:30:48 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:30:48.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:30:48 smithi115 ceph-mon[78593]: pgmap v7642: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:30:49.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:30:48 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:30:49.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:30:48 smithi047 ceph-mon[77147]: pgmap v7642: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:30:50.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:30:49 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:30:50.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:30:49 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:30:50.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:30:49 smithi115 conmon[90393]: 2022-05-14T03:30:49.609+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:30:51.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:30:50 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:30:51.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:30:50 smithi047 ceph-mon[77147]: pgmap v7643: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:30:51.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:30:50 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:30:51.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:30:50 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:30:51.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:30:50 smithi115 conmon[90393]: 2022-05-14T03:30:50.616+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:30:51.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:30:50 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:30:51.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:30:50 smithi115 ceph-mon[78593]: pgmap v7643: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:30:51.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:30:50 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:30:51.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:30:50 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:30:52.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:30:51 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:30:52.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:30:51 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 14585 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:30:52.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:30:51 smithi115 conmon[90393]: 2022-05-14T03:30:51.614+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:30:52.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:30:51 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:30:52.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:30:51 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 14585 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:30:53.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:30:52 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:30:53.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:30:52 smithi047 ceph-mon[77147]: pgmap v7644: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:30:53.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:30:52 smithi115 conmon[90393]: 2022-05-14T03:30:52.646+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:30:53.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:30:52 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:30:53.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:30:52 smithi115 ceph-mon[78593]: pgmap v7644: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:30:54.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:30:53 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:30:54.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:30:53 smithi115 conmon[90393]: 2022-05-14T03:30:53.647+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:30:54.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:30:53 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:30:55.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:30:54 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:30:55.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:30:54 smithi047 ceph-mon[77147]: pgmap v7645: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:30:55.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:30:54 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:30:55.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:30:54 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:30:55.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:30:54 smithi115 conmon[90393]: 2022-05-14T03:30:54.682+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:30:55.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:30:54 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:30:55.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:30:54 smithi115 ceph-mon[78593]: pgmap v7645: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:30:55.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:30:54 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:30:55.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:30:54 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:30:56.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:30:55 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:30:56.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:30:55 smithi115 conmon[90393]: 2022-05-14T03:30:55.709+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:30:56.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:30:55 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:30:57.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:30:56 smithi047 ceph-mon[77147]: pgmap v7646: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:30:57.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:30:56 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:30:57.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:30:56 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 14590 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:30:57.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:30:56 smithi115 conmon[90393]: 2022-05-14T03:30:56.755+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:30:57.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:30:56 smithi115 ceph-mon[78593]: pgmap v7646: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:30:57.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:30:56 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:30:57.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:30:56 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 14590 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:30:58.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:30:57 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:30:58.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:30:57 smithi115 conmon[90393]: 2022-05-14T03:30:57.776+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:30:58.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:30:57 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:30:59.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:30:58 smithi047 ceph-mon[77147]: pgmap v7647: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:30:59.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:30:58 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:30:59.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:30:58 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:30:59.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:30:58 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:30:59.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:30:58 smithi115 conmon[90393]: 2022-05-14T03:30:58.740+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:30:59.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:30:58 smithi115 ceph-mon[78593]: pgmap v7647: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:30:59.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:30:58 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:30:59.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:30:58 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:30:59.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:30:58 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:31:00.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:30:59 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:31:00.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:30:59 smithi115 conmon[90393]: 2022-05-14T03:30:59.745+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:31:00.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:30:59 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:31:01.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:31:00 smithi047 ceph-mon[77147]: pgmap v7648: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:31:01.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:31:00 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:31:01.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:31:00 smithi115 conmon[90393]: 2022-05-14T03:31:00.729+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:31:01.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:31:00 smithi115 ceph-mon[78593]: pgmap v7648: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:31:01.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:31:00 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:31:02.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:31:01 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:31:02.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:31:01 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 14595 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:31:02.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:31:01 smithi115 conmon[90393]: 2022-05-14T03:31:01.741+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:31:02.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:31:01 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:31:02.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:31:01 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 14595 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:31:03.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:31:02 smithi047 ceph-mon[77147]: pgmap v7649: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:31:03.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:31:02 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:31:03.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:31:02 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:31:03.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:31:02 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:31:03.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:31:02 smithi115 conmon[90393]: 2022-05-14T03:31:02.780+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:31:03.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:31:02 smithi115 ceph-mon[78593]: pgmap v7649: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:31:03.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:31:02 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:31:03.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:31:02 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:31:03.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:31:02 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:31:04.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:31:03 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:31:04.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:31:03 smithi115 conmon[90393]: 2022-05-14T03:31:03.756+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:31:04.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:31:03 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:31:05.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:31:04 smithi047 ceph-mon[77147]: pgmap v7650: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:31:05.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:31:04 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:31:05.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:31:04 smithi115 conmon[90393]: 2022-05-14T03:31:04.801+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:31:05.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:31:04 smithi115 ceph-mon[78593]: pgmap v7650: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:31:05.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:31:04 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:31:06.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:31:05 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:31:06.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:31:05 smithi115 conmon[90393]: 2022-05-14T03:31:05.808+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:31:06.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:31:05 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:31:07.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:31:06 smithi047 ceph-mon[77147]: pgmap v7651: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:31:07.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:31:06 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:31:07.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:31:06 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 14600 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:31:07.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:31:06 smithi115 conmon[90393]: 2022-05-14T03:31:06.838+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:31:07.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:31:06 smithi115 ceph-mon[78593]: pgmap v7651: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:31:07.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:31:06 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:31:07.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:31:06 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 14600 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:31:08.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:31:07 smithi115 conmon[90393]: 2022-05-14T03:31:07.863+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:31:09.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:31:08 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:31:09.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:31:08 smithi047 ceph-mon[77147]: pgmap v7652: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:31:09.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:31:08 smithi115 conmon[90393]: 2022-05-14T03:31:08.879+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:31:09.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:31:08 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:31:09.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:31:08 smithi115 ceph-mon[78593]: pgmap v7652: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:31:10.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:31:09 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:31:10.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:31:09 smithi115 conmon[90393]: 2022-05-14T03:31:09.831+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:31:10.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:31:09 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:31:11.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:31:10 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:31:11.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:31:10 smithi047 ceph-mon[77147]: pgmap v7653: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:31:11.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:31:10 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:31:11.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:31:10 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:31:11.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:31:10 smithi115 conmon[90393]: 2022-05-14T03:31:10.831+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:31:11.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:31:10 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:31:11.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:31:10 smithi115 ceph-mon[78593]: pgmap v7653: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:31:11.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:31:10 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:31:11.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:31:10 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:31:12.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:31:11 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:31:12.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:31:11 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 14605 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:31:12.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:31:11 smithi115 conmon[90393]: 2022-05-14T03:31:11.856+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:31:12.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:31:11 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:31:12.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:31:11 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 14605 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:31:13.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:31:12 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:31:13.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:31:12 smithi047 ceph-mon[77147]: pgmap v7654: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:31:13.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:31:12 smithi115 conmon[90393]: 2022-05-14T03:31:12.884+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:31:13.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:31:12 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:31:13.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:31:12 smithi115 ceph-mon[78593]: pgmap v7654: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:31:14.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:31:13 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:31:14.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:31:13 smithi115 conmon[90393]: 2022-05-14T03:31:13.907+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:31:14.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:31:13 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:31:15.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:31:14 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:31:15.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:31:14 smithi047 ceph-mon[77147]: pgmap v7655: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:31:15.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:31:14 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:31:15.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:31:14 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:31:15.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:31:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T03:31:15.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:31:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T03:31:15.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:31:14 smithi115 conmon[90393]: 2022-05-14T03:31:14.886+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:31:15.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:31:14 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:31:15.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:31:14 smithi115 ceph-mon[78593]: pgmap v7655: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:31:15.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:31:14 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:31:15.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:31:14 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:31:15.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:31:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T03:31:15.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:31:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T03:31:16.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:31:15 smithi115 conmon[90393]: 2022-05-14T03:31:15.899+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:31:16.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:31:15 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:31:16.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:31:15 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:31:17.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:31:16 smithi115 conmon[90393]: 2022-05-14T03:31:16.931+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:31:17.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:31:16 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:31:17.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:31:16 smithi115 ceph-mon[78593]: pgmap v7656: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:31:17.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:31:16 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 14610 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:31:17.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:31:16 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:31:17.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:31:16 smithi047 ceph-mon[77147]: pgmap v7656: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:31:17.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:31:16 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 14610 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:31:18.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:31:17 smithi115 conmon[90393]: 2022-05-14T03:31:17.898+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:31:18.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:31:17 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:31:18.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:31:17 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:31:19.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:31:18 smithi115 conmon[90393]: 2022-05-14T03:31:18.887+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:31:19.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:31:18 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:31:19.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:31:18 smithi115 ceph-mon[78593]: pgmap v7657: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:31:19.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:31:18 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:31:19.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:31:18 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:31:19.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:31:18 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:31:19.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:31:18 smithi047 ceph-mon[77147]: pgmap v7657: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:31:19.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:31:18 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:31:19.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:31:18 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:31:20.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:31:19 smithi115 conmon[90393]: 2022-05-14T03:31:19.905+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:31:20.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:31:19 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:31:20.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:31:19 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:31:21.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:31:20 smithi115 conmon[90393]: 2022-05-14T03:31:20.955+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:31:21.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:31:20 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:31:21.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:31:20 smithi115 ceph-mon[78593]: pgmap v7658: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:31:21.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:31:20 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:31:21.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:31:20 smithi047 ceph-mon[77147]: pgmap v7658: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:31:22.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:31:21 smithi115 conmon[90393]: 2022-05-14T03:31:21.923+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:31:22.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:31:21 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:31:22.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:31:21 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 14615 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:31:22.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:31:21 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:31:22.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:31:21 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 14615 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:31:23.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:31:22 smithi115 conmon[90393]: 2022-05-14T03:31:22.925+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:31:23.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:31:22 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:31:23.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:31:22 smithi115 ceph-mon[78593]: pgmap v7659: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:31:23.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:31:22 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:31:23.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:31:22 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:31:23.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:31:22 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:31:23.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:31:22 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:31:23.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:31:22 smithi047 ceph-mon[77147]: pgmap v7659: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:31:23.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:31:22 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:31:23.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:31:22 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:31:23.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:31:22 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:31:24.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:31:23 smithi115 conmon[90393]: 2022-05-14T03:31:23.926+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:31:24.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:31:23 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:31:24.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:31:23 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:31:25.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:31:24 smithi115 conmon[90393]: 2022-05-14T03:31:24.974+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:31:25.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:31:24 smithi115 ceph-mon[78593]: pgmap v7660: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:31:25.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:31:24 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:31:25.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:31:24 smithi047 ceph-mon[77147]: pgmap v7660: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:31:25.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:31:24 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:31:26.110 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:31:25 smithi115 conmon[90393]: 2022-05-14T03:31:25.964+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:31:26.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:31:25 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:31:26.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:31:25 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:31:27.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:31:26 smithi115 conmon[90393]: 2022-05-14T03:31:26.936+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:31:27.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:31:26 smithi115 ceph-mon[78593]: pgmap v7661: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:31:27.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:31:26 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:31:27.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:31:26 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 14620 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:31:27.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:31:26 smithi047 ceph-mon[77147]: pgmap v7661: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:31:27.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:31:26 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:31:27.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:31:26 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 14620 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:31:28.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:31:27 smithi115 conmon[90393]: 2022-05-14T03:31:27.898+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:31:28.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:31:27 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:31:28.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:31:27 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:31:29.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:31:28 smithi115 conmon[90393]: 2022-05-14T03:31:28.865+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:31:29.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:31:28 smithi115 ceph-mon[78593]: pgmap v7662: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:31:29.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:31:28 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:31:29.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:31:28 smithi047 ceph-mon[77147]: pgmap v7662: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:31:29.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:31:28 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:31:30.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:31:29 smithi115 conmon[90393]: 2022-05-14T03:31:29.907+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:31:30.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:31:29 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:31:30.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:31:29 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:31:31.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:31:30 smithi115 conmon[90393]: 2022-05-14T03:31:30.933+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:31:31.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:31:30 smithi115 ceph-mon[78593]: pgmap v7663: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:31:31.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:31:30 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:31:31.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:31:30 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:31:31.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:31:30 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:31:31.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:31:30 smithi047 ceph-mon[77147]: pgmap v7663: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:31:31.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:31:30 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:31:31.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:31:30 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:31:31.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:31:30 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:31:32.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:31:31 smithi115 conmon[90393]: 2022-05-14T03:31:31.982+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:31:32.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:31:31 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:31:32.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:31:31 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 14625 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:31:32.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:31:31 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:31:32.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:31:31 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 14625 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:31:33.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:31:32 smithi047 ceph-mon[77147]: pgmap v7664: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:31:33.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:31:32 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:31:33.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:31:32 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T03:31:33.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:31:32 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T03:31:33.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:31:32 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T03:31:33.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:31:32 smithi115 conmon[90393]: 2022-05-14T03:31:32.944+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:31:33.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:31:32 smithi115 ceph-mon[78593]: pgmap v7664: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:31:33.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:31:32 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:31:33.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:31:32 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T03:31:33.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:31:32 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T03:31:33.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:31:32 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T03:31:34.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:31:33 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:31:34.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:31:33 smithi115 conmon[90393]: 2022-05-14T03:31:33.909+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:31:34.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:31:33 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:31:35.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:31:34 smithi047 ceph-mon[77147]: pgmap v7665: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:31:35.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:31:34 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:31:35.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:31:34 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:31:35.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:31:34 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:31:35.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:31:34 smithi115 conmon[90393]: 2022-05-14T03:31:34.933+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:31:35.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:31:34 smithi115 ceph-mon[78593]: pgmap v7665: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:31:35.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:31:34 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:31:35.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:31:34 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:31:35.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:31:34 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:31:36.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:31:35 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:31:36.293 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:31:35 smithi047 conmon[78521]: 2022-05-14T03:31:35.791+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:31:36.293 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:31:35 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T03:31:36.293 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:31:35 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:31:36.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:31:35 smithi047 conmon[78521]: yield 2022-05-14T03:31:36.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:31:35 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:31:36.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:31:35 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:31:36.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:31:35 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:31:36.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:31:35 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T03:31:36.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:31:35 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:31:36.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:31:35 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T03:31:36.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:31:35 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:31:36.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:31:35 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T03:31:36.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:31:35 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:31:36.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:31:35 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T03:31:36.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:31:35 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:31:36.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:31:35 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T03:31:36.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:31:35 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:31:36.297 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:31:35 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:31:36.297 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:31:35 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:31:36.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:31:35 smithi115 conmon[90393]: 2022-05-14T03:31:35.966+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:31:36.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:31:35 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:31:37.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:31:36 smithi047 ceph-mon[77147]: pgmap v7666: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:31:37.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:31:36 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:31:37.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T03:31:37.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:31:37.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T03:31:37.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:31:37.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:31:37.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:31:37.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T03:31:37.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:31:37.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T03:31:37.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:31:37.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T03:31:37.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:31:37.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T03:31:37.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:31:37.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T03:31:37.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:31:37.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:31:37.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:31:37.297 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:31:36 smithi047 ceph-mon[77147]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T03:31:37.297 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:31:36 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:31:37.297 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:31:36 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 14630 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:31:37.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:31:36 smithi115 conmon[90393]: 2022-05-14T03:31:36.995+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:31:37.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:31:36 smithi115 ceph-mon[78593]: pgmap v7666: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:31:37.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:31:36 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:31:37.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T03:31:37.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:31:37.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T03:31:37.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:31:37.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:31:37.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:31:37.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T03:31:37.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:31:37.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T03:31:37.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:31:37.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T03:31:37.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:31:37.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T03:31:37.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:31:37.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T03:31:37.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:31:37.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:31:37.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:31:37.364 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:31:36 smithi115 ceph-mon[78593]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T03:31:37.364 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:31:36 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:31:37.365 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:31:36 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 14630 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:31:38.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:31:37 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:31:38.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:31:37 smithi115 conmon[90393]: 2022-05-14T03:31:37.957+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:31:38.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:31:37 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:31:39.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:31:38 smithi047 ceph-mon[77147]: pgmap v7667: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:31:39.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:31:38 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:31:39.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:31:38 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:31:39.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:31:38 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:31:39.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:31:38 smithi047 conmon[78521]: 2022-05-14T03:31:38.864+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:31:39.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:31:38 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T03:31:39.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:31:38 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:31:39.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:31:38 smithi047 conmon[78521]: yield 2022-05-14T03:31:39.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:31:38 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:31:39.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:31:38 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:31:39.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:31:38 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:31:39.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:31:38 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T03:31:39.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:31:38 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:31:39.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:31:38 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T03:31:39.297 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:31:38 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:31:39.297 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:31:38 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T03:31:39.297 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:31:38 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:31:39.297 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:31:38 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T03:31:39.298 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:31:38 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:31:39.298 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:31:38 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T03:31:39.298 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:31:38 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:31:39.298 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:31:38 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:31:39.299 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:31:38 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:31:39.299 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:31:38 smithi047 conmon[78521]: 2022-05-14T03:31:38.866+0000 7f533f777700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T03:31:39.299 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:31:38 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T03:31:39.299 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:31:38 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:31:39.299 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:31:38 smithi047 conmon[78521]: yield 2022-05-14T03:31:39.300 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:31:38 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:31:39.300 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:31:38 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:31:39.300 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:31:38 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:31:39.300 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:31:38 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T03:31:39.301 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:31:38 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:31:39.301 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:31:38 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T03:31:39.301 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:31:38 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:31:39.301 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:31:38 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T03:31:39.301 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:31:38 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:31:39.302 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:31:38 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T03:31:39.302 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:31:38 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:31:39.302 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:31:38 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T03:31:39.303 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:31:38 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:31:39.303 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:31:38 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:31:39.303 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:31:38 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:31:39.303 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:31:38 smithi047 conmon[78521]: 2022-05-14T03:31:39.304 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:31:38 smithi047 conmon[78521]: During handling of the above exception, another exception occurred: 2022-05-14T03:31:39.304 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:31:38 smithi047 conmon[78521]: 2022-05-14T03:31:39.304 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:31:38 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T03:31:39.305 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:31:38 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T03:31:39.305 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:31:38 smithi047 conmon[78521]: return f(*arg) 2022-05-14T03:31:39.305 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:31:38 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T03:31:39.305 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:31:38 smithi047 conmon[78521]: self._write_client_files(client_files, host) 2022-05-14T03:31:39.306 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:31:38 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T03:31:39.306 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:31:38 smithi047 conmon[78521]: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T03:31:39.307 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:31:38 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T03:31:39.307 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:31:38 smithi047 conmon[78521]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T03:31:39.307 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:31:38 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T03:31:39.307 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:31:38 smithi047 conmon[78521]: return self.event_loop.get_result(coro) 2022-05-14T03:31:39.308 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:31:38 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T03:31:39.308 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:31:38 smithi047 conmon[78521]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T03:31:39.309 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:31:38 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T03:31:39.309 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:31:38 smithi047 conmon[78521]: return self.__get_result() 2022-05-14T03:31:39.309 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:31:38 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T03:31:39.310 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:31:38 smithi047 conmon[78521]: raise self._exception 2022-05-14T03:31:39.310 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:31:38 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T03:31:39.310 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:31:38 smithi047 conmon[78521]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T03:31:39.311 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:31:38 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T03:31:39.311 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:31:38 smithi047 conmon[78521]: conn = await self._remote_connection(host, addr) 2022-05-14T03:31:39.311 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:31:38 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T03:31:39.311 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:31:38 smithi047 conmon[78521]: raise 2022-05-14T03:31:39.312 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:31:38 smithi047 conmon[78521]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T03:31:39.312 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:31:38 smithi047 conmon[78521]: self.gen.throw(type, value, traceback) 2022-05-14T03:31:39.312 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:31:38 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T03:31:39.313 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:31:38 smithi047 conmon[78521]: raise OrchestratorError(msg) 2022-05-14T03:31:39.313 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:31:38 smithi047 conmon[78521]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:31:39.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:31:38 smithi115 conmon[90393]: 2022-05-14T03:31:38.976+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:31:39.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:31:38 smithi115 ceph-mon[78593]: pgmap v7667: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:31:39.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:31:38 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:31:39.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:31:38 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:31:39.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:31:38 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:31:40.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:31:39 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:31:40.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T03:31:40.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:31:40.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T03:31:40.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:31:40.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:31:40.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:31:40.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T03:31:40.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:31:40.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T03:31:40.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:31:40.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T03:31:40.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:31:40.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T03:31:40.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:31:40.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T03:31:40.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:31:40.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:31:40.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:31:40.297 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:31:39 smithi047 ceph-mon[77147]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T03:31:40.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T03:31:40.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:31:40.298 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T03:31:40.298 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:31:40.298 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:31:40.299 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:31:40.299 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T03:31:40.299 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:31:40.299 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T03:31:40.299 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:31:40.300 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T03:31:40.300 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:31:40.300 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T03:31:40.300 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:31:40.301 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T03:31:40.301 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:31:40.301 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:31:40.301 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:31:40.302 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T03:31:40.302 INFO:journalctl@ceph.mon.b.smithi047.stdout: During handling of the above exception, another exception occurred: 2022-05-14T03:31:40.302 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T03:31:40.302 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T03:31:40.302 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T03:31:40.303 INFO:journalctl@ceph.mon.b.smithi047.stdout: return f(*arg) 2022-05-14T03:31:40.303 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T03:31:40.303 INFO:journalctl@ceph.mon.b.smithi047.stdout: self._write_client_files(client_files, host) 2022-05-14T03:31:40.303 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T03:31:40.304 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T03:31:40.304 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T03:31:40.304 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T03:31:40.304 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T03:31:40.304 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.event_loop.get_result(coro) 2022-05-14T03:31:40.305 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T03:31:40.305 INFO:journalctl@ceph.mon.b.smithi047.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T03:31:40.305 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T03:31:40.305 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.__get_result() 2022-05-14T03:31:40.306 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T03:31:40.306 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise self._exception 2022-05-14T03:31:40.306 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T03:31:40.306 INFO:journalctl@ceph.mon.b.smithi047.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T03:31:40.307 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T03:31:40.307 INFO:journalctl@ceph.mon.b.smithi047.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T03:31:40.307 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T03:31:40.307 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise 2022-05-14T03:31:40.307 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T03:31:40.308 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.gen.throw(type, value, traceback) 2022-05-14T03:31:40.308 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T03:31:40.308 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OrchestratorError(msg) 2022-05-14T03:31:40.308 INFO:journalctl@ceph.mon.b.smithi047.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:31:40.309 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:31:39 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:31:40.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:31:39 smithi115 conmon[90393]: 2022-05-14T03:31:39.946+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:31:40.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:31:39 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:31:40.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T03:31:40.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:31:40.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T03:31:40.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:31:40.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:31:40.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:31:40.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T03:31:40.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:31:40.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T03:31:40.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:31:40.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T03:31:40.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:31:40.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T03:31:40.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:31:40.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T03:31:40.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:31:40.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:31:40.365 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:31:40.365 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:31:39 smithi115 ceph-mon[78593]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T03:31:40.365 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T03:31:40.365 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:31:40.366 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T03:31:40.366 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:31:40.366 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:31:40.366 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:31:40.367 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T03:31:40.367 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:31:40.367 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T03:31:40.367 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:31:40.368 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T03:31:40.368 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:31:40.368 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T03:31:40.368 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:31:40.368 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T03:31:40.369 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:31:40.369 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:31:40.369 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:31:40.369 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T03:31:40.370 INFO:journalctl@ceph.mon.c.smithi115.stdout: During handling of the above exception, another exception occurred: 2022-05-14T03:31:40.370 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T03:31:40.370 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T03:31:40.370 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T03:31:40.371 INFO:journalctl@ceph.mon.c.smithi115.stdout: return f(*arg) 2022-05-14T03:31:40.371 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T03:31:40.371 INFO:journalctl@ceph.mon.c.smithi115.stdout: self._write_client_files(client_files, host) 2022-05-14T03:31:40.371 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T03:31:40.372 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T03:31:40.372 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T03:31:40.372 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T03:31:40.372 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T03:31:40.372 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.event_loop.get_result(coro) 2022-05-14T03:31:40.373 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T03:31:40.373 INFO:journalctl@ceph.mon.c.smithi115.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T03:31:40.374 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T03:31:40.374 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.__get_result() 2022-05-14T03:31:40.374 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T03:31:40.375 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise self._exception 2022-05-14T03:31:40.375 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T03:31:40.375 INFO:journalctl@ceph.mon.c.smithi115.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T03:31:40.375 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T03:31:40.376 INFO:journalctl@ceph.mon.c.smithi115.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T03:31:40.376 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T03:31:40.376 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise 2022-05-14T03:31:40.376 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T03:31:40.377 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.gen.throw(type, value, traceback) 2022-05-14T03:31:40.377 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T03:31:40.377 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OrchestratorError(msg) 2022-05-14T03:31:40.377 INFO:journalctl@ceph.mon.c.smithi115.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:31:40.378 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:31:39 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:31:41.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:31:40 smithi047 ceph-mon[77147]: pgmap v7668: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:31:41.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:31:40 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:31:41.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:31:40 smithi115 conmon[90393]: 2022-05-14T03:31:40.897+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:31:41.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:31:40 smithi115 ceph-mon[78593]: pgmap v7668: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:31:41.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:31:40 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:31:42.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:31:41 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:31:42.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:31:41 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 14635 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:31:42.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:31:41 smithi115 conmon[90393]: 2022-05-14T03:31:41.897+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:31:42.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:31:41 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:31:42.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:31:41 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 14635 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:31:43.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:31:42 smithi047 ceph-mon[77147]: pgmap v7669: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:31:43.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:31:42 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:31:43.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:31:42 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:31:43.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:31:42 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:31:43.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:31:42 smithi115 conmon[90393]: 2022-05-14T03:31:42.905+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:31:43.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:31:42 smithi115 ceph-mon[78593]: pgmap v7669: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:31:43.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:31:42 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:31:43.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:31:42 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:31:43.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:31:42 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:31:44.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:31:43 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:31:44.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:31:43 smithi115 conmon[90393]: 2022-05-14T03:31:43.928+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:31:44.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:31:43 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:31:45.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:31:44 smithi047 ceph-mon[77147]: pgmap v7670: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:31:45.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:31:44 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:31:45.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:31:44 smithi115 conmon[90393]: 2022-05-14T03:31:44.975+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:31:45.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:31:44 smithi115 ceph-mon[78593]: pgmap v7670: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:31:45.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:31:44 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:31:46.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:31:45 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:31:46.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:31:45 smithi115 conmon[90393]: 2022-05-14T03:31:45.993+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:31:46.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:31:45 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:31:47.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:31:46 smithi047 ceph-mon[77147]: pgmap v7671: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:31:47.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:31:46 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:31:47.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:31:46 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 14640 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:31:47.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:31:47 smithi115 conmon[90393]: 2022-05-14T03:31:47.010+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:31:47.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:31:46 smithi115 ceph-mon[78593]: pgmap v7671: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:31:47.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:31:46 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:31:47.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:31:46 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 14640 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:31:48.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:31:47 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:31:48.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:31:47 smithi115 conmon[90393]: 2022-05-14T03:31:47.995+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:31:48.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:31:47 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:31:49.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:31:48 smithi047 ceph-mon[77147]: pgmap v7672: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:31:49.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:31:48 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:31:49.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:31:49 smithi115 conmon[90393]: 2022-05-14T03:31:49.031+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:31:49.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:31:48 smithi115 ceph-mon[78593]: pgmap v7672: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:31:49.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:31:48 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:31:50.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:31:49 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:31:50.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:31:50 smithi115 conmon[90393]: 2022-05-14T03:31:50.032+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:31:50.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:31:49 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:31:51.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:31:50 smithi047 ceph-mon[77147]: pgmap v7673: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:31:51.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:31:50 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:31:51.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:31:50 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:31:51.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:31:50 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:31:51.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:31:51 smithi115 conmon[90393]: 2022-05-14T03:31:51.060+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:31:51.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:31:50 smithi115 ceph-mon[78593]: pgmap v7673: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:31:51.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:31:50 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:31:51.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:31:50 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:31:51.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:31:50 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:31:52.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:31:51 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:31:52.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:31:51 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 14645 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:31:52.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:31:52 smithi115 conmon[90393]: 2022-05-14T03:31:52.104+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:31:52.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:31:51 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:31:52.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:31:51 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 14645 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:31:53.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:31:52 smithi047 ceph-mon[77147]: pgmap v7674: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:31:53.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:31:52 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:31:53.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:31:53 smithi115 conmon[90393]: 2022-05-14T03:31:53.064+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:31:53.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:31:52 smithi115 ceph-mon[78593]: pgmap v7674: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:31:53.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:31:52 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:31:54.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:31:53 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:31:54.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:31:54 smithi115 conmon[90393]: 2022-05-14T03:31:54.093+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:31:54.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:31:53 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:31:55.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:31:54 smithi047 ceph-mon[77147]: pgmap v7675: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:31:55.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:31:54 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:31:55.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:31:54 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:31:55.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:31:54 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:31:55.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:31:55 smithi115 conmon[90393]: 2022-05-14T03:31:55.117+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:31:55.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:31:54 smithi115 ceph-mon[78593]: pgmap v7675: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:31:55.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:31:54 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:31:55.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:31:54 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:31:55.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:31:54 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:31:56.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:31:55 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:31:56.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:31:55 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:31:56.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:31:56 smithi115 conmon[90393]: 2022-05-14T03:31:56.137+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:31:57.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:31:56 smithi047 ceph-mon[77147]: pgmap v7676: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:31:57.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:31:56 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:31:57.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:31:56 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 14651 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:31:57.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:31:57 smithi115 conmon[90393]: 2022-05-14T03:31:57.125+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:31:57.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:31:56 smithi115 ceph-mon[78593]: pgmap v7676: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:31:57.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:31:56 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:31:57.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:31:56 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 14651 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:31:58.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:31:57 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:31:58.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:31:57 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:31:58.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:31:58 smithi115 conmon[90393]: 2022-05-14T03:31:58.156+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:31:59.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:31:58 smithi047 ceph-mon[77147]: pgmap v7677: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:31:59.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:31:58 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:31:59.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:31:58 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:31:59.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:31:58 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:31:59.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:31:59 smithi115 conmon[90393]: 2022-05-14T03:31:59.159+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:31:59.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:31:58 smithi115 ceph-mon[78593]: pgmap v7677: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:31:59.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:31:58 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:31:59.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:31:58 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:31:59.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:31:58 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:32:00.297 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:31:59 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:32:00.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:31:59 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:32:00.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:32:00 smithi115 conmon[90393]: 2022-05-14T03:32:00.155+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:32:01.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:32:00 smithi047 ceph-mon[77147]: pgmap v7678: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:32:01.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:32:00 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:32:01.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:32:01 smithi115 conmon[90393]: 2022-05-14T03:32:01.173+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:32:01.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:32:00 smithi115 ceph-mon[78593]: pgmap v7678: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:32:01.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:32:00 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:32:02.218 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:32:01 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:32:02.218 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:32:01 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 14655 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:32:02.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:32:01 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:32:02.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:32:01 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 14655 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:32:02.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:32:02 smithi115 conmon[90393]: 2022-05-14T03:32:02.215+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:32:03.240 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:32:02 smithi115 ceph-mon[78593]: pgmap v7679: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:32:03.240 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:32:02 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:32:03.240 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:32:02 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:32:03.240 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:32:02 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:32:03.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:32:02 smithi047 ceph-mon[77147]: pgmap v7679: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:32:03.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:32:02 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:32:03.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:32:02 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:32:03.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:32:02 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:32:03.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:32:03 smithi115 conmon[90393]: 2022-05-14T03:32:03.237+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:32:04.223 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:32:03 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:32:04.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:32:03 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:32:04.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:32:04 smithi115 conmon[90393]: 2022-05-14T03:32:04.220+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:32:05.235 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:32:04 smithi115 ceph-mon[78593]: pgmap v7680: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:32:05.235 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:32:04 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:32:05.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:32:04 smithi047 ceph-mon[77147]: pgmap v7680: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:32:05.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:32:04 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:32:05.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:32:05 smithi115 conmon[90393]: 2022-05-14T03:32:05.232+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:32:06.272 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:32:05 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:32:06.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:32:05 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:32:06.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:32:06 smithi115 conmon[90393]: 2022-05-14T03:32:06.269+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:32:07.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:32:06 smithi047 ceph-mon[77147]: pgmap v7681: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:32:07.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:32:06 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:32:07.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:32:06 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 14660 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:32:07.317 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:32:06 smithi115 ceph-mon[78593]: pgmap v7681: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:32:07.317 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:32:06 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:32:07.317 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:32:06 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 14660 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:32:07.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:32:07 smithi115 conmon[90393]: 2022-05-14T03:32:07.314+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:32:08.290 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:32:07 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:32:08.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:32:07 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:32:08.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:32:08 smithi115 conmon[90393]: 2022-05-14T03:32:08.287+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:32:09.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:32:08 smithi047 ceph-mon[77147]: pgmap v7682: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:32:09.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:32:08 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:32:09.338 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:32:08 smithi115 ceph-mon[78593]: pgmap v7682: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:32:09.338 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:32:08 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:32:09.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:32:09 smithi115 conmon[90393]: 2022-05-14T03:32:09.335+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:32:10.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:32:09 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:32:10.294 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:32:09 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:32:10.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:32:10 smithi115 conmon[90393]: 2022-05-14T03:32:10.292+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:32:11.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:32:10 smithi047 ceph-mon[77147]: pgmap v7683: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:32:11.295 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:32:10 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:32:11.295 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:32:10 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:32:11.295 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:32:10 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:32:11.316 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:32:10 smithi115 ceph-mon[78593]: pgmap v7683: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:32:11.317 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:32:10 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:32:11.317 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:32:10 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:32:11.317 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:32:10 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:32:11.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:32:11 smithi115 conmon[90393]: 2022-05-14T03:32:11.314+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:32:12.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:32:11 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:32:12.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:32:11 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 14665 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:32:12.299 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:32:11 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:32:12.299 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:32:11 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 14665 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:32:12.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:32:12 smithi115 conmon[90393]: 2022-05-14T03:32:12.296+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:32:13.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:32:12 smithi047 ceph-mon[77147]: pgmap v7684: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:32:13.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:32:12 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:32:13.312 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:32:12 smithi115 ceph-mon[78593]: pgmap v7684: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:32:13.312 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:32:12 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:32:13.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:32:13 smithi115 conmon[90393]: 2022-05-14T03:32:13.309+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:32:14.274 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:32:13 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:32:14.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:32:13 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:32:14.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:32:14 smithi115 conmon[90393]: 2022-05-14T03:32:14.272+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:32:15.238 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:32:14 smithi115 ceph-mon[78593]: pgmap v7685: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:32:15.239 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:32:14 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:32:15.239 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:32:14 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:32:15.239 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:32:14 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:32:15.240 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:32:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T03:32:15.240 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:32:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T03:32:15.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:32:14 smithi047 ceph-mon[77147]: pgmap v7685: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:32:15.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:32:14 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:32:15.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:32:14 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:32:15.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:32:14 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:32:15.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:32:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T03:32:15.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:32:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T03:32:15.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:32:15 smithi115 conmon[90393]: 2022-05-14T03:32:15.236+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:32:16.258 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:32:15 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:32:16.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:32:15 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:32:16.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:32:16 smithi115 conmon[90393]: 2022-05-14T03:32:16.256+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:32:17.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:32:16 smithi047 ceph-mon[77147]: pgmap v7686: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:32:17.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:32:16 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:32:17.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:32:16 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 14670 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:32:17.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:32:17 smithi115 conmon[90393]: 2022-05-14T03:32:17.235+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:32:17.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:32:16 smithi115 ceph-mon[78593]: pgmap v7686: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:32:17.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:32:16 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:32:17.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:32:16 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 14670 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:32:18.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:32:17 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:32:18.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:32:17 smithi047 ceph-mon[77147]: pgmap v7687: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:32:18.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:32:18 smithi115 conmon[90393]: 2022-05-14T03:32:18.202+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:32:18.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:32:17 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:32:18.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:32:17 smithi115 ceph-mon[78593]: pgmap v7687: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:32:19.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:32:18 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:32:19.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:32:18 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:32:19.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:32:18 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:32:19.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:32:19 smithi115 conmon[90393]: 2022-05-14T03:32:19.227+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:32:19.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:32:18 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:32:19.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:32:18 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:32:19.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:32:18 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:32:20.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:32:19 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:32:20.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:32:19 smithi047 ceph-mon[77147]: pgmap v7688: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:32:20.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:32:20 smithi115 conmon[90393]: 2022-05-14T03:32:20.213+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:32:20.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:32:20 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:32:20.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:32:20 smithi115 ceph-mon[78593]: pgmap v7688: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:32:21.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:32:21 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:32:21.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:32:21 smithi115 conmon[90393]: 2022-05-14T03:32:21.235+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:32:21.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:32:21 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:32:22.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:32:22 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:32:22.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:32:22 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 14675 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:32:22.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:32:22 smithi047 ceph-mon[77147]: pgmap v7689: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:32:22.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:32:22 smithi115 conmon[90393]: 2022-05-14T03:32:22.217+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:32:22.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:32:22 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:32:22.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:32:22 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 14675 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:32:22.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:32:22 smithi115 ceph-mon[78593]: pgmap v7689: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:32:23.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:32:23 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:32:23.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:32:23 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:32:23.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:32:23 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:32:23.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:32:23 smithi115 conmon[90393]: 2022-05-14T03:32:23.176+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:32:23.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:32:23 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:32:23.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:32:23 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:32:23.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:32:23 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:32:24.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:32:24 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:32:24.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:32:24 smithi047 ceph-mon[77147]: pgmap v7690: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:32:24.360 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:32:24 smithi115 conmon[90393]: 2022-05-14T03:32:24.208+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:32:24.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:32:24 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:32:24.362 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:32:24 smithi115 ceph-mon[78593]: pgmap v7690: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:32:25.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:32:25 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:32:25.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:32:25 smithi115 conmon[90393]: 2022-05-14T03:32:25.213+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:32:25.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:32:25 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:32:26.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:32:26 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:32:26.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:32:26 smithi047 ceph-mon[77147]: pgmap v7691: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:32:26.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:32:26 smithi115 conmon[90393]: 2022-05-14T03:32:26.209+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:32:26.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:32:26 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:32:26.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:32:26 smithi115 ceph-mon[78593]: pgmap v7691: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:32:27.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:32:27 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:32:27.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:32:27 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 14680 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:32:27.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:32:27 smithi115 conmon[90393]: 2022-05-14T03:32:27.237+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:32:27.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:32:27 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:32:27.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:32:27 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 14680 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:32:28.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:32:28 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:32:28.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:32:28 smithi047 ceph-mon[77147]: pgmap v7692: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:32:28.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:32:28 smithi115 conmon[90393]: 2022-05-14T03:32:28.219+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:32:28.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:32:28 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:32:28.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:32:28 smithi115 ceph-mon[78593]: pgmap v7692: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:32:29.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:32:29 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:32:29.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:32:29 smithi115 conmon[90393]: 2022-05-14T03:32:29.257+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:32:29.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:32:29 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:32:30.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:32:30 smithi115 conmon[90393]: 2022-05-14T03:32:30.231+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:32:30.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:32:30 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:32:30.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:32:30 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:32:30.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:32:30 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:32:30.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:32:30 smithi115 ceph-mon[78593]: pgmap v7693: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:32:30.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:32:30 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:32:30.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:32:30 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:32:30.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:32:30 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:32:30.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:32:30 smithi047 ceph-mon[77147]: pgmap v7693: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:32:31.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:32:31 smithi115 conmon[90393]: 2022-05-14T03:32:31.216+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:32:31.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:32:31 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:32:31.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:32:31 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:32:32.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:32:32 smithi115 conmon[90393]: 2022-05-14T03:32:32.245+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:32:32.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:32:32 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:32:32.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:32:32 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 14685 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:32:32.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:32:32 smithi115 ceph-mon[78593]: pgmap v7694: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:32:32.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:32:32 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:32:32.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:32:32 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 14685 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:32:32.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:32:32 smithi047 ceph-mon[77147]: pgmap v7694: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:32:33.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:32:33 smithi115 conmon[90393]: 2022-05-14T03:32:33.208+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:32:34.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:32:33 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:32:34.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:32:33 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:32:34.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:32:34 smithi115 conmon[90393]: 2022-05-14T03:32:34.257+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:32:35.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:32:34 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:32:35.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:32:34 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:32:35.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:32:34 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:32:35.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:32:34 smithi047 ceph-mon[77147]: pgmap v7695: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:32:35.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:32:34 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:32:35.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:32:34 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:32:35.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:32:34 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:32:35.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:32:34 smithi115 ceph-mon[78593]: pgmap v7695: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:32:35.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:32:35 smithi115 conmon[90393]: 2022-05-14T03:32:35.300+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:32:36.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:32:35 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:32:36.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:32:35 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:32:36.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:32:36 smithi115 conmon[90393]: 2022-05-14T03:32:36.350+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:32:37.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:32:36 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:32:37.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:32:36 smithi047 ceph-mon[77147]: pgmap v7696: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:32:37.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:32:36 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:32:37.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:32:36 smithi115 ceph-mon[78593]: pgmap v7696: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:32:37.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:32:37 smithi115 conmon[90393]: 2022-05-14T03:32:37.349+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:32:38.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:32:37 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:32:38.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:32:37 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 14690 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:32:38.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:32:37 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:32:38.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:32:37 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 14690 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:32:38.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:32:38 smithi115 conmon[90393]: 2022-05-14T03:32:38.340+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:32:38.913 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:32:38 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:32:38.914 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:32:38 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:32:38.914 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:32:38 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:32:38.914 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:32:38 smithi047 ceph-mon[77147]: pgmap v7697: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:32:38.915 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:32:38 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:32:38.915 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:32:38 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:32:38.915 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:32:38 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:32:38.915 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:32:38 smithi115 ceph-mon[78593]: pgmap v7697: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:32:39.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:32:39 smithi115 conmon[90393]: 2022-05-14T03:32:39.295+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:32:39.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:32:39 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:32:39.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:32:39 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T03:32:39.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:32:39 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T03:32:39.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:32:39 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T03:32:40.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:32:39 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:32:40.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:32:39 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T03:32:40.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:32:39 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T03:32:40.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:32:39 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T03:32:40.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:32:40 smithi115 conmon[90393]: 2022-05-14T03:32:40.277+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:32:41.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:32:40 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:32:41.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:32:40 smithi047 ceph-mon[77147]: pgmap v7698: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:32:41.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:32:40 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:32:41.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:32:40 smithi115 ceph-mon[78593]: pgmap v7698: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:32:41.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:32:41 smithi115 conmon[90393]: 2022-05-14T03:32:41.260+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:32:41.971 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:32:41 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:32:42.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:32:41 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:32:42.293 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:32:41 smithi047 conmon[78521]: 2022-05-14T03:32:41.968+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:32:42.293 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:32:41 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T03:32:42.293 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:32:41 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:32:42.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:32:41 smithi047 conmon[78521]: yield 2022-05-14T03:32:42.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:32:41 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:32:42.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:32:41 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:32:42.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:32:41 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:32:42.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:32:41 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T03:32:42.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:32:41 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:32:42.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:32:41 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T03:32:42.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:32:41 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:32:42.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:32:41 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T03:32:42.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:32:41 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:32:42.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:32:41 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T03:32:42.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:32:41 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:32:42.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:32:41 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T03:32:42.297 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:32:41 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:32:42.297 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:32:41 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:32:42.297 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:32:41 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:32:42.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:32:42 smithi115 conmon[90393]: 2022-05-14T03:32:42.306+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:32:43.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:32:42 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:32:43.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:32:42 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:32:43.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:32:42 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:32:43.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:32:42 smithi047 ceph-mon[77147]: pgmap v7699: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:32:43.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:32:42 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:32:43.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T03:32:43.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:32:43.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T03:32:43.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:32:43.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:32:43.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:32:43.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T03:32:43.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:32:43.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T03:32:43.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:32:43.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T03:32:43.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:32:43.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T03:32:43.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:32:43.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T03:32:43.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:32:43.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:32:43.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:32:43.048 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:32:42 smithi047 ceph-mon[77147]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T03:32:43.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:32:42 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:32:43.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:32:42 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:32:43.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:32:42 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:32:43.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:32:42 smithi115 ceph-mon[78593]: pgmap v7699: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:32:43.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:32:42 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:32:43.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T03:32:43.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:32:43.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T03:32:43.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:32:43.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:32:43.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:32:43.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T03:32:43.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:32:43.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T03:32:43.113 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:32:43.113 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T03:32:43.113 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:32:43.113 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T03:32:43.114 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:32:43.114 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T03:32:43.114 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:32:43.114 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:32:43.115 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:32:43.115 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:32:42 smithi115 ceph-mon[78593]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T03:32:43.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:32:43 smithi115 conmon[90393]: 2022-05-14T03:32:43.270+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:32:44.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:32:43 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:32:44.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:32:43 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:32:44.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:32:44 smithi115 conmon[90393]: 2022-05-14T03:32:44.297+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:32:45.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:32:44 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:32:45.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:32:44 smithi047 ceph-mon[77147]: pgmap v7700: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:32:45.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:32:44 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:32:45.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:32:44 smithi115 ceph-mon[78593]: pgmap v7700: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:32:45.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:32:45 smithi047 conmon[78521]: 2022-05-14T03:32:45.041+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:32:45.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:32:45 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T03:32:45.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:32:45 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:32:45.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:32:45 smithi047 conmon[78521]: yield 2022-05-14T03:32:45.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:32:45 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:32:45.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:32:45 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:32:45.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:32:45 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:32:45.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:32:45 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T03:32:45.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:32:45 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:32:45.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:32:45 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T03:32:45.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:32:45 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:32:45.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:32:45 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T03:32:45.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:32:45 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:32:45.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:32:45 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T03:32:45.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:32:45 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:32:45.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:32:45 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T03:32:45.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:32:45 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:32:45.547 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:32:45 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:32:45.547 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:32:45 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:32:45.547 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:32:45 smithi047 conmon[78521]: 2022-05-14T03:32:45.041+0000 7f5341f7c700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T03:32:45.547 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:32:45 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T03:32:45.548 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:32:45 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:32:45.548 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:32:45 smithi047 conmon[78521]: yield 2022-05-14T03:32:45.548 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:32:45 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:32:45.548 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:32:45 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:32:45.548 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:32:45 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:32:45.549 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:32:45 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T03:32:45.549 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:32:45 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:32:45.549 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:32:45 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T03:32:45.549 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:32:45 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:32:45.550 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:32:45 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T03:32:45.550 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:32:45 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:32:45.550 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:32:45 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T03:32:45.550 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:32:45 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:32:45.551 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:32:45 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T03:32:45.551 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:32:45 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:32:45.551 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:32:45 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:32:45.551 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:32:45 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:32:45.551 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:32:45 smithi047 conmon[78521]: 2022-05-14T03:32:45.552 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:32:45 smithi047 conmon[78521]: During handling of the above exception, another exception occurred: 2022-05-14T03:32:45.552 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:32:45 smithi047 conmon[78521]: 2022-05-14T03:32:45.552 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:32:45 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T03:32:45.553 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:32:45 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T03:32:45.553 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:32:45 smithi047 conmon[78521]: return f(*arg) 2022-05-14T03:32:45.553 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:32:45 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T03:32:45.553 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:32:45 smithi047 conmon[78521]: self._write_client_files(client_files, host) 2022-05-14T03:32:45.553 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:32:45 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T03:32:45.554 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:32:45 smithi047 conmon[78521]: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T03:32:45.554 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:32:45 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T03:32:45.554 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:32:45 smithi047 conmon[78521]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T03:32:45.554 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:32:45 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T03:32:45.555 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:32:45 smithi047 conmon[78521]: return self.event_loop.get_result(coro) 2022-05-14T03:32:45.555 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:32:45 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T03:32:45.555 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:32:45 smithi047 conmon[78521]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T03:32:45.555 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:32:45 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T03:32:45.555 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:32:45 smithi047 conmon[78521]: return self.__get_result() 2022-05-14T03:32:45.556 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:32:45 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T03:32:45.556 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:32:45 smithi047 conmon[78521]: raise self._exception 2022-05-14T03:32:45.556 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:32:45 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T03:32:45.556 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:32:45 smithi047 conmon[78521]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T03:32:45.557 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:32:45 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T03:32:45.557 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:32:45 smithi047 conmon[78521]: conn = await self._remote_connection(host, addr) 2022-05-14T03:32:45.557 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:32:45 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T03:32:45.557 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:32:45 smithi047 conmon[78521]: raise 2022-05-14T03:32:45.557 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:32:45 smithi047 conmon[78521]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T03:32:45.558 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:32:45 smithi047 conmon[78521]: self.gen.throw(type, value, traceback) 2022-05-14T03:32:45.558 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:32:45 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T03:32:45.558 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:32:45 smithi047 conmon[78521]: raise OrchestratorError(msg) 2022-05-14T03:32:45.558 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:32:45 smithi047 conmon[78521]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:32:45.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:32:45 smithi115 conmon[90393]: 2022-05-14T03:32:45.273+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:32:46.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:32:45 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:32:46.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:32:45 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:32:46.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:32:46 smithi115 conmon[90393]: 2022-05-14T03:32:46.233+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:32:47.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:32:46 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:32:47.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T03:32:47.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:32:47.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T03:32:47.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:32:47.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:32:47.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:32:47.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T03:32:47.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:32:47.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T03:32:47.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:32:47.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T03:32:47.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:32:47.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T03:32:47.049 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:32:47.049 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T03:32:47.049 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:32:47.050 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:32:47.050 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:32:47.050 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:32:46 smithi047 ceph-mon[77147]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T03:32:47.050 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T03:32:47.050 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:32:47.051 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T03:32:47.051 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:32:47.051 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:32:47.051 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:32:47.051 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T03:32:47.052 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:32:47.052 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T03:32:47.052 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:32:47.052 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T03:32:47.053 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:32:47.053 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T03:32:47.053 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:32:47.053 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T03:32:47.054 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:32:47.054 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:32:47.054 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:32:47.054 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T03:32:47.054 INFO:journalctl@ceph.mon.b.smithi047.stdout: During handling of the above exception, another exception occurred: 2022-05-14T03:32:47.055 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T03:32:47.055 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T03:32:47.055 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T03:32:47.055 INFO:journalctl@ceph.mon.b.smithi047.stdout: return f(*arg) 2022-05-14T03:32:47.056 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T03:32:47.056 INFO:journalctl@ceph.mon.b.smithi047.stdout: self._write_client_files(client_files, host) 2022-05-14T03:32:47.056 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T03:32:47.056 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T03:32:47.056 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T03:32:47.057 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T03:32:47.057 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T03:32:47.057 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.event_loop.get_result(coro) 2022-05-14T03:32:47.057 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T03:32:47.058 INFO:journalctl@ceph.mon.b.smithi047.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T03:32:47.058 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T03:32:47.058 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.__get_result() 2022-05-14T03:32:47.059 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T03:32:47.059 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise self._exception 2022-05-14T03:32:47.060 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T03:32:47.060 INFO:journalctl@ceph.mon.b.smithi047.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T03:32:47.061 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T03:32:47.061 INFO:journalctl@ceph.mon.b.smithi047.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T03:32:47.062 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T03:32:47.062 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise 2022-05-14T03:32:47.062 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T03:32:47.063 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.gen.throw(type, value, traceback) 2022-05-14T03:32:47.063 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T03:32:47.064 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OrchestratorError(msg) 2022-05-14T03:32:47.064 INFO:journalctl@ceph.mon.b.smithi047.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:32:47.065 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:32:46 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:32:47.065 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:32:46 smithi047 ceph-mon[77147]: pgmap v7701: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:32:47.066 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:32:46 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 14695 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:32:47.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:32:46 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:32:47.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T03:32:47.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:32:47.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T03:32:47.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:32:47.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:32:47.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:32:47.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T03:32:47.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:32:47.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T03:32:47.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:32:47.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T03:32:47.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:32:47.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T03:32:47.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:32:47.113 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T03:32:47.113 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:32:47.113 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:32:47.113 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:32:47.114 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:32:46 smithi115 ceph-mon[78593]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T03:32:47.114 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T03:32:47.114 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:32:47.114 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T03:32:47.115 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:32:47.115 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:32:47.115 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:32:47.115 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T03:32:47.115 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:32:47.116 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T03:32:47.116 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:32:47.116 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T03:32:47.116 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:32:47.117 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T03:32:47.117 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:32:47.117 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T03:32:47.117 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:32:47.117 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:32:47.118 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:32:47.118 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T03:32:47.118 INFO:journalctl@ceph.mon.c.smithi115.stdout: During handling of the above exception, another exception occurred: 2022-05-14T03:32:47.118 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T03:32:47.118 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T03:32:47.119 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T03:32:47.119 INFO:journalctl@ceph.mon.c.smithi115.stdout: return f(*arg) 2022-05-14T03:32:47.119 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T03:32:47.119 INFO:journalctl@ceph.mon.c.smithi115.stdout: self._write_client_files(client_files, host) 2022-05-14T03:32:47.120 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T03:32:47.120 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T03:32:47.120 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T03:32:47.120 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T03:32:47.120 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T03:32:47.121 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.event_loop.get_result(coro) 2022-05-14T03:32:47.121 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T03:32:47.121 INFO:journalctl@ceph.mon.c.smithi115.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T03:32:47.121 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T03:32:47.122 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.__get_result() 2022-05-14T03:32:47.122 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T03:32:47.122 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise self._exception 2022-05-14T03:32:47.122 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T03:32:47.122 INFO:journalctl@ceph.mon.c.smithi115.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T03:32:47.123 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T03:32:47.124 INFO:journalctl@ceph.mon.c.smithi115.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T03:32:47.124 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T03:32:47.124 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise 2022-05-14T03:32:47.124 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T03:32:47.125 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.gen.throw(type, value, traceback) 2022-05-14T03:32:47.125 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T03:32:47.125 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OrchestratorError(msg) 2022-05-14T03:32:47.125 INFO:journalctl@ceph.mon.c.smithi115.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:32:47.126 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:32:46 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:32:47.126 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:32:46 smithi115 ceph-mon[78593]: pgmap v7701: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:32:47.126 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:32:46 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 14695 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:32:47.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:32:47 smithi115 conmon[90393]: 2022-05-14T03:32:47.250+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:32:48.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:32:47 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:32:48.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:32:47 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:32:48.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:32:48 smithi115 conmon[90393]: 2022-05-14T03:32:48.204+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:32:49.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:32:48 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:32:49.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:32:48 smithi047 ceph-mon[77147]: pgmap v7702: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:32:49.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:32:48 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:32:49.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:32:48 smithi115 ceph-mon[78593]: pgmap v7702: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:32:49.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:32:49 smithi115 conmon[90393]: 2022-05-14T03:32:49.193+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:32:50.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:32:49 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:32:50.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:32:49 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:32:50.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:32:50 smithi115 conmon[90393]: 2022-05-14T03:32:50.209+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:32:51.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:32:50 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:32:51.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:32:50 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:32:51.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:32:50 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:32:51.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:32:50 smithi047 ceph-mon[77147]: pgmap v7703: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:32:51.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:32:50 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:32:51.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:32:50 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:32:51.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:32:50 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:32:51.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:32:50 smithi115 ceph-mon[78593]: pgmap v7703: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:32:51.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:32:51 smithi115 conmon[90393]: 2022-05-14T03:32:51.222+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:32:52.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:32:51 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:32:52.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:32:51 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 14705 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:32:52.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:32:51 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:32:52.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:32:51 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 14705 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:32:52.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:32:52 smithi115 conmon[90393]: 2022-05-14T03:32:52.173+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:32:53.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:32:52 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:32:53.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:32:52 smithi047 ceph-mon[77147]: pgmap v7704: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:32:53.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:32:52 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:32:53.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:32:52 smithi115 ceph-mon[78593]: pgmap v7704: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:32:53.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:32:53 smithi115 conmon[90393]: 2022-05-14T03:32:53.134+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:32:54.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:32:53 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:32:54.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:32:53 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:32:54.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:32:54 smithi115 conmon[90393]: 2022-05-14T03:32:54.104+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:32:55.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:32:54 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:32:55.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:32:54 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:32:55.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:32:54 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:32:55.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:32:54 smithi047 ceph-mon[77147]: pgmap v7705: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:32:55.099 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:32:54 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:32:55.099 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:32:54 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:32:55.099 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:32:54 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:32:55.100 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:32:54 smithi115 ceph-mon[78593]: pgmap v7705: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:32:55.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:32:55 smithi115 conmon[90393]: 2022-05-14T03:32:55.096+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:32:56.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:32:55 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:32:56.102 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:32:55 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:32:56.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:32:56 smithi115 conmon[90393]: 2022-05-14T03:32:56.099+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:32:57.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:32:56 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:32:57.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:32:56 smithi047 ceph-mon[77147]: pgmap v7706: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:32:57.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:32:56 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:32:57.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:32:56 smithi115 ceph-mon[78593]: pgmap v7706: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:32:57.610 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:32:57 smithi115 conmon[90393]: 2022-05-14T03:32:57.140+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:32:58.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:32:57 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:32:58.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:32:57 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 14711 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:32:58.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:32:57 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:32:58.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:32:57 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 14711 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:32:58.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:32:58 smithi115 conmon[90393]: 2022-05-14T03:32:58.121+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:32:59.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:32:58 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:32:59.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:32:58 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:32:59.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:32:58 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:32:59.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:32:58 smithi047 ceph-mon[77147]: pgmap v7707: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:32:59.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:32:58 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:32:59.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:32:58 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:32:59.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:32:58 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:32:59.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:32:58 smithi115 ceph-mon[78593]: pgmap v7707: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:32:59.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:32:59 smithi115 conmon[90393]: 2022-05-14T03:32:59.139+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:33:00.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:32:59 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:33:00.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:32:59 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:33:00.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:33:00 smithi115 conmon[90393]: 2022-05-14T03:33:00.136+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:33:01.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:33:00 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:33:01.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:33:00 smithi047 ceph-mon[77147]: pgmap v7708: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:33:01.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:33:00 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:33:01.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:33:00 smithi115 ceph-mon[78593]: pgmap v7708: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:33:01.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:33:01 smithi115 conmon[90393]: 2022-05-14T03:33:01.125+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:33:02.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:33:01 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:33:02.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:33:01 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:33:02.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:33:02 smithi115 conmon[90393]: 2022-05-14T03:33:02.163+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:33:03.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:33:02 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:33:03.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:33:02 smithi047 ceph-mon[77147]: pgmap v7709: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:33:03.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:33:02 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:33:03.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:33:02 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:33:03.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:33:02 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:33:03.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:33:02 smithi115 ceph-mon[78593]: pgmap v7709: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:33:03.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:33:02 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:33:03.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:33:02 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:33:03.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:33:03 smithi115 conmon[90393]: 2022-05-14T03:33:03.188+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:33:04.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:33:03 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:33:04.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:33:03 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:33:04.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:33:04 smithi115 conmon[90393]: 2022-05-14T03:33:04.192+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:33:05.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:33:04 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:33:05.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:33:04 smithi047 ceph-mon[77147]: pgmap v7710: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:33:05.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:33:04 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:33:05.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:33:04 smithi115 ceph-mon[78593]: pgmap v7710: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:33:05.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:33:05 smithi115 conmon[90393]: 2022-05-14T03:33:05.188+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:33:06.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:33:05 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:33:06.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:33:05 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:33:06.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:33:06 smithi115 conmon[90393]: 2022-05-14T03:33:06.179+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:33:07.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:33:06 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:33:07.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:33:06 smithi047 ceph-mon[77147]: pgmap v7711: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:33:07.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:33:06 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 14720 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:33:07.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:33:06 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:33:07.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:33:06 smithi115 ceph-mon[78593]: pgmap v7711: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:33:07.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:33:06 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 14720 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:33:07.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:33:07 smithi115 conmon[90393]: 2022-05-14T03:33:07.193+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:33:08.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:33:07 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:33:08.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:33:07 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:33:08.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:33:08 smithi115 conmon[90393]: 2022-05-14T03:33:08.197+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:33:09.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:33:08 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:33:09.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:33:08 smithi047 ceph-mon[77147]: pgmap v7712: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:33:09.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:33:08 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:33:09.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:33:08 smithi115 ceph-mon[78593]: pgmap v7712: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:33:09.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:33:09 smithi115 conmon[90393]: 2022-05-14T03:33:09.228+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:33:10.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:33:09 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:33:10.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:33:09 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:33:10.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:33:10 smithi115 conmon[90393]: 2022-05-14T03:33:10.276+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:33:11.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:33:10 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:33:11.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:33:10 smithi047 ceph-mon[77147]: pgmap v7713: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:33:11.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:33:10 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:33:11.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:33:10 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:33:11.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:33:10 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:33:11.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:33:10 smithi115 ceph-mon[78593]: pgmap v7713: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:33:11.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:33:10 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:33:11.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:33:10 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:33:11.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:33:11 smithi115 conmon[90393]: 2022-05-14T03:33:11.256+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:33:12.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:33:11 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:33:12.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:33:11 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 14725 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:33:12.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:33:11 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:33:12.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:33:11 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 14725 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:33:12.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:33:12 smithi115 conmon[90393]: 2022-05-14T03:33:12.290+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:33:13.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:33:12 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:33:13.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:33:12 smithi047 ceph-mon[77147]: pgmap v7714: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:33:13.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:33:12 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:33:13.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:33:12 smithi115 ceph-mon[78593]: pgmap v7714: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:33:13.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:33:13 smithi115 conmon[90393]: 2022-05-14T03:33:13.289+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:33:14.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:33:13 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:33:14.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:33:13 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:33:14.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:33:14 smithi115 conmon[90393]: 2022-05-14T03:33:14.278+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:33:15.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:33:14 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:33:15.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:33:14 smithi047 ceph-mon[77147]: pgmap v7715: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:33:15.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:33:14 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:33:15.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:33:14 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:33:15.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:33:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T03:33:15.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:33:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T03:33:15.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:33:14 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:33:15.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:33:14 smithi115 ceph-mon[78593]: pgmap v7715: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:33:15.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:33:14 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:33:15.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:33:14 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:33:15.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:33:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T03:33:15.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:33:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T03:33:15.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:33:15 smithi115 conmon[90393]: 2022-05-14T03:33:15.304+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:33:16.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:33:15 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:33:16.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:33:15 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:33:16.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:33:16 smithi115 conmon[90393]: 2022-05-14T03:33:16.269+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:33:17.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:33:16 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:33:17.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:33:16 smithi047 ceph-mon[77147]: pgmap v7716: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 6 op/s 2022-05-14T03:33:17.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:33:16 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 14730 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:33:17.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:33:16 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:33:17.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:33:16 smithi115 ceph-mon[78593]: pgmap v7716: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 6 op/s 2022-05-14T03:33:17.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:33:16 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 14730 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:33:17.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:33:17 smithi115 conmon[90393]: 2022-05-14T03:33:17.230+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:33:18.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:33:17 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:33:18.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:33:17 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:33:18.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:33:18 smithi115 conmon[90393]: 2022-05-14T03:33:18.187+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:33:19.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:33:18 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:33:19.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:33:18 smithi047 ceph-mon[77147]: pgmap v7717: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 4 op/s 2022-05-14T03:33:19.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:33:18 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:33:19.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:33:18 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:33:19.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:33:18 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:33:19.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:33:18 smithi115 ceph-mon[78593]: pgmap v7717: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 4 op/s 2022-05-14T03:33:19.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:33:18 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:33:19.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:33:18 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:33:19.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:33:19 smithi115 conmon[90393]: 2022-05-14T03:33:19.150+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:33:20.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:33:19 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:33:20.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:33:19 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:33:20.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:33:20 smithi115 conmon[90393]: 2022-05-14T03:33:20.150+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:33:21.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:33:20 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:33:21.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:33:20 smithi115 ceph-mon[78593]: pgmap v7718: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 7 op/s 2022-05-14T03:33:21.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:33:20 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:33:21.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:33:20 smithi047 ceph-mon[77147]: pgmap v7718: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 7 op/s 2022-05-14T03:33:21.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:33:21 smithi115 conmon[90393]: 2022-05-14T03:33:21.188+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:33:22.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:33:21 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:33:22.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:33:21 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 14735 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:33:22.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:33:21 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:33:22.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:33:21 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 14735 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:33:22.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:33:22 smithi115 conmon[90393]: 2022-05-14T03:33:22.199+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:33:23.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:33:22 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:33:23.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:33:22 smithi115 ceph-mon[78593]: pgmap v7719: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 4 op/s 2022-05-14T03:33:23.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:33:22 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:33:23.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:33:22 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:33:23.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:33:22 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:33:23.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:33:22 smithi047 ceph-mon[77147]: pgmap v7719: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 4 op/s 2022-05-14T03:33:23.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:33:22 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:33:23.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:33:22 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:33:23.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:33:23 smithi115 conmon[90393]: 2022-05-14T03:33:23.220+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:33:24.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:33:23 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:33:24.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:33:23 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:33:24.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:33:24 smithi115 conmon[90393]: 2022-05-14T03:33:24.193+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:33:25.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:33:24 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:33:25.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:33:24 smithi115 ceph-mon[78593]: pgmap v7720: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 4 op/s 2022-05-14T03:33:25.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:33:24 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:33:25.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:33:24 smithi047 ceph-mon[77147]: pgmap v7720: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 4 op/s 2022-05-14T03:33:25.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:33:25 smithi115 conmon[90393]: 2022-05-14T03:33:25.182+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:33:26.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:33:25 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:33:26.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:33:25 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:33:26.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:33:26 smithi115 conmon[90393]: 2022-05-14T03:33:26.154+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:33:27.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:33:26 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:33:27.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:33:26 smithi115 ceph-mon[78593]: pgmap v7721: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 7 op/s 2022-05-14T03:33:27.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:33:26 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 14740 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:33:27.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:33:26 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:33:27.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:33:26 smithi047 ceph-mon[77147]: pgmap v7721: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 7 op/s 2022-05-14T03:33:27.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:33:26 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 14740 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:33:27.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:33:27 smithi115 conmon[90393]: 2022-05-14T03:33:27.138+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:33:28.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:33:27 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:33:28.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:33:27 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:33:28.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:33:28 smithi115 conmon[90393]: 2022-05-14T03:33:28.136+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:33:29.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:33:28 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:33:29.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:33:28 smithi115 ceph-mon[78593]: pgmap v7722: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:33:29.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:33:28 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:33:29.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:33:28 smithi047 ceph-mon[77147]: pgmap v7722: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:33:29.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:33:29 smithi115 conmon[90393]: 2022-05-14T03:33:29.155+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:33:30.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:33:29 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:33:30.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:33:29 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:33:30.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:33:30 smithi115 conmon[90393]: 2022-05-14T03:33:30.129+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:33:31.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:33:30 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:33:31.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:33:30 smithi115 ceph-mon[78593]: pgmap v7723: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:33:31.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:33:30 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:33:31.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:33:30 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:33:31.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:33:30 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:33:31.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:33:30 smithi047 ceph-mon[77147]: pgmap v7723: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:33:31.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:33:30 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:33:31.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:33:30 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:33:31.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:33:31 smithi115 conmon[90393]: 2022-05-14T03:33:31.143+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:33:32.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:33:31 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:33:32.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:33:31 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 14745 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:33:32.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:33:31 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:33:32.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:33:31 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 14745 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:33:32.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:33:32 smithi115 conmon[90393]: 2022-05-14T03:33:32.124+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:33:33.091 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:33:32 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:33:33.091 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:33:32 smithi115 ceph-mon[78593]: pgmap v7724: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:33:33.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:33:32 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:33:33.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:33:32 smithi047 ceph-mon[77147]: pgmap v7724: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:33:33.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:33:33 smithi115 conmon[90393]: 2022-05-14T03:33:33.088+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:33:34.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:33:33 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:33:34.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:33:34 smithi115 conmon[90393]: 2022-05-14T03:33:34.068+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:33:34.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:33:33 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:33:35.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:33:35 smithi115 conmon[90393]: 2022-05-14T03:33:35.072+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:33:35.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:33:34 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:33:35.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:33:34 smithi115 ceph-mon[78593]: pgmap v7725: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:33:35.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:33:34 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:33:35.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:33:34 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:33:35.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:33:34 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:33:35.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:33:34 smithi047 ceph-mon[77147]: pgmap v7725: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:33:35.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:33:34 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:33:35.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:33:34 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:33:36.101 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:33:35 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:33:36.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:33:35 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:33:36.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:33:36 smithi115 conmon[90393]: 2022-05-14T03:33:36.098+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:33:37.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:33:36 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:33:37.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:33:36 smithi115 ceph-mon[78593]: pgmap v7726: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:33:37.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:33:36 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 14750 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:33:37.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:33:36 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:33:37.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:33:36 smithi047 ceph-mon[77147]: pgmap v7726: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:33:37.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:33:36 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 14750 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:33:37.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:33:37 smithi115 conmon[90393]: 2022-05-14T03:33:37.145+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:33:38.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:33:37 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:33:38.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:33:37 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:33:38.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:33:38 smithi115 conmon[90393]: 2022-05-14T03:33:38.194+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:33:39.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:33:38 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:33:39.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:33:38 smithi115 ceph-mon[78593]: pgmap v7727: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:33:39.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:33:38 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:33:39.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:33:38 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:33:39.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:33:38 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:33:39.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:33:38 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:33:39.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:33:38 smithi047 ceph-mon[77147]: pgmap v7727: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:33:39.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:33:38 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:33:39.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:33:38 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:33:39.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:33:38 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:33:39.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:33:39 smithi115 conmon[90393]: 2022-05-14T03:33:39.242+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:33:40.268 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:33:39 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:33:40.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:33:39 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:33:40.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:33:40 smithi115 conmon[90393]: 2022-05-14T03:33:40.266+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:33:41.237 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:33:40 smithi115 ceph-mon[78593]: pgmap v7728: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:33:41.237 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:33:40 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:33:41.237 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:33:41 smithi115 conmon[90393]: 2022-05-14T03:33:41.234+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:33:41.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:33:40 smithi047 ceph-mon[77147]: pgmap v7728: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:33:41.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:33:40 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:33:42.217 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:33:41 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:33:42.217 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:33:41 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 14755 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:33:42.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:33:41 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:33:42.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:33:41 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 14755 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:33:42.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:33:42 smithi115 conmon[90393]: 2022-05-14T03:33:42.214+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:33:43.192 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:33:42 smithi115 ceph-mon[78593]: pgmap v7729: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:33:43.193 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:33:42 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:33:43.193 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:33:42 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:33:43.193 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:33:42 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:33:43.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:33:42 smithi047 ceph-mon[77147]: pgmap v7729: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:33:43.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:33:42 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:33:43.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:33:42 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:33:43.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:33:42 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:33:43.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:33:43 smithi115 conmon[90393]: 2022-05-14T03:33:43.189+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:33:44.184 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:33:43 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:33:44.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:33:43 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:33:44.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:33:44 smithi115 conmon[90393]: 2022-05-14T03:33:44.181+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:33:45.145 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:33:44 smithi047 ceph-mon[77147]: pgmap v7730: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:33:45.146 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:33:44 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:33:45.146 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:33:44 smithi115 ceph-mon[78593]: pgmap v7730: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:33:45.146 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:33:44 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:33:45.552 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:33:45 smithi115 conmon[90393]: 2022-05-14T03:33:45.230+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:33:45.883 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:33:45 smithi047 conmon[78521]: 2022-05-14T03:33:45.647+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:33:45.884 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:33:45 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T03:33:45.884 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:33:45 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:33:45.884 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:33:45 smithi047 conmon[78521]: yield 2022-05-14T03:33:45.885 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:33:45 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:33:45.885 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:33:45 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:33:45.885 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:33:45 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:33:45.885 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:33:45 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T03:33:45.886 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:33:45 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:33:45.886 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:33:45 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T03:33:45.886 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:33:45 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:33:45.886 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:33:45 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T03:33:45.886 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:33:45 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:33:45.887 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:33:45 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T03:33:45.887 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:33:45 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:33:45.887 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:33:45 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T03:33:45.887 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:33:45 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:33:45.888 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:33:45 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:33:45.888 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:33:45 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:33:46.259 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:33:45 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T03:33:46.260 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:33:45 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T03:33:46.261 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:33:45 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T03:33:46.261 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:33:45 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:33:46.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:33:45 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T03:33:46.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:33:45 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T03:33:46.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:33:45 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T03:33:46.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:33:45 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:33:46.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:33:46 smithi115 conmon[90393]: 2022-05-14T03:33:46.257+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:33:47.225 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:33:46 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:33:47.226 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T03:33:47.226 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:33:47.226 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T03:33:47.226 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:33:47.227 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:33:47.227 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:33:47.227 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T03:33:47.228 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:33:47.228 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T03:33:47.229 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:33:47.229 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T03:33:47.229 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:33:47.229 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T03:33:47.230 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:33:47.230 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T03:33:47.230 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:33:47.230 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:33:47.230 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:33:47.231 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:33:46 smithi115 ceph-mon[78593]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T03:33:47.231 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:33:46 smithi115 ceph-mon[78593]: pgmap v7731: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:33:47.231 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:33:46 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:33:47.231 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:33:46 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 14760 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:33:47.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:33:46 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:33:47.292 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T03:33:47.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:33:47.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T03:33:47.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:33:47.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:33:47.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:33:47.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T03:33:47.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:33:47.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T03:33:47.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:33:47.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T03:33:47.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:33:47.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T03:33:47.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:33:47.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T03:33:47.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:33:47.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:33:47.298 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:33:47.298 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:33:46 smithi047 ceph-mon[77147]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T03:33:47.299 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:33:46 smithi047 ceph-mon[77147]: pgmap v7731: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:33:47.299 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:33:46 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:33:47.299 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:33:46 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 14760 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:33:47.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:33:47 smithi115 conmon[90393]: 2022-05-14T03:33:47.222+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:33:48.233 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:33:47 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:33:48.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:33:47 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:33:48.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:33:48 smithi115 conmon[90393]: 2022-05-14T03:33:48.230+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:33:49.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:33:48 smithi047 ceph-mon[77147]: pgmap v7732: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:33:49.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:33:48 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:33:49.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:33:48 smithi047 conmon[78521]: 2022-05-14T03:33:48.720+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:33:49.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:33:48 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T03:33:49.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:33:48 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:33:49.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:33:48 smithi047 conmon[78521]: yield 2022-05-14T03:33:49.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:33:48 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:33:49.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:33:48 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:33:49.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:33:48 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:33:49.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:33:48 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T03:33:49.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:33:48 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:33:49.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:33:48 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T03:33:49.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:33:48 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:33:49.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:33:48 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T03:33:49.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:33:48 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:33:49.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:33:48 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T03:33:49.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:33:48 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:33:49.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:33:48 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T03:33:49.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:33:48 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:33:49.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:33:48 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:33:49.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:33:48 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:33:49.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:33:48 smithi047 conmon[78521]: 2022-05-14T03:33:48.721+0000 7f5344f82700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T03:33:49.048 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:33:48 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T03:33:49.048 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:33:48 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:33:49.048 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:33:48 smithi047 conmon[78521]: yield 2022-05-14T03:33:49.048 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:33:48 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:33:49.048 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:33:48 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:33:49.049 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:33:48 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:33:49.049 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:33:48 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T03:33:49.049 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:33:48 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:33:49.049 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:33:48 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T03:33:49.050 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:33:48 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:33:49.050 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:33:48 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T03:33:49.050 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:33:48 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:33:49.050 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:33:48 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T03:33:49.050 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:33:48 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:33:49.051 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:33:48 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T03:33:49.051 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:33:48 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:33:49.051 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:33:48 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:33:49.051 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:33:48 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:33:49.052 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:33:48 smithi047 conmon[78521]: 2022-05-14T03:33:49.052 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:33:48 smithi047 conmon[78521]: During handling of the above exception, another exception occurred: 2022-05-14T03:33:49.052 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:33:48 smithi047 conmon[78521]: 2022-05-14T03:33:49.052 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:33:48 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T03:33:49.052 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:33:48 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T03:33:49.053 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:33:48 smithi047 conmon[78521]: return f(*arg) 2022-05-14T03:33:49.053 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:33:48 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T03:33:49.053 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:33:48 smithi047 conmon[78521]: self._write_client_files(client_files, host) 2022-05-14T03:33:49.053 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:33:48 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T03:33:49.054 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:33:48 smithi047 conmon[78521]: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T03:33:49.054 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:33:48 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T03:33:49.054 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:33:48 smithi047 conmon[78521]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T03:33:49.054 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:33:48 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T03:33:49.054 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:33:48 smithi047 conmon[78521]: return self.event_loop.get_result(coro) 2022-05-14T03:33:49.055 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:33:48 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T03:33:49.055 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:33:48 smithi047 conmon[78521]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T03:33:49.055 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:33:48 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T03:33:49.055 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:33:48 smithi047 conmon[78521]: return self.__get_result() 2022-05-14T03:33:49.055 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:33:48 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T03:33:49.056 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:33:48 smithi047 conmon[78521]: raise self._exception 2022-05-14T03:33:49.056 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:33:48 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T03:33:49.056 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:33:48 smithi047 conmon[78521]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T03:33:49.056 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:33:48 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T03:33:49.057 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:33:48 smithi047 conmon[78521]: conn = await self._remote_connection(host, addr) 2022-05-14T03:33:49.057 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:33:48 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T03:33:49.057 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:33:48 smithi047 conmon[78521]: raise 2022-05-14T03:33:49.057 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:33:48 smithi047 conmon[78521]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T03:33:49.057 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:33:48 smithi047 conmon[78521]: self.gen.throw(type, value, traceback) 2022-05-14T03:33:49.058 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:33:48 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T03:33:49.058 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:33:48 smithi047 conmon[78521]: raise OrchestratorError(msg) 2022-05-14T03:33:49.058 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:33:48 smithi047 conmon[78521]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:33:49.201 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:33:48 smithi115 ceph-mon[78593]: pgmap v7732: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:33:49.201 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:33:48 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:33:49.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:33:49 smithi115 conmon[90393]: 2022-05-14T03:33:49.199+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:33:50.175 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:33:49 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:33:50.176 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T03:33:50.176 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:33:50.176 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T03:33:50.177 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:33:50.177 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:33:50.177 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:33:50.177 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T03:33:50.178 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:33:50.178 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T03:33:50.178 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:33:50.178 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T03:33:50.179 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:33:50.179 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T03:33:50.179 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:33:50.179 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T03:33:50.179 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:33:50.180 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:33:50.180 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:33:50.180 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:33:49 smithi115 ceph-mon[78593]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T03:33:50.180 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T03:33:50.181 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:33:50.181 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T03:33:50.181 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:33:50.182 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:33:50.182 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:33:50.182 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T03:33:50.182 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:33:50.183 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T03:33:50.183 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:33:50.183 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T03:33:50.183 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:33:50.184 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T03:33:50.184 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:33:50.184 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T03:33:50.184 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:33:50.185 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:33:50.185 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:33:50.185 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T03:33:50.185 INFO:journalctl@ceph.mon.c.smithi115.stdout: During handling of the above exception, another exception occurred: 2022-05-14T03:33:50.186 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T03:33:50.186 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T03:33:50.186 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T03:33:50.187 INFO:journalctl@ceph.mon.c.smithi115.stdout: return f(*arg) 2022-05-14T03:33:50.187 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T03:33:50.187 INFO:journalctl@ceph.mon.c.smithi115.stdout: self._write_client_files(client_files, host) 2022-05-14T03:33:50.187 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T03:33:50.188 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T03:33:50.188 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T03:33:50.188 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T03:33:50.188 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T03:33:50.189 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.event_loop.get_result(coro) 2022-05-14T03:33:50.189 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T03:33:50.189 INFO:journalctl@ceph.mon.c.smithi115.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T03:33:50.189 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T03:33:50.190 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.__get_result() 2022-05-14T03:33:50.190 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T03:33:50.190 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise self._exception 2022-05-14T03:33:50.190 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T03:33:50.191 INFO:journalctl@ceph.mon.c.smithi115.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T03:33:50.191 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T03:33:50.191 INFO:journalctl@ceph.mon.c.smithi115.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T03:33:50.191 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T03:33:50.192 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise 2022-05-14T03:33:50.192 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T03:33:50.192 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.gen.throw(type, value, traceback) 2022-05-14T03:33:50.192 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T03:33:50.192 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OrchestratorError(msg) 2022-05-14T03:33:50.193 INFO:journalctl@ceph.mon.c.smithi115.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:33:50.193 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:33:49 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:33:50.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:33:49 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:33:50.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T03:33:50.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:33:50.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T03:33:50.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:33:50.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:33:50.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:33:50.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T03:33:50.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:33:50.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T03:33:50.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:33:50.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T03:33:50.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:33:50.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T03:33:50.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:33:50.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T03:33:50.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:33:50.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:33:50.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:33:50.297 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:33:49 smithi047 ceph-mon[77147]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T03:33:50.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T03:33:50.298 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:33:50.298 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T03:33:50.298 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:33:50.298 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:33:50.299 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:33:50.299 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T03:33:50.299 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:33:50.299 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T03:33:50.300 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:33:50.300 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T03:33:50.300 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:33:50.300 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T03:33:50.300 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:33:50.301 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T03:33:50.301 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:33:50.301 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:33:50.301 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:33:50.302 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T03:33:50.302 INFO:journalctl@ceph.mon.b.smithi047.stdout: During handling of the above exception, another exception occurred: 2022-05-14T03:33:50.302 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T03:33:50.302 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T03:33:50.303 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T03:33:50.303 INFO:journalctl@ceph.mon.b.smithi047.stdout: return f(*arg) 2022-05-14T03:33:50.303 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T03:33:50.303 INFO:journalctl@ceph.mon.b.smithi047.stdout: self._write_client_files(client_files, host) 2022-05-14T03:33:50.303 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T03:33:50.304 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T03:33:50.304 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T03:33:50.304 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T03:33:50.304 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T03:33:50.304 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.event_loop.get_result(coro) 2022-05-14T03:33:50.305 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T03:33:50.305 INFO:journalctl@ceph.mon.b.smithi047.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T03:33:50.305 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T03:33:50.305 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.__get_result() 2022-05-14T03:33:50.306 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T03:33:50.306 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise self._exception 2022-05-14T03:33:50.306 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T03:33:50.306 INFO:journalctl@ceph.mon.b.smithi047.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T03:33:50.306 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T03:33:50.307 INFO:journalctl@ceph.mon.b.smithi047.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T03:33:50.307 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T03:33:50.307 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise 2022-05-14T03:33:50.307 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T03:33:50.308 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.gen.throw(type, value, traceback) 2022-05-14T03:33:50.308 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T03:33:50.308 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OrchestratorError(msg) 2022-05-14T03:33:50.308 INFO:journalctl@ceph.mon.b.smithi047.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:33:50.308 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:33:49 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:33:50.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:33:50 smithi115 conmon[90393]: 2022-05-14T03:33:50.172+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:33:51.150 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:33:50 smithi115 ceph-mon[78593]: pgmap v7733: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:33:51.151 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:33:50 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:33:51.151 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:33:50 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:33:51.151 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:33:50 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:33:51.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:33:50 smithi047 ceph-mon[77147]: pgmap v7733: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:33:51.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:33:50 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:33:51.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:33:50 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:33:51.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:33:50 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:33:51.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:33:51 smithi115 conmon[90393]: 2022-05-14T03:33:51.147+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:33:52.166 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:33:51 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:33:52.166 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:33:51 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 14765 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:33:52.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:33:51 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:33:52.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:33:51 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 14765 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:33:52.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:33:52 smithi115 conmon[90393]: 2022-05-14T03:33:52.163+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:33:53.170 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:33:52 smithi115 ceph-mon[78593]: pgmap v7734: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:33:53.170 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:33:52 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:33:53.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:33:52 smithi047 ceph-mon[77147]: pgmap v7734: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:33:53.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:33:52 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:33:53.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:33:53 smithi115 conmon[90393]: 2022-05-14T03:33:53.167+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:33:54.220 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:33:53 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:33:54.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:33:53 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:33:54.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:33:54 smithi115 conmon[90393]: 2022-05-14T03:33:54.217+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:33:55.195 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:33:54 smithi115 ceph-mon[78593]: pgmap v7735: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:33:55.196 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:33:54 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:33:55.196 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:33:54 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:33:55.196 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:33:54 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:33:55.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:33:54 smithi047 ceph-mon[77147]: pgmap v7735: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:33:55.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:33:54 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:33:55.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:33:54 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:33:55.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:33:54 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:33:55.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:33:55 smithi115 conmon[90393]: 2022-05-14T03:33:55.193+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:33:56.200 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:33:55 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:33:56.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:33:55 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:33:56.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:33:56 smithi115 conmon[90393]: 2022-05-14T03:33:56.198+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:33:57.226 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:33:56 smithi115 ceph-mon[78593]: pgmap v7736: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:33:57.226 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:33:56 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:33:57.226 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:33:56 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 14770 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:33:57.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:33:56 smithi047 ceph-mon[77147]: pgmap v7736: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:33:57.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:33:56 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:33:57.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:33:56 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 14770 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:33:57.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:33:57 smithi115 conmon[90393]: 2022-05-14T03:33:57.224+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:33:58.214 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:33:57 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:33:58.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:33:57 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:33:58.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:33:58 smithi115 conmon[90393]: 2022-05-14T03:33:58.211+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:33:59.235 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:33:58 smithi115 ceph-mon[78593]: pgmap v7737: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:33:59.235 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:33:58 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:33:59.235 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:33:58 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:33:59.236 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:33:58 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:33:59.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:33:58 smithi047 ceph-mon[77147]: pgmap v7737: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:33:59.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:33:58 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:33:59.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:33:58 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:33:59.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:33:58 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:33:59.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:33:59 smithi115 conmon[90393]: 2022-05-14T03:33:59.232+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:34:00.249 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:33:59 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:34:00.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:33:59 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:34:00.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:34:00 smithi115 conmon[90393]: 2022-05-14T03:34:00.246+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:34:01.270 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:34:00 smithi115 ceph-mon[78593]: pgmap v7738: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:34:01.270 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:34:00 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:34:01.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:34:00 smithi047 ceph-mon[77147]: pgmap v7738: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:34:01.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:34:00 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:34:01.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:34:01 smithi115 conmon[90393]: 2022-05-14T03:34:01.268+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:34:02.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:34:01 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:34:02.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:34:01 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 14775 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:34:02.305 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:34:01 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:34:02.305 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:34:01 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 14775 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:34:02.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:34:02 smithi115 conmon[90393]: 2022-05-14T03:34:02.303+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:34:03.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:34:02 smithi047 ceph-mon[77147]: pgmap v7739: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:34:03.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:34:02 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:34:03.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:34:02 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:34:03.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:34:02 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:34:03.337 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:34:02 smithi115 ceph-mon[78593]: pgmap v7739: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:34:03.338 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:34:02 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:34:03.338 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:34:02 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:34:03.338 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:34:02 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:34:03.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:34:03 smithi115 conmon[90393]: 2022-05-14T03:34:03.335+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:34:04.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:34:03 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:34:04.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:34:03 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:34:04.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:34:04 smithi115 conmon[90393]: 2022-05-14T03:34:04.361+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:34:05.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:34:04 smithi047 ceph-mon[77147]: pgmap v7740: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:34:05.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:34:04 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:34:05.334 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:34:04 smithi115 ceph-mon[78593]: pgmap v7740: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:34:05.334 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:34:04 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:34:05.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:34:05 smithi115 conmon[90393]: 2022-05-14T03:34:05.332+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:34:06.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:34:05 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:34:06.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:34:05 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:34:06.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:34:06 smithi115 conmon[90393]: 2022-05-14T03:34:06.355+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:34:07.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:34:06 smithi047 ceph-mon[77147]: pgmap v7741: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:34:07.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:34:06 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:34:07.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:34:06 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 14780 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:34:07.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:34:06 smithi115 ceph-mon[78593]: pgmap v7741: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:34:07.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:34:06 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:34:07.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:34:06 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 14780 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:34:07.860 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:34:07 smithi115 conmon[90393]: 2022-05-14T03:34:07.367+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:34:08.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:34:07 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:34:08.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:34:07 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:34:08.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:34:08 smithi115 conmon[90393]: 2022-05-14T03:34:08.409+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:34:09.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:34:08 smithi047 ceph-mon[77147]: pgmap v7742: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:34:09.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:34:08 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:34:09.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:34:08 smithi115 ceph-mon[78593]: pgmap v7742: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:34:09.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:34:08 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:34:09.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:34:09 smithi115 conmon[90393]: 2022-05-14T03:34:09.437+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:34:10.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:34:09 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:34:10.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:34:09 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:34:10.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:34:10 smithi115 conmon[90393]: 2022-05-14T03:34:10.414+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:34:11.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:34:10 smithi047 ceph-mon[77147]: pgmap v7743: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:34:11.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:34:10 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:34:11.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:34:10 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:34:11.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:34:10 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:34:11.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:34:10 smithi115 ceph-mon[78593]: pgmap v7743: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:34:11.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:34:10 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:34:11.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:34:10 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:34:11.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:34:10 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:34:11.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:34:11 smithi115 conmon[90393]: 2022-05-14T03:34:11.409+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:34:12.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:34:11 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:34:12.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:34:11 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 14785 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:34:12.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:34:11 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:34:12.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:34:11 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 14785 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:34:12.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:34:12 smithi115 conmon[90393]: 2022-05-14T03:34:12.416+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:34:13.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:34:12 smithi047 ceph-mon[77147]: pgmap v7744: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:34:13.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:34:12 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:34:13.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:34:12 smithi115 ceph-mon[78593]: pgmap v7744: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:34:13.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:34:12 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:34:13.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:34:13 smithi115 conmon[90393]: 2022-05-14T03:34:13.449+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:34:14.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:34:13 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:34:14.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:34:13 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:34:14.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:34:14 smithi115 conmon[90393]: 2022-05-14T03:34:14.436+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:34:15.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:34:14 smithi047 ceph-mon[77147]: pgmap v7745: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:34:15.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:34:14 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:34:15.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:34:14 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:34:15.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:34:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T03:34:15.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:34:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T03:34:15.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:34:14 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:34:15.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:34:14 smithi115 ceph-mon[78593]: pgmap v7745: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:34:15.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:34:14 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:34:15.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:34:14 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:34:15.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:34:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T03:34:15.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:34:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T03:34:15.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:34:14 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:34:15.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:34:15 smithi115 conmon[90393]: 2022-05-14T03:34:15.485+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:34:16.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:34:15 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:34:16.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:34:15 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:34:16.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:34:16 smithi115 conmon[90393]: 2022-05-14T03:34:16.492+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:34:17.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:34:16 smithi047 ceph-mon[77147]: pgmap v7746: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:34:17.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:34:16 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:34:17.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:34:16 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 14790 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:34:17.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:34:16 smithi115 ceph-mon[78593]: pgmap v7746: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:34:17.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:34:16 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:34:17.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:34:16 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 14790 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:34:17.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:34:17 smithi115 conmon[90393]: 2022-05-14T03:34:17.450+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:34:18.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:34:17 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:34:18.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:34:17 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:34:18.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:34:18 smithi115 conmon[90393]: 2022-05-14T03:34:18.491+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:34:19.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:34:18 smithi047 ceph-mon[77147]: pgmap v7747: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:34:19.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:34:18 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:34:19.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:34:18 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:34:19.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:34:18 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:34:19.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:34:18 smithi115 ceph-mon[78593]: pgmap v7747: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:34:19.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:34:18 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:34:19.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:34:18 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:34:19.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:34:18 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:34:19.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:34:19 smithi115 conmon[90393]: 2022-05-14T03:34:19.473+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:34:20.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:34:19 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:34:20.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:34:19 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:34:20.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:34:20 smithi115 conmon[90393]: 2022-05-14T03:34:20.514+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:34:21.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:34:20 smithi047 ceph-mon[77147]: pgmap v7748: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:34:21.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:34:20 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:34:21.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:34:20 smithi115 ceph-mon[78593]: pgmap v7748: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:34:21.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:34:20 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:34:21.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:34:21 smithi115 conmon[90393]: 2022-05-14T03:34:21.476+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:34:22.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:34:21 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:34:22.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:34:21 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 14795 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:34:22.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:34:21 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:34:22.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:34:21 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 14795 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:34:22.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:34:22 smithi115 conmon[90393]: 2022-05-14T03:34:22.459+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:34:23.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:34:22 smithi047 ceph-mon[77147]: pgmap v7749: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:34:23.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:34:22 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:34:23.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:34:22 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:34:23.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:34:22 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:34:23.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:34:22 smithi115 ceph-mon[78593]: pgmap v7749: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:34:23.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:34:22 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:34:23.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:34:22 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:34:23.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:34:22 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:34:23.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:34:23 smithi115 conmon[90393]: 2022-05-14T03:34:23.495+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:34:24.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:34:24 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:34:24.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:34:24 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:34:24.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:34:24 smithi115 conmon[90393]: 2022-05-14T03:34:24.489+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:34:25.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:34:25 smithi047 ceph-mon[77147]: pgmap v7750: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:34:25.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:34:25 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:34:25.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:34:25 smithi115 ceph-mon[78593]: pgmap v7750: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:34:25.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:34:25 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:34:25.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:34:25 smithi115 conmon[90393]: 2022-05-14T03:34:25.495+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:34:26.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:34:26 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:34:26.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:34:26 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:34:26.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:34:26 smithi115 conmon[90393]: 2022-05-14T03:34:26.462+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:34:27.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:34:27 smithi047 ceph-mon[77147]: pgmap v7751: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:34:27.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:34:27 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:34:27.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:34:27 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 14800 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:34:27.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:34:27 smithi115 ceph-mon[78593]: pgmap v7751: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:34:27.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:34:27 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:34:27.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:34:27 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 14800 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:34:27.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:34:27 smithi115 conmon[90393]: 2022-05-14T03:34:27.418+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:34:28.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:34:28 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:34:28.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:34:28 smithi047 ceph-mon[77147]: pgmap v7752: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:34:28.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:34:28 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:34:28.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:34:28 smithi115 ceph-mon[78593]: pgmap v7752: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:34:28.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:34:28 smithi115 conmon[90393]: 2022-05-14T03:34:28.439+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:34:29.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:34:29 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:34:29.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:34:29 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:34:29.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:34:29 smithi115 conmon[90393]: 2022-05-14T03:34:29.473+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:34:30.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:34:30 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:34:30.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:34:30 smithi047 ceph-mon[77147]: pgmap v7753: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:34:30.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:34:30 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:34:30.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:34:30 smithi115 ceph-mon[78593]: pgmap v7753: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:34:30.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:34:30 smithi115 conmon[90393]: 2022-05-14T03:34:30.484+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:34:31.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:34:31 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:34:31.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:34:31 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:34:31.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:34:31 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:34:31.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:34:31 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:34:31.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:34:31 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:34:31.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:34:31 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:34:31.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:34:31 smithi115 conmon[90393]: 2022-05-14T03:34:31.447+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:34:32.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:34:32 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:34:32.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:34:32 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 14805 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:34:32.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:34:32 smithi047 ceph-mon[77147]: pgmap v7754: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:34:32.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:34:32 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:34:32.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:34:32 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 14805 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:34:32.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:34:32 smithi115 ceph-mon[78593]: pgmap v7754: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:34:32.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:34:32 smithi115 conmon[90393]: 2022-05-14T03:34:32.496+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:34:33.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:34:33 smithi115 conmon[90393]: 2022-05-14T03:34:33.523+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:34:33.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:34:33 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:34:34.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:34:33 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:34:34.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:34:34 smithi115 conmon[90393]: 2022-05-14T03:34:34.516+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:34:34.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:34:34 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:34:34.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:34:34 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:34:34.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:34:34 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:34:34.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:34:34 smithi115 ceph-mon[78593]: pgmap v7755: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:34:35.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:34:34 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:34:35.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:34:34 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:34:35.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:34:34 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:34:35.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:34:34 smithi047 ceph-mon[77147]: pgmap v7755: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:34:35.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:34:35 smithi115 conmon[90393]: 2022-05-14T03:34:35.531+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:34:35.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:34:35 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:34:36.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:34:35 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:34:36.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:34:36 smithi115 conmon[90393]: 2022-05-14T03:34:36.521+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:34:36.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:34:36 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:34:36.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:34:36 smithi115 ceph-mon[78593]: pgmap v7756: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:34:37.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:34:36 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:34:37.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:34:36 smithi047 ceph-mon[77147]: pgmap v7756: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:34:37.864 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:34:37 smithi115 conmon[90393]: 2022-05-14T03:34:37.555+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:34:37.864 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:34:37 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:34:37.865 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:34:37 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 14810 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:34:38.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:34:37 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:34:38.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:34:37 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 14810 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:34:38.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:34:38 smithi115 conmon[90393]: 2022-05-14T03:34:38.512+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:34:38.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:34:38 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:34:38.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:34:38 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:34:38.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:34:38 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:34:38.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:34:38 smithi115 ceph-mon[78593]: pgmap v7757: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:34:39.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:34:38 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:34:39.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:34:38 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:34:39.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:34:38 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:34:39.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:34:38 smithi047 ceph-mon[77147]: pgmap v7757: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:34:39.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:34:39 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:34:39.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:34:39 smithi115 conmon[90393]: 2022-05-14T03:34:39.504+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:34:40.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:34:39 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:34:40.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:34:40 smithi115 conmon[90393]: 2022-05-14T03:34:40.464+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:34:40.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:34:40 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:34:40.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:34:40 smithi115 ceph-mon[78593]: pgmap v7758: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:34:41.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:34:40 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:34:41.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:34:40 smithi047 ceph-mon[77147]: pgmap v7758: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:34:41.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:34:41 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:34:41.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:34:41 smithi115 conmon[90393]: 2022-05-14T03:34:41.472+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:34:42.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:34:41 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:34:42.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:34:42 smithi115 conmon[90393]: 2022-05-14T03:34:42.431+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:34:42.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:34:42 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:34:42.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:34:42 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:34:42.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:34:42 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:34:42.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:34:42 smithi115 ceph-mon[78593]: pgmap v7759: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:34:43.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:34:42 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:34:43.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:34:42 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:34:43.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:34:42 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:34:43.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:34:42 smithi047 ceph-mon[77147]: pgmap v7759: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:34:43.684 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:34:43 smithi115 conmon[90393]: 2022-05-14T03:34:43.398+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:34:44.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:34:43 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:34:44.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:34:43 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:34:44.688 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:34:44 smithi115 conmon[90393]: 2022-05-14T03:34:44.364+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:34:45.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:34:44 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:34:45.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:34:44 smithi047 ceph-mon[77147]: pgmap v7760: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:34:45.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:34:44 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:34:45.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:34:44 smithi115 ceph-mon[78593]: pgmap v7760: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:34:45.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:34:45 smithi115 conmon[90393]: 2022-05-14T03:34:45.346+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:34:46.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:34:45 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:34:46.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:34:45 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:34:46.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:34:46 smithi115 conmon[90393]: 2022-05-14T03:34:46.341+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:34:47.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:34:46 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:34:47.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:34:46 smithi047 ceph-mon[77147]: pgmap v7761: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:34:47.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:34:46 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 14815 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:34:47.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:34:46 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:34:47.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:34:46 smithi115 ceph-mon[78593]: pgmap v7761: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:34:47.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:34:46 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 14815 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:34:47.696 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:34:47 smithi115 conmon[90393]: 2022-05-14T03:34:47.364+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:34:48.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:34:47 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:34:48.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:34:47 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:34:48.701 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:34:48 smithi115 conmon[90393]: 2022-05-14T03:34:48.363+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:34:49.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:34:48 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:34:49.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:34:48 smithi047 ceph-mon[77147]: pgmap v7762: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:34:49.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:34:48 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:34:49.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:34:48 smithi115 ceph-mon[78593]: pgmap v7762: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:34:49.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:34:49 smithi115 conmon[90393]: 2022-05-14T03:34:49.333+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:34:50.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:34:49 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:34:50.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:34:49 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T03:34:50.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:34:49 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T03:34:50.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:34:49 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T03:34:50.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:34:49 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:34:50.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:34:49 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T03:34:50.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:34:49 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T03:34:50.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:34:49 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T03:34:50.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:34:50 smithi115 conmon[90393]: 2022-05-14T03:34:50.303+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:34:51.045 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:34:50 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:34:51.046 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:34:50 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:34:51.046 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:34:50 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:34:51.046 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:34:50 smithi047 ceph-mon[77147]: pgmap v7763: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:34:51.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:34:50 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:34:51.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:34:50 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:34:51.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:34:50 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:34:51.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:34:50 smithi115 ceph-mon[78593]: pgmap v7763: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:34:51.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:34:51 smithi115 conmon[90393]: 2022-05-14T03:34:51.349+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:34:52.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:34:51 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:34:52.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:34:51 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 14825 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:34:52.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:34:51 smithi047 conmon[78521]: 2022-05-14T03:34:51.824+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:34:52.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:34:51 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T03:34:52.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:34:51 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:34:52.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:34:51 smithi047 conmon[78521]: yield 2022-05-14T03:34:52.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:34:51 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:34:52.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:34:51 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:34:52.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:34:51 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:34:52.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:34:51 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T03:34:52.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:34:51 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:34:52.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:34:51 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T03:34:52.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:34:51 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:34:52.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:34:51 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T03:34:52.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:34:51 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:34:52.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:34:51 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T03:34:52.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:34:51 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:34:52.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:34:51 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T03:34:52.048 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:34:51 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:34:52.048 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:34:51 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:34:52.048 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:34:51 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:34:52.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:34:51 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:34:52.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:34:51 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 14825 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:34:52.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:34:52 smithi115 conmon[90393]: 2022-05-14T03:34:52.354+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:34:53.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:34:52 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:34:53.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:34:52 smithi047 ceph-mon[77147]: pgmap v7764: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:34:53.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:34:52 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:34:53.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T03:34:53.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:34:53.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T03:34:53.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:34:53.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:34:53.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:34:53.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T03:34:53.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:34:53.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T03:34:53.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:34:53.049 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T03:34:53.049 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:34:53.050 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T03:34:53.050 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:34:53.050 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T03:34:53.051 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:34:53.051 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:34:53.052 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:34:53.052 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:34:52 smithi047 ceph-mon[77147]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T03:34:53.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:34:52 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:34:53.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:34:52 smithi115 ceph-mon[78593]: pgmap v7764: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:34:53.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:34:52 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:34:53.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T03:34:53.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:34:53.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T03:34:53.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:34:53.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:34:53.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:34:53.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T03:34:53.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:34:53.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T03:34:53.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:34:53.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T03:34:53.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:34:53.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T03:34:53.113 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:34:53.113 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T03:34:53.113 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:34:53.114 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:34:53.114 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:34:53.114 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:34:52 smithi115 ceph-mon[78593]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T03:34:53.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:34:53 smithi115 conmon[90393]: 2022-05-14T03:34:53.313+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:34:54.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:34:53 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:34:54.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:34:53 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:34:54.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:34:54 smithi115 conmon[90393]: 2022-05-14T03:34:54.282+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:34:55.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:34:54 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:34:55.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:34:54 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:34:55.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:34:54 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:34:55.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:34:54 smithi047 ceph-mon[77147]: pgmap v7765: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:34:55.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:34:54 smithi047 conmon[78521]: 2022-05-14T03:34:54.896+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:34:55.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:34:54 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T03:34:55.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:34:54 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:34:55.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:34:54 smithi047 conmon[78521]: yield 2022-05-14T03:34:55.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:34:54 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:34:55.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:34:54 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:34:55.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:34:54 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:34:55.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:34:54 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T03:34:55.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:34:54 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:34:55.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:34:54 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T03:34:55.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:34:54 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:34:55.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:34:54 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T03:34:55.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:34:54 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:34:55.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:34:54 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T03:34:55.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:34:54 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:34:55.048 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:34:54 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T03:34:55.048 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:34:54 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:34:55.048 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:34:54 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:34:55.048 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:34:54 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:34:55.048 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:34:54 smithi047 conmon[78521]: 2022-05-14T03:34:54.897+0000 7f5344781700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T03:34:55.049 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:34:54 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T03:34:55.049 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:34:54 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:34:55.049 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:34:54 smithi047 conmon[78521]: yield 2022-05-14T03:34:55.050 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:34:54 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:34:55.050 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:34:54 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:34:55.050 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:34:54 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:34:55.050 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:34:54 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T03:34:55.051 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:34:54 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:34:55.051 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:34:54 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T03:34:55.051 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:34:54 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:34:55.051 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:34:54 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T03:34:55.052 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:34:54 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:34:55.052 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:34:54 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T03:34:55.052 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:34:54 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:34:55.052 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:34:54 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T03:34:55.053 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:34:54 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:34:55.053 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:34:54 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:34:55.053 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:34:54 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:34:55.053 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:34:54 smithi047 conmon[78521]: 2022-05-14T03:34:55.054 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:34:54 smithi047 conmon[78521]: During handling of the above exception, another exception occurred: 2022-05-14T03:34:55.054 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:34:54 smithi047 conmon[78521]: 2022-05-14T03:34:55.054 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:34:54 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T03:34:55.054 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:34:54 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T03:34:55.055 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:34:54 smithi047 conmon[78521]: return f(*arg) 2022-05-14T03:34:55.055 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:34:54 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T03:34:55.055 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:34:54 smithi047 conmon[78521]: self._write_client_files(client_files, host) 2022-05-14T03:34:55.056 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:34:54 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T03:34:55.056 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:34:54 smithi047 conmon[78521]: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T03:34:55.056 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:34:54 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T03:34:55.056 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:34:54 smithi047 conmon[78521]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T03:34:55.056 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:34:54 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T03:34:55.057 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:34:54 smithi047 conmon[78521]: return self.event_loop.get_result(coro) 2022-05-14T03:34:55.057 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:34:54 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T03:34:55.057 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:34:54 smithi047 conmon[78521]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T03:34:55.057 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:34:54 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T03:34:55.058 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:34:54 smithi047 conmon[78521]: return self.__get_result() 2022-05-14T03:34:55.058 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:34:54 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T03:34:55.058 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:34:54 smithi047 conmon[78521]: raise self._exception 2022-05-14T03:34:55.058 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:34:54 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T03:34:55.058 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:34:54 smithi047 conmon[78521]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T03:34:55.059 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:34:54 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T03:34:55.059 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:34:54 smithi047 conmon[78521]: conn = await self._remote_connection(host, addr) 2022-05-14T03:34:55.059 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:34:54 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T03:34:55.059 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:34:54 smithi047 conmon[78521]: raise 2022-05-14T03:34:55.059 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:34:54 smithi047 conmon[78521]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T03:34:55.060 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:34:54 smithi047 conmon[78521]: self.gen.throw(type, value, traceback) 2022-05-14T03:34:55.060 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:34:54 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T03:34:55.060 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:34:54 smithi047 conmon[78521]: raise OrchestratorError(msg) 2022-05-14T03:34:55.060 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:34:54 smithi047 conmon[78521]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:34:55.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:34:54 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:34:55.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:34:54 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:34:55.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:34:54 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:34:55.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:34:54 smithi115 ceph-mon[78593]: pgmap v7765: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:34:55.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:34:55 smithi115 conmon[90393]: 2022-05-14T03:34:55.245+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:34:56.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:34:55 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:34:56.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:34:55 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:34:56.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T03:34:56.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:34:56.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T03:34:56.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:34:56.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:34:56.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:34:56.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T03:34:56.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:34:56.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T03:34:56.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:34:56.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T03:34:56.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:34:56.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T03:34:56.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:34:56.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T03:34:56.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:34:56.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:34:56.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:34:56.048 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:34:55 smithi047 ceph-mon[77147]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T03:34:56.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T03:34:56.049 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:34:56.049 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T03:34:56.050 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:34:56.050 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:34:56.051 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:34:56.051 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T03:34:56.052 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:34:56.052 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T03:34:56.052 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:34:56.053 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T03:34:56.053 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:34:56.054 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T03:34:56.054 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:34:56.055 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T03:34:56.055 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:34:56.055 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:34:56.056 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:34:56.056 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T03:34:56.057 INFO:journalctl@ceph.mon.b.smithi047.stdout: During handling of the above exception, another exception occurred: 2022-05-14T03:34:56.057 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T03:34:56.058 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T03:34:56.058 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T03:34:56.058 INFO:journalctl@ceph.mon.b.smithi047.stdout: return f(*arg) 2022-05-14T03:34:56.059 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T03:34:56.059 INFO:journalctl@ceph.mon.b.smithi047.stdout: self._write_client_files(client_files, host) 2022-05-14T03:34:56.060 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T03:34:56.060 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T03:34:56.061 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T03:34:56.061 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T03:34:56.062 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T03:34:56.062 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.event_loop.get_result(coro) 2022-05-14T03:34:56.062 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T03:34:56.063 INFO:journalctl@ceph.mon.b.smithi047.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T03:34:56.063 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T03:34:56.064 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.__get_result() 2022-05-14T03:34:56.064 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T03:34:56.064 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise self._exception 2022-05-14T03:34:56.064 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T03:34:56.064 INFO:journalctl@ceph.mon.b.smithi047.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T03:34:56.065 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T03:34:56.065 INFO:journalctl@ceph.mon.b.smithi047.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T03:34:56.065 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T03:34:56.065 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise 2022-05-14T03:34:56.066 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T03:34:56.066 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.gen.throw(type, value, traceback) 2022-05-14T03:34:56.066 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T03:34:56.066 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OrchestratorError(msg) 2022-05-14T03:34:56.066 INFO:journalctl@ceph.mon.b.smithi047.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:34:56.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:34:55 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:34:56.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:34:55 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:34:56.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T03:34:56.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:34:56.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T03:34:56.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:34:56.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:34:56.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:34:56.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T03:34:56.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:34:56.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T03:34:56.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:34:56.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T03:34:56.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:34:56.113 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T03:34:56.113 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:34:56.113 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T03:34:56.113 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:34:56.113 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:34:56.114 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:34:56.114 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:34:55 smithi115 ceph-mon[78593]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T03:34:56.114 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T03:34:56.114 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:34:56.115 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T03:34:56.115 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:34:56.115 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:34:56.115 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:34:56.116 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T03:34:56.116 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:34:56.116 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T03:34:56.116 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:34:56.117 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T03:34:56.117 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:34:56.118 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T03:34:56.118 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:34:56.118 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T03:34:56.119 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:34:56.119 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:34:56.120 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:34:56.120 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T03:34:56.120 INFO:journalctl@ceph.mon.c.smithi115.stdout: During handling of the above exception, another exception occurred: 2022-05-14T03:34:56.121 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T03:34:56.121 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T03:34:56.122 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T03:34:56.122 INFO:journalctl@ceph.mon.c.smithi115.stdout: return f(*arg) 2022-05-14T03:34:56.122 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T03:34:56.123 INFO:journalctl@ceph.mon.c.smithi115.stdout: self._write_client_files(client_files, host) 2022-05-14T03:34:56.123 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T03:34:56.124 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T03:34:56.124 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T03:34:56.124 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T03:34:56.124 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T03:34:56.125 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.event_loop.get_result(coro) 2022-05-14T03:34:56.125 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T03:34:56.125 INFO:journalctl@ceph.mon.c.smithi115.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T03:34:56.125 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T03:34:56.126 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.__get_result() 2022-05-14T03:34:56.126 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T03:34:56.126 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise self._exception 2022-05-14T03:34:56.126 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T03:34:56.127 INFO:journalctl@ceph.mon.c.smithi115.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T03:34:56.127 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T03:34:56.127 INFO:journalctl@ceph.mon.c.smithi115.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T03:34:56.127 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T03:34:56.127 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise 2022-05-14T03:34:56.128 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T03:34:56.128 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.gen.throw(type, value, traceback) 2022-05-14T03:34:56.128 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T03:34:56.129 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OrchestratorError(msg) 2022-05-14T03:34:56.129 INFO:journalctl@ceph.mon.c.smithi115.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:34:56.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:34:56 smithi115 conmon[90393]: 2022-05-14T03:34:56.233+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:34:57.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:34:56 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:34:57.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:34:56 smithi047 ceph-mon[77147]: pgmap v7766: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:34:57.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:34:56 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:34:57.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:34:56 smithi115 ceph-mon[78593]: pgmap v7766: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:34:57.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:34:57 smithi115 conmon[90393]: 2022-05-14T03:34:57.210+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:34:58.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:34:57 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:34:58.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:34:57 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 14830 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:34:58.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:34:57 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:34:58.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:34:57 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 14830 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:34:58.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:34:58 smithi115 conmon[90393]: 2022-05-14T03:34:58.183+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:34:59.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:34:58 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:34:59.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:34:58 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:34:59.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:34:58 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:34:59.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:34:58 smithi047 ceph-mon[77147]: pgmap v7767: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:34:59.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:34:58 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:34:59.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:34:58 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:34:59.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:34:58 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:34:59.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:34:58 smithi115 ceph-mon[78593]: pgmap v7767: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:34:59.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:34:59 smithi115 conmon[90393]: 2022-05-14T03:34:59.233+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:35:00.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:34:59 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:35:00.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:34:59 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:35:00.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:35:00 smithi115 conmon[90393]: 2022-05-14T03:35:00.195+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:35:01.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:35:00 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:35:01.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:35:00 smithi047 ceph-mon[77147]: pgmap v7768: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:35:01.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:35:00 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:35:01.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:35:00 smithi115 ceph-mon[78593]: pgmap v7768: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:35:01.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:35:01 smithi115 conmon[90393]: 2022-05-14T03:35:01.199+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:35:02.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:35:01 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:35:02.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:35:01 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:35:02.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:35:02 smithi115 conmon[90393]: 2022-05-14T03:35:02.216+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:35:03.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:35:02 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:35:03.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:35:02 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:35:03.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:35:02 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:35:03.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:35:02 smithi047 ceph-mon[77147]: pgmap v7769: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:35:03.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:35:02 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:35:03.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:35:02 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:35:03.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:35:02 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:35:03.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:35:02 smithi115 ceph-mon[78593]: pgmap v7769: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:35:03.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:35:03 smithi115 conmon[90393]: 2022-05-14T03:35:03.264+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:35:04.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:35:03 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:35:04.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:35:03 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:35:04.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:35:04 smithi115 conmon[90393]: 2022-05-14T03:35:04.239+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:35:05.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:35:04 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:35:05.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:35:04 smithi047 ceph-mon[77147]: pgmap v7770: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:35:05.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:35:04 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:35:05.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:35:04 smithi115 ceph-mon[78593]: pgmap v7770: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:35:05.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:35:05 smithi115 conmon[90393]: 2022-05-14T03:35:05.216+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:35:06.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:35:05 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:35:06.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:35:05 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:35:06.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:35:06 smithi115 conmon[90393]: 2022-05-14T03:35:06.170+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:35:07.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:35:06 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:35:07.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:35:06 smithi047 ceph-mon[77147]: pgmap v7771: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:35:07.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:35:06 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 14835 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:35:07.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:35:06 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:35:07.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:35:06 smithi115 ceph-mon[78593]: pgmap v7771: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:35:07.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:35:06 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 14835 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:35:07.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:35:07 smithi115 conmon[90393]: 2022-05-14T03:35:07.220+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:35:08.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:35:07 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:35:08.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:35:07 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:35:08.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:35:08 smithi115 conmon[90393]: 2022-05-14T03:35:08.212+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:35:09.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:35:08 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:35:09.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:35:08 smithi047 ceph-mon[77147]: pgmap v7772: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:35:09.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:35:08 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:35:09.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:35:08 smithi115 ceph-mon[78593]: pgmap v7772: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:35:09.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:35:09 smithi115 conmon[90393]: 2022-05-14T03:35:09.205+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:35:10.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:35:09 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:35:10.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:35:09 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:35:10.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:35:10 smithi115 conmon[90393]: 2022-05-14T03:35:10.200+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:35:11.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:35:10 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:35:11.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:35:10 smithi047 ceph-mon[77147]: pgmap v7773: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:35:11.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:35:10 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:35:11.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:35:10 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:35:11.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:35:10 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:35:11.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:35:10 smithi115 ceph-mon[78593]: pgmap v7773: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:35:11.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:35:10 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:35:11.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:35:10 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:35:11.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:35:11 smithi115 conmon[90393]: 2022-05-14T03:35:11.231+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:35:12.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:35:11 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:35:12.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:35:11 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 14845 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:35:12.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:35:11 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:35:12.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:35:11 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 14845 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:35:12.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:35:12 smithi115 conmon[90393]: 2022-05-14T03:35:12.201+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:35:13.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:35:12 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:35:13.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:35:12 smithi047 ceph-mon[77147]: pgmap v7774: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:35:13.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:35:12 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:35:13.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:35:12 smithi115 ceph-mon[78593]: pgmap v7774: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:35:13.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:35:13 smithi115 conmon[90393]: 2022-05-14T03:35:13.205+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:35:14.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:35:13 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:35:14.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:35:13 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:35:14.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:35:14 smithi115 conmon[90393]: 2022-05-14T03:35:14.199+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:35:15.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:35:14 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:35:15.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:35:14 smithi115 ceph-mon[78593]: pgmap v7775: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:35:15.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:35:14 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:35:15.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:35:14 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:35:15.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:35:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T03:35:15.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:35:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T03:35:15.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:35:14 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:35:15.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:35:14 smithi047 ceph-mon[77147]: pgmap v7775: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:35:15.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:35:14 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:35:15.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:35:14 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:35:15.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:35:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T03:35:15.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:35:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T03:35:15.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:35:15 smithi115 conmon[90393]: 2022-05-14T03:35:15.171+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:35:16.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:35:15 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:35:16.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:35:15 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:35:16.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:35:16 smithi115 conmon[90393]: 2022-05-14T03:35:16.198+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:35:17.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:35:16 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:35:17.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:35:16 smithi115 ceph-mon[78593]: pgmap v7776: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:35:17.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:35:16 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 14850 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:35:17.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:35:16 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:35:17.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:35:16 smithi047 ceph-mon[77147]: pgmap v7776: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:35:17.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:35:16 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 14850 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:35:17.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:35:17 smithi115 conmon[90393]: 2022-05-14T03:35:17.174+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:35:18.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:35:17 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:35:18.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:35:17 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:35:18.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:35:18 smithi115 conmon[90393]: 2022-05-14T03:35:18.212+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:35:19.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:35:18 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:35:19.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:35:18 smithi115 ceph-mon[78593]: pgmap v7777: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:35:19.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:35:18 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:35:19.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:35:18 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:35:19.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:35:18 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:35:19.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:35:18 smithi047 ceph-mon[77147]: pgmap v7777: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:35:19.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:35:18 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:35:19.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:35:18 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:35:19.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:35:19 smithi115 conmon[90393]: 2022-05-14T03:35:19.212+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:35:20.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:35:19 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:35:20.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:35:19 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:35:20.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:35:20 smithi115 conmon[90393]: 2022-05-14T03:35:20.254+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:35:21.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:35:20 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:35:21.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:35:20 smithi115 ceph-mon[78593]: pgmap v7778: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:35:21.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:35:20 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:35:21.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:35:20 smithi047 ceph-mon[77147]: pgmap v7778: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:35:21.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:35:21 smithi115 conmon[90393]: 2022-05-14T03:35:21.229+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:35:22.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:35:21 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:35:22.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:35:21 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 14855 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:35:22.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:35:21 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:35:22.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:35:21 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 14855 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:35:22.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:35:22 smithi115 conmon[90393]: 2022-05-14T03:35:22.218+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:35:23.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:35:22 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:35:23.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:35:22 smithi115 ceph-mon[78593]: pgmap v7779: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:35:23.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:35:22 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:35:23.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:35:22 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:35:23.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:35:22 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:35:23.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:35:22 smithi047 ceph-mon[77147]: pgmap v7779: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:35:23.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:35:22 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:35:23.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:35:22 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:35:23.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:35:23 smithi115 conmon[90393]: 2022-05-14T03:35:23.217+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:35:24.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:35:23 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:35:24.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:35:23 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:35:24.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:35:24 smithi115 conmon[90393]: 2022-05-14T03:35:24.235+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:35:25.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:35:24 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:35:25.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:35:24 smithi115 ceph-mon[78593]: pgmap v7780: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:35:25.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:35:24 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:35:25.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:35:24 smithi047 ceph-mon[77147]: pgmap v7780: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:35:25.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:35:25 smithi115 conmon[90393]: 2022-05-14T03:35:25.245+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:35:26.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:35:25 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:35:26.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:35:25 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:35:26.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:35:26 smithi115 conmon[90393]: 2022-05-14T03:35:26.249+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:35:27.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:35:26 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:35:27.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:35:26 smithi115 ceph-mon[78593]: pgmap v7781: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:35:27.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:35:26 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 14860 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:35:27.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:35:26 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:35:27.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:35:26 smithi047 ceph-mon[77147]: pgmap v7781: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:35:27.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:35:26 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 14860 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:35:27.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:35:27 smithi115 conmon[90393]: 2022-05-14T03:35:27.225+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:35:28.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:35:27 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:35:28.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:35:27 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:35:28.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:35:28 smithi115 conmon[90393]: 2022-05-14T03:35:28.271+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:35:29.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:35:28 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:35:29.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:35:28 smithi115 ceph-mon[78593]: pgmap v7782: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:35:29.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:35:28 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:35:29.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:35:28 smithi047 ceph-mon[77147]: pgmap v7782: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:35:29.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:35:29 smithi115 conmon[90393]: 2022-05-14T03:35:29.243+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:35:30.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:35:29 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:35:30.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:35:29 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:35:30.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:35:30 smithi115 conmon[90393]: 2022-05-14T03:35:30.274+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:35:31.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:35:30 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:35:31.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:35:30 smithi115 ceph-mon[78593]: pgmap v7783: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:35:31.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:35:30 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:35:31.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:35:30 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:35:31.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:35:30 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:35:31.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:35:30 smithi047 ceph-mon[77147]: pgmap v7783: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:35:31.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:35:30 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:35:31.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:35:30 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:35:31.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:35:31 smithi115 conmon[90393]: 2022-05-14T03:35:31.290+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:35:32.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:35:31 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:35:32.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:35:31 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 14865 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:35:32.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:35:31 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:35:32.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:35:31 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 14865 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:35:32.616 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:35:32 smithi115 conmon[90393]: 2022-05-14T03:35:32.255+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:35:33.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:35:32 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:35:33.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:35:32 smithi115 ceph-mon[78593]: pgmap v7784: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:35:33.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:35:32 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:35:33.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:35:32 smithi047 ceph-mon[77147]: pgmap v7784: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:35:33.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:35:33 smithi115 conmon[90393]: 2022-05-14T03:35:33.263+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:35:34.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:35:33 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:35:34.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:35:33 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:35:34.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:35:34 smithi115 conmon[90393]: 2022-05-14T03:35:34.239+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:35:35.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:35:34 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:35:35.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:35:34 smithi115 ceph-mon[78593]: pgmap v7785: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:35:35.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:35:34 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:35:35.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:35:34 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:35:35.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:35:34 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:35:35.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:35:34 smithi047 ceph-mon[77147]: pgmap v7785: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:35:35.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:35:34 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:35:35.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:35:34 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:35:35.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:35:35 smithi115 conmon[90393]: 2022-05-14T03:35:35.204+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:35:36.210 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:35:35 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:35:36.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:35:35 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:35:36.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:35:36 smithi115 conmon[90393]: 2022-05-14T03:35:36.207+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:35:37.240 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:35:36 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:35:37.240 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:35:36 smithi115 ceph-mon[78593]: pgmap v7786: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:35:37.241 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:35:36 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 14870 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:35:37.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:35:36 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:35:37.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:35:36 smithi047 ceph-mon[77147]: pgmap v7786: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:35:37.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:35:36 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 14870 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:35:37.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:35:37 smithi115 conmon[90393]: 2022-05-14T03:35:37.237+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:35:38.268 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:35:37 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:35:38.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:35:37 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:35:38.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:35:38 smithi115 conmon[90393]: 2022-05-14T03:35:38.265+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:35:39.249 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:35:38 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:35:39.250 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:35:38 smithi115 ceph-mon[78593]: pgmap v7787: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:35:39.250 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:35:38 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:35:39.250 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:35:38 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:35:39.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:35:38 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:35:39.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:35:38 smithi047 ceph-mon[77147]: pgmap v7787: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:35:39.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:35:38 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:35:39.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:35:38 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:35:39.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:35:39 smithi115 conmon[90393]: 2022-05-14T03:35:39.246+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:35:40.261 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:35:39 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:35:40.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:35:39 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:35:40.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:35:40 smithi115 conmon[90393]: 2022-05-14T03:35:40.258+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:35:41.282 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:35:40 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:35:41.282 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:35:40 smithi115 ceph-mon[78593]: pgmap v7788: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:35:41.282 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:35:40 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:35:41.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:35:40 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:35:41.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:35:40 smithi047 ceph-mon[77147]: pgmap v7788: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:35:41.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:35:40 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:35:41.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:35:41 smithi115 conmon[90393]: 2022-05-14T03:35:41.279+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:35:42.247 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:35:41 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:35:42.248 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:35:41 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 14875 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:35:42.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:35:41 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:35:42.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:35:41 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 14875 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:35:42.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:35:42 smithi115 conmon[90393]: 2022-05-14T03:35:42.245+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:35:43.267 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:35:42 smithi115 ceph-mon[78593]: pgmap v7789: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:35:43.267 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:35:42 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:35:43.267 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:35:42 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:35:43.268 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:35:42 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:35:43.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:35:42 smithi047 ceph-mon[77147]: pgmap v7789: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:35:43.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:35:42 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:35:43.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:35:42 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:35:43.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:35:42 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:35:43.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:35:43 smithi115 conmon[90393]: 2022-05-14T03:35:43.264+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:35:44.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:35:43 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:35:44.298 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:35:43 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:35:44.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:35:44 smithi115 conmon[90393]: 2022-05-14T03:35:44.296+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:35:45.267 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:35:44 smithi115 ceph-mon[78593]: pgmap v7790: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:35:45.267 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:35:44 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:35:45.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:35:44 smithi047 ceph-mon[77147]: pgmap v7790: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:35:45.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:35:44 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:35:45.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:35:45 smithi115 conmon[90393]: 2022-05-14T03:35:45.264+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:35:46.250 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:35:45 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:35:46.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:35:45 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:35:46.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:35:46 smithi115 conmon[90393]: 2022-05-14T03:35:46.248+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:35:47.271 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:35:46 smithi115 ceph-mon[78593]: pgmap v7791: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:35:47.271 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:35:46 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:35:47.272 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:35:46 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 14880 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:35:47.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:35:46 smithi047 ceph-mon[77147]: pgmap v7791: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:35:47.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:35:46 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:35:47.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:35:46 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 14880 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:35:47.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:35:47 smithi115 conmon[90393]: 2022-05-14T03:35:47.269+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:35:48.248 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:35:47 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:35:48.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:35:47 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:35:48.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:35:48 smithi115 conmon[90393]: 2022-05-14T03:35:48.246+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:35:49.234 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:35:48 smithi115 ceph-mon[78593]: pgmap v7792: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:35:49.234 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:35:48 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:35:49.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:35:48 smithi047 ceph-mon[77147]: pgmap v7792: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:35:49.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:35:48 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:35:49.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:35:49 smithi115 conmon[90393]: 2022-05-14T03:35:49.232+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:35:50.195 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:35:49 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:35:50.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:35:49 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:35:50.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:35:50 smithi115 conmon[90393]: 2022-05-14T03:35:50.192+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:35:51.206 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:35:50 smithi115 ceph-mon[78593]: pgmap v7793: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:35:51.207 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:35:50 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:35:51.207 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:35:50 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:35:51.207 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:35:50 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:35:51.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:35:50 smithi047 ceph-mon[77147]: pgmap v7793: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:35:51.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:35:50 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:35:51.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:35:50 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:35:51.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:35:50 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:35:51.613 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:35:51 smithi115 conmon[90393]: 2022-05-14T03:35:51.204+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:35:52.180 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:35:51 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:35:52.180 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:35:51 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 14885 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:35:52.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:35:51 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:35:52.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:35:51 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 14885 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:35:52.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:35:52 smithi115 conmon[90393]: 2022-05-14T03:35:52.178+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:35:53.211 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:35:52 smithi115 ceph-mon[78593]: pgmap v7794: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:35:53.211 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:35:52 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:35:53.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:35:52 smithi047 ceph-mon[77147]: pgmap v7794: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:35:53.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:35:52 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:35:53.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:35:53 smithi115 conmon[90393]: 2022-05-14T03:35:53.209+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:35:54.255 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:35:53 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:35:54.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:35:53 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:35:54.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:35:54 smithi115 conmon[90393]: 2022-05-14T03:35:54.253+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:35:55.286 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:35:54 smithi115 ceph-mon[78593]: pgmap v7795: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:35:55.286 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:35:54 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:35:55.287 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:35:54 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:35:55.287 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:35:54 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:35:55.287 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:35:54 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T03:35:55.287 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:35:54 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T03:35:55.288 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:35:54 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T03:35:55.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:35:54 smithi047 ceph-mon[77147]: pgmap v7795: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:35:55.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:35:54 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:35:55.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:35:54 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:35:55.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:35:54 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:35:55.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:35:54 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T03:35:55.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:35:54 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T03:35:55.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:35:54 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T03:35:55.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:35:55 smithi115 conmon[90393]: 2022-05-14T03:35:55.283+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:35:55.792 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:35:55 smithi047 conmon[78521]: 2022-05-14T03:35:55.375+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:35:55.793 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:35:55 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T03:35:55.793 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:35:55 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:35:55.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:35:55 smithi047 conmon[78521]: yield 2022-05-14T03:35:55.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:35:55 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:35:55.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:35:55 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:35:55.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:35:55 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:35:55.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:35:55 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T03:35:55.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:35:55 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:35:55.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:35:55 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T03:35:55.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:35:55 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:35:55.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:35:55 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T03:35:55.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:35:55 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:35:55.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:35:55 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T03:35:55.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:35:55 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:35:55.797 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:35:55 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T03:35:55.797 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:35:55 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:35:55.797 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:35:55 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:35:55.798 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:35:55 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:35:56.252 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:35:55 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:35:56.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:35:55 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:35:56.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:35:56 smithi115 conmon[90393]: 2022-05-14T03:35:56.250+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:35:57.274 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:35:56 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:35:57.275 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T03:35:57.275 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:35:57.275 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T03:35:57.275 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:35:57.276 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:35:57.276 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:35:57.276 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T03:35:57.276 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:35:57.277 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T03:35:57.277 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:35:57.277 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T03:35:57.277 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:35:57.278 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T03:35:57.278 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:35:57.278 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T03:35:57.278 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:35:57.278 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:35:57.279 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:35:57.279 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:35:56 smithi115 ceph-mon[78593]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T03:35:57.279 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:35:56 smithi115 ceph-mon[78593]: pgmap v7796: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:35:57.279 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:35:56 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:35:57.279 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:35:56 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 14890 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:35:57.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:35:56 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:35:57.292 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T03:35:57.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:35:57.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T03:35:57.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:35:57.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:35:57.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:35:57.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T03:35:57.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:35:57.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T03:35:57.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:35:57.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T03:35:57.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:35:57.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T03:35:57.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:35:57.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T03:35:57.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:35:57.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:35:57.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:35:57.296 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:35:56 smithi047 ceph-mon[77147]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T03:35:57.297 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:35:56 smithi047 ceph-mon[77147]: pgmap v7796: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:35:57.297 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:35:56 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:35:57.297 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:35:56 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 14890 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:35:57.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:35:57 smithi115 conmon[90393]: 2022-05-14T03:35:57.272+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:35:58.289 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:35:57 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:35:58.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:35:57 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:35:58.610 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:35:58 smithi115 conmon[90393]: 2022-05-14T03:35:58.286+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:35:58.793 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:35:58 smithi047 conmon[78521]: 2022-05-14T03:35:58.447+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:35:58.793 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:35:58 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T03:35:58.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:35:58 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:35:58.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:35:58 smithi047 conmon[78521]: yield 2022-05-14T03:35:58.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:35:58 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:35:58.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:35:58 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:35:58.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:35:58 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:35:58.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:35:58 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T03:35:58.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:35:58 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:35:58.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:35:58 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T03:35:58.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:35:58 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:35:58.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:35:58 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T03:35:58.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:35:58 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:35:58.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:35:58 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T03:35:58.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:35:58 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:35:58.797 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:35:58 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T03:35:58.797 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:35:58 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:35:58.797 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:35:58 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:35:58.797 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:35:58 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:35:58.797 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:35:58 smithi047 conmon[78521]: 2022-05-14T03:35:58.448+0000 7f533f777700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T03:35:58.798 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:35:58 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T03:35:58.798 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:35:58 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:35:58.798 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:35:58 smithi047 conmon[78521]: yield 2022-05-14T03:35:58.798 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:35:58 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:35:58.799 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:35:58 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:35:58.799 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:35:58 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:35:58.799 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:35:58 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T03:35:58.799 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:35:58 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:35:58.799 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:35:58 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T03:35:58.800 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:35:58 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:35:58.800 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:35:58 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T03:35:58.800 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:35:58 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:35:58.800 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:35:58 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T03:35:58.801 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:35:58 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:35:58.801 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:35:58 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T03:35:58.801 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:35:58 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:35:58.801 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:35:58 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:35:58.802 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:35:58 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:35:58.802 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:35:58 smithi047 conmon[78521]: 2022-05-14T03:35:58.802 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:35:58 smithi047 conmon[78521]: During handling of the above exception, another exception occurred: 2022-05-14T03:35:58.802 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:35:58 smithi047 conmon[78521]: 2022-05-14T03:35:58.802 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:35:58 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T03:35:58.803 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:35:58 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T03:35:58.803 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:35:58 smithi047 conmon[78521]: return f(*arg) 2022-05-14T03:35:58.803 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:35:58 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T03:35:58.803 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:35:58 smithi047 conmon[78521]: self._write_client_files(client_files, host) 2022-05-14T03:35:58.804 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:35:58 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T03:35:58.804 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:35:58 smithi047 conmon[78521]: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T03:35:58.804 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:35:58 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T03:35:58.804 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:35:58 smithi047 conmon[78521]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T03:35:58.804 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:35:58 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T03:35:58.805 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:35:58 smithi047 conmon[78521]: return self.event_loop.get_result(coro) 2022-05-14T03:35:58.805 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:35:58 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T03:35:58.805 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:35:58 smithi047 conmon[78521]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T03:35:58.805 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:35:58 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T03:35:58.806 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:35:58 smithi047 conmon[78521]: return self.__get_result() 2022-05-14T03:35:58.806 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:35:58 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T03:35:58.806 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:35:58 smithi047 conmon[78521]: raise self._exception 2022-05-14T03:35:58.806 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:35:58 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T03:35:58.806 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:35:58 smithi047 conmon[78521]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T03:35:58.807 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:35:58 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T03:35:58.807 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:35:58 smithi047 conmon[78521]: conn = await self._remote_connection(host, addr) 2022-05-14T03:35:58.807 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:35:58 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T03:35:58.807 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:35:58 smithi047 conmon[78521]: raise 2022-05-14T03:35:58.807 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:35:58 smithi047 conmon[78521]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T03:35:58.808 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:35:58 smithi047 conmon[78521]: self.gen.throw(type, value, traceback) 2022-05-14T03:35:58.808 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:35:58 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T03:35:58.808 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:35:58 smithi047 conmon[78521]: raise OrchestratorError(msg) 2022-05-14T03:35:58.808 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:35:58 smithi047 conmon[78521]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:35:59.275 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:35:58 smithi115 ceph-mon[78593]: pgmap v7797: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:35:59.275 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:35:58 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:35:59.275 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:35:58 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:35:59.275 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:35:58 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:35:59.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:35:58 smithi047 ceph-mon[77147]: pgmap v7797: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:35:59.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:35:58 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:35:59.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:35:58 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:35:59.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:35:58 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:35:59.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:35:59 smithi115 conmon[90393]: 2022-05-14T03:35:59.272+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:36:00.239 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:35:59 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:36:00.239 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T03:36:00.240 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:36:00.240 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T03:36:00.240 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:36:00.240 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:36:00.241 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:36:00.241 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T03:36:00.241 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:36:00.241 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T03:36:00.242 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:36:00.242 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T03:36:00.242 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:36:00.242 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T03:36:00.242 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:36:00.243 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T03:36:00.243 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:36:00.243 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:36:00.243 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:36:00.243 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:35:59 smithi115 ceph-mon[78593]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T03:36:00.244 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T03:36:00.244 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:36:00.244 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T03:36:00.244 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:36:00.244 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:36:00.245 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:36:00.245 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T03:36:00.245 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:36:00.245 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T03:36:00.245 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:36:00.246 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T03:36:00.246 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:36:00.246 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T03:36:00.246 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:36:00.246 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T03:36:00.247 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:36:00.247 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:36:00.247 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:36:00.247 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T03:36:00.248 INFO:journalctl@ceph.mon.c.smithi115.stdout: During handling of the above exception, another exception occurred: 2022-05-14T03:36:00.248 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T03:36:00.248 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T03:36:00.248 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T03:36:00.248 INFO:journalctl@ceph.mon.c.smithi115.stdout: return f(*arg) 2022-05-14T03:36:00.249 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T03:36:00.249 INFO:journalctl@ceph.mon.c.smithi115.stdout: self._write_client_files(client_files, host) 2022-05-14T03:36:00.249 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T03:36:00.249 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T03:36:00.249 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T03:36:00.250 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T03:36:00.250 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T03:36:00.250 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.event_loop.get_result(coro) 2022-05-14T03:36:00.250 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T03:36:00.251 INFO:journalctl@ceph.mon.c.smithi115.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T03:36:00.251 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T03:36:00.251 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.__get_result() 2022-05-14T03:36:00.251 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T03:36:00.251 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise self._exception 2022-05-14T03:36:00.252 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T03:36:00.252 INFO:journalctl@ceph.mon.c.smithi115.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T03:36:00.252 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T03:36:00.252 INFO:journalctl@ceph.mon.c.smithi115.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T03:36:00.252 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T03:36:00.253 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise 2022-05-14T03:36:00.253 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T03:36:00.253 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.gen.throw(type, value, traceback) 2022-05-14T03:36:00.254 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T03:36:00.254 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OrchestratorError(msg) 2022-05-14T03:36:00.254 INFO:journalctl@ceph.mon.c.smithi115.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:36:00.254 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:35:59 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:36:00.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:35:59 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:36:00.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T03:36:00.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:36:00.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T03:36:00.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:36:00.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:36:00.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:36:00.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T03:36:00.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:36:00.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T03:36:00.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:36:00.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T03:36:00.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:36:00.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T03:36:00.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:36:00.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T03:36:00.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:36:00.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:36:00.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:36:00.297 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:35:59 smithi047 ceph-mon[77147]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T03:36:00.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T03:36:00.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:36:00.298 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T03:36:00.298 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:36:00.298 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:36:00.298 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:36:00.298 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T03:36:00.299 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:36:00.299 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T03:36:00.299 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:36:00.299 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T03:36:00.300 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:36:00.300 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T03:36:00.300 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:36:00.300 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T03:36:00.300 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:36:00.301 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:36:00.301 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:36:00.301 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T03:36:00.301 INFO:journalctl@ceph.mon.b.smithi047.stdout: During handling of the above exception, another exception occurred: 2022-05-14T03:36:00.302 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T03:36:00.302 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T03:36:00.302 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T03:36:00.302 INFO:journalctl@ceph.mon.b.smithi047.stdout: return f(*arg) 2022-05-14T03:36:00.302 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T03:36:00.303 INFO:journalctl@ceph.mon.b.smithi047.stdout: self._write_client_files(client_files, host) 2022-05-14T03:36:00.303 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T03:36:00.303 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T03:36:00.304 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T03:36:00.304 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T03:36:00.304 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T03:36:00.304 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.event_loop.get_result(coro) 2022-05-14T03:36:00.304 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T03:36:00.305 INFO:journalctl@ceph.mon.b.smithi047.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T03:36:00.305 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T03:36:00.305 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.__get_result() 2022-05-14T03:36:00.305 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T03:36:00.306 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise self._exception 2022-05-14T03:36:00.306 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T03:36:00.306 INFO:journalctl@ceph.mon.b.smithi047.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T03:36:00.306 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T03:36:00.306 INFO:journalctl@ceph.mon.b.smithi047.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T03:36:00.307 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T03:36:00.307 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise 2022-05-14T03:36:00.307 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T03:36:00.307 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.gen.throw(type, value, traceback) 2022-05-14T03:36:00.307 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T03:36:00.308 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OrchestratorError(msg) 2022-05-14T03:36:00.308 INFO:journalctl@ceph.mon.b.smithi047.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:36:00.308 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:35:59 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:36:00.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:36:00 smithi115 conmon[90393]: 2022-05-14T03:36:00.236+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:36:01.201 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:36:00 smithi115 ceph-mon[78593]: pgmap v7798: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:36:01.202 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:36:00 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:36:01.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:36:00 smithi047 ceph-mon[77147]: pgmap v7798: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:36:01.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:36:00 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:36:01.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:36:01 smithi115 conmon[90393]: 2022-05-14T03:36:01.198+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:36:02.241 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:36:01 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:36:02.241 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:36:01 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 14895 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:36:02.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:36:01 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:36:02.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:36:01 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 14895 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:36:02.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:36:02 smithi115 conmon[90393]: 2022-05-14T03:36:02.238+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:36:03.234 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:36:02 smithi115 ceph-mon[78593]: pgmap v7799: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:36:03.235 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:36:02 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:36:03.235 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:36:02 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:36:03.235 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:36:02 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:36:03.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:36:02 smithi047 ceph-mon[77147]: pgmap v7799: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:36:03.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:36:02 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:36:03.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:36:02 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:36:03.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:36:02 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:36:03.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:36:03 smithi115 conmon[90393]: 2022-05-14T03:36:03.231+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:36:04.210 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:36:03 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:36:04.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:36:03 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:36:04.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:36:04 smithi115 conmon[90393]: 2022-05-14T03:36:04.207+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:36:05.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:36:04 smithi047 ceph-mon[77147]: pgmap v7800: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:36:05.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:36:04 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:36:05.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:36:04 smithi115 ceph-mon[78593]: pgmap v7800: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:36:05.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:36:04 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:36:05.360 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:36:05 smithi115 conmon[90393]: 2022-05-14T03:36:05.198+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:36:06.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:36:05 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:36:06.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:36:05 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:36:06.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:36:06 smithi115 conmon[90393]: 2022-05-14T03:36:06.185+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:36:07.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:36:06 smithi047 ceph-mon[77147]: pgmap v7801: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:36:07.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:36:06 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:36:07.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:36:06 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 14900 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:36:07.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:36:06 smithi115 ceph-mon[78593]: pgmap v7801: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:36:07.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:36:06 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:36:07.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:36:06 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 14900 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:36:07.360 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:36:07 smithi115 conmon[90393]: 2022-05-14T03:36:07.207+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:36:08.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:36:07 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:36:08.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:36:07 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:36:08.360 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:36:08 smithi115 conmon[90393]: 2022-05-14T03:36:08.164+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:36:09.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:36:08 smithi047 ceph-mon[77147]: pgmap v7802: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:36:09.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:36:08 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:36:09.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:36:09 smithi115 conmon[90393]: 2022-05-14T03:36:09.167+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:36:09.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:36:08 smithi115 ceph-mon[78593]: pgmap v7802: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:36:09.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:36:08 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:36:10.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:36:09 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:36:10.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:36:10 smithi115 conmon[90393]: 2022-05-14T03:36:10.208+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:36:10.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:36:09 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:36:11.254 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:36:10 smithi115 ceph-mon[78593]: pgmap v7803: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:36:11.254 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:36:10 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:36:11.255 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:36:10 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:36:11.255 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:36:10 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:36:11.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:36:10 smithi047 ceph-mon[77147]: pgmap v7803: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:36:11.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:36:10 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:36:11.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:36:10 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:36:11.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:36:10 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:36:11.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:36:11 smithi115 conmon[90393]: 2022-05-14T03:36:11.251+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:36:12.265 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:36:11 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:36:12.266 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:36:11 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 14905 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:36:12.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:36:11 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:36:12.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:36:11 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 14905 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:36:12.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:36:12 smithi115 conmon[90393]: 2022-05-14T03:36:12.262+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:36:13.276 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:36:12 smithi115 ceph-mon[78593]: pgmap v7804: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:36:13.276 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:36:12 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:36:13.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:36:12 smithi047 ceph-mon[77147]: pgmap v7804: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:36:13.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:36:12 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:36:13.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:36:13 smithi115 conmon[90393]: 2022-05-14T03:36:13.273+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:36:14.244 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:36:13 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:36:14.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:36:13 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:36:14.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:36:14 smithi115 conmon[90393]: 2022-05-14T03:36:14.241+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:36:15.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:36:14 smithi047 ceph-mon[77147]: pgmap v7805: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:36:15.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:36:14 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:36:15.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:36:14 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:36:15.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:36:14 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:36:15.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:36:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T03:36:15.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:36:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T03:36:15.360 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:36:15 smithi115 conmon[90393]: 2022-05-14T03:36:15.210+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:36:15.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:36:14 smithi115 ceph-mon[78593]: pgmap v7805: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:36:15.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:36:14 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:36:15.362 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:36:14 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:36:15.362 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:36:14 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:36:15.362 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:36:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T03:36:15.363 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:36:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T03:36:16.253 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:36:15 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:36:16.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:36:15 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:36:16.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:36:16 smithi115 conmon[90393]: 2022-05-14T03:36:16.250+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:36:17.286 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:36:16 smithi115 ceph-mon[78593]: pgmap v7806: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:36:17.287 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:36:16 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:36:17.287 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:36:16 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 14910 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:36:17.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:36:16 smithi047 ceph-mon[77147]: pgmap v7806: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:36:17.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:36:16 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:36:17.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:36:16 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 14910 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:36:17.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:36:17 smithi115 conmon[90393]: 2022-05-14T03:36:17.284+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:36:18.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:36:17 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:36:18.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:36:18 smithi115 conmon[90393]: 2022-05-14T03:36:18.241+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:36:18.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:36:18 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:36:19.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:36:19 smithi047 ceph-mon[77147]: pgmap v7807: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:36:19.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:36:19 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:36:19.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:36:19 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:36:19.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:36:19 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:36:19.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:36:19 smithi115 conmon[90393]: 2022-05-14T03:36:19.247+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:36:19.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:36:19 smithi115 ceph-mon[78593]: pgmap v7807: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:36:19.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:36:19 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:36:19.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:36:19 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:36:19.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:36:19 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:36:20.271 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:36:20 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:36:20.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:36:20 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:36:20.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:36:20 smithi115 conmon[90393]: 2022-05-14T03:36:20.268+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:36:21.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:36:21 smithi047 ceph-mon[77147]: pgmap v7808: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:36:21.424 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:36:21 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:36:21.425 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:36:21 smithi115 conmon[90393]: 2022-05-14T03:36:21.220+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:36:21.426 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:36:21 smithi115 ceph-mon[78593]: pgmap v7808: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:36:21.426 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:36:21 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:36:22.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:36:22 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:36:22.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:36:22 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 14915 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:36:22.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:36:22 smithi115 conmon[90393]: 2022-05-14T03:36:22.208+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:36:22.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:36:22 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:36:22.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:36:22 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 14915 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:36:23.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:36:23 smithi047 ceph-mon[77147]: pgmap v7809: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:36:23.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:36:23 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:36:23.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:36:23 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:36:23.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:36:23 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:36:23.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:36:23 smithi115 conmon[90393]: 2022-05-14T03:36:23.199+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:36:23.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:36:23 smithi115 ceph-mon[78593]: pgmap v7809: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:36:23.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:36:23 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:36:23.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:36:23 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:36:23.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:36:23 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:36:24.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:36:24 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:36:24.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:36:24 smithi115 conmon[90393]: 2022-05-14T03:36:24.205+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:36:24.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:36:24 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:36:25.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:36:25 smithi047 ceph-mon[77147]: pgmap v7810: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:36:25.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:36:25 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:36:25.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:36:25 smithi115 conmon[90393]: 2022-05-14T03:36:25.193+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:36:25.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:36:25 smithi115 ceph-mon[78593]: pgmap v7810: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:36:25.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:36:25 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:36:26.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:36:26 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:36:26.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:36:26 smithi115 conmon[90393]: 2022-05-14T03:36:26.243+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:36:26.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:36:26 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:36:27.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:36:27 smithi047 ceph-mon[77147]: pgmap v7811: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:36:27.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:36:27 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:36:27.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:36:27 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 14920 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:36:27.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:36:27 smithi115 conmon[90393]: 2022-05-14T03:36:27.210+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:36:27.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:36:27 smithi115 ceph-mon[78593]: pgmap v7811: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:36:27.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:36:27 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:36:27.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:36:27 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 14920 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:36:28.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:36:28 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:36:28.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:36:28 smithi047 ceph-mon[77147]: pgmap v7812: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:36:28.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:36:28 smithi115 conmon[90393]: 2022-05-14T03:36:28.197+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:36:28.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:36:28 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:36:28.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:36:28 smithi115 ceph-mon[78593]: pgmap v7812: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:36:29.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:36:29 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:36:29.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:36:29 smithi115 conmon[90393]: 2022-05-14T03:36:29.233+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:36:29.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:36:29 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:36:30.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:36:30 smithi115 conmon[90393]: 2022-05-14T03:36:30.195+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:36:30.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:36:30 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:36:30.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:36:30 smithi115 ceph-mon[78593]: pgmap v7813: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:36:30.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:36:30 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:36:30.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:36:30 smithi047 ceph-mon[77147]: pgmap v7813: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:36:31.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:36:31 smithi115 conmon[90393]: 2022-05-14T03:36:31.224+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:36:31.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:36:31 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:36:31.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:36:31 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:36:31.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:36:31 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:36:31.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:36:31 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:36:31.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:36:31 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:36:31.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:36:31 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:36:32.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:36:32 smithi115 conmon[90393]: 2022-05-14T03:36:32.211+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:36:32.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:36:32 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:36:32.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:36:32 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 14925 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:36:32.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:36:32 smithi115 ceph-mon[78593]: pgmap v7814: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:36:32.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:36:32 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:36:32.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:36:32 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 14925 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:36:32.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:36:32 smithi047 ceph-mon[77147]: pgmap v7814: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:36:33.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:36:33 smithi115 conmon[90393]: 2022-05-14T03:36:33.223+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:36:34.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:36:33 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:36:34.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:36:33 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:36:34.610 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:36:34 smithi115 conmon[90393]: 2022-05-14T03:36:34.202+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:36:35.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:36:34 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:36:35.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:36:34 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:36:35.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:36:34 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:36:35.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:36:34 smithi047 ceph-mon[77147]: pgmap v7815: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:36:35.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:36:34 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:36:35.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:36:34 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:36:35.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:36:34 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:36:35.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:36:34 smithi115 ceph-mon[78593]: pgmap v7815: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:36:35.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:36:35 smithi115 conmon[90393]: 2022-05-14T03:36:35.156+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:36:36.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:36:35 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:36:36.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:36:35 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:36:36.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:36:36 smithi115 conmon[90393]: 2022-05-14T03:36:36.135+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:36:37.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:36:36 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:36:37.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:36:36 smithi047 ceph-mon[77147]: pgmap v7816: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:36:37.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:36:36 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:36:37.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:36:36 smithi115 ceph-mon[78593]: pgmap v7816: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:36:37.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:36:37 smithi115 conmon[90393]: 2022-05-14T03:36:37.154+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:36:38.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:36:37 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:36:38.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:36:37 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 14930 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:36:38.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:36:37 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:36:38.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:36:37 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 14930 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:36:38.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:36:38 smithi115 conmon[90393]: 2022-05-14T03:36:38.194+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:36:39.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:36:38 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:36:39.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:36:38 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:36:39.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:36:38 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:36:39.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:36:38 smithi047 ceph-mon[77147]: pgmap v7817: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:36:39.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:36:38 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:36:39.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:36:38 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:36:39.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:36:38 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:36:39.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:36:38 smithi115 ceph-mon[78593]: pgmap v7817: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:36:39.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:36:39 smithi115 conmon[90393]: 2022-05-14T03:36:39.217+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:36:40.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:36:39 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:36:40.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:36:39 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:36:40.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:36:40 smithi115 conmon[90393]: 2022-05-14T03:36:40.218+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:36:41.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:36:40 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:36:41.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:36:40 smithi047 ceph-mon[77147]: pgmap v7818: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:36:41.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:36:40 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:36:41.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:36:40 smithi115 ceph-mon[78593]: pgmap v7818: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:36:41.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:36:41 smithi115 conmon[90393]: 2022-05-14T03:36:41.234+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:36:42.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:36:41 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:36:42.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:36:41 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:36:42.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:36:42 smithi115 conmon[90393]: 2022-05-14T03:36:42.211+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:36:43.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:36:42 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:36:43.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:36:42 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:36:43.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:36:42 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:36:43.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:36:42 smithi047 ceph-mon[77147]: pgmap v7819: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:36:43.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:36:42 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:36:43.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:36:42 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:36:43.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:36:42 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:36:43.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:36:42 smithi115 ceph-mon[78593]: pgmap v7819: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:36:43.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:36:43 smithi115 conmon[90393]: 2022-05-14T03:36:43.244+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:36:44.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:36:43 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:36:44.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:36:43 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:36:44.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:36:44 smithi115 conmon[90393]: 2022-05-14T03:36:44.271+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:36:45.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:36:44 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:36:45.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:36:44 smithi047 ceph-mon[77147]: pgmap v7820: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:36:45.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:36:44 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:36:45.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:36:44 smithi115 ceph-mon[78593]: pgmap v7820: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:36:45.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:36:45 smithi115 conmon[90393]: 2022-05-14T03:36:45.229+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:36:46.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:36:45 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:36:46.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:36:45 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:36:46.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:36:46 smithi115 conmon[90393]: 2022-05-14T03:36:46.262+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:36:47.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:36:46 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:36:47.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:36:46 smithi047 ceph-mon[77147]: pgmap v7821: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:36:47.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:36:46 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 14935 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:36:47.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:36:46 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:36:47.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:36:46 smithi115 ceph-mon[78593]: pgmap v7821: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:36:47.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:36:46 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 14935 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:36:47.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:36:47 smithi115 conmon[90393]: 2022-05-14T03:36:47.292+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:36:48.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:36:47 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:36:48.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:36:47 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:36:48.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:36:48 smithi115 conmon[90393]: 2022-05-14T03:36:48.304+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:36:49.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:36:48 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:36:49.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:36:48 smithi047 ceph-mon[77147]: pgmap v7822: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:36:49.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:36:48 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:36:49.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:36:48 smithi115 ceph-mon[78593]: pgmap v7822: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:36:49.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:36:49 smithi115 conmon[90393]: 2022-05-14T03:36:49.338+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:36:50.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:36:49 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:36:50.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:36:49 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:36:50.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:36:50 smithi115 conmon[90393]: 2022-05-14T03:36:50.326+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:36:51.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:36:50 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:36:51.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:36:50 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:36:51.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:36:50 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:36:51.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:36:50 smithi047 ceph-mon[77147]: pgmap v7823: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:36:51.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:36:50 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:36:51.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:36:50 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:36:51.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:36:50 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:36:51.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:36:50 smithi115 ceph-mon[78593]: pgmap v7823: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:36:51.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:36:51 smithi115 conmon[90393]: 2022-05-14T03:36:51.283+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:36:52.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:36:51 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:36:52.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:36:51 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 14945 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:36:52.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:36:51 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:36:52.125 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:36:51 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 14945 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:36:52.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:36:52 smithi115 conmon[90393]: 2022-05-14T03:36:52.288+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:36:53.049 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:36:52 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:36:53.050 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:36:52 smithi047 ceph-mon[77147]: pgmap v7824: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:36:53.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:36:52 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:36:53.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:36:52 smithi115 ceph-mon[78593]: pgmap v7824: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:36:53.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:36:53 smithi115 conmon[90393]: 2022-05-14T03:36:53.292+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:36:54.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:36:53 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:36:54.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:36:53 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:36:54.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:36:54 smithi115 conmon[90393]: 2022-05-14T03:36:54.282+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:36:55.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:36:54 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:36:55.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:36:54 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:36:55.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:36:54 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:36:55.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:36:54 smithi047 ceph-mon[77147]: pgmap v7825: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:36:55.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:36:54 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:36:55.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:36:54 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:36:55.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:36:54 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:36:55.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:36:54 smithi115 ceph-mon[78593]: pgmap v7825: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:36:55.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:36:55 smithi115 conmon[90393]: 2022-05-14T03:36:55.275+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:36:56.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:36:55 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:36:56.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:36:55 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:36:56.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:36:56 smithi115 conmon[90393]: 2022-05-14T03:36:56.230+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:36:57.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:36:56 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:36:57.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:36:56 smithi047 ceph-mon[77147]: pgmap v7826: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:36:57.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:36:56 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:36:57.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:36:56 smithi115 ceph-mon[78593]: pgmap v7826: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:36:57.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:36:57 smithi115 conmon[90393]: 2022-05-14T03:36:57.210+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:36:58.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:36:57 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:36:58.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:36:57 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 14950 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:36:58.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:36:57 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:36:58.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:36:57 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 14950 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:36:58.497 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:36:58 smithi115 conmon[90393]: 2022-05-14T03:36:58.213+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:36:59.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:36:58 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:36:59.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:36:58 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:36:59.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:36:58 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:36:59.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:36:58 smithi047 ceph-mon[77147]: pgmap v7827: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:36:59.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:36:58 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T03:36:59.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:36:58 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T03:36:59.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:36:58 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T03:36:59.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:36:58 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:36:59.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:36:58 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:36:59.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:36:58 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:36:59.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:36:58 smithi115 ceph-mon[78593]: pgmap v7827: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:36:59.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:36:58 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T03:36:59.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:36:58 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T03:36:59.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:36:58 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T03:36:59.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:36:59 smithi115 conmon[90393]: 2022-05-14T03:36:59.167+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:37:00.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:36:59 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:37:00.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:36:59 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:37:00.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:37:00 smithi115 conmon[90393]: 2022-05-14T03:37:00.130+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:37:01.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:37:00 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:37:01.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:37:00 smithi115 ceph-mon[78593]: pgmap v7828: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:37:01.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:37:00 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:37:01.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:37:00 smithi047 ceph-mon[77147]: pgmap v7828: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:37:01.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:37:01 smithi115 conmon[90393]: 2022-05-14T03:37:01.116+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:37:02.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:37:01 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:37:02.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:37:01 smithi047 conmon[78521]: 2022-05-14T03:37:01.551+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:37:02.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:37:01 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T03:37:02.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:37:01 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:37:02.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:37:01 smithi047 conmon[78521]: yield 2022-05-14T03:37:02.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:37:01 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:37:02.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:37:01 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:37:02.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:37:01 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:37:02.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:37:01 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T03:37:02.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:37:01 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:37:02.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:37:01 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T03:37:02.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:37:01 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:37:02.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:37:01 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T03:37:02.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:37:01 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:37:02.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:37:01 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T03:37:02.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:37:01 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:37:02.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:37:01 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T03:37:02.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:37:01 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:37:02.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:37:01 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:37:02.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:37:01 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:37:02.080 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:37:01 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:37:02.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:37:02 smithi115 conmon[90393]: 2022-05-14T03:37:02.078+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:37:03.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:37:03 smithi115 conmon[90393]: 2022-05-14T03:37:03.029+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:37:03.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:37:02 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:37:03.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:37:02 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:37:03.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T03:37:03.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:37:03.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T03:37:03.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:37:03.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:37:03.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:37:03.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T03:37:03.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:37:03.113 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T03:37:03.113 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:37:03.113 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T03:37:03.113 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:37:03.114 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T03:37:03.114 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:37:03.114 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T03:37:03.114 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:37:03.114 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:37:03.115 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:37:03.115 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:37:02 smithi115 ceph-mon[78593]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T03:37:03.115 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:37:02 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:37:03.115 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:37:02 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:37:03.115 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:37:02 smithi115 ceph-mon[78593]: pgmap v7829: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:37:03.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:37:02 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:37:03.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:37:02 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:37:03.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T03:37:03.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:37:03.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T03:37:03.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:37:03.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:37:03.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:37:03.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T03:37:03.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:37:03.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T03:37:03.298 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:37:03.298 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T03:37:03.298 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:37:03.299 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T03:37:03.299 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:37:03.300 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T03:37:03.301 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:37:03.301 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:37:03.301 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:37:03.302 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:37:02 smithi047 ceph-mon[77147]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T03:37:03.302 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:37:02 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:37:03.303 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:37:02 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:37:03.303 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:37:02 smithi047 ceph-mon[77147]: pgmap v7829: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:37:04.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:37:03 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:37:04.110 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:37:04 smithi115 conmon[90393]: 2022-05-14T03:37:04.044+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:37:04.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:37:03 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:37:05.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:37:04 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:37:05.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:37:04 smithi047 ceph-mon[77147]: pgmap v7830: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:37:05.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:37:04 smithi047 conmon[78521]: 2022-05-14T03:37:04.624+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:37:05.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:37:04 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T03:37:05.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:37:04 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:37:05.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:37:04 smithi047 conmon[78521]: yield 2022-05-14T03:37:05.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:37:04 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:37:05.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:37:04 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:37:05.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:37:04 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:37:05.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:37:04 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T03:37:05.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:37:04 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:37:05.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:37:04 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T03:37:05.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:37:04 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:37:05.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:37:04 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T03:37:05.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:37:04 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:37:05.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:37:04 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T03:37:05.048 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:37:04 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:37:05.048 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:37:04 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T03:37:05.048 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:37:04 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:37:05.048 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:37:04 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:37:05.048 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:37:04 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:37:05.049 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:37:04 smithi047 conmon[78521]: 2022-05-14T03:37:04.625+0000 7f5341f7c700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T03:37:05.049 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:37:04 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T03:37:05.049 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:37:04 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:37:05.049 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:37:04 smithi047 conmon[78521]: yield 2022-05-14T03:37:05.050 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:37:04 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:37:05.050 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:37:04 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:37:05.050 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:37:04 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:37:05.050 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:37:04 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T03:37:05.051 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:37:04 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:37:05.051 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:37:04 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T03:37:05.051 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:37:04 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:37:05.051 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:37:04 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T03:37:05.051 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:37:04 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:37:05.052 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:37:04 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T03:37:05.052 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:37:04 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:37:05.052 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:37:04 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T03:37:05.052 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:37:04 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:37:05.053 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:37:04 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:37:05.053 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:37:04 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:37:05.053 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:37:04 smithi047 conmon[78521]: 2022-05-14T03:37:05.054 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:37:04 smithi047 conmon[78521]: During handling of the above exception, another exception occurred: 2022-05-14T03:37:05.054 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:37:04 smithi047 conmon[78521]: 2022-05-14T03:37:05.054 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:37:04 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T03:37:05.055 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:37:04 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T03:37:05.055 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:37:04 smithi047 conmon[78521]: return f(*arg) 2022-05-14T03:37:05.056 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:37:04 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T03:37:05.056 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:37:04 smithi047 conmon[78521]: self._write_client_files(client_files, host) 2022-05-14T03:37:05.056 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:37:04 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T03:37:05.057 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:37:04 smithi047 conmon[78521]: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T03:37:05.057 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:37:04 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T03:37:05.058 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:37:04 smithi047 conmon[78521]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T03:37:05.058 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:37:04 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T03:37:05.058 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:37:04 smithi047 conmon[78521]: return self.event_loop.get_result(coro) 2022-05-14T03:37:05.059 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:37:04 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T03:37:05.059 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:37:04 smithi047 conmon[78521]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T03:37:05.060 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:37:04 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T03:37:05.060 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:37:04 smithi047 conmon[78521]: return self.__get_result() 2022-05-14T03:37:05.061 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:37:04 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T03:37:05.061 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:37:04 smithi047 conmon[78521]: raise self._exception 2022-05-14T03:37:05.061 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:37:04 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T03:37:05.062 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:37:04 smithi047 conmon[78521]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T03:37:05.062 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:37:04 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T03:37:05.063 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:37:04 smithi047 conmon[78521]: conn = await self._remote_connection(host, addr) 2022-05-14T03:37:05.063 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:37:04 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T03:37:05.063 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:37:04 smithi047 conmon[78521]: raise 2022-05-14T03:37:05.064 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:37:04 smithi047 conmon[78521]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T03:37:05.064 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:37:04 smithi047 conmon[78521]: self.gen.throw(type, value, traceback) 2022-05-14T03:37:05.064 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:37:04 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T03:37:05.065 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:37:04 smithi047 conmon[78521]: raise OrchestratorError(msg) 2022-05-14T03:37:05.065 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:37:04 smithi047 conmon[78521]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:37:05.072 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:37:04 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:37:05.072 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:37:04 smithi115 ceph-mon[78593]: pgmap v7830: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:37:05.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:37:05 smithi115 conmon[90393]: 2022-05-14T03:37:05.069+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:37:06.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:37:05 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:37:06.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:37:05 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:37:06.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T03:37:06.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:37:06.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T03:37:06.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:37:06.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:37:06.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:37:06.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T03:37:06.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:37:06.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T03:37:06.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:37:06.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T03:37:06.113 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:37:06.113 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T03:37:06.113 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:37:06.113 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T03:37:06.114 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:37:06.114 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:37:06.114 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:37:06.114 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:37:05 smithi115 ceph-mon[78593]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T03:37:06.114 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T03:37:06.115 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:37:06.115 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T03:37:06.115 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:37:06.115 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:37:06.116 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:37:06.116 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T03:37:06.116 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:37:06.116 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T03:37:06.116 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:37:06.117 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T03:37:06.117 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:37:06.117 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T03:37:06.117 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:37:06.117 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T03:37:06.118 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:37:06.118 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:37:06.118 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:37:06.118 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T03:37:06.118 INFO:journalctl@ceph.mon.c.smithi115.stdout: During handling of the above exception, another exception occurred: 2022-05-14T03:37:06.119 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T03:37:06.119 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T03:37:06.119 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T03:37:06.119 INFO:journalctl@ceph.mon.c.smithi115.stdout: return f(*arg) 2022-05-14T03:37:06.120 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T03:37:06.120 INFO:journalctl@ceph.mon.c.smithi115.stdout: self._write_client_files(client_files, host) 2022-05-14T03:37:06.120 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T03:37:06.120 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T03:37:06.120 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T03:37:06.121 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T03:37:06.121 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T03:37:06.121 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.event_loop.get_result(coro) 2022-05-14T03:37:06.121 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T03:37:06.121 INFO:journalctl@ceph.mon.c.smithi115.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T03:37:06.122 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T03:37:06.122 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.__get_result() 2022-05-14T03:37:06.122 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T03:37:06.122 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise self._exception 2022-05-14T03:37:06.122 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T03:37:06.123 INFO:journalctl@ceph.mon.c.smithi115.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T03:37:06.123 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T03:37:06.123 INFO:journalctl@ceph.mon.c.smithi115.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T03:37:06.123 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T03:37:06.124 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise 2022-05-14T03:37:06.124 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T03:37:06.124 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.gen.throw(type, value, traceback) 2022-05-14T03:37:06.124 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T03:37:06.124 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OrchestratorError(msg) 2022-05-14T03:37:06.125 INFO:journalctl@ceph.mon.c.smithi115.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:37:06.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:37:05 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:37:06.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:37:05 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:37:06.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T03:37:06.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:37:06.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T03:37:06.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:37:06.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:37:06.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:37:06.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T03:37:06.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:37:06.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T03:37:06.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:37:06.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T03:37:06.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:37:06.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T03:37:06.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:37:06.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T03:37:06.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:37:06.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:37:06.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:37:06.297 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:37:05 smithi047 ceph-mon[77147]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T03:37:06.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T03:37:06.298 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:37:06.298 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T03:37:06.298 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:37:06.298 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:37:06.299 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:37:06.299 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T03:37:06.299 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:37:06.299 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T03:37:06.299 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:37:06.300 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T03:37:06.300 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:37:06.300 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T03:37:06.300 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:37:06.300 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T03:37:06.301 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:37:06.301 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:37:06.301 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:37:06.301 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T03:37:06.302 INFO:journalctl@ceph.mon.b.smithi047.stdout: During handling of the above exception, another exception occurred: 2022-05-14T03:37:06.302 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T03:37:06.302 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T03:37:06.302 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T03:37:06.303 INFO:journalctl@ceph.mon.b.smithi047.stdout: return f(*arg) 2022-05-14T03:37:06.303 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T03:37:06.303 INFO:journalctl@ceph.mon.b.smithi047.stdout: self._write_client_files(client_files, host) 2022-05-14T03:37:06.303 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T03:37:06.303 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T03:37:06.304 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T03:37:06.304 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T03:37:06.304 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T03:37:06.304 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.event_loop.get_result(coro) 2022-05-14T03:37:06.305 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T03:37:06.305 INFO:journalctl@ceph.mon.b.smithi047.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T03:37:06.305 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T03:37:06.306 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.__get_result() 2022-05-14T03:37:06.306 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T03:37:06.306 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise self._exception 2022-05-14T03:37:06.306 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T03:37:06.306 INFO:journalctl@ceph.mon.b.smithi047.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T03:37:06.307 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T03:37:06.307 INFO:journalctl@ceph.mon.b.smithi047.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T03:37:06.307 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T03:37:06.308 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise 2022-05-14T03:37:06.308 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T03:37:06.308 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.gen.throw(type, value, traceback) 2022-05-14T03:37:06.308 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T03:37:06.308 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OrchestratorError(msg) 2022-05-14T03:37:06.309 INFO:journalctl@ceph.mon.b.smithi047.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:37:06.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:37:06 smithi115 conmon[90393]: 2022-05-14T03:37:06.112+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:37:07.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:37:06 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:37:07.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:37:06 smithi115 ceph-mon[78593]: pgmap v7831: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:37:07.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:37:06 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 14960 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:37:07.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:37:06 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:37:07.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:37:06 smithi047 ceph-mon[77147]: pgmap v7831: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:37:07.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:37:06 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 14960 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:37:07.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:37:07 smithi115 conmon[90393]: 2022-05-14T03:37:07.128+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:37:08.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:37:07 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:37:08.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:37:07 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:37:08.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:37:08 smithi115 conmon[90393]: 2022-05-14T03:37:08.168+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:37:09.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:37:08 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:37:09.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:37:08 smithi115 ceph-mon[78593]: pgmap v7832: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:37:09.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:37:08 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:37:09.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:37:08 smithi047 ceph-mon[77147]: pgmap v7832: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:37:09.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:37:09 smithi115 conmon[90393]: 2022-05-14T03:37:09.121+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:37:10.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:37:10 smithi115 conmon[90393]: 2022-05-14T03:37:10.073+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:37:10.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:37:09 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:37:10.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:37:09 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:37:11.096 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:37:10 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:37:11.097 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:37:10 smithi115 ceph-mon[78593]: pgmap v7833: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:37:11.097 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:37:10 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:37:11.097 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:37:10 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:37:11.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:37:10 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:37:11.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:37:10 smithi047 ceph-mon[77147]: pgmap v7833: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:37:11.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:37:10 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:37:11.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:37:10 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:37:11.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:37:11 smithi115 conmon[90393]: 2022-05-14T03:37:11.093+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:37:12.089 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:37:11 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:37:12.090 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:37:11 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 14965 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:37:12.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:37:11 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:37:12.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:37:11 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 14965 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:37:12.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:37:12 smithi115 conmon[90393]: 2022-05-14T03:37:12.086+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:37:13.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:37:12 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:37:13.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:37:12 smithi115 ceph-mon[78593]: pgmap v7834: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:37:13.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:37:12 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:37:13.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:37:12 smithi047 ceph-mon[77147]: pgmap v7834: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:37:13.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:37:13 smithi115 conmon[90393]: 2022-05-14T03:37:13.104+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:37:14.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:37:14 smithi115 conmon[90393]: 2022-05-14T03:37:14.083+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:37:14.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:37:13 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:37:14.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:37:13 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:37:15.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:37:15 smithi115 conmon[90393]: 2022-05-14T03:37:15.083+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:37:15.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:37:14 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:37:15.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:37:14 smithi115 ceph-mon[78593]: pgmap v7835: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:37:15.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:37:14 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:37:15.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:37:14 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:37:15.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:37:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T03:37:15.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:37:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T03:37:15.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:37:14 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:37:15.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:37:14 smithi047 ceph-mon[77147]: pgmap v7835: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:37:15.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:37:14 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:37:15.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:37:14 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:37:15.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:37:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T03:37:15.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:37:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T03:37:16.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:37:16 smithi115 conmon[90393]: 2022-05-14T03:37:16.063+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:37:16.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:37:15 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:37:16.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:37:15 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:37:17.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:37:17 smithi115 conmon[90393]: 2022-05-14T03:37:17.047+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:37:17.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:37:16 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:37:17.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:37:16 smithi115 ceph-mon[78593]: pgmap v7836: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:37:17.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:37:16 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 14970 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:37:17.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:37:16 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:37:17.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:37:16 smithi047 ceph-mon[77147]: pgmap v7836: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:37:17.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:37:16 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 14970 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:37:18.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:37:18 smithi115 conmon[90393]: 2022-05-14T03:37:18.013+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:37:18.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:37:17 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:37:18.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:37:17 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:37:19.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:37:19 smithi115 conmon[90393]: 2022-05-14T03:37:19.062+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:37:19.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:37:18 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:37:19.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:37:18 smithi115 ceph-mon[78593]: pgmap v7837: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:37:19.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:37:18 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:37:19.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:37:18 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:37:19.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:37:18 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:37:19.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:37:18 smithi047 ceph-mon[77147]: pgmap v7837: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:37:19.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:37:18 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:37:19.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:37:18 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:37:20.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:37:19 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:37:20.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:37:20 smithi115 conmon[90393]: 2022-05-14T03:37:20.068+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:37:20.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:37:19 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:37:21.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:37:20 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:37:21.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:37:20 smithi047 ceph-mon[77147]: pgmap v7838: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:37:21.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:37:21 smithi115 conmon[90393]: 2022-05-14T03:37:21.101+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:37:21.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:37:20 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:37:21.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:37:20 smithi115 ceph-mon[78593]: pgmap v7838: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:37:22.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:37:21 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:37:22.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:37:21 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 14974 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:37:22.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:37:22 smithi115 conmon[90393]: 2022-05-14T03:37:22.093+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:37:22.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:37:21 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:37:22.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:37:21 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 14974 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:37:23.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:37:22 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:37:23.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:37:22 smithi047 ceph-mon[77147]: pgmap v7839: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:37:23.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:37:22 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:37:23.295 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:37:22 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:37:23.361 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:37:23 smithi115 conmon[90393]: 2022-05-14T03:37:23.070+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:37:23.362 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:37:22 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:37:23.363 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:37:22 smithi115 ceph-mon[78593]: pgmap v7839: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:37:23.363 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:37:22 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:37:23.364 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:37:22 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:37:24.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:37:23 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:37:24.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:37:24 smithi115 conmon[90393]: 2022-05-14T03:37:24.113+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:37:24.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:37:23 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:37:25.144 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:37:24 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:37:25.144 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:37:24 smithi115 ceph-mon[78593]: pgmap v7840: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:37:25.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:37:24 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:37:25.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:37:24 smithi047 ceph-mon[77147]: pgmap v7840: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:37:25.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:37:25 smithi115 conmon[90393]: 2022-05-14T03:37:25.141+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:37:26.144 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:37:25 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:37:26.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:37:25 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:37:26.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:37:26 smithi115 conmon[90393]: 2022-05-14T03:37:26.141+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:37:27.182 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:37:26 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:37:27.182 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:37:26 smithi115 ceph-mon[78593]: pgmap v7841: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:37:27.183 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:37:26 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 14979 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:37:27.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:37:26 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:37:27.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:37:26 smithi047 ceph-mon[77147]: pgmap v7841: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:37:27.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:37:26 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 14979 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:37:27.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:37:27 smithi115 conmon[90393]: 2022-05-14T03:37:27.179+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:37:28.218 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:37:27 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:37:28.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:37:27 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:37:28.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:37:28 smithi115 conmon[90393]: 2022-05-14T03:37:28.216+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:37:29.225 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:37:28 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:37:29.225 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:37:28 smithi115 ceph-mon[78593]: pgmap v7842: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:37:29.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:37:28 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:37:29.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:37:28 smithi047 ceph-mon[77147]: pgmap v7842: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:37:29.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:37:29 smithi115 conmon[90393]: 2022-05-14T03:37:29.222+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:37:30.245 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:37:29 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:37:30.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:37:29 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:37:30.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:37:30 smithi115 conmon[90393]: 2022-05-14T03:37:30.243+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:37:31.205 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:37:30 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:37:31.205 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:37:30 smithi115 ceph-mon[78593]: pgmap v7843: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:37:31.205 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:37:30 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:37:31.206 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:37:30 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:37:31.206 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:37:30 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:37:31.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:37:30 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:37:31.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:37:30 smithi047 ceph-mon[77147]: pgmap v7843: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:37:31.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:37:30 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:37:31.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:37:30 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:37:31.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:37:30 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:37:31.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:37:31 smithi115 conmon[90393]: 2022-05-14T03:37:31.202+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:37:32.193 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:37:31 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:37:32.194 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:37:31 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 14985 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:37:32.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:37:31 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:37:32.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:37:31 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 14985 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:37:32.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:37:32 smithi115 conmon[90393]: 2022-05-14T03:37:32.190+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:37:33.216 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:37:32 smithi115 ceph-mon[78593]: pgmap v7844: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:37:33.216 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:37:32 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:37:33.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:37:32 smithi047 ceph-mon[77147]: pgmap v7844: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:37:33.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:37:32 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:37:33.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:37:33 smithi115 conmon[90393]: 2022-05-14T03:37:33.213+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:37:34.192 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:37:33 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:37:34.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:37:33 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:37:34.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:37:34 smithi115 conmon[90393]: 2022-05-14T03:37:34.190+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:37:35.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:37:34 smithi047 ceph-mon[77147]: pgmap v7845: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:37:35.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:37:34 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:37:35.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:37:34 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:37:35.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:37:34 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:37:35.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:37:35 smithi115 conmon[90393]: 2022-05-14T03:37:35.153+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:37:35.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:37:34 smithi115 ceph-mon[78593]: pgmap v7845: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:37:35.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:37:34 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:37:35.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:37:34 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:37:35.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:37:34 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:37:36.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:37:35 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:37:36.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:37:36 smithi115 conmon[90393]: 2022-05-14T03:37:36.121+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:37:36.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:37:35 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:37:37.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:37:36 smithi047 ceph-mon[77147]: pgmap v7846: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:37:37.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:37:36 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:37:37.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:37:36 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 14990 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:37:37.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:37:37 smithi115 conmon[90393]: 2022-05-14T03:37:37.151+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:37:37.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:37:36 smithi115 ceph-mon[78593]: pgmap v7846: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:37:37.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:37:36 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:37:37.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:37:36 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 14990 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:37:38.189 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:37:37 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:37:38.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:37:37 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:37:38.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:37:38 smithi115 conmon[90393]: 2022-05-14T03:37:38.187+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:37:39.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:37:38 smithi047 ceph-mon[77147]: pgmap v7847: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:37:39.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:37:38 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:37:39.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:37:38 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:37:39.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:37:38 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:37:39.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:37:39 smithi115 conmon[90393]: 2022-05-14T03:37:39.158+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:37:39.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:37:38 smithi115 ceph-mon[78593]: pgmap v7847: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:37:39.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:37:38 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:37:39.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:37:38 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:37:39.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:37:38 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:37:40.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:37:39 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:37:40.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:37:40 smithi115 conmon[90393]: 2022-05-14T03:37:40.142+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:37:40.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:37:39 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:37:41.186 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:37:40 smithi115 ceph-mon[78593]: pgmap v7848: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:37:41.186 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:37:40 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:37:41.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:37:40 smithi047 ceph-mon[77147]: pgmap v7848: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:37:41.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:37:40 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:37:41.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:37:41 smithi115 conmon[90393]: 2022-05-14T03:37:41.184+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:37:42.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:37:41 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:37:42.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:37:41 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 14995 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:37:42.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:37:42 smithi115 conmon[90393]: 2022-05-14T03:37:42.139+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:37:42.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:37:41 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:37:42.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:37:41 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 14995 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:37:43.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:37:42 smithi047 ceph-mon[77147]: pgmap v7849: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:37:43.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:37:42 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:37:43.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:37:42 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:37:43.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:37:42 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:37:43.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:37:43 smithi115 conmon[90393]: 2022-05-14T03:37:43.120+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:37:43.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:37:42 smithi115 ceph-mon[78593]: pgmap v7849: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:37:43.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:37:42 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:37:43.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:37:42 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:37:43.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:37:42 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:37:44.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:37:43 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:37:44.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:37:44 smithi115 conmon[90393]: 2022-05-14T03:37:44.090+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:37:44.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:37:43 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:37:45.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:37:44 smithi047 ceph-mon[77147]: pgmap v7850: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:37:45.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:37:44 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:37:45.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:37:45 smithi115 conmon[90393]: 2022-05-14T03:37:45.109+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:37:45.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:37:44 smithi115 ceph-mon[78593]: pgmap v7850: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:37:45.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:37:44 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:37:46.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:37:45 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:37:46.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:37:46 smithi115 conmon[90393]: 2022-05-14T03:37:46.159+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:37:46.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:37:45 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:37:47.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:37:46 smithi047 ceph-mon[77147]: pgmap v7851: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:37:47.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:37:46 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:37:47.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:37:46 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 15000 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:37:47.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:37:47 smithi115 conmon[90393]: 2022-05-14T03:37:47.180+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:37:47.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:37:46 smithi115 ceph-mon[78593]: pgmap v7851: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:37:47.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:37:46 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:37:47.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:37:46 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 15000 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:37:48.223 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:37:47 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:37:48.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:37:47 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:37:48.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:37:48 smithi115 conmon[90393]: 2022-05-14T03:37:48.220+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:37:49.250 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:37:48 smithi115 ceph-mon[78593]: pgmap v7852: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:37:49.251 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:37:48 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:37:49.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:37:48 smithi047 ceph-mon[77147]: pgmap v7852: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:37:49.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:37:48 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:37:49.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:37:49 smithi115 conmon[90393]: 2022-05-14T03:37:49.248+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:37:50.281 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:37:49 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:37:50.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:37:49 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:37:50.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:37:50 smithi115 conmon[90393]: 2022-05-14T03:37:50.278+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:37:51.262 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:37:50 smithi115 ceph-mon[78593]: pgmap v7853: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:37:51.263 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:37:50 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:37:51.263 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:37:50 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:37:51.263 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:37:50 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:37:51.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:37:50 smithi047 ceph-mon[77147]: pgmap v7853: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:37:51.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:37:50 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:37:51.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:37:50 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:37:51.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:37:50 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:37:51.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:37:51 smithi115 conmon[90393]: 2022-05-14T03:37:51.259+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:37:52.256 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:37:51 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:37:52.257 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:37:51 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 15005 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:37:52.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:37:51 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:37:52.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:37:51 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 15005 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:37:52.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:37:52 smithi115 conmon[90393]: 2022-05-14T03:37:52.254+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:37:53.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:37:52 smithi047 ceph-mon[77147]: pgmap v7854: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:37:53.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:37:52 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:37:53.298 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:37:52 smithi115 ceph-mon[78593]: pgmap v7854: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:37:53.298 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:37:52 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:37:53.610 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:37:53 smithi115 conmon[90393]: 2022-05-14T03:37:53.295+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:37:54.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:37:53 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:37:54.321 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:37:53 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:37:54.610 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:37:54 smithi115 conmon[90393]: 2022-05-14T03:37:54.313+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:37:55.271 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:37:54 smithi115 ceph-mon[78593]: pgmap v7855: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:37:55.272 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:37:54 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:37:55.272 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:37:54 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:37:55.272 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:37:54 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:37:55.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:37:54 smithi047 ceph-mon[77147]: pgmap v7855: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:37:55.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:37:54 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:37:55.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:37:54 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:37:55.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:37:54 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:37:55.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:37:55 smithi115 conmon[90393]: 2022-05-14T03:37:55.269+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:37:56.244 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:37:55 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:37:56.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:37:55 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:37:56.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:37:56 smithi115 conmon[90393]: 2022-05-14T03:37:56.241+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:37:57.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:37:56 smithi047 ceph-mon[77147]: pgmap v7856: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:37:57.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:37:56 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:37:57.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:37:56 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 15010 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:37:57.294 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:37:56 smithi115 ceph-mon[78593]: pgmap v7856: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:37:57.294 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:37:56 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:37:57.294 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:37:56 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 15010 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:37:57.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:37:57 smithi115 conmon[90393]: 2022-05-14T03:37:57.291+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:37:58.253 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:37:57 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:37:58.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:37:57 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:37:58.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:37:58 smithi115 conmon[90393]: 2022-05-14T03:37:58.243+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:37:59.256 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:37:58 smithi115 ceph-mon[78593]: pgmap v7857: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:37:59.256 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:37:58 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:37:59.256 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:37:58 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:37:59.257 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:37:58 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:37:59.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:37:58 smithi047 ceph-mon[77147]: pgmap v7857: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:37:59.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:37:58 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:37:59.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:37:58 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:37:59.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:37:58 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:37:59.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:37:59 smithi115 conmon[90393]: 2022-05-14T03:37:59.253+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:38:00.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:37:59 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:38:00.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:38:00 smithi115 conmon[90393]: 2022-05-14T03:38:00.224+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:38:00.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:37:59 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:38:01.247 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:38:00 smithi115 ceph-mon[78593]: pgmap v7858: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:38:01.248 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:38:00 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:38:01.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:38:00 smithi047 ceph-mon[77147]: pgmap v7858: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:38:01.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:38:00 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:38:01.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:38:01 smithi115 conmon[90393]: 2022-05-14T03:38:01.245+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:38:02.258 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:38:01 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:38:02.258 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:38:01 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 15015 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:38:02.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:38:01 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:38:02.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:38:01 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 15015 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:38:02.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:38:02 smithi115 conmon[90393]: 2022-05-14T03:38:02.254+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:38:03.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:38:02 smithi047 ceph-mon[77147]: pgmap v7859: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:38:03.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:38:02 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:38:03.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:38:02 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:38:03.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:38:02 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:38:03.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:38:02 smithi115 ceph-mon[78593]: pgmap v7859: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:38:03.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:38:02 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:38:03.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:38:02 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:38:03.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:38:02 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:38:03.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:38:03 smithi115 conmon[90393]: 2022-05-14T03:38:03.293+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:38:04.253 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:38:04 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:38:04.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:38:04 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:38:04.610 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:38:04 smithi115 conmon[90393]: 2022-05-14T03:38:04.251+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:38:05.267 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:38:05 smithi115 ceph-mon[78593]: pgmap v7860: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:38:05.268 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:38:05 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:38:05.268 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:38:05 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T03:38:05.268 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:38:05 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T03:38:05.269 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:38:05 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T03:38:05.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:38:05 smithi047 ceph-mon[77147]: pgmap v7860: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:38:05.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:38:05 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:38:05.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:38:05 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T03:38:05.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:38:05 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T03:38:05.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:38:05 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T03:38:05.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:38:05 smithi115 conmon[90393]: 2022-05-14T03:38:05.264+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:38:06.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:38:06 smithi115 conmon[90393]: 2022-05-14T03:38:06.304+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:38:06.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:38:06 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' 2022-05-14T03:38:06.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:38:06 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' 2022-05-14T03:38:06.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:38:06 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:38:06.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:38:06 smithi115 ceph-mon[78593]: pgmap v7861: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:38:06.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:38:06 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' 2022-05-14T03:38:06.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:38:06 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' 2022-05-14T03:38:06.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:38:06 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:38:06.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:38:06 smithi047 ceph-mon[77147]: pgmap v7861: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:38:07.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:38:07 smithi115 conmon[90393]: 2022-05-14T03:38:07.275+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:38:07.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:38:07 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:38:07.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:38:07 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 15020 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:38:07.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:38:07 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:38:07.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:38:07 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 15020 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:38:08.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:38:07 smithi047 conmon[78521]: 2022-05-14T03:38:07.727+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:38:08.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:38:07 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T03:38:08.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:38:07 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:38:08.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:38:07 smithi047 conmon[78521]: yield 2022-05-14T03:38:08.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:38:07 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:38:08.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:38:07 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:38:08.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:38:07 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:38:08.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:38:07 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T03:38:08.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:38:07 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:38:08.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:38:07 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T03:38:08.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:38:07 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:38:08.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:38:07 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T03:38:08.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:38:07 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:38:08.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:38:07 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T03:38:08.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:38:07 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:38:08.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:38:07 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T03:38:08.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:38:07 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:38:08.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:38:07 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:38:08.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:38:07 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:38:08.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:38:08 smithi115 conmon[90393]: 2022-05-14T03:38:08.263+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:38:09.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:38:08 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:38:09.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:38:08 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' 2022-05-14T03:38:09.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:38:08 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:38:09.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T03:38:09.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:38:09.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T03:38:09.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:38:09.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:38:09.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:38:09.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T03:38:09.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:38:09.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T03:38:09.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:38:09.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T03:38:09.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:38:09.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T03:38:09.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:38:09.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T03:38:09.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:38:09.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:38:09.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:38:09.048 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:38:08 smithi047 ceph-mon[77147]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T03:38:09.048 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:38:08 smithi047 ceph-mon[77147]: pgmap v7862: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:38:09.049 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:38:08 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' 2022-05-14T03:38:09.049 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:38:08 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-05-14T03:38:09.049 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:38:08 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-05-14T03:38:09.049 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:38:08 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-05-14T03:38:09.050 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:38:08 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-05-14T03:38:09.050 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:38:08 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-05-14T03:38:09.050 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:38:08 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-05-14T03:38:09.051 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:38:08 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-05-14T03:38:09.051 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:38:08 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-05-14T03:38:09.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:38:08 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:38:09.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:38:08 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' 2022-05-14T03:38:09.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:38:08 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:38:09.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T03:38:09.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:38:09.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T03:38:09.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:38:09.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:38:09.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:38:09.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T03:38:09.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:38:09.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T03:38:09.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:38:09.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T03:38:09.113 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:38:09.113 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T03:38:09.113 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:38:09.113 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T03:38:09.113 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:38:09.114 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:38:09.114 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:38:09.114 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:38:08 smithi115 ceph-mon[78593]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T03:38:09.114 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:38:08 smithi115 ceph-mon[78593]: pgmap v7862: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:38:09.115 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:38:08 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' 2022-05-14T03:38:09.115 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:38:08 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-05-14T03:38:09.115 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:38:08 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-05-14T03:38:09.116 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:38:08 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-05-14T03:38:09.116 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:38:08 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-05-14T03:38:09.116 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:38:08 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-05-14T03:38:09.117 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:38:08 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-05-14T03:38:09.117 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:38:08 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-05-14T03:38:09.117 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:38:08 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-05-14T03:38:09.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:38:09 smithi115 conmon[90393]: 2022-05-14T03:38:09.301+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:38:10.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:38:09 smithi047 ceph-mon[77147]: Adjusting osd_memory_target on smithi047 to 137.2M 2022-05-14T03:38:10.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:38:09 smithi047 ceph-mon[77147]: Unable to set osd_memory_target on smithi047 to 143868239: error parsing value: Value '143868239' is below minimum 939524096 2022-05-14T03:38:10.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:38:09 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:38:10.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:38:09 smithi047 ceph-mon[77147]: Adjusting osd_memory_target on smithi115 to 197.0M 2022-05-14T03:38:10.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:38:09 smithi047 ceph-mon[77147]: Unable to set osd_memory_target on smithi115 to 206629785: error parsing value: Value '206629785' is below minimum 939524096 2022-05-14T03:38:10.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:38:09 smithi115 ceph-mon[78593]: Adjusting osd_memory_target on smithi047 to 137.2M 2022-05-14T03:38:10.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:38:09 smithi115 ceph-mon[78593]: Unable to set osd_memory_target on smithi047 to 143868239: error parsing value: Value '143868239' is below minimum 939524096 2022-05-14T03:38:10.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:38:09 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:38:10.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:38:09 smithi115 ceph-mon[78593]: Adjusting osd_memory_target on smithi115 to 197.0M 2022-05-14T03:38:10.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:38:09 smithi115 ceph-mon[78593]: Unable to set osd_memory_target on smithi115 to 206629785: error parsing value: Value '206629785' is below minimum 939524096 2022-05-14T03:38:10.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:38:10 smithi115 conmon[90393]: 2022-05-14T03:38:10.343+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:38:11.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:38:10 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:38:11.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:38:10 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:38:11.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:38:10 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:38:11.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:38:10 smithi047 ceph-mon[77147]: pgmap v7863: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:38:11.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:38:10 smithi047 conmon[78521]: 2022-05-14T03:38:10.799+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:38:11.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:38:10 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T03:38:11.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:38:10 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:38:11.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:38:10 smithi047 conmon[78521]: yield 2022-05-14T03:38:11.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:38:10 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:38:11.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:38:10 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:38:11.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:38:10 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:38:11.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:38:10 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T03:38:11.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:38:10 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:38:11.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:38:10 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T03:38:11.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:38:10 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:38:11.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:38:10 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T03:38:11.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:38:10 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:38:11.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:38:10 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T03:38:11.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:38:10 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:38:11.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:38:10 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T03:38:11.048 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:38:10 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:38:11.048 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:38:10 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:38:11.048 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:38:10 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:38:11.048 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:38:10 smithi047 conmon[78521]: 2022-05-14T03:38:10.800+0000 7f5344f82700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T03:38:11.049 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:38:10 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T03:38:11.049 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:38:10 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:38:11.049 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:38:10 smithi047 conmon[78521]: yield 2022-05-14T03:38:11.049 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:38:10 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:38:11.049 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:38:10 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:38:11.050 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:38:10 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:38:11.050 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:38:10 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T03:38:11.050 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:38:10 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:38:11.050 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:38:10 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T03:38:11.051 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:38:10 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:38:11.051 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:38:10 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T03:38:11.051 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:38:10 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:38:11.051 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:38:10 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T03:38:11.051 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:38:10 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:38:11.052 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:38:10 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T03:38:11.052 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:38:10 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:38:11.052 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:38:10 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:38:11.052 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:38:10 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:38:11.053 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:38:10 smithi047 conmon[78521]: 2022-05-14T03:38:11.053 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:38:10 smithi047 conmon[78521]: During handling of the above exception, another exception occurred: 2022-05-14T03:38:11.053 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:38:10 smithi047 conmon[78521]: 2022-05-14T03:38:11.053 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:38:10 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T03:38:11.053 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:38:10 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T03:38:11.054 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:38:10 smithi047 conmon[78521]: return f(*arg) 2022-05-14T03:38:11.054 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:38:10 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T03:38:11.054 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:38:10 smithi047 conmon[78521]: self._write_client_files(client_files, host) 2022-05-14T03:38:11.054 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:38:10 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T03:38:11.054 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:38:10 smithi047 conmon[78521]: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T03:38:11.055 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:38:10 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T03:38:11.055 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:38:10 smithi047 conmon[78521]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T03:38:11.055 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:38:10 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T03:38:11.055 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:38:10 smithi047 conmon[78521]: return self.event_loop.get_result(coro) 2022-05-14T03:38:11.056 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:38:10 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T03:38:11.056 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:38:10 smithi047 conmon[78521]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T03:38:11.056 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:38:10 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T03:38:11.056 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:38:10 smithi047 conmon[78521]: return self.__get_result() 2022-05-14T03:38:11.056 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:38:10 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T03:38:11.057 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:38:10 smithi047 conmon[78521]: raise self._exception 2022-05-14T03:38:11.057 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:38:10 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T03:38:11.057 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:38:10 smithi047 conmon[78521]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T03:38:11.057 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:38:10 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T03:38:11.058 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:38:10 smithi047 conmon[78521]: conn = await self._remote_connection(host, addr) 2022-05-14T03:38:11.058 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:38:10 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T03:38:11.058 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:38:10 smithi047 conmon[78521]: raise 2022-05-14T03:38:11.058 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:38:10 smithi047 conmon[78521]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T03:38:11.058 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:38:10 smithi047 conmon[78521]: self.gen.throw(type, value, traceback) 2022-05-14T03:38:11.059 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:38:10 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T03:38:11.059 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:38:10 smithi047 conmon[78521]: raise OrchestratorError(msg) 2022-05-14T03:38:11.059 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:38:10 smithi047 conmon[78521]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:38:11.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:38:10 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:38:11.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:38:10 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:38:11.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:38:10 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:38:11.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:38:10 smithi115 ceph-mon[78593]: pgmap v7863: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:38:11.711 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:38:11 smithi115 conmon[90393]: 2022-05-14T03:38:11.379+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:38:12.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:38:11 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:38:12.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:38:11 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:38:12.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T03:38:12.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:38:12.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T03:38:12.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:38:12.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:38:12.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:38:12.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T03:38:12.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:38:12.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T03:38:12.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:38:12.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T03:38:12.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:38:12.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T03:38:12.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:38:12.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T03:38:12.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:38:12.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:38:12.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:38:12.047 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:38:11 smithi047 ceph-mon[77147]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T03:38:12.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T03:38:12.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:38:12.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T03:38:12.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:38:12.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:38:12.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:38:12.049 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T03:38:12.049 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:38:12.049 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T03:38:12.049 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:38:12.050 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T03:38:12.050 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:38:12.050 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T03:38:12.050 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:38:12.050 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T03:38:12.051 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:38:12.051 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:38:12.051 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:38:12.051 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T03:38:12.052 INFO:journalctl@ceph.mon.b.smithi047.stdout: During handling of the above exception, another exception occurred: 2022-05-14T03:38:12.052 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T03:38:12.052 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T03:38:12.052 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T03:38:12.052 INFO:journalctl@ceph.mon.b.smithi047.stdout: return f(*arg) 2022-05-14T03:38:12.053 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T03:38:12.053 INFO:journalctl@ceph.mon.b.smithi047.stdout: self._write_client_files(client_files, host) 2022-05-14T03:38:12.053 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T03:38:12.053 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T03:38:12.054 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T03:38:12.054 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T03:38:12.054 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T03:38:12.054 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.event_loop.get_result(coro) 2022-05-14T03:38:12.054 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T03:38:12.055 INFO:journalctl@ceph.mon.b.smithi047.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T03:38:12.055 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T03:38:12.055 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.__get_result() 2022-05-14T03:38:12.055 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T03:38:12.055 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise self._exception 2022-05-14T03:38:12.056 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T03:38:12.056 INFO:journalctl@ceph.mon.b.smithi047.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T03:38:12.056 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T03:38:12.056 INFO:journalctl@ceph.mon.b.smithi047.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T03:38:12.057 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T03:38:12.057 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise 2022-05-14T03:38:12.057 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T03:38:12.057 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.gen.throw(type, value, traceback) 2022-05-14T03:38:12.057 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T03:38:12.058 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OrchestratorError(msg) 2022-05-14T03:38:12.058 INFO:journalctl@ceph.mon.b.smithi047.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:38:12.058 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:38:11 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 15025 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:38:12.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:38:11 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:38:12.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:38:11 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:38:12.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T03:38:12.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:38:12.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T03:38:12.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:38:12.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:38:12.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:38:12.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T03:38:12.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:38:12.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T03:38:12.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:38:12.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T03:38:12.113 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:38:12.113 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T03:38:12.113 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:38:12.113 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T03:38:12.113 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:38:12.114 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:38:12.114 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:38:12.114 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:38:11 smithi115 ceph-mon[78593]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T03:38:12.114 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T03:38:12.115 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:38:12.115 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T03:38:12.115 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:38:12.115 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:38:12.115 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:38:12.116 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T03:38:12.116 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:38:12.116 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T03:38:12.116 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:38:12.116 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T03:38:12.117 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:38:12.117 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T03:38:12.117 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:38:12.117 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T03:38:12.118 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:38:12.118 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:38:12.118 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:38:12.118 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T03:38:12.118 INFO:journalctl@ceph.mon.c.smithi115.stdout: During handling of the above exception, another exception occurred: 2022-05-14T03:38:12.119 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T03:38:12.119 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T03:38:12.119 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T03:38:12.119 INFO:journalctl@ceph.mon.c.smithi115.stdout: return f(*arg) 2022-05-14T03:38:12.120 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T03:38:12.120 INFO:journalctl@ceph.mon.c.smithi115.stdout: self._write_client_files(client_files, host) 2022-05-14T03:38:12.120 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T03:38:12.120 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T03:38:12.120 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T03:38:12.121 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T03:38:12.121 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T03:38:12.121 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.event_loop.get_result(coro) 2022-05-14T03:38:12.121 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T03:38:12.122 INFO:journalctl@ceph.mon.c.smithi115.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T03:38:12.122 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T03:38:12.122 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.__get_result() 2022-05-14T03:38:12.122 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T03:38:12.122 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise self._exception 2022-05-14T03:38:12.123 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T03:38:12.123 INFO:journalctl@ceph.mon.c.smithi115.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T03:38:12.123 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T03:38:12.123 INFO:journalctl@ceph.mon.c.smithi115.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T03:38:12.123 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T03:38:12.124 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise 2022-05-14T03:38:12.124 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T03:38:12.124 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.gen.throw(type, value, traceback) 2022-05-14T03:38:12.124 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T03:38:12.125 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OrchestratorError(msg) 2022-05-14T03:38:12.125 INFO:journalctl@ceph.mon.c.smithi115.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:38:12.125 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:38:11 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 15025 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:38:12.716 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:38:12 smithi115 conmon[90393]: 2022-05-14T03:38:12.360+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:38:13.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:38:12 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:38:13.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:38:12 smithi047 ceph-mon[77147]: pgmap v7864: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:38:13.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:38:12 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:38:13.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:38:12 smithi115 ceph-mon[78593]: pgmap v7864: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:38:13.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:38:13 smithi115 conmon[90393]: 2022-05-14T03:38:13.333+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:38:14.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:38:13 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:38:14.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:38:13 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:38:14.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:38:14 smithi115 conmon[90393]: 2022-05-14T03:38:14.317+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:38:15.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:38:14 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:38:15.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:38:14 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:38:15.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:38:14 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:38:15.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:38:14 smithi047 ceph-mon[77147]: pgmap v7865: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:38:15.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:38:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T03:38:15.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:38:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T03:38:15.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:38:14 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:38:15.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:38:14 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:38:15.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:38:14 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:38:15.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:38:14 smithi115 ceph-mon[78593]: pgmap v7865: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:38:15.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:38:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T03:38:15.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:38:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T03:38:15.728 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:38:15 smithi115 conmon[90393]: 2022-05-14T03:38:15.357+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:38:16.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:38:15 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:38:16.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:38:15 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:38:16.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:38:16 smithi115 conmon[90393]: 2022-05-14T03:38:16.356+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:38:17.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:38:16 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:38:17.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:38:16 smithi047 ceph-mon[77147]: pgmap v7866: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:38:17.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:38:16 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:38:17.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:38:16 smithi115 ceph-mon[78593]: pgmap v7866: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:38:17.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:38:17 smithi115 conmon[90393]: 2022-05-14T03:38:17.345+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:38:18.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:38:17 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:38:18.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:38:17 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 15030 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:38:18.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:38:17 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:38:18.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:38:17 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 15030 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:38:18.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:38:18 smithi115 conmon[90393]: 2022-05-14T03:38:18.348+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:38:19.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:38:18 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:38:19.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:38:18 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:38:19.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:38:18 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:38:19.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:38:18 smithi047 ceph-mon[77147]: pgmap v7867: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:38:19.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:38:18 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:38:19.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:38:18 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:38:19.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:38:18 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:38:19.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:38:18 smithi115 ceph-mon[78593]: pgmap v7867: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:38:19.754 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:38:19 smithi115 conmon[90393]: 2022-05-14T03:38:19.385+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:38:20.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:38:19 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:38:20.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:38:19 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:38:20.759 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:38:20 smithi115 conmon[90393]: 2022-05-14T03:38:20.433+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:38:21.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:38:20 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:38:21.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:38:20 smithi047 ceph-mon[77147]: pgmap v7868: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:38:21.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:38:20 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:38:21.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:38:20 smithi115 ceph-mon[78593]: pgmap v7868: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:38:21.760 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:38:21 smithi115 conmon[90393]: 2022-05-14T03:38:21.452+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:38:22.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:38:21 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:38:22.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:38:21 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:38:22.768 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:38:22 smithi115 conmon[90393]: 2022-05-14T03:38:22.432+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:38:23.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:38:22 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:38:23.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:38:22 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:38:23.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:38:22 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:38:23.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:38:22 smithi047 ceph-mon[77147]: pgmap v7869: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:38:23.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:38:22 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:38:23.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:38:22 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:38:23.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:38:22 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:38:23.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:38:22 smithi115 ceph-mon[78593]: pgmap v7869: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:38:23.770 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:38:23 smithi115 conmon[90393]: 2022-05-14T03:38:23.455+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:38:24.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:38:23 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:38:24.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:38:23 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:38:24.775 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:38:24 smithi115 conmon[90393]: 2022-05-14T03:38:24.498+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:38:25.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:38:24 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:38:25.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:38:24 smithi047 ceph-mon[77147]: pgmap v7870: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:38:25.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:38:24 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:38:25.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:38:24 smithi115 ceph-mon[78593]: pgmap v7870: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:38:25.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:38:25 smithi115 conmon[90393]: 2022-05-14T03:38:25.544+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:38:25.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:38:25 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:38:26.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:38:25 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:38:26.780 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:38:26 smithi115 conmon[90393]: 2022-05-14T03:38:26.522+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:38:27.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:38:26 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:38:27.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:38:26 smithi047 ceph-mon[77147]: pgmap v7871: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:38:27.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:38:26 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 15035 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:38:27.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:38:26 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:38:27.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:38:26 smithi115 ceph-mon[78593]: pgmap v7871: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:38:27.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:38:26 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 15035 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:38:27.782 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:38:27 smithi115 conmon[90393]: 2022-05-14T03:38:27.503+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:38:28.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:38:27 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:38:28.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:38:27 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:38:28.789 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:38:28 smithi115 conmon[90393]: 2022-05-14T03:38:28.469+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:38:29.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:38:28 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:38:29.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:38:28 smithi047 ceph-mon[77147]: pgmap v7872: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:38:29.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:38:28 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:38:29.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:38:28 smithi115 ceph-mon[78593]: pgmap v7872: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:38:29.789 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:38:29 smithi115 conmon[90393]: 2022-05-14T03:38:29.503+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:38:30.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:38:29 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:38:30.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:38:29 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:38:30.800 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:38:30 smithi115 conmon[90393]: 2022-05-14T03:38:30.489+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:38:31.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:38:30 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:38:31.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:38:30 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:38:31.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:38:30 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:38:31.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:38:30 smithi115 ceph-mon[78593]: pgmap v7873: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:38:31.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:38:30 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:38:31.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:38:30 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:38:31.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:38:30 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:38:31.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:38:30 smithi047 ceph-mon[77147]: pgmap v7873: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:38:31.801 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:38:31 smithi115 conmon[90393]: 2022-05-14T03:38:31.539+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:38:32.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:38:31 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:38:32.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:38:31 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 15045 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:38:32.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:38:31 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:38:32.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:38:31 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 15045 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:38:32.806 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:38:32 smithi115 conmon[90393]: 2022-05-14T03:38:32.494+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:38:33.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:38:32 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:38:33.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:38:32 smithi115 ceph-mon[78593]: pgmap v7874: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:38:33.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:38:32 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:38:33.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:38:32 smithi047 ceph-mon[77147]: pgmap v7874: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:38:33.812 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:38:33 smithi115 conmon[90393]: 2022-05-14T03:38:33.481+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:38:34.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:38:33 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:38:34.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:38:33 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:38:34.817 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:38:34 smithi115 conmon[90393]: 2022-05-14T03:38:34.506+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:38:35.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:38:34 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:38:35.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:38:34 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:38:35.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:38:34 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:38:35.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:38:34 smithi115 ceph-mon[78593]: pgmap v7875: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:38:35.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:38:34 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:38:35.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:38:34 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:38:35.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:38:34 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:38:35.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:38:34 smithi047 ceph-mon[77147]: pgmap v7875: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:38:35.819 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:38:35 smithi115 conmon[90393]: 2022-05-14T03:38:35.507+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:38:36.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:38:35 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:38:36.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:38:35 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:38:36.823 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:38:36 smithi115 conmon[90393]: 2022-05-14T03:38:36.493+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:38:37.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:38:36 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:38:37.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:38:36 smithi115 ceph-mon[78593]: pgmap v7876: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:38:37.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:38:36 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:38:37.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:38:36 smithi047 ceph-mon[77147]: pgmap v7876: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:38:37.824 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:38:37 smithi115 conmon[90393]: 2022-05-14T03:38:37.513+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:38:38.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:38:37 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:38:38.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:38:37 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 15050 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:38:38.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:38:37 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:38:38.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:38:37 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 15050 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:38:38.836 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:38:38 smithi115 conmon[90393]: 2022-05-14T03:38:38.480+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:38:39.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:38:38 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:38:39.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:38:38 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:38:39.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:38:38 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:38:39.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:38:38 smithi115 ceph-mon[78593]: pgmap v7877: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:38:39.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:38:38 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:38:39.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:38:38 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:38:39.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:38:38 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:38:39.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:38:38 smithi047 ceph-mon[77147]: pgmap v7877: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:38:39.837 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:38:39 smithi115 conmon[90393]: 2022-05-14T03:38:39.515+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:38:40.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:38:39 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:38:40.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:38:39 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:38:40.842 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:38:40 smithi115 conmon[90393]: 2022-05-14T03:38:40.487+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:38:41.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:38:40 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:38:41.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:38:40 smithi115 ceph-mon[78593]: pgmap v7878: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:38:41.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:38:40 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:38:41.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:38:40 smithi047 ceph-mon[77147]: pgmap v7878: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:38:41.847 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:38:41 smithi115 conmon[90393]: 2022-05-14T03:38:41.465+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:38:42.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:38:41 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:38:42.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:38:41 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:38:42.849 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:38:42 smithi115 conmon[90393]: 2022-05-14T03:38:42.508+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:38:43.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:38:42 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:38:43.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:38:42 smithi115 ceph-mon[78593]: pgmap v7879: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:38:43.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:38:42 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:38:43.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:38:42 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:38:43.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:38:42 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:38:43.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:38:42 smithi047 ceph-mon[77147]: pgmap v7879: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:38:43.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:38:42 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:38:43.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:38:42 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:38:43.854 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:38:43 smithi115 conmon[90393]: 2022-05-14T03:38:43.493+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:38:44.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:38:43 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:38:44.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:38:43 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:38:44.857 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:38:44 smithi115 conmon[90393]: 2022-05-14T03:38:44.497+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:38:45.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:38:44 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:38:45.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:38:44 smithi115 ceph-mon[78593]: pgmap v7880: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:38:45.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:38:44 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:38:45.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:38:44 smithi047 ceph-mon[77147]: pgmap v7880: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:38:45.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:38:45 smithi115 conmon[90393]: 2022-05-14T03:38:45.531+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:38:46.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:38:45 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:38:46.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:38:45 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:38:46.860 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:38:46 smithi115 conmon[90393]: 2022-05-14T03:38:46.533+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:38:47.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:38:46 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:38:47.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:38:46 smithi047 ceph-mon[77147]: pgmap v7881: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:38:47.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:38:46 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 15060 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:38:47.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:38:46 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:38:47.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:38:46 smithi115 ceph-mon[78593]: pgmap v7881: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:38:47.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:38:46 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 15060 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:38:47.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:38:47 smithi115 conmon[90393]: 2022-05-14T03:38:47.545+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:38:48.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:38:47 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:38:48.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:38:47 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:38:48.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:38:48 smithi115 conmon[90393]: 2022-05-14T03:38:48.584+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:38:49.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:38:48 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:38:49.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:38:48 smithi047 ceph-mon[77147]: pgmap v7882: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:38:49.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:38:48 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:38:49.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:38:48 smithi115 ceph-mon[78593]: pgmap v7882: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:38:50.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:38:49 smithi115 conmon[90393]: 2022-05-14T03:38:49.622+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:38:50.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:38:49 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:38:50.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:38:49 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:38:51.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:38:50 smithi115 conmon[90393]: 2022-05-14T03:38:50.664+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:38:51.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:38:50 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:38:51.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:38:50 smithi115 ceph-mon[78593]: pgmap v7883: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:38:51.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:38:50 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:38:51.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:38:50 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:38:51.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:38:50 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:38:51.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:38:50 smithi047 ceph-mon[77147]: pgmap v7883: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:38:51.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:38:50 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:38:51.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:38:50 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:38:51.877 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:38:51 smithi115 conmon[90393]: 2022-05-14T03:38:51.617+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:38:52.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:38:51 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:38:52.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:38:51 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 15065 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:38:52.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:38:51 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:38:52.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:38:51 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 15065 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:38:52.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:38:52 smithi115 conmon[90393]: 2022-05-14T03:38:52.582+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:38:53.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:38:52 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:38:53.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:38:52 smithi047 ceph-mon[77147]: pgmap v7884: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:38:53.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:38:52 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:38:53.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:38:52 smithi115 ceph-mon[78593]: pgmap v7884: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:38:53.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:38:53 smithi115 conmon[90393]: 2022-05-14T03:38:53.592+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:38:54.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:38:53 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:38:54.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:38:53 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:38:54.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:38:54 smithi115 conmon[90393]: 2022-05-14T03:38:54.600+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:38:55.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:38:54 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:38:55.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:38:54 smithi047 ceph-mon[77147]: pgmap v7885: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:38:55.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:38:54 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:38:55.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:38:54 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:38:55.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:38:54 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:38:55.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:38:54 smithi115 ceph-mon[78593]: pgmap v7885: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:38:55.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:38:54 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:38:55.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:38:54 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:38:55.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:38:55 smithi115 conmon[90393]: 2022-05-14T03:38:55.578+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:38:56.295 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:38:55 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:38:56.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:38:55 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:38:56.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:38:56 smithi115 conmon[90393]: 2022-05-14T03:38:56.573+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:38:57.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:38:56 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:38:57.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:38:56 smithi047 ceph-mon[77147]: pgmap v7886: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:38:57.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:38:56 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 15070 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:38:57.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:38:56 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:38:57.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:38:56 smithi115 ceph-mon[78593]: pgmap v7886: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:38:57.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:38:56 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 15070 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:38:57.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:38:57 smithi115 conmon[90393]: 2022-05-14T03:38:57.549+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:38:58.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:38:57 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:38:58.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:38:57 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:38:58.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:38:58 smithi115 conmon[90393]: 2022-05-14T03:38:58.593+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:38:59.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:38:58 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:38:59.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:38:58 smithi047 ceph-mon[77147]: pgmap v7887: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:38:59.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:38:58 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:38:59.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:38:58 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:38:59.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:38:58 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:38:59.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:38:58 smithi115 ceph-mon[78593]: pgmap v7887: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:38:59.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:38:58 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:38:59.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:38:58 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:38:59.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:38:59 smithi115 conmon[90393]: 2022-05-14T03:38:59.563+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:39:00.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:38:59 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:39:00.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:38:59 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:39:00.906 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:39:00 smithi115 conmon[90393]: 2022-05-14T03:39:00.606+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:39:01.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:39:00 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:39:01.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:39:00 smithi047 ceph-mon[77147]: pgmap v7888: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:39:01.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:39:00 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:39:01.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:39:00 smithi115 ceph-mon[78593]: pgmap v7888: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:39:01.906 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:39:01 smithi115 conmon[90393]: 2022-05-14T03:39:01.631+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:39:02.293 INFO:journalctl@ceph.osd.4.smithi047.stdout:May 14 03:39:01 smithi047 conmon[88497]: 2022-05-14T03:39:01.929+0000 7f773fa85700 -1 received signal: Hangup from Kernel ( Could be generated by pthread_kill(), raise(), abort(), alarm() ) UID: 0 2022-05-14T03:39:02.293 INFO:journalctl@ceph.osd.4.smithi047.stdout:May 14 03:39:01 smithi047 conmon[88497]: 2022-05-14T03:39:01.990+0000 7f773fa85700 -1 received signal: Hangup from Kernel ( Could be generated by pthread_kill(), raise(), abort(), alarm() ) UID: 0 2022-05-14T03:39:02.294 INFO:journalctl@ceph.osd.7.smithi047.stdout:May 14 03:39:01 smithi047 conmon[94366]: 2022-05-14T03:39:01.929+0000 7fb53b5e8700 -1 received signal: Hangup from Kernel ( Could be generated by pthread_kill(), raise(), abort(), alarm() ) UID: 0 2022-05-14T03:39:02.294 INFO:journalctl@ceph.osd.7.smithi047.stdout:May 14 03:39:01 smithi047 conmon[94366]: 2022-05-14T03:39:01.990+0000 7fb53b5e8700 -1 received signal: Hangup from Kernel ( Could be generated by pthread_kill(), raise(), abort(), alarm() ) UID: 0 2022-05-14T03:39:02.295 INFO:journalctl@ceph.osd.10.smithi047.stdout:May 14 03:39:01 smithi047 conmon[100262]: 2022-05-14T03:39:01.929+0000 7f5b525fe700 -1 received signal: Hangup from Kernel ( Could be generated by pthread_kill(), raise(), abort(), alarm() ) UID: 0 2022-05-14T03:39:02.295 INFO:journalctl@ceph.osd.10.smithi047.stdout:May 14 03:39:01 smithi047 conmon[100262]: 2022-05-14T03:39:01.990+0000 7f5b525fe700 -1 received signal: Hangup from Kernel ( Could be generated by pthread_kill(), raise(), abort(), alarm() ) UID: 0 2022-05-14T03:39:02.296 INFO:journalctl@ceph.mds.e.smithi047.stdout:May 14 03:39:01 smithi047 conmon[103776]: 2022-05-14T03:39:01.929+0000 7f01c03c5700 -1 received signal: Hangup from Kernel ( Could be generated by pthread_kill(), raise(), abort(), alarm() ) UID: 0 2022-05-14T03:39:02.296 INFO:journalctl@ceph.mds.e.smithi047.stdout:May 14 03:39:01 smithi047 conmon[103776]: 2022-05-14T03:39:01.990+0000 7f01c03c5700 -1 received signal: Hangup from Kernel ( Could be generated by pthread_kill(), raise(), abort(), alarm() ) UID: 0 2022-05-14T03:39:02.297 INFO:journalctl@ceph.mds.k.smithi047.stdout:May 14 03:39:01 smithi047 conmon[104923]: 2022-05-14T03:39:01.929+0000 7f1d4b76c700 -1 received signal: Hangup from Kernel ( Could be generated by pthread_kill(), raise(), abort(), alarm() ) UID: 0 2022-05-14T03:39:02.297 INFO:journalctl@ceph.mds.k.smithi047.stdout:May 14 03:39:01 smithi047 conmon[104923]: 2022-05-14T03:39:01.990+0000 7f1d4b76c700 -1 received signal: Hangup from Kernel ( Could be generated by pthread_kill(), raise(), abort(), alarm() ) UID: 0 2022-05-14T03:39:02.298 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:39:01 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:39:02.298 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:39:01 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 15075 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:39:02.298 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:39:01 smithi047 conmon[77123]: 2022-05-14T03:39:01.929+0000 7f8b276a6700 -1 received signal: Hangup from Kernel ( Could be generated by pthread_kill(), raise(), abort(), alarm() ) UID: 0 2022-05-14T03:39:02.298 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:39:01 smithi047 conmon[77123]: 2022-05-14T03:39:01.929+0000 7f8b276a6700 -1 mon.b@1(leader) e3 *** Got Signal Hangup *** 2022-05-14T03:39:02.299 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:39:01 smithi047 conmon[77123]: 2022-05-14T03:39:01.990+0000 7f8b276a6700 -1 received signal: Hangup from Kernel ( Could be generated by pthread_kill(), raise(), abort(), alarm() ) UID: 0 2022-05-14T03:39:02.299 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:39:01 smithi047 conmon[77123]: 2022-05-14T03:39:01.990+0000 7f8b276a6700 -1 mon.b@1(leader) e3 *** Got Signal Hangup *** 2022-05-14T03:39:02.299 INFO:journalctl@ceph.mds.h.smithi047.stdout:May 14 03:39:01 smithi047 conmon[104342]: 2022-05-14T03:39:01.930+0000 7f4979e97700 -1 received signal: Hangup from Kernel ( Could be generated by pthread_kill(), raise(), abort(), alarm() ) UID: 0 2022-05-14T03:39:02.300 INFO:journalctl@ceph.mds.h.smithi047.stdout:May 14 03:39:01 smithi047 conmon[104342]: 2022-05-14T03:39:01.990+0000 7f4979e97700 -1 received signal: Hangup from Kernel ( Could be generated by pthread_kill(), raise(), abort(), alarm() ) UID: 0 2022-05-14T03:39:02.300 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:39:01 smithi047 conmon[78521]: 2022-05-14T03:39:01.929+0000 7f53a881b700 -1 received signal: Hangup from Kernel ( Could be generated by pthread_kill(), raise(), abort(), alarm() ) UID: 0 2022-05-14T03:39:02.300 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:39:01 smithi047 conmon[78521]: 2022-05-14T03:39:01.990+0000 7f53a881b700 -1 received signal: Hangup from Kernel ( Could be generated by pthread_kill(), raise(), abort(), alarm() ) UID: 0 2022-05-14T03:39:02.301 INFO:journalctl@ceph.osd.1.smithi047.stdout:May 14 03:39:01 smithi047 conmon[82670]: 2022-05-14T03:39:01.929+0000 7f9bd39a2700 -1 received signal: Hangup from Kernel ( Could be generated by pthread_kill(), raise(), abort(), alarm() ) UID: 0 2022-05-14T03:39:02.301 INFO:journalctl@ceph.osd.1.smithi047.stdout:May 14 03:39:01 smithi047 conmon[82670]: 2022-05-14T03:39:01.990+0000 7f9bd39a2700 -1 received signal: Hangup from Kernel ( Could be generated by pthread_kill(), raise(), abort(), alarm() ) UID: 0 2022-05-14T03:39:02.302 INFO:journalctl@ceph.mds.b.smithi047.stdout:May 14 03:39:01 smithi047 conmon[103208]: 2022-05-14T03:39:01.929+0000 7f1510519700 -1 received signal: Hangup from Kernel ( Could be generated by pthread_kill(), raise(), abort(), alarm() ) UID: 0 2022-05-14T03:39:02.302 INFO:journalctl@ceph.mds.b.smithi047.stdout:May 14 03:39:01 smithi047 conmon[103208]: 2022-05-14T03:39:01.990+0000 7f1510519700 -1 received signal: Hangup from Kernel ( Could be generated by pthread_kill(), raise(), abort(), alarm() ) UID: 0 2022-05-14T03:39:02.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:39:01 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:39:02.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:39:01 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 15075 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:39:02.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:39:02 smithi115 conmon[90393]: 2022-05-14T03:39:02.585+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:39:03.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:39:02 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:39:03.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:39:02 smithi047 ceph-mon[77147]: pgmap v7889: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:39:03.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:39:02 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:39:03.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:39:02 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:39:03.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:39:02 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:39:03.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:39:02 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:39:03.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:39:02 smithi115 ceph-mon[78593]: pgmap v7889: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:39:03.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:39:02 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:39:03.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:39:02 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:39:03.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:39:02 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:39:03.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:39:03 smithi115 conmon[90393]: 2022-05-14T03:39:03.591+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:39:04.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:39:03 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:39:04.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:39:03 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:39:04.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:39:04 smithi115 conmon[90393]: 2022-05-14T03:39:04.591+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:39:05.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:39:04 smithi047 ceph-mon[77147]: pgmap v7890: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:39:05.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:39:04 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:39:05.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:39:04 smithi115 ceph-mon[78593]: pgmap v7890: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:39:05.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:39:04 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:39:05.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:39:05 smithi115 conmon[90393]: 2022-05-14T03:39:05.548+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:39:06.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:39:05 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:39:06.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:39:05 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:39:06.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:39:06 smithi115 conmon[90393]: 2022-05-14T03:39:06.574+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:39:07.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:39:06 smithi047 ceph-mon[77147]: pgmap v7891: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:39:07.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:39:06 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:39:07.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:39:06 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 15080 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:39:07.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:39:06 smithi115 ceph-mon[78593]: pgmap v7891: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:39:07.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:39:06 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:39:07.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:39:06 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 15080 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:39:07.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:39:07 smithi115 conmon[90393]: 2022-05-14T03:39:07.586+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:39:08.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:39:07 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:39:08.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:39:07 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:39:08.930 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:39:08 smithi115 conmon[90393]: 2022-05-14T03:39:08.621+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:39:09.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:39:08 smithi047 ceph-mon[77147]: pgmap v7892: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:39:09.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:39:08 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:39:09.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:39:08 smithi115 ceph-mon[78593]: pgmap v7892: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:39:09.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:39:08 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:39:09.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:39:09 smithi115 conmon[90393]: 2022-05-14T03:39:09.594+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:39:10.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:39:09 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:39:10.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:39:09 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:39:10.848 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:39:10 smithi115 conmon[90393]: 2022-05-14T03:39:10.583+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:39:11.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:39:10 smithi115 ceph-mon[78593]: pgmap v7893: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:39:11.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:39:10 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:39:11.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:39:10 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:39:11.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:39:10 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:39:11.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:39:10 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T03:39:11.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:39:10 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T03:39:11.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:39:10 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T03:39:11.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:39:10 smithi047 ceph-mon[77147]: pgmap v7893: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:39:11.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:39:10 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:39:11.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:39:10 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:39:11.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:39:10 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:39:11.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:39:10 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T03:39:11.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:39:10 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T03:39:11.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:39:10 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T03:39:11.793 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:39:11 smithi047 conmon[78521]: 2022-05-14T03:39:11.407+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:39:11.793 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:39:11 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T03:39:11.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:39:11 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:39:11.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:39:11 smithi047 conmon[78521]: yield 2022-05-14T03:39:11.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:39:11 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:39:11.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:39:11 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:39:11.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:39:11 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:39:11.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:39:11 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T03:39:11.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:39:11 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:39:11.797 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:39:11 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T03:39:11.797 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:39:11 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:39:11.797 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:39:11 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T03:39:11.798 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:39:11 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:39:11.798 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:39:11 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T03:39:11.799 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:39:11 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:39:11.799 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:39:11 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T03:39:11.799 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:39:11 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:39:11.800 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:39:11 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:39:11.800 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:39:11 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:39:11.861 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:39:11 smithi115 conmon[90393]: 2022-05-14T03:39:11.581+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:39:12.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:39:11 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:39:12.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:39:11 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 15085 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:39:12.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:39:11 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:39:12.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:39:11 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 15085 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:39:12.937 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:39:12 smithi115 conmon[90393]: 2022-05-14T03:39:12.615+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:39:13.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:39:12 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:39:13.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T03:39:13.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:39:13.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T03:39:13.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:39:13.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:39:13.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:39:13.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T03:39:13.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:39:13.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T03:39:13.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:39:13.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T03:39:13.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:39:13.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T03:39:13.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:39:13.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T03:39:13.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:39:13.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:39:13.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:39:13.297 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:39:12 smithi047 ceph-mon[77147]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T03:39:13.298 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:39:12 smithi047 ceph-mon[77147]: pgmap v7894: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:39:13.298 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:39:12 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:39:13.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:39:12 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:39:13.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T03:39:13.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:39:13.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T03:39:13.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:39:13.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:39:13.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:39:13.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T03:39:13.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:39:13.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T03:39:13.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:39:13.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T03:39:13.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:39:13.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T03:39:13.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:39:13.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T03:39:13.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:39:13.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:39:13.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:39:13.364 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:39:12 smithi115 ceph-mon[78593]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T03:39:13.364 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:39:12 smithi115 ceph-mon[78593]: pgmap v7894: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:39:13.364 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:39:12 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:39:13.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:39:13 smithi115 conmon[90393]: 2022-05-14T03:39:13.568+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:39:14.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:39:13 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:39:14.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:39:13 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:39:14.793 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:39:14 smithi047 conmon[78521]: 2022-05-14T03:39:14.479+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:39:14.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:39:14 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T03:39:14.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:39:14 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:39:14.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:39:14 smithi047 conmon[78521]: yield 2022-05-14T03:39:14.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:39:14 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:39:14.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:39:14 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:39:14.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:39:14 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:39:14.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:39:14 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T03:39:14.797 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:39:14 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:39:14.797 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:39:14 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T03:39:14.798 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:39:14 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:39:14.798 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:39:14 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T03:39:14.798 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:39:14 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:39:14.799 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:39:14 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T03:39:14.799 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:39:14 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:39:14.799 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:39:14 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T03:39:14.799 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:39:14 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:39:14.799 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:39:14 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:39:14.800 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:39:14 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:39:14.800 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:39:14 smithi047 conmon[78521]: 2022-05-14T03:39:14.479+0000 7f5344781700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T03:39:14.800 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:39:14 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T03:39:14.800 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:39:14 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:39:14.801 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:39:14 smithi047 conmon[78521]: yield 2022-05-14T03:39:14.801 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:39:14 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:39:14.801 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:39:14 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:39:14.801 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:39:14 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:39:14.802 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:39:14 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T03:39:14.802 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:39:14 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:39:14.802 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:39:14 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T03:39:14.802 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:39:14 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:39:14.803 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:39:14 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T03:39:14.803 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:39:14 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:39:14.803 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:39:14 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T03:39:14.803 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:39:14 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:39:14.804 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:39:14 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T03:39:14.804 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:39:14 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:39:14.804 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:39:14 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:39:14.804 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:39:14 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:39:14.804 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:39:14 smithi047 conmon[78521]: 2022-05-14T03:39:14.805 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:39:14 smithi047 conmon[78521]: During handling of the above exception, another exception occurred: 2022-05-14T03:39:14.805 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:39:14 smithi047 conmon[78521]: 2022-05-14T03:39:14.805 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:39:14 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T03:39:14.806 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:39:14 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T03:39:14.806 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:39:14 smithi047 conmon[78521]: return f(*arg) 2022-05-14T03:39:14.806 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:39:14 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T03:39:14.806 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:39:14 smithi047 conmon[78521]: self._write_client_files(client_files, host) 2022-05-14T03:39:14.806 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:39:14 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T03:39:14.807 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:39:14 smithi047 conmon[78521]: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T03:39:14.807 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:39:14 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T03:39:14.807 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:39:14 smithi047 conmon[78521]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T03:39:14.807 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:39:14 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T03:39:14.808 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:39:14 smithi047 conmon[78521]: return self.event_loop.get_result(coro) 2022-05-14T03:39:14.808 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:39:14 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T03:39:14.808 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:39:14 smithi047 conmon[78521]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T03:39:14.808 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:39:14 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T03:39:14.808 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:39:14 smithi047 conmon[78521]: return self.__get_result() 2022-05-14T03:39:14.809 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:39:14 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T03:39:14.809 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:39:14 smithi047 conmon[78521]: raise self._exception 2022-05-14T03:39:14.810 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:39:14 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T03:39:14.810 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:39:14 smithi047 conmon[78521]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T03:39:14.810 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:39:14 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T03:39:14.810 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:39:14 smithi047 conmon[78521]: conn = await self._remote_connection(host, addr) 2022-05-14T03:39:14.811 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:39:14 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T03:39:14.811 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:39:14 smithi047 conmon[78521]: raise 2022-05-14T03:39:14.811 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:39:14 smithi047 conmon[78521]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T03:39:14.811 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:39:14 smithi047 conmon[78521]: self.gen.throw(type, value, traceback) 2022-05-14T03:39:14.812 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:39:14 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T03:39:14.812 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:39:14 smithi047 conmon[78521]: raise OrchestratorError(msg) 2022-05-14T03:39:14.812 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:39:14 smithi047 conmon[78521]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:39:14.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:39:14 smithi115 conmon[90393]: 2022-05-14T03:39:14.569+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:39:15.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:39:14 smithi047 ceph-mon[77147]: pgmap v7895: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:39:15.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:39:14 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:39:15.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:39:14 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:39:15.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:39:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T03:39:15.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:39:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T03:39:15.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:39:14 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:39:15.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:39:14 smithi115 ceph-mon[78593]: pgmap v7895: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:39:15.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:39:14 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:39:15.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:39:14 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:39:15.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:39:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T03:39:15.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:39:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T03:39:15.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:39:14 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:39:15.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:39:15 smithi115 conmon[90393]: 2022-05-14T03:39:15.569+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:39:16.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:39:15 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:39:16.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T03:39:16.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:39:16.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T03:39:16.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:39:16.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:39:16.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:39:16.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T03:39:16.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:39:16.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T03:39:16.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:39:16.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T03:39:16.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:39:16.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T03:39:16.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:39:16.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T03:39:16.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:39:16.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:39:16.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:39:16.297 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:39:15 smithi047 ceph-mon[77147]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T03:39:16.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T03:39:16.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:39:16.298 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T03:39:16.298 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:39:16.298 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:39:16.298 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:39:16.299 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T03:39:16.299 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:39:16.299 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T03:39:16.300 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:39:16.300 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T03:39:16.300 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:39:16.301 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T03:39:16.301 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:39:16.301 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T03:39:16.301 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:39:16.302 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:39:16.302 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:39:16.303 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T03:39:16.303 INFO:journalctl@ceph.mon.b.smithi047.stdout: During handling of the above exception, another exception occurred: 2022-05-14T03:39:16.303 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T03:39:16.303 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T03:39:16.304 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T03:39:16.304 INFO:journalctl@ceph.mon.b.smithi047.stdout: return f(*arg) 2022-05-14T03:39:16.304 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T03:39:16.305 INFO:journalctl@ceph.mon.b.smithi047.stdout: self._write_client_files(client_files, host) 2022-05-14T03:39:16.305 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T03:39:16.305 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T03:39:16.305 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T03:39:16.306 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T03:39:16.306 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T03:39:16.306 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.event_loop.get_result(coro) 2022-05-14T03:39:16.306 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T03:39:16.306 INFO:journalctl@ceph.mon.b.smithi047.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T03:39:16.307 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T03:39:16.307 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.__get_result() 2022-05-14T03:39:16.307 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T03:39:16.307 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise self._exception 2022-05-14T03:39:16.308 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T03:39:16.308 INFO:journalctl@ceph.mon.b.smithi047.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T03:39:16.308 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T03:39:16.308 INFO:journalctl@ceph.mon.b.smithi047.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T03:39:16.308 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T03:39:16.309 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise 2022-05-14T03:39:16.309 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T03:39:16.309 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.gen.throw(type, value, traceback) 2022-05-14T03:39:16.310 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T03:39:16.310 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OrchestratorError(msg) 2022-05-14T03:39:16.310 INFO:journalctl@ceph.mon.b.smithi047.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:39:16.310 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:39:15 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:39:16.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:39:15 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:39:16.359 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T03:39:16.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:39:16.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T03:39:16.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:39:16.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:39:16.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:39:16.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T03:39:16.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:39:16.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T03:39:16.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:39:16.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T03:39:16.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:39:16.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T03:39:16.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:39:16.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T03:39:16.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:39:16.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:39:16.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:39:16.365 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:39:15 smithi115 ceph-mon[78593]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T03:39:16.365 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T03:39:16.365 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:39:16.365 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T03:39:16.365 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:39:16.366 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:39:16.366 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:39:16.366 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T03:39:16.366 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:39:16.366 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T03:39:16.367 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:39:16.367 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T03:39:16.367 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:39:16.367 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T03:39:16.368 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:39:16.368 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T03:39:16.368 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:39:16.369 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:39:16.369 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:39:16.369 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T03:39:16.369 INFO:journalctl@ceph.mon.c.smithi115.stdout: During handling of the above exception, another exception occurred: 2022-05-14T03:39:16.369 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T03:39:16.370 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T03:39:16.370 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T03:39:16.371 INFO:journalctl@ceph.mon.c.smithi115.stdout: return f(*arg) 2022-05-14T03:39:16.371 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T03:39:16.371 INFO:journalctl@ceph.mon.c.smithi115.stdout: self._write_client_files(client_files, host) 2022-05-14T03:39:16.371 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T03:39:16.371 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T03:39:16.372 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T03:39:16.372 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T03:39:16.372 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T03:39:16.372 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.event_loop.get_result(coro) 2022-05-14T03:39:16.372 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T03:39:16.373 INFO:journalctl@ceph.mon.c.smithi115.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T03:39:16.373 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T03:39:16.373 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.__get_result() 2022-05-14T03:39:16.373 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T03:39:16.374 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise self._exception 2022-05-14T03:39:16.374 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T03:39:16.374 INFO:journalctl@ceph.mon.c.smithi115.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T03:39:16.374 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T03:39:16.374 INFO:journalctl@ceph.mon.c.smithi115.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T03:39:16.375 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T03:39:16.375 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise 2022-05-14T03:39:16.375 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T03:39:16.375 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.gen.throw(type, value, traceback) 2022-05-14T03:39:16.375 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T03:39:16.376 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OrchestratorError(msg) 2022-05-14T03:39:16.376 INFO:journalctl@ceph.mon.c.smithi115.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:39:16.376 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:39:15 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:39:16.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:39:16 smithi115 conmon[90393]: 2022-05-14T03:39:16.601+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:39:17.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:39:16 smithi047 ceph-mon[77147]: pgmap v7896: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:39:17.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:39:16 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:39:17.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:39:16 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 15090 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:39:17.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:39:16 smithi115 ceph-mon[78593]: pgmap v7896: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:39:17.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:39:16 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:39:17.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:39:16 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 15090 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:39:17.954 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:39:17 smithi115 conmon[90393]: 2022-05-14T03:39:17.629+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:39:18.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:39:17 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:39:18.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:39:17 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:39:18.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:39:18 smithi115 conmon[90393]: 2022-05-14T03:39:18.587+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:39:19.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:39:18 smithi047 ceph-mon[77147]: pgmap v7897: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:39:19.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:39:18 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:39:19.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:39:18 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:39:19.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:39:18 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:39:19.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:39:18 smithi115 ceph-mon[78593]: pgmap v7897: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:39:19.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:39:18 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:39:19.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:39:18 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:39:19.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:39:18 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:39:19.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:39:19 smithi115 conmon[90393]: 2022-05-14T03:39:19.584+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:39:20.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:39:19 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:39:20.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:39:19 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:39:20.960 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:39:20 smithi115 conmon[90393]: 2022-05-14T03:39:20.607+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:39:21.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:39:20 smithi047 ceph-mon[77147]: pgmap v7898: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:39:21.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:39:20 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:39:21.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:39:20 smithi115 ceph-mon[78593]: pgmap v7898: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:39:21.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:39:20 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:39:21.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:39:21 smithi115 conmon[90393]: 2022-05-14T03:39:21.601+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:39:22.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:39:21 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:39:22.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:39:21 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 15095 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:39:22.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:39:21 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:39:22.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:39:21 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 15095 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:39:22.966 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:39:22 smithi115 conmon[90393]: 2022-05-14T03:39:22.611+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:39:23.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:39:22 smithi047 ceph-mon[77147]: pgmap v7899: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:39:23.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:39:22 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:39:23.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:39:22 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:39:23.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:39:22 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:39:23.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:39:22 smithi115 ceph-mon[78593]: pgmap v7899: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:39:23.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:39:22 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:39:23.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:39:22 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:39:23.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:39:22 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:39:23.972 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:39:23 smithi115 conmon[90393]: 2022-05-14T03:39:23.633+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:39:24.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:39:23 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:39:24.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:39:23 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:39:24.971 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:39:24 smithi115 conmon[90393]: 2022-05-14T03:39:24.645+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:39:25.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:39:24 smithi047 ceph-mon[77147]: pgmap v7900: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:39:25.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:39:24 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:39:25.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:39:24 smithi115 ceph-mon[78593]: pgmap v7900: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:39:25.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:39:24 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:39:25.978 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:39:25 smithi115 conmon[90393]: 2022-05-14T03:39:25.650+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:39:26.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:39:25 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:39:26.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:39:25 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:39:26.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:39:26 smithi115 conmon[90393]: 2022-05-14T03:39:26.601+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:39:27.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:39:26 smithi047 ceph-mon[77147]: pgmap v7901: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:39:27.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:39:26 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:39:27.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:39:26 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 15100 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:39:27.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:39:26 smithi115 ceph-mon[78593]: pgmap v7901: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:39:27.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:39:26 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:39:27.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:39:26 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 15100 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:39:27.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:39:27 smithi115 conmon[90393]: 2022-05-14T03:39:27.559+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:39:28.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:39:27 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:39:28.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:39:27 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:39:28.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:39:28 smithi115 conmon[90393]: 2022-05-14T03:39:28.536+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:39:29.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:39:28 smithi047 ceph-mon[77147]: pgmap v7902: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:39:29.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:39:28 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:39:29.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:39:28 smithi115 ceph-mon[78593]: pgmap v7902: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:39:29.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:39:28 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:39:29.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:39:29 smithi115 conmon[90393]: 2022-05-14T03:39:29.557+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:39:30.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:39:29 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:39:30.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:39:29 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:39:30.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:39:30 smithi115 conmon[90393]: 2022-05-14T03:39:30.599+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:39:31.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:39:30 smithi047 ceph-mon[77147]: pgmap v7903: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:39:31.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:39:30 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:39:31.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:39:30 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:39:31.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:39:30 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:39:31.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:39:30 smithi115 ceph-mon[78593]: pgmap v7903: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:39:31.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:39:30 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:39:31.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:39:30 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:39:31.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:39:30 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:39:31.990 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:39:31 smithi115 conmon[90393]: 2022-05-14T03:39:31.642+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:39:32.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:39:31 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:39:32.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:39:31 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 15105 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:39:32.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:39:31 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:39:32.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:39:31 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 15105 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:39:32.991 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:39:32 smithi115 conmon[90393]: 2022-05-14T03:39:32.654+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:39:33.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:39:32 smithi047 ceph-mon[77147]: pgmap v7904: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:39:33.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:39:32 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:39:33.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:39:32 smithi115 ceph-mon[78593]: pgmap v7904: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:39:33.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:39:32 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:39:33.998 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:39:33 smithi115 conmon[90393]: 2022-05-14T03:39:33.695+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:39:34.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:39:33 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:39:34.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:39:33 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:39:34.996 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:39:34 smithi115 conmon[90393]: 2022-05-14T03:39:34.728+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:39:35.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:39:34 smithi047 ceph-mon[77147]: pgmap v7905: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:39:35.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:39:34 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:39:35.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:39:34 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:39:35.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:39:34 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:39:35.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:39:34 smithi115 ceph-mon[78593]: pgmap v7905: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:39:35.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:39:34 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:39:35.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:39:34 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:39:35.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:39:34 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:39:36.003 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:39:35 smithi115 conmon[90393]: 2022-05-14T03:39:35.686+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:39:36.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:39:35 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:39:36.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:39:36 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:39:37.004 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:39:36 smithi115 conmon[90393]: 2022-05-14T03:39:36.654+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:39:37.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:39:37 smithi047 ceph-mon[77147]: pgmap v7906: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:39:37.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:39:37 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:39:37.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:39:37 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 15110 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:39:37.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:39:37 smithi115 ceph-mon[78593]: pgmap v7906: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:39:37.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:39:37 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:39:37.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:39:37 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 15110 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:39:38.009 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:39:37 smithi115 conmon[90393]: 2022-05-14T03:39:37.677+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:39:38.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:39:38 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:39:38.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:39:38 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:39:39.008 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:39:38 smithi115 conmon[90393]: 2022-05-14T03:39:38.719+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:39:39.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:39:39 smithi047 ceph-mon[77147]: pgmap v7907: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:39:39.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:39:39 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:39:39.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:39:39 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:39:39.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:39:39 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:39:39.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:39:39 smithi115 ceph-mon[78593]: pgmap v7907: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:39:39.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:39:39 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:39:39.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:39:39 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:39:39.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:39:39 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:39:40.015 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:39:39 smithi115 conmon[90393]: 2022-05-14T03:39:39.719+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:39:40.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:39:40 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:39:40.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:39:40 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:39:41.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:39:40 smithi115 conmon[90393]: 2022-05-14T03:39:40.766+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:39:41.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:39:41 smithi115 ceph-mon[78593]: pgmap v7908: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:39:41.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:39:41 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:39:41.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:39:41 smithi047 ceph-mon[77147]: pgmap v7908: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:39:41.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:39:41 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:39:42.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:39:41 smithi115 conmon[90393]: 2022-05-14T03:39:41.773+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:39:42.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:39:42 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 15115 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:39:42.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:39:42 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:39:42.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:39:42 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 15115 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:39:42.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:39:42 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:39:43.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:39:42 smithi115 conmon[90393]: 2022-05-14T03:39:42.818+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:39:43.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:39:43 smithi115 ceph-mon[78593]: pgmap v7909: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:39:43.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:39:43 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:39:43.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:39:43 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:39:43.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:39:43 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:39:43.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:39:43 smithi047 ceph-mon[77147]: pgmap v7909: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:39:43.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:39:43 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:39:43.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:39:43 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:39:43.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:39:43 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:39:44.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:39:43 smithi115 conmon[90393]: 2022-05-14T03:39:43.787+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:39:44.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:39:44 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:39:44.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:39:44 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:39:45.026 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:39:44 smithi115 conmon[90393]: 2022-05-14T03:39:44.737+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:39:45.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:39:45 smithi047 ceph-mon[77147]: pgmap v7910: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:39:45.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:39:45 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:39:45.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:39:45 smithi115 ceph-mon[78593]: pgmap v7910: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:39:45.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:39:45 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:39:46.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:39:45 smithi115 conmon[90393]: 2022-05-14T03:39:45.782+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:39:46.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:39:46 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:39:46.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:39:46 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:39:47.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:39:46 smithi115 conmon[90393]: 2022-05-14T03:39:46.825+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:39:47.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:39:47 smithi115 ceph-mon[78593]: pgmap v7911: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:39:47.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:39:47 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 15120 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:39:47.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:39:47 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:39:47.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:39:47 smithi047 ceph-mon[77147]: pgmap v7911: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:39:47.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:39:47 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 15120 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:39:47.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:39:47 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:39:48.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:39:48 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:39:48.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:39:47 smithi115 conmon[90393]: 2022-05-14T03:39:47.865+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:39:48.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:39:48 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:39:49.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:39:48 smithi115 conmon[90393]: 2022-05-14T03:39:48.850+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:39:49.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:39:49 smithi115 ceph-mon[78593]: pgmap v7912: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:39:49.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:39:49 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:39:49.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:39:49 smithi047 ceph-mon[77147]: pgmap v7912: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:39:49.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:39:49 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:39:50.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:39:49 smithi115 conmon[90393]: 2022-05-14T03:39:49.844+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:39:50.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:39:50 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:39:50.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:39:50 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:39:51.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:39:50 smithi115 conmon[90393]: 2022-05-14T03:39:50.818+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:39:51.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:39:51 smithi115 ceph-mon[78593]: pgmap v7913: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:39:51.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:39:51 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:39:51.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:39:51 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:39:51.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:39:51 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:39:51.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:39:51 smithi047 ceph-mon[77147]: pgmap v7913: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:39:51.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:39:51 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:39:51.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:39:51 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:39:51.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:39:51 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:39:52.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:39:51 smithi115 conmon[90393]: 2022-05-14T03:39:51.865+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:39:52.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:39:52 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 15125 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:39:52.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:39:52 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:39:52.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:39:52 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 15125 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:39:52.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:39:52 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:39:53.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:39:52 smithi115 conmon[90393]: 2022-05-14T03:39:52.829+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:39:53.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:39:53 smithi115 ceph-mon[78593]: pgmap v7914: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:39:53.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:39:53 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:39:53.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:39:53 smithi047 ceph-mon[77147]: pgmap v7914: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:39:53.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:39:53 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:39:54.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:39:54 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:39:54.110 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:39:53 smithi115 conmon[90393]: 2022-05-14T03:39:53.825+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:39:54.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:39:54 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:39:55.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:39:54 smithi115 conmon[90393]: 2022-05-14T03:39:54.822+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:39:55.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:39:55 smithi115 ceph-mon[78593]: pgmap v7915: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:39:55.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:39:55 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:39:55.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:39:55 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:39:55.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:39:55 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:39:55.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:39:55 smithi047 ceph-mon[77147]: pgmap v7915: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:39:55.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:39:55 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:39:55.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:39:55 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:39:55.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:39:55 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:39:56.056 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:39:55 smithi115 conmon[90393]: 2022-05-14T03:39:55.786+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:39:56.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:39:56 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:39:56.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:39:56 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:39:57.057 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:39:56 smithi115 conmon[90393]: 2022-05-14T03:39:56.738+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:39:57.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:39:57 smithi115 ceph-mon[78593]: pgmap v7916: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:39:57.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:39:57 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 15130 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:39:57.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:39:57 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:39:57.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:39:57 smithi047 ceph-mon[77147]: pgmap v7916: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:39:57.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:39:57 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 15130 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:39:57.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:39:57 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:39:58.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:39:57 smithi115 conmon[90393]: 2022-05-14T03:39:57.737+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:39:59.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:39:58 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:39:59.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:39:58 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:39:59.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:39:58 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:39:59.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:39:58 smithi047 ceph-mon[77147]: pgmap v7917: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:39:59.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:39:58 smithi115 conmon[90393]: 2022-05-14T03:39:58.704+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:39:59.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:39:58 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:39:59.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:39:58 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:39:59.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:39:58 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:39:59.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:39:58 smithi115 ceph-mon[78593]: pgmap v7917: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:40:00.002 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:39:59 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:40:00.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:39:59 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:40:00.110 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:39:59 smithi115 conmon[90393]: 2022-05-14T03:39:59.703+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:40:00.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:40:00 smithi047 conmon[77123]: 2022-05-14T03:39:59.998+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : Health detail: HEALTH_ERR 1 filesystem is degraded; 1 mds daemon damaged; 1 MDSs are read only; 1 MDSs report slow metadata IOs; 1/3 mons down, quorum b,c; Reduced data availability: 1 pg inactive; 9 slow ops, oldest one blocked for 15130 sec, osd.5 has slow ops 2022-05-14T03:40:00.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:40:00 smithi047 conmon[77123]: 2022-05-14T03:39:59.998+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : [WRN] FS_DEGRADED: 1 filesystem is degraded 2022-05-14T03:40:00.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:40:00 smithi047 conmon[77123]: 2022-05-14T03:39:59.998+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : fs cephfs is degraded 2022-05-14T03:40:00.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:40:00 smithi047 conmon[77123]: 2022-05-14T03:39:59.998+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : [ERR] MDS_DAMAGE: 1 mds daemon damaged 2022-05-14T03:40:00.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:40:00 smithi047 conmon[77123]: 2022-05-14T03:39:59.998+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : fs cephfs mds.2 is damaged 2022-05-14T03:40:00.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:40:00 smithi047 conmon[77123]: 2022-05-14T03:39:59.998+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : [WRN] MDS_READ_ONLY: 1 MDSs are read only 2022-05-14T03:40:00.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:40:00 smithi047 conmon[77123]: 2022-05-14T03:39:59.998+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : mds.l(mds.0): MDS in read-only mode 2022-05-14T03:40:00.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:40:00 smithi047 conmon[77123]: 2022-05-14T03:39:59.998+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : [WRN] MDS_SLOW_METADATA_IO: 1 MDSs report slow metadata IOs 2022-05-14T03:40:00.295 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:40:00 smithi047 conmon[77123]: 2022-05-14T03:39:59.998+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : mds.l(mds.0): 4 slow metadata IOs are blocked > 180 secs, oldest blocked for 15433 secs 2022-05-14T03:40:00.295 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:40:00 smithi047 conmon[77123]: 2022-05-14T03:39:59.998+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : [WRN] MON_DOWN: 1/3 mons down, quorum b,c 2022-05-14T03:40:00.295 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:40:00 smithi047 conmon[77123]: 2022-05-14T03:39:59.998+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : mon.a (rank 0) addr [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] is down (out of quorum) 2022-05-14T03:40:00.295 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:40:00 smithi047 conmon[77123]: 2022-05-14T03:39:59.998+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : [WRN] PG_AVAILABILITY: Reduced data availability: 1 pg inactive 2022-05-14T03:40:00.296 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:40:00 smithi047 conmon[77123]: 2022-05-14T03:39:59.998+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : pg 2.1 is stuck inactive for 4h, current state unknown, last acting [] 2022-05-14T03:40:00.296 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:40:00 smithi047 conmon[77123]: 2022-05-14T03:39:59.998+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : [WRN] SLOW_OPS: 9 slow ops, oldest one blocked for 15130 sec, osd.5 has slow ops 2022-05-14T03:40:01.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:40:00 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:40:01.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:40:00 smithi047 ceph-mon[77147]: pgmap v7918: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:40:01.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:40:00 smithi047 ceph-mon[77147]: Health detail: HEALTH_ERR 1 filesystem is degraded; 1 mds daemon damaged; 1 MDSs are read only; 1 MDSs report slow metadata IOs; 1/3 mons down, quorum b,c; Reduced data availability: 1 pg inactive; 9 slow ops, oldest one blocked for 15130 sec, osd.5 has slow ops 2022-05-14T03:40:01.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:40:00 smithi047 ceph-mon[77147]: [WRN] FS_DEGRADED: 1 filesystem is degraded 2022-05-14T03:40:01.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:40:00 smithi047 ceph-mon[77147]: fs cephfs is degraded 2022-05-14T03:40:01.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:40:00 smithi047 ceph-mon[77147]: [ERR] MDS_DAMAGE: 1 mds daemon damaged 2022-05-14T03:40:01.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:40:00 smithi047 ceph-mon[77147]: fs cephfs mds.2 is damaged 2022-05-14T03:40:01.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:40:00 smithi047 ceph-mon[77147]: [WRN] MDS_READ_ONLY: 1 MDSs are read only 2022-05-14T03:40:01.045 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:40:00 smithi047 ceph-mon[77147]: mds.l(mds.0): MDS in read-only mode 2022-05-14T03:40:01.045 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:40:00 smithi047 ceph-mon[77147]: [WRN] MDS_SLOW_METADATA_IO: 1 MDSs report slow metadata IOs 2022-05-14T03:40:01.045 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:40:00 smithi047 ceph-mon[77147]: mds.l(mds.0): 4 slow metadata IOs are blocked > 180 secs, oldest blocked for 15433 secs 2022-05-14T03:40:01.045 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:40:00 smithi047 ceph-mon[77147]: [WRN] MON_DOWN: 1/3 mons down, quorum b,c 2022-05-14T03:40:01.046 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:40:00 smithi047 ceph-mon[77147]: mon.a (rank 0) addr [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] is down (out of quorum) 2022-05-14T03:40:01.046 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:40:00 smithi047 ceph-mon[77147]: [WRN] PG_AVAILABILITY: Reduced data availability: 1 pg inactive 2022-05-14T03:40:01.046 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:40:00 smithi047 ceph-mon[77147]: pg 2.1 is stuck inactive for 4h, current state unknown, last acting [] 2022-05-14T03:40:01.046 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:40:00 smithi047 ceph-mon[77147]: [WRN] SLOW_OPS: 9 slow ops, oldest one blocked for 15130 sec, osd.5 has slow ops 2022-05-14T03:40:01.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:40:00 smithi115 conmon[90393]: 2022-05-14T03:40:00.675+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:40:01.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:40:00 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:40:01.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:40:00 smithi115 ceph-mon[78593]: pgmap v7918: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:40:01.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:40:00 smithi115 ceph-mon[78593]: Health detail: HEALTH_ERR 1 filesystem is degraded; 1 mds daemon damaged; 1 MDSs are read only; 1 MDSs report slow metadata IOs; 1/3 mons down, quorum b,c; Reduced data availability: 1 pg inactive; 9 slow ops, oldest one blocked for 15130 sec, osd.5 has slow ops 2022-05-14T03:40:01.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:40:00 smithi115 ceph-mon[78593]: [WRN] FS_DEGRADED: 1 filesystem is degraded 2022-05-14T03:40:01.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:40:00 smithi115 ceph-mon[78593]: fs cephfs is degraded 2022-05-14T03:40:01.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:40:00 smithi115 ceph-mon[78593]: [ERR] MDS_DAMAGE: 1 mds daemon damaged 2022-05-14T03:40:01.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:40:00 smithi115 ceph-mon[78593]: fs cephfs mds.2 is damaged 2022-05-14T03:40:01.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:40:00 smithi115 ceph-mon[78593]: [WRN] MDS_READ_ONLY: 1 MDSs are read only 2022-05-14T03:40:01.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:40:00 smithi115 ceph-mon[78593]: mds.l(mds.0): MDS in read-only mode 2022-05-14T03:40:01.112 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:40:00 smithi115 ceph-mon[78593]: [WRN] MDS_SLOW_METADATA_IO: 1 MDSs report slow metadata IOs 2022-05-14T03:40:01.112 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:40:00 smithi115 ceph-mon[78593]: mds.l(mds.0): 4 slow metadata IOs are blocked > 180 secs, oldest blocked for 15433 secs 2022-05-14T03:40:01.112 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:40:00 smithi115 ceph-mon[78593]: [WRN] MON_DOWN: 1/3 mons down, quorum b,c 2022-05-14T03:40:01.112 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:40:00 smithi115 ceph-mon[78593]: mon.a (rank 0) addr [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] is down (out of quorum) 2022-05-14T03:40:01.113 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:40:00 smithi115 ceph-mon[78593]: [WRN] PG_AVAILABILITY: Reduced data availability: 1 pg inactive 2022-05-14T03:40:01.113 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:40:00 smithi115 ceph-mon[78593]: pg 2.1 is stuck inactive for 4h, current state unknown, last acting [] 2022-05-14T03:40:01.113 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:40:00 smithi115 ceph-mon[78593]: [WRN] SLOW_OPS: 9 slow ops, oldest one blocked for 15130 sec, osd.5 has slow ops 2022-05-14T03:40:02.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:40:01 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:40:02.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:40:01 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 15135 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:40:02.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:40:01 smithi115 conmon[90393]: 2022-05-14T03:40:01.651+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:40:02.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:40:01 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:40:02.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:40:01 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 15135 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:40:03.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:40:02 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:40:03.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:40:02 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:40:03.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:40:02 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:40:03.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:40:02 smithi047 ceph-mon[77147]: pgmap v7919: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:40:03.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:40:02 smithi115 conmon[90393]: 2022-05-14T03:40:02.650+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:40:03.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:40:02 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:40:03.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:40:02 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:40:03.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:40:02 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:40:03.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:40:02 smithi115 ceph-mon[78593]: pgmap v7919: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:40:04.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:40:03 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:40:04.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:40:03 smithi115 conmon[90393]: 2022-05-14T03:40:03.680+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:40:04.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:40:03 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:40:05.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:40:04 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:40:05.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:40:04 smithi047 ceph-mon[77147]: pgmap v7920: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:40:05.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:40:04 smithi115 conmon[90393]: 2022-05-14T03:40:04.706+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:40:05.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:40:04 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:40:05.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:40:04 smithi115 ceph-mon[78593]: pgmap v7920: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:40:06.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:40:05 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:40:06.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:40:05 smithi115 conmon[90393]: 2022-05-14T03:40:05.692+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:40:06.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:40:05 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:40:07.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:40:06 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:40:07.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:40:06 smithi047 ceph-mon[77147]: pgmap v7921: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:40:07.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:40:06 smithi115 conmon[90393]: 2022-05-14T03:40:06.656+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:40:07.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:40:06 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:40:07.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:40:06 smithi115 ceph-mon[78593]: pgmap v7921: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:40:08.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:40:07 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:40:08.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:40:07 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 15140 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:40:08.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:40:07 smithi115 conmon[90393]: 2022-05-14T03:40:07.632+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:40:08.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:40:07 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:40:08.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:40:07 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 15140 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:40:09.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:40:08 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:40:09.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:40:08 smithi047 ceph-mon[77147]: pgmap v7922: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:40:09.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:40:08 smithi115 conmon[90393]: 2022-05-14T03:40:08.629+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:40:09.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:40:08 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:40:09.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:40:08 smithi115 ceph-mon[78593]: pgmap v7922: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:40:10.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:40:09 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:40:10.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:40:09 smithi115 conmon[90393]: 2022-05-14T03:40:09.645+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:40:10.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:40:09 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:40:10.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:40:10 smithi115 conmon[90393]: 2022-05-14T03:40:10.603+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:40:10.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:40:10 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:40:10.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:40:10 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:40:10.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:40:10 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:40:10.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:40:10 smithi115 ceph-mon[78593]: pgmap v7923: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:40:11.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:40:10 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:40:11.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:40:10 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:40:11.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:40:10 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:40:11.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:40:10 smithi047 ceph-mon[77147]: pgmap v7923: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:40:11.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:40:11 smithi115 conmon[90393]: 2022-05-14T03:40:11.567+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:40:11.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:40:11 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:40:12.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:40:11 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:40:12.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:40:12 smithi115 conmon[90393]: 2022-05-14T03:40:12.575+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:40:12.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:40:12 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:40:12.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:40:12 smithi115 ceph-mon[78593]: pgmap v7924: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:40:13.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:40:12 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:40:13.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:40:12 smithi047 ceph-mon[77147]: pgmap v7924: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:40:13.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:40:13 smithi115 conmon[90393]: 2022-05-14T03:40:13.554+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:40:13.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:40:13 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:40:14.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:40:13 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:40:14.791 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:40:14 smithi115 conmon[90393]: 2022-05-14T03:40:14.552+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:40:14.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:40:14 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:40:14.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:40:14 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:40:15.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:40:14 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:40:15.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:40:14 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:40:15.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:40:14 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:40:15.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:40:14 smithi047 ceph-mon[77147]: pgmap v7925: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:40:15.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:40:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T03:40:15.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:40:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T03:40:15.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:40:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T03:40:15.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:40:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T03:40:15.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:40:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T03:40:15.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:40:14 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:40:15.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:40:14 smithi115 ceph-mon[78593]: pgmap v7925: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:40:15.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:40:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T03:40:15.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:40:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T03:40:15.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:40:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T03:40:15.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:40:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T03:40:15.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:40:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T03:40:15.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:40:15 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:40:15.860 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:40:15 smithi115 conmon[90393]: 2022-05-14T03:40:15.544+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:40:16.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:40:15 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:40:16.797 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:40:16 smithi115 conmon[90393]: 2022-05-14T03:40:16.519+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:40:17.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:40:16 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:40:17.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:40:16 smithi115 ceph-mon[78593]: pgmap v7926: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:40:17.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:40:16 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 15145 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:40:17.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:40:16 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:40:17.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:40:16 smithi047 ceph-mon[77147]: pgmap v7926: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:40:17.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:40:16 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 15145 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:40:17.803 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:40:17 smithi115 conmon[90393]: 2022-05-14T03:40:17.521+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:40:18.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:40:17 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:40:18.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:40:17 smithi047 conmon[78521]: 2022-05-14T03:40:17.584+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:40:18.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:40:17 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T03:40:18.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:40:17 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:40:18.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:40:17 smithi047 conmon[78521]: yield 2022-05-14T03:40:18.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:40:17 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:40:18.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:40:17 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:40:18.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:40:17 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:40:18.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:40:17 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T03:40:18.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:40:17 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:40:18.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:40:17 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T03:40:18.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:40:17 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:40:18.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:40:17 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T03:40:18.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:40:17 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:40:18.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:40:17 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T03:40:18.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:40:17 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:40:18.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:40:17 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T03:40:18.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:40:17 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:40:18.048 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:40:17 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:40:18.048 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:40:17 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:40:18.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:40:17 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:40:18.808 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:40:18 smithi115 conmon[90393]: 2022-05-14T03:40:18.502+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:40:18.809 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:40:18 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:40:18.810 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:40:18 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:40:18.810 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T03:40:18.811 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:40:18.811 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T03:40:18.811 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:40:18.812 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:40:18.812 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:40:18.813 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T03:40:18.813 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:40:18.814 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T03:40:18.814 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:40:18.814 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T03:40:18.815 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:40:18.815 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T03:40:18.816 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:40:18.816 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T03:40:18.816 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:40:18.817 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:40:18.817 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:40:18.817 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:40:18 smithi115 ceph-mon[78593]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T03:40:18.818 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:40:18 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:40:18.818 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:40:18 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:40:18.819 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:40:18 smithi115 ceph-mon[78593]: pgmap v7927: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:40:19.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:40:18 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:40:19.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:40:18 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:40:19.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T03:40:19.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:40:19.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T03:40:19.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:40:19.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:40:19.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:40:19.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T03:40:19.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:40:19.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T03:40:19.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:40:19.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T03:40:19.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:40:19.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T03:40:19.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:40:19.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T03:40:19.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:40:19.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:40:19.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:40:19.297 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:40:18 smithi047 ceph-mon[77147]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T03:40:19.297 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:40:18 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:40:19.297 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:40:18 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:40:19.298 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:40:18 smithi047 ceph-mon[77147]: pgmap v7927: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:40:19.808 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:40:19 smithi115 conmon[90393]: 2022-05-14T03:40:19.457+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:40:20.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:40:19 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:40:20.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:40:19 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:40:20.813 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:40:20 smithi115 conmon[90393]: 2022-05-14T03:40:20.411+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:40:21.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:40:20 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:40:21.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:40:20 smithi047 ceph-mon[77147]: pgmap v7928: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:40:21.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:40:20 smithi047 conmon[78521]: 2022-05-14T03:40:20.656+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:40:21.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:40:20 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T03:40:21.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:40:20 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:40:21.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:40:20 smithi047 conmon[78521]: yield 2022-05-14T03:40:21.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:40:20 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:40:21.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:40:20 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:40:21.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:40:20 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:40:21.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:40:20 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T03:40:21.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:40:20 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:40:21.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:40:20 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T03:40:21.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:40:20 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:40:21.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:40:20 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T03:40:21.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:40:20 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:40:21.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:40:20 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T03:40:21.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:40:20 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:40:21.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:40:20 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T03:40:21.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:40:20 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:40:21.048 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:40:20 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:40:21.048 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:40:20 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:40:21.048 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:40:20 smithi047 conmon[78521]: 2022-05-14T03:40:20.657+0000 7f533f777700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T03:40:21.048 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:40:20 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T03:40:21.048 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:40:20 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:40:21.049 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:40:20 smithi047 conmon[78521]: yield 2022-05-14T03:40:21.049 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:40:20 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:40:21.049 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:40:20 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:40:21.049 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:40:20 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:40:21.050 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:40:20 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T03:40:21.050 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:40:20 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:40:21.050 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:40:20 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T03:40:21.050 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:40:20 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:40:21.051 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:40:20 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T03:40:21.051 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:40:20 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:40:21.051 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:40:20 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T03:40:21.051 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:40:20 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:40:21.051 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:40:20 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T03:40:21.052 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:40:20 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:40:21.052 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:40:20 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:40:21.052 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:40:20 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:40:21.052 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:40:20 smithi047 conmon[78521]: 2022-05-14T03:40:21.053 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:40:20 smithi047 conmon[78521]: During handling of the above exception, another exception occurred: 2022-05-14T03:40:21.053 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:40:20 smithi047 conmon[78521]: 2022-05-14T03:40:21.053 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:40:20 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T03:40:21.053 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:40:20 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T03:40:21.053 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:40:20 smithi047 conmon[78521]: return f(*arg) 2022-05-14T03:40:21.054 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:40:20 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T03:40:21.054 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:40:20 smithi047 conmon[78521]: self._write_client_files(client_files, host) 2022-05-14T03:40:21.054 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:40:20 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T03:40:21.054 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:40:20 smithi047 conmon[78521]: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T03:40:21.055 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:40:20 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T03:40:21.055 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:40:20 smithi047 conmon[78521]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T03:40:21.055 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:40:20 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T03:40:21.055 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:40:20 smithi047 conmon[78521]: return self.event_loop.get_result(coro) 2022-05-14T03:40:21.055 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:40:20 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T03:40:21.056 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:40:20 smithi047 conmon[78521]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T03:40:21.056 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:40:20 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T03:40:21.056 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:40:20 smithi047 conmon[78521]: return self.__get_result() 2022-05-14T03:40:21.056 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:40:20 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T03:40:21.056 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:40:20 smithi047 conmon[78521]: raise self._exception 2022-05-14T03:40:21.057 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:40:20 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T03:40:21.057 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:40:20 smithi047 conmon[78521]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T03:40:21.057 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:40:20 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T03:40:21.057 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:40:20 smithi047 conmon[78521]: conn = await self._remote_connection(host, addr) 2022-05-14T03:40:21.058 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:40:20 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T03:40:21.058 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:40:20 smithi047 conmon[78521]: raise 2022-05-14T03:40:21.058 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:40:20 smithi047 conmon[78521]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T03:40:21.058 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:40:20 smithi047 conmon[78521]: self.gen.throw(type, value, traceback) 2022-05-14T03:40:21.058 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:40:20 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T03:40:21.059 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:40:20 smithi047 conmon[78521]: raise OrchestratorError(msg) 2022-05-14T03:40:21.059 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:40:20 smithi047 conmon[78521]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:40:21.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:40:20 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:40:21.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:40:20 smithi115 ceph-mon[78593]: pgmap v7928: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:40:21.815 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:40:21 smithi115 conmon[90393]: 2022-05-14T03:40:21.455+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:40:22.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:40:21 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:40:22.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:40:21 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:40:22.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T03:40:22.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:40:22.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T03:40:22.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:40:22.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:40:22.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:40:22.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T03:40:22.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:40:22.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T03:40:22.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:40:22.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T03:40:22.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:40:22.113 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T03:40:22.113 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:40:22.113 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T03:40:22.113 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:40:22.113 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:40:22.114 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:40:22.114 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:40:21 smithi115 ceph-mon[78593]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T03:40:22.114 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T03:40:22.114 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:40:22.115 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T03:40:22.115 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:40:22.115 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:40:22.115 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:40:22.115 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T03:40:22.116 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:40:22.116 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T03:40:22.116 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:40:22.116 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T03:40:22.116 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:40:22.117 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T03:40:22.117 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:40:22.117 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T03:40:22.117 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:40:22.118 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:40:22.118 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:40:22.118 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T03:40:22.118 INFO:journalctl@ceph.mon.c.smithi115.stdout: During handling of the above exception, another exception occurred: 2022-05-14T03:40:22.118 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T03:40:22.119 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T03:40:22.119 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T03:40:22.119 INFO:journalctl@ceph.mon.c.smithi115.stdout: return f(*arg) 2022-05-14T03:40:22.119 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T03:40:22.119 INFO:journalctl@ceph.mon.c.smithi115.stdout: self._write_client_files(client_files, host) 2022-05-14T03:40:22.120 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T03:40:22.120 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T03:40:22.120 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T03:40:22.120 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T03:40:22.121 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T03:40:22.121 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.event_loop.get_result(coro) 2022-05-14T03:40:22.121 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T03:40:22.121 INFO:journalctl@ceph.mon.c.smithi115.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T03:40:22.121 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T03:40:22.122 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.__get_result() 2022-05-14T03:40:22.122 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T03:40:22.122 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise self._exception 2022-05-14T03:40:22.122 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T03:40:22.123 INFO:journalctl@ceph.mon.c.smithi115.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T03:40:22.123 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T03:40:22.123 INFO:journalctl@ceph.mon.c.smithi115.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T03:40:22.123 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T03:40:22.123 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise 2022-05-14T03:40:22.124 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T03:40:22.124 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.gen.throw(type, value, traceback) 2022-05-14T03:40:22.124 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T03:40:22.124 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OrchestratorError(msg) 2022-05-14T03:40:22.124 INFO:journalctl@ceph.mon.c.smithi115.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:40:22.125 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:40:21 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 15155 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:40:22.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:40:21 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:40:22.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:40:21 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:40:22.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T03:40:22.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:40:22.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T03:40:22.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:40:22.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:40:22.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:40:22.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T03:40:22.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:40:22.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T03:40:22.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:40:22.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T03:40:22.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:40:22.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T03:40:22.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:40:22.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T03:40:22.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:40:22.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:40:22.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:40:22.297 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:40:21 smithi047 ceph-mon[77147]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T03:40:22.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T03:40:22.298 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:40:22.298 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T03:40:22.298 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:40:22.298 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:40:22.298 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:40:22.299 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T03:40:22.299 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:40:22.299 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T03:40:22.299 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:40:22.300 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T03:40:22.300 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:40:22.300 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T03:40:22.300 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:40:22.300 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T03:40:22.301 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:40:22.301 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:40:22.301 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:40:22.301 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T03:40:22.302 INFO:journalctl@ceph.mon.b.smithi047.stdout: During handling of the above exception, another exception occurred: 2022-05-14T03:40:22.302 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T03:40:22.302 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T03:40:22.302 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T03:40:22.302 INFO:journalctl@ceph.mon.b.smithi047.stdout: return f(*arg) 2022-05-14T03:40:22.303 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T03:40:22.303 INFO:journalctl@ceph.mon.b.smithi047.stdout: self._write_client_files(client_files, host) 2022-05-14T03:40:22.303 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T03:40:22.303 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T03:40:22.303 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T03:40:22.304 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T03:40:22.304 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T03:40:22.304 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.event_loop.get_result(coro) 2022-05-14T03:40:22.304 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T03:40:22.305 INFO:journalctl@ceph.mon.b.smithi047.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T03:40:22.305 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T03:40:22.305 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.__get_result() 2022-05-14T03:40:22.305 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T03:40:22.305 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise self._exception 2022-05-14T03:40:22.306 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T03:40:22.306 INFO:journalctl@ceph.mon.b.smithi047.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T03:40:22.306 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T03:40:22.306 INFO:journalctl@ceph.mon.b.smithi047.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T03:40:22.306 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T03:40:22.307 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise 2022-05-14T03:40:22.307 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T03:40:22.307 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.gen.throw(type, value, traceback) 2022-05-14T03:40:22.307 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T03:40:22.307 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OrchestratorError(msg) 2022-05-14T03:40:22.308 INFO:journalctl@ceph.mon.b.smithi047.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:40:22.308 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:40:21 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 15155 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:40:22.824 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:40:22 smithi115 conmon[90393]: 2022-05-14T03:40:22.505+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:40:23.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:40:22 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:40:23.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:40:22 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:40:23.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:40:22 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:40:23.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:40:22 smithi115 ceph-mon[78593]: pgmap v7929: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:40:23.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:40:22 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:40:23.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:40:22 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:40:23.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:40:22 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:40:23.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:40:22 smithi047 ceph-mon[77147]: pgmap v7929: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:40:23.825 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:40:23 smithi115 conmon[90393]: 2022-05-14T03:40:23.492+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:40:24.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:40:23 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:40:24.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:40:23 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:40:24.831 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:40:24 smithi115 conmon[90393]: 2022-05-14T03:40:24.523+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:40:24.831 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:40:24 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:40:25.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:40:24 smithi115 ceph-mon[78593]: pgmap v7930: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:40:25.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:40:24 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:40:25.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:40:24 smithi047 ceph-mon[77147]: pgmap v7930: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:40:25.832 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:40:25 smithi115 conmon[90393]: 2022-05-14T03:40:25.567+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:40:26.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:40:25 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:40:26.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:40:25 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:40:26.836 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:40:26 smithi115 conmon[90393]: 2022-05-14T03:40:26.543+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:40:27.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:40:26 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:40:27.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:40:26 smithi115 ceph-mon[78593]: pgmap v7931: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:40:27.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:40:26 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:40:27.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:40:26 smithi047 ceph-mon[77147]: pgmap v7931: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:40:27.838 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:40:27 smithi115 conmon[90393]: 2022-05-14T03:40:27.569+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:40:28.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:40:27 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:40:28.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:40:27 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 15160 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:40:28.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:40:27 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:40:28.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:40:27 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 15160 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:40:29.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:40:28 smithi115 conmon[90393]: 2022-05-14T03:40:28.613+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:40:29.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:40:28 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:40:29.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:40:28 smithi115 ceph-mon[78593]: pgmap v7932: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:40:29.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:40:28 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:40:29.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:40:28 smithi047 ceph-mon[77147]: pgmap v7932: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:40:30.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:40:29 smithi115 conmon[90393]: 2022-05-14T03:40:29.628+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:40:30.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:40:29 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:40:30.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:40:29 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:40:31.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:40:30 smithi115 conmon[90393]: 2022-05-14T03:40:30.663+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:40:31.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:40:30 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:40:31.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:40:30 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:40:31.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:40:30 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:40:31.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:40:30 smithi115 ceph-mon[78593]: pgmap v7933: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:40:31.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:40:30 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:40:31.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:40:30 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:40:31.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:40:30 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:40:31.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:40:30 smithi047 ceph-mon[77147]: pgmap v7933: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:40:32.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:40:31 smithi115 conmon[90393]: 2022-05-14T03:40:31.661+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:40:32.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:40:31 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:40:32.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:40:31 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:40:33.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:40:32 smithi115 conmon[90393]: 2022-05-14T03:40:32.705+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:40:33.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:40:32 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:40:33.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:40:32 smithi115 ceph-mon[78593]: pgmap v7934: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:40:33.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:40:32 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:40:33.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:40:32 smithi047 ceph-mon[77147]: pgmap v7934: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:40:34.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:40:33 smithi115 conmon[90393]: 2022-05-14T03:40:33.738+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:40:34.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:40:33 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:40:34.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:40:33 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:40:35.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:40:34 smithi115 conmon[90393]: 2022-05-14T03:40:34.787+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:40:35.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:40:34 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:40:35.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:40:34 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:40:35.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:40:34 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:40:35.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:40:34 smithi115 ceph-mon[78593]: pgmap v7935: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:40:35.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:40:34 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:40:35.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:40:34 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:40:35.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:40:34 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:40:35.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:40:34 smithi047 ceph-mon[77147]: pgmap v7935: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:40:36.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:40:35 smithi115 conmon[90393]: 2022-05-14T03:40:35.772+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:40:36.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:40:35 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:40:36.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:40:35 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:40:37.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:40:36 smithi115 conmon[90393]: 2022-05-14T03:40:36.796+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:40:37.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:40:36 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:40:37.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:40:36 smithi115 ceph-mon[78593]: pgmap v7936: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:40:37.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:40:36 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 15170 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:40:37.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:40:36 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:40:37.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:40:36 smithi047 ceph-mon[77147]: pgmap v7936: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:40:37.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:40:36 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 15170 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:40:38.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:40:37 smithi115 conmon[90393]: 2022-05-14T03:40:37.795+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:40:38.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:40:37 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:40:38.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:40:37 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:40:39.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:40:38 smithi115 conmon[90393]: 2022-05-14T03:40:38.818+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:40:39.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:40:38 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:40:39.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:40:38 smithi115 ceph-mon[78593]: pgmap v7937: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:40:39.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:40:38 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:40:39.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:40:38 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:40:39.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:40:38 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:40:39.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:40:38 smithi047 ceph-mon[77147]: pgmap v7937: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:40:39.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:40:38 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:40:39.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:40:38 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:40:40.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:40:39 smithi115 conmon[90393]: 2022-05-14T03:40:39.771+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:40:40.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:40:39 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:40:40.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:40:39 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:40:41.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:40:40 smithi115 conmon[90393]: 2022-05-14T03:40:40.775+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:40:41.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:40:40 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:40:41.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:40:40 smithi115 ceph-mon[78593]: pgmap v7938: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:40:41.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:40:40 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:40:41.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:40:40 smithi047 ceph-mon[77147]: pgmap v7938: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:40:42.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:40:41 smithi115 conmon[90393]: 2022-05-14T03:40:41.813+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:40:42.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:40:41 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:40:42.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:40:41 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 15175 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:40:42.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:40:41 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:40:42.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:40:41 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 15175 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:40:43.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:40:42 smithi115 conmon[90393]: 2022-05-14T03:40:42.841+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:40:43.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:40:42 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:40:43.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:40:42 smithi115 ceph-mon[78593]: pgmap v7939: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:40:43.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:40:42 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:40:43.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:40:42 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:40:43.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:40:42 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:40:43.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:40:42 smithi047 ceph-mon[77147]: pgmap v7939: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:40:43.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:40:42 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:40:43.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:40:42 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:40:44.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:40:43 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:40:44.110 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:40:43 smithi115 conmon[90393]: 2022-05-14T03:40:43.837+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:40:44.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:40:43 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:40:45.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:40:44 smithi115 conmon[90393]: 2022-05-14T03:40:44.825+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:40:45.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:40:44 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:40:45.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:40:44 smithi115 ceph-mon[78593]: pgmap v7940: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:40:45.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:40:44 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:40:45.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:40:44 smithi047 ceph-mon[77147]: pgmap v7940: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:40:46.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:40:45 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:40:46.111 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:40:45 smithi115 conmon[90393]: 2022-05-14T03:40:45.851+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:40:46.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:40:45 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:40:47.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:40:46 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:40:47.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:40:46 smithi047 ceph-mon[77147]: pgmap v7941: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:40:47.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:40:46 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 15180 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:40:47.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:40:46 smithi115 conmon[90393]: 2022-05-14T03:40:46.899+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:40:47.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:40:46 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:40:47.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:40:46 smithi115 ceph-mon[78593]: pgmap v7941: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:40:47.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:40:46 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 15180 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:40:48.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:40:47 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:40:48.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:40:47 smithi115 conmon[90393]: 2022-05-14T03:40:47.859+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:40:48.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:40:47 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:40:49.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:40:48 smithi115 conmon[90393]: 2022-05-14T03:40:48.817+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:40:49.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:40:48 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:40:49.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:40:48 smithi115 ceph-mon[78593]: pgmap v7942: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:40:49.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:40:48 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:40:49.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:40:48 smithi047 ceph-mon[77147]: pgmap v7942: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:40:50.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:40:49 smithi115 conmon[90393]: 2022-05-14T03:40:49.817+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:40:50.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:40:49 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:40:50.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:40:49 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:40:51.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:40:50 smithi115 conmon[90393]: 2022-05-14T03:40:50.793+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:40:51.112 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:40:50 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:40:51.112 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:40:50 smithi115 ceph-mon[78593]: pgmap v7943: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:40:51.112 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:40:50 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:40:51.112 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:40:50 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:40:51.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:40:50 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:40:51.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:40:50 smithi047 ceph-mon[77147]: pgmap v7943: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:40:51.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:40:50 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:40:51.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:40:50 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:40:52.110 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:40:51 smithi115 conmon[90393]: 2022-05-14T03:40:51.769+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:40:52.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:40:51 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:40:52.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:40:51 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 15185 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:40:52.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:40:51 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:40:52.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:40:51 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 15185 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:40:53.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:40:52 smithi115 conmon[90393]: 2022-05-14T03:40:52.776+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:40:53.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:40:52 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:40:53.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:40:52 smithi115 ceph-mon[78593]: pgmap v7944: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:40:53.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:40:52 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:40:53.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:40:52 smithi047 ceph-mon[77147]: pgmap v7944: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:40:54.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:40:53 smithi115 conmon[90393]: 2022-05-14T03:40:53.787+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:40:54.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:40:53 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:40:54.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:40:53 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:40:55.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:40:54 smithi115 conmon[90393]: 2022-05-14T03:40:54.780+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:40:55.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:40:54 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:40:55.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:40:54 smithi115 ceph-mon[78593]: pgmap v7945: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:40:55.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:40:54 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:40:55.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:40:54 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:40:55.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:40:54 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:40:55.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:40:54 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:40:55.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:40:54 smithi047 ceph-mon[77147]: pgmap v7945: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:40:55.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:40:54 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:40:55.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:40:54 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:40:55.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:40:54 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:40:56.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:40:55 smithi115 conmon[90393]: 2022-05-14T03:40:55.771+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:40:56.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:40:55 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:40:56.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:40:55 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:40:57.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:40:56 smithi115 conmon[90393]: 2022-05-14T03:40:56.817+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:40:57.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:40:56 smithi115 ceph-mon[78593]: pgmap v7946: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:40:57.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:40:56 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 15190 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:40:57.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:40:56 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:40:57.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:40:56 smithi047 ceph-mon[77147]: pgmap v7946: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:40:57.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:40:56 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 15190 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:40:57.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:40:56 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:40:58.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:40:57 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:40:58.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:40:57 smithi115 conmon[90393]: 2022-05-14T03:40:57.856+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:40:58.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:40:57 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:40:59.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:40:58 smithi047 ceph-mon[77147]: pgmap v7947: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:40:59.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:40:58 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:40:59.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:40:58 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:40:59.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:40:58 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:40:59.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:40:58 smithi115 conmon[90393]: 2022-05-14T03:40:58.865+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:40:59.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:40:58 smithi115 ceph-mon[78593]: pgmap v7947: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:40:59.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:40:58 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:40:59.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:40:58 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:40:59.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:40:58 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:41:00.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:40:59 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:41:00.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:40:59 smithi115 conmon[90393]: 2022-05-14T03:40:59.857+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:41:00.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:40:59 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:41:01.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:41:00 smithi115 conmon[90393]: 2022-05-14T03:41:00.814+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:41:01.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:41:00 smithi115 ceph-mon[78593]: pgmap v7948: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:41:01.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:41:00 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:41:01.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:41:00 smithi047 ceph-mon[77147]: pgmap v7948: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:41:01.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:41:00 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:41:01.960 INFO:journalctl@ceph.osd.11.smithi115.stdout:May 14 03:41:01 smithi115 conmon[102145]: 2022-05-14T03:41:01.780+0000 7f98e16bf700 -1 received signal: Hangup from Kernel ( Could be generated by pthread_kill(), raise(), abort(), alarm() ) UID: 0 2022-05-14T03:41:01.960 INFO:journalctl@ceph.osd.11.smithi115.stdout:May 14 03:41:01 smithi115 conmon[102145]: 2022-05-14T03:41:01.846+0000 7f98e16bf700 -1 received signal: Hangup from Kernel ( Could be generated by pthread_kill(), raise(), abort(), alarm() ) UID: 0 2022-05-14T03:41:01.961 INFO:journalctl@ceph.mgr.z.smithi115.stdout:May 14 03:41:01 smithi115 conmon[80399]: 2022-05-14T03:41:01.780+0000 7f6205d37700 -1 received signal: Hangup from Kernel ( Could be generated by pthread_kill(), raise(), abort(), alarm() ) UID: 0 2022-05-14T03:41:01.961 INFO:journalctl@ceph.mgr.z.smithi115.stdout:May 14 03:41:01 smithi115 conmon[80399]: 2022-05-14T03:41:01.846+0000 7f6205d37700 -1 received signal: Hangup from Kernel ( Could be generated by pthread_kill(), raise(), abort(), alarm() ) UID: 0 2022-05-14T03:41:01.962 INFO:journalctl@ceph.osd.2.smithi115.stdout:May 14 03:41:01 smithi115 conmon[84560]: 2022-05-14T03:41:01.780+0000 7ff4a43da700 -1 received signal: Hangup from Kernel ( Could be generated by pthread_kill(), raise(), abort(), alarm() ) UID: 0 2022-05-14T03:41:01.962 INFO:journalctl@ceph.osd.2.smithi115.stdout:May 14 03:41:01 smithi115 conmon[84560]: 2022-05-14T03:41:01.846+0000 7ff4a43da700 -1 received signal: Hangup from Kernel ( Could be generated by pthread_kill(), raise(), abort(), alarm() ) UID: 0 2022-05-14T03:41:01.963 INFO:journalctl@ceph.mds.f.smithi115.stdout:May 14 03:41:01 smithi115 conmon[105998]: 2022-05-14T03:41:01.781+0000 7ff925564700 -1 received signal: Hangup from Kernel ( Could be generated by pthread_kill(), raise(), abort(), alarm() ) UID: 0 2022-05-14T03:41:01.963 INFO:journalctl@ceph.mds.f.smithi115.stdout:May 14 03:41:01 smithi115 conmon[105998]: 2022-05-14T03:41:01.846+0000 7ff925564700 -1 received signal: Hangup from Kernel ( Could be generated by pthread_kill(), raise(), abort(), alarm() ) UID: 0 2022-05-14T03:41:01.963 INFO:journalctl@ceph.mds.i.smithi115.stdout:May 14 03:41:01 smithi115 conmon[106567]: 2022-05-14T03:41:01.780+0000 7f4ea00fc700 -1 received signal: Hangup from Kernel ( Could be generated by pthread_kill(), raise(), abort(), alarm() ) UID: 0 2022-05-14T03:41:01.964 INFO:journalctl@ceph.mds.i.smithi115.stdout:May 14 03:41:01 smithi115 conmon[106567]: 2022-05-14T03:41:01.846+0000 7f4ea00fc700 -1 received signal: Hangup from Kernel ( Could be generated by pthread_kill(), raise(), abort(), alarm() ) UID: 0 2022-05-14T03:41:01.964 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:41:01 smithi115 conmon[90393]: 2022-05-14T03:41:01.780+0000 7fa08c875700 -1 received signal: Hangup from Kernel ( Could be generated by pthread_kill(), raise(), abort(), alarm() ) UID: 0 2022-05-14T03:41:01.964 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:41:01 smithi115 conmon[90393]: 2022-05-14T03:41:01.785+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:41:01.965 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:41:01 smithi115 conmon[90393]: 2022-05-14T03:41:01.846+0000 7fa08c875700 -1 received signal: Hangup from Kernel ( Could be generated by pthread_kill(), raise(), abort(), alarm() ) UID: 0 2022-05-14T03:41:01.965 INFO:journalctl@ceph.osd.8.smithi115.stdout:May 14 03:41:01 smithi115 conmon[96252]: 2022-05-14T03:41:01.780+0000 7f226266c700 -1 received signal: Hangup from Kernel ( Could be generated by pthread_kill(), raise(), abort(), alarm() ) UID: 0 2022-05-14T03:41:01.966 INFO:journalctl@ceph.osd.8.smithi115.stdout:May 14 03:41:01 smithi115 conmon[96252]: 2022-05-14T03:41:01.846+0000 7f226266c700 -1 received signal: Hangup from Kernel ( Could be generated by pthread_kill(), raise(), abort(), alarm() ) UID: 0 2022-05-14T03:41:01.966 INFO:journalctl@ceph.mds.c.smithi115.stdout:May 14 03:41:01 smithi115 conmon[105427]: 2022-05-14T03:41:01.780+0000 7f13a98e9700 -1 received signal: Hangup from Kernel ( Could be generated by pthread_kill(), raise(), abort(), alarm() ) UID: 0 2022-05-14T03:41:01.966 INFO:journalctl@ceph.mds.c.smithi115.stdout:May 14 03:41:01 smithi115 conmon[105427]: 2022-05-14T03:41:01.846+0000 7f13a98e9700 -1 received signal: Hangup from Kernel ( Could be generated by pthread_kill(), raise(), abort(), alarm() ) UID: 0 2022-05-14T03:41:01.967 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:41:01 smithi115 conmon[78569]: 2022-05-14T03:41:01.780+0000 7fd6b3773700 -1 received signal: Hangup from Kernel ( Could be generated by pthread_kill(), raise(), abort(), alarm() ) UID: 0 2022-05-14T03:41:01.967 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:41:01 smithi115 conmon[78569]: 2022-05-14T03:41:01.780+0000 7fd6b3773700 -1 mon.c@2(peon) e3 *** Got Signal Hangup *** 2022-05-14T03:41:01.967 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:41:01 smithi115 conmon[78569]: 2022-05-14T03:41:01.846+0000 7fd6b3773700 -1 received signal: Hangup from Kernel ( Could be generated by pthread_kill(), raise(), abort(), alarm() ) UID: 0 2022-05-14T03:41:01.968 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:41:01 smithi115 conmon[78569]: 2022-05-14T03:41:01.846+0000 7fd6b3773700 -1 mon.c@2(peon) e3 *** Got Signal Hangup *** 2022-05-14T03:41:01.968 INFO:journalctl@ceph.mds.l.smithi115.stdout:May 14 03:41:01 smithi115 conmon[107138]: 2022-05-14T03:41:01.780+0000 7f8df55c9700 -1 received signal: Hangup from Kernel ( Could be generated by pthread_kill(), raise(), abort(), alarm() ) UID: 0 2022-05-14T03:41:01.968 INFO:journalctl@ceph.mds.l.smithi115.stdout:May 14 03:41:01 smithi115 conmon[107138]: 2022-05-14T03:41:01.846+0000 7f8df55c9700 -1 received signal: Hangup from Kernel ( Could be generated by pthread_kill(), raise(), abort(), alarm() ) UID: 0 2022-05-14T03:41:02.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:41:01 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 15195 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:41:02.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:41:01 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:41:02.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:41:01 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 15195 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:41:02.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:41:01 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:41:03.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:41:02 smithi115 conmon[90393]: 2022-05-14T03:41:02.771+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:41:03.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:41:02 smithi115 ceph-mon[78593]: pgmap v7949: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:41:03.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:41:02 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:41:03.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:41:02 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:41:03.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:41:02 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:41:03.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:41:02 smithi047 ceph-mon[77147]: pgmap v7949: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:41:03.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:41:02 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:41:03.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:41:02 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:41:03.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:41:02 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:41:04.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:41:03 smithi115 conmon[90393]: 2022-05-14T03:41:03.802+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:41:04.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:41:03 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:41:04.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:41:03 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:41:05.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:41:04 smithi115 conmon[90393]: 2022-05-14T03:41:04.809+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:41:05.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:41:04 smithi115 ceph-mon[78593]: pgmap v7950: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:41:05.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:41:04 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:41:05.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:41:04 smithi047 ceph-mon[77147]: pgmap v7950: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:41:05.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:41:04 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:41:06.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:41:05 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:41:06.110 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:41:05 smithi115 conmon[90393]: 2022-05-14T03:41:05.854+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:41:06.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:41:05 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:41:07.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:41:06 smithi115 conmon[90393]: 2022-05-14T03:41:06.814+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:41:07.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:41:06 smithi115 ceph-mon[78593]: pgmap v7951: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:41:07.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:41:06 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 15200 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:41:07.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:41:06 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:41:07.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:41:06 smithi047 ceph-mon[77147]: pgmap v7951: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:41:07.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:41:06 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 15200 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:41:07.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:41:06 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:41:08.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:41:07 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:41:08.110 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:41:07 smithi115 conmon[90393]: 2022-05-14T03:41:07.851+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:41:08.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:41:07 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:41:09.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:41:08 smithi047 ceph-mon[77147]: pgmap v7952: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:41:09.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:41:08 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:41:09.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:41:08 smithi115 conmon[90393]: 2022-05-14T03:41:08.885+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:41:09.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:41:08 smithi115 ceph-mon[78593]: pgmap v7952: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:41:09.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:41:08 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:41:10.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:41:09 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:41:10.110 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:41:09 smithi115 conmon[90393]: 2022-05-14T03:41:09.838+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:41:10.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:41:09 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:41:11.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:41:10 smithi115 conmon[90393]: 2022-05-14T03:41:10.804+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:41:11.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:41:10 smithi115 ceph-mon[78593]: pgmap v7953: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:41:11.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:41:10 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:41:11.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:41:10 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:41:11.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:41:10 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:41:11.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:41:10 smithi047 ceph-mon[77147]: pgmap v7953: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:41:11.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:41:10 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:41:11.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:41:10 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:41:11.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:41:10 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:41:12.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:41:11 smithi115 conmon[90393]: 2022-05-14T03:41:11.760+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:41:12.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:41:11 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 15205 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:41:12.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:41:11 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:41:12.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:41:11 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 15205 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:41:12.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:41:11 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:41:13.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:41:12 smithi115 conmon[90393]: 2022-05-14T03:41:12.757+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:41:13.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:41:12 smithi115 ceph-mon[78593]: pgmap v7954: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:41:13.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:41:12 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:41:13.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:41:12 smithi047 ceph-mon[77147]: pgmap v7954: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:41:13.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:41:12 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:41:14.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:41:13 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:41:14.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:41:13 smithi115 conmon[90393]: 2022-05-14T03:41:13.740+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:41:14.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:41:13 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:41:15.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:41:14 smithi115 conmon[90393]: 2022-05-14T03:41:14.761+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:41:15.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:41:14 smithi115 ceph-mon[78593]: pgmap v7955: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:41:15.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:41:14 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:41:15.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:41:14 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:41:15.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:41:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T03:41:15.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:41:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T03:41:15.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:41:14 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:41:15.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:41:14 smithi047 ceph-mon[77147]: pgmap v7955: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:41:15.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:41:14 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:41:15.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:41:14 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:41:15.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:41:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T03:41:15.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:41:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T03:41:15.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:41:14 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:41:16.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:41:15 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:41:16.110 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:41:15 smithi115 conmon[90393]: 2022-05-14T03:41:15.790+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:41:16.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:41:15 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:41:17.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:41:16 smithi115 conmon[90393]: 2022-05-14T03:41:16.788+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:41:17.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:41:16 smithi115 ceph-mon[78593]: pgmap v7956: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:41:17.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:41:16 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 15210 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:41:17.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:41:16 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:41:17.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:41:16 smithi047 ceph-mon[77147]: pgmap v7956: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:41:17.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:41:16 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 15210 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:41:17.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:41:16 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:41:18.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:41:17 smithi115 conmon[90393]: 2022-05-14T03:41:17.775+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:41:18.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:41:18 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:41:18.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:41:18 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:41:19.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:41:18 smithi115 conmon[90393]: 2022-05-14T03:41:18.761+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:41:19.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:41:19 smithi115 ceph-mon[78593]: pgmap v7957: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:41:19.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:41:19 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:41:19.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:41:19 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:41:19.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:41:19 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:41:19.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:41:19 smithi047 ceph-mon[77147]: pgmap v7957: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:41:19.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:41:19 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:41:19.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:41:19 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:41:19.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:41:19 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:41:20.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:41:20 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:41:20.110 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:41:19 smithi115 conmon[90393]: 2022-05-14T03:41:19.768+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:41:20.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:41:20 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:41:21.010 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:41:20 smithi115 conmon[90393]: 2022-05-14T03:41:20.742+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:41:21.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:41:21 smithi047 ceph-mon[77147]: pgmap v7958: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:41:21.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:41:21 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T03:41:21.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:41:21 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T03:41:21.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:41:21 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T03:41:21.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:41:21 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:41:21.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:41:21 smithi047 conmon[78521]: 2022-05-14T03:41:21.135+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:41:21.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:41:21 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T03:41:21.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:41:21 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:41:21.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:41:21 smithi047 conmon[78521]: yield 2022-05-14T03:41:21.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:41:21 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:41:21.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:41:21 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:41:21.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:41:21 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:41:21.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:41:21 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T03:41:21.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:41:21 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:41:21.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:41:21 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T03:41:21.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:41:21 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:41:21.297 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:41:21 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T03:41:21.297 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:41:21 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:41:21.297 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:41:21 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T03:41:21.297 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:41:21 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:41:21.298 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:41:21 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T03:41:21.298 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:41:21 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:41:21.298 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:41:21 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:41:21.298 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:41:21 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:41:21.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:41:21 smithi115 ceph-mon[78593]: pgmap v7958: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:41:21.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:41:21 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T03:41:21.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:41:21 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T03:41:21.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:41:21 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T03:41:21.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:41:21 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:41:22.014 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:41:21 smithi115 conmon[90393]: 2022-05-14T03:41:21.712+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:41:22.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:41:22 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:41:22.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:41:22 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 15215 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:41:22.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:41:22 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:41:22.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:41:22 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 15215 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:41:23.015 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:41:22 smithi115 conmon[90393]: 2022-05-14T03:41:22.745+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:41:23.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:41:23 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:41:23.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T03:41:23.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:41:23.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T03:41:23.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:41:23.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:41:23.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:41:23.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T03:41:23.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:41:23.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T03:41:23.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:41:23.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T03:41:23.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:41:23.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T03:41:23.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:41:23.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T03:41:23.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:41:23.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:41:23.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:41:23.297 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:41:23 smithi047 ceph-mon[77147]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T03:41:23.297 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:41:23 smithi047 ceph-mon[77147]: pgmap v7959: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:41:23.297 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:41:23 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:41:23.298 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:41:23 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:41:23.298 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:41:23 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:41:23.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:41:23 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:41:23.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T03:41:23.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:41:23.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T03:41:23.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:41:23.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:41:23.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:41:23.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T03:41:23.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:41:23.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T03:41:23.365 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:41:23.365 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T03:41:23.366 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:41:23.366 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T03:41:23.366 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:41:23.367 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T03:41:23.367 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:41:23.368 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:41:23.368 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:41:23.368 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:41:23 smithi115 ceph-mon[78593]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T03:41:23.368 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:41:23 smithi115 ceph-mon[78593]: pgmap v7959: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:41:23.369 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:41:23 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:41:23.369 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:41:23 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:41:23.369 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:41:23 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:41:24.020 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:41:23 smithi115 conmon[90393]: 2022-05-14T03:41:23.717+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:41:24.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:41:24 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:41:24.293 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:41:24 smithi047 conmon[78521]: 2022-05-14T03:41:24.207+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:41:24.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:41:24 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T03:41:24.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:41:24 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:41:24.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:41:24 smithi047 conmon[78521]: yield 2022-05-14T03:41:24.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:41:24 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:41:24.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:41:24 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:41:24.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:41:24 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:41:24.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:41:24 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T03:41:24.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:41:24 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:41:24.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:41:24 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T03:41:24.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:41:24 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:41:24.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:41:24 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T03:41:24.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:41:24 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:41:24.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:41:24 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T03:41:24.297 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:41:24 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:41:24.297 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:41:24 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T03:41:24.297 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:41:24 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:41:24.297 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:41:24 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:41:24.298 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:41:24 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:41:24.298 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:41:24 smithi047 conmon[78521]: 2022-05-14T03:41:24.208+0000 7f5341f7c700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T03:41:24.298 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:41:24 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T03:41:24.299 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:41:24 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:41:24.299 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:41:24 smithi047 conmon[78521]: yield 2022-05-14T03:41:24.299 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:41:24 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:41:24.299 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:41:24 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:41:24.299 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:41:24 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:41:24.300 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:41:24 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T03:41:24.300 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:41:24 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:41:24.300 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:41:24 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T03:41:24.300 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:41:24 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:41:24.301 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:41:24 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T03:41:24.301 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:41:24 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:41:24.301 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:41:24 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T03:41:24.301 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:41:24 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:41:24.302 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:41:24 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T03:41:24.302 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:41:24 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:41:24.302 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:41:24 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:41:24.302 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:41:24 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:41:24.302 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:41:24 smithi047 conmon[78521]: 2022-05-14T03:41:24.303 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:41:24 smithi047 conmon[78521]: During handling of the above exception, another exception occurred: 2022-05-14T03:41:24.303 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:41:24 smithi047 conmon[78521]: 2022-05-14T03:41:24.303 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:41:24 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T03:41:24.303 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:41:24 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T03:41:24.304 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:41:24 smithi047 conmon[78521]: return f(*arg) 2022-05-14T03:41:24.304 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:41:24 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T03:41:24.304 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:41:24 smithi047 conmon[78521]: self._write_client_files(client_files, host) 2022-05-14T03:41:24.304 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:41:24 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T03:41:24.305 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:41:24 smithi047 conmon[78521]: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T03:41:24.305 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:41:24 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T03:41:24.305 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:41:24 smithi047 conmon[78521]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T03:41:24.306 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:41:24 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T03:41:24.306 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:41:24 smithi047 conmon[78521]: return self.event_loop.get_result(coro) 2022-05-14T03:41:24.306 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:41:24 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T03:41:24.306 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:41:24 smithi047 conmon[78521]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T03:41:24.307 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:41:24 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T03:41:24.307 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:41:24 smithi047 conmon[78521]: return self.__get_result() 2022-05-14T03:41:24.307 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:41:24 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T03:41:24.307 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:41:24 smithi047 conmon[78521]: raise self._exception 2022-05-14T03:41:24.308 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:41:24 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T03:41:24.308 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:41:24 smithi047 conmon[78521]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T03:41:24.308 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:41:24 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T03:41:24.308 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:41:24 smithi047 conmon[78521]: conn = await self._remote_connection(host, addr) 2022-05-14T03:41:24.308 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:41:24 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T03:41:24.309 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:41:24 smithi047 conmon[78521]: raise 2022-05-14T03:41:24.309 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:41:24 smithi047 conmon[78521]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T03:41:24.309 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:41:24 smithi047 conmon[78521]: self.gen.throw(type, value, traceback) 2022-05-14T03:41:24.310 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:41:24 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T03:41:24.310 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:41:24 smithi047 conmon[78521]: raise OrchestratorError(msg) 2022-05-14T03:41:24.310 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:41:24 smithi047 conmon[78521]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:41:24.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:41:24 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:41:25.021 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:41:24 smithi115 conmon[90393]: 2022-05-14T03:41:24.723+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:41:25.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:41:25 smithi047 ceph-mon[77147]: pgmap v7960: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:41:25.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:41:25 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:41:25.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:41:25 smithi115 ceph-mon[78593]: pgmap v7960: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:41:25.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:41:25 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:41:26.026 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:41:25 smithi115 conmon[90393]: 2022-05-14T03:41:25.717+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:41:26.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:41:26 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:41:26.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T03:41:26.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:41:26.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T03:41:26.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:41:26.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:41:26.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:41:26.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T03:41:26.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:41:26.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T03:41:26.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:41:26.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T03:41:26.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:41:26.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T03:41:26.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:41:26.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T03:41:26.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:41:26.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:41:26.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:41:26.298 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:41:26 smithi047 ceph-mon[77147]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T03:41:26.298 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T03:41:26.298 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:41:26.298 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T03:41:26.299 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:41:26.299 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:41:26.299 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:41:26.299 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T03:41:26.300 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:41:26.300 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T03:41:26.300 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:41:26.300 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T03:41:26.300 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:41:26.301 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T03:41:26.301 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:41:26.301 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T03:41:26.301 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:41:26.302 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:41:26.302 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:41:26.302 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T03:41:26.302 INFO:journalctl@ceph.mon.b.smithi047.stdout: During handling of the above exception, another exception occurred: 2022-05-14T03:41:26.302 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T03:41:26.303 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T03:41:26.303 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T03:41:26.303 INFO:journalctl@ceph.mon.b.smithi047.stdout: return f(*arg) 2022-05-14T03:41:26.303 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T03:41:26.303 INFO:journalctl@ceph.mon.b.smithi047.stdout: self._write_client_files(client_files, host) 2022-05-14T03:41:26.304 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T03:41:26.304 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T03:41:26.304 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T03:41:26.304 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T03:41:26.305 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T03:41:26.305 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.event_loop.get_result(coro) 2022-05-14T03:41:26.305 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T03:41:26.305 INFO:journalctl@ceph.mon.b.smithi047.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T03:41:26.306 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T03:41:26.306 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.__get_result() 2022-05-14T03:41:26.306 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T03:41:26.306 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise self._exception 2022-05-14T03:41:26.306 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T03:41:26.307 INFO:journalctl@ceph.mon.b.smithi047.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T03:41:26.307 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T03:41:26.307 INFO:journalctl@ceph.mon.b.smithi047.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T03:41:26.307 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T03:41:26.307 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise 2022-05-14T03:41:26.308 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T03:41:26.308 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.gen.throw(type, value, traceback) 2022-05-14T03:41:26.308 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T03:41:26.308 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OrchestratorError(msg) 2022-05-14T03:41:26.308 INFO:journalctl@ceph.mon.b.smithi047.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:41:26.309 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:41:26 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:41:26.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:41:26 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:41:26.359 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T03:41:26.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:41:26.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T03:41:26.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:41:26.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:41:26.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:41:26.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T03:41:26.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:41:26.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T03:41:26.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:41:26.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T03:41:26.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:41:26.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T03:41:26.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:41:26.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T03:41:26.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:41:26.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:41:26.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:41:26.363 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:41:26 smithi115 ceph-mon[78593]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T03:41:26.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T03:41:26.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:41:26.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T03:41:26.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:41:26.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:41:26.365 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:41:26.365 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T03:41:26.365 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:41:26.365 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T03:41:26.365 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:41:26.366 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T03:41:26.366 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:41:26.366 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T03:41:26.366 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:41:26.367 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T03:41:26.367 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:41:26.367 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:41:26.367 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:41:26.367 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T03:41:26.368 INFO:journalctl@ceph.mon.c.smithi115.stdout: During handling of the above exception, another exception occurred: 2022-05-14T03:41:26.368 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T03:41:26.368 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T03:41:26.368 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T03:41:26.368 INFO:journalctl@ceph.mon.c.smithi115.stdout: return f(*arg) 2022-05-14T03:41:26.369 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T03:41:26.369 INFO:journalctl@ceph.mon.c.smithi115.stdout: self._write_client_files(client_files, host) 2022-05-14T03:41:26.369 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T03:41:26.369 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T03:41:26.370 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T03:41:26.370 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T03:41:26.370 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T03:41:26.370 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.event_loop.get_result(coro) 2022-05-14T03:41:26.370 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T03:41:26.371 INFO:journalctl@ceph.mon.c.smithi115.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T03:41:26.371 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T03:41:26.372 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.__get_result() 2022-05-14T03:41:26.372 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T03:41:26.373 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise self._exception 2022-05-14T03:41:26.373 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T03:41:26.373 INFO:journalctl@ceph.mon.c.smithi115.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T03:41:26.373 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T03:41:26.374 INFO:journalctl@ceph.mon.c.smithi115.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T03:41:26.374 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T03:41:26.374 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise 2022-05-14T03:41:26.374 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T03:41:26.374 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.gen.throw(type, value, traceback) 2022-05-14T03:41:26.375 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T03:41:26.375 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OrchestratorError(msg) 2022-05-14T03:41:26.375 INFO:journalctl@ceph.mon.c.smithi115.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:41:26.375 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:41:26 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:41:27.027 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:41:26 smithi115 conmon[90393]: 2022-05-14T03:41:26.722+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:41:27.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:41:27 smithi047 ceph-mon[77147]: pgmap v7961: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:41:27.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:41:27 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:41:27.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:41:27 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 15220 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:41:27.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:41:27 smithi115 ceph-mon[78593]: pgmap v7961: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:41:27.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:41:27 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:41:27.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:41:27 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 15220 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:41:28.032 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:41:27 smithi115 conmon[90393]: 2022-05-14T03:41:27.676+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:41:28.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:41:28 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:41:28.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:41:28 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:41:29.032 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:41:28 smithi115 conmon[90393]: 2022-05-14T03:41:28.682+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:41:29.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:41:29 smithi047 ceph-mon[77147]: pgmap v7962: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:41:29.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:41:29 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:41:29.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:41:29 smithi115 ceph-mon[78593]: pgmap v7962: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:41:29.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:41:29 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:41:30.038 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:41:29 smithi115 conmon[90393]: 2022-05-14T03:41:29.693+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:41:30.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:41:30 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:41:30.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:41:30 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:41:31.038 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:41:30 smithi115 conmon[90393]: 2022-05-14T03:41:30.671+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:41:31.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:41:31 smithi047 ceph-mon[77147]: pgmap v7963: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:41:31.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:41:31 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:41:31.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:41:31 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:41:31.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:41:31 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:41:31.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:41:31 smithi115 ceph-mon[78593]: pgmap v7963: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:41:31.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:41:31 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:41:31.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:41:31 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:41:31.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:41:31 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:41:32.042 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:41:31 smithi115 conmon[90393]: 2022-05-14T03:41:31.704+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:41:32.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:41:32 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:41:32.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:41:32 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 15225 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:41:32.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:41:32 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:41:32.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:41:32 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 15225 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:41:33.041 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:41:32 smithi115 conmon[90393]: 2022-05-14T03:41:32.717+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:41:33.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:41:33 smithi047 ceph-mon[77147]: pgmap v7964: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:41:33.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:41:33 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:41:33.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:41:33 smithi115 ceph-mon[78593]: pgmap v7964: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:41:33.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:41:33 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:41:34.046 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:41:33 smithi115 conmon[90393]: 2022-05-14T03:41:33.745+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:41:34.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:41:34 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:41:34.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:41:34 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:41:35.046 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:41:34 smithi115 conmon[90393]: 2022-05-14T03:41:34.726+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:41:35.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:41:35 smithi115 ceph-mon[78593]: pgmap v7965: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:41:35.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:41:35 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:41:35.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:41:35 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:41:35.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:41:35 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:41:35.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:41:35 smithi047 ceph-mon[77147]: pgmap v7965: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:41:35.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:41:35 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:41:35.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:41:35 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:41:35.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:41:35 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:41:36.051 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:41:35 smithi115 conmon[90393]: 2022-05-14T03:41:35.723+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:41:36.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:41:36 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:41:36.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:41:36 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:41:37.052 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:41:36 smithi115 conmon[90393]: 2022-05-14T03:41:36.760+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:41:37.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:41:37 smithi115 ceph-mon[78593]: pgmap v7966: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:41:37.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:41:37 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 15230 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:41:37.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:41:37 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:41:37.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:41:37 smithi047 ceph-mon[77147]: pgmap v7966: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:41:37.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:41:37 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 15230 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:41:37.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:41:37 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:41:38.057 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:41:37 smithi115 conmon[90393]: 2022-05-14T03:41:37.786+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:41:38.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:41:38 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:41:38.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:41:38 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:41:39.058 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:41:38 smithi115 conmon[90393]: 2022-05-14T03:41:38.752+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:41:39.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:41:39 smithi115 ceph-mon[78593]: pgmap v7967: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:41:39.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:41:39 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:41:39.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:41:39 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:41:39.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:41:39 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:41:39.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:41:39 smithi047 ceph-mon[77147]: pgmap v7967: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:41:39.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:41:39 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:41:39.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:41:39 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:41:39.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:41:39 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:41:40.063 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:41:39 smithi115 conmon[90393]: 2022-05-14T03:41:39.748+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:41:40.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:41:40 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:41:40.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:41:40 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:41:41.064 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:41:40 smithi115 conmon[90393]: 2022-05-14T03:41:40.748+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:41:41.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:41:41 smithi115 ceph-mon[78593]: pgmap v7968: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:41:41.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:41:41 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:41:41.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:41:41 smithi047 ceph-mon[77147]: pgmap v7968: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:41:41.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:41:41 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:41:42.068 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:41:41 smithi115 conmon[90393]: 2022-05-14T03:41:41.780+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:41:42.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:41:42 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:41:42.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:41:42 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:41:42.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:41:42 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 15235 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:41:42.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:41:42 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:41:42.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:41:42 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:41:42.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:41:42 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:41:42.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:41:42 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 15235 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:41:42.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:41:42 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:41:43.071 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:41:42 smithi115 conmon[90393]: 2022-05-14T03:41:42.789+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:41:43.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:41:43 smithi115 ceph-mon[78593]: pgmap v7969: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:41:43.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:41:43 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:41:43.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:41:43 smithi047 ceph-mon[77147]: pgmap v7969: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:41:43.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:41:43 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:41:44.074 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:41:43 smithi115 conmon[90393]: 2022-05-14T03:41:43.774+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:41:44.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:41:44 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:41:44.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:41:44 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:41:45.077 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:41:44 smithi115 conmon[90393]: 2022-05-14T03:41:44.727+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:41:45.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:41:45 smithi115 ceph-mon[78593]: pgmap v7970: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:41:45.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:41:45 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:41:45.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:41:45 smithi047 ceph-mon[77147]: pgmap v7970: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:41:45.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:41:45 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:41:46.080 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:41:45 smithi115 conmon[90393]: 2022-05-14T03:41:45.739+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:41:46.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:41:46 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:41:46.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:41:46 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:41:47.081 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:41:46 smithi115 conmon[90393]: 2022-05-14T03:41:46.723+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:41:47.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:41:47 smithi115 ceph-mon[78593]: pgmap v7971: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:41:47.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:41:47 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:41:47.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:41:47 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 15240 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:41:47.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:41:47 smithi047 ceph-mon[77147]: pgmap v7971: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:41:47.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:41:47 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:41:47.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:41:47 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 15240 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:41:48.086 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:41:47 smithi115 conmon[90393]: 2022-05-14T03:41:47.713+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:41:48.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:41:48 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:41:48.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:41:48 smithi115 ceph-mon[78593]: pgmap v7972: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:41:48.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:41:48 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:41:48.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:41:48 smithi047 ceph-mon[77147]: pgmap v7972: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:41:49.087 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:41:48 smithi115 conmon[90393]: 2022-05-14T03:41:48.692+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:41:49.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:41:49 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:41:49.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:41:49 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:41:50.095 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:41:49 smithi115 conmon[90393]: 2022-05-14T03:41:49.718+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:41:50.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:41:50 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:41:50.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:41:50 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:41:50.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:41:50 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:41:50.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:41:50 smithi115 ceph-mon[78593]: pgmap v7973: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:41:50.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:41:50 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:41:50.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:41:50 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:41:50.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:41:50 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:41:50.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:41:50 smithi047 ceph-mon[77147]: pgmap v7973: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:41:51.096 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:41:50 smithi115 conmon[90393]: 2022-05-14T03:41:50.680+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:41:51.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:41:51 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:41:51.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:41:51 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:41:52.098 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:41:51 smithi115 conmon[90393]: 2022-05-14T03:41:51.697+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:41:52.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:41:52 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:41:52.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:41:52 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 15245 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:41:52.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:41:52 smithi115 ceph-mon[78593]: pgmap v7974: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:41:52.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:41:52 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:41:52.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:41:52 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 15245 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:41:52.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:41:52 smithi047 ceph-mon[77147]: pgmap v7974: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:41:53.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:41:52 smithi115 conmon[90393]: 2022-05-14T03:41:52.668+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:41:54.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:41:53 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:41:54.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:41:53 smithi115 conmon[90393]: 2022-05-14T03:41:53.636+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:41:54.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:41:53 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:41:55.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:41:54 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:41:55.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:41:54 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:41:55.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:41:54 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:41:55.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:41:54 smithi047 ceph-mon[77147]: pgmap v7975: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:41:55.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:41:54 smithi115 conmon[90393]: 2022-05-14T03:41:54.614+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:41:55.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:41:54 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:41:55.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:41:54 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:41:55.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:41:54 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:41:55.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:41:54 smithi115 ceph-mon[78593]: pgmap v7975: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:41:56.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:41:55 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:41:56.110 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:41:55 smithi115 conmon[90393]: 2022-05-14T03:41:55.621+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:41:56.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:41:55 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:41:56.866 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:41:56 smithi115 conmon[90393]: 2022-05-14T03:41:56.600+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:41:56.867 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:41:56 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:41:56.867 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:41:56 smithi115 ceph-mon[78593]: pgmap v7976: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:41:57.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:41:56 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:41:57.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:41:56 smithi047 ceph-mon[77147]: pgmap v7976: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:41:58.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:41:57 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:41:58.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:41:57 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 15250 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:41:58.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:41:57 smithi115 conmon[90393]: 2022-05-14T03:41:57.621+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:41:58.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:41:57 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:41:58.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:41:57 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 15250 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:41:58.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:41:58 smithi115 conmon[90393]: 2022-05-14T03:41:58.574+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:41:58.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:41:58 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:41:58.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:41:58 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:41:58.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:41:58 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:41:58.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:41:58 smithi115 ceph-mon[78593]: pgmap v7977: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:41:59.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:41:58 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:41:59.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:41:58 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:41:59.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:41:58 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:41:59.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:41:58 smithi047 ceph-mon[77147]: pgmap v7977: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:41:59.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:41:59 smithi115 conmon[90393]: 2022-05-14T03:41:59.543+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:41:59.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:41:59 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:42:00.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:41:59 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:42:00.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:42:00 smithi115 conmon[90393]: 2022-05-14T03:42:00.559+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:42:00.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:42:00 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:42:00.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:42:00 smithi115 ceph-mon[78593]: pgmap v7978: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:42:01.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:42:00 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:42:01.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:42:00 smithi047 ceph-mon[77147]: pgmap v7978: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:42:01.780 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:42:01 smithi115 conmon[90393]: 2022-05-14T03:42:01.510+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:42:02.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:42:01 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:42:02.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:42:01 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:42:02.782 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:42:02 smithi115 conmon[90393]: 2022-05-14T03:42:02.491+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:42:03.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:42:02 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:42:03.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:42:02 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:42:03.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:42:02 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:42:03.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:42:02 smithi047 ceph-mon[77147]: pgmap v7979: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:42:03.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:42:02 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:42:03.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:42:02 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:42:03.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:42:02 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:42:03.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:42:02 smithi115 ceph-mon[78593]: pgmap v7979: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:42:03.783 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:42:03 smithi115 conmon[90393]: 2022-05-14T03:42:03.472+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:42:04.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:42:03 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:42:04.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:42:03 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:42:04.788 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:42:04 smithi115 conmon[90393]: 2022-05-14T03:42:04.499+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:42:05.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:42:04 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:42:05.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:42:04 smithi047 ceph-mon[77147]: pgmap v7980: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:42:05.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:42:04 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:42:05.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:42:04 smithi115 ceph-mon[78593]: pgmap v7980: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:42:05.790 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:42:05 smithi115 conmon[90393]: 2022-05-14T03:42:05.498+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:42:06.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:42:05 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:42:06.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:42:05 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:42:06.794 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:42:06 smithi115 conmon[90393]: 2022-05-14T03:42:06.513+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:42:07.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:42:06 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:42:07.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:42:06 smithi115 ceph-mon[78593]: pgmap v7981: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:42:07.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:42:06 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 15255 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:42:07.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:42:06 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:42:07.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:42:06 smithi047 ceph-mon[77147]: pgmap v7981: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:42:07.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:42:06 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 15255 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:42:07.795 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:42:07 smithi115 conmon[90393]: 2022-05-14T03:42:07.495+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:42:08.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:42:07 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:42:08.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:42:07 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:42:08.800 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:42:08 smithi115 conmon[90393]: 2022-05-14T03:42:08.477+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:42:09.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:42:08 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:42:09.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:42:08 smithi115 ceph-mon[78593]: pgmap v7982: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:42:09.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:42:08 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:42:09.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:42:08 smithi047 ceph-mon[77147]: pgmap v7982: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:42:09.802 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:42:09 smithi115 conmon[90393]: 2022-05-14T03:42:09.503+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:42:10.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:42:09 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:42:10.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:42:09 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:42:10.810 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:42:10 smithi115 conmon[90393]: 2022-05-14T03:42:10.532+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:42:11.114 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:42:10 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:42:11.115 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:42:10 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:42:11.116 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:42:10 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:42:11.116 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:42:10 smithi115 ceph-mon[78593]: pgmap v7983: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:42:11.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:42:10 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:42:11.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:42:10 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:42:11.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:42:10 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:42:11.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:42:10 smithi047 ceph-mon[77147]: pgmap v7983: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:42:11.813 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:42:11 smithi115 conmon[90393]: 2022-05-14T03:42:11.545+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:42:12.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:42:11 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:42:12.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:42:11 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 15265 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:42:12.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:42:11 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:42:12.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:42:11 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 15265 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:42:12.822 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:42:12 smithi115 conmon[90393]: 2022-05-14T03:42:12.549+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:42:13.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:42:12 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:42:13.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:42:12 smithi115 ceph-mon[78593]: pgmap v7984: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:42:13.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:42:12 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:42:13.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:42:12 smithi047 ceph-mon[77147]: pgmap v7984: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:42:13.824 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:42:13 smithi115 conmon[90393]: 2022-05-14T03:42:13.568+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:42:14.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:42:13 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:42:14.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:42:13 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:42:14.832 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:42:14 smithi115 conmon[90393]: 2022-05-14T03:42:14.533+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:42:15.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:42:14 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:42:15.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:42:14 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:42:15.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:42:14 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:42:15.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:42:14 smithi115 ceph-mon[78593]: pgmap v7985: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:42:15.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:42:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T03:42:15.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:42:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T03:42:15.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:42:14 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:42:15.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:42:14 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:42:15.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:42:14 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:42:15.295 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:42:14 smithi047 ceph-mon[77147]: pgmap v7985: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:42:15.295 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:42:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T03:42:15.295 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:42:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T03:42:15.830 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:42:15 smithi115 conmon[90393]: 2022-05-14T03:42:15.526+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:42:16.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:42:15 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:42:16.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:42:15 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:42:16.837 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:42:16 smithi115 conmon[90393]: 2022-05-14T03:42:16.533+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:42:17.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:42:16 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:42:17.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:42:16 smithi115 ceph-mon[78593]: pgmap v7986: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:42:17.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:42:16 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:42:17.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:42:16 smithi047 ceph-mon[77147]: pgmap v7986: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:42:17.837 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:42:17 smithi115 conmon[90393]: 2022-05-14T03:42:17.575+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:42:18.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:42:17 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:42:18.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:42:17 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 15270 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:42:18.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:42:17 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:42:18.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:42:17 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 15270 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:42:18.847 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:42:18 smithi115 conmon[90393]: 2022-05-14T03:42:18.528+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:42:19.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:42:18 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:42:19.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:42:18 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:42:19.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:42:18 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:42:19.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:42:18 smithi115 ceph-mon[78593]: pgmap v7987: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:42:19.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:42:18 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:42:19.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:42:18 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:42:19.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:42:18 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:42:19.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:42:18 smithi047 ceph-mon[77147]: pgmap v7987: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:42:19.853 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:42:19 smithi115 conmon[90393]: 2022-05-14T03:42:19.485+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:42:20.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:42:19 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:42:20.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:42:19 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:42:20.854 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:42:20 smithi115 conmon[90393]: 2022-05-14T03:42:20.475+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:42:21.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:42:20 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:42:21.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:42:20 smithi115 ceph-mon[78593]: pgmap v7988: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:42:21.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:42:20 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:42:21.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:42:20 smithi047 ceph-mon[77147]: pgmap v7988: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:42:21.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:42:21 smithi115 conmon[90393]: 2022-05-14T03:42:21.467+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:42:22.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:42:21 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:42:22.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:42:21 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:42:22.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:42:22 smithi115 conmon[90393]: 2022-05-14T03:42:22.473+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:42:23.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:42:22 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:42:23.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:42:22 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:42:23.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:42:22 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:42:23.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:42:22 smithi047 ceph-mon[77147]: pgmap v7989: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:42:23.363 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:42:22 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:42:23.363 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:42:22 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:42:23.364 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:42:22 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:42:23.364 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:42:22 smithi115 ceph-mon[78593]: pgmap v7989: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:42:23.863 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:42:23 smithi115 conmon[90393]: 2022-05-14T03:42:23.443+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:42:24.258 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:42:23 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:42:24.261 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:42:23 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:42:24.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:42:24 smithi115 conmon[90393]: 2022-05-14T03:42:24.416+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:42:25.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:42:24 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:42:25.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:42:24 smithi047 ceph-mon[77147]: pgmap v7990: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:42:25.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:42:24 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T03:42:25.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:42:24 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T03:42:25.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:42:24 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T03:42:25.112 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:42:24 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:42:25.112 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:42:24 smithi115 ceph-mon[78593]: pgmap v7990: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:42:25.112 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:42:24 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T03:42:25.113 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:42:24 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T03:42:25.113 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:42:24 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T03:42:25.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:42:25 smithi115 conmon[90393]: 2022-05-14T03:42:25.387+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:42:26.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:42:25 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:42:26.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:42:25 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:42:26.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:42:26 smithi115 conmon[90393]: 2022-05-14T03:42:26.389+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:42:27.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:42:26 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:42:27.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:42:26 smithi047 ceph-mon[77147]: pgmap v7991: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:42:27.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:42:26 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 15275 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:42:27.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:42:26 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:42:27.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:42:26 smithi115 ceph-mon[78593]: pgmap v7991: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:42:27.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:42:26 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 15275 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:42:27.793 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:42:27 smithi047 conmon[78521]: 2022-05-14T03:42:27.312+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:42:27.793 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:42:27 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T03:42:27.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:42:27 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:42:27.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:42:27 smithi047 conmon[78521]: yield 2022-05-14T03:42:27.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:42:27 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:42:27.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:42:27 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:42:27.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:42:27 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:42:27.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:42:27 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T03:42:27.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:42:27 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:42:27.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:42:27 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T03:42:27.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:42:27 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:42:27.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:42:27 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T03:42:27.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:42:27 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:42:27.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:42:27 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T03:42:27.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:42:27 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:42:27.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:42:27 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T03:42:27.797 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:42:27 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:42:27.797 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:42:27 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:42:27.798 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:42:27 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:42:27.860 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:42:27 smithi115 conmon[90393]: 2022-05-14T03:42:27.437+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:42:28.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:42:27 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:42:28.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:42:27 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:42:28.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:42:28 smithi115 conmon[90393]: 2022-05-14T03:42:28.462+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:42:29.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:42:28 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:42:29.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T03:42:29.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:42:29.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T03:42:29.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:42:29.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:42:29.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:42:29.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T03:42:29.298 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:42:29.298 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T03:42:29.299 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:42:29.299 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T03:42:29.299 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:42:29.299 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T03:42:29.299 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:42:29.300 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T03:42:29.300 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:42:29.301 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:42:29.301 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:42:29.301 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:42:28 smithi047 ceph-mon[77147]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T03:42:29.301 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:42:28 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:42:29.302 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:42:28 smithi047 ceph-mon[77147]: pgmap v7992: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:42:29.362 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:42:28 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:42:29.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T03:42:29.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:42:29.365 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T03:42:29.365 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:42:29.365 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:42:29.365 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:42:29.366 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T03:42:29.366 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:42:29.366 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T03:42:29.366 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:42:29.367 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T03:42:29.367 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:42:29.367 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T03:42:29.367 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:42:29.367 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T03:42:29.368 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:42:29.368 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:42:29.368 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:42:29.368 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:42:28 smithi115 ceph-mon[78593]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T03:42:29.369 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:42:28 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:42:29.369 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:42:28 smithi115 ceph-mon[78593]: pgmap v7992: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:42:29.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:42:29 smithi115 conmon[90393]: 2022-05-14T03:42:29.477+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:42:30.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:42:29 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:42:30.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:42:29 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:42:30.793 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:42:30 smithi047 conmon[78521]: 2022-05-14T03:42:30.384+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:42:30.793 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:42:30 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T03:42:30.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:42:30 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:42:30.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:42:30 smithi047 conmon[78521]: yield 2022-05-14T03:42:30.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:42:30 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:42:30.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:42:30 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:42:30.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:42:30 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:42:30.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:42:30 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T03:42:30.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:42:30 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:42:30.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:42:30 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T03:42:30.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:42:30 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:42:30.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:42:30 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T03:42:30.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:42:30 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:42:30.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:42:30 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T03:42:30.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:42:30 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:42:30.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:42:30 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T03:42:30.797 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:42:30 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:42:30.797 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:42:30 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:42:30.799 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:42:30 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:42:30.799 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:42:30 smithi047 conmon[78521]: 2022-05-14T03:42:30.385+0000 7f5344f82700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T03:42:30.800 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:42:30 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T03:42:30.805 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:42:30 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:42:30.806 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:42:30 smithi047 conmon[78521]: yield 2022-05-14T03:42:30.806 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:42:30 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:42:30.806 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:42:30 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:42:30.807 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:42:30 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:42:30.807 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:42:30 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T03:42:30.807 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:42:30 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:42:30.807 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:42:30 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T03:42:30.808 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:42:30 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:42:30.808 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:42:30 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T03:42:30.808 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:42:30 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:42:30.808 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:42:30 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T03:42:30.809 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:42:30 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:42:30.809 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:42:30 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T03:42:30.812 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:42:30 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:42:30.813 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:42:30 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:42:30.813 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:42:30 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:42:30.813 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:42:30 smithi047 conmon[78521]: 2022-05-14T03:42:30.813 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:42:30 smithi047 conmon[78521]: During handling of the above exception, another exception occurred: 2022-05-14T03:42:30.814 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:42:30 smithi047 conmon[78521]: 2022-05-14T03:42:30.814 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:42:30 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T03:42:30.814 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:42:30 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T03:42:30.814 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:42:30 smithi047 conmon[78521]: return f(*arg) 2022-05-14T03:42:30.814 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:42:30 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T03:42:30.815 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:42:30 smithi047 conmon[78521]: self._write_client_files(client_files, host) 2022-05-14T03:42:30.815 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:42:30 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T03:42:30.815 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:42:30 smithi047 conmon[78521]: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T03:42:30.815 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:42:30 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T03:42:30.816 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:42:30 smithi047 conmon[78521]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T03:42:30.816 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:42:30 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T03:42:30.816 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:42:30 smithi047 conmon[78521]: return self.event_loop.get_result(coro) 2022-05-14T03:42:30.816 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:42:30 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T03:42:30.816 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:42:30 smithi047 conmon[78521]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T03:42:30.817 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:42:30 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T03:42:30.817 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:42:30 smithi047 conmon[78521]: return self.__get_result() 2022-05-14T03:42:30.817 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:42:30 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T03:42:30.817 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:42:30 smithi047 conmon[78521]: raise self._exception 2022-05-14T03:42:30.818 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:42:30 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T03:42:30.818 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:42:30 smithi047 conmon[78521]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T03:42:30.818 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:42:30 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T03:42:30.819 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:42:30 smithi047 conmon[78521]: conn = await self._remote_connection(host, addr) 2022-05-14T03:42:30.820 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:42:30 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T03:42:30.820 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:42:30 smithi047 conmon[78521]: raise 2022-05-14T03:42:30.821 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:42:30 smithi047 conmon[78521]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T03:42:30.821 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:42:30 smithi047 conmon[78521]: self.gen.throw(type, value, traceback) 2022-05-14T03:42:30.822 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:42:30 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T03:42:30.822 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:42:30 smithi047 conmon[78521]: raise OrchestratorError(msg) 2022-05-14T03:42:30.822 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:42:30 smithi047 conmon[78521]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:42:30.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:42:30 smithi115 conmon[90393]: 2022-05-14T03:42:30.486+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:42:31.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:42:30 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:42:31.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:42:30 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:42:31.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:42:30 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:42:31.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:42:30 smithi047 ceph-mon[77147]: pgmap v7993: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:42:31.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:42:30 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:42:31.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:42:30 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:42:31.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:42:30 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:42:31.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:42:30 smithi115 ceph-mon[78593]: pgmap v7993: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:42:31.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:42:31 smithi115 conmon[90393]: 2022-05-14T03:42:31.458+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:42:32.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:42:31 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:42:32.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T03:42:32.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:42:32.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T03:42:32.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:42:32.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:42:32.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:42:32.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T03:42:32.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:42:32.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T03:42:32.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:42:32.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T03:42:32.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:42:32.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T03:42:32.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:42:32.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T03:42:32.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:42:32.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:42:32.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:42:32.298 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:42:31 smithi047 ceph-mon[77147]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T03:42:32.298 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T03:42:32.298 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:42:32.298 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T03:42:32.298 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:42:32.299 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:42:32.299 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:42:32.299 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T03:42:32.299 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:42:32.300 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T03:42:32.300 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:42:32.300 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T03:42:32.300 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:42:32.301 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T03:42:32.301 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:42:32.301 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T03:42:32.301 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:42:32.301 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:42:32.302 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:42:32.302 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T03:42:32.302 INFO:journalctl@ceph.mon.b.smithi047.stdout: During handling of the above exception, another exception occurred: 2022-05-14T03:42:32.303 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T03:42:32.303 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T03:42:32.303 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T03:42:32.303 INFO:journalctl@ceph.mon.b.smithi047.stdout: return f(*arg) 2022-05-14T03:42:32.303 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T03:42:32.304 INFO:journalctl@ceph.mon.b.smithi047.stdout: self._write_client_files(client_files, host) 2022-05-14T03:42:32.304 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T03:42:32.304 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T03:42:32.304 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T03:42:32.305 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T03:42:32.305 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T03:42:32.305 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.event_loop.get_result(coro) 2022-05-14T03:42:32.305 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T03:42:32.305 INFO:journalctl@ceph.mon.b.smithi047.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T03:42:32.306 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T03:42:32.306 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.__get_result() 2022-05-14T03:42:32.306 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T03:42:32.306 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise self._exception 2022-05-14T03:42:32.307 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T03:42:32.307 INFO:journalctl@ceph.mon.b.smithi047.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T03:42:32.307 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T03:42:32.307 INFO:journalctl@ceph.mon.b.smithi047.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T03:42:32.309 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T03:42:32.309 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise 2022-05-14T03:42:32.309 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T03:42:32.309 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.gen.throw(type, value, traceback) 2022-05-14T03:42:32.309 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T03:42:32.310 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OrchestratorError(msg) 2022-05-14T03:42:32.310 INFO:journalctl@ceph.mon.b.smithi047.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:42:32.310 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:42:31 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:42:32.310 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:42:31 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 15285 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:42:32.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:42:31 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:42:32.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T03:42:32.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:42:32.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T03:42:32.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:42:32.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:42:32.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:42:32.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T03:42:32.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:42:32.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T03:42:32.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:42:32.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T03:42:32.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:42:32.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T03:42:32.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:42:32.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T03:42:32.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:42:32.365 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:42:32.365 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:42:32.365 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:42:31 smithi115 ceph-mon[78593]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T03:42:32.365 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T03:42:32.365 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:42:32.366 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T03:42:32.366 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:42:32.366 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:42:32.366 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:42:32.367 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T03:42:32.367 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:42:32.367 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T03:42:32.367 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:42:32.367 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T03:42:32.368 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:42:32.368 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T03:42:32.369 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:42:32.369 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T03:42:32.369 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:42:32.369 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:42:32.370 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:42:32.370 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T03:42:32.370 INFO:journalctl@ceph.mon.c.smithi115.stdout: During handling of the above exception, another exception occurred: 2022-05-14T03:42:32.370 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T03:42:32.370 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T03:42:32.371 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T03:42:32.371 INFO:journalctl@ceph.mon.c.smithi115.stdout: return f(*arg) 2022-05-14T03:42:32.371 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T03:42:32.371 INFO:journalctl@ceph.mon.c.smithi115.stdout: self._write_client_files(client_files, host) 2022-05-14T03:42:32.372 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T03:42:32.372 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T03:42:32.372 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T03:42:32.372 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T03:42:32.372 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T03:42:32.373 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.event_loop.get_result(coro) 2022-05-14T03:42:32.373 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T03:42:32.374 INFO:journalctl@ceph.mon.c.smithi115.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T03:42:32.375 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T03:42:32.375 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.__get_result() 2022-05-14T03:42:32.375 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T03:42:32.375 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise self._exception 2022-05-14T03:42:32.375 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T03:42:32.376 INFO:journalctl@ceph.mon.c.smithi115.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T03:42:32.376 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T03:42:32.376 INFO:journalctl@ceph.mon.c.smithi115.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T03:42:32.376 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T03:42:32.376 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise 2022-05-14T03:42:32.377 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T03:42:32.377 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.gen.throw(type, value, traceback) 2022-05-14T03:42:32.377 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T03:42:32.377 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OrchestratorError(msg) 2022-05-14T03:42:32.378 INFO:journalctl@ceph.mon.c.smithi115.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:42:32.378 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:42:31 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:42:32.378 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:42:31 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 15285 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:42:32.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:42:32 smithi115 conmon[90393]: 2022-05-14T03:42:32.446+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:42:33.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:42:32 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:42:33.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:42:32 smithi047 ceph-mon[77147]: pgmap v7994: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:42:33.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:42:32 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:42:33.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:42:32 smithi115 ceph-mon[78593]: pgmap v7994: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:42:33.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:42:33 smithi115 conmon[90393]: 2022-05-14T03:42:33.432+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:42:34.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:42:33 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:42:34.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:42:33 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:42:34.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:42:34 smithi115 conmon[90393]: 2022-05-14T03:42:34.435+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:42:35.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:42:34 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:42:35.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:42:34 smithi047 ceph-mon[77147]: pgmap v7995: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:42:35.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:42:34 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:42:35.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:42:34 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:42:35.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:42:34 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:42:35.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:42:34 smithi115 ceph-mon[78593]: pgmap v7995: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:42:35.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:42:34 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:42:35.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:42:34 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:42:35.860 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:42:35 smithi115 conmon[90393]: 2022-05-14T03:42:35.456+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:42:36.298 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:42:35 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:42:36.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:42:35 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:42:36.864 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:42:36 smithi115 conmon[90393]: 2022-05-14T03:42:36.462+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:42:37.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:42:36 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:42:37.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:42:36 smithi047 ceph-mon[77147]: pgmap v7996: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:42:37.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:42:36 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 15290 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:42:37.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:42:36 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:42:37.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:42:36 smithi115 ceph-mon[78593]: pgmap v7996: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:42:37.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:42:36 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 15290 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:42:37.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:42:37 smithi115 conmon[90393]: 2022-05-14T03:42:37.425+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:42:38.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:42:37 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:42:38.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:42:37 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:42:38.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:42:38 smithi115 conmon[90393]: 2022-05-14T03:42:38.470+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:42:39.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:42:38 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:42:39.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:42:38 smithi047 ceph-mon[77147]: pgmap v7997: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:42:39.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:42:38 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:42:39.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:42:38 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:42:39.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:42:38 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:42:39.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:42:38 smithi115 ceph-mon[78593]: pgmap v7997: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:42:39.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:42:38 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:42:39.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:42:38 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:42:39.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:42:39 smithi115 conmon[90393]: 2022-05-14T03:42:39.502+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:42:40.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:42:39 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:42:40.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:42:39 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:42:40.860 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:42:40 smithi115 conmon[90393]: 2022-05-14T03:42:40.465+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:42:41.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:42:40 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:42:41.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:42:40 smithi047 ceph-mon[77147]: pgmap v7998: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:42:41.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:42:40 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:42:41.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:42:40 smithi115 ceph-mon[78593]: pgmap v7998: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:42:41.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:42:41 smithi115 conmon[90393]: 2022-05-14T03:42:41.508+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:42:42.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:42:41 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:42:42.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:42:41 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 15295 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:42:42.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:42:41 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:42:42.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:42:41 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 15295 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:42:42.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:42:42 smithi115 conmon[90393]: 2022-05-14T03:42:42.470+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:42:43.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:42:42 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:42:43.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:42:42 smithi047 ceph-mon[77147]: pgmap v7999: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:42:43.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:42:42 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:42:43.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:42:42 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:42:43.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:42:42 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:42:43.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:42:42 smithi115 ceph-mon[78593]: pgmap v7999: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:42:43.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:42:42 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:42:43.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:42:42 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:42:43.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:42:43 smithi115 conmon[90393]: 2022-05-14T03:42:43.443+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:42:44.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:42:43 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:42:44.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:42:43 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:42:44.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:42:44 smithi115 conmon[90393]: 2022-05-14T03:42:44.467+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:42:45.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:42:44 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:42:45.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:42:44 smithi047 ceph-mon[77147]: pgmap v8000: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:42:45.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:42:44 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:42:45.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:42:44 smithi115 ceph-mon[78593]: pgmap v8000: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:42:45.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:42:45 smithi115 conmon[90393]: 2022-05-14T03:42:45.511+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:42:46.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:42:45 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:42:46.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:42:45 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:42:46.860 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:42:46 smithi115 conmon[90393]: 2022-05-14T03:42:46.552+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:42:47.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:42:46 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:42:47.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:42:46 smithi047 ceph-mon[77147]: pgmap v8001: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:42:47.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:42:46 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 15300 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:42:47.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:42:46 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:42:47.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:42:46 smithi115 ceph-mon[78593]: pgmap v8001: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:42:47.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:42:46 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 15300 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:42:47.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:42:47 smithi115 conmon[90393]: 2022-05-14T03:42:47.558+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:42:48.299 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:42:47 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:42:48.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:42:47 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:42:48.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:42:48 smithi115 conmon[90393]: 2022-05-14T03:42:48.535+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:42:49.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:42:48 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:42:49.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:42:48 smithi047 ceph-mon[77147]: pgmap v8002: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:42:49.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:42:48 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:42:49.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:42:48 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:42:49.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:42:48 smithi115 ceph-mon[78593]: pgmap v8002: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:42:49.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:42:48 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:42:49.866 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:42:49 smithi115 conmon[90393]: 2022-05-14T03:42:49.496+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:42:50.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:42:49 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:42:50.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:42:49 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:42:50.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:42:50 smithi115 conmon[90393]: 2022-05-14T03:42:50.510+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:42:51.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:42:50 smithi047 ceph-mon[77147]: pgmap v8003: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:42:51.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:42:50 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:42:51.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:42:50 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:42:51.295 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:42:50 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:42:51.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:42:50 smithi115 ceph-mon[78593]: pgmap v8003: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:42:51.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:42:50 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:42:51.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:42:50 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:42:51.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:42:50 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:42:51.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:42:51 smithi115 conmon[90393]: 2022-05-14T03:42:51.471+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:42:52.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:42:51 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:42:52.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:42:51 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 15305 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:42:52.382 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:42:51 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:42:52.383 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:42:51 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 15305 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:42:52.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:42:52 smithi115 conmon[90393]: 2022-05-14T03:42:52.422+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:42:53.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:42:52 smithi047 ceph-mon[77147]: pgmap v8004: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:42:53.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:42:52 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:42:53.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:42:52 smithi115 ceph-mon[78593]: pgmap v8004: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:42:53.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:42:52 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:42:53.860 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:42:53 smithi115 conmon[90393]: 2022-05-14T03:42:53.453+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:42:54.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:42:53 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:42:54.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:42:53 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:42:54.862 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:42:54 smithi115 conmon[90393]: 2022-05-14T03:42:54.503+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:42:55.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:42:54 smithi047 ceph-mon[77147]: pgmap v8005: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:42:55.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:42:54 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:42:55.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:42:54 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:42:55.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:42:54 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:42:55.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:42:54 smithi115 ceph-mon[78593]: pgmap v8005: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:42:55.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:42:54 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:42:55.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:42:54 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:42:55.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:42:54 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:42:55.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:42:55 smithi115 conmon[90393]: 2022-05-14T03:42:55.477+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:42:56.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:42:55 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:42:56.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:42:55 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:42:56.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:42:56 smithi115 conmon[90393]: 2022-05-14T03:42:56.459+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:42:57.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:42:56 smithi047 ceph-mon[77147]: pgmap v8006: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:42:57.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:42:56 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:42:57.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:42:56 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 15310 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:42:57.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:42:56 smithi115 ceph-mon[78593]: pgmap v8006: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:42:57.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:42:56 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:42:57.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:42:56 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 15310 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:42:57.862 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:42:57 smithi115 conmon[90393]: 2022-05-14T03:42:57.475+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:42:58.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:42:57 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:42:58.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:42:57 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:42:58.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:42:58 smithi115 conmon[90393]: 2022-05-14T03:42:58.444+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:42:59.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:42:58 smithi047 ceph-mon[77147]: pgmap v8007: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:42:59.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:42:58 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:42:59.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:42:58 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:42:59.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:42:58 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:42:59.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:42:58 smithi115 ceph-mon[78593]: pgmap v8007: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:42:59.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:42:58 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:42:59.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:42:58 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:42:59.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:42:58 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:42:59.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:42:59 smithi115 conmon[90393]: 2022-05-14T03:42:59.420+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:43:00.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:42:59 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:43:00.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:42:59 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:43:00.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:43:00 smithi115 conmon[90393]: 2022-05-14T03:43:00.418+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:43:01.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:43:00 smithi047 ceph-mon[77147]: pgmap v8008: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:43:01.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:43:00 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:43:01.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:43:00 smithi115 ceph-mon[78593]: pgmap v8008: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:43:01.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:43:00 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:43:01.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:43:01 smithi115 conmon[90393]: 2022-05-14T03:43:01.454+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:43:02.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:43:01 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:43:02.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:43:01 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 15315 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:43:02.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:43:01 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:43:02.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:43:01 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 15315 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:43:02.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:43:02 smithi115 conmon[90393]: 2022-05-14T03:43:02.424+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:43:03.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:43:03 smithi047 ceph-mon[77147]: pgmap v8009: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:43:03.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:43:03 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:43:03.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:43:03 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:43:03.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:43:03 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:43:03.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:43:03 smithi115 ceph-mon[78593]: pgmap v8009: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:43:03.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:43:03 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:43:03.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:43:03 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:43:03.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:43:03 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:43:03.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:43:03 smithi115 conmon[90393]: 2022-05-14T03:43:03.438+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:43:04.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:43:04 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:43:04.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:43:04 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:43:04.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:43:04 smithi115 conmon[90393]: 2022-05-14T03:43:04.467+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:43:05.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:43:05 smithi047 ceph-mon[77147]: pgmap v8010: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:43:05.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:43:05 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:43:05.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:43:05 smithi115 ceph-mon[78593]: pgmap v8010: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:43:05.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:43:05 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:43:05.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:43:05 smithi115 conmon[90393]: 2022-05-14T03:43:05.466+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:43:06.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:43:06 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:43:06.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:43:06 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:43:06.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:43:06 smithi115 conmon[90393]: 2022-05-14T03:43:06.512+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:43:07.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:43:07 smithi047 ceph-mon[77147]: pgmap v8011: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:43:07.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:43:07 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:43:07.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:43:07 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 15320 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:43:07.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:43:07 smithi115 ceph-mon[78593]: pgmap v8011: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:43:07.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:43:07 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:43:07.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:43:07 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 15320 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:43:07.860 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:43:07 smithi115 conmon[90393]: 2022-05-14T03:43:07.472+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:43:08.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:43:08 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:43:08.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:43:08 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:43:08.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:43:08 smithi115 conmon[90393]: 2022-05-14T03:43:08.484+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:43:09.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:43:09 smithi047 ceph-mon[77147]: pgmap v8012: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:43:09.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:43:09 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:43:09.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:43:09 smithi115 ceph-mon[78593]: pgmap v8012: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:43:09.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:43:09 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:43:09.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:43:09 smithi115 conmon[90393]: 2022-05-14T03:43:09.500+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:43:10.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:43:10 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:43:10.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:43:10 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:43:10.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:43:10 smithi115 conmon[90393]: 2022-05-14T03:43:10.481+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:43:11.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:43:11 smithi047 ceph-mon[77147]: pgmap v8013: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:43:11.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:43:11 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:43:11.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:43:11 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:43:11.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:43:11 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:43:11.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:43:11 smithi115 ceph-mon[78593]: pgmap v8013: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:43:11.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:43:11 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:43:11.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:43:11 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:43:11.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:43:11 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:43:11.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:43:11 smithi115 conmon[90393]: 2022-05-14T03:43:11.476+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:43:12.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:43:12 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:43:12.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:43:12 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 15325 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:43:12.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:43:12 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:43:12.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:43:12 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 15325 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:43:12.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:43:12 smithi115 conmon[90393]: 2022-05-14T03:43:12.427+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:43:13.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:43:13 smithi047 ceph-mon[77147]: pgmap v8014: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:43:13.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:43:13 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:43:13.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:43:13 smithi115 ceph-mon[78593]: pgmap v8014: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:43:13.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:43:13 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:43:13.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:43:13 smithi115 conmon[90393]: 2022-05-14T03:43:13.430+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:43:14.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:43:14 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:43:14.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:43:14 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:43:14.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:43:14 smithi115 conmon[90393]: 2022-05-14T03:43:14.396+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:43:15.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:43:15 smithi047 ceph-mon[77147]: pgmap v8015: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:43:15.295 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:43:15 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:43:15.295 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:43:15 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:43:15.295 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:43:15 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:43:15.296 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:43:15 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T03:43:15.296 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:43:15 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T03:43:15.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:43:15 smithi115 ceph-mon[78593]: pgmap v8015: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:43:15.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:43:15 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:43:15.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:43:15 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:43:15.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:43:15 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:43:15.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:43:15 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T03:43:15.362 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:43:15 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T03:43:15.860 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:43:15 smithi115 conmon[90393]: 2022-05-14T03:43:15.408+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:43:16.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:43:16 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:43:16.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:43:16 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:43:16.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:43:16 smithi115 conmon[90393]: 2022-05-14T03:43:16.363+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:43:17.321 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:43:17 smithi115 ceph-mon[78593]: pgmap v8016: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:43:17.321 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:43:17 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:43:17.321 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:43:17 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 15330 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:43:17.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:43:17 smithi047 ceph-mon[77147]: pgmap v8016: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:43:17.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:43:17 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:43:17.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:43:17 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 15330 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:43:17.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:43:17 smithi115 conmon[90393]: 2022-05-14T03:43:17.317+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:43:18.322 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:43:18 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:43:18.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:43:18 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:43:18.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:43:18 smithi115 conmon[90393]: 2022-05-14T03:43:18.319+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:43:19.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:43:19 smithi115 conmon[90393]: 2022-05-14T03:43:19.292+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:43:19.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:43:19 smithi115 ceph-mon[78593]: pgmap v8017: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:43:19.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:43:19 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:43:19.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:43:19 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:43:19.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:43:19 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:43:19.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:43:19 smithi047 ceph-mon[77147]: pgmap v8017: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:43:19.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:43:19 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:43:19.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:43:19 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:43:19.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:43:19 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:43:20.360 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:43:20 smithi115 conmon[90393]: 2022-05-14T03:43:20.258+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:43:20.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:43:20 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:43:20.547 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:43:20 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:43:21.361 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:43:21 smithi115 conmon[90393]: 2022-05-14T03:43:21.218+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:43:21.362 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:43:21 smithi115 ceph-mon[78593]: pgmap v8018: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:43:21.362 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:43:21 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:43:21.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:43:21 smithi047 ceph-mon[77147]: pgmap v8018: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:43:21.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:43:21 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:43:22.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:43:22 smithi115 conmon[90393]: 2022-05-14T03:43:22.228+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:43:22.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:43:22 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:43:22.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:43:22 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 15335 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:43:22.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:43:22 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:43:22.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:43:22 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 15335 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:43:23.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:43:23 smithi115 conmon[90393]: 2022-05-14T03:43:23.247+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:43:23.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:43:23 smithi115 ceph-mon[78593]: pgmap v8019: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:43:23.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:43:23 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:43:23.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:43:23 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:43:23.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:43:23 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:43:23.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:43:23 smithi047 ceph-mon[77147]: pgmap v8019: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:43:23.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:43:23 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:43:23.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:43:23 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:43:23.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:43:23 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:43:24.360 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:43:24 smithi115 conmon[90393]: 2022-05-14T03:43:24.240+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:43:24.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:43:24 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:43:24.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:43:24 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:43:25.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:43:25 smithi115 conmon[90393]: 2022-05-14T03:43:25.234+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:43:25.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:43:25 smithi115 ceph-mon[78593]: pgmap v8020: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:43:25.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:43:25 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:43:25.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:43:25 smithi047 ceph-mon[77147]: pgmap v8020: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:43:25.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:43:25 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:43:26.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:43:26 smithi115 conmon[90393]: 2022-05-14T03:43:26.276+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:43:26.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:43:26 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:43:26.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:43:26 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:43:27.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:43:27 smithi115 conmon[90393]: 2022-05-14T03:43:27.309+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:43:27.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:43:27 smithi115 ceph-mon[78593]: pgmap v8021: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 7 op/s 2022-05-14T03:43:27.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:43:27 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:43:27.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:43:27 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 15340 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:43:27.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:43:27 smithi047 ceph-mon[77147]: pgmap v8021: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 7 op/s 2022-05-14T03:43:27.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:43:27 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:43:27.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:43:27 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 15340 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:43:28.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:43:28 smithi115 conmon[90393]: 2022-05-14T03:43:28.294+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:43:28.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:43:28 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:43:28.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:43:28 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:43:29.360 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:43:29 smithi115 conmon[90393]: 2022-05-14T03:43:29.306+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:43:29.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:43:29 smithi115 ceph-mon[78593]: pgmap v8022: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 4 op/s 2022-05-14T03:43:29.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:43:29 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:43:29.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:43:29 smithi047 ceph-mon[77147]: pgmap v8022: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 4 op/s 2022-05-14T03:43:29.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:43:29 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:43:30.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:43:30 smithi115 conmon[90393]: 2022-05-14T03:43:30.311+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:43:30.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:43:30 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:43:30.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:43:30 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:43:30.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:43:30 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:43:30.433 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:43:30 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:43:30.433 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:43:30 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:43:30.434 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:43:30 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:43:31.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:43:31 smithi047 ceph-mon[77147]: pgmap v8023: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 7 op/s 2022-05-14T03:43:31.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:43:31 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:43:31.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:43:31 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T03:43:31.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:43:31 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T03:43:31.295 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:43:31 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T03:43:31.325 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:43:31 smithi115 ceph-mon[78593]: pgmap v8023: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 7 op/s 2022-05-14T03:43:31.326 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:43:31 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:43:31.326 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:43:31 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T03:43:31.326 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:43:31 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T03:43:31.327 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:43:31 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T03:43:31.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:43:31 smithi115 conmon[90393]: 2022-05-14T03:43:31.323+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:43:32.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:43:32 smithi115 conmon[90393]: 2022-05-14T03:43:32.320+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:43:32.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:43:32 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:43:32.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:43:32 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 15345 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:43:32.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:43:32 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:43:32.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:43:32 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 15345 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:43:33.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:43:33 smithi115 conmon[90393]: 2022-05-14T03:43:33.325+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:43:33.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:43:33 smithi115 ceph-mon[78593]: pgmap v8024: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 4 op/s 2022-05-14T03:43:33.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:43:33 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:43:33.491 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:43:33 smithi047 ceph-mon[77147]: pgmap v8024: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 4 op/s 2022-05-14T03:43:33.491 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:43:33 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:43:33.793 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:43:33 smithi047 conmon[78521]: 2022-05-14T03:43:33.487+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:43:33.793 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:43:33 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T03:43:33.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:43:33 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:43:33.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:43:33 smithi047 conmon[78521]: yield 2022-05-14T03:43:33.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:43:33 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:43:33.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:43:33 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:43:33.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:43:33 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:43:33.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:43:33 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T03:43:33.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:43:33 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:43:33.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:43:33 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T03:43:33.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:43:33 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:43:33.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:43:33 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T03:43:33.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:43:33 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:43:33.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:43:33 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T03:43:33.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:43:33 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:43:33.797 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:43:33 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T03:43:33.797 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:43:33 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:43:33.797 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:43:33 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:43:33.797 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:43:33 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:43:34.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:43:34 smithi115 conmon[90393]: 2022-05-14T03:43:34.319+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:43:35.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:43:34 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:43:35.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:43:34 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:43:35.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T03:43:35.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:43:35.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T03:43:35.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:43:35.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:43:35.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:43:35.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T03:43:35.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:43:35.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T03:43:35.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:43:35.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T03:43:35.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:43:35.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T03:43:35.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:43:35.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T03:43:35.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:43:35.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:43:35.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:43:35.048 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:43:34 smithi047 ceph-mon[77147]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T03:43:35.048 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:43:34 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:43:35.049 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:43:34 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:43:35.049 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:43:34 smithi047 ceph-mon[77147]: pgmap v8025: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 4 op/s 2022-05-14T03:43:35.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:43:34 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:43:35.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:43:34 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:43:35.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T03:43:35.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:43:35.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T03:43:35.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:43:35.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:43:35.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:43:35.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T03:43:35.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:43:35.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T03:43:35.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:43:35.113 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T03:43:35.113 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:43:35.113 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T03:43:35.113 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:43:35.114 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T03:43:35.114 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:43:35.114 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:43:35.114 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:43:35.114 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:43:34 smithi115 ceph-mon[78593]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T03:43:35.115 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:43:34 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:43:35.115 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:43:34 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:43:35.115 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:43:34 smithi115 ceph-mon[78593]: pgmap v8025: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 4 op/s 2022-05-14T03:43:35.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:43:35 smithi115 conmon[90393]: 2022-05-14T03:43:35.291+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:43:36.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:43:35 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:43:36.114 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:43:35 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:43:36.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:43:36 smithi115 conmon[90393]: 2022-05-14T03:43:36.313+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:43:37.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:43:36 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:43:37.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:43:36 smithi047 ceph-mon[77147]: pgmap v8026: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 7 op/s 2022-05-14T03:43:37.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:43:36 smithi047 conmon[78521]: 2022-05-14T03:43:36.559+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:43:37.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:43:36 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T03:43:37.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:43:36 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:43:37.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:43:36 smithi047 conmon[78521]: yield 2022-05-14T03:43:37.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:43:36 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:43:37.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:43:36 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:43:37.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:43:36 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:43:37.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:43:36 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T03:43:37.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:43:36 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:43:37.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:43:36 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T03:43:37.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:43:36 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:43:37.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:43:36 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T03:43:37.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:43:36 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:43:37.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:43:36 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T03:43:37.048 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:43:36 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:43:37.048 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:43:36 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T03:43:37.048 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:43:36 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:43:37.048 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:43:36 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:43:37.048 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:43:36 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:43:37.049 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:43:36 smithi047 conmon[78521]: 2022-05-14T03:43:36.560+0000 7f5344781700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T03:43:37.049 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:43:36 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T03:43:37.049 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:43:36 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:43:37.049 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:43:36 smithi047 conmon[78521]: yield 2022-05-14T03:43:37.049 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:43:36 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:43:37.050 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:43:36 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:43:37.050 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:43:36 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:43:37.050 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:43:36 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T03:43:37.050 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:43:36 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:43:37.051 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:43:36 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T03:43:37.051 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:43:36 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:43:37.051 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:43:36 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T03:43:37.051 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:43:36 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:43:37.051 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:43:36 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T03:43:37.052 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:43:36 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:43:37.052 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:43:36 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T03:43:37.052 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:43:36 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:43:37.052 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:43:36 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:43:37.052 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:43:36 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:43:37.053 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:43:36 smithi047 conmon[78521]: 2022-05-14T03:43:37.053 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:43:36 smithi047 conmon[78521]: During handling of the above exception, another exception occurred: 2022-05-14T03:43:37.053 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:43:36 smithi047 conmon[78521]: 2022-05-14T03:43:37.053 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:43:36 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T03:43:37.054 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:43:36 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T03:43:37.054 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:43:36 smithi047 conmon[78521]: return f(*arg) 2022-05-14T03:43:37.054 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:43:36 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T03:43:37.054 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:43:36 smithi047 conmon[78521]: self._write_client_files(client_files, host) 2022-05-14T03:43:37.054 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:43:36 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T03:43:37.055 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:43:36 smithi047 conmon[78521]: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T03:43:37.055 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:43:36 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T03:43:37.055 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:43:36 smithi047 conmon[78521]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T03:43:37.056 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:43:36 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T03:43:37.056 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:43:36 smithi047 conmon[78521]: return self.event_loop.get_result(coro) 2022-05-14T03:43:37.057 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:43:36 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T03:43:37.057 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:43:36 smithi047 conmon[78521]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T03:43:37.058 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:43:36 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T03:43:37.058 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:43:36 smithi047 conmon[78521]: return self.__get_result() 2022-05-14T03:43:37.058 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:43:36 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T03:43:37.059 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:43:36 smithi047 conmon[78521]: raise self._exception 2022-05-14T03:43:37.059 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:43:36 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T03:43:37.059 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:43:36 smithi047 conmon[78521]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T03:43:37.060 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:43:36 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T03:43:37.060 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:43:36 smithi047 conmon[78521]: conn = await self._remote_connection(host, addr) 2022-05-14T03:43:37.060 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:43:36 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T03:43:37.060 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:43:36 smithi047 conmon[78521]: raise 2022-05-14T03:43:37.060 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:43:36 smithi047 conmon[78521]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T03:43:37.061 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:43:36 smithi047 conmon[78521]: self.gen.throw(type, value, traceback) 2022-05-14T03:43:37.061 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:43:36 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T03:43:37.062 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:43:36 smithi047 conmon[78521]: raise OrchestratorError(msg) 2022-05-14T03:43:37.062 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:43:36 smithi047 conmon[78521]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:43:37.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:43:36 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:43:37.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:43:36 smithi115 ceph-mon[78593]: pgmap v8026: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 7 op/s 2022-05-14T03:43:37.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:43:37 smithi115 conmon[90393]: 2022-05-14T03:43:37.354+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:43:38.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:43:37 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:43:38.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:43:37 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:43:38.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T03:43:38.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:43:38.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T03:43:38.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:43:38.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:43:38.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:43:38.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T03:43:38.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:43:38.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T03:43:38.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:43:38.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T03:43:38.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:43:38.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T03:43:38.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:43:38.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T03:43:38.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:43:38.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:43:38.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:43:38.048 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:43:37 smithi047 ceph-mon[77147]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T03:43:38.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T03:43:38.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:43:38.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T03:43:38.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:43:38.049 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:43:38.049 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:43:38.049 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T03:43:38.049 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:43:38.049 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T03:43:38.050 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:43:38.050 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T03:43:38.050 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:43:38.050 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T03:43:38.051 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:43:38.051 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T03:43:38.051 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:43:38.051 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:43:38.051 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:43:38.052 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T03:43:38.052 INFO:journalctl@ceph.mon.b.smithi047.stdout: During handling of the above exception, another exception occurred: 2022-05-14T03:43:38.052 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T03:43:38.052 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T03:43:38.052 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T03:43:38.053 INFO:journalctl@ceph.mon.b.smithi047.stdout: return f(*arg) 2022-05-14T03:43:38.053 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T03:43:38.053 INFO:journalctl@ceph.mon.b.smithi047.stdout: self._write_client_files(client_files, host) 2022-05-14T03:43:38.053 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T03:43:38.054 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T03:43:38.054 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T03:43:38.054 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T03:43:38.054 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T03:43:38.054 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.event_loop.get_result(coro) 2022-05-14T03:43:38.055 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T03:43:38.055 INFO:journalctl@ceph.mon.b.smithi047.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T03:43:38.055 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T03:43:38.055 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.__get_result() 2022-05-14T03:43:38.055 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T03:43:38.056 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise self._exception 2022-05-14T03:43:38.056 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T03:43:38.056 INFO:journalctl@ceph.mon.b.smithi047.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T03:43:38.056 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T03:43:38.056 INFO:journalctl@ceph.mon.b.smithi047.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T03:43:38.057 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T03:43:38.057 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise 2022-05-14T03:43:38.057 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T03:43:38.057 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.gen.throw(type, value, traceback) 2022-05-14T03:43:38.058 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T03:43:38.058 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OrchestratorError(msg) 2022-05-14T03:43:38.058 INFO:journalctl@ceph.mon.b.smithi047.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:43:38.058 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:43:37 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 15350 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:43:38.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:43:37 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:43:38.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:43:37 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:43:38.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T03:43:38.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:43:38.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T03:43:38.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:43:38.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:43:38.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:43:38.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T03:43:38.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:43:38.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T03:43:38.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:43:38.113 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T03:43:38.113 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:43:38.113 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T03:43:38.113 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:43:38.114 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T03:43:38.114 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:43:38.114 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:43:38.114 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:43:38.114 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:43:37 smithi115 ceph-mon[78593]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T03:43:38.115 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T03:43:38.115 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:43:38.115 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T03:43:38.115 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:43:38.116 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:43:38.116 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:43:38.116 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T03:43:38.116 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:43:38.116 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T03:43:38.117 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:43:38.117 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T03:43:38.117 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:43:38.117 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T03:43:38.118 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:43:38.118 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T03:43:38.118 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:43:38.118 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:43:38.118 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:43:38.119 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T03:43:38.119 INFO:journalctl@ceph.mon.c.smithi115.stdout: During handling of the above exception, another exception occurred: 2022-05-14T03:43:38.119 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T03:43:38.119 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T03:43:38.119 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T03:43:38.120 INFO:journalctl@ceph.mon.c.smithi115.stdout: return f(*arg) 2022-05-14T03:43:38.120 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T03:43:38.120 INFO:journalctl@ceph.mon.c.smithi115.stdout: self._write_client_files(client_files, host) 2022-05-14T03:43:38.120 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T03:43:38.121 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T03:43:38.121 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T03:43:38.121 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T03:43:38.121 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T03:43:38.121 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.event_loop.get_result(coro) 2022-05-14T03:43:38.122 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T03:43:38.122 INFO:journalctl@ceph.mon.c.smithi115.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T03:43:38.122 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T03:43:38.122 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.__get_result() 2022-05-14T03:43:38.123 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T03:43:38.123 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise self._exception 2022-05-14T03:43:38.123 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T03:43:38.123 INFO:journalctl@ceph.mon.c.smithi115.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T03:43:38.123 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T03:43:38.124 INFO:journalctl@ceph.mon.c.smithi115.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T03:43:38.124 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T03:43:38.124 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise 2022-05-14T03:43:38.124 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T03:43:38.124 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.gen.throw(type, value, traceback) 2022-05-14T03:43:38.125 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T03:43:38.125 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OrchestratorError(msg) 2022-05-14T03:43:38.125 INFO:journalctl@ceph.mon.c.smithi115.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:43:38.125 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:43:37 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 15350 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:43:38.790 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:43:38 smithi115 conmon[90393]: 2022-05-14T03:43:38.362+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:43:39.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:43:38 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:43:39.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:43:38 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:43:39.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:43:38 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:43:39.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:43:38 smithi047 ceph-mon[77147]: pgmap v8027: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:43:39.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:43:38 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:43:39.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:43:38 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:43:39.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:43:38 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:43:39.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:43:38 smithi115 ceph-mon[78593]: pgmap v8027: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:43:39.797 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:43:39 smithi115 conmon[90393]: 2022-05-14T03:43:39.395+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:43:40.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:43:39 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:43:40.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:43:39 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:43:40.796 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:43:40 smithi115 conmon[90393]: 2022-05-14T03:43:40.388+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:43:41.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:43:40 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:43:41.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:43:40 smithi115 ceph-mon[78593]: pgmap v8028: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:43:41.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:43:40 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:43:41.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:43:40 smithi047 ceph-mon[77147]: pgmap v8028: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:43:41.801 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:43:41 smithi115 conmon[90393]: 2022-05-14T03:43:41.427+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:43:42.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:43:41 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:43:42.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:43:41 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:43:42.806 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:43:42 smithi115 conmon[90393]: 2022-05-14T03:43:42.438+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:43:43.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:43:42 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:43:43.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:43:42 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:43:43.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:43:42 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:43:43.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:43:42 smithi115 ceph-mon[78593]: pgmap v8029: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:43:43.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:43:42 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:43:43.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:43:42 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:43:43.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:43:42 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:43:43.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:43:42 smithi047 ceph-mon[77147]: pgmap v8029: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:43:43.808 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:43:43 smithi115 conmon[90393]: 2022-05-14T03:43:43.464+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:43:44.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:43:43 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:43:44.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:43:43 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:43:44.812 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:43:44 smithi115 conmon[90393]: 2022-05-14T03:43:44.443+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:43:45.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:43:44 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:43:45.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:43:44 smithi115 ceph-mon[78593]: pgmap v8030: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:43:45.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:43:44 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:43:45.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:43:44 smithi047 ceph-mon[77147]: pgmap v8030: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:43:45.814 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:43:45 smithi115 conmon[90393]: 2022-05-14T03:43:45.475+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:43:46.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:43:45 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:43:46.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:43:45 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:43:46.823 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:43:46 smithi115 conmon[90393]: 2022-05-14T03:43:46.481+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:43:47.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:43:46 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:43:47.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:43:46 smithi115 ceph-mon[78593]: pgmap v8031: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:43:47.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:43:46 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 15355 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:43:47.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:43:46 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:43:47.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:43:46 smithi047 ceph-mon[77147]: pgmap v8031: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:43:47.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:43:46 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 15355 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:43:47.822 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:43:47 smithi115 conmon[90393]: 2022-05-14T03:43:47.507+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:43:48.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:43:47 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:43:48.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:43:47 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:43:48.828 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:43:48 smithi115 conmon[90393]: 2022-05-14T03:43:48.466+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:43:49.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:43:48 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:43:49.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:43:48 smithi115 ceph-mon[78593]: pgmap v8032: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:43:49.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:43:48 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:43:49.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:43:48 smithi047 ceph-mon[77147]: pgmap v8032: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:43:49.833 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:43:49 smithi115 conmon[90393]: 2022-05-14T03:43:49.455+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:43:50.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:43:49 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:43:50.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:43:49 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:43:50.841 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:43:50 smithi115 conmon[90393]: 2022-05-14T03:43:50.444+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:43:51.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:43:50 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:43:51.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:43:50 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:43:51.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:43:50 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:43:51.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:43:50 smithi115 ceph-mon[78593]: pgmap v8033: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:43:51.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:43:50 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:43:51.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:43:50 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:43:51.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:43:50 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:43:51.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:43:50 smithi047 ceph-mon[77147]: pgmap v8033: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:43:51.840 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:43:51 smithi115 conmon[90393]: 2022-05-14T03:43:51.473+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:43:52.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:43:51 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:43:52.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:43:51 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 15365 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:43:52.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:43:51 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:43:52.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:43:51 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 15365 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:43:52.847 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:43:52 smithi115 conmon[90393]: 2022-05-14T03:43:52.454+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:43:53.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:43:52 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:43:53.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:43:52 smithi115 ceph-mon[78593]: pgmap v8034: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:43:53.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:43:52 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:43:53.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:43:52 smithi047 ceph-mon[77147]: pgmap v8034: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:43:53.853 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:43:53 smithi115 conmon[90393]: 2022-05-14T03:43:53.407+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:43:54.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:43:53 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:43:54.301 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:43:53 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:43:54.852 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:43:54 smithi115 conmon[90393]: 2022-05-14T03:43:54.446+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:43:55.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:43:54 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:43:55.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:43:54 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:43:55.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:43:54 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:43:55.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:43:54 smithi115 ceph-mon[78593]: pgmap v8035: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:43:55.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:43:54 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:43:55.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:43:54 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:43:55.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:43:54 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:43:55.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:43:54 smithi047 ceph-mon[77147]: pgmap v8035: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:43:55.860 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:43:55 smithi115 conmon[90393]: 2022-05-14T03:43:55.444+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:43:56.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:43:55 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:43:56.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:43:55 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:43:56.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:43:56 smithi115 conmon[90393]: 2022-05-14T03:43:56.433+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:43:57.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:43:56 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:43:57.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:43:56 smithi047 ceph-mon[77147]: pgmap v8036: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:43:57.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:43:56 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:43:57.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:43:56 smithi115 ceph-mon[78593]: pgmap v8036: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:43:57.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:43:57 smithi115 conmon[90393]: 2022-05-14T03:43:57.476+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:43:58.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:43:57 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:43:58.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:43:57 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 15370 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:43:58.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:43:57 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:43:58.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:43:57 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 15370 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:43:58.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:43:58 smithi115 conmon[90393]: 2022-05-14T03:43:58.440+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:43:59.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:43:58 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:43:59.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:43:58 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:43:59.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:43:58 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:43:59.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:43:58 smithi047 ceph-mon[77147]: pgmap v8037: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:43:59.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:43:58 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:43:59.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:43:58 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:43:59.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:43:58 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:43:59.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:43:58 smithi115 ceph-mon[78593]: pgmap v8037: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:43:59.860 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:43:59 smithi115 conmon[90393]: 2022-05-14T03:43:59.477+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:44:00.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:43:59 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:44:00.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:43:59 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:44:00.863 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:44:00 smithi115 conmon[90393]: 2022-05-14T03:44:00.526+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:44:01.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:44:00 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:44:01.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:44:00 smithi047 ceph-mon[77147]: pgmap v8038: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:44:01.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:44:00 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:44:01.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:44:00 smithi115 ceph-mon[78593]: pgmap v8038: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:44:01.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:44:01 smithi115 conmon[90393]: 2022-05-14T03:44:01.502+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:44:02.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:44:01 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:44:02.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:44:01 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:44:02.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:44:02 smithi115 conmon[90393]: 2022-05-14T03:44:02.486+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:44:03.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:44:02 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:44:03.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:44:02 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:44:03.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:44:02 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:44:03.295 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:44:02 smithi047 ceph-mon[77147]: pgmap v8039: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:44:03.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:44:02 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:44:03.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:44:02 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:44:03.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:44:02 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:44:03.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:44:02 smithi115 ceph-mon[78593]: pgmap v8039: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:44:03.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:44:03 smithi115 conmon[90393]: 2022-05-14T03:44:03.514+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:44:04.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:44:03 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:44:04.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:44:03 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:44:04.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:44:04 smithi115 conmon[90393]: 2022-05-14T03:44:04.479+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:44:05.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:44:04 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:44:05.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:44:04 smithi047 ceph-mon[77147]: pgmap v8040: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:44:05.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:44:04 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:44:05.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:44:04 smithi115 ceph-mon[78593]: pgmap v8040: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:44:05.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:44:05 smithi115 conmon[90393]: 2022-05-14T03:44:05.468+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:44:06.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:44:05 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:44:06.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:44:05 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:44:06.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:44:06 smithi115 conmon[90393]: 2022-05-14T03:44:06.421+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:44:07.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:44:06 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:44:07.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:44:06 smithi047 ceph-mon[77147]: pgmap v8041: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:44:07.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:44:06 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 15375 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:44:07.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:44:06 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:44:07.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:44:06 smithi115 ceph-mon[78593]: pgmap v8041: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:44:07.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:44:06 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 15375 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:44:07.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:44:07 smithi115 conmon[90393]: 2022-05-14T03:44:07.411+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:44:08.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:44:07 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:44:08.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:44:07 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:44:08.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:44:08 smithi115 conmon[90393]: 2022-05-14T03:44:08.396+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:44:09.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:44:08 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:44:09.295 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:44:08 smithi047 ceph-mon[77147]: pgmap v8042: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:44:09.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:44:08 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:44:09.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:44:08 smithi115 ceph-mon[78593]: pgmap v8042: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:44:09.862 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:44:09 smithi115 conmon[90393]: 2022-05-14T03:44:09.366+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:44:10.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:44:09 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:44:10.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:44:09 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:44:10.860 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:44:10 smithi115 conmon[90393]: 2022-05-14T03:44:10.411+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:44:11.295 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:44:10 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:44:11.296 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:44:10 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:44:11.296 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:44:10 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:44:11.296 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:44:10 smithi047 ceph-mon[77147]: pgmap v8043: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:44:11.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:44:10 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:44:11.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:44:10 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:44:11.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:44:10 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:44:11.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:44:10 smithi115 ceph-mon[78593]: pgmap v8043: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:44:11.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:44:11 smithi115 conmon[90393]: 2022-05-14T03:44:11.397+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:44:12.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:44:11 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:44:12.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:44:11 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 15385 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:44:12.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:44:11 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:44:12.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:44:11 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 15385 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:44:12.860 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:44:12 smithi115 conmon[90393]: 2022-05-14T03:44:12.386+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:44:13.295 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:44:12 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:44:13.295 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:44:12 smithi047 ceph-mon[77147]: pgmap v8044: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:44:13.343 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:44:12 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:44:13.343 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:44:12 smithi115 ceph-mon[78593]: pgmap v8044: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:44:13.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:44:13 smithi115 conmon[90393]: 2022-05-14T03:44:13.339+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:44:14.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:44:13 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:44:14.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:44:13 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:44:14.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:44:14 smithi115 conmon[90393]: 2022-05-14T03:44:14.357+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:44:15.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:44:14 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:44:15.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:44:14 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:44:15.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:44:14 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:44:15.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:44:14 smithi047 ceph-mon[77147]: pgmap v8045: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:44:15.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:44:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T03:44:15.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:44:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T03:44:15.350 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:44:14 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:44:15.350 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:44:14 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:44:15.351 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:44:14 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:44:15.351 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:44:14 smithi115 ceph-mon[78593]: pgmap v8045: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:44:15.351 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:44:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T03:44:15.351 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:44:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T03:44:15.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:44:15 smithi115 conmon[90393]: 2022-05-14T03:44:15.347+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:44:16.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:44:15 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:44:16.309 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:44:15 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:44:16.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:44:16 smithi115 conmon[90393]: 2022-05-14T03:44:16.306+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:44:17.277 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:44:16 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:44:17.278 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:44:16 smithi115 ceph-mon[78593]: pgmap v8046: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:44:17.278 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:44:16 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 15390 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:44:17.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:44:16 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:44:17.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:44:16 smithi047 ceph-mon[77147]: pgmap v8046: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:44:17.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:44:16 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 15390 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:44:17.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:44:17 smithi115 conmon[90393]: 2022-05-14T03:44:17.266+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:44:18.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:44:17 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:44:18.306 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:44:17 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:44:18.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:44:18 smithi115 conmon[90393]: 2022-05-14T03:44:18.304+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:44:19.258 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:44:18 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:44:19.259 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:44:18 smithi115 ceph-mon[78593]: pgmap v8047: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:44:19.260 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:44:18 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:44:19.260 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:44:18 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:44:19.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:44:18 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:44:19.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:44:18 smithi047 ceph-mon[77147]: pgmap v8047: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:44:19.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:44:18 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:44:19.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:44:18 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:44:19.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:44:19 smithi115 conmon[90393]: 2022-05-14T03:44:19.255+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:44:20.279 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:44:19 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:44:20.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:44:19 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:44:20.610 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:44:20 smithi115 conmon[90393]: 2022-05-14T03:44:20.276+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:44:21.238 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:44:20 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:44:21.238 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:44:20 smithi115 ceph-mon[78593]: pgmap v8048: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:44:21.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:44:20 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:44:21.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:44:20 smithi047 ceph-mon[77147]: pgmap v8048: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:44:21.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:44:21 smithi115 conmon[90393]: 2022-05-14T03:44:21.235+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:44:22.238 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:44:21 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:44:22.238 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:44:21 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 15395 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:44:22.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:44:21 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:44:22.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:44:21 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 15395 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:44:22.610 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:44:22 smithi115 conmon[90393]: 2022-05-14T03:44:22.235+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:44:23.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:44:22 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:44:23.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:44:22 smithi047 ceph-mon[77147]: pgmap v8049: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:44:23.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:44:22 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:44:23.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:44:22 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:44:23.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:44:23 smithi115 conmon[90393]: 2022-05-14T03:44:23.212+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:44:23.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:44:22 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:44:23.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:44:22 smithi115 ceph-mon[78593]: pgmap v8049: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:44:23.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:44:22 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:44:23.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:44:22 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:44:24.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:44:23 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:44:24.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:44:24 smithi115 conmon[90393]: 2022-05-14T03:44:24.196+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.2.197:6 2.1 2.96a7dda1 (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e150) 2022-05-14T03:44:24.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:44:23 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:44:25.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:44:24 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:44:25.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:44:24 smithi047 ceph-mon[77147]: pgmap v8050: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:44:25.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:44:24 smithi047 ceph-mon[77147]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2022-05-14T03:44:25.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:44:25 smithi115 conmon[90393]: 2022-05-14T03:44:25.164+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:44:25.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:44:24 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:44:25.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:44:24 smithi115 ceph-mon[78593]: pgmap v8050: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:44:25.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:44:24 smithi115 ceph-mon[78593]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2022-05-14T03:44:26.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:44:25 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:44:26.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:44:25 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:44:26.362 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:44:26 smithi115 conmon[90393]: 2022-05-14T03:44:26.134+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:44:27.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:44:26 smithi047 ceph-mon[77147]: pgmap v8051: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:44:27.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:44:26 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:44:27.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:44:26 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 15400 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:44:27.360 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:44:27 smithi115 conmon[90393]: 2022-05-14T03:44:27.113+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:44:27.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:44:26 smithi115 ceph-mon[78593]: pgmap v8051: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:44:27.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:44:26 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:44:27.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:44:26 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 15400 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:44:28.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:44:27 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:44:28.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:44:27 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:44:28.360 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:44:28 smithi115 conmon[90393]: 2022-05-14T03:44:28.146+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:44:29.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:44:28 smithi047 ceph-mon[77147]: pgmap v8052: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:44:29.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:44:28 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:44:29.361 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:44:29 smithi115 conmon[90393]: 2022-05-14T03:44:29.180+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:44:29.362 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:44:28 smithi115 ceph-mon[78593]: pgmap v8052: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:44:29.362 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:44:28 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:44:30.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:44:29 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:44:30.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:44:30 smithi115 conmon[90393]: 2022-05-14T03:44:30.222+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:44:30.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:44:29 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:44:31.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:44:30 smithi047 ceph-mon[77147]: pgmap v8053: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:44:31.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:44:30 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:44:31.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:44:30 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:44:31.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:44:30 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:44:31.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:44:31 smithi115 conmon[90393]: 2022-05-14T03:44:31.241+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:44:31.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:44:30 smithi115 ceph-mon[78593]: pgmap v8053: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:44:31.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:44:30 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:44:31.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:44:30 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:44:31.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:44:30 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:44:32.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:44:31 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:44:32.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:44:31 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 15405 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:44:32.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:44:32 smithi115 conmon[90393]: 2022-05-14T03:44:32.243+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:44:32.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:44:31 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:44:32.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:44:31 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 15405 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:44:33.266 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:44:32 smithi115 ceph-mon[78593]: pgmap v8054: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:44:33.266 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:44:32 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:44:33.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:44:32 smithi047 ceph-mon[77147]: pgmap v8054: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:44:33.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:44:32 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:44:33.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:44:33 smithi115 conmon[90393]: 2022-05-14T03:44:33.263+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:44:34.272 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:44:34 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:44:34.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:44:34 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:44:34.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:44:34 smithi115 conmon[90393]: 2022-05-14T03:44:34.268+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:44:35.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:44:35 smithi047 ceph-mon[77147]: pgmap v8055: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:44:35.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:44:35 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:44:35.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:44:35 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:44:35.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:44:35 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:44:35.360 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:44:35 smithi115 conmon[90393]: 2022-05-14T03:44:35.236+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:44:35.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:44:35 smithi115 ceph-mon[78593]: pgmap v8055: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:44:35.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:44:35 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:44:35.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:44:35 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:44:35.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:44:35 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:44:36.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:44:36 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:44:36.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:44:36 smithi115 conmon[90393]: 2022-05-14T03:44:36.257+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:44:36.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:44:36 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:44:37.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:44:37 smithi047 ceph-mon[77147]: pgmap v8056: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:44:37.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:44:37 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:44:37.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:44:37 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T03:44:37.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:44:37 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T03:44:37.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:44:37 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T03:44:37.295 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:44:37 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 15410 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:44:37.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:44:37 smithi047 conmon[78521]: 2022-05-14T03:44:37.167+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:44:37.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:44:37 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T03:44:37.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:44:37 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:44:37.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:44:37 smithi047 conmon[78521]: yield 2022-05-14T03:44:37.297 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:44:37 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:44:37.297 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:44:37 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:44:37.297 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:44:37 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:44:37.297 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:44:37 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T03:44:37.298 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:44:37 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:44:37.298 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:44:37 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T03:44:37.298 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:44:37 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:44:37.299 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:44:37 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T03:44:37.299 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:44:37 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:44:37.300 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:44:37 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T03:44:37.300 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:44:37 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:44:37.300 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:44:37 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T03:44:37.300 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:44:37 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:44:37.300 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:44:37 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:44:37.301 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:44:37 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:44:37.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:44:37 smithi115 conmon[90393]: 2022-05-14T03:44:37.222+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:44:37.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:44:37 smithi115 ceph-mon[78593]: pgmap v8056: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:44:37.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:44:37 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:44:37.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:44:37 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T03:44:37.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:44:37 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T03:44:37.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:44:37 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T03:44:37.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:44:37 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 15410 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:44:38.296 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:44:38 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:44:38.360 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:44:38 smithi115 conmon[90393]: 2022-05-14T03:44:38.200+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:44:38.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:44:38 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:44:39.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:44:39 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:44:39.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T03:44:39.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:44:39.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T03:44:39.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:44:39.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:44:39.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:44:39.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T03:44:39.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:44:39.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T03:44:39.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:44:39.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T03:44:39.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:44:39.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T03:44:39.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:44:39.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T03:44:39.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:44:39.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:44:39.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:44:39.298 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:44:39 smithi047 ceph-mon[77147]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T03:44:39.298 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:44:39 smithi047 ceph-mon[77147]: pgmap v8057: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:44:39.298 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:44:39 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:44:39.298 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:44:39 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:44:39.299 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:44:39 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:44:39.361 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:44:39 smithi115 conmon[90393]: 2022-05-14T03:44:39.153+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:44:39.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:44:39 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:44:39.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T03:44:39.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:44:39.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T03:44:39.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:44:39.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:44:39.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:44:39.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T03:44:39.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:44:39.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T03:44:39.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:44:39.365 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T03:44:39.365 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:44:39.365 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T03:44:39.365 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:44:39.366 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T03:44:39.366 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:44:39.366 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:44:39.366 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:44:39.366 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:44:39 smithi115 ceph-mon[78593]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T03:44:39.368 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:44:39 smithi115 ceph-mon[78593]: pgmap v8057: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:44:39.369 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:44:39 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:44:39.369 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:44:39 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:44:39.369 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:44:39 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:44:40.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:44:40 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:44:40.293 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:44:40 smithi047 conmon[78521]: 2022-05-14T03:44:40.239+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:44:40.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:44:40 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T03:44:40.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:44:40 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:44:40.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:44:40 smithi047 conmon[78521]: yield 2022-05-14T03:44:40.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:44:40 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:44:40.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:44:40 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:44:40.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:44:40 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:44:40.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:44:40 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T03:44:40.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:44:40 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:44:40.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:44:40 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T03:44:40.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:44:40 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:44:40.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:44:40 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T03:44:40.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:44:40 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:44:40.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:44:40 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T03:44:40.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:44:40 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:44:40.297 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:44:40 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T03:44:40.297 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:44:40 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:44:40.297 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:44:40 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:44:40.297 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:44:40 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:44:40.298 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:44:40 smithi047 conmon[78521]: 2022-05-14T03:44:40.240+0000 7f533f777700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T03:44:40.298 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:44:40 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T03:44:40.298 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:44:40 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:44:40.298 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:44:40 smithi047 conmon[78521]: yield 2022-05-14T03:44:40.298 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:44:40 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:44:40.299 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:44:40 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:44:40.299 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:44:40 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:44:40.299 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:44:40 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T03:44:40.300 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:44:40 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:44:40.300 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:44:40 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T03:44:40.300 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:44:40 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:44:40.300 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:44:40 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T03:44:40.300 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:44:40 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:44:40.301 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:44:40 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T03:44:40.301 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:44:40 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:44:40.301 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:44:40 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T03:44:40.301 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:44:40 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:44:40.301 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:44:40 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:44:40.302 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:44:40 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:44:40.302 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:44:40 smithi047 conmon[78521]: 2022-05-14T03:44:40.303 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:44:40 smithi047 conmon[78521]: During handling of the above exception, another exception occurred: 2022-05-14T03:44:40.303 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:44:40 smithi047 conmon[78521]: 2022-05-14T03:44:40.303 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:44:40 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T03:44:40.303 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:44:40 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T03:44:40.304 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:44:40 smithi047 conmon[78521]: return f(*arg) 2022-05-14T03:44:40.304 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:44:40 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T03:44:40.304 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:44:40 smithi047 conmon[78521]: self._write_client_files(client_files, host) 2022-05-14T03:44:40.304 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:44:40 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T03:44:40.305 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:44:40 smithi047 conmon[78521]: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T03:44:40.305 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:44:40 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T03:44:40.305 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:44:40 smithi047 conmon[78521]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T03:44:40.305 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:44:40 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T03:44:40.306 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:44:40 smithi047 conmon[78521]: return self.event_loop.get_result(coro) 2022-05-14T03:44:40.306 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:44:40 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T03:44:40.306 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:44:40 smithi047 conmon[78521]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T03:44:40.306 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:44:40 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T03:44:40.306 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:44:40 smithi047 conmon[78521]: return self.__get_result() 2022-05-14T03:44:40.307 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:44:40 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T03:44:40.307 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:44:40 smithi047 conmon[78521]: raise self._exception 2022-05-14T03:44:40.307 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:44:40 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T03:44:40.307 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:44:40 smithi047 conmon[78521]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T03:44:40.308 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:44:40 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T03:44:40.308 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:44:40 smithi047 conmon[78521]: conn = await self._remote_connection(host, addr) 2022-05-14T03:44:40.308 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:44:40 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T03:44:40.308 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:44:40 smithi047 conmon[78521]: raise 2022-05-14T03:44:40.308 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:44:40 smithi047 conmon[78521]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T03:44:40.309 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:44:40 smithi047 conmon[78521]: self.gen.throw(type, value, traceback) 2022-05-14T03:44:40.309 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:44:40 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T03:44:40.309 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:44:40 smithi047 conmon[78521]: raise OrchestratorError(msg) 2022-05-14T03:44:40.309 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:44:40 smithi047 conmon[78521]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:44:40.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:44:40 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:44:40.360 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:44:40 smithi115 conmon[90393]: 2022-05-14T03:44:40.155+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:44:41.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:44:41 smithi047 ceph-mon[77147]: pgmap v8058: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:44:41.295 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:44:41 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:44:41.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:44:41 smithi115 conmon[90393]: 2022-05-14T03:44:41.168+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:44:41.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:44:41 smithi115 ceph-mon[78593]: pgmap v8058: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:44:41.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:44:41 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:44:42.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:44:42 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:44:42.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T03:44:42.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:44:42.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T03:44:42.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:44:42.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:44:42.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:44:42.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T03:44:42.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:44:42.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T03:44:42.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:44:42.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T03:44:42.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:44:42.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T03:44:42.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:44:42.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T03:44:42.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:44:42.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:44:42.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:44:42.297 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:44:42 smithi047 ceph-mon[77147]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T03:44:42.298 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T03:44:42.298 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:44:42.298 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T03:44:42.298 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:44:42.298 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:44:42.299 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:44:42.299 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T03:44:42.299 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:44:42.300 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T03:44:42.300 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:44:42.300 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T03:44:42.300 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:44:42.301 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T03:44:42.301 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:44:42.302 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T03:44:42.302 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:44:42.302 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:44:42.303 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:44:42.303 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T03:44:42.303 INFO:journalctl@ceph.mon.b.smithi047.stdout: During handling of the above exception, another exception occurred: 2022-05-14T03:44:42.303 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T03:44:42.303 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T03:44:42.304 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T03:44:42.304 INFO:journalctl@ceph.mon.b.smithi047.stdout: return f(*arg) 2022-05-14T03:44:42.304 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T03:44:42.304 INFO:journalctl@ceph.mon.b.smithi047.stdout: self._write_client_files(client_files, host) 2022-05-14T03:44:42.304 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T03:44:42.305 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T03:44:42.305 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T03:44:42.305 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T03:44:42.305 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T03:44:42.307 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.event_loop.get_result(coro) 2022-05-14T03:44:42.307 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T03:44:42.307 INFO:journalctl@ceph.mon.b.smithi047.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T03:44:42.308 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T03:44:42.308 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.__get_result() 2022-05-14T03:44:42.308 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T03:44:42.308 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise self._exception 2022-05-14T03:44:42.309 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T03:44:42.309 INFO:journalctl@ceph.mon.b.smithi047.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T03:44:42.309 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T03:44:42.309 INFO:journalctl@ceph.mon.b.smithi047.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T03:44:42.310 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T03:44:42.310 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise 2022-05-14T03:44:42.310 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T03:44:42.310 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.gen.throw(type, value, traceback) 2022-05-14T03:44:42.311 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T03:44:42.311 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OrchestratorError(msg) 2022-05-14T03:44:42.311 INFO:journalctl@ceph.mon.b.smithi047.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:44:42.311 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:44:42 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:44:42.311 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:44:42 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 15415 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:44:42.360 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:44:42 smithi115 conmon[90393]: 2022-05-14T03:44:42.139+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:44:42.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:44:42 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:44:42.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T03:44:42.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:44:42.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T03:44:42.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:44:42.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:44:42.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:44:42.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T03:44:42.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:44:42.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T03:44:42.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:44:42.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T03:44:42.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:44:42.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T03:44:42.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:44:42.365 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T03:44:42.365 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:44:42.365 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:44:42.366 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:44:42.366 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:44:42 smithi115 ceph-mon[78593]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T03:44:42.366 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T03:44:42.366 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:44:42.367 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T03:44:42.367 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:44:42.367 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:44:42.367 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:44:42.368 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T03:44:42.368 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:44:42.368 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T03:44:42.369 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:44:42.369 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T03:44:42.370 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:44:42.370 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T03:44:42.370 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:44:42.370 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T03:44:42.371 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:44:42.371 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:44:42.371 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:44:42.371 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T03:44:42.371 INFO:journalctl@ceph.mon.c.smithi115.stdout: During handling of the above exception, another exception occurred: 2022-05-14T03:44:42.372 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T03:44:42.372 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T03:44:42.373 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T03:44:42.373 INFO:journalctl@ceph.mon.c.smithi115.stdout: return f(*arg) 2022-05-14T03:44:42.374 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T03:44:42.374 INFO:journalctl@ceph.mon.c.smithi115.stdout: self._write_client_files(client_files, host) 2022-05-14T03:44:42.375 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T03:44:42.375 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T03:44:42.375 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T03:44:42.376 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T03:44:42.376 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T03:44:42.377 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.event_loop.get_result(coro) 2022-05-14T03:44:42.377 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T03:44:42.378 INFO:journalctl@ceph.mon.c.smithi115.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T03:44:42.378 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T03:44:42.378 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.__get_result() 2022-05-14T03:44:42.379 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T03:44:42.379 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise self._exception 2022-05-14T03:44:42.379 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T03:44:42.380 INFO:journalctl@ceph.mon.c.smithi115.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T03:44:42.380 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T03:44:42.381 INFO:journalctl@ceph.mon.c.smithi115.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T03:44:42.381 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T03:44:42.382 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise 2022-05-14T03:44:42.382 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T03:44:42.383 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.gen.throw(type, value, traceback) 2022-05-14T03:44:42.383 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T03:44:42.384 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OrchestratorError(msg) 2022-05-14T03:44:42.384 INFO:journalctl@ceph.mon.c.smithi115.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:44:42.385 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:44:42 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:44:42.385 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:44:42 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 15415 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:44:43.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:44:43 smithi047 ceph-mon[77147]: pgmap v8059: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:44:43.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:44:43 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:44:43.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:44:43 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:44:43.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:44:43 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:44:43.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:44:43 smithi115 conmon[90393]: 2022-05-14T03:44:43.113+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:44:43.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:44:43 smithi115 ceph-mon[78593]: pgmap v8059: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:44:43.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:44:43 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:44:43.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:44:43 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:44:43.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:44:43 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:44:44.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:44:44 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:44:44.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:44:44 smithi115 conmon[90393]: 2022-05-14T03:44:44.084+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:44:44.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:44:44 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:44:45.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:44:45 smithi047 ceph-mon[77147]: pgmap v8060: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:44:45.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:44:45 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:44:45.360 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:44:45 smithi115 conmon[90393]: 2022-05-14T03:44:45.102+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:44:45.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:44:45 smithi115 ceph-mon[78593]: pgmap v8060: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:44:45.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:44:45 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:44:46.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:44:46 smithi115 conmon[90393]: 2022-05-14T03:44:46.061+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:44:46.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:44:46 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:44:46.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:44:46 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:44:47.366 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:44:47 smithi115 conmon[90393]: 2022-05-14T03:44:47.101+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:44:47.367 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:44:47 smithi115 ceph-mon[78593]: pgmap v8061: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:44:47.367 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:44:47 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:44:47.367 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:44:47 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 15420 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:44:47.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:44:47 smithi047 ceph-mon[77147]: pgmap v8061: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:44:47.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:44:47 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:44:47.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:44:47 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 15420 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:44:48.360 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:44:48 smithi115 conmon[90393]: 2022-05-14T03:44:48.105+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:44:48.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:44:48 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:44:48.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:44:48 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:44:49.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:44:49 smithi115 conmon[90393]: 2022-05-14T03:44:49.099+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:44:49.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:44:49 smithi115 ceph-mon[78593]: pgmap v8062: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:44:49.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:44:49 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:44:49.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:44:49 smithi047 ceph-mon[77147]: pgmap v8062: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:44:49.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:44:49 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:44:50.361 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:44:50 smithi115 conmon[90393]: 2022-05-14T03:44:50.091+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:44:50.363 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:44:50 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:44:50.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:44:50 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:44:51.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:44:51 smithi115 conmon[90393]: 2022-05-14T03:44:51.095+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:44:51.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:44:51 smithi115 ceph-mon[78593]: pgmap v8063: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:44:51.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:44:51 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:44:51.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:44:51 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:44:51.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:44:51 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:44:51.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:44:51 smithi047 ceph-mon[77147]: pgmap v8063: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:44:51.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:44:51 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:44:51.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:44:51 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:44:51.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:44:51 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:44:52.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:44:52 smithi115 conmon[90393]: 2022-05-14T03:44:52.117+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:44:52.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:44:52 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:44:52.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:44:52 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 15426 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:44:52.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:44:52 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:44:52.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:44:52 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 15426 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:44:53.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:44:53 smithi115 conmon[90393]: 2022-05-14T03:44:53.099+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:44:53.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:44:53 smithi115 ceph-mon[78593]: pgmap v8064: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:44:53.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:44:53 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:44:53.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:44:53 smithi047 ceph-mon[77147]: pgmap v8064: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:44:53.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:44:53 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:44:54.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:44:54 smithi115 conmon[90393]: 2022-05-14T03:44:54.115+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:44:54.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:44:54 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:44:54.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:44:54 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:44:55.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:44:55 smithi115 conmon[90393]: 2022-05-14T03:44:55.104+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:44:55.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:44:55 smithi115 ceph-mon[78593]: pgmap v8065: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:44:55.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:44:55 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:44:55.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:44:55 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:44:55.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:44:55 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:44:55.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:44:55 smithi047 ceph-mon[77147]: pgmap v8065: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:44:55.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:44:55 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:44:55.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:44:55 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:44:55.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:44:55 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:44:56.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:44:56 smithi115 conmon[90393]: 2022-05-14T03:44:56.112+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:44:56.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:44:56 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:44:56.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:44:56 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:44:57.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:44:57 smithi115 conmon[90393]: 2022-05-14T03:44:57.111+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:44:57.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:44:57 smithi115 ceph-mon[78593]: pgmap v8066: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:44:57.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:44:57 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:44:57.365 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:44:57 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 15430 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:44:57.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:44:57 smithi047 ceph-mon[77147]: pgmap v8066: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:44:57.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:44:57 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:44:57.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:44:57 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 15430 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:44:58.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:44:58 smithi115 conmon[90393]: 2022-05-14T03:44:58.101+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:44:59.114 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:44:58 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:44:59.114 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:44:58 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:44:59.114 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:44:58 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:44:59.115 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:44:58 smithi115 ceph-mon[78593]: pgmap v8067: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:44:59.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:44:58 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:44:59.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:44:58 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:44:59.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:44:58 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:44:59.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:44:58 smithi047 ceph-mon[77147]: pgmap v8067: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:44:59.610 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:44:59 smithi115 conmon[90393]: 2022-05-14T03:44:59.132+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:45:00.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:44:59 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:45:00.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:44:59 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:45:00.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:45:00 smithi115 conmon[90393]: 2022-05-14T03:45:00.151+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:45:01.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:45:00 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:45:01.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:45:00 smithi115 ceph-mon[78593]: pgmap v8068: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:45:01.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:45:00 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:45:01.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:45:00 smithi047 ceph-mon[77147]: pgmap v8068: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:45:01.611 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:45:01 smithi115 conmon[90393]: 2022-05-14T03:45:01.197+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:45:02.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:45:01 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:45:02.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:45:01 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 15435 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:45:02.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:45:01 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:45:02.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:45:01 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 15435 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:45:02.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:45:02 smithi115 conmon[90393]: 2022-05-14T03:45:02.197+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:45:03.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:45:02 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:45:03.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:45:02 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:45:03.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:45:02 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:45:03.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:45:02 smithi115 ceph-mon[78593]: pgmap v8069: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:45:03.295 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:45:02 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:45:03.295 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:45:02 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:45:03.296 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:45:02 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:45:03.296 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:45:02 smithi047 ceph-mon[77147]: pgmap v8069: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:45:03.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:45:03 smithi115 conmon[90393]: 2022-05-14T03:45:03.182+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:45:04.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:45:03 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:45:04.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:45:03 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:45:04.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:45:04 smithi115 conmon[90393]: 2022-05-14T03:45:04.150+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:45:05.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:45:04 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:45:05.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:45:04 smithi115 ceph-mon[78593]: pgmap v8070: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:45:05.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:45:04 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:45:05.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:45:04 smithi047 ceph-mon[77147]: pgmap v8070: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:45:05.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:45:05 smithi115 conmon[90393]: 2022-05-14T03:45:05.132+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:45:06.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:45:05 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:45:06.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:45:05 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:45:06.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:45:06 smithi115 conmon[90393]: 2022-05-14T03:45:06.153+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:45:07.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:45:06 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:45:07.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:45:06 smithi115 ceph-mon[78593]: pgmap v8071: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:45:07.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:45:06 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:45:07.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:45:06 smithi047 ceph-mon[77147]: pgmap v8071: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:45:07.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:45:07 smithi115 conmon[90393]: 2022-05-14T03:45:07.111+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:45:08.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:45:07 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:45:08.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:45:07 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 15440 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:45:08.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:45:07 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:45:08.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:45:07 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 15440 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:45:08.610 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:45:08 smithi115 conmon[90393]: 2022-05-14T03:45:08.125+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:45:09.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:45:08 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:45:09.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:45:08 smithi115 ceph-mon[78593]: pgmap v8072: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:45:09.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:45:08 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:45:09.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:45:08 smithi047 ceph-mon[77147]: pgmap v8072: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:45:09.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:45:09 smithi115 conmon[90393]: 2022-05-14T03:45:09.112+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:45:10.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:45:09 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:45:10.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:45:09 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:45:10.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:45:10 smithi115 conmon[90393]: 2022-05-14T03:45:10.122+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:45:11.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:45:10 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:45:11.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:45:10 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:45:11.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:45:10 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:45:11.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:45:10 smithi115 ceph-mon[78593]: pgmap v8073: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:45:11.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:45:10 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:45:11.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:45:10 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:45:11.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:45:10 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:45:11.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:45:10 smithi047 ceph-mon[77147]: pgmap v8073: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:45:11.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:45:11 smithi115 conmon[90393]: 2022-05-14T03:45:11.135+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:45:12.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:45:11 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:45:12.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:45:11 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:45:12.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:45:12 smithi115 conmon[90393]: 2022-05-14T03:45:12.122+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:45:13.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:45:13 smithi115 conmon[90393]: 2022-05-14T03:45:13.078+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:45:13.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:45:12 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:45:13.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:45:12 smithi115 ceph-mon[78593]: pgmap v8074: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:45:13.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:45:12 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:45:13.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:45:12 smithi047 ceph-mon[77147]: pgmap v8074: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:45:14.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:45:14 smithi115 conmon[90393]: 2022-05-14T03:45:14.032+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:45:14.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:45:13 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:45:14.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:45:13 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:45:15.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:45:14 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:45:15.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:45:14 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:45:15.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:45:14 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:45:15.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:45:14 smithi047 ceph-mon[77147]: pgmap v8075: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:45:15.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:45:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T03:45:15.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:45:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T03:45:15.360 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:45:14 smithi115 conmon[90393]: 2022-05-14T03:45:14.984+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:45:15.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:45:14 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:45:15.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:45:14 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:45:15.362 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:45:14 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:45:15.362 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:45:14 smithi115 ceph-mon[78593]: pgmap v8075: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:45:15.362 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:45:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T03:45:15.363 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:45:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T03:45:16.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:45:15 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:45:16.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:45:15 smithi115 conmon[90393]: 2022-05-14T03:45:15.961+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:45:16.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:45:15 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:45:17.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:45:16 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:45:17.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:45:16 smithi047 ceph-mon[77147]: pgmap v8076: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:45:17.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:45:16 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:45:17.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:45:16 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 15445 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:45:17.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:45:16 smithi115 conmon[90393]: 2022-05-14T03:45:16.948+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:45:17.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:45:16 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:45:17.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:45:16 smithi115 ceph-mon[78593]: pgmap v8076: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:45:17.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:45:16 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:45:17.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:45:16 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 15445 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:45:18.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:45:17 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:45:18.362 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:45:17 smithi115 conmon[90393]: 2022-05-14T03:45:17.991+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:45:18.363 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:45:17 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:45:19.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:45:18 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:45:19.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:45:18 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:45:19.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:45:18 smithi047 ceph-mon[77147]: pgmap v8077: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:45:19.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:45:18 smithi115 conmon[90393]: 2022-05-14T03:45:18.984+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:45:19.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:45:18 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:45:19.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:45:18 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:45:19.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:45:18 smithi115 ceph-mon[78593]: pgmap v8077: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:45:20.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:45:19 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:45:20.360 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:45:19 smithi115 conmon[90393]: 2022-05-14T03:45:19.981+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:45:20.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:45:19 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:45:21.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:45:20 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:45:21.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:45:20 smithi047 ceph-mon[77147]: pgmap v8078: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:45:21.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:45:20 smithi115 conmon[90393]: 2022-05-14T03:45:20.993+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:45:21.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:45:20 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:45:21.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:45:20 smithi115 ceph-mon[78593]: pgmap v8078: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:45:22.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:45:21 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:45:22.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:45:21 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 15455 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:45:22.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:45:22 smithi115 conmon[90393]: 2022-05-14T03:45:21.999+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:45:22.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:45:21 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:45:22.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:45:21 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 15455 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:45:23.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:45:22 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:45:23.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:45:22 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:45:23.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:45:22 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:45:23.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:45:22 smithi047 ceph-mon[77147]: pgmap v8079: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:45:23.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:45:23 smithi115 conmon[90393]: 2022-05-14T03:45:23.030+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:45:23.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:45:22 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:45:23.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:45:22 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:45:23.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:45:22 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:45:23.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:45:22 smithi115 ceph-mon[78593]: pgmap v8079: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:45:24.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:45:23 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:45:24.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:45:24 smithi115 conmon[90393]: 2022-05-14T03:45:24.064+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:45:24.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:45:23 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:45:25.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:45:24 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:45:25.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:45:24 smithi047 ceph-mon[77147]: pgmap v8080: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:45:25.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:45:25 smithi115 conmon[90393]: 2022-05-14T03:45:25.064+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:45:25.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:45:24 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:45:25.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:45:24 smithi115 ceph-mon[78593]: pgmap v8080: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:45:26.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:45:25 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:45:26.361 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:45:26 smithi115 conmon[90393]: 2022-05-14T03:45:26.081+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:45:26.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:45:25 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:45:27.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:45:26 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:45:27.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:45:26 smithi047 ceph-mon[77147]: pgmap v8081: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:45:27.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:45:27 smithi115 conmon[90393]: 2022-05-14T03:45:27.072+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:45:27.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:45:26 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:45:27.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:45:26 smithi115 ceph-mon[78593]: pgmap v8081: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:45:28.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:45:27 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:45:28.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:45:27 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 15460 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:45:28.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:45:27 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:45:28.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:45:27 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 15460 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:45:28.360 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:45:28 smithi115 conmon[90393]: 2022-05-14T03:45:28.051+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:45:29.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:45:28 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:45:29.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:45:28 smithi047 ceph-mon[77147]: pgmap v8082: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:45:29.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:45:29 smithi115 conmon[90393]: 2022-05-14T03:45:29.060+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:45:29.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:45:28 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:45:29.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:45:28 smithi115 ceph-mon[78593]: pgmap v8082: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:45:30.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:45:29 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:45:30.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:45:29 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:45:30.360 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:45:30 smithi115 conmon[90393]: 2022-05-14T03:45:30.026+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:45:31.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:45:30 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:45:31.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:45:30 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:45:31.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:45:30 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:45:31.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:45:30 smithi047 ceph-mon[77147]: pgmap v8083: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:45:31.361 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:45:31 smithi115 conmon[90393]: 2022-05-14T03:45:31.075+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:45:31.362 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:45:30 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:45:31.363 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:45:30 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:45:31.363 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:45:30 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:45:31.363 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:45:30 smithi115 ceph-mon[78593]: pgmap v8083: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:45:32.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:45:31 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:45:32.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:45:32 smithi115 conmon[90393]: 2022-05-14T03:45:32.090+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:45:32.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:45:31 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:45:33.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:45:32 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:45:33.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:45:32 smithi047 ceph-mon[77147]: pgmap v8084: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:45:33.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:45:33 smithi115 conmon[90393]: 2022-05-14T03:45:33.070+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:45:33.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:45:32 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:45:33.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:45:32 smithi115 ceph-mon[78593]: pgmap v8084: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:45:34.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:45:33 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:45:34.360 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:45:34 smithi115 conmon[90393]: 2022-05-14T03:45:34.024+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:45:34.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:45:33 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:45:35.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:45:34 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:45:35.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:45:34 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:45:35.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:45:34 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:45:35.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:45:34 smithi047 ceph-mon[77147]: pgmap v8085: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:45:35.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:45:35 smithi115 conmon[90393]: 2022-05-14T03:45:35.058+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:45:35.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:45:34 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:45:35.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:45:34 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:45:35.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:45:34 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:45:35.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:45:34 smithi115 ceph-mon[78593]: pgmap v8085: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:45:36.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:45:35 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:45:36.360 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:45:36 smithi115 conmon[90393]: 2022-05-14T03:45:36.070+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:45:36.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:45:35 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:45:37.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:45:36 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:45:37.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:45:36 smithi047 ceph-mon[77147]: pgmap v8086: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:45:37.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:45:36 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 15465 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:45:37.362 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:45:37 smithi115 conmon[90393]: 2022-05-14T03:45:37.113+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:45:37.363 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:45:36 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:45:37.363 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:45:36 smithi115 ceph-mon[78593]: pgmap v8086: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:45:37.363 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:45:36 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 15465 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:45:38.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:45:37 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:45:38.362 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:45:37 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:45:38.364 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:45:38 smithi115 conmon[90393]: 2022-05-14T03:45:38.115+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:45:39.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:45:38 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:45:39.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:45:38 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:45:39.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:45:38 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:45:39.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:45:38 smithi047 ceph-mon[77147]: pgmap v8087: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:45:39.361 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:45:39 smithi115 conmon[90393]: 2022-05-14T03:45:39.146+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:45:39.362 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:45:38 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:45:39.363 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:45:38 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:45:39.364 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:45:38 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:45:39.366 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:45:38 smithi115 ceph-mon[78593]: pgmap v8087: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:45:40.288 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:45:39 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:45:40.289 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:45:40 smithi115 conmon[90393]: 2022-05-14T03:45:40.147+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:45:40.289 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:45:39 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:45:41.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:45:40 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:45:41.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:45:40 smithi047 ceph-mon[77147]: pgmap v8088: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:45:41.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:45:40 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T03:45:41.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:45:40 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T03:45:41.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:45:40 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T03:45:41.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:45:40 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:45:41.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:45:40 smithi115 ceph-mon[78593]: pgmap v8088: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:45:41.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:45:40 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T03:45:41.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:45:40 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T03:45:41.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:45:40 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T03:45:41.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:45:41 smithi115 conmon[90393]: 2022-05-14T03:45:41.144+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:45:42.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:45:41 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:45:42.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:45:41 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 15475 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:45:42.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:45:42 smithi115 conmon[90393]: 2022-05-14T03:45:42.125+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:45:42.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:45:41 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:45:42.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:45:41 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 15475 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:45:43.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:45:42 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:45:43.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:45:42 smithi047 ceph-mon[77147]: pgmap v8089: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:45:43.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:45:42 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:45:43.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:45:42 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:45:43.361 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:45:43 smithi115 conmon[90393]: 2022-05-14T03:45:43.122+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:45:43.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:45:42 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:45:43.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:45:42 smithi115 ceph-mon[78593]: pgmap v8089: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:45:43.362 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:45:42 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:45:43.362 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:45:42 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:45:43.793 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:45:43 smithi047 conmon[78521]: 2022-05-14T03:45:43.344+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:45:43.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:45:43 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T03:45:43.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:45:43 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:45:43.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:45:43 smithi047 conmon[78521]: yield 2022-05-14T03:45:43.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:45:43 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:45:43.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:45:43 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:45:43.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:45:43 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:45:43.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:45:43 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T03:45:43.797 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:45:43 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:45:43.797 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:45:43 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T03:45:43.798 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:45:43 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:45:43.798 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:45:43 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T03:45:43.799 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:45:43 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:45:43.799 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:45:43 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T03:45:43.799 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:45:43 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:45:43.800 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:45:43 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T03:45:43.800 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:45:43 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:45:43.801 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:45:43 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:45:43.801 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:45:43 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:45:44.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:45:43 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:45:44.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:45:44 smithi115 conmon[90393]: 2022-05-14T03:45:44.118+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:45:44.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:45:43 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:45:45.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:45:44 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:45:45.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:45:44 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:45:45.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T03:45:45.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:45:45.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T03:45:45.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:45:45.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:45:45.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:45:45.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T03:45:45.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:45:45.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T03:45:45.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:45:45.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T03:45:45.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:45:45.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T03:45:45.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:45:45.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T03:45:45.298 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:45:45.298 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:45:45.298 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:45:45.299 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:45:44 smithi047 ceph-mon[77147]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T03:45:45.299 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:45:44 smithi047 ceph-mon[77147]: pgmap v8090: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:45:45.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:45:45 smithi115 conmon[90393]: 2022-05-14T03:45:45.143+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:45:45.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:45:44 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:45:45.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:45:44 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:45:45.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T03:45:45.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:45:45.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T03:45:45.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:45:45.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:45:45.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:45:45.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T03:45:45.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:45:45.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T03:45:45.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:45:45.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T03:45:45.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:45:45.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T03:45:45.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:45:45.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T03:45:45.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:45:45.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:45:45.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:45:45.364 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:45:44 smithi115 ceph-mon[78593]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T03:45:45.365 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:45:44 smithi115 ceph-mon[78593]: pgmap v8090: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:45:46.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:45:45 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:45:46.360 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:45:46 smithi115 conmon[90393]: 2022-05-14T03:45:46.108+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:45:46.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:45:45 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:45:46.793 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:45:46 smithi047 conmon[78521]: 2022-05-14T03:45:46.416+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:45:46.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:45:46 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T03:45:46.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:45:46 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:45:46.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:45:46 smithi047 conmon[78521]: yield 2022-05-14T03:45:46.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:45:46 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:45:46.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:45:46 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:45:46.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:45:46 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:45:46.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:45:46 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T03:45:46.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:45:46 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:45:46.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:45:46 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T03:45:46.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:45:46 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:45:46.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:45:46 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T03:45:46.797 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:45:46 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:45:46.797 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:45:46 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T03:45:46.797 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:45:46 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:45:46.797 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:45:46 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T03:45:46.798 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:45:46 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:45:46.798 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:45:46 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:45:46.798 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:45:46 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:45:46.798 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:45:46 smithi047 conmon[78521]: 2022-05-14T03:45:46.417+0000 7f5341f7c700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T03:45:46.799 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:45:46 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T03:45:46.799 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:45:46 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:45:46.799 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:45:46 smithi047 conmon[78521]: yield 2022-05-14T03:45:46.799 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:45:46 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:45:46.799 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:45:46 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:45:46.800 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:45:46 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:45:46.800 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:45:46 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T03:45:46.800 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:45:46 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:45:46.800 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:45:46 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T03:45:46.801 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:45:46 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:45:46.801 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:45:46 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T03:45:46.801 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:45:46 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:45:46.801 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:45:46 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T03:45:46.802 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:45:46 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:45:46.802 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:45:46 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T03:45:46.802 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:45:46 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:45:46.802 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:45:46 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:45:46.803 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:45:46 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:45:46.803 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:45:46 smithi047 conmon[78521]: 2022-05-14T03:45:46.804 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:45:46 smithi047 conmon[78521]: During handling of the above exception, another exception occurred: 2022-05-14T03:45:46.804 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:45:46 smithi047 conmon[78521]: 2022-05-14T03:45:46.804 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:45:46 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T03:45:46.804 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:45:46 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T03:45:46.805 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:45:46 smithi047 conmon[78521]: return f(*arg) 2022-05-14T03:45:46.805 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:45:46 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T03:45:46.805 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:45:46 smithi047 conmon[78521]: self._write_client_files(client_files, host) 2022-05-14T03:45:46.805 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:45:46 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T03:45:46.805 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:45:46 smithi047 conmon[78521]: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T03:45:46.806 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:45:46 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T03:45:46.806 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:45:46 smithi047 conmon[78521]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T03:45:46.806 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:45:46 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T03:45:46.807 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:45:46 smithi047 conmon[78521]: return self.event_loop.get_result(coro) 2022-05-14T03:45:46.807 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:45:46 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T03:45:46.807 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:45:46 smithi047 conmon[78521]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T03:45:46.808 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:45:46 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T03:45:46.808 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:45:46 smithi047 conmon[78521]: return self.__get_result() 2022-05-14T03:45:46.808 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:45:46 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T03:45:46.808 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:45:46 smithi047 conmon[78521]: raise self._exception 2022-05-14T03:45:46.809 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:45:46 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T03:45:46.809 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:45:46 smithi047 conmon[78521]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T03:45:46.809 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:45:46 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T03:45:46.809 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:45:46 smithi047 conmon[78521]: conn = await self._remote_connection(host, addr) 2022-05-14T03:45:46.810 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:45:46 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T03:45:46.810 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:45:46 smithi047 conmon[78521]: raise 2022-05-14T03:45:46.810 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:45:46 smithi047 conmon[78521]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T03:45:46.810 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:45:46 smithi047 conmon[78521]: self.gen.throw(type, value, traceback) 2022-05-14T03:45:46.810 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:45:46 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T03:45:46.811 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:45:46 smithi047 conmon[78521]: raise OrchestratorError(msg) 2022-05-14T03:45:46.811 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:45:46 smithi047 conmon[78521]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:45:47.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:45:46 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:45:47.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:45:46 smithi047 ceph-mon[77147]: pgmap v8091: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:45:47.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:45:46 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:45:47.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:45:46 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 15480 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:45:47.360 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:45:47 smithi115 conmon[90393]: 2022-05-14T03:45:47.096+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:45:47.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:45:46 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:45:47.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:45:46 smithi115 ceph-mon[78593]: pgmap v8091: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:45:47.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:45:46 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:45:47.362 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:45:46 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 15480 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:45:48.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:45:47 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:45:48.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T03:45:48.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:45:48.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T03:45:48.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:45:48.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:45:48.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:45:48.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T03:45:48.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:45:48.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T03:45:48.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:45:48.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T03:45:48.298 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:45:48.298 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T03:45:48.299 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:45:48.299 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T03:45:48.299 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:45:48.299 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:45:48.300 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:45:48.300 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:45:47 smithi047 ceph-mon[77147]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T03:45:48.300 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T03:45:48.300 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:45:48.300 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T03:45:48.301 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:45:48.301 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:45:48.301 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:45:48.301 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T03:45:48.302 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:45:48.302 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T03:45:48.302 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:45:48.303 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T03:45:48.303 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:45:48.304 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T03:45:48.304 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:45:48.304 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T03:45:48.304 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:45:48.304 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:45:48.305 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:45:48.305 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T03:45:48.305 INFO:journalctl@ceph.mon.b.smithi047.stdout: During handling of the above exception, another exception occurred: 2022-05-14T03:45:48.305 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T03:45:48.306 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T03:45:48.307 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T03:45:48.307 INFO:journalctl@ceph.mon.b.smithi047.stdout: return f(*arg) 2022-05-14T03:45:48.307 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T03:45:48.307 INFO:journalctl@ceph.mon.b.smithi047.stdout: self._write_client_files(client_files, host) 2022-05-14T03:45:48.308 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T03:45:48.308 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T03:45:48.308 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T03:45:48.308 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T03:45:48.309 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T03:45:48.309 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.event_loop.get_result(coro) 2022-05-14T03:45:48.309 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T03:45:48.309 INFO:journalctl@ceph.mon.b.smithi047.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T03:45:48.310 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T03:45:48.310 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.__get_result() 2022-05-14T03:45:48.311 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T03:45:48.311 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise self._exception 2022-05-14T03:45:48.311 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T03:45:48.312 INFO:journalctl@ceph.mon.b.smithi047.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T03:45:48.312 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T03:45:48.312 INFO:journalctl@ceph.mon.b.smithi047.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T03:45:48.312 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T03:45:48.313 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise 2022-05-14T03:45:48.313 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T03:45:48.313 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.gen.throw(type, value, traceback) 2022-05-14T03:45:48.313 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T03:45:48.314 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OrchestratorError(msg) 2022-05-14T03:45:48.314 INFO:journalctl@ceph.mon.b.smithi047.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:45:48.314 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:45:47 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:45:48.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:45:48 smithi115 conmon[90393]: 2022-05-14T03:45:48.111+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:45:48.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:45:47 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:45:48.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T03:45:48.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:45:48.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T03:45:48.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:45:48.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:45:48.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:45:48.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T03:45:48.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:45:48.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T03:45:48.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:45:48.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T03:45:48.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:45:48.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T03:45:48.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:45:48.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T03:45:48.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:45:48.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:45:48.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:45:48.365 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:45:47 smithi115 ceph-mon[78593]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T03:45:48.365 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T03:45:48.365 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:45:48.365 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T03:45:48.366 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:45:48.366 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:45:48.366 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:45:48.366 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T03:45:48.366 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:45:48.367 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T03:45:48.367 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:45:48.367 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T03:45:48.367 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:45:48.368 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T03:45:48.368 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:45:48.368 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T03:45:48.368 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:45:48.368 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:45:48.369 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:45:48.369 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T03:45:48.369 INFO:journalctl@ceph.mon.c.smithi115.stdout: During handling of the above exception, another exception occurred: 2022-05-14T03:45:48.369 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T03:45:48.370 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T03:45:48.370 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T03:45:48.370 INFO:journalctl@ceph.mon.c.smithi115.stdout: return f(*arg) 2022-05-14T03:45:48.370 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T03:45:48.370 INFO:journalctl@ceph.mon.c.smithi115.stdout: self._write_client_files(client_files, host) 2022-05-14T03:45:48.371 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T03:45:48.371 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T03:45:48.371 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T03:45:48.371 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T03:45:48.371 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T03:45:48.372 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.event_loop.get_result(coro) 2022-05-14T03:45:48.372 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T03:45:48.372 INFO:journalctl@ceph.mon.c.smithi115.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T03:45:48.372 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T03:45:48.373 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.__get_result() 2022-05-14T03:45:48.373 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T03:45:48.373 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise self._exception 2022-05-14T03:45:48.373 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T03:45:48.374 INFO:journalctl@ceph.mon.c.smithi115.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T03:45:48.374 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T03:45:48.374 INFO:journalctl@ceph.mon.c.smithi115.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T03:45:48.374 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T03:45:48.377 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise 2022-05-14T03:45:48.378 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T03:45:48.378 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.gen.throw(type, value, traceback) 2022-05-14T03:45:48.378 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T03:45:48.378 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OrchestratorError(msg) 2022-05-14T03:45:48.379 INFO:journalctl@ceph.mon.c.smithi115.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:45:48.379 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:45:47 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:45:49.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:45:48 smithi047 ceph-mon[77147]: pgmap v8092: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:45:49.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:45:48 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:45:49.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:45:49 smithi115 conmon[90393]: 2022-05-14T03:45:49.064+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:45:49.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:45:48 smithi115 ceph-mon[78593]: pgmap v8092: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:45:49.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:45:48 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:45:50.295 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:45:49 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:45:50.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:45:50 smithi115 conmon[90393]: 2022-05-14T03:45:50.063+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:45:50.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:45:49 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:45:51.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:45:50 smithi047 ceph-mon[77147]: pgmap v8093: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:45:51.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:45:50 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:45:51.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:45:50 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:45:51.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:45:50 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:45:51.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:45:51 smithi115 conmon[90393]: 2022-05-14T03:45:51.079+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:45:51.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:45:50 smithi115 ceph-mon[78593]: pgmap v8093: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:45:51.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:45:50 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:45:51.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:45:50 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:45:51.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:45:50 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:45:52.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:45:51 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:45:52.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:45:51 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 15485 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:45:52.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:45:52 smithi115 conmon[90393]: 2022-05-14T03:45:52.096+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:45:52.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:45:51 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:45:52.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:45:51 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 15485 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:45:53.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:45:52 smithi047 ceph-mon[77147]: pgmap v8094: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:45:53.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:45:52 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:45:53.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:45:53 smithi115 conmon[90393]: 2022-05-14T03:45:53.094+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:45:53.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:45:52 smithi115 ceph-mon[78593]: pgmap v8094: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:45:53.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:45:52 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:45:54.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:45:54 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:45:54.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:45:54 smithi115 conmon[90393]: 2022-05-14T03:45:54.139+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:45:54.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:45:54 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:45:55.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:45:55 smithi047 ceph-mon[77147]: pgmap v8095: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:45:55.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:45:55 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:45:55.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:45:55 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:45:55.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:45:55 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:45:55.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:45:55 smithi115 conmon[90393]: 2022-05-14T03:45:55.142+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:45:55.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:45:55 smithi115 ceph-mon[78593]: pgmap v8095: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:45:55.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:45:55 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:45:55.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:45:55 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:45:55.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:45:55 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:45:56.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:45:56 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:45:56.361 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:45:56 smithi115 conmon[90393]: 2022-05-14T03:45:56.175+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:45:56.363 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:45:56 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:45:57.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:45:57 smithi047 ceph-mon[77147]: pgmap v8096: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:45:57.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:45:57 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:45:57.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:45:57 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 15490 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:45:57.360 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:45:57 smithi115 conmon[90393]: 2022-05-14T03:45:57.175+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:45:57.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:45:57 smithi115 ceph-mon[78593]: pgmap v8096: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:45:57.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:45:57 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:45:57.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:45:57 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 15490 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:45:58.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:45:58 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:45:58.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:45:58 smithi115 conmon[90393]: 2022-05-14T03:45:58.208+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:45:58.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:45:58 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:45:59.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:45:59 smithi047 ceph-mon[77147]: pgmap v8097: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:45:59.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:45:59 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:45:59.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:45:59 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:45:59.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:45:59 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:45:59.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:45:59 smithi115 conmon[90393]: 2022-05-14T03:45:59.183+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:45:59.362 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:45:59 smithi115 ceph-mon[78593]: pgmap v8097: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:45:59.363 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:45:59 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:45:59.363 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:45:59 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:45:59.363 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:45:59 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:46:00.296 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:46:00 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:46:00.360 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:46:00 smithi115 conmon[90393]: 2022-05-14T03:46:00.163+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:46:00.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:46:00 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:46:01.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:46:01 smithi047 ceph-mon[77147]: pgmap v8098: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:46:01.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:46:01 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:46:01.362 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:46:01 smithi115 conmon[90393]: 2022-05-14T03:46:01.134+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:46:01.362 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:46:01 smithi115 ceph-mon[78593]: pgmap v8098: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:46:01.364 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:46:01 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:46:02.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:46:02 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:46:02.295 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:46:02 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 15495 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:46:02.366 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:46:02 smithi115 conmon[90393]: 2022-05-14T03:46:02.112+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:46:02.366 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:46:02 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:46:02.366 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:46:02 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 15495 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:46:03.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:46:03 smithi047 ceph-mon[77147]: pgmap v8099: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:46:03.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:46:03 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:46:03.295 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:46:03 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:46:03.295 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:46:03 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:46:03.360 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:46:03 smithi115 conmon[90393]: 2022-05-14T03:46:03.147+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:46:03.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:46:03 smithi115 ceph-mon[78593]: pgmap v8099: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:46:03.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:46:03 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:46:03.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:46:03 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:46:03.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:46:03 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:46:04.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:46:04 smithi115 conmon[90393]: 2022-05-14T03:46:04.154+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:46:04.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:46:04 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:46:04.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:46:04 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:46:05.368 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:46:05 smithi115 conmon[90393]: 2022-05-14T03:46:05.152+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:46:05.368 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:46:05 smithi115 ceph-mon[78593]: pgmap v8100: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:46:05.369 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:46:05 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:46:05.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:46:05 smithi047 ceph-mon[77147]: pgmap v8100: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:46:05.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:46:05 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:46:06.360 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:46:06 smithi115 conmon[90393]: 2022-05-14T03:46:06.193+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:46:06.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:46:06 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:46:06.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:46:06 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:46:07.362 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:46:07 smithi115 conmon[90393]: 2022-05-14T03:46:07.160+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:46:07.363 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:46:07 smithi115 ceph-mon[78593]: pgmap v8101: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:46:07.363 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:46:07 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:46:07.363 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:46:07 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 15500 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:46:07.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:46:07 smithi047 ceph-mon[77147]: pgmap v8101: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:46:07.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:46:07 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:46:07.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:46:07 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 15500 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:46:08.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:46:08 smithi115 conmon[90393]: 2022-05-14T03:46:08.194+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:46:08.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:46:08 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:46:08.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:46:08 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:46:09.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:46:09 smithi115 conmon[90393]: 2022-05-14T03:46:09.227+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:46:09.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:46:09 smithi115 ceph-mon[78593]: pgmap v8102: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:46:09.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:46:09 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:46:09.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:46:09 smithi047 ceph-mon[77147]: pgmap v8102: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:46:09.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:46:09 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:46:10.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:46:10 smithi115 conmon[90393]: 2022-05-14T03:46:10.179+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:46:10.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:46:10 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:46:10.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:46:10 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:46:11.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:46:11 smithi115 conmon[90393]: 2022-05-14T03:46:11.166+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:46:11.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:46:11 smithi115 ceph-mon[78593]: pgmap v8103: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:46:11.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:46:11 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:46:11.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:46:11 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:46:11.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:46:11 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:46:11.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:46:11 smithi047 ceph-mon[77147]: pgmap v8103: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:46:11.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:46:11 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:46:11.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:46:11 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:46:11.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:46:11 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:46:12.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:46:12 smithi115 conmon[90393]: 2022-05-14T03:46:12.160+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:46:12.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:46:12 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:46:12.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:46:12 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 15505 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:46:12.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:46:12 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:46:12.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:46:12 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 15505 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:46:13.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:46:13 smithi115 conmon[90393]: 2022-05-14T03:46:13.166+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:46:13.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:46:13 smithi115 ceph-mon[78593]: pgmap v8104: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:46:13.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:46:13 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:46:13.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:46:13 smithi047 ceph-mon[77147]: pgmap v8104: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:46:13.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:46:13 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:46:14.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:46:14 smithi115 conmon[90393]: 2022-05-14T03:46:14.138+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:46:14.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:46:14 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:46:14.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:46:14 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:46:15.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:46:15 smithi115 conmon[90393]: 2022-05-14T03:46:15.124+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:46:15.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:46:15 smithi115 ceph-mon[78593]: pgmap v8105: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:46:15.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:46:15 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:46:15.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:46:15 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:46:15.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:46:15 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:46:15.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:46:15 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T03:46:15.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:46:15 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T03:46:15.546 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:46:15 smithi047 ceph-mon[77147]: pgmap v8105: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:46:15.546 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:46:15 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:46:15.547 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:46:15 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:46:15.547 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:46:15 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:46:15.547 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:46:15 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T03:46:15.547 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:46:15 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T03:46:16.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:46:16 smithi115 conmon[90393]: 2022-05-14T03:46:16.146+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:46:16.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:46:16 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:46:16.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:46:16 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:46:17.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:46:17 smithi115 conmon[90393]: 2022-05-14T03:46:17.179+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:46:17.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:46:17 smithi115 ceph-mon[78593]: pgmap v8106: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:46:17.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:46:17 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:46:17.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:46:17 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 15510 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:46:17.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:46:17 smithi047 ceph-mon[77147]: pgmap v8106: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:46:17.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:46:17 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:46:17.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:46:17 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 15510 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:46:18.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:46:18 smithi115 conmon[90393]: 2022-05-14T03:46:18.186+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:46:19.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:46:18 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:46:19.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:46:18 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:46:19.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:46:18 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:46:19.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:46:18 smithi115 ceph-mon[78593]: pgmap v8107: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:46:19.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:46:18 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:46:19.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:46:18 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:46:19.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:46:18 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:46:19.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:46:18 smithi047 ceph-mon[77147]: pgmap v8107: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:46:19.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:46:19 smithi115 conmon[90393]: 2022-05-14T03:46:19.218+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:46:20.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:46:19 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:46:20.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:46:19 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:46:20.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:46:20 smithi115 conmon[90393]: 2022-05-14T03:46:20.188+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:46:21.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:46:20 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:46:21.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:46:20 smithi115 ceph-mon[78593]: pgmap v8108: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:46:21.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:46:20 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:46:21.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:46:20 smithi047 ceph-mon[77147]: pgmap v8108: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:46:21.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:46:21 smithi115 conmon[90393]: 2022-05-14T03:46:21.157+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:46:22.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:46:21 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:46:22.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:46:21 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 15515 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:46:22.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:46:21 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:46:22.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:46:21 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 15515 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:46:22.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:46:22 smithi115 conmon[90393]: 2022-05-14T03:46:22.165+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:46:23.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:46:22 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:46:23.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:46:22 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:46:23.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:46:22 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:46:23.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:46:22 smithi115 ceph-mon[78593]: pgmap v8109: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:46:23.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:46:22 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:46:23.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:46:22 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:46:23.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:46:22 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:46:23.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:46:22 smithi047 ceph-mon[77147]: pgmap v8109: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:46:23.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:46:23 smithi115 conmon[90393]: 2022-05-14T03:46:23.215+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:46:24.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:46:23 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:46:24.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:46:23 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:46:24.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:46:24 smithi115 conmon[90393]: 2022-05-14T03:46:24.211+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:46:25.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:46:24 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:46:25.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:46:24 smithi115 ceph-mon[78593]: pgmap v8110: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:46:25.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:46:24 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:46:25.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:46:24 smithi047 ceph-mon[77147]: pgmap v8110: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:46:25.610 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:46:25 smithi115 conmon[90393]: 2022-05-14T03:46:25.207+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:46:26.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:46:25 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:46:26.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:46:25 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:46:26.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:46:26 smithi115 conmon[90393]: 2022-05-14T03:46:26.253+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:46:27.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:46:26 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:46:27.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:46:26 smithi115 ceph-mon[78593]: pgmap v8111: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:46:27.295 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:46:26 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:46:27.295 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:46:26 smithi047 ceph-mon[77147]: pgmap v8111: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:46:27.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:46:27 smithi115 conmon[90393]: 2022-05-14T03:46:27.294+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:46:28.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:46:27 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:46:28.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:46:27 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 15520 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:46:28.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:46:27 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:46:28.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:46:27 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 15520 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:46:28.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:46:28 smithi115 conmon[90393]: 2022-05-14T03:46:28.297+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:46:29.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:46:28 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:46:29.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:46:28 smithi115 ceph-mon[78593]: pgmap v8112: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:46:29.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:46:28 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:46:29.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:46:28 smithi047 ceph-mon[77147]: pgmap v8112: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:46:29.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:46:29 smithi115 conmon[90393]: 2022-05-14T03:46:29.253+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:46:30.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:46:29 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:46:30.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:46:29 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:46:30.610 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:46:30 smithi115 conmon[90393]: 2022-05-14T03:46:30.250+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:46:31.209 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:46:30 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:46:31.209 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:46:30 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:46:31.210 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:46:30 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:46:31.210 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:46:30 smithi115 ceph-mon[78593]: pgmap v8113: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:46:31.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:46:30 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:46:31.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:46:30 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:46:31.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:46:30 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:46:31.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:46:30 smithi047 ceph-mon[77147]: pgmap v8113: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:46:31.610 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:46:31 smithi115 conmon[90393]: 2022-05-14T03:46:31.206+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:46:32.206 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:46:31 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:46:32.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:46:31 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:46:32.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:46:32 smithi115 conmon[90393]: 2022-05-14T03:46:32.203+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:46:33.225 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:46:32 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:46:33.226 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:46:32 smithi115 ceph-mon[78593]: pgmap v8114: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:46:33.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:46:32 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:46:33.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:46:32 smithi047 ceph-mon[77147]: pgmap v8114: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:46:33.610 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:46:33 smithi115 conmon[90393]: 2022-05-14T03:46:33.222+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:46:34.181 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:46:33 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:46:34.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:46:33 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:46:34.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:46:34 smithi115 conmon[90393]: 2022-05-14T03:46:34.178+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:46:35.164 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:46:34 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:46:35.164 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:46:34 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:46:35.164 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:46:34 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:46:35.165 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:46:34 smithi115 ceph-mon[78593]: pgmap v8115: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:46:35.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:46:34 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:46:35.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:46:34 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:46:35.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:46:34 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:46:35.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:46:34 smithi047 ceph-mon[77147]: pgmap v8115: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:46:35.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:46:35 smithi115 conmon[90393]: 2022-05-14T03:46:35.160+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:46:36.154 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:46:35 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:46:36.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:46:35 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:46:36.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:46:36 smithi115 conmon[90393]: 2022-05-14T03:46:36.150+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:46:37.150 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:46:36 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:46:37.150 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:46:36 smithi115 ceph-mon[78593]: pgmap v8116: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:46:37.151 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:46:36 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 15525 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:46:37.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:46:36 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:46:37.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:46:36 smithi047 ceph-mon[77147]: pgmap v8116: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:46:37.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:46:36 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 15525 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:46:37.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:46:37 smithi115 conmon[90393]: 2022-05-14T03:46:37.147+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:46:38.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:46:37 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:46:38.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:46:38 smithi115 conmon[90393]: 2022-05-14T03:46:38.130+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:46:38.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:46:37 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:46:39.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:46:38 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:46:39.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:46:38 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:46:39.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:46:38 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:46:39.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:46:38 smithi047 ceph-mon[77147]: pgmap v8117: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:46:39.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:46:39 smithi115 conmon[90393]: 2022-05-14T03:46:39.126+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:46:39.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:46:38 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:46:39.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:46:38 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:46:39.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:46:38 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:46:39.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:46:38 smithi115 ceph-mon[78593]: pgmap v8117: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:46:40.156 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:46:39 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:46:40.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:46:39 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:46:40.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:46:40 smithi115 conmon[90393]: 2022-05-14T03:46:40.153+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:46:41.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:46:40 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:46:41.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:46:40 smithi047 ceph-mon[77147]: pgmap v8118: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:46:41.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:46:41 smithi115 conmon[90393]: 2022-05-14T03:46:41.129+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:46:41.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:46:40 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:46:41.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:46:40 smithi115 ceph-mon[78593]: pgmap v8118: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:46:42.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:46:41 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:46:42.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:46:41 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 15535 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:46:42.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:46:42 smithi115 conmon[90393]: 2022-05-14T03:46:42.156+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:46:42.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:46:41 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:46:42.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:46:41 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 15535 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:46:43.207 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:46:42 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:46:43.207 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:46:42 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:46:43.207 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:46:42 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:46:43.208 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:46:42 smithi115 ceph-mon[78593]: pgmap v8119: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:46:43.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:46:42 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:46:43.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:46:42 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:46:43.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:46:42 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:46:43.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:46:42 smithi047 ceph-mon[77147]: pgmap v8119: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:46:43.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:46:43 smithi115 conmon[90393]: 2022-05-14T03:46:43.204+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:46:44.230 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:46:43 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:46:44.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:46:43 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:46:44.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:46:44 smithi115 conmon[90393]: 2022-05-14T03:46:44.227+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:46:45.213 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:46:44 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:46:45.214 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:46:44 smithi115 ceph-mon[78593]: pgmap v8120: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:46:45.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:46:44 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:46:45.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:46:44 smithi047 ceph-mon[77147]: pgmap v8120: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:46:45.610 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:46:45 smithi115 conmon[90393]: 2022-05-14T03:46:45.210+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:46:46.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:46:45 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:46:46.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:46:46 smithi115 conmon[90393]: 2022-05-14T03:46:46.163+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:46:46.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:46:45 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:46:47.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:46:46 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:46:47.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:46:46 smithi047 ceph-mon[77147]: pgmap v8121: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:46:47.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:46:46 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T03:46:47.295 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:46:46 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T03:46:47.295 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:46:46 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T03:46:47.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:46:46 smithi047 conmon[78521]: 2022-05-14T03:46:46.895+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:46:47.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:46:46 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T03:46:47.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:46:46 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:46:47.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:46:46 smithi047 conmon[78521]: yield 2022-05-14T03:46:47.297 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:46:46 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:46:47.297 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:46:46 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:46:47.297 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:46:46 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:46:47.298 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:46:46 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T03:46:47.298 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:46:46 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:46:47.298 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:46:46 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T03:46:47.298 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:46:46 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:46:47.298 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:46:46 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T03:46:47.299 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:46:46 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:46:47.299 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:46:46 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T03:46:47.299 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:46:46 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:46:47.299 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:46:46 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T03:46:47.300 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:46:46 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:46:47.300 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:46:46 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:46:47.300 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:46:46 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:46:47.361 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:46:47 smithi115 conmon[90393]: 2022-05-14T03:46:47.117+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:46:47.362 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:46:46 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:46:47.362 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:46:46 smithi115 ceph-mon[78593]: pgmap v8121: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:46:47.362 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:46:46 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T03:46:47.363 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:46:46 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T03:46:47.363 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:46:46 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T03:46:48.183 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:46:47 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:46:48.183 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:46:47 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:46:48.183 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T03:46:48.184 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:46:48.184 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T03:46:48.184 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:46:48.184 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:46:48.185 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:46:48.185 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T03:46:48.185 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:46:48.185 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T03:46:48.185 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:46:48.186 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T03:46:48.186 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:46:48.186 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T03:46:48.186 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:46:48.187 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T03:46:48.187 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:46:48.187 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:46:48.187 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:46:48.187 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:46:47 smithi047 ceph-mon[77147]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T03:46:48.188 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:46:47 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 15540 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:46:48.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:46:48 smithi115 conmon[90393]: 2022-05-14T03:46:48.161+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:46:48.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:46:47 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:46:48.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:46:47 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:46:48.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T03:46:48.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:46:48.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T03:46:48.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:46:48.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:46:48.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:46:48.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T03:46:48.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:46:48.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T03:46:48.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:46:48.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T03:46:48.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:46:48.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T03:46:48.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:46:48.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T03:46:48.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:46:48.365 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:46:48.365 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:46:48.365 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:46:47 smithi115 ceph-mon[78593]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T03:46:48.365 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:46:47 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 15540 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:46:49.283 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:46:49 smithi115 conmon[90393]: 2022-05-14T03:46:49.147+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:46:49.283 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:46:48 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:46:49.284 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:46:48 smithi115 ceph-mon[78593]: pgmap v8122: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:46:49.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:46:48 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:46:49.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:46:48 smithi047 ceph-mon[77147]: pgmap v8122: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:46:50.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:46:49 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:46:50.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:46:49 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' 2022-05-14T03:46:50.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:46:49 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' 2022-05-14T03:46:50.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:46:49 smithi047 conmon[78521]: 2022-05-14T03:46:49.967+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:46:50.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:46:49 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T03:46:50.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:46:49 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:46:50.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:46:49 smithi047 conmon[78521]: yield 2022-05-14T03:46:50.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:46:49 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:46:50.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:46:49 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:46:50.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:46:49 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:46:50.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:46:49 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T03:46:50.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:46:49 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:46:50.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:46:49 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T03:46:50.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:46:49 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:46:50.297 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:46:49 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T03:46:50.297 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:46:49 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:46:50.297 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:46:49 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T03:46:50.297 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:46:49 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:46:50.298 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:46:49 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T03:46:50.298 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:46:49 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:46:50.298 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:46:49 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:46:50.298 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:46:49 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:46:50.299 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:46:49 smithi047 conmon[78521]: 2022-05-14T03:46:49.968+0000 7f5344f82700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T03:46:50.300 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:46:49 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T03:46:50.300 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:46:49 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:46:50.300 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:46:49 smithi047 conmon[78521]: yield 2022-05-14T03:46:50.300 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:46:49 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:46:50.300 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:46:49 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:46:50.301 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:46:49 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:46:50.301 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:46:49 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T03:46:50.301 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:46:49 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:46:50.301 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:46:49 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T03:46:50.302 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:46:49 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:46:50.302 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:46:49 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T03:46:50.302 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:46:49 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:46:50.302 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:46:49 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T03:46:50.303 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:46:49 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:46:50.303 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:46:49 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T03:46:50.303 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:46:49 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:46:50.303 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:46:49 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:46:50.304 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:46:49 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:46:50.304 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:46:49 smithi047 conmon[78521]: 2022-05-14T03:46:50.304 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:46:49 smithi047 conmon[78521]: During handling of the above exception, another exception occurred: 2022-05-14T03:46:50.304 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:46:49 smithi047 conmon[78521]: 2022-05-14T03:46:50.305 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:46:49 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T03:46:50.305 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:46:49 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T03:46:50.305 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:46:49 smithi047 conmon[78521]: return f(*arg) 2022-05-14T03:46:50.305 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:46:49 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T03:46:50.305 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:46:49 smithi047 conmon[78521]: self._write_client_files(client_files, host) 2022-05-14T03:46:50.306 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:46:49 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T03:46:50.306 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:46:49 smithi047 conmon[78521]: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T03:46:50.306 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:46:49 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T03:46:50.306 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:46:49 smithi047 conmon[78521]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T03:46:50.307 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:46:49 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T03:46:50.307 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:46:49 smithi047 conmon[78521]: return self.event_loop.get_result(coro) 2022-05-14T03:46:50.307 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:46:49 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T03:46:50.307 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:46:49 smithi047 conmon[78521]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T03:46:50.307 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:46:49 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T03:46:50.308 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:46:49 smithi047 conmon[78521]: return self.__get_result() 2022-05-14T03:46:50.308 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:46:49 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T03:46:50.308 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:46:49 smithi047 conmon[78521]: raise self._exception 2022-05-14T03:46:50.308 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:46:49 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T03:46:50.309 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:46:49 smithi047 conmon[78521]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T03:46:50.309 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:46:49 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T03:46:50.309 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:46:49 smithi047 conmon[78521]: conn = await self._remote_connection(host, addr) 2022-05-14T03:46:50.309 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:46:49 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T03:46:50.310 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:46:49 smithi047 conmon[78521]: raise 2022-05-14T03:46:50.310 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:46:49 smithi047 conmon[78521]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T03:46:50.310 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:46:49 smithi047 conmon[78521]: self.gen.throw(type, value, traceback) 2022-05-14T03:46:50.311 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:46:49 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T03:46:50.311 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:46:49 smithi047 conmon[78521]: raise OrchestratorError(msg) 2022-05-14T03:46:50.311 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:46:49 smithi047 conmon[78521]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:46:50.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:46:50 smithi115 conmon[90393]: 2022-05-14T03:46:50.152+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:46:50.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:46:49 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:46:50.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:46:49 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' 2022-05-14T03:46:50.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:46:49 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' 2022-05-14T03:46:51.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:46:50 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:46:51.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:46:50 smithi047 ceph-mon[77147]: Detected new or changed devices on smithi115 2022-05-14T03:46:51.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:46:50 smithi047 ceph-mon[77147]: Detected new or changed devices on smithi047 2022-05-14T03:46:51.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:46:50 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:46:51.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:46:50 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:46:51.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:46:50 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:46:51.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T03:46:51.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:46:51.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T03:46:51.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:46:51.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:46:51.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:46:51.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T03:46:51.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:46:51.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T03:46:51.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:46:51.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T03:46:51.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:46:51.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T03:46:51.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:46:51.298 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T03:46:51.298 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:46:51.298 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:46:51.298 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:46:51.298 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:46:50 smithi047 ceph-mon[77147]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T03:46:51.299 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T03:46:51.299 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:46:51.299 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T03:46:51.299 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:46:51.300 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:46:51.300 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:46:51.300 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T03:46:51.300 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:46:51.300 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T03:46:51.301 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:46:51.301 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T03:46:51.301 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:46:51.301 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T03:46:51.302 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:46:51.302 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T03:46:51.302 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:46:51.302 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:46:51.302 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:46:51.303 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T03:46:51.303 INFO:journalctl@ceph.mon.b.smithi047.stdout: During handling of the above exception, another exception occurred: 2022-05-14T03:46:51.303 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T03:46:51.303 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T03:46:51.304 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T03:46:51.304 INFO:journalctl@ceph.mon.b.smithi047.stdout: return f(*arg) 2022-05-14T03:46:51.304 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T03:46:51.304 INFO:journalctl@ceph.mon.b.smithi047.stdout: self._write_client_files(client_files, host) 2022-05-14T03:46:51.304 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T03:46:51.305 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T03:46:51.305 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T03:46:51.305 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T03:46:51.305 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T03:46:51.305 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.event_loop.get_result(coro) 2022-05-14T03:46:51.306 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T03:46:51.306 INFO:journalctl@ceph.mon.b.smithi047.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T03:46:51.306 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T03:46:51.306 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.__get_result() 2022-05-14T03:46:51.307 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T03:46:51.307 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise self._exception 2022-05-14T03:46:51.307 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T03:46:51.307 INFO:journalctl@ceph.mon.b.smithi047.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T03:46:51.307 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T03:46:51.308 INFO:journalctl@ceph.mon.b.smithi047.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T03:46:51.308 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T03:46:51.308 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise 2022-05-14T03:46:51.308 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T03:46:51.309 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.gen.throw(type, value, traceback) 2022-05-14T03:46:51.309 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T03:46:51.310 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OrchestratorError(msg) 2022-05-14T03:46:51.310 INFO:journalctl@ceph.mon.b.smithi047.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:46:51.310 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:46:50 smithi047 ceph-mon[77147]: pgmap v8123: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:46:51.360 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:46:51 smithi115 conmon[90393]: 2022-05-14T03:46:51.129+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:46:51.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:46:50 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:46:51.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:46:50 smithi115 ceph-mon[78593]: Detected new or changed devices on smithi115 2022-05-14T03:46:51.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:46:50 smithi115 ceph-mon[78593]: Detected new or changed devices on smithi047 2022-05-14T03:46:51.362 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:46:50 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:46:51.362 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:46:50 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:46:51.362 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:46:50 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:46:51.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T03:46:51.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:46:51.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T03:46:51.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:46:51.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:46:51.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:46:51.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T03:46:51.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:46:51.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T03:46:51.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:46:51.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T03:46:51.365 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:46:51.365 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T03:46:51.365 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:46:51.365 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T03:46:51.366 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:46:51.366 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:46:51.366 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:46:51.366 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:46:50 smithi115 ceph-mon[78593]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T03:46:51.366 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T03:46:51.367 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:46:51.367 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T03:46:51.367 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:46:51.367 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:46:51.367 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:46:51.368 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T03:46:51.368 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:46:51.368 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T03:46:51.368 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:46:51.369 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T03:46:51.370 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:46:51.370 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T03:46:51.371 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:46:51.371 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T03:46:51.371 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:46:51.372 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:46:51.372 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:46:51.373 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T03:46:51.373 INFO:journalctl@ceph.mon.c.smithi115.stdout: During handling of the above exception, another exception occurred: 2022-05-14T03:46:51.373 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T03:46:51.374 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T03:46:51.374 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T03:46:51.374 INFO:journalctl@ceph.mon.c.smithi115.stdout: return f(*arg) 2022-05-14T03:46:51.375 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T03:46:51.375 INFO:journalctl@ceph.mon.c.smithi115.stdout: self._write_client_files(client_files, host) 2022-05-14T03:46:51.376 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T03:46:51.376 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T03:46:51.376 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T03:46:51.377 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T03:46:51.377 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T03:46:51.377 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.event_loop.get_result(coro) 2022-05-14T03:46:51.378 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T03:46:51.378 INFO:journalctl@ceph.mon.c.smithi115.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T03:46:51.378 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T03:46:51.378 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.__get_result() 2022-05-14T03:46:51.378 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T03:46:51.379 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise self._exception 2022-05-14T03:46:51.379 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T03:46:51.379 INFO:journalctl@ceph.mon.c.smithi115.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T03:46:51.380 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T03:46:51.380 INFO:journalctl@ceph.mon.c.smithi115.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T03:46:51.380 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T03:46:51.381 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise 2022-05-14T03:46:51.381 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T03:46:51.382 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.gen.throw(type, value, traceback) 2022-05-14T03:46:51.382 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T03:46:51.382 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OrchestratorError(msg) 2022-05-14T03:46:51.382 INFO:journalctl@ceph.mon.c.smithi115.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:46:51.383 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:46:50 smithi115 ceph-mon[78593]: pgmap v8123: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:46:52.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:46:51 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:46:52.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:46:51 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:46:52.360 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:46:52 smithi115 conmon[90393]: 2022-05-14T03:46:52.108+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:46:53.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:46:52 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:46:53.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:46:52 smithi047 ceph-mon[77147]: pgmap v8124: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:46:53.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:46:53 smithi115 conmon[90393]: 2022-05-14T03:46:53.083+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:46:53.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:46:52 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:46:53.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:46:52 smithi115 ceph-mon[78593]: pgmap v8124: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:46:54.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:46:53 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:46:54.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:46:53 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:46:54.360 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:46:54 smithi115 conmon[90393]: 2022-05-14T03:46:54.060+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:46:55.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:46:54 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:46:55.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:46:54 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:46:55.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:46:54 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:46:55.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:46:54 smithi047 ceph-mon[77147]: pgmap v8125: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:46:55.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:46:55 smithi115 conmon[90393]: 2022-05-14T03:46:55.012+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:46:55.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:46:54 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:46:55.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:46:54 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:46:55.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:46:54 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:46:55.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:46:54 smithi115 ceph-mon[78593]: pgmap v8125: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:46:56.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:46:55 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:46:56.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:46:55 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:46:56.360 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:46:56 smithi115 conmon[90393]: 2022-05-14T03:46:56.044+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:46:57.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:46:56 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:46:57.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:46:56 smithi047 ceph-mon[77147]: pgmap v8126: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:46:57.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:46:56 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 15545 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:46:57.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:46:57 smithi115 conmon[90393]: 2022-05-14T03:46:57.009+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:46:57.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:46:56 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:46:57.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:46:56 smithi115 ceph-mon[78593]: pgmap v8126: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:46:57.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:46:56 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 15545 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:46:58.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:46:57 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:46:58.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:46:57 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:46:58.360 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:46:58 smithi115 conmon[90393]: 2022-05-14T03:46:58.034+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:46:59.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:46:58 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:46:59.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:46:58 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:46:59.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:46:58 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:46:59.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:46:58 smithi047 ceph-mon[77147]: pgmap v8127: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:46:59.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:46:59 smithi115 conmon[90393]: 2022-05-14T03:46:59.028+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:46:59.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:46:58 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:46:59.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:46:58 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:46:59.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:46:58 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:46:59.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:46:58 smithi115 ceph-mon[78593]: pgmap v8127: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:47:00.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:46:59 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:47:00.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:46:59 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:47:00.360 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:47:00 smithi115 conmon[90393]: 2022-05-14T03:47:00.015+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:47:01.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:47:00 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:47:01.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:47:00 smithi047 ceph-mon[77147]: pgmap v8128: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:47:01.361 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:47:00 smithi115 conmon[90393]: 2022-05-14T03:47:00.995+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:47:01.362 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:47:00 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:47:01.363 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:47:00 smithi115 ceph-mon[78593]: pgmap v8128: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:47:02.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:47:01 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:47:02.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:47:01 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:47:02.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:47:01 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 15555 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:47:02.360 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:47:01 smithi115 conmon[90393]: 2022-05-14T03:47:01.959+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:47:02.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:47:01 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:47:02.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:47:01 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:47:02.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:47:01 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 15555 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:47:03.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:47:02 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:47:03.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:47:02 smithi047 ceph-mon[77147]: pgmap v8129: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:47:03.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:47:02 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:47:03.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:47:02 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:47:03.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:47:02 smithi115 conmon[90393]: 2022-05-14T03:47:02.947+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:47:03.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:47:02 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:47:03.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:47:02 smithi115 ceph-mon[78593]: pgmap v8129: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:47:03.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:47:02 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:47:03.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:47:02 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:47:04.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:47:03 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:47:04.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:47:03 smithi115 conmon[90393]: 2022-05-14T03:47:03.906+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:47:04.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:47:03 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:47:05.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:47:04 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:47:05.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:47:04 smithi047 ceph-mon[77147]: pgmap v8130: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:47:05.360 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:47:04 smithi115 conmon[90393]: 2022-05-14T03:47:04.909+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:47:05.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:47:04 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:47:05.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:47:04 smithi115 ceph-mon[78593]: pgmap v8130: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:47:06.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:47:06 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:47:06.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:47:05 smithi115 conmon[90393]: 2022-05-14T03:47:05.906+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:47:06.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:47:06 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:47:07.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:47:07 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:47:07.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:47:07 smithi047 ceph-mon[77147]: pgmap v8131: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:47:07.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:47:07 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 15560 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:47:07.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:47:07 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:47:07.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:47:06 smithi115 conmon[90393]: 2022-05-14T03:47:06.939+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:47:07.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:47:07 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:47:07.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:47:07 smithi115 ceph-mon[78593]: pgmap v8131: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:47:07.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:47:07 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 15560 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:47:07.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:47:07 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:47:08.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:47:08 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:47:08.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:47:07 smithi115 conmon[90393]: 2022-05-14T03:47:07.922+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:47:08.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:47:08 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:47:09.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:47:09 smithi047 ceph-mon[77147]: pgmap v8132: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:47:09.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:47:09 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:47:09.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:47:08 smithi115 conmon[90393]: 2022-05-14T03:47:08.948+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:47:09.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:47:09 smithi115 ceph-mon[78593]: pgmap v8132: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:47:09.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:47:09 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:47:10.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:47:10 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:47:10.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:47:09 smithi115 conmon[90393]: 2022-05-14T03:47:09.921+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:47:10.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:47:10 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:47:11.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:47:11 smithi047 ceph-mon[77147]: pgmap v8133: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:47:11.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:47:11 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:47:11.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:47:11 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:47:11.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:47:11 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:47:11.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:47:10 smithi115 conmon[90393]: 2022-05-14T03:47:10.967+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:47:11.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:47:11 smithi115 ceph-mon[78593]: pgmap v8133: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:47:11.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:47:11 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:47:11.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:47:11 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:47:11.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:47:11 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:47:12.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:47:12 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 15565 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:47:12.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:47:12 smithi115 conmon[90393]: 2022-05-14T03:47:12.005+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:47:12.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:47:12 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 15565 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:47:13.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:47:13 smithi047 ceph-mon[77147]: pgmap v8134: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:47:13.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:47:13 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:47:13.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:47:13 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:47:13.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:47:12 smithi115 conmon[90393]: 2022-05-14T03:47:12.976+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:47:13.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:47:13 smithi115 ceph-mon[78593]: pgmap v8134: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:47:13.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:47:13 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:47:13.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:47:13 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:47:14.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:47:14 smithi115 conmon[90393]: 2022-05-14T03:47:14.015+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:47:15.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:47:15 smithi047 ceph-mon[77147]: pgmap v8135: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:47:15.295 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:47:15 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:47:15.295 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:47:15 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:47:15.295 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:47:15 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:47:15.295 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:47:15 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T03:47:15.295 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:47:15 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T03:47:15.296 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:47:15 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:47:15.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:47:14 smithi115 conmon[90393]: 2022-05-14T03:47:14.981+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:47:15.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:47:15 smithi115 ceph-mon[78593]: pgmap v8135: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:47:15.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:47:15 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:47:15.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:47:15 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:47:15.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:47:15 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:47:15.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:47:15 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T03:47:15.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:47:15 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T03:47:15.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:47:15 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:47:16.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:47:16 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:47:16.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:47:15 smithi115 conmon[90393]: 2022-05-14T03:47:15.944+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:47:16.366 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:47:16 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:47:17.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:47:17 smithi047 ceph-mon[77147]: pgmap v8136: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:47:17.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:47:17 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 15570 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:47:17.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:47:17 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:47:17.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:47:16 smithi115 conmon[90393]: 2022-05-14T03:47:16.995+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:47:17.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:47:17 smithi115 ceph-mon[78593]: pgmap v8136: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:47:17.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:47:17 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 15570 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:47:17.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:47:17 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:47:18.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:47:18 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:47:18.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:47:17 smithi115 conmon[90393]: 2022-05-14T03:47:17.950+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:47:18.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:47:18 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:47:19.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:47:19 smithi047 ceph-mon[77147]: pgmap v8137: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:47:19.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:47:19 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:47:19.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:47:19 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:47:19.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:47:19 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:47:19.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:47:18 smithi115 conmon[90393]: 2022-05-14T03:47:18.934+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:47:19.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:47:19 smithi115 ceph-mon[78593]: pgmap v8137: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:47:19.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:47:19 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:47:19.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:47:19 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:47:19.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:47:19 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:47:20.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:47:19 smithi115 conmon[90393]: 2022-05-14T03:47:19.917+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:47:20.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:47:20 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:47:20.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:47:20 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:47:21.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:47:20 smithi115 conmon[90393]: 2022-05-14T03:47:20.936+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:47:21.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:47:21 smithi115 ceph-mon[78593]: pgmap v8138: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:47:21.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:47:21 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:47:21.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:47:21 smithi047 ceph-mon[77147]: pgmap v8138: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:47:21.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:47:21 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:47:22.363 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:47:21 smithi115 conmon[90393]: 2022-05-14T03:47:21.931+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:47:22.364 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:47:22 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 15575 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:47:22.365 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:47:22 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:47:22.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:47:22 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 15575 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:47:22.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:47:22 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:47:23.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:47:22 smithi115 conmon[90393]: 2022-05-14T03:47:22.894+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:47:23.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:47:23 smithi115 ceph-mon[78593]: pgmap v8139: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:47:23.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:47:23 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:47:23.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:47:23 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:47:23.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:47:23 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:47:23.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:47:23 smithi047 ceph-mon[77147]: pgmap v8139: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:47:23.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:47:23 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:47:23.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:47:23 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:47:23.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:47:23 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:47:24.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:47:23 smithi115 conmon[90393]: 2022-05-14T03:47:23.868+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:47:24.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:47:24 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:47:24.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:47:24 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:47:25.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:47:24 smithi115 conmon[90393]: 2022-05-14T03:47:24.861+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:47:25.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:47:25 smithi115 ceph-mon[78593]: pgmap v8140: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:47:25.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:47:25 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:47:25.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:47:25 smithi047 ceph-mon[77147]: pgmap v8140: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:47:25.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:47:25 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:47:26.111 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:47:25 smithi115 conmon[90393]: 2022-05-14T03:47:25.848+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:47:26.112 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:47:26 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:47:26.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:47:26 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:47:27.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:47:26 smithi115 conmon[90393]: 2022-05-14T03:47:26.823+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:47:27.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:47:27 smithi115 ceph-mon[78593]: pgmap v8141: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:47:27.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:47:27 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 15580 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:47:27.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:47:27 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:47:27.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:47:27 smithi047 ceph-mon[77147]: pgmap v8141: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:47:27.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:47:27 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 15580 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:47:27.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:47:27 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:47:28.074 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:47:27 smithi115 conmon[90393]: 2022-05-14T03:47:27.773+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:47:28.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:47:28 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:47:28.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:47:28 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:47:29.075 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:47:28 smithi115 conmon[90393]: 2022-05-14T03:47:28.786+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:47:29.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:47:29 smithi115 ceph-mon[78593]: pgmap v8142: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:47:29.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:47:29 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:47:29.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:47:29 smithi047 ceph-mon[77147]: pgmap v8142: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:47:29.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:47:29 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:47:30.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:47:29 smithi115 conmon[90393]: 2022-05-14T03:47:29.791+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:47:30.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:47:30 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:47:30.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:47:30 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:47:30.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:47:30 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:47:30.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:47:30 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:47:30.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:47:30 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:47:30.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:47:30 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:47:31.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:47:30 smithi115 conmon[90393]: 2022-05-14T03:47:30.779+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:47:31.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:47:31 smithi047 ceph-mon[77147]: pgmap v8143: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:47:31.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:47:31 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:47:31.614 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:47:31 smithi115 ceph-mon[78593]: pgmap v8143: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:47:31.614 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:47:31 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:47:32.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:47:31 smithi115 conmon[90393]: 2022-05-14T03:47:31.825+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:47:32.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:47:32 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 15585 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:47:32.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:47:32 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:47:32.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:47:32 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 15585 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:47:32.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:47:32 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:47:33.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:47:32 smithi115 conmon[90393]: 2022-05-14T03:47:32.825+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:47:33.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:47:33 smithi047 ceph-mon[77147]: pgmap v8144: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:47:33.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:47:33 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:47:33.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:47:33 smithi115 ceph-mon[78593]: pgmap v8144: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:47:33.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:47:33 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:47:34.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:47:33 smithi115 conmon[90393]: 2022-05-14T03:47:33.861+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:47:35.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:47:34 smithi115 conmon[90393]: 2022-05-14T03:47:34.881+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:47:35.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:47:34 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:47:35.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:47:34 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:47:35.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:47:34 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:47:35.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:47:34 smithi115 ceph-mon[78593]: pgmap v8145: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:47:35.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:47:34 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:47:35.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:47:34 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:47:35.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:47:34 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:47:35.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:47:34 smithi047 ceph-mon[77147]: pgmap v8145: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:47:36.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:47:35 smithi115 conmon[90393]: 2022-05-14T03:47:35.893+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:47:36.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:47:35 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:47:36.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:47:35 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:47:37.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:47:36 smithi115 conmon[90393]: 2022-05-14T03:47:36.907+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:47:37.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:47:36 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:47:37.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:47:36 smithi115 ceph-mon[78593]: pgmap v8146: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:47:37.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:47:36 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:47:37.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:47:36 smithi047 ceph-mon[77147]: pgmap v8146: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:47:38.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:47:37 smithi115 conmon[90393]: 2022-05-14T03:47:37.871+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:47:38.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:47:37 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 15590 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:47:38.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:47:37 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:47:38.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:47:37 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 15590 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:47:38.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:47:37 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:47:39.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:47:38 smithi115 conmon[90393]: 2022-05-14T03:47:38.880+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:47:39.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:47:38 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:47:39.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:47:38 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:47:39.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:47:38 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:47:39.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:47:38 smithi115 ceph-mon[78593]: pgmap v8147: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:47:39.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:47:38 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:47:39.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:47:38 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:47:39.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:47:38 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:47:39.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:47:38 smithi047 ceph-mon[77147]: pgmap v8147: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:47:40.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:47:39 smithi115 conmon[90393]: 2022-05-14T03:47:39.832+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:47:40.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:47:39 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:47:40.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:47:39 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:47:41.110 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:47:40 smithi115 conmon[90393]: 2022-05-14T03:47:40.871+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:47:41.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:47:40 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:47:41.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:47:40 smithi115 ceph-mon[78593]: pgmap v8148: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:47:41.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:47:40 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:47:41.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:47:40 smithi047 ceph-mon[77147]: pgmap v8148: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:47:42.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:47:41 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:47:42.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:47:41 smithi115 conmon[90393]: 2022-05-14T03:47:41.892+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:47:42.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:47:41 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:47:43.110 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:47:42 smithi115 conmon[90393]: 2022-05-14T03:47:42.843+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:47:43.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:47:42 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:47:43.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:47:42 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:47:43.112 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:47:42 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:47:43.112 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:47:42 smithi115 ceph-mon[78593]: pgmap v8149: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:47:43.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:47:42 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:47:43.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:47:42 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:47:43.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:47:42 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:47:43.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:47:42 smithi047 ceph-mon[77147]: pgmap v8149: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:47:44.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:47:43 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:47:44.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:47:43 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:47:44.360 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:47:43 smithi115 conmon[90393]: 2022-05-14T03:47:43.882+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:47:45.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:47:44 smithi115 conmon[90393]: 2022-05-14T03:47:44.848+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:47:45.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:47:44 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:47:45.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:47:44 smithi115 ceph-mon[78593]: pgmap v8150: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:47:45.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:47:44 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:47:45.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:47:44 smithi047 ceph-mon[77147]: pgmap v8150: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:47:46.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:47:45 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:47:46.110 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:47:45 smithi115 conmon[90393]: 2022-05-14T03:47:45.848+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:47:46.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:47:45 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:47:47.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:47:46 smithi115 conmon[90393]: 2022-05-14T03:47:46.849+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:47:47.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:47:46 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:47:47.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:47:46 smithi115 ceph-mon[78593]: pgmap v8151: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:47:47.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:47:46 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 15595 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:47:47.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:47:46 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:47:47.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:47:46 smithi047 ceph-mon[77147]: pgmap v8151: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:47:47.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:47:46 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 15595 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:47:48.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:47:47 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:47:48.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:47:47 smithi115 conmon[90393]: 2022-05-14T03:47:47.868+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:47:48.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:47:47 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:47:49.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:47:48 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:47:49.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:47:48 smithi047 ceph-mon[77147]: pgmap v8152: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:47:49.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:47:48 smithi115 conmon[90393]: 2022-05-14T03:47:48.886+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:47:49.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:47:48 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:47:49.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:47:48 smithi115 ceph-mon[78593]: pgmap v8152: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:47:50.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:47:49 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:47:50.360 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:47:49 smithi115 conmon[90393]: 2022-05-14T03:47:49.930+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:47:50.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:47:49 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:47:51.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:47:50 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:47:51.297 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:47:50 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:47:51.298 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:47:50 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:47:51.298 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:47:50 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T03:47:51.299 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:47:50 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T03:47:51.299 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:47:50 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T03:47:51.300 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:47:50 smithi047 ceph-mon[77147]: pgmap v8153: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:47:51.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:47:50 smithi115 conmon[90393]: 2022-05-14T03:47:50.907+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:47:51.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:47:50 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:47:51.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:47:50 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:47:51.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:47:50 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:47:51.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:47:50 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T03:47:51.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:47:50 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T03:47:51.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:47:50 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T03:47:51.362 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:47:50 smithi115 ceph-mon[78593]: pgmap v8153: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:47:52.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:47:51 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:47:52.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:47:51 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 15605 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:47:52.360 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:47:51 smithi115 conmon[90393]: 2022-05-14T03:47:51.938+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:47:52.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:47:51 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:47:52.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:47:51 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 15605 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:47:53.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:47:52 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:47:53.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:47:52 smithi047 ceph-mon[77147]: pgmap v8154: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:47:53.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:47:53 smithi047 conmon[78521]: 2022-05-14T03:47:53.072+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:47:53.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:47:53 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T03:47:53.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:47:53 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:47:53.297 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:47:53 smithi047 conmon[78521]: yield 2022-05-14T03:47:53.298 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:47:53 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:47:53.298 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:47:53 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:47:53.299 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:47:53 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:47:53.299 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:47:53 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T03:47:53.299 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:47:53 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:47:53.300 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:47:53 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T03:47:53.300 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:47:53 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:47:53.300 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:47:53 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T03:47:53.301 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:47:53 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:47:53.301 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:47:53 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T03:47:53.302 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:47:53 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:47:53.302 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:47:53 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T03:47:53.302 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:47:53 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:47:53.303 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:47:53 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:47:53.303 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:47:53 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:47:53.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:47:52 smithi115 conmon[90393]: 2022-05-14T03:47:52.926+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:47:53.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:47:52 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:47:53.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:47:52 smithi115 ceph-mon[78593]: pgmap v8154: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:47:54.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:47:53 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:47:54.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:47:53 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:47:54.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T03:47:54.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:47:54.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T03:47:54.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:47:54.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:47:54.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:47:54.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T03:47:54.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:47:54.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T03:47:54.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:47:54.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T03:47:54.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:47:54.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T03:47:54.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:47:54.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T03:47:54.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:47:54.298 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:47:54.298 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:47:54.298 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:47:53 smithi047 ceph-mon[77147]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T03:47:54.360 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:47:53 smithi115 conmon[90393]: 2022-05-14T03:47:53.966+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:47:54.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:47:53 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:47:54.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:47:53 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:47:54.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T03:47:54.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:47:54.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T03:47:54.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:47:54.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:47:54.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:47:54.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T03:47:54.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:47:54.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T03:47:54.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:47:54.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T03:47:54.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:47:54.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T03:47:54.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:47:54.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T03:47:54.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:47:54.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:47:54.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:47:54.365 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:47:53 smithi115 ceph-mon[78593]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T03:47:55.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:47:54 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:47:55.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:47:54 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:47:55.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:47:54 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:47:55.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:47:54 smithi047 ceph-mon[77147]: pgmap v8155: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:47:55.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:47:54 smithi115 conmon[90393]: 2022-05-14T03:47:54.962+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:47:55.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:47:54 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:47:55.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:47:54 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:47:55.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:47:54 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:47:55.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:47:54 smithi115 ceph-mon[78593]: pgmap v8155: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:47:56.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:47:55 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:47:56.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:47:56 smithi047 conmon[78521]: 2022-05-14T03:47:56.144+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:47:56.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:47:56 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T03:47:56.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:47:56 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:47:56.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:47:56 smithi047 conmon[78521]: yield 2022-05-14T03:47:56.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:47:56 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:47:56.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:47:56 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:47:56.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:47:56 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:47:56.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:47:56 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T03:47:56.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:47:56 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:47:56.297 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:47:56 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T03:47:56.297 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:47:56 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:47:56.297 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:47:56 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T03:47:56.297 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:47:56 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:47:56.298 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:47:56 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T03:47:56.298 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:47:56 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:47:56.298 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:47:56 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T03:47:56.298 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:47:56 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:47:56.299 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:47:56 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:47:56.299 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:47:56 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:47:56.300 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:47:56 smithi047 conmon[78521]: 2022-05-14T03:47:56.145+0000 7f533ff78700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T03:47:56.300 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:47:56 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T03:47:56.300 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:47:56 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:47:56.300 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:47:56 smithi047 conmon[78521]: yield 2022-05-14T03:47:56.301 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:47:56 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:47:56.301 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:47:56 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:47:56.301 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:47:56 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:47:56.301 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:47:56 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T03:47:56.302 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:47:56 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:47:56.302 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:47:56 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T03:47:56.302 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:47:56 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:47:56.303 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:47:56 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T03:47:56.303 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:47:56 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:47:56.303 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:47:56 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T03:47:56.303 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:47:56 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:47:56.304 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:47:56 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T03:47:56.304 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:47:56 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:47:56.304 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:47:56 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:47:56.304 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:47:56 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:47:56.304 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:47:56 smithi047 conmon[78521]: 2022-05-14T03:47:56.305 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:47:56 smithi047 conmon[78521]: During handling of the above exception, another exception occurred: 2022-05-14T03:47:56.306 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:47:56 smithi047 conmon[78521]: 2022-05-14T03:47:56.306 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:47:56 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T03:47:56.306 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:47:56 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T03:47:56.306 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:47:56 smithi047 conmon[78521]: return f(*arg) 2022-05-14T03:47:56.307 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:47:56 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T03:47:56.307 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:47:56 smithi047 conmon[78521]: self._write_client_files(client_files, host) 2022-05-14T03:47:56.307 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:47:56 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T03:47:56.307 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:47:56 smithi047 conmon[78521]: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T03:47:56.307 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:47:56 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T03:47:56.308 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:47:56 smithi047 conmon[78521]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T03:47:56.308 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:47:56 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T03:47:56.308 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:47:56 smithi047 conmon[78521]: return self.event_loop.get_result(coro) 2022-05-14T03:47:56.308 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:47:56 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T03:47:56.309 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:47:56 smithi047 conmon[78521]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T03:47:56.309 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:47:56 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T03:47:56.309 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:47:56 smithi047 conmon[78521]: return self.__get_result() 2022-05-14T03:47:56.309 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:47:56 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T03:47:56.310 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:47:56 smithi047 conmon[78521]: raise self._exception 2022-05-14T03:47:56.310 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:47:56 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T03:47:56.310 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:47:56 smithi047 conmon[78521]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T03:47:56.310 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:47:56 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T03:47:56.310 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:47:56 smithi047 conmon[78521]: conn = await self._remote_connection(host, addr) 2022-05-14T03:47:56.311 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:47:56 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T03:47:56.311 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:47:56 smithi047 conmon[78521]: raise 2022-05-14T03:47:56.311 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:47:56 smithi047 conmon[78521]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T03:47:56.311 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:47:56 smithi047 conmon[78521]: self.gen.throw(type, value, traceback) 2022-05-14T03:47:56.311 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:47:56 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T03:47:56.312 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:47:56 smithi047 conmon[78521]: raise OrchestratorError(msg) 2022-05-14T03:47:56.312 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:47:56 smithi047 conmon[78521]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:47:56.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:47:55 smithi115 conmon[90393]: 2022-05-14T03:47:55.941+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:47:56.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:47:55 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:47:57.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:47:56 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:47:57.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:47:56 smithi047 ceph-mon[77147]: pgmap v8156: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:47:57.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:47:56 smithi115 conmon[90393]: 2022-05-14T03:47:56.972+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:47:57.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:47:56 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:47:57.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:47:56 smithi115 ceph-mon[78593]: pgmap v8156: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:47:58.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:47:57 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:47:58.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T03:47:58.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:47:58.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T03:47:58.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:47:58.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:47:58.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:47:58.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T03:47:58.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:47:58.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T03:47:58.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:47:58.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T03:47:58.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:47:58.301 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T03:47:58.301 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:47:58.301 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T03:47:58.302 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:47:58.302 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:47:58.302 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:47:58.302 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:47:57 smithi047 ceph-mon[77147]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T03:47:58.303 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T03:47:58.303 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:47:58.303 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T03:47:58.303 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:47:58.304 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:47:58.304 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:47:58.304 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T03:47:58.304 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:47:58.305 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T03:47:58.306 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:47:58.306 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T03:47:58.306 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:47:58.306 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T03:47:58.307 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:47:58.307 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T03:47:58.307 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:47:58.307 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:47:58.307 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:47:58.308 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T03:47:58.308 INFO:journalctl@ceph.mon.b.smithi047.stdout: During handling of the above exception, another exception occurred: 2022-05-14T03:47:58.308 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T03:47:58.308 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T03:47:58.309 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T03:47:58.309 INFO:journalctl@ceph.mon.b.smithi047.stdout: return f(*arg) 2022-05-14T03:47:58.309 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T03:47:58.310 INFO:journalctl@ceph.mon.b.smithi047.stdout: self._write_client_files(client_files, host) 2022-05-14T03:47:58.310 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T03:47:58.310 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T03:47:58.310 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T03:47:58.311 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T03:47:58.311 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T03:47:58.312 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.event_loop.get_result(coro) 2022-05-14T03:47:58.312 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T03:47:58.312 INFO:journalctl@ceph.mon.b.smithi047.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T03:47:58.312 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T03:47:58.312 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.__get_result() 2022-05-14T03:47:58.313 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T03:47:58.313 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise self._exception 2022-05-14T03:47:58.313 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T03:47:58.313 INFO:journalctl@ceph.mon.b.smithi047.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T03:47:58.314 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T03:47:58.314 INFO:journalctl@ceph.mon.b.smithi047.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T03:47:58.314 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T03:47:58.314 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise 2022-05-14T03:47:58.314 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T03:47:58.315 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.gen.throw(type, value, traceback) 2022-05-14T03:47:58.315 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T03:47:58.315 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OrchestratorError(msg) 2022-05-14T03:47:58.315 INFO:journalctl@ceph.mon.b.smithi047.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:47:58.316 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:47:57 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 15610 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:47:58.316 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:47:57 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:47:58.360 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:47:57 smithi115 conmon[90393]: 2022-05-14T03:47:57.953+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:47:58.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:47:57 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:47:58.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T03:47:58.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:47:58.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T03:47:58.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:47:58.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:47:58.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:47:58.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T03:47:58.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:47:58.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T03:47:58.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:47:58.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T03:47:58.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:47:58.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T03:47:58.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:47:58.365 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T03:47:58.365 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:47:58.365 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:47:58.366 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:47:58.366 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:47:57 smithi115 ceph-mon[78593]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T03:47:58.366 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T03:47:58.367 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:47:58.367 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T03:47:58.368 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:47:58.368 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:47:58.369 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:47:58.370 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T03:47:58.370 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:47:58.370 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T03:47:58.371 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:47:58.371 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T03:47:58.371 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:47:58.371 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T03:47:58.371 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:47:58.372 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T03:47:58.372 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:47:58.372 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:47:58.372 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:47:58.373 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T03:47:58.373 INFO:journalctl@ceph.mon.c.smithi115.stdout: During handling of the above exception, another exception occurred: 2022-05-14T03:47:58.373 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T03:47:58.373 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T03:47:58.374 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T03:47:58.374 INFO:journalctl@ceph.mon.c.smithi115.stdout: return f(*arg) 2022-05-14T03:47:58.374 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T03:47:58.374 INFO:journalctl@ceph.mon.c.smithi115.stdout: self._write_client_files(client_files, host) 2022-05-14T03:47:58.374 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T03:47:58.375 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T03:47:58.375 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T03:47:58.375 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T03:47:58.376 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T03:47:58.376 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.event_loop.get_result(coro) 2022-05-14T03:47:58.376 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T03:47:58.377 INFO:journalctl@ceph.mon.c.smithi115.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T03:47:58.377 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T03:47:58.377 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.__get_result() 2022-05-14T03:47:58.378 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T03:47:58.378 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise self._exception 2022-05-14T03:47:58.378 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T03:47:58.378 INFO:journalctl@ceph.mon.c.smithi115.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T03:47:58.379 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T03:47:58.379 INFO:journalctl@ceph.mon.c.smithi115.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T03:47:58.379 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T03:47:58.379 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise 2022-05-14T03:47:58.379 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T03:47:58.380 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.gen.throw(type, value, traceback) 2022-05-14T03:47:58.380 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T03:47:58.380 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OrchestratorError(msg) 2022-05-14T03:47:58.381 INFO:journalctl@ceph.mon.c.smithi115.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:47:58.381 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:47:57 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 15610 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:47:58.381 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:47:57 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:47:59.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:47:58 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:47:59.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:47:58 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:47:59.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:47:58 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:47:59.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:47:58 smithi047 ceph-mon[77147]: pgmap v8157: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:47:59.360 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:47:58 smithi115 conmon[90393]: 2022-05-14T03:47:58.954+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:47:59.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:47:58 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:47:59.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:47:58 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:47:59.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:47:58 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:47:59.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:47:58 smithi115 ceph-mon[78593]: pgmap v8157: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:48:00.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:47:59 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:48:00.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:47:59 smithi115 conmon[90393]: 2022-05-14T03:47:59.944+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:48:00.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:47:59 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:48:01.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:48:00 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:48:01.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:48:00 smithi047 ceph-mon[77147]: pgmap v8158: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:48:01.360 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:48:00 smithi115 conmon[90393]: 2022-05-14T03:48:00.930+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:48:01.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:48:00 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:48:01.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:48:00 smithi115 ceph-mon[78593]: pgmap v8158: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:48:02.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:48:01 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:48:02.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:48:01 smithi115 conmon[90393]: 2022-05-14T03:48:01.954+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:48:02.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:48:01 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:48:03.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:48:02 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:48:03.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:48:02 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:48:03.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:48:02 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:48:03.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:48:02 smithi047 ceph-mon[77147]: pgmap v8159: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:48:03.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:48:02 smithi115 conmon[90393]: 2022-05-14T03:48:02.938+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:48:03.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:48:02 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:48:03.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:48:02 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:48:03.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:48:02 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:48:03.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:48:02 smithi115 ceph-mon[78593]: pgmap v8159: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:48:04.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:48:03 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:48:04.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:48:03 smithi115 conmon[90393]: 2022-05-14T03:48:03.975+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:48:04.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:48:03 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:48:05.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:48:04 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:48:05.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:48:04 smithi047 ceph-mon[77147]: pgmap v8160: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:48:05.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:48:05 smithi115 conmon[90393]: 2022-05-14T03:48:05.005+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:48:05.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:48:04 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:48:05.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:48:04 smithi115 ceph-mon[78593]: pgmap v8160: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:48:06.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:48:05 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:48:06.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:48:06 smithi115 conmon[90393]: 2022-05-14T03:48:06.021+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:48:06.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:48:05 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:48:07.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:48:06 smithi047 ceph-mon[77147]: pgmap v8161: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:48:07.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:48:06 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 15615 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:48:07.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:48:07 smithi115 conmon[90393]: 2022-05-14T03:48:07.015+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:48:07.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:48:06 smithi115 ceph-mon[78593]: pgmap v8161: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:48:07.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:48:06 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 15615 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:48:08.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:48:07 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:48:08.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:48:08 smithi115 conmon[90393]: 2022-05-14T03:48:08.003+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:48:08.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:48:07 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:48:09.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:48:08 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:48:09.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:48:08 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:48:09.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:48:08 smithi047 ceph-mon[77147]: pgmap v8162: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:48:09.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:48:08 smithi115 conmon[90393]: 2022-05-14T03:48:08.961+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:48:09.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:48:08 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:48:09.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:48:08 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:48:09.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:48:08 smithi115 ceph-mon[78593]: pgmap v8162: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:48:10.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:48:09 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:48:10.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:48:09 smithi115 conmon[90393]: 2022-05-14T03:48:09.938+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:48:10.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:48:09 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:48:11.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:48:10 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:48:11.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:48:10 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:48:11.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:48:10 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:48:11.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:48:10 smithi047 ceph-mon[77147]: pgmap v8163: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:48:11.361 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:48:10 smithi115 conmon[90393]: 2022-05-14T03:48:10.900+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:48:11.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:48:10 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:48:11.362 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:48:10 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:48:11.362 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:48:10 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:48:11.362 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:48:10 smithi115 ceph-mon[78593]: pgmap v8163: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:48:12.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:48:11 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:48:12.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:48:11 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 15625 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:48:12.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:48:11 smithi115 conmon[90393]: 2022-05-14T03:48:11.882+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:48:12.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:48:11 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:48:12.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:48:11 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 15625 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:48:13.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:48:12 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:48:13.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:48:12 smithi047 ceph-mon[77147]: pgmap v8164: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:48:13.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:48:12 smithi115 conmon[90393]: 2022-05-14T03:48:12.883+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:48:13.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:48:12 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:48:13.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:48:12 smithi115 ceph-mon[78593]: pgmap v8164: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:48:14.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:48:13 smithi115 conmon[90393]: 2022-05-14T03:48:13.850+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:48:14.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:48:13 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:48:14.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:48:13 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:48:15.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:48:14 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:48:15.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:48:14 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:48:15.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:48:14 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:48:15.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:48:14 smithi047 ceph-mon[77147]: pgmap v8165: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:48:15.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:48:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T03:48:15.295 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:48:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T03:48:15.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:48:14 smithi115 conmon[90393]: 2022-05-14T03:48:14.894+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:48:15.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:48:14 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:48:15.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:48:14 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:48:15.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:48:14 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:48:15.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:48:14 smithi115 ceph-mon[78593]: pgmap v8165: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:48:15.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:48:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T03:48:15.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:48:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T03:48:16.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:48:15 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:48:16.360 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:48:15 smithi115 conmon[90393]: 2022-05-14T03:48:15.891+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:48:16.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:48:15 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:48:17.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:48:17 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:48:17.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:48:17 smithi047 ceph-mon[77147]: pgmap v8166: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:48:17.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:48:16 smithi115 conmon[90393]: 2022-05-14T03:48:16.941+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:48:17.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:48:17 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:48:17.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:48:17 smithi115 ceph-mon[78593]: pgmap v8166: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:48:18.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:48:18 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:48:18.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:48:18 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 15630 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:48:18.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:48:17 smithi115 conmon[90393]: 2022-05-14T03:48:17.970+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:48:18.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:48:18 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:48:18.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:48:18 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 15630 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:48:19.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:48:19 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:48:19.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:48:19 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:48:19.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:48:19 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:48:19.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:48:19 smithi047 ceph-mon[77147]: pgmap v8167: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:48:19.360 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:48:18 smithi115 conmon[90393]: 2022-05-14T03:48:18.975+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:48:19.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:48:19 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:48:19.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:48:19 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:48:19.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:48:19 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:48:19.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:48:19 smithi115 ceph-mon[78593]: pgmap v8167: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:48:20.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:48:20 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:48:20.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:48:20 smithi115 conmon[90393]: 2022-05-14T03:48:20.009+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:48:20.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:48:20 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:48:21.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:48:21 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:48:21.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:48:21 smithi047 ceph-mon[77147]: pgmap v8168: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:48:21.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:48:21 smithi115 conmon[90393]: 2022-05-14T03:48:21.058+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:48:21.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:48:21 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:48:21.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:48:21 smithi115 ceph-mon[78593]: pgmap v8168: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:48:22.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:48:22 smithi115 conmon[90393]: 2022-05-14T03:48:22.022+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:48:23.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:48:23 smithi115 conmon[90393]: 2022-05-14T03:48:23.016+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:48:23.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:48:22 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:48:23.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:48:22 smithi115 ceph-mon[78593]: pgmap v8169: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:48:23.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:48:22 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:48:23.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:48:22 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:48:23.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:48:22 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:48:23.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:48:22 smithi047 ceph-mon[77147]: pgmap v8169: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:48:23.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:48:22 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:48:23.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:48:22 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:48:24.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:48:24 smithi115 conmon[90393]: 2022-05-14T03:48:24.032+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:48:24.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:48:23 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:48:24.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:48:23 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:48:25.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:48:25 smithi115 conmon[90393]: 2022-05-14T03:48:25.006+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:48:25.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:48:24 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:48:25.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:48:24 smithi115 ceph-mon[78593]: pgmap v8170: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:48:25.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:48:24 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:48:25.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:48:24 smithi047 ceph-mon[77147]: pgmap v8170: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:48:26.110 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:48:25 smithi115 conmon[90393]: 2022-05-14T03:48:25.966+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:48:26.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:48:25 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:48:26.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:48:25 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:48:26.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:48:25 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:48:26.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:48:25 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:48:27.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:48:26 smithi115 conmon[90393]: 2022-05-14T03:48:26.956+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:48:27.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:48:26 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:48:27.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:48:26 smithi115 ceph-mon[78593]: pgmap v8171: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:48:27.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:48:26 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 15635 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:48:27.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:48:26 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:48:27.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:48:26 smithi047 ceph-mon[77147]: pgmap v8171: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:48:27.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:48:26 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 15635 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:48:28.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:48:27 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:48:28.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:48:28 smithi115 conmon[90393]: 2022-05-14T03:48:28.006+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:48:28.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:48:27 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:48:29.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:48:28 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:48:29.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:48:28 smithi047 ceph-mon[77147]: pgmap v8172: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:48:29.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:48:29 smithi115 conmon[90393]: 2022-05-14T03:48:29.031+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:48:29.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:48:28 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:48:29.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:48:28 smithi115 ceph-mon[78593]: pgmap v8172: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:48:30.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:48:30 smithi115 conmon[90393]: 2022-05-14T03:48:30.017+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:48:31.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:48:30 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:48:31.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:48:30 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:48:31.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:48:30 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:48:31.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:48:30 smithi047 ceph-mon[77147]: pgmap v8173: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:48:31.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:48:31 smithi115 conmon[90393]: 2022-05-14T03:48:31.012+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:48:31.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:48:30 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:48:31.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:48:30 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:48:31.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:48:30 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:48:31.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:48:30 smithi115 ceph-mon[78593]: pgmap v8173: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:48:32.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:48:31 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:48:32.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:48:31 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 15645 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:48:32.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:48:32 smithi115 conmon[90393]: 2022-05-14T03:48:32.031+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:48:32.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:48:31 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:48:32.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:48:31 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 15645 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:48:33.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:48:32 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:48:33.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:48:32 smithi047 ceph-mon[77147]: pgmap v8174: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:48:33.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:48:33 smithi115 conmon[90393]: 2022-05-14T03:48:33.070+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:48:33.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:48:32 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:48:33.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:48:32 smithi115 ceph-mon[78593]: pgmap v8174: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:48:34.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:48:33 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:48:34.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:48:34 smithi115 conmon[90393]: 2022-05-14T03:48:34.102+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:48:34.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:48:33 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:48:35.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:48:34 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:48:35.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:48:34 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:48:35.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:48:34 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:48:35.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:48:34 smithi047 ceph-mon[77147]: pgmap v8175: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:48:35.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:48:35 smithi115 conmon[90393]: 2022-05-14T03:48:35.113+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:48:35.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:48:34 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:48:35.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:48:34 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:48:35.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:48:34 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:48:35.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:48:34 smithi115 ceph-mon[78593]: pgmap v8175: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:48:36.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:48:35 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:48:36.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:48:36 smithi115 conmon[90393]: 2022-05-14T03:48:36.104+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:48:36.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:48:35 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:48:37.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:48:36 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:48:37.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:48:36 smithi047 ceph-mon[77147]: pgmap v8176: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:48:37.363 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:48:37 smithi115 conmon[90393]: 2022-05-14T03:48:37.072+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:48:37.364 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:48:36 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:48:37.364 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:48:36 smithi115 ceph-mon[78593]: pgmap v8176: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:48:38.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:48:37 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:48:38.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:48:37 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 15649 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:48:38.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:48:38 smithi115 conmon[90393]: 2022-05-14T03:48:38.068+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:48:38.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:48:37 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:48:38.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:48:37 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 15649 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:48:39.295 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:48:38 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:48:39.295 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:48:38 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:48:39.295 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:48:38 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:48:39.296 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:48:38 smithi047 ceph-mon[77147]: pgmap v8177: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:48:39.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:48:39 smithi115 conmon[90393]: 2022-05-14T03:48:39.025+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:48:39.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:48:38 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:48:39.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:48:38 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:48:39.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:48:38 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:48:39.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:48:38 smithi115 ceph-mon[78593]: pgmap v8177: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:48:40.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:48:39 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:48:40.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:48:40 smithi115 conmon[90393]: 2022-05-14T03:48:40.050+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:48:40.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:48:39 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:48:41.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:48:40 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:48:41.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:48:40 smithi047 ceph-mon[77147]: pgmap v8178: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:48:41.361 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:48:41 smithi115 conmon[90393]: 2022-05-14T03:48:41.026+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:48:41.362 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:48:40 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:48:41.362 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:48:40 smithi115 ceph-mon[78593]: pgmap v8178: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:48:42.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:48:41 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:48:42.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:48:42 smithi115 conmon[90393]: 2022-05-14T03:48:42.006+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:48:42.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:48:41 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:48:43.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:48:42 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:48:43.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:48:42 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:48:43.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:48:42 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:48:43.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:48:42 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:48:43.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:48:42 smithi047 ceph-mon[77147]: pgmap v8179: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:48:43.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:48:42 smithi115 conmon[90393]: 2022-05-14T03:48:42.977+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:48:43.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:48:42 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:48:43.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:48:42 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:48:43.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:48:42 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:48:43.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:48:42 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:48:43.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:48:42 smithi115 ceph-mon[78593]: pgmap v8179: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:48:44.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:48:43 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:48:44.360 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:48:44 smithi115 conmon[90393]: 2022-05-14T03:48:44.016+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:48:44.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:48:43 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:48:45.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:48:44 smithi047 ceph-mon[77147]: pgmap v8180: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:48:45.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:48:45 smithi115 conmon[90393]: 2022-05-14T03:48:45.020+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:48:45.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:48:44 smithi115 ceph-mon[78593]: pgmap v8180: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:48:46.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:48:45 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:48:46.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:48:45 smithi115 conmon[90393]: 2022-05-14T03:48:45.973+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:48:46.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:48:45 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:48:47.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:48:46 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:48:47.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:48:46 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:48:47.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:48:46 smithi047 ceph-mon[77147]: pgmap v8181: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:48:47.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:48:46 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 15655 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:48:47.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:48:46 smithi115 conmon[90393]: 2022-05-14T03:48:46.951+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:48:47.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:48:46 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:48:47.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:48:46 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:48:47.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:48:46 smithi115 ceph-mon[78593]: pgmap v8181: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:48:47.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:48:46 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 15655 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:48:48.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:48:47 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:48:48.360 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:48:47 smithi115 conmon[90393]: 2022-05-14T03:48:47.936+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:48:48.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:48:47 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:48:49.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:48:48 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:48:49.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:48:48 smithi047 ceph-mon[77147]: pgmap v8182: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:48:49.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:48:48 smithi115 conmon[90393]: 2022-05-14T03:48:48.916+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:48:49.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:48:48 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:48:49.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:48:48 smithi115 ceph-mon[78593]: pgmap v8182: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:48:50.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:48:49 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:48:50.360 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:48:49 smithi115 conmon[90393]: 2022-05-14T03:48:49.889+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:48:50.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:48:49 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:48:51.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:48:50 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:48:51.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:48:50 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:48:51.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:48:50 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:48:51.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:48:50 smithi047 ceph-mon[77147]: pgmap v8183: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:48:51.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:48:50 smithi115 conmon[90393]: 2022-05-14T03:48:50.868+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:48:51.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:48:50 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:48:51.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:48:50 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:48:51.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:48:50 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:48:51.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:48:50 smithi115 ceph-mon[78593]: pgmap v8183: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:48:52.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:48:51 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:48:52.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:48:51 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 15665 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:48:52.362 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:48:51 smithi115 conmon[90393]: 2022-05-14T03:48:51.860+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:48:52.362 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:48:51 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:48:52.362 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:48:51 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 15665 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:48:53.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:48:52 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:48:53.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:48:52 smithi047 ceph-mon[77147]: pgmap v8184: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:48:53.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:48:52 smithi115 conmon[90393]: 2022-05-14T03:48:52.888+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:48:53.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:48:52 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:48:53.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:48:52 smithi115 ceph-mon[78593]: pgmap v8184: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:48:54.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:48:53 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:48:54.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:48:53 smithi115 conmon[90393]: 2022-05-14T03:48:53.925+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:48:54.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:48:53 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:48:55.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:48:54 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:48:55.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:48:54 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:48:55.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:48:54 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:48:55.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:48:54 smithi047 ceph-mon[77147]: pgmap v8185: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:48:55.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:48:54 smithi115 conmon[90393]: 2022-05-14T03:48:54.954+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:48:55.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:48:54 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:48:55.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:48:54 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:48:55.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:48:54 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:48:55.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:48:54 smithi115 ceph-mon[78593]: pgmap v8185: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:48:56.194 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:48:55 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:48:56.195 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:48:55 smithi115 conmon[90393]: 2022-05-14T03:48:55.932+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:48:56.225 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:48:55 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:48:57.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:48:56 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:48:57.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:48:56 smithi047 ceph-mon[77147]: pgmap v8186: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:48:57.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:48:56 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T03:48:57.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:48:56 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T03:48:57.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:48:56 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T03:48:57.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:48:56 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' 2022-05-14T03:48:57.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:48:56 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' 2022-05-14T03:48:57.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:48:56 smithi115 conmon[90393]: 2022-05-14T03:48:56.920+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:48:57.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:48:56 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:48:57.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:48:56 smithi115 ceph-mon[78593]: pgmap v8186: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:48:57.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:48:56 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T03:48:57.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:48:56 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T03:48:57.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:48:56 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T03:48:57.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:48:56 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' 2022-05-14T03:48:57.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:48:56 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' 2022-05-14T03:48:58.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:48:57 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:48:58.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:48:57 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 15670 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:48:58.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:48:57 smithi115 conmon[90393]: 2022-05-14T03:48:57.903+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:48:58.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:48:57 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:48:58.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:48:57 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 15670 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:48:59.211 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:48:58 smithi115 conmon[90393]: 2022-05-14T03:48:58.931+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:48:59.212 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:48:58 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:48:59.212 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:48:58 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:48:59.213 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:48:58 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:48:59.213 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:48:58 smithi115 ceph-mon[78593]: pgmap v8187: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:48:59.252 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:48:58 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:48:59.252 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:48:58 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:48:59.252 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:48:58 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:48:59.253 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:48:58 smithi047 ceph-mon[77147]: pgmap v8187: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:48:59.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:48:59 smithi047 conmon[78521]: 2022-05-14T03:48:59.247+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:48:59.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:48:59 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T03:48:59.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:48:59 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:48:59.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:48:59 smithi047 conmon[78521]: yield 2022-05-14T03:48:59.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:48:59 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:48:59.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:48:59 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:48:59.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:48:59 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:48:59.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:48:59 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T03:48:59.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:48:59 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:48:59.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:48:59 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T03:48:59.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:48:59 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:48:59.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:48:59 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T03:48:59.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:48:59 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:48:59.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:48:59 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T03:48:59.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:48:59 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:48:59.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:48:59 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T03:48:59.547 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:48:59 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:48:59.547 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:48:59 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:48:59.547 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:48:59 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:49:00.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:48:59 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:49:00.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:48:59 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' 2022-05-14T03:49:00.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:48:59 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' 2022-05-14T03:49:00.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:48:59 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-05-14T03:49:00.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:48:59 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-05-14T03:49:00.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:48:59 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-05-14T03:49:00.295 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:48:59 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-05-14T03:49:00.295 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:48:59 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-05-14T03:49:00.295 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:48:59 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-05-14T03:49:00.295 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:48:59 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-05-14T03:49:00.296 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:48:59 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-05-14T03:49:00.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:48:59 smithi115 conmon[90393]: 2022-05-14T03:48:59.964+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:49:00.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:48:59 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:49:00.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:48:59 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' 2022-05-14T03:49:00.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:48:59 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' 2022-05-14T03:49:00.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:48:59 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-05-14T03:49:00.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:48:59 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-05-14T03:49:00.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:48:59 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-05-14T03:49:00.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:48:59 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-05-14T03:49:00.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:48:59 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-05-14T03:49:00.362 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:48:59 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-05-14T03:49:00.362 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:48:59 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-05-14T03:49:00.362 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:48:59 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-05-14T03:49:01.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:49:00 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:49:01.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T03:49:01.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:49:01.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T03:49:01.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:49:01.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:49:01.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:49:01.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T03:49:01.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:49:01.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T03:49:01.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:49:01.298 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T03:49:01.298 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:49:01.299 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T03:49:01.299 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:49:01.299 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T03:49:01.300 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:49:01.300 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:49:01.300 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:49:01.301 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:49:00 smithi047 ceph-mon[77147]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T03:49:01.302 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:49:00 smithi047 ceph-mon[77147]: Adjusting osd_memory_target on smithi115 to 197.0M 2022-05-14T03:49:01.303 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:49:00 smithi047 ceph-mon[77147]: Unable to set osd_memory_target on smithi115 to 206629785: error parsing value: Value '206629785' is below minimum 939524096 2022-05-14T03:49:01.303 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:49:00 smithi047 ceph-mon[77147]: Adjusting osd_memory_target on smithi047 to 198.1M 2022-05-14T03:49:01.303 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:49:00 smithi047 ceph-mon[77147]: Unable to set osd_memory_target on smithi047 to 207755878: error parsing value: Value '207755878' is below minimum 939524096 2022-05-14T03:49:01.303 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:49:00 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:49:01.304 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:49:00 smithi047 ceph-mon[77147]: pgmap v8188: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:49:01.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:49:01 smithi115 conmon[90393]: 2022-05-14T03:49:01.012+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:49:01.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:49:00 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:49:01.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T03:49:01.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:49:01.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T03:49:01.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:49:01.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:49:01.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:49:01.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T03:49:01.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:49:01.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T03:49:01.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:49:01.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T03:49:01.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:49:01.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T03:49:01.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:49:01.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T03:49:01.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:49:01.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:49:01.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:49:01.364 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:49:00 smithi115 ceph-mon[78593]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T03:49:01.365 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:49:00 smithi115 ceph-mon[78593]: Adjusting osd_memory_target on smithi115 to 197.0M 2022-05-14T03:49:01.365 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:49:00 smithi115 ceph-mon[78593]: Unable to set osd_memory_target on smithi115 to 206629785: error parsing value: Value '206629785' is below minimum 939524096 2022-05-14T03:49:01.365 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:49:00 smithi115 ceph-mon[78593]: Adjusting osd_memory_target on smithi047 to 198.1M 2022-05-14T03:49:01.365 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:49:00 smithi115 ceph-mon[78593]: Unable to set osd_memory_target on smithi047 to 207755878: error parsing value: Value '207755878' is below minimum 939524096 2022-05-14T03:49:01.366 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:49:00 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:49:01.366 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:49:00 smithi115 ceph-mon[78593]: pgmap v8188: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:49:02.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:49:01 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:49:02.361 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:49:01 smithi115 conmon[90393]: 2022-05-14T03:49:01.976+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:49:02.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:49:01 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:49:02.793 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:49:02 smithi047 conmon[78521]: 2022-05-14T03:49:02.319+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:49:02.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:49:02 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T03:49:02.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:49:02 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:49:02.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:49:02 smithi047 conmon[78521]: yield 2022-05-14T03:49:02.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:49:02 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:49:02.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:49:02 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:49:02.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:49:02 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:49:02.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:49:02 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T03:49:02.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:49:02 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:49:02.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:49:02 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T03:49:02.797 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:49:02 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:49:02.797 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:49:02 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T03:49:02.798 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:49:02 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:49:02.798 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:49:02 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T03:49:02.799 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:49:02 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:49:02.799 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:49:02 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T03:49:02.799 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:49:02 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:49:02.800 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:49:02 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:49:02.800 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:49:02 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:49:02.800 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:49:02 smithi047 conmon[78521]: 2022-05-14T03:49:02.320+0000 7f533f777700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T03:49:02.800 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:49:02 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T03:49:02.801 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:49:02 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:49:02.801 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:49:02 smithi047 conmon[78521]: yield 2022-05-14T03:49:02.801 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:49:02 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:49:02.801 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:49:02 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:49:02.802 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:49:02 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:49:02.802 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:49:02 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T03:49:02.802 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:49:02 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:49:02.802 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:49:02 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T03:49:02.803 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:49:02 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:49:02.803 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:49:02 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T03:49:02.803 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:49:02 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:49:02.803 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:49:02 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T03:49:02.803 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:49:02 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:49:02.804 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:49:02 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T03:49:02.804 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:49:02 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:49:02.804 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:49:02 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:49:02.805 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:49:02 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:49:02.806 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:49:02 smithi047 conmon[78521]: 2022-05-14T03:49:02.807 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:49:02 smithi047 conmon[78521]: During handling of the above exception, another exception occurred: 2022-05-14T03:49:02.807 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:49:02 smithi047 conmon[78521]: 2022-05-14T03:49:02.808 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:49:02 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T03:49:02.808 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:49:02 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T03:49:02.809 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:49:02 smithi047 conmon[78521]: return f(*arg) 2022-05-14T03:49:02.811 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:49:02 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T03:49:02.812 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:49:02 smithi047 conmon[78521]: self._write_client_files(client_files, host) 2022-05-14T03:49:02.812 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:49:02 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T03:49:02.812 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:49:02 smithi047 conmon[78521]: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T03:49:02.812 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:49:02 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T03:49:02.813 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:49:02 smithi047 conmon[78521]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T03:49:02.813 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:49:02 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T03:49:02.813 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:49:02 smithi047 conmon[78521]: return self.event_loop.get_result(coro) 2022-05-14T03:49:02.813 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:49:02 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T03:49:02.814 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:49:02 smithi047 conmon[78521]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T03:49:02.814 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:49:02 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T03:49:02.814 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:49:02 smithi047 conmon[78521]: return self.__get_result() 2022-05-14T03:49:02.814 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:49:02 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T03:49:02.815 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:49:02 smithi047 conmon[78521]: raise self._exception 2022-05-14T03:49:02.815 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:49:02 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T03:49:02.815 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:49:02 smithi047 conmon[78521]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T03:49:02.815 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:49:02 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T03:49:02.815 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:49:02 smithi047 conmon[78521]: conn = await self._remote_connection(host, addr) 2022-05-14T03:49:02.816 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:49:02 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T03:49:02.816 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:49:02 smithi047 conmon[78521]: raise 2022-05-14T03:49:02.816 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:49:02 smithi047 conmon[78521]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T03:49:02.816 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:49:02 smithi047 conmon[78521]: self.gen.throw(type, value, traceback) 2022-05-14T03:49:02.817 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:49:02 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T03:49:02.817 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:49:02 smithi047 conmon[78521]: raise OrchestratorError(msg) 2022-05-14T03:49:02.817 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:49:02 smithi047 conmon[78521]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:49:03.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:49:03 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:49:03.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:49:03 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:49:03.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:49:03 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:49:03.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:49:03 smithi047 ceph-mon[77147]: pgmap v8189: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:49:03.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:49:02 smithi115 conmon[90393]: 2022-05-14T03:49:02.938+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:49:03.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:49:03 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:49:03.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:49:03 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:49:03.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:49:03 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:49:03.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:49:03 smithi115 ceph-mon[78593]: pgmap v8189: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:49:04.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:49:04 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:49:04.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T03:49:04.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:49:04.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T03:49:04.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:49:04.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:49:04.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:49:04.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T03:49:04.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:49:04.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T03:49:04.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:49:04.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T03:49:04.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:49:04.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T03:49:04.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:49:04.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T03:49:04.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:49:04.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:49:04.298 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:49:04.298 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:49:04 smithi047 ceph-mon[77147]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T03:49:04.298 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T03:49:04.298 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:49:04.299 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T03:49:04.299 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:49:04.299 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:49:04.299 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:49:04.299 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T03:49:04.300 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:49:04.300 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T03:49:04.300 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:49:04.300 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T03:49:04.301 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:49:04.301 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T03:49:04.301 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:49:04.301 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T03:49:04.302 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:49:04.302 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:49:04.302 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:49:04.302 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T03:49:04.302 INFO:journalctl@ceph.mon.b.smithi047.stdout: During handling of the above exception, another exception occurred: 2022-05-14T03:49:04.303 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T03:49:04.303 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T03:49:04.303 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T03:49:04.303 INFO:journalctl@ceph.mon.b.smithi047.stdout: return f(*arg) 2022-05-14T03:49:04.304 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T03:49:04.304 INFO:journalctl@ceph.mon.b.smithi047.stdout: self._write_client_files(client_files, host) 2022-05-14T03:49:04.304 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T03:49:04.304 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T03:49:04.305 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T03:49:04.305 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T03:49:04.305 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T03:49:04.305 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.event_loop.get_result(coro) 2022-05-14T03:49:04.305 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T03:49:04.306 INFO:journalctl@ceph.mon.b.smithi047.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T03:49:04.306 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T03:49:04.306 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.__get_result() 2022-05-14T03:49:04.306 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T03:49:04.307 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise self._exception 2022-05-14T03:49:04.307 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T03:49:04.307 INFO:journalctl@ceph.mon.b.smithi047.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T03:49:04.307 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T03:49:04.307 INFO:journalctl@ceph.mon.b.smithi047.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T03:49:04.308 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T03:49:04.308 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise 2022-05-14T03:49:04.308 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T03:49:04.308 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.gen.throw(type, value, traceback) 2022-05-14T03:49:04.309 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T03:49:04.309 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OrchestratorError(msg) 2022-05-14T03:49:04.309 INFO:journalctl@ceph.mon.b.smithi047.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:49:04.309 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:49:04 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:49:04.360 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:49:03 smithi115 conmon[90393]: 2022-05-14T03:49:03.888+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:49:04.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:49:04 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:49:04.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T03:49:04.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:49:04.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T03:49:04.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:49:04.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:49:04.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:49:04.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T03:49:04.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:49:04.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T03:49:04.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:49:04.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T03:49:04.365 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:49:04.365 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T03:49:04.365 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:49:04.365 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T03:49:04.366 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:49:04.366 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:49:04.366 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:49:04.366 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:49:04 smithi115 ceph-mon[78593]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T03:49:04.366 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T03:49:04.367 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:49:04.367 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T03:49:04.367 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:49:04.367 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:49:04.367 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:49:04.368 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T03:49:04.368 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:49:04.368 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T03:49:04.368 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:49:04.369 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T03:49:04.369 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:49:04.369 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T03:49:04.369 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:49:04.370 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T03:49:04.370 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:49:04.370 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:49:04.370 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:49:04.370 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T03:49:04.371 INFO:journalctl@ceph.mon.c.smithi115.stdout: During handling of the above exception, another exception occurred: 2022-05-14T03:49:04.371 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T03:49:04.371 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T03:49:04.371 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T03:49:04.371 INFO:journalctl@ceph.mon.c.smithi115.stdout: return f(*arg) 2022-05-14T03:49:04.372 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T03:49:04.372 INFO:journalctl@ceph.mon.c.smithi115.stdout: self._write_client_files(client_files, host) 2022-05-14T03:49:04.372 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T03:49:04.372 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T03:49:04.373 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T03:49:04.373 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T03:49:04.373 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T03:49:04.373 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.event_loop.get_result(coro) 2022-05-14T03:49:04.373 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T03:49:04.374 INFO:journalctl@ceph.mon.c.smithi115.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T03:49:04.374 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T03:49:04.374 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.__get_result() 2022-05-14T03:49:04.374 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T03:49:04.375 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise self._exception 2022-05-14T03:49:04.375 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T03:49:04.375 INFO:journalctl@ceph.mon.c.smithi115.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T03:49:04.375 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T03:49:04.375 INFO:journalctl@ceph.mon.c.smithi115.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T03:49:04.378 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T03:49:04.378 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise 2022-05-14T03:49:04.379 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T03:49:04.379 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.gen.throw(type, value, traceback) 2022-05-14T03:49:04.379 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T03:49:04.380 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OrchestratorError(msg) 2022-05-14T03:49:04.380 INFO:journalctl@ceph.mon.c.smithi115.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:49:04.380 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:49:04 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:49:05.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:49:05 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:49:05.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:49:05 smithi047 ceph-mon[77147]: pgmap v8190: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:49:05.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:49:04 smithi115 conmon[90393]: 2022-05-14T03:49:04.864+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:49:05.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:49:05 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:49:05.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:49:05 smithi115 ceph-mon[78593]: pgmap v8190: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:49:06.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:49:05 smithi115 conmon[90393]: 2022-05-14T03:49:05.835+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:49:06.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:49:06 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:49:06.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:49:06 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:49:07.110 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:49:06 smithi115 conmon[90393]: 2022-05-14T03:49:06.834+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:49:07.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:49:07 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:49:07.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:49:07 smithi115 ceph-mon[78593]: pgmap v8191: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:49:07.115 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:49:07 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 15675 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:49:07.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:49:07 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:49:07.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:49:07 smithi047 ceph-mon[77147]: pgmap v8191: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:49:07.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:49:07 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 15675 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:49:08.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:49:07 smithi115 conmon[90393]: 2022-05-14T03:49:07.827+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:49:08.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:49:08 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:49:08.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:49:08 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:49:09.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:49:09 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:49:09.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:49:09 smithi047 ceph-mon[77147]: pgmap v8192: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:49:09.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:49:08 smithi115 conmon[90393]: 2022-05-14T03:49:08.870+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:49:09.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:49:09 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:49:09.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:49:09 smithi115 ceph-mon[78593]: pgmap v8192: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:49:10.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:49:10 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:49:10.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:49:10 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:49:10.360 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:49:09 smithi115 conmon[90393]: 2022-05-14T03:49:09.904+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:49:10.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:49:10 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:49:10.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:49:10 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:49:11.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:49:11 smithi047 ceph-mon[77147]: pgmap v8193: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:49:11.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:49:11 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:49:11.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:49:11 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:49:11.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:49:11 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:49:11.360 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:49:10 smithi115 conmon[90393]: 2022-05-14T03:49:10.893+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:49:11.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:49:11 smithi115 ceph-mon[78593]: pgmap v8193: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:49:11.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:49:11 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:49:11.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:49:11 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:49:11.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:49:11 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:49:12.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:49:12 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 15685 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:49:12.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:49:12 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:49:12.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:49:11 smithi115 conmon[90393]: 2022-05-14T03:49:11.897+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:49:12.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:49:12 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 15685 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:49:12.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:49:12 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:49:13.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:49:13 smithi047 ceph-mon[77147]: pgmap v8194: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:49:13.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:49:13 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:49:13.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:49:12 smithi115 conmon[90393]: 2022-05-14T03:49:12.916+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:49:13.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:49:13 smithi115 ceph-mon[78593]: pgmap v8194: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:49:13.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:49:13 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:49:14.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:49:14 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:49:14.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:49:13 smithi115 conmon[90393]: 2022-05-14T03:49:13.950+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:49:14.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:49:14 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:49:15.360 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:49:14 smithi115 conmon[90393]: 2022-05-14T03:49:14.989+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:49:15.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:49:15 smithi115 ceph-mon[78593]: pgmap v8195: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:49:15.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:49:15 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:49:15.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:49:15 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:49:15.362 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:49:15 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T03:49:15.362 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:49:15 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T03:49:15.362 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:49:15 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:49:15.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:49:15 smithi047 ceph-mon[77147]: pgmap v8195: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:49:15.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:49:15 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:49:15.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:49:15 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:49:15.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:49:15 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T03:49:15.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:49:15 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T03:49:15.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:49:15 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:49:16.360 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:49:16 smithi115 conmon[90393]: 2022-05-14T03:49:16.014+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:49:16.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:49:16 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:49:16.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:49:16 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:49:17.360 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:49:16 smithi115 conmon[90393]: 2022-05-14T03:49:16.967+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:49:17.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:49:17 smithi115 ceph-mon[78593]: pgmap v8196: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:49:17.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:49:17 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 15690 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:49:17.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:49:17 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:49:17.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:49:17 smithi047 ceph-mon[77147]: pgmap v8196: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:49:17.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:49:17 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 15690 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:49:17.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:49:17 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:49:18.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:49:17 smithi115 conmon[90393]: 2022-05-14T03:49:17.925+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:49:18.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:49:18 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:49:18.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:49:18 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:49:19.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:49:18 smithi115 conmon[90393]: 2022-05-14T03:49:18.958+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:49:19.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:49:19 smithi115 ceph-mon[78593]: pgmap v8197: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:49:19.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:49:19 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:49:19.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:49:19 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:49:19.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:49:19 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:49:19.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:49:19 smithi047 ceph-mon[77147]: pgmap v8197: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:49:19.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:49:19 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:49:19.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:49:19 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:49:19.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:49:19 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:49:20.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:49:19 smithi115 conmon[90393]: 2022-05-14T03:49:19.950+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:49:20.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:49:20 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:49:20.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:49:20 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:49:21.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:49:20 smithi115 conmon[90393]: 2022-05-14T03:49:20.909+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:49:21.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:49:21 smithi115 ceph-mon[78593]: pgmap v8198: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:49:21.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:49:21 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:49:21.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:49:21 smithi047 ceph-mon[77147]: pgmap v8198: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:49:21.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:49:21 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:49:22.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:49:21 smithi115 conmon[90393]: 2022-05-14T03:49:21.942+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:49:22.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:49:22 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 15695 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:49:22.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:49:22 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:49:22.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:49:22 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 15695 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:49:22.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:49:22 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:49:23.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:49:22 smithi115 conmon[90393]: 2022-05-14T03:49:22.944+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:49:23.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:49:23 smithi115 ceph-mon[78593]: pgmap v8199: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:49:23.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:49:23 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:49:23.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:49:23 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:49:23.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:49:23 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:49:23.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:49:23 smithi047 ceph-mon[77147]: pgmap v8199: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:49:23.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:49:23 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:49:23.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:49:23 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:49:23.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:49:23 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:49:24.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:49:23 smithi115 conmon[90393]: 2022-05-14T03:49:23.926+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:49:24.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:49:24 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:49:24.546 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:49:24 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:49:25.362 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:49:24 smithi115 conmon[90393]: 2022-05-14T03:49:24.934+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:49:25.363 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:49:25 smithi115 ceph-mon[78593]: pgmap v8200: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:49:25.363 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:49:25 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:49:25.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:49:25 smithi047 ceph-mon[77147]: pgmap v8200: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:49:25.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:49:25 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:49:26.362 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:49:25 smithi115 conmon[90393]: 2022-05-14T03:49:25.892+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:49:26.363 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:49:26 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:49:26.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:49:26 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:49:27.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:49:26 smithi115 conmon[90393]: 2022-05-14T03:49:26.880+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:49:27.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:49:27 smithi115 ceph-mon[78593]: pgmap v8201: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:49:27.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:49:27 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 15700 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:49:27.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:49:27 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:49:27.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:49:27 smithi047 ceph-mon[77147]: pgmap v8201: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:49:27.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:49:27 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 15700 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:49:27.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:49:27 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:49:28.360 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:49:27 smithi115 conmon[90393]: 2022-05-14T03:49:27.895+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:49:28.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:49:28 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:49:28.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:49:28 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:49:29.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:49:28 smithi115 conmon[90393]: 2022-05-14T03:49:28.901+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:49:29.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:49:29 smithi115 ceph-mon[78593]: pgmap v8202: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:49:29.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:49:29 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:49:29.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:49:29 smithi047 ceph-mon[77147]: pgmap v8202: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:49:29.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:49:29 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:49:30.360 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:49:29 smithi115 conmon[90393]: 2022-05-14T03:49:29.917+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:49:30.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:49:30 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:49:30.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:49:30 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:49:30.362 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:49:30 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:49:30.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:49:30 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:49:30.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:49:30 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:49:30.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:49:30 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:49:31.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:49:30 smithi115 conmon[90393]: 2022-05-14T03:49:30.885+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:49:31.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:49:31 smithi115 ceph-mon[78593]: pgmap v8203: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:49:31.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:49:31 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:49:31.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:49:31 smithi047 ceph-mon[77147]: pgmap v8203: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:49:31.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:49:31 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:49:32.360 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:49:31 smithi115 conmon[90393]: 2022-05-14T03:49:31.872+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:49:32.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:49:32 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 15705 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:49:32.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:49:32 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:49:32.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:49:32 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 15705 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:49:32.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:49:32 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:49:33.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:49:32 smithi115 conmon[90393]: 2022-05-14T03:49:32.838+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:49:33.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:49:33 smithi115 ceph-mon[78593]: pgmap v8204: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:49:33.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:49:33 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:49:33.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:49:33 smithi047 ceph-mon[77147]: pgmap v8204: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:49:33.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:49:33 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:49:34.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:49:33 smithi115 conmon[90393]: 2022-05-14T03:49:33.880+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:49:35.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:49:34 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:49:35.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:49:34 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:49:35.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:49:34 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:49:35.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:49:34 smithi047 ceph-mon[77147]: pgmap v8205: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:49:35.363 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:49:34 smithi115 conmon[90393]: 2022-05-14T03:49:34.905+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:49:35.364 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:49:34 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:49:35.366 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:49:34 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:49:35.366 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:49:34 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:49:35.366 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:49:34 smithi115 ceph-mon[78593]: pgmap v8205: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:49:36.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:49:35 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:49:36.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:49:35 smithi115 conmon[90393]: 2022-05-14T03:49:35.929+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:49:36.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:49:35 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:49:37.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:49:36 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:49:37.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:49:36 smithi047 ceph-mon[77147]: pgmap v8206: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:49:37.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:49:36 smithi115 conmon[90393]: 2022-05-14T03:49:36.935+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:49:37.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:49:36 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:49:37.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:49:36 smithi115 ceph-mon[78593]: pgmap v8206: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:49:38.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:49:37 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 15710 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:49:38.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:49:37 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:49:38.360 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:49:37 smithi115 conmon[90393]: 2022-05-14T03:49:37.958+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:49:38.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:49:37 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 15710 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:49:38.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:49:37 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:49:39.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:49:38 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:49:39.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:49:38 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:49:39.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:49:38 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:49:39.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:49:38 smithi047 ceph-mon[77147]: pgmap v8207: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:49:39.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:49:39 smithi115 conmon[90393]: 2022-05-14T03:49:39.002+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:49:39.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:49:38 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:49:39.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:49:38 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:49:39.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:49:38 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:49:39.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:49:38 smithi115 ceph-mon[78593]: pgmap v8207: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:49:40.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:49:39 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:49:40.367 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:49:39 smithi115 conmon[90393]: 2022-05-14T03:49:39.968+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:49:40.367 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:49:39 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:49:41.296 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:49:40 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:49:41.296 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:49:40 smithi047 ceph-mon[77147]: pgmap v8208: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:49:41.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:49:40 smithi115 conmon[90393]: 2022-05-14T03:49:40.936+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:49:41.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:49:40 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:49:41.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:49:40 smithi115 ceph-mon[78593]: pgmap v8208: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:49:42.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:49:41 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:49:42.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:49:41 smithi115 conmon[90393]: 2022-05-14T03:49:41.909+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:49:42.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:49:41 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:49:43.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:49:42 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:49:43.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:49:42 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:49:43.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:49:42 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:49:43.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:49:42 smithi047 ceph-mon[77147]: pgmap v8209: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:49:43.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:49:42 smithi115 conmon[90393]: 2022-05-14T03:49:42.952+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:49:43.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:49:42 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:49:43.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:49:42 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:49:43.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:49:42 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:49:43.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:49:42 smithi115 ceph-mon[78593]: pgmap v8209: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:49:44.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:49:43 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:49:44.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:49:43 smithi115 conmon[90393]: 2022-05-14T03:49:43.962+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:49:44.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:49:43 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:49:45.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:49:44 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:49:45.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:49:44 smithi047 ceph-mon[77147]: pgmap v8210: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:49:45.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:49:45 smithi115 conmon[90393]: 2022-05-14T03:49:45.001+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:49:45.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:49:44 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:49:45.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:49:44 smithi115 ceph-mon[78593]: pgmap v8210: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:49:46.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:49:45 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:49:46.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:49:45 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:49:46.360 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:49:45 smithi115 conmon[90393]: 2022-05-14T03:49:45.965+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:49:47.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:49:46 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:49:47.331 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:49:46 smithi047 ceph-mon[77147]: pgmap v8211: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:49:47.331 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:49:46 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 15715 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:49:47.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:49:46 smithi115 conmon[90393]: 2022-05-14T03:49:46.976+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:49:47.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:49:46 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:49:47.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:49:46 smithi115 ceph-mon[78593]: pgmap v8211: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:49:47.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:49:46 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 15715 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:49:48.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:49:47 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:49:48.360 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:49:47 smithi115 conmon[90393]: 2022-05-14T03:49:47.932+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:49:48.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:49:47 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:49:49.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:49:48 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:49:49.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:49:48 smithi047 ceph-mon[77147]: pgmap v8212: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:49:49.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:49:48 smithi115 conmon[90393]: 2022-05-14T03:49:48.949+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:49:49.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:49:48 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:49:49.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:49:48 smithi115 ceph-mon[78593]: pgmap v8212: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:49:50.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:49:49 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:49:50.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:49:49 smithi115 conmon[90393]: 2022-05-14T03:49:49.937+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:49:50.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:49:49 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:49:51.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:49:50 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:49:51.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:49:50 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:49:51.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:49:50 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:49:51.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:49:50 smithi047 ceph-mon[77147]: pgmap v8213: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:49:51.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:49:50 smithi115 conmon[90393]: 2022-05-14T03:49:50.911+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:49:51.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:49:50 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:49:51.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:49:50 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:49:51.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:49:50 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:49:51.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:49:50 smithi115 ceph-mon[78593]: pgmap v8213: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:49:52.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:49:51 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:49:52.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:49:51 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 15725 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:49:52.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:49:51 smithi115 conmon[90393]: 2022-05-14T03:49:51.956+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:49:52.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:49:51 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:49:52.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:49:51 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 15725 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:49:53.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:49:52 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:49:53.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:49:52 smithi047 ceph-mon[77147]: pgmap v8214: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:49:53.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:49:52 smithi115 conmon[90393]: 2022-05-14T03:49:52.982+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:49:53.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:49:52 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:49:53.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:49:52 smithi115 ceph-mon[78593]: pgmap v8214: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:49:54.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:49:53 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:49:54.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:49:54 smithi115 conmon[90393]: 2022-05-14T03:49:54.024+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:49:54.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:49:53 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:49:55.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:49:54 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:49:55.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:49:54 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:49:55.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:49:54 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:49:55.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:49:54 smithi047 ceph-mon[77147]: pgmap v8215: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:49:55.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:49:55 smithi115 conmon[90393]: 2022-05-14T03:49:55.026+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:49:55.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:49:54 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:49:55.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:49:54 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:49:55.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:49:54 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:49:55.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:49:54 smithi115 ceph-mon[78593]: pgmap v8215: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:49:56.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:49:55 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:49:56.360 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:49:56 smithi115 conmon[90393]: 2022-05-14T03:49:55.999+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:49:56.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:49:55 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:49:57.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:49:56 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:49:57.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:49:56 smithi047 ceph-mon[77147]: pgmap v8216: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:49:57.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:49:56 smithi115 conmon[90393]: 2022-05-14T03:49:56.970+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:49:57.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:49:56 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:49:57.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:49:56 smithi115 ceph-mon[78593]: pgmap v8216: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:49:58.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:49:57 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 15730 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:49:58.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:49:57 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:49:58.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:49:57 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 15730 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:49:58.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:49:57 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:49:58.361 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:49:57 smithi115 conmon[90393]: 2022-05-14T03:49:57.922+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:49:59.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:49:58 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:49:59.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:49:58 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:49:59.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:49:58 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:49:59.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:49:58 smithi047 ceph-mon[77147]: pgmap v8217: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:49:59.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:49:58 smithi115 conmon[90393]: 2022-05-14T03:49:58.895+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:49:59.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:49:58 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:49:59.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:49:58 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:49:59.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:49:58 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:49:59.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:49:58 smithi115 ceph-mon[78593]: pgmap v8217: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:50:00.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:49:59 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:50:00.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:50:00 smithi047 conmon[77123]: 2022-05-14T03:49:59.999+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : Health detail: HEALTH_ERR 1 filesystem is degraded; 1 mds daemon damaged; 1 MDSs are read only; 1 MDSs report slow metadata IOs; 1/3 mons down, quorum b,c; Reduced data availability: 1 pg inactive; 9 slow ops, oldest one blocked for 15730 sec, osd.5 has slow ops 2022-05-14T03:50:00.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:50:00 smithi047 conmon[77123]: 2022-05-14T03:49:59.999+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : [WRN] FS_DEGRADED: 1 filesystem is degraded 2022-05-14T03:50:00.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:50:00 smithi047 conmon[77123]: 2022-05-14T03:49:59.999+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : fs cephfs is degraded 2022-05-14T03:50:00.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:50:00 smithi047 conmon[77123]: 2022-05-14T03:49:59.999+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : [ERR] MDS_DAMAGE: 1 mds daemon damaged 2022-05-14T03:50:00.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:50:00 smithi047 conmon[77123]: 2022-05-14T03:49:59.999+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : fs cephfs mds.2 is damaged 2022-05-14T03:50:00.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:50:00 smithi047 conmon[77123]: 2022-05-14T03:49:59.999+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : [WRN] MDS_READ_ONLY: 1 MDSs are read only 2022-05-14T03:50:00.295 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:50:00 smithi047 conmon[77123]: 2022-05-14T03:49:59.999+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : mds.l(mds.0): MDS in read-only mode 2022-05-14T03:50:00.295 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:50:00 smithi047 conmon[77123]: 2022-05-14T03:49:59.999+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : [WRN] MDS_SLOW_METADATA_IO: 1 MDSs report slow metadata IOs 2022-05-14T03:50:00.295 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:50:00 smithi047 conmon[77123]: 2022-05-14T03:49:59.999+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : mds.l(mds.0): 4 slow metadata IOs are blocked > 180 secs, oldest blocked for 16033 secs 2022-05-14T03:50:00.295 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:50:00 smithi047 conmon[77123]: 2022-05-14T03:49:59.999+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : [WRN] MON_DOWN: 1/3 mons down, quorum b,c 2022-05-14T03:50:00.296 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:50:00 smithi047 conmon[77123]: 2022-05-14T03:49:59.999+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : mon.a (rank 0) addr [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] is down (out of quorum) 2022-05-14T03:50:00.296 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:50:00 smithi047 conmon[77123]: 2022-05-14T03:49:59.999+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : [WRN] PG_AVAILABILITY: Reduced data availability: 1 pg inactive 2022-05-14T03:50:00.296 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:50:00 smithi047 conmon[77123]: 2022-05-14T03:49:59.999+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : pg 2.1 is stuck inactive for 4h, current state unknown, last acting [] 2022-05-14T03:50:00.297 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:50:00 smithi047 conmon[77123]: 2022-05-14T03:49:59.999+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : [WRN] SLOW_OPS: 9 slow ops, oldest one blocked for 15730 sec, osd.5 has slow ops 2022-05-14T03:50:00.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:49:59 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:50:00.360 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:49:59 smithi115 conmon[90393]: 2022-05-14T03:49:59.873+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:50:01.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:50:00 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:50:01.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:50:00 smithi047 ceph-mon[77147]: Health detail: HEALTH_ERR 1 filesystem is degraded; 1 mds daemon damaged; 1 MDSs are read only; 1 MDSs report slow metadata IOs; 1/3 mons down, quorum b,c; Reduced data availability: 1 pg inactive; 9 slow ops, oldest one blocked for 15730 sec, osd.5 has slow ops 2022-05-14T03:50:01.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:50:00 smithi047 ceph-mon[77147]: [WRN] FS_DEGRADED: 1 filesystem is degraded 2022-05-14T03:50:01.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:50:00 smithi047 ceph-mon[77147]: fs cephfs is degraded 2022-05-14T03:50:01.295 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:50:00 smithi047 ceph-mon[77147]: [ERR] MDS_DAMAGE: 1 mds daemon damaged 2022-05-14T03:50:01.295 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:50:00 smithi047 ceph-mon[77147]: fs cephfs mds.2 is damaged 2022-05-14T03:50:01.295 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:50:00 smithi047 ceph-mon[77147]: [WRN] MDS_READ_ONLY: 1 MDSs are read only 2022-05-14T03:50:01.295 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:50:00 smithi047 ceph-mon[77147]: mds.l(mds.0): MDS in read-only mode 2022-05-14T03:50:01.296 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:50:00 smithi047 ceph-mon[77147]: [WRN] MDS_SLOW_METADATA_IO: 1 MDSs report slow metadata IOs 2022-05-14T03:50:01.296 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:50:00 smithi047 ceph-mon[77147]: mds.l(mds.0): 4 slow metadata IOs are blocked > 180 secs, oldest blocked for 16033 secs 2022-05-14T03:50:01.296 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:50:00 smithi047 ceph-mon[77147]: [WRN] MON_DOWN: 1/3 mons down, quorum b,c 2022-05-14T03:50:01.297 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:50:00 smithi047 ceph-mon[77147]: mon.a (rank 0) addr [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] is down (out of quorum) 2022-05-14T03:50:01.298 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:50:00 smithi047 ceph-mon[77147]: [WRN] PG_AVAILABILITY: Reduced data availability: 1 pg inactive 2022-05-14T03:50:01.298 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:50:00 smithi047 ceph-mon[77147]: pg 2.1 is stuck inactive for 4h, current state unknown, last acting [] 2022-05-14T03:50:01.298 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:50:00 smithi047 ceph-mon[77147]: [WRN] SLOW_OPS: 9 slow ops, oldest one blocked for 15730 sec, osd.5 has slow ops 2022-05-14T03:50:01.298 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:50:00 smithi047 ceph-mon[77147]: pgmap v8218: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:50:01.360 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:50:00 smithi115 conmon[90393]: 2022-05-14T03:50:00.916+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:50:01.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:50:00 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:50:01.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:50:00 smithi115 ceph-mon[78593]: Health detail: HEALTH_ERR 1 filesystem is degraded; 1 mds daemon damaged; 1 MDSs are read only; 1 MDSs report slow metadata IOs; 1/3 mons down, quorum b,c; Reduced data availability: 1 pg inactive; 9 slow ops, oldest one blocked for 15730 sec, osd.5 has slow ops 2022-05-14T03:50:01.363 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:50:00 smithi115 ceph-mon[78593]: [WRN] FS_DEGRADED: 1 filesystem is degraded 2022-05-14T03:50:01.363 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:50:00 smithi115 ceph-mon[78593]: fs cephfs is degraded 2022-05-14T03:50:01.364 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:50:00 smithi115 ceph-mon[78593]: [ERR] MDS_DAMAGE: 1 mds daemon damaged 2022-05-14T03:50:01.364 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:50:00 smithi115 ceph-mon[78593]: fs cephfs mds.2 is damaged 2022-05-14T03:50:01.364 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:50:00 smithi115 ceph-mon[78593]: [WRN] MDS_READ_ONLY: 1 MDSs are read only 2022-05-14T03:50:01.365 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:50:00 smithi115 ceph-mon[78593]: mds.l(mds.0): MDS in read-only mode 2022-05-14T03:50:01.365 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:50:00 smithi115 ceph-mon[78593]: [WRN] MDS_SLOW_METADATA_IO: 1 MDSs report slow metadata IOs 2022-05-14T03:50:01.366 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:50:00 smithi115 ceph-mon[78593]: mds.l(mds.0): 4 slow metadata IOs are blocked > 180 secs, oldest blocked for 16033 secs 2022-05-14T03:50:01.366 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:50:00 smithi115 ceph-mon[78593]: [WRN] MON_DOWN: 1/3 mons down, quorum b,c 2022-05-14T03:50:01.367 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:50:00 smithi115 ceph-mon[78593]: mon.a (rank 0) addr [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] is down (out of quorum) 2022-05-14T03:50:01.367 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:50:00 smithi115 ceph-mon[78593]: [WRN] PG_AVAILABILITY: Reduced data availability: 1 pg inactive 2022-05-14T03:50:01.368 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:50:00 smithi115 ceph-mon[78593]: pg 2.1 is stuck inactive for 4h, current state unknown, last acting [] 2022-05-14T03:50:01.368 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:50:00 smithi115 ceph-mon[78593]: [WRN] SLOW_OPS: 9 slow ops, oldest one blocked for 15730 sec, osd.5 has slow ops 2022-05-14T03:50:01.368 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:50:00 smithi115 ceph-mon[78593]: pgmap v8218: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:50:02.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:50:01 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:50:02.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:50:01 smithi115 conmon[90393]: 2022-05-14T03:50:01.953+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:50:02.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:50:01 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:50:03.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:50:02 smithi115 conmon[90393]: 2022-05-14T03:50:02.993+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:50:03.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:50:02 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:50:03.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:50:02 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:50:03.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:50:02 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:50:03.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:50:02 smithi115 ceph-mon[78593]: pgmap v8219: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:50:03.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:50:02 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T03:50:03.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:50:02 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T03:50:03.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:50:02 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T03:50:03.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:50:02 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:50:03.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:50:02 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:50:03.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:50:02 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:50:03.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:50:02 smithi047 ceph-mon[77147]: pgmap v8219: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:50:03.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:50:02 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T03:50:03.295 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:50:02 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T03:50:03.295 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:50:02 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T03:50:03.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:50:02 smithi047 conmon[78521]: 2022-05-14T03:50:02.927+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:50:03.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:50:02 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T03:50:03.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:50:02 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:50:03.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:50:02 smithi047 conmon[78521]: yield 2022-05-14T03:50:03.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:50:02 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:50:03.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:50:02 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:50:03.297 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:50:02 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:50:03.297 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:50:02 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T03:50:03.297 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:50:02 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:50:03.297 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:50:02 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T03:50:03.298 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:50:02 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:50:03.298 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:50:02 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T03:50:03.298 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:50:02 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:50:03.298 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:50:02 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T03:50:03.299 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:50:02 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:50:03.299 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:50:02 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T03:50:03.299 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:50:02 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:50:03.299 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:50:02 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:50:03.299 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:50:02 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:50:04.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:50:03 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:50:04.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T03:50:04.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:50:04.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T03:50:04.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:50:04.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:50:04.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:50:04.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T03:50:04.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:50:04.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T03:50:04.298 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:50:04.298 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T03:50:04.298 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:50:04.299 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T03:50:04.299 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:50:04.299 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T03:50:04.300 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:50:04.300 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:50:04.301 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:50:04.301 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:50:03 smithi047 ceph-mon[77147]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T03:50:04.301 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:50:03 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:50:04.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:50:03 smithi115 conmon[90393]: 2022-05-14T03:50:03.978+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:50:04.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:50:03 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:50:04.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T03:50:04.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:50:04.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T03:50:04.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:50:04.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:50:04.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:50:04.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T03:50:04.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:50:04.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T03:50:04.365 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:50:04.365 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T03:50:04.366 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:50:04.366 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T03:50:04.367 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:50:04.367 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T03:50:04.368 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:50:04.368 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:50:04.369 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:50:04.369 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:50:03 smithi115 ceph-mon[78593]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T03:50:04.370 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:50:03 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:50:05.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:50:04 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:50:05.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:50:04 smithi047 ceph-mon[77147]: pgmap v8220: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:50:05.360 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:50:05 smithi115 conmon[90393]: 2022-05-14T03:50:05.026+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:50:05.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:50:04 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:50:05.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:50:04 smithi115 ceph-mon[78593]: pgmap v8220: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:50:06.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:50:05 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:50:06.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:50:06 smithi047 conmon[78521]: 2022-05-14T03:50:05.999+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:50:06.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:50:06 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T03:50:06.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:50:06 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:50:06.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:50:06 smithi047 conmon[78521]: yield 2022-05-14T03:50:06.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:50:06 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:50:06.297 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:50:06 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:50:06.297 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:50:06 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:50:06.297 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:50:06 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T03:50:06.297 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:50:06 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:50:06.298 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:50:06 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T03:50:06.298 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:50:06 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:50:06.298 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:50:06 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T03:50:06.298 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:50:06 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:50:06.298 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:50:06 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T03:50:06.299 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:50:06 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:50:06.299 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:50:06 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T03:50:06.299 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:50:06 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:50:06.299 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:50:06 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:50:06.300 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:50:06 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:50:06.300 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:50:06 smithi047 conmon[78521]: 2022-05-14T03:50:06.000+0000 7f5341f7c700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T03:50:06.300 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:50:06 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T03:50:06.300 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:50:06 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:50:06.300 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:50:06 smithi047 conmon[78521]: yield 2022-05-14T03:50:06.301 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:50:06 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:50:06.301 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:50:06 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:50:06.301 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:50:06 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:50:06.301 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:50:06 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T03:50:06.302 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:50:06 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:50:06.302 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:50:06 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T03:50:06.302 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:50:06 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:50:06.302 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:50:06 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T03:50:06.302 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:50:06 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:50:06.303 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:50:06 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T03:50:06.303 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:50:06 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:50:06.303 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:50:06 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T03:50:06.304 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:50:06 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:50:06.304 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:50:06 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:50:06.304 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:50:06 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:50:06.304 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:50:06 smithi047 conmon[78521]: 2022-05-14T03:50:06.305 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:50:06 smithi047 conmon[78521]: During handling of the above exception, another exception occurred: 2022-05-14T03:50:06.305 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:50:06 smithi047 conmon[78521]: 2022-05-14T03:50:06.305 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:50:06 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T03:50:06.306 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:50:06 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T03:50:06.306 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:50:06 smithi047 conmon[78521]: return f(*arg) 2022-05-14T03:50:06.307 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:50:06 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T03:50:06.307 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:50:06 smithi047 conmon[78521]: self._write_client_files(client_files, host) 2022-05-14T03:50:06.307 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:50:06 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T03:50:06.307 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:50:06 smithi047 conmon[78521]: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T03:50:06.308 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:50:06 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T03:50:06.308 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:50:06 smithi047 conmon[78521]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T03:50:06.308 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:50:06 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T03:50:06.308 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:50:06 smithi047 conmon[78521]: return self.event_loop.get_result(coro) 2022-05-14T03:50:06.308 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:50:06 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T03:50:06.309 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:50:06 smithi047 conmon[78521]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T03:50:06.309 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:50:06 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T03:50:06.310 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:50:06 smithi047 conmon[78521]: return self.__get_result() 2022-05-14T03:50:06.310 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:50:06 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T03:50:06.310 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:50:06 smithi047 conmon[78521]: raise self._exception 2022-05-14T03:50:06.310 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:50:06 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T03:50:06.311 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:50:06 smithi047 conmon[78521]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T03:50:06.311 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:50:06 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T03:50:06.312 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:50:06 smithi047 conmon[78521]: conn = await self._remote_connection(host, addr) 2022-05-14T03:50:06.312 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:50:06 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T03:50:06.312 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:50:06 smithi047 conmon[78521]: raise 2022-05-14T03:50:06.312 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:50:06 smithi047 conmon[78521]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T03:50:06.313 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:50:06 smithi047 conmon[78521]: self.gen.throw(type, value, traceback) 2022-05-14T03:50:06.313 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:50:06 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T03:50:06.313 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:50:06 smithi047 conmon[78521]: raise OrchestratorError(msg) 2022-05-14T03:50:06.313 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:50:06 smithi047 conmon[78521]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:50:06.370 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:50:06 smithi115 conmon[90393]: 2022-05-14T03:50:06.017+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:50:06.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:50:05 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:50:07.295 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:50:06 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:50:07.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T03:50:07.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:50:07.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T03:50:07.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:50:07.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:50:07.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:50:07.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T03:50:07.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:50:07.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T03:50:07.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:50:07.298 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T03:50:07.298 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:50:07.298 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T03:50:07.298 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:50:07.299 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T03:50:07.299 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:50:07.299 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:50:07.299 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:50:07.300 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:50:06 smithi047 ceph-mon[77147]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T03:50:07.300 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T03:50:07.300 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:50:07.300 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T03:50:07.301 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:50:07.301 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:50:07.301 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:50:07.302 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T03:50:07.302 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:50:07.302 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T03:50:07.302 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:50:07.303 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T03:50:07.303 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:50:07.303 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T03:50:07.303 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:50:07.303 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T03:50:07.304 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:50:07.304 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:50:07.304 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:50:07.304 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T03:50:07.304 INFO:journalctl@ceph.mon.b.smithi047.stdout: During handling of the above exception, another exception occurred: 2022-05-14T03:50:07.305 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T03:50:07.305 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T03:50:07.305 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T03:50:07.305 INFO:journalctl@ceph.mon.b.smithi047.stdout: return f(*arg) 2022-05-14T03:50:07.306 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T03:50:07.306 INFO:journalctl@ceph.mon.b.smithi047.stdout: self._write_client_files(client_files, host) 2022-05-14T03:50:07.306 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T03:50:07.306 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T03:50:07.306 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T03:50:07.307 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T03:50:07.307 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T03:50:07.307 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.event_loop.get_result(coro) 2022-05-14T03:50:07.307 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T03:50:07.308 INFO:journalctl@ceph.mon.b.smithi047.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T03:50:07.308 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T03:50:07.308 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.__get_result() 2022-05-14T03:50:07.308 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T03:50:07.308 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise self._exception 2022-05-14T03:50:07.309 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T03:50:07.309 INFO:journalctl@ceph.mon.b.smithi047.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T03:50:07.309 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T03:50:07.309 INFO:journalctl@ceph.mon.b.smithi047.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T03:50:07.309 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T03:50:07.310 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise 2022-05-14T03:50:07.310 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T03:50:07.310 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.gen.throw(type, value, traceback) 2022-05-14T03:50:07.310 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T03:50:07.311 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OrchestratorError(msg) 2022-05-14T03:50:07.311 INFO:journalctl@ceph.mon.b.smithi047.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:50:07.311 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:50:06 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:50:07.311 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:50:06 smithi047 ceph-mon[77147]: pgmap v8221: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:50:07.311 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:50:06 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 15735 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:50:07.360 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:50:07 smithi115 conmon[90393]: 2022-05-14T03:50:07.017+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:50:07.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:50:06 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:50:07.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T03:50:07.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:50:07.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T03:50:07.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:50:07.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:50:07.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:50:07.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T03:50:07.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:50:07.365 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T03:50:07.368 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:50:07.369 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T03:50:07.370 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:50:07.370 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T03:50:07.370 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:50:07.371 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T03:50:07.371 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:50:07.372 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:50:07.372 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:50:07.373 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:50:06 smithi115 ceph-mon[78593]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T03:50:07.373 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T03:50:07.374 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:50:07.374 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T03:50:07.374 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:50:07.375 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:50:07.375 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:50:07.376 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T03:50:07.376 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:50:07.376 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T03:50:07.377 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:50:07.377 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T03:50:07.378 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:50:07.378 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T03:50:07.378 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:50:07.379 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T03:50:07.379 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:50:07.380 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:50:07.380 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:50:07.381 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T03:50:07.381 INFO:journalctl@ceph.mon.c.smithi115.stdout: During handling of the above exception, another exception occurred: 2022-05-14T03:50:07.381 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T03:50:07.381 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T03:50:07.382 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T03:50:07.382 INFO:journalctl@ceph.mon.c.smithi115.stdout: return f(*arg) 2022-05-14T03:50:07.382 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T03:50:07.382 INFO:journalctl@ceph.mon.c.smithi115.stdout: self._write_client_files(client_files, host) 2022-05-14T03:50:07.382 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T03:50:07.383 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T03:50:07.383 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T03:50:07.383 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T03:50:07.383 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T03:50:07.384 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.event_loop.get_result(coro) 2022-05-14T03:50:07.384 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T03:50:07.384 INFO:journalctl@ceph.mon.c.smithi115.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T03:50:07.384 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T03:50:07.384 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.__get_result() 2022-05-14T03:50:07.385 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T03:50:07.385 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise self._exception 2022-05-14T03:50:07.385 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T03:50:07.385 INFO:journalctl@ceph.mon.c.smithi115.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T03:50:07.386 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T03:50:07.386 INFO:journalctl@ceph.mon.c.smithi115.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T03:50:07.386 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T03:50:07.386 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise 2022-05-14T03:50:07.386 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T03:50:07.387 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.gen.throw(type, value, traceback) 2022-05-14T03:50:07.387 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T03:50:07.387 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OrchestratorError(msg) 2022-05-14T03:50:07.387 INFO:journalctl@ceph.mon.c.smithi115.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:50:07.387 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:50:06 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:50:07.388 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:50:06 smithi115 ceph-mon[78593]: pgmap v8221: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:50:07.388 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:50:06 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 15735 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:50:08.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:50:07 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:50:08.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:50:07 smithi115 conmon[90393]: 2022-05-14T03:50:07.979+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:50:08.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:50:07 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:50:09.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:50:09 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:50:09.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:50:09 smithi047 ceph-mon[77147]: pgmap v8222: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:50:09.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:50:09 smithi115 conmon[90393]: 2022-05-14T03:50:09.016+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:50:09.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:50:09 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:50:09.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:50:09 smithi115 ceph-mon[78593]: pgmap v8222: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:50:10.296 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:50:10 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:50:10.360 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:50:10 smithi115 conmon[90393]: 2022-05-14T03:50:10.011+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:50:10.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:50:10 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:50:11.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:50:11 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:50:11.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:50:11 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:50:11.295 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:50:11 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:50:11.295 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:50:11 smithi047 ceph-mon[77147]: pgmap v8223: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:50:11.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:50:10 smithi115 conmon[90393]: 2022-05-14T03:50:10.985+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:50:11.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:50:11 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:50:11.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:50:11 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:50:11.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:50:11 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:50:11.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:50:11 smithi115 ceph-mon[78593]: pgmap v8223: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:50:12.297 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:50:12 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:50:12.297 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:50:12 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 15745 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:50:12.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:50:11 smithi115 conmon[90393]: 2022-05-14T03:50:11.993+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:50:12.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:50:12 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:50:12.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:50:12 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 15745 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:50:13.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:50:13 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:50:13.295 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:50:13 smithi047 ceph-mon[77147]: pgmap v8224: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:50:13.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:50:13 smithi115 conmon[90393]: 2022-05-14T03:50:13.040+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:50:13.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:50:13 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:50:13.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:50:13 smithi115 ceph-mon[78593]: pgmap v8224: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:50:14.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:50:14 smithi115 conmon[90393]: 2022-05-14T03:50:14.089+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:50:15.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:50:15 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:50:15.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:50:15 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:50:15.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:50:15 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:50:15.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:50:15 smithi047 ceph-mon[77147]: pgmap v8225: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:50:15.295 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:50:15 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T03:50:15.295 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:50:15 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T03:50:15.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:50:15 smithi115 conmon[90393]: 2022-05-14T03:50:15.042+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:50:15.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:50:15 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:50:15.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:50:15 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:50:15.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:50:15 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:50:15.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:50:15 smithi115 ceph-mon[78593]: pgmap v8225: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:50:15.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:50:15 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T03:50:15.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:50:15 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T03:50:16.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:50:16 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:50:16.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:50:16 smithi115 conmon[90393]: 2022-05-14T03:50:16.037+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:50:16.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:50:16 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:50:17.296 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:50:17 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:50:17.296 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:50:17 smithi047 ceph-mon[77147]: pgmap v8226: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:50:17.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:50:17 smithi115 conmon[90393]: 2022-05-14T03:50:17.050+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:50:17.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:50:17 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:50:17.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:50:17 smithi115 ceph-mon[78593]: pgmap v8226: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:50:18.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:50:18 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:50:18.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:50:18 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 15749 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:50:18.364 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:50:18 smithi115 conmon[90393]: 2022-05-14T03:50:18.046+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:50:18.365 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:50:18 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:50:18.365 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:50:18 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 15749 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:50:19.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:50:19 smithi115 conmon[90393]: 2022-05-14T03:50:19.057+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:50:19.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:50:19 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:50:19.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:50:19 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:50:19.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:50:19 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:50:19.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:50:19 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:50:19.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:50:19 smithi115 ceph-mon[78593]: pgmap v8227: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:50:19.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:50:19 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:50:19.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:50:19 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:50:19.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:50:19 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:50:19.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:50:19 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:50:19.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:50:19 smithi047 ceph-mon[77147]: pgmap v8227: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:50:20.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:50:20 smithi115 conmon[90393]: 2022-05-14T03:50:20.057+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:50:20.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:50:20 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:50:20.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:50:20 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:50:21.360 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:50:21 smithi115 conmon[90393]: 2022-05-14T03:50:21.069+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:50:21.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:50:21 smithi115 ceph-mon[78593]: pgmap v8228: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:50:21.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:50:21 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:50:21.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:50:21 smithi047 ceph-mon[77147]: pgmap v8228: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:50:21.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:50:21 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:50:22.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:50:22 smithi115 conmon[90393]: 2022-05-14T03:50:22.020+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:50:23.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:50:22 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:50:23.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:50:22 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:50:23.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:50:22 smithi047 ceph-mon[77147]: pgmap v8229: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:50:23.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:50:22 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:50:23.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:50:22 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:50:23.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:50:23 smithi115 conmon[90393]: 2022-05-14T03:50:23.000+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:50:23.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:50:22 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:50:23.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:50:22 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:50:23.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:50:22 smithi115 ceph-mon[78593]: pgmap v8229: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:50:23.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:50:22 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:50:23.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:50:22 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:50:24.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:50:23 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:50:24.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:50:23 smithi115 conmon[90393]: 2022-05-14T03:50:23.974+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:50:24.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:50:23 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:50:25.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:50:24 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:50:25.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:50:24 smithi047 ceph-mon[77147]: pgmap v8230: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-05-14T03:50:25.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:50:24 smithi115 conmon[90393]: 2022-05-14T03:50:24.953+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:50:25.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:50:24 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:50:25.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:50:24 smithi115 ceph-mon[78593]: pgmap v8230: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-05-14T03:50:26.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:50:25 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:50:26.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:50:25 smithi115 conmon[90393]: 2022-05-14T03:50:25.957+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:50:26.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:50:25 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:50:27.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:50:26 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:50:27.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:50:26 smithi047 ceph-mon[77147]: pgmap v8231: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:50:27.295 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:50:26 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 15754 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:50:27.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:50:26 smithi115 conmon[90393]: 2022-05-14T03:50:26.995+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:50:27.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:50:26 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:50:27.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:50:26 smithi115 ceph-mon[78593]: pgmap v8231: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:50:27.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:50:26 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 15754 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:50:28.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:50:27 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:50:28.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:50:27 smithi115 conmon[90393]: 2022-05-14T03:50:27.946+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:50:28.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:50:27 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:50:29.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:50:28 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:50:29.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:50:28 smithi047 ceph-mon[77147]: pgmap v8232: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:50:29.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:50:28 smithi115 conmon[90393]: 2022-05-14T03:50:28.929+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:50:29.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:50:28 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:50:29.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:50:28 smithi115 ceph-mon[78593]: pgmap v8232: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:50:30.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:50:29 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:50:30.360 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:50:29 smithi115 conmon[90393]: 2022-05-14T03:50:29.915+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:50:30.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:50:29 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:50:31.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:50:30 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:50:31.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:50:30 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:50:31.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:50:30 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:50:31.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:50:30 smithi047 ceph-mon[77147]: pgmap v8233: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:50:31.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:50:30 smithi115 conmon[90393]: 2022-05-14T03:50:30.921+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:50:31.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:50:30 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:50:31.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:50:30 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:50:31.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:50:30 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:50:31.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:50:30 smithi115 ceph-mon[78593]: pgmap v8233: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:50:32.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:50:31 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:50:32.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:50:31 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 15765 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:50:32.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:50:31 smithi115 conmon[90393]: 2022-05-14T03:50:31.912+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:50:32.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:50:31 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:50:32.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:50:31 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 15765 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:50:33.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:50:32 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:50:33.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:50:32 smithi047 ceph-mon[77147]: pgmap v8234: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:50:33.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:50:32 smithi115 conmon[90393]: 2022-05-14T03:50:32.928+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:50:33.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:50:32 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:50:33.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:50:32 smithi115 ceph-mon[78593]: pgmap v8234: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:50:34.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:50:33 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:50:34.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:50:33 smithi115 conmon[90393]: 2022-05-14T03:50:33.930+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:50:34.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:50:33 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:50:35.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:50:34 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:50:35.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:50:34 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:50:35.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:50:34 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:50:35.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:50:34 smithi047 ceph-mon[77147]: pgmap v8235: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2022-05-14T03:50:35.360 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:50:34 smithi115 conmon[90393]: 2022-05-14T03:50:34.928+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:50:35.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:50:34 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:50:35.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:50:34 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:50:35.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:50:34 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:50:35.362 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:50:34 smithi115 ceph-mon[78593]: pgmap v8235: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2022-05-14T03:50:36.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:50:35 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:50:36.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:50:35 smithi115 conmon[90393]: 2022-05-14T03:50:35.970+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:50:36.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:50:35 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:50:37.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:50:36 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:50:37.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:50:36 smithi047 ceph-mon[77147]: pgmap v8236: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2022-05-14T03:50:37.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:50:36 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 15770 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:50:37.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:50:36 smithi115 conmon[90393]: 2022-05-14T03:50:36.996+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:50:37.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:50:36 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:50:37.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:50:36 smithi115 ceph-mon[78593]: pgmap v8236: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2022-05-14T03:50:37.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:50:36 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 15770 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:50:38.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:50:37 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:50:38.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:50:37 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:50:38.360 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:50:37 smithi115 conmon[90393]: 2022-05-14T03:50:37.961+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:50:39.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:50:38 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:50:39.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:50:38 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:50:39.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:50:38 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:50:39.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:50:38 smithi047 ceph-mon[77147]: pgmap v8237: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:50:39.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:50:38 smithi115 conmon[90393]: 2022-05-14T03:50:38.918+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:50:39.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:50:38 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:50:39.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:50:38 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:50:39.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:50:38 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:50:39.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:50:38 smithi115 ceph-mon[78593]: pgmap v8237: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:50:40.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:50:39 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:50:40.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:50:39 smithi115 conmon[90393]: 2022-05-14T03:50:39.952+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:50:40.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:50:39 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:50:41.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:50:40 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:50:41.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:50:40 smithi047 ceph-mon[77147]: pgmap v8238: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:50:41.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:50:40 smithi115 conmon[90393]: 2022-05-14T03:50:40.963+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:50:41.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:50:40 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:50:41.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:50:40 smithi115 ceph-mon[78593]: pgmap v8238: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:50:42.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:50:41 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:50:42.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:50:41 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 15775 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:50:42.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:50:41 smithi115 conmon[90393]: 2022-05-14T03:50:41.981+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:50:42.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:50:41 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:50:42.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:50:41 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 15775 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:50:43.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:50:42 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:50:43.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:50:42 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:50:43.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:50:42 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:50:43.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:50:42 smithi047 ceph-mon[77147]: pgmap v8239: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:50:43.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:50:43 smithi115 conmon[90393]: 2022-05-14T03:50:43.008+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:50:43.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:50:42 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:50:43.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:50:42 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:50:43.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:50:42 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:50:43.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:50:42 smithi115 ceph-mon[78593]: pgmap v8239: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:50:44.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:50:43 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:50:44.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:50:44 smithi115 conmon[90393]: 2022-05-14T03:50:44.008+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:50:44.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:50:43 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:50:45.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:50:44 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:50:45.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:50:44 smithi047 ceph-mon[77147]: pgmap v8240: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2022-05-14T03:50:45.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:50:44 smithi115 conmon[90393]: 2022-05-14T03:50:44.964+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:50:45.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:50:44 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:50:45.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:50:44 smithi115 ceph-mon[78593]: pgmap v8240: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2022-05-14T03:50:46.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:50:45 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:50:46.361 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:50:45 smithi115 conmon[90393]: 2022-05-14T03:50:45.970+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:50:46.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:50:45 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:50:47.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:50:46 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:50:47.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:50:46 smithi047 ceph-mon[77147]: pgmap v8241: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2022-05-14T03:50:47.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:50:46 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 15780 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:50:47.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:50:47 smithi115 conmon[90393]: 2022-05-14T03:50:47.010+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:50:47.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:50:46 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:50:47.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:50:46 smithi115 ceph-mon[78593]: pgmap v8241: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2022-05-14T03:50:47.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:50:46 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 15780 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:50:48.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:50:47 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:50:48.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:50:47 smithi115 conmon[90393]: 2022-05-14T03:50:47.990+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:50:48.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:50:47 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:50:49.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:50:48 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:50:49.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:50:48 smithi047 ceph-mon[77147]: pgmap v8242: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:50:49.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:50:48 smithi115 conmon[90393]: 2022-05-14T03:50:48.969+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:50:49.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:50:48 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:50:49.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:50:48 smithi115 ceph-mon[78593]: pgmap v8242: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:50:50.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:50:49 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:50:50.361 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:50:49 smithi115 conmon[90393]: 2022-05-14T03:50:49.944+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:50:50.362 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:50:49 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:50:51.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:50:50 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:50:51.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:50:50 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:50:51.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:50:50 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:50:51.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:50:50 smithi047 ceph-mon[77147]: pgmap v8243: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:50:51.360 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:50:50 smithi115 conmon[90393]: 2022-05-14T03:50:50.987+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:50:51.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:50:50 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:50:51.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:50:50 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:50:51.362 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:50:50 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:50:51.362 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:50:50 smithi115 ceph-mon[78593]: pgmap v8243: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:50:52.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:50:51 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:50:52.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:50:51 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 15785 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:50:52.360 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:50:51 smithi115 conmon[90393]: 2022-05-14T03:50:51.996+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:50:52.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:50:51 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:50:52.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:50:51 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 15785 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:50:53.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:50:52 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:50:53.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:50:52 smithi047 ceph-mon[77147]: pgmap v8244: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:50:53.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:50:52 smithi115 conmon[90393]: 2022-05-14T03:50:52.962+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:50:53.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:50:52 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:50:53.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:50:52 smithi115 ceph-mon[78593]: pgmap v8244: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:50:54.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:50:53 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:50:54.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:50:53 smithi115 conmon[90393]: 2022-05-14T03:50:53.939+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:50:54.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:50:53 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:50:55.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:50:55 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:50:55.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:50:55 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:50:55.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:50:55 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:50:55.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:50:55 smithi047 ceph-mon[77147]: pgmap v8245: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2022-05-14T03:50:55.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:50:54 smithi115 conmon[90393]: 2022-05-14T03:50:54.910+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:50:55.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:50:55 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:50:55.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:50:55 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:50:55.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:50:55 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:50:55.362 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:50:55 smithi115 ceph-mon[78593]: pgmap v8245: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2022-05-14T03:50:56.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:50:56 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:50:56.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:50:55 smithi115 conmon[90393]: 2022-05-14T03:50:55.911+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:50:56.362 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:50:56 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:50:57.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:50:57 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:50:57.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:50:57 smithi047 ceph-mon[77147]: pgmap v8246: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2022-05-14T03:50:57.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:50:57 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 15790 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:50:57.360 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:50:56 smithi115 conmon[90393]: 2022-05-14T03:50:56.911+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:50:57.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:50:57 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:50:57.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:50:57 smithi115 ceph-mon[78593]: pgmap v8246: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2022-05-14T03:50:57.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:50:57 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 15790 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:50:58.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:50:58 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:50:58.360 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:50:57 smithi115 conmon[90393]: 2022-05-14T03:50:57.936+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:50:58.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:50:58 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:50:59.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:50:59 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:50:59.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:50:59 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:50:59.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:50:59 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:50:59.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:50:59 smithi047 ceph-mon[77147]: pgmap v8247: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:50:59.360 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:50:58 smithi115 conmon[90393]: 2022-05-14T03:50:58.930+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:50:59.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:50:59 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:50:59.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:50:59 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:50:59.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:50:59 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:50:59.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:50:59 smithi115 ceph-mon[78593]: pgmap v8247: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:51:00.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:51:00 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:51:00.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:50:59 smithi115 conmon[90393]: 2022-05-14T03:50:59.908+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:51:00.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:51:00 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:51:01.298 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:51:01 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:51:01.298 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:51:01 smithi047 ceph-mon[77147]: pgmap v8248: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:51:01.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:51:00 smithi115 conmon[90393]: 2022-05-14T03:51:00.883+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:51:01.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:51:01 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:51:01.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:51:01 smithi115 ceph-mon[78593]: pgmap v8248: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:51:02.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:51:02 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:51:02.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:51:02 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 15795 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:51:02.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:51:01 smithi115 conmon[90393]: 2022-05-14T03:51:01.931+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:51:02.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:51:02 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:51:02.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:51:02 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 15795 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:51:03.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:51:03 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:51:03.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:51:03 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:51:03.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:51:03 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:51:03.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:51:03 smithi047 ceph-mon[77147]: pgmap v8249: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:51:03.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:51:02 smithi115 conmon[90393]: 2022-05-14T03:51:02.969+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:51:03.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:51:03 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:51:03.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:51:03 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:51:03.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:51:03 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:51:03.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:51:03 smithi115 ceph-mon[78593]: pgmap v8249: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:51:04.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:51:04 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:51:04.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:51:04 smithi115 conmon[90393]: 2022-05-14T03:51:04.019+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:51:04.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:51:04 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:51:05.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:51:05 smithi115 conmon[90393]: 2022-05-14T03:51:05.045+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:51:05.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:51:05 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:51:05.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:51:05 smithi115 ceph-mon[78593]: pgmap v8250: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2022-05-14T03:51:05.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:51:05 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:51:05.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:51:05 smithi047 ceph-mon[77147]: pgmap v8250: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2022-05-14T03:51:06.360 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:51:05 smithi115 conmon[90393]: 2022-05-14T03:51:05.998+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:51:06.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:51:06 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T03:51:06.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:51:06 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T03:51:06.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:51:06 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T03:51:06.530 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:51:06 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T03:51:06.530 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:51:06 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T03:51:06.531 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:51:06 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T03:51:07.360 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:51:06 smithi115 conmon[90393]: 2022-05-14T03:51:06.962+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:51:07.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:51:07 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:51:07.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:51:07 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:51:07.362 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:51:07 smithi115 ceph-mon[78593]: pgmap v8251: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2022-05-14T03:51:07.362 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:51:07 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 15800 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:51:07.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:51:07 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:51:07.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:51:07 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:51:07.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:51:07 smithi047 ceph-mon[77147]: pgmap v8251: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2022-05-14T03:51:07.545 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:51:07 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 15800 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:51:08.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:51:08 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:51:08.360 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:51:08 smithi115 conmon[90393]: 2022-05-14T03:51:08.011+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:51:08.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:51:08 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:51:09.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:51:09 smithi115 conmon[90393]: 2022-05-14T03:51:09.004+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:51:09.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:51:09 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:51:09.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:51:09 smithi115 ceph-mon[78593]: pgmap v8252: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:51:09.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:51:09 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:51:09.547 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:51:09 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:51:09.547 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:51:09 smithi047 ceph-mon[77147]: pgmap v8252: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:51:09.547 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:51:09 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:51:09.548 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:51:09 smithi047 conmon[78521]: 2022-05-14T03:51:09.104+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:51:09.548 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:51:09 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T03:51:09.548 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:51:09 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:51:09.548 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:51:09 smithi047 conmon[78521]: yield 2022-05-14T03:51:09.549 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:51:09 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:51:09.549 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:51:09 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:51:09.549 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:51:09 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:51:09.549 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:51:09 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T03:51:09.550 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:51:09 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:51:09.550 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:51:09 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T03:51:09.550 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:51:09 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:51:09.550 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:51:09 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T03:51:09.550 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:51:09 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:51:09.551 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:51:09 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T03:51:09.551 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:51:09 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:51:09.552 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:51:09 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T03:51:09.552 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:51:09 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:51:09.552 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:51:09 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:51:09.553 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:51:09 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:51:10.360 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:51:10 smithi115 conmon[90393]: 2022-05-14T03:51:10.002+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:51:10.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:51:10 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:51:10.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T03:51:10.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:51:10.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T03:51:10.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:51:10.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:51:10.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:51:10.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T03:51:10.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:51:10.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T03:51:10.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:51:10.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T03:51:10.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:51:10.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T03:51:10.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:51:10.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T03:51:10.365 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:51:10.365 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:51:10.365 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:51:10.366 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:51:10 smithi115 ceph-mon[78593]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T03:51:10.366 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:51:10 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:51:10.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:51:10 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:51:10.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T03:51:10.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:51:10.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T03:51:10.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:51:10.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:51:10.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:51:10.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T03:51:10.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:51:10.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T03:51:10.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:51:10.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T03:51:10.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:51:10.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T03:51:10.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:51:10.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T03:51:10.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:51:10.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:51:10.548 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:51:10.548 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:51:10 smithi047 ceph-mon[77147]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T03:51:10.548 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:51:10 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:51:11.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:51:10 smithi115 conmon[90393]: 2022-05-14T03:51:10.959+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:51:11.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:51:11 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:51:11.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:51:11 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:51:11.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:51:11 smithi115 ceph-mon[78593]: pgmap v8253: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:51:11.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:51:11 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:51:11.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:51:11 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:51:11.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:51:11 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:51:11.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:51:11 smithi047 ceph-mon[77147]: pgmap v8253: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:51:11.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:51:11 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:51:12.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:51:12 smithi115 conmon[90393]: 2022-05-14T03:51:12.005+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:51:12.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:51:12 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 15805 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:51:12.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:51:12 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:51:12.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:51:12 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 15805 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:51:12.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:51:12 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:51:12.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:51:12 smithi047 conmon[78521]: 2022-05-14T03:51:12.176+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:51:12.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:51:12 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T03:51:12.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:51:12 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:51:12.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:51:12 smithi047 conmon[78521]: yield 2022-05-14T03:51:12.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:51:12 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:51:12.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:51:12 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:51:12.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:51:12 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:51:12.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:51:12 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T03:51:12.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:51:12 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:51:12.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:51:12 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T03:51:12.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:51:12 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:51:12.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:51:12 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T03:51:12.547 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:51:12 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:51:12.547 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:51:12 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T03:51:12.547 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:51:12 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:51:12.547 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:51:12 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T03:51:12.547 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:51:12 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:51:12.548 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:51:12 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:51:12.548 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:51:12 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:51:12.548 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:51:12 smithi047 conmon[78521]: 2022-05-14T03:51:12.177+0000 7f5344f82700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T03:51:12.548 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:51:12 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T03:51:12.548 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:51:12 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:51:12.549 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:51:12 smithi047 conmon[78521]: yield 2022-05-14T03:51:12.549 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:51:12 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:51:12.549 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:51:12 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:51:12.549 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:51:12 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:51:12.550 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:51:12 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T03:51:12.550 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:51:12 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:51:12.550 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:51:12 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T03:51:12.550 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:51:12 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:51:12.550 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:51:12 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T03:51:12.551 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:51:12 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:51:12.551 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:51:12 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T03:51:12.551 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:51:12 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:51:12.551 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:51:12 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T03:51:12.551 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:51:12 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:51:12.552 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:51:12 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:51:12.552 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:51:12 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:51:12.552 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:51:12 smithi047 conmon[78521]: 2022-05-14T03:51:12.552 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:51:12 smithi047 conmon[78521]: During handling of the above exception, another exception occurred: 2022-05-14T03:51:12.553 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:51:12 smithi047 conmon[78521]: 2022-05-14T03:51:12.553 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:51:12 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T03:51:12.553 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:51:12 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T03:51:12.553 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:51:12 smithi047 conmon[78521]: return f(*arg) 2022-05-14T03:51:12.553 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:51:12 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T03:51:12.554 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:51:12 smithi047 conmon[78521]: self._write_client_files(client_files, host) 2022-05-14T03:51:12.554 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:51:12 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T03:51:12.554 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:51:12 smithi047 conmon[78521]: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T03:51:12.554 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:51:12 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T03:51:12.555 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:51:12 smithi047 conmon[78521]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T03:51:12.555 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:51:12 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T03:51:12.555 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:51:12 smithi047 conmon[78521]: return self.event_loop.get_result(coro) 2022-05-14T03:51:12.555 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:51:12 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T03:51:12.555 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:51:12 smithi047 conmon[78521]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T03:51:12.556 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:51:12 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T03:51:12.556 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:51:12 smithi047 conmon[78521]: return self.__get_result() 2022-05-14T03:51:12.556 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:51:12 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T03:51:12.556 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:51:12 smithi047 conmon[78521]: raise self._exception 2022-05-14T03:51:12.556 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:51:12 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T03:51:12.557 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:51:12 smithi047 conmon[78521]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T03:51:12.557 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:51:12 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T03:51:12.557 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:51:12 smithi047 conmon[78521]: conn = await self._remote_connection(host, addr) 2022-05-14T03:51:12.557 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:51:12 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T03:51:12.558 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:51:12 smithi047 conmon[78521]: raise 2022-05-14T03:51:12.558 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:51:12 smithi047 conmon[78521]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T03:51:12.558 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:51:12 smithi047 conmon[78521]: self.gen.throw(type, value, traceback) 2022-05-14T03:51:12.558 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:51:12 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T03:51:12.558 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:51:12 smithi047 conmon[78521]: raise OrchestratorError(msg) 2022-05-14T03:51:12.559 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:51:12 smithi047 conmon[78521]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:51:13.362 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:51:13 smithi115 conmon[90393]: 2022-05-14T03:51:13.006+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:51:13.362 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:51:13 smithi115 ceph-mon[78593]: pgmap v8254: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:51:13.363 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:51:13 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:51:13.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:51:13 smithi047 ceph-mon[77147]: pgmap v8254: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:51:13.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:51:13 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:51:14.360 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:51:14 smithi115 conmon[90393]: 2022-05-14T03:51:14.034+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:51:14.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:51:14 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:51:14.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T03:51:14.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:51:14.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T03:51:14.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:51:14.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:51:14.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:51:14.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T03:51:14.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:51:14.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T03:51:14.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:51:14.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T03:51:14.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:51:14.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T03:51:14.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:51:14.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T03:51:14.365 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:51:14.365 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:51:14.365 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:51:14.366 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:51:14 smithi115 ceph-mon[78593]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T03:51:14.366 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T03:51:14.366 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:51:14.366 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T03:51:14.367 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:51:14.367 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:51:14.367 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:51:14.367 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T03:51:14.367 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:51:14.368 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T03:51:14.368 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:51:14.368 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T03:51:14.368 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:51:14.369 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T03:51:14.369 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:51:14.369 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T03:51:14.369 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:51:14.370 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:51:14.370 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:51:14.371 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T03:51:14.371 INFO:journalctl@ceph.mon.c.smithi115.stdout: During handling of the above exception, another exception occurred: 2022-05-14T03:51:14.371 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T03:51:14.372 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T03:51:14.372 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T03:51:14.372 INFO:journalctl@ceph.mon.c.smithi115.stdout: return f(*arg) 2022-05-14T03:51:14.372 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T03:51:14.372 INFO:journalctl@ceph.mon.c.smithi115.stdout: self._write_client_files(client_files, host) 2022-05-14T03:51:14.373 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T03:51:14.373 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T03:51:14.373 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T03:51:14.373 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T03:51:14.374 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T03:51:14.374 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.event_loop.get_result(coro) 2022-05-14T03:51:14.374 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T03:51:14.374 INFO:journalctl@ceph.mon.c.smithi115.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T03:51:14.374 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T03:51:14.375 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.__get_result() 2022-05-14T03:51:14.375 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T03:51:14.375 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise self._exception 2022-05-14T03:51:14.375 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T03:51:14.376 INFO:journalctl@ceph.mon.c.smithi115.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T03:51:14.376 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T03:51:14.376 INFO:journalctl@ceph.mon.c.smithi115.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T03:51:14.376 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T03:51:14.376 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise 2022-05-14T03:51:14.377 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T03:51:14.377 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.gen.throw(type, value, traceback) 2022-05-14T03:51:14.377 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T03:51:14.377 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OrchestratorError(msg) 2022-05-14T03:51:14.378 INFO:journalctl@ceph.mon.c.smithi115.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:51:14.378 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:51:14 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:51:14.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:51:14 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:51:14.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T03:51:14.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:51:14.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T03:51:14.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:51:14.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:51:14.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:51:14.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T03:51:14.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:51:14.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T03:51:14.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:51:14.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T03:51:14.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:51:14.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T03:51:14.548 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:51:14.548 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T03:51:14.548 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:51:14.548 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:51:14.549 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:51:14.549 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:51:14 smithi047 ceph-mon[77147]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T03:51:14.549 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T03:51:14.549 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:51:14.551 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T03:51:14.551 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:51:14.551 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:51:14.552 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:51:14.552 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T03:51:14.552 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:51:14.553 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T03:51:14.554 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:51:14.554 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T03:51:14.554 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:51:14.554 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T03:51:14.554 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:51:14.555 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T03:51:14.555 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:51:14.555 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:51:14.555 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:51:14.555 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T03:51:14.556 INFO:journalctl@ceph.mon.b.smithi047.stdout: During handling of the above exception, another exception occurred: 2022-05-14T03:51:14.556 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T03:51:14.556 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T03:51:14.556 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T03:51:14.557 INFO:journalctl@ceph.mon.b.smithi047.stdout: return f(*arg) 2022-05-14T03:51:14.557 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T03:51:14.557 INFO:journalctl@ceph.mon.b.smithi047.stdout: self._write_client_files(client_files, host) 2022-05-14T03:51:14.557 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T03:51:14.557 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T03:51:14.558 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T03:51:14.558 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T03:51:14.558 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T03:51:14.558 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.event_loop.get_result(coro) 2022-05-14T03:51:14.559 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T03:51:14.559 INFO:journalctl@ceph.mon.b.smithi047.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T03:51:14.559 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T03:51:14.559 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.__get_result() 2022-05-14T03:51:14.559 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T03:51:14.560 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise self._exception 2022-05-14T03:51:14.560 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T03:51:14.560 INFO:journalctl@ceph.mon.b.smithi047.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T03:51:14.560 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T03:51:14.560 INFO:journalctl@ceph.mon.b.smithi047.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T03:51:14.561 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T03:51:14.561 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise 2022-05-14T03:51:14.561 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T03:51:14.561 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.gen.throw(type, value, traceback) 2022-05-14T03:51:14.561 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T03:51:14.562 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OrchestratorError(msg) 2022-05-14T03:51:14.562 INFO:journalctl@ceph.mon.b.smithi047.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:51:14.562 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:51:14 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:51:15.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:51:15 smithi115 conmon[90393]: 2022-05-14T03:51:15.034+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:51:15.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:51:15 smithi115 ceph-mon[78593]: pgmap v8255: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2022-05-14T03:51:15.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:51:15 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:51:15.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:51:15 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:51:15.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:51:15 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T03:51:15.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:51:15 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T03:51:15.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:51:15 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:51:15.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:51:15 smithi047 ceph-mon[77147]: pgmap v8255: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2022-05-14T03:51:15.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:51:15 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:51:15.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:51:15 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:51:15.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:51:15 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T03:51:15.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:51:15 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T03:51:15.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:51:15 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:51:16.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:51:16 smithi115 conmon[90393]: 2022-05-14T03:51:16.026+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:51:16.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:51:16 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:51:16.547 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:51:16 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:51:17.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:51:16 smithi115 conmon[90393]: 2022-05-14T03:51:16.979+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:51:17.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:51:17 smithi115 ceph-mon[78593]: pgmap v8256: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2022-05-14T03:51:17.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:51:17 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 15810 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:51:17.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:51:17 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:51:17.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:51:17 smithi047 ceph-mon[77147]: pgmap v8256: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2022-05-14T03:51:17.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:51:17 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 15810 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:51:17.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:51:17 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:51:18.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:51:17 smithi115 conmon[90393]: 2022-05-14T03:51:17.982+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:51:18.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:51:18 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:51:18.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:51:18 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:51:19.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:51:18 smithi115 conmon[90393]: 2022-05-14T03:51:18.952+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:51:19.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:51:19 smithi115 ceph-mon[78593]: pgmap v8257: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:51:19.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:51:19 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:51:19.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:51:19 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:51:19.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:51:19 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:51:19.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:51:19 smithi047 ceph-mon[77147]: pgmap v8257: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:51:19.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:51:19 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:51:19.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:51:19 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:51:19.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:51:19 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:51:20.360 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:51:19 smithi115 conmon[90393]: 2022-05-14T03:51:19.914+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:51:20.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:51:20 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:51:20.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:51:20 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:51:21.360 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:51:20 smithi115 conmon[90393]: 2022-05-14T03:51:20.884+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:51:21.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:51:21 smithi115 ceph-mon[78593]: pgmap v8258: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:51:21.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:51:21 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:51:21.546 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:51:21 smithi047 ceph-mon[77147]: pgmap v8258: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:51:21.546 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:51:21 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:51:22.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:51:21 smithi115 conmon[90393]: 2022-05-14T03:51:21.853+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:51:22.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:51:22 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:51:22.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:51:22 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:51:22.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:51:22 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:51:22.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:51:22 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 15815 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:51:22.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:51:22 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:51:22.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:51:22 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:51:22.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:51:22 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:51:22.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:51:22 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 15815 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:51:23.360 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:51:22 smithi115 conmon[90393]: 2022-05-14T03:51:22.888+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:51:23.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:51:23 smithi115 ceph-mon[78593]: pgmap v8259: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:51:23.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:51:23 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:51:23.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:51:23 smithi047 ceph-mon[77147]: pgmap v8259: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:51:23.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:51:23 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:51:24.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:51:23 smithi115 conmon[90393]: 2022-05-14T03:51:23.900+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:51:24.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:51:24 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:51:24.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:51:24 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:51:25.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:51:24 smithi115 conmon[90393]: 2022-05-14T03:51:24.872+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:51:25.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:51:25 smithi115 ceph-mon[78593]: pgmap v8260: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2022-05-14T03:51:25.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:51:25 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:51:25.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:51:25 smithi047 ceph-mon[77147]: pgmap v8260: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2022-05-14T03:51:25.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:51:25 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:51:26.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:51:25 smithi115 conmon[90393]: 2022-05-14T03:51:25.876+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:51:26.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:51:26 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:51:26.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:51:26 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:51:27.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:51:26 smithi115 conmon[90393]: 2022-05-14T03:51:26.896+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:51:27.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:51:27 smithi115 ceph-mon[78593]: pgmap v8261: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2022-05-14T03:51:27.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:51:27 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 15820 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:51:27.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:51:27 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:51:27.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:51:27 smithi047 ceph-mon[77147]: pgmap v8261: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2022-05-14T03:51:27.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:51:27 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 15820 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:51:27.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:51:27 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:51:28.360 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:51:27 smithi115 conmon[90393]: 2022-05-14T03:51:27.885+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:51:28.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:51:28 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:51:28.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:51:28 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:51:29.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:51:28 smithi115 conmon[90393]: 2022-05-14T03:51:28.875+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:51:29.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:51:29 smithi115 ceph-mon[78593]: pgmap v8262: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:51:29.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:51:29 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:51:29.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:51:29 smithi047 ceph-mon[77147]: pgmap v8262: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:51:29.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:51:29 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:51:30.110 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:51:29 smithi115 conmon[90393]: 2022-05-14T03:51:29.855+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:51:30.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:51:30 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:51:30.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:51:30 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:51:30.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:51:30 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:51:30.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:51:30 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:51:30.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:51:30 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:51:30.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:51:30 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:51:31.110 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:51:30 smithi115 conmon[90393]: 2022-05-14T03:51:30.850+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:51:31.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:51:31 smithi047 ceph-mon[77147]: pgmap v8263: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:51:31.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:51:31 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:51:31.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:51:31 smithi115 ceph-mon[78593]: pgmap v8263: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:51:31.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:51:31 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:51:32.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:51:31 smithi115 conmon[90393]: 2022-05-14T03:51:31.846+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:51:32.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:51:32 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:51:32.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:51:32 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 15825 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:51:32.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:51:32 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:51:32.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:51:32 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 15825 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:51:33.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:51:32 smithi115 conmon[90393]: 2022-05-14T03:51:32.829+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:51:33.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:51:33 smithi047 ceph-mon[77147]: pgmap v8264: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:51:33.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:51:33 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:51:33.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:51:33 smithi115 ceph-mon[78593]: pgmap v8264: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:51:33.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:51:33 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:51:34.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:51:33 smithi115 conmon[90393]: 2022-05-14T03:51:33.840+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:51:35.110 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:51:34 smithi115 conmon[90393]: 2022-05-14T03:51:34.854+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:51:35.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:51:34 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:51:35.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:51:34 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:51:35.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:51:34 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:51:35.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:51:34 smithi115 ceph-mon[78593]: pgmap v8265: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2022-05-14T03:51:35.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:51:34 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:51:35.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:51:34 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:51:35.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:51:34 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:51:35.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:51:34 smithi047 ceph-mon[77147]: pgmap v8265: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2022-05-14T03:51:36.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:51:35 smithi115 conmon[90393]: 2022-05-14T03:51:35.826+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:51:36.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:51:35 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:51:36.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:51:35 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:51:37.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:51:36 smithi115 conmon[90393]: 2022-05-14T03:51:36.819+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:51:37.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:51:36 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:51:37.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:51:36 smithi115 ceph-mon[78593]: pgmap v8266: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2022-05-14T03:51:37.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:51:36 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 15830 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:51:37.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:51:36 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:51:37.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:51:36 smithi047 ceph-mon[77147]: pgmap v8266: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2022-05-14T03:51:37.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:51:36 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 15830 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:51:38.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:51:37 smithi115 conmon[90393]: 2022-05-14T03:51:37.784+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:51:38.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:51:37 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:51:38.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:51:37 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:51:39.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:51:38 smithi115 conmon[90393]: 2022-05-14T03:51:38.744+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:51:39.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:51:38 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:51:39.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:51:38 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:51:39.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:51:38 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:51:39.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:51:38 smithi115 ceph-mon[78593]: pgmap v8267: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:51:39.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:51:38 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:51:39.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:51:38 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:51:39.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:51:38 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:51:39.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:51:38 smithi047 ceph-mon[77147]: pgmap v8267: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:51:40.110 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:51:39 smithi115 conmon[90393]: 2022-05-14T03:51:39.769+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:51:40.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:51:39 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:51:40.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:51:39 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:51:41.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:51:40 smithi115 conmon[90393]: 2022-05-14T03:51:40.757+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:51:41.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:51:40 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:51:41.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:51:40 smithi115 ceph-mon[78593]: pgmap v8268: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:51:41.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:51:40 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:51:41.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:51:40 smithi047 ceph-mon[77147]: pgmap v8268: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:51:42.111 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:51:41 smithi115 conmon[90393]: 2022-05-14T03:51:41.749+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:51:42.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:51:41 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:51:42.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:51:41 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 15835 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:51:42.295 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:51:41 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:51:42.295 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:51:41 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 15835 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:51:43.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:51:42 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:51:43.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:51:42 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:51:43.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:51:42 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:51:43.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:51:42 smithi115 ceph-mon[78593]: pgmap v8269: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:51:43.111 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:51:42 smithi115 conmon[90393]: 2022-05-14T03:51:42.719+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:51:43.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:51:42 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:51:43.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:51:42 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:51:43.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:51:42 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:51:43.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:51:42 smithi047 ceph-mon[77147]: pgmap v8269: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:51:44.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:51:43 smithi115 conmon[90393]: 2022-05-14T03:51:43.721+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:51:44.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:51:43 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:51:44.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:51:43 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:51:45.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:51:44 smithi115 conmon[90393]: 2022-05-14T03:51:44.727+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:51:45.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:51:44 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:51:45.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:51:44 smithi115 ceph-mon[78593]: pgmap v8270: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2022-05-14T03:51:45.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:51:44 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:51:45.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:51:44 smithi047 ceph-mon[77147]: pgmap v8270: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2022-05-14T03:51:46.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:51:45 smithi115 conmon[90393]: 2022-05-14T03:51:45.765+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:51:46.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:51:45 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:51:46.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:51:45 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:51:47.112 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:51:46 smithi115 conmon[90393]: 2022-05-14T03:51:46.796+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:51:47.113 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:51:46 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:51:47.113 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:51:46 smithi115 ceph-mon[78593]: pgmap v8271: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2022-05-14T03:51:47.114 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:51:46 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 15840 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:51:47.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:51:46 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:51:47.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:51:46 smithi047 ceph-mon[77147]: pgmap v8271: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2022-05-14T03:51:47.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:51:46 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 15840 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:51:48.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:51:47 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:51:48.111 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:51:47 smithi115 conmon[90393]: 2022-05-14T03:51:47.783+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:51:48.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:51:47 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:51:49.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:51:48 smithi115 conmon[90393]: 2022-05-14T03:51:48.746+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:51:49.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:51:48 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:51:49.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:51:48 smithi115 ceph-mon[78593]: pgmap v8272: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:51:49.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:51:48 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:51:49.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:51:48 smithi047 ceph-mon[77147]: pgmap v8272: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:51:50.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:51:49 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:51:50.110 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:51:49 smithi115 conmon[90393]: 2022-05-14T03:51:49.724+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:51:50.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:51:49 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:51:51.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:51:50 smithi115 conmon[90393]: 2022-05-14T03:51:50.756+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:51:51.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:51:50 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:51:51.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:51:50 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:51:51.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:51:50 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:51:51.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:51:50 smithi115 ceph-mon[78593]: pgmap v8273: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:51:51.297 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:51:50 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:51:51.298 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:51:50 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:51:51.298 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:51:50 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:51:51.298 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:51:50 smithi047 ceph-mon[77147]: pgmap v8273: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:51:52.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:51:51 smithi115 conmon[90393]: 2022-05-14T03:51:51.728+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:51:52.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:51:51 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:51:52.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:51:51 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 15845 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:51:52.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:51:51 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:51:52.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:51:51 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 15845 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:51:52.971 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:51:52 smithi115 conmon[90393]: 2022-05-14T03:51:52.718+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:51:53.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:51:52 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:51:53.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:51:52 smithi047 ceph-mon[77147]: pgmap v8274: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:51:53.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:51:52 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:51:53.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:51:52 smithi115 ceph-mon[78593]: pgmap v8274: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:51:54.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:51:53 smithi115 conmon[90393]: 2022-05-14T03:51:53.763+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:51:54.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:51:53 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:51:54.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:51:53 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:51:55.110 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:51:54 smithi115 conmon[90393]: 2022-05-14T03:51:54.754+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:51:55.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:51:54 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:51:55.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:51:54 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:51:55.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:51:54 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:51:55.112 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:51:54 smithi115 ceph-mon[78593]: pgmap v8275: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2022-05-14T03:51:55.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:51:54 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:51:55.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:51:54 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:51:55.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:51:54 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:51:55.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:51:54 smithi047 ceph-mon[77147]: pgmap v8275: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2022-05-14T03:51:56.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:51:55 smithi115 conmon[90393]: 2022-05-14T03:51:55.755+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:51:56.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:51:55 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:51:56.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:51:55 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:51:56.986 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:51:56 smithi115 conmon[90393]: 2022-05-14T03:51:56.723+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:51:57.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:51:56 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:51:57.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:51:56 smithi047 ceph-mon[77147]: pgmap v8276: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2022-05-14T03:51:57.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:51:56 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 15850 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:51:57.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:51:56 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:51:57.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:51:56 smithi115 ceph-mon[78593]: pgmap v8276: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2022-05-14T03:51:57.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:51:56 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 15850 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:51:58.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:51:57 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:51:58.110 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:51:57 smithi115 conmon[90393]: 2022-05-14T03:51:57.771+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:51:58.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:51:57 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:51:59.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:51:58 smithi115 conmon[90393]: 2022-05-14T03:51:58.816+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:51:59.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:51:58 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:51:59.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:51:58 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:51:59.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:51:58 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:51:59.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:51:58 smithi115 ceph-mon[78593]: pgmap v8277: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:51:59.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:51:58 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:51:59.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:51:58 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:51:59.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:51:58 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:51:59.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:51:58 smithi047 ceph-mon[77147]: pgmap v8277: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:52:00.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:51:59 smithi115 conmon[90393]: 2022-05-14T03:51:59.813+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:52:00.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:51:59 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:52:00.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:51:59 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:52:01.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:52:00 smithi115 conmon[90393]: 2022-05-14T03:52:00.794+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:52:01.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:52:01 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:52:01.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:52:01 smithi115 ceph-mon[78593]: pgmap v8278: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:52:01.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:52:01 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:52:01.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:52:01 smithi047 ceph-mon[77147]: pgmap v8278: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:52:02.110 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:52:01 smithi115 conmon[90393]: 2022-05-14T03:52:01.827+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:52:02.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:52:02 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:52:02.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:52:02 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 15855 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:52:02.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:52:02 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:52:02.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:52:02 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 15855 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:52:03.110 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:52:02 smithi115 conmon[90393]: 2022-05-14T03:52:02.779+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:52:03.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:52:03 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:52:03.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:52:03 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:52:03.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:52:03 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:52:03.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:52:03 smithi115 ceph-mon[78593]: pgmap v8279: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:52:03.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:52:03 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:52:03.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:52:03 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:52:03.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:52:03 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:52:03.295 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:52:03 smithi047 ceph-mon[77147]: pgmap v8279: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:52:04.015 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:52:03 smithi115 conmon[90393]: 2022-05-14T03:52:03.747+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:52:04.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:52:04 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:52:04.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:52:04 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:52:05.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:52:04 smithi115 conmon[90393]: 2022-05-14T03:52:04.797+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:52:05.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:52:05 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:52:05.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:52:05 smithi115 ceph-mon[78593]: pgmap v8280: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2022-05-14T03:52:05.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:52:05 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:52:05.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:52:05 smithi047 ceph-mon[77147]: pgmap v8280: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2022-05-14T03:52:06.022 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:52:05 smithi115 conmon[90393]: 2022-05-14T03:52:05.754+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:52:06.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:52:06 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:52:06.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:52:06 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:52:07.026 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:52:06 smithi115 conmon[90393]: 2022-05-14T03:52:06.737+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:52:07.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:52:07 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:52:07.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:52:07 smithi047 ceph-mon[77147]: pgmap v8281: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2022-05-14T03:52:07.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:52:07 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 15860 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:52:07.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:52:07 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:52:07.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:52:07 smithi115 ceph-mon[78593]: pgmap v8281: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2022-05-14T03:52:07.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:52:07 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 15860 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:52:08.028 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:52:07 smithi115 conmon[90393]: 2022-05-14T03:52:07.770+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:52:08.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:52:08 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:52:08.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:52:08 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:52:09.032 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:52:08 smithi115 conmon[90393]: 2022-05-14T03:52:08.741+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:52:09.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:52:09 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:52:09.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:52:09 smithi047 ceph-mon[77147]: pgmap v8282: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:52:09.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:52:09 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:52:09.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:52:09 smithi115 ceph-mon[78593]: pgmap v8282: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:52:10.034 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:52:09 smithi115 conmon[90393]: 2022-05-14T03:52:09.743+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:52:10.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:52:10 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:52:10.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:52:10 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:52:11.043 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:52:10 smithi115 conmon[90393]: 2022-05-14T03:52:10.710+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:52:11.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:52:11 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:52:11.300 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:52:11 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:52:11.301 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:52:11 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:52:11.301 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:52:11 smithi047 ceph-mon[77147]: pgmap v8283: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:52:11.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:52:11 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:52:11.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:52:11 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:52:11.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:52:11 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:52:11.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:52:11 smithi115 ceph-mon[78593]: pgmap v8283: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:52:12.044 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:52:11 smithi115 conmon[90393]: 2022-05-14T03:52:11.696+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:52:12.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:52:12 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:52:12.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:52:12 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 15865 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:52:12.294 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:52:12 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:52:12.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:52:12 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 15865 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:52:13.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:52:12 smithi047 conmon[78521]: 2022-05-14T03:52:12.655+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:52:13.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:52:12 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T03:52:13.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:52:12 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:52:13.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:52:12 smithi047 conmon[78521]: yield 2022-05-14T03:52:13.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:52:12 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:52:13.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:52:12 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:52:13.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:52:12 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:52:13.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:52:12 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T03:52:13.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:52:12 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:52:13.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:52:12 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T03:52:13.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:52:12 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:52:13.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:52:12 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T03:52:13.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:52:12 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:52:13.048 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:52:12 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T03:52:13.049 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:52:12 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:52:13.049 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:52:12 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T03:52:13.049 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:52:12 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:52:13.050 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:52:12 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:52:13.050 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:52:12 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:52:13.051 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:52:12 smithi115 conmon[90393]: 2022-05-14T03:52:12.730+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:52:13.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:52:13 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:52:13.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:52:13 smithi115 ceph-mon[78593]: pgmap v8284: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:52:13.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:52:13 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T03:52:13.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:52:13 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T03:52:13.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:52:13 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T03:52:13.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:52:13 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:52:13.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:52:13 smithi047 ceph-mon[77147]: pgmap v8284: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:52:13.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:52:13 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T03:52:13.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:52:13 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T03:52:13.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:52:13 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T03:52:14.053 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:52:13 smithi115 conmon[90393]: 2022-05-14T03:52:13.724+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:52:14.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:52:14 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:52:14.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T03:52:14.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:52:14.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T03:52:14.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:52:14.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:52:14.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:52:14.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T03:52:14.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:52:14.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T03:52:14.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:52:14.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T03:52:14.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:52:14.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T03:52:14.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:52:14.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T03:52:14.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:52:14.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:52:14.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:52:14.364 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:52:14 smithi115 ceph-mon[78593]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T03:52:14.364 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:52:14 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:52:14.364 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:52:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' 2022-05-14T03:52:14.365 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:52:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' 2022-05-14T03:52:14.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:52:14 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:52:14.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T03:52:14.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:52:14.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T03:52:14.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:52:14.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:52:14.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:52:14.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T03:52:14.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:52:14.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T03:52:14.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:52:14.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T03:52:14.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:52:14.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T03:52:14.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:52:14.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T03:52:14.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:52:14.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:52:14.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:52:14.547 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:52:14 smithi047 ceph-mon[77147]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T03:52:14.547 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:52:14 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:52:14.548 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:52:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' 2022-05-14T03:52:14.548 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:52:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' 2022-05-14T03:52:15.059 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:52:14 smithi115 conmon[90393]: 2022-05-14T03:52:14.774+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:52:15.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:52:15 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:52:15.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:52:15 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:52:15.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:52:15 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:52:15.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:52:15 smithi115 ceph-mon[78593]: pgmap v8285: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2022-05-14T03:52:15.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:52:15 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T03:52:15.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:52:15 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T03:52:15.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:52:15 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:52:15.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:52:15 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:52:15.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:52:15 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:52:15.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:52:15 smithi047 ceph-mon[77147]: pgmap v8285: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2022-05-14T03:52:15.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:52:15 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T03:52:15.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:52:15 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T03:52:16.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:52:15 smithi047 conmon[78521]: 2022-05-14T03:52:15.727+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:52:16.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:52:15 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T03:52:16.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:52:15 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:52:16.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:52:15 smithi047 conmon[78521]: yield 2022-05-14T03:52:16.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:52:15 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:52:16.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:52:15 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:52:16.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:52:15 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:52:16.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:52:15 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T03:52:16.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:52:15 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:52:16.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:52:15 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T03:52:16.048 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:52:15 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:52:16.048 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:52:15 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T03:52:16.048 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:52:15 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:52:16.048 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:52:15 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T03:52:16.048 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:52:15 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:52:16.049 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:52:15 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T03:52:16.049 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:52:15 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:52:16.049 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:52:15 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:52:16.049 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:52:15 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:52:16.049 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:52:15 smithi047 conmon[78521]: 2022-05-14T03:52:15.728+0000 7f533ff78700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T03:52:16.050 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:52:15 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T03:52:16.050 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:52:15 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:52:16.050 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:52:15 smithi047 conmon[78521]: yield 2022-05-14T03:52:16.050 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:52:15 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:52:16.050 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:52:15 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:52:16.051 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:52:15 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:52:16.051 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:52:15 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T03:52:16.051 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:52:15 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:52:16.051 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:52:15 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T03:52:16.052 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:52:15 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:52:16.052 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:52:15 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T03:52:16.052 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:52:15 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:52:16.052 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:52:15 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T03:52:16.052 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:52:15 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:52:16.053 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:52:15 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T03:52:16.053 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:52:15 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:52:16.053 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:52:15 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:52:16.053 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:52:15 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:52:16.053 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:52:15 smithi047 conmon[78521]: 2022-05-14T03:52:16.054 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:52:15 smithi047 conmon[78521]: During handling of the above exception, another exception occurred: 2022-05-14T03:52:16.054 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:52:15 smithi047 conmon[78521]: 2022-05-14T03:52:16.054 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:52:15 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T03:52:16.058 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:52:15 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T03:52:16.059 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:52:15 smithi047 conmon[78521]: return f(*arg) 2022-05-14T03:52:16.059 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:52:15 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T03:52:16.059 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:52:15 smithi047 conmon[78521]: self._write_client_files(client_files, host) 2022-05-14T03:52:16.060 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:52:15 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T03:52:16.060 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:52:15 smithi047 conmon[78521]: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T03:52:16.060 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:52:15 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T03:52:16.061 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:52:15 smithi047 conmon[78521]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T03:52:16.061 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:52:15 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T03:52:16.061 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:52:15 smithi047 conmon[78521]: return self.event_loop.get_result(coro) 2022-05-14T03:52:16.061 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:52:15 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T03:52:16.062 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:52:15 smithi047 conmon[78521]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T03:52:16.062 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:52:15 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T03:52:16.062 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:52:15 smithi047 conmon[78521]: return self.__get_result() 2022-05-14T03:52:16.062 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:52:15 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T03:52:16.062 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:52:15 smithi047 conmon[78521]: raise self._exception 2022-05-14T03:52:16.063 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:52:15 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T03:52:16.063 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:52:15 smithi047 conmon[78521]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T03:52:16.063 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:52:15 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T03:52:16.063 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:52:15 smithi047 conmon[78521]: conn = await self._remote_connection(host, addr) 2022-05-14T03:52:16.064 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:52:15 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T03:52:16.064 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:52:15 smithi047 conmon[78521]: raise 2022-05-14T03:52:16.064 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:52:15 smithi047 conmon[78521]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T03:52:16.064 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:52:15 smithi047 conmon[78521]: self.gen.throw(type, value, traceback) 2022-05-14T03:52:16.064 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:52:15 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T03:52:16.065 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:52:15 smithi047 conmon[78521]: raise OrchestratorError(msg) 2022-05-14T03:52:16.065 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:52:15 smithi047 conmon[78521]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:52:16.066 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:52:15 smithi115 conmon[90393]: 2022-05-14T03:52:15.765+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:52:16.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:52:16 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:52:16.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:52:16 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:52:17.064 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:52:16 smithi115 conmon[90393]: 2022-05-14T03:52:16.768+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:52:17.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:52:17 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:52:17.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T03:52:17.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:52:17.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T03:52:17.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:52:17.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:52:17.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:52:17.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T03:52:17.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:52:17.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T03:52:17.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:52:17.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T03:52:17.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:52:17.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T03:52:17.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:52:17.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T03:52:17.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:52:17.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:52:17.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:52:17.364 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:52:17 smithi115 ceph-mon[78593]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T03:52:17.365 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T03:52:17.365 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:52:17.365 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T03:52:17.365 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:52:17.366 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:52:17.366 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:52:17.366 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T03:52:17.366 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:52:17.366 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T03:52:17.367 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:52:17.367 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T03:52:17.367 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:52:17.367 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T03:52:17.368 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:52:17.368 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T03:52:17.368 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:52:17.368 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:52:17.368 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:52:17.369 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T03:52:17.369 INFO:journalctl@ceph.mon.c.smithi115.stdout: During handling of the above exception, another exception occurred: 2022-05-14T03:52:17.369 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T03:52:17.369 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T03:52:17.370 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T03:52:17.370 INFO:journalctl@ceph.mon.c.smithi115.stdout: return f(*arg) 2022-05-14T03:52:17.370 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T03:52:17.370 INFO:journalctl@ceph.mon.c.smithi115.stdout: self._write_client_files(client_files, host) 2022-05-14T03:52:17.370 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T03:52:17.371 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T03:52:17.371 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T03:52:17.371 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T03:52:17.371 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T03:52:17.371 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.event_loop.get_result(coro) 2022-05-14T03:52:17.372 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T03:52:17.372 INFO:journalctl@ceph.mon.c.smithi115.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T03:52:17.372 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T03:52:17.372 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.__get_result() 2022-05-14T03:52:17.372 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T03:52:17.373 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise self._exception 2022-05-14T03:52:17.373 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T03:52:17.373 INFO:journalctl@ceph.mon.c.smithi115.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T03:52:17.373 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T03:52:17.373 INFO:journalctl@ceph.mon.c.smithi115.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T03:52:17.374 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T03:52:17.374 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise 2022-05-14T03:52:17.374 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T03:52:17.374 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.gen.throw(type, value, traceback) 2022-05-14T03:52:17.375 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T03:52:17.375 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OrchestratorError(msg) 2022-05-14T03:52:17.375 INFO:journalctl@ceph.mon.c.smithi115.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:52:17.375 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:52:17 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:52:17.375 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:52:17 smithi115 ceph-mon[78593]: pgmap v8286: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2022-05-14T03:52:17.376 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:52:17 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 15870 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:52:17.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:52:17 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:52:17.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T03:52:17.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:52:17.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T03:52:17.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:52:17.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:52:17.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:52:17.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T03:52:17.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:52:17.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T03:52:17.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:52:17.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T03:52:17.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:52:17.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T03:52:17.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:52:17.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T03:52:17.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:52:17.548 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:52:17.548 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:52:17.548 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:52:17 smithi047 ceph-mon[77147]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T03:52:17.548 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T03:52:17.548 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:52:17.549 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T03:52:17.549 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:52:17.549 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:52:17.549 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:52:17.550 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T03:52:17.550 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:52:17.550 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T03:52:17.550 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:52:17.550 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T03:52:17.551 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:52:17.551 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T03:52:17.551 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:52:17.551 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T03:52:17.551 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:52:17.552 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:52:17.552 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:52:17.552 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T03:52:17.552 INFO:journalctl@ceph.mon.b.smithi047.stdout: During handling of the above exception, another exception occurred: 2022-05-14T03:52:17.552 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T03:52:17.553 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T03:52:17.553 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T03:52:17.553 INFO:journalctl@ceph.mon.b.smithi047.stdout: return f(*arg) 2022-05-14T03:52:17.553 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T03:52:17.553 INFO:journalctl@ceph.mon.b.smithi047.stdout: self._write_client_files(client_files, host) 2022-05-14T03:52:17.554 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T03:52:17.554 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T03:52:17.554 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T03:52:17.556 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T03:52:17.556 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T03:52:17.556 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.event_loop.get_result(coro) 2022-05-14T03:52:17.556 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T03:52:17.557 INFO:journalctl@ceph.mon.b.smithi047.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T03:52:17.557 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T03:52:17.557 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.__get_result() 2022-05-14T03:52:17.557 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T03:52:17.558 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise self._exception 2022-05-14T03:52:17.558 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T03:52:17.558 INFO:journalctl@ceph.mon.b.smithi047.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T03:52:17.558 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T03:52:17.558 INFO:journalctl@ceph.mon.b.smithi047.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T03:52:17.559 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T03:52:17.559 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise 2022-05-14T03:52:17.559 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T03:52:17.559 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.gen.throw(type, value, traceback) 2022-05-14T03:52:17.560 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T03:52:17.560 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OrchestratorError(msg) 2022-05-14T03:52:17.560 INFO:journalctl@ceph.mon.b.smithi047.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:52:17.560 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:52:17 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:52:17.561 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:52:17 smithi047 ceph-mon[77147]: pgmap v8286: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2022-05-14T03:52:17.561 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:52:17 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 15870 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:52:18.065 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:52:17 smithi115 conmon[90393]: 2022-05-14T03:52:17.729+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:52:18.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:52:18 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:52:18.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:52:18 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:52:18.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:52:18 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:52:18.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:52:18 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:52:19.074 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:52:18 smithi115 conmon[90393]: 2022-05-14T03:52:18.760+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:52:19.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:52:19 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:52:19.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:52:19 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:52:19.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:52:19 smithi115 ceph-mon[78593]: pgmap v8287: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:52:19.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:52:19 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:52:19.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:52:19 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:52:19.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:52:19 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:52:19.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:52:19 smithi047 ceph-mon[77147]: pgmap v8287: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:52:19.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:52:19 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:52:20.075 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:52:19 smithi115 conmon[90393]: 2022-05-14T03:52:19.716+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:52:20.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:52:20 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:52:20.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:52:20 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:52:21.080 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:52:20 smithi115 conmon[90393]: 2022-05-14T03:52:20.723+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:52:21.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:52:21 smithi115 ceph-mon[78593]: pgmap v8288: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:52:21.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:52:21 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:52:21.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:52:21 smithi047 ceph-mon[77147]: pgmap v8288: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:52:21.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:52:21 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:52:22.086 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:52:21 smithi115 conmon[90393]: 2022-05-14T03:52:21.771+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:52:22.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:52:22 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:52:22.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:52:22 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 15875 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:52:22.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:52:22 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:52:22.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:52:22 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 15875 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:52:23.091 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:52:22 smithi115 conmon[90393]: 2022-05-14T03:52:22.778+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:52:23.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:52:23 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:52:23.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:52:23 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:52:23.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:52:23 smithi115 ceph-mon[78593]: pgmap v8289: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:52:23.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:52:23 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:52:23.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:52:23 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:52:23.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:52:23 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:52:23.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:52:23 smithi047 ceph-mon[77147]: pgmap v8289: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:52:23.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:52:23 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:52:24.096 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:52:23 smithi115 conmon[90393]: 2022-05-14T03:52:23.779+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:52:24.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:52:24 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:52:24.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:52:24 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:52:25.098 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:52:24 smithi115 conmon[90393]: 2022-05-14T03:52:24.772+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:52:25.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:52:25 smithi115 ceph-mon[78593]: pgmap v8290: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2022-05-14T03:52:25.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:52:25 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:52:25.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:52:25 smithi047 ceph-mon[77147]: pgmap v8290: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2022-05-14T03:52:25.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:52:25 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:52:26.104 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:52:25 smithi115 conmon[90393]: 2022-05-14T03:52:25.727+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:52:26.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:52:26 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:52:26.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:52:26 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:52:27.104 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:52:26 smithi115 conmon[90393]: 2022-05-14T03:52:26.700+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:52:27.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:52:27 smithi115 ceph-mon[78593]: pgmap v8291: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2022-05-14T03:52:27.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:52:27 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:52:27.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:52:27 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 15880 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:52:27.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:52:27 smithi047 ceph-mon[77147]: pgmap v8291: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2022-05-14T03:52:27.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:52:27 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:52:27.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:52:27 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 15880 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:52:28.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:52:27 smithi115 conmon[90393]: 2022-05-14T03:52:27.724+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:52:28.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:52:28 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:52:28.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:52:28 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:52:29.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:52:28 smithi115 conmon[90393]: 2022-05-14T03:52:28.744+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:52:29.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:52:29 smithi047 ceph-mon[77147]: pgmap v8292: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:52:29.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:52:29 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:52:29.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:52:29 smithi115 ceph-mon[78593]: pgmap v8292: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:52:29.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:52:29 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:52:30.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:52:29 smithi115 conmon[90393]: 2022-05-14T03:52:29.727+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:52:30.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:52:30 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:52:30.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:52:30 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:52:30.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:52:30 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:52:30.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:52:30 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:52:30.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:52:30 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:52:30.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:52:30 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:52:31.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:52:30 smithi115 conmon[90393]: 2022-05-14T03:52:30.709+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:52:31.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:52:31 smithi047 ceph-mon[77147]: pgmap v8293: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:52:31.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:52:31 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:52:31.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:52:31 smithi115 ceph-mon[78593]: pgmap v8293: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:52:31.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:52:31 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:52:32.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:52:31 smithi115 conmon[90393]: 2022-05-14T03:52:31.668+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:52:32.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:52:32 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:52:32.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:52:32 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 15885 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:52:32.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:52:32 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:52:32.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:52:32 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 15885 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:52:33.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:52:32 smithi115 conmon[90393]: 2022-05-14T03:52:32.629+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:52:33.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:52:33 smithi047 ceph-mon[77147]: pgmap v8294: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:52:33.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:52:33 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:52:33.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:52:33 smithi115 ceph-mon[78593]: pgmap v8294: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:52:33.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:52:33 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:52:34.110 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:52:33 smithi115 conmon[90393]: 2022-05-14T03:52:33.638+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:52:34.917 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:52:34 smithi115 conmon[90393]: 2022-05-14T03:52:34.648+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:52:35.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:52:34 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:52:35.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:52:34 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:52:35.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:52:34 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:52:35.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:52:34 smithi047 ceph-mon[77147]: pgmap v8295: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2022-05-14T03:52:35.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:52:34 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:52:35.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:52:34 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:52:35.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:52:34 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:52:35.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:52:34 smithi115 ceph-mon[78593]: pgmap v8295: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2022-05-14T03:52:35.918 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:52:35 smithi115 conmon[90393]: 2022-05-14T03:52:35.618+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:52:36.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:52:35 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:52:36.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:52:35 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:52:36.860 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:52:36 smithi115 conmon[90393]: 2022-05-14T03:52:36.602+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:52:37.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:52:36 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:52:37.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:52:36 smithi047 ceph-mon[77147]: pgmap v8296: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2022-05-14T03:52:37.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:52:36 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 15890 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:52:37.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:52:36 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:52:37.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:52:36 smithi115 ceph-mon[78593]: pgmap v8296: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2022-05-14T03:52:37.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:52:36 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 15890 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:52:37.860 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:52:37 smithi115 conmon[90393]: 2022-05-14T03:52:37.558+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:52:38.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:52:37 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:52:38.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:52:37 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:52:38.861 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:52:38 smithi115 conmon[90393]: 2022-05-14T03:52:38.528+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:52:39.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:52:38 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:52:39.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:52:38 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:52:39.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:52:38 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:52:39.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:52:38 smithi047 ceph-mon[77147]: pgmap v8297: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:52:39.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:52:38 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:52:39.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:52:38 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:52:39.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:52:38 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:52:39.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:52:38 smithi115 ceph-mon[78593]: pgmap v8297: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:52:39.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:52:39 smithi115 conmon[90393]: 2022-05-14T03:52:39.532+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:52:40.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:52:39 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:52:40.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:52:39 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:52:40.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:52:40 smithi115 conmon[90393]: 2022-05-14T03:52:40.549+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:52:41.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:52:40 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:52:41.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:52:40 smithi047 ceph-mon[77147]: pgmap v8298: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:52:41.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:52:40 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:52:41.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:52:40 smithi115 ceph-mon[78593]: pgmap v8298: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:52:41.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:52:41 smithi115 conmon[90393]: 2022-05-14T03:52:41.568+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:52:42.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:52:41 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:52:42.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:52:41 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 15895 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:52:42.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:52:41 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:52:42.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:52:41 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 15895 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:52:42.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:52:42 smithi115 conmon[90393]: 2022-05-14T03:52:42.572+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:52:43.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:52:42 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:52:43.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:52:42 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:52:43.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:52:42 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:52:43.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:52:42 smithi047 ceph-mon[77147]: pgmap v8299: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:52:43.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:52:42 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:52:43.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:52:42 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:52:43.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:52:42 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:52:43.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:52:42 smithi115 ceph-mon[78593]: pgmap v8299: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:52:43.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:52:43 smithi115 conmon[90393]: 2022-05-14T03:52:43.570+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:52:44.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:52:43 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:52:44.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:52:43 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:52:44.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:52:44 smithi115 conmon[90393]: 2022-05-14T03:52:44.577+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:52:45.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:52:44 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:52:45.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:52:44 smithi047 ceph-mon[77147]: pgmap v8300: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2022-05-14T03:52:45.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:52:44 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:52:45.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:52:44 smithi115 ceph-mon[78593]: pgmap v8300: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2022-05-14T03:52:45.960 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:52:45 smithi115 conmon[90393]: 2022-05-14T03:52:45.607+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:52:46.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:52:45 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:52:46.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:52:45 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:52:46.964 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:52:46 smithi115 conmon[90393]: 2022-05-14T03:52:46.622+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:52:47.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:52:46 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:52:47.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:52:46 smithi047 ceph-mon[77147]: pgmap v8301: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2022-05-14T03:52:47.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:52:46 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 15900 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:52:47.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:52:46 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:52:47.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:52:46 smithi115 ceph-mon[78593]: pgmap v8301: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2022-05-14T03:52:47.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:52:46 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 15900 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:52:47.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:52:47 smithi115 conmon[90393]: 2022-05-14T03:52:47.601+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:52:48.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:52:47 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:52:48.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:52:47 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:52:48.973 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:52:48 smithi115 conmon[90393]: 2022-05-14T03:52:48.612+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:52:49.295 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:52:48 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:52:49.296 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:52:48 smithi047 ceph-mon[77147]: pgmap v8302: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:52:49.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:52:48 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:52:49.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:52:48 smithi115 ceph-mon[78593]: pgmap v8302: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:52:49.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:52:49 smithi115 conmon[90393]: 2022-05-14T03:52:49.564+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:52:50.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:52:49 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:52:50.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:52:49 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:52:50.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:52:50 smithi115 conmon[90393]: 2022-05-14T03:52:50.583+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:52:51.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:52:50 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:52:51.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:52:50 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:52:51.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:52:50 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:52:51.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:52:50 smithi047 ceph-mon[77147]: pgmap v8303: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:52:51.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:52:50 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:52:51.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:52:50 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:52:51.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:52:50 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:52:51.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:52:50 smithi115 ceph-mon[78593]: pgmap v8303: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:52:51.987 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:52:51 smithi115 conmon[90393]: 2022-05-14T03:52:51.614+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:52:52.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:52:51 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:52:52.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:52:51 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 15905 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:52:52.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:52:51 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:52:52.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:52:51 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 15905 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:52:52.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:52:52 smithi115 conmon[90393]: 2022-05-14T03:52:52.589+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:52:53.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:52:52 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:52:53.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:52:52 smithi047 ceph-mon[77147]: pgmap v8304: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:52:53.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:52:52 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:52:53.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:52:52 smithi115 ceph-mon[78593]: pgmap v8304: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:52:53.860 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:52:53 smithi115 conmon[90393]: 2022-05-14T03:52:53.552+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:52:54.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:52:53 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:52:54.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:52:53 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:52:54.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:52:54 smithi115 conmon[90393]: 2022-05-14T03:52:54.551+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:52:55.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:52:55 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:52:55.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:52:55 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:52:55.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:52:55 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:52:55.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:52:55 smithi047 ceph-mon[77147]: pgmap v8305: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2022-05-14T03:52:55.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:52:55 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:52:55.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:52:55 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:52:55.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:52:55 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:52:55.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:52:55 smithi115 ceph-mon[78593]: pgmap v8305: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2022-05-14T03:52:55.861 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:52:55 smithi115 conmon[90393]: 2022-05-14T03:52:55.577+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:52:56.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:52:56 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:52:56.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:52:56 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:52:56.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:52:56 smithi115 conmon[90393]: 2022-05-14T03:52:56.555+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:52:57.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:52:57 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:52:57.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:52:57 smithi047 ceph-mon[77147]: pgmap v8306: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2022-05-14T03:52:57.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:52:57 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 15910 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:52:57.362 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:52:57 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:52:57.363 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:52:57 smithi115 ceph-mon[78593]: pgmap v8306: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2022-05-14T03:52:57.363 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:52:57 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 15910 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:52:57.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:52:57 smithi115 conmon[90393]: 2022-05-14T03:52:57.539+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:52:58.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:52:58 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:52:58.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:52:58 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:52:58.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:52:58 smithi115 conmon[90393]: 2022-05-14T03:52:58.523+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:52:59.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:52:59 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:52:59.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:52:59 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:52:59.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:52:59 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:52:59.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:52:59 smithi047 ceph-mon[77147]: pgmap v8307: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:52:59.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:52:59 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:52:59.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:52:59 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:52:59.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:52:59 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:52:59.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:52:59 smithi115 ceph-mon[78593]: pgmap v8307: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:52:59.860 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:52:59 smithi115 conmon[90393]: 2022-05-14T03:52:59.498+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:53:00.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:53:00 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:53:00.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:53:00 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:53:00.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:53:00 smithi115 conmon[90393]: 2022-05-14T03:53:00.532+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:53:01.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:53:01 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:53:01.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:53:01 smithi047 ceph-mon[77147]: pgmap v8308: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:53:01.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:53:01 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:53:01.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:53:01 smithi115 ceph-mon[78593]: pgmap v8308: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:53:01.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:53:01 smithi115 conmon[90393]: 2022-05-14T03:53:01.580+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:53:02.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:53:02 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:53:02.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:53:02 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 15915 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:53:02.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:53:02 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:53:02.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:53:02 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 15915 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:53:02.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:53:02 smithi115 conmon[90393]: 2022-05-14T03:53:02.573+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:53:03.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:53:03 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:53:03.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:53:03 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:53:03.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:53:03 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:53:03.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:53:03 smithi047 ceph-mon[77147]: pgmap v8309: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:53:03.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:53:03 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:53:03.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:53:03 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:53:03.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:53:03 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:53:03.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:53:03 smithi115 ceph-mon[78593]: pgmap v8309: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:53:03.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:53:03 smithi115 conmon[90393]: 2022-05-14T03:53:03.532+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:53:04.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:53:04 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:53:04.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:53:04 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:53:04.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:53:04 smithi115 conmon[90393]: 2022-05-14T03:53:04.558+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:53:05.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:53:05 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:53:05.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:53:05 smithi115 ceph-mon[78593]: pgmap v8310: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2022-05-14T03:53:05.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:53:05 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:53:05.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:53:05 smithi047 ceph-mon[77147]: pgmap v8310: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2022-05-14T03:53:06.048 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:53:05 smithi115 conmon[90393]: 2022-05-14T03:53:05.606+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:53:06.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:53:06 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:53:06.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:53:06 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:53:06.860 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:53:06 smithi115 conmon[90393]: 2022-05-14T03:53:06.589+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:53:07.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:53:07 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:53:07.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:53:07 smithi115 ceph-mon[78593]: pgmap v8311: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2022-05-14T03:53:07.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:53:07 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 15920 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:53:07.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:53:07 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:53:07.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:53:07 smithi047 ceph-mon[77147]: pgmap v8311: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2022-05-14T03:53:07.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:53:07 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 15920 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:53:07.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:53:07 smithi115 conmon[90393]: 2022-05-14T03:53:07.593+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:53:08.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:53:08 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:53:08.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:53:08 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:53:09.059 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:53:08 smithi115 conmon[90393]: 2022-05-14T03:53:08.635+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:53:09.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:53:09 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:53:09.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:53:09 smithi115 ceph-mon[78593]: pgmap v8312: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:53:09.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:53:09 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:53:09.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:53:09 smithi047 ceph-mon[77147]: pgmap v8312: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:53:10.065 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:53:09 smithi115 conmon[90393]: 2022-05-14T03:53:09.648+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:53:10.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:53:10 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:53:10.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:53:10 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:53:11.066 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:53:10 smithi115 conmon[90393]: 2022-05-14T03:53:10.665+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:53:11.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:53:11 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:53:11.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:53:11 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:53:11.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:53:11 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:53:11.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:53:11 smithi115 ceph-mon[78593]: pgmap v8313: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:53:11.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:53:11 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:53:11.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:53:11 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:53:11.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:53:11 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:53:11.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:53:11 smithi047 ceph-mon[77147]: pgmap v8313: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:53:12.072 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:53:11 smithi115 conmon[90393]: 2022-05-14T03:53:11.629+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:53:12.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:53:12 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:53:12.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:53:12 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:53:12.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:53:12 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 15925 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:53:12.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:53:12 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:53:12.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:53:12 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:53:12.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:53:12 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 15925 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:53:13.077 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:53:12 smithi115 conmon[90393]: 2022-05-14T03:53:12.626+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:53:13.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:53:13 smithi115 ceph-mon[78593]: pgmap v8314: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:53:13.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:53:13 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:53:13.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:53:13 smithi047 ceph-mon[77147]: pgmap v8314: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:53:13.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:53:13 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:53:14.082 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:53:13 smithi115 conmon[90393]: 2022-05-14T03:53:13.624+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:53:14.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:53:14 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:53:14.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:53:14 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:53:15.084 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:53:14 smithi115 conmon[90393]: 2022-05-14T03:53:14.659+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:53:15.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:53:15 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:53:15.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:53:15 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:53:15.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:53:15 smithi115 ceph-mon[78593]: pgmap v8315: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2022-05-14T03:53:15.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:53:15 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T03:53:15.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:53:15 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T03:53:15.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:53:15 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:53:15.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:53:15 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:53:15.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:53:15 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:53:15.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:53:15 smithi047 ceph-mon[77147]: pgmap v8315: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2022-05-14T03:53:15.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:53:15 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T03:53:15.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:53:15 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T03:53:15.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:53:15 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:53:16.089 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:53:15 smithi115 conmon[90393]: 2022-05-14T03:53:15.660+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:53:16.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:53:16 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:53:16.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:53:16 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T03:53:16.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:53:16 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T03:53:16.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:53:16 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T03:53:16.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:53:16 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:53:16.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:53:16 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T03:53:16.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:53:16 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T03:53:16.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:53:16 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T03:53:17.094 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:53:16 smithi115 conmon[90393]: 2022-05-14T03:53:16.615+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:53:17.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:53:17 smithi115 ceph-mon[78593]: pgmap v8316: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2022-05-14T03:53:17.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:53:17 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:53:17.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:53:17 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 15930 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:53:17.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:53:17 smithi047 ceph-mon[77147]: pgmap v8316: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2022-05-14T03:53:17.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:53:17 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:53:17.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:53:17 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 15930 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:53:17.865 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:53:17 smithi115 conmon[90393]: 2022-05-14T03:53:17.600+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:53:18.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:53:18 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:53:18.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:53:18 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:53:18.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:53:18 smithi115 conmon[90393]: 2022-05-14T03:53:18.552+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:53:19.102 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:53:18 smithi047 conmon[78521]: 2022-05-14T03:53:18.832+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:53:19.102 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:53:18 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T03:53:19.103 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:53:18 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:53:19.103 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:53:18 smithi047 conmon[78521]: yield 2022-05-14T03:53:19.103 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:53:18 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:53:19.103 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:53:18 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:53:19.103 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:53:18 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:53:19.104 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:53:18 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T03:53:19.104 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:53:18 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:53:19.104 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:53:18 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T03:53:19.104 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:53:18 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:53:19.105 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:53:18 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T03:53:19.105 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:53:18 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:53:19.105 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:53:18 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T03:53:19.105 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:53:18 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:53:19.106 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:53:18 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T03:53:19.106 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:53:18 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:53:19.106 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:53:18 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:53:19.106 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:53:18 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:53:19.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:53:19 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:53:19.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:53:19 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:53:19.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:53:19 smithi115 ceph-mon[78593]: pgmap v8317: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:53:19.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:53:19 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:53:19.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:53:19 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:53:19.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:53:19 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:53:19.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:53:19 smithi047 ceph-mon[77147]: pgmap v8317: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:53:19.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:53:19 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:53:19.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:53:19 smithi115 conmon[90393]: 2022-05-14T03:53:19.599+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:53:20.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:53:20 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:53:20.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T03:53:20.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:53:20.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T03:53:20.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:53:20.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:53:20.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:53:20.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T03:53:20.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:53:20.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T03:53:20.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:53:20.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T03:53:20.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:53:20.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T03:53:20.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:53:20.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T03:53:20.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:53:20.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:53:20.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:53:20.364 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:53:20 smithi115 ceph-mon[78593]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T03:53:20.364 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:53:20 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:53:20.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:53:20 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:53:20.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T03:53:20.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:53:20.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T03:53:20.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:53:20.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:53:20.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:53:20.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T03:53:20.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:53:20.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T03:53:20.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:53:20.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T03:53:20.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:53:20.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T03:53:20.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:53:20.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T03:53:20.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:53:20.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:53:20.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:53:20.547 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:53:20 smithi047 ceph-mon[77147]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T03:53:20.547 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:53:20 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:53:20.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:53:20 smithi115 conmon[90393]: 2022-05-14T03:53:20.567+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:53:21.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:53:21 smithi047 ceph-mon[77147]: pgmap v8318: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:53:21.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:53:21 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:53:21.580 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:53:21 smithi115 ceph-mon[78593]: pgmap v8318: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:53:21.580 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:53:21 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:53:21.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:53:21 smithi115 conmon[90393]: 2022-05-14T03:53:21.577+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:53:22.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:53:22 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:53:22.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:53:22 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:53:22.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:53:22 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:53:22.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:53:22 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 15935 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:53:22.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:53:21 smithi047 conmon[78521]: 2022-05-14T03:53:21.904+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:53:22.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:53:21 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T03:53:22.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:53:21 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:53:22.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:53:21 smithi047 conmon[78521]: yield 2022-05-14T03:53:22.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:53:21 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:53:22.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:53:21 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:53:22.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:53:21 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:53:22.297 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:53:21 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T03:53:22.297 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:53:21 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:53:22.297 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:53:21 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T03:53:22.297 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:53:21 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:53:22.298 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:53:21 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T03:53:22.298 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:53:21 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:53:22.298 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:53:21 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T03:53:22.298 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:53:21 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:53:22.299 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:53:21 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T03:53:22.299 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:53:21 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:53:22.299 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:53:21 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:53:22.299 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:53:21 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:53:22.300 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:53:21 smithi047 conmon[78521]: 2022-05-14T03:53:21.905+0000 7f533f777700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T03:53:22.300 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:53:21 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T03:53:22.300 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:53:21 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:53:22.300 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:53:21 smithi047 conmon[78521]: yield 2022-05-14T03:53:22.300 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:53:21 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:53:22.301 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:53:21 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:53:22.301 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:53:21 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:53:22.301 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:53:21 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T03:53:22.301 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:53:21 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:53:22.302 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:53:21 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T03:53:22.302 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:53:21 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:53:22.302 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:53:21 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T03:53:22.302 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:53:21 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:53:22.303 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:53:21 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T03:53:22.303 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:53:21 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:53:22.303 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:53:21 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T03:53:22.303 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:53:21 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:53:22.303 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:53:21 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:53:22.304 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:53:21 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:53:22.304 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:53:21 smithi047 conmon[78521]: 2022-05-14T03:53:22.304 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:53:21 smithi047 conmon[78521]: During handling of the above exception, another exception occurred: 2022-05-14T03:53:22.304 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:53:21 smithi047 conmon[78521]: 2022-05-14T03:53:22.305 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:53:21 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T03:53:22.305 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:53:21 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T03:53:22.305 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:53:21 smithi047 conmon[78521]: return f(*arg) 2022-05-14T03:53:22.305 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:53:21 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T03:53:22.306 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:53:21 smithi047 conmon[78521]: self._write_client_files(client_files, host) 2022-05-14T03:53:22.306 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:53:21 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T03:53:22.306 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:53:21 smithi047 conmon[78521]: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T03:53:22.306 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:53:21 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T03:53:22.307 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:53:21 smithi047 conmon[78521]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T03:53:22.307 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:53:21 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T03:53:22.307 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:53:21 smithi047 conmon[78521]: return self.event_loop.get_result(coro) 2022-05-14T03:53:22.307 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:53:21 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T03:53:22.307 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:53:21 smithi047 conmon[78521]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T03:53:22.308 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:53:21 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T03:53:22.308 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:53:21 smithi047 conmon[78521]: return self.__get_result() 2022-05-14T03:53:22.308 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:53:21 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T03:53:22.308 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:53:21 smithi047 conmon[78521]: raise self._exception 2022-05-14T03:53:22.308 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:53:21 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T03:53:22.309 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:53:21 smithi047 conmon[78521]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T03:53:22.309 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:53:21 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T03:53:22.309 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:53:21 smithi047 conmon[78521]: conn = await self._remote_connection(host, addr) 2022-05-14T03:53:22.309 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:53:21 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T03:53:22.310 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:53:21 smithi047 conmon[78521]: raise 2022-05-14T03:53:22.310 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:53:21 smithi047 conmon[78521]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T03:53:22.310 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:53:21 smithi047 conmon[78521]: self.gen.throw(type, value, traceback) 2022-05-14T03:53:22.310 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:53:21 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T03:53:22.310 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:53:21 smithi047 conmon[78521]: raise OrchestratorError(msg) 2022-05-14T03:53:22.311 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:53:21 smithi047 conmon[78521]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:53:22.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:53:22 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:53:22.631 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:53:22 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:53:22.631 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:53:22 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:53:22.631 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:53:22 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 15935 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:53:23.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:53:22 smithi115 conmon[90393]: 2022-05-14T03:53:22.608+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:53:23.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:53:23 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:53:23.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T03:53:23.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:53:23.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T03:53:23.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:53:23.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:53:23.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:53:23.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T03:53:23.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:53:23.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T03:53:23.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:53:23.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T03:53:23.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:53:23.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T03:53:23.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:53:23.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T03:53:23.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:53:23.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:53:23.548 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:53:23.548 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:53:23 smithi047 ceph-mon[77147]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T03:53:23.548 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T03:53:23.548 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:53:23.548 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T03:53:23.549 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:53:23.549 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:53:23.549 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:53:23.549 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T03:53:23.550 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:53:23.550 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T03:53:23.550 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:53:23.550 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T03:53:23.551 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:53:23.551 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T03:53:23.551 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:53:23.551 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T03:53:23.552 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:53:23.552 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:53:23.552 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:53:23.553 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T03:53:23.553 INFO:journalctl@ceph.mon.b.smithi047.stdout: During handling of the above exception, another exception occurred: 2022-05-14T03:53:23.553 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T03:53:23.553 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T03:53:23.554 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T03:53:23.554 INFO:journalctl@ceph.mon.b.smithi047.stdout: return f(*arg) 2022-05-14T03:53:23.554 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T03:53:23.554 INFO:journalctl@ceph.mon.b.smithi047.stdout: self._write_client_files(client_files, host) 2022-05-14T03:53:23.555 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T03:53:23.555 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T03:53:23.555 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T03:53:23.555 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T03:53:23.555 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T03:53:23.556 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.event_loop.get_result(coro) 2022-05-14T03:53:23.556 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T03:53:23.556 INFO:journalctl@ceph.mon.b.smithi047.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T03:53:23.556 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T03:53:23.556 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.__get_result() 2022-05-14T03:53:23.557 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T03:53:23.557 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise self._exception 2022-05-14T03:53:23.557 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T03:53:23.557 INFO:journalctl@ceph.mon.b.smithi047.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T03:53:23.558 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T03:53:23.558 INFO:journalctl@ceph.mon.b.smithi047.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T03:53:23.558 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T03:53:23.558 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise 2022-05-14T03:53:23.558 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T03:53:23.559 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.gen.throw(type, value, traceback) 2022-05-14T03:53:23.559 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T03:53:23.559 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OrchestratorError(msg) 2022-05-14T03:53:23.560 INFO:journalctl@ceph.mon.b.smithi047.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:53:23.560 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:53:23 smithi047 ceph-mon[77147]: pgmap v8319: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:53:23.560 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:53:23 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:53:23.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:53:23 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:53:23.608 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T03:53:23.608 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:53:23.608 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T03:53:23.608 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:53:23.609 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:53:23.609 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:53:23.609 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T03:53:23.609 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:53:23.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T03:53:23.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:53:23.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T03:53:23.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:53:23.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T03:53:23.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:53:23.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T03:53:23.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:53:23.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:53:23.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:53:23.612 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:53:23 smithi115 ceph-mon[78593]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T03:53:23.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T03:53:23.614 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:53:23.614 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T03:53:23.614 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:53:23.614 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:53:23.615 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:53:23.615 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T03:53:23.615 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:53:23.615 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T03:53:23.616 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:53:23.616 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T03:53:23.616 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:53:23.616 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T03:53:23.616 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:53:23.617 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T03:53:23.617 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:53:23.617 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:53:23.617 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:53:23.618 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T03:53:23.618 INFO:journalctl@ceph.mon.c.smithi115.stdout: During handling of the above exception, another exception occurred: 2022-05-14T03:53:23.618 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T03:53:23.618 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T03:53:23.618 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T03:53:23.619 INFO:journalctl@ceph.mon.c.smithi115.stdout: return f(*arg) 2022-05-14T03:53:23.619 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T03:53:23.619 INFO:journalctl@ceph.mon.c.smithi115.stdout: self._write_client_files(client_files, host) 2022-05-14T03:53:23.620 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T03:53:23.620 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T03:53:23.620 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T03:53:23.621 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T03:53:23.621 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T03:53:23.621 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.event_loop.get_result(coro) 2022-05-14T03:53:23.621 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T03:53:23.622 INFO:journalctl@ceph.mon.c.smithi115.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T03:53:23.622 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T03:53:23.622 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.__get_result() 2022-05-14T03:53:23.622 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T03:53:23.623 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise self._exception 2022-05-14T03:53:23.623 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T03:53:23.623 INFO:journalctl@ceph.mon.c.smithi115.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T03:53:23.624 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T03:53:23.624 INFO:journalctl@ceph.mon.c.smithi115.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T03:53:23.625 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T03:53:23.625 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise 2022-05-14T03:53:23.625 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T03:53:23.626 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.gen.throw(type, value, traceback) 2022-05-14T03:53:23.626 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T03:53:23.626 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OrchestratorError(msg) 2022-05-14T03:53:23.626 INFO:journalctl@ceph.mon.c.smithi115.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:53:23.627 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:53:23 smithi115 ceph-mon[78593]: pgmap v8319: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:53:23.627 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:53:23 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:53:23.861 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:53:23 smithi115 conmon[90393]: 2022-05-14T03:53:23.604+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:53:24.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:53:24 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:53:24.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:53:24 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:53:25.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:53:24 smithi115 conmon[90393]: 2022-05-14T03:53:24.634+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:53:25.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:53:25 smithi047 ceph-mon[77147]: pgmap v8320: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-05-14T03:53:25.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:53:25 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:53:25.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:53:25 smithi115 ceph-mon[78593]: pgmap v8320: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-05-14T03:53:25.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:53:25 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:53:26.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:53:25 smithi115 conmon[90393]: 2022-05-14T03:53:25.674+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:53:26.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:53:26 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:53:26.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:53:26 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:53:27.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:53:26 smithi115 conmon[90393]: 2022-05-14T03:53:26.692+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:53:27.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:53:27 smithi047 ceph-mon[77147]: pgmap v8321: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 5 op/s 2022-05-14T03:53:27.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:53:27 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:53:27.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:53:27 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 15940 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:53:27.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:53:27 smithi115 ceph-mon[78593]: pgmap v8321: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 5 op/s 2022-05-14T03:53:27.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:53:27 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:53:27.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:53:27 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 15940 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:53:28.110 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:53:27 smithi115 conmon[90393]: 2022-05-14T03:53:27.680+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:53:28.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:53:28 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:53:28.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:53:28 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:53:29.110 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:53:28 smithi115 conmon[90393]: 2022-05-14T03:53:28.641+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:53:29.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:53:29 smithi047 ceph-mon[77147]: pgmap v8322: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:53:29.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:53:29 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:53:29.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:53:29 smithi115 ceph-mon[78593]: pgmap v8322: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:53:29.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:53:29 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:53:30.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:53:29 smithi115 conmon[90393]: 2022-05-14T03:53:29.620+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:53:30.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:53:30 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:53:30.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:53:30 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:53:30.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:53:30 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:53:30.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:53:30 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:53:30.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:53:30 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:53:30.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:53:30 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:53:31.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:53:30 smithi115 conmon[90393]: 2022-05-14T03:53:30.643+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:53:31.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:53:31 smithi047 ceph-mon[77147]: pgmap v8323: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:53:31.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:53:31 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:53:31.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:53:31 smithi115 ceph-mon[78593]: pgmap v8323: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:53:31.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:53:31 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:53:32.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:53:31 smithi115 conmon[90393]: 2022-05-14T03:53:31.613+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:53:32.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:53:32 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:53:32.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:53:32 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 15945 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:53:32.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:53:32 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:53:32.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:53:32 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 15945 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:53:33.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:53:32 smithi115 conmon[90393]: 2022-05-14T03:53:32.659+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:53:33.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:53:33 smithi047 ceph-mon[77147]: pgmap v8324: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:53:33.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:53:33 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:53:33.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:53:33 smithi115 ceph-mon[78593]: pgmap v8324: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:53:33.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:53:33 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:53:34.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:53:33 smithi115 conmon[90393]: 2022-05-14T03:53:33.639+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:53:34.930 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:53:34 smithi115 conmon[90393]: 2022-05-14T03:53:34.645+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:53:35.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:53:34 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:53:35.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:53:34 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:53:35.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:53:34 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:53:35.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:53:34 smithi047 ceph-mon[77147]: pgmap v8325: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-05-14T03:53:35.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:53:34 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:53:35.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:53:34 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:53:35.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:53:34 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:53:35.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:53:34 smithi115 ceph-mon[78593]: pgmap v8325: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 5 op/s 2022-05-14T03:53:36.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:53:35 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:53:36.110 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:53:35 smithi115 conmon[90393]: 2022-05-14T03:53:35.679+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:53:36.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:53:35 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:53:36.935 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:53:36 smithi115 conmon[90393]: 2022-05-14T03:53:36.631+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:53:37.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:53:36 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:53:37.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:53:36 smithi047 ceph-mon[77147]: pgmap v8326: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2022-05-14T03:53:37.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:53:36 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 15950 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:53:37.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:53:36 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:53:37.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:53:36 smithi115 ceph-mon[78593]: pgmap v8326: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2022-05-14T03:53:37.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:53:36 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 15950 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:53:37.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:53:37 smithi115 conmon[90393]: 2022-05-14T03:53:37.592+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:53:38.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:53:37 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:53:38.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:53:37 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:53:38.945 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:53:38 smithi115 conmon[90393]: 2022-05-14T03:53:38.607+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:53:39.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:53:38 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:53:39.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:53:38 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:53:39.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:53:38 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:53:39.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:53:38 smithi047 ceph-mon[77147]: pgmap v8327: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:53:39.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:53:38 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:53:39.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:53:38 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:53:39.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:53:38 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:53:39.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:53:38 smithi115 ceph-mon[78593]: pgmap v8327: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:53:39.860 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:53:39 smithi115 conmon[90393]: 2022-05-14T03:53:39.604+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:53:40.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:53:39 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:53:40.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:53:39 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:53:40.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:53:40 smithi115 conmon[90393]: 2022-05-14T03:53:40.571+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:53:41.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:53:40 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:53:41.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:53:40 smithi047 ceph-mon[77147]: pgmap v8328: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:53:41.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:53:40 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:53:41.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:53:40 smithi115 ceph-mon[78593]: pgmap v8328: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:53:41.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:53:41 smithi115 conmon[90393]: 2022-05-14T03:53:41.577+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:53:42.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:53:41 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:53:42.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:53:41 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 15955 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:53:42.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:53:41 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:53:42.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:53:41 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 15955 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:53:42.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:53:42 smithi115 conmon[90393]: 2022-05-14T03:53:42.547+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:53:43.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:53:42 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:53:43.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:53:42 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:53:43.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:53:42 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:53:43.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:53:42 smithi047 ceph-mon[77147]: pgmap v8329: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:53:43.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:53:42 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:53:43.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:53:42 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:53:43.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:53:42 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:53:43.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:53:42 smithi115 ceph-mon[78593]: pgmap v8329: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:53:43.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:53:43 smithi115 conmon[90393]: 2022-05-14T03:53:43.536+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:53:44.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:53:43 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:53:44.364 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:53:43 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:53:44.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:53:44 smithi115 conmon[90393]: 2022-05-14T03:53:44.519+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:53:45.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:53:44 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:53:45.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:53:44 smithi047 ceph-mon[77147]: pgmap v8330: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2022-05-14T03:53:45.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:53:44 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:53:45.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:53:44 smithi115 ceph-mon[78593]: pgmap v8330: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2022-05-14T03:53:45.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:53:45 smithi115 conmon[90393]: 2022-05-14T03:53:45.512+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:53:46.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:53:45 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:53:46.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:53:45 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:53:46.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:53:46 smithi115 conmon[90393]: 2022-05-14T03:53:46.540+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:53:47.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:53:46 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:53:47.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:53:46 smithi047 ceph-mon[77147]: pgmap v8331: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2022-05-14T03:53:47.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:53:46 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 15960 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:53:47.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:53:46 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:53:47.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:53:46 smithi115 ceph-mon[78593]: pgmap v8331: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2022-05-14T03:53:47.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:53:46 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 15960 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:53:47.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:53:47 smithi115 conmon[90393]: 2022-05-14T03:53:47.515+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:53:48.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:53:47 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:53:48.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:53:47 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:53:48.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:53:48 smithi115 conmon[90393]: 2022-05-14T03:53:48.523+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:53:49.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:53:48 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:53:49.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:53:48 smithi047 ceph-mon[77147]: pgmap v8332: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:53:49.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:53:48 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:53:49.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:53:48 smithi115 ceph-mon[78593]: pgmap v8332: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:53:49.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:53:49 smithi115 conmon[90393]: 2022-05-14T03:53:49.475+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:53:50.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:53:49 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:53:50.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:53:49 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:53:50.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:53:50 smithi115 conmon[90393]: 2022-05-14T03:53:50.463+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:53:51.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:53:50 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:53:51.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:53:50 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:53:51.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:53:50 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:53:51.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:53:50 smithi047 ceph-mon[77147]: pgmap v8333: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:53:51.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:53:50 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:53:51.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:53:50 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:53:51.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:53:50 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:53:51.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:53:50 smithi115 ceph-mon[78593]: pgmap v8333: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:53:51.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:53:51 smithi115 conmon[90393]: 2022-05-14T03:53:51.481+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:53:52.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:53:51 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:53:52.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:53:51 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 15965 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:53:52.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:53:51 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:53:52.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:53:51 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 15965 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:53:52.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:53:52 smithi115 conmon[90393]: 2022-05-14T03:53:52.506+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:53:53.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:53:52 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:53:53.362 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:53:52 smithi047 ceph-mon[77147]: pgmap v8334: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:53:53.363 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:53:53 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:53:53.363 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:53:53 smithi115 ceph-mon[78593]: pgmap v8334: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:53:53.860 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:53:53 smithi115 conmon[90393]: 2022-05-14T03:53:53.547+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:53:54.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:53:54 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:53:54.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:53:54 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:53:54.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:53:54 smithi115 conmon[90393]: 2022-05-14T03:53:54.564+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:53:55.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:53:55 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:53:55.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:53:55 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:53:55.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:53:55 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:53:55.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:53:55 smithi047 ceph-mon[77147]: pgmap v8335: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2022-05-14T03:53:55.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:53:55 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:53:55.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:53:55 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:53:55.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:53:55 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:53:55.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:53:55 smithi115 ceph-mon[78593]: pgmap v8335: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2022-05-14T03:53:55.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:53:55 smithi115 conmon[90393]: 2022-05-14T03:53:55.561+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:53:56.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:53:56 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:53:56.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:53:56 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:53:56.860 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:53:56 smithi115 conmon[90393]: 2022-05-14T03:53:56.537+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:53:57.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:53:57 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:53:57.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:53:57 smithi047 ceph-mon[77147]: pgmap v8336: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2022-05-14T03:53:57.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:53:57 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 15970 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:53:57.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:53:57 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:53:57.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:53:57 smithi115 ceph-mon[78593]: pgmap v8336: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2022-05-14T03:53:57.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:53:57 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 15970 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:53:57.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:53:57 smithi115 conmon[90393]: 2022-05-14T03:53:57.527+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:53:58.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:53:58 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:53:58.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:53:58 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:53:58.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:53:58 smithi115 conmon[90393]: 2022-05-14T03:53:58.547+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:53:59.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:53:59 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:53:59.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:53:59 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:53:59.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:53:59 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:53:59.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:53:59 smithi047 ceph-mon[77147]: pgmap v8337: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:53:59.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:53:59 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:53:59.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:53:59 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:53:59.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:53:59 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:53:59.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:53:59 smithi115 ceph-mon[78593]: pgmap v8337: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:53:59.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:53:59 smithi115 conmon[90393]: 2022-05-14T03:53:59.515+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:54:00.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:54:00 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:54:00.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:54:00 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:54:00.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:54:00 smithi115 conmon[90393]: 2022-05-14T03:54:00.504+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:54:01.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:54:01 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:54:01.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:54:01 smithi047 ceph-mon[77147]: pgmap v8338: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:54:01.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:54:01 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:54:01.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:54:01 smithi115 ceph-mon[78593]: pgmap v8338: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:54:01.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:54:01 smithi115 conmon[90393]: 2022-05-14T03:54:01.493+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:54:02.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:54:02 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:54:02.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:54:02 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 15975 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:54:02.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:54:02 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:54:02.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:54:02 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 15975 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:54:02.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:54:02 smithi115 conmon[90393]: 2022-05-14T03:54:02.456+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:54:03.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:54:03 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:54:03.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:54:03 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:54:03.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:54:03 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:54:03.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:54:03 smithi115 ceph-mon[78593]: pgmap v8339: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:54:03.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:54:03 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:54:03.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:54:03 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:54:03.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:54:03 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:54:03.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:54:03 smithi047 ceph-mon[77147]: pgmap v8339: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:54:03.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:54:03 smithi115 conmon[90393]: 2022-05-14T03:54:03.418+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:54:04.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:54:04 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:54:04.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:54:04 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:54:04.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:54:04 smithi115 conmon[90393]: 2022-05-14T03:54:04.390+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:54:05.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:54:05 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:54:05.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:54:05 smithi115 ceph-mon[78593]: pgmap v8340: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2022-05-14T03:54:05.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:54:05 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:54:05.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:54:05 smithi047 ceph-mon[77147]: pgmap v8340: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2022-05-14T03:54:05.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:54:05 smithi115 conmon[90393]: 2022-05-14T03:54:05.440+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:54:06.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:54:06 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:54:06.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:54:06 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:54:06.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:54:06 smithi115 conmon[90393]: 2022-05-14T03:54:06.420+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:54:07.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:54:07 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:54:07.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:54:07 smithi115 ceph-mon[78593]: pgmap v8341: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2022-05-14T03:54:07.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:54:07 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 15980 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:54:07.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:54:07 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:54:07.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:54:07 smithi047 ceph-mon[77147]: pgmap v8341: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2022-05-14T03:54:07.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:54:07 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 15980 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:54:07.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:54:07 smithi115 conmon[90393]: 2022-05-14T03:54:07.449+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:54:08.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:54:08 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:54:08.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:54:08 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:54:08.860 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:54:08 smithi115 conmon[90393]: 2022-05-14T03:54:08.442+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:54:09.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:54:09 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:54:09.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:54:09 smithi115 ceph-mon[78593]: pgmap v8342: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:54:09.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:54:09 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:54:09.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:54:09 smithi047 ceph-mon[77147]: pgmap v8342: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:54:09.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:54:09 smithi115 conmon[90393]: 2022-05-14T03:54:09.470+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:54:10.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:54:10 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:54:10.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:54:10 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:54:10.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:54:10 smithi115 conmon[90393]: 2022-05-14T03:54:10.510+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:54:11.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:54:11 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:54:11.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:54:11 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:54:11.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:54:11 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:54:11.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:54:11 smithi115 ceph-mon[78593]: pgmap v8343: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:54:11.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:54:11 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:54:11.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:54:11 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:54:11.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:54:11 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:54:11.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:54:11 smithi047 ceph-mon[77147]: pgmap v8343: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:54:11.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:54:11 smithi115 conmon[90393]: 2022-05-14T03:54:11.488+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:54:12.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:54:12 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:54:12.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:54:12 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 15985 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:54:12.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:54:12 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:54:12.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:54:12 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 15985 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:54:12.860 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:54:12 smithi115 conmon[90393]: 2022-05-14T03:54:12.493+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:54:13.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:54:13 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:54:13.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:54:13 smithi115 ceph-mon[78593]: pgmap v8344: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:54:13.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:54:13 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:54:13.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:54:13 smithi047 ceph-mon[77147]: pgmap v8344: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:54:13.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:54:13 smithi115 conmon[90393]: 2022-05-14T03:54:13.471+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:54:14.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:54:14 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:54:14.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:54:14 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:54:14.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:54:14 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:54:14.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:54:14 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:54:14.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:54:14 smithi115 conmon[90393]: 2022-05-14T03:54:14.505+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:54:15.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:54:15 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:54:15.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:54:15 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:54:15.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:54:15 smithi115 ceph-mon[78593]: pgmap v8345: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2022-05-14T03:54:15.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:54:15 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T03:54:15.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:54:15 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T03:54:15.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:54:15 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:54:15.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:54:15 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:54:15.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:54:15 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:54:15.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:54:15 smithi047 ceph-mon[77147]: pgmap v8345: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2022-05-14T03:54:15.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:54:15 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T03:54:15.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:54:15 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T03:54:15.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:54:15 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:54:15.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:54:15 smithi115 conmon[90393]: 2022-05-14T03:54:15.505+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:54:16.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:54:16 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:54:16.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:54:16 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:54:16.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:54:16 smithi115 conmon[90393]: 2022-05-14T03:54:16.518+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:54:17.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:54:17 smithi115 ceph-mon[78593]: pgmap v8346: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2022-05-14T03:54:17.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:54:17 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:54:17.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:54:17 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 15990 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:54:17.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:54:17 smithi047 ceph-mon[77147]: pgmap v8346: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2022-05-14T03:54:17.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:54:17 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:54:17.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:54:17 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 15990 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:54:17.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:54:17 smithi115 conmon[90393]: 2022-05-14T03:54:17.471+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:54:18.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:54:18 smithi115 conmon[90393]: 2022-05-14T03:54:18.507+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:54:19.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:54:18 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:54:19.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:54:18 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:54:19.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:54:18 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:54:19.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:54:18 smithi047 ceph-mon[77147]: pgmap v8347: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:54:19.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:54:18 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:54:19.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:54:18 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:54:19.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:54:18 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:54:19.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:54:18 smithi115 ceph-mon[78593]: pgmap v8347: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:54:19.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:54:19 smithi115 conmon[90393]: 2022-05-14T03:54:19.464+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:54:20.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:54:19 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:54:20.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:54:19 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:54:20.860 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:54:20 smithi115 conmon[90393]: 2022-05-14T03:54:20.501+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:54:21.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:54:20 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:54:21.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:54:20 smithi047 ceph-mon[77147]: pgmap v8348: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:54:21.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:54:20 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:54:21.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:54:20 smithi115 ceph-mon[78593]: pgmap v8348: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:54:21.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:54:21 smithi115 conmon[90393]: 2022-05-14T03:54:21.495+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:54:22.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:54:21 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:54:22.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:54:21 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T03:54:22.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:54:21 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T03:54:22.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:54:21 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T03:54:22.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:54:21 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 15995 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:54:22.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:54:21 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:54:22.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:54:21 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T03:54:22.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:54:21 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T03:54:22.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:54:21 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T03:54:22.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:54:21 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 15995 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:54:22.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:54:22 smithi115 conmon[90393]: 2022-05-14T03:54:22.473+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:54:23.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:54:22 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:54:23.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:54:22 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:54:23.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:54:22 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:54:23.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:54:22 smithi047 ceph-mon[77147]: pgmap v8349: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:54:23.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:54:22 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:54:23.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:54:22 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:54:23.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:54:22 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:54:23.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:54:22 smithi115 ceph-mon[78593]: pgmap v8349: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:54:23.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:54:23 smithi115 conmon[90393]: 2022-05-14T03:54:23.503+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:54:24.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:54:23 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:54:24.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:54:23 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:54:24.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:54:24 smithi115 conmon[90393]: 2022-05-14T03:54:24.549+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:54:25.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:54:24 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:54:25.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:54:24 smithi047 ceph-mon[77147]: pgmap v8350: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2022-05-14T03:54:25.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:54:25 smithi047 conmon[78521]: 2022-05-14T03:54:25.007+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:54:25.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:54:25 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T03:54:25.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:54:25 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:54:25.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:54:25 smithi047 conmon[78521]: yield 2022-05-14T03:54:25.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:54:25 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:54:25.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:54:25 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:54:25.297 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:54:25 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:54:25.297 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:54:25 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T03:54:25.298 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:54:25 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:54:25.298 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:54:25 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T03:54:25.298 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:54:25 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:54:25.298 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:54:25 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T03:54:25.299 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:54:25 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:54:25.299 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:54:25 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T03:54:25.300 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:54:25 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:54:25.301 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:54:25 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T03:54:25.301 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:54:25 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:54:25.301 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:54:25 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:54:25.302 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:54:25 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:54:25.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:54:24 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:54:25.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:54:24 smithi115 ceph-mon[78593]: pgmap v8350: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2022-05-14T03:54:25.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:54:25 smithi115 conmon[90393]: 2022-05-14T03:54:25.564+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:54:26.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:54:25 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:54:26.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:54:25 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:54:26.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T03:54:26.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:54:26.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T03:54:26.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:54:26.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:54:26.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:54:26.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T03:54:26.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:54:26.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T03:54:26.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:54:26.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T03:54:26.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:54:26.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T03:54:26.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:54:26.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T03:54:26.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:54:26.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:54:26.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:54:26.297 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:54:25 smithi047 ceph-mon[77147]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T03:54:26.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:54:25 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:54:26.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:54:25 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:54:26.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T03:54:26.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:54:26.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T03:54:26.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:54:26.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:54:26.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:54:26.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T03:54:26.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:54:26.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T03:54:26.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:54:26.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T03:54:26.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:54:26.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T03:54:26.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:54:26.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T03:54:26.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:54:26.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:54:26.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:54:26.364 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:54:25 smithi115 ceph-mon[78593]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T03:54:26.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:54:26 smithi115 conmon[90393]: 2022-05-14T03:54:26.523+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:54:27.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:54:26 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:54:27.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:54:26 smithi047 ceph-mon[77147]: pgmap v8351: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 5 op/s 2022-05-14T03:54:27.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:54:26 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 16000 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:54:27.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:54:26 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:54:27.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:54:26 smithi115 ceph-mon[78593]: pgmap v8351: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 5 op/s 2022-05-14T03:54:27.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:54:26 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 16000 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:54:27.860 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:54:27 smithi115 conmon[90393]: 2022-05-14T03:54:27.561+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:54:28.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:54:27 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:54:28.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:54:28 smithi047 conmon[78521]: 2022-05-14T03:54:28.079+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:54:28.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:54:28 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T03:54:28.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:54:28 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:54:28.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:54:28 smithi047 conmon[78521]: yield 2022-05-14T03:54:28.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:54:28 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:54:28.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:54:28 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:54:28.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:54:28 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:54:28.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:54:28 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T03:54:28.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:54:28 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:54:28.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:54:28 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T03:54:28.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:54:28 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:54:28.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:54:28 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T03:54:28.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:54:28 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:54:28.297 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:54:28 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T03:54:28.297 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:54:28 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:54:28.297 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:54:28 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T03:54:28.297 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:54:28 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:54:28.298 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:54:28 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:54:28.298 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:54:28 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:54:28.298 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:54:28 smithi047 conmon[78521]: 2022-05-14T03:54:28.080+0000 7f5341f7c700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T03:54:28.298 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:54:28 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T03:54:28.298 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:54:28 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:54:28.299 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:54:28 smithi047 conmon[78521]: yield 2022-05-14T03:54:28.299 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:54:28 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:54:28.299 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:54:28 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:54:28.299 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:54:28 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:54:28.300 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:54:28 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T03:54:28.300 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:54:28 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:54:28.300 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:54:28 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T03:54:28.300 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:54:28 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:54:28.301 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:54:28 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T03:54:28.301 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:54:28 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:54:28.301 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:54:28 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T03:54:28.301 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:54:28 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:54:28.301 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:54:28 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T03:54:28.302 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:54:28 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:54:28.302 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:54:28 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:54:28.302 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:54:28 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:54:28.302 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:54:28 smithi047 conmon[78521]: 2022-05-14T03:54:28.303 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:54:28 smithi047 conmon[78521]: During handling of the above exception, another exception occurred: 2022-05-14T03:54:28.303 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:54:28 smithi047 conmon[78521]: 2022-05-14T03:54:28.303 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:54:28 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T03:54:28.303 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:54:28 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T03:54:28.303 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:54:28 smithi047 conmon[78521]: return f(*arg) 2022-05-14T03:54:28.304 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:54:28 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T03:54:28.304 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:54:28 smithi047 conmon[78521]: self._write_client_files(client_files, host) 2022-05-14T03:54:28.304 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:54:28 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T03:54:28.304 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:54:28 smithi047 conmon[78521]: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T03:54:28.304 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:54:28 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T03:54:28.305 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:54:28 smithi047 conmon[78521]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T03:54:28.305 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:54:28 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T03:54:28.305 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:54:28 smithi047 conmon[78521]: return self.event_loop.get_result(coro) 2022-05-14T03:54:28.305 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:54:28 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T03:54:28.306 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:54:28 smithi047 conmon[78521]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T03:54:28.306 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:54:28 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T03:54:28.306 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:54:28 smithi047 conmon[78521]: return self.__get_result() 2022-05-14T03:54:28.306 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:54:28 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T03:54:28.306 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:54:28 smithi047 conmon[78521]: raise self._exception 2022-05-14T03:54:28.307 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:54:28 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T03:54:28.307 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:54:28 smithi047 conmon[78521]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T03:54:28.307 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:54:28 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T03:54:28.307 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:54:28 smithi047 conmon[78521]: conn = await self._remote_connection(host, addr) 2022-05-14T03:54:28.308 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:54:28 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T03:54:28.308 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:54:28 smithi047 conmon[78521]: raise 2022-05-14T03:54:28.308 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:54:28 smithi047 conmon[78521]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T03:54:28.309 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:54:28 smithi047 conmon[78521]: self.gen.throw(type, value, traceback) 2022-05-14T03:54:28.309 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:54:28 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T03:54:28.309 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:54:28 smithi047 conmon[78521]: raise OrchestratorError(msg) 2022-05-14T03:54:28.309 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:54:28 smithi047 conmon[78521]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:54:28.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:54:27 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:54:28.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:54:28 smithi115 conmon[90393]: 2022-05-14T03:54:28.515+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:54:29.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:54:28 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:54:29.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:54:28 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:54:29.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T03:54:29.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:54:29.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T03:54:29.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:54:29.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:54:29.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:54:29.298 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T03:54:29.298 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:54:29.299 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T03:54:29.299 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:54:29.300 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T03:54:29.300 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:54:29.300 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T03:54:29.300 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:54:29.301 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T03:54:29.301 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:54:29.302 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:54:29.302 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:54:29.302 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:54:28 smithi047 ceph-mon[77147]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T03:54:29.303 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T03:54:29.303 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:54:29.303 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T03:54:29.304 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:54:29.304 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:54:29.305 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:54:29.305 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T03:54:29.305 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:54:29.306 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T03:54:29.306 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:54:29.306 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T03:54:29.307 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:54:29.307 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T03:54:29.307 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:54:29.308 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T03:54:29.308 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:54:29.308 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:54:29.309 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:54:29.309 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T03:54:29.310 INFO:journalctl@ceph.mon.b.smithi047.stdout: During handling of the above exception, another exception occurred: 2022-05-14T03:54:29.310 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T03:54:29.310 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T03:54:29.312 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T03:54:29.313 INFO:journalctl@ceph.mon.b.smithi047.stdout: return f(*arg) 2022-05-14T03:54:29.313 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T03:54:29.313 INFO:journalctl@ceph.mon.b.smithi047.stdout: self._write_client_files(client_files, host) 2022-05-14T03:54:29.314 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T03:54:29.314 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T03:54:29.315 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T03:54:29.315 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T03:54:29.315 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T03:54:29.315 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.event_loop.get_result(coro) 2022-05-14T03:54:29.316 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T03:54:29.316 INFO:journalctl@ceph.mon.b.smithi047.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T03:54:29.316 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T03:54:29.316 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.__get_result() 2022-05-14T03:54:29.316 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T03:54:29.317 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise self._exception 2022-05-14T03:54:29.317 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T03:54:29.317 INFO:journalctl@ceph.mon.b.smithi047.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T03:54:29.317 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T03:54:29.318 INFO:journalctl@ceph.mon.b.smithi047.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T03:54:29.318 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T03:54:29.318 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise 2022-05-14T03:54:29.318 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T03:54:29.318 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.gen.throw(type, value, traceback) 2022-05-14T03:54:29.319 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T03:54:29.319 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OrchestratorError(msg) 2022-05-14T03:54:29.319 INFO:journalctl@ceph.mon.b.smithi047.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:54:29.319 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:54:28 smithi047 ceph-mon[77147]: pgmap v8352: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 4 op/s 2022-05-14T03:54:29.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:54:28 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:54:29.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:54:28 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:54:29.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T03:54:29.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:54:29.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T03:54:29.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:54:29.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:54:29.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:54:29.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T03:54:29.365 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:54:29.365 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T03:54:29.366 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:54:29.366 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T03:54:29.367 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:54:29.367 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T03:54:29.368 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:54:29.368 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T03:54:29.369 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:54:29.369 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:54:29.369 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:54:29.370 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:54:28 smithi115 ceph-mon[78593]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T03:54:29.370 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T03:54:29.371 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:54:29.371 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T03:54:29.372 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:54:29.372 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:54:29.373 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:54:29.373 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T03:54:29.374 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:54:29.374 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T03:54:29.374 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:54:29.375 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T03:54:29.375 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:54:29.376 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T03:54:29.376 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:54:29.377 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T03:54:29.377 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:54:29.378 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:54:29.378 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:54:29.378 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T03:54:29.379 INFO:journalctl@ceph.mon.c.smithi115.stdout: During handling of the above exception, another exception occurred: 2022-05-14T03:54:29.379 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T03:54:29.380 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T03:54:29.380 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T03:54:29.381 INFO:journalctl@ceph.mon.c.smithi115.stdout: return f(*arg) 2022-05-14T03:54:29.381 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T03:54:29.382 INFO:journalctl@ceph.mon.c.smithi115.stdout: self._write_client_files(client_files, host) 2022-05-14T03:54:29.382 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T03:54:29.383 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T03:54:29.383 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T03:54:29.383 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T03:54:29.384 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T03:54:29.384 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.event_loop.get_result(coro) 2022-05-14T03:54:29.385 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T03:54:29.385 INFO:journalctl@ceph.mon.c.smithi115.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T03:54:29.386 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T03:54:29.386 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.__get_result() 2022-05-14T03:54:29.387 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T03:54:29.387 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise self._exception 2022-05-14T03:54:29.387 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T03:54:29.387 INFO:journalctl@ceph.mon.c.smithi115.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T03:54:29.388 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T03:54:29.388 INFO:journalctl@ceph.mon.c.smithi115.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T03:54:29.388 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T03:54:29.388 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise 2022-05-14T03:54:29.389 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T03:54:29.389 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.gen.throw(type, value, traceback) 2022-05-14T03:54:29.389 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T03:54:29.389 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OrchestratorError(msg) 2022-05-14T03:54:29.390 INFO:journalctl@ceph.mon.c.smithi115.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:54:29.390 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:54:28 smithi115 ceph-mon[78593]: pgmap v8352: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 4 op/s 2022-05-14T03:54:29.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:54:29 smithi115 conmon[90393]: 2022-05-14T03:54:29.500+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:54:30.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:54:29 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:54:30.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:54:29 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:54:30.861 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:54:30 smithi115 conmon[90393]: 2022-05-14T03:54:30.521+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:54:31.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:54:30 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:54:31.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:54:30 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:54:31.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:54:30 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:54:31.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:54:30 smithi047 ceph-mon[77147]: pgmap v8353: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 7 op/s 2022-05-14T03:54:31.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:54:30 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:54:31.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:54:30 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:54:31.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:54:30 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:54:31.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:54:30 smithi115 ceph-mon[78593]: pgmap v8353: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 7 op/s 2022-05-14T03:54:31.860 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:54:31 smithi115 conmon[90393]: 2022-05-14T03:54:31.499+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:54:32.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:54:31 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:54:32.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:54:31 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 16005 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:54:32.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:54:31 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:54:32.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:54:31 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 16005 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:54:32.860 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:54:32 smithi115 conmon[90393]: 2022-05-14T03:54:32.455+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:54:33.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:54:32 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:54:33.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:54:32 smithi047 ceph-mon[77147]: pgmap v8354: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 4 op/s 2022-05-14T03:54:33.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:54:32 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:54:33.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:54:32 smithi115 ceph-mon[78593]: pgmap v8354: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 4 op/s 2022-05-14T03:54:33.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:54:33 smithi115 conmon[90393]: 2022-05-14T03:54:33.414+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:54:34.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:54:33 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:54:34.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:54:33 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:54:34.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:54:34 smithi115 conmon[90393]: 2022-05-14T03:54:34.407+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:54:35.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:54:35 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:54:35.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:54:35 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:54:35.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:54:35 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:54:35.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:54:35 smithi047 ceph-mon[77147]: pgmap v8355: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 5 op/s 2022-05-14T03:54:35.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:54:35 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:54:35.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:54:35 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:54:35.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:54:35 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:54:35.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:54:35 smithi115 ceph-mon[78593]: pgmap v8355: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 5 op/s 2022-05-14T03:54:35.860 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:54:35 smithi115 conmon[90393]: 2022-05-14T03:54:35.362+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:54:36.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:54:36 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:54:36.344 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:54:36 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:54:36.610 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:54:36 smithi115 conmon[90393]: 2022-05-14T03:54:36.341+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:54:37.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:54:37 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:54:37.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:54:37 smithi047 ceph-mon[77147]: pgmap v8356: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 5 op/s 2022-05-14T03:54:37.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:54:37 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 16010 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:54:37.322 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:54:37 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:54:37.322 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:54:37 smithi115 ceph-mon[78593]: pgmap v8356: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 5 op/s 2022-05-14T03:54:37.322 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:54:37 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 16010 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:54:37.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:54:37 smithi115 conmon[90393]: 2022-05-14T03:54:37.319+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:54:38.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:54:38 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:54:38.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:54:38 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:54:38.860 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:54:38 smithi115 conmon[90393]: 2022-05-14T03:54:38.357+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:54:39.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:54:39 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:54:39.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:54:39 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:54:39.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:54:39 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:54:39.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:54:39 smithi047 ceph-mon[77147]: pgmap v8357: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:54:39.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:54:39 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:54:39.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:54:39 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:54:39.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:54:39 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:54:39.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:54:39 smithi115 ceph-mon[78593]: pgmap v8357: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:54:39.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:54:39 smithi115 conmon[90393]: 2022-05-14T03:54:39.371+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:54:40.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:54:40 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:54:40.353 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:54:40 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:54:40.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:54:40 smithi115 conmon[90393]: 2022-05-14T03:54:40.349+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:54:41.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:54:41 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:54:41.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:54:41 smithi047 ceph-mon[77147]: pgmap v8358: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:54:41.327 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:54:41 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:54:41.327 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:54:41 smithi115 ceph-mon[78593]: pgmap v8358: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:54:41.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:54:41 smithi115 conmon[90393]: 2022-05-14T03:54:41.323+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:54:42.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:54:42 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:54:42.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:54:42 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 16015 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:54:42.327 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:54:42 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:54:42.327 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:54:42 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 16015 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:54:42.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:54:42 smithi115 conmon[90393]: 2022-05-14T03:54:42.323+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:54:43.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:54:43 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:54:43.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:54:43 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:54:43.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:54:43 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:54:43.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:54:43 smithi047 ceph-mon[77147]: pgmap v8359: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:54:43.306 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:54:43 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:54:43.306 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:54:43 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:54:43.306 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:54:43 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:54:43.306 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:54:43 smithi115 ceph-mon[78593]: pgmap v8359: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:54:43.610 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:54:43 smithi115 conmon[90393]: 2022-05-14T03:54:43.302+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:54:44.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:54:44 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:54:44.338 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:54:44 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:54:44.610 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:54:44 smithi115 conmon[90393]: 2022-05-14T03:54:44.335+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:54:45.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:54:45 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:54:45.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:54:45 smithi047 ceph-mon[77147]: pgmap v8360: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2022-05-14T03:54:45.312 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:54:45 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:54:45.312 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:54:45 smithi115 ceph-mon[78593]: pgmap v8360: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2022-05-14T03:54:45.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:54:45 smithi115 conmon[90393]: 2022-05-14T03:54:45.309+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:54:46.331 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:54:46 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:54:46.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:54:46 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:54:46.610 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:54:46 smithi115 conmon[90393]: 2022-05-14T03:54:46.328+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:54:47.334 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:54:47 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:54:47.334 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:54:47 smithi115 ceph-mon[78593]: pgmap v8361: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2022-05-14T03:54:47.334 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:54:47 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 16020 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:54:47.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:54:47 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:54:47.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:54:47 smithi047 ceph-mon[77147]: pgmap v8361: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2022-05-14T03:54:47.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:54:47 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 16020 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:54:47.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:54:47 smithi115 conmon[90393]: 2022-05-14T03:54:47.330+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:54:48.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:54:48 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:54:48.360 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:54:48 smithi115 conmon[90393]: 2022-05-14T03:54:48.283+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:54:48.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:54:48 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:54:49.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:54:49 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:54:49.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:54:49 smithi115 ceph-mon[78593]: pgmap v8362: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:54:49.360 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:54:49 smithi115 conmon[90393]: 2022-05-14T03:54:49.249+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:54:49.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:54:49 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:54:49.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:54:49 smithi047 ceph-mon[77147]: pgmap v8362: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:54:50.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:54:50 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:54:50.360 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:54:50 smithi115 conmon[90393]: 2022-05-14T03:54:50.248+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:54:50.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:54:50 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:54:51.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:54:51 smithi115 conmon[90393]: 2022-05-14T03:54:51.240+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:54:51.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:54:51 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:54:51.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:54:51 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:54:51.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:54:51 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:54:51.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:54:51 smithi115 ceph-mon[78593]: pgmap v8363: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:54:51.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:54:51 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:54:51.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:54:51 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:54:51.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:54:51 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:54:51.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:54:51 smithi047 ceph-mon[77147]: pgmap v8363: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:54:52.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:54:52 smithi115 conmon[90393]: 2022-05-14T03:54:52.279+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:54:52.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:54:52 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:54:52.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:54:52 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 16025 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:54:52.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:54:52 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:54:52.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:54:52 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 16025 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:54:53.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:54:53 smithi115 conmon[90393]: 2022-05-14T03:54:53.239+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:54:53.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:54:53 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:54:53.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:54:53 smithi115 ceph-mon[78593]: pgmap v8364: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:54:53.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:54:53 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:54:53.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:54:53 smithi047 ceph-mon[77147]: pgmap v8364: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:54:54.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:54:54 smithi115 conmon[90393]: 2022-05-14T03:54:54.195+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:54:54.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:54:54 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:54:54.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:54:54 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:54:55.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:54:55 smithi115 conmon[90393]: 2022-05-14T03:54:55.181+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:54:55.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:54:55 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:54:55.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:54:55 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:54:55.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:54:55 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:54:55.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:54:55 smithi115 ceph-mon[78593]: pgmap v8365: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2022-05-14T03:54:55.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:54:55 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:54:55.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:54:55 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:54:55.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:54:55 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:54:55.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:54:55 smithi047 ceph-mon[77147]: pgmap v8365: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2022-05-14T03:54:56.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:54:56 smithi115 conmon[90393]: 2022-05-14T03:54:56.154+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:54:56.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:54:56 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:54:56.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:54:56 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:54:57.360 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:54:57 smithi115 conmon[90393]: 2022-05-14T03:54:57.188+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:54:57.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:54:57 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:54:57.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:54:57 smithi115 ceph-mon[78593]: pgmap v8366: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2022-05-14T03:54:57.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:54:57 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 16030 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:54:57.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:54:57 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:54:57.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:54:57 smithi047 ceph-mon[77147]: pgmap v8366: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2022-05-14T03:54:57.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:54:57 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 16030 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:54:58.360 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:54:58 smithi115 conmon[90393]: 2022-05-14T03:54:58.165+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:54:58.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:54:58 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:54:58.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:54:58 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:54:58.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:54:58 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:54:58.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:54:58 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:54:59.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:54:59 smithi115 conmon[90393]: 2022-05-14T03:54:59.204+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:54:59.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:54:59 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:54:59.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:54:59 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:54:59.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:54:59 smithi115 ceph-mon[78593]: pgmap v8367: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:54:59.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:54:59 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:54:59.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:54:59 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:54:59.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:54:59 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:54:59.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:54:59 smithi047 ceph-mon[77147]: pgmap v8367: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:54:59.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:54:59 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:55:00.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:55:00 smithi115 conmon[90393]: 2022-05-14T03:55:00.217+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:55:00.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:55:00 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:55:00.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:55:00 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:55:01.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:55:01 smithi047 ceph-mon[77147]: pgmap v8368: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:55:01.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:55:01 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:55:01.610 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:55:01 smithi115 conmon[90393]: 2022-05-14T03:55:01.222+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:55:01.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:55:01 smithi115 ceph-mon[78593]: pgmap v8368: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:55:01.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:55:01 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:55:02.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:55:02 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:55:02.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:55:02 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:55:02.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:55:02 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:55:02.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:55:02 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 16035 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:55:02.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:55:02 smithi115 conmon[90393]: 2022-05-14T03:55:02.232+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:55:02.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:55:02 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:55:02.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:55:02 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:55:02.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:55:02 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:55:02.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:55:02 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 16035 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:55:03.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:55:03 smithi047 ceph-mon[77147]: pgmap v8369: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:55:03.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:55:03 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:55:03.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:55:03 smithi115 conmon[90393]: 2022-05-14T03:55:03.235+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:55:03.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:55:03 smithi115 ceph-mon[78593]: pgmap v8369: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:55:03.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:55:03 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:55:04.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:55:04 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:55:04.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:55:04 smithi115 conmon[90393]: 2022-05-14T03:55:04.208+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:55:04.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:55:04 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:55:05.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:55:05 smithi047 ceph-mon[77147]: pgmap v8370: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2022-05-14T03:55:05.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:55:05 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:55:05.610 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:55:05 smithi115 conmon[90393]: 2022-05-14T03:55:05.257+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:55:05.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:55:05 smithi115 ceph-mon[78593]: pgmap v8370: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2022-05-14T03:55:05.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:55:05 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:55:06.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:55:06 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:55:06.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:55:06 smithi115 conmon[90393]: 2022-05-14T03:55:06.277+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:55:06.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:55:06 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:55:07.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:55:07 smithi047 ceph-mon[77147]: pgmap v8371: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2022-05-14T03:55:07.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:55:07 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:55:07.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:55:07 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 16040 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:55:07.610 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:55:07 smithi115 conmon[90393]: 2022-05-14T03:55:07.322+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:55:07.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:55:07 smithi115 ceph-mon[78593]: pgmap v8371: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2022-05-14T03:55:07.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:55:07 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:55:07.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:55:07 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 16040 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:55:08.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:55:08 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:55:08.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:55:08 smithi115 conmon[90393]: 2022-05-14T03:55:08.281+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:55:08.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:55:08 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:55:09.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:55:09 smithi047 ceph-mon[77147]: pgmap v8372: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:55:09.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:55:09 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:55:09.610 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:55:09 smithi115 conmon[90393]: 2022-05-14T03:55:09.277+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:55:09.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:55:09 smithi115 ceph-mon[78593]: pgmap v8372: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:55:09.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:55:09 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:55:10.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:55:10 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:55:10.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:55:10 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:55:10.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:55:10 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:55:10.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:55:10 smithi115 conmon[90393]: 2022-05-14T03:55:10.316+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:55:10.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:55:10 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:55:10.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:55:10 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:55:10.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:55:10 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:55:11.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:55:11 smithi047 ceph-mon[77147]: pgmap v8373: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:55:11.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:55:11 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:55:11.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:55:11 smithi115 conmon[90393]: 2022-05-14T03:55:11.326+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:55:11.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:55:11 smithi115 ceph-mon[78593]: pgmap v8373: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:55:11.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:55:11 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:55:12.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:55:12 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:55:12.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:55:12 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 16045 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:55:12.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:55:12 smithi115 conmon[90393]: 2022-05-14T03:55:12.300+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:55:12.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:55:12 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:55:12.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:55:12 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 16045 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:55:13.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:55:13 smithi047 ceph-mon[77147]: pgmap v8374: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:55:13.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:55:13 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:55:13.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:55:13 smithi115 conmon[90393]: 2022-05-14T03:55:13.265+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:55:13.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:55:13 smithi115 ceph-mon[78593]: pgmap v8374: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:55:13.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:55:13 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:55:14.610 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:55:14 smithi115 conmon[90393]: 2022-05-14T03:55:14.224+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:55:15.232 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:55:14 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:55:15.233 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:55:14 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:55:15.233 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:55:14 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:55:15.233 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:55:14 smithi115 ceph-mon[78593]: pgmap v8375: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2022-05-14T03:55:15.233 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:55:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T03:55:15.234 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:55:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T03:55:15.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:55:14 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:55:15.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:55:14 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:55:15.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:55:14 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:55:15.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:55:14 smithi047 ceph-mon[77147]: pgmap v8375: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2022-05-14T03:55:15.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:55:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T03:55:15.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:55:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T03:55:15.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:55:15 smithi115 conmon[90393]: 2022-05-14T03:55:15.230+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:55:16.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:55:15 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:55:16.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:55:15 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:55:16.360 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:55:16 smithi115 conmon[90393]: 2022-05-14T03:55:16.202+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:55:17.253 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:55:16 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:55:17.254 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:55:16 smithi115 ceph-mon[78593]: pgmap v8376: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2022-05-14T03:55:17.254 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:55:16 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 16050 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:55:17.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:55:16 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:55:17.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:55:16 smithi047 ceph-mon[77147]: pgmap v8376: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2022-05-14T03:55:17.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:55:16 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 16050 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:55:17.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:55:17 smithi115 conmon[90393]: 2022-05-14T03:55:17.251+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:55:18.242 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:55:17 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:55:18.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:55:17 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:55:18.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:55:18 smithi115 conmon[90393]: 2022-05-14T03:55:18.239+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:55:19.236 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:55:18 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:55:19.237 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:55:18 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:55:19.237 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:55:18 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:55:19.237 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:55:18 smithi115 ceph-mon[78593]: pgmap v8377: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:55:19.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:55:18 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:55:19.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:55:18 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:55:19.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:55:18 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:55:19.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:55:18 smithi047 ceph-mon[77147]: pgmap v8377: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:55:19.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:55:19 smithi115 conmon[90393]: 2022-05-14T03:55:19.233+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:55:20.229 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:55:19 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:55:20.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:55:19 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:55:20.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:55:20 smithi115 conmon[90393]: 2022-05-14T03:55:20.226+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:55:21.242 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:55:20 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:55:21.242 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:55:20 smithi115 ceph-mon[78593]: pgmap v8378: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:55:21.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:55:20 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:55:21.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:55:20 smithi047 ceph-mon[77147]: pgmap v8378: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:55:21.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:55:21 smithi115 conmon[90393]: 2022-05-14T03:55:21.239+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:55:22.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:55:21 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:55:22.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:55:21 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 16055 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:55:22.360 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:55:22 smithi115 conmon[90393]: 2022-05-14T03:55:22.215+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:55:22.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:55:21 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:55:22.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:55:21 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 16055 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:55:23.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:55:22 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:55:23.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:55:22 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:55:23.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:55:22 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:55:23.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:55:22 smithi047 ceph-mon[77147]: pgmap v8379: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:55:23.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:55:23 smithi115 conmon[90393]: 2022-05-14T03:55:23.183+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:55:23.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:55:22 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:55:23.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:55:22 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:55:23.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:55:22 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:55:23.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:55:22 smithi115 ceph-mon[78593]: pgmap v8379: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:55:24.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:55:23 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:55:24.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:55:23 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:55:24.360 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:55:24 smithi115 conmon[90393]: 2022-05-14T03:55:24.163+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:55:25.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:55:24 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:55:25.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:55:24 smithi047 ceph-mon[77147]: pgmap v8380: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2022-05-14T03:55:25.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:55:24 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:55:25.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:55:24 smithi115 ceph-mon[78593]: pgmap v8380: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2022-05-14T03:55:25.360 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:55:25 smithi115 conmon[90393]: 2022-05-14T03:55:25.205+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:55:26.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:55:25 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:55:26.360 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:55:26 smithi115 conmon[90393]: 2022-05-14T03:55:26.169+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:55:26.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:55:25 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:55:27.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:55:27 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:55:27.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:55:27 smithi047 ceph-mon[77147]: pgmap v8381: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2022-05-14T03:55:27.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:55:27 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 16060 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:55:27.360 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:55:27 smithi115 conmon[90393]: 2022-05-14T03:55:27.177+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:55:27.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:55:27 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:55:27.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:55:27 smithi115 ceph-mon[78593]: pgmap v8381: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2022-05-14T03:55:27.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:55:27 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 16060 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:55:28.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:55:28 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:55:28.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:55:28 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:55:28.360 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:55:28 smithi115 conmon[90393]: 2022-05-14T03:55:28.205+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:55:29.008 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:55:28 smithi047 conmon[78521]: 2022-05-14T03:55:28.687+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:55:29.008 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:55:28 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T03:55:29.009 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:55:28 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:55:29.009 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:55:28 smithi047 conmon[78521]: yield 2022-05-14T03:55:29.009 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:55:28 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:55:29.009 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:55:28 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:55:29.010 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:55:28 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:55:29.010 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:55:28 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T03:55:29.010 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:55:28 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:55:29.010 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:55:28 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T03:55:29.010 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:55:28 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:55:29.011 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:55:28 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T03:55:29.011 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:55:28 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:55:29.011 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:55:28 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T03:55:29.011 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:55:28 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:55:29.012 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:55:28 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T03:55:29.012 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:55:28 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:55:29.012 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:55:28 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:55:29.012 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:55:28 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:55:29.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:55:29 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:55:29.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:55:29 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T03:55:29.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:55:29 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T03:55:29.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:55:29 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T03:55:29.295 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:55:29 smithi047 ceph-mon[77147]: pgmap v8382: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:55:29.360 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:55:29 smithi115 conmon[90393]: 2022-05-14T03:55:29.168+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:55:29.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:55:29 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:55:29.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:55:29 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T03:55:29.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:55:29 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T03:55:29.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:55:29 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T03:55:29.362 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:55:29 smithi115 ceph-mon[78593]: pgmap v8382: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:55:30.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:55:30 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:55:30.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:55:30 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:55:30.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T03:55:30.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:55:30.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T03:55:30.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:55:30.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:55:30.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:55:30.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T03:55:30.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:55:30.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T03:55:30.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:55:30.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T03:55:30.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:55:30.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T03:55:30.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:55:30.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T03:55:30.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:55:30.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:55:30.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:55:30.297 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:55:30 smithi047 ceph-mon[77147]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T03:55:30.360 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:55:30 smithi115 conmon[90393]: 2022-05-14T03:55:30.213+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:55:30.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:55:30 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:55:30.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:55:30 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:55:30.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T03:55:30.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:55:30.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T03:55:30.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:55:30.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:55:30.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:55:30.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T03:55:30.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:55:30.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T03:55:30.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:55:30.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T03:55:30.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:55:30.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T03:55:30.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:55:30.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T03:55:30.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:55:30.365 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:55:30.365 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:55:30.365 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:55:30 smithi115 ceph-mon[78593]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T03:55:31.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:55:31 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:55:31.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:55:31 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:55:31.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:55:31 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:55:31.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:55:31 smithi047 ceph-mon[77147]: pgmap v8383: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:55:31.360 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:55:31 smithi115 conmon[90393]: 2022-05-14T03:55:31.189+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:55:31.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:55:31 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:55:31.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:55:31 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:55:31.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:55:31 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:55:31.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:55:31 smithi115 ceph-mon[78593]: pgmap v8383: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:55:32.021 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:55:31 smithi047 conmon[78521]: 2022-05-14T03:55:31.759+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:55:32.021 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:55:31 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T03:55:32.021 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:55:31 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:55:32.022 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:55:31 smithi047 conmon[78521]: yield 2022-05-14T03:55:32.022 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:55:31 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:55:32.022 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:55:31 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:55:32.023 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:55:31 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:55:32.023 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:55:31 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T03:55:32.023 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:55:31 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:55:32.023 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:55:31 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T03:55:32.024 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:55:31 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:55:32.024 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:55:31 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T03:55:32.024 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:55:31 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:55:32.024 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:55:31 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T03:55:32.025 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:55:31 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:55:32.025 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:55:31 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T03:55:32.025 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:55:31 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:55:32.025 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:55:31 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:55:32.025 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:55:31 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:55:32.026 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:55:31 smithi047 conmon[78521]: 2022-05-14T03:55:31.760+0000 7f5344f82700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T03:55:32.026 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:55:31 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T03:55:32.026 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:55:31 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:55:32.026 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:55:31 smithi047 conmon[78521]: yield 2022-05-14T03:55:32.026 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:55:31 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:55:32.027 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:55:31 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:55:32.027 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:55:31 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:55:32.027 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:55:31 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T03:55:32.027 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:55:31 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:55:32.028 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:55:31 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T03:55:32.028 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:55:31 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:55:32.028 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:55:31 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T03:55:32.028 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:55:31 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:55:32.029 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:55:31 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T03:55:32.029 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:55:31 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:55:32.029 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:55:31 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T03:55:32.029 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:55:31 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:55:32.030 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:55:31 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:55:32.030 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:55:31 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:55:32.030 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:55:31 smithi047 conmon[78521]: 2022-05-14T03:55:32.030 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:55:31 smithi047 conmon[78521]: During handling of the above exception, another exception occurred: 2022-05-14T03:55:32.031 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:55:31 smithi047 conmon[78521]: 2022-05-14T03:55:32.031 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:55:31 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T03:55:32.031 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:55:31 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T03:55:32.031 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:55:31 smithi047 conmon[78521]: return f(*arg) 2022-05-14T03:55:32.031 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:55:31 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T03:55:32.032 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:55:31 smithi047 conmon[78521]: self._write_client_files(client_files, host) 2022-05-14T03:55:32.032 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:55:31 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T03:55:32.032 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:55:31 smithi047 conmon[78521]: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T03:55:32.032 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:55:31 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T03:55:32.032 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:55:31 smithi047 conmon[78521]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T03:55:32.033 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:55:31 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T03:55:32.033 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:55:31 smithi047 conmon[78521]: return self.event_loop.get_result(coro) 2022-05-14T03:55:32.033 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:55:31 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T03:55:32.033 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:55:31 smithi047 conmon[78521]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T03:55:32.033 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:55:31 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T03:55:32.034 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:55:31 smithi047 conmon[78521]: return self.__get_result() 2022-05-14T03:55:32.034 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:55:31 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T03:55:32.034 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:55:31 smithi047 conmon[78521]: raise self._exception 2022-05-14T03:55:32.034 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:55:31 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T03:55:32.035 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:55:31 smithi047 conmon[78521]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T03:55:32.035 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:55:31 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T03:55:32.035 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:55:31 smithi047 conmon[78521]: conn = await self._remote_connection(host, addr) 2022-05-14T03:55:32.035 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:55:31 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T03:55:32.035 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:55:31 smithi047 conmon[78521]: raise 2022-05-14T03:55:32.036 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:55:31 smithi047 conmon[78521]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T03:55:32.036 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:55:31 smithi047 conmon[78521]: self.gen.throw(type, value, traceback) 2022-05-14T03:55:32.036 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:55:31 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T03:55:32.036 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:55:31 smithi047 conmon[78521]: raise OrchestratorError(msg) 2022-05-14T03:55:32.037 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:55:31 smithi047 conmon[78521]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:55:32.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:55:32 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:55:32.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:55:32 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 16065 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:55:32.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:55:32 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:55:32.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:55:32 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 16065 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:55:32.360 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:55:32 smithi115 conmon[90393]: 2022-05-14T03:55:32.234+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:55:33.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:55:33 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:55:33.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:55:33 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:55:33.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T03:55:33.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:55:33.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T03:55:33.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:55:33.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:55:33.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:55:33.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T03:55:33.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:55:33.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T03:55:33.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:55:33.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T03:55:33.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:55:33.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T03:55:33.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:55:33.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T03:55:33.298 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:55:33.298 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:55:33.298 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:55:33.298 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:55:33 smithi047 ceph-mon[77147]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T03:55:33.299 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T03:55:33.299 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:55:33.299 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T03:55:33.299 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:55:33.299 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:55:33.300 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:55:33.300 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T03:55:33.300 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:55:33.300 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T03:55:33.300 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:55:33.301 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T03:55:33.301 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:55:33.301 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T03:55:33.301 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:55:33.302 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T03:55:33.302 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:55:33.302 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:55:33.302 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:55:33.302 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T03:55:33.303 INFO:journalctl@ceph.mon.b.smithi047.stdout: During handling of the above exception, another exception occurred: 2022-05-14T03:55:33.303 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T03:55:33.303 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T03:55:33.303 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T03:55:33.304 INFO:journalctl@ceph.mon.b.smithi047.stdout: return f(*arg) 2022-05-14T03:55:33.304 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T03:55:33.304 INFO:journalctl@ceph.mon.b.smithi047.stdout: self._write_client_files(client_files, host) 2022-05-14T03:55:33.304 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T03:55:33.304 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T03:55:33.305 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T03:55:33.305 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T03:55:33.305 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T03:55:33.305 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.event_loop.get_result(coro) 2022-05-14T03:55:33.305 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T03:55:33.306 INFO:journalctl@ceph.mon.b.smithi047.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T03:55:33.306 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T03:55:33.306 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.__get_result() 2022-05-14T03:55:33.306 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T03:55:33.306 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise self._exception 2022-05-14T03:55:33.307 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T03:55:33.307 INFO:journalctl@ceph.mon.b.smithi047.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T03:55:33.307 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T03:55:33.307 INFO:journalctl@ceph.mon.b.smithi047.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T03:55:33.308 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T03:55:33.308 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise 2022-05-14T03:55:33.308 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T03:55:33.308 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.gen.throw(type, value, traceback) 2022-05-14T03:55:33.308 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T03:55:33.309 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OrchestratorError(msg) 2022-05-14T03:55:33.309 INFO:journalctl@ceph.mon.b.smithi047.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:55:33.309 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:55:33 smithi047 ceph-mon[77147]: pgmap v8384: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:55:33.360 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:55:33 smithi115 conmon[90393]: 2022-05-14T03:55:33.226+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:55:33.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:55:33 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:55:33.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:55:33 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:55:33.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T03:55:33.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:55:33.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T03:55:33.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:55:33.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:55:33.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:55:33.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T03:55:33.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:55:33.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T03:55:33.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:55:33.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T03:55:33.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:55:33.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T03:55:33.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:55:33.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T03:55:33.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:55:33.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:55:33.365 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:55:33.365 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:55:33 smithi115 ceph-mon[78593]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T03:55:33.365 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T03:55:33.365 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:55:33.366 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T03:55:33.366 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:55:33.366 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:55:33.366 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:55:33.366 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T03:55:33.367 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:55:33.367 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T03:55:33.367 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:55:33.367 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T03:55:33.368 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:55:33.368 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T03:55:33.368 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:55:33.368 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T03:55:33.368 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:55:33.369 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:55:33.369 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:55:33.369 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T03:55:33.369 INFO:journalctl@ceph.mon.c.smithi115.stdout: During handling of the above exception, another exception occurred: 2022-05-14T03:55:33.370 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T03:55:33.370 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T03:55:33.370 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T03:55:33.370 INFO:journalctl@ceph.mon.c.smithi115.stdout: return f(*arg) 2022-05-14T03:55:33.370 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T03:55:33.371 INFO:journalctl@ceph.mon.c.smithi115.stdout: self._write_client_files(client_files, host) 2022-05-14T03:55:33.371 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T03:55:33.371 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T03:55:33.371 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T03:55:33.371 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T03:55:33.372 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T03:55:33.372 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.event_loop.get_result(coro) 2022-05-14T03:55:33.372 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T03:55:33.372 INFO:journalctl@ceph.mon.c.smithi115.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T03:55:33.373 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T03:55:33.373 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.__get_result() 2022-05-14T03:55:33.373 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T03:55:33.373 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise self._exception 2022-05-14T03:55:33.373 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T03:55:33.374 INFO:journalctl@ceph.mon.c.smithi115.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T03:55:33.374 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T03:55:33.374 INFO:journalctl@ceph.mon.c.smithi115.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T03:55:33.374 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T03:55:33.375 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise 2022-05-14T03:55:33.375 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T03:55:33.375 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.gen.throw(type, value, traceback) 2022-05-14T03:55:33.375 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T03:55:33.375 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OrchestratorError(msg) 2022-05-14T03:55:33.376 INFO:journalctl@ceph.mon.c.smithi115.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:55:33.376 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:55:33 smithi115 ceph-mon[78593]: pgmap v8384: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:55:34.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:55:34 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:55:34.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:55:34 smithi115 conmon[90393]: 2022-05-14T03:55:34.267+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:55:34.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:55:34 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:55:35.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:55:35 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:55:35.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:55:35 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:55:35.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:55:35 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:55:35.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:55:35 smithi047 ceph-mon[77147]: pgmap v8385: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2022-05-14T03:55:35.360 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:55:35 smithi115 conmon[90393]: 2022-05-14T03:55:35.274+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:55:35.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:55:35 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:55:35.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:55:35 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:55:35.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:55:35 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:55:35.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:55:35 smithi115 ceph-mon[78593]: pgmap v8385: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2022-05-14T03:55:36.288 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:55:36 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:55:36.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:55:36 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:55:36.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:55:36 smithi115 conmon[90393]: 2022-05-14T03:55:36.284+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:55:37.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:55:37 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:55:37.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:55:37 smithi047 ceph-mon[77147]: pgmap v8386: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2022-05-14T03:55:37.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:55:37 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 16070 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:55:37.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:55:37 smithi115 conmon[90393]: 2022-05-14T03:55:37.265+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:55:37.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:55:37 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:55:37.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:55:37 smithi115 ceph-mon[78593]: pgmap v8386: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2022-05-14T03:55:37.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:55:37 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 16070 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:55:38.315 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:55:38 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:55:38.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:55:38 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:55:38.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:55:38 smithi115 conmon[90393]: 2022-05-14T03:55:38.312+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:55:39.346 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:55:39 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:55:39.346 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:55:39 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:55:39.347 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:55:39 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:55:39.347 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:55:39 smithi115 ceph-mon[78593]: pgmap v8387: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:55:39.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:55:39 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:55:39.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:55:39 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:55:39.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:55:39 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:55:39.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:55:39 smithi047 ceph-mon[77147]: pgmap v8387: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:55:39.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:55:39 smithi115 conmon[90393]: 2022-05-14T03:55:39.341+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:55:40.326 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:55:40 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:55:40.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:55:40 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:55:40.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:55:40 smithi115 conmon[90393]: 2022-05-14T03:55:40.323+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:55:41.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:55:41 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:55:41.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:55:41 smithi115 ceph-mon[78593]: pgmap v8388: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:55:41.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:55:41 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:55:41.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:55:41 smithi047 ceph-mon[77147]: pgmap v8388: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:55:41.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:55:41 smithi115 conmon[90393]: 2022-05-14T03:55:41.369+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:55:42.329 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:55:42 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:55:42.330 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:55:42 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 16075 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:55:42.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:55:42 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:55:42.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:55:42 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 16075 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:55:42.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:55:42 smithi115 conmon[90393]: 2022-05-14T03:55:42.326+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:55:43.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:55:43 smithi115 conmon[90393]: 2022-05-14T03:55:43.303+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:55:43.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:55:43 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:55:43.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:55:43 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:55:43.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:55:43 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:55:43.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:55:43 smithi115 ceph-mon[78593]: pgmap v8389: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:55:43.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:55:43 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:55:43.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:55:43 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:55:43.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:55:43 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:55:43.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:55:43 smithi047 ceph-mon[77147]: pgmap v8389: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:55:44.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:55:44 smithi115 conmon[90393]: 2022-05-14T03:55:44.289+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:55:44.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:55:44 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:55:44.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:55:44 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:55:45.360 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:55:45 smithi115 conmon[90393]: 2022-05-14T03:55:45.275+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:55:45.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:55:45 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:55:45.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:55:45 smithi115 ceph-mon[78593]: pgmap v8390: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2022-05-14T03:55:45.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:55:45 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:55:45.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:55:45 smithi047 ceph-mon[77147]: pgmap v8390: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2022-05-14T03:55:46.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:55:46 smithi115 conmon[90393]: 2022-05-14T03:55:46.244+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:55:46.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:55:46 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:55:46.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:55:46 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:55:47.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:55:47 smithi115 conmon[90393]: 2022-05-14T03:55:47.235+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:55:47.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:55:47 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:55:47.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:55:47 smithi115 ceph-mon[78593]: pgmap v8391: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2022-05-14T03:55:47.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:55:47 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 16080 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:55:47.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:55:47 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:55:47.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:55:47 smithi047 ceph-mon[77147]: pgmap v8391: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2022-05-14T03:55:47.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:55:47 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 16080 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:55:48.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:55:48 smithi115 conmon[90393]: 2022-05-14T03:55:48.270+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:55:48.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:55:48 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:55:48.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:55:48 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:55:49.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:55:49 smithi115 conmon[90393]: 2022-05-14T03:55:49.311+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:55:49.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:55:49 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:55:49.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:55:49 smithi115 ceph-mon[78593]: pgmap v8392: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:55:49.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:55:49 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:55:49.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:55:49 smithi047 ceph-mon[77147]: pgmap v8392: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:55:50.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:55:50 smithi115 conmon[90393]: 2022-05-14T03:55:50.311+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:55:50.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:55:50 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:55:50.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:55:50 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:55:51.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:55:51 smithi115 conmon[90393]: 2022-05-14T03:55:51.317+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:55:51.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:55:51 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:55:51.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:55:51 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:55:51.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:55:51 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:55:51.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:55:51 smithi115 ceph-mon[78593]: pgmap v8393: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:55:51.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:55:51 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:55:51.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:55:51 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:55:51.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:55:51 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:55:51.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:55:51 smithi047 ceph-mon[77147]: pgmap v8393: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:55:52.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:55:52 smithi115 conmon[90393]: 2022-05-14T03:55:52.278+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:55:52.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:55:52 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:55:52.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:55:52 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 16085 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:55:52.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:55:52 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:55:52.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:55:52 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 16085 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:55:53.360 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:55:53 smithi115 conmon[90393]: 2022-05-14T03:55:53.248+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:55:53.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:55:53 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:55:53.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:55:53 smithi115 ceph-mon[78593]: pgmap v8394: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:55:53.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:55:53 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:55:53.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:55:53 smithi047 ceph-mon[77147]: pgmap v8394: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:55:54.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:55:54 smithi115 conmon[90393]: 2022-05-14T03:55:54.236+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:55:54.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:55:54 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:55:54.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:55:54 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:55:54.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:55:54 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:55:54.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:55:54 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:55:54.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:55:54 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:55:54.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:55:54 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:55:54.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:55:54 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:55:54.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:55:54 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:55:55.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:55:55 smithi047 ceph-mon[77147]: pgmap v8395: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2022-05-14T03:55:55.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:55:55 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:55:55.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:55:55 smithi115 conmon[90393]: 2022-05-14T03:55:55.191+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:55:55.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:55:55 smithi115 ceph-mon[78593]: pgmap v8395: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2022-05-14T03:55:55.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:55:55 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:55:56.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:55:56 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:55:56.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:55:56 smithi115 conmon[90393]: 2022-05-14T03:55:56.227+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:55:56.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:55:56 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:55:57.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:55:57 smithi047 ceph-mon[77147]: pgmap v8396: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2022-05-14T03:55:57.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:55:57 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:55:57.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:55:57 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 16090 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:55:57.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:55:57 smithi115 conmon[90393]: 2022-05-14T03:55:57.209+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:55:57.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:55:57 smithi115 ceph-mon[78593]: pgmap v8396: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2022-05-14T03:55:57.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:55:57 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:55:57.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:55:57 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 16090 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:55:58.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:55:58 smithi115 conmon[90393]: 2022-05-14T03:55:58.160+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:55:59.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:55:58 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:55:59.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:55:58 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:55:59.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:55:58 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:55:59.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:55:58 smithi047 ceph-mon[77147]: pgmap v8397: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:55:59.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:55:59 smithi115 conmon[90393]: 2022-05-14T03:55:59.125+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:55:59.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:55:58 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:55:59.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:55:58 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:55:59.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:55:58 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:55:59.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:55:58 smithi115 ceph-mon[78593]: pgmap v8397: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:56:00.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:55:59 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:56:00.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:56:00 smithi115 conmon[90393]: 2022-05-14T03:56:00.130+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:56:00.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:55:59 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:56:01.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:56:00 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:56:01.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:56:00 smithi047 ceph-mon[77147]: pgmap v8398: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:56:01.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:56:01 smithi115 conmon[90393]: 2022-05-14T03:56:01.084+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:56:01.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:56:00 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:56:01.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:56:00 smithi115 ceph-mon[78593]: pgmap v8398: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:56:02.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:56:01 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:56:02.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:56:01 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:56:02.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:56:01 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 16095 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:56:02.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:56:02 smithi115 conmon[90393]: 2022-05-14T03:56:02.047+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:56:02.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:56:01 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:56:02.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:56:01 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:56:02.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:56:01 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 16095 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:56:03.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:56:02 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:56:03.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:56:02 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:56:03.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:56:02 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:56:03.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:56:02 smithi047 ceph-mon[77147]: pgmap v8399: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:56:03.360 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:56:03 smithi115 conmon[90393]: 2022-05-14T03:56:03.005+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:56:03.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:56:02 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:56:03.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:56:02 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:56:03.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:56:02 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:56:03.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:56:02 smithi115 ceph-mon[78593]: pgmap v8399: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:56:04.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:56:03 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:56:04.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:56:03 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:56:04.360 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:56:03 smithi115 conmon[90393]: 2022-05-14T03:56:03.965+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:56:05.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:56:04 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:56:05.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:56:04 smithi047 ceph-mon[77147]: pgmap v8400: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2022-05-14T03:56:05.360 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:56:05 smithi115 conmon[90393]: 2022-05-14T03:56:05.005+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:56:05.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:56:04 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:56:05.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:56:04 smithi115 ceph-mon[78593]: pgmap v8400: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2022-05-14T03:56:06.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:56:05 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:56:06.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:56:05 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:56:06.360 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:56:05 smithi115 conmon[90393]: 2022-05-14T03:56:05.989+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:56:07.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:56:06 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:56:07.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:56:06 smithi047 ceph-mon[77147]: pgmap v8401: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2022-05-14T03:56:07.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:56:06 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 16100 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:56:07.360 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:56:06 smithi115 conmon[90393]: 2022-05-14T03:56:06.983+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:56:07.362 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:56:06 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:56:07.362 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:56:06 smithi115 ceph-mon[78593]: pgmap v8401: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2022-05-14T03:56:07.363 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:56:06 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 16100 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:56:08.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:56:07 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:56:08.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:56:07 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:56:08.360 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:56:08 smithi115 conmon[90393]: 2022-05-14T03:56:08.011+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:56:09.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:56:09 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:56:09.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:56:09 smithi047 ceph-mon[77147]: pgmap v8402: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:56:09.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:56:08 smithi115 conmon[90393]: 2022-05-14T03:56:08.979+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:56:09.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:56:09 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:56:09.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:56:09 smithi115 ceph-mon[78593]: pgmap v8402: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:56:10.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:56:10 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:56:10.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:56:10 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:56:10.360 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:56:09 smithi115 conmon[90393]: 2022-05-14T03:56:09.948+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:56:11.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:56:11 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:56:11.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:56:11 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:56:11.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:56:11 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:56:11.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:56:11 smithi047 ceph-mon[77147]: pgmap v8403: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:56:11.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:56:10 smithi115 conmon[90393]: 2022-05-14T03:56:10.975+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:56:11.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:56:11 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:56:11.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:56:11 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:56:11.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:56:11 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:56:11.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:56:11 smithi115 ceph-mon[78593]: pgmap v8403: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:56:12.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:56:12 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:56:12.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:56:12 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 16105 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:56:12.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:56:12 smithi115 conmon[90393]: 2022-05-14T03:56:12.009+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:56:12.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:56:12 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:56:12.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:56:12 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 16105 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:56:13.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:56:13 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:56:13.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:56:13 smithi047 ceph-mon[77147]: pgmap v8404: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:56:13.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:56:13 smithi115 conmon[90393]: 2022-05-14T03:56:13.014+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:56:13.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:56:13 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:56:13.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:56:13 smithi115 ceph-mon[78593]: pgmap v8404: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:56:14.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:56:14 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:56:14.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:56:14 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:56:14.360 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:56:13 smithi115 conmon[90393]: 2022-05-14T03:56:13.976+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:56:15.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:56:15 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:56:15.306 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:56:15 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:56:15.306 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:56:15 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:56:15.307 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:56:15 smithi047 ceph-mon[77147]: pgmap v8405: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2022-05-14T03:56:15.307 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:56:15 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T03:56:15.307 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:56:15 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T03:56:15.360 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:56:15 smithi115 conmon[90393]: 2022-05-14T03:56:15.010+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:56:15.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:56:15 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:56:15.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:56:15 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:56:15.362 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:56:15 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:56:15.362 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:56:15 smithi115 ceph-mon[78593]: pgmap v8405: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2022-05-14T03:56:15.363 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:56:15 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T03:56:15.363 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:56:15 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T03:56:16.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:56:16 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:56:16.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:56:16 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:56:16.360 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:56:16 smithi115 conmon[90393]: 2022-05-14T03:56:16.047+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:56:17.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:56:17 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:56:17.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:56:17 smithi047 ceph-mon[77147]: pgmap v8406: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2022-05-14T03:56:17.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:56:17 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 16110 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:56:17.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:56:17 smithi115 conmon[90393]: 2022-05-14T03:56:17.078+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:56:17.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:56:17 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:56:17.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:56:17 smithi115 ceph-mon[78593]: pgmap v8406: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2022-05-14T03:56:17.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:56:17 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 16110 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:56:18.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:56:18 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:56:18.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:56:18 smithi115 conmon[90393]: 2022-05-14T03:56:18.088+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:56:18.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:56:18 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:56:19.360 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:56:19 smithi115 conmon[90393]: 2022-05-14T03:56:19.113+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:56:19.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:56:19 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:56:19.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:56:19 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:56:19.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:56:19 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:56:19.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:56:19 smithi115 ceph-mon[78593]: pgmap v8407: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:56:19.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:56:19 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:56:19.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:56:19 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:56:19.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:56:19 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:56:19.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:56:19 smithi047 ceph-mon[77147]: pgmap v8407: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:56:20.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:56:20 smithi115 conmon[90393]: 2022-05-14T03:56:20.066+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:56:20.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:56:20 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:56:20.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:56:20 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:56:20.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:56:20 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:56:20.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:56:20 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:56:21.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:56:21 smithi115 ceph-mon[78593]: pgmap v8408: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:56:21.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:56:21 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:56:21.360 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:56:21 smithi115 conmon[90393]: 2022-05-14T03:56:21.097+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:56:21.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:56:21 smithi047 ceph-mon[77147]: pgmap v8408: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:56:21.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:56:21 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:56:22.360 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:56:22 smithi115 conmon[90393]: 2022-05-14T03:56:22.068+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:56:22.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:56:22 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:56:22.362 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:56:22 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:56:22.362 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:56:22 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 16115 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:56:22.362 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:56:22 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:56:22.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:56:22 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:56:22.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:56:22 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:56:22.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:56:22 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 16115 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:56:22.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:56:22 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:56:23.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:56:23 smithi047 ceph-mon[77147]: pgmap v8409: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:56:23.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:56:23 smithi115 ceph-mon[78593]: pgmap v8409: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:56:23.610 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:56:23 smithi115 conmon[90393]: 2022-05-14T03:56:23.118+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:56:24.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:56:24 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:56:24.610 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:56:24 smithi115 conmon[90393]: 2022-05-14T03:56:24.131+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:56:24.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:56:24 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:56:25.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:56:25 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:56:25.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:56:25 smithi047 ceph-mon[77147]: pgmap v8410: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2022-05-14T03:56:25.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:56:25 smithi115 conmon[90393]: 2022-05-14T03:56:25.125+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:56:25.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:56:25 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:56:25.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:56:25 smithi115 ceph-mon[78593]: pgmap v8410: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2022-05-14T03:56:26.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:56:26 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:56:26.360 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:56:26 smithi115 conmon[90393]: 2022-05-14T03:56:26.104+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:56:26.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:56:26 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:56:27.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:56:27 smithi115 conmon[90393]: 2022-05-14T03:56:27.082+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:56:27.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:56:27 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:56:27.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:56:27 smithi115 ceph-mon[78593]: pgmap v8411: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2022-05-14T03:56:27.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:56:27 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 16120 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:56:27.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:56:27 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:56:27.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:56:27 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:56:27.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:56:27 smithi047 ceph-mon[77147]: pgmap v8411: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2022-05-14T03:56:27.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:56:27 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 16120 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:56:27.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:56:27 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:56:28.610 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:56:28 smithi115 conmon[90393]: 2022-05-14T03:56:28.109+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:56:29.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:56:29 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:56:29.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:56:29 smithi047 ceph-mon[77147]: pgmap v8412: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:56:29.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:56:29 smithi115 conmon[90393]: 2022-05-14T03:56:29.114+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:56:29.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:56:29 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:56:29.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:56:29 smithi115 ceph-mon[78593]: pgmap v8412: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:56:30.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:56:30 smithi115 conmon[90393]: 2022-05-14T03:56:30.065+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:56:30.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:56:30 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:56:30.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:56:30 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:56:30.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:56:30 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:56:30.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:56:30 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:56:30.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:56:30 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:56:30.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:56:30 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:56:30.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:56:30 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:56:30.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:56:30 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:56:31.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:56:31 smithi115 ceph-mon[78593]: pgmap v8413: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:56:31.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:56:31 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:56:31.360 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:56:31 smithi115 conmon[90393]: 2022-05-14T03:56:31.019+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:56:31.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:56:31 smithi047 ceph-mon[77147]: pgmap v8413: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:56:31.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:56:31 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:56:32.110 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:56:31 smithi115 conmon[90393]: 2022-05-14T03:56:31.971+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:56:32.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:56:32 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T03:56:32.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:56:32 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T03:56:32.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:56:32 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T03:56:32.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:56:32 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 16125 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:56:32.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:56:32 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:56:32.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:56:32 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T03:56:32.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:56:32 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T03:56:32.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:56:32 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T03:56:32.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:56:32 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 16125 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:56:32.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:56:32 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:56:33.360 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:56:33 smithi115 conmon[90393]: 2022-05-14T03:56:33.000+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:56:33.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:56:33 smithi115 ceph-mon[78593]: pgmap v8414: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:56:33.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:56:33 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:56:33.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:56:33 smithi047 ceph-mon[77147]: pgmap v8414: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:56:33.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:56:33 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:56:34.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:56:33 smithi115 conmon[90393]: 2022-05-14T03:56:33.963+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:56:35.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:56:34 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:56:35.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:56:34 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:56:35.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:56:34 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:56:35.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:56:34 smithi047 ceph-mon[77147]: pgmap v8415: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2022-05-14T03:56:35.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:56:34 smithi047 conmon[78521]: 2022-05-14T03:56:34.864+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:56:35.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:56:34 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T03:56:35.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:56:34 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:56:35.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:56:34 smithi047 conmon[78521]: yield 2022-05-14T03:56:35.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:56:34 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:56:35.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:56:34 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:56:35.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:56:34 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:56:35.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:56:34 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T03:56:35.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:56:34 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:56:35.297 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:56:34 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T03:56:35.297 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:56:34 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:56:35.297 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:56:34 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T03:56:35.297 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:56:34 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:56:35.297 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:56:34 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T03:56:35.298 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:56:34 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:56:35.298 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:56:34 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T03:56:35.298 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:56:34 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:56:35.298 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:56:34 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:56:35.299 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:56:34 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:56:35.360 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:56:35 smithi115 conmon[90393]: 2022-05-14T03:56:35.006+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:56:35.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:56:34 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:56:35.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:56:34 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:56:35.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:56:34 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:56:35.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:56:34 smithi115 ceph-mon[78593]: pgmap v8415: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2022-05-14T03:56:36.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:56:35 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:56:36.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T03:56:36.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:56:36.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T03:56:36.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:56:36.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:56:36.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:56:36.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T03:56:36.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:56:36.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T03:56:36.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:56:36.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T03:56:36.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:56:36.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T03:56:36.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:56:36.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T03:56:36.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:56:36.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:56:36.298 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:56:36.298 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:56:35 smithi047 ceph-mon[77147]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T03:56:36.298 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:56:35 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:56:36.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:56:36 smithi115 conmon[90393]: 2022-05-14T03:56:36.008+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:56:36.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:56:35 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:56:36.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T03:56:36.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:56:36.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T03:56:36.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:56:36.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:56:36.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:56:36.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T03:56:36.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:56:36.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T03:56:36.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:56:36.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T03:56:36.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:56:36.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T03:56:36.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:56:36.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T03:56:36.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:56:36.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:56:36.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:56:36.364 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:56:35 smithi115 ceph-mon[78593]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T03:56:36.365 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:56:35 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:56:37.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:56:36 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:56:37.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:56:36 smithi047 ceph-mon[77147]: pgmap v8416: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2022-05-14T03:56:37.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:56:36 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 16130 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:56:37.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:56:36 smithi115 conmon[90393]: 2022-05-14T03:56:36.987+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:56:37.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:56:36 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:56:37.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:56:36 smithi115 ceph-mon[78593]: pgmap v8416: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2022-05-14T03:56:37.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:56:36 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 16130 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:56:38.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:56:37 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:56:38.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:56:37 smithi047 conmon[78521]: 2022-05-14T03:56:37.936+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:56:38.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:56:37 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T03:56:38.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:56:37 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:56:38.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:56:37 smithi047 conmon[78521]: yield 2022-05-14T03:56:38.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:56:37 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:56:38.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:56:37 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:56:38.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:56:37 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:56:38.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:56:37 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T03:56:38.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:56:37 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:56:38.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:56:37 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T03:56:38.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:56:37 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:56:38.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:56:37 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T03:56:38.297 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:56:37 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:56:38.297 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:56:37 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T03:56:38.297 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:56:37 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:56:38.297 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:56:37 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T03:56:38.298 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:56:37 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:56:38.298 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:56:37 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:56:38.298 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:56:37 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:56:38.298 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:56:37 smithi047 conmon[78521]: 2022-05-14T03:56:37.937+0000 7f533ef76700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T03:56:38.299 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:56:37 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T03:56:38.299 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:56:37 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:56:38.299 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:56:37 smithi047 conmon[78521]: yield 2022-05-14T03:56:38.299 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:56:37 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:56:38.300 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:56:37 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:56:38.300 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:56:37 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:56:38.300 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:56:37 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T03:56:38.301 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:56:37 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:56:38.301 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:56:37 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T03:56:38.301 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:56:37 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:56:38.301 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:56:37 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T03:56:38.302 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:56:37 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:56:38.302 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:56:37 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T03:56:38.302 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:56:37 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:56:38.302 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:56:37 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T03:56:38.303 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:56:37 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:56:38.303 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:56:37 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:56:38.303 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:56:37 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:56:38.303 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:56:37 smithi047 conmon[78521]: 2022-05-14T03:56:38.303 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:56:37 smithi047 conmon[78521]: During handling of the above exception, another exception occurred: 2022-05-14T03:56:38.304 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:56:37 smithi047 conmon[78521]: 2022-05-14T03:56:38.304 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:56:37 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T03:56:38.304 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:56:37 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T03:56:38.305 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:56:37 smithi047 conmon[78521]: return f(*arg) 2022-05-14T03:56:38.305 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:56:37 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T03:56:38.305 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:56:37 smithi047 conmon[78521]: self._write_client_files(client_files, host) 2022-05-14T03:56:38.305 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:56:37 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T03:56:38.305 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:56:37 smithi047 conmon[78521]: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T03:56:38.306 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:56:37 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T03:56:38.306 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:56:37 smithi047 conmon[78521]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T03:56:38.306 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:56:37 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T03:56:38.306 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:56:37 smithi047 conmon[78521]: return self.event_loop.get_result(coro) 2022-05-14T03:56:38.307 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:56:37 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T03:56:38.307 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:56:37 smithi047 conmon[78521]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T03:56:38.307 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:56:37 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T03:56:38.307 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:56:37 smithi047 conmon[78521]: return self.__get_result() 2022-05-14T03:56:38.307 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:56:37 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T03:56:38.308 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:56:37 smithi047 conmon[78521]: raise self._exception 2022-05-14T03:56:38.308 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:56:37 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T03:56:38.308 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:56:37 smithi047 conmon[78521]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T03:56:38.308 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:56:37 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T03:56:38.309 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:56:37 smithi047 conmon[78521]: conn = await self._remote_connection(host, addr) 2022-05-14T03:56:38.309 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:56:37 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T03:56:38.309 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:56:37 smithi047 conmon[78521]: raise 2022-05-14T03:56:38.309 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:56:37 smithi047 conmon[78521]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T03:56:38.309 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:56:37 smithi047 conmon[78521]: self.gen.throw(type, value, traceback) 2022-05-14T03:56:38.310 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:56:37 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T03:56:38.310 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:56:37 smithi047 conmon[78521]: raise OrchestratorError(msg) 2022-05-14T03:56:38.310 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:56:37 smithi047 conmon[78521]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:56:38.360 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:56:37 smithi115 conmon[90393]: 2022-05-14T03:56:37.960+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:56:38.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:56:37 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:56:39.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:56:38 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:56:39.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T03:56:39.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:56:39.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T03:56:39.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:56:39.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:56:39.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:56:39.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T03:56:39.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:56:39.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T03:56:39.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:56:39.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T03:56:39.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:56:39.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T03:56:39.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:56:39.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T03:56:39.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:56:39.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:56:39.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:56:39.298 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:56:38 smithi047 ceph-mon[77147]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T03:56:39.298 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T03:56:39.298 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:56:39.298 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T03:56:39.298 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:56:39.299 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:56:39.299 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:56:39.299 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T03:56:39.299 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:56:39.299 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T03:56:39.300 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:56:39.300 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T03:56:39.300 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:56:39.300 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T03:56:39.301 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:56:39.301 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T03:56:39.301 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:56:39.302 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:56:39.302 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:56:39.302 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T03:56:39.303 INFO:journalctl@ceph.mon.b.smithi047.stdout: During handling of the above exception, another exception occurred: 2022-05-14T03:56:39.303 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T03:56:39.303 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T03:56:39.303 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T03:56:39.303 INFO:journalctl@ceph.mon.b.smithi047.stdout: return f(*arg) 2022-05-14T03:56:39.304 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T03:56:39.304 INFO:journalctl@ceph.mon.b.smithi047.stdout: self._write_client_files(client_files, host) 2022-05-14T03:56:39.304 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T03:56:39.304 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T03:56:39.305 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T03:56:39.305 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T03:56:39.305 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T03:56:39.305 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.event_loop.get_result(coro) 2022-05-14T03:56:39.305 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T03:56:39.306 INFO:journalctl@ceph.mon.b.smithi047.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T03:56:39.306 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T03:56:39.306 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.__get_result() 2022-05-14T03:56:39.306 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T03:56:39.306 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise self._exception 2022-05-14T03:56:39.307 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T03:56:39.307 INFO:journalctl@ceph.mon.b.smithi047.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T03:56:39.307 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T03:56:39.307 INFO:journalctl@ceph.mon.b.smithi047.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T03:56:39.307 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T03:56:39.308 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise 2022-05-14T03:56:39.308 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T03:56:39.308 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.gen.throw(type, value, traceback) 2022-05-14T03:56:39.308 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T03:56:39.308 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OrchestratorError(msg) 2022-05-14T03:56:39.309 INFO:journalctl@ceph.mon.b.smithi047.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:56:39.309 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:56:38 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:56:39.309 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:56:38 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:56:39.309 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:56:38 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:56:39.310 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:56:38 smithi047 ceph-mon[77147]: pgmap v8417: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:56:39.360 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:56:38 smithi115 conmon[90393]: 2022-05-14T03:56:38.986+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:56:39.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:56:38 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:56:39.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T03:56:39.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:56:39.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T03:56:39.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:56:39.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:56:39.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:56:39.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T03:56:39.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:56:39.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T03:56:39.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:56:39.365 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T03:56:39.365 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:56:39.366 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T03:56:39.366 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:56:39.367 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T03:56:39.367 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:56:39.367 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:56:39.368 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:56:39.368 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:56:38 smithi115 ceph-mon[78593]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T03:56:39.368 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T03:56:39.369 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:56:39.369 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T03:56:39.370 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:56:39.370 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:56:39.370 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:56:39.371 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T03:56:39.371 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:56:39.372 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T03:56:39.372 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:56:39.373 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T03:56:39.373 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:56:39.373 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T03:56:39.374 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:56:39.374 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T03:56:39.374 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:56:39.374 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:56:39.375 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:56:39.375 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T03:56:39.375 INFO:journalctl@ceph.mon.c.smithi115.stdout: During handling of the above exception, another exception occurred: 2022-05-14T03:56:39.376 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T03:56:39.376 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T03:56:39.376 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T03:56:39.376 INFO:journalctl@ceph.mon.c.smithi115.stdout: return f(*arg) 2022-05-14T03:56:39.377 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T03:56:39.377 INFO:journalctl@ceph.mon.c.smithi115.stdout: self._write_client_files(client_files, host) 2022-05-14T03:56:39.377 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T03:56:39.378 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T03:56:39.378 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T03:56:39.378 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T03:56:39.378 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T03:56:39.379 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.event_loop.get_result(coro) 2022-05-14T03:56:39.379 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T03:56:39.379 INFO:journalctl@ceph.mon.c.smithi115.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T03:56:39.380 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T03:56:39.380 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.__get_result() 2022-05-14T03:56:39.380 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T03:56:39.381 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise self._exception 2022-05-14T03:56:39.381 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T03:56:39.381 INFO:journalctl@ceph.mon.c.smithi115.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T03:56:39.381 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T03:56:39.382 INFO:journalctl@ceph.mon.c.smithi115.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T03:56:39.382 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T03:56:39.382 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise 2022-05-14T03:56:39.382 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T03:56:39.383 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.gen.throw(type, value, traceback) 2022-05-14T03:56:39.383 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T03:56:39.383 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OrchestratorError(msg) 2022-05-14T03:56:39.384 INFO:journalctl@ceph.mon.c.smithi115.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:56:39.384 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:56:38 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:56:39.384 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:56:38 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:56:39.384 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:56:38 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:56:39.384 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:56:38 smithi115 ceph-mon[78593]: pgmap v8417: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:56:40.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:56:39 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:56:40.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:56:40 smithi115 conmon[90393]: 2022-05-14T03:56:40.007+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:56:40.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:56:39 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:56:41.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:56:40 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:56:41.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:56:40 smithi047 ceph-mon[77147]: pgmap v8418: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:56:41.360 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:56:41 smithi115 conmon[90393]: 2022-05-14T03:56:41.025+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:56:41.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:56:40 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:56:41.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:56:40 smithi115 ceph-mon[78593]: pgmap v8418: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:56:42.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:56:42 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:56:42.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:56:42 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 16135 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:56:42.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:56:41 smithi115 conmon[90393]: 2022-05-14T03:56:41.995+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:56:42.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:56:42 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:56:42.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:56:42 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 16135 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:56:43.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:56:43 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:56:43.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:56:43 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:56:43.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:56:43 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:56:43.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:56:43 smithi047 ceph-mon[77147]: pgmap v8419: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:56:43.360 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:56:43 smithi115 conmon[90393]: 2022-05-14T03:56:43.019+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:56:43.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:56:43 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:56:43.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:56:43 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:56:43.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:56:43 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:56:43.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:56:43 smithi115 ceph-mon[78593]: pgmap v8419: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:56:44.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:56:44 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:56:44.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:56:44 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:56:44.360 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:56:44 smithi115 conmon[90393]: 2022-05-14T03:56:44.015+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:56:45.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:56:45 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:56:45.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:56:45 smithi047 ceph-mon[77147]: pgmap v8420: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2022-05-14T03:56:45.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:56:44 smithi115 conmon[90393]: 2022-05-14T03:56:44.983+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:56:45.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:56:45 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:56:45.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:56:45 smithi115 ceph-mon[78593]: pgmap v8420: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2022-05-14T03:56:46.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:56:46 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:56:46.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:56:46 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:56:46.360 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:56:46 smithi115 conmon[90393]: 2022-05-14T03:56:46.030+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:56:47.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:56:47 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:56:47.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:56:47 smithi047 ceph-mon[77147]: pgmap v8421: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2022-05-14T03:56:47.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:56:47 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 16140 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:56:47.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:56:47 smithi115 conmon[90393]: 2022-05-14T03:56:47.037+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:56:47.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:56:47 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:56:47.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:56:47 smithi115 ceph-mon[78593]: pgmap v8421: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2022-05-14T03:56:47.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:56:47 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 16140 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:56:48.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:56:48 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:56:48.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:56:48 smithi115 conmon[90393]: 2022-05-14T03:56:48.001+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:56:48.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:56:48 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:56:49.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:56:49 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:56:49.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:56:49 smithi047 ceph-mon[77147]: pgmap v8422: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:56:49.360 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:56:49 smithi115 conmon[90393]: 2022-05-14T03:56:49.030+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:56:49.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:56:49 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:56:49.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:56:49 smithi115 ceph-mon[78593]: pgmap v8422: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:56:50.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:56:50 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:56:50.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:56:50 smithi115 conmon[90393]: 2022-05-14T03:56:50.047+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:56:50.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:56:50 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:56:51.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:56:51 smithi115 conmon[90393]: 2022-05-14T03:56:51.058+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:56:51.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:56:51 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:56:51.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:56:51 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:56:51.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:56:51 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:56:51.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:56:51 smithi115 ceph-mon[78593]: pgmap v8423: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:56:51.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:56:51 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:56:51.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:56:51 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:56:51.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:56:51 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:56:51.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:56:51 smithi047 ceph-mon[77147]: pgmap v8423: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:56:52.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:56:52 smithi115 conmon[90393]: 2022-05-14T03:56:52.026+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:56:52.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:56:52 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:56:52.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:56:52 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 16145 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:56:52.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:56:52 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:56:52.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:56:52 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 16145 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:56:53.360 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:56:53 smithi115 conmon[90393]: 2022-05-14T03:56:53.013+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:56:53.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:56:53 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:56:53.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:56:53 smithi115 ceph-mon[78593]: pgmap v8424: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:56:53.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:56:53 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:56:53.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:56:53 smithi047 ceph-mon[77147]: pgmap v8424: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:56:54.360 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:56:54 smithi115 conmon[90393]: 2022-05-14T03:56:54.028+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:56:54.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:56:54 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:56:54.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:56:54 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:56:55.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:56:55 smithi115 conmon[90393]: 2022-05-14T03:56:55.034+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:56:55.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:56:55 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:56:55.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:56:55 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:56:55.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:56:55 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:56:55.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:56:55 smithi115 ceph-mon[78593]: pgmap v8425: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2022-05-14T03:56:55.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:56:55 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:56:55.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:56:55 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:56:55.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:56:55 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:56:55.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:56:55 smithi047 ceph-mon[77147]: pgmap v8425: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2022-05-14T03:56:56.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:56:56 smithi115 conmon[90393]: 2022-05-14T03:56:56.076+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:56:56.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:56:56 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:56:56.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:56:56 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:56:57.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:56:57 smithi115 conmon[90393]: 2022-05-14T03:56:57.091+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:56:57.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:56:57 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:56:57.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:56:57 smithi115 ceph-mon[78593]: pgmap v8426: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2022-05-14T03:56:57.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:56:57 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 16150 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:56:57.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:56:57 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:56:57.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:56:57 smithi047 ceph-mon[77147]: pgmap v8426: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2022-05-14T03:56:57.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:56:57 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 16150 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:56:58.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:56:58 smithi115 conmon[90393]: 2022-05-14T03:56:58.121+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:56:58.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:56:58 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:56:58.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:56:58 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:56:59.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:56:59 smithi115 conmon[90393]: 2022-05-14T03:56:59.081+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:56:59.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:56:59 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:56:59.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:56:59 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:56:59.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:56:59 smithi115 ceph-mon[78593]: pgmap v8427: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:56:59.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:56:59 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:56:59.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:56:59 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:56:59.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:56:59 smithi047 ceph-mon[77147]: pgmap v8427: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:57:00.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:57:00 smithi115 conmon[90393]: 2022-05-14T03:57:00.097+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:57:00.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:57:00 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:57:00.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:57:00 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:57:00.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:57:00 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:57:00.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:57:00 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:57:01.360 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:57:01 smithi115 conmon[90393]: 2022-05-14T03:57:01.122+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:57:01.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:57:01 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:57:01.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:57:01 smithi115 ceph-mon[78593]: pgmap v8428: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:57:01.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:57:01 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:57:01.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:57:01 smithi047 ceph-mon[77147]: pgmap v8428: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:57:02.360 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:57:02 smithi115 conmon[90393]: 2022-05-14T03:57:02.147+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:57:02.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:57:02 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 16155 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:57:02.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:57:02 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 16155 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:57:03.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:57:03 smithi115 conmon[90393]: 2022-05-14T03:57:03.160+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:57:03.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:57:03 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:57:03.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:57:03 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:57:03.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:57:03 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:57:03.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:57:03 smithi115 ceph-mon[78593]: pgmap v8429: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:57:03.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:57:03 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:57:03.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:57:03 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:57:03.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:57:03 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:57:03.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:57:03 smithi047 ceph-mon[77147]: pgmap v8429: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:57:04.360 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:57:04 smithi115 conmon[90393]: 2022-05-14T03:57:04.158+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:57:04.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:57:04 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:57:04.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:57:04 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:57:05.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:57:05 smithi115 conmon[90393]: 2022-05-14T03:57:05.170+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:57:05.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:57:05 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:57:05.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:57:05 smithi115 ceph-mon[78593]: pgmap v8430: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2022-05-14T03:57:05.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:57:05 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:57:05.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:57:05 smithi047 ceph-mon[77147]: pgmap v8430: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2022-05-14T03:57:06.360 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:57:06 smithi115 conmon[90393]: 2022-05-14T03:57:06.164+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:57:06.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:57:06 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:57:06.545 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:57:06 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:57:07.360 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:57:07 smithi115 conmon[90393]: 2022-05-14T03:57:07.136+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:57:07.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:57:07 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:57:07.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:57:07 smithi115 ceph-mon[78593]: pgmap v8431: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2022-05-14T03:57:07.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:57:07 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 16160 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:57:07.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:57:07 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:57:07.545 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:57:07 smithi047 ceph-mon[77147]: pgmap v8431: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2022-05-14T03:57:07.545 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:57:07 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 16160 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:57:08.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:57:08 smithi115 conmon[90393]: 2022-05-14T03:57:08.173+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:57:08.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:57:08 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:57:08.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:57:08 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:57:09.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:57:09 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:57:09.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:57:09 smithi047 ceph-mon[77147]: pgmap v8432: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:57:09.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:57:09 smithi115 conmon[90393]: 2022-05-14T03:57:09.153+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:57:09.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:57:09 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:57:09.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:57:09 smithi115 ceph-mon[78593]: pgmap v8432: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:57:10.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:57:10 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:57:10.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:57:10 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:57:10.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:57:10 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:57:10.610 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:57:10 smithi115 conmon[90393]: 2022-05-14T03:57:10.165+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:57:10.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:57:10 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:57:10.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:57:10 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:57:10.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:57:10 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:57:11.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:57:11 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:57:11.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:57:11 smithi047 ceph-mon[77147]: pgmap v8433: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:57:11.610 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:57:11 smithi115 conmon[90393]: 2022-05-14T03:57:11.174+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:57:11.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:57:11 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:57:11.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:57:11 smithi115 ceph-mon[78593]: pgmap v8433: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:57:12.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:57:12 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:57:12.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:57:12 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:57:12.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:57:12 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 16165 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:57:12.610 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:57:12 smithi115 conmon[90393]: 2022-05-14T03:57:12.150+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:57:12.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:57:12 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:57:12.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:57:12 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:57:12.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:57:12 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 16165 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:57:13.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:57:13 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:57:13.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:57:13 smithi047 ceph-mon[77147]: pgmap v8434: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:57:13.610 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:57:13 smithi115 conmon[90393]: 2022-05-14T03:57:13.155+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:57:13.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:57:13 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:57:13.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:57:13 smithi115 ceph-mon[78593]: pgmap v8434: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:57:14.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:57:14 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:57:14.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:57:14 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:57:14.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:57:14 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:57:14.610 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:57:14 smithi115 conmon[90393]: 2022-05-14T03:57:14.183+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:57:14.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:57:14 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:57:14.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:57:14 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:57:14.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:57:14 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:57:15.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:57:15 smithi047 ceph-mon[77147]: pgmap v8435: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2022-05-14T03:57:15.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:57:15 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:57:15.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:57:15 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T03:57:15.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:57:15 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T03:57:15.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:57:15 smithi115 conmon[90393]: 2022-05-14T03:57:15.168+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:57:15.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:57:15 smithi115 ceph-mon[78593]: pgmap v8435: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2022-05-14T03:57:15.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:57:15 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:57:15.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:57:15 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T03:57:15.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:57:15 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T03:57:16.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:57:16 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:57:16.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:57:16 smithi115 conmon[90393]: 2022-05-14T03:57:16.138+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:57:16.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:57:16 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:57:17.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:57:17 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:57:17.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:57:17 smithi047 ceph-mon[77147]: pgmap v8436: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2022-05-14T03:57:17.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:57:17 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 16170 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:57:17.610 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:57:17 smithi115 conmon[90393]: 2022-05-14T03:57:17.172+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:57:17.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:57:17 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:57:17.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:57:17 smithi115 ceph-mon[78593]: pgmap v8436: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2022-05-14T03:57:17.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:57:17 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 16170 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:57:18.610 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:57:18 smithi115 conmon[90393]: 2022-05-14T03:57:18.132+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:57:19.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:57:18 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:57:19.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:57:18 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:57:19.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:57:18 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:57:19.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:57:18 smithi047 ceph-mon[77147]: pgmap v8437: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:57:19.360 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:57:19 smithi115 conmon[90393]: 2022-05-14T03:57:19.101+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:57:19.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:57:18 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:57:19.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:57:18 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:57:19.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:57:18 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:57:19.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:57:18 smithi115 ceph-mon[78593]: pgmap v8437: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:57:20.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:57:19 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:57:20.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:57:19 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:57:20.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:57:20 smithi115 conmon[90393]: 2022-05-14T03:57:20.078+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:57:20.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:57:19 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:57:20.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:57:19 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:57:21.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:57:20 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:57:21.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:57:20 smithi047 ceph-mon[77147]: pgmap v8438: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:57:21.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:57:21 smithi115 conmon[90393]: 2022-05-14T03:57:21.104+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:57:21.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:57:20 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:57:21.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:57:20 smithi115 ceph-mon[78593]: pgmap v8438: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:57:22.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:57:21 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:57:22.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:57:21 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 16175 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:57:22.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:57:22 smithi115 conmon[90393]: 2022-05-14T03:57:22.066+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:57:22.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:57:21 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:57:22.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:57:21 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 16175 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:57:23.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:57:22 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:57:23.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:57:22 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:57:23.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:57:22 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:57:23.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:57:22 smithi047 ceph-mon[77147]: pgmap v8439: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:57:23.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:57:23 smithi115 conmon[90393]: 2022-05-14T03:57:23.039+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:57:23.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:57:22 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:57:23.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:57:22 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:57:23.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:57:22 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:57:23.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:57:22 smithi115 ceph-mon[78593]: pgmap v8439: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:57:24.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:57:23 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:57:24.360 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:57:24 smithi115 conmon[90393]: 2022-05-14T03:57:24.044+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:57:24.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:57:23 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:57:25.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:57:25 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:57:25.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:57:25 smithi047 ceph-mon[77147]: pgmap v8440: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:57:25.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:57:25 smithi115 conmon[90393]: 2022-05-14T03:57:25.090+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:57:25.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:57:25 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:57:25.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:57:25 smithi115 ceph-mon[78593]: pgmap v8440: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:57:26.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:57:26 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:57:26.360 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:57:26 smithi115 conmon[90393]: 2022-05-14T03:57:26.062+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:57:26.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:57:26 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:57:27.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:57:27 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:57:27.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:57:27 smithi047 ceph-mon[77147]: pgmap v8441: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2022-05-14T03:57:27.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:57:27 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 16180 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:57:27.360 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:57:27 smithi115 conmon[90393]: 2022-05-14T03:57:27.051+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:57:27.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:57:27 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:57:27.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:57:27 smithi115 ceph-mon[78593]: pgmap v8441: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2022-05-14T03:57:27.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:57:27 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 16180 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:57:28.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:57:28 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:57:28.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:57:28 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:57:28.360 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:57:28 smithi115 conmon[90393]: 2022-05-14T03:57:28.073+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:57:29.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:57:29 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:57:29.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:57:29 smithi047 ceph-mon[77147]: pgmap v8442: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:57:29.360 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:57:29 smithi115 conmon[90393]: 2022-05-14T03:57:29.072+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:57:29.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:57:29 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:57:29.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:57:29 smithi115 ceph-mon[78593]: pgmap v8442: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:57:30.296 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:57:30 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:57:30.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:57:30 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:57:30.360 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:57:30 smithi115 conmon[90393]: 2022-05-14T03:57:30.058+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:57:31.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:57:31 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:57:31.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:57:31 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:57:31.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:57:31 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:57:31.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:57:31 smithi047 ceph-mon[77147]: pgmap v8443: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:57:31.360 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:57:31 smithi115 conmon[90393]: 2022-05-14T03:57:31.034+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:57:31.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:57:31 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:57:31.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:57:31 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:57:31.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:57:31 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:57:31.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:57:31 smithi115 ceph-mon[78593]: pgmap v8443: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:57:32.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:57:32 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:57:32.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:57:32 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 16185 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:57:32.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:57:32 smithi115 conmon[90393]: 2022-05-14T03:57:32.044+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:57:32.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:57:32 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:57:32.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:57:32 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 16185 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:57:33.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:57:33 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:57:33.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:57:33 smithi047 ceph-mon[77147]: pgmap v8444: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:57:33.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:57:33 smithi115 conmon[90393]: 2022-05-14T03:57:33.003+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:57:33.362 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:57:33 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:57:33.362 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:57:33 smithi115 ceph-mon[78593]: pgmap v8444: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:57:34.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:57:34 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:57:34.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:57:34 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:57:34.361 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:57:33 smithi115 conmon[90393]: 2022-05-14T03:57:33.977+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:57:35.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:57:35 smithi115 conmon[90393]: 2022-05-14T03:57:35.011+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:57:35.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:57:35 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:57:35.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:57:35 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:57:35.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:57:35 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:57:35.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:57:35 smithi115 ceph-mon[78593]: pgmap v8445: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:57:35.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:57:35 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:57:35.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:57:35 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:57:35.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:57:35 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:57:35.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:57:35 smithi047 ceph-mon[77147]: pgmap v8445: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:57:36.360 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:57:36 smithi115 conmon[90393]: 2022-05-14T03:57:36.032+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:57:36.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:57:36 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:57:36.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:57:36 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:57:37.360 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:57:36 smithi115 conmon[90393]: 2022-05-14T03:57:36.993+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:57:37.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:57:37 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:57:37.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:57:37 smithi115 ceph-mon[78593]: pgmap v8446: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:57:37.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:57:37 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 16190 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:57:37.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:57:37 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:57:37.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:57:37 smithi047 ceph-mon[77147]: pgmap v8446: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:57:37.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:57:37 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 16190 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:57:38.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:57:38 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:57:38.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:57:38 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T03:57:38.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:57:38 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T03:57:38.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:57:38 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T03:57:38.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:57:37 smithi115 conmon[90393]: 2022-05-14T03:57:37.993+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:57:38.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:57:38 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:57:38.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:57:38 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T03:57:38.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:57:38 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T03:57:38.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:57:38 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T03:57:38.793 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:57:38 smithi047 conmon[78521]: 2022-05-14T03:57:38.415+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:57:38.793 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:57:38 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T03:57:38.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:57:38 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:57:38.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:57:38 smithi047 conmon[78521]: yield 2022-05-14T03:57:38.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:57:38 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:57:38.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:57:38 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:57:38.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:57:38 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:57:38.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:57:38 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T03:57:38.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:57:38 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:57:38.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:57:38 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T03:57:38.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:57:38 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:57:38.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:57:38 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T03:57:38.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:57:38 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:57:38.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:57:38 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T03:57:38.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:57:38 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:57:38.797 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:57:38 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T03:57:38.797 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:57:38 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:57:38.797 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:57:38 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:57:38.797 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:57:38 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:57:39.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:57:38 smithi115 conmon[90393]: 2022-05-14T03:57:38.965+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:57:39.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:57:39 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:57:39.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:57:39 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:57:39.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:57:39 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:57:39.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:57:39 smithi115 ceph-mon[78593]: pgmap v8447: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:57:39.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:57:39 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:57:39.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:57:39 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:57:39.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:57:39 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:57:39.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:57:39 smithi047 ceph-mon[77147]: pgmap v8447: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:57:40.360 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:57:39 smithi115 conmon[90393]: 2022-05-14T03:57:39.987+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:57:40.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:57:40 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:57:40.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T03:57:40.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:57:40.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T03:57:40.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:57:40.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:57:40.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:57:40.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T03:57:40.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:57:40.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T03:57:40.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:57:40.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T03:57:40.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:57:40.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T03:57:40.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:57:40.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T03:57:40.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:57:40.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:57:40.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:57:40.365 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:57:40 smithi115 ceph-mon[78593]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T03:57:40.365 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:57:40 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:57:40.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:57:40 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:57:40.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T03:57:40.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:57:40.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T03:57:40.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:57:40.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:57:40.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:57:40.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T03:57:40.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:57:40.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T03:57:40.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:57:40.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T03:57:40.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:57:40.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T03:57:40.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:57:40.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T03:57:40.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:57:40.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:57:40.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:57:40.547 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:57:40 smithi047 ceph-mon[77147]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T03:57:40.548 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:57:40 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:57:41.360 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:57:40 smithi115 conmon[90393]: 2022-05-14T03:57:40.957+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:57:41.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:57:41 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:57:41.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:57:41 smithi115 ceph-mon[78593]: pgmap v8448: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:57:41.491 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:57:41 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:57:41.492 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:57:41 smithi047 ceph-mon[77147]: pgmap v8448: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:57:41.492 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:57:41 smithi047 conmon[78521]: 2022-05-14T03:57:41.487+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:57:41.492 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:57:41 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T03:57:41.492 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:57:41 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:57:41.793 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:57:41 smithi047 conmon[78521]: yield 2022-05-14T03:57:41.793 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:57:41 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:57:41.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:57:41 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:57:41.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:57:41 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:57:41.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:57:41 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T03:57:41.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:57:41 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:57:41.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:57:41 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T03:57:41.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:57:41 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:57:41.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:57:41 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T03:57:41.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:57:41 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:57:41.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:57:41 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T03:57:41.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:57:41 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:57:41.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:57:41 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T03:57:41.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:57:41 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:57:41.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:57:41 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:57:41.797 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:57:41 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:57:41.797 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:57:41 smithi047 conmon[78521]: 2022-05-14T03:57:41.488+0000 7f5344781700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T03:57:41.797 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:57:41 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T03:57:41.797 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:57:41 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:57:41.797 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:57:41 smithi047 conmon[78521]: yield 2022-05-14T03:57:41.798 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:57:41 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:57:41.798 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:57:41 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:57:41.798 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:57:41 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:57:41.798 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:57:41 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T03:57:41.799 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:57:41 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:57:41.799 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:57:41 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T03:57:41.799 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:57:41 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:57:41.799 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:57:41 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T03:57:41.799 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:57:41 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:57:41.800 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:57:41 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T03:57:41.800 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:57:41 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:57:41.800 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:57:41 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T03:57:41.800 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:57:41 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:57:41.800 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:57:41 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:57:41.801 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:57:41 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:57:41.801 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:57:41 smithi047 conmon[78521]: 2022-05-14T03:57:41.801 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:57:41 smithi047 conmon[78521]: During handling of the above exception, another exception occurred: 2022-05-14T03:57:41.801 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:57:41 smithi047 conmon[78521]: 2022-05-14T03:57:41.802 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:57:41 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T03:57:41.802 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:57:41 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T03:57:41.802 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:57:41 smithi047 conmon[78521]: return f(*arg) 2022-05-14T03:57:41.802 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:57:41 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T03:57:41.802 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:57:41 smithi047 conmon[78521]: self._write_client_files(client_files, host) 2022-05-14T03:57:41.803 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:57:41 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T03:57:41.803 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:57:41 smithi047 conmon[78521]: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T03:57:41.803 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:57:41 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T03:57:41.803 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:57:41 smithi047 conmon[78521]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T03:57:41.804 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:57:41 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T03:57:41.804 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:57:41 smithi047 conmon[78521]: return self.event_loop.get_result(coro) 2022-05-14T03:57:41.804 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:57:41 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T03:57:41.804 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:57:41 smithi047 conmon[78521]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T03:57:41.804 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:57:41 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T03:57:41.805 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:57:41 smithi047 conmon[78521]: return self.__get_result() 2022-05-14T03:57:41.805 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:57:41 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T03:57:41.805 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:57:41 smithi047 conmon[78521]: raise self._exception 2022-05-14T03:57:41.805 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:57:41 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T03:57:41.805 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:57:41 smithi047 conmon[78521]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T03:57:41.806 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:57:41 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T03:57:41.806 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:57:41 smithi047 conmon[78521]: conn = await self._remote_connection(host, addr) 2022-05-14T03:57:41.806 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:57:41 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T03:57:41.806 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:57:41 smithi047 conmon[78521]: raise 2022-05-14T03:57:41.807 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:57:41 smithi047 conmon[78521]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T03:57:41.807 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:57:41 smithi047 conmon[78521]: self.gen.throw(type, value, traceback) 2022-05-14T03:57:41.807 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:57:41 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T03:57:41.807 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:57:41 smithi047 conmon[78521]: raise OrchestratorError(msg) 2022-05-14T03:57:41.807 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:57:41 smithi047 conmon[78521]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:57:42.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:57:41 smithi115 conmon[90393]: 2022-05-14T03:57:41.967+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:57:42.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:57:42 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:57:42.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:57:42 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 16195 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:57:42.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:57:42 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:57:42.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:57:42 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 16195 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:57:43.360 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:57:43 smithi115 conmon[90393]: 2022-05-14T03:57:43.005+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:57:43.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:57:43 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:57:43.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T03:57:43.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:57:43.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T03:57:43.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:57:43.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:57:43.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:57:43.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T03:57:43.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:57:43.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T03:57:43.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:57:43.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T03:57:43.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:57:43.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T03:57:43.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:57:43.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T03:57:43.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:57:43.365 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:57:43.365 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:57:43.365 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:57:43 smithi115 ceph-mon[78593]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T03:57:43.366 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T03:57:43.366 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:57:43.366 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T03:57:43.366 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:57:43.366 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:57:43.367 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:57:43.367 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T03:57:43.367 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:57:43.367 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T03:57:43.368 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:57:43.368 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T03:57:43.368 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:57:43.368 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T03:57:43.368 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:57:43.369 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T03:57:43.369 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:57:43.369 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:57:43.369 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:57:43.370 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T03:57:43.370 INFO:journalctl@ceph.mon.c.smithi115.stdout: During handling of the above exception, another exception occurred: 2022-05-14T03:57:43.370 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T03:57:43.370 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T03:57:43.370 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T03:57:43.371 INFO:journalctl@ceph.mon.c.smithi115.stdout: return f(*arg) 2022-05-14T03:57:43.371 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T03:57:43.371 INFO:journalctl@ceph.mon.c.smithi115.stdout: self._write_client_files(client_files, host) 2022-05-14T03:57:43.371 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T03:57:43.372 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T03:57:43.372 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T03:57:43.372 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T03:57:43.372 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T03:57:43.372 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.event_loop.get_result(coro) 2022-05-14T03:57:43.373 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T03:57:43.373 INFO:journalctl@ceph.mon.c.smithi115.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T03:57:43.373 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T03:57:43.373 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.__get_result() 2022-05-14T03:57:43.373 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T03:57:43.374 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise self._exception 2022-05-14T03:57:43.374 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T03:57:43.374 INFO:journalctl@ceph.mon.c.smithi115.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T03:57:43.374 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T03:57:43.375 INFO:journalctl@ceph.mon.c.smithi115.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T03:57:43.375 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T03:57:43.375 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise 2022-05-14T03:57:43.375 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T03:57:43.375 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.gen.throw(type, value, traceback) 2022-05-14T03:57:43.376 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T03:57:43.376 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OrchestratorError(msg) 2022-05-14T03:57:43.376 INFO:journalctl@ceph.mon.c.smithi115.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:57:43.376 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:57:43 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:57:43.377 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:57:43 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:57:43.377 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:57:43 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:57:43.377 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:57:43 smithi115 ceph-mon[78593]: pgmap v8449: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:57:43.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:57:43 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:57:43.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T03:57:43.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:57:43.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T03:57:43.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:57:43.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:57:43.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:57:43.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T03:57:43.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:57:43.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T03:57:43.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:57:43.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T03:57:43.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:57:43.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T03:57:43.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:57:43.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T03:57:43.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:57:43.548 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:57:43.548 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:57:43.548 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:57:43 smithi047 ceph-mon[77147]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T03:57:43.548 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T03:57:43.549 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:57:43.549 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T03:57:43.549 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:57:43.549 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:57:43.549 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:57:43.550 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T03:57:43.550 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:57:43.550 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T03:57:43.550 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:57:43.550 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T03:57:43.551 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:57:43.551 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T03:57:43.551 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:57:43.551 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T03:57:43.552 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:57:43.552 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:57:43.552 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:57:43.552 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T03:57:43.552 INFO:journalctl@ceph.mon.b.smithi047.stdout: During handling of the above exception, another exception occurred: 2022-05-14T03:57:43.553 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T03:57:43.553 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T03:57:43.553 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T03:57:43.553 INFO:journalctl@ceph.mon.b.smithi047.stdout: return f(*arg) 2022-05-14T03:57:43.554 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T03:57:43.554 INFO:journalctl@ceph.mon.b.smithi047.stdout: self._write_client_files(client_files, host) 2022-05-14T03:57:43.554 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T03:57:43.554 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T03:57:43.554 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T03:57:43.555 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T03:57:43.555 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T03:57:43.555 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.event_loop.get_result(coro) 2022-05-14T03:57:43.555 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T03:57:43.555 INFO:journalctl@ceph.mon.b.smithi047.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T03:57:43.556 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T03:57:43.556 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.__get_result() 2022-05-14T03:57:43.556 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T03:57:43.556 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise self._exception 2022-05-14T03:57:43.556 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T03:57:43.557 INFO:journalctl@ceph.mon.b.smithi047.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T03:57:43.557 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T03:57:43.557 INFO:journalctl@ceph.mon.b.smithi047.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T03:57:43.557 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T03:57:43.558 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise 2022-05-14T03:57:43.558 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T03:57:43.558 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.gen.throw(type, value, traceback) 2022-05-14T03:57:43.558 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T03:57:43.559 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OrchestratorError(msg) 2022-05-14T03:57:43.559 INFO:journalctl@ceph.mon.b.smithi047.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:57:43.559 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:57:43 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:57:43.559 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:57:43 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:57:43.559 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:57:43 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:57:43.560 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:57:43 smithi047 ceph-mon[77147]: pgmap v8449: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:57:44.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:57:43 smithi115 conmon[90393]: 2022-05-14T03:57:43.986+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:57:44.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:57:44 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:57:44.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:57:44 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:57:45.360 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:57:44 smithi115 conmon[90393]: 2022-05-14T03:57:44.987+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:57:45.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:57:45 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:57:45.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:57:45 smithi115 ceph-mon[78593]: pgmap v8450: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:57:45.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:57:45 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:57:45.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:57:45 smithi047 ceph-mon[77147]: pgmap v8450: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:57:46.360 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:57:46 smithi115 conmon[90393]: 2022-05-14T03:57:46.030+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:57:46.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:57:46 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:57:46.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:57:46 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:57:47.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:57:46 smithi115 conmon[90393]: 2022-05-14T03:57:46.980+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:57:47.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:57:47 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:57:47.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:57:47 smithi115 ceph-mon[78593]: pgmap v8451: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:57:47.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:57:47 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 16200 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:57:47.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:57:47 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:57:47.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:57:47 smithi047 ceph-mon[77147]: pgmap v8451: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:57:47.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:57:47 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 16200 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:57:48.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:57:47 smithi115 conmon[90393]: 2022-05-14T03:57:47.932+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:57:48.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:57:48 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:57:48.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:57:48 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:57:49.360 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:57:48 smithi115 conmon[90393]: 2022-05-14T03:57:48.934+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:57:49.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:57:49 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:57:49.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:57:49 smithi115 ceph-mon[78593]: pgmap v8452: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:57:49.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:57:49 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:57:49.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:57:49 smithi047 ceph-mon[77147]: pgmap v8452: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:57:50.361 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:57:49 smithi115 conmon[90393]: 2022-05-14T03:57:49.886+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:57:50.485 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:57:50 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:57:50.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:57:50 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:57:51.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:57:50 smithi115 conmon[90393]: 2022-05-14T03:57:50.918+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:57:51.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:57:51 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:57:51.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:57:51 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:57:51.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:57:51 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:57:51.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:57:51 smithi115 ceph-mon[78593]: pgmap v8453: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:57:51.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:57:51 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:57:51.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:57:51 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:57:51.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:57:51 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:57:51.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:57:51 smithi047 ceph-mon[77147]: pgmap v8453: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:57:52.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:57:51 smithi115 conmon[90393]: 2022-05-14T03:57:51.959+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:57:52.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:57:52 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:57:52.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:57:52 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 16205 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:57:52.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:57:52 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:57:52.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:57:52 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 16205 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:57:53.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:57:52 smithi115 conmon[90393]: 2022-05-14T03:57:52.994+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:57:53.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:57:53 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:57:53.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:57:53 smithi115 ceph-mon[78593]: pgmap v8454: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:57:53.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:57:53 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:57:53.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:57:53 smithi047 ceph-mon[77147]: pgmap v8454: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:57:54.360 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:57:54 smithi115 conmon[90393]: 2022-05-14T03:57:54.040+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:57:54.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:57:54 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:57:54.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:57:54 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:57:54.362 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:57:54 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:57:54.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:57:54 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:57:54.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:57:54 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:57:54.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:57:54 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:57:55.360 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:57:55 smithi115 conmon[90393]: 2022-05-14T03:57:55.084+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:57:55.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:57:55 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:57:55.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:57:55 smithi115 ceph-mon[78593]: pgmap v8455: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:57:55.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:57:55 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:57:55.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:57:55 smithi047 ceph-mon[77147]: pgmap v8455: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:57:56.360 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:57:56 smithi115 conmon[90393]: 2022-05-14T03:57:56.104+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:57:56.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:57:56 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:57:56.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:57:56 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:57:56.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:57:56 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:57:56.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:57:56 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:57:57.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:57:57 smithi047 ceph-mon[77147]: pgmap v8456: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:57:57.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:57:57 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 16210 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:57:57.610 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:57:57 smithi115 conmon[90393]: 2022-05-14T03:57:57.133+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:57:57.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:57:57 smithi115 ceph-mon[78593]: pgmap v8456: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:57:57.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:57:57 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 16210 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:57:58.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:57:58 smithi115 conmon[90393]: 2022-05-14T03:57:58.104+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:57:59.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:57:58 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:57:59.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:57:58 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:57:59.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:57:58 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:57:59.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:57:58 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:57:59.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:57:58 smithi047 ceph-mon[77147]: pgmap v8457: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:57:59.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:57:59 smithi115 conmon[90393]: 2022-05-14T03:57:59.138+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:57:59.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:57:58 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:57:59.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:57:58 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:57:59.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:57:58 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:57:59.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:57:58 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:57:59.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:57:58 smithi115 ceph-mon[78593]: pgmap v8457: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:58:00.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:58:00 smithi115 conmon[90393]: 2022-05-14T03:58:00.100+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:58:00.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:58:00 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:58:00.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:58:00 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:58:00.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:58:00 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:58:00.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:58:00 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:58:01.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:58:01 smithi115 conmon[90393]: 2022-05-14T03:58:01.087+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:58:01.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:58:01 smithi115 ceph-mon[78593]: pgmap v8458: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:58:01.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:58:01 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:58:01.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:58:01 smithi047 ceph-mon[77147]: pgmap v8458: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:58:01.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:58:01 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:58:02.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:58:02 smithi115 conmon[90393]: 2022-05-14T03:58:02.091+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:58:02.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:58:02 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:58:02.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:58:02 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:58:02.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:58:02 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 16215 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:58:02.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:58:02 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:58:02.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:58:02 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:58:02.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:58:02 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:58:02.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:58:02 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 16215 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:58:02.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:58:02 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:58:03.360 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:58:03 smithi115 conmon[90393]: 2022-05-14T03:58:03.082+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:58:03.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:58:03 smithi115 ceph-mon[78593]: pgmap v8459: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:58:03.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:58:03 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:58:03.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:58:03 smithi047 ceph-mon[77147]: pgmap v8459: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:58:03.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:58:03 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:58:04.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:58:04 smithi115 conmon[90393]: 2022-05-14T03:58:04.045+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:58:04.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:58:04 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:58:04.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:58:04 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:58:05.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:58:05 smithi115 conmon[90393]: 2022-05-14T03:58:05.011+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:58:05.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:58:05 smithi115 ceph-mon[78593]: pgmap v8460: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:58:05.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:58:05 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:58:05.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:58:05 smithi047 ceph-mon[77147]: pgmap v8460: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:58:05.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:58:05 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:58:06.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:58:06 smithi115 conmon[90393]: 2022-05-14T03:58:06.009+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:58:06.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:58:06 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:58:06.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:58:06 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:58:07.360 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:58:07 smithi115 conmon[90393]: 2022-05-14T03:58:07.044+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:58:07.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:58:07 smithi115 ceph-mon[78593]: pgmap v8461: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:58:07.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:58:07 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 16220 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:58:07.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:58:07 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:58:07.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:58:07 smithi047 ceph-mon[77147]: pgmap v8461: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:58:07.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:58:07 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 16220 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:58:07.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:58:07 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:58:08.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:58:08 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:58:08.360 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:58:08 smithi115 conmon[90393]: 2022-05-14T03:58:08.064+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:58:08.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:58:08 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:58:09.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:58:09 smithi115 ceph-mon[78593]: pgmap v8462: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:58:09.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:58:09 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:58:09.360 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:58:09 smithi115 conmon[90393]: 2022-05-14T03:58:09.087+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:58:09.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:58:09 smithi047 ceph-mon[77147]: pgmap v8462: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:58:09.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:58:09 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:58:10.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:58:10 smithi115 conmon[90393]: 2022-05-14T03:58:10.040+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:58:10.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:58:10 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:58:10.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:58:10 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:58:10.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:58:10 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:58:10.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:58:10 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:58:10.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:58:10 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:58:10.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:58:10 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:58:11.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:58:11 smithi115 conmon[90393]: 2022-05-14T03:58:11.029+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:58:11.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:58:11 smithi115 ceph-mon[78593]: pgmap v8463: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:58:11.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:58:11 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:58:11.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:58:11 smithi047 ceph-mon[77147]: pgmap v8463: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:58:11.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:58:11 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:58:12.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:58:12 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 16226 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:58:12.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:58:12 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:58:12.361 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:58:12 smithi115 conmon[90393]: 2022-05-14T03:58:12.060+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:58:12.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:58:12 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 16226 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:58:12.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:58:12 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:58:13.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:58:13 smithi115 ceph-mon[78593]: pgmap v8464: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:58:13.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:58:13 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:58:13.360 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:58:13 smithi115 conmon[90393]: 2022-05-14T03:58:13.089+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:58:13.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:58:13 smithi047 ceph-mon[77147]: pgmap v8464: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:58:13.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:58:13 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:58:14.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:58:14 smithi115 conmon[90393]: 2022-05-14T03:58:14.133+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:58:15.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:58:14 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:58:15.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:58:14 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:58:15.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:58:14 smithi047 ceph-mon[77147]: pgmap v8465: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:58:15.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:58:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T03:58:15.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:58:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T03:58:15.360 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:58:15 smithi115 conmon[90393]: 2022-05-14T03:58:15.108+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:58:15.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:58:14 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:58:15.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:58:14 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:58:15.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:58:14 smithi115 ceph-mon[78593]: pgmap v8465: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:58:15.362 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:58:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T03:58:15.362 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:58:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T03:58:16.302 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:58:15 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:58:16.302 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:58:15 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:58:16.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:58:16 smithi115 conmon[90393]: 2022-05-14T03:58:16.058+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:58:16.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:58:15 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:58:16.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:58:15 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:58:17.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:58:17 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:58:17.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:58:17 smithi047 ceph-mon[77147]: pgmap v8466: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:58:17.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:58:17 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 16230 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:58:17.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:58:17 smithi115 conmon[90393]: 2022-05-14T03:58:17.043+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:58:17.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:58:17 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:58:17.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:58:17 smithi115 ceph-mon[78593]: pgmap v8466: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:58:17.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:58:17 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 16230 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:58:18.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:58:18 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:58:18.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:58:18 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:58:18.360 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:58:18 smithi115 conmon[90393]: 2022-05-14T03:58:18.054+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:58:19.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:58:19 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:58:19.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:58:19 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:58:19.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:58:19 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:58:19.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:58:19 smithi047 ceph-mon[77147]: pgmap v8467: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:58:19.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:58:19 smithi115 conmon[90393]: 2022-05-14T03:58:19.041+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:58:19.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:58:19 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:58:19.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:58:19 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:58:19.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:58:19 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:58:19.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:58:19 smithi115 ceph-mon[78593]: pgmap v8467: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:58:20.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:58:20 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:58:20.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:58:20 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:58:20.360 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:58:20 smithi115 conmon[90393]: 2022-05-14T03:58:20.015+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:58:21.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:58:21 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:58:21.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:58:21 smithi047 ceph-mon[77147]: pgmap v8468: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:58:21.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:58:20 smithi115 conmon[90393]: 2022-05-14T03:58:20.998+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:58:21.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:58:21 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:58:21.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:58:21 smithi115 ceph-mon[78593]: pgmap v8468: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:58:22.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:58:22 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:58:22.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:58:22 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 16235 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:58:22.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:58:22 smithi115 conmon[90393]: 2022-05-14T03:58:22.014+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:58:22.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:58:22 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:58:22.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:58:22 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 16235 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:58:23.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:58:23 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:58:23.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:58:23 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:58:23.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:58:23 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:58:23.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:58:23 smithi047 ceph-mon[77147]: pgmap v8469: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:58:23.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:58:23 smithi115 conmon[90393]: 2022-05-14T03:58:23.047+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:58:23.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:58:23 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:58:23.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:58:23 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:58:23.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:58:23 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:58:23.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:58:23 smithi115 ceph-mon[78593]: pgmap v8469: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:58:24.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:58:24 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:58:24.360 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:58:24 smithi115 conmon[90393]: 2022-05-14T03:58:24.019+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:58:24.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:58:24 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:58:25.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:58:25 smithi115 conmon[90393]: 2022-05-14T03:58:25.054+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:58:25.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:58:25 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:58:25.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:58:25 smithi115 ceph-mon[78593]: pgmap v8470: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:58:25.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:58:25 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:58:25.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:58:25 smithi047 ceph-mon[77147]: pgmap v8470: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:58:26.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:58:26 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:58:26.361 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:58:26 smithi115 conmon[90393]: 2022-05-14T03:58:26.069+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:58:26.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:58:26 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:58:27.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:58:27 smithi115 conmon[90393]: 2022-05-14T03:58:27.062+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:58:27.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:58:27 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:58:27.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:58:27 smithi115 ceph-mon[78593]: pgmap v8471: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:58:27.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:58:27 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 16240 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:58:27.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:58:27 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:58:27.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:58:27 smithi047 ceph-mon[77147]: pgmap v8471: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:58:27.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:58:27 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 16240 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:58:28.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:58:28 smithi115 conmon[90393]: 2022-05-14T03:58:28.108+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:58:28.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:58:28 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:58:28.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:58:28 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:58:29.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:58:29 smithi115 conmon[90393]: 2022-05-14T03:58:29.117+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:58:29.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:58:29 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:58:29.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:58:29 smithi115 ceph-mon[78593]: pgmap v8472: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:58:29.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:58:29 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:58:29.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:58:29 smithi047 ceph-mon[77147]: pgmap v8472: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:58:30.360 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:58:30 smithi115 conmon[90393]: 2022-05-14T03:58:30.159+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:58:30.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:58:30 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:58:30.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:58:30 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:58:31.360 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:58:31 smithi115 conmon[90393]: 2022-05-14T03:58:31.122+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:58:31.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:58:31 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:58:31.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:58:31 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:58:31.362 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:58:31 smithi115 ceph-mon[78593]: pgmap v8473: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:58:31.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:58:31 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:58:31.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:58:31 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:58:31.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:58:31 smithi047 ceph-mon[77147]: pgmap v8473: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:58:32.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:58:32 smithi115 conmon[90393]: 2022-05-14T03:58:32.139+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:58:32.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:58:32 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:58:32.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:58:32 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:58:32.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:58:32 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 16245 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:58:32.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:58:32 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:58:32.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:58:32 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:58:32.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:58:32 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 16245 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:58:33.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:58:33 smithi115 conmon[90393]: 2022-05-14T03:58:33.093+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:58:33.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:58:33 smithi115 ceph-mon[78593]: pgmap v8474: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:58:33.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:58:33 smithi047 ceph-mon[77147]: pgmap v8474: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:58:34.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:58:34 smithi115 conmon[90393]: 2022-05-14T03:58:34.082+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:58:34.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:58:34 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:58:34.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:58:34 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:58:34.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:58:34 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:58:34.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:58:34 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:58:35.360 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:58:35 smithi115 conmon[90393]: 2022-05-14T03:58:35.106+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:58:35.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:58:35 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:58:35.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:58:35 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:58:35.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:58:35 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:58:35.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:58:35 smithi115 ceph-mon[78593]: pgmap v8475: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:58:35.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:58:35 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:58:35.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:58:35 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:58:35.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:58:35 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:58:35.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:58:35 smithi047 ceph-mon[77147]: pgmap v8475: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:58:36.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:58:36 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:58:36.360 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:58:36 smithi115 conmon[90393]: 2022-05-14T03:58:36.154+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:58:36.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:58:36 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:58:37.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:58:37 smithi115 conmon[90393]: 2022-05-14T03:58:37.200+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:58:37.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:58:37 smithi115 ceph-mon[78593]: pgmap v8476: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:58:37.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:58:37 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 16250 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:58:37.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:58:37 smithi047 ceph-mon[77147]: pgmap v8476: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:58:37.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:58:37 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 16250 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:58:38.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:58:38 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:58:38.360 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:58:38 smithi115 conmon[90393]: 2022-05-14T03:58:38.180+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:58:38.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:58:38 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:58:39.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:58:39 smithi115 conmon[90393]: 2022-05-14T03:58:39.201+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:58:39.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:58:39 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:58:39.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:58:39 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:58:39.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:58:39 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:58:39.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:58:39 smithi115 ceph-mon[78593]: pgmap v8477: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:58:39.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:58:39 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:58:39.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:58:39 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:58:39.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:58:39 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:58:39.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:58:39 smithi047 ceph-mon[77147]: pgmap v8477: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:58:40.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:58:40 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:58:40.360 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:58:40 smithi115 conmon[90393]: 2022-05-14T03:58:40.232+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:58:40.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:58:40 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:58:41.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:58:41 smithi115 conmon[90393]: 2022-05-14T03:58:41.249+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:58:41.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:58:41 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:58:41.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:58:41 smithi115 ceph-mon[78593]: pgmap v8478: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:58:41.536 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:58:41 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:58:41.536 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:58:41 smithi047 ceph-mon[77147]: pgmap v8478: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:58:42.293 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:58:42 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:58:42.293 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:58:42 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T03:58:42.294 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:58:42 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T03:58:42.294 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:58:42 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T03:58:42.294 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:58:42 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 16255 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:58:42.295 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:58:42 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:58:42.295 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:58:42 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T03:58:42.295 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:58:42 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T03:58:42.296 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:58:42 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T03:58:42.296 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:58:42 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 16255 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:58:42.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:58:42 smithi115 conmon[90393]: 2022-05-14T03:58:42.290+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:58:43.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:58:43 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:58:43.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:58:43 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:58:43.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:58:43 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:58:43.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:58:43 smithi047 ceph-mon[77147]: pgmap v8479: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:58:43.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:58:43 smithi115 conmon[90393]: 2022-05-14T03:58:43.334+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:58:43.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:58:43 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:58:43.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:58:43 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:58:43.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:58:43 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:58:43.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:58:43 smithi115 ceph-mon[78593]: pgmap v8479: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:58:44.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:58:44 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:58:44.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:58:44 smithi115 conmon[90393]: 2022-05-14T03:58:44.306+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:58:44.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:58:44 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:58:45.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:58:44 smithi047 conmon[78521]: 2022-05-14T03:58:44.592+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:58:45.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:58:44 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T03:58:45.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:58:44 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:58:45.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:58:44 smithi047 conmon[78521]: yield 2022-05-14T03:58:45.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:58:44 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:58:45.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:58:44 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:58:45.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:58:44 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:58:45.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:58:44 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T03:58:45.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:58:44 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:58:45.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:58:44 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T03:58:45.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:58:44 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:58:45.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:58:44 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T03:58:45.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:58:44 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:58:45.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:58:44 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T03:58:45.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:58:44 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:58:45.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:58:44 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T03:58:45.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:58:44 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:58:45.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:58:44 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:58:45.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:58:44 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:58:45.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:58:45 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:58:45.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:58:45 smithi047 ceph-mon[77147]: pgmap v8480: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:58:45.610 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:58:45 smithi115 conmon[90393]: 2022-05-14T03:58:45.328+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:58:45.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:58:45 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:58:45.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:58:45 smithi115 ceph-mon[78593]: pgmap v8480: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:58:46.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:58:46 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:58:46.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:58:46 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:58:46.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T03:58:46.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:58:46.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T03:58:46.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:58:46.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:58:46.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:58:46.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T03:58:46.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:58:46.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T03:58:46.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:58:46.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T03:58:46.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:58:46.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T03:58:46.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:58:46.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T03:58:46.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:58:46.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:58:46.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:58:46.548 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:58:46 smithi047 ceph-mon[77147]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T03:58:46.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:58:46 smithi115 conmon[90393]: 2022-05-14T03:58:46.286+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:58:46.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:58:46 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:58:46.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:58:46 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:58:46.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T03:58:46.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:58:46.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T03:58:46.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:58:46.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:58:46.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:58:46.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T03:58:46.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:58:46.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T03:58:46.613 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:58:46.613 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T03:58:46.613 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:58:46.613 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T03:58:46.614 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:58:46.614 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T03:58:46.614 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:58:46.614 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:58:46.614 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:58:46.615 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:58:46 smithi115 ceph-mon[78593]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T03:58:47.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:58:47 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:58:47.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:58:47 smithi047 ceph-mon[77147]: pgmap v8481: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:58:47.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:58:47 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 16260 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:58:47.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:58:47 smithi115 conmon[90393]: 2022-05-14T03:58:47.328+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:58:47.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:58:47 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:58:47.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:58:47 smithi115 ceph-mon[78593]: pgmap v8481: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:58:47.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:58:47 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 16260 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:58:48.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:58:47 smithi047 conmon[78521]: 2022-05-14T03:58:47.664+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:58:48.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:58:47 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T03:58:48.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:58:47 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:58:48.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:58:47 smithi047 conmon[78521]: yield 2022-05-14T03:58:48.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:58:47 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:58:48.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:58:47 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:58:48.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:58:47 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:58:48.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:58:47 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T03:58:48.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:58:47 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:58:48.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:58:47 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T03:58:48.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:58:47 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:58:48.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:58:47 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T03:58:48.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:58:47 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:58:48.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:58:47 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T03:58:48.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:58:47 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:58:48.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:58:47 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T03:58:48.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:58:47 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:58:48.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:58:47 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:58:48.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:58:47 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:58:48.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:58:47 smithi047 conmon[78521]: 2022-05-14T03:58:47.665+0000 7f5341f7c700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T03:58:48.048 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:58:47 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T03:58:48.048 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:58:47 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:58:48.048 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:58:47 smithi047 conmon[78521]: yield 2022-05-14T03:58:48.048 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:58:47 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:58:48.048 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:58:47 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:58:48.049 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:58:47 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:58:48.049 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:58:47 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T03:58:48.049 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:58:47 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:58:48.049 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:58:47 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T03:58:48.049 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:58:47 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:58:48.050 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:58:47 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T03:58:48.050 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:58:47 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:58:48.050 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:58:47 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T03:58:48.050 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:58:47 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:58:48.050 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:58:47 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T03:58:48.051 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:58:47 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:58:48.051 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:58:47 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:58:48.051 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:58:47 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:58:48.051 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:58:47 smithi047 conmon[78521]: 2022-05-14T03:58:48.052 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:58:47 smithi047 conmon[78521]: During handling of the above exception, another exception occurred: 2022-05-14T03:58:48.052 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:58:47 smithi047 conmon[78521]: 2022-05-14T03:58:48.052 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:58:47 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T03:58:48.053 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:58:47 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T03:58:48.053 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:58:47 smithi047 conmon[78521]: return f(*arg) 2022-05-14T03:58:48.053 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:58:47 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T03:58:48.053 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:58:47 smithi047 conmon[78521]: self._write_client_files(client_files, host) 2022-05-14T03:58:48.053 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:58:47 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T03:58:48.054 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:58:47 smithi047 conmon[78521]: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T03:58:48.054 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:58:47 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T03:58:48.054 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:58:47 smithi047 conmon[78521]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T03:58:48.054 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:58:47 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T03:58:48.054 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:58:47 smithi047 conmon[78521]: return self.event_loop.get_result(coro) 2022-05-14T03:58:48.055 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:58:47 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T03:58:48.055 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:58:47 smithi047 conmon[78521]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T03:58:48.055 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:58:47 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T03:58:48.055 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:58:47 smithi047 conmon[78521]: return self.__get_result() 2022-05-14T03:58:48.055 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:58:47 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T03:58:48.056 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:58:47 smithi047 conmon[78521]: raise self._exception 2022-05-14T03:58:48.056 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:58:47 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T03:58:48.056 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:58:47 smithi047 conmon[78521]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T03:58:48.056 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:58:47 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T03:58:48.057 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:58:47 smithi047 conmon[78521]: conn = await self._remote_connection(host, addr) 2022-05-14T03:58:48.057 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:58:47 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T03:58:48.057 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:58:47 smithi047 conmon[78521]: raise 2022-05-14T03:58:48.057 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:58:47 smithi047 conmon[78521]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T03:58:48.057 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:58:47 smithi047 conmon[78521]: self.gen.throw(type, value, traceback) 2022-05-14T03:58:48.058 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:58:47 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T03:58:48.058 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:58:47 smithi047 conmon[78521]: raise OrchestratorError(msg) 2022-05-14T03:58:48.058 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:58:47 smithi047 conmon[78521]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:58:48.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:58:48 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:58:48.610 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:58:48 smithi115 conmon[90393]: 2022-05-14T03:58:48.289+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:58:48.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:58:48 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:58:49.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:58:49 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:58:49.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:58:49 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:58:49.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T03:58:49.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:58:49.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T03:58:49.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:58:49.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:58:49.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:58:49.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T03:58:49.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:58:49.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T03:58:49.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:58:49.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T03:58:49.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:58:49.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T03:58:49.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:58:49.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T03:58:49.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:58:49.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:58:49.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:58:49.547 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:58:49 smithi047 ceph-mon[77147]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T03:58:49.548 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T03:58:49.548 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:58:49.548 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T03:58:49.549 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:58:49.549 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:58:49.549 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:58:49.549 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T03:58:49.550 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:58:49.550 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T03:58:49.550 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:58:49.550 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T03:58:49.551 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:58:49.551 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T03:58:49.552 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:58:49.552 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T03:58:49.552 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:58:49.553 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:58:49.553 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:58:49.553 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T03:58:49.553 INFO:journalctl@ceph.mon.b.smithi047.stdout: During handling of the above exception, another exception occurred: 2022-05-14T03:58:49.554 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T03:58:49.554 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T03:58:49.554 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T03:58:49.554 INFO:journalctl@ceph.mon.b.smithi047.stdout: return f(*arg) 2022-05-14T03:58:49.554 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T03:58:49.555 INFO:journalctl@ceph.mon.b.smithi047.stdout: self._write_client_files(client_files, host) 2022-05-14T03:58:49.555 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T03:58:49.555 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T03:58:49.555 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T03:58:49.555 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T03:58:49.556 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T03:58:49.556 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.event_loop.get_result(coro) 2022-05-14T03:58:49.556 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T03:58:49.556 INFO:journalctl@ceph.mon.b.smithi047.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T03:58:49.556 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T03:58:49.557 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.__get_result() 2022-05-14T03:58:49.557 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T03:58:49.557 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise self._exception 2022-05-14T03:58:49.557 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T03:58:49.557 INFO:journalctl@ceph.mon.b.smithi047.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T03:58:49.558 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T03:58:49.558 INFO:journalctl@ceph.mon.b.smithi047.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T03:58:49.558 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T03:58:49.558 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise 2022-05-14T03:58:49.558 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T03:58:49.559 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.gen.throw(type, value, traceback) 2022-05-14T03:58:49.559 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T03:58:49.559 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OrchestratorError(msg) 2022-05-14T03:58:49.559 INFO:journalctl@ceph.mon.b.smithi047.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:58:49.560 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:58:49 smithi047 ceph-mon[77147]: pgmap v8482: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:58:49.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:58:49 smithi115 conmon[90393]: 2022-05-14T03:58:49.246+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:58:49.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:58:49 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:58:49.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:58:49 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:58:49.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T03:58:49.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:58:49.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T03:58:49.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:58:49.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:58:49.613 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:58:49.613 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T03:58:49.613 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:58:49.613 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T03:58:49.613 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:58:49.614 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T03:58:49.614 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:58:49.614 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T03:58:49.614 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:58:49.615 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T03:58:49.615 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:58:49.615 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:58:49.615 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:58:49.615 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:58:49 smithi115 ceph-mon[78593]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T03:58:49.616 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T03:58:49.616 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:58:49.616 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T03:58:49.616 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:58:49.616 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:58:49.617 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:58:49.617 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T03:58:49.617 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:58:49.617 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T03:58:49.618 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:58:49.618 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T03:58:49.618 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:58:49.618 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T03:58:49.618 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:58:49.619 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T03:58:49.619 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:58:49.619 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:58:49.619 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:58:49.619 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T03:58:49.620 INFO:journalctl@ceph.mon.c.smithi115.stdout: During handling of the above exception, another exception occurred: 2022-05-14T03:58:49.620 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T03:58:49.620 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T03:58:49.620 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T03:58:49.621 INFO:journalctl@ceph.mon.c.smithi115.stdout: return f(*arg) 2022-05-14T03:58:49.621 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T03:58:49.621 INFO:journalctl@ceph.mon.c.smithi115.stdout: self._write_client_files(client_files, host) 2022-05-14T03:58:49.622 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T03:58:49.622 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T03:58:49.622 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T03:58:49.622 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T03:58:49.623 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T03:58:49.623 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.event_loop.get_result(coro) 2022-05-14T03:58:49.623 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T03:58:49.623 INFO:journalctl@ceph.mon.c.smithi115.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T03:58:49.624 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T03:58:49.624 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.__get_result() 2022-05-14T03:58:49.624 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T03:58:49.624 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise self._exception 2022-05-14T03:58:49.624 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T03:58:49.625 INFO:journalctl@ceph.mon.c.smithi115.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T03:58:49.625 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T03:58:49.625 INFO:journalctl@ceph.mon.c.smithi115.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T03:58:49.625 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T03:58:49.625 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise 2022-05-14T03:58:49.626 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T03:58:49.626 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.gen.throw(type, value, traceback) 2022-05-14T03:58:49.626 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T03:58:49.626 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OrchestratorError(msg) 2022-05-14T03:58:49.627 INFO:journalctl@ceph.mon.c.smithi115.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:58:49.627 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:58:49 smithi115 ceph-mon[78593]: pgmap v8482: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:58:50.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:58:50 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:58:50.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:58:50 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:58:50.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:58:50 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:58:50.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:58:50 smithi115 conmon[90393]: 2022-05-14T03:58:50.256+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:58:50.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:58:50 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:58:50.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:58:50 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:58:50.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:58:50 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:58:51.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:58:51 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:58:51.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:58:51 smithi047 ceph-mon[77147]: pgmap v8483: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:58:51.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:58:51 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:58:51.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:58:51 smithi115 conmon[90393]: 2022-05-14T03:58:51.286+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:58:51.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:58:51 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:58:51.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:58:51 smithi115 ceph-mon[78593]: pgmap v8483: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:58:51.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:58:51 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:58:52.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:58:52 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:58:52.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:58:52 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 16265 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:58:52.614 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:58:52 smithi115 conmon[90393]: 2022-05-14T03:58:52.280+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:58:52.615 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:58:52 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:58:52.615 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:58:52 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 16265 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:58:53.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:58:53 smithi047 ceph-mon[77147]: pgmap v8484: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:58:53.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:58:53 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:58:53.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:58:53 smithi115 conmon[90393]: 2022-05-14T03:58:53.254+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:58:53.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:58:53 smithi115 ceph-mon[78593]: pgmap v8484: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:58:53.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:58:53 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:58:54.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:58:54 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:58:54.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:58:54 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:58:54.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:58:54 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:58:54.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:58:54 smithi115 conmon[90393]: 2022-05-14T03:58:54.261+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:58:54.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:58:54 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:58:54.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:58:54 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:58:54.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:58:54 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:58:55.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:58:55 smithi047 ceph-mon[77147]: pgmap v8485: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:58:55.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:58:55 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:58:55.612 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:58:55 smithi115 conmon[90393]: 2022-05-14T03:58:55.277+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:58:55.613 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:58:55 smithi115 ceph-mon[78593]: pgmap v8485: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:58:55.613 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:58:55 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:58:56.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:58:56 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:58:56.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:58:56 smithi115 conmon[90393]: 2022-05-14T03:58:56.228+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:58:56.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:58:56 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:58:57.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:58:57 smithi047 ceph-mon[77147]: pgmap v8486: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:58:57.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:58:57 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:58:57.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:58:57 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 16270 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:58:57.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:58:57 smithi115 conmon[90393]: 2022-05-14T03:58:57.181+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:58:57.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:58:57 smithi115 ceph-mon[78593]: pgmap v8486: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:58:57.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:58:57 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:58:57.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:58:57 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 16270 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:58:58.610 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:58:58 smithi115 conmon[90393]: 2022-05-14T03:58:58.180+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:58:59.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:58:59 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:58:59.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:58:59 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:58:59.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:58:59 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:58:59.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:58:59 smithi047 ceph-mon[77147]: pgmap v8487: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:58:59.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:58:59 smithi115 conmon[90393]: 2022-05-14T03:58:59.226+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:58:59.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:58:59 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:58:59.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:58:59 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:58:59.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:58:59 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:58:59.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:58:59 smithi115 ceph-mon[78593]: pgmap v8487: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:59:00.277 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:59:00 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:59:00.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:59:00 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:59:00.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:59:00 smithi115 conmon[90393]: 2022-05-14T03:59:00.274+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:59:01.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:59:01 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:59:01.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:59:01 smithi047 ceph-mon[77147]: pgmap v8488: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:59:01.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:59:01 smithi115 conmon[90393]: 2022-05-14T03:59:01.256+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:59:01.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:59:01 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:59:01.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:59:01 smithi115 ceph-mon[78593]: pgmap v8488: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:59:02.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:59:02 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:59:02.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:59:02 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 16275 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:59:02.302 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:59:02 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:59:02.303 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:59:02 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 16275 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:59:02.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:59:02 smithi115 conmon[90393]: 2022-05-14T03:59:02.299+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:59:03.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:59:03 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:59:03.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:59:03 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:59:03.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:59:03 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:59:03.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:59:03 smithi047 ceph-mon[77147]: pgmap v8489: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:59:03.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:59:03 smithi115 conmon[90393]: 2022-05-14T03:59:03.259+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:59:03.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:59:03 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:59:03.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:59:03 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:59:03.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:59:03 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:59:03.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:59:03 smithi115 ceph-mon[78593]: pgmap v8489: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:59:04.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:59:04 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:59:04.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:59:04 smithi115 conmon[90393]: 2022-05-14T03:59:04.237+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:59:04.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:59:04 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:59:05.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:59:05 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:59:05.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:59:05 smithi047 ceph-mon[77147]: pgmap v8490: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:59:05.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:59:05 smithi115 conmon[90393]: 2022-05-14T03:59:05.203+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:59:05.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:59:05 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:59:05.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:59:05 smithi115 ceph-mon[78593]: pgmap v8490: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:59:06.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:59:06 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:59:06.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:59:06 smithi115 conmon[90393]: 2022-05-14T03:59:06.214+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:59:06.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:59:06 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:59:07.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:59:07 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:59:07.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:59:07 smithi047 ceph-mon[77147]: pgmap v8491: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:59:07.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:59:07 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 16280 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:59:07.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:59:07 smithi115 conmon[90393]: 2022-05-14T03:59:07.256+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:59:07.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:59:07 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:59:07.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:59:07 smithi115 ceph-mon[78593]: pgmap v8491: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:59:07.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:59:07 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 16280 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:59:08.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:59:08 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:59:08.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:59:08 smithi115 conmon[90393]: 2022-05-14T03:59:08.246+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:59:08.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:59:08 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:59:09.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:59:09 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:59:09.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:59:09 smithi047 ceph-mon[77147]: pgmap v8492: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:59:09.360 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:59:09 smithi115 conmon[90393]: 2022-05-14T03:59:09.270+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:59:09.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:59:09 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:59:09.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:59:09 smithi115 ceph-mon[78593]: pgmap v8492: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:59:10.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:59:10 smithi115 conmon[90393]: 2022-05-14T03:59:10.256+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:59:10.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:59:10 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:59:10.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:59:10 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:59:11.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:59:11 smithi115 conmon[90393]: 2022-05-14T03:59:11.210+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:59:11.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:59:11 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:59:11.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:59:11 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:59:11.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:59:11 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:59:11.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:59:11 smithi115 ceph-mon[78593]: pgmap v8493: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:59:11.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:59:11 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:59:11.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:59:11 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:59:11.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:59:11 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:59:11.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:59:11 smithi047 ceph-mon[77147]: pgmap v8493: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:59:12.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:59:12 smithi115 conmon[90393]: 2022-05-14T03:59:12.259+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:59:12.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:59:12 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:59:12.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:59:12 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 16285 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:59:12.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:59:12 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:59:12.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:59:12 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 16285 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:59:13.360 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:59:13 smithi115 conmon[90393]: 2022-05-14T03:59:13.227+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:59:13.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:59:13 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:59:13.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:59:13 smithi115 ceph-mon[78593]: pgmap v8494: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:59:13.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:59:13 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:59:13.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:59:13 smithi047 ceph-mon[77147]: pgmap v8494: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:59:14.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:59:14 smithi115 conmon[90393]: 2022-05-14T03:59:14.241+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:59:14.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:59:14 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:59:14.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:59:14 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:59:15.360 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:59:15 smithi115 conmon[90393]: 2022-05-14T03:59:15.223+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:59:15.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:59:15 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:59:15.362 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:59:15 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:59:15.362 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:59:15 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:59:15.362 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:59:15 smithi115 ceph-mon[78593]: pgmap v8495: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:59:15.363 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:59:15 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T03:59:15.363 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:59:15 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T03:59:15.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:59:15 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:59:15.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:59:15 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:59:15.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:59:15 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:59:15.545 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:59:15 smithi047 ceph-mon[77147]: pgmap v8495: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:59:15.545 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:59:15 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T03:59:15.546 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:59:15 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T03:59:16.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:59:16 smithi115 conmon[90393]: 2022-05-14T03:59:16.213+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:59:16.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:59:16 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:59:16.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:59:16 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:59:17.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:59:17 smithi115 conmon[90393]: 2022-05-14T03:59:17.167+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:59:17.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:59:17 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:59:17.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:59:17 smithi115 ceph-mon[78593]: pgmap v8496: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:59:17.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:59:17 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 16290 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:59:17.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:59:17 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:59:17.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:59:17 smithi047 ceph-mon[77147]: pgmap v8496: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:59:17.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:59:17 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 16290 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:59:18.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:59:18 smithi115 conmon[90393]: 2022-05-14T03:59:18.199+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:59:18.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:59:18 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:59:18.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:59:18 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:59:19.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:59:19 smithi115 conmon[90393]: 2022-05-14T03:59:19.227+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:59:19.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:59:19 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:59:19.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:59:19 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:59:19.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:59:19 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:59:19.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:59:19 smithi115 ceph-mon[78593]: pgmap v8497: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:59:19.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:59:19 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:59:19.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:59:19 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:59:19.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:59:19 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:59:19.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:59:19 smithi047 ceph-mon[77147]: pgmap v8497: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:59:20.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:59:20 smithi115 conmon[90393]: 2022-05-14T03:59:20.254+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:59:20.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:59:20 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:59:20.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:59:20 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:59:21.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:59:21 smithi115 conmon[90393]: 2022-05-14T03:59:21.244+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:59:21.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:59:21 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:59:21.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:59:21 smithi115 ceph-mon[78593]: pgmap v8498: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:59:21.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:59:21 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:59:21.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:59:21 smithi047 ceph-mon[77147]: pgmap v8498: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:59:22.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:59:22 smithi115 conmon[90393]: 2022-05-14T03:59:22.258+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:59:22.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:59:22 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:59:22.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:59:22 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 16295 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:59:22.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:59:22 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:59:22.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:59:22 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 16295 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:59:23.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:59:23 smithi115 conmon[90393]: 2022-05-14T03:59:23.212+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:59:23.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:59:23 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:59:23.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:59:23 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:59:23.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:59:23 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:59:23.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:59:23 smithi115 ceph-mon[78593]: pgmap v8499: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:59:23.545 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:59:23 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:59:23.545 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:59:23 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:59:23.546 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:59:23 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:59:23.546 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:59:23 smithi047 ceph-mon[77147]: pgmap v8499: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:59:24.360 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:59:24 smithi115 conmon[90393]: 2022-05-14T03:59:24.198+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:59:24.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:59:24 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:59:24.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:59:24 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:59:25.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:59:25 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:59:25.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:59:25 smithi047 ceph-mon[77147]: pgmap v8500: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:59:25.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:59:25 smithi115 conmon[90393]: 2022-05-14T03:59:25.201+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:59:25.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:59:25 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:59:25.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:59:25 smithi115 ceph-mon[78593]: pgmap v8500: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:59:26.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:59:26 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:59:26.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:59:26 smithi115 conmon[90393]: 2022-05-14T03:59:26.154+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:59:26.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:59:26 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:59:27.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:59:27 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:59:27.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:59:27 smithi047 ceph-mon[77147]: pgmap v8501: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:59:27.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:59:27 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 16300 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:59:27.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:59:27 smithi115 conmon[90393]: 2022-05-14T03:59:27.128+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:59:27.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:59:27 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:59:27.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:59:27 smithi115 ceph-mon[78593]: pgmap v8501: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:59:27.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:59:27 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 16300 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:59:28.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:59:28 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:59:28.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:59:28 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:59:28.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:59:28 smithi115 conmon[90393]: 2022-05-14T03:59:28.151+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:59:28.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:59:28 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:59:28.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:59:28 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:59:29.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:59:29 smithi047 ceph-mon[77147]: pgmap v8502: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:59:29.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:59:29 smithi115 ceph-mon[78593]: pgmap v8502: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:59:29.610 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:59:29 smithi115 conmon[90393]: 2022-05-14T03:59:29.154+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:59:30.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:59:30 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:59:30.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:59:30 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:59:30.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:59:30 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:59:30.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:59:30 smithi115 conmon[90393]: 2022-05-14T03:59:30.111+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:59:30.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:59:30 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:59:30.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:59:30 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:59:30.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:59:30 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:59:31.360 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:59:31 smithi115 conmon[90393]: 2022-05-14T03:59:31.097+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:59:31.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:59:31 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:59:31.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:59:31 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:59:31.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:59:31 smithi115 ceph-mon[78593]: pgmap v8503: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:59:31.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:59:31 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:59:31.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:59:31 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:59:31.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:59:31 smithi047 ceph-mon[77147]: pgmap v8503: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:59:32.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:59:32 smithi115 conmon[90393]: 2022-05-14T03:59:32.070+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:59:32.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:59:32 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:59:32.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:59:32 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 16305 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:59:32.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:59:32 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:59:32.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:59:32 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 16305 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:59:33.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:59:33 smithi115 conmon[90393]: 2022-05-14T03:59:33.074+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:59:33.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:59:33 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:59:33.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:59:33 smithi115 ceph-mon[78593]: pgmap v8504: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:59:33.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:59:33 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:59:33.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:59:33 smithi047 ceph-mon[77147]: pgmap v8504: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:59:34.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:59:34 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:59:34.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:59:34 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:59:34.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:59:34 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:59:34.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:59:34 smithi115 conmon[90393]: 2022-05-14T03:59:34.117+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:59:34.612 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:59:34 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:59:34.613 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:59:34 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:59:34.613 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:59:34 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:59:35.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:59:35 smithi115 conmon[90393]: 2022-05-14T03:59:35.102+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:59:35.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:59:35 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:59:35.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:59:35 smithi115 ceph-mon[78593]: pgmap v8505: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:59:35.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:59:35 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:59:35.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:59:35 smithi047 ceph-mon[77147]: pgmap v8505: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:59:36.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:59:36 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:59:36.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:59:36 smithi115 conmon[90393]: 2022-05-14T03:59:36.123+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:59:36.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:59:36 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:59:37.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:59:37 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:59:37.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:59:37 smithi047 ceph-mon[77147]: pgmap v8506: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:59:37.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:59:37 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 16309 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:59:37.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:59:37 smithi115 conmon[90393]: 2022-05-14T03:59:37.162+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:59:37.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:59:37 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:59:37.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:59:37 smithi115 ceph-mon[78593]: pgmap v8506: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:59:37.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:59:37 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 16309 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:59:38.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:59:38 smithi115 conmon[90393]: 2022-05-14T03:59:38.165+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:59:39.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:59:39 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:59:39.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:59:39 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:59:39.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:59:39 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:59:39.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:59:39 smithi047 ceph-mon[77147]: pgmap v8507: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:59:39.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:59:39 smithi115 conmon[90393]: 2022-05-14T03:59:39.165+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:59:39.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:59:39 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:59:39.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:59:39 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:59:39.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:59:39 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:59:39.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:59:39 smithi115 ceph-mon[78593]: pgmap v8507: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:59:40.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:59:40 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:59:40.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:59:40 smithi115 conmon[90393]: 2022-05-14T03:59:40.188+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:59:40.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:59:40 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:59:41.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:59:41 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:59:41.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:59:41 smithi047 ceph-mon[77147]: pgmap v8508: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:59:41.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:59:41 smithi115 conmon[90393]: 2022-05-14T03:59:41.204+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:59:41.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:59:41 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:59:41.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:59:41 smithi115 ceph-mon[78593]: pgmap v8508: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:59:42.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:59:42 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:59:42.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:59:42 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 16315 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:59:42.360 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:59:42 smithi115 conmon[90393]: 2022-05-14T03:59:42.192+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:59:42.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:59:42 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:59:42.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:59:42 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 16315 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:59:43.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:59:43 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:59:43.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:59:43 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:59:43.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:59:43 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:59:43.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:59:43 smithi047 ceph-mon[77147]: pgmap v8509: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:59:43.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:59:43 smithi115 conmon[90393]: 2022-05-14T03:59:43.160+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:59:43.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:59:43 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:59:43.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:59:43 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:59:43.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:59:43 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:59:43.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:59:43 smithi115 ceph-mon[78593]: pgmap v8509: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:59:44.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:59:44 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:59:44.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:59:44 smithi115 conmon[90393]: 2022-05-14T03:59:44.155+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:59:44.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:59:44 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:59:45.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:59:45 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:59:45.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:59:45 smithi047 ceph-mon[77147]: pgmap v8510: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:59:45.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:59:45 smithi115 conmon[90393]: 2022-05-14T03:59:45.141+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:59:45.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:59:45 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:59:45.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:59:45 smithi115 ceph-mon[78593]: pgmap v8510: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:59:46.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:59:46 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:59:46.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:59:46 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:59:46.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:59:46 smithi115 conmon[90393]: 2022-05-14T03:59:46.113+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:59:46.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:59:46 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:59:46.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:59:46 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:59:47.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:59:47 smithi115 conmon[90393]: 2022-05-14T03:59:47.066+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:59:47.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:59:47 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:59:47.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:59:47 smithi115 ceph-mon[78593]: pgmap v8511: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:59:47.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:59:47 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 16320 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:59:47.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:59:47 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:59:47.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:59:47 smithi047 ceph-mon[77147]: pgmap v8511: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:59:47.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:59:47 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 16320 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:59:48.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:59:48 smithi115 conmon[90393]: 2022-05-14T03:59:48.095+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:59:48.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:59:48 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:59:48.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:59:48 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T03:59:48.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:59:48 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T03:59:48.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:59:48 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T03:59:48.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:59:48 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:59:48.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:59:48 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T03:59:48.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:59:48 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T03:59:48.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:59:48 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T03:59:49.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:59:49 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:59:49.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:59:49 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' 2022-05-14T03:59:49.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:59:49 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' 2022-05-14T03:59:49.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:59:49 smithi047 ceph-mon[77147]: pgmap v8512: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:59:49.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:59:49 smithi115 conmon[90393]: 2022-05-14T03:59:49.139+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:59:49.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:59:49 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:59:49.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:59:49 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' 2022-05-14T03:59:49.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:59:49 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' 2022-05-14T03:59:49.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:59:49 smithi115 ceph-mon[78593]: pgmap v8512: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:59:50.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:59:50 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:59:50.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:59:50 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:59:50.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:59:50 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:59:50.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:59:50 smithi115 conmon[90393]: 2022-05-14T03:59:50.178+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:59:50.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:59:50 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:59:50.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:59:50 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:59:50.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:59:50 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:59:51.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:59:50 smithi047 conmon[78521]: 2022-05-14T03:59:50.767+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:59:51.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:59:50 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T03:59:51.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:59:50 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:59:51.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:59:50 smithi047 conmon[78521]: yield 2022-05-14T03:59:51.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:59:50 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:59:51.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:59:50 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:59:51.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:59:50 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:59:51.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:59:50 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T03:59:51.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:59:50 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:59:51.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:59:50 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T03:59:51.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:59:50 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:59:51.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:59:50 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T03:59:51.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:59:50 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:59:51.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:59:50 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T03:59:51.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:59:50 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:59:51.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:59:50 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T03:59:51.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:59:50 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:59:51.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:59:50 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:59:51.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:59:50 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:59:51.168 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:59:51 smithi115 ceph-mon[78593]: pgmap v8513: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:59:51.169 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:59:51 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' 2022-05-14T03:59:51.169 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:59:51 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' 2022-05-14T03:59:51.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:59:51 smithi047 ceph-mon[77147]: pgmap v8513: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:59:51.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:59:51 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' 2022-05-14T03:59:51.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:59:51 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' 2022-05-14T03:59:51.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:59:51 smithi115 conmon[90393]: 2022-05-14T03:59:51.165+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:59:52.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:59:52 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:59:52.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:59:52 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:59:52.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T03:59:52.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:59:52.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T03:59:52.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:59:52.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:59:52.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:59:52.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T03:59:52.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:59:52.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T03:59:52.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:59:52.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T03:59:52.548 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:59:52.548 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T03:59:52.548 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:59:52.548 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T03:59:52.549 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:59:52.549 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:59:52.549 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:59:52.549 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:59:52 smithi047 ceph-mon[77147]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T03:59:52.549 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:59:52 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:59:52.550 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:59:52 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-05-14T03:59:52.550 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:59:52 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-05-14T03:59:52.550 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:59:52 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-05-14T03:59:52.550 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:59:52 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-05-14T03:59:52.551 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:59:52 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-05-14T03:59:52.551 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:59:52 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-05-14T03:59:52.551 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:59:52 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-05-14T03:59:52.551 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:59:52 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-05-14T03:59:52.552 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:59:52 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 16325 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:59:52.615 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:59:52 smithi115 conmon[90393]: 2022-05-14T03:59:52.181+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:59:52.616 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:59:52 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:59:52.616 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:59:52 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:59:52.617 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T03:59:52.617 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:59:52.617 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T03:59:52.618 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:59:52.618 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:59:52.619 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:59:52.619 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T03:59:52.619 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:59:52.620 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T03:59:52.620 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:59:52.621 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T03:59:52.621 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:59:52.622 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T03:59:52.622 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:59:52.623 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T03:59:52.623 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:59:52.624 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:59:52.624 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:59:52.625 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:59:52 smithi115 ceph-mon[78593]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T03:59:52.625 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:59:52 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:59:52.625 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:59:52 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-05-14T03:59:52.625 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:59:52 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-05-14T03:59:52.626 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:59:52 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-05-14T03:59:52.626 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:59:52 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-05-14T03:59:52.626 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:59:52 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-05-14T03:59:52.626 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:59:52 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-05-14T03:59:52.626 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:59:52 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-05-14T03:59:52.627 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:59:52 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-05-14T03:59:52.627 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:59:52 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 16325 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:59:53.545 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:59:53 smithi047 ceph-mon[77147]: Adjusting osd_memory_target on smithi047 to 198.1M 2022-05-14T03:59:53.546 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:59:53 smithi047 ceph-mon[77147]: Unable to set osd_memory_target on smithi047 to 207755878: error parsing value: Value '207755878' is below minimum 939524096 2022-05-14T03:59:53.546 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:59:53 smithi047 ceph-mon[77147]: Adjusting osd_memory_target on smithi115 to 197.0M 2022-05-14T03:59:53.546 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:59:53 smithi047 ceph-mon[77147]: Unable to set osd_memory_target on smithi115 to 206629785: error parsing value: Value '206629785' is below minimum 939524096 2022-05-14T03:59:53.547 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:59:53 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:59:53.547 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:59:53 smithi047 ceph-mon[77147]: pgmap v8514: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:59:53.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:59:53 smithi115 conmon[90393]: 2022-05-14T03:59:53.180+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:59:53.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:59:53 smithi115 ceph-mon[78593]: Adjusting osd_memory_target on smithi047 to 198.1M 2022-05-14T03:59:53.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:59:53 smithi115 ceph-mon[78593]: Unable to set osd_memory_target on smithi047 to 207755878: error parsing value: Value '207755878' is below minimum 939524096 2022-05-14T03:59:53.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:59:53 smithi115 ceph-mon[78593]: Adjusting osd_memory_target on smithi115 to 197.0M 2022-05-14T03:59:53.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:59:53 smithi115 ceph-mon[78593]: Unable to set osd_memory_target on smithi115 to 206629785: error parsing value: Value '206629785' is below minimum 939524096 2022-05-14T03:59:53.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:59:53 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:59:53.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:59:53 smithi115 ceph-mon[78593]: pgmap v8514: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:59:54.156 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:59:53 smithi047 conmon[78521]: 2022-05-14T03:59:53.839+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:59:54.156 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:59:53 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T03:59:54.157 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:59:53 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:59:54.157 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:59:53 smithi047 conmon[78521]: yield 2022-05-14T03:59:54.157 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:59:53 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:59:54.157 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:59:53 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:59:54.158 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:59:53 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:59:54.158 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:59:53 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T03:59:54.158 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:59:53 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:59:54.158 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:59:53 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T03:59:54.158 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:59:53 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:59:54.159 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:59:53 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T03:59:54.159 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:59:53 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:59:54.159 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:59:53 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T03:59:54.159 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:59:53 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:59:54.160 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:59:53 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T03:59:54.160 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:59:53 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:59:54.160 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:59:53 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:59:54.160 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:59:53 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:59:54.161 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:59:53 smithi047 conmon[78521]: 2022-05-14T03:59:53.840+0000 7f5344f82700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T03:59:54.161 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:59:53 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T03:59:54.161 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:59:53 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:59:54.161 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:59:53 smithi047 conmon[78521]: yield 2022-05-14T03:59:54.162 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:59:53 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:59:54.162 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:59:53 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:59:54.162 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:59:53 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:59:54.163 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:59:53 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T03:59:54.163 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:59:53 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:59:54.163 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:59:53 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T03:59:54.163 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:59:53 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:59:54.164 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:59:53 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T03:59:54.164 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:59:53 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:59:54.164 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:59:53 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T03:59:54.164 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:59:53 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:59:54.164 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:59:53 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T03:59:54.165 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:59:53 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:59:54.165 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:59:53 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:59:54.165 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:59:53 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:59:54.165 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:59:53 smithi047 conmon[78521]: 2022-05-14T03:59:54.166 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:59:53 smithi047 conmon[78521]: During handling of the above exception, another exception occurred: 2022-05-14T03:59:54.166 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:59:53 smithi047 conmon[78521]: 2022-05-14T03:59:54.166 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:59:53 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T03:59:54.166 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:59:53 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T03:59:54.167 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:59:53 smithi047 conmon[78521]: return f(*arg) 2022-05-14T03:59:54.167 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:59:53 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T03:59:54.167 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:59:53 smithi047 conmon[78521]: self._write_client_files(client_files, host) 2022-05-14T03:59:54.167 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:59:53 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T03:59:54.167 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:59:53 smithi047 conmon[78521]: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T03:59:54.168 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:59:53 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T03:59:54.168 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:59:53 smithi047 conmon[78521]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T03:59:54.168 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:59:53 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T03:59:54.168 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:59:53 smithi047 conmon[78521]: return self.event_loop.get_result(coro) 2022-05-14T03:59:54.168 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:59:53 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T03:59:54.169 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:59:53 smithi047 conmon[78521]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T03:59:54.169 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:59:53 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T03:59:54.169 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:59:53 smithi047 conmon[78521]: return self.__get_result() 2022-05-14T03:59:54.169 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:59:53 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T03:59:54.170 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:59:53 smithi047 conmon[78521]: raise self._exception 2022-05-14T03:59:54.170 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:59:53 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T03:59:54.170 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:59:53 smithi047 conmon[78521]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T03:59:54.170 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:59:53 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T03:59:54.170 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:59:53 smithi047 conmon[78521]: conn = await self._remote_connection(host, addr) 2022-05-14T03:59:54.171 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:59:53 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T03:59:54.171 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:59:53 smithi047 conmon[78521]: raise 2022-05-14T03:59:54.171 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:59:53 smithi047 conmon[78521]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T03:59:54.171 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:59:53 smithi047 conmon[78521]: self.gen.throw(type, value, traceback) 2022-05-14T03:59:54.172 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:59:53 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T03:59:54.172 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:59:53 smithi047 conmon[78521]: raise OrchestratorError(msg) 2022-05-14T03:59:54.172 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 03:59:53 smithi047 conmon[78521]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:59:54.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:59:54 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:59:54.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:59:54 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:59:54.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:59:54 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:59:54.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:59:54 smithi115 conmon[90393]: 2022-05-14T03:59:54.209+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:59:54.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:59:54 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:59:54.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:59:54 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:59:54.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:59:54 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:59:55.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:59:55 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:59:55.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T03:59:55.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:59:55.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T03:59:55.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:59:55.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:59:55.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:59:55.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T03:59:55.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:59:55.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T03:59:55.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:59:55.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T03:59:55.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:59:55.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T03:59:55.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:59:55.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T03:59:55.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:59:55.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:59:55.548 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:59:55.548 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:59:55 smithi047 ceph-mon[77147]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T03:59:55.548 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T03:59:55.548 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:59:55.548 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T03:59:55.549 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:59:55.549 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:59:55.549 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:59:55.549 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T03:59:55.550 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:59:55.550 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T03:59:55.550 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:59:55.550 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T03:59:55.550 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:59:55.551 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T03:59:55.551 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:59:55.551 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T03:59:55.551 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:59:55.552 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:59:55.552 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:59:55.552 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T03:59:55.552 INFO:journalctl@ceph.mon.b.smithi047.stdout: During handling of the above exception, another exception occurred: 2022-05-14T03:59:55.552 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T03:59:55.553 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T03:59:55.553 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T03:59:55.553 INFO:journalctl@ceph.mon.b.smithi047.stdout: return f(*arg) 2022-05-14T03:59:55.553 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T03:59:55.553 INFO:journalctl@ceph.mon.b.smithi047.stdout: self._write_client_files(client_files, host) 2022-05-14T03:59:55.554 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T03:59:55.554 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T03:59:55.554 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T03:59:55.554 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T03:59:55.555 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T03:59:55.555 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.event_loop.get_result(coro) 2022-05-14T03:59:55.555 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T03:59:55.555 INFO:journalctl@ceph.mon.b.smithi047.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T03:59:55.555 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T03:59:55.556 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.__get_result() 2022-05-14T03:59:55.556 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T03:59:55.556 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise self._exception 2022-05-14T03:59:55.556 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T03:59:55.556 INFO:journalctl@ceph.mon.b.smithi047.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T03:59:55.557 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T03:59:55.557 INFO:journalctl@ceph.mon.b.smithi047.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T03:59:55.557 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T03:59:55.557 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise 2022-05-14T03:59:55.557 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T03:59:55.558 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.gen.throw(type, value, traceback) 2022-05-14T03:59:55.558 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T03:59:55.558 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OrchestratorError(msg) 2022-05-14T03:59:55.558 INFO:journalctl@ceph.mon.b.smithi047.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:59:55.559 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:59:55 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:59:55.559 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:59:55 smithi047 ceph-mon[77147]: pgmap v8515: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:59:55.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:59:55 smithi115 conmon[90393]: 2022-05-14T03:59:55.210+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:59:55.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:59:55 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:59:55.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T03:59:55.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:59:55.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T03:59:55.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:59:55.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:59:55.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:59:55.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T03:59:55.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:59:55.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T03:59:55.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:59:55.613 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T03:59:55.613 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:59:55.613 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T03:59:55.613 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:59:55.614 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T03:59:55.614 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:59:55.614 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:59:55.614 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:59:55.614 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:59:55 smithi115 ceph-mon[78593]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T03:59:55.615 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T03:59:55.615 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T03:59:55.615 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T03:59:55.615 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T03:59:55.615 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T03:59:55.616 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T03:59:55.616 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T03:59:55.616 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T03:59:55.616 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T03:59:55.617 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T03:59:55.617 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T03:59:55.617 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T03:59:55.617 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T03:59:55.617 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T03:59:55.618 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T03:59:55.618 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T03:59:55.618 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T03:59:55.618 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:59:55.618 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T03:59:55.619 INFO:journalctl@ceph.mon.c.smithi115.stdout: During handling of the above exception, another exception occurred: 2022-05-14T03:59:55.619 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T03:59:55.619 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T03:59:55.619 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T03:59:55.620 INFO:journalctl@ceph.mon.c.smithi115.stdout: return f(*arg) 2022-05-14T03:59:55.620 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T03:59:55.620 INFO:journalctl@ceph.mon.c.smithi115.stdout: self._write_client_files(client_files, host) 2022-05-14T03:59:55.620 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T03:59:55.620 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T03:59:55.621 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T03:59:55.621 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T03:59:55.621 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T03:59:55.621 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.event_loop.get_result(coro) 2022-05-14T03:59:55.622 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T03:59:55.622 INFO:journalctl@ceph.mon.c.smithi115.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T03:59:55.622 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T03:59:55.622 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.__get_result() 2022-05-14T03:59:55.623 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T03:59:55.623 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise self._exception 2022-05-14T03:59:55.623 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T03:59:55.623 INFO:journalctl@ceph.mon.c.smithi115.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T03:59:55.623 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T03:59:55.624 INFO:journalctl@ceph.mon.c.smithi115.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T03:59:55.624 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T03:59:55.624 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise 2022-05-14T03:59:55.624 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T03:59:55.624 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.gen.throw(type, value, traceback) 2022-05-14T03:59:55.625 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T03:59:55.625 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OrchestratorError(msg) 2022-05-14T03:59:55.625 INFO:journalctl@ceph.mon.c.smithi115.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T03:59:55.625 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:59:55 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:59:55.626 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:59:55 smithi115 ceph-mon[78593]: pgmap v8515: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T03:59:56.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:59:56 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:59:56.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:59:56 smithi115 conmon[90393]: 2022-05-14T03:59:56.214+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:59:56.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:59:56 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:59:57.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:59:57 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:59:57.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:59:57 smithi047 ceph-mon[77147]: pgmap v8516: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:59:57.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:59:57 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 16330 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:59:57.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:59:57 smithi115 conmon[90393]: 2022-05-14T03:59:57.237+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:59:57.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:59:57 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:59:57.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:59:57 smithi115 ceph-mon[78593]: pgmap v8516: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:59:57.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:59:57 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 16330 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T03:59:58.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:59:58 smithi115 conmon[90393]: 2022-05-14T03:59:58.287+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T03:59:59.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:59:59 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:59:59.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:59:59 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:59:59.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:59:59 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:59:59.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 03:59:59 smithi047 ceph-mon[77147]: pgmap v8517: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:59:59.340 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:59:59 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T03:59:59.340 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:59:59 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T03:59:59.341 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:59:59 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T03:59:59.341 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 03:59:59 smithi115 ceph-mon[78593]: pgmap v8517: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T03:59:59.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 03:59:59 smithi115 conmon[90393]: 2022-05-14T03:59:59.337+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:00:00.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:00:00 smithi047 conmon[77123]: 2022-05-14T03:59:59.998+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : Health detail: HEALTH_ERR 1 filesystem is degraded; 1 mds daemon damaged; 1 MDSs are read only; 1 MDSs report slow metadata IOs; 1/3 mons down, quorum b,c; Reduced data availability: 1 pg inactive; 9 slow ops, oldest one blocked for 16330 sec, osd.5 has slow ops 2022-05-14T04:00:00.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:00:00 smithi047 conmon[77123]: 2022-05-14T03:59:59.998+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : [WRN] FS_DEGRADED: 1 filesystem is degraded 2022-05-14T04:00:00.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:00:00 smithi047 conmon[77123]: 2022-05-14T03:59:59.998+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : fs cephfs is degraded 2022-05-14T04:00:00.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:00:00 smithi047 conmon[77123]: 2022-05-14T03:59:59.998+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : [ERR] MDS_DAMAGE: 1 mds daemon damaged 2022-05-14T04:00:00.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:00:00 smithi047 conmon[77123]: 2022-05-14T03:59:59.998+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : fs cephfs mds.2 is damaged 2022-05-14T04:00:00.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:00:00 smithi047 conmon[77123]: 2022-05-14T03:59:59.999+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : [WRN] MDS_READ_ONLY: 1 MDSs are read only 2022-05-14T04:00:00.295 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:00:00 smithi047 conmon[77123]: 2022-05-14T03:59:59.999+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : mds.l(mds.0): MDS in read-only mode 2022-05-14T04:00:00.295 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:00:00 smithi047 conmon[77123]: 2022-05-14T03:59:59.999+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : [WRN] MDS_SLOW_METADATA_IO: 1 MDSs report slow metadata IOs 2022-05-14T04:00:00.295 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:00:00 smithi047 conmon[77123]: 2022-05-14T03:59:59.999+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : mds.l(mds.0): 4 slow metadata IOs are blocked > 180 secs, oldest blocked for 16633 secs 2022-05-14T04:00:00.295 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:00:00 smithi047 conmon[77123]: 2022-05-14T03:59:59.999+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : [WRN] MON_DOWN: 1/3 mons down, quorum b,c 2022-05-14T04:00:00.296 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:00:00 smithi047 conmon[77123]: 2022-05-14T03:59:59.999+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : mon.a (rank 0) addr [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] is down (out of quorum) 2022-05-14T04:00:00.296 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:00:00 smithi047 conmon[77123]: 2022-05-14T03:59:59.999+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : [WRN] PG_AVAILABILITY: Reduced data availability: 1 pg inactive 2022-05-14T04:00:00.296 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:00:00 smithi047 conmon[77123]: 2022-05-14T03:59:59.999+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : pg 2.1 is stuck inactive for 4h, current state unknown, last acting [] 2022-05-14T04:00:00.296 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:00:00 smithi047 conmon[77123]: 2022-05-14T03:59:59.999+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : [WRN] SLOW_OPS: 9 slow ops, oldest one blocked for 16330 sec, osd.5 has slow ops 2022-05-14T04:00:00.297 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:00:00 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:00:00.297 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:00:00 smithi047 ceph-mon[77147]: Health detail: HEALTH_ERR 1 filesystem is degraded; 1 mds daemon damaged; 1 MDSs are read only; 1 MDSs report slow metadata IOs; 1/3 mons down, quorum b,c; Reduced data availability: 1 pg inactive; 9 slow ops, oldest one blocked for 16330 sec, osd.5 has slow ops 2022-05-14T04:00:00.297 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:00:00 smithi047 ceph-mon[77147]: [WRN] FS_DEGRADED: 1 filesystem is degraded 2022-05-14T04:00:00.298 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:00:00 smithi047 ceph-mon[77147]: fs cephfs is degraded 2022-05-14T04:00:00.298 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:00:00 smithi047 ceph-mon[77147]: [ERR] MDS_DAMAGE: 1 mds daemon damaged 2022-05-14T04:00:00.298 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:00:00 smithi047 ceph-mon[77147]: fs cephfs mds.2 is damaged 2022-05-14T04:00:00.298 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:00:00 smithi047 ceph-mon[77147]: [WRN] MDS_READ_ONLY: 1 MDSs are read only 2022-05-14T04:00:00.299 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:00:00 smithi047 ceph-mon[77147]: mds.l(mds.0): MDS in read-only mode 2022-05-14T04:00:00.299 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:00:00 smithi047 ceph-mon[77147]: [WRN] MDS_SLOW_METADATA_IO: 1 MDSs report slow metadata IOs 2022-05-14T04:00:00.299 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:00:00 smithi047 ceph-mon[77147]: mds.l(mds.0): 4 slow metadata IOs are blocked > 180 secs, oldest blocked for 16633 secs 2022-05-14T04:00:00.299 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:00:00 smithi047 ceph-mon[77147]: [WRN] MON_DOWN: 1/3 mons down, quorum b,c 2022-05-14T04:00:00.300 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:00:00 smithi047 ceph-mon[77147]: mon.a (rank 0) addr [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] is down (out of quorum) 2022-05-14T04:00:00.300 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:00:00 smithi047 ceph-mon[77147]: [WRN] PG_AVAILABILITY: Reduced data availability: 1 pg inactive 2022-05-14T04:00:00.300 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:00:00 smithi047 ceph-mon[77147]: pg 2.1 is stuck inactive for 4h, current state unknown, last acting [] 2022-05-14T04:00:00.300 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:00:00 smithi047 ceph-mon[77147]: [WRN] SLOW_OPS: 9 slow ops, oldest one blocked for 16330 sec, osd.5 has slow ops 2022-05-14T04:00:00.315 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:00:00 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:00:00.316 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:00:00 smithi115 ceph-mon[78593]: Health detail: HEALTH_ERR 1 filesystem is degraded; 1 mds daemon damaged; 1 MDSs are read only; 1 MDSs report slow metadata IOs; 1/3 mons down, quorum b,c; Reduced data availability: 1 pg inactive; 9 slow ops, oldest one blocked for 16330 sec, osd.5 has slow ops 2022-05-14T04:00:00.316 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:00:00 smithi115 ceph-mon[78593]: [WRN] FS_DEGRADED: 1 filesystem is degraded 2022-05-14T04:00:00.316 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:00:00 smithi115 ceph-mon[78593]: fs cephfs is degraded 2022-05-14T04:00:00.317 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:00:00 smithi115 ceph-mon[78593]: [ERR] MDS_DAMAGE: 1 mds daemon damaged 2022-05-14T04:00:00.317 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:00:00 smithi115 ceph-mon[78593]: fs cephfs mds.2 is damaged 2022-05-14T04:00:00.317 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:00:00 smithi115 ceph-mon[78593]: [WRN] MDS_READ_ONLY: 1 MDSs are read only 2022-05-14T04:00:00.317 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:00:00 smithi115 ceph-mon[78593]: mds.l(mds.0): MDS in read-only mode 2022-05-14T04:00:00.318 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:00:00 smithi115 ceph-mon[78593]: [WRN] MDS_SLOW_METADATA_IO: 1 MDSs report slow metadata IOs 2022-05-14T04:00:00.318 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:00:00 smithi115 ceph-mon[78593]: mds.l(mds.0): 4 slow metadata IOs are blocked > 180 secs, oldest blocked for 16633 secs 2022-05-14T04:00:00.319 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:00:00 smithi115 ceph-mon[78593]: [WRN] MON_DOWN: 1/3 mons down, quorum b,c 2022-05-14T04:00:00.319 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:00:00 smithi115 ceph-mon[78593]: mon.a (rank 0) addr [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] is down (out of quorum) 2022-05-14T04:00:00.319 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:00:00 smithi115 ceph-mon[78593]: [WRN] PG_AVAILABILITY: Reduced data availability: 1 pg inactive 2022-05-14T04:00:00.319 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:00:00 smithi115 ceph-mon[78593]: pg 2.1 is stuck inactive for 4h, current state unknown, last acting [] 2022-05-14T04:00:00.320 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:00:00 smithi115 ceph-mon[78593]: [WRN] SLOW_OPS: 9 slow ops, oldest one blocked for 16330 sec, osd.5 has slow ops 2022-05-14T04:00:00.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:00:00 smithi115 conmon[90393]: 2022-05-14T04:00:00.312+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:00:01.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:00:01 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:00:01.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:00:01 smithi047 ceph-mon[77147]: pgmap v8518: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:00:01.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:00:01 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:00:01.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:00:01 smithi115 ceph-mon[78593]: pgmap v8518: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:00:01.360 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:00:01 smithi115 conmon[90393]: 2022-05-14T04:00:01.269+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:00:02.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:00:02 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:00:02.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:00:02 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 16335 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:00:02.299 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:00:02 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:00:02.299 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:00:02 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 16335 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:00:02.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:00:02 smithi115 conmon[90393]: 2022-05-14T04:00:02.296+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:00:03.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:00:03 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:00:03.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:00:03 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:00:03.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:00:03 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:00:03.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:00:03 smithi047 ceph-mon[77147]: pgmap v8519: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:00:03.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:00:03 smithi115 conmon[90393]: 2022-05-14T04:00:03.258+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:00:03.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:00:03 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:00:03.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:00:03 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:00:03.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:00:03 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:00:03.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:00:03 smithi115 ceph-mon[78593]: pgmap v8519: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:00:04.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:00:04 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:00:04.304 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:00:04 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:00:04.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:00:04 smithi115 conmon[90393]: 2022-05-14T04:00:04.301+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:00:05.321 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:00:05 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:00:05.321 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:00:05 smithi115 ceph-mon[78593]: pgmap v8520: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:00:05.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:00:05 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:00:05.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:00:05 smithi047 ceph-mon[77147]: pgmap v8520: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:00:05.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:00:05 smithi115 conmon[90393]: 2022-05-14T04:00:05.317+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:00:06.315 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:00:06 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:00:06.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:00:06 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:00:06.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:00:06 smithi115 conmon[90393]: 2022-05-14T04:00:06.312+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:00:07.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:00:07 smithi115 conmon[90393]: 2022-05-14T04:00:07.267+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:00:07.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:00:07 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:00:07.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:00:07 smithi115 ceph-mon[78593]: pgmap v8521: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:00:07.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:00:07 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 16340 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:00:07.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:00:07 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:00:07.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:00:07 smithi047 ceph-mon[77147]: pgmap v8521: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:00:07.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:00:07 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 16340 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:00:08.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:00:08 smithi115 conmon[90393]: 2022-05-14T04:00:08.236+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:00:08.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:00:08 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:00:08.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:00:08 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:00:09.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:00:09 smithi115 conmon[90393]: 2022-05-14T04:00:09.280+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:00:09.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:00:09 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:00:09.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:00:09 smithi115 ceph-mon[78593]: pgmap v8522: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:00:09.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:00:09 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:00:09.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:00:09 smithi047 ceph-mon[77147]: pgmap v8522: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:00:10.322 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:00:10 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:00:10.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:00:10 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:00:10.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:00:10 smithi115 conmon[90393]: 2022-05-14T04:00:10.318+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:00:11.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:00:11 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:00:11.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:00:11 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:00:11.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:00:11 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:00:11.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:00:11 smithi115 ceph-mon[78593]: pgmap v8523: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:00:11.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:00:11 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:00:11.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:00:11 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:00:11.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:00:11 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:00:11.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:00:11 smithi047 ceph-mon[77147]: pgmap v8523: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:00:11.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:00:11 smithi115 conmon[90393]: 2022-05-14T04:00:11.359+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:00:12.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:00:12 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:00:12.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:00:12 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 16345 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:00:12.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:00:12 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:00:12.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:00:12 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 16345 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:00:12.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:00:12 smithi115 conmon[90393]: 2022-05-14T04:00:12.398+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:00:13.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:00:13 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:00:13.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:00:13 smithi115 ceph-mon[78593]: pgmap v8524: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:00:13.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:00:13 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:00:13.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:00:13 smithi047 ceph-mon[77147]: pgmap v8524: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:00:13.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:00:13 smithi115 conmon[90393]: 2022-05-14T04:00:13.360+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:00:14.365 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:00:14 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:00:14.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:00:14 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:00:14.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:00:14 smithi115 conmon[90393]: 2022-05-14T04:00:14.381+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:00:15.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:00:15 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:00:15.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:00:15 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:00:15.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:00:15 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:00:15.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:00:15 smithi115 ceph-mon[78593]: pgmap v8525: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:00:15.362 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:00:15 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T04:00:15.362 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:00:15 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T04:00:15.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:00:15 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:00:15.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:00:15 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:00:15.545 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:00:15 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:00:15.545 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:00:15 smithi047 ceph-mon[77147]: pgmap v8525: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:00:15.545 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:00:15 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T04:00:15.546 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:00:15 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T04:00:15.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:00:15 smithi115 conmon[90393]: 2022-05-14T04:00:15.381+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:00:16.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:00:16 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:00:16.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:00:16 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:00:16.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:00:16 smithi115 conmon[90393]: 2022-05-14T04:00:16.404+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:00:17.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:00:17 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:00:17.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:00:17 smithi115 ceph-mon[78593]: pgmap v8526: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:00:17.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:00:17 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 16350 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:00:17.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:00:17 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:00:17.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:00:17 smithi047 ceph-mon[77147]: pgmap v8526: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:00:17.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:00:17 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 16350 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:00:17.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:00:17 smithi115 conmon[90393]: 2022-05-14T04:00:17.410+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:00:18.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:00:18 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:00:18.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:00:18 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:00:18.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:00:18 smithi115 conmon[90393]: 2022-05-14T04:00:18.410+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:00:19.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:00:19 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:00:19.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:00:19 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:00:19.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:00:19 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:00:19.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:00:19 smithi115 ceph-mon[78593]: pgmap v8527: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:00:19.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:00:19 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:00:19.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:00:19 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:00:19.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:00:19 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:00:19.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:00:19 smithi047 ceph-mon[77147]: pgmap v8527: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:00:19.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:00:19 smithi115 conmon[90393]: 2022-05-14T04:00:19.391+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:00:20.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:00:20 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:00:20.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:00:20 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:00:20.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:00:20 smithi115 conmon[90393]: 2022-05-14T04:00:20.441+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:00:21.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:00:21 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:00:21.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:00:21 smithi115 ceph-mon[78593]: pgmap v8528: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:00:21.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:00:21 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:00:21.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:00:21 smithi047 ceph-mon[77147]: pgmap v8528: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:00:21.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:00:21 smithi115 conmon[90393]: 2022-05-14T04:00:21.397+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:00:22.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:00:22 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:00:22.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:00:22 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 16355 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:00:22.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:00:22 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:00:22.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:00:22 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 16355 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:00:22.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:00:22 smithi115 conmon[90393]: 2022-05-14T04:00:22.357+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:00:23.409 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:00:23 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:00:23.409 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:00:23 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:00:23.409 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:00:23 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:00:23.410 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:00:23 smithi115 ceph-mon[78593]: pgmap v8529: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:00:23.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:00:23 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:00:23.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:00:23 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:00:23.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:00:23 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:00:23.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:00:23 smithi047 ceph-mon[77147]: pgmap v8529: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:00:23.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:00:23 smithi115 conmon[90393]: 2022-05-14T04:00:23.406+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:00:24.389 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:00:24 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:00:24.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:00:24 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:00:24.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:00:24 smithi115 conmon[90393]: 2022-05-14T04:00:24.386+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:00:25.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:00:25 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:00:25.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:00:25 smithi047 ceph-mon[77147]: pgmap v8530: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:00:25.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:00:25 smithi115 conmon[90393]: 2022-05-14T04:00:25.370+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:00:25.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:00:25 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:00:25.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:00:25 smithi115 ceph-mon[78593]: pgmap v8530: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:00:26.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:00:26 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:00:26.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:00:26 smithi115 conmon[90393]: 2022-05-14T04:00:26.321+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:00:26.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:00:26 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:00:27.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:00:27 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:00:27.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:00:27 smithi047 ceph-mon[77147]: pgmap v8531: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:00:27.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:00:27 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 16360 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:00:27.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:00:27 smithi115 conmon[90393]: 2022-05-14T04:00:27.327+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:00:27.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:00:27 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:00:27.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:00:27 smithi115 ceph-mon[78593]: pgmap v8531: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:00:27.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:00:27 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 16360 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:00:28.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:00:28 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:00:28.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:00:28 smithi115 conmon[90393]: 2022-05-14T04:00:28.320+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:00:28.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:00:28 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:00:29.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:00:29 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:00:29.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:00:29 smithi047 ceph-mon[77147]: pgmap v8532: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:00:29.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:00:29 smithi115 conmon[90393]: 2022-05-14T04:00:29.329+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:00:29.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:00:29 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:00:29.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:00:29 smithi115 ceph-mon[78593]: pgmap v8532: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:00:30.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:00:30 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:00:30.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:00:30 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:00:30.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:00:30 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:00:30.610 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:00:30 smithi115 conmon[90393]: 2022-05-14T04:00:30.291+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:00:30.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:00:30 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:00:30.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:00:30 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:00:30.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:00:30 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:00:31.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:00:31 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:00:31.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:00:31 smithi047 ceph-mon[77147]: pgmap v8533: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:00:31.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:00:31 smithi115 conmon[90393]: 2022-05-14T04:00:31.280+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:00:31.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:00:31 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:00:31.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:00:31 smithi115 ceph-mon[78593]: pgmap v8533: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:00:32.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:00:32 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:00:32.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:00:32 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 16365 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:00:32.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:00:32 smithi115 conmon[90393]: 2022-05-14T04:00:32.284+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:00:32.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:00:32 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:00:32.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:00:32 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 16365 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:00:33.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:00:33 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:00:33.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:00:33 smithi047 ceph-mon[77147]: pgmap v8534: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:00:33.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:00:33 smithi115 conmon[90393]: 2022-05-14T04:00:33.247+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:00:33.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:00:33 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:00:33.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:00:33 smithi115 ceph-mon[78593]: pgmap v8534: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:00:34.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:00:34 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:00:34.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:00:34 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:00:34.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:00:34 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:00:34.610 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:00:34 smithi115 conmon[90393]: 2022-05-14T04:00:34.205+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:00:34.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:00:34 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:00:34.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:00:34 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:00:34.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:00:34 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:00:35.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:00:35 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:00:35.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:00:35 smithi047 ceph-mon[77147]: pgmap v8535: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:00:35.610 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:00:35 smithi115 conmon[90393]: 2022-05-14T04:00:35.247+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:00:35.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:00:35 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:00:35.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:00:35 smithi115 ceph-mon[78593]: pgmap v8535: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:00:36.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:00:36 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:00:36.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:00:36 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:00:36.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:00:36 smithi115 conmon[90393]: 2022-05-14T04:00:36.284+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:00:36.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:00:36 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:00:36.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:00:36 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:00:37.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:00:37 smithi047 ceph-mon[77147]: pgmap v8536: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:00:37.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:00:37 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:00:37.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:00:37 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 16370 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:00:37.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:00:37 smithi115 conmon[90393]: 2022-05-14T04:00:37.304+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:00:37.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:00:37 smithi115 ceph-mon[78593]: pgmap v8536: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:00:37.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:00:37 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:00:37.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:00:37 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 16370 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:00:38.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:00:38 smithi115 conmon[90393]: 2022-05-14T04:00:38.260+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:00:39.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:00:39 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:00:39.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:00:39 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:00:39.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:00:39 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:00:39.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:00:39 smithi047 ceph-mon[77147]: pgmap v8537: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:00:39.294 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:00:39 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:00:39.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:00:39 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:00:39.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:00:39 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:00:39.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:00:39 smithi115 ceph-mon[78593]: pgmap v8537: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:00:39.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:00:39 smithi115 conmon[90393]: 2022-05-14T04:00:39.291+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:00:40.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:00:40 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:00:40.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:00:40 smithi115 conmon[90393]: 2022-05-14T04:00:40.245+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:00:40.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:00:40 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:00:41.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:00:41 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:00:41.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:00:41 smithi047 ceph-mon[77147]: pgmap v8538: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:00:41.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:00:41 smithi115 conmon[90393]: 2022-05-14T04:00:41.280+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:00:41.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:00:41 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:00:41.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:00:41 smithi115 ceph-mon[78593]: pgmap v8538: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:00:42.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:00:42 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:00:42.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:00:42 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 16375 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:00:42.360 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:00:42 smithi115 conmon[90393]: 2022-05-14T04:00:42.278+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:00:42.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:00:42 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:00:42.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:00:42 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 16375 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:00:43.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:00:43 smithi115 conmon[90393]: 2022-05-14T04:00:43.231+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:00:43.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:00:43 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:00:43.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:00:43 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:00:43.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:00:43 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:00:43.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:00:43 smithi115 ceph-mon[78593]: pgmap v8539: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:00:43.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:00:43 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:00:43.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:00:43 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:00:43.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:00:43 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:00:43.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:00:43 smithi047 ceph-mon[77147]: pgmap v8539: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:00:44.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:00:44 smithi115 conmon[90393]: 2022-05-14T04:00:44.264+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:00:44.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:00:44 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:00:44.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:00:44 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:00:45.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:00:45 smithi115 conmon[90393]: 2022-05-14T04:00:45.243+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:00:45.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:00:45 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:00:45.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:00:45 smithi115 ceph-mon[78593]: pgmap v8540: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:00:45.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:00:45 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:00:45.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:00:45 smithi047 ceph-mon[77147]: pgmap v8540: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:00:46.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:00:46 smithi115 conmon[90393]: 2022-05-14T04:00:46.196+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:00:46.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:00:46 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:00:46.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:00:46 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:00:47.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:00:47 smithi115 conmon[90393]: 2022-05-14T04:00:47.197+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:00:47.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:00:47 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:00:47.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:00:47 smithi115 ceph-mon[78593]: pgmap v8541: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:00:47.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:00:47 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 16380 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:00:47.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:00:47 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:00:47.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:00:47 smithi047 ceph-mon[77147]: pgmap v8541: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:00:47.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:00:47 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 16380 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:00:48.360 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:00:48 smithi115 conmon[90393]: 2022-05-14T04:00:48.229+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:00:48.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:00:48 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:00:48.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:00:48 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:00:49.361 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:00:49 smithi115 conmon[90393]: 2022-05-14T04:00:49.199+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:00:49.362 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:00:49 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:00:49.363 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:00:49 smithi115 ceph-mon[78593]: pgmap v8542: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:00:49.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:00:49 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:00:49.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:00:49 smithi047 ceph-mon[77147]: pgmap v8542: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:00:50.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:00:50 smithi115 conmon[90393]: 2022-05-14T04:00:50.174+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:00:50.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:00:50 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:00:50.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:00:50 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:00:51.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:00:51 smithi115 conmon[90393]: 2022-05-14T04:00:51.133+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:00:51.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:00:51 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:00:51.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:00:51 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:00:51.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:00:51 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:00:51.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:00:51 smithi115 ceph-mon[78593]: pgmap v8543: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:00:51.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:00:51 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:00:51.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:00:51 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:00:51.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:00:51 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:00:51.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:00:51 smithi047 ceph-mon[77147]: pgmap v8543: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:00:52.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:00:52 smithi115 conmon[90393]: 2022-05-14T04:00:52.130+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:00:52.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:00:52 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:00:52.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:00:52 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:00:52.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:00:52 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 16385 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:00:52.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:00:52 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:00:52.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:00:52 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:00:52.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:00:52 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 16385 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:00:53.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:00:53 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:00:53.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:00:53 smithi115 ceph-mon[78593]: pgmap v8544: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:00:53.361 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:00:53 smithi115 conmon[90393]: 2022-05-14T04:00:53.164+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:00:53.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:00:53 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:00:53.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:00:53 smithi047 ceph-mon[77147]: pgmap v8544: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:00:54.149 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:00:54 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T04:00:54.150 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:00:54 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T04:00:54.150 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:00:54 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T04:00:54.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:00:54 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T04:00:54.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:00:54 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T04:00:54.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:00:54 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T04:00:54.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:00:54 smithi115 conmon[90393]: 2022-05-14T04:00:54.146+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:00:54.793 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:00:54 smithi047 conmon[78521]: 2022-05-14T04:00:54.447+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:00:54.793 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:00:54 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T04:00:54.793 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:00:54 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:00:54.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:00:54 smithi047 conmon[78521]: yield 2022-05-14T04:00:54.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:00:54 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:00:54.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:00:54 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:00:54.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:00:54 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:00:54.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:00:54 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T04:00:54.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:00:54 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:00:54.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:00:54 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T04:00:54.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:00:54 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:00:54.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:00:54 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T04:00:54.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:00:54 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:00:54.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:00:54 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T04:00:54.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:00:54 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:00:54.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:00:54 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T04:00:54.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:00:54 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:00:54.797 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:00:54 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:00:54.797 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:00:54 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:00:55.360 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:00:55 smithi115 conmon[90393]: 2022-05-14T04:00:55.147+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:00:55.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:00:55 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:00:55.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:00:55 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:00:55.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:00:55 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:00:55.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:00:55 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:00:55.362 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:00:55 smithi115 ceph-mon[78593]: pgmap v8545: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:00:55.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:00:55 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:00:55.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:00:55 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:00:55.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:00:55 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:00:55.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:00:55 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:00:55.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:00:55 smithi047 ceph-mon[77147]: pgmap v8545: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:00:56.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:00:56 smithi115 conmon[90393]: 2022-05-14T04:00:56.188+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:00:56.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:00:56 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:00:56.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T04:00:56.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:00:56.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T04:00:56.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:00:56.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:00:56.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:00:56.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T04:00:56.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:00:56.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T04:00:56.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:00:56.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T04:00:56.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:00:56.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T04:00:56.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:00:56.365 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T04:00:56.365 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:00:56.365 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:00:56.366 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:00:56.366 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:00:56 smithi115 ceph-mon[78593]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T04:00:56.366 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:00:56 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:00:56.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:00:56 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:00:56.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T04:00:56.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:00:56.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T04:00:56.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:00:56.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:00:56.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:00:56.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T04:00:56.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:00:56.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T04:00:56.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:00:56.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T04:00:56.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:00:56.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T04:00:56.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:00:56.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T04:00:56.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:00:56.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:00:56.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:00:56.548 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:00:56 smithi047 ceph-mon[77147]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T04:00:56.548 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:00:56 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:00:57.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:00:57 smithi115 conmon[90393]: 2022-05-14T04:00:57.189+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:00:57.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:00:57 smithi115 ceph-mon[78593]: pgmap v8546: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:00:57.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:00:57 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 16390 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:00:57.523 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:00:57 smithi047 ceph-mon[77147]: pgmap v8546: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:00:57.523 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:00:57 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 16390 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:00:57.793 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:00:57 smithi047 conmon[78521]: 2022-05-14T04:00:57.519+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:00:57.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:00:57 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T04:00:57.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:00:57 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:00:57.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:00:57 smithi047 conmon[78521]: yield 2022-05-14T04:00:57.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:00:57 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:00:57.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:00:57 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:00:57.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:00:57 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:00:57.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:00:57 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T04:00:57.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:00:57 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:00:57.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:00:57 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T04:00:57.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:00:57 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:00:57.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:00:57 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T04:00:57.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:00:57 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:00:57.797 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:00:57 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T04:00:57.797 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:00:57 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:00:57.797 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:00:57 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T04:00:57.797 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:00:57 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:00:57.798 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:00:57 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:00:57.798 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:00:57 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:00:57.798 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:00:57 smithi047 conmon[78521]: 2022-05-14T04:00:57.520+0000 7f533ef76700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T04:00:57.799 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:00:57 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T04:00:57.799 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:00:57 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:00:57.799 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:00:57 smithi047 conmon[78521]: yield 2022-05-14T04:00:57.800 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:00:57 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:00:57.800 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:00:57 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:00:57.800 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:00:57 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:00:57.800 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:00:57 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T04:00:57.801 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:00:57 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:00:57.801 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:00:57 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T04:00:57.801 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:00:57 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:00:57.801 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:00:57 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T04:00:57.802 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:00:57 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:00:57.802 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:00:57 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T04:00:57.802 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:00:57 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:00:57.802 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:00:57 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T04:00:57.803 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:00:57 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:00:57.803 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:00:57 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:00:57.803 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:00:57 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:00:57.803 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:00:57 smithi047 conmon[78521]: 2022-05-14T04:00:57.804 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:00:57 smithi047 conmon[78521]: During handling of the above exception, another exception occurred: 2022-05-14T04:00:57.804 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:00:57 smithi047 conmon[78521]: 2022-05-14T04:00:57.804 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:00:57 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T04:00:57.804 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:00:57 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T04:00:57.804 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:00:57 smithi047 conmon[78521]: return f(*arg) 2022-05-14T04:00:57.805 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:00:57 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T04:00:57.805 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:00:57 smithi047 conmon[78521]: self._write_client_files(client_files, host) 2022-05-14T04:00:57.805 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:00:57 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T04:00:57.805 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:00:57 smithi047 conmon[78521]: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T04:00:57.806 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:00:57 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T04:00:57.806 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:00:57 smithi047 conmon[78521]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T04:00:57.806 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:00:57 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T04:00:57.806 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:00:57 smithi047 conmon[78521]: return self.event_loop.get_result(coro) 2022-05-14T04:00:57.806 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:00:57 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T04:00:57.807 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:00:57 smithi047 conmon[78521]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T04:00:57.807 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:00:57 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T04:00:57.807 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:00:57 smithi047 conmon[78521]: return self.__get_result() 2022-05-14T04:00:57.807 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:00:57 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T04:00:57.808 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:00:57 smithi047 conmon[78521]: raise self._exception 2022-05-14T04:00:57.808 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:00:57 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T04:00:57.808 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:00:57 smithi047 conmon[78521]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T04:00:57.808 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:00:57 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T04:00:57.808 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:00:57 smithi047 conmon[78521]: conn = await self._remote_connection(host, addr) 2022-05-14T04:00:57.809 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:00:57 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T04:00:57.809 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:00:57 smithi047 conmon[78521]: raise 2022-05-14T04:00:57.809 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:00:57 smithi047 conmon[78521]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T04:00:57.809 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:00:57 smithi047 conmon[78521]: self.gen.throw(type, value, traceback) 2022-05-14T04:00:57.810 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:00:57 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T04:00:57.810 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:00:57 smithi047 conmon[78521]: raise OrchestratorError(msg) 2022-05-14T04:00:57.810 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:00:57 smithi047 conmon[78521]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:00:58.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:00:58 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:00:58.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:00:58 smithi115 conmon[90393]: 2022-05-14T04:00:58.203+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:00:58.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:00:58 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:00:59.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:00:59 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:00:59.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:00:59 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:00:59.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T04:00:59.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:00:59.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T04:00:59.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:00:59.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:00:59.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:00:59.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T04:00:59.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:00:59.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T04:00:59.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:00:59.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T04:00:59.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:00:59.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T04:00:59.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:00:59.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T04:00:59.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:00:59.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:00:59.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:00:59.548 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:00:59 smithi047 ceph-mon[77147]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T04:00:59.548 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T04:00:59.548 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:00:59.548 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T04:00:59.548 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:00:59.549 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:00:59.549 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:00:59.549 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T04:00:59.549 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:00:59.550 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T04:00:59.550 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:00:59.550 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T04:00:59.550 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:00:59.550 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T04:00:59.551 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:00:59.551 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T04:00:59.551 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:00:59.551 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:00:59.552 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:00:59.552 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T04:00:59.552 INFO:journalctl@ceph.mon.b.smithi047.stdout: During handling of the above exception, another exception occurred: 2022-05-14T04:00:59.553 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T04:00:59.553 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T04:00:59.553 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T04:00:59.553 INFO:journalctl@ceph.mon.b.smithi047.stdout: return f(*arg) 2022-05-14T04:00:59.554 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T04:00:59.554 INFO:journalctl@ceph.mon.b.smithi047.stdout: self._write_client_files(client_files, host) 2022-05-14T04:00:59.554 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T04:00:59.554 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T04:00:59.555 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T04:00:59.555 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T04:00:59.555 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T04:00:59.556 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.event_loop.get_result(coro) 2022-05-14T04:00:59.556 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T04:00:59.556 INFO:journalctl@ceph.mon.b.smithi047.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T04:00:59.557 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T04:00:59.557 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.__get_result() 2022-05-14T04:00:59.558 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T04:00:59.558 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise self._exception 2022-05-14T04:00:59.559 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T04:00:59.560 INFO:journalctl@ceph.mon.b.smithi047.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T04:00:59.560 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T04:00:59.560 INFO:journalctl@ceph.mon.b.smithi047.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T04:00:59.561 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T04:00:59.561 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise 2022-05-14T04:00:59.561 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T04:00:59.562 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.gen.throw(type, value, traceback) 2022-05-14T04:00:59.562 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T04:00:59.563 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OrchestratorError(msg) 2022-05-14T04:00:59.563 INFO:journalctl@ceph.mon.b.smithi047.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:00:59.564 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:00:59 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:00:59.564 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:00:59 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:00:59.564 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:00:59 smithi047 ceph-mon[77147]: pgmap v8547: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:00:59.610 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:00:59 smithi115 conmon[90393]: 2022-05-14T04:00:59.181+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:00:59.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:00:59 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:00:59.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:00:59 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:00:59.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T04:00:59.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:00:59.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T04:00:59.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:00:59.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:00:59.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:00:59.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T04:00:59.613 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:00:59.613 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T04:00:59.613 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:00:59.614 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T04:00:59.614 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:00:59.614 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T04:00:59.614 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:00:59.615 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T04:00:59.615 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:00:59.615 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:00:59.615 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:00:59.615 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:00:59 smithi115 ceph-mon[78593]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T04:00:59.616 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T04:00:59.616 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:00:59.616 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T04:00:59.616 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:00:59.616 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:00:59.617 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:00:59.617 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T04:00:59.617 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:00:59.618 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T04:00:59.618 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:00:59.618 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T04:00:59.618 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:00:59.619 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T04:00:59.619 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:00:59.619 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T04:00:59.619 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:00:59.619 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:00:59.620 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:00:59.620 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T04:00:59.620 INFO:journalctl@ceph.mon.c.smithi115.stdout: During handling of the above exception, another exception occurred: 2022-05-14T04:00:59.620 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T04:00:59.621 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T04:00:59.621 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T04:00:59.621 INFO:journalctl@ceph.mon.c.smithi115.stdout: return f(*arg) 2022-05-14T04:00:59.621 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T04:00:59.621 INFO:journalctl@ceph.mon.c.smithi115.stdout: self._write_client_files(client_files, host) 2022-05-14T04:00:59.622 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T04:00:59.622 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T04:00:59.622 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T04:00:59.622 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T04:00:59.623 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T04:00:59.623 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.event_loop.get_result(coro) 2022-05-14T04:00:59.623 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T04:00:59.623 INFO:journalctl@ceph.mon.c.smithi115.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T04:00:59.623 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T04:00:59.624 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.__get_result() 2022-05-14T04:00:59.624 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T04:00:59.624 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise self._exception 2022-05-14T04:00:59.624 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T04:00:59.625 INFO:journalctl@ceph.mon.c.smithi115.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T04:00:59.625 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T04:00:59.625 INFO:journalctl@ceph.mon.c.smithi115.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T04:00:59.625 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T04:00:59.625 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise 2022-05-14T04:00:59.626 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T04:00:59.626 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.gen.throw(type, value, traceback) 2022-05-14T04:00:59.626 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T04:00:59.626 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OrchestratorError(msg) 2022-05-14T04:00:59.627 INFO:journalctl@ceph.mon.c.smithi115.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:00:59.627 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:00:59 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:00:59.627 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:00:59 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:00:59.627 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:00:59 smithi115 ceph-mon[78593]: pgmap v8547: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:01:00.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:01:00 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:01:00.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:01:00 smithi115 conmon[90393]: 2022-05-14T04:01:00.152+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:01:00.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:01:00 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:01:01.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:01:01 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:01:01.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:01:01 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:01:01.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:01:01 smithi047 ceph-mon[77147]: pgmap v8548: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:01:01.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:01:01 smithi115 conmon[90393]: 2022-05-14T04:01:01.138+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:01:01.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:01:01 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:01:01.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:01:01 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:01:01.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:01:01 smithi115 ceph-mon[78593]: pgmap v8548: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:01:02.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:01:02 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:01:02.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:01:02 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:01:02.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:01:02 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:01:02.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:01:02 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 16395 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:01:02.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:01:02 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:01:02.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:01:02 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:01:02.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:01:02 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:01:02.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:01:02 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 16395 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:01:02.611 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:01:02 smithi115 conmon[90393]: 2022-05-14T04:01:02.148+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:01:03.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:01:03 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:01:03.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:01:03 smithi047 ceph-mon[77147]: pgmap v8549: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:01:03.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:01:03 smithi115 conmon[90393]: 2022-05-14T04:01:03.142+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:01:03.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:01:03 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:01:03.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:01:03 smithi115 ceph-mon[78593]: pgmap v8549: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:01:04.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:01:04 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:01:04.611 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:01:04 smithi115 conmon[90393]: 2022-05-14T04:01:04.117+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:01:04.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:01:04 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:01:05.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:01:05 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:01:05.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:01:05 smithi047 ceph-mon[77147]: pgmap v8550: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:01:05.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:01:05 smithi115 conmon[90393]: 2022-05-14T04:01:05.109+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:01:05.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:01:05 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:01:05.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:01:05 smithi115 ceph-mon[78593]: pgmap v8550: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:01:06.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:01:06 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:01:06.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:01:06 smithi115 conmon[90393]: 2022-05-14T04:01:06.134+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:01:06.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:01:06 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:01:07.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:01:07 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:01:07.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:01:07 smithi047 ceph-mon[77147]: pgmap v8551: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:01:07.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:01:07 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 16400 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:01:07.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:01:07 smithi115 conmon[90393]: 2022-05-14T04:01:07.115+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:01:07.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:01:07 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:01:07.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:01:07 smithi115 ceph-mon[78593]: pgmap v8551: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:01:07.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:01:07 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 16400 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:01:08.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:01:08 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:01:08.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:01:08 smithi115 conmon[90393]: 2022-05-14T04:01:08.163+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:01:08.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:01:08 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:01:09.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:01:09 smithi047 ceph-mon[77147]: pgmap v8552: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:01:09.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:01:09 smithi115 conmon[90393]: 2022-05-14T04:01:09.170+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:01:09.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:01:09 smithi115 ceph-mon[78593]: pgmap v8552: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:01:10.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:01:10 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:01:10.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:01:10 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:01:10.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:01:10 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:01:10.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:01:10 smithi115 conmon[90393]: 2022-05-14T04:01:10.194+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:01:10.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:01:10 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:01:10.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:01:10 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:01:10.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:01:10 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:01:11.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:01:11 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:01:11.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:01:11 smithi047 ceph-mon[77147]: pgmap v8553: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:01:11.610 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:01:11 smithi115 conmon[90393]: 2022-05-14T04:01:11.151+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:01:11.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:01:11 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:01:11.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:01:11 smithi115 ceph-mon[78593]: pgmap v8553: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:01:12.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:01:12 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:01:12.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:01:12 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:01:12.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:01:12 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 16405 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:01:12.610 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:01:12 smithi115 conmon[90393]: 2022-05-14T04:01:12.155+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:01:12.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:01:12 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:01:12.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:01:12 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:01:12.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:01:12 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 16405 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:01:13.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:01:13 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:01:13.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:01:13 smithi047 ceph-mon[77147]: pgmap v8554: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:01:13.610 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:01:13 smithi115 conmon[90393]: 2022-05-14T04:01:13.117+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:01:13.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:01:13 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:01:13.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:01:13 smithi115 ceph-mon[78593]: pgmap v8554: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:01:14.360 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:01:14 smithi115 conmon[90393]: 2022-05-14T04:01:14.074+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:01:14.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:01:14 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:01:14.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:01:14 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:01:14.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:01:14 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:01:14.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:01:14 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:01:14.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:01:14 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:01:14.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:01:14 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:01:15.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:01:15 smithi115 conmon[90393]: 2022-05-14T04:01:15.038+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:01:15.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:01:15 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:01:15.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:01:15 smithi115 ceph-mon[78593]: pgmap v8555: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:01:15.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:01:15 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T04:01:15.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:01:15 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T04:01:15.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:01:15 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:01:15.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:01:15 smithi047 ceph-mon[77147]: pgmap v8555: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:01:15.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:01:15 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T04:01:15.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:01:15 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T04:01:16.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:01:16 smithi115 conmon[90393]: 2022-05-14T04:01:16.078+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:01:16.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:01:16 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:01:16.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:01:16 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:01:16.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:01:16 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:01:16.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:01:16 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:01:17.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:01:17 smithi115 conmon[90393]: 2022-05-14T04:01:17.105+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:01:17.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:01:17 smithi115 ceph-mon[78593]: pgmap v8556: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:01:17.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:01:17 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 16410 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:01:17.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:01:17 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:01:17.545 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:01:17 smithi047 ceph-mon[77147]: pgmap v8556: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:01:17.545 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:01:17 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 16410 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:01:17.545 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:01:17 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:01:18.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:01:18 smithi115 conmon[90393]: 2022-05-14T04:01:18.121+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:01:19.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:01:19 smithi115 conmon[90393]: 2022-05-14T04:01:19.166+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:01:19.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:01:19 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:01:19.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:01:19 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:01:19.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:01:19 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:01:19.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:01:19 smithi115 ceph-mon[78593]: pgmap v8557: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:01:19.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:01:19 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:01:19.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:01:19 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:01:19.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:01:19 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:01:19.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:01:19 smithi047 ceph-mon[77147]: pgmap v8557: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:01:20.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:01:20 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:01:20.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:01:20 smithi115 conmon[90393]: 2022-05-14T04:01:20.174+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:01:20.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:01:20 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:01:21.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:01:21 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:01:21.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:01:21 smithi047 ceph-mon[77147]: pgmap v8558: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:01:21.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:01:21 smithi115 conmon[90393]: 2022-05-14T04:01:21.176+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:01:21.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:01:21 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:01:21.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:01:21 smithi115 ceph-mon[78593]: pgmap v8558: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:01:22.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:01:22 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:01:22.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:01:22 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:01:22.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:01:22 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:01:22.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:01:22 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 16415 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:01:22.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:01:22 smithi115 conmon[90393]: 2022-05-14T04:01:22.216+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:01:22.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:01:22 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:01:22.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:01:22 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:01:22.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:01:22 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:01:22.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:01:22 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 16415 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:01:23.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:01:23 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:01:23.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:01:23 smithi047 ceph-mon[77147]: pgmap v8559: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:01:23.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:01:23 smithi115 conmon[90393]: 2022-05-14T04:01:23.240+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:01:23.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:01:23 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:01:23.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:01:23 smithi115 ceph-mon[78593]: pgmap v8559: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:01:24.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:01:24 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:01:24.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:01:24 smithi115 conmon[90393]: 2022-05-14T04:01:24.286+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:01:24.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:01:24 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:01:25.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:01:25 smithi047 ceph-mon[77147]: pgmap v8560: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:01:25.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:01:25 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:01:25.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:01:25 smithi115 conmon[90393]: 2022-05-14T04:01:25.301+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:01:25.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:01:25 smithi115 ceph-mon[78593]: pgmap v8560: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:01:25.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:01:25 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:01:26.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:01:26 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:01:26.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:01:26 smithi115 conmon[90393]: 2022-05-14T04:01:26.342+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:01:26.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:01:26 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:01:27.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:01:27 smithi047 ceph-mon[77147]: pgmap v8561: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:01:27.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:01:27 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:01:27.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:01:27 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 16420 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:01:27.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:01:27 smithi115 conmon[90393]: 2022-05-14T04:01:27.389+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:01:27.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:01:27 smithi115 ceph-mon[78593]: pgmap v8561: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:01:27.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:01:27 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:01:27.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:01:27 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 16420 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:01:28.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:01:28 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:01:28.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:01:28 smithi115 conmon[90393]: 2022-05-14T04:01:28.431+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:01:28.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:01:28 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:01:29.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:01:29 smithi047 ceph-mon[77147]: pgmap v8562: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:01:29.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:01:29 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:01:29.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:01:29 smithi115 conmon[90393]: 2022-05-14T04:01:29.412+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:01:29.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:01:29 smithi115 ceph-mon[78593]: pgmap v8562: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:01:29.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:01:29 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:01:30.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:01:30 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:01:30.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:01:30 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:01:30.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:01:30 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:01:30.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:01:30 smithi115 conmon[90393]: 2022-05-14T04:01:30.405+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:01:30.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:01:30 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:01:30.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:01:30 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:01:30.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:01:30 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:01:31.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:01:31 smithi047 ceph-mon[77147]: pgmap v8563: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:01:31.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:01:31 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:01:31.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:01:31 smithi115 conmon[90393]: 2022-05-14T04:01:31.413+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:01:31.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:01:31 smithi115 ceph-mon[78593]: pgmap v8563: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:01:31.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:01:31 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:01:32.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:01:32 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:01:32.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:01:32 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 16425 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:01:32.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:01:32 smithi115 conmon[90393]: 2022-05-14T04:01:32.415+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:01:32.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:01:32 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:01:32.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:01:32 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 16425 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:01:33.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:01:33 smithi047 ceph-mon[77147]: pgmap v8564: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:01:33.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:01:33 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:01:33.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:01:33 smithi115 conmon[90393]: 2022-05-14T04:01:33.430+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:01:33.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:01:33 smithi115 ceph-mon[78593]: pgmap v8564: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:01:33.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:01:33 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:01:34.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:01:34 smithi115 conmon[90393]: 2022-05-14T04:01:34.455+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:01:35.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:01:35 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:01:35.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:01:35 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:01:35.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:01:35 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:01:35.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:01:35 smithi115 ceph-mon[78593]: pgmap v8565: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:01:35.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:01:35 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:01:35.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:01:35 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:01:35.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:01:35 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:01:35.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:01:35 smithi047 ceph-mon[77147]: pgmap v8565: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:01:35.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:01:35 smithi115 conmon[90393]: 2022-05-14T04:01:35.449+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:01:36.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:01:36 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:01:36.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:01:36 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:01:36.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:01:36 smithi115 conmon[90393]: 2022-05-14T04:01:36.455+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:01:37.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:01:37 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:01:37.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:01:37 smithi115 ceph-mon[78593]: pgmap v8566: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:01:37.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:01:37 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 16430 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:01:37.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:01:37 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:01:37.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:01:37 smithi047 ceph-mon[77147]: pgmap v8566: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:01:37.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:01:37 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 16430 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:01:37.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:01:37 smithi115 conmon[90393]: 2022-05-14T04:01:37.461+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:01:38.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:01:38 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:01:38.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:01:38 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:01:38.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:01:38 smithi115 conmon[90393]: 2022-05-14T04:01:38.458+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:01:39.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:01:39 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:01:39.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:01:39 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:01:39.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:01:39 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:01:39.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:01:39 smithi115 ceph-mon[78593]: pgmap v8567: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:01:39.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:01:39 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:01:39.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:01:39 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:01:39.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:01:39 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:01:39.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:01:39 smithi047 ceph-mon[77147]: pgmap v8567: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:01:39.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:01:39 smithi115 conmon[90393]: 2022-05-14T04:01:39.465+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:01:40.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:01:40 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:01:40.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:01:40 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:01:40.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:01:40 smithi115 conmon[90393]: 2022-05-14T04:01:40.447+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:01:41.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:01:41 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:01:41.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:01:41 smithi115 ceph-mon[78593]: pgmap v8568: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:01:41.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:01:41 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:01:41.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:01:41 smithi047 ceph-mon[77147]: pgmap v8568: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:01:41.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:01:41 smithi115 conmon[90393]: 2022-05-14T04:01:41.466+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:01:42.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:01:42 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:01:42.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:01:42 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 16435 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:01:42.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:01:42 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:01:42.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:01:42 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 16435 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:01:42.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:01:42 smithi115 conmon[90393]: 2022-05-14T04:01:42.494+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:01:43.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:01:43 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:01:43.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:01:43 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:01:43.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:01:43 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:01:43.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:01:43 smithi115 ceph-mon[78593]: pgmap v8569: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:01:43.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:01:43 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:01:43.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:01:43 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:01:43.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:01:43 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:01:43.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:01:43 smithi047 ceph-mon[77147]: pgmap v8569: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:01:43.862 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:01:43 smithi115 conmon[90393]: 2022-05-14T04:01:43.494+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:01:44.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:01:44 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:01:44.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:01:44 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:01:44.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:01:44 smithi115 conmon[90393]: 2022-05-14T04:01:44.450+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:01:45.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:01:45 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:01:45.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:01:45 smithi115 ceph-mon[78593]: pgmap v8570: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:01:45.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:01:45 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:01:45.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:01:45 smithi047 ceph-mon[77147]: pgmap v8570: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:01:45.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:01:45 smithi115 conmon[90393]: 2022-05-14T04:01:45.489+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:01:46.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:01:46 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:01:46.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:01:46 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:01:46.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:01:46 smithi115 conmon[90393]: 2022-05-14T04:01:46.539+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:01:47.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:01:47 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:01:47.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:01:47 smithi115 ceph-mon[78593]: pgmap v8571: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:01:47.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:01:47 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 16440 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:01:47.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:01:47 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:01:47.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:01:47 smithi047 ceph-mon[77147]: pgmap v8571: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:01:47.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:01:47 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 16440 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:01:47.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:01:47 smithi115 conmon[90393]: 2022-05-14T04:01:47.526+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:01:48.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:01:48 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:01:48.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:01:48 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:01:48.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:01:48 smithi115 conmon[90393]: 2022-05-14T04:01:48.491+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:01:49.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:01:49 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:01:49.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:01:49 smithi115 ceph-mon[78593]: pgmap v8572: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:01:49.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:01:49 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:01:49.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:01:49 smithi047 ceph-mon[77147]: pgmap v8572: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:01:49.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:01:49 smithi115 conmon[90393]: 2022-05-14T04:01:49.453+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:01:50.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:01:50 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:01:50.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:01:50 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:01:50.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:01:50 smithi115 conmon[90393]: 2022-05-14T04:01:50.420+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:01:51.414 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:01:51 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:01:51.415 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:01:51 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:01:51.416 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:01:51 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:01:51.416 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:01:51 smithi115 ceph-mon[78593]: pgmap v8573: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:01:51.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:01:51 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:01:51.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:01:51 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:01:51.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:01:51 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:01:51.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:01:51 smithi047 ceph-mon[77147]: pgmap v8573: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:01:51.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:01:51 smithi115 conmon[90393]: 2022-05-14T04:01:51.411+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:01:52.422 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:01:52 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:01:52.423 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:01:52 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 16445 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:01:52.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:01:52 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:01:52.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:01:52 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 16445 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:01:52.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:01:52 smithi115 conmon[90393]: 2022-05-14T04:01:52.419+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:01:53.430 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:01:53 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:01:53.430 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:01:53 smithi115 ceph-mon[78593]: pgmap v8574: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:01:53.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:01:53 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:01:53.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:01:53 smithi047 ceph-mon[77147]: pgmap v8574: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:01:53.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:01:53 smithi115 conmon[90393]: 2022-05-14T04:01:53.427+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:01:54.457 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:01:54 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:01:54.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:01:54 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:01:54.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:01:54 smithi115 conmon[90393]: 2022-05-14T04:01:54.454+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:01:55.425 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:01:55 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:01:55.426 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:01:55 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:01:55.426 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:01:55 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:01:55.426 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:01:55 smithi115 ceph-mon[78593]: pgmap v8575: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:01:55.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:01:55 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:01:55.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:01:55 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:01:55.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:01:55 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:01:55.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:01:55 smithi047 ceph-mon[77147]: pgmap v8575: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:01:55.860 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:01:55 smithi115 conmon[90393]: 2022-05-14T04:01:55.422+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:01:56.435 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:01:56 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:01:56.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:01:56 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:01:56.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:01:56 smithi115 conmon[90393]: 2022-05-14T04:01:56.432+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:01:57.446 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:01:57 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:01:57.447 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:01:57 smithi115 ceph-mon[78593]: pgmap v8576: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:01:57.447 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:01:57 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 16450 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:01:57.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:01:57 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:01:57.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:01:57 smithi047 ceph-mon[77147]: pgmap v8576: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:01:57.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:01:57 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 16450 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:01:57.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:01:57 smithi115 conmon[90393]: 2022-05-14T04:01:57.443+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:01:58.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:01:58 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:01:58.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:01:58 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T04:01:58.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:01:58 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T04:01:58.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:01:58 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T04:01:58.362 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:01:58 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:01:58.362 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:01:58 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:01:58.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:01:58 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:01:58.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:01:58 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T04:01:58.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:01:58 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T04:01:58.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:01:58 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T04:01:58.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:01:58 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:01:58.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:01:58 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:01:58.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:01:58 smithi115 conmon[90393]: 2022-05-14T04:01:58.491+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:01:59.447 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:01:59 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:01:59.448 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:01:59 smithi115 ceph-mon[78593]: pgmap v8577: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:01:59.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:01:59 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:01:59.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:01:59 smithi047 ceph-mon[77147]: pgmap v8577: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:01:59.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:01:59 smithi115 conmon[90393]: 2022-05-14T04:01:59.445+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:02:00.475 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:02:00 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:02:00.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:02:00 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:02:00.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:02:00 smithi115 conmon[90393]: 2022-05-14T04:02:00.471+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:02:01.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:02:00 smithi047 conmon[78521]: 2022-05-14T04:02:00.624+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:02:01.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:02:00 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T04:02:01.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:02:00 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:02:01.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:02:00 smithi047 conmon[78521]: yield 2022-05-14T04:02:01.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:02:00 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:02:01.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:02:00 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:02:01.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:02:00 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:02:01.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:02:00 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T04:02:01.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:02:00 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:02:01.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:02:00 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T04:02:01.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:02:00 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:02:01.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:02:00 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T04:02:01.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:02:00 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:02:01.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:02:00 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T04:02:01.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:02:00 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:02:01.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:02:00 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T04:02:01.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:02:00 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:02:01.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:02:00 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:02:01.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:02:00 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:02:01.431 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:02:01 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:02:01.432 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:02:01 smithi115 ceph-mon[78593]: pgmap v8578: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:02:01.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:02:01 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:02:01.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:02:01 smithi047 ceph-mon[77147]: pgmap v8578: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:02:01.860 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:02:01 smithi115 conmon[90393]: 2022-05-14T04:02:01.428+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:02:02.415 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:02:02 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:02:02.415 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:02:02 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:02:02.415 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T04:02:02.416 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:02:02.416 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T04:02:02.416 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:02:02.417 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:02:02.417 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:02:02.417 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T04:02:02.417 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:02:02.418 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T04:02:02.418 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:02:02.418 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T04:02:02.418 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:02:02.419 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T04:02:02.419 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:02:02.419 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T04:02:02.419 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:02:02.420 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:02:02.420 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:02:02.420 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:02:02 smithi115 ceph-mon[78593]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T04:02:02.420 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:02:02 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:02:02.420 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:02:02 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:02:02.421 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:02:02 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 16455 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:02:02.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:02:02 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:02:02.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:02:02 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:02:02.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T04:02:02.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:02:02.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T04:02:02.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:02:02.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:02:02.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:02:02.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T04:02:02.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:02:02.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T04:02:02.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:02:02.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T04:02:02.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:02:02.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T04:02:02.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:02:02.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T04:02:02.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:02:02.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:02:02.548 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:02:02.548 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:02:02 smithi047 ceph-mon[77147]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T04:02:02.548 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:02:02 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:02:02.548 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:02:02 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:02:02.549 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:02:02 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 16455 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:02:02.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:02:02 smithi115 conmon[90393]: 2022-05-14T04:02:02.411+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:02:03.460 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:02:03 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:02:03.460 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:02:03 smithi115 ceph-mon[78593]: pgmap v8579: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:02:03.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:02:03 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:02:03.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:02:03 smithi047 ceph-mon[77147]: pgmap v8579: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:02:03.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:02:03 smithi115 conmon[90393]: 2022-05-14T04:02:03.457+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:02:04.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:02:03 smithi047 conmon[78521]: 2022-05-14T04:02:03.696+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:02:04.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:02:03 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T04:02:04.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:02:03 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:02:04.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:02:03 smithi047 conmon[78521]: yield 2022-05-14T04:02:04.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:02:03 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:02:04.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:02:03 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:02:04.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:02:03 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:02:04.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:02:03 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T04:02:04.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:02:03 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:02:04.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:02:03 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T04:02:04.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:02:03 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:02:04.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:02:03 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T04:02:04.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:02:03 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:02:04.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:02:03 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T04:02:04.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:02:03 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:02:04.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:02:03 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T04:02:04.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:02:03 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:02:04.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:02:03 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:02:04.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:02:03 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:02:04.048 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:02:03 smithi047 conmon[78521]: 2022-05-14T04:02:03.697+0000 7f5344781700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T04:02:04.048 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:02:03 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T04:02:04.048 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:02:03 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:02:04.048 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:02:03 smithi047 conmon[78521]: yield 2022-05-14T04:02:04.048 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:02:03 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:02:04.049 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:02:03 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:02:04.049 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:02:03 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:02:04.049 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:02:03 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T04:02:04.049 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:02:03 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:02:04.050 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:02:03 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T04:02:04.050 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:02:03 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:02:04.050 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:02:03 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T04:02:04.050 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:02:03 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:02:04.050 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:02:03 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T04:02:04.051 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:02:03 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:02:04.051 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:02:03 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T04:02:04.051 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:02:03 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:02:04.051 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:02:03 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:02:04.052 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:02:03 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:02:04.053 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:02:03 smithi047 conmon[78521]: 2022-05-14T04:02:04.053 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:02:03 smithi047 conmon[78521]: During handling of the above exception, another exception occurred: 2022-05-14T04:02:04.053 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:02:03 smithi047 conmon[78521]: 2022-05-14T04:02:04.054 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:02:03 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T04:02:04.054 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:02:03 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T04:02:04.055 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:02:03 smithi047 conmon[78521]: return f(*arg) 2022-05-14T04:02:04.055 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:02:03 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T04:02:04.056 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:02:03 smithi047 conmon[78521]: self._write_client_files(client_files, host) 2022-05-14T04:02:04.056 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:02:03 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T04:02:04.057 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:02:03 smithi047 conmon[78521]: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T04:02:04.057 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:02:03 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T04:02:04.058 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:02:03 smithi047 conmon[78521]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T04:02:04.058 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:02:03 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T04:02:04.058 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:02:03 smithi047 conmon[78521]: return self.event_loop.get_result(coro) 2022-05-14T04:02:04.059 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:02:03 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T04:02:04.059 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:02:03 smithi047 conmon[78521]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T04:02:04.060 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:02:03 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T04:02:04.060 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:02:03 smithi047 conmon[78521]: return self.__get_result() 2022-05-14T04:02:04.061 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:02:03 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T04:02:04.061 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:02:03 smithi047 conmon[78521]: raise self._exception 2022-05-14T04:02:04.062 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:02:03 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T04:02:04.062 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:02:03 smithi047 conmon[78521]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T04:02:04.063 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:02:03 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T04:02:04.063 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:02:03 smithi047 conmon[78521]: conn = await self._remote_connection(host, addr) 2022-05-14T04:02:04.063 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:02:03 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T04:02:04.064 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:02:03 smithi047 conmon[78521]: raise 2022-05-14T04:02:04.064 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:02:03 smithi047 conmon[78521]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T04:02:04.065 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:02:03 smithi047 conmon[78521]: self.gen.throw(type, value, traceback) 2022-05-14T04:02:04.065 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:02:03 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T04:02:04.066 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:02:03 smithi047 conmon[78521]: raise OrchestratorError(msg) 2022-05-14T04:02:04.066 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:02:03 smithi047 conmon[78521]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:02:04.445 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:02:04 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:02:04.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:02:04 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:02:04.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:02:04 smithi115 conmon[90393]: 2022-05-14T04:02:04.442+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:02:05.478 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:02:05 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:02:05.479 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:02:05 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:02:05.479 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T04:02:05.479 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:02:05.480 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T04:02:05.480 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:02:05.480 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:02:05.480 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:02:05.480 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T04:02:05.481 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:02:05.481 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T04:02:05.481 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:02:05.482 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T04:02:05.482 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:02:05.482 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T04:02:05.482 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:02:05.482 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T04:02:05.483 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:02:05.483 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:02:05.483 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:02:05.483 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:02:05 smithi115 ceph-mon[78593]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T04:02:05.484 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T04:02:05.484 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:02:05.484 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T04:02:05.484 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:02:05.485 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:02:05.485 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:02:05.485 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T04:02:05.485 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:02:05.486 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T04:02:05.486 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:02:05.486 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T04:02:05.486 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:02:05.487 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T04:02:05.487 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:02:05.487 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T04:02:05.487 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:02:05.488 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:02:05.488 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:02:05.488 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T04:02:05.488 INFO:journalctl@ceph.mon.c.smithi115.stdout: During handling of the above exception, another exception occurred: 2022-05-14T04:02:05.488 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T04:02:05.489 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T04:02:05.489 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T04:02:05.489 INFO:journalctl@ceph.mon.c.smithi115.stdout: return f(*arg) 2022-05-14T04:02:05.490 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T04:02:05.490 INFO:journalctl@ceph.mon.c.smithi115.stdout: self._write_client_files(client_files, host) 2022-05-14T04:02:05.490 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T04:02:05.491 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T04:02:05.491 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T04:02:05.491 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T04:02:05.491 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T04:02:05.492 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.event_loop.get_result(coro) 2022-05-14T04:02:05.492 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T04:02:05.492 INFO:journalctl@ceph.mon.c.smithi115.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T04:02:05.493 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T04:02:05.493 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.__get_result() 2022-05-14T04:02:05.493 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T04:02:05.493 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise self._exception 2022-05-14T04:02:05.494 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T04:02:05.494 INFO:journalctl@ceph.mon.c.smithi115.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T04:02:05.494 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T04:02:05.494 INFO:journalctl@ceph.mon.c.smithi115.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T04:02:05.494 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T04:02:05.495 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise 2022-05-14T04:02:05.495 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T04:02:05.495 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.gen.throw(type, value, traceback) 2022-05-14T04:02:05.495 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T04:02:05.496 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OrchestratorError(msg) 2022-05-14T04:02:05.496 INFO:journalctl@ceph.mon.c.smithi115.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:02:05.496 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:02:05 smithi115 ceph-mon[78593]: pgmap v8580: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:02:05.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:02:05 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:02:05.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:02:05 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:02:05.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T04:02:05.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:02:05.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T04:02:05.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:02:05.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:02:05.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:02:05.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T04:02:05.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:02:05.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T04:02:05.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:02:05.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T04:02:05.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:02:05.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T04:02:05.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:02:05.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T04:02:05.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:02:05.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:02:05.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:02:05.547 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:02:05 smithi047 ceph-mon[77147]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T04:02:05.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T04:02:05.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:02:05.548 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T04:02:05.548 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:02:05.548 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:02:05.548 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:02:05.549 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T04:02:05.549 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:02:05.549 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T04:02:05.549 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:02:05.549 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T04:02:05.550 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:02:05.550 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T04:02:05.550 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:02:05.550 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T04:02:05.550 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:02:05.551 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:02:05.551 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:02:05.551 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T04:02:05.551 INFO:journalctl@ceph.mon.b.smithi047.stdout: During handling of the above exception, another exception occurred: 2022-05-14T04:02:05.552 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T04:02:05.552 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T04:02:05.552 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T04:02:05.552 INFO:journalctl@ceph.mon.b.smithi047.stdout: return f(*arg) 2022-05-14T04:02:05.552 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T04:02:05.553 INFO:journalctl@ceph.mon.b.smithi047.stdout: self._write_client_files(client_files, host) 2022-05-14T04:02:05.553 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T04:02:05.553 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T04:02:05.553 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T04:02:05.553 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T04:02:05.554 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T04:02:05.554 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.event_loop.get_result(coro) 2022-05-14T04:02:05.554 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T04:02:05.554 INFO:journalctl@ceph.mon.b.smithi047.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T04:02:05.554 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T04:02:05.555 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.__get_result() 2022-05-14T04:02:05.555 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T04:02:05.555 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise self._exception 2022-05-14T04:02:05.555 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T04:02:05.555 INFO:journalctl@ceph.mon.b.smithi047.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T04:02:05.556 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T04:02:05.556 INFO:journalctl@ceph.mon.b.smithi047.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T04:02:05.556 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T04:02:05.556 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise 2022-05-14T04:02:05.556 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T04:02:05.557 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.gen.throw(type, value, traceback) 2022-05-14T04:02:05.557 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T04:02:05.557 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OrchestratorError(msg) 2022-05-14T04:02:05.557 INFO:journalctl@ceph.mon.b.smithi047.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:02:05.558 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:02:05 smithi047 ceph-mon[77147]: pgmap v8580: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:02:05.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:02:05 smithi115 conmon[90393]: 2022-05-14T04:02:05.475+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:02:06.515 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:02:06 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:02:06.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:02:06 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:02:06.860 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:02:06 smithi115 conmon[90393]: 2022-05-14T04:02:06.512+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:02:07.512 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:02:07 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:02:07.512 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:02:07 smithi115 ceph-mon[78593]: pgmap v8581: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:02:07.512 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:02:07 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 16460 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:02:07.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:02:07 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:02:07.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:02:07 smithi047 ceph-mon[77147]: pgmap v8581: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:02:07.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:02:07 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 16460 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:02:07.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:02:07 smithi115 conmon[90393]: 2022-05-14T04:02:07.509+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:02:08.531 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:02:08 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:02:08.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:02:08 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:02:08.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:02:08 smithi115 conmon[90393]: 2022-05-14T04:02:08.528+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:02:09.531 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:02:09 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:02:09.532 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:02:09 smithi115 ceph-mon[78593]: pgmap v8582: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:02:09.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:02:09 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:02:09.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:02:09 smithi047 ceph-mon[77147]: pgmap v8582: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:02:09.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:02:09 smithi115 conmon[90393]: 2022-05-14T04:02:09.529+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:02:10.519 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:02:10 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:02:10.520 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:02:10 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:02:10.520 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:02:10 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:02:10.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:02:10 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:02:10.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:02:10 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:02:10.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:02:10 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:02:10.861 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:02:10 smithi115 conmon[90393]: 2022-05-14T04:02:10.516+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:02:11.528 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:02:11 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:02:11.528 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:02:11 smithi115 ceph-mon[78593]: pgmap v8583: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:02:11.529 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:02:11 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:02:11.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:02:11 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:02:11.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:02:11 smithi047 ceph-mon[77147]: pgmap v8583: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:02:11.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:02:11 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:02:11.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:02:11 smithi115 conmon[90393]: 2022-05-14T04:02:11.525+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:02:12.484 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:02:12 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:02:12.485 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:02:12 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 16465 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:02:12.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:02:12 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:02:12.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:02:12 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 16465 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:02:12.860 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:02:12 smithi115 conmon[90393]: 2022-05-14T04:02:12.481+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:02:13.446 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:02:13 smithi115 ceph-mon[78593]: pgmap v8584: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:02:13.446 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:02:13 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:02:13.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:02:13 smithi047 ceph-mon[77147]: pgmap v8584: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:02:13.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:02:13 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:02:13.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:02:13 smithi115 conmon[90393]: 2022-05-14T04:02:13.443+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:02:14.481 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:02:14 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:02:14.482 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:02:14 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:02:14.482 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:02:14 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:02:14.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:02:14 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:02:14.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:02:14 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:02:14.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:02:14 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:02:14.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:02:14 smithi115 conmon[90393]: 2022-05-14T04:02:14.479+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:02:15.477 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:02:15 smithi115 ceph-mon[78593]: pgmap v8585: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:02:15.478 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:02:15 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:02:15.478 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:02:15 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T04:02:15.478 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:02:15 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T04:02:15.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:02:15 smithi047 ceph-mon[77147]: pgmap v8585: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:02:15.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:02:15 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:02:15.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:02:15 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T04:02:15.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:02:15 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T04:02:15.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:02:15 smithi115 conmon[90393]: 2022-05-14T04:02:15.475+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:02:16.476 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:02:16 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:02:16.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:02:16 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:02:16.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:02:16 smithi115 conmon[90393]: 2022-05-14T04:02:16.474+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:02:17.466 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:02:17 smithi115 ceph-mon[78593]: pgmap v8586: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:02:17.466 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:02:17 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:02:17.467 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:02:17 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 16470 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:02:17.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:02:17 smithi047 ceph-mon[77147]: pgmap v8586: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:02:17.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:02:17 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:02:17.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:02:17 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 16470 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:02:17.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:02:17 smithi115 conmon[90393]: 2022-05-14T04:02:17.464+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:02:18.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:02:18 smithi115 conmon[90393]: 2022-05-14T04:02:18.438+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:02:19.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:02:19 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:02:19.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:02:19 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:02:19.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:02:19 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:02:19.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:02:19 smithi115 ceph-mon[78593]: pgmap v8587: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:02:19.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:02:19 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:02:19.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:02:19 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:02:19.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:02:19 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:02:19.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:02:19 smithi047 ceph-mon[77147]: pgmap v8587: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:02:19.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:02:19 smithi115 conmon[90393]: 2022-05-14T04:02:19.466+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:02:20.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:02:20 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:02:20.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:02:20 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:02:20.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:02:20 smithi115 conmon[90393]: 2022-05-14T04:02:20.457+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:02:21.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:02:21 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:02:21.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:02:21 smithi115 ceph-mon[78593]: pgmap v8588: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:02:21.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:02:21 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:02:21.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:02:21 smithi047 ceph-mon[77147]: pgmap v8588: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:02:21.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:02:21 smithi115 conmon[90393]: 2022-05-14T04:02:21.407+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:02:22.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:02:22 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:02:22.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:02:22 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 16475 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:02:22.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:02:22 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:02:22.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:02:22 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 16475 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:02:22.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:02:22 smithi115 conmon[90393]: 2022-05-14T04:02:22.414+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:02:23.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:02:23 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:02:23.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:02:23 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:02:23.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:02:23 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:02:23.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:02:23 smithi115 ceph-mon[78593]: pgmap v8589: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:02:23.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:02:23 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:02:23.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:02:23 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:02:23.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:02:23 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:02:23.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:02:23 smithi047 ceph-mon[77147]: pgmap v8589: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:02:23.860 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:02:23 smithi115 conmon[90393]: 2022-05-14T04:02:23.459+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:02:24.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:02:24 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:02:24.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:02:24 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:02:24.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:02:24 smithi115 conmon[90393]: 2022-05-14T04:02:24.433+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:02:25.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:02:25 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:02:25.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:02:25 smithi115 ceph-mon[78593]: pgmap v8590: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:02:25.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:02:25 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:02:25.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:02:25 smithi047 ceph-mon[77147]: pgmap v8590: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:02:25.860 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:02:25 smithi115 conmon[90393]: 2022-05-14T04:02:25.416+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:02:26.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:02:26 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:02:26.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:02:26 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:02:26.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:02:26 smithi115 conmon[90393]: 2022-05-14T04:02:26.464+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:02:27.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:02:27 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:02:27.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:02:27 smithi115 ceph-mon[78593]: pgmap v8591: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:02:27.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:02:27 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 16480 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:02:27.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:02:27 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:02:27.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:02:27 smithi047 ceph-mon[77147]: pgmap v8591: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:02:27.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:02:27 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 16480 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:02:27.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:02:27 smithi115 conmon[90393]: 2022-05-14T04:02:27.435+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:02:28.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:02:28 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:02:28.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:02:28 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:02:28.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:02:28 smithi115 conmon[90393]: 2022-05-14T04:02:28.448+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:02:29.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:02:29 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:02:29.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:02:29 smithi115 ceph-mon[78593]: pgmap v8592: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:02:29.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:02:29 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:02:29.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:02:29 smithi047 ceph-mon[77147]: pgmap v8592: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:02:29.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:02:29 smithi115 conmon[90393]: 2022-05-14T04:02:29.476+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:02:30.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:02:30 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:02:30.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:02:30 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:02:30.866 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:02:30 smithi115 conmon[90393]: 2022-05-14T04:02:30.444+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:02:31.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:02:31 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:02:31.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:02:31 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:02:31.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:02:31 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:02:31.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:02:31 smithi115 ceph-mon[78593]: pgmap v8593: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:02:31.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:02:31 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:02:31.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:02:31 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:02:31.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:02:31 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:02:31.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:02:31 smithi047 ceph-mon[77147]: pgmap v8593: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:02:31.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:02:31 smithi115 conmon[90393]: 2022-05-14T04:02:31.462+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:02:32.440 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:02:32 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:02:32.440 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:02:32 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 16485 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:02:32.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:02:32 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:02:32.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:02:32 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 16485 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:02:32.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:02:32 smithi115 conmon[90393]: 2022-05-14T04:02:32.437+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:02:33.454 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:02:33 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:02:33.455 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:02:33 smithi115 ceph-mon[78593]: pgmap v8594: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:02:33.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:02:33 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:02:33.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:02:33 smithi047 ceph-mon[77147]: pgmap v8594: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:02:33.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:02:33 smithi115 conmon[90393]: 2022-05-14T04:02:33.451+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:02:34.441 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:02:34 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:02:34.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:02:34 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:02:34.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:02:34 smithi115 conmon[90393]: 2022-05-14T04:02:34.435+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:02:35.475 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:02:35 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:02:35.476 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:02:35 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:02:35.476 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:02:35 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:02:35.476 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:02:35 smithi115 ceph-mon[78593]: pgmap v8595: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:02:35.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:02:35 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:02:35.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:02:35 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:02:35.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:02:35 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:02:35.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:02:35 smithi047 ceph-mon[77147]: pgmap v8595: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:02:35.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:02:35 smithi115 conmon[90393]: 2022-05-14T04:02:35.472+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:02:36.520 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:02:36 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:02:36.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:02:36 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:02:36.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:02:36 smithi115 conmon[90393]: 2022-05-14T04:02:36.517+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:02:37.482 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:02:37 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:02:37.482 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:02:37 smithi115 ceph-mon[78593]: pgmap v8596: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:02:37.482 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:02:37 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 16490 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:02:37.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:02:37 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:02:37.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:02:37 smithi047 ceph-mon[77147]: pgmap v8596: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:02:37.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:02:37 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 16490 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:02:37.860 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:02:37 smithi115 conmon[90393]: 2022-05-14T04:02:37.478+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:02:38.444 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:02:38 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:02:38.444 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:02:38 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:02:38.445 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:02:38 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:02:38.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:02:38 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:02:38.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:02:38 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:02:38.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:02:38 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:02:38.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:02:38 smithi115 conmon[90393]: 2022-05-14T04:02:38.441+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:02:39.489 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:02:39 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:02:39.490 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:02:39 smithi115 ceph-mon[78593]: pgmap v8597: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:02:39.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:02:39 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:02:39.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:02:39 smithi047 ceph-mon[77147]: pgmap v8597: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:02:39.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:02:39 smithi115 conmon[90393]: 2022-05-14T04:02:39.486+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:02:40.465 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:02:40 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:02:40.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:02:40 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:02:40.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:02:40 smithi115 conmon[90393]: 2022-05-14T04:02:40.461+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:02:41.508 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:02:41 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:02:41.508 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:02:41 smithi115 ceph-mon[78593]: pgmap v8598: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:02:41.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:02:41 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:02:41.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:02:41 smithi047 ceph-mon[77147]: pgmap v8598: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:02:41.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:02:41 smithi115 conmon[90393]: 2022-05-14T04:02:41.505+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:02:42.493 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:02:42 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:02:42.500 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:02:42 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:02:42.500 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:02:42 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:02:42.500 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:02:42 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 16495 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:02:42.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:02:42 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:02:42.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:02:42 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:02:42.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:02:42 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:02:42.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:02:42 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 16495 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:02:42.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:02:42 smithi115 conmon[90393]: 2022-05-14T04:02:42.490+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:02:43.489 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:02:43 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:02:43.490 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:02:43 smithi115 ceph-mon[78593]: pgmap v8599: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:02:43.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:02:43 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:02:43.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:02:43 smithi047 ceph-mon[77147]: pgmap v8599: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:02:43.860 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:02:43 smithi115 conmon[90393]: 2022-05-14T04:02:43.486+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:02:44.452 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:02:44 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:02:44.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:02:44 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:02:44.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:02:44 smithi115 conmon[90393]: 2022-05-14T04:02:44.449+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:02:45.465 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:02:45 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:02:45.466 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:02:45 smithi115 ceph-mon[78593]: pgmap v8600: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:02:45.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:02:45 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:02:45.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:02:45 smithi047 ceph-mon[77147]: pgmap v8600: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:02:45.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:02:45 smithi115 conmon[90393]: 2022-05-14T04:02:45.462+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:02:46.485 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:02:46 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:02:46.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:02:46 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:02:46.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:02:46 smithi115 conmon[90393]: 2022-05-14T04:02:46.482+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:02:47.515 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:02:47 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:02:47.516 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:02:47 smithi115 ceph-mon[78593]: pgmap v8601: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:02:47.516 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:02:47 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 16500 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:02:47.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:02:47 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:02:47.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:02:47 smithi047 ceph-mon[77147]: pgmap v8601: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:02:47.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:02:47 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 16500 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:02:47.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:02:47 smithi115 conmon[90393]: 2022-05-14T04:02:47.512+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:02:48.522 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:02:48 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:02:48.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:02:48 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:02:48.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:02:48 smithi115 conmon[90393]: 2022-05-14T04:02:48.520+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:02:49.513 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:02:49 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:02:49.514 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:02:49 smithi115 ceph-mon[78593]: pgmap v8602: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:02:49.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:02:49 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:02:49.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:02:49 smithi047 ceph-mon[77147]: pgmap v8602: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:02:49.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:02:49 smithi115 conmon[90393]: 2022-05-14T04:02:49.511+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:02:50.504 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:02:50 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:02:50.504 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:02:50 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:02:50.504 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:02:50 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:02:50.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:02:50 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:02:50.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:02:50 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:02:50.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:02:50 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:02:50.860 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:02:50 smithi115 conmon[90393]: 2022-05-14T04:02:50.501+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:02:51.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:02:51 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:02:51.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:02:51 smithi047 ceph-mon[77147]: pgmap v8603: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:02:51.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:02:51 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:02:51.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:02:51 smithi115 ceph-mon[78593]: pgmap v8603: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:02:51.861 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:02:51 smithi115 conmon[90393]: 2022-05-14T04:02:51.544+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:02:52.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:02:52 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:02:52.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:02:52 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 16505 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:02:52.591 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:02:52 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:02:52.591 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:02:52 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 16505 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:02:52.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:02:52 smithi115 conmon[90393]: 2022-05-14T04:02:52.588+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:02:53.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:02:53 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:02:53.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:02:53 smithi047 ceph-mon[77147]: pgmap v8604: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:02:53.606 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:02:53 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:02:53.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:02:53 smithi115 ceph-mon[78593]: pgmap v8604: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:02:53.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:02:53 smithi115 conmon[90393]: 2022-05-14T04:02:53.604+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:02:54.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:02:54 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:02:54.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:02:54 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:02:54.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:02:54 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:02:54.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:02:54 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:02:54.560 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:02:54 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:02:54.560 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:02:54 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:02:54.561 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:02:54 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:02:54.561 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:02:54 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:02:54.860 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:02:54 smithi115 conmon[90393]: 2022-05-14T04:02:54.557+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:02:55.530 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:02:55 smithi115 ceph-mon[78593]: pgmap v8605: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:02:55.530 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:02:55 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:02:55.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:02:55 smithi047 ceph-mon[77147]: pgmap v8605: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:02:55.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:02:55 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:02:55.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:02:55 smithi115 conmon[90393]: 2022-05-14T04:02:55.527+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:02:56.488 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:02:56 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:02:56.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:02:56 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:02:56.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:02:56 smithi115 conmon[90393]: 2022-05-14T04:02:56.484+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:02:57.532 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:02:57 smithi115 ceph-mon[78593]: pgmap v8606: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:02:57.532 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:02:57 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:02:57.533 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:02:57 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 16510 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:02:57.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:02:57 smithi047 ceph-mon[77147]: pgmap v8606: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:02:57.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:02:57 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:02:57.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:02:57 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 16510 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:02:57.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:02:57 smithi115 conmon[90393]: 2022-05-14T04:02:57.529+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:02:58.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:02:58 smithi115 conmon[90393]: 2022-05-14T04:02:58.542+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:02:59.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:02:59 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:02:59.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:02:59 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:02:59.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:02:59 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:02:59.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:02:59 smithi115 ceph-mon[78593]: pgmap v8607: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:02:59.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:02:59 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:02:59.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:02:59 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:02:59.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:02:59 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:02:59.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:02:59 smithi047 ceph-mon[77147]: pgmap v8607: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:02:59.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:02:59 smithi115 conmon[90393]: 2022-05-14T04:02:59.533+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:03:00.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:03:00 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:03:00.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:03:00 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:03:00.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:03:00 smithi115 conmon[90393]: 2022-05-14T04:03:00.546+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:03:01.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:03:01 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:03:01.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:03:01 smithi115 ceph-mon[78593]: pgmap v8608: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:03:01.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:03:01 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:03:01.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:03:01 smithi047 ceph-mon[77147]: pgmap v8608: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:03:01.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:03:01 smithi115 conmon[90393]: 2022-05-14T04:03:01.513+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:03:02.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:03:02 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:03:02.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:03:02 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 16515 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:03:02.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:03:02 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:03:02.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:03:02 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 16515 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:03:02.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:03:02 smithi115 conmon[90393]: 2022-05-14T04:03:02.510+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:03:03.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:03:03 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:03:03.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:03:03 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:03:03.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:03:03 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:03:03.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:03:03 smithi115 ceph-mon[78593]: pgmap v8609: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:03:03.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:03:03 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:03:03.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:03:03 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:03:03.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:03:03 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:03:03.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:03:03 smithi047 ceph-mon[77147]: pgmap v8609: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:03:03.756 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:03:03 smithi115 conmon[90393]: 2022-05-14T04:03:03.463+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:03:04.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:03:04 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:03:04.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:03:04 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T04:03:04.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:03:04 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T04:03:04.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:03:04 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T04:03:04.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:03:04 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:03:04.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:03:04 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T04:03:04.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:03:04 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T04:03:04.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:03:04 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T04:03:04.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:03:04 smithi047 conmon[78521]: 2022-05-14T04:03:04.175+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:03:04.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:03:04 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T04:03:04.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:03:04 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:03:04.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:03:04 smithi047 conmon[78521]: yield 2022-05-14T04:03:04.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:03:04 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:03:04.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:03:04 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:03:04.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:03:04 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:03:04.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:03:04 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T04:03:04.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:03:04 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:03:04.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:03:04 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T04:03:04.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:03:04 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:03:04.547 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:03:04 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T04:03:04.547 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:03:04 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:03:04.547 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:03:04 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T04:03:04.548 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:03:04 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:03:04.548 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:03:04 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T04:03:04.548 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:03:04 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:03:04.548 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:03:04 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:03:04.549 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:03:04 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:03:04.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:03:04 smithi115 conmon[90393]: 2022-05-14T04:03:04.422+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:03:05.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:03:05 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:03:05.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:03:05 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:03:05.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T04:03:05.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:03:05.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T04:03:05.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:03:05.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:03:05.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:03:05.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T04:03:05.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:03:05.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T04:03:05.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:03:05.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T04:03:05.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:03:05.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T04:03:05.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:03:05.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T04:03:05.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:03:05.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:03:05.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:03:05.364 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:03:05 smithi115 ceph-mon[78593]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T04:03:05.365 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:03:05 smithi115 ceph-mon[78593]: pgmap v8610: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:03:05.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:03:05 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:03:05.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:03:05 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:03:05.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T04:03:05.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:03:05.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T04:03:05.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:03:05.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:03:05.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:03:05.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T04:03:05.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:03:05.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T04:03:05.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:03:05.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T04:03:05.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:03:05.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T04:03:05.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:03:05.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T04:03:05.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:03:05.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:03:05.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:03:05.547 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:03:05 smithi047 ceph-mon[77147]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T04:03:05.548 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:03:05 smithi047 ceph-mon[77147]: pgmap v8610: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:03:05.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:03:05 smithi115 conmon[90393]: 2022-05-14T04:03:05.377+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:03:06.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:03:06 smithi115 conmon[90393]: 2022-05-14T04:03:06.343+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:03:06.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:03:06 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:03:06.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:03:06 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:03:07.361 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:03:07 smithi115 conmon[90393]: 2022-05-14T04:03:07.338+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:03:07.362 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:03:07 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:03:07.362 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:03:07 smithi115 ceph-mon[78593]: pgmap v8611: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:03:07.363 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:03:07 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 16520 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:03:07.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:03:07 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:03:07.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:03:07 smithi047 ceph-mon[77147]: pgmap v8611: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:03:07.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:03:07 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 16520 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:03:07.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:03:07 smithi047 conmon[78521]: 2022-05-14T04:03:07.247+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:03:07.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:03:07 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T04:03:07.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:03:07 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:03:07.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:03:07 smithi047 conmon[78521]: yield 2022-05-14T04:03:07.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:03:07 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:03:07.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:03:07 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:03:07.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:03:07 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:03:07.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:03:07 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T04:03:07.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:03:07 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:03:07.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:03:07 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T04:03:07.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:03:07 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:03:07.547 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:03:07 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T04:03:07.547 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:03:07 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:03:07.547 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:03:07 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T04:03:07.547 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:03:07 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:03:07.547 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:03:07 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T04:03:07.548 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:03:07 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:03:07.548 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:03:07 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:03:07.548 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:03:07 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:03:07.548 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:03:07 smithi047 conmon[78521]: 2022-05-14T04:03:07.248+0000 7f533ff78700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T04:03:07.549 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:03:07 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T04:03:07.549 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:03:07 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:03:07.549 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:03:07 smithi047 conmon[78521]: yield 2022-05-14T04:03:07.549 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:03:07 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:03:07.549 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:03:07 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:03:07.550 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:03:07 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:03:07.550 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:03:07 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T04:03:07.550 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:03:07 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:03:07.550 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:03:07 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T04:03:07.551 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:03:07 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:03:07.551 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:03:07 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T04:03:07.551 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:03:07 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:03:07.551 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:03:07 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T04:03:07.551 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:03:07 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:03:07.552 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:03:07 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T04:03:07.552 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:03:07 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:03:07.552 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:03:07 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:03:07.552 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:03:07 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:03:07.552 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:03:07 smithi047 conmon[78521]: 2022-05-14T04:03:07.553 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:03:07 smithi047 conmon[78521]: During handling of the above exception, another exception occurred: 2022-05-14T04:03:07.553 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:03:07 smithi047 conmon[78521]: 2022-05-14T04:03:07.553 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:03:07 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T04:03:07.553 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:03:07 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T04:03:07.554 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:03:07 smithi047 conmon[78521]: return f(*arg) 2022-05-14T04:03:07.554 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:03:07 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T04:03:07.554 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:03:07 smithi047 conmon[78521]: self._write_client_files(client_files, host) 2022-05-14T04:03:07.554 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:03:07 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T04:03:07.554 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:03:07 smithi047 conmon[78521]: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T04:03:07.555 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:03:07 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T04:03:07.555 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:03:07 smithi047 conmon[78521]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T04:03:07.555 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:03:07 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T04:03:07.555 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:03:07 smithi047 conmon[78521]: return self.event_loop.get_result(coro) 2022-05-14T04:03:07.556 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:03:07 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T04:03:07.556 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:03:07 smithi047 conmon[78521]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T04:03:07.556 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:03:07 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T04:03:07.556 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:03:07 smithi047 conmon[78521]: return self.__get_result() 2022-05-14T04:03:07.556 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:03:07 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T04:03:07.557 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:03:07 smithi047 conmon[78521]: raise self._exception 2022-05-14T04:03:07.557 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:03:07 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T04:03:07.557 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:03:07 smithi047 conmon[78521]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T04:03:07.557 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:03:07 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T04:03:07.557 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:03:07 smithi047 conmon[78521]: conn = await self._remote_connection(host, addr) 2022-05-14T04:03:07.558 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:03:07 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T04:03:07.558 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:03:07 smithi047 conmon[78521]: raise 2022-05-14T04:03:07.558 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:03:07 smithi047 conmon[78521]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T04:03:07.558 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:03:07 smithi047 conmon[78521]: self.gen.throw(type, value, traceback) 2022-05-14T04:03:07.559 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:03:07 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T04:03:07.559 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:03:07 smithi047 conmon[78521]: raise OrchestratorError(msg) 2022-05-14T04:03:07.559 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:03:07 smithi047 conmon[78521]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:03:08.353 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:03:08 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:03:08.379 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:03:08 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:03:08.379 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T04:03:08.380 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:03:08.380 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T04:03:08.380 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:03:08.380 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:03:08.380 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:03:08.381 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T04:03:08.381 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:03:08.381 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T04:03:08.382 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:03:08.382 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T04:03:08.382 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:03:08.382 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T04:03:08.383 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:03:08.383 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T04:03:08.383 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:03:08.383 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:03:08.384 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:03:08.384 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:03:08 smithi115 ceph-mon[78593]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T04:03:08.384 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T04:03:08.385 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:03:08.385 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T04:03:08.385 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:03:08.385 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:03:08.385 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:03:08.386 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T04:03:08.386 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:03:08.386 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T04:03:08.386 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:03:08.387 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T04:03:08.387 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:03:08.387 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T04:03:08.387 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:03:08.388 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T04:03:08.388 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:03:08.388 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:03:08.388 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:03:08.389 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T04:03:08.389 INFO:journalctl@ceph.mon.c.smithi115.stdout: During handling of the above exception, another exception occurred: 2022-05-14T04:03:08.389 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T04:03:08.390 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T04:03:08.390 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T04:03:08.390 INFO:journalctl@ceph.mon.c.smithi115.stdout: return f(*arg) 2022-05-14T04:03:08.390 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T04:03:08.391 INFO:journalctl@ceph.mon.c.smithi115.stdout: self._write_client_files(client_files, host) 2022-05-14T04:03:08.391 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T04:03:08.391 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T04:03:08.391 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T04:03:08.392 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T04:03:08.392 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T04:03:08.392 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.event_loop.get_result(coro) 2022-05-14T04:03:08.392 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T04:03:08.392 INFO:journalctl@ceph.mon.c.smithi115.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T04:03:08.393 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T04:03:08.393 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.__get_result() 2022-05-14T04:03:08.393 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T04:03:08.393 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise self._exception 2022-05-14T04:03:08.394 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T04:03:08.394 INFO:journalctl@ceph.mon.c.smithi115.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T04:03:08.394 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T04:03:08.394 INFO:journalctl@ceph.mon.c.smithi115.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T04:03:08.395 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T04:03:08.395 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise 2022-05-14T04:03:08.395 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T04:03:08.395 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.gen.throw(type, value, traceback) 2022-05-14T04:03:08.396 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T04:03:08.396 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OrchestratorError(msg) 2022-05-14T04:03:08.396 INFO:journalctl@ceph.mon.c.smithi115.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:03:08.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:03:08 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:03:08.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:03:08 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:03:08.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T04:03:08.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:03:08.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T04:03:08.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:03:08.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:03:08.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:03:08.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T04:03:08.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:03:08.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T04:03:08.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:03:08.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T04:03:08.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:03:08.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T04:03:08.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:03:08.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T04:03:08.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:03:08.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:03:08.548 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:03:08.548 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:03:08 smithi047 ceph-mon[77147]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T04:03:08.548 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T04:03:08.548 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:03:08.548 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T04:03:08.549 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:03:08.549 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:03:08.550 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:03:08.550 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T04:03:08.550 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:03:08.550 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T04:03:08.551 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:03:08.551 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T04:03:08.551 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:03:08.554 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T04:03:08.554 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:03:08.554 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T04:03:08.555 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:03:08.555 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:03:08.555 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:03:08.555 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T04:03:08.555 INFO:journalctl@ceph.mon.b.smithi047.stdout: During handling of the above exception, another exception occurred: 2022-05-14T04:03:08.556 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T04:03:08.556 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T04:03:08.556 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T04:03:08.556 INFO:journalctl@ceph.mon.b.smithi047.stdout: return f(*arg) 2022-05-14T04:03:08.557 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T04:03:08.557 INFO:journalctl@ceph.mon.b.smithi047.stdout: self._write_client_files(client_files, host) 2022-05-14T04:03:08.557 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T04:03:08.558 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T04:03:08.558 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T04:03:08.558 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T04:03:08.558 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T04:03:08.559 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.event_loop.get_result(coro) 2022-05-14T04:03:08.559 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T04:03:08.559 INFO:journalctl@ceph.mon.b.smithi047.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T04:03:08.559 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T04:03:08.560 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.__get_result() 2022-05-14T04:03:08.560 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T04:03:08.560 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise self._exception 2022-05-14T04:03:08.560 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T04:03:08.560 INFO:journalctl@ceph.mon.b.smithi047.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T04:03:08.561 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T04:03:08.561 INFO:journalctl@ceph.mon.b.smithi047.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T04:03:08.561 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T04:03:08.561 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise 2022-05-14T04:03:08.562 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T04:03:08.562 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.gen.throw(type, value, traceback) 2022-05-14T04:03:08.562 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T04:03:08.562 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OrchestratorError(msg) 2022-05-14T04:03:08.563 INFO:journalctl@ceph.mon.b.smithi047.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:03:08.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:03:08 smithi115 conmon[90393]: 2022-05-14T04:03:08.350+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:03:09.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:03:09 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:03:09.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:03:09 smithi115 ceph-mon[78593]: pgmap v8612: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:03:09.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:03:09 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:03:09.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:03:09 smithi047 ceph-mon[77147]: pgmap v8612: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:03:09.860 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:03:09 smithi115 conmon[90393]: 2022-05-14T04:03:09.384+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:03:10.546 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:03:10 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:03:10.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:03:10 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:03:10.611 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:03:10 smithi115 conmon[90393]: 2022-05-14T04:03:10.341+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:03:11.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:03:11 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:03:11.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:03:11 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:03:11.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:03:11 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:03:11.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:03:11 smithi047 ceph-mon[77147]: pgmap v8613: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:03:11.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:03:11 smithi115 conmon[90393]: 2022-05-14T04:03:11.327+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:03:11.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:03:11 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:03:11.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:03:11 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:03:11.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:03:11 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:03:11.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:03:11 smithi115 ceph-mon[78593]: pgmap v8613: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:03:12.377 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:03:12 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:03:12.377 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:03:12 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 16525 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:03:12.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:03:12 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:03:12.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:03:12 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 16525 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:03:12.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:03:12 smithi115 conmon[90393]: 2022-05-14T04:03:12.374+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:03:13.419 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:03:13 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:03:13.419 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:03:13 smithi115 ceph-mon[78593]: pgmap v8614: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:03:13.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:03:13 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:03:13.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:03:13 smithi047 ceph-mon[77147]: pgmap v8614: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:03:13.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:03:13 smithi115 conmon[90393]: 2022-05-14T04:03:13.416+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:03:14.403 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:03:14 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:03:14.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:03:14 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:03:14.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:03:14 smithi115 conmon[90393]: 2022-05-14T04:03:14.400+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:03:15.424 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:03:15 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:03:15.424 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:03:15 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:03:15.424 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:03:15 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:03:15.425 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:03:15 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T04:03:15.425 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:03:15 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T04:03:15.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:03:15 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:03:15.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:03:15 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:03:15.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:03:15 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:03:15.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:03:15 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T04:03:15.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:03:15 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T04:03:15.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:03:15 smithi115 conmon[90393]: 2022-05-14T04:03:15.420+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:03:16.434 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:03:16 smithi115 ceph-mon[78593]: pgmap v8615: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:03:16.434 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:03:16 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:03:16.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:03:16 smithi047 ceph-mon[77147]: pgmap v8615: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:03:16.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:03:16 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:03:16.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:03:16 smithi115 conmon[90393]: 2022-05-14T04:03:16.431+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:03:17.404 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:03:17 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:03:17.405 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:03:17 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 16530 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:03:17.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:03:17 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:03:17.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:03:17 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 16530 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:03:17.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:03:17 smithi115 conmon[90393]: 2022-05-14T04:03:17.401+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:03:18.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:03:18 smithi047 ceph-mon[77147]: pgmap v8616: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:03:18.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:03:18 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:03:18.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:03:18 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:03:18.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:03:18 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:03:18.610 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:03:18 smithi115 conmon[90393]: 2022-05-14T04:03:18.378+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:03:18.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:03:18 smithi115 ceph-mon[78593]: pgmap v8616: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:03:18.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:03:18 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:03:18.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:03:18 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:03:18.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:03:18 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:03:19.401 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:03:19 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:03:19.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:03:19 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:03:19.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:03:19 smithi115 conmon[90393]: 2022-05-14T04:03:19.397+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:03:20.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:03:20 smithi047 ceph-mon[77147]: pgmap v8617: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:03:20.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:03:20 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:03:20.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:03:20 smithi115 conmon[90393]: 2022-05-14T04:03:20.384+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:03:20.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:03:20 smithi115 ceph-mon[78593]: pgmap v8617: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:03:20.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:03:20 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:03:21.410 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:03:21 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:03:21.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:03:21 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:03:21.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:03:21 smithi115 conmon[90393]: 2022-05-14T04:03:21.407+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:03:22.446 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:03:22 smithi115 ceph-mon[78593]: pgmap v8618: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:03:22.447 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:03:22 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:03:22.447 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:03:22 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:03:22.447 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:03:22 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:03:22.447 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:03:22 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 16535 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:03:22.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:03:22 smithi047 ceph-mon[77147]: pgmap v8618: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:03:22.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:03:22 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:03:22.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:03:22 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:03:22.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:03:22 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:03:22.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:03:22 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 16535 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:03:22.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:03:22 smithi115 conmon[90393]: 2022-05-14T04:03:22.443+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:03:23.462 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:03:23 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:03:23.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:03:23 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:03:23.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:03:23 smithi115 conmon[90393]: 2022-05-14T04:03:23.459+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:03:24.449 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:03:24 smithi115 ceph-mon[78593]: pgmap v8619: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:03:24.450 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:03:24 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:03:24.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:03:24 smithi047 ceph-mon[77147]: pgmap v8619: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:03:24.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:03:24 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:03:24.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:03:24 smithi115 conmon[90393]: 2022-05-14T04:03:24.446+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:03:25.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:03:25 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:03:25.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:03:25 smithi115 conmon[90393]: 2022-05-14T04:03:25.403+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:03:25.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:03:25 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:03:26.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:03:26 smithi047 ceph-mon[77147]: pgmap v8620: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:03:26.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:03:26 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:03:26.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:03:26 smithi115 conmon[90393]: 2022-05-14T04:03:26.404+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:03:26.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:03:26 smithi115 ceph-mon[78593]: pgmap v8620: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:03:26.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:03:26 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:03:27.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:03:27 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:03:27.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:03:27 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 16540 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:03:27.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:03:27 smithi115 conmon[90393]: 2022-05-14T04:03:27.357+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:03:27.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:03:27 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:03:27.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:03:27 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 16540 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:03:28.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:03:28 smithi047 ceph-mon[77147]: pgmap v8621: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:03:28.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:03:28 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:03:28.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:03:28 smithi115 conmon[90393]: 2022-05-14T04:03:28.397+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:03:28.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:03:28 smithi115 ceph-mon[78593]: pgmap v8621: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:03:28.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:03:28 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:03:29.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:03:29 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:03:29.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:03:29 smithi115 conmon[90393]: 2022-05-14T04:03:29.413+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:03:29.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:03:29 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:03:30.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:03:30 smithi047 ceph-mon[77147]: pgmap v8622: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:03:30.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:03:30 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:03:30.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:03:30 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:03:30.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:03:30 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:03:30.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:03:30 smithi115 conmon[90393]: 2022-05-14T04:03:30.422+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:03:30.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:03:30 smithi115 ceph-mon[78593]: pgmap v8622: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:03:30.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:03:30 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:03:30.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:03:30 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:03:30.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:03:30 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:03:31.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:03:31 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:03:31.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:03:31 smithi115 conmon[90393]: 2022-05-14T04:03:31.374+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:03:31.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:03:31 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:03:32.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:03:32 smithi047 ceph-mon[77147]: pgmap v8623: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 7 op/s 2022-05-14T04:03:32.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:03:32 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:03:32.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:03:32 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 16545 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:03:32.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:03:32 smithi115 conmon[90393]: 2022-05-14T04:03:32.340+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:03:32.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:03:32 smithi115 ceph-mon[78593]: pgmap v8623: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 7 op/s 2022-05-14T04:03:32.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:03:32 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:03:32.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:03:32 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 16545 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:03:33.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:03:33 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:03:33.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:03:33 smithi115 conmon[90393]: 2022-05-14T04:03:33.336+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:03:33.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:03:33 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:03:34.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:03:34 smithi047 ceph-mon[77147]: pgmap v8624: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 4 op/s 2022-05-14T04:03:34.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:03:34 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:03:34.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:03:34 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:03:34.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:03:34 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:03:34.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:03:34 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:03:34.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:03:34 smithi115 conmon[90393]: 2022-05-14T04:03:34.359+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:03:34.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:03:34 smithi115 ceph-mon[78593]: pgmap v8624: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 4 op/s 2022-05-14T04:03:34.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:03:34 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:03:34.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:03:34 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:03:34.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:03:34 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:03:34.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:03:34 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:03:35.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:03:35 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:03:35.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:03:35 smithi115 conmon[90393]: 2022-05-14T04:03:35.376+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:03:35.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:03:35 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:03:36.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:03:36 smithi047 ceph-mon[77147]: pgmap v8625: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 7 op/s 2022-05-14T04:03:36.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:03:36 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:03:36.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:03:36 smithi115 conmon[90393]: 2022-05-14T04:03:36.422+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:03:36.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:03:36 smithi115 ceph-mon[78593]: pgmap v8625: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 7 op/s 2022-05-14T04:03:36.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:03:36 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:03:37.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:03:37 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:03:37.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:03:37 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 16550 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:03:37.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:03:37 smithi115 conmon[90393]: 2022-05-14T04:03:37.388+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:03:37.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:03:37 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:03:37.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:03:37 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 16550 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:03:38.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:03:38 smithi047 ceph-mon[77147]: pgmap v8626: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 4 op/s 2022-05-14T04:03:38.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:03:38 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:03:38.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:03:38 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:03:38.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:03:38 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:03:38.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:03:38 smithi115 conmon[90393]: 2022-05-14T04:03:38.358+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:03:38.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:03:38 smithi115 ceph-mon[78593]: pgmap v8626: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 4 op/s 2022-05-14T04:03:38.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:03:38 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:03:38.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:03:38 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:03:38.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:03:38 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:03:39.545 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:03:39 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:03:39.611 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:03:39 smithi115 conmon[90393]: 2022-05-14T04:03:39.335+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:03:39.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:03:39 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:03:40.547 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:03:40 smithi047 ceph-mon[77147]: pgmap v8627: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 4 op/s 2022-05-14T04:03:40.547 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:03:40 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:03:40.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:03:40 smithi115 conmon[90393]: 2022-05-14T04:03:40.304+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:03:40.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:03:40 smithi115 ceph-mon[78593]: pgmap v8627: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 4 op/s 2022-05-14T04:03:40.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:03:40 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:03:41.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:03:41 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:03:41.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:03:41 smithi115 conmon[90393]: 2022-05-14T04:03:41.259+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:03:41.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:03:41 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:03:42.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:03:42 smithi047 ceph-mon[77147]: pgmap v8628: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 7 op/s 2022-05-14T04:03:42.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:03:42 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:03:42.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:03:42 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:03:42.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:03:42 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:03:42.545 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:03:42 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 16555 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:03:42.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:03:42 smithi115 conmon[90393]: 2022-05-14T04:03:42.212+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:03:42.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:03:42 smithi115 ceph-mon[78593]: pgmap v8628: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 7 op/s 2022-05-14T04:03:42.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:03:42 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:03:42.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:03:42 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:03:42.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:03:42 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:03:42.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:03:42 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 16555 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:03:43.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:03:43 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:03:43.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:03:43 smithi115 conmon[90393]: 2022-05-14T04:03:43.216+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:03:43.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:03:43 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:03:44.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:03:44 smithi047 ceph-mon[77147]: pgmap v8629: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:03:44.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:03:44 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:03:44.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:03:44 smithi115 conmon[90393]: 2022-05-14T04:03:44.190+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:03:44.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:03:44 smithi115 ceph-mon[78593]: pgmap v8629: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:03:44.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:03:44 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:03:45.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:03:45 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:03:45.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:03:45 smithi115 conmon[90393]: 2022-05-14T04:03:45.226+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:03:45.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:03:45 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:03:46.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:03:46 smithi047 ceph-mon[77147]: pgmap v8630: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:03:46.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:03:46 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:03:46.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:03:46 smithi115 conmon[90393]: 2022-05-14T04:03:46.194+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:03:46.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:03:46 smithi115 ceph-mon[78593]: pgmap v8630: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:03:46.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:03:46 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:03:47.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:03:47 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:03:47.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:03:47 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 16560 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:03:47.611 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:03:47 smithi115 conmon[90393]: 2022-05-14T04:03:47.191+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:03:47.612 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:03:47 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:03:47.612 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:03:47 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 16560 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:03:48.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:03:48 smithi047 ceph-mon[77147]: pgmap v8631: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:03:48.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:03:48 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:03:48.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:03:48 smithi115 conmon[90393]: 2022-05-14T04:03:48.224+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:03:48.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:03:48 smithi115 ceph-mon[78593]: pgmap v8631: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:03:48.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:03:48 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:03:49.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:03:49 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:03:49.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:03:49 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:03:49.610 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:03:49 smithi115 conmon[90393]: 2022-05-14T04:03:49.236+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:03:50.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:03:50 smithi047 ceph-mon[77147]: pgmap v8632: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:03:50.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:03:50 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:03:50.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:03:50 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:03:50.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:03:50 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:03:50.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:03:50 smithi115 conmon[90393]: 2022-05-14T04:03:50.211+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:03:50.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:03:50 smithi115 ceph-mon[78593]: pgmap v8632: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:03:50.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:03:50 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:03:50.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:03:50 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:03:50.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:03:50 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:03:51.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:03:51 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:03:51.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:03:51 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:03:51.610 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:03:51 smithi115 conmon[90393]: 2022-05-14T04:03:51.244+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:03:52.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:03:52 smithi047 ceph-mon[77147]: pgmap v8633: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:03:52.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:03:52 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:03:52.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:03:52 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 16565 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:03:52.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:03:52 smithi115 conmon[90393]: 2022-05-14T04:03:52.252+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:03:52.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:03:52 smithi115 ceph-mon[78593]: pgmap v8633: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:03:52.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:03:52 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:03:52.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:03:52 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 16565 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:03:53.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:03:53 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:03:53.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:03:53 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:03:53.610 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:03:53 smithi115 conmon[90393]: 2022-05-14T04:03:53.272+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:03:54.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:03:54 smithi047 ceph-mon[77147]: pgmap v8634: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:03:54.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:03:54 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:03:54.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:03:54 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:03:54.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:03:54 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:03:54.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:03:54 smithi115 conmon[90393]: 2022-05-14T04:03:54.247+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:03:54.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:03:54 smithi115 ceph-mon[78593]: pgmap v8634: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:03:54.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:03:54 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:03:54.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:03:54 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:03:54.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:03:54 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:03:55.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:03:55 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:03:55.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:03:55 smithi115 conmon[90393]: 2022-05-14T04:03:55.265+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:03:55.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:03:55 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:03:56.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:03:56 smithi047 ceph-mon[77147]: pgmap v8635: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:03:56.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:03:56 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:03:56.610 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:03:56 smithi115 conmon[90393]: 2022-05-14T04:03:56.217+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:03:56.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:03:56 smithi115 ceph-mon[78593]: pgmap v8635: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:03:56.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:03:56 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:03:57.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:03:57 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:03:57.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:03:57 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 16570 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:03:57.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:03:57 smithi115 conmon[90393]: 2022-05-14T04:03:57.243+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:03:57.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:03:57 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:03:57.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:03:57 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 16570 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:03:58.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:03:58 smithi047 ceph-mon[77147]: pgmap v8636: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:03:58.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:03:58 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:03:58.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:03:58 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:03:58.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:03:58 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:03:58.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:03:58 smithi115 conmon[90393]: 2022-05-14T04:03:58.235+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:03:58.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:03:58 smithi115 ceph-mon[78593]: pgmap v8636: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:03:58.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:03:58 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:03:58.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:03:58 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:03:58.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:03:58 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:03:59.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:03:59 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:03:59.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:03:59 smithi115 conmon[90393]: 2022-05-14T04:03:59.220+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:03:59.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:03:59 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:04:00.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:04:00 smithi047 ceph-mon[77147]: pgmap v8637: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:04:00.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:04:00 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:04:00.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:04:00 smithi115 conmon[90393]: 2022-05-14T04:04:00.234+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:04:00.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:04:00 smithi115 ceph-mon[78593]: pgmap v8637: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:04:00.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:04:00 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:04:01.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:04:01 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:04:01.610 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:04:01 smithi115 conmon[90393]: 2022-05-14T04:04:01.259+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:04:01.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:04:01 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:04:02.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:04:02 smithi047 ceph-mon[77147]: pgmap v8638: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:04:02.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:04:02 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:04:02.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:04:02 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:04:02.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:04:02 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:04:02.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:04:02 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 16575 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:04:02.610 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:04:02 smithi115 conmon[90393]: 2022-05-14T04:04:02.265+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:04:02.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:04:02 smithi115 ceph-mon[78593]: pgmap v8638: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:04:02.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:04:02 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:04:02.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:04:02 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:04:02.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:04:02 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:04:02.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:04:02 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 16575 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:04:03.545 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:04:03 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:04:03.610 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:04:03 smithi115 conmon[90393]: 2022-05-14T04:04:03.253+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:04:03.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:04:03 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:04:04.610 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:04:04 smithi115 conmon[90393]: 2022-05-14T04:04:04.275+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:04:04.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:04:04 smithi115 ceph-mon[78593]: pgmap v8639: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:04:04.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:04:04 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:04:04.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:04:04 smithi047 ceph-mon[77147]: pgmap v8639: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:04:04.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:04:04 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:04:05.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:04:05 smithi115 conmon[90393]: 2022-05-14T04:04:05.226+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:04:05.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:04:05 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:04:05.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:04:05 smithi115 ceph-mon[78593]: pgmap v8640: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:04:05.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:04:05 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:04:05.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:04:05 smithi047 ceph-mon[77147]: pgmap v8640: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:04:06.610 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:04:06 smithi115 conmon[90393]: 2022-05-14T04:04:06.276+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:04:06.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:04:06 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:04:06.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:04:06 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:04:07.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:04:07 smithi115 conmon[90393]: 2022-05-14T04:04:07.309+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:04:07.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:04:07 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:04:07.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:04:07 smithi115 ceph-mon[78593]: pgmap v8641: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:04:07.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:04:07 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 16580 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:04:07.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:04:07 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T04:04:07.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:04:07 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T04:04:07.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:04:07 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T04:04:07.772 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:04:07 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:04:07.773 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:04:07 smithi047 ceph-mon[77147]: pgmap v8641: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:04:07.773 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:04:07 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 16580 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:04:07.774 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:04:07 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T04:04:07.774 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:04:07 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T04:04:07.774 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:04:07 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T04:04:08.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:04:08 smithi115 conmon[90393]: 2022-05-14T04:04:08.328+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:04:08.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:04:08 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:04:08.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:04:08 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:04:09.610 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:04:09 smithi115 conmon[90393]: 2022-05-14T04:04:09.285+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:04:09.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:04:09 smithi115 ceph-mon[78593]: pgmap v8642: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:04:09.612 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:04:09 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:04:09.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:04:09 smithi047 ceph-mon[77147]: pgmap v8642: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:04:09.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:04:09 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:04:10.610 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:04:10 smithi115 conmon[90393]: 2022-05-14T04:04:10.275+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:04:10.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:04:10 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:04:10.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:04:10 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:04:10.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:04:10 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:04:10.795 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:04:10 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:04:10.795 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:04:10 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:04:10.796 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:04:10 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:04:10.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:04:10 smithi047 conmon[78521]: 2022-05-14T04:04:10.352+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:04:10.797 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:04:10 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T04:04:10.797 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:04:10 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:04:10.798 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:04:10 smithi047 conmon[78521]: yield 2022-05-14T04:04:10.798 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:04:10 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:04:10.798 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:04:10 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:04:10.798 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:04:10 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:04:10.798 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:04:10 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T04:04:10.799 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:04:10 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:04:10.799 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:04:10 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T04:04:10.799 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:04:10 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:04:10.800 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:04:10 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T04:04:10.800 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:04:10 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:04:10.800 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:04:10 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T04:04:10.800 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:04:10 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:04:10.802 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:04:10 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T04:04:10.803 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:04:10 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:04:10.803 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:04:10 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:04:10.804 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:04:10 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:04:11.610 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:04:11 smithi115 conmon[90393]: 2022-05-14T04:04:11.315+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:04:11.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:04:11 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:04:11.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:04:11 smithi115 ceph-mon[78593]: pgmap v8643: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:04:11.612 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:04:11 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:04:11.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T04:04:11.613 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:04:11.613 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T04:04:11.613 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:04:11.614 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:04:11.614 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:04:11.614 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T04:04:11.615 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:04:11.615 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T04:04:11.616 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:04:11.616 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T04:04:11.617 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:04:11.617 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T04:04:11.618 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:04:11.618 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T04:04:11.619 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:04:11.620 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:04:11.621 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:04:11.621 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:04:11 smithi115 ceph-mon[78593]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T04:04:11.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:04:11 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:04:11.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:04:11 smithi047 ceph-mon[77147]: pgmap v8643: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:04:11.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:04:11 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:04:11.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T04:04:11.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:04:11.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T04:04:11.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:04:11.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:04:11.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:04:11.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T04:04:11.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:04:11.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T04:04:11.797 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:04:11.797 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T04:04:11.797 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:04:11.797 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T04:04:11.798 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:04:11.798 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T04:04:11.798 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:04:11.798 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:04:11.798 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:04:11.799 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:04:11 smithi047 ceph-mon[77147]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T04:04:12.610 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:04:12 smithi115 conmon[90393]: 2022-05-14T04:04:12.308+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:04:12.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:04:12 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:04:12.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:04:12 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 16585 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:04:12.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:04:12 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:04:12.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:04:12 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 16585 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:04:13.432 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:04:13 smithi047 conmon[78521]: 2022-05-14T04:04:13.424+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:04:13.432 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:04:13 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T04:04:13.432 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:04:13 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:04:13.432 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:04:13 smithi047 conmon[78521]: yield 2022-05-14T04:04:13.433 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:04:13 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:04:13.433 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:04:13 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:04:13.433 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:04:13 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:04:13.433 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:04:13 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T04:04:13.434 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:04:13 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:04:13.434 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:04:13 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T04:04:13.434 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:04:13 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:04:13.434 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:04:13 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T04:04:13.435 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:04:13 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:04:13.435 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:04:13 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T04:04:13.435 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:04:13 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:04:13.435 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:04:13 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T04:04:13.435 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:04:13 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:04:13.436 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:04:13 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:04:13.436 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:04:13 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:04:13.436 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:04:13 smithi047 conmon[78521]: 2022-05-14T04:04:13.425+0000 7f5344f82700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T04:04:13.436 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:04:13 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T04:04:13.436 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:04:13 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:04:13.437 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:04:13 smithi047 conmon[78521]: yield 2022-05-14T04:04:13.437 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:04:13 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:04:13.437 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:04:13 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:04:13.437 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:04:13 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:04:13.438 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:04:13 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T04:04:13.438 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:04:13 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:04:13.438 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:04:13 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T04:04:13.438 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:04:13 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:04:13.438 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:04:13 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T04:04:13.439 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:04:13 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:04:13.439 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:04:13 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T04:04:13.439 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:04:13 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:04:13.439 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:04:13 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T04:04:13.439 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:04:13 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:04:13.440 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:04:13 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:04:13.440 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:04:13 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:04:13.440 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:04:13 smithi047 conmon[78521]: 2022-05-14T04:04:13.440 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:04:13 smithi047 conmon[78521]: During handling of the above exception, another exception occurred: 2022-05-14T04:04:13.440 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:04:13 smithi047 conmon[78521]: 2022-05-14T04:04:13.441 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:04:13 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T04:04:13.441 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:04:13 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T04:04:13.441 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:04:13 smithi047 conmon[78521]: return f(*arg) 2022-05-14T04:04:13.441 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:04:13 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T04:04:13.442 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:04:13 smithi047 conmon[78521]: self._write_client_files(client_files, host) 2022-05-14T04:04:13.442 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:04:13 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T04:04:13.442 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:04:13 smithi047 conmon[78521]: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T04:04:13.442 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:04:13 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T04:04:13.442 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:04:13 smithi047 conmon[78521]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T04:04:13.443 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:04:13 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T04:04:13.443 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:04:13 smithi047 conmon[78521]: return self.event_loop.get_result(coro) 2022-05-14T04:04:13.443 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:04:13 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T04:04:13.443 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:04:13 smithi047 conmon[78521]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T04:04:13.443 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:04:13 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T04:04:13.444 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:04:13 smithi047 conmon[78521]: return self.__get_result() 2022-05-14T04:04:13.444 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:04:13 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T04:04:13.444 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:04:13 smithi047 conmon[78521]: raise self._exception 2022-05-14T04:04:13.444 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:04:13 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T04:04:13.444 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:04:13 smithi047 conmon[78521]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T04:04:13.445 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:04:13 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T04:04:13.445 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:04:13 smithi047 conmon[78521]: conn = await self._remote_connection(host, addr) 2022-05-14T04:04:13.445 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:04:13 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T04:04:13.445 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:04:13 smithi047 conmon[78521]: raise 2022-05-14T04:04:13.445 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:04:13 smithi047 conmon[78521]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T04:04:13.448 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:04:13 smithi047 conmon[78521]: self.gen.throw(type, value, traceback) 2022-05-14T04:04:13.449 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:04:13 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T04:04:13.449 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:04:13 smithi047 conmon[78521]: raise OrchestratorError(msg) 2022-05-14T04:04:13.449 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:04:13 smithi047 conmon[78521]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:04:13.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:04:13 smithi115 conmon[90393]: 2022-05-14T04:04:13.269+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:04:14.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:04:14 smithi115 conmon[90393]: 2022-05-14T04:04:14.238+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:04:14.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:04:14 smithi115 ceph-mon[78593]: pgmap v8644: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:04:14.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:04:14 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:04:14.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:04:14 smithi047 ceph-mon[77147]: pgmap v8644: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:04:14.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:04:14 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:04:15.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:04:15 smithi115 conmon[90393]: 2022-05-14T04:04:15.228+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:04:15.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:04:15 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:04:15.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:04:15 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:04:15.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T04:04:15.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:04:15.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T04:04:15.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:04:15.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:04:15.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:04:15.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T04:04:15.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:04:15.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T04:04:15.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:04:15.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T04:04:15.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:04:15.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T04:04:15.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:04:15.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T04:04:15.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:04:15.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:04:15.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:04:15.365 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:04:15 smithi115 ceph-mon[78593]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T04:04:15.365 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T04:04:15.365 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:04:15.365 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T04:04:15.366 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:04:15.366 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:04:15.366 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:04:15.366 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T04:04:15.366 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:04:15.367 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T04:04:15.367 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:04:15.367 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T04:04:15.368 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:04:15.368 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T04:04:15.368 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:04:15.368 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T04:04:15.369 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:04:15.369 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:04:15.369 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:04:15.369 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T04:04:15.370 INFO:journalctl@ceph.mon.c.smithi115.stdout: During handling of the above exception, another exception occurred: 2022-05-14T04:04:15.370 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T04:04:15.370 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T04:04:15.370 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T04:04:15.371 INFO:journalctl@ceph.mon.c.smithi115.stdout: return f(*arg) 2022-05-14T04:04:15.371 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T04:04:15.371 INFO:journalctl@ceph.mon.c.smithi115.stdout: self._write_client_files(client_files, host) 2022-05-14T04:04:15.371 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T04:04:15.372 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T04:04:15.372 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T04:04:15.372 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T04:04:15.373 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T04:04:15.373 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.event_loop.get_result(coro) 2022-05-14T04:04:15.373 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T04:04:15.374 INFO:journalctl@ceph.mon.c.smithi115.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T04:04:15.374 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T04:04:15.375 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.__get_result() 2022-05-14T04:04:15.375 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T04:04:15.375 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise self._exception 2022-05-14T04:04:15.376 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T04:04:15.376 INFO:journalctl@ceph.mon.c.smithi115.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T04:04:15.376 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T04:04:15.377 INFO:journalctl@ceph.mon.c.smithi115.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T04:04:15.377 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T04:04:15.378 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise 2022-05-14T04:04:15.378 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T04:04:15.379 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.gen.throw(type, value, traceback) 2022-05-14T04:04:15.379 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T04:04:15.379 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OrchestratorError(msg) 2022-05-14T04:04:15.380 INFO:journalctl@ceph.mon.c.smithi115.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:04:15.380 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:04:15 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:04:15.380 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:04:15 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:04:15.381 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:04:15 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T04:04:15.381 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:04:15 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T04:04:15.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:04:15 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:04:15.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:04:15 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:04:15.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T04:04:15.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:04:15.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T04:04:15.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:04:15.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:04:15.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:04:15.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T04:04:15.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:04:15.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T04:04:15.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:04:15.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T04:04:15.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:04:15.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T04:04:15.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:04:15.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T04:04:15.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:04:15.548 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:04:15.548 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:04:15.548 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:04:15 smithi047 ceph-mon[77147]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T04:04:15.548 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T04:04:15.549 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:04:15.549 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T04:04:15.549 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:04:15.550 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:04:15.550 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:04:15.550 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T04:04:15.550 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:04:15.550 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T04:04:15.551 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:04:15.551 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T04:04:15.551 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:04:15.551 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T04:04:15.552 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:04:15.552 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T04:04:15.552 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:04:15.553 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:04:15.553 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:04:15.553 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T04:04:15.554 INFO:journalctl@ceph.mon.b.smithi047.stdout: During handling of the above exception, another exception occurred: 2022-05-14T04:04:15.554 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T04:04:15.555 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T04:04:15.555 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T04:04:15.555 INFO:journalctl@ceph.mon.b.smithi047.stdout: return f(*arg) 2022-05-14T04:04:15.556 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T04:04:15.556 INFO:journalctl@ceph.mon.b.smithi047.stdout: self._write_client_files(client_files, host) 2022-05-14T04:04:15.556 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T04:04:15.557 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T04:04:15.557 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T04:04:15.557 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T04:04:15.558 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T04:04:15.558 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.event_loop.get_result(coro) 2022-05-14T04:04:15.558 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T04:04:15.559 INFO:journalctl@ceph.mon.b.smithi047.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T04:04:15.559 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T04:04:15.560 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.__get_result() 2022-05-14T04:04:15.560 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T04:04:15.560 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise self._exception 2022-05-14T04:04:15.561 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T04:04:15.561 INFO:journalctl@ceph.mon.b.smithi047.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T04:04:15.561 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T04:04:15.562 INFO:journalctl@ceph.mon.b.smithi047.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T04:04:15.562 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T04:04:15.562 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise 2022-05-14T04:04:15.563 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T04:04:15.563 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.gen.throw(type, value, traceback) 2022-05-14T04:04:15.563 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T04:04:15.564 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OrchestratorError(msg) 2022-05-14T04:04:15.564 INFO:journalctl@ceph.mon.b.smithi047.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:04:15.564 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:04:15 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:04:15.565 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:04:15 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:04:15.565 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:04:15 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T04:04:15.566 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:04:15 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T04:04:16.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:04:16 smithi115 conmon[90393]: 2022-05-14T04:04:16.269+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:04:16.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:04:16 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:04:16.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:04:16 smithi115 ceph-mon[78593]: pgmap v8645: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:04:16.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:04:16 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:04:16.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:04:16 smithi047 ceph-mon[77147]: pgmap v8645: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:04:17.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:04:17 smithi115 conmon[90393]: 2022-05-14T04:04:17.278+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:04:17.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:04:17 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:04:17.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:04:17 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 16590 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:04:17.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:04:17 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:04:17.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:04:17 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 16590 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:04:18.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:04:18 smithi115 conmon[90393]: 2022-05-14T04:04:18.240+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:04:18.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:04:18 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:04:18.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:04:18 smithi115 ceph-mon[78593]: pgmap v8646: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:04:18.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:04:18 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:04:18.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:04:18 smithi047 ceph-mon[77147]: pgmap v8646: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:04:19.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:04:19 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:04:19.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:04:19 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:04:19.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:04:19 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:04:19.610 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:04:19 smithi115 conmon[90393]: 2022-05-14T04:04:19.242+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:04:19.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:04:19 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:04:19.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:04:19 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:04:19.612 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:04:19 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:04:20.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:04:20 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:04:20.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:04:20 smithi047 ceph-mon[77147]: pgmap v8647: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:04:20.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:04:20 smithi115 conmon[90393]: 2022-05-14T04:04:20.206+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:04:20.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:04:20 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:04:20.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:04:20 smithi115 ceph-mon[78593]: pgmap v8647: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:04:21.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:04:21 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:04:21.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:04:21 smithi115 conmon[90393]: 2022-05-14T04:04:21.252+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:04:21.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:04:21 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:04:22.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:04:22 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:04:22.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:04:22 smithi047 ceph-mon[77147]: pgmap v8648: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:04:22.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:04:22 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 16595 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:04:22.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:04:22 smithi115 conmon[90393]: 2022-05-14T04:04:22.210+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:04:22.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:04:22 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:04:22.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:04:22 smithi115 ceph-mon[78593]: pgmap v8648: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:04:22.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:04:22 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 16595 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:04:23.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:04:23 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:04:23.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:04:23 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:04:23.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:04:23 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:04:23.610 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:04:23 smithi115 conmon[90393]: 2022-05-14T04:04:23.228+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:04:23.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:04:23 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:04:23.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:04:23 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:04:23.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:04:23 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:04:24.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:04:24 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:04:24.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:04:24 smithi047 ceph-mon[77147]: pgmap v8649: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:04:24.610 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:04:24 smithi115 conmon[90393]: 2022-05-14T04:04:24.214+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:04:24.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:04:24 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:04:24.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:04:24 smithi115 ceph-mon[78593]: pgmap v8649: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:04:25.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:04:25 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:04:25.610 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:04:25 smithi115 conmon[90393]: 2022-05-14T04:04:25.244+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:04:25.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:04:25 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:04:26.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:04:26 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:04:26.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:04:26 smithi047 ceph-mon[77147]: pgmap v8650: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:04:26.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:04:26 smithi115 conmon[90393]: 2022-05-14T04:04:26.213+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:04:26.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:04:26 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:04:26.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:04:26 smithi115 ceph-mon[78593]: pgmap v8650: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:04:27.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:04:27 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:04:27.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:04:27 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 16600 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:04:27.611 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:04:27 smithi115 conmon[90393]: 2022-05-14T04:04:27.233+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:04:27.612 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:04:27 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:04:27.613 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:04:27 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 16600 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:04:28.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:04:28 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:04:28.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:04:28 smithi047 ceph-mon[77147]: pgmap v8651: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:04:28.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:04:28 smithi115 conmon[90393]: 2022-05-14T04:04:28.207+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:04:28.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:04:28 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:04:28.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:04:28 smithi115 ceph-mon[78593]: pgmap v8651: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:04:29.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:04:29 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:04:29.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:04:29 smithi115 conmon[90393]: 2022-05-14T04:04:29.165+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:04:29.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:04:29 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:04:30.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:04:30 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:04:30.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:04:30 smithi047 ceph-mon[77147]: pgmap v8652: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:04:30.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:04:30 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:04:30.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:04:30 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:04:30.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:04:30 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:04:30.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:04:30 smithi115 conmon[90393]: 2022-05-14T04:04:30.149+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:04:30.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:04:30 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:04:30.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:04:30 smithi115 ceph-mon[78593]: pgmap v8652: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:04:30.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:04:30 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:04:30.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:04:30 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:04:30.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:04:30 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:04:31.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:04:31 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:04:31.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:04:31 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:04:31.610 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:04:31 smithi115 conmon[90393]: 2022-05-14T04:04:31.167+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:04:32.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:04:32 smithi047 ceph-mon[77147]: pgmap v8653: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:04:32.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:04:32 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:04:32.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:04:32 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 16605 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:04:32.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:04:32 smithi115 conmon[90393]: 2022-05-14T04:04:32.170+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:04:32.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:04:32 smithi115 ceph-mon[78593]: pgmap v8653: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:04:32.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:04:32 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:04:32.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:04:32 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 16605 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:04:33.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:04:33 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:04:33.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:04:33 smithi115 conmon[90393]: 2022-05-14T04:04:33.196+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:04:33.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:04:33 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:04:34.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:04:34 smithi047 ceph-mon[77147]: pgmap v8654: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:04:34.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:04:34 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:04:34.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:04:34 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:04:34.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:04:34 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:04:34.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:04:34 smithi115 conmon[90393]: 2022-05-14T04:04:34.172+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:04:34.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:04:34 smithi115 ceph-mon[78593]: pgmap v8654: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:04:34.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:04:34 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:04:34.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:04:34 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:04:34.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:04:34 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:04:35.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:04:35 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:04:35.619 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:04:35 smithi115 conmon[90393]: 2022-05-14T04:04:35.193+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:04:35.619 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:04:35 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:04:36.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:04:36 smithi047 ceph-mon[77147]: pgmap v8655: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:04:36.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:04:36 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:04:36.610 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:04:36 smithi115 conmon[90393]: 2022-05-14T04:04:36.158+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:04:36.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:04:36 smithi115 ceph-mon[78593]: pgmap v8655: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:04:36.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:04:36 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:04:37.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:04:37 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:04:37.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:04:37 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 16610 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:04:37.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:04:37 smithi115 conmon[90393]: 2022-05-14T04:04:37.160+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:04:37.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:04:37 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:04:37.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:04:37 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 16610 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:04:38.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:04:38 smithi047 ceph-mon[77147]: pgmap v8656: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:04:38.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:04:38 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:04:38.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:04:38 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:04:38.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:04:38 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:04:38.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:04:38 smithi115 conmon[90393]: 2022-05-14T04:04:38.194+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:04:38.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:04:38 smithi115 ceph-mon[78593]: pgmap v8656: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:04:38.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:04:38 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:04:38.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:04:38 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:04:38.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:04:38 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:04:39.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:04:39 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:04:39.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:04:39 smithi115 conmon[90393]: 2022-05-14T04:04:39.212+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:04:39.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:04:39 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:04:40.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:04:40 smithi047 ceph-mon[77147]: pgmap v8657: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:04:40.611 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:04:40 smithi115 conmon[90393]: 2022-05-14T04:04:40.258+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:04:40.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:04:40 smithi115 ceph-mon[78593]: pgmap v8657: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:04:41.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:04:41 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:04:41.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:04:41 smithi115 conmon[90393]: 2022-05-14T04:04:41.273+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:04:41.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:04:41 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:04:42.545 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:04:42 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:04:42.546 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:04:42 smithi047 ceph-mon[77147]: pgmap v8658: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:04:42.546 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:04:42 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:04:42.546 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:04:42 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:04:42.546 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:04:42 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 16615 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:04:42.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:04:42 smithi115 conmon[90393]: 2022-05-14T04:04:42.255+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:04:42.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:04:42 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:04:42.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:04:42 smithi115 ceph-mon[78593]: pgmap v8658: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:04:42.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:04:42 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:04:42.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:04:42 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:04:42.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:04:42 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 16615 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:04:43.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:04:43 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:04:43.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:04:43 smithi115 conmon[90393]: 2022-05-14T04:04:43.211+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:04:43.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:04:43 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:04:44.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:04:44 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:04:44.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:04:44 smithi047 ceph-mon[77147]: pgmap v8659: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:04:44.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:04:44 smithi115 conmon[90393]: 2022-05-14T04:04:44.254+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:04:44.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:04:44 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:04:44.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:04:44 smithi115 ceph-mon[78593]: pgmap v8659: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:04:45.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:04:45 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:04:45.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:04:45 smithi115 conmon[90393]: 2022-05-14T04:04:45.216+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:04:45.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:04:45 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:04:46.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:04:46 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:04:46.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:04:46 smithi047 ceph-mon[77147]: pgmap v8660: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:04:46.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:04:46 smithi115 conmon[90393]: 2022-05-14T04:04:46.176+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:04:46.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:04:46 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:04:46.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:04:46 smithi115 ceph-mon[78593]: pgmap v8660: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:04:47.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:04:47 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:04:47.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:04:47 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:04:47.545 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:04:47 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 16620 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:04:47.545 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:04:47 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:04:47.610 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:04:47 smithi115 conmon[90393]: 2022-05-14T04:04:47.179+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:04:47.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:04:47 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:04:47.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:04:47 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:04:47.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:04:47 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 16620 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:04:47.612 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:04:47 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:04:48.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:04:48 smithi047 ceph-mon[77147]: pgmap v8661: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:04:48.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:04:48 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:04:48.610 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:04:48 smithi115 conmon[90393]: 2022-05-14T04:04:48.131+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:04:48.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:04:48 smithi115 ceph-mon[78593]: pgmap v8661: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:04:48.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:04:48 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:04:49.360 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:04:49 smithi115 conmon[90393]: 2022-05-14T04:04:49.098+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:04:49.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:04:49 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:04:49.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:04:49 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:04:50.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:04:50 smithi047 ceph-mon[77147]: pgmap v8662: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:04:50.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:04:50 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:04:50.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:04:50 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:04:50.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:04:50 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:04:50.610 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:04:50 smithi115 conmon[90393]: 2022-05-14T04:04:50.140+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:04:50.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:04:50 smithi115 ceph-mon[78593]: pgmap v8662: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:04:50.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:04:50 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:04:50.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:04:50 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:04:50.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:04:50 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:04:51.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:04:51 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:04:51.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:04:51 smithi115 conmon[90393]: 2022-05-14T04:04:51.152+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:04:51.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:04:51 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:04:52.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:04:52 smithi047 ceph-mon[77147]: pgmap v8663: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:04:52.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:04:52 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 16626 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:04:52.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:04:52 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:04:52.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:04:52 smithi115 conmon[90393]: 2022-05-14T04:04:52.197+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:04:52.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:04:52 smithi115 ceph-mon[78593]: pgmap v8663: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:04:52.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:04:52 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 16626 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:04:52.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:04:52 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:04:53.610 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:04:53 smithi115 conmon[90393]: 2022-05-14T04:04:53.239+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:04:54.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:04:54 smithi047 ceph-mon[77147]: pgmap v8664: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:04:54.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:04:54 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:04:54.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:04:54 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:04:54.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:04:54 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:04:54.610 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:04:54 smithi115 conmon[90393]: 2022-05-14T04:04:54.233+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:04:54.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:04:54 smithi115 ceph-mon[78593]: pgmap v8664: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:04:54.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:04:54 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:04:54.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:04:54 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:04:54.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:04:54 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:04:55.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:04:55 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:04:55.610 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:04:55 smithi115 conmon[90393]: 2022-05-14T04:04:55.212+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:04:55.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:04:55 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:04:56.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:04:56 smithi047 ceph-mon[77147]: pgmap v8665: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:04:56.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:04:56 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:04:56.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:04:56 smithi115 conmon[90393]: 2022-05-14T04:04:56.242+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:04:56.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:04:56 smithi115 ceph-mon[78593]: pgmap v8665: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:04:56.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:04:56 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:04:57.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:04:57 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:04:57.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:04:57 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 16630 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:04:57.613 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:04:57 smithi115 conmon[90393]: 2022-05-14T04:04:57.213+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:04:57.614 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:04:57 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:04:57.614 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:04:57 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 16630 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:04:58.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:04:58 smithi047 ceph-mon[77147]: pgmap v8666: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:04:58.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:04:58 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:04:58.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:04:58 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:04:58.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:04:58 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:04:58.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:04:58 smithi115 conmon[90393]: 2022-05-14T04:04:58.218+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:04:58.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:04:58 smithi115 ceph-mon[78593]: pgmap v8666: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:04:58.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:04:58 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:04:58.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:04:58 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:04:58.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:04:58 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:04:59.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:04:59 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:04:59.613 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:04:59 smithi115 conmon[90393]: 2022-05-14T04:04:59.258+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:04:59.616 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:04:59 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:05:00.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:05:00 smithi047 ceph-mon[77147]: pgmap v8667: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:05:00.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:05:00 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:05:00.610 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:05:00 smithi115 conmon[90393]: 2022-05-14T04:05:00.271+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:05:00.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:05:00 smithi115 ceph-mon[78593]: pgmap v8667: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:05:00.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:05:00 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:05:01.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:05:01 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:05:01.611 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:05:01 smithi115 conmon[90393]: 2022-05-14T04:05:01.230+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:05:01.612 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:05:01 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:05:02.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:05:02 smithi047 ceph-mon[77147]: pgmap v8668: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:05:02.546 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:05:02 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:05:02.546 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:05:02 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:05:02.546 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:05:02 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:05:02.547 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:05:02 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 16635 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:05:02.547 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:05:02 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:05:02.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:05:02 smithi115 conmon[90393]: 2022-05-14T04:05:02.189+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:05:02.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:05:02 smithi115 ceph-mon[78593]: pgmap v8668: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:05:02.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:05:02 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:05:02.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:05:02 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:05:02.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:05:02 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:05:02.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:05:02 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 16635 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:05:02.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:05:02 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:05:03.546 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:05:03 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:05:03.610 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:05:03 smithi115 conmon[90393]: 2022-05-14T04:05:03.168+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:05:03.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:05:03 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:05:04.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:05:04 smithi047 ceph-mon[77147]: pgmap v8669: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:05:04.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:05:04 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:05:04.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:05:04 smithi115 conmon[90393]: 2022-05-14T04:05:04.193+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:05:04.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:05:04 smithi115 ceph-mon[78593]: pgmap v8669: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:05:04.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:05:04 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:05:05.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:05:05 smithi115 conmon[90393]: 2022-05-14T04:05:05.224+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:05:06.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:05:06 smithi047 ceph-mon[77147]: pgmap v8670: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:05:06.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:05:06 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:05:06.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:05:06 smithi115 conmon[90393]: 2022-05-14T04:05:06.238+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:05:06.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:05:06 smithi115 ceph-mon[78593]: pgmap v8670: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:05:06.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:05:06 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:05:07.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:05:07 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:05:07.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:05:07 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 16640 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:05:07.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:05:07 smithi115 conmon[90393]: 2022-05-14T04:05:07.262+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:05:07.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:05:07 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:05:07.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:05:07 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 16640 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:05:08.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:05:08 smithi047 ceph-mon[77147]: pgmap v8671: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:05:08.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:05:08 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:05:08.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:05:08 smithi115 conmon[90393]: 2022-05-14T04:05:08.214+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:05:08.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:05:08 smithi115 ceph-mon[78593]: pgmap v8671: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:05:08.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:05:08 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:05:09.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:05:09 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:05:09.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:05:09 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:05:09.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:05:09 smithi115 conmon[90393]: 2022-05-14T04:05:09.197+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:05:09.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:05:09 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:05:09.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:05:09 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:05:10.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:05:10 smithi047 ceph-mon[77147]: pgmap v8672: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:05:10.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:05:10 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:05:10.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:05:10 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:05:10.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:05:10 smithi115 conmon[90393]: 2022-05-14T04:05:10.223+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:05:10.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:05:10 smithi115 ceph-mon[78593]: pgmap v8672: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:05:10.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:05:10 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:05:10.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:05:10 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:05:11.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:05:11 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:05:11.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:05:11 smithi115 conmon[90393]: 2022-05-14T04:05:11.208+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:05:11.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:05:11 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:05:12.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:05:12 smithi047 ceph-mon[77147]: pgmap v8673: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:05:12.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:05:12 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:05:12.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:05:12 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 16645 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:05:12.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:05:12 smithi115 conmon[90393]: 2022-05-14T04:05:12.236+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:05:12.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:05:12 smithi115 ceph-mon[78593]: pgmap v8673: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:05:12.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:05:12 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:05:12.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:05:12 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 16645 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:05:13.530 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:05:13 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:05:13.530 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:05:13 smithi115 conmon[90393]: 2022-05-14T04:05:13.265+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:05:13.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:05:13 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:05:14.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:05:14 smithi047 ceph-mon[77147]: pgmap v8674: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:05:14.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:05:14 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:05:14.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:05:14 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:05:14.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:05:14 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:05:14.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:05:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T04:05:14.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:05:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T04:05:14.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:05:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T04:05:14.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:05:14 smithi115 conmon[90393]: 2022-05-14T04:05:14.252+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:05:14.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:05:14 smithi115 ceph-mon[78593]: pgmap v8674: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:05:14.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:05:14 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:05:14.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:05:14 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:05:14.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:05:14 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:05:14.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:05:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T04:05:14.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:05:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T04:05:14.612 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:05:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T04:05:15.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:05:15 smithi115 conmon[90393]: 2022-05-14T04:05:15.300+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:05:15.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:05:15 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:05:15.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:05:15 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T04:05:15.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:05:15 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T04:05:15.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:05:15 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:05:15.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:05:15 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T04:05:15.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:05:15 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T04:05:16.611 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:05:16 smithi115 conmon[90393]: 2022-05-14T04:05:16.289+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:05:16.612 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:05:16 smithi115 ceph-mon[78593]: pgmap v8675: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:05:16.612 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:05:16 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:05:16.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:05:16 smithi047 ceph-mon[77147]: pgmap v8675: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:05:16.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:05:16 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:05:16.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:05:16 smithi047 conmon[78521]: 2022-05-14T04:05:16.527+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:05:16.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:05:16 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T04:05:16.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:05:16 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:05:16.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:05:16 smithi047 conmon[78521]: yield 2022-05-14T04:05:16.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:05:16 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:05:16.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:05:16 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:05:16.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:05:16 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:05:16.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:05:16 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T04:05:16.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:05:16 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:05:16.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:05:16 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T04:05:16.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:05:16 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:05:16.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:05:16 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T04:05:16.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:05:16 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:05:16.797 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:05:16 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T04:05:16.797 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:05:16 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:05:16.797 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:05:16 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T04:05:16.797 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:05:16 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:05:16.797 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:05:16 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:05:16.798 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:05:16 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:05:17.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:05:17 smithi115 conmon[90393]: 2022-05-14T04:05:17.276+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:05:17.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:05:17 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:05:17.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:05:17 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 16650 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:05:17.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:05:17 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:05:17.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:05:17 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 16650 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:05:18.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:05:18 smithi115 conmon[90393]: 2022-05-14T04:05:18.282+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:05:18.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:05:18 smithi115 ceph-mon[78593]: pgmap v8676: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:05:18.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:05:18 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:05:18.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T04:05:18.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:05:18.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T04:05:18.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:05:18.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:05:18.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:05:18.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T04:05:18.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:05:18.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T04:05:18.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:05:18.613 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T04:05:18.613 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:05:18.613 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T04:05:18.613 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:05:18.613 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T04:05:18.614 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:05:18.614 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:05:18.614 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:05:18.614 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:05:18 smithi115 ceph-mon[78593]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T04:05:18.614 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:05:18 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:05:18.615 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:05:18 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:05:18.615 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:05:18 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:05:18.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:05:18 smithi047 ceph-mon[77147]: pgmap v8676: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:05:18.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:05:18 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:05:18.793 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T04:05:18.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:05:18.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T04:05:18.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:05:18.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:05:18.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:05:18.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T04:05:18.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:05:18.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T04:05:18.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:05:18.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T04:05:18.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:05:18.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T04:05:18.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:05:18.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T04:05:18.797 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:05:18.797 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:05:18.797 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:05:18.797 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:05:18 smithi047 ceph-mon[77147]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T04:05:18.797 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:05:18 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:05:18.798 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:05:18 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:05:18.798 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:05:18 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:05:19.603 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:05:19 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:05:19.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:05:19 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:05:19.610 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:05:19 smithi115 conmon[90393]: 2022-05-14T04:05:19.309+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:05:20.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:05:19 smithi047 conmon[78521]: 2022-05-14T04:05:19.599+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:05:20.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:05:19 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T04:05:20.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:05:19 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:05:20.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:05:19 smithi047 conmon[78521]: yield 2022-05-14T04:05:20.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:05:19 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:05:20.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:05:19 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:05:20.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:05:19 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:05:20.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:05:19 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T04:05:20.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:05:19 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:05:20.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:05:19 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T04:05:20.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:05:19 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:05:20.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:05:19 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T04:05:20.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:05:19 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:05:20.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:05:19 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T04:05:20.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:05:19 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:05:20.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:05:19 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T04:05:20.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:05:19 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:05:20.048 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:05:19 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:05:20.048 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:05:19 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:05:20.048 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:05:19 smithi047 conmon[78521]: 2022-05-14T04:05:19.600+0000 7f533ef76700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T04:05:20.048 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:05:19 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T04:05:20.049 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:05:19 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:05:20.049 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:05:19 smithi047 conmon[78521]: yield 2022-05-14T04:05:20.049 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:05:19 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:05:20.049 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:05:19 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:05:20.050 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:05:19 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:05:20.050 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:05:19 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T04:05:20.050 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:05:19 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:05:20.050 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:05:19 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T04:05:20.050 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:05:19 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:05:20.051 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:05:19 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T04:05:20.051 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:05:19 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:05:20.051 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:05:19 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T04:05:20.051 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:05:19 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:05:20.052 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:05:19 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T04:05:20.052 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:05:19 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:05:20.052 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:05:19 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:05:20.052 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:05:19 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:05:20.052 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:05:19 smithi047 conmon[78521]: 2022-05-14T04:05:20.053 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:05:19 smithi047 conmon[78521]: During handling of the above exception, another exception occurred: 2022-05-14T04:05:20.053 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:05:19 smithi047 conmon[78521]: 2022-05-14T04:05:20.053 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:05:19 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T04:05:20.053 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:05:19 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T04:05:20.054 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:05:19 smithi047 conmon[78521]: return f(*arg) 2022-05-14T04:05:20.054 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:05:19 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T04:05:20.054 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:05:19 smithi047 conmon[78521]: self._write_client_files(client_files, host) 2022-05-14T04:05:20.055 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:05:19 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T04:05:20.055 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:05:19 smithi047 conmon[78521]: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T04:05:20.055 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:05:19 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T04:05:20.055 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:05:19 smithi047 conmon[78521]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T04:05:20.056 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:05:19 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T04:05:20.056 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:05:19 smithi047 conmon[78521]: return self.event_loop.get_result(coro) 2022-05-14T04:05:20.056 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:05:19 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T04:05:20.056 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:05:19 smithi047 conmon[78521]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T04:05:20.056 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:05:19 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T04:05:20.057 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:05:19 smithi047 conmon[78521]: return self.__get_result() 2022-05-14T04:05:20.057 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:05:19 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T04:05:20.057 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:05:19 smithi047 conmon[78521]: raise self._exception 2022-05-14T04:05:20.057 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:05:19 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T04:05:20.058 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:05:19 smithi047 conmon[78521]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T04:05:20.058 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:05:19 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T04:05:20.058 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:05:19 smithi047 conmon[78521]: conn = await self._remote_connection(host, addr) 2022-05-14T04:05:20.058 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:05:19 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T04:05:20.059 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:05:19 smithi047 conmon[78521]: raise 2022-05-14T04:05:20.059 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:05:19 smithi047 conmon[78521]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T04:05:20.059 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:05:19 smithi047 conmon[78521]: self.gen.throw(type, value, traceback) 2022-05-14T04:05:20.059 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:05:19 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T04:05:20.059 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:05:19 smithi047 conmon[78521]: raise OrchestratorError(msg) 2022-05-14T04:05:20.060 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:05:19 smithi047 conmon[78521]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:05:20.610 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:05:20 smithi115 conmon[90393]: 2022-05-14T04:05:20.340+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:05:20.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:05:20 smithi115 ceph-mon[78593]: pgmap v8677: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:05:20.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:05:20 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:05:20.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:05:20 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:05:20.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T04:05:20.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:05:20.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T04:05:20.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:05:20.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:05:20.613 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:05:20.613 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T04:05:20.613 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:05:20.614 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T04:05:20.614 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:05:20.614 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T04:05:20.616 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:05:20.617 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T04:05:20.617 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:05:20.618 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T04:05:20.618 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:05:20.619 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:05:20.619 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:05:20.619 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:05:20 smithi115 ceph-mon[78593]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T04:05:20.620 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T04:05:20.620 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:05:20.621 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T04:05:20.621 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:05:20.622 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:05:20.622 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:05:20.622 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T04:05:20.623 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:05:20.623 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T04:05:20.623 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:05:20.624 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T04:05:20.624 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:05:20.624 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T04:05:20.625 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:05:20.625 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T04:05:20.625 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:05:20.626 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:05:20.626 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:05:20.627 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T04:05:20.627 INFO:journalctl@ceph.mon.c.smithi115.stdout: During handling of the above exception, another exception occurred: 2022-05-14T04:05:20.627 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T04:05:20.628 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T04:05:20.628 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T04:05:20.628 INFO:journalctl@ceph.mon.c.smithi115.stdout: return f(*arg) 2022-05-14T04:05:20.630 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T04:05:20.630 INFO:journalctl@ceph.mon.c.smithi115.stdout: self._write_client_files(client_files, host) 2022-05-14T04:05:20.630 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T04:05:20.630 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T04:05:20.630 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T04:05:20.631 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T04:05:20.631 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T04:05:20.632 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.event_loop.get_result(coro) 2022-05-14T04:05:20.632 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T04:05:20.632 INFO:journalctl@ceph.mon.c.smithi115.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T04:05:20.632 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T04:05:20.633 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.__get_result() 2022-05-14T04:05:20.633 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T04:05:20.633 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise self._exception 2022-05-14T04:05:20.633 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T04:05:20.633 INFO:journalctl@ceph.mon.c.smithi115.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T04:05:20.634 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T04:05:20.634 INFO:journalctl@ceph.mon.c.smithi115.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T04:05:20.634 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T04:05:20.634 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise 2022-05-14T04:05:20.635 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T04:05:20.635 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.gen.throw(type, value, traceback) 2022-05-14T04:05:20.635 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T04:05:20.635 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OrchestratorError(msg) 2022-05-14T04:05:20.636 INFO:journalctl@ceph.mon.c.smithi115.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:05:20.795 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:05:20 smithi047 ceph-mon[77147]: pgmap v8677: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:05:20.795 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:05:20 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:05:20.795 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:05:20 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:05:20.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T04:05:20.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:05:20.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T04:05:20.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:05:20.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:05:20.797 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:05:20.797 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T04:05:20.797 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:05:20.797 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T04:05:20.798 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:05:20.798 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T04:05:20.798 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:05:20.798 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T04:05:20.798 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:05:20.799 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T04:05:20.799 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:05:20.799 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:05:20.799 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:05:20.800 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:05:20 smithi047 ceph-mon[77147]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T04:05:20.800 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T04:05:20.800 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:05:20.800 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T04:05:20.800 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:05:20.801 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:05:20.801 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:05:20.801 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T04:05:20.801 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:05:20.802 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T04:05:20.802 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:05:20.802 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T04:05:20.802 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:05:20.803 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T04:05:20.803 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:05:20.803 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T04:05:20.803 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:05:20.803 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:05:20.804 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:05:20.804 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T04:05:20.804 INFO:journalctl@ceph.mon.b.smithi047.stdout: During handling of the above exception, another exception occurred: 2022-05-14T04:05:20.804 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T04:05:20.805 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T04:05:20.805 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T04:05:20.805 INFO:journalctl@ceph.mon.b.smithi047.stdout: return f(*arg) 2022-05-14T04:05:20.805 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T04:05:20.806 INFO:journalctl@ceph.mon.b.smithi047.stdout: self._write_client_files(client_files, host) 2022-05-14T04:05:20.806 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T04:05:20.806 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T04:05:20.806 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T04:05:20.807 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T04:05:20.807 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T04:05:20.807 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.event_loop.get_result(coro) 2022-05-14T04:05:20.807 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T04:05:20.807 INFO:journalctl@ceph.mon.b.smithi047.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T04:05:20.808 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T04:05:20.808 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.__get_result() 2022-05-14T04:05:20.808 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T04:05:20.808 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise self._exception 2022-05-14T04:05:20.808 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T04:05:20.809 INFO:journalctl@ceph.mon.b.smithi047.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T04:05:20.809 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T04:05:20.813 INFO:journalctl@ceph.mon.b.smithi047.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T04:05:20.814 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T04:05:20.814 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise 2022-05-14T04:05:20.814 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T04:05:20.814 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.gen.throw(type, value, traceback) 2022-05-14T04:05:20.814 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T04:05:20.815 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OrchestratorError(msg) 2022-05-14T04:05:20.815 INFO:journalctl@ceph.mon.b.smithi047.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:05:21.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:05:21 smithi115 conmon[90393]: 2022-05-14T04:05:21.332+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:05:21.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:05:21 smithi115 ceph-mon[78593]: pgmap v8678: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:05:21.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:05:21 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:05:21.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:05:21 smithi047 ceph-mon[77147]: pgmap v8678: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:05:21.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:05:21 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:05:22.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:05:22 smithi115 conmon[90393]: 2022-05-14T04:05:22.378+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:05:22.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:05:22 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:05:22.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:05:22 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:05:22.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:05:22 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:05:22.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:05:22 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 16655 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:05:22.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:05:22 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:05:22.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:05:22 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:05:22.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:05:22 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:05:22.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:05:22 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 16655 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:05:23.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:05:23 smithi115 conmon[90393]: 2022-05-14T04:05:23.403+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:05:23.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:05:23 smithi115 ceph-mon[78593]: pgmap v8679: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:05:23.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:05:23 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:05:23.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:05:23 smithi047 ceph-mon[77147]: pgmap v8679: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:05:23.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:05:23 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:05:24.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:05:24 smithi115 conmon[90393]: 2022-05-14T04:05:24.410+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:05:24.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:05:24 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:05:24.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:05:24 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:05:25.797 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:05:25 smithi047 ceph-mon[77147]: pgmap v8680: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:05:25.797 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:05:25 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:05:25.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:05:25 smithi115 conmon[90393]: 2022-05-14T04:05:25.407+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:05:25.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:05:25 smithi115 ceph-mon[78593]: pgmap v8680: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:05:25.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:05:25 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:05:26.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:05:26 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:05:26.861 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:05:26 smithi115 conmon[90393]: 2022-05-14T04:05:26.361+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:05:26.862 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:05:26 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:05:27.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:05:27 smithi047 ceph-mon[77147]: pgmap v8681: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:05:27.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:05:27 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:05:27.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:05:27 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 16660 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:05:27.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:05:27 smithi115 conmon[90393]: 2022-05-14T04:05:27.382+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:05:27.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:05:27 smithi115 ceph-mon[78593]: pgmap v8681: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:05:27.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:05:27 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:05:27.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:05:27 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 16660 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:05:28.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:05:28 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:05:28.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:05:28 smithi115 conmon[90393]: 2022-05-14T04:05:28.430+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:05:28.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:05:28 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:05:29.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:05:29 smithi047 ceph-mon[77147]: pgmap v8682: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:05:29.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:05:29 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:05:29.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:05:29 smithi115 conmon[90393]: 2022-05-14T04:05:29.434+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:05:29.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:05:29 smithi115 ceph-mon[78593]: pgmap v8682: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:05:29.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:05:29 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:05:30.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:05:30 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:05:30.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:05:30 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:05:30.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:05:30 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:05:30.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:05:30 smithi115 conmon[90393]: 2022-05-14T04:05:30.398+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:05:30.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:05:30 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:05:30.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:05:30 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:05:30.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:05:30 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:05:31.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:05:31 smithi047 ceph-mon[77147]: pgmap v8683: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:05:31.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:05:31 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:05:31.862 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:05:31 smithi115 conmon[90393]: 2022-05-14T04:05:31.372+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:05:31.863 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:05:31 smithi115 ceph-mon[78593]: pgmap v8683: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:05:31.863 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:05:31 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:05:32.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:05:32 smithi115 conmon[90393]: 2022-05-14T04:05:32.336+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:05:32.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:05:32 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:05:32.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:05:32 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 16665 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:05:32.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:05:32 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:05:32.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:05:32 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 16665 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:05:33.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:05:33 smithi115 conmon[90393]: 2022-05-14T04:05:33.302+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:05:34.362 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:05:34 smithi115 conmon[90393]: 2022-05-14T04:05:34.341+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:05:34.363 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:05:34 smithi115 ceph-mon[78593]: pgmap v8684: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:05:34.363 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:05:34 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:05:34.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:05:34 smithi047 ceph-mon[77147]: pgmap v8684: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:05:34.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:05:34 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:05:35.385 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:05:35 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:05:35.385 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:05:35 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:05:35.386 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:05:35 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:05:35.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:05:35 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:05:35.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:05:35 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:05:35.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:05:35 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:05:35.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:05:35 smithi115 conmon[90393]: 2022-05-14T04:05:35.381+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:05:36.434 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:05:36 smithi115 ceph-mon[78593]: pgmap v8685: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:05:36.434 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:05:36 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:05:36.550 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:05:36 smithi047 ceph-mon[77147]: pgmap v8685: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:05:36.550 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:05:36 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:05:36.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:05:36 smithi115 conmon[90393]: 2022-05-14T04:05:36.429+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:05:37.400 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:05:37 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:05:37.400 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:05:37 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 16670 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:05:37.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:05:37 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:05:37.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:05:37 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 16670 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:05:37.864 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:05:37 smithi115 conmon[90393]: 2022-05-14T04:05:37.397+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:05:38.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:05:38 smithi047 ceph-mon[77147]: pgmap v8686: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:05:38.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:05:38 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:05:38.610 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:05:38 smithi115 conmon[90393]: 2022-05-14T04:05:38.350+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:05:38.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:05:38 smithi115 ceph-mon[78593]: pgmap v8686: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:05:38.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:05:38 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:05:39.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:05:39 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:05:39.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:05:39 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:05:39.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:05:39 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:05:39.610 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:05:39 smithi115 conmon[90393]: 2022-05-14T04:05:39.349+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:05:39.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:05:39 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:05:39.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:05:39 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:05:39.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:05:39 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:05:40.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:05:40 smithi047 ceph-mon[77147]: pgmap v8687: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:05:40.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:05:40 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:05:40.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:05:40 smithi115 conmon[90393]: 2022-05-14T04:05:40.356+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:05:40.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:05:40 smithi115 ceph-mon[78593]: pgmap v8687: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:05:40.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:05:40 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:05:41.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:05:41 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:05:41.610 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:05:41 smithi115 conmon[90393]: 2022-05-14T04:05:41.357+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:05:41.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:05:41 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:05:42.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:05:42 smithi047 ceph-mon[77147]: pgmap v8688: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:05:42.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:05:42 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:05:42.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:05:42 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:05:42.545 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:05:42 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:05:42.545 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:05:42 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 16675 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:05:42.610 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:05:42 smithi115 conmon[90393]: 2022-05-14T04:05:42.312+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:05:42.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:05:42 smithi115 ceph-mon[78593]: pgmap v8688: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:05:42.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:05:42 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:05:42.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:05:42 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:05:42.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:05:42 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:05:42.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:05:42 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 16675 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:05:43.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:05:43 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:05:43.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:05:43 smithi115 conmon[90393]: 2022-05-14T04:05:43.314+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:05:43.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:05:43 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:05:44.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:05:44 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:05:44.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:05:44 smithi047 ceph-mon[77147]: pgmap v8689: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:05:44.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:05:44 smithi115 conmon[90393]: 2022-05-14T04:05:44.288+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:05:44.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:05:44 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:05:44.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:05:44 smithi115 ceph-mon[78593]: pgmap v8689: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:05:45.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:05:45 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:05:45.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:05:45 smithi115 conmon[90393]: 2022-05-14T04:05:45.267+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:05:45.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:05:45 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:05:46.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:05:46 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:05:46.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:05:46 smithi047 ceph-mon[77147]: pgmap v8690: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:05:46.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:05:46 smithi115 conmon[90393]: 2022-05-14T04:05:46.270+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:05:46.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:05:46 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:05:46.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:05:46 smithi115 ceph-mon[78593]: pgmap v8690: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:05:47.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:05:47 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:05:47.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:05:47 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 16680 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:05:47.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:05:47 smithi115 conmon[90393]: 2022-05-14T04:05:47.228+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:05:47.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:05:47 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:05:47.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:05:47 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 16680 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:05:48.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:05:48 smithi047 ceph-mon[77147]: 2 slow requests (by type [ 'queued for pg' : 2 ] most affected pool [ 'cephfs_metadata' : 2 ]) 2022-05-14T04:05:48.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:05:48 smithi047 ceph-mon[77147]: pgmap v8691: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:05:48.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:05:48 smithi115 conmon[90393]: 2022-05-14T04:05:48.202+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:05:48.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:05:48 smithi115 ceph-mon[78593]: 2 slow requests (by type [ 'queued for pg' : 2 ] most affected pool [ 'cephfs_metadata' : 2 ]) 2022-05-14T04:05:48.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:05:48 smithi115 ceph-mon[78593]: pgmap v8691: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:05:49.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:05:49 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:05:49.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:05:49 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:05:49.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:05:49 smithi115 conmon[90393]: 2022-05-14T04:05:49.219+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:05:49.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:05:49 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:05:49.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:05:49 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:05:50.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:05:50 smithi047 ceph-mon[77147]: pgmap v8692: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:05:50.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:05:50 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:05:50.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:05:50 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:05:50.615 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:05:50 smithi115 conmon[90393]: 2022-05-14T04:05:50.184+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:05:50.616 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:05:50 smithi115 ceph-mon[78593]: pgmap v8692: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:05:50.616 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:05:50 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:05:50.617 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:05:50 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:05:51.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:05:51 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:05:51.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:05:51 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:05:51.612 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:05:51 smithi115 conmon[90393]: 2022-05-14T04:05:51.234+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:05:51.613 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:05:51 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:05:51.613 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:05:51 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:05:52.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:05:52 smithi047 ceph-mon[77147]: pgmap v8693: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:05:52.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:05:52 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 16685 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:05:52.612 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:05:52 smithi115 ceph-mon[78593]: pgmap v8693: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:05:52.612 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:05:52 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 16685 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:05:53.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:05:53 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:05:53.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:05:53 smithi115 conmon[90393]: 2022-05-14T04:05:53.311+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:05:53.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:05:53 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:05:54.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:05:54 smithi047 ceph-mon[77147]: pgmap v8694: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:05:54.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:05:54 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:05:54.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:05:54 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:05:54.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:05:54 smithi115 conmon[90393]: 2022-05-14T04:05:54.282+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:05:54.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:05:54 smithi115 ceph-mon[78593]: pgmap v8694: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:05:54.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:05:54 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:05:54.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:05:54 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:05:55.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:05:55 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:05:55.613 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:05:55 smithi115 conmon[90393]: 2022-05-14T04:05:55.251+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:05:55.614 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:05:55 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:05:56.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:05:56 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:05:56.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:05:56 smithi047 ceph-mon[77147]: pgmap v8695: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:05:56.610 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:05:56 smithi115 conmon[90393]: 2022-05-14T04:05:56.257+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:05:56.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:05:56 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:05:56.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:05:56 smithi115 ceph-mon[78593]: pgmap v8695: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:05:57.546 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:05:57 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:05:57.546 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:05:57 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 16690 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:05:57.610 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:05:57 smithi115 conmon[90393]: 2022-05-14T04:05:57.286+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:05:57.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:05:57 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:05:57.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:05:57 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 16690 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:05:58.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:05:58 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:05:58.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:05:58 smithi047 ceph-mon[77147]: pgmap v8696: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:05:58.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:05:58 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:05:58.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:05:58 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:05:58.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:05:58 smithi115 conmon[90393]: 2022-05-14T04:05:58.332+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:05:58.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:05:58 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:05:58.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:05:58 smithi115 ceph-mon[78593]: pgmap v8696: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:05:58.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:05:58 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:05:58.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:05:58 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:05:59.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:05:59 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:05:59.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:05:59 smithi115 conmon[90393]: 2022-05-14T04:05:59.294+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:05:59.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:05:59 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:06:00.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:06:00 smithi047 ceph-mon[77147]: pgmap v8697: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:06:00.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:06:00 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:06:00.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:06:00 smithi115 conmon[90393]: 2022-05-14T04:06:00.322+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:06:00.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:06:00 smithi115 ceph-mon[78593]: pgmap v8697: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:06:00.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:06:00 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:06:01.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:06:01 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:06:01.610 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:06:01 smithi115 conmon[90393]: 2022-05-14T04:06:01.326+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:06:01.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:06:01 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:06:02.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:06:02 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:06:02.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:06:02 smithi047 ceph-mon[77147]: pgmap v8698: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:06:02.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:06:02 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:06:02.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:06:02 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:06:02.546 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:06:02 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 16695 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:06:02.610 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:06:02 smithi115 conmon[90393]: 2022-05-14T04:06:02.359+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:06:02.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:06:02 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:06:02.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:06:02 smithi115 ceph-mon[78593]: pgmap v8698: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:06:02.612 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:06:02 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:06:02.614 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:06:02 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:06:02.614 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:06:02 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 16695 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:06:03.545 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:06:03 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:06:03.610 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:06:03 smithi115 conmon[90393]: 2022-05-14T04:06:03.407+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:06:03.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:06:03 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:06:04.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:06:04 smithi047 ceph-mon[77147]: pgmap v8699: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:06:04.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:06:04 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:06:04.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:06:04 smithi115 conmon[90393]: 2022-05-14T04:06:04.434+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:06:04.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:06:04 smithi115 ceph-mon[78593]: pgmap v8699: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:06:04.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:06:04 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:06:05.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:06:05 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:06:05.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:06:05 smithi115 conmon[90393]: 2022-05-14T04:06:05.452+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:06:05.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:06:05 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:06:06.488 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:06:06 smithi115 ceph-mon[78593]: pgmap v8700: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:06:06.489 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:06:06 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:06:06.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:06:06 smithi047 ceph-mon[77147]: pgmap v8700: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:06:06.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:06:06 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:06:06.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:06:06 smithi115 conmon[90393]: 2022-05-14T04:06:06.484+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:06:07.530 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:06:07 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:06:07.530 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:06:07 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 16700 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:06:07.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:06:07 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:06:07.545 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:06:07 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 16700 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:06:07.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:06:07 smithi115 conmon[90393]: 2022-05-14T04:06:07.523+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:06:08.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:06:08 smithi047 ceph-mon[77147]: pgmap v8701: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:06:08.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:06:08 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:06:08.545 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:06:08 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:06:08.563 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:06:08 smithi115 ceph-mon[78593]: pgmap v8701: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:06:08.563 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:06:08 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:06:08.564 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:06:08 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:06:08.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:06:08 smithi115 conmon[90393]: 2022-05-14T04:06:08.559+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:06:09.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:06:09 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:06:09.572 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:06:09 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:06:09.860 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:06:09 smithi115 conmon[90393]: 2022-05-14T04:06:09.569+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:06:10.545 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:06:10 smithi047 ceph-mon[77147]: pgmap v8702: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:06:10.547 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:06:10 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:06:10.548 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:06:10 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:06:10.548 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:06:10 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:06:10.604 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:06:10 smithi115 ceph-mon[78593]: pgmap v8702: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:06:10.605 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:06:10 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:06:10.605 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:06:10 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:06:10.605 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:06:10 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:06:10.865 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:06:10 smithi115 conmon[90393]: 2022-05-14T04:06:10.600+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:06:11.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:06:11 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:06:11.580 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:06:11 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:06:11.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:06:11 smithi115 conmon[90393]: 2022-05-14T04:06:11.576+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:06:12.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:06:12 smithi047 ceph-mon[77147]: pgmap v8703: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:06:12.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:06:12 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:06:12.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:06:12 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 16705 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:06:12.593 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:06:12 smithi115 ceph-mon[78593]: pgmap v8703: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:06:12.593 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:06:12 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:06:12.594 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:06:12 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 16705 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:06:12.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:06:12 smithi115 conmon[90393]: 2022-05-14T04:06:12.588+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:06:13.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:06:13 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:06:13.551 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:06:13 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:06:13.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:06:13 smithi115 conmon[90393]: 2022-05-14T04:06:13.548+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:06:14.513 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:06:14 smithi115 ceph-mon[78593]: pgmap v8704: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:06:14.513 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:06:14 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:06:14.514 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:06:14 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:06:14.514 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:06:14 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:06:14.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:06:14 smithi047 ceph-mon[77147]: pgmap v8704: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:06:14.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:06:14 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:06:14.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:06:14 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:06:14.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:06:14 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:06:14.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:06:14 smithi115 conmon[90393]: 2022-05-14T04:06:14.510+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:06:15.514 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:06:15 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:06:15.514 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:06:15 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T04:06:15.514 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:06:15 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T04:06:15.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:06:15 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:06:15.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:06:15 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T04:06:15.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:06:15 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T04:06:15.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:06:15 smithi115 conmon[90393]: 2022-05-14T04:06:15.510+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:06:16.545 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:06:16 smithi047 ceph-mon[77147]: pgmap v8705: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:06:16.545 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:06:16 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:06:16.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:06:16 smithi115 conmon[90393]: 2022-05-14T04:06:16.496+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:06:16.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:06:16 smithi115 ceph-mon[78593]: pgmap v8705: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:06:16.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:06:16 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:06:17.513 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:06:17 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:06:17.514 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:06:17 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 16710 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:06:17.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:06:17 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:06:17.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:06:17 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 16710 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:06:17.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:06:17 smithi115 conmon[90393]: 2022-05-14T04:06:17.510+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:06:18.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:06:18 smithi047 ceph-mon[77147]: pgmap v8706: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:06:18.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:06:18 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:06:18.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:06:18 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:06:18.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:06:18 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:06:18.610 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:06:18 smithi115 conmon[90393]: 2022-05-14T04:06:18.496+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:06:18.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:06:18 smithi115 ceph-mon[78593]: pgmap v8706: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:06:18.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:06:18 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:06:18.612 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:06:18 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:06:18.612 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:06:18 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:06:19.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:06:19 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:06:19.612 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:06:19 smithi115 conmon[90393]: 2022-05-14T04:06:19.466+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:06:19.612 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:06:19 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:06:20.437 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:06:20 smithi115 ceph-mon[78593]: pgmap v8707: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:06:20.437 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:06:20 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:06:20.438 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:06:20 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T04:06:20.438 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:06:20 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T04:06:20.438 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:06:20 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T04:06:20.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:06:20 smithi047 ceph-mon[77147]: pgmap v8707: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:06:20.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:06:20 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:06:20.545 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:06:20 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T04:06:20.545 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:06:20 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T04:06:20.545 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:06:20 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T04:06:20.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:06:20 smithi047 conmon[78521]: 2022-05-14T04:06:20.207+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:06:20.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:06:20 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T04:06:20.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:06:20 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:06:20.547 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:06:20 smithi047 conmon[78521]: yield 2022-05-14T04:06:20.547 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:06:20 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:06:20.547 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:06:20 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:06:20.547 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:06:20 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:06:20.548 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:06:20 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T04:06:20.549 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:06:20 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:06:20.549 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:06:20 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T04:06:20.549 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:06:20 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:06:20.549 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:06:20 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T04:06:20.549 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:06:20 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:06:20.550 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:06:20 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T04:06:20.550 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:06:20 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:06:20.550 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:06:20 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T04:06:20.550 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:06:20 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:06:20.551 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:06:20 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:06:20.551 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:06:20 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:06:20.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:06:20 smithi115 conmon[90393]: 2022-05-14T04:06:20.434+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:06:21.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:06:21 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:06:21.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T04:06:21.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:06:21.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T04:06:21.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:06:21.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:06:21.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:06:21.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T04:06:21.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:06:21.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T04:06:21.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:06:21.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T04:06:21.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:06:21.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T04:06:21.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:06:21.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T04:06:21.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:06:21.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:06:21.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:06:21.547 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:06:21 smithi047 ceph-mon[77147]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T04:06:21.547 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:06:21 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:06:21.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:06:21 smithi115 conmon[90393]: 2022-05-14T04:06:21.479+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:06:21.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:06:21 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:06:21.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T04:06:21.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:06:21.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T04:06:21.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:06:21.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:06:21.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:06:21.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T04:06:21.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:06:21.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T04:06:21.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:06:21.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T04:06:21.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:06:21.613 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T04:06:21.613 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:06:21.613 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T04:06:21.613 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:06:21.613 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:06:21.614 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:06:21.614 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:06:21 smithi115 ceph-mon[78593]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T04:06:21.614 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:06:21 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:06:22.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:06:22 smithi047 ceph-mon[77147]: pgmap v8708: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:06:22.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:06:22 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:06:22.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:06:22 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:06:22.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:06:22 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:06:22.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:06:22 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 16715 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:06:22.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:06:22 smithi115 conmon[90393]: 2022-05-14T04:06:22.471+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:06:22.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:06:22 smithi115 ceph-mon[78593]: pgmap v8708: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:06:22.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:06:22 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:06:22.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:06:22 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:06:22.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:06:22 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:06:22.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:06:22 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 16715 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:06:23.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:06:23 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:06:23.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:06:23 smithi047 conmon[78521]: 2022-05-14T04:06:23.279+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:06:23.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:06:23 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T04:06:23.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:06:23 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:06:23.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:06:23 smithi047 conmon[78521]: yield 2022-05-14T04:06:23.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:06:23 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:06:23.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:06:23 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:06:23.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:06:23 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:06:23.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:06:23 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T04:06:23.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:06:23 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:06:23.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:06:23 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T04:06:23.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:06:23 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:06:23.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:06:23 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T04:06:23.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:06:23 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:06:23.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:06:23 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T04:06:23.547 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:06:23 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:06:23.547 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:06:23 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T04:06:23.547 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:06:23 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:06:23.547 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:06:23 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:06:23.548 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:06:23 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:06:23.548 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:06:23 smithi047 conmon[78521]: 2022-05-14T04:06:23.280+0000 7f5344781700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T04:06:23.548 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:06:23 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T04:06:23.548 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:06:23 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:06:23.548 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:06:23 smithi047 conmon[78521]: yield 2022-05-14T04:06:23.549 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:06:23 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:06:23.549 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:06:23 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:06:23.549 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:06:23 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:06:23.550 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:06:23 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T04:06:23.550 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:06:23 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:06:23.550 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:06:23 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T04:06:23.550 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:06:23 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:06:23.551 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:06:23 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T04:06:23.551 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:06:23 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:06:23.551 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:06:23 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T04:06:23.551 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:06:23 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:06:23.552 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:06:23 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T04:06:23.552 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:06:23 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:06:23.552 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:06:23 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:06:23.552 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:06:23 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:06:23.553 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:06:23 smithi047 conmon[78521]: 2022-05-14T04:06:23.553 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:06:23 smithi047 conmon[78521]: During handling of the above exception, another exception occurred: 2022-05-14T04:06:23.553 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:06:23 smithi047 conmon[78521]: 2022-05-14T04:06:23.554 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:06:23 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T04:06:23.554 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:06:23 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T04:06:23.554 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:06:23 smithi047 conmon[78521]: return f(*arg) 2022-05-14T04:06:23.554 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:06:23 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T04:06:23.554 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:06:23 smithi047 conmon[78521]: self._write_client_files(client_files, host) 2022-05-14T04:06:23.555 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:06:23 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T04:06:23.555 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:06:23 smithi047 conmon[78521]: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T04:06:23.555 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:06:23 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T04:06:23.555 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:06:23 smithi047 conmon[78521]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T04:06:23.555 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:06:23 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T04:06:23.556 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:06:23 smithi047 conmon[78521]: return self.event_loop.get_result(coro) 2022-05-14T04:06:23.556 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:06:23 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T04:06:23.556 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:06:23 smithi047 conmon[78521]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T04:06:23.556 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:06:23 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T04:06:23.556 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:06:23 smithi047 conmon[78521]: return self.__get_result() 2022-05-14T04:06:23.557 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:06:23 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T04:06:23.557 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:06:23 smithi047 conmon[78521]: raise self._exception 2022-05-14T04:06:23.557 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:06:23 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T04:06:23.557 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:06:23 smithi047 conmon[78521]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T04:06:23.558 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:06:23 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T04:06:23.558 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:06:23 smithi047 conmon[78521]: conn = await self._remote_connection(host, addr) 2022-05-14T04:06:23.558 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:06:23 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T04:06:23.558 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:06:23 smithi047 conmon[78521]: raise 2022-05-14T04:06:23.558 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:06:23 smithi047 conmon[78521]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T04:06:23.559 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:06:23 smithi047 conmon[78521]: self.gen.throw(type, value, traceback) 2022-05-14T04:06:23.559 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:06:23 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T04:06:23.559 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:06:23 smithi047 conmon[78521]: raise OrchestratorError(msg) 2022-05-14T04:06:23.560 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:06:23 smithi047 conmon[78521]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:06:23.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:06:23 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:06:23.610 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:06:23 smithi115 conmon[90393]: 2022-05-14T04:06:23.469+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:06:24.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:06:24 smithi047 ceph-mon[77147]: pgmap v8709: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:06:24.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:06:24 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:06:24.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T04:06:24.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:06:24.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T04:06:24.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:06:24.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:06:24.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:06:24.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T04:06:24.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:06:24.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T04:06:24.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:06:24.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T04:06:24.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:06:24.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T04:06:24.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:06:24.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T04:06:24.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:06:24.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:06:24.548 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:06:24.548 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:06:24 smithi047 ceph-mon[77147]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T04:06:24.548 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T04:06:24.548 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:06:24.549 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T04:06:24.549 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:06:24.549 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:06:24.549 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:06:24.550 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T04:06:24.550 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:06:24.550 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T04:06:24.550 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:06:24.550 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T04:06:24.551 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:06:24.551 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T04:06:24.551 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:06:24.551 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T04:06:24.552 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:06:24.552 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:06:24.552 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:06:24.552 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T04:06:24.553 INFO:journalctl@ceph.mon.b.smithi047.stdout: During handling of the above exception, another exception occurred: 2022-05-14T04:06:24.553 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T04:06:24.554 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T04:06:24.554 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T04:06:24.554 INFO:journalctl@ceph.mon.b.smithi047.stdout: return f(*arg) 2022-05-14T04:06:24.554 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T04:06:24.555 INFO:journalctl@ceph.mon.b.smithi047.stdout: self._write_client_files(client_files, host) 2022-05-14T04:06:24.555 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T04:06:24.555 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T04:06:24.555 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T04:06:24.556 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T04:06:24.556 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T04:06:24.556 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.event_loop.get_result(coro) 2022-05-14T04:06:24.557 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T04:06:24.557 INFO:journalctl@ceph.mon.b.smithi047.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T04:06:24.557 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T04:06:24.557 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.__get_result() 2022-05-14T04:06:24.557 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T04:06:24.558 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise self._exception 2022-05-14T04:06:24.558 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T04:06:24.558 INFO:journalctl@ceph.mon.b.smithi047.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T04:06:24.559 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T04:06:24.559 INFO:journalctl@ceph.mon.b.smithi047.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T04:06:24.559 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T04:06:24.560 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise 2022-05-14T04:06:24.560 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T04:06:24.560 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.gen.throw(type, value, traceback) 2022-05-14T04:06:24.560 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T04:06:24.561 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OrchestratorError(msg) 2022-05-14T04:06:24.561 INFO:journalctl@ceph.mon.b.smithi047.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:06:24.561 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:06:24 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:06:24.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:06:24 smithi115 conmon[90393]: 2022-05-14T04:06:24.427+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:06:24.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:06:24 smithi115 ceph-mon[78593]: pgmap v8709: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:06:24.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:06:24 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:06:24.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T04:06:24.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:06:24.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T04:06:24.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:06:24.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:06:24.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:06:24.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T04:06:24.613 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:06:24.613 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T04:06:24.613 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:06:24.613 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T04:06:24.613 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:06:24.614 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T04:06:24.614 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:06:24.614 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T04:06:24.614 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:06:24.615 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:06:24.615 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:06:24.615 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:06:24 smithi115 ceph-mon[78593]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T04:06:24.615 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T04:06:24.615 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:06:24.616 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T04:06:24.616 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:06:24.617 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:06:24.617 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:06:24.618 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T04:06:24.618 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:06:24.618 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T04:06:24.618 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:06:24.619 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T04:06:24.619 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:06:24.619 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T04:06:24.619 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:06:24.619 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T04:06:24.620 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:06:24.620 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:06:24.621 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:06:24.621 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T04:06:24.621 INFO:journalctl@ceph.mon.c.smithi115.stdout: During handling of the above exception, another exception occurred: 2022-05-14T04:06:24.621 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T04:06:24.622 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T04:06:24.622 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T04:06:24.622 INFO:journalctl@ceph.mon.c.smithi115.stdout: return f(*arg) 2022-05-14T04:06:24.622 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T04:06:24.623 INFO:journalctl@ceph.mon.c.smithi115.stdout: self._write_client_files(client_files, host) 2022-05-14T04:06:24.625 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T04:06:24.625 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T04:06:24.625 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T04:06:24.625 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T04:06:24.626 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T04:06:24.626 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.event_loop.get_result(coro) 2022-05-14T04:06:24.626 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T04:06:24.626 INFO:journalctl@ceph.mon.c.smithi115.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T04:06:24.627 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T04:06:24.627 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.__get_result() 2022-05-14T04:06:24.627 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T04:06:24.628 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise self._exception 2022-05-14T04:06:24.628 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T04:06:24.630 INFO:journalctl@ceph.mon.c.smithi115.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T04:06:24.630 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T04:06:24.630 INFO:journalctl@ceph.mon.c.smithi115.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T04:06:24.631 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T04:06:24.631 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise 2022-05-14T04:06:24.631 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T04:06:24.632 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.gen.throw(type, value, traceback) 2022-05-14T04:06:24.632 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T04:06:24.632 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OrchestratorError(msg) 2022-05-14T04:06:24.632 INFO:journalctl@ceph.mon.c.smithi115.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:06:24.633 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:06:24 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:06:25.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:06:25 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:06:25.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:06:25 smithi115 conmon[90393]: 2022-05-14T04:06:25.401+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:06:25.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:06:25 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:06:26.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:06:26 smithi047 ceph-mon[77147]: pgmap v8710: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:06:26.545 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:06:26 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:06:26.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:06:26 smithi115 conmon[90393]: 2022-05-14T04:06:26.425+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:06:26.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:06:26 smithi115 ceph-mon[78593]: pgmap v8710: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:06:26.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:06:26 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:06:27.546 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:06:27 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:06:27.546 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:06:27 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 16720 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:06:27.610 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:06:27 smithi115 conmon[90393]: 2022-05-14T04:06:27.389+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:06:27.612 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:06:27 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:06:27.612 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:06:27 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 16720 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:06:28.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:06:28 smithi047 ceph-mon[77147]: pgmap v8711: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:06:28.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:06:28 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:06:28.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:06:28 smithi115 conmon[90393]: 2022-05-14T04:06:28.344+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:06:28.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:06:28 smithi115 ceph-mon[78593]: pgmap v8711: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:06:28.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:06:28 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:06:29.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:06:29 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:06:29.611 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:06:29 smithi115 conmon[90393]: 2022-05-14T04:06:29.319+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:06:29.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:06:29 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:06:30.612 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:06:30 smithi115 conmon[90393]: 2022-05-14T04:06:30.357+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:06:30.612 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:06:30 smithi115 ceph-mon[78593]: pgmap v8712: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:06:30.613 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:06:30 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:06:30.613 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:06:30 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:06:30.613 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:06:30 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:06:30.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:06:30 smithi047 ceph-mon[77147]: pgmap v8712: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:06:30.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:06:30 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:06:30.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:06:30 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:06:30.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:06:30 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:06:31.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:06:31 smithi115 conmon[90393]: 2022-05-14T04:06:31.343+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:06:31.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:06:31 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:06:31.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:06:31 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:06:32.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:06:32 smithi115 conmon[90393]: 2022-05-14T04:06:32.322+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:06:32.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:06:32 smithi115 ceph-mon[78593]: pgmap v8713: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:06:32.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:06:32 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:06:32.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:06:32 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 16725 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:06:32.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:06:32 smithi047 ceph-mon[77147]: pgmap v8713: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:06:32.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:06:32 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:06:32.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:06:32 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 16725 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:06:33.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:06:33 smithi115 conmon[90393]: 2022-05-14T04:06:33.355+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:06:33.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:06:33 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:06:33.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:06:33 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:06:34.612 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:06:34 smithi115 conmon[90393]: 2022-05-14T04:06:34.398+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:06:34.613 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:06:34 smithi115 ceph-mon[78593]: pgmap v8714: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:06:34.613 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:06:34 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:06:34.613 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:06:34 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:06:34.613 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:06:34 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:06:34.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:06:34 smithi047 ceph-mon[77147]: pgmap v8714: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:06:34.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:06:34 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:06:34.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:06:34 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:06:34.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:06:34 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:06:35.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:06:35 smithi115 conmon[90393]: 2022-05-14T04:06:35.401+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:06:35.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:06:35 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:06:35.796 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:06:35 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:06:36.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:06:36 smithi115 conmon[90393]: 2022-05-14T04:06:36.354+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:06:36.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:06:36 smithi115 ceph-mon[78593]: pgmap v8715: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:06:36.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:06:36 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:06:36.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:06:36 smithi047 ceph-mon[77147]: pgmap v8715: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:06:36.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:06:36 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:06:37.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:06:37 smithi115 conmon[90393]: 2022-05-14T04:06:37.400+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:06:37.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:06:37 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:06:37.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:06:37 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 16730 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:06:37.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:06:37 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:06:37.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:06:37 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 16730 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:06:38.610 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:06:38 smithi115 conmon[90393]: 2022-05-14T04:06:38.365+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:06:38.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:06:38 smithi115 ceph-mon[78593]: pgmap v8716: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:06:38.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:06:38 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:06:38.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:06:38 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:06:38.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:06:38 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:06:38.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:06:38 smithi047 ceph-mon[77147]: pgmap v8716: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:06:38.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:06:38 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:06:38.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:06:38 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:06:38.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:06:38 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:06:39.611 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:06:39 smithi115 conmon[90393]: 2022-05-14T04:06:39.395+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:06:39.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:06:39 smithi115 ceph-mon[78593]: pgmap v8717: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:06:39.612 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:06:39 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:06:39.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:06:39 smithi047 ceph-mon[77147]: pgmap v8717: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:06:39.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:06:39 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:06:40.610 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:06:40 smithi115 conmon[90393]: 2022-05-14T04:06:40.366+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:06:40.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:06:40 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:06:40.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:06:40 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:06:41.610 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:06:41 smithi115 conmon[90393]: 2022-05-14T04:06:41.397+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:06:41.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:06:41 smithi115 ceph-mon[78593]: pgmap v8718: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:06:41.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:06:41 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:06:41.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:06:41 smithi047 ceph-mon[77147]: pgmap v8718: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:06:41.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:06:41 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:06:42.610 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:06:42 smithi115 conmon[90393]: 2022-05-14T04:06:42.380+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:06:42.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:06:42 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:06:42.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:06:42 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:06:42.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:06:42 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:06:42.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:06:42 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 16735 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:06:42.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:06:42 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:06:42.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:06:42 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:06:42.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:06:42 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:06:42.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:06:42 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 16735 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:06:43.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:06:43 smithi115 conmon[90393]: 2022-05-14T04:06:43.419+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:06:43.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:06:43 smithi115 ceph-mon[78593]: pgmap v8719: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:06:43.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:06:43 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:06:43.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:06:43 smithi047 ceph-mon[77147]: pgmap v8719: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:06:43.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:06:43 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:06:44.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:06:44 smithi115 conmon[90393]: 2022-05-14T04:06:44.459+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:06:44.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:06:44 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:06:44.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:06:44 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:06:45.611 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:06:45 smithi115 conmon[90393]: 2022-05-14T04:06:45.464+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:06:45.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:06:45 smithi115 ceph-mon[78593]: pgmap v8720: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:06:45.612 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:06:45 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:06:45.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:06:45 smithi047 ceph-mon[77147]: pgmap v8720: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:06:45.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:06:45 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:06:46.613 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:06:46 smithi115 conmon[90393]: 2022-05-14T04:06:46.469+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:06:46.614 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:06:46 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:06:46.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:06:46 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:06:47.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:06:47 smithi115 conmon[90393]: 2022-05-14T04:06:47.483+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:06:47.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:06:47 smithi115 ceph-mon[78593]: pgmap v8721: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:06:47.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:06:47 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:06:47.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:06:47 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 16740 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:06:47.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:06:47 smithi047 ceph-mon[77147]: pgmap v8721: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:06:47.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:06:47 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:06:47.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:06:47 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 16740 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:06:48.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:06:48 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:06:48.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:06:48 smithi115 conmon[90393]: 2022-05-14T04:06:48.482+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:06:48.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:06:48 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:06:49.798 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:06:49 smithi047 ceph-mon[77147]: pgmap v8722: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:06:49.798 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:06:49 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:06:49.863 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:06:49 smithi115 conmon[90393]: 2022-05-14T04:06:49.497+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:06:49.864 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:06:49 smithi115 ceph-mon[78593]: pgmap v8722: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:06:49.870 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:06:49 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:06:50.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:06:50 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:06:50.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:06:50 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:06:50.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:06:50 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:06:50.862 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:06:50 smithi115 conmon[90393]: 2022-05-14T04:06:50.498+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:06:50.863 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:06:50 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:06:50.863 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:06:50 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:06:50.864 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:06:50 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:06:51.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:06:51 smithi047 ceph-mon[77147]: pgmap v8723: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:06:51.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:06:51 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:06:51.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:06:51 smithi115 conmon[90393]: 2022-05-14T04:06:51.540+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:06:51.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:06:51 smithi115 ceph-mon[78593]: pgmap v8723: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:06:51.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:06:51 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:06:52.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:06:52 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:06:52.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:06:52 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 16745 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:06:52.864 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:06:52 smithi115 conmon[90393]: 2022-05-14T04:06:52.589+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:06:52.865 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:06:52 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:06:52.865 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:06:52 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 16745 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:06:53.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:06:53 smithi115 conmon[90393]: 2022-05-14T04:06:53.602+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:06:54.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:06:54 smithi047 ceph-mon[77147]: pgmap v8724: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:06:54.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:06:54 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:06:54.580 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:06:54 smithi115 ceph-mon[78593]: pgmap v8724: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:06:54.581 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:06:54 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:06:54.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:06:54 smithi115 conmon[90393]: 2022-05-14T04:06:54.577+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:06:55.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:06:55 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:06:55.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:06:55 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:06:55.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:06:55 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:06:55.573 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:06:55 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:06:55.573 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:06:55 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:06:55.573 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:06:55 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:06:55.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:06:55 smithi115 conmon[90393]: 2022-05-14T04:06:55.570+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:06:56.546 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:06:56 smithi047 ceph-mon[77147]: pgmap v8725: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:06:56.546 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:06:56 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:06:56.601 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:06:56 smithi115 ceph-mon[78593]: pgmap v8725: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:06:56.602 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:06:56 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:06:56.860 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:06:56 smithi115 conmon[90393]: 2022-05-14T04:06:56.593+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:06:57.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:06:57 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:06:57.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:06:57 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 16750 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:06:57.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:06:57 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:06:57.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:06:57 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 16750 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:06:58.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:06:57 smithi115 conmon[90393]: 2022-05-14T04:06:57.614+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:06:58.547 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:06:58 smithi047 ceph-mon[77147]: pgmap v8726: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:06:58.547 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:06:58 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:06:58.548 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:06:58 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:06:58.548 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:06:58 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:06:58.571 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:06:58 smithi115 ceph-mon[78593]: pgmap v8726: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:06:58.571 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:06:58 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:06:58.571 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:06:58 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:06:58.572 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:06:58 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:06:58.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:06:58 smithi115 conmon[90393]: 2022-05-14T04:06:58.567+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:06:59.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:06:59 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:06:59.592 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:06:59 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:06:59.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:06:59 smithi115 conmon[90393]: 2022-05-14T04:06:59.588+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:07:00.549 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:07:00 smithi047 ceph-mon[77147]: pgmap v8727: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:07:00.550 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:07:00 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:07:00.602 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:07:00 smithi115 ceph-mon[78593]: pgmap v8727: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:07:00.602 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:07:00 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:07:00.873 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:07:00 smithi115 conmon[90393]: 2022-05-14T04:07:00.598+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:07:01.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:07:01 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:07:01.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:07:01 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:07:02.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:07:01 smithi115 conmon[90393]: 2022-05-14T04:07:01.632+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:07:02.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:07:02 smithi047 ceph-mon[77147]: pgmap v8728: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:07:02.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:07:02 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:07:02.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:07:02 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:07:02.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:07:02 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:07:02.545 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:07:02 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 16755 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:07:02.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:07:02 smithi115 ceph-mon[78593]: pgmap v8728: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:07:02.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:07:02 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:07:02.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:07:02 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:07:02.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:07:02 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:07:02.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:07:02 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 16755 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:07:03.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:07:02 smithi115 conmon[90393]: 2022-05-14T04:07:02.649+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:07:03.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:07:03 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:07:03.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:07:03 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:07:04.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:07:03 smithi115 conmon[90393]: 2022-05-14T04:07:03.648+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:07:04.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:07:04 smithi047 ceph-mon[77147]: pgmap v8729: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:07:04.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:07:04 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:07:04.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:07:04 smithi115 ceph-mon[78593]: pgmap v8729: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:07:04.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:07:04 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:07:05.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:07:04 smithi115 conmon[90393]: 2022-05-14T04:07:04.682+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:07:05.554 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:07:05 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:07:05.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:07:05 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:07:06.110 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:07:05 smithi115 conmon[90393]: 2022-05-14T04:07:05.692+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:07:06.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:07:06 smithi047 ceph-mon[77147]: pgmap v8730: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:07:06.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:07:06 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:07:06.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:07:06 smithi115 ceph-mon[78593]: pgmap v8730: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:07:06.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:07:06 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:07:07.115 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:07:06 smithi115 conmon[90393]: 2022-05-14T04:07:06.651+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:07:07.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:07:07 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:07:07.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:07:07 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 16760 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:07:07.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:07:07 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:07:07.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:07:07 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 16760 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:07:08.110 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:07:07 smithi115 conmon[90393]: 2022-05-14T04:07:07.662+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:07:08.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:07:08 smithi047 ceph-mon[77147]: pgmap v8731: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:07:08.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:07:08 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:07:08.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:07:08 smithi115 ceph-mon[78593]: pgmap v8731: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:07:08.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:07:08 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:07:09.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:07:08 smithi115 conmon[90393]: 2022-05-14T04:07:08.643+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:07:09.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:07:09 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:07:09.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:07:09 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:07:10.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:07:09 smithi115 conmon[90393]: 2022-05-14T04:07:09.638+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:07:10.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:07:10 smithi047 ceph-mon[77147]: pgmap v8732: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:07:10.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:07:10 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:07:10.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:07:10 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:07:10.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:07:10 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:07:10.606 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:07:10 smithi115 ceph-mon[78593]: pgmap v8732: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:07:10.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:07:10 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:07:10.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:07:10 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:07:10.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:07:10 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:07:10.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:07:10 smithi115 conmon[90393]: 2022-05-14T04:07:10.603+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:07:11.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:07:11 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:07:11.582 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:07:11 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:07:11.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:07:11 smithi115 conmon[90393]: 2022-05-14T04:07:11.579+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:07:12.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:07:12 smithi047 ceph-mon[77147]: pgmap v8733: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:07:12.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:07:12 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:07:12.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:07:12 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 16765 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:07:12.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:07:12 smithi115 ceph-mon[78593]: pgmap v8733: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:07:12.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:07:12 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:07:12.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:07:12 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 16765 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:07:13.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:07:12 smithi115 conmon[90393]: 2022-05-14T04:07:12.619+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:07:13.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:07:13 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:07:13.590 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:07:13 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:07:13.860 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:07:13 smithi115 conmon[90393]: 2022-05-14T04:07:13.587+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:07:14.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:07:14 smithi047 ceph-mon[77147]: pgmap v8734: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:07:14.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:07:14 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:07:14.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:07:14 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:07:14.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:07:14 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:07:14.559 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:07:14 smithi115 ceph-mon[78593]: pgmap v8734: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:07:14.560 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:07:14 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:07:14.560 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:07:14 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:07:14.560 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:07:14 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:07:14.860 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:07:14 smithi115 conmon[90393]: 2022-05-14T04:07:14.556+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:07:15.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:07:15 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:07:15.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:07:15 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T04:07:15.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:07:15 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T04:07:15.570 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:07:15 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:07:15.570 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:07:15 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T04:07:15.570 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:07:15 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T04:07:15.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:07:15 smithi115 conmon[90393]: 2022-05-14T04:07:15.565+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:07:16.531 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:07:16 smithi115 ceph-mon[78593]: pgmap v8735: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:07:16.531 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:07:16 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:07:16.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:07:16 smithi047 ceph-mon[77147]: pgmap v8735: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:07:16.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:07:16 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:07:16.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:07:16 smithi115 conmon[90393]: 2022-05-14T04:07:16.527+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:07:17.506 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:07:17 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:07:17.506 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:07:17 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 16770 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:07:17.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:07:17 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:07:17.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:07:17 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 16770 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:07:17.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:07:17 smithi115 conmon[90393]: 2022-05-14T04:07:17.503+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:07:18.464 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:07:18 smithi115 ceph-mon[78593]: pgmap v8736: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:07:18.465 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:07:18 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:07:18.465 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:07:18 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:07:18.465 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:07:18 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:07:18.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:07:18 smithi047 ceph-mon[77147]: pgmap v8736: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:07:18.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:07:18 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:07:18.545 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:07:18 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:07:18.545 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:07:18 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:07:18.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:07:18 smithi115 conmon[90393]: 2022-05-14T04:07:18.461+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:07:19.476 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:07:19 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:07:19.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:07:19 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:07:19.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:07:19 smithi115 conmon[90393]: 2022-05-14T04:07:19.473+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:07:20.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:07:20 smithi047 ceph-mon[77147]: pgmap v8737: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:07:20.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:07:20 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:07:20.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:07:20 smithi115 conmon[90393]: 2022-05-14T04:07:20.463+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:07:20.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:07:20 smithi115 ceph-mon[78593]: pgmap v8737: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:07:20.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:07:20 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:07:21.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:07:21 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:07:21.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:07:21 smithi115 conmon[90393]: 2022-05-14T04:07:21.424+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:07:21.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:07:21 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:07:22.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:07:22 smithi047 ceph-mon[77147]: pgmap v8738: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:07:22.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:07:22 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:07:22.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:07:22 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:07:22.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:07:22 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:07:22.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:07:22 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 16775 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:07:22.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:07:22 smithi115 conmon[90393]: 2022-05-14T04:07:22.451+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:07:22.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:07:22 smithi115 ceph-mon[78593]: pgmap v8738: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:07:22.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:07:22 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:07:22.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:07:22 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:07:22.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:07:22 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:07:22.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:07:22 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 16775 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:07:23.481 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:07:23 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:07:23.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:07:23 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:07:23.795 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:07:23 smithi115 conmon[90393]: 2022-05-14T04:07:23.477+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:07:24.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:07:24 smithi047 ceph-mon[77147]: pgmap v8739: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:07:24.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:07:24 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:07:24.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:07:24 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T04:07:24.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:07:24 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T04:07:24.545 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:07:24 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T04:07:24.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:07:24 smithi115 conmon[90393]: 2022-05-14T04:07:24.455+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:07:24.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:07:24 smithi115 ceph-mon[78593]: pgmap v8739: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:07:24.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:07:24 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:07:24.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:07:24 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T04:07:24.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:07:24 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T04:07:24.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:07:24 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T04:07:25.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:07:25 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:07:25.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:07:25 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:07:25.610 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:07:25 smithi115 conmon[90393]: 2022-05-14T04:07:25.476+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:07:26.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:07:26 smithi047 ceph-mon[77147]: pgmap v8740: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:07:26.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:07:26 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:07:26.545 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:07:26 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:07:26.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:07:26 smithi047 conmon[78521]: 2022-05-14T04:07:26.384+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:07:26.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:07:26 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T04:07:26.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:07:26 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:07:26.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:07:26 smithi047 conmon[78521]: yield 2022-05-14T04:07:26.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:07:26 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:07:26.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:07:26 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:07:26.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:07:26 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:07:26.547 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:07:26 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T04:07:26.547 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:07:26 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:07:26.547 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:07:26 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T04:07:26.547 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:07:26 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:07:26.548 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:07:26 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T04:07:26.548 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:07:26 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:07:26.548 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:07:26 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T04:07:26.548 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:07:26 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:07:26.549 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:07:26 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T04:07:26.549 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:07:26 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:07:26.549 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:07:26 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:07:26.549 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:07:26 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:07:26.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:07:26 smithi115 conmon[90393]: 2022-05-14T04:07:26.477+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:07:26.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:07:26 smithi115 ceph-mon[78593]: pgmap v8740: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:07:26.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:07:26 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:07:26.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:07:26 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:07:27.494 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:07:27 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:07:27.494 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:07:27 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 16780 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:07:27.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:07:27 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:07:27.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:07:27 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 16780 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:07:27.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:07:27 smithi115 conmon[90393]: 2022-05-14T04:07:27.491+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:07:28.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:07:28 smithi047 ceph-mon[77147]: pgmap v8741: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:07:28.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:07:28 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:07:28.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T04:07:28.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:07:28.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T04:07:28.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:07:28.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:07:28.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:07:28.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T04:07:28.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:07:28.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T04:07:28.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:07:28.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T04:07:28.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:07:28.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T04:07:28.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:07:28.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T04:07:28.548 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:07:28.548 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:07:28.548 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:07:28.548 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:07:28 smithi047 ceph-mon[77147]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T04:07:28.548 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:07:28 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:07:28.610 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:07:28 smithi115 conmon[90393]: 2022-05-14T04:07:28.484+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:07:28.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:07:28 smithi115 ceph-mon[78593]: pgmap v8741: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:07:28.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:07:28 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:07:28.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T04:07:28.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:07:28.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T04:07:28.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:07:28.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:07:28.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:07:28.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T04:07:28.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:07:28.614 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T04:07:28.614 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:07:28.614 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T04:07:28.614 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:07:28.615 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T04:07:28.615 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:07:28.615 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T04:07:28.615 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:07:28.616 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:07:28.616 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:07:28.616 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:07:28 smithi115 ceph-mon[78593]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T04:07:28.616 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:07:28 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:07:29.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:07:29 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:07:29.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:07:29 smithi047 conmon[78521]: 2022-05-14T04:07:29.456+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:07:29.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:07:29 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T04:07:29.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:07:29 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:07:29.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:07:29 smithi047 conmon[78521]: yield 2022-05-14T04:07:29.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:07:29 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:07:29.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:07:29 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:07:29.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:07:29 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:07:29.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:07:29 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T04:07:29.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:07:29 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:07:29.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:07:29 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T04:07:29.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:07:29 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:07:29.547 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:07:29 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T04:07:29.547 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:07:29 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:07:29.547 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:07:29 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T04:07:29.547 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:07:29 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:07:29.548 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:07:29 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T04:07:29.548 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:07:29 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:07:29.548 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:07:29 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:07:29.548 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:07:29 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:07:29.548 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:07:29 smithi047 conmon[78521]: 2022-05-14T04:07:29.457+0000 7f533ff78700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T04:07:29.549 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:07:29 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T04:07:29.549 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:07:29 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:07:29.549 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:07:29 smithi047 conmon[78521]: yield 2022-05-14T04:07:29.549 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:07:29 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:07:29.550 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:07:29 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:07:29.550 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:07:29 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:07:29.550 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:07:29 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T04:07:29.550 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:07:29 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:07:29.550 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:07:29 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T04:07:29.551 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:07:29 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:07:29.551 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:07:29 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T04:07:29.551 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:07:29 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:07:29.551 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:07:29 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T04:07:29.551 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:07:29 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:07:29.552 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:07:29 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T04:07:29.552 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:07:29 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:07:29.552 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:07:29 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:07:29.552 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:07:29 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:07:29.552 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:07:29 smithi047 conmon[78521]: 2022-05-14T04:07:29.555 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:07:29 smithi047 conmon[78521]: During handling of the above exception, another exception occurred: 2022-05-14T04:07:29.555 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:07:29 smithi047 conmon[78521]: 2022-05-14T04:07:29.555 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:07:29 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T04:07:29.555 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:07:29 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T04:07:29.556 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:07:29 smithi047 conmon[78521]: return f(*arg) 2022-05-14T04:07:29.556 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:07:29 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T04:07:29.556 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:07:29 smithi047 conmon[78521]: self._write_client_files(client_files, host) 2022-05-14T04:07:29.556 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:07:29 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T04:07:29.557 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:07:29 smithi047 conmon[78521]: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T04:07:29.557 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:07:29 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T04:07:29.557 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:07:29 smithi047 conmon[78521]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T04:07:29.557 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:07:29 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T04:07:29.557 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:07:29 smithi047 conmon[78521]: return self.event_loop.get_result(coro) 2022-05-14T04:07:29.558 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:07:29 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T04:07:29.558 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:07:29 smithi047 conmon[78521]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T04:07:29.558 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:07:29 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T04:07:29.559 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:07:29 smithi047 conmon[78521]: return self.__get_result() 2022-05-14T04:07:29.559 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:07:29 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T04:07:29.560 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:07:29 smithi047 conmon[78521]: raise self._exception 2022-05-14T04:07:29.560 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:07:29 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T04:07:29.560 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:07:29 smithi047 conmon[78521]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T04:07:29.561 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:07:29 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T04:07:29.561 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:07:29 smithi047 conmon[78521]: conn = await self._remote_connection(host, addr) 2022-05-14T04:07:29.561 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:07:29 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T04:07:29.561 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:07:29 smithi047 conmon[78521]: raise 2022-05-14T04:07:29.561 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:07:29 smithi047 conmon[78521]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T04:07:29.562 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:07:29 smithi047 conmon[78521]: self.gen.throw(type, value, traceback) 2022-05-14T04:07:29.563 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:07:29 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T04:07:29.563 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:07:29 smithi047 conmon[78521]: raise OrchestratorError(msg) 2022-05-14T04:07:29.563 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:07:29 smithi047 conmon[78521]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:07:29.610 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:07:29 smithi115 conmon[90393]: 2022-05-14T04:07:29.435+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:07:29.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:07:29 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:07:30.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:07:30 smithi047 ceph-mon[77147]: pgmap v8742: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:07:30.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:07:30 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:07:30.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:07:30 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:07:30.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:07:30 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:07:30.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:07:30 smithi115 conmon[90393]: 2022-05-14T04:07:30.468+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:07:30.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:07:30 smithi115 ceph-mon[78593]: pgmap v8742: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:07:30.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:07:30 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:07:30.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:07:30 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:07:30.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:07:30 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:07:31.517 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:07:31 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:07:31.517 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T04:07:31.518 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:07:31.518 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T04:07:31.518 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:07:31.518 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:07:31.519 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:07:31.519 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T04:07:31.519 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:07:31.519 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T04:07:31.519 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:07:31.520 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T04:07:31.520 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:07:31.520 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T04:07:31.520 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:07:31.520 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T04:07:31.521 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:07:31.521 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:07:31.521 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:07:31.521 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:07:31 smithi115 ceph-mon[78593]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T04:07:31.522 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T04:07:31.522 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:07:31.522 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T04:07:31.522 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:07:31.522 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:07:31.523 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:07:31.523 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T04:07:31.523 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:07:31.523 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T04:07:31.523 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:07:31.524 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T04:07:31.524 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:07:31.524 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T04:07:31.524 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:07:31.524 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T04:07:31.525 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:07:31.525 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:07:31.525 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:07:31.525 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T04:07:31.526 INFO:journalctl@ceph.mon.c.smithi115.stdout: During handling of the above exception, another exception occurred: 2022-05-14T04:07:31.526 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T04:07:31.526 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T04:07:31.526 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T04:07:31.526 INFO:journalctl@ceph.mon.c.smithi115.stdout: return f(*arg) 2022-05-14T04:07:31.527 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T04:07:31.527 INFO:journalctl@ceph.mon.c.smithi115.stdout: self._write_client_files(client_files, host) 2022-05-14T04:07:31.527 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T04:07:31.527 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T04:07:31.527 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T04:07:31.528 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T04:07:31.528 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T04:07:31.528 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.event_loop.get_result(coro) 2022-05-14T04:07:31.528 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T04:07:31.529 INFO:journalctl@ceph.mon.c.smithi115.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T04:07:31.529 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T04:07:31.529 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.__get_result() 2022-05-14T04:07:31.529 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T04:07:31.529 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise self._exception 2022-05-14T04:07:31.530 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T04:07:31.530 INFO:journalctl@ceph.mon.c.smithi115.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T04:07:31.530 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T04:07:31.530 INFO:journalctl@ceph.mon.c.smithi115.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T04:07:31.530 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T04:07:31.531 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise 2022-05-14T04:07:31.531 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T04:07:31.531 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.gen.throw(type, value, traceback) 2022-05-14T04:07:31.531 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T04:07:31.531 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OrchestratorError(msg) 2022-05-14T04:07:31.532 INFO:journalctl@ceph.mon.c.smithi115.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:07:31.532 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:07:31 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:07:31.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:07:31 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:07:31.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T04:07:31.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:07:31.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T04:07:31.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:07:31.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:07:31.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:07:31.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T04:07:31.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:07:31.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T04:07:31.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:07:31.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T04:07:31.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:07:31.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T04:07:31.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:07:31.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T04:07:31.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:07:31.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:07:31.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:07:31.547 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:07:31 smithi047 ceph-mon[77147]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T04:07:31.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T04:07:31.548 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:07:31.548 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T04:07:31.548 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:07:31.548 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:07:31.548 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:07:31.549 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T04:07:31.549 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:07:31.549 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T04:07:31.549 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:07:31.549 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T04:07:31.550 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:07:31.550 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T04:07:31.550 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:07:31.550 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T04:07:31.551 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:07:31.551 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:07:31.551 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:07:31.551 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T04:07:31.551 INFO:journalctl@ceph.mon.b.smithi047.stdout: During handling of the above exception, another exception occurred: 2022-05-14T04:07:31.552 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T04:07:31.552 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T04:07:31.552 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T04:07:31.552 INFO:journalctl@ceph.mon.b.smithi047.stdout: return f(*arg) 2022-05-14T04:07:31.552 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T04:07:31.553 INFO:journalctl@ceph.mon.b.smithi047.stdout: self._write_client_files(client_files, host) 2022-05-14T04:07:31.553 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T04:07:31.553 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T04:07:31.553 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T04:07:31.554 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T04:07:31.554 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T04:07:31.554 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.event_loop.get_result(coro) 2022-05-14T04:07:31.554 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T04:07:31.555 INFO:journalctl@ceph.mon.b.smithi047.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T04:07:31.555 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T04:07:31.555 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.__get_result() 2022-05-14T04:07:31.555 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T04:07:31.555 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise self._exception 2022-05-14T04:07:31.556 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T04:07:31.556 INFO:journalctl@ceph.mon.b.smithi047.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T04:07:31.556 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T04:07:31.556 INFO:journalctl@ceph.mon.b.smithi047.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T04:07:31.556 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T04:07:31.557 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise 2022-05-14T04:07:31.557 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T04:07:31.557 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.gen.throw(type, value, traceback) 2022-05-14T04:07:31.557 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T04:07:31.558 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OrchestratorError(msg) 2022-05-14T04:07:31.558 INFO:journalctl@ceph.mon.b.smithi047.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:07:31.558 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:07:31 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:07:31.861 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:07:31 smithi115 conmon[90393]: 2022-05-14T04:07:31.514+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:07:32.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:07:32 smithi047 ceph-mon[77147]: pgmap v8743: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:07:32.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:07:32 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:07:32.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:07:32 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 16785 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:07:32.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:07:32 smithi115 conmon[90393]: 2022-05-14T04:07:32.466+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:07:32.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:07:32 smithi115 ceph-mon[78593]: pgmap v8743: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:07:32.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:07:32 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:07:32.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:07:32 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 16785 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:07:33.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:07:33 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:07:33.610 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:07:33 smithi115 conmon[90393]: 2022-05-14T04:07:33.495+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:07:33.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:07:33 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:07:34.533 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:07:34 smithi115 ceph-mon[78593]: pgmap v8744: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:07:34.533 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:07:34 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:07:34.533 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:07:34 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:07:34.533 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:07:34 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:07:34.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:07:34 smithi047 ceph-mon[77147]: pgmap v8744: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:07:34.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:07:34 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:07:34.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:07:34 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:07:34.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:07:34 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:07:34.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:07:34 smithi115 conmon[90393]: 2022-05-14T04:07:34.530+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:07:35.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:07:35 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:07:35.611 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:07:35 smithi115 conmon[90393]: 2022-05-14T04:07:35.490+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:07:35.612 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:07:35 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:07:36.531 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:07:36 smithi115 ceph-mon[78593]: pgmap v8745: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:07:36.532 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:07:36 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:07:36.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:07:36 smithi047 ceph-mon[77147]: pgmap v8745: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:07:36.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:07:36 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:07:36.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:07:36 smithi115 conmon[90393]: 2022-05-14T04:07:36.529+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:07:37.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:07:37 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:07:37.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:07:37 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 16790 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:07:37.550 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:07:37 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:07:37.550 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:07:37 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 16790 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:07:37.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:07:37 smithi115 conmon[90393]: 2022-05-14T04:07:37.547+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:07:38.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:07:38 smithi047 ceph-mon[77147]: pgmap v8746: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:07:38.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:07:38 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:07:38.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:07:38 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:07:38.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:07:38 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:07:38.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:07:38 smithi115 conmon[90393]: 2022-05-14T04:07:38.516+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:07:38.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:07:38 smithi115 ceph-mon[78593]: pgmap v8746: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:07:38.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:07:38 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:07:38.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:07:38 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:07:38.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:07:38 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:07:39.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:07:39 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:07:39.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:07:39 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:07:39.610 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:07:39 smithi115 conmon[90393]: 2022-05-14T04:07:39.473+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:07:40.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:07:40 smithi047 ceph-mon[77147]: pgmap v8747: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:07:40.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:07:40 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:07:40.610 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:07:40 smithi115 conmon[90393]: 2022-05-14T04:07:40.450+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:07:40.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:07:40 smithi115 ceph-mon[78593]: pgmap v8747: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:07:40.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:07:40 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:07:41.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:07:41 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:07:41.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:07:41 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:07:41.610 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:07:41 smithi115 conmon[90393]: 2022-05-14T04:07:41.447+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:07:42.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:07:42 smithi047 ceph-mon[77147]: pgmap v8748: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:07:42.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:07:42 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:07:42.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:07:42 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:07:42.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:07:42 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:07:42.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:07:42 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 16795 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:07:42.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:07:42 smithi115 conmon[90393]: 2022-05-14T04:07:42.412+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:07:42.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:07:42 smithi115 ceph-mon[78593]: pgmap v8748: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:07:42.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:07:42 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:07:42.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:07:42 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:07:42.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:07:42 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:07:42.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:07:42 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 16795 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:07:43.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:07:43 smithi115 conmon[90393]: 2022-05-14T04:07:43.369+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:07:43.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:07:43 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:07:43.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:07:43 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:07:44.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:07:44 smithi115 conmon[90393]: 2022-05-14T04:07:44.368+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:07:44.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:07:44 smithi115 ceph-mon[78593]: pgmap v8749: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:07:44.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:07:44 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:07:44.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:07:44 smithi047 ceph-mon[77147]: pgmap v8749: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:07:44.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:07:44 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:07:45.610 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:07:45 smithi115 conmon[90393]: 2022-05-14T04:07:45.374+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:07:45.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:07:45 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:07:45.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:07:45 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:07:46.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:07:46 smithi115 conmon[90393]: 2022-05-14T04:07:46.338+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:07:46.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:07:46 smithi115 ceph-mon[78593]: pgmap v8750: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:07:46.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:07:46 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:07:46.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:07:46 smithi047 ceph-mon[77147]: pgmap v8750: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:07:46.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:07:46 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:07:47.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:07:47 smithi115 conmon[90393]: 2022-05-14T04:07:47.323+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:07:47.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:07:47 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:07:47.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:07:47 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 16800 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:07:47.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:07:47 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:07:47.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:07:47 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 16800 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:07:48.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:07:48 smithi115 conmon[90393]: 2022-05-14T04:07:48.349+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:07:48.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:07:48 smithi115 ceph-mon[78593]: pgmap v8751: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:07:48.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:07:48 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:07:48.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:07:48 smithi047 ceph-mon[77147]: pgmap v8751: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:07:48.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:07:48 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:07:49.610 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:07:49 smithi115 conmon[90393]: 2022-05-14T04:07:49.350+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:07:49.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:07:49 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:07:49.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:07:49 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:07:50.616 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:07:50 smithi115 conmon[90393]: 2022-05-14T04:07:50.335+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:07:50.617 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:07:50 smithi115 ceph-mon[78593]: pgmap v8752: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:07:50.618 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:07:50 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:07:50.618 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:07:50 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:07:50.619 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:07:50 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:07:50.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:07:50 smithi047 ceph-mon[77147]: pgmap v8752: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:07:50.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:07:50 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:07:50.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:07:50 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:07:50.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:07:50 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:07:51.610 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:07:51 smithi115 conmon[90393]: 2022-05-14T04:07:51.368+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:07:51.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:07:51 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:07:51.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:07:51 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:07:52.610 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:07:52 smithi115 conmon[90393]: 2022-05-14T04:07:52.353+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:07:52.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:07:52 smithi115 ceph-mon[78593]: pgmap v8753: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:07:52.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:07:52 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:07:52.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:07:52 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 16805 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:07:52.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:07:52 smithi047 ceph-mon[77147]: pgmap v8753: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:07:52.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:07:52 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:07:52.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:07:52 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 16805 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:07:53.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:07:53 smithi115 conmon[90393]: 2022-05-14T04:07:53.349+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:07:53.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:07:53 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:07:53.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:07:53 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:07:54.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:07:54 smithi115 conmon[90393]: 2022-05-14T04:07:54.343+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:07:54.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:07:54 smithi115 ceph-mon[78593]: pgmap v8754: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:07:54.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:07:54 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:07:54.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:07:54 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:07:54.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:07:54 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:07:54.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:07:54 smithi047 ceph-mon[77147]: pgmap v8754: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:07:54.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:07:54 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:07:54.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:07:54 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:07:54.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:07:54 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:07:55.610 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:07:55 smithi115 conmon[90393]: 2022-05-14T04:07:55.299+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:07:55.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:07:55 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:07:55.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:07:55 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:07:56.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:07:56 smithi115 conmon[90393]: 2022-05-14T04:07:56.295+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:07:56.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:07:56 smithi115 ceph-mon[78593]: pgmap v8755: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:07:56.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:07:56 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:07:56.799 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:07:56 smithi047 ceph-mon[77147]: pgmap v8755: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:07:56.800 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:07:56 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:07:57.610 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:07:57 smithi115 conmon[90393]: 2022-05-14T04:07:57.314+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:07:57.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:07:57 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:07:57.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:07:57 smithi115 ceph-mon[78593]: pgmap v8756: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:07:57.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:07:57 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 16810 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:07:57.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:07:57 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:07:57.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:07:57 smithi047 ceph-mon[77147]: pgmap v8756: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:07:57.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:07:57 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 16810 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:07:58.610 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:07:58 smithi115 conmon[90393]: 2022-05-14T04:07:58.341+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:07:58.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:07:58 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:07:58.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:07:58 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:07:58.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:07:58 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:07:58.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:07:58 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:07:58.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:07:58 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:07:58.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:07:58 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:07:59.611 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:07:59 smithi115 conmon[90393]: 2022-05-14T04:07:59.381+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:07:59.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:07:59 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:07:59.612 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:07:59 smithi115 ceph-mon[78593]: pgmap v8757: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:07:59.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:07:59 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:07:59.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:07:59 smithi047 ceph-mon[77147]: pgmap v8757: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:08:00.613 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:08:00 smithi115 conmon[90393]: 2022-05-14T04:08:00.348+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:08:00.613 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:08:00 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:08:00.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:08:00 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:08:01.614 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:08:01 smithi115 conmon[90393]: 2022-05-14T04:08:01.370+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:08:01.614 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:08:01 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:08:01.615 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:08:01 smithi115 ceph-mon[78593]: pgmap v8758: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:08:01.797 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:08:01 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:08:01.797 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:08:01 smithi047 ceph-mon[77147]: pgmap v8758: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:08:02.618 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:08:02 smithi115 conmon[90393]: 2022-05-14T04:08:02.364+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:08:02.618 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:08:02 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:08:02.619 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:08:02 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:08:02.619 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:08:02 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:08:02.619 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:08:02 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 16815 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:08:02.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:08:02 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:08:02.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:08:02 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:08:02.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:08:02 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:08:02.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:08:02 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 16815 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:08:03.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:08:03 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:08:03.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:08:03 smithi047 ceph-mon[77147]: pgmap v8759: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:08:03.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:08:03 smithi115 conmon[90393]: 2022-05-14T04:08:03.381+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:08:03.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:08:03 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:08:03.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:08:03 smithi115 ceph-mon[78593]: pgmap v8759: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:08:04.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:08:04 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:08:04.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:08:04 smithi115 conmon[90393]: 2022-05-14T04:08:04.371+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:08:04.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:08:04 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:08:05.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:08:05 smithi115 conmon[90393]: 2022-05-14T04:08:05.334+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:08:05.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:08:05 smithi115 ceph-mon[78593]: pgmap v8760: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:08:05.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:08:05 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:08:05.796 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:08:05 smithi047 ceph-mon[77147]: pgmap v8760: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:08:05.796 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:08:05 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:08:06.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:08:06 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:08:06.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:08:06 smithi115 conmon[90393]: 2022-05-14T04:08:06.378+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:08:06.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:08:06 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:08:07.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:08:07 smithi047 ceph-mon[77147]: pgmap v8761: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:08:07.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:08:07 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:08:07.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:08:07 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 16820 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:08:07.860 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:08:07 smithi115 conmon[90393]: 2022-05-14T04:08:07.425+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:08:07.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:08:07 smithi115 ceph-mon[78593]: pgmap v8761: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:08:07.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:08:07 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:08:07.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:08:07 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 16820 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:08:08.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:08:08 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:08:08.861 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:08:08 smithi115 conmon[90393]: 2022-05-14T04:08:08.390+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:08:08.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:08:08 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:08:09.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:08:09 smithi047 ceph-mon[77147]: pgmap v8762: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:08:09.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:08:09 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:08:09.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:08:09 smithi115 conmon[90393]: 2022-05-14T04:08:09.392+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:08:09.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:08:09 smithi115 ceph-mon[78593]: pgmap v8762: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:08:09.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:08:09 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:08:10.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:08:10 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:08:10.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:08:10 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:08:10.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:08:10 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:08:10.860 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:08:10 smithi115 conmon[90393]: 2022-05-14T04:08:10.374+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:08:10.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:08:10 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:08:10.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:08:10 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:08:10.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:08:10 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:08:11.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:08:11 smithi115 conmon[90393]: 2022-05-14T04:08:11.343+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:08:11.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:08:11 smithi115 ceph-mon[78593]: pgmap v8763: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:08:11.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:08:11 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:08:11.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:08:11 smithi047 ceph-mon[77147]: pgmap v8763: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:08:11.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:08:11 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:08:12.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:08:12 smithi115 conmon[90393]: 2022-05-14T04:08:12.345+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:08:12.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:08:12 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:08:12.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:08:12 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 16825 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:08:12.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:08:12 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:08:12.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:08:12 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 16825 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:08:13.860 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:08:13 smithi115 conmon[90393]: 2022-05-14T04:08:13.369+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:08:14.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:08:14 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:08:14.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:08:14 smithi047 ceph-mon[77147]: pgmap v8764: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:08:14.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:08:14 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:08:14.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:08:14 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:08:14.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:08:14 smithi115 conmon[90393]: 2022-05-14T04:08:14.373+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:08:14.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:08:14 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:08:14.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:08:14 smithi115 ceph-mon[78593]: pgmap v8764: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:08:14.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:08:14 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:08:14.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:08:14 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:08:15.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:08:15 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:08:15.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:08:15 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T04:08:15.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:08:15 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T04:08:15.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:08:15 smithi115 conmon[90393]: 2022-05-14T04:08:15.339+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:08:15.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:08:15 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:08:15.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:08:15 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T04:08:15.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:08:15 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T04:08:16.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:08:16 smithi047 ceph-mon[77147]: pgmap v8765: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:08:16.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:08:16 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:08:16.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:08:16 smithi115 conmon[90393]: 2022-05-14T04:08:16.330+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:08:16.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:08:16 smithi115 ceph-mon[78593]: pgmap v8765: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:08:16.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:08:16 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:08:17.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:08:17 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:08:17.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:08:17 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 16830 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:08:17.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:08:17 smithi115 conmon[90393]: 2022-05-14T04:08:17.326+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:08:17.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:08:17 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:08:17.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:08:17 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 16830 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:08:18.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:08:18 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:08:18.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:08:18 smithi047 ceph-mon[77147]: pgmap v8766: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:08:18.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:08:18 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:08:18.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:08:18 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:08:18.611 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:08:18 smithi115 conmon[90393]: 2022-05-14T04:08:18.352+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:08:18.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:08:18 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:08:18.612 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:08:18 smithi115 ceph-mon[78593]: pgmap v8766: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:08:18.612 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:08:18 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:08:18.612 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:08:18 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:08:19.547 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:08:19 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:08:19.610 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:08:19 smithi115 conmon[90393]: 2022-05-14T04:08:19.313+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:08:19.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:08:19 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:08:20.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:08:20 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:08:20.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:08:20 smithi047 ceph-mon[77147]: pgmap v8767: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:08:20.613 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:08:20 smithi115 conmon[90393]: 2022-05-14T04:08:20.270+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:08:20.679 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:08:20 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:08:20.680 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:08:20 smithi115 ceph-mon[78593]: pgmap v8767: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:08:21.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:08:21 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:08:21.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:08:21 smithi115 conmon[90393]: 2022-05-14T04:08:21.310+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:08:21.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:08:21 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:08:22.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:08:22 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:08:22.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:08:22 smithi047 ceph-mon[77147]: pgmap v8768: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:08:22.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:08:22 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:08:22.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:08:22 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:08:22.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:08:22 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 16835 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:08:22.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:08:22 smithi115 conmon[90393]: 2022-05-14T04:08:22.340+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:08:22.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:08:22 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:08:22.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:08:22 smithi115 ceph-mon[78593]: pgmap v8768: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:08:22.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:08:22 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:08:22.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:08:22 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:08:22.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:08:22 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 16835 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:08:23.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:08:23 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:08:23.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:08:23 smithi115 conmon[90393]: 2022-05-14T04:08:23.304+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:08:23.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:08:23 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:08:24.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:08:24 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:08:24.545 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:08:24 smithi047 ceph-mon[77147]: pgmap v8769: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:08:24.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:08:24 smithi115 conmon[90393]: 2022-05-14T04:08:24.330+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:08:24.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:08:24 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:08:24.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:08:24 smithi115 ceph-mon[78593]: pgmap v8769: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:08:25.545 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:08:25 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:08:25.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:08:25 smithi115 conmon[90393]: 2022-05-14T04:08:25.287+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:08:25.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:08:25 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:08:26.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:08:26 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:08:26.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:08:26 smithi047 ceph-mon[77147]: pgmap v8770: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:08:26.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:08:26 smithi115 conmon[90393]: 2022-05-14T04:08:26.261+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:08:26.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:08:26 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:08:26.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:08:26 smithi115 ceph-mon[78593]: pgmap v8770: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:08:27.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:08:27 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:08:27.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:08:27 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 16840 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:08:27.610 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:08:27 smithi115 conmon[90393]: 2022-05-14T04:08:27.297+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:08:27.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:08:27 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:08:27.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:08:27 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 16840 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:08:28.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:08:28 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:08:28.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:08:28 smithi047 ceph-mon[77147]: pgmap v8771: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:08:28.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:08:28 smithi115 conmon[90393]: 2022-05-14T04:08:28.334+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:08:28.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:08:28 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:08:28.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:08:28 smithi115 ceph-mon[78593]: pgmap v8771: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:08:29.514 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:08:29 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:08:29.515 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:08:29 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:08:29.515 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:08:29 smithi115 conmon[90393]: 2022-05-14T04:08:29.362+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:08:30.196 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:08:29 smithi047 conmon[78521]: 2022-05-14T04:08:29.935+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:08:30.197 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:08:29 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T04:08:30.197 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:08:29 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:08:30.197 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:08:29 smithi047 conmon[78521]: yield 2022-05-14T04:08:30.197 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:08:29 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:08:30.198 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:08:29 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:08:30.198 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:08:29 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:08:30.198 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:08:29 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T04:08:30.198 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:08:29 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:08:30.199 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:08:29 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T04:08:30.199 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:08:29 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:08:30.199 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:08:29 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T04:08:30.200 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:08:29 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:08:30.200 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:08:29 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T04:08:30.200 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:08:29 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:08:30.200 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:08:29 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T04:08:30.200 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:08:29 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:08:30.201 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:08:29 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:08:30.201 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:08:29 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:08:30.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:08:30 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:08:30.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:08:30 smithi115 ceph-mon[78593]: pgmap v8772: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:08:30.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:08:30 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:08:30.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:08:30 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:08:30.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:08:30 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T04:08:30.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:08:30 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T04:08:30.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:08:30 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T04:08:30.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:08:30 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:08:30.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:08:30 smithi047 ceph-mon[77147]: pgmap v8772: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:08:30.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:08:30 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:08:30.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:08:30 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:08:30.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:08:30 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T04:08:30.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:08:30 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T04:08:30.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:08:30 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T04:08:30.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:08:30 smithi115 conmon[90393]: 2022-05-14T04:08:30.370+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:08:31.546 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:08:31 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:08:31.546 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:08:31 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:08:31.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T04:08:31.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:08:31.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T04:08:31.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:08:31.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:08:31.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:08:31.548 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T04:08:31.548 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:08:31.548 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T04:08:31.548 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:08:31.548 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T04:08:31.549 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:08:31.549 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T04:08:31.550 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:08:31.550 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T04:08:31.550 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:08:31.550 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:08:31.551 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:08:31.551 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:08:31 smithi047 ceph-mon[77147]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T04:08:31.610 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:08:31 smithi115 conmon[90393]: 2022-05-14T04:08:31.402+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:08:31.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:08:31 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:08:31.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:08:31 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:08:31.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T04:08:31.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:08:31.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T04:08:31.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:08:31.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:08:31.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:08:31.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T04:08:31.614 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:08:31.614 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T04:08:31.614 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:08:31.615 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T04:08:31.615 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:08:31.615 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T04:08:31.615 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:08:31.616 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T04:08:31.616 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:08:31.616 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:08:31.616 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:08:31.616 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:08:31 smithi115 ceph-mon[78593]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T04:08:32.546 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:08:32 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:08:32.546 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:08:32 smithi047 ceph-mon[77147]: pgmap v8773: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:08:32.546 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:08:32 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 16845 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:08:32.610 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:08:32 smithi115 conmon[90393]: 2022-05-14T04:08:32.418+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:08:32.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:08:32 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:08:32.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:08:32 smithi115 ceph-mon[78593]: pgmap v8773: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:08:32.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:08:32 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 16845 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:08:33.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:08:33 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:08:33.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:08:33 smithi047 conmon[78521]: 2022-05-14T04:08:33.007+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:08:33.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:08:33 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T04:08:33.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:08:33 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:08:33.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:08:33 smithi047 conmon[78521]: yield 2022-05-14T04:08:33.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:08:33 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:08:33.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:08:33 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:08:33.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:08:33 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:08:33.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:08:33 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T04:08:33.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:08:33 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:08:33.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:08:33 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T04:08:33.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:08:33 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:08:33.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:08:33 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T04:08:33.297 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:08:33 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:08:33.297 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:08:33 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T04:08:33.297 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:08:33 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:08:33.297 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:08:33 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T04:08:33.298 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:08:33 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:08:33.298 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:08:33 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:08:33.298 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:08:33 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:08:33.298 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:08:33 smithi047 conmon[78521]: 2022-05-14T04:08:33.008+0000 7f5344f82700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T04:08:33.298 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:08:33 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T04:08:33.299 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:08:33 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:08:33.299 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:08:33 smithi047 conmon[78521]: yield 2022-05-14T04:08:33.299 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:08:33 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:08:33.299 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:08:33 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:08:33.299 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:08:33 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:08:33.300 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:08:33 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T04:08:33.300 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:08:33 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:08:33.300 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:08:33 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T04:08:33.300 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:08:33 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:08:33.301 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:08:33 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T04:08:33.301 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:08:33 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:08:33.301 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:08:33 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T04:08:33.301 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:08:33 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:08:33.301 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:08:33 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T04:08:33.302 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:08:33 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:08:33.302 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:08:33 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:08:33.303 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:08:33 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:08:33.303 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:08:33 smithi047 conmon[78521]: 2022-05-14T04:08:33.303 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:08:33 smithi047 conmon[78521]: During handling of the above exception, another exception occurred: 2022-05-14T04:08:33.303 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:08:33 smithi047 conmon[78521]: 2022-05-14T04:08:33.303 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:08:33 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T04:08:33.304 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:08:33 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T04:08:33.304 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:08:33 smithi047 conmon[78521]: return f(*arg) 2022-05-14T04:08:33.304 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:08:33 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T04:08:33.304 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:08:33 smithi047 conmon[78521]: self._write_client_files(client_files, host) 2022-05-14T04:08:33.305 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:08:33 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T04:08:33.305 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:08:33 smithi047 conmon[78521]: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T04:08:33.305 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:08:33 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T04:08:33.305 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:08:33 smithi047 conmon[78521]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T04:08:33.305 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:08:33 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T04:08:33.306 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:08:33 smithi047 conmon[78521]: return self.event_loop.get_result(coro) 2022-05-14T04:08:33.306 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:08:33 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T04:08:33.306 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:08:33 smithi047 conmon[78521]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T04:08:33.306 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:08:33 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T04:08:33.306 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:08:33 smithi047 conmon[78521]: return self.__get_result() 2022-05-14T04:08:33.307 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:08:33 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T04:08:33.307 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:08:33 smithi047 conmon[78521]: raise self._exception 2022-05-14T04:08:33.307 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:08:33 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T04:08:33.307 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:08:33 smithi047 conmon[78521]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T04:08:33.308 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:08:33 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T04:08:33.308 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:08:33 smithi047 conmon[78521]: conn = await self._remote_connection(host, addr) 2022-05-14T04:08:33.308 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:08:33 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T04:08:33.308 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:08:33 smithi047 conmon[78521]: raise 2022-05-14T04:08:33.308 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:08:33 smithi047 conmon[78521]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T04:08:33.309 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:08:33 smithi047 conmon[78521]: self.gen.throw(type, value, traceback) 2022-05-14T04:08:33.309 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:08:33 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T04:08:33.309 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:08:33 smithi047 conmon[78521]: raise OrchestratorError(msg) 2022-05-14T04:08:33.309 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:08:33 smithi047 conmon[78521]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:08:33.610 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:08:33 smithi115 conmon[90393]: 2022-05-14T04:08:33.452+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:08:33.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:08:33 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:08:34.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:08:34 smithi047 ceph-mon[77147]: pgmap v8774: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:08:34.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:08:34 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:08:34.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:08:34 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:08:34.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T04:08:34.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:08:34.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T04:08:34.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:08:34.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:08:34.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:08:34.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T04:08:34.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:08:34.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T04:08:34.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:08:34.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T04:08:34.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:08:34.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T04:08:34.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:08:34.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T04:08:34.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:08:34.548 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:08:34.548 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:08:34.548 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:08:34 smithi047 ceph-mon[77147]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T04:08:34.548 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T04:08:34.549 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:08:34.549 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T04:08:34.549 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:08:34.549 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:08:34.549 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:08:34.550 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T04:08:34.550 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:08:34.551 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T04:08:34.551 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:08:34.551 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T04:08:34.551 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:08:34.552 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T04:08:34.553 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:08:34.554 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T04:08:34.554 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:08:34.554 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:08:34.554 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:08:34.555 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T04:08:34.555 INFO:journalctl@ceph.mon.b.smithi047.stdout: During handling of the above exception, another exception occurred: 2022-05-14T04:08:34.555 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T04:08:34.555 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T04:08:34.556 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T04:08:34.556 INFO:journalctl@ceph.mon.b.smithi047.stdout: return f(*arg) 2022-05-14T04:08:34.556 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T04:08:34.556 INFO:journalctl@ceph.mon.b.smithi047.stdout: self._write_client_files(client_files, host) 2022-05-14T04:08:34.556 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T04:08:34.557 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T04:08:34.557 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T04:08:34.560 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T04:08:34.560 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T04:08:34.560 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.event_loop.get_result(coro) 2022-05-14T04:08:34.560 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T04:08:34.561 INFO:journalctl@ceph.mon.b.smithi047.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T04:08:34.561 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T04:08:34.561 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.__get_result() 2022-05-14T04:08:34.561 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T04:08:34.562 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise self._exception 2022-05-14T04:08:34.562 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T04:08:34.562 INFO:journalctl@ceph.mon.b.smithi047.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T04:08:34.562 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T04:08:34.562 INFO:journalctl@ceph.mon.b.smithi047.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T04:08:34.563 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T04:08:34.563 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise 2022-05-14T04:08:34.563 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T04:08:34.564 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.gen.throw(type, value, traceback) 2022-05-14T04:08:34.564 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T04:08:34.564 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OrchestratorError(msg) 2022-05-14T04:08:34.564 INFO:journalctl@ceph.mon.b.smithi047.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:08:34.565 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:08:34 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:08:34.565 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:08:34 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:08:34.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:08:34 smithi115 conmon[90393]: 2022-05-14T04:08:34.442+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:08:34.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:08:34 smithi115 ceph-mon[78593]: pgmap v8774: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:08:34.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:08:34 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:08:34.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:08:34 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:08:34.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T04:08:34.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:08:34.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T04:08:34.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:08:34.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:08:34.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:08:34.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T04:08:34.614 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:08:34.614 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T04:08:34.614 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:08:34.614 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T04:08:34.615 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:08:34.615 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T04:08:34.615 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:08:34.615 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T04:08:34.615 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:08:34.616 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:08:34.616 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:08:34.616 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:08:34 smithi115 ceph-mon[78593]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T04:08:34.616 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T04:08:34.617 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:08:34.617 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T04:08:34.617 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:08:34.617 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:08:34.617 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:08:34.618 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T04:08:34.618 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:08:34.618 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T04:08:34.618 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:08:34.618 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T04:08:34.619 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:08:34.619 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T04:08:34.619 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:08:34.619 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T04:08:34.620 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:08:34.620 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:08:34.620 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:08:34.620 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T04:08:34.620 INFO:journalctl@ceph.mon.c.smithi115.stdout: During handling of the above exception, another exception occurred: 2022-05-14T04:08:34.621 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T04:08:34.621 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T04:08:34.621 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T04:08:34.622 INFO:journalctl@ceph.mon.c.smithi115.stdout: return f(*arg) 2022-05-14T04:08:34.622 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T04:08:34.623 INFO:journalctl@ceph.mon.c.smithi115.stdout: self._write_client_files(client_files, host) 2022-05-14T04:08:34.623 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T04:08:34.623 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T04:08:34.623 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T04:08:34.624 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T04:08:34.624 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T04:08:34.624 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.event_loop.get_result(coro) 2022-05-14T04:08:34.624 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T04:08:34.624 INFO:journalctl@ceph.mon.c.smithi115.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T04:08:34.625 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T04:08:34.625 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.__get_result() 2022-05-14T04:08:34.625 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T04:08:34.625 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise self._exception 2022-05-14T04:08:34.626 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T04:08:34.626 INFO:journalctl@ceph.mon.c.smithi115.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T04:08:34.626 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T04:08:34.626 INFO:journalctl@ceph.mon.c.smithi115.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T04:08:34.626 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T04:08:34.627 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise 2022-05-14T04:08:34.627 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T04:08:34.627 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.gen.throw(type, value, traceback) 2022-05-14T04:08:34.627 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T04:08:34.627 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OrchestratorError(msg) 2022-05-14T04:08:34.628 INFO:journalctl@ceph.mon.c.smithi115.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:08:34.628 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:08:34 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:08:34.628 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:08:34 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:08:35.483 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:08:35 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:08:35.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:08:35 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:08:35.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:08:35 smithi115 conmon[90393]: 2022-05-14T04:08:35.480+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:08:36.500 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:08:36 smithi115 ceph-mon[78593]: pgmap v8775: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:08:36.501 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:08:36 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:08:36.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:08:36 smithi047 ceph-mon[77147]: pgmap v8775: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:08:36.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:08:36 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:08:36.860 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:08:36 smithi115 conmon[90393]: 2022-05-14T04:08:36.496+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:08:37.483 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:08:37 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:08:37.484 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:08:37 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 16850 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:08:37.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:08:37 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:08:37.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:08:37 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 16850 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:08:37.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:08:37 smithi115 conmon[90393]: 2022-05-14T04:08:37.480+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:08:38.477 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:08:38 smithi115 ceph-mon[78593]: pgmap v8776: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:08:38.478 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:08:38 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:08:38.478 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:08:38 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:08:38.478 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:08:38 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:08:38.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:08:38 smithi047 ceph-mon[77147]: pgmap v8776: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:08:38.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:08:38 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:08:38.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:08:38 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:08:38.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:08:38 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:08:38.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:08:38 smithi115 conmon[90393]: 2022-05-14T04:08:38.474+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:08:39.499 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:08:39 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:08:39.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:08:39 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:08:39.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:08:39 smithi115 conmon[90393]: 2022-05-14T04:08:39.496+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:08:40.509 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:08:40 smithi115 ceph-mon[78593]: pgmap v8777: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:08:40.509 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:08:40 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:08:40.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:08:40 smithi047 ceph-mon[77147]: pgmap v8777: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:08:40.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:08:40 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:08:40.863 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:08:40 smithi115 conmon[90393]: 2022-05-14T04:08:40.505+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:08:41.516 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:08:41 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:08:41.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:08:41 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:08:41.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:08:41 smithi115 conmon[90393]: 2022-05-14T04:08:41.513+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:08:42.537 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:08:42 smithi115 ceph-mon[78593]: pgmap v8778: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:08:42.538 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:08:42 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:08:42.538 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:08:42 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:08:42.538 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:08:42 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:08:42.538 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:08:42 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 16855 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:08:42.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:08:42 smithi047 ceph-mon[77147]: pgmap v8778: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:08:42.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:08:42 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:08:42.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:08:42 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:08:42.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:08:42 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:08:42.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:08:42 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 16855 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:08:42.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:08:42 smithi115 conmon[90393]: 2022-05-14T04:08:42.533+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:08:43.532 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:08:43 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:08:43.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:08:43 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:08:43.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:08:43 smithi115 conmon[90393]: 2022-05-14T04:08:43.529+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:08:44.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:08:44 smithi047 ceph-mon[77147]: pgmap v8779: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:08:44.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:08:44 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:08:44.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:08:44 smithi115 conmon[90393]: 2022-05-14T04:08:44.483+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:08:44.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:08:44 smithi115 ceph-mon[78593]: pgmap v8779: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:08:44.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:08:44 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:08:45.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:08:45 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:08:45.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:08:45 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:08:45.612 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:08:45 smithi115 conmon[90393]: 2022-05-14T04:08:45.474+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:08:46.506 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:08:46 smithi115 ceph-mon[78593]: pgmap v8780: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:08:46.507 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:08:46 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:08:46.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:08:46 smithi047 ceph-mon[77147]: pgmap v8780: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:08:46.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:08:46 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:08:46.864 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:08:46 smithi115 conmon[90393]: 2022-05-14T04:08:46.503+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:08:47.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:08:47 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:08:47.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:08:47 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 16860 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:08:47.553 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:08:47 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:08:47.553 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:08:47 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 16860 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:08:47.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:08:47 smithi115 conmon[90393]: 2022-05-14T04:08:47.550+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:08:48.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:08:48 smithi047 ceph-mon[77147]: pgmap v8781: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:08:48.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:08:48 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:08:48.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:08:48 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:08:48.579 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:08:48 smithi115 ceph-mon[78593]: pgmap v8781: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:08:48.579 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:08:48 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:08:48.580 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:08:48 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:08:48.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:08:48 smithi115 conmon[90393]: 2022-05-14T04:08:48.576+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:08:49.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:08:49 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:08:49.587 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:08:49 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:08:49.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:08:49 smithi115 conmon[90393]: 2022-05-14T04:08:49.583+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:08:50.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:08:50 smithi047 ceph-mon[77147]: pgmap v8782: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:08:50.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:08:50 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:08:50.545 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:08:50 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:08:50.545 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:08:50 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:08:50.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:08:50 smithi115 ceph-mon[78593]: pgmap v8782: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:08:50.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:08:50 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:08:50.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:08:50 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:08:50.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:08:50 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:08:51.110 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:08:50 smithi115 conmon[90393]: 2022-05-14T04:08:50.633+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:08:51.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:08:51 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:08:51.600 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:08:51 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:08:51.861 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:08:51 smithi115 conmon[90393]: 2022-05-14T04:08:51.597+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:08:52.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:08:52 smithi047 ceph-mon[77147]: pgmap v8783: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:08:52.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:08:52 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:08:52.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:08:52 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 16865 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:08:52.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:08:52 smithi115 ceph-mon[78593]: pgmap v8783: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:08:52.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:08:52 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:08:52.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:08:52 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 16865 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:08:53.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:08:52 smithi115 conmon[90393]: 2022-05-14T04:08:52.621+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:08:53.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:08:53 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:08:53.592 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:08:53 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:08:53.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:08:53 smithi115 conmon[90393]: 2022-05-14T04:08:53.589+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:08:54.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:08:54 smithi047 ceph-mon[77147]: pgmap v8784: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:08:54.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:08:54 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:08:54.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:08:54 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:08:54.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:08:54 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:08:54.584 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:08:54 smithi115 ceph-mon[78593]: pgmap v8784: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:08:54.585 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:08:54 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:08:54.585 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:08:54 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:08:54.585 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:08:54 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:08:54.860 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:08:54 smithi115 conmon[90393]: 2022-05-14T04:08:54.581+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:08:55.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:08:55 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:08:55.563 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:08:55 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:08:55.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:08:55 smithi115 conmon[90393]: 2022-05-14T04:08:55.560+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:08:56.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:08:56 smithi115 ceph-mon[78593]: pgmap v8785: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:08:56.542 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:08:56 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:08:56.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:08:56 smithi047 ceph-mon[77147]: pgmap v8785: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:08:56.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:08:56 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:08:56.860 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:08:56 smithi115 conmon[90393]: 2022-05-14T04:08:56.538+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:08:57.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:08:57 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:08:57.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:08:57 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 16870 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:08:57.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:08:57 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:08:57.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:08:57 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 16870 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:08:57.610 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:08:57 smithi115 conmon[90393]: 2022-05-14T04:08:57.497+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:08:58.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:08:58 smithi047 ceph-mon[77147]: pgmap v8786: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:08:58.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:08:58 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:08:58.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:08:58 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:08:58.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:08:58 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:08:58.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:08:58 smithi115 conmon[90393]: 2022-05-14T04:08:58.488+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:08:58.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:08:58 smithi115 ceph-mon[78593]: pgmap v8786: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:08:58.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:08:58 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:08:58.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:08:58 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:08:58.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:08:58 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:08:59.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:08:59 smithi115 conmon[90393]: 2022-05-14T04:08:59.456+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:08:59.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:08:59 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:08:59.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:08:59 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:09:00.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:09:00 smithi115 conmon[90393]: 2022-05-14T04:09:00.429+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:09:00.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:09:00 smithi115 ceph-mon[78593]: pgmap v8787: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:09:00.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:09:00 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:09:00.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:09:00 smithi047 ceph-mon[77147]: pgmap v8787: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:09:00.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:09:00 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:09:01.610 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:09:01 smithi115 conmon[90393]: 2022-05-14T04:09:01.379+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:09:01.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:09:01 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:09:01.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:09:01 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:09:02.610 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:09:02 smithi115 conmon[90393]: 2022-05-14T04:09:02.374+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:09:02.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:09:02 smithi115 ceph-mon[78593]: pgmap v8788: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:09:02.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:09:02 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:09:02.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:09:02 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:09:02.612 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:09:02 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:09:02.612 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:09:02 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 16875 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:09:02.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:09:02 smithi047 ceph-mon[77147]: pgmap v8788: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:09:02.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:09:02 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:09:02.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:09:02 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:09:02.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:09:02 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:09:02.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:09:02 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 16875 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:09:03.610 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:09:03 smithi115 conmon[90393]: 2022-05-14T04:09:03.376+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:09:03.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:09:03 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:09:03.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:09:03 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:09:04.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:09:04 smithi115 conmon[90393]: 2022-05-14T04:09:04.400+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:09:04.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:09:04 smithi115 ceph-mon[78593]: pgmap v8789: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:09:04.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:09:04 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:09:04.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:09:04 smithi047 ceph-mon[77147]: pgmap v8789: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:09:04.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:09:04 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:09:05.610 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:09:05 smithi115 conmon[90393]: 2022-05-14T04:09:05.363+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:09:05.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:09:05 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:09:05.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:09:05 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:09:06.611 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:09:06 smithi115 conmon[90393]: 2022-05-14T04:09:06.374+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:09:06.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:09:06 smithi115 ceph-mon[78593]: pgmap v8790: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:09:06.612 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:09:06 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:09:06.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:09:06 smithi047 ceph-mon[77147]: pgmap v8790: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:09:06.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:09:06 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:09:07.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:09:07 smithi115 conmon[90393]: 2022-05-14T04:09:07.348+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:09:07.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:09:07 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:09:07.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:09:07 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 16880 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:09:07.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:09:07 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:09:07.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:09:07 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 16880 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:09:08.610 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:09:08 smithi115 conmon[90393]: 2022-05-14T04:09:08.302+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:09:08.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:09:08 smithi115 ceph-mon[78593]: pgmap v8791: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:09:08.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:09:08 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:09:08.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:09:08 smithi047 ceph-mon[77147]: pgmap v8791: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:09:08.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:09:08 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:09:09.610 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:09:09 smithi115 conmon[90393]: 2022-05-14T04:09:09.313+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:09:09.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:09:09 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:09:09.798 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:09:09 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:09:10.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:09:10 smithi115 conmon[90393]: 2022-05-14T04:09:10.293+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:09:10.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:09:10 smithi115 ceph-mon[78593]: pgmap v8792: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:09:10.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:09:10 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:09:10.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:09:10 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:09:10.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:09:10 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:09:10.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:09:10 smithi047 ceph-mon[77147]: pgmap v8792: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:09:10.797 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:09:10 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:09:10.797 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:09:10 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:09:10.798 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:09:10 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:09:11.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:09:11 smithi115 conmon[90393]: 2022-05-14T04:09:11.337+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:09:11.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:09:11 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:09:11.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:09:11 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:09:12.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:09:12 smithi115 conmon[90393]: 2022-05-14T04:09:12.312+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:09:12.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:09:12 smithi115 ceph-mon[78593]: pgmap v8793: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:09:12.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:09:12 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:09:12.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:09:12 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 16885 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:09:12.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:09:12 smithi047 ceph-mon[77147]: pgmap v8793: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:09:12.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:09:12 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:09:12.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:09:12 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 16885 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:09:13.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:09:13 smithi115 conmon[90393]: 2022-05-14T04:09:13.270+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:09:13.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:09:13 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:09:13.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:09:13 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:09:14.611 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:09:14 smithi115 conmon[90393]: 2022-05-14T04:09:14.230+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:09:14.612 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:09:14 smithi115 ceph-mon[78593]: pgmap v8794: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:09:14.613 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:09:14 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:09:14.613 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:09:14 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:09:14.614 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:09:14 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:09:14.614 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:09:14 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:09:14.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:09:14 smithi047 ceph-mon[77147]: pgmap v8794: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:09:14.795 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:09:14 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:09:14.795 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:09:14 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:09:14.796 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:09:14 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:09:14.796 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:09:14 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:09:15.610 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:09:15 smithi115 conmon[90393]: 2022-05-14T04:09:15.279+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:09:15.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:09:15 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T04:09:15.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:09:15 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T04:09:15.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:09:15 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T04:09:15.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:09:15 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T04:09:16.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:09:16 smithi115 conmon[90393]: 2022-05-14T04:09:16.231+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:09:16.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:09:16 smithi115 ceph-mon[78593]: pgmap v8795: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:09:16.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:09:16 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:09:16.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:09:16 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:09:16.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:09:16 smithi047 ceph-mon[77147]: pgmap v8795: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:09:16.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:09:16 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:09:16.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:09:16 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:09:17.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:09:17 smithi115 conmon[90393]: 2022-05-14T04:09:17.280+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:09:17.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:09:17 smithi115 ceph-mon[78593]: pgmap v8796: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:09:17.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:09:17 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 16890 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:09:17.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:09:17 smithi047 ceph-mon[77147]: pgmap v8796: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:09:17.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:09:17 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 16890 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:09:18.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:09:18 smithi115 conmon[90393]: 2022-05-14T04:09:18.252+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:09:18.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:09:18 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:09:18.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:09:18 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:09:18.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:09:18 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:09:18.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:09:18 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:09:18.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:09:18 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:09:18.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:09:18 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:09:19.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:09:19 smithi115 conmon[90393]: 2022-05-14T04:09:19.303+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:09:19.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:09:19 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:09:19.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:09:19 smithi115 ceph-mon[78593]: pgmap v8797: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:09:19.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:09:19 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:09:19.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:09:19 smithi047 ceph-mon[77147]: pgmap v8797: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:09:20.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:09:20 smithi115 conmon[90393]: 2022-05-14T04:09:20.304+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:09:20.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:09:20 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:09:20.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:09:20 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:09:21.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:09:21 smithi115 conmon[90393]: 2022-05-14T04:09:21.279+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:09:21.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:09:21 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:09:21.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:09:21 smithi115 ceph-mon[78593]: pgmap v8798: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:09:21.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:09:21 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:09:21.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:09:21 smithi047 ceph-mon[77147]: pgmap v8798: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:09:22.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:09:22 smithi115 conmon[90393]: 2022-05-14T04:09:22.232+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:09:22.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:09:22 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:09:22.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:09:22 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:09:22.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:09:22 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:09:22.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:09:22 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 16895 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:09:22.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:09:22 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:09:22.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:09:22 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:09:22.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:09:22 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:09:22.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:09:22 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:09:22.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:09:22 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 16895 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:09:22.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:09:22 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:09:23.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:09:23 smithi115 conmon[90393]: 2022-05-14T04:09:23.204+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:09:23.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:09:23 smithi115 ceph-mon[78593]: pgmap v8799: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:09:23.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:09:23 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:09:23.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:09:23 smithi047 ceph-mon[77147]: pgmap v8799: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:09:23.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:09:23 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:09:24.610 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:09:24 smithi115 conmon[90393]: 2022-05-14T04:09:24.226+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:09:24.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:09:24 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:09:24.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:09:24 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:09:25.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:09:25 smithi115 conmon[90393]: 2022-05-14T04:09:25.228+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:09:25.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:09:25 smithi115 ceph-mon[78593]: pgmap v8800: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:09:25.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:09:25 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:09:25.797 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:09:25 smithi047 ceph-mon[77147]: pgmap v8800: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:09:25.798 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:09:25 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:09:26.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:09:26 smithi115 conmon[90393]: 2022-05-14T04:09:26.244+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:09:26.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:09:26 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:09:26.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:09:26 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:09:27.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:09:27 smithi115 conmon[90393]: 2022-05-14T04:09:27.288+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:09:27.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:09:27 smithi115 ceph-mon[78593]: pgmap v8801: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:09:27.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:09:27 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 16900 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:09:27.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:09:27 smithi047 ceph-mon[77147]: pgmap v8801: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:09:27.795 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:09:27 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 16900 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:09:28.611 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:09:28 smithi115 conmon[90393]: 2022-05-14T04:09:28.289+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:09:28.612 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:09:28 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:09:28.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:09:28 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:09:29.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:09:29 smithi115 conmon[90393]: 2022-05-14T04:09:29.296+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:09:29.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:09:29 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:09:29.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:09:29 smithi115 ceph-mon[78593]: pgmap v8802: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:09:29.795 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:09:29 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:09:29.796 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:09:29 smithi047 ceph-mon[77147]: pgmap v8802: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:09:30.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:09:30 smithi115 conmon[90393]: 2022-05-14T04:09:30.289+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:09:30.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:09:30 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:09:30.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:09:30 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:09:30.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:09:30 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:09:30.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:09:30 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:09:30.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:09:30 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:09:30.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:09:30 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:09:31.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:09:31 smithi115 conmon[90393]: 2022-05-14T04:09:31.318+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:09:31.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:09:31 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:09:31.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:09:31 smithi115 ceph-mon[78593]: pgmap v8803: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:09:31.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:09:31 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:09:31.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:09:31 smithi047 ceph-mon[77147]: pgmap v8803: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:09:32.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:09:32 smithi115 conmon[90393]: 2022-05-14T04:09:32.286+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:09:32.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:09:32 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:09:32.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:09:32 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 16905 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:09:32.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:09:32 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:09:32.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:09:32 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 16905 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:09:33.610 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:09:33 smithi115 conmon[90393]: 2022-05-14T04:09:33.329+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:09:33.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:09:33 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:09:33.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:09:33 smithi115 ceph-mon[78593]: pgmap v8804: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:09:33.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:09:33 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T04:09:33.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:09:33 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T04:09:33.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:09:33 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T04:09:33.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:09:33 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:09:33.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:09:33 smithi047 ceph-mon[77147]: pgmap v8804: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:09:33.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:09:33 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T04:09:33.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:09:33 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T04:09:33.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:09:33 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T04:09:34.797 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:09:34 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:09:34.798 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:09:34 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:09:34.799 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:09:34 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:09:34.860 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:09:34 smithi115 conmon[90393]: 2022-05-14T04:09:34.370+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:09:34.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:09:34 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:09:34.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:09:34 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:09:34.862 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:09:34 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:09:35.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:09:35 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:09:35.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:09:35 smithi047 ceph-mon[77147]: pgmap v8805: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:09:35.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:09:35 smithi115 conmon[90393]: 2022-05-14T04:09:35.400+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:09:35.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:09:35 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:09:35.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:09:35 smithi115 ceph-mon[78593]: pgmap v8805: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:09:36.470 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:09:36 smithi047 conmon[78521]: 2022-05-14T04:09:36.112+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:09:36.470 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:09:36 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T04:09:36.471 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:09:36 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:09:36.473 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:09:36 smithi047 conmon[78521]: yield 2022-05-14T04:09:36.474 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:09:36 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:09:36.474 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:09:36 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:09:36.474 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:09:36 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:09:36.475 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:09:36 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T04:09:36.475 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:09:36 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:09:36.475 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:09:36 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T04:09:36.476 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:09:36 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:09:36.476 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:09:36 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T04:09:36.476 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:09:36 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:09:36.476 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:09:36 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T04:09:36.476 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:09:36 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:09:36.477 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:09:36 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T04:09:36.477 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:09:36 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:09:36.477 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:09:36 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:09:36.477 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:09:36 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:09:36.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:09:36 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:09:36.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:09:36 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:09:36.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T04:09:36.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:09:36.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T04:09:36.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:09:36.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:09:36.797 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:09:36.797 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T04:09:36.798 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:09:36.799 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T04:09:36.800 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:09:36.800 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T04:09:36.800 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:09:36.800 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T04:09:36.801 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:09:36.801 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T04:09:36.801 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:09:36.801 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:09:36.802 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:09:36.802 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:09:36 smithi047 ceph-mon[77147]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T04:09:36.862 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:09:36 smithi115 conmon[90393]: 2022-05-14T04:09:36.440+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:09:36.862 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:09:36 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:09:36.863 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:09:36 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:09:36.863 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T04:09:36.863 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:09:36.864 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T04:09:36.864 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:09:36.864 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:09:36.864 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:09:36.865 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T04:09:36.865 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:09:36.865 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T04:09:36.865 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:09:36.866 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T04:09:36.866 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:09:36.866 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T04:09:36.866 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:09:36.867 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T04:09:36.867 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:09:36.867 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:09:36.867 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:09:36.868 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:09:36 smithi115 ceph-mon[78593]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T04:09:37.797 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:09:37 smithi047 ceph-mon[77147]: pgmap v8806: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:09:37.797 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:09:37 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:09:37.798 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:09:37 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 16910 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:09:37.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:09:37 smithi115 conmon[90393]: 2022-05-14T04:09:37.406+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:09:37.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:09:37 smithi115 ceph-mon[78593]: pgmap v8806: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:09:37.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:09:37 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:09:37.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:09:37 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 16910 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:09:38.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:09:38 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:09:38.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:09:38 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:09:38.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:09:38 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:09:38.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:09:38 smithi115 conmon[90393]: 2022-05-14T04:09:38.447+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:09:38.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:09:38 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:09:38.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:09:38 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:09:38.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:09:38 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:09:39.483 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:09:39 smithi047 conmon[78521]: 2022-05-14T04:09:39.184+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:09:39.484 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:09:39 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T04:09:39.484 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:09:39 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:09:39.484 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:09:39 smithi047 conmon[78521]: yield 2022-05-14T04:09:39.484 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:09:39 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:09:39.485 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:09:39 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:09:39.485 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:09:39 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:09:39.485 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:09:39 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T04:09:39.485 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:09:39 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:09:39.485 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:09:39 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T04:09:39.486 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:09:39 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:09:39.486 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:09:39 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T04:09:39.486 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:09:39 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:09:39.486 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:09:39 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T04:09:39.487 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:09:39 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:09:39.487 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:09:39 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T04:09:39.487 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:09:39 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:09:39.487 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:09:39 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:09:39.487 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:09:39 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:09:39.488 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:09:39 smithi047 conmon[78521]: 2022-05-14T04:09:39.185+0000 7f533ef76700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T04:09:39.488 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:09:39 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T04:09:39.488 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:09:39 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:09:39.488 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:09:39 smithi047 conmon[78521]: yield 2022-05-14T04:09:39.488 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:09:39 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:09:39.489 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:09:39 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:09:39.489 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:09:39 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:09:39.489 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:09:39 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T04:09:39.489 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:09:39 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:09:39.490 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:09:39 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T04:09:39.490 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:09:39 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:09:39.490 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:09:39 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T04:09:39.490 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:09:39 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:09:39.490 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:09:39 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T04:09:39.491 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:09:39 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:09:39.491 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:09:39 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T04:09:39.491 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:09:39 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:09:39.491 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:09:39 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:09:39.491 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:09:39 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:09:39.492 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:09:39 smithi047 conmon[78521]: 2022-05-14T04:09:39.492 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:09:39 smithi047 conmon[78521]: During handling of the above exception, another exception occurred: 2022-05-14T04:09:39.492 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:09:39 smithi047 conmon[78521]: 2022-05-14T04:09:39.492 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:09:39 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T04:09:39.493 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:09:39 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T04:09:39.493 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:09:39 smithi047 conmon[78521]: return f(*arg) 2022-05-14T04:09:39.493 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:09:39 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T04:09:39.494 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:09:39 smithi047 conmon[78521]: self._write_client_files(client_files, host) 2022-05-14T04:09:39.494 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:09:39 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T04:09:39.494 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:09:39 smithi047 conmon[78521]: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T04:09:39.494 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:09:39 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T04:09:39.495 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:09:39 smithi047 conmon[78521]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T04:09:39.495 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:09:39 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T04:09:39.495 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:09:39 smithi047 conmon[78521]: return self.event_loop.get_result(coro) 2022-05-14T04:09:39.495 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:09:39 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T04:09:39.496 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:09:39 smithi047 conmon[78521]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T04:09:39.496 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:09:39 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T04:09:39.496 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:09:39 smithi047 conmon[78521]: return self.__get_result() 2022-05-14T04:09:39.496 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:09:39 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T04:09:39.497 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:09:39 smithi047 conmon[78521]: raise self._exception 2022-05-14T04:09:39.497 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:09:39 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T04:09:39.497 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:09:39 smithi047 conmon[78521]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T04:09:39.497 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:09:39 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T04:09:39.498 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:09:39 smithi047 conmon[78521]: conn = await self._remote_connection(host, addr) 2022-05-14T04:09:39.498 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:09:39 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T04:09:39.498 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:09:39 smithi047 conmon[78521]: raise 2022-05-14T04:09:39.498 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:09:39 smithi047 conmon[78521]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T04:09:39.499 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:09:39 smithi047 conmon[78521]: self.gen.throw(type, value, traceback) 2022-05-14T04:09:39.499 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:09:39 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T04:09:39.499 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:09:39 smithi047 conmon[78521]: raise OrchestratorError(msg) 2022-05-14T04:09:39.499 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:09:39 smithi047 conmon[78521]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:09:39.800 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:09:39 smithi047 ceph-mon[77147]: pgmap v8807: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:09:39.800 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:09:39 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:09:39.800 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:09:39 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:09:39.801 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T04:09:39.801 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:09:39.801 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T04:09:39.802 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:09:39.802 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:09:39.802 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:09:39.802 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T04:09:39.802 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:09:39.803 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T04:09:39.803 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:09:39.803 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T04:09:39.803 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:09:39.803 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T04:09:39.804 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:09:39.804 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T04:09:39.804 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:09:39.804 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:09:39.805 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:09:39.805 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:09:39 smithi047 ceph-mon[77147]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T04:09:39.805 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T04:09:39.806 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:09:39.806 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T04:09:39.806 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:09:39.806 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:09:39.806 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:09:39.807 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T04:09:39.807 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:09:39.807 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T04:09:39.807 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:09:39.807 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T04:09:39.808 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:09:39.808 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T04:09:39.808 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:09:39.808 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T04:09:39.809 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:09:39.809 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:09:39.809 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:09:39.809 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T04:09:39.809 INFO:journalctl@ceph.mon.b.smithi047.stdout: During handling of the above exception, another exception occurred: 2022-05-14T04:09:39.810 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T04:09:39.810 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T04:09:39.810 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T04:09:39.810 INFO:journalctl@ceph.mon.b.smithi047.stdout: return f(*arg) 2022-05-14T04:09:39.810 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T04:09:39.811 INFO:journalctl@ceph.mon.b.smithi047.stdout: self._write_client_files(client_files, host) 2022-05-14T04:09:39.811 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T04:09:39.811 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T04:09:39.811 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T04:09:39.812 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T04:09:39.812 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T04:09:39.812 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.event_loop.get_result(coro) 2022-05-14T04:09:39.812 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T04:09:39.812 INFO:journalctl@ceph.mon.b.smithi047.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T04:09:39.813 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T04:09:39.813 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.__get_result() 2022-05-14T04:09:39.813 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T04:09:39.813 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise self._exception 2022-05-14T04:09:39.814 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T04:09:39.814 INFO:journalctl@ceph.mon.b.smithi047.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T04:09:39.814 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T04:09:39.814 INFO:journalctl@ceph.mon.b.smithi047.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T04:09:39.814 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T04:09:39.815 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise 2022-05-14T04:09:39.815 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T04:09:39.815 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.gen.throw(type, value, traceback) 2022-05-14T04:09:39.815 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T04:09:39.815 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OrchestratorError(msg) 2022-05-14T04:09:39.816 INFO:journalctl@ceph.mon.b.smithi047.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:09:39.862 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:09:39 smithi115 conmon[90393]: 2022-05-14T04:09:39.411+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:09:39.863 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:09:39 smithi115 ceph-mon[78593]: pgmap v8807: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:09:39.863 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:09:39 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:09:39.864 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:09:39 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:09:39.864 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T04:09:39.865 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:09:39.865 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T04:09:39.865 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:09:39.866 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:09:39.866 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:09:39.867 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T04:09:39.867 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:09:39.868 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T04:09:39.868 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:09:39.869 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T04:09:39.869 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:09:39.870 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T04:09:39.872 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:09:39.872 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T04:09:39.873 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:09:39.873 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:09:39.874 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:09:39.874 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:09:39 smithi115 ceph-mon[78593]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T04:09:39.875 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T04:09:39.875 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:09:39.876 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T04:09:39.877 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:09:39.877 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:09:39.877 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:09:39.878 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T04:09:39.878 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:09:39.879 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T04:09:39.879 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:09:39.880 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T04:09:39.880 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:09:39.881 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T04:09:39.882 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:09:39.882 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T04:09:39.882 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:09:39.882 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:09:39.883 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:09:39.883 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T04:09:39.883 INFO:journalctl@ceph.mon.c.smithi115.stdout: During handling of the above exception, another exception occurred: 2022-05-14T04:09:39.883 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T04:09:39.884 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T04:09:39.884 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T04:09:39.884 INFO:journalctl@ceph.mon.c.smithi115.stdout: return f(*arg) 2022-05-14T04:09:39.884 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T04:09:39.884 INFO:journalctl@ceph.mon.c.smithi115.stdout: self._write_client_files(client_files, host) 2022-05-14T04:09:39.885 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T04:09:39.885 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T04:09:39.885 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T04:09:39.885 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T04:09:39.886 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T04:09:39.886 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.event_loop.get_result(coro) 2022-05-14T04:09:39.887 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T04:09:39.887 INFO:journalctl@ceph.mon.c.smithi115.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T04:09:39.887 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T04:09:39.887 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.__get_result() 2022-05-14T04:09:39.888 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T04:09:39.888 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise self._exception 2022-05-14T04:09:39.888 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T04:09:39.888 INFO:journalctl@ceph.mon.c.smithi115.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T04:09:39.888 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T04:09:39.889 INFO:journalctl@ceph.mon.c.smithi115.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T04:09:39.889 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T04:09:39.889 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise 2022-05-14T04:09:39.889 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T04:09:39.890 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.gen.throw(type, value, traceback) 2022-05-14T04:09:39.890 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T04:09:39.890 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OrchestratorError(msg) 2022-05-14T04:09:39.890 INFO:journalctl@ceph.mon.c.smithi115.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:09:40.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:09:40 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:09:40.860 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:09:40 smithi115 conmon[90393]: 2022-05-14T04:09:40.399+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:09:40.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:09:40 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:09:41.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:09:41 smithi047 ceph-mon[77147]: pgmap v8808: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:09:41.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:09:41 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:09:41.860 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:09:41 smithi115 conmon[90393]: 2022-05-14T04:09:41.440+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:09:41.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:09:41 smithi115 ceph-mon[78593]: pgmap v8808: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:09:41.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:09:41 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:09:42.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:09:42 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:09:42.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:09:42 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:09:42.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:09:42 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:09:42.795 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:09:42 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 16915 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:09:42.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:09:42 smithi115 conmon[90393]: 2022-05-14T04:09:42.416+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:09:42.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:09:42 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:09:42.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:09:42 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:09:42.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:09:42 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:09:42.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:09:42 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 16915 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:09:43.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:09:43 smithi115 conmon[90393]: 2022-05-14T04:09:43.455+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:09:44.486 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:09:44 smithi115 ceph-mon[78593]: pgmap v8809: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:09:44.486 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:09:44 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:09:44.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:09:44 smithi047 ceph-mon[77147]: pgmap v8809: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:09:44.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:09:44 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:09:44.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:09:44 smithi115 conmon[90393]: 2022-05-14T04:09:44.482+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:09:45.490 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:09:45 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:09:45.549 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:09:45 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:09:45.862 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:09:45 smithi115 conmon[90393]: 2022-05-14T04:09:45.486+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:09:46.533 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:09:46 smithi115 ceph-mon[78593]: pgmap v8810: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:09:46.533 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:09:46 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:09:46.534 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:09:46 smithi115 conmon[90393]: 2022-05-14T04:09:46.530+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:09:46.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:09:46 smithi047 ceph-mon[77147]: pgmap v8810: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:09:46.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:09:46 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:09:47.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:09:47 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:09:47.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:09:47 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 16920 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:09:47.579 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:09:47 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:09:47.580 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:09:47 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 16920 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:09:47.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:09:47 smithi115 conmon[90393]: 2022-05-14T04:09:47.575+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:09:48.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:09:48 smithi047 ceph-mon[77147]: pgmap v8811: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:09:48.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:09:48 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:09:48.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:09:48 smithi115 ceph-mon[78593]: pgmap v8811: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:09:48.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:09:48 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:09:49.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:09:48 smithi115 conmon[90393]: 2022-05-14T04:09:48.622+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:09:49.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:09:49 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:09:49.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:09:49 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:09:50.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:09:49 smithi115 conmon[90393]: 2022-05-14T04:09:49.643+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:09:50.550 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:09:50 smithi047 ceph-mon[77147]: pgmap v8812: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:09:50.551 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:09:50 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:09:50.551 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:09:50 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:09:50.551 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:09:50 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:09:50.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:09:50 smithi115 ceph-mon[78593]: pgmap v8812: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:09:50.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:09:50 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:09:50.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:09:50 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:09:50.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:09:50 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:09:51.110 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:09:50 smithi115 conmon[90393]: 2022-05-14T04:09:50.692+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:09:51.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:09:51 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:09:51.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:09:51 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:09:52.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:09:51 smithi115 conmon[90393]: 2022-05-14T04:09:51.695+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:09:52.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:09:52 smithi047 ceph-mon[77147]: pgmap v8813: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:09:52.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:09:52 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:09:52.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:09:52 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 16925 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:09:52.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:09:52 smithi115 ceph-mon[78593]: pgmap v8813: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:09:52.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:09:52 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:09:52.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:09:52 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 16925 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:09:53.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:09:52 smithi115 conmon[90393]: 2022-05-14T04:09:52.736+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:09:53.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:09:53 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:09:53.615 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:09:53 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:09:54.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:09:53 smithi115 conmon[90393]: 2022-05-14T04:09:53.737+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:09:54.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:09:54 smithi047 ceph-mon[77147]: pgmap v8814: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:09:54.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:09:54 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:09:54.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:09:54 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:09:54.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:09:54 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:09:54.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:09:54 smithi115 ceph-mon[78593]: pgmap v8814: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:09:54.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:09:54 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:09:54.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:09:54 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:09:54.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:09:54 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:09:55.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:09:54 smithi115 conmon[90393]: 2022-05-14T04:09:54.713+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:09:55.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:09:55 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:09:55.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:09:55 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:09:56.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:09:55 smithi115 conmon[90393]: 2022-05-14T04:09:55.693+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:09:56.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:09:56 smithi047 ceph-mon[77147]: pgmap v8815: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:09:56.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:09:56 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:09:56.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:09:56 smithi115 ceph-mon[78593]: pgmap v8815: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:09:56.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:09:56 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:09:57.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:09:56 smithi115 conmon[90393]: 2022-05-14T04:09:56.732+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:09:57.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:09:57 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:09:57.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:09:57 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 16930 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:09:57.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:09:57 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:09:57.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:09:57 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 16930 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:09:58.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:09:57 smithi115 conmon[90393]: 2022-05-14T04:09:57.727+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:09:58.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:09:58 smithi047 ceph-mon[77147]: pgmap v8816: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:09:58.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:09:58 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:09:58.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:09:58 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:09:58.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:09:58 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:09:58.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:09:58 smithi115 ceph-mon[78593]: pgmap v8816: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:09:58.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:09:58 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:09:58.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:09:58 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:09:58.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:09:58 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:09:59.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:09:58 smithi115 conmon[90393]: 2022-05-14T04:09:58.731+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:09:59.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:09:59 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:09:59.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:09:59 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:10:00.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:09:59 smithi115 conmon[90393]: 2022-05-14T04:09:59.732+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:10:00.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:10:00 smithi047 conmon[77123]: 2022-05-14T04:09:59.999+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : Health detail: HEALTH_ERR 1 filesystem is degraded; 1 mds daemon damaged; 1 MDSs are read only; 1 MDSs report slow metadata IOs; 1/3 mons down, quorum b,c; Reduced data availability: 1 pg inactive; 9 slow ops, oldest one blocked for 16930 sec, osd.5 has slow ops 2022-05-14T04:10:00.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:10:00 smithi047 conmon[77123]: 2022-05-14T04:09:59.999+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : [WRN] FS_DEGRADED: 1 filesystem is degraded 2022-05-14T04:10:00.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:10:00 smithi047 conmon[77123]: 2022-05-14T04:10:00.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:10:00 smithi047 conmon[77123]: 2022-05-14T04:09:59.999+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : fs cephfs is degraded 2022-05-14T04:10:00.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:10:00 smithi047 conmon[77123]: 2022-05-14T04:09:59.999+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : [ERR] MDS_DAMAGE: 1 mds daemon damaged 2022-05-14T04:10:00.295 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:10:00 smithi047 conmon[77123]: 2022-05-14T04:09:59.999+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : fs cephfs mds.2 is damaged 2022-05-14T04:10:00.295 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:10:00 smithi047 conmon[77123]: 2022-05-14T04:09:59.999+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : [WRN] MDS_READ_ONLY: 1 MDSs are read only 2022-05-14T04:10:00.295 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:10:00 smithi047 conmon[77123]: 2022-05-14T04:09:59.999+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : mds.l(mds.0): MDS in read-only mode 2022-05-14T04:10:00.295 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:10:00 smithi047 conmon[77123]: 2022-05-14T04:09:59.999+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : [WRN] MDS_SLOW_METADATA_IO: 1 MDSs report slow metadata IOs 2022-05-14T04:10:00.296 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:10:00 smithi047 conmon[77123]: 2022-05-14T04:09:59.999+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : mds.l(mds.0): 4 slow metadata IOs are blocked > 180 secs, oldest blocked for 17233 secs 2022-05-14T04:10:00.296 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:10:00 smithi047 conmon[77123]: 2022-05-14T04:09:59.999+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : [WRN] MON_DOWN: 1/3 mons down, quorum b,c 2022-05-14T04:10:00.296 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:10:00 smithi047 conmon[77123]: 2022-05-14T04:09:59.999+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : mon.a (rank 0) addr [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] is down (out of quorum) 2022-05-14T04:10:00.297 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:10:00 smithi047 conmon[77123]: 2022-05-14T04:09:59.999+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : [WRN] PG_AVAILABILITY: Reduced data availability: 1 pg inactive 2022-05-14T04:10:00.297 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:10:00 smithi047 conmon[77123]: 2022-05-14T04:09:59.999+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : pg 2.1 is stuck inactive for 4h, current state unknown, last acting [] 2022-05-14T04:10:00.297 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:10:00 smithi047 conmon[77123]: 2022-05-14T04:09:59.999+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : [WRN] SLOW_OPS: 9 slow ops, oldest one blocked for 16930 sec, osd.5 has slow ops 2022-05-14T04:10:00.297 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:10:00 smithi047 ceph-mon[77147]: pgmap v8817: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:10:00.298 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:10:00 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:10:00.298 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:10:00 smithi047 ceph-mon[77147]: Health detail: HEALTH_ERR 1 filesystem is degraded; 1 mds daemon damaged; 1 MDSs are read only; 1 MDSs report slow metadata IOs; 1/3 mons down, quorum b,c; Reduced data availability: 1 pg inactive; 9 slow ops, oldest one blocked for 16930 sec, osd.5 has slow ops 2022-05-14T04:10:00.298 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:10:00 smithi047 ceph-mon[77147]: [WRN] FS_DEGRADED: 1 filesystem is degraded 2022-05-14T04:10:00.298 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:10:00 smithi047 ceph-mon[77147]: fs cephfs is degraded 2022-05-14T04:10:00.299 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:10:00 smithi047 ceph-mon[77147]: [ERR] MDS_DAMAGE: 1 mds daemon damaged 2022-05-14T04:10:00.299 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:10:00 smithi047 ceph-mon[77147]: fs cephfs mds.2 is damaged 2022-05-14T04:10:00.299 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:10:00 smithi047 ceph-mon[77147]: [WRN] MDS_READ_ONLY: 1 MDSs are read only 2022-05-14T04:10:00.299 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:10:00 smithi047 ceph-mon[77147]: mds.l(mds.0): MDS in read-only mode 2022-05-14T04:10:00.300 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:10:00 smithi047 ceph-mon[77147]: [WRN] MDS_SLOW_METADATA_IO: 1 MDSs report slow metadata IOs 2022-05-14T04:10:00.300 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:10:00 smithi047 ceph-mon[77147]: mds.l(mds.0): 4 slow metadata IOs are blocked > 180 secs, oldest blocked for 17233 secs 2022-05-14T04:10:00.300 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:10:00 smithi047 ceph-mon[77147]: [WRN] MON_DOWN: 1/3 mons down, quorum b,c 2022-05-14T04:10:00.300 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:10:00 smithi047 ceph-mon[77147]: mon.a (rank 0) addr [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] is down (out of quorum) 2022-05-14T04:10:00.301 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:10:00 smithi047 ceph-mon[77147]: [WRN] PG_AVAILABILITY: Reduced data availability: 1 pg inactive 2022-05-14T04:10:00.301 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:10:00 smithi047 ceph-mon[77147]: pg 2.1 is stuck inactive for 4h, current state unknown, last acting [] 2022-05-14T04:10:00.301 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:10:00 smithi047 ceph-mon[77147]: [WRN] SLOW_OPS: 9 slow ops, oldest one blocked for 16930 sec, osd.5 has slow ops 2022-05-14T04:10:00.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:10:00 smithi115 ceph-mon[78593]: pgmap v8817: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:10:00.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:10:00 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:10:00.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:10:00 smithi115 ceph-mon[78593]: Health detail: HEALTH_ERR 1 filesystem is degraded; 1 mds daemon damaged; 1 MDSs are read only; 1 MDSs report slow metadata IOs; 1/3 mons down, quorum b,c; Reduced data availability: 1 pg inactive; 9 slow ops, oldest one blocked for 16930 sec, osd.5 has slow ops 2022-05-14T04:10:00.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:10:00 smithi115 ceph-mon[78593]: [WRN] FS_DEGRADED: 1 filesystem is degraded 2022-05-14T04:10:00.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:10:00 smithi115 ceph-mon[78593]: fs cephfs is degraded 2022-05-14T04:10:00.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:10:00 smithi115 ceph-mon[78593]: [ERR] MDS_DAMAGE: 1 mds daemon damaged 2022-05-14T04:10:00.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:10:00 smithi115 ceph-mon[78593]: fs cephfs mds.2 is damaged 2022-05-14T04:10:00.612 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:10:00 smithi115 ceph-mon[78593]: [WRN] MDS_READ_ONLY: 1 MDSs are read only 2022-05-14T04:10:00.612 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:10:00 smithi115 ceph-mon[78593]: mds.l(mds.0): MDS in read-only mode 2022-05-14T04:10:00.612 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:10:00 smithi115 ceph-mon[78593]: [WRN] MDS_SLOW_METADATA_IO: 1 MDSs report slow metadata IOs 2022-05-14T04:10:00.612 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:10:00 smithi115 ceph-mon[78593]: mds.l(mds.0): 4 slow metadata IOs are blocked > 180 secs, oldest blocked for 17233 secs 2022-05-14T04:10:00.612 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:10:00 smithi115 ceph-mon[78593]: [WRN] MON_DOWN: 1/3 mons down, quorum b,c 2022-05-14T04:10:00.613 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:10:00 smithi115 ceph-mon[78593]: mon.a (rank 0) addr [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] is down (out of quorum) 2022-05-14T04:10:00.613 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:10:00 smithi115 ceph-mon[78593]: [WRN] PG_AVAILABILITY: Reduced data availability: 1 pg inactive 2022-05-14T04:10:00.613 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:10:00 smithi115 ceph-mon[78593]: pg 2.1 is stuck inactive for 4h, current state unknown, last acting [] 2022-05-14T04:10:00.613 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:10:00 smithi115 ceph-mon[78593]: [WRN] SLOW_OPS: 9 slow ops, oldest one blocked for 16930 sec, osd.5 has slow ops 2022-05-14T04:10:01.110 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:10:00 smithi115 conmon[90393]: 2022-05-14T04:10:00.689+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:10:01.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:10:01 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:10:01.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:10:01 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:10:02.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:10:01 smithi115 conmon[90393]: 2022-05-14T04:10:01.669+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:10:02.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:10:02 smithi047 ceph-mon[77147]: pgmap v8818: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:10:02.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:10:02 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:10:02.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:10:02 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:10:02.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:10:02 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:10:02.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:10:02 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 16935 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:10:02.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:10:02 smithi115 ceph-mon[78593]: pgmap v8818: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:10:02.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:10:02 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:10:02.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:10:02 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:10:02.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:10:02 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:10:02.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:10:02 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 16935 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:10:03.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:10:02 smithi115 conmon[90393]: 2022-05-14T04:10:02.642+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:10:03.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:10:03 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:10:03.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:10:03 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:10:04.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:10:03 smithi115 conmon[90393]: 2022-05-14T04:10:03.678+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:10:04.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:10:04 smithi047 ceph-mon[77147]: pgmap v8819: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:10:04.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:10:04 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:10:04.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:10:04 smithi115 ceph-mon[78593]: pgmap v8819: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:10:04.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:10:04 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:10:05.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:10:04 smithi115 conmon[90393]: 2022-05-14T04:10:04.670+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:10:05.546 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:10:05 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:10:05.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:10:05 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:10:06.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:10:05 smithi115 conmon[90393]: 2022-05-14T04:10:05.680+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:10:06.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:10:06 smithi047 ceph-mon[77147]: pgmap v8820: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:10:06.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:10:06 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:10:06.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:10:06 smithi115 ceph-mon[78593]: pgmap v8820: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:10:06.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:10:06 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:10:07.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:10:06 smithi115 conmon[90393]: 2022-05-14T04:10:06.662+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:10:07.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:10:07 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:10:07.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:10:07 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 16940 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:10:07.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:10:07 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:10:07.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:10:07 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 16940 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:10:08.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:10:07 smithi115 conmon[90393]: 2022-05-14T04:10:07.690+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:10:08.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:10:08 smithi047 ceph-mon[77147]: pgmap v8821: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:10:08.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:10:08 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:10:08.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:10:08 smithi115 ceph-mon[78593]: pgmap v8821: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:10:08.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:10:08 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:10:09.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:10:08 smithi115 conmon[90393]: 2022-05-14T04:10:08.666+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:10:09.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:10:09 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:10:09.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:10:09 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:10:10.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:10:09 smithi115 conmon[90393]: 2022-05-14T04:10:09.688+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:10:10.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:10:10 smithi047 ceph-mon[77147]: pgmap v8822: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:10:10.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:10:10 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:10:10.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:10:10 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:10:10.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:10:10 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:10:10.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:10:10 smithi115 ceph-mon[78593]: pgmap v8822: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:10:10.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:10:10 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:10:10.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:10:10 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:10:10.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:10:10 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:10:11.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:10:10 smithi115 conmon[90393]: 2022-05-14T04:10:10.655+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:10:11.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:10:11 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:10:11.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:10:11 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:10:12.110 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:10:11 smithi115 conmon[90393]: 2022-05-14T04:10:11.614+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:10:12.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:10:12 smithi047 ceph-mon[77147]: pgmap v8823: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:10:12.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:10:12 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:10:12.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:10:12 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:10:12.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:10:12 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 16945 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:10:12.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:10:12 smithi115 ceph-mon[78593]: pgmap v8823: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:10:12.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:10:12 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:10:12.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:10:12 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:10:12.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:10:12 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 16945 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:10:13.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:10:12 smithi115 conmon[90393]: 2022-05-14T04:10:12.642+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:10:13.546 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:10:13 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:10:13.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:10:13 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:10:14.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:10:13 smithi115 conmon[90393]: 2022-05-14T04:10:13.621+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:10:14.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:10:14 smithi047 ceph-mon[77147]: pgmap v8824: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:10:14.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:10:14 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:10:14.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:10:14 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:10:14.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:10:14 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:10:14.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:10:14 smithi115 ceph-mon[78593]: pgmap v8824: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:10:14.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:10:14 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:10:14.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:10:14 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:10:14.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:10:14 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:10:15.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:10:14 smithi115 conmon[90393]: 2022-05-14T04:10:14.672+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:10:15.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:10:15 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T04:10:15.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:10:15 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T04:10:15.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:10:15 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:10:15.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:10:15 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T04:10:15.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:10:15 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T04:10:15.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:10:15 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:10:16.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:10:15 smithi115 conmon[90393]: 2022-05-14T04:10:15.645+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:10:16.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:10:16 smithi047 ceph-mon[77147]: pgmap v8825: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:10:16.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:10:16 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:10:16.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:10:16 smithi115 ceph-mon[78593]: pgmap v8825: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:10:16.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:10:16 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:10:17.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:10:16 smithi115 conmon[90393]: 2022-05-14T04:10:16.648+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:10:17.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:10:17 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:10:17.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:10:17 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 16950 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:10:17.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:10:17 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:10:17.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:10:17 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 16950 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:10:18.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:10:17 smithi115 conmon[90393]: 2022-05-14T04:10:17.630+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:10:18.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:10:18 smithi047 ceph-mon[77147]: pgmap v8826: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:10:18.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:10:18 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:10:18.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:10:18 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:10:18.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:10:18 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:10:18.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:10:18 smithi115 ceph-mon[78593]: pgmap v8826: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:10:18.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:10:18 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:10:18.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:10:18 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:10:18.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:10:18 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:10:19.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:10:18 smithi115 conmon[90393]: 2022-05-14T04:10:18.627+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:10:19.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:10:19 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:10:19.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:10:19 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:10:20.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:10:19 smithi115 conmon[90393]: 2022-05-14T04:10:19.619+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:10:20.578 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:10:20 smithi115 ceph-mon[78593]: pgmap v8827: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:10:20.578 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:10:20 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:10:20.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:10:20 smithi047 ceph-mon[77147]: pgmap v8827: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:10:20.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:10:20 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:10:20.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:10:20 smithi115 conmon[90393]: 2022-05-14T04:10:20.575+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:10:21.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:10:21 smithi115 conmon[90393]: 2022-05-14T04:10:21.548+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:10:21.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:10:21 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:10:21.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:10:21 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:10:22.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:10:22 smithi115 conmon[90393]: 2022-05-14T04:10:22.541+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:10:22.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:10:22 smithi115 ceph-mon[78593]: pgmap v8828: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:10:22.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:10:22 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:10:22.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:10:22 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:10:22.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:10:22 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:10:22.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:10:22 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 16955 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:10:22.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:10:22 smithi047 ceph-mon[77147]: pgmap v8828: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:10:22.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:10:22 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:10:22.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:10:22 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:10:22.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:10:22 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:10:22.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:10:22 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 16955 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:10:23.594 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:10:23 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:10:23.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:10:23 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:10:23.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:10:23 smithi115 conmon[90393]: 2022-05-14T04:10:23.590+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:10:24.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:10:24 smithi115 conmon[90393]: 2022-05-14T04:10:24.552+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:10:24.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:10:24 smithi115 ceph-mon[78593]: pgmap v8829: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:10:24.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:10:24 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:10:24.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:10:24 smithi047 ceph-mon[77147]: pgmap v8829: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:10:24.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:10:24 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:10:25.591 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:10:25 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:10:25.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:10:25 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:10:25.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:10:25 smithi115 conmon[90393]: 2022-05-14T04:10:25.588+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:10:26.597 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:10:26 smithi115 ceph-mon[78593]: pgmap v8830: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:10:26.598 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:10:26 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:10:26.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:10:26 smithi047 ceph-mon[77147]: pgmap v8830: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:10:26.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:10:26 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:10:26.860 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:10:26 smithi115 conmon[90393]: 2022-05-14T04:10:26.594+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:10:27.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:10:27 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:10:27.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:10:27 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 16960 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:10:27.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:10:27 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:10:27.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:10:27 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 16960 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:10:27.861 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:10:27 smithi115 conmon[90393]: 2022-05-14T04:10:27.604+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:10:28.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:10:28 smithi115 ceph-mon[78593]: pgmap v8831: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:10:28.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:10:28 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:10:28.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:10:28 smithi047 ceph-mon[77147]: pgmap v8831: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:10:28.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:10:28 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:10:29.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:10:28 smithi115 conmon[90393]: 2022-05-14T04:10:28.619+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:10:29.582 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:10:29 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:10:29.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:10:29 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:10:29.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:10:29 smithi115 conmon[90393]: 2022-05-14T04:10:29.579+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:10:30.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:10:30 smithi115 ceph-mon[78593]: pgmap v8832: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:10:30.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:10:30 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:10:30.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:10:30 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:10:30.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:10:30 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:10:30.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:10:30 smithi047 ceph-mon[77147]: pgmap v8832: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:10:30.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:10:30 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:10:30.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:10:30 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:10:30.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:10:30 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:10:31.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:10:30 smithi115 conmon[90393]: 2022-05-14T04:10:30.622+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:10:31.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:10:31 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:10:31.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:10:31 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:10:32.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:10:31 smithi115 conmon[90393]: 2022-05-14T04:10:31.649+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:10:32.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:10:32 smithi115 ceph-mon[78593]: pgmap v8833: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:10:32.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:10:32 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:10:32.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:10:32 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 16965 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:10:32.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:10:32 smithi047 ceph-mon[77147]: pgmap v8833: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:10:32.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:10:32 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:10:32.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:10:32 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 16965 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:10:33.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:10:32 smithi115 conmon[90393]: 2022-05-14T04:10:32.648+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:10:33.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:10:33 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:10:33.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:10:33 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:10:34.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:10:33 smithi115 conmon[90393]: 2022-05-14T04:10:33.641+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:10:34.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:10:34 smithi115 ceph-mon[78593]: pgmap v8834: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:10:34.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:10:34 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:10:34.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:10:34 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:10:34.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:10:34 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:10:34.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:10:34 smithi047 ceph-mon[77147]: pgmap v8834: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:10:34.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:10:34 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:10:34.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:10:34 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:10:34.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:10:34 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:10:35.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:10:34 smithi115 conmon[90393]: 2022-05-14T04:10:34.667+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:10:35.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:10:35 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:10:35.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:10:35 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:10:36.110 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:10:35 smithi115 conmon[90393]: 2022-05-14T04:10:35.681+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:10:36.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:10:36 smithi115 ceph-mon[78593]: pgmap v8835: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:10:36.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:10:36 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:10:36.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:10:36 smithi047 ceph-mon[77147]: pgmap v8835: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:10:36.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:10:36 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:10:37.110 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:10:36 smithi115 conmon[90393]: 2022-05-14T04:10:36.640+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:10:37.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:10:37 smithi115 conmon[90393]: 2022-05-14T04:10:37.598+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:10:37.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:10:37 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:10:37.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:10:37 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 16970 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:10:37.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:10:37 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:10:37.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:10:37 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 16970 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:10:38.610 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:10:38 smithi115 conmon[90393]: 2022-05-14T04:10:38.588+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:10:38.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:10:38 smithi115 ceph-mon[78593]: pgmap v8836: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:10:38.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:10:38 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:10:38.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:10:38 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:10:38.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:10:38 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:10:38.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:10:38 smithi047 ceph-mon[77147]: pgmap v8836: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:10:38.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:10:38 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:10:38.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:10:38 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:10:38.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:10:38 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:10:39.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:10:39 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:10:39.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:10:39 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T04:10:39.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:10:39 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T04:10:39.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:10:39 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T04:10:39.583 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:10:39 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:10:39.584 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:10:39 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T04:10:39.584 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:10:39 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T04:10:39.584 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:10:39 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T04:10:39.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:10:39 smithi115 conmon[90393]: 2022-05-14T04:10:39.580+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:10:40.618 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:10:40 smithi115 ceph-mon[78593]: pgmap v8837: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:10:40.619 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:10:40 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:10:40.619 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:10:40 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' 2022-05-14T04:10:40.619 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:10:40 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' 2022-05-14T04:10:40.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:10:40 smithi047 ceph-mon[77147]: pgmap v8837: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:10:40.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:10:40 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:10:40.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:10:40 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' 2022-05-14T04:10:40.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:10:40 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' 2022-05-14T04:10:41.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:10:40 smithi115 conmon[90393]: 2022-05-14T04:10:40.615+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:10:41.649 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:10:41 smithi115 ceph-mon[78593]: pgmap v8838: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:10:41.649 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:10:41 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:10:41.798 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:10:41 smithi047 ceph-mon[77147]: pgmap v8838: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:10:41.799 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:10:41 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:10:42.110 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:10:41 smithi115 conmon[90393]: 2022-05-14T04:10:41.646+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:10:42.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:10:42 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:10:42.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:10:42 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:10:42.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:10:42 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:10:42.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:10:42 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' 2022-05-14T04:10:42.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:10:42 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 16975 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:10:42.545 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:10:42 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' 2022-05-14T04:10:42.545 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:10:42 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:10:42.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T04:10:42.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:10:42.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T04:10:42.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:10:42.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:10:42.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:10:42.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T04:10:42.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:10:42.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T04:10:42.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:10:42.548 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T04:10:42.548 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:10:42.548 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T04:10:42.548 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:10:42.548 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T04:10:42.549 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:10:42.549 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:10:42.549 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:10:42.549 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:10:42 smithi047 ceph-mon[77147]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T04:10:42.550 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:10:42 smithi047 conmon[78521]: 2022-05-14T04:10:42.287+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:10:42.550 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:10:42 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T04:10:42.550 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:10:42 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:10:42.551 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:10:42 smithi047 conmon[78521]: yield 2022-05-14T04:10:42.551 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:10:42 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:10:42.551 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:10:42 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:10:42.552 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:10:42 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:10:42.552 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:10:42 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T04:10:42.552 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:10:42 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:10:42.552 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:10:42 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T04:10:42.553 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:10:42 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:10:42.553 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:10:42 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T04:10:42.553 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:10:42 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:10:42.554 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:10:42 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T04:10:42.554 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:10:42 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:10:42.554 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:10:42 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T04:10:42.554 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:10:42 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:10:42.554 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:10:42 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:10:42.555 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:10:42 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:10:42.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:10:42 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:10:42.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:10:42 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:10:42.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:10:42 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:10:42.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:10:42 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' 2022-05-14T04:10:42.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:10:42 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 16975 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:10:42.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:10:42 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' 2022-05-14T04:10:42.612 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:10:42 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:10:42.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T04:10:42.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:10:42.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T04:10:42.613 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:10:42.613 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:10:42.613 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:10:42.613 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T04:10:42.614 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:10:42.614 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T04:10:42.614 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:10:42.614 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T04:10:42.615 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:10:42.615 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T04:10:42.615 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:10:42.615 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T04:10:42.615 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:10:42.616 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:10:42.616 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:10:42.616 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:10:42 smithi115 ceph-mon[78593]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T04:10:43.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:10:42 smithi115 conmon[90393]: 2022-05-14T04:10:42.621+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:10:43.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:10:43 smithi047 ceph-mon[77147]: pgmap v8839: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:10:43.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:10:43 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:10:43.795 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:10:43 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-05-14T04:10:43.795 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:10:43 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-05-14T04:10:43.795 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:10:43 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-05-14T04:10:43.795 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:10:43 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-05-14T04:10:43.796 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:10:43 smithi047 ceph-mon[77147]: Adjusting osd_memory_target on smithi047 to 198.1M 2022-05-14T04:10:43.796 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:10:43 smithi047 ceph-mon[77147]: Unable to set osd_memory_target on smithi047 to 207755878: error parsing value: Value '207755878' is below minimum 939524096 2022-05-14T04:10:43.796 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:10:43 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-05-14T04:10:43.796 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:10:43 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-05-14T04:10:43.796 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:10:43 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-05-14T04:10:43.797 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:10:43 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-05-14T04:10:43.797 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:10:43 smithi047 ceph-mon[77147]: Adjusting osd_memory_target on smithi115 to 197.0M 2022-05-14T04:10:43.797 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:10:43 smithi047 ceph-mon[77147]: Unable to set osd_memory_target on smithi115 to 206629785: error parsing value: Value '206629785' is below minimum 939524096 2022-05-14T04:10:43.862 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:10:43 smithi115 conmon[90393]: 2022-05-14T04:10:43.577+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:10:43.863 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:10:43 smithi115 ceph-mon[78593]: pgmap v8839: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:10:43.863 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:10:43 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:10:43.864 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:10:43 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-05-14T04:10:43.864 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:10:43 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-05-14T04:10:43.864 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:10:43 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-05-14T04:10:43.865 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:10:43 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-05-14T04:10:43.865 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:10:43 smithi115 ceph-mon[78593]: Adjusting osd_memory_target on smithi047 to 198.1M 2022-05-14T04:10:43.865 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:10:43 smithi115 ceph-mon[78593]: Unable to set osd_memory_target on smithi047 to 207755878: error parsing value: Value '207755878' is below minimum 939524096 2022-05-14T04:10:43.866 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:10:43 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-05-14T04:10:43.866 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:10:43 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-05-14T04:10:43.867 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:10:43 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-05-14T04:10:43.867 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:10:43 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-05-14T04:10:43.868 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:10:43 smithi115 ceph-mon[78593]: Adjusting osd_memory_target on smithi115 to 197.0M 2022-05-14T04:10:43.868 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:10:43 smithi115 ceph-mon[78593]: Unable to set osd_memory_target on smithi115 to 206629785: error parsing value: Value '206629785' is below minimum 939524096 2022-05-14T04:10:44.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:10:44 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:10:44.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:10:44 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:10:44.860 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:10:44 smithi115 conmon[90393]: 2022-05-14T04:10:44.547+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:10:45.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:10:45 smithi047 ceph-mon[77147]: pgmap v8840: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:10:45.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:10:45 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:10:45.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:10:45 smithi047 conmon[78521]: 2022-05-14T04:10:45.359+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:10:45.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:10:45 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T04:10:45.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:10:45 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:10:45.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:10:45 smithi047 conmon[78521]: yield 2022-05-14T04:10:45.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:10:45 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:10:45.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:10:45 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:10:45.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:10:45 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:10:45.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:10:45 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T04:10:45.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:10:45 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:10:45.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:10:45 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T04:10:45.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:10:45 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:10:45.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:10:45 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T04:10:45.797 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:10:45 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:10:45.800 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:10:45 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T04:10:45.800 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:10:45 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:10:45.801 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:10:45 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T04:10:45.801 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:10:45 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:10:45.801 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:10:45 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:10:45.801 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:10:45 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:10:45.802 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:10:45 smithi047 conmon[78521]: 2022-05-14T04:10:45.360+0000 7f5344781700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T04:10:45.802 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:10:45 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T04:10:45.802 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:10:45 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:10:45.803 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:10:45 smithi047 conmon[78521]: yield 2022-05-14T04:10:45.803 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:10:45 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:10:45.803 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:10:45 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:10:45.803 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:10:45 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:10:45.803 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:10:45 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T04:10:45.804 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:10:45 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:10:45.804 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:10:45 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T04:10:45.804 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:10:45 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:10:45.804 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:10:45 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T04:10:45.804 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:10:45 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:10:45.805 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:10:45 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T04:10:45.805 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:10:45 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:10:45.805 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:10:45 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T04:10:45.805 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:10:45 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:10:45.806 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:10:45 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:10:45.806 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:10:45 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:10:45.806 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:10:45 smithi047 conmon[78521]: 2022-05-14T04:10:45.807 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:10:45 smithi047 conmon[78521]: During handling of the above exception, another exception occurred: 2022-05-14T04:10:45.807 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:10:45 smithi047 conmon[78521]: 2022-05-14T04:10:45.807 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:10:45 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T04:10:45.807 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:10:45 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T04:10:45.808 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:10:45 smithi047 conmon[78521]: return f(*arg) 2022-05-14T04:10:45.808 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:10:45 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T04:10:45.808 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:10:45 smithi047 conmon[78521]: self._write_client_files(client_files, host) 2022-05-14T04:10:45.808 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:10:45 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T04:10:45.808 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:10:45 smithi047 conmon[78521]: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T04:10:45.809 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:10:45 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T04:10:45.809 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:10:45 smithi047 conmon[78521]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T04:10:45.809 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:10:45 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T04:10:45.809 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:10:45 smithi047 conmon[78521]: return self.event_loop.get_result(coro) 2022-05-14T04:10:45.810 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:10:45 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T04:10:45.810 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:10:45 smithi047 conmon[78521]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T04:10:45.810 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:10:45 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T04:10:45.810 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:10:45 smithi047 conmon[78521]: return self.__get_result() 2022-05-14T04:10:45.811 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:10:45 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T04:10:45.812 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:10:45 smithi047 conmon[78521]: raise self._exception 2022-05-14T04:10:45.812 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:10:45 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T04:10:45.813 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:10:45 smithi047 conmon[78521]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T04:10:45.813 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:10:45 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T04:10:45.813 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:10:45 smithi047 conmon[78521]: conn = await self._remote_connection(host, addr) 2022-05-14T04:10:45.813 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:10:45 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T04:10:45.814 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:10:45 smithi047 conmon[78521]: raise 2022-05-14T04:10:45.814 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:10:45 smithi047 conmon[78521]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T04:10:45.814 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:10:45 smithi047 conmon[78521]: self.gen.throw(type, value, traceback) 2022-05-14T04:10:45.814 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:10:45 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T04:10:45.814 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:10:45 smithi047 conmon[78521]: raise OrchestratorError(msg) 2022-05-14T04:10:45.815 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:10:45 smithi047 conmon[78521]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:10:45.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:10:45 smithi115 conmon[90393]: 2022-05-14T04:10:45.510+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:10:45.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:10:45 smithi115 ceph-mon[78593]: pgmap v8840: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:10:45.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:10:45 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:10:46.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:10:46 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:10:46.793 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T04:10:46.793 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:10:46.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T04:10:46.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:10:46.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:10:46.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:10:46.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T04:10:46.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:10:46.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T04:10:46.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:10:46.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T04:10:46.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:10:46.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T04:10:46.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:10:46.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T04:10:46.797 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:10:46.797 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:10:46.797 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:10:46.797 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:10:46 smithi047 ceph-mon[77147]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T04:10:46.797 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T04:10:46.798 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:10:46.798 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T04:10:46.798 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:10:46.798 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:10:46.799 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:10:46.799 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T04:10:46.799 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:10:46.799 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T04:10:46.799 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:10:46.800 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T04:10:46.800 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:10:46.800 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T04:10:46.800 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:10:46.800 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T04:10:46.801 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:10:46.801 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:10:46.801 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:10:46.801 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T04:10:46.802 INFO:journalctl@ceph.mon.b.smithi047.stdout: During handling of the above exception, another exception occurred: 2022-05-14T04:10:46.802 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T04:10:46.802 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T04:10:46.802 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T04:10:46.802 INFO:journalctl@ceph.mon.b.smithi047.stdout: return f(*arg) 2022-05-14T04:10:46.803 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T04:10:46.803 INFO:journalctl@ceph.mon.b.smithi047.stdout: self._write_client_files(client_files, host) 2022-05-14T04:10:46.803 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T04:10:46.803 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T04:10:46.804 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T04:10:46.804 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T04:10:46.804 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T04:10:46.804 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.event_loop.get_result(coro) 2022-05-14T04:10:46.804 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T04:10:46.805 INFO:journalctl@ceph.mon.b.smithi047.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T04:10:46.805 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T04:10:46.805 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.__get_result() 2022-05-14T04:10:46.805 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T04:10:46.805 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise self._exception 2022-05-14T04:10:46.806 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T04:10:46.806 INFO:journalctl@ceph.mon.b.smithi047.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T04:10:46.806 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T04:10:46.806 INFO:journalctl@ceph.mon.b.smithi047.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T04:10:46.807 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T04:10:46.807 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise 2022-05-14T04:10:46.807 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T04:10:46.807 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.gen.throw(type, value, traceback) 2022-05-14T04:10:46.807 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T04:10:46.808 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OrchestratorError(msg) 2022-05-14T04:10:46.808 INFO:journalctl@ceph.mon.b.smithi047.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:10:46.808 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:10:46 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:10:46.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:10:46 smithi115 conmon[90393]: 2022-05-14T04:10:46.518+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:10:46.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:10:46 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:10:46.860 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T04:10:46.860 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:10:46.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T04:10:46.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:10:46.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:10:46.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:10:46.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T04:10:46.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:10:46.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T04:10:46.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:10:46.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T04:10:46.863 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:10:46.863 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T04:10:46.863 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:10:46.863 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T04:10:46.864 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:10:46.864 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:10:46.864 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:10:46.864 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:10:46 smithi115 ceph-mon[78593]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T04:10:46.864 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T04:10:46.865 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:10:46.865 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T04:10:46.865 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:10:46.865 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:10:46.866 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:10:46.866 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T04:10:46.866 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:10:46.866 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T04:10:46.866 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:10:46.867 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T04:10:46.867 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:10:46.867 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T04:10:46.867 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:10:46.867 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T04:10:46.868 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:10:46.868 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:10:46.868 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:10:46.868 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T04:10:46.869 INFO:journalctl@ceph.mon.c.smithi115.stdout: During handling of the above exception, another exception occurred: 2022-05-14T04:10:46.869 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T04:10:46.869 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T04:10:46.869 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T04:10:46.870 INFO:journalctl@ceph.mon.c.smithi115.stdout: return f(*arg) 2022-05-14T04:10:46.870 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T04:10:46.870 INFO:journalctl@ceph.mon.c.smithi115.stdout: self._write_client_files(client_files, host) 2022-05-14T04:10:46.870 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T04:10:46.871 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T04:10:46.871 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T04:10:46.871 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T04:10:46.871 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T04:10:46.871 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.event_loop.get_result(coro) 2022-05-14T04:10:46.872 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T04:10:46.872 INFO:journalctl@ceph.mon.c.smithi115.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T04:10:46.872 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T04:10:46.872 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.__get_result() 2022-05-14T04:10:46.873 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T04:10:46.873 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise self._exception 2022-05-14T04:10:46.873 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T04:10:46.873 INFO:journalctl@ceph.mon.c.smithi115.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T04:10:46.874 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T04:10:46.874 INFO:journalctl@ceph.mon.c.smithi115.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T04:10:46.874 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T04:10:46.875 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise 2022-05-14T04:10:46.875 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T04:10:46.875 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.gen.throw(type, value, traceback) 2022-05-14T04:10:46.875 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T04:10:46.875 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OrchestratorError(msg) 2022-05-14T04:10:46.876 INFO:journalctl@ceph.mon.c.smithi115.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:10:46.876 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:10:46 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:10:47.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:10:47 smithi047 ceph-mon[77147]: pgmap v8841: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:10:47.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:10:47 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:10:47.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:10:47 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 16980 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:10:47.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:10:47 smithi115 conmon[90393]: 2022-05-14T04:10:47.515+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:10:47.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:10:47 smithi115 ceph-mon[78593]: pgmap v8841: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:10:47.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:10:47 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:10:47.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:10:47 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 16980 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:10:48.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:10:48 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:10:48.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:10:48 smithi115 conmon[90393]: 2022-05-14T04:10:48.501+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:10:48.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:10:48 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:10:49.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:10:49 smithi047 ceph-mon[77147]: pgmap v8842: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:10:49.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:10:49 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:10:49.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:10:49 smithi115 conmon[90393]: 2022-05-14T04:10:49.452+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:10:49.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:10:49 smithi115 ceph-mon[78593]: pgmap v8842: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:10:49.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:10:49 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:10:50.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:10:50 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:10:50.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:10:50 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:10:50.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:10:50 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:10:50.862 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:10:50 smithi115 conmon[90393]: 2022-05-14T04:10:50.462+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:10:50.863 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:10:50 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:10:50.863 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:10:50 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:10:50.863 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:10:50 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:10:51.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:10:51 smithi047 ceph-mon[77147]: pgmap v8843: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:10:51.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:10:51 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:10:51.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:10:51 smithi115 conmon[90393]: 2022-05-14T04:10:51.427+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:10:51.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:10:51 smithi115 ceph-mon[78593]: pgmap v8843: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:10:51.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:10:51 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:10:52.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:10:52 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:10:52.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:10:52 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 16985 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:10:52.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:10:52 smithi115 conmon[90393]: 2022-05-14T04:10:52.397+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:10:52.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:10:52 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:10:52.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:10:52 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 16985 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:10:53.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:10:53 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:10:53.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:10:53 smithi047 ceph-mon[77147]: pgmap v8844: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:10:53.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:10:53 smithi115 conmon[90393]: 2022-05-14T04:10:53.387+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:10:53.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:10:53 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:10:53.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:10:53 smithi115 ceph-mon[78593]: pgmap v8844: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:10:54.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:10:54 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:10:54.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:10:54 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:10:54.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:10:54 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:10:54.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:10:54 smithi115 conmon[90393]: 2022-05-14T04:10:54.401+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:10:54.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:10:54 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:10:54.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:10:54 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:10:54.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:10:54 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:10:55.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:10:55 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:10:55.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:10:55 smithi047 ceph-mon[77147]: pgmap v8845: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:10:55.860 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:10:55 smithi115 conmon[90393]: 2022-05-14T04:10:55.443+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:10:55.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:10:55 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:10:55.862 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:10:55 smithi115 ceph-mon[78593]: pgmap v8845: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:10:56.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:10:56 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:10:56.866 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:10:56 smithi115 conmon[90393]: 2022-05-14T04:10:56.442+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:10:56.867 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:10:56 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:10:57.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:10:57 smithi047 ceph-mon[77147]: pgmap v8846: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:10:57.795 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:10:57 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:10:57.796 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:10:57 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 16990 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:10:57.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:10:57 smithi115 conmon[90393]: 2022-05-14T04:10:57.454+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:10:57.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:10:57 smithi115 ceph-mon[78593]: pgmap v8846: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:10:57.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:10:57 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:10:57.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:10:57 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 16990 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:10:58.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:10:58 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:10:58.795 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:10:58 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:10:58.795 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:10:58 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:10:58.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:10:58 smithi115 conmon[90393]: 2022-05-14T04:10:58.474+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:10:58.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:10:58 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:10:58.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:10:58 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:10:58.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:10:58 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:10:59.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:10:59 smithi047 ceph-mon[77147]: pgmap v8847: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:10:59.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:10:59 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:10:59.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:10:59 smithi115 conmon[90393]: 2022-05-14T04:10:59.453+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:10:59.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:10:59 smithi115 ceph-mon[78593]: pgmap v8847: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:10:59.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:10:59 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:11:00.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:11:00 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:11:00.861 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:11:00 smithi115 conmon[90393]: 2022-05-14T04:11:00.440+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:11:00.862 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:11:00 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:11:01.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:11:01 smithi047 ceph-mon[77147]: pgmap v8848: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:11:01.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:11:01 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:11:01.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:11:01 smithi115 conmon[90393]: 2022-05-14T04:11:01.436+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:11:01.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:11:01 smithi115 ceph-mon[78593]: pgmap v8848: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:11:01.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:11:01 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:11:02.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:11:02 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:11:02.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:11:02 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:11:02.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:11:02 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:11:02.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:11:02 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 16995 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:11:02.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:11:02 smithi115 conmon[90393]: 2022-05-14T04:11:02.482+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:11:02.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:11:02 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:11:02.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:11:02 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:11:02.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:11:02 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:11:02.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:11:02 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 16995 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:11:03.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:11:03 smithi115 conmon[90393]: 2022-05-14T04:11:03.468+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:11:04.460 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:11:04 smithi115 ceph-mon[78593]: pgmap v8849: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:11:04.461 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:11:04 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:11:04.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:11:04 smithi047 ceph-mon[77147]: pgmap v8849: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:11:04.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:11:04 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:11:04.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:11:04 smithi115 conmon[90393]: 2022-05-14T04:11:04.456+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:11:05.508 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:11:05 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:11:05.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:11:05 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:11:05.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:11:05 smithi115 conmon[90393]: 2022-05-14T04:11:05.506+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:11:06.527 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:11:06 smithi115 ceph-mon[78593]: pgmap v8850: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:11:06.527 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:11:06 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:11:06.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:11:06 smithi047 ceph-mon[77147]: pgmap v8850: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:11:06.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:11:06 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:11:06.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:11:06 smithi115 conmon[90393]: 2022-05-14T04:11:06.524+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:11:07.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:11:07 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:11:07.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:11:07 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 17000 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:11:07.573 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:11:07 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:11:07.573 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:11:07 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 17000 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:11:07.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:11:07 smithi115 conmon[90393]: 2022-05-14T04:11:07.571+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:11:08.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:11:08 smithi047 ceph-mon[77147]: pgmap v8851: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:11:08.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:11:08 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:11:08.554 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:11:08 smithi115 ceph-mon[78593]: pgmap v8851: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:11:08.555 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:11:08 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:11:08.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:11:08 smithi115 conmon[90393]: 2022-05-14T04:11:08.552+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:11:09.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:11:09 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:11:09.602 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:11:09 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:11:09.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:11:09 smithi115 conmon[90393]: 2022-05-14T04:11:09.600+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:11:10.545 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:11:10 smithi047 ceph-mon[77147]: pgmap v8852: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:11:10.546 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:11:10 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:11:10.546 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:11:10 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:11:10.546 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:11:10 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:11:10.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:11:10 smithi115 ceph-mon[78593]: pgmap v8852: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:11:10.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:11:10 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:11:10.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:11:10 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:11:10.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:11:10 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:11:11.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:11:10 smithi115 conmon[90393]: 2022-05-14T04:11:10.626+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:11:11.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:11:11 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:11:11.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:11:11 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:11:12.110 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:11:11 smithi115 conmon[90393]: 2022-05-14T04:11:11.628+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:11:12.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:11:12 smithi047 ceph-mon[77147]: pgmap v8853: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:11:12.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:11:12 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:11:12.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:11:12 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 17005 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:11:12.591 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:11:12 smithi115 ceph-mon[78593]: pgmap v8853: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:11:12.591 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:11:12 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:11:12.591 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:11:12 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 17005 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:11:12.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:11:12 smithi115 conmon[90393]: 2022-05-14T04:11:12.589+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:11:13.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:11:13 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:11:13.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:11:13 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:11:14.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:11:13 smithi115 conmon[90393]: 2022-05-14T04:11:13.618+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:11:14.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:11:14 smithi047 ceph-mon[77147]: pgmap v8854: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:11:14.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:11:14 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:11:14.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:11:14 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:11:14.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:11:14 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:11:14.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:11:14 smithi115 ceph-mon[78593]: pgmap v8854: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:11:14.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:11:14 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:11:14.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:11:14 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:11:14.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:11:14 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:11:15.110 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:11:14 smithi115 conmon[90393]: 2022-05-14T04:11:14.652+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:11:15.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:11:15 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:11:15.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:11:15 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T04:11:15.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:11:15 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T04:11:15.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:11:15 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:11:15.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:11:15 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T04:11:15.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:11:15 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T04:11:16.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:11:15 smithi115 conmon[90393]: 2022-05-14T04:11:15.681+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:11:16.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:11:16 smithi047 ceph-mon[77147]: pgmap v8855: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:11:16.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:11:16 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:11:16.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:11:16 smithi115 ceph-mon[78593]: pgmap v8855: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:11:16.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:11:16 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:11:17.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:11:16 smithi115 conmon[90393]: 2022-05-14T04:11:16.683+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:11:17.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:11:17 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:11:17.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:11:17 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 17010 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:11:17.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:11:17 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:11:17.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:11:17 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 17010 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:11:18.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:11:17 smithi115 conmon[90393]: 2022-05-14T04:11:17.690+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:11:18.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:11:18 smithi047 ceph-mon[77147]: pgmap v8856: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:11:18.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:11:18 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:11:18.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:11:18 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:11:18.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:11:18 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:11:18.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:11:18 smithi115 ceph-mon[78593]: pgmap v8856: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:11:18.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:11:18 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:11:18.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:11:18 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:11:18.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:11:18 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:11:19.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:11:18 smithi115 conmon[90393]: 2022-05-14T04:11:18.642+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:11:19.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:11:19 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:11:19.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:11:19 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:11:20.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:11:19 smithi115 conmon[90393]: 2022-05-14T04:11:19.627+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:11:20.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:11:20 smithi047 ceph-mon[77147]: pgmap v8857: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:11:20.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:11:20 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:11:20.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:11:20 smithi115 ceph-mon[78593]: pgmap v8857: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:11:20.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:11:20 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:11:21.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:11:20 smithi115 conmon[90393]: 2022-05-14T04:11:20.669+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:11:21.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:11:21 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:11:21.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:11:21 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:11:22.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:11:21 smithi115 conmon[90393]: 2022-05-14T04:11:21.664+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:11:22.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:11:22 smithi047 ceph-mon[77147]: pgmap v8858: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:11:22.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:11:22 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:11:22.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:11:22 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:11:22.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:11:22 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:11:22.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:11:22 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 17015 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:11:22.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:11:22 smithi115 ceph-mon[78593]: pgmap v8858: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:11:22.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:11:22 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:11:22.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:11:22 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:11:22.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:11:22 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:11:22.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:11:22 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 17015 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:11:23.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:11:22 smithi115 conmon[90393]: 2022-05-14T04:11:22.637+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:11:23.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:11:23 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:11:23.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:11:23 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:11:24.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:11:23 smithi115 conmon[90393]: 2022-05-14T04:11:23.653+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:11:24.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:11:24 smithi047 ceph-mon[77147]: pgmap v8859: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:11:24.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:11:24 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:11:24.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:11:24 smithi115 ceph-mon[78593]: pgmap v8859: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:11:24.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:11:24 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:11:25.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:11:24 smithi115 conmon[90393]: 2022-05-14T04:11:24.662+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:11:25.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:11:25 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:11:25.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:11:25 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:11:26.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:11:25 smithi115 conmon[90393]: 2022-05-14T04:11:25.643+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:11:26.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:11:26 smithi047 ceph-mon[77147]: pgmap v8860: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:11:26.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:11:26 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:11:26.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:11:26 smithi115 ceph-mon[78593]: pgmap v8860: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:11:26.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:11:26 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:11:27.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:11:26 smithi115 conmon[90393]: 2022-05-14T04:11:26.626+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:11:27.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:11:27 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:11:27.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:11:27 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 17020 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:11:27.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:11:27 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:11:27.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:11:27 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 17020 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:11:28.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:11:27 smithi115 conmon[90393]: 2022-05-14T04:11:27.607+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:11:28.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:11:28 smithi047 ceph-mon[77147]: pgmap v8861: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:11:28.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:11:28 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:11:28.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:11:28 smithi115 ceph-mon[78593]: pgmap v8861: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:11:28.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:11:28 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:11:29.110 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:11:28 smithi115 conmon[90393]: 2022-05-14T04:11:28.623+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:11:29.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:11:29 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:11:29.595 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:11:29 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:11:29.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:11:29 smithi115 conmon[90393]: 2022-05-14T04:11:29.592+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:11:30.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:11:30 smithi047 ceph-mon[77147]: pgmap v8862: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:11:30.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:11:30 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:11:30.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:11:30 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:11:30.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:11:30 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:11:30.558 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:11:30 smithi115 ceph-mon[78593]: pgmap v8862: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:11:30.559 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:11:30 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:11:30.559 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:11:30 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:11:30.559 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:11:30 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:11:30.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:11:30 smithi115 conmon[90393]: 2022-05-14T04:11:30.555+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:11:31.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:11:31 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:11:31.605 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:11:31 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:11:31.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:11:31 smithi115 conmon[90393]: 2022-05-14T04:11:31.602+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:11:32.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:11:32 smithi047 ceph-mon[77147]: pgmap v8863: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:11:32.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:11:32 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:11:32.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:11:32 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 17025 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:11:32.590 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:11:32 smithi115 ceph-mon[78593]: pgmap v8863: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:11:32.590 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:11:32 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:11:32.591 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:11:32 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 17025 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:11:32.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:11:32 smithi115 conmon[90393]: 2022-05-14T04:11:32.587+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:11:33.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:11:33 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:11:33.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:11:33 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:11:34.110 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:11:33 smithi115 conmon[90393]: 2022-05-14T04:11:33.632+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:11:34.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:11:34 smithi047 ceph-mon[77147]: pgmap v8864: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:11:34.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:11:34 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:11:34.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:11:34 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:11:34.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:11:34 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:11:34.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:11:34 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:11:34.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:11:34 smithi115 ceph-mon[78593]: pgmap v8864: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:11:34.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:11:34 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:11:34.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:11:34 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:11:34.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:11:34 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:11:34.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:11:34 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:11:35.110 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:11:34 smithi115 conmon[90393]: 2022-05-14T04:11:34.664+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:11:35.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:11:35 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:11:35.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:11:35 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:11:36.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:11:35 smithi115 conmon[90393]: 2022-05-14T04:11:35.709+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:11:36.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:11:36 smithi047 ceph-mon[77147]: pgmap v8865: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:11:36.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:11:36 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:11:36.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:11:36 smithi115 ceph-mon[78593]: pgmap v8865: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:11:36.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:11:36 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:11:37.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:11:36 smithi115 conmon[90393]: 2022-05-14T04:11:36.693+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:11:37.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:11:37 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:11:37.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:11:37 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 17030 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:11:37.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:11:37 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:11:37.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:11:37 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 17030 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:11:38.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:11:37 smithi115 conmon[90393]: 2022-05-14T04:11:37.694+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:11:38.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:11:38 smithi115 ceph-mon[78593]: pgmap v8866: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:11:38.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:11:38 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:11:38.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:11:38 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:11:38.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:11:38 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:11:38.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:11:38 smithi047 ceph-mon[77147]: pgmap v8866: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:11:38.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:11:38 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:11:38.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:11:38 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:11:38.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:11:38 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:11:39.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:11:38 smithi115 conmon[90393]: 2022-05-14T04:11:38.721+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:11:39.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:11:39 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:11:39.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:11:39 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:11:40.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:11:39 smithi115 conmon[90393]: 2022-05-14T04:11:39.688+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:11:40.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:11:40 smithi115 ceph-mon[78593]: pgmap v8867: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:11:40.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:11:40 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:11:40.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:11:40 smithi047 ceph-mon[77147]: pgmap v8867: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:11:40.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:11:40 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:11:41.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:11:40 smithi115 conmon[90393]: 2022-05-14T04:11:40.734+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:11:41.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:11:41 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:11:41.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:11:41 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:11:42.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:11:41 smithi115 conmon[90393]: 2022-05-14T04:11:41.715+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:11:42.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:11:42 smithi115 ceph-mon[78593]: pgmap v8868: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:11:42.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:11:42 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:11:42.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:11:42 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:11:42.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:11:42 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:11:42.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:11:42 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 17035 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:11:42.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:11:42 smithi047 ceph-mon[77147]: pgmap v8868: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:11:42.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:11:42 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:11:42.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:11:42 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:11:42.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:11:42 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:11:42.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:11:42 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 17035 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:11:43.110 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:11:42 smithi115 conmon[90393]: 2022-05-14T04:11:42.714+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:11:43.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:11:43 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:11:43.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:11:43 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:11:44.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:11:43 smithi115 conmon[90393]: 2022-05-14T04:11:43.760+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:11:44.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:11:44 smithi115 ceph-mon[78593]: pgmap v8869: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:11:44.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:11:44 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:11:44.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:11:44 smithi047 ceph-mon[77147]: pgmap v8869: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:11:44.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:11:44 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:11:45.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:11:44 smithi115 conmon[90393]: 2022-05-14T04:11:44.732+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:11:45.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:11:45 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:11:45.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:11:45 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:11:46.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:11:45 smithi115 conmon[90393]: 2022-05-14T04:11:45.743+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:11:46.293 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:11:45 smithi047 conmon[78521]: 2022-05-14T04:11:45.967+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:11:46.293 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:11:45 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T04:11:46.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:11:45 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:11:46.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:11:45 smithi047 conmon[78521]: yield 2022-05-14T04:11:46.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:11:45 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:11:46.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:11:45 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:11:46.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:11:45 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:11:46.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:11:45 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T04:11:46.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:11:45 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:11:46.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:11:45 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T04:11:46.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:11:45 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:11:46.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:11:45 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T04:11:46.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:11:45 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:11:46.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:11:45 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T04:11:46.297 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:11:45 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:11:46.297 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:11:45 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T04:11:46.298 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:11:45 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:11:46.298 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:11:45 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:11:46.298 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:11:45 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:11:46.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:11:46 smithi115 ceph-mon[78593]: pgmap v8870: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:11:46.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:11:46 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T04:11:46.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:11:46 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T04:11:46.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:11:46 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T04:11:46.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:11:46 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:11:46.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:11:46 smithi047 ceph-mon[77147]: pgmap v8870: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:11:46.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:11:46 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T04:11:46.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:11:46 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T04:11:46.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:11:46 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T04:11:46.795 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:11:46 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:11:47.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:11:46 smithi115 conmon[90393]: 2022-05-14T04:11:46.704+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:11:47.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:11:47 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:11:47.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T04:11:47.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:11:47.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T04:11:47.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:11:47.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:11:47.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:11:47.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T04:11:47.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:11:47.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T04:11:47.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:11:47.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T04:11:47.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:11:47.613 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T04:11:47.613 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:11:47.613 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T04:11:47.613 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:11:47.614 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:11:47.614 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:11:47.614 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:11:47 smithi115 ceph-mon[78593]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T04:11:47.614 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:11:47 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:11:47.614 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:11:47 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 17040 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:11:47.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:11:47 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:11:47.793 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T04:11:47.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:11:47.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T04:11:47.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:11:47.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:11:47.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:11:47.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T04:11:47.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:11:47.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T04:11:47.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:11:47.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T04:11:47.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:11:47.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T04:11:47.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:11:47.797 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T04:11:47.797 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:11:47.797 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:11:47.797 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:11:47.798 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:11:47 smithi047 ceph-mon[77147]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T04:11:47.798 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:11:47 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:11:47.798 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:11:47 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 17040 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:11:48.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:11:47 smithi115 conmon[90393]: 2022-05-14T04:11:47.691+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:11:48.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:11:48 smithi115 ceph-mon[78593]: pgmap v8871: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:11:48.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:11:48 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:11:48.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:11:48 smithi047 ceph-mon[77147]: pgmap v8871: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:11:48.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:11:48 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:11:49.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:11:48 smithi115 conmon[90393]: 2022-05-14T04:11:48.693+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:11:49.336 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:11:49 smithi047 conmon[78521]: 2022-05-14T04:11:49.039+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:11:49.336 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:11:49 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T04:11:49.336 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:11:49 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:11:49.337 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:11:49 smithi047 conmon[78521]: yield 2022-05-14T04:11:49.337 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:11:49 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:11:49.337 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:11:49 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:11:49.337 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:11:49 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:11:49.338 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:11:49 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T04:11:49.338 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:11:49 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:11:49.338 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:11:49 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T04:11:49.338 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:11:49 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:11:49.339 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:11:49 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T04:11:49.339 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:11:49 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:11:49.339 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:11:49 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T04:11:49.339 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:11:49 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:11:49.339 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:11:49 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T04:11:49.340 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:11:49 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:11:49.340 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:11:49 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:11:49.340 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:11:49 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:11:49.340 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:11:49 smithi047 conmon[78521]: 2022-05-14T04:11:49.040+0000 7f533ff78700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T04:11:49.341 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:11:49 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T04:11:49.341 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:11:49 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:11:49.342 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:11:49 smithi047 conmon[78521]: yield 2022-05-14T04:11:49.342 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:11:49 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:11:49.343 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:11:49 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:11:49.343 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:11:49 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:11:49.345 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:11:49 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T04:11:49.345 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:11:49 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:11:49.345 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:11:49 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T04:11:49.346 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:11:49 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:11:49.346 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:11:49 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T04:11:49.346 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:11:49 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:11:49.346 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:11:49 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T04:11:49.346 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:11:49 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:11:49.347 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:11:49 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T04:11:49.347 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:11:49 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:11:49.347 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:11:49 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:11:49.347 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:11:49 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:11:49.348 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:11:49 smithi047 conmon[78521]: 2022-05-14T04:11:49.348 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:11:49 smithi047 conmon[78521]: During handling of the above exception, another exception occurred: 2022-05-14T04:11:49.348 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:11:49 smithi047 conmon[78521]: 2022-05-14T04:11:49.349 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:11:49 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T04:11:49.349 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:11:49 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T04:11:49.349 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:11:49 smithi047 conmon[78521]: return f(*arg) 2022-05-14T04:11:49.349 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:11:49 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T04:11:49.349 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:11:49 smithi047 conmon[78521]: self._write_client_files(client_files, host) 2022-05-14T04:11:49.350 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:11:49 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T04:11:49.350 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:11:49 smithi047 conmon[78521]: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T04:11:49.350 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:11:49 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T04:11:49.350 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:11:49 smithi047 conmon[78521]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T04:11:49.351 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:11:49 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T04:11:49.351 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:11:49 smithi047 conmon[78521]: return self.event_loop.get_result(coro) 2022-05-14T04:11:49.351 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:11:49 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T04:11:49.351 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:11:49 smithi047 conmon[78521]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T04:11:49.351 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:11:49 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T04:11:49.352 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:11:49 smithi047 conmon[78521]: return self.__get_result() 2022-05-14T04:11:49.352 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:11:49 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T04:11:49.352 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:11:49 smithi047 conmon[78521]: raise self._exception 2022-05-14T04:11:49.352 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:11:49 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T04:11:49.352 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:11:49 smithi047 conmon[78521]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T04:11:49.353 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:11:49 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T04:11:49.353 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:11:49 smithi047 conmon[78521]: conn = await self._remote_connection(host, addr) 2022-05-14T04:11:49.353 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:11:49 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T04:11:49.354 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:11:49 smithi047 conmon[78521]: raise 2022-05-14T04:11:49.354 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:11:49 smithi047 conmon[78521]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T04:11:49.354 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:11:49 smithi047 conmon[78521]: self.gen.throw(type, value, traceback) 2022-05-14T04:11:49.355 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:11:49 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T04:11:49.355 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:11:49 smithi047 conmon[78521]: raise OrchestratorError(msg) 2022-05-14T04:11:49.355 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:11:49 smithi047 conmon[78521]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:11:49.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:11:49 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:11:49.798 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:11:49 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:11:50.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:11:49 smithi115 conmon[90393]: 2022-05-14T04:11:49.711+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:11:50.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:11:50 smithi115 ceph-mon[78593]: pgmap v8872: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:11:50.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:11:50 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:11:50.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T04:11:50.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:11:50.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T04:11:50.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:11:50.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:11:50.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:11:50.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T04:11:50.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:11:50.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T04:11:50.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:11:50.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T04:11:50.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:11:50.613 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T04:11:50.613 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:11:50.613 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T04:11:50.613 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:11:50.614 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:11:50.614 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:11:50.614 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:11:50 smithi115 ceph-mon[78593]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T04:11:50.615 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T04:11:50.615 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:11:50.615 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T04:11:50.616 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:11:50.616 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:11:50.617 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:11:50.617 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T04:11:50.617 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:11:50.618 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T04:11:50.618 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:11:50.618 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T04:11:50.619 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:11:50.619 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T04:11:50.619 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:11:50.619 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T04:11:50.620 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:11:50.620 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:11:50.620 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:11:50.620 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T04:11:50.621 INFO:journalctl@ceph.mon.c.smithi115.stdout: During handling of the above exception, another exception occurred: 2022-05-14T04:11:50.621 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T04:11:50.621 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T04:11:50.622 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T04:11:50.622 INFO:journalctl@ceph.mon.c.smithi115.stdout: return f(*arg) 2022-05-14T04:11:50.622 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T04:11:50.622 INFO:journalctl@ceph.mon.c.smithi115.stdout: self._write_client_files(client_files, host) 2022-05-14T04:11:50.623 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T04:11:50.623 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T04:11:50.623 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T04:11:50.623 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T04:11:50.623 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T04:11:50.624 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.event_loop.get_result(coro) 2022-05-14T04:11:50.624 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T04:11:50.624 INFO:journalctl@ceph.mon.c.smithi115.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T04:11:50.624 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T04:11:50.625 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.__get_result() 2022-05-14T04:11:50.625 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T04:11:50.625 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise self._exception 2022-05-14T04:11:50.625 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T04:11:50.626 INFO:journalctl@ceph.mon.c.smithi115.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T04:11:50.626 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T04:11:50.627 INFO:journalctl@ceph.mon.c.smithi115.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T04:11:50.627 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T04:11:50.628 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise 2022-05-14T04:11:50.628 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T04:11:50.628 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.gen.throw(type, value, traceback) 2022-05-14T04:11:50.628 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T04:11:50.629 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OrchestratorError(msg) 2022-05-14T04:11:50.629 INFO:journalctl@ceph.mon.c.smithi115.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:11:50.629 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:11:50 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:11:50.629 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:11:50 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:11:50.630 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:11:50 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:11:50.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:11:50 smithi047 ceph-mon[77147]: pgmap v8872: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:11:50.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:11:50 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:11:50.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T04:11:50.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:11:50.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T04:11:50.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:11:50.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:11:50.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:11:50.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T04:11:50.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:11:50.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T04:11:50.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:11:50.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T04:11:50.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:11:50.797 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T04:11:50.797 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:11:50.797 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T04:11:50.797 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:11:50.798 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:11:50.798 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:11:50.798 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:11:50 smithi047 ceph-mon[77147]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T04:11:50.799 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T04:11:50.799 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:11:50.799 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T04:11:50.799 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:11:50.799 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:11:50.800 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:11:50.800 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T04:11:50.800 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:11:50.800 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T04:11:50.800 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:11:50.801 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T04:11:50.801 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:11:50.801 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T04:11:50.801 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:11:50.802 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T04:11:50.802 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:11:50.802 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:11:50.802 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:11:50.803 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T04:11:50.803 INFO:journalctl@ceph.mon.b.smithi047.stdout: During handling of the above exception, another exception occurred: 2022-05-14T04:11:50.803 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T04:11:50.803 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T04:11:50.804 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T04:11:50.804 INFO:journalctl@ceph.mon.b.smithi047.stdout: return f(*arg) 2022-05-14T04:11:50.804 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T04:11:50.804 INFO:journalctl@ceph.mon.b.smithi047.stdout: self._write_client_files(client_files, host) 2022-05-14T04:11:50.804 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T04:11:50.805 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T04:11:50.805 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T04:11:50.805 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T04:11:50.805 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T04:11:50.806 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.event_loop.get_result(coro) 2022-05-14T04:11:50.806 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T04:11:50.806 INFO:journalctl@ceph.mon.b.smithi047.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T04:11:50.806 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T04:11:50.806 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.__get_result() 2022-05-14T04:11:50.807 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T04:11:50.807 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise self._exception 2022-05-14T04:11:50.807 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T04:11:50.807 INFO:journalctl@ceph.mon.b.smithi047.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T04:11:50.807 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T04:11:50.808 INFO:journalctl@ceph.mon.b.smithi047.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T04:11:50.808 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T04:11:50.808 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise 2022-05-14T04:11:50.809 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T04:11:50.809 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.gen.throw(type, value, traceback) 2022-05-14T04:11:50.809 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T04:11:50.810 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OrchestratorError(msg) 2022-05-14T04:11:50.810 INFO:journalctl@ceph.mon.b.smithi047.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:11:50.810 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:11:50 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:11:50.810 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:11:50 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:11:50.810 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:11:50 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:11:51.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:11:50 smithi115 conmon[90393]: 2022-05-14T04:11:50.684+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:11:51.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:11:51 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:11:51.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:11:51 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:11:52.110 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:11:51 smithi115 conmon[90393]: 2022-05-14T04:11:51.651+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:11:52.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:11:52 smithi115 ceph-mon[78593]: pgmap v8873: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:11:52.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:11:52 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:11:52.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:11:52 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 17045 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:11:52.795 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:11:52 smithi047 ceph-mon[77147]: pgmap v8873: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:11:52.795 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:11:52 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:11:52.796 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:11:52 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 17045 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:11:53.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:11:52 smithi115 conmon[90393]: 2022-05-14T04:11:52.690+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:11:53.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:11:53 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:11:53.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:11:53 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:11:54.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:11:53 smithi115 conmon[90393]: 2022-05-14T04:11:53.661+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:11:54.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:11:54 smithi115 ceph-mon[78593]: pgmap v8874: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:11:54.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:11:54 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:11:54.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:11:54 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:11:54.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:11:54 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:11:54.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:11:54 smithi047 ceph-mon[77147]: pgmap v8874: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:11:54.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:11:54 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:11:54.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:11:54 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:11:54.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:11:54 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:11:55.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:11:54 smithi115 conmon[90393]: 2022-05-14T04:11:54.642+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:11:55.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:11:55 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:11:55.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:11:55 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:11:56.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:11:55 smithi115 conmon[90393]: 2022-05-14T04:11:55.679+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:11:56.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:11:56 smithi115 ceph-mon[78593]: pgmap v8875: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:11:56.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:11:56 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:11:56.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:11:56 smithi047 ceph-mon[77147]: pgmap v8875: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:11:56.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:11:56 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:11:57.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:11:56 smithi115 conmon[90393]: 2022-05-14T04:11:56.651+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:11:57.624 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:11:57 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:11:57.625 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:11:57 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 17050 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:11:57.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:11:57 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:11:57.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:11:57 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 17050 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:11:58.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:11:57 smithi115 conmon[90393]: 2022-05-14T04:11:57.621+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:11:58.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:11:58 smithi047 ceph-mon[77147]: pgmap v8876: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:11:58.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:11:58 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:11:58.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:11:58 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:11:58.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:11:58 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:11:58.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:11:58 smithi115 conmon[90393]: 2022-05-14T04:11:58.606+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:11:58.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:11:58 smithi115 ceph-mon[78593]: pgmap v8876: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:11:58.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:11:58 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:11:58.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:11:58 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:11:58.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:11:58 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:11:59.628 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:11:59 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:11:59.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:11:59 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:12:00.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:11:59 smithi115 conmon[90393]: 2022-05-14T04:11:59.625+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:12:00.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:12:00 smithi047 ceph-mon[77147]: pgmap v8877: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:12:00.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:12:00 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:12:00.871 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:12:00 smithi115 conmon[90393]: 2022-05-14T04:12:00.592+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:12:00.871 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:12:00 smithi115 ceph-mon[78593]: pgmap v8877: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:12:00.872 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:12:00 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:12:01.640 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:12:01 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:12:01.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:12:01 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:12:02.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:12:01 smithi115 conmon[90393]: 2022-05-14T04:12:01.637+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:12:02.641 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:12:02 smithi115 ceph-mon[78593]: pgmap v8878: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:12:02.642 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:12:02 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:12:02.642 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:12:02 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:12:02.642 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:12:02 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:12:02.642 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:12:02 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 17055 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:12:02.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:12:02 smithi047 ceph-mon[77147]: pgmap v8878: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:12:02.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:12:02 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:12:02.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:12:02 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:12:02.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:12:02 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:12:02.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:12:02 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 17055 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:12:03.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:12:02 smithi115 conmon[90393]: 2022-05-14T04:12:02.639+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:12:03.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:12:03 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:12:03.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:12:03 smithi115 conmon[90393]: 2022-05-14T04:12:03.629+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:12:03.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:12:03 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:12:04.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:12:04 smithi047 ceph-mon[77147]: pgmap v8879: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:12:04.797 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:12:04 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:12:04.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:12:04 smithi115 conmon[90393]: 2022-05-14T04:12:04.601+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:12:04.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:12:04 smithi115 ceph-mon[78593]: pgmap v8879: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:12:04.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:12:04 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:12:05.646 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:12:05 smithi115 ceph-mon[78593]: pgmap v8880: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:12:05.647 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:12:05 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:12:05.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:12:05 smithi047 ceph-mon[77147]: pgmap v8880: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:12:05.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:12:05 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:12:06.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:12:05 smithi115 conmon[90393]: 2022-05-14T04:12:05.641+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:12:06.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:12:06 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:12:06.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:12:06 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:12:06.860 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:12:06 smithi115 conmon[90393]: 2022-05-14T04:12:06.625+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:12:07.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:12:07 smithi047 ceph-mon[77147]: pgmap v8881: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:12:07.795 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:12:07 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:12:07.795 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:12:07 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 17060 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:12:07.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:12:07 smithi115 conmon[90393]: 2022-05-14T04:12:07.591+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:12:07.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:12:07 smithi115 ceph-mon[78593]: pgmap v8881: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:12:07.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:12:07 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:12:07.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:12:07 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 17060 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:12:08.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:12:08 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:12:08.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:12:08 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:12:08.860 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:12:08 smithi115 conmon[90393]: 2022-05-14T04:12:08.599+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:12:09.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:12:09 smithi047 ceph-mon[77147]: pgmap v8882: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:12:09.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:12:09 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:12:09.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:12:09 smithi115 conmon[90393]: 2022-05-14T04:12:09.606+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:12:09.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:12:09 smithi115 ceph-mon[78593]: pgmap v8882: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:12:09.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:12:09 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:12:10.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:12:10 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:12:10.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:12:10 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:12:10.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:12:10 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:12:10.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:12:10 smithi115 conmon[90393]: 2022-05-14T04:12:10.623+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:12:10.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:12:10 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:12:10.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:12:10 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:12:10.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:12:10 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:12:11.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:12:11 smithi047 ceph-mon[77147]: pgmap v8883: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:12:11.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:12:11 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:12:11.860 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:12:11 smithi115 conmon[90393]: 2022-05-14T04:12:11.598+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:12:11.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:12:11 smithi115 ceph-mon[78593]: pgmap v8883: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:12:11.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:12:11 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:12:12.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:12:12 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:12:12.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:12:12 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 17065 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:12:12.860 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:12:12 smithi115 conmon[90393]: 2022-05-14T04:12:12.592+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:12:12.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:12:12 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:12:12.862 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:12:12 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 17065 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:12:13.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:12:13 smithi047 ceph-mon[77147]: pgmap v8884: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:12:13.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:12:13 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:12:13.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:12:13 smithi115 conmon[90393]: 2022-05-14T04:12:13.599+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:12:13.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:12:13 smithi115 ceph-mon[78593]: pgmap v8884: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:12:13.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:12:13 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:12:14.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:12:14 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:12:14.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:12:14 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:12:14.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:12:14 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:12:14.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:12:14 smithi115 conmon[90393]: 2022-05-14T04:12:14.620+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:12:14.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:12:14 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:12:14.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:12:14 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:12:14.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:12:14 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:12:15.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:12:15 smithi047 ceph-mon[77147]: pgmap v8885: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:12:15.795 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:12:15 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T04:12:15.795 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:12:15 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T04:12:15.796 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:12:15 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:12:15.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:12:15 smithi115 conmon[90393]: 2022-05-14T04:12:15.597+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:12:15.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:12:15 smithi115 ceph-mon[78593]: pgmap v8885: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:12:15.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:12:15 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T04:12:15.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:12:15 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T04:12:15.862 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:12:15 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:12:16.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:12:16 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:12:16.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:12:16 smithi115 conmon[90393]: 2022-05-14T04:12:16.611+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:12:16.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:12:16 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:12:17.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:12:17 smithi047 ceph-mon[77147]: pgmap v8886: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:12:17.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:12:17 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:12:17.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:12:17 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 17070 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:12:17.861 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:12:17 smithi115 conmon[90393]: 2022-05-14T04:12:17.594+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:12:17.862 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:12:17 smithi115 ceph-mon[78593]: pgmap v8886: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:12:17.863 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:12:17 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:12:17.863 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:12:17 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 17070 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:12:18.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:12:18 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:12:18.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:12:18 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:12:18.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:12:18 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:12:18.860 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:12:18 smithi115 conmon[90393]: 2022-05-14T04:12:18.567+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:12:18.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:12:18 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:12:18.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:12:18 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:12:18.863 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:12:18 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:12:19.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:12:19 smithi047 ceph-mon[77147]: pgmap v8887: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:12:19.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:12:19 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:12:19.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:12:19 smithi115 conmon[90393]: 2022-05-14T04:12:19.538+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:12:19.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:12:19 smithi115 ceph-mon[78593]: pgmap v8887: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:12:19.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:12:19 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:12:20.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:12:20 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:12:20.860 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:12:20 smithi115 conmon[90393]: 2022-05-14T04:12:20.574+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:12:20.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:12:20 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:12:21.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:12:21 smithi047 ceph-mon[77147]: pgmap v8888: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:12:21.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:12:21 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:12:21.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:12:21 smithi115 conmon[90393]: 2022-05-14T04:12:21.569+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:12:21.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:12:21 smithi115 ceph-mon[78593]: pgmap v8888: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:12:21.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:12:21 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:12:22.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:12:22 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:12:22.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:12:22 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:12:22.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:12:22 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:12:22.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:12:22 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 17075 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:12:22.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:12:22 smithi115 conmon[90393]: 2022-05-14T04:12:22.595+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:12:22.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:12:22 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:12:22.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:12:22 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:12:22.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:12:22 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:12:22.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:12:22 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 17075 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:12:23.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:12:23 smithi047 ceph-mon[77147]: pgmap v8889: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:12:23.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:12:23 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:12:23.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:12:23 smithi115 conmon[90393]: 2022-05-14T04:12:23.560+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:12:23.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:12:23 smithi115 ceph-mon[78593]: pgmap v8889: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:12:23.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:12:23 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:12:24.795 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:12:24 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:12:24.860 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:12:24 smithi115 conmon[90393]: 2022-05-14T04:12:24.573+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:12:24.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:12:24 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:12:25.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:12:25 smithi047 ceph-mon[77147]: pgmap v8890: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:12:25.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:12:25 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:12:25.866 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:12:25 smithi115 conmon[90393]: 2022-05-14T04:12:25.550+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:12:25.867 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:12:25 smithi115 ceph-mon[78593]: pgmap v8890: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:12:25.867 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:12:25 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:12:26.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:12:26 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:12:26.864 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:12:26 smithi115 conmon[90393]: 2022-05-14T04:12:26.534+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:12:26.865 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:12:26 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:12:27.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:12:27 smithi047 ceph-mon[77147]: pgmap v8891: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:12:27.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:12:27 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:12:27.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:12:27 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 17080 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:12:27.860 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:12:27 smithi115 conmon[90393]: 2022-05-14T04:12:27.488+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:12:27.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:12:27 smithi115 ceph-mon[78593]: pgmap v8891: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:12:27.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:12:27 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:12:27.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:12:27 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 17080 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:12:28.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:12:28 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:12:28.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:12:28 smithi115 conmon[90393]: 2022-05-14T04:12:28.535+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:12:28.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:12:28 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:12:29.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:12:29 smithi047 ceph-mon[77147]: pgmap v8892: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:12:29.795 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:12:29 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:12:29.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:12:29 smithi115 conmon[90393]: 2022-05-14T04:12:29.516+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:12:29.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:12:29 smithi115 ceph-mon[78593]: pgmap v8892: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:12:29.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:12:29 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:12:30.795 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:12:30 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:12:30.795 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:12:30 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:12:30.796 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:12:30 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:12:30.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:12:30 smithi115 conmon[90393]: 2022-05-14T04:12:30.504+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:12:30.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:12:30 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:12:30.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:12:30 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:12:30.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:12:30 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:12:31.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:12:31 smithi047 ceph-mon[77147]: pgmap v8893: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:12:31.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:12:31 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:12:31.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:12:31 smithi115 conmon[90393]: 2022-05-14T04:12:31.500+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:12:31.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:12:31 smithi115 ceph-mon[78593]: pgmap v8893: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:12:31.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:12:31 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:12:32.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:12:32 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:12:32.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:12:32 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 17085 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:12:32.860 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:12:32 smithi115 conmon[90393]: 2022-05-14T04:12:32.490+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:12:32.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:12:32 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:12:32.862 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:12:32 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 17085 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:12:33.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:12:33 smithi115 conmon[90393]: 2022-05-14T04:12:33.442+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:12:34.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:12:34 smithi047 ceph-mon[77147]: pgmap v8894: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:12:34.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:12:34 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:12:34.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:12:34 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:12:34.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:12:34 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:12:34.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:12:34 smithi115 conmon[90393]: 2022-05-14T04:12:34.441+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:12:34.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:12:34 smithi115 ceph-mon[78593]: pgmap v8894: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:12:34.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:12:34 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:12:34.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:12:34 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:12:34.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:12:34 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:12:35.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:12:35 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:12:35.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:12:35 smithi115 conmon[90393]: 2022-05-14T04:12:35.418+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:12:35.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:12:35 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:12:36.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:12:36 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:12:36.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:12:36 smithi047 ceph-mon[77147]: pgmap v8895: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:12:36.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:12:36 smithi115 conmon[90393]: 2022-05-14T04:12:36.402+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:12:36.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:12:36 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:12:36.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:12:36 smithi115 ceph-mon[78593]: pgmap v8895: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:12:37.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:12:37 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:12:37.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:12:37 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 17090 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:12:37.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:12:37 smithi115 conmon[90393]: 2022-05-14T04:12:37.425+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:12:37.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:12:37 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:12:37.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:12:37 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 17090 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:12:38.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:12:38 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:12:38.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:12:38 smithi047 ceph-mon[77147]: pgmap v8896: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:12:38.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:12:38 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:12:38.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:12:38 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:12:38.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:12:38 smithi115 conmon[90393]: 2022-05-14T04:12:38.419+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:12:38.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:12:38 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:12:38.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:12:38 smithi115 ceph-mon[78593]: pgmap v8896: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:12:38.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:12:38 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:12:38.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:12:38 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:12:39.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:12:39 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:12:39.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:12:39 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:12:39.610 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:12:39 smithi115 conmon[90393]: 2022-05-14T04:12:39.408+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:12:40.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:12:40 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:12:40.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:12:40 smithi047 ceph-mon[77147]: pgmap v8897: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:12:40.610 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:12:40 smithi115 conmon[90393]: 2022-05-14T04:12:40.374+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:12:40.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:12:40 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:12:40.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:12:40 smithi115 ceph-mon[78593]: pgmap v8897: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:12:41.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:12:41 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:12:41.613 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:12:41 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:12:41.613 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:12:41 smithi115 conmon[90393]: 2022-05-14T04:12:41.371+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:12:42.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:12:42 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:12:42.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:12:42 smithi047 ceph-mon[77147]: pgmap v8898: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:12:42.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:12:42 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:12:42.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:12:42 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:12:42.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:12:42 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 17095 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:12:42.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:12:42 smithi115 conmon[90393]: 2022-05-14T04:12:42.416+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:12:42.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:12:42 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:12:42.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:12:42 smithi115 ceph-mon[78593]: pgmap v8898: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:12:42.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:12:42 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:12:42.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:12:42 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:12:42.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:12:42 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 17095 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:12:43.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:12:43 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:12:43.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:12:43 smithi115 conmon[90393]: 2022-05-14T04:12:43.432+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:12:43.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:12:43 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:12:44.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:12:44 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:12:44.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:12:44 smithi047 ceph-mon[77147]: pgmap v8899: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:12:44.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:12:44 smithi115 conmon[90393]: 2022-05-14T04:12:44.399+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:12:44.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:12:44 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:12:44.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:12:44 smithi115 ceph-mon[78593]: pgmap v8899: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:12:45.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:12:45 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:12:45.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:12:45 smithi115 conmon[90393]: 2022-05-14T04:12:45.392+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:12:45.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:12:45 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:12:46.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:12:46 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:12:46.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:12:46 smithi047 ceph-mon[77147]: pgmap v8900: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:12:46.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:12:46 smithi115 conmon[90393]: 2022-05-14T04:12:46.418+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:12:46.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:12:46 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:12:46.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:12:46 smithi115 ceph-mon[78593]: pgmap v8900: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:12:47.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:12:47 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:12:47.545 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:12:47 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 17100 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:12:47.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:12:47 smithi115 conmon[90393]: 2022-05-14T04:12:47.453+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:12:47.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:12:47 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:12:47.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:12:47 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 17100 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:12:48.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:12:48 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:12:48.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:12:48 smithi047 ceph-mon[77147]: pgmap v8901: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:12:48.610 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:12:48 smithi115 conmon[90393]: 2022-05-14T04:12:48.437+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:12:48.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:12:48 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:12:48.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:12:48 smithi115 ceph-mon[78593]: pgmap v8901: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:12:49.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:12:49 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:12:49.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:12:49 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T04:12:49.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:12:49 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T04:12:49.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:12:49 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T04:12:49.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:12:49 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:12:49.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:12:49 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T04:12:49.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:12:49 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T04:12:49.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:12:49 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T04:12:49.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:12:49 smithi115 conmon[90393]: 2022-05-14T04:12:49.476+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:12:50.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:12:50 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:12:50.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:12:50 smithi047 ceph-mon[77147]: pgmap v8902: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:12:50.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:12:50 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:12:50.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:12:50 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:12:50.611 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:12:50 smithi115 conmon[90393]: 2022-05-14T04:12:50.485+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:12:50.612 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:12:50 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:12:50.612 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:12:50 smithi115 ceph-mon[78593]: pgmap v8902: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:12:50.612 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:12:50 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:12:50.613 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:12:50 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:12:51.531 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:12:51 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:12:51.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:12:51 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:12:51.860 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:12:51 smithi115 conmon[90393]: 2022-05-14T04:12:51.529+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:12:52.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:12:52 smithi047 ceph-mon[77147]: pgmap v8903: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:12:52.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:12:52 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:12:52.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:12:52 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 17105 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:12:52.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:12:52 smithi047 conmon[78521]: 2022-05-14T04:12:52.144+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:12:52.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:12:52 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T04:12:52.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:12:52 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:12:52.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:12:52 smithi047 conmon[78521]: yield 2022-05-14T04:12:52.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:12:52 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:12:52.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:12:52 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:12:52.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:12:52 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:12:52.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:12:52 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T04:12:52.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:12:52 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:12:52.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:12:52 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T04:12:52.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:12:52 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:12:52.547 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:12:52 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T04:12:52.547 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:12:52 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:12:52.547 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:12:52 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T04:12:52.547 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:12:52 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:12:52.547 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:12:52 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T04:12:52.548 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:12:52 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:12:52.548 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:12:52 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:12:52.548 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:12:52 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:12:52.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:12:52 smithi115 conmon[90393]: 2022-05-14T04:12:52.514+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:12:52.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:12:52 smithi115 ceph-mon[78593]: pgmap v8903: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:12:52.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:12:52 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:12:52.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:12:52 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 17105 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:12:53.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:12:53 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:12:53.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:12:53 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:12:53.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T04:12:53.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:12:53.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T04:12:53.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:12:53.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:12:53.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:12:53.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T04:12:53.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:12:53.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T04:12:53.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:12:53.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T04:12:53.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:12:53.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T04:12:53.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:12:53.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T04:12:53.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:12:53.548 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:12:53.548 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:12:53.548 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:12:53 smithi047 ceph-mon[77147]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T04:12:53.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:12:53 smithi115 conmon[90393]: 2022-05-14T04:12:53.504+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:12:53.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:12:53 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:12:53.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:12:53 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:12:53.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T04:12:53.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:12:53.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T04:12:53.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:12:53.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:12:53.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:12:53.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T04:12:53.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:12:53.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T04:12:53.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:12:53.613 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T04:12:53.613 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:12:53.613 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T04:12:53.613 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:12:53.613 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T04:12:53.614 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:12:53.614 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:12:53.614 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:12:53.614 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:12:53 smithi115 ceph-mon[78593]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T04:12:54.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:12:54 smithi047 ceph-mon[77147]: pgmap v8904: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:12:54.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:12:54 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:12:54.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:12:54 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:12:54.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:12:54 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:12:54.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:12:54 smithi115 conmon[90393]: 2022-05-14T04:12:54.468+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:12:54.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:12:54 smithi115 ceph-mon[78593]: pgmap v8904: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:12:54.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:12:54 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:12:54.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:12:54 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:12:54.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:12:54 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:12:55.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:12:55 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:12:55.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:12:55 smithi047 conmon[78521]: 2022-05-14T04:12:55.216+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:12:55.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:12:55 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T04:12:55.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:12:55 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:12:55.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:12:55 smithi047 conmon[78521]: yield 2022-05-14T04:12:55.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:12:55 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:12:55.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:12:55 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:12:55.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:12:55 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:12:55.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:12:55 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T04:12:55.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:12:55 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:12:55.547 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:12:55 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T04:12:55.547 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:12:55 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:12:55.548 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:12:55 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T04:12:55.548 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:12:55 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:12:55.548 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:12:55 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T04:12:55.549 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:12:55 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:12:55.549 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:12:55 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T04:12:55.549 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:12:55 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:12:55.549 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:12:55 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:12:55.550 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:12:55 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:12:55.550 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:12:55 smithi047 conmon[78521]: 2022-05-14T04:12:55.217+0000 7f5344f82700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T04:12:55.550 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:12:55 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T04:12:55.550 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:12:55 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:12:55.550 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:12:55 smithi047 conmon[78521]: yield 2022-05-14T04:12:55.551 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:12:55 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:12:55.551 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:12:55 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:12:55.551 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:12:55 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:12:55.551 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:12:55 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T04:12:55.552 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:12:55 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:12:55.552 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:12:55 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T04:12:55.552 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:12:55 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:12:55.552 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:12:55 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T04:12:55.553 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:12:55 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:12:55.553 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:12:55 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T04:12:55.554 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:12:55 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:12:55.554 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:12:55 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T04:12:55.555 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:12:55 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:12:55.555 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:12:55 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:12:55.556 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:12:55 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:12:55.556 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:12:55 smithi047 conmon[78521]: 2022-05-14T04:12:55.556 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:12:55 smithi047 conmon[78521]: During handling of the above exception, another exception occurred: 2022-05-14T04:12:55.556 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:12:55 smithi047 conmon[78521]: 2022-05-14T04:12:55.557 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:12:55 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T04:12:55.557 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:12:55 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T04:12:55.558 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:12:55 smithi047 conmon[78521]: return f(*arg) 2022-05-14T04:12:55.558 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:12:55 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T04:12:55.558 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:12:55 smithi047 conmon[78521]: self._write_client_files(client_files, host) 2022-05-14T04:12:55.559 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:12:55 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T04:12:55.559 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:12:55 smithi047 conmon[78521]: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T04:12:55.559 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:12:55 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T04:12:55.559 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:12:55 smithi047 conmon[78521]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T04:12:55.560 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:12:55 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T04:12:55.560 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:12:55 smithi047 conmon[78521]: return self.event_loop.get_result(coro) 2022-05-14T04:12:55.560 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:12:55 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T04:12:55.560 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:12:55 smithi047 conmon[78521]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T04:12:55.560 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:12:55 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T04:12:55.561 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:12:55 smithi047 conmon[78521]: return self.__get_result() 2022-05-14T04:12:55.561 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:12:55 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T04:12:55.561 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:12:55 smithi047 conmon[78521]: raise self._exception 2022-05-14T04:12:55.561 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:12:55 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T04:12:55.562 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:12:55 smithi047 conmon[78521]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T04:12:55.562 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:12:55 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T04:12:55.562 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:12:55 smithi047 conmon[78521]: conn = await self._remote_connection(host, addr) 2022-05-14T04:12:55.562 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:12:55 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T04:12:55.562 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:12:55 smithi047 conmon[78521]: raise 2022-05-14T04:12:55.563 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:12:55 smithi047 conmon[78521]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T04:12:55.563 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:12:55 smithi047 conmon[78521]: self.gen.throw(type, value, traceback) 2022-05-14T04:12:55.566 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:12:55 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T04:12:55.566 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:12:55 smithi047 conmon[78521]: raise OrchestratorError(msg) 2022-05-14T04:12:55.566 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:12:55 smithi047 conmon[78521]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:12:55.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:12:55 smithi115 conmon[90393]: 2022-05-14T04:12:55.473+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:12:55.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:12:55 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:12:56.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:12:56 smithi047 ceph-mon[77147]: pgmap v8905: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:12:56.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:12:56 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:12:56.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:12:56 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:12:56.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T04:12:56.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:12:56.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T04:12:56.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:12:56.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:12:56.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:12:56.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T04:12:56.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:12:56.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T04:12:56.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:12:56.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T04:12:56.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:12:56.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T04:12:56.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:12:56.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T04:12:56.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:12:56.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:12:56.548 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:12:56.549 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:12:56 smithi047 ceph-mon[77147]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T04:12:56.549 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T04:12:56.550 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:12:56.550 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T04:12:56.550 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:12:56.550 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:12:56.551 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:12:56.551 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T04:12:56.551 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:12:56.551 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T04:12:56.551 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:12:56.552 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T04:12:56.552 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:12:56.552 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T04:12:56.552 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:12:56.552 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T04:12:56.553 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:12:56.553 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:12:56.553 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:12:56.553 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T04:12:56.554 INFO:journalctl@ceph.mon.b.smithi047.stdout: During handling of the above exception, another exception occurred: 2022-05-14T04:12:56.554 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T04:12:56.554 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T04:12:56.554 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T04:12:56.554 INFO:journalctl@ceph.mon.b.smithi047.stdout: return f(*arg) 2022-05-14T04:12:56.555 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T04:12:56.555 INFO:journalctl@ceph.mon.b.smithi047.stdout: self._write_client_files(client_files, host) 2022-05-14T04:12:56.555 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T04:12:56.556 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T04:12:56.556 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T04:12:56.557 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T04:12:56.557 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T04:12:56.558 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.event_loop.get_result(coro) 2022-05-14T04:12:56.558 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T04:12:56.559 INFO:journalctl@ceph.mon.b.smithi047.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T04:12:56.559 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T04:12:56.559 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.__get_result() 2022-05-14T04:12:56.561 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T04:12:56.562 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise self._exception 2022-05-14T04:12:56.562 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T04:12:56.562 INFO:journalctl@ceph.mon.b.smithi047.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T04:12:56.563 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T04:12:56.563 INFO:journalctl@ceph.mon.b.smithi047.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T04:12:56.563 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T04:12:56.563 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise 2022-05-14T04:12:56.564 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T04:12:56.564 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.gen.throw(type, value, traceback) 2022-05-14T04:12:56.564 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T04:12:56.564 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OrchestratorError(msg) 2022-05-14T04:12:56.564 INFO:journalctl@ceph.mon.b.smithi047.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:12:56.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:12:56 smithi115 conmon[90393]: 2022-05-14T04:12:56.465+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:12:56.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:12:56 smithi115 ceph-mon[78593]: pgmap v8905: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:12:56.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:12:56 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:12:56.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:12:56 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:12:56.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T04:12:56.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:12:56.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T04:12:56.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:12:56.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:12:56.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:12:56.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T04:12:56.613 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:12:56.613 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T04:12:56.614 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:12:56.614 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T04:12:56.614 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:12:56.614 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T04:12:56.614 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:12:56.615 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T04:12:56.615 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:12:56.615 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:12:56.615 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:12:56.616 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:12:56 smithi115 ceph-mon[78593]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T04:12:56.616 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T04:12:56.616 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:12:56.616 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T04:12:56.617 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:12:56.617 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:12:56.617 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:12:56.617 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T04:12:56.618 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:12:56.618 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T04:12:56.618 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:12:56.618 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T04:12:56.618 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:12:56.619 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T04:12:56.619 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:12:56.619 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T04:12:56.619 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:12:56.620 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:12:56.620 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:12:56.620 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T04:12:56.620 INFO:journalctl@ceph.mon.c.smithi115.stdout: During handling of the above exception, another exception occurred: 2022-05-14T04:12:56.621 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T04:12:56.623 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T04:12:56.624 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T04:12:56.624 INFO:journalctl@ceph.mon.c.smithi115.stdout: return f(*arg) 2022-05-14T04:12:56.624 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T04:12:56.624 INFO:journalctl@ceph.mon.c.smithi115.stdout: self._write_client_files(client_files, host) 2022-05-14T04:12:56.625 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T04:12:56.625 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T04:12:56.625 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T04:12:56.625 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T04:12:56.626 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T04:12:56.626 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.event_loop.get_result(coro) 2022-05-14T04:12:56.626 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T04:12:56.626 INFO:journalctl@ceph.mon.c.smithi115.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T04:12:56.627 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T04:12:56.627 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.__get_result() 2022-05-14T04:12:56.627 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T04:12:56.627 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise self._exception 2022-05-14T04:12:56.628 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T04:12:56.628 INFO:journalctl@ceph.mon.c.smithi115.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T04:12:56.628 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T04:12:56.628 INFO:journalctl@ceph.mon.c.smithi115.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T04:12:56.629 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T04:12:56.629 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise 2022-05-14T04:12:56.629 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T04:12:56.629 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.gen.throw(type, value, traceback) 2022-05-14T04:12:56.630 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T04:12:56.630 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OrchestratorError(msg) 2022-05-14T04:12:56.630 INFO:journalctl@ceph.mon.c.smithi115.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:12:57.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:12:57 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:12:57.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:12:57 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 17110 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:12:57.612 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:12:57 smithi115 conmon[90393]: 2022-05-14T04:12:57.472+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:12:57.613 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:12:57 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:12:57.613 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:12:57 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 17110 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:12:58.545 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:12:58 smithi047 ceph-mon[77147]: pgmap v8906: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:12:58.836 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:12:58 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:12:58.837 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:12:58 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:12:58.837 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:12:58 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:12:58.839 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:12:58 smithi115 conmon[90393]: 2022-05-14T04:12:58.428+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:12:58.840 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:12:58 smithi115 ceph-mon[78593]: pgmap v8906: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:12:58.840 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:12:58 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:12:58.840 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:12:58 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:12:58.840 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:12:58 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:12:59.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:12:59 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:12:59.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:12:59 smithi115 conmon[90393]: 2022-05-14T04:12:59.410+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:12:59.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:12:59 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:13:00.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:13:00 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:13:00.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:13:00 smithi047 ceph-mon[77147]: pgmap v8907: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:13:00.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:13:00 smithi115 conmon[90393]: 2022-05-14T04:13:00.412+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:13:00.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:13:00 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:13:00.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:13:00 smithi115 ceph-mon[78593]: pgmap v8907: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:13:01.610 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:13:01 smithi115 conmon[90393]: 2022-05-14T04:13:01.418+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:13:01.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:13:01 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:13:01.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:13:01 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:13:02.610 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:13:02 smithi115 conmon[90393]: 2022-05-14T04:13:02.415+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:13:02.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:13:02 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:13:02.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:13:02 smithi115 ceph-mon[78593]: pgmap v8908: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:13:02.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:13:02 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:13:02.612 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:13:02 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:13:02.612 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:13:02 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 17115 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:13:02.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:13:02 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:13:02.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:13:02 smithi047 ceph-mon[77147]: pgmap v8908: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:13:02.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:13:02 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:13:02.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:13:02 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:13:02.795 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:13:02 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 17115 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:13:03.610 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:13:03 smithi115 conmon[90393]: 2022-05-14T04:13:03.431+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:13:03.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:13:03 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:13:03.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:13:03 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:13:04.610 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:13:04 smithi115 conmon[90393]: 2022-05-14T04:13:04.451+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:13:04.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:13:04 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:13:04.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:13:04 smithi115 ceph-mon[78593]: pgmap v8909: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:13:04.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:13:04 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:13:04.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:13:04 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:13:04.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:13:04 smithi047 ceph-mon[77147]: pgmap v8909: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:13:04.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:13:04 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:13:05.610 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:13:05 smithi115 conmon[90393]: 2022-05-14T04:13:05.432+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:13:05.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:13:05 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:13:05.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:13:05 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:13:06.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:13:06 smithi115 conmon[90393]: 2022-05-14T04:13:06.428+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:13:06.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:13:06 smithi115 ceph-mon[78593]: pgmap v8910: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:13:06.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:13:06 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:13:06.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:13:06 smithi047 ceph-mon[77147]: pgmap v8910: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:13:06.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:13:06 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:13:07.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:13:07 smithi115 conmon[90393]: 2022-05-14T04:13:07.403+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:13:07.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:13:07 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:13:07.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:13:07 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 17120 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:13:07.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:13:07 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:13:07.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:13:07 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 17120 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:13:08.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:13:08 smithi115 conmon[90393]: 2022-05-14T04:13:08.360+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:13:08.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:13:08 smithi115 ceph-mon[78593]: pgmap v8911: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:13:08.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:13:08 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:13:08.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:13:08 smithi047 ceph-mon[77147]: pgmap v8911: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:13:08.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:13:08 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:13:09.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:13:09 smithi115 conmon[90393]: 2022-05-14T04:13:09.317+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:13:09.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:13:09 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:13:09.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:13:09 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:13:10.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:13:10 smithi115 conmon[90393]: 2022-05-14T04:13:10.340+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:13:10.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:13:10 smithi115 ceph-mon[78593]: pgmap v8912: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:13:10.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:13:10 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:13:10.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:13:10 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:13:10.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:13:10 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:13:10.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:13:10 smithi047 ceph-mon[77147]: pgmap v8912: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:13:10.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:13:10 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:13:10.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:13:10 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:13:10.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:13:10 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:13:11.610 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:13:11 smithi115 conmon[90393]: 2022-05-14T04:13:11.349+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:13:11.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:13:11 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:13:11.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:13:11 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:13:12.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:13:12 smithi115 conmon[90393]: 2022-05-14T04:13:12.381+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:13:12.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:13:12 smithi115 ceph-mon[78593]: pgmap v8913: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:13:12.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:13:12 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:13:12.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:13:12 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 17125 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:13:12.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:13:12 smithi047 ceph-mon[77147]: pgmap v8913: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:13:12.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:13:12 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:13:12.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:13:12 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 17125 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:13:13.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:13:13 smithi115 conmon[90393]: 2022-05-14T04:13:13.404+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:13:13.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:13:13 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:13:13.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:13:13 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:13:14.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:13:14 smithi115 conmon[90393]: 2022-05-14T04:13:14.426+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:13:14.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:13:14 smithi115 ceph-mon[78593]: pgmap v8914: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:13:14.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:13:14 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:13:14.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:13:14 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:13:14.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:13:14 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:13:14.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:13:14 smithi047 ceph-mon[77147]: pgmap v8914: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:13:14.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:13:14 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:13:14.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:13:14 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:13:14.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:13:14 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:13:15.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:13:15 smithi115 conmon[90393]: 2022-05-14T04:13:15.462+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:13:15.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:13:15 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:13:15.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:13:15 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T04:13:15.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:13:15 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T04:13:15.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:13:15 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:13:15.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:13:15 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T04:13:15.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:13:15 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T04:13:16.610 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:13:16 smithi115 conmon[90393]: 2022-05-14T04:13:16.415+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:13:16.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:13:16 smithi115 ceph-mon[78593]: pgmap v8915: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:13:16.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:13:16 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:13:16.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:13:16 smithi047 ceph-mon[77147]: pgmap v8915: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:13:16.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:13:16 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:13:17.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:13:17 smithi115 conmon[90393]: 2022-05-14T04:13:17.401+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:13:17.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:13:17 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:13:17.612 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:13:17 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 17130 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:13:17.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:13:17 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:13:17.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:13:17 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 17130 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:13:18.610 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:13:18 smithi115 conmon[90393]: 2022-05-14T04:13:18.395+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:13:18.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:13:18 smithi115 ceph-mon[78593]: pgmap v8916: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:13:18.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:13:18 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:13:18.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:13:18 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:13:18.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:13:18 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:13:18.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:13:18 smithi047 ceph-mon[77147]: pgmap v8916: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:13:18.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:13:18 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:13:18.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:13:18 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:13:18.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:13:18 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:13:19.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:13:19 smithi115 conmon[90393]: 2022-05-14T04:13:19.412+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:13:19.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:13:19 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:13:19.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:13:19 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:13:20.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:13:20 smithi115 conmon[90393]: 2022-05-14T04:13:20.423+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:13:20.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:13:20 smithi115 ceph-mon[78593]: pgmap v8917: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:13:20.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:13:20 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:13:20.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:13:20 smithi047 ceph-mon[77147]: pgmap v8917: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:13:20.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:13:20 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:13:21.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:13:21 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:13:21.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:13:21 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:13:21.860 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:13:21 smithi115 conmon[90393]: 2022-05-14T04:13:21.453+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:13:22.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:13:22 smithi047 ceph-mon[77147]: pgmap v8918: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:13:22.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:13:22 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:13:22.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:13:22 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:13:22.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:13:22 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:13:22.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:13:22 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 17135 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:13:22.860 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:13:22 smithi115 conmon[90393]: 2022-05-14T04:13:22.443+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:13:22.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:13:22 smithi115 ceph-mon[78593]: pgmap v8918: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:13:22.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:13:22 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:13:22.862 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:13:22 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:13:22.862 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:13:22 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:13:22.863 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:13:22 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 17135 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:13:23.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:13:23 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:13:23.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:13:23 smithi115 conmon[90393]: 2022-05-14T04:13:23.475+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:13:23.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:13:23 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:13:24.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:13:24 smithi047 ceph-mon[77147]: pgmap v8919: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:13:24.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:13:24 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:13:24.860 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:13:24 smithi115 conmon[90393]: 2022-05-14T04:13:24.428+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:13:24.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:13:24 smithi115 ceph-mon[78593]: pgmap v8919: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:13:24.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:13:24 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:13:25.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:13:25 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:13:25.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:13:25 smithi115 conmon[90393]: 2022-05-14T04:13:25.413+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:13:25.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:13:25 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:13:26.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:13:26 smithi047 ceph-mon[77147]: pgmap v8920: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:13:26.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:13:26 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:13:26.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:13:26 smithi115 conmon[90393]: 2022-05-14T04:13:26.455+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:13:26.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:13:26 smithi115 ceph-mon[78593]: pgmap v8920: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:13:26.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:13:26 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:13:27.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:13:27 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:13:27.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:13:27 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 17140 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:13:27.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:13:27 smithi115 conmon[90393]: 2022-05-14T04:13:27.469+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:13:27.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:13:27 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:13:27.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:13:27 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 17140 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:13:28.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:13:28 smithi047 ceph-mon[77147]: pgmap v8921: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:13:28.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:13:28 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:13:28.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:13:28 smithi115 conmon[90393]: 2022-05-14T04:13:28.481+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:13:28.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:13:28 smithi115 ceph-mon[78593]: pgmap v8921: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:13:28.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:13:28 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:13:29.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:13:29 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:13:29.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:13:29 smithi115 conmon[90393]: 2022-05-14T04:13:29.478+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:13:29.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:13:29 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:13:30.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:13:30 smithi047 ceph-mon[77147]: pgmap v8922: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:13:30.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:13:30 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:13:30.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:13:30 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:13:30.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:13:30 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:13:30.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:13:30 smithi115 conmon[90393]: 2022-05-14T04:13:30.488+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:13:30.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:13:30 smithi115 ceph-mon[78593]: pgmap v8922: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:13:30.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:13:30 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:13:30.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:13:30 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:13:30.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:13:30 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:13:31.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:13:31 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:13:31.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:13:31 smithi115 conmon[90393]: 2022-05-14T04:13:31.457+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:13:31.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:13:31 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:13:32.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:13:32 smithi047 ceph-mon[77147]: pgmap v8923: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:13:32.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:13:32 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:13:32.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:13:32 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 17145 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:13:32.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:13:32 smithi115 conmon[90393]: 2022-05-14T04:13:32.491+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:13:32.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:13:32 smithi115 ceph-mon[78593]: pgmap v8923: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:13:32.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:13:32 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:13:32.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:13:32 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 17145 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:13:33.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:13:33 smithi047 ceph-mon[77147]: pgmap v8924: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:13:33.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:13:33 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:13:33.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:13:33 smithi115 conmon[90393]: 2022-05-14T04:13:33.462+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:13:33.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:13:33 smithi115 ceph-mon[78593]: pgmap v8924: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:13:33.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:13:33 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:13:34.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:13:34 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:13:34.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:13:34 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:13:34.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:13:34 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:13:34.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:13:34 smithi115 conmon[90393]: 2022-05-14T04:13:34.445+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:13:34.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:13:34 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:13:34.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:13:34 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:13:34.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:13:34 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:13:35.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:13:35 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:13:35.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:13:35 smithi047 ceph-mon[77147]: pgmap v8925: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:13:35.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:13:35 smithi115 conmon[90393]: 2022-05-14T04:13:35.442+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:13:35.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:13:35 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:13:35.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:13:35 smithi115 ceph-mon[78593]: pgmap v8925: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:13:36.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:13:36 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:13:36.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:13:36 smithi115 conmon[90393]: 2022-05-14T04:13:36.429+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:13:36.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:13:36 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:13:37.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:13:37 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:13:37.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:13:37 smithi047 ceph-mon[77147]: pgmap v8926: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:13:37.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:13:37 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 17150 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:13:37.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:13:37 smithi115 conmon[90393]: 2022-05-14T04:13:37.422+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:13:37.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:13:37 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:13:37.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:13:37 smithi115 ceph-mon[78593]: pgmap v8926: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:13:37.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:13:37 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 17150 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:13:38.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:13:38 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:13:38.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:13:38 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:13:38.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:13:38 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:13:38.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:13:38 smithi115 conmon[90393]: 2022-05-14T04:13:38.397+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:13:38.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:13:38 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:13:38.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:13:38 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:13:38.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:13:38 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:13:39.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:13:39 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:13:39.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:13:39 smithi047 ceph-mon[77147]: pgmap v8927: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:13:39.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:13:39 smithi115 conmon[90393]: 2022-05-14T04:13:39.437+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:13:39.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:13:39 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:13:39.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:13:39 smithi115 ceph-mon[78593]: pgmap v8927: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:13:40.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:13:40 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:13:40.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:13:40 smithi115 conmon[90393]: 2022-05-14T04:13:40.403+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:13:40.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:13:40 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:13:41.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:13:41 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:13:41.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:13:41 smithi047 ceph-mon[77147]: pgmap v8928: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:13:41.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:13:41 smithi115 conmon[90393]: 2022-05-14T04:13:41.427+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:13:41.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:13:41 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:13:41.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:13:41 smithi115 ceph-mon[78593]: pgmap v8928: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:13:42.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:13:42 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:13:42.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:13:42 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:13:42.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:13:42 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:13:42.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:13:42 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 17155 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:13:42.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:13:42 smithi115 conmon[90393]: 2022-05-14T04:13:42.464+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:13:42.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:13:42 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:13:42.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:13:42 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:13:42.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:13:42 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:13:42.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:13:42 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 17155 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:13:43.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:13:43 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:13:43.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:13:43 smithi047 ceph-mon[77147]: pgmap v8929: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:13:43.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:13:43 smithi115 conmon[90393]: 2022-05-14T04:13:43.451+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:13:43.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:13:43 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:13:43.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:13:43 smithi115 ceph-mon[78593]: pgmap v8929: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:13:44.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:13:44 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:13:44.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:13:44 smithi115 conmon[90393]: 2022-05-14T04:13:44.488+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:13:44.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:13:44 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:13:45.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:13:45 smithi047 ceph-mon[77147]: pgmap v8930: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:13:45.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:13:45 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:13:45.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:13:45 smithi115 conmon[90393]: 2022-05-14T04:13:45.443+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:13:45.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:13:45 smithi115 ceph-mon[78593]: pgmap v8930: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:13:45.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:13:45 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:13:46.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:13:46 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:13:46.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:13:46 smithi115 conmon[90393]: 2022-05-14T04:13:46.405+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:13:46.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:13:46 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:13:47.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:13:47 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:13:47.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:13:47 smithi047 ceph-mon[77147]: pgmap v8931: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:13:47.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:13:47 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 17160 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:13:47.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:13:47 smithi115 conmon[90393]: 2022-05-14T04:13:47.386+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:13:47.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:13:47 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:13:47.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:13:47 smithi115 ceph-mon[78593]: pgmap v8931: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:13:47.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:13:47 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 17160 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:13:48.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:13:48 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:13:48.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:13:48 smithi115 conmon[90393]: 2022-05-14T04:13:48.398+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:13:48.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:13:48 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:13:49.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:13:49 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:13:49.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:13:49 smithi047 ceph-mon[77147]: pgmap v8932: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:13:49.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:13:49 smithi115 conmon[90393]: 2022-05-14T04:13:49.408+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:13:49.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:13:49 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:13:49.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:13:49 smithi115 ceph-mon[78593]: pgmap v8932: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:13:50.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:13:50 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:13:50.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:13:50 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:13:50.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:13:50 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:13:50.863 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:13:50 smithi115 conmon[90393]: 2022-05-14T04:13:50.363+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:13:50.864 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:13:50 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:13:50.864 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:13:50 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:13:50.864 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:13:50 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:13:51.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:13:51 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:13:51.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:13:51 smithi047 ceph-mon[77147]: pgmap v8933: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:13:51.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:13:51 smithi115 conmon[90393]: 2022-05-14T04:13:51.358+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:13:51.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:13:51 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:13:51.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:13:51 smithi115 ceph-mon[78593]: pgmap v8933: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:13:52.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:13:52 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:13:52.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:13:52 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 17165 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:13:52.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:13:52 smithi115 conmon[90393]: 2022-05-14T04:13:52.381+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:13:52.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:13:52 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:13:52.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:13:52 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 17165 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:13:53.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:13:53 smithi115 conmon[90393]: 2022-05-14T04:13:53.331+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:13:53.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:13:53 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:13:53.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:13:53 smithi115 ceph-mon[78593]: pgmap v8934: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:13:53.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:13:53 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:13:53.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:13:53 smithi047 ceph-mon[77147]: pgmap v8934: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:13:54.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:13:54 smithi115 conmon[90393]: 2022-05-14T04:13:54.315+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:13:54.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:13:54 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:13:54.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:13:54 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:13:54.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:13:54 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:13:54.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:13:54 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:13:54.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:13:54 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:13:54.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:13:54 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:13:55.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:13:55 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:13:55.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:13:55 smithi047 ceph-mon[77147]: pgmap v8935: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:13:55.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:13:55 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T04:13:55.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:13:55 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T04:13:55.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:13:55 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T04:13:55.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:13:55 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:13:55.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:13:55 smithi115 conmon[90393]: 2022-05-14T04:13:55.296+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:13:55.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:13:55 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:13:55.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:13:55 smithi115 ceph-mon[78593]: pgmap v8935: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:13:55.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:13:55 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T04:13:55.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:13:55 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T04:13:55.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:13:55 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T04:13:55.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:13:55 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:13:56.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:13:55 smithi047 conmon[78521]: 2022-05-14T04:13:55.696+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:13:56.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:13:55 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T04:13:56.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:13:55 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:13:56.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:13:55 smithi047 conmon[78521]: yield 2022-05-14T04:13:56.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:13:55 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:13:56.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:13:55 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:13:56.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:13:55 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:13:56.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:13:55 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T04:13:56.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:13:55 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:13:56.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:13:55 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T04:13:56.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:13:55 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:13:56.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:13:55 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T04:13:56.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:13:55 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:13:56.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:13:55 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T04:13:56.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:13:55 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:13:56.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:13:55 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T04:13:56.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:13:55 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:13:56.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:13:55 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:13:56.048 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:13:55 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:13:56.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:13:56 smithi115 conmon[90393]: 2022-05-14T04:13:56.311+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:13:56.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:13:56 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:13:56.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T04:13:56.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:13:56.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T04:13:56.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:13:56.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:13:56.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:13:56.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T04:13:56.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:13:56.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T04:13:56.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:13:56.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T04:13:56.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:13:56.613 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T04:13:56.613 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:13:56.613 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T04:13:56.613 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:13:56.614 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:13:56.614 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:13:56.614 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:13:56 smithi115 ceph-mon[78593]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T04:13:56.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:13:56 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:13:56.793 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T04:13:56.793 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:13:56.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T04:13:56.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:13:56.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:13:56.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:13:56.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T04:13:56.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:13:56.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T04:13:56.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:13:56.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T04:13:56.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:13:56.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T04:13:56.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:13:56.797 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T04:13:56.797 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:13:56.797 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:13:56.797 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:13:56.798 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:13:56 smithi047 ceph-mon[77147]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T04:13:57.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:13:57 smithi115 conmon[90393]: 2022-05-14T04:13:57.285+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:13:57.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:13:57 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:13:57.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:13:57 smithi115 ceph-mon[78593]: pgmap v8936: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:13:57.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:13:57 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 17170 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:13:57.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:13:57 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:13:57.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:13:57 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:13:57.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:13:57 smithi047 ceph-mon[77147]: pgmap v8936: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:13:57.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:13:57 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 17170 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:13:57.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:13:57 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:13:58.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:13:58 smithi115 conmon[90393]: 2022-05-14T04:13:58.268+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:13:58.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:13:58 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:13:58.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:13:58 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:13:58.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:13:58 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:13:58.771 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:13:58 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:13:58.771 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:13:58 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:13:58.771 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:13:58 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:13:59.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:13:58 smithi047 conmon[78521]: 2022-05-14T04:13:58.767+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:13:59.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:13:58 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T04:13:59.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:13:58 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:13:59.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:13:58 smithi047 conmon[78521]: yield 2022-05-14T04:13:59.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:13:58 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:13:59.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:13:58 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:13:59.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:13:58 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:13:59.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:13:58 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T04:13:59.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:13:58 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:13:59.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:13:58 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T04:13:59.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:13:58 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:13:59.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:13:58 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T04:13:59.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:13:58 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:13:59.048 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:13:58 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T04:13:59.048 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:13:58 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:13:59.048 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:13:58 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T04:13:59.049 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:13:58 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:13:59.049 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:13:58 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:13:59.049 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:13:58 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:13:59.049 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:13:58 smithi047 conmon[78521]: 2022-05-14T04:13:58.768+0000 7f533ef76700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T04:13:59.050 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:13:58 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T04:13:59.050 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:13:58 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:13:59.050 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:13:58 smithi047 conmon[78521]: yield 2022-05-14T04:13:59.050 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:13:58 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:13:59.050 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:13:58 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:13:59.051 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:13:58 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:13:59.051 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:13:58 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T04:13:59.051 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:13:58 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:13:59.051 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:13:58 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T04:13:59.052 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:13:58 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:13:59.052 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:13:58 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T04:13:59.052 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:13:58 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:13:59.052 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:13:58 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T04:13:59.052 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:13:58 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:13:59.053 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:13:58 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T04:13:59.053 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:13:58 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:13:59.053 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:13:58 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:13:59.053 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:13:58 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:13:59.053 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:13:58 smithi047 conmon[78521]: 2022-05-14T04:13:59.054 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:13:58 smithi047 conmon[78521]: During handling of the above exception, another exception occurred: 2022-05-14T04:13:59.054 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:13:58 smithi047 conmon[78521]: 2022-05-14T04:13:59.054 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:13:58 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T04:13:59.054 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:13:58 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T04:13:59.055 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:13:58 smithi047 conmon[78521]: return f(*arg) 2022-05-14T04:13:59.055 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:13:58 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T04:13:59.055 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:13:58 smithi047 conmon[78521]: self._write_client_files(client_files, host) 2022-05-14T04:13:59.055 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:13:58 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T04:13:59.055 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:13:58 smithi047 conmon[78521]: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T04:13:59.056 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:13:58 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T04:13:59.056 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:13:58 smithi047 conmon[78521]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T04:13:59.056 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:13:58 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T04:13:59.056 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:13:58 smithi047 conmon[78521]: return self.event_loop.get_result(coro) 2022-05-14T04:13:59.057 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:13:58 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T04:13:59.057 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:13:58 smithi047 conmon[78521]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T04:13:59.057 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:13:58 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T04:13:59.057 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:13:58 smithi047 conmon[78521]: return self.__get_result() 2022-05-14T04:13:59.057 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:13:58 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T04:13:59.058 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:13:58 smithi047 conmon[78521]: raise self._exception 2022-05-14T04:13:59.058 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:13:58 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T04:13:59.058 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:13:58 smithi047 conmon[78521]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T04:13:59.058 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:13:58 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T04:13:59.059 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:13:58 smithi047 conmon[78521]: conn = await self._remote_connection(host, addr) 2022-05-14T04:13:59.059 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:13:58 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T04:13:59.059 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:13:58 smithi047 conmon[78521]: raise 2022-05-14T04:13:59.059 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:13:58 smithi047 conmon[78521]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T04:13:59.060 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:13:58 smithi047 conmon[78521]: self.gen.throw(type, value, traceback) 2022-05-14T04:13:59.060 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:13:58 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T04:13:59.060 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:13:58 smithi047 conmon[78521]: raise OrchestratorError(msg) 2022-05-14T04:13:59.060 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:13:58 smithi047 conmon[78521]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:13:59.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:13:59 smithi115 conmon[90393]: 2022-05-14T04:13:59.302+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:13:59.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:13:59 smithi115 ceph-mon[78593]: pgmap v8937: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:13:59.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:13:59 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:13:59.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T04:13:59.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:13:59.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T04:13:59.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:13:59.613 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:13:59.613 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:13:59.614 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T04:13:59.614 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:13:59.614 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T04:13:59.614 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:13:59.615 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T04:13:59.615 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:13:59.615 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T04:13:59.616 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:13:59.616 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T04:13:59.616 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:13:59.616 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:13:59.616 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:13:59.617 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:13:59 smithi115 ceph-mon[78593]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T04:13:59.617 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T04:13:59.617 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:13:59.617 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T04:13:59.618 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:13:59.618 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:13:59.618 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:13:59.618 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T04:13:59.618 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:13:59.619 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T04:13:59.619 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:13:59.619 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T04:13:59.619 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:13:59.619 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T04:13:59.620 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:13:59.620 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T04:13:59.620 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:13:59.620 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:13:59.621 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:13:59.621 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T04:13:59.621 INFO:journalctl@ceph.mon.c.smithi115.stdout: During handling of the above exception, another exception occurred: 2022-05-14T04:13:59.621 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T04:13:59.622 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T04:13:59.622 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T04:13:59.622 INFO:journalctl@ceph.mon.c.smithi115.stdout: return f(*arg) 2022-05-14T04:13:59.623 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T04:13:59.623 INFO:journalctl@ceph.mon.c.smithi115.stdout: self._write_client_files(client_files, host) 2022-05-14T04:13:59.623 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T04:13:59.623 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T04:13:59.624 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T04:13:59.625 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T04:13:59.625 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T04:13:59.625 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.event_loop.get_result(coro) 2022-05-14T04:13:59.625 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T04:13:59.626 INFO:journalctl@ceph.mon.c.smithi115.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T04:13:59.626 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T04:13:59.626 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.__get_result() 2022-05-14T04:13:59.626 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T04:13:59.626 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise self._exception 2022-05-14T04:13:59.627 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T04:13:59.627 INFO:journalctl@ceph.mon.c.smithi115.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T04:13:59.627 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T04:13:59.627 INFO:journalctl@ceph.mon.c.smithi115.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T04:13:59.628 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T04:13:59.628 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise 2022-05-14T04:13:59.628 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T04:13:59.628 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.gen.throw(type, value, traceback) 2022-05-14T04:13:59.628 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T04:13:59.629 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OrchestratorError(msg) 2022-05-14T04:13:59.629 INFO:journalctl@ceph.mon.c.smithi115.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:13:59.629 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:13:59 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:13:59.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:13:59 smithi047 ceph-mon[77147]: pgmap v8937: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:13:59.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:13:59 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:13:59.793 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T04:13:59.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:13:59.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T04:13:59.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:13:59.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:13:59.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:13:59.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T04:13:59.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:13:59.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T04:13:59.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:13:59.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T04:13:59.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:13:59.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T04:13:59.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:13:59.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T04:13:59.797 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:13:59.797 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:13:59.798 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:13:59.798 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:13:59 smithi047 ceph-mon[77147]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T04:13:59.798 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T04:13:59.798 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:13:59.798 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T04:13:59.799 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:13:59.799 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:13:59.799 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:13:59.800 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T04:13:59.800 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:13:59.800 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T04:13:59.800 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:13:59.801 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T04:13:59.801 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:13:59.801 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T04:13:59.802 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:13:59.802 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T04:13:59.802 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:13:59.802 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:13:59.802 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:13:59.803 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T04:13:59.803 INFO:journalctl@ceph.mon.b.smithi047.stdout: During handling of the above exception, another exception occurred: 2022-05-14T04:13:59.804 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T04:13:59.804 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T04:13:59.805 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T04:13:59.805 INFO:journalctl@ceph.mon.b.smithi047.stdout: return f(*arg) 2022-05-14T04:13:59.806 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T04:13:59.806 INFO:journalctl@ceph.mon.b.smithi047.stdout: self._write_client_files(client_files, host) 2022-05-14T04:13:59.806 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T04:13:59.807 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T04:13:59.807 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T04:13:59.808 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T04:13:59.808 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T04:13:59.808 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.event_loop.get_result(coro) 2022-05-14T04:13:59.809 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T04:13:59.809 INFO:journalctl@ceph.mon.b.smithi047.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T04:13:59.810 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T04:13:59.810 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.__get_result() 2022-05-14T04:13:59.810 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T04:13:59.811 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise self._exception 2022-05-14T04:13:59.811 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T04:13:59.812 INFO:journalctl@ceph.mon.b.smithi047.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T04:13:59.812 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T04:13:59.812 INFO:journalctl@ceph.mon.b.smithi047.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T04:13:59.813 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T04:13:59.813 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise 2022-05-14T04:13:59.814 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T04:13:59.814 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.gen.throw(type, value, traceback) 2022-05-14T04:13:59.814 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T04:13:59.815 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OrchestratorError(msg) 2022-05-14T04:13:59.815 INFO:journalctl@ceph.mon.b.smithi047.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:13:59.815 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:13:59 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:14:00.610 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:14:00 smithi115 conmon[90393]: 2022-05-14T04:14:00.303+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:14:00.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:14:00 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:14:00.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:14:00 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:14:01.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:14:01 smithi115 conmon[90393]: 2022-05-14T04:14:01.340+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:14:01.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:14:01 smithi115 ceph-mon[78593]: pgmap v8938: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:14:01.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:14:01 smithi047 ceph-mon[77147]: pgmap v8938: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:14:02.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:14:02 smithi115 conmon[90393]: 2022-05-14T04:14:02.309+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:14:02.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:14:02 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:14:02.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:14:02 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:14:02.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:14:02 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:14:02.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:14:02 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 17175 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:14:02.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:14:02 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:14:02.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:14:02 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:14:02.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:14:02 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:14:02.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:14:02 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 17175 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:14:03.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:14:03 smithi115 conmon[90393]: 2022-05-14T04:14:03.295+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:14:04.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:14:04 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:14:04.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:14:04 smithi047 ceph-mon[77147]: pgmap v8939: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:14:04.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:14:04 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:14:04.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:14:04 smithi115 conmon[90393]: 2022-05-14T04:14:04.309+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:14:04.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:14:04 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:14:04.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:14:04 smithi115 ceph-mon[78593]: pgmap v8939: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:14:04.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:14:04 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:14:05.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:14:05 smithi115 conmon[90393]: 2022-05-14T04:14:05.265+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:14:06.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:14:06 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:14:06.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:14:06 smithi047 ceph-mon[77147]: pgmap v8940: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:14:06.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:14:06 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:14:06.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:14:06 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:14:06.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:14:06 smithi115 ceph-mon[78593]: pgmap v8940: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:14:06.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:14:06 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:14:06.610 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:14:06 smithi115 conmon[90393]: 2022-05-14T04:14:06.298+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:14:07.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:14:07 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:14:07.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:14:07 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 17180 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:14:07.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:14:07 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:14:07.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:14:07 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 17180 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:14:07.610 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:14:07 smithi115 conmon[90393]: 2022-05-14T04:14:07.308+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:14:08.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:14:08 smithi047 ceph-mon[77147]: pgmap v8941: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:14:08.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:14:08 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:14:08.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:14:08 smithi115 ceph-mon[78593]: pgmap v8941: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:14:08.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:14:08 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:14:08.610 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:14:08 smithi115 conmon[90393]: 2022-05-14T04:14:08.299+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:14:09.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:14:09 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:14:09.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:14:09 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:14:09.610 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:14:09 smithi115 conmon[90393]: 2022-05-14T04:14:09.320+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:14:10.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:14:10 smithi047 ceph-mon[77147]: pgmap v8942: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:14:10.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:14:10 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:14:10.545 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:14:10 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:14:10.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:14:10 smithi115 ceph-mon[78593]: pgmap v8942: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:14:10.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:14:10 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:14:10.612 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:14:10 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:14:10.612 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:14:10 smithi115 conmon[90393]: 2022-05-14T04:14:10.314+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:14:11.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:14:11 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:14:11.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:14:11 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:14:11.610 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:14:11 smithi115 conmon[90393]: 2022-05-14T04:14:11.312+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:14:12.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:14:12 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:14:12.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:14:12 smithi047 ceph-mon[77147]: pgmap v8943: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:14:12.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:14:12 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 17185 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:14:12.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:14:12 smithi115 conmon[90393]: 2022-05-14T04:14:12.343+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:14:12.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:14:12 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:14:12.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:14:12 smithi115 ceph-mon[78593]: pgmap v8943: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:14:12.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:14:12 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 17185 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:14:13.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:14:13 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:14:13.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:14:13 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:14:13.610 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:14:13 smithi115 conmon[90393]: 2022-05-14T04:14:13.328+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:14:14.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:14:14 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:14:14.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:14:14 smithi047 ceph-mon[77147]: pgmap v8944: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:14:14.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:14:14 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:14:14.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:14:14 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:14:14.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:14:14 smithi115 conmon[90393]: 2022-05-14T04:14:14.343+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:14:14.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:14:14 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:14:14.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:14:14 smithi115 ceph-mon[78593]: pgmap v8944: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:14:14.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:14:14 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:14:14.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:14:14 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:14:15.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:14:15 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:14:15.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:14:15 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T04:14:15.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:14:15 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T04:14:15.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:14:15 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:14:15.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:14:15 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T04:14:15.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:14:15 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T04:14:15.610 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:14:15 smithi115 conmon[90393]: 2022-05-14T04:14:15.357+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:14:16.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:14:16 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:14:16.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:14:16 smithi047 ceph-mon[77147]: pgmap v8945: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:14:16.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:14:16 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:14:16.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:14:16 smithi115 ceph-mon[78593]: pgmap v8945: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:14:16.610 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:14:16 smithi115 conmon[90393]: 2022-05-14T04:14:16.381+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:14:17.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:14:17 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:14:17.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:14:17 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 17190 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:14:17.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:14:17 smithi115 conmon[90393]: 2022-05-14T04:14:17.427+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:14:17.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:14:17 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:14:17.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:14:17 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 17190 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:14:18.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:14:18 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:14:18.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:14:18 smithi047 ceph-mon[77147]: pgmap v8946: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:14:18.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:14:18 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:14:18.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:14:18 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:14:18.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:14:18 smithi115 conmon[90393]: 2022-05-14T04:14:18.445+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:14:18.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:14:18 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:14:18.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:14:18 smithi115 ceph-mon[78593]: pgmap v8946: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:14:18.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:14:18 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:14:18.612 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:14:18 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:14:19.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:14:19 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:14:19.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:14:19 smithi115 conmon[90393]: 2022-05-14T04:14:19.428+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:14:19.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:14:19 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:14:20.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:14:20 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:14:20.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:14:20 smithi047 ceph-mon[77147]: pgmap v8947: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:14:20.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:14:20 smithi115 conmon[90393]: 2022-05-14T04:14:20.437+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:14:20.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:14:20 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:14:20.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:14:20 smithi115 ceph-mon[78593]: pgmap v8947: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:14:21.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:14:21 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:14:21.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:14:21 smithi115 conmon[90393]: 2022-05-14T04:14:21.424+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:14:21.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:14:21 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:14:22.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:14:22 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:14:22.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:14:22 smithi047 ceph-mon[77147]: pgmap v8948: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 7 op/s 2022-05-14T04:14:22.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:14:22 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:14:22.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:14:22 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:14:22.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:14:22 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 17195 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:14:22.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:14:22 smithi115 conmon[90393]: 2022-05-14T04:14:22.394+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:14:22.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:14:22 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:14:22.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:14:22 smithi115 ceph-mon[78593]: pgmap v8948: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 7 op/s 2022-05-14T04:14:22.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:14:22 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:14:22.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:14:22 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:14:22.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:14:22 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 17195 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:14:23.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:14:23 smithi115 conmon[90393]: 2022-05-14T04:14:23.358+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:14:23.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:14:23 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:14:23.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:14:23 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:14:24.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:14:24 smithi115 conmon[90393]: 2022-05-14T04:14:24.350+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:14:24.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:14:24 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:14:24.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:14:24 smithi115 ceph-mon[78593]: pgmap v8949: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 4 op/s 2022-05-14T04:14:24.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:14:24 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:14:24.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:14:24 smithi047 ceph-mon[77147]: pgmap v8949: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 4 op/s 2022-05-14T04:14:25.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:14:25 smithi115 conmon[90393]: 2022-05-14T04:14:25.354+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:14:25.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:14:25 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:14:25.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:14:25 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:14:26.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:14:26 smithi115 conmon[90393]: 2022-05-14T04:14:26.372+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:14:26.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:14:26 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:14:26.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:14:26 smithi115 ceph-mon[78593]: pgmap v8950: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 7 op/s 2022-05-14T04:14:26.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:14:26 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:14:26.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:14:26 smithi047 ceph-mon[77147]: pgmap v8950: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 7 op/s 2022-05-14T04:14:27.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:14:27 smithi115 conmon[90393]: 2022-05-14T04:14:27.414+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:14:27.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:14:27 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:14:27.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:14:27 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 17200 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:14:27.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:14:27 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:14:27.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:14:27 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 17200 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:14:28.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:14:28 smithi115 conmon[90393]: 2022-05-14T04:14:28.412+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:14:28.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:14:28 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:14:28.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:14:28 smithi115 ceph-mon[78593]: pgmap v8951: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 4 op/s 2022-05-14T04:14:28.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:14:28 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:14:28.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:14:28 smithi047 ceph-mon[77147]: pgmap v8951: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 4 op/s 2022-05-14T04:14:29.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:14:29 smithi115 conmon[90393]: 2022-05-14T04:14:29.416+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:14:29.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:14:29 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:14:29.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:14:29 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:14:30.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:14:30 smithi115 conmon[90393]: 2022-05-14T04:14:30.421+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:14:30.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:14:30 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:14:30.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:14:30 smithi115 ceph-mon[78593]: pgmap v8952: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 4 op/s 2022-05-14T04:14:30.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:14:30 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:14:30.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:14:30 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:14:30.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:14:30 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:14:30.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:14:30 smithi047 ceph-mon[77147]: pgmap v8952: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 4 op/s 2022-05-14T04:14:30.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:14:30 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:14:30.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:14:30 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:14:31.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:14:31 smithi115 conmon[90393]: 2022-05-14T04:14:31.444+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:14:31.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:14:31 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:14:31.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:14:31 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:14:32.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:14:32 smithi115 conmon[90393]: 2022-05-14T04:14:32.399+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:14:32.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:14:32 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:14:32.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:14:32 smithi115 ceph-mon[78593]: pgmap v8953: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 7 op/s 2022-05-14T04:14:32.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:14:32 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:14:32.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:14:32 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 17205 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:14:32.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:14:32 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:14:32.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:14:32 smithi047 ceph-mon[77147]: pgmap v8953: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 7 op/s 2022-05-14T04:14:32.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:14:32 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:14:32.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:14:32 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 17205 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:14:33.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:14:33 smithi115 conmon[90393]: 2022-05-14T04:14:33.397+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:14:33.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:14:33 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:14:33.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:14:33 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:14:34.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:14:34 smithi115 conmon[90393]: 2022-05-14T04:14:34.416+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:14:34.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:14:34 smithi115 ceph-mon[78593]: pgmap v8954: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:14:34.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:14:34 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:14:34.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:14:34 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:14:34.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:14:34 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:14:34.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:14:34 smithi047 ceph-mon[77147]: pgmap v8954: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:14:34.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:14:34 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:14:34.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:14:34 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:14:34.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:14:34 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:14:35.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:14:35 smithi115 conmon[90393]: 2022-05-14T04:14:35.452+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:14:35.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:14:35 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:14:35.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:14:35 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:14:36.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:14:36 smithi115 conmon[90393]: 2022-05-14T04:14:36.459+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:14:36.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:14:36 smithi115 ceph-mon[78593]: pgmap v8955: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:14:36.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:14:36 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:14:36.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:14:36 smithi047 ceph-mon[77147]: pgmap v8955: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:14:36.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:14:36 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:14:37.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:14:37 smithi115 conmon[90393]: 2022-05-14T04:14:37.463+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:14:37.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:14:37 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:14:37.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:14:37 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 17210 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:14:37.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:14:37 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:14:37.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:14:37 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 17210 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:14:38.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:14:38 smithi115 conmon[90393]: 2022-05-14T04:14:38.466+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:14:38.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:14:38 smithi115 ceph-mon[78593]: pgmap v8956: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:14:38.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:14:38 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:14:38.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:14:38 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:14:38.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:14:38 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:14:38.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:14:38 smithi047 ceph-mon[77147]: pgmap v8956: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:14:38.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:14:38 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:14:38.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:14:38 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:14:38.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:14:38 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:14:39.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:14:39 smithi115 conmon[90393]: 2022-05-14T04:14:39.513+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:14:39.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:14:39 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:14:39.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:14:39 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:14:40.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:14:40 smithi115 conmon[90393]: 2022-05-14T04:14:40.540+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:14:40.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:14:40 smithi115 ceph-mon[78593]: pgmap v8957: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:14:40.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:14:40 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:14:40.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:14:40 smithi047 ceph-mon[77147]: pgmap v8957: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:14:40.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:14:40 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:14:41.610 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:14:41 smithi115 conmon[90393]: 2022-05-14T04:14:41.524+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:14:41.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:14:41 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:14:41.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:14:41 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:14:42.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:14:42 smithi115 conmon[90393]: 2022-05-14T04:14:42.546+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:14:42.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:14:42 smithi115 ceph-mon[78593]: pgmap v8958: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:14:42.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:14:42 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:14:42.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:14:42 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:14:42.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:14:42 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:14:42.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:14:42 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 17215 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:14:42.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:14:42 smithi047 ceph-mon[77147]: pgmap v8958: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:14:42.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:14:42 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:14:42.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:14:42 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:14:42.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:14:42 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:14:42.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:14:42 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 17215 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:14:43.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:14:43 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:14:43.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:14:43 smithi115 conmon[90393]: 2022-05-14T04:14:43.575+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:14:43.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:14:43 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:14:44.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:14:44 smithi047 ceph-mon[77147]: pgmap v8959: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:14:44.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:14:44 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:14:44.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:14:44 smithi115 conmon[90393]: 2022-05-14T04:14:44.579+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:14:44.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:14:44 smithi115 ceph-mon[78593]: pgmap v8959: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:14:44.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:14:44 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:14:45.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:14:45 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:14:45.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:14:45 smithi115 conmon[90393]: 2022-05-14T04:14:45.534+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:14:45.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:14:45 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:14:46.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:14:46 smithi047 ceph-mon[77147]: pgmap v8960: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:14:46.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:14:46 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:14:46.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:14:46 smithi115 conmon[90393]: 2022-05-14T04:14:46.542+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:14:46.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:14:46 smithi115 ceph-mon[78593]: pgmap v8960: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:14:46.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:14:46 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:14:47.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:14:47 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:14:47.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:14:47 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 17220 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:14:47.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:14:47 smithi115 conmon[90393]: 2022-05-14T04:14:47.541+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:14:47.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:14:47 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:14:47.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:14:47 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 17220 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:14:48.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:14:48 smithi047 ceph-mon[77147]: pgmap v8961: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:14:48.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:14:48 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:14:48.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:14:48 smithi115 conmon[90393]: 2022-05-14T04:14:48.540+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:14:48.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:14:48 smithi115 ceph-mon[78593]: pgmap v8961: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:14:48.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:14:48 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:14:49.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:14:49 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:14:49.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:14:49 smithi115 conmon[90393]: 2022-05-14T04:14:49.553+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:14:49.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:14:49 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:14:50.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:14:50 smithi047 ceph-mon[77147]: pgmap v8962: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:14:50.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:14:50 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:14:50.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:14:50 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:14:50.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:14:50 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:14:50.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:14:50 smithi115 conmon[90393]: 2022-05-14T04:14:50.550+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:14:50.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:14:50 smithi115 ceph-mon[78593]: pgmap v8962: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:14:50.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:14:50 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:14:50.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:14:50 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:14:50.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:14:50 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:14:51.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:14:51 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:14:51.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:14:51 smithi115 conmon[90393]: 2022-05-14T04:14:51.533+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:14:51.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:14:51 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:14:52.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:14:52 smithi047 ceph-mon[77147]: pgmap v8963: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:14:52.795 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:14:52 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:14:52.795 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:14:52 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 17225 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:14:52.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:14:52 smithi115 conmon[90393]: 2022-05-14T04:14:52.552+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:14:52.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:14:52 smithi115 ceph-mon[78593]: pgmap v8963: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:14:52.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:14:52 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:14:52.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:14:52 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 17225 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:14:53.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:14:53 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:14:53.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:14:53 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:14:53.860 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:14:53 smithi115 conmon[90393]: 2022-05-14T04:14:53.561+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:14:54.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:14:54 smithi047 ceph-mon[77147]: pgmap v8964: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:14:54.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:14:54 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:14:54.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:14:54 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:14:54.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:14:54 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:14:54.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:14:54 smithi115 conmon[90393]: 2022-05-14T04:14:54.541+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:14:54.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:14:54 smithi115 ceph-mon[78593]: pgmap v8964: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:14:54.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:14:54 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:14:54.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:14:54 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:14:54.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:14:54 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:14:55.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:14:55 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:14:55.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:14:55 smithi115 conmon[90393]: 2022-05-14T04:14:55.549+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:14:55.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:14:55 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:14:56.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:14:56 smithi047 ceph-mon[77147]: pgmap v8965: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:14:56.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:14:56 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:14:56.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:14:56 smithi115 conmon[90393]: 2022-05-14T04:14:56.554+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:14:56.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:14:56 smithi115 ceph-mon[78593]: pgmap v8965: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:14:56.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:14:56 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:14:57.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:14:57 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:14:57.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:14:57 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 17230 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:14:57.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:14:57 smithi115 conmon[90393]: 2022-05-14T04:14:57.539+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:14:57.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:14:57 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:14:57.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:14:57 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 17230 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:14:58.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:14:58 smithi047 ceph-mon[77147]: pgmap v8966: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:14:58.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:14:58 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:14:58.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:14:58 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:14:58.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:14:58 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:14:58.817 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:14:58 smithi115 conmon[90393]: 2022-05-14T04:14:58.527+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:14:58.818 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:14:58 smithi115 ceph-mon[78593]: pgmap v8966: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:14:58.818 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:14:58 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:14:58.818 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:14:58 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:14:58.819 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:14:58 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:14:59.561 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:14:59 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:14:59.561 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:14:59 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T04:14:59.562 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:14:59 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T04:14:59.562 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:14:59 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T04:14:59.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:14:59 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:14:59.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:14:59 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T04:14:59.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:14:59 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T04:14:59.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:14:59 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T04:14:59.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:14:59 smithi115 conmon[90393]: 2022-05-14T04:14:59.557+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:15:00.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:15:00 smithi047 ceph-mon[77147]: pgmap v8967: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:15:00.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:15:00 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:15:00.860 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:15:00 smithi115 conmon[90393]: 2022-05-14T04:15:00.589+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:15:00.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:15:00 smithi115 ceph-mon[78593]: pgmap v8967: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:15:00.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:15:00 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:15:01.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:15:01 smithi047 ceph-mon[77147]: pgmap v8968: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:15:01.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:15:01 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:15:01.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:15:01 smithi115 conmon[90393]: 2022-05-14T04:15:01.629+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:15:01.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:15:01 smithi115 ceph-mon[78593]: pgmap v8968: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:15:01.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:15:01 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:15:02.293 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:15:01 smithi047 conmon[78521]: 2022-05-14T04:15:01.871+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:15:02.300 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:15:01 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T04:15:02.301 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:15:01 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:15:02.301 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:15:01 smithi047 conmon[78521]: yield 2022-05-14T04:15:02.301 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:15:01 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:15:02.301 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:15:01 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:15:02.302 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:15:01 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:15:02.302 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:15:01 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T04:15:02.302 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:15:01 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:15:02.302 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:15:01 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T04:15:02.303 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:15:01 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:15:02.303 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:15:01 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T04:15:02.303 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:15:01 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:15:02.303 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:15:01 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T04:15:02.304 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:15:01 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:15:02.304 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:15:01 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T04:15:02.304 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:15:01 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:15:02.304 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:15:01 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:15:02.305 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:15:01 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:15:02.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:15:02 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:15:02.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:15:02 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:15:02.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:15:02 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:15:02.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:15:02 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:15:02.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T04:15:02.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:15:02.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T04:15:02.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:15:02.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:15:02.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:15:02.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T04:15:02.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:15:02.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T04:15:02.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:15:02.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T04:15:02.797 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:15:02.797 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T04:15:02.797 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:15:02.797 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T04:15:02.798 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:15:02.798 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:15:02.798 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:15:02.798 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:15:02 smithi047 ceph-mon[77147]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T04:15:02.798 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:15:02 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 17235 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:15:02.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:15:02 smithi115 conmon[90393]: 2022-05-14T04:15:02.622+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:15:02.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:15:02 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:15:02.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:15:02 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:15:02.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:15:02 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:15:02.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:15:02 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:15:02.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T04:15:02.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:15:02.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T04:15:02.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:15:02.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:15:02.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:15:02.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T04:15:02.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:15:02.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T04:15:02.863 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:15:02.863 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T04:15:02.863 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:15:02.863 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T04:15:02.863 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:15:02.864 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T04:15:02.864 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:15:02.864 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:15:02.864 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:15:02.864 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:15:02 smithi115 ceph-mon[78593]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T04:15:02.865 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:15:02 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 17235 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:15:03.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:15:03 smithi047 ceph-mon[77147]: pgmap v8969: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:15:03.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:15:03 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:15:03.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:15:03 smithi115 conmon[90393]: 2022-05-14T04:15:03.613+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:15:03.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:15:03 smithi115 ceph-mon[78593]: pgmap v8969: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:15:03.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:15:03 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:15:04.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:15:04 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:15:04.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:15:04 smithi115 conmon[90393]: 2022-05-14T04:15:04.566+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:15:04.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:15:04 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:15:05.293 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:15:04 smithi047 conmon[78521]: 2022-05-14T04:15:04.945+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:15:05.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:15:04 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T04:15:05.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:15:04 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:15:05.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:15:04 smithi047 conmon[78521]: yield 2022-05-14T04:15:05.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:15:04 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:15:05.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:15:04 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:15:05.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:15:04 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:15:05.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:15:04 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T04:15:05.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:15:04 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:15:05.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:15:04 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T04:15:05.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:15:04 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:15:05.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:15:04 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T04:15:05.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:15:04 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:15:05.297 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:15:04 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T04:15:05.297 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:15:04 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:15:05.297 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:15:04 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T04:15:05.298 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:15:04 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:15:05.298 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:15:04 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:15:05.298 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:15:04 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:15:05.298 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:15:04 smithi047 conmon[78521]: 2022-05-14T04:15:04.945+0000 7f5344781700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T04:15:05.299 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:15:04 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T04:15:05.299 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:15:04 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:15:05.299 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:15:04 smithi047 conmon[78521]: yield 2022-05-14T04:15:05.299 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:15:04 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:15:05.299 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:15:04 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:15:05.300 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:15:04 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:15:05.300 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:15:04 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T04:15:05.300 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:15:04 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:15:05.300 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:15:04 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T04:15:05.301 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:15:04 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:15:05.301 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:15:04 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T04:15:05.301 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:15:04 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:15:05.301 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:15:04 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T04:15:05.301 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:15:04 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:15:05.302 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:15:04 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T04:15:05.302 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:15:04 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:15:05.302 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:15:04 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:15:05.302 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:15:04 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:15:05.303 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:15:04 smithi047 conmon[78521]: 2022-05-14T04:15:05.303 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:15:04 smithi047 conmon[78521]: During handling of the above exception, another exception occurred: 2022-05-14T04:15:05.303 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:15:04 smithi047 conmon[78521]: 2022-05-14T04:15:05.303 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:15:04 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T04:15:05.304 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:15:04 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T04:15:05.304 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:15:04 smithi047 conmon[78521]: return f(*arg) 2022-05-14T04:15:05.304 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:15:04 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T04:15:05.304 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:15:04 smithi047 conmon[78521]: self._write_client_files(client_files, host) 2022-05-14T04:15:05.305 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:15:04 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T04:15:05.305 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:15:04 smithi047 conmon[78521]: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T04:15:05.305 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:15:04 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T04:15:05.305 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:15:04 smithi047 conmon[78521]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T04:15:05.305 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:15:04 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T04:15:05.306 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:15:04 smithi047 conmon[78521]: return self.event_loop.get_result(coro) 2022-05-14T04:15:05.306 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:15:04 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T04:15:05.306 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:15:04 smithi047 conmon[78521]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T04:15:05.306 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:15:04 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T04:15:05.307 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:15:04 smithi047 conmon[78521]: return self.__get_result() 2022-05-14T04:15:05.307 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:15:04 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T04:15:05.307 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:15:04 smithi047 conmon[78521]: raise self._exception 2022-05-14T04:15:05.307 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:15:04 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T04:15:05.308 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:15:04 smithi047 conmon[78521]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T04:15:05.308 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:15:04 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T04:15:05.308 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:15:04 smithi047 conmon[78521]: conn = await self._remote_connection(host, addr) 2022-05-14T04:15:05.308 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:15:04 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T04:15:05.308 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:15:04 smithi047 conmon[78521]: raise 2022-05-14T04:15:05.309 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:15:04 smithi047 conmon[78521]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T04:15:05.309 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:15:04 smithi047 conmon[78521]: self.gen.throw(type, value, traceback) 2022-05-14T04:15:05.309 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:15:04 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T04:15:05.310 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:15:04 smithi047 conmon[78521]: raise OrchestratorError(msg) 2022-05-14T04:15:05.310 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:15:04 smithi047 conmon[78521]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:15:05.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:15:05 smithi047 ceph-mon[77147]: pgmap v8970: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:15:05.795 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:15:05 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:15:05.795 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:15:05 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:15:05.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T04:15:05.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:15:05.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T04:15:05.797 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:15:05.797 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:15:05.797 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:15:05.798 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T04:15:05.798 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:15:05.798 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T04:15:05.798 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:15:05.798 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T04:15:05.799 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:15:05.799 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T04:15:05.799 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:15:05.799 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T04:15:05.799 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:15:05.800 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:15:05.800 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:15:05.800 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:15:05 smithi047 ceph-mon[77147]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T04:15:05.800 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T04:15:05.801 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:15:05.801 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T04:15:05.801 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:15:05.802 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:15:05.802 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:15:05.802 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T04:15:05.802 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:15:05.802 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T04:15:05.803 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:15:05.803 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T04:15:05.803 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:15:05.803 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T04:15:05.803 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:15:05.804 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T04:15:05.804 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:15:05.804 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:15:05.804 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:15:05.804 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T04:15:05.805 INFO:journalctl@ceph.mon.b.smithi047.stdout: During handling of the above exception, another exception occurred: 2022-05-14T04:15:05.805 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T04:15:05.805 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T04:15:05.805 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T04:15:05.806 INFO:journalctl@ceph.mon.b.smithi047.stdout: return f(*arg) 2022-05-14T04:15:05.806 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T04:15:05.806 INFO:journalctl@ceph.mon.b.smithi047.stdout: self._write_client_files(client_files, host) 2022-05-14T04:15:05.806 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T04:15:05.806 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T04:15:05.807 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T04:15:05.807 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T04:15:05.807 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T04:15:05.807 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.event_loop.get_result(coro) 2022-05-14T04:15:05.808 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T04:15:05.808 INFO:journalctl@ceph.mon.b.smithi047.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T04:15:05.808 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T04:15:05.808 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.__get_result() 2022-05-14T04:15:05.808 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T04:15:05.809 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise self._exception 2022-05-14T04:15:05.809 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T04:15:05.809 INFO:journalctl@ceph.mon.b.smithi047.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T04:15:05.809 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T04:15:05.810 INFO:journalctl@ceph.mon.b.smithi047.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T04:15:05.810 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T04:15:05.810 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise 2022-05-14T04:15:05.810 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T04:15:05.811 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.gen.throw(type, value, traceback) 2022-05-14T04:15:05.811 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T04:15:05.811 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OrchestratorError(msg) 2022-05-14T04:15:05.811 INFO:journalctl@ceph.mon.b.smithi047.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:15:05.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:15:05 smithi115 conmon[90393]: 2022-05-14T04:15:05.586+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:15:05.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:15:05 smithi115 ceph-mon[78593]: pgmap v8970: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:15:05.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:15:05 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:15:05.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:15:05 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:15:05.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T04:15:05.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:15:05.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T04:15:05.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:15:05.863 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:15:05.863 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:15:05.863 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T04:15:05.863 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:15:05.863 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T04:15:05.864 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:15:05.864 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T04:15:05.864 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:15:05.865 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T04:15:05.865 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:15:05.866 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T04:15:05.866 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:15:05.866 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:15:05.867 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:15:05.867 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:15:05 smithi115 ceph-mon[78593]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T04:15:05.867 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T04:15:05.868 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:15:05.868 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T04:15:05.868 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:15:05.869 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:15:05.869 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:15:05.869 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T04:15:05.870 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:15:05.870 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T04:15:05.870 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:15:05.871 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T04:15:05.871 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:15:05.872 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T04:15:05.872 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:15:05.872 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T04:15:05.872 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:15:05.873 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:15:05.873 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:15:05.873 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T04:15:05.874 INFO:journalctl@ceph.mon.c.smithi115.stdout: During handling of the above exception, another exception occurred: 2022-05-14T04:15:05.874 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T04:15:05.874 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T04:15:05.874 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T04:15:05.875 INFO:journalctl@ceph.mon.c.smithi115.stdout: return f(*arg) 2022-05-14T04:15:05.875 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T04:15:05.875 INFO:journalctl@ceph.mon.c.smithi115.stdout: self._write_client_files(client_files, host) 2022-05-14T04:15:05.875 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T04:15:05.876 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T04:15:05.876 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T04:15:05.876 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T04:15:05.877 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T04:15:05.877 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.event_loop.get_result(coro) 2022-05-14T04:15:05.877 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T04:15:05.877 INFO:journalctl@ceph.mon.c.smithi115.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T04:15:05.877 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T04:15:05.878 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.__get_result() 2022-05-14T04:15:05.878 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T04:15:05.878 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise self._exception 2022-05-14T04:15:05.878 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T04:15:05.878 INFO:journalctl@ceph.mon.c.smithi115.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T04:15:05.879 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T04:15:05.879 INFO:journalctl@ceph.mon.c.smithi115.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T04:15:05.879 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T04:15:05.879 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise 2022-05-14T04:15:05.880 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T04:15:05.880 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.gen.throw(type, value, traceback) 2022-05-14T04:15:05.880 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T04:15:05.880 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OrchestratorError(msg) 2022-05-14T04:15:05.880 INFO:journalctl@ceph.mon.c.smithi115.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:15:06.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:15:06 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:15:06.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:15:06 smithi115 conmon[90393]: 2022-05-14T04:15:06.555+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:15:06.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:15:06 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:15:07.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:15:07 smithi047 ceph-mon[77147]: pgmap v8971: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:15:07.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:15:07 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:15:07.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:15:07 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 17240 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:15:07.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:15:07 smithi115 conmon[90393]: 2022-05-14T04:15:07.598+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:15:07.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:15:07 smithi115 ceph-mon[78593]: pgmap v8971: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:15:07.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:15:07 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:15:07.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:15:07 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 17240 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:15:08.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:15:08 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:15:08.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:15:08 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:15:08.860 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:15:08 smithi115 conmon[90393]: 2022-05-14T04:15:08.570+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:15:09.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:15:09 smithi047 ceph-mon[77147]: pgmap v8972: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:15:09.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:15:09 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:15:09.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:15:09 smithi115 conmon[90393]: 2022-05-14T04:15:09.597+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:15:09.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:15:09 smithi115 ceph-mon[78593]: pgmap v8972: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:15:09.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:15:09 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:15:10.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:15:10 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:15:10.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:15:10 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:15:10.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:15:10 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:15:10.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:15:10 smithi115 conmon[90393]: 2022-05-14T04:15:10.603+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:15:10.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:15:10 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:15:10.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:15:10 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:15:10.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:15:10 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:15:11.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:15:11 smithi047 ceph-mon[77147]: pgmap v8973: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:15:11.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:15:11 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:15:11.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:15:11 smithi115 conmon[90393]: 2022-05-14T04:15:11.563+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:15:11.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:15:11 smithi115 ceph-mon[78593]: pgmap v8973: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:15:11.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:15:11 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:15:12.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:15:12 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:15:12.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:15:12 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 17245 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:15:12.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:15:12 smithi115 conmon[90393]: 2022-05-14T04:15:12.576+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:15:12.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:15:12 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:15:12.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:15:12 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 17245 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:15:13.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:15:13 smithi047 ceph-mon[77147]: pgmap v8974: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:15:13.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:15:13 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:15:13.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:15:13 smithi115 conmon[90393]: 2022-05-14T04:15:13.624+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:15:13.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:15:13 smithi115 ceph-mon[78593]: pgmap v8974: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:15:13.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:15:13 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:15:14.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:15:14 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:15:14.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:15:14 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:15:14.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:15:14 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:15:14.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:15:14 smithi115 conmon[90393]: 2022-05-14T04:15:14.639+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:15:14.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:15:14 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:15:14.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:15:14 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:15:14.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:15:14 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:15:15.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:15:15 smithi047 ceph-mon[77147]: pgmap v8975: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:15:15.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:15:15 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T04:15:15.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:15:15 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T04:15:15.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:15:15 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:15:15.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:15:15 smithi115 conmon[90393]: 2022-05-14T04:15:15.610+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:15:15.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:15:15 smithi115 ceph-mon[78593]: pgmap v8975: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:15:15.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:15:15 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T04:15:15.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:15:15 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T04:15:15.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:15:15 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:15:16.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:15:16 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:15:16.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:15:16 smithi115 conmon[90393]: 2022-05-14T04:15:16.632+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:15:16.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:15:16 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:15:17.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:15:17 smithi047 ceph-mon[77147]: pgmap v8976: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:15:17.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:15:17 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:15:17.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:15:17 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 17250 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:15:17.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:15:17 smithi115 conmon[90393]: 2022-05-14T04:15:17.638+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:15:17.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:15:17 smithi115 ceph-mon[78593]: pgmap v8976: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:15:17.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:15:17 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:15:17.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:15:17 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 17250 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:15:18.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:15:18 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:15:18.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:15:18 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:15:18.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:15:18 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:15:18.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:15:18 smithi115 conmon[90393]: 2022-05-14T04:15:18.667+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:15:18.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:15:18 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:15:18.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:15:18 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:15:18.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:15:18 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:15:19.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:15:19 smithi047 ceph-mon[77147]: pgmap v8977: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:15:19.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:15:19 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:15:19.860 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:15:19 smithi115 conmon[90393]: 2022-05-14T04:15:19.691+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:15:19.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:15:19 smithi115 ceph-mon[78593]: pgmap v8977: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:15:19.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:15:19 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:15:20.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:15:20 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:15:20.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:15:20 smithi115 conmon[90393]: 2022-05-14T04:15:20.644+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:15:20.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:15:20 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:15:21.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:15:21 smithi047 ceph-mon[77147]: pgmap v8978: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:15:21.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:15:21 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:15:21.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:15:21 smithi115 conmon[90393]: 2022-05-14T04:15:21.622+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:15:21.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:15:21 smithi115 ceph-mon[78593]: pgmap v8978: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:15:21.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:15:21 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:15:22.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:15:22 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:15:22.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:15:22 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:15:22.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:15:22 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:15:22.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:15:22 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 17255 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:15:22.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:15:22 smithi115 conmon[90393]: 2022-05-14T04:15:22.576+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:15:22.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:15:22 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:15:22.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:15:22 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:15:22.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:15:22 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:15:22.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:15:22 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 17255 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:15:23.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:15:23 smithi047 ceph-mon[77147]: pgmap v8979: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:15:23.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:15:23 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:15:23.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:15:23 smithi115 conmon[90393]: 2022-05-14T04:15:23.626+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:15:23.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:15:23 smithi115 ceph-mon[78593]: pgmap v8979: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:15:23.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:15:23 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:15:24.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:15:24 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:15:24.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:15:24 smithi115 conmon[90393]: 2022-05-14T04:15:24.651+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:15:24.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:15:24 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:15:25.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:15:25 smithi047 ceph-mon[77147]: pgmap v8980: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:15:25.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:15:25 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:15:25.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:15:25 smithi115 conmon[90393]: 2022-05-14T04:15:25.684+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:15:25.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:15:25 smithi115 ceph-mon[78593]: pgmap v8980: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:15:25.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:15:25 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:15:26.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:15:26 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:15:26.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:15:26 smithi115 conmon[90393]: 2022-05-14T04:15:26.708+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:15:26.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:15:26 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:15:27.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:15:27 smithi047 ceph-mon[77147]: pgmap v8981: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:15:27.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:15:27 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:15:27.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:15:27 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 17260 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:15:27.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:15:27 smithi115 conmon[90393]: 2022-05-14T04:15:27.702+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:15:27.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:15:27 smithi115 ceph-mon[78593]: pgmap v8981: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:15:27.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:15:27 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:15:27.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:15:27 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 17260 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:15:28.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:15:28 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:15:28.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:15:28 smithi115 conmon[90393]: 2022-05-14T04:15:28.711+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:15:28.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:15:28 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:15:29.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:15:29 smithi115 conmon[90393]: 2022-05-14T04:15:29.685+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:15:29.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:15:29 smithi115 ceph-mon[78593]: pgmap v8982: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:15:29.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:15:29 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:15:30.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:15:29 smithi047 ceph-mon[77147]: pgmap v8982: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:15:30.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:15:29 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:15:30.860 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:15:30 smithi115 conmon[90393]: 2022-05-14T04:15:30.644+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:15:30.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:15:30 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:15:30.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:15:30 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:15:30.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:15:30 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:15:31.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:15:30 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:15:31.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:15:30 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:15:31.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:15:30 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:15:31.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:15:31 smithi115 conmon[90393]: 2022-05-14T04:15:31.622+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:15:31.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:15:31 smithi115 ceph-mon[78593]: pgmap v8983: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:15:31.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:15:31 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:15:32.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:15:31 smithi047 ceph-mon[77147]: pgmap v8983: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:15:32.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:15:31 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:15:32.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:15:32 smithi115 conmon[90393]: 2022-05-14T04:15:32.593+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:15:32.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:15:32 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:15:32.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:15:32 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 17265 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:15:33.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:15:32 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:15:33.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:15:32 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 17265 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:15:33.860 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:15:33 smithi115 conmon[90393]: 2022-05-14T04:15:33.550+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:15:34.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:15:34 smithi047 ceph-mon[77147]: pgmap v8984: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:15:34.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:15:34 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:15:34.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:15:34 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:15:34.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:15:34 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:15:34.599 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:15:34 smithi115 ceph-mon[78593]: pgmap v8984: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:15:34.599 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:15:34 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:15:34.599 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:15:34 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:15:34.600 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:15:34 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:15:34.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:15:34 smithi115 conmon[90393]: 2022-05-14T04:15:34.596+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:15:35.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:15:35 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:15:35.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:15:35 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:15:36.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:15:35 smithi115 conmon[90393]: 2022-05-14T04:15:35.622+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:15:36.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:15:36 smithi047 ceph-mon[77147]: pgmap v8985: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:15:36.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:15:36 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:15:36.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:15:36 smithi115 ceph-mon[78593]: pgmap v8985: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:15:36.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:15:36 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:15:37.110 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:15:36 smithi115 conmon[90393]: 2022-05-14T04:15:36.623+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:15:37.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:15:37 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:15:37.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:15:37 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 17270 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:15:37.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:15:37 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:15:37.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:15:37 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 17270 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:15:38.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:15:37 smithi115 conmon[90393]: 2022-05-14T04:15:37.632+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:15:38.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:15:38 smithi047 ceph-mon[77147]: pgmap v8986: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:15:38.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:15:38 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:15:38.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:15:38 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:15:38.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:15:38 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:15:38.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:15:38 smithi115 ceph-mon[78593]: pgmap v8986: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:15:38.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:15:38 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:15:38.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:15:38 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:15:38.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:15:38 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:15:39.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:15:38 smithi115 conmon[90393]: 2022-05-14T04:15:38.641+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:15:39.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:15:39 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:15:39.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:15:39 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:15:40.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:15:39 smithi115 conmon[90393]: 2022-05-14T04:15:39.648+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:15:40.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:15:40 smithi047 ceph-mon[77147]: pgmap v8987: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:15:40.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:15:40 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:15:40.603 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:15:40 smithi115 ceph-mon[78593]: pgmap v8987: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:15:40.604 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:15:40 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:15:40.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:15:40 smithi115 conmon[90393]: 2022-05-14T04:15:40.600+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:15:41.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:15:41 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:15:41.594 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:15:41 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:15:41.860 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:15:41 smithi115 conmon[90393]: 2022-05-14T04:15:41.591+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:15:42.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:15:42 smithi047 ceph-mon[77147]: pgmap v8988: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:15:42.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:15:42 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:15:42.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:15:42 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:15:42.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:15:42 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:15:42.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:15:42 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 17275 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:15:42.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:15:42 smithi115 ceph-mon[78593]: pgmap v8988: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:15:42.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:15:42 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:15:42.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:15:42 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:15:42.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:15:42 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:15:42.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:15:42 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 17275 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:15:43.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:15:42 smithi115 conmon[90393]: 2022-05-14T04:15:42.617+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:15:43.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:15:43 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:15:43.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:15:43 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:15:44.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:15:43 smithi115 conmon[90393]: 2022-05-14T04:15:43.612+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:15:44.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:15:44 smithi047 ceph-mon[77147]: pgmap v8989: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:15:44.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:15:44 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:15:44.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:15:44 smithi115 ceph-mon[78593]: pgmap v8989: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:15:44.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:15:44 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:15:45.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:15:44 smithi115 conmon[90393]: 2022-05-14T04:15:44.630+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:15:45.606 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:15:45 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:15:45.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:15:45 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:15:45.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:15:45 smithi115 conmon[90393]: 2022-05-14T04:15:45.624+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:15:46.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:15:46 smithi115 ceph-mon[78593]: pgmap v8990: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:15:46.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:15:46 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:15:46.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:15:46 smithi047 ceph-mon[77147]: pgmap v8990: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:15:46.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:15:46 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:15:47.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:15:46 smithi115 conmon[90393]: 2022-05-14T04:15:46.636+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:15:47.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:15:47 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:15:47.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:15:47 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 17280 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:15:47.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:15:47 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:15:47.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:15:47 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 17280 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:15:48.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:15:47 smithi115 conmon[90393]: 2022-05-14T04:15:47.621+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:15:48.590 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:15:48 smithi115 ceph-mon[78593]: pgmap v8991: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:15:48.590 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:15:48 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:15:48.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:15:48 smithi047 ceph-mon[77147]: pgmap v8991: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:15:48.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:15:48 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:15:48.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:15:48 smithi115 conmon[90393]: 2022-05-14T04:15:48.586+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:15:49.572 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:15:49 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:15:49.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:15:49 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:15:49.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:15:49 smithi115 conmon[90393]: 2022-05-14T04:15:49.569+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:15:50.580 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:15:50 smithi115 ceph-mon[78593]: pgmap v8992: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:15:50.581 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:15:50 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:15:50.581 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:15:50 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:15:50.581 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:15:50 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:15:50.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:15:50 smithi047 ceph-mon[77147]: pgmap v8992: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:15:50.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:15:50 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:15:50.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:15:50 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:15:50.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:15:50 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:15:50.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:15:50 smithi115 conmon[90393]: 2022-05-14T04:15:50.577+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:15:51.593 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:15:51 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:15:51.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:15:51 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:15:51.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:15:51 smithi115 conmon[90393]: 2022-05-14T04:15:51.590+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:15:52.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:15:52 smithi115 conmon[90393]: 2022-05-14T04:15:52.553+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:15:52.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:15:52 smithi115 ceph-mon[78593]: pgmap v8993: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:15:52.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:15:52 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:15:52.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:15:52 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 17285 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:15:52.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:15:52 smithi047 ceph-mon[77147]: pgmap v8993: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:15:52.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:15:52 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:15:52.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:15:52 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 17285 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:15:53.578 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:15:53 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:15:53.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:15:53 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:15:53.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:15:53 smithi115 conmon[90393]: 2022-05-14T04:15:53.575+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:15:54.605 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:15:54 smithi115 ceph-mon[78593]: pgmap v8994: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:15:54.605 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:15:54 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:15:54.606 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:15:54 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:15:54.606 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:15:54 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:15:54.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:15:54 smithi047 ceph-mon[77147]: pgmap v8994: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:15:54.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:15:54 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:15:54.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:15:54 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:15:54.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:15:54 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:15:54.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:15:54 smithi115 conmon[90393]: 2022-05-14T04:15:54.602+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:15:55.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:15:55 smithi115 conmon[90393]: 2022-05-14T04:15:55.556+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:15:55.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:15:55 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:15:55.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:15:55 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:15:56.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:15:56 smithi115 conmon[90393]: 2022-05-14T04:15:56.547+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:15:56.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:15:56 smithi115 ceph-mon[78593]: pgmap v8995: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:15:56.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:15:56 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:15:56.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:15:56 smithi047 ceph-mon[77147]: pgmap v8995: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:15:56.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:15:56 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:15:57.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:15:57 smithi115 conmon[90393]: 2022-05-14T04:15:57.574+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:15:57.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:15:57 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:15:57.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:15:57 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 17290 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:15:57.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:15:57 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:15:57.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:15:57 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 17290 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:15:58.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:15:58 smithi115 conmon[90393]: 2022-05-14T04:15:58.577+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:15:58.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:15:58 smithi115 ceph-mon[78593]: pgmap v8996: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:15:58.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:15:58 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:15:58.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:15:58 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:15:58.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:15:58 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:15:58.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:15:58 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:15:58.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:15:58 smithi047 ceph-mon[77147]: pgmap v8996: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:15:58.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:15:58 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:15:58.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:15:58 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:15:58.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:15:58 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:15:58.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:15:58 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:15:59.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:15:59 smithi115 conmon[90393]: 2022-05-14T04:15:59.555+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:15:59.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:15:59 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:15:59.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:15:59 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:16:00.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:16:00 smithi115 ceph-mon[78593]: pgmap v8997: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:16:00.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:16:00 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:16:00.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:16:00 smithi047 ceph-mon[77147]: pgmap v8997: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:16:00.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:16:00 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:16:00.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:16:00 smithi115 conmon[90393]: 2022-05-14T04:16:00.605+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:16:01.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:16:01 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:16:01.610 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:16:01 smithi115 conmon[90393]: 2022-05-14T04:16:01.575+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:16:01.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:16:01 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:16:02.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:16:02 smithi115 conmon[90393]: 2022-05-14T04:16:02.576+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:16:02.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:16:02 smithi115 ceph-mon[78593]: pgmap v8998: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:16:02.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:16:02 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:16:02.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:16:02 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:16:02.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:16:02 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:16:02.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:16:02 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 17295 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:16:02.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:16:02 smithi047 ceph-mon[77147]: pgmap v8998: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:16:02.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:16:02 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:16:02.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:16:02 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:16:02.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:16:02 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:16:02.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:16:02 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 17295 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:16:03.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:16:03 smithi115 conmon[90393]: 2022-05-14T04:16:03.534+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:16:03.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:16:03 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:16:03.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:16:03 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:16:04.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:16:04 smithi115 conmon[90393]: 2022-05-14T04:16:04.579+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:16:04.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:16:04 smithi115 ceph-mon[78593]: pgmap v8999: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:16:04.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:16:04 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:16:04.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:16:04 smithi047 ceph-mon[77147]: pgmap v8999: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:16:04.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:16:04 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:16:05.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:16:05 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:16:05.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:16:05 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T04:16:05.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:16:05 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T04:16:05.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:16:05 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T04:16:05.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:16:05 smithi115 conmon[90393]: 2022-05-14T04:16:05.575+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:16:05.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:16:05 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:16:05.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:16:05 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T04:16:05.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:16:05 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T04:16:05.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:16:05 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T04:16:06.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:16:06 smithi047 ceph-mon[77147]: pgmap v9000: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:16:06.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:16:06 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:16:06.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:16:06 smithi115 conmon[90393]: 2022-05-14T04:16:06.577+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:16:06.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:16:06 smithi115 ceph-mon[78593]: pgmap v9000: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:16:06.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:16:06 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:16:07.622 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:16:07 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:16:07.622 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:16:07 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 17300 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:16:07.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:16:07 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:16:07.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:16:07 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 17300 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:16:08.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:16:07 smithi115 conmon[90393]: 2022-05-14T04:16:07.619+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:16:08.369 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:16:08 smithi047 conmon[78521]: 2022-05-14T04:16:08.047+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:16:08.369 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:16:08 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T04:16:08.369 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:16:08 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:16:08.370 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:16:08 smithi047 conmon[78521]: yield 2022-05-14T04:16:08.370 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:16:08 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:16:08.370 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:16:08 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:16:08.370 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:16:08 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:16:08.371 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:16:08 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T04:16:08.371 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:16:08 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:16:08.371 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:16:08 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T04:16:08.371 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:16:08 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:16:08.372 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:16:08 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T04:16:08.372 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:16:08 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:16:08.372 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:16:08 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T04:16:08.372 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:16:08 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:16:08.373 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:16:08 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T04:16:08.373 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:16:08 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:16:08.373 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:16:08 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:16:08.373 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:16:08 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:16:08.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:16:08 smithi047 ceph-mon[77147]: pgmap v9001: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:16:08.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:16:08 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:16:08.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:16:08 smithi115 conmon[90393]: 2022-05-14T04:16:08.600+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:16:08.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:16:08 smithi115 ceph-mon[78593]: pgmap v9001: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:16:08.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:16:08 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:16:09.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:16:09 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:16:09.793 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T04:16:09.793 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:16:09.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T04:16:09.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:16:09.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:16:09.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:16:09.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T04:16:09.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:16:09.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T04:16:09.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:16:09.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T04:16:09.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:16:09.797 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T04:16:09.797 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:16:09.797 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T04:16:09.797 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:16:09.798 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:16:09.798 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:16:09.798 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:16:09 smithi047 ceph-mon[77147]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T04:16:09.798 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:16:09 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:16:09.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:16:09 smithi115 conmon[90393]: 2022-05-14T04:16:09.609+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:16:09.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:16:09 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:16:09.860 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T04:16:09.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:16:09.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T04:16:09.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:16:09.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:16:09.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:16:09.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T04:16:09.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:16:09.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T04:16:09.863 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:16:09.863 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T04:16:09.863 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:16:09.863 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T04:16:09.864 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:16:09.864 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T04:16:09.864 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:16:09.864 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:16:09.864 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:16:09.865 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:16:09 smithi115 ceph-mon[78593]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T04:16:09.865 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:16:09 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:16:10.654 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:16:10 smithi115 ceph-mon[78593]: pgmap v9002: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:16:10.655 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:16:10 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:16:10.655 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:16:10 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:16:10.655 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:16:10 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:16:10.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:16:10 smithi047 ceph-mon[77147]: pgmap v9002: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:16:10.795 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:16:10 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:16:10.795 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:16:10 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:16:10.795 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:16:10 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:16:11.110 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:16:10 smithi115 conmon[90393]: 2022-05-14T04:16:10.652+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:16:11.382 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:16:11 smithi047 conmon[78521]: 2022-05-14T04:16:11.119+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:16:11.382 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:16:11 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T04:16:11.382 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:16:11 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:16:11.382 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:16:11 smithi047 conmon[78521]: yield 2022-05-14T04:16:11.383 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:16:11 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:16:11.383 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:16:11 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:16:11.383 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:16:11 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:16:11.383 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:16:11 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T04:16:11.384 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:16:11 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:16:11.384 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:16:11 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T04:16:11.384 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:16:11 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:16:11.384 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:16:11 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T04:16:11.385 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:16:11 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:16:11.385 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:16:11 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T04:16:11.385 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:16:11 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:16:11.385 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:16:11 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T04:16:11.386 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:16:11 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:16:11.386 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:16:11 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:16:11.386 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:16:11 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:16:11.386 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:16:11 smithi047 conmon[78521]: 2022-05-14T04:16:11.120+0000 7f533ff78700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T04:16:11.387 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:16:11 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T04:16:11.387 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:16:11 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:16:11.387 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:16:11 smithi047 conmon[78521]: yield 2022-05-14T04:16:11.387 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:16:11 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:16:11.387 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:16:11 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:16:11.388 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:16:11 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:16:11.388 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:16:11 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T04:16:11.388 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:16:11 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:16:11.388 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:16:11 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T04:16:11.389 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:16:11 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:16:11.389 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:16:11 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T04:16:11.389 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:16:11 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:16:11.389 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:16:11 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T04:16:11.390 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:16:11 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:16:11.390 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:16:11 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T04:16:11.390 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:16:11 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:16:11.390 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:16:11 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:16:11.391 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:16:11 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:16:11.391 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:16:11 smithi047 conmon[78521]: 2022-05-14T04:16:11.391 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:16:11 smithi047 conmon[78521]: During handling of the above exception, another exception occurred: 2022-05-14T04:16:11.392 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:16:11 smithi047 conmon[78521]: 2022-05-14T04:16:11.392 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:16:11 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T04:16:11.392 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:16:11 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T04:16:11.392 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:16:11 smithi047 conmon[78521]: return f(*arg) 2022-05-14T04:16:11.392 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:16:11 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T04:16:11.393 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:16:11 smithi047 conmon[78521]: self._write_client_files(client_files, host) 2022-05-14T04:16:11.393 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:16:11 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T04:16:11.393 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:16:11 smithi047 conmon[78521]: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T04:16:11.393 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:16:11 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T04:16:11.394 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:16:11 smithi047 conmon[78521]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T04:16:11.394 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:16:11 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T04:16:11.394 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:16:11 smithi047 conmon[78521]: return self.event_loop.get_result(coro) 2022-05-14T04:16:11.394 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:16:11 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T04:16:11.394 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:16:11 smithi047 conmon[78521]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T04:16:11.395 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:16:11 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T04:16:11.395 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:16:11 smithi047 conmon[78521]: return self.__get_result() 2022-05-14T04:16:11.395 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:16:11 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T04:16:11.395 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:16:11 smithi047 conmon[78521]: raise self._exception 2022-05-14T04:16:11.395 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:16:11 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T04:16:11.396 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:16:11 smithi047 conmon[78521]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T04:16:11.396 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:16:11 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T04:16:11.396 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:16:11 smithi047 conmon[78521]: conn = await self._remote_connection(host, addr) 2022-05-14T04:16:11.396 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:16:11 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T04:16:11.396 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:16:11 smithi047 conmon[78521]: raise 2022-05-14T04:16:11.397 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:16:11 smithi047 conmon[78521]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T04:16:11.397 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:16:11 smithi047 conmon[78521]: self.gen.throw(type, value, traceback) 2022-05-14T04:16:11.397 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:16:11 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T04:16:11.397 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:16:11 smithi047 conmon[78521]: raise OrchestratorError(msg) 2022-05-14T04:16:11.398 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:16:11 smithi047 conmon[78521]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:16:11.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:16:11 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:16:11.860 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:16:11 smithi115 conmon[90393]: 2022-05-14T04:16:11.615+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:16:11.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:16:11 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:16:12.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:16:12 smithi047 ceph-mon[77147]: pgmap v9003: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:16:12.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:16:12 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:16:12.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T04:16:12.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:16:12.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T04:16:12.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:16:12.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:16:12.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:16:12.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T04:16:12.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:16:12.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T04:16:12.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:16:12.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T04:16:12.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:16:12.797 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T04:16:12.797 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:16:12.797 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T04:16:12.797 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:16:12.798 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:16:12.798 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:16:12.798 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:16:12 smithi047 ceph-mon[77147]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T04:16:12.798 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T04:16:12.799 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:16:12.799 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T04:16:12.799 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:16:12.799 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:16:12.799 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:16:12.800 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T04:16:12.800 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:16:12.800 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T04:16:12.800 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:16:12.800 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T04:16:12.801 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:16:12.801 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T04:16:12.801 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:16:12.801 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T04:16:12.802 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:16:12.802 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:16:12.802 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:16:12.802 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T04:16:12.802 INFO:journalctl@ceph.mon.b.smithi047.stdout: During handling of the above exception, another exception occurred: 2022-05-14T04:16:12.803 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T04:16:12.803 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T04:16:12.803 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T04:16:12.804 INFO:journalctl@ceph.mon.b.smithi047.stdout: return f(*arg) 2022-05-14T04:16:12.804 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T04:16:12.804 INFO:journalctl@ceph.mon.b.smithi047.stdout: self._write_client_files(client_files, host) 2022-05-14T04:16:12.804 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T04:16:12.804 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T04:16:12.805 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T04:16:12.805 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T04:16:12.805 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T04:16:12.805 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.event_loop.get_result(coro) 2022-05-14T04:16:12.805 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T04:16:12.806 INFO:journalctl@ceph.mon.b.smithi047.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T04:16:12.806 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T04:16:12.806 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.__get_result() 2022-05-14T04:16:12.807 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T04:16:12.807 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise self._exception 2022-05-14T04:16:12.807 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T04:16:12.807 INFO:journalctl@ceph.mon.b.smithi047.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T04:16:12.808 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T04:16:12.808 INFO:journalctl@ceph.mon.b.smithi047.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T04:16:12.808 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T04:16:12.809 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise 2022-05-14T04:16:12.809 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T04:16:12.809 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.gen.throw(type, value, traceback) 2022-05-14T04:16:12.809 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T04:16:12.809 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OrchestratorError(msg) 2022-05-14T04:16:12.810 INFO:journalctl@ceph.mon.b.smithi047.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:16:12.810 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:16:12 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:16:12.810 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:16:12 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 17305 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:16:12.860 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:16:12 smithi115 conmon[90393]: 2022-05-14T04:16:12.618+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:16:12.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:16:12 smithi115 ceph-mon[78593]: pgmap v9003: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:16:12.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:16:12 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:16:12.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T04:16:12.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:16:12.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T04:16:12.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:16:12.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:16:12.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:16:12.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T04:16:12.863 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:16:12.863 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T04:16:12.863 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:16:12.864 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T04:16:12.864 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:16:12.864 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T04:16:12.865 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:16:12.865 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T04:16:12.866 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:16:12.866 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:16:12.866 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:16:12.866 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:16:12 smithi115 ceph-mon[78593]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T04:16:12.867 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T04:16:12.867 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:16:12.867 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T04:16:12.867 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:16:12.868 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:16:12.868 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:16:12.868 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T04:16:12.868 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:16:12.868 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T04:16:12.869 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:16:12.869 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T04:16:12.869 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:16:12.869 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T04:16:12.870 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:16:12.870 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T04:16:12.870 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:16:12.870 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:16:12.870 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:16:12.871 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T04:16:12.871 INFO:journalctl@ceph.mon.c.smithi115.stdout: During handling of the above exception, another exception occurred: 2022-05-14T04:16:12.871 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T04:16:12.872 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T04:16:12.872 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T04:16:12.872 INFO:journalctl@ceph.mon.c.smithi115.stdout: return f(*arg) 2022-05-14T04:16:12.872 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T04:16:12.872 INFO:journalctl@ceph.mon.c.smithi115.stdout: self._write_client_files(client_files, host) 2022-05-14T04:16:12.873 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T04:16:12.873 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T04:16:12.873 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T04:16:12.873 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T04:16:12.873 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T04:16:12.874 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.event_loop.get_result(coro) 2022-05-14T04:16:12.874 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T04:16:12.874 INFO:journalctl@ceph.mon.c.smithi115.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T04:16:12.875 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T04:16:12.875 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.__get_result() 2022-05-14T04:16:12.875 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T04:16:12.875 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise self._exception 2022-05-14T04:16:12.876 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T04:16:12.876 INFO:journalctl@ceph.mon.c.smithi115.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T04:16:12.876 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T04:16:12.876 INFO:journalctl@ceph.mon.c.smithi115.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T04:16:12.877 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T04:16:12.877 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise 2022-05-14T04:16:12.877 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T04:16:12.877 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.gen.throw(type, value, traceback) 2022-05-14T04:16:12.878 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T04:16:12.878 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OrchestratorError(msg) 2022-05-14T04:16:12.878 INFO:journalctl@ceph.mon.c.smithi115.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:16:12.878 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:16:12 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:16:12.878 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:16:12 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 17305 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:16:13.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:16:13 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:16:13.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:16:13 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:16:13.860 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:16:13 smithi115 conmon[90393]: 2022-05-14T04:16:13.574+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:16:14.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:16:14 smithi047 ceph-mon[77147]: pgmap v9004: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:16:14.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:16:14 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:16:14.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:16:14 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:16:14.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:16:14 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:16:14.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:16:14 smithi115 conmon[90393]: 2022-05-14T04:16:14.624+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:16:14.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:16:14 smithi115 ceph-mon[78593]: pgmap v9004: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:16:14.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:16:14 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:16:14.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:16:14 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:16:14.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:16:14 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:16:15.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:16:15 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T04:16:15.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:16:15 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T04:16:15.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:16:15 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:16:15.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:16:15 smithi115 conmon[90393]: 2022-05-14T04:16:15.632+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:16:15.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:16:15 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T04:16:15.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:16:15 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T04:16:15.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:16:15 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:16:16.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:16:16 smithi047 ceph-mon[77147]: pgmap v9005: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:16:16.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:16:16 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:16:16.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:16:16 smithi115 conmon[90393]: 2022-05-14T04:16:16.614+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:16:16.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:16:16 smithi115 ceph-mon[78593]: pgmap v9005: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:16:16.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:16:16 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:16:17.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:16:17 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:16:17.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:16:17 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 17310 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:16:17.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:16:17 smithi115 conmon[90393]: 2022-05-14T04:16:17.602+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:16:17.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:16:17 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:16:17.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:16:17 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 17310 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:16:18.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:16:18 smithi047 ceph-mon[77147]: pgmap v9006: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:16:18.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:16:18 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:16:18.795 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:16:18 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:16:18.795 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:16:18 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:16:18.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:16:18 smithi115 conmon[90393]: 2022-05-14T04:16:18.594+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:16:18.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:16:18 smithi115 ceph-mon[78593]: pgmap v9006: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:16:18.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:16:18 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:16:18.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:16:18 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:16:18.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:16:18 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:16:19.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:16:19 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:16:19.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:16:19 smithi115 conmon[90393]: 2022-05-14T04:16:19.552+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:16:19.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:16:19 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:16:20.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:16:20 smithi047 ceph-mon[77147]: pgmap v9007: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:16:20.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:16:20 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:16:20.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:16:20 smithi115 conmon[90393]: 2022-05-14T04:16:20.582+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:16:20.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:16:20 smithi115 ceph-mon[78593]: pgmap v9007: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:16:20.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:16:20 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:16:21.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:16:21 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:16:21.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:16:21 smithi115 conmon[90393]: 2022-05-14T04:16:21.623+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:16:21.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:16:21 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:16:22.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:16:22 smithi047 ceph-mon[77147]: pgmap v9008: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:16:22.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:16:22 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:16:22.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:16:22 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:16:22.795 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:16:22 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:16:22.795 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:16:22 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 17315 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:16:22.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:16:22 smithi115 conmon[90393]: 2022-05-14T04:16:22.623+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:16:22.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:16:22 smithi115 ceph-mon[78593]: pgmap v9008: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:16:22.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:16:22 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:16:22.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:16:22 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:16:22.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:16:22 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:16:22.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:16:22 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 17315 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:16:23.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:16:23 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:16:23.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:16:23 smithi115 conmon[90393]: 2022-05-14T04:16:23.670+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:16:23.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:16:23 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:16:24.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:16:24 smithi047 ceph-mon[77147]: pgmap v9009: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:16:24.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:16:24 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:16:24.860 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:16:24 smithi115 conmon[90393]: 2022-05-14T04:16:24.664+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:16:24.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:16:24 smithi115 ceph-mon[78593]: pgmap v9009: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:16:24.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:16:24 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:16:25.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:16:25 smithi047 ceph-mon[77147]: pgmap v9010: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:16:25.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:16:25 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:16:25.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:16:25 smithi115 conmon[90393]: 2022-05-14T04:16:25.649+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:16:25.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:16:25 smithi115 ceph-mon[78593]: pgmap v9010: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:16:25.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:16:25 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:16:26.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:16:26 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:16:26.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:16:26 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:16:26.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:16:26 smithi115 conmon[90393]: 2022-05-14T04:16:26.619+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:16:27.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:16:27 smithi047 ceph-mon[77147]: pgmap v9011: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:16:27.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:16:27 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:16:27.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:16:27 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 17320 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:16:27.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:16:27 smithi115 conmon[90393]: 2022-05-14T04:16:27.634+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:16:27.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:16:27 smithi115 ceph-mon[78593]: pgmap v9011: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:16:27.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:16:27 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:16:27.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:16:27 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 17320 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:16:28.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:16:28 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:16:28.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:16:28 smithi115 conmon[90393]: 2022-05-14T04:16:28.650+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:16:28.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:16:28 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:16:29.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:16:29 smithi047 ceph-mon[77147]: pgmap v9012: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:16:29.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:16:29 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:16:29.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:16:29 smithi115 conmon[90393]: 2022-05-14T04:16:29.690+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:16:29.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:16:29 smithi115 ceph-mon[78593]: pgmap v9012: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:16:29.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:16:29 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:16:30.722 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:16:30 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:16:30.722 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:16:30 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:16:30.722 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:16:30 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:16:30.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:16:30 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:16:30.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:16:30 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:16:30.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:16:30 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:16:31.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:16:30 smithi115 conmon[90393]: 2022-05-14T04:16:30.718+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:16:31.758 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:16:31 smithi115 ceph-mon[78593]: pgmap v9013: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:16:31.758 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:16:31 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:16:31.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:16:31 smithi047 ceph-mon[77147]: pgmap v9013: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:16:31.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:16:31 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:16:32.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:16:31 smithi115 conmon[90393]: 2022-05-14T04:16:31.755+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:16:32.734 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:16:32 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:16:32.734 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:16:32 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 17325 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:16:32.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:16:32 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:16:32.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:16:32 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 17325 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:16:33.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:16:32 smithi115 conmon[90393]: 2022-05-14T04:16:32.732+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:16:33.763 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:16:33 smithi115 ceph-mon[78593]: pgmap v9014: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:16:33.764 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:16:33 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:16:33.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:16:33 smithi047 ceph-mon[77147]: pgmap v9014: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:16:33.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:16:33 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:16:34.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:16:33 smithi115 conmon[90393]: 2022-05-14T04:16:33.760+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:16:34.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:16:34 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:16:34.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:16:34 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:16:34.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:16:34 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:16:34.801 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:16:34 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:16:34.802 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:16:34 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:16:34.802 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:16:34 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:16:35.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:16:34 smithi115 conmon[90393]: 2022-05-14T04:16:34.798+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:16:35.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:16:35 smithi047 ceph-mon[77147]: pgmap v9015: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:16:35.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:16:35 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:16:35.802 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:16:35 smithi115 ceph-mon[78593]: pgmap v9015: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:16:35.802 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:16:35 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:16:36.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:16:35 smithi115 conmon[90393]: 2022-05-14T04:16:35.799+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:16:36.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:16:36 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:16:36.849 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:16:36 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:16:37.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:16:36 smithi115 conmon[90393]: 2022-05-14T04:16:36.846+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:16:37.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:16:37 smithi047 ceph-mon[77147]: pgmap v9016: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:16:37.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:16:37 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:16:37.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:16:37 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 17330 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:16:37.856 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:16:37 smithi115 ceph-mon[78593]: pgmap v9016: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:16:37.856 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:16:37 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:16:37.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:16:37 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 17330 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:16:38.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:16:37 smithi115 conmon[90393]: 2022-05-14T04:16:37.853+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:16:38.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:16:38 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:16:38.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:16:38 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:16:38.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:16:38 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:16:38.849 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:16:38 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:16:38.849 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:16:38 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:16:38.850 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:16:38 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:16:39.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:16:38 smithi115 conmon[90393]: 2022-05-14T04:16:38.846+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:16:39.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:16:39 smithi047 ceph-mon[77147]: pgmap v9017: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:16:39.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:16:39 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:16:39.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:16:39 smithi115 ceph-mon[78593]: pgmap v9017: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:16:39.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:16:39 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:16:40.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:16:39 smithi115 conmon[90393]: 2022-05-14T04:16:39.854+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:16:40.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:16:40 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:16:40.845 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:16:40 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:16:41.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:16:40 smithi115 conmon[90393]: 2022-05-14T04:16:40.841+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:16:41.800 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:16:41 smithi047 ceph-mon[77147]: pgmap v9018: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:16:41.801 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:16:41 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:16:41.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:16:41 smithi115 ceph-mon[78593]: pgmap v9018: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:16:41.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:16:41 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:16:42.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:16:41 smithi115 conmon[90393]: 2022-05-14T04:16:41.881+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:16:42.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:16:42 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:16:42.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:16:42 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:16:42.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:16:42 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:16:42.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:16:42 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 17335 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:16:42.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:16:42 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:16:42.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:16:42 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:16:42.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:16:42 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:16:42.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:16:42 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 17335 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:16:43.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:16:42 smithi115 conmon[90393]: 2022-05-14T04:16:42.865+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:16:43.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:16:43 smithi047 ceph-mon[77147]: pgmap v9019: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:16:43.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:16:43 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:16:43.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:16:43 smithi115 ceph-mon[78593]: pgmap v9019: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:16:43.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:16:43 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:16:44.360 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:16:43 smithi115 conmon[90393]: 2022-05-14T04:16:43.859+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:16:44.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:16:44 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:16:44.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:16:44 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:16:45.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:16:44 smithi115 conmon[90393]: 2022-05-14T04:16:44.861+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:16:45.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:16:45 smithi047 ceph-mon[77147]: pgmap v9020: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:16:45.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:16:45 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:16:45.829 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:16:45 smithi115 ceph-mon[78593]: pgmap v9020: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:16:45.829 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:16:45 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:16:46.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:16:45 smithi115 conmon[90393]: 2022-05-14T04:16:45.825+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:16:46.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:16:46 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:16:46.837 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:16:46 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:16:47.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:16:46 smithi115 conmon[90393]: 2022-05-14T04:16:46.834+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:16:47.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:16:47 smithi047 ceph-mon[77147]: pgmap v9021: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:16:47.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:16:47 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:16:47.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:16:47 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 17340 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:16:47.819 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:16:47 smithi115 ceph-mon[78593]: pgmap v9021: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:16:47.819 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:16:47 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:16:47.820 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:16:47 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 17340 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:16:48.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:16:47 smithi115 conmon[90393]: 2022-05-14T04:16:47.816+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:16:48.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:16:48 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:16:48.851 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:16:48 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:16:49.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:16:48 smithi115 conmon[90393]: 2022-05-14T04:16:48.848+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:16:49.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:16:49 smithi047 ceph-mon[77147]: pgmap v9022: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:16:49.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:16:49 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:16:49.835 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:16:49 smithi115 ceph-mon[78593]: pgmap v9022: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:16:49.836 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:16:49 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:16:50.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:16:49 smithi115 conmon[90393]: 2022-05-14T04:16:49.832+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:16:50.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:16:50 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:16:50.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:16:50 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:16:50.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:16:50 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:16:50.818 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:16:50 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:16:50.819 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:16:50 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:16:50.819 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:16:50 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:16:51.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:16:50 smithi115 conmon[90393]: 2022-05-14T04:16:50.816+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:16:51.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:16:51 smithi047 ceph-mon[77147]: pgmap v9023: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:16:51.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:16:51 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:16:51.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:16:51 smithi115 ceph-mon[78593]: pgmap v9023: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:16:51.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:16:51 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:16:52.110 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:16:51 smithi115 conmon[90393]: 2022-05-14T04:16:51.855+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:16:52.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:16:52 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:16:52.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:16:52 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 17345 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:16:52.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:16:52 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:16:52.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:16:52 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 17345 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:16:53.360 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:16:52 smithi115 conmon[90393]: 2022-05-14T04:16:52.894+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:16:53.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:16:53 smithi047 ceph-mon[77147]: pgmap v9024: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:16:53.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:16:53 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:16:53.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:16:53 smithi115 ceph-mon[78593]: pgmap v9024: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:16:53.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:16:53 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:16:54.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:16:53 smithi115 conmon[90393]: 2022-05-14T04:16:53.903+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:16:54.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:16:54 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:16:54.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:16:54 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:16:54.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:16:54 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:16:54.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:16:54 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:16:54.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:16:54 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:16:54.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:16:54 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:16:55.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:16:54 smithi115 conmon[90393]: 2022-05-14T04:16:54.879+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:16:55.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:16:55 smithi047 ceph-mon[77147]: pgmap v9025: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:16:55.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:16:55 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:16:55.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:16:55 smithi115 ceph-mon[78593]: pgmap v9025: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:16:55.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:16:55 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:16:56.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:16:55 smithi115 conmon[90393]: 2022-05-14T04:16:55.910+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:16:56.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:16:56 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:16:56.863 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:16:56 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:16:57.360 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:16:56 smithi115 conmon[90393]: 2022-05-14T04:16:56.949+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:16:57.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:16:57 smithi115 ceph-mon[78593]: pgmap v9026: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:16:57.862 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:16:57 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:16:57.862 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:16:57 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 17350 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:16:58.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:16:57 smithi047 ceph-mon[77147]: pgmap v9026: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:16:58.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:16:57 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:16:58.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:16:57 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 17350 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:16:58.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:16:57 smithi115 conmon[90393]: 2022-05-14T04:16:57.963+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:16:58.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:16:58 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:16:58.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:16:58 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:16:58.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:16:58 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:16:59.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:16:58 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:16:59.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:16:58 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:16:59.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:16:58 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:16:59.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:16:58 smithi115 conmon[90393]: 2022-05-14T04:16:58.993+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:16:59.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:16:59 smithi115 ceph-mon[78593]: pgmap v9027: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:16:59.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:16:59 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:17:00.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:16:59 smithi047 ceph-mon[77147]: pgmap v9027: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:17:00.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:16:59 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:17:00.360 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:16:59 smithi115 conmon[90393]: 2022-05-14T04:16:59.973+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:17:00.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:17:00 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:17:01.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:17:00 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:17:01.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:17:00 smithi115 conmon[90393]: 2022-05-14T04:17:00.982+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:17:01.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:17:01 smithi115 ceph-mon[78593]: pgmap v9028: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:17:01.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:17:01 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:17:02.045 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:17:01 smithi047 ceph-mon[77147]: pgmap v9028: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:17:02.045 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:17:01 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:17:02.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:17:01 smithi115 conmon[90393]: 2022-05-14T04:17:01.997+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:17:02.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:17:02 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:17:02.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:17:02 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:17:02.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:17:02 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:17:02.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:17:02 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 17355 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:17:03.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:17:02 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:17:03.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:17:02 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:17:03.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:17:02 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:17:03.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:17:02 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 17355 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:17:03.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:17:02 smithi115 conmon[90393]: 2022-05-14T04:17:02.987+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:17:04.277 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:17:03 smithi115 conmon[90393]: 2022-05-14T04:17:03.985+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:17:04.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:17:04 smithi047 ceph-mon[77147]: pgmap v9029: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:17:04.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:17:04 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:17:04.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:17:04 smithi115 ceph-mon[78593]: pgmap v9029: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:17:04.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:17:04 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:17:05.284 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:17:05 smithi115 conmon[90393]: 2022-05-14T04:17:05.013+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:17:05.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:17:05 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:17:05.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:17:05 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:17:06.283 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:17:05 smithi115 conmon[90393]: 2022-05-14T04:17:05.995+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:17:06.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:17:06 smithi047 ceph-mon[77147]: pgmap v9030: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:17:06.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:17:06 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:17:06.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:17:06 smithi115 ceph-mon[78593]: pgmap v9030: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:17:06.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:17:06 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:17:07.292 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:17:06 smithi115 conmon[90393]: 2022-05-14T04:17:06.976+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:17:07.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:17:07 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:17:07.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:17:07 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 17360 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:17:07.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:17:07 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:17:07.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:17:07 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 17360 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:17:08.292 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:17:07 smithi115 conmon[90393]: 2022-05-14T04:17:07.931+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:17:08.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:17:08 smithi047 ceph-mon[77147]: pgmap v9031: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:17:08.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:17:08 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:17:08.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:17:08 smithi115 ceph-mon[78593]: pgmap v9031: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:17:08.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:17:08 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:17:09.297 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:17:08 smithi115 conmon[90393]: 2022-05-14T04:17:08.925+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:17:09.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:17:09 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:17:09.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:17:09 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:17:10.298 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:17:09 smithi115 conmon[90393]: 2022-05-14T04:17:09.908+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:17:10.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:17:10 smithi115 ceph-mon[78593]: pgmap v9032: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:17:10.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:17:10 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:17:10.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:17:10 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:17:10.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:17:10 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:17:10.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:17:10 smithi047 ceph-mon[77147]: pgmap v9032: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:17:10.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:17:10 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:17:10.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:17:10 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:17:10.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:17:10 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:17:11.170 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:17:10 smithi115 conmon[90393]: 2022-05-14T04:17:10.906+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:17:11.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:17:11 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:17:11.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:17:11 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T04:17:11.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:17:11 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T04:17:11.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:17:11 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T04:17:11.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:17:11 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:17:11.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:17:11 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T04:17:11.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:17:11 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T04:17:11.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:17:11 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T04:17:12.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:17:11 smithi047 conmon[78521]: 2022-05-14T04:17:11.727+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:17:12.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:17:11 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T04:17:12.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:17:11 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:17:12.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:17:11 smithi047 conmon[78521]: yield 2022-05-14T04:17:12.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:17:11 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:17:12.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:17:11 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:17:12.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:17:11 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:17:12.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:17:11 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T04:17:12.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:17:11 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:17:12.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:17:11 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T04:17:12.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:17:11 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:17:12.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:17:11 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T04:17:12.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:17:11 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:17:12.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:17:11 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T04:17:12.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:17:11 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:17:12.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:17:11 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T04:17:12.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:17:11 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:17:12.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:17:11 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:17:12.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:17:11 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:17:12.312 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:17:11 smithi115 conmon[90393]: 2022-05-14T04:17:11.933+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:17:12.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:17:12 smithi115 ceph-mon[78593]: pgmap v9033: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:17:12.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:17:12 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:17:12.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:17:12 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 17365 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:17:12.710 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:17:12 smithi047 ceph-mon[77147]: pgmap v9033: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:17:12.710 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:17:12 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:17:12.710 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:17:12 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 17365 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:17:13.106 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:17:12 smithi115 conmon[90393]: 2022-05-14T04:17:12.903+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:17:13.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:17:13 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:17:13.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T04:17:13.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:17:13.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T04:17:13.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:17:13.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:17:13.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:17:13.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T04:17:13.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:17:13.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T04:17:13.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:17:13.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T04:17:13.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:17:13.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T04:17:13.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:17:13.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T04:17:13.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:17:13.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:17:13.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:17:13.364 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:17:13 smithi115 ceph-mon[78593]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T04:17:13.364 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:17:13 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:17:13.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:17:13 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:17:13.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T04:17:13.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:17:13.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T04:17:13.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:17:13.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:17:13.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:17:13.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T04:17:13.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:17:13.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T04:17:13.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:17:13.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T04:17:13.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:17:13.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T04:17:13.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:17:13.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T04:17:13.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:17:13.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:17:13.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:17:13.548 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:17:13 smithi047 ceph-mon[77147]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T04:17:13.548 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:17:13 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:17:14.274 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:17:13 smithi115 conmon[90393]: 2022-05-14T04:17:13.912+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:17:14.393 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:17:14 smithi047 ceph-mon[77147]: pgmap v9034: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:17:14.393 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:17:14 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:17:14.393 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:17:14 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:17:14.394 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:17:14 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:17:14.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:17:14 smithi115 ceph-mon[78593]: pgmap v9034: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:17:14.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:17:14 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:17:14.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:17:14 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:17:14.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:17:14 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:17:15.293 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:17:14 smithi047 conmon[78521]: 2022-05-14T04:17:14.799+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:17:15.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:17:14 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T04:17:15.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:17:14 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:17:15.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:17:14 smithi047 conmon[78521]: yield 2022-05-14T04:17:15.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:17:14 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:17:15.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:17:14 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:17:15.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:17:14 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:17:15.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:17:14 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T04:17:15.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:17:14 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:17:15.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:17:14 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T04:17:15.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:17:14 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:17:15.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:17:14 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T04:17:15.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:17:14 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:17:15.297 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:17:14 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T04:17:15.297 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:17:14 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:17:15.297 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:17:14 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T04:17:15.297 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:17:14 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:17:15.298 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:17:14 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:17:15.298 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:17:14 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:17:15.298 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:17:14 smithi047 conmon[78521]: 2022-05-14T04:17:14.800+0000 7f5344f82700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T04:17:15.298 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:17:14 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T04:17:15.298 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:17:14 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:17:15.299 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:17:14 smithi047 conmon[78521]: yield 2022-05-14T04:17:15.299 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:17:14 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:17:15.299 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:17:14 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:17:15.299 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:17:14 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:17:15.300 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:17:14 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T04:17:15.301 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:17:14 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:17:15.302 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:17:14 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T04:17:15.302 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:17:14 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:17:15.302 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:17:14 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T04:17:15.302 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:17:14 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:17:15.303 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:17:14 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T04:17:15.303 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:17:14 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:17:15.303 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:17:14 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T04:17:15.303 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:17:14 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:17:15.303 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:17:14 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:17:15.304 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:17:14 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:17:15.304 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:17:14 smithi047 conmon[78521]: 2022-05-14T04:17:15.304 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:17:14 smithi047 conmon[78521]: During handling of the above exception, another exception occurred: 2022-05-14T04:17:15.304 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:17:14 smithi047 conmon[78521]: 2022-05-14T04:17:15.305 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:17:14 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T04:17:15.305 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:17:14 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T04:17:15.305 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:17:14 smithi047 conmon[78521]: return f(*arg) 2022-05-14T04:17:15.305 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:17:14 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T04:17:15.306 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:17:14 smithi047 conmon[78521]: self._write_client_files(client_files, host) 2022-05-14T04:17:15.306 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:17:14 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T04:17:15.306 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:17:14 smithi047 conmon[78521]: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T04:17:15.306 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:17:14 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T04:17:15.307 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:17:14 smithi047 conmon[78521]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T04:17:15.307 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:17:14 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T04:17:15.307 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:17:14 smithi047 conmon[78521]: return self.event_loop.get_result(coro) 2022-05-14T04:17:15.307 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:17:14 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T04:17:15.307 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:17:14 smithi047 conmon[78521]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T04:17:15.308 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:17:14 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T04:17:15.308 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:17:14 smithi047 conmon[78521]: return self.__get_result() 2022-05-14T04:17:15.308 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:17:14 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T04:17:15.309 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:17:14 smithi047 conmon[78521]: raise self._exception 2022-05-14T04:17:15.309 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:17:14 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T04:17:15.309 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:17:14 smithi047 conmon[78521]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T04:17:15.309 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:17:14 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T04:17:15.309 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:17:14 smithi047 conmon[78521]: conn = await self._remote_connection(host, addr) 2022-05-14T04:17:15.310 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:17:14 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T04:17:15.310 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:17:14 smithi047 conmon[78521]: raise 2022-05-14T04:17:15.310 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:17:14 smithi047 conmon[78521]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T04:17:15.310 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:17:14 smithi047 conmon[78521]: self.gen.throw(type, value, traceback) 2022-05-14T04:17:15.311 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:17:14 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T04:17:15.311 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:17:14 smithi047 conmon[78521]: raise OrchestratorError(msg) 2022-05-14T04:17:15.311 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:17:14 smithi047 conmon[78521]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:17:15.318 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:17:14 smithi115 conmon[90393]: 2022-05-14T04:17:14.960+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:17:15.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:17:15 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:17:15.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:17:15 smithi115 ceph-mon[78593]: Detected new or changed devices on smithi115 2022-05-14T04:17:15.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:17:15 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' 2022-05-14T04:17:15.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:17:15 smithi115 ceph-mon[78593]: Detected new or changed devices on smithi047 2022-05-14T04:17:15.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:17:15 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' 2022-05-14T04:17:15.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:17:15 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T04:17:15.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:17:15 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T04:17:15.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:17:15 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:17:15.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:17:15 smithi047 ceph-mon[77147]: Detected new or changed devices on smithi115 2022-05-14T04:17:15.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:17:15 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' 2022-05-14T04:17:15.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:17:15 smithi047 ceph-mon[77147]: Detected new or changed devices on smithi047 2022-05-14T04:17:15.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:17:15 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' 2022-05-14T04:17:15.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:17:15 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T04:17:15.795 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:17:15 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T04:17:16.320 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:17:15 smithi115 conmon[90393]: 2022-05-14T04:17:15.963+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:17:16.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:17:16 smithi115 ceph-mon[78593]: pgmap v9035: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:17:16.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:17:16 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:17:16.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T04:17:16.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:17:16.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T04:17:16.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:17:16.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:17:16.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:17:16.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T04:17:16.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:17:16.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T04:17:16.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:17:16.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T04:17:16.613 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:17:16.613 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T04:17:16.613 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:17:16.613 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T04:17:16.614 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:17:16.614 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:17:16.614 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:17:16.614 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:17:16 smithi115 ceph-mon[78593]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T04:17:16.614 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T04:17:16.615 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:17:16.615 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T04:17:16.615 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:17:16.616 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:17:16.616 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:17:16.616 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T04:17:16.616 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:17:16.617 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T04:17:16.617 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:17:16.617 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T04:17:16.617 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:17:16.618 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T04:17:16.618 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:17:16.618 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T04:17:16.618 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:17:16.619 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:17:16.619 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:17:16.619 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T04:17:16.620 INFO:journalctl@ceph.mon.c.smithi115.stdout: During handling of the above exception, another exception occurred: 2022-05-14T04:17:16.620 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T04:17:16.620 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T04:17:16.620 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T04:17:16.620 INFO:journalctl@ceph.mon.c.smithi115.stdout: return f(*arg) 2022-05-14T04:17:16.621 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T04:17:16.621 INFO:journalctl@ceph.mon.c.smithi115.stdout: self._write_client_files(client_files, host) 2022-05-14T04:17:16.621 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T04:17:16.621 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T04:17:16.622 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T04:17:16.622 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T04:17:16.622 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T04:17:16.622 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.event_loop.get_result(coro) 2022-05-14T04:17:16.622 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T04:17:16.623 INFO:journalctl@ceph.mon.c.smithi115.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T04:17:16.623 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T04:17:16.623 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.__get_result() 2022-05-14T04:17:16.623 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T04:17:16.624 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise self._exception 2022-05-14T04:17:16.624 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T04:17:16.624 INFO:journalctl@ceph.mon.c.smithi115.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T04:17:16.624 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T04:17:16.625 INFO:journalctl@ceph.mon.c.smithi115.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T04:17:16.625 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T04:17:16.625 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise 2022-05-14T04:17:16.625 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T04:17:16.626 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.gen.throw(type, value, traceback) 2022-05-14T04:17:16.626 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T04:17:16.626 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OrchestratorError(msg) 2022-05-14T04:17:16.627 INFO:journalctl@ceph.mon.c.smithi115.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:17:16.627 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:17:16 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:17:16.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:17:16 smithi047 ceph-mon[77147]: pgmap v9035: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:17:16.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:17:16 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:17:16.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T04:17:16.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:17:16.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T04:17:16.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:17:16.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:17:16.797 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:17:16.797 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T04:17:16.797 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:17:16.797 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T04:17:16.798 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:17:16.798 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T04:17:16.798 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:17:16.798 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T04:17:16.799 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:17:16.799 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T04:17:16.799 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:17:16.799 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:17:16.799 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:17:16.800 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:17:16 smithi047 ceph-mon[77147]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T04:17:16.800 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T04:17:16.800 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:17:16.800 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T04:17:16.801 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:17:16.801 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:17:16.801 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:17:16.801 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T04:17:16.802 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:17:16.802 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T04:17:16.802 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:17:16.803 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T04:17:16.803 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:17:16.803 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T04:17:16.803 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:17:16.804 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T04:17:16.804 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:17:16.804 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:17:16.804 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:17:16.805 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T04:17:16.805 INFO:journalctl@ceph.mon.b.smithi047.stdout: During handling of the above exception, another exception occurred: 2022-05-14T04:17:16.805 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T04:17:16.806 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T04:17:16.806 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T04:17:16.806 INFO:journalctl@ceph.mon.b.smithi047.stdout: return f(*arg) 2022-05-14T04:17:16.807 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T04:17:16.807 INFO:journalctl@ceph.mon.b.smithi047.stdout: self._write_client_files(client_files, host) 2022-05-14T04:17:16.807 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T04:17:16.808 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T04:17:16.808 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T04:17:16.808 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T04:17:16.809 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T04:17:16.809 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.event_loop.get_result(coro) 2022-05-14T04:17:16.809 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T04:17:16.810 INFO:journalctl@ceph.mon.b.smithi047.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T04:17:16.810 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T04:17:16.810 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.__get_result() 2022-05-14T04:17:16.810 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T04:17:16.811 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise self._exception 2022-05-14T04:17:16.811 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T04:17:16.811 INFO:journalctl@ceph.mon.b.smithi047.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T04:17:16.812 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T04:17:16.812 INFO:journalctl@ceph.mon.b.smithi047.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T04:17:16.812 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T04:17:16.813 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise 2022-05-14T04:17:16.814 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T04:17:16.814 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.gen.throw(type, value, traceback) 2022-05-14T04:17:16.814 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T04:17:16.814 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OrchestratorError(msg) 2022-05-14T04:17:16.815 INFO:journalctl@ceph.mon.b.smithi047.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:17:16.815 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:17:16 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:17:17.324 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:17:17 smithi115 conmon[90393]: 2022-05-14T04:17:17.001+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:17:17.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:17:17 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:17:17.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:17:17 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 17370 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:17:17.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:17:17 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:17:17.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:17:17 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 17370 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:17:18.326 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:17:18 smithi115 conmon[90393]: 2022-05-14T04:17:18.021+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:17:18.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:17:18 smithi115 ceph-mon[78593]: pgmap v9036: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:17:18.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:17:18 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:17:18.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:17:18 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:17:18.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:17:18 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:17:18.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:17:18 smithi047 ceph-mon[77147]: pgmap v9036: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:17:18.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:17:18 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:17:18.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:17:18 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:17:18.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:17:18 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:17:19.330 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:17:19 smithi115 conmon[90393]: 2022-05-14T04:17:19.035+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:17:19.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:17:19 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:17:19.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:17:19 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:17:20.332 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:17:19 smithi115 conmon[90393]: 2022-05-14T04:17:19.995+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:17:20.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:17:20 smithi115 ceph-mon[78593]: pgmap v9037: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:17:20.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:17:20 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:17:20.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:17:20 smithi047 ceph-mon[77147]: pgmap v9037: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:17:20.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:17:20 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:17:21.338 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:17:20 smithi115 conmon[90393]: 2022-05-14T04:17:20.985+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:17:21.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:17:21 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:17:21.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:17:21 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:17:22.339 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:17:21 smithi115 conmon[90393]: 2022-05-14T04:17:21.956+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:17:22.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:17:22 smithi115 ceph-mon[78593]: pgmap v9038: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:17:22.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:17:22 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:17:22.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:17:22 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:17:22.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:17:22 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:17:22.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:17:22 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 17375 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:17:22.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:17:22 smithi047 ceph-mon[77147]: pgmap v9038: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:17:22.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:17:22 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:17:22.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:17:22 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:17:22.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:17:22 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:17:22.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:17:22 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 17375 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:17:23.346 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:17:22 smithi115 conmon[90393]: 2022-05-14T04:17:22.987+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:17:23.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:17:23 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:17:23.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:17:23 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:17:24.354 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:17:23 smithi115 conmon[90393]: 2022-05-14T04:17:23.947+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:17:24.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:17:24 smithi115 ceph-mon[78593]: pgmap v9039: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:17:24.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:17:24 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:17:24.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:17:24 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:17:24.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:17:24 smithi047 ceph-mon[77147]: pgmap v9039: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:17:24.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:17:24 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:17:24.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:17:24 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:17:25.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:17:24 smithi115 conmon[90393]: 2022-05-14T04:17:24.932+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:17:25.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:17:25 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:17:25.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:17:25 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:17:26.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:17:25 smithi115 conmon[90393]: 2022-05-14T04:17:25.914+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:17:26.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:17:26 smithi047 ceph-mon[77147]: pgmap v9040: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:17:26.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:17:26 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:17:26.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:17:26 smithi115 ceph-mon[78593]: pgmap v9040: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:17:26.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:17:26 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:17:27.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:17:26 smithi115 conmon[90393]: 2022-05-14T04:17:26.879+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:17:27.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:17:27 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:17:27.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:17:27 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 17380 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:17:27.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:17:27 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:17:27.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:17:27 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 17380 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:17:28.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:17:27 smithi115 conmon[90393]: 2022-05-14T04:17:27.891+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:17:28.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:17:28 smithi047 ceph-mon[77147]: pgmap v9041: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:17:28.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:17:28 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:17:28.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:17:28 smithi115 ceph-mon[78593]: pgmap v9041: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:17:28.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:17:28 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:17:29.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:17:28 smithi115 conmon[90393]: 2022-05-14T04:17:28.928+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:17:29.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:17:29 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:17:29.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:17:29 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:17:30.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:17:29 smithi115 conmon[90393]: 2022-05-14T04:17:29.883+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:17:30.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:17:30 smithi047 ceph-mon[77147]: pgmap v9042: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:17:30.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:17:30 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:17:30.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:17:30 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:17:30.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:17:30 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:17:30.863 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:17:30 smithi115 ceph-mon[78593]: pgmap v9042: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:17:30.863 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:17:30 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:17:30.864 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:17:30 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:17:30.864 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:17:30 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:17:31.361 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:17:30 smithi115 conmon[90393]: 2022-05-14T04:17:30.883+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:17:31.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:17:31 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:17:31.844 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:17:31 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:17:32.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:17:31 smithi115 conmon[90393]: 2022-05-14T04:17:31.841+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:17:32.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:17:32 smithi047 ceph-mon[77147]: pgmap v9043: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:17:32.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:17:32 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:17:32.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:17:32 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 17385 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:17:32.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:17:32 smithi115 ceph-mon[78593]: pgmap v9043: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:17:32.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:17:32 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:17:32.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:17:32 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 17385 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:17:33.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:17:32 smithi115 conmon[90393]: 2022-05-14T04:17:32.793+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:17:33.771 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:17:33 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:17:33.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:17:33 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:17:34.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:17:33 smithi115 conmon[90393]: 2022-05-14T04:17:33.768+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:17:34.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:17:34 smithi047 ceph-mon[77147]: pgmap v9044: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:17:34.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:17:34 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:17:34.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:17:34 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:17:34.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:17:34 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:17:34.810 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:17:34 smithi115 ceph-mon[78593]: pgmap v9044: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:17:34.810 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:17:34 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:17:34.810 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:17:34 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:17:34.810 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:17:34 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:17:35.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:17:34 smithi115 conmon[90393]: 2022-05-14T04:17:34.807+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:17:35.789 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:17:35 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:17:35.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:17:35 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:17:36.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:17:35 smithi115 conmon[90393]: 2022-05-14T04:17:35.786+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:17:36.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:17:36 smithi047 ceph-mon[77147]: pgmap v9045: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:17:36.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:17:36 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:17:36.816 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:17:36 smithi115 ceph-mon[78593]: pgmap v9045: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:17:36.816 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:17:36 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:17:37.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:17:36 smithi115 conmon[90393]: 2022-05-14T04:17:36.813+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:17:37.777 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:17:37 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:17:37.778 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:17:37 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 17390 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:17:37.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:17:37 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:17:37.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:17:37 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 17390 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:17:38.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:17:37 smithi115 conmon[90393]: 2022-05-14T04:17:37.774+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:17:38.790 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:17:38 smithi115 ceph-mon[78593]: pgmap v9046: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:17:38.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:17:38 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:17:38.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:17:38 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:17:38.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:17:38 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:17:38.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:17:38 smithi047 ceph-mon[77147]: pgmap v9046: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:17:38.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:17:38 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:17:38.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:17:38 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:17:38.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:17:38 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:17:39.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:17:38 smithi115 conmon[90393]: 2022-05-14T04:17:38.787+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:17:39.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:17:39 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:17:39.815 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:17:39 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:17:40.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:17:39 smithi115 conmon[90393]: 2022-05-14T04:17:39.812+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:17:40.772 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:17:40 smithi115 ceph-mon[78593]: pgmap v9047: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:17:40.773 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:17:40 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:17:40.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:17:40 smithi047 ceph-mon[77147]: pgmap v9047: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:17:40.795 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:17:40 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:17:41.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:17:40 smithi115 conmon[90393]: 2022-05-14T04:17:40.769+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:17:41.746 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:17:41 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:17:41.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:17:41 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:17:42.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:17:41 smithi115 conmon[90393]: 2022-05-14T04:17:41.743+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:17:42.732 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:17:42 smithi115 ceph-mon[78593]: pgmap v9048: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:17:42.732 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:17:42 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:17:42.733 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:17:42 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:17:42.733 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:17:42 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:17:42.733 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:17:42 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 17395 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:17:42.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:17:42 smithi047 ceph-mon[77147]: pgmap v9048: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:17:42.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:17:42 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:17:42.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:17:42 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:17:42.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:17:42 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:17:42.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:17:42 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 17395 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:17:43.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:17:42 smithi115 conmon[90393]: 2022-05-14T04:17:42.728+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:17:43.756 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:17:43 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:17:43.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:17:43 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:17:44.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:17:43 smithi115 conmon[90393]: 2022-05-14T04:17:43.753+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:17:44.709 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:17:44 smithi115 ceph-mon[78593]: pgmap v9049: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:17:44.709 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:17:44 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:17:44.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:17:44 smithi047 ceph-mon[77147]: pgmap v9049: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:17:44.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:17:44 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:17:45.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:17:44 smithi115 conmon[90393]: 2022-05-14T04:17:44.706+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:17:45.693 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:17:45 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:17:45.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:17:45 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:17:46.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:17:45 smithi115 conmon[90393]: 2022-05-14T04:17:45.690+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:17:46.737 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:17:46 smithi115 ceph-mon[78593]: pgmap v9050: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:17:46.737 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:17:46 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:17:46.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:17:46 smithi047 ceph-mon[77147]: pgmap v9050: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:17:46.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:17:46 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:17:47.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:17:46 smithi115 conmon[90393]: 2022-05-14T04:17:46.733+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:17:47.690 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:17:47 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:17:47.690 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:17:47 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 17400 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:17:47.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:17:47 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:17:47.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:17:47 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 17400 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:17:48.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:17:47 smithi115 conmon[90393]: 2022-05-14T04:17:47.687+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:17:48.697 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:17:48 smithi115 ceph-mon[78593]: pgmap v9051: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:17:48.697 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:17:48 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:17:48.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:17:48 smithi047 ceph-mon[77147]: pgmap v9051: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:17:48.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:17:48 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:17:49.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:17:48 smithi115 conmon[90393]: 2022-05-14T04:17:48.694+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:17:49.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:17:49 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:17:49.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:17:49 smithi115 conmon[90393]: 2022-05-14T04:17:49.672+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:17:49.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:17:49 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:17:50.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:17:50 smithi047 ceph-mon[77147]: pgmap v9052: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:17:50.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:17:50 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:17:50.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:17:50 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:17:50.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:17:50 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:17:50.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:17:50 smithi115 conmon[90393]: 2022-05-14T04:17:50.673+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:17:50.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:17:50 smithi115 ceph-mon[78593]: pgmap v9052: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:17:50.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:17:50 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:17:50.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:17:50 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:17:50.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:17:50 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:17:51.715 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:17:51 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:17:51.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:17:51 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:17:52.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:17:51 smithi115 conmon[90393]: 2022-05-14T04:17:51.712+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:17:52.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:17:52 smithi047 ceph-mon[77147]: pgmap v9053: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:17:52.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:17:52 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:17:52.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:17:52 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 17405 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:17:52.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:17:52 smithi115 conmon[90393]: 2022-05-14T04:17:52.682+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:17:52.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:17:52 smithi115 ceph-mon[78593]: pgmap v9053: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:17:52.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:17:52 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:17:52.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:17:52 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 17405 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:17:53.727 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:17:53 smithi115 ceph-mon[78593]: pgmap v9054: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:17:53.727 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:17:53 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:17:53.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:17:53 smithi047 ceph-mon[77147]: pgmap v9054: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:17:53.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:17:53 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:17:54.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:17:53 smithi115 conmon[90393]: 2022-05-14T04:17:53.724+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:17:54.733 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:17:54 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:17:54.733 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:17:54 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:17:54.733 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:17:54 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:17:54.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:17:54 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:17:54.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:17:54 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:17:54.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:17:54 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:17:55.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:17:54 smithi115 conmon[90393]: 2022-05-14T04:17:54.730+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:17:55.710 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:17:55 smithi115 ceph-mon[78593]: pgmap v9055: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:17:55.710 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:17:55 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:17:55.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:17:55 smithi047 ceph-mon[77147]: pgmap v9055: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:17:55.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:17:55 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:17:56.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:17:55 smithi115 conmon[90393]: 2022-05-14T04:17:55.707+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:17:56.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:17:56 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:17:56.860 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:17:56 smithi115 conmon[90393]: 2022-05-14T04:17:56.694+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:17:56.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:17:56 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:17:57.727 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:17:57 smithi115 ceph-mon[78593]: pgmap v9056: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:17:57.728 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:17:57 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:17:57.728 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:17:57 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 17410 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:17:57.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:17:57 smithi047 ceph-mon[77147]: pgmap v9056: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:17:57.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:17:57 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:17:57.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:17:57 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 17410 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:17:58.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:17:57 smithi115 conmon[90393]: 2022-05-14T04:17:57.724+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:17:58.725 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:17:58 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:17:58.726 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:17:58 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:17:58.726 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:17:58 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:17:58.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:17:58 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:17:58.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:17:58 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:17:58.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:17:58 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:17:59.110 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:17:58 smithi115 conmon[90393]: 2022-05-14T04:17:58.722+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:17:59.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:17:59 smithi047 ceph-mon[77147]: pgmap v9057: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:17:59.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:17:59 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:17:59.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:17:59 smithi115 conmon[90393]: 2022-05-14T04:17:59.710+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:17:59.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:17:59 smithi115 ceph-mon[78593]: pgmap v9057: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:17:59.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:17:59 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:18:00.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:18:00 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:18:00.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:18:00 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:18:00.860 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:18:00 smithi115 conmon[90393]: 2022-05-14T04:18:00.662+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:18:01.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:18:01 smithi047 ceph-mon[77147]: pgmap v9058: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:18:01.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:18:01 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:18:01.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:18:01 smithi115 conmon[90393]: 2022-05-14T04:18:01.705+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:18:01.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:18:01 smithi115 ceph-mon[78593]: pgmap v9058: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:18:01.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:18:01 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:18:02.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:18:02 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:18:02.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:18:02 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:18:02.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:18:02 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:18:02.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:18:02 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 17415 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:18:02.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:18:02 smithi115 conmon[90393]: 2022-05-14T04:18:02.656+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:18:02.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:18:02 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:18:02.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:18:02 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:18:02.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:18:02 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:18:02.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:18:02 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 17415 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:18:03.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:18:03 smithi047 ceph-mon[77147]: pgmap v9059: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:18:03.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:18:03 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:18:03.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:18:03 smithi115 ceph-mon[78593]: pgmap v9059: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:18:03.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:18:03 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:18:03.860 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:18:03 smithi115 conmon[90393]: 2022-05-14T04:18:03.648+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:18:04.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:18:04 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:18:04.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:18:04 smithi115 conmon[90393]: 2022-05-14T04:18:04.662+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:18:04.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:18:04 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:18:05.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:18:05 smithi047 ceph-mon[77147]: pgmap v9060: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:18:05.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:18:05 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:18:05.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:18:05 smithi115 conmon[90393]: 2022-05-14T04:18:05.635+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:18:05.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:18:05 smithi115 ceph-mon[78593]: pgmap v9060: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:18:05.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:18:05 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:18:06.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:18:06 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:18:06.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:18:06 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:18:06.860 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:18:06 smithi115 conmon[90393]: 2022-05-14T04:18:06.664+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:18:07.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:18:07 smithi047 ceph-mon[77147]: pgmap v9061: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:18:07.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:18:07 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:18:07.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:18:07 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 17420 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:18:07.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:18:07 smithi115 conmon[90393]: 2022-05-14T04:18:07.712+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:18:07.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:18:07 smithi115 ceph-mon[78593]: pgmap v9061: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:18:07.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:18:07 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:18:07.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:18:07 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 17420 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:18:08.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:18:08 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:18:08.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:18:08 smithi115 conmon[90393]: 2022-05-14T04:18:08.673+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:18:08.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:18:08 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:18:09.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:18:09 smithi047 ceph-mon[77147]: pgmap v9062: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:18:09.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:18:09 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:18:09.860 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:18:09 smithi115 conmon[90393]: 2022-05-14T04:18:09.667+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:18:09.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:18:09 smithi115 ceph-mon[78593]: pgmap v9062: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:18:09.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:18:09 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:18:10.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:18:10 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:18:10.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:18:10 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:18:10.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:18:10 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:18:10.860 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:18:10 smithi115 conmon[90393]: 2022-05-14T04:18:10.692+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:18:10.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:18:10 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:18:10.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:18:10 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:18:10.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:18:10 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:18:11.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:18:11 smithi047 ceph-mon[77147]: pgmap v9063: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:18:11.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:18:11 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:18:11.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:18:11 smithi115 conmon[90393]: 2022-05-14T04:18:11.718+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:18:11.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:18:11 smithi115 ceph-mon[78593]: pgmap v9063: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:18:11.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:18:11 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:18:12.766 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:18:12 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:18:12.766 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:18:12 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 17425 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:18:12.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:18:12 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:18:12.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:18:12 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 17425 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:18:13.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:18:12 smithi115 conmon[90393]: 2022-05-14T04:18:12.763+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:18:13.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:18:13 smithi047 ceph-mon[77147]: pgmap v9064: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:18:13.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:18:13 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:18:13.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:18:13 smithi115 conmon[90393]: 2022-05-14T04:18:13.726+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:18:13.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:18:13 smithi115 ceph-mon[78593]: pgmap v9064: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:18:13.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:18:13 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:18:14.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:18:14 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:18:14.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:18:14 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:18:14.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:18:14 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:18:14.854 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:18:14 smithi115 conmon[90393]: 2022-05-14T04:18:14.698+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:18:14.855 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:18:14 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:18:14.855 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:18:14 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:18:14.855 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:18:14 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:18:15.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:18:15 smithi115 ceph-mon[78593]: pgmap v9065: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:18:15.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:18:15 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T04:18:15.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:18:15 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T04:18:15.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:18:15 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:18:15.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:18:15 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T04:18:15.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:18:15 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T04:18:15.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:18:15 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T04:18:15.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:18:15 smithi047 ceph-mon[77147]: pgmap v9065: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:18:15.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:18:15 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T04:18:15.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:18:15 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T04:18:15.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:18:15 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:18:15.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:18:15 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T04:18:15.802 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:18:15 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T04:18:15.803 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:18:15 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T04:18:16.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:18:15 smithi115 conmon[90393]: 2022-05-14T04:18:15.656+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:18:16.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:18:16 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:18:16.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:18:16 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:18:16.860 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:18:16 smithi115 conmon[90393]: 2022-05-14T04:18:16.671+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:18:17.535 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:18:17 smithi115 ceph-mon[78593]: pgmap v9066: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:18:17.536 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:18:17 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:18:17.536 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:18:17 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 17430 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:18:17.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:18:17 smithi047 ceph-mon[77147]: pgmap v9066: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:18:17.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:18:17 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:18:17.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:18:17 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 17430 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:18:17.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:18:17 smithi115 conmon[90393]: 2022-05-14T04:18:17.655+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:18:18.293 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:18:17 smithi047 conmon[78521]: 2022-05-14T04:18:17.904+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:18:18.293 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:18:17 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T04:18:18.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:18:17 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:18:18.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:18:17 smithi047 conmon[78521]: yield 2022-05-14T04:18:18.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:18:17 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:18:18.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:18:17 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:18:18.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:18:17 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:18:18.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:18:17 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T04:18:18.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:18:17 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:18:18.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:18:17 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T04:18:18.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:18:17 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:18:18.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:18:17 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T04:18:18.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:18:17 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:18:18.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:18:17 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T04:18:18.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:18:17 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:18:18.297 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:18:17 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T04:18:18.297 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:18:17 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:18:18.297 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:18:17 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:18:18.298 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:18:17 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:18:18.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:18:18 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:18:18.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:18:18 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:18:18.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:18:18 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:18:18.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:18:18 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:18:18.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T04:18:18.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:18:18.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T04:18:18.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:18:18.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:18:18.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:18:18.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T04:18:18.797 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:18:18.797 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T04:18:18.797 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:18:18.797 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T04:18:18.798 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:18:18.798 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T04:18:18.798 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:18:18.798 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T04:18:18.799 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:18:18.799 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:18:18.799 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:18:18.799 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:18:18 smithi047 ceph-mon[77147]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T04:18:18.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:18:18 smithi115 conmon[90393]: 2022-05-14T04:18:18.658+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:18:18.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:18:18 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:18:18.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:18:18 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:18:18.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:18:18 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:18:18.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:18:18 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:18:18.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T04:18:18.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:18:18.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T04:18:18.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:18:18.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:18:18.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:18:18.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T04:18:18.863 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:18:18.863 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T04:18:18.863 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:18:18.863 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T04:18:18.864 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:18:18.864 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T04:18:18.864 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:18:18.864 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T04:18:18.864 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:18:18.865 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:18:18.865 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:18:18.865 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:18:18 smithi115 ceph-mon[78593]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T04:18:19.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:18:19 smithi047 ceph-mon[77147]: pgmap v9067: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:18:19.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:18:19 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:18:19.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:18:19 smithi115 ceph-mon[78593]: pgmap v9067: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:18:19.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:18:19 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:18:19.860 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:18:19 smithi115 conmon[90393]: 2022-05-14T04:18:19.634+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:18:20.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:18:20 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:18:20.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:18:20 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:18:20.860 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:18:20 smithi115 conmon[90393]: 2022-05-14T04:18:20.653+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:18:21.293 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:18:20 smithi047 conmon[78521]: 2022-05-14T04:18:20.976+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:18:21.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:18:20 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T04:18:21.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:18:20 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:18:21.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:18:20 smithi047 conmon[78521]: yield 2022-05-14T04:18:21.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:18:20 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:18:21.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:18:20 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:18:21.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:18:20 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:18:21.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:18:20 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T04:18:21.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:18:20 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:18:21.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:18:20 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T04:18:21.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:18:20 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:18:21.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:18:20 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T04:18:21.297 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:18:20 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:18:21.297 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:18:20 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T04:18:21.297 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:18:20 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:18:21.297 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:18:20 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T04:18:21.298 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:18:20 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:18:21.298 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:18:20 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:18:21.298 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:18:20 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:18:21.298 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:18:20 smithi047 conmon[78521]: 2022-05-14T04:18:20.977+0000 7f533ef76700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T04:18:21.298 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:18:20 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T04:18:21.299 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:18:20 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:18:21.299 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:18:20 smithi047 conmon[78521]: yield 2022-05-14T04:18:21.299 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:18:20 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:18:21.299 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:18:20 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:18:21.300 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:18:20 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:18:21.300 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:18:20 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T04:18:21.300 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:18:20 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:18:21.300 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:18:20 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T04:18:21.300 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:18:20 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:18:21.301 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:18:20 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T04:18:21.301 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:18:20 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:18:21.301 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:18:20 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T04:18:21.301 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:18:20 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:18:21.302 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:18:20 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T04:18:21.302 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:18:20 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:18:21.302 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:18:20 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:18:21.302 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:18:20 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:18:21.303 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:18:20 smithi047 conmon[78521]: 2022-05-14T04:18:21.303 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:18:20 smithi047 conmon[78521]: During handling of the above exception, another exception occurred: 2022-05-14T04:18:21.303 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:18:20 smithi047 conmon[78521]: 2022-05-14T04:18:21.303 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:18:20 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T04:18:21.304 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:18:20 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T04:18:21.304 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:18:20 smithi047 conmon[78521]: return f(*arg) 2022-05-14T04:18:21.304 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:18:20 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T04:18:21.304 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:18:20 smithi047 conmon[78521]: self._write_client_files(client_files, host) 2022-05-14T04:18:21.304 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:18:20 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T04:18:21.305 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:18:20 smithi047 conmon[78521]: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T04:18:21.305 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:18:20 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T04:18:21.305 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:18:20 smithi047 conmon[78521]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T04:18:21.305 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:18:20 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T04:18:21.306 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:18:20 smithi047 conmon[78521]: return self.event_loop.get_result(coro) 2022-05-14T04:18:21.306 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:18:20 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T04:18:21.306 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:18:20 smithi047 conmon[78521]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T04:18:21.306 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:18:20 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T04:18:21.307 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:18:20 smithi047 conmon[78521]: return self.__get_result() 2022-05-14T04:18:21.307 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:18:20 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T04:18:21.307 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:18:20 smithi047 conmon[78521]: raise self._exception 2022-05-14T04:18:21.307 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:18:20 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T04:18:21.307 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:18:20 smithi047 conmon[78521]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T04:18:21.308 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:18:20 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T04:18:21.308 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:18:20 smithi047 conmon[78521]: conn = await self._remote_connection(host, addr) 2022-05-14T04:18:21.308 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:18:20 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T04:18:21.308 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:18:20 smithi047 conmon[78521]: raise 2022-05-14T04:18:21.308 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:18:20 smithi047 conmon[78521]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T04:18:21.309 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:18:20 smithi047 conmon[78521]: self.gen.throw(type, value, traceback) 2022-05-14T04:18:21.309 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:18:20 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T04:18:21.309 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:18:20 smithi047 conmon[78521]: raise OrchestratorError(msg) 2022-05-14T04:18:21.309 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:18:20 smithi047 conmon[78521]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:18:21.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:18:21 smithi115 conmon[90393]: 2022-05-14T04:18:21.642+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:18:21.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:18:21 smithi115 ceph-mon[78593]: pgmap v9068: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:18:21.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:18:21 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:18:21.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:18:21 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:18:21.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T04:18:21.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:18:21.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T04:18:21.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:18:21.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:18:21.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:18:21.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T04:18:21.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:18:21.863 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T04:18:21.863 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:18:21.863 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T04:18:21.863 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:18:21.863 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T04:18:21.864 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:18:21.864 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T04:18:21.865 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:18:21.865 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:18:21.865 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:18:21.866 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:18:21 smithi115 ceph-mon[78593]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T04:18:21.866 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T04:18:21.866 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:18:21.866 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T04:18:21.866 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:18:21.867 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:18:21.867 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:18:21.867 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T04:18:21.867 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:18:21.868 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T04:18:21.868 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:18:21.868 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T04:18:21.868 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:18:21.868 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T04:18:21.869 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:18:21.869 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T04:18:21.869 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:18:21.870 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:18:21.870 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:18:21.870 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T04:18:21.871 INFO:journalctl@ceph.mon.c.smithi115.stdout: During handling of the above exception, another exception occurred: 2022-05-14T04:18:21.871 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T04:18:21.871 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T04:18:21.871 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T04:18:21.871 INFO:journalctl@ceph.mon.c.smithi115.stdout: return f(*arg) 2022-05-14T04:18:21.872 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T04:18:21.872 INFO:journalctl@ceph.mon.c.smithi115.stdout: self._write_client_files(client_files, host) 2022-05-14T04:18:21.872 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T04:18:21.873 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T04:18:21.873 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T04:18:21.873 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T04:18:21.873 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T04:18:21.874 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.event_loop.get_result(coro) 2022-05-14T04:18:21.874 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T04:18:21.874 INFO:journalctl@ceph.mon.c.smithi115.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T04:18:21.874 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T04:18:21.875 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.__get_result() 2022-05-14T04:18:21.876 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T04:18:21.876 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise self._exception 2022-05-14T04:18:21.876 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T04:18:21.876 INFO:journalctl@ceph.mon.c.smithi115.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T04:18:21.876 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T04:18:21.877 INFO:journalctl@ceph.mon.c.smithi115.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T04:18:21.877 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T04:18:21.877 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise 2022-05-14T04:18:21.878 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T04:18:21.878 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.gen.throw(type, value, traceback) 2022-05-14T04:18:21.878 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T04:18:21.879 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OrchestratorError(msg) 2022-05-14T04:18:21.879 INFO:journalctl@ceph.mon.c.smithi115.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:18:22.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:18:21 smithi047 ceph-mon[77147]: pgmap v9068: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:18:22.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:18:21 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:18:22.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:18:21 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:18:22.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T04:18:22.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:18:22.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T04:18:22.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:18:22.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:18:22.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:18:22.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T04:18:22.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:18:22.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T04:18:22.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:18:22.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T04:18:22.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:18:22.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T04:18:22.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:18:22.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T04:18:22.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:18:22.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:18:22.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:18:22.048 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:18:21 smithi047 ceph-mon[77147]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T04:18:22.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T04:18:22.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:18:22.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T04:18:22.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:18:22.049 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:18:22.049 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:18:22.049 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T04:18:22.049 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:18:22.050 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T04:18:22.050 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:18:22.050 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T04:18:22.050 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:18:22.050 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T04:18:22.051 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:18:22.051 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T04:18:22.051 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:18:22.051 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:18:22.051 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:18:22.052 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T04:18:22.052 INFO:journalctl@ceph.mon.b.smithi047.stdout: During handling of the above exception, another exception occurred: 2022-05-14T04:18:22.052 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T04:18:22.052 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T04:18:22.053 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T04:18:22.053 INFO:journalctl@ceph.mon.b.smithi047.stdout: return f(*arg) 2022-05-14T04:18:22.053 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T04:18:22.053 INFO:journalctl@ceph.mon.b.smithi047.stdout: self._write_client_files(client_files, host) 2022-05-14T04:18:22.053 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T04:18:22.054 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T04:18:22.054 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T04:18:22.054 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T04:18:22.054 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T04:18:22.054 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.event_loop.get_result(coro) 2022-05-14T04:18:22.055 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T04:18:22.055 INFO:journalctl@ceph.mon.b.smithi047.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T04:18:22.055 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T04:18:22.055 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.__get_result() 2022-05-14T04:18:22.055 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T04:18:22.056 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise self._exception 2022-05-14T04:18:22.056 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T04:18:22.056 INFO:journalctl@ceph.mon.b.smithi047.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T04:18:22.056 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T04:18:22.057 INFO:journalctl@ceph.mon.b.smithi047.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T04:18:22.057 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T04:18:22.057 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise 2022-05-14T04:18:22.057 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T04:18:22.057 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.gen.throw(type, value, traceback) 2022-05-14T04:18:22.058 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T04:18:22.058 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OrchestratorError(msg) 2022-05-14T04:18:22.058 INFO:journalctl@ceph.mon.b.smithi047.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:18:22.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:18:22 smithi115 conmon[90393]: 2022-05-14T04:18:22.656+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:18:22.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:18:22 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:18:22.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:18:22 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:18:22.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:18:22 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:18:22.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:18:22 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 17435 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:18:23.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:18:22 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:18:23.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:18:22 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:18:23.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:18:22 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:18:23.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:18:22 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 17435 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:18:23.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:18:23 smithi115 ceph-mon[78593]: pgmap v9069: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:18:23.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:18:23 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:18:23.860 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:18:23 smithi115 conmon[90393]: 2022-05-14T04:18:23.636+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:18:24.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:18:23 smithi047 ceph-mon[77147]: pgmap v9069: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:18:24.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:18:23 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:18:24.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:18:24 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:18:24.861 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:18:24 smithi115 conmon[90393]: 2022-05-14T04:18:24.671+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:18:25.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:18:24 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:18:25.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:18:25 smithi115 conmon[90393]: 2022-05-14T04:18:25.668+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:18:25.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:18:25 smithi115 ceph-mon[78593]: pgmap v9070: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:18:25.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:18:25 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:18:26.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:18:25 smithi047 ceph-mon[77147]: pgmap v9070: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:18:26.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:18:25 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:18:26.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:18:26 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:18:26.860 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:18:26 smithi115 conmon[90393]: 2022-05-14T04:18:26.674+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:18:27.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:18:26 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:18:27.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:18:27 smithi115 conmon[90393]: 2022-05-14T04:18:27.664+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:18:27.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:18:27 smithi115 ceph-mon[78593]: pgmap v9071: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:18:27.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:18:27 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:18:27.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:18:27 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 17440 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:18:28.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:18:27 smithi047 ceph-mon[77147]: pgmap v9071: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:18:28.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:18:27 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:18:28.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:18:27 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 17440 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:18:28.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:18:28 smithi115 conmon[90393]: 2022-05-14T04:18:28.702+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:18:28.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:18:28 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:18:29.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:18:28 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:18:30.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:18:29 smithi047 ceph-mon[77147]: pgmap v9072: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:18:30.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:18:29 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:18:30.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:18:29 smithi115 conmon[90393]: 2022-05-14T04:18:29.718+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:18:30.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:18:29 smithi115 ceph-mon[78593]: pgmap v9072: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:18:30.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:18:29 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:18:31.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:18:30 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:18:31.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:18:30 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:18:31.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:18:30 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:18:31.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:18:30 smithi115 conmon[90393]: 2022-05-14T04:18:30.748+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:18:31.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:18:30 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:18:31.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:18:30 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:18:31.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:18:30 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:18:32.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:18:31 smithi047 ceph-mon[77147]: pgmap v9073: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:18:32.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:18:31 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:18:32.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:18:31 smithi115 conmon[90393]: 2022-05-14T04:18:31.718+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:18:32.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:18:31 smithi115 ceph-mon[78593]: pgmap v9073: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:18:32.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:18:31 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:18:33.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:18:32 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:18:33.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:18:32 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 17445 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:18:33.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:18:32 smithi115 conmon[90393]: 2022-05-14T04:18:32.708+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:18:33.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:18:32 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:18:33.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:18:32 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 17445 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:18:34.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:18:33 smithi115 conmon[90393]: 2022-05-14T04:18:33.706+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:18:34.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:18:34 smithi115 ceph-mon[78593]: pgmap v9074: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:18:34.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:18:34 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:18:34.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:18:34 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:18:34.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:18:34 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:18:34.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:18:34 smithi047 ceph-mon[77147]: pgmap v9074: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:18:34.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:18:34 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:18:34.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:18:34 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:18:34.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:18:34 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:18:35.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:18:34 smithi115 conmon[90393]: 2022-05-14T04:18:34.690+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:18:35.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:18:35 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:18:35.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:18:35 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:18:36.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:18:35 smithi115 conmon[90393]: 2022-05-14T04:18:35.730+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:18:36.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:18:36 smithi115 ceph-mon[78593]: pgmap v9075: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:18:36.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:18:36 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:18:36.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:18:36 smithi047 ceph-mon[77147]: pgmap v9075: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:18:36.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:18:36 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:18:37.110 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:18:36 smithi115 conmon[90393]: 2022-05-14T04:18:36.741+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:18:37.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:18:37 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:18:37.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:18:37 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 17450 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:18:37.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:18:37 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:18:37.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:18:37 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 17450 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:18:38.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:18:37 smithi115 conmon[90393]: 2022-05-14T04:18:37.727+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:18:38.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:18:38 smithi115 ceph-mon[78593]: pgmap v9076: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:18:38.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:18:38 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:18:38.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:18:38 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:18:38.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:18:38 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:18:38.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:18:38 smithi047 ceph-mon[77147]: pgmap v9076: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:18:38.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:18:38 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:18:38.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:18:38 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:18:38.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:18:38 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:18:39.110 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:18:38 smithi115 conmon[90393]: 2022-05-14T04:18:38.697+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:18:39.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:18:39 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:18:39.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:18:39 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:18:40.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:18:39 smithi115 conmon[90393]: 2022-05-14T04:18:39.719+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:18:40.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:18:40 smithi115 ceph-mon[78593]: pgmap v9077: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:18:40.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:18:40 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:18:40.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:18:40 smithi047 ceph-mon[77147]: pgmap v9077: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:18:40.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:18:40 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:18:41.110 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:18:40 smithi115 conmon[90393]: 2022-05-14T04:18:40.729+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:18:41.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:18:41 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:18:41.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:18:41 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:18:42.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:18:41 smithi115 conmon[90393]: 2022-05-14T04:18:41.770+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:18:42.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:18:42 smithi115 ceph-mon[78593]: pgmap v9078: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:18:42.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:18:42 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:18:42.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:18:42 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:18:42.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:18:42 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:18:42.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:18:42 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 17455 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:18:42.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:18:42 smithi047 ceph-mon[77147]: pgmap v9078: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:18:42.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:18:42 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:18:42.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:18:42 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:18:42.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:18:42 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:18:42.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:18:42 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 17455 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:18:43.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:18:42 smithi115 conmon[90393]: 2022-05-14T04:18:42.742+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:18:43.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:18:43 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:18:43.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:18:43 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:18:44.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:18:43 smithi115 conmon[90393]: 2022-05-14T04:18:43.782+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:18:44.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:18:44 smithi115 ceph-mon[78593]: pgmap v9079: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:18:44.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:18:44 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:18:44.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:18:44 smithi047 ceph-mon[77147]: pgmap v9079: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:18:44.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:18:44 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:18:45.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:18:44 smithi115 conmon[90393]: 2022-05-14T04:18:44.788+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:18:45.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:18:45 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:18:45.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:18:45 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:18:46.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:18:45 smithi115 conmon[90393]: 2022-05-14T04:18:45.793+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:18:46.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:18:46 smithi115 ceph-mon[78593]: pgmap v9080: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:18:46.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:18:46 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:18:46.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:18:46 smithi047 ceph-mon[77147]: pgmap v9080: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:18:46.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:18:46 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:18:47.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:18:46 smithi115 conmon[90393]: 2022-05-14T04:18:46.760+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:18:47.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:18:47 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:18:47.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:18:47 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 17460 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:18:47.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:18:47 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:18:47.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:18:47 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 17460 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:18:48.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:18:47 smithi115 conmon[90393]: 2022-05-14T04:18:47.747+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:18:48.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:18:48 smithi115 ceph-mon[78593]: pgmap v9081: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:18:48.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:18:48 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:18:48.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:18:48 smithi047 ceph-mon[77147]: pgmap v9081: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:18:48.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:18:48 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:18:49.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:18:48 smithi115 conmon[90393]: 2022-05-14T04:18:48.781+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:18:49.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:18:49 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:18:49.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:18:49 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:18:50.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:18:49 smithi115 conmon[90393]: 2022-05-14T04:18:49.804+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:18:50.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:18:50 smithi115 ceph-mon[78593]: pgmap v9082: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:18:50.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:18:50 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:18:50.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:18:50 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:18:50.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:18:50 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:18:50.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:18:50 smithi047 ceph-mon[77147]: pgmap v9082: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:18:50.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:18:50 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:18:50.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:18:50 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:18:50.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:18:50 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:18:51.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:18:50 smithi115 conmon[90393]: 2022-05-14T04:18:50.818+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:18:51.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:18:51 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:18:51.839 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:18:51 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:18:52.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:18:51 smithi115 conmon[90393]: 2022-05-14T04:18:51.836+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:18:52.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:18:52 smithi047 ceph-mon[77147]: pgmap v9083: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:18:52.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:18:52 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:18:52.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:18:52 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 17465 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:18:52.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:18:52 smithi115 ceph-mon[78593]: pgmap v9083: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:18:52.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:18:52 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:18:52.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:18:52 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 17465 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:18:53.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:18:52 smithi115 conmon[90393]: 2022-05-14T04:18:52.877+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:18:53.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:18:53 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:18:53.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:18:53 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:18:54.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:18:53 smithi115 conmon[90393]: 2022-05-14T04:18:53.917+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:18:54.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:18:54 smithi047 ceph-mon[77147]: pgmap v9084: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:18:54.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:18:54 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:18:54.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:18:54 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:18:54.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:18:54 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:18:54.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:18:54 smithi115 ceph-mon[78593]: pgmap v9084: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:18:54.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:18:54 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:18:54.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:18:54 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:18:54.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:18:54 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:18:55.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:18:54 smithi115 conmon[90393]: 2022-05-14T04:18:54.904+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:18:55.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:18:55 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:18:55.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:18:55 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:18:56.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:18:55 smithi115 conmon[90393]: 2022-05-14T04:18:55.929+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:18:56.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:18:56 smithi047 ceph-mon[77147]: pgmap v9085: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:18:56.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:18:56 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:18:56.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:18:56 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:18:56.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:18:56 smithi115 ceph-mon[78593]: pgmap v9085: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:18:56.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:18:56 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:18:56.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:18:56 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:18:57.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:18:56 smithi115 conmon[90393]: 2022-05-14T04:18:56.884+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:18:57.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:18:57 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:18:57.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:18:57 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 17470 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:18:57.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:18:57 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:18:57.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:18:57 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 17470 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:18:58.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:18:57 smithi115 conmon[90393]: 2022-05-14T04:18:57.919+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:18:58.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:18:58 smithi047 ceph-mon[77147]: pgmap v9086: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:18:58.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:18:58 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:18:58.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:18:58 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:18:58.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:18:58 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:18:58.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:18:58 smithi115 ceph-mon[78593]: pgmap v9086: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:18:58.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:18:58 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:18:58.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:18:58 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:18:58.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:18:58 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:18:59.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:18:58 smithi115 conmon[90393]: 2022-05-14T04:18:58.889+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:18:59.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:18:59 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:18:59.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:18:59 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:19:00.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:18:59 smithi115 conmon[90393]: 2022-05-14T04:18:59.923+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:19:00.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:19:00 smithi047 ceph-mon[77147]: pgmap v9087: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:19:00.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:19:00 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:19:00.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:19:00 smithi115 ceph-mon[78593]: pgmap v9087: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:19:00.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:19:00 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:19:01.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:19:00 smithi115 conmon[90393]: 2022-05-14T04:19:00.895+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:19:01.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:19:01 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:19:01.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:19:01 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:19:02.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:19:01 smithi115 conmon[90393]: 2022-05-14T04:19:01.857+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:19:02.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:19:02 smithi047 ceph-mon[77147]: pgmap v9088: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:19:02.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:19:02 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:19:02.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:19:02 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:19:02.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:19:02 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:19:02.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:19:02 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 17475 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:19:02.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:19:02 smithi115 ceph-mon[78593]: pgmap v9088: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:19:02.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:19:02 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:19:02.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:19:02 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:19:02.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:19:02 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:19:02.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:19:02 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 17475 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:19:03.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:19:02 smithi115 conmon[90393]: 2022-05-14T04:19:02.904+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:19:03.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:19:03 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:19:03.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:19:03 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:19:04.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:19:03 smithi115 conmon[90393]: 2022-05-14T04:19:03.880+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:19:04.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:19:04 smithi047 ceph-mon[77147]: pgmap v9089: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:19:04.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:19:04 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:19:04.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:19:04 smithi115 ceph-mon[78593]: pgmap v9089: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:19:04.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:19:04 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:19:05.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:19:04 smithi115 conmon[90393]: 2022-05-14T04:19:04.863+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:19:05.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:19:05 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:19:05.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:19:05 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:19:06.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:19:05 smithi115 conmon[90393]: 2022-05-14T04:19:05.866+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:19:06.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:19:06 smithi047 ceph-mon[77147]: pgmap v9090: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:19:06.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:19:06 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:19:06.854 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:19:06 smithi115 ceph-mon[78593]: pgmap v9090: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:19:06.854 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:19:06 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:19:07.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:19:06 smithi115 conmon[90393]: 2022-05-14T04:19:06.851+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:19:07.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:19:07 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:19:07.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:19:07 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 17480 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:19:07.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:19:07 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:19:07.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:19:07 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 17480 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:19:08.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:19:07 smithi115 conmon[90393]: 2022-05-14T04:19:07.868+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:19:08.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:19:08 smithi047 ceph-mon[77147]: pgmap v9091: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:19:08.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:19:08 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:19:08.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:19:08 smithi115 ceph-mon[78593]: pgmap v9091: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:19:08.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:19:08 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:19:09.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:19:08 smithi115 conmon[90393]: 2022-05-14T04:19:08.903+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:19:09.797 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:19:09 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:19:09.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:19:09 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:19:10.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:19:09 smithi115 conmon[90393]: 2022-05-14T04:19:09.934+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:19:10.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:19:10 smithi047 ceph-mon[77147]: pgmap v9092: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:19:10.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:19:10 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:19:10.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:19:10 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:19:10.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:19:10 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:19:10.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:19:10 smithi115 ceph-mon[78593]: pgmap v9092: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:19:10.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:19:10 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:19:10.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:19:10 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:19:10.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:19:10 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:19:11.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:19:10 smithi115 conmon[90393]: 2022-05-14T04:19:10.892+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:19:11.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:19:11 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:19:11.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:19:11 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:19:12.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:19:11 smithi115 conmon[90393]: 2022-05-14T04:19:11.879+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:19:12.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:19:12 smithi047 ceph-mon[77147]: pgmap v9093: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:19:12.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:19:12 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:19:12.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:19:12 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 17485 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:19:12.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:19:12 smithi115 ceph-mon[78593]: pgmap v9093: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:19:12.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:19:12 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:19:12.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:19:12 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 17485 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:19:13.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:19:12 smithi115 conmon[90393]: 2022-05-14T04:19:12.869+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:19:13.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:19:13 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:19:13.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:19:13 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:19:14.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:19:13 smithi115 conmon[90393]: 2022-05-14T04:19:13.901+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:19:14.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:19:14 smithi047 ceph-mon[77147]: pgmap v9094: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:19:14.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:19:14 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:19:14.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:19:14 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:19:14.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:19:14 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:19:14.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:19:14 smithi115 ceph-mon[78593]: pgmap v9094: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:19:14.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:19:14 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:19:14.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:19:14 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:19:14.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:19:14 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:19:15.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:19:14 smithi115 conmon[90393]: 2022-05-14T04:19:14.928+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:19:15.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:19:15 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T04:19:15.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:19:15 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T04:19:15.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:19:15 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:19:15.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:19:15 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T04:19:15.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:19:15 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T04:19:15.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:19:15 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:19:16.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:19:15 smithi115 conmon[90393]: 2022-05-14T04:19:15.934+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:19:16.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:19:16 smithi047 ceph-mon[77147]: pgmap v9095: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:19:16.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:19:16 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:19:16.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:19:16 smithi115 ceph-mon[78593]: pgmap v9095: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:19:16.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:19:16 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:19:17.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:19:16 smithi115 conmon[90393]: 2022-05-14T04:19:16.918+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:19:17.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:19:17 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:19:17.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:19:17 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 17490 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:19:17.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:19:17 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:19:17.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:19:17 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 17490 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:19:18.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:19:17 smithi115 conmon[90393]: 2022-05-14T04:19:17.901+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:19:18.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:19:18 smithi047 ceph-mon[77147]: pgmap v9096: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:19:18.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:19:18 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:19:18.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:19:18 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:19:18.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:19:18 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:19:18.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:19:18 smithi115 ceph-mon[78593]: pgmap v9096: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:19:18.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:19:18 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:19:18.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:19:18 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:19:18.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:19:18 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:19:19.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:19:18 smithi115 conmon[90393]: 2022-05-14T04:19:18.867+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:19:19.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:19:19 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:19:19.823 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:19:19 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:19:20.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:19:19 smithi115 conmon[90393]: 2022-05-14T04:19:19.820+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:19:20.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:19:20 smithi047 ceph-mon[77147]: pgmap v9097: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:19:20.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:19:20 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:19:20.831 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:19:20 smithi115 ceph-mon[78593]: pgmap v9097: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:19:20.832 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:19:20 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:19:21.091 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:19:20 smithi115 conmon[90393]: 2022-05-14T04:19:20.828+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:19:21.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:19:21 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:19:21.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:19:21 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T04:19:21.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:19:21 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T04:19:21.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:19:21 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T04:19:21.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:19:21 smithi047 conmon[78521]: 2022-05-14T04:19:21.456+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:19:21.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:19:21 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T04:19:21.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:19:21 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:19:21.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:19:21 smithi047 conmon[78521]: yield 2022-05-14T04:19:21.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:19:21 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:19:21.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:19:21 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:19:21.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:19:21 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:19:21.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:19:21 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T04:19:21.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:19:21 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:19:21.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:19:21 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T04:19:21.797 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:19:21 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:19:21.797 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:19:21 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T04:19:21.797 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:19:21 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:19:21.798 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:19:21 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T04:19:21.799 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:19:21 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:19:21.799 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:19:21 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T04:19:21.799 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:19:21 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:19:21.799 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:19:21 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:19:21.800 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:19:21 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:19:21.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:19:21 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:19:21.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:19:21 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T04:19:21.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:19:21 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T04:19:21.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:19:21 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T04:19:22.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:19:21 smithi115 conmon[90393]: 2022-05-14T04:19:21.856+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:19:22.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:19:22 smithi047 ceph-mon[77147]: pgmap v9098: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:19:22.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:19:22 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:19:22.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:19:22 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:19:22.795 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:19:22 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:19:22.795 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:19:22 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 17495 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:19:22.818 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:19:22 smithi115 ceph-mon[78593]: pgmap v9098: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:19:22.819 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:19:22 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:19:22.820 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:19:22 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:19:22.820 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:19:22 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:19:22.821 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:19:22 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 17495 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:19:23.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:19:22 smithi115 conmon[90393]: 2022-05-14T04:19:22.815+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:19:23.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:19:23 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:19:23.792 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T04:19:23.793 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:19:23.793 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T04:19:23.793 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:19:23.793 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:19:23.794 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:19:23.794 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T04:19:23.794 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:19:23.795 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T04:19:23.795 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:19:23.795 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T04:19:23.795 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:19:23.795 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T04:19:23.796 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:19:23.796 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T04:19:23.796 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:19:23.796 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:19:23.796 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:19:23.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:19:23 smithi115 ceph-mon[78593]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T04:19:23.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:19:23 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:19:23.798 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:19:23 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:19:23.798 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T04:19:23.798 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:19:23.798 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T04:19:23.799 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:19:23.799 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:19:23.799 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:19:23.799 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T04:19:23.800 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:19:23.800 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T04:19:23.800 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:19:23.800 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T04:19:23.801 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:19:23.801 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T04:19:23.801 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:19:23.801 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T04:19:23.802 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:19:23.802 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:19:23.802 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:19:23.803 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:19:23 smithi047 ceph-mon[77147]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T04:19:23.803 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:19:23 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:19:24.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:19:23 smithi115 conmon[90393]: 2022-05-14T04:19:23.789+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:19:24.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:19:24 smithi115 ceph-mon[78593]: pgmap v9099: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:19:24.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:19:24 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:19:24.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:19:24 smithi047 ceph-mon[77147]: pgmap v9099: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:19:24.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:19:24 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:19:24.793 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:19:24 smithi047 conmon[78521]: 2022-05-14T04:19:24.527+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:19:24.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:19:24 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T04:19:24.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:19:24 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:19:24.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:19:24 smithi047 conmon[78521]: yield 2022-05-14T04:19:24.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:19:24 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:19:24.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:19:24 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:19:24.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:19:24 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:19:24.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:19:24 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T04:19:24.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:19:24 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:19:24.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:19:24 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T04:19:24.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:19:24 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:19:24.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:19:24 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T04:19:24.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:19:24 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:19:24.797 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:19:24 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T04:19:24.797 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:19:24 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:19:24.797 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:19:24 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T04:19:24.797 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:19:24 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:19:24.797 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:19:24 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:19:24.798 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:19:24 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:19:24.798 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:19:24 smithi047 conmon[78521]: 2022-05-14T04:19:24.528+0000 7f5344781700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T04:19:24.798 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:19:24 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T04:19:24.798 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:19:24 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:19:24.799 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:19:24 smithi047 conmon[78521]: yield 2022-05-14T04:19:24.799 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:19:24 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:19:24.799 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:19:24 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:19:24.799 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:19:24 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:19:24.800 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:19:24 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T04:19:24.800 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:19:24 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:19:24.800 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:19:24 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T04:19:24.800 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:19:24 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:19:24.800 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:19:24 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T04:19:24.801 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:19:24 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:19:24.801 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:19:24 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T04:19:24.801 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:19:24 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:19:24.801 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:19:24 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T04:19:24.802 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:19:24 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:19:24.802 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:19:24 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:19:24.802 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:19:24 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:19:24.803 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:19:24 smithi047 conmon[78521]: 2022-05-14T04:19:24.803 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:19:24 smithi047 conmon[78521]: During handling of the above exception, another exception occurred: 2022-05-14T04:19:24.803 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:19:24 smithi047 conmon[78521]: 2022-05-14T04:19:24.803 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:19:24 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T04:19:24.803 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:19:24 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T04:19:24.804 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:19:24 smithi047 conmon[78521]: return f(*arg) 2022-05-14T04:19:24.804 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:19:24 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T04:19:24.804 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:19:24 smithi047 conmon[78521]: self._write_client_files(client_files, host) 2022-05-14T04:19:24.805 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:19:24 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T04:19:24.805 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:19:24 smithi047 conmon[78521]: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T04:19:24.806 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:19:24 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T04:19:24.806 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:19:24 smithi047 conmon[78521]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T04:19:24.806 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:19:24 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T04:19:24.806 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:19:24 smithi047 conmon[78521]: return self.event_loop.get_result(coro) 2022-05-14T04:19:24.807 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:19:24 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T04:19:24.807 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:19:24 smithi047 conmon[78521]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T04:19:24.807 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:19:24 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T04:19:24.807 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:19:24 smithi047 conmon[78521]: return self.__get_result() 2022-05-14T04:19:24.808 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:19:24 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T04:19:24.808 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:19:24 smithi047 conmon[78521]: raise self._exception 2022-05-14T04:19:24.808 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:19:24 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T04:19:24.808 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:19:24 smithi047 conmon[78521]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T04:19:24.809 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:19:24 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T04:19:24.809 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:19:24 smithi047 conmon[78521]: conn = await self._remote_connection(host, addr) 2022-05-14T04:19:24.809 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:19:24 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T04:19:24.809 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:19:24 smithi047 conmon[78521]: raise 2022-05-14T04:19:24.810 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:19:24 smithi047 conmon[78521]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T04:19:24.810 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:19:24 smithi047 conmon[78521]: self.gen.throw(type, value, traceback) 2022-05-14T04:19:24.810 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:19:24 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T04:19:24.810 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:19:24 smithi047 conmon[78521]: raise OrchestratorError(msg) 2022-05-14T04:19:24.810 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:19:24 smithi047 conmon[78521]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:19:25.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:19:24 smithi115 conmon[90393]: 2022-05-14T04:19:24.788+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:19:25.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:19:25 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:19:25.793 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T04:19:25.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:19:25.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T04:19:25.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:19:25.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:19:25.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:19:25.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T04:19:25.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:19:25.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T04:19:25.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:19:25.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T04:19:25.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:19:25.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T04:19:25.797 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:19:25.797 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T04:19:25.797 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:19:25.798 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:19:25.799 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:19:25.799 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:19:25 smithi047 ceph-mon[77147]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T04:19:25.799 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T04:19:25.800 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:19:25.800 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T04:19:25.801 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:19:25.801 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:19:25.802 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:19:25.802 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T04:19:25.802 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:19:25.803 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T04:19:25.803 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:19:25.804 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T04:19:25.804 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:19:25.804 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T04:19:25.805 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:19:25.805 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T04:19:25.806 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:19:25.806 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:19:25.807 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:19:25.807 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T04:19:25.807 INFO:journalctl@ceph.mon.b.smithi047.stdout: During handling of the above exception, another exception occurred: 2022-05-14T04:19:25.808 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T04:19:25.808 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T04:19:25.809 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T04:19:25.809 INFO:journalctl@ceph.mon.b.smithi047.stdout: return f(*arg) 2022-05-14T04:19:25.809 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T04:19:25.810 INFO:journalctl@ceph.mon.b.smithi047.stdout: self._write_client_files(client_files, host) 2022-05-14T04:19:25.810 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T04:19:25.810 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T04:19:25.811 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T04:19:25.811 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T04:19:25.812 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T04:19:25.812 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.event_loop.get_result(coro) 2022-05-14T04:19:25.813 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T04:19:25.814 INFO:journalctl@ceph.mon.b.smithi047.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T04:19:25.814 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T04:19:25.815 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.__get_result() 2022-05-14T04:19:25.815 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T04:19:25.815 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise self._exception 2022-05-14T04:19:25.816 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T04:19:25.816 INFO:journalctl@ceph.mon.b.smithi047.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T04:19:25.816 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T04:19:25.816 INFO:journalctl@ceph.mon.b.smithi047.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T04:19:25.817 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T04:19:25.817 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise 2022-05-14T04:19:25.817 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T04:19:25.817 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.gen.throw(type, value, traceback) 2022-05-14T04:19:25.817 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T04:19:25.818 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OrchestratorError(msg) 2022-05-14T04:19:25.818 INFO:journalctl@ceph.mon.b.smithi047.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:19:25.818 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:19:25 smithi047 ceph-mon[77147]: pgmap v9100: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:19:25.818 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:19:25 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:19:25.819 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:19:25 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:19:25.819 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T04:19:25.820 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:19:25.820 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T04:19:25.820 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:19:25.820 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:19:25.821 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:19:25.821 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T04:19:25.821 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:19:25.821 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T04:19:25.821 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:19:25.822 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T04:19:25.822 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:19:25.822 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T04:19:25.822 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:19:25.823 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T04:19:25.823 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:19:25.823 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:19:25.823 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:19:25.824 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:19:25 smithi115 ceph-mon[78593]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T04:19:25.824 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T04:19:25.824 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:19:25.824 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T04:19:25.824 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:19:25.825 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:19:25.825 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:19:25.825 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T04:19:25.825 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:19:25.825 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T04:19:25.826 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:19:25.826 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T04:19:25.826 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:19:25.826 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T04:19:25.826 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:19:25.827 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T04:19:25.827 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:19:25.827 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:19:25.827 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:19:25.828 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T04:19:25.828 INFO:journalctl@ceph.mon.c.smithi115.stdout: During handling of the above exception, another exception occurred: 2022-05-14T04:19:25.828 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T04:19:25.828 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T04:19:25.829 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T04:19:25.829 INFO:journalctl@ceph.mon.c.smithi115.stdout: return f(*arg) 2022-05-14T04:19:25.829 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T04:19:25.829 INFO:journalctl@ceph.mon.c.smithi115.stdout: self._write_client_files(client_files, host) 2022-05-14T04:19:25.830 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T04:19:25.830 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T04:19:25.830 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T04:19:25.830 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T04:19:25.831 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T04:19:25.831 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.event_loop.get_result(coro) 2022-05-14T04:19:25.831 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T04:19:25.831 INFO:journalctl@ceph.mon.c.smithi115.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T04:19:25.831 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T04:19:25.832 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.__get_result() 2022-05-14T04:19:25.832 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T04:19:25.832 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise self._exception 2022-05-14T04:19:25.832 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T04:19:25.833 INFO:journalctl@ceph.mon.c.smithi115.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T04:19:25.833 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T04:19:25.833 INFO:journalctl@ceph.mon.c.smithi115.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T04:19:25.833 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T04:19:25.834 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise 2022-05-14T04:19:25.834 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T04:19:25.834 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.gen.throw(type, value, traceback) 2022-05-14T04:19:25.834 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T04:19:25.834 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OrchestratorError(msg) 2022-05-14T04:19:25.835 INFO:journalctl@ceph.mon.c.smithi115.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:19:25.835 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:19:25 smithi115 ceph-mon[78593]: pgmap v9100: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:19:25.835 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:19:25 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:19:26.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:19:25 smithi115 conmon[90393]: 2022-05-14T04:19:25.815+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:19:26.768 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:19:26 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:19:26.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:19:26 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:19:27.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:19:26 smithi115 conmon[90393]: 2022-05-14T04:19:26.765+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:19:27.725 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:19:27 smithi115 ceph-mon[78593]: pgmap v9101: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:19:27.725 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:19:27 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:19:27.726 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:19:27 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 17500 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:19:27.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:19:27 smithi047 ceph-mon[77147]: pgmap v9101: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:19:27.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:19:27 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:19:27.795 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:19:27 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 17500 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:19:28.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:19:27 smithi115 conmon[90393]: 2022-05-14T04:19:27.721+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:19:28.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:19:28 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:19:28.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:19:28 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:19:28.860 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:19:28 smithi115 conmon[90393]: 2022-05-14T04:19:28.676+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:19:29.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:19:29 smithi047 ceph-mon[77147]: pgmap v9102: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:19:29.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:19:29 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:19:29.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:19:29 smithi115 ceph-mon[78593]: pgmap v9102: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:19:29.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:19:29 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:19:29.860 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:19:29 smithi115 conmon[90393]: 2022-05-14T04:19:29.714+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:19:30.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:19:30 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:19:30.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:19:30 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:19:30.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:19:30 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:19:30.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:19:30 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:19:30.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:19:30 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:19:30.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:19:30 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:19:30.861 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:19:30 smithi115 conmon[90393]: 2022-05-14T04:19:30.675+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:19:31.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:19:31 smithi047 ceph-mon[77147]: pgmap v9103: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:19:31.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:19:31 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:19:31.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:19:31 smithi115 ceph-mon[78593]: pgmap v9103: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:19:31.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:19:31 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:19:31.860 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:19:31 smithi115 conmon[90393]: 2022-05-14T04:19:31.709+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:19:32.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:19:32 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:19:32.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:19:32 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 17505 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:19:32.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:19:32 smithi115 conmon[90393]: 2022-05-14T04:19:32.670+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:19:32.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:19:32 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:19:32.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:19:32 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 17505 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:19:33.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:19:33 smithi047 ceph-mon[77147]: pgmap v9104: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:19:33.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:19:33 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:19:33.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:19:33 smithi115 conmon[90393]: 2022-05-14T04:19:33.627+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:19:33.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:19:33 smithi115 ceph-mon[78593]: pgmap v9104: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:19:33.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:19:33 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:19:34.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:19:34 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:19:34.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:19:34 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:19:34.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:19:34 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:19:34.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:19:34 smithi115 conmon[90393]: 2022-05-14T04:19:34.645+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:19:34.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:19:34 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:19:34.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:19:34 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:19:34.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:19:34 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:19:35.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:19:35 smithi047 ceph-mon[77147]: pgmap v9105: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:19:35.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:19:35 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:19:35.863 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:19:35 smithi115 conmon[90393]: 2022-05-14T04:19:35.602+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:19:35.863 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:19:35 smithi115 ceph-mon[78593]: pgmap v9105: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:19:35.864 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:19:35 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:19:36.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:19:36 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:19:36.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:19:36 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:19:36.860 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:19:36 smithi115 conmon[90393]: 2022-05-14T04:19:36.552+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:19:37.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:19:37 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:19:37.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:19:37 smithi047 ceph-mon[77147]: pgmap v9106: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:19:37.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:19:37 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 17510 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:19:37.860 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:19:37 smithi115 conmon[90393]: 2022-05-14T04:19:37.602+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:19:37.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:19:37 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:19:37.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:19:37 smithi115 ceph-mon[78593]: pgmap v9106: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:19:37.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:19:37 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 17510 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:19:38.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:19:38 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:19:38.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:19:38 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:19:38.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:19:38 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:19:38.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:19:38 smithi115 conmon[90393]: 2022-05-14T04:19:38.629+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:19:38.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:19:38 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:19:38.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:19:38 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:19:38.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:19:38 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:19:39.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:19:39 smithi047 ceph-mon[77147]: pgmap v9107: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:19:39.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:19:39 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:19:39.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:19:39 smithi115 conmon[90393]: 2022-05-14T04:19:39.608+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:19:39.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:19:39 smithi115 ceph-mon[78593]: pgmap v9107: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:19:39.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:19:39 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:19:40.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:19:40 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:19:40.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:19:40 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:19:40.860 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:19:40 smithi115 conmon[90393]: 2022-05-14T04:19:40.604+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:19:41.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:19:41 smithi047 ceph-mon[77147]: pgmap v9108: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:19:41.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:19:41 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:19:41.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:19:41 smithi115 conmon[90393]: 2022-05-14T04:19:41.605+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:19:41.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:19:41 smithi115 ceph-mon[78593]: pgmap v9108: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:19:41.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:19:41 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:19:42.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:19:42 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:19:42.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:19:42 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:19:42.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:19:42 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:19:42.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:19:42 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 17515 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:19:42.860 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:19:42 smithi115 conmon[90393]: 2022-05-14T04:19:42.637+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:19:42.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:19:42 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:19:42.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:19:42 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:19:42.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:19:42 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:19:42.862 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:19:42 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 17515 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:19:43.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:19:43 smithi047 ceph-mon[77147]: pgmap v9109: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:19:43.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:19:43 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:19:43.860 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:19:43 smithi115 conmon[90393]: 2022-05-14T04:19:43.644+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:19:43.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:19:43 smithi115 ceph-mon[78593]: pgmap v9109: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:19:43.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:19:43 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:19:44.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:19:44 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:19:44.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:19:44 smithi115 conmon[90393]: 2022-05-14T04:19:44.693+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:19:44.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:19:44 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:19:45.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:19:45 smithi047 ceph-mon[77147]: pgmap v9110: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:19:45.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:19:45 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:19:45.860 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:19:45 smithi115 conmon[90393]: 2022-05-14T04:19:45.674+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:19:45.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:19:45 smithi115 ceph-mon[78593]: pgmap v9110: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:19:45.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:19:45 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:19:46.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:19:46 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:19:46.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:19:46 smithi115 conmon[90393]: 2022-05-14T04:19:46.705+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:19:46.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:19:46 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:19:47.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:19:47 smithi047 ceph-mon[77147]: pgmap v9111: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:19:47.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:19:47 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:19:47.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:19:47 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 17520 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:19:47.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:19:47 smithi115 conmon[90393]: 2022-05-14T04:19:47.677+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:19:47.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:19:47 smithi115 ceph-mon[78593]: pgmap v9111: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:19:47.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:19:47 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:19:47.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:19:47 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 17520 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:19:48.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:19:48 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:19:48.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:19:48 smithi115 conmon[90393]: 2022-05-14T04:19:48.725+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:19:48.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:19:48 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:19:49.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:19:49 smithi115 conmon[90393]: 2022-05-14T04:19:49.774+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:19:49.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:19:49 smithi115 ceph-mon[78593]: pgmap v9112: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:19:49.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:19:49 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:19:50.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:19:49 smithi047 ceph-mon[77147]: pgmap v9112: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:19:50.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:19:49 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:19:50.860 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:19:50 smithi115 conmon[90393]: 2022-05-14T04:19:50.779+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:19:50.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:19:50 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:19:50.862 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:19:50 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:19:50.862 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:19:50 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:19:51.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:19:50 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:19:51.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:19:50 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:19:51.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:19:50 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:19:51.827 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:19:51 smithi115 ceph-mon[78593]: pgmap v9113: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:19:51.828 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:19:51 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:19:52.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:19:51 smithi047 ceph-mon[77147]: pgmap v9113: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:19:52.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:19:51 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:19:52.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:19:51 smithi115 conmon[90393]: 2022-05-14T04:19:51.824+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:19:52.837 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:19:52 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:19:52.837 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:19:52 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 17525 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:19:53.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:19:52 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:19:53.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:19:52 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 17525 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:19:53.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:19:52 smithi115 conmon[90393]: 2022-05-14T04:19:52.834+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:19:53.851 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:19:53 smithi115 ceph-mon[78593]: pgmap v9114: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:19:53.851 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:19:53 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:19:54.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:19:53 smithi047 ceph-mon[77147]: pgmap v9114: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:19:54.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:19:53 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:19:54.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:19:53 smithi115 conmon[90393]: 2022-05-14T04:19:53.848+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:19:54.809 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:19:54 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:19:54.810 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:19:54 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:19:54.810 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:19:54 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:19:55.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:19:54 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:19:55.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:19:54 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:19:55.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:19:54 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:19:55.110 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:19:54 smithi115 conmon[90393]: 2022-05-14T04:19:54.806+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:19:55.818 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:19:55 smithi115 ceph-mon[78593]: pgmap v9115: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:19:55.818 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:19:55 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:19:56.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:19:55 smithi047 ceph-mon[77147]: pgmap v9115: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:19:56.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:19:55 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:19:56.110 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:19:55 smithi115 conmon[90393]: 2022-05-14T04:19:55.814+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:19:56.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:19:56 smithi115 conmon[90393]: 2022-05-14T04:19:56.799+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:19:56.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:19:56 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:19:57.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:19:56 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:19:57.821 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:19:57 smithi115 ceph-mon[78593]: pgmap v9116: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:19:57.822 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:19:57 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:19:57.822 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:19:57 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 17530 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:19:58.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:19:57 smithi047 ceph-mon[77147]: pgmap v9116: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:19:58.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:19:57 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:19:58.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:19:57 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 17530 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:19:58.110 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:19:57 smithi115 conmon[90393]: 2022-05-14T04:19:57.818+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:19:58.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:19:58 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:19:58.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:19:58 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:19:58.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:19:58 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:19:58.860 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:19:58 smithi115 conmon[90393]: 2022-05-14T04:19:58.800+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:19:59.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:19:58 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:19:59.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:19:58 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:19:59.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:19:58 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:19:59.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:19:59 smithi115 conmon[90393]: 2022-05-14T04:19:59.772+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:19:59.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:19:59 smithi115 ceph-mon[78593]: pgmap v9117: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:19:59.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:19:59 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:20:00.002 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:19:59 smithi047 ceph-mon[77147]: pgmap v9117: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:20:00.002 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:19:59 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:20:00.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:20:00 smithi047 conmon[77123]: 2022-05-14T04:19:59.999+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : Health detail: HEALTH_ERR 1 filesystem is degraded; 1 mds daemon damaged; 1 MDSs are read only; 1 MDSs report slow metadata IOs; 1/3 mons down, quorum b,c; Reduced data availability: 1 pg inactive; 9 slow ops, oldest one blocked for 17530 sec, osd.5 has slow ops 2022-05-14T04:20:00.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:20:00 smithi047 conmon[77123]: 2022-05-14T04:19:59.999+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : [WRN] FS_DEGRADED: 1 filesystem is degraded 2022-05-14T04:20:00.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:20:00 smithi047 conmon[77123]: 2022-05-14T04:20:00.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:20:00 smithi047 conmon[77123]: 2022-05-14T04:19:59.999+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : fs cephfs is degraded 2022-05-14T04:20:00.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:20:00 smithi047 conmon[77123]: 2022-05-14T04:19:59.999+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : [ERR] MDS_DAMAGE: 1 mds daemon damaged 2022-05-14T04:20:00.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:20:00 smithi047 conmon[77123]: 2022-05-14T04:19:59.999+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : fs cephfs mds.2 is damaged 2022-05-14T04:20:00.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:20:00 smithi047 conmon[77123]: 2022-05-14T04:19:59.999+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : [WRN] MDS_READ_ONLY: 1 MDSs are read only 2022-05-14T04:20:00.295 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:20:00 smithi047 conmon[77123]: 2022-05-14T04:19:59.999+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : mds.l(mds.0): MDS in read-only mode 2022-05-14T04:20:00.295 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:20:00 smithi047 conmon[77123]: 2022-05-14T04:19:59.999+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : [WRN] MDS_SLOW_METADATA_IO: 1 MDSs report slow metadata IOs 2022-05-14T04:20:00.295 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:20:00 smithi047 conmon[77123]: 2022-05-14T04:20:00.000+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : mds.l(mds.0): 4 slow metadata IOs are blocked > 180 secs, oldest blocked for 17833 secs 2022-05-14T04:20:00.295 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:20:00 smithi047 conmon[77123]: 2022-05-14T04:20:00.000+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : [WRN] MON_DOWN: 1/3 mons down, quorum b,c 2022-05-14T04:20:00.296 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:20:00 smithi047 conmon[77123]: 2022-05-14T04:20:00.000+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : mon.a (rank 0) addr [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] is down (out of quorum) 2022-05-14T04:20:00.296 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:20:00 smithi047 conmon[77123]: 2022-05-14T04:20:00.000+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : [WRN] PG_AVAILABILITY: Reduced data availability: 1 pg inactive 2022-05-14T04:20:00.296 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:20:00 smithi047 conmon[77123]: 2022-05-14T04:20:00.000+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : pg 2.1 is stuck inactive for 5h, current state unknown, last acting [] 2022-05-14T04:20:00.296 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:20:00 smithi047 conmon[77123]: 2022-05-14T04:20:00.000+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : [WRN] SLOW_OPS: 9 slow ops, oldest one blocked for 17530 sec, osd.5 has slow ops 2022-05-14T04:20:00.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:20:00 smithi115 conmon[90393]: 2022-05-14T04:20:00.725+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:20:00.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:20:00 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:20:00.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:20:00 smithi115 ceph-mon[78593]: Health detail: HEALTH_ERR 1 filesystem is degraded; 1 mds daemon damaged; 1 MDSs are read only; 1 MDSs report slow metadata IOs; 1/3 mons down, quorum b,c; Reduced data availability: 1 pg inactive; 9 slow ops, oldest one blocked for 17530 sec, osd.5 has slow ops 2022-05-14T04:20:00.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:20:00 smithi115 ceph-mon[78593]: [WRN] FS_DEGRADED: 1 filesystem is degraded 2022-05-14T04:20:00.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:20:00 smithi115 ceph-mon[78593]: fs cephfs is degraded 2022-05-14T04:20:00.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:20:00 smithi115 ceph-mon[78593]: [ERR] MDS_DAMAGE: 1 mds daemon damaged 2022-05-14T04:20:00.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:20:00 smithi115 ceph-mon[78593]: fs cephfs mds.2 is damaged 2022-05-14T04:20:00.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:20:00 smithi115 ceph-mon[78593]: [WRN] MDS_READ_ONLY: 1 MDSs are read only 2022-05-14T04:20:00.862 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:20:00 smithi115 ceph-mon[78593]: mds.l(mds.0): MDS in read-only mode 2022-05-14T04:20:00.862 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:20:00 smithi115 ceph-mon[78593]: [WRN] MDS_SLOW_METADATA_IO: 1 MDSs report slow metadata IOs 2022-05-14T04:20:00.862 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:20:00 smithi115 ceph-mon[78593]: mds.l(mds.0): 4 slow metadata IOs are blocked > 180 secs, oldest blocked for 17833 secs 2022-05-14T04:20:00.862 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:20:00 smithi115 ceph-mon[78593]: [WRN] MON_DOWN: 1/3 mons down, quorum b,c 2022-05-14T04:20:00.863 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:20:00 smithi115 ceph-mon[78593]: mon.a (rank 0) addr [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] is down (out of quorum) 2022-05-14T04:20:00.863 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:20:00 smithi115 ceph-mon[78593]: [WRN] PG_AVAILABILITY: Reduced data availability: 1 pg inactive 2022-05-14T04:20:00.863 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:20:00 smithi115 ceph-mon[78593]: pg 2.1 is stuck inactive for 5h, current state unknown, last acting [] 2022-05-14T04:20:00.863 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:20:00 smithi115 ceph-mon[78593]: [WRN] SLOW_OPS: 9 slow ops, oldest one blocked for 17530 sec, osd.5 has slow ops 2022-05-14T04:20:01.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:20:00 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:20:01.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:20:00 smithi047 ceph-mon[77147]: Health detail: HEALTH_ERR 1 filesystem is degraded; 1 mds daemon damaged; 1 MDSs are read only; 1 MDSs report slow metadata IOs; 1/3 mons down, quorum b,c; Reduced data availability: 1 pg inactive; 9 slow ops, oldest one blocked for 17530 sec, osd.5 has slow ops 2022-05-14T04:20:01.045 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:20:00 smithi047 ceph-mon[77147]: [WRN] FS_DEGRADED: 1 filesystem is degraded 2022-05-14T04:20:01.046 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:20:00 smithi047 ceph-mon[77147]: fs cephfs is degraded 2022-05-14T04:20:01.047 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:20:00 smithi047 ceph-mon[77147]: [ERR] MDS_DAMAGE: 1 mds daemon damaged 2022-05-14T04:20:01.047 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:20:00 smithi047 ceph-mon[77147]: fs cephfs mds.2 is damaged 2022-05-14T04:20:01.047 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:20:00 smithi047 ceph-mon[77147]: [WRN] MDS_READ_ONLY: 1 MDSs are read only 2022-05-14T04:20:01.047 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:20:00 smithi047 ceph-mon[77147]: mds.l(mds.0): MDS in read-only mode 2022-05-14T04:20:01.048 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:20:00 smithi047 ceph-mon[77147]: [WRN] MDS_SLOW_METADATA_IO: 1 MDSs report slow metadata IOs 2022-05-14T04:20:01.048 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:20:00 smithi047 ceph-mon[77147]: mds.l(mds.0): 4 slow metadata IOs are blocked > 180 secs, oldest blocked for 17833 secs 2022-05-14T04:20:01.048 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:20:00 smithi047 ceph-mon[77147]: [WRN] MON_DOWN: 1/3 mons down, quorum b,c 2022-05-14T04:20:01.049 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:20:00 smithi047 ceph-mon[77147]: mon.a (rank 0) addr [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] is down (out of quorum) 2022-05-14T04:20:01.049 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:20:00 smithi047 ceph-mon[77147]: [WRN] PG_AVAILABILITY: Reduced data availability: 1 pg inactive 2022-05-14T04:20:01.049 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:20:00 smithi047 ceph-mon[77147]: pg 2.1 is stuck inactive for 5h, current state unknown, last acting [] 2022-05-14T04:20:01.049 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:20:00 smithi047 ceph-mon[77147]: [WRN] SLOW_OPS: 9 slow ops, oldest one blocked for 17530 sec, osd.5 has slow ops 2022-05-14T04:20:01.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:20:01 smithi115 conmon[90393]: 2022-05-14T04:20:01.758+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:20:01.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:20:01 smithi115 ceph-mon[78593]: pgmap v9118: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:20:01.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:20:01 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:20:02.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:20:01 smithi047 ceph-mon[77147]: pgmap v9118: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:20:02.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:20:01 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:20:02.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:20:02 smithi115 conmon[90393]: 2022-05-14T04:20:02.760+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:20:02.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:20:02 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:20:02.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:20:02 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:20:02.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:20:02 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:20:02.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:20:02 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 17535 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:20:03.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:20:02 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:20:03.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:20:02 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:20:03.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:20:02 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:20:03.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:20:02 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 17535 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:20:03.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:20:03 smithi115 conmon[90393]: 2022-05-14T04:20:03.807+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:20:03.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:20:03 smithi115 ceph-mon[78593]: pgmap v9119: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:20:03.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:20:03 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:20:04.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:20:03 smithi047 ceph-mon[77147]: pgmap v9119: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:20:04.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:20:03 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:20:04.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:20:04 smithi115 conmon[90393]: 2022-05-14T04:20:04.782+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:20:04.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:20:04 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:20:05.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:20:04 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:20:05.860 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:20:05 smithi115 conmon[90393]: 2022-05-14T04:20:05.780+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:20:05.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:20:05 smithi115 ceph-mon[78593]: pgmap v9120: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:20:05.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:20:05 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:20:06.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:20:05 smithi047 ceph-mon[77147]: pgmap v9120: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:20:06.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:20:05 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:20:06.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:20:06 smithi115 conmon[90393]: 2022-05-14T04:20:06.734+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:20:06.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:20:06 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:20:07.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:20:06 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:20:07.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:20:07 smithi115 conmon[90393]: 2022-05-14T04:20:07.698+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:20:07.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:20:07 smithi115 ceph-mon[78593]: pgmap v9121: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:20:07.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:20:07 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:20:07.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:20:07 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 17540 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:20:08.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:20:07 smithi047 ceph-mon[77147]: pgmap v9121: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:20:08.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:20:07 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:20:08.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:20:07 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 17540 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:20:08.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:20:08 smithi115 conmon[90393]: 2022-05-14T04:20:08.697+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:20:08.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:20:08 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:20:09.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:20:08 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:20:09.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:20:09 smithi115 conmon[90393]: 2022-05-14T04:20:09.702+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:20:09.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:20:09 smithi115 ceph-mon[78593]: pgmap v9122: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:20:09.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:20:09 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:20:10.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:20:09 smithi047 ceph-mon[77147]: pgmap v9122: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:20:10.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:20:09 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:20:11.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:20:10 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:20:11.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:20:10 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:20:11.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:20:10 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:20:11.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:20:10 smithi115 conmon[90393]: 2022-05-14T04:20:10.707+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:20:11.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:20:10 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:20:11.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:20:10 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:20:11.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:20:10 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:20:12.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:20:11 smithi047 ceph-mon[77147]: pgmap v9123: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:20:12.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:20:11 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:20:12.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:20:11 smithi115 conmon[90393]: 2022-05-14T04:20:11.699+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:20:12.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:20:11 smithi115 ceph-mon[78593]: pgmap v9123: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:20:12.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:20:11 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:20:13.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:20:12 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:20:13.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:20:12 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 17545 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:20:13.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:20:12 smithi115 conmon[90393]: 2022-05-14T04:20:12.738+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:20:13.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:20:12 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:20:13.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:20:12 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 17545 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:20:14.110 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:20:13 smithi115 conmon[90393]: 2022-05-14T04:20:13.788+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:20:14.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:20:14 smithi115 ceph-mon[78593]: pgmap v9124: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:20:14.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:20:14 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:20:14.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:20:14 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:20:14.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:20:14 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:20:14.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:20:14 smithi047 ceph-mon[77147]: pgmap v9124: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:20:14.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:20:14 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:20:14.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:20:14 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:20:14.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:20:14 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:20:15.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:20:14 smithi115 conmon[90393]: 2022-05-14T04:20:14.832+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:20:15.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:20:15 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:20:15.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:20:15 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T04:20:15.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:20:15 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T04:20:15.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:20:15 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:20:15.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:20:15 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T04:20:15.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:20:15 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T04:20:16.330 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:20:15 smithi115 conmon[90393]: 2022-05-14T04:20:15.867+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:20:16.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:20:16 smithi115 ceph-mon[78593]: pgmap v9125: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:20:16.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:20:16 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:20:16.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:20:16 smithi047 ceph-mon[77147]: pgmap v9125: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:20:16.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:20:16 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:20:17.334 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:20:16 smithi115 conmon[90393]: 2022-05-14T04:20:16.879+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:20:17.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:20:17 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:20:17.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:20:17 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 17550 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:20:17.795 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:20:17 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:20:17.795 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:20:17 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 17550 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:20:18.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:20:17 smithi115 conmon[90393]: 2022-05-14T04:20:17.851+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:20:18.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:20:18 smithi115 ceph-mon[78593]: pgmap v9126: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:20:18.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:20:18 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:20:18.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:20:18 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:20:18.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:20:18 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:20:18.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:20:18 smithi047 ceph-mon[77147]: pgmap v9126: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:20:18.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:20:18 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:20:18.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:20:18 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:20:18.795 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:20:18 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:20:19.340 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:20:18 smithi115 conmon[90393]: 2022-05-14T04:20:18.899+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:20:19.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:20:19 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:20:19.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:20:19 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:20:20.342 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:20:19 smithi115 conmon[90393]: 2022-05-14T04:20:19.877+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:20:20.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:20:20 smithi115 ceph-mon[78593]: pgmap v9127: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:20:20.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:20:20 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:20:20.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:20:20 smithi047 ceph-mon[77147]: pgmap v9127: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:20:20.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:20:20 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:20:21.348 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:20:20 smithi115 conmon[90393]: 2022-05-14T04:20:20.920+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:20:21.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:20:21 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:20:21.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:20:21 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:20:22.349 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:20:21 smithi115 conmon[90393]: 2022-05-14T04:20:21.933+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:20:22.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:20:22 smithi115 ceph-mon[78593]: pgmap v9128: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:20:22.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:20:22 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:20:22.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:20:22 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:20:22.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:20:22 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:20:22.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:20:22 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 17555 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:20:22.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:20:22 smithi047 ceph-mon[77147]: pgmap v9128: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:20:22.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:20:22 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:20:22.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:20:22 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:20:22.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:20:22 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:20:22.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:20:22 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 17555 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:20:23.354 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:20:22 smithi115 conmon[90393]: 2022-05-14T04:20:22.910+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:20:23.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:20:23 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:20:23.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:20:23 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:20:24.356 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:20:23 smithi115 conmon[90393]: 2022-05-14T04:20:23.889+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:20:24.605 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:20:24 smithi047 ceph-mon[77147]: pgmap v9129: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:20:24.606 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:20:24 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:20:24.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:20:24 smithi115 ceph-mon[78593]: pgmap v9129: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:20:24.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:20:24 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:20:25.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:20:24 smithi115 conmon[90393]: 2022-05-14T04:20:24.873+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:20:25.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:20:25 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:20:25.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:20:25 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T04:20:25.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:20:25 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T04:20:25.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:20:25 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T04:20:25.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:20:25 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:20:25.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:20:25 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T04:20:25.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:20:25 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T04:20:25.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:20:25 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T04:20:26.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:20:25 smithi115 conmon[90393]: 2022-05-14T04:20:25.862+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:20:26.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:20:26 smithi047 ceph-mon[77147]: pgmap v9130: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:20:26.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:20:26 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:20:26.818 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:20:26 smithi115 ceph-mon[78593]: pgmap v9130: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:20:26.818 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:20:26 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:20:27.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:20:26 smithi115 conmon[90393]: 2022-05-14T04:20:26.815+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:20:27.636 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:20:27 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:20:27.636 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:20:27 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 17560 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:20:27.769 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:20:27 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:20:27.770 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:20:27 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 17560 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:20:28.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:20:27 smithi047 conmon[78521]: 2022-05-14T04:20:27.631+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:20:28.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:20:27 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T04:20:28.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:20:27 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:20:28.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:20:27 smithi047 conmon[78521]: yield 2022-05-14T04:20:28.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:20:27 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:20:28.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:20:27 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:20:28.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:20:27 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:20:28.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:20:27 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T04:20:28.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:20:27 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:20:28.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:20:27 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T04:20:28.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:20:27 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:20:28.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:20:27 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T04:20:28.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:20:27 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:20:28.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:20:27 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T04:20:28.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:20:27 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:20:28.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:20:27 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T04:20:28.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:20:27 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:20:28.048 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:20:27 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:20:28.048 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:20:27 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:20:28.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:20:27 smithi115 conmon[90393]: 2022-05-14T04:20:27.766+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:20:28.749 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:20:28 smithi115 ceph-mon[78593]: pgmap v9131: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:20:28.749 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:20:28 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:20:28.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:20:28 smithi047 ceph-mon[77147]: pgmap v9131: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:20:28.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:20:28 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:20:29.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:20:28 smithi115 conmon[90393]: 2022-05-14T04:20:28.746+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:20:29.740 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:20:29 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:20:29.740 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T04:20:29.741 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:20:29.741 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T04:20:29.741 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:20:29.741 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:20:29.741 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:20:29.742 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T04:20:29.742 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:20:29.742 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T04:20:29.742 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:20:29.743 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T04:20:29.743 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:20:29.743 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T04:20:29.743 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:20:29.743 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T04:20:29.744 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:20:29.744 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:20:29.744 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:20:29.744 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:20:29 smithi115 ceph-mon[78593]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T04:20:29.744 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:20:29 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:20:29.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:20:29 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:20:29.793 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T04:20:29.793 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:20:29.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T04:20:29.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:20:29.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:20:29.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:20:29.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T04:20:29.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:20:29.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T04:20:29.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:20:29.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T04:20:29.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:20:29.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T04:20:29.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:20:29.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T04:20:29.797 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:20:29.797 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:20:29.797 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:20:29.797 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:20:29 smithi047 ceph-mon[77147]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T04:20:29.797 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:20:29 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:20:30.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:20:29 smithi115 conmon[90393]: 2022-05-14T04:20:29.729+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:20:30.686 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:20:30 smithi115 ceph-mon[78593]: pgmap v9132: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:20:30.687 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:20:30 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:20:30.687 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:20:30 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:20:30.687 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:20:30 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:20:30.707 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:20:30 smithi047 ceph-mon[77147]: pgmap v9132: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:20:30.707 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:20:30 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:20:30.707 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:20:30 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:20:30.708 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:20:30 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:20:31.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:20:30 smithi047 conmon[78521]: 2022-05-14T04:20:30.703+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:20:31.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:20:30 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T04:20:31.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:20:30 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:20:31.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:20:30 smithi047 conmon[78521]: yield 2022-05-14T04:20:31.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:20:30 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:20:31.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:20:30 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:20:31.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:20:30 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:20:31.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:20:30 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T04:20:31.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:20:30 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:20:31.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:20:30 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T04:20:31.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:20:30 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:20:31.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:20:30 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T04:20:31.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:20:30 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:20:31.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:20:30 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T04:20:31.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:20:30 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:20:31.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:20:30 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T04:20:31.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:20:30 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:20:31.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:20:30 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:20:31.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:20:30 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:20:31.048 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:20:30 smithi047 conmon[78521]: 2022-05-14T04:20:30.704+0000 7f533ff78700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T04:20:31.048 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:20:30 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T04:20:31.048 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:20:30 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:20:31.048 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:20:30 smithi047 conmon[78521]: yield 2022-05-14T04:20:31.049 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:20:30 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:20:31.049 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:20:30 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:20:31.049 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:20:30 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:20:31.049 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:20:30 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T04:20:31.049 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:20:30 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:20:31.050 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:20:30 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T04:20:31.050 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:20:30 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:20:31.050 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:20:30 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T04:20:31.050 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:20:30 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:20:31.051 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:20:30 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T04:20:31.051 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:20:30 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:20:31.051 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:20:30 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T04:20:31.051 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:20:30 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:20:31.051 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:20:30 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:20:31.052 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:20:30 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:20:31.052 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:20:30 smithi047 conmon[78521]: 2022-05-14T04:20:31.052 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:20:30 smithi047 conmon[78521]: During handling of the above exception, another exception occurred: 2022-05-14T04:20:31.052 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:20:30 smithi047 conmon[78521]: 2022-05-14T04:20:31.052 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:20:30 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T04:20:31.053 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:20:30 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T04:20:31.053 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:20:30 smithi047 conmon[78521]: return f(*arg) 2022-05-14T04:20:31.053 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:20:30 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T04:20:31.053 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:20:30 smithi047 conmon[78521]: self._write_client_files(client_files, host) 2022-05-14T04:20:31.054 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:20:30 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T04:20:31.054 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:20:30 smithi047 conmon[78521]: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T04:20:31.054 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:20:30 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T04:20:31.054 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:20:30 smithi047 conmon[78521]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T04:20:31.055 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:20:30 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T04:20:31.055 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:20:30 smithi047 conmon[78521]: return self.event_loop.get_result(coro) 2022-05-14T04:20:31.055 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:20:30 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T04:20:31.055 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:20:30 smithi047 conmon[78521]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T04:20:31.055 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:20:30 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T04:20:31.056 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:20:30 smithi047 conmon[78521]: return self.__get_result() 2022-05-14T04:20:31.056 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:20:30 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T04:20:31.056 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:20:30 smithi047 conmon[78521]: raise self._exception 2022-05-14T04:20:31.056 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:20:30 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T04:20:31.056 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:20:30 smithi047 conmon[78521]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T04:20:31.057 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:20:30 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T04:20:31.057 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:20:30 smithi047 conmon[78521]: conn = await self._remote_connection(host, addr) 2022-05-14T04:20:31.057 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:20:30 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T04:20:31.057 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:20:30 smithi047 conmon[78521]: raise 2022-05-14T04:20:31.058 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:20:30 smithi047 conmon[78521]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T04:20:31.058 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:20:30 smithi047 conmon[78521]: self.gen.throw(type, value, traceback) 2022-05-14T04:20:31.058 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:20:30 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T04:20:31.058 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:20:30 smithi047 conmon[78521]: raise OrchestratorError(msg) 2022-05-14T04:20:31.058 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:20:30 smithi047 conmon[78521]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:20:31.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:20:30 smithi115 conmon[90393]: 2022-05-14T04:20:30.684+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:20:31.732 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:20:31 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:20:31.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:20:31 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:20:32.110 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:20:31 smithi115 conmon[90393]: 2022-05-14T04:20:31.728+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:20:32.710 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:20:32 smithi115 ceph-mon[78593]: pgmap v9133: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:20:32.711 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:20:32 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:20:32.711 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:20:32 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:20:32.711 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T04:20:32.711 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:20:32.712 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T04:20:32.712 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:20:32.713 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:20:32.713 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:20:32.713 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T04:20:32.713 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:20:32.714 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T04:20:32.714 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:20:32.714 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T04:20:32.714 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:20:32.714 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T04:20:32.715 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:20:32.715 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T04:20:32.715 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:20:32.716 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:20:32.716 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:20:32.717 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:20:32 smithi115 ceph-mon[78593]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T04:20:32.717 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T04:20:32.717 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:20:32.717 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T04:20:32.718 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:20:32.718 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:20:32.718 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:20:32.719 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T04:20:32.719 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:20:32.719 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T04:20:32.719 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:20:32.719 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T04:20:32.720 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:20:32.720 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T04:20:32.720 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:20:32.720 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T04:20:32.720 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:20:32.721 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:20:32.721 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:20:32.721 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T04:20:32.721 INFO:journalctl@ceph.mon.c.smithi115.stdout: During handling of the above exception, another exception occurred: 2022-05-14T04:20:32.721 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T04:20:32.722 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T04:20:32.722 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T04:20:32.722 INFO:journalctl@ceph.mon.c.smithi115.stdout: return f(*arg) 2022-05-14T04:20:32.722 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T04:20:32.723 INFO:journalctl@ceph.mon.c.smithi115.stdout: self._write_client_files(client_files, host) 2022-05-14T04:20:32.723 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T04:20:32.723 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T04:20:32.723 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T04:20:32.723 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T04:20:32.724 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T04:20:32.724 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.event_loop.get_result(coro) 2022-05-14T04:20:32.724 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T04:20:32.724 INFO:journalctl@ceph.mon.c.smithi115.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T04:20:32.725 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T04:20:32.725 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.__get_result() 2022-05-14T04:20:32.725 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T04:20:32.725 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise self._exception 2022-05-14T04:20:32.725 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T04:20:32.726 INFO:journalctl@ceph.mon.c.smithi115.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T04:20:32.726 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T04:20:32.727 INFO:journalctl@ceph.mon.c.smithi115.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T04:20:32.727 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T04:20:32.727 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise 2022-05-14T04:20:32.727 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T04:20:32.728 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.gen.throw(type, value, traceback) 2022-05-14T04:20:32.728 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T04:20:32.728 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OrchestratorError(msg) 2022-05-14T04:20:32.728 INFO:journalctl@ceph.mon.c.smithi115.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:20:32.728 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:20:32 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 17565 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:20:32.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:20:32 smithi047 ceph-mon[77147]: pgmap v9133: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:20:32.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:20:32 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:20:32.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:20:32 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:20:32.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T04:20:32.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:20:32.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T04:20:32.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:20:32.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:20:32.797 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:20:32.797 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T04:20:32.797 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:20:32.798 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T04:20:32.799 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:20:32.799 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T04:20:32.799 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:20:32.799 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T04:20:32.800 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:20:32.801 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T04:20:32.802 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:20:32.802 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:20:32.802 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:20:32.802 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:20:32 smithi047 ceph-mon[77147]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T04:20:32.802 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T04:20:32.803 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:20:32.803 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T04:20:32.803 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:20:32.803 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:20:32.803 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:20:32.804 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T04:20:32.804 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:20:32.804 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T04:20:32.805 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:20:32.805 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T04:20:32.805 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:20:32.805 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T04:20:32.805 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:20:32.806 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T04:20:32.806 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:20:32.806 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:20:32.806 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:20:32.807 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T04:20:32.807 INFO:journalctl@ceph.mon.b.smithi047.stdout: During handling of the above exception, another exception occurred: 2022-05-14T04:20:32.807 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T04:20:32.807 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T04:20:32.808 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T04:20:32.808 INFO:journalctl@ceph.mon.b.smithi047.stdout: return f(*arg) 2022-05-14T04:20:32.808 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T04:20:32.809 INFO:journalctl@ceph.mon.b.smithi047.stdout: self._write_client_files(client_files, host) 2022-05-14T04:20:32.809 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T04:20:32.809 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T04:20:32.809 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T04:20:32.810 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T04:20:32.810 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T04:20:32.810 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.event_loop.get_result(coro) 2022-05-14T04:20:32.810 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T04:20:32.810 INFO:journalctl@ceph.mon.b.smithi047.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T04:20:32.812 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T04:20:32.812 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.__get_result() 2022-05-14T04:20:32.812 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T04:20:32.812 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise self._exception 2022-05-14T04:20:32.813 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T04:20:32.813 INFO:journalctl@ceph.mon.b.smithi047.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T04:20:32.813 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T04:20:32.813 INFO:journalctl@ceph.mon.b.smithi047.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T04:20:32.813 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T04:20:32.814 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise 2022-05-14T04:20:32.814 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T04:20:32.814 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.gen.throw(type, value, traceback) 2022-05-14T04:20:32.814 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T04:20:32.815 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OrchestratorError(msg) 2022-05-14T04:20:32.815 INFO:journalctl@ceph.mon.b.smithi047.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:20:32.815 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:20:32 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 17565 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:20:33.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:20:32 smithi115 conmon[90393]: 2022-05-14T04:20:32.707+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:20:33.697 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:20:33 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:20:33.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:20:33 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:20:34.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:20:33 smithi115 conmon[90393]: 2022-05-14T04:20:33.693+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:20:34.718 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:20:34 smithi115 ceph-mon[78593]: pgmap v9134: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:20:34.718 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:20:34 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:20:34.718 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:20:34 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:20:34.718 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:20:34 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:20:34.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:20:34 smithi047 ceph-mon[77147]: pgmap v9134: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:20:34.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:20:34 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:20:34.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:20:34 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:20:34.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:20:34 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:20:35.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:20:34 smithi115 conmon[90393]: 2022-05-14T04:20:34.714+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:20:35.757 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:20:35 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:20:35.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:20:35 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:20:36.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:20:35 smithi115 conmon[90393]: 2022-05-14T04:20:35.754+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:20:36.747 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:20:36 smithi115 ceph-mon[78593]: pgmap v9135: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:20:36.748 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:20:36 smithi115 ceph-mon[78593]: 8 slow requests (by type [ 'queued for pg' : 8 ] most affected pool [ 'cephfs_metadata' : 8 ]) 2022-05-14T04:20:36.749 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:20:36 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:20:36.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:20:36 smithi047 ceph-mon[77147]: pgmap v9135: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:20:36.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:20:36 smithi047 ceph-mon[77147]: 8 slow requests (by type [ 'queued for pg' : 8 ] most affected pool [ 'cephfs_metadata' : 8 ]) 2022-05-14T04:20:36.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:20:36 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:20:37.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:20:36 smithi115 conmon[90393]: 2022-05-14T04:20:36.743+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:20:37.747 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:20:37 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:20:37.747 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:20:37 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 17570 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:20:37.797 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:20:37 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:20:37.797 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:20:37 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 17570 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:20:38.110 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:20:37 smithi115 conmon[90393]: 2022-05-14T04:20:37.744+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:20:38.760 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:20:38 smithi115 ceph-mon[78593]: pgmap v9136: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:20:38.760 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:20:38 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:20:38.760 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:20:38 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:20:38.760 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:20:38 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:20:38.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:20:38 smithi047 ceph-mon[77147]: pgmap v9136: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:20:38.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:20:38 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:20:38.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:20:38 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:20:38.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:20:38 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:20:39.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:20:38 smithi115 conmon[90393]: 2022-05-14T04:20:38.756+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:20:39.783 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:20:39 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:20:39.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:20:39 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:20:40.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:20:39 smithi115 conmon[90393]: 2022-05-14T04:20:39.779+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:20:40.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:20:40 smithi047 ceph-mon[77147]: pgmap v9137: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:20:40.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:20:40 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:20:40.807 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:20:40 smithi115 ceph-mon[78593]: pgmap v9137: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:20:40.807 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:20:40 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:20:41.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:20:40 smithi115 conmon[90393]: 2022-05-14T04:20:40.804+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:20:41.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:20:41 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:20:41.856 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:20:41 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:20:42.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:20:41 smithi115 conmon[90393]: 2022-05-14T04:20:41.853+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:20:42.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:20:42 smithi047 ceph-mon[77147]: pgmap v9138: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:20:42.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:20:42 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:20:42.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:20:42 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:20:42.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:20:42 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:20:42.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:20:42 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 17575 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:20:42.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:20:42 smithi115 ceph-mon[78593]: pgmap v9138: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:20:42.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:20:42 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:20:42.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:20:42 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:20:42.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:20:42 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:20:42.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:20:42 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 17575 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:20:43.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:20:42 smithi115 conmon[90393]: 2022-05-14T04:20:42.866+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:20:43.795 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:20:43 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:20:43.855 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:20:43 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:20:44.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:20:43 smithi115 conmon[90393]: 2022-05-14T04:20:43.852+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:20:44.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:20:44 smithi047 ceph-mon[77147]: pgmap v9139: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:20:44.796 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:20:44 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:20:44.824 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:20:44 smithi115 ceph-mon[78593]: pgmap v9139: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:20:44.824 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:20:44 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:20:45.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:20:44 smithi115 conmon[90393]: 2022-05-14T04:20:44.821+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:20:45.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:20:45 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:20:45.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:20:45 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:20:46.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:20:45 smithi115 conmon[90393]: 2022-05-14T04:20:45.794+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:20:46.762 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:20:46 smithi115 ceph-mon[78593]: pgmap v9140: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:20:46.762 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:20:46 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:20:46.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:20:46 smithi047 ceph-mon[77147]: pgmap v9140: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:20:46.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:20:46 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:20:47.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:20:46 smithi115 conmon[90393]: 2022-05-14T04:20:46.759+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:20:47.786 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:20:47 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:20:47.787 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:20:47 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 17580 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:20:47.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:20:47 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:20:47.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:20:47 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 17580 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:20:48.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:20:47 smithi115 conmon[90393]: 2022-05-14T04:20:47.784+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:20:48.768 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:20:48 smithi115 ceph-mon[78593]: pgmap v9141: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:20:48.768 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:20:48 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:20:48.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:20:48 smithi047 ceph-mon[77147]: pgmap v9141: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:20:48.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:20:48 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:20:49.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:20:48 smithi115 conmon[90393]: 2022-05-14T04:20:48.765+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:20:49.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:20:49 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:20:49.806 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:20:49 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:20:50.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:20:49 smithi115 conmon[90393]: 2022-05-14T04:20:49.803+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:20:50.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:20:50 smithi047 ceph-mon[77147]: pgmap v9142: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:20:50.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:20:50 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:20:50.795 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:20:50 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:20:50.795 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:20:50 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:20:50.839 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:20:50 smithi115 ceph-mon[78593]: pgmap v9142: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:20:50.839 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:20:50 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:20:50.840 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:20:50 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:20:50.840 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:20:50 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:20:51.114 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:20:50 smithi115 conmon[90393]: 2022-05-14T04:20:50.836+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:20:51.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:20:51 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:20:51.864 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:20:51 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:20:52.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:20:51 smithi115 conmon[90393]: 2022-05-14T04:20:51.881+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:20:52.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:20:52 smithi047 ceph-mon[77147]: pgmap v9143: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:20:52.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:20:52 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:20:52.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:20:52 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 17585 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:20:52.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:20:52 smithi115 ceph-mon[78593]: pgmap v9143: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:20:52.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:20:52 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:20:52.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:20:52 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 17585 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:20:53.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:20:52 smithi115 conmon[90393]: 2022-05-14T04:20:52.873+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:20:53.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:20:53 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:20:53.862 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:20:53 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:20:54.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:20:53 smithi115 conmon[90393]: 2022-05-14T04:20:53.871+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:20:54.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:20:54 smithi047 ceph-mon[77147]: pgmap v9144: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:20:54.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:20:54 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:20:54.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:20:54 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:20:54.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:20:54 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:20:54.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:20:54 smithi115 ceph-mon[78593]: pgmap v9144: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:20:54.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:20:54 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:20:54.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:20:54 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:20:54.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:20:54 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:20:55.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:20:54 smithi115 conmon[90393]: 2022-05-14T04:20:54.883+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:20:55.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:20:55 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:20:55.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:20:55 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:20:56.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:20:55 smithi115 conmon[90393]: 2022-05-14T04:20:55.892+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:20:56.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:20:56 smithi047 ceph-mon[77147]: pgmap v9145: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:20:56.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:20:56 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:20:56.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:20:56 smithi115 ceph-mon[78593]: pgmap v9145: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:20:56.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:20:56 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:20:57.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:20:56 smithi115 conmon[90393]: 2022-05-14T04:20:56.933+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:20:57.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:20:57 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:20:57.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:20:57 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 17590 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:20:57.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:20:57 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:20:57.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:20:57 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 17590 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:20:58.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:20:57 smithi115 conmon[90393]: 2022-05-14T04:20:57.973+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:20:58.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:20:58 smithi047 ceph-mon[77147]: pgmap v9146: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:20:58.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:20:58 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:20:58.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:20:58 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:20:58.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:20:58 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:20:58.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:20:58 smithi115 ceph-mon[78593]: pgmap v9146: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:20:58.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:20:58 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:20:58.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:20:58 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:20:58.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:20:58 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:20:59.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:20:58 smithi115 conmon[90393]: 2022-05-14T04:20:58.985+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:20:59.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:20:59 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:20:59.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:20:59 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:21:00.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:20:59 smithi115 conmon[90393]: 2022-05-14T04:20:59.975+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:21:00.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:21:00 smithi047 ceph-mon[77147]: pgmap v9147: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:21:00.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:21:00 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:21:00.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:21:00 smithi115 ceph-mon[78593]: pgmap v9147: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:21:00.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:21:00 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:21:01.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:21:01 smithi115 conmon[90393]: 2022-05-14T04:21:01.004+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:21:01.806 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:21:01 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:21:01.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:21:01 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:21:02.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:21:01 smithi115 conmon[90393]: 2022-05-14T04:21:01.996+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:21:02.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:21:02 smithi047 ceph-mon[77147]: pgmap v9148: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:21:02.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:21:02 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:21:02.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:21:02 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:21:02.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:21:02 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:21:02.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:21:02 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 17595 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:21:02.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:21:02 smithi115 ceph-mon[78593]: pgmap v9148: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:21:02.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:21:02 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:21:02.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:21:02 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:21:02.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:21:02 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:21:02.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:21:02 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 17595 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:21:03.360 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:21:02 smithi115 conmon[90393]: 2022-05-14T04:21:02.988+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:21:03.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:21:03 smithi047 ceph-mon[77147]: pgmap v9149: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:21:03.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:21:03 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:21:03.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:21:03 smithi115 ceph-mon[78593]: pgmap v9149: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:21:03.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:21:03 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:21:04.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:21:04 smithi115 conmon[90393]: 2022-05-14T04:21:04.010+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:21:04.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:21:04 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:21:04.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:21:04 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:21:05.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:21:05 smithi115 conmon[90393]: 2022-05-14T04:21:05.001+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:21:05.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:21:05 smithi047 ceph-mon[77147]: pgmap v9150: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:21:05.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:21:05 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:21:05.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:21:05 smithi115 ceph-mon[78593]: pgmap v9150: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:21:05.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:21:05 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:21:06.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:21:06 smithi115 conmon[90393]: 2022-05-14T04:21:06.037+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:21:06.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:21:06 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:21:06.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:21:06 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:21:07.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:21:07 smithi115 conmon[90393]: 2022-05-14T04:21:07.037+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:21:07.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:21:07 smithi047 ceph-mon[77147]: pgmap v9151: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:21:07.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:21:07 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:21:07.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:21:07 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 17600 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:21:07.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:21:07 smithi115 ceph-mon[78593]: pgmap v9151: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:21:07.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:21:07 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:21:07.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:21:07 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 17600 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:21:08.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:21:08 smithi115 conmon[90393]: 2022-05-14T04:21:08.012+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:21:08.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:21:08 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:21:08.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:21:08 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:21:09.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:21:09 smithi115 conmon[90393]: 2022-05-14T04:21:09.011+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:21:09.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:21:09 smithi047 ceph-mon[77147]: pgmap v9152: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:21:09.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:21:09 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:21:09.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:21:09 smithi115 ceph-mon[78593]: pgmap v9152: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:21:09.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:21:09 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:21:10.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:21:10 smithi115 conmon[90393]: 2022-05-14T04:21:10.001+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:21:10.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:21:10 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:21:10.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:21:10 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:21:10.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:21:10 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:21:10.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:21:10 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:21:10.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:21:10 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:21:10.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:21:10 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:21:11.360 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:21:10 smithi115 conmon[90393]: 2022-05-14T04:21:10.979+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:21:11.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:21:11 smithi047 ceph-mon[77147]: pgmap v9153: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:21:11.796 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:21:11 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:21:11.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:21:11 smithi115 ceph-mon[78593]: pgmap v9153: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:21:11.862 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:21:11 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:21:12.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:21:12 smithi115 conmon[90393]: 2022-05-14T04:21:12.022+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:21:12.798 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:21:12 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:21:12.798 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:21:12 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 17605 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:21:12.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:21:12 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:21:12.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:21:12 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 17605 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:21:13.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:21:13 smithi115 conmon[90393]: 2022-05-14T04:21:13.011+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:21:13.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:21:13 smithi047 ceph-mon[77147]: pgmap v9154: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:21:13.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:21:13 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:21:13.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:21:13 smithi115 ceph-mon[78593]: pgmap v9154: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:21:13.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:21:13 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:21:14.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:21:13 smithi115 conmon[90393]: 2022-05-14T04:21:13.971+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:21:14.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:21:14 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:21:14.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:21:14 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:21:14.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:21:14 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:21:14.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:21:14 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:21:14.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:21:14 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:21:14.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:21:14 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:21:15.361 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:21:14 smithi115 conmon[90393]: 2022-05-14T04:21:14.986+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:21:15.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:21:15 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T04:21:15.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:21:15 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T04:21:15.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:21:15 smithi047 ceph-mon[77147]: pgmap v9155: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:21:15.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:21:15 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:21:15.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:21:15 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T04:21:15.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:21:15 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T04:21:15.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:21:15 smithi115 ceph-mon[78593]: pgmap v9155: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:21:15.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:21:15 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:21:16.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:21:16 smithi115 conmon[90393]: 2022-05-14T04:21:16.013+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:21:16.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:21:16 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:21:16.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:21:16 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:21:17.360 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:21:16 smithi115 conmon[90393]: 2022-05-14T04:21:16.992+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:21:17.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:21:17 smithi047 ceph-mon[77147]: pgmap v9156: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:21:17.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:21:17 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:21:17.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:21:17 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 17610 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:21:17.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:21:17 smithi115 ceph-mon[78593]: pgmap v9156: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:21:17.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:21:17 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:21:17.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:21:17 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 17610 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:21:18.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:21:17 smithi115 conmon[90393]: 2022-05-14T04:21:17.971+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:21:18.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:21:18 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:21:18.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:21:18 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:21:18.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:21:18 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:21:18.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:21:18 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:21:18.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:21:18 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:21:18.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:21:18 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:21:19.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:21:18 smithi115 conmon[90393]: 2022-05-14T04:21:18.927+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:21:19.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:21:19 smithi047 ceph-mon[77147]: pgmap v9157: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:21:19.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:21:19 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:21:19.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:21:19 smithi115 ceph-mon[78593]: pgmap v9157: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:21:19.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:21:19 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:21:20.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:21:19 smithi115 conmon[90393]: 2022-05-14T04:21:19.920+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:21:20.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:21:20 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:21:20.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:21:20 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:21:21.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:21:20 smithi115 conmon[90393]: 2022-05-14T04:21:20.923+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:21:21.862 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:21:21 smithi115 ceph-mon[78593]: pgmap v9158: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:21:21.862 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:21:21 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:21:22.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:21:21 smithi047 ceph-mon[77147]: pgmap v9158: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:21:22.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:21:21 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:21:22.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:21:21 smithi115 conmon[90393]: 2022-05-14T04:21:21.883+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:21:22.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:21:22 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:21:22.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:21:22 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:21:22.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:21:22 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:21:22.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:21:22 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 17615 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:21:23.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:21:22 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:21:23.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:21:22 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:21:23.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:21:22 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:21:23.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:21:22 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 17615 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:21:23.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:21:22 smithi115 conmon[90393]: 2022-05-14T04:21:22.913+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:21:23.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:21:23 smithi115 ceph-mon[78593]: pgmap v9159: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:21:23.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:21:23 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:21:24.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:21:23 smithi047 ceph-mon[77147]: pgmap v9159: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:21:24.045 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:21:23 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:21:24.360 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:21:23 smithi115 conmon[90393]: 2022-05-14T04:21:23.929+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:21:24.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:21:24 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:21:25.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:21:24 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:21:25.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:21:24 smithi115 conmon[90393]: 2022-05-14T04:21:24.893+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:21:25.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:21:25 smithi115 ceph-mon[78593]: pgmap v9160: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:21:25.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:21:25 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:21:26.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:21:25 smithi047 ceph-mon[77147]: pgmap v9160: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:21:26.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:21:25 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:21:26.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:21:25 smithi115 conmon[90393]: 2022-05-14T04:21:25.911+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:21:26.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:21:26 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:21:27.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:21:26 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:21:27.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:21:26 smithi115 conmon[90393]: 2022-05-14T04:21:26.932+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:21:27.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:21:27 smithi115 ceph-mon[78593]: pgmap v9161: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:21:27.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:21:27 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:21:27.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:21:27 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 17620 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:21:28.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:21:27 smithi047 ceph-mon[77147]: pgmap v9161: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:21:28.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:21:27 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:21:28.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:21:27 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 17620 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:21:28.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:21:27 smithi115 conmon[90393]: 2022-05-14T04:21:27.931+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:21:28.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:21:28 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:21:29.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:21:28 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:21:29.362 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:21:28 smithi115 conmon[90393]: 2022-05-14T04:21:28.939+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:21:29.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:21:29 smithi115 ceph-mon[78593]: pgmap v9162: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:21:29.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:21:29 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:21:30.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:21:29 smithi047 ceph-mon[77147]: pgmap v9162: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:21:30.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:21:29 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:21:30.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:21:29 smithi115 conmon[90393]: 2022-05-14T04:21:29.941+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:21:30.839 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:21:30 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:21:30.840 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:21:30 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:21:30.840 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:21:30 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:21:30.840 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:21:30 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:21:30.841 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:21:30 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:21:30.841 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:21:30 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:21:31.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:21:30 smithi115 conmon[90393]: 2022-05-14T04:21:30.958+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:21:31.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:21:31 smithi115 ceph-mon[78593]: pgmap v9163: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:21:31.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:21:31 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T04:21:31.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:21:31 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T04:21:31.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:21:31 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T04:21:31.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:21:31 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:21:31.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:21:31 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' 2022-05-14T04:21:31.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:21:31 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' 2022-05-14T04:21:32.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:21:31 smithi047 ceph-mon[77147]: pgmap v9163: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:21:32.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:21:31 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T04:21:32.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:21:31 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T04:21:32.045 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:21:31 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T04:21:32.045 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:21:31 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:21:32.045 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:21:31 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' 2022-05-14T04:21:32.046 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:21:31 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' 2022-05-14T04:21:32.360 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:21:31 smithi115 conmon[90393]: 2022-05-14T04:21:31.969+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:21:32.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:21:32 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:21:32.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:21:32 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 17625 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:21:33.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:21:32 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:21:33.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:21:32 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 17625 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:21:33.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:21:33 smithi115 conmon[90393]: 2022-05-14T04:21:33.007+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:21:33.841 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:21:33 smithi115 ceph-mon[78593]: pgmap v9164: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:21:33.843 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:21:33 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:21:33.859 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:21:33 smithi047 ceph-mon[77147]: pgmap v9164: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:21:33.859 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:21:33 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:21:33.860 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:21:33 smithi047 conmon[78521]: 2022-05-14T04:21:33.808+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:21:33.860 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:21:33 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T04:21:33.860 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:21:33 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:21:33.861 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:21:33 smithi047 conmon[78521]: yield 2022-05-14T04:21:33.861 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:21:33 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:21:33.862 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:21:33 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:21:33.862 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:21:33 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:21:33.862 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:21:33 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T04:21:33.862 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:21:33 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:21:33.863 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:21:33 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T04:21:33.863 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:21:33 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:21:33.864 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:21:33 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T04:21:33.864 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:21:33 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:21:33.864 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:21:33 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T04:21:33.864 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:21:33 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:21:33.865 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:21:33 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T04:21:33.865 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:21:33 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:21:33.865 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:21:33 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:21:33.865 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:21:33 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:21:34.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:21:34 smithi115 conmon[90393]: 2022-05-14T04:21:34.033+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:21:34.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:21:34 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:21:34.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:21:34 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:21:34.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:21:34 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' 2022-05-14T04:21:34.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:21:34 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:21:34.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T04:21:34.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:21:34.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T04:21:34.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:21:34.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:21:34.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:21:34.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T04:21:34.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:21:34.863 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T04:21:34.863 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:21:34.863 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T04:21:34.863 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:21:34.863 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T04:21:34.864 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:21:34.864 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T04:21:34.864 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:21:34.864 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:21:34.865 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:21:34.865 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:21:34 smithi115 ceph-mon[78593]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T04:21:34.865 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:21:34 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' 2022-05-14T04:21:34.865 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:21:34 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:21:34.865 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:21:34 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-05-14T04:21:34.866 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:21:34 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-05-14T04:21:34.866 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:21:34 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-05-14T04:21:34.866 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:21:34 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-05-14T04:21:34.867 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:21:34 smithi115 ceph-mon[78593]: Adjusting osd_memory_target on smithi047 to 198.1M 2022-05-14T04:21:34.867 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:21:34 smithi115 ceph-mon[78593]: Unable to set osd_memory_target on smithi047 to 207755878: error parsing value: Value '207755878' is below minimum 939524096 2022-05-14T04:21:34.867 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:21:34 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-05-14T04:21:34.867 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:21:34 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-05-14T04:21:34.868 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:21:34 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-05-14T04:21:34.868 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:21:34 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-05-14T04:21:34.868 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:21:34 smithi115 ceph-mon[78593]: Adjusting osd_memory_target on smithi115 to 197.0M 2022-05-14T04:21:34.868 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:21:34 smithi115 ceph-mon[78593]: Unable to set osd_memory_target on smithi115 to 206629785: error parsing value: Value '206629785' is below minimum 939524096 2022-05-14T04:21:35.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:21:34 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:21:35.045 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:21:34 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:21:35.045 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:21:34 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' 2022-05-14T04:21:35.045 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:21:34 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:21:35.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T04:21:35.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:21:35.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T04:21:35.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:21:35.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:21:35.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:21:35.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T04:21:35.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:21:35.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T04:21:35.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:21:35.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T04:21:35.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:21:35.050 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T04:21:35.051 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:21:35.052 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T04:21:35.052 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:21:35.052 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:21:35.052 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:21:35.053 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:21:34 smithi047 ceph-mon[77147]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T04:21:35.053 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:21:34 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' 2022-05-14T04:21:35.053 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:21:34 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:21:35.053 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:21:34 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-05-14T04:21:35.053 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:21:34 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-05-14T04:21:35.054 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:21:34 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-05-14T04:21:35.054 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:21:34 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-05-14T04:21:35.055 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:21:34 smithi047 ceph-mon[77147]: Adjusting osd_memory_target on smithi047 to 198.1M 2022-05-14T04:21:35.055 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:21:34 smithi047 ceph-mon[77147]: Unable to set osd_memory_target on smithi047 to 207755878: error parsing value: Value '207755878' is below minimum 939524096 2022-05-14T04:21:35.055 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:21:34 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-05-14T04:21:35.055 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:21:34 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-05-14T04:21:35.056 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:21:34 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-05-14T04:21:35.056 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:21:34 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-05-14T04:21:35.056 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:21:34 smithi047 ceph-mon[77147]: Adjusting osd_memory_target on smithi115 to 197.0M 2022-05-14T04:21:35.056 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:21:34 smithi047 ceph-mon[77147]: Unable to set osd_memory_target on smithi115 to 206629785: error parsing value: Value '206629785' is below minimum 939524096 2022-05-14T04:21:35.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:21:35 smithi115 conmon[90393]: 2022-05-14T04:21:35.041+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:21:35.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:21:35 smithi115 ceph-mon[78593]: pgmap v9165: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:21:35.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:21:35 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:21:36.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:21:35 smithi047 ceph-mon[77147]: pgmap v9165: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:21:36.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:21:35 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:21:36.361 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:21:36 smithi115 conmon[90393]: 2022-05-14T04:21:36.086+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:21:36.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:21:36 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:21:36.883 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:21:36 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:21:37.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:21:36 smithi047 conmon[78521]: 2022-05-14T04:21:36.879+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:21:37.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:21:36 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T04:21:37.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:21:36 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:21:37.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:21:36 smithi047 conmon[78521]: yield 2022-05-14T04:21:37.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:21:36 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:21:37.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:21:36 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:21:37.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:21:36 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:21:37.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:21:36 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T04:21:37.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:21:36 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:21:37.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:21:36 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T04:21:37.297 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:21:36 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:21:37.297 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:21:36 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T04:21:37.297 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:21:36 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:21:37.297 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:21:36 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T04:21:37.298 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:21:36 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:21:37.298 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:21:36 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T04:21:37.298 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:21:36 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:21:37.298 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:21:36 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:21:37.298 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:21:36 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:21:37.299 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:21:36 smithi047 conmon[78521]: 2022-05-14T04:21:36.880+0000 7f5344f82700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T04:21:37.299 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:21:36 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T04:21:37.299 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:21:36 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:21:37.299 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:21:36 smithi047 conmon[78521]: yield 2022-05-14T04:21:37.300 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:21:36 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:21:37.300 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:21:36 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:21:37.300 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:21:36 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:21:37.300 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:21:36 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T04:21:37.300 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:21:36 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:21:37.301 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:21:36 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T04:21:37.301 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:21:36 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:21:37.301 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:21:36 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T04:21:37.301 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:21:36 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:21:37.302 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:21:36 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T04:21:37.302 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:21:36 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:21:37.302 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:21:36 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T04:21:37.302 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:21:36 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:21:37.302 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:21:36 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:21:37.303 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:21:36 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:21:37.303 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:21:36 smithi047 conmon[78521]: 2022-05-14T04:21:37.303 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:21:36 smithi047 conmon[78521]: During handling of the above exception, another exception occurred: 2022-05-14T04:21:37.303 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:21:36 smithi047 conmon[78521]: 2022-05-14T04:21:37.304 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:21:36 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T04:21:37.304 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:21:36 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T04:21:37.304 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:21:36 smithi047 conmon[78521]: return f(*arg) 2022-05-14T04:21:37.304 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:21:36 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T04:21:37.304 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:21:36 smithi047 conmon[78521]: self._write_client_files(client_files, host) 2022-05-14T04:21:37.305 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:21:36 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T04:21:37.305 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:21:36 smithi047 conmon[78521]: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T04:21:37.305 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:21:36 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T04:21:37.305 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:21:36 smithi047 conmon[78521]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T04:21:37.305 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:21:36 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T04:21:37.306 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:21:36 smithi047 conmon[78521]: return self.event_loop.get_result(coro) 2022-05-14T04:21:37.306 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:21:36 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T04:21:37.306 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:21:36 smithi047 conmon[78521]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T04:21:37.306 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:21:36 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T04:21:37.307 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:21:36 smithi047 conmon[78521]: return self.__get_result() 2022-05-14T04:21:37.307 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:21:36 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T04:21:37.307 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:21:36 smithi047 conmon[78521]: raise self._exception 2022-05-14T04:21:37.307 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:21:36 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T04:21:37.308 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:21:36 smithi047 conmon[78521]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T04:21:37.308 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:21:36 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T04:21:37.308 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:21:36 smithi047 conmon[78521]: conn = await self._remote_connection(host, addr) 2022-05-14T04:21:37.308 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:21:36 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T04:21:37.308 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:21:36 smithi047 conmon[78521]: raise 2022-05-14T04:21:37.309 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:21:36 smithi047 conmon[78521]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T04:21:37.309 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:21:36 smithi047 conmon[78521]: self.gen.throw(type, value, traceback) 2022-05-14T04:21:37.309 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:21:36 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T04:21:37.309 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:21:36 smithi047 conmon[78521]: raise OrchestratorError(msg) 2022-05-14T04:21:37.310 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:21:36 smithi047 conmon[78521]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:21:37.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:21:37 smithi115 conmon[90393]: 2022-05-14T04:21:37.078+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:21:37.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:21:37 smithi115 ceph-mon[78593]: pgmap v9166: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:21:37.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:21:37 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:21:37.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T04:21:37.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:21:37.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T04:21:37.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:21:37.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:21:37.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:21:37.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T04:21:37.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:21:37.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T04:21:37.863 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:21:37.863 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T04:21:37.863 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:21:37.863 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T04:21:37.864 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:21:37.864 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T04:21:37.864 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:21:37.864 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:21:37.865 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:21:37.865 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:21:37 smithi115 ceph-mon[78593]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T04:21:37.865 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T04:21:37.865 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:21:37.865 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T04:21:37.866 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:21:37.866 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:21:37.866 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:21:37.866 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T04:21:37.867 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:21:37.867 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T04:21:37.867 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:21:37.867 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T04:21:37.868 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:21:37.868 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T04:21:37.868 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:21:37.868 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T04:21:37.868 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:21:37.869 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:21:37.869 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:21:37.869 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T04:21:37.869 INFO:journalctl@ceph.mon.c.smithi115.stdout: During handling of the above exception, another exception occurred: 2022-05-14T04:21:37.870 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T04:21:37.870 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T04:21:37.870 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T04:21:37.871 INFO:journalctl@ceph.mon.c.smithi115.stdout: return f(*arg) 2022-05-14T04:21:37.871 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T04:21:37.871 INFO:journalctl@ceph.mon.c.smithi115.stdout: self._write_client_files(client_files, host) 2022-05-14T04:21:37.871 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T04:21:37.871 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T04:21:37.872 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T04:21:37.872 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T04:21:37.872 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T04:21:37.872 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.event_loop.get_result(coro) 2022-05-14T04:21:37.873 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T04:21:37.873 INFO:journalctl@ceph.mon.c.smithi115.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T04:21:37.873 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T04:21:37.873 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.__get_result() 2022-05-14T04:21:37.873 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T04:21:37.874 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise self._exception 2022-05-14T04:21:37.874 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T04:21:37.874 INFO:journalctl@ceph.mon.c.smithi115.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T04:21:37.874 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T04:21:37.874 INFO:journalctl@ceph.mon.c.smithi115.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T04:21:37.875 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T04:21:37.875 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise 2022-05-14T04:21:37.875 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T04:21:37.875 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.gen.throw(type, value, traceback) 2022-05-14T04:21:37.876 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T04:21:37.876 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OrchestratorError(msg) 2022-05-14T04:21:37.876 INFO:journalctl@ceph.mon.c.smithi115.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:21:37.876 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:21:37 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:21:37.876 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:21:37 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 17630 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:21:38.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:21:37 smithi047 ceph-mon[77147]: pgmap v9166: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:21:38.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:21:37 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:21:38.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T04:21:38.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:21:38.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T04:21:38.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:21:38.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:21:38.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:21:38.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T04:21:38.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:21:38.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T04:21:38.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:21:38.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T04:21:38.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:21:38.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T04:21:38.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:21:38.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T04:21:38.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:21:38.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:21:38.049 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:21:38.049 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:21:37 smithi047 ceph-mon[77147]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T04:21:38.049 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T04:21:38.049 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:21:38.049 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T04:21:38.050 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:21:38.050 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:21:38.050 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:21:38.050 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T04:21:38.051 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:21:38.051 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T04:21:38.051 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:21:38.051 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T04:21:38.052 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:21:38.052 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T04:21:38.052 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:21:38.053 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T04:21:38.053 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:21:38.053 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:21:38.053 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:21:38.053 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T04:21:38.054 INFO:journalctl@ceph.mon.b.smithi047.stdout: During handling of the above exception, another exception occurred: 2022-05-14T04:21:38.054 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T04:21:38.054 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T04:21:38.054 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T04:21:38.054 INFO:journalctl@ceph.mon.b.smithi047.stdout: return f(*arg) 2022-05-14T04:21:38.055 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T04:21:38.055 INFO:journalctl@ceph.mon.b.smithi047.stdout: self._write_client_files(client_files, host) 2022-05-14T04:21:38.055 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T04:21:38.055 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T04:21:38.056 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T04:21:38.056 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T04:21:38.056 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T04:21:38.056 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.event_loop.get_result(coro) 2022-05-14T04:21:38.057 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T04:21:38.057 INFO:journalctl@ceph.mon.b.smithi047.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T04:21:38.057 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T04:21:38.057 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.__get_result() 2022-05-14T04:21:38.058 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T04:21:38.058 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise self._exception 2022-05-14T04:21:38.058 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T04:21:38.058 INFO:journalctl@ceph.mon.b.smithi047.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T04:21:38.059 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T04:21:38.059 INFO:journalctl@ceph.mon.b.smithi047.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T04:21:38.059 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T04:21:38.059 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise 2022-05-14T04:21:38.059 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T04:21:38.060 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.gen.throw(type, value, traceback) 2022-05-14T04:21:38.060 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T04:21:38.060 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OrchestratorError(msg) 2022-05-14T04:21:38.060 INFO:journalctl@ceph.mon.b.smithi047.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:21:38.060 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:21:37 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:21:38.061 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:21:37 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 17630 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:21:38.599 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:21:38 smithi115 conmon[90393]: 2022-05-14T04:21:38.127+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:21:38.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:21:38 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:21:38.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:21:38 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:21:38.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:21:38 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:21:39.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:21:38 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:21:39.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:21:38 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:21:39.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:21:38 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:21:39.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:21:39 smithi115 conmon[90393]: 2022-05-14T04:21:39.137+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:21:39.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:21:39 smithi115 ceph-mon[78593]: pgmap v9167: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:21:39.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:21:39 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:21:40.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:21:39 smithi047 ceph-mon[77147]: pgmap v9167: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:21:40.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:21:39 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:21:40.606 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:21:40 smithi115 conmon[90393]: 2022-05-14T04:21:40.113+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:21:40.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:21:40 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:21:41.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:21:40 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:21:41.610 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:21:41 smithi115 conmon[90393]: 2022-05-14T04:21:41.118+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:21:42.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:21:41 smithi047 ceph-mon[77147]: pgmap v9168: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:21:42.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:21:41 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:21:42.098 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:21:41 smithi115 ceph-mon[78593]: pgmap v9168: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:21:42.098 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:21:41 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:21:42.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:21:42 smithi115 conmon[90393]: 2022-05-14T04:21:42.095+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:21:43.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:21:42 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:21:43.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:21:42 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:21:43.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:21:42 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:21:43.045 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:21:42 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 17635 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:21:43.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:21:42 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:21:43.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:21:42 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:21:43.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:21:42 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:21:43.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:21:42 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 17635 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:21:43.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:21:43 smithi115 conmon[90393]: 2022-05-14T04:21:43.120+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:21:44.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:21:43 smithi047 ceph-mon[77147]: pgmap v9169: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:21:44.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:21:43 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:21:44.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:21:43 smithi115 ceph-mon[78593]: pgmap v9169: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:21:44.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:21:43 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:21:44.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:21:44 smithi115 conmon[90393]: 2022-05-14T04:21:44.160+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:21:45.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:21:44 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:21:45.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:21:44 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:21:45.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:21:45 smithi115 conmon[90393]: 2022-05-14T04:21:45.189+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:21:46.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:21:45 smithi047 ceph-mon[77147]: pgmap v9170: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:21:46.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:21:45 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:21:46.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:21:45 smithi115 ceph-mon[78593]: pgmap v9170: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:21:46.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:21:45 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:21:46.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:21:46 smithi115 conmon[90393]: 2022-05-14T04:21:46.216+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:21:47.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:21:46 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:21:47.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:21:46 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:21:47.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:21:47 smithi115 conmon[90393]: 2022-05-14T04:21:47.224+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:21:48.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:21:47 smithi047 ceph-mon[77147]: pgmap v9171: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:21:48.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:21:47 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:21:48.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:21:47 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 17640 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:21:48.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:21:47 smithi115 ceph-mon[78593]: pgmap v9171: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:21:48.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:21:47 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:21:48.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:21:47 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 17640 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:21:48.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:21:48 smithi115 conmon[90393]: 2022-05-14T04:21:48.222+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:21:49.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:21:48 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:21:49.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:21:48 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:21:49.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:21:49 smithi115 conmon[90393]: 2022-05-14T04:21:49.184+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:21:50.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:21:49 smithi047 ceph-mon[77147]: pgmap v9172: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:21:50.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:21:49 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:21:50.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:21:49 smithi115 ceph-mon[78593]: pgmap v9172: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:21:50.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:21:49 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:21:50.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:21:50 smithi115 conmon[90393]: 2022-05-14T04:21:50.192+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:21:51.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:21:50 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:21:51.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:21:50 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:21:51.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:21:50 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:21:51.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:21:50 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:21:51.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:21:50 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:21:51.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:21:50 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:21:51.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:21:51 smithi115 conmon[90393]: 2022-05-14T04:21:51.170+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:21:52.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:21:51 smithi047 ceph-mon[77147]: pgmap v9173: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:21:52.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:21:51 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:21:52.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:21:51 smithi115 ceph-mon[78593]: pgmap v9173: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:21:52.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:21:51 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:21:52.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:21:52 smithi115 conmon[90393]: 2022-05-14T04:21:52.155+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:21:53.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:21:52 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:21:53.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:21:52 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 17645 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:21:53.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:21:52 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:21:53.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:21:52 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 17645 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:21:53.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:21:53 smithi115 conmon[90393]: 2022-05-14T04:21:53.126+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:21:54.348 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:21:54 smithi115 conmon[90393]: 2022-05-14T04:21:54.090+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:21:54.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:21:54 smithi115 ceph-mon[78593]: pgmap v9174: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:21:54.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:21:54 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:21:54.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:21:54 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:21:54.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:21:54 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:21:54.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:21:54 smithi047 ceph-mon[77147]: pgmap v9174: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:21:54.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:21:54 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:21:54.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:21:54 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:21:54.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:21:54 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:21:55.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:21:55 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:21:55.610 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:21:55 smithi115 conmon[90393]: 2022-05-14T04:21:55.113+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:21:55.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:21:55 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:21:56.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:21:56 smithi115 conmon[90393]: 2022-05-14T04:21:56.149+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:21:56.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:21:56 smithi115 ceph-mon[78593]: pgmap v9175: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:21:56.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:21:56 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:21:56.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:21:56 smithi047 ceph-mon[77147]: pgmap v9175: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:21:56.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:21:56 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:21:57.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:21:57 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:21:57.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:21:57 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 17650 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:21:57.611 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:21:57 smithi115 conmon[90393]: 2022-05-14T04:21:57.181+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:21:57.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:21:57 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:21:57.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:21:57 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 17650 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:21:58.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:21:58 smithi115 conmon[90393]: 2022-05-14T04:21:58.132+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:21:58.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:21:58 smithi115 ceph-mon[78593]: pgmap v9176: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:21:58.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:21:58 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:21:58.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:21:58 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:21:58.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:21:58 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:21:58.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:21:58 smithi047 ceph-mon[77147]: pgmap v9176: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:21:58.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:21:58 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:21:58.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:21:58 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:21:58.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:21:58 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:21:59.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:21:59 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:21:59.610 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:21:59 smithi115 conmon[90393]: 2022-05-14T04:21:59.123+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:21:59.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:21:59 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:22:00.610 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:22:00 smithi115 conmon[90393]: 2022-05-14T04:22:00.161+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:22:00.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:22:00 smithi115 ceph-mon[78593]: pgmap v9177: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:22:00.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:22:00 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:22:00.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:22:00 smithi047 ceph-mon[77147]: pgmap v9177: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:22:00.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:22:00 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:22:01.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:22:01 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:22:01.610 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:22:01 smithi115 conmon[90393]: 2022-05-14T04:22:01.158+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:22:01.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:22:01 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:22:02.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:22:02 smithi115 conmon[90393]: 2022-05-14T04:22:02.155+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:22:02.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:22:02 smithi115 ceph-mon[78593]: pgmap v9178: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:22:02.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:22:02 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:22:02.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:22:02 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:22:02.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:22:02 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:22:02.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:22:02 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 17655 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:22:02.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:22:02 smithi047 ceph-mon[77147]: pgmap v9178: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:22:02.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:22:02 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:22:02.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:22:02 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:22:02.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:22:02 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:22:02.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:22:02 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 17655 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:22:03.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:22:03 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:22:03.610 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:22:03 smithi115 conmon[90393]: 2022-05-14T04:22:03.135+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:22:03.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:22:03 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:22:04.616 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:22:04 smithi115 conmon[90393]: 2022-05-14T04:22:04.171+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:22:04.617 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:22:04 smithi115 ceph-mon[78593]: pgmap v9179: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:22:04.617 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:22:04 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:22:04.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:22:04 smithi047 ceph-mon[77147]: pgmap v9179: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:22:04.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:22:04 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:22:05.383 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:22:05 smithi115 conmon[90393]: 2022-05-14T04:22:05.124+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:22:05.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:22:05 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:22:05.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:22:05 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:22:06.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:22:06 smithi115 conmon[90393]: 2022-05-14T04:22:06.086+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:22:06.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:22:06 smithi047 ceph-mon[77147]: pgmap v9180: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:22:06.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:22:06 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:22:06.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:22:06 smithi115 ceph-mon[78593]: pgmap v9180: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:22:06.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:22:06 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:22:07.389 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:22:07 smithi115 conmon[90393]: 2022-05-14T04:22:07.109+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:22:07.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:22:07 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:22:07.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:22:07 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 17660 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:22:07.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:22:07 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:22:07.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:22:07 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 17660 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:22:08.614 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:22:08 smithi115 conmon[90393]: 2022-05-14T04:22:08.146+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:22:08.615 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:22:08 smithi115 ceph-mon[78593]: pgmap v9181: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:22:08.615 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:22:08 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:22:08.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:22:08 smithi047 ceph-mon[77147]: pgmap v9181: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:22:08.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:22:08 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:22:09.396 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:22:09 smithi115 conmon[90393]: 2022-05-14T04:22:09.133+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:22:09.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:22:09 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:22:09.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:22:09 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:22:10.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:22:10 smithi115 conmon[90393]: 2022-05-14T04:22:10.151+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:22:10.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:22:10 smithi115 ceph-mon[78593]: pgmap v9182: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:22:10.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:22:10 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:22:10.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:22:10 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:22:10.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:22:10 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:22:10.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:22:10 smithi047 ceph-mon[77147]: pgmap v9182: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:22:10.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:22:10 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:22:10.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:22:10 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:22:10.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:22:10 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:22:11.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:22:11 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:22:11.610 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:22:11 smithi115 conmon[90393]: 2022-05-14T04:22:11.181+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:22:11.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:22:11 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:22:12.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:22:12 smithi115 conmon[90393]: 2022-05-14T04:22:12.211+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:22:12.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:22:12 smithi115 ceph-mon[78593]: pgmap v9183: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:22:12.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:22:12 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:22:12.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:22:12 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 17665 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:22:12.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:22:12 smithi047 ceph-mon[77147]: pgmap v9183: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:22:12.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:22:12 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:22:12.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:22:12 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 17665 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:22:13.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:22:13 smithi115 conmon[90393]: 2022-05-14T04:22:13.191+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:22:13.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:22:13 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:22:13.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:22:13 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:22:13.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:22:13 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:22:13.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:22:13 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:22:14.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:22:14 smithi115 conmon[90393]: 2022-05-14T04:22:14.235+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:22:14.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:22:14 smithi115 ceph-mon[78593]: pgmap v9184: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:22:14.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:22:14 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:22:14.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:22:14 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:22:14.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:22:14 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:22:14.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:22:14 smithi047 ceph-mon[77147]: pgmap v9184: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:22:14.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:22:14 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:22:14.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:22:14 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:22:14.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:22:14 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:22:15.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:22:15 smithi115 conmon[90393]: 2022-05-14T04:22:15.239+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:22:15.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:22:15 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T04:22:15.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:22:15 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T04:22:15.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:22:15 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:22:15.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:22:15 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T04:22:15.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:22:15 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T04:22:15.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:22:15 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:22:16.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:22:16 smithi115 conmon[90393]: 2022-05-14T04:22:16.208+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:22:16.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:22:16 smithi115 ceph-mon[78593]: pgmap v9185: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:22:16.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:22:16 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:22:16.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:22:16 smithi047 ceph-mon[77147]: pgmap v9185: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:22:16.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:22:16 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:22:17.610 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:22:17 smithi115 conmon[90393]: 2022-05-14T04:22:17.220+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:22:17.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:22:17 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:22:17.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:22:17 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 17670 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:22:17.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:22:17 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:22:17.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:22:17 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 17670 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:22:18.610 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:22:18 smithi115 conmon[90393]: 2022-05-14T04:22:18.221+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:22:18.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:22:18 smithi115 ceph-mon[78593]: pgmap v9186: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:22:18.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:22:18 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:22:18.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:22:18 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:22:18.612 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:22:18 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:22:18.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:22:18 smithi047 ceph-mon[77147]: pgmap v9186: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:22:18.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:22:18 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:22:18.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:22:18 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:22:18.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:22:18 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:22:19.610 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:22:19 smithi115 conmon[90393]: 2022-05-14T04:22:19.211+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:22:19.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:22:19 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:22:19.795 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:22:19 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:22:20.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:22:20 smithi115 conmon[90393]: 2022-05-14T04:22:20.216+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:22:20.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:22:20 smithi115 ceph-mon[78593]: pgmap v9187: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:22:20.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:22:20 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:22:20.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:22:20 smithi047 ceph-mon[77147]: pgmap v9187: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:22:20.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:22:20 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:22:21.610 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:22:21 smithi115 conmon[90393]: 2022-05-14T04:22:21.190+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:22:21.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:22:21 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:22:21.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:22:21 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:22:22.444 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:22:22 smithi115 conmon[90393]: 2022-05-14T04:22:22.158+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:22:22.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:22:22 smithi047 ceph-mon[77147]: pgmap v9188: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:22:22.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:22:22 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:22:22.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:22:22 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:22:22.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:22:22 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:22:22.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:22:22 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 17675 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:22:22.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:22:22 smithi115 ceph-mon[78593]: pgmap v9188: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:22:22.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:22:22 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:22:22.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:22:22 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:22:22.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:22:22 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:22:22.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:22:22 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 17675 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:22:23.447 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:22:23 smithi115 conmon[90393]: 2022-05-14T04:22:23.156+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:22:23.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:22:23 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:22:23.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:22:23 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:22:24.449 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:22:24 smithi115 conmon[90393]: 2022-05-14T04:22:24.151+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:22:24.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:22:24 smithi047 ceph-mon[77147]: pgmap v9189: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:22:24.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:22:24 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:22:24.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:22:24 smithi115 ceph-mon[78593]: pgmap v9189: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:22:24.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:22:24 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:22:25.453 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:22:25 smithi115 conmon[90393]: 2022-05-14T04:22:25.132+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:22:25.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:22:25 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:22:25.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:22:25 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:22:26.455 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:22:26 smithi115 conmon[90393]: 2022-05-14T04:22:26.137+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:22:26.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:22:26 smithi047 ceph-mon[77147]: pgmap v9190: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:22:26.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:22:26 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:22:26.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:22:26 smithi115 ceph-mon[78593]: pgmap v9190: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:22:26.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:22:26 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:22:27.459 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:22:27 smithi115 conmon[90393]: 2022-05-14T04:22:27.167+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:22:27.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:22:27 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:22:27.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:22:27 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 17680 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:22:27.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:22:27 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:22:27.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:22:27 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 17680 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:22:28.461 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:22:28 smithi115 conmon[90393]: 2022-05-14T04:22:28.171+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:22:28.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:22:28 smithi047 ceph-mon[77147]: pgmap v9191: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:22:28.796 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:22:28 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:22:28.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:22:28 smithi115 ceph-mon[78593]: pgmap v9191: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:22:28.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:22:28 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:22:29.467 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:22:29 smithi115 conmon[90393]: 2022-05-14T04:22:29.185+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:22:29.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:22:29 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:22:29.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:22:29 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:22:30.467 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:22:30 smithi115 conmon[90393]: 2022-05-14T04:22:30.178+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:22:30.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:22:30 smithi047 ceph-mon[77147]: pgmap v9192: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:22:30.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:22:30 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:22:30.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:22:30 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:22:30.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:22:30 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:22:30.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:22:30 smithi115 ceph-mon[78593]: pgmap v9192: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:22:30.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:22:30 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:22:30.862 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:22:30 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:22:30.862 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:22:30 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:22:31.473 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:22:31 smithi115 conmon[90393]: 2022-05-14T04:22:31.138+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:22:31.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:22:31 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:22:31.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:22:31 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:22:32.474 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:22:32 smithi115 conmon[90393]: 2022-05-14T04:22:32.138+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:22:32.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:22:32 smithi047 ceph-mon[77147]: pgmap v9193: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:22:32.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:22:32 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:22:32.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:22:32 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 17685 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:22:32.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:22:32 smithi115 ceph-mon[78593]: pgmap v9193: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:22:32.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:22:32 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:22:32.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:22:32 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 17685 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:22:33.479 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:22:33 smithi115 conmon[90393]: 2022-05-14T04:22:33.150+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:22:33.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:22:33 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:22:33.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:22:33 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:22:34.480 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:22:34 smithi115 conmon[90393]: 2022-05-14T04:22:34.179+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:22:34.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:22:34 smithi047 ceph-mon[77147]: pgmap v9194: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:22:34.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:22:34 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:22:34.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:22:34 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:22:34.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:22:34 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:22:34.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:22:34 smithi115 ceph-mon[78593]: pgmap v9194: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:22:34.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:22:34 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:22:34.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:22:34 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:22:34.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:22:34 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:22:35.485 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:22:35 smithi115 conmon[90393]: 2022-05-14T04:22:35.196+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:22:35.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:22:35 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:22:35.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:22:35 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:22:36.486 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:22:36 smithi115 conmon[90393]: 2022-05-14T04:22:36.182+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:22:36.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:22:36 smithi047 ceph-mon[77147]: pgmap v9195: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:22:36.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:22:36 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:22:36.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:22:36 smithi115 ceph-mon[78593]: pgmap v9195: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:22:36.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:22:36 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:22:37.484 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:22:37 smithi115 conmon[90393]: 2022-05-14T04:22:37.228+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:22:37.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:22:37 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T04:22:37.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:22:37 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T04:22:37.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:22:37 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T04:22:37.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:22:37 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:22:37.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:22:37 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 17690 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:22:37.795 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:22:37 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T04:22:37.797 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:22:37 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T04:22:37.797 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:22:37 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T04:22:37.798 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:22:37 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:22:37.798 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:22:37 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 17690 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:22:37.799 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:22:37 smithi047 conmon[78521]: 2022-05-14T04:22:37.488+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:22:37.799 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:22:37 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T04:22:37.800 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:22:37 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:22:37.800 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:22:37 smithi047 conmon[78521]: yield 2022-05-14T04:22:37.800 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:22:37 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:22:37.801 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:22:37 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:22:37.802 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:22:37 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:22:37.802 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:22:37 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T04:22:37.803 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:22:37 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:22:37.803 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:22:37 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T04:22:37.803 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:22:37 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:22:37.808 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:22:37 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T04:22:37.809 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:22:37 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:22:37.809 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:22:37 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T04:22:37.809 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:22:37 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:22:37.809 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:22:37 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T04:22:37.810 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:22:37 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:22:37.810 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:22:37 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:22:37.810 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:22:37 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:22:38.494 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:22:38 smithi115 conmon[90393]: 2022-05-14T04:22:38.185+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:22:38.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:22:38 smithi047 ceph-mon[77147]: pgmap v9196: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:22:38.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:22:38 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:22:38.795 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:22:38 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:22:38.795 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:22:38 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' 2022-05-14T04:22:38.796 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:22:38 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' 2022-05-14T04:22:38.796 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:22:38 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:22:38.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:22:38 smithi115 ceph-mon[78593]: pgmap v9196: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:22:38.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:22:38 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:22:38.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:22:38 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:22:38.862 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:22:38 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' 2022-05-14T04:22:38.862 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:22:38 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' 2022-05-14T04:22:38.863 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:22:38 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:22:39.498 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:22:39 smithi115 conmon[90393]: 2022-05-14T04:22:39.182+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:22:39.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:22:39 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:22:39.793 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T04:22:39.793 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:22:39.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T04:22:39.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:22:39.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:22:39.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:22:39.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T04:22:39.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:22:39.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T04:22:39.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:22:39.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T04:22:39.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:22:39.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T04:22:39.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:22:39.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T04:22:39.797 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:22:39.797 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:22:39.797 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:22:39.798 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:22:39 smithi047 ceph-mon[77147]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T04:22:39.798 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:22:39 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:22:39.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:22:39 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:22:39.860 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T04:22:39.860 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:22:39.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T04:22:39.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:22:39.863 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:22:39.864 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:22:39.865 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T04:22:39.865 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:22:39.865 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T04:22:39.866 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:22:39.866 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T04:22:39.867 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:22:39.867 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T04:22:39.867 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:22:39.868 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T04:22:39.868 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:22:39.869 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:22:39.869 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:22:39.869 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:22:39 smithi115 ceph-mon[78593]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T04:22:39.870 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:22:39 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:22:40.498 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:22:40 smithi115 conmon[90393]: 2022-05-14T04:22:40.200+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:22:40.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:22:40 smithi047 ceph-mon[77147]: pgmap v9197: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:22:40.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:22:40 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:22:40.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:22:40 smithi047 conmon[78521]: 2022-05-14T04:22:40.559+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:22:40.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:22:40 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T04:22:40.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:22:40 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:22:40.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:22:40 smithi047 conmon[78521]: yield 2022-05-14T04:22:40.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:22:40 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:22:40.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:22:40 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:22:40.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:22:40 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:22:40.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:22:40 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T04:22:40.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:22:40 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:22:40.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:22:40 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T04:22:40.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:22:40 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:22:40.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:22:40 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T04:22:40.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:22:40 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:22:40.797 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:22:40 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T04:22:40.797 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:22:40 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:22:40.797 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:22:40 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T04:22:40.797 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:22:40 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:22:40.797 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:22:40 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:22:40.798 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:22:40 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:22:40.798 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:22:40 smithi047 conmon[78521]: 2022-05-14T04:22:40.560+0000 7f533ef76700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T04:22:40.798 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:22:40 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T04:22:40.798 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:22:40 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:22:40.799 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:22:40 smithi047 conmon[78521]: yield 2022-05-14T04:22:40.799 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:22:40 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:22:40.799 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:22:40 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:22:40.799 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:22:40 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:22:40.799 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:22:40 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T04:22:40.800 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:22:40 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:22:40.800 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:22:40 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T04:22:40.800 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:22:40 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:22:40.800 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:22:40 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T04:22:40.800 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:22:40 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:22:40.801 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:22:40 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T04:22:40.801 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:22:40 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:22:40.801 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:22:40 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T04:22:40.802 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:22:40 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:22:40.804 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:22:40 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:22:40.804 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:22:40 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:22:40.804 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:22:40 smithi047 conmon[78521]: 2022-05-14T04:22:40.804 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:22:40 smithi047 conmon[78521]: During handling of the above exception, another exception occurred: 2022-05-14T04:22:40.805 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:22:40 smithi047 conmon[78521]: 2022-05-14T04:22:40.805 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:22:40 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T04:22:40.805 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:22:40 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T04:22:40.805 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:22:40 smithi047 conmon[78521]: return f(*arg) 2022-05-14T04:22:40.806 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:22:40 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T04:22:40.806 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:22:40 smithi047 conmon[78521]: self._write_client_files(client_files, host) 2022-05-14T04:22:40.806 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:22:40 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T04:22:40.807 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:22:40 smithi047 conmon[78521]: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T04:22:40.807 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:22:40 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T04:22:40.807 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:22:40 smithi047 conmon[78521]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T04:22:40.807 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:22:40 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T04:22:40.807 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:22:40 smithi047 conmon[78521]: return self.event_loop.get_result(coro) 2022-05-14T04:22:40.808 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:22:40 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T04:22:40.808 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:22:40 smithi047 conmon[78521]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T04:22:40.808 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:22:40 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T04:22:40.808 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:22:40 smithi047 conmon[78521]: return self.__get_result() 2022-05-14T04:22:40.808 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:22:40 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T04:22:40.809 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:22:40 smithi047 conmon[78521]: raise self._exception 2022-05-14T04:22:40.809 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:22:40 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T04:22:40.809 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:22:40 smithi047 conmon[78521]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T04:22:40.809 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:22:40 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T04:22:40.810 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:22:40 smithi047 conmon[78521]: conn = await self._remote_connection(host, addr) 2022-05-14T04:22:40.810 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:22:40 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T04:22:40.810 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:22:40 smithi047 conmon[78521]: raise 2022-05-14T04:22:40.810 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:22:40 smithi047 conmon[78521]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T04:22:40.811 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:22:40 smithi047 conmon[78521]: self.gen.throw(type, value, traceback) 2022-05-14T04:22:40.811 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:22:40 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T04:22:40.812 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:22:40 smithi047 conmon[78521]: raise OrchestratorError(msg) 2022-05-14T04:22:40.812 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:22:40 smithi047 conmon[78521]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:22:40.880 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:22:40 smithi115 ceph-mon[78593]: pgmap v9197: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:22:40.881 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:22:40 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:22:41.506 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:22:41 smithi115 conmon[90393]: 2022-05-14T04:22:41.222+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:22:41.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:22:41 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:22:41.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T04:22:41.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:22:41.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T04:22:41.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:22:41.797 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:22:41.797 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:22:41.797 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T04:22:41.798 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:22:41.798 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T04:22:41.799 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:22:41.800 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T04:22:41.800 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:22:41.800 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T04:22:41.800 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:22:41.801 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T04:22:41.801 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:22:41.802 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:22:41.802 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:22:41.802 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:22:41 smithi047 ceph-mon[77147]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T04:22:41.802 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T04:22:41.803 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:22:41.803 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T04:22:41.803 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:22:41.803 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:22:41.804 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:22:41.804 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T04:22:41.805 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:22:41.805 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T04:22:41.805 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:22:41.806 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T04:22:41.806 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:22:41.806 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T04:22:41.806 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:22:41.807 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T04:22:41.807 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:22:41.807 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:22:41.807 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:22:41.807 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T04:22:41.808 INFO:journalctl@ceph.mon.b.smithi047.stdout: During handling of the above exception, another exception occurred: 2022-05-14T04:22:41.808 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T04:22:41.808 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T04:22:41.809 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T04:22:41.810 INFO:journalctl@ceph.mon.b.smithi047.stdout: return f(*arg) 2022-05-14T04:22:41.810 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T04:22:41.811 INFO:journalctl@ceph.mon.b.smithi047.stdout: self._write_client_files(client_files, host) 2022-05-14T04:22:41.811 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T04:22:41.812 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T04:22:41.812 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T04:22:41.816 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T04:22:41.816 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T04:22:41.816 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.event_loop.get_result(coro) 2022-05-14T04:22:41.817 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T04:22:41.817 INFO:journalctl@ceph.mon.b.smithi047.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T04:22:41.817 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T04:22:41.817 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.__get_result() 2022-05-14T04:22:41.818 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T04:22:41.818 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise self._exception 2022-05-14T04:22:41.818 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T04:22:41.818 INFO:journalctl@ceph.mon.b.smithi047.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T04:22:41.819 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T04:22:41.819 INFO:journalctl@ceph.mon.b.smithi047.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T04:22:41.819 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T04:22:41.819 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise 2022-05-14T04:22:41.819 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T04:22:41.820 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.gen.throw(type, value, traceback) 2022-05-14T04:22:41.820 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T04:22:41.820 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OrchestratorError(msg) 2022-05-14T04:22:41.820 INFO:journalctl@ceph.mon.b.smithi047.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:22:41.821 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:22:41 smithi047 ceph-mon[77147]: pgmap v9198: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:22:41.821 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:22:41 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:22:41.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:22:41 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:22:41.860 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T04:22:41.860 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:22:41.860 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T04:22:41.860 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:22:41.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:22:41.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:22:41.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T04:22:41.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:22:41.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T04:22:41.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:22:41.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T04:22:41.863 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:22:41.863 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T04:22:41.864 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:22:41.864 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T04:22:41.865 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:22:41.865 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:22:41.865 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:22:41.866 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:22:41 smithi115 ceph-mon[78593]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T04:22:41.866 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T04:22:41.866 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:22:41.867 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T04:22:41.867 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:22:41.867 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:22:41.868 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:22:41.868 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T04:22:41.869 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:22:41.870 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T04:22:41.870 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:22:41.870 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T04:22:41.871 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:22:41.871 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T04:22:41.871 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:22:41.871 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T04:22:41.872 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:22:41.872 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:22:41.872 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:22:41.872 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T04:22:41.872 INFO:journalctl@ceph.mon.c.smithi115.stdout: During handling of the above exception, another exception occurred: 2022-05-14T04:22:41.873 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T04:22:41.873 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T04:22:41.874 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T04:22:41.875 INFO:journalctl@ceph.mon.c.smithi115.stdout: return f(*arg) 2022-05-14T04:22:41.875 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T04:22:41.875 INFO:journalctl@ceph.mon.c.smithi115.stdout: self._write_client_files(client_files, host) 2022-05-14T04:22:41.875 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T04:22:41.876 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T04:22:41.876 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T04:22:41.876 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T04:22:41.876 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T04:22:41.877 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.event_loop.get_result(coro) 2022-05-14T04:22:41.877 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T04:22:41.877 INFO:journalctl@ceph.mon.c.smithi115.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T04:22:41.877 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T04:22:41.878 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.__get_result() 2022-05-14T04:22:41.878 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T04:22:41.879 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise self._exception 2022-05-14T04:22:41.879 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T04:22:41.879 INFO:journalctl@ceph.mon.c.smithi115.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T04:22:41.880 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T04:22:41.880 INFO:journalctl@ceph.mon.c.smithi115.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T04:22:41.881 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T04:22:41.881 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise 2022-05-14T04:22:41.882 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T04:22:41.882 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.gen.throw(type, value, traceback) 2022-05-14T04:22:41.883 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T04:22:41.883 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OrchestratorError(msg) 2022-05-14T04:22:41.884 INFO:journalctl@ceph.mon.c.smithi115.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:22:41.884 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:22:41 smithi115 ceph-mon[78593]: pgmap v9198: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:22:41.884 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:22:41 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:22:42.506 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:22:42 smithi115 conmon[90393]: 2022-05-14T04:22:42.227+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:22:42.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:22:42 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:22:42.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:22:42 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:22:42.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:22:42 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:22:42.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:22:42 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 17695 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:22:42.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:22:42 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:22:42.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:22:42 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:22:42.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:22:42 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:22:42.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:22:42 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 17695 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:22:43.511 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:22:43 smithi115 conmon[90393]: 2022-05-14T04:22:43.242+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:22:43.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:22:43 smithi047 ceph-mon[77147]: pgmap v9199: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:22:43.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:22:43 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:22:43.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:22:43 smithi115 ceph-mon[78593]: pgmap v9199: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:22:43.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:22:43 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:22:44.514 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:22:44 smithi115 conmon[90393]: 2022-05-14T04:22:44.194+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:22:44.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:22:44 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:22:44.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:22:44 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:22:45.518 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:22:45 smithi115 conmon[90393]: 2022-05-14T04:22:45.176+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:22:45.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:22:45 smithi047 ceph-mon[77147]: pgmap v9200: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:22:45.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:22:45 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:22:45.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:22:45 smithi115 ceph-mon[78593]: pgmap v9200: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:22:45.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:22:45 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:22:46.519 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:22:46 smithi115 conmon[90393]: 2022-05-14T04:22:46.167+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:22:46.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:22:46 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:22:46.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:22:46 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:22:47.524 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:22:47 smithi115 conmon[90393]: 2022-05-14T04:22:47.203+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:22:47.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:22:47 smithi047 ceph-mon[77147]: pgmap v9201: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:22:47.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:22:47 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:22:47.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:22:47 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 17700 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:22:47.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:22:47 smithi115 ceph-mon[78593]: pgmap v9201: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:22:47.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:22:47 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:22:47.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:22:47 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 17700 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:22:48.526 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:22:48 smithi115 conmon[90393]: 2022-05-14T04:22:48.162+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:22:48.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:22:48 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:22:48.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:22:48 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:22:49.531 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:22:49 smithi115 conmon[90393]: 2022-05-14T04:22:49.127+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:22:49.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:22:49 smithi047 ceph-mon[77147]: pgmap v9202: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:22:49.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:22:49 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:22:49.862 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:22:49 smithi115 ceph-mon[78593]: pgmap v9202: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:22:49.862 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:22:49 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:22:50.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:22:50 smithi115 conmon[90393]: 2022-05-14T04:22:50.095+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:22:50.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:22:50 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:22:50.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:22:50 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:22:50.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:22:50 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:22:50.863 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:22:50 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:22:50.863 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:22:50 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:22:50.864 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:22:50 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:22:51.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:22:51 smithi115 conmon[90393]: 2022-05-14T04:22:51.100+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:22:51.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:22:51 smithi047 ceph-mon[77147]: pgmap v9203: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:22:51.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:22:51 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:22:51.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:22:51 smithi115 ceph-mon[78593]: pgmap v9203: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:22:51.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:22:51 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:22:52.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:22:52 smithi115 conmon[90393]: 2022-05-14T04:22:52.077+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:22:52.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:22:52 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:22:52.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:22:52 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 17705 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:22:52.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:22:52 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:22:52.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:22:52 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 17705 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:22:53.362 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:22:53 smithi115 conmon[90393]: 2022-05-14T04:22:53.075+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:22:53.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:22:53 smithi115 ceph-mon[78593]: pgmap v9204: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:22:53.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:22:53 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:22:54.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:22:53 smithi047 ceph-mon[77147]: pgmap v9204: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:22:54.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:22:53 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:22:54.547 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:22:54 smithi115 conmon[90393]: 2022-05-14T04:22:54.124+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:22:54.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:22:54 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:22:54.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:22:54 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:22:54.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:22:54 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:22:55.050 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:22:54 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:22:55.050 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:22:54 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:22:55.051 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:22:54 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:22:55.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:22:55 smithi115 conmon[90393]: 2022-05-14T04:22:55.094+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:22:55.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:22:55 smithi115 ceph-mon[78593]: pgmap v9205: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:22:55.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:22:55 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:22:56.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:22:55 smithi047 ceph-mon[77147]: pgmap v9205: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:22:56.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:22:55 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:22:56.552 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:22:56 smithi115 conmon[90393]: 2022-05-14T04:22:56.124+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:22:56.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:22:56 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:22:57.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:22:56 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:22:57.558 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:22:57 smithi115 conmon[90393]: 2022-05-14T04:22:57.144+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:22:57.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:22:57 smithi115 ceph-mon[78593]: pgmap v9206: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:22:57.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:22:57 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:22:57.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:22:57 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 17709 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:22:58.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:22:57 smithi047 ceph-mon[77147]: pgmap v9206: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:22:58.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:22:57 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:22:58.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:22:57 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 17709 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:22:58.558 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:22:58 smithi115 conmon[90393]: 2022-05-14T04:22:58.134+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:22:58.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:22:58 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:22:58.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:22:58 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:22:58.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:22:58 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:22:59.045 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:22:58 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:22:59.045 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:22:58 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:22:59.045 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:22:58 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:22:59.361 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:22:59 smithi115 conmon[90393]: 2022-05-14T04:22:59.097+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:22:59.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:22:59 smithi115 ceph-mon[78593]: pgmap v9207: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:22:59.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:22:59 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:23:00.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:22:59 smithi047 ceph-mon[77147]: pgmap v9207: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:23:00.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:22:59 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:23:00.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:23:00 smithi115 conmon[90393]: 2022-05-14T04:23:00.054+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:23:00.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:23:00 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:23:01.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:23:00 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:23:01.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:23:01 smithi115 conmon[90393]: 2022-05-14T04:23:01.072+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:23:01.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:23:01 smithi115 ceph-mon[78593]: pgmap v9208: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:23:01.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:23:01 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:23:02.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:23:01 smithi047 ceph-mon[77147]: pgmap v9208: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:23:02.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:23:01 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:23:02.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:23:02 smithi115 conmon[90393]: 2022-05-14T04:23:02.075+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:23:02.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:23:02 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:23:02.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:23:02 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:23:02.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:23:02 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:23:02.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:23:02 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 17715 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:23:03.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:23:02 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:23:03.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:23:02 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:23:03.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:23:02 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:23:03.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:23:02 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 17715 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:23:03.577 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:23:03 smithi115 conmon[90393]: 2022-05-14T04:23:03.111+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:23:03.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:23:03 smithi115 ceph-mon[78593]: pgmap v9209: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:23:03.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:23:03 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:23:04.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:23:03 smithi047 ceph-mon[77147]: pgmap v9209: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:23:04.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:23:03 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:23:04.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:23:04 smithi115 conmon[90393]: 2022-05-14T04:23:04.065+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:23:04.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:23:04 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:23:05.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:23:04 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:23:05.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:23:05 smithi115 conmon[90393]: 2022-05-14T04:23:05.052+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:23:05.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:23:05 smithi115 ceph-mon[78593]: pgmap v9210: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:23:05.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:23:05 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:23:06.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:23:05 smithi047 ceph-mon[77147]: pgmap v9210: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:23:06.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:23:05 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:23:06.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:23:06 smithi115 conmon[90393]: 2022-05-14T04:23:06.090+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:23:06.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:23:06 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:23:07.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:23:06 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:23:07.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:23:07 smithi115 conmon[90393]: 2022-05-14T04:23:07.105+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:23:07.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:23:07 smithi115 ceph-mon[78593]: pgmap v9211: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:23:07.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:23:07 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:23:07.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:23:07 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 17720 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:23:08.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:23:07 smithi047 ceph-mon[77147]: pgmap v9211: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:23:08.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:23:07 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:23:08.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:23:07 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 17720 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:23:08.591 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:23:08 smithi115 conmon[90393]: 2022-05-14T04:23:08.132+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:23:08.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:23:08 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:23:09.045 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:23:08 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:23:09.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:23:09 smithi115 conmon[90393]: 2022-05-14T04:23:09.094+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:23:09.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:23:09 smithi115 ceph-mon[78593]: pgmap v9212: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:23:09.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:23:09 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:23:10.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:23:09 smithi047 ceph-mon[77147]: pgmap v9212: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:23:10.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:23:09 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:23:10.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:23:10 smithi115 conmon[90393]: 2022-05-14T04:23:10.078+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:23:10.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:23:10 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:23:10.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:23:10 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:23:10.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:23:10 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:23:11.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:23:10 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:23:11.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:23:10 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:23:11.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:23:10 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:23:11.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:23:11 smithi115 conmon[90393]: 2022-05-14T04:23:11.074+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:23:11.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:23:11 smithi115 ceph-mon[78593]: pgmap v9213: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:23:11.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:23:11 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:23:12.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:23:11 smithi047 ceph-mon[77147]: pgmap v9213: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:23:12.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:23:11 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:23:12.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:23:12 smithi115 conmon[90393]: 2022-05-14T04:23:12.038+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:23:12.862 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:23:12 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:23:12.863 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:23:12 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 17725 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:23:13.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:23:12 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:23:13.045 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:23:12 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 17725 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:23:13.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:23:13 smithi115 conmon[90393]: 2022-05-14T04:23:13.026+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:23:14.037 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:23:13 smithi115 ceph-mon[78593]: pgmap v9214: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:23:14.038 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:23:13 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:23:14.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:23:13 smithi047 ceph-mon[77147]: pgmap v9214: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:23:14.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:23:13 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:23:14.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:23:14 smithi115 conmon[90393]: 2022-05-14T04:23:14.034+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:23:14.995 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:23:14 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:23:14.996 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:23:14 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:23:14.996 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:23:14 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:23:15.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:23:14 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:23:15.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:23:14 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:23:15.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:23:14 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:23:15.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:23:14 smithi115 conmon[90393]: 2022-05-14T04:23:14.993+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:23:15.977 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:23:15 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T04:23:15.977 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:23:15 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T04:23:15.978 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:23:15 smithi115 ceph-mon[78593]: pgmap v9215: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:23:15.978 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:23:15 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:23:16.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:23:15 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T04:23:16.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:23:15 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T04:23:16.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:23:15 smithi047 ceph-mon[77147]: pgmap v9215: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:23:16.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:23:15 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:23:16.373 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:23:15 smithi115 conmon[90393]: 2022-05-14T04:23:15.974+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:23:16.984 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:23:16 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:23:17.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:23:16 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:23:17.361 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:23:16 smithi115 conmon[90393]: 2022-05-14T04:23:16.981+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:23:17.960 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:23:17 smithi115 ceph-mon[78593]: pgmap v9216: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:23:17.960 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:23:17 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:23:17.960 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:23:17 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 17730 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:23:18.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:23:17 smithi047 ceph-mon[77147]: pgmap v9216: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:23:18.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:23:17 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:23:18.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:23:17 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 17730 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:23:18.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:23:17 smithi115 conmon[90393]: 2022-05-14T04:23:17.951+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:23:18.918 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:23:18 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:23:18.918 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:23:18 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:23:18.919 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:23:18 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:23:19.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:23:18 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:23:19.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:23:18 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:23:19.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:23:18 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:23:19.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:23:18 smithi115 conmon[90393]: 2022-05-14T04:23:18.915+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:23:19.960 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:23:19 smithi115 ceph-mon[78593]: pgmap v9217: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:23:19.960 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:23:19 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:23:20.046 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:23:19 smithi047 ceph-mon[77147]: pgmap v9217: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:23:20.046 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:23:19 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:23:20.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:23:19 smithi115 conmon[90393]: 2022-05-14T04:23:19.957+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:23:20.954 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:23:20 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:23:21.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:23:20 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:23:21.360 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:23:20 smithi115 conmon[90393]: 2022-05-14T04:23:20.952+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:23:21.983 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:23:21 smithi115 ceph-mon[78593]: pgmap v9218: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:23:21.983 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:23:21 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:23:22.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:23:21 smithi047 ceph-mon[77147]: pgmap v9218: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:23:22.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:23:21 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:23:22.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:23:21 smithi115 conmon[90393]: 2022-05-14T04:23:21.980+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:23:22.998 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:23:22 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:23:22.999 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:23:22 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:23:22.999 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:23:22 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:23:22.999 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:23:22 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 17735 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:23:23.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:23:22 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:23:23.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:23:22 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:23:23.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:23:22 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:23:23.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:23:22 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 17735 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:23:23.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:23:22 smithi115 conmon[90393]: 2022-05-14T04:23:22.996+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:23:24.018 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:23:23 smithi115 ceph-mon[78593]: pgmap v9219: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:23:24.018 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:23:23 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:23:24.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:23:23 smithi047 ceph-mon[77147]: pgmap v9219: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:23:24.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:23:23 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:23:24.360 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:23:24 smithi115 conmon[90393]: 2022-05-14T04:23:24.015+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:23:25.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:23:24 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:23:25.055 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:23:24 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:23:25.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:23:25 smithi115 conmon[90393]: 2022-05-14T04:23:25.053+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:23:26.009 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:23:25 smithi115 ceph-mon[78593]: pgmap v9220: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:23:26.009 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:23:25 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:23:26.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:23:25 smithi047 ceph-mon[77147]: pgmap v9220: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:23:26.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:23:25 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:23:26.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:23:26 smithi115 conmon[90393]: 2022-05-14T04:23:26.005+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:23:26.977 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:23:26 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:23:27.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:23:26 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:23:27.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:23:26 smithi115 conmon[90393]: 2022-05-14T04:23:26.974+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:23:28.001 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:23:27 smithi115 ceph-mon[78593]: pgmap v9221: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:23:28.002 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:23:27 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:23:28.002 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:23:27 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 17740 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:23:28.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:23:27 smithi047 ceph-mon[77147]: pgmap v9221: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:23:28.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:23:27 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:23:28.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:23:27 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 17740 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:23:28.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:23:27 smithi115 conmon[90393]: 2022-05-14T04:23:27.998+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:23:29.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:23:28 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:23:29.052 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:23:28 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:23:29.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:23:29 smithi115 conmon[90393]: 2022-05-14T04:23:29.045+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:23:30.036 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:23:29 smithi115 ceph-mon[78593]: pgmap v9222: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:23:30.036 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:23:29 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:23:30.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:23:29 smithi047 ceph-mon[77147]: pgmap v9222: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:23:30.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:23:29 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:23:30.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:23:30 smithi115 conmon[90393]: 2022-05-14T04:23:30.033+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:23:31.028 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:23:30 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:23:31.028 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:23:30 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:23:31.029 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:23:30 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:23:31.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:23:30 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:23:31.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:23:30 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:23:31.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:23:30 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:23:31.364 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:23:31 smithi115 conmon[90393]: 2022-05-14T04:23:31.025+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:23:32.021 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:23:31 smithi115 ceph-mon[78593]: pgmap v9223: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:23:32.022 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:23:31 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:23:32.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:23:31 smithi047 ceph-mon[77147]: pgmap v9223: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:23:32.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:23:31 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:23:32.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:23:32 smithi115 conmon[90393]: 2022-05-14T04:23:32.018+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:23:33.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:23:32 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:23:33.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:23:32 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 17745 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:23:33.062 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:23:32 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:23:33.062 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:23:32 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 17745 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:23:33.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:23:33 smithi115 conmon[90393]: 2022-05-14T04:23:33.059+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:23:34.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:23:34 smithi115 conmon[90393]: 2022-05-14T04:23:34.039+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:23:34.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:23:34 smithi047 ceph-mon[77147]: pgmap v9224: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:23:34.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:23:34 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:23:34.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:23:34 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:23:34.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:23:34 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:23:34.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:23:34 smithi115 ceph-mon[78593]: pgmap v9224: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:23:34.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:23:34 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:23:34.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:23:34 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:23:34.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:23:34 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:23:35.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:23:35 smithi115 conmon[90393]: 2022-05-14T04:23:35.025+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:23:35.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:23:35 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:23:35.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:23:35 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:23:36.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:23:35 smithi115 conmon[90393]: 2022-05-14T04:23:35.994+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:23:36.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:23:36 smithi047 ceph-mon[77147]: pgmap v9225: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:23:36.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:23:36 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:23:36.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:23:36 smithi115 ceph-mon[78593]: pgmap v9225: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:23:36.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:23:36 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:23:37.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:23:37 smithi115 conmon[90393]: 2022-05-14T04:23:37.014+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:23:37.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:23:37 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:23:37.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:23:37 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 17750 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:23:37.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:23:37 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:23:37.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:23:37 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 17750 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:23:38.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:23:38 smithi115 conmon[90393]: 2022-05-14T04:23:38.046+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:23:38.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:23:38 smithi047 ceph-mon[77147]: pgmap v9226: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:23:38.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:23:38 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:23:38.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:23:38 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:23:38.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:23:38 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:23:38.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:23:38 smithi115 ceph-mon[78593]: pgmap v9226: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:23:38.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:23:38 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:23:38.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:23:38 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:23:38.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:23:38 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:23:39.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:23:39 smithi115 conmon[90393]: 2022-05-14T04:23:39.058+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:23:39.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:23:39 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:23:39.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:23:39 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:23:40.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:23:40 smithi115 conmon[90393]: 2022-05-14T04:23:40.073+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:23:40.642 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:23:40 smithi115 ceph-mon[78593]: pgmap v9227: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:23:40.642 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:23:40 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:23:40.642 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:23:40 smithi047 ceph-mon[77147]: pgmap v9227: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:23:40.643 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:23:40 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:23:41.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:23:41 smithi115 conmon[90393]: 2022-05-14T04:23:41.038+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:23:41.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:23:41 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:23:41.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:23:41 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T04:23:41.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:23:41 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T04:23:41.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:23:41 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T04:23:41.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:23:41 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:23:41.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:23:41 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T04:23:41.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:23:41 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T04:23:41.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:23:41 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T04:23:42.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:23:42 smithi115 conmon[90393]: 2022-05-14T04:23:42.032+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:23:42.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:23:42 smithi047 ceph-mon[77147]: pgmap v9228: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:23:42.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:23:42 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:23:42.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:23:42 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:23:42.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:23:42 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:23:42.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:23:42 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 17755 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:23:42.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:23:42 smithi115 ceph-mon[78593]: pgmap v9228: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:23:42.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:23:42 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:23:42.862 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:23:42 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:23:42.862 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:23:42 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:23:42.862 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:23:42 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 17755 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:23:43.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:23:43 smithi115 conmon[90393]: 2022-05-14T04:23:43.007+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:23:43.667 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:23:43 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:23:43.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:23:43 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:23:44.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:23:43 smithi047 conmon[78521]: 2022-05-14T04:23:43.664+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:23:44.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:23:43 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T04:23:44.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:23:43 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:23:44.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:23:43 smithi047 conmon[78521]: yield 2022-05-14T04:23:44.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:23:43 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:23:44.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:23:43 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:23:44.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:23:43 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:23:44.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:23:43 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T04:23:44.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:23:43 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:23:44.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:23:43 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T04:23:44.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:23:43 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:23:44.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:23:43 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T04:23:44.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:23:43 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:23:44.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:23:43 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T04:23:44.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:23:43 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:23:44.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:23:43 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T04:23:44.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:23:43 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:23:44.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:23:43 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:23:44.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:23:43 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:23:44.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:23:44 smithi115 conmon[90393]: 2022-05-14T04:23:43.999+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:23:44.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:23:44 smithi047 ceph-mon[77147]: pgmap v9229: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:23:44.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:23:44 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:23:44.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:23:44 smithi115 ceph-mon[78593]: pgmap v9229: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:23:44.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:23:44 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:23:45.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:23:44 smithi115 conmon[90393]: 2022-05-14T04:23:44.981+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:23:45.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:23:45 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:23:45.793 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T04:23:45.793 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:23:45.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T04:23:45.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:23:45.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:23:45.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:23:45.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T04:23:45.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:23:45.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T04:23:45.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:23:45.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T04:23:45.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:23:45.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T04:23:45.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:23:45.797 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T04:23:45.797 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:23:45.797 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:23:45.798 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:23:45.798 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:23:45 smithi047 ceph-mon[77147]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T04:23:45.798 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:23:45 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:23:45.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:23:45 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:23:45.859 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T04:23:45.860 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:23:45.860 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T04:23:45.860 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:23:45.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:23:45.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:23:45.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T04:23:45.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:23:45.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T04:23:45.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:23:45.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T04:23:45.863 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:23:45.863 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T04:23:45.863 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:23:45.863 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T04:23:45.864 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:23:45.864 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:23:45.864 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:23:45.864 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:23:45 smithi115 ceph-mon[78593]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T04:23:45.864 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:23:45 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:23:46.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:23:45 smithi115 conmon[90393]: 2022-05-14T04:23:45.985+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:23:46.739 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:23:46 smithi047 ceph-mon[77147]: pgmap v9230: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:23:46.739 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:23:46 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:23:46.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:23:46 smithi115 ceph-mon[78593]: pgmap v9230: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:23:46.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:23:46 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:23:47.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:23:46 smithi047 conmon[78521]: 2022-05-14T04:23:46.736+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:23:47.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:23:46 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T04:23:47.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:23:46 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:23:47.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:23:46 smithi047 conmon[78521]: yield 2022-05-14T04:23:47.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:23:46 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:23:47.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:23:46 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:23:47.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:23:46 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:23:47.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:23:46 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T04:23:47.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:23:46 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:23:47.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:23:46 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T04:23:47.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:23:46 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:23:47.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:23:46 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T04:23:47.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:23:46 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:23:47.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:23:46 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T04:23:47.048 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:23:46 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:23:47.048 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:23:46 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T04:23:47.048 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:23:46 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:23:47.048 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:23:46 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:23:47.048 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:23:46 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:23:47.049 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:23:46 smithi047 conmon[78521]: 2022-05-14T04:23:46.737+0000 7f5344781700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T04:23:47.049 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:23:46 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T04:23:47.049 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:23:46 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:23:47.049 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:23:46 smithi047 conmon[78521]: yield 2022-05-14T04:23:47.050 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:23:46 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:23:47.050 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:23:46 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:23:47.050 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:23:46 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:23:47.050 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:23:46 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T04:23:47.050 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:23:46 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:23:47.051 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:23:46 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T04:23:47.051 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:23:46 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:23:47.051 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:23:46 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T04:23:47.051 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:23:46 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:23:47.051 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:23:46 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T04:23:47.052 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:23:46 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:23:47.052 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:23:46 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T04:23:47.052 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:23:46 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:23:47.053 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:23:46 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:23:47.053 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:23:46 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:23:47.053 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:23:46 smithi047 conmon[78521]: 2022-05-14T04:23:47.053 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:23:46 smithi047 conmon[78521]: During handling of the above exception, another exception occurred: 2022-05-14T04:23:47.053 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:23:46 smithi047 conmon[78521]: 2022-05-14T04:23:47.054 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:23:46 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T04:23:47.054 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:23:46 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T04:23:47.054 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:23:46 smithi047 conmon[78521]: return f(*arg) 2022-05-14T04:23:47.054 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:23:46 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T04:23:47.055 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:23:46 smithi047 conmon[78521]: self._write_client_files(client_files, host) 2022-05-14T04:23:47.055 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:23:46 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T04:23:47.055 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:23:46 smithi047 conmon[78521]: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T04:23:47.055 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:23:46 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T04:23:47.055 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:23:46 smithi047 conmon[78521]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T04:23:47.056 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:23:46 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T04:23:47.056 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:23:46 smithi047 conmon[78521]: return self.event_loop.get_result(coro) 2022-05-14T04:23:47.056 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:23:46 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T04:23:47.057 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:23:46 smithi047 conmon[78521]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T04:23:47.057 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:23:46 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T04:23:47.058 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:23:46 smithi047 conmon[78521]: return self.__get_result() 2022-05-14T04:23:47.058 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:23:46 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T04:23:47.058 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:23:46 smithi047 conmon[78521]: raise self._exception 2022-05-14T04:23:47.058 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:23:46 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T04:23:47.059 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:23:46 smithi047 conmon[78521]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T04:23:47.059 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:23:46 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T04:23:47.059 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:23:46 smithi047 conmon[78521]: conn = await self._remote_connection(host, addr) 2022-05-14T04:23:47.059 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:23:46 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T04:23:47.060 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:23:46 smithi047 conmon[78521]: raise 2022-05-14T04:23:47.060 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:23:46 smithi047 conmon[78521]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T04:23:47.060 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:23:46 smithi047 conmon[78521]: self.gen.throw(type, value, traceback) 2022-05-14T04:23:47.060 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:23:46 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T04:23:47.060 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:23:46 smithi047 conmon[78521]: raise OrchestratorError(msg) 2022-05-14T04:23:47.061 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:23:46 smithi047 conmon[78521]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:23:47.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:23:46 smithi115 conmon[90393]: 2022-05-14T04:23:46.994+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:23:47.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:23:47 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:23:47.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:23:47 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 17760 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:23:47.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:23:47 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:23:47.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:23:47 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 17760 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:23:48.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:23:48 smithi115 conmon[90393]: 2022-05-14T04:23:48.008+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:23:48.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:23:48 smithi047 ceph-mon[77147]: pgmap v9231: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:23:48.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:23:48 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:23:48.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T04:23:48.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:23:48.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T04:23:48.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:23:48.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:23:48.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:23:48.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T04:23:48.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:23:48.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T04:23:48.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:23:48.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T04:23:48.797 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:23:48.797 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T04:23:48.798 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:23:48.798 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T04:23:48.798 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:23:48.798 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:23:48.798 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:23:48.799 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:23:48 smithi047 ceph-mon[77147]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T04:23:48.799 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T04:23:48.799 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:23:48.799 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T04:23:48.800 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:23:48.800 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:23:48.800 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:23:48.800 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T04:23:48.800 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:23:48.801 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T04:23:48.801 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:23:48.801 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T04:23:48.801 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:23:48.801 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T04:23:48.802 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:23:48.802 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T04:23:48.802 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:23:48.802 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:23:48.802 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:23:48.803 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T04:23:48.803 INFO:journalctl@ceph.mon.b.smithi047.stdout: During handling of the above exception, another exception occurred: 2022-05-14T04:23:48.803 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T04:23:48.803 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T04:23:48.803 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T04:23:48.804 INFO:journalctl@ceph.mon.b.smithi047.stdout: return f(*arg) 2022-05-14T04:23:48.804 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T04:23:48.804 INFO:journalctl@ceph.mon.b.smithi047.stdout: self._write_client_files(client_files, host) 2022-05-14T04:23:48.804 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T04:23:48.805 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T04:23:48.805 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T04:23:48.805 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T04:23:48.805 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T04:23:48.805 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.event_loop.get_result(coro) 2022-05-14T04:23:48.806 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T04:23:48.806 INFO:journalctl@ceph.mon.b.smithi047.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T04:23:48.806 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T04:23:48.806 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.__get_result() 2022-05-14T04:23:48.806 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T04:23:48.807 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise self._exception 2022-05-14T04:23:48.807 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T04:23:48.807 INFO:journalctl@ceph.mon.b.smithi047.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T04:23:48.807 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T04:23:48.807 INFO:journalctl@ceph.mon.b.smithi047.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T04:23:48.808 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T04:23:48.808 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise 2022-05-14T04:23:48.809 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T04:23:48.809 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.gen.throw(type, value, traceback) 2022-05-14T04:23:48.809 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T04:23:48.809 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OrchestratorError(msg) 2022-05-14T04:23:48.810 INFO:journalctl@ceph.mon.b.smithi047.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:23:48.810 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:23:48 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:23:48.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:23:48 smithi115 ceph-mon[78593]: pgmap v9231: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:23:48.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:23:48 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:23:48.860 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T04:23:48.860 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:23:48.860 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T04:23:48.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:23:48.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:23:48.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:23:48.863 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T04:23:48.863 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:23:48.863 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T04:23:48.863 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:23:48.863 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T04:23:48.864 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:23:48.864 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T04:23:48.864 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:23:48.864 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T04:23:48.865 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:23:48.865 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:23:48.865 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:23:48.865 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:23:48 smithi115 ceph-mon[78593]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T04:23:48.865 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T04:23:48.866 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:23:48.866 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T04:23:48.866 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:23:48.866 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:23:48.867 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:23:48.867 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T04:23:48.867 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:23:48.867 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T04:23:48.867 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:23:48.868 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T04:23:48.868 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:23:48.868 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T04:23:48.868 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:23:48.869 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T04:23:48.869 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:23:48.869 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:23:48.869 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:23:48.869 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T04:23:48.870 INFO:journalctl@ceph.mon.c.smithi115.stdout: During handling of the above exception, another exception occurred: 2022-05-14T04:23:48.870 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T04:23:48.870 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T04:23:48.871 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T04:23:48.871 INFO:journalctl@ceph.mon.c.smithi115.stdout: return f(*arg) 2022-05-14T04:23:48.871 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T04:23:48.872 INFO:journalctl@ceph.mon.c.smithi115.stdout: self._write_client_files(client_files, host) 2022-05-14T04:23:48.872 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T04:23:48.872 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T04:23:48.872 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T04:23:48.872 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T04:23:48.873 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T04:23:48.873 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.event_loop.get_result(coro) 2022-05-14T04:23:48.873 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T04:23:48.873 INFO:journalctl@ceph.mon.c.smithi115.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T04:23:48.873 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T04:23:48.874 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.__get_result() 2022-05-14T04:23:48.874 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T04:23:48.874 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise self._exception 2022-05-14T04:23:48.874 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T04:23:48.875 INFO:journalctl@ceph.mon.c.smithi115.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T04:23:48.875 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T04:23:48.875 INFO:journalctl@ceph.mon.c.smithi115.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T04:23:48.875 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T04:23:48.875 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise 2022-05-14T04:23:48.876 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T04:23:48.876 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.gen.throw(type, value, traceback) 2022-05-14T04:23:48.876 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T04:23:48.876 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OrchestratorError(msg) 2022-05-14T04:23:48.876 INFO:journalctl@ceph.mon.c.smithi115.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:23:48.877 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:23:48 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:23:49.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:23:49 smithi115 conmon[90393]: 2022-05-14T04:23:49.042+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:23:49.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:23:49 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:23:49.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:23:49 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:23:50.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:23:50 smithi115 conmon[90393]: 2022-05-14T04:23:50.047+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:23:50.795 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:23:50 smithi047 ceph-mon[77147]: pgmap v9232: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:23:50.795 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:23:50 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:23:50.795 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:23:50 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:23:50.796 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:23:50 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:23:50.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:23:50 smithi115 ceph-mon[78593]: pgmap v9232: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:23:50.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:23:50 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:23:50.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:23:50 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:23:50.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:23:50 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:23:51.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:23:51 smithi115 conmon[90393]: 2022-05-14T04:23:51.037+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:23:51.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:23:51 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:23:51.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:23:51 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:23:52.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:23:52 smithi115 conmon[90393]: 2022-05-14T04:23:52.030+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:23:52.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:23:52 smithi047 ceph-mon[77147]: pgmap v9233: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:23:52.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:23:52 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:23:52.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:23:52 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:23:52.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:23:52 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 17765 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:23:52.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:23:52 smithi115 ceph-mon[78593]: pgmap v9233: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:23:52.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:23:52 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:23:52.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:23:52 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:23:52.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:23:52 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 17765 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:23:53.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:23:53 smithi115 conmon[90393]: 2022-05-14T04:23:53.060+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:23:53.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:23:53 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:23:53.864 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:23:53 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:23:54.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:23:54 smithi115 conmon[90393]: 2022-05-14T04:23:54.089+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:23:54.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:23:54 smithi047 ceph-mon[77147]: pgmap v9234: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:23:54.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:23:54 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:23:54.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:23:54 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:23:54.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:23:54 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:23:54.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:23:54 smithi115 ceph-mon[78593]: pgmap v9234: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:23:54.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:23:54 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:23:54.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:23:54 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:23:54.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:23:54 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:23:55.361 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:23:55 smithi115 conmon[90393]: 2022-05-14T04:23:55.065+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:23:55.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:23:55 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:23:55.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:23:55 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:23:56.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:23:56 smithi115 conmon[90393]: 2022-05-14T04:23:56.084+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:23:56.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:23:56 smithi047 ceph-mon[77147]: pgmap v9235: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:23:56.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:23:56 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:23:56.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:23:56 smithi115 ceph-mon[78593]: pgmap v9235: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:23:56.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:23:56 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:23:57.447 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:23:57 smithi115 conmon[90393]: 2022-05-14T04:23:57.125+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:23:57.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:23:57 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:23:57.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:23:57 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 17769 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:23:57.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:23:57 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:23:57.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:23:57 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 17769 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:23:58.448 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:23:58 smithi115 conmon[90393]: 2022-05-14T04:23:58.109+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:23:58.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:23:58 smithi047 ceph-mon[77147]: pgmap v9236: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:23:58.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:23:58 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:23:58.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:23:58 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:23:58.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:23:58 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:23:58.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:23:58 smithi115 ceph-mon[78593]: pgmap v9236: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:23:58.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:23:58 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:23:58.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:23:58 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:23:58.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:23:58 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:23:59.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:23:59 smithi115 conmon[90393]: 2022-05-14T04:23:59.095+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:23:59.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:23:59 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:23:59.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:23:59 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:24:00.361 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:24:00 smithi115 conmon[90393]: 2022-05-14T04:24:00.066+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:24:00.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:24:00 smithi047 ceph-mon[77147]: pgmap v9237: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:24:00.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:24:00 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:24:00.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:24:00 smithi115 ceph-mon[78593]: pgmap v9237: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:24:00.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:24:00 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:24:01.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:24:01 smithi115 conmon[90393]: 2022-05-14T04:24:01.051+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:24:01.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:24:01 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:24:01.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:24:01 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:24:02.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:24:02 smithi115 conmon[90393]: 2022-05-14T04:24:02.026+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:24:02.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:24:02 smithi047 ceph-mon[77147]: pgmap v9238: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:24:02.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:24:02 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:24:02.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:24:02 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:24:02.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:24:02 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:24:02.795 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:24:02 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 17775 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:24:02.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:24:02 smithi115 ceph-mon[78593]: pgmap v9238: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:24:02.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:24:02 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:24:02.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:24:02 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:24:02.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:24:02 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:24:02.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:24:02 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 17775 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:24:03.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:24:03 smithi115 conmon[90393]: 2022-05-14T04:24:03.024+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:24:03.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:24:03 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:24:03.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:24:03 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:24:04.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:24:04 smithi115 conmon[90393]: 2022-05-14T04:24:04.025+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:24:04.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:24:04 smithi047 ceph-mon[77147]: pgmap v9239: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:24:04.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:24:04 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:24:04.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:24:04 smithi115 ceph-mon[78593]: pgmap v9239: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:24:04.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:24:04 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:24:05.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:24:05 smithi115 conmon[90393]: 2022-05-14T04:24:05.067+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:24:05.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:24:05 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:24:05.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:24:05 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:24:06.478 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:24:06 smithi115 conmon[90393]: 2022-05-14T04:24:06.107+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:24:06.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:24:06 smithi047 ceph-mon[77147]: pgmap v9240: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:24:06.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:24:06 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:24:06.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:24:06 smithi115 ceph-mon[78593]: pgmap v9240: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:24:06.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:24:06 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:24:07.482 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:24:07 smithi115 conmon[90393]: 2022-05-14T04:24:07.153+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:24:07.483 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:24:07 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:24:07.484 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:24:07 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 17780 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:24:07.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:24:07 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:24:07.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:24:07 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 17780 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:24:08.484 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:24:08 smithi115 conmon[90393]: 2022-05-14T04:24:08.136+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:24:08.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:24:08 smithi047 ceph-mon[77147]: pgmap v9241: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:24:08.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:24:08 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:24:08.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:24:08 smithi115 ceph-mon[78593]: pgmap v9241: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:24:08.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:24:08 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:24:09.489 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:24:09 smithi115 conmon[90393]: 2022-05-14T04:24:09.132+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:24:09.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:24:09 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:24:09.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:24:09 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:24:10.491 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:24:10 smithi115 conmon[90393]: 2022-05-14T04:24:10.140+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:24:10.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:24:10 smithi047 ceph-mon[77147]: pgmap v9242: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:24:10.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:24:10 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:24:10.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:24:10 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:24:10.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:24:10 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:24:10.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:24:10 smithi115 ceph-mon[78593]: pgmap v9242: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:24:10.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:24:10 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:24:10.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:24:10 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:24:10.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:24:10 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:24:11.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:24:11 smithi115 conmon[90393]: 2022-05-14T04:24:11.092+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:24:11.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:24:11 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:24:11.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:24:11 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:24:12.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:24:12 smithi115 conmon[90393]: 2022-05-14T04:24:12.076+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:24:12.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:24:12 smithi047 ceph-mon[77147]: pgmap v9243: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:24:12.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:24:12 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:24:12.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:24:12 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 17785 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:24:12.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:24:12 smithi115 ceph-mon[78593]: pgmap v9243: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:24:12.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:24:12 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:24:12.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:24:12 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 17785 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:24:13.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:24:13 smithi115 conmon[90393]: 2022-05-14T04:24:13.032+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:24:13.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:24:13 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:24:13.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:24:13 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:24:14.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:24:14 smithi115 conmon[90393]: 2022-05-14T04:24:14.018+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:24:14.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:24:14 smithi047 ceph-mon[77147]: pgmap v9244: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:24:14.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:24:14 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:24:14.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:24:14 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:24:14.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:24:14 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:24:14.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:24:14 smithi115 ceph-mon[78593]: pgmap v9244: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:24:14.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:24:14 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:24:14.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:24:14 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:24:14.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:24:14 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:24:15.360 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:24:15 smithi115 conmon[90393]: 2022-05-14T04:24:15.025+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:24:15.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:24:15 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T04:24:15.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:24:15 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T04:24:15.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:24:15 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:24:15.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:24:15 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T04:24:15.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:24:15 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T04:24:15.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:24:15 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:24:16.360 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:24:16 smithi115 conmon[90393]: 2022-05-14T04:24:16.038+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:24:16.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:24:16 smithi047 ceph-mon[77147]: pgmap v9245: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:24:16.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:24:16 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:24:16.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:24:16 smithi115 ceph-mon[78593]: pgmap v9245: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:24:16.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:24:16 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:24:17.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:24:17 smithi115 conmon[90393]: 2022-05-14T04:24:17.002+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:24:17.797 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:24:17 smithi047 ceph-mon[77147]: pgmap v9246: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:24:17.797 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:24:17 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:24:17.798 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:24:17 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 17790 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:24:17.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:24:17 smithi115 ceph-mon[78593]: pgmap v9246: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:24:17.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:24:17 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:24:17.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:24:17 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 17790 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:24:18.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:24:17 smithi115 conmon[90393]: 2022-05-14T04:24:17.980+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:24:18.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:24:18 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:24:18.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:24:18 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:24:18.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:24:18 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:24:18.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:24:18 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:24:18.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:24:18 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:24:18.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:24:18 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:24:19.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:24:18 smithi115 conmon[90393]: 2022-05-14T04:24:18.953+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:24:19.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:24:19 smithi047 ceph-mon[77147]: pgmap v9247: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:24:19.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:24:19 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:24:19.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:24:19 smithi115 ceph-mon[78593]: pgmap v9247: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:24:19.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:24:19 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:24:20.360 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:24:19 smithi115 conmon[90393]: 2022-05-14T04:24:19.986+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:24:20.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:24:20 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:24:20.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:24:20 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:24:21.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:24:21 smithi115 conmon[90393]: 2022-05-14T04:24:21.030+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:24:21.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:24:21 smithi047 ceph-mon[77147]: pgmap v9248: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:24:21.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:24:21 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:24:21.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:24:21 smithi115 ceph-mon[78593]: pgmap v9248: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:24:21.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:24:21 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:24:22.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:24:21 smithi115 conmon[90393]: 2022-05-14T04:24:21.985+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:24:22.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:24:22 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:24:22.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:24:22 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:24:22.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:24:22 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:24:22.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:24:22 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 17795 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:24:22.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:24:22 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:24:22.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:24:22 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:24:22.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:24:22 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:24:22.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:24:22 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 17795 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:24:23.362 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:24:22 smithi115 conmon[90393]: 2022-05-14T04:24:22.955+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:24:23.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:24:23 smithi047 ceph-mon[77147]: pgmap v9249: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:24:23.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:24:23 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:24:23.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:24:23 smithi115 ceph-mon[78593]: pgmap v9249: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:24:23.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:24:23 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:24:24.361 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:24:23 smithi115 conmon[90393]: 2022-05-14T04:24:23.997+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:24:24.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:24:24 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:24:24.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:24:24 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:24:25.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:24:25 smithi115 conmon[90393]: 2022-05-14T04:24:25.012+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:24:25.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:24:25 smithi047 ceph-mon[77147]: pgmap v9250: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:24:25.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:24:25 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:24:25.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:24:25 smithi115 ceph-mon[78593]: pgmap v9250: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:24:25.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:24:25 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:24:26.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:24:26 smithi115 conmon[90393]: 2022-05-14T04:24:26.035+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:24:26.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:24:26 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:24:26.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:24:26 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:24:27.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:24:27 smithi115 conmon[90393]: 2022-05-14T04:24:27.073+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:24:27.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:24:27 smithi115 ceph-mon[78593]: pgmap v9251: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:24:27.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:24:27 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:24:27.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:24:27 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 17800 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:24:28.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:24:27 smithi047 ceph-mon[77147]: pgmap v9251: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:24:28.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:24:27 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:24:28.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:24:27 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 17800 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:24:28.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:24:28 smithi115 conmon[90393]: 2022-05-14T04:24:28.089+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:24:28.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:24:28 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:24:29.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:24:28 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:24:29.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:24:29 smithi115 conmon[90393]: 2022-05-14T04:24:29.097+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:24:29.862 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:24:29 smithi115 ceph-mon[78593]: pgmap v9252: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:24:29.863 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:24:29 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:24:30.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:24:29 smithi047 ceph-mon[77147]: pgmap v9252: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:24:30.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:24:29 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:24:30.553 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:24:30 smithi115 conmon[90393]: 2022-05-14T04:24:30.128+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:24:30.870 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:24:30 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:24:30.870 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:24:30 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:24:30.870 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:24:30 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:24:31.046 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:24:30 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:24:31.046 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:24:30 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:24:31.047 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:24:30 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:24:31.557 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:24:31 smithi115 conmon[90393]: 2022-05-14T04:24:31.147+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:24:31.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:24:31 smithi115 ceph-mon[78593]: pgmap v9253: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:24:31.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:24:31 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:24:32.051 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:24:31 smithi047 ceph-mon[77147]: pgmap v9253: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:24:32.052 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:24:31 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:24:32.562 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:24:32 smithi115 conmon[90393]: 2022-05-14T04:24:32.150+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:24:32.871 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:24:32 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:24:32.872 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:24:32 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 17805 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:24:33.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:24:32 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:24:33.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:24:32 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 17805 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:24:33.567 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:24:33 smithi115 conmon[90393]: 2022-05-14T04:24:33.140+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:24:33.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:24:33 smithi115 ceph-mon[78593]: pgmap v9254: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:24:33.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:24:33 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:24:34.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:24:33 smithi047 ceph-mon[77147]: pgmap v9254: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:24:34.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:24:33 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:24:34.568 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:24:34 smithi115 conmon[90393]: 2022-05-14T04:24:34.111+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:24:34.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:24:34 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:24:34.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:24:34 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:24:34.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:24:34 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:24:35.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:24:34 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:24:35.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:24:34 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:24:35.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:24:34 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:24:35.567 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:24:35 smithi115 conmon[90393]: 2022-05-14T04:24:35.119+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:24:35.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:24:35 smithi115 ceph-mon[78593]: pgmap v9255: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:24:35.862 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:24:35 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:24:36.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:24:35 smithi047 ceph-mon[77147]: pgmap v9255: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:24:36.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:24:35 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:24:36.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:24:36 smithi115 conmon[90393]: 2022-05-14T04:24:36.100+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:24:36.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:24:36 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:24:37.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:24:36 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:24:37.584 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:24:37 smithi115 conmon[90393]: 2022-05-14T04:24:37.145+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:24:37.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:24:37 smithi115 ceph-mon[78593]: pgmap v9256: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:24:37.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:24:37 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:24:37.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:24:37 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 17810 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:24:38.046 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:24:37 smithi047 ceph-mon[77147]: pgmap v9256: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:24:38.047 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:24:37 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:24:38.047 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:24:37 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 17810 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:24:38.584 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:24:38 smithi115 conmon[90393]: 2022-05-14T04:24:38.155+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:24:38.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:24:38 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:24:38.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:24:38 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:24:38.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:24:38 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:24:39.045 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:24:38 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:24:39.046 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:24:38 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:24:39.046 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:24:38 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:24:39.588 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:24:39 smithi115 conmon[90393]: 2022-05-14T04:24:39.158+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:24:39.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:24:39 smithi115 ceph-mon[78593]: pgmap v9257: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:24:39.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:24:39 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:24:40.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:24:39 smithi047 ceph-mon[77147]: pgmap v9257: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:24:40.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:24:39 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:24:40.590 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:24:40 smithi115 conmon[90393]: 2022-05-14T04:24:40.120+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:24:40.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:24:40 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:24:41.046 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:24:40 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:24:41.596 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:24:41 smithi115 conmon[90393]: 2022-05-14T04:24:41.123+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:24:41.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:24:41 smithi115 ceph-mon[78593]: pgmap v9258: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:24:41.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:24:41 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:24:42.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:24:41 smithi047 ceph-mon[77147]: pgmap v9258: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:24:42.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:24:41 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:24:42.602 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:24:42 smithi115 conmon[90393]: 2022-05-14T04:24:42.151+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:24:42.862 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:24:42 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:24:42.863 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:24:42 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:24:42.863 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:24:42 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:24:42.863 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:24:42 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 17815 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:24:43.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:24:42 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:24:43.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:24:42 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:24:43.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:24:42 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:24:43.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:24:42 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 17815 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:24:43.605 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:24:43 smithi115 conmon[90393]: 2022-05-14T04:24:43.157+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:24:43.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:24:43 smithi115 ceph-mon[78593]: pgmap v9259: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:24:43.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:24:43 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:24:44.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:24:43 smithi047 ceph-mon[77147]: pgmap v9259: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:24:44.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:24:43 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:24:44.604 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:24:44 smithi115 conmon[90393]: 2022-05-14T04:24:44.207+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:24:44.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:24:44 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:24:45.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:24:44 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:24:45.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:24:45 smithi115 conmon[90393]: 2022-05-14T04:24:45.186+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:24:46.057 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:24:45 smithi047 ceph-mon[77147]: pgmap v9260: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:24:46.058 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:24:45 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:24:46.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:24:45 smithi115 ceph-mon[78593]: pgmap v9260: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:24:46.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:24:45 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:24:46.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:24:46 smithi115 conmon[90393]: 2022-05-14T04:24:46.161+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:24:46.875 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:24:46 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:24:46.875 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:24:46 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:24:47.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:24:47 smithi047 conmon[78521]: 2022-05-14T04:24:47.215+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:24:47.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:24:47 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T04:24:47.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:24:47 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:24:47.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:24:47 smithi047 conmon[78521]: yield 2022-05-14T04:24:47.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:24:47 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:24:47.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:24:47 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:24:47.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:24:47 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:24:47.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:24:47 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T04:24:47.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:24:47 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:24:47.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:24:47 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T04:24:47.547 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:24:47 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:24:47.547 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:24:47 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T04:24:47.547 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:24:47 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:24:47.547 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:24:47 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T04:24:47.548 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:24:47 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:24:47.548 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:24:47 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T04:24:47.549 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:24:47 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:24:47.549 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:24:47 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:24:47.549 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:24:47 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:24:47.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:24:47 smithi115 conmon[90393]: 2022-05-14T04:24:47.187+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:24:48.045 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:24:47 smithi047 ceph-mon[77147]: pgmap v9261: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:24:48.046 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:24:47 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T04:24:48.046 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:24:47 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T04:24:48.046 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:24:47 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T04:24:48.047 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:24:47 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:24:48.047 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:24:47 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:24:48.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T04:24:48.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:24:48.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T04:24:48.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:24:48.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:24:48.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:24:48.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T04:24:48.049 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:24:48.049 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T04:24:48.049 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:24:48.049 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T04:24:48.050 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:24:48.050 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T04:24:48.050 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:24:48.050 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T04:24:48.050 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:24:48.051 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:24:48.051 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:24:48.051 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:24:47 smithi047 ceph-mon[77147]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T04:24:48.051 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:24:47 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 17820 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:24:48.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:24:47 smithi115 ceph-mon[78593]: pgmap v9261: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:24:48.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:24:47 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T04:24:48.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:24:47 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T04:24:48.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:24:47 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T04:24:48.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:24:47 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:24:48.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:24:47 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:24:48.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T04:24:48.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:24:48.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T04:24:48.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:24:48.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:24:48.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:24:48.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T04:24:48.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:24:48.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T04:24:48.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:24:48.113 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T04:24:48.113 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:24:48.113 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T04:24:48.113 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:24:48.114 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T04:24:48.114 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:24:48.114 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:24:48.114 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:24:48.114 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:24:47 smithi115 ceph-mon[78593]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T04:24:48.115 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:24:47 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 17820 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:24:48.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:24:48 smithi115 conmon[90393]: 2022-05-14T04:24:48.199+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:24:49.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:24:48 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:24:49.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:24:48 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:24:49.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:24:49 smithi115 conmon[90393]: 2022-05-14T04:24:49.198+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:24:50.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:24:49 smithi047 ceph-mon[77147]: pgmap v9262: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:24:50.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:24:49 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:24:50.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:24:49 smithi115 ceph-mon[78593]: pgmap v9262: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:24:50.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:24:49 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:24:50.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:24:50 smithi047 conmon[78521]: 2022-05-14T04:24:50.287+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:24:50.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:24:50 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T04:24:50.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:24:50 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:24:50.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:24:50 smithi047 conmon[78521]: yield 2022-05-14T04:24:50.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:24:50 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:24:50.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:24:50 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:24:50.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:24:50 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:24:50.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:24:50 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T04:24:50.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:24:50 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:24:50.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:24:50 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T04:24:50.547 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:24:50 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:24:50.547 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:24:50 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T04:24:50.547 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:24:50 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:24:50.547 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:24:50 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T04:24:50.548 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:24:50 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:24:50.548 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:24:50 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T04:24:50.548 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:24:50 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:24:50.548 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:24:50 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:24:50.548 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:24:50 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:24:50.549 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:24:50 smithi047 conmon[78521]: 2022-05-14T04:24:50.288+0000 7f533ff78700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T04:24:50.549 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:24:50 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T04:24:50.549 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:24:50 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:24:50.549 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:24:50 smithi047 conmon[78521]: yield 2022-05-14T04:24:50.550 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:24:50 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:24:50.550 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:24:50 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:24:50.550 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:24:50 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:24:50.550 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:24:50 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T04:24:50.550 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:24:50 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:24:50.551 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:24:50 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T04:24:50.551 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:24:50 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:24:50.551 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:24:50 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T04:24:50.551 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:24:50 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:24:50.552 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:24:50 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T04:24:50.552 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:24:50 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:24:50.553 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:24:50 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T04:24:50.554 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:24:50 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:24:50.554 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:24:50 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:24:50.554 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:24:50 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:24:50.554 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:24:50 smithi047 conmon[78521]: 2022-05-14T04:24:50.555 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:24:50 smithi047 conmon[78521]: During handling of the above exception, another exception occurred: 2022-05-14T04:24:50.555 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:24:50 smithi047 conmon[78521]: 2022-05-14T04:24:50.555 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:24:50 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T04:24:50.555 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:24:50 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T04:24:50.556 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:24:50 smithi047 conmon[78521]: return f(*arg) 2022-05-14T04:24:50.556 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:24:50 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T04:24:50.556 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:24:50 smithi047 conmon[78521]: self._write_client_files(client_files, host) 2022-05-14T04:24:50.556 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:24:50 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T04:24:50.557 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:24:50 smithi047 conmon[78521]: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T04:24:50.557 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:24:50 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T04:24:50.557 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:24:50 smithi047 conmon[78521]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T04:24:50.557 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:24:50 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T04:24:50.557 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:24:50 smithi047 conmon[78521]: return self.event_loop.get_result(coro) 2022-05-14T04:24:50.558 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:24:50 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T04:24:50.558 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:24:50 smithi047 conmon[78521]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T04:24:50.558 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:24:50 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T04:24:50.558 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:24:50 smithi047 conmon[78521]: return self.__get_result() 2022-05-14T04:24:50.559 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:24:50 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T04:24:50.559 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:24:50 smithi047 conmon[78521]: raise self._exception 2022-05-14T04:24:50.559 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:24:50 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T04:24:50.559 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:24:50 smithi047 conmon[78521]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T04:24:50.560 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:24:50 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T04:24:50.560 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:24:50 smithi047 conmon[78521]: conn = await self._remote_connection(host, addr) 2022-05-14T04:24:50.560 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:24:50 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T04:24:50.560 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:24:50 smithi047 conmon[78521]: raise 2022-05-14T04:24:50.560 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:24:50 smithi047 conmon[78521]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T04:24:50.561 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:24:50 smithi047 conmon[78521]: self.gen.throw(type, value, traceback) 2022-05-14T04:24:50.561 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:24:50 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T04:24:50.561 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:24:50 smithi047 conmon[78521]: raise OrchestratorError(msg) 2022-05-14T04:24:50.561 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:24:50 smithi047 conmon[78521]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:24:50.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:24:50 smithi115 conmon[90393]: 2022-05-14T04:24:50.173+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:24:51.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:24:50 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:24:51.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:24:50 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:24:51.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:24:50 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:24:51.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:24:50 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:24:51.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T04:24:51.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:24:51.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T04:24:51.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:24:51.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:24:51.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:24:51.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T04:24:51.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:24:51.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T04:24:51.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:24:51.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T04:24:51.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:24:51.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T04:24:51.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:24:51.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T04:24:51.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:24:51.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:24:51.049 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:24:51.050 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:24:50 smithi047 ceph-mon[77147]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T04:24:51.050 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T04:24:51.050 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:24:51.050 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T04:24:51.051 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:24:51.051 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:24:51.051 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:24:51.051 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T04:24:51.052 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:24:51.052 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T04:24:51.053 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:24:51.053 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T04:24:51.053 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:24:51.054 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T04:24:51.054 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:24:51.054 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T04:24:51.054 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:24:51.054 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:24:51.055 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:24:51.055 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T04:24:51.055 INFO:journalctl@ceph.mon.b.smithi047.stdout: During handling of the above exception, another exception occurred: 2022-05-14T04:24:51.055 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T04:24:51.056 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T04:24:51.056 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T04:24:51.056 INFO:journalctl@ceph.mon.b.smithi047.stdout: return f(*arg) 2022-05-14T04:24:51.056 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T04:24:51.057 INFO:journalctl@ceph.mon.b.smithi047.stdout: self._write_client_files(client_files, host) 2022-05-14T04:24:51.057 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T04:24:51.057 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T04:24:51.057 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T04:24:51.058 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T04:24:51.058 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T04:24:51.058 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.event_loop.get_result(coro) 2022-05-14T04:24:51.058 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T04:24:51.059 INFO:journalctl@ceph.mon.b.smithi047.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T04:24:51.059 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T04:24:51.061 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.__get_result() 2022-05-14T04:24:51.061 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T04:24:51.061 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise self._exception 2022-05-14T04:24:51.062 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T04:24:51.062 INFO:journalctl@ceph.mon.b.smithi047.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T04:24:51.062 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T04:24:51.063 INFO:journalctl@ceph.mon.b.smithi047.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T04:24:51.063 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T04:24:51.063 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise 2022-05-14T04:24:51.063 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T04:24:51.064 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.gen.throw(type, value, traceback) 2022-05-14T04:24:51.064 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T04:24:51.064 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OrchestratorError(msg) 2022-05-14T04:24:51.064 INFO:journalctl@ceph.mon.b.smithi047.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:24:51.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:24:50 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:24:51.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:24:50 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:24:51.112 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:24:50 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:24:51.112 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:24:50 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:24:51.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T04:24:51.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:24:51.113 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T04:24:51.113 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:24:51.113 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:24:51.113 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:24:51.114 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T04:24:51.114 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:24:51.114 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T04:24:51.114 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:24:51.114 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T04:24:51.115 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:24:51.115 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T04:24:51.115 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:24:51.115 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T04:24:51.115 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:24:51.116 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:24:51.116 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:24:51.116 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:24:50 smithi115 ceph-mon[78593]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T04:24:51.116 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T04:24:51.116 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:24:51.117 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T04:24:51.117 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:24:51.118 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:24:51.118 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:24:51.118 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T04:24:51.119 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:24:51.119 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T04:24:51.119 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:24:51.119 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T04:24:51.119 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:24:51.120 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T04:24:51.120 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:24:51.120 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T04:24:51.121 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:24:51.121 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:24:51.121 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:24:51.121 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T04:24:51.122 INFO:journalctl@ceph.mon.c.smithi115.stdout: During handling of the above exception, another exception occurred: 2022-05-14T04:24:51.122 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T04:24:51.122 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T04:24:51.122 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T04:24:51.122 INFO:journalctl@ceph.mon.c.smithi115.stdout: return f(*arg) 2022-05-14T04:24:51.123 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T04:24:51.123 INFO:journalctl@ceph.mon.c.smithi115.stdout: self._write_client_files(client_files, host) 2022-05-14T04:24:51.123 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T04:24:51.123 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T04:24:51.123 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T04:24:51.124 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T04:24:51.124 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T04:24:51.124 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.event_loop.get_result(coro) 2022-05-14T04:24:51.124 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T04:24:51.124 INFO:journalctl@ceph.mon.c.smithi115.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T04:24:51.125 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T04:24:51.125 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.__get_result() 2022-05-14T04:24:51.125 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T04:24:51.125 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise self._exception 2022-05-14T04:24:51.126 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T04:24:51.126 INFO:journalctl@ceph.mon.c.smithi115.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T04:24:51.126 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T04:24:51.126 INFO:journalctl@ceph.mon.c.smithi115.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T04:24:51.126 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T04:24:51.127 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise 2022-05-14T04:24:51.127 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T04:24:51.127 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.gen.throw(type, value, traceback) 2022-05-14T04:24:51.128 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T04:24:51.128 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OrchestratorError(msg) 2022-05-14T04:24:51.128 INFO:journalctl@ceph.mon.c.smithi115.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:24:51.610 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:24:51 smithi115 conmon[90393]: 2022-05-14T04:24:51.223+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:24:52.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:24:51 smithi047 ceph-mon[77147]: pgmap v9263: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:24:52.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:24:51 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:24:52.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:24:51 smithi115 ceph-mon[78593]: pgmap v9263: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:24:52.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:24:51 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:24:52.611 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:24:52 smithi115 conmon[90393]: 2022-05-14T04:24:52.198+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:24:53.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:24:52 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:24:53.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:24:52 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 17825 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:24:53.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:24:52 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:24:53.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:24:52 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 17825 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:24:53.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:24:53 smithi115 conmon[90393]: 2022-05-14T04:24:53.241+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:24:54.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:24:53 smithi047 ceph-mon[77147]: pgmap v9264: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:24:54.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:24:53 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:24:54.119 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:24:53 smithi115 ceph-mon[78593]: pgmap v9264: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:24:54.119 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:24:53 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:24:54.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:24:54 smithi115 conmon[90393]: 2022-05-14T04:24:54.257+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:24:55.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:24:54 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:24:55.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:24:54 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:24:55.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:24:54 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:24:55.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:24:54 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:24:55.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:24:54 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:24:55.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:24:54 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:24:55.610 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:24:55 smithi115 conmon[90393]: 2022-05-14T04:24:55.248+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:24:56.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:24:55 smithi047 ceph-mon[77147]: pgmap v9265: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:24:56.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:24:55 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:24:56.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:24:55 smithi115 ceph-mon[78593]: pgmap v9265: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:24:56.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:24:55 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:24:56.611 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:24:56 smithi115 conmon[90393]: 2022-05-14T04:24:56.259+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:24:57.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:24:56 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:24:57.113 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:24:56 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:24:57.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:24:57 smithi115 conmon[90393]: 2022-05-14T04:24:57.300+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:24:58.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:24:57 smithi047 ceph-mon[77147]: pgmap v9266: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:24:58.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:24:57 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:24:58.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:24:57 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 17830 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:24:58.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:24:57 smithi115 ceph-mon[78593]: pgmap v9266: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:24:58.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:24:57 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:24:58.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:24:57 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 17830 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:24:58.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:24:58 smithi115 conmon[90393]: 2022-05-14T04:24:58.322+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:24:59.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:24:58 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:24:59.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:24:58 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:24:59.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:24:58 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:24:59.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:24:58 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:24:59.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:24:58 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:24:59.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:24:58 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:24:59.610 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:24:59 smithi115 conmon[90393]: 2022-05-14T04:24:59.318+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:25:00.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:24:59 smithi047 ceph-mon[77147]: pgmap v9267: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:25:00.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:24:59 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:25:00.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:24:59 smithi115 ceph-mon[78593]: pgmap v9267: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:25:00.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:24:59 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:25:00.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:25:00 smithi115 conmon[90393]: 2022-05-14T04:25:00.336+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:25:01.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:25:00 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:25:01.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:25:00 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:25:01.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:25:01 smithi115 conmon[90393]: 2022-05-14T04:25:01.299+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:25:02.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:25:01 smithi047 ceph-mon[77147]: pgmap v9268: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:25:02.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:25:01 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:25:02.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:25:01 smithi115 ceph-mon[78593]: pgmap v9268: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:25:02.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:25:01 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:25:02.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:25:02 smithi115 conmon[90393]: 2022-05-14T04:25:02.271+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:25:03.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:25:02 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:25:03.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:25:02 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:25:03.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:25:02 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:25:03.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:25:02 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 17835 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:25:03.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:25:02 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:25:03.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:25:02 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:25:03.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:25:02 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:25:03.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:25:02 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 17835 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:25:03.611 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:25:03 smithi115 conmon[90393]: 2022-05-14T04:25:03.275+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:25:04.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:25:03 smithi047 ceph-mon[77147]: pgmap v9269: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:25:04.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:25:03 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:25:04.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:25:03 smithi115 ceph-mon[78593]: pgmap v9269: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:25:04.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:25:03 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:25:04.610 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:25:04 smithi115 conmon[90393]: 2022-05-14T04:25:04.287+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:25:05.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:25:04 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:25:05.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:25:04 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:25:05.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:25:05 smithi115 conmon[90393]: 2022-05-14T04:25:05.282+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:25:06.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:25:05 smithi047 ceph-mon[77147]: pgmap v9270: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:25:06.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:25:05 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:25:06.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:25:05 smithi115 ceph-mon[78593]: pgmap v9270: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:25:06.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:25:05 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:25:06.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:25:06 smithi115 conmon[90393]: 2022-05-14T04:25:06.266+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:25:07.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:25:06 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:25:07.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:25:06 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:25:07.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:25:07 smithi115 conmon[90393]: 2022-05-14T04:25:07.254+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:25:08.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:25:07 smithi047 ceph-mon[77147]: pgmap v9271: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:25:08.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:25:07 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:25:08.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:25:07 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 17840 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:25:08.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:25:07 smithi115 ceph-mon[78593]: pgmap v9271: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:25:08.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:25:07 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:25:08.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:25:07 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 17840 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:25:08.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:25:08 smithi115 conmon[90393]: 2022-05-14T04:25:08.301+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:25:09.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:25:08 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:25:09.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:25:08 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:25:09.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:25:09 smithi115 conmon[90393]: 2022-05-14T04:25:09.256+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:25:10.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:25:09 smithi047 ceph-mon[77147]: pgmap v9272: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:25:10.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:25:09 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:25:10.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:25:09 smithi115 ceph-mon[78593]: pgmap v9272: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:25:10.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:25:09 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:25:10.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:25:10 smithi115 conmon[90393]: 2022-05-14T04:25:10.296+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:25:11.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:25:10 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:25:11.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:25:10 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:25:11.045 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:25:10 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:25:11.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:25:10 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:25:11.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:25:10 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:25:11.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:25:10 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:25:11.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:25:11 smithi115 conmon[90393]: 2022-05-14T04:25:11.299+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:25:12.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:25:11 smithi047 ceph-mon[77147]: pgmap v9273: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:25:12.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:25:11 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:25:12.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:25:11 smithi115 ceph-mon[78593]: pgmap v9273: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:25:12.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:25:11 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:25:12.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:25:12 smithi115 conmon[90393]: 2022-05-14T04:25:12.326+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:25:13.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:25:12 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:25:13.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:25:12 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 17845 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:25:13.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:25:12 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:25:13.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:25:12 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 17845 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:25:13.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:25:13 smithi115 conmon[90393]: 2022-05-14T04:25:13.308+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:25:14.610 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:25:14 smithi115 conmon[90393]: 2022-05-14T04:25:14.283+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:25:14.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:25:14 smithi115 ceph-mon[78593]: pgmap v9274: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:25:14.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:25:14 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:25:14.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:25:14 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:25:14.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:25:14 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:25:14.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:25:14 smithi047 ceph-mon[77147]: pgmap v9274: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:25:14.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:25:14 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:25:14.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:25:14 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:25:14.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:25:14 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:25:15.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:25:15 smithi115 conmon[90393]: 2022-05-14T04:25:15.265+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:25:15.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:25:15 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:25:15.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:25:15 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T04:25:15.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:25:15 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T04:25:15.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:25:15 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:25:15.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:25:15 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T04:25:15.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:25:15 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T04:25:16.610 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:25:16 smithi115 conmon[90393]: 2022-05-14T04:25:16.242+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:25:16.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:25:16 smithi115 ceph-mon[78593]: pgmap v9275: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 7 op/s 2022-05-14T04:25:16.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:25:16 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:25:16.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:25:16 smithi047 ceph-mon[77147]: pgmap v9275: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 7 op/s 2022-05-14T04:25:16.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:25:16 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:25:17.610 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:25:17 smithi115 conmon[90393]: 2022-05-14T04:25:17.207+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:25:17.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:25:17 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:25:17.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:25:17 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 17850 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:25:17.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:25:17 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:25:17.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:25:17 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 17850 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:25:18.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:25:18 smithi115 conmon[90393]: 2022-05-14T04:25:18.241+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:25:18.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:25:18 smithi115 ceph-mon[78593]: pgmap v9276: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 4 op/s 2022-05-14T04:25:18.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:25:18 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:25:18.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:25:18 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:25:18.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:25:18 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:25:18.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:25:18 smithi047 ceph-mon[77147]: pgmap v9276: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 4 op/s 2022-05-14T04:25:18.795 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:25:18 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:25:18.795 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:25:18 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:25:18.795 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:25:18 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:25:19.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:25:19 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:25:19.610 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:25:19 smithi115 conmon[90393]: 2022-05-14T04:25:19.210+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:25:19.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:25:19 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:25:20.610 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:25:20 smithi115 conmon[90393]: 2022-05-14T04:25:20.235+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:25:20.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:25:20 smithi115 ceph-mon[78593]: pgmap v9277: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 4 op/s 2022-05-14T04:25:20.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:25:20 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:25:20.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:25:20 smithi047 ceph-mon[77147]: pgmap v9277: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 4 op/s 2022-05-14T04:25:20.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:25:20 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:25:21.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:25:21 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:25:21.610 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:25:21 smithi115 conmon[90393]: 2022-05-14T04:25:21.267+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:25:21.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:25:21 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:25:22.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:25:22 smithi115 conmon[90393]: 2022-05-14T04:25:22.244+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:25:22.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:25:22 smithi115 ceph-mon[78593]: pgmap v9278: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 7 op/s 2022-05-14T04:25:22.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:25:22 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:25:22.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:25:22 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:25:22.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:25:22 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:25:22.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:25:22 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 17855 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:25:22.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:25:22 smithi047 ceph-mon[77147]: pgmap v9278: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 7 op/s 2022-05-14T04:25:22.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:25:22 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:25:22.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:25:22 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:25:22.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:25:22 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:25:22.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:25:22 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 17855 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:25:23.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:25:23 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:25:23.610 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:25:23 smithi115 conmon[90393]: 2022-05-14T04:25:23.218+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:25:23.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:25:23 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:25:24.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:25:24 smithi115 conmon[90393]: 2022-05-14T04:25:24.237+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:25:24.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:25:24 smithi115 ceph-mon[78593]: pgmap v9279: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 4 op/s 2022-05-14T04:25:24.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:25:24 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:25:24.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:25:24 smithi047 ceph-mon[77147]: pgmap v9279: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 4 op/s 2022-05-14T04:25:24.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:25:24 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:25:25.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:25:25 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:25:25.610 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:25:25 smithi115 conmon[90393]: 2022-05-14T04:25:25.267+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:25:25.795 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:25:25 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:25:26.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:25:26 smithi115 ceph-mon[78593]: pgmap v9280: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 7 op/s 2022-05-14T04:25:26.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:25:26 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:25:26.610 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:25:26 smithi115 conmon[90393]: 2022-05-14T04:25:26.265+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:25:26.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:25:26 smithi047 ceph-mon[77147]: pgmap v9280: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 7 op/s 2022-05-14T04:25:26.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:25:26 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:25:27.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:25:27 smithi115 conmon[90393]: 2022-05-14T04:25:27.277+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:25:27.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:25:27 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:25:27.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:25:27 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 17860 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:25:27.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:25:27 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:25:27.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:25:27 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 17860 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:25:28.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:25:28 smithi115 conmon[90393]: 2022-05-14T04:25:28.269+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:25:28.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:25:28 smithi115 ceph-mon[78593]: pgmap v9281: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:25:28.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:25:28 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:25:28.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:25:28 smithi047 ceph-mon[77147]: pgmap v9281: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:25:28.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:25:28 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:25:29.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:25:29 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:25:29.610 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:25:29 smithi115 conmon[90393]: 2022-05-14T04:25:29.232+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:25:29.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:25:29 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:25:30.449 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:25:30 smithi115 conmon[90393]: 2022-05-14T04:25:30.191+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:25:30.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:25:30 smithi047 ceph-mon[77147]: pgmap v9282: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:25:30.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:25:30 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:25:30.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:25:30 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:25:30.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:25:30 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:25:30.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:25:30 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:25:30.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:25:30 smithi115 ceph-mon[78593]: pgmap v9282: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:25:30.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:25:30 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:25:30.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:25:30 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:25:30.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:25:30 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:25:30.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:25:30 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:25:31.454 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:25:31 smithi115 conmon[90393]: 2022-05-14T04:25:31.188+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:25:31.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:25:31 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:25:31.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:25:31 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:25:32.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:25:32 smithi115 conmon[90393]: 2022-05-14T04:25:32.231+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:25:32.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:25:32 smithi115 ceph-mon[78593]: pgmap v9283: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:25:32.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:25:32 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:25:32.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:25:32 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 17865 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:25:32.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:25:32 smithi047 ceph-mon[77147]: pgmap v9283: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:25:32.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:25:32 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:25:32.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:25:32 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 17865 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:25:33.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:25:33 smithi115 conmon[90393]: 2022-05-14T04:25:33.280+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:25:33.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:25:33 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:25:33.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:25:33 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:25:34.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:25:34 smithi115 conmon[90393]: 2022-05-14T04:25:34.283+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:25:34.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:25:34 smithi115 ceph-mon[78593]: pgmap v9284: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:25:34.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:25:34 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:25:34.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:25:34 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:25:34.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:25:34 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:25:34.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:25:34 smithi047 ceph-mon[77147]: pgmap v9284: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:25:34.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:25:34 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:25:34.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:25:34 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:25:34.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:25:34 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:25:35.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:25:35 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:25:35.610 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:25:35 smithi115 conmon[90393]: 2022-05-14T04:25:35.328+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:25:35.782 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:25:35 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:25:36.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:25:36 smithi115 ceph-mon[78593]: pgmap v9285: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:25:36.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:25:36 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:25:36.610 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:25:36 smithi115 conmon[90393]: 2022-05-14T04:25:36.287+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:25:36.655 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:25:36 smithi047 ceph-mon[77147]: pgmap v9285: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:25:36.655 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:25:36 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:25:37.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:25:37 smithi115 conmon[90393]: 2022-05-14T04:25:37.264+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:25:37.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:25:37 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:25:37.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:25:37 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 17870 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:25:37.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:25:37 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:25:37.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:25:37 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 17870 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:25:38.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:25:38 smithi115 conmon[90393]: 2022-05-14T04:25:38.222+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:25:38.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:25:38 smithi115 ceph-mon[78593]: pgmap v9286: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:25:38.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:25:38 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:25:38.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:25:38 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:25:38.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:25:38 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:25:38.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:25:38 smithi047 ceph-mon[77147]: pgmap v9286: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:25:38.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:25:38 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:25:38.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:25:38 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:25:38.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:25:38 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:25:39.610 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:25:39 smithi115 conmon[90393]: 2022-05-14T04:25:39.251+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:25:39.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:25:39 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:25:39.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:25:39 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:25:40.480 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:25:40 smithi115 conmon[90393]: 2022-05-14T04:25:40.206+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:25:40.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:25:40 smithi047 ceph-mon[77147]: pgmap v9287: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:25:40.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:25:40 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:25:40.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:25:40 smithi115 ceph-mon[78593]: pgmap v9287: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:25:40.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:25:40 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:25:41.486 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:25:41 smithi115 conmon[90393]: 2022-05-14T04:25:41.183+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:25:41.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:25:41 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:25:41.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:25:41 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:25:42.487 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:25:42 smithi115 conmon[90393]: 2022-05-14T04:25:42.148+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:25:42.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:25:42 smithi047 ceph-mon[77147]: pgmap v9288: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:25:42.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:25:42 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:25:42.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:25:42 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:25:42.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:25:42 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:25:42.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:25:42 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 17875 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:25:42.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:25:42 smithi115 ceph-mon[78593]: pgmap v9288: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:25:42.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:25:42 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:25:42.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:25:42 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:25:42.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:25:42 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:25:42.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:25:42 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 17875 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:25:43.492 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:25:43 smithi115 conmon[90393]: 2022-05-14T04:25:43.141+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:25:43.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:25:43 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:25:43.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:25:43 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:25:44.494 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:25:44 smithi115 conmon[90393]: 2022-05-14T04:25:44.157+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:25:44.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:25:44 smithi047 ceph-mon[77147]: pgmap v9289: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:25:44.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:25:44 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:25:44.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:25:44 smithi115 ceph-mon[78593]: pgmap v9289: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:25:44.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:25:44 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:25:45.501 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:25:45 smithi115 conmon[90393]: 2022-05-14T04:25:45.192+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:25:45.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:25:45 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:25:45.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:25:45 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:25:46.500 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:25:46 smithi115 conmon[90393]: 2022-05-14T04:25:46.196+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:25:46.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:25:46 smithi047 ceph-mon[77147]: pgmap v9290: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:25:46.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:25:46 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:25:46.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:25:46 smithi115 ceph-mon[78593]: pgmap v9290: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:25:46.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:25:46 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:25:47.509 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:25:47 smithi115 conmon[90393]: 2022-05-14T04:25:47.156+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:25:47.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:25:47 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:25:47.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:25:47 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 17880 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:25:47.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:25:47 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:25:47.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:25:47 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 17880 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:25:48.509 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:25:48 smithi115 conmon[90393]: 2022-05-14T04:25:48.164+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:25:48.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:25:48 smithi047 ceph-mon[77147]: pgmap v9291: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:25:48.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:25:48 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:25:48.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:25:48 smithi115 ceph-mon[78593]: pgmap v9291: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:25:48.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:25:48 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:25:49.516 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:25:49 smithi115 conmon[90393]: 2022-05-14T04:25:49.143+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:25:49.795 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:25:49 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:25:49.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:25:49 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:25:50.416 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:25:50 smithi115 conmon[90393]: 2022-05-14T04:25:50.142+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:25:50.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:25:50 smithi047 ceph-mon[77147]: pgmap v9292: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:25:50.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:25:50 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:25:50.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:25:50 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:25:50.795 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:25:50 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:25:50.795 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:25:50 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T04:25:50.795 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:25:50 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T04:25:50.795 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:25:50 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T04:25:50.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:25:50 smithi115 ceph-mon[78593]: pgmap v9292: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:25:50.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:25:50 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:25:50.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:25:50 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:25:50.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:25:50 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:25:50.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:25:50 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T04:25:50.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:25:50 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T04:25:50.798 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:25:50 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T04:25:51.519 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:25:51 smithi115 conmon[90393]: 2022-05-14T04:25:51.182+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:25:51.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:25:51 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:25:51.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:25:51 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:25:52.520 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:25:52 smithi115 conmon[90393]: 2022-05-14T04:25:52.162+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:25:52.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:25:52 smithi047 ceph-mon[77147]: pgmap v9293: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:25:52.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:25:52 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:25:52.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:25:52 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 17885 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:25:52.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:25:52 smithi115 ceph-mon[78593]: pgmap v9293: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:25:52.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:25:52 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:25:52.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:25:52 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 17885 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:25:53.526 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:25:53 smithi115 conmon[90393]: 2022-05-14T04:25:53.138+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:25:53.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:25:53 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:25:53.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:25:53 smithi047 conmon[78521]: 2022-05-14T04:25:53.391+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:25:53.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:25:53 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T04:25:53.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:25:53 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:25:53.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:25:53 smithi047 conmon[78521]: yield 2022-05-14T04:25:53.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:25:53 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:25:53.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:25:53 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:25:53.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:25:53 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:25:53.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:25:53 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T04:25:53.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:25:53 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:25:53.797 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:25:53 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T04:25:53.797 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:25:53 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:25:53.797 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:25:53 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T04:25:53.797 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:25:53 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:25:53.798 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:25:53 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T04:25:53.798 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:25:53 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:25:53.798 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:25:53 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T04:25:53.798 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:25:53 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:25:53.799 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:25:53 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:25:53.799 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:25:53 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:25:53.862 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:25:53 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:25:54.527 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:25:54 smithi115 conmon[90393]: 2022-05-14T04:25:54.178+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:25:54.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:25:54 smithi047 ceph-mon[77147]: pgmap v9294: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:25:54.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:25:54 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:25:54.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T04:25:54.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:25:54.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T04:25:54.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:25:54.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:25:54.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:25:54.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T04:25:54.797 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:25:54.797 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T04:25:54.797 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:25:54.798 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T04:25:54.798 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:25:54.798 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T04:25:54.800 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:25:54.800 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T04:25:54.800 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:25:54.801 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:25:54.801 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:25:54.802 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:25:54 smithi047 ceph-mon[77147]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T04:25:54.802 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:25:54 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:25:54.802 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:25:54 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:25:54.803 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:25:54 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:25:54.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:25:54 smithi115 ceph-mon[78593]: pgmap v9294: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:25:54.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:25:54 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:25:54.860 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T04:25:54.860 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:25:54.860 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T04:25:54.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:25:54.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:25:54.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:25:54.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T04:25:54.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:25:54.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T04:25:54.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:25:54.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T04:25:54.863 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:25:54.863 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T04:25:54.863 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:25:54.863 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T04:25:54.863 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:25:54.864 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:25:54.864 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:25:54.864 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:25:54 smithi115 ceph-mon[78593]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T04:25:54.864 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:25:54 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:25:54.864 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:25:54 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:25:54.865 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:25:54 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:25:55.532 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:25:55 smithi115 conmon[90393]: 2022-05-14T04:25:55.149+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:25:55.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:25:55 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:25:55.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:25:55 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:25:56.534 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:25:56 smithi115 conmon[90393]: 2022-05-14T04:25:56.174+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:25:56.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:25:56 smithi047 ceph-mon[77147]: pgmap v9295: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:25:56.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:25:56 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:25:56.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:25:56 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:25:56.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T04:25:56.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:25:56.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T04:25:56.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:25:56.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:25:56.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:25:56.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T04:25:56.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:25:56.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T04:25:56.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:25:56.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T04:25:56.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:25:56.797 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T04:25:56.797 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:25:56.797 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T04:25:56.797 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:25:56.798 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:25:56.798 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:25:56.798 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:25:56 smithi047 ceph-mon[77147]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T04:25:56.798 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T04:25:56.798 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:25:56.799 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T04:25:56.799 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:25:56.799 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:25:56.799 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:25:56.800 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T04:25:56.800 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:25:56.800 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T04:25:56.800 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:25:56.801 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T04:25:56.801 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:25:56.801 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T04:25:56.801 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:25:56.802 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T04:25:56.802 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:25:56.803 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:25:56.803 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:25:56.803 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T04:25:56.804 INFO:journalctl@ceph.mon.b.smithi047.stdout: During handling of the above exception, another exception occurred: 2022-05-14T04:25:56.804 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T04:25:56.804 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T04:25:56.804 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T04:25:56.805 INFO:journalctl@ceph.mon.b.smithi047.stdout: return f(*arg) 2022-05-14T04:25:56.805 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T04:25:56.805 INFO:journalctl@ceph.mon.b.smithi047.stdout: self._write_client_files(client_files, host) 2022-05-14T04:25:56.805 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T04:25:56.806 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T04:25:56.806 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T04:25:56.806 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T04:25:56.806 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T04:25:56.806 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.event_loop.get_result(coro) 2022-05-14T04:25:56.807 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T04:25:56.807 INFO:journalctl@ceph.mon.b.smithi047.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T04:25:56.807 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T04:25:56.807 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.__get_result() 2022-05-14T04:25:56.807 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T04:25:56.808 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise self._exception 2022-05-14T04:25:56.808 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T04:25:56.808 INFO:journalctl@ceph.mon.b.smithi047.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T04:25:56.808 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T04:25:56.808 INFO:journalctl@ceph.mon.b.smithi047.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T04:25:56.809 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T04:25:56.809 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise 2022-05-14T04:25:56.809 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T04:25:56.809 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.gen.throw(type, value, traceback) 2022-05-14T04:25:56.809 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T04:25:56.810 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OrchestratorError(msg) 2022-05-14T04:25:56.810 INFO:journalctl@ceph.mon.b.smithi047.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:25:56.811 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:25:56 smithi047 conmon[78521]: 2022-05-14T04:25:56.463+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:25:56.811 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:25:56 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T04:25:56.811 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:25:56 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:25:56.811 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:25:56 smithi047 conmon[78521]: yield 2022-05-14T04:25:56.811 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:25:56 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:25:56.812 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:25:56 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:25:56.812 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:25:56 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:25:56.812 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:25:56 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T04:25:56.813 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:25:56 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:25:56.813 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:25:56 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T04:25:56.813 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:25:56 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:25:56.813 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:25:56 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T04:25:56.814 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:25:56 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:25:56.814 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:25:56 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T04:25:56.814 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:25:56 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:25:56.814 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:25:56 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T04:25:56.814 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:25:56 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:25:56.815 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:25:56 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:25:56.815 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:25:56 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:25:56.815 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:25:56 smithi047 conmon[78521]: 2022-05-14T04:25:56.464+0000 7f5344f82700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T04:25:56.815 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:25:56 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T04:25:56.816 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:25:56 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:25:56.816 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:25:56 smithi047 conmon[78521]: yield 2022-05-14T04:25:56.816 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:25:56 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:25:56.816 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:25:56 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:25:56.816 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:25:56 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:25:56.817 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:25:56 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T04:25:56.817 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:25:56 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:25:56.817 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:25:56 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T04:25:56.817 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:25:56 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:25:56.817 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:25:56 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T04:25:56.818 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:25:56 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:25:56.818 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:25:56 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T04:25:56.818 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:25:56 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:25:56.818 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:25:56 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T04:25:56.819 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:25:56 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:25:56.819 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:25:56 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:25:56.819 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:25:56 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:25:56.819 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:25:56 smithi047 conmon[78521]: 2022-05-14T04:25:56.819 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:25:56 smithi047 conmon[78521]: During handling of the above exception, another exception occurred: 2022-05-14T04:25:56.820 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:25:56 smithi047 conmon[78521]: 2022-05-14T04:25:56.820 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:25:56 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T04:25:56.820 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:25:56 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T04:25:56.820 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:25:56 smithi047 conmon[78521]: return f(*arg) 2022-05-14T04:25:56.820 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:25:56 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T04:25:56.821 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:25:56 smithi047 conmon[78521]: self._write_client_files(client_files, host) 2022-05-14T04:25:56.821 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:25:56 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T04:25:56.821 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:25:56 smithi047 conmon[78521]: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T04:25:56.821 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:25:56 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T04:25:56.821 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:25:56 smithi047 conmon[78521]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T04:25:56.822 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:25:56 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T04:25:56.822 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:25:56 smithi047 conmon[78521]: return self.event_loop.get_result(coro) 2022-05-14T04:25:56.822 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:25:56 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T04:25:56.822 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:25:56 smithi047 conmon[78521]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T04:25:56.823 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:25:56 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T04:25:56.823 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:25:56 smithi047 conmon[78521]: return self.__get_result() 2022-05-14T04:25:56.823 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:25:56 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T04:25:56.824 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:25:56 smithi047 conmon[78521]: raise self._exception 2022-05-14T04:25:56.824 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:25:56 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T04:25:56.824 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:25:56 smithi047 conmon[78521]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T04:25:56.824 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:25:56 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T04:25:56.825 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:25:56 smithi047 conmon[78521]: conn = await self._remote_connection(host, addr) 2022-05-14T04:25:56.825 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:25:56 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T04:25:56.825 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:25:56 smithi047 conmon[78521]: raise 2022-05-14T04:25:56.825 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:25:56 smithi047 conmon[78521]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T04:25:56.825 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:25:56 smithi047 conmon[78521]: self.gen.throw(type, value, traceback) 2022-05-14T04:25:56.826 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:25:56 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T04:25:56.826 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:25:56 smithi047 conmon[78521]: raise OrchestratorError(msg) 2022-05-14T04:25:56.826 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:25:56 smithi047 conmon[78521]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:25:56.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:25:56 smithi115 ceph-mon[78593]: pgmap v9295: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:25:56.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:25:56 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:25:56.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:25:56 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:25:56.860 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T04:25:56.860 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:25:56.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T04:25:56.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:25:56.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:25:56.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:25:56.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T04:25:56.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:25:56.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T04:25:56.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:25:56.863 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T04:25:56.863 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:25:56.863 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T04:25:56.863 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:25:56.863 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T04:25:56.864 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:25:56.864 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:25:56.864 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:25:56.864 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:25:56 smithi115 ceph-mon[78593]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T04:25:56.865 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T04:25:56.865 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:25:56.865 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T04:25:56.865 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:25:56.866 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:25:56.866 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:25:56.866 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T04:25:56.866 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:25:56.866 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T04:25:56.867 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:25:56.867 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T04:25:56.867 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:25:56.867 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T04:25:56.868 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:25:56.868 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T04:25:56.868 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:25:56.868 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:25:56.868 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:25:56.869 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T04:25:56.869 INFO:journalctl@ceph.mon.c.smithi115.stdout: During handling of the above exception, another exception occurred: 2022-05-14T04:25:56.869 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T04:25:56.869 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T04:25:56.869 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T04:25:56.870 INFO:journalctl@ceph.mon.c.smithi115.stdout: return f(*arg) 2022-05-14T04:25:56.870 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T04:25:56.870 INFO:journalctl@ceph.mon.c.smithi115.stdout: self._write_client_files(client_files, host) 2022-05-14T04:25:56.870 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T04:25:56.871 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T04:25:56.871 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T04:25:56.871 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T04:25:56.871 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T04:25:56.871 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.event_loop.get_result(coro) 2022-05-14T04:25:56.871 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T04:25:56.872 INFO:journalctl@ceph.mon.c.smithi115.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T04:25:56.872 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T04:25:56.872 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.__get_result() 2022-05-14T04:25:56.872 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T04:25:56.873 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise self._exception 2022-05-14T04:25:56.873 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T04:25:56.873 INFO:journalctl@ceph.mon.c.smithi115.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T04:25:56.873 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T04:25:56.873 INFO:journalctl@ceph.mon.c.smithi115.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T04:25:56.874 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T04:25:56.874 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise 2022-05-14T04:25:56.874 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T04:25:56.874 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.gen.throw(type, value, traceback) 2022-05-14T04:25:56.875 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T04:25:56.875 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OrchestratorError(msg) 2022-05-14T04:25:56.875 INFO:journalctl@ceph.mon.c.smithi115.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:25:57.538 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:25:57 smithi115 conmon[90393]: 2022-05-14T04:25:57.192+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:25:57.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:25:57 smithi047 ceph-mon[77147]: pgmap v9296: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:25:57.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:25:57 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:25:57.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:25:57 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 17890 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:25:57.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:25:57 smithi115 ceph-mon[78593]: pgmap v9296: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:25:57.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:25:57 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:25:57.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:25:57 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 17890 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:25:58.539 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:25:58 smithi115 conmon[90393]: 2022-05-14T04:25:58.167+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:25:58.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:25:58 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:25:58.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:25:58 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:25:58.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:25:58 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:25:58.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:25:58 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:25:58.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:25:58 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:25:58.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:25:58 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:25:59.546 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:25:59 smithi115 conmon[90393]: 2022-05-14T04:25:59.187+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:25:59.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:25:59 smithi115 ceph-mon[78593]: pgmap v9297: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:25:59.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:25:59 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:26:00.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:25:59 smithi047 ceph-mon[77147]: pgmap v9297: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:26:00.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:25:59 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:26:00.546 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:26:00 smithi115 conmon[90393]: 2022-05-14T04:26:00.227+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:26:00.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:26:00 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:26:01.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:26:00 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:26:01.551 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:26:01 smithi115 conmon[90393]: 2022-05-14T04:26:01.201+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:26:01.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:26:01 smithi115 ceph-mon[78593]: pgmap v9298: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:26:01.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:26:01 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:26:02.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:26:01 smithi047 ceph-mon[77147]: pgmap v9298: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:26:02.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:26:01 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:26:02.552 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:26:02 smithi115 conmon[90393]: 2022-05-14T04:26:02.195+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:26:02.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:26:02 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:26:02.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:26:02 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:26:02.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:26:02 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:26:02.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:26:02 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 17895 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:26:03.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:26:02 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:26:03.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:26:02 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:26:03.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:26:02 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:26:03.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:26:02 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 17895 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:26:03.559 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:26:03 smithi115 conmon[90393]: 2022-05-14T04:26:03.224+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:26:03.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:26:03 smithi115 ceph-mon[78593]: pgmap v9299: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:26:03.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:26:03 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:26:04.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:26:03 smithi047 ceph-mon[77147]: pgmap v9299: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:26:04.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:26:03 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:26:04.559 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:26:04 smithi115 conmon[90393]: 2022-05-14T04:26:04.184+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:26:04.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:26:04 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:26:05.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:26:04 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:26:05.564 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:26:05 smithi115 conmon[90393]: 2022-05-14T04:26:05.180+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:26:05.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:26:05 smithi115 ceph-mon[78593]: pgmap v9300: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:26:05.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:26:05 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:26:06.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:26:05 smithi047 ceph-mon[77147]: pgmap v9300: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:26:06.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:26:05 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:26:06.565 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:26:06 smithi115 conmon[90393]: 2022-05-14T04:26:06.212+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:26:06.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:26:06 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:26:07.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:26:06 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:26:07.571 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:26:07 smithi115 conmon[90393]: 2022-05-14T04:26:07.242+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:26:07.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:26:07 smithi115 ceph-mon[78593]: pgmap v9301: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:26:07.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:26:07 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:26:07.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:26:07 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 17900 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:26:08.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:26:07 smithi047 ceph-mon[77147]: pgmap v9301: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:26:08.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:26:07 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:26:08.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:26:07 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 17900 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:26:08.571 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:26:08 smithi115 conmon[90393]: 2022-05-14T04:26:08.256+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:26:08.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:26:08 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:26:09.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:26:08 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:26:09.577 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:26:09 smithi115 conmon[90393]: 2022-05-14T04:26:09.225+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:26:09.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:26:09 smithi115 ceph-mon[78593]: pgmap v9302: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:26:09.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:26:09 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:26:10.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:26:09 smithi047 ceph-mon[77147]: pgmap v9302: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:26:10.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:26:09 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:26:10.578 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:26:10 smithi115 conmon[90393]: 2022-05-14T04:26:10.273+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:26:10.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:26:10 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:26:10.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:26:10 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:26:10.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:26:10 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:26:11.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:26:10 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:26:11.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:26:10 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:26:11.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:26:10 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:26:11.583 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:26:11 smithi115 conmon[90393]: 2022-05-14T04:26:11.311+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:26:11.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:26:11 smithi115 ceph-mon[78593]: pgmap v9303: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:26:11.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:26:11 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:26:12.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:26:11 smithi047 ceph-mon[77147]: pgmap v9303: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:26:12.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:26:11 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:26:12.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:26:12 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:26:12.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:26:12 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 17905 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:26:12.610 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:26:12 smithi115 conmon[90393]: 2022-05-14T04:26:12.341+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:26:13.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:26:12 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:26:13.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:26:12 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 17905 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:26:13.614 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:26:13 smithi115 ceph-mon[78593]: pgmap v9304: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:26:13.614 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:26:13 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:26:13.614 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:26:13 smithi115 conmon[90393]: 2022-05-14T04:26:13.341+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:26:14.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:26:13 smithi047 ceph-mon[77147]: pgmap v9304: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:26:14.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:26:13 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:26:14.591 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:26:14 smithi115 conmon[90393]: 2022-05-14T04:26:14.321+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:26:14.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:26:14 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:26:14.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:26:14 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:26:14.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:26:14 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:26:15.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:26:14 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:26:15.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:26:14 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:26:15.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:26:14 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:26:15.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:26:15 smithi115 conmon[90393]: 2022-05-14T04:26:15.361+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:26:15.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:26:15 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T04:26:15.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:26:15 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T04:26:15.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:26:15 smithi115 ceph-mon[78593]: pgmap v9305: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:26:15.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:26:15 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:26:16.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:26:15 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T04:26:16.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:26:15 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T04:26:16.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:26:15 smithi047 ceph-mon[77147]: pgmap v9305: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:26:16.045 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:26:15 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:26:16.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:26:16 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:26:16.860 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:26:16 smithi115 conmon[90393]: 2022-05-14T04:26:16.372+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:26:17.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:26:16 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:26:17.606 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:26:17 smithi115 conmon[90393]: 2022-05-14T04:26:17.324+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:26:17.863 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:26:17 smithi115 ceph-mon[78593]: pgmap v9306: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:26:17.864 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:26:17 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:26:17.864 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:26:17 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 17910 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:26:18.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:26:17 smithi047 ceph-mon[77147]: pgmap v9306: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:26:18.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:26:17 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:26:18.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:26:17 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 17910 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:26:18.606 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:26:18 smithi115 conmon[90393]: 2022-05-14T04:26:18.327+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:26:18.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:26:18 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:26:18.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:26:18 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:26:18.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:26:18 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:26:19.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:26:18 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:26:19.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:26:18 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:26:19.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:26:18 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:26:19.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:26:19 smithi115 conmon[90393]: 2022-05-14T04:26:19.343+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:26:20.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:26:19 smithi047 ceph-mon[77147]: pgmap v9307: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:26:20.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:26:19 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:26:20.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:26:19 smithi115 ceph-mon[78593]: pgmap v9307: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:26:20.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:26:19 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:26:20.613 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:26:20 smithi115 conmon[90393]: 2022-05-14T04:26:20.356+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:26:21.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:26:20 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:26:21.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:26:20 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:26:21.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:26:21 smithi115 conmon[90393]: 2022-05-14T04:26:21.347+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:26:22.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:26:21 smithi047 ceph-mon[77147]: pgmap v9308: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:26:22.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:26:21 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:26:22.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:26:21 smithi115 ceph-mon[78593]: pgmap v9308: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:26:22.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:26:21 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:26:22.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:26:22 smithi115 conmon[90393]: 2022-05-14T04:26:22.386+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:26:22.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:26:22 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:26:22.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:26:22 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:26:22.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:26:22 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:26:22.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:26:22 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 17915 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:26:23.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:26:22 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:26:23.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:26:22 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:26:23.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:26:22 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:26:23.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:26:22 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 17915 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:26:23.624 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:26:23 smithi115 conmon[90393]: 2022-05-14T04:26:23.357+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:26:24.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:26:23 smithi047 ceph-mon[77147]: pgmap v9309: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:26:24.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:26:23 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:26:24.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:26:23 smithi115 ceph-mon[78593]: pgmap v9309: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:26:24.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:26:23 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:26:24.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:26:24 smithi115 conmon[90393]: 2022-05-14T04:26:24.354+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:26:25.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:26:24 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:26:25.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:26:24 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:26:25.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:26:25 smithi115 conmon[90393]: 2022-05-14T04:26:25.350+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:26:26.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:26:25 smithi047 ceph-mon[77147]: pgmap v9310: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:26:26.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:26:25 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:26:26.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:26:25 smithi115 ceph-mon[78593]: pgmap v9310: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:26:26.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:26:25 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:26:26.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:26:26 smithi115 conmon[90393]: 2022-05-14T04:26:26.339+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:26:27.054 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:26:26 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:26:27.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:26:26 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:26:27.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:26:27 smithi115 conmon[90393]: 2022-05-14T04:26:27.352+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:26:28.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:26:27 smithi047 ceph-mon[77147]: pgmap v9311: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:26:28.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:26:27 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:26:28.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:26:27 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 17920 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:26:28.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:26:27 smithi115 ceph-mon[78593]: pgmap v9311: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:26:28.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:26:27 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:26:28.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:26:27 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 17920 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:26:28.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:26:28 smithi115 conmon[90393]: 2022-05-14T04:26:28.328+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:26:29.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:26:28 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:26:29.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:26:28 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:26:29.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:26:29 smithi115 conmon[90393]: 2022-05-14T04:26:29.323+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:26:30.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:26:29 smithi047 ceph-mon[77147]: pgmap v9312: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:26:30.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:26:29 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:26:30.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:26:29 smithi115 ceph-mon[78593]: pgmap v9312: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:26:30.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:26:29 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:26:30.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:26:30 smithi115 conmon[90393]: 2022-05-14T04:26:30.333+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:26:31.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:26:30 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:26:31.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:26:30 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:26:31.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:26:30 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:26:31.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:26:30 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:26:31.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:26:30 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:26:31.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:26:30 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:26:31.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:26:31 smithi115 conmon[90393]: 2022-05-14T04:26:31.354+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:26:32.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:26:31 smithi047 ceph-mon[77147]: pgmap v9313: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:26:32.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:26:31 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:26:32.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:26:31 smithi115 ceph-mon[78593]: pgmap v9313: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:26:32.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:26:31 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:26:32.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:26:32 smithi115 conmon[90393]: 2022-05-14T04:26:32.347+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:26:33.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:26:32 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:26:33.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:26:32 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 17925 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:26:33.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:26:32 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:26:33.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:26:32 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 17925 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:26:33.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:26:33 smithi115 conmon[90393]: 2022-05-14T04:26:33.336+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:26:34.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:26:33 smithi047 ceph-mon[77147]: pgmap v9314: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:26:34.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:26:33 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:26:34.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:26:33 smithi115 ceph-mon[78593]: pgmap v9314: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:26:34.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:26:33 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:26:34.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:26:34 smithi115 conmon[90393]: 2022-05-14T04:26:34.308+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:26:35.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:26:34 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:26:35.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:26:34 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:26:35.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:26:34 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:26:35.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:26:34 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:26:35.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:26:34 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:26:35.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:26:34 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:26:35.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:26:35 smithi115 conmon[90393]: 2022-05-14T04:26:35.302+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:26:36.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:26:35 smithi047 ceph-mon[77147]: pgmap v9315: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:26:36.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:26:35 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:26:36.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:26:35 smithi115 ceph-mon[78593]: pgmap v9315: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:26:36.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:26:35 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:26:36.610 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:26:36 smithi115 conmon[90393]: 2022-05-14T04:26:36.319+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:26:37.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:26:36 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:26:37.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:26:36 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:26:37.675 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:26:37 smithi115 conmon[90393]: 2022-05-14T04:26:37.367+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:26:38.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:26:37 smithi047 ceph-mon[77147]: pgmap v9316: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:26:38.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:26:37 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:26:38.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:26:37 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 17930 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:26:38.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:26:37 smithi115 ceph-mon[78593]: pgmap v9316: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:26:38.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:26:37 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:26:38.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:26:37 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 17930 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:26:38.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:26:38 smithi115 conmon[90393]: 2022-05-14T04:26:38.336+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:26:39.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:26:38 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:26:39.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:26:38 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:26:39.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:26:38 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:26:39.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:26:38 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:26:39.158 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:26:38 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:26:39.158 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:26:38 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:26:39.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:26:39 smithi115 conmon[90393]: 2022-05-14T04:26:39.345+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:26:40.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:26:39 smithi047 ceph-mon[77147]: pgmap v9317: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:26:40.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:26:39 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:26:40.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:26:39 smithi115 ceph-mon[78593]: pgmap v9317: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:26:40.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:26:39 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:26:40.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:26:40 smithi115 conmon[90393]: 2022-05-14T04:26:40.325+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:26:41.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:26:40 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:26:41.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:26:40 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:26:41.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:26:41 smithi115 conmon[90393]: 2022-05-14T04:26:41.301+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:26:42.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:26:41 smithi047 ceph-mon[77147]: pgmap v9318: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:26:42.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:26:41 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:26:42.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:26:41 smithi115 ceph-mon[78593]: pgmap v9318: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:26:42.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:26:41 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:26:42.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:26:42 smithi115 conmon[90393]: 2022-05-14T04:26:42.261+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:26:43.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:26:42 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:26:43.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:26:42 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:26:43.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:26:42 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:26:43.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:26:42 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 17935 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:26:43.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:26:42 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:26:43.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:26:42 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:26:43.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:26:42 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:26:43.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:26:42 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 17935 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:26:43.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:26:43 smithi115 conmon[90393]: 2022-05-14T04:26:43.270+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:26:44.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:26:43 smithi047 ceph-mon[77147]: pgmap v9319: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:26:44.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:26:43 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:26:44.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:26:43 smithi115 ceph-mon[78593]: pgmap v9319: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:26:44.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:26:43 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:26:44.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:26:44 smithi115 conmon[90393]: 2022-05-14T04:26:44.260+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:26:45.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:26:44 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:26:45.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:26:44 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:26:45.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:26:45 smithi115 conmon[90393]: 2022-05-14T04:26:45.309+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:26:46.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:26:45 smithi047 ceph-mon[77147]: pgmap v9320: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:26:46.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:26:45 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:26:46.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:26:45 smithi115 ceph-mon[78593]: pgmap v9320: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:26:46.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:26:45 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:26:46.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:26:46 smithi115 conmon[90393]: 2022-05-14T04:26:46.291+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:26:47.050 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:26:46 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:26:47.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:26:46 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:26:47.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:26:47 smithi115 conmon[90393]: 2022-05-14T04:26:47.281+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:26:48.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:26:47 smithi047 ceph-mon[77147]: pgmap v9321: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:26:48.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:26:47 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:26:48.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:26:47 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 17940 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:26:48.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:26:47 smithi115 ceph-mon[78593]: pgmap v9321: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:26:48.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:26:47 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:26:48.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:26:47 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 17940 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:26:48.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:26:48 smithi115 conmon[90393]: 2022-05-14T04:26:48.308+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:26:49.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:26:48 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:26:49.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:26:48 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:26:49.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:26:49 smithi115 conmon[90393]: 2022-05-14T04:26:49.321+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:26:50.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:26:49 smithi047 ceph-mon[77147]: pgmap v9322: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:26:50.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:26:49 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:26:50.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:26:49 smithi115 ceph-mon[78593]: pgmap v9322: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:26:50.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:26:49 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:26:50.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:26:50 smithi115 conmon[90393]: 2022-05-14T04:26:50.323+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:26:51.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:26:50 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:26:51.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:26:50 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:26:51.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:26:50 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:26:51.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:26:50 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:26:51.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:26:50 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:26:51.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:26:50 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:26:51.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:26:51 smithi115 conmon[90393]: 2022-05-14T04:26:51.349+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:26:52.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:26:51 smithi047 ceph-mon[77147]: pgmap v9323: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:26:52.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:26:51 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:26:52.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:26:51 smithi115 ceph-mon[78593]: pgmap v9323: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:26:52.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:26:51 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:26:52.721 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:26:52 smithi115 conmon[90393]: 2022-05-14T04:26:52.384+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:26:53.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:26:52 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:26:53.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:26:52 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 17945 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:26:53.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:26:52 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:26:53.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:26:52 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 17945 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:26:53.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:26:53 smithi115 conmon[90393]: 2022-05-14T04:26:53.386+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:26:54.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:26:54 smithi115 conmon[90393]: 2022-05-14T04:26:54.339+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:26:54.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:26:54 smithi115 ceph-mon[78593]: pgmap v9324: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:26:54.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:26:54 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:26:54.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:26:54 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:26:54.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:26:54 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:26:54.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:26:54 smithi047 ceph-mon[77147]: pgmap v9324: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:26:54.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:26:54 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:26:54.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:26:54 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:26:54.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:26:54 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:26:55.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:26:55 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:26:55.610 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:26:55 smithi115 conmon[90393]: 2022-05-14T04:26:55.343+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:26:55.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:26:55 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:26:56.566 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:26:56 smithi115 conmon[90393]: 2022-05-14T04:26:56.295+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:26:56.566 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:26:56 smithi115 ceph-mon[78593]: pgmap v9325: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:26:56.567 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:26:56 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:26:56.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:26:56 smithi047 ceph-mon[77147]: pgmap v9325: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:26:56.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:26:56 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:26:57.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:26:57 smithi115 conmon[90393]: 2022-05-14T04:26:57.248+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:26:57.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:26:57 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:26:57.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:26:57 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T04:26:57.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:26:57 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T04:26:57.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:26:57 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T04:26:57.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:26:57 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 17950 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:26:57.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:26:57 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:26:57.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:26:57 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T04:26:57.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:26:57 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T04:26:57.795 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:26:57 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T04:26:57.795 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:26:57 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 17950 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:26:58.610 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:26:58 smithi115 conmon[90393]: 2022-05-14T04:26:58.292+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:26:58.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:26:58 smithi115 ceph-mon[78593]: pgmap v9326: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:26:58.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:26:58 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:26:58.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:26:58 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:26:58.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:26:58 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:26:58.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:26:58 smithi047 ceph-mon[77147]: pgmap v9326: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:26:58.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:26:58 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:26:58.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:26:58 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:26:58.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:26:58 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:26:59.610 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:26:59 smithi115 conmon[90393]: 2022-05-14T04:26:59.315+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:26:59.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:26:59 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:26:59.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:26:59 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:26:59.793 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:26:59 smithi047 conmon[78521]: 2022-05-14T04:26:59.568+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:26:59.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:26:59 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T04:26:59.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:26:59 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:26:59.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:26:59 smithi047 conmon[78521]: yield 2022-05-14T04:26:59.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:26:59 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:26:59.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:26:59 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:26:59.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:26:59 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:26:59.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:26:59 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T04:26:59.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:26:59 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:26:59.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:26:59 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T04:26:59.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:26:59 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:26:59.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:26:59 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T04:26:59.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:26:59 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:26:59.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:26:59 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T04:26:59.797 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:26:59 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:26:59.797 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:26:59 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T04:26:59.797 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:26:59 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:26:59.797 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:26:59 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:26:59.797 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:26:59 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:27:00.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:27:00 smithi047 ceph-mon[77147]: pgmap v9327: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:27:00.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:27:00 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:27:00.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:27:00 smithi115 conmon[90393]: 2022-05-14T04:27:00.358+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:27:00.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:27:00 smithi115 ceph-mon[78593]: pgmap v9327: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:27:00.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:27:00 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:27:01.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:27:01 smithi115 conmon[90393]: 2022-05-14T04:27:01.314+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:27:01.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:27:01 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:27:01.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T04:27:01.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:27:01.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T04:27:01.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:27:01.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:27:01.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:27:01.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T04:27:01.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:27:01.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T04:27:01.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:27:01.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T04:27:01.613 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:27:01.613 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T04:27:01.613 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:27:01.613 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T04:27:01.613 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:27:01.614 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:27:01.614 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:27:01.614 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:27:01 smithi115 ceph-mon[78593]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T04:27:01.614 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:27:01 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:27:01.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:27:01 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:27:01.793 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T04:27:01.793 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:27:01.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T04:27:01.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:27:01.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:27:01.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:27:01.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T04:27:01.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:27:01.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T04:27:01.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:27:01.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T04:27:01.797 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:27:01.797 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T04:27:01.797 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:27:01.797 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T04:27:01.798 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:27:01.798 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:27:01.798 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:27:01.798 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:27:01 smithi047 ceph-mon[77147]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T04:27:01.798 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:27:01 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:27:02.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:27:02 smithi115 conmon[90393]: 2022-05-14T04:27:02.341+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:27:02.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:27:02 smithi115 ceph-mon[78593]: pgmap v9328: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:27:02.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:27:02 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:27:02.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:27:02 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:27:02.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:27:02 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:27:02.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:27:02 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 17955 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:27:02.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:27:02 smithi047 ceph-mon[77147]: pgmap v9328: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:27:02.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:27:02 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:27:02.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:27:02 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:27:02.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:27:02 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:27:02.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:27:02 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 17955 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:27:02.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:27:02 smithi047 conmon[78521]: 2022-05-14T04:27:02.640+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:27:02.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:27:02 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T04:27:02.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:27:02 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:27:02.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:27:02 smithi047 conmon[78521]: yield 2022-05-14T04:27:02.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:27:02 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:27:02.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:27:02 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:27:02.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:27:02 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:27:02.797 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:27:02 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T04:27:02.797 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:27:02 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:27:02.797 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:27:02 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T04:27:02.797 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:27:02 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:27:02.798 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:27:02 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T04:27:02.798 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:27:02 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:27:02.798 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:27:02 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T04:27:02.798 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:27:02 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:27:02.799 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:27:02 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T04:27:02.799 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:27:02 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:27:02.799 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:27:02 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:27:02.799 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:27:02 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:27:02.799 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:27:02 smithi047 conmon[78521]: 2022-05-14T04:27:02.641+0000 7f533ef76700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T04:27:02.800 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:27:02 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T04:27:02.800 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:27:02 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:27:02.800 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:27:02 smithi047 conmon[78521]: yield 2022-05-14T04:27:02.800 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:27:02 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:27:02.801 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:27:02 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:27:02.801 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:27:02 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:27:02.801 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:27:02 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T04:27:02.801 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:27:02 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:27:02.802 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:27:02 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T04:27:02.802 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:27:02 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:27:02.802 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:27:02 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T04:27:02.802 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:27:02 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:27:02.802 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:27:02 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T04:27:02.803 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:27:02 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:27:02.803 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:27:02 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T04:27:02.803 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:27:02 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:27:02.803 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:27:02 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:27:02.804 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:27:02 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:27:02.804 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:27:02 smithi047 conmon[78521]: 2022-05-14T04:27:02.804 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:27:02 smithi047 conmon[78521]: During handling of the above exception, another exception occurred: 2022-05-14T04:27:02.804 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:27:02 smithi047 conmon[78521]: 2022-05-14T04:27:02.804 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:27:02 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T04:27:02.805 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:27:02 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T04:27:02.805 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:27:02 smithi047 conmon[78521]: return f(*arg) 2022-05-14T04:27:02.805 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:27:02 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T04:27:02.805 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:27:02 smithi047 conmon[78521]: self._write_client_files(client_files, host) 2022-05-14T04:27:02.805 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:27:02 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T04:27:02.806 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:27:02 smithi047 conmon[78521]: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T04:27:02.806 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:27:02 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T04:27:02.806 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:27:02 smithi047 conmon[78521]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T04:27:02.806 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:27:02 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T04:27:02.806 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:27:02 smithi047 conmon[78521]: return self.event_loop.get_result(coro) 2022-05-14T04:27:02.807 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:27:02 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T04:27:02.807 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:27:02 smithi047 conmon[78521]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T04:27:02.807 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:27:02 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T04:27:02.807 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:27:02 smithi047 conmon[78521]: return self.__get_result() 2022-05-14T04:27:02.807 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:27:02 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T04:27:02.808 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:27:02 smithi047 conmon[78521]: raise self._exception 2022-05-14T04:27:02.808 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:27:02 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T04:27:02.808 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:27:02 smithi047 conmon[78521]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T04:27:02.808 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:27:02 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T04:27:02.808 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:27:02 smithi047 conmon[78521]: conn = await self._remote_connection(host, addr) 2022-05-14T04:27:02.809 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:27:02 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T04:27:02.809 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:27:02 smithi047 conmon[78521]: raise 2022-05-14T04:27:02.809 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:27:02 smithi047 conmon[78521]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T04:27:02.809 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:27:02 smithi047 conmon[78521]: self.gen.throw(type, value, traceback) 2022-05-14T04:27:02.810 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:27:02 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T04:27:02.810 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:27:02 smithi047 conmon[78521]: raise OrchestratorError(msg) 2022-05-14T04:27:02.810 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:27:02 smithi047 conmon[78521]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:27:03.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:27:03 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:27:03.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:27:03 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:27:03.860 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:27:03 smithi115 conmon[90393]: 2022-05-14T04:27:03.384+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:27:04.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:27:04 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:27:04.825 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T04:27:04.826 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:27:04.826 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T04:27:04.826 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:27:04.827 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:27:04.827 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:27:04.827 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T04:27:04.827 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:27:04.828 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T04:27:04.828 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:27:04.828 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T04:27:04.828 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:27:04.829 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T04:27:04.829 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:27:04.829 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T04:27:04.829 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:27:04.830 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:27:04.830 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:27:04.830 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:27:04 smithi047 ceph-mon[77147]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T04:27:04.830 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T04:27:04.830 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:27:04.831 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T04:27:04.831 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:27:04.831 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:27:04.831 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:27:04.832 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T04:27:04.832 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:27:04.832 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T04:27:04.832 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:27:04.833 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T04:27:04.833 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:27:04.833 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T04:27:04.833 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:27:04.833 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T04:27:04.834 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:27:04.834 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:27:04.834 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:27:04.834 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T04:27:04.835 INFO:journalctl@ceph.mon.b.smithi047.stdout: During handling of the above exception, another exception occurred: 2022-05-14T04:27:04.835 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T04:27:04.835 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T04:27:04.835 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T04:27:04.836 INFO:journalctl@ceph.mon.b.smithi047.stdout: return f(*arg) 2022-05-14T04:27:04.836 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T04:27:04.836 INFO:journalctl@ceph.mon.b.smithi047.stdout: self._write_client_files(client_files, host) 2022-05-14T04:27:04.836 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T04:27:04.836 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T04:27:04.837 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T04:27:04.837 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T04:27:04.837 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T04:27:04.837 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.event_loop.get_result(coro) 2022-05-14T04:27:04.838 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T04:27:04.838 INFO:journalctl@ceph.mon.b.smithi047.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T04:27:04.838 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T04:27:04.838 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.__get_result() 2022-05-14T04:27:04.838 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T04:27:04.839 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise self._exception 2022-05-14T04:27:04.839 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T04:27:04.839 INFO:journalctl@ceph.mon.b.smithi047.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T04:27:04.839 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T04:27:04.839 INFO:journalctl@ceph.mon.b.smithi047.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T04:27:04.840 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T04:27:04.840 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise 2022-05-14T04:27:04.840 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T04:27:04.840 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.gen.throw(type, value, traceback) 2022-05-14T04:27:04.841 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T04:27:04.841 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OrchestratorError(msg) 2022-05-14T04:27:04.841 INFO:journalctl@ceph.mon.b.smithi047.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:27:04.841 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:27:04 smithi047 ceph-mon[77147]: pgmap v9329: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:27:04.842 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:27:04 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:27:04.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:27:04 smithi115 conmon[90393]: 2022-05-14T04:27:04.426+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:27:04.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:27:04 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:27:04.860 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T04:27:04.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:27:04.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T04:27:04.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:27:04.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:27:04.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:27:04.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T04:27:04.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:27:04.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T04:27:04.863 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:27:04.863 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T04:27:04.863 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:27:04.863 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T04:27:04.864 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:27:04.864 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T04:27:04.864 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:27:04.865 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:27:04.865 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:27:04.865 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:27:04 smithi115 ceph-mon[78593]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T04:27:04.866 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T04:27:04.866 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:27:04.867 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T04:27:04.867 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:27:04.867 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:27:04.867 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:27:04.868 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T04:27:04.868 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:27:04.868 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T04:27:04.868 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:27:04.868 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T04:27:04.869 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:27:04.869 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T04:27:04.869 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:27:04.869 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T04:27:04.870 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:27:04.870 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:27:04.870 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:27:04.870 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T04:27:04.870 INFO:journalctl@ceph.mon.c.smithi115.stdout: During handling of the above exception, another exception occurred: 2022-05-14T04:27:04.871 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T04:27:04.871 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T04:27:04.871 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T04:27:04.871 INFO:journalctl@ceph.mon.c.smithi115.stdout: return f(*arg) 2022-05-14T04:27:04.872 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T04:27:04.872 INFO:journalctl@ceph.mon.c.smithi115.stdout: self._write_client_files(client_files, host) 2022-05-14T04:27:04.872 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T04:27:04.872 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T04:27:04.872 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T04:27:04.873 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T04:27:04.873 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T04:27:04.873 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.event_loop.get_result(coro) 2022-05-14T04:27:04.873 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T04:27:04.874 INFO:journalctl@ceph.mon.c.smithi115.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T04:27:04.874 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T04:27:04.874 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.__get_result() 2022-05-14T04:27:04.874 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T04:27:04.875 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise self._exception 2022-05-14T04:27:04.875 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T04:27:04.875 INFO:journalctl@ceph.mon.c.smithi115.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T04:27:04.875 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T04:27:04.875 INFO:journalctl@ceph.mon.c.smithi115.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T04:27:04.876 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T04:27:04.876 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise 2022-05-14T04:27:04.876 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T04:27:04.876 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.gen.throw(type, value, traceback) 2022-05-14T04:27:04.876 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T04:27:04.877 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OrchestratorError(msg) 2022-05-14T04:27:04.877 INFO:journalctl@ceph.mon.c.smithi115.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:27:04.877 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:27:04 smithi115 ceph-mon[78593]: pgmap v9329: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:27:04.877 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:27:04 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:27:05.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:27:05 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:27:05.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:27:05 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:27:05.860 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:27:05 smithi115 conmon[90393]: 2022-05-14T04:27:05.413+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:27:06.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:27:06 smithi047 ceph-mon[77147]: pgmap v9330: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:27:06.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:27:06 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:27:06.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:27:06 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:27:06.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:27:06 smithi115 conmon[90393]: 2022-05-14T04:27:06.428+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:27:06.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:27:06 smithi115 ceph-mon[78593]: pgmap v9330: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:27:06.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:27:06 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:27:06.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:27:06 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:27:07.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:27:07 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 17960 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:27:07.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:27:07 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 17960 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:27:07.860 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:27:07 smithi115 conmon[90393]: 2022-05-14T04:27:07.463+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:27:08.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:27:08 smithi047 ceph-mon[77147]: pgmap v9331: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:27:08.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:27:08 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:27:08.860 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:27:08 smithi115 conmon[90393]: 2022-05-14T04:27:08.466+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:27:08.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:27:08 smithi115 ceph-mon[78593]: pgmap v9331: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:27:08.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:27:08 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:27:09.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:27:09 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:27:09.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:27:09 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:27:09.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:27:09 smithi115 conmon[90393]: 2022-05-14T04:27:09.447+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:27:09.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:27:09 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:27:09.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:27:09 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:27:10.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:27:10 smithi047 ceph-mon[77147]: pgmap v9332: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:27:10.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:27:10 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:27:10.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:27:10 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:27:10.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:27:10 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:27:10.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:27:10 smithi115 conmon[90393]: 2022-05-14T04:27:10.399+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:27:10.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:27:10 smithi115 ceph-mon[78593]: pgmap v9332: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:27:10.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:27:10 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:27:10.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:27:10 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:27:10.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:27:10 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:27:11.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:27:11 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:27:11.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:27:11 smithi115 conmon[90393]: 2022-05-14T04:27:11.408+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:27:11.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:27:11 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:27:12.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:27:12 smithi047 ceph-mon[77147]: pgmap v9333: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:27:12.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:27:12 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 17965 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:27:12.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:27:12 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:27:12.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:27:12 smithi115 conmon[90393]: 2022-05-14T04:27:12.430+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:27:12.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:27:12 smithi115 ceph-mon[78593]: pgmap v9333: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:27:12.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:27:12 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 17965 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:27:12.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:27:12 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:27:13.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:27:13 smithi115 conmon[90393]: 2022-05-14T04:27:13.467+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:27:14.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:27:14 smithi047 ceph-mon[77147]: pgmap v9334: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:27:14.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:27:14 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:27:14.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:27:14 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:27:14.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:27:14 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:27:14.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:27:14 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:27:14.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:27:14 smithi115 conmon[90393]: 2022-05-14T04:27:14.432+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:27:14.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:27:14 smithi115 ceph-mon[78593]: pgmap v9334: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:27:14.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:27:14 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:27:14.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:27:14 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:27:14.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:27:14 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:27:14.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:27:14 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:27:15.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:27:15 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T04:27:15.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:27:15 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T04:27:15.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:27:15 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:27:15.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:27:15 smithi115 conmon[90393]: 2022-05-14T04:27:15.425+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:27:15.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:27:15 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T04:27:15.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:27:15 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T04:27:15.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:27:15 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:27:16.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:27:16 smithi047 ceph-mon[77147]: pgmap v9335: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:27:16.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:27:16 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:27:16.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:27:16 smithi115 conmon[90393]: 2022-05-14T04:27:16.438+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:27:16.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:27:16 smithi115 ceph-mon[78593]: pgmap v9335: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:27:16.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:27:16 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:27:17.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:27:17 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 17970 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:27:17.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:27:17 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:27:17.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:27:17 smithi115 conmon[90393]: 2022-05-14T04:27:17.421+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:27:17.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:27:17 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 17970 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:27:17.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:27:17 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:27:18.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:27:18 smithi047 ceph-mon[77147]: pgmap v9336: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:27:18.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:27:18 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:27:18.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:27:18 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:27:18.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:27:18 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:27:18.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:27:18 smithi115 conmon[90393]: 2022-05-14T04:27:18.378+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:27:18.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:27:18 smithi115 ceph-mon[78593]: pgmap v9336: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:27:18.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:27:18 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:27:18.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:27:18 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:27:18.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:27:18 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:27:19.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:27:19 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:27:19.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:27:19 smithi115 conmon[90393]: 2022-05-14T04:27:19.360+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:27:19.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:27:19 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:27:20.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:27:20 smithi115 conmon[90393]: 2022-05-14T04:27:20.312+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:27:20.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:27:20 smithi115 ceph-mon[78593]: pgmap v9337: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:27:20.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:27:20 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:27:20.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:27:20 smithi047 ceph-mon[77147]: pgmap v9337: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:27:20.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:27:20 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:27:21.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:27:21 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:27:21.610 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:27:21 smithi115 conmon[90393]: 2022-05-14T04:27:21.350+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:27:21.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:27:21 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:27:22.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:27:22 smithi115 conmon[90393]: 2022-05-14T04:27:22.324+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:27:22.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:27:22 smithi115 ceph-mon[78593]: pgmap v9338: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:27:22.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:27:22 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:27:22.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:27:22 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:27:22.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:27:22 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:27:22.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:27:22 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 17975 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:27:22.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:27:22 smithi047 ceph-mon[77147]: pgmap v9338: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:27:22.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:27:22 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:27:22.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:27:22 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:27:22.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:27:22 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:27:22.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:27:22 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 17975 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:27:23.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:27:23 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:27:23.610 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:27:23 smithi115 conmon[90393]: 2022-05-14T04:27:23.285+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:27:23.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:27:23 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:27:24.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:27:24 smithi115 ceph-mon[78593]: pgmap v9339: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:27:24.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:27:24 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:27:24.610 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:27:24 smithi115 conmon[90393]: 2022-05-14T04:27:24.326+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:27:24.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:27:24 smithi047 ceph-mon[77147]: pgmap v9339: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:27:24.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:27:24 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:27:25.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:27:25 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:27:25.610 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:27:25 smithi115 conmon[90393]: 2022-05-14T04:27:25.292+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:27:25.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:27:25 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:27:26.534 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:27:26 smithi115 conmon[90393]: 2022-05-14T04:27:26.274+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:27:26.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:27:26 smithi047 ceph-mon[77147]: pgmap v9340: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:27:26.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:27:26 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:27:26.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:27:26 smithi115 ceph-mon[78593]: pgmap v9340: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:27:26.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:27:26 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:27:27.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:27:27 smithi115 conmon[90393]: 2022-05-14T04:27:27.304+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:27:27.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:27:27 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:27:27.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:27:27 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 17980 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:27:27.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:27:27 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:27:27.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:27:27 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 17980 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:27:28.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:27:28 smithi115 conmon[90393]: 2022-05-14T04:27:28.349+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:27:28.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:27:28 smithi115 ceph-mon[78593]: pgmap v9341: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:27:28.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:27:28 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:27:28.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:27:28 smithi047 ceph-mon[77147]: pgmap v9341: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:27:28.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:27:28 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:27:29.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:27:29 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:27:29.610 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:27:29 smithi115 conmon[90393]: 2022-05-14T04:27:29.335+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:27:30.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:27:29 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:27:30.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:27:30 smithi115 conmon[90393]: 2022-05-14T04:27:30.295+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:27:30.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:27:30 smithi115 ceph-mon[78593]: pgmap v9342: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:27:30.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:27:30 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:27:30.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:27:30 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:27:30.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:27:30 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:27:31.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:27:30 smithi047 ceph-mon[77147]: pgmap v9342: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:27:31.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:27:30 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:27:31.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:27:30 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:27:31.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:27:30 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:27:31.554 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:27:31 smithi115 conmon[90393]: 2022-05-14T04:27:31.289+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:27:31.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:27:31 smithi115 ceph-mon[78593]: pgmap v9343: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:27:31.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:27:31 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:27:32.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:27:31 smithi047 ceph-mon[77147]: pgmap v9343: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:27:32.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:27:31 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:27:32.558 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:27:32 smithi115 conmon[90393]: 2022-05-14T04:27:32.269+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:27:32.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:27:32 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:27:32.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:27:32 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 17985 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:27:33.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:27:32 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:27:33.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:27:32 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 17985 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:27:33.559 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:27:33 smithi115 conmon[90393]: 2022-05-14T04:27:33.296+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:27:33.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:27:33 smithi115 ceph-mon[78593]: pgmap v9344: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:27:33.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:27:33 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:27:34.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:27:33 smithi047 ceph-mon[77147]: pgmap v9344: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:27:34.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:27:33 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:27:34.560 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:27:34 smithi115 conmon[90393]: 2022-05-14T04:27:34.262+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:27:34.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:27:34 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:27:34.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:27:34 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:27:34.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:27:34 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:27:35.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:27:34 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:27:35.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:27:34 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:27:35.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:27:34 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:27:35.565 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:27:35 smithi115 conmon[90393]: 2022-05-14T04:27:35.270+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:27:35.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:27:35 smithi115 ceph-mon[78593]: pgmap v9345: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:27:35.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:27:35 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:27:36.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:27:35 smithi047 ceph-mon[77147]: pgmap v9345: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:27:36.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:27:35 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:27:36.566 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:27:36 smithi115 conmon[90393]: 2022-05-14T04:27:36.275+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:27:36.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:27:36 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:27:37.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:27:36 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:27:37.571 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:27:37 smithi115 conmon[90393]: 2022-05-14T04:27:37.305+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:27:37.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:27:37 smithi115 ceph-mon[78593]: pgmap v9346: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:27:37.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:27:37 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:27:37.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:27:37 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 17990 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:27:38.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:27:37 smithi047 ceph-mon[77147]: pgmap v9346: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:27:38.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:27:37 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:27:38.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:27:37 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 17990 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:27:38.573 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:27:38 smithi115 conmon[90393]: 2022-05-14T04:27:38.260+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:27:38.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:27:38 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:27:38.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:27:38 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:27:38.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:27:38 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:27:39.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:27:38 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:27:39.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:27:38 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:27:39.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:27:38 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:27:39.577 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:27:39 smithi115 conmon[90393]: 2022-05-14T04:27:39.280+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:27:39.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:27:39 smithi115 ceph-mon[78593]: pgmap v9347: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:27:39.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:27:39 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:27:40.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:27:39 smithi047 ceph-mon[77147]: pgmap v9347: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:27:40.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:27:39 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:27:40.579 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:27:40 smithi115 conmon[90393]: 2022-05-14T04:27:40.305+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:27:40.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:27:40 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:27:41.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:27:40 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:27:41.585 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:27:41 smithi115 ceph-mon[78593]: pgmap v9348: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:27:41.586 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:27:41 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:27:41.586 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:27:41 smithi115 conmon[90393]: 2022-05-14T04:27:41.296+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:27:42.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:27:41 smithi047 ceph-mon[77147]: pgmap v9348: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:27:42.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:27:41 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:27:42.586 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:27:42 smithi115 conmon[90393]: 2022-05-14T04:27:42.299+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:27:42.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:27:42 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:27:42.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:27:42 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:27:42.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:27:42 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:27:42.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:27:42 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 17995 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:27:43.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:27:42 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:27:43.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:27:42 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:27:43.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:27:42 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:27:43.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:27:42 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 17995 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:27:43.590 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:27:43 smithi115 conmon[90393]: 2022-05-14T04:27:43.287+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:27:43.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:27:43 smithi115 ceph-mon[78593]: pgmap v9349: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:27:43.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:27:43 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:27:44.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:27:43 smithi047 ceph-mon[77147]: pgmap v9349: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:27:44.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:27:43 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:27:44.592 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:27:44 smithi115 conmon[90393]: 2022-05-14T04:27:44.255+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:27:44.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:27:44 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:27:45.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:27:44 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:27:45.597 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:27:45 smithi115 conmon[90393]: 2022-05-14T04:27:45.251+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:27:45.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:27:45 smithi115 ceph-mon[78593]: pgmap v9350: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:27:45.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:27:45 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:27:46.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:27:45 smithi047 ceph-mon[77147]: pgmap v9350: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:27:46.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:27:45 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:27:46.598 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:27:46 smithi115 conmon[90393]: 2022-05-14T04:27:46.271+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:27:46.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:27:46 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:27:47.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:27:46 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:27:47.602 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:27:47 smithi115 conmon[90393]: 2022-05-14T04:27:47.249+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:27:47.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:27:47 smithi115 ceph-mon[78593]: pgmap v9351: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:27:47.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:27:47 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:27:47.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:27:47 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 18000 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:27:48.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:27:47 smithi047 ceph-mon[77147]: pgmap v9351: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:27:48.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:27:47 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:27:48.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:27:47 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 18000 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:27:48.604 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:27:48 smithi115 conmon[90393]: 2022-05-14T04:27:48.268+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:27:48.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:27:48 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:27:49.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:27:48 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:27:49.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:27:49 smithi115 conmon[90393]: 2022-05-14T04:27:49.317+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:27:50.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:27:49 smithi047 ceph-mon[77147]: pgmap v9352: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:27:50.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:27:49 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:27:50.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:27:49 smithi115 ceph-mon[78593]: pgmap v9352: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:27:50.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:27:49 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:27:50.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:27:50 smithi115 conmon[90393]: 2022-05-14T04:27:50.290+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:27:51.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:27:50 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:27:51.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:27:50 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:27:51.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:27:50 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:27:51.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:27:50 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:27:51.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:27:50 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:27:51.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:27:50 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:27:51.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:27:51 smithi115 conmon[90393]: 2022-05-14T04:27:51.294+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:27:52.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:27:51 smithi047 ceph-mon[77147]: pgmap v9353: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:27:52.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:27:51 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:27:52.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:27:51 smithi115 ceph-mon[78593]: pgmap v9353: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:27:52.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:27:51 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:27:52.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:27:52 smithi115 conmon[90393]: 2022-05-14T04:27:52.252+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:27:53.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:27:52 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:27:53.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:27:52 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 18005 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:27:53.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:27:52 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:27:53.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:27:52 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 18005 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:27:53.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:27:53 smithi115 conmon[90393]: 2022-05-14T04:27:53.284+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:27:54.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:27:53 smithi047 ceph-mon[77147]: pgmap v9354: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:27:54.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:27:53 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:27:54.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:27:53 smithi115 ceph-mon[78593]: pgmap v9354: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:27:54.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:27:53 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:27:54.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:27:54 smithi115 conmon[90393]: 2022-05-14T04:27:54.244+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:27:55.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:27:54 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:27:55.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:27:54 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:27:55.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:27:54 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:27:55.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:27:54 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:27:55.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:27:54 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:27:55.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:27:54 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:27:55.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:27:55 smithi115 conmon[90393]: 2022-05-14T04:27:55.210+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:27:56.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:27:55 smithi047 ceph-mon[77147]: pgmap v9355: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:27:56.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:27:55 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:27:56.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:27:55 smithi115 ceph-mon[78593]: pgmap v9355: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:27:56.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:27:55 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:27:56.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:27:56 smithi115 conmon[90393]: 2022-05-14T04:27:56.258+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:27:57.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:27:56 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:27:57.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:27:56 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:27:57.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:27:57 smithi115 conmon[90393]: 2022-05-14T04:27:57.249+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:27:58.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:27:57 smithi047 ceph-mon[77147]: pgmap v9356: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:27:58.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:27:57 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:27:58.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:27:57 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 18010 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:27:58.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:27:57 smithi115 ceph-mon[78593]: pgmap v9356: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:27:58.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:27:57 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:27:58.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:27:57 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 18010 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:27:58.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:27:58 smithi115 conmon[90393]: 2022-05-14T04:27:58.292+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:27:59.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:27:58 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:27:59.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:27:58 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:27:59.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:27:58 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:27:59.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:27:58 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:27:59.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:27:58 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:27:59.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:27:58 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:27:59.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:27:59 smithi115 conmon[90393]: 2022-05-14T04:27:59.271+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:28:00.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:27:59 smithi047 ceph-mon[77147]: pgmap v9357: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:28:00.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:27:59 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:28:00.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:27:59 smithi115 ceph-mon[78593]: pgmap v9357: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:28:00.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:27:59 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:28:00.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:28:00 smithi115 conmon[90393]: 2022-05-14T04:28:00.311+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:28:01.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:28:00 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:28:01.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:28:00 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:28:01.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:28:01 smithi115 conmon[90393]: 2022-05-14T04:28:01.277+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:28:02.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:28:01 smithi047 ceph-mon[77147]: pgmap v9358: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:28:02.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:28:01 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:28:02.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:28:01 smithi115 ceph-mon[78593]: pgmap v9358: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:28:02.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:28:01 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:28:02.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:28:02 smithi115 conmon[90393]: 2022-05-14T04:28:02.286+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:28:03.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:28:02 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:28:03.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:28:02 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:28:03.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:28:02 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:28:03.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:28:02 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 18015 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:28:03.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:28:02 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T04:28:03.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:28:02 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T04:28:03.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:28:02 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T04:28:03.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:28:02 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:28:03.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:28:02 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:28:03.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:28:02 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:28:03.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:28:02 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 18015 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:28:03.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:28:02 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T04:28:03.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:28:02 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T04:28:03.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:28:02 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T04:28:03.542 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:28:03 smithi047 conmon[78521]: 2022-05-14T04:28:03.247+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:28:03.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:28:03 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T04:28:03.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:28:03 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:28:03.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:28:03 smithi047 conmon[78521]: yield 2022-05-14T04:28:03.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:28:03 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:28:03.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:28:03 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:28:03.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:28:03 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:28:03.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:28:03 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T04:28:03.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:28:03 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:28:03.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:28:03 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T04:28:03.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:28:03 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:28:03.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:28:03 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T04:28:03.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:28:03 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:28:03.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:28:03 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T04:28:03.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:28:03 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:28:03.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:28:03 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T04:28:03.547 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:28:03 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:28:03.547 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:28:03 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:28:03.547 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:28:03 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:28:03.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:28:03 smithi115 conmon[90393]: 2022-05-14T04:28:03.253+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:28:04.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:28:03 smithi047 ceph-mon[77147]: pgmap v9359: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:28:04.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:28:03 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:28:04.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T04:28:04.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:28:04.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T04:28:04.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:28:04.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:28:04.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:28:04.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T04:28:04.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:28:04.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T04:28:04.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:28:04.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T04:28:04.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:28:04.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T04:28:04.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:28:04.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T04:28:04.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:28:04.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:28:04.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:28:04.048 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:28:03 smithi047 ceph-mon[77147]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T04:28:04.048 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:28:03 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:28:04.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:28:03 smithi115 ceph-mon[78593]: pgmap v9359: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:28:04.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:28:03 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:28:04.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T04:28:04.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:28:04.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T04:28:04.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:28:04.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:28:04.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:28:04.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T04:28:04.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:28:04.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T04:28:04.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:28:04.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T04:28:04.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:28:04.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T04:28:04.113 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:28:04.113 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T04:28:04.113 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:28:04.113 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:28:04.113 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:28:04.114 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:28:03 smithi115 ceph-mon[78593]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T04:28:04.114 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:28:03 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:28:04.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:28:04 smithi115 conmon[90393]: 2022-05-14T04:28:04.297+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:28:05.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:28:04 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:28:05.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:28:04 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:28:05.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:28:05 smithi115 conmon[90393]: 2022-05-14T04:28:05.319+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:28:06.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:28:05 smithi047 ceph-mon[77147]: pgmap v9360: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:28:06.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:28:05 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:28:06.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:28:05 smithi115 ceph-mon[78593]: pgmap v9360: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:28:06.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:28:05 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:28:06.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:28:06 smithi115 conmon[90393]: 2022-05-14T04:28:06.294+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:28:06.663 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:28:06 smithi047 conmon[78521]: 2022-05-14T04:28:06.319+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:28:06.664 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:28:06 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T04:28:06.664 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:28:06 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:28:06.664 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:28:06 smithi047 conmon[78521]: yield 2022-05-14T04:28:06.665 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:28:06 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:28:06.665 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:28:06 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:28:06.665 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:28:06 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:28:06.665 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:28:06 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T04:28:06.666 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:28:06 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:28:06.666 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:28:06 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T04:28:06.666 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:28:06 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:28:06.667 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:28:06 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T04:28:06.667 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:28:06 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:28:06.667 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:28:06 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T04:28:06.667 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:28:06 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:28:06.667 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:28:06 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T04:28:06.668 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:28:06 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:28:06.668 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:28:06 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:28:06.668 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:28:06 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:28:06.668 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:28:06 smithi047 conmon[78521]: 2022-05-14T04:28:06.320+0000 7f5344781700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T04:28:06.669 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:28:06 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T04:28:06.669 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:28:06 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:28:06.669 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:28:06 smithi047 conmon[78521]: yield 2022-05-14T04:28:06.669 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:28:06 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:28:06.670 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:28:06 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:28:06.670 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:28:06 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:28:06.670 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:28:06 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T04:28:06.670 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:28:06 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:28:06.670 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:28:06 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T04:28:06.671 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:28:06 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:28:06.671 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:28:06 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T04:28:06.671 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:28:06 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:28:06.671 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:28:06 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T04:28:06.672 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:28:06 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:28:06.672 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:28:06 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T04:28:06.672 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:28:06 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:28:06.672 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:28:06 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:28:06.673 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:28:06 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:28:06.673 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:28:06 smithi047 conmon[78521]: 2022-05-14T04:28:06.673 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:28:06 smithi047 conmon[78521]: During handling of the above exception, another exception occurred: 2022-05-14T04:28:06.673 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:28:06 smithi047 conmon[78521]: 2022-05-14T04:28:06.674 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:28:06 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T04:28:06.674 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:28:06 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T04:28:06.674 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:28:06 smithi047 conmon[78521]: return f(*arg) 2022-05-14T04:28:06.674 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:28:06 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T04:28:06.674 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:28:06 smithi047 conmon[78521]: self._write_client_files(client_files, host) 2022-05-14T04:28:06.675 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:28:06 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T04:28:06.675 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:28:06 smithi047 conmon[78521]: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T04:28:06.675 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:28:06 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T04:28:06.675 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:28:06 smithi047 conmon[78521]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T04:28:06.676 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:28:06 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T04:28:06.676 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:28:06 smithi047 conmon[78521]: return self.event_loop.get_result(coro) 2022-05-14T04:28:06.676 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:28:06 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T04:28:06.676 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:28:06 smithi047 conmon[78521]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T04:28:06.677 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:28:06 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T04:28:06.677 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:28:06 smithi047 conmon[78521]: return self.__get_result() 2022-05-14T04:28:06.677 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:28:06 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T04:28:06.677 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:28:06 smithi047 conmon[78521]: raise self._exception 2022-05-14T04:28:06.678 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:28:06 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T04:28:06.678 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:28:06 smithi047 conmon[78521]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T04:28:06.678 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:28:06 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T04:28:06.678 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:28:06 smithi047 conmon[78521]: conn = await self._remote_connection(host, addr) 2022-05-14T04:28:06.678 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:28:06 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T04:28:06.679 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:28:06 smithi047 conmon[78521]: raise 2022-05-14T04:28:06.679 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:28:06 smithi047 conmon[78521]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T04:28:06.679 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:28:06 smithi047 conmon[78521]: self.gen.throw(type, value, traceback) 2022-05-14T04:28:06.679 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:28:06 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T04:28:06.680 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:28:06 smithi047 conmon[78521]: raise OrchestratorError(msg) 2022-05-14T04:28:06.680 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:28:06 smithi047 conmon[78521]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:28:07.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:28:06 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:28:07.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:28:06 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:28:07.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T04:28:07.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:28:07.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T04:28:07.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:28:07.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:28:07.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:28:07.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T04:28:07.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:28:07.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T04:28:07.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:28:07.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T04:28:07.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:28:07.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T04:28:07.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:28:07.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T04:28:07.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:28:07.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:28:07.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:28:07.048 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:28:06 smithi047 ceph-mon[77147]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T04:28:07.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T04:28:07.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:28:07.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T04:28:07.049 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:28:07.050 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:28:07.050 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:28:07.050 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T04:28:07.051 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:28:07.051 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T04:28:07.051 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:28:07.051 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T04:28:07.052 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:28:07.052 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T04:28:07.052 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:28:07.052 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T04:28:07.053 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:28:07.053 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:28:07.053 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:28:07.053 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T04:28:07.054 INFO:journalctl@ceph.mon.b.smithi047.stdout: During handling of the above exception, another exception occurred: 2022-05-14T04:28:07.054 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T04:28:07.054 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T04:28:07.054 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T04:28:07.055 INFO:journalctl@ceph.mon.b.smithi047.stdout: return f(*arg) 2022-05-14T04:28:07.055 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T04:28:07.055 INFO:journalctl@ceph.mon.b.smithi047.stdout: self._write_client_files(client_files, host) 2022-05-14T04:28:07.055 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T04:28:07.056 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T04:28:07.056 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T04:28:07.056 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T04:28:07.057 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T04:28:07.057 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.event_loop.get_result(coro) 2022-05-14T04:28:07.057 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T04:28:07.058 INFO:journalctl@ceph.mon.b.smithi047.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T04:28:07.058 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T04:28:07.058 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.__get_result() 2022-05-14T04:28:07.058 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T04:28:07.059 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise self._exception 2022-05-14T04:28:07.059 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T04:28:07.059 INFO:journalctl@ceph.mon.b.smithi047.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T04:28:07.059 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T04:28:07.060 INFO:journalctl@ceph.mon.b.smithi047.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T04:28:07.060 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T04:28:07.060 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise 2022-05-14T04:28:07.060 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T04:28:07.061 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.gen.throw(type, value, traceback) 2022-05-14T04:28:07.061 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T04:28:07.061 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OrchestratorError(msg) 2022-05-14T04:28:07.061 INFO:journalctl@ceph.mon.b.smithi047.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:28:07.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:28:06 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:28:07.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:28:06 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:28:07.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T04:28:07.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:28:07.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T04:28:07.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:28:07.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:28:07.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:28:07.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T04:28:07.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:28:07.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T04:28:07.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:28:07.114 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T04:28:07.114 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:28:07.114 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T04:28:07.115 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:28:07.115 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T04:28:07.115 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:28:07.115 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:28:07.116 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:28:07.116 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:28:06 smithi115 ceph-mon[78593]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T04:28:07.116 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T04:28:07.116 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:28:07.116 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T04:28:07.117 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:28:07.117 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:28:07.117 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:28:07.117 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T04:28:07.118 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:28:07.118 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T04:28:07.118 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:28:07.118 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T04:28:07.118 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:28:07.119 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T04:28:07.119 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:28:07.119 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T04:28:07.119 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:28:07.119 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:28:07.120 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:28:07.120 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T04:28:07.120 INFO:journalctl@ceph.mon.c.smithi115.stdout: During handling of the above exception, another exception occurred: 2022-05-14T04:28:07.120 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T04:28:07.120 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T04:28:07.121 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T04:28:07.121 INFO:journalctl@ceph.mon.c.smithi115.stdout: return f(*arg) 2022-05-14T04:28:07.121 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T04:28:07.121 INFO:journalctl@ceph.mon.c.smithi115.stdout: self._write_client_files(client_files, host) 2022-05-14T04:28:07.121 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T04:28:07.122 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T04:28:07.122 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T04:28:07.122 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T04:28:07.122 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T04:28:07.123 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.event_loop.get_result(coro) 2022-05-14T04:28:07.123 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T04:28:07.123 INFO:journalctl@ceph.mon.c.smithi115.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T04:28:07.123 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T04:28:07.123 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.__get_result() 2022-05-14T04:28:07.124 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T04:28:07.124 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise self._exception 2022-05-14T04:28:07.124 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T04:28:07.124 INFO:journalctl@ceph.mon.c.smithi115.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T04:28:07.124 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T04:28:07.125 INFO:journalctl@ceph.mon.c.smithi115.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T04:28:07.125 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T04:28:07.125 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise 2022-05-14T04:28:07.125 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T04:28:07.125 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.gen.throw(type, value, traceback) 2022-05-14T04:28:07.126 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T04:28:07.126 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OrchestratorError(msg) 2022-05-14T04:28:07.126 INFO:journalctl@ceph.mon.c.smithi115.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:28:07.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:28:07 smithi115 conmon[90393]: 2022-05-14T04:28:07.291+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:28:08.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:28:07 smithi047 ceph-mon[77147]: pgmap v9361: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:28:08.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:28:07 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:28:08.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:28:07 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 18020 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:28:08.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:28:07 smithi115 ceph-mon[78593]: pgmap v9361: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:28:08.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:28:07 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:28:08.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:28:07 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 18020 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:28:08.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:28:08 smithi115 conmon[90393]: 2022-05-14T04:28:08.243+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:28:09.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:28:08 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:28:09.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:28:08 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:28:09.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:28:09 smithi115 conmon[90393]: 2022-05-14T04:28:09.203+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:28:10.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:28:09 smithi047 ceph-mon[77147]: pgmap v9362: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:28:10.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:28:09 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:28:10.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:28:09 smithi115 ceph-mon[78593]: pgmap v9362: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:28:10.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:28:09 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:28:10.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:28:10 smithi115 conmon[90393]: 2022-05-14T04:28:10.247+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:28:11.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:28:10 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:28:11.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:28:10 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:28:11.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:28:10 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:28:11.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:28:10 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:28:11.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:28:10 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:28:11.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:28:10 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:28:11.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:28:11 smithi115 conmon[90393]: 2022-05-14T04:28:11.257+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:28:12.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:28:11 smithi047 ceph-mon[77147]: pgmap v9363: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:28:12.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:28:11 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:28:12.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:28:11 smithi115 ceph-mon[78593]: pgmap v9363: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:28:12.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:28:11 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:28:12.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:28:12 smithi115 conmon[90393]: 2022-05-14T04:28:12.253+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:28:13.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:28:12 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:28:13.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:28:12 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 18025 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:28:13.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:28:12 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:28:13.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:28:12 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 18025 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:28:13.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:28:13 smithi115 conmon[90393]: 2022-05-14T04:28:13.287+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:28:14.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:28:13 smithi047 ceph-mon[77147]: pgmap v9364: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:28:14.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:28:13 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:28:14.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:28:13 smithi115 ceph-mon[78593]: pgmap v9364: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:28:14.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:28:13 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:28:14.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:28:14 smithi115 conmon[90393]: 2022-05-14T04:28:14.292+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:28:15.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:28:14 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:28:15.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:28:14 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:28:15.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:28:14 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:28:15.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:28:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T04:28:15.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:28:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T04:28:15.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:28:14 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:28:15.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:28:14 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:28:15.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:28:14 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:28:15.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:28:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T04:28:15.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:28:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T04:28:15.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:28:15 smithi115 conmon[90393]: 2022-05-14T04:28:15.272+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:28:16.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:28:15 smithi047 ceph-mon[77147]: pgmap v9365: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:28:16.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:28:15 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:28:16.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:28:15 smithi115 ceph-mon[78593]: pgmap v9365: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:28:16.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:28:15 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:28:16.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:28:16 smithi115 conmon[90393]: 2022-05-14T04:28:16.264+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:28:17.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:28:16 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:28:17.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:28:16 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:28:17.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:28:17 smithi115 conmon[90393]: 2022-05-14T04:28:17.280+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:28:18.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:28:17 smithi047 ceph-mon[77147]: pgmap v9366: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:28:18.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:28:17 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:28:18.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:28:17 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 18030 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:28:18.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:28:17 smithi115 ceph-mon[78593]: pgmap v9366: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:28:18.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:28:17 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:28:18.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:28:17 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 18030 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:28:18.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:28:18 smithi115 conmon[90393]: 2022-05-14T04:28:18.285+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:28:19.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:28:18 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:28:19.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:28:18 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:28:19.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:28:18 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:28:19.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:28:18 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:28:19.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:28:18 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:28:19.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:28:18 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:28:19.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:28:19 smithi115 conmon[90393]: 2022-05-14T04:28:19.296+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:28:20.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:28:19 smithi047 ceph-mon[77147]: pgmap v9367: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:28:20.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:28:19 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:28:20.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:28:19 smithi115 ceph-mon[78593]: pgmap v9367: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:28:20.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:28:19 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:28:20.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:28:20 smithi115 conmon[90393]: 2022-05-14T04:28:20.250+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:28:21.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:28:20 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:28:21.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:28:20 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:28:21.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:28:21 smithi115 conmon[90393]: 2022-05-14T04:28:21.279+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:28:22.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:28:21 smithi047 ceph-mon[77147]: pgmap v9368: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:28:22.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:28:21 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:28:22.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:28:21 smithi115 ceph-mon[78593]: pgmap v9368: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:28:22.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:28:21 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:28:22.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:28:22 smithi115 conmon[90393]: 2022-05-14T04:28:22.256+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:28:23.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:28:22 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:28:23.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:28:22 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:28:23.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:28:22 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:28:23.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:28:22 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 18035 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:28:23.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:28:22 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:28:23.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:28:22 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:28:23.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:28:22 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:28:23.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:28:22 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 18035 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:28:23.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:28:23 smithi115 conmon[90393]: 2022-05-14T04:28:23.211+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:28:24.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:28:23 smithi047 ceph-mon[77147]: pgmap v9369: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:28:24.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:28:23 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:28:24.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:28:23 smithi115 ceph-mon[78593]: pgmap v9369: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:28:24.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:28:23 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:28:24.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:28:24 smithi115 conmon[90393]: 2022-05-14T04:28:24.240+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:28:25.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:28:24 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:28:25.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:28:24 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:28:25.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:28:25 smithi115 conmon[90393]: 2022-05-14T04:28:25.222+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:28:26.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:28:25 smithi047 ceph-mon[77147]: pgmap v9370: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:28:26.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:28:25 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:28:26.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:28:25 smithi115 ceph-mon[78593]: pgmap v9370: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:28:26.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:28:25 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:28:26.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:28:26 smithi115 conmon[90393]: 2022-05-14T04:28:26.180+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:28:27.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:28:26 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:28:27.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:28:26 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:28:27.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:28:27 smithi115 conmon[90393]: 2022-05-14T04:28:27.169+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:28:28.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:28:27 smithi047 ceph-mon[77147]: pgmap v9371: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:28:28.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:28:27 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:28:28.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:28:27 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 18040 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:28:28.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:28:27 smithi115 ceph-mon[78593]: pgmap v9371: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:28:28.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:28:27 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:28:28.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:28:27 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 18040 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:28:28.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:28:28 smithi115 conmon[90393]: 2022-05-14T04:28:28.168+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:28:29.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:28:28 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:28:29.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:28:28 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:28:29.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:28:29 smithi115 conmon[90393]: 2022-05-14T04:28:29.171+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:28:30.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:28:29 smithi047 ceph-mon[77147]: pgmap v9372: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:28:30.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:28:29 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:28:30.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:28:29 smithi115 ceph-mon[78593]: pgmap v9372: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:28:30.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:28:29 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:28:30.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:28:30 smithi115 conmon[90393]: 2022-05-14T04:28:30.179+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:28:31.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:28:30 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:28:31.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:28:30 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:28:31.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:28:30 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:28:31.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:28:30 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:28:31.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:28:30 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:28:31.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:28:30 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:28:31.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:28:31 smithi115 conmon[90393]: 2022-05-14T04:28:31.216+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:28:32.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:28:31 smithi047 ceph-mon[77147]: pgmap v9373: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:28:32.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:28:31 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:28:32.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:28:31 smithi115 ceph-mon[78593]: pgmap v9373: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:28:32.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:28:31 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:28:32.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:28:32 smithi115 conmon[90393]: 2022-05-14T04:28:32.194+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:28:33.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:28:32 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:28:33.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:28:32 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 18045 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:28:33.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:28:32 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:28:33.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:28:32 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 18045 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:28:33.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:28:33 smithi115 conmon[90393]: 2022-05-14T04:28:33.175+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:28:34.446 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:28:34 smithi115 conmon[90393]: 2022-05-14T04:28:34.160+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:28:34.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:28:34 smithi047 ceph-mon[77147]: pgmap v9374: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:28:34.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:28:34 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:28:34.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:28:34 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:28:34.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:28:34 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:28:34.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:28:34 smithi115 ceph-mon[78593]: pgmap v9374: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:28:34.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:28:34 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:28:34.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:28:34 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:28:34.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:28:34 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:28:35.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:28:35 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:28:35.610 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:28:35 smithi115 conmon[90393]: 2022-05-14T04:28:35.203+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:28:35.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:28:35 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:28:36.452 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:28:36 smithi115 conmon[90393]: 2022-05-14T04:28:36.154+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:28:36.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:28:36 smithi047 ceph-mon[77147]: pgmap v9375: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:28:36.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:28:36 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:28:36.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:28:36 smithi115 ceph-mon[78593]: pgmap v9375: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:28:36.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:28:36 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:28:37.457 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:28:37 smithi115 conmon[90393]: 2022-05-14T04:28:37.138+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:28:37.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:28:37 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:28:37.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:28:37 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 18050 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:28:37.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:28:37 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:28:37.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:28:37 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 18050 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:28:38.458 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:28:38 smithi115 conmon[90393]: 2022-05-14T04:28:38.146+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:28:38.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:28:38 smithi047 ceph-mon[77147]: pgmap v9376: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:28:38.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:28:38 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:28:38.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:28:38 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:28:38.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:28:38 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:28:38.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:28:38 smithi115 ceph-mon[78593]: pgmap v9376: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:28:38.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:28:38 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:28:38.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:28:38 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:28:38.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:28:38 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:28:39.465 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:28:39 smithi115 conmon[90393]: 2022-05-14T04:28:39.111+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:28:39.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:28:39 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:28:39.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:28:39 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:28:40.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:28:40 smithi115 conmon[90393]: 2022-05-14T04:28:40.102+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:28:40.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:28:40 smithi047 ceph-mon[77147]: pgmap v9377: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:28:40.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:28:40 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:28:40.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:28:40 smithi115 ceph-mon[78593]: pgmap v9377: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:28:40.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:28:40 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:28:41.474 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:28:41 smithi115 conmon[90393]: 2022-05-14T04:28:41.131+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:28:41.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:28:41 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:28:41.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:28:41 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:28:42.474 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:28:42 smithi115 conmon[90393]: 2022-05-14T04:28:42.163+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:28:42.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:28:42 smithi047 ceph-mon[77147]: pgmap v9378: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:28:42.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:28:42 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:28:42.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:28:42 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:28:42.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:28:42 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:28:42.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:28:42 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 18055 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:28:42.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:28:42 smithi115 ceph-mon[78593]: pgmap v9378: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:28:42.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:28:42 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:28:42.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:28:42 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:28:42.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:28:42 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:28:42.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:28:42 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 18055 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:28:43.479 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:28:43 smithi115 conmon[90393]: 2022-05-14T04:28:43.171+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:28:43.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:28:43 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:28:43.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:28:43 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:28:44.480 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:28:44 smithi115 conmon[90393]: 2022-05-14T04:28:44.213+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:28:44.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:28:44 smithi047 ceph-mon[77147]: pgmap v9379: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:28:44.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:28:44 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:28:44.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:28:44 smithi115 ceph-mon[78593]: pgmap v9379: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:28:44.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:28:44 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:28:45.487 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:28:45 smithi115 conmon[90393]: 2022-05-14T04:28:45.229+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:28:45.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:28:45 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:28:45.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:28:45 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:28:46.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:28:46 smithi115 conmon[90393]: 2022-05-14T04:28:46.266+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:28:46.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:28:46 smithi115 ceph-mon[78593]: pgmap v9380: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:28:46.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:28:46 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:28:46.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:28:46 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:28:46.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:28:46 smithi047 ceph-mon[77147]: pgmap v9380: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:28:46.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:28:46 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:28:46.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:28:46 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:28:47.490 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:28:47 smithi115 conmon[90393]: 2022-05-14T04:28:47.227+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:28:47.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:28:47 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:28:47.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:28:47 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 18060 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:28:47.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:28:47 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:28:47.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:28:47 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 18060 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:28:48.492 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:28:48 smithi115 conmon[90393]: 2022-05-14T04:28:48.220+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:28:48.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:28:48 smithi047 ceph-mon[77147]: pgmap v9381: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:28:48.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:28:48 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:28:48.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:28:48 smithi115 ceph-mon[78593]: pgmap v9381: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:28:48.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:28:48 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:28:49.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:28:49 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:28:49.610 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:28:49 smithi115 conmon[90393]: 2022-05-14T04:28:49.252+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:28:49.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:28:49 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:28:50.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:28:50 smithi115 ceph-mon[78593]: pgmap v9382: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:28:50.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:28:50 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:28:50.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:28:50 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:28:50.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:28:50 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:28:50.610 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:28:50 smithi115 conmon[90393]: 2022-05-14T04:28:50.252+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:28:50.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:28:50 smithi047 ceph-mon[77147]: pgmap v9382: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:28:50.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:28:50 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:28:50.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:28:50 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:28:50.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:28:50 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:28:51.506 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:28:51 smithi115 conmon[90393]: 2022-05-14T04:28:51.218+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:28:51.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:28:51 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:28:51.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:28:51 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:28:52.507 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:28:52 smithi115 conmon[90393]: 2022-05-14T04:28:52.218+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:28:52.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:28:52 smithi047 ceph-mon[77147]: pgmap v9383: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:28:52.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:28:52 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:28:52.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:28:52 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 18065 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:28:52.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:28:52 smithi115 ceph-mon[78593]: pgmap v9383: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:28:52.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:28:52 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:28:52.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:28:52 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 18065 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:28:53.513 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:28:53 smithi115 conmon[90393]: 2022-05-14T04:28:53.256+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:28:53.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:28:53 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:28:53.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:28:53 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:28:54.513 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:28:54 smithi115 conmon[90393]: 2022-05-14T04:28:54.259+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:28:54.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:28:54 smithi047 ceph-mon[77147]: pgmap v9384: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:28:54.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:28:54 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:28:54.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:28:54 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:28:54.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:28:54 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:28:54.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:28:54 smithi115 ceph-mon[78593]: pgmap v9384: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:28:54.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:28:54 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:28:54.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:28:54 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:28:54.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:28:54 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:28:55.518 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:28:55 smithi115 conmon[90393]: 2022-05-14T04:28:55.229+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:28:55.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:28:55 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:28:55.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:28:55 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:28:56.519 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:28:56 smithi115 conmon[90393]: 2022-05-14T04:28:56.182+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:28:56.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:28:56 smithi047 ceph-mon[77147]: pgmap v9385: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:28:56.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:28:56 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:28:56.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:28:56 smithi115 ceph-mon[78593]: pgmap v9385: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:28:56.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:28:56 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:28:57.526 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:28:57 smithi115 conmon[90393]: 2022-05-14T04:28:57.207+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:28:57.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:28:57 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:28:57.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:28:57 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 18070 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:28:57.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:28:57 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:28:57.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:28:57 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 18070 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:28:58.526 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:28:58 smithi115 conmon[90393]: 2022-05-14T04:28:58.182+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:28:58.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:28:58 smithi047 ceph-mon[77147]: pgmap v9386: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:28:58.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:28:58 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:28:58.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:28:58 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:28:58.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:28:58 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:28:58.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:28:58 smithi115 ceph-mon[78593]: pgmap v9386: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:28:58.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:28:58 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:28:58.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:28:58 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:28:58.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:28:58 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:28:59.533 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:28:59 smithi115 conmon[90393]: 2022-05-14T04:28:59.225+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:28:59.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:28:59 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:28:59.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:28:59 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:29:00.532 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:29:00 smithi115 conmon[90393]: 2022-05-14T04:29:00.199+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:29:00.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:29:00 smithi047 ceph-mon[77147]: pgmap v9387: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:29:00.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:29:00 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:29:00.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:29:00 smithi115 ceph-mon[78593]: pgmap v9387: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:29:00.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:29:00 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:29:01.539 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:29:01 smithi115 conmon[90393]: 2022-05-14T04:29:01.245+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:29:01.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:29:01 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:29:01.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:29:01 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:29:02.540 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:29:02 smithi115 conmon[90393]: 2022-05-14T04:29:02.230+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:29:02.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:29:02 smithi047 ceph-mon[77147]: pgmap v9388: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:29:02.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:29:02 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:29:02.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:29:02 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:29:02.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:29:02 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:29:02.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:29:02 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 18075 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:29:02.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:29:02 smithi115 ceph-mon[78593]: pgmap v9388: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:29:02.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:29:02 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:29:02.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:29:02 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:29:02.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:29:02 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:29:02.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:29:02 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 18075 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:29:03.545 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:29:03 smithi115 conmon[90393]: 2022-05-14T04:29:03.228+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:29:03.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:29:03 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:29:04.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:29:03 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:29:04.546 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:29:04 smithi115 conmon[90393]: 2022-05-14T04:29:04.203+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:29:04.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:29:04 smithi115 ceph-mon[78593]: pgmap v9389: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:29:04.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:29:04 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:29:05.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:29:04 smithi047 ceph-mon[77147]: pgmap v9389: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:29:05.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:29:04 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:29:05.551 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:29:05 smithi115 conmon[90393]: 2022-05-14T04:29:05.209+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:29:05.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:29:05 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:29:06.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:29:05 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:29:06.425 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:29:06 smithi115 conmon[90393]: 2022-05-14T04:29:06.170+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:29:06.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:29:06 smithi047 ceph-mon[77147]: pgmap v9390: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:29:06.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:29:06 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:29:06.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:29:06 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T04:29:06.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:29:06 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T04:29:06.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:29:06 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T04:29:06.822 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:29:06 smithi115 ceph-mon[78593]: pgmap v9390: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:29:06.823 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:29:06 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:29:06.823 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:29:06 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T04:29:06.823 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:29:06 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T04:29:06.823 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:29:06 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T04:29:07.556 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:29:07 smithi115 conmon[90393]: 2022-05-14T04:29:07.206+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:29:07.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:29:07 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:29:07.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:29:07 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 18080 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:29:08.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:29:07 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:29:08.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:29:07 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 18080 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:29:08.557 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:29:08 smithi115 conmon[90393]: 2022-05-14T04:29:08.192+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:29:08.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:29:08 smithi115 ceph-mon[78593]: pgmap v9391: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:29:08.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:29:08 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:29:09.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:29:08 smithi047 ceph-mon[77147]: pgmap v9391: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:29:09.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:29:08 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:29:09.563 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:29:09 smithi115 conmon[90393]: 2022-05-14T04:29:09.239+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:29:09.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:29:09 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:29:09.793 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:29:09 smithi047 conmon[78521]: 2022-05-14T04:29:09.424+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:29:09.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:29:09 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T04:29:09.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:29:09 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:29:09.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:29:09 smithi047 conmon[78521]: yield 2022-05-14T04:29:09.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:29:09 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:29:09.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:29:09 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:29:09.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:29:09 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:29:09.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:29:09 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T04:29:09.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:29:09 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:29:09.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:29:09 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T04:29:09.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:29:09 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:29:09.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:29:09 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T04:29:09.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:29:09 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:29:09.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:29:09 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T04:29:09.797 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:29:09 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:29:09.797 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:29:09 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T04:29:09.797 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:29:09 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:29:09.797 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:29:09 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:29:09.798 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:29:09 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:29:09.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:29:09 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:29:10.564 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:29:10 smithi115 conmon[90393]: 2022-05-14T04:29:10.229+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:29:10.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:29:10 smithi115 ceph-mon[78593]: pgmap v9392: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:29:10.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:29:10 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:29:10.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T04:29:10.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:29:10.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T04:29:10.863 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:29:10.863 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:29:10.864 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:29:10.864 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T04:29:10.865 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:29:10.865 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T04:29:10.866 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:29:10.866 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T04:29:10.867 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:29:10.867 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T04:29:10.867 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:29:10.868 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T04:29:10.868 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:29:10.869 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:29:10.870 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:29:10.870 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:29:10 smithi115 ceph-mon[78593]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T04:29:10.871 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:29:10 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:29:10.871 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:29:10 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:29:10.872 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:29:10 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:29:11.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:29:10 smithi047 ceph-mon[77147]: pgmap v9392: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:29:11.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:29:10 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:29:11.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T04:29:11.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:29:11.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T04:29:11.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:29:11.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:29:11.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:29:11.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T04:29:11.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:29:11.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T04:29:11.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:29:11.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T04:29:11.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:29:11.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T04:29:11.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:29:11.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T04:29:11.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:29:11.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:29:11.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:29:11.047 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:29:10 smithi047 ceph-mon[77147]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T04:29:11.048 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:29:10 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:29:11.048 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:29:10 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:29:11.048 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:29:10 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:29:11.569 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:29:11 smithi115 conmon[90393]: 2022-05-14T04:29:11.259+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:29:11.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:29:11 smithi115 ceph-mon[78593]: pgmap v9393: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:29:11.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:29:11 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:29:12.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:29:11 smithi047 ceph-mon[77147]: pgmap v9393: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:29:12.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:29:11 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:29:12.570 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:29:12 smithi115 conmon[90393]: 2022-05-14T04:29:12.251+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:29:12.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:29:12 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:29:12.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:29:12 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 18085 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:29:12.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:29:12 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:29:12.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T04:29:12.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:29:12.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T04:29:12.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:29:12.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:29:12.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:29:12.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T04:29:12.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:29:12.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T04:29:12.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:29:12.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T04:29:12.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:29:12.797 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T04:29:12.797 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:29:12.797 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T04:29:12.797 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:29:12.798 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:29:12.798 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:29:12.798 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:29:12 smithi047 ceph-mon[77147]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T04:29:12.798 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T04:29:12.798 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:29:12.799 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T04:29:12.799 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:29:12.799 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:29:12.799 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:29:12.799 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T04:29:12.800 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:29:12.800 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T04:29:12.800 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:29:12.800 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T04:29:12.801 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:29:12.801 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T04:29:12.801 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:29:12.801 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T04:29:12.802 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:29:12.802 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:29:12.802 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:29:12.802 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T04:29:12.803 INFO:journalctl@ceph.mon.b.smithi047.stdout: During handling of the above exception, another exception occurred: 2022-05-14T04:29:12.803 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T04:29:12.803 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T04:29:12.803 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T04:29:12.803 INFO:journalctl@ceph.mon.b.smithi047.stdout: return f(*arg) 2022-05-14T04:29:12.804 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T04:29:12.804 INFO:journalctl@ceph.mon.b.smithi047.stdout: self._write_client_files(client_files, host) 2022-05-14T04:29:12.804 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T04:29:12.804 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T04:29:12.805 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T04:29:12.805 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T04:29:12.805 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T04:29:12.805 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.event_loop.get_result(coro) 2022-05-14T04:29:12.805 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T04:29:12.806 INFO:journalctl@ceph.mon.b.smithi047.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T04:29:12.806 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T04:29:12.806 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.__get_result() 2022-05-14T04:29:12.806 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T04:29:12.807 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise self._exception 2022-05-14T04:29:12.807 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T04:29:12.807 INFO:journalctl@ceph.mon.b.smithi047.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T04:29:12.807 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T04:29:12.807 INFO:journalctl@ceph.mon.b.smithi047.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T04:29:12.808 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T04:29:12.808 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise 2022-05-14T04:29:12.808 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T04:29:12.808 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.gen.throw(type, value, traceback) 2022-05-14T04:29:12.809 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T04:29:12.809 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OrchestratorError(msg) 2022-05-14T04:29:12.809 INFO:journalctl@ceph.mon.b.smithi047.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:29:12.810 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:29:12 smithi047 conmon[78521]: 2022-05-14T04:29:12.496+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:29:12.810 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:29:12 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T04:29:12.810 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:29:12 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:29:12.810 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:29:12 smithi047 conmon[78521]: yield 2022-05-14T04:29:12.811 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:29:12 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:29:12.811 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:29:12 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:29:12.811 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:29:12 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:29:12.811 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:29:12 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T04:29:12.812 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:29:12 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:29:12.812 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:29:12 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T04:29:12.812 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:29:12 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:29:12.812 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:29:12 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T04:29:12.812 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:29:12 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:29:12.813 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:29:12 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T04:29:12.813 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:29:12 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:29:12.813 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:29:12 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T04:29:12.813 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:29:12 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:29:12.814 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:29:12 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:29:12.814 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:29:12 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:29:12.814 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:29:12 smithi047 conmon[78521]: 2022-05-14T04:29:12.497+0000 7f533ff78700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T04:29:12.814 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:29:12 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T04:29:12.815 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:29:12 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:29:12.815 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:29:12 smithi047 conmon[78521]: yield 2022-05-14T04:29:12.815 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:29:12 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:29:12.815 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:29:12 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:29:12.815 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:29:12 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:29:12.816 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:29:12 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T04:29:12.816 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:29:12 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:29:12.816 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:29:12 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T04:29:12.816 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:29:12 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:29:12.816 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:29:12 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T04:29:12.817 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:29:12 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:29:12.817 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:29:12 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T04:29:12.817 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:29:12 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:29:12.817 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:29:12 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T04:29:12.818 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:29:12 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:29:12.818 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:29:12 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:29:12.818 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:29:12 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:29:12.818 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:29:12 smithi047 conmon[78521]: 2022-05-14T04:29:12.819 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:29:12 smithi047 conmon[78521]: During handling of the above exception, another exception occurred: 2022-05-14T04:29:12.819 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:29:12 smithi047 conmon[78521]: 2022-05-14T04:29:12.819 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:29:12 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T04:29:12.819 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:29:12 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T04:29:12.819 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:29:12 smithi047 conmon[78521]: return f(*arg) 2022-05-14T04:29:12.820 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:29:12 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T04:29:12.820 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:29:12 smithi047 conmon[78521]: self._write_client_files(client_files, host) 2022-05-14T04:29:12.820 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:29:12 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T04:29:12.820 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:29:12 smithi047 conmon[78521]: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T04:29:12.820 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:29:12 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T04:29:12.821 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:29:12 smithi047 conmon[78521]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T04:29:12.821 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:29:12 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T04:29:12.821 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:29:12 smithi047 conmon[78521]: return self.event_loop.get_result(coro) 2022-05-14T04:29:12.821 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:29:12 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T04:29:12.822 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:29:12 smithi047 conmon[78521]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T04:29:12.822 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:29:12 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T04:29:12.822 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:29:12 smithi047 conmon[78521]: return self.__get_result() 2022-05-14T04:29:12.823 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:29:12 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T04:29:12.823 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:29:12 smithi047 conmon[78521]: raise self._exception 2022-05-14T04:29:12.823 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:29:12 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T04:29:12.823 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:29:12 smithi047 conmon[78521]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T04:29:12.823 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:29:12 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T04:29:12.824 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:29:12 smithi047 conmon[78521]: conn = await self._remote_connection(host, addr) 2022-05-14T04:29:12.824 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:29:12 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T04:29:12.824 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:29:12 smithi047 conmon[78521]: raise 2022-05-14T04:29:12.824 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:29:12 smithi047 conmon[78521]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T04:29:12.825 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:29:12 smithi047 conmon[78521]: self.gen.throw(type, value, traceback) 2022-05-14T04:29:12.825 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:29:12 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T04:29:12.825 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:29:12 smithi047 conmon[78521]: raise OrchestratorError(msg) 2022-05-14T04:29:12.825 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:29:12 smithi047 conmon[78521]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:29:12.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:29:12 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:29:12.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:29:12 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 18085 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:29:12.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:29:12 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:29:12.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T04:29:12.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:29:12.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T04:29:12.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:29:12.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:29:12.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:29:12.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T04:29:12.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:29:12.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T04:29:12.863 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:29:12.863 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T04:29:12.863 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:29:12.864 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T04:29:12.864 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:29:12.864 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T04:29:12.864 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:29:12.865 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:29:12.865 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:29:12.865 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:29:12 smithi115 ceph-mon[78593]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T04:29:12.865 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T04:29:12.866 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:29:12.866 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T04:29:12.866 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:29:12.866 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:29:12.867 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:29:12.867 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T04:29:12.867 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:29:12.867 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T04:29:12.867 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:29:12.868 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T04:29:12.868 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:29:12.868 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T04:29:12.868 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:29:12.868 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T04:29:12.869 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:29:12.869 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:29:12.869 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:29:12.869 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T04:29:12.870 INFO:journalctl@ceph.mon.c.smithi115.stdout: During handling of the above exception, another exception occurred: 2022-05-14T04:29:12.870 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T04:29:12.870 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T04:29:12.870 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T04:29:12.871 INFO:journalctl@ceph.mon.c.smithi115.stdout: return f(*arg) 2022-05-14T04:29:12.871 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T04:29:12.871 INFO:journalctl@ceph.mon.c.smithi115.stdout: self._write_client_files(client_files, host) 2022-05-14T04:29:12.871 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T04:29:12.871 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T04:29:12.872 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T04:29:12.872 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T04:29:12.872 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T04:29:12.872 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.event_loop.get_result(coro) 2022-05-14T04:29:12.873 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T04:29:12.873 INFO:journalctl@ceph.mon.c.smithi115.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T04:29:12.873 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T04:29:12.873 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.__get_result() 2022-05-14T04:29:12.874 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T04:29:12.874 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise self._exception 2022-05-14T04:29:12.874 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T04:29:12.874 INFO:journalctl@ceph.mon.c.smithi115.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T04:29:12.874 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T04:29:12.875 INFO:journalctl@ceph.mon.c.smithi115.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T04:29:12.875 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T04:29:12.875 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise 2022-05-14T04:29:12.875 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T04:29:12.876 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.gen.throw(type, value, traceback) 2022-05-14T04:29:12.876 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T04:29:12.876 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OrchestratorError(msg) 2022-05-14T04:29:12.876 INFO:journalctl@ceph.mon.c.smithi115.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:29:13.576 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:29:13 smithi115 conmon[90393]: 2022-05-14T04:29:13.203+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:29:13.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:29:13 smithi115 ceph-mon[78593]: pgmap v9394: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:29:13.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:29:13 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:29:14.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:29:13 smithi047 ceph-mon[77147]: pgmap v9394: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:29:14.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:29:13 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:29:14.577 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:29:14 smithi115 conmon[90393]: 2022-05-14T04:29:14.225+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:29:14.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:29:14 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:29:14.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:29:14 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:29:14.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:29:14 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:29:15.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:29:14 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:29:15.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:29:14 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:29:15.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:29:14 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:29:15.584 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:29:15 smithi115 conmon[90393]: 2022-05-14T04:29:15.254+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:29:15.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:29:15 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T04:29:15.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:29:15 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T04:29:15.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:29:15 smithi115 ceph-mon[78593]: pgmap v9395: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:29:15.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:29:15 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:29:16.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:29:15 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T04:29:16.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:29:15 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T04:29:16.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:29:15 smithi047 ceph-mon[77147]: pgmap v9395: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:29:16.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:29:15 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:29:16.584 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:29:16 smithi115 conmon[90393]: 2022-05-14T04:29:16.283+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:29:16.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:29:16 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:29:17.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:29:16 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:29:17.589 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:29:17 smithi115 conmon[90393]: 2022-05-14T04:29:17.314+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:29:17.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:29:17 smithi115 ceph-mon[78593]: pgmap v9396: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:29:17.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:29:17 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:29:17.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:29:17 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 18090 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:29:18.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:29:17 smithi047 ceph-mon[77147]: pgmap v9396: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:29:18.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:29:17 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:29:18.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:29:17 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 18090 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:29:18.590 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:29:18 smithi115 conmon[90393]: 2022-05-14T04:29:18.282+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:29:18.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:29:18 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:29:18.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:29:18 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:29:18.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:29:18 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:29:19.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:29:18 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:29:19.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:29:18 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:29:19.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:29:18 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:29:19.595 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:29:19 smithi115 conmon[90393]: 2022-05-14T04:29:19.239+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:29:19.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:29:19 smithi115 ceph-mon[78593]: pgmap v9397: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:29:19.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:29:19 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:29:20.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:29:19 smithi047 ceph-mon[77147]: pgmap v9397: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:29:20.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:29:19 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:29:20.597 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:29:20 smithi115 conmon[90393]: 2022-05-14T04:29:20.283+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:29:20.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:29:20 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:29:21.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:29:20 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:29:21.602 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:29:21 smithi115 conmon[90393]: 2022-05-14T04:29:21.326+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:29:21.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:29:21 smithi115 ceph-mon[78593]: pgmap v9398: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:29:21.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:29:21 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:29:22.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:29:21 smithi047 ceph-mon[77147]: pgmap v9398: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:29:22.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:29:21 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:29:22.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:29:22 smithi115 conmon[90393]: 2022-05-14T04:29:22.376+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:29:22.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:29:22 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:29:22.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:29:22 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:29:22.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:29:22 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:29:22.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:29:22 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 18095 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:29:23.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:29:22 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:29:23.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:29:22 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:29:23.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:29:22 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:29:23.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:29:22 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 18095 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:29:23.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:29:23 smithi115 conmon[90393]: 2022-05-14T04:29:23.343+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:29:23.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:29:23 smithi115 ceph-mon[78593]: pgmap v9399: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:29:23.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:29:23 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:29:24.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:29:23 smithi047 ceph-mon[77147]: pgmap v9399: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:29:24.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:29:23 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:29:24.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:29:24 smithi115 conmon[90393]: 2022-05-14T04:29:24.314+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:29:25.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:29:24 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:29:25.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:29:24 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:29:25.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:29:25 smithi115 conmon[90393]: 2022-05-14T04:29:25.294+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:29:26.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:29:25 smithi047 ceph-mon[77147]: pgmap v9400: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:29:26.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:29:25 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:29:26.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:29:25 smithi115 ceph-mon[78593]: pgmap v9400: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:29:26.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:29:25 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:29:26.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:29:26 smithi115 conmon[90393]: 2022-05-14T04:29:26.295+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:29:27.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:29:26 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:29:27.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:29:26 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:29:27.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:29:27 smithi115 conmon[90393]: 2022-05-14T04:29:27.248+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:29:28.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:29:27 smithi047 ceph-mon[77147]: pgmap v9401: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:29:28.112 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:29:27 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:29:28.113 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:29:27 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 18100 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:29:28.114 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:29:27 smithi115 ceph-mon[78593]: pgmap v9401: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:29:28.114 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:29:27 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:29:28.114 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:29:27 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 18100 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:29:28.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:29:28 smithi115 conmon[90393]: 2022-05-14T04:29:28.232+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:29:29.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:29:28 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:29:29.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:29:28 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:29:29.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:29:29 smithi115 conmon[90393]: 2022-05-14T04:29:29.272+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:29:30.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:29:29 smithi047 ceph-mon[77147]: pgmap v9402: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:29:30.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:29:29 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:29:30.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:29:29 smithi115 ceph-mon[78593]: pgmap v9402: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:29:30.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:29:29 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:29:30.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:29:30 smithi115 conmon[90393]: 2022-05-14T04:29:30.276+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:29:31.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:29:30 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:29:31.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:29:30 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:29:31.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:29:30 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:29:31.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:29:30 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:29:31.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:29:30 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:29:31.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:29:30 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:29:31.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:29:31 smithi115 conmon[90393]: 2022-05-14T04:29:31.309+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:29:32.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:29:31 smithi047 ceph-mon[77147]: pgmap v9403: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:29:32.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:29:31 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:29:32.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:29:31 smithi115 ceph-mon[78593]: pgmap v9403: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:29:32.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:29:31 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:29:32.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:29:32 smithi115 conmon[90393]: 2022-05-14T04:29:32.281+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:29:33.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:29:32 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:29:33.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:29:32 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 18105 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:29:33.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:29:32 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:29:33.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:29:32 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 18105 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:29:33.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:29:33 smithi115 conmon[90393]: 2022-05-14T04:29:33.265+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:29:34.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:29:33 smithi047 ceph-mon[77147]: pgmap v9404: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:29:34.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:29:33 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:29:34.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:29:33 smithi115 ceph-mon[78593]: pgmap v9404: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:29:34.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:29:33 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:29:34.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:29:34 smithi115 conmon[90393]: 2022-05-14T04:29:34.248+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:29:35.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:29:34 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:29:35.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:29:34 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:29:35.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:29:34 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:29:35.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:29:34 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:29:35.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:29:34 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:29:35.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:29:34 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:29:35.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:29:35 smithi115 conmon[90393]: 2022-05-14T04:29:35.276+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:29:36.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:29:35 smithi047 ceph-mon[77147]: pgmap v9405: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:29:36.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:29:35 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:29:36.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:29:35 smithi115 ceph-mon[78593]: pgmap v9405: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:29:36.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:29:35 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:29:36.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:29:36 smithi115 conmon[90393]: 2022-05-14T04:29:36.275+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:29:37.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:29:36 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:29:37.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:29:36 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:29:37.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:29:37 smithi115 conmon[90393]: 2022-05-14T04:29:37.249+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:29:38.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:29:37 smithi047 ceph-mon[77147]: pgmap v9406: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:29:38.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:29:37 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:29:38.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:29:37 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 18110 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:29:38.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:29:37 smithi115 ceph-mon[78593]: pgmap v9406: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:29:38.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:29:37 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:29:38.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:29:37 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 18110 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:29:38.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:29:38 smithi115 conmon[90393]: 2022-05-14T04:29:38.271+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:29:39.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:29:38 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:29:39.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:29:38 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:29:39.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:29:38 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:29:39.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:29:38 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:29:39.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:29:38 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:29:39.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:29:38 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:29:39.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:29:39 smithi115 conmon[90393]: 2022-05-14T04:29:39.253+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:29:40.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:29:39 smithi047 ceph-mon[77147]: pgmap v9407: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:29:40.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:29:39 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:29:40.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:29:39 smithi115 ceph-mon[78593]: pgmap v9407: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:29:40.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:29:39 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:29:40.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:29:40 smithi115 conmon[90393]: 2022-05-14T04:29:40.256+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:29:41.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:29:40 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:29:41.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:29:40 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:29:41.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:29:41 smithi115 conmon[90393]: 2022-05-14T04:29:41.212+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:29:42.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:29:41 smithi047 ceph-mon[77147]: pgmap v9408: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:29:42.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:29:41 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:29:42.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:29:41 smithi115 ceph-mon[78593]: pgmap v9408: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:29:42.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:29:41 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:29:42.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:29:42 smithi115 conmon[90393]: 2022-05-14T04:29:42.185+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:29:43.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:29:42 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:29:43.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:29:42 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:29:43.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:29:42 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:29:43.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:29:42 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 18115 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:29:43.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:29:42 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:29:43.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:29:42 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:29:43.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:29:42 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:29:43.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:29:42 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 18115 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:29:43.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:29:43 smithi115 conmon[90393]: 2022-05-14T04:29:43.179+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:29:44.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:29:43 smithi047 ceph-mon[77147]: pgmap v9409: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:29:44.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:29:43 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:29:44.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:29:43 smithi115 ceph-mon[78593]: pgmap v9409: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:29:44.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:29:43 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:29:44.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:29:44 smithi115 conmon[90393]: 2022-05-14T04:29:44.222+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:29:45.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:29:44 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:29:45.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:29:44 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:29:45.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:29:45 smithi115 conmon[90393]: 2022-05-14T04:29:45.189+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:29:46.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:29:45 smithi047 ceph-mon[77147]: pgmap v9410: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:29:46.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:29:45 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:29:46.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:29:45 smithi115 ceph-mon[78593]: pgmap v9410: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:29:46.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:29:45 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:29:46.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:29:46 smithi115 conmon[90393]: 2022-05-14T04:29:46.178+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:29:47.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:29:46 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:29:47.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:29:46 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:29:47.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:29:47 smithi115 conmon[90393]: 2022-05-14T04:29:47.130+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:29:48.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:29:47 smithi047 ceph-mon[77147]: pgmap v9411: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:29:48.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:29:47 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:29:48.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:29:47 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 18120 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:29:48.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:29:47 smithi115 ceph-mon[78593]: pgmap v9411: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:29:48.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:29:47 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:29:48.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:29:47 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 18120 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:29:48.610 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:29:48 smithi115 conmon[90393]: 2022-05-14T04:29:48.151+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:29:49.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:29:48 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:29:49.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:29:48 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:29:49.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:29:49 smithi115 conmon[90393]: 2022-05-14T04:29:49.105+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:29:50.045 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:29:49 smithi047 ceph-mon[77147]: pgmap v9412: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:29:50.046 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:29:49 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:29:50.088 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:29:49 smithi115 ceph-mon[78593]: pgmap v9412: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:29:50.089 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:29:49 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:29:50.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:29:50 smithi115 conmon[90393]: 2022-05-14T04:29:50.086+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:29:51.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:29:50 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:29:51.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:29:50 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:29:51.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:29:50 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:29:51.085 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:29:50 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:29:51.085 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:29:50 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:29:51.086 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:29:50 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:29:51.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:29:51 smithi115 conmon[90393]: 2022-05-14T04:29:51.082+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:29:52.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:29:51 smithi047 ceph-mon[77147]: pgmap v9413: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:29:52.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:29:51 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:29:52.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:29:51 smithi115 ceph-mon[78593]: pgmap v9413: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:29:52.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:29:51 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:29:52.612 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:29:52 smithi115 conmon[90393]: 2022-05-14T04:29:52.132+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:29:53.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:29:52 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:29:53.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:29:52 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 18126 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:29:53.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:29:52 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:29:53.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:29:52 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 18126 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:29:53.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:29:53 smithi115 conmon[90393]: 2022-05-14T04:29:53.162+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:29:54.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:29:53 smithi047 ceph-mon[77147]: pgmap v9414: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:29:54.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:29:53 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:29:54.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:29:53 smithi115 ceph-mon[78593]: pgmap v9414: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:29:54.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:29:53 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:29:54.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:29:54 smithi115 conmon[90393]: 2022-05-14T04:29:54.182+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:29:55.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:29:54 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:29:55.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:29:54 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:29:55.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:29:54 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:29:55.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:29:54 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:29:55.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:29:54 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:29:55.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:29:54 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:29:55.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:29:55 smithi115 conmon[90393]: 2022-05-14T04:29:55.152+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:29:56.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:29:55 smithi047 ceph-mon[77147]: pgmap v9415: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:29:56.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:29:55 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:29:56.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:29:55 smithi115 ceph-mon[78593]: pgmap v9415: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:29:56.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:29:55 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:29:56.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:29:56 smithi115 conmon[90393]: 2022-05-14T04:29:56.148+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:29:57.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:29:56 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:29:57.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:29:56 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:29:57.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:29:57 smithi115 conmon[90393]: 2022-05-14T04:29:57.170+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:29:58.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:29:57 smithi047 ceph-mon[77147]: pgmap v9416: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:29:58.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:29:57 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:29:58.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:29:57 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 18130 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:29:58.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:29:57 smithi115 ceph-mon[78593]: pgmap v9416: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:29:58.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:29:57 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:29:58.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:29:57 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 18130 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:29:58.610 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:29:58 smithi115 conmon[90393]: 2022-05-14T04:29:58.128+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:29:59.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:29:58 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:29:59.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:29:58 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:29:59.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:29:58 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:29:59.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:29:58 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:29:59.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:29:58 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:29:59.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:29:58 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:29:59.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:29:59 smithi115 conmon[90393]: 2022-05-14T04:29:59.166+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:30:00.002 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:29:59 smithi047 ceph-mon[77147]: pgmap v9417: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:30:00.003 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:29:59 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:30:00.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:29:59 smithi115 ceph-mon[78593]: pgmap v9417: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:30:00.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:29:59 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:30:00.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:30:00 smithi047 conmon[77123]: 2022-05-14T04:29:59.999+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : Health detail: HEALTH_ERR 1 filesystem is degraded; 1 mds daemon damaged; 1 MDSs are read only; 1 MDSs report slow metadata IOs; 1/3 mons down, quorum b,c; Reduced data availability: 1 pg inactive; 9 slow ops, oldest one blocked for 18130 sec, osd.5 has slow ops 2022-05-14T04:30:00.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:30:00 smithi047 conmon[77123]: 2022-05-14T04:29:59.999+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : [WRN] FS_DEGRADED: 1 filesystem is degraded 2022-05-14T04:30:00.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:30:00 smithi047 conmon[77123]: 2022-05-14T04:29:59.999+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : fs cephfs is degraded 2022-05-14T04:30:00.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:30:00 smithi047 conmon[77123]: 2022-05-14T04:29:59.999+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : [ERR] MDS_DAMAGE: 1 mds daemon damaged 2022-05-14T04:30:00.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:30:00 smithi047 conmon[77123]: 2022-05-14T04:29:59.999+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : fs cephfs mds.2 is damaged 2022-05-14T04:30:00.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:30:00 smithi047 conmon[77123]: 2022-05-14T04:30:00.000+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : [WRN] MDS_READ_ONLY: 1 MDSs are read only 2022-05-14T04:30:00.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:30:00 smithi047 conmon[77123]: 2022-05-14T04:30:00.000+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : mds.l(mds.0): MDS in read-only mode 2022-05-14T04:30:00.295 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:30:00 smithi047 conmon[77123]: 2022-05-14T04:30:00.000+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : [WRN] MDS_SLOW_METADATA_IO: 1 MDSs report slow metadata IOs 2022-05-14T04:30:00.295 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:30:00 smithi047 conmon[77123]: 2022-05-14T04:30:00.000+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : mds.l(mds.0): 4 slow metadata IOs are blocked > 180 secs, oldest blocked for 18433 secs 2022-05-14T04:30:00.295 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:30:00 smithi047 conmon[77123]: 2022-05-14T04:30:00.000+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : [WRN] MON_DOWN: 1/3 mons down, quorum b,c 2022-05-14T04:30:00.296 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:30:00 smithi047 conmon[77123]: 2022-05-14T04:30:00.000+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : mon.a (rank 0) addr [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] is down (out of quorum) 2022-05-14T04:30:00.296 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:30:00 smithi047 conmon[77123]: 2022-05-14T04:30:00.000+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : [WRN] PG_AVAILABILITY: Reduced data availability: 1 pg inactive 2022-05-14T04:30:00.296 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:30:00 smithi047 conmon[77123]: 2022-05-14T04:30:00.000+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : pg 2.1 is stuck inactive for 5h, current state unknown, last acting [] 2022-05-14T04:30:00.297 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:30:00 smithi047 conmon[77123]: 2022-05-14T04:30:00.000+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : [WRN] SLOW_OPS: 9 slow ops, oldest one blocked for 18130 sec, osd.5 has slow ops 2022-05-14T04:30:00.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:30:00 smithi115 conmon[90393]: 2022-05-14T04:30:00.192+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:30:01.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:30:00 smithi047 ceph-mon[77147]: Health detail: HEALTH_ERR 1 filesystem is degraded; 1 mds daemon damaged; 1 MDSs are read only; 1 MDSs report slow metadata IOs; 1/3 mons down, quorum b,c; Reduced data availability: 1 pg inactive; 9 slow ops, oldest one blocked for 18130 sec, osd.5 has slow ops 2022-05-14T04:30:01.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:30:00 smithi047 ceph-mon[77147]: [WRN] FS_DEGRADED: 1 filesystem is degraded 2022-05-14T04:30:01.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:30:00 smithi047 ceph-mon[77147]: fs cephfs is degraded 2022-05-14T04:30:01.045 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:30:00 smithi047 ceph-mon[77147]: [ERR] MDS_DAMAGE: 1 mds daemon damaged 2022-05-14T04:30:01.045 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:30:00 smithi047 ceph-mon[77147]: fs cephfs mds.2 is damaged 2022-05-14T04:30:01.045 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:30:00 smithi047 ceph-mon[77147]: [WRN] MDS_READ_ONLY: 1 MDSs are read only 2022-05-14T04:30:01.045 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:30:00 smithi047 ceph-mon[77147]: mds.l(mds.0): MDS in read-only mode 2022-05-14T04:30:01.045 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:30:00 smithi047 ceph-mon[77147]: [WRN] MDS_SLOW_METADATA_IO: 1 MDSs report slow metadata IOs 2022-05-14T04:30:01.046 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:30:00 smithi047 ceph-mon[77147]: mds.l(mds.0): 4 slow metadata IOs are blocked > 180 secs, oldest blocked for 18433 secs 2022-05-14T04:30:01.046 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:30:00 smithi047 ceph-mon[77147]: [WRN] MON_DOWN: 1/3 mons down, quorum b,c 2022-05-14T04:30:01.046 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:30:00 smithi047 ceph-mon[77147]: mon.a (rank 0) addr [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] is down (out of quorum) 2022-05-14T04:30:01.047 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:30:00 smithi047 ceph-mon[77147]: [WRN] PG_AVAILABILITY: Reduced data availability: 1 pg inactive 2022-05-14T04:30:01.047 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:30:00 smithi047 ceph-mon[77147]: pg 2.1 is stuck inactive for 5h, current state unknown, last acting [] 2022-05-14T04:30:01.047 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:30:00 smithi047 ceph-mon[77147]: [WRN] SLOW_OPS: 9 slow ops, oldest one blocked for 18130 sec, osd.5 has slow ops 2022-05-14T04:30:01.047 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:30:00 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:30:01.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:30:00 smithi115 ceph-mon[78593]: Health detail: HEALTH_ERR 1 filesystem is degraded; 1 mds daemon damaged; 1 MDSs are read only; 1 MDSs report slow metadata IOs; 1/3 mons down, quorum b,c; Reduced data availability: 1 pg inactive; 9 slow ops, oldest one blocked for 18130 sec, osd.5 has slow ops 2022-05-14T04:30:01.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:30:00 smithi115 ceph-mon[78593]: [WRN] FS_DEGRADED: 1 filesystem is degraded 2022-05-14T04:30:01.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:30:00 smithi115 ceph-mon[78593]: fs cephfs is degraded 2022-05-14T04:30:01.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:30:00 smithi115 ceph-mon[78593]: [ERR] MDS_DAMAGE: 1 mds daemon damaged 2022-05-14T04:30:01.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:30:00 smithi115 ceph-mon[78593]: fs cephfs mds.2 is damaged 2022-05-14T04:30:01.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:30:00 smithi115 ceph-mon[78593]: [WRN] MDS_READ_ONLY: 1 MDSs are read only 2022-05-14T04:30:01.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:30:00 smithi115 ceph-mon[78593]: mds.l(mds.0): MDS in read-only mode 2022-05-14T04:30:01.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:30:00 smithi115 ceph-mon[78593]: [WRN] MDS_SLOW_METADATA_IO: 1 MDSs report slow metadata IOs 2022-05-14T04:30:01.112 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:30:00 smithi115 ceph-mon[78593]: mds.l(mds.0): 4 slow metadata IOs are blocked > 180 secs, oldest blocked for 18433 secs 2022-05-14T04:30:01.112 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:30:00 smithi115 ceph-mon[78593]: [WRN] MON_DOWN: 1/3 mons down, quorum b,c 2022-05-14T04:30:01.112 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:30:00 smithi115 ceph-mon[78593]: mon.a (rank 0) addr [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] is down (out of quorum) 2022-05-14T04:30:01.112 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:30:00 smithi115 ceph-mon[78593]: [WRN] PG_AVAILABILITY: Reduced data availability: 1 pg inactive 2022-05-14T04:30:01.112 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:30:00 smithi115 ceph-mon[78593]: pg 2.1 is stuck inactive for 5h, current state unknown, last acting [] 2022-05-14T04:30:01.113 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:30:00 smithi115 ceph-mon[78593]: [WRN] SLOW_OPS: 9 slow ops, oldest one blocked for 18130 sec, osd.5 has slow ops 2022-05-14T04:30:01.113 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:30:00 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:30:01.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:30:01 smithi115 conmon[90393]: 2022-05-14T04:30:01.239+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:30:02.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:30:01 smithi047 ceph-mon[77147]: pgmap v9418: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:30:02.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:30:01 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:30:02.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:30:01 smithi115 ceph-mon[78593]: pgmap v9418: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:30:02.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:30:01 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:30:02.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:30:02 smithi115 conmon[90393]: 2022-05-14T04:30:02.272+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:30:03.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:30:02 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:30:03.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:30:02 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:30:03.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:30:02 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:30:03.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:30:02 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 18135 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:30:03.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:30:02 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:30:03.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:30:02 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:30:03.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:30:02 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:30:03.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:30:02 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 18135 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:30:03.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:30:03 smithi115 conmon[90393]: 2022-05-14T04:30:03.300+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:30:04.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:30:03 smithi047 ceph-mon[77147]: pgmap v9419: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:30:04.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:30:03 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:30:04.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:30:03 smithi115 ceph-mon[78593]: pgmap v9419: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:30:04.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:30:03 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:30:04.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:30:04 smithi115 conmon[90393]: 2022-05-14T04:30:04.339+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:30:05.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:30:04 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:30:05.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:30:04 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:30:05.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:30:05 smithi115 conmon[90393]: 2022-05-14T04:30:05.329+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:30:06.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:30:05 smithi047 ceph-mon[77147]: pgmap v9420: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:30:06.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:30:05 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:30:06.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:30:05 smithi115 ceph-mon[78593]: pgmap v9420: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:30:06.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:30:05 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:30:06.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:30:06 smithi115 conmon[90393]: 2022-05-14T04:30:06.306+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:30:07.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:30:06 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:30:07.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:30:06 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:30:07.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:30:07 smithi115 conmon[90393]: 2022-05-14T04:30:07.350+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:30:08.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:30:07 smithi047 ceph-mon[77147]: pgmap v9421: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:30:08.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:30:07 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:30:08.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:30:07 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 18140 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:30:08.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:30:07 smithi115 ceph-mon[78593]: pgmap v9421: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:30:08.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:30:07 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:30:08.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:30:07 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 18140 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:30:08.755 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:30:08 smithi115 conmon[90393]: 2022-05-14T04:30:08.380+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:30:09.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:30:08 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:30:09.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:30:08 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:30:09.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:30:09 smithi115 conmon[90393]: 2022-05-14T04:30:09.354+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:30:10.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:30:09 smithi047 ceph-mon[77147]: pgmap v9422: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:30:10.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:30:09 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:30:10.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:30:09 smithi115 ceph-mon[78593]: pgmap v9422: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:30:10.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:30:09 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:30:10.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:30:10 smithi115 conmon[90393]: 2022-05-14T04:30:10.312+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:30:11.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:30:10 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:30:11.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:30:10 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:30:11.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:30:10 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:30:11.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:30:10 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:30:11.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:30:10 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:30:11.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:30:10 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:30:11.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:30:11 smithi115 conmon[90393]: 2022-05-14T04:30:11.272+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:30:12.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:30:11 smithi047 ceph-mon[77147]: pgmap v9423: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:30:12.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:30:11 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:30:12.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:30:11 smithi115 ceph-mon[78593]: pgmap v9423: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:30:12.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:30:11 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:30:12.586 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:30:12 smithi115 conmon[90393]: 2022-05-14T04:30:12.313+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:30:12.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:30:12 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:30:12.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:30:12 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 18145 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:30:12.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:30:12 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T04:30:12.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:30:12 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T04:30:12.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:30:12 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T04:30:12.979 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:30:12 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:30:12.980 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:30:12 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 18145 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:30:12.980 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:30:12 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T04:30:12.980 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:30:12 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T04:30:12.980 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:30:12 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T04:30:13.293 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:30:12 smithi047 conmon[78521]: 2022-05-14T04:30:12.976+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:30:13.293 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:30:12 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T04:30:13.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:30:12 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:30:13.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:30:12 smithi047 conmon[78521]: yield 2022-05-14T04:30:13.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:30:12 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:30:13.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:30:12 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:30:13.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:30:12 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:30:13.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:30:12 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T04:30:13.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:30:12 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:30:13.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:30:12 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T04:30:13.297 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:30:12 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:30:13.297 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:30:12 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T04:30:13.298 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:30:12 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:30:13.298 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:30:12 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T04:30:13.298 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:30:12 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:30:13.299 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:30:12 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T04:30:13.299 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:30:12 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:30:13.300 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:30:12 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:30:13.300 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:30:12 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:30:13.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:30:13 smithi115 conmon[90393]: 2022-05-14T04:30:13.348+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:30:14.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:30:13 smithi047 ceph-mon[77147]: pgmap v9424: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:30:14.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:30:13 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:30:14.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T04:30:14.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:30:14.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T04:30:14.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:30:14.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:30:14.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:30:14.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T04:30:14.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:30:14.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T04:30:14.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:30:14.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T04:30:14.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:30:14.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T04:30:14.049 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:30:14.049 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T04:30:14.050 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:30:14.050 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:30:14.051 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:30:14.051 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:30:13 smithi047 ceph-mon[77147]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T04:30:14.052 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:30:13 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:30:14.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:30:13 smithi115 ceph-mon[78593]: pgmap v9424: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:30:14.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:30:13 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:30:14.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T04:30:14.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:30:14.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T04:30:14.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:30:14.113 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:30:14.113 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:30:14.113 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T04:30:14.114 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:30:14.114 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T04:30:14.114 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:30:14.115 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T04:30:14.115 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:30:14.115 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T04:30:14.116 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:30:14.116 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T04:30:14.116 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:30:14.117 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:30:14.117 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:30:14.117 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:30:13 smithi115 ceph-mon[78593]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T04:30:14.119 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:30:13 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:30:14.776 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:30:14 smithi115 conmon[90393]: 2022-05-14T04:30:14.394+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:30:15.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:30:14 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:30:15.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:30:14 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:30:15.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:30:14 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:30:15.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:30:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T04:30:15.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:30:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T04:30:15.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:30:14 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:30:15.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:30:14 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:30:15.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:30:14 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:30:15.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:30:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T04:30:15.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:30:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T04:30:15.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:30:15 smithi115 conmon[90393]: 2022-05-14T04:30:15.352+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:30:16.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:30:15 smithi047 ceph-mon[77147]: pgmap v9425: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:30:16.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:30:15 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:30:16.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:30:15 smithi115 ceph-mon[78593]: pgmap v9425: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:30:16.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:30:15 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:30:16.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:30:16 smithi047 conmon[78521]: 2022-05-14T04:30:16.047+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:30:16.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:30:16 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T04:30:16.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:30:16 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:30:16.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:30:16 smithi047 conmon[78521]: yield 2022-05-14T04:30:16.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:30:16 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:30:16.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:30:16 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:30:16.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:30:16 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:30:16.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:30:16 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T04:30:16.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:30:16 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:30:16.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:30:16 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T04:30:16.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:30:16 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:30:16.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:30:16 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T04:30:16.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:30:16 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:30:16.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:30:16 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T04:30:16.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:30:16 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:30:16.547 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:30:16 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T04:30:16.547 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:30:16 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:30:16.547 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:30:16 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:30:16.547 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:30:16 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:30:16.547 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:30:16 smithi047 conmon[78521]: 2022-05-14T04:30:16.048+0000 7f5344f82700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T04:30:16.548 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:30:16 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T04:30:16.548 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:30:16 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:30:16.548 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:30:16 smithi047 conmon[78521]: yield 2022-05-14T04:30:16.548 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:30:16 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:30:16.548 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:30:16 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:30:16.549 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:30:16 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:30:16.549 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:30:16 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T04:30:16.549 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:30:16 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:30:16.549 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:30:16 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T04:30:16.549 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:30:16 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:30:16.550 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:30:16 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T04:30:16.550 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:30:16 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:30:16.550 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:30:16 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T04:30:16.550 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:30:16 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:30:16.551 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:30:16 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T04:30:16.551 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:30:16 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:30:16.551 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:30:16 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:30:16.551 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:30:16 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:30:16.551 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:30:16 smithi047 conmon[78521]: 2022-05-14T04:30:16.552 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:30:16 smithi047 conmon[78521]: During handling of the above exception, another exception occurred: 2022-05-14T04:30:16.552 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:30:16 smithi047 conmon[78521]: 2022-05-14T04:30:16.552 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:30:16 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T04:30:16.553 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:30:16 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T04:30:16.553 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:30:16 smithi047 conmon[78521]: return f(*arg) 2022-05-14T04:30:16.553 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:30:16 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T04:30:16.553 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:30:16 smithi047 conmon[78521]: self._write_client_files(client_files, host) 2022-05-14T04:30:16.554 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:30:16 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T04:30:16.554 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:30:16 smithi047 conmon[78521]: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T04:30:16.554 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:30:16 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T04:30:16.554 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:30:16 smithi047 conmon[78521]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T04:30:16.554 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:30:16 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T04:30:16.555 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:30:16 smithi047 conmon[78521]: return self.event_loop.get_result(coro) 2022-05-14T04:30:16.555 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:30:16 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T04:30:16.555 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:30:16 smithi047 conmon[78521]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T04:30:16.555 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:30:16 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T04:30:16.556 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:30:16 smithi047 conmon[78521]: return self.__get_result() 2022-05-14T04:30:16.556 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:30:16 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T04:30:16.556 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:30:16 smithi047 conmon[78521]: raise self._exception 2022-05-14T04:30:16.556 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:30:16 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T04:30:16.556 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:30:16 smithi047 conmon[78521]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T04:30:16.557 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:30:16 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T04:30:16.557 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:30:16 smithi047 conmon[78521]: conn = await self._remote_connection(host, addr) 2022-05-14T04:30:16.557 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:30:16 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T04:30:16.557 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:30:16 smithi047 conmon[78521]: raise 2022-05-14T04:30:16.557 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:30:16 smithi047 conmon[78521]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T04:30:16.558 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:30:16 smithi047 conmon[78521]: self.gen.throw(type, value, traceback) 2022-05-14T04:30:16.558 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:30:16 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T04:30:16.558 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:30:16 smithi047 conmon[78521]: raise OrchestratorError(msg) 2022-05-14T04:30:16.558 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:30:16 smithi047 conmon[78521]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:30:16.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:30:16 smithi115 conmon[90393]: 2022-05-14T04:30:16.319+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:30:17.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:30:16 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:30:17.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T04:30:17.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:30:17.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T04:30:17.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:30:17.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:30:17.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:30:17.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T04:30:17.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:30:17.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T04:30:17.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:30:17.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T04:30:17.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:30:17.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T04:30:17.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:30:17.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T04:30:17.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:30:17.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:30:17.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:30:17.048 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:30:16 smithi047 ceph-mon[77147]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T04:30:17.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T04:30:17.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:30:17.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T04:30:17.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:30:17.049 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:30:17.049 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:30:17.049 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T04:30:17.049 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:30:17.049 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T04:30:17.050 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:30:17.050 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T04:30:17.050 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:30:17.050 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T04:30:17.050 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:30:17.051 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T04:30:17.051 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:30:17.051 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:30:17.051 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:30:17.051 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T04:30:17.052 INFO:journalctl@ceph.mon.b.smithi047.stdout: During handling of the above exception, another exception occurred: 2022-05-14T04:30:17.052 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T04:30:17.052 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T04:30:17.052 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T04:30:17.052 INFO:journalctl@ceph.mon.b.smithi047.stdout: return f(*arg) 2022-05-14T04:30:17.053 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T04:30:17.053 INFO:journalctl@ceph.mon.b.smithi047.stdout: self._write_client_files(client_files, host) 2022-05-14T04:30:17.053 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T04:30:17.054 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T04:30:17.054 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T04:30:17.054 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T04:30:17.054 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T04:30:17.055 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.event_loop.get_result(coro) 2022-05-14T04:30:17.055 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T04:30:17.055 INFO:journalctl@ceph.mon.b.smithi047.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T04:30:17.055 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T04:30:17.055 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.__get_result() 2022-05-14T04:30:17.056 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T04:30:17.056 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise self._exception 2022-05-14T04:30:17.056 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T04:30:17.056 INFO:journalctl@ceph.mon.b.smithi047.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T04:30:17.056 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T04:30:17.057 INFO:journalctl@ceph.mon.b.smithi047.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T04:30:17.057 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T04:30:17.057 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise 2022-05-14T04:30:17.057 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T04:30:17.058 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.gen.throw(type, value, traceback) 2022-05-14T04:30:17.058 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T04:30:17.058 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OrchestratorError(msg) 2022-05-14T04:30:17.058 INFO:journalctl@ceph.mon.b.smithi047.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:30:17.058 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:30:16 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:30:17.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:30:16 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:30:17.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T04:30:17.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:30:17.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T04:30:17.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:30:17.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:30:17.113 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:30:17.113 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T04:30:17.115 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:30:17.115 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T04:30:17.115 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:30:17.116 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T04:30:17.116 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:30:17.116 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T04:30:17.117 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:30:17.117 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T04:30:17.117 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:30:17.117 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:30:17.117 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:30:17.118 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:30:16 smithi115 ceph-mon[78593]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T04:30:17.118 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T04:30:17.118 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:30:17.118 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T04:30:17.119 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:30:17.119 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:30:17.119 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:30:17.119 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T04:30:17.119 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:30:17.120 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T04:30:17.120 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:30:17.120 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T04:30:17.120 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:30:17.121 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T04:30:17.121 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:30:17.121 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T04:30:17.121 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:30:17.121 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:30:17.122 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:30:17.122 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T04:30:17.122 INFO:journalctl@ceph.mon.c.smithi115.stdout: During handling of the above exception, another exception occurred: 2022-05-14T04:30:17.122 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T04:30:17.123 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T04:30:17.123 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T04:30:17.123 INFO:journalctl@ceph.mon.c.smithi115.stdout: return f(*arg) 2022-05-14T04:30:17.124 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T04:30:17.124 INFO:journalctl@ceph.mon.c.smithi115.stdout: self._write_client_files(client_files, host) 2022-05-14T04:30:17.124 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T04:30:17.124 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T04:30:17.125 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T04:30:17.125 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T04:30:17.125 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T04:30:17.125 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.event_loop.get_result(coro) 2022-05-14T04:30:17.125 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T04:30:17.126 INFO:journalctl@ceph.mon.c.smithi115.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T04:30:17.126 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T04:30:17.126 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.__get_result() 2022-05-14T04:30:17.126 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T04:30:17.126 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise self._exception 2022-05-14T04:30:17.127 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T04:30:17.127 INFO:journalctl@ceph.mon.c.smithi115.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T04:30:17.127 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T04:30:17.127 INFO:journalctl@ceph.mon.c.smithi115.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T04:30:17.128 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T04:30:17.128 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise 2022-05-14T04:30:17.128 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T04:30:17.129 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.gen.throw(type, value, traceback) 2022-05-14T04:30:17.130 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T04:30:17.130 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OrchestratorError(msg) 2022-05-14T04:30:17.130 INFO:journalctl@ceph.mon.c.smithi115.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:30:17.131 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:30:16 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:30:17.610 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:30:17 smithi115 conmon[90393]: 2022-05-14T04:30:17.327+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:30:18.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:30:17 smithi047 ceph-mon[77147]: pgmap v9426: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:30:18.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:30:17 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:30:18.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:30:17 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 18150 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:30:18.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:30:17 smithi115 ceph-mon[78593]: pgmap v9426: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:30:18.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:30:17 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:30:18.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:30:17 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 18150 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:30:18.794 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:30:18 smithi115 conmon[90393]: 2022-05-14T04:30:18.363+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:30:19.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:30:18 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:30:19.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:30:18 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:30:19.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:30:18 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:30:19.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:30:18 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:30:19.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:30:18 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:30:19.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:30:18 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:30:19.796 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:30:19 smithi115 conmon[90393]: 2022-05-14T04:30:19.381+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:30:20.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:30:19 smithi115 ceph-mon[78593]: pgmap v9427: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:30:20.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:30:19 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:30:20.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:30:19 smithi047 ceph-mon[77147]: pgmap v9427: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:30:20.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:30:19 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:30:20.801 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:30:20 smithi115 conmon[90393]: 2022-05-14T04:30:20.405+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:30:21.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:30:20 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:30:21.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:30:20 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:30:21.810 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:30:21 smithi115 conmon[90393]: 2022-05-14T04:30:21.443+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:30:22.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:30:21 smithi115 ceph-mon[78593]: pgmap v9428: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:30:22.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:30:21 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:30:22.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:30:21 smithi047 ceph-mon[77147]: pgmap v9428: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:30:22.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:30:21 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:30:22.811 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:30:22 smithi115 conmon[90393]: 2022-05-14T04:30:22.444+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:30:23.116 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:30:22 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:30:23.116 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:30:22 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:30:23.117 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:30:22 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:30:23.117 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:30:22 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 18155 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:30:23.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:30:22 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:30:23.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:30:22 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:30:23.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:30:22 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:30:23.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:30:22 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 18155 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:30:23.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:30:23 smithi115 conmon[90393]: 2022-05-14T04:30:23.467+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:30:24.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:30:24 smithi047 ceph-mon[77147]: pgmap v9429: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:30:24.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:30:24 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:30:24.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:30:24 smithi115 conmon[90393]: 2022-05-14T04:30:24.505+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:30:24.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:30:24 smithi115 ceph-mon[78593]: pgmap v9429: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:30:24.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:30:24 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:30:25.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:30:25 smithi115 conmon[90393]: 2022-05-14T04:30:25.532+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:30:26.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:30:26 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:30:26.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:30:26 smithi047 ceph-mon[77147]: pgmap v9430: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:30:26.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:30:26 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:30:26.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:30:26 smithi115 ceph-mon[78593]: pgmap v9430: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:30:26.860 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:30:26 smithi115 conmon[90393]: 2022-05-14T04:30:26.538+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:30:27.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:30:27 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:30:27.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:30:27 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 18160 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:30:27.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:30:27 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:30:27.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:30:27 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 18160 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:30:27.862 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:30:27 smithi115 conmon[90393]: 2022-05-14T04:30:27.553+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:30:28.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:30:28 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:30:28.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:30:28 smithi047 ceph-mon[77147]: pgmap v9431: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:30:28.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:30:28 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:30:28.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:30:28 smithi115 ceph-mon[78593]: pgmap v9431: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:30:28.860 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:30:28 smithi115 conmon[90393]: 2022-05-14T04:30:28.541+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:30:29.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:30:29 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:30:29.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:30:29 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:30:29.860 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:30:29 smithi115 conmon[90393]: 2022-05-14T04:30:29.506+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:30:30.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:30:30 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:30:30.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:30:30 smithi047 ceph-mon[77147]: pgmap v9432: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:30:30.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:30:30 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:30:30.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:30:30 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:30:30.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:30:30 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:30:30.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:30:30 smithi115 ceph-mon[78593]: pgmap v9432: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:30:30.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:30:30 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:30:30.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:30:30 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:30:30.861 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:30:30 smithi115 conmon[90393]: 2022-05-14T04:30:30.534+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:30:31.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:30:31 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:30:31.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:30:31 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:30:31.860 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:30:31 smithi115 conmon[90393]: 2022-05-14T04:30:31.567+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:30:32.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:30:32 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:30:32.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:30:32 smithi047 ceph-mon[77147]: pgmap v9433: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:30:32.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:30:32 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 18165 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:30:32.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:30:32 smithi115 conmon[90393]: 2022-05-14T04:30:32.557+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:30:32.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:30:32 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:30:32.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:30:32 smithi115 ceph-mon[78593]: pgmap v9433: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:30:32.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:30:32 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 18165 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:30:33.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:30:33 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:30:33.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:30:33 smithi115 conmon[90393]: 2022-05-14T04:30:33.571+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:30:33.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:30:33 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:30:34.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:30:34 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:30:34.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:30:34 smithi047 ceph-mon[77147]: pgmap v9434: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:30:34.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:30:34 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:30:34.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:30:34 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:30:34.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:30:34 smithi115 conmon[90393]: 2022-05-14T04:30:34.523+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:30:34.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:30:34 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:30:34.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:30:34 smithi115 ceph-mon[78593]: pgmap v9434: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:30:34.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:30:34 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:30:34.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:30:34 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:30:35.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:30:35 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:30:35.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:30:35 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:30:35.860 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:30:35 smithi115 conmon[90393]: 2022-05-14T04:30:35.518+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:30:36.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:30:36 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:30:36.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:30:36 smithi047 ceph-mon[77147]: pgmap v9435: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:30:36.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:30:36 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:30:36.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:30:36 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:30:36.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:30:36 smithi115 conmon[90393]: 2022-05-14T04:30:36.478+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:30:36.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:30:36 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:30:36.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:30:36 smithi115 ceph-mon[78593]: pgmap v9435: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:30:36.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:30:36 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:30:36.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:30:36 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:30:37.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:30:37 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 18170 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:30:37.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:30:37 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 18170 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:30:37.860 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:30:37 smithi115 conmon[90393]: 2022-05-14T04:30:37.504+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:30:38.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:30:38 smithi047 ceph-mon[77147]: pgmap v9436: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:30:38.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:30:38 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:30:38.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:30:38 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:30:38.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:30:38 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:30:38.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:30:38 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:30:38.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:30:38 smithi115 conmon[90393]: 2022-05-14T04:30:38.460+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:30:38.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:30:38 smithi115 ceph-mon[78593]: pgmap v9436: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:30:38.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:30:38 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:30:38.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:30:38 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:30:38.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:30:38 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:30:38.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:30:38 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:30:39.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:30:39 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:30:39.863 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:30:39 smithi115 conmon[90393]: 2022-05-14T04:30:39.410+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:30:39.863 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:30:39 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:30:40.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:30:40 smithi047 ceph-mon[77147]: pgmap v9437: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:30:40.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:30:40 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:30:40.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:30:40 smithi115 conmon[90393]: 2022-05-14T04:30:40.439+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:30:40.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:30:40 smithi115 ceph-mon[78593]: pgmap v9437: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:30:40.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:30:40 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:30:41.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:30:41 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:30:41.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:30:41 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:30:41.860 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:30:41 smithi115 conmon[90393]: 2022-05-14T04:30:41.413+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:30:42.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:30:42 smithi047 ceph-mon[77147]: pgmap v9438: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:30:42.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:30:42 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:30:42.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:30:42 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:30:42.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:30:42 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:30:42.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:30:42 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 18175 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:30:42.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:30:42 smithi115 conmon[90393]: 2022-05-14T04:30:42.448+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:30:42.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:30:42 smithi115 ceph-mon[78593]: pgmap v9438: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:30:42.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:30:42 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:30:42.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:30:42 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:30:42.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:30:42 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:30:42.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:30:42 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 18175 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:30:43.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:30:43 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:30:43.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:30:43 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:30:43.860 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:30:43 smithi115 conmon[90393]: 2022-05-14T04:30:43.470+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:30:44.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:30:44 smithi047 ceph-mon[77147]: pgmap v9439: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:30:44.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:30:44 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:30:44.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:30:44 smithi115 ceph-mon[78593]: pgmap v9439: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:30:44.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:30:44 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:30:44.861 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:30:44 smithi115 conmon[90393]: 2022-05-14T04:30:44.497+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:30:45.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:30:45 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:30:45.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:30:45 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:30:45.860 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:30:45 smithi115 conmon[90393]: 2022-05-14T04:30:45.486+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:30:46.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:30:46 smithi047 ceph-mon[77147]: pgmap v9440: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:30:46.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:30:46 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:30:46.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:30:46 smithi115 ceph-mon[78593]: pgmap v9440: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:30:46.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:30:46 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:30:46.860 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:30:46 smithi115 conmon[90393]: 2022-05-14T04:30:46.501+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:30:47.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:30:47 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 18180 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:30:47.860 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:30:47 smithi115 conmon[90393]: 2022-05-14T04:30:47.507+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:30:48.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:30:47 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 18180 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:30:48.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:30:48 smithi115 conmon[90393]: 2022-05-14T04:30:48.550+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:30:48.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:30:48 smithi115 ceph-mon[78593]: pgmap v9441: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:30:48.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:30:48 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:30:49.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:30:48 smithi047 ceph-mon[77147]: pgmap v9441: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:30:49.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:30:48 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:30:49.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:30:49 smithi115 conmon[90393]: 2022-05-14T04:30:49.541+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:30:49.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:30:49 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:30:50.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:30:49 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:30:50.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:30:50 smithi115 conmon[90393]: 2022-05-14T04:30:50.577+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:30:50.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:30:50 smithi115 ceph-mon[78593]: pgmap v9442: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:30:50.862 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:30:50 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:30:50.862 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:30:50 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:30:50.863 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:30:50 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:30:51.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:30:50 smithi047 ceph-mon[77147]: pgmap v9442: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:30:51.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:30:50 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:30:51.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:30:50 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:30:51.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:30:50 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:30:51.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:30:51 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:30:51.860 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:30:51 smithi115 conmon[90393]: 2022-05-14T04:30:51.551+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:30:52.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:30:51 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:30:52.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:30:52 smithi115 ceph-mon[78593]: pgmap v9443: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:30:52.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:30:52 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:30:52.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:30:52 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 18185 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:30:52.860 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:30:52 smithi115 conmon[90393]: 2022-05-14T04:30:52.561+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:30:53.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:30:52 smithi047 ceph-mon[77147]: pgmap v9443: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:30:53.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:30:52 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:30:53.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:30:52 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 18185 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:30:53.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:30:53 smithi115 conmon[90393]: 2022-05-14T04:30:53.554+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:30:53.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:30:53 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:30:54.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:30:53 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:30:54.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:30:54 smithi115 conmon[90393]: 2022-05-14T04:30:54.557+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:30:54.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:30:54 smithi115 ceph-mon[78593]: pgmap v9444: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:30:54.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:30:54 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:30:54.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:30:54 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:30:54.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:30:54 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:30:55.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:30:54 smithi047 ceph-mon[77147]: pgmap v9444: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:30:55.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:30:54 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:30:55.045 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:30:54 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:30:55.045 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:30:54 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:30:55.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:30:55 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:30:55.860 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:30:55 smithi115 conmon[90393]: 2022-05-14T04:30:55.594+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:30:56.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:30:55 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:30:56.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:30:56 smithi115 ceph-mon[78593]: pgmap v9445: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:30:56.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:30:56 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:30:56.861 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:30:56 smithi115 conmon[90393]: 2022-05-14T04:30:56.636+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:30:57.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:30:56 smithi047 ceph-mon[77147]: pgmap v9445: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:30:57.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:30:56 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:30:57.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:30:57 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:30:57.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:30:57 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 18190 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:30:57.861 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:30:57 smithi115 conmon[90393]: 2022-05-14T04:30:57.661+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:30:58.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:30:57 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:30:58.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:30:57 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 18190 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:30:58.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:30:58 smithi115 ceph-mon[78593]: pgmap v9446: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:30:58.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:30:58 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:30:58.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:30:58 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:30:58.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:30:58 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:30:58.861 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:30:58 smithi115 conmon[90393]: 2022-05-14T04:30:58.665+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:30:59.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:30:58 smithi047 ceph-mon[77147]: pgmap v9446: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:30:59.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:30:58 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:30:59.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:30:58 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:30:59.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:30:58 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:30:59.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:30:59 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:30:59.860 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:30:59 smithi115 conmon[90393]: 2022-05-14T04:30:59.617+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:31:00.047 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:30:59 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:31:00.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:31:00 smithi115 ceph-mon[78593]: pgmap v9447: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:31:00.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:31:00 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:31:00.861 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:31:00 smithi115 conmon[90393]: 2022-05-14T04:31:00.619+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:31:01.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:31:00 smithi047 ceph-mon[77147]: pgmap v9447: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:31:01.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:31:00 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:31:01.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:31:01 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:31:01.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:31:01 smithi115 ceph-mon[78593]: pgmap v9448: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:31:01.861 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:31:01 smithi115 conmon[90393]: 2022-05-14T04:31:01.665+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:31:02.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:31:01 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:31:02.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:31:01 smithi047 ceph-mon[77147]: pgmap v9448: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:31:02.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:31:02 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:31:02.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:31:02 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:31:02.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:31:02 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:31:02.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:31:02 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 18195 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:31:02.861 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:31:02 smithi115 conmon[90393]: 2022-05-14T04:31:02.628+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:31:03.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:31:02 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:31:03.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:31:02 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:31:03.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:31:02 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:31:03.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:31:02 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 18195 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:31:03.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:31:03 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:31:03.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:31:03 smithi115 ceph-mon[78593]: pgmap v9449: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:31:03.861 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:31:03 smithi115 conmon[90393]: 2022-05-14T04:31:03.646+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:31:04.046 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:31:03 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:31:04.046 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:31:03 smithi047 ceph-mon[77147]: pgmap v9449: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:31:04.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:31:04 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:31:04.860 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:31:04 smithi115 conmon[90393]: 2022-05-14T04:31:04.675+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:31:05.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:31:04 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:31:05.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:31:05 smithi115 conmon[90393]: 2022-05-14T04:31:05.712+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:31:05.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:31:05 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:31:05.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:31:05 smithi115 ceph-mon[78593]: pgmap v9450: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:31:06.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:31:05 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:31:06.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:31:05 smithi047 ceph-mon[77147]: pgmap v9450: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:31:06.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:31:06 smithi115 conmon[90393]: 2022-05-14T04:31:06.689+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:31:06.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:31:06 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:31:07.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:31:06 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:31:08.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:31:07 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:31:08.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:31:07 smithi047 ceph-mon[77147]: pgmap v9451: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:31:08.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:31:07 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 18200 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:31:08.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:31:07 smithi115 conmon[90393]: 2022-05-14T04:31:07.698+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:31:08.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:31:07 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:31:08.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:31:07 smithi115 ceph-mon[78593]: pgmap v9451: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:31:08.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:31:07 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 18200 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:31:09.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:31:08 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:31:09.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:31:08 smithi115 conmon[90393]: 2022-05-14T04:31:08.671+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:31:09.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:31:08 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:31:10.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:31:09 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:31:10.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:31:09 smithi047 ceph-mon[77147]: pgmap v9452: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:31:10.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:31:09 smithi115 conmon[90393]: 2022-05-14T04:31:09.632+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:31:10.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:31:09 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:31:10.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:31:09 smithi115 ceph-mon[78593]: pgmap v9452: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:31:11.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:31:10 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:31:11.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:31:10 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:31:11.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:31:10 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:31:11.110 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:31:10 smithi115 conmon[90393]: 2022-05-14T04:31:10.674+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:31:11.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:31:10 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:31:11.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:31:10 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:31:11.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:31:10 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:31:12.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:31:11 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:31:12.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:31:11 smithi047 ceph-mon[77147]: pgmap v9453: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:31:12.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:31:11 smithi115 conmon[90393]: 2022-05-14T04:31:11.652+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:31:12.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:31:11 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:31:12.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:31:11 smithi115 ceph-mon[78593]: pgmap v9453: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:31:13.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:31:12 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:31:13.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:31:12 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 18205 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:31:13.112 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:31:12 smithi115 conmon[90393]: 2022-05-14T04:31:12.638+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:31:13.117 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:31:12 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:31:13.118 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:31:12 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 18205 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:31:14.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:31:13 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:31:14.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:31:13 smithi047 ceph-mon[77147]: pgmap v9454: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:31:14.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:31:13 smithi115 conmon[90393]: 2022-05-14T04:31:13.669+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:31:14.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:31:13 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:31:14.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:31:13 smithi115 ceph-mon[78593]: pgmap v9454: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:31:15.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:31:14 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:31:15.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:31:14 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:31:15.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:31:14 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:31:15.110 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:31:14 smithi115 conmon[90393]: 2022-05-14T04:31:14.713+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:31:15.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:31:14 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:31:15.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:31:14 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:31:15.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:31:14 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:31:16.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:31:15 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T04:31:16.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:31:15 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T04:31:16.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:31:15 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:31:16.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:31:15 smithi047 ceph-mon[77147]: pgmap v9455: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:31:16.099 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:31:15 smithi115 conmon[90393]: 2022-05-14T04:31:15.677+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:31:16.100 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:31:15 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T04:31:16.100 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:31:15 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T04:31:16.101 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:31:15 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:31:16.101 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:31:15 smithi115 ceph-mon[78593]: pgmap v9455: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:31:16.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:31:16 smithi115 conmon[90393]: 2022-05-14T04:31:16.672+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:31:16.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:31:16 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:31:16.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:31:16 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T04:31:16.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:31:16 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T04:31:16.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:31:16 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T04:31:17.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:31:16 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:31:17.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:31:16 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T04:31:17.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:31:16 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T04:31:17.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:31:16 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T04:31:18.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:31:17 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:31:18.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:31:17 smithi047 ceph-mon[77147]: pgmap v9456: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:31:18.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:31:17 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 18210 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:31:18.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:31:17 smithi115 conmon[90393]: 2022-05-14T04:31:17.711+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:31:18.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:31:17 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:31:18.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:31:17 smithi115 ceph-mon[78593]: pgmap v9456: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:31:18.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:31:17 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 18210 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:31:19.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:31:18 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:31:19.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:31:18 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:31:19.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:31:18 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:31:19.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:31:18 smithi115 conmon[90393]: 2022-05-14T04:31:18.739+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:31:19.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:31:18 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:31:19.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:31:18 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:31:19.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:31:18 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:31:19.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:31:19 smithi047 conmon[78521]: 2022-05-14T04:31:19.151+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:31:19.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:31:19 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T04:31:19.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:31:19 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:31:19.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:31:19 smithi047 conmon[78521]: yield 2022-05-14T04:31:19.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:31:19 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:31:19.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:31:19 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:31:19.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:31:19 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:31:19.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:31:19 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T04:31:19.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:31:19 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:31:19.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:31:19 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T04:31:19.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:31:19 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:31:19.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:31:19 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T04:31:19.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:31:19 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:31:19.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:31:19 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T04:31:19.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:31:19 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:31:19.547 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:31:19 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T04:31:19.547 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:31:19 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:31:19.547 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:31:19 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:31:19.547 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:31:19 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:31:20.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:31:19 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:31:20.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:31:19 smithi047 ceph-mon[77147]: pgmap v9457: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:31:20.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:31:19 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:31:20.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T04:31:20.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:31:20.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T04:31:20.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:31:20.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:31:20.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:31:20.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T04:31:20.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:31:20.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T04:31:20.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:31:20.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T04:31:20.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:31:20.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T04:31:20.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:31:20.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T04:31:20.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:31:20.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:31:20.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:31:20.048 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:31:19 smithi047 ceph-mon[77147]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T04:31:20.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:31:19 smithi115 conmon[90393]: 2022-05-14T04:31:19.723+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:31:20.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:31:19 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:31:20.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:31:19 smithi115 ceph-mon[78593]: pgmap v9457: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:31:20.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:31:19 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:31:20.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T04:31:20.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:31:20.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T04:31:20.113 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:31:20.113 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:31:20.113 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:31:20.113 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T04:31:20.114 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:31:20.114 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T04:31:20.114 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:31:20.114 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T04:31:20.115 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:31:20.115 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T04:31:20.115 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:31:20.115 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T04:31:20.115 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:31:20.116 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:31:20.116 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:31:20.116 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:31:19 smithi115 ceph-mon[78593]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T04:31:21.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:31:20 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:31:21.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:31:20 smithi115 conmon[90393]: 2022-05-14T04:31:20.759+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:31:21.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:31:20 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:31:22.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:31:21 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:31:22.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:31:21 smithi047 ceph-mon[77147]: pgmap v9458: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:31:22.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:31:21 smithi115 conmon[90393]: 2022-05-14T04:31:21.732+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:31:22.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:31:21 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:31:22.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:31:21 smithi115 ceph-mon[78593]: pgmap v9458: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:31:22.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:31:22 smithi047 conmon[78521]: 2022-05-14T04:31:22.223+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:31:22.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:31:22 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T04:31:22.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:31:22 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:31:22.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:31:22 smithi047 conmon[78521]: yield 2022-05-14T04:31:22.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:31:22 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:31:22.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:31:22 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:31:22.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:31:22 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:31:22.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:31:22 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T04:31:22.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:31:22 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:31:22.548 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:31:22 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T04:31:22.548 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:31:22 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:31:22.549 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:31:22 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T04:31:22.549 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:31:22 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:31:22.549 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:31:22 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T04:31:22.549 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:31:22 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:31:22.549 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:31:22 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T04:31:22.550 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:31:22 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:31:22.550 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:31:22 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:31:22.550 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:31:22 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:31:22.550 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:31:22 smithi047 conmon[78521]: 2022-05-14T04:31:22.224+0000 7f533ef76700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T04:31:22.551 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:31:22 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T04:31:22.551 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:31:22 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:31:22.551 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:31:22 smithi047 conmon[78521]: yield 2022-05-14T04:31:22.551 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:31:22 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:31:22.551 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:31:22 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:31:22.552 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:31:22 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:31:22.552 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:31:22 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T04:31:22.552 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:31:22 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:31:22.552 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:31:22 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T04:31:22.553 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:31:22 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:31:22.553 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:31:22 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T04:31:22.553 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:31:22 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:31:22.553 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:31:22 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T04:31:22.554 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:31:22 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:31:22.554 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:31:22 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T04:31:22.554 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:31:22 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:31:22.554 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:31:22 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:31:22.554 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:31:22 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:31:22.555 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:31:22 smithi047 conmon[78521]: 2022-05-14T04:31:22.555 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:31:22 smithi047 conmon[78521]: During handling of the above exception, another exception occurred: 2022-05-14T04:31:22.555 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:31:22 smithi047 conmon[78521]: 2022-05-14T04:31:22.555 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:31:22 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T04:31:22.556 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:31:22 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T04:31:22.556 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:31:22 smithi047 conmon[78521]: return f(*arg) 2022-05-14T04:31:22.556 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:31:22 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T04:31:22.556 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:31:22 smithi047 conmon[78521]: self._write_client_files(client_files, host) 2022-05-14T04:31:22.557 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:31:22 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T04:31:22.557 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:31:22 smithi047 conmon[78521]: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T04:31:22.557 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:31:22 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T04:31:22.557 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:31:22 smithi047 conmon[78521]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T04:31:22.557 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:31:22 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T04:31:22.558 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:31:22 smithi047 conmon[78521]: return self.event_loop.get_result(coro) 2022-05-14T04:31:22.558 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:31:22 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T04:31:22.558 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:31:22 smithi047 conmon[78521]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T04:31:22.558 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:31:22 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T04:31:22.558 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:31:22 smithi047 conmon[78521]: return self.__get_result() 2022-05-14T04:31:22.559 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:31:22 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T04:31:22.559 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:31:22 smithi047 conmon[78521]: raise self._exception 2022-05-14T04:31:22.559 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:31:22 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T04:31:22.559 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:31:22 smithi047 conmon[78521]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T04:31:22.559 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:31:22 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T04:31:22.560 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:31:22 smithi047 conmon[78521]: conn = await self._remote_connection(host, addr) 2022-05-14T04:31:22.560 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:31:22 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T04:31:22.560 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:31:22 smithi047 conmon[78521]: raise 2022-05-14T04:31:22.560 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:31:22 smithi047 conmon[78521]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T04:31:22.560 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:31:22 smithi047 conmon[78521]: self.gen.throw(type, value, traceback) 2022-05-14T04:31:22.561 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:31:22 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T04:31:22.561 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:31:22 smithi047 conmon[78521]: raise OrchestratorError(msg) 2022-05-14T04:31:22.561 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:31:22 smithi047 conmon[78521]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:31:23.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:31:22 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:31:23.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:31:22 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:31:23.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:31:22 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:31:23.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:31:22 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:31:23.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T04:31:23.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:31:23.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T04:31:23.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:31:23.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:31:23.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:31:23.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T04:31:23.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:31:23.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T04:31:23.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:31:23.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T04:31:23.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:31:23.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T04:31:23.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:31:23.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T04:31:23.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:31:23.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:31:23.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:31:23.048 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:31:22 smithi047 ceph-mon[77147]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T04:31:23.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T04:31:23.049 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:31:23.049 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T04:31:23.049 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:31:23.049 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:31:23.050 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:31:23.050 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T04:31:23.050 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:31:23.050 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T04:31:23.050 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:31:23.051 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T04:31:23.051 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:31:23.051 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T04:31:23.052 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:31:23.052 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T04:31:23.052 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:31:23.052 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:31:23.053 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:31:23.053 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T04:31:23.053 INFO:journalctl@ceph.mon.b.smithi047.stdout: During handling of the above exception, another exception occurred: 2022-05-14T04:31:23.053 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T04:31:23.054 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T04:31:23.056 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T04:31:23.057 INFO:journalctl@ceph.mon.b.smithi047.stdout: return f(*arg) 2022-05-14T04:31:23.058 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T04:31:23.059 INFO:journalctl@ceph.mon.b.smithi047.stdout: self._write_client_files(client_files, host) 2022-05-14T04:31:23.059 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T04:31:23.059 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T04:31:23.059 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T04:31:23.060 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T04:31:23.060 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T04:31:23.060 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.event_loop.get_result(coro) 2022-05-14T04:31:23.060 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T04:31:23.061 INFO:journalctl@ceph.mon.b.smithi047.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T04:31:23.061 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T04:31:23.061 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.__get_result() 2022-05-14T04:31:23.061 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T04:31:23.061 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise self._exception 2022-05-14T04:31:23.062 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T04:31:23.062 INFO:journalctl@ceph.mon.b.smithi047.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T04:31:23.062 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T04:31:23.062 INFO:journalctl@ceph.mon.b.smithi047.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T04:31:23.063 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T04:31:23.063 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise 2022-05-14T04:31:23.063 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T04:31:23.063 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.gen.throw(type, value, traceback) 2022-05-14T04:31:23.063 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T04:31:23.064 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OrchestratorError(msg) 2022-05-14T04:31:23.064 INFO:journalctl@ceph.mon.b.smithi047.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:31:23.064 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:31:22 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 18215 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:31:23.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:31:22 smithi115 conmon[90393]: 2022-05-14T04:31:22.747+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:31:23.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:31:22 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:31:23.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:31:22 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:31:23.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:31:22 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:31:23.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:31:22 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:31:23.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T04:31:23.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:31:23.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T04:31:23.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:31:23.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:31:23.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:31:23.113 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T04:31:23.113 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:31:23.113 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T04:31:23.113 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:31:23.114 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T04:31:23.114 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:31:23.114 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T04:31:23.114 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:31:23.114 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T04:31:23.115 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:31:23.115 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:31:23.115 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:31:23.115 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:31:22 smithi115 ceph-mon[78593]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T04:31:23.115 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T04:31:23.116 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:31:23.116 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T04:31:23.116 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:31:23.116 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:31:23.117 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:31:23.117 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T04:31:23.117 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:31:23.117 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T04:31:23.117 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:31:23.118 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T04:31:23.118 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:31:23.118 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T04:31:23.118 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:31:23.118 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T04:31:23.119 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:31:23.119 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:31:23.119 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:31:23.119 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T04:31:23.120 INFO:journalctl@ceph.mon.c.smithi115.stdout: During handling of the above exception, another exception occurred: 2022-05-14T04:31:23.120 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T04:31:23.120 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T04:31:23.120 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T04:31:23.120 INFO:journalctl@ceph.mon.c.smithi115.stdout: return f(*arg) 2022-05-14T04:31:23.121 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T04:31:23.121 INFO:journalctl@ceph.mon.c.smithi115.stdout: self._write_client_files(client_files, host) 2022-05-14T04:31:23.121 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T04:31:23.121 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T04:31:23.122 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T04:31:23.122 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T04:31:23.122 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T04:31:23.122 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.event_loop.get_result(coro) 2022-05-14T04:31:23.122 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T04:31:23.123 INFO:journalctl@ceph.mon.c.smithi115.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T04:31:23.123 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T04:31:23.123 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.__get_result() 2022-05-14T04:31:23.123 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T04:31:23.124 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise self._exception 2022-05-14T04:31:23.124 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T04:31:23.124 INFO:journalctl@ceph.mon.c.smithi115.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T04:31:23.124 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T04:31:23.124 INFO:journalctl@ceph.mon.c.smithi115.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T04:31:23.125 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T04:31:23.125 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise 2022-05-14T04:31:23.125 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T04:31:23.125 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.gen.throw(type, value, traceback) 2022-05-14T04:31:23.125 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T04:31:23.126 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OrchestratorError(msg) 2022-05-14T04:31:23.126 INFO:journalctl@ceph.mon.c.smithi115.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:31:23.126 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:31:22 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 18215 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:31:24.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:31:23 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:31:24.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:31:23 smithi047 ceph-mon[77147]: pgmap v9459: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:31:24.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:31:23 smithi115 conmon[90393]: 2022-05-14T04:31:23.769+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:31:24.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:31:23 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:31:24.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:31:23 smithi115 ceph-mon[78593]: pgmap v9459: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:31:25.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:31:24 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:31:25.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:31:24 smithi115 conmon[90393]: 2022-05-14T04:31:24.743+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:31:25.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:31:24 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:31:26.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:31:25 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:31:26.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:31:25 smithi047 ceph-mon[77147]: pgmap v9460: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:31:26.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:31:25 smithi115 conmon[90393]: 2022-05-14T04:31:25.780+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:31:26.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:31:25 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:31:26.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:31:25 smithi115 ceph-mon[78593]: pgmap v9460: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:31:27.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:31:26 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:31:27.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:31:26 smithi115 conmon[90393]: 2022-05-14T04:31:26.804+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:31:27.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:31:26 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:31:28.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:31:27 smithi047 ceph-mon[77147]: pgmap v9461: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:31:28.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:31:27 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:31:28.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:31:27 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 18220 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:31:28.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:31:27 smithi115 conmon[90393]: 2022-05-14T04:31:27.772+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:31:28.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:31:27 smithi115 ceph-mon[78593]: pgmap v9461: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:31:28.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:31:27 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:31:28.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:31:27 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 18220 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:31:29.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:31:28 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:31:29.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:31:28 smithi115 conmon[90393]: 2022-05-14T04:31:28.780+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:31:29.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:31:28 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:31:30.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:31:29 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:31:30.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:31:29 smithi047 ceph-mon[77147]: pgmap v9462: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:31:30.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:31:29 smithi115 conmon[90393]: 2022-05-14T04:31:29.796+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:31:30.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:31:29 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:31:30.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:31:29 smithi115 ceph-mon[78593]: pgmap v9462: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:31:31.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:31:30 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:31:31.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:31:30 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:31:31.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:31:30 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:31:31.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:31:30 smithi115 conmon[90393]: 2022-05-14T04:31:30.817+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:31:31.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:31:30 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:31:31.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:31:30 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:31:31.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:31:30 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:31:32.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:31:31 smithi047 ceph-mon[77147]: pgmap v9463: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:31:32.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:31:31 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:31:32.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:31:31 smithi115 conmon[90393]: 2022-05-14T04:31:31.834+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:31:32.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:31:31 smithi115 ceph-mon[78593]: pgmap v9463: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:31:32.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:31:31 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:31:33.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:31:32 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:31:33.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:31:32 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 18225 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:31:33.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:31:32 smithi115 conmon[90393]: 2022-05-14T04:31:32.806+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:31:33.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:31:32 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:31:33.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:31:32 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 18225 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:31:34.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:31:33 smithi047 ceph-mon[77147]: pgmap v9464: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:31:34.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:31:33 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:31:34.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:31:33 smithi115 ceph-mon[78593]: pgmap v9464: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:31:34.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:31:33 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:31:34.110 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:31:33 smithi115 conmon[90393]: 2022-05-14T04:31:33.811+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:31:35.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:31:34 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:31:35.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:31:34 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:31:35.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:31:34 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:31:35.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:31:34 smithi115 conmon[90393]: 2022-05-14T04:31:34.847+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:31:35.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:31:34 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:31:35.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:31:34 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:31:35.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:31:34 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:31:36.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:31:35 smithi047 ceph-mon[77147]: pgmap v9465: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:31:36.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:31:35 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:31:36.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:31:35 smithi115 ceph-mon[78593]: pgmap v9465: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:31:36.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:31:35 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:31:36.110 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:31:35 smithi115 conmon[90393]: 2022-05-14T04:31:35.886+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:31:37.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:31:36 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:31:37.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:31:36 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:31:37.110 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:31:36 smithi115 conmon[90393]: 2022-05-14T04:31:36.872+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:31:38.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:31:37 smithi047 ceph-mon[77147]: pgmap v9466: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:31:38.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:31:37 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:31:38.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:31:37 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 18230 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:31:38.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:31:37 smithi115 conmon[90393]: 2022-05-14T04:31:37.890+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:31:38.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:31:37 smithi115 ceph-mon[78593]: pgmap v9466: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:31:38.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:31:37 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:31:38.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:31:37 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 18230 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:31:39.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:31:38 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:31:39.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:31:38 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:31:39.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:31:38 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:31:39.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:31:38 smithi115 conmon[90393]: 2022-05-14T04:31:38.903+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:31:39.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:31:38 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:31:39.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:31:38 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:31:39.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:31:38 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:31:40.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:31:39 smithi047 ceph-mon[77147]: pgmap v9467: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:31:40.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:31:39 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:31:40.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:31:39 smithi115 conmon[90393]: 2022-05-14T04:31:39.893+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:31:40.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:31:39 smithi115 ceph-mon[78593]: pgmap v9467: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:31:40.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:31:39 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:31:41.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:31:40 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:31:41.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:31:40 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:31:41.110 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:31:40 smithi115 conmon[90393]: 2022-05-14T04:31:40.901+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:31:42.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:31:41 smithi047 ceph-mon[77147]: pgmap v9468: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:31:42.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:31:41 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:31:42.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:31:41 smithi115 conmon[90393]: 2022-05-14T04:31:41.929+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:31:42.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:31:41 smithi115 ceph-mon[78593]: pgmap v9468: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:31:42.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:31:41 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:31:43.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:31:42 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:31:43.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:31:42 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:31:43.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:31:42 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:31:43.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:31:42 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 18235 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:31:43.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:31:42 smithi115 conmon[90393]: 2022-05-14T04:31:42.883+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:31:43.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:31:42 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:31:43.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:31:42 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:31:43.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:31:42 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:31:43.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:31:42 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 18235 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:31:44.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:31:43 smithi047 ceph-mon[77147]: pgmap v9469: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:31:44.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:31:43 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:31:44.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:31:43 smithi115 conmon[90393]: 2022-05-14T04:31:43.923+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:31:44.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:31:43 smithi115 ceph-mon[78593]: pgmap v9469: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:31:44.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:31:43 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:31:45.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:31:44 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:31:45.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:31:44 smithi115 conmon[90393]: 2022-05-14T04:31:44.962+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:31:45.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:31:44 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:31:46.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:31:45 smithi047 ceph-mon[77147]: pgmap v9470: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:31:46.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:31:45 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:31:46.110 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:31:45 smithi115 conmon[90393]: 2022-05-14T04:31:45.938+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:31:46.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:31:45 smithi115 ceph-mon[78593]: pgmap v9470: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:31:46.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:31:45 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:31:47.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:31:46 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:31:47.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:31:46 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:31:47.110 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:31:46 smithi115 conmon[90393]: 2022-05-14T04:31:46.952+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:31:48.001 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:31:47 smithi115 ceph-mon[78593]: pgmap v9471: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:31:48.002 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:31:47 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:31:48.002 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:31:47 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 18240 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:31:48.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:31:47 smithi047 ceph-mon[77147]: pgmap v9471: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:31:48.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:31:47 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:31:48.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:31:47 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 18240 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:31:48.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:31:47 smithi115 conmon[90393]: 2022-05-14T04:31:47.998+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:31:49.008 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:31:48 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:31:49.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:31:48 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:31:49.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:31:49 smithi115 conmon[90393]: 2022-05-14T04:31:49.005+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:31:50.021 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:31:49 smithi115 ceph-mon[78593]: pgmap v9472: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:31:50.021 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:31:49 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:31:50.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:31:49 smithi047 ceph-mon[77147]: pgmap v9472: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:31:50.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:31:49 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:31:50.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:31:50 smithi115 conmon[90393]: 2022-05-14T04:31:50.018+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:31:51.038 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:31:50 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:31:51.038 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:31:50 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:31:51.038 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:31:50 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:31:51.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:31:50 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:31:51.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:31:50 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:31:51.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:31:50 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:31:51.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:31:51 smithi115 conmon[90393]: 2022-05-14T04:31:51.034+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:31:52.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:31:51 smithi047 ceph-mon[77147]: pgmap v9473: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:31:52.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:31:51 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:31:52.055 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:31:51 smithi115 ceph-mon[78593]: pgmap v9473: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:31:52.056 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:31:51 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:31:52.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:31:52 smithi115 conmon[90393]: 2022-05-14T04:31:52.052+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:31:53.043 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:31:52 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:31:53.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:31:52 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 18245 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:31:53.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:31:52 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:31:53.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:31:52 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 18245 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:31:53.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:31:53 smithi115 conmon[90393]: 2022-05-14T04:31:53.039+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:31:54.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:31:53 smithi047 ceph-mon[77147]: pgmap v9474: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:31:54.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:31:53 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:31:54.069 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:31:53 smithi115 ceph-mon[78593]: pgmap v9474: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:31:54.070 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:31:53 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:31:54.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:31:54 smithi115 conmon[90393]: 2022-05-14T04:31:54.066+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:31:55.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:31:54 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:31:55.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:31:54 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:31:55.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:31:54 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:31:55.066 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:31:54 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:31:55.066 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:31:54 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:31:55.066 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:31:54 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:31:55.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:31:55 smithi115 conmon[90393]: 2022-05-14T04:31:55.063+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:31:56.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:31:55 smithi047 ceph-mon[77147]: pgmap v9475: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:31:56.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:31:55 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:31:56.061 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:31:55 smithi115 ceph-mon[78593]: pgmap v9475: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:31:56.061 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:31:55 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:31:56.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:31:56 smithi115 conmon[90393]: 2022-05-14T04:31:56.058+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:31:57.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:31:56 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:31:57.095 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:31:56 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:31:57.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:31:57 smithi115 conmon[90393]: 2022-05-14T04:31:57.092+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:31:58.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:31:57 smithi047 ceph-mon[77147]: pgmap v9476: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:31:58.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:31:57 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:31:58.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:31:57 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 18250 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:31:58.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:31:57 smithi115 ceph-mon[78593]: pgmap v9476: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:31:58.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:31:57 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:31:58.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:31:57 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 18250 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:31:58.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:31:58 smithi115 conmon[90393]: 2022-05-14T04:31:58.116+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:31:59.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:31:58 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:31:59.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:31:58 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:31:59.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:31:58 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:31:59.075 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:31:58 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:31:59.076 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:31:58 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:31:59.076 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:31:58 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:31:59.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:31:59 smithi115 conmon[90393]: 2022-05-14T04:31:59.073+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:32:00.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:31:59 smithi047 ceph-mon[77147]: pgmap v9477: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:32:00.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:31:59 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:32:00.064 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:31:59 smithi115 ceph-mon[78593]: pgmap v9477: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:32:00.064 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:31:59 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:32:00.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:32:00 smithi115 conmon[90393]: 2022-05-14T04:32:00.061+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:32:01.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:32:00 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:32:01.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:32:00 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:32:01.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:32:01 smithi115 conmon[90393]: 2022-05-14T04:32:01.039+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:32:02.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:32:01 smithi047 ceph-mon[77147]: pgmap v9478: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:32:02.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:32:01 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:32:02.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:32:02 smithi115 conmon[90393]: 2022-05-14T04:32:02.039+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:32:02.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:32:01 smithi115 ceph-mon[78593]: pgmap v9478: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:32:02.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:32:01 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:32:03.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:32:02 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:32:03.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:32:02 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:32:03.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:32:02 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:32:03.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:32:02 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 18255 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:32:03.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:32:03 smithi115 conmon[90393]: 2022-05-14T04:32:03.039+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:32:03.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:32:02 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:32:03.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:32:02 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:32:03.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:32:02 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:32:03.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:32:02 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 18255 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:32:04.080 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:32:03 smithi115 ceph-mon[78593]: pgmap v9479: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:32:04.080 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:32:03 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:32:04.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:32:03 smithi047 ceph-mon[77147]: pgmap v9479: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:32:04.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:32:03 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:32:04.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:32:04 smithi115 conmon[90393]: 2022-05-14T04:32:04.077+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:32:05.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:32:04 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:32:05.110 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:32:05 smithi115 conmon[90393]: 2022-05-14T04:32:05.041+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:32:05.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:32:04 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:32:06.088 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:32:05 smithi115 ceph-mon[78593]: pgmap v9480: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:32:06.088 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:32:05 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:32:06.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:32:05 smithi047 ceph-mon[77147]: pgmap v9480: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:32:06.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:32:05 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:32:06.360 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:32:06 smithi115 conmon[90393]: 2022-05-14T04:32:06.084+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:32:07.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:32:06 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:32:07.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:32:06 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:32:07.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:32:07 smithi115 conmon[90393]: 2022-05-14T04:32:07.116+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:32:08.089 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:32:07 smithi115 ceph-mon[78593]: pgmap v9481: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:32:08.090 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:32:07 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:32:08.090 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:32:07 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 18260 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:32:08.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:32:07 smithi047 ceph-mon[77147]: pgmap v9481: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:32:08.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:32:07 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:32:08.295 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:32:07 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 18260 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:32:08.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:32:08 smithi115 conmon[90393]: 2022-05-14T04:32:08.087+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:32:09.085 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:32:08 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:32:09.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:32:08 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:32:09.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:32:09 smithi115 conmon[90393]: 2022-05-14T04:32:09.082+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:32:10.128 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:32:09 smithi115 ceph-mon[78593]: pgmap v9482: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:32:10.129 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:32:09 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:32:10.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:32:09 smithi047 ceph-mon[77147]: pgmap v9482: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:32:10.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:32:09 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:32:10.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:32:10 smithi115 conmon[90393]: 2022-05-14T04:32:10.125+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:32:11.150 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:32:10 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:32:11.151 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:32:10 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:32:11.151 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:32:10 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:32:11.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:32:10 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:32:11.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:32:10 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:32:11.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:32:10 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:32:11.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:32:11 smithi115 conmon[90393]: 2022-05-14T04:32:11.147+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:32:12.145 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:32:11 smithi115 ceph-mon[78593]: pgmap v9483: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:32:12.146 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:32:11 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:32:12.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:32:11 smithi047 ceph-mon[77147]: pgmap v9483: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:32:12.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:32:11 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:32:12.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:32:12 smithi115 conmon[90393]: 2022-05-14T04:32:12.140+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:32:13.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:32:12 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:32:13.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:32:12 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 18265 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:32:13.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:32:13 smithi115 conmon[90393]: 2022-05-14T04:32:13.105+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:32:13.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:32:12 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:32:13.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:32:12 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 18265 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:32:14.501 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:32:14 smithi115 conmon[90393]: 2022-05-14T04:32:14.135+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:32:14.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:32:14 smithi047 ceph-mon[77147]: pgmap v9484: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:32:14.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:32:14 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:32:14.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:32:14 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:32:14.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:32:14 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:32:14.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:32:14 smithi115 ceph-mon[78593]: pgmap v9484: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:32:14.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:32:14 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:32:14.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:32:14 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:32:14.862 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:32:14 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:32:15.505 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:32:15 smithi115 conmon[90393]: 2022-05-14T04:32:15.144+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:32:15.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:32:15 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:32:15.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:32:15 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T04:32:15.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:32:15 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T04:32:15.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:32:15 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:32:15.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:32:15 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T04:32:15.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:32:15 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T04:32:16.507 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:32:16 smithi115 conmon[90393]: 2022-05-14T04:32:16.137+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:32:16.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:32:16 smithi047 ceph-mon[77147]: pgmap v9485: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:32:16.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:32:16 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:32:16.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:32:16 smithi115 ceph-mon[78593]: pgmap v9485: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:32:16.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:32:16 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:32:17.511 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:32:17 smithi115 conmon[90393]: 2022-05-14T04:32:17.126+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:32:17.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:32:17 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:32:17.837 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:32:17 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 18270 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:32:17.866 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:32:17 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:32:17.866 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:32:17 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 18270 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:32:18.512 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:32:18 smithi115 conmon[90393]: 2022-05-14T04:32:18.141+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:32:18.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:32:18 smithi047 ceph-mon[77147]: pgmap v9486: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:32:18.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:32:18 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:32:18.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:32:18 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:32:18.795 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:32:18 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:32:18.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:32:18 smithi115 ceph-mon[78593]: pgmap v9486: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:32:18.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:32:18 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:32:18.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:32:18 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:32:18.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:32:18 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:32:19.520 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:32:19 smithi115 conmon[90393]: 2022-05-14T04:32:19.170+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:32:19.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:32:19 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:32:19.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:32:19 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:32:20.519 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:32:20 smithi115 conmon[90393]: 2022-05-14T04:32:20.146+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:32:20.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:32:20 smithi047 ceph-mon[77147]: pgmap v9487: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:32:20.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:32:20 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:32:20.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:32:20 smithi115 ceph-mon[78593]: pgmap v9487: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:32:20.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:32:20 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:32:21.525 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:32:21 smithi115 conmon[90393]: 2022-05-14T04:32:21.147+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:32:21.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:32:21 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:32:21.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:32:21 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:32:22.525 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:32:22 smithi115 conmon[90393]: 2022-05-14T04:32:22.130+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:32:22.776 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:32:22 smithi047 ceph-mon[77147]: pgmap v9488: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:32:22.776 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:32:22 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:32:22.777 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:32:22 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:32:22.777 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:32:22 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:32:22.778 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:32:22 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:32:22.778 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:32:22 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T04:32:22.779 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:32:22 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T04:32:22.779 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:32:22 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T04:32:22.779 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:32:22 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 18275 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:32:22.782 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:32:22 smithi115 ceph-mon[78593]: pgmap v9488: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:32:22.782 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:32:22 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:32:22.782 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:32:22 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:32:22.783 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:32:22 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:32:22.783 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:32:22 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:32:22.783 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:32:22 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T04:32:22.784 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:32:22 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T04:32:22.784 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:32:22 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T04:32:22.785 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:32:22 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 18275 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:32:23.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:32:23 smithi115 conmon[90393]: 2022-05-14T04:32:23.102+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:32:24.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:32:23 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' 2022-05-14T04:32:24.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:32:23 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' 2022-05-14T04:32:24.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:32:23 smithi047 ceph-mon[77147]: pgmap v9489: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:32:24.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:32:23 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:32:24.085 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:32:23 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' 2022-05-14T04:32:24.085 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:32:23 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' 2022-05-14T04:32:24.085 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:32:23 smithi115 ceph-mon[78593]: pgmap v9489: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:32:24.086 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:32:23 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:32:24.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:32:24 smithi115 conmon[90393]: 2022-05-14T04:32:24.081+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:32:25.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:32:24 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:32:25.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:32:24 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:32:25.312 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:32:25 smithi115 conmon[90393]: 2022-05-14T04:32:25.043+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:32:25.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:32:25 smithi047 conmon[78521]: 2022-05-14T04:32:25.328+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:32:25.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:32:25 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T04:32:25.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:32:25 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:32:25.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:32:25 smithi047 conmon[78521]: yield 2022-05-14T04:32:25.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:32:25 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:32:25.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:32:25 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:32:25.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:32:25 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:32:25.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:32:25 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T04:32:25.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:32:25 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:32:25.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:32:25 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T04:32:25.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:32:25 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:32:25.797 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:32:25 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T04:32:25.797 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:32:25 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:32:25.797 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:32:25 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T04:32:25.797 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:32:25 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:32:25.798 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:32:25 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T04:32:25.798 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:32:25 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:32:25.798 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:32:25 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:32:25.798 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:32:25 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:32:26.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:32:26 smithi115 conmon[90393]: 2022-05-14T04:32:26.049+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:32:26.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:32:26 smithi115 ceph-mon[78593]: pgmap v9490: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:32:26.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:32:26 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:32:26.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:32:26 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' 2022-05-14T04:32:26.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:32:26 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' 2022-05-14T04:32:26.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:32:26 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:32:26.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T04:32:26.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:32:26.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T04:32:26.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:32:26.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:32:26.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:32:26.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T04:32:26.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:32:26.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T04:32:26.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:32:26.365 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T04:32:26.365 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:32:26.365 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T04:32:26.365 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:32:26.366 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T04:32:26.366 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:32:26.366 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:32:26.366 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:32:26.367 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:32:26 smithi115 ceph-mon[78593]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T04:32:26.367 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:32:26 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-05-14T04:32:26.367 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:32:26 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-05-14T04:32:26.367 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:32:26 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-05-14T04:32:26.367 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:32:26 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-05-14T04:32:26.368 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:32:26 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-05-14T04:32:26.368 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:32:26 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-05-14T04:32:26.368 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:32:26 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-05-14T04:32:26.369 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:32:26 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-05-14T04:32:26.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:32:26 smithi047 ceph-mon[77147]: pgmap v9490: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:32:26.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:32:26 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:32:26.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:32:26 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' 2022-05-14T04:32:26.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:32:26 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' 2022-05-14T04:32:26.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:32:26 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:32:26.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T04:32:26.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:32:26.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T04:32:26.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:32:26.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:32:26.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:32:26.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T04:32:26.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:32:26.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T04:32:26.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:32:26.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T04:32:26.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:32:26.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T04:32:26.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:32:26.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T04:32:26.548 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:32:26.548 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:32:26.548 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:32:26.548 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:32:26 smithi047 ceph-mon[77147]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T04:32:26.548 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:32:26 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-05-14T04:32:26.549 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:32:26 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-05-14T04:32:26.549 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:32:26 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-05-14T04:32:26.549 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:32:26 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-05-14T04:32:26.549 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:32:26 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-05-14T04:32:26.550 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:32:26 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-05-14T04:32:26.550 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:32:26 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-05-14T04:32:26.550 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:32:26 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-05-14T04:32:27.360 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:32:27 smithi115 conmon[90393]: 2022-05-14T04:32:27.068+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:32:27.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:32:27 smithi115 ceph-mon[78593]: Adjusting osd_memory_target on smithi115 to 197.0M 2022-05-14T04:32:27.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:32:27 smithi115 ceph-mon[78593]: Unable to set osd_memory_target on smithi115 to 206629785: error parsing value: Value '206629785' is below minimum 939524096 2022-05-14T04:32:27.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:32:27 smithi115 ceph-mon[78593]: Adjusting osd_memory_target on smithi047 to 198.1M 2022-05-14T04:32:27.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:32:27 smithi115 ceph-mon[78593]: Unable to set osd_memory_target on smithi047 to 207755878: error parsing value: Value '207755878' is below minimum 939524096 2022-05-14T04:32:27.362 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:32:27 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:32:27.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:32:27 smithi047 ceph-mon[77147]: Adjusting osd_memory_target on smithi115 to 197.0M 2022-05-14T04:32:27.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:32:27 smithi047 ceph-mon[77147]: Unable to set osd_memory_target on smithi115 to 206629785: error parsing value: Value '206629785' is below minimum 939524096 2022-05-14T04:32:27.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:32:27 smithi047 ceph-mon[77147]: Adjusting osd_memory_target on smithi047 to 198.1M 2022-05-14T04:32:27.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:32:27 smithi047 ceph-mon[77147]: Unable to set osd_memory_target on smithi047 to 207755878: error parsing value: Value '207755878' is below minimum 939524096 2022-05-14T04:32:27.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:32:27 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:32:28.275 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:32:28 smithi115 conmon[90393]: 2022-05-14T04:32:28.021+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:32:28.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:32:28 smithi047 ceph-mon[77147]: pgmap v9491: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:32:28.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:32:28 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:32:28.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:32:28 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 18280 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:32:28.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:32:28 smithi047 conmon[78521]: 2022-05-14T04:32:28.400+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:32:28.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:32:28 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T04:32:28.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:32:28 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:32:28.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:32:28 smithi047 conmon[78521]: yield 2022-05-14T04:32:28.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:32:28 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:32:28.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:32:28 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:32:28.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:32:28 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:32:28.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:32:28 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T04:32:28.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:32:28 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:32:28.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:32:28 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T04:32:28.547 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:32:28 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:32:28.547 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:32:28 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T04:32:28.547 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:32:28 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:32:28.547 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:32:28 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T04:32:28.548 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:32:28 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:32:28.548 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:32:28 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T04:32:28.548 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:32:28 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:32:28.548 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:32:28 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:32:28.548 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:32:28 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:32:28.549 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:32:28 smithi047 conmon[78521]: 2022-05-14T04:32:28.401+0000 7f5344781700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T04:32:28.549 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:32:28 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T04:32:28.549 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:32:28 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:32:28.549 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:32:28 smithi047 conmon[78521]: yield 2022-05-14T04:32:28.549 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:32:28 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:32:28.550 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:32:28 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:32:28.550 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:32:28 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:32:28.550 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:32:28 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T04:32:28.550 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:32:28 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:32:28.551 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:32:28 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T04:32:28.551 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:32:28 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:32:28.551 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:32:28 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T04:32:28.551 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:32:28 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:32:28.551 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:32:28 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T04:32:28.552 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:32:28 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:32:28.552 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:32:28 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T04:32:28.552 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:32:28 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:32:28.552 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:32:28 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:32:28.552 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:32:28 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:32:28.553 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:32:28 smithi047 conmon[78521]: 2022-05-14T04:32:28.553 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:32:28 smithi047 conmon[78521]: During handling of the above exception, another exception occurred: 2022-05-14T04:32:28.553 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:32:28 smithi047 conmon[78521]: 2022-05-14T04:32:28.553 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:32:28 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T04:32:28.553 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:32:28 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T04:32:28.554 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:32:28 smithi047 conmon[78521]: return f(*arg) 2022-05-14T04:32:28.554 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:32:28 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T04:32:28.554 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:32:28 smithi047 conmon[78521]: self._write_client_files(client_files, host) 2022-05-14T04:32:28.554 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:32:28 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T04:32:28.554 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:32:28 smithi047 conmon[78521]: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T04:32:28.555 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:32:28 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T04:32:28.555 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:32:28 smithi047 conmon[78521]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T04:32:28.555 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:32:28 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T04:32:28.555 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:32:28 smithi047 conmon[78521]: return self.event_loop.get_result(coro) 2022-05-14T04:32:28.556 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:32:28 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T04:32:28.556 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:32:28 smithi047 conmon[78521]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T04:32:28.556 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:32:28 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T04:32:28.556 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:32:28 smithi047 conmon[78521]: return self.__get_result() 2022-05-14T04:32:28.556 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:32:28 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T04:32:28.557 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:32:28 smithi047 conmon[78521]: raise self._exception 2022-05-14T04:32:28.557 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:32:28 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T04:32:28.557 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:32:28 smithi047 conmon[78521]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T04:32:28.557 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:32:28 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T04:32:28.557 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:32:28 smithi047 conmon[78521]: conn = await self._remote_connection(host, addr) 2022-05-14T04:32:28.558 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:32:28 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T04:32:28.558 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:32:28 smithi047 conmon[78521]: raise 2022-05-14T04:32:28.558 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:32:28 smithi047 conmon[78521]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T04:32:28.558 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:32:28 smithi047 conmon[78521]: self.gen.throw(type, value, traceback) 2022-05-14T04:32:28.558 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:32:28 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T04:32:28.559 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:32:28 smithi047 conmon[78521]: raise OrchestratorError(msg) 2022-05-14T04:32:28.559 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:32:28 smithi047 conmon[78521]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:32:28.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:32:28 smithi115 ceph-mon[78593]: pgmap v9491: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:32:28.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:32:28 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:32:28.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:32:28 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 18280 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:32:29.280 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:32:29 smithi115 conmon[90393]: 2022-05-14T04:32:29.026+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:32:29.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:32:29 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:32:29.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:32:29 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:32:29.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T04:32:29.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:32:29.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T04:32:29.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:32:29.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:32:29.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:32:29.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T04:32:29.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:32:29.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T04:32:29.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:32:29.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T04:32:29.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:32:29.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T04:32:29.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:32:29.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T04:32:29.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:32:29.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:32:29.548 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:32:29.548 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:32:29 smithi047 ceph-mon[77147]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T04:32:29.548 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T04:32:29.548 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:32:29.548 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T04:32:29.549 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:32:29.549 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:32:29.549 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:32:29.549 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T04:32:29.550 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:32:29.550 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T04:32:29.550 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:32:29.550 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T04:32:29.550 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:32:29.551 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T04:32:29.552 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:32:29.552 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T04:32:29.552 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:32:29.552 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:32:29.553 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:32:29.553 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T04:32:29.553 INFO:journalctl@ceph.mon.b.smithi047.stdout: During handling of the above exception, another exception occurred: 2022-05-14T04:32:29.553 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T04:32:29.554 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T04:32:29.554 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T04:32:29.554 INFO:journalctl@ceph.mon.b.smithi047.stdout: return f(*arg) 2022-05-14T04:32:29.554 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T04:32:29.554 INFO:journalctl@ceph.mon.b.smithi047.stdout: self._write_client_files(client_files, host) 2022-05-14T04:32:29.555 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T04:32:29.555 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T04:32:29.555 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T04:32:29.555 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T04:32:29.556 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T04:32:29.556 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.event_loop.get_result(coro) 2022-05-14T04:32:29.556 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T04:32:29.556 INFO:journalctl@ceph.mon.b.smithi047.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T04:32:29.556 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T04:32:29.557 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.__get_result() 2022-05-14T04:32:29.557 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T04:32:29.559 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise self._exception 2022-05-14T04:32:29.559 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T04:32:29.560 INFO:journalctl@ceph.mon.b.smithi047.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T04:32:29.560 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T04:32:29.560 INFO:journalctl@ceph.mon.b.smithi047.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T04:32:29.560 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T04:32:29.561 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise 2022-05-14T04:32:29.561 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T04:32:29.561 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.gen.throw(type, value, traceback) 2022-05-14T04:32:29.561 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T04:32:29.562 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OrchestratorError(msg) 2022-05-14T04:32:29.562 INFO:journalctl@ceph.mon.b.smithi047.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:32:29.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:32:29 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:32:29.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:32:29 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:32:29.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T04:32:29.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:32:29.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T04:32:29.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:32:29.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:32:29.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:32:29.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T04:32:29.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:32:29.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T04:32:29.613 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:32:29.613 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T04:32:29.613 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:32:29.613 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T04:32:29.614 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:32:29.614 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T04:32:29.614 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:32:29.614 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:32:29.614 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:32:29.615 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:32:29 smithi115 ceph-mon[78593]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T04:32:29.615 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T04:32:29.615 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:32:29.616 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T04:32:29.616 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:32:29.616 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:32:29.616 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:32:29.616 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T04:32:29.617 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:32:29.617 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T04:32:29.617 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:32:29.617 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T04:32:29.618 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:32:29.618 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T04:32:29.618 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:32:29.619 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T04:32:29.619 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:32:29.619 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:32:29.619 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:32:29.620 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T04:32:29.620 INFO:journalctl@ceph.mon.c.smithi115.stdout: During handling of the above exception, another exception occurred: 2022-05-14T04:32:29.620 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T04:32:29.620 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T04:32:29.620 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T04:32:29.621 INFO:journalctl@ceph.mon.c.smithi115.stdout: return f(*arg) 2022-05-14T04:32:29.621 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T04:32:29.621 INFO:journalctl@ceph.mon.c.smithi115.stdout: self._write_client_files(client_files, host) 2022-05-14T04:32:29.621 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T04:32:29.622 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T04:32:29.622 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T04:32:29.622 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T04:32:29.623 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T04:32:29.623 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.event_loop.get_result(coro) 2022-05-14T04:32:29.623 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T04:32:29.623 INFO:journalctl@ceph.mon.c.smithi115.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T04:32:29.623 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T04:32:29.624 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.__get_result() 2022-05-14T04:32:29.624 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T04:32:29.624 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise self._exception 2022-05-14T04:32:29.625 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T04:32:29.625 INFO:journalctl@ceph.mon.c.smithi115.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T04:32:29.625 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T04:32:29.625 INFO:journalctl@ceph.mon.c.smithi115.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T04:32:29.625 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T04:32:29.626 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise 2022-05-14T04:32:29.626 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T04:32:29.626 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.gen.throw(type, value, traceback) 2022-05-14T04:32:29.626 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T04:32:29.627 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OrchestratorError(msg) 2022-05-14T04:32:29.627 INFO:journalctl@ceph.mon.c.smithi115.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:32:30.281 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:32:30 smithi115 conmon[90393]: 2022-05-14T04:32:30.009+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:32:30.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:32:30 smithi047 ceph-mon[77147]: pgmap v9492: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:32:30.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:32:30 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:32:30.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:32:30 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:32:30.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:32:30 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:32:30.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:32:30 smithi115 ceph-mon[78593]: pgmap v9492: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:32:30.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:32:30 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:32:30.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:32:30 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:32:30.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:32:30 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:32:31.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:32:31 smithi115 conmon[90393]: 2022-05-14T04:32:31.046+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:32:31.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:32:31 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:32:31.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:32:31 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:32:32.288 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:32:32 smithi115 conmon[90393]: 2022-05-14T04:32:32.011+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:32:32.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:32:32 smithi047 ceph-mon[77147]: pgmap v9493: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:32:32.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:32:32 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:32:32.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:32:32 smithi115 ceph-mon[78593]: pgmap v9493: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:32:32.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:32:32 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:32:33.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:32:33 smithi115 conmon[90393]: 2022-05-14T04:32:33.042+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:32:33.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:32:33 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:32:33.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:32:33 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 18285 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:32:33.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:32:33 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:32:33.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:32:33 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 18285 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:32:34.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:32:34 smithi115 conmon[90393]: 2022-05-14T04:32:34.011+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:32:34.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:32:34 smithi047 ceph-mon[77147]: pgmap v9494: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:32:34.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:32:34 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:32:34.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:32:34 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:32:34.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:32:34 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:32:34.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:32:34 smithi115 ceph-mon[78593]: pgmap v9494: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:32:34.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:32:34 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:32:34.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:32:34 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:32:34.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:32:34 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:32:35.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:32:35 smithi115 conmon[90393]: 2022-05-14T04:32:35.019+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:32:35.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:32:35 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:32:35.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:32:35 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:32:36.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:32:36 smithi115 conmon[90393]: 2022-05-14T04:32:36.025+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:32:36.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:32:36 smithi047 ceph-mon[77147]: pgmap v9495: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:32:36.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:32:36 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:32:36.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:32:36 smithi115 ceph-mon[78593]: pgmap v9495: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:32:36.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:32:36 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:32:37.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:32:37 smithi115 conmon[90393]: 2022-05-14T04:32:37.040+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:32:37.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:32:37 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:32:37.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:32:37 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 18290 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:32:37.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:32:37 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:32:37.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:32:37 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 18290 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:32:38.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:32:38 smithi115 conmon[90393]: 2022-05-14T04:32:38.018+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:32:38.795 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:32:38 smithi047 ceph-mon[77147]: pgmap v9496: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:32:38.795 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:32:38 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:32:38.795 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:32:38 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:32:38.796 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:32:38 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:32:38.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:32:38 smithi115 ceph-mon[78593]: pgmap v9496: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:32:38.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:32:38 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:32:38.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:32:38 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:32:38.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:32:38 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:32:39.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:32:39 smithi115 conmon[90393]: 2022-05-14T04:32:39.046+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:32:39.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:32:39 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:32:39.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:32:39 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:32:40.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:32:40 smithi115 conmon[90393]: 2022-05-14T04:32:40.060+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:32:40.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:32:40 smithi047 ceph-mon[77147]: pgmap v9497: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:32:40.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:32:40 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:32:40.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:32:40 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:32:40.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:32:40 smithi115 ceph-mon[78593]: pgmap v9497: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:32:40.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:32:40 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:32:40.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:32:40 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:32:41.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:32:41 smithi115 conmon[90393]: 2022-05-14T04:32:41.027+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:32:41.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:32:41 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:32:41.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:32:41 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:32:42.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:32:42 smithi115 conmon[90393]: 2022-05-14T04:32:42.029+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:32:42.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:32:42 smithi047 ceph-mon[77147]: pgmap v9498: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:32:42.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:32:42 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:32:42.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:32:42 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:32:42.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:32:42 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:32:42.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:32:42 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 18295 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:32:42.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:32:42 smithi115 ceph-mon[78593]: pgmap v9498: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:32:42.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:32:42 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:32:42.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:32:42 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:32:42.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:32:42 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:32:42.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:32:42 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 18295 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:32:43.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:32:43 smithi115 conmon[90393]: 2022-05-14T04:32:43.074+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:32:43.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:32:43 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:32:43.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:32:43 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:32:44.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:32:44 smithi115 conmon[90393]: 2022-05-14T04:32:44.089+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:32:44.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:32:44 smithi047 ceph-mon[77147]: pgmap v9499: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:32:44.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:32:44 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:32:44.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:32:44 smithi115 ceph-mon[78593]: pgmap v9499: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:32:44.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:32:44 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:32:45.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:32:45 smithi115 conmon[90393]: 2022-05-14T04:32:45.041+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:32:45.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:32:45 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:32:46.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:32:45 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:32:46.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:32:46 smithi115 conmon[90393]: 2022-05-14T04:32:46.012+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:32:46.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:32:46 smithi115 ceph-mon[78593]: pgmap v9500: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:32:46.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:32:46 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:32:47.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:32:46 smithi047 ceph-mon[77147]: pgmap v9500: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:32:47.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:32:46 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:32:47.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:32:46 smithi115 conmon[90393]: 2022-05-14T04:32:46.975+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:32:47.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:32:47 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:32:47.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:32:47 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 18301 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:32:48.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:32:47 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:32:48.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:32:47 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 18301 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:32:48.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:32:47 smithi115 conmon[90393]: 2022-05-14T04:32:47.974+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:32:48.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:32:48 smithi115 ceph-mon[78593]: pgmap v9501: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:32:48.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:32:48 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:32:49.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:32:48 smithi047 ceph-mon[77147]: pgmap v9501: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:32:49.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:32:48 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:32:49.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:32:48 smithi115 conmon[90393]: 2022-05-14T04:32:48.974+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:32:49.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:32:49 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:32:50.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:32:49 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:32:50.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:32:49 smithi115 conmon[90393]: 2022-05-14T04:32:49.973+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:32:50.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:32:50 smithi115 ceph-mon[78593]: pgmap v9502: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:32:50.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:32:50 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:32:50.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:32:50 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:32:50.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:32:50 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:32:51.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:32:50 smithi047 ceph-mon[77147]: pgmap v9502: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:32:51.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:32:50 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:32:51.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:32:50 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:32:51.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:32:50 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:32:51.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:32:50 smithi115 conmon[90393]: 2022-05-14T04:32:50.969+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:32:51.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:32:51 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:32:52.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:32:51 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:32:52.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:32:51 smithi115 conmon[90393]: 2022-05-14T04:32:51.991+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:32:52.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:32:52 smithi115 ceph-mon[78593]: pgmap v9503: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:32:52.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:32:52 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:32:52.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:32:52 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 18305 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:32:53.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:32:52 smithi047 ceph-mon[77147]: pgmap v9503: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:32:53.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:32:52 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:32:53.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:32:52 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 18305 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:32:53.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:32:52 smithi115 conmon[90393]: 2022-05-14T04:32:52.957+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:32:53.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:32:53 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:32:54.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:32:53 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:32:54.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:32:54 smithi115 conmon[90393]: 2022-05-14T04:32:54.007+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:32:54.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:32:54 smithi115 ceph-mon[78593]: pgmap v9504: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:32:54.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:32:54 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:32:54.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:32:54 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:32:54.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:32:54 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:32:55.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:32:54 smithi047 ceph-mon[77147]: pgmap v9504: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:32:55.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:32:54 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:32:55.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:32:54 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:32:55.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:32:54 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:32:55.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:32:54 smithi115 conmon[90393]: 2022-05-14T04:32:54.974+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:32:55.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:32:55 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:32:56.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:32:55 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:32:56.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:32:55 smithi115 conmon[90393]: 2022-05-14T04:32:55.976+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:32:56.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:32:56 smithi115 ceph-mon[78593]: pgmap v9505: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:32:56.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:32:56 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:32:57.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:32:56 smithi047 ceph-mon[77147]: pgmap v9505: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:32:57.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:32:56 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:32:57.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:32:56 smithi115 conmon[90393]: 2022-05-14T04:32:56.935+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:32:57.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:32:57 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:32:57.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:32:57 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 18310 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:32:58.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:32:57 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:32:58.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:32:57 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 18310 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:32:58.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:32:57 smithi115 conmon[90393]: 2022-05-14T04:32:57.918+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:32:58.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:32:58 smithi115 ceph-mon[78593]: pgmap v9506: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:32:58.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:32:58 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:32:58.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:32:58 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:32:58.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:32:58 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:32:59.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:32:58 smithi047 ceph-mon[77147]: pgmap v9506: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:32:59.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:32:58 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:32:59.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:32:58 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:32:59.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:32:58 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:32:59.366 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:32:58 smithi115 conmon[90393]: 2022-05-14T04:32:58.946+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:32:59.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:32:59 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:33:00.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:32:59 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:33:00.314 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:32:59 smithi115 conmon[90393]: 2022-05-14T04:32:59.940+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:33:00.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:33:00 smithi115 ceph-mon[78593]: pgmap v9507: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:33:00.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:33:00 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:33:01.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:33:00 smithi047 ceph-mon[77147]: pgmap v9507: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:33:01.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:33:00 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:33:01.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:33:00 smithi115 conmon[90393]: 2022-05-14T04:33:00.932+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:33:01.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:33:01 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:33:02.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:33:01 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:33:02.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:33:01 smithi115 conmon[90393]: 2022-05-14T04:33:01.934+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:33:02.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:33:02 smithi115 ceph-mon[78593]: pgmap v9508: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:33:02.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:33:02 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:33:02.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:33:02 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:33:02.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:33:02 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:33:02.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:33:02 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 18315 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:33:03.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:33:02 smithi047 ceph-mon[77147]: pgmap v9508: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:33:03.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:33:02 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:33:03.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:33:02 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:33:03.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:33:02 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:33:03.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:33:02 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 18315 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:33:03.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:33:02 smithi115 conmon[90393]: 2022-05-14T04:33:02.951+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:33:03.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:33:03 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:33:04.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:33:03 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:33:04.360 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:33:03 smithi115 conmon[90393]: 2022-05-14T04:33:03.949+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:33:04.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:33:04 smithi115 ceph-mon[78593]: pgmap v9509: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:33:04.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:33:04 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:33:05.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:33:04 smithi047 ceph-mon[77147]: pgmap v9509: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:33:05.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:33:04 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:33:05.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:33:04 smithi115 conmon[90393]: 2022-05-14T04:33:04.962+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:33:05.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:33:05 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:33:06.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:33:05 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:33:06.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:33:05 smithi115 conmon[90393]: 2022-05-14T04:33:05.986+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:33:07.002 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:33:06 smithi115 ceph-mon[78593]: pgmap v9510: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:33:07.003 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:33:06 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:33:07.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:33:06 smithi047 ceph-mon[77147]: pgmap v9510: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:33:07.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:33:06 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:33:07.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:33:07 smithi115 conmon[90393]: 2022-05-14T04:33:06.999+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:33:07.964 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:33:07 smithi115 ceph-mon[78593]: pgmap v9511: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:33:07.964 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:33:07 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:33:07.965 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:33:07 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 18320 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:33:08.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:33:07 smithi047 ceph-mon[77147]: pgmap v9511: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:33:08.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:33:07 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:33:08.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:33:07 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 18320 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:33:08.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:33:08 smithi115 conmon[90393]: 2022-05-14T04:33:08.024+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:33:09.032 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:33:08 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:33:09.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:33:08 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:33:09.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:33:09 smithi115 conmon[90393]: 2022-05-14T04:33:09.028+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:33:10.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:33:09 smithi047 ceph-mon[77147]: pgmap v9512: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:33:10.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:33:09 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:33:10.070 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:33:09 smithi115 ceph-mon[78593]: pgmap v9512: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:33:10.070 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:33:09 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:33:10.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:33:10 smithi115 conmon[90393]: 2022-05-14T04:33:10.064+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:33:11.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:33:10 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:33:11.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:33:10 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:33:11.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:33:10 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:33:11.097 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:33:10 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:33:11.097 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:33:10 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:33:11.097 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:33:10 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:33:11.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:33:11 smithi115 conmon[90393]: 2022-05-14T04:33:11.094+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:33:12.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:33:11 smithi047 ceph-mon[77147]: pgmap v9513: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:33:12.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:33:11 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:33:12.085 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:33:11 smithi115 ceph-mon[78593]: pgmap v9513: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:33:12.086 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:33:11 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:33:12.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:33:12 smithi115 conmon[90393]: 2022-05-14T04:33:12.082+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:33:12.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:33:12 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:33:12.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:33:12 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 18325 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:33:13.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:33:12 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:33:13.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:33:12 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 18325 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:33:13.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:33:13 smithi115 conmon[90393]: 2022-05-14T04:33:13.116+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:33:14.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:33:13 smithi047 ceph-mon[77147]: pgmap v9514: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:33:14.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:33:13 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:33:14.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:33:13 smithi115 ceph-mon[78593]: pgmap v9514: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:33:14.142 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:33:13 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:33:14.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:33:14 smithi115 conmon[90393]: 2022-05-14T04:33:14.164+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:33:15.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:33:14 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:33:15.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:33:14 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:33:15.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:33:14 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:33:15.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:33:14 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:33:15.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:33:14 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:33:15.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:33:14 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:33:15.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:33:15 smithi115 conmon[90393]: 2022-05-14T04:33:15.118+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:33:16.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:33:15 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T04:33:16.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:33:15 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T04:33:16.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:33:15 smithi047 ceph-mon[77147]: pgmap v9515: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:33:16.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:33:15 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:33:16.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:33:15 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T04:33:16.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:33:15 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T04:33:16.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:33:15 smithi115 ceph-mon[78593]: pgmap v9515: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:33:16.112 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:33:15 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:33:16.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:33:16 smithi115 conmon[90393]: 2022-05-14T04:33:16.114+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:33:17.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:33:16 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:33:17.080 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:33:16 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:33:17.362 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:33:17 smithi115 conmon[90393]: 2022-05-14T04:33:17.077+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:33:18.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:33:17 smithi047 ceph-mon[77147]: pgmap v9516: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:33:18.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:33:17 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:33:18.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:33:17 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 18330 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:33:18.093 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:33:17 smithi115 ceph-mon[78593]: pgmap v9516: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:33:18.093 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:33:17 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:33:18.094 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:33:17 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 18330 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:33:18.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:33:18 smithi115 conmon[90393]: 2022-05-14T04:33:18.089+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:33:19.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:33:18 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:33:19.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:33:18 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:33:19.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:33:18 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:33:19.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:33:18 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:33:19.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:33:18 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:33:19.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:33:18 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:33:19.611 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:33:19 smithi115 conmon[90393]: 2022-05-14T04:33:19.120+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:33:20.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:33:19 smithi047 ceph-mon[77147]: pgmap v9517: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:33:20.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:33:19 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:33:20.077 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:33:19 smithi115 ceph-mon[78593]: pgmap v9517: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:33:20.078 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:33:19 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:33:20.361 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:33:20 smithi115 conmon[90393]: 2022-05-14T04:33:20.075+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:33:21.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:33:20 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:33:21.051 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:33:20 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:33:21.360 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:33:21 smithi115 conmon[90393]: 2022-05-14T04:33:21.049+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:33:22.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:33:21 smithi047 ceph-mon[77147]: pgmap v9518: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:33:22.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:33:21 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:33:22.081 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:33:21 smithi115 ceph-mon[78593]: pgmap v9518: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:33:22.082 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:33:21 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:33:22.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:33:22 smithi115 conmon[90393]: 2022-05-14T04:33:22.079+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:33:23.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:33:22 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:33:23.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:33:22 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:33:23.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:33:22 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:33:23.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:33:22 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 18336 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:33:23.063 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:33:22 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:33:23.064 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:33:22 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:33:23.064 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:33:22 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:33:23.064 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:33:22 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 18336 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:33:23.360 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:33:23 smithi115 conmon[90393]: 2022-05-14T04:33:23.060+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:33:24.020 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:33:23 smithi115 ceph-mon[78593]: pgmap v9519: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:33:24.020 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:33:23 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:33:24.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:33:23 smithi047 ceph-mon[77147]: pgmap v9519: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:33:24.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:33:23 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:33:24.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:33:24 smithi115 conmon[90393]: 2022-05-14T04:33:24.017+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:33:24.994 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:33:24 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:33:25.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:33:24 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:33:25.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:33:24 smithi115 conmon[90393]: 2022-05-14T04:33:24.991+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:33:25.945 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:33:25 smithi115 ceph-mon[78593]: pgmap v9520: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:33:25.946 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:33:25 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:33:26.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:33:25 smithi047 ceph-mon[77147]: pgmap v9520: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:33:26.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:33:25 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:33:26.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:33:25 smithi115 conmon[90393]: 2022-05-14T04:33:25.943+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:33:26.977 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:33:26 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:33:27.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:33:26 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:33:27.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:33:26 smithi115 conmon[90393]: 2022-05-14T04:33:26.975+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:33:28.003 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:33:27 smithi115 ceph-mon[78593]: pgmap v9521: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:33:28.003 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:33:27 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:33:28.003 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:33:27 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 18340 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:33:28.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:33:27 smithi047 ceph-mon[77147]: pgmap v9521: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:33:28.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:33:27 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:33:28.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:33:27 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 18340 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:33:28.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:33:28 smithi115 conmon[90393]: 2022-05-14T04:33:28.000+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:33:28.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:33:28 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:33:28.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:33:28 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T04:33:28.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:33:28 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T04:33:28.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:33:28 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T04:33:28.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:33:28 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:33:28.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:33:28 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T04:33:28.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:33:28 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T04:33:28.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:33:28 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T04:33:29.293 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:33:29 smithi047 conmon[78521]: 2022-05-14T04:33:29.007+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:33:29.293 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:33:29 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T04:33:29.293 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:33:29 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:33:29.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:33:29 smithi047 conmon[78521]: yield 2022-05-14T04:33:29.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:33:29 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:33:29.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:33:29 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:33:29.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:33:29 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:33:29.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:33:29 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T04:33:29.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:33:29 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:33:29.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:33:29 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T04:33:29.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:33:29 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:33:29.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:33:29 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T04:33:29.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:33:29 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:33:29.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:33:29 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T04:33:29.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:33:29 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:33:29.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:33:29 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T04:33:29.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:33:29 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:33:29.297 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:33:29 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:33:29.297 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:33:29 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:33:29.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:33:29 smithi115 conmon[90393]: 2022-05-14T04:33:29.002+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:33:30.007 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:33:29 smithi115 ceph-mon[78593]: pgmap v9522: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:33:30.008 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:33:29 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:33:30.008 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:33:29 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:33:30.008 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T04:33:30.009 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:33:30.009 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T04:33:30.010 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:33:30.010 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:33:30.010 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:33:30.011 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T04:33:30.012 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:33:30.012 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T04:33:30.012 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:33:30.013 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T04:33:30.014 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:33:30.014 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T04:33:30.015 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:33:30.015 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T04:33:30.016 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:33:30.017 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:33:30.017 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:33:30.018 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:33:29 smithi115 ceph-mon[78593]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T04:33:30.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:33:29 smithi047 ceph-mon[77147]: pgmap v9522: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:33:30.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:33:29 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:33:30.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:33:29 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:33:30.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T04:33:30.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:33:30.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T04:33:30.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:33:30.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:33:30.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:33:30.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T04:33:30.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:33:30.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T04:33:30.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:33:30.049 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T04:33:30.049 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:33:30.050 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T04:33:30.050 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:33:30.050 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T04:33:30.050 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:33:30.051 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:33:30.051 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:33:30.051 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:33:29 smithi047 ceph-mon[77147]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T04:33:30.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:33:30 smithi115 conmon[90393]: 2022-05-14T04:33:30.004+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:33:31.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:33:30 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:33:31.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:33:30 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:33:31.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:33:30 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:33:31.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:33:30 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:33:31.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:33:30 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:33:31.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:33:30 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:33:31.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:33:31 smithi115 conmon[90393]: 2022-05-14T04:33:31.041+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:33:32.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:33:31 smithi047 ceph-mon[77147]: pgmap v9523: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:33:32.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:33:31 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:33:32.043 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:33:31 smithi115 ceph-mon[78593]: pgmap v9523: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:33:32.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:33:31 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:33:32.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:33:32 smithi115 conmon[90393]: 2022-05-14T04:33:32.039+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:33:32.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:33:32 smithi047 conmon[78521]: 2022-05-14T04:33:32.079+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:33:32.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:33:32 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T04:33:32.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:33:32 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:33:32.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:33:32 smithi047 conmon[78521]: yield 2022-05-14T04:33:32.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:33:32 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:33:32.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:33:32 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:33:32.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:33:32 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:33:32.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:33:32 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T04:33:32.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:33:32 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:33:32.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:33:32 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T04:33:32.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:33:32 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:33:32.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:33:32 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T04:33:32.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:33:32 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:33:32.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:33:32 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T04:33:32.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:33:32 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:33:32.547 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:33:32 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T04:33:32.547 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:33:32 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:33:32.547 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:33:32 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:33:32.547 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:33:32 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:33:32.547 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:33:32 smithi047 conmon[78521]: 2022-05-14T04:33:32.080+0000 7f533ff78700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T04:33:32.548 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:33:32 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T04:33:32.548 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:33:32 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:33:32.548 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:33:32 smithi047 conmon[78521]: yield 2022-05-14T04:33:32.548 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:33:32 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:33:32.551 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:33:32 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:33:32.551 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:33:32 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:33:32.551 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:33:32 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T04:33:32.551 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:33:32 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:33:32.552 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:33:32 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T04:33:32.552 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:33:32 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:33:32.552 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:33:32 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T04:33:32.552 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:33:32 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:33:32.552 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:33:32 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T04:33:32.553 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:33:32 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:33:32.553 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:33:32 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T04:33:32.553 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:33:32 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:33:32.554 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:33:32 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:33:32.554 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:33:32 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:33:32.554 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:33:32 smithi047 conmon[78521]: 2022-05-14T04:33:32.554 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:33:32 smithi047 conmon[78521]: During handling of the above exception, another exception occurred: 2022-05-14T04:33:32.555 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:33:32 smithi047 conmon[78521]: 2022-05-14T04:33:32.555 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:33:32 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T04:33:32.555 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:33:32 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T04:33:32.555 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:33:32 smithi047 conmon[78521]: return f(*arg) 2022-05-14T04:33:32.556 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:33:32 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T04:33:32.556 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:33:32 smithi047 conmon[78521]: self._write_client_files(client_files, host) 2022-05-14T04:33:32.556 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:33:32 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T04:33:32.556 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:33:32 smithi047 conmon[78521]: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T04:33:32.556 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:33:32 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T04:33:32.557 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:33:32 smithi047 conmon[78521]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T04:33:32.557 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:33:32 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T04:33:32.557 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:33:32 smithi047 conmon[78521]: return self.event_loop.get_result(coro) 2022-05-14T04:33:32.557 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:33:32 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T04:33:32.558 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:33:32 smithi047 conmon[78521]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T04:33:32.558 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:33:32 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T04:33:32.558 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:33:32 smithi047 conmon[78521]: return self.__get_result() 2022-05-14T04:33:32.558 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:33:32 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T04:33:32.559 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:33:32 smithi047 conmon[78521]: raise self._exception 2022-05-14T04:33:32.559 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:33:32 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T04:33:32.559 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:33:32 smithi047 conmon[78521]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T04:33:32.559 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:33:32 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T04:33:32.560 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:33:32 smithi047 conmon[78521]: conn = await self._remote_connection(host, addr) 2022-05-14T04:33:32.561 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:33:32 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T04:33:32.561 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:33:32 smithi047 conmon[78521]: raise 2022-05-14T04:33:32.561 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:33:32 smithi047 conmon[78521]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T04:33:32.562 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:33:32 smithi047 conmon[78521]: self.gen.throw(type, value, traceback) 2022-05-14T04:33:32.562 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:33:32 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T04:33:32.562 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:33:32 smithi047 conmon[78521]: raise OrchestratorError(msg) 2022-05-14T04:33:32.562 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:33:32 smithi047 conmon[78521]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:33:33.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:33:32 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:33:33.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:33:32 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:33:33.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T04:33:33.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:33:33.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T04:33:33.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:33:33.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:33:33.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:33:33.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T04:33:33.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:33:33.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T04:33:33.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:33:33.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T04:33:33.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:33:33.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T04:33:33.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:33:33.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T04:33:33.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:33:33.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:33:33.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:33:33.047 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:33:32 smithi047 ceph-mon[77147]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T04:33:33.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T04:33:33.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:33:33.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T04:33:33.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:33:33.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:33:33.049 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:33:33.049 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T04:33:33.049 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:33:33.049 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T04:33:33.050 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:33:33.050 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T04:33:33.050 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:33:33.050 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T04:33:33.050 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:33:33.051 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T04:33:33.051 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:33:33.051 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:33:33.051 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:33:33.051 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T04:33:33.052 INFO:journalctl@ceph.mon.b.smithi047.stdout: During handling of the above exception, another exception occurred: 2022-05-14T04:33:33.052 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T04:33:33.052 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T04:33:33.052 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T04:33:33.053 INFO:journalctl@ceph.mon.b.smithi047.stdout: return f(*arg) 2022-05-14T04:33:33.053 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T04:33:33.053 INFO:journalctl@ceph.mon.b.smithi047.stdout: self._write_client_files(client_files, host) 2022-05-14T04:33:33.053 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T04:33:33.054 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T04:33:33.054 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T04:33:33.054 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T04:33:33.054 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T04:33:33.054 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.event_loop.get_result(coro) 2022-05-14T04:33:33.055 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T04:33:33.055 INFO:journalctl@ceph.mon.b.smithi047.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T04:33:33.055 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T04:33:33.055 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.__get_result() 2022-05-14T04:33:33.055 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T04:33:33.056 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise self._exception 2022-05-14T04:33:33.056 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T04:33:33.056 INFO:journalctl@ceph.mon.b.smithi047.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T04:33:33.056 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T04:33:33.057 INFO:journalctl@ceph.mon.b.smithi047.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T04:33:33.057 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T04:33:33.057 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise 2022-05-14T04:33:33.057 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T04:33:33.057 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.gen.throw(type, value, traceback) 2022-05-14T04:33:33.058 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T04:33:33.058 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OrchestratorError(msg) 2022-05-14T04:33:33.058 INFO:journalctl@ceph.mon.b.smithi047.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:33:33.058 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:33:32 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 18345 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:33:33.059 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:33:32 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:33:33.059 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:33:32 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:33:33.060 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T04:33:33.060 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:33:33.060 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T04:33:33.060 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:33:33.061 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:33:33.061 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:33:33.061 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T04:33:33.061 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:33:33.061 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T04:33:33.062 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:33:33.062 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T04:33:33.062 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:33:33.062 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T04:33:33.063 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:33:33.063 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T04:33:33.063 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:33:33.063 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:33:33.063 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:33:33.064 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:33:32 smithi115 ceph-mon[78593]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T04:33:33.064 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T04:33:33.064 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:33:33.065 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T04:33:33.065 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:33:33.066 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:33:33.067 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:33:33.067 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T04:33:33.068 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:33:33.068 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T04:33:33.068 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:33:33.068 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T04:33:33.069 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:33:33.069 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T04:33:33.069 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:33:33.069 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T04:33:33.070 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:33:33.070 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:33:33.070 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:33:33.070 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T04:33:33.070 INFO:journalctl@ceph.mon.c.smithi115.stdout: During handling of the above exception, another exception occurred: 2022-05-14T04:33:33.071 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T04:33:33.071 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T04:33:33.071 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T04:33:33.071 INFO:journalctl@ceph.mon.c.smithi115.stdout: return f(*arg) 2022-05-14T04:33:33.072 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T04:33:33.072 INFO:journalctl@ceph.mon.c.smithi115.stdout: self._write_client_files(client_files, host) 2022-05-14T04:33:33.072 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T04:33:33.072 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T04:33:33.072 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T04:33:33.073 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T04:33:33.073 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T04:33:33.073 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.event_loop.get_result(coro) 2022-05-14T04:33:33.073 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T04:33:33.074 INFO:journalctl@ceph.mon.c.smithi115.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T04:33:33.074 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T04:33:33.074 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.__get_result() 2022-05-14T04:33:33.075 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T04:33:33.075 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise self._exception 2022-05-14T04:33:33.075 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T04:33:33.075 INFO:journalctl@ceph.mon.c.smithi115.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T04:33:33.075 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T04:33:33.076 INFO:journalctl@ceph.mon.c.smithi115.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T04:33:33.076 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T04:33:33.076 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise 2022-05-14T04:33:33.076 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T04:33:33.077 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.gen.throw(type, value, traceback) 2022-05-14T04:33:33.077 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T04:33:33.077 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OrchestratorError(msg) 2022-05-14T04:33:33.077 INFO:journalctl@ceph.mon.c.smithi115.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:33:33.077 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:33:32 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 18345 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:33:33.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:33:33 smithi115 conmon[90393]: 2022-05-14T04:33:33.044+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:33:34.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:33:33 smithi047 ceph-mon[77147]: pgmap v9524: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:33:34.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:33:33 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:33:34.049 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:33:33 smithi115 ceph-mon[78593]: pgmap v9524: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:33:34.049 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:33:33 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:33:34.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:33:34 smithi115 conmon[90393]: 2022-05-14T04:33:34.046+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:33:35.003 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:33:34 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:33:35.003 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:33:34 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:33:35.003 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:33:34 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:33:35.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:33:34 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:33:35.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:33:34 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:33:35.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:33:34 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:33:35.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:33:35 smithi115 conmon[90393]: 2022-05-14T04:33:35.000+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:33:36.003 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:33:35 smithi115 ceph-mon[78593]: pgmap v9525: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:33:36.003 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:33:35 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:33:36.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:33:35 smithi047 ceph-mon[77147]: pgmap v9525: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:33:36.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:33:35 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:33:36.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:33:36 smithi115 conmon[90393]: 2022-05-14T04:33:36.000+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:33:36.977 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:33:36 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:33:37.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:33:36 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:33:37.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:33:36 smithi115 conmon[90393]: 2022-05-14T04:33:36.974+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:33:37.976 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:33:37 smithi115 ceph-mon[78593]: pgmap v9526: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:33:37.977 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:33:37 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:33:37.977 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:33:37 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 18350 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:33:38.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:33:37 smithi047 ceph-mon[77147]: pgmap v9526: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:33:38.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:33:37 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:33:38.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:33:37 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 18350 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:33:38.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:33:37 smithi115 conmon[90393]: 2022-05-14T04:33:37.973+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:33:39.007 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:33:38 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:33:39.008 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:33:38 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:33:39.008 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:33:38 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:33:39.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:33:38 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:33:39.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:33:38 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:33:39.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:33:38 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:33:39.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:33:39 smithi115 conmon[90393]: 2022-05-14T04:33:39.004+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:33:40.016 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:33:39 smithi115 ceph-mon[78593]: pgmap v9527: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:33:40.016 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:33:39 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:33:40.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:33:39 smithi047 ceph-mon[77147]: pgmap v9527: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:33:40.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:33:39 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:33:40.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:33:40 smithi115 conmon[90393]: 2022-05-14T04:33:40.013+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:33:41.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:33:40 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:33:41.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:33:40 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:33:41.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:33:41 smithi115 conmon[90393]: 2022-05-14T04:33:41.038+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:33:42.015 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:33:41 smithi115 ceph-mon[78593]: pgmap v9528: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:33:42.015 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:33:41 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:33:42.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:33:41 smithi047 ceph-mon[77147]: pgmap v9528: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:33:42.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:33:41 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:33:42.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:33:42 smithi115 conmon[90393]: 2022-05-14T04:33:42.012+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:33:43.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:33:42 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:33:43.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:33:42 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:33:43.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:33:42 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:33:43.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:33:42 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 18355 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:33:43.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:33:42 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:33:43.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:33:42 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:33:43.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:33:42 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:33:43.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:33:42 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 18355 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:33:43.360 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:33:43 smithi115 conmon[90393]: 2022-05-14T04:33:43.041+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:33:44.014 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:33:43 smithi115 ceph-mon[78593]: pgmap v9529: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:33:44.015 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:33:43 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:33:44.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:33:43 smithi047 ceph-mon[77147]: pgmap v9529: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:33:44.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:33:43 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:33:44.360 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:33:44 smithi115 conmon[90393]: 2022-05-14T04:33:44.011+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:33:45.016 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:33:44 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:33:45.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:33:44 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:33:45.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:33:45 smithi115 conmon[90393]: 2022-05-14T04:33:45.012+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:33:46.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:33:45 smithi047 ceph-mon[77147]: pgmap v9530: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:33:46.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:33:45 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:33:46.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:33:45 smithi115 ceph-mon[78593]: pgmap v9530: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:33:46.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:33:45 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:33:46.360 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:33:46 smithi115 conmon[90393]: 2022-05-14T04:33:46.042+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:33:47.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:33:46 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:33:47.095 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:33:46 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:33:47.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:33:47 smithi115 conmon[90393]: 2022-05-14T04:33:47.091+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:33:48.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:33:47 smithi047 ceph-mon[77147]: pgmap v9531: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:33:48.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:33:47 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:33:48.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:33:47 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 18360 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:33:48.076 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:33:47 smithi115 ceph-mon[78593]: pgmap v9531: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:33:48.077 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:33:47 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:33:48.077 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:33:47 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 18360 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:33:48.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:33:48 smithi115 conmon[90393]: 2022-05-14T04:33:48.073+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:33:49.042 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:33:48 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:33:49.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:33:48 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:33:49.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:33:49 smithi115 conmon[90393]: 2022-05-14T04:33:49.039+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:33:50.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:33:49 smithi047 ceph-mon[77147]: pgmap v9532: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:33:50.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:33:49 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:33:50.092 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:33:49 smithi115 ceph-mon[78593]: pgmap v9532: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:33:50.092 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:33:49 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:33:50.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:33:50 smithi115 conmon[90393]: 2022-05-14T04:33:50.089+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:33:51.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:33:50 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:33:51.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:33:50 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:33:51.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:33:50 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:33:51.096 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:33:50 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:33:51.096 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:33:50 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:33:51.096 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:33:50 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:33:51.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:33:51 smithi115 conmon[90393]: 2022-05-14T04:33:51.093+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:33:52.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:33:51 smithi047 ceph-mon[77147]: pgmap v9533: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:33:52.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:33:51 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:33:52.049 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:33:51 smithi115 ceph-mon[78593]: pgmap v9533: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:33:52.049 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:33:51 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:33:52.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:33:52 smithi115 conmon[90393]: 2022-05-14T04:33:52.046+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:33:53.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:33:52 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:33:53.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:33:52 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 18365 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:33:53.051 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:33:52 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:33:53.052 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:33:52 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 18365 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:33:53.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:33:53 smithi115 conmon[90393]: 2022-05-14T04:33:53.048+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:33:54.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:33:53 smithi047 ceph-mon[77147]: pgmap v9534: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:33:54.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:33:53 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:33:54.099 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:33:53 smithi115 ceph-mon[78593]: pgmap v9534: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:33:54.100 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:33:53 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:33:54.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:33:54 smithi115 conmon[90393]: 2022-05-14T04:33:54.096+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:33:55.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:33:54 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:33:55.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:33:54 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:33:55.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:33:54 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:33:55.197 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:33:54 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:33:55.198 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:33:54 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:33:55.198 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:33:54 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:33:55.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:33:55 smithi115 conmon[90393]: 2022-05-14T04:33:55.047+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:33:56.038 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:33:55 smithi115 ceph-mon[78593]: pgmap v9535: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:33:56.039 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:33:55 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:33:56.046 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:33:55 smithi047 ceph-mon[77147]: pgmap v9535: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:33:56.046 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:33:55 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:33:56.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:33:56 smithi115 conmon[90393]: 2022-05-14T04:33:56.034+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:33:57.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:33:56 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:33:57.113 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:33:56 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:33:57.114 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:33:56 smithi115 conmon[90393]: 2022-05-14T04:33:56.986+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:33:58.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:33:57 smithi047 ceph-mon[77147]: pgmap v9536: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:33:58.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:33:57 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:33:58.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:33:57 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 18371 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:33:58.110 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:33:57 smithi115 conmon[90393]: 2022-05-14T04:33:57.977+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:33:58.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:33:57 smithi115 ceph-mon[78593]: pgmap v9536: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:33:58.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:33:57 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:33:58.112 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:33:57 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 18371 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:33:59.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:33:58 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:33:59.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:33:58 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:33:59.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:33:58 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:33:59.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:33:58 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:33:59.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:33:58 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:33:59.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:33:58 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:33:59.110 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:33:59 smithi115 conmon[90393]: 2022-05-14T04:33:59.015+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:34:00.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:33:59 smithi047 ceph-mon[77147]: pgmap v9537: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:34:00.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:33:59 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:34:00.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:34:00 smithi115 conmon[90393]: 2022-05-14T04:34:00.014+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:34:00.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:33:59 smithi115 ceph-mon[78593]: pgmap v9537: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:34:00.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:33:59 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:34:01.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:34:00 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:34:01.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:34:00 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:34:01.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:34:01 smithi115 conmon[90393]: 2022-05-14T04:34:01.041+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:34:02.056 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:34:01 smithi115 ceph-mon[78593]: pgmap v9538: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:34:02.057 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:34:01 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:34:02.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:34:01 smithi047 ceph-mon[77147]: pgmap v9538: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:34:02.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:34:01 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:34:02.362 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:34:02 smithi115 conmon[90393]: 2022-05-14T04:34:02.053+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:34:03.094 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:34:02 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:34:03.094 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:34:02 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:34:03.094 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:34:02 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:34:03.094 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:34:02 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 18375 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:34:03.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:34:02 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:34:03.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:34:02 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:34:03.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:34:02 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:34:03.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:34:02 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 18375 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:34:03.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:34:03 smithi115 conmon[90393]: 2022-05-14T04:34:03.090+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:34:04.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:34:03 smithi115 ceph-mon[78593]: pgmap v9539: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:34:04.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:34:03 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:34:04.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:34:03 smithi047 ceph-mon[77147]: pgmap v9539: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:34:04.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:34:03 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:34:04.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:34:04 smithi115 conmon[90393]: 2022-05-14T04:34:04.107+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:34:05.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:34:04 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:34:05.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:34:04 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:34:05.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:34:05 smithi115 conmon[90393]: 2022-05-14T04:34:05.125+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:34:06.101 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:34:05 smithi115 ceph-mon[78593]: pgmap v9540: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:34:06.102 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:34:05 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:34:06.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:34:05 smithi047 ceph-mon[77147]: pgmap v9540: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:34:06.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:34:05 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:34:06.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:34:06 smithi115 conmon[90393]: 2022-05-14T04:34:06.099+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:34:07.106 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:34:06 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:34:07.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:34:06 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:34:07.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:34:07 smithi115 conmon[90393]: 2022-05-14T04:34:07.103+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:34:08.092 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:34:07 smithi115 ceph-mon[78593]: pgmap v9541: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:34:08.093 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:34:07 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:34:08.093 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:34:07 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 18381 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:34:08.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:34:07 smithi047 ceph-mon[77147]: pgmap v9541: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:34:08.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:34:07 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:34:08.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:34:07 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 18381 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:34:08.360 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:34:08 smithi115 conmon[90393]: 2022-05-14T04:34:08.089+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:34:09.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:34:08 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:34:09.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:34:08 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:34:09.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:34:09 smithi115 conmon[90393]: 2022-05-14T04:34:09.121+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:34:10.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:34:09 smithi115 ceph-mon[78593]: pgmap v9542: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:34:10.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:34:09 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:34:10.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:34:09 smithi047 ceph-mon[77147]: pgmap v9542: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:34:10.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:34:09 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:34:10.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:34:10 smithi115 conmon[90393]: 2022-05-14T04:34:10.148+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:34:11.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:34:10 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:34:11.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:34:10 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:34:11.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:34:10 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:34:11.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:34:10 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:34:11.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:34:10 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:34:11.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:34:10 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:34:11.610 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:34:11 smithi115 conmon[90393]: 2022-05-14T04:34:11.149+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:34:12.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:34:11 smithi115 ceph-mon[78593]: pgmap v9543: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:34:12.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:34:11 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:34:12.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:34:11 smithi047 ceph-mon[77147]: pgmap v9543: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:34:12.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:34:11 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:34:12.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:34:12 smithi115 conmon[90393]: 2022-05-14T04:34:12.181+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:34:13.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:34:12 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:34:13.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:34:12 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 18386 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:34:13.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:34:12 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:34:13.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:34:12 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 18386 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:34:13.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:34:13 smithi115 conmon[90393]: 2022-05-14T04:34:13.205+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:34:14.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:34:13 smithi115 ceph-mon[78593]: pgmap v9544: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:34:14.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:34:13 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:34:14.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:34:13 smithi047 ceph-mon[77147]: pgmap v9544: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:34:14.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:34:13 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:34:14.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:34:14 smithi115 conmon[90393]: 2022-05-14T04:34:14.227+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:34:15.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:34:14 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:34:15.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:34:14 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:34:15.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:34:14 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:34:15.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:34:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T04:34:15.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:34:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T04:34:15.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:34:14 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:34:15.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:34:14 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:34:15.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:34:14 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:34:15.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:34:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T04:34:15.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:34:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T04:34:15.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:34:15 smithi115 conmon[90393]: 2022-05-14T04:34:15.273+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:34:16.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:34:15 smithi115 ceph-mon[78593]: pgmap v9545: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:34:16.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:34:15 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:34:16.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:34:15 smithi047 ceph-mon[77147]: pgmap v9545: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:34:16.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:34:15 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:34:16.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:34:16 smithi115 conmon[90393]: 2022-05-14T04:34:16.233+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:34:17.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:34:16 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:34:17.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:34:16 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:34:17.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:34:17 smithi115 conmon[90393]: 2022-05-14T04:34:17.186+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:34:18.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:34:17 smithi115 ceph-mon[78593]: pgmap v9546: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:34:18.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:34:17 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:34:18.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:34:17 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 18390 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:34:18.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:34:17 smithi047 ceph-mon[77147]: pgmap v9546: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:34:18.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:34:17 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:34:18.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:34:17 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 18390 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:34:18.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:34:18 smithi115 conmon[90393]: 2022-05-14T04:34:18.205+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:34:19.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:34:18 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:34:19.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:34:18 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:34:19.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:34:18 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:34:19.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:34:18 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:34:19.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:34:18 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:34:19.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:34:18 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:34:19.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:34:19 smithi115 conmon[90393]: 2022-05-14T04:34:19.171+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:34:20.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:34:19 smithi115 ceph-mon[78593]: pgmap v9547: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:34:20.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:34:19 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:34:20.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:34:19 smithi047 ceph-mon[77147]: pgmap v9547: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:34:20.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:34:19 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:34:20.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:34:20 smithi115 conmon[90393]: 2022-05-14T04:34:20.167+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:34:21.208 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:34:20 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:34:21.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:34:20 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:34:21.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:34:21 smithi115 conmon[90393]: 2022-05-14T04:34:21.206+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:34:22.233 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:34:21 smithi115 ceph-mon[78593]: pgmap v9548: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 7 op/s 2022-05-14T04:34:22.234 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:34:21 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:34:22.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:34:21 smithi047 ceph-mon[77147]: pgmap v9548: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 7 op/s 2022-05-14T04:34:22.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:34:21 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:34:22.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:34:22 smithi115 conmon[90393]: 2022-05-14T04:34:22.231+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:34:23.213 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:34:22 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:34:23.214 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:34:22 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:34:23.214 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:34:22 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:34:23.214 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:34:22 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 18395 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:34:23.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:34:22 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:34:23.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:34:22 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:34:23.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:34:22 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:34:23.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:34:22 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 18395 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:34:23.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:34:23 smithi115 conmon[90393]: 2022-05-14T04:34:23.211+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:34:24.534 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:34:24 smithi115 conmon[90393]: 2022-05-14T04:34:24.238+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:34:24.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:34:24 smithi047 ceph-mon[77147]: pgmap v9549: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 4 op/s 2022-05-14T04:34:24.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:34:24 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:34:24.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:34:24 smithi115 ceph-mon[78593]: pgmap v9549: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 4 op/s 2022-05-14T04:34:24.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:34:24 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:34:25.540 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:34:25 smithi115 conmon[90393]: 2022-05-14T04:34:25.241+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:34:25.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:34:25 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:34:25.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:34:25 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:34:26.540 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:34:26 smithi115 conmon[90393]: 2022-05-14T04:34:26.267+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:34:26.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:34:26 smithi047 ceph-mon[77147]: pgmap v9550: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 7 op/s 2022-05-14T04:34:26.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:34:26 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:34:26.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:34:26 smithi115 ceph-mon[78593]: pgmap v9550: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 7 op/s 2022-05-14T04:34:26.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:34:26 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:34:27.616 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:34:27 smithi115 conmon[90393]: 2022-05-14T04:34:27.314+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:34:27.616 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:34:27 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:34:27.616 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:34:27 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 18400 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:34:28.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:34:27 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:34:28.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:34:27 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 18400 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:34:28.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:34:28 smithi115 conmon[90393]: 2022-05-14T04:34:28.312+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:34:28.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:34:28 smithi115 ceph-mon[78593]: pgmap v9551: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 4 op/s 2022-05-14T04:34:28.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:34:28 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:34:29.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:34:28 smithi047 ceph-mon[77147]: pgmap v9551: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 4 op/s 2022-05-14T04:34:29.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:34:28 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:34:29.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:34:29 smithi115 conmon[90393]: 2022-05-14T04:34:29.346+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:34:29.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:34:29 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:34:29.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:34:29 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:34:30.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:34:29 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:34:30.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:34:29 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:34:30.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:34:30 smithi115 conmon[90393]: 2022-05-14T04:34:30.366+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:34:30.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:34:30 smithi115 ceph-mon[78593]: pgmap v9552: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 4 op/s 2022-05-14T04:34:30.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:34:30 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:34:30.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:34:30 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:34:30.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:34:30 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:34:31.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:34:30 smithi047 ceph-mon[77147]: pgmap v9552: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 4 op/s 2022-05-14T04:34:31.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:34:30 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:34:31.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:34:30 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:34:31.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:34:30 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:34:31.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:34:31 smithi115 conmon[90393]: 2022-05-14T04:34:31.337+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:34:31.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:34:31 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:34:32.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:34:31 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:34:32.562 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:34:32 smithi115 conmon[90393]: 2022-05-14T04:34:32.330+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:34:32.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:34:32 smithi115 ceph-mon[78593]: pgmap v9553: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 7 op/s 2022-05-14T04:34:32.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:34:32 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T04:34:32.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:34:32 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T04:34:32.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:34:32 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T04:34:32.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:34:32 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:34:32.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:34:32 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 18405 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:34:33.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:34:32 smithi047 ceph-mon[77147]: pgmap v9553: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 7 op/s 2022-05-14T04:34:33.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:34:32 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T04:34:33.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:34:32 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T04:34:33.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:34:32 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T04:34:33.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:34:32 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:34:33.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:34:32 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 18405 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:34:33.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:34:33 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:34:33.610 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:34:33 smithi115 conmon[90393]: 2022-05-14T04:34:33.354+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:34:34.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:34:33 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:34:34.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:34:34 smithi115 conmon[90393]: 2022-05-14T04:34:34.352+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:34:34.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:34:34 smithi115 ceph-mon[78593]: pgmap v9554: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:34:34.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:34:34 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:34:34.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:34:34 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:34:34.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:34:34 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:34:35.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:34:34 smithi047 ceph-mon[77147]: pgmap v9554: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:34:35.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:34:34 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:34:35.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:34:34 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:34:35.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:34:34 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:34:35.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:34:35 smithi047 conmon[78521]: 2022-05-14T04:34:35.183+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:34:35.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:34:35 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T04:34:35.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:34:35 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:34:35.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:34:35 smithi047 conmon[78521]: yield 2022-05-14T04:34:35.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:34:35 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:34:35.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:34:35 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:34:35.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:34:35 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:34:35.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:34:35 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T04:34:35.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:34:35 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:34:35.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:34:35 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T04:34:35.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:34:35 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:34:35.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:34:35 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T04:34:35.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:34:35 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:34:35.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:34:35 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T04:34:35.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:34:35 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:34:35.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:34:35 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T04:34:35.547 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:34:35 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:34:35.547 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:34:35 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:34:35.547 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:34:35 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:34:35.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:34:35 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:34:35.610 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:34:35 smithi115 conmon[90393]: 2022-05-14T04:34:35.348+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:34:36.045 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:34:35 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:34:36.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:34:36 smithi115 conmon[90393]: 2022-05-14T04:34:36.346+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:34:36.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:34:36 smithi115 ceph-mon[78593]: pgmap v9555: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:34:36.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:34:36 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:34:36.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T04:34:36.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:34:36.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T04:34:36.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:34:36.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:34:36.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:34:36.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T04:34:36.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:34:36.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T04:34:36.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:34:36.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T04:34:36.613 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:34:36.613 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T04:34:36.613 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:34:36.613 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T04:34:36.614 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:34:36.614 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:34:36.614 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:34:36.614 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:34:36 smithi115 ceph-mon[78593]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T04:34:36.614 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:34:36 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:34:37.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:34:36 smithi047 ceph-mon[77147]: pgmap v9555: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:34:37.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:34:36 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:34:37.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T04:34:37.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:34:37.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T04:34:37.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:34:37.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:34:37.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:34:37.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T04:34:37.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:34:37.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T04:34:37.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:34:37.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T04:34:37.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:34:37.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T04:34:37.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:34:37.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T04:34:37.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:34:37.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:34:37.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:34:37.048 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:34:36 smithi047 ceph-mon[77147]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T04:34:37.048 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:34:36 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:34:37.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:34:37 smithi115 conmon[90393]: 2022-05-14T04:34:37.349+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:34:37.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:34:37 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:34:37.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:34:37 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 18410 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:34:38.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:34:37 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:34:38.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:34:37 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 18410 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:34:38.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:34:38 smithi047 conmon[78521]: 2022-05-14T04:34:38.257+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:34:38.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:34:38 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T04:34:38.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:34:38 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:34:38.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:34:38 smithi047 conmon[78521]: yield 2022-05-14T04:34:38.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:34:38 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:34:38.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:34:38 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:34:38.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:34:38 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:34:38.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:34:38 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T04:34:38.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:34:38 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:34:38.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:34:38 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T04:34:38.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:34:38 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:34:38.547 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:34:38 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T04:34:38.547 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:34:38 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:34:38.547 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:34:38 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T04:34:38.547 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:34:38 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:34:38.548 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:34:38 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T04:34:38.548 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:34:38 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:34:38.548 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:34:38 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:34:38.548 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:34:38 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:34:38.549 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:34:38 smithi047 conmon[78521]: 2022-05-14T04:34:38.257+0000 7f5344f82700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T04:34:38.549 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:34:38 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T04:34:38.549 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:34:38 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:34:38.549 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:34:38 smithi047 conmon[78521]: yield 2022-05-14T04:34:38.550 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:34:38 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:34:38.550 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:34:38 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:34:38.550 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:34:38 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:34:38.550 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:34:38 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T04:34:38.551 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:34:38 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:34:38.551 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:34:38 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T04:34:38.551 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:34:38 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:34:38.551 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:34:38 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T04:34:38.552 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:34:38 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:34:38.552 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:34:38 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T04:34:38.553 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:34:38 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:34:38.553 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:34:38 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T04:34:38.553 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:34:38 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:34:38.554 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:34:38 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:34:38.554 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:34:38 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:34:38.554 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:34:38 smithi047 conmon[78521]: 2022-05-14T04:34:38.554 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:34:38 smithi047 conmon[78521]: During handling of the above exception, another exception occurred: 2022-05-14T04:34:38.555 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:34:38 smithi047 conmon[78521]: 2022-05-14T04:34:38.555 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:34:38 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T04:34:38.555 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:34:38 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T04:34:38.555 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:34:38 smithi047 conmon[78521]: return f(*arg) 2022-05-14T04:34:38.556 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:34:38 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T04:34:38.556 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:34:38 smithi047 conmon[78521]: self._write_client_files(client_files, host) 2022-05-14T04:34:38.557 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:34:38 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T04:34:38.557 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:34:38 smithi047 conmon[78521]: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T04:34:38.557 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:34:38 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T04:34:38.557 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:34:38 smithi047 conmon[78521]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T04:34:38.558 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:34:38 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T04:34:38.558 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:34:38 smithi047 conmon[78521]: return self.event_loop.get_result(coro) 2022-05-14T04:34:38.558 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:34:38 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T04:34:38.558 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:34:38 smithi047 conmon[78521]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T04:34:38.559 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:34:38 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T04:34:38.559 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:34:38 smithi047 conmon[78521]: return self.__get_result() 2022-05-14T04:34:38.559 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:34:38 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T04:34:38.559 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:34:38 smithi047 conmon[78521]: raise self._exception 2022-05-14T04:34:38.560 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:34:38 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T04:34:38.560 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:34:38 smithi047 conmon[78521]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T04:34:38.560 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:34:38 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T04:34:38.560 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:34:38 smithi047 conmon[78521]: conn = await self._remote_connection(host, addr) 2022-05-14T04:34:38.561 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:34:38 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T04:34:38.561 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:34:38 smithi047 conmon[78521]: raise 2022-05-14T04:34:38.561 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:34:38 smithi047 conmon[78521]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T04:34:38.561 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:34:38 smithi047 conmon[78521]: self.gen.throw(type, value, traceback) 2022-05-14T04:34:38.562 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:34:38 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T04:34:38.562 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:34:38 smithi047 conmon[78521]: raise OrchestratorError(msg) 2022-05-14T04:34:38.562 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:34:38 smithi047 conmon[78521]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:34:38.581 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:34:38 smithi115 conmon[90393]: 2022-05-14T04:34:38.305+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:34:38.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:34:38 smithi115 ceph-mon[78593]: pgmap v9556: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:34:38.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:34:38 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:34:38.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:34:38 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:34:38.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:34:38 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:34:39.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:34:38 smithi047 ceph-mon[77147]: pgmap v9556: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:34:39.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:34:38 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:34:39.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:34:38 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:34:39.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:34:38 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:34:39.586 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:34:39 smithi115 conmon[90393]: 2022-05-14T04:34:39.282+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:34:39.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:34:39 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:34:39.860 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T04:34:39.860 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:34:39.860 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T04:34:39.860 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:34:39.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:34:39.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:34:39.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T04:34:39.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:34:39.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T04:34:39.863 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:34:39.863 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T04:34:39.863 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:34:39.864 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T04:34:39.864 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:34:39.864 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T04:34:39.864 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:34:39.865 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:34:39.865 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:34:39.865 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:34:39 smithi115 ceph-mon[78593]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T04:34:39.866 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T04:34:39.866 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:34:39.866 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T04:34:39.866 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:34:39.867 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:34:39.867 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:34:39.867 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T04:34:39.867 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:34:39.868 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T04:34:39.868 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:34:39.868 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T04:34:39.868 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:34:39.868 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T04:34:39.869 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:34:39.869 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T04:34:39.869 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:34:39.869 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:34:39.870 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:34:39.870 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T04:34:39.870 INFO:journalctl@ceph.mon.c.smithi115.stdout: During handling of the above exception, another exception occurred: 2022-05-14T04:34:39.870 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T04:34:39.870 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T04:34:39.871 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T04:34:39.871 INFO:journalctl@ceph.mon.c.smithi115.stdout: return f(*arg) 2022-05-14T04:34:39.871 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T04:34:39.871 INFO:journalctl@ceph.mon.c.smithi115.stdout: self._write_client_files(client_files, host) 2022-05-14T04:34:39.872 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T04:34:39.872 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T04:34:39.872 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T04:34:39.872 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T04:34:39.872 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T04:34:39.873 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.event_loop.get_result(coro) 2022-05-14T04:34:39.873 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T04:34:39.873 INFO:journalctl@ceph.mon.c.smithi115.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T04:34:39.874 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T04:34:39.874 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.__get_result() 2022-05-14T04:34:39.874 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T04:34:39.874 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise self._exception 2022-05-14T04:34:39.874 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T04:34:39.875 INFO:journalctl@ceph.mon.c.smithi115.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T04:34:39.875 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T04:34:39.875 INFO:journalctl@ceph.mon.c.smithi115.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T04:34:39.875 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T04:34:39.876 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise 2022-05-14T04:34:39.876 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T04:34:39.876 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.gen.throw(type, value, traceback) 2022-05-14T04:34:39.876 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T04:34:39.877 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OrchestratorError(msg) 2022-05-14T04:34:39.877 INFO:journalctl@ceph.mon.c.smithi115.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:34:39.877 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:34:39 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:34:40.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:34:39 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:34:40.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T04:34:40.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:34:40.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T04:34:40.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:34:40.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:34:40.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:34:40.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T04:34:40.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:34:40.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T04:34:40.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:34:40.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T04:34:40.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:34:40.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T04:34:40.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:34:40.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T04:34:40.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:34:40.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:34:40.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:34:40.048 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:34:39 smithi047 ceph-mon[77147]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T04:34:40.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T04:34:40.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:34:40.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T04:34:40.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:34:40.049 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:34:40.049 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:34:40.049 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T04:34:40.049 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:34:40.050 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T04:34:40.050 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:34:40.050 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T04:34:40.050 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:34:40.050 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T04:34:40.051 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:34:40.051 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T04:34:40.051 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:34:40.051 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:34:40.052 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:34:40.052 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T04:34:40.052 INFO:journalctl@ceph.mon.b.smithi047.stdout: During handling of the above exception, another exception occurred: 2022-05-14T04:34:40.052 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T04:34:40.052 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T04:34:40.053 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T04:34:40.053 INFO:journalctl@ceph.mon.b.smithi047.stdout: return f(*arg) 2022-05-14T04:34:40.053 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T04:34:40.053 INFO:journalctl@ceph.mon.b.smithi047.stdout: self._write_client_files(client_files, host) 2022-05-14T04:34:40.054 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T04:34:40.054 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T04:34:40.054 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T04:34:40.054 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T04:34:40.054 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T04:34:40.055 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.event_loop.get_result(coro) 2022-05-14T04:34:40.055 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T04:34:40.055 INFO:journalctl@ceph.mon.b.smithi047.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T04:34:40.055 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T04:34:40.056 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.__get_result() 2022-05-14T04:34:40.056 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T04:34:40.056 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise self._exception 2022-05-14T04:34:40.056 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T04:34:40.056 INFO:journalctl@ceph.mon.b.smithi047.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T04:34:40.057 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T04:34:40.057 INFO:journalctl@ceph.mon.b.smithi047.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T04:34:40.057 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T04:34:40.057 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise 2022-05-14T04:34:40.058 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T04:34:40.058 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.gen.throw(type, value, traceback) 2022-05-14T04:34:40.058 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T04:34:40.058 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OrchestratorError(msg) 2022-05-14T04:34:40.058 INFO:journalctl@ceph.mon.b.smithi047.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:34:40.059 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:34:39 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:34:40.587 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:34:40 smithi115 conmon[90393]: 2022-05-14T04:34:40.238+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:34:40.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:34:40 smithi115 ceph-mon[78593]: pgmap v9557: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:34:40.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:34:40 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:34:41.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:34:40 smithi047 ceph-mon[77147]: pgmap v9557: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:34:41.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:34:40 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:34:41.593 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:34:41 smithi115 conmon[90393]: 2022-05-14T04:34:41.208+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:34:41.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:34:41 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:34:42.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:34:41 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:34:42.594 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:34:42 smithi115 conmon[90393]: 2022-05-14T04:34:42.231+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:34:42.864 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:34:42 smithi115 ceph-mon[78593]: pgmap v9558: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:34:42.864 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:34:42 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:34:42.864 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:34:42 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:34:42.865 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:34:42 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:34:42.865 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:34:42 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 18415 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:34:43.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:34:42 smithi047 ceph-mon[77147]: pgmap v9558: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:34:43.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:34:42 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:34:43.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:34:42 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:34:43.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:34:42 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:34:43.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:34:42 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 18415 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:34:43.599 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:34:43 smithi115 conmon[90393]: 2022-05-14T04:34:43.261+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:34:43.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:34:43 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:34:44.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:34:43 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:34:44.600 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:34:44 smithi115 conmon[90393]: 2022-05-14T04:34:44.251+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:34:44.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:34:44 smithi115 ceph-mon[78593]: pgmap v9559: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:34:44.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:34:44 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:34:45.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:34:44 smithi047 ceph-mon[77147]: pgmap v9559: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:34:45.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:34:44 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:34:45.605 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:34:45 smithi115 conmon[90393]: 2022-05-14T04:34:45.298+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:34:45.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:34:45 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:34:46.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:34:45 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:34:46.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:34:46 smithi115 conmon[90393]: 2022-05-14T04:34:46.309+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:34:46.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:34:46 smithi115 ceph-mon[78593]: pgmap v9560: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:34:46.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:34:46 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:34:47.054 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:34:46 smithi047 ceph-mon[77147]: pgmap v9560: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:34:47.054 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:34:46 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:34:47.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:34:47 smithi115 conmon[90393]: 2022-05-14T04:34:47.269+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:34:48.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:34:47 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:34:48.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:34:47 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 18420 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:34:48.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:34:47 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:34:48.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:34:47 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 18420 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:34:48.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:34:48 smithi115 conmon[90393]: 2022-05-14T04:34:48.252+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:34:49.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:34:48 smithi047 ceph-mon[77147]: pgmap v9561: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:34:49.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:34:48 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:34:49.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:34:48 smithi115 ceph-mon[78593]: pgmap v9561: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:34:49.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:34:48 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:34:49.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:34:49 smithi115 conmon[90393]: 2022-05-14T04:34:49.251+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:34:50.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:34:49 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:34:50.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:34:49 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:34:50.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:34:50 smithi115 conmon[90393]: 2022-05-14T04:34:50.279+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:34:51.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:34:50 smithi047 ceph-mon[77147]: pgmap v9562: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:34:51.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:34:50 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:34:51.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:34:50 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:34:51.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:34:50 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:34:51.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:34:50 smithi115 ceph-mon[78593]: pgmap v9562: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:34:51.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:34:50 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:34:51.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:34:50 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:34:51.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:34:50 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:34:51.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:34:51 smithi115 conmon[90393]: 2022-05-14T04:34:51.241+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:34:52.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:34:51 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:34:52.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:34:51 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:34:52.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:34:52 smithi115 conmon[90393]: 2022-05-14T04:34:52.223+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:34:53.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:34:52 smithi047 ceph-mon[77147]: pgmap v9563: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:34:53.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:34:52 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:34:53.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:34:52 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 18425 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:34:53.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:34:52 smithi115 ceph-mon[78593]: pgmap v9563: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:34:53.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:34:52 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:34:53.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:34:52 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 18425 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:34:53.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:34:53 smithi115 conmon[90393]: 2022-05-14T04:34:53.182+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:34:54.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:34:53 smithi047 ceph-mon[77147]: pgmap v9564: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:34:54.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:34:53 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:34:54.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:34:53 smithi115 ceph-mon[78593]: pgmap v9564: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:34:54.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:34:53 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:34:54.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:34:54 smithi115 conmon[90393]: 2022-05-14T04:34:54.135+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:34:55.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:34:54 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:34:55.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:34:54 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:34:55.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:34:54 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:34:55.112 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:34:54 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:34:55.112 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:34:54 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:34:55.113 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:34:54 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:34:55.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:34:55 smithi115 conmon[90393]: 2022-05-14T04:34:55.133+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:34:56.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:34:55 smithi047 ceph-mon[77147]: pgmap v9565: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:34:56.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:34:55 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:34:56.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:34:55 smithi115 ceph-mon[78593]: pgmap v9565: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:34:56.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:34:55 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:34:56.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:34:56 smithi115 conmon[90393]: 2022-05-14T04:34:56.126+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:34:57.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:34:56 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:34:57.084 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:34:56 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:34:57.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:34:57 smithi115 conmon[90393]: 2022-05-14T04:34:57.081+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:34:58.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:34:57 smithi047 ceph-mon[77147]: pgmap v9566: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:34:58.045 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:34:57 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:34:58.045 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:34:57 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 18431 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:34:58.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:34:57 smithi115 ceph-mon[78593]: pgmap v9566: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:34:58.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:34:57 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:34:58.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:34:57 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 18431 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:34:58.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:34:58 smithi115 conmon[90393]: 2022-05-14T04:34:58.131+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:34:59.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:34:58 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:34:59.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:34:58 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:34:59.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:34:58 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:34:59.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:34:58 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:34:59.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:34:58 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:34:59.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:34:58 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:34:59.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:34:59 smithi115 conmon[90393]: 2022-05-14T04:34:59.124+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:35:00.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:34:59 smithi047 ceph-mon[77147]: pgmap v9567: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:35:00.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:34:59 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:35:00.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:34:59 smithi115 ceph-mon[78593]: pgmap v9567: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:35:00.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:34:59 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:35:00.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:35:00 smithi115 conmon[90393]: 2022-05-14T04:35:00.118+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:35:01.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:35:00 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:35:01.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:35:00 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:35:01.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:35:01 smithi115 conmon[90393]: 2022-05-14T04:35:01.121+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:35:02.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:35:01 smithi047 ceph-mon[77147]: pgmap v9568: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:35:02.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:35:01 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:35:02.081 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:35:01 smithi115 ceph-mon[78593]: pgmap v9568: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:35:02.081 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:35:01 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:35:02.360 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:35:02 smithi115 conmon[90393]: 2022-05-14T04:35:02.078+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:35:03.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:35:02 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:35:03.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:35:02 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:35:03.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:35:02 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:35:03.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:35:02 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 18436 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:35:03.090 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:35:02 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:35:03.090 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:35:02 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:35:03.091 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:35:02 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:35:03.091 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:35:02 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 18436 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:35:03.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:35:03 smithi115 conmon[90393]: 2022-05-14T04:35:03.087+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:35:04.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:35:03 smithi047 ceph-mon[77147]: pgmap v9569: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:35:04.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:35:03 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:35:04.080 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:35:03 smithi115 ceph-mon[78593]: pgmap v9569: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:35:04.080 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:35:03 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:35:04.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:35:04 smithi115 conmon[90393]: 2022-05-14T04:35:04.077+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:35:05.040 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:35:04 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:35:05.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:35:04 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:35:05.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:35:05 smithi115 conmon[90393]: 2022-05-14T04:35:05.038+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:35:06.005 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:35:05 smithi115 ceph-mon[78593]: pgmap v9570: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:35:06.005 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:35:05 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:35:06.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:35:05 smithi047 ceph-mon[77147]: pgmap v9570: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:35:06.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:35:05 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:35:06.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:35:06 smithi115 conmon[90393]: 2022-05-14T04:35:06.002+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:35:06.981 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:35:06 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:35:07.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:35:06 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:35:07.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:35:06 smithi115 conmon[90393]: 2022-05-14T04:35:06.977+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:35:07.989 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:35:07 smithi115 ceph-mon[78593]: pgmap v9571: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:35:07.990 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:35:07 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:35:07.990 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:35:07 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 18440 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:35:08.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:35:07 smithi047 ceph-mon[77147]: pgmap v9571: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:35:08.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:35:07 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:35:08.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:35:07 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 18440 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:35:08.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:35:07 smithi115 conmon[90393]: 2022-05-14T04:35:07.987+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:35:08.974 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:35:08 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:35:09.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:35:08 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:35:09.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:35:08 smithi115 conmon[90393]: 2022-05-14T04:35:08.972+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:35:10.013 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:35:09 smithi115 ceph-mon[78593]: pgmap v9572: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:35:10.014 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:35:09 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:35:10.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:35:09 smithi047 ceph-mon[77147]: pgmap v9572: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:35:10.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:35:09 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:35:10.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:35:10 smithi115 conmon[90393]: 2022-05-14T04:35:10.011+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:35:10.974 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:35:10 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:35:10.974 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:35:10 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:35:10.975 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:35:10 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:35:11.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:35:10 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:35:11.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:35:10 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:35:11.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:35:10 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:35:11.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:35:10 smithi115 conmon[90393]: 2022-05-14T04:35:10.971+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:35:11.985 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:35:11 smithi115 ceph-mon[78593]: pgmap v9573: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:35:11.986 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:35:11 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:35:12.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:35:11 smithi047 ceph-mon[77147]: pgmap v9573: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:35:12.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:35:11 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:35:12.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:35:11 smithi115 conmon[90393]: 2022-05-14T04:35:11.983+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:35:12.951 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:35:12 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:35:12.952 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:35:12 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 18445 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:35:13.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:35:12 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:35:13.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:35:12 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 18445 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:35:13.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:35:12 smithi115 conmon[90393]: 2022-05-14T04:35:12.949+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:35:14.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:35:13 smithi047 ceph-mon[77147]: pgmap v9574: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:35:14.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:35:13 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:35:14.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:35:13 smithi115 conmon[90393]: 2022-05-14T04:35:13.942+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:35:14.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:35:13 smithi115 ceph-mon[78593]: pgmap v9574: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:35:14.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:35:13 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:35:14.956 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:35:14 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:35:14.957 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:35:14 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:35:14.957 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:35:14 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:35:14.957 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:35:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T04:35:14.958 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:35:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T04:35:15.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:35:14 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:35:15.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:35:14 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:35:15.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:35:14 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:35:15.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:35:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T04:35:15.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:35:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T04:35:15.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:35:14 smithi115 conmon[90393]: 2022-05-14T04:35:14.954+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:35:15.964 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:35:15 smithi115 ceph-mon[78593]: pgmap v9575: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:35:15.965 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:35:15 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:35:16.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:35:15 smithi047 ceph-mon[77147]: pgmap v9575: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:35:16.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:35:15 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:35:16.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:35:15 smithi115 conmon[90393]: 2022-05-14T04:35:15.962+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:35:17.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:35:16 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:35:17.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:35:16 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:35:17.110 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:35:16 smithi115 conmon[90393]: 2022-05-14T04:35:16.932+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:35:17.971 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:35:17 smithi115 ceph-mon[78593]: pgmap v9576: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:35:17.972 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:35:17 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:35:17.972 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:35:17 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 18450 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:35:18.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:35:17 smithi047 ceph-mon[77147]: pgmap v9576: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:35:18.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:35:17 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:35:18.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:35:17 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 18450 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:35:18.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:35:17 smithi115 conmon[90393]: 2022-05-14T04:35:17.968+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:35:19.005 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:35:18 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:35:19.005 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:35:18 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:35:19.006 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:35:18 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:35:19.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:35:18 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:35:19.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:35:18 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:35:19.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:35:18 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:35:19.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:35:19 smithi115 conmon[90393]: 2022-05-14T04:35:19.002+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:35:19.977 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:35:19 smithi115 ceph-mon[78593]: pgmap v9577: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:35:19.977 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:35:19 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:35:20.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:35:19 smithi047 ceph-mon[77147]: pgmap v9577: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:35:20.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:35:19 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:35:20.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:35:19 smithi115 conmon[90393]: 2022-05-14T04:35:19.974+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:35:21.012 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:35:20 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:35:21.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:35:20 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:35:21.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:35:21 smithi115 conmon[90393]: 2022-05-14T04:35:21.009+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:35:22.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:35:21 smithi047 ceph-mon[77147]: pgmap v9578: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:35:22.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:35:21 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:35:22.058 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:35:21 smithi115 ceph-mon[78593]: pgmap v9578: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:35:22.058 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:35:21 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:35:22.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:35:22 smithi115 conmon[90393]: 2022-05-14T04:35:22.055+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:35:23.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:35:22 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:35:23.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:35:22 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:35:23.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:35:22 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:35:23.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:35:22 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 18455 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:35:23.065 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:35:22 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:35:23.065 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:35:22 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:35:23.065 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:35:22 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:35:23.066 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:35:22 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 18455 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:35:23.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:35:23 smithi115 conmon[90393]: 2022-05-14T04:35:23.061+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:35:24.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:35:23 smithi047 ceph-mon[77147]: pgmap v9579: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:35:24.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:35:23 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:35:24.106 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:35:23 smithi115 ceph-mon[78593]: pgmap v9579: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:35:24.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:35:23 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:35:24.360 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:35:24 smithi115 conmon[90393]: 2022-05-14T04:35:24.103+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:35:25.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:35:24 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:35:25.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:35:24 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:35:25.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:35:25 smithi115 conmon[90393]: 2022-05-14T04:35:25.126+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:35:26.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:35:25 smithi047 ceph-mon[77147]: pgmap v9580: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:35:26.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:35:25 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:35:26.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:35:25 smithi115 ceph-mon[78593]: pgmap v9580: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:35:26.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:35:25 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:35:26.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:35:26 smithi115 conmon[90393]: 2022-05-14T04:35:26.141+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:35:27.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:35:26 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:35:27.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:35:26 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:35:27.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:35:27 smithi115 conmon[90393]: 2022-05-14T04:35:27.168+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:35:28.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:35:27 smithi047 ceph-mon[77147]: pgmap v9581: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:35:28.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:35:27 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:35:28.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:35:27 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 18461 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:35:28.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:35:27 smithi115 ceph-mon[78593]: pgmap v9581: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:35:28.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:35:27 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:35:28.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:35:27 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 18461 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:35:28.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:35:28 smithi115 conmon[90393]: 2022-05-14T04:35:28.158+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:35:29.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:35:28 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:35:29.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:35:28 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:35:29.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:35:29 smithi115 conmon[90393]: 2022-05-14T04:35:29.129+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:35:30.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:35:29 smithi047 ceph-mon[77147]: pgmap v9582: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:35:30.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:35:29 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:35:30.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:35:29 smithi115 ceph-mon[78593]: pgmap v9582: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:35:30.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:35:29 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:35:30.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:35:30 smithi115 conmon[90393]: 2022-05-14T04:35:30.158+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:35:31.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:35:30 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:35:31.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:35:30 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:35:31.045 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:35:30 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:35:31.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:35:30 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:35:31.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:35:30 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:35:31.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:35:30 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:35:31.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:35:31 smithi115 conmon[90393]: 2022-05-14T04:35:31.137+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:35:32.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:35:31 smithi047 ceph-mon[77147]: pgmap v9583: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:35:32.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:35:31 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:35:32.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:35:31 smithi115 ceph-mon[78593]: pgmap v9583: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:35:32.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:35:31 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:35:32.610 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:35:32 smithi115 conmon[90393]: 2022-05-14T04:35:32.156+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:35:33.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:35:32 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:35:33.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:35:32 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 18466 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:35:33.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:35:32 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:35:33.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:35:32 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 18466 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:35:33.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:35:33 smithi115 conmon[90393]: 2022-05-14T04:35:33.121+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:35:34.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:35:33 smithi047 ceph-mon[77147]: pgmap v9584: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:35:34.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:35:33 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:35:34.092 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:35:33 smithi115 ceph-mon[78593]: pgmap v9584: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:35:34.092 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:35:33 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:35:34.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:35:34 smithi115 conmon[90393]: 2022-05-14T04:35:34.089+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:35:35.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:35:34 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:35:35.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:35:34 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:35:35.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:35:34 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:35:35.054 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:35:34 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:35:35.054 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:35:34 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:35:35.054 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:35:34 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:35:35.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:35:35 smithi115 conmon[90393]: 2022-05-14T04:35:35.050+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:35:36.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:35:35 smithi047 ceph-mon[77147]: pgmap v9585: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:35:36.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:35:35 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:35:36.053 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:35:35 smithi115 ceph-mon[78593]: pgmap v9585: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:35:36.054 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:35:35 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:35:36.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:35:36 smithi115 conmon[90393]: 2022-05-14T04:35:36.050+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:35:37.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:35:36 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:35:37.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:35:37 smithi115 conmon[90393]: 2022-05-14T04:35:37.005+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:35:37.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:35:36 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:35:38.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:35:37 smithi047 ceph-mon[77147]: pgmap v9586: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:35:38.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:35:37 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:35:38.045 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:35:37 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 18471 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:35:38.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:35:37 smithi115 conmon[90393]: 2022-05-14T04:35:37.991+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:35:38.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:35:37 smithi115 ceph-mon[78593]: pgmap v9586: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:35:38.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:35:37 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:35:38.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:35:37 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 18471 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:35:38.778 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:35:38 smithi047 conmon[78521]: 2022-05-14T04:35:38.736+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:35:38.779 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:35:38 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T04:35:38.779 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:35:38 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:35:38.779 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:35:38 smithi047 conmon[78521]: yield 2022-05-14T04:35:38.779 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:35:38 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:35:38.780 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:35:38 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:35:38.780 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:35:38 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:35:38.780 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:35:38 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T04:35:38.780 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:35:38 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:35:38.780 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:35:38 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T04:35:38.781 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:35:38 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:35:38.781 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:35:38 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T04:35:38.781 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:35:38 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:35:38.781 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:35:38 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T04:35:38.782 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:35:38 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:35:38.782 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:35:38 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T04:35:38.782 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:35:38 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:35:38.782 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:35:38 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:35:38.782 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:35:38 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:35:38.783 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:35:38 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:35:38.783 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:35:38 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:35:38.784 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:35:38 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:35:38.784 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:35:38 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T04:35:38.784 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:35:38 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T04:35:38.784 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:35:38 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T04:35:39.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:35:39 smithi115 conmon[90393]: 2022-05-14T04:35:39.011+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:35:39.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:35:38 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:35:39.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:35:38 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:35:39.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:35:38 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:35:39.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:35:38 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T04:35:39.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:35:38 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T04:35:39.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:35:38 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T04:35:40.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:35:39 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:35:40.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T04:35:40.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:35:40.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T04:35:40.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:35:40.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:35:40.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:35:40.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T04:35:40.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:35:40.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T04:35:40.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:35:40.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T04:35:40.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:35:40.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T04:35:40.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:35:40.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T04:35:40.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:35:40.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:35:40.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:35:40.048 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:35:39 smithi047 ceph-mon[77147]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T04:35:40.048 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:35:39 smithi047 ceph-mon[77147]: pgmap v9587: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:35:40.048 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:35:39 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:35:40.049 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:35:39 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:35:40.049 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T04:35:40.049 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:35:40.050 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T04:35:40.050 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:35:40.050 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:35:40.050 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:35:40.050 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T04:35:40.051 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:35:40.051 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T04:35:40.051 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:35:40.052 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T04:35:40.052 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:35:40.052 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T04:35:40.052 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:35:40.053 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T04:35:40.053 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:35:40.053 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:35:40.054 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:35:40.054 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:35:39 smithi115 ceph-mon[78593]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T04:35:40.054 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:35:39 smithi115 ceph-mon[78593]: pgmap v9587: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:35:40.055 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:35:39 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:35:40.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:35:40 smithi115 conmon[90393]: 2022-05-14T04:35:40.041+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:35:41.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:35:40 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:35:41.114 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:35:41 smithi115 conmon[90393]: 2022-05-14T04:35:41.023+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:35:41.114 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:35:40 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:35:42.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:35:41 smithi047 ceph-mon[77147]: pgmap v9588: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:35:42.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:35:41 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:35:42.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:35:41 smithi047 conmon[78521]: 2022-05-14T04:35:41.808+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:35:42.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:35:41 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T04:35:42.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:35:41 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:35:42.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:35:41 smithi047 conmon[78521]: yield 2022-05-14T04:35:42.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:35:41 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:35:42.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:35:41 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:35:42.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:35:41 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:35:42.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:35:41 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T04:35:42.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:35:41 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:35:42.048 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:35:41 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T04:35:42.048 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:35:41 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:35:42.048 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:35:41 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T04:35:42.049 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:35:41 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:35:42.049 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:35:41 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T04:35:42.050 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:35:41 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:35:42.050 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:35:41 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T04:35:42.050 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:35:41 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:35:42.054 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:35:41 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:35:42.054 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:35:41 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:35:42.054 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:35:41 smithi047 conmon[78521]: 2022-05-14T04:35:41.809+0000 7f533ef76700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T04:35:42.055 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:35:41 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T04:35:42.055 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:35:41 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:35:42.056 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:35:41 smithi047 conmon[78521]: yield 2022-05-14T04:35:42.056 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:35:41 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:35:42.057 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:35:41 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:35:42.057 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:35:41 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:35:42.058 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:35:41 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T04:35:42.058 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:35:41 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:35:42.058 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:35:41 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T04:35:42.059 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:35:41 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:35:42.059 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:35:41 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T04:35:42.060 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:35:41 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:35:42.060 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:35:41 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T04:35:42.061 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:35:41 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:35:42.061 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:35:41 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T04:35:42.062 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:35:41 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:35:42.062 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:35:41 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:35:42.063 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:35:41 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:35:42.063 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:35:41 smithi047 conmon[78521]: 2022-05-14T04:35:42.064 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:35:41 smithi047 conmon[78521]: During handling of the above exception, another exception occurred: 2022-05-14T04:35:42.064 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:35:41 smithi047 conmon[78521]: 2022-05-14T04:35:42.064 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:35:41 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T04:35:42.065 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:35:41 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T04:35:42.065 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:35:41 smithi047 conmon[78521]: return f(*arg) 2022-05-14T04:35:42.066 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:35:41 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T04:35:42.066 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:35:41 smithi047 conmon[78521]: self._write_client_files(client_files, host) 2022-05-14T04:35:42.067 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:35:41 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T04:35:42.067 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:35:41 smithi047 conmon[78521]: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T04:35:42.067 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:35:41 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T04:35:42.068 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:35:41 smithi047 conmon[78521]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T04:35:42.068 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:35:41 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T04:35:42.068 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:35:41 smithi047 conmon[78521]: return self.event_loop.get_result(coro) 2022-05-14T04:35:42.068 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:35:41 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T04:35:42.068 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:35:41 smithi047 conmon[78521]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T04:35:42.069 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:35:41 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T04:35:42.069 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:35:41 smithi047 conmon[78521]: return self.__get_result() 2022-05-14T04:35:42.069 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:35:41 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T04:35:42.069 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:35:41 smithi047 conmon[78521]: raise self._exception 2022-05-14T04:35:42.069 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:35:41 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T04:35:42.070 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:35:41 smithi047 conmon[78521]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T04:35:42.070 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:35:41 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T04:35:42.070 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:35:41 smithi047 conmon[78521]: conn = await self._remote_connection(host, addr) 2022-05-14T04:35:42.070 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:35:41 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T04:35:42.071 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:35:41 smithi047 conmon[78521]: raise 2022-05-14T04:35:42.071 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:35:41 smithi047 conmon[78521]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T04:35:42.071 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:35:41 smithi047 conmon[78521]: self.gen.throw(type, value, traceback) 2022-05-14T04:35:42.071 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:35:41 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T04:35:42.071 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:35:41 smithi047 conmon[78521]: raise OrchestratorError(msg) 2022-05-14T04:35:42.072 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:35:41 smithi047 conmon[78521]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:35:42.114 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:35:41 smithi115 ceph-mon[78593]: pgmap v9588: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:35:42.117 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:35:41 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:35:42.118 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:35:42 smithi115 conmon[90393]: 2022-05-14T04:35:42.030+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:35:43.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:35:42 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:35:43.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:35:42 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:35:43.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:35:42 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:35:43.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T04:35:43.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:35:43.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T04:35:43.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:35:43.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:35:43.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:35:43.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T04:35:43.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:35:43.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T04:35:43.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:35:43.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T04:35:43.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:35:43.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T04:35:43.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:35:43.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T04:35:43.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:35:43.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:35:43.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:35:43.048 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:35:42 smithi047 ceph-mon[77147]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T04:35:43.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T04:35:43.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:35:43.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T04:35:43.049 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:35:43.049 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:35:43.049 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:35:43.049 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T04:35:43.050 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:35:43.050 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T04:35:43.050 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:35:43.050 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T04:35:43.050 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:35:43.051 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T04:35:43.051 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:35:43.051 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T04:35:43.051 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:35:43.052 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:35:43.052 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:35:43.052 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T04:35:43.052 INFO:journalctl@ceph.mon.b.smithi047.stdout: During handling of the above exception, another exception occurred: 2022-05-14T04:35:43.052 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T04:35:43.053 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T04:35:43.053 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T04:35:43.054 INFO:journalctl@ceph.mon.b.smithi047.stdout: return f(*arg) 2022-05-14T04:35:43.055 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T04:35:43.055 INFO:journalctl@ceph.mon.b.smithi047.stdout: self._write_client_files(client_files, host) 2022-05-14T04:35:43.056 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T04:35:43.056 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T04:35:43.057 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T04:35:43.057 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T04:35:43.057 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T04:35:43.058 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.event_loop.get_result(coro) 2022-05-14T04:35:43.058 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T04:35:43.059 INFO:journalctl@ceph.mon.b.smithi047.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T04:35:43.059 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T04:35:43.060 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.__get_result() 2022-05-14T04:35:43.060 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T04:35:43.060 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise self._exception 2022-05-14T04:35:43.061 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T04:35:43.061 INFO:journalctl@ceph.mon.b.smithi047.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T04:35:43.062 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T04:35:43.062 INFO:journalctl@ceph.mon.b.smithi047.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T04:35:43.063 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T04:35:43.063 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise 2022-05-14T04:35:43.063 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T04:35:43.064 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.gen.throw(type, value, traceback) 2022-05-14T04:35:43.064 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T04:35:43.065 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OrchestratorError(msg) 2022-05-14T04:35:43.065 INFO:journalctl@ceph.mon.b.smithi047.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:35:43.066 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:35:42 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:35:43.066 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:35:42 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 18475 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:35:43.080 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:35:42 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:35:43.080 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:35:42 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:35:43.081 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:35:42 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:35:43.081 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T04:35:43.082 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:35:43.082 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T04:35:43.082 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:35:43.083 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:35:43.083 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:35:43.084 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T04:35:43.084 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:35:43.085 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T04:35:43.085 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:35:43.085 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T04:35:43.086 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:35:43.086 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T04:35:43.087 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:35:43.087 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T04:35:43.087 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:35:43.088 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:35:43.088 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:35:43.088 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:35:42 smithi115 ceph-mon[78593]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T04:35:43.089 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T04:35:43.089 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:35:43.090 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T04:35:43.090 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:35:43.090 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:35:43.091 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:35:43.091 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T04:35:43.092 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:35:43.092 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T04:35:43.092 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:35:43.093 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T04:35:43.093 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:35:43.094 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T04:35:43.094 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:35:43.094 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T04:35:43.095 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:35:43.095 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:35:43.095 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:35:43.096 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T04:35:43.096 INFO:journalctl@ceph.mon.c.smithi115.stdout: During handling of the above exception, another exception occurred: 2022-05-14T04:35:43.096 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T04:35:43.097 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T04:35:43.097 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T04:35:43.098 INFO:journalctl@ceph.mon.c.smithi115.stdout: return f(*arg) 2022-05-14T04:35:43.098 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T04:35:43.098 INFO:journalctl@ceph.mon.c.smithi115.stdout: self._write_client_files(client_files, host) 2022-05-14T04:35:43.099 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T04:35:43.099 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T04:35:43.100 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T04:35:43.100 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T04:35:43.100 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T04:35:43.101 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.event_loop.get_result(coro) 2022-05-14T04:35:43.101 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T04:35:43.102 INFO:journalctl@ceph.mon.c.smithi115.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T04:35:43.102 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T04:35:43.102 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.__get_result() 2022-05-14T04:35:43.103 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T04:35:43.103 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise self._exception 2022-05-14T04:35:43.104 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T04:35:43.104 INFO:journalctl@ceph.mon.c.smithi115.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T04:35:43.104 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T04:35:43.105 INFO:journalctl@ceph.mon.c.smithi115.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T04:35:43.105 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T04:35:43.105 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise 2022-05-14T04:35:43.106 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T04:35:43.106 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.gen.throw(type, value, traceback) 2022-05-14T04:35:43.106 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T04:35:43.107 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OrchestratorError(msg) 2022-05-14T04:35:43.107 INFO:journalctl@ceph.mon.c.smithi115.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:35:43.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:35:42 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:35:43.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:35:42 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 18475 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:35:43.362 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:35:43 smithi115 conmon[90393]: 2022-05-14T04:35:43.076+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:35:44.080 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:35:43 smithi115 ceph-mon[78593]: pgmap v9589: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:35:44.081 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:35:43 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:35:44.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:35:43 smithi047 ceph-mon[77147]: pgmap v9589: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:35:44.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:35:43 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:35:44.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:35:44 smithi115 conmon[90393]: 2022-05-14T04:35:44.077+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:35:45.061 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:35:44 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:35:45.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:35:44 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:35:45.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:35:45 smithi115 conmon[90393]: 2022-05-14T04:35:45.054+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:35:46.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:35:46 smithi115 conmon[90393]: 2022-05-14T04:35:46.030+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:35:46.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:35:45 smithi115 ceph-mon[78593]: pgmap v9590: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:35:46.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:35:45 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:35:46.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:35:45 smithi047 ceph-mon[77147]: pgmap v9590: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:35:46.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:35:45 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:35:47.062 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:35:46 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:35:47.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:35:46 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:35:47.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:35:47 smithi115 conmon[90393]: 2022-05-14T04:35:47.060+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:35:48.101 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:35:47 smithi115 ceph-mon[78593]: pgmap v9591: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:35:48.102 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:35:47 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:35:48.102 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:35:47 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 18480 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:35:48.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:35:47 smithi047 ceph-mon[77147]: pgmap v9591: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:35:48.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:35:47 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:35:48.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:35:47 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 18480 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:35:48.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:35:48 smithi115 conmon[90393]: 2022-05-14T04:35:48.099+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:35:49.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:35:48 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:35:49.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:35:48 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:35:49.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:35:49 smithi115 conmon[90393]: 2022-05-14T04:35:49.109+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:35:50.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:35:49 smithi115 ceph-mon[78593]: pgmap v9592: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:35:50.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:35:49 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:35:50.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:35:49 smithi047 ceph-mon[77147]: pgmap v9592: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:35:50.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:35:49 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:35:50.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:35:50 smithi115 conmon[90393]: 2022-05-14T04:35:50.112+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:35:51.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:35:50 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:35:51.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:35:50 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:35:51.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:35:50 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:35:51.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:35:50 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:35:51.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:35:50 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:35:51.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:35:50 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:35:51.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:35:51 smithi115 conmon[90393]: 2022-05-14T04:35:51.160+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:35:52.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:35:51 smithi115 ceph-mon[78593]: pgmap v9593: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:35:52.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:35:51 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:35:52.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:35:51 smithi047 ceph-mon[77147]: pgmap v9593: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:35:52.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:35:51 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:35:52.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:35:52 smithi115 conmon[90393]: 2022-05-14T04:35:52.138+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:35:52.828 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:35:52 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:35:53.104 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:35:52 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 18486 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:35:53.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:35:52 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:35:53.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:35:52 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 18486 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:35:53.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:35:53 smithi115 conmon[90393]: 2022-05-14T04:35:53.101+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:35:54.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:35:53 smithi115 ceph-mon[78593]: pgmap v9594: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:35:54.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:35:53 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:35:54.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:35:53 smithi047 ceph-mon[77147]: pgmap v9594: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:35:54.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:35:53 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:35:54.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:35:54 smithi115 conmon[90393]: 2022-05-14T04:35:54.150+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:35:55.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:35:54 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:35:55.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:35:54 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:35:55.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:35:54 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:35:55.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:35:54 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:35:55.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:35:54 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:35:55.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:35:54 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:35:55.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:35:55 smithi115 conmon[90393]: 2022-05-14T04:35:55.136+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:35:56.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:35:55 smithi115 ceph-mon[78593]: pgmap v9595: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:35:56.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:35:55 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:35:56.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:35:55 smithi047 ceph-mon[77147]: pgmap v9595: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:35:56.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:35:55 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:35:56.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:35:56 smithi115 conmon[90393]: 2022-05-14T04:35:56.112+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:35:57.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:35:56 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:35:57.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:35:56 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:35:57.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:35:57 smithi115 conmon[90393]: 2022-05-14T04:35:57.116+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:35:58.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:35:57 smithi115 ceph-mon[78593]: pgmap v9596: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:35:58.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:35:57 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:35:58.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:35:57 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 18490 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:35:58.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:35:57 smithi047 ceph-mon[77147]: pgmap v9596: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:35:58.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:35:57 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:35:58.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:35:57 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 18490 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:35:58.365 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:35:58 smithi115 conmon[90393]: 2022-05-14T04:35:58.098+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:35:59.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:35:58 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:35:59.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:35:58 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:35:59.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:35:58 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:35:59.110 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:35:59 smithi115 conmon[90393]: 2022-05-14T04:35:59.053+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:35:59.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:35:58 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:35:59.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:35:58 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:35:59.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:35:58 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:36:00.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:35:59 smithi115 ceph-mon[78593]: pgmap v9597: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:36:00.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:35:59 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:36:00.111 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:36:00 smithi115 conmon[90393]: 2022-05-14T04:36:00.094+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:36:00.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:35:59 smithi047 ceph-mon[77147]: pgmap v9597: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:36:00.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:35:59 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:36:01.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:36:00 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:36:01.111 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:36:01 smithi115 conmon[90393]: 2022-05-14T04:36:01.063+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:36:01.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:36:00 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:36:02.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:36:01 smithi047 ceph-mon[77147]: pgmap v9598: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:36:02.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:36:01 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:36:02.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:36:01 smithi115 ceph-mon[78593]: pgmap v9598: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:36:02.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:36:01 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:36:02.360 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:36:02 smithi115 conmon[90393]: 2022-05-14T04:36:02.093+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:36:03.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:36:02 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:36:03.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:36:02 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:36:03.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:36:02 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:36:03.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:36:02 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 18495 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:36:03.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:36:03 smithi115 conmon[90393]: 2022-05-14T04:36:03.045+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:36:03.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:36:02 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:36:03.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:36:02 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:36:03.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:36:02 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:36:03.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:36:02 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 18495 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:36:04.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:36:03 smithi047 ceph-mon[77147]: pgmap v9599: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:36:04.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:36:03 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:36:04.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:36:04 smithi115 conmon[90393]: 2022-05-14T04:36:04.050+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:36:04.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:36:03 smithi115 ceph-mon[78593]: pgmap v9599: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:36:04.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:36:03 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:36:05.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:36:04 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:36:05.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:36:04 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:36:05.360 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:36:05 smithi115 conmon[90393]: 2022-05-14T04:36:05.056+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:36:06.295 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:36:05 smithi047 ceph-mon[77147]: pgmap v9600: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:36:06.295 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:36:05 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:36:06.362 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:36:06 smithi115 conmon[90393]: 2022-05-14T04:36:06.082+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:36:06.362 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:36:05 smithi115 ceph-mon[78593]: pgmap v9600: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:36:06.362 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:36:05 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:36:07.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:36:06 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:36:07.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:36:06 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:36:07.360 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:36:07 smithi115 conmon[90393]: 2022-05-14T04:36:07.091+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:36:08.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:36:07 smithi047 ceph-mon[77147]: pgmap v9601: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:36:08.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:36:07 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:36:08.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:36:07 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 18500 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:36:08.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:36:08 smithi115 conmon[90393]: 2022-05-14T04:36:08.093+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:36:08.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:36:07 smithi115 ceph-mon[78593]: pgmap v9601: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:36:08.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:36:07 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:36:08.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:36:07 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 18500 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:36:09.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:36:08 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:36:09.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:36:08 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:36:09.360 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:36:09 smithi115 conmon[90393]: 2022-05-14T04:36:09.063+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:36:10.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:36:09 smithi047 ceph-mon[77147]: pgmap v9602: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:36:10.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:36:09 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:36:10.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:36:10 smithi115 conmon[90393]: 2022-05-14T04:36:10.106+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:36:10.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:36:09 smithi115 ceph-mon[78593]: pgmap v9602: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:36:10.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:36:09 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:36:11.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:36:10 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:36:11.295 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:36:10 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:36:11.295 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:36:10 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:36:11.360 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:36:11 smithi115 conmon[90393]: 2022-05-14T04:36:11.108+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:36:11.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:36:10 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:36:11.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:36:10 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:36:11.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:36:10 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:36:12.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:36:11 smithi047 ceph-mon[77147]: pgmap v9603: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:36:12.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:36:11 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:36:12.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:36:12 smithi115 conmon[90393]: 2022-05-14T04:36:12.081+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:36:12.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:36:11 smithi115 ceph-mon[78593]: pgmap v9603: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:36:12.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:36:11 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:36:13.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:36:12 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:36:13.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:36:12 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 18505 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:36:13.362 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:36:13 smithi115 conmon[90393]: 2022-05-14T04:36:13.054+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:36:13.362 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:36:12 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:36:13.363 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:36:12 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 18505 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:36:14.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:36:14 smithi115 conmon[90393]: 2022-05-14T04:36:14.091+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:36:14.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:36:14 smithi115 ceph-mon[78593]: pgmap v9604: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:36:14.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:36:14 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:36:14.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:36:14 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:36:14.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:36:14 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:36:15.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:36:14 smithi047 ceph-mon[77147]: pgmap v9604: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:36:15.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:36:14 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:36:15.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:36:14 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:36:15.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:36:14 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:36:15.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:36:15 smithi115 conmon[90393]: 2022-05-14T04:36:15.065+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:36:15.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:36:15 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:36:15.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:36:15 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T04:36:15.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:36:15 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T04:36:16.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:36:15 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:36:16.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:36:15 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T04:36:16.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:36:15 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T04:36:16.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:36:16 smithi115 conmon[90393]: 2022-05-14T04:36:16.016+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:36:16.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:36:16 smithi115 ceph-mon[78593]: pgmap v9605: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:36:16.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:36:16 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:36:17.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:36:16 smithi047 ceph-mon[77147]: pgmap v9605: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:36:17.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:36:16 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:36:17.360 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:36:16 smithi115 conmon[90393]: 2022-05-14T04:36:16.990+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:36:17.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:36:17 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:36:17.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:36:17 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 18511 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:36:18.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:36:17 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:36:18.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:36:17 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 18511 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:36:18.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:36:18 smithi115 conmon[90393]: 2022-05-14T04:36:18.024+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:36:18.864 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:36:18 smithi115 ceph-mon[78593]: pgmap v9606: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:36:18.864 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:36:18 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:36:18.864 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:36:18 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:36:18.864 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:36:18 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:36:18.865 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:36:18 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:36:19.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:36:18 smithi047 ceph-mon[77147]: pgmap v9606: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:36:19.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:36:18 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:36:19.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:36:18 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:36:19.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:36:18 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:36:19.045 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:36:18 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:36:19.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:36:19 smithi115 conmon[90393]: 2022-05-14T04:36:19.051+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:36:19.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:36:19 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:36:20.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:36:19 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:36:20.361 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:36:20 smithi115 conmon[90393]: 2022-05-14T04:36:20.017+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:36:20.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:36:20 smithi115 ceph-mon[78593]: pgmap v9607: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:36:20.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:36:20 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:36:21.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:36:20 smithi047 ceph-mon[77147]: pgmap v9607: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:36:21.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:36:20 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:36:21.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:36:21 smithi115 conmon[90393]: 2022-05-14T04:36:21.058+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:36:21.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:36:21 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:36:22.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:36:21 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:36:22.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:36:22 smithi115 conmon[90393]: 2022-05-14T04:36:22.029+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:36:22.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:36:22 smithi115 ceph-mon[78593]: pgmap v9608: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:36:22.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:36:22 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:36:22.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:36:22 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:36:22.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:36:22 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:36:22.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:36:22 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 18515 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:36:23.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:36:22 smithi047 ceph-mon[77147]: pgmap v9608: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:36:23.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:36:22 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:36:23.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:36:22 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:36:23.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:36:22 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:36:23.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:36:22 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 18515 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:36:23.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:36:23 smithi115 conmon[90393]: 2022-05-14T04:36:23.031+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:36:23.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:36:23 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:36:24.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:36:23 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:36:24.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:36:24 smithi115 conmon[90393]: 2022-05-14T04:36:24.040+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:36:24.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:36:24 smithi115 ceph-mon[78593]: pgmap v9609: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:36:24.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:36:24 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:36:25.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:36:24 smithi047 ceph-mon[77147]: pgmap v9609: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:36:25.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:36:24 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:36:25.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:36:25 smithi115 conmon[90393]: 2022-05-14T04:36:25.080+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:36:25.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:36:25 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:36:26.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:36:25 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:36:26.600 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:36:26 smithi115 conmon[90393]: 2022-05-14T04:36:26.124+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:36:26.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:36:26 smithi115 ceph-mon[78593]: pgmap v9610: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:36:26.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:36:26 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:36:27.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:36:26 smithi047 ceph-mon[77147]: pgmap v9610: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:36:27.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:36:26 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:36:27.361 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:36:27 smithi115 conmon[90393]: 2022-05-14T04:36:27.096+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:36:27.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:36:27 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:36:27.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:36:27 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 18520 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:36:28.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:36:27 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:36:28.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:36:27 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 18520 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:36:28.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:36:28 smithi115 conmon[90393]: 2022-05-14T04:36:28.089+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:36:28.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:36:28 smithi115 ceph-mon[78593]: pgmap v9611: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:36:28.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:36:28 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:36:29.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:36:28 smithi047 ceph-mon[77147]: pgmap v9611: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:36:29.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:36:28 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:36:29.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:36:29 smithi115 conmon[90393]: 2022-05-14T04:36:29.113+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:36:30.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:36:29 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:36:30.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:36:29 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:36:30.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:36:30 smithi115 conmon[90393]: 2022-05-14T04:36:30.116+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:36:31.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:36:30 smithi047 ceph-mon[77147]: pgmap v9612: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:36:31.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:36:30 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:36:31.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:36:30 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:36:31.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:36:30 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:36:31.071 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:36:30 smithi115 ceph-mon[78593]: pgmap v9612: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:36:31.072 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:36:30 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:36:31.072 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:36:30 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:36:31.072 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:36:30 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:36:31.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:36:31 smithi115 conmon[90393]: 2022-05-14T04:36:31.068+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:36:32.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:36:31 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:36:32.087 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:36:31 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:36:32.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:36:32 smithi115 conmon[90393]: 2022-05-14T04:36:32.085+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:36:33.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:36:32 smithi047 ceph-mon[77147]: pgmap v9613: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:36:33.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:36:32 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:36:33.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:36:32 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 18526 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:36:33.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:36:32 smithi115 ceph-mon[78593]: pgmap v9613: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:36:33.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:36:32 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:36:33.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:36:32 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 18526 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:36:33.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:36:33 smithi115 conmon[90393]: 2022-05-14T04:36:33.134+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:36:34.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:36:33 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:36:34.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:36:33 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:36:34.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:36:34 smithi115 conmon[90393]: 2022-05-14T04:36:34.106+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:36:35.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:36:34 smithi047 ceph-mon[77147]: pgmap v9614: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:36:35.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:36:34 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:36:35.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:36:34 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:36:35.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:36:34 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:36:35.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:36:34 smithi115 ceph-mon[78593]: pgmap v9614: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:36:35.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:36:34 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:36:35.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:36:34 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:36:35.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:36:34 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:36:35.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:36:35 smithi115 conmon[90393]: 2022-05-14T04:36:35.148+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:36:36.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:36:35 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:36:36.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:36:35 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:36:36.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:36:36 smithi115 conmon[90393]: 2022-05-14T04:36:36.167+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:36:37.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:36:36 smithi047 ceph-mon[77147]: pgmap v9615: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:36:37.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:36:36 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:36:37.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:36:36 smithi115 ceph-mon[78593]: pgmap v9615: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:36:37.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:36:36 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:36:37.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:36:37 smithi115 conmon[90393]: 2022-05-14T04:36:37.150+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:36:38.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:36:37 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:36:38.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:36:37 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 18531 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:36:38.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:36:37 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:36:38.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:36:37 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 18531 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:36:38.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:36:38 smithi115 conmon[90393]: 2022-05-14T04:36:38.138+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:36:39.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:36:38 smithi047 ceph-mon[77147]: pgmap v9616: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:36:39.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:36:38 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:36:39.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:36:38 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:36:39.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:36:38 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:36:39.105 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:36:38 smithi115 ceph-mon[78593]: pgmap v9616: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:36:39.105 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:36:38 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:36:39.105 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:36:38 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:36:39.106 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:36:38 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:36:39.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:36:39 smithi115 conmon[90393]: 2022-05-14T04:36:39.099+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:36:40.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:36:39 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:36:40.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:36:39 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:36:40.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:36:40 smithi115 conmon[90393]: 2022-05-14T04:36:40.113+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:36:41.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:36:40 smithi047 ceph-mon[77147]: pgmap v9617: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:36:41.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:36:40 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:36:41.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:36:40 smithi115 ceph-mon[78593]: pgmap v9617: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:36:41.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:36:40 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:36:41.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:36:41 smithi115 conmon[90393]: 2022-05-14T04:36:41.134+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:36:41.910 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:36:41 smithi047 ceph-mon[77147]: pgmap v9618: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:36:41.911 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:36:41 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:36:41.912 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:36:41 smithi115 ceph-mon[78593]: pgmap v9618: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:36:41.912 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:36:41 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:36:42.319 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:36:42 smithi115 conmon[90393]: 2022-05-14T04:36:42.097+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:36:43.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:36:42 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:36:43.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:36:42 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:36:43.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:36:42 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T04:36:43.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:36:42 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T04:36:43.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:36:42 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T04:36:43.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:36:42 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:36:43.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:36:42 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 18536 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:36:43.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:36:42 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:36:43.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:36:42 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:36:43.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:36:42 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T04:36:43.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:36:42 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T04:36:43.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:36:42 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T04:36:43.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:36:42 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:36:43.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:36:42 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 18536 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:36:43.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:36:43 smithi115 conmon[90393]: 2022-05-14T04:36:43.140+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:36:44.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:36:43 smithi047 ceph-mon[77147]: pgmap v9619: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:36:44.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:36:43 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:36:44.102 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:36:43 smithi115 ceph-mon[78593]: pgmap v9619: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:36:44.102 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:36:43 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:36:44.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:36:44 smithi115 conmon[90393]: 2022-05-14T04:36:44.099+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:36:44.915 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:36:44 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:36:45.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:36:44 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:36:45.293 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:36:44 smithi047 conmon[78521]: 2022-05-14T04:36:44.911+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:36:45.293 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:36:44 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T04:36:45.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:36:44 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:36:45.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:36:44 smithi047 conmon[78521]: yield 2022-05-14T04:36:45.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:36:44 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:36:45.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:36:44 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:36:45.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:36:44 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:36:45.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:36:44 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T04:36:45.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:36:44 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:36:45.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:36:44 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T04:36:45.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:36:44 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:36:45.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:36:44 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T04:36:45.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:36:44 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:36:45.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:36:44 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T04:36:45.297 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:36:44 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:36:45.297 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:36:44 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T04:36:45.297 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:36:44 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:36:45.297 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:36:44 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:36:45.298 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:36:44 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:36:45.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:36:45 smithi115 conmon[90393]: 2022-05-14T04:36:45.146+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:36:46.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:36:45 smithi047 ceph-mon[77147]: pgmap v9620: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:36:46.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:36:45 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:36:46.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T04:36:46.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:36:46.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T04:36:46.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:36:46.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:36:46.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:36:46.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T04:36:46.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:36:46.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T04:36:46.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:36:46.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T04:36:46.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:36:46.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T04:36:46.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:36:46.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T04:36:46.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:36:46.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:36:46.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:36:46.048 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:36:45 smithi047 ceph-mon[77147]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T04:36:46.048 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:36:45 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:36:46.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:36:45 smithi115 ceph-mon[78593]: pgmap v9620: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:36:46.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:36:45 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:36:46.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T04:36:46.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:36:46.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T04:36:46.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:36:46.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:36:46.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:36:46.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T04:36:46.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:36:46.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T04:36:46.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:36:46.113 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T04:36:46.113 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:36:46.113 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T04:36:46.113 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:36:46.113 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T04:36:46.114 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:36:46.114 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:36:46.114 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:36:46.115 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:36:45 smithi115 ceph-mon[78593]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T04:36:46.115 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:36:45 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:36:46.613 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:36:46 smithi115 conmon[90393]: 2022-05-14T04:36:46.181+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:36:47.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:36:46 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:36:47.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:36:46 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:36:47.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:36:47 smithi115 conmon[90393]: 2022-05-14T04:36:47.166+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:36:47.988 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:36:47 smithi047 ceph-mon[77147]: pgmap v9621: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:36:47.988 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:36:47 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:36:47.989 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:36:47 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 18541 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:36:48.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:36:47 smithi115 ceph-mon[78593]: pgmap v9621: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:36:48.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:36:47 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:36:48.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:36:47 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 18541 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:36:48.293 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:36:47 smithi047 conmon[78521]: 2022-05-14T04:36:47.984+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:36:48.293 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:36:47 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T04:36:48.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:36:47 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:36:48.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:36:47 smithi047 conmon[78521]: yield 2022-05-14T04:36:48.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:36:47 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:36:48.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:36:47 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:36:48.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:36:47 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:36:48.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:36:47 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T04:36:48.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:36:47 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:36:48.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:36:47 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T04:36:48.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:36:47 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:36:48.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:36:47 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T04:36:48.297 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:36:47 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:36:48.297 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:36:47 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T04:36:48.297 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:36:47 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:36:48.297 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:36:47 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T04:36:48.298 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:36:47 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:36:48.299 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:36:47 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:36:48.299 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:36:47 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:36:48.299 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:36:47 smithi047 conmon[78521]: 2022-05-14T04:36:47.984+0000 7f5344781700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T04:36:48.300 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:36:47 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T04:36:48.300 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:36:47 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:36:48.301 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:36:47 smithi047 conmon[78521]: yield 2022-05-14T04:36:48.301 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:36:47 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:36:48.302 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:36:47 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:36:48.302 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:36:47 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:36:48.302 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:36:47 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T04:36:48.303 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:36:47 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:36:48.303 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:36:47 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T04:36:48.303 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:36:47 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:36:48.304 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:36:47 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T04:36:48.304 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:36:47 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:36:48.304 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:36:47 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T04:36:48.305 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:36:47 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:36:48.305 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:36:47 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T04:36:48.305 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:36:47 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:36:48.305 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:36:47 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:36:48.306 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:36:47 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:36:48.306 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:36:47 smithi047 conmon[78521]: 2022-05-14T04:36:48.306 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:36:47 smithi047 conmon[78521]: During handling of the above exception, another exception occurred: 2022-05-14T04:36:48.307 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:36:47 smithi047 conmon[78521]: 2022-05-14T04:36:48.307 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:36:47 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T04:36:48.307 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:36:47 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T04:36:48.308 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:36:47 smithi047 conmon[78521]: return f(*arg) 2022-05-14T04:36:48.308 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:36:47 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T04:36:48.308 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:36:47 smithi047 conmon[78521]: self._write_client_files(client_files, host) 2022-05-14T04:36:48.310 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:36:47 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T04:36:48.310 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:36:47 smithi047 conmon[78521]: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T04:36:48.310 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:36:47 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T04:36:48.311 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:36:47 smithi047 conmon[78521]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T04:36:48.311 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:36:47 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T04:36:48.311 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:36:47 smithi047 conmon[78521]: return self.event_loop.get_result(coro) 2022-05-14T04:36:48.311 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:36:47 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T04:36:48.312 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:36:47 smithi047 conmon[78521]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T04:36:48.312 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:36:47 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T04:36:48.312 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:36:47 smithi047 conmon[78521]: return self.__get_result() 2022-05-14T04:36:48.312 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:36:47 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T04:36:48.313 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:36:47 smithi047 conmon[78521]: raise self._exception 2022-05-14T04:36:48.313 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:36:47 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T04:36:48.313 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:36:47 smithi047 conmon[78521]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T04:36:48.313 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:36:47 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T04:36:48.314 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:36:47 smithi047 conmon[78521]: conn = await self._remote_connection(host, addr) 2022-05-14T04:36:48.314 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:36:47 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T04:36:48.314 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:36:47 smithi047 conmon[78521]: raise 2022-05-14T04:36:48.315 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:36:47 smithi047 conmon[78521]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T04:36:48.315 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:36:47 smithi047 conmon[78521]: self.gen.throw(type, value, traceback) 2022-05-14T04:36:48.315 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:36:47 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T04:36:48.315 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:36:47 smithi047 conmon[78521]: raise OrchestratorError(msg) 2022-05-14T04:36:48.316 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:36:47 smithi047 conmon[78521]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:36:48.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:36:48 smithi115 conmon[90393]: 2022-05-14T04:36:48.161+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:36:49.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:36:48 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:36:49.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T04:36:49.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:36:49.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T04:36:49.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:36:49.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:36:49.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:36:49.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T04:36:49.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:36:49.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T04:36:49.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:36:49.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T04:36:49.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:36:49.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T04:36:49.050 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:36:49.050 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T04:36:49.050 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:36:49.050 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:36:49.051 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:36:49.051 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:36:48 smithi047 ceph-mon[77147]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T04:36:49.051 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T04:36:49.051 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:36:49.052 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T04:36:49.052 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:36:49.052 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:36:49.052 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:36:49.053 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T04:36:49.053 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:36:49.053 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T04:36:49.053 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:36:49.054 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T04:36:49.054 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:36:49.054 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T04:36:49.054 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:36:49.055 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T04:36:49.055 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:36:49.055 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:36:49.055 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:36:49.056 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T04:36:49.056 INFO:journalctl@ceph.mon.b.smithi047.stdout: During handling of the above exception, another exception occurred: 2022-05-14T04:36:49.056 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T04:36:49.056 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T04:36:49.057 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T04:36:49.057 INFO:journalctl@ceph.mon.b.smithi047.stdout: return f(*arg) 2022-05-14T04:36:49.057 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T04:36:49.057 INFO:journalctl@ceph.mon.b.smithi047.stdout: self._write_client_files(client_files, host) 2022-05-14T04:36:49.058 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T04:36:49.058 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T04:36:49.058 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T04:36:49.059 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T04:36:49.059 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T04:36:49.059 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.event_loop.get_result(coro) 2022-05-14T04:36:49.059 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T04:36:49.059 INFO:journalctl@ceph.mon.b.smithi047.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T04:36:49.060 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T04:36:49.060 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.__get_result() 2022-05-14T04:36:49.060 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T04:36:49.060 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise self._exception 2022-05-14T04:36:49.060 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T04:36:49.061 INFO:journalctl@ceph.mon.b.smithi047.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T04:36:49.061 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T04:36:49.061 INFO:journalctl@ceph.mon.b.smithi047.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T04:36:49.061 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T04:36:49.062 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise 2022-05-14T04:36:49.062 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T04:36:49.062 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.gen.throw(type, value, traceback) 2022-05-14T04:36:49.062 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T04:36:49.063 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OrchestratorError(msg) 2022-05-14T04:36:49.063 INFO:journalctl@ceph.mon.b.smithi047.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:36:49.063 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:36:48 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:36:49.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:36:48 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:36:49.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T04:36:49.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:36:49.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T04:36:49.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:36:49.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:36:49.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:36:49.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T04:36:49.113 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:36:49.113 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T04:36:49.113 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:36:49.114 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T04:36:49.114 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:36:49.114 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T04:36:49.114 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:36:49.115 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T04:36:49.115 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:36:49.115 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:36:49.115 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:36:49.115 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:36:48 smithi115 ceph-mon[78593]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T04:36:49.116 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T04:36:49.116 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:36:49.116 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T04:36:49.116 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:36:49.116 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:36:49.117 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:36:49.117 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T04:36:49.117 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:36:49.117 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T04:36:49.118 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:36:49.118 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T04:36:49.118 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:36:49.118 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T04:36:49.119 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:36:49.119 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T04:36:49.119 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:36:49.119 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:36:49.120 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:36:49.120 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T04:36:49.120 INFO:journalctl@ceph.mon.c.smithi115.stdout: During handling of the above exception, another exception occurred: 2022-05-14T04:36:49.121 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T04:36:49.121 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T04:36:49.121 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T04:36:49.121 INFO:journalctl@ceph.mon.c.smithi115.stdout: return f(*arg) 2022-05-14T04:36:49.122 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T04:36:49.122 INFO:journalctl@ceph.mon.c.smithi115.stdout: self._write_client_files(client_files, host) 2022-05-14T04:36:49.122 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T04:36:49.122 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T04:36:49.122 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T04:36:49.123 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T04:36:49.123 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T04:36:49.123 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.event_loop.get_result(coro) 2022-05-14T04:36:49.123 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T04:36:49.124 INFO:journalctl@ceph.mon.c.smithi115.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T04:36:49.124 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T04:36:49.124 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.__get_result() 2022-05-14T04:36:49.124 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T04:36:49.125 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise self._exception 2022-05-14T04:36:49.125 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T04:36:49.125 INFO:journalctl@ceph.mon.c.smithi115.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T04:36:49.125 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T04:36:49.126 INFO:journalctl@ceph.mon.c.smithi115.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T04:36:49.126 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T04:36:49.126 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise 2022-05-14T04:36:49.126 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T04:36:49.126 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.gen.throw(type, value, traceback) 2022-05-14T04:36:49.127 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T04:36:49.127 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OrchestratorError(msg) 2022-05-14T04:36:49.127 INFO:journalctl@ceph.mon.c.smithi115.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:36:49.127 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:36:48 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:36:49.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:36:49 smithi115 conmon[90393]: 2022-05-14T04:36:49.119+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:36:50.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:36:49 smithi047 ceph-mon[77147]: pgmap v9622: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:36:50.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:36:49 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:36:50.079 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:36:49 smithi115 ceph-mon[78593]: pgmap v9622: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:36:50.080 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:36:49 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:36:50.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:36:50 smithi115 conmon[90393]: 2022-05-14T04:36:50.076+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:36:51.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:36:50 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:36:51.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:36:50 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:36:51.045 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:36:50 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:36:51.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:36:50 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:36:51.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:36:50 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:36:51.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:36:50 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:36:51.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:36:51 smithi115 conmon[90393]: 2022-05-14T04:36:51.113+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:36:52.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:36:51 smithi047 ceph-mon[77147]: pgmap v9623: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:36:52.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:36:51 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:36:52.106 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:36:51 smithi115 ceph-mon[78593]: pgmap v9623: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:36:52.106 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:36:51 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:36:52.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:36:52 smithi115 conmon[90393]: 2022-05-14T04:36:52.104+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:36:53.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:36:52 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:36:53.045 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:36:52 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 18546 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:36:53.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:36:52 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:36:53.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:36:52 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 18546 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:36:53.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:36:53 smithi115 conmon[90393]: 2022-05-14T04:36:53.149+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:36:54.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:36:53 smithi047 ceph-mon[77147]: pgmap v9624: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:36:54.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:36:53 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:36:54.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:36:53 smithi115 ceph-mon[78593]: pgmap v9624: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:36:54.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:36:53 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:36:54.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:36:54 smithi115 conmon[90393]: 2022-05-14T04:36:54.186+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:36:55.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:36:54 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:36:55.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:36:54 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:36:55.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:36:54 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:36:55.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:36:54 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:36:55.138 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:36:54 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:36:55.138 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:36:54 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:36:55.610 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:36:55 smithi115 conmon[90393]: 2022-05-14T04:36:55.167+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:36:56.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:36:55 smithi047 ceph-mon[77147]: pgmap v9625: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:36:56.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:36:55 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:36:56.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:36:55 smithi115 ceph-mon[78593]: pgmap v9625: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:36:56.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:36:55 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:36:56.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:36:56 smithi115 conmon[90393]: 2022-05-14T04:36:56.180+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:36:57.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:36:56 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:36:57.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:36:56 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:36:57.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:36:57 smithi115 conmon[90393]: 2022-05-14T04:36:57.182+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:36:58.050 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:36:57 smithi047 ceph-mon[77147]: pgmap v9626: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:36:58.051 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:36:57 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:36:58.051 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:36:57 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 18550 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:36:58.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:36:57 smithi115 ceph-mon[78593]: pgmap v9626: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:36:58.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:36:57 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:36:58.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:36:57 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 18550 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:36:58.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:36:58 smithi115 conmon[90393]: 2022-05-14T04:36:58.148+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:36:59.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:36:58 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:36:59.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:36:58 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:36:59.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:36:58 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:36:59.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:36:58 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:36:59.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:36:58 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:36:59.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:36:58 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:36:59.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:36:59 smithi115 conmon[90393]: 2022-05-14T04:36:59.122+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:37:00.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:36:59 smithi047 ceph-mon[77147]: pgmap v9627: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:37:00.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:36:59 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:37:00.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:36:59 smithi115 ceph-mon[78593]: pgmap v9627: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:37:00.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:36:59 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:37:00.610 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:37:00 smithi115 conmon[90393]: 2022-05-14T04:37:00.146+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:37:01.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:37:00 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:37:01.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:37:00 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:37:01.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:37:01 smithi115 conmon[90393]: 2022-05-14T04:37:01.195+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:37:02.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:37:01 smithi047 ceph-mon[77147]: pgmap v9628: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:37:02.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:37:01 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:37:02.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:37:01 smithi115 ceph-mon[78593]: pgmap v9628: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:37:02.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:37:01 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:37:02.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:37:02 smithi115 conmon[90393]: 2022-05-14T04:37:02.244+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:37:03.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:37:02 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:37:03.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:37:02 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:37:03.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:37:02 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:37:03.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:37:02 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 18556 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:37:03.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:37:02 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:37:03.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:37:02 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:37:03.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:37:02 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:37:03.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:37:02 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 18556 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:37:03.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:37:03 smithi115 conmon[90393]: 2022-05-14T04:37:03.211+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:37:04.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:37:03 smithi047 ceph-mon[77147]: pgmap v9629: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:37:04.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:37:03 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:37:04.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:37:03 smithi115 ceph-mon[78593]: pgmap v9629: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:37:04.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:37:03 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:37:04.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:37:04 smithi115 conmon[90393]: 2022-05-14T04:37:04.222+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:37:05.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:37:04 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:37:05.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:37:04 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:37:05.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:37:05 smithi115 conmon[90393]: 2022-05-14T04:37:05.214+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:37:06.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:37:05 smithi047 ceph-mon[77147]: pgmap v9630: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:37:06.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:37:05 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:37:06.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:37:05 smithi115 ceph-mon[78593]: pgmap v9630: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:37:06.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:37:05 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:37:06.610 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:37:06 smithi115 conmon[90393]: 2022-05-14T04:37:06.259+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:37:07.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:37:06 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:37:07.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:37:06 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:37:07.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:37:07 smithi115 conmon[90393]: 2022-05-14T04:37:07.242+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:37:08.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:37:07 smithi047 ceph-mon[77147]: pgmap v9631: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:37:08.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:37:07 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:37:08.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:37:07 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 18560 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:37:08.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:37:07 smithi115 ceph-mon[78593]: pgmap v9631: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:37:08.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:37:07 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:37:08.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:37:07 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 18560 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:37:08.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:37:08 smithi115 conmon[90393]: 2022-05-14T04:37:08.223+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:37:09.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:37:08 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:37:09.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:37:08 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:37:09.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:37:09 smithi115 conmon[90393]: 2022-05-14T04:37:09.255+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:37:10.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:37:09 smithi047 ceph-mon[77147]: pgmap v9632: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:37:10.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:37:09 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:37:10.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:37:09 smithi115 ceph-mon[78593]: pgmap v9632: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:37:10.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:37:09 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:37:10.610 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:37:10 smithi115 conmon[90393]: 2022-05-14T04:37:10.289+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:37:11.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:37:10 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:37:11.045 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:37:10 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:37:11.045 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:37:10 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:37:11.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:37:10 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:37:11.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:37:10 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:37:11.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:37:10 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:37:11.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:37:11 smithi115 conmon[90393]: 2022-05-14T04:37:11.275+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:37:12.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:37:11 smithi047 ceph-mon[77147]: pgmap v9633: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:37:12.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:37:11 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:37:12.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:37:11 smithi115 ceph-mon[78593]: pgmap v9633: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:37:12.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:37:11 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:37:12.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:37:12 smithi115 conmon[90393]: 2022-05-14T04:37:12.325+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:37:13.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:37:12 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:37:13.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:37:12 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 18565 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:37:13.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:37:12 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:37:13.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:37:12 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 18565 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:37:13.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:37:13 smithi115 conmon[90393]: 2022-05-14T04:37:13.349+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:37:14.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:37:13 smithi047 ceph-mon[77147]: pgmap v9634: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:37:14.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:37:13 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:37:14.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:37:13 smithi115 ceph-mon[78593]: pgmap v9634: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:37:14.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:37:13 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:37:14.759 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:37:14 smithi115 conmon[90393]: 2022-05-14T04:37:14.380+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:37:15.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:37:14 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:37:15.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:37:14 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:37:15.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:37:14 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:37:15.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:37:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T04:37:15.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:37:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T04:37:15.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:37:14 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:37:15.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:37:14 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:37:15.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:37:14 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:37:15.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:37:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T04:37:15.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:37:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T04:37:15.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:37:15 smithi115 conmon[90393]: 2022-05-14T04:37:15.339+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:37:16.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:37:15 smithi047 ceph-mon[77147]: pgmap v9635: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:37:16.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:37:15 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:37:16.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:37:15 smithi115 ceph-mon[78593]: pgmap v9635: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:37:16.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:37:15 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:37:16.766 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:37:16 smithi115 conmon[90393]: 2022-05-14T04:37:16.361+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:37:16.767 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:37:16 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:37:17.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:37:16 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:37:17.773 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:37:17 smithi115 conmon[90393]: 2022-05-14T04:37:17.378+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:37:18.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:37:17 smithi047 ceph-mon[77147]: pgmap v9636: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:37:18.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:37:17 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:37:18.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:37:17 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 18570 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:37:18.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:37:17 smithi115 ceph-mon[78593]: pgmap v9636: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:37:18.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:37:17 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:37:18.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:37:17 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 18570 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:37:18.772 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:37:18 smithi115 conmon[90393]: 2022-05-14T04:37:18.405+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:37:19.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:37:18 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:37:19.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:37:18 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:37:19.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:37:18 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:37:19.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:37:18 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:37:19.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:37:18 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:37:19.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:37:18 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:37:19.777 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:37:19 smithi115 conmon[90393]: 2022-05-14T04:37:19.369+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:37:20.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:37:19 smithi047 ceph-mon[77147]: pgmap v9637: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:37:20.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:37:19 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:37:20.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:37:19 smithi115 ceph-mon[78593]: pgmap v9637: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:37:20.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:37:19 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:37:20.779 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:37:20 smithi115 conmon[90393]: 2022-05-14T04:37:20.415+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:37:21.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:37:20 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:37:21.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:37:20 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:37:21.784 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:37:21 smithi115 conmon[90393]: 2022-05-14T04:37:21.367+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:37:22.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:37:21 smithi047 ceph-mon[77147]: pgmap v9638: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:37:22.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:37:21 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:37:22.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:37:21 smithi115 ceph-mon[78593]: pgmap v9638: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:37:22.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:37:21 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:37:22.786 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:37:22 smithi115 conmon[90393]: 2022-05-14T04:37:22.359+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:37:23.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:37:22 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:37:23.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:37:22 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:37:23.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:37:22 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:37:23.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:37:22 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 18575 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:37:23.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:37:22 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:37:23.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:37:22 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:37:23.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:37:22 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:37:23.112 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:37:22 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 18575 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:37:23.791 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:37:23 smithi115 conmon[90393]: 2022-05-14T04:37:23.397+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:37:24.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:37:23 smithi047 ceph-mon[77147]: pgmap v9639: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:37:24.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:37:23 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:37:24.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:37:23 smithi115 ceph-mon[78593]: pgmap v9639: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:37:24.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:37:23 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:37:24.793 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:37:24 smithi115 conmon[90393]: 2022-05-14T04:37:24.367+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:37:25.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:37:24 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:37:25.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:37:24 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:37:25.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:37:25 smithi115 conmon[90393]: 2022-05-14T04:37:25.341+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:37:26.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:37:25 smithi115 ceph-mon[78593]: pgmap v9640: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:37:26.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:37:25 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:37:26.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:37:25 smithi047 ceph-mon[77147]: pgmap v9640: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:37:26.453 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:37:25 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:37:26.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:37:26 smithi115 conmon[90393]: 2022-05-14T04:37:26.321+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:37:27.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:37:26 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:37:27.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:37:26 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:37:27.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:37:27 smithi115 conmon[90393]: 2022-05-14T04:37:27.284+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:37:28.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:37:27 smithi115 ceph-mon[78593]: pgmap v9641: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:37:28.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:37:27 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:37:28.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:37:27 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 18580 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:37:28.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:37:27 smithi047 ceph-mon[77147]: pgmap v9641: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:37:28.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:37:27 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:37:28.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:37:27 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 18580 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:37:28.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:37:28 smithi115 conmon[90393]: 2022-05-14T04:37:28.329+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:37:29.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:37:28 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:37:29.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:37:28 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:37:29.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:37:29 smithi115 conmon[90393]: 2022-05-14T04:37:29.305+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:37:30.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:37:29 smithi115 ceph-mon[78593]: pgmap v9642: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:37:30.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:37:29 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:37:30.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:37:29 smithi047 ceph-mon[77147]: pgmap v9642: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:37:30.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:37:29 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:37:30.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:37:30 smithi115 conmon[90393]: 2022-05-14T04:37:30.318+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:37:31.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:37:30 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:37:31.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:37:30 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:37:31.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:37:30 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:37:31.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:37:30 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:37:31.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:37:30 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:37:31.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:37:30 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:37:31.816 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:37:31 smithi115 conmon[90393]: 2022-05-14T04:37:31.367+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:37:32.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:37:31 smithi115 ceph-mon[78593]: pgmap v9643: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:37:32.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:37:31 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:37:32.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:37:31 smithi047 ceph-mon[77147]: pgmap v9643: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:37:32.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:37:31 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:37:32.818 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:37:32 smithi115 conmon[90393]: 2022-05-14T04:37:32.370+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:37:33.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:37:32 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:37:33.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:37:32 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 18585 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:37:33.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:37:32 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:37:33.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:37:32 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 18585 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:37:33.823 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:37:33 smithi115 conmon[90393]: 2022-05-14T04:37:33.382+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:37:34.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:37:33 smithi115 ceph-mon[78593]: pgmap v9644: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:37:34.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:37:33 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:37:34.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:37:33 smithi047 ceph-mon[77147]: pgmap v9644: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:37:34.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:37:33 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:37:34.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:37:34 smithi115 conmon[90393]: 2022-05-14T04:37:34.356+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:37:35.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:37:34 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:37:35.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:37:34 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:37:35.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:37:34 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:37:35.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:37:34 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:37:35.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:37:34 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:37:35.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:37:34 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:37:35.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:37:35 smithi115 conmon[90393]: 2022-05-14T04:37:35.333+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:37:36.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:37:35 smithi115 ceph-mon[78593]: pgmap v9645: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:37:36.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:37:35 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:37:36.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:37:35 smithi047 ceph-mon[77147]: pgmap v9645: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:37:36.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:37:35 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:37:36.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:37:36 smithi115 conmon[90393]: 2022-05-14T04:37:36.298+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:37:37.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:37:36 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:37:37.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:37:36 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:37:37.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:37:37 smithi115 conmon[90393]: 2022-05-14T04:37:37.273+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:37:38.121 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:37:37 smithi115 ceph-mon[78593]: pgmap v9646: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:37:38.121 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:37:37 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:37:38.122 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:37:37 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 18590 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:37:38.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:37:37 smithi047 ceph-mon[77147]: pgmap v9646: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:37:38.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:37:37 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:37:38.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:37:37 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 18590 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:37:38.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:37:38 smithi115 conmon[90393]: 2022-05-14T04:37:38.244+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:37:39.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:37:38 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:37:39.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:37:38 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:37:39.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:37:38 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:37:39.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:37:38 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:37:39.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:37:38 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:37:39.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:37:38 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:37:39.610 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:37:39 smithi115 conmon[90393]: 2022-05-14T04:37:39.201+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:37:40.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:37:39 smithi115 ceph-mon[78593]: pgmap v9647: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:37:40.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:37:39 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:37:40.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:37:39 smithi047 ceph-mon[77147]: pgmap v9647: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:37:40.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:37:39 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:37:40.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:37:40 smithi115 conmon[90393]: 2022-05-14T04:37:40.173+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:37:41.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:37:40 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:37:41.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:37:40 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:37:41.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:37:41 smithi115 conmon[90393]: 2022-05-14T04:37:41.198+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:37:42.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:37:41 smithi115 ceph-mon[78593]: pgmap v9648: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:37:42.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:37:41 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:37:42.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:37:41 smithi047 ceph-mon[77147]: pgmap v9648: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:37:42.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:37:41 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:37:42.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:37:42 smithi115 conmon[90393]: 2022-05-14T04:37:42.191+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:37:43.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:37:42 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:37:43.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:37:42 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:37:43.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:37:42 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:37:43.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:37:42 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 18595 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:37:43.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:37:42 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:37:43.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:37:42 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:37:43.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:37:42 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:37:43.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:37:42 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 18595 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:37:43.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:37:43 smithi115 conmon[90393]: 2022-05-14T04:37:43.200+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:37:44.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:37:43 smithi115 ceph-mon[78593]: pgmap v9649: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:37:44.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:37:43 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:37:44.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:37:43 smithi047 ceph-mon[77147]: pgmap v9649: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:37:44.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:37:43 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:37:44.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:37:44 smithi115 conmon[90393]: 2022-05-14T04:37:44.150+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:37:45.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:37:44 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:37:45.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:37:44 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:37:45.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:37:45 smithi115 conmon[90393]: 2022-05-14T04:37:45.107+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:37:46.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:37:45 smithi047 ceph-mon[77147]: pgmap v9650: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:37:46.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:37:45 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:37:46.360 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:37:46 smithi115 conmon[90393]: 2022-05-14T04:37:46.081+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:37:46.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:37:45 smithi115 ceph-mon[78593]: pgmap v9650: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:37:46.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:37:45 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:37:47.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:37:46 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:37:47.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:37:47 smithi115 conmon[90393]: 2022-05-14T04:37:47.046+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:37:47.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:37:46 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:37:48.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:37:47 smithi047 ceph-mon[77147]: pgmap v9651: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:37:48.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:37:47 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:37:48.295 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:37:47 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 18601 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:37:48.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:37:48 smithi115 conmon[90393]: 2022-05-14T04:37:48.013+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:37:48.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:37:47 smithi115 ceph-mon[78593]: pgmap v9651: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:37:48.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:37:47 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:37:48.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:37:47 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 18601 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:37:49.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:37:48 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T04:37:49.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:37:48 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T04:37:49.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:37:48 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T04:37:49.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:37:48 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:37:49.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:37:48 smithi115 conmon[90393]: 2022-05-14T04:37:48.994+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:37:49.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:37:48 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T04:37:49.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:37:48 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T04:37:49.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:37:48 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T04:37:49.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:37:48 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:37:50.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:37:49 smithi047 ceph-mon[77147]: pgmap v9652: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:37:50.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:37:49 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:37:50.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:37:49 smithi115 ceph-mon[78593]: pgmap v9652: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:37:50.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:37:49 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:37:50.360 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:37:50 smithi115 conmon[90393]: 2022-05-14T04:37:50.021+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:37:51.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:37:50 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:37:51.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:37:50 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:37:51.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:37:50 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:37:51.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:37:51 smithi047 conmon[78521]: 2022-05-14T04:37:51.088+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:37:51.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:37:51 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T04:37:51.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:37:51 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:37:51.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:37:51 smithi047 conmon[78521]: yield 2022-05-14T04:37:51.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:37:51 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:37:51.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:37:51 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:37:51.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:37:51 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:37:51.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:37:51 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T04:37:51.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:37:51 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:37:51.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:37:51 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T04:37:51.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:37:51 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:37:51.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:37:51 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T04:37:51.297 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:37:51 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:37:51.297 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:37:51 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T04:37:51.297 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:37:51 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:37:51.297 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:37:51 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T04:37:51.298 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:37:51 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:37:51.298 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:37:51 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:37:51.298 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:37:51 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:37:51.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:37:51 smithi115 conmon[90393]: 2022-05-14T04:37:51.068+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:37:51.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:37:50 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:37:51.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:37:50 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:37:51.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:37:50 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:37:52.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:37:51 smithi047 ceph-mon[77147]: pgmap v9653: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:37:52.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:37:51 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:37:52.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:37:51 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:37:52.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T04:37:52.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:37:52.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T04:37:52.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:37:52.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:37:52.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:37:52.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T04:37:52.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:37:52.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T04:37:52.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:37:52.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T04:37:52.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:37:52.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T04:37:52.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:37:52.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T04:37:52.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:37:52.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:37:52.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:37:52.297 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:37:51 smithi047 ceph-mon[77147]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T04:37:52.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:37:52 smithi115 conmon[90393]: 2022-05-14T04:37:52.078+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:37:52.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:37:51 smithi115 ceph-mon[78593]: pgmap v9653: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:37:52.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:37:51 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:37:52.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:37:51 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:37:52.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T04:37:52.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:37:52.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T04:37:52.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:37:52.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:37:52.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:37:52.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T04:37:52.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:37:52.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T04:37:52.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:37:52.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T04:37:52.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:37:52.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T04:37:52.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:37:52.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T04:37:52.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:37:52.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:37:52.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:37:52.364 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:37:51 smithi115 ceph-mon[78593]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T04:37:53.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:37:52 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:37:53.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:37:52 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 18605 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:37:53.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:37:53 smithi115 conmon[90393]: 2022-05-14T04:37:53.117+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:37:53.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:37:52 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:37:53.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:37:52 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 18605 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:37:54.146 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:37:53 smithi115 ceph-mon[78593]: pgmap v9654: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:37:54.146 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:37:53 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:37:54.163 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:37:53 smithi047 ceph-mon[77147]: pgmap v9654: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:37:54.163 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:37:53 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:37:54.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:37:54 smithi047 conmon[78521]: 2022-05-14T04:37:54.160+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:37:54.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:37:54 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T04:37:54.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:37:54 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:37:54.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:37:54 smithi047 conmon[78521]: yield 2022-05-14T04:37:54.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:37:54 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:37:54.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:37:54 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:37:54.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:37:54 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:37:54.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:37:54 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T04:37:54.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:37:54 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:37:54.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:37:54 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T04:37:54.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:37:54 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:37:54.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:37:54 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T04:37:54.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:37:54 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:37:54.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:37:54 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T04:37:54.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:37:54 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:37:54.547 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:37:54 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T04:37:54.547 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:37:54 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:37:54.547 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:37:54 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:37:54.547 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:37:54 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:37:54.547 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:37:54 smithi047 conmon[78521]: 2022-05-14T04:37:54.161+0000 7f5340779700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T04:37:54.548 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:37:54 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T04:37:54.548 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:37:54 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:37:54.548 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:37:54 smithi047 conmon[78521]: yield 2022-05-14T04:37:54.548 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:37:54 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:37:54.548 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:37:54 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:37:54.549 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:37:54 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:37:54.549 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:37:54 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T04:37:54.549 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:37:54 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:37:54.549 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:37:54 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T04:37:54.549 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:37:54 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:37:54.550 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:37:54 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T04:37:54.550 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:37:54 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:37:54.550 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:37:54 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T04:37:54.552 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:37:54 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:37:54.552 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:37:54 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T04:37:54.553 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:37:54 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:37:54.553 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:37:54 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:37:54.553 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:37:54 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:37:54.554 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:37:54 smithi047 conmon[78521]: 2022-05-14T04:37:54.554 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:37:54 smithi047 conmon[78521]: During handling of the above exception, another exception occurred: 2022-05-14T04:37:54.555 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:37:54 smithi047 conmon[78521]: 2022-05-14T04:37:54.555 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:37:54 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T04:37:54.556 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:37:54 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T04:37:54.556 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:37:54 smithi047 conmon[78521]: return f(*arg) 2022-05-14T04:37:54.556 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:37:54 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T04:37:54.557 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:37:54 smithi047 conmon[78521]: self._write_client_files(client_files, host) 2022-05-14T04:37:54.557 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:37:54 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T04:37:54.558 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:37:54 smithi047 conmon[78521]: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T04:37:54.558 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:37:54 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T04:37:54.559 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:37:54 smithi047 conmon[78521]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T04:37:54.560 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:37:54 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T04:37:54.560 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:37:54 smithi047 conmon[78521]: return self.event_loop.get_result(coro) 2022-05-14T04:37:54.560 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:37:54 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T04:37:54.561 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:37:54 smithi047 conmon[78521]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T04:37:54.561 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:37:54 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T04:37:54.562 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:37:54 smithi047 conmon[78521]: return self.__get_result() 2022-05-14T04:37:54.562 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:37:54 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T04:37:54.562 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:37:54 smithi047 conmon[78521]: raise self._exception 2022-05-14T04:37:54.562 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:37:54 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T04:37:54.563 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:37:54 smithi047 conmon[78521]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T04:37:54.563 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:37:54 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T04:37:54.563 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:37:54 smithi047 conmon[78521]: conn = await self._remote_connection(host, addr) 2022-05-14T04:37:54.563 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:37:54 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T04:37:54.564 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:37:54 smithi047 conmon[78521]: raise 2022-05-14T04:37:54.570 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:37:54 smithi047 conmon[78521]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T04:37:54.570 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:37:54 smithi047 conmon[78521]: self.gen.throw(type, value, traceback) 2022-05-14T04:37:54.570 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:37:54 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T04:37:54.570 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:37:54 smithi047 conmon[78521]: raise OrchestratorError(msg) 2022-05-14T04:37:54.571 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:37:54 smithi047 conmon[78521]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:37:54.610 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:37:54 smithi115 conmon[90393]: 2022-05-14T04:37:54.143+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:37:55.295 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:37:54 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:37:55.295 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:37:54 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:37:55.295 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:37:54 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:37:55.295 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:37:54 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:37:55.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T04:37:55.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:37:55.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T04:37:55.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:37:55.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:37:55.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:37:55.298 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T04:37:55.299 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:37:55.299 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T04:37:55.299 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:37:55.299 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T04:37:55.300 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:37:55.300 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T04:37:55.300 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:37:55.300 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T04:37:55.300 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:37:55.301 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:37:55.301 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:37:55.301 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:37:54 smithi047 ceph-mon[77147]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T04:37:55.301 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T04:37:55.302 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:37:55.302 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T04:37:55.302 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:37:55.302 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:37:55.303 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:37:55.303 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T04:37:55.303 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:37:55.303 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T04:37:55.303 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:37:55.304 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T04:37:55.304 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:37:55.304 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T04:37:55.304 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:37:55.305 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T04:37:55.305 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:37:55.305 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:37:55.305 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:37:55.306 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T04:37:55.306 INFO:journalctl@ceph.mon.b.smithi047.stdout: During handling of the above exception, another exception occurred: 2022-05-14T04:37:55.306 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T04:37:55.306 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T04:37:55.306 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T04:37:55.307 INFO:journalctl@ceph.mon.b.smithi047.stdout: return f(*arg) 2022-05-14T04:37:55.307 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T04:37:55.307 INFO:journalctl@ceph.mon.b.smithi047.stdout: self._write_client_files(client_files, host) 2022-05-14T04:37:55.307 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T04:37:55.308 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T04:37:55.308 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T04:37:55.311 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T04:37:55.311 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T04:37:55.311 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.event_loop.get_result(coro) 2022-05-14T04:37:55.311 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T04:37:55.312 INFO:journalctl@ceph.mon.b.smithi047.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T04:37:55.312 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T04:37:55.312 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.__get_result() 2022-05-14T04:37:55.312 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T04:37:55.312 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise self._exception 2022-05-14T04:37:55.313 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T04:37:55.313 INFO:journalctl@ceph.mon.b.smithi047.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T04:37:55.313 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T04:37:55.313 INFO:journalctl@ceph.mon.b.smithi047.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T04:37:55.314 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T04:37:55.314 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise 2022-05-14T04:37:55.314 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T04:37:55.314 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.gen.throw(type, value, traceback) 2022-05-14T04:37:55.314 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T04:37:55.315 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OrchestratorError(msg) 2022-05-14T04:37:55.315 INFO:journalctl@ceph.mon.b.smithi047.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:37:55.360 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:37:55 smithi115 conmon[90393]: 2022-05-14T04:37:55.111+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:37:55.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:37:54 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:37:55.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:37:54 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:37:55.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:37:54 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:37:55.362 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:37:54 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:37:55.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T04:37:55.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:37:55.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T04:37:55.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:37:55.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:37:55.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:37:55.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T04:37:55.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:37:55.365 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T04:37:55.365 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:37:55.365 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T04:37:55.365 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:37:55.365 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T04:37:55.366 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:37:55.366 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T04:37:55.366 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:37:55.366 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:37:55.369 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:37:55.370 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:37:54 smithi115 ceph-mon[78593]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T04:37:55.370 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T04:37:55.370 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:37:55.370 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T04:37:55.371 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:37:55.371 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:37:55.371 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:37:55.371 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T04:37:55.371 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:37:55.372 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T04:37:55.372 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:37:55.372 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T04:37:55.372 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:37:55.373 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T04:37:55.373 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:37:55.373 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T04:37:55.373 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:37:55.373 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:37:55.374 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:37:55.374 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T04:37:55.374 INFO:journalctl@ceph.mon.c.smithi115.stdout: During handling of the above exception, another exception occurred: 2022-05-14T04:37:55.374 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T04:37:55.375 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T04:37:55.375 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T04:37:55.375 INFO:journalctl@ceph.mon.c.smithi115.stdout: return f(*arg) 2022-05-14T04:37:55.375 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T04:37:55.375 INFO:journalctl@ceph.mon.c.smithi115.stdout: self._write_client_files(client_files, host) 2022-05-14T04:37:55.376 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T04:37:55.376 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T04:37:55.376 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T04:37:55.376 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T04:37:55.377 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T04:37:55.377 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.event_loop.get_result(coro) 2022-05-14T04:37:55.377 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T04:37:55.377 INFO:journalctl@ceph.mon.c.smithi115.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T04:37:55.377 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T04:37:55.378 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.__get_result() 2022-05-14T04:37:55.378 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T04:37:55.378 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise self._exception 2022-05-14T04:37:55.378 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T04:37:55.378 INFO:journalctl@ceph.mon.c.smithi115.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T04:37:55.379 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T04:37:55.379 INFO:journalctl@ceph.mon.c.smithi115.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T04:37:55.379 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T04:37:55.379 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise 2022-05-14T04:37:55.380 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T04:37:55.380 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.gen.throw(type, value, traceback) 2022-05-14T04:37:55.380 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T04:37:55.380 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OrchestratorError(msg) 2022-05-14T04:37:55.380 INFO:journalctl@ceph.mon.c.smithi115.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:37:56.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:37:55 smithi047 ceph-mon[77147]: pgmap v9655: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:37:56.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:37:55 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:37:56.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:37:55 smithi115 ceph-mon[78593]: pgmap v9655: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:37:56.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:37:55 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:37:56.360 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:37:56 smithi115 conmon[90393]: 2022-05-14T04:37:56.142+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:37:57.151 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:37:56 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:37:57.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:37:56 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:37:57.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:37:57 smithi115 conmon[90393]: 2022-05-14T04:37:57.149+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:37:58.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:37:57 smithi047 ceph-mon[77147]: pgmap v9656: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:37:58.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:37:57 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:37:58.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:37:57 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 18611 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:37:58.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:37:58 smithi115 conmon[90393]: 2022-05-14T04:37:58.120+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:37:58.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:37:57 smithi115 ceph-mon[78593]: pgmap v9656: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:37:58.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:37:57 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:37:58.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:37:57 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 18611 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:37:59.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:37:58 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:37:59.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:37:58 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:37:59.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:37:58 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:37:59.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:37:59 smithi115 conmon[90393]: 2022-05-14T04:37:59.132+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:37:59.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:37:58 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:37:59.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:37:58 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:37:59.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:37:58 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:38:00.296 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:37:59 smithi047 ceph-mon[77147]: pgmap v9657: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:38:00.296 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:37:59 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:38:00.362 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:37:59 smithi115 ceph-mon[78593]: pgmap v9657: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:38:00.362 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:37:59 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:38:00.362 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:38:00 smithi115 conmon[90393]: 2022-05-14T04:38:00.144+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:38:01.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:38:00 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:38:01.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:38:00 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:38:01.360 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:38:01 smithi115 conmon[90393]: 2022-05-14T04:38:01.119+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:38:02.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:38:01 smithi047 ceph-mon[77147]: pgmap v9658: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:38:02.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:38:01 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:38:02.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:38:01 smithi115 ceph-mon[78593]: pgmap v9658: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:38:02.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:38:01 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:38:02.360 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:38:02 smithi115 conmon[90393]: 2022-05-14T04:38:02.119+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:38:03.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:38:02 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:38:03.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:38:02 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:38:03.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:38:02 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:38:03.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:38:02 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 18616 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:38:03.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:38:03 smithi115 conmon[90393]: 2022-05-14T04:38:03.160+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:38:03.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:38:02 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:38:03.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:38:02 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:38:03.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:38:02 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:38:03.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:38:02 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 18616 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:38:04.587 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:38:04 smithi115 conmon[90393]: 2022-05-14T04:38:04.130+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:38:04.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:38:04 smithi115 ceph-mon[78593]: pgmap v9659: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:38:04.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:38:04 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:38:05.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:38:04 smithi047 ceph-mon[77147]: pgmap v9659: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:38:05.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:38:04 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:38:05.595 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:38:05 smithi115 conmon[90393]: 2022-05-14T04:38:05.110+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:38:05.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:38:05 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:38:06.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:38:05 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:38:06.595 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:38:06 smithi115 conmon[90393]: 2022-05-14T04:38:06.137+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:38:06.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:38:06 smithi115 ceph-mon[78593]: pgmap v9660: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:38:06.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:38:06 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:38:06.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:38:06 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:38:07.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:38:06 smithi047 ceph-mon[77147]: pgmap v9660: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:38:07.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:38:06 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:38:07.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:38:06 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:38:07.598 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:38:07 smithi115 conmon[90393]: 2022-05-14T04:38:07.161+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:38:07.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:38:07 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:38:07.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:38:07 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 18621 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:38:08.048 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:38:07 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:38:08.049 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:38:07 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 18621 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:38:08.600 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:38:08 smithi115 conmon[90393]: 2022-05-14T04:38:08.199+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:38:08.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:38:08 smithi115 ceph-mon[78593]: pgmap v9661: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:38:08.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:38:08 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:38:09.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:38:08 smithi047 ceph-mon[77147]: pgmap v9661: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:38:09.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:38:08 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:38:09.605 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:38:09 smithi115 conmon[90393]: 2022-05-14T04:38:09.190+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:38:09.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:38:09 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:38:10.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:38:09 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:38:10.607 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:38:10 smithi115 conmon[90393]: 2022-05-14T04:38:10.150+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:38:10.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:38:10 smithi115 ceph-mon[78593]: pgmap v9662: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:38:10.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:38:10 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:38:10.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:38:10 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:38:10.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:38:10 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:38:11.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:38:10 smithi047 ceph-mon[77147]: pgmap v9662: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:38:11.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:38:10 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:38:11.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:38:10 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:38:11.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:38:10 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:38:11.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:38:11 smithi115 conmon[90393]: 2022-05-14T04:38:11.116+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:38:12.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:38:11 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:38:12.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:38:11 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:38:12.610 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:38:12 smithi115 conmon[90393]: 2022-05-14T04:38:12.151+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:38:13.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:38:12 smithi047 ceph-mon[77147]: pgmap v9663: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:38:13.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:38:12 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:38:13.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:38:12 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 18625 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:38:13.112 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:38:12 smithi115 ceph-mon[78593]: pgmap v9663: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:38:13.113 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:38:12 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:38:13.113 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:38:12 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 18625 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:38:13.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:38:13 smithi115 conmon[90393]: 2022-05-14T04:38:13.125+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:38:14.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:38:13 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:38:14.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:38:13 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:38:14.611 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:38:14 smithi115 conmon[90393]: 2022-05-14T04:38:14.150+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:38:15.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:38:14 smithi047 ceph-mon[77147]: pgmap v9664: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:38:15.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:38:14 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:38:15.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:38:14 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:38:15.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:38:14 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:38:15.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:38:14 smithi115 ceph-mon[78593]: pgmap v9664: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:38:15.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:38:14 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:38:15.112 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:38:14 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:38:15.112 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:38:14 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:38:15.610 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:38:15 smithi115 conmon[90393]: 2022-05-14T04:38:15.123+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:38:16.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:38:15 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T04:38:16.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:38:15 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T04:38:16.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:38:15 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:38:16.095 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:38:15 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T04:38:16.096 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:38:15 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T04:38:16.096 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:38:15 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:38:16.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:38:16 smithi115 conmon[90393]: 2022-05-14T04:38:16.092+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:38:17.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:38:16 smithi047 ceph-mon[77147]: pgmap v9665: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:38:17.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:38:16 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:38:17.056 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:38:16 smithi115 ceph-mon[78593]: pgmap v9665: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:38:17.056 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:38:16 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:38:17.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:38:17 smithi115 conmon[90393]: 2022-05-14T04:38:17.053+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:38:18.025 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:38:17 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:38:18.026 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:38:17 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 18631 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:38:18.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:38:17 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:38:18.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:38:17 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 18631 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:38:18.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:38:18 smithi115 conmon[90393]: 2022-05-14T04:38:18.022+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:38:19.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:38:18 smithi047 ceph-mon[77147]: pgmap v9666: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:38:19.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:38:18 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:38:19.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:38:18 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:38:19.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:38:18 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:38:19.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:38:18 smithi115 ceph-mon[78593]: pgmap v9666: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:38:19.048 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:38:18 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:38:19.048 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:38:18 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:38:19.048 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:38:18 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:38:19.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:38:19 smithi115 conmon[90393]: 2022-05-14T04:38:19.044+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:38:20.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:38:19 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:38:20.064 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:38:19 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:38:20.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:38:20 smithi115 conmon[90393]: 2022-05-14T04:38:20.061+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:38:21.025 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:38:20 smithi115 ceph-mon[78593]: pgmap v9667: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:38:21.025 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:38:20 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:38:21.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:38:20 smithi047 ceph-mon[77147]: pgmap v9667: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:38:21.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:38:20 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:38:21.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:38:21 smithi115 conmon[90393]: 2022-05-14T04:38:21.022+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:38:22.008 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:38:21 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:38:22.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:38:21 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:38:22.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:38:22 smithi115 conmon[90393]: 2022-05-14T04:38:22.004+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:38:22.987 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:38:22 smithi115 ceph-mon[78593]: pgmap v9668: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:38:22.988 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:38:22 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:38:22.988 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:38:22 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:38:22.988 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:38:22 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:38:22.989 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:38:22 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 18635 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:38:23.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:38:22 smithi047 ceph-mon[77147]: pgmap v9668: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:38:23.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:38:22 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:38:23.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:38:22 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:38:23.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:38:22 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:38:23.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:38:22 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 18635 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:38:23.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:38:22 smithi115 conmon[90393]: 2022-05-14T04:38:22.985+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:38:24.003 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:38:23 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:38:24.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:38:23 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:38:24.362 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:38:24 smithi115 conmon[90393]: 2022-05-14T04:38:24.000+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:38:24.998 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:38:24 smithi115 ceph-mon[78593]: pgmap v9669: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:38:24.998 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:38:24 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:38:25.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:38:24 smithi047 ceph-mon[77147]: pgmap v9669: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:38:25.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:38:24 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:38:25.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:38:24 smithi115 conmon[90393]: 2022-05-14T04:38:24.995+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:38:26.025 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:38:25 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:38:26.045 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:38:25 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:38:26.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:38:26 smithi115 conmon[90393]: 2022-05-14T04:38:26.022+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:38:27.006 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:38:26 smithi115 ceph-mon[78593]: pgmap v9670: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:38:27.007 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:38:26 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:38:27.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:38:26 smithi047 ceph-mon[77147]: pgmap v9670: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:38:27.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:38:26 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:38:27.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:38:27 smithi115 conmon[90393]: 2022-05-14T04:38:27.004+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:38:28.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:38:27 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:38:28.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:38:27 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 18640 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:38:28.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:38:27 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:38:28.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:38:27 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 18640 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:38:28.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:38:28 smithi115 conmon[90393]: 2022-05-14T04:38:28.041+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:38:29.006 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:38:28 smithi115 ceph-mon[78593]: pgmap v9671: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:38:29.007 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:38:28 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:38:29.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:38:28 smithi047 ceph-mon[77147]: pgmap v9671: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:38:29.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:38:28 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:38:29.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:38:29 smithi115 conmon[90393]: 2022-05-14T04:38:29.002+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:38:30.016 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:38:29 smithi115 ceph-mon[78593]: pgmap v9672: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:38:30.017 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:38:29 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:38:30.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:38:29 smithi047 ceph-mon[77147]: pgmap v9672: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:38:30.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:38:29 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:38:30.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:38:30 smithi115 conmon[90393]: 2022-05-14T04:38:30.013+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:38:30.979 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:38:30 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:38:30.979 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:38:30 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:38:30.980 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:38:30 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:38:31.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:38:30 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:38:31.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:38:30 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:38:31.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:38:30 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:38:31.360 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:38:30 smithi115 conmon[90393]: 2022-05-14T04:38:30.974+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:38:31.993 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:38:31 smithi115 ceph-mon[78593]: pgmap v9673: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:38:31.994 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:38:31 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:38:32.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:38:31 smithi047 ceph-mon[77147]: pgmap v9673: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:38:32.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:38:31 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:38:32.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:38:31 smithi115 conmon[90393]: 2022-05-14T04:38:31.990+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:38:32.945 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:38:32 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:38:32.945 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:38:32 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 18645 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:38:33.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:38:32 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:38:33.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:38:32 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 18645 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:38:33.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:38:32 smithi115 conmon[90393]: 2022-05-14T04:38:32.942+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:38:34.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:38:33 smithi047 ceph-mon[77147]: pgmap v9674: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:38:34.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:38:33 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:38:34.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:38:33 smithi115 conmon[90393]: 2022-05-14T04:38:33.894+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:38:34.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:38:33 smithi115 ceph-mon[78593]: pgmap v9674: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:38:34.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:38:33 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:38:34.945 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:38:34 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:38:34.946 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:38:34 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:38:34.946 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:38:34 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:38:35.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:38:34 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:38:35.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:38:34 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:38:35.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:38:34 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:38:35.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:38:34 smithi115 conmon[90393]: 2022-05-14T04:38:34.942+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:38:35.991 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:38:35 smithi115 ceph-mon[78593]: pgmap v9675: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:38:35.991 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:38:35 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:38:36.047 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:38:35 smithi047 ceph-mon[77147]: pgmap v9675: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:38:36.047 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:38:35 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:38:36.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:38:35 smithi115 conmon[90393]: 2022-05-14T04:38:35.988+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:38:36.958 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:38:36 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:38:37.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:38:36 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:38:37.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:38:36 smithi115 conmon[90393]: 2022-05-14T04:38:36.956+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:38:38.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:38:37 smithi047 ceph-mon[77147]: pgmap v9676: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:38:38.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:38:37 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:38:38.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:38:37 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 18650 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:38:38.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:38:37 smithi115 conmon[90393]: 2022-05-14T04:38:37.910+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:38:38.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:38:37 smithi115 ceph-mon[78593]: pgmap v9676: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:38:38.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:38:37 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:38:38.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:38:37 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 18650 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:38:39.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:38:38 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:38:39.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:38:38 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:38:39.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:38:38 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:38:39.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:38:38 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:38:39.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:38:38 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:38:39.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:38:38 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:38:39.110 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:38:38 smithi115 conmon[90393]: 2022-05-14T04:38:38.895+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:38:40.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:38:39 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:38:40.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:38:39 smithi047 ceph-mon[77147]: pgmap v9677: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:38:40.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:38:39 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:38:40.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:38:39 smithi115 ceph-mon[78593]: pgmap v9677: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:38:40.110 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:38:39 smithi115 conmon[90393]: 2022-05-14T04:38:39.861+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:38:41.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:38:40 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:38:41.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:38:40 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:38:41.110 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:38:40 smithi115 conmon[90393]: 2022-05-14T04:38:40.834+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:38:42.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:38:41 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:38:42.045 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:38:41 smithi047 ceph-mon[77147]: pgmap v9678: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:38:42.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:38:41 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:38:42.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:38:41 smithi115 ceph-mon[78593]: pgmap v9678: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:38:42.111 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:38:41 smithi115 conmon[90393]: 2022-05-14T04:38:41.876+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:38:43.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:38:42 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:38:43.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:38:42 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:38:43.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:38:42 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:38:43.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:38:42 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 18655 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:38:43.110 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:38:42 smithi115 conmon[90393]: 2022-05-14T04:38:42.908+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:38:43.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:38:42 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:38:43.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:38:42 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:38:43.112 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:38:42 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:38:43.112 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:38:42 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 18655 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:38:44.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:38:43 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:38:44.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:38:43 smithi047 ceph-mon[77147]: pgmap v9679: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:38:44.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:38:43 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:38:44.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:38:43 smithi115 ceph-mon[78593]: pgmap v9679: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:38:44.110 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:38:43 smithi115 conmon[90393]: 2022-05-14T04:38:43.864+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:38:45.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:38:44 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:38:45.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:38:44 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:38:45.111 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:38:44 smithi115 conmon[90393]: 2022-05-14T04:38:44.832+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:38:46.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:38:45 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:38:46.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:38:45 smithi047 ceph-mon[77147]: pgmap v9680: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:38:46.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:38:45 smithi115 conmon[90393]: 2022-05-14T04:38:45.875+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:38:46.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:38:45 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:38:46.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:38:45 smithi115 ceph-mon[78593]: pgmap v9680: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:38:47.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:38:46 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:38:47.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:38:46 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:38:47.110 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:38:46 smithi115 conmon[90393]: 2022-05-14T04:38:46.891+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:38:48.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:38:47 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:38:48.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:38:47 smithi047 ceph-mon[77147]: pgmap v9681: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:38:48.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:38:47 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 18660 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:38:48.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:38:47 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:38:48.112 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:38:47 smithi115 ceph-mon[78593]: pgmap v9681: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:38:48.112 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:38:47 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 18660 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:38:48.112 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:38:47 smithi115 conmon[90393]: 2022-05-14T04:38:47.875+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:38:49.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:38:48 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:38:49.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:38:48 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:38:49.110 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:38:48 smithi115 conmon[90393]: 2022-05-14T04:38:48.876+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:38:50.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:38:49 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:38:50.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:38:49 smithi047 ceph-mon[77147]: pgmap v9682: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:38:50.114 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:38:49 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:38:50.114 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:38:49 smithi115 ceph-mon[78593]: pgmap v9682: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:38:50.115 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:38:49 smithi115 conmon[90393]: 2022-05-14T04:38:49.922+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:38:51.050 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:38:50 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:38:51.051 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:38:50 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:38:51.051 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:38:50 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:38:51.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:38:50 smithi115 conmon[90393]: 2022-05-14T04:38:50.873+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:38:51.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:38:50 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:38:51.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:38:50 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:38:51.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:38:50 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:38:52.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:38:51 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:38:52.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:38:51 smithi047 ceph-mon[77147]: pgmap v9683: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:38:52.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:38:51 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:38:52.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:38:51 smithi115 ceph-mon[78593]: pgmap v9683: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:38:52.110 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:38:51 smithi115 conmon[90393]: 2022-05-14T04:38:51.888+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:38:53.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:38:52 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:38:53.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:38:52 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 18665 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:38:53.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:38:52 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:38:53.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:38:52 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 18665 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:38:53.110 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:38:52 smithi115 conmon[90393]: 2022-05-14T04:38:52.910+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:38:54.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:38:53 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:38:54.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:38:53 smithi047 ceph-mon[77147]: pgmap v9684: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:38:54.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:38:53 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:38:54.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:38:53 smithi115 ceph-mon[78593]: pgmap v9684: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:38:54.110 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:38:53 smithi115 conmon[90393]: 2022-05-14T04:38:53.949+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:38:54.984 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:38:54 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:38:54.984 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:38:54 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:38:54.985 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:38:54 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:38:54.985 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:38:54 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T04:38:54.985 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:38:54 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T04:38:54.985 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:38:54 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T04:38:55.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:38:54 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:38:55.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:38:54 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:38:55.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:38:54 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:38:55.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:38:54 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T04:38:55.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:38:54 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T04:38:55.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:38:54 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T04:38:55.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:38:54 smithi047 conmon[78521]: 2022-05-14T04:38:54.767+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:38:55.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:38:54 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T04:38:55.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:38:54 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:38:55.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:38:54 smithi047 conmon[78521]: yield 2022-05-14T04:38:55.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:38:54 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:38:55.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:38:54 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:38:55.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:38:54 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:38:55.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:38:54 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T04:38:55.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:38:54 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:38:55.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:38:54 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T04:38:55.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:38:54 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:38:55.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:38:54 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T04:38:55.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:38:54 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:38:55.048 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:38:54 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T04:38:55.048 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:38:54 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:38:55.048 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:38:54 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T04:38:55.048 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:38:54 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:38:55.049 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:38:54 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:38:55.049 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:38:54 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:38:55.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:38:54 smithi115 conmon[90393]: 2022-05-14T04:38:54.981+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:38:56.024 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:38:55 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:38:56.025 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T04:38:56.025 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:38:56.025 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T04:38:56.025 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:38:56.026 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:38:56.026 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:38:56.026 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T04:38:56.026 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:38:56.027 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T04:38:56.027 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:38:56.027 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T04:38:56.027 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:38:56.027 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T04:38:56.028 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:38:56.028 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T04:38:56.028 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:38:56.028 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:38:56.029 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:38:56.029 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:38:55 smithi115 ceph-mon[78593]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T04:38:56.033 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:38:55 smithi115 ceph-mon[78593]: pgmap v9685: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:38:56.034 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:38:55 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:38:56.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:38:55 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:38:56.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T04:38:56.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:38:56.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T04:38:56.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:38:56.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:38:56.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:38:56.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T04:38:56.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:38:56.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T04:38:56.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:38:56.049 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T04:38:56.049 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:38:56.051 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T04:38:56.051 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:38:56.051 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T04:38:56.052 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:38:56.052 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:38:56.052 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:38:56.052 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:38:55 smithi047 ceph-mon[77147]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T04:38:56.053 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:38:55 smithi047 ceph-mon[77147]: pgmap v9685: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:38:56.053 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:38:55 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:38:56.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:38:56 smithi115 conmon[90393]: 2022-05-14T04:38:56.021+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:38:57.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:38:56 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:38:57.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:38:57 smithi115 conmon[90393]: 2022-05-14T04:38:56.999+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:38:57.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:38:56 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:38:58.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:38:57 smithi047 ceph-mon[77147]: pgmap v9686: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:38:58.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:38:57 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:38:58.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:38:57 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 18670 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:38:58.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:38:57 smithi047 conmon[78521]: 2022-05-14T04:38:57.839+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:38:58.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:38:57 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T04:38:58.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:38:57 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:38:58.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:38:57 smithi047 conmon[78521]: yield 2022-05-14T04:38:58.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:38:57 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:38:58.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:38:57 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:38:58.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:38:57 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:38:58.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:38:57 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T04:38:58.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:38:57 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:38:58.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:38:57 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T04:38:58.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:38:57 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:38:58.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:38:57 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T04:38:58.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:38:57 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:38:58.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:38:57 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T04:38:58.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:38:57 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:38:58.048 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:38:57 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T04:38:58.048 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:38:57 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:38:58.048 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:38:57 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:38:58.048 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:38:57 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:38:58.049 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:38:57 smithi047 conmon[78521]: 2022-05-14T04:38:57.840+0000 7f5344f82700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T04:38:58.049 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:38:57 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T04:38:58.049 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:38:57 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:38:58.049 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:38:57 smithi047 conmon[78521]: yield 2022-05-14T04:38:58.049 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:38:57 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:38:58.050 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:38:57 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:38:58.050 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:38:57 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:38:58.050 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:38:57 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T04:38:58.050 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:38:57 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:38:58.051 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:38:57 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T04:38:58.051 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:38:57 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:38:58.052 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:38:57 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T04:38:58.052 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:38:57 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:38:58.052 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:38:57 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T04:38:58.052 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:38:57 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:38:58.053 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:38:57 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T04:38:58.053 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:38:57 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:38:58.053 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:38:57 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:38:58.053 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:38:57 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:38:58.054 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:38:57 smithi047 conmon[78521]: 2022-05-14T04:38:58.054 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:38:57 smithi047 conmon[78521]: During handling of the above exception, another exception occurred: 2022-05-14T04:38:58.054 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:38:57 smithi047 conmon[78521]: 2022-05-14T04:38:58.054 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:38:57 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T04:38:58.054 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:38:57 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T04:38:58.055 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:38:57 smithi047 conmon[78521]: return f(*arg) 2022-05-14T04:38:58.055 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:38:57 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T04:38:58.055 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:38:57 smithi047 conmon[78521]: self._write_client_files(client_files, host) 2022-05-14T04:38:58.055 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:38:57 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T04:38:58.055 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:38:57 smithi047 conmon[78521]: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T04:38:58.056 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:38:57 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T04:38:58.056 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:38:57 smithi047 conmon[78521]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T04:38:58.056 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:38:57 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T04:38:58.056 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:38:57 smithi047 conmon[78521]: return self.event_loop.get_result(coro) 2022-05-14T04:38:58.057 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:38:57 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T04:38:58.057 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:38:57 smithi047 conmon[78521]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T04:38:58.057 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:38:57 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T04:38:58.057 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:38:57 smithi047 conmon[78521]: return self.__get_result() 2022-05-14T04:38:58.058 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:38:57 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T04:38:58.058 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:38:57 smithi047 conmon[78521]: raise self._exception 2022-05-14T04:38:58.058 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:38:57 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T04:38:58.059 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:38:57 smithi047 conmon[78521]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T04:38:58.059 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:38:57 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T04:38:58.059 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:38:57 smithi047 conmon[78521]: conn = await self._remote_connection(host, addr) 2022-05-14T04:38:58.059 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:38:57 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T04:38:58.059 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:38:57 smithi047 conmon[78521]: raise 2022-05-14T04:38:58.060 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:38:57 smithi047 conmon[78521]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T04:38:58.060 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:38:57 smithi047 conmon[78521]: self.gen.throw(type, value, traceback) 2022-05-14T04:38:58.060 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:38:57 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T04:38:58.060 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:38:57 smithi047 conmon[78521]: raise OrchestratorError(msg) 2022-05-14T04:38:58.060 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:38:57 smithi047 conmon[78521]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:38:58.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:38:57 smithi115 conmon[90393]: 2022-05-14T04:38:57.992+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:38:58.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:38:57 smithi115 ceph-mon[78593]: pgmap v9686: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:38:58.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:38:57 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:38:58.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:38:57 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 18670 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:38:59.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:38:58 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:38:59.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:38:58 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:38:59.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:38:58 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:38:59.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T04:38:59.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:38:59.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T04:38:59.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:38:59.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:38:59.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:38:59.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T04:38:59.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:38:59.049 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T04:38:59.049 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:38:59.049 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T04:38:59.050 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:38:59.050 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T04:38:59.051 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:38:59.051 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T04:38:59.052 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:38:59.053 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:38:59.053 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:38:59.054 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:38:58 smithi047 ceph-mon[77147]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T04:38:59.054 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T04:38:59.054 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:38:59.054 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T04:38:59.055 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:38:59.055 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:38:59.055 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:38:59.055 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T04:38:59.055 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:38:59.056 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T04:38:59.056 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:38:59.056 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T04:38:59.056 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:38:59.056 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T04:38:59.057 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:38:59.057 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T04:38:59.057 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:38:59.057 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:38:59.058 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:38:59.058 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T04:38:59.058 INFO:journalctl@ceph.mon.b.smithi047.stdout: During handling of the above exception, another exception occurred: 2022-05-14T04:38:59.058 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T04:38:59.059 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T04:38:59.059 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T04:38:59.059 INFO:journalctl@ceph.mon.b.smithi047.stdout: return f(*arg) 2022-05-14T04:38:59.059 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T04:38:59.060 INFO:journalctl@ceph.mon.b.smithi047.stdout: self._write_client_files(client_files, host) 2022-05-14T04:38:59.060 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T04:38:59.060 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T04:38:59.060 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T04:38:59.060 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T04:38:59.061 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T04:38:59.061 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.event_loop.get_result(coro) 2022-05-14T04:38:59.062 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T04:38:59.062 INFO:journalctl@ceph.mon.b.smithi047.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T04:38:59.062 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T04:38:59.062 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.__get_result() 2022-05-14T04:38:59.063 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T04:38:59.063 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise self._exception 2022-05-14T04:38:59.063 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T04:38:59.063 INFO:journalctl@ceph.mon.b.smithi047.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T04:38:59.064 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T04:38:59.065 INFO:journalctl@ceph.mon.b.smithi047.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T04:38:59.065 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T04:38:59.065 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise 2022-05-14T04:38:59.066 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T04:38:59.066 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.gen.throw(type, value, traceback) 2022-05-14T04:38:59.066 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T04:38:59.067 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OrchestratorError(msg) 2022-05-14T04:38:59.067 INFO:journalctl@ceph.mon.b.smithi047.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:38:59.067 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:38:58 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:38:59.116 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:38:58 smithi115 conmon[90393]: 2022-05-14T04:38:58.975+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:38:59.118 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:38:58 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:38:59.118 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:38:58 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:38:59.118 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:38:58 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:38:59.118 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T04:38:59.119 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:38:59.119 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T04:38:59.119 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:38:59.119 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:38:59.120 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:38:59.120 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T04:38:59.120 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:38:59.120 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T04:38:59.120 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:38:59.121 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T04:38:59.121 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:38:59.121 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T04:38:59.121 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:38:59.122 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T04:38:59.122 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:38:59.122 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:38:59.122 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:38:59.122 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:38:58 smithi115 ceph-mon[78593]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T04:38:59.123 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T04:38:59.123 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:38:59.123 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T04:38:59.123 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:38:59.124 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:38:59.124 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:38:59.124 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T04:38:59.124 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:38:59.124 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T04:38:59.126 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:38:59.126 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T04:38:59.126 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:38:59.127 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T04:38:59.127 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:38:59.127 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T04:38:59.127 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:38:59.128 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:38:59.128 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:38:59.128 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T04:38:59.128 INFO:journalctl@ceph.mon.c.smithi115.stdout: During handling of the above exception, another exception occurred: 2022-05-14T04:38:59.128 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T04:38:59.129 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T04:38:59.129 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T04:38:59.129 INFO:journalctl@ceph.mon.c.smithi115.stdout: return f(*arg) 2022-05-14T04:38:59.130 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T04:38:59.130 INFO:journalctl@ceph.mon.c.smithi115.stdout: self._write_client_files(client_files, host) 2022-05-14T04:38:59.130 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T04:38:59.130 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T04:38:59.130 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T04:38:59.131 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T04:38:59.131 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T04:38:59.131 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.event_loop.get_result(coro) 2022-05-14T04:38:59.131 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T04:38:59.131 INFO:journalctl@ceph.mon.c.smithi115.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T04:38:59.132 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T04:38:59.132 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.__get_result() 2022-05-14T04:38:59.132 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T04:38:59.132 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise self._exception 2022-05-14T04:38:59.133 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T04:38:59.133 INFO:journalctl@ceph.mon.c.smithi115.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T04:38:59.133 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T04:38:59.133 INFO:journalctl@ceph.mon.c.smithi115.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T04:38:59.134 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T04:38:59.134 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise 2022-05-14T04:38:59.134 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T04:38:59.135 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.gen.throw(type, value, traceback) 2022-05-14T04:38:59.135 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T04:38:59.135 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OrchestratorError(msg) 2022-05-14T04:38:59.135 INFO:journalctl@ceph.mon.c.smithi115.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:38:59.135 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:38:58 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:39:00.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:38:59 smithi047 ceph-mon[77147]: pgmap v9687: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:39:00.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:38:59 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:39:00.111 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:39:00 smithi115 conmon[90393]: 2022-05-14T04:39:00.000+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:39:00.112 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:38:59 smithi115 ceph-mon[78593]: pgmap v9687: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:39:00.113 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:38:59 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:39:01.026 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:39:00 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:39:01.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:39:00 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:39:01.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:39:01 smithi115 conmon[90393]: 2022-05-14T04:39:01.023+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:39:02.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:39:01 smithi047 ceph-mon[77147]: pgmap v9688: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:39:02.045 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:39:01 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:39:02.070 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:39:01 smithi115 ceph-mon[78593]: pgmap v9688: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:39:02.071 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:39:01 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:39:02.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:39:02 smithi115 conmon[90393]: 2022-05-14T04:39:02.067+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:39:03.032 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:39:02 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:39:03.033 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:39:02 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:39:03.033 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:39:02 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:39:03.034 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:39:02 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 18675 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:39:03.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:39:02 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:39:03.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:39:02 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:39:03.045 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:39:02 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:39:03.045 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:39:02 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 18675 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:39:03.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:39:03 smithi115 conmon[90393]: 2022-05-14T04:39:03.028+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:39:04.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:39:03 smithi047 ceph-mon[77147]: pgmap v9689: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:39:04.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:39:03 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:39:04.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:39:03 smithi115 ceph-mon[78593]: pgmap v9689: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:39:04.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:39:03 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:39:04.110 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:39:04 smithi115 conmon[90393]: 2022-05-14T04:39:04.028+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:39:05.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:39:04 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:39:05.077 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:39:04 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:39:05.360 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:39:05 smithi115 conmon[90393]: 2022-05-14T04:39:05.074+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:39:06.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:39:05 smithi047 ceph-mon[77147]: pgmap v9690: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:39:06.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:39:05 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:39:06.098 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:39:05 smithi115 ceph-mon[78593]: pgmap v9690: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:39:06.099 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:39:05 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:39:06.360 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:39:06 smithi115 conmon[90393]: 2022-05-14T04:39:06.095+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:39:07.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:39:06 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:39:07.072 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:39:06 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:39:07.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:39:07 smithi115 conmon[90393]: 2022-05-14T04:39:07.069+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:39:08.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:39:07 smithi115 ceph-mon[78593]: pgmap v9691: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:39:08.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:39:07 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:39:08.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:39:07 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 18680 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:39:08.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:39:07 smithi047 ceph-mon[77147]: pgmap v9691: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:39:08.295 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:39:07 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:39:08.295 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:39:07 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 18680 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:39:08.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:39:08 smithi115 conmon[90393]: 2022-05-14T04:39:08.105+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:39:09.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:39:08 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:39:09.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:39:08 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:39:09.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:39:09 smithi115 conmon[90393]: 2022-05-14T04:39:09.113+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:39:10.076 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:39:09 smithi115 ceph-mon[78593]: pgmap v9692: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:39:10.076 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:39:09 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:39:10.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:39:09 smithi047 ceph-mon[77147]: pgmap v9692: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:39:10.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:39:09 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:39:10.361 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:39:10 smithi115 conmon[90393]: 2022-05-14T04:39:10.073+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:39:11.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:39:11 smithi115 conmon[90393]: 2022-05-14T04:39:11.031+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:39:11.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:39:10 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:39:11.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:39:10 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:39:11.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:39:10 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:39:11.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:39:10 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:39:11.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:39:10 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:39:11.304 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:39:10 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:39:12.110 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:39:12 smithi115 conmon[90393]: 2022-05-14T04:39:12.047+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:39:12.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:39:11 smithi115 ceph-mon[78593]: pgmap v9693: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:39:12.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:39:11 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:39:12.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:39:11 smithi047 ceph-mon[77147]: pgmap v9693: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:39:12.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:39:11 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:39:13.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:39:13 smithi115 conmon[90393]: 2022-05-14T04:39:13.045+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:39:13.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:39:12 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:39:13.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:39:12 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 18686 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:39:13.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:39:12 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:39:13.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:39:12 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 18686 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:39:14.110 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:39:14 smithi115 conmon[90393]: 2022-05-14T04:39:14.068+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:39:14.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:39:13 smithi115 ceph-mon[78593]: pgmap v9694: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:39:14.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:39:13 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:39:14.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:39:13 smithi047 ceph-mon[77147]: pgmap v9694: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:39:14.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:39:13 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:39:15.102 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:39:14 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:39:15.103 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:39:14 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:39:15.103 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:39:14 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:39:15.104 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:39:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T04:39:15.104 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:39:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T04:39:15.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:39:14 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:39:15.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:39:14 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:39:15.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:39:14 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:39:15.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:39:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T04:39:15.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:39:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T04:39:15.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:39:15 smithi115 conmon[90393]: 2022-05-14T04:39:15.099+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:39:16.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:39:15 smithi115 ceph-mon[78593]: pgmap v9695: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:39:16.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:39:15 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:39:16.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:39:15 smithi047 ceph-mon[77147]: pgmap v9695: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:39:16.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:39:15 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:39:16.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:39:16 smithi115 conmon[90393]: 2022-05-14T04:39:16.104+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:39:17.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:39:16 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:39:17.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:39:16 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:39:17.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:39:17 smithi115 conmon[90393]: 2022-05-14T04:39:17.109+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:39:18.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:39:17 smithi115 ceph-mon[78593]: pgmap v9696: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:39:18.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:39:17 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:39:18.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:39:17 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 18690 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:39:18.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:39:17 smithi047 ceph-mon[77147]: pgmap v9696: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:39:18.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:39:17 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:39:18.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:39:17 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 18690 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:39:18.610 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:39:18 smithi115 conmon[90393]: 2022-05-14T04:39:18.139+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:39:19.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:39:18 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:39:19.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:39:18 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:39:19.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:39:18 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:39:19.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:39:18 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:39:19.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:39:18 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:39:19.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:39:18 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:39:19.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:39:19 smithi115 conmon[90393]: 2022-05-14T04:39:19.177+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:39:20.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:39:19 smithi115 ceph-mon[78593]: pgmap v9697: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:39:20.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:39:19 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:39:20.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:39:19 smithi047 ceph-mon[77147]: pgmap v9697: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:39:20.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:39:19 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:39:20.610 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:39:20 smithi115 conmon[90393]: 2022-05-14T04:39:20.175+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:39:21.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:39:20 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:39:21.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:39:20 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:39:21.610 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:39:21 smithi115 conmon[90393]: 2022-05-14T04:39:21.146+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:39:22.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:39:21 smithi115 ceph-mon[78593]: pgmap v9698: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:39:22.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:39:21 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:39:22.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:39:21 smithi047 ceph-mon[77147]: pgmap v9698: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:39:22.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:39:21 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:39:22.610 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:39:22 smithi115 conmon[90393]: 2022-05-14T04:39:22.180+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:39:23.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:39:22 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:39:23.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:39:22 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:39:23.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:39:22 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:39:23.112 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:39:22 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 18695 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:39:23.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:39:22 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:39:23.295 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:39:22 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:39:23.295 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:39:22 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:39:23.295 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:39:22 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 18695 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:39:23.613 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:39:23 smithi115 conmon[90393]: 2022-05-14T04:39:23.227+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:39:24.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:39:23 smithi115 ceph-mon[78593]: pgmap v9699: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:39:24.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:39:23 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:39:24.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:39:23 smithi047 ceph-mon[77147]: pgmap v9699: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:39:24.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:39:23 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:39:24.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:39:24 smithi115 conmon[90393]: 2022-05-14T04:39:24.181+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:39:25.117 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:39:24 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:39:25.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:39:24 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:39:25.616 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:39:25 smithi115 conmon[90393]: 2022-05-14T04:39:25.198+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:39:26.182 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:39:25 smithi115 ceph-mon[78593]: pgmap v9700: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:39:26.182 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:39:25 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:39:26.296 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:39:25 smithi047 ceph-mon[77147]: pgmap v9700: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:39:26.296 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:39:25 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:39:26.610 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:39:26 smithi115 conmon[90393]: 2022-05-14T04:39:26.179+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:39:27.162 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:39:26 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:39:27.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:39:26 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:39:27.612 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:39:27 smithi115 conmon[90393]: 2022-05-14T04:39:27.160+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:39:28.184 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:39:27 smithi115 ceph-mon[78593]: pgmap v9701: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:39:28.184 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:39:27 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:39:28.184 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:39:27 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 18700 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:39:28.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:39:27 smithi047 ceph-mon[77147]: pgmap v9701: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:39:28.301 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:39:27 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:39:28.302 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:39:27 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 18700 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:39:28.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:39:28 smithi115 conmon[90393]: 2022-05-14T04:39:28.181+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:39:29.220 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:39:28 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:39:29.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:39:28 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:39:29.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:39:29 smithi115 conmon[90393]: 2022-05-14T04:39:29.217+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:39:30.224 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:39:29 smithi115 ceph-mon[78593]: pgmap v9702: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:39:30.224 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:39:29 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:39:30.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:39:29 smithi047 ceph-mon[77147]: pgmap v9702: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:39:30.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:39:29 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:39:30.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:39:30 smithi115 conmon[90393]: 2022-05-14T04:39:30.221+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:39:31.194 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:39:30 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:39:31.195 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:39:30 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:39:31.195 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:39:30 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:39:31.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:39:30 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:39:31.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:39:30 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:39:31.295 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:39:30 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:39:31.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:39:31 smithi115 conmon[90393]: 2022-05-14T04:39:31.192+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:39:32.208 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:39:31 smithi115 ceph-mon[78593]: pgmap v9703: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:39:32.208 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:39:31 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:39:32.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:39:31 smithi047 ceph-mon[77147]: pgmap v9703: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:39:32.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:39:31 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:39:32.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:39:32 smithi115 conmon[90393]: 2022-05-14T04:39:32.206+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:39:33.189 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:39:32 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:39:33.189 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:39:32 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 18705 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:39:33.295 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:39:32 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:39:33.295 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:39:32 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 18705 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:39:33.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:39:33 smithi115 conmon[90393]: 2022-05-14T04:39:33.186+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:39:34.140 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:39:33 smithi115 ceph-mon[78593]: pgmap v9704: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:39:34.140 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:39:33 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:39:34.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:39:33 smithi047 ceph-mon[77147]: pgmap v9704: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:39:34.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:39:33 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:39:34.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:39:34 smithi115 conmon[90393]: 2022-05-14T04:39:34.138+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:39:35.163 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:39:34 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:39:35.163 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:39:34 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:39:35.163 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:39:34 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:39:35.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:39:34 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:39:35.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:39:34 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:39:35.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:39:34 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:39:35.610 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:39:35 smithi115 conmon[90393]: 2022-05-14T04:39:35.158+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:39:36.148 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:39:35 smithi115 ceph-mon[78593]: pgmap v9705: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:39:36.149 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:39:35 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:39:36.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:39:35 smithi047 ceph-mon[77147]: pgmap v9705: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:39:36.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:39:35 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:39:36.610 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:39:36 smithi115 conmon[90393]: 2022-05-14T04:39:36.145+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:39:37.168 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:39:36 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:39:37.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:39:36 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:39:37.612 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:39:37 smithi115 conmon[90393]: 2022-05-14T04:39:37.164+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:39:38.161 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:39:37 smithi115 ceph-mon[78593]: pgmap v9706: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:39:38.161 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:39:37 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:39:38.162 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:39:37 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 18711 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:39:38.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:39:37 smithi047 ceph-mon[77147]: pgmap v9706: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:39:38.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:39:37 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:39:38.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:39:37 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 18711 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:39:38.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:39:38 smithi115 conmon[90393]: 2022-05-14T04:39:38.158+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:39:39.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:39:38 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:39:39.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:39:38 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:39:39.295 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:39:38 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:39:39.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:39:39 smithi115 conmon[90393]: 2022-05-14T04:39:39.145+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:39:39.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:39:38 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:39:39.366 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:39:38 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:39:39.366 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:39:38 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:39:40.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:39:39 smithi047 ceph-mon[77147]: pgmap v9707: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:39:40.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:39:39 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:39:40.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:39:39 smithi115 ceph-mon[78593]: pgmap v9707: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:39:40.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:39:39 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:39:40.361 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:39:40 smithi115 conmon[90393]: 2022-05-14T04:39:40.096+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:39:41.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:39:40 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:39:41.362 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:39:40 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:39:41.362 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:39:41 smithi115 conmon[90393]: 2022-05-14T04:39:41.071+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:39:42.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:39:41 smithi047 ceph-mon[77147]: pgmap v9708: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:39:42.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:39:41 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:39:42.361 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:39:42 smithi115 conmon[90393]: 2022-05-14T04:39:42.035+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:39:42.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:39:41 smithi115 ceph-mon[78593]: pgmap v9708: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:39:42.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:39:41 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:39:43.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:39:42 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:39:43.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:39:42 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:39:43.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:39:42 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:39:43.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:39:42 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 18716 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:39:43.360 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:39:42 smithi115 conmon[90393]: 2022-05-14T04:39:42.995+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:39:43.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:39:42 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:39:43.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:39:42 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:39:43.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:39:42 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:39:43.362 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:39:42 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 18716 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:39:44.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:39:43 smithi047 ceph-mon[77147]: pgmap v9709: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:39:44.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:39:43 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:39:44.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:39:44 smithi115 conmon[90393]: 2022-05-14T04:39:44.006+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:39:44.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:39:43 smithi115 ceph-mon[78593]: pgmap v9709: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:39:44.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:39:43 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:39:45.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:39:44 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:39:45.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:39:44 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:39:45.362 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:39:45 smithi115 conmon[90393]: 2022-05-14T04:39:45.041+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:39:46.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:39:45 smithi047 ceph-mon[77147]: pgmap v9710: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:39:46.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:39:45 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:39:46.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:39:45 smithi115 conmon[90393]: 2022-05-14T04:39:45.997+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:39:46.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:39:45 smithi115 ceph-mon[78593]: pgmap v9710: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:39:46.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:39:45 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:39:47.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:39:46 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:39:47.366 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:39:46 smithi115 conmon[90393]: 2022-05-14T04:39:46.982+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:39:47.366 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:39:46 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:39:48.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:39:47 smithi047 ceph-mon[77147]: pgmap v9711: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:39:48.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:39:47 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:39:48.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:39:47 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 18720 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:39:48.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:39:48 smithi115 conmon[90393]: 2022-05-14T04:39:48.017+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:39:48.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:39:47 smithi115 ceph-mon[78593]: pgmap v9711: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:39:48.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:39:47 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:39:48.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:39:47 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 18720 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:39:49.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:39:48 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:39:49.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:39:49 smithi115 conmon[90393]: 2022-05-14T04:39:49.020+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:39:49.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:39:48 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:39:50.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:39:49 smithi047 ceph-mon[77147]: pgmap v9712: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:39:50.297 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:39:49 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:39:50.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:39:50 smithi115 conmon[90393]: 2022-05-14T04:39:50.065+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:39:50.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:39:49 smithi115 ceph-mon[78593]: pgmap v9712: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:39:50.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:39:49 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:39:51.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:39:50 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:39:51.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:39:50 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:39:51.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:39:50 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:39:51.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:39:51 smithi115 conmon[90393]: 2022-05-14T04:39:51.039+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:39:51.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:39:50 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:39:51.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:39:50 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:39:51.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:39:50 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:39:52.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:39:51 smithi047 ceph-mon[77147]: pgmap v9713: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:39:52.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:39:51 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:39:52.360 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:39:52 smithi115 conmon[90393]: 2022-05-14T04:39:52.037+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:39:52.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:39:51 smithi115 ceph-mon[78593]: pgmap v9713: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:39:52.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:39:51 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:39:53.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:39:52 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:39:53.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:39:52 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 18725 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:39:53.360 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:39:53 smithi115 conmon[90393]: 2022-05-14T04:39:53.009+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:39:53.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:39:52 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:39:53.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:39:52 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 18725 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:39:54.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:39:53 smithi047 ceph-mon[77147]: pgmap v9714: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:39:54.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:39:53 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:39:54.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:39:54 smithi115 conmon[90393]: 2022-05-14T04:39:54.014+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:39:54.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:39:53 smithi115 ceph-mon[78593]: pgmap v9714: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:39:54.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:39:53 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:39:55.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:39:54 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:39:55.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:39:54 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:39:55.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:39:54 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:39:55.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:39:55 smithi115 conmon[90393]: 2022-05-14T04:39:55.035+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:39:55.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:39:54 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:39:55.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:39:54 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:39:55.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:39:54 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:39:56.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:39:56 smithi047 ceph-mon[77147]: pgmap v9715: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:39:56.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:39:56 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:39:56.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:39:56 smithi115 conmon[90393]: 2022-05-14T04:39:56.058+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:39:56.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:39:56 smithi115 ceph-mon[78593]: pgmap v9715: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:39:56.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:39:56 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:39:57.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:39:57 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:39:57.360 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:39:57 smithi115 conmon[90393]: 2022-05-14T04:39:57.078+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:39:57.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:39:57 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:39:58.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:39:58 smithi047 ceph-mon[77147]: pgmap v9716: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:39:58.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:39:58 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:39:58.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:39:58 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 18730 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:39:58.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:39:58 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T04:39:58.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:39:58 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T04:39:58.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:39:58 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T04:39:58.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:39:58 smithi115 conmon[90393]: 2022-05-14T04:39:58.093+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:39:58.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:39:58 smithi115 ceph-mon[78593]: pgmap v9716: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:39:58.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:39:58 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:39:58.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:39:58 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 18730 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:39:58.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:39:58 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T04:39:58.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:39:58 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T04:39:58.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:39:58 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T04:39:59.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:39:59 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:39:59.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:39:59 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:39:59.295 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:39:59 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:39:59.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:39:59 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:39:59.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:39:59 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:39:59.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:39:59 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:39:59.361 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:39:59 smithi115 conmon[90393]: 2022-05-14T04:39:59.135+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:40:00.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:40:00 smithi047 conmon[77123]: 2022-05-14T04:39:59.999+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : Health detail: HEALTH_ERR 1 filesystem is degraded; 1 mds daemon damaged; 1 MDSs are read only; 1 MDSs report slow metadata IOs; 1/3 mons down, quorum b,c; Reduced data availability: 1 pg inactive; 9 slow ops, oldest one blocked for 18730 sec, osd.5 has slow ops 2022-05-14T04:40:00.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:40:00 smithi047 conmon[77123]: 2022-05-14T04:39:59.999+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : [WRN] FS_DEGRADED: 1 filesystem is degraded 2022-05-14T04:40:00.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:40:00 smithi047 conmon[77123]: 2022-05-14T04:39:59.999+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : fs cephfs is degraded 2022-05-14T04:40:00.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:40:00 smithi047 conmon[77123]: 2022-05-14T04:39:59.999+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : [ERR] MDS_DAMAGE: 1 mds daemon damaged 2022-05-14T04:40:00.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:40:00 smithi047 conmon[77123]: 2022-05-14T04:39:59.999+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : fs cephfs mds.2 is damaged 2022-05-14T04:40:00.295 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:40:00 smithi047 conmon[77123]: 2022-05-14T04:39:59.999+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : [WRN] MDS_READ_ONLY: 1 MDSs are read only 2022-05-14T04:40:00.295 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:40:00 smithi047 conmon[77123]: 2022-05-14T04:39:59.999+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : mds.l(mds.0): MDS in read-only mode 2022-05-14T04:40:00.295 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:40:00 smithi047 conmon[77123]: 2022-05-14T04:39:59.999+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : [WRN] MDS_SLOW_METADATA_IO: 1 MDSs report slow metadata IOs 2022-05-14T04:40:00.295 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:40:00 smithi047 conmon[77123]: 2022-05-14T04:39:59.999+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : mds.l(mds.0): 4 slow metadata IOs are blocked > 180 secs, oldest blocked for 19033 secs 2022-05-14T04:40:00.295 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:40:00 smithi047 conmon[77123]: 2022-05-14T04:40:00.296 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:40:00 smithi047 conmon[77123]: 2022-05-14T04:39:59.999+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : [WRN] MON_DOWN: 1/3 mons down, quorum b,c 2022-05-14T04:40:00.296 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:40:00 smithi047 conmon[77123]: 2022-05-14T04:40:00.296 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:40:00 smithi047 conmon[77123]: 2022-05-14T04:39:59.999+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : mon.a (rank 0) addr [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] is down (out of quorum) 2022-05-14T04:40:00.296 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:40:00 smithi047 conmon[77123]: 2022-05-14T04:39:59.999+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : [WRN] PG_AVAILABILITY: Reduced data availability: 1 pg inactive 2022-05-14T04:40:00.297 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:40:00 smithi047 conmon[77123]: 2022-05-14T04:39:59.999+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : pg 2.1 is stuck inactive for 5h, current state unknown, last acting [] 2022-05-14T04:40:00.297 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:40:00 smithi047 conmon[77123]: 2022-05-14T04:39:59.999+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : [WRN] SLOW_OPS: 9 slow ops, oldest one blocked for 18730 sec, osd.5 has slow ops 2022-05-14T04:40:00.297 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:40:00 smithi047 ceph-mon[77147]: pgmap v9717: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:40:00.297 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:40:00 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:40:00.297 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:40:00 smithi047 ceph-mon[77147]: Health detail: HEALTH_ERR 1 filesystem is degraded; 1 mds daemon damaged; 1 MDSs are read only; 1 MDSs report slow metadata IOs; 1/3 mons down, quorum b,c; Reduced data availability: 1 pg inactive; 9 slow ops, oldest one blocked for 18730 sec, osd.5 has slow ops 2022-05-14T04:40:00.298 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:40:00 smithi047 ceph-mon[77147]: [WRN] FS_DEGRADED: 1 filesystem is degraded 2022-05-14T04:40:00.298 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:40:00 smithi047 ceph-mon[77147]: fs cephfs is degraded 2022-05-14T04:40:00.298 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:40:00 smithi047 ceph-mon[77147]: [ERR] MDS_DAMAGE: 1 mds daemon damaged 2022-05-14T04:40:00.298 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:40:00 smithi047 ceph-mon[77147]: fs cephfs mds.2 is damaged 2022-05-14T04:40:00.299 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:40:00 smithi047 ceph-mon[77147]: [WRN] MDS_READ_ONLY: 1 MDSs are read only 2022-05-14T04:40:00.299 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:40:00 smithi047 ceph-mon[77147]: mds.l(mds.0): MDS in read-only mode 2022-05-14T04:40:00.299 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:40:00 smithi047 ceph-mon[77147]: [WRN] MDS_SLOW_METADATA_IO: 1 MDSs report slow metadata IOs 2022-05-14T04:40:00.299 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:40:00 smithi047 ceph-mon[77147]: mds.l(mds.0): 4 slow metadata IOs are blocked > 180 secs, oldest blocked for 19033 secs 2022-05-14T04:40:00.300 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:40:00 smithi047 ceph-mon[77147]: [WRN] MON_DOWN: 1/3 mons down, quorum b,c 2022-05-14T04:40:00.300 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:40:00 smithi047 ceph-mon[77147]: mon.a (rank 0) addr [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] is down (out of quorum) 2022-05-14T04:40:00.300 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:40:00 smithi047 ceph-mon[77147]: [WRN] PG_AVAILABILITY: Reduced data availability: 1 pg inactive 2022-05-14T04:40:00.300 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:40:00 smithi047 ceph-mon[77147]: pg 2.1 is stuck inactive for 5h, current state unknown, last acting [] 2022-05-14T04:40:00.300 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:40:00 smithi047 ceph-mon[77147]: [WRN] SLOW_OPS: 9 slow ops, oldest one blocked for 18730 sec, osd.5 has slow ops 2022-05-14T04:40:00.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:40:00 smithi115 conmon[90393]: 2022-05-14T04:40:00.111+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:40:00.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:40:00 smithi115 ceph-mon[78593]: pgmap v9717: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:40:00.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:40:00 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:40:00.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:40:00 smithi115 ceph-mon[78593]: Health detail: HEALTH_ERR 1 filesystem is degraded; 1 mds daemon damaged; 1 MDSs are read only; 1 MDSs report slow metadata IOs; 1/3 mons down, quorum b,c; Reduced data availability: 1 pg inactive; 9 slow ops, oldest one blocked for 18730 sec, osd.5 has slow ops 2022-05-14T04:40:00.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:40:00 smithi115 ceph-mon[78593]: [WRN] FS_DEGRADED: 1 filesystem is degraded 2022-05-14T04:40:00.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:40:00 smithi115 ceph-mon[78593]: fs cephfs is degraded 2022-05-14T04:40:00.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:40:00 smithi115 ceph-mon[78593]: [ERR] MDS_DAMAGE: 1 mds daemon damaged 2022-05-14T04:40:00.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:40:00 smithi115 ceph-mon[78593]: fs cephfs mds.2 is damaged 2022-05-14T04:40:00.362 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:40:00 smithi115 ceph-mon[78593]: [WRN] MDS_READ_ONLY: 1 MDSs are read only 2022-05-14T04:40:00.362 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:40:00 smithi115 ceph-mon[78593]: mds.l(mds.0): MDS in read-only mode 2022-05-14T04:40:00.362 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:40:00 smithi115 ceph-mon[78593]: [WRN] MDS_SLOW_METADATA_IO: 1 MDSs report slow metadata IOs 2022-05-14T04:40:00.362 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:40:00 smithi115 ceph-mon[78593]: mds.l(mds.0): 4 slow metadata IOs are blocked > 180 secs, oldest blocked for 19033 secs 2022-05-14T04:40:00.362 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:40:00 smithi115 ceph-mon[78593]: [WRN] MON_DOWN: 1/3 mons down, quorum b,c 2022-05-14T04:40:00.363 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:40:00 smithi115 ceph-mon[78593]: mon.a (rank 0) addr [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] is down (out of quorum) 2022-05-14T04:40:00.363 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:40:00 smithi115 ceph-mon[78593]: [WRN] PG_AVAILABILITY: Reduced data availability: 1 pg inactive 2022-05-14T04:40:00.363 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:40:00 smithi115 ceph-mon[78593]: pg 2.1 is stuck inactive for 5h, current state unknown, last acting [] 2022-05-14T04:40:00.363 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:40:00 smithi115 ceph-mon[78593]: [WRN] SLOW_OPS: 9 slow ops, oldest one blocked for 18730 sec, osd.5 has slow ops 2022-05-14T04:40:01.296 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:40:01 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:40:01.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:40:00 smithi047 conmon[78521]: 2022-05-14T04:40:00.943+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:40:01.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:40:00 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T04:40:01.297 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:40:00 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:40:01.297 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:40:00 smithi047 conmon[78521]: yield 2022-05-14T04:40:01.297 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:40:00 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:40:01.297 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:40:00 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:40:01.298 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:40:00 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:40:01.298 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:40:00 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T04:40:01.298 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:40:00 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:40:01.298 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:40:00 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T04:40:01.298 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:40:00 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:40:01.299 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:40:00 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T04:40:01.299 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:40:00 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:40:01.299 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:40:00 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T04:40:01.299 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:40:00 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:40:01.300 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:40:00 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T04:40:01.300 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:40:00 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:40:01.300 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:40:00 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:40:01.300 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:40:00 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:40:01.360 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:40:01 smithi115 conmon[90393]: 2022-05-14T04:40:01.147+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:40:01.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:40:01 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:40:02.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:40:02 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:40:02.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T04:40:02.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:40:02.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T04:40:02.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:40:02.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:40:02.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:40:02.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T04:40:02.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:40:02.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T04:40:02.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:40:02.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T04:40:02.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:40:02.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T04:40:02.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:40:02.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T04:40:02.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:40:02.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:40:02.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:40:02.298 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:40:02 smithi047 ceph-mon[77147]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T04:40:02.298 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:40:02 smithi047 ceph-mon[77147]: pgmap v9718: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:40:02.298 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:40:02 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:40:02.360 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:40:02 smithi115 conmon[90393]: 2022-05-14T04:40:02.146+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:40:02.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:40:02 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:40:02.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T04:40:02.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:40:02.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T04:40:02.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:40:02.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:40:02.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:40:02.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T04:40:02.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:40:02.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T04:40:02.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:40:02.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T04:40:02.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:40:02.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T04:40:02.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:40:02.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T04:40:02.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:40:02.365 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:40:02.365 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:40:02.365 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:40:02 smithi115 ceph-mon[78593]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T04:40:02.366 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:40:02 smithi115 ceph-mon[78593]: pgmap v9718: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:40:02.366 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:40:02 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:40:03.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:40:03 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:40:03.295 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:40:03 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:40:03.295 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:40:03 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:40:03.296 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:40:03 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 18736 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:40:03.367 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:40:03 smithi115 conmon[90393]: 2022-05-14T04:40:03.121+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:40:03.368 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:40:03 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:40:03.368 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:40:03 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:40:03.369 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:40:03 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:40:03.369 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:40:03 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 18736 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:40:04.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:40:04 smithi047 ceph-mon[77147]: pgmap v9719: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:40:04.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:40:04 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:40:04.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:40:04 smithi047 conmon[78521]: 2022-05-14T04:40:04.015+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:40:04.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:40:04 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T04:40:04.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:40:04 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:40:04.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:40:04 smithi047 conmon[78521]: yield 2022-05-14T04:40:04.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:40:04 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:40:04.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:40:04 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:40:04.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:40:04 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:40:04.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:40:04 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T04:40:04.297 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:40:04 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:40:04.299 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:40:04 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T04:40:04.299 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:40:04 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:40:04.299 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:40:04 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T04:40:04.299 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:40:04 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:40:04.299 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:40:04 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T04:40:04.300 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:40:04 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:40:04.300 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:40:04 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T04:40:04.300 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:40:04 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:40:04.300 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:40:04 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:40:04.300 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:40:04 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:40:04.301 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:40:04 smithi047 conmon[78521]: 2022-05-14T04:40:04.016+0000 7f533ef76700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T04:40:04.301 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:40:04 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T04:40:04.301 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:40:04 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:40:04.301 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:40:04 smithi047 conmon[78521]: yield 2022-05-14T04:40:04.302 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:40:04 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:40:04.302 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:40:04 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:40:04.302 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:40:04 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:40:04.302 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:40:04 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T04:40:04.306 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:40:04 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:40:04.306 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:40:04 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T04:40:04.306 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:40:04 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:40:04.306 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:40:04 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T04:40:04.307 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:40:04 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:40:04.307 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:40:04 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T04:40:04.307 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:40:04 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:40:04.307 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:40:04 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T04:40:04.307 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:40:04 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:40:04.308 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:40:04 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:40:04.308 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:40:04 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:40:04.308 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:40:04 smithi047 conmon[78521]: 2022-05-14T04:40:04.309 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:40:04 smithi047 conmon[78521]: During handling of the above exception, another exception occurred: 2022-05-14T04:40:04.309 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:40:04 smithi047 conmon[78521]: 2022-05-14T04:40:04.309 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:40:04 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T04:40:04.309 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:40:04 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T04:40:04.310 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:40:04 smithi047 conmon[78521]: return f(*arg) 2022-05-14T04:40:04.310 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:40:04 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T04:40:04.310 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:40:04 smithi047 conmon[78521]: self._write_client_files(client_files, host) 2022-05-14T04:40:04.310 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:40:04 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T04:40:04.310 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:40:04 smithi047 conmon[78521]: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T04:40:04.311 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:40:04 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T04:40:04.311 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:40:04 smithi047 conmon[78521]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T04:40:04.311 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:40:04 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T04:40:04.311 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:40:04 smithi047 conmon[78521]: return self.event_loop.get_result(coro) 2022-05-14T04:40:04.311 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:40:04 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T04:40:04.312 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:40:04 smithi047 conmon[78521]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T04:40:04.312 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:40:04 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T04:40:04.312 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:40:04 smithi047 conmon[78521]: return self.__get_result() 2022-05-14T04:40:04.312 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:40:04 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T04:40:04.313 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:40:04 smithi047 conmon[78521]: raise self._exception 2022-05-14T04:40:04.313 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:40:04 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T04:40:04.315 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:40:04 smithi047 conmon[78521]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T04:40:04.315 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:40:04 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T04:40:04.316 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:40:04 smithi047 conmon[78521]: conn = await self._remote_connection(host, addr) 2022-05-14T04:40:04.316 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:40:04 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T04:40:04.316 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:40:04 smithi047 conmon[78521]: raise 2022-05-14T04:40:04.316 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:40:04 smithi047 conmon[78521]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T04:40:04.317 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:40:04 smithi047 conmon[78521]: self.gen.throw(type, value, traceback) 2022-05-14T04:40:04.317 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:40:04 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T04:40:04.317 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:40:04 smithi047 conmon[78521]: raise OrchestratorError(msg) 2022-05-14T04:40:04.317 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:40:04 smithi047 conmon[78521]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:40:04.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:40:04 smithi115 conmon[90393]: 2022-05-14T04:40:04.115+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:40:04.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:40:04 smithi115 ceph-mon[78593]: pgmap v9719: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:40:04.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:40:04 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:40:05.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:40:05 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:40:05.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T04:40:05.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:40:05.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T04:40:05.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:40:05.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:40:05.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:40:05.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T04:40:05.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:40:05.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T04:40:05.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:40:05.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T04:40:05.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:40:05.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T04:40:05.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:40:05.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T04:40:05.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:40:05.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:40:05.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:40:05.297 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:40:05 smithi047 ceph-mon[77147]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T04:40:05.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T04:40:05.298 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:40:05.298 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T04:40:05.298 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:40:05.298 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:40:05.299 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:40:05.299 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T04:40:05.299 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:40:05.299 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T04:40:05.299 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:40:05.300 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T04:40:05.300 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:40:05.300 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T04:40:05.300 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:40:05.301 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T04:40:05.301 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:40:05.302 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:40:05.303 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:40:05.303 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T04:40:05.303 INFO:journalctl@ceph.mon.b.smithi047.stdout: During handling of the above exception, another exception occurred: 2022-05-14T04:40:05.303 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T04:40:05.304 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T04:40:05.304 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T04:40:05.304 INFO:journalctl@ceph.mon.b.smithi047.stdout: return f(*arg) 2022-05-14T04:40:05.304 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T04:40:05.305 INFO:journalctl@ceph.mon.b.smithi047.stdout: self._write_client_files(client_files, host) 2022-05-14T04:40:05.305 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T04:40:05.305 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T04:40:05.305 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T04:40:05.305 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T04:40:05.306 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T04:40:05.306 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.event_loop.get_result(coro) 2022-05-14T04:40:05.306 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T04:40:05.306 INFO:journalctl@ceph.mon.b.smithi047.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T04:40:05.307 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T04:40:05.307 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.__get_result() 2022-05-14T04:40:05.307 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T04:40:05.307 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise self._exception 2022-05-14T04:40:05.307 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T04:40:05.308 INFO:journalctl@ceph.mon.b.smithi047.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T04:40:05.308 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T04:40:05.308 INFO:journalctl@ceph.mon.b.smithi047.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T04:40:05.308 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T04:40:05.308 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise 2022-05-14T04:40:05.309 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T04:40:05.309 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.gen.throw(type, value, traceback) 2022-05-14T04:40:05.310 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T04:40:05.310 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OrchestratorError(msg) 2022-05-14T04:40:05.310 INFO:journalctl@ceph.mon.b.smithi047.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:40:05.310 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:40:05 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:40:05.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:40:05 smithi115 conmon[90393]: 2022-05-14T04:40:05.079+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:40:05.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:40:05 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:40:05.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T04:40:05.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:40:05.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T04:40:05.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:40:05.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:40:05.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:40:05.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T04:40:05.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:40:05.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T04:40:05.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:40:05.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T04:40:05.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:40:05.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T04:40:05.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:40:05.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T04:40:05.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:40:05.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:40:05.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:40:05.364 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:40:05 smithi115 ceph-mon[78593]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T04:40:05.365 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T04:40:05.365 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:40:05.365 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T04:40:05.365 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:40:05.366 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:40:05.366 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:40:05.366 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T04:40:05.366 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:40:05.366 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T04:40:05.367 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:40:05.367 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T04:40:05.367 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:40:05.367 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T04:40:05.368 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:40:05.368 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T04:40:05.368 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:40:05.368 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:40:05.368 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:40:05.369 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T04:40:05.369 INFO:journalctl@ceph.mon.c.smithi115.stdout: During handling of the above exception, another exception occurred: 2022-05-14T04:40:05.370 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T04:40:05.370 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T04:40:05.370 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T04:40:05.371 INFO:journalctl@ceph.mon.c.smithi115.stdout: return f(*arg) 2022-05-14T04:40:05.371 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T04:40:05.371 INFO:journalctl@ceph.mon.c.smithi115.stdout: self._write_client_files(client_files, host) 2022-05-14T04:40:05.371 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T04:40:05.371 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T04:40:05.372 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T04:40:05.372 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T04:40:05.372 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T04:40:05.372 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.event_loop.get_result(coro) 2022-05-14T04:40:05.372 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T04:40:05.373 INFO:journalctl@ceph.mon.c.smithi115.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T04:40:05.373 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T04:40:05.373 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.__get_result() 2022-05-14T04:40:05.373 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T04:40:05.374 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise self._exception 2022-05-14T04:40:05.374 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T04:40:05.374 INFO:journalctl@ceph.mon.c.smithi115.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T04:40:05.374 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T04:40:05.374 INFO:journalctl@ceph.mon.c.smithi115.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T04:40:05.375 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T04:40:05.375 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise 2022-05-14T04:40:05.375 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T04:40:05.375 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.gen.throw(type, value, traceback) 2022-05-14T04:40:05.375 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T04:40:05.376 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OrchestratorError(msg) 2022-05-14T04:40:05.376 INFO:journalctl@ceph.mon.c.smithi115.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:40:05.376 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:40:05 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:40:06.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:40:06 smithi115 conmon[90393]: 2022-05-14T04:40:06.105+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:40:06.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:40:06 smithi115 ceph-mon[78593]: pgmap v9720: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:40:06.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:40:06 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:40:06.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:40:06 smithi047 ceph-mon[77147]: pgmap v9720: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:40:06.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:40:06 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:40:07.365 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:40:07 smithi115 conmon[90393]: 2022-05-14T04:40:07.121+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:40:07.366 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:40:07 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:40:07.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:40:07 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:40:08.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:40:08 smithi115 conmon[90393]: 2022-05-14T04:40:08.081+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:40:08.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:40:08 smithi115 ceph-mon[78593]: pgmap v9721: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:40:08.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:40:08 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:40:08.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:40:08 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 18741 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:40:08.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:40:08 smithi047 ceph-mon[77147]: pgmap v9721: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:40:08.545 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:40:08 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:40:08.545 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:40:08 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 18741 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:40:09.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:40:09 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:40:09.360 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:40:09 smithi115 conmon[90393]: 2022-05-14T04:40:09.129+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:40:09.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:40:09 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:40:10.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:40:10 smithi115 conmon[90393]: 2022-05-14T04:40:10.107+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:40:10.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:40:10 smithi115 ceph-mon[78593]: pgmap v9722: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:40:10.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:40:10 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:40:10.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:40:10 smithi047 ceph-mon[77147]: pgmap v9722: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:40:10.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:40:10 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:40:11.360 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:40:11 smithi115 conmon[90393]: 2022-05-14T04:40:11.096+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:40:11.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:40:11 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:40:11.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:40:11 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:40:11.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:40:11 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:40:11.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:40:11 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:40:11.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:40:11 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:40:11.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:40:11 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:40:12.360 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:40:12 smithi115 conmon[90393]: 2022-05-14T04:40:12.142+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:40:12.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:40:12 smithi115 ceph-mon[78593]: pgmap v9723: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:40:12.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:40:12 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:40:12.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:40:12 smithi047 ceph-mon[77147]: pgmap v9723: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:40:12.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:40:12 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:40:13.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:40:13 smithi115 conmon[90393]: 2022-05-14T04:40:13.137+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:40:13.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:40:13 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:40:13.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:40:13 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 18746 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:40:13.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:40:13 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:40:13.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:40:13 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 18746 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:40:14.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:40:14 smithi115 conmon[90393]: 2022-05-14T04:40:14.153+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:40:14.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:40:14 smithi115 ceph-mon[78593]: pgmap v9724: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:40:14.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:40:14 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:40:14.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:40:14 smithi047 ceph-mon[77147]: pgmap v9724: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:40:14.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:40:14 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:40:15.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:40:15 smithi115 conmon[90393]: 2022-05-14T04:40:15.189+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:40:15.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:40:15 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:40:15.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:40:15 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:40:15.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:40:15 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:40:15.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:40:15 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T04:40:15.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:40:15 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T04:40:15.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:40:15 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:40:15.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:40:15 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:40:15.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:40:15 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:40:15.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:40:15 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T04:40:15.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:40:15 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T04:40:16.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:40:16 smithi115 conmon[90393]: 2022-05-14T04:40:16.234+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:40:16.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:40:16 smithi115 ceph-mon[78593]: pgmap v9725: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:40:16.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:40:16 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:40:16.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:40:16 smithi047 ceph-mon[77147]: pgmap v9725: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:40:16.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:40:16 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:40:17.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:40:17 smithi115 conmon[90393]: 2022-05-14T04:40:17.233+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:40:17.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:40:17 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:40:17.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:40:17 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:40:18.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:40:18 smithi115 conmon[90393]: 2022-05-14T04:40:18.223+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:40:18.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:40:18 smithi115 ceph-mon[78593]: pgmap v9726: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:40:18.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:40:18 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:40:18.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:40:18 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 18751 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:40:18.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:40:18 smithi047 ceph-mon[77147]: pgmap v9726: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:40:18.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:40:18 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:40:18.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:40:18 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 18751 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:40:19.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:40:19 smithi115 conmon[90393]: 2022-05-14T04:40:19.272+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:40:19.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:40:19 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:40:19.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:40:19 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:40:19.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:40:19 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:40:19.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:40:19 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:40:19.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:40:19 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:40:19.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:40:19 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:40:20.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:40:20 smithi115 conmon[90393]: 2022-05-14T04:40:20.235+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:40:20.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:40:20 smithi115 ceph-mon[78593]: pgmap v9727: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:40:20.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:40:20 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:40:20.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:40:20 smithi047 ceph-mon[77147]: pgmap v9727: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:40:20.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:40:20 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:40:21.362 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:40:21 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:40:21.363 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:40:21 smithi115 conmon[90393]: 2022-05-14T04:40:21.192+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:40:21.545 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:40:21 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:40:22.545 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:40:22 smithi047 ceph-mon[77147]: pgmap v9728: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:40:22.546 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:40:22 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:40:22.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:40:22 smithi115 conmon[90393]: 2022-05-14T04:40:22.234+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:40:22.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:40:22 smithi115 ceph-mon[78593]: pgmap v9728: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:40:22.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:40:22 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:40:23.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:40:23 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:40:23.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:40:23 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:40:23.545 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:40:23 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:40:23.545 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:40:23 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 18755 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:40:23.615 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:40:23 smithi115 conmon[90393]: 2022-05-14T04:40:23.231+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:40:23.616 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:40:23 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:40:23.616 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:40:23 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:40:23.616 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:40:23 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:40:23.617 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:40:23 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 18755 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:40:24.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:40:24 smithi047 ceph-mon[77147]: pgmap v9729: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:40:24.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:40:24 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:40:24.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:40:24 smithi115 conmon[90393]: 2022-05-14T04:40:24.225+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:40:24.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:40:24 smithi115 ceph-mon[78593]: pgmap v9729: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:40:24.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:40:24 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:40:25.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:40:25 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:40:25.610 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:40:25 smithi115 conmon[90393]: 2022-05-14T04:40:25.265+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:40:25.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:40:25 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:40:26.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:40:26 smithi047 ceph-mon[77147]: pgmap v9730: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:40:26.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:40:26 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:40:26.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:40:26 smithi115 conmon[90393]: 2022-05-14T04:40:26.270+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:40:26.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:40:26 smithi115 ceph-mon[78593]: pgmap v9730: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:40:26.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:40:26 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:40:27.545 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:40:27 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:40:27.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:40:27 smithi115 conmon[90393]: 2022-05-14T04:40:27.228+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:40:27.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:40:27 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:40:28.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:40:28 smithi047 ceph-mon[77147]: pgmap v9731: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:40:28.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:40:28 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:40:28.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:40:28 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 18760 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:40:28.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:40:28 smithi115 conmon[90393]: 2022-05-14T04:40:28.249+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:40:28.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:40:28 smithi115 ceph-mon[78593]: pgmap v9731: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:40:28.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:40:28 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:40:28.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:40:28 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 18760 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:40:29.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:40:29 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:40:29.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:40:29 smithi115 conmon[90393]: 2022-05-14T04:40:29.287+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:40:29.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:40:29 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:40:30.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:40:30 smithi047 ceph-mon[77147]: pgmap v9732: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:40:30.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:40:30 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:40:30.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:40:30 smithi115 ceph-mon[78593]: pgmap v9732: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:40:30.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:40:30 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:40:30.610 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:40:30 smithi115 conmon[90393]: 2022-05-14T04:40:30.240+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:40:31.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:40:31 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:40:31.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:40:31 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:40:31.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:40:31 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:40:31.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:40:31 smithi115 conmon[90393]: 2022-05-14T04:40:31.272+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:40:31.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:40:31 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:40:31.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:40:31 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:40:31.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:40:31 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:40:32.546 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:40:32 smithi047 ceph-mon[77147]: pgmap v9733: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:40:32.546 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:40:32 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:40:32.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:40:32 smithi115 conmon[90393]: 2022-05-14T04:40:32.227+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:40:32.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:40:32 smithi115 ceph-mon[78593]: pgmap v9733: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:40:32.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:40:32 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:40:33.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:40:33 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:40:33.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:40:33 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 18765 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:40:33.610 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:40:33 smithi115 conmon[90393]: 2022-05-14T04:40:33.237+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:40:33.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:40:33 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:40:33.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:40:33 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 18765 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:40:34.545 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:40:34 smithi047 ceph-mon[77147]: pgmap v9734: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:40:34.546 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:40:34 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:40:34.610 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:40:34 smithi115 conmon[90393]: 2022-05-14T04:40:34.237+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:40:34.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:40:34 smithi115 ceph-mon[78593]: pgmap v9734: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:40:34.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:40:34 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:40:35.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:40:35 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:40:35.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:40:35 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:40:35.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:40:35 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:40:35.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:40:35 smithi115 conmon[90393]: 2022-05-14T04:40:35.274+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:40:35.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:40:35 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:40:35.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:40:35 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:40:35.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:40:35 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:40:36.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:40:36 smithi047 ceph-mon[77147]: pgmap v9735: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:40:36.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:40:36 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:40:36.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:40:36 smithi115 ceph-mon[78593]: pgmap v9735: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:40:36.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:40:36 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:40:36.612 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:40:36 smithi115 conmon[90393]: 2022-05-14T04:40:36.232+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:40:37.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:40:37 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:40:37.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:40:37 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:40:37.611 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:40:37 smithi115 conmon[90393]: 2022-05-14T04:40:37.272+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:40:38.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:40:38 smithi047 ceph-mon[77147]: pgmap v9736: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:40:38.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:40:38 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:40:38.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:40:38 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 18770 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:40:38.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:40:38 smithi115 ceph-mon[78593]: pgmap v9736: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:40:38.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:40:38 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:40:38.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:40:38 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 18770 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:40:38.611 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:40:38 smithi115 conmon[90393]: 2022-05-14T04:40:38.262+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:40:39.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:40:39 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:40:39.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:40:39 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:40:39.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:40:39 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:40:39.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:40:39 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:40:39.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:40:39 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:40:39.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:40:39 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:40:39.611 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:40:39 smithi115 conmon[90393]: 2022-05-14T04:40:39.216+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:40:40.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:40:40 smithi047 ceph-mon[77147]: pgmap v9737: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:40:40.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:40:40 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:40:40.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:40:40 smithi115 conmon[90393]: 2022-05-14T04:40:40.181+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:40:40.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:40:40 smithi115 ceph-mon[78593]: pgmap v9737: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:40:40.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:40:40 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:40:41.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:40:41 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:40:41.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:40:41 smithi115 conmon[90393]: 2022-05-14T04:40:41.148+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:40:41.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:40:41 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:40:42.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:40:42 smithi047 ceph-mon[77147]: pgmap v9738: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:40:42.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:40:42 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:40:42.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:40:42 smithi115 conmon[90393]: 2022-05-14T04:40:42.160+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:40:42.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:40:42 smithi115 ceph-mon[78593]: pgmap v9738: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:40:42.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:40:42 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:40:43.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:40:43 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:40:43.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:40:43 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:40:43.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:40:43 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:40:43.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:40:43 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 18775 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:40:43.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:40:43 smithi115 conmon[90393]: 2022-05-14T04:40:43.200+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:40:43.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:40:43 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:40:43.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:40:43 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:40:43.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:40:43 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:40:43.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:40:43 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 18775 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:40:44.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:40:44 smithi047 ceph-mon[77147]: pgmap v9739: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:40:44.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:40:44 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:40:44.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:40:44 smithi115 conmon[90393]: 2022-05-14T04:40:44.192+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:40:44.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:40:44 smithi115 ceph-mon[78593]: pgmap v9739: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:40:44.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:40:44 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:40:45.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:40:45 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:40:45.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:40:45 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:40:45.610 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:40:45 smithi115 conmon[90393]: 2022-05-14T04:40:45.204+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:40:46.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:40:46 smithi047 ceph-mon[77147]: pgmap v9740: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:40:46.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:40:46 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:40:46.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:40:46 smithi115 conmon[90393]: 2022-05-14T04:40:46.230+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:40:46.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:40:46 smithi115 ceph-mon[78593]: pgmap v9740: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:40:46.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:40:46 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:40:47.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:40:47 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:40:47.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:40:47 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:40:47.611 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:40:47 smithi115 conmon[90393]: 2022-05-14T04:40:47.278+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:40:48.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:40:48 smithi047 ceph-mon[77147]: pgmap v9741: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:40:48.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:40:48 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:40:48.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:40:48 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 18780 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:40:48.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:40:48 smithi115 conmon[90393]: 2022-05-14T04:40:48.241+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:40:48.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:40:48 smithi115 ceph-mon[78593]: pgmap v9741: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:40:48.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:40:48 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:40:48.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:40:48 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 18780 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:40:49.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:40:49 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:40:49.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:40:49 smithi115 conmon[90393]: 2022-05-14T04:40:49.282+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:40:49.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:40:49 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:40:50.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:40:50 smithi047 ceph-mon[77147]: pgmap v9742: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:40:50.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:40:50 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:40:50.611 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:40:50 smithi115 conmon[90393]: 2022-05-14T04:40:50.320+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:40:50.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:40:50 smithi115 ceph-mon[78593]: pgmap v9742: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:40:50.612 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:40:50 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:40:51.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:40:51 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:40:51.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:40:51 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:40:51.545 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:40:51 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:40:51.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:40:51 smithi115 conmon[90393]: 2022-05-14T04:40:51.344+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:40:51.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:40:51 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:40:51.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:40:51 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:40:51.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:40:51 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:40:52.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:40:52 smithi047 ceph-mon[77147]: pgmap v9743: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:40:52.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:40:52 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:40:52.610 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:40:52 smithi115 conmon[90393]: 2022-05-14T04:40:52.349+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:40:52.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:40:52 smithi115 ceph-mon[78593]: pgmap v9743: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:40:52.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:40:52 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:40:53.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:40:53 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:40:53.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:40:53 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 18785 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:40:53.612 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:40:53 smithi115 conmon[90393]: 2022-05-14T04:40:53.317+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:40:53.613 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:40:53 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:40:53.613 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:40:53 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 18785 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:40:54.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:40:54 smithi047 ceph-mon[77147]: pgmap v9744: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:40:54.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:40:54 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:40:54.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:40:54 smithi115 conmon[90393]: 2022-05-14T04:40:54.363+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:40:54.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:40:54 smithi115 ceph-mon[78593]: pgmap v9744: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:40:54.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:40:54 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:40:55.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:40:55 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:40:55.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:40:55 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:40:55.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:40:55 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:40:55.610 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:40:55 smithi115 conmon[90393]: 2022-05-14T04:40:55.390+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:40:55.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:40:55 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:40:55.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:40:55 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:40:55.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:40:55 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:40:56.546 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:40:56 smithi047 ceph-mon[77147]: pgmap v9745: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:40:56.546 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:40:56 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:40:56.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:40:56 smithi115 conmon[90393]: 2022-05-14T04:40:56.397+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:40:56.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:40:56 smithi115 ceph-mon[78593]: pgmap v9745: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:40:56.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:40:56 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:40:57.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:40:57 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:40:57.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:40:57 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:40:57.612 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:40:57 smithi115 conmon[90393]: 2022-05-14T04:40:57.442+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:40:58.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:40:58 smithi047 ceph-mon[77147]: pgmap v9746: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:40:58.545 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:40:58 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:40:58.545 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:40:58 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:40:58.545 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:40:58 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:40:58.545 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:40:58 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 18790 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:40:58.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:40:58 smithi115 conmon[90393]: 2022-05-14T04:40:58.446+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:40:58.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:40:58 smithi115 ceph-mon[78593]: pgmap v9746: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:40:58.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:40:58 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:40:58.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:40:58 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:40:58.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:40:58 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:40:58.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:40:58 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 18790 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:40:59.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:40:59 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:40:59.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:40:59 smithi115 conmon[90393]: 2022-05-14T04:40:59.423+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:40:59.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:40:59 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:41:00.545 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:41:00 smithi047 ceph-mon[77147]: pgmap v9747: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:41:00.546 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:41:00 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:41:00.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:41:00 smithi115 conmon[90393]: 2022-05-14T04:41:00.395+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:41:00.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:41:00 smithi115 ceph-mon[78593]: pgmap v9747: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:41:00.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:41:00 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:41:01.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:41:01 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:41:01.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:41:01 smithi115 conmon[90393]: 2022-05-14T04:41:01.372+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:41:01.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:41:01 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:41:02.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:41:02 smithi047 ceph-mon[77147]: pgmap v9748: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:41:02.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:41:02 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:41:02.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:41:02 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:41:02.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:41:02 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:41:02.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:41:02 smithi115 conmon[90393]: 2022-05-14T04:41:02.331+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:41:02.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:41:02 smithi115 ceph-mon[78593]: pgmap v9748: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:41:02.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:41:02 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:41:02.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:41:02 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:41:02.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:41:02 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:41:03.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:41:03 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:41:03.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:41:03 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 18795 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:41:03.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:41:03 smithi115 conmon[90393]: 2022-05-14T04:41:03.335+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:41:03.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:41:03 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:41:03.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:41:03 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 18795 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:41:04.334 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:41:04 smithi115 ceph-mon[78593]: pgmap v9749: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:41:04.517 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:41:04 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:41:04.518 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:41:04 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T04:41:04.518 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:41:04 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T04:41:04.519 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:41:04 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T04:41:04.520 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:41:04 smithi047 ceph-mon[77147]: pgmap v9749: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:41:04.521 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:41:04 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:41:04.521 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:41:04 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T04:41:04.521 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:41:04 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T04:41:04.523 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:41:04 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T04:41:04.612 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:41:04 smithi115 conmon[90393]: 2022-05-14T04:41:04.331+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:41:04.793 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:41:04 smithi047 conmon[78521]: 2022-05-14T04:41:04.497+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:41:04.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:41:04 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T04:41:04.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:41:04 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:41:04.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:41:04 smithi047 conmon[78521]: yield 2022-05-14T04:41:04.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:41:04 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:41:04.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:41:04 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:41:04.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:41:04 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:41:04.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:41:04 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T04:41:04.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:41:04 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:41:04.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:41:04 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T04:41:04.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:41:04 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:41:04.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:41:04 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T04:41:04.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:41:04 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:41:04.797 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:41:04 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T04:41:04.797 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:41:04 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:41:04.797 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:41:04 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T04:41:04.798 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:41:04 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:41:04.798 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:41:04 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:41:04.798 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:41:04 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:41:05.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:41:05 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:41:05.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:41:05 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:41:05.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T04:41:05.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:41:05.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T04:41:05.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:41:05.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:41:05.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:41:05.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T04:41:05.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:41:05.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T04:41:05.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:41:05.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T04:41:05.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:41:05.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T04:41:05.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:41:05.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T04:41:05.548 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:41:05.548 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:41:05.548 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:41:05.549 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:41:05 smithi047 ceph-mon[77147]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T04:41:05.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:41:05 smithi115 conmon[90393]: 2022-05-14T04:41:05.284+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:41:05.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:41:05 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:41:05.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:41:05 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:41:05.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T04:41:05.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:41:05.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T04:41:05.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:41:05.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:41:05.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:41:05.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T04:41:05.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:41:05.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T04:41:05.613 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:41:05.613 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T04:41:05.613 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:41:05.613 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T04:41:05.614 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:41:05.614 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T04:41:05.614 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:41:05.614 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:41:05.614 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:41:05.615 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:41:05 smithi115 ceph-mon[78593]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T04:41:06.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:41:06 smithi047 ceph-mon[77147]: pgmap v9750: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:41:06.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:41:06 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:41:06.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:41:06 smithi115 conmon[90393]: 2022-05-14T04:41:06.330+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:41:06.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:41:06 smithi115 ceph-mon[78593]: pgmap v9750: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:41:06.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:41:06 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:41:07.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:41:07 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:41:07.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:41:07 smithi115 conmon[90393]: 2022-05-14T04:41:07.284+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:41:07.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:41:07 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:41:08.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:41:07 smithi047 conmon[78521]: 2022-05-14T04:41:07.632+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:41:08.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:41:07 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T04:41:08.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:41:07 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:41:08.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:41:07 smithi047 conmon[78521]: yield 2022-05-14T04:41:08.048 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:41:07 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:41:08.048 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:41:07 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:41:08.048 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:41:07 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:41:08.048 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:41:07 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T04:41:08.048 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:41:07 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:41:08.049 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:41:07 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T04:41:08.049 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:41:07 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:41:08.049 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:41:07 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T04:41:08.049 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:41:07 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:41:08.050 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:41:07 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T04:41:08.052 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:41:07 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:41:08.052 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:41:07 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T04:41:08.052 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:41:07 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:41:08.053 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:41:07 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:41:08.053 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:41:07 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:41:08.054 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:41:07 smithi047 conmon[78521]: 2022-05-14T04:41:07.633+0000 7f5344781700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T04:41:08.054 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:41:07 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T04:41:08.055 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:41:07 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:41:08.055 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:41:07 smithi047 conmon[78521]: yield 2022-05-14T04:41:08.056 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:41:07 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:41:08.056 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:41:07 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:41:08.057 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:41:07 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:41:08.057 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:41:07 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T04:41:08.060 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:41:07 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:41:08.060 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:41:07 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T04:41:08.060 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:41:07 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:41:08.061 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:41:07 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T04:41:08.061 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:41:07 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:41:08.061 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:41:07 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T04:41:08.061 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:41:07 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:41:08.062 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:41:07 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T04:41:08.062 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:41:07 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:41:08.062 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:41:07 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:41:08.062 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:41:07 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:41:08.062 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:41:07 smithi047 conmon[78521]: 2022-05-14T04:41:08.063 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:41:07 smithi047 conmon[78521]: During handling of the above exception, another exception occurred: 2022-05-14T04:41:08.063 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:41:07 smithi047 conmon[78521]: 2022-05-14T04:41:08.063 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:41:07 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T04:41:08.063 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:41:07 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T04:41:08.064 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:41:07 smithi047 conmon[78521]: return f(*arg) 2022-05-14T04:41:08.064 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:41:07 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T04:41:08.064 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:41:07 smithi047 conmon[78521]: self._write_client_files(client_files, host) 2022-05-14T04:41:08.064 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:41:07 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T04:41:08.064 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:41:07 smithi047 conmon[78521]: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T04:41:08.065 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:41:07 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T04:41:08.065 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:41:07 smithi047 conmon[78521]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T04:41:08.065 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:41:07 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T04:41:08.065 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:41:07 smithi047 conmon[78521]: return self.event_loop.get_result(coro) 2022-05-14T04:41:08.065 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:41:07 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T04:41:08.066 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:41:07 smithi047 conmon[78521]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T04:41:08.066 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:41:07 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T04:41:08.066 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:41:07 smithi047 conmon[78521]: return self.__get_result() 2022-05-14T04:41:08.066 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:41:07 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T04:41:08.067 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:41:07 smithi047 conmon[78521]: raise self._exception 2022-05-14T04:41:08.067 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:41:07 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T04:41:08.067 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:41:07 smithi047 conmon[78521]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T04:41:08.067 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:41:07 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T04:41:08.067 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:41:07 smithi047 conmon[78521]: conn = await self._remote_connection(host, addr) 2022-05-14T04:41:08.068 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:41:07 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T04:41:08.068 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:41:07 smithi047 conmon[78521]: raise 2022-05-14T04:41:08.068 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:41:07 smithi047 conmon[78521]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T04:41:08.068 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:41:07 smithi047 conmon[78521]: self.gen.throw(type, value, traceback) 2022-05-14T04:41:08.068 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:41:07 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T04:41:08.069 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:41:07 smithi047 conmon[78521]: raise OrchestratorError(msg) 2022-05-14T04:41:08.069 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:41:07 smithi047 conmon[78521]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:41:08.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:41:08 smithi047 ceph-mon[77147]: pgmap v9751: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:41:08.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:41:08 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:41:08.546 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:41:08 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 18800 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:41:08.546 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:41:08 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:41:08.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T04:41:08.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:41:08.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T04:41:08.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:41:08.548 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:41:08.548 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:41:08.548 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T04:41:08.548 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:41:08.549 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T04:41:08.549 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:41:08.549 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T04:41:08.549 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:41:08.550 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T04:41:08.550 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:41:08.550 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T04:41:08.551 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:41:08.551 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:41:08.552 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:41:08.552 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:41:08 smithi047 ceph-mon[77147]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T04:41:08.553 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T04:41:08.553 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:41:08.554 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T04:41:08.554 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:41:08.555 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:41:08.555 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:41:08.555 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T04:41:08.556 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:41:08.557 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T04:41:08.557 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:41:08.558 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T04:41:08.558 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:41:08.559 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T04:41:08.559 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:41:08.559 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T04:41:08.560 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:41:08.560 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:41:08.561 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:41:08.561 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T04:41:08.562 INFO:journalctl@ceph.mon.b.smithi047.stdout: During handling of the above exception, another exception occurred: 2022-05-14T04:41:08.562 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T04:41:08.563 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T04:41:08.563 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T04:41:08.564 INFO:journalctl@ceph.mon.b.smithi047.stdout: return f(*arg) 2022-05-14T04:41:08.564 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T04:41:08.564 INFO:journalctl@ceph.mon.b.smithi047.stdout: self._write_client_files(client_files, host) 2022-05-14T04:41:08.565 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T04:41:08.567 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T04:41:08.567 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T04:41:08.568 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T04:41:08.568 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T04:41:08.569 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.event_loop.get_result(coro) 2022-05-14T04:41:08.570 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T04:41:08.572 INFO:journalctl@ceph.mon.b.smithi047.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T04:41:08.572 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T04:41:08.573 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.__get_result() 2022-05-14T04:41:08.573 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T04:41:08.573 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise self._exception 2022-05-14T04:41:08.574 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T04:41:08.574 INFO:journalctl@ceph.mon.b.smithi047.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T04:41:08.574 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T04:41:08.574 INFO:journalctl@ceph.mon.b.smithi047.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T04:41:08.574 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T04:41:08.575 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise 2022-05-14T04:41:08.576 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T04:41:08.576 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.gen.throw(type, value, traceback) 2022-05-14T04:41:08.576 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T04:41:08.576 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OrchestratorError(msg) 2022-05-14T04:41:08.576 INFO:journalctl@ceph.mon.b.smithi047.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:41:08.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:41:08 smithi115 conmon[90393]: 2022-05-14T04:41:08.308+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:41:08.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:41:08 smithi115 ceph-mon[78593]: pgmap v9751: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:41:08.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:41:08 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:41:08.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:41:08 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 18800 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:41:08.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:41:08 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:41:08.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T04:41:08.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:41:08.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T04:41:08.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:41:08.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:41:08.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:41:08.613 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T04:41:08.613 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:41:08.613 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T04:41:08.614 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:41:08.614 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T04:41:08.614 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:41:08.614 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T04:41:08.615 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:41:08.615 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T04:41:08.615 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:41:08.615 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:41:08.616 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:41:08.616 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:41:08 smithi115 ceph-mon[78593]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T04:41:08.617 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T04:41:08.617 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:41:08.617 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T04:41:08.617 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:41:08.618 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:41:08.618 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:41:08.618 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T04:41:08.618 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:41:08.619 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T04:41:08.619 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:41:08.619 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T04:41:08.619 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:41:08.620 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T04:41:08.620 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:41:08.620 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T04:41:08.620 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:41:08.620 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:41:08.621 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:41:08.624 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T04:41:08.624 INFO:journalctl@ceph.mon.c.smithi115.stdout: During handling of the above exception, another exception occurred: 2022-05-14T04:41:08.624 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T04:41:08.625 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T04:41:08.625 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T04:41:08.625 INFO:journalctl@ceph.mon.c.smithi115.stdout: return f(*arg) 2022-05-14T04:41:08.626 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T04:41:08.626 INFO:journalctl@ceph.mon.c.smithi115.stdout: self._write_client_files(client_files, host) 2022-05-14T04:41:08.626 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T04:41:08.626 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T04:41:08.626 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T04:41:08.627 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T04:41:08.627 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T04:41:08.627 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.event_loop.get_result(coro) 2022-05-14T04:41:08.627 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T04:41:08.628 INFO:journalctl@ceph.mon.c.smithi115.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T04:41:08.628 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T04:41:08.628 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.__get_result() 2022-05-14T04:41:08.628 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T04:41:08.628 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise self._exception 2022-05-14T04:41:08.629 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T04:41:08.629 INFO:journalctl@ceph.mon.c.smithi115.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T04:41:08.629 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T04:41:08.630 INFO:journalctl@ceph.mon.c.smithi115.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T04:41:08.630 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T04:41:08.630 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise 2022-05-14T04:41:08.630 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T04:41:08.630 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.gen.throw(type, value, traceback) 2022-05-14T04:41:08.631 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T04:41:08.631 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OrchestratorError(msg) 2022-05-14T04:41:08.631 INFO:journalctl@ceph.mon.c.smithi115.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:41:09.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:41:09 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:41:09.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:41:09 smithi115 conmon[90393]: 2022-05-14T04:41:09.330+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:41:09.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:41:09 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:41:10.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:41:10 smithi047 ceph-mon[77147]: pgmap v9752: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:41:10.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:41:10 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:41:10.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:41:10 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:41:10.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:41:10 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:41:10.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:41:10 smithi115 conmon[90393]: 2022-05-14T04:41:10.331+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:41:10.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:41:10 smithi115 ceph-mon[78593]: pgmap v9752: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:41:10.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:41:10 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:41:10.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:41:10 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:41:10.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:41:10 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:41:11.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:41:11 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:41:11.610 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:41:11 smithi115 conmon[90393]: 2022-05-14T04:41:11.321+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:41:11.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:41:11 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:41:12.545 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:41:12 smithi047 ceph-mon[77147]: pgmap v9753: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:41:12.545 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:41:12 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:41:12.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:41:12 smithi115 conmon[90393]: 2022-05-14T04:41:12.369+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:41:12.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:41:12 smithi115 ceph-mon[78593]: pgmap v9753: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:41:12.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:41:12 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:41:13.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:41:13 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:41:13.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:41:13 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 18805 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:41:13.610 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:41:13 smithi115 conmon[90393]: 2022-05-14T04:41:13.377+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:41:13.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:41:13 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:41:13.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:41:13 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 18805 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:41:14.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:41:14 smithi047 ceph-mon[77147]: pgmap v9754: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:41:14.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:41:14 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:41:14.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:41:14 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:41:14.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:41:14 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:41:14.611 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:41:14 smithi115 conmon[90393]: 2022-05-14T04:41:14.412+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:41:14.612 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:41:14 smithi115 ceph-mon[78593]: pgmap v9754: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:41:14.613 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:41:14 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:41:14.613 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:41:14 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:41:14.614 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:41:14 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:41:15.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:41:15 smithi115 conmon[90393]: 2022-05-14T04:41:15.454+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:41:15.612 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:41:15 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:41:15.612 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:41:15 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T04:41:15.612 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:41:15 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T04:41:15.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:41:15 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:41:15.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:41:15 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T04:41:15.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:41:15 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T04:41:16.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:41:16 smithi115 conmon[90393]: 2022-05-14T04:41:16.449+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:41:16.618 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:41:16 smithi115 ceph-mon[78593]: pgmap v9755: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:41:16.619 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:41:16 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:41:16.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:41:16 smithi047 ceph-mon[77147]: pgmap v9755: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:41:16.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:41:16 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:41:17.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:41:17 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:41:17.610 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:41:17 smithi115 conmon[90393]: 2022-05-14T04:41:17.478+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:41:17.795 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:41:17 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:41:18.610 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:41:18 smithi115 conmon[90393]: 2022-05-14T04:41:18.450+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:41:18.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:41:18 smithi115 ceph-mon[78593]: pgmap v9756: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:41:18.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:41:18 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:41:18.612 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:41:18 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:41:18.612 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:41:18 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:41:18.612 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:41:18 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 18810 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:41:18.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:41:18 smithi047 ceph-mon[77147]: pgmap v9756: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:41:18.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:41:18 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:41:18.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:41:18 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:41:18.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:41:18 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:41:18.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:41:18 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 18810 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:41:19.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:41:19 smithi115 conmon[90393]: 2022-05-14T04:41:19.474+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:41:19.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:41:19 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:41:19.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:41:19 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:41:20.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:41:20 smithi115 conmon[90393]: 2022-05-14T04:41:20.517+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:41:20.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:41:20 smithi115 ceph-mon[78593]: pgmap v9757: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:41:20.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:41:20 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:41:20.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:41:20 smithi047 ceph-mon[77147]: pgmap v9757: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:41:20.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:41:20 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:41:21.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:41:21 smithi115 conmon[90393]: 2022-05-14T04:41:21.530+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:41:21.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:41:21 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:41:21.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:41:21 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:41:22.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:41:22 smithi115 conmon[90393]: 2022-05-14T04:41:22.556+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:41:22.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:41:22 smithi115 ceph-mon[78593]: pgmap v9758: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:41:22.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:41:22 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:41:22.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:41:22 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:41:22.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:41:22 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:41:22.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:41:22 smithi047 ceph-mon[77147]: pgmap v9758: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:41:22.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:41:22 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:41:22.795 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:41:22 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:41:22.795 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:41:22 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:41:23.610 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:41:23 smithi115 conmon[90393]: 2022-05-14T04:41:23.519+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:41:23.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:41:23 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:41:23.612 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:41:23 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 18815 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:41:23.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:41:23 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:41:23.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:41:23 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 18815 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:41:24.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:41:24 smithi115 conmon[90393]: 2022-05-14T04:41:24.513+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:41:24.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:41:24 smithi115 ceph-mon[78593]: pgmap v9759: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:41:24.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:41:24 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:41:24.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:41:24 smithi047 ceph-mon[77147]: pgmap v9759: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:41:24.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:41:24 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:41:25.610 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:41:25 smithi115 conmon[90393]: 2022-05-14T04:41:25.528+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:41:25.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:41:25 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:41:25.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:41:25 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:41:26.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:41:26 smithi115 conmon[90393]: 2022-05-14T04:41:26.551+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:41:26.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:41:26 smithi115 ceph-mon[78593]: pgmap v9760: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:41:26.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:41:26 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:41:26.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:41:26 smithi047 ceph-mon[77147]: pgmap v9760: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:41:26.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:41:26 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:41:27.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:41:27 smithi115 conmon[90393]: 2022-05-14T04:41:27.558+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:41:27.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:41:27 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:41:27.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:41:27 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:41:28.610 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:41:28 smithi115 conmon[90393]: 2022-05-14T04:41:28.538+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:41:28.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:41:28 smithi115 ceph-mon[78593]: pgmap v9761: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:41:28.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:41:28 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:41:28.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:41:28 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 18820 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:41:28.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:41:28 smithi047 ceph-mon[77147]: pgmap v9761: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:41:28.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:41:28 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:41:28.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:41:28 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 18820 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:41:29.610 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:41:29 smithi115 conmon[90393]: 2022-05-14T04:41:29.548+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:41:29.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:41:29 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:41:29.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:41:29 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:41:30.589 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:41:30 smithi115 ceph-mon[78593]: pgmap v9762: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:41:30.589 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:41:30 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:41:30.589 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:41:30 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:41:30.590 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:41:30 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:41:30.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:41:30 smithi047 ceph-mon[77147]: pgmap v9762: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:41:30.795 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:41:30 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:41:30.795 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:41:30 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:41:30.795 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:41:30 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:41:30.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:41:30 smithi115 conmon[90393]: 2022-05-14T04:41:30.587+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:41:31.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:41:31 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:41:31.610 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:41:31 smithi115 conmon[90393]: 2022-05-14T04:41:31.556+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:41:31.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:41:31 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:41:32.598 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:41:32 smithi115 ceph-mon[78593]: pgmap v9763: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:41:32.598 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:41:32 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:41:32.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:41:32 smithi047 ceph-mon[77147]: pgmap v9763: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:41:32.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:41:32 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:41:32.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:41:32 smithi115 conmon[90393]: 2022-05-14T04:41:32.595+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:41:33.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:41:33 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:41:33.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:41:33 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 18825 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:41:33.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:41:33 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:41:33.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:41:33 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 18825 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:41:34.110 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:41:33 smithi115 conmon[90393]: 2022-05-14T04:41:33.636+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:41:34.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:41:34 smithi115 conmon[90393]: 2022-05-14T04:41:34.604+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:41:34.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:41:34 smithi115 ceph-mon[78593]: pgmap v9764: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:41:34.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:41:34 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:41:34.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:41:34 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:41:34.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:41:34 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:41:34.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:41:34 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:41:35.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:41:34 smithi047 ceph-mon[77147]: pgmap v9764: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:41:35.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:41:34 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:41:35.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:41:34 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:41:35.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:41:34 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:41:35.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:41:34 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:41:35.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:41:35 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:41:35.860 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:41:35 smithi115 conmon[90393]: 2022-05-14T04:41:35.573+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:41:36.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:41:35 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:41:36.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:41:36 smithi115 conmon[90393]: 2022-05-14T04:41:36.561+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:41:36.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:41:36 smithi115 ceph-mon[78593]: pgmap v9765: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:41:36.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:41:36 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:41:37.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:41:36 smithi047 ceph-mon[77147]: pgmap v9765: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:41:37.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:41:36 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:41:37.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:41:37 smithi115 conmon[90393]: 2022-05-14T04:41:37.552+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:41:37.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:41:37 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:41:37.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:41:37 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 18830 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:41:38.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:41:37 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:41:38.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:41:37 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 18830 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:41:38.860 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:41:38 smithi115 conmon[90393]: 2022-05-14T04:41:38.598+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:41:38.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:41:38 smithi115 ceph-mon[78593]: pgmap v9766: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:41:38.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:41:38 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:41:38.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:41:38 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:41:38.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:41:38 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:41:39.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:41:38 smithi047 ceph-mon[77147]: pgmap v9766: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:41:39.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:41:38 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:41:39.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:41:38 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:41:39.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:41:38 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:41:39.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:41:39 smithi115 conmon[90393]: 2022-05-14T04:41:39.601+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:41:39.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:41:39 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:41:40.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:41:39 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:41:40.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:41:40 smithi115 conmon[90393]: 2022-05-14T04:41:40.561+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:41:40.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:41:40 smithi115 ceph-mon[78593]: pgmap v9767: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:41:40.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:41:40 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:41:41.050 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:41:40 smithi047 ceph-mon[77147]: pgmap v9767: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:41:41.050 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:41:40 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:41:41.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:41:41 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:41:41.860 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:41:41 smithi115 conmon[90393]: 2022-05-14T04:41:41.598+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:41:42.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:41:41 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:41:43.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:41:42 smithi047 ceph-mon[77147]: pgmap v9768: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:41:43.045 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:41:42 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:41:43.045 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:41:42 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:41:43.046 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:41:42 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:41:43.046 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:41:42 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 18835 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:41:43.110 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:41:42 smithi115 conmon[90393]: 2022-05-14T04:41:42.611+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:41:43.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:41:42 smithi115 ceph-mon[78593]: pgmap v9768: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:41:43.112 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:41:42 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:41:43.112 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:41:42 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:41:43.113 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:41:42 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:41:43.113 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:41:42 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 18835 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:41:44.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:41:43 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:41:44.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:41:43 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:41:44.110 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:41:43 smithi115 conmon[90393]: 2022-05-14T04:41:43.613+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:41:44.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:41:44 smithi115 conmon[90393]: 2022-05-14T04:41:44.599+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:41:44.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:41:44 smithi115 ceph-mon[78593]: pgmap v9769: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:41:44.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:41:44 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:41:45.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:41:44 smithi047 ceph-mon[77147]: pgmap v9769: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:41:45.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:41:44 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:41:46.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:41:45 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:41:46.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:41:45 smithi115 conmon[90393]: 2022-05-14T04:41:45.626+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:41:46.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:41:45 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:41:47.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:41:46 smithi047 ceph-mon[77147]: pgmap v9770: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:41:47.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:41:46 smithi115 conmon[90393]: 2022-05-14T04:41:46.653+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:41:47.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:41:46 smithi115 ceph-mon[78593]: pgmap v9770: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:41:48.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:41:47 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:41:48.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:41:47 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 18840 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:41:48.110 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:41:47 smithi115 conmon[90393]: 2022-05-14T04:41:47.696+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:41:48.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:41:47 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:41:48.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:41:47 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 18840 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:41:49.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:41:48 smithi047 ceph-mon[77147]: pgmap v9771: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:41:49.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:41:48 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:41:49.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:41:48 smithi115 conmon[90393]: 2022-05-14T04:41:48.667+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:41:49.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:41:48 smithi115 ceph-mon[78593]: pgmap v9771: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:41:49.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:41:48 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:41:50.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:41:49 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:41:50.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:41:49 smithi115 conmon[90393]: 2022-05-14T04:41:49.678+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:41:50.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:41:49 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:41:51.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:41:50 smithi047 ceph-mon[77147]: pgmap v9772: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:41:51.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:41:50 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:41:51.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:41:50 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:41:51.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:41:50 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:41:51.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:41:50 smithi115 conmon[90393]: 2022-05-14T04:41:50.645+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:41:51.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:41:50 smithi115 ceph-mon[78593]: pgmap v9772: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:41:51.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:41:50 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:41:51.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:41:50 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:41:51.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:41:50 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:41:52.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:41:51 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:41:52.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:41:51 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:41:52.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:41:51 smithi115 conmon[90393]: 2022-05-14T04:41:51.612+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:41:52.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:41:51 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:41:52.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:41:51 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:41:52.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:41:52 smithi115 conmon[90393]: 2022-05-14T04:41:52.603+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:41:52.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:41:52 smithi115 ceph-mon[78593]: pgmap v9773: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:41:52.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:41:52 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:41:52.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:41:52 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 18845 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:41:53.046 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:41:52 smithi047 ceph-mon[77147]: pgmap v9773: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:41:53.046 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:41:52 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:41:53.047 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:41:52 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 18845 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:41:53.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:41:53 smithi115 conmon[90393]: 2022-05-14T04:41:53.576+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:41:53.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:41:53 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:41:54.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:41:53 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:41:55.048 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:41:54 smithi047 ceph-mon[77147]: pgmap v9774: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:41:55.049 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:41:54 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:41:55.049 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:41:54 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:41:55.049 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:41:54 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:41:55.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:41:54 smithi115 conmon[90393]: 2022-05-14T04:41:54.610+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:41:55.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:41:54 smithi115 ceph-mon[78593]: pgmap v9774: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:41:55.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:41:54 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:41:55.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:41:54 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:41:55.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:41:54 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:41:55.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:41:55 smithi115 conmon[90393]: 2022-05-14T04:41:55.596+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:41:55.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:41:55 smithi115 ceph-mon[78593]: pgmap v9775: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:41:55.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:41:55 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:41:56.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:41:55 smithi047 ceph-mon[77147]: pgmap v9775: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:41:56.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:41:55 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:41:57.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:41:56 smithi115 conmon[90393]: 2022-05-14T04:41:56.632+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:41:57.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:41:57 smithi115 conmon[90393]: 2022-05-14T04:41:57.590+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:41:58.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:41:58 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:41:58.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:41:58 smithi047 ceph-mon[77147]: pgmap v9776: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:41:58.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:41:58 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:41:58.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:41:58 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 18850 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:41:58.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:41:58 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:41:58.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:41:58 smithi115 ceph-mon[78593]: pgmap v9776: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:41:58.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:41:58 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:41:58.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:41:58 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 18850 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:41:58.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:41:58 smithi115 conmon[90393]: 2022-05-14T04:41:58.543+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:41:59.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:41:59 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:41:59.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:41:59 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:41:59.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:41:59 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:41:59.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:41:59 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:41:59.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:41:59 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:41:59.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:41:59 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:41:59.860 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:41:59 smithi115 conmon[90393]: 2022-05-14T04:41:59.513+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:42:00.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:42:00 smithi115 ceph-mon[78593]: pgmap v9777: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:42:00.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:42:00 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:42:00.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:42:00 smithi047 ceph-mon[77147]: pgmap v9777: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:42:00.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:42:00 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:42:00.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:42:00 smithi115 conmon[90393]: 2022-05-14T04:42:00.532+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:42:01.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:42:01 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:42:01.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:42:01 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:42:01.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:42:01 smithi115 conmon[90393]: 2022-05-14T04:42:01.572+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:42:02.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:42:02 smithi115 ceph-mon[78593]: pgmap v9778: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:42:02.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:42:02 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:42:02.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:42:02 smithi047 ceph-mon[77147]: pgmap v9778: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:42:02.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:42:02 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:42:02.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:42:02 smithi115 conmon[90393]: 2022-05-14T04:42:02.582+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:42:03.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:42:03 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:42:03.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:42:03 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:42:03.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:42:03 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:42:03.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:42:03 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 18855 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:42:03.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:42:03 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:42:03.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:42:03 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:42:03.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:42:03 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:42:03.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:42:03 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 18855 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:42:03.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:42:03 smithi115 conmon[90393]: 2022-05-14T04:42:03.541+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:42:04.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:42:04 smithi115 ceph-mon[78593]: pgmap v9779: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:42:04.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:42:04 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:42:04.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:42:04 smithi047 ceph-mon[77147]: pgmap v9779: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:42:04.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:42:04 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:42:04.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:42:04 smithi115 conmon[90393]: 2022-05-14T04:42:04.562+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:42:05.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:42:05 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:42:05.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:42:05 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:42:05.860 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:42:05 smithi115 conmon[90393]: 2022-05-14T04:42:05.513+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:42:06.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:42:06 smithi115 ceph-mon[78593]: pgmap v9780: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:42:06.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:42:06 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:42:06.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:42:06 smithi047 ceph-mon[77147]: pgmap v9780: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:42:06.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:42:06 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:42:06.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:42:06 smithi115 conmon[90393]: 2022-05-14T04:42:06.543+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:42:07.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:42:07 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:42:07.546 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:42:07 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:42:07.769 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:42:07 smithi115 conmon[90393]: 2022-05-14T04:42:07.513+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:42:08.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:42:08 smithi115 ceph-mon[78593]: pgmap v9781: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:42:08.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:42:08 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:42:08.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:42:08 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T04:42:08.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:42:08 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 18860 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:42:08.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:42:08 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T04:42:08.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:42:08 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T04:42:08.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:42:08 smithi047 ceph-mon[77147]: pgmap v9781: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:42:08.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:42:08 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:42:08.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:42:08 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T04:42:08.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:42:08 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 18860 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:42:08.545 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:42:08 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T04:42:08.545 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:42:08 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T04:42:08.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:42:08 smithi115 conmon[90393]: 2022-05-14T04:42:08.535+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:42:09.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:42:09 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:42:09.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:42:09 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:42:09.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:42:09 smithi115 conmon[90393]: 2022-05-14T04:42:09.516+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:42:10.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:42:10 smithi115 ceph-mon[78593]: pgmap v9782: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:42:10.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:42:10 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:42:10.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:42:10 smithi047 ceph-mon[77147]: pgmap v9782: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:42:10.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:42:10 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:42:10.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:42:10 smithi115 conmon[90393]: 2022-05-14T04:42:10.544+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:42:11.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:42:10 smithi047 conmon[78521]: 2022-05-14T04:42:10.735+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:42:11.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:42:10 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T04:42:11.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:42:10 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:42:11.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:42:10 smithi047 conmon[78521]: yield 2022-05-14T04:42:11.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:42:10 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:42:11.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:42:10 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:42:11.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:42:10 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:42:11.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:42:10 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T04:42:11.048 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:42:10 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:42:11.048 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:42:10 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T04:42:11.048 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:42:10 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:42:11.048 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:42:10 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T04:42:11.049 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:42:10 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:42:11.049 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:42:10 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T04:42:11.049 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:42:10 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:42:11.049 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:42:10 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T04:42:11.050 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:42:10 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:42:11.050 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:42:10 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:42:11.050 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:42:10 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:42:11.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:42:11 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:42:11.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:42:11 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:42:11.363 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:42:11 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:42:11.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:42:11 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:42:11.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:42:11 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:42:11.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:42:11 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:42:11.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:42:11 smithi115 conmon[90393]: 2022-05-14T04:42:11.548+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:42:12.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:42:12 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:42:12.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T04:42:12.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:42:12.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T04:42:12.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:42:12.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:42:12.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:42:12.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T04:42:12.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:42:12.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T04:42:12.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:42:12.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T04:42:12.365 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:42:12.365 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T04:42:12.366 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:42:12.366 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T04:42:12.366 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:42:12.366 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:42:12.367 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:42:12.367 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:42:12 smithi115 ceph-mon[78593]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T04:42:12.367 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:42:12 smithi115 ceph-mon[78593]: pgmap v9783: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:42:12.368 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:42:12 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:42:12.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:42:12 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:42:12.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T04:42:12.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:42:12.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T04:42:12.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:42:12.549 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:42:12.549 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:42:12.550 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T04:42:12.550 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:42:12.550 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T04:42:12.550 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:42:12.551 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T04:42:12.551 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:42:12.551 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T04:42:12.551 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:42:12.551 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T04:42:12.552 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:42:12.552 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:42:12.552 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:42:12.552 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:42:12 smithi047 ceph-mon[77147]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T04:42:12.553 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:42:12 smithi047 ceph-mon[77147]: pgmap v9783: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:42:12.553 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:42:12 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:42:12.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:42:12 smithi115 conmon[90393]: 2022-05-14T04:42:12.587+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:42:13.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:42:13 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:42:13.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:42:13 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 18865 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:42:13.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:42:13 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:42:13.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:42:13 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 18865 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:42:13.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:42:13 smithi115 conmon[90393]: 2022-05-14T04:42:13.540+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:42:14.106 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:42:13 smithi047 conmon[78521]: 2022-05-14T04:42:13.807+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:42:14.106 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:42:13 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T04:42:14.107 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:42:13 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:42:14.107 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:42:13 smithi047 conmon[78521]: yield 2022-05-14T04:42:14.107 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:42:13 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:42:14.107 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:42:13 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:42:14.108 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:42:13 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:42:14.108 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:42:13 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T04:42:14.108 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:42:13 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:42:14.108 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:42:13 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T04:42:14.109 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:42:13 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:42:14.109 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:42:13 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T04:42:14.109 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:42:13 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:42:14.110 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:42:13 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T04:42:14.110 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:42:13 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:42:14.110 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:42:13 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T04:42:14.111 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:42:13 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:42:14.111 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:42:13 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:42:14.111 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:42:13 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:42:14.112 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:42:13 smithi047 conmon[78521]: 2022-05-14T04:42:13.808+0000 7f5340779700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T04:42:14.112 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:42:13 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T04:42:14.112 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:42:13 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:42:14.112 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:42:13 smithi047 conmon[78521]: yield 2022-05-14T04:42:14.113 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:42:13 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:42:14.113 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:42:13 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:42:14.113 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:42:13 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:42:14.113 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:42:13 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T04:42:14.114 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:42:13 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:42:14.114 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:42:13 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T04:42:14.114 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:42:13 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:42:14.114 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:42:13 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T04:42:14.115 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:42:13 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:42:14.115 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:42:13 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T04:42:14.115 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:42:13 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:42:14.115 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:42:13 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T04:42:14.116 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:42:13 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:42:14.116 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:42:13 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:42:14.116 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:42:13 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:42:14.116 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:42:13 smithi047 conmon[78521]: 2022-05-14T04:42:14.117 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:42:13 smithi047 conmon[78521]: During handling of the above exception, another exception occurred: 2022-05-14T04:42:14.117 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:42:13 smithi047 conmon[78521]: 2022-05-14T04:42:14.117 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:42:13 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T04:42:14.117 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:42:13 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T04:42:14.118 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:42:13 smithi047 conmon[78521]: return f(*arg) 2022-05-14T04:42:14.119 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:42:13 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T04:42:14.119 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:42:13 smithi047 conmon[78521]: self._write_client_files(client_files, host) 2022-05-14T04:42:14.119 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:42:13 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T04:42:14.120 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:42:13 smithi047 conmon[78521]: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T04:42:14.120 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:42:13 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T04:42:14.120 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:42:13 smithi047 conmon[78521]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T04:42:14.120 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:42:13 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T04:42:14.121 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:42:13 smithi047 conmon[78521]: return self.event_loop.get_result(coro) 2022-05-14T04:42:14.121 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:42:13 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T04:42:14.121 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:42:13 smithi047 conmon[78521]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T04:42:14.121 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:42:13 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T04:42:14.121 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:42:13 smithi047 conmon[78521]: return self.__get_result() 2022-05-14T04:42:14.122 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:42:13 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T04:42:14.122 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:42:13 smithi047 conmon[78521]: raise self._exception 2022-05-14T04:42:14.122 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:42:13 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T04:42:14.122 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:42:13 smithi047 conmon[78521]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T04:42:14.123 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:42:13 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T04:42:14.123 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:42:13 smithi047 conmon[78521]: conn = await self._remote_connection(host, addr) 2022-05-14T04:42:14.123 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:42:13 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T04:42:14.123 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:42:13 smithi047 conmon[78521]: raise 2022-05-14T04:42:14.123 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:42:13 smithi047 conmon[78521]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T04:42:14.124 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:42:13 smithi047 conmon[78521]: self.gen.throw(type, value, traceback) 2022-05-14T04:42:14.124 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:42:13 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T04:42:14.124 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:42:13 smithi047 conmon[78521]: raise OrchestratorError(msg) 2022-05-14T04:42:14.124 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:42:13 smithi047 conmon[78521]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:42:14.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:42:14 smithi115 ceph-mon[78593]: pgmap v9784: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:42:14.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:42:14 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:42:14.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:42:14 smithi047 ceph-mon[77147]: pgmap v9784: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:42:14.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:42:14 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:42:14.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:42:14 smithi115 conmon[90393]: 2022-05-14T04:42:14.583+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:42:15.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:42:15 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:42:15.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T04:42:15.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:42:15.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T04:42:15.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:42:15.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:42:15.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:42:15.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T04:42:15.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:42:15.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T04:42:15.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:42:15.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T04:42:15.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:42:15.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T04:42:15.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:42:15.548 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T04:42:15.548 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:42:15.548 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:42:15.548 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:42:15.549 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:42:15 smithi047 ceph-mon[77147]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T04:42:15.549 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T04:42:15.549 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:42:15.549 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T04:42:15.550 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:42:15.550 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:42:15.550 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:42:15.550 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T04:42:15.550 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:42:15.551 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T04:42:15.551 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:42:15.551 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T04:42:15.551 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:42:15.552 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T04:42:15.552 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:42:15.552 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T04:42:15.552 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:42:15.552 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:42:15.553 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:42:15.553 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T04:42:15.553 INFO:journalctl@ceph.mon.b.smithi047.stdout: During handling of the above exception, another exception occurred: 2022-05-14T04:42:15.553 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T04:42:15.554 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T04:42:15.554 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T04:42:15.554 INFO:journalctl@ceph.mon.b.smithi047.stdout: return f(*arg) 2022-05-14T04:42:15.554 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T04:42:15.555 INFO:journalctl@ceph.mon.b.smithi047.stdout: self._write_client_files(client_files, host) 2022-05-14T04:42:15.555 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T04:42:15.555 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T04:42:15.555 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T04:42:15.555 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T04:42:15.556 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T04:42:15.556 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.event_loop.get_result(coro) 2022-05-14T04:42:15.556 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T04:42:15.556 INFO:journalctl@ceph.mon.b.smithi047.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T04:42:15.556 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T04:42:15.557 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.__get_result() 2022-05-14T04:42:15.557 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T04:42:15.557 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise self._exception 2022-05-14T04:42:15.557 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T04:42:15.558 INFO:journalctl@ceph.mon.b.smithi047.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T04:42:15.558 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T04:42:15.558 INFO:journalctl@ceph.mon.b.smithi047.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T04:42:15.558 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T04:42:15.559 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise 2022-05-14T04:42:15.559 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T04:42:15.559 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.gen.throw(type, value, traceback) 2022-05-14T04:42:15.559 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T04:42:15.559 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OrchestratorError(msg) 2022-05-14T04:42:15.560 INFO:journalctl@ceph.mon.b.smithi047.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:42:15.560 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:42:15 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:42:15.560 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:42:15 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:42:15.560 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:42:15 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:42:15.561 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:42:15 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T04:42:15.561 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:42:15 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T04:42:15.579 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:42:15 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:42:15.580 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T04:42:15.580 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:42:15.580 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T04:42:15.580 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:42:15.580 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:42:15.581 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:42:15.581 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T04:42:15.581 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:42:15.581 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T04:42:15.582 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:42:15.582 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T04:42:15.582 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:42:15.582 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T04:42:15.582 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:42:15.583 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T04:42:15.583 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:42:15.583 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:42:15.583 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:42:15.584 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:42:15 smithi115 ceph-mon[78593]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T04:42:15.584 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T04:42:15.584 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:42:15.584 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T04:42:15.584 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:42:15.585 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:42:15.585 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:42:15.585 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T04:42:15.585 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:42:15.585 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T04:42:15.586 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:42:15.586 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T04:42:15.586 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:42:15.586 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T04:42:15.586 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:42:15.587 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T04:42:15.587 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:42:15.587 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:42:15.587 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:42:15.587 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T04:42:15.588 INFO:journalctl@ceph.mon.c.smithi115.stdout: During handling of the above exception, another exception occurred: 2022-05-14T04:42:15.588 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T04:42:15.588 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T04:42:15.589 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T04:42:15.589 INFO:journalctl@ceph.mon.c.smithi115.stdout: return f(*arg) 2022-05-14T04:42:15.589 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T04:42:15.589 INFO:journalctl@ceph.mon.c.smithi115.stdout: self._write_client_files(client_files, host) 2022-05-14T04:42:15.590 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T04:42:15.590 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T04:42:15.590 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T04:42:15.590 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T04:42:15.591 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T04:42:15.591 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.event_loop.get_result(coro) 2022-05-14T04:42:15.591 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T04:42:15.591 INFO:journalctl@ceph.mon.c.smithi115.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T04:42:15.592 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T04:42:15.592 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.__get_result() 2022-05-14T04:42:15.592 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T04:42:15.592 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise self._exception 2022-05-14T04:42:15.592 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T04:42:15.593 INFO:journalctl@ceph.mon.c.smithi115.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T04:42:15.593 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T04:42:15.593 INFO:journalctl@ceph.mon.c.smithi115.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T04:42:15.593 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T04:42:15.594 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise 2022-05-14T04:42:15.594 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T04:42:15.594 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.gen.throw(type, value, traceback) 2022-05-14T04:42:15.594 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T04:42:15.595 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OrchestratorError(msg) 2022-05-14T04:42:15.595 INFO:journalctl@ceph.mon.c.smithi115.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:42:15.595 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:42:15 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:42:15.595 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:42:15 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:42:15.596 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:42:15 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:42:15.596 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:42:15 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T04:42:15.596 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:42:15 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T04:42:15.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:42:15 smithi115 conmon[90393]: 2022-05-14T04:42:15.576+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:42:16.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:42:16 smithi047 ceph-mon[77147]: pgmap v9785: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:42:16.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:42:16 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:42:16.574 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:42:16 smithi115 ceph-mon[78593]: pgmap v9785: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:42:16.574 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:42:16 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:42:16.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:42:16 smithi115 conmon[90393]: 2022-05-14T04:42:16.571+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:42:17.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:42:17 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:42:17.599 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:42:17 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:42:17.860 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:42:17 smithi115 conmon[90393]: 2022-05-14T04:42:17.596+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:42:18.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:42:18 smithi047 ceph-mon[77147]: pgmap v9786: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:42:18.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:42:18 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:42:18.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:42:18 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 18870 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:42:18.616 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:42:18 smithi115 ceph-mon[78593]: pgmap v9786: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:42:18.617 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:42:18 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:42:18.617 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:42:18 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 18870 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:42:19.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:42:18 smithi115 conmon[90393]: 2022-05-14T04:42:18.644+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:42:19.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:42:19 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:42:19.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:42:19 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:42:19.597 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:42:19 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:42:19.598 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:42:19 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:42:19.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:42:19 smithi115 conmon[90393]: 2022-05-14T04:42:19.595+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:42:20.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:42:20 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:42:20.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:42:20 smithi047 ceph-mon[77147]: pgmap v9787: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:42:20.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:42:20 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:42:20.573 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:42:20 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:42:20.573 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:42:20 smithi115 ceph-mon[78593]: pgmap v9787: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:42:20.573 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:42:20 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:42:20.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:42:20 smithi115 conmon[90393]: 2022-05-14T04:42:20.571+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:42:21.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:42:21 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:42:21.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:42:21 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:42:21.862 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:42:21 smithi115 conmon[90393]: 2022-05-14T04:42:21.605+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:42:22.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:42:22 smithi047 ceph-mon[77147]: pgmap v9788: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:42:22.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:42:22 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:42:22.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:42:22 smithi115 ceph-mon[78593]: pgmap v9788: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:42:22.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:42:22 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:42:23.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:42:22 smithi115 conmon[90393]: 2022-05-14T04:42:22.644+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:42:23.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:42:23 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:42:23.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:42:23 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:42:23.545 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:42:23 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 18875 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:42:23.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:42:23 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:42:23.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:42:23 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:42:23.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:42:23 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 18875 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:42:24.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:42:23 smithi115 conmon[90393]: 2022-05-14T04:42:23.657+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:42:24.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:42:24 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:42:24.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:42:24 smithi047 ceph-mon[77147]: pgmap v9789: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:42:24.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:42:24 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:42:24.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:42:24 smithi115 ceph-mon[78593]: pgmap v9789: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:42:25.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:42:24 smithi115 conmon[90393]: 2022-05-14T04:42:24.685+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:42:25.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:42:25 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:42:25.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:42:25 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:42:26.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:42:25 smithi115 conmon[90393]: 2022-05-14T04:42:25.682+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:42:26.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:42:26 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:42:26.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:42:26 smithi047 ceph-mon[77147]: pgmap v9790: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:42:26.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:42:26 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:42:26.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:42:26 smithi115 ceph-mon[78593]: pgmap v9790: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:42:27.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:42:26 smithi115 conmon[90393]: 2022-05-14T04:42:26.654+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:42:27.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:42:27 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:42:27.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:42:27 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:42:28.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:42:27 smithi115 conmon[90393]: 2022-05-14T04:42:27.644+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:42:28.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:42:28 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:42:28.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:42:28 smithi047 ceph-mon[77147]: pgmap v9791: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:42:28.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:42:28 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 18880 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:42:28.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:42:28 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:42:28.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:42:28 smithi115 ceph-mon[78593]: pgmap v9791: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:42:28.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:42:28 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 18880 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:42:29.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:42:28 smithi115 conmon[90393]: 2022-05-14T04:42:28.613+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:42:29.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:42:29 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:42:29.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:42:29 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:42:29.590 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:42:29 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:42:29.591 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:42:29 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:42:29.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:42:29 smithi115 conmon[90393]: 2022-05-14T04:42:29.587+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:42:30.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:42:30 smithi047 ceph-mon[77147]: pgmap v9792: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:42:30.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:42:30 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:42:30.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:42:30 smithi115 ceph-mon[78593]: pgmap v9792: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:42:30.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:42:30 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:42:31.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:42:30 smithi115 conmon[90393]: 2022-05-14T04:42:30.636+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:42:31.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:42:31 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:42:31.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:42:31 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:42:31.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:42:31 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:42:31.600 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:42:31 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:42:31.601 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:42:31 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:42:31.601 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:42:31 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:42:31.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:42:31 smithi115 conmon[90393]: 2022-05-14T04:42:31.597+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:42:32.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:42:32 smithi047 ceph-mon[77147]: pgmap v9793: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:42:32.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:42:32 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:42:32.568 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:42:32 smithi115 ceph-mon[78593]: pgmap v9793: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:42:32.568 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:42:32 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:42:32.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:42:32 smithi115 conmon[90393]: 2022-05-14T04:42:32.564+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:42:33.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:42:33 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:42:33.545 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:42:33 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 18885 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:42:33.585 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:42:33 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:42:33.585 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:42:33 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 18885 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:42:33.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:42:33 smithi115 conmon[90393]: 2022-05-14T04:42:33.582+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:42:34.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:42:34 smithi047 ceph-mon[77147]: pgmap v9794: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:42:34.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:42:34 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:42:34.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:42:34 smithi115 ceph-mon[78593]: pgmap v9794: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:42:34.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:42:34 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:42:35.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:42:34 smithi115 conmon[90393]: 2022-05-14T04:42:34.609+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:42:35.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:42:35 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:42:35.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:42:35 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:42:35.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:42:35 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:42:35.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:42:35 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:42:35.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:42:35 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:42:35.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:42:35 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:42:36.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:42:35 smithi115 conmon[90393]: 2022-05-14T04:42:35.628+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:42:36.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:42:36 smithi047 ceph-mon[77147]: pgmap v9795: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:42:36.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:42:36 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:42:36.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:42:36 smithi115 ceph-mon[78593]: pgmap v9795: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:42:36.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:42:36 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:42:37.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:42:36 smithi115 conmon[90393]: 2022-05-14T04:42:36.674+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:42:38.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:42:37 smithi115 conmon[90393]: 2022-05-14T04:42:37.635+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:42:38.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:42:38 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:42:38.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:42:38 smithi047 ceph-mon[77147]: pgmap v9796: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:42:38.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:42:38 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:42:38.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:42:38 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 18890 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:42:38.606 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:42:38 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:42:38.606 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:42:38 smithi115 ceph-mon[78593]: pgmap v9796: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:42:38.606 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:42:38 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:42:38.606 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:42:38 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 18890 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:42:38.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:42:38 smithi115 conmon[90393]: 2022-05-14T04:42:38.603+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:42:39.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:42:39 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:42:39.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:42:39 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:42:39.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:42:39 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:42:39.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:42:39 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:42:39.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:42:39 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:42:39.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:42:39 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:42:40.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:42:39 smithi115 conmon[90393]: 2022-05-14T04:42:39.644+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:42:40.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:42:40 smithi047 ceph-mon[77147]: pgmap v9797: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:42:40.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:42:40 smithi115 ceph-mon[78593]: pgmap v9797: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:42:41.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:42:40 smithi115 conmon[90393]: 2022-05-14T04:42:40.688+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:42:41.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:42:41 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:42:41.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:42:41 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:42:42.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:42:41 smithi115 conmon[90393]: 2022-05-14T04:42:41.664+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:42:42.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:42:42 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:42:42.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:42:42 smithi047 ceph-mon[77147]: pgmap v9798: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:42:42.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:42:42 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:42:42.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:42:42 smithi115 ceph-mon[78593]: pgmap v9798: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:42:43.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:42:42 smithi115 conmon[90393]: 2022-05-14T04:42:42.638+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:42:43.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:42:43 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:42:43.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:42:43 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:42:43.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:42:43 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:42:43.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:42:43 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:42:43.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:42:43 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 18895 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:42:43.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:42:43 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:42:43.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:42:43 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:42:43.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:42:43 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:42:43.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:42:43 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:42:43.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:42:43 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 18895 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:42:44.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:42:43 smithi115 conmon[90393]: 2022-05-14T04:42:43.661+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:42:45.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:42:44 smithi047 ceph-mon[77147]: pgmap v9799: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:42:45.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:42:44 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:42:45.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:42:44 smithi115 conmon[90393]: 2022-05-14T04:42:44.651+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:42:45.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:42:44 smithi115 ceph-mon[78593]: pgmap v9799: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:42:45.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:42:44 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:42:46.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:42:45 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:42:46.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:42:45 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:42:46.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:42:45 smithi115 conmon[90393]: 2022-05-14T04:42:45.636+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:42:46.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:42:45 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:42:46.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:42:45 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:42:46.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:42:46 smithi115 conmon[90393]: 2022-05-14T04:42:46.604+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:42:46.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:42:46 smithi115 ceph-mon[78593]: pgmap v9800: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:42:46.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:42:46 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:42:47.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:42:46 smithi047 ceph-mon[77147]: pgmap v9800: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:42:47.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:42:46 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:42:48.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:42:47 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 18900 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:42:48.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:42:47 smithi115 conmon[90393]: 2022-05-14T04:42:47.653+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:42:48.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:42:47 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 18900 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:42:49.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:42:48 smithi047 ceph-mon[77147]: pgmap v9801: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:42:49.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:42:48 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:42:49.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:42:48 smithi115 conmon[90393]: 2022-05-14T04:42:48.669+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:42:49.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:42:48 smithi115 ceph-mon[78593]: pgmap v9801: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:42:49.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:42:48 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:42:50.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:42:49 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:42:50.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:42:49 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:42:50.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:42:49 smithi115 conmon[90393]: 2022-05-14T04:42:49.625+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:42:50.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:42:49 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:42:50.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:42:49 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:42:50.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:42:50 smithi115 conmon[90393]: 2022-05-14T04:42:50.595+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:42:50.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:42:50 smithi115 ceph-mon[78593]: pgmap v9802: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:42:50.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:42:50 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:42:50.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:42:50 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:42:50.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:42:50 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:42:51.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:42:50 smithi047 ceph-mon[77147]: pgmap v9802: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:42:51.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:42:50 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:42:51.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:42:50 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:42:51.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:42:50 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:42:51.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:42:51 smithi115 conmon[90393]: 2022-05-14T04:42:51.587+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:42:51.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:42:51 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:42:52.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:42:51 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:42:52.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:42:52 smithi115 conmon[90393]: 2022-05-14T04:42:52.574+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:42:52.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:42:52 smithi115 ceph-mon[78593]: pgmap v9803: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:42:52.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:42:52 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:42:52.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:42:52 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 18905 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:42:53.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:42:52 smithi047 ceph-mon[77147]: pgmap v9803: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:42:53.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:42:52 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:42:53.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:42:52 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 18905 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:42:53.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:42:53 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:42:53.860 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:42:53 smithi115 conmon[90393]: 2022-05-14T04:42:53.544+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:42:54.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:42:53 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:42:54.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:42:54 smithi115 conmon[90393]: 2022-05-14T04:42:54.507+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:42:54.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:42:54 smithi115 ceph-mon[78593]: pgmap v9804: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:42:54.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:42:54 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:42:54.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:42:54 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:42:54.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:42:54 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:42:55.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:42:54 smithi047 ceph-mon[77147]: pgmap v9804: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:42:55.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:42:54 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:42:55.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:42:54 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:42:55.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:42:54 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:42:55.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:42:55 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:42:55.860 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:42:55 smithi115 conmon[90393]: 2022-05-14T04:42:55.519+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:42:56.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:42:55 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:42:56.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:42:56 smithi115 ceph-mon[78593]: pgmap v9805: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:42:56.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:42:56 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:42:56.860 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:42:56 smithi115 conmon[90393]: 2022-05-14T04:42:56.495+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:42:57.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:42:56 smithi047 ceph-mon[77147]: pgmap v9805: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:42:57.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:42:56 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:42:57.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:42:57 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:42:57.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:42:57 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 18910 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:42:57.861 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:42:57 smithi115 conmon[90393]: 2022-05-14T04:42:57.519+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:42:58.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:42:57 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:42:58.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:42:57 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 18910 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:42:58.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:42:58 smithi115 conmon[90393]: 2022-05-14T04:42:58.476+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:42:58.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:42:58 smithi115 ceph-mon[78593]: pgmap v9806: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:42:58.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:42:58 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:42:58.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:42:58 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:42:58.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:42:58 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:42:59.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:42:58 smithi047 ceph-mon[77147]: pgmap v9806: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:42:59.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:42:58 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:42:59.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:42:58 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:42:59.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:42:58 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:42:59.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:42:59 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:42:59.860 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:42:59 smithi115 conmon[90393]: 2022-05-14T04:42:59.474+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:43:00.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:42:59 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:43:00.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:43:00 smithi115 ceph-mon[78593]: pgmap v9807: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:43:00.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:43:00 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:43:00.860 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:43:00 smithi115 conmon[90393]: 2022-05-14T04:43:00.477+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:43:01.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:43:00 smithi047 ceph-mon[77147]: pgmap v9807: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:43:01.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:43:00 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:43:01.710 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:43:01 smithi115 conmon[90393]: 2022-05-14T04:43:01.432+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:43:02.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:43:01 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:43:02.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:43:01 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:43:02.711 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:43:02 smithi115 conmon[90393]: 2022-05-14T04:43:02.428+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:43:03.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:43:02 smithi047 ceph-mon[77147]: pgmap v9808: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:43:03.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:43:02 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:43:03.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:43:02 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:43:03.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:43:02 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:43:03.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:43:02 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 18915 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:43:03.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:43:02 smithi115 ceph-mon[78593]: pgmap v9808: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:43:03.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:43:02 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:43:03.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:43:02 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:43:03.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:43:02 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:43:03.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:43:02 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 18915 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:43:03.716 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:43:03 smithi115 conmon[90393]: 2022-05-14T04:43:03.420+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:43:04.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:43:03 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:43:04.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:43:03 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:43:04.717 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:43:04 smithi115 conmon[90393]: 2022-05-14T04:43:04.398+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:43:05.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:43:04 smithi047 ceph-mon[77147]: pgmap v9809: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:43:05.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:43:04 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:43:05.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:43:04 smithi115 ceph-mon[78593]: pgmap v9809: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:43:05.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:43:04 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:43:05.722 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:43:05 smithi115 conmon[90393]: 2022-05-14T04:43:05.363+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:43:06.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:43:05 smithi047 ceph-mon[77147]: pgmap v9810: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:43:06.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:43:05 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:43:06.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:43:05 smithi115 ceph-mon[78593]: pgmap v9810: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:43:06.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:43:05 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:43:06.726 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:43:06 smithi115 conmon[90393]: 2022-05-14T04:43:06.381+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:43:07.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:43:06 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:43:07.114 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:43:06 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:43:07.729 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:43:07 smithi115 conmon[90393]: 2022-05-14T04:43:07.400+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:43:08.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:43:07 smithi047 ceph-mon[77147]: pgmap v9811: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:43:08.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:43:07 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:43:08.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:43:07 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 18920 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:43:08.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:43:07 smithi115 ceph-mon[78593]: pgmap v9811: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:43:08.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:43:07 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:43:08.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:43:07 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 18920 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:43:08.730 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:43:08 smithi115 conmon[90393]: 2022-05-14T04:43:08.373+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:43:09.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:43:08 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:43:09.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:43:08 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:43:09.741 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:43:09 smithi115 conmon[90393]: 2022-05-14T04:43:09.404+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:43:10.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:43:09 smithi047 ceph-mon[77147]: pgmap v9812: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:43:10.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:43:09 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:43:10.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:43:09 smithi115 ceph-mon[78593]: pgmap v9812: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:43:10.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:43:09 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:43:10.736 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:43:10 smithi115 conmon[90393]: 2022-05-14T04:43:10.392+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:43:11.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:43:10 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:43:11.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:43:10 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:43:11.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:43:10 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:43:11.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:43:10 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:43:11.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:43:10 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:43:11.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:43:10 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:43:11.741 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:43:11 smithi115 conmon[90393]: 2022-05-14T04:43:11.429+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:43:12.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:43:11 smithi047 ceph-mon[77147]: pgmap v9813: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:43:12.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:43:11 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:43:12.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:43:11 smithi115 ceph-mon[78593]: pgmap v9813: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:43:12.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:43:11 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:43:12.742 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:43:12 smithi115 conmon[90393]: 2022-05-14T04:43:12.465+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:43:13.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:43:12 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:43:13.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:43:12 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 18925 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:43:13.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:43:12 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:43:13.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:43:12 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 18925 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:43:13.748 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:43:13 smithi115 conmon[90393]: 2022-05-14T04:43:13.493+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:43:14.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:43:13 smithi047 ceph-mon[77147]: pgmap v9814: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:43:14.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:43:13 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:43:14.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:43:13 smithi115 ceph-mon[78593]: pgmap v9814: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:43:14.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:43:13 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:43:14.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:43:14 smithi115 conmon[90393]: 2022-05-14T04:43:14.531+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:43:14.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:43:14 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:43:14.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:43:14 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:43:14.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:43:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T04:43:14.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:43:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T04:43:14.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:43:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T04:43:14.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:43:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' 2022-05-14T04:43:14.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:43:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' 2022-05-14T04:43:14.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:43:14 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:43:14.862 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:43:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T04:43:14.862 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:43:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T04:43:15.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:43:14 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:43:15.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:43:14 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:43:15.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:43:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T04:43:15.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:43:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T04:43:15.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:43:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T04:43:15.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:43:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' 2022-05-14T04:43:15.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:43:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' 2022-05-14T04:43:15.045 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:43:14 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:43:15.045 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:43:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T04:43:15.045 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:43:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T04:43:15.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:43:15 smithi115 conmon[90393]: 2022-05-14T04:43:15.512+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:43:15.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:43:15 smithi115 ceph-mon[78593]: pgmap v9815: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:43:15.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:43:15 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:43:16.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:43:15 smithi047 ceph-mon[77147]: pgmap v9815: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:43:16.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:43:15 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:43:16.840 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:43:16 smithi115 conmon[90393]: 2022-05-14T04:43:16.560+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:43:16.840 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:43:16 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:43:17.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:43:16 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:43:17.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:43:16 smithi047 conmon[78521]: 2022-05-14T04:43:16.914+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:43:17.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:43:16 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T04:43:17.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:43:16 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:43:17.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:43:16 smithi047 conmon[78521]: yield 2022-05-14T04:43:17.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:43:16 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:43:17.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:43:16 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:43:17.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:43:16 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:43:17.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:43:16 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T04:43:17.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:43:16 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:43:17.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:43:16 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T04:43:17.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:43:16 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:43:17.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:43:16 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T04:43:17.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:43:16 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:43:17.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:43:16 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T04:43:17.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:43:16 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:43:17.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:43:16 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T04:43:17.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:43:16 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:43:17.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:43:16 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:43:17.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:43:16 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:43:17.845 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:43:17 smithi115 conmon[90393]: 2022-05-14T04:43:17.586+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:43:18.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:43:17 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' 2022-05-14T04:43:18.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:43:17 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' 2022-05-14T04:43:18.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:43:17 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:43:18.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T04:43:18.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:43:18.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T04:43:18.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:43:18.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:43:18.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:43:18.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T04:43:18.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:43:18.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T04:43:18.113 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:43:18.113 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T04:43:18.113 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:43:18.113 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T04:43:18.114 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:43:18.114 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T04:43:18.114 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:43:18.114 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:43:18.114 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:43:18.115 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:43:17 smithi115 ceph-mon[78593]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T04:43:18.115 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:43:17 smithi115 ceph-mon[78593]: pgmap v9816: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:43:18.115 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:43:17 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-05-14T04:43:18.115 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:43:17 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-05-14T04:43:18.116 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:43:17 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-05-14T04:43:18.116 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:43:17 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-05-14T04:43:18.116 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:43:17 smithi115 ceph-mon[78593]: Adjusting osd_memory_target on smithi115 to 197.0M 2022-05-14T04:43:18.116 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:43:17 smithi115 ceph-mon[78593]: Unable to set osd_memory_target on smithi115 to 206629785: error parsing value: Value '206629785' is below minimum 939524096 2022-05-14T04:43:18.116 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:43:17 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-05-14T04:43:18.117 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:43:17 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-05-14T04:43:18.117 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:43:17 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-05-14T04:43:18.117 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:43:17 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-05-14T04:43:18.117 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:43:17 smithi115 ceph-mon[78593]: Adjusting osd_memory_target on smithi047 to 198.1M 2022-05-14T04:43:18.118 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:43:17 smithi115 ceph-mon[78593]: Unable to set osd_memory_target on smithi047 to 207755878: error parsing value: Value '207755878' is below minimum 939524096 2022-05-14T04:43:18.118 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:43:17 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:43:18.118 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:43:17 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 18930 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:43:18.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:43:17 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' 2022-05-14T04:43:18.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:43:17 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' 2022-05-14T04:43:18.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:43:17 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:43:18.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T04:43:18.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:43:18.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T04:43:18.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:43:18.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:43:18.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:43:18.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T04:43:18.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:43:18.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T04:43:18.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:43:18.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T04:43:18.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:43:18.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T04:43:18.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:43:18.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T04:43:18.298 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:43:18.298 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:43:18.298 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:43:18.298 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:43:17 smithi047 ceph-mon[77147]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T04:43:18.299 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:43:17 smithi047 ceph-mon[77147]: pgmap v9816: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:43:18.299 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:43:17 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-05-14T04:43:18.299 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:43:17 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-05-14T04:43:18.299 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:43:17 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-05-14T04:43:18.300 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:43:17 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-05-14T04:43:18.300 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:43:17 smithi047 ceph-mon[77147]: Adjusting osd_memory_target on smithi115 to 197.0M 2022-05-14T04:43:18.300 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:43:17 smithi047 ceph-mon[77147]: Unable to set osd_memory_target on smithi115 to 206629785: error parsing value: Value '206629785' is below minimum 939524096 2022-05-14T04:43:18.300 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:43:17 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-05-14T04:43:18.300 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:43:17 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-05-14T04:43:18.301 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:43:17 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-05-14T04:43:18.301 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:43:17 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-05-14T04:43:18.301 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:43:17 smithi047 ceph-mon[77147]: Adjusting osd_memory_target on smithi047 to 198.1M 2022-05-14T04:43:18.301 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:43:17 smithi047 ceph-mon[77147]: Unable to set osd_memory_target on smithi047 to 207755878: error parsing value: Value '207755878' is below minimum 939524096 2022-05-14T04:43:18.302 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:43:17 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:43:18.302 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:43:17 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 18930 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:43:18.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:43:18 smithi115 conmon[90393]: 2022-05-14T04:43:18.561+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:43:19.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:43:18 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:43:19.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:43:18 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:43:19.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:43:18 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:43:19.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:43:18 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:43:19.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:43:18 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:43:19.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:43:18 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:43:19.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:43:19 smithi115 conmon[90393]: 2022-05-14T04:43:19.554+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:43:20.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:43:19 smithi047 ceph-mon[77147]: pgmap v9817: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:43:20.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:43:19 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:43:20.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:43:19 smithi047 conmon[78521]: 2022-05-14T04:43:19.984+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:43:20.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:43:19 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T04:43:20.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:43:19 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:43:20.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:43:19 smithi047 conmon[78521]: yield 2022-05-14T04:43:20.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:43:19 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:43:20.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:43:19 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:43:20.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:43:19 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:43:20.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:43:19 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T04:43:20.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:43:19 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:43:20.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:43:19 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T04:43:20.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:43:19 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:43:20.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:43:19 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T04:43:20.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:43:19 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:43:20.297 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:43:19 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T04:43:20.297 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:43:19 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:43:20.297 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:43:19 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T04:43:20.297 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:43:19 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:43:20.297 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:43:19 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:43:20.298 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:43:19 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:43:20.298 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:43:19 smithi047 conmon[78521]: 2022-05-14T04:43:19.985+0000 7f5344f82700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T04:43:20.298 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:43:19 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T04:43:20.298 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:43:19 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:43:20.299 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:43:19 smithi047 conmon[78521]: yield 2022-05-14T04:43:20.299 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:43:19 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:43:20.299 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:43:19 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:43:20.299 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:43:19 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:43:20.299 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:43:19 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T04:43:20.300 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:43:19 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:43:20.300 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:43:19 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T04:43:20.300 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:43:19 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:43:20.300 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:43:19 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T04:43:20.301 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:43:19 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:43:20.301 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:43:19 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T04:43:20.301 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:43:19 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:43:20.301 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:43:19 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T04:43:20.302 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:43:19 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:43:20.302 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:43:19 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:43:20.302 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:43:19 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:43:20.302 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:43:19 smithi047 conmon[78521]: 2022-05-14T04:43:20.303 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:43:19 smithi047 conmon[78521]: During handling of the above exception, another exception occurred: 2022-05-14T04:43:20.303 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:43:19 smithi047 conmon[78521]: 2022-05-14T04:43:20.303 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:43:19 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T04:43:20.303 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:43:19 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T04:43:20.303 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:43:19 smithi047 conmon[78521]: return f(*arg) 2022-05-14T04:43:20.304 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:43:19 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T04:43:20.304 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:43:19 smithi047 conmon[78521]: self._write_client_files(client_files, host) 2022-05-14T04:43:20.304 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:43:19 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T04:43:20.304 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:43:19 smithi047 conmon[78521]: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T04:43:20.305 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:43:19 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T04:43:20.305 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:43:19 smithi047 conmon[78521]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T04:43:20.305 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:43:19 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T04:43:20.305 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:43:19 smithi047 conmon[78521]: return self.event_loop.get_result(coro) 2022-05-14T04:43:20.305 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:43:19 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T04:43:20.306 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:43:19 smithi047 conmon[78521]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T04:43:20.306 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:43:19 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T04:43:20.306 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:43:19 smithi047 conmon[78521]: return self.__get_result() 2022-05-14T04:43:20.306 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:43:19 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T04:43:20.307 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:43:19 smithi047 conmon[78521]: raise self._exception 2022-05-14T04:43:20.307 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:43:19 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T04:43:20.307 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:43:19 smithi047 conmon[78521]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T04:43:20.307 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:43:19 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T04:43:20.307 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:43:19 smithi047 conmon[78521]: conn = await self._remote_connection(host, addr) 2022-05-14T04:43:20.308 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:43:19 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T04:43:20.308 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:43:19 smithi047 conmon[78521]: raise 2022-05-14T04:43:20.308 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:43:19 smithi047 conmon[78521]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T04:43:20.308 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:43:19 smithi047 conmon[78521]: self.gen.throw(type, value, traceback) 2022-05-14T04:43:20.308 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:43:19 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T04:43:20.309 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:43:19 smithi047 conmon[78521]: raise OrchestratorError(msg) 2022-05-14T04:43:20.309 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:43:19 smithi047 conmon[78521]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:43:20.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:43:19 smithi115 ceph-mon[78593]: pgmap v9817: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:43:20.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:43:19 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:43:20.860 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:43:20 smithi115 conmon[90393]: 2022-05-14T04:43:20.533+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:43:21.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:43:20 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:43:21.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T04:43:21.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:43:21.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T04:43:21.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:43:21.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:43:21.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:43:21.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T04:43:21.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:43:21.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T04:43:21.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:43:21.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T04:43:21.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:43:21.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T04:43:21.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:43:21.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T04:43:21.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:43:21.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:43:21.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:43:21.297 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:43:20 smithi047 ceph-mon[77147]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T04:43:21.298 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T04:43:21.298 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:43:21.298 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T04:43:21.298 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:43:21.298 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:43:21.299 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:43:21.299 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T04:43:21.299 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:43:21.299 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T04:43:21.299 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:43:21.300 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T04:43:21.300 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:43:21.300 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T04:43:21.300 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:43:21.300 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T04:43:21.301 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:43:21.301 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:43:21.301 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:43:21.301 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T04:43:21.302 INFO:journalctl@ceph.mon.b.smithi047.stdout: During handling of the above exception, another exception occurred: 2022-05-14T04:43:21.302 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T04:43:21.302 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T04:43:21.302 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T04:43:21.302 INFO:journalctl@ceph.mon.b.smithi047.stdout: return f(*arg) 2022-05-14T04:43:21.303 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T04:43:21.303 INFO:journalctl@ceph.mon.b.smithi047.stdout: self._write_client_files(client_files, host) 2022-05-14T04:43:21.303 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T04:43:21.303 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T04:43:21.303 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T04:43:21.304 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T04:43:21.304 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T04:43:21.304 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.event_loop.get_result(coro) 2022-05-14T04:43:21.304 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T04:43:21.304 INFO:journalctl@ceph.mon.b.smithi047.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T04:43:21.305 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T04:43:21.305 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.__get_result() 2022-05-14T04:43:21.305 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T04:43:21.305 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise self._exception 2022-05-14T04:43:21.305 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T04:43:21.306 INFO:journalctl@ceph.mon.b.smithi047.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T04:43:21.306 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T04:43:21.306 INFO:journalctl@ceph.mon.b.smithi047.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T04:43:21.306 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T04:43:21.307 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise 2022-05-14T04:43:21.307 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T04:43:21.307 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.gen.throw(type, value, traceback) 2022-05-14T04:43:21.307 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T04:43:21.307 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OrchestratorError(msg) 2022-05-14T04:43:21.308 INFO:journalctl@ceph.mon.b.smithi047.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:43:21.308 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:43:20 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:43:21.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:43:20 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:43:21.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T04:43:21.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:43:21.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T04:43:21.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:43:21.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:43:21.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:43:21.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T04:43:21.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:43:21.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T04:43:21.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:43:21.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T04:43:21.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:43:21.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T04:43:21.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:43:21.365 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T04:43:21.366 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:43:21.366 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:43:21.366 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:43:21.366 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:43:20 smithi115 ceph-mon[78593]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T04:43:21.367 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T04:43:21.367 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:43:21.367 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T04:43:21.367 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:43:21.368 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:43:21.368 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:43:21.368 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T04:43:21.368 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:43:21.368 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T04:43:21.369 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:43:21.369 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T04:43:21.369 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:43:21.369 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T04:43:21.369 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:43:21.370 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T04:43:21.370 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:43:21.370 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:43:21.371 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:43:21.372 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T04:43:21.372 INFO:journalctl@ceph.mon.c.smithi115.stdout: During handling of the above exception, another exception occurred: 2022-05-14T04:43:21.372 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T04:43:21.372 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T04:43:21.372 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T04:43:21.373 INFO:journalctl@ceph.mon.c.smithi115.stdout: return f(*arg) 2022-05-14T04:43:21.373 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T04:43:21.373 INFO:journalctl@ceph.mon.c.smithi115.stdout: self._write_client_files(client_files, host) 2022-05-14T04:43:21.373 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T04:43:21.374 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T04:43:21.374 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T04:43:21.374 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T04:43:21.374 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T04:43:21.374 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.event_loop.get_result(coro) 2022-05-14T04:43:21.375 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T04:43:21.375 INFO:journalctl@ceph.mon.c.smithi115.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T04:43:21.375 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T04:43:21.375 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.__get_result() 2022-05-14T04:43:21.375 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T04:43:21.376 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise self._exception 2022-05-14T04:43:21.376 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T04:43:21.376 INFO:journalctl@ceph.mon.c.smithi115.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T04:43:21.376 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T04:43:21.376 INFO:journalctl@ceph.mon.c.smithi115.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T04:43:21.377 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T04:43:21.377 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise 2022-05-14T04:43:21.377 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T04:43:21.377 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.gen.throw(type, value, traceback) 2022-05-14T04:43:21.378 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T04:43:21.378 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OrchestratorError(msg) 2022-05-14T04:43:21.378 INFO:journalctl@ceph.mon.c.smithi115.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:43:21.378 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:43:20 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:43:21.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:43:21 smithi115 conmon[90393]: 2022-05-14T04:43:21.521+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:43:22.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:43:21 smithi047 ceph-mon[77147]: pgmap v9818: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:43:22.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:43:21 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:43:22.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:43:21 smithi115 ceph-mon[78593]: pgmap v9818: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:43:22.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:43:21 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:43:22.860 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:43:22 smithi115 conmon[90393]: 2022-05-14T04:43:22.534+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:43:23.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:43:22 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:43:23.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:43:22 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:43:23.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:43:22 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:43:23.295 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:43:22 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 18935 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:43:23.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:43:22 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:43:23.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:43:22 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:43:23.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:43:22 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:43:23.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:43:22 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 18935 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:43:23.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:43:23 smithi115 conmon[90393]: 2022-05-14T04:43:23.552+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:43:24.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:43:23 smithi047 ceph-mon[77147]: pgmap v9819: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:43:24.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:43:23 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:43:24.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:43:23 smithi115 ceph-mon[78593]: pgmap v9819: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:43:24.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:43:23 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:43:24.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:43:24 smithi115 conmon[90393]: 2022-05-14T04:43:24.537+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:43:25.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:43:24 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:43:25.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:43:24 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:43:25.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:43:25 smithi115 conmon[90393]: 2022-05-14T04:43:25.584+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:43:26.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:43:25 smithi047 ceph-mon[77147]: pgmap v9820: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:43:26.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:43:25 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:43:26.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:43:25 smithi115 ceph-mon[78593]: pgmap v9820: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:43:26.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:43:25 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:43:26.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:43:26 smithi115 conmon[90393]: 2022-05-14T04:43:26.600+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:43:27.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:43:26 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:43:27.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:43:26 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:43:27.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:43:27 smithi115 conmon[90393]: 2022-05-14T04:43:27.552+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:43:28.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:43:27 smithi047 ceph-mon[77147]: pgmap v9821: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:43:28.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:43:27 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:43:28.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:43:27 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 18940 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:43:28.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:43:27 smithi115 ceph-mon[78593]: pgmap v9821: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:43:28.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:43:27 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:43:28.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:43:27 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 18940 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:43:28.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:43:28 smithi115 conmon[90393]: 2022-05-14T04:43:28.546+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:43:29.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:43:28 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:43:29.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:43:28 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:43:29.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:43:29 smithi115 conmon[90393]: 2022-05-14T04:43:29.540+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:43:30.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:43:29 smithi047 ceph-mon[77147]: pgmap v9822: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:43:30.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:43:29 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:43:30.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:43:29 smithi115 ceph-mon[78593]: pgmap v9822: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:43:30.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:43:29 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:43:30.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:43:30 smithi115 conmon[90393]: 2022-05-14T04:43:30.519+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:43:31.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:43:30 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:43:31.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:43:30 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:43:31.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:43:30 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:43:31.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:43:30 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:43:31.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:43:30 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:43:31.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:43:30 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:43:31.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:43:31 smithi115 conmon[90393]: 2022-05-14T04:43:31.558+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:43:32.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:43:31 smithi047 ceph-mon[77147]: pgmap v9823: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:43:32.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:43:31 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:43:32.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:43:31 smithi115 ceph-mon[78593]: pgmap v9823: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:43:32.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:43:31 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:43:32.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:43:32 smithi115 conmon[90393]: 2022-05-14T04:43:32.560+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:43:33.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:43:32 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:43:33.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:43:32 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 18945 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:43:33.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:43:32 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:43:33.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:43:32 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 18945 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:43:33.861 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:43:33 smithi115 conmon[90393]: 2022-05-14T04:43:33.583+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:43:34.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:43:33 smithi047 ceph-mon[77147]: pgmap v9824: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:43:34.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:43:33 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:43:34.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:43:33 smithi115 ceph-mon[78593]: pgmap v9824: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:43:34.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:43:33 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:43:34.941 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:43:34 smithi115 conmon[90393]: 2022-05-14T04:43:34.608+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:43:35.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:43:34 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:43:35.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:43:34 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:43:35.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:43:34 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:43:35.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:43:34 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:43:35.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:43:34 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:43:35.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:43:34 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:43:35.947 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:43:35 smithi115 conmon[90393]: 2022-05-14T04:43:35.630+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:43:36.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:43:35 smithi047 ceph-mon[77147]: pgmap v9825: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:43:36.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:43:35 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:43:36.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:43:35 smithi115 ceph-mon[78593]: pgmap v9825: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:43:36.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:43:35 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:43:36.948 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:43:36 smithi115 conmon[90393]: 2022-05-14T04:43:36.643+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:43:37.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:43:36 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:43:37.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:43:36 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:43:37.953 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:43:37 smithi115 conmon[90393]: 2022-05-14T04:43:37.677+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:43:38.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:43:37 smithi047 ceph-mon[77147]: pgmap v9826: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:43:38.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:43:37 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 18950 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:43:38.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:43:37 smithi115 ceph-mon[78593]: pgmap v9826: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:43:38.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:43:37 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 18950 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:43:39.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:43:38 smithi115 conmon[90393]: 2022-05-14T04:43:38.727+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:43:39.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:43:38 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:43:39.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:43:38 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:43:39.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:43:38 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:43:39.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:43:38 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:43:39.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:43:38 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:43:39.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:43:38 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:43:39.959 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:43:39 smithi115 conmon[90393]: 2022-05-14T04:43:39.703+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:43:40.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:43:39 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:43:40.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:43:39 smithi047 ceph-mon[77147]: pgmap v9827: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:43:40.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:43:39 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:43:40.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:43:39 smithi115 ceph-mon[78593]: pgmap v9827: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:43:41.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:43:40 smithi115 conmon[90393]: 2022-05-14T04:43:40.717+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:43:41.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:43:40 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:43:41.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:43:40 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:43:42.110 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:43:41 smithi115 conmon[90393]: 2022-05-14T04:43:41.734+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:43:42.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:43:41 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:43:42.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:43:41 smithi115 ceph-mon[78593]: pgmap v9828: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:43:42.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:43:41 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:43:42.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:43:41 smithi047 ceph-mon[77147]: pgmap v9828: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:43:43.110 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:43:42 smithi115 conmon[90393]: 2022-05-14T04:43:42.728+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:43:43.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:43:42 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:43:43.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:43:42 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:43:43.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:43:42 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:43:43.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:43:42 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 18955 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:43:43.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:43:42 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:43:43.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:43:42 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:43:43.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:43:42 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:43:43.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:43:42 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 18955 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:43:44.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:43:43 smithi115 conmon[90393]: 2022-05-14T04:43:43.766+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:43:44.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:43:43 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:43:44.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:43:43 smithi115 ceph-mon[78593]: pgmap v9829: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:43:44.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:43:43 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:43:44.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:43:43 smithi047 ceph-mon[77147]: pgmap v9829: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:43:45.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:43:44 smithi115 conmon[90393]: 2022-05-14T04:43:44.767+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:43:45.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:43:44 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:43:45.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:43:44 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:43:46.110 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:43:45 smithi115 conmon[90393]: 2022-05-14T04:43:45.753+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:43:46.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:43:45 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:43:46.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:43:45 smithi115 ceph-mon[78593]: pgmap v9830: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:43:46.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:43:45 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:43:46.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:43:45 smithi047 ceph-mon[77147]: pgmap v9830: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:43:47.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:43:46 smithi115 conmon[90393]: 2022-05-14T04:43:46.745+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:43:47.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:43:46 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:43:47.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:43:46 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:43:47.989 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:43:47 smithi115 conmon[90393]: 2022-05-14T04:43:47.702+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:43:48.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:43:47 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:43:48.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:43:47 smithi047 ceph-mon[77147]: pgmap v9831: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:43:48.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:43:47 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 18960 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:43:48.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:43:47 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:43:48.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:43:47 smithi115 ceph-mon[78593]: pgmap v9831: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:43:48.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:43:47 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 18960 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:43:48.988 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:43:48 smithi115 conmon[90393]: 2022-05-14T04:43:48.654+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:43:49.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:43:48 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:43:49.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:43:48 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:43:49.995 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:43:49 smithi115 conmon[90393]: 2022-05-14T04:43:49.674+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:43:50.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:43:49 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:43:50.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:43:49 smithi047 ceph-mon[77147]: pgmap v9832: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:43:50.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:43:49 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:43:50.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:43:49 smithi115 ceph-mon[78593]: pgmap v9832: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:43:50.995 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:43:50 smithi115 conmon[90393]: 2022-05-14T04:43:50.631+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:43:51.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:43:50 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:43:51.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:43:50 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:43:51.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:43:50 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:43:51.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:43:50 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:43:51.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:43:50 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:43:51.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:43:50 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:43:51.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:43:50 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:43:51.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:43:50 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:43:51.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:43:51 smithi115 conmon[90393]: 2022-05-14T04:43:51.596+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:43:52.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:43:51 smithi047 ceph-mon[77147]: pgmap v9833: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:43:52.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:43:51 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:43:52.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:43:51 smithi115 ceph-mon[78593]: pgmap v9833: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:43:52.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:43:51 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:43:52.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:43:52 smithi115 conmon[90393]: 2022-05-14T04:43:52.601+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:43:53.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:43:52 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:43:53.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:43:52 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 18965 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:43:53.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:43:53 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:43:53.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:43:53 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 18965 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:43:53.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:43:53 smithi115 conmon[90393]: 2022-05-14T04:43:53.579+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:43:54.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:43:54 smithi047 ceph-mon[77147]: pgmap v9834: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:43:54.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:43:54 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:43:54.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:43:54 smithi115 ceph-mon[78593]: pgmap v9834: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:43:54.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:43:54 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:43:54.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:43:54 smithi115 conmon[90393]: 2022-05-14T04:43:54.597+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:43:55.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:43:55 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:43:55.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:43:55 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:43:55.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:43:55 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:43:55.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:43:55 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:43:55.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:43:55 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:43:55.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:43:55 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:43:55.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:43:55 smithi115 conmon[90393]: 2022-05-14T04:43:55.601+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:43:56.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:43:56 smithi047 ceph-mon[77147]: pgmap v9835: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:43:56.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:43:56 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:43:56.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:43:56 smithi115 ceph-mon[78593]: pgmap v9835: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:43:56.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:43:56 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:43:57.015 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:43:56 smithi115 conmon[90393]: 2022-05-14T04:43:56.637+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:43:57.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:43:57 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:43:57.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:43:57 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:43:58.020 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:43:57 smithi115 conmon[90393]: 2022-05-14T04:43:57.631+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:43:58.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:43:58 smithi047 ceph-mon[77147]: pgmap v9836: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:43:58.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:43:58 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:43:58.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:43:58 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 18970 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:43:58.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:43:58 smithi115 ceph-mon[78593]: pgmap v9836: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:43:58.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:43:58 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:43:58.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:43:58 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 18970 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:43:59.021 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:43:58 smithi115 conmon[90393]: 2022-05-14T04:43:58.610+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:43:59.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:43:59 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:43:59.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:43:59 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:43:59.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:43:59 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:43:59.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:43:59 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:43:59.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:43:59 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:43:59.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:43:59 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:43:59.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:43:59 smithi115 conmon[90393]: 2022-05-14T04:43:59.583+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:44:00.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:44:00 smithi047 ceph-mon[77147]: pgmap v9837: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:44:00.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:44:00 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:44:00.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:44:00 smithi115 ceph-mon[78593]: pgmap v9837: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:44:00.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:44:00 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:44:01.027 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:44:00 smithi115 conmon[90393]: 2022-05-14T04:44:00.633+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:44:01.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:44:01 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:44:01.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:44:01 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:44:02.088 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:44:01 smithi115 conmon[90393]: 2022-05-14T04:44:01.654+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:44:02.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:44:02 smithi115 ceph-mon[78593]: pgmap v9838: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:44:02.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:44:02 smithi047 ceph-mon[77147]: pgmap v9838: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:44:03.088 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:44:02 smithi115 conmon[90393]: 2022-05-14T04:44:02.621+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:44:03.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:44:03 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:44:03.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:44:03 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:44:03.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:44:03 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:44:03.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:44:03 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:44:03.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:44:03 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 18975 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:44:03.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:44:03 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:44:03.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:44:03 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:44:03.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:44:03 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:44:03.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:44:03 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:44:03.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:44:03 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 18975 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:44:04.088 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:44:03 smithi115 conmon[90393]: 2022-05-14T04:44:03.615+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:44:04.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:44:04 smithi115 ceph-mon[78593]: pgmap v9839: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:44:04.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:44:04 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:44:04.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:44:04 smithi047 ceph-mon[77147]: pgmap v9839: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:44:04.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:44:04 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:44:05.093 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:44:04 smithi115 conmon[90393]: 2022-05-14T04:44:04.636+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:44:05.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:44:05 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:44:05.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:44:05 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:44:06.094 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:44:05 smithi115 conmon[90393]: 2022-05-14T04:44:05.632+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:44:06.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:44:06 smithi115 ceph-mon[78593]: pgmap v9840: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:44:06.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:44:06 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:44:06.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:44:06 smithi047 ceph-mon[77147]: pgmap v9840: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:44:06.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:44:06 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:44:07.099 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:44:06 smithi115 conmon[90393]: 2022-05-14T04:44:06.632+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:44:07.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:44:07 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:44:07.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:44:07 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:44:07.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:44:07 smithi115 conmon[90393]: 2022-05-14T04:44:07.603+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:44:08.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:44:08 smithi115 ceph-mon[78593]: pgmap v9841: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:44:08.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:44:08 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:44:08.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:44:08 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 18980 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:44:08.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:44:08 smithi047 ceph-mon[77147]: pgmap v9841: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:44:08.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:44:08 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:44:08.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:44:08 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 18980 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:44:09.105 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:44:08 smithi115 conmon[90393]: 2022-05-14T04:44:08.611+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:44:09.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:44:09 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:44:09.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:44:09 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:44:09.862 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:44:09 smithi115 conmon[90393]: 2022-05-14T04:44:09.603+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:44:10.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:44:10 smithi115 ceph-mon[78593]: pgmap v9842: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:44:10.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:44:10 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:44:10.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:44:10 smithi047 ceph-mon[77147]: pgmap v9842: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:44:10.547 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:44:10 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:44:11.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:44:10 smithi115 conmon[90393]: 2022-05-14T04:44:10.652+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:44:11.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:44:11 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:44:11.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:44:11 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:44:11.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:44:11 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:44:11.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:44:11 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:44:11.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:44:11 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:44:11.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:44:11 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:44:12.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:44:11 smithi115 conmon[90393]: 2022-05-14T04:44:11.611+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:44:12.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:44:12 smithi047 ceph-mon[77147]: pgmap v9843: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:44:12.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:44:12 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:44:12.599 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:44:12 smithi115 ceph-mon[78593]: pgmap v9843: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:44:12.599 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:44:12 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:44:12.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:44:12 smithi115 conmon[90393]: 2022-05-14T04:44:12.596+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:44:13.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:44:13 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:44:13.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:44:13 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 18985 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:44:13.579 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:44:13 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:44:13.579 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:44:13 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 18985 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:44:13.861 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:44:13 smithi115 conmon[90393]: 2022-05-14T04:44:13.575+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:44:14.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:44:14 smithi047 ceph-mon[77147]: pgmap v9844: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:44:14.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:44:14 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:44:14.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:44:14 smithi115 ceph-mon[78593]: pgmap v9844: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:44:14.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:44:14 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:44:14.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:44:14 smithi115 conmon[90393]: 2022-05-14T04:44:14.603+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:44:15.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:44:15 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:44:15.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:44:15 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:44:15.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:44:15 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:44:15.545 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:44:15 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T04:44:15.545 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:44:15 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T04:44:15.558 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:44:15 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:44:15.558 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:44:15 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:44:15.558 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:44:15 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:44:15.558 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:44:15 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T04:44:15.559 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:44:15 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T04:44:15.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:44:15 smithi115 conmon[90393]: 2022-05-14T04:44:15.555+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:44:16.533 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:44:16 smithi115 ceph-mon[78593]: pgmap v9845: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:44:16.533 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:44:16 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:44:16.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:44:16 smithi047 ceph-mon[77147]: pgmap v9845: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:44:16.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:44:16 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:44:16.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:44:16 smithi115 conmon[90393]: 2022-05-14T04:44:16.530+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:44:17.529 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:44:17 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:44:17.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:44:17 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:44:17.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:44:17 smithi115 conmon[90393]: 2022-05-14T04:44:17.526+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:44:18.521 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:44:18 smithi115 ceph-mon[78593]: pgmap v9846: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:44:18.522 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:44:18 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:44:18.522 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:44:18 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 18990 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:44:18.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:44:18 smithi047 ceph-mon[77147]: pgmap v9846: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:44:18.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:44:18 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:44:18.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:44:18 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 18990 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:44:18.860 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:44:18 smithi115 conmon[90393]: 2022-05-14T04:44:18.518+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:44:19.506 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:44:19 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:44:19.507 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:44:19 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:44:19.507 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:44:19 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:44:19.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:44:19 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:44:19.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:44:19 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:44:19.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:44:19 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:44:19.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:44:19 smithi115 conmon[90393]: 2022-05-14T04:44:19.503+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:44:20.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:44:20 smithi047 ceph-mon[77147]: pgmap v9847: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:44:20.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:44:20 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:44:20.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:44:20 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T04:44:20.295 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:44:20 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T04:44:20.295 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:44:20 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T04:44:20.365 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:44:20 smithi115 ceph-mon[78593]: pgmap v9847: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:44:20.365 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:44:20 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:44:20.366 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:44:20 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T04:44:20.366 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:44:20 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T04:44:20.366 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:44:20 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T04:44:20.793 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:44:20 smithi047 conmon[78521]: 2022-05-14T04:44:20.592+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:44:20.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:44:20 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T04:44:20.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:44:20 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:44:20.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:44:20 smithi047 conmon[78521]: yield 2022-05-14T04:44:20.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:44:20 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:44:20.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:44:20 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:44:20.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:44:20 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:44:20.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:44:20 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T04:44:20.797 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:44:20 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:44:20.797 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:44:20 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T04:44:20.797 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:44:20 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:44:20.798 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:44:20 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T04:44:20.799 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:44:20 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:44:20.799 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:44:20 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T04:44:20.799 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:44:20 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:44:20.800 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:44:20 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T04:44:20.800 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:44:20 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:44:20.800 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:44:20 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:44:20.801 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:44:20 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:44:20.860 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:44:20 smithi115 conmon[90393]: 2022-05-14T04:44:20.535+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:44:21.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:44:21 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:44:21.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:44:21 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:44:21.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T04:44:21.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:44:21.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T04:44:21.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:44:21.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:44:21.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:44:21.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T04:44:21.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:44:21.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T04:44:21.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:44:21.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T04:44:21.549 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:44:21.550 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T04:44:21.550 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:44:21.550 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T04:44:21.550 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:44:21.551 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:44:21.551 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:44:21.551 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:44:21 smithi047 ceph-mon[77147]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T04:44:21.585 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:44:21 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:44:21.586 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:44:21 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:44:21.586 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T04:44:21.586 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:44:21.586 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T04:44:21.587 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:44:21.587 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:44:21.587 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:44:21.587 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T04:44:21.588 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:44:21.588 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T04:44:21.588 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:44:21.588 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T04:44:21.588 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:44:21.589 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T04:44:21.589 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:44:21.589 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T04:44:21.589 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:44:21.589 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:44:21.590 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:44:21.590 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:44:21 smithi115 ceph-mon[78593]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T04:44:21.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:44:21 smithi115 conmon[90393]: 2022-05-14T04:44:21.582+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:44:22.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:44:22 smithi047 ceph-mon[77147]: pgmap v9848: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:44:22.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:44:22 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:44:22.559 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:44:22 smithi115 ceph-mon[78593]: pgmap v9848: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:44:22.559 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:44:22 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:44:22.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:44:22 smithi115 conmon[90393]: 2022-05-14T04:44:22.556+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:44:23.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:44:23 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:44:23.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:44:23 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:44:23.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:44:23 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:44:23.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:44:23 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 18995 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:44:23.569 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:44:23 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:44:23.569 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:44:23 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:44:23.570 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:44:23 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:44:23.570 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:44:23 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 18995 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:44:23.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:44:23 smithi115 conmon[90393]: 2022-05-14T04:44:23.566+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:44:24.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:44:23 smithi047 conmon[78521]: 2022-05-14T04:44:23.663+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:44:24.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:44:23 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T04:44:24.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:44:23 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:44:24.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:44:23 smithi047 conmon[78521]: yield 2022-05-14T04:44:24.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:44:23 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:44:24.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:44:23 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:44:24.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:44:23 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:44:24.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:44:23 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T04:44:24.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:44:23 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:44:24.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:44:23 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T04:44:24.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:44:23 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:44:24.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:44:23 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T04:44:24.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:44:23 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:44:24.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:44:23 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T04:44:24.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:44:23 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:44:24.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:44:23 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T04:44:24.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:44:23 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:44:24.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:44:23 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:44:24.048 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:44:23 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:44:24.048 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:44:23 smithi047 conmon[78521]: 2022-05-14T04:44:23.664+0000 7f533ef76700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T04:44:24.048 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:44:23 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T04:44:24.048 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:44:23 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:44:24.048 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:44:23 smithi047 conmon[78521]: yield 2022-05-14T04:44:24.049 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:44:23 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:44:24.049 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:44:23 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:44:24.049 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:44:23 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:44:24.049 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:44:23 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T04:44:24.050 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:44:23 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:44:24.050 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:44:23 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T04:44:24.050 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:44:23 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:44:24.050 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:44:23 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T04:44:24.050 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:44:23 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:44:24.051 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:44:23 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T04:44:24.051 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:44:23 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:44:24.051 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:44:23 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T04:44:24.051 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:44:23 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:44:24.052 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:44:23 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:44:24.052 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:44:23 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:44:24.052 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:44:23 smithi047 conmon[78521]: 2022-05-14T04:44:24.052 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:44:23 smithi047 conmon[78521]: During handling of the above exception, another exception occurred: 2022-05-14T04:44:24.052 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:44:23 smithi047 conmon[78521]: 2022-05-14T04:44:24.053 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:44:23 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T04:44:24.053 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:44:23 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T04:44:24.053 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:44:23 smithi047 conmon[78521]: return f(*arg) 2022-05-14T04:44:24.053 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:44:23 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T04:44:24.054 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:44:23 smithi047 conmon[78521]: self._write_client_files(client_files, host) 2022-05-14T04:44:24.054 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:44:23 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T04:44:24.054 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:44:23 smithi047 conmon[78521]: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T04:44:24.054 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:44:23 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T04:44:24.054 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:44:23 smithi047 conmon[78521]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T04:44:24.055 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:44:23 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T04:44:24.055 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:44:23 smithi047 conmon[78521]: return self.event_loop.get_result(coro) 2022-05-14T04:44:24.055 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:44:23 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T04:44:24.055 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:44:23 smithi047 conmon[78521]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T04:44:24.056 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:44:23 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T04:44:24.056 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:44:23 smithi047 conmon[78521]: return self.__get_result() 2022-05-14T04:44:24.056 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:44:23 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T04:44:24.056 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:44:23 smithi047 conmon[78521]: raise self._exception 2022-05-14T04:44:24.057 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:44:23 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T04:44:24.057 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:44:23 smithi047 conmon[78521]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T04:44:24.057 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:44:23 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T04:44:24.057 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:44:23 smithi047 conmon[78521]: conn = await self._remote_connection(host, addr) 2022-05-14T04:44:24.057 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:44:23 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T04:44:24.058 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:44:23 smithi047 conmon[78521]: raise 2022-05-14T04:44:24.058 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:44:23 smithi047 conmon[78521]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T04:44:24.058 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:44:23 smithi047 conmon[78521]: self.gen.throw(type, value, traceback) 2022-05-14T04:44:24.058 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:44:23 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T04:44:24.059 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:44:23 smithi047 conmon[78521]: raise OrchestratorError(msg) 2022-05-14T04:44:24.059 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:44:23 smithi047 conmon[78521]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:44:24.528 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:44:24 smithi115 ceph-mon[78593]: pgmap v9849: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:44:24.528 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:44:24 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:44:24.529 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:44:24 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:44:24.529 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T04:44:24.529 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:44:24.529 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T04:44:24.529 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:44:24.530 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:44:24.530 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:44:24.530 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T04:44:24.530 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:44:24.531 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T04:44:24.531 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:44:24.531 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T04:44:24.531 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:44:24.531 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T04:44:24.532 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:44:24.532 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T04:44:24.532 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:44:24.532 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:44:24.532 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:44:24.533 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:44:24 smithi115 ceph-mon[78593]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T04:44:24.533 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T04:44:24.533 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:44:24.533 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T04:44:24.534 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:44:24.534 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:44:24.534 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:44:24.534 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T04:44:24.535 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:44:24.535 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T04:44:24.535 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:44:24.535 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T04:44:24.535 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:44:24.536 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T04:44:24.536 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:44:24.536 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T04:44:24.536 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:44:24.536 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:44:24.537 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:44:24.537 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T04:44:24.537 INFO:journalctl@ceph.mon.c.smithi115.stdout: During handling of the above exception, another exception occurred: 2022-05-14T04:44:24.537 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T04:44:24.537 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T04:44:24.538 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T04:44:24.538 INFO:journalctl@ceph.mon.c.smithi115.stdout: return f(*arg) 2022-05-14T04:44:24.538 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T04:44:24.538 INFO:journalctl@ceph.mon.c.smithi115.stdout: self._write_client_files(client_files, host) 2022-05-14T04:44:24.538 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T04:44:24.539 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T04:44:24.539 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T04:44:24.539 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T04:44:24.539 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T04:44:24.541 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.event_loop.get_result(coro) 2022-05-14T04:44:24.542 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T04:44:24.542 INFO:journalctl@ceph.mon.c.smithi115.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T04:44:24.542 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T04:44:24.542 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.__get_result() 2022-05-14T04:44:24.542 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T04:44:24.543 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise self._exception 2022-05-14T04:44:24.543 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T04:44:24.543 INFO:journalctl@ceph.mon.c.smithi115.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T04:44:24.543 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T04:44:24.544 INFO:journalctl@ceph.mon.c.smithi115.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T04:44:24.544 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T04:44:24.544 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise 2022-05-14T04:44:24.544 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T04:44:24.544 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.gen.throw(type, value, traceback) 2022-05-14T04:44:24.545 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T04:44:24.545 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OrchestratorError(msg) 2022-05-14T04:44:24.545 INFO:journalctl@ceph.mon.c.smithi115.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:44:24.546 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:44:24 smithi047 ceph-mon[77147]: pgmap v9849: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:44:24.546 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:44:24 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:44:24.546 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:44:24 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:44:24.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T04:44:24.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:44:24.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T04:44:24.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:44:24.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:44:24.548 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:44:24.548 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T04:44:24.548 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:44:24.548 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T04:44:24.549 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:44:24.549 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T04:44:24.549 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:44:24.550 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T04:44:24.550 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:44:24.550 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T04:44:24.551 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:44:24.551 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:44:24.552 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:44:24.553 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:44:24 smithi047 ceph-mon[77147]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T04:44:24.553 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T04:44:24.553 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:44:24.553 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T04:44:24.553 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:44:24.554 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:44:24.554 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:44:24.554 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T04:44:24.554 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:44:24.554 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T04:44:24.555 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:44:24.555 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T04:44:24.555 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:44:24.555 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T04:44:24.556 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:44:24.556 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T04:44:24.556 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:44:24.556 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:44:24.556 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:44:24.557 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T04:44:24.557 INFO:journalctl@ceph.mon.b.smithi047.stdout: During handling of the above exception, another exception occurred: 2022-05-14T04:44:24.557 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T04:44:24.557 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T04:44:24.557 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T04:44:24.558 INFO:journalctl@ceph.mon.b.smithi047.stdout: return f(*arg) 2022-05-14T04:44:24.558 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T04:44:24.558 INFO:journalctl@ceph.mon.b.smithi047.stdout: self._write_client_files(client_files, host) 2022-05-14T04:44:24.559 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T04:44:24.559 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T04:44:24.559 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T04:44:24.559 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T04:44:24.560 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T04:44:24.560 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.event_loop.get_result(coro) 2022-05-14T04:44:24.560 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T04:44:24.560 INFO:journalctl@ceph.mon.b.smithi047.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T04:44:24.561 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T04:44:24.561 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.__get_result() 2022-05-14T04:44:24.561 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T04:44:24.561 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise self._exception 2022-05-14T04:44:24.561 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T04:44:24.562 INFO:journalctl@ceph.mon.b.smithi047.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T04:44:24.562 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T04:44:24.562 INFO:journalctl@ceph.mon.b.smithi047.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T04:44:24.562 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T04:44:24.563 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise 2022-05-14T04:44:24.563 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T04:44:24.563 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.gen.throw(type, value, traceback) 2022-05-14T04:44:24.563 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T04:44:24.563 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OrchestratorError(msg) 2022-05-14T04:44:24.564 INFO:journalctl@ceph.mon.b.smithi047.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:44:24.862 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:44:24 smithi115 conmon[90393]: 2022-05-14T04:44:24.524+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:44:25.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:44:25 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:44:25.557 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:44:25 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:44:25.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:44:25 smithi115 conmon[90393]: 2022-05-14T04:44:25.554+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:44:26.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:44:26 smithi047 ceph-mon[77147]: pgmap v9850: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:44:26.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:44:26 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:44:26.550 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:44:26 smithi115 ceph-mon[78593]: pgmap v9850: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:44:26.551 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:44:26 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:44:26.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:44:26 smithi115 conmon[90393]: 2022-05-14T04:44:26.547+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:44:27.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:44:27 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:44:27.549 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:44:27 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:44:27.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:44:27 smithi115 conmon[90393]: 2022-05-14T04:44:27.546+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:44:28.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:44:28 smithi047 ceph-mon[77147]: pgmap v9851: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:44:28.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:44:28 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:44:28.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:44:28 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 19000 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:44:28.590 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:44:28 smithi115 ceph-mon[78593]: pgmap v9851: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:44:28.590 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:44:28 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:44:28.591 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:44:28 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 19000 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:44:28.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:44:28 smithi115 conmon[90393]: 2022-05-14T04:44:28.587+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:44:29.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:44:29 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:44:29.588 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:44:29 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:44:29.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:44:29 smithi115 conmon[90393]: 2022-05-14T04:44:29.585+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:44:30.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:44:30 smithi047 ceph-mon[77147]: pgmap v9852: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:44:30.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:44:30 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:44:30.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:44:30 smithi115 ceph-mon[78593]: pgmap v9852: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:44:30.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:44:30 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:44:31.110 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:44:30 smithi115 conmon[90393]: 2022-05-14T04:44:30.628+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:44:31.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:44:31 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:44:31.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:44:31 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:44:31.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:44:31 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:44:31.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:44:31 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:44:31.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:44:31 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:44:31.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:44:31 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:44:32.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:44:31 smithi115 conmon[90393]: 2022-05-14T04:44:31.627+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:44:32.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:44:32 smithi047 ceph-mon[77147]: pgmap v9853: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:44:32.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:44:32 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:44:32.583 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:44:32 smithi115 ceph-mon[78593]: pgmap v9853: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:44:32.584 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:44:32 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:44:32.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:44:32 smithi115 conmon[90393]: 2022-05-14T04:44:32.580+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:44:33.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:44:33 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:44:33.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:44:33 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 19005 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:44:33.554 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:44:33 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:44:33.554 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:44:33 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 19005 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:44:33.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:44:33 smithi115 conmon[90393]: 2022-05-14T04:44:33.551+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:44:34.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:44:34 smithi047 ceph-mon[77147]: pgmap v9854: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:44:34.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:44:34 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:44:34.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:44:34 smithi115 ceph-mon[78593]: pgmap v9854: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:44:34.546 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:44:34 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:44:34.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:44:34 smithi115 conmon[90393]: 2022-05-14T04:44:34.543+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:44:35.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:44:35 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:44:35.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:44:35 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:44:35.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:44:35 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:44:35.585 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:44:35 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:44:35.585 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:44:35 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:44:35.586 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:44:35 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:44:35.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:44:35 smithi115 conmon[90393]: 2022-05-14T04:44:35.582+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:44:36.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:44:36 smithi047 ceph-mon[77147]: pgmap v9855: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:44:36.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:44:36 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:44:36.602 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:44:36 smithi115 ceph-mon[78593]: pgmap v9855: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:44:36.602 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:44:36 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:44:36.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:44:36 smithi115 conmon[90393]: 2022-05-14T04:44:36.599+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:44:37.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:44:37 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:44:37.578 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:44:37 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:44:37.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:44:37 smithi115 conmon[90393]: 2022-05-14T04:44:37.574+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:44:38.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:44:38 smithi047 ceph-mon[77147]: pgmap v9856: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:44:38.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:44:38 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:44:38.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:44:38 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 19010 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:44:38.596 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:44:38 smithi115 ceph-mon[78593]: pgmap v9856: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:44:38.596 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:44:38 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:44:38.596 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:44:38 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 19010 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:44:38.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:44:38 smithi115 conmon[90393]: 2022-05-14T04:44:38.592+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:44:39.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:44:39 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:44:39.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:44:39 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:44:39.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:44:39 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:44:39.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:44:39 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:44:39.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:44:39 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:44:39.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:44:39 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:44:40.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:44:39 smithi115 conmon[90393]: 2022-05-14T04:44:39.608+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:44:40.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:44:40 smithi047 ceph-mon[77147]: pgmap v9857: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:44:40.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:44:40 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:44:40.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:44:40 smithi115 ceph-mon[78593]: pgmap v9857: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:44:40.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:44:40 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:44:41.110 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:44:40 smithi115 conmon[90393]: 2022-05-14T04:44:40.634+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:44:41.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:44:41 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:44:41.630 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:44:41 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:44:42.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:44:41 smithi115 conmon[90393]: 2022-05-14T04:44:41.669+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:44:42.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:44:42 smithi047 ceph-mon[77147]: pgmap v9858: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:44:42.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:44:42 smithi115 ceph-mon[78593]: pgmap v9858: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:44:43.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:44:42 smithi115 conmon[90393]: 2022-05-14T04:44:42.713+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:44:43.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:44:43 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:44:43.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:44:43 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:44:43.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:44:43 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:44:43.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:44:43 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 19015 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:44:43.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:44:43 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:44:43.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:44:43 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:44:43.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:44:43 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:44:43.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:44:43 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 19015 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:44:44.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:44:43 smithi115 conmon[90393]: 2022-05-14T04:44:43.674+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:44:44.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:44:44 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:44:44.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:44:44 smithi047 ceph-mon[77147]: pgmap v9859: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:44:44.614 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:44:44 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:44:44.615 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:44:44 smithi115 ceph-mon[78593]: pgmap v9859: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:44:45.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:44:44 smithi115 conmon[90393]: 2022-05-14T04:44:44.631+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:44:45.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:44:45 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:44:45.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:44:45 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:44:45.595 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:44:45 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:44:45.596 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:44:45 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:44:45.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:44:45 smithi115 conmon[90393]: 2022-05-14T04:44:45.593+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:44:46.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:44:46 smithi047 ceph-mon[77147]: pgmap v9860: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:44:46.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:44:46 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:44:46.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:44:46 smithi115 ceph-mon[78593]: pgmap v9860: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:44:46.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:44:46 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:44:47.110 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:44:46 smithi115 conmon[90393]: 2022-05-14T04:44:46.613+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:44:47.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:44:47 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:44:47.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:44:47 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:44:48.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:44:47 smithi115 conmon[90393]: 2022-05-14T04:44:47.658+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:44:48.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:44:48 smithi047 ceph-mon[77147]: pgmap v9861: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:44:48.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:44:48 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:44:48.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:44:48 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 19020 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:44:48.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:44:48 smithi115 ceph-mon[78593]: pgmap v9861: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:44:48.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:44:48 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:44:48.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:44:48 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 19020 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:44:49.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:44:48 smithi115 conmon[90393]: 2022-05-14T04:44:48.689+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:44:50.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:44:49 smithi115 conmon[90393]: 2022-05-14T04:44:49.729+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:44:50.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:44:50 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:44:50.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:44:50 smithi047 ceph-mon[77147]: pgmap v9862: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:44:50.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:44:50 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:44:50.545 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:44:50 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:44:50.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:44:50 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:44:50.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:44:50 smithi115 ceph-mon[78593]: pgmap v9862: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:44:50.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:44:50 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:44:50.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:44:50 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:44:51.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:44:50 smithi115 conmon[90393]: 2022-05-14T04:44:50.738+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:44:51.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:44:51 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:44:51.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:44:51 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:44:52.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:44:51 smithi115 conmon[90393]: 2022-05-14T04:44:51.712+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:44:52.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:44:52 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:44:52.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:44:52 smithi047 ceph-mon[77147]: pgmap v9863: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:44:52.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:44:52 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:44:52.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:44:52 smithi115 ceph-mon[78593]: pgmap v9863: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:44:53.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:44:52 smithi115 conmon[90393]: 2022-05-14T04:44:52.708+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:44:53.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:44:53 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:44:53.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:44:53 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 19025 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:44:53.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:44:53 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:44:53.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:44:53 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 19025 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:44:54.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:44:53 smithi115 conmon[90393]: 2022-05-14T04:44:53.705+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:44:54.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:44:54 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:44:54.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:44:54 smithi047 ceph-mon[77147]: pgmap v9864: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:44:54.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:44:54 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:44:54.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:44:54 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:44:54.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:44:54 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:44:54.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:44:54 smithi115 ceph-mon[78593]: pgmap v9864: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:44:54.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:44:54 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:44:54.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:44:54 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:44:55.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:44:54 smithi115 conmon[90393]: 2022-05-14T04:44:54.663+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:44:55.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:44:55 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:44:55.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:44:55 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:44:55.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:44:55 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:44:55.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:44:55 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:44:56.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:44:55 smithi115 conmon[90393]: 2022-05-14T04:44:55.687+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:44:56.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:44:56 smithi047 ceph-mon[77147]: pgmap v9865: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:44:56.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:44:56 smithi115 ceph-mon[78593]: pgmap v9865: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:44:57.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:44:56 smithi115 conmon[90393]: 2022-05-14T04:44:56.650+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:44:57.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:44:57 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:44:57.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:44:57 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:44:57.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:44:57 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:44:57.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:44:57 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:44:58.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:44:57 smithi115 conmon[90393]: 2022-05-14T04:44:57.673+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:44:58.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:44:58 smithi047 ceph-mon[77147]: pgmap v9866: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:44:58.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:44:58 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:44:58.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:44:58 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:44:58.545 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:44:58 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 19030 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:44:58.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:44:58 smithi115 ceph-mon[78593]: pgmap v9866: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:44:58.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:44:58 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:44:58.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:44:58 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:44:58.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:44:58 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 19030 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:44:59.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:44:58 smithi115 conmon[90393]: 2022-05-14T04:44:58.639+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:44:59.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:44:59 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:44:59.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:44:59 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:44:59.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:44:59 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:44:59.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:44:59 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:45:00.114 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:44:59 smithi115 conmon[90393]: 2022-05-14T04:44:59.642+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:45:00.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:45:00 smithi047 ceph-mon[77147]: pgmap v9867: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:45:00.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:45:00 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:45:00.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:45:00 smithi115 ceph-mon[78593]: pgmap v9867: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:45:00.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:45:00 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:45:01.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:45:00 smithi115 conmon[90393]: 2022-05-14T04:45:00.688+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:45:02.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:45:01 smithi115 conmon[90393]: 2022-05-14T04:45:01.675+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:45:02.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:45:02 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:45:02.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:45:02 smithi115 ceph-mon[78593]: pgmap v9868: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:45:02.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:45:02 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:45:02.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:45:02 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:45:02.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:45:02 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:45:02.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:45:02 smithi047 ceph-mon[77147]: pgmap v9868: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:45:02.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:45:02 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:45:02.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:45:02 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:45:03.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:45:02 smithi115 conmon[90393]: 2022-05-14T04:45:02.672+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:45:03.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:45:03 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:45:03.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:45:03 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 19035 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:45:03.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:45:03 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:45:03.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:45:03 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 19035 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:45:04.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:45:03 smithi115 conmon[90393]: 2022-05-14T04:45:03.638+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:45:04.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:45:04 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:45:04.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:45:04 smithi115 ceph-mon[78593]: pgmap v9869: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:45:04.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:45:04 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:45:04.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:45:04 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:45:04.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:45:04 smithi047 ceph-mon[77147]: pgmap v9869: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:45:04.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:45:04 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:45:05.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:45:04 smithi115 conmon[90393]: 2022-05-14T04:45:04.683+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:45:06.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:45:05 smithi115 conmon[90393]: 2022-05-14T04:45:05.699+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:45:06.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:45:06 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:45:06.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:45:06 smithi115 ceph-mon[78593]: pgmap v9870: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:45:06.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:45:06 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:45:06.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:45:06 smithi047 ceph-mon[77147]: pgmap v9870: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:45:07.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:45:06 smithi115 conmon[90393]: 2022-05-14T04:45:06.660+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:45:07.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:45:07 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:45:07.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:45:07 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:45:07.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:45:07 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:45:07.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:45:07 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:45:08.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:45:07 smithi115 conmon[90393]: 2022-05-14T04:45:07.703+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:45:08.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:45:08 smithi115 ceph-mon[78593]: pgmap v9871: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:45:08.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:45:08 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 19040 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:45:08.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:45:08 smithi047 ceph-mon[77147]: pgmap v9871: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:45:08.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:45:08 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 19040 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:45:09.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:45:08 smithi115 conmon[90393]: 2022-05-14T04:45:08.725+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:45:09.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:45:09 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:45:09.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:45:09 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:45:10.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:45:09 smithi115 conmon[90393]: 2022-05-14T04:45:09.712+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:45:10.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:45:10 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:45:10.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:45:10 smithi115 ceph-mon[78593]: pgmap v9872: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:45:10.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:45:10 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:45:10.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:45:10 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:45:10.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:45:10 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:45:10.795 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:45:10 smithi047 ceph-mon[77147]: pgmap v9872: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:45:10.795 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:45:10 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:45:10.795 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:45:10 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:45:11.110 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:45:10 smithi115 conmon[90393]: 2022-05-14T04:45:10.760+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:45:11.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:45:11 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:45:11.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:45:11 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:45:12.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:45:11 smithi115 conmon[90393]: 2022-05-14T04:45:11.790+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:45:12.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:45:12 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:45:12.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:45:12 smithi115 ceph-mon[78593]: pgmap v9873: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:45:12.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:45:12 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:45:12.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:45:12 smithi047 ceph-mon[77147]: pgmap v9873: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:45:13.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:45:12 smithi115 conmon[90393]: 2022-05-14T04:45:12.757+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:45:13.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:45:13 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:45:13.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:45:13 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 19045 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:45:13.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:45:13 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:45:13.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:45:13 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 19045 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:45:14.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:45:13 smithi115 conmon[90393]: 2022-05-14T04:45:13.726+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:45:14.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:45:14 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:45:14.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:45:14 smithi115 ceph-mon[78593]: pgmap v9874: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:45:14.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:45:14 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:45:14.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:45:14 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:45:14.632 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:45:14 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:45:14.632 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:45:14 smithi047 ceph-mon[77147]: pgmap v9874: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:45:14.632 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:45:14 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:45:14.633 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:45:14 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:45:15.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:45:14 smithi115 conmon[90393]: 2022-05-14T04:45:14.721+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:45:15.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:45:15 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:45:15.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:45:15 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T04:45:15.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:45:15 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T04:45:15.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:45:15 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:45:15.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:45:15 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T04:45:15.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:45:15 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T04:45:16.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:45:15 smithi115 conmon[90393]: 2022-05-14T04:45:15.718+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:45:16.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:45:16 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:45:16.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:45:16 smithi115 ceph-mon[78593]: pgmap v9875: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 7 op/s 2022-05-14T04:45:16.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:45:16 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:45:16.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:45:16 smithi047 ceph-mon[77147]: pgmap v9875: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 7 op/s 2022-05-14T04:45:17.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:45:16 smithi115 conmon[90393]: 2022-05-14T04:45:16.677+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:45:17.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:45:17 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:45:17.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:45:17 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:45:18.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:45:17 smithi115 conmon[90393]: 2022-05-14T04:45:17.652+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:45:18.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:45:18 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:45:18.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:45:18 smithi115 ceph-mon[78593]: pgmap v9876: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 4 op/s 2022-05-14T04:45:18.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:45:18 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:45:18.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:45:18 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:45:18.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:45:18 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:45:18.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:45:18 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 19050 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:45:18.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:45:18 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:45:18.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:45:18 smithi047 ceph-mon[77147]: pgmap v9876: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 4 op/s 2022-05-14T04:45:18.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:45:18 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:45:18.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:45:18 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:45:18.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:45:18 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:45:18.795 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:45:18 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 19050 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:45:19.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:45:18 smithi115 conmon[90393]: 2022-05-14T04:45:18.669+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:45:20.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:45:19 smithi115 conmon[90393]: 2022-05-14T04:45:19.621+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:45:20.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:45:20 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:45:20.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:45:20 smithi115 ceph-mon[78593]: pgmap v9877: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 4 op/s 2022-05-14T04:45:20.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:45:20 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:45:20.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:45:20 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:45:20.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:45:20 smithi047 ceph-mon[77147]: pgmap v9877: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 4 op/s 2022-05-14T04:45:20.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:45:20 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:45:21.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:45:20 smithi115 conmon[90393]: 2022-05-14T04:45:20.608+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:45:21.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:45:21 smithi115 conmon[90393]: 2022-05-14T04:45:21.573+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:45:21.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:45:21 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:45:21.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:45:21 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:45:22.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:45:22 smithi115 conmon[90393]: 2022-05-14T04:45:22.578+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:45:22.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:45:22 smithi115 ceph-mon[78593]: pgmap v9878: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 7 op/s 2022-05-14T04:45:22.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:45:22 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:45:22.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:45:22 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:45:22.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:45:22 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:45:22.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:45:22 smithi047 ceph-mon[77147]: pgmap v9878: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 7 op/s 2022-05-14T04:45:22.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:45:22 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:45:22.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:45:22 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:45:22.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:45:22 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:45:23.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:45:23 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:45:23.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:45:23 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 19055 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:45:23.715 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:45:23 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:45:23.715 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:45:23 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 19055 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:45:24.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:45:23 smithi115 conmon[90393]: 2022-05-14T04:45:23.619+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:45:24.487 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:45:24 smithi047 ceph-mon[77147]: pgmap v9879: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 4 op/s 2022-05-14T04:45:24.487 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:45:24 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:45:24.487 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:45:24 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T04:45:24.488 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:45:24 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T04:45:24.488 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:45:24 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T04:45:24.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:45:24 smithi115 ceph-mon[78593]: pgmap v9879: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 4 op/s 2022-05-14T04:45:24.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:45:24 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:45:24.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:45:24 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T04:45:24.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:45:24 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T04:45:24.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:45:24 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T04:45:25.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:45:24 smithi115 conmon[90393]: 2022-05-14T04:45:24.650+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:45:25.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:45:25 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:45:25.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:45:25 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:45:26.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:45:25 smithi115 conmon[90393]: 2022-05-14T04:45:25.642+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:45:26.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:45:26 smithi115 ceph-mon[78593]: pgmap v9880: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 7 op/s 2022-05-14T04:45:26.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:45:26 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:45:26.772 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:45:26 smithi047 ceph-mon[77147]: pgmap v9880: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 7 op/s 2022-05-14T04:45:26.773 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:45:26 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:45:27.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:45:26 smithi047 conmon[78521]: 2022-05-14T04:45:26.767+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:45:27.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:45:26 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T04:45:27.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:45:26 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:45:27.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:45:26 smithi047 conmon[78521]: yield 2022-05-14T04:45:27.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:45:26 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:45:27.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:45:26 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:45:27.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:45:26 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:45:27.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:45:26 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T04:45:27.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:45:26 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:45:27.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:45:26 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T04:45:27.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:45:26 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:45:27.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:45:26 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T04:45:27.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:45:26 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:45:27.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:45:26 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T04:45:27.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:45:26 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:45:27.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:45:26 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T04:45:27.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:45:26 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:45:27.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:45:26 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:45:27.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:45:26 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:45:27.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:45:26 smithi115 conmon[90393]: 2022-05-14T04:45:26.652+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:45:27.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:45:27 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:45:27.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:45:27 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:45:28.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:45:27 smithi115 conmon[90393]: 2022-05-14T04:45:27.629+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:45:28.627 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:45:28 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:45:28.627 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T04:45:28.628 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:45:28.628 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T04:45:28.628 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:45:28.628 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:45:28.629 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:45:28.629 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T04:45:28.629 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:45:28.629 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T04:45:28.629 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:45:28.630 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T04:45:28.630 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:45:28.630 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T04:45:28.630 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:45:28.631 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T04:45:28.631 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:45:28.631 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:45:28.631 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:45:28.632 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:45:28 smithi115 ceph-mon[78593]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T04:45:28.632 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:45:28 smithi115 ceph-mon[78593]: pgmap v9881: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:45:28.632 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:45:28 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:45:28.632 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:45:28 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 19060 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:45:28.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:45:28 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:45:28.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T04:45:28.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:45:28.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T04:45:28.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:45:28.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:45:28.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:45:28.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T04:45:28.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:45:28.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T04:45:28.797 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:45:28.797 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T04:45:28.798 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:45:28.798 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T04:45:28.798 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:45:28.798 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T04:45:28.799 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:45:28.799 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:45:28.799 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:45:28.799 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:45:28 smithi047 ceph-mon[77147]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T04:45:28.800 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:45:28 smithi047 ceph-mon[77147]: pgmap v9881: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:45:28.800 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:45:28 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:45:28.800 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:45:28 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 19060 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:45:29.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:45:28 smithi115 conmon[90393]: 2022-05-14T04:45:28.624+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:45:29.629 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:45:29 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:45:29.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:45:29 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:45:30.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:45:29 smithi115 conmon[90393]: 2022-05-14T04:45:29.626+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:45:30.293 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:45:29 smithi047 conmon[78521]: 2022-05-14T04:45:29.840+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:45:30.293 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:45:29 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T04:45:30.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:45:29 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:45:30.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:45:29 smithi047 conmon[78521]: yield 2022-05-14T04:45:30.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:45:29 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:45:30.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:45:29 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:45:30.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:45:29 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:45:30.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:45:29 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T04:45:30.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:45:29 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:45:30.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:45:29 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T04:45:30.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:45:29 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:45:30.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:45:29 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T04:45:30.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:45:29 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:45:30.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:45:29 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T04:45:30.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:45:29 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:45:30.297 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:45:29 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T04:45:30.297 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:45:29 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:45:30.297 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:45:29 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:45:30.297 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:45:29 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:45:30.297 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:45:29 smithi047 conmon[78521]: 2022-05-14T04:45:29.840+0000 7f5344781700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T04:45:30.298 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:45:29 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T04:45:30.298 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:45:29 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:45:30.298 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:45:29 smithi047 conmon[78521]: yield 2022-05-14T04:45:30.299 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:45:29 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:45:30.299 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:45:29 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:45:30.299 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:45:29 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:45:30.299 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:45:29 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T04:45:30.300 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:45:29 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:45:30.300 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:45:29 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T04:45:30.300 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:45:29 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:45:30.300 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:45:29 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T04:45:30.301 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:45:29 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:45:30.301 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:45:29 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T04:45:30.301 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:45:29 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:45:30.301 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:45:29 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T04:45:30.302 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:45:29 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:45:30.302 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:45:29 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:45:30.302 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:45:29 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:45:30.302 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:45:29 smithi047 conmon[78521]: 2022-05-14T04:45:30.303 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:45:29 smithi047 conmon[78521]: During handling of the above exception, another exception occurred: 2022-05-14T04:45:30.303 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:45:29 smithi047 conmon[78521]: 2022-05-14T04:45:30.303 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:45:29 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T04:45:30.303 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:45:29 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T04:45:30.303 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:45:29 smithi047 conmon[78521]: return f(*arg) 2022-05-14T04:45:30.304 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:45:29 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T04:45:30.304 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:45:29 smithi047 conmon[78521]: self._write_client_files(client_files, host) 2022-05-14T04:45:30.304 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:45:29 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T04:45:30.304 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:45:29 smithi047 conmon[78521]: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T04:45:30.305 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:45:29 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T04:45:30.305 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:45:29 smithi047 conmon[78521]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T04:45:30.305 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:45:29 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T04:45:30.305 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:45:29 smithi047 conmon[78521]: return self.event_loop.get_result(coro) 2022-05-14T04:45:30.305 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:45:29 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T04:45:30.306 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:45:29 smithi047 conmon[78521]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T04:45:30.306 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:45:29 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T04:45:30.306 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:45:29 smithi047 conmon[78521]: return self.__get_result() 2022-05-14T04:45:30.306 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:45:29 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T04:45:30.306 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:45:29 smithi047 conmon[78521]: raise self._exception 2022-05-14T04:45:30.307 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:45:29 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T04:45:30.307 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:45:29 smithi047 conmon[78521]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T04:45:30.307 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:45:29 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T04:45:30.307 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:45:29 smithi047 conmon[78521]: conn = await self._remote_connection(host, addr) 2022-05-14T04:45:30.307 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:45:29 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T04:45:30.308 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:45:29 smithi047 conmon[78521]: raise 2022-05-14T04:45:30.308 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:45:29 smithi047 conmon[78521]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T04:45:30.308 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:45:29 smithi047 conmon[78521]: self.gen.throw(type, value, traceback) 2022-05-14T04:45:30.308 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:45:29 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T04:45:30.309 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:45:29 smithi047 conmon[78521]: raise OrchestratorError(msg) 2022-05-14T04:45:30.309 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:45:29 smithi047 conmon[78521]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:45:30.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:45:30 smithi047 ceph-mon[77147]: pgmap v9882: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:45:30.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:45:30 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:45:30.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:45:30 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:45:30.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:45:30 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:45:30.862 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:45:30 smithi115 conmon[90393]: 2022-05-14T04:45:30.611+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:45:30.863 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:45:30 smithi115 ceph-mon[78593]: pgmap v9882: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:45:30.863 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:45:30 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:45:30.863 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:45:30 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:45:30.863 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:45:30 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:45:31.658 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:45:31 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:45:31.659 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T04:45:31.659 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:45:31.659 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T04:45:31.660 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:45:31.660 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:45:31.660 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:45:31.660 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T04:45:31.661 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:45:31.661 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T04:45:31.661 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:45:31.661 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T04:45:31.662 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:45:31.662 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T04:45:31.662 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:45:31.662 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T04:45:31.663 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:45:31.663 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:45:31.663 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:45:31.663 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:45:31 smithi115 ceph-mon[78593]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T04:45:31.663 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T04:45:31.664 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:45:31.664 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T04:45:31.664 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:45:31.664 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:45:31.665 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:45:31.665 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T04:45:31.669 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:45:31.669 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T04:45:31.670 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:45:31.670 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T04:45:31.670 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:45:31.670 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T04:45:31.671 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:45:31.671 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T04:45:31.671 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:45:31.671 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:45:31.671 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:45:31.672 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T04:45:31.672 INFO:journalctl@ceph.mon.c.smithi115.stdout: During handling of the above exception, another exception occurred: 2022-05-14T04:45:31.672 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T04:45:31.672 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T04:45:31.672 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T04:45:31.673 INFO:journalctl@ceph.mon.c.smithi115.stdout: return f(*arg) 2022-05-14T04:45:31.673 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T04:45:31.673 INFO:journalctl@ceph.mon.c.smithi115.stdout: self._write_client_files(client_files, host) 2022-05-14T04:45:31.673 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T04:45:31.674 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T04:45:31.674 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T04:45:31.674 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T04:45:31.674 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T04:45:31.674 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.event_loop.get_result(coro) 2022-05-14T04:45:31.675 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T04:45:31.675 INFO:journalctl@ceph.mon.c.smithi115.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T04:45:31.675 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T04:45:31.675 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.__get_result() 2022-05-14T04:45:31.676 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T04:45:31.676 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise self._exception 2022-05-14T04:45:31.676 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T04:45:31.676 INFO:journalctl@ceph.mon.c.smithi115.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T04:45:31.676 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T04:45:31.677 INFO:journalctl@ceph.mon.c.smithi115.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T04:45:31.677 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T04:45:31.677 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise 2022-05-14T04:45:31.677 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T04:45:31.677 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.gen.throw(type, value, traceback) 2022-05-14T04:45:31.678 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T04:45:31.678 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OrchestratorError(msg) 2022-05-14T04:45:31.678 INFO:journalctl@ceph.mon.c.smithi115.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:45:31.678 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:45:31 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:45:31.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:45:31 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:45:31.793 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T04:45:31.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:45:31.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T04:45:31.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:45:31.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:45:31.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:45:31.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T04:45:31.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:45:31.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T04:45:31.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:45:31.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T04:45:31.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:45:31.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T04:45:31.797 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:45:31.797 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T04:45:31.797 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:45:31.797 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:45:31.797 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:45:31.798 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:45:31 smithi047 ceph-mon[77147]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T04:45:31.798 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T04:45:31.798 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:45:31.798 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T04:45:31.799 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:45:31.799 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:45:31.799 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:45:31.799 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T04:45:31.799 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:45:31.800 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T04:45:31.800 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:45:31.800 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T04:45:31.800 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:45:31.801 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T04:45:31.801 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:45:31.801 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T04:45:31.801 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:45:31.802 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:45:31.802 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:45:31.802 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T04:45:31.802 INFO:journalctl@ceph.mon.b.smithi047.stdout: During handling of the above exception, another exception occurred: 2022-05-14T04:45:31.802 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T04:45:31.803 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T04:45:31.803 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T04:45:31.803 INFO:journalctl@ceph.mon.b.smithi047.stdout: return f(*arg) 2022-05-14T04:45:31.803 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T04:45:31.804 INFO:journalctl@ceph.mon.b.smithi047.stdout: self._write_client_files(client_files, host) 2022-05-14T04:45:31.804 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T04:45:31.805 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T04:45:31.805 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T04:45:31.805 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T04:45:31.805 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T04:45:31.806 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.event_loop.get_result(coro) 2022-05-14T04:45:31.806 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T04:45:31.806 INFO:journalctl@ceph.mon.b.smithi047.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T04:45:31.806 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T04:45:31.806 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.__get_result() 2022-05-14T04:45:31.807 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T04:45:31.807 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise self._exception 2022-05-14T04:45:31.807 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T04:45:31.807 INFO:journalctl@ceph.mon.b.smithi047.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T04:45:31.808 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T04:45:31.808 INFO:journalctl@ceph.mon.b.smithi047.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T04:45:31.808 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T04:45:31.808 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise 2022-05-14T04:45:31.808 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T04:45:31.809 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.gen.throw(type, value, traceback) 2022-05-14T04:45:31.809 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T04:45:31.809 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OrchestratorError(msg) 2022-05-14T04:45:31.809 INFO:journalctl@ceph.mon.b.smithi047.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:45:31.810 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:45:31 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:45:32.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:45:31 smithi115 conmon[90393]: 2022-05-14T04:45:31.655+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:45:32.702 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:45:32 smithi115 ceph-mon[78593]: pgmap v9883: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:45:32.702 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:45:32 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:45:32.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:45:32 smithi047 ceph-mon[77147]: pgmap v9883: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:45:32.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:45:32 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:45:33.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:45:32 smithi115 conmon[90393]: 2022-05-14T04:45:32.698+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:45:33.698 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:45:33 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 19065 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:45:33.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:45:33 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 19065 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:45:34.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:45:33 smithi115 conmon[90393]: 2022-05-14T04:45:33.695+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:45:34.725 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:45:34 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:45:34.726 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:45:34 smithi115 ceph-mon[78593]: pgmap v9884: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:45:34.726 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:45:34 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:45:34.726 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:45:34 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:45:34.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:45:34 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:45:34.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:45:34 smithi047 ceph-mon[77147]: pgmap v9884: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:45:34.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:45:34 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:45:34.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:45:34 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:45:35.111 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:45:34 smithi115 conmon[90393]: 2022-05-14T04:45:34.722+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:45:35.718 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:45:35 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:45:35.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:45:35 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:45:36.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:45:35 smithi115 conmon[90393]: 2022-05-14T04:45:35.715+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:45:36.753 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:45:36 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:45:36.754 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:45:36 smithi115 ceph-mon[78593]: pgmap v9885: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:45:36.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:45:36 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:45:36.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:45:36 smithi047 ceph-mon[77147]: pgmap v9885: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:45:37.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:45:36 smithi115 conmon[90393]: 2022-05-14T04:45:36.751+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:45:37.708 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:45:37 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:45:37.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:45:37 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:45:38.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:45:37 smithi115 conmon[90393]: 2022-05-14T04:45:37.705+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:45:38.683 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:45:38 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:45:38.683 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:45:38 smithi115 ceph-mon[78593]: pgmap v9886: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:45:38.684 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:45:38 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:45:38.684 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:45:38 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:45:38.684 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:45:38 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 19070 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:45:38.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:45:38 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:45:38.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:45:38 smithi047 ceph-mon[77147]: pgmap v9886: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:45:38.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:45:38 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:45:38.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:45:38 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:45:38.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:45:38 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 19070 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:45:39.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:45:38 smithi115 conmon[90393]: 2022-05-14T04:45:38.680+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:45:39.727 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:45:39 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:45:39.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:45:39 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:45:40.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:45:39 smithi115 conmon[90393]: 2022-05-14T04:45:39.724+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:45:40.741 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:45:40 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:45:40.742 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:45:40 smithi115 ceph-mon[78593]: pgmap v9887: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:45:40.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:45:40 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:45:40.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:45:40 smithi047 ceph-mon[77147]: pgmap v9887: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:45:41.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:45:40 smithi115 conmon[90393]: 2022-05-14T04:45:40.739+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:45:41.710 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:45:41 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:45:41.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:45:41 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:45:42.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:45:41 smithi115 conmon[90393]: 2022-05-14T04:45:41.694+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:45:42.689 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:45:42 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:45:42.689 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:45:42 smithi115 ceph-mon[78593]: pgmap v9888: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:45:42.690 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:45:42 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:45:42.690 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:45:42 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:45:42.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:45:42 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:45:42.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:45:42 smithi047 ceph-mon[77147]: pgmap v9888: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:45:42.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:45:42 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:45:42.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:45:42 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:45:43.110 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:45:42 smithi115 conmon[90393]: 2022-05-14T04:45:42.686+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:45:43.665 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:45:43 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:45:43.665 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:45:43 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 19075 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:45:43.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:45:43 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:45:43.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:45:43 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 19075 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:45:44.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:45:43 smithi115 conmon[90393]: 2022-05-14T04:45:43.662+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:45:44.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:45:44 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:45:44.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:45:44 smithi047 ceph-mon[77147]: pgmap v9889: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:45:44.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:45:44 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:45:44.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:45:44 smithi115 conmon[90393]: 2022-05-14T04:45:44.633+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:45:44.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:45:44 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:45:44.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:45:44 smithi115 ceph-mon[78593]: pgmap v9889: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:45:44.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:45:44 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:45:45.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:45:45 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:45:45.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:45:45 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:45:45.860 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:45:45 smithi115 conmon[90393]: 2022-05-14T04:45:45.630+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:45:46.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:45:46 smithi047 ceph-mon[77147]: pgmap v9890: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:45:46.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:45:46 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:45:46.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:45:46 smithi115 ceph-mon[78593]: pgmap v9890: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:45:46.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:45:46 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:45:46.861 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:45:46 smithi115 conmon[90393]: 2022-05-14T04:45:46.656+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:45:47.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:45:47 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:45:47.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:45:47 smithi115 conmon[90393]: 2022-05-14T04:45:47.651+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:45:47.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:45:47 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:45:48.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:45:48 smithi047 ceph-mon[77147]: pgmap v9891: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:45:48.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:45:48 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:45:48.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:45:48 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 19080 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:45:48.860 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:45:48 smithi115 conmon[90393]: 2022-05-14T04:45:48.640+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:45:48.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:45:48 smithi115 ceph-mon[78593]: pgmap v9891: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:45:48.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:45:48 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:45:48.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:45:48 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 19080 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:45:49.681 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:45:49 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:45:49.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:45:49 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:45:50.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:45:49 smithi115 conmon[90393]: 2022-05-14T04:45:49.679+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:45:50.705 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:45:50 smithi115 ceph-mon[78593]: pgmap v9892: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:45:50.705 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:45:50 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:45:50.706 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:45:50 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:45:50.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:45:50 smithi047 ceph-mon[77147]: pgmap v9892: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:45:50.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:45:50 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:45:50.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:45:50 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:45:51.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:45:50 smithi115 conmon[90393]: 2022-05-14T04:45:50.703+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:45:51.687 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:45:51 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:45:51.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:45:51 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:45:52.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:45:51 smithi115 conmon[90393]: 2022-05-14T04:45:51.684+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:45:52.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:45:52 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:45:52.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:45:52 smithi047 ceph-mon[77147]: pgmap v9893: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:45:52.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:45:52 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:45:52.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:45:52 smithi115 ceph-mon[78593]: pgmap v9893: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:45:52.860 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:45:52 smithi115 conmon[90393]: 2022-05-14T04:45:52.651+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:45:53.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:45:53 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:45:53.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:45:53 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:45:53.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:45:53 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 19085 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:45:53.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:45:53 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:45:53.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:45:53 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:45:53.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:45:53 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 19085 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:45:53.860 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:45:53 smithi115 conmon[90393]: 2022-05-14T04:45:53.675+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:45:54.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:45:54 smithi047 ceph-mon[77147]: pgmap v9894: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:45:54.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:45:54 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:45:54.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:45:54 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:45:54.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:45:54 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:45:54.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:45:54 smithi115 ceph-mon[78593]: pgmap v9894: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:45:54.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:45:54 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:45:54.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:45:54 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:45:54.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:45:54 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:45:54.861 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:45:54 smithi115 conmon[90393]: 2022-05-14T04:45:54.675+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:45:55.714 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:45:55 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:45:55.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:45:55 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:45:56.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:45:55 smithi115 conmon[90393]: 2022-05-14T04:45:55.712+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:45:56.711 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:45:56 smithi115 ceph-mon[78593]: pgmap v9895: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:45:56.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:45:56 smithi047 ceph-mon[77147]: pgmap v9895: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:45:57.111 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:45:56 smithi115 conmon[90393]: 2022-05-14T04:45:56.709+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:45:57.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:45:57 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:45:57.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:45:57 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:45:57.860 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:45:57 smithi115 conmon[90393]: 2022-05-14T04:45:57.671+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:45:58.698 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:45:58 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:45:58.698 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:45:58 smithi115 ceph-mon[78593]: pgmap v9896: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:45:58.699 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:45:58 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:45:58.699 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:45:58 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:45:58.700 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:45:58 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:45:58.700 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:45:58 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 19090 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:45:58.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:45:58 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:45:58.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:45:58 smithi047 ceph-mon[77147]: pgmap v9896: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:45:58.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:45:58 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:45:58.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:45:58 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:45:58.795 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:45:58 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:45:58.795 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:45:58 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 19090 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:45:59.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:45:58 smithi115 conmon[90393]: 2022-05-14T04:45:58.694+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:46:00.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:45:59 smithi115 conmon[90393]: 2022-05-14T04:45:59.681+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:46:00.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:46:00 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:46:00.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:46:00 smithi047 ceph-mon[77147]: pgmap v9897: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:46:00.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:46:00 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:46:00.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:46:00 smithi115 ceph-mon[78593]: pgmap v9897: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:46:00.860 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:46:00 smithi115 conmon[90393]: 2022-05-14T04:46:00.672+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:46:01.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:46:01 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:46:01.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:46:01 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:46:01.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:46:01 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:46:01.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:46:01 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:46:01.797 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:46:01 smithi115 conmon[90393]: 2022-05-14T04:46:01.646+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:46:02.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:46:02 smithi047 ceph-mon[77147]: pgmap v9898: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:46:02.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:46:02 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:46:02.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:46:02 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:46:02.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:46:02 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:46:02.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:46:02 smithi115 ceph-mon[78593]: pgmap v9898: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:46:02.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:46:02 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:46:02.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:46:02 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:46:02.863 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:46:02 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:46:02.863 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:46:02 smithi115 conmon[90393]: 2022-05-14T04:46:02.656+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:46:03.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:46:03 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:46:03.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:46:03 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 19095 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:46:03.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:46:03 smithi115 conmon[90393]: 2022-05-14T04:46:03.639+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:46:03.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:46:03 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:46:03.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:46:03 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 19095 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:46:04.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:46:04 smithi047 ceph-mon[77147]: pgmap v9899: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:46:04.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:46:04 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:46:04.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:46:04 smithi115 conmon[90393]: 2022-05-14T04:46:04.634+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:46:04.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:46:04 smithi115 ceph-mon[78593]: pgmap v9899: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:46:04.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:46:04 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:46:05.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:46:05 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:46:05.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:46:05 smithi115 conmon[90393]: 2022-05-14T04:46:05.601+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:46:05.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:46:05 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:46:06.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:46:06 smithi047 ceph-mon[77147]: pgmap v9900: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:46:06.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:46:06 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:46:06.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:46:06 smithi115 conmon[90393]: 2022-05-14T04:46:06.605+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:46:06.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:46:06 smithi115 ceph-mon[78593]: pgmap v9900: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:46:06.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:46:06 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:46:07.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:46:07 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:46:07.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:46:07 smithi115 conmon[90393]: 2022-05-14T04:46:07.647+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:46:07.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:46:07 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:46:08.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:46:08 smithi047 ceph-mon[77147]: pgmap v9901: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:46:08.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:46:08 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:46:08.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:46:08 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 19100 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:46:08.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:46:08 smithi115 conmon[90393]: 2022-05-14T04:46:08.644+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:46:08.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:46:08 smithi115 ceph-mon[78593]: pgmap v9901: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:46:08.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:46:08 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:46:08.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:46:08 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 19100 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:46:09.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:46:09 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:46:09.860 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:46:09 smithi115 conmon[90393]: 2022-05-14T04:46:09.690+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:46:09.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:46:09 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:46:10.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:46:10 smithi047 ceph-mon[77147]: pgmap v9902: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:46:10.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:46:10 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:46:10.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:46:10 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:46:10.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:46:10 smithi115 conmon[90393]: 2022-05-14T04:46:10.653+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:46:10.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:46:10 smithi115 ceph-mon[78593]: pgmap v9902: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:46:10.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:46:10 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:46:10.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:46:10 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:46:11.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:46:11 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:46:11.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:46:11 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:46:11.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:46:11 smithi115 conmon[90393]: 2022-05-14T04:46:11.674+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:46:11.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:46:11 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:46:11.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:46:11 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:46:12.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:46:12 smithi047 ceph-mon[77147]: pgmap v9903: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:46:12.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:46:12 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:46:12.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:46:12 smithi115 conmon[90393]: 2022-05-14T04:46:12.693+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:46:12.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:46:12 smithi115 ceph-mon[78593]: pgmap v9903: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:46:12.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:46:12 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:46:13.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:46:13 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 19105 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:46:13.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:46:13 smithi115 conmon[90393]: 2022-05-14T04:46:13.695+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:46:13.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:46:13 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 19105 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:46:14.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:46:14 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:46:14.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:46:14 smithi047 ceph-mon[77147]: pgmap v9904: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:46:14.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:46:14 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:46:14.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:46:14 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:46:14.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:46:14 smithi115 conmon[90393]: 2022-05-14T04:46:14.732+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:46:14.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:46:14 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:46:14.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:46:14 smithi115 ceph-mon[78593]: pgmap v9904: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:46:14.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:46:14 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:46:14.862 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:46:14 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:46:15.798 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:46:15 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:46:15.798 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:46:15 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T04:46:15.798 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:46:15 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T04:46:15.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:46:15 smithi115 conmon[90393]: 2022-05-14T04:46:15.698+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:46:15.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:46:15 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:46:15.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:46:15 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T04:46:15.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:46:15 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T04:46:16.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:46:16 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:46:16.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:46:16 smithi047 ceph-mon[77147]: pgmap v9905: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:46:16.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:46:16 smithi115 conmon[90393]: 2022-05-14T04:46:16.734+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:46:16.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:46:16 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:46:16.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:46:16 smithi115 ceph-mon[78593]: pgmap v9905: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:46:17.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:46:17 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:46:17.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:46:17 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:46:17.860 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:46:17 smithi115 conmon[90393]: 2022-05-14T04:46:17.734+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:46:18.757 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:46:18 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:46:18.758 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:46:18 smithi115 ceph-mon[78593]: pgmap v9906: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:46:18.758 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:46:18 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:46:18.758 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:46:18 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:46:18.758 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:46:18 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 19110 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:46:18.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:46:18 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:46:18.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:46:18 smithi047 ceph-mon[77147]: pgmap v9906: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:46:18.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:46:18 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:46:18.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:46:18 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:46:18.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:46:18 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 19110 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:46:19.110 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:46:18 smithi115 conmon[90393]: 2022-05-14T04:46:18.754+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:46:19.766 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:46:19 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:46:19.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:46:19 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:46:20.110 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:46:19 smithi115 conmon[90393]: 2022-05-14T04:46:19.763+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:46:20.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:46:20 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:46:20.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:46:20 smithi047 ceph-mon[77147]: pgmap v9907: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:46:20.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:46:20 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:46:20.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:46:20 smithi115 ceph-mon[78593]: pgmap v9907: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:46:21.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:46:20 smithi115 conmon[90393]: 2022-05-14T04:46:20.793+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:46:21.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:46:21 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:46:21.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:46:21 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:46:21.861 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:46:21 smithi115 conmon[90393]: 2022-05-14T04:46:21.750+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:46:22.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:46:22 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:46:22.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:46:22 smithi047 ceph-mon[77147]: pgmap v9908: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:46:22.798 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:46:22 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:46:22.798 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:46:22 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:46:22.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:46:22 smithi115 conmon[90393]: 2022-05-14T04:46:22.760+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:46:22.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:46:22 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:46:22.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:46:22 smithi115 ceph-mon[78593]: pgmap v9908: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:46:22.862 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:46:22 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:46:22.862 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:46:22 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:46:23.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:46:23 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:46:23.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:46:23 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 19115 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:46:23.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:46:23 smithi115 conmon[90393]: 2022-05-14T04:46:23.730+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:46:23.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:46:23 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:46:23.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:46:23 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 19115 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:46:24.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:46:24 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:46:24.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:46:24 smithi047 ceph-mon[77147]: pgmap v9909: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:46:24.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:46:24 smithi115 conmon[90393]: 2022-05-14T04:46:24.683+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:46:24.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:46:24 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:46:24.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:46:24 smithi115 ceph-mon[78593]: pgmap v9909: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:46:25.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:46:25 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:46:25.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:46:25 smithi115 conmon[90393]: 2022-05-14T04:46:25.679+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:46:25.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:46:25 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:46:26.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:46:26 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:46:26.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:46:26 smithi047 ceph-mon[77147]: pgmap v9910: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:46:26.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:46:26 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:46:26.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:46:26 smithi115 conmon[90393]: 2022-05-14T04:46:26.697+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:46:26.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:46:26 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:46:26.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:46:26 smithi115 ceph-mon[78593]: pgmap v9910: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:46:26.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:46:26 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:46:28.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:46:27 smithi115 conmon[90393]: 2022-05-14T04:46:27.728+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:46:28.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:46:28 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:46:28.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:46:28 smithi047 ceph-mon[77147]: pgmap v9911: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:46:28.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:46:28 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 19120 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:46:28.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:46:28 smithi115 conmon[90393]: 2022-05-14T04:46:28.697+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:46:28.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:46:28 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:46:28.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:46:28 smithi115 ceph-mon[78593]: pgmap v9911: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:46:28.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:46:28 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 19120 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:46:29.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:46:29 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:46:29.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:46:29 smithi115 conmon[90393]: 2022-05-14T04:46:29.729+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:46:29.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:46:29 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:46:30.769 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:46:30 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:46:30.769 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:46:30 smithi115 ceph-mon[78593]: pgmap v9912: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:46:30.770 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:46:30 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:46:30.770 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:46:30 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:46:30.771 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:46:30 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T04:46:30.771 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:46:30 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T04:46:30.771 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:46:30 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T04:46:30.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:46:30 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:46:30.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:46:30 smithi047 ceph-mon[77147]: pgmap v9912: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:46:30.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:46:30 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:46:30.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:46:30 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:46:30.795 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:46:30 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T04:46:30.795 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:46:30 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T04:46:30.796 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:46:30 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T04:46:30.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:46:30 smithi047 conmon[78521]: 2022-05-14T04:46:30.320+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:46:30.797 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:46:30 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T04:46:30.797 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:46:30 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:46:30.798 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:46:30 smithi047 conmon[78521]: yield 2022-05-14T04:46:30.798 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:46:30 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:46:30.798 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:46:30 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:46:30.799 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:46:30 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:46:30.799 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:46:30 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T04:46:30.800 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:46:30 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:46:30.800 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:46:30 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T04:46:30.800 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:46:30 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:46:30.801 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:46:30 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T04:46:30.802 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:46:30 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:46:30.802 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:46:30 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T04:46:30.802 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:46:30 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:46:30.803 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:46:30 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T04:46:30.803 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:46:30 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:46:30.804 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:46:30 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:46:30.805 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:46:30 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:46:31.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:46:30 smithi115 conmon[90393]: 2022-05-14T04:46:30.766+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:46:31.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:46:31 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:46:31.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:46:31 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:46:31.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T04:46:31.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:46:31.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T04:46:31.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:46:31.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:46:31.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:46:31.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T04:46:31.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:46:31.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T04:46:31.797 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:46:31.797 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T04:46:31.797 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:46:31.797 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T04:46:31.798 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:46:31.798 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T04:46:31.799 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:46:31.799 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:46:31.799 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:46:31.800 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:46:31 smithi047 ceph-mon[77147]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T04:46:31.860 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:46:31 smithi115 conmon[90393]: 2022-05-14T04:46:31.728+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:46:31.862 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:46:31 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:46:31.862 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:46:31 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:46:31.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T04:46:31.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:46:31.863 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T04:46:31.863 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:46:31.863 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:46:31.863 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:46:31.863 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T04:46:31.864 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:46:31.864 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T04:46:31.864 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:46:31.864 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T04:46:31.865 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:46:31.865 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T04:46:31.865 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:46:31.865 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T04:46:31.866 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:46:31.866 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:46:31.866 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:46:31.866 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:46:31 smithi115 ceph-mon[78593]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T04:46:32.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:46:32 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:46:32.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:46:32 smithi047 ceph-mon[77147]: pgmap v9913: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:46:32.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:46:32 smithi115 conmon[90393]: 2022-05-14T04:46:32.740+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:46:32.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:46:32 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:46:32.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:46:32 smithi115 ceph-mon[78593]: pgmap v9913: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:46:33.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:46:33 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:46:33.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:46:33 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 19125 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:46:33.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:46:33 smithi047 conmon[78521]: 2022-05-14T04:46:33.456+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:46:33.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:46:33 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T04:46:33.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:46:33 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:46:33.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:46:33 smithi047 conmon[78521]: yield 2022-05-14T04:46:33.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:46:33 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:46:33.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:46:33 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:46:33.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:46:33 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:46:33.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:46:33 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T04:46:33.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:46:33 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:46:33.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:46:33 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T04:46:33.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:46:33 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:46:33.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:46:33 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T04:46:33.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:46:33 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:46:33.797 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:46:33 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T04:46:33.797 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:46:33 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:46:33.797 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:46:33 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T04:46:33.797 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:46:33 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:46:33.798 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:46:33 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:46:33.798 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:46:33 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:46:33.798 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:46:33 smithi047 conmon[78521]: 2022-05-14T04:46:33.457+0000 7f5340f7a700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T04:46:33.798 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:46:33 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T04:46:33.799 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:46:33 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:46:33.799 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:46:33 smithi047 conmon[78521]: yield 2022-05-14T04:46:33.799 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:46:33 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:46:33.799 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:46:33 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:46:33.800 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:46:33 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:46:33.800 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:46:33 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T04:46:33.800 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:46:33 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:46:33.800 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:46:33 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T04:46:33.800 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:46:33 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:46:33.801 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:46:33 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T04:46:33.801 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:46:33 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:46:33.801 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:46:33 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T04:46:33.801 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:46:33 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:46:33.801 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:46:33 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T04:46:33.802 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:46:33 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:46:33.802 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:46:33 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:46:33.802 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:46:33 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:46:33.802 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:46:33 smithi047 conmon[78521]: 2022-05-14T04:46:33.802 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:46:33 smithi047 conmon[78521]: During handling of the above exception, another exception occurred: 2022-05-14T04:46:33.803 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:46:33 smithi047 conmon[78521]: 2022-05-14T04:46:33.803 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:46:33 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T04:46:33.803 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:46:33 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T04:46:33.803 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:46:33 smithi047 conmon[78521]: return f(*arg) 2022-05-14T04:46:33.804 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:46:33 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T04:46:33.804 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:46:33 smithi047 conmon[78521]: self._write_client_files(client_files, host) 2022-05-14T04:46:33.804 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:46:33 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T04:46:33.804 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:46:33 smithi047 conmon[78521]: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T04:46:33.804 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:46:33 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T04:46:33.805 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:46:33 smithi047 conmon[78521]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T04:46:33.805 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:46:33 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T04:46:33.805 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:46:33 smithi047 conmon[78521]: return self.event_loop.get_result(coro) 2022-05-14T04:46:33.805 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:46:33 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T04:46:33.805 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:46:33 smithi047 conmon[78521]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T04:46:33.806 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:46:33 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T04:46:33.806 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:46:33 smithi047 conmon[78521]: return self.__get_result() 2022-05-14T04:46:33.806 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:46:33 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T04:46:33.806 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:46:33 smithi047 conmon[78521]: raise self._exception 2022-05-14T04:46:33.807 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:46:33 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T04:46:33.807 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:46:33 smithi047 conmon[78521]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T04:46:33.807 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:46:33 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T04:46:33.807 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:46:33 smithi047 conmon[78521]: conn = await self._remote_connection(host, addr) 2022-05-14T04:46:33.807 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:46:33 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T04:46:33.808 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:46:33 smithi047 conmon[78521]: raise 2022-05-14T04:46:33.808 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:46:33 smithi047 conmon[78521]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T04:46:33.808 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:46:33 smithi047 conmon[78521]: self.gen.throw(type, value, traceback) 2022-05-14T04:46:33.808 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:46:33 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T04:46:33.808 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:46:33 smithi047 conmon[78521]: raise OrchestratorError(msg) 2022-05-14T04:46:33.809 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:46:33 smithi047 conmon[78521]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:46:33.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:46:33 smithi115 conmon[90393]: 2022-05-14T04:46:33.764+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:46:33.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:46:33 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:46:33.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:46:33 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 19125 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:46:34.860 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:46:34 smithi115 conmon[90393]: 2022-05-14T04:46:34.769+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:46:34.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:46:34 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:46:34.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:46:34 smithi115 ceph-mon[78593]: pgmap v9914: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:46:34.862 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:46:34 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:46:34.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T04:46:34.863 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:46:34.863 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T04:46:34.864 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:46:34.864 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:46:34.865 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:46:34.865 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T04:46:34.866 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:46:34.866 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T04:46:34.867 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:46:34.867 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T04:46:34.868 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:46:34.868 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T04:46:34.869 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:46:34.869 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T04:46:34.870 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:46:34.870 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:46:34.871 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:46:34.872 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:46:34 smithi115 ceph-mon[78593]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T04:46:34.872 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T04:46:34.872 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:46:34.873 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T04:46:34.873 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:46:34.874 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:46:34.874 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:46:34.875 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T04:46:34.876 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:46:34.876 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T04:46:34.876 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:46:34.877 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T04:46:34.877 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:46:34.877 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T04:46:34.877 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:46:34.878 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T04:46:34.878 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:46:34.878 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:46:34.878 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:46:34.878 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T04:46:34.879 INFO:journalctl@ceph.mon.c.smithi115.stdout: During handling of the above exception, another exception occurred: 2022-05-14T04:46:34.879 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T04:46:34.879 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T04:46:34.879 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T04:46:34.880 INFO:journalctl@ceph.mon.c.smithi115.stdout: return f(*arg) 2022-05-14T04:46:34.880 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T04:46:34.880 INFO:journalctl@ceph.mon.c.smithi115.stdout: self._write_client_files(client_files, host) 2022-05-14T04:46:34.880 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T04:46:34.880 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T04:46:34.881 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T04:46:34.881 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T04:46:34.881 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T04:46:34.881 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.event_loop.get_result(coro) 2022-05-14T04:46:34.882 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T04:46:34.882 INFO:journalctl@ceph.mon.c.smithi115.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T04:46:34.883 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T04:46:34.883 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.__get_result() 2022-05-14T04:46:34.884 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T04:46:34.884 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise self._exception 2022-05-14T04:46:34.885 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T04:46:34.885 INFO:journalctl@ceph.mon.c.smithi115.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T04:46:34.886 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T04:46:34.886 INFO:journalctl@ceph.mon.c.smithi115.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T04:46:34.887 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T04:46:34.887 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise 2022-05-14T04:46:34.887 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T04:46:34.888 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.gen.throw(type, value, traceback) 2022-05-14T04:46:34.888 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T04:46:34.889 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OrchestratorError(msg) 2022-05-14T04:46:34.889 INFO:journalctl@ceph.mon.c.smithi115.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:46:34.890 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:46:34 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:46:34.890 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:46:34 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:46:35.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:46:34 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:46:35.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:46:34 smithi047 ceph-mon[77147]: pgmap v9914: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:46:35.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:46:34 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:46:35.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T04:46:35.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:46:35.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T04:46:35.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:46:35.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:46:35.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:46:35.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T04:46:35.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:46:35.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T04:46:35.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:46:35.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T04:46:35.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:46:35.049 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T04:46:35.049 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:46:35.050 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T04:46:35.050 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:46:35.051 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:46:35.051 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:46:35.052 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:46:34 smithi047 ceph-mon[77147]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T04:46:35.052 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T04:46:35.053 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:46:35.053 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T04:46:35.054 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:46:35.054 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:46:35.054 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:46:35.055 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T04:46:35.055 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:46:35.056 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T04:46:35.057 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:46:35.064 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T04:46:35.064 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:46:35.065 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T04:46:35.065 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:46:35.066 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T04:46:35.067 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:46:35.067 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:46:35.067 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:46:35.067 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T04:46:35.068 INFO:journalctl@ceph.mon.b.smithi047.stdout: During handling of the above exception, another exception occurred: 2022-05-14T04:46:35.068 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T04:46:35.068 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T04:46:35.068 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T04:46:35.068 INFO:journalctl@ceph.mon.b.smithi047.stdout: return f(*arg) 2022-05-14T04:46:35.069 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T04:46:35.069 INFO:journalctl@ceph.mon.b.smithi047.stdout: self._write_client_files(client_files, host) 2022-05-14T04:46:35.070 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T04:46:35.070 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T04:46:35.070 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T04:46:35.070 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T04:46:35.071 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T04:46:35.071 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.event_loop.get_result(coro) 2022-05-14T04:46:35.071 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T04:46:35.071 INFO:journalctl@ceph.mon.b.smithi047.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T04:46:35.071 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T04:46:35.072 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.__get_result() 2022-05-14T04:46:35.072 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T04:46:35.072 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise self._exception 2022-05-14T04:46:35.072 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T04:46:35.073 INFO:journalctl@ceph.mon.b.smithi047.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T04:46:35.073 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T04:46:35.073 INFO:journalctl@ceph.mon.b.smithi047.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T04:46:35.074 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T04:46:35.074 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise 2022-05-14T04:46:35.074 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T04:46:35.075 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.gen.throw(type, value, traceback) 2022-05-14T04:46:35.075 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T04:46:35.075 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OrchestratorError(msg) 2022-05-14T04:46:35.076 INFO:journalctl@ceph.mon.b.smithi047.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:46:35.076 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:46:34 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:46:35.076 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:46:34 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:46:35.860 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:46:35 smithi115 conmon[90393]: 2022-05-14T04:46:35.750+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:46:35.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:46:35 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:46:36.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:46:35 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:46:36.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:46:36 smithi115 conmon[90393]: 2022-05-14T04:46:36.735+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:46:36.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:46:36 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:46:36.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:46:36 smithi115 ceph-mon[78593]: pgmap v9915: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:46:37.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:46:36 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:46:37.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:46:36 smithi047 ceph-mon[77147]: pgmap v9915: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:46:37.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:46:37 smithi115 conmon[90393]: 2022-05-14T04:46:37.720+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:46:37.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:46:37 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:46:38.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:46:37 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:46:38.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:46:38 smithi115 conmon[90393]: 2022-05-14T04:46:38.738+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:46:38.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:46:38 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:46:38.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:46:38 smithi115 ceph-mon[78593]: pgmap v9916: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:46:38.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:46:38 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:46:38.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:46:38 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:46:38.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:46:38 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 19130 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:46:39.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:46:38 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:46:39.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:46:38 smithi047 ceph-mon[77147]: pgmap v9916: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:46:39.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:46:38 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:46:39.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:46:38 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:46:39.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:46:38 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 19130 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:46:39.860 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:46:39 smithi115 conmon[90393]: 2022-05-14T04:46:39.700+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:46:39.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:46:39 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:46:40.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:46:39 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:46:40.860 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:46:40 smithi115 conmon[90393]: 2022-05-14T04:46:40.721+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:46:40.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:46:40 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:46:40.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:46:40 smithi115 ceph-mon[78593]: pgmap v9917: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:46:41.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:46:40 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:46:41.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:46:40 smithi047 ceph-mon[77147]: pgmap v9917: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:46:41.860 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:46:41 smithi115 conmon[90393]: 2022-05-14T04:46:41.708+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:46:41.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:46:41 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:46:42.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:46:41 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:46:42.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:46:42 smithi115 conmon[90393]: 2022-05-14T04:46:42.749+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:46:42.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:46:42 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:46:42.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:46:42 smithi115 ceph-mon[78593]: pgmap v9918: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:46:42.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:46:42 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:46:42.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:46:42 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:46:43.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:46:42 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:46:43.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:46:42 smithi047 ceph-mon[77147]: pgmap v9918: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:46:43.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:46:42 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:46:43.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:46:42 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:46:43.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:46:43 smithi115 conmon[90393]: 2022-05-14T04:46:43.745+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:46:43.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:46:43 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:46:43.863 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:46:43 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 19135 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:46:44.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:46:43 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:46:44.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:46:43 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 19135 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:46:44.860 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:46:44 smithi115 conmon[90393]: 2022-05-14T04:46:44.722+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:46:44.907 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:46:44 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:46:44.908 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:46:44 smithi115 ceph-mon[78593]: pgmap v9919: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:46:45.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:46:44 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:46:45.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:46:44 smithi047 ceph-mon[77147]: pgmap v9919: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:46:45.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:46:45 smithi115 conmon[90393]: 2022-05-14T04:46:45.709+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:46:45.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:46:45 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:46:46.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:46:45 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:46:46.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:46:46 smithi115 conmon[90393]: 2022-05-14T04:46:46.756+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:46:46.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:46:46 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:46:46.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:46:46 smithi115 ceph-mon[78593]: pgmap v9920: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:46:47.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:46:46 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:46:47.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:46:46 smithi047 ceph-mon[77147]: pgmap v9920: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:46:47.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:46:47 smithi115 conmon[90393]: 2022-05-14T04:46:47.765+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:46:47.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:46:47 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:46:48.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:46:47 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:46:48.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:46:48 smithi115 conmon[90393]: 2022-05-14T04:46:48.759+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:46:48.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:46:48 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:46:48.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:46:48 smithi115 ceph-mon[78593]: pgmap v9921: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:46:48.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:46:48 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 19140 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:46:49.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:46:48 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:46:49.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:46:48 smithi047 ceph-mon[77147]: pgmap v9921: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:46:49.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:46:48 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 19140 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:46:49.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:46:49 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:46:49.862 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:46:49 smithi115 conmon[90393]: 2022-05-14T04:46:49.723+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:46:50.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:46:49 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:46:50.861 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:46:50 smithi115 conmon[90393]: 2022-05-14T04:46:50.772+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:46:50.862 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:46:50 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:46:50.865 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:46:50 smithi115 ceph-mon[78593]: pgmap v9922: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:46:50.865 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:46:50 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:46:50.865 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:46:50 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:46:51.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:46:50 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:46:51.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:46:50 smithi047 ceph-mon[77147]: pgmap v9922: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:46:51.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:46:50 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:46:51.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:46:50 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:46:51.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:46:51 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:46:51.860 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:46:51 smithi115 conmon[90393]: 2022-05-14T04:46:51.785+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:46:52.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:46:51 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:46:52.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:46:52 smithi115 conmon[90393]: 2022-05-14T04:46:52.807+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:46:52.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:46:52 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:46:52.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:46:52 smithi115 ceph-mon[78593]: pgmap v9923: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:46:53.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:46:52 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:46:53.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:46:52 smithi047 ceph-mon[77147]: pgmap v9923: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:46:53.861 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:46:53 smithi115 conmon[90393]: 2022-05-14T04:46:53.844+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:46:53.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:46:53 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:46:53.862 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:46:53 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 19145 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:46:54.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:46:53 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:46:54.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:46:53 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 19145 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:46:54.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:46:54 smithi115 conmon[90393]: 2022-05-14T04:46:54.795+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:46:54.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:46:54 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:46:54.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:46:54 smithi115 ceph-mon[78593]: pgmap v9924: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:46:54.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:46:54 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:46:54.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:46:54 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:46:55.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:46:54 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:46:55.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:46:54 smithi047 ceph-mon[77147]: pgmap v9924: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:46:55.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:46:54 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:46:55.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:46:54 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:46:55.861 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:46:55 smithi115 conmon[90393]: 2022-05-14T04:46:55.826+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:46:55.862 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:46:55 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:46:56.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:46:55 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:46:56.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:46:56 smithi115 conmon[90393]: 2022-05-14T04:46:56.790+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:46:56.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:46:56 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:46:56.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:46:56 smithi115 ceph-mon[78593]: pgmap v9925: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:46:57.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:46:56 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:46:57.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:46:56 smithi047 ceph-mon[77147]: pgmap v9925: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:46:58.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:46:57 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:46:58.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:46:57 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:46:58.110 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:46:57 smithi115 conmon[90393]: 2022-05-14T04:46:57.742+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:46:59.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:46:58 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:46:59.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:46:58 smithi047 ceph-mon[77147]: pgmap v9926: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:46:59.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:46:58 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:46:59.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:46:58 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:46:59.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:46:58 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 19150 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:46:59.110 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:46:58 smithi115 conmon[90393]: 2022-05-14T04:46:58.757+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:46:59.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:46:58 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:46:59.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:46:58 smithi115 ceph-mon[78593]: pgmap v9926: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:46:59.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:46:58 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:46:59.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:46:58 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:46:59.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:46:58 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 19150 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:47:00.050 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:46:59 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:47:00.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:46:59 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:47:00.110 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:46:59 smithi115 conmon[90393]: 2022-05-14T04:46:59.737+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:47:01.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:47:00 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:47:01.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:47:00 smithi047 ceph-mon[77147]: pgmap v9927: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:47:01.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:47:00 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:47:01.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:47:00 smithi115 ceph-mon[78593]: pgmap v9927: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:47:01.110 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:47:00 smithi115 conmon[90393]: 2022-05-14T04:47:00.720+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:47:02.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:47:01 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:47:02.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:47:01 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:47:02.110 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:47:01 smithi115 conmon[90393]: 2022-05-14T04:47:01.756+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:47:03.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:47:02 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:47:03.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:47:02 smithi047 ceph-mon[77147]: pgmap v9928: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:47:03.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:47:02 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:47:03.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:47:02 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:47:03.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:47:02 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:47:03.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:47:02 smithi115 ceph-mon[78593]: pgmap v9928: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:47:03.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:47:02 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:47:03.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:47:02 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:47:03.111 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:47:02 smithi115 conmon[90393]: 2022-05-14T04:47:02.733+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:47:04.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:47:03 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:47:04.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:47:03 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 19155 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:47:04.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:47:03 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:47:04.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:47:03 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 19155 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:47:04.110 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:47:03 smithi115 conmon[90393]: 2022-05-14T04:47:03.714+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:47:05.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:47:04 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:47:05.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:47:04 smithi047 ceph-mon[77147]: pgmap v9929: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:47:05.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:47:04 smithi115 conmon[90393]: 2022-05-14T04:47:04.692+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:47:05.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:47:04 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:47:05.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:47:04 smithi115 ceph-mon[78593]: pgmap v9929: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:47:06.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:47:05 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:47:06.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:47:05 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:47:06.110 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:47:05 smithi115 conmon[90393]: 2022-05-14T04:47:05.720+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:47:07.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:47:06 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:47:07.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:47:06 smithi047 ceph-mon[77147]: pgmap v9930: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:47:07.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:47:06 smithi115 conmon[90393]: 2022-05-14T04:47:06.676+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:47:07.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:47:06 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:47:07.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:47:06 smithi115 ceph-mon[78593]: pgmap v9930: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:47:08.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:47:07 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:47:08.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:47:07 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:47:08.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:47:07 smithi115 conmon[90393]: 2022-05-14T04:47:07.641+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:47:08.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:47:07 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:47:08.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:47:07 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:47:09.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:47:08 smithi047 ceph-mon[77147]: pgmap v9931: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:47:09.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:47:08 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:47:09.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:47:08 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 19160 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:47:09.118 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:47:08 smithi115 conmon[90393]: 2022-05-14T04:47:08.655+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:47:09.119 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:47:08 smithi115 ceph-mon[78593]: pgmap v9931: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:47:09.119 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:47:08 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:47:09.120 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:47:08 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 19160 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:47:10.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:47:09 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:47:10.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:47:09 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:47:10.110 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:47:09 smithi115 conmon[90393]: 2022-05-14T04:47:09.636+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:47:11.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:47:10 smithi047 ceph-mon[77147]: pgmap v9932: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:47:11.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:47:10 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:47:11.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:47:10 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:47:11.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:47:10 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:47:11.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:47:10 smithi115 conmon[90393]: 2022-05-14T04:47:10.616+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:47:11.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:47:10 smithi115 ceph-mon[78593]: pgmap v9932: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:47:11.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:47:10 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:47:11.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:47:10 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:47:11.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:47:10 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:47:12.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:47:11 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:47:12.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:47:11 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:47:12.110 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:47:11 smithi115 conmon[90393]: 2022-05-14T04:47:11.607+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:47:13.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:47:12 smithi047 ceph-mon[77147]: pgmap v9933: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:47:13.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:47:12 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:47:13.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:47:12 smithi115 conmon[90393]: 2022-05-14T04:47:12.612+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:47:13.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:47:12 smithi115 ceph-mon[78593]: pgmap v9933: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:47:13.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:47:12 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:47:14.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:47:13 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:47:14.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:47:13 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 19165 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:47:14.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:47:13 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:47:14.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:47:13 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 19165 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:47:14.110 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:47:13 smithi115 conmon[90393]: 2022-05-14T04:47:13.629+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:47:15.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:47:14 smithi047 ceph-mon[77147]: pgmap v9934: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:47:15.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:47:14 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:47:15.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:47:14 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:47:15.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:47:14 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:47:15.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:47:14 smithi115 conmon[90393]: 2022-05-14T04:47:14.629+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:47:15.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:47:14 smithi115 ceph-mon[78593]: pgmap v9934: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:47:15.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:47:14 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:47:15.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:47:14 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:47:15.112 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:47:14 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:47:16.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:47:15 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:47:16.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:47:15 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T04:47:16.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:47:15 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T04:47:16.114 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:47:15 smithi115 conmon[90393]: 2022-05-14T04:47:15.627+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:47:16.114 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:47:15 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:47:16.115 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:47:15 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T04:47:16.115 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:47:15 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T04:47:17.045 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:47:16 smithi047 ceph-mon[77147]: pgmap v9935: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:47:17.045 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:47:16 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:47:17.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:47:16 smithi115 ceph-mon[78593]: pgmap v9935: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:47:17.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:47:16 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:47:17.110 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:47:16 smithi115 conmon[90393]: 2022-05-14T04:47:16.664+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:47:18.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:47:17 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:47:18.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:47:17 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:47:18.110 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:47:17 smithi115 conmon[90393]: 2022-05-14T04:47:17.627+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:47:18.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:47:18 smithi115 ceph-mon[78593]: pgmap v9936: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:47:18.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:47:18 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:47:18.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:47:18 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:47:18.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:47:18 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:47:18.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:47:18 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 19170 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:47:18.862 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:47:18 smithi115 conmon[90393]: 2022-05-14T04:47:18.605+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:47:19.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:47:18 smithi047 ceph-mon[77147]: pgmap v9936: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:47:19.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:47:18 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:47:19.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:47:18 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:47:19.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:47:18 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:47:19.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:47:18 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 19170 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:47:19.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:47:19 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:47:19.860 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:47:19 smithi115 conmon[90393]: 2022-05-14T04:47:19.578+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:47:20.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:47:19 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:47:20.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:47:20 smithi115 conmon[90393]: 2022-05-14T04:47:20.585+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:47:20.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:47:20 smithi115 ceph-mon[78593]: pgmap v9937: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:47:20.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:47:20 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:47:21.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:47:20 smithi047 ceph-mon[77147]: pgmap v9937: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:47:21.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:47:20 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:47:21.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:47:21 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:47:21.860 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:47:21 smithi115 conmon[90393]: 2022-05-14T04:47:21.557+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:47:22.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:47:21 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:47:22.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:47:22 smithi115 conmon[90393]: 2022-05-14T04:47:22.527+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:47:22.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:47:22 smithi115 ceph-mon[78593]: pgmap v9938: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:47:22.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:47:22 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:47:22.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:47:22 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:47:22.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:47:22 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:47:23.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:47:22 smithi047 ceph-mon[77147]: pgmap v9938: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:47:23.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:47:22 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:47:23.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:47:22 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:47:23.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:47:22 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:47:23.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:47:23 smithi115 conmon[90393]: 2022-05-14T04:47:23.567+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:47:23.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:47:23 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:47:23.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:47:23 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 19175 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:47:24.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:47:23 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:47:24.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:47:23 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 19175 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:47:24.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:47:24 smithi115 conmon[90393]: 2022-05-14T04:47:24.583+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:47:24.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:47:24 smithi115 ceph-mon[78593]: pgmap v9939: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:47:24.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:47:24 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:47:25.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:47:24 smithi047 ceph-mon[77147]: pgmap v9939: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:47:25.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:47:24 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:47:25.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:47:25 smithi115 conmon[90393]: 2022-05-14T04:47:25.540+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:47:25.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:47:25 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:47:26.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:47:25 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:47:26.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:47:26 smithi115 ceph-mon[78593]: pgmap v9940: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:47:26.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:47:26 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:47:26.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:47:26 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:47:26.860 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:47:26 smithi115 conmon[90393]: 2022-05-14T04:47:26.538+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:47:27.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:47:26 smithi047 ceph-mon[77147]: pgmap v9940: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:47:27.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:47:26 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:47:27.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:47:26 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:47:27.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:47:27 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:47:27.860 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:47:27 smithi115 conmon[90393]: 2022-05-14T04:47:27.555+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:47:28.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:47:27 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:47:28.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:47:28 smithi115 conmon[90393]: 2022-05-14T04:47:28.527+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:47:28.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:47:28 smithi115 ceph-mon[78593]: pgmap v9941: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:47:28.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:47:28 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 19180 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:47:28.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:47:28 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:47:29.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:47:28 smithi047 ceph-mon[77147]: pgmap v9941: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:47:29.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:47:28 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 19180 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:47:29.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:47:28 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:47:29.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:47:29 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:47:29.860 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:47:29 smithi115 conmon[90393]: 2022-05-14T04:47:29.552+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:47:30.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:47:29 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:47:30.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:47:30 smithi115 conmon[90393]: 2022-05-14T04:47:30.593+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:47:30.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:47:30 smithi115 ceph-mon[78593]: pgmap v9942: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:47:30.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:47:30 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:47:30.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:47:30 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:47:30.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:47:30 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:47:31.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:47:30 smithi047 ceph-mon[77147]: pgmap v9942: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:47:31.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:47:30 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:47:31.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:47:30 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:47:31.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:47:30 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:47:32.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:47:31 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:47:32.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:47:31 smithi115 conmon[90393]: 2022-05-14T04:47:31.609+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:47:32.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:47:31 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:47:32.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:47:32 smithi115 conmon[90393]: 2022-05-14T04:47:32.599+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:47:32.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:47:32 smithi115 ceph-mon[78593]: pgmap v9943: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:47:32.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:47:32 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:47:32.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:47:32 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 19185 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:47:33.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:47:32 smithi047 ceph-mon[77147]: pgmap v9943: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:47:33.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:47:32 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:47:33.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:47:32 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 19185 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:47:33.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:47:33 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T04:47:33.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:47:33 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T04:47:33.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:47:33 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T04:47:33.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:47:33 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:47:33.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:47:33 smithi115 conmon[90393]: 2022-05-14T04:47:33.570+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:47:33.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:47:33 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T04:47:33.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:47:33 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T04:47:33.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:47:33 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T04:47:33.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:47:33 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:47:34.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:47:34 smithi115 conmon[90393]: 2022-05-14T04:47:34.521+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:47:34.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:47:34 smithi115 ceph-mon[78593]: pgmap v9944: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:47:34.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:47:34 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:47:34.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:47:34 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:47:34.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:47:34 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:47:35.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:47:34 smithi047 ceph-mon[77147]: pgmap v9944: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:47:35.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:47:34 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:47:35.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:47:34 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:47:35.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:47:34 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:47:35.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:47:35 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:47:35.860 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:47:35 smithi115 conmon[90393]: 2022-05-14T04:47:35.558+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:47:36.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:47:35 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:47:36.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:47:36 smithi115 conmon[90393]: 2022-05-14T04:47:36.590+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:47:36.713 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:47:36 smithi047 conmon[78521]: 2022-05-14T04:47:36.560+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:47:36.713 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:47:36 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T04:47:36.714 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:47:36 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:47:36.714 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:47:36 smithi047 conmon[78521]: yield 2022-05-14T04:47:36.715 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:47:36 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:47:36.715 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:47:36 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:47:36.715 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:47:36 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:47:36.715 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:47:36 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T04:47:36.716 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:47:36 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:47:36.716 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:47:36 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T04:47:36.716 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:47:36 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:47:36.717 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:47:36 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T04:47:36.717 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:47:36 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:47:36.717 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:47:36 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T04:47:36.718 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:47:36 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:47:36.718 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:47:36 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T04:47:36.719 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:47:36 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:47:36.719 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:47:36 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:47:36.719 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:47:36 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:47:37.050 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:47:36 smithi047 ceph-mon[77147]: pgmap v9945: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:47:37.050 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:47:36 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:47:37.051 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:47:36 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' 2022-05-14T04:47:37.051 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:47:36 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' 2022-05-14T04:47:37.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:47:36 smithi115 ceph-mon[78593]: pgmap v9945: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:47:37.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:47:36 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:47:37.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:47:36 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' 2022-05-14T04:47:37.112 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:47:36 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' 2022-05-14T04:47:38.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:47:37 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:47:38.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T04:47:38.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:47:38.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T04:47:38.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:47:38.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:47:38.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:47:38.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T04:47:38.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:47:38.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T04:47:38.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:47:38.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T04:47:38.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:47:38.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T04:47:38.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:47:38.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T04:47:38.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:47:38.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:47:38.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:47:38.047 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:47:37 smithi047 ceph-mon[77147]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T04:47:38.047 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:47:37 smithi047 ceph-mon[77147]: Detected new or changed devices on smithi115 2022-05-14T04:47:38.048 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:47:37 smithi047 ceph-mon[77147]: Detected new or changed devices on smithi047 2022-05-14T04:47:38.048 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:47:37 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:47:38.048 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:47:37 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 19190 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:47:38.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:47:37 smithi115 conmon[90393]: 2022-05-14T04:47:37.626+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:47:38.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:47:37 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:47:38.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T04:47:38.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:47:38.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T04:47:38.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:47:38.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:47:38.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:47:38.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T04:47:38.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:47:38.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T04:47:38.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:47:38.113 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T04:47:38.113 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:47:38.113 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T04:47:38.113 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:47:38.114 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T04:47:38.114 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:47:38.114 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:47:38.114 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:47:38.114 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:47:37 smithi115 ceph-mon[78593]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T04:47:38.115 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:47:37 smithi115 ceph-mon[78593]: Detected new or changed devices on smithi115 2022-05-14T04:47:38.115 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:47:37 smithi115 ceph-mon[78593]: Detected new or changed devices on smithi047 2022-05-14T04:47:38.115 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:47:37 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:47:38.115 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:47:37 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 19190 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:47:39.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:47:38 smithi047 ceph-mon[77147]: pgmap v9946: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:47:39.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:47:38 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:47:39.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:47:38 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:47:39.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:47:38 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:47:39.110 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:47:38 smithi115 conmon[90393]: 2022-05-14T04:47:38.667+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:47:39.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:47:38 smithi115 ceph-mon[78593]: pgmap v9946: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:47:39.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:47:38 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:47:39.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:47:38 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:47:39.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:47:38 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:47:40.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:47:39 smithi047 conmon[78521]: 2022-05-14T04:47:39.631+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:47:40.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:47:39 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T04:47:40.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:47:39 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:47:40.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:47:39 smithi047 conmon[78521]: yield 2022-05-14T04:47:40.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:47:39 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:47:40.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:47:39 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:47:40.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:47:39 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:47:40.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:47:39 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T04:47:40.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:47:39 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:47:40.049 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:47:39 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T04:47:40.049 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:47:39 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:47:40.049 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:47:39 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T04:47:40.050 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:47:39 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:47:40.050 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:47:39 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T04:47:40.050 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:47:39 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:47:40.050 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:47:39 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T04:47:40.050 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:47:39 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:47:40.051 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:47:39 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:47:40.051 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:47:39 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:47:40.051 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:47:39 smithi047 conmon[78521]: 2022-05-14T04:47:39.632+0000 7f5344f82700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T04:47:40.052 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:47:39 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T04:47:40.052 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:47:39 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:47:40.053 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:47:39 smithi047 conmon[78521]: yield 2022-05-14T04:47:40.053 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:47:39 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:47:40.053 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:47:39 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:47:40.054 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:47:39 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:47:40.054 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:47:39 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T04:47:40.054 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:47:39 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:47:40.054 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:47:39 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T04:47:40.055 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:47:39 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:47:40.055 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:47:39 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T04:47:40.055 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:47:39 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:47:40.055 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:47:39 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T04:47:40.055 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:47:39 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:47:40.056 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:47:39 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T04:47:40.056 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:47:39 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:47:40.056 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:47:39 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:47:40.056 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:47:39 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:47:40.057 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:47:39 smithi047 conmon[78521]: 2022-05-14T04:47:40.057 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:47:39 smithi047 conmon[78521]: During handling of the above exception, another exception occurred: 2022-05-14T04:47:40.057 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:47:39 smithi047 conmon[78521]: 2022-05-14T04:47:40.057 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:47:39 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T04:47:40.058 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:47:39 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T04:47:40.058 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:47:39 smithi047 conmon[78521]: return f(*arg) 2022-05-14T04:47:40.058 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:47:39 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T04:47:40.058 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:47:39 smithi047 conmon[78521]: self._write_client_files(client_files, host) 2022-05-14T04:47:40.059 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:47:39 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T04:47:40.059 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:47:39 smithi047 conmon[78521]: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T04:47:40.059 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:47:39 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T04:47:40.059 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:47:39 smithi047 conmon[78521]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T04:47:40.060 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:47:39 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T04:47:40.060 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:47:39 smithi047 conmon[78521]: return self.event_loop.get_result(coro) 2022-05-14T04:47:40.060 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:47:39 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T04:47:40.060 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:47:39 smithi047 conmon[78521]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T04:47:40.061 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:47:39 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T04:47:40.061 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:47:39 smithi047 conmon[78521]: return self.__get_result() 2022-05-14T04:47:40.061 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:47:39 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T04:47:40.061 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:47:39 smithi047 conmon[78521]: raise self._exception 2022-05-14T04:47:40.061 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:47:39 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T04:47:40.062 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:47:39 smithi047 conmon[78521]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T04:47:40.062 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:47:39 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T04:47:40.062 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:47:39 smithi047 conmon[78521]: conn = await self._remote_connection(host, addr) 2022-05-14T04:47:40.062 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:47:39 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T04:47:40.063 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:47:39 smithi047 conmon[78521]: raise 2022-05-14T04:47:40.063 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:47:39 smithi047 conmon[78521]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T04:47:40.063 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:47:39 smithi047 conmon[78521]: self.gen.throw(type, value, traceback) 2022-05-14T04:47:40.063 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:47:39 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T04:47:40.063 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:47:39 smithi047 conmon[78521]: raise OrchestratorError(msg) 2022-05-14T04:47:40.064 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:47:39 smithi047 conmon[78521]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:47:40.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:47:39 smithi115 conmon[90393]: 2022-05-14T04:47:39.699+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:47:41.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:47:40 smithi047 ceph-mon[77147]: pgmap v9947: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:47:41.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:47:40 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:47:41.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T04:47:41.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:47:41.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T04:47:41.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:47:41.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:47:41.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:47:41.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T04:47:41.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:47:41.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T04:47:41.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:47:41.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T04:47:41.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:47:41.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T04:47:41.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:47:41.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T04:47:41.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:47:41.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:47:41.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:47:41.048 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:47:40 smithi047 ceph-mon[77147]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T04:47:41.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T04:47:41.049 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:47:41.049 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T04:47:41.049 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:47:41.049 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:47:41.050 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:47:41.050 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T04:47:41.050 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:47:41.050 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T04:47:41.051 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:47:41.051 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T04:47:41.051 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:47:41.051 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T04:47:41.052 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:47:41.052 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T04:47:41.052 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:47:41.052 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:47:41.053 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:47:41.053 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T04:47:41.053 INFO:journalctl@ceph.mon.b.smithi047.stdout: During handling of the above exception, another exception occurred: 2022-05-14T04:47:41.053 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T04:47:41.053 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T04:47:41.054 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T04:47:41.054 INFO:journalctl@ceph.mon.b.smithi047.stdout: return f(*arg) 2022-05-14T04:47:41.054 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T04:47:41.054 INFO:journalctl@ceph.mon.b.smithi047.stdout: self._write_client_files(client_files, host) 2022-05-14T04:47:41.055 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T04:47:41.055 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T04:47:41.055 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T04:47:41.055 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T04:47:41.056 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T04:47:41.056 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.event_loop.get_result(coro) 2022-05-14T04:47:41.056 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T04:47:41.056 INFO:journalctl@ceph.mon.b.smithi047.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T04:47:41.056 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T04:47:41.057 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.__get_result() 2022-05-14T04:47:41.057 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T04:47:41.057 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise self._exception 2022-05-14T04:47:41.057 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T04:47:41.057 INFO:journalctl@ceph.mon.b.smithi047.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T04:47:41.058 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T04:47:41.058 INFO:journalctl@ceph.mon.b.smithi047.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T04:47:41.058 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T04:47:41.058 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise 2022-05-14T04:47:41.059 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T04:47:41.059 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.gen.throw(type, value, traceback) 2022-05-14T04:47:41.059 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T04:47:41.059 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OrchestratorError(msg) 2022-05-14T04:47:41.059 INFO:journalctl@ceph.mon.b.smithi047.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:47:41.061 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:47:40 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:47:41.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:47:40 smithi115 conmon[90393]: 2022-05-14T04:47:40.704+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:47:41.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:47:40 smithi115 ceph-mon[78593]: pgmap v9947: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:47:41.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:47:40 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:47:41.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T04:47:41.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:47:41.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T04:47:41.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:47:41.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:47:41.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:47:41.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T04:47:41.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:47:41.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T04:47:41.113 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:47:41.113 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T04:47:41.113 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:47:41.113 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T04:47:41.113 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:47:41.114 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T04:47:41.114 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:47:41.114 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:47:41.114 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:47:41.115 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:47:40 smithi115 ceph-mon[78593]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T04:47:41.115 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T04:47:41.115 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:47:41.115 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T04:47:41.115 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:47:41.116 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:47:41.116 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:47:41.116 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T04:47:41.116 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:47:41.116 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T04:47:41.117 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:47:41.117 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T04:47:41.117 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:47:41.117 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T04:47:41.118 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:47:41.118 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T04:47:41.118 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:47:41.118 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:47:41.118 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:47:41.119 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T04:47:41.119 INFO:journalctl@ceph.mon.c.smithi115.stdout: During handling of the above exception, another exception occurred: 2022-05-14T04:47:41.119 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T04:47:41.119 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T04:47:41.119 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T04:47:41.120 INFO:journalctl@ceph.mon.c.smithi115.stdout: return f(*arg) 2022-05-14T04:47:41.120 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T04:47:41.120 INFO:journalctl@ceph.mon.c.smithi115.stdout: self._write_client_files(client_files, host) 2022-05-14T04:47:41.120 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T04:47:41.120 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T04:47:41.121 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T04:47:41.121 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T04:47:41.121 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T04:47:41.121 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.event_loop.get_result(coro) 2022-05-14T04:47:41.122 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T04:47:41.122 INFO:journalctl@ceph.mon.c.smithi115.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T04:47:41.122 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T04:47:41.122 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.__get_result() 2022-05-14T04:47:41.122 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T04:47:41.123 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise self._exception 2022-05-14T04:47:41.123 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T04:47:41.123 INFO:journalctl@ceph.mon.c.smithi115.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T04:47:41.123 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T04:47:41.123 INFO:journalctl@ceph.mon.c.smithi115.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T04:47:41.124 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T04:47:41.124 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise 2022-05-14T04:47:41.124 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T04:47:41.124 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.gen.throw(type, value, traceback) 2022-05-14T04:47:41.124 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T04:47:41.125 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OrchestratorError(msg) 2022-05-14T04:47:41.125 INFO:journalctl@ceph.mon.c.smithi115.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:47:41.125 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:47:40 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:47:42.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:47:41 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:47:42.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:47:41 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:47:42.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:47:41 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:47:42.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:47:41 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:47:42.121 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:47:41 smithi115 conmon[90393]: 2022-05-14T04:47:41.674+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:47:43.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:47:42 smithi047 ceph-mon[77147]: pgmap v9948: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:47:43.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:47:42 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:47:43.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:47:42 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:47:43.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:47:42 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:47:43.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:47:42 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 19195 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:47:43.110 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:47:42 smithi115 conmon[90393]: 2022-05-14T04:47:42.676+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:47:43.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:47:42 smithi115 ceph-mon[78593]: pgmap v9948: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:47:43.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:47:42 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:47:43.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:47:42 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:47:43.112 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:47:42 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:47:43.112 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:47:42 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 19195 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:47:44.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:47:43 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:47:44.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:47:43 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:47:44.110 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:47:43 smithi115 conmon[90393]: 2022-05-14T04:47:43.656+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:47:45.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:47:44 smithi047 ceph-mon[77147]: pgmap v9949: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:47:45.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:47:44 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:47:45.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:47:44 smithi115 ceph-mon[78593]: pgmap v9949: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:47:45.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:47:44 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:47:45.110 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:47:44 smithi115 conmon[90393]: 2022-05-14T04:47:44.617+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:47:45.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:47:45 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:47:45.860 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:47:45 smithi115 conmon[90393]: 2022-05-14T04:47:45.597+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:47:46.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:47:45 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:47:47.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:47:46 smithi047 ceph-mon[77147]: pgmap v9950: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:47:47.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:47:46 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:47:47.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:47:46 smithi115 ceph-mon[78593]: pgmap v9950: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:47:47.113 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:47:46 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:47:47.114 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:47:46 smithi115 conmon[90393]: 2022-05-14T04:47:46.623+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:47:48.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:47:47 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:47:48.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:47:47 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 19200 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:47:48.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:47:47 smithi115 conmon[90393]: 2022-05-14T04:47:47.615+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:47:48.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:47:47 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:47:48.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:47:47 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 19200 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:47:48.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:47:48 smithi115 conmon[90393]: 2022-05-14T04:47:48.583+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:47:48.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:47:48 smithi115 ceph-mon[78593]: pgmap v9951: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:47:48.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:47:48 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:47:49.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:47:48 smithi047 ceph-mon[77147]: pgmap v9951: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:47:49.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:47:48 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:47:49.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:47:49 smithi115 ceph-mon[78593]: pgmap v9952: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:47:49.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:47:49 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:47:49.860 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:47:49 smithi115 conmon[90393]: 2022-05-14T04:47:49.566+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:47:50.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:47:49 smithi047 ceph-mon[77147]: pgmap v9952: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:47:50.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:47:49 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:47:50.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:47:50 smithi115 conmon[90393]: 2022-05-14T04:47:50.581+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:47:50.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:47:50 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:47:50.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:47:50 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:47:50.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:47:50 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:47:51.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:47:50 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:47:51.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:47:50 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:47:51.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:47:50 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:47:52.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:47:51 smithi047 ceph-mon[77147]: pgmap v9953: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:47:52.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:47:51 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:47:52.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:47:51 smithi115 conmon[90393]: 2022-05-14T04:47:51.612+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:47:52.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:47:51 smithi115 ceph-mon[78593]: pgmap v9953: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:47:52.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:47:51 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:47:53.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:47:52 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:47:53.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:47:52 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 19205 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:47:53.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:47:52 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:47:53.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:47:52 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 19205 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:47:53.110 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:47:52 smithi115 conmon[90393]: 2022-05-14T04:47:52.631+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:47:53.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:47:53 smithi115 ceph-mon[78593]: pgmap v9954: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:47:53.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:47:53 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:47:53.861 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:47:53 smithi115 conmon[90393]: 2022-05-14T04:47:53.588+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:47:54.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:47:53 smithi047 ceph-mon[77147]: pgmap v9954: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:47:54.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:47:53 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:47:54.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:47:54 smithi115 conmon[90393]: 2022-05-14T04:47:54.578+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:47:54.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:47:54 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:47:54.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:47:54 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:47:54.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:47:54 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:47:55.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:47:54 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:47:55.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:47:54 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:47:55.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:47:54 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:47:55.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:47:55 smithi115 ceph-mon[78593]: pgmap v9955: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:47:55.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:47:55 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:47:55.860 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:47:55 smithi115 conmon[90393]: 2022-05-14T04:47:55.556+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:47:56.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:47:55 smithi047 ceph-mon[77147]: pgmap v9955: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:47:56.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:47:55 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:47:56.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:47:56 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:47:56.860 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:47:56 smithi115 conmon[90393]: 2022-05-14T04:47:56.574+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:47:57.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:47:56 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:47:57.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:47:57 smithi115 ceph-mon[78593]: pgmap v9956: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:47:57.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:47:57 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:47:57.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:47:57 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 19210 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:47:57.862 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:47:57 smithi115 conmon[90393]: 2022-05-14T04:47:57.570+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:47:58.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:47:57 smithi047 ceph-mon[77147]: pgmap v9956: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:47:58.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:47:57 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:47:58.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:47:57 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 19210 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:47:58.800 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:47:58 smithi115 conmon[90393]: 2022-05-14T04:47:58.527+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:47:59.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:47:58 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:47:59.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:47:58 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:47:59.112 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:47:58 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:47:59.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:47:58 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:47:59.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:47:58 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:47:59.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:47:58 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:47:59.805 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:47:59 smithi115 conmon[90393]: 2022-05-14T04:47:59.514+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:48:00.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:47:59 smithi115 ceph-mon[78593]: pgmap v9957: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:48:00.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:47:59 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:48:00.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:47:59 smithi047 ceph-mon[77147]: pgmap v9957: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:48:00.295 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:47:59 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:48:00.811 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:48:00 smithi115 conmon[90393]: 2022-05-14T04:48:00.534+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:48:01.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:48:00 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:48:01.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:48:00 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:48:01.812 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:48:01 smithi115 conmon[90393]: 2022-05-14T04:48:01.522+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:48:02.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:48:01 smithi115 ceph-mon[78593]: pgmap v9958: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:48:02.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:48:01 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:48:02.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:48:01 smithi047 ceph-mon[77147]: pgmap v9958: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:48:02.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:48:01 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:48:02.813 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:48:02 smithi115 conmon[90393]: 2022-05-14T04:48:02.522+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:48:03.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:48:02 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:48:03.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:48:02 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:48:03.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:48:02 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:48:03.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:48:02 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 19215 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:48:03.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:48:02 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:48:03.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:48:02 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:48:03.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:48:02 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:48:03.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:48:02 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 19215 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:48:03.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:48:03 smithi115 ceph-mon[78593]: pgmap v9959: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:48:03.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:48:03 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:48:03.861 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:48:03 smithi115 conmon[90393]: 2022-05-14T04:48:03.572+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:48:04.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:48:03 smithi047 ceph-mon[77147]: pgmap v9959: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:48:04.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:48:03 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:48:04.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:48:04 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:48:04.860 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:48:04 smithi115 conmon[90393]: 2022-05-14T04:48:04.598+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:48:05.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:48:04 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:48:06.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:48:05 smithi115 ceph-mon[78593]: pgmap v9960: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:48:06.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:48:05 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:48:06.111 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:48:05 smithi115 conmon[90393]: 2022-05-14T04:48:05.623+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:48:06.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:48:05 smithi047 ceph-mon[77147]: pgmap v9960: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:48:06.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:48:05 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:48:06.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:48:06 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:48:06.860 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:48:06 smithi115 conmon[90393]: 2022-05-14T04:48:06.604+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:48:07.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:48:06 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:48:08.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:48:07 smithi115 conmon[90393]: 2022-05-14T04:48:07.621+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:48:08.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:48:07 smithi115 ceph-mon[78593]: pgmap v9961: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:48:08.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:48:07 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:48:08.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:48:07 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 19220 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:48:08.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:48:07 smithi047 ceph-mon[77147]: pgmap v9961: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:48:08.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:48:07 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:48:08.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:48:07 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 19220 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:48:09.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:48:08 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:48:09.110 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:48:08 smithi115 conmon[90393]: 2022-05-14T04:48:08.636+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:48:09.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:48:08 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:48:10.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:48:09 smithi115 ceph-mon[78593]: pgmap v9962: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:48:10.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:48:09 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:48:10.110 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:48:09 smithi115 conmon[90393]: 2022-05-14T04:48:09.660+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:48:10.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:48:09 smithi047 ceph-mon[77147]: pgmap v9962: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:48:10.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:48:09 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:48:11.110 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:48:10 smithi115 conmon[90393]: 2022-05-14T04:48:10.635+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:48:11.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:48:10 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:48:11.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:48:10 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:48:11.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:48:10 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:48:11.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:48:10 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:48:11.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:48:10 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:48:11.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:48:10 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:48:12.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:48:11 smithi115 ceph-mon[78593]: pgmap v9963: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:48:12.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:48:11 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:48:12.110 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:48:11 smithi115 conmon[90393]: 2022-05-14T04:48:11.636+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:48:12.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:48:11 smithi047 ceph-mon[77147]: pgmap v9963: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:48:12.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:48:11 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:48:13.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:48:12 smithi115 conmon[90393]: 2022-05-14T04:48:12.677+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:48:13.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:48:12 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:48:13.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:48:12 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 19225 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:48:13.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:48:12 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:48:13.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:48:12 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 19225 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:48:14.110 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:48:13 smithi115 conmon[90393]: 2022-05-14T04:48:13.639+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:48:14.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:48:13 smithi115 ceph-mon[78593]: pgmap v9964: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:48:14.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:48:13 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:48:14.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:48:13 smithi047 ceph-mon[77147]: pgmap v9964: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:48:14.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:48:13 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:48:15.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:48:14 smithi115 conmon[90393]: 2022-05-14T04:48:14.622+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:48:15.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:48:14 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:48:15.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:48:14 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:48:15.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:48:14 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:48:15.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:48:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T04:48:15.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:48:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T04:48:15.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:48:14 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:48:15.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:48:14 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:48:15.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:48:14 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:48:15.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:48:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T04:48:15.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:48:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T04:48:16.110 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:48:15 smithi115 conmon[90393]: 2022-05-14T04:48:15.665+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:48:16.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:48:15 smithi115 ceph-mon[78593]: pgmap v9965: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:48:16.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:48:15 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:48:16.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:48:15 smithi047 ceph-mon[77147]: pgmap v9965: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:48:16.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:48:15 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:48:17.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:48:16 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:48:17.110 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:48:16 smithi115 conmon[90393]: 2022-05-14T04:48:16.673+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:48:17.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:48:16 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:48:18.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:48:17 smithi115 ceph-mon[78593]: pgmap v9966: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:48:18.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:48:17 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 19230 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:48:18.110 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:48:17 smithi115 conmon[90393]: 2022-05-14T04:48:17.721+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:48:18.295 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:48:17 smithi047 ceph-mon[77147]: pgmap v9966: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:48:18.296 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:48:17 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 19230 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:48:19.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:48:18 smithi115 conmon[90393]: 2022-05-14T04:48:18.674+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:48:19.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:48:18 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:48:19.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:48:18 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:48:19.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:48:18 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:48:19.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:48:18 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:48:19.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:48:18 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:48:19.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:48:18 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:48:19.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:48:18 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:48:19.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:48:18 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:48:20.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:48:19 smithi115 ceph-mon[78593]: pgmap v9967: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:48:20.110 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:48:19 smithi115 conmon[90393]: 2022-05-14T04:48:19.712+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:48:20.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:48:19 smithi047 ceph-mon[77147]: pgmap v9967: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:48:21.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:48:20 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:48:21.110 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:48:20 smithi115 conmon[90393]: 2022-05-14T04:48:20.736+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:48:21.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:48:20 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:48:22.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:48:21 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:48:22.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:48:21 smithi115 ceph-mon[78593]: pgmap v9968: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:48:22.110 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:48:21 smithi115 conmon[90393]: 2022-05-14T04:48:21.763+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:48:22.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:48:21 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:48:22.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:48:21 smithi047 ceph-mon[77147]: pgmap v9968: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:48:23.110 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:48:22 smithi115 conmon[90393]: 2022-05-14T04:48:22.811+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:48:23.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:48:22 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:48:23.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:48:22 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:48:23.112 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:48:22 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:48:23.112 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:48:22 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 19235 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:48:23.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:48:22 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:48:23.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:48:22 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:48:23.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:48:22 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:48:23.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:48:22 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 19235 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:48:24.110 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:48:23 smithi115 conmon[90393]: 2022-05-14T04:48:23.787+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:48:24.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:48:23 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:48:24.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:48:23 smithi115 ceph-mon[78593]: pgmap v9969: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:48:24.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:48:23 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:48:24.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:48:23 smithi047 ceph-mon[77147]: pgmap v9969: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:48:25.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:48:24 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:48:25.110 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:48:24 smithi115 conmon[90393]: 2022-05-14T04:48:24.758+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:48:25.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:48:24 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:48:26.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:48:25 smithi115 conmon[90393]: 2022-05-14T04:48:25.756+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:48:26.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:48:25 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:48:26.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:48:25 smithi115 ceph-mon[78593]: pgmap v9970: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:48:26.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:48:25 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:48:26.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:48:25 smithi047 ceph-mon[77147]: pgmap v9970: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:48:27.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:48:26 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:48:27.110 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:48:26 smithi115 conmon[90393]: 2022-05-14T04:48:26.777+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:48:27.296 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:48:26 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:48:28.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:48:27 smithi115 conmon[90393]: 2022-05-14T04:48:27.814+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:48:28.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:48:27 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:48:28.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:48:27 smithi115 ceph-mon[78593]: pgmap v9971: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:48:28.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:48:27 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 19240 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:48:28.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:48:27 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:48:28.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:48:27 smithi047 ceph-mon[77147]: pgmap v9971: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:48:28.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:48:27 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 19240 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:48:29.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:48:28 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:48:29.110 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:48:28 smithi115 conmon[90393]: 2022-05-14T04:48:28.786+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:48:29.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:48:28 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:48:30.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:48:29 smithi115 conmon[90393]: 2022-05-14T04:48:29.830+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:48:30.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:48:29 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:48:30.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:48:29 smithi115 ceph-mon[78593]: pgmap v9972: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:48:30.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:48:29 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:48:30.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:48:29 smithi047 ceph-mon[77147]: pgmap v9972: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:48:31.110 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:48:30 smithi115 conmon[90393]: 2022-05-14T04:48:30.811+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:48:31.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:48:30 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:48:31.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:48:30 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:48:31.112 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:48:30 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:48:31.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:48:30 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:48:31.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:48:30 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:48:31.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:48:30 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:48:32.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:48:31 smithi115 conmon[90393]: 2022-05-14T04:48:31.763+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:48:32.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:48:31 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:48:32.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:48:31 smithi115 ceph-mon[78593]: pgmap v9973: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:48:32.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:48:31 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:48:32.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:48:31 smithi047 ceph-mon[77147]: pgmap v9973: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:48:33.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:48:32 smithi115 conmon[90393]: 2022-05-14T04:48:32.809+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:48:33.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:48:32 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:48:33.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:48:32 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 19245 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:48:33.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:48:32 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:48:33.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:48:32 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 19245 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:48:34.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:48:33 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:48:34.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:48:33 smithi115 ceph-mon[78593]: pgmap v9974: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:48:34.110 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:48:33 smithi115 conmon[90393]: 2022-05-14T04:48:33.778+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:48:34.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:48:33 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:48:34.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:48:33 smithi047 ceph-mon[77147]: pgmap v9974: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:48:35.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:48:34 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:48:35.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:48:34 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:48:35.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:48:34 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:48:35.112 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:48:34 smithi115 conmon[90393]: 2022-05-14T04:48:34.772+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:48:35.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:48:34 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:48:35.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:48:34 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:48:35.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:48:34 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:48:36.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:48:35 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:48:36.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:48:35 smithi115 ceph-mon[78593]: pgmap v9975: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:48:36.110 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:48:35 smithi115 conmon[90393]: 2022-05-14T04:48:35.763+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:48:36.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:48:35 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:48:36.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:48:35 smithi047 ceph-mon[77147]: pgmap v9975: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:48:37.110 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:48:36 smithi115 conmon[90393]: 2022-05-14T04:48:36.765+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:48:37.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:48:36 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:48:37.298 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:48:36 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:48:38.110 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:48:37 smithi115 conmon[90393]: 2022-05-14T04:48:37.778+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:48:38.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:48:37 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:48:38.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:48:37 smithi115 ceph-mon[78593]: pgmap v9976: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:48:38.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:48:37 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 19250 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:48:38.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:48:37 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:48:38.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:48:37 smithi047 ceph-mon[77147]: pgmap v9976: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:48:38.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:48:37 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 19250 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:48:39.113 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:48:38 smithi115 conmon[90393]: 2022-05-14T04:48:38.771+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:48:39.114 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:48:38 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:48:39.114 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:48:38 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:48:39.115 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:48:38 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:48:39.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:48:38 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:48:39.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:48:38 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:48:39.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:48:38 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:48:39.938 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:48:39 smithi115 conmon[90393]: 2022-05-14T04:48:39.732+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:48:40.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:48:39 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:48:40.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:48:39 smithi047 ceph-mon[77147]: pgmap v9977: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:48:40.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:48:39 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T04:48:40.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:48:39 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T04:48:40.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:48:39 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T04:48:40.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:48:39 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:48:40.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:48:39 smithi115 ceph-mon[78593]: pgmap v9977: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:48:40.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:48:39 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T04:48:40.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:48:39 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T04:48:40.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:48:39 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T04:48:41.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:48:40 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:48:41.111 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:48:40 smithi115 conmon[90393]: 2022-05-14T04:48:40.732+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:48:41.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:48:40 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:48:42.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:48:41 smithi115 conmon[90393]: 2022-05-14T04:48:41.747+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:48:42.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:48:41 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:48:42.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:48:41 smithi115 ceph-mon[78593]: pgmap v9978: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:48:42.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:48:41 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:48:42.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:48:41 smithi047 ceph-mon[77147]: pgmap v9978: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:48:43.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:48:42 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:48:43.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:48:42 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:48:43.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:48:42 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:48:43.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:48:42 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:48:43.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:48:42 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 19255 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:48:43.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:48:42 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:48:43.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T04:48:43.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:48:43.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T04:48:43.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:48:43.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:48:43.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:48:43.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T04:48:43.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:48:43.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T04:48:43.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:48:43.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T04:48:43.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:48:43.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T04:48:43.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:48:43.049 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T04:48:43.049 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:48:43.049 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:48:43.049 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:48:43.050 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:48:42 smithi047 ceph-mon[77147]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T04:48:43.050 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:48:42 smithi047 conmon[78521]: 2022-05-14T04:48:42.736+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:48:43.051 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:48:42 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T04:48:43.051 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:48:42 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:48:43.051 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:48:42 smithi047 conmon[78521]: yield 2022-05-14T04:48:43.051 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:48:42 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:48:43.052 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:48:42 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:48:43.052 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:48:42 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:48:43.052 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:48:42 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T04:48:43.052 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:48:42 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:48:43.053 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:48:42 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T04:48:43.053 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:48:42 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:48:43.053 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:48:42 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T04:48:43.053 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:48:42 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:48:43.053 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:48:42 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T04:48:43.054 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:48:42 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:48:43.054 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:48:42 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T04:48:43.054 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:48:42 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:48:43.054 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:48:42 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:48:43.055 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:48:42 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:48:43.113 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:48:42 smithi115 conmon[90393]: 2022-05-14T04:48:42.700+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:48:43.114 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:48:42 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:48:43.115 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:48:42 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:48:43.115 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:48:42 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:48:43.115 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:48:42 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:48:43.115 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:48:42 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 19255 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:48:43.116 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:48:42 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:48:43.116 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T04:48:43.116 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:48:43.116 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T04:48:43.117 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:48:43.117 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:48:43.117 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:48:43.117 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T04:48:43.118 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:48:43.118 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T04:48:43.118 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:48:43.118 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T04:48:43.118 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:48:43.119 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T04:48:43.119 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:48:43.119 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T04:48:43.119 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:48:43.119 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:48:43.120 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:48:43.120 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:48:42 smithi115 ceph-mon[78593]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T04:48:43.950 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:48:43 smithi115 conmon[90393]: 2022-05-14T04:48:43.669+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:48:44.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:48:43 smithi047 ceph-mon[77147]: pgmap v9979: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:48:44.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:48:43 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:48:44.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:48:43 smithi115 ceph-mon[78593]: pgmap v9979: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:48:44.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:48:43 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:48:44.952 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:48:44 smithi115 conmon[90393]: 2022-05-14T04:48:44.668+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:48:45.301 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:48:44 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:48:45.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:48:44 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:48:45.957 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:48:45 smithi115 conmon[90393]: 2022-05-14T04:48:45.655+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:48:46.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:48:45 smithi047 ceph-mon[77147]: pgmap v9980: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:48:46.295 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:48:45 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:48:46.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:48:45 smithi047 conmon[78521]: 2022-05-14T04:48:45.808+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:48:46.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:48:45 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T04:48:46.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:48:45 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:48:46.297 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:48:45 smithi047 conmon[78521]: yield 2022-05-14T04:48:46.297 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:48:45 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:48:46.297 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:48:45 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:48:46.297 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:48:45 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:48:46.298 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:48:45 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T04:48:46.298 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:48:45 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:48:46.298 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:48:45 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T04:48:46.298 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:48:45 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:48:46.299 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:48:45 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T04:48:46.299 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:48:45 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:48:46.299 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:48:45 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T04:48:46.299 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:48:45 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:48:46.299 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:48:45 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T04:48:46.300 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:48:45 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:48:46.300 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:48:45 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:48:46.300 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:48:45 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:48:46.300 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:48:45 smithi047 conmon[78521]: 2022-05-14T04:48:45.809+0000 7f533ef76700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T04:48:46.301 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:48:45 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T04:48:46.301 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:48:45 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:48:46.301 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:48:45 smithi047 conmon[78521]: yield 2022-05-14T04:48:46.301 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:48:45 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:48:46.302 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:48:45 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:48:46.302 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:48:45 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:48:46.302 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:48:45 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T04:48:46.302 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:48:45 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:48:46.302 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:48:45 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T04:48:46.303 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:48:45 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:48:46.303 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:48:45 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T04:48:46.303 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:48:45 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:48:46.303 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:48:45 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T04:48:46.303 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:48:45 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:48:46.304 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:48:45 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T04:48:46.304 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:48:45 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:48:46.304 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:48:45 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:48:46.304 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:48:45 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:48:46.305 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:48:45 smithi047 conmon[78521]: 2022-05-14T04:48:46.305 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:48:45 smithi047 conmon[78521]: During handling of the above exception, another exception occurred: 2022-05-14T04:48:46.305 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:48:45 smithi047 conmon[78521]: 2022-05-14T04:48:46.306 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:48:45 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T04:48:46.306 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:48:45 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T04:48:46.306 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:48:45 smithi047 conmon[78521]: return f(*arg) 2022-05-14T04:48:46.307 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:48:45 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T04:48:46.307 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:48:45 smithi047 conmon[78521]: self._write_client_files(client_files, host) 2022-05-14T04:48:46.307 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:48:45 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T04:48:46.307 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:48:45 smithi047 conmon[78521]: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T04:48:46.308 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:48:45 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T04:48:46.308 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:48:45 smithi047 conmon[78521]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T04:48:46.308 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:48:45 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T04:48:46.308 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:48:45 smithi047 conmon[78521]: return self.event_loop.get_result(coro) 2022-05-14T04:48:46.308 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:48:45 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T04:48:46.309 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:48:45 smithi047 conmon[78521]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T04:48:46.309 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:48:45 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T04:48:46.309 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:48:45 smithi047 conmon[78521]: return self.__get_result() 2022-05-14T04:48:46.309 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:48:45 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T04:48:46.309 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:48:45 smithi047 conmon[78521]: raise self._exception 2022-05-14T04:48:46.310 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:48:45 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T04:48:46.310 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:48:45 smithi047 conmon[78521]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T04:48:46.310 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:48:45 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T04:48:46.310 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:48:45 smithi047 conmon[78521]: conn = await self._remote_connection(host, addr) 2022-05-14T04:48:46.311 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:48:45 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T04:48:46.311 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:48:45 smithi047 conmon[78521]: raise 2022-05-14T04:48:46.311 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:48:45 smithi047 conmon[78521]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T04:48:46.311 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:48:45 smithi047 conmon[78521]: self.gen.throw(type, value, traceback) 2022-05-14T04:48:46.311 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:48:45 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T04:48:46.312 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:48:45 smithi047 conmon[78521]: raise OrchestratorError(msg) 2022-05-14T04:48:46.312 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:48:45 smithi047 conmon[78521]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:48:46.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:48:45 smithi115 ceph-mon[78593]: pgmap v9980: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:48:46.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:48:45 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:48:46.959 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:48:46 smithi115 conmon[90393]: 2022-05-14T04:48:46.696+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:48:47.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:48:46 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:48:47.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T04:48:47.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:48:47.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T04:48:47.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:48:47.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:48:47.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:48:47.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T04:48:47.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:48:47.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T04:48:47.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:48:47.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T04:48:47.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:48:47.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T04:48:47.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:48:47.298 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T04:48:47.298 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:48:47.298 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:48:47.298 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:48:47.299 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:48:46 smithi047 ceph-mon[77147]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T04:48:47.299 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T04:48:47.299 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:48:47.299 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T04:48:47.299 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:48:47.300 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:48:47.300 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:48:47.300 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T04:48:47.300 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:48:47.301 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T04:48:47.301 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:48:47.301 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T04:48:47.302 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:48:47.302 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T04:48:47.303 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:48:47.303 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T04:48:47.303 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:48:47.303 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:48:47.304 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:48:47.305 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T04:48:47.306 INFO:journalctl@ceph.mon.b.smithi047.stdout: During handling of the above exception, another exception occurred: 2022-05-14T04:48:47.306 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T04:48:47.306 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T04:48:47.306 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T04:48:47.306 INFO:journalctl@ceph.mon.b.smithi047.stdout: return f(*arg) 2022-05-14T04:48:47.307 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T04:48:47.307 INFO:journalctl@ceph.mon.b.smithi047.stdout: self._write_client_files(client_files, host) 2022-05-14T04:48:47.307 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T04:48:47.307 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T04:48:47.308 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T04:48:47.308 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T04:48:47.308 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T04:48:47.308 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.event_loop.get_result(coro) 2022-05-14T04:48:47.308 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T04:48:47.309 INFO:journalctl@ceph.mon.b.smithi047.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T04:48:47.309 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T04:48:47.309 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.__get_result() 2022-05-14T04:48:47.309 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T04:48:47.310 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise self._exception 2022-05-14T04:48:47.310 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T04:48:47.310 INFO:journalctl@ceph.mon.b.smithi047.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T04:48:47.310 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T04:48:47.310 INFO:journalctl@ceph.mon.b.smithi047.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T04:48:47.311 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T04:48:47.311 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise 2022-05-14T04:48:47.311 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T04:48:47.311 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.gen.throw(type, value, traceback) 2022-05-14T04:48:47.311 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T04:48:47.312 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OrchestratorError(msg) 2022-05-14T04:48:47.312 INFO:journalctl@ceph.mon.b.smithi047.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:48:47.312 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:48:46 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:48:47.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:48:46 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:48:47.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T04:48:47.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:48:47.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T04:48:47.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:48:47.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:48:47.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:48:47.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T04:48:47.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:48:47.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T04:48:47.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:48:47.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T04:48:47.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:48:47.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T04:48:47.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:48:47.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T04:48:47.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:48:47.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:48:47.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:48:47.364 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:48:46 smithi115 ceph-mon[78593]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T04:48:47.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T04:48:47.365 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:48:47.365 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T04:48:47.365 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:48:47.365 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:48:47.366 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:48:47.366 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T04:48:47.367 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:48:47.367 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T04:48:47.367 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:48:47.367 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T04:48:47.368 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:48:47.368 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T04:48:47.368 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:48:47.368 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T04:48:47.368 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:48:47.369 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:48:47.369 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:48:47.369 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T04:48:47.369 INFO:journalctl@ceph.mon.c.smithi115.stdout: During handling of the above exception, another exception occurred: 2022-05-14T04:48:47.370 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T04:48:47.370 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T04:48:47.370 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T04:48:47.370 INFO:journalctl@ceph.mon.c.smithi115.stdout: return f(*arg) 2022-05-14T04:48:47.370 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T04:48:47.371 INFO:journalctl@ceph.mon.c.smithi115.stdout: self._write_client_files(client_files, host) 2022-05-14T04:48:47.371 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T04:48:47.371 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T04:48:47.371 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T04:48:47.371 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T04:48:47.372 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T04:48:47.372 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.event_loop.get_result(coro) 2022-05-14T04:48:47.372 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T04:48:47.372 INFO:journalctl@ceph.mon.c.smithi115.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T04:48:47.372 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T04:48:47.373 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.__get_result() 2022-05-14T04:48:47.373 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T04:48:47.373 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise self._exception 2022-05-14T04:48:47.373 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T04:48:47.373 INFO:journalctl@ceph.mon.c.smithi115.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T04:48:47.374 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T04:48:47.374 INFO:journalctl@ceph.mon.c.smithi115.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T04:48:47.374 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T04:48:47.374 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise 2022-05-14T04:48:47.375 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T04:48:47.375 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.gen.throw(type, value, traceback) 2022-05-14T04:48:47.375 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T04:48:47.375 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OrchestratorError(msg) 2022-05-14T04:48:47.375 INFO:journalctl@ceph.mon.c.smithi115.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:48:47.376 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:48:46 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:48:47.964 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:48:47 smithi115 conmon[90393]: 2022-05-14T04:48:47.661+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:48:48.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:48:47 smithi047 ceph-mon[77147]: pgmap v9981: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:48:48.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:48:47 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:48:48.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:48:47 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 19260 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:48:48.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:48:47 smithi115 ceph-mon[78593]: pgmap v9981: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:48:48.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:48:47 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:48:48.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:48:47 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 19260 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:48:48.965 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:48:48 smithi115 conmon[90393]: 2022-05-14T04:48:48.689+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:48:49.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:48:48 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:48:49.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:48:48 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:48:49.973 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:48:49 smithi115 conmon[90393]: 2022-05-14T04:48:49.695+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:48:50.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:48:49 smithi047 ceph-mon[77147]: pgmap v9982: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:48:50.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:48:49 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:48:50.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:48:49 smithi115 ceph-mon[78593]: pgmap v9982: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:48:50.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:48:49 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:48:50.972 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:48:50 smithi115 conmon[90393]: 2022-05-14T04:48:50.711+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:48:51.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:48:50 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:48:51.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:48:50 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:48:51.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:48:50 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:48:51.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:48:50 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:48:52.110 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:48:51 smithi115 conmon[90393]: 2022-05-14T04:48:51.725+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:48:52.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:48:51 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:48:52.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:48:51 smithi115 ceph-mon[78593]: pgmap v9983: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:48:52.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:48:51 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:48:52.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:48:51 smithi047 ceph-mon[77147]: pgmap v9983: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:48:52.978 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:48:52 smithi115 conmon[90393]: 2022-05-14T04:48:52.720+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:48:53.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:48:52 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:48:53.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:48:52 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 19265 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:48:53.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:48:52 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:48:53.408 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:48:52 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 19265 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:48:53.985 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:48:53 smithi115 conmon[90393]: 2022-05-14T04:48:53.707+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:48:54.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:48:53 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:48:54.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:48:53 smithi047 ceph-mon[77147]: pgmap v9984: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:48:54.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:48:53 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:48:54.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:48:53 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:48:54.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:48:53 smithi115 ceph-mon[78593]: pgmap v9984: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:48:54.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:48:53 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:48:55.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:48:54 smithi115 conmon[90393]: 2022-05-14T04:48:54.739+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:48:55.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:48:54 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:48:55.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:48:54 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:48:55.296 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:48:54 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:48:55.297 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:48:54 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:48:56.110 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:48:55 smithi115 conmon[90393]: 2022-05-14T04:48:55.759+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:48:56.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:48:55 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:48:56.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:48:55 smithi115 ceph-mon[78593]: pgmap v9985: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:48:56.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:48:55 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:48:56.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:48:55 smithi047 ceph-mon[77147]: pgmap v9985: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:48:57.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:48:56 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:48:57.110 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:48:56 smithi115 conmon[90393]: 2022-05-14T04:48:56.785+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:48:57.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:48:56 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:48:58.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:48:57 smithi115 conmon[90393]: 2022-05-14T04:48:57.774+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:48:58.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:48:57 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:48:58.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:48:57 smithi115 ceph-mon[78593]: pgmap v9986: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:48:58.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:48:57 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 19270 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:48:58.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:48:57 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:48:58.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:48:57 smithi047 ceph-mon[77147]: pgmap v9986: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:48:58.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:48:57 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 19270 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:48:58.998 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:48:58 smithi115 conmon[90393]: 2022-05-14T04:48:58.744+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:48:59.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:48:58 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:48:59.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:48:58 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:48:59.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:48:58 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:48:59.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:48:58 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:48:59.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:48:58 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:48:59.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:48:58 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:49:00.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:49:00 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:49:00.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:49:00 smithi115 ceph-mon[78593]: pgmap v9987: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:49:00.111 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:48:59 smithi115 conmon[90393]: 2022-05-14T04:48:59.786+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:49:00.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:49:00 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:49:00.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:49:00 smithi047 ceph-mon[77147]: pgmap v9987: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:49:01.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:49:01 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:49:01.110 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:49:00 smithi115 conmon[90393]: 2022-05-14T04:49:00.759+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:49:01.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:49:01 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:49:02.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:49:02 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:49:02.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:49:02 smithi115 ceph-mon[78593]: pgmap v9988: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:49:02.110 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:49:01 smithi115 conmon[90393]: 2022-05-14T04:49:01.781+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:49:02.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:49:02 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:49:02.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:49:02 smithi047 ceph-mon[77147]: pgmap v9988: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:49:03.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:49:02 smithi115 conmon[90393]: 2022-05-14T04:49:02.825+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:49:03.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:49:03 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:49:03.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:49:03 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:49:03.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:49:03 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:49:03.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:49:03 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 19275 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:49:03.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:49:03 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:49:03.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:49:03 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:49:03.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:49:03 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:49:03.296 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:49:03 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 19275 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:49:04.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:49:04 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:49:04.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:49:04 smithi047 ceph-mon[77147]: pgmap v9989: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:49:04.363 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:49:03 smithi115 conmon[90393]: 2022-05-14T04:49:03.860+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:49:04.369 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:49:04 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:49:04.369 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:49:04 smithi115 ceph-mon[78593]: pgmap v9989: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:49:05.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:49:05 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:49:05.110 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:49:04 smithi115 conmon[90393]: 2022-05-14T04:49:04.818+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:49:05.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:49:05 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:49:06.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:49:06 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:49:06.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:49:06 smithi115 ceph-mon[78593]: pgmap v9990: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:49:06.110 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:49:05 smithi115 conmon[90393]: 2022-05-14T04:49:05.852+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:49:06.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:49:06 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:49:06.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:49:06 smithi047 ceph-mon[77147]: pgmap v9990: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:49:07.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:49:06 smithi115 conmon[90393]: 2022-05-14T04:49:06.803+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:49:07.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:49:07 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:49:07.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:49:07 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:49:08.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:49:07 smithi115 conmon[90393]: 2022-05-14T04:49:07.847+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:49:08.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:49:08 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:49:08.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:49:08 smithi115 ceph-mon[78593]: pgmap v9991: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:49:08.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:49:08 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 19280 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:49:08.298 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:49:08 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:49:08.298 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:49:08 smithi047 ceph-mon[77147]: pgmap v9991: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:49:08.298 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:49:08 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 19280 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:49:09.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:49:09 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:49:09.110 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:49:08 smithi115 conmon[90393]: 2022-05-14T04:49:08.801+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:49:09.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:49:09 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:49:10.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:49:10 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:49:10.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:49:10 smithi115 ceph-mon[78593]: pgmap v9992: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:49:10.112 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:49:09 smithi115 conmon[90393]: 2022-05-14T04:49:09.794+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:49:10.302 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:49:10 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:49:10.302 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:49:10 smithi047 ceph-mon[77147]: pgmap v9992: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:49:11.040 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:49:10 smithi115 conmon[90393]: 2022-05-14T04:49:10.778+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:49:11.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:49:11 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:49:11.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:49:11 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:49:11.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:49:11 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:49:11.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:49:11 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:49:11.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:49:11 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:49:11.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:49:11 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:49:12.045 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:49:11 smithi115 conmon[90393]: 2022-05-14T04:49:11.744+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:49:12.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:49:12 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:49:12.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:49:12 smithi115 ceph-mon[78593]: pgmap v9993: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:49:12.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:49:12 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:49:12.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:49:12 smithi047 ceph-mon[77147]: pgmap v9993: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:49:13.046 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:49:12 smithi115 conmon[90393]: 2022-05-14T04:49:12.783+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:49:13.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:49:13 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:49:13.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:49:13 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 19285 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:49:13.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:49:13 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:49:13.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:49:13 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 19285 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:49:14.052 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:49:13 smithi115 conmon[90393]: 2022-05-14T04:49:13.758+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:49:14.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:49:14 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:49:14.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:49:14 smithi115 ceph-mon[78593]: pgmap v9994: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:49:14.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:49:14 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:49:14.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:49:14 smithi047 ceph-mon[77147]: pgmap v9994: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:49:15.054 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:49:14 smithi115 conmon[90393]: 2022-05-14T04:49:14.740+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:49:15.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:49:15 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:49:15.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:49:15 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:49:15.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:49:15 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:49:15.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:49:15 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T04:49:15.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:49:15 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T04:49:15.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:49:15 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:49:15.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:49:15 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:49:15.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:49:15 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:49:15.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:49:15 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T04:49:15.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:49:15 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T04:49:16.059 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:49:15 smithi115 conmon[90393]: 2022-05-14T04:49:15.730+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:49:16.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:49:16 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:49:16.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:49:16 smithi115 ceph-mon[78593]: pgmap v9995: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:49:16.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:49:16 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:49:16.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:49:16 smithi047 ceph-mon[77147]: pgmap v9995: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:49:17.060 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:49:16 smithi115 conmon[90393]: 2022-05-14T04:49:16.698+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:49:17.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:49:17 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:49:17.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:49:17 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:49:17.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:49:17 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:49:17.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:49:17 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:49:18.066 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:49:17 smithi115 conmon[90393]: 2022-05-14T04:49:17.663+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:49:18.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:49:18 smithi115 ceph-mon[78593]: pgmap v9996: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:49:18.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:49:18 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:49:18.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:49:18 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 19290 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:49:18.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:49:18 smithi047 ceph-mon[77147]: pgmap v9996: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:49:18.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:49:18 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:49:18.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:49:18 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 19290 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:49:19.067 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:49:18 smithi115 conmon[90393]: 2022-05-14T04:49:18.616+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:49:19.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:49:19 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:49:19.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:49:19 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:49:19.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:49:19 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:49:19.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:49:19 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:49:19.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:49:19 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:49:19.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:49:19 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:49:20.074 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:49:19 smithi115 conmon[90393]: 2022-05-14T04:49:19.643+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:49:20.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:49:20 smithi115 ceph-mon[78593]: pgmap v9997: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:49:20.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:49:20 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:49:20.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:49:20 smithi047 ceph-mon[77147]: pgmap v9997: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:49:20.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:49:20 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:49:21.077 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:49:20 smithi115 conmon[90393]: 2022-05-14T04:49:20.610+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:49:21.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:49:21 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:49:21.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:49:21 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:49:21.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:49:21 smithi115 conmon[90393]: 2022-05-14T04:49:21.579+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:49:22.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:49:22 smithi115 ceph-mon[78593]: pgmap v9998: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:49:22.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:49:22 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:49:22.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:49:22 smithi047 ceph-mon[77147]: pgmap v9998: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:49:22.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:49:22 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:49:22.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:49:22 smithi115 conmon[90393]: 2022-05-14T04:49:22.552+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:49:23.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:49:23 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:49:23.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:49:23 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:49:23.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:49:23 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:49:23.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:49:23 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 19295 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:49:23.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:49:23 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:49:23.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:49:23 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:49:23.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:49:23 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:49:23.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:49:23 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 19295 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:49:23.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:49:23 smithi115 conmon[90393]: 2022-05-14T04:49:23.576+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:49:24.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:49:24 smithi115 ceph-mon[78593]: pgmap v9999: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:49:24.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:49:24 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:49:24.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:49:24 smithi047 ceph-mon[77147]: pgmap v9999: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:49:24.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:49:24 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:49:24.862 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:49:24 smithi115 conmon[90393]: 2022-05-14T04:49:24.567+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:49:25.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:49:25 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:49:25.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:49:25 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:49:25.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:49:25 smithi115 conmon[90393]: 2022-05-14T04:49:25.586+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:49:26.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:49:26 smithi115 ceph-mon[78593]: pgmap v10000: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:49:26.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:49:26 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:49:26.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:49:26 smithi047 ceph-mon[77147]: pgmap v10000: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:49:26.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:49:26 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:49:26.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:49:26 smithi115 conmon[90393]: 2022-05-14T04:49:26.568+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:49:27.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:49:27 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:49:27.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:49:27 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:49:27.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:49:27 smithi115 conmon[90393]: 2022-05-14T04:49:27.582+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:49:28.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:49:28 smithi115 ceph-mon[78593]: pgmap v10001: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:49:28.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:49:28 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:49:28.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:49:28 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 19300 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:49:28.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:49:28 smithi047 ceph-mon[77147]: pgmap v10001: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:49:28.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:49:28 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:49:28.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:49:28 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 19300 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:49:28.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:49:28 smithi115 conmon[90393]: 2022-05-14T04:49:28.591+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:49:29.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:49:29 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:49:29.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:49:29 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:49:29.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:49:29 smithi115 conmon[90393]: 2022-05-14T04:49:29.546+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:49:30.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:49:30 smithi115 ceph-mon[78593]: pgmap v10002: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:49:30.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:49:30 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:49:30.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:49:30 smithi047 ceph-mon[77147]: pgmap v10002: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:49:30.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:49:30 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:49:30.860 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:49:30 smithi115 conmon[90393]: 2022-05-14T04:49:30.581+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:49:31.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:49:31 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:49:31.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:49:31 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:49:31.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:49:31 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:49:31.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:49:31 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:49:31.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:49:31 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:49:31.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:49:31 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:49:32.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:49:31 smithi115 conmon[90393]: 2022-05-14T04:49:31.621+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:49:32.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:49:32 smithi047 ceph-mon[77147]: pgmap v10003: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:49:32.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:49:32 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:49:32.579 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:49:32 smithi115 ceph-mon[78593]: pgmap v10003: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:49:32.579 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:49:32 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:49:32.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:49:32 smithi115 conmon[90393]: 2022-05-14T04:49:32.575+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:49:33.533 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:49:33 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:49:33.533 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:49:33 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 19305 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:49:33.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:49:33 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:49:33.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:49:33 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 19305 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:49:33.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:49:33 smithi115 conmon[90393]: 2022-05-14T04:49:33.529+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:49:34.512 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:49:34 smithi115 ceph-mon[78593]: pgmap v10004: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:49:34.512 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:49:34 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:49:34.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:49:34 smithi047 ceph-mon[77147]: pgmap v10004: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:49:34.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:49:34 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:49:34.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:49:34 smithi115 conmon[90393]: 2022-05-14T04:49:34.509+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:49:35.496 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:49:35 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:49:35.496 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:49:35 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:49:35.497 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:49:35 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:49:35.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:49:35 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:49:35.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:49:35 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:49:35.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:49:35 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:49:35.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:49:35 smithi115 conmon[90393]: 2022-05-14T04:49:35.493+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:49:36.507 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:49:36 smithi115 ceph-mon[78593]: pgmap v10005: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:49:36.508 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:49:36 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:49:36.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:49:36 smithi047 ceph-mon[77147]: pgmap v10005: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:49:36.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:49:36 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:49:36.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:49:36 smithi115 conmon[90393]: 2022-05-14T04:49:36.504+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:49:37.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:49:37 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:49:37.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:49:37 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:49:37.860 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:49:37 smithi115 conmon[90393]: 2022-05-14T04:49:37.537+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:49:38.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:49:38 smithi047 ceph-mon[77147]: pgmap v10006: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:49:38.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:49:38 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:49:38.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:49:38 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 19310 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:49:38.576 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:49:38 smithi115 ceph-mon[78593]: pgmap v10006: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:49:38.576 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:49:38 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:49:38.577 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:49:38 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 19310 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:49:38.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:49:38 smithi115 conmon[90393]: 2022-05-14T04:49:38.573+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:49:39.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:49:39 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:49:39.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:49:39 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:49:39.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:49:39 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:49:39.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:49:39 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:49:39.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:49:39 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:49:39.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:49:39 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:49:40.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:49:39 smithi115 conmon[90393]: 2022-05-14T04:49:39.609+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:49:40.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:49:40 smithi047 ceph-mon[77147]: pgmap v10007: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:49:40.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:49:40 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:49:40.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:49:40 smithi115 ceph-mon[78593]: pgmap v10007: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:49:40.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:49:40 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:49:41.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:49:40 smithi115 conmon[90393]: 2022-05-14T04:49:40.615+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:49:41.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:49:41 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:49:41.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:49:41 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:49:42.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:49:41 smithi115 conmon[90393]: 2022-05-14T04:49:41.664+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:49:42.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:49:42 smithi047 ceph-mon[77147]: pgmap v10008: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:49:42.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:49:42 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:49:42.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:49:42 smithi115 ceph-mon[78593]: pgmap v10008: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:49:42.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:49:42 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:49:43.111 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:49:42 smithi115 conmon[90393]: 2022-05-14T04:49:42.683+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:49:43.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:49:43 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:49:43.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:49:43 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:49:43.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:49:43 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:49:43.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:49:43 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 19315 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:49:43.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:49:43 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:49:43.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:49:43 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:49:43.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:49:43 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:49:43.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:49:43 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 19315 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:49:44.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:49:43 smithi115 conmon[90393]: 2022-05-14T04:49:43.672+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:49:44.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:49:44 smithi047 ceph-mon[77147]: pgmap v10009: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:49:44.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:49:44 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:49:44.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:49:44 smithi115 ceph-mon[78593]: pgmap v10009: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:49:44.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:49:44 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:49:45.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:49:44 smithi115 conmon[90393]: 2022-05-14T04:49:44.640+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:49:45.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:49:45 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:49:45.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:49:45 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:49:45.942 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:49:45 smithi115 conmon[90393]: 2022-05-14T04:49:45.687+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:49:46.326 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:49:46 smithi115 ceph-mon[78593]: pgmap v10010: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:49:46.327 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:49:46 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:49:46.327 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:49:46 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T04:49:46.328 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:49:46 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T04:49:46.328 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:49:46 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T04:49:46.419 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:49:46 smithi047 ceph-mon[77147]: pgmap v10010: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:49:46.420 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:49:46 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:49:46.420 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:49:46 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T04:49:46.420 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:49:46 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T04:49:46.421 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:49:46 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T04:49:46.793 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:49:46 smithi047 conmon[78521]: 2022-05-14T04:49:46.416+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:49:46.793 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:49:46 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T04:49:46.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:49:46 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:49:46.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:49:46 smithi047 conmon[78521]: yield 2022-05-14T04:49:46.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:49:46 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:49:46.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:49:46 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:49:46.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:49:46 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:49:46.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:49:46 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T04:49:46.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:49:46 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:49:46.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:49:46 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T04:49:46.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:49:46 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:49:46.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:49:46 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T04:49:46.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:49:46 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:49:46.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:49:46 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T04:49:46.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:49:46 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:49:46.797 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:49:46 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T04:49:46.797 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:49:46 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:49:46.797 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:49:46 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:49:46.797 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:49:46 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:49:47.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:49:46 smithi115 conmon[90393]: 2022-05-14T04:49:46.703+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:49:47.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:49:47 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:49:47.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T04:49:47.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:49:47.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T04:49:47.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:49:47.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:49:47.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:49:47.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T04:49:47.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:49:47.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T04:49:47.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:49:47.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T04:49:47.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:49:47.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T04:49:47.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:49:47.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T04:49:47.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:49:47.548 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:49:47.548 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:49:47.548 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:49:47 smithi047 ceph-mon[77147]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T04:49:47.548 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:49:47 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:49:47.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:49:47 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:49:47.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T04:49:47.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:49:47.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T04:49:47.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:49:47.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:49:47.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:49:47.613 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T04:49:47.613 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:49:47.613 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T04:49:47.613 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:49:47.613 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T04:49:47.614 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:49:47.614 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T04:49:47.614 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:49:47.614 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T04:49:47.615 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:49:47.615 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:49:47.615 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:49:47.615 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:49:47 smithi115 ceph-mon[78593]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T04:49:47.615 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:49:47 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:49:48.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:49:47 smithi115 conmon[90393]: 2022-05-14T04:49:47.657+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:49:48.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:49:48 smithi047 ceph-mon[77147]: pgmap v10011: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:49:48.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:49:48 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:49:48.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:49:48 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 19320 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:49:48.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:49:48 smithi115 ceph-mon[78593]: pgmap v10011: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:49:48.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:49:48 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:49:48.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:49:48 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 19320 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:49:49.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:49:48 smithi115 conmon[90393]: 2022-05-14T04:49:48.685+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:49:49.491 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:49:49 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:49:49.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:49:49 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:49:49.793 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:49:49 smithi047 conmon[78521]: 2022-05-14T04:49:49.488+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:49:49.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:49:49 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T04:49:49.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:49:49 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:49:49.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:49:49 smithi047 conmon[78521]: yield 2022-05-14T04:49:49.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:49:49 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:49:49.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:49:49 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:49:49.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:49:49 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:49:49.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:49:49 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T04:49:49.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:49:49 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:49:49.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:49:49 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T04:49:49.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:49:49 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:49:49.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:49:49 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T04:49:49.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:49:49 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:49:49.797 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:49:49 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T04:49:49.797 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:49:49 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:49:49.797 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:49:49 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T04:49:49.797 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:49:49 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:49:49.798 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:49:49 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:49:49.798 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:49:49 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:49:49.798 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:49:49 smithi047 conmon[78521]: 2022-05-14T04:49:49.491+0000 7f5344781700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T04:49:49.798 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:49:49 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T04:49:49.798 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:49:49 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:49:49.799 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:49:49 smithi047 conmon[78521]: yield 2022-05-14T04:49:49.799 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:49:49 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:49:49.799 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:49:49 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:49:49.799 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:49:49 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:49:49.800 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:49:49 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T04:49:49.800 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:49:49 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:49:49.800 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:49:49 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T04:49:49.800 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:49:49 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:49:49.800 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:49:49 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T04:49:49.801 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:49:49 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:49:49.801 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:49:49 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T04:49:49.801 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:49:49 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:49:49.801 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:49:49 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T04:49:49.802 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:49:49 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:49:49.802 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:49:49 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:49:49.802 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:49:49 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:49:49.802 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:49:49 smithi047 conmon[78521]: 2022-05-14T04:49:49.803 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:49:49 smithi047 conmon[78521]: During handling of the above exception, another exception occurred: 2022-05-14T04:49:49.803 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:49:49 smithi047 conmon[78521]: 2022-05-14T04:49:49.803 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:49:49 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T04:49:49.803 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:49:49 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T04:49:49.803 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:49:49 smithi047 conmon[78521]: return f(*arg) 2022-05-14T04:49:49.804 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:49:49 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T04:49:49.804 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:49:49 smithi047 conmon[78521]: self._write_client_files(client_files, host) 2022-05-14T04:49:49.804 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:49:49 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T04:49:49.804 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:49:49 smithi047 conmon[78521]: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T04:49:49.805 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:49:49 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T04:49:49.805 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:49:49 smithi047 conmon[78521]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T04:49:49.805 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:49:49 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T04:49:49.805 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:49:49 smithi047 conmon[78521]: return self.event_loop.get_result(coro) 2022-05-14T04:49:49.805 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:49:49 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T04:49:49.806 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:49:49 smithi047 conmon[78521]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T04:49:49.806 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:49:49 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T04:49:49.806 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:49:49 smithi047 conmon[78521]: return self.__get_result() 2022-05-14T04:49:49.806 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:49:49 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T04:49:49.807 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:49:49 smithi047 conmon[78521]: raise self._exception 2022-05-14T04:49:49.807 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:49:49 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T04:49:49.807 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:49:49 smithi047 conmon[78521]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T04:49:49.807 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:49:49 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T04:49:49.807 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:49:49 smithi047 conmon[78521]: conn = await self._remote_connection(host, addr) 2022-05-14T04:49:49.808 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:49:49 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T04:49:49.808 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:49:49 smithi047 conmon[78521]: raise 2022-05-14T04:49:49.808 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:49:49 smithi047 conmon[78521]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T04:49:49.808 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:49:49 smithi047 conmon[78521]: self.gen.throw(type, value, traceback) 2022-05-14T04:49:49.808 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:49:49 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T04:49:49.809 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:49:49 smithi047 conmon[78521]: raise OrchestratorError(msg) 2022-05-14T04:49:49.809 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:49:49 smithi047 conmon[78521]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:49:50.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:49:49 smithi115 conmon[90393]: 2022-05-14T04:49:49.694+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:49:50.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:49:50 smithi047 ceph-mon[77147]: pgmap v10012: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:49:50.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:49:50 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:49:50.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T04:49:50.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:49:50.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T04:49:50.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:49:50.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:49:50.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:49:50.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T04:49:50.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:49:50.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T04:49:50.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:49:50.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T04:49:50.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:49:50.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T04:49:50.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:49:50.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T04:49:50.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:49:50.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:49:50.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:49:50.548 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:49:50 smithi047 ceph-mon[77147]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T04:49:50.548 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T04:49:50.548 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:49:50.548 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T04:49:50.548 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:49:50.549 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:49:50.549 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:49:50.549 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T04:49:50.549 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:49:50.550 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T04:49:50.550 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:49:50.550 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T04:49:50.550 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:49:50.550 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T04:49:50.551 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:49:50.551 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T04:49:50.551 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:49:50.551 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:49:50.551 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:49:50.552 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T04:49:50.552 INFO:journalctl@ceph.mon.b.smithi047.stdout: During handling of the above exception, another exception occurred: 2022-05-14T04:49:50.552 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T04:49:50.552 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T04:49:50.553 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T04:49:50.553 INFO:journalctl@ceph.mon.b.smithi047.stdout: return f(*arg) 2022-05-14T04:49:50.553 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T04:49:50.553 INFO:journalctl@ceph.mon.b.smithi047.stdout: self._write_client_files(client_files, host) 2022-05-14T04:49:50.553 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T04:49:50.554 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T04:49:50.554 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T04:49:50.554 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T04:49:50.554 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T04:49:50.555 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.event_loop.get_result(coro) 2022-05-14T04:49:50.555 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T04:49:50.555 INFO:journalctl@ceph.mon.b.smithi047.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T04:49:50.555 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T04:49:50.555 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.__get_result() 2022-05-14T04:49:50.556 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T04:49:50.556 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise self._exception 2022-05-14T04:49:50.556 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T04:49:50.556 INFO:journalctl@ceph.mon.b.smithi047.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T04:49:50.556 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T04:49:50.557 INFO:journalctl@ceph.mon.b.smithi047.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T04:49:50.557 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T04:49:50.557 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise 2022-05-14T04:49:50.557 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T04:49:50.557 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.gen.throw(type, value, traceback) 2022-05-14T04:49:50.558 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T04:49:50.558 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OrchestratorError(msg) 2022-05-14T04:49:50.558 INFO:journalctl@ceph.mon.b.smithi047.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:49:50.558 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:49:50 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:49:50.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:49:50 smithi115 ceph-mon[78593]: pgmap v10012: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:49:50.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:49:50 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:49:50.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T04:49:50.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:49:50.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T04:49:50.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:49:50.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:49:50.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:49:50.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T04:49:50.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:49:50.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T04:49:50.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:49:50.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T04:49:50.613 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:49:50.613 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T04:49:50.613 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:49:50.613 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T04:49:50.614 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:49:50.614 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:49:50.614 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:49:50.614 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:49:50 smithi115 ceph-mon[78593]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T04:49:50.614 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T04:49:50.615 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:49:50.615 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T04:49:50.615 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:49:50.615 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:49:50.616 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:49:50.616 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T04:49:50.616 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:49:50.616 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T04:49:50.616 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:49:50.617 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T04:49:50.617 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:49:50.617 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T04:49:50.617 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:49:50.618 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T04:49:50.618 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:49:50.618 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:49:50.618 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:49:50.618 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T04:49:50.619 INFO:journalctl@ceph.mon.c.smithi115.stdout: During handling of the above exception, another exception occurred: 2022-05-14T04:49:50.619 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T04:49:50.619 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T04:49:50.619 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T04:49:50.620 INFO:journalctl@ceph.mon.c.smithi115.stdout: return f(*arg) 2022-05-14T04:49:50.620 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T04:49:50.620 INFO:journalctl@ceph.mon.c.smithi115.stdout: self._write_client_files(client_files, host) 2022-05-14T04:49:50.620 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T04:49:50.620 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T04:49:50.621 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T04:49:50.621 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T04:49:50.621 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T04:49:50.621 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.event_loop.get_result(coro) 2022-05-14T04:49:50.622 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T04:49:50.622 INFO:journalctl@ceph.mon.c.smithi115.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T04:49:50.622 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T04:49:50.622 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.__get_result() 2022-05-14T04:49:50.622 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T04:49:50.623 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise self._exception 2022-05-14T04:49:50.623 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T04:49:50.623 INFO:journalctl@ceph.mon.c.smithi115.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T04:49:50.623 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T04:49:50.624 INFO:journalctl@ceph.mon.c.smithi115.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T04:49:50.624 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T04:49:50.624 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise 2022-05-14T04:49:50.624 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T04:49:50.624 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.gen.throw(type, value, traceback) 2022-05-14T04:49:50.625 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T04:49:50.625 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OrchestratorError(msg) 2022-05-14T04:49:50.625 INFO:journalctl@ceph.mon.c.smithi115.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:49:50.625 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:49:50 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:49:51.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:49:50 smithi115 conmon[90393]: 2022-05-14T04:49:50.689+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:49:51.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:49:51 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:49:51.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:49:51 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:49:51.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:49:51 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:49:51.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:49:51 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:49:51.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:49:51 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:49:51.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:49:51 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:49:52.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:49:51 smithi115 conmon[90393]: 2022-05-14T04:49:51.727+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:49:52.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:49:52 smithi047 ceph-mon[77147]: pgmap v10013: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:49:52.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:49:52 smithi115 ceph-mon[78593]: pgmap v10013: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:49:53.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:49:52 smithi115 conmon[90393]: 2022-05-14T04:49:52.765+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:49:53.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:49:53 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:49:53.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:49:53 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 19325 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:49:53.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:49:53 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:49:53.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:49:53 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 19325 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:49:54.110 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:49:53 smithi115 conmon[90393]: 2022-05-14T04:49:53.733+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:49:54.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:49:54 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:49:54.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:49:54 smithi047 ceph-mon[77147]: pgmap v10014: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:49:54.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:49:54 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:49:54.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:49:54 smithi115 ceph-mon[78593]: pgmap v10014: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:49:55.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:49:54 smithi115 conmon[90393]: 2022-05-14T04:49:54.737+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:49:55.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:49:55 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:49:55.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:49:55 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:49:55.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:49:55 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:49:55.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:49:55 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:49:55.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:49:55 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:49:55.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:49:55 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:49:56.110 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:49:55 smithi115 conmon[90393]: 2022-05-14T04:49:55.764+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:49:56.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:49:56 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:49:56.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:49:56 smithi047 ceph-mon[77147]: pgmap v10015: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:49:56.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:49:56 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:49:56.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:49:56 smithi115 ceph-mon[78593]: pgmap v10015: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:49:57.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:49:56 smithi115 conmon[90393]: 2022-05-14T04:49:56.732+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:49:57.546 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:49:57 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:49:57.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:49:57 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:49:58.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:49:57 smithi115 conmon[90393]: 2022-05-14T04:49:57.761+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:49:58.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:49:58 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:49:58.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:49:58 smithi047 ceph-mon[77147]: pgmap v10016: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:49:58.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:49:58 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 19330 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:49:58.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:49:58 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:49:58.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:49:58 smithi115 ceph-mon[78593]: pgmap v10016: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:49:58.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:49:58 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 19330 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:49:59.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:49:58 smithi115 conmon[90393]: 2022-05-14T04:49:58.768+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:49:59.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:49:59 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:49:59.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:49:59 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:49:59.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:49:59 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:49:59.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:49:59 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:49:59.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:49:59 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:49:59.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:49:59 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:50:00.110 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:49:59 smithi115 conmon[90393]: 2022-05-14T04:49:59.732+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:50:00.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:50:00 smithi047 conmon[77123]: 2022-05-14T04:49:59.998+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : Health detail: HEALTH_ERR 1 filesystem is degraded; 1 mds daemon damaged; 1 MDSs are read only; 1 MDSs report slow metadata IOs; 1/3 mons down, quorum b,c; Reduced data availability: 1 pg inactive; 9 slow ops, oldest one blocked for 19335 sec, osd.5 has slow ops 2022-05-14T04:50:00.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:50:00 smithi047 conmon[77123]: 2022-05-14T04:49:59.998+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : [WRN] FS_DEGRADED: 1 filesystem is degraded 2022-05-14T04:50:00.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:50:00 smithi047 conmon[77123]: 2022-05-14T04:49:59.998+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : fs cephfs is degraded 2022-05-14T04:50:00.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:50:00 smithi047 conmon[77123]: 2022-05-14T04:49:59.998+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : [ERR] MDS_DAMAGE: 1 mds daemon damaged 2022-05-14T04:50:00.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:50:00 smithi047 conmon[77123]: 2022-05-14T04:49:59.998+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : fs cephfs mds.2 is damaged 2022-05-14T04:50:00.295 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:50:00 smithi047 conmon[77123]: 2022-05-14T04:49:59.998+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : [WRN] MDS_READ_ONLY: 1 MDSs are read only 2022-05-14T04:50:00.295 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:50:00 smithi047 conmon[77123]: 2022-05-14T04:49:59.998+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : mds.l(mds.0): MDS in read-only mode 2022-05-14T04:50:00.295 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:50:00 smithi047 conmon[77123]: 2022-05-14T04:49:59.998+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : [WRN] MDS_SLOW_METADATA_IO: 1 MDSs report slow metadata IOs 2022-05-14T04:50:00.295 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:50:00 smithi047 conmon[77123]: 2022-05-14T04:49:59.998+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : mds.l(mds.0): 4 slow metadata IOs are blocked > 180 secs, oldest blocked for 19634 secs 2022-05-14T04:50:00.295 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:50:00 smithi047 conmon[77123]: 2022-05-14T04:49:59.998+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : [WRN] MON_DOWN: 1/3 mons down, quorum b,c 2022-05-14T04:50:00.296 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:50:00 smithi047 conmon[77123]: 2022-05-14T04:49:59.998+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : mon.a (rank 0) addr [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] is down (out of quorum) 2022-05-14T04:50:00.296 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:50:00 smithi047 conmon[77123]: 2022-05-14T04:49:59.998+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : [WRN] PG_AVAILABILITY: Reduced data availability: 1 pg inactive 2022-05-14T04:50:00.296 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:50:00 smithi047 conmon[77123]: 2022-05-14T04:49:59.998+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : pg 2.1 is stuck inactive for 5h, current state unknown, last acting [] 2022-05-14T04:50:00.296 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:50:00 smithi047 conmon[77123]: 2022-05-14T04:49:59.998+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : [WRN] SLOW_OPS: 9 slow ops, oldest one blocked for 19335 sec, osd.5 has slow ops 2022-05-14T04:50:00.297 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:50:00 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:50:00.297 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:50:00 smithi047 ceph-mon[77147]: pgmap v10017: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:50:00.297 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:50:00 smithi047 ceph-mon[77147]: Health detail: HEALTH_ERR 1 filesystem is degraded; 1 mds daemon damaged; 1 MDSs are read only; 1 MDSs report slow metadata IOs; 1/3 mons down, quorum b,c; Reduced data availability: 1 pg inactive; 9 slow ops, oldest one blocked for 19335 sec, osd.5 has slow ops 2022-05-14T04:50:00.297 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:50:00 smithi047 ceph-mon[77147]: [WRN] FS_DEGRADED: 1 filesystem is degraded 2022-05-14T04:50:00.297 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:50:00 smithi047 ceph-mon[77147]: fs cephfs is degraded 2022-05-14T04:50:00.298 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:50:00 smithi047 ceph-mon[77147]: [ERR] MDS_DAMAGE: 1 mds daemon damaged 2022-05-14T04:50:00.298 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:50:00 smithi047 ceph-mon[77147]: fs cephfs mds.2 is damaged 2022-05-14T04:50:00.298 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:50:00 smithi047 ceph-mon[77147]: [WRN] MDS_READ_ONLY: 1 MDSs are read only 2022-05-14T04:50:00.298 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:50:00 smithi047 ceph-mon[77147]: mds.l(mds.0): MDS in read-only mode 2022-05-14T04:50:00.299 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:50:00 smithi047 ceph-mon[77147]: [WRN] MDS_SLOW_METADATA_IO: 1 MDSs report slow metadata IOs 2022-05-14T04:50:00.299 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:50:00 smithi047 ceph-mon[77147]: mds.l(mds.0): 4 slow metadata IOs are blocked > 180 secs, oldest blocked for 19634 secs 2022-05-14T04:50:00.299 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:50:00 smithi047 ceph-mon[77147]: [WRN] MON_DOWN: 1/3 mons down, quorum b,c 2022-05-14T04:50:00.299 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:50:00 smithi047 ceph-mon[77147]: mon.a (rank 0) addr [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] is down (out of quorum) 2022-05-14T04:50:00.299 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:50:00 smithi047 ceph-mon[77147]: [WRN] PG_AVAILABILITY: Reduced data availability: 1 pg inactive 2022-05-14T04:50:00.300 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:50:00 smithi047 ceph-mon[77147]: pg 2.1 is stuck inactive for 5h, current state unknown, last acting [] 2022-05-14T04:50:00.300 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:50:00 smithi047 ceph-mon[77147]: [WRN] SLOW_OPS: 9 slow ops, oldest one blocked for 19335 sec, osd.5 has slow ops 2022-05-14T04:50:00.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:50:00 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:50:00.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:50:00 smithi115 ceph-mon[78593]: pgmap v10017: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:50:00.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:50:00 smithi115 ceph-mon[78593]: Health detail: HEALTH_ERR 1 filesystem is degraded; 1 mds daemon damaged; 1 MDSs are read only; 1 MDSs report slow metadata IOs; 1/3 mons down, quorum b,c; Reduced data availability: 1 pg inactive; 9 slow ops, oldest one blocked for 19335 sec, osd.5 has slow ops 2022-05-14T04:50:00.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:50:00 smithi115 ceph-mon[78593]: [WRN] FS_DEGRADED: 1 filesystem is degraded 2022-05-14T04:50:00.612 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:50:00 smithi115 ceph-mon[78593]: fs cephfs is degraded 2022-05-14T04:50:00.612 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:50:00 smithi115 ceph-mon[78593]: [ERR] MDS_DAMAGE: 1 mds daemon damaged 2022-05-14T04:50:00.612 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:50:00 smithi115 ceph-mon[78593]: fs cephfs mds.2 is damaged 2022-05-14T04:50:00.612 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:50:00 smithi115 ceph-mon[78593]: [WRN] MDS_READ_ONLY: 1 MDSs are read only 2022-05-14T04:50:00.613 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:50:00 smithi115 ceph-mon[78593]: mds.l(mds.0): MDS in read-only mode 2022-05-14T04:50:00.613 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:50:00 smithi115 ceph-mon[78593]: [WRN] MDS_SLOW_METADATA_IO: 1 MDSs report slow metadata IOs 2022-05-14T04:50:00.613 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:50:00 smithi115 ceph-mon[78593]: mds.l(mds.0): 4 slow metadata IOs are blocked > 180 secs, oldest blocked for 19634 secs 2022-05-14T04:50:00.613 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:50:00 smithi115 ceph-mon[78593]: [WRN] MON_DOWN: 1/3 mons down, quorum b,c 2022-05-14T04:50:00.613 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:50:00 smithi115 ceph-mon[78593]: mon.a (rank 0) addr [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] is down (out of quorum) 2022-05-14T04:50:00.614 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:50:00 smithi115 ceph-mon[78593]: [WRN] PG_AVAILABILITY: Reduced data availability: 1 pg inactive 2022-05-14T04:50:00.614 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:50:00 smithi115 ceph-mon[78593]: pg 2.1 is stuck inactive for 5h, current state unknown, last acting [] 2022-05-14T04:50:00.614 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:50:00 smithi115 ceph-mon[78593]: [WRN] SLOW_OPS: 9 slow ops, oldest one blocked for 19335 sec, osd.5 has slow ops 2022-05-14T04:50:01.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:50:00 smithi115 conmon[90393]: 2022-05-14T04:50:00.694+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:50:01.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:50:01 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:50:01.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:50:01 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:50:01.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:50:01 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:50:01.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:50:01 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:50:02.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:50:01 smithi115 conmon[90393]: 2022-05-14T04:50:01.702+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:50:02.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:50:02 smithi047 ceph-mon[77147]: pgmap v10018: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:50:02.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:50:02 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:50:02.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:50:02 smithi115 ceph-mon[78593]: pgmap v10018: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:50:02.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:50:02 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:50:03.110 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:50:02 smithi115 conmon[90393]: 2022-05-14T04:50:02.667+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:50:03.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:50:03 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:50:03.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:50:03 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:50:03.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:50:03 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:50:03.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:50:03 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 19335 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:50:03.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:50:03 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:50:03.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:50:03 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:50:03.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:50:03 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:50:03.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:50:03 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 19335 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:50:04.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:50:03 smithi115 conmon[90393]: 2022-05-14T04:50:03.672+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:50:04.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:50:04 smithi047 ceph-mon[77147]: pgmap v10019: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:50:04.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:50:04 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:50:04.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:50:04 smithi115 ceph-mon[78593]: pgmap v10019: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:50:04.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:50:04 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:50:05.110 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:50:04 smithi115 conmon[90393]: 2022-05-14T04:50:04.716+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:50:05.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:50:05 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:50:05.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:50:05 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:50:06.110 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:50:05 smithi115 conmon[90393]: 2022-05-14T04:50:05.670+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:50:06.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:50:06 smithi047 ceph-mon[77147]: pgmap v10020: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:50:06.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:50:06 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:50:06.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:50:06 smithi115 ceph-mon[78593]: pgmap v10020: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:50:06.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:50:06 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:50:07.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:50:06 smithi115 conmon[90393]: 2022-05-14T04:50:06.667+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:50:07.546 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:50:07 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:50:07.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:50:07 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:50:08.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:50:07 smithi115 conmon[90393]: 2022-05-14T04:50:07.619+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:50:08.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:50:08 smithi047 ceph-mon[77147]: pgmap v10021: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:50:08.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:50:08 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:50:08.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:50:08 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 19340 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:50:08.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:50:08 smithi115 ceph-mon[78593]: pgmap v10021: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:50:08.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:50:08 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:50:08.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:50:08 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 19340 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:50:09.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:50:08 smithi115 conmon[90393]: 2022-05-14T04:50:08.637+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:50:09.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:50:09 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:50:09.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:50:09 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:50:10.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:50:09 smithi115 conmon[90393]: 2022-05-14T04:50:09.622+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:50:10.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:50:10 smithi047 ceph-mon[77147]: pgmap v10022: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:50:10.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:50:10 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:50:10.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:50:10 smithi115 ceph-mon[78593]: pgmap v10022: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:50:10.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:50:10 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:50:11.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:50:10 smithi115 conmon[90393]: 2022-05-14T04:50:10.651+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:50:11.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:50:11 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:50:11.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:50:11 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:50:11.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:50:11 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:50:11.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:50:11 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:50:11.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:50:11 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:50:11.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:50:11 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:50:12.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:50:11 smithi115 conmon[90393]: 2022-05-14T04:50:11.688+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:50:12.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:50:12 smithi047 ceph-mon[77147]: pgmap v10023: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:50:12.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:50:12 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:50:12.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:50:12 smithi115 ceph-mon[78593]: pgmap v10023: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:50:12.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:50:12 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:50:13.110 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:50:12 smithi115 conmon[90393]: 2022-05-14T04:50:12.654+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:50:13.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:50:13 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:50:13.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:50:13 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 19345 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:50:13.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:50:13 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:50:13.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:50:13 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 19345 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:50:14.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:50:13 smithi115 conmon[90393]: 2022-05-14T04:50:13.638+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:50:14.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:50:14 smithi047 ceph-mon[77147]: pgmap v10024: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:50:14.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:50:14 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:50:14.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:50:14 smithi115 ceph-mon[78593]: pgmap v10024: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:50:14.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:50:14 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:50:15.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:50:14 smithi115 conmon[90393]: 2022-05-14T04:50:14.647+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:50:15.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:50:15 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:50:15.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:50:15 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:50:15.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:50:15 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:50:15.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:50:15 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T04:50:15.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:50:15 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T04:50:15.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:50:15 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:50:15.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:50:15 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:50:15.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:50:15 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:50:15.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:50:15 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T04:50:15.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:50:15 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T04:50:16.110 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:50:15 smithi115 conmon[90393]: 2022-05-14T04:50:15.635+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:50:16.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:50:16 smithi047 ceph-mon[77147]: pgmap v10025: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:50:16.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:50:16 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:50:16.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:50:16 smithi115 ceph-mon[78593]: pgmap v10025: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:50:16.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:50:16 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:50:16.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:50:16 smithi115 conmon[90393]: 2022-05-14T04:50:16.606+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:50:17.548 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:50:17 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:50:17.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:50:17 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:50:18.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:50:17 smithi115 conmon[90393]: 2022-05-14T04:50:17.622+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:50:18.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:50:18 smithi047 ceph-mon[77147]: pgmap v10026: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:50:18.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:50:18 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:50:18.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:50:18 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:50:18.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:50:18 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:50:18.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:50:18 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 19350 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:50:18.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:50:18 smithi115 ceph-mon[78593]: pgmap v10026: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:50:18.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:50:18 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:50:18.612 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:50:18 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:50:18.612 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:50:18 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:50:18.613 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:50:18 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 19350 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:50:19.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:50:18 smithi115 conmon[90393]: 2022-05-14T04:50:18.639+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:50:19.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:50:19 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:50:19.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:50:19 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:50:20.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:50:19 smithi115 conmon[90393]: 2022-05-14T04:50:19.651+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:50:20.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:50:20 smithi047 ceph-mon[77147]: pgmap v10027: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:50:20.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:50:20 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:50:20.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:50:20 smithi115 ceph-mon[78593]: pgmap v10027: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:50:20.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:50:20 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:50:21.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:50:20 smithi115 conmon[90393]: 2022-05-14T04:50:20.624+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:50:21.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:50:21 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:50:21.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:50:21 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:50:22.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:50:21 smithi115 conmon[90393]: 2022-05-14T04:50:21.651+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:50:22.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:50:22 smithi047 ceph-mon[77147]: pgmap v10028: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:50:22.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:50:22 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:50:22.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:50:22 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:50:22.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:50:22 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:50:22.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:50:22 smithi115 ceph-mon[78593]: pgmap v10028: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:50:22.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:50:22 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:50:22.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:50:22 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:50:22.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:50:22 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:50:23.110 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:50:22 smithi115 conmon[90393]: 2022-05-14T04:50:22.624+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:50:23.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:50:23 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:50:23.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:50:23 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 19355 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:50:23.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:50:23 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:50:23.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:50:23 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 19355 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:50:24.110 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:50:23 smithi115 conmon[90393]: 2022-05-14T04:50:23.655+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:50:24.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:50:24 smithi047 ceph-mon[77147]: pgmap v10029: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:50:24.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:50:24 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:50:24.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:50:24 smithi115 ceph-mon[78593]: pgmap v10029: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:50:24.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:50:24 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:50:25.110 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:50:24 smithi115 conmon[90393]: 2022-05-14T04:50:24.606+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:50:25.586 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:50:25 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:50:25.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:50:25 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:50:25.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:50:25 smithi115 conmon[90393]: 2022-05-14T04:50:25.582+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:50:26.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:50:26 smithi115 ceph-mon[78593]: pgmap v10030: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:50:26.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:50:26 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:50:26.610 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:50:26 smithi115 conmon[90393]: 2022-05-14T04:50:26.534+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:50:26.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:50:26 smithi047 ceph-mon[77147]: pgmap v10030: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:50:26.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:50:26 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:50:27.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:50:27 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:50:27.610 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:50:27 smithi115 conmon[90393]: 2022-05-14T04:50:27.524+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:50:27.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:50:27 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:50:28.568 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:50:28 smithi115 ceph-mon[78593]: pgmap v10031: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:50:28.569 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:50:28 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:50:28.569 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:50:28 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 19360 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:50:28.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:50:28 smithi047 ceph-mon[77147]: pgmap v10031: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:50:28.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:50:28 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:50:28.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:50:28 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 19360 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:50:28.860 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:50:28 smithi115 conmon[90393]: 2022-05-14T04:50:28.565+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:50:29.568 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:50:29 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:50:29.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:50:29 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:50:29.862 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:50:29 smithi115 conmon[90393]: 2022-05-14T04:50:29.565+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:50:30.584 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:50:30 smithi115 ceph-mon[78593]: pgmap v10032: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:50:30.584 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:50:30 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:50:30.585 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:50:30 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:50:30.585 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:50:30 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:50:30.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:50:30 smithi047 ceph-mon[77147]: pgmap v10032: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:50:30.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:50:30 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:50:30.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:50:30 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:50:30.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:50:30 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:50:30.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:50:30 smithi115 conmon[90393]: 2022-05-14T04:50:30.581+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:50:31.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:50:31 smithi115 conmon[90393]: 2022-05-14T04:50:31.562+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:50:31.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:50:31 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:50:31.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:50:31 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:50:32.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:50:32 smithi115 ceph-mon[78593]: pgmap v10033: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:50:32.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:50:32 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:50:32.611 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:50:32 smithi115 conmon[90393]: 2022-05-14T04:50:32.530+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:50:32.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:50:32 smithi047 ceph-mon[77147]: pgmap v10033: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:50:32.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:50:32 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:50:33.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:50:33 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:50:33.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:50:33 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 19365 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:50:33.611 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:50:33 smithi115 conmon[90393]: 2022-05-14T04:50:33.517+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:50:33.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:50:33 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:50:33.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:50:33 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 19365 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:50:34.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:50:34 smithi115 ceph-mon[78593]: pgmap v10034: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:50:34.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:50:34 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:50:34.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:50:34 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:50:34.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:50:34 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:50:34.611 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:50:34 smithi115 conmon[90393]: 2022-05-14T04:50:34.557+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:50:34.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:50:34 smithi047 ceph-mon[77147]: pgmap v10034: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:50:34.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:50:34 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:50:34.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:50:34 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:50:34.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:50:34 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:50:35.610 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:50:35 smithi115 conmon[90393]: 2022-05-14T04:50:35.553+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:50:35.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:50:35 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:50:35.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:50:35 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:50:36.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:50:36 smithi115 conmon[90393]: 2022-05-14T04:50:36.568+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:50:36.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:50:36 smithi115 ceph-mon[78593]: pgmap v10035: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:50:36.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:50:36 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:50:36.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:50:36 smithi047 ceph-mon[77147]: pgmap v10035: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:50:36.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:50:36 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:50:37.611 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:50:37 smithi115 conmon[90393]: 2022-05-14T04:50:37.530+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:50:37.612 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:50:37 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:50:37.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:50:37 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:50:38.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:50:38 smithi115 conmon[90393]: 2022-05-14T04:50:38.534+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:50:38.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:50:38 smithi115 ceph-mon[78593]: pgmap v10036: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:50:38.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:50:38 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:50:38.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:50:38 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:50:38.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:50:38 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:50:38.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:50:38 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 19370 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:50:38.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:50:38 smithi047 ceph-mon[77147]: pgmap v10036: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:50:38.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:50:38 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:50:38.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:50:38 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:50:38.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:50:38 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:50:38.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:50:38 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 19370 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:50:39.610 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:50:39 smithi115 conmon[90393]: 2022-05-14T04:50:39.499+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:50:39.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:50:39 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:50:39.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:50:39 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:50:40.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:50:40 smithi115 conmon[90393]: 2022-05-14T04:50:40.496+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:50:40.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:50:40 smithi115 ceph-mon[78593]: pgmap v10037: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:50:40.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:50:40 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:50:40.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:50:40 smithi047 ceph-mon[77147]: pgmap v10037: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:50:40.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:50:40 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:50:41.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:50:41 smithi115 conmon[90393]: 2022-05-14T04:50:41.466+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:50:41.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:50:41 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:50:41.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:50:41 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:50:42.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:50:42 smithi115 conmon[90393]: 2022-05-14T04:50:42.420+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:50:42.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:50:42 smithi115 ceph-mon[78593]: pgmap v10038: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:50:42.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:50:42 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:50:42.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:50:42 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:50:42.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:50:42 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:50:42.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:50:42 smithi047 ceph-mon[77147]: pgmap v10038: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:50:42.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:50:42 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:50:42.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:50:42 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:50:42.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:50:42 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:50:43.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:50:43 smithi115 conmon[90393]: 2022-05-14T04:50:43.459+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:50:43.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:50:43 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:50:43.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:50:43 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 19375 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:50:43.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:50:43 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:50:43.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:50:43 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 19375 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:50:44.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:50:44 smithi115 conmon[90393]: 2022-05-14T04:50:44.491+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:50:44.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:50:44 smithi115 ceph-mon[78593]: pgmap v10039: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:50:44.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:50:44 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:50:44.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:50:44 smithi047 ceph-mon[77147]: pgmap v10039: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:50:44.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:50:44 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:50:45.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:50:45 smithi115 conmon[90393]: 2022-05-14T04:50:45.498+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:50:45.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:50:45 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:50:45.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:50:45 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:50:46.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:50:46 smithi047 ceph-mon[77147]: pgmap v10040: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:50:46.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:50:46 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:50:46.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:50:46 smithi115 conmon[90393]: 2022-05-14T04:50:46.466+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:50:46.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:50:46 smithi115 ceph-mon[78593]: pgmap v10040: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:50:46.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:50:46 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:50:47.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:50:47 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:50:47.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:50:47 smithi115 conmon[90393]: 2022-05-14T04:50:47.470+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:50:47.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:50:47 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:50:48.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:50:48 smithi047 ceph-mon[77147]: pgmap v10041: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:50:48.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:50:48 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:50:48.795 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:50:48 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 19380 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:50:48.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:50:48 smithi115 conmon[90393]: 2022-05-14T04:50:48.437+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:50:48.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:50:48 smithi115 ceph-mon[78593]: pgmap v10041: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:50:48.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:50:48 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:50:48.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:50:48 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 19380 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:50:49.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:50:49 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:50:49.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:50:49 smithi115 conmon[90393]: 2022-05-14T04:50:49.454+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:50:49.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:50:49 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:50:50.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:50:50 smithi047 ceph-mon[77147]: pgmap v10042: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:50:50.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:50:50 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:50:50.795 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:50:50 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:50:50.795 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:50:50 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:50:50.795 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:50:50 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T04:50:50.795 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:50:50 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T04:50:50.796 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:50:50 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T04:50:50.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:50:50 smithi115 conmon[90393]: 2022-05-14T04:50:50.435+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:50:50.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:50:50 smithi115 ceph-mon[78593]: pgmap v10042: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:50:50.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:50:50 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:50:50.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:50:50 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:50:50.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:50:50 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:50:50.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:50:50 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T04:50:50.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:50:50 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T04:50:50.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:50:50 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T04:50:51.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:50:51 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:50:51.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:50:51 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:50:51.860 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:50:51 smithi115 conmon[90393]: 2022-05-14T04:50:51.400+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:50:52.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:50:52 smithi047 ceph-mon[77147]: pgmap v10043: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:50:52.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:50:52 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:50:52.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:50:52 smithi047 conmon[78521]: 2022-05-14T04:50:52.591+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:50:52.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:50:52 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T04:50:52.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:50:52 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:50:52.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:50:52 smithi047 conmon[78521]: yield 2022-05-14T04:50:52.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:50:52 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:50:52.797 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:50:52 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:50:52.801 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:50:52 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:50:52.802 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:50:52 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T04:50:52.802 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:50:52 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:50:52.802 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:50:52 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T04:50:52.802 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:50:52 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:50:52.802 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:50:52 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T04:50:52.803 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:50:52 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:50:52.803 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:50:52 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T04:50:52.803 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:50:52 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:50:52.803 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:50:52 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T04:50:52.804 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:50:52 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:50:52.804 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:50:52 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:50:52.804 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:50:52 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:50:52.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:50:52 smithi115 conmon[90393]: 2022-05-14T04:50:52.447+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:50:52.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:50:52 smithi115 ceph-mon[78593]: pgmap v10043: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:50:52.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:50:52 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:50:53.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:50:53 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:50:53.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:50:53 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:50:53.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T04:50:53.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:50:53.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T04:50:53.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:50:53.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:50:53.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:50:53.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T04:50:53.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:50:53.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T04:50:53.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:50:53.797 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T04:50:53.797 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:50:53.797 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T04:50:53.797 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:50:53.798 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T04:50:53.798 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:50:53.798 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:50:53.799 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:50:53.799 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:50:53 smithi047 ceph-mon[77147]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T04:50:53.799 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:50:53 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 19385 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:50:53.866 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:50:53 smithi115 conmon[90393]: 2022-05-14T04:50:53.437+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:50:53.866 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:50:53 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:50:53.867 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:50:53 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:50:53.867 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T04:50:53.867 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:50:53.867 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T04:50:53.867 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:50:53.868 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:50:53.868 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:50:53.868 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T04:50:53.868 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:50:53.869 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T04:50:53.869 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:50:53.869 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T04:50:53.870 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:50:53.871 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T04:50:53.871 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:50:53.872 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T04:50:53.872 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:50:53.872 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:50:53.872 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:50:53.872 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:50:53 smithi115 ceph-mon[78593]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T04:50:53.873 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:50:53 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 19385 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:50:54.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:50:54 smithi047 ceph-mon[77147]: pgmap v10044: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:50:54.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:50:54 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:50:54.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:50:54 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:50:54.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:50:54 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:50:54.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:50:54 smithi115 conmon[90393]: 2022-05-14T04:50:54.454+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:50:54.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:50:54 smithi115 ceph-mon[78593]: pgmap v10044: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:50:54.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:50:54 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:50:54.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:50:54 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:50:54.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:50:54 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:50:55.667 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:50:55 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:50:55.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:50:55 smithi115 conmon[90393]: 2022-05-14T04:50:55.478+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:50:55.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:50:55 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:50:56.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:50:55 smithi047 conmon[78521]: 2022-05-14T04:50:55.663+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:50:56.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:50:55 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T04:50:56.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:50:55 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:50:56.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:50:55 smithi047 conmon[78521]: yield 2022-05-14T04:50:56.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:50:55 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:50:56.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:50:55 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:50:56.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:50:55 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:50:56.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:50:55 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T04:50:56.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:50:55 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:50:56.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:50:55 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T04:50:56.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:50:55 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:50:56.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:50:55 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T04:50:56.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:50:55 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:50:56.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:50:55 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T04:50:56.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:50:55 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:50:56.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:50:55 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T04:50:56.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:50:55 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:50:56.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:50:55 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:50:56.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:50:55 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:50:56.048 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:50:55 smithi047 conmon[78521]: 2022-05-14T04:50:55.664+0000 7f5340f7a700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T04:50:56.048 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:50:55 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T04:50:56.048 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:50:55 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:50:56.048 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:50:55 smithi047 conmon[78521]: yield 2022-05-14T04:50:56.049 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:50:55 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:50:56.049 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:50:55 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:50:56.049 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:50:55 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:50:56.049 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:50:55 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T04:50:56.049 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:50:55 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:50:56.050 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:50:55 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T04:50:56.050 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:50:55 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:50:56.050 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:50:55 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T04:50:56.050 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:50:55 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:50:56.050 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:50:55 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T04:50:56.051 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:50:55 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:50:56.051 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:50:55 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T04:50:56.051 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:50:55 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:50:56.051 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:50:55 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:50:56.052 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:50:55 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:50:56.052 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:50:55 smithi047 conmon[78521]: 2022-05-14T04:50:56.052 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:50:55 smithi047 conmon[78521]: During handling of the above exception, another exception occurred: 2022-05-14T04:50:56.052 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:50:55 smithi047 conmon[78521]: 2022-05-14T04:50:56.052 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:50:55 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T04:50:56.053 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:50:55 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T04:50:56.053 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:50:55 smithi047 conmon[78521]: return f(*arg) 2022-05-14T04:50:56.053 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:50:55 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T04:50:56.053 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:50:55 smithi047 conmon[78521]: self._write_client_files(client_files, host) 2022-05-14T04:50:56.054 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:50:55 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T04:50:56.054 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:50:55 smithi047 conmon[78521]: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T04:50:56.054 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:50:55 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T04:50:56.054 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:50:55 smithi047 conmon[78521]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T04:50:56.056 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:50:55 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T04:50:56.056 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:50:55 smithi047 conmon[78521]: return self.event_loop.get_result(coro) 2022-05-14T04:50:56.057 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:50:55 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T04:50:56.057 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:50:55 smithi047 conmon[78521]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T04:50:56.057 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:50:55 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T04:50:56.057 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:50:55 smithi047 conmon[78521]: return self.__get_result() 2022-05-14T04:50:56.058 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:50:55 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T04:50:56.058 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:50:55 smithi047 conmon[78521]: raise self._exception 2022-05-14T04:50:56.058 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:50:55 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T04:50:56.058 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:50:55 smithi047 conmon[78521]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T04:50:56.059 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:50:55 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T04:50:56.059 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:50:55 smithi047 conmon[78521]: conn = await self._remote_connection(host, addr) 2022-05-14T04:50:56.059 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:50:55 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T04:50:56.059 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:50:55 smithi047 conmon[78521]: raise 2022-05-14T04:50:56.060 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:50:55 smithi047 conmon[78521]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T04:50:56.062 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:50:55 smithi047 conmon[78521]: self.gen.throw(type, value, traceback) 2022-05-14T04:50:56.062 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:50:55 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T04:50:56.062 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:50:55 smithi047 conmon[78521]: raise OrchestratorError(msg) 2022-05-14T04:50:56.062 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:50:55 smithi047 conmon[78521]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:50:56.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:50:56 smithi047 ceph-mon[77147]: pgmap v10045: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:50:56.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:50:56 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:50:56.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:50:56 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:50:56.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T04:50:56.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:50:56.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T04:50:56.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:50:56.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:50:56.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:50:56.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T04:50:56.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:50:56.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T04:50:56.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:50:56.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T04:50:56.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:50:56.797 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T04:50:56.797 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:50:56.797 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T04:50:56.797 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:50:56.797 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:50:56.798 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:50:56.798 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:50:56 smithi047 ceph-mon[77147]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T04:50:56.798 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T04:50:56.798 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:50:56.799 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T04:50:56.799 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:50:56.799 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:50:56.799 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:50:56.800 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T04:50:56.801 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:50:56.801 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T04:50:56.801 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:50:56.802 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T04:50:56.802 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:50:56.802 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T04:50:56.802 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:50:56.802 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T04:50:56.803 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:50:56.803 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:50:56.803 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:50:56.803 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T04:50:56.803 INFO:journalctl@ceph.mon.b.smithi047.stdout: During handling of the above exception, another exception occurred: 2022-05-14T04:50:56.804 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T04:50:56.804 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T04:50:56.804 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T04:50:56.804 INFO:journalctl@ceph.mon.b.smithi047.stdout: return f(*arg) 2022-05-14T04:50:56.805 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T04:50:56.805 INFO:journalctl@ceph.mon.b.smithi047.stdout: self._write_client_files(client_files, host) 2022-05-14T04:50:56.805 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T04:50:56.805 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T04:50:56.806 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T04:50:56.806 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T04:50:56.806 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T04:50:56.806 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.event_loop.get_result(coro) 2022-05-14T04:50:56.806 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T04:50:56.807 INFO:journalctl@ceph.mon.b.smithi047.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T04:50:56.807 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T04:50:56.808 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.__get_result() 2022-05-14T04:50:56.808 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T04:50:56.808 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise self._exception 2022-05-14T04:50:56.808 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T04:50:56.808 INFO:journalctl@ceph.mon.b.smithi047.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T04:50:56.809 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T04:50:56.809 INFO:journalctl@ceph.mon.b.smithi047.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T04:50:56.809 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T04:50:56.809 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise 2022-05-14T04:50:56.810 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T04:50:56.810 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.gen.throw(type, value, traceback) 2022-05-14T04:50:56.810 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T04:50:56.810 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OrchestratorError(msg) 2022-05-14T04:50:56.810 INFO:journalctl@ceph.mon.b.smithi047.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:50:56.860 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:50:56 smithi115 conmon[90393]: 2022-05-14T04:50:56.527+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:50:56.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:50:56 smithi115 ceph-mon[78593]: pgmap v10045: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:50:56.862 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:50:56 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:50:56.862 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:50:56 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:50:56.863 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T04:50:56.863 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:50:56.863 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T04:50:56.864 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:50:56.864 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:50:56.865 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:50:56.866 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T04:50:56.866 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:50:56.866 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T04:50:56.867 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:50:56.867 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T04:50:56.867 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:50:56.868 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T04:50:56.868 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:50:56.868 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T04:50:56.869 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:50:56.869 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:50:56.869 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:50:56.869 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:50:56 smithi115 ceph-mon[78593]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T04:50:56.870 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T04:50:56.870 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:50:56.870 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T04:50:56.871 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:50:56.871 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:50:56.871 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:50:56.871 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T04:50:56.871 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:50:56.872 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T04:50:56.872 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:50:56.872 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T04:50:56.872 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:50:56.873 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T04:50:56.873 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:50:56.873 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T04:50:56.873 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:50:56.874 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:50:56.874 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:50:56.874 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T04:50:56.874 INFO:journalctl@ceph.mon.c.smithi115.stdout: During handling of the above exception, another exception occurred: 2022-05-14T04:50:56.874 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T04:50:56.875 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T04:50:56.875 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T04:50:56.875 INFO:journalctl@ceph.mon.c.smithi115.stdout: return f(*arg) 2022-05-14T04:50:56.875 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T04:50:56.876 INFO:journalctl@ceph.mon.c.smithi115.stdout: self._write_client_files(client_files, host) 2022-05-14T04:50:56.876 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T04:50:56.876 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T04:50:56.876 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T04:50:56.876 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T04:50:56.877 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T04:50:56.877 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.event_loop.get_result(coro) 2022-05-14T04:50:56.877 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T04:50:56.877 INFO:journalctl@ceph.mon.c.smithi115.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T04:50:56.877 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T04:50:56.878 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.__get_result() 2022-05-14T04:50:56.879 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T04:50:56.879 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise self._exception 2022-05-14T04:50:56.879 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T04:50:56.879 INFO:journalctl@ceph.mon.c.smithi115.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T04:50:56.879 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T04:50:56.880 INFO:journalctl@ceph.mon.c.smithi115.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T04:50:56.880 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T04:50:56.880 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise 2022-05-14T04:50:56.880 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T04:50:56.881 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.gen.throw(type, value, traceback) 2022-05-14T04:50:56.881 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T04:50:56.881 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OrchestratorError(msg) 2022-05-14T04:50:56.881 INFO:journalctl@ceph.mon.c.smithi115.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:50:57.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:50:57 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:50:57.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:50:57 smithi115 conmon[90393]: 2022-05-14T04:50:57.551+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:50:57.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:50:57 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:50:58.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:50:58 smithi047 ceph-mon[77147]: pgmap v10046: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:50:58.795 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:50:58 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:50:58.795 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:50:58 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:50:58.795 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:50:58 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:50:58.796 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:50:58 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 19390 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:50:58.860 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:50:58 smithi115 conmon[90393]: 2022-05-14T04:50:58.567+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:50:58.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:50:58 smithi115 ceph-mon[78593]: pgmap v10046: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:50:58.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:50:58 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:50:58.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:50:58 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:50:58.862 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:50:58 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:50:58.862 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:50:58 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 19390 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:50:59.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:50:59 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:50:59.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:50:59 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:50:59.860 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:50:59 smithi115 conmon[90393]: 2022-05-14T04:50:59.592+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:51:00.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:51:00 smithi047 ceph-mon[77147]: pgmap v10047: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:51:00.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:51:00 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:51:00.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:51:00 smithi115 conmon[90393]: 2022-05-14T04:51:00.567+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:51:00.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:51:00 smithi115 ceph-mon[78593]: pgmap v10047: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:51:00.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:51:00 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:51:01.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:51:01 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:51:01.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:51:01 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:51:01.860 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:51:01 smithi115 conmon[90393]: 2022-05-14T04:51:01.563+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:51:02.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:51:02 smithi047 ceph-mon[77147]: pgmap v10048: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:51:02.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:51:02 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:51:02.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:51:02 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:51:02.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:51:02 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:51:02.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:51:02 smithi115 conmon[90393]: 2022-05-14T04:51:02.541+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:51:02.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:51:02 smithi115 ceph-mon[78593]: pgmap v10048: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:51:02.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:51:02 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:51:02.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:51:02 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:51:02.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:51:02 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:51:03.797 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:51:03 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:51:03.798 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:51:03 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 19395 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:51:03.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:51:03 smithi115 conmon[90393]: 2022-05-14T04:51:03.545+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:51:03.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:51:03 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:51:03.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:51:03 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 19395 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:51:04.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:51:04 smithi047 ceph-mon[77147]: pgmap v10049: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:51:04.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:51:04 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:51:04.860 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:51:04 smithi115 conmon[90393]: 2022-05-14T04:51:04.576+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:51:04.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:51:04 smithi115 ceph-mon[78593]: pgmap v10049: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:51:04.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:51:04 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:51:05.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:51:05 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:51:05.865 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:51:05 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:51:05.865 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:51:05 smithi115 conmon[90393]: 2022-05-14T04:51:05.552+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:51:06.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:51:06 smithi047 ceph-mon[77147]: pgmap v10050: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:51:06.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:51:06 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:51:06.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:51:06 smithi115 conmon[90393]: 2022-05-14T04:51:06.595+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:51:06.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:51:06 smithi115 ceph-mon[78593]: pgmap v10050: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:51:06.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:51:06 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:51:07.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:51:07 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:51:07.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:51:07 smithi115 conmon[90393]: 2022-05-14T04:51:07.612+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:51:07.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:51:07 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:51:08.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:51:08 smithi047 ceph-mon[77147]: pgmap v10051: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:51:08.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:51:08 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:51:08.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:51:08 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 19400 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:51:08.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:51:08 smithi115 conmon[90393]: 2022-05-14T04:51:08.569+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:51:08.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:51:08 smithi115 ceph-mon[78593]: pgmap v10051: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:51:08.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:51:08 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:51:08.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:51:08 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 19400 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:51:09.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:51:09 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:51:09.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:51:09 smithi115 conmon[90393]: 2022-05-14T04:51:09.536+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:51:09.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:51:09 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:51:10.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:51:10 smithi047 ceph-mon[77147]: pgmap v10052: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:51:10.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:51:10 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:51:10.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:51:10 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:51:10.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:51:10 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:51:10.862 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:51:10 smithi115 conmon[90393]: 2022-05-14T04:51:10.521+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:51:10.863 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:51:10 smithi115 ceph-mon[78593]: pgmap v10052: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:51:10.863 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:51:10 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:51:10.864 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:51:10 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:51:10.864 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:51:10 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:51:11.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:51:11 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:51:11.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:51:11 smithi115 conmon[90393]: 2022-05-14T04:51:11.567+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:51:11.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:51:11 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:51:12.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:51:12 smithi047 ceph-mon[77147]: pgmap v10053: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:51:12.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:51:12 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:51:12.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:51:12 smithi115 conmon[90393]: 2022-05-14T04:51:12.550+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:51:12.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:51:12 smithi115 ceph-mon[78593]: pgmap v10053: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:51:12.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:51:12 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:51:13.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:51:13 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:51:13.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:51:13 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 19405 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:51:13.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:51:13 smithi115 conmon[90393]: 2022-05-14T04:51:13.522+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:51:13.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:51:13 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:51:13.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:51:13 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 19405 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:51:14.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:51:14 smithi047 ceph-mon[77147]: pgmap v10054: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:51:14.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:51:14 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:51:14.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:51:14 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:51:14.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:51:14 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:51:14.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:51:14 smithi115 conmon[90393]: 2022-05-14T04:51:14.506+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:51:14.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:51:14 smithi115 ceph-mon[78593]: pgmap v10054: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:51:14.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:51:14 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:51:14.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:51:14 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:51:14.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:51:14 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:51:15.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:51:15 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:51:15.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:51:15 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T04:51:15.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:51:15 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T04:51:15.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:51:15 smithi115 conmon[90393]: 2022-05-14T04:51:15.468+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:51:15.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:51:15 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:51:15.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:51:15 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T04:51:15.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:51:15 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T04:51:16.801 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:51:16 smithi047 ceph-mon[77147]: pgmap v10055: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:51:16.801 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:51:16 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:51:16.860 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:51:16 smithi115 conmon[90393]: 2022-05-14T04:51:16.504+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:51:16.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:51:16 smithi115 ceph-mon[78593]: pgmap v10055: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:51:16.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:51:16 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:51:17.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:51:17 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:51:17.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:51:17 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:51:17.860 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:51:17 smithi115 conmon[90393]: 2022-05-14T04:51:17.524+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:51:18.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:51:18 smithi047 ceph-mon[77147]: pgmap v10056: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:51:18.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:51:18 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:51:18.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:51:18 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:51:18.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:51:18 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:51:18.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:51:18 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 19410 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:51:18.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:51:18 smithi115 conmon[90393]: 2022-05-14T04:51:18.497+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:51:18.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:51:18 smithi115 ceph-mon[78593]: pgmap v10056: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:51:18.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:51:18 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:51:18.862 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:51:18 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:51:18.862 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:51:18 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:51:18.862 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:51:18 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 19410 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:51:19.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:51:19 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:51:19.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:51:19 smithi115 conmon[90393]: 2022-05-14T04:51:19.496+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:51:19.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:51:19 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:51:20.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:51:20 smithi047 ceph-mon[77147]: pgmap v10057: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:51:20.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:51:20 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:51:20.863 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:51:20 smithi115 conmon[90393]: 2022-05-14T04:51:20.455+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:51:20.864 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:51:20 smithi115 ceph-mon[78593]: pgmap v10057: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:51:20.864 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:51:20 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:51:21.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:51:21 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:51:21.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:51:21 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:51:21.860 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:51:21 smithi115 conmon[90393]: 2022-05-14T04:51:21.455+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:51:22.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:51:22 smithi047 ceph-mon[77147]: pgmap v10058: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:51:22.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:51:22 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:51:22.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:51:22 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:51:22.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:51:22 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:51:22.860 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:51:22 smithi115 conmon[90393]: 2022-05-14T04:51:22.431+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:51:22.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:51:22 smithi115 ceph-mon[78593]: pgmap v10058: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:51:22.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:51:22 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:51:22.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:51:22 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:51:22.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:51:22 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:51:23.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:51:23 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:51:23.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:51:23 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 19415 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:51:23.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:51:23 smithi115 conmon[90393]: 2022-05-14T04:51:23.429+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:51:23.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:51:23 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:51:23.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:51:23 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 19415 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:51:24.795 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:51:24 smithi047 ceph-mon[77147]: pgmap v10059: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:51:24.796 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:51:24 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:51:24.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:51:24 smithi115 conmon[90393]: 2022-05-14T04:51:24.465+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:51:24.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:51:24 smithi115 ceph-mon[78593]: pgmap v10059: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:51:24.862 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:51:24 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:51:25.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:51:25 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:51:25.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:51:25 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:51:25.860 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:51:25 smithi115 conmon[90393]: 2022-05-14T04:51:25.436+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:51:26.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:51:26 smithi047 ceph-mon[77147]: pgmap v10060: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:51:26.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:51:26 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:51:26.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:51:26 smithi115 conmon[90393]: 2022-05-14T04:51:26.447+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:51:26.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:51:26 smithi115 ceph-mon[78593]: pgmap v10060: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:51:26.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:51:26 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:51:27.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:51:27 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:51:27.860 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:51:27 smithi115 conmon[90393]: 2022-05-14T04:51:27.468+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:51:27.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:51:27 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:51:28.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:51:28 smithi047 ceph-mon[77147]: pgmap v10061: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:51:28.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:51:28 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:51:28.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:51:28 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 19420 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:51:28.860 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:51:28 smithi115 conmon[90393]: 2022-05-14T04:51:28.435+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:51:28.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:51:28 smithi115 ceph-mon[78593]: pgmap v10061: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:51:28.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:51:28 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:51:28.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:51:28 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 19420 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:51:29.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:51:29 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:51:29.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:51:29 smithi115 conmon[90393]: 2022-05-14T04:51:29.483+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:51:29.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:51:29 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:51:30.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:51:30 smithi047 ceph-mon[77147]: pgmap v10062: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:51:30.797 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:51:30 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:51:30.798 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:51:30 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:51:30.798 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:51:30 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:51:30.862 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:51:30 smithi115 conmon[90393]: 2022-05-14T04:51:30.509+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:51:30.864 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:51:30 smithi115 ceph-mon[78593]: pgmap v10062: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:51:30.864 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:51:30 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:51:30.864 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:51:30 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:51:30.865 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:51:30 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:51:31.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:51:31 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:51:31.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:51:31 smithi115 conmon[90393]: 2022-05-14T04:51:31.504+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:51:31.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:51:31 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:51:32.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:51:32 smithi047 ceph-mon[77147]: pgmap v10063: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:51:32.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:51:32 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:51:32.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:51:32 smithi115 conmon[90393]: 2022-05-14T04:51:32.471+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:51:32.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:51:32 smithi115 ceph-mon[78593]: pgmap v10063: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:51:32.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:51:32 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:51:33.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:51:33 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:51:33.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:51:33 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 19425 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:51:33.860 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:51:33 smithi115 conmon[90393]: 2022-05-14T04:51:33.486+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:51:33.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:51:33 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:51:33.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:51:33 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 19425 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:51:34.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:51:34 smithi047 ceph-mon[77147]: pgmap v10064: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:51:34.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:51:34 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:51:34.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:51:34 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:51:34.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:51:34 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:51:34.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:51:34 smithi115 conmon[90393]: 2022-05-14T04:51:34.473+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:51:34.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:51:34 smithi115 ceph-mon[78593]: pgmap v10064: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:51:34.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:51:34 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:51:34.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:51:34 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:51:34.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:51:34 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:51:35.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:51:35 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:51:35.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:51:35 smithi115 conmon[90393]: 2022-05-14T04:51:35.446+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:51:35.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:51:35 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:51:36.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:51:36 smithi047 ceph-mon[77147]: pgmap v10065: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:51:36.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:51:36 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:51:36.860 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:51:36 smithi115 conmon[90393]: 2022-05-14T04:51:36.424+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:51:36.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:51:36 smithi115 ceph-mon[78593]: pgmap v10065: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:51:36.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:51:36 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:51:37.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:51:37 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:51:37.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:51:37 smithi115 conmon[90393]: 2022-05-14T04:51:37.456+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:51:37.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:51:37 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:51:38.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:51:38 smithi047 ceph-mon[77147]: pgmap v10066: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:51:38.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:51:38 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:51:38.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:51:38 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:51:38.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:51:38 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:51:38.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:51:38 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 19430 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:51:38.861 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:51:38 smithi115 conmon[90393]: 2022-05-14T04:51:38.425+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:51:38.862 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:51:38 smithi115 ceph-mon[78593]: pgmap v10066: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:51:38.862 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:51:38 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:51:38.863 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:51:38 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:51:38.863 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:51:38 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:51:38.863 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:51:38 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 19430 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:51:39.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:51:39 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:51:39.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:51:39 smithi115 conmon[90393]: 2022-05-14T04:51:39.382+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:51:39.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:51:39 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:51:40.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:51:40 smithi047 ceph-mon[77147]: pgmap v10067: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:51:40.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:51:40 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:51:40.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:51:40 smithi115 conmon[90393]: 2022-05-14T04:51:40.408+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:51:40.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:51:40 smithi115 ceph-mon[78593]: pgmap v10067: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:51:40.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:51:40 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:51:41.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:51:41 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:51:41.861 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:51:41 smithi115 conmon[90393]: 2022-05-14T04:51:41.383+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:51:41.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:51:41 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:51:42.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:51:42 smithi047 ceph-mon[77147]: pgmap v10068: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:51:42.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:51:42 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:51:42.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:51:42 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:51:42.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:51:42 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:51:42.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:51:42 smithi115 conmon[90393]: 2022-05-14T04:51:42.364+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:51:42.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:51:42 smithi115 ceph-mon[78593]: pgmap v10068: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:51:42.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:51:42 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:51:42.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:51:42 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:51:42.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:51:42 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:51:43.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:51:43 smithi115 conmon[90393]: 2022-05-14T04:51:43.319+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:51:43.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:51:43 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:51:43.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:51:43 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 19435 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:51:43.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:51:43 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:51:43.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:51:43 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 19435 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:51:44.534 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:51:44 smithi115 conmon[90393]: 2022-05-14T04:51:44.274+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:51:44.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:51:44 smithi047 ceph-mon[77147]: pgmap v10069: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:51:44.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:51:44 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:51:44.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:51:44 smithi115 ceph-mon[78593]: pgmap v10069: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:51:44.862 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:51:44 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:51:45.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:51:45 smithi115 conmon[90393]: 2022-05-14T04:51:45.321+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:51:45.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:51:45 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:51:45.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:51:45 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:51:46.610 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:51:46 smithi115 conmon[90393]: 2022-05-14T04:51:46.340+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:51:46.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:51:46 smithi115 ceph-mon[78593]: pgmap v10070: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:51:46.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:51:46 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:51:46.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:51:46 smithi047 ceph-mon[77147]: pgmap v10070: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:51:46.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:51:46 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:51:47.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:51:47 smithi115 conmon[90393]: 2022-05-14T04:51:47.336+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:51:47.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:51:47 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:51:48.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:51:47 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:51:48.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:51:48 smithi115 conmon[90393]: 2022-05-14T04:51:48.383+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:51:48.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:51:48 smithi115 ceph-mon[78593]: pgmap v10071: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:51:48.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:51:48 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:51:48.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:51:48 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 19440 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:51:49.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:51:48 smithi047 ceph-mon[77147]: pgmap v10071: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:51:49.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:51:48 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:51:49.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:51:48 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 19440 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:51:49.862 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:51:49 smithi115 conmon[90393]: 2022-05-14T04:51:49.414+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:51:49.862 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:51:49 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:51:50.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:51:49 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:51:50.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:51:50 smithi115 conmon[90393]: 2022-05-14T04:51:50.409+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:51:50.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:51:50 smithi115 ceph-mon[78593]: pgmap v10072: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:51:50.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:51:50 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:51:50.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:51:50 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:51:50.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:51:50 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:51:51.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:51:50 smithi047 ceph-mon[77147]: pgmap v10072: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:51:51.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:51:50 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:51:51.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:51:50 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:51:51.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:51:50 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:51:51.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:51:51 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:51:51.860 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:51:51 smithi115 conmon[90393]: 2022-05-14T04:51:51.405+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:51:52.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:51:51 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:51:52.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:51:52 smithi115 conmon[90393]: 2022-05-14T04:51:52.406+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:51:52.992 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:51:52 smithi115 ceph-mon[78593]: pgmap v10073: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:51:52.992 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:51:52 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:51:53.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:51:52 smithi047 ceph-mon[77147]: pgmap v10073: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:51:53.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:51:52 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:51:53.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:51:53 smithi115 conmon[90393]: 2022-05-14T04:51:53.438+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:51:53.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:51:53 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:51:53.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:51:53 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 19445 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:51:54.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:51:53 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:51:54.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:51:53 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 19445 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:51:54.860 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:51:54 smithi115 conmon[90393]: 2022-05-14T04:51:54.399+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:51:54.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:51:54 smithi115 ceph-mon[78593]: pgmap v10074: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:51:54.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:51:54 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:51:54.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:51:54 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:51:54.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:51:54 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:51:55.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:51:54 smithi047 ceph-mon[77147]: pgmap v10074: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:51:55.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:51:54 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:51:55.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:51:54 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:51:55.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:51:54 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:51:55.723 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:51:55 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:51:55.724 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:51:55 smithi115 conmon[90393]: 2022-05-14T04:51:55.445+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:51:56.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:51:55 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:51:56.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:51:56 smithi047 conmon[78521]: 2022-05-14T04:51:56.145+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:51:56.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:51:56 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T04:51:56.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:51:56 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:51:56.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:51:56 smithi047 conmon[78521]: yield 2022-05-14T04:51:56.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:51:56 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:51:56.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:51:56 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:51:56.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:51:56 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:51:56.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:51:56 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T04:51:56.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:51:56 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:51:56.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:51:56 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T04:51:56.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:51:56 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:51:56.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:51:56 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T04:51:56.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:51:56 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:51:56.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:51:56 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T04:51:56.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:51:56 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:51:56.547 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:51:56 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T04:51:56.547 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:51:56 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:51:56.547 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:51:56 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:51:56.547 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:51:56 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:51:56.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:51:56 smithi115 conmon[90393]: 2022-05-14T04:51:56.440+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:51:56.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:51:56 smithi115 ceph-mon[78593]: pgmap v10075: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:51:56.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:51:56 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:51:56.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:51:56 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T04:51:56.862 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:51:56 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T04:51:56.862 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:51:56 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T04:51:57.046 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:51:56 smithi047 ceph-mon[77147]: pgmap v10075: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:51:57.046 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:51:56 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:51:57.046 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:51:56 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T04:51:57.047 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:51:56 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T04:51:57.047 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:51:56 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T04:51:57.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:51:57 smithi115 conmon[90393]: 2022-05-14T04:51:57.453+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:51:57.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:51:57 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:51:57.860 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T04:51:57.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:51:57.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T04:51:57.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:51:57.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:51:57.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:51:57.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T04:51:57.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:51:57.863 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T04:51:57.863 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:51:57.863 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T04:51:57.863 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:51:57.864 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T04:51:57.864 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:51:57.864 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T04:51:57.865 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:51:57.865 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:51:57.865 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:51:57.865 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:51:57 smithi115 ceph-mon[78593]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T04:51:57.866 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:51:57 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:51:58.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:51:57 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:51:58.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T04:51:58.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:51:58.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T04:51:58.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:51:58.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:51:58.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:51:58.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T04:51:58.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:51:58.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T04:51:58.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:51:58.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T04:51:58.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:51:58.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T04:51:58.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:51:58.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T04:51:58.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:51:58.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:51:58.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:51:58.047 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:51:57 smithi047 ceph-mon[77147]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T04:51:58.047 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:51:57 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:51:58.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:51:58 smithi115 conmon[90393]: 2022-05-14T04:51:58.433+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:51:58.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:51:58 smithi115 ceph-mon[78593]: pgmap v10076: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:51:58.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:51:58 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:51:58.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:51:58 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:51:58.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:51:58 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:51:58.862 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:51:58 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 19450 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:51:59.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:51:58 smithi047 ceph-mon[77147]: pgmap v10076: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:51:59.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:51:58 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:51:59.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:51:58 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:51:59.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:51:58 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:51:59.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:51:58 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 19450 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:51:59.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:51:59 smithi047 conmon[78521]: 2022-05-14T04:51:59.216+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:51:59.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:51:59 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T04:51:59.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:51:59 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:51:59.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:51:59 smithi047 conmon[78521]: yield 2022-05-14T04:51:59.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:51:59 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:51:59.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:51:59 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:51:59.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:51:59 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:51:59.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:51:59 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T04:51:59.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:51:59 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:51:59.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:51:59 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T04:51:59.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:51:59 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:51:59.547 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:51:59 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T04:51:59.547 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:51:59 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:51:59.547 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:51:59 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T04:51:59.547 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:51:59 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:51:59.548 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:51:59 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T04:51:59.548 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:51:59 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:51:59.548 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:51:59 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:51:59.548 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:51:59 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:51:59.548 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:51:59 smithi047 conmon[78521]: 2022-05-14T04:51:59.217+0000 7f5344f82700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T04:51:59.549 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:51:59 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T04:51:59.549 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:51:59 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:51:59.549 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:51:59 smithi047 conmon[78521]: yield 2022-05-14T04:51:59.549 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:51:59 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:51:59.549 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:51:59 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:51:59.550 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:51:59 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:51:59.550 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:51:59 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T04:51:59.550 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:51:59 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:51:59.550 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:51:59 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T04:51:59.551 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:51:59 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:51:59.551 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:51:59 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T04:51:59.551 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:51:59 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:51:59.551 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:51:59 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T04:51:59.552 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:51:59 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:51:59.552 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:51:59 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T04:51:59.552 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:51:59 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:51:59.552 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:51:59 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:51:59.553 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:51:59 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:51:59.553 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:51:59 smithi047 conmon[78521]: 2022-05-14T04:51:59.553 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:51:59 smithi047 conmon[78521]: During handling of the above exception, another exception occurred: 2022-05-14T04:51:59.553 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:51:59 smithi047 conmon[78521]: 2022-05-14T04:51:59.554 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:51:59 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T04:51:59.554 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:51:59 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T04:51:59.554 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:51:59 smithi047 conmon[78521]: return f(*arg) 2022-05-14T04:51:59.554 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:51:59 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T04:51:59.555 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:51:59 smithi047 conmon[78521]: self._write_client_files(client_files, host) 2022-05-14T04:51:59.555 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:51:59 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T04:51:59.555 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:51:59 smithi047 conmon[78521]: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T04:51:59.555 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:51:59 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T04:51:59.555 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:51:59 smithi047 conmon[78521]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T04:51:59.556 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:51:59 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T04:51:59.556 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:51:59 smithi047 conmon[78521]: return self.event_loop.get_result(coro) 2022-05-14T04:51:59.556 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:51:59 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T04:51:59.556 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:51:59 smithi047 conmon[78521]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T04:51:59.557 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:51:59 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T04:51:59.557 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:51:59 smithi047 conmon[78521]: return self.__get_result() 2022-05-14T04:51:59.557 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:51:59 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T04:51:59.557 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:51:59 smithi047 conmon[78521]: raise self._exception 2022-05-14T04:51:59.557 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:51:59 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T04:51:59.558 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:51:59 smithi047 conmon[78521]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T04:51:59.558 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:51:59 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T04:51:59.558 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:51:59 smithi047 conmon[78521]: conn = await self._remote_connection(host, addr) 2022-05-14T04:51:59.558 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:51:59 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T04:51:59.558 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:51:59 smithi047 conmon[78521]: raise 2022-05-14T04:51:59.559 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:51:59 smithi047 conmon[78521]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T04:51:59.559 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:51:59 smithi047 conmon[78521]: self.gen.throw(type, value, traceback) 2022-05-14T04:51:59.559 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:51:59 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T04:51:59.559 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:51:59 smithi047 conmon[78521]: raise OrchestratorError(msg) 2022-05-14T04:51:59.560 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:51:59 smithi047 conmon[78521]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:51:59.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:51:59 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:51:59.860 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:51:59 smithi115 conmon[90393]: 2022-05-14T04:51:59.478+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:52:00.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:51:59 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:52:00.860 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:52:00 smithi115 conmon[90393]: 2022-05-14T04:52:00.463+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:52:00.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:52:00 smithi115 ceph-mon[78593]: pgmap v10077: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:52:00.863 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:52:00 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:52:00.863 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T04:52:00.863 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:52:00.863 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T04:52:00.864 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:52:00.864 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:52:00.864 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:52:00.864 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T04:52:00.864 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:52:00.865 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T04:52:00.865 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:52:00.866 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T04:52:00.866 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:52:00.866 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T04:52:00.867 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:52:00.867 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T04:52:00.868 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:52:00.868 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:52:00.869 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:52:00.869 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:52:00 smithi115 ceph-mon[78593]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T04:52:00.869 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T04:52:00.870 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:52:00.870 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T04:52:00.870 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:52:00.871 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:52:00.871 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:52:00.871 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T04:52:00.872 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:52:00.872 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T04:52:00.872 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:52:00.873 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T04:52:00.873 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:52:00.874 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T04:52:00.874 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:52:00.874 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T04:52:00.875 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:52:00.875 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:52:00.875 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:52:00.875 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T04:52:00.876 INFO:journalctl@ceph.mon.c.smithi115.stdout: During handling of the above exception, another exception occurred: 2022-05-14T04:52:00.876 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T04:52:00.876 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T04:52:00.876 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T04:52:00.877 INFO:journalctl@ceph.mon.c.smithi115.stdout: return f(*arg) 2022-05-14T04:52:00.877 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T04:52:00.877 INFO:journalctl@ceph.mon.c.smithi115.stdout: self._write_client_files(client_files, host) 2022-05-14T04:52:00.877 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T04:52:00.878 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T04:52:00.878 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T04:52:00.878 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T04:52:00.878 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T04:52:00.878 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.event_loop.get_result(coro) 2022-05-14T04:52:00.879 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T04:52:00.879 INFO:journalctl@ceph.mon.c.smithi115.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T04:52:00.879 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T04:52:00.879 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.__get_result() 2022-05-14T04:52:00.879 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T04:52:00.880 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise self._exception 2022-05-14T04:52:00.880 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T04:52:00.880 INFO:journalctl@ceph.mon.c.smithi115.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T04:52:00.880 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T04:52:00.881 INFO:journalctl@ceph.mon.c.smithi115.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T04:52:00.881 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T04:52:00.881 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise 2022-05-14T04:52:00.882 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T04:52:00.882 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.gen.throw(type, value, traceback) 2022-05-14T04:52:00.882 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T04:52:00.882 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OrchestratorError(msg) 2022-05-14T04:52:00.883 INFO:journalctl@ceph.mon.c.smithi115.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:52:00.883 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:52:00 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:52:01.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:52:00 smithi047 ceph-mon[77147]: pgmap v10077: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:52:01.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:52:00 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:52:01.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T04:52:01.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:52:01.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T04:52:01.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:52:01.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:52:01.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:52:01.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T04:52:01.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:52:01.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T04:52:01.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:52:01.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T04:52:01.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:52:01.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T04:52:01.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:52:01.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T04:52:01.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:52:01.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:52:01.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:52:01.047 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:52:00 smithi047 ceph-mon[77147]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T04:52:01.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T04:52:01.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:52:01.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T04:52:01.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:52:01.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:52:01.049 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:52:01.049 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T04:52:01.049 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:52:01.049 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T04:52:01.049 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:52:01.050 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T04:52:01.050 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:52:01.050 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T04:52:01.050 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:52:01.050 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T04:52:01.051 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:52:01.051 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:52:01.051 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:52:01.051 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T04:52:01.052 INFO:journalctl@ceph.mon.b.smithi047.stdout: During handling of the above exception, another exception occurred: 2022-05-14T04:52:01.052 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T04:52:01.052 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T04:52:01.052 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T04:52:01.052 INFO:journalctl@ceph.mon.b.smithi047.stdout: return f(*arg) 2022-05-14T04:52:01.053 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T04:52:01.053 INFO:journalctl@ceph.mon.b.smithi047.stdout: self._write_client_files(client_files, host) 2022-05-14T04:52:01.053 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T04:52:01.053 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T04:52:01.053 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T04:52:01.054 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T04:52:01.054 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T04:52:01.054 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.event_loop.get_result(coro) 2022-05-14T04:52:01.054 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T04:52:01.055 INFO:journalctl@ceph.mon.b.smithi047.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T04:52:01.055 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T04:52:01.055 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.__get_result() 2022-05-14T04:52:01.055 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T04:52:01.055 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise self._exception 2022-05-14T04:52:01.056 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T04:52:01.056 INFO:journalctl@ceph.mon.b.smithi047.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T04:52:01.056 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T04:52:01.056 INFO:journalctl@ceph.mon.b.smithi047.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T04:52:01.056 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T04:52:01.057 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise 2022-05-14T04:52:01.057 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T04:52:01.057 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.gen.throw(type, value, traceback) 2022-05-14T04:52:01.057 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T04:52:01.058 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OrchestratorError(msg) 2022-05-14T04:52:01.058 INFO:journalctl@ceph.mon.b.smithi047.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:52:01.058 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:52:00 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:52:01.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:52:01 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:52:01.860 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:52:01 smithi115 conmon[90393]: 2022-05-14T04:52:01.489+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:52:02.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:52:01 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:52:02.860 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:52:02 smithi115 conmon[90393]: 2022-05-14T04:52:02.479+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:52:02.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:52:02 smithi115 ceph-mon[78593]: pgmap v10078: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:52:02.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:52:02 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:52:02.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:52:02 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:52:02.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:52:02 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:52:03.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:52:02 smithi047 ceph-mon[77147]: pgmap v10078: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:52:03.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:52:02 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:52:03.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:52:02 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:52:03.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:52:02 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:52:03.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:52:03 smithi115 conmon[90393]: 2022-05-14T04:52:03.458+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:52:03.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:52:03 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:52:03.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:52:03 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 19455 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:52:04.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:52:03 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:52:04.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:52:03 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 19455 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:52:04.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:52:04 smithi115 conmon[90393]: 2022-05-14T04:52:04.499+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:52:04.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:52:04 smithi115 ceph-mon[78593]: pgmap v10079: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:52:04.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:52:04 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:52:05.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:52:04 smithi047 ceph-mon[77147]: pgmap v10079: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:52:05.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:52:04 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:52:05.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:52:05 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:52:05.860 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:52:05 smithi115 conmon[90393]: 2022-05-14T04:52:05.504+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:52:06.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:52:05 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:52:06.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:52:06 smithi115 conmon[90393]: 2022-05-14T04:52:06.500+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:52:06.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:52:06 smithi115 ceph-mon[78593]: pgmap v10080: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:52:06.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:52:06 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:52:07.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:52:06 smithi047 ceph-mon[77147]: pgmap v10080: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:52:07.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:52:06 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:52:07.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:52:07 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:52:07.860 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:52:07 smithi115 conmon[90393]: 2022-05-14T04:52:07.493+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:52:08.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:52:07 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:52:08.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:52:08 smithi115 conmon[90393]: 2022-05-14T04:52:08.496+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:52:08.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:52:08 smithi115 ceph-mon[78593]: pgmap v10081: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:52:08.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:52:08 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:52:08.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:52:08 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 19460 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:52:09.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:52:08 smithi047 ceph-mon[77147]: pgmap v10081: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:52:09.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:52:08 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:52:09.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:52:08 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 19460 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:52:09.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:52:09 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:52:09.860 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:52:09 smithi115 conmon[90393]: 2022-05-14T04:52:09.499+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:52:10.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:52:09 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:52:10.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:52:10 smithi115 conmon[90393]: 2022-05-14T04:52:10.543+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:52:10.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:52:10 smithi115 ceph-mon[78593]: pgmap v10082: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:52:10.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:52:10 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:52:10.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:52:10 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:52:10.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:52:10 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:52:11.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:52:10 smithi047 ceph-mon[77147]: pgmap v10082: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:52:11.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:52:10 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:52:11.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:52:10 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:52:11.045 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:52:10 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:52:11.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:52:11 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:52:11.860 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:52:11 smithi115 conmon[90393]: 2022-05-14T04:52:11.559+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:52:12.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:52:11 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:52:12.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:52:12 smithi115 conmon[90393]: 2022-05-14T04:52:12.516+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:52:12.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:52:12 smithi115 ceph-mon[78593]: pgmap v10083: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:52:12.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:52:12 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:52:13.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:52:12 smithi047 ceph-mon[77147]: pgmap v10083: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:52:13.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:52:12 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:52:13.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:52:13 smithi115 conmon[90393]: 2022-05-14T04:52:13.560+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:52:13.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:52:13 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:52:13.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:52:13 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 19465 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:52:14.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:52:13 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:52:14.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:52:13 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 19465 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:52:14.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:52:14 smithi115 conmon[90393]: 2022-05-14T04:52:14.514+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:52:14.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:52:14 smithi115 ceph-mon[78593]: pgmap v10084: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:52:14.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:52:14 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:52:14.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:52:14 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:52:14.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:52:14 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:52:15.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:52:14 smithi047 ceph-mon[77147]: pgmap v10084: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:52:15.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:52:14 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:52:15.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:52:14 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:52:15.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:52:14 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:52:15.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:52:15 smithi115 conmon[90393]: 2022-05-14T04:52:15.476+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:52:15.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:52:15 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:52:15.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:52:15 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T04:52:15.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:52:15 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T04:52:16.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:52:15 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:52:16.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:52:15 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T04:52:16.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:52:15 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T04:52:16.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:52:16 smithi115 conmon[90393]: 2022-05-14T04:52:16.463+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:52:16.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:52:16 smithi115 ceph-mon[78593]: pgmap v10085: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:52:16.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:52:16 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:52:17.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:52:16 smithi047 ceph-mon[77147]: pgmap v10085: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:52:17.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:52:16 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:52:17.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:52:17 smithi115 conmon[90393]: 2022-05-14T04:52:17.490+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:52:17.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:52:17 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:52:18.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:52:17 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:52:18.860 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:52:18 smithi115 conmon[90393]: 2022-05-14T04:52:18.498+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:52:18.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:52:18 smithi115 ceph-mon[78593]: pgmap v10086: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:52:18.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:52:18 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:52:18.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:52:18 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:52:18.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:52:18 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:52:18.862 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:52:18 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 19470 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:52:19.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:52:18 smithi047 ceph-mon[77147]: pgmap v10086: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:52:19.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:52:18 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:52:19.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:52:18 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:52:19.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:52:18 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:52:19.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:52:18 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 19470 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:52:19.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:52:19 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:52:19.860 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:52:19 smithi115 conmon[90393]: 2022-05-14T04:52:19.477+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:52:20.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:52:19 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:52:20.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:52:20 smithi115 ceph-mon[78593]: pgmap v10087: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:52:20.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:52:20 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:52:20.861 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:52:20 smithi115 conmon[90393]: 2022-05-14T04:52:20.501+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:52:21.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:52:20 smithi047 ceph-mon[77147]: pgmap v10087: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:52:21.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:52:20 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:52:21.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:52:21 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:52:21.860 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:52:21 smithi115 conmon[90393]: 2022-05-14T04:52:21.520+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:52:22.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:52:21 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:52:22.860 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:52:22 smithi115 conmon[90393]: 2022-05-14T04:52:22.482+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:52:22.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:52:22 smithi115 ceph-mon[78593]: pgmap v10088: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:52:22.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:52:22 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:52:22.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:52:22 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:52:22.862 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:52:22 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:52:23.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:52:22 smithi047 ceph-mon[77147]: pgmap v10088: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:52:23.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:52:22 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:52:23.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:52:22 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:52:23.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:52:22 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:52:23.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:52:23 smithi115 conmon[90393]: 2022-05-14T04:52:23.530+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:52:23.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:52:23 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:52:23.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:52:23 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 19475 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:52:24.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:52:23 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:52:24.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:52:23 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 19475 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:52:24.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:52:24 smithi115 ceph-mon[78593]: pgmap v10089: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:52:24.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:52:24 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:52:24.861 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:52:24 smithi115 conmon[90393]: 2022-05-14T04:52:24.539+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:52:25.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:52:24 smithi047 ceph-mon[77147]: pgmap v10089: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:52:25.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:52:24 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:52:25.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:52:25 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:52:25.860 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:52:25 smithi115 conmon[90393]: 2022-05-14T04:52:25.519+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:52:26.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:52:25 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:52:26.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:52:26 smithi115 conmon[90393]: 2022-05-14T04:52:26.559+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:52:26.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:52:26 smithi115 ceph-mon[78593]: pgmap v10090: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:52:26.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:52:26 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:52:27.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:52:26 smithi047 ceph-mon[77147]: pgmap v10090: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:52:27.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:52:26 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:52:28.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:52:27 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:52:28.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:52:27 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:52:28.110 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:52:27 smithi115 conmon[90393]: 2022-05-14T04:52:27.608+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:52:28.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:52:28 smithi115 conmon[90393]: 2022-05-14T04:52:28.577+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:52:28.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:52:28 smithi115 ceph-mon[78593]: pgmap v10091: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:52:28.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:52:28 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:52:28.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:52:28 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 19480 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:52:29.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:52:28 smithi047 ceph-mon[77147]: pgmap v10091: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:52:29.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:52:28 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:52:29.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:52:28 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 19480 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:52:29.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:52:29 smithi115 conmon[90393]: 2022-05-14T04:52:29.585+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:52:29.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:52:29 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:52:30.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:52:29 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:52:30.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:52:30 smithi115 conmon[90393]: 2022-05-14T04:52:30.573+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:52:30.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:52:30 smithi115 ceph-mon[78593]: pgmap v10092: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:52:30.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:52:30 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:52:30.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:52:30 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:52:30.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:52:30 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:52:31.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:52:30 smithi047 ceph-mon[77147]: pgmap v10092: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:52:31.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:52:30 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:52:31.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:52:30 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:52:31.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:52:30 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:52:31.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:52:31 smithi115 conmon[90393]: 2022-05-14T04:52:31.590+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:52:31.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:52:31 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:52:32.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:52:31 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:52:32.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:52:32 smithi115 conmon[90393]: 2022-05-14T04:52:32.570+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:52:32.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:52:32 smithi115 ceph-mon[78593]: pgmap v10093: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:52:32.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:52:32 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:52:33.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:52:32 smithi047 ceph-mon[77147]: pgmap v10093: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:52:33.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:52:32 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:52:33.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:52:33 smithi115 conmon[90393]: 2022-05-14T04:52:33.585+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:52:33.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:52:33 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:52:33.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:52:33 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 19485 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:52:34.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:52:33 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:52:34.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:52:33 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 19485 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:52:35.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:52:34 smithi047 ceph-mon[77147]: pgmap v10094: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:52:35.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:52:34 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:52:35.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:52:34 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:52:35.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:52:34 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:52:35.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:52:34 smithi115 conmon[90393]: 2022-05-14T04:52:34.634+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:52:35.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:52:34 smithi115 ceph-mon[78593]: pgmap v10094: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:52:35.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:52:34 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:52:35.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:52:34 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:52:35.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:52:34 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:52:35.860 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:52:35 smithi115 conmon[90393]: 2022-05-14T04:52:35.600+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:52:35.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:52:35 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:52:36.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:52:35 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:52:36.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:52:36 smithi115 conmon[90393]: 2022-05-14T04:52:36.596+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:52:36.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:52:36 smithi115 ceph-mon[78593]: pgmap v10095: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:52:36.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:52:36 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:52:37.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:52:36 smithi047 ceph-mon[77147]: pgmap v10095: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:52:37.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:52:36 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:52:37.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:52:37 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:52:37.862 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:52:37 smithi115 conmon[90393]: 2022-05-14T04:52:37.587+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:52:38.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:52:37 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:52:38.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:52:38 smithi115 conmon[90393]: 2022-05-14T04:52:38.594+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:52:38.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:52:38 smithi115 ceph-mon[78593]: pgmap v10096: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:52:38.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:52:38 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:52:38.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:52:38 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:52:38.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:52:38 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:52:38.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:52:38 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 19490 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:52:39.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:52:38 smithi047 ceph-mon[77147]: pgmap v10096: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:52:39.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:52:38 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:52:39.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:52:38 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:52:39.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:52:38 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:52:39.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:52:38 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 19490 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:52:39.860 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:52:39 smithi115 conmon[90393]: 2022-05-14T04:52:39.569+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:52:39.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:52:39 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:52:40.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:52:39 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:52:41.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:52:40 smithi047 ceph-mon[77147]: pgmap v10097: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:52:41.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:52:40 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:52:41.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:52:40 smithi115 conmon[90393]: 2022-05-14T04:52:40.607+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:52:41.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:52:40 smithi115 ceph-mon[78593]: pgmap v10097: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:52:41.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:52:40 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:52:41.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:52:41 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:52:41.860 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:52:41 smithi115 conmon[90393]: 2022-05-14T04:52:41.604+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:52:42.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:52:41 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:52:42.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:52:42 smithi115 conmon[90393]: 2022-05-14T04:52:42.595+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:52:42.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:52:42 smithi115 ceph-mon[78593]: pgmap v10098: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:52:42.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:52:42 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:52:42.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:52:42 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:52:42.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:52:42 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:52:43.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:52:42 smithi047 ceph-mon[77147]: pgmap v10098: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:52:43.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:52:42 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:52:43.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:52:42 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:52:43.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:52:42 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:52:44.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:52:43 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:52:44.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:52:43 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 19495 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:52:44.110 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:52:43 smithi115 conmon[90393]: 2022-05-14T04:52:43.617+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:52:44.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:52:43 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:52:44.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:52:43 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 19495 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:52:44.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:52:44 smithi115 conmon[90393]: 2022-05-14T04:52:44.593+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:52:44.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:52:44 smithi115 ceph-mon[78593]: pgmap v10099: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:52:44.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:52:44 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:52:45.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:52:44 smithi047 ceph-mon[77147]: pgmap v10099: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:52:45.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:52:44 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:52:45.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:52:45 smithi115 conmon[90393]: 2022-05-14T04:52:45.586+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:52:45.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:52:45 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:52:46.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:52:45 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:52:46.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:52:46 smithi115 conmon[90393]: 2022-05-14T04:52:46.555+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:52:46.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:52:46 smithi115 ceph-mon[78593]: pgmap v10100: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:52:46.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:52:46 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:52:47.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:52:46 smithi047 ceph-mon[77147]: pgmap v10100: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:52:47.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:52:46 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:52:47.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:52:47 smithi115 conmon[90393]: 2022-05-14T04:52:47.587+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:52:47.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:52:47 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:52:48.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:52:47 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:52:49.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:52:48 smithi047 ceph-mon[77147]: pgmap v10101: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:52:49.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:52:48 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:52:49.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:52:48 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 19500 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:52:49.110 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:52:48 smithi115 conmon[90393]: 2022-05-14T04:52:48.624+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:52:49.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:52:48 smithi115 ceph-mon[78593]: pgmap v10101: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:52:49.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:52:48 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:52:49.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:52:48 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 19500 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:52:50.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:52:49 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:52:50.110 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:52:49 smithi115 conmon[90393]: 2022-05-14T04:52:49.636+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:52:50.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:52:49 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:52:50.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:52:50 smithi115 conmon[90393]: 2022-05-14T04:52:50.604+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:52:50.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:52:50 smithi115 ceph-mon[78593]: pgmap v10102: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:52:50.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:52:50 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:52:50.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:52:50 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:52:50.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:52:50 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:52:51.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:52:50 smithi047 ceph-mon[77147]: pgmap v10102: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:52:51.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:52:50 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:52:51.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:52:50 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:52:51.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:52:50 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:52:52.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:52:51 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:52:52.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:52:51 smithi115 conmon[90393]: 2022-05-14T04:52:51.652+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:52:52.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:52:51 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:52:53.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:52:52 smithi047 ceph-mon[77147]: pgmap v10103: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:52:53.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:52:52 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:52:53.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:52:52 smithi115 conmon[90393]: 2022-05-14T04:52:52.690+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:52:53.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:52:52 smithi115 ceph-mon[78593]: pgmap v10103: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:52:53.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:52:52 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:52:54.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:52:53 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:52:54.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:52:53 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 19505 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:52:54.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:52:53 smithi115 conmon[90393]: 2022-05-14T04:52:53.668+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:52:54.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:52:53 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:52:54.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:52:53 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 19505 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:52:55.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:52:54 smithi047 ceph-mon[77147]: pgmap v10104: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:52:55.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:52:54 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:52:55.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:52:54 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:52:55.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:52:54 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:52:55.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:52:54 smithi115 conmon[90393]: 2022-05-14T04:52:54.636+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:52:55.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:52:54 smithi115 ceph-mon[78593]: pgmap v10104: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:52:55.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:52:54 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:52:55.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:52:54 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:52:55.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:52:54 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:52:56.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:52:55 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:52:56.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:52:55 smithi115 conmon[90393]: 2022-05-14T04:52:55.674+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:52:56.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:52:55 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:52:57.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:52:56 smithi047 ceph-mon[77147]: pgmap v10105: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:52:57.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:52:56 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:52:57.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:52:56 smithi115 conmon[90393]: 2022-05-14T04:52:56.712+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:52:57.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:52:56 smithi115 ceph-mon[78593]: pgmap v10105: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:52:57.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:52:56 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:52:58.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:52:57 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:52:58.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:52:57 smithi115 conmon[90393]: 2022-05-14T04:52:57.702+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:52:58.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:52:57 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:52:59.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:52:58 smithi047 ceph-mon[77147]: pgmap v10106: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:52:59.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:52:58 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:52:59.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:52:58 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:52:59.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:52:58 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:52:59.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:52:58 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 19510 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:52:59.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:52:58 smithi115 conmon[90393]: 2022-05-14T04:52:58.714+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:52:59.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:52:58 smithi115 ceph-mon[78593]: pgmap v10106: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:52:59.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:52:58 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:52:59.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:52:58 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:52:59.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:52:58 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:52:59.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:52:58 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 19510 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:53:00.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:52:59 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:53:00.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:52:59 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T04:53:00.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:52:59 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T04:53:00.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:52:59 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T04:53:00.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:52:59 smithi115 conmon[90393]: 2022-05-14T04:52:59.696+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:53:00.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:52:59 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:53:00.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:52:59 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T04:53:00.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:52:59 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T04:53:00.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:52:59 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T04:53:01.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:53:00 smithi047 ceph-mon[77147]: pgmap v10107: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:53:01.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:53:00 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:53:01.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:53:00 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' 2022-05-14T04:53:01.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:53:00 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' 2022-05-14T04:53:01.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:53:00 smithi115 conmon[90393]: 2022-05-14T04:53:00.681+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:53:01.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:53:00 smithi115 ceph-mon[78593]: pgmap v10107: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:53:01.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:53:00 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:53:01.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:53:00 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' 2022-05-14T04:53:01.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:53:00 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' 2022-05-14T04:53:02.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:53:01 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:53:02.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:53:01 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:53:02.110 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:53:01 smithi115 conmon[90393]: 2022-05-14T04:53:01.726+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:53:02.769 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:53:02 smithi047 conmon[78521]: 2022-05-14T04:53:02.319+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:53:02.769 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:53:02 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T04:53:02.770 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:53:02 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:53:02.770 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:53:02 smithi047 conmon[78521]: yield 2022-05-14T04:53:02.770 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:53:02 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:53:02.770 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:53:02 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:53:02.770 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:53:02 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:53:02.771 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:53:02 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T04:53:02.771 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:53:02 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:53:02.771 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:53:02 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T04:53:02.771 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:53:02 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:53:02.772 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:53:02 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T04:53:02.772 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:53:02 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:53:02.772 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:53:02 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T04:53:02.772 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:53:02 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:53:02.772 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:53:02 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T04:53:02.773 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:53:02 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:53:02.773 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:53:02 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:53:02.773 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:53:02 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:53:03.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:53:02 smithi047 ceph-mon[77147]: pgmap v10108: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:53:03.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:53:02 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:53:03.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:53:02 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:53:03.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:53:02 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:53:03.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:53:02 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:53:03.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:53:02 smithi115 conmon[90393]: 2022-05-14T04:53:02.729+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:53:03.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:53:02 smithi115 ceph-mon[78593]: pgmap v10108: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:53:03.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:53:02 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:53:03.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:53:02 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:53:03.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:53:02 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:53:03.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:53:02 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:53:04.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:53:03 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:53:04.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T04:53:04.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:53:04.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T04:53:04.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:53:04.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:53:04.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:53:04.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T04:53:04.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:53:04.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T04:53:04.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:53:04.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T04:53:04.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:53:04.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T04:53:04.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:53:04.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T04:53:04.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:53:04.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:53:04.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:53:04.047 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:53:03 smithi047 ceph-mon[77147]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T04:53:04.048 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:53:03 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 19515 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:53:04.048 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:53:03 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:53:04.110 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:53:03 smithi115 conmon[90393]: 2022-05-14T04:53:03.712+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:53:04.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:53:03 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:53:04.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T04:53:04.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:53:04.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T04:53:04.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:53:04.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:53:04.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:53:04.113 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T04:53:04.113 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:53:04.113 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T04:53:04.113 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:53:04.113 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T04:53:04.114 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:53:04.114 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T04:53:04.114 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:53:04.114 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T04:53:04.115 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:53:04.115 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:53:04.115 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:53:04.115 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:53:03 smithi115 ceph-mon[78593]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T04:53:04.115 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:53:03 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 19515 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:53:04.116 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:53:03 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:53:05.110 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:53:04 smithi115 conmon[90393]: 2022-05-14T04:53:04.691+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:53:05.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:53:04 smithi115 ceph-mon[78593]: pgmap v10109: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:53:05.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:53:04 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:53:05.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:53:04 smithi047 ceph-mon[77147]: pgmap v10109: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:53:05.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:53:04 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:53:05.793 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:53:05 smithi047 conmon[78521]: 2022-05-14T04:53:05.391+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:53:05.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:53:05 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T04:53:05.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:53:05 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:53:05.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:53:05 smithi047 conmon[78521]: yield 2022-05-14T04:53:05.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:53:05 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:53:05.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:53:05 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:53:05.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:53:05 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:53:05.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:53:05 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T04:53:05.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:53:05 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:53:05.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:53:05 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T04:53:05.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:53:05 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:53:05.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:53:05 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T04:53:05.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:53:05 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:53:05.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:53:05 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T04:53:05.797 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:53:05 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:53:05.797 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:53:05 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T04:53:05.797 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:53:05 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:53:05.797 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:53:05 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:53:05.798 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:53:05 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:53:05.798 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:53:05 smithi047 conmon[78521]: 2022-05-14T04:53:05.392+0000 7f533ef76700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T04:53:05.798 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:53:05 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T04:53:05.799 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:53:05 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:53:05.799 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:53:05 smithi047 conmon[78521]: yield 2022-05-14T04:53:05.799 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:53:05 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:53:05.799 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:53:05 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:53:05.800 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:53:05 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:53:05.800 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:53:05 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T04:53:05.800 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:53:05 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:53:05.800 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:53:05 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T04:53:05.800 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:53:05 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:53:05.801 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:53:05 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T04:53:05.801 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:53:05 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:53:05.801 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:53:05 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T04:53:05.801 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:53:05 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:53:05.802 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:53:05 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T04:53:05.802 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:53:05 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:53:05.802 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:53:05 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:53:05.802 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:53:05 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:53:05.802 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:53:05 smithi047 conmon[78521]: 2022-05-14T04:53:05.803 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:53:05 smithi047 conmon[78521]: During handling of the above exception, another exception occurred: 2022-05-14T04:53:05.803 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:53:05 smithi047 conmon[78521]: 2022-05-14T04:53:05.803 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:53:05 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T04:53:05.803 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:53:05 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T04:53:05.803 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:53:05 smithi047 conmon[78521]: return f(*arg) 2022-05-14T04:53:05.804 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:53:05 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T04:53:05.804 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:53:05 smithi047 conmon[78521]: self._write_client_files(client_files, host) 2022-05-14T04:53:05.804 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:53:05 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T04:53:05.804 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:53:05 smithi047 conmon[78521]: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T04:53:05.805 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:53:05 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T04:53:05.805 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:53:05 smithi047 conmon[78521]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T04:53:05.805 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:53:05 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T04:53:05.805 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:53:05 smithi047 conmon[78521]: return self.event_loop.get_result(coro) 2022-05-14T04:53:05.805 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:53:05 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T04:53:05.806 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:53:05 smithi047 conmon[78521]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T04:53:05.806 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:53:05 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T04:53:05.806 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:53:05 smithi047 conmon[78521]: return self.__get_result() 2022-05-14T04:53:05.806 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:53:05 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T04:53:05.806 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:53:05 smithi047 conmon[78521]: raise self._exception 2022-05-14T04:53:05.807 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:53:05 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T04:53:05.807 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:53:05 smithi047 conmon[78521]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T04:53:05.807 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:53:05 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T04:53:05.807 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:53:05 smithi047 conmon[78521]: conn = await self._remote_connection(host, addr) 2022-05-14T04:53:05.808 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:53:05 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T04:53:05.808 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:53:05 smithi047 conmon[78521]: raise 2022-05-14T04:53:05.808 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:53:05 smithi047 conmon[78521]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T04:53:05.808 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:53:05 smithi047 conmon[78521]: self.gen.throw(type, value, traceback) 2022-05-14T04:53:05.808 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:53:05 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T04:53:05.809 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:53:05 smithi047 conmon[78521]: raise OrchestratorError(msg) 2022-05-14T04:53:05.809 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:53:05 smithi047 conmon[78521]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:53:06.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:53:05 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:53:06.110 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:53:05 smithi115 conmon[90393]: 2022-05-14T04:53:05.663+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:53:06.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:53:05 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:53:07.110 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:53:06 smithi115 conmon[90393]: 2022-05-14T04:53:06.637+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:53:07.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:53:06 smithi115 ceph-mon[78593]: pgmap v10110: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:53:07.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:53:06 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:53:07.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T04:53:07.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:53:07.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T04:53:07.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:53:07.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:53:07.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:53:07.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T04:53:07.113 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:53:07.113 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T04:53:07.113 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:53:07.113 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T04:53:07.114 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:53:07.114 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T04:53:07.114 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:53:07.114 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T04:53:07.114 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:53:07.115 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:53:07.115 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:53:07.115 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:53:06 smithi115 ceph-mon[78593]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T04:53:07.115 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T04:53:07.115 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:53:07.116 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T04:53:07.116 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:53:07.116 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:53:07.116 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:53:07.117 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T04:53:07.117 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:53:07.118 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T04:53:07.118 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:53:07.119 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T04:53:07.119 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:53:07.119 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T04:53:07.119 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:53:07.120 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T04:53:07.120 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:53:07.120 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:53:07.120 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:53:07.120 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T04:53:07.121 INFO:journalctl@ceph.mon.c.smithi115.stdout: During handling of the above exception, another exception occurred: 2022-05-14T04:53:07.121 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T04:53:07.121 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T04:53:07.121 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T04:53:07.122 INFO:journalctl@ceph.mon.c.smithi115.stdout: return f(*arg) 2022-05-14T04:53:07.122 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T04:53:07.122 INFO:journalctl@ceph.mon.c.smithi115.stdout: self._write_client_files(client_files, host) 2022-05-14T04:53:07.123 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T04:53:07.123 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T04:53:07.123 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T04:53:07.124 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T04:53:07.124 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T04:53:07.124 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.event_loop.get_result(coro) 2022-05-14T04:53:07.124 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T04:53:07.125 INFO:journalctl@ceph.mon.c.smithi115.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T04:53:07.125 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T04:53:07.125 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.__get_result() 2022-05-14T04:53:07.125 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T04:53:07.125 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise self._exception 2022-05-14T04:53:07.126 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T04:53:07.126 INFO:journalctl@ceph.mon.c.smithi115.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T04:53:07.126 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T04:53:07.126 INFO:journalctl@ceph.mon.c.smithi115.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T04:53:07.126 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T04:53:07.127 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise 2022-05-14T04:53:07.127 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T04:53:07.127 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.gen.throw(type, value, traceback) 2022-05-14T04:53:07.127 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T04:53:07.127 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OrchestratorError(msg) 2022-05-14T04:53:07.128 INFO:journalctl@ceph.mon.c.smithi115.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:53:07.128 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:53:06 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:53:07.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:53:06 smithi047 ceph-mon[77147]: pgmap v10110: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:53:07.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:53:06 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:53:07.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T04:53:07.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:53:07.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T04:53:07.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:53:07.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:53:07.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:53:07.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T04:53:07.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:53:07.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T04:53:07.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:53:07.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T04:53:07.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:53:07.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T04:53:07.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:53:07.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T04:53:07.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:53:07.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:53:07.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:53:07.297 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:53:06 smithi047 ceph-mon[77147]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T04:53:07.298 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T04:53:07.300 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:53:07.300 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T04:53:07.300 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:53:07.301 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:53:07.301 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:53:07.301 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T04:53:07.301 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:53:07.302 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T04:53:07.302 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:53:07.302 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T04:53:07.302 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:53:07.302 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T04:53:07.303 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:53:07.304 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T04:53:07.304 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:53:07.304 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:53:07.304 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:53:07.305 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T04:53:07.305 INFO:journalctl@ceph.mon.b.smithi047.stdout: During handling of the above exception, another exception occurred: 2022-05-14T04:53:07.305 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T04:53:07.305 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T04:53:07.305 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T04:53:07.306 INFO:journalctl@ceph.mon.b.smithi047.stdout: return f(*arg) 2022-05-14T04:53:07.306 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T04:53:07.306 INFO:journalctl@ceph.mon.b.smithi047.stdout: self._write_client_files(client_files, host) 2022-05-14T04:53:07.306 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T04:53:07.307 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T04:53:07.307 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T04:53:07.307 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T04:53:07.307 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T04:53:07.307 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.event_loop.get_result(coro) 2022-05-14T04:53:07.308 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T04:53:07.308 INFO:journalctl@ceph.mon.b.smithi047.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T04:53:07.308 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T04:53:07.308 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.__get_result() 2022-05-14T04:53:07.309 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T04:53:07.309 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise self._exception 2022-05-14T04:53:07.309 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T04:53:07.309 INFO:journalctl@ceph.mon.b.smithi047.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T04:53:07.310 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T04:53:07.310 INFO:journalctl@ceph.mon.b.smithi047.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T04:53:07.310 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T04:53:07.311 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise 2022-05-14T04:53:07.311 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T04:53:07.311 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.gen.throw(type, value, traceback) 2022-05-14T04:53:07.311 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T04:53:07.312 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OrchestratorError(msg) 2022-05-14T04:53:07.312 INFO:journalctl@ceph.mon.b.smithi047.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:53:07.312 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:53:06 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:53:07.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:53:07 smithi115 conmon[90393]: 2022-05-14T04:53:07.590+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:53:07.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:53:07 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:53:07.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:53:07 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 19520 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:53:08.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:53:07 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:53:08.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:53:07 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 19520 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:53:09.112 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:53:08 smithi115 conmon[90393]: 2022-05-14T04:53:08.618+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:53:09.114 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:53:08 smithi115 ceph-mon[78593]: pgmap v10111: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:53:09.114 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:53:08 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:53:09.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:53:08 smithi047 ceph-mon[77147]: pgmap v10111: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:53:09.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:53:08 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:53:10.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:53:09 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:53:10.110 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:53:09 smithi115 conmon[90393]: 2022-05-14T04:53:09.619+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:53:10.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:53:09 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:53:11.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:53:10 smithi115 conmon[90393]: 2022-05-14T04:53:10.635+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:53:11.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:53:10 smithi115 ceph-mon[78593]: pgmap v10112: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:53:11.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:53:10 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:53:11.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:53:10 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:53:11.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:53:10 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:53:11.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:53:10 smithi047 ceph-mon[77147]: pgmap v10112: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:53:11.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:53:10 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:53:11.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:53:10 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:53:11.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:53:10 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:53:11.823 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:53:11 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:53:11.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:53:11 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:53:11.862 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:53:11 smithi115 conmon[90393]: 2022-05-14T04:53:11.592+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:53:12.824 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:53:12 smithi115 conmon[90393]: 2022-05-14T04:53:12.557+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:53:13.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:53:12 smithi115 ceph-mon[78593]: pgmap v10113: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:53:13.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:53:12 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:53:13.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:53:12 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 19525 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:53:13.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:53:12 smithi047 ceph-mon[77147]: pgmap v10113: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:53:13.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:53:12 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:53:13.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:53:12 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 19525 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:53:13.831 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:53:13 smithi115 conmon[90393]: 2022-05-14T04:53:13.519+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:53:14.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:53:13 smithi115 ceph-mon[78593]: pgmap v10114: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:53:14.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:53:13 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:53:14.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:53:13 smithi047 ceph-mon[77147]: pgmap v10114: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:53:14.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:53:13 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:53:14.832 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:53:14 smithi115 conmon[90393]: 2022-05-14T04:53:14.562+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:53:15.113 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:53:14 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:53:15.114 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:53:14 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:53:15.114 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:53:14 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:53:15.114 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:53:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T04:53:15.114 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:53:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T04:53:15.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:53:14 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:53:15.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:53:14 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:53:15.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:53:14 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:53:15.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:53:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T04:53:15.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:53:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T04:53:15.835 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:53:15 smithi115 conmon[90393]: 2022-05-14T04:53:15.565+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:53:16.113 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:53:15 smithi115 ceph-mon[78593]: pgmap v10115: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:53:16.113 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:53:15 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:53:16.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:53:15 smithi047 ceph-mon[77147]: pgmap v10115: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:53:16.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:53:15 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:53:16.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:53:16 smithi115 conmon[90393]: 2022-05-14T04:53:16.595+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:53:16.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:53:16 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:53:17.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:53:16 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:53:18.110 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:53:17 smithi115 conmon[90393]: 2022-05-14T04:53:17.615+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:53:18.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:53:17 smithi115 ceph-mon[78593]: pgmap v10116: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:53:18.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:53:17 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:53:18.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:53:17 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 19530 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:53:18.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:53:17 smithi047 ceph-mon[77147]: pgmap v10116: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:53:18.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:53:17 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:53:18.295 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:53:17 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 19530 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:53:19.113 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:53:18 smithi115 conmon[90393]: 2022-05-14T04:53:18.640+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:53:19.114 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:53:18 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:53:19.114 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:53:18 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:53:19.114 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:53:18 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:53:19.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:53:18 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:53:19.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:53:18 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:53:19.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:53:18 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:53:20.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:53:19 smithi115 ceph-mon[78593]: pgmap v10117: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:53:20.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:53:19 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:53:20.111 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:53:19 smithi115 conmon[90393]: 2022-05-14T04:53:19.682+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:53:20.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:53:19 smithi047 ceph-mon[77147]: pgmap v10117: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:53:20.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:53:19 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:53:21.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:53:20 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:53:21.110 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:53:20 smithi115 conmon[90393]: 2022-05-14T04:53:20.716+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:53:21.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:53:20 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:53:22.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:53:21 smithi115 ceph-mon[78593]: pgmap v10118: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:53:22.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:53:21 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:53:22.110 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:53:21 smithi115 conmon[90393]: 2022-05-14T04:53:21.741+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:53:22.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:53:21 smithi047 ceph-mon[77147]: pgmap v10118: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:53:22.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:53:21 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:53:23.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:53:22 smithi115 conmon[90393]: 2022-05-14T04:53:22.698+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:53:23.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:53:22 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:53:23.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:53:22 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:53:23.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:53:22 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:53:23.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:53:22 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 19535 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:53:23.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:53:22 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:53:23.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:53:22 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:53:23.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:53:22 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:53:23.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:53:22 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 19535 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:53:24.110 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:53:23 smithi115 conmon[90393]: 2022-05-14T04:53:23.741+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:53:24.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:53:23 smithi115 ceph-mon[78593]: pgmap v10119: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:53:24.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:53:23 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:53:24.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:53:23 smithi047 ceph-mon[77147]: pgmap v10119: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:53:24.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:53:23 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:53:25.110 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:53:24 smithi115 conmon[90393]: 2022-05-14T04:53:24.791+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:53:26.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:53:25 smithi115 conmon[90393]: 2022-05-14T04:53:25.831+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:53:26.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:53:26 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:53:26.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:53:26 smithi047 ceph-mon[77147]: pgmap v10120: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:53:26.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:53:26 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:53:26.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:53:26 smithi115 ceph-mon[78593]: pgmap v10120: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:53:27.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:53:26 smithi115 conmon[90393]: 2022-05-14T04:53:26.856+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:53:27.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:53:27 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:53:27.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:53:27 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:53:28.258 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:53:27 smithi115 conmon[90393]: 2022-05-14T04:53:27.875+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:53:28.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:53:28 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:53:28.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:53:28 smithi047 ceph-mon[77147]: pgmap v10121: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:53:28.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:53:28 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 19540 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:53:28.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:53:28 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:53:28.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:53:28 smithi115 ceph-mon[78593]: pgmap v10121: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:53:28.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:53:28 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 19540 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:53:29.264 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:53:28 smithi115 conmon[90393]: 2022-05-14T04:53:28.898+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:53:29.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:53:29 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:53:29.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:53:29 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:53:30.265 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:53:29 smithi115 conmon[90393]: 2022-05-14T04:53:29.893+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:53:30.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:53:30 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:53:30.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:53:30 smithi047 ceph-mon[77147]: pgmap v10122: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:53:30.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:53:30 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:53:30.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:53:30 smithi115 ceph-mon[78593]: pgmap v10122: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:53:31.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:53:30 smithi115 conmon[90393]: 2022-05-14T04:53:30.849+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:53:31.548 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:53:31 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:53:31.549 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:53:31 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:53:31.549 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:53:31 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:53:31.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:53:31 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:53:31.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:53:31 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:53:31.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:53:31 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:53:32.277 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:53:31 smithi115 conmon[90393]: 2022-05-14T04:53:31.869+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:53:32.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:53:32 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:53:32.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:53:32 smithi047 ceph-mon[77147]: pgmap v10123: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:53:32.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:53:32 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:53:32.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:53:32 smithi115 ceph-mon[78593]: pgmap v10123: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:53:33.291 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:53:32 smithi115 conmon[90393]: 2022-05-14T04:53:32.860+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:53:33.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:53:33 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:53:33.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:53:33 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 19545 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:53:33.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:53:33 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:53:33.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:53:33 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 19545 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:53:34.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:53:33 smithi115 conmon[90393]: 2022-05-14T04:53:33.816+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:53:34.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:53:34 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:53:34.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:53:34 smithi047 ceph-mon[77147]: pgmap v10124: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:53:34.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:53:34 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:53:34.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:53:34 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:53:34.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:53:34 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:53:34.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:53:34 smithi115 ceph-mon[78593]: pgmap v10124: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:53:34.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:53:34 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:53:34.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:53:34 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:53:35.110 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:53:34 smithi115 conmon[90393]: 2022-05-14T04:53:34.837+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:53:35.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:53:35 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:53:35.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:53:35 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:53:36.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:53:35 smithi115 conmon[90393]: 2022-05-14T04:53:35.837+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:53:36.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:53:36 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:53:36.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:53:36 smithi047 ceph-mon[77147]: pgmap v10125: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:53:36.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:53:36 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:53:36.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:53:36 smithi115 ceph-mon[78593]: pgmap v10125: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:53:37.291 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:53:36 smithi115 conmon[90393]: 2022-05-14T04:53:36.861+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:53:37.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:53:37 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:53:37.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:53:37 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:53:38.110 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:53:37 smithi115 conmon[90393]: 2022-05-14T04:53:37.819+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:53:38.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:53:38 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:53:38.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:53:38 smithi047 ceph-mon[77147]: pgmap v10126: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:53:38.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:53:38 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:53:38.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:53:38 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:53:38.545 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:53:38 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 19550 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:53:38.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:53:38 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:53:38.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:53:38 smithi115 ceph-mon[78593]: pgmap v10126: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:53:38.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:53:38 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:53:38.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:53:38 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:53:38.612 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:53:38 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 19550 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:53:39.298 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:53:38 smithi115 conmon[90393]: 2022-05-14T04:53:38.857+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:53:39.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:53:39 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:53:39.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:53:39 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:53:40.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:53:39 smithi115 conmon[90393]: 2022-05-14T04:53:39.822+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:53:40.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:53:40 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:53:40.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:53:40 smithi115 ceph-mon[78593]: pgmap v10127: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:53:40.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:53:40 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:53:40.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:53:40 smithi047 ceph-mon[77147]: pgmap v10127: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:53:41.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:53:40 smithi115 conmon[90393]: 2022-05-14T04:53:40.843+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:53:41.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:53:41 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:53:41.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:53:41 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:53:42.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:53:41 smithi115 conmon[90393]: 2022-05-14T04:53:41.804+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:53:42.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:53:42 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:53:42.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:53:42 smithi115 ceph-mon[78593]: pgmap v10128: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:53:42.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:53:42 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:53:42.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:53:42 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:53:42.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:53:42 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:53:42.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:53:42 smithi047 ceph-mon[77147]: pgmap v10128: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:53:42.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:53:42 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:53:42.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:53:42 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:53:43.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:53:42 smithi115 conmon[90393]: 2022-05-14T04:53:42.827+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:53:43.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:53:43 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:53:43.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:53:43 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 19555 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:53:43.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:53:43 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:53:43.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:53:43 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 19555 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:53:44.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:53:43 smithi115 conmon[90393]: 2022-05-14T04:53:43.792+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:53:44.612 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:53:44 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:53:44.612 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:53:44 smithi115 ceph-mon[78593]: pgmap v10129: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:53:44.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:53:44 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:53:44.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:53:44 smithi047 ceph-mon[77147]: pgmap v10129: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:53:45.112 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:53:44 smithi115 conmon[90393]: 2022-05-14T04:53:44.762+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:53:45.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:53:45 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:53:45.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:53:45 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:53:45.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:53:45 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:53:45.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:53:45 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:53:46.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:53:45 smithi115 conmon[90393]: 2022-05-14T04:53:45.787+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:53:46.612 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:53:46 smithi115 ceph-mon[78593]: pgmap v10130: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:53:46.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:53:46 smithi047 ceph-mon[77147]: pgmap v10130: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:53:47.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:53:46 smithi115 conmon[90393]: 2022-05-14T04:53:46.834+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:53:47.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:53:47 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:53:47.795 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:53:47 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:53:48.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:53:47 smithi115 conmon[90393]: 2022-05-14T04:53:47.844+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:53:48.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:53:48 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:53:48.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:53:48 smithi115 ceph-mon[78593]: pgmap v10131: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:53:48.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:53:48 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 19560 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:53:48.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:53:48 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:53:48.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:53:48 smithi047 ceph-mon[77147]: pgmap v10131: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:53:48.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:53:48 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 19560 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:53:49.330 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:53:48 smithi115 conmon[90393]: 2022-05-14T04:53:48.866+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:53:49.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:53:49 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:53:49.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:53:49 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:53:50.328 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:53:49 smithi115 conmon[90393]: 2022-05-14T04:53:49.887+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:53:50.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:53:50 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:53:50.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:53:50 smithi115 ceph-mon[78593]: pgmap v10132: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:53:50.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:53:50 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:53:50.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:53:50 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:53:50.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:53:50 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:53:50.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:53:50 smithi047 ceph-mon[77147]: pgmap v10132: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:53:50.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:53:50 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:53:50.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:53:50 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:53:51.336 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:53:50 smithi115 conmon[90393]: 2022-05-14T04:53:50.924+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:53:51.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:53:51 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:53:51.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:53:51 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:53:52.335 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:53:51 smithi115 conmon[90393]: 2022-05-14T04:53:51.881+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:53:52.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:53:52 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:53:52.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:53:52 smithi115 ceph-mon[78593]: pgmap v10133: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:53:52.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:53:52 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:53:52.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:53:52 smithi047 ceph-mon[77147]: pgmap v10133: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:53:53.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:53:52 smithi115 conmon[90393]: 2022-05-14T04:53:52.852+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:53:53.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:53:53 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:53:53.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:53:53 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 19565 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:53:53.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:53:53 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:53:53.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:53:53 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 19565 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:53:54.338 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:53:53 smithi115 conmon[90393]: 2022-05-14T04:53:53.871+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:53:54.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:53:54 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:53:54.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:53:54 smithi115 ceph-mon[78593]: pgmap v10134: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:53:54.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:53:54 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:53:54.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:53:54 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:53:54.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:53:54 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:53:54.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:53:54 smithi047 ceph-mon[77147]: pgmap v10134: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:53:54.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:53:54 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:53:54.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:53:54 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:53:55.344 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:53:54 smithi115 conmon[90393]: 2022-05-14T04:53:54.889+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:53:55.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:53:55 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:53:55.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:53:55 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:53:56.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:53:55 smithi115 conmon[90393]: 2022-05-14T04:53:55.847+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:53:56.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:53:56 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:53:56.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:53:56 smithi115 ceph-mon[78593]: pgmap v10135: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:53:56.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:53:56 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:53:56.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:53:56 smithi047 ceph-mon[77147]: pgmap v10135: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:53:57.351 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:53:56 smithi115 conmon[90393]: 2022-05-14T04:53:56.880+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:53:57.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:53:57 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:53:57.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:53:57 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:53:58.352 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:53:57 smithi115 conmon[90393]: 2022-05-14T04:53:57.871+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:53:58.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:53:58 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:53:58.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:53:58 smithi115 ceph-mon[78593]: pgmap v10136: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:53:58.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:53:58 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:53:58.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:53:58 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:53:58.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:53:58 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 19570 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:53:58.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:53:58 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:53:58.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:53:58 smithi047 ceph-mon[77147]: pgmap v10136: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:53:58.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:53:58 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:53:58.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:53:58 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:53:58.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:53:58 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 19570 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:53:59.358 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:53:58 smithi115 conmon[90393]: 2022-05-14T04:53:58.900+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:53:59.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:53:59 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:53:59.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:53:59 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:54:00.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:53:59 smithi115 conmon[90393]: 2022-05-14T04:53:59.851+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:54:00.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:54:00 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:54:00.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:54:00 smithi047 ceph-mon[77147]: pgmap v10137: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:54:00.832 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:54:00 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:54:00.832 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:54:00 smithi115 ceph-mon[78593]: pgmap v10137: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:54:01.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:54:00 smithi115 conmon[90393]: 2022-05-14T04:54:00.829+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:54:01.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:54:01 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:54:01.823 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:54:01 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:54:02.110 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:54:01 smithi115 conmon[90393]: 2022-05-14T04:54:01.820+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:54:02.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:54:02 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:54:02.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:54:02 smithi047 ceph-mon[77147]: pgmap v10138: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:54:02.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:54:02 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:54:02.795 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:54:02 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:54:02.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:54:02 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:54:02.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:54:02 smithi115 ceph-mon[78593]: pgmap v10138: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:54:02.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:54:02 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:54:02.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:54:02 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:54:03.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:54:02 smithi115 conmon[90393]: 2022-05-14T04:54:02.867+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:54:03.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:54:03 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:54:03.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:54:03 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 19575 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:54:03.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:54:03 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:54:03.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:54:03 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 19575 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:54:04.360 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:54:03 smithi115 conmon[90393]: 2022-05-14T04:54:03.877+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:54:04.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:54:04 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:54:04.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:54:04 smithi047 ceph-mon[77147]: pgmap v10139: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:54:04.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:54:04 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:54:04.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:54:04 smithi115 ceph-mon[78593]: pgmap v10139: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:54:05.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:54:04 smithi115 conmon[90393]: 2022-05-14T04:54:04.904+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:54:05.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:54:05 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:54:05.853 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:54:05 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:54:06.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:54:05 smithi115 conmon[90393]: 2022-05-14T04:54:05.906+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:54:06.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:54:06 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:54:06.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:54:06 smithi047 ceph-mon[77147]: pgmap v10140: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:54:06.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:54:06 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T04:54:06.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:54:06 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T04:54:06.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:54:06 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T04:54:06.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:54:06 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' 2022-05-14T04:54:06.795 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:54:06 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' 2022-05-14T04:54:06.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:54:06 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:54:06.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:54:06 smithi115 ceph-mon[78593]: pgmap v10140: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:54:06.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:54:06 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T04:54:06.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:54:06 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T04:54:06.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:54:06 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T04:54:06.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:54:06 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' 2022-05-14T04:54:06.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:54:06 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' 2022-05-14T04:54:07.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:54:06 smithi115 conmon[90393]: 2022-05-14T04:54:06.908+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:54:07.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:54:07 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:54:07.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:54:07 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:54:08.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:54:07 smithi115 conmon[90393]: 2022-05-14T04:54:07.905+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:54:08.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:54:08 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:54:08.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:54:08 smithi047 ceph-mon[77147]: pgmap v10141: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:54:08.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:54:08 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 19580 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:54:08.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:54:08 smithi047 conmon[78521]: 2022-05-14T04:54:08.496+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:54:08.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:54:08 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T04:54:08.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:54:08 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:54:08.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:54:08 smithi047 conmon[78521]: yield 2022-05-14T04:54:08.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:54:08 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:54:08.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:54:08 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:54:08.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:54:08 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:54:08.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:54:08 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T04:54:08.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:54:08 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:54:08.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:54:08 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T04:54:08.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:54:08 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:54:08.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:54:08 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T04:54:08.797 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:54:08 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:54:08.797 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:54:08 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T04:54:08.797 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:54:08 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:54:08.797 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:54:08 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T04:54:08.798 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:54:08 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:54:08.798 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:54:08 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:54:08.798 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:54:08 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:54:08.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:54:08 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:54:08.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:54:08 smithi115 ceph-mon[78593]: pgmap v10141: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:54:08.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:54:08 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 19580 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:54:09.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:54:08 smithi115 conmon[90393]: 2022-05-14T04:54:08.917+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:54:09.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:54:09 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:54:09.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:54:09 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' 2022-05-14T04:54:09.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:54:09 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' 2022-05-14T04:54:09.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:54:09 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:54:09.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T04:54:09.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:54:09.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T04:54:09.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:54:09.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:54:09.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:54:09.797 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T04:54:09.797 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:54:09.798 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T04:54:09.798 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:54:09.799 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T04:54:09.799 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:54:09.800 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T04:54:09.800 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:54:09.800 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T04:54:09.801 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:54:09.801 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:54:09.802 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:54:09.802 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:54:09 smithi047 ceph-mon[77147]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T04:54:09.803 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:54:09 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-05-14T04:54:09.803 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:54:09 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-05-14T04:54:09.804 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:54:09 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-05-14T04:54:09.804 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:54:09 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-05-14T04:54:09.804 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:54:09 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-05-14T04:54:09.805 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:54:09 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-05-14T04:54:09.805 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:54:09 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-05-14T04:54:09.806 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:54:09 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-05-14T04:54:09.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:54:09 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:54:09.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:54:09 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' 2022-05-14T04:54:09.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:54:09 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' 2022-05-14T04:54:09.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:54:09 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:54:09.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T04:54:09.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:54:09.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T04:54:09.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:54:09.863 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:54:09.863 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:54:09.864 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T04:54:09.864 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:54:09.865 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T04:54:09.865 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:54:09.866 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T04:54:09.866 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:54:09.866 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T04:54:09.867 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:54:09.867 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T04:54:09.868 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:54:09.868 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:54:09.868 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:54:09.869 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:54:09 smithi115 ceph-mon[78593]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T04:54:09.869 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:54:09 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-05-14T04:54:09.869 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:54:09 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-05-14T04:54:09.870 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:54:09 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-05-14T04:54:09.870 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:54:09 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-05-14T04:54:09.870 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:54:09 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-05-14T04:54:09.870 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:54:09 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-05-14T04:54:09.870 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:54:09 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-05-14T04:54:09.871 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:54:09 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-05-14T04:54:10.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:54:09 smithi115 conmon[90393]: 2022-05-14T04:54:09.914+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:54:10.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:54:10 smithi047 ceph-mon[77147]: Adjusting osd_memory_target on smithi115 to 197.0M 2022-05-14T04:54:10.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:54:10 smithi047 ceph-mon[77147]: Unable to set osd_memory_target on smithi115 to 206629785: error parsing value: Value '206629785' is below minimum 939524096 2022-05-14T04:54:10.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:54:10 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:54:10.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:54:10 smithi047 ceph-mon[77147]: Adjusting osd_memory_target on smithi047 to 198.1M 2022-05-14T04:54:10.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:54:10 smithi047 ceph-mon[77147]: Unable to set osd_memory_target on smithi047 to 207755878: error parsing value: Value '207755878' is below minimum 939524096 2022-05-14T04:54:10.795 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:54:10 smithi047 ceph-mon[77147]: pgmap v10142: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:54:10.796 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:54:10 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:54:10.796 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:54:10 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:54:10.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:54:10 smithi115 ceph-mon[78593]: Adjusting osd_memory_target on smithi115 to 197.0M 2022-05-14T04:54:10.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:54:10 smithi115 ceph-mon[78593]: Unable to set osd_memory_target on smithi115 to 206629785: error parsing value: Value '206629785' is below minimum 939524096 2022-05-14T04:54:10.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:54:10 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:54:10.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:54:10 smithi115 ceph-mon[78593]: Adjusting osd_memory_target on smithi047 to 198.1M 2022-05-14T04:54:10.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:54:10 smithi115 ceph-mon[78593]: Unable to set osd_memory_target on smithi047 to 207755878: error parsing value: Value '207755878' is below minimum 939524096 2022-05-14T04:54:10.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:54:10 smithi115 ceph-mon[78593]: pgmap v10142: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:54:10.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:54:10 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:54:10.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:54:10 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:54:11.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:54:10 smithi115 conmon[90393]: 2022-05-14T04:54:10.907+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:54:11.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:54:11 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:54:11.793 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:54:11 smithi047 conmon[78521]: 2022-05-14T04:54:11.567+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:54:11.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:54:11 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T04:54:11.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:54:11 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:54:11.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:54:11 smithi047 conmon[78521]: yield 2022-05-14T04:54:11.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:54:11 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:54:11.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:54:11 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:54:11.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:54:11 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:54:11.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:54:11 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T04:54:11.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:54:11 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:54:11.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:54:11 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T04:54:11.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:54:11 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:54:11.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:54:11 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T04:54:11.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:54:11 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:54:11.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:54:11 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T04:54:11.797 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:54:11 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:54:11.797 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:54:11 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T04:54:11.797 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:54:11 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:54:11.797 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:54:11 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:54:11.797 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:54:11 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:54:11.798 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:54:11 smithi047 conmon[78521]: 2022-05-14T04:54:11.568+0000 7f5344781700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T04:54:11.798 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:54:11 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T04:54:11.798 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:54:11 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:54:11.798 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:54:11 smithi047 conmon[78521]: yield 2022-05-14T04:54:11.799 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:54:11 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:54:11.799 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:54:11 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:54:11.799 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:54:11 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:54:11.800 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:54:11 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T04:54:11.800 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:54:11 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:54:11.800 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:54:11 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T04:54:11.800 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:54:11 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:54:11.800 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:54:11 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T04:54:11.801 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:54:11 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:54:11.801 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:54:11 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T04:54:11.801 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:54:11 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:54:11.801 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:54:11 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T04:54:11.802 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:54:11 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:54:11.802 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:54:11 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:54:11.802 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:54:11 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:54:11.802 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:54:11 smithi047 conmon[78521]: 2022-05-14T04:54:11.802 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:54:11 smithi047 conmon[78521]: During handling of the above exception, another exception occurred: 2022-05-14T04:54:11.803 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:54:11 smithi047 conmon[78521]: 2022-05-14T04:54:11.803 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:54:11 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T04:54:11.803 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:54:11 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T04:54:11.803 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:54:11 smithi047 conmon[78521]: return f(*arg) 2022-05-14T04:54:11.803 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:54:11 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T04:54:11.804 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:54:11 smithi047 conmon[78521]: self._write_client_files(client_files, host) 2022-05-14T04:54:11.804 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:54:11 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T04:54:11.804 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:54:11 smithi047 conmon[78521]: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T04:54:11.804 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:54:11 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T04:54:11.805 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:54:11 smithi047 conmon[78521]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T04:54:11.805 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:54:11 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T04:54:11.805 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:54:11 smithi047 conmon[78521]: return self.event_loop.get_result(coro) 2022-05-14T04:54:11.805 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:54:11 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T04:54:11.806 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:54:11 smithi047 conmon[78521]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T04:54:11.806 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:54:11 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T04:54:11.806 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:54:11 smithi047 conmon[78521]: return self.__get_result() 2022-05-14T04:54:11.806 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:54:11 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T04:54:11.806 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:54:11 smithi047 conmon[78521]: raise self._exception 2022-05-14T04:54:11.808 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:54:11 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T04:54:11.808 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:54:11 smithi047 conmon[78521]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T04:54:11.808 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:54:11 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T04:54:11.809 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:54:11 smithi047 conmon[78521]: conn = await self._remote_connection(host, addr) 2022-05-14T04:54:11.809 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:54:11 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T04:54:11.809 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:54:11 smithi047 conmon[78521]: raise 2022-05-14T04:54:11.810 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:54:11 smithi047 conmon[78521]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T04:54:11.810 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:54:11 smithi047 conmon[78521]: self.gen.throw(type, value, traceback) 2022-05-14T04:54:11.810 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:54:11 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T04:54:11.811 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:54:11 smithi047 conmon[78521]: raise OrchestratorError(msg) 2022-05-14T04:54:11.811 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:54:11 smithi047 conmon[78521]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:54:11.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:54:11 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:54:12.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:54:11 smithi115 conmon[90393]: 2022-05-14T04:54:11.862+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:54:12.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:54:12 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:54:12.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:54:12 smithi047 ceph-mon[77147]: pgmap v10143: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:54:12.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:54:12 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:54:12.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T04:54:12.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:54:12.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T04:54:12.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:54:12.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:54:12.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:54:12.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T04:54:12.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:54:12.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T04:54:12.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:54:12.797 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T04:54:12.797 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:54:12.797 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T04:54:12.798 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:54:12.798 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T04:54:12.798 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:54:12.798 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:54:12.798 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:54:12.799 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:54:12 smithi047 ceph-mon[77147]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T04:54:12.799 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T04:54:12.799 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:54:12.799 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T04:54:12.800 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:54:12.800 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:54:12.801 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:54:12.801 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T04:54:12.801 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:54:12.801 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T04:54:12.802 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:54:12.802 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T04:54:12.802 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:54:12.802 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T04:54:12.802 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:54:12.803 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T04:54:12.803 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:54:12.803 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:54:12.803 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:54:12.803 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T04:54:12.804 INFO:journalctl@ceph.mon.b.smithi047.stdout: During handling of the above exception, another exception occurred: 2022-05-14T04:54:12.804 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T04:54:12.804 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T04:54:12.804 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T04:54:12.804 INFO:journalctl@ceph.mon.b.smithi047.stdout: return f(*arg) 2022-05-14T04:54:12.805 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T04:54:12.805 INFO:journalctl@ceph.mon.b.smithi047.stdout: self._write_client_files(client_files, host) 2022-05-14T04:54:12.805 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T04:54:12.805 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T04:54:12.805 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T04:54:12.806 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T04:54:12.806 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T04:54:12.806 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.event_loop.get_result(coro) 2022-05-14T04:54:12.806 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T04:54:12.807 INFO:journalctl@ceph.mon.b.smithi047.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T04:54:12.807 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T04:54:12.807 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.__get_result() 2022-05-14T04:54:12.807 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T04:54:12.807 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise self._exception 2022-05-14T04:54:12.808 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T04:54:12.808 INFO:journalctl@ceph.mon.b.smithi047.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T04:54:12.808 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T04:54:12.808 INFO:journalctl@ceph.mon.b.smithi047.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T04:54:12.808 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T04:54:12.809 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise 2022-05-14T04:54:12.809 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T04:54:12.809 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.gen.throw(type, value, traceback) 2022-05-14T04:54:12.809 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T04:54:12.810 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OrchestratorError(msg) 2022-05-14T04:54:12.810 INFO:journalctl@ceph.mon.b.smithi047.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:54:12.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:54:12 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:54:12.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:54:12 smithi115 ceph-mon[78593]: pgmap v10143: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:54:12.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:54:12 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:54:12.860 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T04:54:12.860 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:54:12.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T04:54:12.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:54:12.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:54:12.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:54:12.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T04:54:12.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:54:12.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T04:54:12.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:54:12.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T04:54:12.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:54:12.863 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T04:54:12.863 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:54:12.863 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T04:54:12.863 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:54:12.863 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:54:12.864 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:54:12.864 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:54:12 smithi115 ceph-mon[78593]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T04:54:12.864 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T04:54:12.864 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:54:12.864 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T04:54:12.865 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:54:12.865 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:54:12.865 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:54:12.865 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T04:54:12.866 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:54:12.866 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T04:54:12.866 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:54:12.866 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T04:54:12.866 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:54:12.867 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T04:54:12.867 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:54:12.867 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T04:54:12.867 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:54:12.867 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:54:12.868 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:54:12.868 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T04:54:12.868 INFO:journalctl@ceph.mon.c.smithi115.stdout: During handling of the above exception, another exception occurred: 2022-05-14T04:54:12.868 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T04:54:12.868 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T04:54:12.868 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T04:54:12.869 INFO:journalctl@ceph.mon.c.smithi115.stdout: return f(*arg) 2022-05-14T04:54:12.869 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T04:54:12.869 INFO:journalctl@ceph.mon.c.smithi115.stdout: self._write_client_files(client_files, host) 2022-05-14T04:54:12.869 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T04:54:12.870 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T04:54:12.870 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T04:54:12.870 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T04:54:12.870 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T04:54:12.870 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.event_loop.get_result(coro) 2022-05-14T04:54:12.871 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T04:54:12.871 INFO:journalctl@ceph.mon.c.smithi115.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T04:54:12.871 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T04:54:12.871 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.__get_result() 2022-05-14T04:54:12.871 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T04:54:12.872 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise self._exception 2022-05-14T04:54:12.872 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T04:54:12.872 INFO:journalctl@ceph.mon.c.smithi115.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T04:54:12.872 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T04:54:12.872 INFO:journalctl@ceph.mon.c.smithi115.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T04:54:12.873 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T04:54:12.873 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise 2022-05-14T04:54:12.873 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T04:54:12.873 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.gen.throw(type, value, traceback) 2022-05-14T04:54:12.873 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T04:54:12.874 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OrchestratorError(msg) 2022-05-14T04:54:12.874 INFO:journalctl@ceph.mon.c.smithi115.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:54:13.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:54:12 smithi115 conmon[90393]: 2022-05-14T04:54:12.898+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:54:13.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:54:13 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:54:13.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:54:13 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 19585 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:54:13.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:54:13 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:54:13.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:54:13 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 19585 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:54:14.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:54:13 smithi115 conmon[90393]: 2022-05-14T04:54:13.885+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:54:14.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:54:14 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:54:14.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:54:14 smithi047 ceph-mon[77147]: pgmap v10144: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:54:14.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:54:14 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:54:14.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:54:14 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:54:14.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:54:14 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:54:14.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:54:14 smithi115 ceph-mon[78593]: pgmap v10144: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:54:14.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:54:14 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:54:14.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:54:14 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:54:15.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:54:14 smithi115 conmon[90393]: 2022-05-14T04:54:14.880+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:54:15.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:54:15 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:54:15.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:54:15 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T04:54:15.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:54:15 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T04:54:15.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:54:15 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:54:15.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:54:15 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T04:54:15.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:54:15 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T04:54:16.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:54:15 smithi115 conmon[90393]: 2022-05-14T04:54:15.864+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:54:16.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:54:16 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:54:16.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:54:16 smithi047 ceph-mon[77147]: pgmap v10145: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:54:16.838 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:54:16 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:54:16.839 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:54:16 smithi115 ceph-mon[78593]: pgmap v10145: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:54:17.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:54:16 smithi115 conmon[90393]: 2022-05-14T04:54:16.834+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:54:17.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:54:17 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:54:17.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:54:17 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:54:18.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:54:17 smithi115 conmon[90393]: 2022-05-14T04:54:17.794+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:54:18.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:54:18 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:54:18.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:54:18 smithi047 ceph-mon[77147]: pgmap v10146: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:54:18.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:54:18 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:54:18.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:54:18 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:54:18.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:54:18 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 19590 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:54:18.824 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:54:18 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:54:18.825 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:54:18 smithi115 ceph-mon[78593]: pgmap v10146: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:54:18.825 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:54:18 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:54:18.825 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:54:18 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:54:18.826 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:54:18 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 19590 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:54:19.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:54:18 smithi115 conmon[90393]: 2022-05-14T04:54:18.821+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:54:19.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:54:19 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:54:19.829 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:54:19 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:54:20.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:54:19 smithi115 conmon[90393]: 2022-05-14T04:54:19.826+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:54:20.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:54:20 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:54:20.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:54:20 smithi047 ceph-mon[77147]: pgmap v10147: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:54:20.814 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:54:20 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:54:20.814 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:54:20 smithi115 ceph-mon[78593]: pgmap v10147: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:54:21.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:54:20 smithi115 conmon[90393]: 2022-05-14T04:54:20.811+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:54:21.765 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:54:21 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:54:21.796 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:54:21 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:54:22.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:54:21 smithi115 conmon[90393]: 2022-05-14T04:54:21.762+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:54:22.766 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:54:22 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:54:22.767 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:54:22 smithi115 ceph-mon[78593]: pgmap v10148: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:54:22.767 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:54:22 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:54:22.767 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:54:22 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:54:22.768 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:54:22 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:54:22.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:54:22 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:54:22.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:54:22 smithi047 ceph-mon[77147]: pgmap v10148: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:54:22.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:54:22 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:54:22.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:54:22 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:54:22.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:54:22 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:54:23.110 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:54:22 smithi115 conmon[90393]: 2022-05-14T04:54:22.763+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:54:23.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:54:23 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:54:23.795 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:54:23 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 19595 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:54:23.801 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:54:23 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:54:23.801 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:54:23 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 19595 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:54:24.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:54:23 smithi115 conmon[90393]: 2022-05-14T04:54:23.797+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:54:25.110 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:54:24 smithi115 conmon[90393]: 2022-05-14T04:54:24.750+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:54:25.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:54:24 smithi115 ceph-mon[78593]: pgmap v10149: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:54:25.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:54:24 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:54:25.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:54:24 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:54:25.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:54:24 smithi047 ceph-mon[77147]: pgmap v10149: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:54:25.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:54:24 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:54:25.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:54:24 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:54:26.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:54:25 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:54:26.110 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:54:25 smithi115 conmon[90393]: 2022-05-14T04:54:25.713+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:54:26.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:54:25 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:54:27.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:54:26 smithi115 conmon[90393]: 2022-05-14T04:54:26.684+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:54:27.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:54:26 smithi115 ceph-mon[78593]: pgmap v10150: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:54:27.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:54:26 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:54:27.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:54:26 smithi047 ceph-mon[77147]: pgmap v10150: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:54:27.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:54:26 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:54:28.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:54:27 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:54:28.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:54:27 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 19600 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:54:28.110 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:54:27 smithi115 conmon[90393]: 2022-05-14T04:54:27.722+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:54:28.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:54:27 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:54:28.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:54:27 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 19600 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:54:29.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:54:28 smithi115 ceph-mon[78593]: pgmap v10151: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:54:29.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:54:28 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:54:29.110 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:54:28 smithi115 conmon[90393]: 2022-05-14T04:54:28.760+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:54:29.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:54:28 smithi047 ceph-mon[77147]: pgmap v10151: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:54:29.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:54:28 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:54:30.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:54:29 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:54:30.110 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:54:29 smithi115 conmon[90393]: 2022-05-14T04:54:29.710+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:54:30.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:54:29 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:54:31.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:54:30 smithi115 conmon[90393]: 2022-05-14T04:54:30.712+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:54:31.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:54:30 smithi115 ceph-mon[78593]: pgmap v10152: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:54:31.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:54:30 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:54:31.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:54:30 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:54:31.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:54:30 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:54:31.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:54:30 smithi047 ceph-mon[77147]: pgmap v10152: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:54:31.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:54:30 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:54:31.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:54:30 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:54:31.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:54:30 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:54:32.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:54:31 smithi115 ceph-mon[78593]: pgmap v10153: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:54:32.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:54:31 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:54:32.111 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:54:31 smithi115 conmon[90393]: 2022-05-14T04:54:31.670+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:54:32.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:54:31 smithi047 ceph-mon[77147]: pgmap v10153: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:54:32.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:54:31 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:54:33.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:54:32 smithi115 conmon[90393]: 2022-05-14T04:54:32.651+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:54:33.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:54:32 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:54:33.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:54:32 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 19605 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:54:33.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:54:32 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:54:33.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:54:32 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 19605 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:54:34.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:54:33 smithi115 conmon[90393]: 2022-05-14T04:54:33.656+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:54:34.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:54:33 smithi115 ceph-mon[78593]: pgmap v10154: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:54:34.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:54:33 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:54:34.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:54:33 smithi047 ceph-mon[77147]: pgmap v10154: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:54:34.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:54:33 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:54:35.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:54:34 smithi115 conmon[90393]: 2022-05-14T04:54:34.688+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:54:35.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:54:34 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:54:35.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:54:34 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:54:35.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:54:34 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:54:35.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:54:34 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:54:35.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:54:34 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:54:35.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:54:34 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:54:36.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:54:35 smithi115 conmon[90393]: 2022-05-14T04:54:35.685+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:54:36.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:54:35 smithi115 ceph-mon[78593]: pgmap v10155: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:54:36.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:54:35 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:54:36.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:54:35 smithi047 ceph-mon[77147]: pgmap v10155: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:54:36.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:54:35 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:54:37.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:54:36 smithi115 conmon[90393]: 2022-05-14T04:54:36.718+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:54:37.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:54:36 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:54:37.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:54:36 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:54:38.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:54:37 smithi115 conmon[90393]: 2022-05-14T04:54:37.694+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:54:38.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:54:37 smithi115 ceph-mon[78593]: pgmap v10156: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:54:38.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:54:37 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:54:38.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:54:37 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 19610 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:54:38.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:54:37 smithi047 ceph-mon[77147]: pgmap v10156: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:54:38.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:54:37 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:54:38.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:54:37 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 19610 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:54:39.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:54:38 smithi115 conmon[90393]: 2022-05-14T04:54:38.665+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:54:39.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:54:38 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:54:39.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:54:38 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:54:39.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:54:38 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:54:39.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:54:38 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:54:39.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:54:38 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:54:39.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:54:38 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:54:40.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:54:39 smithi115 conmon[90393]: 2022-05-14T04:54:39.711+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:54:40.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:54:39 smithi115 ceph-mon[78593]: pgmap v10157: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:54:40.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:54:39 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:54:40.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:54:39 smithi047 ceph-mon[77147]: pgmap v10157: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:54:40.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:54:39 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:54:41.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:54:40 smithi115 conmon[90393]: 2022-05-14T04:54:40.713+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:54:41.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:54:40 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:54:41.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:54:40 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:54:42.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:54:41 smithi115 conmon[90393]: 2022-05-14T04:54:41.753+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:54:42.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:54:41 smithi115 ceph-mon[78593]: pgmap v10158: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:54:42.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:54:41 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:54:42.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:54:41 smithi047 ceph-mon[77147]: pgmap v10158: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:54:42.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:54:41 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:54:43.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:54:42 smithi115 conmon[90393]: 2022-05-14T04:54:42.770+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:54:43.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:54:42 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:54:43.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:54:42 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:54:43.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:54:42 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:54:43.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:54:42 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 19615 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:54:43.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:54:42 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:54:43.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:54:42 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:54:43.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:54:42 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:54:43.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:54:42 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 19615 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:54:44.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:54:43 smithi115 conmon[90393]: 2022-05-14T04:54:43.786+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:54:44.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:54:43 smithi115 ceph-mon[78593]: pgmap v10159: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:54:44.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:54:43 smithi047 ceph-mon[77147]: pgmap v10159: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:54:45.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:54:44 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:54:45.110 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:54:44 smithi115 conmon[90393]: 2022-05-14T04:54:44.827+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:54:45.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:54:44 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:54:46.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:54:45 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:54:46.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:54:45 smithi115 ceph-mon[78593]: pgmap v10160: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:54:46.111 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:54:45 smithi115 conmon[90393]: 2022-05-14T04:54:45.816+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:54:46.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:54:45 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:54:46.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:54:45 smithi047 ceph-mon[77147]: pgmap v10160: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:54:47.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:54:46 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:54:47.110 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:54:46 smithi115 conmon[90393]: 2022-05-14T04:54:46.788+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:54:47.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:54:46 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:54:48.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:54:47 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:54:48.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:54:47 smithi115 ceph-mon[78593]: pgmap v10161: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:54:48.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:54:47 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:54:48.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:54:47 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 19620 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:54:48.111 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:54:47 smithi115 conmon[90393]: 2022-05-14T04:54:47.763+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:54:48.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:54:47 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:54:48.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:54:47 smithi047 ceph-mon[77147]: pgmap v10161: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:54:48.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:54:47 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:54:48.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:54:47 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 19620 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:54:49.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:54:48 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:54:49.110 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:54:48 smithi115 conmon[90393]: 2022-05-14T04:54:48.749+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:54:49.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:54:48 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:54:50.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:54:49 smithi115 ceph-mon[78593]: pgmap v10162: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:54:50.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:54:49 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:54:50.110 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:54:49 smithi115 conmon[90393]: 2022-05-14T04:54:49.767+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:54:50.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:54:49 smithi047 ceph-mon[77147]: pgmap v10162: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:54:50.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:54:49 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:54:51.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:54:50 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:54:51.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:54:50 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:54:51.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:54:50 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:54:51.111 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:54:50 smithi115 conmon[90393]: 2022-05-14T04:54:50.720+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:54:51.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:54:50 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:54:51.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:54:50 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:54:51.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:54:50 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:54:52.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:54:51 smithi115 ceph-mon[78593]: pgmap v10163: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:54:52.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:54:51 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:54:52.110 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:54:51 smithi115 conmon[90393]: 2022-05-14T04:54:51.720+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:54:52.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:54:51 smithi047 ceph-mon[77147]: pgmap v10163: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:54:52.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:54:51 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:54:53.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:54:52 smithi115 conmon[90393]: 2022-05-14T04:54:52.691+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:54:53.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:54:52 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:54:53.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:54:52 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 19625 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:54:53.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:54:52 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:54:53.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:54:52 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 19625 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:54:54.110 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:54:53 smithi115 conmon[90393]: 2022-05-14T04:54:53.697+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:54:54.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:54:53 smithi115 ceph-mon[78593]: pgmap v10164: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:54:54.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:54:53 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:54:54.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:54:53 smithi047 ceph-mon[77147]: pgmap v10164: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:54:54.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:54:53 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:54:55.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:54:54 smithi115 conmon[90393]: 2022-05-14T04:54:54.668+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:54:55.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:54:54 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:54:55.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:54:54 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:54:55.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:54:54 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:54:55.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:54:54 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:54:55.296 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:54:54 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:54:55.296 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:54:54 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:54:56.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:54:55 smithi115 conmon[90393]: 2022-05-14T04:54:55.715+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:54:56.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:54:55 smithi115 ceph-mon[78593]: pgmap v10165: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:54:56.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:54:55 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:54:56.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:54:55 smithi047 ceph-mon[77147]: pgmap v10165: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:54:56.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:54:55 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:54:57.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:54:56 smithi115 conmon[90393]: 2022-05-14T04:54:56.740+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:54:57.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:54:56 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:54:57.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:54:56 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:54:58.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:54:57 smithi115 conmon[90393]: 2022-05-14T04:54:57.761+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:54:58.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:54:57 smithi115 ceph-mon[78593]: pgmap v10166: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:54:58.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:54:57 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:54:58.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:54:57 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 19630 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:54:58.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:54:57 smithi047 ceph-mon[77147]: pgmap v10166: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:54:58.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:54:57 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:54:58.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:54:57 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 19630 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:54:59.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:54:58 smithi115 conmon[90393]: 2022-05-14T04:54:58.739+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:54:59.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:54:58 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:54:59.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:54:58 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:54:59.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:54:58 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:54:59.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:54:58 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:54:59.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:54:58 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:54:59.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:54:58 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:55:00.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:54:59 smithi115 conmon[90393]: 2022-05-14T04:54:59.740+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:55:00.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:54:59 smithi115 ceph-mon[78593]: pgmap v10167: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:55:00.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:54:59 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:55:00.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:54:59 smithi047 ceph-mon[77147]: pgmap v10167: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:55:00.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:54:59 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:55:01.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:55:00 smithi115 conmon[90393]: 2022-05-14T04:55:00.697+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:55:01.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:55:00 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:55:01.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:55:00 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:55:02.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:55:01 smithi115 conmon[90393]: 2022-05-14T04:55:01.704+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:55:02.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:55:01 smithi115 ceph-mon[78593]: pgmap v10168: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:55:02.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:55:01 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:55:02.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:55:01 smithi047 ceph-mon[77147]: pgmap v10168: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:55:02.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:55:01 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:55:02.940 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:55:02 smithi115 conmon[90393]: 2022-05-14T04:55:02.660+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:55:03.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:55:02 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:55:03.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:55:02 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:55:03.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:55:02 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:55:03.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:55:02 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 19635 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:55:03.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:55:02 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:55:03.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:55:02 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:55:03.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:55:02 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:55:03.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:55:02 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 19635 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:55:03.944 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:55:03 smithi115 conmon[90393]: 2022-05-14T04:55:03.669+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:55:04.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:55:03 smithi047 ceph-mon[77147]: pgmap v10169: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:55:04.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:55:03 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:55:04.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:55:03 smithi115 ceph-mon[78593]: pgmap v10169: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:55:04.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:55:03 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:55:04.946 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:55:04 smithi115 conmon[90393]: 2022-05-14T04:55:04.676+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:55:05.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:55:04 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:55:05.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:55:04 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:55:06.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:55:05 smithi115 conmon[90393]: 2022-05-14T04:55:05.699+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:55:06.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:55:05 smithi115 ceph-mon[78593]: pgmap v10170: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:55:06.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:55:05 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:55:06.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:55:05 smithi047 ceph-mon[77147]: pgmap v10170: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:55:06.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:55:05 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:55:07.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:55:06 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:55:07.110 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:55:06 smithi115 conmon[90393]: 2022-05-14T04:55:06.735+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:55:07.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:55:06 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:55:08.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:55:07 smithi115 conmon[90393]: 2022-05-14T04:55:07.736+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:55:08.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:55:07 smithi115 ceph-mon[78593]: pgmap v10171: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:55:08.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:55:07 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:55:08.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:55:07 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 19640 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:55:08.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:55:07 smithi047 ceph-mon[77147]: pgmap v10171: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:55:08.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:55:07 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:55:08.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:55:07 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 19640 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:55:09.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:55:08 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:55:09.110 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:55:08 smithi115 conmon[90393]: 2022-05-14T04:55:08.709+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:55:09.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:55:08 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:55:09.969 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:55:09 smithi115 conmon[90393]: 2022-05-14T04:55:09.676+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:55:10.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:55:09 smithi047 ceph-mon[77147]: pgmap v10172: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:55:10.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:55:09 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:55:10.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:55:09 smithi115 ceph-mon[78593]: pgmap v10172: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:55:10.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:55:09 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:55:10.968 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:55:10 smithi115 conmon[90393]: 2022-05-14T04:55:10.639+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:55:11.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:55:10 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:55:11.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:55:10 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:55:11.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:55:10 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:55:11.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:55:10 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:55:11.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:55:10 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:55:11.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:55:10 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:55:11.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:55:11 smithi115 conmon[90393]: 2022-05-14T04:55:11.595+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:55:12.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:55:11 smithi047 ceph-mon[77147]: pgmap v10173: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:55:12.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:55:11 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T04:55:12.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:55:11 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T04:55:12.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:55:11 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T04:55:12.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:55:11 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:55:12.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:55:12 smithi047 conmon[78521]: 2022-05-14T04:55:12.176+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:55:12.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:55:12 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T04:55:12.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:55:12 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:55:12.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:55:12 smithi047 conmon[78521]: yield 2022-05-14T04:55:12.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:55:12 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:55:12.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:55:12 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:55:12.297 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:55:12 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:55:12.297 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:55:12 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T04:55:12.297 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:55:12 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:55:12.297 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:55:12 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T04:55:12.298 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:55:12 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:55:12.298 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:55:12 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T04:55:12.298 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:55:12 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:55:12.298 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:55:12 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T04:55:12.298 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:55:12 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:55:12.299 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:55:12 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T04:55:12.299 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:55:12 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:55:12.299 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:55:12 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:55:12.299 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:55:12 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:55:12.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:55:11 smithi115 ceph-mon[78593]: pgmap v10173: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:55:12.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:55:11 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T04:55:12.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:55:11 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T04:55:12.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:55:11 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T04:55:12.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:55:11 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:55:12.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:55:12 smithi115 conmon[90393]: 2022-05-14T04:55:12.593+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:55:13.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:55:12 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:55:13.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T04:55:13.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:55:13.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T04:55:13.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:55:13.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:55:13.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:55:13.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T04:55:13.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:55:13.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T04:55:13.298 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:55:13.298 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T04:55:13.299 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:55:13.299 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T04:55:13.299 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:55:13.300 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T04:55:13.300 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:55:13.301 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:55:13.302 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:55:13.302 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:55:12 smithi047 ceph-mon[77147]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T04:55:13.302 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:55:12 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:55:13.303 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:55:12 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 19645 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:55:13.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:55:12 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:55:13.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T04:55:13.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:55:13.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T04:55:13.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:55:13.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:55:13.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:55:13.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T04:55:13.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:55:13.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T04:55:13.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:55:13.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T04:55:13.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:55:13.365 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T04:55:13.365 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:55:13.366 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T04:55:13.366 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:55:13.367 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:55:13.367 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:55:13.368 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:55:12 smithi115 ceph-mon[78593]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T04:55:13.368 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:55:12 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:55:13.369 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:55:12 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 19645 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:55:13.978 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:55:13 smithi115 conmon[90393]: 2022-05-14T04:55:13.636+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:55:14.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:55:13 smithi047 ceph-mon[77147]: pgmap v10174: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:55:14.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:55:13 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:55:14.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:55:13 smithi115 ceph-mon[78593]: pgmap v10174: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:55:14.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:55:13 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:55:14.864 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:55:14 smithi115 conmon[90393]: 2022-05-14T04:55:14.603+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:55:15.253 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:55:14 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:55:15.253 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:55:14 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:55:15.254 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:55:14 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:55:15.254 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:55:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T04:55:15.255 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:55:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T04:55:15.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:55:14 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:55:15.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:55:14 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:55:15.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:55:14 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:55:15.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:55:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T04:55:15.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:55:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T04:55:15.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:55:15 smithi047 conmon[78521]: 2022-05-14T04:55:15.248+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:55:15.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:55:15 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T04:55:15.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:55:15 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:55:15.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:55:15 smithi047 conmon[78521]: yield 2022-05-14T04:55:15.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:55:15 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:55:15.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:55:15 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:55:15.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:55:15 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:55:15.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:55:15 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T04:55:15.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:55:15 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:55:15.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:55:15 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T04:55:15.547 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:55:15 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:55:15.548 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:55:15 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T04:55:15.548 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:55:15 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:55:15.548 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:55:15 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T04:55:15.548 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:55:15 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:55:15.548 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:55:15 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T04:55:15.549 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:55:15 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:55:15.549 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:55:15 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:55:15.549 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:55:15 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:55:15.549 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:55:15 smithi047 conmon[78521]: 2022-05-14T04:55:15.249+0000 7f5340f7a700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T04:55:15.550 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:55:15 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T04:55:15.550 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:55:15 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:55:15.550 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:55:15 smithi047 conmon[78521]: yield 2022-05-14T04:55:15.550 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:55:15 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:55:15.550 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:55:15 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:55:15.551 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:55:15 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:55:15.551 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:55:15 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T04:55:15.551 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:55:15 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:55:15.551 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:55:15 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T04:55:15.551 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:55:15 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:55:15.552 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:55:15 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T04:55:15.552 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:55:15 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:55:15.552 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:55:15 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T04:55:15.552 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:55:15 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:55:15.552 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:55:15 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T04:55:15.553 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:55:15 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:55:15.553 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:55:15 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:55:15.553 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:55:15 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:55:15.553 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:55:15 smithi047 conmon[78521]: 2022-05-14T04:55:15.554 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:55:15 smithi047 conmon[78521]: During handling of the above exception, another exception occurred: 2022-05-14T04:55:15.554 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:55:15 smithi047 conmon[78521]: 2022-05-14T04:55:15.554 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:55:15 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T04:55:15.555 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:55:15 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T04:55:15.555 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:55:15 smithi047 conmon[78521]: return f(*arg) 2022-05-14T04:55:15.555 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:55:15 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T04:55:15.555 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:55:15 smithi047 conmon[78521]: self._write_client_files(client_files, host) 2022-05-14T04:55:15.556 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:55:15 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T04:55:15.556 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:55:15 smithi047 conmon[78521]: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T04:55:15.556 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:55:15 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T04:55:15.556 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:55:15 smithi047 conmon[78521]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T04:55:15.556 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:55:15 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T04:55:15.557 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:55:15 smithi047 conmon[78521]: return self.event_loop.get_result(coro) 2022-05-14T04:55:15.557 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:55:15 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T04:55:15.557 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:55:15 smithi047 conmon[78521]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T04:55:15.557 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:55:15 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T04:55:15.557 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:55:15 smithi047 conmon[78521]: return self.__get_result() 2022-05-14T04:55:15.558 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:55:15 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T04:55:15.558 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:55:15 smithi047 conmon[78521]: raise self._exception 2022-05-14T04:55:15.558 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:55:15 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T04:55:15.558 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:55:15 smithi047 conmon[78521]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T04:55:15.559 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:55:15 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T04:55:15.559 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:55:15 smithi047 conmon[78521]: conn = await self._remote_connection(host, addr) 2022-05-14T04:55:15.559 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:55:15 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T04:55:15.559 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:55:15 smithi047 conmon[78521]: raise 2022-05-14T04:55:15.559 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:55:15 smithi047 conmon[78521]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T04:55:15.560 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:55:15 smithi047 conmon[78521]: self.gen.throw(type, value, traceback) 2022-05-14T04:55:15.560 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:55:15 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T04:55:15.560 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:55:15 smithi047 conmon[78521]: raise OrchestratorError(msg) 2022-05-14T04:55:15.560 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:55:15 smithi047 conmon[78521]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:55:15.860 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:55:15 smithi115 conmon[90393]: 2022-05-14T04:55:15.595+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:55:16.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:55:15 smithi047 ceph-mon[77147]: pgmap v10175: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:55:16.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:55:15 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:55:16.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T04:55:16.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:55:16.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T04:55:16.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:55:16.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:55:16.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:55:16.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T04:55:16.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:55:16.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T04:55:16.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:55:16.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T04:55:16.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:55:16.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T04:55:16.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:55:16.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T04:55:16.298 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:55:16.298 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:55:16.299 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:55:16.299 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:55:15 smithi047 ceph-mon[77147]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T04:55:16.299 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T04:55:16.300 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:55:16.300 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T04:55:16.300 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:55:16.301 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:55:16.301 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:55:16.302 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T04:55:16.302 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:55:16.302 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T04:55:16.303 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:55:16.304 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T04:55:16.304 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:55:16.304 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T04:55:16.305 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:55:16.305 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T04:55:16.306 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:55:16.306 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:55:16.307 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:55:16.307 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T04:55:16.307 INFO:journalctl@ceph.mon.b.smithi047.stdout: During handling of the above exception, another exception occurred: 2022-05-14T04:55:16.308 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T04:55:16.308 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T04:55:16.308 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T04:55:16.309 INFO:journalctl@ceph.mon.b.smithi047.stdout: return f(*arg) 2022-05-14T04:55:16.309 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T04:55:16.310 INFO:journalctl@ceph.mon.b.smithi047.stdout: self._write_client_files(client_files, host) 2022-05-14T04:55:16.310 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T04:55:16.310 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T04:55:16.310 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T04:55:16.311 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T04:55:16.311 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T04:55:16.311 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.event_loop.get_result(coro) 2022-05-14T04:55:16.311 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T04:55:16.312 INFO:journalctl@ceph.mon.b.smithi047.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T04:55:16.312 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T04:55:16.312 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.__get_result() 2022-05-14T04:55:16.312 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T04:55:16.312 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise self._exception 2022-05-14T04:55:16.313 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T04:55:16.313 INFO:journalctl@ceph.mon.b.smithi047.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T04:55:16.313 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T04:55:16.313 INFO:journalctl@ceph.mon.b.smithi047.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T04:55:16.314 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T04:55:16.314 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise 2022-05-14T04:55:16.314 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T04:55:16.314 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.gen.throw(type, value, traceback) 2022-05-14T04:55:16.315 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T04:55:16.315 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OrchestratorError(msg) 2022-05-14T04:55:16.315 INFO:journalctl@ceph.mon.b.smithi047.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:55:16.315 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:55:15 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:55:16.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:55:15 smithi115 ceph-mon[78593]: pgmap v10175: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:55:16.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:55:15 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:55:16.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T04:55:16.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:55:16.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T04:55:16.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:55:16.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:55:16.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:55:16.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T04:55:16.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:55:16.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T04:55:16.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:55:16.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T04:55:16.365 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:55:16.365 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T04:55:16.365 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:55:16.365 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T04:55:16.366 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:55:16.366 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:55:16.366 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:55:16.366 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:55:15 smithi115 ceph-mon[78593]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T04:55:16.366 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T04:55:16.367 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:55:16.367 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T04:55:16.367 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:55:16.368 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:55:16.369 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:55:16.369 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T04:55:16.369 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:55:16.369 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T04:55:16.370 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:55:16.370 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T04:55:16.370 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:55:16.370 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T04:55:16.370 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:55:16.371 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T04:55:16.371 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:55:16.371 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:55:16.371 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:55:16.372 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T04:55:16.372 INFO:journalctl@ceph.mon.c.smithi115.stdout: During handling of the above exception, another exception occurred: 2022-05-14T04:55:16.372 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T04:55:16.372 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T04:55:16.372 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T04:55:16.373 INFO:journalctl@ceph.mon.c.smithi115.stdout: return f(*arg) 2022-05-14T04:55:16.373 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T04:55:16.373 INFO:journalctl@ceph.mon.c.smithi115.stdout: self._write_client_files(client_files, host) 2022-05-14T04:55:16.373 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T04:55:16.373 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T04:55:16.374 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T04:55:16.374 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T04:55:16.374 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T04:55:16.374 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.event_loop.get_result(coro) 2022-05-14T04:55:16.375 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T04:55:16.375 INFO:journalctl@ceph.mon.c.smithi115.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T04:55:16.375 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T04:55:16.375 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.__get_result() 2022-05-14T04:55:16.375 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T04:55:16.376 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise self._exception 2022-05-14T04:55:16.376 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T04:55:16.376 INFO:journalctl@ceph.mon.c.smithi115.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T04:55:16.377 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T04:55:16.377 INFO:journalctl@ceph.mon.c.smithi115.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T04:55:16.377 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T04:55:16.378 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise 2022-05-14T04:55:16.378 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T04:55:16.378 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.gen.throw(type, value, traceback) 2022-05-14T04:55:16.379 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T04:55:16.379 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OrchestratorError(msg) 2022-05-14T04:55:16.379 INFO:journalctl@ceph.mon.c.smithi115.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:55:16.379 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:55:15 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:55:16.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:55:16 smithi115 conmon[90393]: 2022-05-14T04:55:16.565+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:55:17.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:55:16 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:55:17.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:55:16 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:55:17.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:55:17 smithi115 conmon[90393]: 2022-05-14T04:55:17.521+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:55:18.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:55:17 smithi047 ceph-mon[77147]: pgmap v10176: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:55:18.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:55:17 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:55:18.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:55:17 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 19650 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:55:18.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:55:17 smithi115 ceph-mon[78593]: pgmap v10176: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:55:18.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:55:17 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:55:18.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:55:17 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 19650 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:55:18.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:55:18 smithi115 conmon[90393]: 2022-05-14T04:55:18.494+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:55:19.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:55:18 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:55:19.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:55:18 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:55:19.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:55:18 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:55:19.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:55:18 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:55:19.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:55:18 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:55:19.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:55:18 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:55:19.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:55:19 smithi115 conmon[90393]: 2022-05-14T04:55:19.479+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:55:20.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:55:19 smithi047 ceph-mon[77147]: pgmap v10177: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 6 op/s 2022-05-14T04:55:20.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:55:19 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:55:20.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:55:19 smithi115 ceph-mon[78593]: pgmap v10177: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 6 op/s 2022-05-14T04:55:20.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:55:19 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:55:20.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:55:20 smithi115 conmon[90393]: 2022-05-14T04:55:20.430+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:55:21.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:55:20 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:55:21.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:55:20 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:55:21.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:55:21 smithi115 conmon[90393]: 2022-05-14T04:55:21.464+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:55:22.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:55:22 smithi047 ceph-mon[77147]: pgmap v10178: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 4 op/s 2022-05-14T04:55:22.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:55:22 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:55:22.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:55:22 smithi115 ceph-mon[78593]: pgmap v10178: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 4 op/s 2022-05-14T04:55:22.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:55:22 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:55:22.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:55:22 smithi115 conmon[90393]: 2022-05-14T04:55:22.433+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:55:23.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:55:23 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:55:23.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:55:23 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:55:23.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:55:23 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:55:23.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:55:23 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 19655 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:55:23.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:55:23 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:55:23.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:55:23 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:55:23.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:55:23 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:55:23.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:55:23 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 19655 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:55:23.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:55:23 smithi115 conmon[90393]: 2022-05-14T04:55:23.475+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:55:24.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:55:24 smithi047 ceph-mon[77147]: pgmap v10179: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 4 op/s 2022-05-14T04:55:24.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:55:24 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:55:24.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:55:24 smithi115 ceph-mon[78593]: pgmap v10179: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 4 op/s 2022-05-14T04:55:24.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:55:24 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:55:24.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:55:24 smithi115 conmon[90393]: 2022-05-14T04:55:24.441+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:55:25.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:55:25 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:55:25.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:55:25 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:55:25.860 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:55:25 smithi115 conmon[90393]: 2022-05-14T04:55:25.435+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:55:26.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:55:26 smithi047 ceph-mon[77147]: pgmap v10180: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 7 op/s 2022-05-14T04:55:26.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:55:26 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:55:26.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:55:26 smithi115 ceph-mon[78593]: pgmap v10180: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 7 op/s 2022-05-14T04:55:26.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:55:26 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:55:26.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:55:26 smithi115 conmon[90393]: 2022-05-14T04:55:26.444+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:55:27.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:55:27 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:55:27.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:55:27 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:55:27.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:55:27 smithi115 conmon[90393]: 2022-05-14T04:55:27.460+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:55:28.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:55:28 smithi047 ceph-mon[77147]: pgmap v10181: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 4 op/s 2022-05-14T04:55:28.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:55:28 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:55:28.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:55:28 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 19660 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:55:28.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:55:28 smithi115 ceph-mon[78593]: pgmap v10181: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 4 op/s 2022-05-14T04:55:28.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:55:28 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:55:28.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:55:28 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 19660 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:55:28.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:55:28 smithi115 conmon[90393]: 2022-05-14T04:55:28.477+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:55:29.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:55:29 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:55:29.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:55:29 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:55:29.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:55:29 smithi115 conmon[90393]: 2022-05-14T04:55:29.441+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:55:30.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:55:30 smithi047 ceph-mon[77147]: pgmap v10182: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 7 op/s 2022-05-14T04:55:30.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:55:30 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:55:30.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:55:30 smithi115 ceph-mon[78593]: pgmap v10182: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 7 op/s 2022-05-14T04:55:30.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:55:30 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:55:30.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:55:30 smithi115 conmon[90393]: 2022-05-14T04:55:30.489+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:55:31.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:55:31 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:55:31.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:55:31 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:55:31.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:55:31 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:55:31.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:55:31 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:55:31.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:55:31 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:55:31.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:55:31 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:55:31.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:55:31 smithi115 conmon[90393]: 2022-05-14T04:55:31.517+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:55:32.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:55:32 smithi047 ceph-mon[77147]: pgmap v10183: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:55:32.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:55:32 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:55:32.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:55:32 smithi115 ceph-mon[78593]: pgmap v10183: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:55:32.362 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:55:32 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:55:32.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:55:32 smithi115 conmon[90393]: 2022-05-14T04:55:32.469+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:55:33.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:55:33 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:55:33.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:55:33 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 19665 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:55:33.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:55:33 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:55:33.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:55:33 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 19665 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:55:33.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:55:33 smithi115 conmon[90393]: 2022-05-14T04:55:33.493+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:55:34.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:55:34 smithi115 ceph-mon[78593]: pgmap v10184: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:55:34.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:55:34 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:55:34.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:55:34 smithi047 ceph-mon[77147]: pgmap v10184: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:55:34.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:55:34 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:55:34.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:55:34 smithi115 conmon[90393]: 2022-05-14T04:55:34.455+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:55:35.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:55:35 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:55:35.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:55:35 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:55:35.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:55:35 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:55:35.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:55:35 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:55:35.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:55:35 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:55:35.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:55:35 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:55:35.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:55:35 smithi115 conmon[90393]: 2022-05-14T04:55:35.425+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:55:36.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:55:36 smithi115 ceph-mon[78593]: pgmap v10185: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:55:36.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:55:36 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:55:36.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:55:36 smithi047 ceph-mon[77147]: pgmap v10185: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:55:36.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:55:36 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:55:36.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:55:36 smithi115 conmon[90393]: 2022-05-14T04:55:36.424+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:55:37.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:55:37 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:55:37.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:55:37 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:55:37.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:55:37 smithi115 conmon[90393]: 2022-05-14T04:55:37.407+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:55:38.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:55:38 smithi115 ceph-mon[78593]: pgmap v10186: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:55:38.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:55:38 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:55:38.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:55:38 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 19670 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:55:38.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:55:38 smithi047 ceph-mon[77147]: pgmap v10186: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:55:38.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:55:38 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:55:38.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:55:38 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 19670 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:55:38.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:55:38 smithi115 conmon[90393]: 2022-05-14T04:55:38.446+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:55:39.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:55:39 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:55:39.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:55:39 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:55:39.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:55:39 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:55:39.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:55:39 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:55:39.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:55:39 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:55:39.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:55:39 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:55:39.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:55:39 smithi115 conmon[90393]: 2022-05-14T04:55:39.441+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:55:40.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:55:40 smithi115 ceph-mon[78593]: pgmap v10187: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:55:40.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:55:40 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:55:40.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:55:40 smithi047 ceph-mon[77147]: pgmap v10187: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:55:40.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:55:40 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:55:40.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:55:40 smithi115 conmon[90393]: 2022-05-14T04:55:40.421+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:55:41.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:55:41 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:55:41.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:55:41 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:55:41.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:55:41 smithi115 conmon[90393]: 2022-05-14T04:55:41.460+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:55:42.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:55:42 smithi115 ceph-mon[78593]: pgmap v10188: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:55:42.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:55:42 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:55:42.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:55:42 smithi047 ceph-mon[77147]: pgmap v10188: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:55:42.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:55:42 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:55:42.860 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:55:42 smithi115 conmon[90393]: 2022-05-14T04:55:42.437+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:55:43.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:55:43 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:55:43.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:55:43 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:55:43.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:55:43 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:55:43.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:55:43 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 19675 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:55:43.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:55:43 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:55:43.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:55:43 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:55:43.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:55:43 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:55:43.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:55:43 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 19675 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:55:43.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:55:43 smithi115 conmon[90393]: 2022-05-14T04:55:43.435+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:55:44.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:55:44 smithi115 ceph-mon[78593]: pgmap v10189: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:55:44.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:55:44 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:55:44.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:55:44 smithi047 ceph-mon[77147]: pgmap v10189: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:55:44.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:55:44 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:55:44.860 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:55:44 smithi115 conmon[90393]: 2022-05-14T04:55:44.426+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:55:45.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:55:45 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:55:45.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:55:45 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:55:45.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:55:45 smithi115 conmon[90393]: 2022-05-14T04:55:45.441+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:55:46.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:55:46 smithi115 ceph-mon[78593]: pgmap v10190: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:55:46.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:55:46 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:55:46.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:55:46 smithi047 ceph-mon[77147]: pgmap v10190: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:55:46.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:55:46 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:55:46.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:55:46 smithi115 conmon[90393]: 2022-05-14T04:55:46.433+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:55:47.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:55:47 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:55:47.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:55:47 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:55:47.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:55:47 smithi115 conmon[90393]: 2022-05-14T04:55:47.458+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:55:48.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:55:48 smithi115 ceph-mon[78593]: pgmap v10191: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:55:48.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:55:48 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:55:48.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:55:48 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 19680 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:55:48.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:55:48 smithi047 ceph-mon[77147]: pgmap v10191: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:55:48.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:55:48 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:55:48.545 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:55:48 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 19680 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:55:48.860 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:55:48 smithi115 conmon[90393]: 2022-05-14T04:55:48.453+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:55:49.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:55:49 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:55:49.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:55:49 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:55:49.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:55:49 smithi115 conmon[90393]: 2022-05-14T04:55:49.405+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:55:50.454 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:55:50 smithi115 ceph-mon[78593]: pgmap v10192: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:55:50.454 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:55:50 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:55:50.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:55:50 smithi047 ceph-mon[77147]: pgmap v10192: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:55:50.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:55:50 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:55:50.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:55:50 smithi115 conmon[90393]: 2022-05-14T04:55:50.451+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:55:51.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:55:51 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:55:51.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:55:51 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:55:51.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:55:51 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:55:51.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:55:51 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:55:51.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:55:51 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:55:51.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:55:51 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:55:51.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:55:51 smithi115 conmon[90393]: 2022-05-14T04:55:51.459+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:55:52.470 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:55:52 smithi115 ceph-mon[78593]: pgmap v10193: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:55:52.470 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:55:52 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:55:52.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:55:52 smithi047 ceph-mon[77147]: pgmap v10193: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:55:52.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:55:52 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:55:52.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:55:52 smithi115 conmon[90393]: 2022-05-14T04:55:52.466+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:55:53.495 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:55:53 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:55:53.495 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:55:53 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 19685 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:55:53.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:55:53 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:55:53.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:55:53 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 19685 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:55:53.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:55:53 smithi115 conmon[90393]: 2022-05-14T04:55:53.491+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:55:54.487 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:55:54 smithi115 ceph-mon[78593]: pgmap v10194: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:55:54.487 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:55:54 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:55:54.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:55:54 smithi047 ceph-mon[77147]: pgmap v10194: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:55:54.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:55:54 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:55:54.860 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:55:54 smithi115 conmon[90393]: 2022-05-14T04:55:54.483+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:55:55.441 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:55:55 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:55:55.441 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:55:55 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:55:55.441 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:55:55 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:55:55.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:55:55 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:55:55.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:55:55 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:55:55.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:55:55 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:55:55.860 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:55:55 smithi115 conmon[90393]: 2022-05-14T04:55:55.437+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:55:56.428 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:55:56 smithi115 ceph-mon[78593]: pgmap v10195: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:55:56.428 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:55:56 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:55:56.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:55:56 smithi047 ceph-mon[77147]: pgmap v10195: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:55:56.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:55:56 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:55:56.860 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:55:56 smithi115 conmon[90393]: 2022-05-14T04:55:56.424+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:55:57.469 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:55:57 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:55:57.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:55:57 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:55:57.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:55:57 smithi115 conmon[90393]: 2022-05-14T04:55:57.465+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:55:58.444 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:55:58 smithi115 ceph-mon[78593]: pgmap v10196: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:55:58.445 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:55:58 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:55:58.445 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:55:58 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 19690 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:55:58.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:55:58 smithi047 ceph-mon[77147]: pgmap v10196: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:55:58.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:55:58 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:55:58.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:55:58 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 19690 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:55:58.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:55:58 smithi115 conmon[90393]: 2022-05-14T04:55:58.441+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:55:59.418 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:55:59 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:55:59.418 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:55:59 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:55:59.418 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:55:59 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:55:59.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:55:59 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:55:59.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:55:59 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:55:59.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:55:59 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:55:59.871 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:55:59 smithi115 conmon[90393]: 2022-05-14T04:55:59.415+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:56:00.431 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:56:00 smithi115 ceph-mon[78593]: pgmap v10197: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:56:00.431 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:56:00 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:56:00.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:56:00 smithi047 ceph-mon[77147]: pgmap v10197: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:56:00.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:56:00 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:56:00.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:56:00 smithi115 conmon[90393]: 2022-05-14T04:56:00.428+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:56:01.457 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:56:01 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:56:01.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:56:01 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:56:01.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:56:01 smithi115 conmon[90393]: 2022-05-14T04:56:01.454+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:56:02.483 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:56:02 smithi115 ceph-mon[78593]: pgmap v10198: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:56:02.484 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:56:02 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:56:02.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:56:02 smithi047 ceph-mon[77147]: pgmap v10198: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:56:02.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:56:02 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:56:02.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:56:02 smithi115 conmon[90393]: 2022-05-14T04:56:02.479+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:56:03.477 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:56:03 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:56:03.478 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:56:03 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:56:03.478 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:56:03 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:56:03.479 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:56:03 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 19695 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:56:03.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:56:03 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:56:03.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:56:03 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:56:03.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:56:03 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:56:03.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:56:03 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 19695 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:56:03.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:56:03 smithi115 conmon[90393]: 2022-05-14T04:56:03.473+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:56:04.464 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:56:04 smithi115 ceph-mon[78593]: pgmap v10199: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:56:04.464 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:56:04 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:56:04.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:56:04 smithi047 ceph-mon[77147]: pgmap v10199: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:56:04.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:56:04 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:56:04.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:56:04 smithi115 conmon[90393]: 2022-05-14T04:56:04.461+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:56:05.462 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:56:05 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:56:05.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:56:05 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:56:05.862 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:56:05 smithi115 conmon[90393]: 2022-05-14T04:56:05.459+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:56:06.459 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:56:06 smithi115 ceph-mon[78593]: pgmap v10200: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:56:06.459 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:56:06 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:56:06.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:56:06 smithi047 ceph-mon[77147]: pgmap v10200: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:56:06.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:56:06 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:56:06.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:56:06 smithi115 conmon[90393]: 2022-05-14T04:56:06.456+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:56:07.495 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:56:07 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:56:07.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:56:07 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:56:07.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:56:07 smithi115 conmon[90393]: 2022-05-14T04:56:07.492+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:56:08.533 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:56:08 smithi115 ceph-mon[78593]: pgmap v10201: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:56:08.534 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:56:08 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:56:08.535 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:56:08 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 19700 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:56:08.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:56:08 smithi047 ceph-mon[77147]: pgmap v10201: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:56:08.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:56:08 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:56:08.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:56:08 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 19700 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:56:08.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:56:08 smithi115 conmon[90393]: 2022-05-14T04:56:08.530+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:56:09.487 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:56:09 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:56:09.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:56:09 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:56:09.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:56:09 smithi115 conmon[90393]: 2022-05-14T04:56:09.485+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:56:10.485 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:56:10 smithi115 ceph-mon[78593]: pgmap v10202: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:56:10.485 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:56:10 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:56:10.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:56:10 smithi047 ceph-mon[77147]: pgmap v10202: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:56:10.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:56:10 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:56:10.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:56:10 smithi115 conmon[90393]: 2022-05-14T04:56:10.481+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:56:11.507 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:56:11 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:56:11.508 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:56:11 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:56:11.508 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:56:11 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:56:11.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:56:11 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:56:11.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:56:11 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:56:11.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:56:11 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:56:11.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:56:11 smithi115 conmon[90393]: 2022-05-14T04:56:11.504+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:56:12.498 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:56:12 smithi115 ceph-mon[78593]: pgmap v10203: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:56:12.498 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:56:12 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:56:12.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:56:12 smithi047 ceph-mon[77147]: pgmap v10203: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:56:12.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:56:12 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:56:12.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:56:12 smithi115 conmon[90393]: 2022-05-14T04:56:12.495+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:56:13.526 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:56:13 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:56:13.527 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:56:13 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 19705 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:56:13.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:56:13 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:56:13.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:56:13 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 19705 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:56:13.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:56:13 smithi115 conmon[90393]: 2022-05-14T04:56:13.523+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:56:14.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:56:14 smithi047 ceph-mon[77147]: pgmap v10204: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:56:14.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:56:14 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:56:14.551 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:56:14 smithi115 ceph-mon[78593]: pgmap v10204: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:56:14.551 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:56:14 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:56:14.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:56:14 smithi115 conmon[90393]: 2022-05-14T04:56:14.547+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:56:15.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:56:15 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:56:15.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:56:15 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:56:15.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:56:15 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:56:15.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:56:15 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T04:56:15.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:56:15 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T04:56:15.572 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:56:15 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:56:15.573 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:56:15 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:56:15.573 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:56:15 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:56:15.573 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:56:15 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T04:56:15.573 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:56:15 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T04:56:15.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:56:15 smithi115 conmon[90393]: 2022-05-14T04:56:15.569+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:56:16.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:56:16 smithi047 ceph-mon[77147]: pgmap v10205: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:56:16.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:56:16 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T04:56:16.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:56:16 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T04:56:16.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:56:16 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T04:56:16.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:56:16 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:56:16.583 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:56:16 smithi115 ceph-mon[78593]: pgmap v10205: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:56:16.583 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:56:16 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T04:56:16.583 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:56:16 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T04:56:16.584 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:56:16 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T04:56:16.584 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:56:16 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:56:16.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:56:16 smithi115 conmon[90393]: 2022-05-14T04:56:16.579+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:56:17.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:56:17 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:56:17.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:56:17 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:56:18.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:56:17 smithi115 conmon[90393]: 2022-05-14T04:56:17.617+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:56:18.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:56:18 smithi047 ceph-mon[77147]: pgmap v10206: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:56:18.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:56:18 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:56:18.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:56:18 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 19710 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:56:18.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:56:18 smithi047 conmon[78521]: 2022-05-14T04:56:18.351+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:56:18.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:56:18 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T04:56:18.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:56:18 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:56:18.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:56:18 smithi047 conmon[78521]: yield 2022-05-14T04:56:18.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:56:18 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:56:18.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:56:18 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:56:18.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:56:18 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:56:18.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:56:18 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T04:56:18.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:56:18 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:56:18.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:56:18 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T04:56:18.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:56:18 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:56:18.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:56:18 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T04:56:18.547 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:56:18 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:56:18.547 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:56:18 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T04:56:18.547 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:56:18 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:56:18.547 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:56:18 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T04:56:18.547 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:56:18 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:56:18.548 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:56:18 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:56:18.548 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:56:18 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:56:18.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:56:18 smithi115 ceph-mon[78593]: pgmap v10206: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:56:18.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:56:18 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:56:18.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:56:18 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 19710 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:56:19.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:56:18 smithi115 conmon[90393]: 2022-05-14T04:56:18.639+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:56:19.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:56:19 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:56:19.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:56:19 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:56:19.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:56:19 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:56:19.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T04:56:19.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:56:19.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T04:56:19.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:56:19.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:56:19.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:56:19.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T04:56:19.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:56:19.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T04:56:19.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:56:19.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T04:56:19.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:56:19.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T04:56:19.548 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:56:19.548 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T04:56:19.548 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:56:19.548 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:56:19.548 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:56:19.549 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:56:19 smithi047 ceph-mon[77147]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T04:56:19.549 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:56:19 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:56:19.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:56:19 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:56:19.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:56:19 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:56:19.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:56:19 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:56:19.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T04:56:19.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:56:19.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T04:56:19.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:56:19.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:56:19.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:56:19.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T04:56:19.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:56:19.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T04:56:19.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:56:19.613 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T04:56:19.613 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:56:19.613 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T04:56:19.613 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:56:19.614 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T04:56:19.614 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:56:19.614 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:56:19.614 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:56:19.615 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:56:19 smithi115 ceph-mon[78593]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T04:56:19.615 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:56:19 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:56:20.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:56:19 smithi115 conmon[90393]: 2022-05-14T04:56:19.644+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:56:20.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:56:20 smithi047 ceph-mon[77147]: pgmap v10207: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:56:20.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:56:20 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:56:20.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:56:20 smithi115 ceph-mon[78593]: pgmap v10207: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:56:20.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:56:20 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:56:21.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:56:20 smithi115 conmon[90393]: 2022-05-14T04:56:20.616+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:56:21.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:56:21 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:56:21.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:56:21 smithi047 conmon[78521]: 2022-05-14T04:56:21.423+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:56:21.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:56:21 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T04:56:21.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:56:21 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:56:21.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:56:21 smithi047 conmon[78521]: yield 2022-05-14T04:56:21.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:56:21 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:56:21.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:56:21 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:56:21.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:56:21 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:56:21.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:56:21 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T04:56:21.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:56:21 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:56:21.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:56:21 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T04:56:21.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:56:21 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:56:21.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:56:21 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T04:56:21.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:56:21 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:56:21.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:56:21 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T04:56:21.547 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:56:21 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:56:21.547 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:56:21 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T04:56:21.547 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:56:21 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:56:21.547 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:56:21 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:56:21.547 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:56:21 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:56:21.548 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:56:21 smithi047 conmon[78521]: 2022-05-14T04:56:21.424+0000 7f5344f82700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T04:56:21.548 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:56:21 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T04:56:21.548 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:56:21 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:56:21.548 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:56:21 smithi047 conmon[78521]: yield 2022-05-14T04:56:21.549 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:56:21 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:56:21.549 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:56:21 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:56:21.549 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:56:21 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:56:21.549 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:56:21 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T04:56:21.550 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:56:21 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:56:21.550 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:56:21 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T04:56:21.550 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:56:21 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:56:21.550 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:56:21 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T04:56:21.550 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:56:21 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:56:21.551 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:56:21 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T04:56:21.551 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:56:21 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:56:21.551 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:56:21 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T04:56:21.551 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:56:21 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:56:21.552 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:56:21 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:56:21.552 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:56:21 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:56:21.552 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:56:21 smithi047 conmon[78521]: 2022-05-14T04:56:21.552 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:56:21 smithi047 conmon[78521]: During handling of the above exception, another exception occurred: 2022-05-14T04:56:21.553 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:56:21 smithi047 conmon[78521]: 2022-05-14T04:56:21.553 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:56:21 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T04:56:21.553 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:56:21 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T04:56:21.553 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:56:21 smithi047 conmon[78521]: return f(*arg) 2022-05-14T04:56:21.554 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:56:21 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T04:56:21.554 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:56:21 smithi047 conmon[78521]: self._write_client_files(client_files, host) 2022-05-14T04:56:21.554 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:56:21 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T04:56:21.554 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:56:21 smithi047 conmon[78521]: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T04:56:21.554 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:56:21 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T04:56:21.555 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:56:21 smithi047 conmon[78521]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T04:56:21.555 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:56:21 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T04:56:21.555 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:56:21 smithi047 conmon[78521]: return self.event_loop.get_result(coro) 2022-05-14T04:56:21.555 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:56:21 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T04:56:21.556 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:56:21 smithi047 conmon[78521]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T04:56:21.556 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:56:21 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T04:56:21.556 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:56:21 smithi047 conmon[78521]: return self.__get_result() 2022-05-14T04:56:21.556 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:56:21 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T04:56:21.556 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:56:21 smithi047 conmon[78521]: raise self._exception 2022-05-14T04:56:21.557 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:56:21 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T04:56:21.557 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:56:21 smithi047 conmon[78521]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T04:56:21.557 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:56:21 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T04:56:21.557 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:56:21 smithi047 conmon[78521]: conn = await self._remote_connection(host, addr) 2022-05-14T04:56:21.558 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:56:21 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T04:56:21.558 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:56:21 smithi047 conmon[78521]: raise 2022-05-14T04:56:21.558 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:56:21 smithi047 conmon[78521]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T04:56:21.558 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:56:21 smithi047 conmon[78521]: self.gen.throw(type, value, traceback) 2022-05-14T04:56:21.559 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:56:21 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T04:56:21.559 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:56:21 smithi047 conmon[78521]: raise OrchestratorError(msg) 2022-05-14T04:56:21.559 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:56:21 smithi047 conmon[78521]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:56:21.596 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:56:21 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:56:21.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:56:21 smithi115 conmon[90393]: 2022-05-14T04:56:21.593+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:56:22.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:56:22 smithi047 ceph-mon[77147]: pgmap v10208: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:56:22.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:56:22 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:56:22.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T04:56:22.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:56:22.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T04:56:22.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:56:22.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:56:22.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:56:22.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T04:56:22.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:56:22.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T04:56:22.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:56:22.548 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T04:56:22.548 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:56:22.548 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T04:56:22.549 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:56:22.549 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T04:56:22.550 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:56:22.550 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:56:22.551 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:56:22.551 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:56:22 smithi047 ceph-mon[77147]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T04:56:22.551 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T04:56:22.552 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:56:22.552 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T04:56:22.553 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:56:22.553 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:56:22.554 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:56:22.554 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T04:56:22.554 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:56:22.555 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T04:56:22.555 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:56:22.556 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T04:56:22.556 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:56:22.557 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T04:56:22.557 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:56:22.557 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T04:56:22.557 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:56:22.558 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:56:22.558 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:56:22.558 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T04:56:22.558 INFO:journalctl@ceph.mon.b.smithi047.stdout: During handling of the above exception, another exception occurred: 2022-05-14T04:56:22.558 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T04:56:22.559 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T04:56:22.559 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T04:56:22.559 INFO:journalctl@ceph.mon.b.smithi047.stdout: return f(*arg) 2022-05-14T04:56:22.559 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T04:56:22.559 INFO:journalctl@ceph.mon.b.smithi047.stdout: self._write_client_files(client_files, host) 2022-05-14T04:56:22.560 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T04:56:22.560 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T04:56:22.560 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T04:56:22.560 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T04:56:22.560 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T04:56:22.561 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.event_loop.get_result(coro) 2022-05-14T04:56:22.561 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T04:56:22.561 INFO:journalctl@ceph.mon.b.smithi047.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T04:56:22.561 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T04:56:22.562 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.__get_result() 2022-05-14T04:56:22.562 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T04:56:22.562 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise self._exception 2022-05-14T04:56:22.562 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T04:56:22.562 INFO:journalctl@ceph.mon.b.smithi047.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T04:56:22.563 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T04:56:22.563 INFO:journalctl@ceph.mon.b.smithi047.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T04:56:22.563 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T04:56:22.563 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise 2022-05-14T04:56:22.563 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T04:56:22.564 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.gen.throw(type, value, traceback) 2022-05-14T04:56:22.564 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T04:56:22.564 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OrchestratorError(msg) 2022-05-14T04:56:22.564 INFO:journalctl@ceph.mon.b.smithi047.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:56:22.565 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:56:22 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:56:22.605 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:56:22 smithi115 ceph-mon[78593]: pgmap v10208: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:56:22.605 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:56:22 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:56:22.606 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T04:56:22.606 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:56:22.606 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T04:56:22.607 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:56:22.607 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:56:22.607 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:56:22.607 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T04:56:22.608 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:56:22.608 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T04:56:22.608 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:56:22.608 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T04:56:22.609 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:56:22.609 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T04:56:22.609 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:56:22.609 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T04:56:22.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:56:22.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:56:22.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:56:22.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:56:22 smithi115 ceph-mon[78593]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T04:56:22.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T04:56:22.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:56:22.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T04:56:22.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:56:22.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:56:22.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:56:22.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T04:56:22.613 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:56:22.613 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T04:56:22.613 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:56:22.613 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T04:56:22.613 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:56:22.614 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T04:56:22.614 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:56:22.614 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T04:56:22.614 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:56:22.615 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:56:22.615 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:56:22.615 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T04:56:22.615 INFO:journalctl@ceph.mon.c.smithi115.stdout: During handling of the above exception, another exception occurred: 2022-05-14T04:56:22.615 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T04:56:22.616 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T04:56:22.616 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T04:56:22.616 INFO:journalctl@ceph.mon.c.smithi115.stdout: return f(*arg) 2022-05-14T04:56:22.616 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T04:56:22.617 INFO:journalctl@ceph.mon.c.smithi115.stdout: self._write_client_files(client_files, host) 2022-05-14T04:56:22.617 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T04:56:22.617 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T04:56:22.617 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T04:56:22.617 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T04:56:22.618 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T04:56:22.618 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.event_loop.get_result(coro) 2022-05-14T04:56:22.618 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T04:56:22.618 INFO:journalctl@ceph.mon.c.smithi115.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T04:56:22.619 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T04:56:22.619 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.__get_result() 2022-05-14T04:56:22.619 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T04:56:22.619 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise self._exception 2022-05-14T04:56:22.619 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T04:56:22.620 INFO:journalctl@ceph.mon.c.smithi115.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T04:56:22.620 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T04:56:22.620 INFO:journalctl@ceph.mon.c.smithi115.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T04:56:22.621 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T04:56:22.621 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise 2022-05-14T04:56:22.621 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T04:56:22.621 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.gen.throw(type, value, traceback) 2022-05-14T04:56:22.622 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T04:56:22.622 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OrchestratorError(msg) 2022-05-14T04:56:22.622 INFO:journalctl@ceph.mon.c.smithi115.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:56:22.622 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:56:22 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:56:22.860 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:56:22 smithi115 conmon[90393]: 2022-05-14T04:56:22.602+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:56:23.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:56:23 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:56:23.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:56:23 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:56:23.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:56:23 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:56:23.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:56:23 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 19715 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:56:23.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:56:23 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:56:23.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:56:23 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:56:23.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:56:23 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:56:23.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:56:23 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 19715 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:56:24.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:56:23 smithi115 conmon[90393]: 2022-05-14T04:56:23.611+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:56:24.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:56:24 smithi047 ceph-mon[77147]: pgmap v10209: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:56:24.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:56:24 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:56:24.587 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:56:24 smithi115 ceph-mon[78593]: pgmap v10209: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:56:24.588 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:56:24 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:56:24.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:56:24 smithi115 conmon[90393]: 2022-05-14T04:56:24.585+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:56:25.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:56:25 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:56:25.543 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:56:25 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:56:25.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:56:25 smithi115 conmon[90393]: 2022-05-14T04:56:25.541+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:56:26.502 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:56:26 smithi115 ceph-mon[78593]: pgmap v10210: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:56:26.502 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:56:26 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:56:26.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:56:26 smithi047 ceph-mon[77147]: pgmap v10210: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:56:26.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:56:26 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:56:26.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:56:26 smithi115 conmon[90393]: 2022-05-14T04:56:26.499+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:56:27.540 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:56:27 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:56:27.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:56:27 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:56:27.860 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:56:27 smithi115 conmon[90393]: 2022-05-14T04:56:27.537+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:56:28.534 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:56:28 smithi115 ceph-mon[78593]: pgmap v10211: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:56:28.534 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:56:28 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:56:28.535 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:56:28 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 19720 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:56:28.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:56:28 smithi047 ceph-mon[77147]: pgmap v10211: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:56:28.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:56:28 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:56:28.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:56:28 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 19720 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:56:28.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:56:28 smithi115 conmon[90393]: 2022-05-14T04:56:28.529+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:56:29.511 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:56:29 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:56:29.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:56:29 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:56:29.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:56:29 smithi115 conmon[90393]: 2022-05-14T04:56:29.508+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:56:30.507 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:56:30 smithi115 ceph-mon[78593]: pgmap v10212: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:56:30.508 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:56:30 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:56:30.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:56:30 smithi047 ceph-mon[77147]: pgmap v10212: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:56:30.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:56:30 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:56:30.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:56:30 smithi115 conmon[90393]: 2022-05-14T04:56:30.505+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:56:31.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:56:31 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:56:31.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:56:31 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:56:31.541 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:56:31 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:56:31.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:56:31 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:56:31.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:56:31 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:56:31.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:56:31 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:56:31.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:56:31 smithi115 conmon[90393]: 2022-05-14T04:56:31.538+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:56:32.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:56:32 smithi047 ceph-mon[77147]: pgmap v10213: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:56:32.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:56:32 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:56:32.582 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:56:32 smithi115 ceph-mon[78593]: pgmap v10213: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:56:32.582 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:56:32 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:56:32.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:56:32 smithi115 conmon[90393]: 2022-05-14T04:56:32.579+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:56:33.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:56:33 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:56:33.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:56:33 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 19725 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:56:33.577 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:56:33 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:56:33.577 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:56:33 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 19725 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:56:33.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:56:33 smithi115 conmon[90393]: 2022-05-14T04:56:33.573+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:56:34.530 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:56:34 smithi115 ceph-mon[78593]: pgmap v10214: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:56:34.530 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:56:34 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:56:34.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:56:34 smithi047 ceph-mon[77147]: pgmap v10214: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:56:34.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:56:34 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:56:34.860 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:56:34 smithi115 conmon[90393]: 2022-05-14T04:56:34.527+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:56:35.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:56:35 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:56:35.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:56:35 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:56:35.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:56:35 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:56:35.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:56:35 smithi115 conmon[90393]: 2022-05-14T04:56:35.494+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:56:35.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:56:35 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:56:35.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:56:35 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:56:35.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:56:35 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:56:36.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:56:36 smithi047 ceph-mon[77147]: pgmap v10215: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:56:36.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:56:36 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:56:36.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:56:36 smithi115 ceph-mon[78593]: pgmap v10215: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:56:36.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:56:36 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:56:36.610 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:56:36 smithi115 conmon[90393]: 2022-05-14T04:56:36.505+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:56:37.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:56:37 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:56:37.549 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:56:37 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:56:37.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:56:37 smithi115 conmon[90393]: 2022-05-14T04:56:37.547+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:56:38.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:56:38 smithi047 ceph-mon[77147]: pgmap v10216: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:56:38.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:56:38 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:56:38.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:56:38 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 19730 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:56:38.594 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:56:38 smithi115 ceph-mon[78593]: pgmap v10216: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:56:38.595 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:56:38 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:56:38.595 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:56:38 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 19730 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:56:38.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:56:38 smithi115 conmon[90393]: 2022-05-14T04:56:38.591+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:56:39.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:56:39 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:56:39.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:56:39 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:56:39.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:56:39 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:56:39.591 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:56:39 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:56:39.591 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:56:39 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:56:39.592 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:56:39 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:56:39.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:56:39 smithi115 conmon[90393]: 2022-05-14T04:56:39.589+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:56:40.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:56:40 smithi047 ceph-mon[77147]: pgmap v10217: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:56:40.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:56:40 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:56:40.560 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:56:40 smithi115 ceph-mon[78593]: pgmap v10217: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:56:40.561 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:56:40 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:56:40.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:56:40 smithi115 conmon[90393]: 2022-05-14T04:56:40.558+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:56:41.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:56:41 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:56:41.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:56:41 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:56:41.610 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:56:41 smithi115 conmon[90393]: 2022-05-14T04:56:41.516+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:56:42.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:56:42 smithi047 ceph-mon[77147]: pgmap v10218: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:56:42.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:56:42 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:56:42.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:56:42 smithi115 ceph-mon[78593]: pgmap v10218: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:56:42.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:56:42 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:56:42.610 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:56:42 smithi115 conmon[90393]: 2022-05-14T04:56:42.506+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:56:43.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:56:43 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:56:43.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:56:43 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:56:43.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:56:43 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:56:43.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:56:43 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 19735 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:56:43.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:56:43 smithi115 conmon[90393]: 2022-05-14T04:56:43.519+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:56:43.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:56:43 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:56:43.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:56:43 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:56:43.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:56:43 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:56:43.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:56:43 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 19735 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:56:44.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:56:44 smithi047 ceph-mon[77147]: pgmap v10219: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:56:44.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:56:44 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:56:44.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:56:44 smithi115 conmon[90393]: 2022-05-14T04:56:44.524+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:56:44.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:56:44 smithi115 ceph-mon[78593]: pgmap v10219: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:56:44.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:56:44 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:56:45.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:56:45 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:56:45.610 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:56:45 smithi115 conmon[90393]: 2022-05-14T04:56:45.487+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:56:45.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:56:45 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:56:46.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:56:46 smithi047 ceph-mon[77147]: pgmap v10220: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:56:46.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:56:46 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:56:46.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:56:46 smithi115 conmon[90393]: 2022-05-14T04:56:46.497+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:56:46.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:56:46 smithi115 ceph-mon[78593]: pgmap v10220: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:56:46.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:56:46 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:56:47.550 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:56:47 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:56:47.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:56:47 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:56:47.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:56:47 smithi115 conmon[90393]: 2022-05-14T04:56:47.546+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:56:48.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:56:48 smithi115 conmon[90393]: 2022-05-14T04:56:48.515+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:56:48.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:56:48 smithi115 ceph-mon[78593]: pgmap v10221: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:56:48.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:56:48 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:56:48.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:56:48 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 19740 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:56:48.795 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:56:48 smithi047 ceph-mon[77147]: pgmap v10221: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:56:48.795 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:56:48 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:56:48.796 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:56:48 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 19740 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:56:49.557 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:56:49 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:56:49.558 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:56:49 smithi115 conmon[90393]: 2022-05-14T04:56:49.535+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:56:49.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:56:49 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:56:50.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:56:50 smithi115 conmon[90393]: 2022-05-14T04:56:50.541+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:56:50.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:56:50 smithi115 ceph-mon[78593]: pgmap v10222: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:56:50.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:56:50 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:56:50.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:56:50 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:56:50.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:56:50 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:56:50.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:56:50 smithi047 ceph-mon[77147]: pgmap v10222: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:56:50.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:56:50 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:56:50.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:56:50 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:56:50.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:56:50 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:56:51.540 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:56:51 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:56:51.795 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:56:51 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:56:51.821 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:56:51 smithi115 conmon[90393]: 2022-05-14T04:56:51.537+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:56:52.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:56:52 smithi115 ceph-mon[78593]: pgmap v10223: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:56:52.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:56:52 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:56:52.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:56:52 smithi047 ceph-mon[77147]: pgmap v10223: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:56:52.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:56:52 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:56:52.860 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:56:52 smithi115 conmon[90393]: 2022-05-14T04:56:52.568+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:56:53.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:56:53 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:56:53.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:56:53 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 19745 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:56:53.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:56:53 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:56:53.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:56:53 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 19745 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:56:54.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:56:53 smithi115 conmon[90393]: 2022-05-14T04:56:53.608+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:56:54.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:56:54 smithi115 ceph-mon[78593]: pgmap v10224: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:56:54.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:56:54 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:56:54.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:56:54 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:56:54.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:56:54 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:56:54.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:56:54 smithi047 ceph-mon[77147]: pgmap v10224: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:56:54.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:56:54 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:56:54.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:56:54 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:56:54.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:56:54 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:56:55.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:56:54 smithi115 conmon[90393]: 2022-05-14T04:56:54.614+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:56:55.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:56:55 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:56:55.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:56:55 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:56:56.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:56:55 smithi115 conmon[90393]: 2022-05-14T04:56:55.614+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:56:56.575 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:56:56 smithi115 ceph-mon[78593]: pgmap v10225: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:56:56.575 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:56:56 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:56:56.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:56:56 smithi047 ceph-mon[77147]: pgmap v10225: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:56:56.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:56:56 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:56:56.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:56:56 smithi115 conmon[90393]: 2022-05-14T04:56:56.572+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:56:57.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:56:57 smithi115 conmon[90393]: 2022-05-14T04:56:57.559+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:56:57.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:56:57 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:56:57.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:56:57 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:56:58.577 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:56:58 smithi115 ceph-mon[78593]: pgmap v10226: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:56:58.578 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:56:58 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:56:58.578 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:56:58 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:56:58.578 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:56:58 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:56:58.578 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:56:58 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 19750 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:56:58.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:56:58 smithi047 ceph-mon[77147]: pgmap v10226: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:56:58.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:56:58 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:56:58.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:56:58 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:56:58.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:56:58 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:56:58.795 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:56:58 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 19750 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:56:58.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:56:58 smithi115 conmon[90393]: 2022-05-14T04:56:58.574+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:56:59.604 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:56:59 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:56:59.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:56:59 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:56:59.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:56:59 smithi115 conmon[90393]: 2022-05-14T04:56:59.601+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:57:00.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:57:00 smithi115 ceph-mon[78593]: pgmap v10227: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:57:00.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:57:00 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:57:00.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:57:00 smithi047 ceph-mon[77147]: pgmap v10227: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:57:00.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:57:00 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:57:01.110 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:57:00 smithi115 conmon[90393]: 2022-05-14T04:57:00.647+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:57:01.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:57:01 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:57:01.795 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:57:01 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:57:02.110 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:57:01 smithi115 conmon[90393]: 2022-05-14T04:57:01.655+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:57:02.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:57:02 smithi115 ceph-mon[78593]: pgmap v10228: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:57:02.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:57:02 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:57:02.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:57:02 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:57:02.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:57:02 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:57:02.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:57:02 smithi047 ceph-mon[77147]: pgmap v10228: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:57:02.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:57:02 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:57:02.795 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:57:02 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:57:02.795 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:57:02 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:57:03.111 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:57:02 smithi115 conmon[90393]: 2022-05-14T04:57:02.689+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:57:03.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:57:03 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:57:03.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:57:03 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 19755 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:57:03.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:57:03 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:57:03.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:57:03 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 19755 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:57:04.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:57:03 smithi115 conmon[90393]: 2022-05-14T04:57:03.709+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:57:04.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:57:04 smithi115 ceph-mon[78593]: pgmap v10229: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:57:04.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:57:04 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:57:04.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:57:04 smithi047 ceph-mon[77147]: pgmap v10229: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:57:04.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:57:04 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:57:05.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:57:04 smithi115 conmon[90393]: 2022-05-14T04:57:04.718+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:57:05.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:57:05 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:57:05.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:57:05 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:57:06.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:57:05 smithi115 conmon[90393]: 2022-05-14T04:57:05.715+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:57:06.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:57:06 smithi115 ceph-mon[78593]: pgmap v10230: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:57:06.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:57:06 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:57:06.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:57:06 smithi047 ceph-mon[77147]: pgmap v10230: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:57:06.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:57:06 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:57:07.110 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:57:06 smithi115 conmon[90393]: 2022-05-14T04:57:06.698+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:57:07.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:57:07 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:57:07.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:57:07 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:57:08.113 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:57:07 smithi115 conmon[90393]: 2022-05-14T04:57:07.740+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:57:08.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:57:08 smithi115 ceph-mon[78593]: pgmap v10231: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:57:08.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:57:08 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:57:08.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:57:08 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 19760 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:57:08.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:57:08 smithi047 ceph-mon[77147]: pgmap v10231: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:57:08.795 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:57:08 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:57:08.795 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:57:08 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 19760 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:57:09.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:57:08 smithi115 conmon[90393]: 2022-05-14T04:57:08.706+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:57:09.751 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:57:09 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:57:09.795 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:57:09 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:57:10.110 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:57:09 smithi115 conmon[90393]: 2022-05-14T04:57:09.748+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:57:10.764 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:57:10 smithi115 ceph-mon[78593]: pgmap v10232: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:57:10.764 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:57:10 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:57:10.765 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:57:10 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:57:10.765 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:57:10 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:57:10.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:57:10 smithi047 ceph-mon[77147]: pgmap v10232: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:57:10.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:57:10 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:57:10.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:57:10 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:57:10.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:57:10 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:57:11.110 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:57:10 smithi115 conmon[90393]: 2022-05-14T04:57:10.760+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:57:11.765 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:57:11 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:57:11.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:57:11 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:57:12.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:57:11 smithi115 conmon[90393]: 2022-05-14T04:57:11.763+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:57:12.778 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:57:12 smithi115 ceph-mon[78593]: pgmap v10233: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:57:12.778 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:57:12 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:57:12.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:57:12 smithi047 ceph-mon[77147]: pgmap v10233: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:57:12.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:57:12 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:57:13.110 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:57:12 smithi115 conmon[90393]: 2022-05-14T04:57:12.774+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:57:13.762 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:57:13 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:57:13.763 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:57:13 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 19765 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:57:13.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:57:13 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:57:13.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:57:13 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 19765 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:57:14.110 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:57:13 smithi115 conmon[90393]: 2022-05-14T04:57:13.759+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:57:14.753 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:57:14 smithi115 ceph-mon[78593]: pgmap v10234: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:57:14.754 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:57:14 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:57:14.754 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:57:14 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:57:14.754 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:57:14 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:57:14.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:57:14 smithi047 ceph-mon[77147]: pgmap v10234: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:57:14.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:57:14 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:57:14.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:57:14 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:57:14.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:57:14 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:57:15.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:57:14 smithi115 conmon[90393]: 2022-05-14T04:57:14.750+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:57:15.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:57:15 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T04:57:15.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:57:15 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:57:15.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:57:15 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T04:57:15.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:57:15 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T04:57:15.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:57:15 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:57:15.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:57:15 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T04:57:16.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:57:15 smithi115 conmon[90393]: 2022-05-14T04:57:15.789+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:57:16.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:57:16 smithi047 ceph-mon[77147]: pgmap v10235: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:57:16.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:57:16 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:57:16.817 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:57:16 smithi115 ceph-mon[78593]: pgmap v10235: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:57:16.817 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:57:16 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:57:17.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:57:16 smithi115 conmon[90393]: 2022-05-14T04:57:16.814+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:57:18.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:57:17 smithi115 conmon[90393]: 2022-05-14T04:57:17.841+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:57:18.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:57:18 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:57:18.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:57:18 smithi047 ceph-mon[77147]: pgmap v10236: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:57:18.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:57:18 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:57:18.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:57:18 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:57:18.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:57:18 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 19770 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:57:18.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:57:18 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:57:18.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:57:18 smithi115 ceph-mon[78593]: pgmap v10236: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:57:18.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:57:18 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:57:18.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:57:18 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:57:18.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:57:18 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 19770 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:57:19.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:57:18 smithi115 conmon[90393]: 2022-05-14T04:57:18.880+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:57:19.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:57:19 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:57:19.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:57:19 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:57:20.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:57:19 smithi115 conmon[90393]: 2022-05-14T04:57:19.871+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:57:20.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:57:20 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:57:20.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:57:20 smithi047 ceph-mon[77147]: pgmap v10237: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:57:20.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:57:20 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:57:20.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:57:20 smithi115 ceph-mon[78593]: pgmap v10237: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:57:21.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:57:20 smithi115 conmon[90393]: 2022-05-14T04:57:20.922+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:57:21.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:57:21 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:57:21.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:57:21 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:57:22.293 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:57:21 smithi047 conmon[78521]: 2022-05-14T04:57:21.903+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:57:22.293 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:57:21 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T04:57:22.293 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:57:21 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:57:22.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:57:21 smithi047 conmon[78521]: yield 2022-05-14T04:57:22.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:57:21 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:57:22.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:57:21 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:57:22.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:57:21 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:57:22.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:57:21 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T04:57:22.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:57:21 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:57:22.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:57:21 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T04:57:22.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:57:21 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:57:22.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:57:21 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T04:57:22.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:57:21 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:57:22.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:57:21 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T04:57:22.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:57:21 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:57:22.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:57:21 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T04:57:22.297 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:57:21 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:57:22.297 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:57:21 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:57:22.297 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:57:21 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:57:22.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:57:21 smithi115 conmon[90393]: 2022-05-14T04:57:21.874+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:57:22.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:57:22 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:57:22.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:57:22 smithi047 ceph-mon[77147]: pgmap v10238: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:57:22.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:57:22 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:57:22.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:57:22 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:57:22.795 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:57:22 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T04:57:22.795 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:57:22 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T04:57:22.795 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:57:22 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T04:57:22.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:57:22 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:57:22.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:57:22 smithi115 ceph-mon[78593]: pgmap v10238: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:57:22.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:57:22 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:57:22.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:57:22 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:57:22.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:57:22 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T04:57:22.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:57:22 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T04:57:22.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:57:22 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T04:57:23.360 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:57:22 smithi115 conmon[90393]: 2022-05-14T04:57:22.888+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:57:23.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:57:23 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:57:23.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:57:23 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:57:23.793 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T04:57:23.793 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:57:23.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T04:57:23.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:57:23.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:57:23.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:57:23.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T04:57:23.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:57:23.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T04:57:23.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:57:23.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T04:57:23.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:57:23.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T04:57:23.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:57:23.797 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T04:57:23.797 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:57:23.797 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:57:23.797 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:57:23.797 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:57:23 smithi047 ceph-mon[77147]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T04:57:23.798 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:57:23 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 19775 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:57:23.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:57:23 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:57:23.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:57:23 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:57:23.860 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T04:57:23.860 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:57:23.860 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T04:57:23.860 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:57:23.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:57:23.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:57:23.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T04:57:23.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:57:23.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T04:57:23.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:57:23.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T04:57:23.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:57:23.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T04:57:23.863 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:57:23.863 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T04:57:23.864 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:57:23.864 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:57:23.864 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:57:23.864 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:57:23 smithi115 ceph-mon[78593]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T04:57:23.864 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:57:23 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 19775 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:57:24.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:57:23 smithi115 conmon[90393]: 2022-05-14T04:57:23.892+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:57:24.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:57:24 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:57:24.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:57:24 smithi047 ceph-mon[77147]: pgmap v10239: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:57:24.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:57:24 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:57:24.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:57:24 smithi115 ceph-mon[78593]: pgmap v10239: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:57:25.293 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:57:24 smithi047 conmon[78521]: 2022-05-14T04:57:24.976+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:57:25.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:57:24 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T04:57:25.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:57:24 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:57:25.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:57:24 smithi047 conmon[78521]: yield 2022-05-14T04:57:25.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:57:24 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:57:25.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:57:24 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:57:25.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:57:24 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:57:25.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:57:24 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T04:57:25.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:57:24 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:57:25.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:57:24 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T04:57:25.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:57:24 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:57:25.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:57:24 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T04:57:25.298 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:57:24 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:57:25.298 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:57:24 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T04:57:25.298 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:57:24 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:57:25.298 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:57:24 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T04:57:25.298 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:57:24 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:57:25.299 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:57:24 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:57:25.299 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:57:24 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:57:25.299 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:57:24 smithi047 conmon[78521]: 2022-05-14T04:57:24.976+0000 7f533ef76700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T04:57:25.299 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:57:24 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T04:57:25.300 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:57:24 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:57:25.300 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:57:24 smithi047 conmon[78521]: yield 2022-05-14T04:57:25.300 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:57:24 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:57:25.300 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:57:24 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:57:25.300 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:57:24 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:57:25.301 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:57:24 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T04:57:25.301 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:57:24 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:57:25.301 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:57:24 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T04:57:25.301 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:57:24 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:57:25.302 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:57:24 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T04:57:25.302 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:57:24 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:57:25.302 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:57:24 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T04:57:25.302 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:57:24 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:57:25.303 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:57:24 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T04:57:25.303 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:57:24 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:57:25.303 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:57:24 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:57:25.303 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:57:24 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:57:25.304 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:57:24 smithi047 conmon[78521]: 2022-05-14T04:57:25.304 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:57:24 smithi047 conmon[78521]: During handling of the above exception, another exception occurred: 2022-05-14T04:57:25.304 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:57:24 smithi047 conmon[78521]: 2022-05-14T04:57:25.304 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:57:24 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T04:57:25.305 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:57:24 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T04:57:25.305 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:57:24 smithi047 conmon[78521]: return f(*arg) 2022-05-14T04:57:25.305 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:57:24 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T04:57:25.305 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:57:24 smithi047 conmon[78521]: self._write_client_files(client_files, host) 2022-05-14T04:57:25.305 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:57:24 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T04:57:25.306 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:57:24 smithi047 conmon[78521]: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T04:57:25.306 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:57:24 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T04:57:25.306 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:57:24 smithi047 conmon[78521]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T04:57:25.306 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:57:24 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T04:57:25.306 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:57:24 smithi047 conmon[78521]: return self.event_loop.get_result(coro) 2022-05-14T04:57:25.307 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:57:24 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T04:57:25.307 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:57:24 smithi047 conmon[78521]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T04:57:25.307 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:57:24 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T04:57:25.307 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:57:24 smithi047 conmon[78521]: return self.__get_result() 2022-05-14T04:57:25.308 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:57:24 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T04:57:25.308 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:57:24 smithi047 conmon[78521]: raise self._exception 2022-05-14T04:57:25.308 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:57:24 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T04:57:25.308 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:57:24 smithi047 conmon[78521]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T04:57:25.308 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:57:24 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T04:57:25.309 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:57:24 smithi047 conmon[78521]: conn = await self._remote_connection(host, addr) 2022-05-14T04:57:25.309 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:57:24 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T04:57:25.309 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:57:24 smithi047 conmon[78521]: raise 2022-05-14T04:57:25.309 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:57:24 smithi047 conmon[78521]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T04:57:25.309 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:57:24 smithi047 conmon[78521]: self.gen.throw(type, value, traceback) 2022-05-14T04:57:25.310 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:57:24 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T04:57:25.310 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:57:24 smithi047 conmon[78521]: raise OrchestratorError(msg) 2022-05-14T04:57:25.310 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:57:24 smithi047 conmon[78521]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:57:25.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:57:24 smithi115 conmon[90393]: 2022-05-14T04:57:24.905+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:57:25.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:57:25 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:57:25.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:57:25 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:57:26.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:57:25 smithi115 conmon[90393]: 2022-05-14T04:57:25.874+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:57:26.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:57:26 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:57:26.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:57:26 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:57:26.793 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T04:57:26.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:57:26.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T04:57:26.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:57:26.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:57:26.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:57:26.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T04:57:26.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:57:26.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T04:57:26.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:57:26.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T04:57:26.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:57:26.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T04:57:26.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:57:26.797 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T04:57:26.797 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:57:26.797 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:57:26.797 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:57:26.798 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:57:26 smithi047 ceph-mon[77147]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T04:57:26.798 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T04:57:26.798 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:57:26.798 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T04:57:26.798 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:57:26.799 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:57:26.799 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:57:26.799 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T04:57:26.799 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:57:26.800 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T04:57:26.800 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:57:26.800 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T04:57:26.800 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:57:26.800 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T04:57:26.801 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:57:26.801 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T04:57:26.801 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:57:26.801 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:57:26.801 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:57:26.802 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T04:57:26.802 INFO:journalctl@ceph.mon.b.smithi047.stdout: During handling of the above exception, another exception occurred: 2022-05-14T04:57:26.802 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T04:57:26.802 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T04:57:26.803 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T04:57:26.803 INFO:journalctl@ceph.mon.b.smithi047.stdout: return f(*arg) 2022-05-14T04:57:26.803 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T04:57:26.803 INFO:journalctl@ceph.mon.b.smithi047.stdout: self._write_client_files(client_files, host) 2022-05-14T04:57:26.803 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T04:57:26.804 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T04:57:26.804 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T04:57:26.804 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T04:57:26.805 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T04:57:26.805 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.event_loop.get_result(coro) 2022-05-14T04:57:26.805 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T04:57:26.806 INFO:journalctl@ceph.mon.b.smithi047.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T04:57:26.806 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T04:57:26.806 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.__get_result() 2022-05-14T04:57:26.806 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T04:57:26.807 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise self._exception 2022-05-14T04:57:26.807 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T04:57:26.807 INFO:journalctl@ceph.mon.b.smithi047.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T04:57:26.807 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T04:57:26.807 INFO:journalctl@ceph.mon.b.smithi047.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T04:57:26.808 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T04:57:26.808 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise 2022-05-14T04:57:26.808 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T04:57:26.808 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.gen.throw(type, value, traceback) 2022-05-14T04:57:26.809 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T04:57:26.809 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OrchestratorError(msg) 2022-05-14T04:57:26.809 INFO:journalctl@ceph.mon.b.smithi047.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:57:26.809 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:57:26 smithi047 ceph-mon[77147]: pgmap v10240: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:57:26.830 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:57:26 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:57:26.831 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:57:26 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:57:26.831 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T04:57:26.831 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:57:26.831 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T04:57:26.832 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:57:26.832 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:57:26.832 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:57:26.832 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T04:57:26.833 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:57:26.833 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T04:57:26.833 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:57:26.833 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T04:57:26.833 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:57:26.834 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T04:57:26.834 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:57:26.834 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T04:57:26.834 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:57:26.834 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:57:26.835 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:57:26.835 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:57:26 smithi115 ceph-mon[78593]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T04:57:26.835 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T04:57:26.835 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:57:26.836 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T04:57:26.836 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:57:26.836 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:57:26.836 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:57:26.836 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T04:57:26.837 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:57:26.837 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T04:57:26.837 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:57:26.837 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T04:57:26.837 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:57:26.838 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T04:57:26.838 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:57:26.839 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T04:57:26.839 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:57:26.839 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:57:26.839 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:57:26.839 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T04:57:26.840 INFO:journalctl@ceph.mon.c.smithi115.stdout: During handling of the above exception, another exception occurred: 2022-05-14T04:57:26.840 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T04:57:26.840 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T04:57:26.840 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T04:57:26.841 INFO:journalctl@ceph.mon.c.smithi115.stdout: return f(*arg) 2022-05-14T04:57:26.841 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T04:57:26.841 INFO:journalctl@ceph.mon.c.smithi115.stdout: self._write_client_files(client_files, host) 2022-05-14T04:57:26.841 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T04:57:26.841 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T04:57:26.842 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T04:57:26.842 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T04:57:26.842 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T04:57:26.842 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.event_loop.get_result(coro) 2022-05-14T04:57:26.842 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T04:57:26.843 INFO:journalctl@ceph.mon.c.smithi115.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T04:57:26.843 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T04:57:26.843 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.__get_result() 2022-05-14T04:57:26.843 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T04:57:26.843 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise self._exception 2022-05-14T04:57:26.844 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T04:57:26.844 INFO:journalctl@ceph.mon.c.smithi115.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T04:57:26.844 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T04:57:26.844 INFO:journalctl@ceph.mon.c.smithi115.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T04:57:26.845 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T04:57:26.845 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise 2022-05-14T04:57:26.845 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T04:57:26.845 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.gen.throw(type, value, traceback) 2022-05-14T04:57:26.845 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T04:57:26.846 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OrchestratorError(msg) 2022-05-14T04:57:26.846 INFO:journalctl@ceph.mon.c.smithi115.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:57:26.846 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:57:26 smithi115 ceph-mon[78593]: pgmap v10240: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:57:27.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:57:26 smithi115 conmon[90393]: 2022-05-14T04:57:26.827+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:57:27.798 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:57:27 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:57:27.810 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:57:27 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:57:28.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:57:27 smithi115 conmon[90393]: 2022-05-14T04:57:27.808+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:57:28.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:57:28 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:57:28.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:57:28 smithi047 ceph-mon[77147]: pgmap v10241: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:57:28.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:57:28 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:57:28.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:57:28 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 19780 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:57:28.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:57:28 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:57:28.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:57:28 smithi115 ceph-mon[78593]: pgmap v10241: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:57:28.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:57:28 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:57:28.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:57:28 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 19780 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:57:29.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:57:28 smithi115 conmon[90393]: 2022-05-14T04:57:28.856+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:57:30.362 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:57:29 smithi115 conmon[90393]: 2022-05-14T04:57:29.858+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:57:30.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:57:30 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:57:30.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:57:30 smithi047 ceph-mon[77147]: pgmap v10242: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:57:30.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:57:30 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:57:30.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:57:30 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:57:30.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:57:30 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:57:30.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:57:30 smithi115 ceph-mon[78593]: pgmap v10242: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:57:30.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:57:30 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:57:30.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:57:30 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:57:31.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:57:30 smithi115 conmon[90393]: 2022-05-14T04:57:30.889+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:57:31.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:57:31 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:57:31.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:57:31 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:57:32.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:57:31 smithi115 conmon[90393]: 2022-05-14T04:57:31.894+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:57:32.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:57:32 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:57:32.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:57:32 smithi047 ceph-mon[77147]: pgmap v10243: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:57:32.849 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:57:32 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:57:32.849 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:57:32 smithi115 ceph-mon[78593]: pgmap v10243: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:57:33.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:57:32 smithi115 conmon[90393]: 2022-05-14T04:57:32.846+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:57:33.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:57:33 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:57:33.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:57:33 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 19785 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:57:33.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:57:33 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:57:33.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:57:33 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 19785 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:57:34.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:57:33 smithi115 conmon[90393]: 2022-05-14T04:57:33.889+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:57:34.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:57:34 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:57:34.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:57:34 smithi047 ceph-mon[77147]: pgmap v10244: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:57:34.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:57:34 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:57:34.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:57:34 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:57:34.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:57:34 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:57:34.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:57:34 smithi115 ceph-mon[78593]: pgmap v10244: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:57:34.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:57:34 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:57:34.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:57:34 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:57:35.360 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:57:34 smithi115 conmon[90393]: 2022-05-14T04:57:34.875+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:57:35.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:57:35 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:57:35.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:57:35 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:57:36.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:57:35 smithi115 conmon[90393]: 2022-05-14T04:57:35.886+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:57:36.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:57:36 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:57:36.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:57:36 smithi047 ceph-mon[77147]: pgmap v10245: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:57:36.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:57:36 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:57:36.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:57:36 smithi115 ceph-mon[78593]: pgmap v10245: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:57:37.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:57:36 smithi115 conmon[90393]: 2022-05-14T04:57:36.859+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:57:37.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:57:37 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:57:37.853 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:57:37 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:57:38.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:57:37 smithi115 conmon[90393]: 2022-05-14T04:57:37.849+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:57:38.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:57:38 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:57:38.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:57:38 smithi047 ceph-mon[77147]: pgmap v10246: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:57:38.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:57:38 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:57:38.795 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:57:38 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:57:38.795 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:57:38 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 19790 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:57:38.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:57:38 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:57:38.862 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:57:38 smithi115 ceph-mon[78593]: pgmap v10246: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:57:38.862 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:57:38 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:57:38.862 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:57:38 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:57:38.862 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:57:38 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 19790 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:57:39.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:57:38 smithi115 conmon[90393]: 2022-05-14T04:57:38.865+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:57:39.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:57:39 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:57:39.827 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:57:39 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:57:40.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:57:39 smithi115 conmon[90393]: 2022-05-14T04:57:39.820+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:57:40.778 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:57:40 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:57:40.778 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:57:40 smithi115 ceph-mon[78593]: pgmap v10247: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:57:40.795 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:57:40 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:57:40.795 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:57:40 smithi047 ceph-mon[77147]: pgmap v10247: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:57:41.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:57:40 smithi115 conmon[90393]: 2022-05-14T04:57:40.774+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:57:41.765 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:57:41 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:57:41.766 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:57:41 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:57:41.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:57:41 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:57:41.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:57:41 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:57:42.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:57:41 smithi115 conmon[90393]: 2022-05-14T04:57:41.762+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:57:42.774 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:57:42 smithi115 ceph-mon[78593]: pgmap v10248: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:57:42.774 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:57:42 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:57:42.774 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:57:42 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:57:42.774 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:57:42 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:57:42.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:57:42 smithi047 ceph-mon[77147]: pgmap v10248: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:57:42.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:57:42 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:57:42.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:57:42 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:57:42.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:57:42 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:57:43.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:57:42 smithi115 conmon[90393]: 2022-05-14T04:57:42.770+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:57:43.770 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:57:43 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:57:43.770 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:57:43 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 19795 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:57:43.796 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:57:43 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:57:43.796 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:57:43 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 19795 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:57:44.110 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:57:43 smithi115 conmon[90393]: 2022-05-14T04:57:43.767+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:57:44.745 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:57:44 smithi115 ceph-mon[78593]: pgmap v10249: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:57:44.746 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:57:44 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:57:44.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:57:44 smithi047 ceph-mon[77147]: pgmap v10249: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:57:44.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:57:44 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:57:45.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:57:44 smithi115 conmon[90393]: 2022-05-14T04:57:44.741+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:57:45.754 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:57:45 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:57:45.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:57:45 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:57:46.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:57:45 smithi115 conmon[90393]: 2022-05-14T04:57:45.747+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:57:46.725 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:57:46 smithi115 ceph-mon[78593]: pgmap v10250: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:57:46.726 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:57:46 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:57:46.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:57:46 smithi047 ceph-mon[77147]: pgmap v10250: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:57:46.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:57:46 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:57:47.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:57:46 smithi115 conmon[90393]: 2022-05-14T04:57:46.722+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:57:47.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:57:47 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:57:47.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:57:47 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:57:47.860 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:57:47 smithi115 conmon[90393]: 2022-05-14T04:57:47.711+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:57:48.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:57:48 smithi047 ceph-mon[77147]: pgmap v10251: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:57:48.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:57:48 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:57:48.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:57:48 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 19800 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:57:48.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:57:48 smithi115 ceph-mon[78593]: pgmap v10251: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:57:48.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:57:48 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:57:48.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:57:48 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 19800 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:57:48.860 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:57:48 smithi115 conmon[90393]: 2022-05-14T04:57:48.694+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:57:49.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:57:49 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:57:49.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:57:49 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:57:49.861 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:57:49 smithi115 conmon[90393]: 2022-05-14T04:57:49.707+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:57:50.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:57:50 smithi047 ceph-mon[77147]: pgmap v10252: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:57:50.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:57:50 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:57:50.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:57:50 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:57:50.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:57:50 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:57:50.860 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:57:50 smithi115 conmon[90393]: 2022-05-14T04:57:50.668+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:57:50.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:57:50 smithi115 ceph-mon[78593]: pgmap v10252: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:57:50.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:57:50 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:57:50.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:57:50 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:57:50.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:57:50 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:57:51.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:57:51 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:57:51.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:57:51 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:57:51.860 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:57:51 smithi115 conmon[90393]: 2022-05-14T04:57:51.668+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:57:52.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:57:52 smithi047 ceph-mon[77147]: pgmap v10253: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:57:52.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:57:52 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:57:52.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:57:52 smithi115 conmon[90393]: 2022-05-14T04:57:52.703+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:57:52.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:57:52 smithi115 ceph-mon[78593]: pgmap v10253: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:57:52.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:57:52 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:57:53.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:57:53 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:57:53.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:57:53 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 19805 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:57:53.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:57:53 smithi115 conmon[90393]: 2022-05-14T04:57:53.705+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:57:53.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:57:53 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:57:53.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:57:53 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 19805 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:57:54.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:57:54 smithi047 ceph-mon[77147]: pgmap v10254: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:57:54.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:57:54 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:57:54.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:57:54 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:57:54.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:57:54 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:57:54.862 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:57:54 smithi115 conmon[90393]: 2022-05-14T04:57:54.728+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:57:54.862 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:57:54 smithi115 ceph-mon[78593]: pgmap v10254: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:57:54.863 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:57:54 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:57:54.863 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:57:54 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:57:54.863 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:57:54 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:57:55.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:57:55 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:57:55.860 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:57:55 smithi115 conmon[90393]: 2022-05-14T04:57:55.739+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:57:55.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:57:55 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:57:56.798 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:57:56 smithi047 ceph-mon[77147]: pgmap v10255: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:57:56.799 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:57:56 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:57:56.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:57:56 smithi115 conmon[90393]: 2022-05-14T04:57:56.721+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:57:56.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:57:56 smithi115 ceph-mon[78593]: pgmap v10255: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:57:56.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:57:56 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:57:57.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:57:57 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:57:57.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:57:57 smithi115 conmon[90393]: 2022-05-14T04:57:57.677+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:57:57.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:57:57 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:57:58.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:57:58 smithi047 ceph-mon[77147]: pgmap v10256: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:57:58.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:57:58 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:57:58.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:57:58 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:57:58.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:57:58 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:57:58.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:57:58 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 19810 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:57:58.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:57:58 smithi115 conmon[90393]: 2022-05-14T04:57:58.681+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:57:58.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:57:58 smithi115 ceph-mon[78593]: pgmap v10256: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:57:58.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:57:58 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:57:58.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:57:58 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:57:58.862 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:57:58 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:57:58.863 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:57:58 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 19810 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:57:59.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:57:59 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:57:59.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:57:59 smithi115 conmon[90393]: 2022-05-14T04:57:59.671+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:57:59.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:57:59 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:58:00.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:58:00 smithi047 ceph-mon[77147]: pgmap v10257: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:58:00.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:58:00 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:58:00.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:58:00 smithi115 ceph-mon[78593]: pgmap v10257: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:58:00.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:58:00 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:58:00.861 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:58:00 smithi115 conmon[90393]: 2022-05-14T04:58:00.629+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:58:01.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:58:01 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:58:01.863 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:58:01 smithi115 conmon[90393]: 2022-05-14T04:58:01.676+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:58:01.869 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:58:01 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:58:02.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:58:02 smithi047 ceph-mon[77147]: pgmap v10258: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:58:02.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:58:02 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:58:02.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:58:02 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:58:02.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:58:02 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:58:02.860 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:58:02 smithi115 conmon[90393]: 2022-05-14T04:58:02.677+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:58:02.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:58:02 smithi115 ceph-mon[78593]: pgmap v10258: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:58:02.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:58:02 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:58:02.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:58:02 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:58:02.862 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:58:02 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:58:03.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:58:03 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:58:03.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:58:03 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 19815 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:58:03.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:58:03 smithi115 conmon[90393]: 2022-05-14T04:58:03.658+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:58:03.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:58:03 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:58:03.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:58:03 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 19815 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:58:04.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:58:04 smithi047 ceph-mon[77147]: pgmap v10259: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:58:04.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:58:04 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:58:04.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:58:04 smithi115 conmon[90393]: 2022-05-14T04:58:04.654+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:58:04.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:58:04 smithi115 ceph-mon[78593]: pgmap v10259: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:58:04.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:58:04 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:58:05.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:58:05 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:58:05.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:58:05 smithi115 conmon[90393]: 2022-05-14T04:58:05.695+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:58:05.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:58:05 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:58:06.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:58:06 smithi047 ceph-mon[77147]: pgmap v10260: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:58:06.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:58:06 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:58:06.860 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:58:06 smithi115 conmon[90393]: 2022-05-14T04:58:06.670+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:58:06.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:58:06 smithi115 ceph-mon[78593]: pgmap v10260: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:58:06.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:58:06 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:58:07.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:58:07 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:58:07.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:58:07 smithi115 conmon[90393]: 2022-05-14T04:58:07.692+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:58:07.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:58:07 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:58:08.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:58:08 smithi047 ceph-mon[77147]: pgmap v10261: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:58:08.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:58:08 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:58:08.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:58:08 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 19820 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:58:08.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:58:08 smithi115 conmon[90393]: 2022-05-14T04:58:08.734+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:58:08.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:58:08 smithi115 ceph-mon[78593]: pgmap v10261: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:58:08.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:58:08 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:58:08.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:58:08 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 19820 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:58:09.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:58:09 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:58:09.860 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:58:09 smithi115 conmon[90393]: 2022-05-14T04:58:09.745+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:58:09.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:58:09 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:58:10.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:58:10 smithi047 ceph-mon[77147]: pgmap v10262: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:58:10.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:58:10 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:58:10.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:58:10 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:58:10.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:58:10 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:58:10.861 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:58:10 smithi115 conmon[90393]: 2022-05-14T04:58:10.765+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:58:10.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:58:10 smithi115 ceph-mon[78593]: pgmap v10262: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:58:10.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:58:10 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:58:10.862 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:58:10 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:58:10.862 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:58:10 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:58:11.862 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:58:11 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:58:11.863 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:58:11 smithi115 conmon[90393]: 2022-05-14T04:58:11.774+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:58:12.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:58:11 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:58:12.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:58:12 smithi115 ceph-mon[78593]: pgmap v10263: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:58:12.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:58:12 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:58:12.860 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:58:12 smithi115 conmon[90393]: 2022-05-14T04:58:12.782+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:58:13.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:58:12 smithi047 ceph-mon[77147]: pgmap v10263: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:58:13.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:58:12 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:58:13.865 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:58:13 smithi115 conmon[90393]: 2022-05-14T04:58:13.772+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:58:13.866 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:58:13 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:58:13.866 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:58:13 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 19825 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:58:14.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:58:13 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:58:14.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:58:13 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 19825 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:58:14.810 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:58:14 smithi115 ceph-mon[78593]: pgmap v10264: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:58:14.810 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:58:14 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:58:14.810 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:58:14 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:58:14.810 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:58:14 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:58:15.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:58:14 smithi047 ceph-mon[77147]: pgmap v10264: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:58:15.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:58:14 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:58:15.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:58:14 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:58:15.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:58:14 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:58:15.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:58:14 smithi115 conmon[90393]: 2022-05-14T04:58:14.803+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:58:15.825 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:58:15 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T04:58:15.826 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:58:15 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T04:58:15.826 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:58:15 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:58:16.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:58:15 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T04:58:16.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:58:15 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T04:58:16.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:58:15 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:58:16.110 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:58:15 smithi115 conmon[90393]: 2022-05-14T04:58:15.822+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:58:16.857 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:58:16 smithi115 ceph-mon[78593]: pgmap v10265: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:58:16.858 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:58:16 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:58:17.046 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:58:16 smithi047 ceph-mon[77147]: pgmap v10265: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:58:17.046 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:58:16 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:58:17.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:58:16 smithi115 conmon[90393]: 2022-05-14T04:58:16.851+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:58:18.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:58:17 smithi115 conmon[90393]: 2022-05-14T04:58:17.888+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:58:18.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:58:18 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:58:18.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:58:18 smithi115 ceph-mon[78593]: pgmap v10266: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:58:18.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:58:18 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:58:18.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:58:18 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:58:18.862 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:58:18 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 19830 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:58:19.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:58:18 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:58:19.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:58:18 smithi047 ceph-mon[77147]: pgmap v10266: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:58:19.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:58:18 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:58:19.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:58:18 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:58:19.045 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:58:18 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 19830 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:58:19.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:58:18 smithi115 conmon[90393]: 2022-05-14T04:58:18.867+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:58:19.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:58:19 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:58:20.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:58:19 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:58:20.365 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:58:19 smithi115 conmon[90393]: 2022-05-14T04:58:19.859+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:58:20.862 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:58:20 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:58:20.863 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:58:20 smithi115 ceph-mon[78593]: pgmap v10267: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:58:20.863 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:58:20 smithi115 conmon[90393]: 2022-05-14T04:58:20.814+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:58:21.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:58:20 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:58:21.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:58:20 smithi047 ceph-mon[77147]: pgmap v10267: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:58:21.861 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:58:21 smithi115 conmon[90393]: 2022-05-14T04:58:21.793+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:58:21.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:58:21 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:58:21.862 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:58:21 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:58:22.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:58:21 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:58:22.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:58:21 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:58:22.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:58:22 smithi115 conmon[90393]: 2022-05-14T04:58:22.803+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:58:22.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:58:22 smithi115 ceph-mon[78593]: pgmap v10268: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:58:22.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:58:22 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:58:22.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:58:22 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:58:22.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:58:22 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:58:23.045 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:58:22 smithi047 ceph-mon[77147]: pgmap v10268: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:58:23.046 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:58:22 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:58:23.046 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:58:22 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:58:23.048 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:58:22 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:58:23.840 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:58:23 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:58:23.840 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:58:23 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 19835 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:58:24.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:58:23 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:58:24.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:58:23 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 19835 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:58:24.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:58:23 smithi115 conmon[90393]: 2022-05-14T04:58:23.837+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:58:24.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:58:24 smithi115 ceph-mon[78593]: pgmap v10269: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:58:25.027 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:58:24 smithi047 ceph-mon[77147]: pgmap v10269: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:58:25.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:58:24 smithi115 conmon[90393]: 2022-05-14T04:58:24.874+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:58:25.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:58:25 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:58:25.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:58:25 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T04:58:25.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:58:25 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T04:58:25.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:58:25 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T04:58:25.862 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:58:25 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:58:25.862 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:58:25 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T04:58:25.862 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:58:25 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T04:58:25.863 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:58:25 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T04:58:26.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:58:25 smithi115 conmon[90393]: 2022-05-14T04:58:25.877+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:58:26.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:58:26 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:58:26.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:58:26 smithi115 ceph-mon[78593]: pgmap v10270: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:58:27.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:58:26 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:58:27.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:58:26 smithi047 ceph-mon[77147]: pgmap v10270: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:58:27.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:58:26 smithi115 conmon[90393]: 2022-05-14T04:58:26.884+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:58:27.853 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:58:27 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:58:28.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:58:27 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:58:28.110 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:58:27 smithi115 conmon[90393]: 2022-05-14T04:58:27.838+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:58:28.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:58:28 smithi047 conmon[78521]: 2022-05-14T04:58:28.080+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:58:28.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:58:28 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T04:58:28.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:58:28 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:58:28.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:58:28 smithi047 conmon[78521]: yield 2022-05-14T04:58:28.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:58:28 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:58:28.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:58:28 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:58:28.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:58:28 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:58:28.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:58:28 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T04:58:28.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:58:28 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:58:28.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:58:28 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T04:58:28.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:58:28 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:58:28.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:58:28 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T04:58:28.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:58:28 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:58:28.547 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:58:28 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T04:58:28.547 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:58:28 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:58:28.547 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:58:28 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T04:58:28.547 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:58:28 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:58:28.547 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:58:28 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:58:28.548 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:58:28 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:58:28.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:58:28 smithi115 conmon[90393]: 2022-05-14T04:58:28.811+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:58:28.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:58:28 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:58:28.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:58:28 smithi115 ceph-mon[78593]: pgmap v10271: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:58:28.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:58:28 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 19840 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:58:29.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:58:28 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:58:29.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:58:28 smithi047 ceph-mon[77147]: pgmap v10271: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:58:29.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:58:28 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 19840 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:58:29.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:58:29 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:58:29.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:58:29 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:58:29.860 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T04:58:29.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:58:29.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T04:58:29.863 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:58:29.869 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:58:29.869 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:58:29.870 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T04:58:29.870 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:58:29.870 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T04:58:29.871 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:58:29.871 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T04:58:29.872 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:58:29.872 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T04:58:29.873 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:58:29.873 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T04:58:29.873 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:58:29.874 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:58:29.874 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:58:29.875 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:58:29 smithi115 ceph-mon[78593]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T04:58:29.875 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:58:29 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:58:30.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:58:29 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:58:30.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:58:29 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:58:30.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T04:58:30.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:58:30.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T04:58:30.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:58:30.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:58:30.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:58:30.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T04:58:30.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:58:30.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T04:58:30.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:58:30.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T04:58:30.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:58:30.052 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T04:58:30.053 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:58:30.053 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T04:58:30.053 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:58:30.053 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:58:30.054 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:58:30.054 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:58:29 smithi047 ceph-mon[77147]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T04:58:30.054 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:58:29 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:58:30.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:58:29 smithi115 conmon[90393]: 2022-05-14T04:58:29.858+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:58:30.861 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:58:30 smithi115 conmon[90393]: 2022-05-14T04:58:30.809+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:58:30.862 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:58:30 smithi115 ceph-mon[78593]: pgmap v10272: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:58:30.862 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:58:30 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:58:30.863 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:58:30 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:58:31.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:58:30 smithi047 ceph-mon[77147]: pgmap v10272: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:58:31.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:58:30 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:58:31.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:58:30 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:58:31.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:58:31 smithi047 conmon[78521]: 2022-05-14T04:58:31.152+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:58:31.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:58:31 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T04:58:31.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:58:31 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:58:31.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:58:31 smithi047 conmon[78521]: yield 2022-05-14T04:58:31.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:58:31 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:58:31.547 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:58:31 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:58:31.547 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:58:31 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:58:31.548 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:58:31 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T04:58:31.548 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:58:31 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:58:31.548 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:58:31 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T04:58:31.549 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:58:31 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:58:31.549 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:58:31 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T04:58:31.549 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:58:31 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:58:31.550 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:58:31 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T04:58:31.550 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:58:31 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:58:31.550 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:58:31 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T04:58:31.551 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:58:31 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:58:31.551 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:58:31 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:58:31.551 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:58:31 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:58:31.551 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:58:31 smithi047 conmon[78521]: 2022-05-14T04:58:31.153+0000 7f5344781700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T04:58:31.552 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:58:31 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T04:58:31.552 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:58:31 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:58:31.552 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:58:31 smithi047 conmon[78521]: yield 2022-05-14T04:58:31.552 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:58:31 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:58:31.553 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:58:31 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:58:31.553 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:58:31 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:58:31.554 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:58:31 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T04:58:31.554 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:58:31 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:58:31.554 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:58:31 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T04:58:31.554 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:58:31 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:58:31.555 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:58:31 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T04:58:31.555 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:58:31 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:58:31.555 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:58:31 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T04:58:31.556 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:58:31 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:58:31.556 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:58:31 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T04:58:31.556 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:58:31 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:58:31.557 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:58:31 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:58:31.557 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:58:31 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:58:31.557 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:58:31 smithi047 conmon[78521]: 2022-05-14T04:58:31.559 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:58:31 smithi047 conmon[78521]: During handling of the above exception, another exception occurred: 2022-05-14T04:58:31.560 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:58:31 smithi047 conmon[78521]: 2022-05-14T04:58:31.560 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:58:31 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T04:58:31.560 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:58:31 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T04:58:31.560 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:58:31 smithi047 conmon[78521]: return f(*arg) 2022-05-14T04:58:31.561 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:58:31 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T04:58:31.561 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:58:31 smithi047 conmon[78521]: self._write_client_files(client_files, host) 2022-05-14T04:58:31.561 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:58:31 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T04:58:31.562 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:58:31 smithi047 conmon[78521]: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T04:58:31.562 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:58:31 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T04:58:31.562 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:58:31 smithi047 conmon[78521]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T04:58:31.563 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:58:31 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T04:58:31.563 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:58:31 smithi047 conmon[78521]: return self.event_loop.get_result(coro) 2022-05-14T04:58:31.563 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:58:31 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T04:58:31.563 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:58:31 smithi047 conmon[78521]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T04:58:31.563 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:58:31 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T04:58:31.564 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:58:31 smithi047 conmon[78521]: return self.__get_result() 2022-05-14T04:58:31.564 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:58:31 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T04:58:31.564 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:58:31 smithi047 conmon[78521]: raise self._exception 2022-05-14T04:58:31.564 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:58:31 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T04:58:31.565 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:58:31 smithi047 conmon[78521]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T04:58:31.568 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:58:31 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T04:58:31.568 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:58:31 smithi047 conmon[78521]: conn = await self._remote_connection(host, addr) 2022-05-14T04:58:31.568 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:58:31 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T04:58:31.569 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:58:31 smithi047 conmon[78521]: raise 2022-05-14T04:58:31.569 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:58:31 smithi047 conmon[78521]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T04:58:31.569 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:58:31 smithi047 conmon[78521]: self.gen.throw(type, value, traceback) 2022-05-14T04:58:31.569 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:58:31 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T04:58:31.570 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:58:31 smithi047 conmon[78521]: raise OrchestratorError(msg) 2022-05-14T04:58:31.570 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:58:31 smithi047 conmon[78521]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:58:31.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:58:31 smithi115 conmon[90393]: 2022-05-14T04:58:31.820+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:58:31.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:58:31 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:58:31.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:58:31 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:58:32.048 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:58:31 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:58:32.049 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:58:31 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:58:32.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:58:32 smithi115 conmon[90393]: 2022-05-14T04:58:32.796+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:58:32.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:58:32 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:58:32.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T04:58:32.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:58:32.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T04:58:32.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:58:32.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:58:32.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:58:32.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T04:58:32.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:58:32.863 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T04:58:32.864 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:58:32.864 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T04:58:32.864 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:58:32.864 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T04:58:32.864 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:58:32.865 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T04:58:32.865 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:58:32.865 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:58:32.865 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:58:32.866 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:58:32 smithi115 ceph-mon[78593]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T04:58:32.866 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T04:58:32.866 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:58:32.866 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T04:58:32.866 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:58:32.867 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:58:32.867 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:58:32.867 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T04:58:32.867 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:58:32.868 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T04:58:32.868 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:58:32.868 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T04:58:32.869 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:58:32.869 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T04:58:32.870 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:58:32.870 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T04:58:32.870 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:58:32.870 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:58:32.870 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:58:32.871 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T04:58:32.871 INFO:journalctl@ceph.mon.c.smithi115.stdout: During handling of the above exception, another exception occurred: 2022-05-14T04:58:32.871 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T04:58:32.871 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T04:58:32.872 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T04:58:32.872 INFO:journalctl@ceph.mon.c.smithi115.stdout: return f(*arg) 2022-05-14T04:58:32.872 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T04:58:32.872 INFO:journalctl@ceph.mon.c.smithi115.stdout: self._write_client_files(client_files, host) 2022-05-14T04:58:32.873 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T04:58:32.873 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T04:58:32.874 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T04:58:32.875 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T04:58:32.875 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T04:58:32.875 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.event_loop.get_result(coro) 2022-05-14T04:58:32.875 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T04:58:32.876 INFO:journalctl@ceph.mon.c.smithi115.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T04:58:32.876 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T04:58:32.876 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.__get_result() 2022-05-14T04:58:32.876 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T04:58:32.876 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise self._exception 2022-05-14T04:58:32.877 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T04:58:32.877 INFO:journalctl@ceph.mon.c.smithi115.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T04:58:32.877 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T04:58:32.877 INFO:journalctl@ceph.mon.c.smithi115.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T04:58:32.878 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T04:58:32.878 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise 2022-05-14T04:58:32.878 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T04:58:32.878 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.gen.throw(type, value, traceback) 2022-05-14T04:58:32.878 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T04:58:32.879 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OrchestratorError(msg) 2022-05-14T04:58:32.879 INFO:journalctl@ceph.mon.c.smithi115.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:58:32.879 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:58:32 smithi115 ceph-mon[78593]: pgmap v10273: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:58:32.879 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:58:32 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:58:33.047 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:58:32 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:58:33.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T04:58:33.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:58:33.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T04:58:33.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:58:33.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:58:33.049 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:58:33.049 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T04:58:33.049 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:58:33.049 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T04:58:33.050 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:58:33.050 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T04:58:33.050 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:58:33.050 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T04:58:33.056 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:58:33.056 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T04:58:33.057 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:58:33.057 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:58:33.057 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:58:33.058 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:58:32 smithi047 ceph-mon[77147]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T04:58:33.058 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T04:58:33.058 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:58:33.058 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T04:58:33.058 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:58:33.059 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:58:33.059 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:58:33.059 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T04:58:33.059 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:58:33.060 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T04:58:33.060 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:58:33.060 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T04:58:33.060 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:58:33.061 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T04:58:33.061 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:58:33.061 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T04:58:33.061 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:58:33.061 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:58:33.062 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:58:33.062 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T04:58:33.062 INFO:journalctl@ceph.mon.b.smithi047.stdout: During handling of the above exception, another exception occurred: 2022-05-14T04:58:33.062 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T04:58:33.063 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T04:58:33.063 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T04:58:33.063 INFO:journalctl@ceph.mon.b.smithi047.stdout: return f(*arg) 2022-05-14T04:58:33.063 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T04:58:33.064 INFO:journalctl@ceph.mon.b.smithi047.stdout: self._write_client_files(client_files, host) 2022-05-14T04:58:33.064 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T04:58:33.064 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T04:58:33.064 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T04:58:33.064 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T04:58:33.065 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T04:58:33.065 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.event_loop.get_result(coro) 2022-05-14T04:58:33.065 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T04:58:33.065 INFO:journalctl@ceph.mon.b.smithi047.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T04:58:33.065 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T04:58:33.066 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.__get_result() 2022-05-14T04:58:33.066 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T04:58:33.066 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise self._exception 2022-05-14T04:58:33.067 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T04:58:33.067 INFO:journalctl@ceph.mon.b.smithi047.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T04:58:33.067 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T04:58:33.067 INFO:journalctl@ceph.mon.b.smithi047.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T04:58:33.067 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T04:58:33.068 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise 2022-05-14T04:58:33.068 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T04:58:33.068 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.gen.throw(type, value, traceback) 2022-05-14T04:58:33.068 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T04:58:33.069 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OrchestratorError(msg) 2022-05-14T04:58:33.070 INFO:journalctl@ceph.mon.b.smithi047.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:58:33.070 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:58:32 smithi047 ceph-mon[77147]: pgmap v10273: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:58:33.070 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:58:32 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:58:33.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:58:33 smithi115 conmon[90393]: 2022-05-14T04:58:33.838+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:58:33.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:58:33 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:58:33.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:58:33 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 19845 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:58:34.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:58:33 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:58:34.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:58:33 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 19845 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:58:34.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:58:34 smithi115 ceph-mon[78593]: pgmap v10274: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:58:34.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:58:34 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:58:34.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:58:34 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:58:35.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:58:34 smithi047 ceph-mon[77147]: pgmap v10274: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:58:35.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:58:34 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:58:35.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:58:34 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:58:35.361 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:58:34 smithi115 conmon[90393]: 2022-05-14T04:58:34.872+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:58:35.886 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:58:35 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:58:36.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:58:35 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:58:36.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:58:35 smithi115 conmon[90393]: 2022-05-14T04:58:35.883+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:58:36.902 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:58:36 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:58:36.902 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:58:36 smithi115 ceph-mon[78593]: pgmap v10275: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:58:37.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:58:36 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:58:37.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:58:36 smithi047 ceph-mon[77147]: pgmap v10275: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:58:37.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:58:36 smithi115 conmon[90393]: 2022-05-14T04:58:36.899+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:58:37.889 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:58:37 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:58:38.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:58:37 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:58:38.361 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:58:37 smithi115 conmon[90393]: 2022-05-14T04:58:37.885+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:58:39.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:58:38 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:58:39.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:58:38 smithi047 ceph-mon[77147]: pgmap v10276: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:58:39.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:58:38 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:58:39.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:58:38 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:58:39.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:58:38 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 19850 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:58:39.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:58:38 smithi115 conmon[90393]: 2022-05-14T04:58:38.850+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:58:39.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:58:38 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:58:39.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:58:38 smithi115 ceph-mon[78593]: pgmap v10276: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:58:39.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:58:38 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:58:39.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:58:38 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:58:39.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:58:38 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 19850 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:58:39.876 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:58:39 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:58:40.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:58:39 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:58:40.362 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:58:39 smithi115 conmon[90393]: 2022-05-14T04:58:39.873+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:58:41.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:58:40 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:58:41.045 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:58:40 smithi047 ceph-mon[77147]: pgmap v10277: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:58:41.110 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:58:40 smithi115 conmon[90393]: 2022-05-14T04:58:40.867+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:58:41.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:58:40 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:58:41.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:58:40 smithi115 ceph-mon[78593]: pgmap v10277: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:58:42.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:58:41 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:58:42.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:58:41 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:58:42.117 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:58:41 smithi115 conmon[90393]: 2022-05-14T04:58:41.825+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:58:43.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:58:42 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:58:43.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:58:42 smithi047 ceph-mon[77147]: pgmap v10278: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:58:43.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:58:42 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:58:43.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:58:42 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:58:43.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:58:42 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:58:43.110 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:58:42 smithi115 conmon[90393]: 2022-05-14T04:58:42.823+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:58:43.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:58:42 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:58:43.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:58:42 smithi115 ceph-mon[78593]: pgmap v10278: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:58:43.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:58:42 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:58:43.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:58:42 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:58:43.112 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:58:42 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:58:44.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:58:43 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:58:44.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:58:43 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 19855 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:58:44.118 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:58:43 smithi115 conmon[90393]: 2022-05-14T04:58:43.776+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:58:44.119 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:58:43 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:58:44.120 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:58:43 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 19855 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:58:45.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:58:44 smithi047 ceph-mon[77147]: pgmap v10279: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:58:45.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:58:44 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:58:45.115 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:58:44 smithi115 conmon[90393]: 2022-05-14T04:58:44.789+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:58:45.116 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:58:44 smithi115 ceph-mon[78593]: pgmap v10279: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:58:45.117 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:58:44 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:58:46.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:58:45 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:58:46.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:58:45 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:58:46.110 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:58:45 smithi115 conmon[90393]: 2022-05-14T04:58:45.763+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:58:47.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:58:46 smithi047 ceph-mon[77147]: pgmap v10280: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:58:47.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:58:46 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:58:47.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:58:46 smithi115 conmon[90393]: 2022-05-14T04:58:46.776+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:58:47.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:58:46 smithi115 ceph-mon[78593]: pgmap v10280: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:58:47.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:58:46 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:58:48.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:58:47 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:58:48.267 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:58:47 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:58:48.268 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:58:47 smithi115 conmon[90393]: 2022-05-14T04:58:47.820+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:58:49.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:58:48 smithi047 ceph-mon[77147]: pgmap v10281: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:58:49.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:58:48 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:58:49.045 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:58:48 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 19860 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:58:49.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:58:48 smithi115 conmon[90393]: 2022-05-14T04:58:48.812+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:58:49.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:58:48 smithi115 ceph-mon[78593]: pgmap v10281: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:58:49.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:58:48 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:58:49.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:58:48 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 19860 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:58:50.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:58:49 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:58:50.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:58:49 smithi115 conmon[90393]: 2022-05-14T04:58:49.795+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:58:50.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:58:49 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:58:51.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:58:50 smithi047 ceph-mon[77147]: pgmap v10282: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:58:51.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:58:50 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:58:51.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:58:50 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:58:51.045 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:58:50 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:58:51.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:58:50 smithi115 conmon[90393]: 2022-05-14T04:58:50.795+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:58:51.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:58:50 smithi115 ceph-mon[78593]: pgmap v10282: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:58:51.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:58:50 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:58:51.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:58:50 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:58:51.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:58:50 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:58:52.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:58:51 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:58:52.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:58:51 smithi115 conmon[90393]: 2022-05-14T04:58:51.797+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:58:52.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:58:51 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:58:53.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:58:52 smithi047 ceph-mon[77147]: pgmap v10283: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:58:53.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:58:52 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:58:53.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:58:52 smithi115 ceph-mon[78593]: pgmap v10283: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:58:53.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:58:52 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:58:53.111 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:58:52 smithi115 conmon[90393]: 2022-05-14T04:58:52.768+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:58:54.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:58:53 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:58:54.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:58:53 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 19865 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:58:54.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:58:53 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:58:54.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:58:53 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 19865 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:58:54.110 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:58:53 smithi115 conmon[90393]: 2022-05-14T04:58:53.800+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:58:55.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:58:54 smithi047 ceph-mon[77147]: pgmap v10284: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:58:55.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:58:54 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:58:55.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:58:54 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:58:55.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:58:54 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:58:55.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:58:54 smithi115 conmon[90393]: 2022-05-14T04:58:54.827+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:58:55.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:58:54 smithi115 ceph-mon[78593]: pgmap v10284: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:58:55.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:58:54 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:58:55.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:58:54 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:58:55.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:58:54 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:58:56.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:58:55 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:58:56.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:58:55 smithi115 conmon[90393]: 2022-05-14T04:58:55.792+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:58:56.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:58:55 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:58:57.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:58:56 smithi047 ceph-mon[77147]: pgmap v10285: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:58:57.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:58:56 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:58:57.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:58:56 smithi115 conmon[90393]: 2022-05-14T04:58:56.839+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:58:57.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:58:56 smithi115 ceph-mon[78593]: pgmap v10285: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:58:57.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:58:56 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:58:58.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:58:57 smithi115 conmon[90393]: 2022-05-14T04:58:57.836+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:58:59.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:58:58 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:58:59.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:58:58 smithi047 ceph-mon[77147]: pgmap v10286: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:58:59.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:58:58 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:58:59.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:58:58 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:58:59.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:58:58 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 19870 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:58:59.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:58:58 smithi115 conmon[90393]: 2022-05-14T04:58:58.793+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:58:59.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:58:58 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:58:59.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:58:58 smithi115 ceph-mon[78593]: pgmap v10286: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:58:59.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:58:58 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:58:59.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:58:58 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:58:59.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:58:58 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 19870 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:59:00.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:58:59 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:59:00.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:58:59 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:59:00.110 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:58:59 smithi115 conmon[90393]: 2022-05-14T04:58:59.815+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:59:00.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:58:59 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:59:00.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:58:59 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:59:01.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:59:00 smithi047 ceph-mon[77147]: pgmap v10287: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:59:01.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:59:00 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:59:01.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:59:00 smithi115 conmon[90393]: 2022-05-14T04:59:00.847+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:59:01.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:59:00 smithi115 ceph-mon[78593]: pgmap v10287: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:59:01.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:59:00 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:59:02.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:59:01 smithi115 conmon[90393]: 2022-05-14T04:59:01.855+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:59:03.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:59:02 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:59:03.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:59:02 smithi047 ceph-mon[77147]: pgmap v10288: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:59:03.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:59:02 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:59:03.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:59:02 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:59:03.110 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:59:02 smithi115 conmon[90393]: 2022-05-14T04:59:02.881+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:59:03.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:59:02 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:59:03.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:59:02 smithi115 ceph-mon[78593]: pgmap v10288: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:59:03.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:59:02 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:59:03.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:59:02 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:59:04.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:59:03 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:59:04.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:59:03 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 19875 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:59:04.110 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:59:03 smithi115 conmon[90393]: 2022-05-14T04:59:03.875+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:59:04.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:59:03 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:59:04.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:59:03 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 19875 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:59:05.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:59:04 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:59:05.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:59:04 smithi047 ceph-mon[77147]: pgmap v10289: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:59:05.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:59:04 smithi115 conmon[90393]: 2022-05-14T04:59:04.834+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:59:05.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:59:04 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:59:05.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:59:04 smithi115 ceph-mon[78593]: pgmap v10289: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:59:06.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:59:05 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:59:06.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:59:05 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:59:06.110 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:59:05 smithi115 conmon[90393]: 2022-05-14T04:59:05.863+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:59:07.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:59:06 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:59:07.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:59:06 smithi047 ceph-mon[77147]: pgmap v10290: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:59:07.113 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:59:06 smithi115 conmon[90393]: 2022-05-14T04:59:06.853+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:59:07.114 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:59:06 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:59:07.115 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:59:06 smithi115 ceph-mon[78593]: pgmap v10290: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:59:08.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:59:07 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:59:08.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:59:07 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:59:08.110 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:59:07 smithi115 conmon[90393]: 2022-05-14T04:59:07.840+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:59:09.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:59:08 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:59:09.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:59:08 smithi047 ceph-mon[77147]: pgmap v10291: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:59:09.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:59:08 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 19880 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:59:09.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:59:08 smithi115 conmon[90393]: 2022-05-14T04:59:08.811+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:59:09.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:59:08 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:59:09.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:59:08 smithi115 ceph-mon[78593]: pgmap v10291: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:59:09.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:59:08 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 19880 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:59:10.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:59:09 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:59:10.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:59:09 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:59:10.110 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:59:09 smithi115 conmon[90393]: 2022-05-14T04:59:09.811+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:59:10.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:59:09 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:59:10.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:59:09 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:59:11.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:59:10 smithi047 ceph-mon[77147]: pgmap v10292: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:59:11.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:59:10 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:59:11.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:59:10 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:59:11.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:59:10 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:59:11.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:59:10 smithi115 conmon[90393]: 2022-05-14T04:59:10.830+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:59:11.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:59:10 smithi115 ceph-mon[78593]: pgmap v10292: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:59:11.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:59:10 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:59:11.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:59:10 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:59:11.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:59:10 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:59:12.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:59:11 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:59:12.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:59:11 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:59:12.110 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:59:11 smithi115 conmon[90393]: 2022-05-14T04:59:11.841+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:59:13.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:59:12 smithi047 ceph-mon[77147]: pgmap v10293: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:59:13.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:59:12 smithi115 conmon[90393]: 2022-05-14T04:59:12.864+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:59:13.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:59:12 smithi115 ceph-mon[78593]: pgmap v10293: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:59:14.046 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:59:13 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:59:14.046 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:59:13 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 19885 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:59:14.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:59:13 smithi115 conmon[90393]: 2022-05-14T04:59:13.875+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:59:14.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:59:13 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:59:14.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:59:13 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 19885 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:59:15.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:59:14 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:59:15.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:59:14 smithi047 ceph-mon[77147]: pgmap v10294: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:59:15.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:59:14 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:59:15.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:59:14 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:59:15.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:59:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T04:59:15.110 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:59:14 smithi115 conmon[90393]: 2022-05-14T04:59:14.840+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:59:15.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:59:14 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:59:15.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:59:14 smithi115 ceph-mon[78593]: pgmap v10294: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:59:15.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:59:14 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:59:15.112 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:59:14 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:59:15.112 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:59:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T04:59:16.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:59:15 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:59:16.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:59:15 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T04:59:16.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:59:15 smithi115 conmon[90393]: 2022-05-14T04:59:15.889+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:59:16.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:59:15 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:59:16.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:59:15 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T04:59:17.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:59:16 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:59:17.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:59:16 smithi047 ceph-mon[77147]: pgmap v10295: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:59:17.110 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:59:16 smithi115 conmon[90393]: 2022-05-14T04:59:16.909+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:59:17.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:59:16 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:59:17.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:59:16 smithi115 ceph-mon[78593]: pgmap v10295: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:59:18.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:59:17 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:59:18.110 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:59:17 smithi115 conmon[90393]: 2022-05-14T04:59:17.912+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:59:18.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:59:17 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:59:19.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:59:18 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:59:19.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:59:18 smithi047 ceph-mon[77147]: pgmap v10296: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:59:19.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:59:18 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:59:19.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:59:18 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:59:19.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:59:18 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 19890 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:59:19.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:59:18 smithi115 conmon[90393]: 2022-05-14T04:59:18.901+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:59:19.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:59:18 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:59:19.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:59:18 smithi115 ceph-mon[78593]: pgmap v10296: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:59:19.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:59:18 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:59:19.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:59:18 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:59:19.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:59:18 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 19890 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:59:20.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:59:19 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:59:20.119 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:59:19 smithi115 conmon[90393]: 2022-05-14T04:59:19.928+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:59:20.120 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:59:19 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:59:21.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:59:20 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:59:21.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:59:20 smithi047 ceph-mon[77147]: pgmap v10297: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:59:21.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:59:20 smithi115 conmon[90393]: 2022-05-14T04:59:20.895+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:59:21.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:59:20 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:59:21.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:59:20 smithi115 ceph-mon[78593]: pgmap v10297: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:59:22.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:59:21 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:59:22.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:59:21 smithi115 conmon[90393]: 2022-05-14T04:59:21.855+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:59:22.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:59:21 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:59:23.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:59:22 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:59:23.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:59:22 smithi047 ceph-mon[77147]: pgmap v10298: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:59:23.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:59:22 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:59:23.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:59:22 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:59:23.114 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:59:22 smithi115 conmon[90393]: 2022-05-14T04:59:22.841+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:59:23.115 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:59:22 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:59:23.115 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:59:22 smithi115 ceph-mon[78593]: pgmap v10298: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:59:23.115 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:59:22 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:59:23.115 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:59:22 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:59:24.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:59:23 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:59:24.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:59:23 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 19895 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:59:24.114 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:59:23 smithi115 conmon[90393]: 2022-05-14T04:59:23.874+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:59:24.115 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:59:23 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:59:24.115 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:59:23 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 19895 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:59:25.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:59:24 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:59:25.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:59:24 smithi047 ceph-mon[77147]: pgmap v10299: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:59:25.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:59:24 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:59:25.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:59:24 smithi115 ceph-mon[78593]: pgmap v10299: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:59:25.110 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:59:24 smithi115 conmon[90393]: 2022-05-14T04:59:24.858+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:59:26.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:59:25 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:59:26.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:59:25 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:59:26.110 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:59:25 smithi115 conmon[90393]: 2022-05-14T04:59:25.849+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:59:27.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:59:26 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:59:27.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:59:26 smithi047 ceph-mon[77147]: pgmap v10300: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:59:27.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:59:26 smithi115 conmon[90393]: 2022-05-14T04:59:26.873+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:59:27.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:59:26 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:59:27.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:59:26 smithi115 ceph-mon[78593]: pgmap v10300: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:59:28.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:59:27 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:59:28.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:59:27 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:59:28.110 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:59:27 smithi115 conmon[90393]: 2022-05-14T04:59:27.830+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:59:29.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:59:28 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:59:29.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:59:28 smithi047 ceph-mon[77147]: pgmap v10301: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:59:29.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:59:28 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:59:29.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:59:28 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 19900 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:59:29.110 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:59:28 smithi115 conmon[90393]: 2022-05-14T04:59:28.866+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:59:29.112 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:59:28 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:59:29.112 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:59:28 smithi115 ceph-mon[78593]: pgmap v10301: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:59:29.113 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:59:28 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:59:29.113 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:59:28 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 19900 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:59:30.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:59:29 smithi115 conmon[90393]: 2022-05-14T04:59:29.896+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:59:31.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:59:30 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:59:31.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:59:30 smithi047 ceph-mon[77147]: pgmap v10302: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:59:31.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:59:30 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:59:31.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:59:30 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:59:31.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:59:30 smithi115 conmon[90393]: 2022-05-14T04:59:30.873+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:59:31.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:59:30 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:59:31.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:59:30 smithi115 ceph-mon[78593]: pgmap v10302: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:59:31.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:59:30 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:59:31.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:59:30 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:59:31.916 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:59:31 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:59:31.916 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:59:31 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T04:59:31.916 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:59:31 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T04:59:31.916 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:59:31 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T04:59:32.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:59:31 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:59:32.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:59:31 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T04:59:32.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:59:31 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T04:59:32.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:59:31 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T04:59:32.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:59:31 smithi115 conmon[90393]: 2022-05-14T04:59:31.912+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:59:33.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:59:32 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:59:33.054 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:59:32 smithi047 ceph-mon[77147]: pgmap v10303: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:59:33.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:59:32 smithi115 conmon[90393]: 2022-05-14T04:59:32.864+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:59:33.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:59:32 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:59:33.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:59:32 smithi115 ceph-mon[78593]: pgmap v10303: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:59:34.046 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:59:33 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:59:34.047 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:59:33 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 19905 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:59:34.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:59:33 smithi115 conmon[90393]: 2022-05-14T04:59:33.853+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:59:34.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:59:33 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:59:34.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:59:33 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 19905 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:59:34.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:59:34 smithi047 conmon[78521]: 2022-05-14T04:59:34.255+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:59:34.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:59:34 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T04:59:34.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:59:34 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:59:34.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:59:34 smithi047 conmon[78521]: yield 2022-05-14T04:59:34.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:59:34 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:59:34.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:59:34 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:59:34.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:59:34 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:59:34.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:59:34 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T04:59:34.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:59:34 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:59:34.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:59:34 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T04:59:34.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:59:34 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:59:34.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:59:34 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T04:59:34.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:59:34 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:59:34.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:59:34 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T04:59:34.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:59:34 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:59:34.547 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:59:34 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T04:59:34.547 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:59:34 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:59:34.547 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:59:34 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:59:34.547 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:59:34 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:59:35.045 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:59:34 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:59:35.046 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:59:34 smithi047 ceph-mon[77147]: pgmap v10304: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:59:35.046 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:59:34 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:59:35.046 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:59:34 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:59:35.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:59:34 smithi115 conmon[90393]: 2022-05-14T04:59:34.826+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:59:35.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:59:34 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:59:35.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:59:34 smithi115 ceph-mon[78593]: pgmap v10304: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:59:35.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:59:34 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:59:35.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:59:34 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:59:36.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:59:35 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:59:36.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:59:35 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:59:36.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T04:59:36.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:59:36.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T04:59:36.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:59:36.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:59:36.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:59:36.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T04:59:36.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:59:36.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T04:59:36.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:59:36.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T04:59:36.049 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:59:36.049 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T04:59:36.049 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:59:36.050 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T04:59:36.050 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:59:36.051 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:59:36.051 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:59:36.051 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:59:35 smithi047 ceph-mon[77147]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T04:59:36.052 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:59:35 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:59:36.111 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:59:35 smithi115 conmon[90393]: 2022-05-14T04:59:35.816+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:59:36.113 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:59:35 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:59:36.113 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:59:35 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:59:36.114 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T04:59:36.114 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:59:36.114 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T04:59:36.114 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:59:36.115 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:59:36.115 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:59:36.115 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T04:59:36.115 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:59:36.115 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T04:59:36.116 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:59:36.116 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T04:59:36.116 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:59:36.116 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T04:59:36.116 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:59:36.117 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T04:59:36.117 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:59:36.117 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:59:36.117 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:59:36.118 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:59:35 smithi115 ceph-mon[78593]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T04:59:36.118 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:59:35 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:59:37.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:59:36 smithi047 ceph-mon[77147]: pgmap v10305: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:59:37.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:59:36 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:59:37.110 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:59:36 smithi115 conmon[90393]: 2022-05-14T04:59:36.865+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:59:37.112 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:59:36 smithi115 ceph-mon[78593]: pgmap v10305: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:59:37.112 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:59:36 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:59:37.793 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:59:37 smithi047 conmon[78521]: 2022-05-14T04:59:37.328+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:59:37.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:59:37 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T04:59:37.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:59:37 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:59:37.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:59:37 smithi047 conmon[78521]: yield 2022-05-14T04:59:37.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:59:37 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:59:37.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:59:37 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:59:37.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:59:37 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:59:37.797 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:59:37 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T04:59:37.797 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:59:37 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:59:37.797 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:59:37 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T04:59:37.798 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:59:37 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:59:37.798 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:59:37 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T04:59:37.798 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:59:37 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:59:37.798 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:59:37 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T04:59:37.799 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:59:37 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:59:37.799 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:59:37 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T04:59:37.799 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:59:37 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:59:37.799 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:59:37 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:59:37.799 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:59:37 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:59:37.802 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:59:37 smithi047 conmon[78521]: 2022-05-14T04:59:37.328+0000 7f5340779700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T04:59:37.802 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:59:37 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T04:59:37.802 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:59:37 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:59:37.802 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:59:37 smithi047 conmon[78521]: yield 2022-05-14T04:59:37.803 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:59:37 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:59:37.803 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:59:37 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:59:37.803 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:59:37 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:59:37.803 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:59:37 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T04:59:37.804 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:59:37 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:59:37.804 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:59:37 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T04:59:37.804 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:59:37 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:59:37.804 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:59:37 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T04:59:37.804 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:59:37 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:59:37.805 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:59:37 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T04:59:37.805 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:59:37 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:59:37.805 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:59:37 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T04:59:37.805 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:59:37 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:59:37.806 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:59:37 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:59:37.806 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:59:37 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:59:37.806 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:59:37 smithi047 conmon[78521]: 2022-05-14T04:59:37.806 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:59:37 smithi047 conmon[78521]: During handling of the above exception, another exception occurred: 2022-05-14T04:59:37.806 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:59:37 smithi047 conmon[78521]: 2022-05-14T04:59:37.807 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:59:37 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T04:59:37.807 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:59:37 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T04:59:37.807 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:59:37 smithi047 conmon[78521]: return f(*arg) 2022-05-14T04:59:37.807 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:59:37 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T04:59:37.808 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:59:37 smithi047 conmon[78521]: self._write_client_files(client_files, host) 2022-05-14T04:59:37.808 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:59:37 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T04:59:37.808 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:59:37 smithi047 conmon[78521]: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T04:59:37.808 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:59:37 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T04:59:37.808 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:59:37 smithi047 conmon[78521]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T04:59:37.809 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:59:37 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T04:59:37.809 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:59:37 smithi047 conmon[78521]: return self.event_loop.get_result(coro) 2022-05-14T04:59:37.809 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:59:37 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T04:59:37.809 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:59:37 smithi047 conmon[78521]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T04:59:37.810 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:59:37 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T04:59:37.810 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:59:37 smithi047 conmon[78521]: return self.__get_result() 2022-05-14T04:59:37.810 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:59:37 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T04:59:37.810 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:59:37 smithi047 conmon[78521]: raise self._exception 2022-05-14T04:59:37.811 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:59:37 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T04:59:37.811 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:59:37 smithi047 conmon[78521]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T04:59:37.811 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:59:37 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T04:59:37.812 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:59:37 smithi047 conmon[78521]: conn = await self._remote_connection(host, addr) 2022-05-14T04:59:37.812 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:59:37 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T04:59:37.812 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:59:37 smithi047 conmon[78521]: raise 2022-05-14T04:59:37.813 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:59:37 smithi047 conmon[78521]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T04:59:37.813 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:59:37 smithi047 conmon[78521]: self.gen.throw(type, value, traceback) 2022-05-14T04:59:37.813 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:59:37 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T04:59:37.813 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:59:37 smithi047 conmon[78521]: raise OrchestratorError(msg) 2022-05-14T04:59:37.813 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 04:59:37 smithi047 conmon[78521]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:59:38.368 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:59:37 smithi115 conmon[90393]: 2022-05-14T04:59:37.866+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:59:39.111 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:59:38 smithi115 conmon[90393]: 2022-05-14T04:59:38.876+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:59:39.112 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:59:38 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:59:39.113 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:59:38 smithi115 ceph-mon[78593]: pgmap v10306: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:59:39.113 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:59:38 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:59:39.113 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T04:59:39.114 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:59:39.115 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T04:59:39.115 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:59:39.116 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:59:39.116 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:59:39.116 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T04:59:39.116 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:59:39.117 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T04:59:39.117 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:59:39.118 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T04:59:39.118 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:59:39.118 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T04:59:39.118 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:59:39.119 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T04:59:39.119 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:59:39.119 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:59:39.119 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:59:39.120 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:59:38 smithi115 ceph-mon[78593]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T04:59:39.120 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T04:59:39.120 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:59:39.120 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T04:59:39.121 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:59:39.121 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:59:39.121 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:59:39.121 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T04:59:39.122 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:59:39.122 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T04:59:39.123 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:59:39.123 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T04:59:39.123 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:59:39.123 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T04:59:39.123 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:59:39.124 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T04:59:39.124 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:59:39.124 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:59:39.124 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:59:39.124 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T04:59:39.125 INFO:journalctl@ceph.mon.c.smithi115.stdout: During handling of the above exception, another exception occurred: 2022-05-14T04:59:39.125 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T04:59:39.125 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T04:59:39.125 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T04:59:39.126 INFO:journalctl@ceph.mon.c.smithi115.stdout: return f(*arg) 2022-05-14T04:59:39.126 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T04:59:39.126 INFO:journalctl@ceph.mon.c.smithi115.stdout: self._write_client_files(client_files, host) 2022-05-14T04:59:39.127 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T04:59:39.127 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T04:59:39.127 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T04:59:39.127 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T04:59:39.128 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T04:59:39.128 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.event_loop.get_result(coro) 2022-05-14T04:59:39.128 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T04:59:39.128 INFO:journalctl@ceph.mon.c.smithi115.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T04:59:39.129 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T04:59:39.129 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.__get_result() 2022-05-14T04:59:39.129 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T04:59:39.129 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise self._exception 2022-05-14T04:59:39.130 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T04:59:39.130 INFO:journalctl@ceph.mon.c.smithi115.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T04:59:39.130 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T04:59:39.130 INFO:journalctl@ceph.mon.c.smithi115.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T04:59:39.131 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T04:59:39.131 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise 2022-05-14T04:59:39.132 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T04:59:39.132 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.gen.throw(type, value, traceback) 2022-05-14T04:59:39.132 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T04:59:39.133 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OrchestratorError(msg) 2022-05-14T04:59:39.133 INFO:journalctl@ceph.mon.c.smithi115.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:59:39.133 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:59:38 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:59:39.134 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:59:38 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:59:39.134 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:59:38 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 19910 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:59:39.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:59:38 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:59:39.340 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:59:38 smithi047 ceph-mon[77147]: pgmap v10306: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:59:39.340 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:59:38 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:59:39.341 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T04:59:39.341 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:59:39.341 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T04:59:39.341 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:59:39.342 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:59:39.342 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:59:39.342 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T04:59:39.342 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:59:39.342 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T04:59:39.343 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:59:39.343 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T04:59:39.343 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:59:39.343 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T04:59:39.344 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:59:39.344 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T04:59:39.344 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:59:39.344 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:59:39.345 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:59:39.345 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:59:38 smithi047 ceph-mon[77147]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T04:59:39.345 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T04:59:39.345 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T04:59:39.346 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T04:59:39.346 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T04:59:39.346 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T04:59:39.347 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T04:59:39.347 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T04:59:39.347 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T04:59:39.347 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T04:59:39.348 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T04:59:39.348 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T04:59:39.348 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T04:59:39.348 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T04:59:39.348 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T04:59:39.349 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T04:59:39.350 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T04:59:39.350 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T04:59:39.351 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:59:39.351 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T04:59:39.352 INFO:journalctl@ceph.mon.b.smithi047.stdout: During handling of the above exception, another exception occurred: 2022-05-14T04:59:39.352 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T04:59:39.353 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T04:59:39.353 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T04:59:39.354 INFO:journalctl@ceph.mon.b.smithi047.stdout: return f(*arg) 2022-05-14T04:59:39.354 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T04:59:39.354 INFO:journalctl@ceph.mon.b.smithi047.stdout: self._write_client_files(client_files, host) 2022-05-14T04:59:39.355 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T04:59:39.355 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T04:59:39.355 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T04:59:39.355 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T04:59:39.356 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T04:59:39.356 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.event_loop.get_result(coro) 2022-05-14T04:59:39.356 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T04:59:39.356 INFO:journalctl@ceph.mon.b.smithi047.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T04:59:39.357 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T04:59:39.357 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.__get_result() 2022-05-14T04:59:39.358 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T04:59:39.358 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise self._exception 2022-05-14T04:59:39.358 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T04:59:39.358 INFO:journalctl@ceph.mon.b.smithi047.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T04:59:39.358 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T04:59:39.359 INFO:journalctl@ceph.mon.b.smithi047.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T04:59:39.359 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T04:59:39.360 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise 2022-05-14T04:59:39.360 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T04:59:39.360 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.gen.throw(type, value, traceback) 2022-05-14T04:59:39.360 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T04:59:39.361 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OrchestratorError(msg) 2022-05-14T04:59:39.361 INFO:journalctl@ceph.mon.b.smithi047.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T04:59:39.361 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:59:38 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:59:39.362 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:59:38 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:59:39.362 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:59:38 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 19910 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:59:40.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:59:39 smithi115 conmon[90393]: 2022-05-14T04:59:39.874+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:59:40.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:59:39 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:59:40.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:59:39 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:59:41.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:59:40 smithi115 conmon[90393]: 2022-05-14T04:59:40.921+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:59:41.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:59:40 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:59:41.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:59:40 smithi115 ceph-mon[78593]: pgmap v10307: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:59:41.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:59:40 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:59:41.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:59:40 smithi047 ceph-mon[77147]: pgmap v10307: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:59:42.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:59:41 smithi115 conmon[90393]: 2022-05-14T04:59:41.891+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:59:42.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:59:41 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:59:42.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:59:41 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:59:43.114 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:59:42 smithi115 conmon[90393]: 2022-05-14T04:59:42.845+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:59:43.114 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:59:42 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:59:43.115 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:59:42 smithi115 ceph-mon[78593]: pgmap v10308: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:59:43.115 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:59:42 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:59:43.115 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:59:42 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:59:43.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:59:42 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:59:43.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:59:42 smithi047 ceph-mon[77147]: pgmap v10308: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:59:43.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:59:42 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:59:43.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:59:42 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:59:44.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:59:43 smithi115 conmon[90393]: 2022-05-14T04:59:43.854+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:59:44.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:59:43 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:59:44.112 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:59:43 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 19915 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:59:44.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:59:43 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:59:44.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:59:43 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 19915 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:59:45.110 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:59:44 smithi115 conmon[90393]: 2022-05-14T04:59:44.888+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:59:45.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:59:44 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:59:45.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:59:44 smithi115 ceph-mon[78593]: pgmap v10309: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:59:45.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:59:44 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:59:45.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:59:44 smithi047 ceph-mon[77147]: pgmap v10309: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:59:46.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:59:45 smithi115 conmon[90393]: 2022-05-14T04:59:45.870+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:59:46.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:59:45 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:59:46.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:59:45 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:59:47.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:59:46 smithi115 conmon[90393]: 2022-05-14T04:59:46.860+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:59:47.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:59:46 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:59:47.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:59:46 smithi115 ceph-mon[78593]: pgmap v10310: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:59:47.295 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:59:46 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:59:47.295 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:59:46 smithi047 ceph-mon[77147]: pgmap v10310: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:59:48.112 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:59:47 smithi115 conmon[90393]: 2022-05-14T04:59:47.867+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:59:48.112 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:59:47 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:59:48.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:59:47 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:59:49.116 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:59:48 smithi115 conmon[90393]: 2022-05-14T04:59:48.846+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:59:49.116 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:59:48 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:59:49.117 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:59:48 smithi115 ceph-mon[78593]: pgmap v10311: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:59:49.117 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:59:48 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 19920 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:59:49.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:59:48 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:59:49.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:59:48 smithi047 ceph-mon[77147]: pgmap v10311: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:59:49.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:59:48 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 19920 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:59:50.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:59:49 smithi115 conmon[90393]: 2022-05-14T04:59:49.834+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:59:50.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:59:49 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:59:50.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:59:49 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:59:51.111 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:59:50 smithi115 conmon[90393]: 2022-05-14T04:59:50.865+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:59:51.113 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:59:50 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:59:51.113 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:59:50 smithi115 ceph-mon[78593]: pgmap v10312: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:59:51.113 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:59:50 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:59:51.113 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:59:50 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:59:51.114 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:59:50 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:59:51.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:59:50 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:59:51.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:59:50 smithi047 ceph-mon[77147]: pgmap v10312: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:59:51.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:59:50 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:59:51.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:59:50 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:59:51.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:59:50 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:59:52.360 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:59:51 smithi115 conmon[90393]: 2022-05-14T04:59:51.857+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:59:53.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:59:52 smithi115 conmon[90393]: 2022-05-14T04:59:52.844+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:59:53.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:59:52 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:59:53.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:59:52 smithi115 ceph-mon[78593]: pgmap v10313: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:59:53.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:59:52 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:59:53.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:59:52 smithi047 ceph-mon[77147]: pgmap v10313: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:59:54.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:59:53 smithi115 conmon[90393]: 2022-05-14T04:59:53.842+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:59:54.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:59:53 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:59:54.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:59:53 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 19925 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:59:54.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:59:53 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:59:54.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:59:53 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 19925 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:59:55.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:59:54 smithi115 conmon[90393]: 2022-05-14T04:59:54.888+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:59:55.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:59:54 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:59:55.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:59:54 smithi115 ceph-mon[78593]: pgmap v10314: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:59:55.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:59:54 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:59:55.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:59:54 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:59:55.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:59:54 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:59:55.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:59:54 smithi047 ceph-mon[77147]: pgmap v10314: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:59:55.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:59:54 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:59:55.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:59:54 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:59:56.110 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:59:55 smithi115 conmon[90393]: 2022-05-14T04:59:55.936+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:59:56.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:59:55 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:59:56.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:59:55 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:59:57.110 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:59:56 smithi115 conmon[90393]: 2022-05-14T04:59:56.892+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:59:57.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:59:56 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:59:57.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:59:56 smithi115 ceph-mon[78593]: pgmap v10315: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:59:57.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:59:56 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:59:57.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:59:56 smithi047 ceph-mon[77147]: pgmap v10315: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T04:59:58.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:59:57 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:59:58.111 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:59:57 smithi115 conmon[90393]: 2022-05-14T04:59:57.939+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:59:58.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:59:57 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:59:59.110 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:59:58 smithi115 conmon[90393]: 2022-05-14T04:59:58.928+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T04:59:59.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:59:58 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:59:59.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:59:58 smithi115 ceph-mon[78593]: pgmap v10316: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:59:59.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:59:58 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:59:59.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:59:58 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:59:59.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:59:58 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 19930 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:59:59.112 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:59:58 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:59:59.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:59:58 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T04:59:59.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:59:58 smithi047 ceph-mon[77147]: pgmap v10316: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T04:59:59.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:59:58 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T04:59:59.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:59:58 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T04:59:59.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:59:58 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 19930 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T04:59:59.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:59:58 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:00:00.113 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 04:59:59 smithi115 conmon[90393]: 2022-05-14T04:59:59.918+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:00:00.114 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 04:59:59 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:00:00.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 04:59:59 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:00:00.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:00:00 smithi047 conmon[77123]: 2022-05-14T04:59:59.999+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : Health detail: HEALTH_ERR 1 filesystem is degraded; 1 mds daemon damaged; 1 MDSs are read only; 1 MDSs report slow metadata IOs; 1/3 mons down, quorum b,c; Reduced data availability: 1 pg inactive; 9 slow ops, oldest one blocked for 19935 sec, osd.5 has slow ops 2022-05-14T05:00:00.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:00:00 smithi047 conmon[77123]: 2022-05-14T04:59:59.999+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : [WRN] FS_DEGRADED: 1 filesystem is degraded 2022-05-14T05:00:00.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:00:00 smithi047 conmon[77123]: 2022-05-14T04:59:59.999+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : fs cephfs is degraded 2022-05-14T05:00:00.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:00:00 smithi047 conmon[77123]: 2022-05-14T04:59:59.999+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : [ERR] MDS_DAMAGE: 1 mds daemon damaged 2022-05-14T05:00:00.295 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:00:00 smithi047 conmon[77123]: 2022-05-14T04:59:59.999+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : fs cephfs mds.2 is damaged 2022-05-14T05:00:00.295 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:00:00 smithi047 conmon[77123]: 2022-05-14T04:59:59.999+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : [WRN] MDS_READ_ONLY: 1 MDSs are read only 2022-05-14T05:00:00.295 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:00:00 smithi047 conmon[77123]: 2022-05-14T04:59:59.999+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : mds.l(mds.0): MDS in read-only mode 2022-05-14T05:00:00.295 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:00:00 smithi047 conmon[77123]: 2022-05-14T04:59:59.999+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : [WRN] MDS_SLOW_METADATA_IO: 1 MDSs report slow metadata IOs 2022-05-14T05:00:00.295 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:00:00 smithi047 conmon[77123]: 2022-05-14T04:59:59.999+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : mds.l(mds.0): 4 slow metadata IOs are blocked > 180 secs, oldest blocked for 20234 secs 2022-05-14T05:00:00.296 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:00:00 smithi047 conmon[77123]: 2022-05-14T04:59:59.999+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : [WRN] MON_DOWN: 1/3 mons down, quorum b,c 2022-05-14T05:00:00.297 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:00:00 smithi047 conmon[77123]: 2022-05-14T05:00:00.000+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : mon.a (rank 0) addr [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] is down (out of quorum) 2022-05-14T05:00:00.298 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:00:00 smithi047 conmon[77123]: 2022-05-14T05:00:00.000+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : [WRN] PG_AVAILABILITY: Reduced data availability: 1 pg inactive 2022-05-14T05:00:00.298 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:00:00 smithi047 conmon[77123]: 2022-05-14T05:00:00.000+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : pg 2.1 is stuck inactive for 5h, current state unknown, last acting [] 2022-05-14T05:00:00.299 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:00:00 smithi047 conmon[77123]: 2022-05-14T05:00:00.000+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : [WRN] SLOW_OPS: 9 slow ops, oldest one blocked for 19935 sec, osd.5 has slow ops 2022-05-14T05:00:01.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:00:00 smithi115 conmon[90393]: 2022-05-14T05:00:00.891+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:00:01.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:00:00 smithi115 ceph-mon[78593]: pgmap v10317: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T05:00:01.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:00:00 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:00:01.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:00:00 smithi115 ceph-mon[78593]: Health detail: HEALTH_ERR 1 filesystem is degraded; 1 mds daemon damaged; 1 MDSs are read only; 1 MDSs report slow metadata IOs; 1/3 mons down, quorum b,c; Reduced data availability: 1 pg inactive; 9 slow ops, oldest one blocked for 19935 sec, osd.5 has slow ops 2022-05-14T05:00:01.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:00:00 smithi115 ceph-mon[78593]: [WRN] FS_DEGRADED: 1 filesystem is degraded 2022-05-14T05:00:01.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:00:00 smithi115 ceph-mon[78593]: fs cephfs is degraded 2022-05-14T05:00:01.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:00:00 smithi115 ceph-mon[78593]: [ERR] MDS_DAMAGE: 1 mds daemon damaged 2022-05-14T05:00:01.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:00:00 smithi115 ceph-mon[78593]: fs cephfs mds.2 is damaged 2022-05-14T05:00:01.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:00:00 smithi115 ceph-mon[78593]: [WRN] MDS_READ_ONLY: 1 MDSs are read only 2022-05-14T05:00:01.112 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:00:00 smithi115 ceph-mon[78593]: mds.l(mds.0): MDS in read-only mode 2022-05-14T05:00:01.112 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:00:00 smithi115 ceph-mon[78593]: [WRN] MDS_SLOW_METADATA_IO: 1 MDSs report slow metadata IOs 2022-05-14T05:00:01.112 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:00:00 smithi115 ceph-mon[78593]: mds.l(mds.0): 4 slow metadata IOs are blocked > 180 secs, oldest blocked for 20234 secs 2022-05-14T05:00:01.112 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:00:00 smithi115 ceph-mon[78593]: [WRN] MON_DOWN: 1/3 mons down, quorum b,c 2022-05-14T05:00:01.112 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:00:00 smithi115 ceph-mon[78593]: mon.a (rank 0) addr [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] is down (out of quorum) 2022-05-14T05:00:01.113 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:00:00 smithi115 ceph-mon[78593]: [WRN] PG_AVAILABILITY: Reduced data availability: 1 pg inactive 2022-05-14T05:00:01.113 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:00:00 smithi115 ceph-mon[78593]: pg 2.1 is stuck inactive for 5h, current state unknown, last acting [] 2022-05-14T05:00:01.123 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:00:00 smithi115 ceph-mon[78593]: [WRN] SLOW_OPS: 9 slow ops, oldest one blocked for 19935 sec, osd.5 has slow ops 2022-05-14T05:00:01.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:00:00 smithi047 ceph-mon[77147]: pgmap v10317: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T05:00:01.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:00:00 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:00:01.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:00:00 smithi047 ceph-mon[77147]: Health detail: HEALTH_ERR 1 filesystem is degraded; 1 mds daemon damaged; 1 MDSs are read only; 1 MDSs report slow metadata IOs; 1/3 mons down, quorum b,c; Reduced data availability: 1 pg inactive; 9 slow ops, oldest one blocked for 19935 sec, osd.5 has slow ops 2022-05-14T05:00:01.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:00:00 smithi047 ceph-mon[77147]: [WRN] FS_DEGRADED: 1 filesystem is degraded 2022-05-14T05:00:01.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:00:00 smithi047 ceph-mon[77147]: fs cephfs is degraded 2022-05-14T05:00:01.295 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:00:00 smithi047 ceph-mon[77147]: [ERR] MDS_DAMAGE: 1 mds daemon damaged 2022-05-14T05:00:01.295 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:00:00 smithi047 ceph-mon[77147]: fs cephfs mds.2 is damaged 2022-05-14T05:00:01.295 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:00:00 smithi047 ceph-mon[77147]: [WRN] MDS_READ_ONLY: 1 MDSs are read only 2022-05-14T05:00:01.296 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:00:00 smithi047 ceph-mon[77147]: mds.l(mds.0): MDS in read-only mode 2022-05-14T05:00:01.297 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:00:00 smithi047 ceph-mon[77147]: [WRN] MDS_SLOW_METADATA_IO: 1 MDSs report slow metadata IOs 2022-05-14T05:00:01.297 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:00:00 smithi047 ceph-mon[77147]: mds.l(mds.0): 4 slow metadata IOs are blocked > 180 secs, oldest blocked for 20234 secs 2022-05-14T05:00:01.297 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:00:00 smithi047 ceph-mon[77147]: [WRN] MON_DOWN: 1/3 mons down, quorum b,c 2022-05-14T05:00:01.297 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:00:00 smithi047 ceph-mon[77147]: mon.a (rank 0) addr [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] is down (out of quorum) 2022-05-14T05:00:01.298 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:00:00 smithi047 ceph-mon[77147]: [WRN] PG_AVAILABILITY: Reduced data availability: 1 pg inactive 2022-05-14T05:00:01.298 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:00:00 smithi047 ceph-mon[77147]: pg 2.1 is stuck inactive for 5h, current state unknown, last acting [] 2022-05-14T05:00:01.298 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:00:00 smithi047 ceph-mon[77147]: [WRN] SLOW_OPS: 9 slow ops, oldest one blocked for 19935 sec, osd.5 has slow ops 2022-05-14T05:00:02.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:00:01 smithi115 conmon[90393]: 2022-05-14T05:00:01.848+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:00:02.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:00:01 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:00:02.297 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:00:01 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:00:03.110 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:00:02 smithi115 conmon[90393]: 2022-05-14T05:00:02.834+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:00:03.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:00:02 smithi115 ceph-mon[78593]: pgmap v10318: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:00:03.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:00:02 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:00:03.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:00:02 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T05:00:03.112 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:00:02 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T05:00:03.112 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:00:02 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:00:03.296 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:00:02 smithi047 ceph-mon[77147]: pgmap v10318: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:00:03.296 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:00:02 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:00:03.297 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:00:02 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T05:00:03.302 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:00:02 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T05:00:03.303 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:00:02 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:00:04.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:00:03 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 19935 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T05:00:04.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:00:03 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 19935 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T05:00:04.360 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:00:03 smithi115 conmon[90393]: 2022-05-14T05:00:03.857+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:00:05.116 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:00:04 smithi115 conmon[90393]: 2022-05-14T05:00:04.829+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:00:05.117 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:00:04 smithi115 ceph-mon[78593]: pgmap v10319: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:00:05.117 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:00:04 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:00:05.118 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:00:04 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:00:05.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:00:04 smithi047 ceph-mon[77147]: pgmap v10319: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:00:05.299 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:00:04 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:00:05.299 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:00:04 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:00:06.362 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:00:05 smithi115 conmon[90393]: 2022-05-14T05:00:05.872+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:00:07.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:00:06 smithi115 conmon[90393]: 2022-05-14T05:00:06.852+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:00:07.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:00:06 smithi115 ceph-mon[78593]: pgmap v10320: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T05:00:07.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:00:06 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:00:07.299 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:00:06 smithi047 ceph-mon[77147]: pgmap v10320: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T05:00:07.299 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:00:06 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:00:08.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:00:07 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:00:08.360 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:00:07 smithi115 conmon[90393]: 2022-05-14T05:00:07.862+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:00:08.367 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:00:07 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:00:09.112 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:00:08 smithi115 conmon[90393]: 2022-05-14T05:00:08.834+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:00:09.113 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:00:08 smithi115 ceph-mon[78593]: pgmap v10321: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:00:09.114 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:00:08 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:00:09.114 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:00:08 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 19940 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T05:00:09.114 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:00:08 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:00:09.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:00:08 smithi047 ceph-mon[77147]: pgmap v10321: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:00:09.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:00:08 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:00:09.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:00:08 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 19940 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T05:00:09.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:00:08 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:00:10.360 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:00:09 smithi115 conmon[90393]: 2022-05-14T05:00:09.882+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:00:11.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:00:10 smithi047 ceph-mon[77147]: pgmap v10322: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T05:00:11.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:00:10 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T05:00:11.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:00:10 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T05:00:11.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:00:10 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:00:11.360 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:00:10 smithi115 conmon[90393]: 2022-05-14T05:00:10.926+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:00:11.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:00:10 smithi115 ceph-mon[78593]: pgmap v10322: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T05:00:11.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:00:10 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T05:00:11.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:00:10 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T05:00:11.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:00:10 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:00:12.296 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:00:11 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:00:12.360 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:00:11 smithi115 conmon[90393]: 2022-05-14T05:00:11.901+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:00:12.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:00:11 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:00:13.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:00:12 smithi047 ceph-mon[77147]: pgmap v10323: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:00:13.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:00:12 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:00:13.361 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:00:12 smithi115 conmon[90393]: 2022-05-14T05:00:12.947+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:00:13.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:00:12 smithi115 ceph-mon[78593]: pgmap v10323: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:00:13.362 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:00:12 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:00:14.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:00:13 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 19945 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T05:00:14.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:00:13 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:00:14.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:00:13 smithi047 ceph-mon[77147]: pgmap v10324: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:00:14.361 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:00:13 smithi115 conmon[90393]: 2022-05-14T05:00:13.947+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:00:14.362 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:00:13 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 19945 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T05:00:14.363 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:00:13 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:00:14.363 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:00:13 smithi115 ceph-mon[78593]: pgmap v10324: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:00:15.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:00:14 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T05:00:15.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:00:14 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T05:00:15.295 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:00:14 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:00:15.295 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:00:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T05:00:15.295 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:00:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T05:00:15.368 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:00:14 smithi115 conmon[90393]: 2022-05-14T05:00:14.986+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:00:15.368 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:00:14 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T05:00:15.369 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:00:14 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T05:00:15.369 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:00:14 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:00:15.369 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:00:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T05:00:15.369 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:00:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T05:00:16.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:00:15 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:00:16.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:00:15 smithi047 ceph-mon[77147]: pgmap v10325: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T05:00:16.361 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:00:15 smithi115 conmon[90393]: 2022-05-14T05:00:15.974+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:00:16.362 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:00:15 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:00:16.362 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:00:15 smithi115 ceph-mon[78593]: pgmap v10325: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T05:00:17.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:00:16 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:00:17.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:00:16 smithi115 conmon[90393]: 2022-05-14T05:00:16.979+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:00:17.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:00:16 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:00:18.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:00:17 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:00:18.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:00:17 smithi047 ceph-mon[77147]: pgmap v10326: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:00:18.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:00:17 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 19950 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T05:00:18.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:00:17 smithi115 conmon[90393]: 2022-05-14T05:00:17.963+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:00:18.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:00:17 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:00:18.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:00:17 smithi115 ceph-mon[78593]: pgmap v10326: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:00:18.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:00:17 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 19950 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T05:00:19.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:00:18 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T05:00:19.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:00:18 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T05:00:19.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:00:18 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:00:19.361 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:00:18 smithi115 conmon[90393]: 2022-05-14T05:00:18.997+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:00:19.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:00:18 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T05:00:19.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:00:18 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T05:00:19.362 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:00:18 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:00:20.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:00:19 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:00:20.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:00:19 smithi047 ceph-mon[77147]: pgmap v10327: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T05:00:20.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:00:19 smithi115 conmon[90393]: 2022-05-14T05:00:19.969+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:00:20.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:00:19 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:00:20.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:00:19 smithi115 ceph-mon[78593]: pgmap v10327: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T05:00:21.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:00:20 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:00:21.365 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:00:21 smithi115 conmon[90393]: 2022-05-14T05:00:21.017+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:00:21.366 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:00:20 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:00:22.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:00:21 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:00:22.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:00:21 smithi047 ceph-mon[77147]: pgmap v10328: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:00:22.360 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:00:21 smithi115 conmon[90393]: 2022-05-14T05:00:21.990+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:00:22.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:00:21 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:00:22.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:00:21 smithi115 ceph-mon[78593]: pgmap v10328: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:00:23.295 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:00:22 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T05:00:23.295 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:00:22 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T05:00:23.296 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:00:22 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:00:23.296 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:00:22 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 19955 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T05:00:23.360 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:00:22 smithi115 conmon[90393]: 2022-05-14T05:00:22.973+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:00:23.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:00:22 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T05:00:23.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:00:22 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T05:00:23.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:00:22 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:00:23.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:00:22 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 19955 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T05:00:24.297 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:00:23 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:00:24.299 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:00:23 smithi047 ceph-mon[77147]: pgmap v10329: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:00:24.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:00:23 smithi115 conmon[90393]: 2022-05-14T05:00:23.973+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:00:24.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:00:23 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:00:24.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:00:23 smithi115 ceph-mon[78593]: pgmap v10329: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:00:25.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:00:24 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:00:25.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:00:25 smithi115 conmon[90393]: 2022-05-14T05:00:25.019+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:00:25.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:00:24 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:00:26.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:00:25 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:00:26.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:00:25 smithi047 ceph-mon[77147]: pgmap v10330: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T05:00:26.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:00:26 smithi115 conmon[90393]: 2022-05-14T05:00:26.016+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:00:26.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:00:25 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:00:26.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:00:25 smithi115 ceph-mon[78593]: pgmap v10330: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T05:00:27.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:00:26 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:00:27.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:00:27 smithi115 conmon[90393]: 2022-05-14T05:00:27.058+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:00:27.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:00:26 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:00:28.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:00:27 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:00:28.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:00:27 smithi047 ceph-mon[77147]: pgmap v10331: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:00:28.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:00:27 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 19960 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T05:00:28.360 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:00:28 smithi115 conmon[90393]: 2022-05-14T05:00:28.053+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:00:28.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:00:27 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:00:28.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:00:27 smithi115 ceph-mon[78593]: pgmap v10331: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:00:28.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:00:27 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 19960 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T05:00:29.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:00:28 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:00:29.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:00:29 smithi115 conmon[90393]: 2022-05-14T05:00:29.037+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:00:29.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:00:28 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:00:30.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:00:29 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:00:30.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:00:29 smithi047 ceph-mon[77147]: pgmap v10332: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T05:00:30.360 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:00:30 smithi115 conmon[90393]: 2022-05-14T05:00:30.054+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:00:30.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:00:29 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:00:30.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:00:29 smithi115 ceph-mon[78593]: pgmap v10332: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T05:00:31.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:00:30 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T05:00:31.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:00:30 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T05:00:31.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:00:30 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:00:31.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:00:31 smithi115 conmon[90393]: 2022-05-14T05:00:31.094+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:00:31.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:00:30 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T05:00:31.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:00:30 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T05:00:31.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:00:30 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:00:32.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:00:31 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:00:32.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:00:31 smithi047 ceph-mon[77147]: pgmap v10333: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:00:32.361 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:00:32 smithi115 conmon[90393]: 2022-05-14T05:00:32.078+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:00:32.362 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:00:31 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:00:32.362 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:00:31 smithi115 ceph-mon[78593]: pgmap v10333: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:00:33.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:00:32 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:00:33.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:00:32 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 19965 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T05:00:33.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:00:33 smithi115 conmon[90393]: 2022-05-14T05:00:33.031+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:00:33.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:00:32 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:00:33.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:00:32 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 19965 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T05:00:34.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:00:33 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:00:34.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:00:33 smithi047 ceph-mon[77147]: pgmap v10334: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:00:34.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:00:34 smithi115 conmon[90393]: 2022-05-14T05:00:34.034+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:00:34.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:00:33 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:00:34.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:00:33 smithi115 ceph-mon[78593]: pgmap v10334: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:00:35.302 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:00:34 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T05:00:35.302 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:00:34 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T05:00:35.302 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:00:34 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:00:35.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:00:34 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T05:00:35.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:00:34 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T05:00:35.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:00:34 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:00:35.362 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:00:35 smithi115 conmon[90393]: 2022-05-14T05:00:35.059+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:00:36.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:00:35 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:00:36.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:00:35 smithi047 ceph-mon[77147]: pgmap v10335: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T05:00:36.362 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:00:36 smithi115 conmon[90393]: 2022-05-14T05:00:36.054+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:00:36.363 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:00:35 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:00:36.364 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:00:35 smithi115 ceph-mon[78593]: pgmap v10335: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T05:00:37.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:00:36 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:00:37.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:00:37 smithi115 conmon[90393]: 2022-05-14T05:00:37.050+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:00:37.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:00:36 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:00:38.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:00:38 smithi115 conmon[90393]: 2022-05-14T05:00:38.066+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:00:38.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:00:37 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:00:38.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:00:37 smithi115 ceph-mon[78593]: pgmap v10336: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:00:38.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:00:37 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T05:00:38.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:00:37 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T05:00:38.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:00:37 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T05:00:38.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:00:37 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 19970 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T05:00:38.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:00:37 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:00:38.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:00:37 smithi047 ceph-mon[77147]: pgmap v10336: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:00:38.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:00:37 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T05:00:38.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:00:37 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T05:00:38.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:00:37 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T05:00:38.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:00:37 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 19970 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T05:00:38.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:00:37 smithi047 conmon[78521]: 2022-05-14T05:00:37.936+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T05:00:38.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:00:37 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T05:00:38.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:00:37 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T05:00:38.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:00:37 smithi047 conmon[78521]: yield 2022-05-14T05:00:38.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:00:37 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T05:00:38.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:00:37 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T05:00:38.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:00:37 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T05:00:38.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:00:37 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T05:00:38.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:00:37 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T05:00:38.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:00:37 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T05:00:38.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:00:37 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T05:00:38.297 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:00:37 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T05:00:38.297 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:00:37 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T05:00:38.297 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:00:37 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T05:00:38.297 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:00:37 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T05:00:38.298 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:00:37 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T05:00:38.298 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:00:37 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T05:00:38.298 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:00:37 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T05:00:38.298 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:00:37 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T05:00:39.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:00:38 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T05:00:39.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T05:00:39.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T05:00:39.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T05:00:39.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T05:00:39.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T05:00:39.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T05:00:39.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T05:00:39.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T05:00:39.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T05:00:39.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T05:00:39.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T05:00:39.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T05:00:39.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T05:00:39.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T05:00:39.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T05:00:39.298 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T05:00:39.298 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T05:00:39.298 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T05:00:39.298 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:00:38 smithi047 ceph-mon[77147]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T05:00:39.298 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:00:38 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T05:00:39.299 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:00:38 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T05:00:39.299 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:00:38 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:00:39.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:00:39 smithi115 conmon[90393]: 2022-05-14T05:00:39.101+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:00:39.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:00:38 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T05:00:39.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T05:00:39.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T05:00:39.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T05:00:39.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T05:00:39.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T05:00:39.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T05:00:39.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T05:00:39.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T05:00:39.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T05:00:39.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T05:00:39.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T05:00:39.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T05:00:39.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T05:00:39.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T05:00:39.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T05:00:39.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T05:00:39.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T05:00:39.365 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T05:00:39.365 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:00:38 smithi115 ceph-mon[78593]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T05:00:39.365 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:00:38 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T05:00:39.366 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:00:38 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T05:00:39.366 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:00:38 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:00:40.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:00:39 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:00:40.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:00:39 smithi047 ceph-mon[77147]: pgmap v10337: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T05:00:40.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:00:40 smithi115 conmon[90393]: 2022-05-14T05:00:40.111+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:00:40.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:00:39 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:00:40.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:00:39 smithi115 ceph-mon[78593]: pgmap v10337: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T05:00:41.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:00:40 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:00:41.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:00:41 smithi047 conmon[78521]: 2022-05-14T05:00:41.008+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T05:00:41.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:00:41 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T05:00:41.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:00:41 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T05:00:41.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:00:41 smithi047 conmon[78521]: yield 2022-05-14T05:00:41.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:00:41 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T05:00:41.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:00:41 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T05:00:41.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:00:41 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T05:00:41.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:00:41 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T05:00:41.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:00:41 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T05:00:41.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:00:41 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T05:00:41.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:00:41 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T05:00:41.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:00:41 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T05:00:41.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:00:41 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T05:00:41.297 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:00:41 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T05:00:41.297 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:00:41 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T05:00:41.297 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:00:41 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T05:00:41.297 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:00:41 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T05:00:41.297 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:00:41 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T05:00:41.298 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:00:41 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T05:00:41.298 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:00:41 smithi047 conmon[78521]: 2022-05-14T05:00:41.009+0000 7f533ff78700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T05:00:41.298 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:00:41 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T05:00:41.298 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:00:41 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T05:00:41.299 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:00:41 smithi047 conmon[78521]: yield 2022-05-14T05:00:41.299 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:00:41 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T05:00:41.299 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:00:41 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T05:00:41.299 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:00:41 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T05:00:41.300 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:00:41 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T05:00:41.302 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:00:41 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T05:00:41.303 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:00:41 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T05:00:41.303 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:00:41 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T05:00:41.303 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:00:41 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T05:00:41.303 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:00:41 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T05:00:41.304 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:00:41 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T05:00:41.304 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:00:41 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T05:00:41.304 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:00:41 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T05:00:41.304 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:00:41 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T05:00:41.305 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:00:41 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T05:00:41.305 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:00:41 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T05:00:41.305 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:00:41 smithi047 conmon[78521]: 2022-05-14T05:00:41.305 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:00:41 smithi047 conmon[78521]: During handling of the above exception, another exception occurred: 2022-05-14T05:00:41.305 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:00:41 smithi047 conmon[78521]: 2022-05-14T05:00:41.306 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:00:41 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T05:00:41.306 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:00:41 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T05:00:41.306 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:00:41 smithi047 conmon[78521]: return f(*arg) 2022-05-14T05:00:41.306 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:00:41 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T05:00:41.307 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:00:41 smithi047 conmon[78521]: self._write_client_files(client_files, host) 2022-05-14T05:00:41.307 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:00:41 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T05:00:41.307 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:00:41 smithi047 conmon[78521]: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T05:00:41.307 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:00:41 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T05:00:41.308 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:00:41 smithi047 conmon[78521]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T05:00:41.308 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:00:41 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T05:00:41.308 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:00:41 smithi047 conmon[78521]: return self.event_loop.get_result(coro) 2022-05-14T05:00:41.308 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:00:41 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T05:00:41.309 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:00:41 smithi047 conmon[78521]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T05:00:41.309 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:00:41 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T05:00:41.309 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:00:41 smithi047 conmon[78521]: return self.__get_result() 2022-05-14T05:00:41.309 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:00:41 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T05:00:41.309 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:00:41 smithi047 conmon[78521]: raise self._exception 2022-05-14T05:00:41.310 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:00:41 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T05:00:41.310 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:00:41 smithi047 conmon[78521]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T05:00:41.310 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:00:41 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T05:00:41.310 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:00:41 smithi047 conmon[78521]: conn = await self._remote_connection(host, addr) 2022-05-14T05:00:41.311 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:00:41 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T05:00:41.311 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:00:41 smithi047 conmon[78521]: raise 2022-05-14T05:00:41.311 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:00:41 smithi047 conmon[78521]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T05:00:41.311 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:00:41 smithi047 conmon[78521]: self.gen.throw(type, value, traceback) 2022-05-14T05:00:41.311 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:00:41 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T05:00:41.312 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:00:41 smithi047 conmon[78521]: raise OrchestratorError(msg) 2022-05-14T05:00:41.312 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:00:41 smithi047 conmon[78521]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T05:00:41.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:00:41 smithi115 conmon[90393]: 2022-05-14T05:00:41.084+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:00:41.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:00:40 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:00:42.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:00:41 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T05:00:42.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T05:00:42.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T05:00:42.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T05:00:42.298 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T05:00:42.298 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T05:00:42.298 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T05:00:42.298 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T05:00:42.298 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T05:00:42.299 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T05:00:42.299 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T05:00:42.299 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T05:00:42.299 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T05:00:42.300 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T05:00:42.300 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T05:00:42.300 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T05:00:42.300 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T05:00:42.300 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T05:00:42.301 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T05:00:42.301 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:00:41 smithi047 ceph-mon[77147]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T05:00:42.301 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T05:00:42.301 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T05:00:42.302 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T05:00:42.302 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T05:00:42.302 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T05:00:42.302 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T05:00:42.302 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T05:00:42.303 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T05:00:42.303 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T05:00:42.303 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T05:00:42.303 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T05:00:42.304 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T05:00:42.304 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T05:00:42.304 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T05:00:42.304 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T05:00:42.304 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T05:00:42.305 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T05:00:42.305 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T05:00:42.305 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T05:00:42.305 INFO:journalctl@ceph.mon.b.smithi047.stdout: During handling of the above exception, another exception occurred: 2022-05-14T05:00:42.306 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T05:00:42.306 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T05:00:42.306 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T05:00:42.306 INFO:journalctl@ceph.mon.b.smithi047.stdout: return f(*arg) 2022-05-14T05:00:42.307 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T05:00:42.307 INFO:journalctl@ceph.mon.b.smithi047.stdout: self._write_client_files(client_files, host) 2022-05-14T05:00:42.308 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T05:00:42.308 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T05:00:42.308 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T05:00:42.308 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T05:00:42.309 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T05:00:42.309 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.event_loop.get_result(coro) 2022-05-14T05:00:42.309 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T05:00:42.309 INFO:journalctl@ceph.mon.b.smithi047.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T05:00:42.309 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T05:00:42.310 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.__get_result() 2022-05-14T05:00:42.310 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T05:00:42.310 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise self._exception 2022-05-14T05:00:42.310 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T05:00:42.311 INFO:journalctl@ceph.mon.b.smithi047.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T05:00:42.311 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T05:00:42.311 INFO:journalctl@ceph.mon.b.smithi047.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T05:00:42.311 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T05:00:42.311 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise 2022-05-14T05:00:42.312 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T05:00:42.312 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.gen.throw(type, value, traceback) 2022-05-14T05:00:42.312 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T05:00:42.312 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OrchestratorError(msg) 2022-05-14T05:00:42.313 INFO:journalctl@ceph.mon.b.smithi047.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T05:00:42.313 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:00:41 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:00:42.313 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:00:41 smithi047 ceph-mon[77147]: pgmap v10338: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:00:42.360 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:00:42 smithi115 conmon[90393]: 2022-05-14T05:00:42.058+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:00:42.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:00:41 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T05:00:42.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T05:00:42.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T05:00:42.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T05:00:42.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T05:00:42.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T05:00:42.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T05:00:42.365 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T05:00:42.365 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T05:00:42.366 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T05:00:42.366 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T05:00:42.367 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T05:00:42.367 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T05:00:42.368 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T05:00:42.368 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T05:00:42.369 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T05:00:42.369 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T05:00:42.370 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T05:00:42.370 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T05:00:42.371 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:00:41 smithi115 ceph-mon[78593]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T05:00:42.371 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T05:00:42.372 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T05:00:42.372 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T05:00:42.373 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T05:00:42.373 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T05:00:42.374 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T05:00:42.374 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T05:00:42.374 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T05:00:42.375 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T05:00:42.375 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T05:00:42.376 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T05:00:42.377 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T05:00:42.377 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T05:00:42.377 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T05:00:42.378 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T05:00:42.378 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T05:00:42.379 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T05:00:42.379 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T05:00:42.380 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T05:00:42.380 INFO:journalctl@ceph.mon.c.smithi115.stdout: During handling of the above exception, another exception occurred: 2022-05-14T05:00:42.381 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T05:00:42.381 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T05:00:42.382 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T05:00:42.382 INFO:journalctl@ceph.mon.c.smithi115.stdout: return f(*arg) 2022-05-14T05:00:42.383 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T05:00:42.383 INFO:journalctl@ceph.mon.c.smithi115.stdout: self._write_client_files(client_files, host) 2022-05-14T05:00:42.384 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T05:00:42.384 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T05:00:42.385 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T05:00:42.386 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T05:00:42.386 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T05:00:42.387 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.event_loop.get_result(coro) 2022-05-14T05:00:42.387 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T05:00:42.388 INFO:journalctl@ceph.mon.c.smithi115.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T05:00:42.388 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T05:00:42.388 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.__get_result() 2022-05-14T05:00:42.389 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T05:00:42.389 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise self._exception 2022-05-14T05:00:42.390 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T05:00:42.390 INFO:journalctl@ceph.mon.c.smithi115.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T05:00:42.391 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T05:00:42.391 INFO:journalctl@ceph.mon.c.smithi115.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T05:00:42.391 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T05:00:42.392 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise 2022-05-14T05:00:42.392 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T05:00:42.393 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.gen.throw(type, value, traceback) 2022-05-14T05:00:42.393 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T05:00:42.393 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OrchestratorError(msg) 2022-05-14T05:00:42.393 INFO:journalctl@ceph.mon.c.smithi115.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T05:00:42.394 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:00:41 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:00:42.394 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:00:41 smithi115 ceph-mon[78593]: pgmap v10338: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:00:43.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:00:42 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T05:00:43.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:00:42 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T05:00:43.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:00:42 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:00:43.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:00:42 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 19976 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T05:00:43.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:00:43 smithi115 conmon[90393]: 2022-05-14T05:00:43.090+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:00:43.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:00:42 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T05:00:43.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:00:42 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T05:00:43.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:00:42 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:00:43.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:00:42 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 19976 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T05:00:44.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:00:43 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:00:44.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:00:43 smithi047 ceph-mon[77147]: pgmap v10339: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:00:44.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:00:44 smithi115 conmon[90393]: 2022-05-14T05:00:44.133+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:00:44.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:00:43 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:00:44.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:00:43 smithi115 ceph-mon[78593]: pgmap v10339: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:00:45.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:00:44 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:00:45.360 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:00:45 smithi115 conmon[90393]: 2022-05-14T05:00:45.165+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:00:45.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:00:44 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:00:46.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:00:46 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:00:46.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:00:46 smithi047 ceph-mon[77147]: pgmap v10340: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T05:00:46.361 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:00:46 smithi115 conmon[90393]: 2022-05-14T05:00:46.164+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:00:46.362 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:00:46 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:00:46.362 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:00:46 smithi115 ceph-mon[78593]: pgmap v10340: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T05:00:47.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:00:47 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:00:47.360 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:00:47 smithi115 conmon[90393]: 2022-05-14T05:00:47.182+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:00:47.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:00:47 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:00:48.298 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:00:48 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:00:48.298 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:00:48 smithi047 ceph-mon[77147]: pgmap v10341: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:00:48.298 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:00:48 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 19980 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T05:00:48.363 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:00:48 smithi115 conmon[90393]: 2022-05-14T05:00:48.195+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:00:48.363 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:00:48 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:00:48.364 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:00:48 smithi115 ceph-mon[78593]: pgmap v10341: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:00:48.364 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:00:48 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 19980 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T05:00:49.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:00:49 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:00:49.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:00:49 smithi115 conmon[90393]: 2022-05-14T05:00:49.241+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:00:49.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:00:49 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:00:50.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:00:50 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:00:50.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:00:50 smithi047 ceph-mon[77147]: pgmap v10342: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T05:00:50.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:00:50 smithi115 conmon[90393]: 2022-05-14T05:00:50.212+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:00:50.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:00:50 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:00:50.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:00:50 smithi115 ceph-mon[78593]: pgmap v10342: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T05:00:51.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:00:51 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T05:00:51.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:00:51 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T05:00:51.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:00:51 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:00:51.361 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:00:51 smithi115 conmon[90393]: 2022-05-14T05:00:51.200+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:00:51.362 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:00:51 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T05:00:51.362 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:00:51 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T05:00:51.362 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:00:51 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:00:52.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:00:52 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:00:52.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:00:52 smithi047 ceph-mon[77147]: pgmap v10343: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:00:52.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:00:52 smithi115 conmon[90393]: 2022-05-14T05:00:52.229+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:00:52.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:00:52 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:00:52.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:00:52 smithi115 ceph-mon[78593]: pgmap v10343: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:00:53.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:00:53 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:00:53.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:00:53 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 19985 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T05:00:53.360 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:00:53 smithi115 conmon[90393]: 2022-05-14T05:00:53.270+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:00:53.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:00:53 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:00:53.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:00:53 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 19985 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T05:00:54.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:00:54 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:00:54.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:00:54 smithi047 ceph-mon[77147]: pgmap v10344: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:00:54.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:00:54 smithi115 conmon[90393]: 2022-05-14T05:00:54.274+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:00:54.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:00:54 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:00:54.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:00:54 smithi115 ceph-mon[78593]: pgmap v10344: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:00:55.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:00:55 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T05:00:55.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:00:55 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T05:00:55.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:00:55 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:00:55.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:00:55 smithi115 conmon[90393]: 2022-05-14T05:00:55.229+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:00:55.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:00:55 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T05:00:55.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:00:55 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T05:00:55.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:00:55 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:00:56.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:00:56 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:00:56.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:00:56 smithi047 ceph-mon[77147]: pgmap v10345: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T05:00:56.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:00:56 smithi115 conmon[90393]: 2022-05-14T05:00:56.223+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:00:56.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:00:56 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:00:56.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:00:56 smithi115 ceph-mon[78593]: pgmap v10345: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T05:00:57.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:00:57 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:00:57.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:00:57 smithi115 conmon[90393]: 2022-05-14T05:00:57.200+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:00:57.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:00:57 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:00:58.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:00:58 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:00:58.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:00:58 smithi047 ceph-mon[77147]: pgmap v10346: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:00:58.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:00:58 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 19990 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T05:00:58.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:00:58 smithi115 conmon[90393]: 2022-05-14T05:00:58.200+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:00:58.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:00:58 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:00:58.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:00:58 smithi115 ceph-mon[78593]: pgmap v10346: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:00:58.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:00:58 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 19990 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T05:00:59.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:00:59 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T05:00:59.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:00:59 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T05:00:59.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:00:59 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:00:59.366 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:00:59 smithi115 conmon[90393]: 2022-05-14T05:00:59.169+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:00:59.366 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:00:59 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T05:00:59.367 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:00:59 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T05:00:59.367 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:00:59 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:01:00.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:01:00 smithi115 conmon[90393]: 2022-05-14T05:01:00.128+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:01:00.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:01:00 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:01:00.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:01:00 smithi115 ceph-mon[78593]: pgmap v10347: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T05:01:00.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:01:00 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:01:00.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:01:00 smithi047 ceph-mon[77147]: pgmap v10347: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T05:01:01.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:01:01 smithi115 conmon[90393]: 2022-05-14T05:01:01.159+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:01:01.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:01:01 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:01:01.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:01:01 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:01:02.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:01:02 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:01:02.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:01:02 smithi047 ceph-mon[77147]: pgmap v10348: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:01:02.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:01:02 smithi115 conmon[90393]: 2022-05-14T05:01:02.111+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:01:02.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:01:02 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:01:02.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:01:02 smithi115 ceph-mon[78593]: pgmap v10348: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:01:03.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:01:03 smithi115 conmon[90393]: 2022-05-14T05:01:03.116+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:01:03.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:01:03 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T05:01:03.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:01:03 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T05:01:03.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:01:03 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:01:03.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:01:03 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 19995 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T05:01:03.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:01:03 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T05:01:03.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:01:03 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T05:01:03.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:01:03 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:01:03.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:01:03 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 19995 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T05:01:04.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:01:04 smithi115 conmon[90393]: 2022-05-14T05:01:04.153+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:01:04.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:01:04 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:01:04.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:01:04 smithi115 ceph-mon[78593]: pgmap v10349: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:01:04.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:01:04 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:01:04.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:01:04 smithi047 ceph-mon[77147]: pgmap v10349: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:01:05.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:01:05 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:01:05.360 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:01:05 smithi115 conmon[90393]: 2022-05-14T05:01:05.140+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:01:05.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:01:05 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:01:06.360 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:01:06 smithi115 conmon[90393]: 2022-05-14T05:01:06.101+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:01:06.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:01:06 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:01:06.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:01:06 smithi115 ceph-mon[78593]: pgmap v10350: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T05:01:06.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:01:06 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:01:06.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:01:06 smithi047 ceph-mon[77147]: pgmap v10350: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T05:01:07.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:01:07 smithi115 conmon[90393]: 2022-05-14T05:01:07.058+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:01:07.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:01:07 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:01:07.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:01:07 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:01:08.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:01:08 smithi115 conmon[90393]: 2022-05-14T05:01:08.070+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:01:08.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:01:08 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:01:08.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:01:08 smithi115 ceph-mon[78593]: pgmap v10351: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:01:08.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:01:08 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 20000 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T05:01:08.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:01:08 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:01:08.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:01:08 smithi047 ceph-mon[77147]: pgmap v10351: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:01:08.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:01:08 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 20000 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T05:01:09.360 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:01:09 smithi115 conmon[90393]: 2022-05-14T05:01:09.089+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:01:09.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:01:09 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:01:09.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:01:09 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:01:10.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:01:10 smithi115 conmon[90393]: 2022-05-14T05:01:10.064+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:01:10.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:01:10 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:01:10.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:01:10 smithi115 ceph-mon[78593]: pgmap v10352: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T05:01:10.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:01:10 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:01:10.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:01:10 smithi047 ceph-mon[77147]: pgmap v10352: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T05:01:11.360 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:01:11 smithi115 conmon[90393]: 2022-05-14T05:01:11.048+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:01:11.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:01:11 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:01:11.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:01:11 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T05:01:11.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:01:11 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T05:01:11.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:01:11 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:01:11.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:01:11 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T05:01:11.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:01:11 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T05:01:12.363 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:01:12 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:01:12.365 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:01:12 smithi115 ceph-mon[78593]: pgmap v10353: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:01:12.366 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:01:12 smithi115 conmon[90393]: 2022-05-14T05:01:12.044+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:01:12.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:01:12 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:01:12.545 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:01:12 smithi047 ceph-mon[77147]: pgmap v10353: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:01:13.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:01:13 smithi115 conmon[90393]: 2022-05-14T05:01:13.028+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:01:13.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:01:13 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:01:13.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:01:13 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 20006 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T05:01:13.546 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:01:13 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:01:13.547 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:01:13 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 20006 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T05:01:14.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:01:14 smithi115 conmon[90393]: 2022-05-14T05:01:14.002+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:01:14.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:01:14 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:01:14.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:01:14 smithi115 ceph-mon[78593]: pgmap v10354: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:01:14.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:01:14 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:01:14.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:01:14 smithi047 ceph-mon[77147]: pgmap v10354: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:01:15.362 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:01:14 smithi115 conmon[90393]: 2022-05-14T05:01:14.977+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:01:15.362 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:01:15 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:01:15.363 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:01:15 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T05:01:15.363 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:01:15 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T05:01:15.363 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:01:15 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T05:01:15.364 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:01:15 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T05:01:15.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:01:15 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:01:15.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:01:15 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T05:01:15.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:01:15 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T05:01:15.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:01:15 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T05:01:15.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:01:15 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T05:01:16.372 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:01:15 smithi115 conmon[90393]: 2022-05-14T05:01:15.962+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:01:16.373 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:01:16 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:01:16.373 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:01:16 smithi115 ceph-mon[78593]: pgmap v10355: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T05:01:16.546 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:01:16 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:01:16.546 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:01:16 smithi047 ceph-mon[77147]: pgmap v10355: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T05:01:17.363 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:01:16 smithi115 conmon[90393]: 2022-05-14T05:01:16.930+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:01:17.364 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:01:17 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:01:17.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:01:17 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:01:18.360 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:01:17 smithi115 conmon[90393]: 2022-05-14T05:01:17.901+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:01:18.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:01:18 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:01:18.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:01:18 smithi115 ceph-mon[78593]: pgmap v10356: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:01:18.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:01:18 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 20010 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T05:01:18.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:01:18 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:01:18.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:01:18 smithi047 ceph-mon[77147]: pgmap v10356: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:01:18.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:01:18 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 20010 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T05:01:19.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:01:18 smithi115 conmon[90393]: 2022-05-14T05:01:18.934+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:01:19.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:01:19 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:01:19.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:01:19 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T05:01:19.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:01:19 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T05:01:19.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:01:19 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:01:19.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:01:19 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T05:01:19.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:01:19 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T05:01:20.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:01:19 smithi115 conmon[90393]: 2022-05-14T05:01:19.888+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:01:20.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:01:20 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:01:20.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:01:20 smithi115 ceph-mon[78593]: pgmap v10357: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T05:01:20.551 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:01:20 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:01:20.551 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:01:20 smithi047 ceph-mon[77147]: pgmap v10357: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T05:01:21.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:01:20 smithi115 conmon[90393]: 2022-05-14T05:01:20.842+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:01:21.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:01:21 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:01:21.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:01:21 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:01:21.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:01:21 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:01:21.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:01:21 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:01:22.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:01:21 smithi115 conmon[90393]: 2022-05-14T05:01:21.840+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:01:22.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:01:22 smithi047 ceph-mon[77147]: pgmap v10358: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:01:22.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:01:22 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:01:22.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:01:22 smithi115 ceph-mon[78593]: pgmap v10358: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:01:22.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:01:22 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:01:23.111 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:01:22 smithi115 conmon[90393]: 2022-05-14T05:01:22.831+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:01:23.557 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:01:23 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T05:01:23.557 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:01:23 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T05:01:23.557 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:01:23 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:01:23.558 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:01:23 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 20015 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T05:01:23.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:01:23 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T05:01:23.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:01:23 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T05:01:23.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:01:23 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:01:23.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:01:23 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 20015 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T05:01:24.127 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:01:23 smithi115 conmon[90393]: 2022-05-14T05:01:23.868+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:01:24.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:01:24 smithi047 ceph-mon[77147]: pgmap v10359: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:01:24.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:01:24 smithi115 ceph-mon[78593]: pgmap v10359: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:01:25.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:01:24 smithi115 conmon[90393]: 2022-05-14T05:01:24.915+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:01:25.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:01:25 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:01:25.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:01:25 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:01:26.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:01:25 smithi115 conmon[90393]: 2022-05-14T05:01:25.954+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:01:26.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:01:26 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:01:26.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:01:26 smithi115 ceph-mon[78593]: pgmap v10360: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T05:01:26.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:01:26 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:01:26.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:01:26 smithi047 ceph-mon[77147]: pgmap v10360: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T05:01:27.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:01:26 smithi115 conmon[90393]: 2022-05-14T05:01:26.909+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:01:27.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:01:27 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:01:27.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:01:27 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:01:28.362 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:01:27 smithi115 conmon[90393]: 2022-05-14T05:01:27.920+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:01:28.363 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:01:28 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:01:28.363 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:01:28 smithi115 ceph-mon[78593]: pgmap v10361: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:01:28.363 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:01:28 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 20020 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T05:01:28.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:01:28 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:01:28.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:01:28 smithi047 ceph-mon[77147]: pgmap v10361: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:01:28.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:01:28 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 20020 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T05:01:29.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:01:28 smithi115 conmon[90393]: 2022-05-14T05:01:28.958+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:01:29.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:01:29 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:01:29.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:01:29 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:01:30.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:01:29 smithi115 conmon[90393]: 2022-05-14T05:01:29.985+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:01:30.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:01:30 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:01:30.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:01:30 smithi115 ceph-mon[78593]: pgmap v10362: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T05:01:30.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:01:30 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:01:30.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:01:30 smithi047 ceph-mon[77147]: pgmap v10362: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T05:01:31.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:01:30 smithi115 conmon[90393]: 2022-05-14T05:01:30.941+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:01:31.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:01:31 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:01:31.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:01:31 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T05:01:31.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:01:31 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T05:01:31.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:01:31 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:01:31.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:01:31 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T05:01:31.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:01:31 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T05:01:32.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:01:31 smithi115 conmon[90393]: 2022-05-14T05:01:31.966+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:01:32.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:01:32 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:01:32.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:01:32 smithi115 ceph-mon[78593]: pgmap v10363: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:01:32.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:01:32 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:01:32.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:01:32 smithi047 ceph-mon[77147]: pgmap v10363: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:01:33.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:01:33 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:01:33.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:01:33 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 20025 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T05:01:33.360 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:01:32 smithi115 conmon[90393]: 2022-05-14T05:01:32.936+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:01:33.545 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:01:33 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:01:33.546 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:01:33 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 20025 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T05:01:34.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:01:33 smithi115 conmon[90393]: 2022-05-14T05:01:33.957+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:01:34.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:01:34 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:01:34.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:01:34 smithi115 ceph-mon[78593]: pgmap v10364: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:01:34.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:01:34 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:01:34.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:01:34 smithi047 ceph-mon[77147]: pgmap v10364: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:01:35.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:01:34 smithi115 conmon[90393]: 2022-05-14T05:01:34.919+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:01:35.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:01:35 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:01:35.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:01:35 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T05:01:35.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:01:35 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T05:01:35.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:01:35 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:01:35.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:01:35 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T05:01:35.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:01:35 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T05:01:36.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:01:35 smithi115 conmon[90393]: 2022-05-14T05:01:35.935+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:01:36.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:01:36 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:01:36.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:01:36 smithi115 ceph-mon[78593]: pgmap v10365: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T05:01:36.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:01:36 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:01:36.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:01:36 smithi047 ceph-mon[77147]: pgmap v10365: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T05:01:37.381 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:01:36 smithi115 conmon[90393]: 2022-05-14T05:01:36.967+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:01:37.382 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:01:37 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:01:37.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:01:37 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:01:38.367 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:01:37 smithi115 conmon[90393]: 2022-05-14T05:01:37.973+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:01:38.368 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:01:38 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:01:38.369 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:01:38 smithi115 ceph-mon[78593]: pgmap v10366: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:01:38.369 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:01:38 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 20030 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T05:01:38.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:01:38 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:01:38.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:01:38 smithi047 ceph-mon[77147]: pgmap v10366: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:01:38.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:01:38 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 20030 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T05:01:39.360 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:01:38 smithi115 conmon[90393]: 2022-05-14T05:01:38.957+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:01:39.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:01:39 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:01:39.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:01:39 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T05:01:39.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:01:39 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T05:01:39.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:01:39 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:01:39.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:01:39 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T05:01:39.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:01:39 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T05:01:40.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:01:40 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:01:40.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:01:40 smithi115 ceph-mon[78593]: pgmap v10367: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T05:01:40.360 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:01:39 smithi115 conmon[90393]: 2022-05-14T05:01:39.938+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:01:40.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:01:40 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:01:40.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:01:40 smithi047 ceph-mon[77147]: pgmap v10367: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T05:01:41.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:01:40 smithi115 conmon[90393]: 2022-05-14T05:01:40.938+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:01:41.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:01:41 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:01:41.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:01:41 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T05:01:41.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:01:41 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T05:01:41.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:01:41 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T05:01:41.546 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:01:41 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:01:41.546 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:01:41 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T05:01:41.547 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:01:41 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T05:01:41.547 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:01:41 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T05:01:42.187 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:01:41 smithi115 conmon[90393]: 2022-05-14T05:01:41.919+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:01:42.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:01:42 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:01:42.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:01:42 smithi047 ceph-mon[77147]: pgmap v10368: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:01:42.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:01:42 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:01:42.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:01:42 smithi115 ceph-mon[78593]: pgmap v10368: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:01:43.186 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:01:42 smithi115 conmon[90393]: 2022-05-14T05:01:42.912+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:01:43.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:01:43 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:01:43.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:01:43 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T05:01:43.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:01:43 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T05:01:43.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:01:43 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 20035 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T05:01:43.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:01:43 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:01:43.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:01:43 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T05:01:43.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:01:43 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T05:01:43.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:01:43 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 20035 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T05:01:44.191 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:01:43 smithi115 conmon[90393]: 2022-05-14T05:01:43.938+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:01:44.546 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:01:44 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:01:44.546 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:01:44 smithi047 ceph-mon[77147]: pgmap v10369: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:01:44.547 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:01:44 smithi047 conmon[78521]: 2022-05-14T05:01:44.111+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T05:01:44.547 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:01:44 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T05:01:44.547 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:01:44 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T05:01:44.547 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:01:44 smithi047 conmon[78521]: yield 2022-05-14T05:01:44.548 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:01:44 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T05:01:44.549 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:01:44 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T05:01:44.550 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:01:44 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T05:01:44.550 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:01:44 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T05:01:44.550 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:01:44 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T05:01:44.550 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:01:44 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T05:01:44.551 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:01:44 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T05:01:44.551 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:01:44 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T05:01:44.551 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:01:44 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T05:01:44.551 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:01:44 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T05:01:44.551 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:01:44 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T05:01:44.552 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:01:44 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T05:01:44.552 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:01:44 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T05:01:44.552 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:01:44 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T05:01:44.552 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:01:44 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T05:01:44.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:01:44 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:01:44.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:01:44 smithi115 ceph-mon[78593]: pgmap v10369: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:01:45.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:01:44 smithi115 conmon[90393]: 2022-05-14T05:01:44.963+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:01:45.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:01:45 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:01:45.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:01:45 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T05:01:45.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T05:01:45.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T05:01:45.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T05:01:45.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T05:01:45.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T05:01:45.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T05:01:45.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T05:01:45.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T05:01:45.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T05:01:45.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T05:01:45.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T05:01:45.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T05:01:45.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T05:01:45.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T05:01:45.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T05:01:45.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T05:01:45.365 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T05:01:45.365 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T05:01:45.365 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:01:45 smithi115 ceph-mon[78593]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T05:01:45.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:01:45 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:01:45.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:01:45 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T05:01:45.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T05:01:45.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T05:01:45.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T05:01:45.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T05:01:45.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T05:01:45.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T05:01:45.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T05:01:45.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T05:01:45.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T05:01:45.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T05:01:45.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T05:01:45.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T05:01:45.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T05:01:45.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T05:01:45.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T05:01:45.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T05:01:45.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T05:01:45.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T05:01:45.547 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:01:45 smithi047 ceph-mon[77147]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T05:01:46.360 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:01:45 smithi115 conmon[90393]: 2022-05-14T05:01:45.997+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:01:46.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:01:46 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:01:46.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:01:46 smithi115 ceph-mon[78593]: pgmap v10370: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T05:01:46.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:01:46 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:01:46.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:01:46 smithi047 ceph-mon[77147]: pgmap v10370: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T05:01:47.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:01:47 smithi115 conmon[90393]: 2022-05-14T05:01:47.023+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:01:47.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:01:47 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:01:47.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:01:47 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:01:47.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:01:47 smithi047 conmon[78521]: 2022-05-14T05:01:47.184+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T05:01:47.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:01:47 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T05:01:47.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:01:47 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T05:01:47.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:01:47 smithi047 conmon[78521]: yield 2022-05-14T05:01:47.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:01:47 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T05:01:47.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:01:47 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T05:01:47.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:01:47 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T05:01:47.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:01:47 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T05:01:47.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:01:47 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T05:01:47.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:01:47 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T05:01:47.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:01:47 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T05:01:47.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:01:47 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T05:01:47.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:01:47 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T05:01:47.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:01:47 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T05:01:47.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:01:47 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T05:01:47.547 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:01:47 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T05:01:47.547 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:01:47 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T05:01:47.548 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:01:47 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T05:01:47.548 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:01:47 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T05:01:47.548 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:01:47 smithi047 conmon[78521]: 2022-05-14T05:01:47.184+0000 7f533ef76700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T05:01:47.548 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:01:47 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T05:01:47.549 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:01:47 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T05:01:47.549 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:01:47 smithi047 conmon[78521]: yield 2022-05-14T05:01:47.549 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:01:47 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T05:01:47.549 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:01:47 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T05:01:47.549 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:01:47 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T05:01:47.550 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:01:47 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T05:01:47.550 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:01:47 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T05:01:47.550 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:01:47 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T05:01:47.550 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:01:47 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T05:01:47.550 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:01:47 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T05:01:47.551 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:01:47 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T05:01:47.551 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:01:47 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T05:01:47.551 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:01:47 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T05:01:47.551 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:01:47 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T05:01:47.552 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:01:47 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T05:01:47.552 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:01:47 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T05:01:47.552 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:01:47 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T05:01:47.552 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:01:47 smithi047 conmon[78521]: 2022-05-14T05:01:47.552 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:01:47 smithi047 conmon[78521]: During handling of the above exception, another exception occurred: 2022-05-14T05:01:47.553 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:01:47 smithi047 conmon[78521]: 2022-05-14T05:01:47.553 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:01:47 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T05:01:47.553 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:01:47 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T05:01:47.554 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:01:47 smithi047 conmon[78521]: return f(*arg) 2022-05-14T05:01:47.554 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:01:47 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T05:01:47.554 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:01:47 smithi047 conmon[78521]: self._write_client_files(client_files, host) 2022-05-14T05:01:47.554 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:01:47 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T05:01:47.554 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:01:47 smithi047 conmon[78521]: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T05:01:47.555 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:01:47 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T05:01:47.555 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:01:47 smithi047 conmon[78521]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T05:01:47.556 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:01:47 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T05:01:47.556 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:01:47 smithi047 conmon[78521]: return self.event_loop.get_result(coro) 2022-05-14T05:01:47.556 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:01:47 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T05:01:47.556 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:01:47 smithi047 conmon[78521]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T05:01:47.556 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:01:47 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T05:01:47.557 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:01:47 smithi047 conmon[78521]: return self.__get_result() 2022-05-14T05:01:47.557 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:01:47 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T05:01:47.557 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:01:47 smithi047 conmon[78521]: raise self._exception 2022-05-14T05:01:47.557 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:01:47 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T05:01:47.558 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:01:47 smithi047 conmon[78521]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T05:01:47.558 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:01:47 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T05:01:47.558 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:01:47 smithi047 conmon[78521]: conn = await self._remote_connection(host, addr) 2022-05-14T05:01:47.558 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:01:47 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T05:01:47.558 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:01:47 smithi047 conmon[78521]: raise 2022-05-14T05:01:47.559 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:01:47 smithi047 conmon[78521]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T05:01:47.559 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:01:47 smithi047 conmon[78521]: self.gen.throw(type, value, traceback) 2022-05-14T05:01:47.559 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:01:47 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T05:01:47.559 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:01:47 smithi047 conmon[78521]: raise OrchestratorError(msg) 2022-05-14T05:01:47.559 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:01:47 smithi047 conmon[78521]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T05:01:48.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:01:48 smithi115 conmon[90393]: 2022-05-14T05:01:48.062+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:01:48.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:01:48 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:01:48.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:01:48 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T05:01:48.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T05:01:48.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T05:01:48.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T05:01:48.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T05:01:48.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T05:01:48.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T05:01:48.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T05:01:48.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T05:01:48.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T05:01:48.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T05:01:48.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T05:01:48.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T05:01:48.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T05:01:48.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T05:01:48.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T05:01:48.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T05:01:48.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T05:01:48.365 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T05:01:48.365 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:01:48 smithi115 ceph-mon[78593]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T05:01:48.365 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T05:01:48.365 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T05:01:48.365 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T05:01:48.366 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T05:01:48.366 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T05:01:48.366 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T05:01:48.366 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T05:01:48.366 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T05:01:48.367 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T05:01:48.367 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T05:01:48.367 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T05:01:48.367 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T05:01:48.367 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T05:01:48.368 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T05:01:48.368 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T05:01:48.368 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T05:01:48.368 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T05:01:48.369 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T05:01:48.369 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T05:01:48.369 INFO:journalctl@ceph.mon.c.smithi115.stdout: During handling of the above exception, another exception occurred: 2022-05-14T05:01:48.369 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T05:01:48.370 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T05:01:48.370 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T05:01:48.370 INFO:journalctl@ceph.mon.c.smithi115.stdout: return f(*arg) 2022-05-14T05:01:48.370 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T05:01:48.370 INFO:journalctl@ceph.mon.c.smithi115.stdout: self._write_client_files(client_files, host) 2022-05-14T05:01:48.371 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T05:01:48.371 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T05:01:48.371 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T05:01:48.371 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T05:01:48.372 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T05:01:48.372 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.event_loop.get_result(coro) 2022-05-14T05:01:48.372 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T05:01:48.372 INFO:journalctl@ceph.mon.c.smithi115.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T05:01:48.372 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T05:01:48.373 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.__get_result() 2022-05-14T05:01:48.373 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T05:01:48.373 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise self._exception 2022-05-14T05:01:48.374 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T05:01:48.375 INFO:journalctl@ceph.mon.c.smithi115.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T05:01:48.375 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T05:01:48.375 INFO:journalctl@ceph.mon.c.smithi115.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T05:01:48.375 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T05:01:48.376 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise 2022-05-14T05:01:48.376 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T05:01:48.376 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.gen.throw(type, value, traceback) 2022-05-14T05:01:48.376 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T05:01:48.376 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OrchestratorError(msg) 2022-05-14T05:01:48.377 INFO:journalctl@ceph.mon.c.smithi115.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T05:01:48.377 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:01:48 smithi115 ceph-mon[78593]: pgmap v10371: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:01:48.377 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:01:48 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 20040 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T05:01:48.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:01:48 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:01:48.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:01:48 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T05:01:48.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T05:01:48.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T05:01:48.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T05:01:48.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T05:01:48.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T05:01:48.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T05:01:48.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T05:01:48.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T05:01:48.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T05:01:48.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T05:01:48.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T05:01:48.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T05:01:48.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T05:01:48.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T05:01:48.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T05:01:48.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T05:01:48.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T05:01:48.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T05:01:48.548 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:01:48 smithi047 ceph-mon[77147]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T05:01:48.548 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T05:01:48.548 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T05:01:48.548 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T05:01:48.548 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T05:01:48.549 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T05:01:48.549 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T05:01:48.549 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T05:01:48.549 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T05:01:48.550 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T05:01:48.550 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T05:01:48.550 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T05:01:48.550 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T05:01:48.551 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T05:01:48.551 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T05:01:48.551 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T05:01:48.551 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T05:01:48.552 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T05:01:48.552 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T05:01:48.552 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T05:01:48.553 INFO:journalctl@ceph.mon.b.smithi047.stdout: During handling of the above exception, another exception occurred: 2022-05-14T05:01:48.553 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T05:01:48.553 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T05:01:48.554 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T05:01:48.554 INFO:journalctl@ceph.mon.b.smithi047.stdout: return f(*arg) 2022-05-14T05:01:48.554 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T05:01:48.554 INFO:journalctl@ceph.mon.b.smithi047.stdout: self._write_client_files(client_files, host) 2022-05-14T05:01:48.555 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T05:01:48.555 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T05:01:48.555 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T05:01:48.555 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T05:01:48.555 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T05:01:48.556 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.event_loop.get_result(coro) 2022-05-14T05:01:48.556 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T05:01:48.556 INFO:journalctl@ceph.mon.b.smithi047.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T05:01:48.556 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T05:01:48.556 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.__get_result() 2022-05-14T05:01:48.557 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T05:01:48.557 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise self._exception 2022-05-14T05:01:48.557 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T05:01:48.557 INFO:journalctl@ceph.mon.b.smithi047.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T05:01:48.558 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T05:01:48.558 INFO:journalctl@ceph.mon.b.smithi047.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T05:01:48.558 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T05:01:48.558 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise 2022-05-14T05:01:48.558 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T05:01:48.559 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.gen.throw(type, value, traceback) 2022-05-14T05:01:48.559 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T05:01:48.559 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OrchestratorError(msg) 2022-05-14T05:01:48.559 INFO:journalctl@ceph.mon.b.smithi047.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T05:01:48.559 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:01:48 smithi047 ceph-mon[77147]: pgmap v10371: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:01:48.560 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:01:48 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 20040 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T05:01:49.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:01:49 smithi115 conmon[90393]: 2022-05-14T05:01:49.037+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:01:49.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:01:49 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:01:49.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:01:49 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:01:50.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:01:50 smithi115 conmon[90393]: 2022-05-14T05:01:50.027+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:01:50.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:01:50 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:01:50.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:01:50 smithi115 ceph-mon[78593]: pgmap v10372: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T05:01:50.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:01:50 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:01:50.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:01:50 smithi047 ceph-mon[77147]: pgmap v10372: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T05:01:51.360 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:01:51 smithi115 conmon[90393]: 2022-05-14T05:01:51.011+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:01:51.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:01:51 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:01:51.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:01:51 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T05:01:51.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:01:51 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T05:01:51.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:01:51 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:01:51.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:01:51 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T05:01:51.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:01:51 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T05:01:52.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:01:52 smithi115 conmon[90393]: 2022-05-14T05:01:52.058+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:01:52.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:01:52 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:01:52.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:01:52 smithi115 ceph-mon[78593]: pgmap v10373: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:01:52.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:01:52 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:01:52.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:01:52 smithi047 ceph-mon[77147]: pgmap v10373: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:01:53.360 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:01:53 smithi115 conmon[90393]: 2022-05-14T05:01:53.100+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:01:53.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:01:53 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:01:53.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:01:53 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 20045 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T05:01:53.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:01:53 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:01:53.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:01:53 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 20045 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T05:01:54.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:01:54 smithi115 conmon[90393]: 2022-05-14T05:01:54.072+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:01:54.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:01:54 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:01:54.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:01:54 smithi115 ceph-mon[78593]: pgmap v10374: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:01:54.545 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:01:54 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:01:54.550 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:01:54 smithi047 ceph-mon[77147]: pgmap v10374: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:01:55.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:01:55 smithi115 conmon[90393]: 2022-05-14T05:01:55.075+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:01:55.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:01:55 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:01:55.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:01:55 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T05:01:55.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:01:55 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T05:01:55.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:01:55 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:01:55.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:01:55 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T05:01:55.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:01:55 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T05:01:56.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:01:56 smithi115 conmon[90393]: 2022-05-14T05:01:56.100+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:01:56.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:01:56 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:01:56.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:01:56 smithi115 ceph-mon[78593]: pgmap v10375: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T05:01:56.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:01:56 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:01:56.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:01:56 smithi047 ceph-mon[77147]: pgmap v10375: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T05:01:57.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:01:57 smithi115 conmon[90393]: 2022-05-14T05:01:57.080+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:01:57.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:01:57 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:01:57.546 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:01:57 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:01:58.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:01:58 smithi115 conmon[90393]: 2022-05-14T05:01:58.051+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:01:58.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:01:58 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:01:58.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:01:58 smithi115 ceph-mon[78593]: pgmap v10376: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:01:58.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:01:58 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 20050 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T05:01:58.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:01:58 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:01:58.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:01:58 smithi047 ceph-mon[77147]: pgmap v10376: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:01:58.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:01:58 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 20050 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T05:01:59.360 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:01:59 smithi115 conmon[90393]: 2022-05-14T05:01:59.066+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:01:59.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:01:59 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:01:59.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:01:59 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T05:01:59.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:01:59 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T05:01:59.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:01:59 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:01:59.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:01:59 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T05:01:59.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:01:59 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T05:02:00.360 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:02:00 smithi115 conmon[90393]: 2022-05-14T05:02:00.033+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:02:00.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:02:00 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:02:00.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:02:00 smithi115 ceph-mon[78593]: pgmap v10377: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T05:02:00.547 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:02:00 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:02:00.547 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:02:00 smithi047 ceph-mon[77147]: pgmap v10377: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T05:02:01.361 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:02:01 smithi115 conmon[90393]: 2022-05-14T05:02:01.014+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:02:01.362 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:02:01 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:02:01.545 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:02:01 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:02:02.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:02:01 smithi115 conmon[90393]: 2022-05-14T05:02:01.999+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:02:02.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:02:02 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:02:02.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:02:02 smithi115 ceph-mon[78593]: pgmap v10378: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:02:02.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:02:02 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:02:02.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:02:02 smithi047 ceph-mon[77147]: pgmap v10378: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:02:03.254 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:02:02 smithi115 conmon[90393]: 2022-05-14T05:02:02.989+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:02:03.546 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:02:03 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:02:03.546 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:02:03 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T05:02:03.546 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:02:03 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T05:02:03.547 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:02:03 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 20055 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T05:02:03.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:02:03 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:02:03.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:02:03 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T05:02:03.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:02:03 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T05:02:03.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:02:03 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 20055 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T05:02:04.362 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:02:04 smithi115 conmon[90393]: 2022-05-14T05:02:04.033+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:02:04.362 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:02:04 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:02:04.363 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:02:04 smithi115 ceph-mon[78593]: pgmap v10379: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:02:04.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:02:04 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:02:04.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:02:04 smithi047 ceph-mon[77147]: pgmap v10379: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:02:05.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:02:05 smithi115 conmon[90393]: 2022-05-14T05:02:05.021+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:02:05.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:02:05 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:02:05.545 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:02:05 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:02:06.264 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:02:05 smithi115 conmon[90393]: 2022-05-14T05:02:05.972+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:02:06.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:02:06 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:02:06.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:02:06 smithi047 ceph-mon[77147]: pgmap v10380: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T05:02:06.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:02:06 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:02:06.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:02:06 smithi115 ceph-mon[78593]: pgmap v10380: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T05:02:07.266 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:02:06 smithi115 conmon[90393]: 2022-05-14T05:02:06.961+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:02:07.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:02:07 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:02:07.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:02:07 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:02:08.270 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:02:07 smithi115 conmon[90393]: 2022-05-14T05:02:07.961+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:02:08.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:02:08 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:02:08.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:02:08 smithi047 ceph-mon[77147]: pgmap v10381: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:02:08.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:02:08 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 20060 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T05:02:08.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:02:08 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:02:08.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:02:08 smithi115 ceph-mon[78593]: pgmap v10381: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:02:08.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:02:08 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 20060 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T05:02:09.272 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:02:09 smithi115 conmon[90393]: 2022-05-14T05:02:09.002+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:02:09.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:02:09 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:02:09.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:02:09 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:02:10.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:02:10 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:02:10.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:02:10 smithi115 ceph-mon[78593]: pgmap v10382: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T05:02:10.361 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:02:10 smithi115 conmon[90393]: 2022-05-14T05:02:10.045+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:02:10.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:02:10 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:02:10.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:02:10 smithi047 ceph-mon[77147]: pgmap v10382: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T05:02:11.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:02:11 smithi115 conmon[90393]: 2022-05-14T05:02:11.050+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:02:11.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:02:11 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:02:11.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:02:11 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T05:02:11.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:02:11 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T05:02:11.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:02:11 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:02:11.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:02:11 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T05:02:11.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:02:11 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T05:02:12.362 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:02:12 smithi115 conmon[90393]: 2022-05-14T05:02:12.041+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:02:12.363 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:02:12 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:02:12.363 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:02:12 smithi115 ceph-mon[78593]: pgmap v10383: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:02:12.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:02:12 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:02:12.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:02:12 smithi047 ceph-mon[77147]: pgmap v10383: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:02:13.360 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:02:13 smithi115 conmon[90393]: 2022-05-14T05:02:13.055+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:02:13.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:02:13 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:02:13.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:02:13 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 20065 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T05:02:13.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:02:13 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:02:13.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:02:13 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 20065 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T05:02:14.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:02:14 smithi115 conmon[90393]: 2022-05-14T05:02:14.059+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:02:14.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:02:14 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:02:14.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:02:14 smithi115 ceph-mon[78593]: pgmap v10384: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:02:14.545 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:02:14 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:02:14.546 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:02:14 smithi047 ceph-mon[77147]: pgmap v10384: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:02:15.294 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:02:15 smithi115 conmon[90393]: 2022-05-14T05:02:15.020+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:02:15.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:02:15 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:02:15.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:02:15 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T05:02:15.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:02:15 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T05:02:15.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:02:15 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T05:02:15.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:02:15 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T05:02:15.612 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:02:15 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:02:15.613 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:02:15 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T05:02:15.613 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:02:15 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T05:02:15.614 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:02:15 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T05:02:15.614 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:02:15 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T05:02:16.296 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:02:15 smithi115 conmon[90393]: 2022-05-14T05:02:15.978+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:02:16.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:02:16 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:02:16.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:02:16 smithi115 ceph-mon[78593]: pgmap v10385: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T05:02:16.797 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:02:16 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:02:16.798 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:02:16 smithi047 ceph-mon[77147]: pgmap v10385: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T05:02:17.298 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:02:16 smithi115 conmon[90393]: 2022-05-14T05:02:16.967+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:02:17.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:02:17 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:02:17.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:02:17 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:02:18.308 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:02:17 smithi115 conmon[90393]: 2022-05-14T05:02:17.970+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:02:18.615 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:02:18 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:02:18.616 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:02:18 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T05:02:18.616 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:02:18 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T05:02:18.617 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:02:18 smithi115 ceph-mon[78593]: pgmap v10386: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:02:18.617 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:02:18 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 20070 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T05:02:18.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:02:18 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:02:18.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:02:18 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T05:02:18.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:02:18 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T05:02:18.795 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:02:18 smithi047 ceph-mon[77147]: pgmap v10386: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:02:18.795 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:02:18 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 20070 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T05:02:19.308 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:02:19 smithi115 conmon[90393]: 2022-05-14T05:02:19.020+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:02:19.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:02:19 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:02:19.796 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:02:19 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:02:20.313 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:02:20 smithi115 conmon[90393]: 2022-05-14T05:02:20.004+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:02:20.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:02:20 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:02:20.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:02:20 smithi115 ceph-mon[78593]: pgmap v10387: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T05:02:20.800 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:02:20 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:02:20.801 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:02:20 smithi047 ceph-mon[77147]: pgmap v10387: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T05:02:21.314 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:02:20 smithi115 conmon[90393]: 2022-05-14T05:02:20.992+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:02:21.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:02:21 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:02:21.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:02:21 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:02:22.323 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:02:22 smithi115 conmon[90393]: 2022-05-14T05:02:22.030+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:02:22.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:02:22 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:02:22.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:02:22 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T05:02:22.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:02:22 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T05:02:22.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:02:22 smithi115 ceph-mon[78593]: pgmap v10388: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:02:22.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:02:22 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:02:22.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:02:22 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T05:02:22.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:02:22 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T05:02:22.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:02:22 smithi047 ceph-mon[77147]: pgmap v10388: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:02:23.327 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:02:23 smithi115 conmon[90393]: 2022-05-14T05:02:23.053+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:02:23.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:02:23 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:02:23.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:02:23 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 20075 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T05:02:23.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:02:23 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:02:23.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:02:23 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 20075 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T05:02:24.362 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:02:24 smithi115 conmon[90393]: 2022-05-14T05:02:24.085+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:02:24.362 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:02:24 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:02:24.363 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:02:24 smithi115 ceph-mon[78593]: pgmap v10389: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:02:24.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:02:24 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:02:24.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:02:24 smithi047 ceph-mon[77147]: pgmap v10389: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:02:25.331 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:02:25 smithi115 conmon[90393]: 2022-05-14T05:02:25.063+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:02:25.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:02:25 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:02:25.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:02:25 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:02:26.336 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:02:26 smithi115 conmon[90393]: 2022-05-14T05:02:26.017+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:02:26.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:02:26 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:02:26.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:02:26 smithi115 ceph-mon[78593]: pgmap v10390: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T05:02:26.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:02:26 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:02:26.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:02:26 smithi047 ceph-mon[77147]: pgmap v10390: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T05:02:27.338 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:02:27 smithi115 conmon[90393]: 2022-05-14T05:02:27.055+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:02:27.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:02:27 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:02:27.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:02:27 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:02:28.342 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:02:28 smithi115 conmon[90393]: 2022-05-14T05:02:28.081+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:02:28.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:02:28 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:02:28.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:02:28 smithi115 ceph-mon[78593]: pgmap v10391: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:02:28.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:02:28 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 20080 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T05:02:28.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:02:28 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:02:28.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:02:28 smithi047 ceph-mon[77147]: pgmap v10391: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:02:28.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:02:28 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 20080 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T05:02:29.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:02:29 smithi115 conmon[90393]: 2022-05-14T05:02:29.107+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:02:29.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:02:29 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:02:29.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:02:29 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:02:30.353 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:02:30 smithi115 conmon[90393]: 2022-05-14T05:02:30.099+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:02:30.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:02:30 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:02:30.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:02:30 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T05:02:30.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:02:30 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T05:02:30.612 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:02:30 smithi115 ceph-mon[78593]: pgmap v10392: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T05:02:30.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:02:30 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:02:30.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:02:30 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T05:02:30.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:02:30 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T05:02:30.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:02:30 smithi047 ceph-mon[77147]: pgmap v10392: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T05:02:31.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:02:31 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:02:31.610 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:02:31 smithi115 conmon[90393]: 2022-05-14T05:02:31.129+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:02:31.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:02:31 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:02:32.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:02:32 smithi115 conmon[90393]: 2022-05-14T05:02:32.144+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:02:32.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:02:32 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:02:32.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:02:32 smithi115 ceph-mon[78593]: pgmap v10393: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:02:32.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:02:32 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:02:32.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:02:32 smithi047 ceph-mon[77147]: pgmap v10393: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:02:33.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:02:33 smithi115 conmon[90393]: 2022-05-14T05:02:33.126+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:02:33.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:02:33 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:02:33.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:02:33 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 20086 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T05:02:33.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:02:33 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:02:33.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:02:33 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 20086 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T05:02:34.613 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:02:34 smithi115 conmon[90393]: 2022-05-14T05:02:34.144+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:02:34.614 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:02:34 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:02:34.614 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:02:34 smithi115 ceph-mon[78593]: pgmap v10394: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:02:34.614 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:02:34 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T05:02:34.615 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:02:34 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T05:02:34.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:02:34 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:02:34.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:02:34 smithi047 ceph-mon[77147]: pgmap v10394: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:02:34.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:02:34 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T05:02:34.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:02:34 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T05:02:35.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:02:35 smithi115 conmon[90393]: 2022-05-14T05:02:35.154+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:02:35.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:02:35 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:02:35.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:02:35 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:02:36.610 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:02:36 smithi115 conmon[90393]: 2022-05-14T05:02:36.137+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:02:36.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:02:36 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:02:36.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:02:36 smithi115 ceph-mon[78593]: pgmap v10395: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T05:02:36.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:02:36 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:02:36.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:02:36 smithi047 ceph-mon[77147]: pgmap v10395: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T05:02:37.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:02:37 smithi115 conmon[90393]: 2022-05-14T05:02:37.101+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:02:37.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:02:37 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:02:37.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:02:37 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:02:38.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:02:38 smithi115 conmon[90393]: 2022-05-14T05:02:38.071+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:02:38.796 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:02:38 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:02:38.797 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:02:38 smithi047 ceph-mon[77147]: pgmap v10396: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:02:38.797 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:02:38 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T05:02:38.798 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:02:38 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T05:02:38.798 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:02:38 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 20090 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T05:02:38.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:02:38 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:02:38.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:02:38 smithi115 ceph-mon[78593]: pgmap v10396: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:02:38.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:02:38 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T05:02:38.862 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:02:38 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T05:02:38.862 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:02:38 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 20090 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T05:02:39.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:02:39 smithi115 conmon[90393]: 2022-05-14T05:02:39.070+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:02:39.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:02:39 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:02:39.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:02:39 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:02:40.363 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:02:40 smithi115 conmon[90393]: 2022-05-14T05:02:40.064+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:02:40.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:02:40 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:02:40.795 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:02:40 smithi047 ceph-mon[77147]: pgmap v10397: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T05:02:40.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:02:40 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:02:40.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:02:40 smithi115 ceph-mon[78593]: pgmap v10397: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T05:02:41.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:02:41 smithi115 conmon[90393]: 2022-05-14T05:02:41.078+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:02:41.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:02:41 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:02:41.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:02:41 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:02:42.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:02:42 smithi115 conmon[90393]: 2022-05-14T05:02:42.054+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:02:42.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:02:42 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:02:42.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:02:42 smithi047 ceph-mon[77147]: pgmap v10398: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:02:42.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:02:42 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T05:02:42.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:02:42 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T05:02:42.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:02:42 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:02:42.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:02:42 smithi115 ceph-mon[78593]: pgmap v10398: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:02:42.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:02:42 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T05:02:42.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:02:42 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T05:02:43.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:02:43 smithi115 conmon[90393]: 2022-05-14T05:02:43.005+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:02:43.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:02:43 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:02:43.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:02:43 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 20095 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T05:02:43.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:02:43 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:02:43.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:02:43 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 20095 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T05:02:44.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:02:44 smithi115 conmon[90393]: 2022-05-14T05:02:44.052+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:02:44.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:02:44 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:02:44.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:02:44 smithi047 ceph-mon[77147]: pgmap v10399: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:02:44.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:02:44 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:02:44.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:02:44 smithi115 ceph-mon[78593]: pgmap v10399: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:02:45.360 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:02:45 smithi115 conmon[90393]: 2022-05-14T05:02:45.004+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:02:45.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:02:45 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:02:45.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:02:45 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:02:46.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:02:46 smithi115 conmon[90393]: 2022-05-14T05:02:46.010+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:02:46.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:02:46 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:02:46.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:02:46 smithi047 ceph-mon[77147]: pgmap v10400: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T05:02:46.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:02:46 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:02:46.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:02:46 smithi115 ceph-mon[78593]: pgmap v10400: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T05:02:47.273 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:02:47 smithi115 conmon[90393]: 2022-05-14T05:02:47.013+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:02:47.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:02:47 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:02:47.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:02:47 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T05:02:47.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:02:47 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T05:02:47.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:02:47 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T05:02:47.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:02:47 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:02:47.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:02:47 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T05:02:47.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:02:47 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T05:02:47.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:02:47 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T05:02:48.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:02:47 smithi047 conmon[78521]: 2022-05-14T05:02:47.663+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T05:02:48.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:02:47 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T05:02:48.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:02:47 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T05:02:48.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:02:47 smithi047 conmon[78521]: yield 2022-05-14T05:02:48.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:02:47 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T05:02:48.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:02:47 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T05:02:48.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:02:47 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T05:02:48.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:02:47 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T05:02:48.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:02:47 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T05:02:48.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:02:47 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T05:02:48.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:02:47 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T05:02:48.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:02:47 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T05:02:48.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:02:47 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T05:02:48.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:02:47 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T05:02:48.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:02:47 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T05:02:48.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:02:47 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T05:02:48.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:02:47 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T05:02:48.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:02:47 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T05:02:48.056 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:02:47 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T05:02:48.360 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:02:48 smithi115 conmon[90393]: 2022-05-14T05:02:48.056+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:02:48.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:02:48 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:02:48.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:02:48 smithi047 ceph-mon[77147]: pgmap v10401: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:02:48.795 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:02:48 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T05:02:48.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T05:02:48.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T05:02:48.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T05:02:48.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T05:02:48.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T05:02:48.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T05:02:48.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T05:02:48.797 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T05:02:48.797 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T05:02:48.798 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T05:02:48.798 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T05:02:48.798 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T05:02:48.798 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T05:02:48.799 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T05:02:48.799 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T05:02:48.799 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T05:02:48.799 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T05:02:48.800 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T05:02:48.800 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:02:48 smithi047 ceph-mon[77147]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T05:02:48.801 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:02:48 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 20100 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T05:02:48.862 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:02:48 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:02:48.864 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:02:48 smithi115 ceph-mon[78593]: pgmap v10401: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:02:48.864 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:02:48 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T05:02:48.864 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T05:02:48.865 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T05:02:48.865 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T05:02:48.865 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T05:02:48.866 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T05:02:48.866 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T05:02:48.866 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T05:02:48.867 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T05:02:48.867 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T05:02:48.867 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T05:02:48.868 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T05:02:48.868 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T05:02:48.869 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T05:02:48.869 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T05:02:48.869 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T05:02:48.870 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T05:02:48.870 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T05:02:48.871 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T05:02:48.871 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:02:48 smithi115 ceph-mon[78593]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T05:02:48.871 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:02:48 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 20100 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T05:02:49.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:02:49 smithi115 conmon[90393]: 2022-05-14T05:02:49.067+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:02:49.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:02:49 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:02:49.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:02:49 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:02:50.415 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:02:50 smithi115 conmon[90393]: 2022-05-14T05:02:50.108+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:02:50.739 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:02:50 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:02:50.739 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:02:50 smithi047 ceph-mon[77147]: pgmap v10402: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T05:02:50.739 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:02:50 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T05:02:50.740 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:02:50 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T05:02:50.862 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:02:50 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:02:50.862 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:02:50 smithi115 ceph-mon[78593]: pgmap v10402: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T05:02:50.863 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:02:50 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T05:02:50.863 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:02:50 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T05:02:51.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:02:50 smithi047 conmon[78521]: 2022-05-14T05:02:50.735+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T05:02:51.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:02:50 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T05:02:51.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:02:50 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T05:02:51.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:02:50 smithi047 conmon[78521]: yield 2022-05-14T05:02:51.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:02:50 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T05:02:51.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:02:50 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T05:02:51.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:02:50 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T05:02:51.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:02:50 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T05:02:51.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:02:50 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T05:02:51.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:02:50 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T05:02:51.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:02:50 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T05:02:51.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:02:50 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T05:02:51.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:02:50 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T05:02:51.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:02:50 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T05:02:51.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:02:50 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T05:02:51.048 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:02:50 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T05:02:51.048 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:02:50 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T05:02:51.048 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:02:50 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T05:02:51.049 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:02:50 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T05:02:51.049 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:02:50 smithi047 conmon[78521]: 2022-05-14T05:02:50.736+0000 7f5344781700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T05:02:51.050 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:02:50 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T05:02:51.051 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:02:50 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T05:02:51.051 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:02:50 smithi047 conmon[78521]: yield 2022-05-14T05:02:51.052 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:02:50 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T05:02:51.052 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:02:50 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T05:02:51.053 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:02:50 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T05:02:51.053 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:02:50 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T05:02:51.054 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:02:50 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T05:02:51.054 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:02:50 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T05:02:51.056 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:02:50 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T05:02:51.057 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:02:50 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T05:02:51.057 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:02:50 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T05:02:51.058 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:02:50 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T05:02:51.058 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:02:50 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T05:02:51.058 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:02:50 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T05:02:51.060 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:02:50 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T05:02:51.061 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:02:50 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T05:02:51.061 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:02:50 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T05:02:51.061 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:02:50 smithi047 conmon[78521]: 2022-05-14T05:02:51.062 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:02:50 smithi047 conmon[78521]: During handling of the above exception, another exception occurred: 2022-05-14T05:02:51.062 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:02:50 smithi047 conmon[78521]: 2022-05-14T05:02:51.062 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:02:50 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T05:02:51.062 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:02:50 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T05:02:51.063 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:02:50 smithi047 conmon[78521]: return f(*arg) 2022-05-14T05:02:51.063 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:02:50 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T05:02:51.063 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:02:50 smithi047 conmon[78521]: self._write_client_files(client_files, host) 2022-05-14T05:02:51.063 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:02:50 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T05:02:51.063 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:02:50 smithi047 conmon[78521]: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T05:02:51.064 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:02:50 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T05:02:51.064 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:02:50 smithi047 conmon[78521]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T05:02:51.064 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:02:50 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T05:02:51.064 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:02:50 smithi047 conmon[78521]: return self.event_loop.get_result(coro) 2022-05-14T05:02:51.064 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:02:50 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T05:02:51.065 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:02:50 smithi047 conmon[78521]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T05:02:51.065 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:02:50 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T05:02:51.065 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:02:50 smithi047 conmon[78521]: return self.__get_result() 2022-05-14T05:02:51.065 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:02:50 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T05:02:51.066 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:02:50 smithi047 conmon[78521]: raise self._exception 2022-05-14T05:02:51.066 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:02:50 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T05:02:51.067 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:02:50 smithi047 conmon[78521]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T05:02:51.067 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:02:50 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T05:02:51.067 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:02:50 smithi047 conmon[78521]: conn = await self._remote_connection(host, addr) 2022-05-14T05:02:51.067 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:02:50 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T05:02:51.068 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:02:50 smithi047 conmon[78521]: raise 2022-05-14T05:02:51.068 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:02:50 smithi047 conmon[78521]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T05:02:51.068 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:02:50 smithi047 conmon[78521]: self.gen.throw(type, value, traceback) 2022-05-14T05:02:51.068 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:02:50 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T05:02:51.069 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:02:50 smithi047 conmon[78521]: raise OrchestratorError(msg) 2022-05-14T05:02:51.069 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:02:50 smithi047 conmon[78521]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T05:02:51.421 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:02:51 smithi115 conmon[90393]: 2022-05-14T05:02:51.149+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:02:51.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:02:51 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:02:51.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:02:51 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T05:02:51.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T05:02:51.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T05:02:51.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T05:02:51.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T05:02:51.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T05:02:51.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T05:02:51.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T05:02:51.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T05:02:51.797 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T05:02:51.797 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T05:02:51.797 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T05:02:51.798 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T05:02:51.798 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T05:02:51.799 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T05:02:51.799 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T05:02:51.799 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T05:02:51.800 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T05:02:51.800 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T05:02:51.801 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:02:51 smithi047 ceph-mon[77147]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T05:02:51.801 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T05:02:51.801 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T05:02:51.801 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T05:02:51.802 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T05:02:51.802 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T05:02:51.802 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T05:02:51.802 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T05:02:51.803 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T05:02:51.803 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T05:02:51.803 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T05:02:51.803 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T05:02:51.804 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T05:02:51.804 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T05:02:51.804 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T05:02:51.804 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T05:02:51.804 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T05:02:51.806 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T05:02:51.806 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T05:02:51.807 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T05:02:51.807 INFO:journalctl@ceph.mon.b.smithi047.stdout: During handling of the above exception, another exception occurred: 2022-05-14T05:02:51.807 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T05:02:51.808 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T05:02:51.808 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T05:02:51.808 INFO:journalctl@ceph.mon.b.smithi047.stdout: return f(*arg) 2022-05-14T05:02:51.809 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T05:02:51.809 INFO:journalctl@ceph.mon.b.smithi047.stdout: self._write_client_files(client_files, host) 2022-05-14T05:02:51.809 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T05:02:51.810 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T05:02:51.810 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T05:02:51.810 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T05:02:51.811 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T05:02:51.811 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.event_loop.get_result(coro) 2022-05-14T05:02:51.812 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T05:02:51.812 INFO:journalctl@ceph.mon.b.smithi047.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T05:02:51.812 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T05:02:51.813 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.__get_result() 2022-05-14T05:02:51.813 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T05:02:51.814 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise self._exception 2022-05-14T05:02:51.815 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T05:02:51.815 INFO:journalctl@ceph.mon.b.smithi047.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T05:02:51.816 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T05:02:51.816 INFO:journalctl@ceph.mon.b.smithi047.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T05:02:51.816 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T05:02:51.817 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise 2022-05-14T05:02:51.817 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T05:02:51.818 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.gen.throw(type, value, traceback) 2022-05-14T05:02:51.818 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T05:02:51.818 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OrchestratorError(msg) 2022-05-14T05:02:51.818 INFO:journalctl@ceph.mon.b.smithi047.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T05:02:51.862 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:02:51 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:02:51.862 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:02:51 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T05:02:51.863 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T05:02:51.871 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T05:02:51.872 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T05:02:51.872 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T05:02:51.873 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T05:02:51.873 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T05:02:51.874 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T05:02:51.874 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T05:02:51.875 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T05:02:51.875 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T05:02:51.875 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T05:02:51.876 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T05:02:51.877 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T05:02:51.877 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T05:02:51.878 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T05:02:51.878 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T05:02:51.879 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T05:02:51.879 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T05:02:51.880 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:02:51 smithi115 ceph-mon[78593]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T05:02:51.880 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T05:02:51.880 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T05:02:51.881 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T05:02:51.882 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T05:02:51.882 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T05:02:51.883 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T05:02:51.883 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T05:02:51.883 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T05:02:51.884 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T05:02:51.884 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T05:02:51.885 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T05:02:51.885 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T05:02:51.886 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T05:02:51.886 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T05:02:51.886 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T05:02:51.887 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T05:02:51.887 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T05:02:51.888 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T05:02:51.888 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T05:02:51.889 INFO:journalctl@ceph.mon.c.smithi115.stdout: During handling of the above exception, another exception occurred: 2022-05-14T05:02:51.889 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T05:02:51.889 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T05:02:51.890 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T05:02:51.890 INFO:journalctl@ceph.mon.c.smithi115.stdout: return f(*arg) 2022-05-14T05:02:51.891 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T05:02:51.892 INFO:journalctl@ceph.mon.c.smithi115.stdout: self._write_client_files(client_files, host) 2022-05-14T05:02:51.892 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T05:02:51.892 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T05:02:51.893 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T05:02:51.893 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T05:02:51.894 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T05:02:51.894 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.event_loop.get_result(coro) 2022-05-14T05:02:51.895 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T05:02:51.895 INFO:journalctl@ceph.mon.c.smithi115.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T05:02:51.895 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T05:02:51.896 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.__get_result() 2022-05-14T05:02:51.896 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T05:02:51.897 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise self._exception 2022-05-14T05:02:51.897 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T05:02:51.898 INFO:journalctl@ceph.mon.c.smithi115.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T05:02:51.898 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T05:02:51.898 INFO:journalctl@ceph.mon.c.smithi115.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T05:02:51.899 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T05:02:51.899 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise 2022-05-14T05:02:51.900 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T05:02:51.900 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.gen.throw(type, value, traceback) 2022-05-14T05:02:51.900 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T05:02:51.901 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OrchestratorError(msg) 2022-05-14T05:02:51.901 INFO:journalctl@ceph.mon.c.smithi115.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T05:02:52.424 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:02:52 smithi115 conmon[90393]: 2022-05-14T05:02:52.128+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:02:52.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:02:52 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:02:52.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:02:52 smithi047 ceph-mon[77147]: pgmap v10403: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:02:52.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:02:52 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:02:52.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:02:52 smithi115 ceph-mon[78593]: pgmap v10403: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:02:53.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:02:53 smithi115 conmon[90393]: 2022-05-14T05:02:53.104+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:02:53.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:02:53 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:02:53.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:02:53 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 20105 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T05:02:53.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:02:53 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:02:53.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:02:53 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 20105 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T05:02:54.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:02:54 smithi115 conmon[90393]: 2022-05-14T05:02:54.061+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:02:54.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:02:54 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:02:54.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:02:54 smithi047 ceph-mon[77147]: pgmap v10404: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:02:54.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:02:54 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T05:02:54.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:02:54 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T05:02:54.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:02:54 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:02:54.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:02:54 smithi115 ceph-mon[78593]: pgmap v10404: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:02:54.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:02:54 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T05:02:54.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:02:54 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T05:02:55.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:02:55 smithi115 conmon[90393]: 2022-05-14T05:02:55.047+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:02:55.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:02:55 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:02:55.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:02:55 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:02:56.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:02:56 smithi115 conmon[90393]: 2022-05-14T05:02:56.002+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:02:56.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:02:56 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:02:56.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:02:56 smithi047 ceph-mon[77147]: pgmap v10405: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T05:02:56.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:02:56 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:02:56.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:02:56 smithi115 ceph-mon[78593]: pgmap v10405: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T05:02:57.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:02:57 smithi115 conmon[90393]: 2022-05-14T05:02:57.037+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:02:57.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:02:57 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:02:57.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:02:57 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:02:58.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:02:58 smithi115 conmon[90393]: 2022-05-14T05:02:58.031+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:02:58.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:02:58 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:02:58.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:02:58 smithi047 ceph-mon[77147]: pgmap v10406: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:02:58.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:02:58 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T05:02:58.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:02:58 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T05:02:58.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:02:58 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 20110 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T05:02:58.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:02:58 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:02:58.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:02:58 smithi115 ceph-mon[78593]: pgmap v10406: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:02:58.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:02:58 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T05:02:58.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:02:58 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T05:02:58.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:02:58 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 20110 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T05:02:59.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:02:59 smithi115 conmon[90393]: 2022-05-14T05:02:59.018+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:02:59.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:02:59 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:02:59.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:02:59 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:03:00.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:02:59 smithi115 conmon[90393]: 2022-05-14T05:02:59.968+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:03:00.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:03:00 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:03:00.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:03:00 smithi047 ceph-mon[77147]: pgmap v10407: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T05:03:00.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:03:00 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:03:00.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:03:00 smithi115 ceph-mon[78593]: pgmap v10407: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T05:03:01.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:03:00 smithi115 conmon[90393]: 2022-05-14T05:03:00.965+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:03:01.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:03:01 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:03:01.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:03:01 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:03:02.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:03:01 smithi115 conmon[90393]: 2022-05-14T05:03:01.945+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:03:02.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:03:02 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:03:02.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:03:02 smithi047 ceph-mon[77147]: pgmap v10408: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:03:02.795 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:03:02 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T05:03:02.795 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:03:02 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T05:03:02.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:03:02 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:03:02.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:03:02 smithi115 ceph-mon[78593]: pgmap v10408: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:03:02.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:03:02 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T05:03:02.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:03:02 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T05:03:03.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:03:02 smithi115 conmon[90393]: 2022-05-14T05:03:02.977+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:03:03.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:03:03 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:03:03.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:03:03 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 20115 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T05:03:03.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:03:03 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:03:03.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:03:03 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 20115 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T05:03:04.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:03:03 smithi115 conmon[90393]: 2022-05-14T05:03:03.969+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:03:04.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:03:04 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:03:04.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:03:04 smithi047 ceph-mon[77147]: pgmap v10409: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:03:04.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:03:04 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:03:04.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:03:04 smithi115 ceph-mon[78593]: pgmap v10409: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:03:05.360 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:03:04 smithi115 conmon[90393]: 2022-05-14T05:03:04.951+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:03:05.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:03:05 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:03:05.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:03:05 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:03:06.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:03:05 smithi115 conmon[90393]: 2022-05-14T05:03:05.913+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:03:06.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:03:06 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:03:06.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:03:06 smithi047 ceph-mon[77147]: pgmap v10410: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T05:03:06.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:03:06 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:03:06.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:03:06 smithi115 ceph-mon[78593]: pgmap v10410: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T05:03:07.360 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:03:06 smithi115 conmon[90393]: 2022-05-14T05:03:06.939+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:03:07.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:03:07 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:03:07.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:03:07 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:03:08.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:03:07 smithi115 conmon[90393]: 2022-05-14T05:03:07.920+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:03:08.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:03:08 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:03:08.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:03:08 smithi047 ceph-mon[77147]: pgmap v10411: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:03:08.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:03:08 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 20120 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T05:03:08.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:03:08 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:03:08.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:03:08 smithi115 ceph-mon[78593]: pgmap v10411: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:03:08.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:03:08 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 20120 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T05:03:09.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:03:08 smithi115 conmon[90393]: 2022-05-14T05:03:08.944+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:03:09.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:03:09 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:03:09.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:03:09 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:03:10.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:03:09 smithi115 conmon[90393]: 2022-05-14T05:03:09.940+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:03:10.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:03:10 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:03:10.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:03:10 smithi047 ceph-mon[77147]: pgmap v10412: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T05:03:10.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:03:10 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T05:03:10.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:03:10 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T05:03:10.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:03:10 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:03:10.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:03:10 smithi115 ceph-mon[78593]: pgmap v10412: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T05:03:10.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:03:10 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T05:03:10.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:03:10 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T05:03:11.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:03:10 smithi115 conmon[90393]: 2022-05-14T05:03:10.950+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:03:11.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:03:11 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:03:11.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:03:11 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:03:12.362 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:03:11 smithi115 conmon[90393]: 2022-05-14T05:03:11.993+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:03:12.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:03:12 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:03:12.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:03:12 smithi047 ceph-mon[77147]: pgmap v10413: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:03:12.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:03:12 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:03:12.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:03:12 smithi115 ceph-mon[78593]: pgmap v10413: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:03:13.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:03:12 smithi115 conmon[90393]: 2022-05-14T05:03:12.967+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:03:13.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:03:13 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:03:13.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:03:13 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 20125 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T05:03:13.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:03:13 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:03:13.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:03:13 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 20125 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T05:03:14.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:03:14 smithi115 conmon[90393]: 2022-05-14T05:03:14.002+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:03:14.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:03:14 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:03:14.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:03:14 smithi047 ceph-mon[77147]: pgmap v10414: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:03:14.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:03:14 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T05:03:14.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:03:14 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T05:03:14.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:03:14 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:03:14.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:03:14 smithi115 ceph-mon[78593]: pgmap v10414: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:03:14.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:03:14 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T05:03:14.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:03:14 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T05:03:15.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:03:14 smithi115 conmon[90393]: 2022-05-14T05:03:14.969+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:03:15.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:03:15 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:03:15.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:03:15 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T05:03:15.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:03:15 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T05:03:15.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:03:15 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:03:15.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:03:15 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T05:03:15.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:03:15 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T05:03:16.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:03:15 smithi115 conmon[90393]: 2022-05-14T05:03:15.967+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:03:16.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:03:16 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:03:16.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:03:16 smithi047 ceph-mon[77147]: pgmap v10415: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T05:03:16.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:03:16 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:03:16.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:03:16 smithi115 ceph-mon[78593]: pgmap v10415: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T05:03:17.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:03:16 smithi115 conmon[90393]: 2022-05-14T05:03:16.967+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:03:17.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:03:17 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:03:17.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:03:17 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:03:18.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:03:17 smithi115 conmon[90393]: 2022-05-14T05:03:17.920+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:03:18.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:03:18 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:03:18.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:03:18 smithi047 ceph-mon[77147]: pgmap v10416: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:03:18.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:03:18 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T05:03:18.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:03:18 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T05:03:18.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:03:18 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 20130 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T05:03:18.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:03:18 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:03:18.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:03:18 smithi115 ceph-mon[78593]: pgmap v10416: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:03:18.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:03:18 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T05:03:18.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:03:18 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T05:03:18.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:03:18 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 20130 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T05:03:19.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:03:18 smithi115 conmon[90393]: 2022-05-14T05:03:18.957+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:03:19.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:03:19 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:03:19.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:03:19 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:03:20.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:03:19 smithi115 conmon[90393]: 2022-05-14T05:03:19.951+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:03:20.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:03:20 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:03:20.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:03:20 smithi047 ceph-mon[77147]: pgmap v10417: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T05:03:20.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:03:20 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:03:20.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:03:20 smithi115 ceph-mon[78593]: pgmap v10417: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T05:03:21.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:03:20 smithi115 conmon[90393]: 2022-05-14T05:03:20.992+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:03:21.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:03:21 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:03:21.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:03:21 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:03:22.361 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:03:21 smithi115 conmon[90393]: 2022-05-14T05:03:21.995+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:03:22.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:03:22 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:03:22.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:03:22 smithi047 ceph-mon[77147]: pgmap v10418: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:03:22.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:03:22 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T05:03:22.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:03:22 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T05:03:22.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:03:22 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:03:22.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:03:22 smithi115 ceph-mon[78593]: pgmap v10418: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:03:22.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:03:22 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T05:03:22.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:03:22 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T05:03:23.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:03:23 smithi115 conmon[90393]: 2022-05-14T05:03:23.005+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:03:23.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:03:23 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:03:23.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:03:23 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 20135 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T05:03:23.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:03:23 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:03:23.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:03:23 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 20135 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T05:03:24.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:03:23 smithi115 conmon[90393]: 2022-05-14T05:03:23.985+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:03:24.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:03:24 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:03:24.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:03:24 smithi047 ceph-mon[77147]: pgmap v10419: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:03:24.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:03:24 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:03:24.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:03:24 smithi115 ceph-mon[78593]: pgmap v10419: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:03:25.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:03:25 smithi115 conmon[90393]: 2022-05-14T05:03:25.016+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:03:25.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:03:25 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:03:25.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:03:25 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:03:26.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:03:25 smithi115 conmon[90393]: 2022-05-14T05:03:25.987+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:03:26.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:03:26 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:03:26.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:03:26 smithi047 ceph-mon[77147]: pgmap v10420: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T05:03:26.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:03:26 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:03:26.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:03:26 smithi115 ceph-mon[78593]: pgmap v10420: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T05:03:27.360 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:03:26 smithi115 conmon[90393]: 2022-05-14T05:03:26.964+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:03:27.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:03:27 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:03:27.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:03:27 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:03:28.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:03:27 smithi115 conmon[90393]: 2022-05-14T05:03:27.943+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:03:28.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:03:28 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:03:28.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:03:28 smithi047 ceph-mon[77147]: pgmap v10421: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:03:28.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:03:28 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 20140 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T05:03:28.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:03:28 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:03:28.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:03:28 smithi115 ceph-mon[78593]: pgmap v10421: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:03:28.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:03:28 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 20140 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T05:03:29.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:03:28 smithi115 conmon[90393]: 2022-05-14T05:03:28.910+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:03:29.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:03:29 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:03:29.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:03:29 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:03:30.360 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:03:29 smithi115 conmon[90393]: 2022-05-14T05:03:29.946+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:03:30.795 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:03:30 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:03:30.795 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:03:30 smithi047 ceph-mon[77147]: pgmap v10422: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T05:03:30.795 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:03:30 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T05:03:30.796 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:03:30 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T05:03:30.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:03:30 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:03:30.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:03:30 smithi115 ceph-mon[78593]: pgmap v10422: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T05:03:30.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:03:30 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T05:03:30.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:03:30 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T05:03:31.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:03:30 smithi115 conmon[90393]: 2022-05-14T05:03:30.923+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:03:31.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:03:31 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:03:31.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:03:31 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:03:32.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:03:31 smithi115 conmon[90393]: 2022-05-14T05:03:31.965+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:03:32.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:03:32 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:03:32.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:03:32 smithi047 ceph-mon[77147]: pgmap v10423: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:03:32.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:03:32 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:03:32.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:03:32 smithi115 ceph-mon[78593]: pgmap v10423: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:03:33.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:03:33 smithi115 conmon[90393]: 2022-05-14T05:03:33.015+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:03:33.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:03:33 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:03:33.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:03:33 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 20145 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T05:03:33.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:03:33 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:03:33.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:03:33 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 20145 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T05:03:34.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:03:34 smithi115 conmon[90393]: 2022-05-14T05:03:34.031+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:03:34.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:03:34 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:03:34.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:03:34 smithi047 ceph-mon[77147]: pgmap v10424: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:03:34.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:03:34 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T05:03:34.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:03:34 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T05:03:34.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:03:34 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:03:34.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:03:34 smithi115 ceph-mon[78593]: pgmap v10424: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:03:34.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:03:34 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T05:03:34.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:03:34 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T05:03:35.360 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:03:35 smithi115 conmon[90393]: 2022-05-14T05:03:35.023+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:03:35.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:03:35 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:03:36.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:03:35 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:03:36.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:03:36 smithi115 conmon[90393]: 2022-05-14T05:03:36.029+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:03:36.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:03:36 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:03:36.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:03:36 smithi115 ceph-mon[78593]: pgmap v10425: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T05:03:37.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:03:36 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:03:37.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:03:36 smithi047 ceph-mon[77147]: pgmap v10425: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T05:03:37.360 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:03:37 smithi115 conmon[90393]: 2022-05-14T05:03:37.015+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:03:37.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:03:37 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:03:38.046 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:03:37 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:03:38.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:03:38 smithi115 conmon[90393]: 2022-05-14T05:03:38.047+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:03:38.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:03:38 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:03:38.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:03:38 smithi115 ceph-mon[78593]: pgmap v10426: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:03:38.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:03:38 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T05:03:38.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:03:38 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T05:03:38.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:03:38 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 20150 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T05:03:39.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:03:38 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:03:39.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:03:38 smithi047 ceph-mon[77147]: pgmap v10426: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:03:39.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:03:38 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T05:03:39.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:03:38 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T05:03:39.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:03:38 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 20150 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T05:03:39.360 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:03:39 smithi115 conmon[90393]: 2022-05-14T05:03:39.025+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:03:39.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:03:39 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:03:40.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:03:39 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:03:40.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:03:40 smithi115 conmon[90393]: 2022-05-14T05:03:40.059+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:03:40.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:03:40 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:03:40.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:03:40 smithi115 ceph-mon[78593]: pgmap v10427: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T05:03:41.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:03:40 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:03:41.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:03:40 smithi047 ceph-mon[77147]: pgmap v10427: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T05:03:41.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:03:41 smithi115 conmon[90393]: 2022-05-14T05:03:41.057+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:03:41.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:03:41 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:03:42.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:03:41 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:03:42.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:03:42 smithi115 conmon[90393]: 2022-05-14T05:03:42.012+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:03:42.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:03:42 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:03:42.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:03:42 smithi115 ceph-mon[78593]: pgmap v10428: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:03:42.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:03:42 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T05:03:42.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:03:42 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T05:03:43.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:03:42 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:03:43.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:03:42 smithi047 ceph-mon[77147]: pgmap v10428: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:03:43.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:03:42 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T05:03:43.045 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:03:42 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T05:03:43.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:03:42 smithi115 conmon[90393]: 2022-05-14T05:03:42.962+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:03:43.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:03:43 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:03:43.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:03:43 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 20155 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T05:03:44.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:03:43 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:03:44.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:03:43 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 20155 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T05:03:44.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:03:44 smithi115 conmon[90393]: 2022-05-14T05:03:44.002+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:03:44.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:03:44 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:03:44.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:03:44 smithi115 ceph-mon[78593]: pgmap v10429: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:03:45.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:03:44 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:03:45.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:03:44 smithi047 ceph-mon[77147]: pgmap v10429: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:03:45.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:03:44 smithi115 conmon[90393]: 2022-05-14T05:03:44.965+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:03:45.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:03:45 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:03:46.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:03:45 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:03:46.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:03:45 smithi115 conmon[90393]: 2022-05-14T05:03:45.932+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:03:46.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:03:46 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:03:46.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:03:46 smithi115 ceph-mon[78593]: pgmap v10430: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T05:03:47.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:03:46 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:03:47.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:03:46 smithi047 ceph-mon[77147]: pgmap v10430: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T05:03:47.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:03:46 smithi115 conmon[90393]: 2022-05-14T05:03:46.957+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:03:47.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:03:47 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:03:48.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:03:47 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:03:48.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:03:47 smithi115 conmon[90393]: 2022-05-14T05:03:47.944+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:03:48.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:03:48 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:03:48.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:03:48 smithi115 ceph-mon[78593]: pgmap v10431: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:03:48.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:03:48 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 20160 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T05:03:49.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:03:48 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:03:49.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:03:48 smithi047 ceph-mon[77147]: pgmap v10431: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:03:49.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:03:48 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 20160 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T05:03:49.360 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:03:48 smithi115 conmon[90393]: 2022-05-14T05:03:48.967+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:03:49.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:03:49 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:03:50.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:03:49 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:03:50.362 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:03:49 smithi115 conmon[90393]: 2022-05-14T05:03:49.919+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:03:50.842 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:03:50 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:03:50.842 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:03:50 smithi047 ceph-mon[77147]: pgmap v10432: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T05:03:50.842 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:03:50 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T05:03:50.843 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:03:50 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T05:03:50.843 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:03:50 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:03:50.843 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:03:50 smithi115 ceph-mon[78593]: pgmap v10432: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T05:03:50.844 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:03:50 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T05:03:50.844 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:03:50 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T05:03:51.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:03:50 smithi115 conmon[90393]: 2022-05-14T05:03:50.963+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:03:51.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:03:51 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:03:51.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:03:51 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T05:03:51.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:03:51 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T05:03:51.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:03:51 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T05:03:52.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:03:51 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:03:52.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:03:51 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T05:03:52.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:03:51 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T05:03:52.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:03:51 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T05:03:52.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:03:51 smithi115 conmon[90393]: 2022-05-14T05:03:51.980+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:03:52.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:03:52 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:03:52.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:03:52 smithi115 ceph-mon[78593]: pgmap v10433: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:03:53.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:03:52 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:03:53.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:03:52 smithi047 ceph-mon[77147]: pgmap v10433: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:03:53.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:03:52 smithi115 conmon[90393]: 2022-05-14T05:03:52.935+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:03:53.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:03:53 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:03:53.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:03:53 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 20165 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T05:03:54.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:03:53 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:03:54.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:03:53 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 20165 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T05:03:54.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:03:53 smithi047 conmon[78521]: 2022-05-14T05:03:53.840+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T05:03:54.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:03:53 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T05:03:54.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:03:53 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T05:03:54.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:03:53 smithi047 conmon[78521]: yield 2022-05-14T05:03:54.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:03:53 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T05:03:54.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:03:53 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T05:03:54.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:03:53 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T05:03:54.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:03:53 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T05:03:54.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:03:53 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T05:03:54.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:03:53 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T05:03:54.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:03:53 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T05:03:54.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:03:53 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T05:03:54.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:03:53 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T05:03:54.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:03:53 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T05:03:54.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:03:53 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T05:03:54.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:03:53 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T05:03:54.048 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:03:53 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T05:03:54.048 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:03:53 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T05:03:54.048 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:03:53 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T05:03:54.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:03:53 smithi115 conmon[90393]: 2022-05-14T05:03:53.958+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:03:54.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:03:54 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:03:54.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:03:54 smithi115 ceph-mon[78593]: pgmap v10434: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:03:54.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:03:54 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T05:03:54.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T05:03:54.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T05:03:54.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T05:03:54.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T05:03:54.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T05:03:54.863 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T05:03:54.863 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T05:03:54.863 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T05:03:54.863 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T05:03:54.864 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T05:03:54.864 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T05:03:54.864 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T05:03:54.864 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T05:03:54.864 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T05:03:54.865 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T05:03:54.865 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T05:03:54.865 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T05:03:54.865 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T05:03:54.866 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:03:54 smithi115 ceph-mon[78593]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T05:03:54.866 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:03:54 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T05:03:54.866 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:03:54 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T05:03:55.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:03:54 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:03:55.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:03:54 smithi047 ceph-mon[77147]: pgmap v10434: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:03:55.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:03:54 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T05:03:55.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T05:03:55.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T05:03:55.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T05:03:55.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T05:03:55.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T05:03:55.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T05:03:55.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T05:03:55.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T05:03:55.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T05:03:55.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T05:03:55.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T05:03:55.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T05:03:55.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T05:03:55.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T05:03:55.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T05:03:55.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T05:03:55.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T05:03:55.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T05:03:55.048 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:03:54 smithi047 ceph-mon[77147]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T05:03:55.048 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:03:54 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T05:03:55.048 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:03:54 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T05:03:55.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:03:54 smithi115 conmon[90393]: 2022-05-14T05:03:54.991+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:03:55.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:03:55 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:03:56.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:03:55 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:03:56.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:03:56 smithi115 conmon[90393]: 2022-05-14T05:03:56.035+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:03:56.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:03:56 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:03:56.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:03:56 smithi115 ceph-mon[78593]: pgmap v10435: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T05:03:56.915 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:03:56 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:03:56.915 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:03:56 smithi047 ceph-mon[77147]: pgmap v10435: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T05:03:57.293 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:03:56 smithi047 conmon[78521]: 2022-05-14T05:03:56.912+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T05:03:57.293 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:03:56 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T05:03:57.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:03:56 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T05:03:57.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:03:56 smithi047 conmon[78521]: yield 2022-05-14T05:03:57.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:03:56 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T05:03:57.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:03:56 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T05:03:57.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:03:56 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T05:03:57.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:03:56 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T05:03:57.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:03:56 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T05:03:57.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:03:56 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T05:03:57.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:03:56 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T05:03:57.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:03:56 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T05:03:57.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:03:56 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T05:03:57.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:03:56 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T05:03:57.297 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:03:56 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T05:03:57.297 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:03:56 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T05:03:57.297 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:03:56 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T05:03:57.297 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:03:56 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T05:03:57.298 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:03:56 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T05:03:57.298 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:03:56 smithi047 conmon[78521]: 2022-05-14T05:03:56.913+0000 7f5340779700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T05:03:57.298 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:03:56 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T05:03:57.298 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:03:56 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T05:03:57.298 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:03:56 smithi047 conmon[78521]: yield 2022-05-14T05:03:57.299 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:03:56 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T05:03:57.299 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:03:56 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T05:03:57.299 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:03:56 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T05:03:57.299 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:03:56 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T05:03:57.300 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:03:56 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T05:03:57.300 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:03:56 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T05:03:57.300 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:03:56 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T05:03:57.300 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:03:56 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T05:03:57.302 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:03:56 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T05:03:57.303 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:03:56 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T05:03:57.303 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:03:56 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T05:03:57.303 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:03:56 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T05:03:57.303 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:03:56 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T05:03:57.304 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:03:56 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T05:03:57.304 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:03:56 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T05:03:57.304 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:03:56 smithi047 conmon[78521]: 2022-05-14T05:03:57.304 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:03:56 smithi047 conmon[78521]: During handling of the above exception, another exception occurred: 2022-05-14T05:03:57.305 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:03:56 smithi047 conmon[78521]: 2022-05-14T05:03:57.305 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:03:56 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T05:03:57.305 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:03:56 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T05:03:57.306 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:03:56 smithi047 conmon[78521]: return f(*arg) 2022-05-14T05:03:57.306 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:03:56 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T05:03:57.306 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:03:56 smithi047 conmon[78521]: self._write_client_files(client_files, host) 2022-05-14T05:03:57.306 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:03:56 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T05:03:57.307 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:03:56 smithi047 conmon[78521]: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T05:03:57.307 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:03:56 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T05:03:57.307 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:03:56 smithi047 conmon[78521]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T05:03:57.308 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:03:56 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T05:03:57.308 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:03:56 smithi047 conmon[78521]: return self.event_loop.get_result(coro) 2022-05-14T05:03:57.308 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:03:56 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T05:03:57.308 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:03:56 smithi047 conmon[78521]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T05:03:57.309 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:03:56 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T05:03:57.309 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:03:56 smithi047 conmon[78521]: return self.__get_result() 2022-05-14T05:03:57.309 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:03:56 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T05:03:57.309 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:03:56 smithi047 conmon[78521]: raise self._exception 2022-05-14T05:03:57.309 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:03:56 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T05:03:57.310 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:03:56 smithi047 conmon[78521]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T05:03:57.310 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:03:56 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T05:03:57.310 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:03:56 smithi047 conmon[78521]: conn = await self._remote_connection(host, addr) 2022-05-14T05:03:57.310 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:03:56 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T05:03:57.311 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:03:56 smithi047 conmon[78521]: raise 2022-05-14T05:03:57.311 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:03:56 smithi047 conmon[78521]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T05:03:57.311 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:03:56 smithi047 conmon[78521]: self.gen.throw(type, value, traceback) 2022-05-14T05:03:57.311 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:03:56 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T05:03:57.312 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:03:56 smithi047 conmon[78521]: raise OrchestratorError(msg) 2022-05-14T05:03:57.312 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:03:56 smithi047 conmon[78521]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T05:03:57.362 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:03:57 smithi115 conmon[90393]: 2022-05-14T05:03:57.010+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:03:58.020 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:03:57 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:03:58.020 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:03:57 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T05:03:58.020 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T05:03:58.020 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T05:03:58.021 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T05:03:58.021 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T05:03:58.021 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T05:03:58.021 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T05:03:58.022 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T05:03:58.022 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T05:03:58.022 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T05:03:58.022 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T05:03:58.023 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T05:03:58.023 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T05:03:58.023 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T05:03:58.023 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T05:03:58.024 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T05:03:58.024 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T05:03:58.024 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T05:03:58.024 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T05:03:58.025 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:03:57 smithi115 ceph-mon[78593]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T05:03:58.025 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T05:03:58.025 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T05:03:58.025 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T05:03:58.025 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T05:03:58.026 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T05:03:58.026 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T05:03:58.026 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T05:03:58.026 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T05:03:58.027 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T05:03:58.027 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T05:03:58.027 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T05:03:58.027 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T05:03:58.028 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T05:03:58.028 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T05:03:58.028 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T05:03:58.028 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T05:03:58.028 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T05:03:58.029 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T05:03:58.029 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T05:03:58.029 INFO:journalctl@ceph.mon.c.smithi115.stdout: During handling of the above exception, another exception occurred: 2022-05-14T05:03:58.029 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T05:03:58.030 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T05:03:58.030 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T05:03:58.030 INFO:journalctl@ceph.mon.c.smithi115.stdout: return f(*arg) 2022-05-14T05:03:58.030 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T05:03:58.030 INFO:journalctl@ceph.mon.c.smithi115.stdout: self._write_client_files(client_files, host) 2022-05-14T05:03:58.031 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T05:03:58.031 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T05:03:58.031 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T05:03:58.031 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T05:03:58.032 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T05:03:58.032 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.event_loop.get_result(coro) 2022-05-14T05:03:58.032 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T05:03:58.032 INFO:journalctl@ceph.mon.c.smithi115.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T05:03:58.033 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T05:03:58.033 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.__get_result() 2022-05-14T05:03:58.033 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T05:03:58.033 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise self._exception 2022-05-14T05:03:58.033 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T05:03:58.034 INFO:journalctl@ceph.mon.c.smithi115.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T05:03:58.034 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T05:03:58.034 INFO:journalctl@ceph.mon.c.smithi115.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T05:03:58.034 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T05:03:58.035 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise 2022-05-14T05:03:58.035 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T05:03:58.035 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.gen.throw(type, value, traceback) 2022-05-14T05:03:58.035 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T05:03:58.035 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OrchestratorError(msg) 2022-05-14T05:03:58.036 INFO:journalctl@ceph.mon.c.smithi115.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T05:03:58.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:03:57 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:03:58.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:03:57 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T05:03:58.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T05:03:58.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T05:03:58.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T05:03:58.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T05:03:58.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T05:03:58.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T05:03:58.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T05:03:58.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T05:03:58.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T05:03:58.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T05:03:58.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T05:03:58.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T05:03:58.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T05:03:58.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T05:03:58.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T05:03:58.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T05:03:58.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T05:03:58.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T05:03:58.048 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:03:57 smithi047 ceph-mon[77147]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T05:03:58.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T05:03:58.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T05:03:58.049 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T05:03:58.049 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T05:03:58.049 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T05:03:58.049 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T05:03:58.050 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T05:03:58.050 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T05:03:58.050 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T05:03:58.050 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T05:03:58.050 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T05:03:58.051 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T05:03:58.051 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T05:03:58.051 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T05:03:58.051 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T05:03:58.051 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T05:03:58.052 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T05:03:58.052 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T05:03:58.052 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T05:03:58.052 INFO:journalctl@ceph.mon.b.smithi047.stdout: During handling of the above exception, another exception occurred: 2022-05-14T05:03:58.053 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T05:03:58.053 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T05:03:58.053 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T05:03:58.053 INFO:journalctl@ceph.mon.b.smithi047.stdout: return f(*arg) 2022-05-14T05:03:58.054 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T05:03:58.054 INFO:journalctl@ceph.mon.b.smithi047.stdout: self._write_client_files(client_files, host) 2022-05-14T05:03:58.055 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T05:03:58.055 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T05:03:58.055 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T05:03:58.055 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T05:03:58.056 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T05:03:58.056 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.event_loop.get_result(coro) 2022-05-14T05:03:58.056 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T05:03:58.056 INFO:journalctl@ceph.mon.b.smithi047.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T05:03:58.056 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T05:03:58.057 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.__get_result() 2022-05-14T05:03:58.057 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T05:03:58.057 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise self._exception 2022-05-14T05:03:58.057 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T05:03:58.058 INFO:journalctl@ceph.mon.b.smithi047.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T05:03:58.058 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T05:03:58.058 INFO:journalctl@ceph.mon.b.smithi047.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T05:03:58.058 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T05:03:58.058 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise 2022-05-14T05:03:58.059 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T05:03:58.059 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.gen.throw(type, value, traceback) 2022-05-14T05:03:58.059 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T05:03:58.059 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OrchestratorError(msg) 2022-05-14T05:03:58.060 INFO:journalctl@ceph.mon.b.smithi047.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T05:03:58.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:03:58 smithi115 conmon[90393]: 2022-05-14T05:03:58.016+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:03:59.006 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:03:58 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:03:59.006 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:03:58 smithi115 ceph-mon[78593]: pgmap v10436: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:03:59.006 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:03:58 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T05:03:59.006 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:03:58 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T05:03:59.007 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:03:58 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 20170 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T05:03:59.050 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:03:58 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:03:59.050 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:03:58 smithi047 ceph-mon[77147]: pgmap v10436: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:03:59.050 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:03:58 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T05:03:59.050 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:03:58 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T05:03:59.051 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:03:58 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 20170 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T05:03:59.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:03:59 smithi115 conmon[90393]: 2022-05-14T05:03:59.002+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:03:59.972 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:03:59 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:04:00.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:03:59 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:04:00.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:03:59 smithi115 conmon[90393]: 2022-05-14T05:03:59.969+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:04:00.978 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:04:00 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:04:00.978 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:04:00 smithi115 ceph-mon[78593]: pgmap v10437: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T05:04:01.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:04:00 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:04:01.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:04:00 smithi047 ceph-mon[77147]: pgmap v10437: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T05:04:01.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:04:00 smithi115 conmon[90393]: 2022-05-14T05:04:00.975+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:04:02.018 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:04:01 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:04:02.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:04:01 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:04:02.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:04:02 smithi115 conmon[90393]: 2022-05-14T05:04:02.014+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:04:03.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:04:02 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:04:03.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:04:02 smithi047 ceph-mon[77147]: pgmap v10438: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:04:03.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:04:02 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T05:04:03.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:04:02 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T05:04:03.059 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:04:02 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:04:03.059 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:04:02 smithi115 ceph-mon[78593]: pgmap v10438: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:04:03.059 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:04:02 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T05:04:03.059 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:04:02 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T05:04:03.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:04:03 smithi115 conmon[90393]: 2022-05-14T05:04:03.055+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:04:04.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:04:03 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:04:04.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:04:03 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 20175 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T05:04:04.092 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:04:03 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:04:04.092 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:04:03 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 20175 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T05:04:04.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:04:04 smithi115 conmon[90393]: 2022-05-14T05:04:04.089+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:04:05.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:04:04 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:04:05.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:04:04 smithi047 ceph-mon[77147]: pgmap v10439: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:04:05.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:04:04 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:04:05.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:04:04 smithi115 ceph-mon[78593]: pgmap v10439: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:04:05.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:04:05 smithi115 conmon[90393]: 2022-05-14T05:04:05.130+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:04:06.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:04:05 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:04:06.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:04:05 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:04:06.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:04:06 smithi115 conmon[90393]: 2022-05-14T05:04:06.104+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:04:07.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:04:06 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:04:07.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:04:06 smithi047 ceph-mon[77147]: pgmap v10440: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T05:04:07.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:04:06 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:04:07.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:04:06 smithi115 ceph-mon[78593]: pgmap v10440: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T05:04:07.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:04:07 smithi115 conmon[90393]: 2022-05-14T05:04:07.132+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:04:08.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:04:07 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:04:08.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:04:07 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:04:08.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:04:08 smithi115 conmon[90393]: 2022-05-14T05:04:08.123+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:04:09.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:04:08 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:04:09.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:04:08 smithi047 ceph-mon[77147]: pgmap v10441: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:04:09.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:04:08 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 20181 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T05:04:09.104 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:04:08 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:04:09.104 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:04:08 smithi115 ceph-mon[78593]: pgmap v10441: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:04:09.104 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:04:08 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 20181 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T05:04:09.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:04:09 smithi115 conmon[90393]: 2022-05-14T05:04:09.100+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:04:10.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:04:09 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:04:10.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:04:09 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:04:10.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:04:10 smithi115 conmon[90393]: 2022-05-14T05:04:10.135+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:04:11.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:04:10 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:04:11.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:04:10 smithi047 ceph-mon[77147]: pgmap v10442: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T05:04:11.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:04:10 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T05:04:11.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:04:10 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T05:04:11.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:04:10 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:04:11.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:04:10 smithi115 ceph-mon[78593]: pgmap v10442: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T05:04:11.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:04:10 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T05:04:11.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:04:10 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T05:04:11.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:04:11 smithi115 conmon[90393]: 2022-05-14T05:04:11.152+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:04:12.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:04:11 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:04:12.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:04:11 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:04:12.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:04:12 smithi115 conmon[90393]: 2022-05-14T05:04:12.159+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:04:13.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:04:12 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:04:13.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:04:12 smithi047 ceph-mon[77147]: pgmap v10443: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:04:13.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:04:12 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:04:13.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:04:12 smithi115 ceph-mon[78593]: pgmap v10443: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:04:13.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:04:13 smithi115 conmon[90393]: 2022-05-14T05:04:13.148+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:04:14.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:04:13 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:04:14.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:04:13 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 20186 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T05:04:14.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:04:13 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:04:14.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:04:13 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 20186 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T05:04:14.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:04:14 smithi115 conmon[90393]: 2022-05-14T05:04:14.105+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:04:15.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:04:14 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:04:15.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:04:14 smithi047 ceph-mon[77147]: pgmap v10444: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:04:15.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:04:14 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T05:04:15.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:04:14 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T05:04:15.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:04:14 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:04:15.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:04:14 smithi115 ceph-mon[78593]: pgmap v10444: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:04:15.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:04:14 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T05:04:15.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:04:14 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T05:04:15.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:04:15 smithi115 conmon[90393]: 2022-05-14T05:04:15.151+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:04:16.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:04:15 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:04:16.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:04:15 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T05:04:16.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:04:15 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T05:04:16.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:04:15 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:04:16.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:04:15 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T05:04:16.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:04:15 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T05:04:16.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:04:16 smithi115 conmon[90393]: 2022-05-14T05:04:16.199+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:04:17.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:04:16 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:04:17.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:04:16 smithi047 ceph-mon[77147]: pgmap v10445: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T05:04:17.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:04:16 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:04:17.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:04:16 smithi115 ceph-mon[78593]: pgmap v10445: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T05:04:17.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:04:17 smithi115 conmon[90393]: 2022-05-14T05:04:17.212+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:04:18.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:04:17 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:04:18.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:04:17 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:04:18.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:04:18 smithi115 conmon[90393]: 2022-05-14T05:04:18.255+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:04:19.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:04:18 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:04:19.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:04:18 smithi047 ceph-mon[77147]: pgmap v10446: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:04:19.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:04:18 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T05:04:19.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:04:18 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T05:04:19.045 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:04:18 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 20191 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T05:04:19.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:04:18 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:04:19.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:04:18 smithi115 ceph-mon[78593]: pgmap v10446: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:04:19.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:04:18 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T05:04:19.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:04:18 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T05:04:19.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:04:18 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 20191 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T05:04:19.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:04:19 smithi115 conmon[90393]: 2022-05-14T05:04:19.238+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:04:20.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:04:19 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:04:20.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:04:19 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:04:20.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:04:20 smithi115 conmon[90393]: 2022-05-14T05:04:20.258+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:04:21.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:04:20 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:04:21.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:04:20 smithi047 ceph-mon[77147]: pgmap v10447: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T05:04:21.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:04:20 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:04:21.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:04:20 smithi115 ceph-mon[78593]: pgmap v10447: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T05:04:21.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:04:21 smithi115 conmon[90393]: 2022-05-14T05:04:21.304+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:04:22.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:04:21 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:04:22.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:04:21 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:04:22.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:04:22 smithi115 conmon[90393]: 2022-05-14T05:04:22.292+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:04:23.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:04:22 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:04:23.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:04:22 smithi047 ceph-mon[77147]: pgmap v10448: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:04:23.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:04:22 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T05:04:23.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:04:22 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T05:04:23.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:04:22 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:04:23.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:04:22 smithi115 ceph-mon[78593]: pgmap v10448: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:04:23.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:04:22 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T05:04:23.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:04:22 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T05:04:23.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:04:23 smithi115 conmon[90393]: 2022-05-14T05:04:23.252+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:04:24.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:04:23 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:04:24.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:04:23 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 20195 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T05:04:24.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:04:23 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:04:24.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:04:23 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 20195 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T05:04:24.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:04:24 smithi115 conmon[90393]: 2022-05-14T05:04:24.290+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:04:25.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:04:24 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:04:25.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:04:24 smithi047 ceph-mon[77147]: pgmap v10449: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:04:25.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:04:24 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:04:25.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:04:24 smithi115 ceph-mon[78593]: pgmap v10449: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:04:25.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:04:25 smithi115 conmon[90393]: 2022-05-14T05:04:25.298+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:04:26.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:04:25 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:04:26.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:04:25 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:04:26.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:04:26 smithi115 conmon[90393]: 2022-05-14T05:04:26.346+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:04:27.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:04:26 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:04:27.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:04:26 smithi047 ceph-mon[77147]: pgmap v10450: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T05:04:27.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:04:26 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:04:27.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:04:26 smithi115 ceph-mon[78593]: pgmap v10450: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T05:04:27.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:04:27 smithi115 conmon[90393]: 2022-05-14T05:04:27.353+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:04:28.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:04:27 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:04:28.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:04:27 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:04:28.700 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:04:28 smithi115 conmon[90393]: 2022-05-14T05:04:28.403+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:04:29.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:04:28 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:04:29.048 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:04:28 smithi047 ceph-mon[77147]: pgmap v10451: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:04:29.048 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:04:28 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 20200 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T05:04:29.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:04:28 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:04:29.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:04:28 smithi115 ceph-mon[78593]: pgmap v10451: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:04:29.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:04:28 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 20200 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T05:04:29.705 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:04:29 smithi115 conmon[90393]: 2022-05-14T05:04:29.448+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:04:30.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:04:29 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:04:30.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:04:29 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:04:30.708 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:04:30 smithi115 conmon[90393]: 2022-05-14T05:04:30.435+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:04:31.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:04:30 smithi047 ceph-mon[77147]: pgmap v10452: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T05:04:31.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:04:30 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:04:31.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:04:30 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T05:04:31.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:04:30 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T05:04:31.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:04:30 smithi115 ceph-mon[78593]: pgmap v10452: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T05:04:31.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:04:30 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:04:31.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:04:30 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T05:04:31.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:04:30 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T05:04:31.721 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:04:31 smithi115 conmon[90393]: 2022-05-14T05:04:31.437+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:04:32.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:04:31 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:04:32.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:04:31 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:04:32.713 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:04:32 smithi115 conmon[90393]: 2022-05-14T05:04:32.422+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:04:33.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:04:32 smithi047 ceph-mon[77147]: pgmap v10453: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:04:33.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:04:32 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:04:33.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:04:32 smithi115 ceph-mon[78593]: pgmap v10453: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:04:33.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:04:32 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:04:33.716 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:04:33 smithi115 conmon[90393]: 2022-05-14T05:04:33.424+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:04:34.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:04:33 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:04:34.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:04:33 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 20205 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T05:04:34.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:04:33 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:04:34.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:04:33 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 20205 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T05:04:34.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:04:34 smithi115 ceph-mon[78593]: pgmap v10454: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:04:34.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:04:34 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:04:34.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:04:34 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T05:04:34.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:04:34 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T05:04:34.861 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:04:34 smithi115 conmon[90393]: 2022-05-14T05:04:34.470+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:04:35.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:04:34 smithi047 ceph-mon[77147]: pgmap v10454: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:04:35.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:04:34 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:04:35.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:04:34 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T05:04:35.045 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:04:34 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T05:04:35.721 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:04:35 smithi115 conmon[90393]: 2022-05-14T05:04:35.432+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:04:36.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:04:35 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:04:36.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:04:35 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:04:36.723 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:04:36 smithi115 conmon[90393]: 2022-05-14T05:04:36.450+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:04:37.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:04:36 smithi047 ceph-mon[77147]: pgmap v10455: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T05:04:37.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:04:36 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:04:37.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:04:36 smithi115 ceph-mon[78593]: pgmap v10455: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T05:04:37.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:04:36 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:04:37.728 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:04:37 smithi115 conmon[90393]: 2022-05-14T05:04:37.405+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:04:38.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:04:37 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:04:38.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:04:37 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:04:38.729 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:04:38 smithi115 conmon[90393]: 2022-05-14T05:04:38.360+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:04:39.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:04:38 smithi047 ceph-mon[77147]: pgmap v10456: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:04:39.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:04:38 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:04:39.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:04:38 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T05:04:39.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:04:38 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T05:04:39.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:04:38 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 20210 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T05:04:39.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:04:38 smithi115 ceph-mon[78593]: pgmap v10456: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:04:39.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:04:38 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:04:39.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:04:38 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T05:04:39.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:04:38 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T05:04:39.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:04:38 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 20210 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T05:04:39.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:04:39 smithi115 conmon[90393]: 2022-05-14T05:04:39.328+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:04:40.045 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:04:39 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:04:40.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:04:39 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:04:40.736 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:04:40 smithi115 conmon[90393]: 2022-05-14T05:04:40.365+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:04:41.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:04:40 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:04:41.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:04:40 smithi047 ceph-mon[77147]: pgmap v10457: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T05:04:41.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:04:40 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:04:41.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:04:40 smithi115 ceph-mon[78593]: pgmap v10457: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T05:04:41.745 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:04:41 smithi115 conmon[90393]: 2022-05-14T05:04:41.361+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:04:42.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:04:41 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:04:42.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:04:41 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:04:42.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:04:42 smithi115 conmon[90393]: 2022-05-14T05:04:42.346+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:04:43.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:04:42 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:04:43.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:04:42 smithi047 ceph-mon[77147]: pgmap v10458: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:04:43.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:04:42 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T05:04:43.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:04:42 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T05:04:43.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:04:42 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:04:43.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:04:42 smithi115 ceph-mon[78593]: pgmap v10458: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:04:43.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:04:42 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T05:04:43.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:04:42 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T05:04:43.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:04:43 smithi115 conmon[90393]: 2022-05-14T05:04:43.312+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:04:44.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:04:43 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:04:44.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:04:43 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 20215 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T05:04:44.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:04:43 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:04:44.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:04:43 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 20215 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T05:04:44.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:04:44 smithi115 conmon[90393]: 2022-05-14T05:04:44.302+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:04:45.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:04:44 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:04:45.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:04:44 smithi047 ceph-mon[77147]: pgmap v10459: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:04:45.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:04:44 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:04:45.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:04:44 smithi115 ceph-mon[78593]: pgmap v10459: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:04:45.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:04:45 smithi115 conmon[90393]: 2022-05-14T05:04:45.258+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:04:46.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:04:45 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:04:46.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:04:45 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:04:46.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:04:46 smithi115 conmon[90393]: 2022-05-14T05:04:46.213+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:04:47.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:04:46 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:04:47.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:04:46 smithi047 ceph-mon[77147]: pgmap v10460: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T05:04:47.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:04:46 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:04:47.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:04:46 smithi115 ceph-mon[78593]: pgmap v10460: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T05:04:47.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:04:47 smithi115 conmon[90393]: 2022-05-14T05:04:47.195+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:04:48.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:04:47 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:04:48.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:04:47 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:04:48.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:04:48 smithi115 conmon[90393]: 2022-05-14T05:04:48.173+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:04:49.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:04:48 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:04:49.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:04:48 smithi047 ceph-mon[77147]: pgmap v10461: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:04:49.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:04:48 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 20220 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T05:04:49.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:04:48 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:04:49.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:04:48 smithi115 ceph-mon[78593]: pgmap v10461: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:04:49.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:04:48 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 20220 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T05:04:49.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:04:49 smithi115 conmon[90393]: 2022-05-14T05:04:49.195+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:04:50.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:04:49 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:04:50.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:04:49 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:04:50.617 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:04:50 smithi115 conmon[90393]: 2022-05-14T05:04:50.240+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:04:51.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:04:50 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:04:51.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:04:50 smithi047 ceph-mon[77147]: pgmap v10462: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T05:04:51.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:04:50 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T05:04:51.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:04:50 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T05:04:51.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:04:50 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:04:51.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:04:50 smithi115 ceph-mon[78593]: pgmap v10462: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T05:04:51.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:04:50 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T05:04:51.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:04:50 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T05:04:51.610 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:04:51 smithi115 conmon[90393]: 2022-05-14T05:04:51.247+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:04:52.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:04:51 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:04:52.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:04:51 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:04:52.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:04:52 smithi115 conmon[90393]: 2022-05-14T05:04:52.216+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:04:53.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:04:52 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:04:53.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:04:52 smithi047 ceph-mon[77147]: pgmap v10463: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:04:53.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:04:52 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:04:53.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:04:52 smithi115 ceph-mon[78593]: pgmap v10463: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:04:53.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:04:53 smithi115 conmon[90393]: 2022-05-14T05:04:53.187+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:04:54.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:04:53 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:04:54.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:04:53 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 20225 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T05:04:54.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:04:53 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:04:54.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:04:53 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 20225 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T05:04:54.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:04:54 smithi115 conmon[90393]: 2022-05-14T05:04:54.183+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:04:55.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:04:54 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:04:55.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:04:54 smithi047 ceph-mon[77147]: pgmap v10464: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:04:55.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:04:54 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T05:04:55.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:04:54 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T05:04:55.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:04:54 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:04:55.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:04:54 smithi115 ceph-mon[78593]: pgmap v10464: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:04:55.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:04:54 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T05:04:55.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:04:54 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T05:04:55.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:04:55 smithi115 conmon[90393]: 2022-05-14T05:04:55.143+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:04:56.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:04:55 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:04:56.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:04:55 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:04:56.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:04:56 smithi115 conmon[90393]: 2022-05-14T05:04:56.151+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:04:57.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:04:56 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:04:57.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:04:56 smithi047 ceph-mon[77147]: pgmap v10465: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T05:04:57.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:04:56 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:04:57.046 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:04:56 smithi115 ceph-mon[78593]: pgmap v10465: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T05:04:57.360 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:04:57 smithi115 conmon[90393]: 2022-05-14T05:04:57.193+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:04:57.788 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:04:57 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:04:57.788 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:04:57 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T05:04:57.789 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:04:57 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T05:04:57.789 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:04:57 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T05:04:57.789 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:04:57 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' 2022-05-14T05:04:57.789 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:04:57 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' 2022-05-14T05:04:58.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:04:57 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:04:58.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:04:57 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T05:04:58.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:04:57 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T05:04:58.045 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:04:57 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T05:04:58.045 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:04:57 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' 2022-05-14T05:04:58.045 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:04:57 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' 2022-05-14T05:04:58.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:04:58 smithi115 conmon[90393]: 2022-05-14T05:04:58.176+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:04:59.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:04:58 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:04:59.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:04:58 smithi047 ceph-mon[77147]: pgmap v10466: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:04:59.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:04:58 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T05:04:59.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:04:58 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T05:04:59.045 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:04:58 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 20230 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T05:04:59.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:04:58 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:04:59.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:04:58 smithi115 ceph-mon[78593]: pgmap v10466: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:04:59.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:04:58 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T05:04:59.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:04:58 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T05:04:59.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:04:58 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 20230 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T05:04:59.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:04:59 smithi115 conmon[90393]: 2022-05-14T05:04:59.178+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:05:00.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:04:59 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:05:00.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:05:00 smithi047 conmon[78521]: 2022-05-14T05:05:00.015+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T05:05:00.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:05:00 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T05:05:00.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:05:00 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T05:05:00.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:05:00 smithi047 conmon[78521]: yield 2022-05-14T05:05:00.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:05:00 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T05:05:00.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:05:00 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T05:05:00.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:05:00 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T05:05:00.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:05:00 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T05:05:00.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:05:00 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T05:05:00.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:05:00 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T05:05:00.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:05:00 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T05:05:00.048 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:05:00 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T05:05:00.048 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:05:00 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T05:05:00.048 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:05:00 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T05:05:00.048 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:05:00 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T05:05:00.048 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:05:00 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T05:05:00.049 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:05:00 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T05:05:00.049 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:05:00 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T05:05:00.050 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:05:00 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T05:05:00.056 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:04:59 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:05:00.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:05:00 smithi115 conmon[90393]: 2022-05-14T05:05:00.204+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:05:01.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:05:00 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:05:01.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:05:00 smithi115 ceph-mon[78593]: pgmap v10467: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T05:05:01.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:05:00 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' 2022-05-14T05:05:01.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:05:00 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' 2022-05-14T05:05:01.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:05:00 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-05-14T05:05:01.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:05:00 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-05-14T05:05:01.112 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:05:00 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-05-14T05:05:01.112 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:05:00 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-05-14T05:05:01.112 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:05:00 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-05-14T05:05:01.112 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:05:00 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-05-14T05:05:01.112 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:05:00 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-05-14T05:05:01.113 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:05:00 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-05-14T05:05:01.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:05:00 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:05:01.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:05:00 smithi047 ceph-mon[77147]: pgmap v10467: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T05:05:01.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:05:00 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' 2022-05-14T05:05:01.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:05:00 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' 2022-05-14T05:05:01.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:05:00 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-05-14T05:05:01.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:05:00 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-05-14T05:05:01.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:05:00 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-05-14T05:05:01.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:05:00 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-05-14T05:05:01.295 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:05:00 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-05-14T05:05:01.295 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:05:00 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-05-14T05:05:01.295 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:05:00 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-05-14T05:05:01.295 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:05:00 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-05-14T05:05:01.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:05:01 smithi115 conmon[90393]: 2022-05-14T05:05:01.212+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:05:02.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:05:01 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T05:05:02.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T05:05:02.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T05:05:02.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T05:05:02.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T05:05:02.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T05:05:02.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T05:05:02.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T05:05:02.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T05:05:02.113 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T05:05:02.113 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T05:05:02.113 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T05:05:02.113 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T05:05:02.113 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T05:05:02.114 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T05:05:02.114 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T05:05:02.115 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T05:05:02.115 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T05:05:02.116 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T05:05:02.116 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:05:01 smithi115 ceph-mon[78593]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T05:05:02.116 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:05:01 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:05:02.116 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:05:01 smithi115 ceph-mon[78593]: Adjusting osd_memory_target on smithi115 to 197.0M 2022-05-14T05:05:02.116 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:05:01 smithi115 ceph-mon[78593]: Unable to set osd_memory_target on smithi115 to 206629785: error parsing value: Value '206629785' is below minimum 939524096 2022-05-14T05:05:02.117 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:05:01 smithi115 ceph-mon[78593]: Adjusting osd_memory_target on smithi047 to 198.1M 2022-05-14T05:05:02.117 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:05:01 smithi115 ceph-mon[78593]: Unable to set osd_memory_target on smithi047 to 207755878: error parsing value: Value '207755878' is below minimum 939524096 2022-05-14T05:05:02.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:05:01 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T05:05:02.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T05:05:02.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T05:05:02.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T05:05:02.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T05:05:02.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T05:05:02.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T05:05:02.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T05:05:02.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T05:05:02.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T05:05:02.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T05:05:02.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T05:05:02.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T05:05:02.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T05:05:02.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T05:05:02.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T05:05:02.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T05:05:02.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T05:05:02.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T05:05:02.297 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:05:01 smithi047 ceph-mon[77147]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T05:05:02.297 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:05:01 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:05:02.298 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:05:01 smithi047 ceph-mon[77147]: Adjusting osd_memory_target on smithi115 to 197.0M 2022-05-14T05:05:02.298 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:05:01 smithi047 ceph-mon[77147]: Unable to set osd_memory_target on smithi115 to 206629785: error parsing value: Value '206629785' is below minimum 939524096 2022-05-14T05:05:02.298 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:05:01 smithi047 ceph-mon[77147]: Adjusting osd_memory_target on smithi047 to 198.1M 2022-05-14T05:05:02.298 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:05:01 smithi047 ceph-mon[77147]: Unable to set osd_memory_target on smithi047 to 207755878: error parsing value: Value '207755878' is below minimum 939524096 2022-05-14T05:05:02.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:05:02 smithi115 conmon[90393]: 2022-05-14T05:05:02.187+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:05:03.091 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:05:02 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:05:03.091 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:05:02 smithi047 ceph-mon[77147]: pgmap v10468: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:05:03.091 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:05:02 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T05:05:03.092 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:05:02 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T05:05:03.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:05:02 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:05:03.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:05:02 smithi115 ceph-mon[78593]: pgmap v10468: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:05:03.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:05:02 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T05:05:03.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:05:02 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T05:05:03.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:05:03 smithi047 conmon[78521]: 2022-05-14T05:05:03.087+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T05:05:03.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:05:03 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T05:05:03.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:05:03 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T05:05:03.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:05:03 smithi047 conmon[78521]: yield 2022-05-14T05:05:03.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:05:03 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T05:05:03.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:05:03 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T05:05:03.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:05:03 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T05:05:03.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:05:03 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T05:05:03.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:05:03 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T05:05:03.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:05:03 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T05:05:03.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:05:03 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T05:05:03.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:05:03 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T05:05:03.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:05:03 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T05:05:03.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:05:03 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T05:05:03.547 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:05:03 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T05:05:03.547 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:05:03 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T05:05:03.548 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:05:03 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T05:05:03.548 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:05:03 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T05:05:03.548 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:05:03 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T05:05:03.549 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:05:03 smithi047 conmon[78521]: 2022-05-14T05:05:03.088+0000 7f533ff78700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T05:05:03.549 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:05:03 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T05:05:03.549 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:05:03 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T05:05:03.550 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:05:03 smithi047 conmon[78521]: yield 2022-05-14T05:05:03.550 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:05:03 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T05:05:03.550 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:05:03 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T05:05:03.550 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:05:03 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T05:05:03.551 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:05:03 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T05:05:03.551 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:05:03 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T05:05:03.551 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:05:03 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T05:05:03.551 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:05:03 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T05:05:03.551 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:05:03 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T05:05:03.552 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:05:03 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T05:05:03.552 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:05:03 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T05:05:03.552 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:05:03 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T05:05:03.552 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:05:03 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T05:05:03.553 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:05:03 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T05:05:03.553 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:05:03 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T05:05:03.553 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:05:03 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T05:05:03.554 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:05:03 smithi047 conmon[78521]: 2022-05-14T05:05:03.554 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:05:03 smithi047 conmon[78521]: During handling of the above exception, another exception occurred: 2022-05-14T05:05:03.554 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:05:03 smithi047 conmon[78521]: 2022-05-14T05:05:03.554 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:05:03 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T05:05:03.554 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:05:03 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T05:05:03.555 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:05:03 smithi047 conmon[78521]: return f(*arg) 2022-05-14T05:05:03.555 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:05:03 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T05:05:03.556 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:05:03 smithi047 conmon[78521]: self._write_client_files(client_files, host) 2022-05-14T05:05:03.556 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:05:03 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T05:05:03.556 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:05:03 smithi047 conmon[78521]: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T05:05:03.556 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:05:03 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T05:05:03.557 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:05:03 smithi047 conmon[78521]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T05:05:03.557 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:05:03 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T05:05:03.557 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:05:03 smithi047 conmon[78521]: return self.event_loop.get_result(coro) 2022-05-14T05:05:03.557 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:05:03 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T05:05:03.557 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:05:03 smithi047 conmon[78521]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T05:05:03.558 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:05:03 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T05:05:03.558 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:05:03 smithi047 conmon[78521]: return self.__get_result() 2022-05-14T05:05:03.558 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:05:03 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T05:05:03.558 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:05:03 smithi047 conmon[78521]: raise self._exception 2022-05-14T05:05:03.559 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:05:03 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T05:05:03.559 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:05:03 smithi047 conmon[78521]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T05:05:03.559 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:05:03 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T05:05:03.560 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:05:03 smithi047 conmon[78521]: conn = await self._remote_connection(host, addr) 2022-05-14T05:05:03.560 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:05:03 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T05:05:03.560 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:05:03 smithi047 conmon[78521]: raise 2022-05-14T05:05:03.560 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:05:03 smithi047 conmon[78521]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T05:05:03.561 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:05:03 smithi047 conmon[78521]: self.gen.throw(type, value, traceback) 2022-05-14T05:05:03.561 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:05:03 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T05:05:03.561 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:05:03 smithi047 conmon[78521]: raise OrchestratorError(msg) 2022-05-14T05:05:03.561 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:05:03 smithi047 conmon[78521]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T05:05:03.610 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:05:03 smithi115 conmon[90393]: 2022-05-14T05:05:03.147+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:05:04.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:05:03 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:05:04.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:05:03 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 20235 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T05:05:04.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:05:03 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:05:04.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:05:03 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 20235 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T05:05:04.610 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:05:04 smithi115 conmon[90393]: 2022-05-14T05:05:04.195+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:05:05.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:05:04 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T05:05:05.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T05:05:05.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T05:05:05.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T05:05:05.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T05:05:05.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T05:05:05.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T05:05:05.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T05:05:05.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T05:05:05.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T05:05:05.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T05:05:05.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T05:05:05.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T05:05:05.113 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T05:05:05.113 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T05:05:05.113 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T05:05:05.113 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T05:05:05.114 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T05:05:05.114 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T05:05:05.114 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:05:04 smithi115 ceph-mon[78593]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T05:05:05.115 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T05:05:05.115 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T05:05:05.115 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T05:05:05.116 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T05:05:05.116 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T05:05:05.116 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T05:05:05.116 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T05:05:05.117 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T05:05:05.117 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T05:05:05.117 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T05:05:05.117 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T05:05:05.117 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T05:05:05.118 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T05:05:05.118 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T05:05:05.118 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T05:05:05.118 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T05:05:05.119 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T05:05:05.119 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T05:05:05.119 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T05:05:05.119 INFO:journalctl@ceph.mon.c.smithi115.stdout: During handling of the above exception, another exception occurred: 2022-05-14T05:05:05.120 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T05:05:05.120 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T05:05:05.120 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T05:05:05.120 INFO:journalctl@ceph.mon.c.smithi115.stdout: return f(*arg) 2022-05-14T05:05:05.121 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T05:05:05.121 INFO:journalctl@ceph.mon.c.smithi115.stdout: self._write_client_files(client_files, host) 2022-05-14T05:05:05.121 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T05:05:05.121 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T05:05:05.122 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T05:05:05.122 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T05:05:05.122 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T05:05:05.122 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.event_loop.get_result(coro) 2022-05-14T05:05:05.122 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T05:05:05.123 INFO:journalctl@ceph.mon.c.smithi115.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T05:05:05.123 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T05:05:05.123 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.__get_result() 2022-05-14T05:05:05.123 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T05:05:05.124 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise self._exception 2022-05-14T05:05:05.124 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T05:05:05.124 INFO:journalctl@ceph.mon.c.smithi115.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T05:05:05.124 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T05:05:05.125 INFO:journalctl@ceph.mon.c.smithi115.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T05:05:05.125 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T05:05:05.125 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise 2022-05-14T05:05:05.125 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T05:05:05.126 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.gen.throw(type, value, traceback) 2022-05-14T05:05:05.126 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T05:05:05.126 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OrchestratorError(msg) 2022-05-14T05:05:05.126 INFO:journalctl@ceph.mon.c.smithi115.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T05:05:05.127 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:05:04 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:05:05.127 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:05:04 smithi115 ceph-mon[78593]: pgmap v10469: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:05:05.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:05:04 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T05:05:05.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T05:05:05.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T05:05:05.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T05:05:05.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T05:05:05.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T05:05:05.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T05:05:05.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T05:05:05.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T05:05:05.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T05:05:05.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T05:05:05.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T05:05:05.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T05:05:05.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T05:05:05.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T05:05:05.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T05:05:05.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T05:05:05.298 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T05:05:05.298 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T05:05:05.298 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:05:04 smithi047 ceph-mon[77147]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T05:05:05.299 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T05:05:05.301 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T05:05:05.302 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T05:05:05.302 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T05:05:05.302 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T05:05:05.303 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T05:05:05.303 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T05:05:05.303 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T05:05:05.304 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T05:05:05.304 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T05:05:05.304 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T05:05:05.305 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T05:05:05.305 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T05:05:05.306 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T05:05:05.306 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T05:05:05.307 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T05:05:05.307 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T05:05:05.309 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T05:05:05.310 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T05:05:05.310 INFO:journalctl@ceph.mon.b.smithi047.stdout: During handling of the above exception, another exception occurred: 2022-05-14T05:05:05.310 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T05:05:05.310 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T05:05:05.310 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T05:05:05.311 INFO:journalctl@ceph.mon.b.smithi047.stdout: return f(*arg) 2022-05-14T05:05:05.311 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T05:05:05.311 INFO:journalctl@ceph.mon.b.smithi047.stdout: self._write_client_files(client_files, host) 2022-05-14T05:05:05.311 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T05:05:05.312 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T05:05:05.312 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T05:05:05.312 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T05:05:05.312 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T05:05:05.313 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.event_loop.get_result(coro) 2022-05-14T05:05:05.313 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T05:05:05.313 INFO:journalctl@ceph.mon.b.smithi047.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T05:05:05.313 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T05:05:05.314 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.__get_result() 2022-05-14T05:05:05.314 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T05:05:05.314 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise self._exception 2022-05-14T05:05:05.314 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T05:05:05.315 INFO:journalctl@ceph.mon.b.smithi047.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T05:05:05.315 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T05:05:05.315 INFO:journalctl@ceph.mon.b.smithi047.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T05:05:05.315 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T05:05:05.316 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise 2022-05-14T05:05:05.316 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T05:05:05.316 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.gen.throw(type, value, traceback) 2022-05-14T05:05:05.316 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T05:05:05.317 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OrchestratorError(msg) 2022-05-14T05:05:05.317 INFO:journalctl@ceph.mon.b.smithi047.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T05:05:05.317 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:05:04 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:05:05.317 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:05:04 smithi047 ceph-mon[77147]: pgmap v10469: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:05:05.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:05:05 smithi115 conmon[90393]: 2022-05-14T05:05:05.235+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:05:06.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:05:05 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:05:06.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:05:05 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:05:06.610 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:05:06 smithi115 conmon[90393]: 2022-05-14T05:05:06.186+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:05:07.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:05:06 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:05:07.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:05:06 smithi115 ceph-mon[78593]: pgmap v10470: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T05:05:07.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:05:06 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:05:07.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:05:06 smithi047 ceph-mon[77147]: pgmap v10470: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T05:05:07.610 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:05:07 smithi115 conmon[90393]: 2022-05-14T05:05:07.223+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:05:08.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:05:07 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:05:08.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:05:07 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:05:08.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:05:08 smithi115 conmon[90393]: 2022-05-14T05:05:08.233+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:05:09.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:05:08 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:05:09.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:05:08 smithi115 ceph-mon[78593]: pgmap v10471: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:05:09.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:05:08 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 20240 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T05:05:09.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:05:08 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:05:09.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:05:08 smithi047 ceph-mon[77147]: pgmap v10471: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:05:09.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:05:08 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 20240 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T05:05:09.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:05:09 smithi115 conmon[90393]: 2022-05-14T05:05:09.259+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:05:10.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:05:09 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:05:10.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:05:09 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:05:10.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:05:10 smithi115 conmon[90393]: 2022-05-14T05:05:10.214+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:05:11.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:05:10 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:05:11.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:05:10 smithi115 ceph-mon[78593]: pgmap v10472: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T05:05:11.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:05:10 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T05:05:11.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:05:10 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T05:05:11.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:05:10 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:05:11.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:05:10 smithi047 ceph-mon[77147]: pgmap v10472: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T05:05:11.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:05:10 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T05:05:11.295 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:05:10 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T05:05:11.610 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:05:11 smithi115 conmon[90393]: 2022-05-14T05:05:11.236+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:05:12.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:05:11 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:05:12.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:05:11 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:05:12.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:05:12 smithi115 conmon[90393]: 2022-05-14T05:05:12.200+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:05:13.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:05:12 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:05:13.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:05:12 smithi115 ceph-mon[78593]: pgmap v10473: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:05:13.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:05:12 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:05:13.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:05:12 smithi047 ceph-mon[77147]: pgmap v10473: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:05:13.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:05:13 smithi115 conmon[90393]: 2022-05-14T05:05:13.198+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:05:14.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:05:13 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:05:14.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:05:13 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 20245 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T05:05:14.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:05:13 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:05:14.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:05:13 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 20245 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T05:05:14.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:05:14 smithi115 conmon[90393]: 2022-05-14T05:05:14.174+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:05:15.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:05:14 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:05:15.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:05:14 smithi115 ceph-mon[78593]: pgmap v10474: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:05:15.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:05:14 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T05:05:15.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:05:14 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T05:05:15.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:05:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T05:05:15.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:05:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T05:05:15.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:05:14 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:05:15.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:05:14 smithi047 ceph-mon[77147]: pgmap v10474: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:05:15.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:05:14 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T05:05:15.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:05:14 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T05:05:15.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:05:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T05:05:15.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:05:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T05:05:15.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:05:15 smithi115 conmon[90393]: 2022-05-14T05:05:15.147+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:05:16.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:05:15 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:05:16.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:05:15 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:05:16.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:05:16 smithi115 conmon[90393]: 2022-05-14T05:05:16.110+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:05:17.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:05:16 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:05:17.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:05:16 smithi115 ceph-mon[78593]: pgmap v10475: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T05:05:17.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:05:16 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:05:17.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:05:16 smithi047 ceph-mon[77147]: pgmap v10475: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T05:05:17.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:05:17 smithi115 conmon[90393]: 2022-05-14T05:05:17.107+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:05:18.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:05:17 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:05:18.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:05:17 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:05:18.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:05:18 smithi115 conmon[90393]: 2022-05-14T05:05:18.147+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:05:19.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:05:18 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:05:19.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:05:18 smithi115 ceph-mon[78593]: pgmap v10476: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:05:19.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:05:18 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T05:05:19.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:05:18 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T05:05:19.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:05:18 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 20250 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T05:05:19.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:05:18 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:05:19.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:05:18 smithi047 ceph-mon[77147]: pgmap v10476: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:05:19.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:05:18 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T05:05:19.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:05:18 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T05:05:19.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:05:18 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 20250 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T05:05:19.610 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:05:19 smithi115 conmon[90393]: 2022-05-14T05:05:19.177+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:05:20.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:05:19 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:05:20.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:05:19 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:05:20.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:05:20 smithi115 conmon[90393]: 2022-05-14T05:05:20.195+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:05:21.112 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:05:20 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:05:21.113 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:05:20 smithi115 ceph-mon[78593]: pgmap v10477: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T05:05:21.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:05:20 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:05:21.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:05:20 smithi047 ceph-mon[77147]: pgmap v10477: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T05:05:21.610 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:05:21 smithi115 conmon[90393]: 2022-05-14T05:05:21.146+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:05:22.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:05:21 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:05:22.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:05:21 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:05:22.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:05:22 smithi115 conmon[90393]: 2022-05-14T05:05:22.166+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:05:23.184 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:05:22 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:05:23.184 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:05:22 smithi115 ceph-mon[78593]: pgmap v10478: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:05:23.184 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:05:22 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T05:05:23.185 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:05:22 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T05:05:23.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:05:22 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:05:23.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:05:22 smithi047 ceph-mon[77147]: pgmap v10478: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:05:23.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:05:22 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T05:05:23.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:05:22 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T05:05:23.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:05:23 smithi115 conmon[90393]: 2022-05-14T05:05:23.181+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:05:24.216 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:05:23 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:05:24.217 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:05:23 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 20255 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T05:05:24.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:05:23 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:05:24.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:05:23 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 20255 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T05:05:24.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:05:24 smithi115 conmon[90393]: 2022-05-14T05:05:24.213+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:05:25.224 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:05:24 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:05:25.224 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:05:24 smithi115 ceph-mon[78593]: pgmap v10479: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:05:25.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:05:24 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:05:25.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:05:24 smithi047 ceph-mon[77147]: pgmap v10479: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:05:25.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:05:25 smithi115 conmon[90393]: 2022-05-14T05:05:25.220+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:05:26.192 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:05:25 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:05:26.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:05:25 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:05:26.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:05:26 smithi115 conmon[90393]: 2022-05-14T05:05:26.190+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:05:27.228 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:05:26 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:05:27.229 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:05:26 smithi115 ceph-mon[78593]: pgmap v10480: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T05:05:27.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:05:26 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:05:27.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:05:26 smithi047 ceph-mon[77147]: pgmap v10480: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T05:05:27.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:05:27 smithi115 conmon[90393]: 2022-05-14T05:05:27.225+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:05:28.185 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:05:27 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:05:28.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:05:27 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:05:28.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:05:28 smithi115 conmon[90393]: 2022-05-14T05:05:28.182+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:05:29.165 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:05:28 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:05:29.166 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:05:28 smithi115 ceph-mon[78593]: pgmap v10481: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:05:29.166 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:05:28 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 20260 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T05:05:29.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:05:28 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:05:29.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:05:28 smithi047 ceph-mon[77147]: pgmap v10481: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:05:29.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:05:28 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 20260 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T05:05:29.610 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:05:29 smithi115 conmon[90393]: 2022-05-14T05:05:29.161+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:05:30.159 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:05:29 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:05:30.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:05:29 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:05:30.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:05:30 smithi115 conmon[90393]: 2022-05-14T05:05:30.156+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:05:31.145 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:05:30 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:05:31.146 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:05:30 smithi115 ceph-mon[78593]: pgmap v10482: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T05:05:31.146 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:05:30 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T05:05:31.146 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:05:30 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T05:05:31.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:05:30 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:05:31.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:05:30 smithi047 ceph-mon[77147]: pgmap v10482: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T05:05:31.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:05:30 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T05:05:31.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:05:30 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T05:05:31.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:05:31 smithi115 conmon[90393]: 2022-05-14T05:05:31.142+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:05:32.153 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:05:31 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:05:32.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:05:31 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:05:32.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:05:32 smithi115 conmon[90393]: 2022-05-14T05:05:32.150+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:05:33.192 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:05:32 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:05:33.192 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:05:32 smithi115 ceph-mon[78593]: pgmap v10483: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:05:33.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:05:32 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:05:33.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:05:32 smithi047 ceph-mon[77147]: pgmap v10483: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:05:33.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:05:33 smithi115 conmon[90393]: 2022-05-14T05:05:33.189+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:05:34.210 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:05:33 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:05:34.211 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:05:33 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 20265 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T05:05:34.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:05:33 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:05:34.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:05:33 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 20265 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T05:05:34.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:05:34 smithi115 conmon[90393]: 2022-05-14T05:05:34.207+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:05:35.168 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:05:34 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:05:35.169 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:05:34 smithi115 ceph-mon[78593]: pgmap v10484: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:05:35.169 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:05:34 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T05:05:35.169 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:05:34 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T05:05:35.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:05:34 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:05:35.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:05:34 smithi047 ceph-mon[77147]: pgmap v10484: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:05:35.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:05:34 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T05:05:35.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:05:34 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T05:05:35.610 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:05:35 smithi115 conmon[90393]: 2022-05-14T05:05:35.166+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:05:36.198 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:05:35 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:05:36.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:05:35 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:05:36.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:05:36 smithi115 conmon[90393]: 2022-05-14T05:05:36.196+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:05:37.177 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:05:36 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:05:37.177 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:05:36 smithi115 ceph-mon[78593]: pgmap v10485: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T05:05:37.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:05:36 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:05:37.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:05:36 smithi047 ceph-mon[77147]: pgmap v10485: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T05:05:37.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:05:37 smithi115 conmon[90393]: 2022-05-14T05:05:37.174+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:05:38.216 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:05:37 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:05:38.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:05:37 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:05:38.610 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:05:38 smithi115 conmon[90393]: 2022-05-14T05:05:38.213+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:05:39.260 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:05:38 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:05:39.260 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:05:38 smithi115 ceph-mon[78593]: pgmap v10486: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:05:39.261 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:05:38 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T05:05:39.261 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:05:38 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T05:05:39.261 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:05:38 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 20270 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T05:05:39.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:05:38 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:05:39.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:05:38 smithi047 ceph-mon[77147]: pgmap v10486: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:05:39.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:05:38 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T05:05:39.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:05:38 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T05:05:39.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:05:38 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 20270 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T05:05:39.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:05:39 smithi115 conmon[90393]: 2022-05-14T05:05:39.258+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:05:40.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:05:39 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:05:40.308 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:05:39 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:05:40.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:05:40 smithi115 conmon[90393]: 2022-05-14T05:05:40.303+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:05:41.276 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:05:40 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:05:41.276 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:05:40 smithi115 ceph-mon[78593]: pgmap v10487: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T05:05:41.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:05:40 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:05:41.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:05:40 smithi047 ceph-mon[77147]: pgmap v10487: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T05:05:41.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:05:41 smithi115 conmon[90393]: 2022-05-14T05:05:41.273+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:05:42.256 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:05:41 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:05:42.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:05:41 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:05:42.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:05:42 smithi115 conmon[90393]: 2022-05-14T05:05:42.253+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:05:43.256 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:05:42 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:05:43.257 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:05:42 smithi115 ceph-mon[78593]: pgmap v10488: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:05:43.257 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:05:42 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T05:05:43.258 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:05:42 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T05:05:43.259 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:05:42 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:05:43.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:05:42 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:05:43.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:05:42 smithi047 ceph-mon[77147]: pgmap v10488: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:05:43.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:05:42 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T05:05:43.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:05:42 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T05:05:43.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:05:42 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:05:43.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:05:43 smithi115 conmon[90393]: 2022-05-14T05:05:43.254+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:05:44.258 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:05:43 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 20275 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T05:05:44.259 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:05:43 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:05:44.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:05:43 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 20275 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T05:05:44.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:05:43 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:05:44.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:05:44 smithi115 conmon[90393]: 2022-05-14T05:05:44.255+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:05:45.220 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:05:44 smithi115 ceph-mon[78593]: pgmap v10489: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:05:45.220 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:05:44 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:05:45.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:05:44 smithi047 ceph-mon[77147]: pgmap v10489: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:05:45.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:05:44 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:05:45.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:05:45 smithi115 conmon[90393]: 2022-05-14T05:05:45.216+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:05:46.252 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:05:45 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:05:46.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:05:45 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:05:46.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:05:46 smithi115 conmon[90393]: 2022-05-14T05:05:46.248+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:05:47.289 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:05:46 smithi115 ceph-mon[78593]: pgmap v10490: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T05:05:47.290 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:05:46 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:05:47.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:05:46 smithi047 ceph-mon[77147]: pgmap v10490: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T05:05:47.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:05:46 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:05:47.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:05:47 smithi115 conmon[90393]: 2022-05-14T05:05:47.286+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:05:48.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:05:47 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:05:48.324 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:05:47 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:05:48.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:05:48 smithi115 conmon[90393]: 2022-05-14T05:05:48.321+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:05:49.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:05:48 smithi047 ceph-mon[77147]: pgmap v10491: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:05:49.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:05:48 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 20280 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T05:05:49.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:05:48 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:05:49.334 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:05:48 smithi115 ceph-mon[78593]: pgmap v10491: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:05:49.336 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:05:48 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 20280 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T05:05:49.337 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:05:48 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:05:49.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:05:49 smithi115 conmon[90393]: 2022-05-14T05:05:49.331+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:05:50.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:05:49 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:05:50.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:05:49 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:05:50.862 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:05:50 smithi115 conmon[90393]: 2022-05-14T05:05:50.365+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:05:51.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:05:50 smithi047 ceph-mon[77147]: pgmap v10492: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T05:05:51.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:05:50 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T05:05:51.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:05:50 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T05:05:51.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:05:50 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:05:51.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:05:50 smithi115 ceph-mon[78593]: pgmap v10492: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T05:05:51.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:05:50 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T05:05:51.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:05:50 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T05:05:51.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:05:50 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:05:51.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:05:51 smithi115 conmon[90393]: 2022-05-14T05:05:51.404+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:05:52.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:05:51 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:05:52.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:05:51 smithi047 ceph-mon[77147]: pgmap v10493: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:05:52.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:05:51 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:05:52.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:05:51 smithi115 ceph-mon[78593]: pgmap v10493: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:05:52.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:05:52 smithi115 conmon[90393]: 2022-05-14T05:05:52.439+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:05:53.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:05:52 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:05:53.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:05:52 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:05:53.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:05:53 smithi115 conmon[90393]: 2022-05-14T05:05:53.397+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:05:54.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:05:53 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 20285 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T05:05:54.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:05:53 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:05:54.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:05:53 smithi047 ceph-mon[77147]: pgmap v10494: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:05:54.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:05:53 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 20285 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T05:05:54.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:05:53 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:05:54.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:05:53 smithi115 ceph-mon[78593]: pgmap v10494: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:05:54.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:05:54 smithi115 conmon[90393]: 2022-05-14T05:05:54.424+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:05:55.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:05:54 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T05:05:55.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:05:54 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T05:05:55.295 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:05:54 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:05:55.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:05:54 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T05:05:55.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:05:54 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T05:05:55.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:05:54 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:05:55.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:05:55 smithi115 conmon[90393]: 2022-05-14T05:05:55.390+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:05:56.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:05:55 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:05:56.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:05:55 smithi047 ceph-mon[77147]: pgmap v10495: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T05:05:56.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:05:55 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:05:56.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:05:55 smithi115 ceph-mon[78593]: pgmap v10495: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T05:05:56.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:05:56 smithi115 conmon[90393]: 2022-05-14T05:05:56.425+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:05:57.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:05:56 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:05:57.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:05:56 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:05:57.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:05:57 smithi115 conmon[90393]: 2022-05-14T05:05:57.377+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:05:58.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:05:57 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:05:58.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:05:57 smithi047 ceph-mon[77147]: pgmap v10496: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:05:58.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:05:57 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 20290 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T05:05:58.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:05:57 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:05:58.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:05:57 smithi115 ceph-mon[78593]: pgmap v10496: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:05:58.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:05:57 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 20290 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T05:05:58.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:05:58 smithi115 conmon[90393]: 2022-05-14T05:05:58.392+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:05:59.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:05:58 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T05:05:59.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:05:58 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T05:05:59.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:05:58 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:05:59.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:05:58 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T05:05:59.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:05:58 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T05:05:59.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:05:58 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:05:59.862 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:05:59 smithi115 conmon[90393]: 2022-05-14T05:05:59.425+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:06:00.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:05:59 smithi047 ceph-mon[77147]: pgmap v10497: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T05:06:00.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:05:59 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:06:00.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:05:59 smithi115 ceph-mon[78593]: pgmap v10497: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T05:06:00.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:05:59 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:06:00.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:06:00 smithi115 conmon[90393]: 2022-05-14T05:06:00.381+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:06:01.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:06:00 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:06:01.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:06:00 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:06:01.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:06:01 smithi115 conmon[90393]: 2022-05-14T05:06:01.363+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:06:02.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:06:02 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:06:02.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:06:02 smithi047 ceph-mon[77147]: pgmap v10498: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:06:02.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:06:02 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:06:02.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:06:02 smithi115 ceph-mon[78593]: pgmap v10498: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:06:02.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:06:02 smithi115 conmon[90393]: 2022-05-14T05:06:02.380+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:06:03.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:06:03 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T05:06:03.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:06:03 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T05:06:03.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:06:03 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:06:03.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:06:03 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 20295 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T05:06:03.350 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:06:03 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T05:06:03.350 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:06:03 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T05:06:03.350 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:06:03 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:06:03.351 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:06:03 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 20295 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T05:06:03.606 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:06:03 smithi115 conmon[90393]: 2022-05-14T05:06:03.345+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:06:04.009 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:06:03 smithi047 conmon[78521]: 2022-05-14T05:06:03.695+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T05:06:04.010 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:06:03 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T05:06:04.010 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:06:03 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T05:06:04.010 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:06:03 smithi047 conmon[78521]: yield 2022-05-14T05:06:04.010 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:06:03 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T05:06:04.011 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:06:03 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T05:06:04.011 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:06:03 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T05:06:04.011 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:06:03 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T05:06:04.011 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:06:03 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T05:06:04.012 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:06:03 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T05:06:04.012 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:06:03 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T05:06:04.012 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:06:03 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T05:06:04.012 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:06:03 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T05:06:04.012 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:06:03 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T05:06:04.013 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:06:03 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T05:06:04.013 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:06:03 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T05:06:04.013 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:06:03 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T05:06:04.013 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:06:03 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T05:06:04.014 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:06:03 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T05:06:04.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:06:04 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T05:06:04.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:06:04 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T05:06:04.295 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:06:04 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T05:06:04.295 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:06:04 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:06:04.296 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:06:04 smithi047 ceph-mon[77147]: pgmap v10499: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:06:04.296 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:06:04 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T05:06:04.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T05:06:04.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T05:06:04.298 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T05:06:04.299 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T05:06:04.299 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T05:06:04.300 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T05:06:04.300 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T05:06:04.301 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T05:06:04.302 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T05:06:04.302 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T05:06:04.302 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T05:06:04.303 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T05:06:04.303 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T05:06:04.304 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T05:06:04.304 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T05:06:04.304 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T05:06:04.305 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T05:06:04.305 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T05:06:04.305 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:06:04 smithi047 ceph-mon[77147]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T05:06:04.321 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:06:04 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T05:06:04.324 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:06:04 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T05:06:04.324 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:06:04 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T05:06:04.324 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:06:04 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:06:04.325 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:06:04 smithi115 ceph-mon[78593]: pgmap v10499: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:06:04.325 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:06:04 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T05:06:04.325 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T05:06:04.325 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T05:06:04.326 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T05:06:04.326 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T05:06:04.326 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T05:06:04.326 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T05:06:04.327 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T05:06:04.327 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T05:06:04.327 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T05:06:04.327 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T05:06:04.327 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T05:06:04.328 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T05:06:04.328 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T05:06:04.328 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T05:06:04.328 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T05:06:04.329 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T05:06:04.329 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T05:06:04.329 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T05:06:04.329 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:06:04 smithi115 ceph-mon[78593]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T05:06:04.610 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:06:04 smithi115 conmon[90393]: 2022-05-14T05:06:04.318+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:06:05.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:06:05 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:06:05.312 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:06:05 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:06:05.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:06:05 smithi115 conmon[90393]: 2022-05-14T05:06:05.309+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:06:06.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:06:06 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:06:06.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:06:06 smithi047 ceph-mon[77147]: pgmap v10500: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T05:06:06.334 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:06:06 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:06:06.334 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:06:06 smithi115 ceph-mon[78593]: pgmap v10500: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T05:06:06.611 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:06:06 smithi115 conmon[90393]: 2022-05-14T05:06:06.331+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:06:07.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:06:07 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:06:07.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:06:07 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T05:06:07.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T05:06:07.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T05:06:07.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T05:06:07.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T05:06:07.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T05:06:07.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T05:06:07.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T05:06:07.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T05:06:07.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T05:06:07.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T05:06:07.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T05:06:07.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T05:06:07.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T05:06:07.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T05:06:07.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T05:06:07.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T05:06:07.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T05:06:07.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T05:06:07.047 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:06:07 smithi047 ceph-mon[77147]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T05:06:07.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T05:06:07.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T05:06:07.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T05:06:07.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T05:06:07.049 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T05:06:07.049 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T05:06:07.049 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T05:06:07.050 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T05:06:07.050 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T05:06:07.050 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T05:06:07.050 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T05:06:07.050 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T05:06:07.051 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T05:06:07.051 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T05:06:07.051 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T05:06:07.051 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T05:06:07.052 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T05:06:07.052 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T05:06:07.052 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T05:06:07.052 INFO:journalctl@ceph.mon.b.smithi047.stdout: During handling of the above exception, another exception occurred: 2022-05-14T05:06:07.053 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T05:06:07.053 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T05:06:07.053 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T05:06:07.054 INFO:journalctl@ceph.mon.b.smithi047.stdout: return f(*arg) 2022-05-14T05:06:07.054 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T05:06:07.054 INFO:journalctl@ceph.mon.b.smithi047.stdout: self._write_client_files(client_files, host) 2022-05-14T05:06:07.054 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T05:06:07.054 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T05:06:07.055 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T05:06:07.055 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T05:06:07.055 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T05:06:07.055 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.event_loop.get_result(coro) 2022-05-14T05:06:07.055 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T05:06:07.056 INFO:journalctl@ceph.mon.b.smithi047.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T05:06:07.056 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T05:06:07.056 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.__get_result() 2022-05-14T05:06:07.056 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T05:06:07.057 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise self._exception 2022-05-14T05:06:07.057 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T05:06:07.057 INFO:journalctl@ceph.mon.b.smithi047.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T05:06:07.057 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T05:06:07.057 INFO:journalctl@ceph.mon.b.smithi047.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T05:06:07.058 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T05:06:07.058 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise 2022-05-14T05:06:07.058 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T05:06:07.058 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.gen.throw(type, value, traceback) 2022-05-14T05:06:07.058 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T05:06:07.059 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OrchestratorError(msg) 2022-05-14T05:06:07.059 INFO:journalctl@ceph.mon.b.smithi047.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T05:06:07.060 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:06:06 smithi047 conmon[78521]: 2022-05-14T05:06:06.767+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T05:06:07.060 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:06:06 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T05:06:07.060 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:06:06 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T05:06:07.060 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:06:06 smithi047 conmon[78521]: yield 2022-05-14T05:06:07.061 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:06:06 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T05:06:07.061 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:06:06 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T05:06:07.061 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:06:06 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T05:06:07.061 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:06:06 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T05:06:07.061 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:06:06 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T05:06:07.062 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:06:06 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T05:06:07.062 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:06:06 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T05:06:07.062 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:06:06 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T05:06:07.062 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:06:06 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T05:06:07.063 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:06:06 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T05:06:07.063 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:06:06 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T05:06:07.063 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:06:06 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T05:06:07.063 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:06:06 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T05:06:07.063 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:06:06 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T05:06:07.064 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:06:06 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T05:06:07.064 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:06:06 smithi047 conmon[78521]: 2022-05-14T05:06:06.768+0000 7f533ef76700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T05:06:07.064 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:06:06 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T05:06:07.064 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:06:06 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T05:06:07.064 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:06:06 smithi047 conmon[78521]: yield 2022-05-14T05:06:07.065 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:06:06 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T05:06:07.065 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:06:06 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T05:06:07.065 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:06:06 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T05:06:07.065 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:06:06 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T05:06:07.065 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:06:06 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T05:06:07.066 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:06:06 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T05:06:07.066 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:06:06 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T05:06:07.066 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:06:06 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T05:06:07.066 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:06:06 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T05:06:07.067 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:06:06 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T05:06:07.067 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:06:06 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T05:06:07.067 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:06:06 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T05:06:07.067 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:06:06 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T05:06:07.067 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:06:06 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T05:06:07.068 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:06:06 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T05:06:07.068 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:06:06 smithi047 conmon[78521]: 2022-05-14T05:06:07.068 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:06:06 smithi047 conmon[78521]: During handling of the above exception, another exception occurred: 2022-05-14T05:06:07.068 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:06:06 smithi047 conmon[78521]: 2022-05-14T05:06:07.069 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:06:06 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T05:06:07.069 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:06:06 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T05:06:07.069 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:06:06 smithi047 conmon[78521]: return f(*arg) 2022-05-14T05:06:07.069 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:06:06 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T05:06:07.069 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:06:06 smithi047 conmon[78521]: self._write_client_files(client_files, host) 2022-05-14T05:06:07.070 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:06:06 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T05:06:07.070 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:06:06 smithi047 conmon[78521]: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T05:06:07.070 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:06:06 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T05:06:07.070 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:06:06 smithi047 conmon[78521]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T05:06:07.070 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:06:06 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T05:06:07.071 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:06:06 smithi047 conmon[78521]: return self.event_loop.get_result(coro) 2022-05-14T05:06:07.071 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:06:06 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T05:06:07.071 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:06:06 smithi047 conmon[78521]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T05:06:07.071 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:06:06 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T05:06:07.072 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:06:06 smithi047 conmon[78521]: return self.__get_result() 2022-05-14T05:06:07.072 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:06:06 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T05:06:07.072 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:06:06 smithi047 conmon[78521]: raise self._exception 2022-05-14T05:06:07.072 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:06:06 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T05:06:07.072 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:06:06 smithi047 conmon[78521]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T05:06:07.073 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:06:06 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T05:06:07.073 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:06:06 smithi047 conmon[78521]: conn = await self._remote_connection(host, addr) 2022-05-14T05:06:07.073 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:06:06 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T05:06:07.073 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:06:06 smithi047 conmon[78521]: raise 2022-05-14T05:06:07.073 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:06:06 smithi047 conmon[78521]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T05:06:07.074 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:06:06 smithi047 conmon[78521]: self.gen.throw(type, value, traceback) 2022-05-14T05:06:07.074 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:06:06 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T05:06:07.074 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:06:06 smithi047 conmon[78521]: raise OrchestratorError(msg) 2022-05-14T05:06:07.074 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:06:06 smithi047 conmon[78521]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T05:06:07.342 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:06:07 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:06:07.342 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:06:07 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T05:06:07.342 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T05:06:07.343 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T05:06:07.343 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T05:06:07.343 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T05:06:07.343 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T05:06:07.344 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T05:06:07.344 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T05:06:07.344 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T05:06:07.344 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T05:06:07.345 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T05:06:07.345 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T05:06:07.345 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T05:06:07.345 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T05:06:07.346 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T05:06:07.346 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T05:06:07.346 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T05:06:07.346 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T05:06:07.346 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T05:06:07.347 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:06:07 smithi115 ceph-mon[78593]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T05:06:07.347 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T05:06:07.347 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T05:06:07.347 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T05:06:07.347 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T05:06:07.348 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T05:06:07.348 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T05:06:07.348 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T05:06:07.348 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T05:06:07.349 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T05:06:07.349 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T05:06:07.349 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T05:06:07.349 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T05:06:07.350 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T05:06:07.350 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T05:06:07.350 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T05:06:07.350 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T05:06:07.350 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T05:06:07.351 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T05:06:07.351 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T05:06:07.351 INFO:journalctl@ceph.mon.c.smithi115.stdout: During handling of the above exception, another exception occurred: 2022-05-14T05:06:07.351 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T05:06:07.352 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T05:06:07.352 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T05:06:07.352 INFO:journalctl@ceph.mon.c.smithi115.stdout: return f(*arg) 2022-05-14T05:06:07.352 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T05:06:07.352 INFO:journalctl@ceph.mon.c.smithi115.stdout: self._write_client_files(client_files, host) 2022-05-14T05:06:07.353 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T05:06:07.353 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T05:06:07.353 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T05:06:07.353 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T05:06:07.354 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T05:06:07.354 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.event_loop.get_result(coro) 2022-05-14T05:06:07.354 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T05:06:07.354 INFO:journalctl@ceph.mon.c.smithi115.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T05:06:07.354 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T05:06:07.355 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.__get_result() 2022-05-14T05:06:07.355 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T05:06:07.355 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise self._exception 2022-05-14T05:06:07.355 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T05:06:07.356 INFO:journalctl@ceph.mon.c.smithi115.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T05:06:07.356 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T05:06:07.356 INFO:journalctl@ceph.mon.c.smithi115.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T05:06:07.356 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T05:06:07.356 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise 2022-05-14T05:06:07.357 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T05:06:07.357 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.gen.throw(type, value, traceback) 2022-05-14T05:06:07.357 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T05:06:07.357 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OrchestratorError(msg) 2022-05-14T05:06:07.358 INFO:journalctl@ceph.mon.c.smithi115.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T05:06:07.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:06:07 smithi115 conmon[90393]: 2022-05-14T05:06:07.339+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:06:08.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:06:08 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:06:08.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:06:08 smithi047 ceph-mon[77147]: pgmap v10501: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:06:08.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:06:08 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 20300 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T05:06:08.342 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:06:08 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:06:08.343 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:06:08 smithi115 ceph-mon[78593]: pgmap v10501: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:06:08.343 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:06:08 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 20300 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T05:06:08.610 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:06:08 smithi115 conmon[90393]: 2022-05-14T05:06:08.339+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:06:09.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:06:09 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:06:09.348 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:06:09 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:06:09.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:06:09 smithi115 conmon[90393]: 2022-05-14T05:06:09.346+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:06:10.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:06:10 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:06:10.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:06:10 smithi047 ceph-mon[77147]: pgmap v10502: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T05:06:10.331 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:06:10 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:06:10.332 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:06:10 smithi115 ceph-mon[78593]: pgmap v10502: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T05:06:10.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:06:10 smithi115 conmon[90393]: 2022-05-14T05:06:10.329+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:06:11.285 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:06:11 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T05:06:11.286 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:06:11 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T05:06:11.286 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:06:11 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:06:11.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:06:11 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T05:06:11.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:06:11 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T05:06:11.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:06:11 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:06:11.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:06:11 smithi115 conmon[90393]: 2022-05-14T05:06:11.281+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:06:12.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:06:12 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:06:12.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:06:12 smithi047 ceph-mon[77147]: pgmap v10503: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:06:12.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:06:12 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:06:12.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:06:12 smithi115 ceph-mon[78593]: pgmap v10503: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:06:12.360 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:06:12 smithi115 conmon[90393]: 2022-05-14T05:06:12.240+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:06:13.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:06:13 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:06:13.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:06:13 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 20305 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T05:06:13.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:06:13 smithi115 conmon[90393]: 2022-05-14T05:06:13.267+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:06:13.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:06:13 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:06:13.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:06:13 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 20305 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T05:06:14.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:06:14 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:06:14.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:06:14 smithi047 ceph-mon[77147]: pgmap v10504: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:06:14.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:06:14 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:06:14.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:06:14 smithi115 ceph-mon[78593]: pgmap v10504: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:06:14.361 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:06:14 smithi115 conmon[90393]: 2022-05-14T05:06:14.283+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:06:15.317 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:06:15 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T05:06:15.317 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:06:15 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T05:06:15.318 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:06:15 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:06:15.318 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:06:15 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T05:06:15.318 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:06:15 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T05:06:15.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:06:15 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T05:06:15.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:06:15 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T05:06:15.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:06:15 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:06:15.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:06:15 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T05:06:15.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:06:15 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T05:06:15.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:06:15 smithi115 conmon[90393]: 2022-05-14T05:06:15.314+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:06:16.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:06:16 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:06:16.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:06:16 smithi115 ceph-mon[78593]: pgmap v10505: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T05:06:16.360 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:06:16 smithi115 conmon[90393]: 2022-05-14T05:06:16.283+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:06:16.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:06:16 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:06:16.545 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:06:16 smithi047 ceph-mon[77147]: pgmap v10505: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T05:06:17.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:06:17 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:06:17.360 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:06:17 smithi115 conmon[90393]: 2022-05-14T05:06:17.290+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:06:17.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:06:17 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:06:18.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:06:18 smithi115 conmon[90393]: 2022-05-14T05:06:18.256+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:06:18.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:06:18 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:06:18.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:06:18 smithi115 ceph-mon[78593]: pgmap v10506: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:06:18.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:06:18 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 20310 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T05:06:18.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:06:18 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:06:18.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:06:18 smithi047 ceph-mon[77147]: pgmap v10506: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:06:18.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:06:18 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 20310 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T05:06:19.309 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:06:19 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T05:06:19.310 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:06:19 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T05:06:19.310 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:06:19 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:06:19.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:06:19 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T05:06:19.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:06:19 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T05:06:19.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:06:19 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:06:19.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:06:19 smithi115 conmon[90393]: 2022-05-14T05:06:19.306+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:06:20.337 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:06:20 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:06:20.338 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:06:20 smithi115 ceph-mon[78593]: pgmap v10507: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 7 op/s 2022-05-14T05:06:20.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:06:20 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:06:20.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:06:20 smithi047 ceph-mon[77147]: pgmap v10507: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 7 op/s 2022-05-14T05:06:20.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:06:20 smithi115 conmon[90393]: 2022-05-14T05:06:20.335+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:06:21.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:06:21 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:06:21.360 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:06:21 smithi115 conmon[90393]: 2022-05-14T05:06:21.308+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:06:21.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:06:21 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:06:22.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:06:22 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:06:22.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:06:22 smithi115 ceph-mon[78593]: pgmap v10508: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 4 op/s 2022-05-14T05:06:22.360 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:06:22 smithi115 conmon[90393]: 2022-05-14T05:06:22.315+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:06:22.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:06:22 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:06:22.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:06:22 smithi047 ceph-mon[77147]: pgmap v10508: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 4 op/s 2022-05-14T05:06:23.330 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:06:23 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T05:06:23.330 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:06:23 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T05:06:23.331 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:06:23 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:06:23.331 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:06:23 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 20315 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T05:06:23.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:06:23 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T05:06:23.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:06:23 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T05:06:23.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:06:23 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:06:23.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:06:23 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 20315 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T05:06:23.611 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:06:23 smithi115 conmon[90393]: 2022-05-14T05:06:23.328+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:06:24.328 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:06:24 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:06:24.328 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:06:24 smithi115 ceph-mon[78593]: pgmap v10509: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 4 op/s 2022-05-14T05:06:24.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:06:24 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:06:24.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:06:24 smithi047 ceph-mon[77147]: pgmap v10509: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 4 op/s 2022-05-14T05:06:24.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:06:24 smithi115 conmon[90393]: 2022-05-14T05:06:24.320+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:06:25.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:06:25 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:06:25.360 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:06:25 smithi115 conmon[90393]: 2022-05-14T05:06:25.301+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:06:25.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:06:25 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:06:26.344 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:06:26 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:06:26.345 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:06:26 smithi115 ceph-mon[78593]: pgmap v10510: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 7 op/s 2022-05-14T05:06:26.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:06:26 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:06:26.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:06:26 smithi047 ceph-mon[77147]: pgmap v10510: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 7 op/s 2022-05-14T05:06:26.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:06:26 smithi115 conmon[90393]: 2022-05-14T05:06:26.341+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:06:27.341 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:06:27 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:06:27.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:06:27 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:06:27.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:06:27 smithi115 conmon[90393]: 2022-05-14T05:06:27.339+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:06:28.365 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:06:28 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:06:28.365 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:06:28 smithi115 ceph-mon[78593]: pgmap v10511: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 4 op/s 2022-05-14T05:06:28.365 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:06:28 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 20320 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T05:06:28.366 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:06:28 smithi115 conmon[90393]: 2022-05-14T05:06:28.304+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:06:28.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:06:28 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:06:28.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:06:28 smithi047 ceph-mon[77147]: pgmap v10511: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 4 op/s 2022-05-14T05:06:28.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:06:28 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 20320 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T05:06:29.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:06:29 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:06:29.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:06:29 smithi115 conmon[90393]: 2022-05-14T05:06:29.282+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:06:29.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:06:29 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:06:30.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:06:30 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:06:30.362 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:06:30 smithi115 ceph-mon[78593]: pgmap v10512: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 7 op/s 2022-05-14T05:06:30.363 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:06:30 smithi115 conmon[90393]: 2022-05-14T05:06:30.296+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:06:30.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:06:30 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:06:30.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:06:30 smithi047 ceph-mon[77147]: pgmap v10512: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 7 op/s 2022-05-14T05:06:31.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:06:31 smithi115 conmon[90393]: 2022-05-14T05:06:31.281+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:06:31.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:06:31 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T05:06:31.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:06:31 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T05:06:31.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:06:31 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:06:31.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:06:31 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T05:06:31.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:06:31 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T05:06:31.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:06:31 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:06:32.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:06:32 smithi115 conmon[90393]: 2022-05-14T05:06:32.308+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:06:32.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:06:32 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:06:32.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:06:32 smithi115 ceph-mon[78593]: pgmap v10513: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:06:32.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:06:32 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:06:32.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:06:32 smithi047 ceph-mon[77147]: pgmap v10513: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:06:33.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:06:33 smithi115 conmon[90393]: 2022-05-14T05:06:33.263+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:06:33.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:06:33 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:06:33.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:06:33 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 20325 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T05:06:33.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:06:33 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:06:33.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:06:33 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 20325 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T05:06:34.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:06:34 smithi115 conmon[90393]: 2022-05-14T05:06:34.279+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:06:34.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:06:34 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:06:34.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:06:34 smithi115 ceph-mon[78593]: pgmap v10514: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:06:34.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:06:34 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:06:34.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:06:34 smithi047 ceph-mon[77147]: pgmap v10514: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:06:35.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:06:35 smithi115 conmon[90393]: 2022-05-14T05:06:35.314+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:06:35.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:06:35 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T05:06:35.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:06:35 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T05:06:35.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:06:35 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:06:35.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:06:35 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T05:06:35.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:06:35 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T05:06:35.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:06:35 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:06:36.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:06:36 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:06:36.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:06:36 smithi047 ceph-mon[77147]: pgmap v10515: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T05:06:36.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:06:36 smithi115 conmon[90393]: 2022-05-14T05:06:36.266+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:06:36.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:06:36 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:06:36.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:06:36 smithi115 ceph-mon[78593]: pgmap v10515: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T05:06:37.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:06:37 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:06:37.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:06:37 smithi115 conmon[90393]: 2022-05-14T05:06:37.288+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:06:37.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:06:37 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:06:38.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:06:38 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:06:38.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:06:38 smithi047 ceph-mon[77147]: pgmap v10516: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:06:38.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:06:38 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 20330 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T05:06:38.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:06:38 smithi115 conmon[90393]: 2022-05-14T05:06:38.261+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:06:38.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:06:38 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:06:38.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:06:38 smithi115 ceph-mon[78593]: pgmap v10516: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:06:38.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:06:38 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 20330 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T05:06:39.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:06:39 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T05:06:39.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:06:39 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T05:06:39.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:06:39 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:06:39.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:06:39 smithi115 conmon[90393]: 2022-05-14T05:06:39.307+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:06:39.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:06:39 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T05:06:39.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:06:39 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T05:06:39.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:06:39 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:06:40.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:06:40 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:06:40.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:06:40 smithi047 ceph-mon[77147]: pgmap v10517: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T05:06:40.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:06:40 smithi115 conmon[90393]: 2022-05-14T05:06:40.350+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:06:40.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:06:40 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:06:40.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:06:40 smithi115 ceph-mon[78593]: pgmap v10517: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T05:06:41.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:06:41 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:06:41.610 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:06:41 smithi115 conmon[90393]: 2022-05-14T05:06:41.345+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:06:41.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:06:41 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:06:42.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:06:42 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:06:42.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:06:42 smithi047 ceph-mon[77147]: pgmap v10518: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:06:42.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:06:42 smithi115 conmon[90393]: 2022-05-14T05:06:42.313+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:06:42.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:06:42 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:06:42.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:06:42 smithi115 ceph-mon[78593]: pgmap v10518: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:06:43.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:06:43 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T05:06:43.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:06:43 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T05:06:43.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:06:43 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:06:43.545 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:06:43 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 20335 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T05:06:43.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:06:43 smithi115 conmon[90393]: 2022-05-14T05:06:43.362+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:06:43.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:06:43 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T05:06:43.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:06:43 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T05:06:43.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:06:43 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:06:43.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:06:43 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 20335 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T05:06:44.397 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:06:44 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:06:44.398 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:06:44 smithi115 ceph-mon[78593]: pgmap v10519: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:06:44.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:06:44 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:06:44.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:06:44 smithi047 ceph-mon[77147]: pgmap v10519: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:06:44.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:06:44 smithi115 conmon[90393]: 2022-05-14T05:06:44.395+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:06:45.405 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:06:45 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:06:45.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:06:45 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:06:45.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:06:45 smithi115 conmon[90393]: 2022-05-14T05:06:45.402+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:06:46.424 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:06:46 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:06:46.424 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:06:46 smithi115 ceph-mon[78593]: pgmap v10520: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T05:06:46.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:06:46 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:06:46.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:06:46 smithi047 ceph-mon[77147]: pgmap v10520: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T05:06:46.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:06:46 smithi115 conmon[90393]: 2022-05-14T05:06:46.421+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:06:47.432 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:06:47 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:06:47.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:06:47 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:06:47.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:06:47 smithi115 conmon[90393]: 2022-05-14T05:06:47.429+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:06:48.407 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:06:48 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:06:48.407 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:06:48 smithi115 ceph-mon[78593]: pgmap v10521: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:06:48.408 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:06:48 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 20340 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T05:06:48.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:06:48 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:06:48.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:06:48 smithi047 ceph-mon[77147]: pgmap v10521: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:06:48.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:06:48 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 20340 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T05:06:48.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:06:48 smithi115 conmon[90393]: 2022-05-14T05:06:48.404+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:06:49.453 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:06:49 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:06:49.546 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:06:49 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:06:49.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:06:49 smithi115 conmon[90393]: 2022-05-14T05:06:49.450+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:06:50.457 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:06:50 smithi115 ceph-mon[78593]: pgmap v10522: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T05:06:50.457 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:06:50 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:06:50.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:06:50 smithi047 ceph-mon[77147]: pgmap v10522: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T05:06:50.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:06:50 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:06:50.861 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:06:50 smithi115 conmon[90393]: 2022-05-14T05:06:50.454+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:06:51.429 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:06:51 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T05:06:51.429 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:06:51 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T05:06:51.430 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:06:51 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:06:51.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:06:51 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T05:06:51.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:06:51 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T05:06:51.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:06:51 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:06:51.861 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:06:51 smithi115 conmon[90393]: 2022-05-14T05:06:51.417+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:06:52.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:06:52 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:06:52.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:06:52 smithi047 ceph-mon[77147]: pgmap v10523: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:06:52.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:06:52 smithi115 conmon[90393]: 2022-05-14T05:06:52.399+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:06:52.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:06:52 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:06:52.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:06:52 smithi115 ceph-mon[78593]: pgmap v10523: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:06:53.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:06:53 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:06:53.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:06:53 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 20345 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T05:06:53.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:06:53 smithi115 conmon[90393]: 2022-05-14T05:06:53.368+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:06:53.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:06:53 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:06:53.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:06:53 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 20345 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T05:06:54.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:06:54 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:06:54.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:06:54 smithi047 ceph-mon[77147]: pgmap v10524: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:06:54.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:06:54 smithi115 conmon[90393]: 2022-05-14T05:06:54.353+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:06:54.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:06:54 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:06:54.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:06:54 smithi115 ceph-mon[78593]: pgmap v10524: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:06:55.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:06:55 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T05:06:55.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:06:55 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T05:06:55.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:06:55 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:06:55.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:06:55 smithi115 conmon[90393]: 2022-05-14T05:06:55.359+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:06:55.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:06:55 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T05:06:55.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:06:55 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T05:06:55.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:06:55 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:06:56.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:06:56 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:06:56.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:06:56 smithi047 ceph-mon[77147]: pgmap v10525: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T05:06:56.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:06:56 smithi115 conmon[90393]: 2022-05-14T05:06:56.334+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:06:56.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:06:56 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:06:56.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:06:56 smithi115 ceph-mon[78593]: pgmap v10525: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T05:06:57.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:06:57 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:06:57.611 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:06:57 smithi115 conmon[90393]: 2022-05-14T05:06:57.294+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:06:57.612 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:06:57 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:06:58.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:06:58 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:06:58.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:06:58 smithi047 ceph-mon[77147]: pgmap v10526: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:06:58.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:06:58 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 20350 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T05:06:58.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:06:58 smithi115 conmon[90393]: 2022-05-14T05:06:58.252+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:06:58.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:06:58 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:06:58.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:06:58 smithi115 ceph-mon[78593]: pgmap v10526: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:06:58.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:06:58 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 20350 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T05:06:59.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:06:59 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T05:06:59.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:06:59 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T05:06:59.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:06:59 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:06:59.610 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:06:59 smithi115 conmon[90393]: 2022-05-14T05:06:59.269+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:06:59.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:06:59 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T05:06:59.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:06:59 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T05:06:59.612 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:06:59 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:07:00.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:07:00 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:07:00.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:07:00 smithi047 ceph-mon[77147]: pgmap v10527: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T05:07:00.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:07:00 smithi115 conmon[90393]: 2022-05-14T05:07:00.289+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:07:00.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:07:00 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:07:00.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:07:00 smithi115 ceph-mon[78593]: pgmap v10527: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T05:07:01.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:07:01 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:07:01.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:07:01 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:07:01.610 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:07:01 smithi115 conmon[90393]: 2022-05-14T05:07:01.338+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:07:02.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:07:02 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:07:02.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:07:02 smithi047 ceph-mon[77147]: pgmap v10528: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:07:02.611 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:07:02 smithi115 conmon[90393]: 2022-05-14T05:07:02.322+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:07:02.612 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:07:02 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:07:02.612 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:07:02 smithi115 ceph-mon[78593]: pgmap v10528: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:07:03.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:07:03 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T05:07:03.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:07:03 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T05:07:03.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:07:03 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:07:03.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:07:03 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 20355 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T05:07:03.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:07:03 smithi115 conmon[90393]: 2022-05-14T05:07:03.313+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:07:03.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:07:03 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T05:07:03.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:07:03 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T05:07:03.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:07:03 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:07:03.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:07:03 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 20355 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T05:07:04.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:07:04 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:07:04.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:07:04 smithi047 ceph-mon[77147]: pgmap v10529: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:07:04.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:07:04 smithi115 conmon[90393]: 2022-05-14T05:07:04.350+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:07:04.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:07:04 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:07:04.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:07:04 smithi115 ceph-mon[78593]: pgmap v10529: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:07:05.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:07:05 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:07:05.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:07:05 smithi115 conmon[90393]: 2022-05-14T05:07:05.368+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:07:05.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:07:05 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:07:06.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:07:06 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:07:06.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:07:06 smithi047 ceph-mon[77147]: pgmap v10530: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T05:07:06.610 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:07:06 smithi115 conmon[90393]: 2022-05-14T05:07:06.396+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:07:06.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:07:06 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:07:06.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:07:06 smithi115 ceph-mon[78593]: pgmap v10530: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T05:07:07.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:07:07 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:07:07.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:07:07 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T05:07:07.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:07:07 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T05:07:07.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:07:07 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T05:07:07.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:07:07 smithi115 conmon[90393]: 2022-05-14T05:07:07.363+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:07:07.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:07:07 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:07:07.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:07:07 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T05:07:07.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:07:07 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T05:07:07.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:07:07 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T05:07:08.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:07:08 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:07:08.547 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:07:08 smithi047 ceph-mon[77147]: pgmap v10531: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:07:08.547 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:07:08 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 20360 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T05:07:08.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:07:08 smithi115 conmon[90393]: 2022-05-14T05:07:08.373+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:07:08.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:07:08 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:07:08.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:07:08 smithi115 ceph-mon[78593]: pgmap v10531: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:07:08.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:07:08 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 20360 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T05:07:09.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:07:09 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:07:09.610 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:07:09 smithi115 conmon[90393]: 2022-05-14T05:07:09.408+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:07:09.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:07:09 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:07:10.223 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:07:09 smithi047 conmon[78521]: 2022-05-14T05:07:09.872+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T05:07:10.223 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:07:09 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T05:07:10.223 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:07:09 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T05:07:10.224 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:07:09 smithi047 conmon[78521]: yield 2022-05-14T05:07:10.224 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:07:09 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T05:07:10.224 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:07:09 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T05:07:10.224 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:07:09 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T05:07:10.225 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:07:09 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T05:07:10.225 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:07:09 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T05:07:10.225 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:07:09 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T05:07:10.225 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:07:09 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T05:07:10.225 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:07:09 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T05:07:10.226 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:07:09 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T05:07:10.226 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:07:09 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T05:07:10.226 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:07:09 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T05:07:10.226 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:07:09 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T05:07:10.226 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:07:09 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T05:07:10.227 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:07:09 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T05:07:10.227 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:07:09 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T05:07:10.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:07:10 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:07:10.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:07:10 smithi047 ceph-mon[77147]: pgmap v10532: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T05:07:10.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:07:10 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T05:07:10.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T05:07:10.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T05:07:10.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T05:07:10.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T05:07:10.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T05:07:10.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T05:07:10.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T05:07:10.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T05:07:10.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T05:07:10.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T05:07:10.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T05:07:10.548 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T05:07:10.548 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T05:07:10.548 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T05:07:10.549 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T05:07:10.549 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T05:07:10.550 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T05:07:10.550 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T05:07:10.550 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:07:10 smithi047 ceph-mon[77147]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T05:07:10.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:07:10 smithi115 conmon[90393]: 2022-05-14T05:07:10.421+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:07:10.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:07:10 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:07:10.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:07:10 smithi115 ceph-mon[78593]: pgmap v10532: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T05:07:10.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:07:10 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T05:07:10.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T05:07:10.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T05:07:10.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T05:07:10.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T05:07:10.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T05:07:10.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T05:07:10.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T05:07:10.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T05:07:10.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T05:07:10.613 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T05:07:10.613 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T05:07:10.613 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T05:07:10.613 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T05:07:10.613 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T05:07:10.614 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T05:07:10.614 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T05:07:10.614 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T05:07:10.614 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T05:07:10.615 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:07:10 smithi115 ceph-mon[78593]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T05:07:11.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:07:11 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T05:07:11.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:07:11 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T05:07:11.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:07:11 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:07:11.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:07:11 smithi115 conmon[90393]: 2022-05-14T05:07:11.450+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:07:11.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:07:11 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T05:07:11.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:07:11 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T05:07:11.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:07:11 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:07:12.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:07:12 smithi047 ceph-mon[77147]: pgmap v10533: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:07:12.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:07:12 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:07:12.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:07:12 smithi115 conmon[90393]: 2022-05-14T05:07:12.473+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:07:12.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:07:12 smithi115 ceph-mon[78593]: pgmap v10533: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:07:12.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:07:12 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:07:13.232 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:07:12 smithi047 conmon[78521]: 2022-05-14T05:07:12.944+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T05:07:13.232 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:07:12 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T05:07:13.233 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:07:12 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T05:07:13.233 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:07:12 smithi047 conmon[78521]: yield 2022-05-14T05:07:13.233 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:07:12 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T05:07:13.233 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:07:12 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T05:07:13.234 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:07:12 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T05:07:13.234 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:07:12 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T05:07:13.234 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:07:12 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T05:07:13.235 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:07:12 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T05:07:13.235 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:07:12 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T05:07:13.235 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:07:12 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T05:07:13.235 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:07:12 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T05:07:13.236 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:07:12 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T05:07:13.236 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:07:12 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T05:07:13.236 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:07:12 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T05:07:13.236 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:07:12 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T05:07:13.236 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:07:12 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T05:07:13.237 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:07:12 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T05:07:13.237 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:07:12 smithi047 conmon[78521]: 2022-05-14T05:07:12.945+0000 7f5344781700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T05:07:13.237 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:07:12 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T05:07:13.238 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:07:12 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T05:07:13.238 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:07:12 smithi047 conmon[78521]: yield 2022-05-14T05:07:13.238 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:07:12 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T05:07:13.238 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:07:12 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T05:07:13.238 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:07:12 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T05:07:13.239 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:07:12 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T05:07:13.239 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:07:12 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T05:07:13.239 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:07:12 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T05:07:13.239 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:07:12 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T05:07:13.239 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:07:12 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T05:07:13.240 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:07:12 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T05:07:13.240 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:07:12 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T05:07:13.240 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:07:12 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T05:07:13.240 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:07:12 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T05:07:13.241 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:07:12 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T05:07:13.241 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:07:12 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T05:07:13.241 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:07:12 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T05:07:13.241 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:07:12 smithi047 conmon[78521]: 2022-05-14T05:07:13.242 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:07:12 smithi047 conmon[78521]: During handling of the above exception, another exception occurred: 2022-05-14T05:07:13.242 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:07:12 smithi047 conmon[78521]: 2022-05-14T05:07:13.242 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:07:12 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T05:07:13.242 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:07:12 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T05:07:13.242 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:07:12 smithi047 conmon[78521]: return f(*arg) 2022-05-14T05:07:13.243 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:07:12 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T05:07:13.243 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:07:12 smithi047 conmon[78521]: self._write_client_files(client_files, host) 2022-05-14T05:07:13.243 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:07:12 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T05:07:13.244 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:07:12 smithi047 conmon[78521]: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T05:07:13.244 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:07:12 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T05:07:13.244 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:07:12 smithi047 conmon[78521]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T05:07:13.244 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:07:12 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T05:07:13.244 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:07:12 smithi047 conmon[78521]: return self.event_loop.get_result(coro) 2022-05-14T05:07:13.245 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:07:12 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T05:07:13.245 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:07:12 smithi047 conmon[78521]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T05:07:13.245 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:07:12 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T05:07:13.245 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:07:12 smithi047 conmon[78521]: return self.__get_result() 2022-05-14T05:07:13.246 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:07:12 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T05:07:13.246 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:07:12 smithi047 conmon[78521]: raise self._exception 2022-05-14T05:07:13.246 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:07:12 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T05:07:13.246 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:07:12 smithi047 conmon[78521]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T05:07:13.247 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:07:12 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T05:07:13.247 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:07:12 smithi047 conmon[78521]: conn = await self._remote_connection(host, addr) 2022-05-14T05:07:13.247 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:07:12 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T05:07:13.247 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:07:12 smithi047 conmon[78521]: raise 2022-05-14T05:07:13.247 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:07:12 smithi047 conmon[78521]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T05:07:13.248 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:07:12 smithi047 conmon[78521]: self.gen.throw(type, value, traceback) 2022-05-14T05:07:13.248 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:07:12 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T05:07:13.248 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:07:12 smithi047 conmon[78521]: raise OrchestratorError(msg) 2022-05-14T05:07:13.248 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:07:12 smithi047 conmon[78521]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T05:07:13.486 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:07:13 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:07:13.487 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:07:13 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T05:07:13.487 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T05:07:13.487 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T05:07:13.487 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T05:07:13.488 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T05:07:13.488 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T05:07:13.488 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T05:07:13.488 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T05:07:13.489 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T05:07:13.489 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T05:07:13.489 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T05:07:13.489 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T05:07:13.489 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T05:07:13.490 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T05:07:13.490 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T05:07:13.490 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T05:07:13.490 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T05:07:13.491 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T05:07:13.491 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T05:07:13.491 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:07:13 smithi115 ceph-mon[78593]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T05:07:13.491 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T05:07:13.492 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T05:07:13.492 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T05:07:13.492 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T05:07:13.492 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T05:07:13.493 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T05:07:13.493 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T05:07:13.493 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T05:07:13.493 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T05:07:13.494 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T05:07:13.494 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T05:07:13.494 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T05:07:13.494 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T05:07:13.494 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T05:07:13.495 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T05:07:13.495 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T05:07:13.495 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T05:07:13.495 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T05:07:13.496 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T05:07:13.496 INFO:journalctl@ceph.mon.c.smithi115.stdout: During handling of the above exception, another exception occurred: 2022-05-14T05:07:13.496 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T05:07:13.496 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T05:07:13.497 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T05:07:13.497 INFO:journalctl@ceph.mon.c.smithi115.stdout: return f(*arg) 2022-05-14T05:07:13.497 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T05:07:13.498 INFO:journalctl@ceph.mon.c.smithi115.stdout: self._write_client_files(client_files, host) 2022-05-14T05:07:13.498 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T05:07:13.498 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T05:07:13.498 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T05:07:13.498 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T05:07:13.499 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T05:07:13.499 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.event_loop.get_result(coro) 2022-05-14T05:07:13.499 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T05:07:13.499 INFO:journalctl@ceph.mon.c.smithi115.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T05:07:13.500 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T05:07:13.500 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.__get_result() 2022-05-14T05:07:13.500 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T05:07:13.500 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise self._exception 2022-05-14T05:07:13.501 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T05:07:13.501 INFO:journalctl@ceph.mon.c.smithi115.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T05:07:13.501 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T05:07:13.501 INFO:journalctl@ceph.mon.c.smithi115.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T05:07:13.501 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T05:07:13.502 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise 2022-05-14T05:07:13.502 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T05:07:13.502 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.gen.throw(type, value, traceback) 2022-05-14T05:07:13.502 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T05:07:13.503 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OrchestratorError(msg) 2022-05-14T05:07:13.503 INFO:journalctl@ceph.mon.c.smithi115.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T05:07:13.503 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:07:13 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 20365 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T05:07:13.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:07:13 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:07:13.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:07:13 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T05:07:13.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T05:07:13.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T05:07:13.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T05:07:13.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T05:07:13.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T05:07:13.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T05:07:13.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T05:07:13.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T05:07:13.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T05:07:13.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T05:07:13.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T05:07:13.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T05:07:13.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T05:07:13.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T05:07:13.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T05:07:13.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T05:07:13.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T05:07:13.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T05:07:13.547 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:07:13 smithi047 ceph-mon[77147]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T05:07:13.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T05:07:13.548 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T05:07:13.548 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T05:07:13.548 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T05:07:13.548 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T05:07:13.548 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T05:07:13.549 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T05:07:13.549 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T05:07:13.549 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T05:07:13.549 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T05:07:13.549 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T05:07:13.550 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T05:07:13.550 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T05:07:13.550 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T05:07:13.550 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T05:07:13.550 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T05:07:13.551 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T05:07:13.551 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T05:07:13.551 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T05:07:13.551 INFO:journalctl@ceph.mon.b.smithi047.stdout: During handling of the above exception, another exception occurred: 2022-05-14T05:07:13.551 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T05:07:13.552 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T05:07:13.552 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T05:07:13.552 INFO:journalctl@ceph.mon.b.smithi047.stdout: return f(*arg) 2022-05-14T05:07:13.552 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T05:07:13.552 INFO:journalctl@ceph.mon.b.smithi047.stdout: self._write_client_files(client_files, host) 2022-05-14T05:07:13.553 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T05:07:13.553 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T05:07:13.553 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T05:07:13.553 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T05:07:13.554 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T05:07:13.554 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.event_loop.get_result(coro) 2022-05-14T05:07:13.554 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T05:07:13.554 INFO:journalctl@ceph.mon.b.smithi047.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T05:07:13.554 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T05:07:13.555 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.__get_result() 2022-05-14T05:07:13.555 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T05:07:13.555 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise self._exception 2022-05-14T05:07:13.555 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T05:07:13.556 INFO:journalctl@ceph.mon.b.smithi047.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T05:07:13.556 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T05:07:13.556 INFO:journalctl@ceph.mon.b.smithi047.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T05:07:13.556 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T05:07:13.556 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise 2022-05-14T05:07:13.557 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T05:07:13.557 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.gen.throw(type, value, traceback) 2022-05-14T05:07:13.557 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T05:07:13.557 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OrchestratorError(msg) 2022-05-14T05:07:13.557 INFO:journalctl@ceph.mon.b.smithi047.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T05:07:13.558 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:07:13 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 20365 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T05:07:13.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:07:13 smithi115 conmon[90393]: 2022-05-14T05:07:13.483+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:07:14.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:07:14 smithi047 ceph-mon[77147]: pgmap v10534: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:07:14.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:07:14 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:07:14.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:07:14 smithi115 conmon[90393]: 2022-05-14T05:07:14.453+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:07:14.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:07:14 smithi115 ceph-mon[78593]: pgmap v10534: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:07:14.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:07:14 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:07:15.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:07:15 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T05:07:15.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:07:15 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T05:07:15.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:07:15 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:07:15.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:07:15 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T05:07:15.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:07:15 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T05:07:15.610 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:07:15 smithi115 conmon[90393]: 2022-05-14T05:07:15.456+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:07:15.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:07:15 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T05:07:15.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:07:15 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T05:07:15.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:07:15 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:07:15.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:07:15 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T05:07:15.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:07:15 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T05:07:16.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:07:16 smithi047 ceph-mon[77147]: pgmap v10535: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T05:07:16.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:07:16 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:07:16.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:07:16 smithi115 ceph-mon[78593]: pgmap v10535: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T05:07:16.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:07:16 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:07:16.610 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:07:16 smithi115 conmon[90393]: 2022-05-14T05:07:16.485+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:07:17.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:07:17 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:07:17.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:07:17 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:07:17.610 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:07:17 smithi115 conmon[90393]: 2022-05-14T05:07:17.489+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:07:18.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:07:18 smithi047 ceph-mon[77147]: pgmap v10536: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:07:18.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:07:18 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:07:18.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:07:18 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 20370 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T05:07:18.610 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:07:18 smithi115 conmon[90393]: 2022-05-14T05:07:18.472+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:07:18.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:07:18 smithi115 ceph-mon[78593]: pgmap v10536: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:07:18.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:07:18 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:07:18.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:07:18 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 20370 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T05:07:19.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:07:19 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T05:07:19.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:07:19 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T05:07:19.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:07:19 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:07:19.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:07:19 smithi115 conmon[90393]: 2022-05-14T05:07:19.494+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:07:19.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:07:19 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T05:07:19.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:07:19 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T05:07:19.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:07:19 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:07:20.514 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:07:20 smithi115 ceph-mon[78593]: pgmap v10537: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T05:07:20.514 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:07:20 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:07:20.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:07:20 smithi047 ceph-mon[77147]: pgmap v10537: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T05:07:20.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:07:20 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:07:20.864 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:07:20 smithi115 conmon[90393]: 2022-05-14T05:07:20.512+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:07:21.523 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:07:21 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:07:21.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:07:21 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:07:21.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:07:21 smithi115 conmon[90393]: 2022-05-14T05:07:21.521+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:07:22.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:07:22 smithi047 ceph-mon[77147]: pgmap v10538: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:07:22.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:07:22 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:07:22.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:07:22 smithi115 conmon[90393]: 2022-05-14T05:07:22.506+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:07:22.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:07:22 smithi115 ceph-mon[78593]: pgmap v10538: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:07:22.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:07:22 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:07:23.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:07:23 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T05:07:23.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:07:23 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T05:07:23.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:07:23 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:07:23.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:07:23 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 20375 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T05:07:23.547 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:07:23 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T05:07:23.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:07:23 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T05:07:23.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:07:23 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:07:23.548 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:07:23 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 20375 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T05:07:23.864 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:07:23 smithi115 conmon[90393]: 2022-05-14T05:07:23.544+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:07:24.521 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:07:24 smithi115 ceph-mon[78593]: pgmap v10539: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:07:24.521 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:07:24 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:07:24.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:07:24 smithi047 ceph-mon[77147]: pgmap v10539: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:07:24.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:07:24 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:07:24.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:07:24 smithi115 conmon[90393]: 2022-05-14T05:07:24.519+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:07:25.526 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:07:25 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:07:25.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:07:25 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:07:25.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:07:25 smithi115 conmon[90393]: 2022-05-14T05:07:25.524+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:07:26.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:07:26 smithi047 ceph-mon[77147]: pgmap v10540: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T05:07:26.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:07:26 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:07:26.568 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:07:26 smithi115 ceph-mon[78593]: pgmap v10540: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T05:07:26.568 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:07:26 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:07:26.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:07:26 smithi115 conmon[90393]: 2022-05-14T05:07:26.565+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:07:27.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:07:27 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:07:27.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:07:27 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:07:28.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:07:27 smithi115 conmon[90393]: 2022-05-14T05:07:27.615+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:07:28.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:07:28 smithi047 ceph-mon[77147]: pgmap v10541: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:07:28.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:07:28 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:07:28.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:07:28 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 20380 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T05:07:28.606 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:07:28 smithi115 ceph-mon[78593]: pgmap v10541: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:07:28.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:07:28 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:07:28.607 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:07:28 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 20380 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T05:07:28.861 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:07:28 smithi115 conmon[90393]: 2022-05-14T05:07:28.604+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:07:29.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:07:29 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:07:29.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:07:29 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:07:30.110 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:07:29 smithi115 conmon[90393]: 2022-05-14T05:07:29.644+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:07:30.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:07:30 smithi047 ceph-mon[77147]: pgmap v10542: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T05:07:30.548 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:07:30 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:07:30.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:07:30 smithi115 ceph-mon[78593]: pgmap v10542: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T05:07:30.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:07:30 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:07:31.110 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:07:30 smithi115 conmon[90393]: 2022-05-14T05:07:30.615+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:07:31.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:07:31 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T05:07:31.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:07:31 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T05:07:31.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:07:31 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:07:31.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:07:31 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T05:07:31.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:07:31 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T05:07:31.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:07:31 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:07:32.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:07:31 smithi115 conmon[90393]: 2022-05-14T05:07:31.643+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:07:32.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:07:32 smithi115 ceph-mon[78593]: pgmap v10543: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:07:32.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:07:32 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:07:32.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:07:32 smithi047 ceph-mon[77147]: pgmap v10543: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:07:32.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:07:32 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:07:33.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:07:32 smithi115 conmon[90393]: 2022-05-14T05:07:32.673+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:07:33.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:07:33 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:07:33.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:07:33 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 20385 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T05:07:33.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:07:33 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:07:33.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:07:33 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 20385 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T05:07:34.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:07:33 smithi115 conmon[90393]: 2022-05-14T05:07:33.625+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:07:34.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:07:34 smithi115 ceph-mon[78593]: pgmap v10544: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:07:34.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:07:34 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:07:34.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:07:34 smithi047 ceph-mon[77147]: pgmap v10544: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:07:34.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:07:34 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:07:35.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:07:34 smithi115 conmon[90393]: 2022-05-14T05:07:34.654+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:07:35.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:07:35 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T05:07:35.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:07:35 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T05:07:35.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:07:35 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:07:35.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:07:35 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T05:07:35.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:07:35 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T05:07:35.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:07:35 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:07:36.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:07:35 smithi115 conmon[90393]: 2022-05-14T05:07:35.607+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:07:36.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:07:36 smithi115 ceph-mon[78593]: pgmap v10545: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T05:07:36.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:07:36 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:07:36.795 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:07:36 smithi047 ceph-mon[77147]: pgmap v10545: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T05:07:36.796 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:07:36 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:07:37.110 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:07:36 smithi115 conmon[90393]: 2022-05-14T05:07:36.620+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:07:37.604 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:07:37 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:07:37.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:07:37 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:07:37.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:07:37 smithi115 conmon[90393]: 2022-05-14T05:07:37.600+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:07:38.578 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:07:38 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T05:07:38.578 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:07:38 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T05:07:38.579 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:07:38 smithi115 ceph-mon[78593]: pgmap v10546: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:07:38.579 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:07:38 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:07:38.579 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:07:38 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 20390 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T05:07:38.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:07:38 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T05:07:38.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:07:38 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T05:07:38.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:07:38 smithi047 ceph-mon[77147]: pgmap v10546: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:07:38.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:07:38 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:07:38.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:07:38 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 20390 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T05:07:38.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:07:38 smithi115 conmon[90393]: 2022-05-14T05:07:38.574+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:07:39.590 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:07:39 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:07:39.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:07:39 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:07:39.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:07:39 smithi115 conmon[90393]: 2022-05-14T05:07:39.585+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:07:40.599 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:07:40 smithi115 ceph-mon[78593]: pgmap v10547: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T05:07:40.600 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:07:40 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:07:40.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:07:40 smithi047 ceph-mon[77147]: pgmap v10547: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T05:07:40.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:07:40 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:07:40.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:07:40 smithi115 conmon[90393]: 2022-05-14T05:07:40.596+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:07:41.603 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:07:41 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:07:41.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:07:41 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:07:41.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:07:41 smithi115 conmon[90393]: 2022-05-14T05:07:41.600+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:07:42.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:07:42 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T05:07:42.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:07:42 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T05:07:42.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:07:42 smithi115 ceph-mon[78593]: pgmap v10548: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:07:42.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:07:42 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:07:42.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:07:42 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T05:07:42.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:07:42 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T05:07:42.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:07:42 smithi047 ceph-mon[77147]: pgmap v10548: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:07:42.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:07:42 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:07:43.111 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:07:42 smithi115 conmon[90393]: 2022-05-14T05:07:42.613+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:07:43.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:07:43 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:07:43.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:07:43 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 20395 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T05:07:43.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:07:43 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:07:43.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:07:43 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 20395 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T05:07:44.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:07:43 smithi115 conmon[90393]: 2022-05-14T05:07:43.621+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:07:44.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:07:44 smithi115 ceph-mon[78593]: pgmap v10549: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:07:44.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:07:44 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:07:44.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:07:44 smithi047 ceph-mon[77147]: pgmap v10549: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:07:44.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:07:44 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:07:45.110 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:07:44 smithi115 conmon[90393]: 2022-05-14T05:07:44.633+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:07:45.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:07:45 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:07:45.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:07:45 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:07:46.110 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:07:45 smithi115 conmon[90393]: 2022-05-14T05:07:45.634+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:07:46.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:07:46 smithi115 ceph-mon[78593]: pgmap v10550: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T05:07:46.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:07:46 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:07:46.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:07:46 smithi047 ceph-mon[77147]: pgmap v10550: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T05:07:46.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:07:46 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:07:47.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:07:46 smithi115 conmon[90393]: 2022-05-14T05:07:46.610+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:07:47.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:07:47 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:07:47.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:07:47 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:07:48.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:07:47 smithi115 conmon[90393]: 2022-05-14T05:07:47.633+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:07:48.646 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:07:48 smithi115 ceph-mon[78593]: pgmap v10551: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:07:48.646 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:07:48 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:07:48.647 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:07:48 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 20400 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T05:07:48.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:07:48 smithi047 ceph-mon[77147]: pgmap v10551: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:07:48.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:07:48 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:07:48.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:07:48 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 20400 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T05:07:49.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:07:48 smithi115 conmon[90393]: 2022-05-14T05:07:48.643+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:07:49.627 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:07:49 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:07:49.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:07:49 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:07:50.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:07:49 smithi115 conmon[90393]: 2022-05-14T05:07:49.624+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:07:50.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:07:50 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T05:07:50.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:07:50 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T05:07:50.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:07:50 smithi047 ceph-mon[77147]: pgmap v10552: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T05:07:50.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:07:50 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:07:50.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:07:50 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T05:07:50.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:07:50 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T05:07:50.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:07:50 smithi115 ceph-mon[78593]: pgmap v10552: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T05:07:50.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:07:50 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:07:50.860 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:07:50 smithi115 conmon[90393]: 2022-05-14T05:07:50.589+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:07:51.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:07:51 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:07:51.860 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:07:51 smithi115 conmon[90393]: 2022-05-14T05:07:51.562+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:07:51.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:07:51 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:07:52.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:07:52 smithi047 ceph-mon[77147]: pgmap v10553: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:07:52.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:07:52 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:07:52.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:07:52 smithi115 conmon[90393]: 2022-05-14T05:07:52.590+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:07:52.896 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:07:52 smithi115 ceph-mon[78593]: pgmap v10553: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:07:52.897 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:07:52 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:07:53.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:07:53 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:07:53.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:07:53 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 20405 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T05:07:53.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:07:53 smithi115 conmon[90393]: 2022-05-14T05:07:53.560+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:07:53.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:07:53 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:07:53.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:07:53 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 20405 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T05:07:54.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:07:54 smithi047 ceph-mon[77147]: pgmap v10554: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:07:54.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:07:54 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:07:54.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:07:54 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T05:07:54.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:07:54 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T05:07:54.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:07:54 smithi115 conmon[90393]: 2022-05-14T05:07:54.582+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:07:54.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:07:54 smithi115 ceph-mon[78593]: pgmap v10554: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:07:54.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:07:54 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:07:54.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:07:54 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T05:07:54.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:07:54 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T05:07:55.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:07:55 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:07:55.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:07:55 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:07:55.860 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:07:55 smithi115 conmon[90393]: 2022-05-14T05:07:55.548+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:07:56.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:07:56 smithi047 ceph-mon[77147]: pgmap v10555: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T05:07:56.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:07:56 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:07:56.862 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:07:56 smithi115 conmon[90393]: 2022-05-14T05:07:56.539+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:07:56.862 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:07:56 smithi115 ceph-mon[78593]: pgmap v10555: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T05:07:56.863 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:07:56 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:07:57.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:07:57 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:07:57.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:07:57 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:07:57.860 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:07:57 smithi115 conmon[90393]: 2022-05-14T05:07:57.587+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:07:58.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:07:58 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T05:07:58.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:07:58 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T05:07:58.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:07:58 smithi047 ceph-mon[77147]: pgmap v10556: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:07:58.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:07:58 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:07:58.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:07:58 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 20410 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T05:07:58.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:07:58 smithi115 conmon[90393]: 2022-05-14T05:07:58.568+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:07:58.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:07:58 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T05:07:58.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:07:58 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T05:07:58.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:07:58 smithi115 ceph-mon[78593]: pgmap v10556: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:07:58.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:07:58 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:07:58.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:07:58 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 20410 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T05:07:59.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:07:59 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:07:59.860 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:07:59 smithi115 conmon[90393]: 2022-05-14T05:07:59.547+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:07:59.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:07:59 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:08:00.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:08:00 smithi047 ceph-mon[77147]: pgmap v10557: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T05:08:00.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:08:00 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:08:00.860 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:08:00 smithi115 conmon[90393]: 2022-05-14T05:08:00.570+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:08:00.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:08:00 smithi115 ceph-mon[78593]: pgmap v10557: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T05:08:00.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:08:00 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:08:01.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:08:01 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:08:01.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:08:01 smithi115 conmon[90393]: 2022-05-14T05:08:01.536+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:08:01.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:08:01 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:08:02.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:08:02 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T05:08:02.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:08:02 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T05:08:02.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:08:02 smithi047 ceph-mon[77147]: pgmap v10558: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:08:02.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:08:02 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:08:02.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:08:02 smithi115 conmon[90393]: 2022-05-14T05:08:02.568+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:08:02.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:08:02 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T05:08:02.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:08:02 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T05:08:02.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:08:02 smithi115 ceph-mon[78593]: pgmap v10558: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:08:02.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:08:02 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:08:03.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:08:03 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:08:03.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:08:03 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 20415 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T05:08:03.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:08:03 smithi115 conmon[90393]: 2022-05-14T05:08:03.555+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:08:03.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:08:03 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:08:03.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:08:03 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 20415 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T05:08:04.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:08:04 smithi047 ceph-mon[77147]: pgmap v10559: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:08:04.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:08:04 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:08:04.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:08:04 smithi115 conmon[90393]: 2022-05-14T05:08:04.550+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:08:04.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:08:04 smithi115 ceph-mon[78593]: pgmap v10559: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:08:04.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:08:04 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:08:05.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:08:05 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:08:05.862 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:08:05 smithi115 conmon[90393]: 2022-05-14T05:08:05.527+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:08:05.862 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:08:05 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:08:06.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:08:06 smithi047 ceph-mon[77147]: pgmap v10560: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T05:08:06.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:08:06 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:08:06.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:08:06 smithi115 conmon[90393]: 2022-05-14T05:08:06.497+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:08:06.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:08:06 smithi115 ceph-mon[78593]: pgmap v10560: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T05:08:06.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:08:06 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:08:07.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:08:07 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:08:07.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:08:07 smithi115 conmon[90393]: 2022-05-14T05:08:07.503+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:08:07.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:08:07 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:08:08.797 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:08:08 smithi047 ceph-mon[77147]: pgmap v10561: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:08:08.797 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:08:08 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:08:08.797 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:08:08 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 20420 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T05:08:08.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:08:08 smithi115 conmon[90393]: 2022-05-14T05:08:08.479+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:08:08.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:08:08 smithi115 ceph-mon[78593]: pgmap v10561: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:08:08.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:08:08 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:08:08.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:08:08 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 20420 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T05:08:09.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:08:09 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:08:09.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:08:09 smithi115 conmon[90393]: 2022-05-14T05:08:09.448+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:08:09.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:08:09 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:08:10.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:08:10 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T05:08:10.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:08:10 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T05:08:10.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:08:10 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:08:10.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:08:10 smithi047 ceph-mon[77147]: pgmap v10562: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T05:08:10.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:08:10 smithi115 conmon[90393]: 2022-05-14T05:08:10.486+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:08:10.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:08:10 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T05:08:10.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:08:10 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T05:08:10.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:08:10 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:08:10.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:08:10 smithi115 ceph-mon[78593]: pgmap v10562: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T05:08:11.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:08:11 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:08:11.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:08:11 smithi115 conmon[90393]: 2022-05-14T05:08:11.490+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:08:11.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:08:11 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:08:12.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:08:12 smithi047 ceph-mon[77147]: pgmap v10563: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:08:12.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:08:12 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:08:12.861 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:08:12 smithi115 conmon[90393]: 2022-05-14T05:08:12.489+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:08:12.862 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:08:12 smithi115 ceph-mon[78593]: pgmap v10563: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:08:12.862 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:08:12 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:08:13.795 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:08:13 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:08:13.796 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:08:13 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T05:08:13.796 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:08:13 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T05:08:13.796 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:08:13 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T05:08:13.797 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:08:13 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 20425 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T05:08:13.797 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:08:13 smithi047 conmon[78521]: 2022-05-14T05:08:13.423+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T05:08:13.798 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:08:13 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T05:08:13.798 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:08:13 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T05:08:13.799 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:08:13 smithi047 conmon[78521]: yield 2022-05-14T05:08:13.799 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:08:13 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T05:08:13.799 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:08:13 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T05:08:13.800 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:08:13 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T05:08:13.800 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:08:13 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T05:08:13.801 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:08:13 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T05:08:13.801 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:08:13 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T05:08:13.802 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:08:13 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T05:08:13.802 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:08:13 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T05:08:13.802 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:08:13 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T05:08:13.803 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:08:13 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T05:08:13.803 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:08:13 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T05:08:13.804 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:08:13 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T05:08:13.804 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:08:13 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T05:08:13.804 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:08:13 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T05:08:13.804 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:08:13 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T05:08:13.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:08:13 smithi115 conmon[90393]: 2022-05-14T05:08:13.481+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:08:13.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:08:13 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:08:13.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:08:13 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T05:08:13.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:08:13 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T05:08:13.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:08:13 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T05:08:13.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:08:13 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 20425 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T05:08:14.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:08:14 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T05:08:14.793 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T05:08:14.793 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T05:08:14.793 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T05:08:14.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T05:08:14.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T05:08:14.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T05:08:14.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T05:08:14.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T05:08:14.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T05:08:14.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T05:08:14.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T05:08:14.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T05:08:14.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T05:08:14.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T05:08:14.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T05:08:14.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T05:08:14.797 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T05:08:14.797 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T05:08:14.797 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:08:14 smithi047 ceph-mon[77147]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T05:08:14.797 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:08:14 smithi047 ceph-mon[77147]: pgmap v10564: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:08:14.798 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:08:14 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:08:14.798 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:08:14 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T05:08:14.798 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:08:14 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T05:08:14.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:08:14 smithi115 conmon[90393]: 2022-05-14T05:08:14.447+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:08:14.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:08:14 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T05:08:14.860 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T05:08:14.860 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T05:08:14.860 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T05:08:14.860 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T05:08:14.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T05:08:14.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T05:08:14.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T05:08:14.863 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T05:08:14.863 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T05:08:14.864 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T05:08:14.864 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T05:08:14.864 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T05:08:14.865 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T05:08:14.865 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T05:08:14.865 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T05:08:14.865 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T05:08:14.866 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T05:08:14.866 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T05:08:14.867 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:08:14 smithi115 ceph-mon[78593]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T05:08:14.867 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:08:14 smithi115 ceph-mon[78593]: pgmap v10564: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:08:14.867 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:08:14 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:08:14.867 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:08:14 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T05:08:14.868 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:08:14 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T05:08:15.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:08:15 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:08:15.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:08:15 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T05:08:15.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:08:15 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T05:08:15.861 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:08:15 smithi115 conmon[90393]: 2022-05-14T05:08:15.437+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:08:15.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:08:15 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:08:15.862 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:08:15 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T05:08:15.862 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:08:15 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T05:08:16.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:08:16 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:08:16.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:08:16 smithi047 ceph-mon[77147]: pgmap v10565: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T05:08:16.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:08:16 smithi047 conmon[78521]: 2022-05-14T05:08:16.495+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T05:08:16.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:08:16 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T05:08:16.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:08:16 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T05:08:16.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:08:16 smithi047 conmon[78521]: yield 2022-05-14T05:08:16.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:08:16 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T05:08:16.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:08:16 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T05:08:16.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:08:16 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T05:08:16.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:08:16 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T05:08:16.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:08:16 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T05:08:16.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:08:16 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T05:08:16.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:08:16 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T05:08:16.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:08:16 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T05:08:16.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:08:16 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T05:08:16.797 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:08:16 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T05:08:16.797 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:08:16 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T05:08:16.797 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:08:16 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T05:08:16.797 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:08:16 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T05:08:16.797 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:08:16 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T05:08:16.798 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:08:16 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T05:08:16.798 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:08:16 smithi047 conmon[78521]: 2022-05-14T05:08:16.496+0000 7f5340779700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T05:08:16.798 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:08:16 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T05:08:16.798 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:08:16 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T05:08:16.799 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:08:16 smithi047 conmon[78521]: yield 2022-05-14T05:08:16.799 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:08:16 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T05:08:16.799 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:08:16 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T05:08:16.799 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:08:16 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T05:08:16.799 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:08:16 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T05:08:16.800 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:08:16 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T05:08:16.800 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:08:16 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T05:08:16.800 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:08:16 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T05:08:16.800 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:08:16 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T05:08:16.801 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:08:16 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T05:08:16.801 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:08:16 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T05:08:16.801 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:08:16 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T05:08:16.801 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:08:16 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T05:08:16.802 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:08:16 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T05:08:16.802 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:08:16 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T05:08:16.802 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:08:16 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T05:08:16.802 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:08:16 smithi047 conmon[78521]: 2022-05-14T05:08:16.802 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:08:16 smithi047 conmon[78521]: During handling of the above exception, another exception occurred: 2022-05-14T05:08:16.803 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:08:16 smithi047 conmon[78521]: 2022-05-14T05:08:16.803 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:08:16 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T05:08:16.803 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:08:16 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T05:08:16.803 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:08:16 smithi047 conmon[78521]: return f(*arg) 2022-05-14T05:08:16.804 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:08:16 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T05:08:16.804 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:08:16 smithi047 conmon[78521]: self._write_client_files(client_files, host) 2022-05-14T05:08:16.804 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:08:16 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T05:08:16.805 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:08:16 smithi047 conmon[78521]: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T05:08:16.805 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:08:16 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T05:08:16.805 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:08:16 smithi047 conmon[78521]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T05:08:16.805 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:08:16 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T05:08:16.806 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:08:16 smithi047 conmon[78521]: return self.event_loop.get_result(coro) 2022-05-14T05:08:16.806 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:08:16 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T05:08:16.806 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:08:16 smithi047 conmon[78521]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T05:08:16.806 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:08:16 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T05:08:16.806 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:08:16 smithi047 conmon[78521]: return self.__get_result() 2022-05-14T05:08:16.807 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:08:16 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T05:08:16.807 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:08:16 smithi047 conmon[78521]: raise self._exception 2022-05-14T05:08:16.807 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:08:16 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T05:08:16.807 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:08:16 smithi047 conmon[78521]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T05:08:16.808 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:08:16 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T05:08:16.808 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:08:16 smithi047 conmon[78521]: conn = await self._remote_connection(host, addr) 2022-05-14T05:08:16.808 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:08:16 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T05:08:16.808 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:08:16 smithi047 conmon[78521]: raise 2022-05-14T05:08:16.808 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:08:16 smithi047 conmon[78521]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T05:08:16.809 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:08:16 smithi047 conmon[78521]: self.gen.throw(type, value, traceback) 2022-05-14T05:08:16.809 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:08:16 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T05:08:16.809 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:08:16 smithi047 conmon[78521]: raise OrchestratorError(msg) 2022-05-14T05:08:16.809 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:08:16 smithi047 conmon[78521]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T05:08:16.860 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:08:16 smithi115 conmon[90393]: 2022-05-14T05:08:16.428+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:08:16.862 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:08:16 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:08:16.862 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:08:16 smithi115 ceph-mon[78593]: pgmap v10565: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T05:08:17.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:08:17 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:08:17.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:08:17 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T05:08:17.793 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T05:08:17.793 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T05:08:17.793 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T05:08:17.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T05:08:17.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T05:08:17.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T05:08:17.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T05:08:17.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T05:08:17.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T05:08:17.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T05:08:17.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T05:08:17.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T05:08:17.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T05:08:17.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T05:08:17.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T05:08:17.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T05:08:17.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T05:08:17.797 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T05:08:17.797 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:08:17 smithi047 ceph-mon[77147]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T05:08:17.797 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T05:08:17.797 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T05:08:17.798 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T05:08:17.798 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T05:08:17.798 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T05:08:17.798 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T05:08:17.798 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T05:08:17.799 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T05:08:17.799 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T05:08:17.799 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T05:08:17.799 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T05:08:17.800 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T05:08:17.800 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T05:08:17.800 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T05:08:17.800 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T05:08:17.800 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T05:08:17.801 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T05:08:17.801 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T05:08:17.801 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T05:08:17.801 INFO:journalctl@ceph.mon.b.smithi047.stdout: During handling of the above exception, another exception occurred: 2022-05-14T05:08:17.801 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T05:08:17.802 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T05:08:17.802 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T05:08:17.802 INFO:journalctl@ceph.mon.b.smithi047.stdout: return f(*arg) 2022-05-14T05:08:17.802 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T05:08:17.802 INFO:journalctl@ceph.mon.b.smithi047.stdout: self._write_client_files(client_files, host) 2022-05-14T05:08:17.803 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T05:08:17.803 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T05:08:17.803 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T05:08:17.803 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T05:08:17.804 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T05:08:17.804 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.event_loop.get_result(coro) 2022-05-14T05:08:17.804 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T05:08:17.804 INFO:journalctl@ceph.mon.b.smithi047.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T05:08:17.804 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T05:08:17.805 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.__get_result() 2022-05-14T05:08:17.805 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T05:08:17.805 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise self._exception 2022-05-14T05:08:17.805 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T05:08:17.805 INFO:journalctl@ceph.mon.b.smithi047.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T05:08:17.806 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T05:08:17.806 INFO:journalctl@ceph.mon.b.smithi047.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T05:08:17.806 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T05:08:17.806 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise 2022-05-14T05:08:17.806 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T05:08:17.807 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.gen.throw(type, value, traceback) 2022-05-14T05:08:17.807 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T05:08:17.807 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OrchestratorError(msg) 2022-05-14T05:08:17.807 INFO:journalctl@ceph.mon.b.smithi047.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T05:08:17.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:08:17 smithi115 conmon[90393]: 2022-05-14T05:08:17.475+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:08:17.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:08:17 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:08:17.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:08:17 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T05:08:17.860 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T05:08:17.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T05:08:17.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T05:08:17.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T05:08:17.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T05:08:17.861 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T05:08:17.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T05:08:17.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T05:08:17.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T05:08:17.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T05:08:17.863 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T05:08:17.863 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T05:08:17.863 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T05:08:17.863 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T05:08:17.863 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T05:08:17.864 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T05:08:17.864 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T05:08:17.864 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T05:08:17.864 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:08:17 smithi115 ceph-mon[78593]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T05:08:17.864 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T05:08:17.865 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T05:08:17.865 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T05:08:17.865 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T05:08:17.865 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T05:08:17.866 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T05:08:17.866 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T05:08:17.866 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T05:08:17.867 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T05:08:17.867 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T05:08:17.867 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T05:08:17.867 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T05:08:17.868 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T05:08:17.868 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T05:08:17.868 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T05:08:17.868 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T05:08:17.868 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T05:08:17.869 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T05:08:17.869 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T05:08:17.869 INFO:journalctl@ceph.mon.c.smithi115.stdout: During handling of the above exception, another exception occurred: 2022-05-14T05:08:17.869 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T05:08:17.869 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T05:08:17.870 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T05:08:17.870 INFO:journalctl@ceph.mon.c.smithi115.stdout: return f(*arg) 2022-05-14T05:08:17.871 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T05:08:17.871 INFO:journalctl@ceph.mon.c.smithi115.stdout: self._write_client_files(client_files, host) 2022-05-14T05:08:17.871 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T05:08:17.871 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T05:08:17.872 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T05:08:17.872 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T05:08:17.872 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T05:08:17.872 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.event_loop.get_result(coro) 2022-05-14T05:08:17.872 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T05:08:17.873 INFO:journalctl@ceph.mon.c.smithi115.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T05:08:17.873 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T05:08:17.873 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.__get_result() 2022-05-14T05:08:17.873 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T05:08:17.874 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise self._exception 2022-05-14T05:08:17.874 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T05:08:17.874 INFO:journalctl@ceph.mon.c.smithi115.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T05:08:17.874 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T05:08:17.875 INFO:journalctl@ceph.mon.c.smithi115.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T05:08:17.875 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T05:08:17.875 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise 2022-05-14T05:08:17.875 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T05:08:17.875 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.gen.throw(type, value, traceback) 2022-05-14T05:08:17.876 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T05:08:17.876 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OrchestratorError(msg) 2022-05-14T05:08:17.876 INFO:journalctl@ceph.mon.c.smithi115.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T05:08:18.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:08:18 smithi047 ceph-mon[77147]: pgmap v10566: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:08:18.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:08:18 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:08:18.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:08:18 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T05:08:18.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:08:18 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T05:08:18.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:08:18 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 20430 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T05:08:18.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:08:18 smithi115 conmon[90393]: 2022-05-14T05:08:18.479+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:08:18.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:08:18 smithi115 ceph-mon[78593]: pgmap v10566: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:08:18.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:08:18 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:08:18.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:08:18 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T05:08:18.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:08:18 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T05:08:18.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:08:18 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 20430 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T05:08:19.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:08:19 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:08:19.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:08:19 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:08:19.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:08:19 smithi115 conmon[90393]: 2022-05-14T05:08:19.512+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:08:20.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:08:20 smithi047 ceph-mon[77147]: pgmap v10567: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T05:08:20.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:08:20 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:08:20.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:08:20 smithi115 conmon[90393]: 2022-05-14T05:08:20.525+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:08:20.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:08:20 smithi115 ceph-mon[78593]: pgmap v10567: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T05:08:20.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:08:20 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:08:21.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:08:21 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:08:21.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:08:21 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:08:21.860 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:08:21 smithi115 conmon[90393]: 2022-05-14T05:08:21.488+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:08:22.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:08:22 smithi047 ceph-mon[77147]: pgmap v10568: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:08:22.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:08:22 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:08:22.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:08:22 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T05:08:22.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:08:22 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T05:08:22.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:08:22 smithi115 conmon[90393]: 2022-05-14T05:08:22.449+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:08:22.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:08:22 smithi115 ceph-mon[78593]: pgmap v10568: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:08:22.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:08:22 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:08:22.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:08:22 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T05:08:22.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:08:22 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T05:08:23.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:08:23 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:08:23.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:08:23 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 20435 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T05:08:23.860 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:08:23 smithi115 conmon[90393]: 2022-05-14T05:08:23.480+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:08:23.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:08:23 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:08:23.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:08:23 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 20435 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T05:08:24.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:08:24 smithi047 ceph-mon[77147]: pgmap v10569: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:08:24.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:08:24 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:08:24.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:08:24 smithi115 conmon[90393]: 2022-05-14T05:08:24.501+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:08:24.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:08:24 smithi115 ceph-mon[78593]: pgmap v10569: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:08:24.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:08:24 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:08:25.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:08:25 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:08:25.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:08:25 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:08:25.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:08:25 smithi115 conmon[90393]: 2022-05-14T05:08:25.511+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:08:26.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:08:26 smithi047 ceph-mon[77147]: pgmap v10570: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T05:08:26.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:08:26 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:08:26.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:08:26 smithi115 conmon[90393]: 2022-05-14T05:08:26.549+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:08:26.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:08:26 smithi115 ceph-mon[78593]: pgmap v10570: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T05:08:26.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:08:26 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:08:27.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:08:27 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:08:27.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:08:27 smithi115 conmon[90393]: 2022-05-14T05:08:27.531+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:08:27.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:08:27 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:08:28.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:08:28 smithi047 ceph-mon[77147]: pgmap v10571: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:08:28.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:08:28 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:08:28.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:08:28 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 20440 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T05:08:28.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:08:28 smithi115 conmon[90393]: 2022-05-14T05:08:28.526+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:08:28.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:08:28 smithi115 ceph-mon[78593]: pgmap v10571: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:08:28.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:08:28 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:08:28.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:08:28 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 20440 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T05:08:29.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:08:29 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:08:29.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:08:29 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:08:29.860 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:08:29 smithi115 conmon[90393]: 2022-05-14T05:08:29.535+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:08:30.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:08:30 smithi047 ceph-mon[77147]: pgmap v10572: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T05:08:30.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:08:30 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T05:08:30.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:08:30 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T05:08:30.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:08:30 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:08:30.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:08:30 smithi115 conmon[90393]: 2022-05-14T05:08:30.497+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:08:30.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:08:30 smithi115 ceph-mon[78593]: pgmap v10572: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T05:08:30.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:08:30 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T05:08:30.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:08:30 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T05:08:30.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:08:30 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:08:31.795 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:08:31 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:08:31.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:08:31 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:08:31.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:08:31 smithi115 conmon[90393]: 2022-05-14T05:08:31.470+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:08:32.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:08:32 smithi047 ceph-mon[77147]: pgmap v10573: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:08:32.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:08:32 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:08:32.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:08:32 smithi115 conmon[90393]: 2022-05-14T05:08:32.435+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:08:32.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:08:32 smithi115 ceph-mon[78593]: pgmap v10573: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:08:32.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:08:32 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:08:33.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:08:33 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:08:33.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:08:33 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 20445 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T05:08:33.860 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:08:33 smithi115 conmon[90393]: 2022-05-14T05:08:33.472+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:08:33.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:08:33 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:08:33.862 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:08:33 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 20445 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T05:08:34.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:08:34 smithi047 ceph-mon[77147]: pgmap v10574: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:08:34.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:08:34 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:08:34.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:08:34 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T05:08:34.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:08:34 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T05:08:34.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:08:34 smithi115 conmon[90393]: 2022-05-14T05:08:34.498+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:08:34.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:08:34 smithi115 ceph-mon[78593]: pgmap v10574: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:08:34.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:08:34 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:08:34.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:08:34 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T05:08:34.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:08:34 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T05:08:35.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:08:35 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:08:35.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:08:35 smithi115 conmon[90393]: 2022-05-14T05:08:35.490+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:08:35.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:08:35 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:08:36.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:08:36 smithi047 ceph-mon[77147]: pgmap v10575: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T05:08:36.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:08:36 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:08:36.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:08:36 smithi115 conmon[90393]: 2022-05-14T05:08:36.533+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:08:36.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:08:36 smithi115 ceph-mon[78593]: pgmap v10575: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T05:08:36.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:08:36 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:08:37.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:08:37 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:08:37.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:08:37 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:08:37.860 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:08:37 smithi115 conmon[90393]: 2022-05-14T05:08:37.580+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:08:38.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:08:38 smithi047 ceph-mon[77147]: pgmap v10576: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:08:38.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:08:38 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T05:08:38.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:08:38 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T05:08:38.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:08:38 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:08:38.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:08:38 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 20450 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T05:08:38.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:08:38 smithi115 conmon[90393]: 2022-05-14T05:08:38.616+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:08:38.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:08:38 smithi115 ceph-mon[78593]: pgmap v10576: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:08:38.862 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:08:38 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T05:08:38.862 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:08:38 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T05:08:38.862 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:08:38 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:08:38.862 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:08:38 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 20450 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T05:08:39.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:08:39 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:08:39.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:08:39 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:08:39.860 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:08:39 smithi115 conmon[90393]: 2022-05-14T05:08:39.581+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:08:40.861 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:08:40 smithi115 conmon[90393]: 2022-05-14T05:08:40.561+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:08:40.862 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:08:40 smithi115 ceph-mon[78593]: pgmap v10577: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T05:08:40.862 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:08:40 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:08:41.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:08:40 smithi047 ceph-mon[77147]: pgmap v10577: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T05:08:41.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:08:40 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:08:41.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:08:41 smithi115 conmon[90393]: 2022-05-14T05:08:41.599+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:08:41.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:08:41 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:08:42.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:08:41 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:08:42.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:08:42 smithi115 conmon[90393]: 2022-05-14T05:08:42.594+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:08:42.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:08:42 smithi115 ceph-mon[78593]: pgmap v10578: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:08:42.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:08:42 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T05:08:42.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:08:42 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T05:08:42.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:08:42 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:08:43.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:08:42 smithi047 ceph-mon[77147]: pgmap v10578: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:08:43.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:08:42 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T05:08:43.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:08:42 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T05:08:43.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:08:42 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:08:43.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:08:43 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:08:43.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:08:43 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 20455 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T05:08:43.860 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:08:43 smithi115 conmon[90393]: 2022-05-14T05:08:43.581+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:08:44.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:08:43 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:08:44.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:08:43 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 20455 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T05:08:44.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:08:44 smithi115 conmon[90393]: 2022-05-14T05:08:44.599+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:08:44.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:08:44 smithi115 ceph-mon[78593]: pgmap v10579: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:08:44.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:08:44 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:08:45.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:08:44 smithi047 ceph-mon[77147]: pgmap v10579: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:08:45.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:08:44 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:08:45.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:08:45 smithi115 conmon[90393]: 2022-05-14T05:08:45.603+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:08:45.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:08:45 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:08:46.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:08:45 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:08:46.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:08:46 smithi115 conmon[90393]: 2022-05-14T05:08:46.571+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:08:46.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:08:46 smithi115 ceph-mon[78593]: pgmap v10580: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T05:08:46.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:08:46 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:08:47.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:08:46 smithi047 ceph-mon[77147]: pgmap v10580: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T05:08:47.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:08:46 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:08:47.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:08:47 smithi115 conmon[90393]: 2022-05-14T05:08:47.585+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:08:47.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:08:47 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:08:48.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:08:47 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:08:48.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:08:48 smithi115 conmon[90393]: 2022-05-14T05:08:48.556+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:08:48.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:08:48 smithi115 ceph-mon[78593]: pgmap v10581: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:08:48.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:08:48 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:08:48.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:08:48 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 20460 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T05:08:49.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:08:48 smithi047 ceph-mon[77147]: pgmap v10581: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:08:49.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:08:48 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:08:49.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:08:48 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 20460 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T05:08:49.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:08:49 smithi115 conmon[90393]: 2022-05-14T05:08:49.557+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:08:49.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:08:49 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:08:50.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:08:49 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:08:50.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:08:50 smithi115 conmon[90393]: 2022-05-14T05:08:50.561+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:08:50.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:08:50 smithi115 ceph-mon[78593]: pgmap v10582: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T05:08:50.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:08:50 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:08:50.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:08:50 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T05:08:50.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:08:50 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T05:08:51.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:08:50 smithi047 ceph-mon[77147]: pgmap v10582: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T05:08:51.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:08:50 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:08:51.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:08:50 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T05:08:51.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:08:50 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T05:08:51.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:08:51 smithi115 conmon[90393]: 2022-05-14T05:08:51.586+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:08:51.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:08:51 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:08:52.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:08:51 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:08:52.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:08:52 smithi115 conmon[90393]: 2022-05-14T05:08:52.573+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:08:52.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:08:52 smithi115 ceph-mon[78593]: pgmap v10583: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:08:52.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:08:52 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:08:53.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:08:52 smithi047 ceph-mon[77147]: pgmap v10583: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:08:53.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:08:52 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:08:53.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:08:53 smithi115 conmon[90393]: 2022-05-14T05:08:53.561+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:08:53.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:08:53 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:08:53.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:08:53 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 20465 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T05:08:54.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:08:53 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:08:54.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:08:53 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 20465 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T05:08:54.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:08:54 smithi115 conmon[90393]: 2022-05-14T05:08:54.526+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:08:54.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:08:54 smithi115 ceph-mon[78593]: pgmap v10584: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:08:54.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:08:54 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:08:54.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:08:54 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T05:08:54.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:08:54 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T05:08:55.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:08:54 smithi047 ceph-mon[77147]: pgmap v10584: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:08:55.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:08:54 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:08:55.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:08:54 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T05:08:55.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:08:54 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T05:08:55.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:08:55 smithi115 conmon[90393]: 2022-05-14T05:08:55.568+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:08:55.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:08:55 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:08:56.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:08:55 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:08:56.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:08:56 smithi115 conmon[90393]: 2022-05-14T05:08:56.562+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:08:56.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:08:56 smithi115 ceph-mon[78593]: pgmap v10585: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T05:08:56.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:08:56 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:08:57.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:08:56 smithi047 ceph-mon[77147]: pgmap v10585: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T05:08:57.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:08:56 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:08:57.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:08:57 smithi115 conmon[90393]: 2022-05-14T05:08:57.545+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:08:57.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:08:57 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:08:58.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:08:57 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:08:58.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:08:58 smithi115 conmon[90393]: 2022-05-14T05:08:58.523+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:08:58.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:08:58 smithi115 ceph-mon[78593]: pgmap v10586: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:08:58.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:08:58 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:08:58.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:08:58 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T05:08:58.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:08:58 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T05:08:58.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:08:58 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 20470 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T05:08:59.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:08:58 smithi047 ceph-mon[77147]: pgmap v10586: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:08:59.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:08:58 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:08:59.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:08:58 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T05:08:59.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:08:58 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T05:08:59.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:08:58 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 20470 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T05:08:59.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:08:59 smithi115 conmon[90393]: 2022-05-14T05:08:59.520+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:08:59.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:08:59 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:09:00.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:08:59 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:09:00.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:09:00 smithi115 conmon[90393]: 2022-05-14T05:09:00.549+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:09:00.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:09:00 smithi115 ceph-mon[78593]: pgmap v10587: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T05:09:00.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:09:00 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:09:01.050 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:09:00 smithi047 ceph-mon[77147]: pgmap v10587: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T05:09:01.051 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:09:00 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:09:01.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:09:01 smithi115 conmon[90393]: 2022-05-14T05:09:01.522+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:09:01.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:09:01 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:09:02.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:09:01 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:09:02.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:09:02 smithi115 conmon[90393]: 2022-05-14T05:09:02.524+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:09:02.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:09:02 smithi115 ceph-mon[78593]: pgmap v10588: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:09:02.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:09:02 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:09:02.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:09:02 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T05:09:02.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:09:02 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T05:09:03.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:09:02 smithi047 ceph-mon[77147]: pgmap v10588: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:09:03.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:09:02 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:09:03.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:09:02 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T05:09:03.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:09:02 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T05:09:03.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:09:03 smithi115 conmon[90393]: 2022-05-14T05:09:03.535+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:09:03.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:09:03 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:09:03.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:09:03 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 20475 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T05:09:04.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:09:03 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:09:04.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:09:03 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 20475 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T05:09:04.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:09:04 smithi115 conmon[90393]: 2022-05-14T05:09:04.584+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:09:04.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:09:04 smithi115 ceph-mon[78593]: pgmap v10589: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:09:04.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:09:04 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:09:05.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:09:04 smithi047 ceph-mon[77147]: pgmap v10589: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:09:05.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:09:04 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:09:05.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:09:05 smithi115 conmon[90393]: 2022-05-14T05:09:05.593+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:09:05.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:09:05 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:09:06.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:09:05 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:09:06.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:09:06 smithi115 conmon[90393]: 2022-05-14T05:09:06.604+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:09:06.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:09:06 smithi115 ceph-mon[78593]: pgmap v10590: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T05:09:06.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:09:06 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:09:07.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:09:06 smithi047 ceph-mon[77147]: pgmap v10590: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T05:09:07.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:09:06 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:09:08.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:09:07 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:09:08.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:09:07 smithi115 conmon[90393]: 2022-05-14T05:09:07.637+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:09:08.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:09:07 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:09:09.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:09:08 smithi047 ceph-mon[77147]: pgmap v10591: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:09:09.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:09:08 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:09:09.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:09:08 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 20480 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T05:09:09.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:09:08 smithi115 conmon[90393]: 2022-05-14T05:09:08.639+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:09:09.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:09:08 smithi115 ceph-mon[78593]: pgmap v10591: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:09:09.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:09:08 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:09:09.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:09:08 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 20480 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T05:09:10.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:09:09 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:09:10.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:09:09 smithi115 conmon[90393]: 2022-05-14T05:09:09.666+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:09:10.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:09:09 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:09:11.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:09:10 smithi047 ceph-mon[77147]: pgmap v10592: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T05:09:11.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:09:10 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T05:09:11.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:09:10 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T05:09:11.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:09:10 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:09:11.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:09:10 smithi115 conmon[90393]: 2022-05-14T05:09:10.661+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:09:11.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:09:10 smithi115 ceph-mon[78593]: pgmap v10592: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T05:09:11.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:09:10 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T05:09:11.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:09:10 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T05:09:11.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:09:10 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:09:12.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:09:11 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:09:12.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:09:11 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:09:12.110 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:09:11 smithi115 conmon[90393]: 2022-05-14T05:09:11.675+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:09:13.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:09:12 smithi047 ceph-mon[77147]: pgmap v10593: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:09:13.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:09:12 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:09:13.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:09:12 smithi115 ceph-mon[78593]: pgmap v10593: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:09:13.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:09:12 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:09:13.110 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:09:12 smithi115 conmon[90393]: 2022-05-14T05:09:12.679+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:09:14.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:09:13 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:09:14.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:09:13 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 20485 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T05:09:14.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:09:13 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:09:14.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:09:13 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 20485 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T05:09:14.110 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:09:13 smithi115 conmon[90393]: 2022-05-14T05:09:13.683+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:09:15.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:09:14 smithi047 ceph-mon[77147]: pgmap v10594: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:09:15.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:09:14 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:09:15.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:09:14 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T05:09:15.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:09:14 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T05:09:15.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:09:14 smithi115 conmon[90393]: 2022-05-14T05:09:14.638+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:09:15.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:09:14 smithi115 ceph-mon[78593]: pgmap v10594: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:09:15.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:09:14 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:09:15.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:09:14 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T05:09:15.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:09:14 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T05:09:16.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:09:15 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:09:16.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:09:15 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T05:09:16.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:09:15 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T05:09:16.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:09:15 smithi115 conmon[90393]: 2022-05-14T05:09:15.643+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:09:16.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:09:15 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:09:16.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:09:15 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T05:09:16.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:09:15 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T05:09:16.860 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:09:16 smithi115 conmon[90393]: 2022-05-14T05:09:16.676+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:09:16.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:09:16 smithi115 ceph-mon[78593]: pgmap v10595: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T05:09:16.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:09:16 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:09:16.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:09:16 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T05:09:16.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:09:16 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T05:09:16.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:09:16 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T05:09:17.019 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:09:16 smithi047 ceph-mon[77147]: pgmap v10595: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T05:09:17.019 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:09:16 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:09:17.020 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:09:16 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T05:09:17.020 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:09:16 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T05:09:17.020 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:09:16 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T05:09:18.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:09:17 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:09:18.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:09:17 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:09:18.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:09:17 smithi115 conmon[90393]: 2022-05-14T05:09:17.654+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:09:19.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:09:18 smithi047 ceph-mon[77147]: pgmap v10596: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:09:19.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:09:18 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T05:09:19.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:09:18 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T05:09:19.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:09:18 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:09:19.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:09:18 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 20490 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T05:09:19.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:09:18 smithi115 conmon[90393]: 2022-05-14T05:09:18.639+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:09:19.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:09:18 smithi115 ceph-mon[78593]: pgmap v10596: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:09:19.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:09:18 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T05:09:19.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:09:18 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T05:09:19.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:09:18 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:09:19.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:09:18 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 20490 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T05:09:20.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:09:19 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:09:20.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:09:19 smithi047 conmon[78521]: 2022-05-14T05:09:19.600+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T05:09:20.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:09:19 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T05:09:20.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:09:19 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T05:09:20.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:09:19 smithi047 conmon[78521]: yield 2022-05-14T05:09:20.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:09:19 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T05:09:20.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:09:19 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T05:09:20.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:09:19 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T05:09:20.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:09:19 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T05:09:20.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:09:19 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T05:09:20.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:09:19 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T05:09:20.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:09:19 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T05:09:20.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:09:19 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T05:09:20.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:09:19 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T05:09:20.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:09:19 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T05:09:20.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:09:19 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T05:09:20.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:09:19 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T05:09:20.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:09:19 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T05:09:20.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:09:19 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T05:09:20.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:09:19 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T05:09:20.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:09:19 smithi115 conmon[90393]: 2022-05-14T05:09:19.677+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:09:20.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:09:19 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:09:21.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:09:20 smithi047 ceph-mon[77147]: pgmap v10597: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T05:09:21.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:09:20 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T05:09:21.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T05:09:21.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T05:09:21.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T05:09:21.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T05:09:21.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T05:09:21.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T05:09:21.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T05:09:21.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T05:09:21.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T05:09:21.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T05:09:21.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T05:09:21.049 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T05:09:21.049 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T05:09:21.049 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T05:09:21.049 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T05:09:21.050 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T05:09:21.050 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T05:09:21.050 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T05:09:21.050 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:09:20 smithi047 ceph-mon[77147]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T05:09:21.051 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:09:20 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:09:21.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:09:20 smithi115 conmon[90393]: 2022-05-14T05:09:20.707+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:09:21.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:09:20 smithi115 ceph-mon[78593]: pgmap v10597: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T05:09:21.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:09:20 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T05:09:21.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T05:09:21.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T05:09:21.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T05:09:21.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T05:09:21.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T05:09:21.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T05:09:21.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T05:09:21.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T05:09:21.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T05:09:21.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T05:09:21.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T05:09:21.113 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T05:09:21.113 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T05:09:21.114 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T05:09:21.114 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T05:09:21.114 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T05:09:21.114 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T05:09:21.115 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T05:09:21.115 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:09:20 smithi115 ceph-mon[78593]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T05:09:21.115 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:09:20 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:09:22.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:09:21 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:09:22.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:09:21 smithi115 conmon[90393]: 2022-05-14T05:09:21.710+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:09:22.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:09:21 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:09:23.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:09:22 smithi047 ceph-mon[77147]: pgmap v10598: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:09:23.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:09:22 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T05:09:23.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:09:22 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T05:09:23.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:09:22 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:09:23.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:09:22 smithi047 conmon[78521]: 2022-05-14T05:09:22.673+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T05:09:23.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:09:22 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T05:09:23.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:09:22 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T05:09:23.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:09:22 smithi047 conmon[78521]: yield 2022-05-14T05:09:23.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:09:22 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T05:09:23.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:09:22 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T05:09:23.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:09:22 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T05:09:23.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:09:22 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T05:09:23.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:09:22 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T05:09:23.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:09:22 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T05:09:23.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:09:22 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T05:09:23.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:09:22 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T05:09:23.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:09:22 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T05:09:23.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:09:22 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T05:09:23.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:09:22 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T05:09:23.048 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:09:22 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T05:09:23.048 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:09:22 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T05:09:23.048 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:09:22 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T05:09:23.048 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:09:22 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T05:09:23.049 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:09:22 smithi047 conmon[78521]: 2022-05-14T05:09:22.674+0000 7f533ff78700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T05:09:23.049 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:09:22 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T05:09:23.049 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:09:22 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T05:09:23.049 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:09:22 smithi047 conmon[78521]: yield 2022-05-14T05:09:23.050 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:09:22 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T05:09:23.050 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:09:22 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T05:09:23.050 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:09:22 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T05:09:23.050 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:09:22 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T05:09:23.051 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:09:22 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T05:09:23.051 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:09:22 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T05:09:23.051 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:09:22 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T05:09:23.051 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:09:22 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T05:09:23.052 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:09:22 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T05:09:23.052 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:09:22 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T05:09:23.052 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:09:22 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T05:09:23.053 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:09:22 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T05:09:23.053 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:09:22 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T05:09:23.053 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:09:22 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T05:09:23.054 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:09:22 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T05:09:23.054 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:09:22 smithi047 conmon[78521]: 2022-05-14T05:09:23.054 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:09:22 smithi047 conmon[78521]: During handling of the above exception, another exception occurred: 2022-05-14T05:09:23.054 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:09:22 smithi047 conmon[78521]: 2022-05-14T05:09:23.055 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:09:22 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T05:09:23.055 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:09:22 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T05:09:23.055 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:09:22 smithi047 conmon[78521]: return f(*arg) 2022-05-14T05:09:23.055 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:09:22 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T05:09:23.056 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:09:22 smithi047 conmon[78521]: self._write_client_files(client_files, host) 2022-05-14T05:09:23.056 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:09:22 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T05:09:23.056 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:09:22 smithi047 conmon[78521]: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T05:09:23.056 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:09:22 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T05:09:23.056 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:09:22 smithi047 conmon[78521]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T05:09:23.057 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:09:22 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T05:09:23.057 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:09:22 smithi047 conmon[78521]: return self.event_loop.get_result(coro) 2022-05-14T05:09:23.057 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:09:22 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T05:09:23.057 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:09:22 smithi047 conmon[78521]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T05:09:23.058 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:09:22 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T05:09:23.058 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:09:22 smithi047 conmon[78521]: return self.__get_result() 2022-05-14T05:09:23.058 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:09:22 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T05:09:23.059 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:09:22 smithi047 conmon[78521]: raise self._exception 2022-05-14T05:09:23.059 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:09:22 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T05:09:23.059 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:09:22 smithi047 conmon[78521]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T05:09:23.059 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:09:22 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T05:09:23.060 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:09:22 smithi047 conmon[78521]: conn = await self._remote_connection(host, addr) 2022-05-14T05:09:23.060 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:09:22 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T05:09:23.060 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:09:22 smithi047 conmon[78521]: raise 2022-05-14T05:09:23.060 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:09:22 smithi047 conmon[78521]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T05:09:23.060 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:09:22 smithi047 conmon[78521]: self.gen.throw(type, value, traceback) 2022-05-14T05:09:23.061 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:09:22 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T05:09:23.061 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:09:22 smithi047 conmon[78521]: raise OrchestratorError(msg) 2022-05-14T05:09:23.061 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:09:22 smithi047 conmon[78521]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T05:09:23.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:09:22 smithi115 conmon[90393]: 2022-05-14T05:09:22.678+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:09:23.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:09:22 smithi115 ceph-mon[78593]: pgmap v10598: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:09:23.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:09:22 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T05:09:23.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:09:22 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T05:09:23.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:09:22 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:09:24.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:09:23 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T05:09:24.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T05:09:24.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T05:09:24.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T05:09:24.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T05:09:24.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T05:09:24.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T05:09:24.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T05:09:24.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T05:09:24.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T05:09:24.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T05:09:24.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T05:09:24.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T05:09:24.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T05:09:24.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T05:09:24.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T05:09:24.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T05:09:24.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T05:09:24.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T05:09:24.047 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:09:23 smithi047 ceph-mon[77147]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T05:09:24.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T05:09:24.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T05:09:24.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T05:09:24.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T05:09:24.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T05:09:24.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T05:09:24.049 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T05:09:24.049 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T05:09:24.049 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T05:09:24.049 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T05:09:24.050 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T05:09:24.050 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T05:09:24.050 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T05:09:24.050 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T05:09:24.050 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T05:09:24.051 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T05:09:24.051 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T05:09:24.051 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T05:09:24.051 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T05:09:24.051 INFO:journalctl@ceph.mon.b.smithi047.stdout: During handling of the above exception, another exception occurred: 2022-05-14T05:09:24.052 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T05:09:24.052 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T05:09:24.052 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T05:09:24.052 INFO:journalctl@ceph.mon.b.smithi047.stdout: return f(*arg) 2022-05-14T05:09:24.052 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T05:09:24.053 INFO:journalctl@ceph.mon.b.smithi047.stdout: self._write_client_files(client_files, host) 2022-05-14T05:09:24.053 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T05:09:24.053 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T05:09:24.053 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T05:09:24.054 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T05:09:24.054 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T05:09:24.054 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.event_loop.get_result(coro) 2022-05-14T05:09:24.054 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T05:09:24.054 INFO:journalctl@ceph.mon.b.smithi047.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T05:09:24.055 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T05:09:24.055 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.__get_result() 2022-05-14T05:09:24.055 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T05:09:24.055 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise self._exception 2022-05-14T05:09:24.055 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T05:09:24.056 INFO:journalctl@ceph.mon.b.smithi047.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T05:09:24.056 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T05:09:24.056 INFO:journalctl@ceph.mon.b.smithi047.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T05:09:24.056 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T05:09:24.057 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise 2022-05-14T05:09:24.057 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T05:09:24.057 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.gen.throw(type, value, traceback) 2022-05-14T05:09:24.057 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T05:09:24.057 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OrchestratorError(msg) 2022-05-14T05:09:24.058 INFO:journalctl@ceph.mon.b.smithi047.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T05:09:24.058 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:09:23 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:09:24.058 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:09:23 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 20495 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T05:09:24.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:09:23 smithi115 conmon[90393]: 2022-05-14T05:09:23.666+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:09:24.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:09:23 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T05:09:24.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T05:09:24.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T05:09:24.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T05:09:24.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T05:09:24.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T05:09:24.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T05:09:24.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T05:09:24.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T05:09:24.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T05:09:24.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T05:09:24.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T05:09:24.113 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T05:09:24.113 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T05:09:24.113 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T05:09:24.113 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T05:09:24.114 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T05:09:24.114 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T05:09:24.114 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T05:09:24.114 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:09:23 smithi115 ceph-mon[78593]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T05:09:24.114 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T05:09:24.115 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T05:09:24.115 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T05:09:24.115 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T05:09:24.115 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T05:09:24.115 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T05:09:24.116 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T05:09:24.116 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T05:09:24.117 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T05:09:24.118 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T05:09:24.118 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T05:09:24.118 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T05:09:24.118 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T05:09:24.119 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T05:09:24.119 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T05:09:24.119 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T05:09:24.119 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T05:09:24.120 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T05:09:24.120 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T05:09:24.120 INFO:journalctl@ceph.mon.c.smithi115.stdout: During handling of the above exception, another exception occurred: 2022-05-14T05:09:24.120 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T05:09:24.120 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T05:09:24.121 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T05:09:24.121 INFO:journalctl@ceph.mon.c.smithi115.stdout: return f(*arg) 2022-05-14T05:09:24.121 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T05:09:24.122 INFO:journalctl@ceph.mon.c.smithi115.stdout: self._write_client_files(client_files, host) 2022-05-14T05:09:24.122 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T05:09:24.122 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T05:09:24.122 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T05:09:24.123 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T05:09:24.123 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T05:09:24.123 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.event_loop.get_result(coro) 2022-05-14T05:09:24.123 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T05:09:24.124 INFO:journalctl@ceph.mon.c.smithi115.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T05:09:24.124 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T05:09:24.125 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.__get_result() 2022-05-14T05:09:24.125 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T05:09:24.125 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise self._exception 2022-05-14T05:09:24.125 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T05:09:24.126 INFO:journalctl@ceph.mon.c.smithi115.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T05:09:24.126 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T05:09:24.126 INFO:journalctl@ceph.mon.c.smithi115.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T05:09:24.126 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T05:09:24.126 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise 2022-05-14T05:09:24.127 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T05:09:24.127 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.gen.throw(type, value, traceback) 2022-05-14T05:09:24.127 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T05:09:24.127 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OrchestratorError(msg) 2022-05-14T05:09:24.127 INFO:journalctl@ceph.mon.c.smithi115.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T05:09:24.128 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:09:23 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:09:24.128 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:09:23 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 20495 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T05:09:25.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:09:24 smithi047 ceph-mon[77147]: pgmap v10599: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:09:25.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:09:24 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:09:25.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:09:24 smithi115 ceph-mon[78593]: pgmap v10599: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:09:25.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:09:24 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:09:25.112 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:09:24 smithi115 conmon[90393]: 2022-05-14T05:09:24.659+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:09:26.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:09:25 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:09:26.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:09:25 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:09:26.110 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:09:25 smithi115 conmon[90393]: 2022-05-14T05:09:25.619+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:09:27.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:09:26 smithi047 ceph-mon[77147]: pgmap v10600: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T05:09:27.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:09:26 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:09:27.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:09:26 smithi115 ceph-mon[78593]: pgmap v10600: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T05:09:27.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:09:26 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:09:27.110 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:09:26 smithi115 conmon[90393]: 2022-05-14T05:09:26.666+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:09:28.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:09:27 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:09:28.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:09:27 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:09:28.111 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:09:27 smithi115 conmon[90393]: 2022-05-14T05:09:27.623+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:09:28.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:09:28 smithi115 conmon[90393]: 2022-05-14T05:09:28.597+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:09:28.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:09:28 smithi115 ceph-mon[78593]: pgmap v10601: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:09:28.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:09:28 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:09:28.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:09:28 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 20500 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T05:09:29.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:09:28 smithi047 ceph-mon[77147]: pgmap v10601: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:09:29.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:09:28 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:09:29.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:09:28 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 20500 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T05:09:29.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:09:29 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:09:29.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:09:29 smithi115 conmon[90393]: 2022-05-14T05:09:29.585+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:09:30.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:09:29 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:09:30.862 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:09:30 smithi115 conmon[90393]: 2022-05-14T05:09:30.588+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:09:30.863 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:09:30 smithi115 ceph-mon[78593]: pgmap v10602: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T05:09:30.863 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:09:30 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:09:30.863 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:09:30 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T05:09:30.864 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:09:30 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T05:09:31.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:09:30 smithi047 ceph-mon[77147]: pgmap v10602: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T05:09:31.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:09:30 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:09:31.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:09:30 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T05:09:31.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:09:30 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T05:09:32.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:09:31 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:09:32.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:09:31 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:09:32.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:09:31 smithi115 conmon[90393]: 2022-05-14T05:09:31.633+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:09:33.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:09:32 smithi047 ceph-mon[77147]: pgmap v10603: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:09:33.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:09:32 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:09:33.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:09:32 smithi115 ceph-mon[78593]: pgmap v10603: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:09:33.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:09:32 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:09:33.110 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:09:32 smithi115 conmon[90393]: 2022-05-14T05:09:32.637+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:09:33.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:09:33 smithi115 conmon[90393]: 2022-05-14T05:09:33.600+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:09:33.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:09:33 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:09:33.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:09:33 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 20505 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T05:09:34.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:09:33 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:09:34.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:09:33 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 20505 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T05:09:34.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:09:34 smithi115 conmon[90393]: 2022-05-14T05:09:34.580+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:09:34.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:09:34 smithi115 ceph-mon[78593]: pgmap v10604: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:09:34.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:09:34 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:09:34.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:09:34 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T05:09:34.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:09:34 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T05:09:35.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:09:34 smithi047 ceph-mon[77147]: pgmap v10604: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:09:35.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:09:34 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:09:35.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:09:34 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T05:09:35.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:09:34 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T05:09:35.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:09:35 smithi115 conmon[90393]: 2022-05-14T05:09:35.574+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:09:35.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:09:35 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:09:36.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:09:35 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:09:36.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:09:36 smithi115 conmon[90393]: 2022-05-14T05:09:36.591+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:09:36.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:09:36 smithi115 ceph-mon[78593]: pgmap v10605: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T05:09:36.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:09:36 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:09:37.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:09:36 smithi047 ceph-mon[77147]: pgmap v10605: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T05:09:37.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:09:36 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:09:37.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:09:37 smithi115 conmon[90393]: 2022-05-14T05:09:37.566+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:09:37.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:09:37 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:09:38.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:09:37 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:09:38.861 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:09:38 smithi115 conmon[90393]: 2022-05-14T05:09:38.521+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:09:38.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:09:38 smithi115 ceph-mon[78593]: pgmap v10606: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:09:38.862 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:09:38 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:09:38.862 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:09:38 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T05:09:38.862 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:09:38 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T05:09:38.862 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:09:38 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 20510 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T05:09:39.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:09:38 smithi047 ceph-mon[77147]: pgmap v10606: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:09:39.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:09:38 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:09:39.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:09:38 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T05:09:39.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:09:38 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T05:09:39.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:09:38 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 20510 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T05:09:39.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:09:39 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:09:39.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:09:39 smithi115 conmon[90393]: 2022-05-14T05:09:39.483+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:09:40.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:09:39 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:09:40.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:09:40 smithi115 conmon[90393]: 2022-05-14T05:09:40.476+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:09:40.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:09:40 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:09:40.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:09:40 smithi115 ceph-mon[78593]: pgmap v10607: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T05:09:41.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:09:40 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:09:41.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:09:40 smithi047 ceph-mon[77147]: pgmap v10607: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T05:09:41.731 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:09:41 smithi115 conmon[90393]: 2022-05-14T05:09:41.445+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:09:42.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:09:41 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:09:42.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:09:41 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:09:42.730 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:09:42 smithi115 conmon[90393]: 2022-05-14T05:09:42.429+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:09:43.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:09:42 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:09:43.046 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:09:42 smithi047 ceph-mon[77147]: pgmap v10608: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:09:43.046 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:09:42 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T05:09:43.046 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:09:42 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T05:09:43.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:09:42 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:09:43.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:09:42 smithi115 ceph-mon[78593]: pgmap v10608: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:09:43.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:09:42 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T05:09:43.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:09:42 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T05:09:43.732 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:09:43 smithi115 conmon[90393]: 2022-05-14T05:09:43.461+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:09:44.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:09:43 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:09:44.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:09:43 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 20515 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T05:09:44.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:09:43 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:09:44.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:09:43 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 20515 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T05:09:44.733 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:09:44 smithi115 conmon[90393]: 2022-05-14T05:09:44.475+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:09:45.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:09:44 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:09:45.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:09:44 smithi047 ceph-mon[77147]: pgmap v10609: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:09:45.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:09:44 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:09:45.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:09:44 smithi115 ceph-mon[78593]: pgmap v10609: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:09:45.862 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:09:45 smithi115 conmon[90393]: 2022-05-14T05:09:45.525+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:09:45.863 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:09:45 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:09:46.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:09:45 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:09:46.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:09:46 smithi115 conmon[90393]: 2022-05-14T05:09:46.490+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:09:46.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:09:46 smithi115 ceph-mon[78593]: pgmap v10610: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T05:09:46.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:09:46 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:09:47.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:09:46 smithi047 ceph-mon[77147]: pgmap v10610: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T05:09:47.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:09:46 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:09:47.746 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:09:47 smithi115 conmon[90393]: 2022-05-14T05:09:47.476+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:09:48.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:09:47 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:09:48.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:09:47 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:09:48.746 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:09:48 smithi115 conmon[90393]: 2022-05-14T05:09:48.426+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:09:49.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:09:48 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:09:49.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:09:48 smithi047 ceph-mon[77147]: pgmap v10611: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:09:49.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:09:48 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 20520 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T05:09:49.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:09:48 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:09:49.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:09:48 smithi115 ceph-mon[78593]: pgmap v10611: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:09:49.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:09:48 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 20520 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T05:09:49.751 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:09:49 smithi115 conmon[90393]: 2022-05-14T05:09:49.460+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:09:50.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:09:49 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:09:50.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:09:49 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:09:50.752 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:09:50 smithi115 conmon[90393]: 2022-05-14T05:09:50.432+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:09:51.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:09:50 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:09:51.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:09:50 smithi047 ceph-mon[77147]: pgmap v10612: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T05:09:51.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:09:50 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T05:09:51.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:09:50 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T05:09:51.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:09:50 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:09:51.112 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:09:50 smithi115 ceph-mon[78593]: pgmap v10612: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T05:09:51.112 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:09:50 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T05:09:51.112 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:09:50 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T05:09:51.759 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:09:51 smithi115 conmon[90393]: 2022-05-14T05:09:51.409+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:09:52.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:09:51 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:09:52.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:09:51 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:09:52.759 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:09:52 smithi115 conmon[90393]: 2022-05-14T05:09:52.434+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:09:53.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:09:52 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:09:53.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:09:52 smithi047 ceph-mon[77147]: pgmap v10613: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:09:53.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:09:52 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:09:53.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:09:52 smithi115 ceph-mon[78593]: pgmap v10613: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:09:53.760 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:09:53 smithi115 conmon[90393]: 2022-05-14T05:09:53.479+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:09:54.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:09:53 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:09:54.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:09:53 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 20525 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T05:09:54.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:09:53 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:09:54.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:09:53 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 20525 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T05:09:54.762 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:09:54 smithi115 conmon[90393]: 2022-05-14T05:09:54.449+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:09:55.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:09:54 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:09:55.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:09:54 smithi047 ceph-mon[77147]: pgmap v10614: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:09:55.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:09:54 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T05:09:55.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:09:54 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T05:09:55.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:09:54 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:09:55.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:09:54 smithi115 ceph-mon[78593]: pgmap v10614: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:09:55.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:09:54 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T05:09:55.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:09:54 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T05:09:55.771 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:09:55 smithi115 conmon[90393]: 2022-05-14T05:09:55.478+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:09:56.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:09:55 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:09:56.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:09:55 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:09:56.773 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:09:56 smithi115 conmon[90393]: 2022-05-14T05:09:56.456+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:09:57.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:09:56 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:09:57.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:09:56 smithi047 ceph-mon[77147]: pgmap v10615: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T05:09:57.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:09:56 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:09:57.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:09:56 smithi115 ceph-mon[78593]: pgmap v10615: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T05:09:57.778 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:09:57 smithi115 conmon[90393]: 2022-05-14T05:09:57.437+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:09:58.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:09:57 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:09:58.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:09:57 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:09:58.780 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:09:58 smithi115 conmon[90393]: 2022-05-14T05:09:58.456+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:09:59.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:09:58 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:09:59.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:09:58 smithi047 ceph-mon[77147]: pgmap v10616: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:09:59.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:09:58 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T05:09:59.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:09:58 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T05:09:59.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:09:58 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 20530 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T05:09:59.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:09:58 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:09:59.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:09:58 smithi115 ceph-mon[78593]: pgmap v10616: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:09:59.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:09:58 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T05:09:59.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:09:58 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T05:09:59.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:09:58 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 20530 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T05:09:59.784 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:09:59 smithi115 conmon[90393]: 2022-05-14T05:09:59.489+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:10:00.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:09:59 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:10:00.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:10:00 smithi047 conmon[77123]: 2022-05-14T05:09:59.998+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : Health detail: HEALTH_ERR 1 filesystem is degraded; 1 mds daemon damaged; 1 MDSs are read only; 1 MDSs report slow metadata IOs; 1/3 mons down, quorum b,c; Reduced data availability: 1 pg inactive; 9 slow ops, oldest one blocked for 20535 sec, osd.5 has slow ops 2022-05-14T05:10:00.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:10:00 smithi047 conmon[77123]: 2022-05-14T05:09:59.998+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : [WRN] FS_DEGRADED: 1 filesystem is degraded 2022-05-14T05:10:00.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:10:00 smithi047 conmon[77123]: 2022-05-14T05:09:59.998+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : fs cephfs is degraded 2022-05-14T05:10:00.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:10:00 smithi047 conmon[77123]: 2022-05-14T05:09:59.998+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : [ERR] MDS_DAMAGE: 1 mds daemon damaged 2022-05-14T05:10:00.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:10:00 smithi047 conmon[77123]: 2022-05-14T05:10:00.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:10:00 smithi047 conmon[77123]: 2022-05-14T05:09:59.998+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : fs cephfs mds.2 is damaged 2022-05-14T05:10:00.045 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:10:00 smithi047 conmon[77123]: 2022-05-14T05:09:59.998+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : [WRN] MDS_READ_ONLY: 1 MDSs are read only 2022-05-14T05:10:00.045 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:10:00 smithi047 conmon[77123]: 2022-05-14T05:09:59.998+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : mds.l(mds.0): MDS in read-only mode 2022-05-14T05:10:00.045 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:10:00 smithi047 conmon[77123]: 2022-05-14T05:09:59.998+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : [WRN] MDS_SLOW_METADATA_IO: 1 MDSs report slow metadata IOs 2022-05-14T05:10:00.045 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:10:00 smithi047 conmon[77123]: 2022-05-14T05:09:59.998+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : mds.l(mds.0): 4 slow metadata IOs are blocked > 180 secs, oldest blocked for 20834 secs 2022-05-14T05:10:00.046 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:10:00 smithi047 conmon[77123]: 2022-05-14T05:09:59.998+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : [WRN] MON_DOWN: 1/3 mons down, quorum b,c 2022-05-14T05:10:00.046 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:10:00 smithi047 conmon[77123]: 2022-05-14T05:09:59.999+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : mon.a (rank 0) addr [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] is down (out of quorum) 2022-05-14T05:10:00.046 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:10:00 smithi047 conmon[77123]: 2022-05-14T05:09:59.999+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : [WRN] PG_AVAILABILITY: Reduced data availability: 1 pg inactive 2022-05-14T05:10:00.046 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:10:00 smithi047 conmon[77123]: 2022-05-14T05:09:59.999+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : pg 2.1 is stuck inactive for 5h, current state unknown, last acting [] 2022-05-14T05:10:00.046 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:10:00 smithi047 conmon[77123]: 2022-05-14T05:09:59.999+0000 7f8b246a0700 -1 log_channel(cluster) log [ERR] : [WRN] SLOW_OPS: 9 slow ops, oldest one blocked for 20535 sec, osd.5 has slow ops 2022-05-14T05:10:00.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:09:59 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:10:00.786 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:10:00 smithi115 conmon[90393]: 2022-05-14T05:10:00.530+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:10:01.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:10:00 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:10:01.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:10:00 smithi047 ceph-mon[77147]: pgmap v10617: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T05:10:01.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:10:00 smithi047 ceph-mon[77147]: Health detail: HEALTH_ERR 1 filesystem is degraded; 1 mds daemon damaged; 1 MDSs are read only; 1 MDSs report slow metadata IOs; 1/3 mons down, quorum b,c; Reduced data availability: 1 pg inactive; 9 slow ops, oldest one blocked for 20535 sec, osd.5 has slow ops 2022-05-14T05:10:01.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:10:00 smithi047 ceph-mon[77147]: [WRN] FS_DEGRADED: 1 filesystem is degraded 2022-05-14T05:10:01.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:10:00 smithi047 ceph-mon[77147]: fs cephfs is degraded 2022-05-14T05:10:01.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:10:00 smithi047 ceph-mon[77147]: [ERR] MDS_DAMAGE: 1 mds daemon damaged 2022-05-14T05:10:01.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:10:00 smithi047 ceph-mon[77147]: fs cephfs mds.2 is damaged 2022-05-14T05:10:01.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:10:00 smithi047 ceph-mon[77147]: [WRN] MDS_READ_ONLY: 1 MDSs are read only 2022-05-14T05:10:01.045 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:10:00 smithi047 ceph-mon[77147]: mds.l(mds.0): MDS in read-only mode 2022-05-14T05:10:01.045 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:10:00 smithi047 ceph-mon[77147]: [WRN] MDS_SLOW_METADATA_IO: 1 MDSs report slow metadata IOs 2022-05-14T05:10:01.045 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:10:00 smithi047 ceph-mon[77147]: mds.l(mds.0): 4 slow metadata IOs are blocked > 180 secs, oldest blocked for 20834 secs 2022-05-14T05:10:01.045 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:10:00 smithi047 ceph-mon[77147]: [WRN] MON_DOWN: 1/3 mons down, quorum b,c 2022-05-14T05:10:01.046 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:10:00 smithi047 ceph-mon[77147]: mon.a (rank 0) addr [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] is down (out of quorum) 2022-05-14T05:10:01.046 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:10:00 smithi047 ceph-mon[77147]: [WRN] PG_AVAILABILITY: Reduced data availability: 1 pg inactive 2022-05-14T05:10:01.046 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:10:00 smithi047 ceph-mon[77147]: pg 2.1 is stuck inactive for 5h, current state unknown, last acting [] 2022-05-14T05:10:01.046 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:10:00 smithi047 ceph-mon[77147]: [WRN] SLOW_OPS: 9 slow ops, oldest one blocked for 20535 sec, osd.5 has slow ops 2022-05-14T05:10:01.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:10:00 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:10:01.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:10:00 smithi115 ceph-mon[78593]: pgmap v10617: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T05:10:01.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:10:00 smithi115 ceph-mon[78593]: Health detail: HEALTH_ERR 1 filesystem is degraded; 1 mds daemon damaged; 1 MDSs are read only; 1 MDSs report slow metadata IOs; 1/3 mons down, quorum b,c; Reduced data availability: 1 pg inactive; 9 slow ops, oldest one blocked for 20535 sec, osd.5 has slow ops 2022-05-14T05:10:01.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:10:00 smithi115 ceph-mon[78593]: [WRN] FS_DEGRADED: 1 filesystem is degraded 2022-05-14T05:10:01.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:10:00 smithi115 ceph-mon[78593]: fs cephfs is degraded 2022-05-14T05:10:01.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:10:00 smithi115 ceph-mon[78593]: [ERR] MDS_DAMAGE: 1 mds daemon damaged 2022-05-14T05:10:01.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:10:00 smithi115 ceph-mon[78593]: fs cephfs mds.2 is damaged 2022-05-14T05:10:01.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:10:00 smithi115 ceph-mon[78593]: [WRN] MDS_READ_ONLY: 1 MDSs are read only 2022-05-14T05:10:01.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:10:00 smithi115 ceph-mon[78593]: mds.l(mds.0): MDS in read-only mode 2022-05-14T05:10:01.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:10:00 smithi115 ceph-mon[78593]: [WRN] MDS_SLOW_METADATA_IO: 1 MDSs report slow metadata IOs 2022-05-14T05:10:01.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:10:00 smithi115 ceph-mon[78593]: mds.l(mds.0): 4 slow metadata IOs are blocked > 180 secs, oldest blocked for 20834 secs 2022-05-14T05:10:01.117 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:10:00 smithi115 ceph-mon[78593]: [WRN] MON_DOWN: 1/3 mons down, quorum b,c 2022-05-14T05:10:01.118 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:10:00 smithi115 ceph-mon[78593]: mon.a (rank 0) addr [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] is down (out of quorum) 2022-05-14T05:10:01.118 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:10:00 smithi115 ceph-mon[78593]: [WRN] PG_AVAILABILITY: Reduced data availability: 1 pg inactive 2022-05-14T05:10:01.118 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:10:00 smithi115 ceph-mon[78593]: pg 2.1 is stuck inactive for 5h, current state unknown, last acting [] 2022-05-14T05:10:01.118 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:10:00 smithi115 ceph-mon[78593]: [WRN] SLOW_OPS: 9 slow ops, oldest one blocked for 20535 sec, osd.5 has slow ops 2022-05-14T05:10:01.791 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:10:01 smithi115 conmon[90393]: 2022-05-14T05:10:01.481+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:10:02.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:10:01 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:10:02.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:10:01 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:10:02.793 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:10:02 smithi115 conmon[90393]: 2022-05-14T05:10:02.515+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:10:03.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:10:02 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:10:03.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:10:02 smithi115 ceph-mon[78593]: pgmap v10618: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:10:03.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:10:02 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T05:10:03.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:10:02 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T05:10:03.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:10:02 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:10:03.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:10:02 smithi047 ceph-mon[77147]: pgmap v10618: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:10:03.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:10:02 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T05:10:03.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:10:02 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T05:10:03.794 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:10:03 smithi115 conmon[90393]: 2022-05-14T05:10:03.482+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:10:04.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:10:03 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:10:04.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:10:03 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 20535 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T05:10:04.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:10:03 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:10:04.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:10:03 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 20535 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T05:10:04.796 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:10:04 smithi115 conmon[90393]: 2022-05-14T05:10:04.457+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:10:05.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:10:04 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:10:05.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:10:04 smithi115 ceph-mon[78593]: pgmap v10619: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:10:05.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:10:04 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:10:05.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:10:04 smithi047 ceph-mon[77147]: pgmap v10619: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:10:05.801 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:10:05 smithi115 conmon[90393]: 2022-05-14T05:10:05.486+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:10:06.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:10:05 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:10:06.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:10:05 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:10:06.802 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:10:06 smithi115 conmon[90393]: 2022-05-14T05:10:06.470+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:10:07.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:10:06 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:10:07.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:10:06 smithi115 ceph-mon[78593]: pgmap v10620: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T05:10:07.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:10:06 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:10:07.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:10:06 smithi047 ceph-mon[77147]: pgmap v10620: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T05:10:07.809 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:10:07 smithi115 conmon[90393]: 2022-05-14T05:10:07.431+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:10:08.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:10:07 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:10:08.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:10:07 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:10:08.809 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:10:08 smithi115 conmon[90393]: 2022-05-14T05:10:08.440+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:10:09.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:10:08 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:10:09.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:10:08 smithi047 ceph-mon[77147]: pgmap v10621: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:10:09.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:10:08 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 20540 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T05:10:09.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:10:08 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:10:09.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:10:08 smithi115 ceph-mon[78593]: pgmap v10621: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:10:09.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:10:08 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 20540 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T05:10:09.811 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:10:09 smithi115 conmon[90393]: 2022-05-14T05:10:09.456+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:10:10.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:10:09 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:10:10.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:10:09 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:10:10.815 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:10:10 smithi115 conmon[90393]: 2022-05-14T05:10:10.503+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:10:11.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:10:10 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:10:11.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:10:10 smithi115 ceph-mon[78593]: pgmap v10622: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T05:10:11.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:10:10 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T05:10:11.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:10:10 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T05:10:11.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:10:10 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:10:11.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:10:10 smithi047 ceph-mon[77147]: pgmap v10622: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T05:10:11.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:10:10 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T05:10:11.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:10:10 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T05:10:11.823 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:10:11 smithi115 conmon[90393]: 2022-05-14T05:10:11.505+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:10:12.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:10:11 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:10:12.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:10:11 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:10:12.822 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:10:12 smithi115 conmon[90393]: 2022-05-14T05:10:12.476+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:10:13.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:10:12 smithi115 ceph-mon[78593]: pgmap v10623: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:10:13.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:10:12 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:10:13.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:10:12 smithi047 ceph-mon[77147]: pgmap v10623: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:10:13.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:10:12 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:10:13.824 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:10:13 smithi115 conmon[90393]: 2022-05-14T05:10:13.502+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:10:14.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:10:13 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:10:14.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:10:13 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 20545 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T05:10:14.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:10:13 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:10:14.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:10:13 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 20545 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T05:10:14.828 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:10:14 smithi115 conmon[90393]: 2022-05-14T05:10:14.500+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:10:15.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:10:14 smithi115 ceph-mon[78593]: pgmap v10624: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:10:15.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:10:14 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:10:15.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:10:14 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T05:10:15.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:10:14 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T05:10:15.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:10:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T05:10:15.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:10:14 smithi047 ceph-mon[77147]: pgmap v10624: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:10:15.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:10:14 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:10:15.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:10:14 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T05:10:15.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:10:14 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T05:10:15.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:10:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T05:10:15.832 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:10:15 smithi115 conmon[90393]: 2022-05-14T05:10:15.457+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:10:16.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:10:15 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:10:16.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:10:15 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T05:10:16.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:10:15 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:10:16.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:10:15 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T05:10:16.833 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:10:16 smithi115 conmon[90393]: 2022-05-14T05:10:16.420+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:10:17.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:10:16 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:10:17.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:10:16 smithi115 ceph-mon[78593]: pgmap v10625: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T05:10:17.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:10:16 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:10:17.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:10:16 smithi047 ceph-mon[77147]: pgmap v10625: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T05:10:17.839 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:10:17 smithi115 conmon[90393]: 2022-05-14T05:10:17.371+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:10:18.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:10:17 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:10:18.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:10:17 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:10:18.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:10:18 smithi115 conmon[90393]: 2022-05-14T05:10:18.348+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:10:19.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:10:18 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:10:19.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:10:18 smithi115 ceph-mon[78593]: pgmap v10626: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:10:19.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:10:18 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T05:10:19.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:10:18 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T05:10:19.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:10:18 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 20550 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T05:10:19.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:10:18 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:10:19.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:10:18 smithi047 ceph-mon[77147]: pgmap v10626: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:10:19.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:10:18 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T05:10:19.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:10:18 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T05:10:19.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:10:18 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 20550 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T05:10:19.843 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:10:19 smithi115 conmon[90393]: 2022-05-14T05:10:19.359+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:10:20.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:10:19 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:10:20.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:10:19 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:10:20.845 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:10:20 smithi115 conmon[90393]: 2022-05-14T05:10:20.388+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:10:21.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:10:20 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:10:21.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:10:20 smithi115 ceph-mon[78593]: pgmap v10627: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T05:10:21.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:10:20 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:10:21.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:10:20 smithi047 ceph-mon[77147]: pgmap v10627: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T05:10:21.851 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:10:21 smithi115 conmon[90393]: 2022-05-14T05:10:21.390+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:10:22.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:10:21 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:10:22.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:10:21 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:10:22.721 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:10:22 smithi115 conmon[90393]: 2022-05-14T05:10:22.423+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:10:23.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:10:22 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:10:23.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:10:22 smithi047 ceph-mon[77147]: pgmap v10628: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:10:23.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:10:22 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T05:10:23.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:10:22 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T05:10:23.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:10:22 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T05:10:23.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:10:22 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T05:10:23.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:10:22 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T05:10:23.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:10:22 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:10:23.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:10:22 smithi115 ceph-mon[78593]: pgmap v10628: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:10:23.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:10:22 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T05:10:23.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:10:22 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T05:10:23.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:10:22 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T05:10:23.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:10:22 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T05:10:23.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:10:22 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T05:10:23.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:10:23 smithi115 conmon[90393]: 2022-05-14T05:10:23.414+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:10:24.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:10:23 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:10:24.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:10:23 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 20555 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T05:10:24.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:10:23 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:10:24.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:10:23 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 20555 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T05:10:24.854 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:10:24 smithi115 conmon[90393]: 2022-05-14T05:10:24.432+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:10:25.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:10:24 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:10:25.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:10:24 smithi115 ceph-mon[78593]: pgmap v10629: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:10:25.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:10:24 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:10:25.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:10:24 smithi047 ceph-mon[77147]: pgmap v10629: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:10:25.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:10:25 smithi115 conmon[90393]: 2022-05-14T05:10:25.437+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:10:26.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:10:25 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:10:26.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:10:25 smithi047 conmon[78521]: 2022-05-14T05:10:25.775+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T05:10:26.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:10:25 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T05:10:26.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:10:25 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T05:10:26.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:10:25 smithi047 conmon[78521]: yield 2022-05-14T05:10:26.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:10:25 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T05:10:26.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:10:25 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T05:10:26.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:10:25 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T05:10:26.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:10:25 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T05:10:26.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:10:25 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T05:10:26.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:10:25 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T05:10:26.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:10:25 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T05:10:26.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:10:25 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T05:10:26.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:10:25 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T05:10:26.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:10:25 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T05:10:26.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:10:25 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T05:10:26.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:10:25 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T05:10:26.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:10:25 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T05:10:26.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:10:25 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T05:10:26.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:10:25 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T05:10:26.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:10:25 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:10:26.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:10:26 smithi115 conmon[90393]: 2022-05-14T05:10:26.435+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:10:27.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:10:26 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:10:27.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:10:26 smithi047 ceph-mon[77147]: pgmap v10630: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T05:10:27.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:10:26 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T05:10:27.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T05:10:27.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T05:10:27.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T05:10:27.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T05:10:27.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T05:10:27.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T05:10:27.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T05:10:27.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T05:10:27.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T05:10:27.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T05:10:27.298 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T05:10:27.298 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T05:10:27.298 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T05:10:27.298 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T05:10:27.299 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T05:10:27.299 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T05:10:27.299 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T05:10:27.299 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T05:10:27.299 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:10:26 smithi047 ceph-mon[77147]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T05:10:27.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:10:26 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:10:27.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:10:26 smithi115 ceph-mon[78593]: pgmap v10630: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T05:10:27.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:10:26 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T05:10:27.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T05:10:27.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T05:10:27.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T05:10:27.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T05:10:27.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T05:10:27.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T05:10:27.365 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T05:10:27.365 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T05:10:27.365 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T05:10:27.366 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T05:10:27.366 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T05:10:27.366 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T05:10:27.366 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T05:10:27.366 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T05:10:27.367 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T05:10:27.367 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T05:10:27.367 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T05:10:27.367 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T05:10:27.368 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:10:26 smithi115 ceph-mon[78593]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T05:10:27.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:10:27 smithi115 conmon[90393]: 2022-05-14T05:10:27.407+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:10:28.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:10:27 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:10:28.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:10:27 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:10:28.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:10:28 smithi115 conmon[90393]: 2022-05-14T05:10:28.442+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:10:29.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:10:28 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:10:29.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:10:28 smithi047 ceph-mon[77147]: pgmap v10631: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:10:29.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:10:28 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 20560 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T05:10:29.294 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:10:28 smithi047 conmon[78521]: 2022-05-14T05:10:28.847+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T05:10:29.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:10:28 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T05:10:29.295 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:10:28 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T05:10:29.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:10:28 smithi047 conmon[78521]: yield 2022-05-14T05:10:29.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:10:28 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T05:10:29.296 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:10:28 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T05:10:29.297 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:10:28 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T05:10:29.297 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:10:28 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T05:10:29.297 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:10:28 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T05:10:29.298 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:10:28 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T05:10:29.298 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:10:28 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T05:10:29.299 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:10:28 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T05:10:29.299 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:10:28 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T05:10:29.299 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:10:28 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T05:10:29.300 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:10:28 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T05:10:29.300 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:10:28 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T05:10:29.300 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:10:28 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T05:10:29.300 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:10:28 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T05:10:29.301 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:10:28 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T05:10:29.301 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:10:28 smithi047 conmon[78521]: 2022-05-14T05:10:28.848+0000 7f533ef76700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T05:10:29.301 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:10:28 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T05:10:29.302 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:10:28 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T05:10:29.302 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:10:28 smithi047 conmon[78521]: yield 2022-05-14T05:10:29.302 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:10:28 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T05:10:29.302 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:10:28 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T05:10:29.302 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:10:28 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T05:10:29.303 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:10:28 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T05:10:29.303 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:10:28 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T05:10:29.303 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:10:28 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T05:10:29.303 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:10:28 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T05:10:29.304 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:10:28 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T05:10:29.304 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:10:28 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T05:10:29.304 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:10:28 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T05:10:29.304 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:10:28 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T05:10:29.304 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:10:28 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T05:10:29.305 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:10:28 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T05:10:29.305 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:10:28 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T05:10:29.305 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:10:28 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T05:10:29.305 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:10:28 smithi047 conmon[78521]: 2022-05-14T05:10:29.305 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:10:28 smithi047 conmon[78521]: During handling of the above exception, another exception occurred: 2022-05-14T05:10:29.306 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:10:28 smithi047 conmon[78521]: 2022-05-14T05:10:29.306 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:10:28 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T05:10:29.306 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:10:28 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T05:10:29.306 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:10:28 smithi047 conmon[78521]: return f(*arg) 2022-05-14T05:10:29.306 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:10:28 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T05:10:29.307 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:10:28 smithi047 conmon[78521]: self._write_client_files(client_files, host) 2022-05-14T05:10:29.307 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:10:28 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T05:10:29.307 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:10:28 smithi047 conmon[78521]: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T05:10:29.307 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:10:28 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T05:10:29.307 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:10:28 smithi047 conmon[78521]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T05:10:29.308 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:10:28 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T05:10:29.308 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:10:28 smithi047 conmon[78521]: return self.event_loop.get_result(coro) 2022-05-14T05:10:29.308 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:10:28 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T05:10:29.308 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:10:28 smithi047 conmon[78521]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T05:10:29.308 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:10:28 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T05:10:29.309 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:10:28 smithi047 conmon[78521]: return self.__get_result() 2022-05-14T05:10:29.309 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:10:28 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T05:10:29.309 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:10:28 smithi047 conmon[78521]: raise self._exception 2022-05-14T05:10:29.309 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:10:28 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T05:10:29.310 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:10:28 smithi047 conmon[78521]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T05:10:29.310 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:10:28 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T05:10:29.310 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:10:28 smithi047 conmon[78521]: conn = await self._remote_connection(host, addr) 2022-05-14T05:10:29.310 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:10:28 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T05:10:29.310 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:10:28 smithi047 conmon[78521]: raise 2022-05-14T05:10:29.311 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:10:28 smithi047 conmon[78521]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T05:10:29.311 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:10:28 smithi047 conmon[78521]: self.gen.throw(type, value, traceback) 2022-05-14T05:10:29.311 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:10:28 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T05:10:29.311 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:10:28 smithi047 conmon[78521]: raise OrchestratorError(msg) 2022-05-14T05:10:29.311 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:10:28 smithi047 conmon[78521]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T05:10:29.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:10:28 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:10:29.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:10:28 smithi115 ceph-mon[78593]: pgmap v10631: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:10:29.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:10:28 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 20560 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T05:10:29.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:10:29 smithi115 conmon[90393]: 2022-05-14T05:10:29.481+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:10:30.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:10:29 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:10:30.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:10:29 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T05:10:30.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T05:10:30.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T05:10:30.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T05:10:30.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T05:10:30.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T05:10:30.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T05:10:30.294 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T05:10:30.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T05:10:30.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T05:10:30.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T05:10:30.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T05:10:30.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T05:10:30.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T05:10:30.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T05:10:30.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T05:10:30.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T05:10:30.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T05:10:30.298 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T05:10:30.298 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:10:29 smithi047 ceph-mon[77147]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T05:10:30.298 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T05:10:30.298 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T05:10:30.298 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T05:10:30.299 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T05:10:30.299 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T05:10:30.299 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T05:10:30.299 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T05:10:30.299 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T05:10:30.300 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T05:10:30.300 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T05:10:30.300 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T05:10:30.300 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T05:10:30.300 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T05:10:30.301 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T05:10:30.301 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T05:10:30.301 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T05:10:30.301 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T05:10:30.302 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T05:10:30.302 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T05:10:30.302 INFO:journalctl@ceph.mon.b.smithi047.stdout: During handling of the above exception, another exception occurred: 2022-05-14T05:10:30.302 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T05:10:30.302 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T05:10:30.303 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T05:10:30.303 INFO:journalctl@ceph.mon.b.smithi047.stdout: return f(*arg) 2022-05-14T05:10:30.303 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T05:10:30.303 INFO:journalctl@ceph.mon.b.smithi047.stdout: self._write_client_files(client_files, host) 2022-05-14T05:10:30.303 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T05:10:30.304 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T05:10:30.304 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T05:10:30.304 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T05:10:30.304 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T05:10:30.305 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.event_loop.get_result(coro) 2022-05-14T05:10:30.305 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T05:10:30.305 INFO:journalctl@ceph.mon.b.smithi047.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T05:10:30.306 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T05:10:30.306 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.__get_result() 2022-05-14T05:10:30.306 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T05:10:30.306 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise self._exception 2022-05-14T05:10:30.306 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T05:10:30.307 INFO:journalctl@ceph.mon.b.smithi047.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T05:10:30.307 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T05:10:30.307 INFO:journalctl@ceph.mon.b.smithi047.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T05:10:30.307 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T05:10:30.307 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise 2022-05-14T05:10:30.308 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T05:10:30.308 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.gen.throw(type, value, traceback) 2022-05-14T05:10:30.308 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T05:10:30.308 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OrchestratorError(msg) 2022-05-14T05:10:30.309 INFO:journalctl@ceph.mon.b.smithi047.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T05:10:30.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:10:29 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:10:30.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:10:29 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T05:10:30.359 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T05:10:30.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T05:10:30.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T05:10:30.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T05:10:30.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T05:10:30.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T05:10:30.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T05:10:30.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T05:10:30.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T05:10:30.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T05:10:30.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T05:10:30.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T05:10:30.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T05:10:30.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T05:10:30.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T05:10:30.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T05:10:30.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T05:10:30.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T05:10:30.363 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:10:29 smithi115 ceph-mon[78593]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T05:10:30.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T05:10:30.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T05:10:30.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T05:10:30.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T05:10:30.365 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T05:10:30.365 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T05:10:30.365 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T05:10:30.365 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T05:10:30.365 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T05:10:30.366 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T05:10:30.366 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T05:10:30.372 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T05:10:30.372 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T05:10:30.372 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T05:10:30.372 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T05:10:30.373 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T05:10:30.373 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T05:10:30.373 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T05:10:30.374 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T05:10:30.374 INFO:journalctl@ceph.mon.c.smithi115.stdout: During handling of the above exception, another exception occurred: 2022-05-14T05:10:30.374 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T05:10:30.374 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T05:10:30.375 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T05:10:30.375 INFO:journalctl@ceph.mon.c.smithi115.stdout: return f(*arg) 2022-05-14T05:10:30.375 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T05:10:30.375 INFO:journalctl@ceph.mon.c.smithi115.stdout: self._write_client_files(client_files, host) 2022-05-14T05:10:30.375 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T05:10:30.376 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T05:10:30.376 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T05:10:30.376 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T05:10:30.376 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T05:10:30.376 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.event_loop.get_result(coro) 2022-05-14T05:10:30.377 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T05:10:30.377 INFO:journalctl@ceph.mon.c.smithi115.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T05:10:30.377 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T05:10:30.377 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.__get_result() 2022-05-14T05:10:30.378 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T05:10:30.378 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise self._exception 2022-05-14T05:10:30.379 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T05:10:30.380 INFO:journalctl@ceph.mon.c.smithi115.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T05:10:30.380 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T05:10:30.380 INFO:journalctl@ceph.mon.c.smithi115.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T05:10:30.381 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T05:10:30.381 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise 2022-05-14T05:10:30.381 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T05:10:30.382 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.gen.throw(type, value, traceback) 2022-05-14T05:10:30.382 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T05:10:30.383 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OrchestratorError(msg) 2022-05-14T05:10:30.383 INFO:journalctl@ceph.mon.c.smithi115.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T05:10:30.860 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:10:30 smithi115 conmon[90393]: 2022-05-14T05:10:30.502+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:10:31.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:10:30 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:10:31.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:10:30 smithi047 ceph-mon[77147]: pgmap v10632: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T05:10:31.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:10:30 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T05:10:31.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:10:30 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T05:10:31.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:10:30 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:10:31.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:10:30 smithi115 ceph-mon[78593]: pgmap v10632: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T05:10:31.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:10:30 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T05:10:31.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:10:30 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T05:10:31.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:10:31 smithi115 conmon[90393]: 2022-05-14T05:10:31.469+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:10:32.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:10:31 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:10:32.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:10:31 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:10:32.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:10:32 smithi115 conmon[90393]: 2022-05-14T05:10:32.488+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:10:33.295 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:10:32 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:10:33.295 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:10:32 smithi047 ceph-mon[77147]: pgmap v10633: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:10:33.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:10:32 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:10:33.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:10:32 smithi115 ceph-mon[78593]: pgmap v10633: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:10:33.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:10:33 smithi115 conmon[90393]: 2022-05-14T05:10:33.493+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:10:34.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:10:33 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:10:34.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:10:33 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 20565 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T05:10:34.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:10:33 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:10:34.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:10:33 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 20565 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T05:10:34.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:10:34 smithi115 conmon[90393]: 2022-05-14T05:10:34.508+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:10:35.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:10:34 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:10:35.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:10:34 smithi047 ceph-mon[77147]: pgmap v10634: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:10:35.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:10:34 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T05:10:35.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:10:34 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T05:10:35.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:10:34 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:10:35.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:10:34 smithi115 ceph-mon[78593]: pgmap v10634: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:10:35.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:10:34 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T05:10:35.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:10:34 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T05:10:35.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:10:35 smithi115 conmon[90393]: 2022-05-14T05:10:35.492+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:10:36.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:10:35 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:10:36.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:10:35 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:10:36.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:10:36 smithi115 conmon[90393]: 2022-05-14T05:10:36.494+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:10:37.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:10:36 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:10:37.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:10:36 smithi047 ceph-mon[77147]: pgmap v10635: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T05:10:37.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:10:36 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:10:37.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:10:36 smithi115 ceph-mon[78593]: pgmap v10635: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T05:10:37.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:10:37 smithi115 conmon[90393]: 2022-05-14T05:10:37.448+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:10:38.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:10:37 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:10:38.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:10:37 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:10:38.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:10:38 smithi115 conmon[90393]: 2022-05-14T05:10:38.494+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:10:39.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:10:38 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:10:39.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:10:38 smithi047 ceph-mon[77147]: pgmap v10636: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:10:39.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:10:38 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T05:10:39.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:10:38 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T05:10:39.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:10:38 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 20570 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T05:10:39.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:10:38 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:10:39.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:10:38 smithi115 ceph-mon[78593]: pgmap v10636: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:10:39.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:10:38 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T05:10:39.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:10:38 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T05:10:39.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:10:38 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 20570 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T05:10:39.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:10:39 smithi115 conmon[90393]: 2022-05-14T05:10:39.526+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:10:40.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:10:39 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:10:40.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:10:39 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:10:40.863 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:10:40 smithi115 conmon[90393]: 2022-05-14T05:10:40.482+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:10:41.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:10:40 smithi047 ceph-mon[77147]: pgmap v10637: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T05:10:41.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:10:40 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:10:41.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:10:40 smithi115 ceph-mon[78593]: pgmap v10637: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T05:10:41.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:10:40 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:10:41.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:10:41 smithi115 conmon[90393]: 2022-05-14T05:10:41.436+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:10:42.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:10:41 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:10:42.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:10:41 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:10:42.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:10:42 smithi115 conmon[90393]: 2022-05-14T05:10:42.431+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:10:43.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:10:42 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:10:43.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:10:42 smithi047 ceph-mon[77147]: pgmap v10638: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:10:43.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:10:42 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T05:10:43.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:10:42 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T05:10:43.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:10:42 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:10:43.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:10:42 smithi115 ceph-mon[78593]: pgmap v10638: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:10:43.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:10:42 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T05:10:43.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:10:42 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T05:10:43.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:10:43 smithi115 conmon[90393]: 2022-05-14T05:10:43.427+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:10:44.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:10:43 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:10:44.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:10:43 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 20575 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T05:10:44.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:10:43 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:10:44.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:10:43 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 20575 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T05:10:44.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:10:44 smithi115 conmon[90393]: 2022-05-14T05:10:44.399+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:10:45.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:10:44 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:10:45.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:10:44 smithi047 ceph-mon[77147]: pgmap v10639: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:10:45.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:10:44 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:10:45.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:10:44 smithi115 ceph-mon[78593]: pgmap v10639: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:10:45.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:10:45 smithi115 conmon[90393]: 2022-05-14T05:10:45.415+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:10:46.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:10:45 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:10:46.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:10:45 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:10:46.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:10:46 smithi115 conmon[90393]: 2022-05-14T05:10:46.427+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:10:47.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:10:46 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:10:47.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:10:46 smithi047 ceph-mon[77147]: pgmap v10640: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T05:10:47.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:10:46 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:10:47.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:10:46 smithi115 ceph-mon[78593]: pgmap v10640: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T05:10:47.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:10:47 smithi115 conmon[90393]: 2022-05-14T05:10:47.427+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:10:48.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:10:47 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:10:48.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:10:47 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:10:48.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:10:48 smithi115 conmon[90393]: 2022-05-14T05:10:48.421+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:10:49.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:10:48 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:10:49.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:10:48 smithi047 ceph-mon[77147]: pgmap v10641: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:10:49.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:10:48 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 20580 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T05:10:49.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:10:48 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:10:49.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:10:48 smithi115 ceph-mon[78593]: pgmap v10641: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:10:49.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:10:48 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 20580 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T05:10:49.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:10:49 smithi115 conmon[90393]: 2022-05-14T05:10:49.406+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:10:50.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:10:49 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:10:50.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:10:49 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:10:50.864 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:10:50 smithi115 conmon[90393]: 2022-05-14T05:10:50.440+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:10:51.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:10:50 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:10:51.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:10:50 smithi047 ceph-mon[77147]: pgmap v10642: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T05:10:51.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:10:50 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T05:10:51.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:10:50 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T05:10:51.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:10:50 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:10:51.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:10:50 smithi115 ceph-mon[78593]: pgmap v10642: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T05:10:51.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:10:50 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T05:10:51.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:10:50 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T05:10:51.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:10:51 smithi115 conmon[90393]: 2022-05-14T05:10:51.409+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:10:52.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:10:51 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:10:52.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:10:51 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:10:52.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:10:52 smithi115 conmon[90393]: 2022-05-14T05:10:52.372+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:10:53.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:10:52 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:10:53.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:10:52 smithi047 ceph-mon[77147]: pgmap v10643: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:10:53.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:10:52 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:10:53.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:10:52 smithi115 ceph-mon[78593]: pgmap v10643: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:10:53.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:10:53 smithi115 conmon[90393]: 2022-05-14T05:10:53.409+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:10:54.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:10:53 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:10:54.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:10:53 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 20585 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T05:10:54.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:10:53 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:10:54.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:10:53 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 20585 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T05:10:54.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:10:54 smithi115 conmon[90393]: 2022-05-14T05:10:54.417+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:10:55.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:10:54 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:10:55.295 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:10:54 smithi047 ceph-mon[77147]: pgmap v10644: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:10:55.296 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:10:54 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T05:10:55.296 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:10:54 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T05:10:55.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:10:54 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:10:55.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:10:54 smithi115 ceph-mon[78593]: pgmap v10644: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:10:55.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:10:54 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T05:10:55.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:10:54 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T05:10:55.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:10:55 smithi115 conmon[90393]: 2022-05-14T05:10:55.391+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:10:56.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:10:55 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:10:56.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:10:55 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:10:56.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:10:56 smithi115 conmon[90393]: 2022-05-14T05:10:56.376+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:10:57.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:10:56 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:10:57.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:10:56 smithi047 ceph-mon[77147]: pgmap v10645: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T05:10:57.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:10:56 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:10:57.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:10:56 smithi115 ceph-mon[78593]: pgmap v10645: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T05:10:57.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:10:57 smithi115 conmon[90393]: 2022-05-14T05:10:57.386+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:10:58.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:10:57 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:10:58.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:10:57 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:10:58.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:10:58 smithi115 conmon[90393]: 2022-05-14T05:10:58.418+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:10:59.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:10:58 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:10:59.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:10:58 smithi047 ceph-mon[77147]: pgmap v10646: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:10:59.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:10:58 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T05:10:59.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:10:58 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T05:10:59.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:10:58 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 20590 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T05:10:59.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:10:58 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:10:59.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:10:58 smithi115 ceph-mon[78593]: pgmap v10646: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:10:59.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:10:58 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T05:10:59.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:10:58 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T05:10:59.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:10:58 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 20590 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T05:10:59.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:10:59 smithi115 conmon[90393]: 2022-05-14T05:10:59.416+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:11:00.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:10:59 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:11:00.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:10:59 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:11:00.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:11:00 smithi115 conmon[90393]: 2022-05-14T05:11:00.413+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:11:01.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:11:00 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:11:01.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:11:00 smithi047 ceph-mon[77147]: pgmap v10647: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T05:11:01.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:11:00 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:11:01.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:11:00 smithi115 ceph-mon[78593]: pgmap v10647: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T05:11:01.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:11:01 smithi115 conmon[90393]: 2022-05-14T05:11:01.415+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:11:02.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:11:01 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:11:02.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:11:01 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:11:02.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:11:02 smithi115 conmon[90393]: 2022-05-14T05:11:02.365+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:11:03.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:11:02 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:11:03.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:11:02 smithi047 ceph-mon[77147]: pgmap v10648: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:11:03.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:11:02 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T05:11:03.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:11:02 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T05:11:03.322 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:11:02 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:11:03.323 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:11:02 smithi115 ceph-mon[78593]: pgmap v10648: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:11:03.323 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:11:02 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T05:11:03.323 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:11:02 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T05:11:03.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:11:03 smithi115 conmon[90393]: 2022-05-14T05:11:03.320+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:11:04.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:11:03 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:11:04.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:11:03 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 20595 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T05:11:04.343 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:11:03 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:11:04.343 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:11:03 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 20595 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T05:11:04.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:11:04 smithi115 conmon[90393]: 2022-05-14T05:11:04.341+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:11:05.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:11:04 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:11:05.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:11:04 smithi047 ceph-mon[77147]: pgmap v10649: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:11:05.342 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:11:04 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:11:05.342 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:11:04 smithi115 ceph-mon[78593]: pgmap v10649: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:11:05.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:11:05 smithi115 conmon[90393]: 2022-05-14T05:11:05.339+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:11:06.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:11:05 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:11:06.346 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:11:05 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:11:06.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:11:06 smithi115 conmon[90393]: 2022-05-14T05:11:06.344+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:11:07.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:11:06 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:11:07.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:11:06 smithi047 ceph-mon[77147]: pgmap v10650: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T05:11:07.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:11:06 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:11:07.309 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:11:06 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:11:07.310 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:11:06 smithi115 ceph-mon[78593]: pgmap v10650: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T05:11:07.310 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:11:06 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:11:07.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:11:07 smithi115 conmon[90393]: 2022-05-14T05:11:07.307+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:11:08.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:11:07 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:11:08.345 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:11:07 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:11:08.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:11:08 smithi115 conmon[90393]: 2022-05-14T05:11:08.343+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:11:09.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:11:08 smithi047 ceph-mon[77147]: pgmap v10651: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:11:09.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:11:08 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 20600 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T05:11:09.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:11:08 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:11:09.307 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:11:08 smithi115 ceph-mon[78593]: pgmap v10651: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:11:09.307 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:11:08 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 20600 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T05:11:09.307 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:11:08 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:11:09.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:11:09 smithi115 conmon[90393]: 2022-05-14T05:11:09.304+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:11:10.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:11:09 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:11:10.336 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:11:09 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:11:10.610 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:11:10 smithi115 conmon[90393]: 2022-05-14T05:11:10.334+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:11:11.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:11:10 smithi047 ceph-mon[77147]: pgmap v10652: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T05:11:11.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:11:10 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T05:11:11.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:11:10 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T05:11:11.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:11:10 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:11:11.309 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:11:10 smithi115 ceph-mon[78593]: pgmap v10652: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T05:11:11.310 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:11:10 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T05:11:11.310 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:11:10 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T05:11:11.310 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:11:10 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:11:11.608 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:11:11 smithi115 conmon[90393]: 2022-05-14T05:11:11.307+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:11:12.287 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:11:11 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:11:12.287 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:11:11 smithi115 ceph-mon[78593]: pgmap v10653: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:11:12.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:11:11 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:11:12.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:11:11 smithi047 ceph-mon[77147]: pgmap v10653: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:11:12.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:11:12 smithi115 conmon[90393]: 2022-05-14T05:11:12.284+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:11:13.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:11:12 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:11:13.307 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:11:12 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:11:13.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:11:13 smithi115 conmon[90393]: 2022-05-14T05:11:13.304+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:11:14.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:11:14 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 20605 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T05:11:14.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:11:14 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:11:14.295 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:11:14 smithi047 ceph-mon[77147]: pgmap v10654: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:11:14.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:11:14 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 20605 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T05:11:14.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:11:14 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:11:14.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:11:14 smithi115 ceph-mon[78593]: pgmap v10654: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:11:14.610 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:11:14 smithi115 conmon[90393]: 2022-05-14T05:11:14.290+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:11:15.334 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:11:15 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T05:11:15.334 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:11:15 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T05:11:15.335 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:11:15 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:11:15.335 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:11:15 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T05:11:15.335 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:11:15 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T05:11:15.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:11:15 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T05:11:15.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:11:15 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T05:11:15.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:11:15 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:11:15.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:11:15 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T05:11:15.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:11:15 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T05:11:15.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:11:15 smithi115 conmon[90393]: 2022-05-14T05:11:15.331+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:11:16.347 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:11:16 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:11:16.347 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:11:16 smithi115 ceph-mon[78593]: pgmap v10655: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T05:11:16.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:11:16 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:11:16.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:11:16 smithi047 ceph-mon[77147]: pgmap v10655: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T05:11:16.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:11:16 smithi115 conmon[90393]: 2022-05-14T05:11:16.344+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:11:17.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:11:17 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:11:17.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:11:17 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:11:17.860 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:11:17 smithi115 conmon[90393]: 2022-05-14T05:11:17.384+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:11:18.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:11:18 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:11:18.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:11:18 smithi115 ceph-mon[78593]: pgmap v10656: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:11:18.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:11:18 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 20610 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T05:11:18.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:11:18 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:11:18.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:11:18 smithi047 ceph-mon[77147]: pgmap v10656: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:11:18.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:11:18 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 20610 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T05:11:18.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:11:18 smithi115 conmon[90393]: 2022-05-14T05:11:18.388+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:11:19.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:11:19 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T05:11:19.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:11:19 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T05:11:19.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:11:19 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:11:19.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:11:19 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T05:11:19.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:11:19 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T05:11:19.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:11:19 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:11:19.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:11:19 smithi115 conmon[90393]: 2022-05-14T05:11:19.430+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:11:20.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:11:20 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:11:20.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:11:20 smithi115 ceph-mon[78593]: pgmap v10657: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T05:11:20.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:11:20 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:11:20.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:11:20 smithi047 ceph-mon[77147]: pgmap v10657: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T05:11:20.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:11:20 smithi115 conmon[90393]: 2022-05-14T05:11:20.457+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:11:21.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:11:21 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:11:21.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:11:21 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:11:21.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:11:21 smithi115 conmon[90393]: 2022-05-14T05:11:21.417+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:11:22.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:11:22 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:11:22.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:11:22 smithi115 ceph-mon[78593]: pgmap v10658: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:11:22.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:11:22 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:11:22.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:11:22 smithi047 ceph-mon[77147]: pgmap v10658: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:11:22.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:11:22 smithi115 conmon[90393]: 2022-05-14T05:11:22.448+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:11:23.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:11:23 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T05:11:23.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:11:23 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T05:11:23.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:11:23 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:11:23.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:11:23 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 20615 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T05:11:23.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:11:23 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T05:11:23.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:11:23 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T05:11:23.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:11:23 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:11:23.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:11:23 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 20615 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T05:11:23.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:11:23 smithi115 conmon[90393]: 2022-05-14T05:11:23.400+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:11:24.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:11:24 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:11:24.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:11:24 smithi115 ceph-mon[78593]: pgmap v10659: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:11:24.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:11:24 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:11:24.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:11:24 smithi047 ceph-mon[77147]: pgmap v10659: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:11:24.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:11:24 smithi115 conmon[90393]: 2022-05-14T05:11:24.373+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:11:25.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:11:25 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:11:25.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:11:25 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:11:25.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:11:25 smithi115 conmon[90393]: 2022-05-14T05:11:25.412+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:11:26.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:11:26 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:11:26.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:11:26 smithi115 ceph-mon[78593]: pgmap v10660: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T05:11:26.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:11:26 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:11:26.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:11:26 smithi047 ceph-mon[77147]: pgmap v10660: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T05:11:26.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:11:26 smithi115 conmon[90393]: 2022-05-14T05:11:26.411+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:11:27.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:11:27 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:11:27.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:11:27 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:11:27.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:11:27 smithi115 conmon[90393]: 2022-05-14T05:11:27.422+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:11:28.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:11:28 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:11:28.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:11:28 smithi115 ceph-mon[78593]: pgmap v10661: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:11:28.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:11:28 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 20620 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T05:11:28.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:11:28 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:11:28.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:11:28 smithi047 ceph-mon[77147]: pgmap v10661: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:11:28.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:11:28 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 20620 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T05:11:28.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:11:28 smithi115 conmon[90393]: 2022-05-14T05:11:28.375+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:11:29.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:11:29 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:11:29.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:11:29 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T05:11:29.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:11:29 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T05:11:29.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:11:29 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T05:11:29.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:11:29 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:11:29.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:11:29 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T05:11:29.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:11:29 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T05:11:29.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:11:29 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T05:11:29.792 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:11:29 smithi047 conmon[78521]: 2022-05-14T05:11:29.455+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T05:11:29.793 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:11:29 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T05:11:29.793 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:11:29 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T05:11:29.793 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:11:29 smithi047 conmon[78521]: yield 2022-05-14T05:11:29.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:11:29 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T05:11:29.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:11:29 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T05:11:29.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:11:29 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T05:11:29.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:11:29 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T05:11:29.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:11:29 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T05:11:29.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:11:29 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T05:11:29.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:11:29 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T05:11:29.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:11:29 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T05:11:29.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:11:29 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T05:11:29.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:11:29 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T05:11:29.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:11:29 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T05:11:29.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:11:29 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T05:11:29.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:11:29 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T05:11:29.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:11:29 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T05:11:29.797 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:11:29 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T05:11:29.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:11:29 smithi115 conmon[90393]: 2022-05-14T05:11:29.383+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:11:30.359 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:11:30 smithi115 conmon[90393]: 2022-05-14T05:11:30.342+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:11:30.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:11:30 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:11:30.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:11:30 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T05:11:30.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T05:11:30.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T05:11:30.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T05:11:30.360 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T05:11:30.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T05:11:30.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T05:11:30.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T05:11:30.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T05:11:30.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T05:11:30.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T05:11:30.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T05:11:30.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T05:11:30.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T05:11:30.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T05:11:30.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T05:11:30.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T05:11:30.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T05:11:30.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T05:11:30.363 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:11:30 smithi115 ceph-mon[78593]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T05:11:30.364 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:11:30 smithi115 ceph-mon[78593]: pgmap v10662: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T05:11:30.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:11:30 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:11:30.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:11:30 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T05:11:30.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T05:11:30.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T05:11:30.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T05:11:30.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T05:11:30.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T05:11:30.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T05:11:30.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T05:11:30.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T05:11:30.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T05:11:30.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T05:11:30.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T05:11:30.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T05:11:30.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T05:11:30.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T05:11:30.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T05:11:30.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T05:11:30.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T05:11:30.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T05:11:30.547 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:11:30 smithi047 ceph-mon[77147]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T05:11:30.547 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:11:30 smithi047 ceph-mon[77147]: pgmap v10662: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T05:11:31.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:11:31 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T05:11:31.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:11:31 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T05:11:31.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:11:31 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:11:31.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:11:31 smithi115 conmon[90393]: 2022-05-14T05:11:31.316+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:11:31.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:11:31 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T05:11:31.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:11:31 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T05:11:31.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:11:31 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:11:32.531 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:11:32 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:11:32.531 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:11:32 smithi047 ceph-mon[77147]: pgmap v10663: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:11:32.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:11:32 smithi115 conmon[90393]: 2022-05-14T05:11:32.330+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:11:32.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:11:32 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:11:32.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:11:32 smithi115 ceph-mon[78593]: pgmap v10663: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:11:32.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:11:32 smithi047 conmon[78521]: 2022-05-14T05:11:32.527+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T05:11:32.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:11:32 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T05:11:32.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:11:32 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T05:11:32.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:11:32 smithi047 conmon[78521]: yield 2022-05-14T05:11:32.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:11:32 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T05:11:32.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:11:32 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T05:11:32.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:11:32 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T05:11:32.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:11:32 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T05:11:32.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:11:32 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T05:11:32.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:11:32 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T05:11:32.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:11:32 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T05:11:32.797 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:11:32 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T05:11:32.797 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:11:32 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T05:11:32.797 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:11:32 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T05:11:32.797 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:11:32 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T05:11:32.798 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:11:32 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T05:11:32.798 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:11:32 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T05:11:32.798 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:11:32 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T05:11:32.798 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:11:32 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T05:11:32.799 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:11:32 smithi047 conmon[78521]: 2022-05-14T05:11:32.528+0000 7f5344781700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T05:11:32.799 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:11:32 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T05:11:32.799 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:11:32 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T05:11:32.799 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:11:32 smithi047 conmon[78521]: yield 2022-05-14T05:11:32.799 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:11:32 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T05:11:32.800 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:11:32 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T05:11:32.800 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:11:32 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T05:11:32.800 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:11:32 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T05:11:32.800 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:11:32 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T05:11:32.801 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:11:32 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T05:11:32.801 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:11:32 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T05:11:32.801 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:11:32 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T05:11:32.801 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:11:32 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T05:11:32.802 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:11:32 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T05:11:32.802 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:11:32 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T05:11:32.802 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:11:32 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T05:11:32.802 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:11:32 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T05:11:32.802 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:11:32 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T05:11:32.803 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:11:32 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T05:11:32.803 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:11:32 smithi047 conmon[78521]: 2022-05-14T05:11:32.803 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:11:32 smithi047 conmon[78521]: During handling of the above exception, another exception occurred: 2022-05-14T05:11:32.803 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:11:32 smithi047 conmon[78521]: 2022-05-14T05:11:32.804 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:11:32 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T05:11:32.804 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:11:32 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T05:11:32.804 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:11:32 smithi047 conmon[78521]: return f(*arg) 2022-05-14T05:11:32.804 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:11:32 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T05:11:32.805 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:11:32 smithi047 conmon[78521]: self._write_client_files(client_files, host) 2022-05-14T05:11:32.805 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:11:32 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T05:11:32.805 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:11:32 smithi047 conmon[78521]: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T05:11:32.805 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:11:32 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T05:11:32.806 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:11:32 smithi047 conmon[78521]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T05:11:32.806 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:11:32 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T05:11:32.806 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:11:32 smithi047 conmon[78521]: return self.event_loop.get_result(coro) 2022-05-14T05:11:32.806 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:11:32 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T05:11:32.806 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:11:32 smithi047 conmon[78521]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T05:11:32.807 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:11:32 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T05:11:32.807 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:11:32 smithi047 conmon[78521]: return self.__get_result() 2022-05-14T05:11:32.807 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:11:32 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T05:11:32.807 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:11:32 smithi047 conmon[78521]: raise self._exception 2022-05-14T05:11:32.808 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:11:32 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T05:11:32.808 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:11:32 smithi047 conmon[78521]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T05:11:32.808 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:11:32 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T05:11:32.808 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:11:32 smithi047 conmon[78521]: conn = await self._remote_connection(host, addr) 2022-05-14T05:11:32.808 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:11:32 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T05:11:32.809 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:11:32 smithi047 conmon[78521]: raise 2022-05-14T05:11:32.809 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:11:32 smithi047 conmon[78521]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T05:11:32.809 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:11:32 smithi047 conmon[78521]: self.gen.throw(type, value, traceback) 2022-05-14T05:11:32.809 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:11:32 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T05:11:32.810 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:11:32 smithi047 conmon[78521]: raise OrchestratorError(msg) 2022-05-14T05:11:32.810 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:11:32 smithi047 conmon[78521]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T05:11:33.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:11:33 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:11:33.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:11:33 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T05:11:33.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T05:11:33.543 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T05:11:33.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T05:11:33.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T05:11:33.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T05:11:33.544 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T05:11:33.545 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T05:11:33.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T05:11:33.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T05:11:33.546 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T05:11:33.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T05:11:33.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T05:11:33.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T05:11:33.547 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T05:11:33.548 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T05:11:33.548 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T05:11:33.548 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T05:11:33.549 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T05:11:33.550 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:11:33 smithi047 ceph-mon[77147]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T05:11:33.550 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T05:11:33.550 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T05:11:33.550 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T05:11:33.551 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T05:11:33.551 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T05:11:33.551 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T05:11:33.551 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T05:11:33.551 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T05:11:33.552 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T05:11:33.552 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T05:11:33.552 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T05:11:33.559 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T05:11:33.559 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T05:11:33.560 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T05:11:33.560 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T05:11:33.560 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T05:11:33.560 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T05:11:33.561 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T05:11:33.561 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T05:11:33.561 INFO:journalctl@ceph.mon.b.smithi047.stdout: During handling of the above exception, another exception occurred: 2022-05-14T05:11:33.561 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T05:11:33.561 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T05:11:33.562 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T05:11:33.562 INFO:journalctl@ceph.mon.b.smithi047.stdout: return f(*arg) 2022-05-14T05:11:33.562 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T05:11:33.562 INFO:journalctl@ceph.mon.b.smithi047.stdout: self._write_client_files(client_files, host) 2022-05-14T05:11:33.563 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T05:11:33.563 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T05:11:33.563 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T05:11:33.563 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T05:11:33.564 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T05:11:33.564 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.event_loop.get_result(coro) 2022-05-14T05:11:33.564 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T05:11:33.564 INFO:journalctl@ceph.mon.b.smithi047.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T05:11:33.564 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T05:11:33.565 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.__get_result() 2022-05-14T05:11:33.565 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T05:11:33.565 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise self._exception 2022-05-14T05:11:33.565 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T05:11:33.566 INFO:journalctl@ceph.mon.b.smithi047.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T05:11:33.566 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T05:11:33.566 INFO:journalctl@ceph.mon.b.smithi047.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T05:11:33.566 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T05:11:33.566 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise 2022-05-14T05:11:33.567 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T05:11:33.567 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.gen.throw(type, value, traceback) 2022-05-14T05:11:33.567 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T05:11:33.567 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OrchestratorError(msg) 2022-05-14T05:11:33.568 INFO:journalctl@ceph.mon.b.smithi047.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T05:11:33.568 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:11:33 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 20625 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T05:11:33.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:11:33 smithi115 conmon[90393]: 2022-05-14T05:11:33.331+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:11:33.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:11:33 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:11:33.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:11:33 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T05:11:33.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T05:11:33.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T05:11:33.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T05:11:33.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T05:11:33.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T05:11:33.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T05:11:33.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T05:11:33.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T05:11:33.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T05:11:33.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T05:11:33.613 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T05:11:33.613 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T05:11:33.613 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T05:11:33.613 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T05:11:33.614 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T05:11:33.614 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T05:11:33.614 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T05:11:33.614 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T05:11:33.614 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:11:33 smithi115 ceph-mon[78593]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T05:11:33.615 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T05:11:33.615 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T05:11:33.615 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T05:11:33.615 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T05:11:33.616 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T05:11:33.616 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T05:11:33.616 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T05:11:33.616 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T05:11:33.616 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T05:11:33.617 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T05:11:33.617 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T05:11:33.617 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T05:11:33.617 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T05:11:33.618 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T05:11:33.618 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T05:11:33.618 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T05:11:33.618 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T05:11:33.618 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T05:11:33.619 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T05:11:33.619 INFO:journalctl@ceph.mon.c.smithi115.stdout: During handling of the above exception, another exception occurred: 2022-05-14T05:11:33.619 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T05:11:33.619 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T05:11:33.619 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T05:11:33.620 INFO:journalctl@ceph.mon.c.smithi115.stdout: return f(*arg) 2022-05-14T05:11:33.620 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T05:11:33.620 INFO:journalctl@ceph.mon.c.smithi115.stdout: self._write_client_files(client_files, host) 2022-05-14T05:11:33.620 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T05:11:33.621 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T05:11:33.621 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T05:11:33.621 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T05:11:33.621 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T05:11:33.621 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.event_loop.get_result(coro) 2022-05-14T05:11:33.622 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T05:11:33.622 INFO:journalctl@ceph.mon.c.smithi115.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T05:11:33.622 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T05:11:33.623 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.__get_result() 2022-05-14T05:11:33.623 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T05:11:33.623 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise self._exception 2022-05-14T05:11:33.623 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T05:11:33.623 INFO:journalctl@ceph.mon.c.smithi115.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T05:11:33.624 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T05:11:33.624 INFO:journalctl@ceph.mon.c.smithi115.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T05:11:33.624 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T05:11:33.624 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise 2022-05-14T05:11:33.625 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T05:11:33.625 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.gen.throw(type, value, traceback) 2022-05-14T05:11:33.625 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T05:11:33.625 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OrchestratorError(msg) 2022-05-14T05:11:33.625 INFO:journalctl@ceph.mon.c.smithi115.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T05:11:33.626 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:11:33 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 20625 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T05:11:34.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:11:34 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:11:34.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:11:34 smithi047 ceph-mon[77147]: pgmap v10664: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:11:34.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:11:34 smithi115 conmon[90393]: 2022-05-14T05:11:34.290+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:11:34.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:11:34 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:11:34.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:11:34 smithi115 ceph-mon[78593]: pgmap v10664: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:11:35.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:11:35 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T05:11:35.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:11:35 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T05:11:35.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:11:35 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:11:35.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:11:35 smithi115 conmon[90393]: 2022-05-14T05:11:35.300+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:11:35.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:11:35 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T05:11:35.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:11:35 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T05:11:35.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:11:35 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:11:36.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:11:36 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:11:36.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:11:36 smithi047 ceph-mon[77147]: pgmap v10665: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T05:11:36.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:11:36 smithi115 conmon[90393]: 2022-05-14T05:11:36.265+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:11:36.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:11:36 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:11:36.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:11:36 smithi115 ceph-mon[78593]: pgmap v10665: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T05:11:37.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:11:37 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:11:37.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:11:37 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:11:37.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:11:37 smithi115 conmon[90393]: 2022-05-14T05:11:37.271+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:11:38.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:11:38 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:11:38.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:11:38 smithi047 ceph-mon[77147]: pgmap v10666: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:11:38.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:11:38 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 20630 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T05:11:38.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:11:38 smithi115 conmon[90393]: 2022-05-14T05:11:38.296+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:11:38.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:11:38 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:11:38.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:11:38 smithi115 ceph-mon[78593]: pgmap v10666: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:11:38.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:11:38 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 20630 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T05:11:39.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:11:39 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T05:11:39.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:11:39 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T05:11:39.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:11:39 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:11:39.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:11:39 smithi115 conmon[90393]: 2022-05-14T05:11:39.308+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:11:39.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:11:39 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T05:11:39.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:11:39 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T05:11:39.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:11:39 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:11:40.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:11:40 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:11:40.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:11:40 smithi047 ceph-mon[77147]: pgmap v10667: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T05:11:40.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:11:40 smithi115 conmon[90393]: 2022-05-14T05:11:40.310+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:11:40.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:11:40 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:11:40.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:11:40 smithi115 ceph-mon[78593]: pgmap v10667: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T05:11:41.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:11:41 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:11:41.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:11:41 smithi115 conmon[90393]: 2022-05-14T05:11:41.345+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:11:41.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:11:41 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:11:42.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:11:42 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:11:42.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:11:42 smithi047 ceph-mon[77147]: pgmap v10668: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:11:42.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:11:42 smithi115 conmon[90393]: 2022-05-14T05:11:42.364+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:11:42.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:11:42 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:11:42.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:11:42 smithi115 ceph-mon[78593]: pgmap v10668: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:11:43.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:11:43 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T05:11:43.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:11:43 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T05:11:43.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:11:43 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:11:43.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:11:43 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 20635 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T05:11:43.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:11:43 smithi115 conmon[90393]: 2022-05-14T05:11:43.315+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:11:43.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:11:43 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T05:11:43.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:11:43 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T05:11:43.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:11:43 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:11:43.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:11:43 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 20635 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T05:11:44.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:11:44 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:11:44.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:11:44 smithi047 ceph-mon[77147]: pgmap v10669: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:11:44.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:11:44 smithi115 conmon[90393]: 2022-05-14T05:11:44.268+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:11:44.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:11:44 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:11:44.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:11:44 smithi115 ceph-mon[78593]: pgmap v10669: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:11:45.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:11:45 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:11:45.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:11:45 smithi115 conmon[90393]: 2022-05-14T05:11:45.268+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:11:45.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:11:45 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:11:46.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:11:46 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:11:46.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:11:46 smithi047 ceph-mon[77147]: pgmap v10670: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T05:11:46.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:11:46 smithi115 conmon[90393]: 2022-05-14T05:11:46.257+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:11:46.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:11:46 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:11:46.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:11:46 smithi115 ceph-mon[78593]: pgmap v10670: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T05:11:47.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:11:47 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:11:47.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:11:47 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:11:47.610 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:11:47 smithi115 conmon[90393]: 2022-05-14T05:11:47.273+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:11:48.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:11:48 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:11:48.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:11:48 smithi047 ceph-mon[77147]: pgmap v10671: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:11:48.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:11:48 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 20640 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T05:11:48.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:11:48 smithi115 conmon[90393]: 2022-05-14T05:11:48.271+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:11:48.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:11:48 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:11:48.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:11:48 smithi115 ceph-mon[78593]: pgmap v10671: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:11:48.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:11:48 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 20640 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T05:11:49.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:11:49 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:11:49.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:11:49 smithi115 conmon[90393]: 2022-05-14T05:11:49.272+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:11:49.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:11:49 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:11:50.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:11:50 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:11:50.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:11:50 smithi047 ceph-mon[77147]: pgmap v10672: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T05:11:50.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:11:50 smithi115 conmon[90393]: 2022-05-14T05:11:50.289+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:11:50.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:11:50 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:11:50.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:11:50 smithi115 ceph-mon[78593]: pgmap v10672: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T05:11:51.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:11:51 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T05:11:51.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:11:51 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T05:11:51.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:11:51 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:11:51.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:11:51 smithi115 conmon[90393]: 2022-05-14T05:11:51.276+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:11:51.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:11:51 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T05:11:51.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:11:51 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T05:11:51.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:11:51 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:11:52.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:11:52 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:11:52.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:11:52 smithi047 ceph-mon[77147]: pgmap v10673: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:11:52.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:11:52 smithi115 conmon[90393]: 2022-05-14T05:11:52.295+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:11:52.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:11:52 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:11:52.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:11:52 smithi115 ceph-mon[78593]: pgmap v10673: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:11:53.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:11:53 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:11:53.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:11:53 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 20645 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T05:11:53.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:11:53 smithi115 conmon[90393]: 2022-05-14T05:11:53.335+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:11:53.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:11:53 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:11:53.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:11:53 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 20645 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T05:11:54.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:11:54 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:11:54.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:11:54 smithi047 ceph-mon[77147]: pgmap v10674: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:11:54.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:11:54 smithi115 conmon[90393]: 2022-05-14T05:11:54.347+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:11:54.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:11:54 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:11:54.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:11:54 smithi115 ceph-mon[78593]: pgmap v10674: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:11:55.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:11:55 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T05:11:55.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:11:55 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T05:11:55.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:11:55 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:11:55.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:11:55 smithi115 conmon[90393]: 2022-05-14T05:11:55.351+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:11:55.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:11:55 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T05:11:55.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:11:55 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T05:11:55.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:11:55 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:11:56.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:11:56 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:11:56.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:11:56 smithi047 ceph-mon[77147]: pgmap v10675: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T05:11:56.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:11:56 smithi115 conmon[90393]: 2022-05-14T05:11:56.355+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:11:56.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:11:56 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:11:56.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:11:56 smithi115 ceph-mon[78593]: pgmap v10675: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T05:11:57.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:11:57 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:11:57.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:11:57 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:11:57.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:11:57 smithi115 conmon[90393]: 2022-05-14T05:11:57.369+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:11:58.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:11:58 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:11:58.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:11:58 smithi047 ceph-mon[77147]: pgmap v10676: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:11:58.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:11:58 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 20650 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T05:11:58.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:11:58 smithi115 conmon[90393]: 2022-05-14T05:11:58.372+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:11:58.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:11:58 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:11:58.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:11:58 smithi115 ceph-mon[78593]: pgmap v10676: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:11:58.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:11:58 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 20650 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T05:11:59.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:11:59 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T05:11:59.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:11:59 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T05:11:59.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:11:59 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:11:59.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:11:59 smithi115 conmon[90393]: 2022-05-14T05:11:59.355+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:11:59.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:11:59 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T05:11:59.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:11:59 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T05:11:59.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:11:59 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:12:00.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:12:00 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:12:00.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:12:00 smithi047 ceph-mon[77147]: pgmap v10677: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T05:12:00.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:12:00 smithi115 conmon[90393]: 2022-05-14T05:12:00.360+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:12:00.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:12:00 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:12:00.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:12:00 smithi115 ceph-mon[78593]: pgmap v10677: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T05:12:01.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:12:01 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:12:01.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:12:01 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:12:01.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:12:01 smithi115 conmon[90393]: 2022-05-14T05:12:01.318+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:12:02.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:12:02 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:12:02.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:12:02 smithi047 ceph-mon[77147]: pgmap v10678: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:12:02.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:12:02 smithi115 conmon[90393]: 2022-05-14T05:12:02.329+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:12:02.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:12:02 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:12:02.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:12:02 smithi115 ceph-mon[78593]: pgmap v10678: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:12:03.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:12:03 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T05:12:03.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:12:03 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T05:12:03.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:12:03 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:12:03.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:12:03 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 20655 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T05:12:03.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:12:03 smithi115 conmon[90393]: 2022-05-14T05:12:03.378+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:12:03.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:12:03 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T05:12:03.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:12:03 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T05:12:03.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:12:03 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:12:03.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:12:03 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 20655 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T05:12:04.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:12:04 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:12:04.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:12:04 smithi047 ceph-mon[77147]: pgmap v10679: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:12:04.611 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:12:04 smithi115 conmon[90393]: 2022-05-14T05:12:04.373+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:12:04.613 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:12:04 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:12:04.613 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:12:04 smithi115 ceph-mon[78593]: pgmap v10679: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:12:05.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:12:05 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:12:05.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:12:05 smithi115 conmon[90393]: 2022-05-14T05:12:05.326+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:12:05.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:12:05 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:12:06.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:12:06 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:12:06.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:12:06 smithi047 ceph-mon[77147]: pgmap v10680: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T05:12:06.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:12:06 smithi115 conmon[90393]: 2022-05-14T05:12:06.335+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:12:06.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:12:06 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:12:06.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:12:06 smithi115 ceph-mon[78593]: pgmap v10680: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T05:12:07.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:12:07 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:12:07.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:12:07 smithi115 conmon[90393]: 2022-05-14T05:12:07.313+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:12:07.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:12:07 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:12:08.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:12:08 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:12:08.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:12:08 smithi047 ceph-mon[77147]: pgmap v10681: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:12:08.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:12:08 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 20660 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T05:12:08.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:12:08 smithi115 conmon[90393]: 2022-05-14T05:12:08.337+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:12:08.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:12:08 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:12:08.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:12:08 smithi115 ceph-mon[78593]: pgmap v10681: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:12:08.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:12:08 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 20660 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T05:12:09.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:12:09 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:12:09.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:12:09 smithi115 conmon[90393]: 2022-05-14T05:12:09.362+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:12:09.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:12:09 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:12:10.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:12:10 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:12:10.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:12:10 smithi047 ceph-mon[77147]: pgmap v10682: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T05:12:10.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:12:10 smithi115 conmon[90393]: 2022-05-14T05:12:10.384+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:12:10.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:12:10 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:12:10.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:12:10 smithi115 ceph-mon[78593]: pgmap v10682: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T05:12:11.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:12:11 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T05:12:11.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:12:11 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T05:12:11.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:12:11 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:12:11.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:12:11 smithi115 conmon[90393]: 2022-05-14T05:12:11.424+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:12:11.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:12:11 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T05:12:11.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:12:11 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T05:12:11.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:12:11 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:12:12.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:12:12 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:12:12.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:12:12 smithi047 ceph-mon[77147]: pgmap v10683: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:12:12.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:12:12 smithi115 conmon[90393]: 2022-05-14T05:12:12.463+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:12:12.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:12:12 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:12:12.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:12:12 smithi115 ceph-mon[78593]: pgmap v10683: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:12:13.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:12:13 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:12:13.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:12:13 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 20665 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T05:12:13.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:12:13 smithi115 conmon[90393]: 2022-05-14T05:12:13.459+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:12:13.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:12:13 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:12:13.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:12:13 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 20665 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T05:12:14.505 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:12:14 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:12:14.506 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:12:14 smithi115 ceph-mon[78593]: pgmap v10684: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:12:14.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:12:14 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:12:14.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:12:14 smithi047 ceph-mon[77147]: pgmap v10684: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:12:14.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:12:14 smithi115 conmon[90393]: 2022-05-14T05:12:14.502+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:12:15.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:12:15 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T05:12:15.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:12:15 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T05:12:15.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:12:15 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:12:15.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:12:15 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T05:12:15.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:12:15 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T05:12:15.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:12:15 smithi115 conmon[90393]: 2022-05-14T05:12:15.467+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:12:15.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:12:15 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T05:12:15.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:12:15 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T05:12:15.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:12:15 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:12:15.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:12:15 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T05:12:15.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:12:15 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T05:12:16.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:12:16 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:12:16.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:12:16 smithi047 ceph-mon[77147]: pgmap v10685: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T05:12:16.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:12:16 smithi115 conmon[90393]: 2022-05-14T05:12:16.424+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:12:16.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:12:16 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:12:16.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:12:16 smithi115 ceph-mon[78593]: pgmap v10685: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T05:12:17.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:12:17 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:12:17.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:12:17 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:12:17.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:12:17 smithi115 conmon[90393]: 2022-05-14T05:12:17.387+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:12:18.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:12:18 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:12:18.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:12:18 smithi047 ceph-mon[77147]: pgmap v10686: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:12:18.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:12:18 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 20670 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T05:12:18.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:12:18 smithi115 conmon[90393]: 2022-05-14T05:12:18.392+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:12:18.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:12:18 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:12:18.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:12:18 smithi115 ceph-mon[78593]: pgmap v10686: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:12:18.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:12:18 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 20670 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T05:12:19.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:12:19 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T05:12:19.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:12:19 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T05:12:19.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:12:19 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:12:19.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:12:19 smithi115 conmon[90393]: 2022-05-14T05:12:19.439+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:12:19.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:12:19 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T05:12:19.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:12:19 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T05:12:19.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:12:19 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:12:20.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:12:20 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:12:20.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:12:20 smithi047 ceph-mon[77147]: pgmap v10687: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T05:12:20.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:12:20 smithi115 conmon[90393]: 2022-05-14T05:12:20.434+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:12:20.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:12:20 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:12:20.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:12:20 smithi115 ceph-mon[78593]: pgmap v10687: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T05:12:21.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:12:21 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:12:21.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:12:21 smithi115 conmon[90393]: 2022-05-14T05:12:21.454+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:12:21.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:12:21 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:12:22.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:12:22 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:12:22.544 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:12:22 smithi047 ceph-mon[77147]: pgmap v10688: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:12:22.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:12:22 smithi115 conmon[90393]: 2022-05-14T05:12:22.483+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:12:22.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:12:22 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:12:22.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:12:22 smithi115 ceph-mon[78593]: pgmap v10688: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:12:23.536 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:12:23 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T05:12:23.536 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:12:23 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T05:12:23.536 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:12:23 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:12:23.536 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:12:23 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 20675 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T05:12:23.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:12:23 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T05:12:23.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:12:23 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T05:12:23.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:12:23 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:12:23.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:12:23 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 20675 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T05:12:23.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:12:23 smithi115 conmon[90393]: 2022-05-14T05:12:23.533+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:12:24.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:12:24 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:12:24.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:12:24 smithi047 ceph-mon[77147]: pgmap v10689: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:12:24.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:12:24 smithi115 conmon[90393]: 2022-05-14T05:12:24.531+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:12:24.615 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:12:24 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:12:24.616 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:12:24 smithi115 ceph-mon[78593]: pgmap v10689: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:12:25.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:12:25 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:12:25.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:12:25 smithi115 conmon[90393]: 2022-05-14T05:12:25.513+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:12:25.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:12:25 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:12:26.565 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:12:26 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:12:26.566 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:12:26 smithi115 ceph-mon[78593]: pgmap v10690: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T05:12:26.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:12:26 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:12:26.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:12:26 smithi047 ceph-mon[77147]: pgmap v10690: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T05:12:26.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:12:26 smithi115 conmon[90393]: 2022-05-14T05:12:26.562+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:12:27.598 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:12:27 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:12:27.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:12:27 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:12:27.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:12:27 smithi115 conmon[90393]: 2022-05-14T05:12:27.594+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:12:28.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:12:28 smithi115 conmon[90393]: 2022-05-14T05:12:28.545+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:12:28.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:12:28 smithi115 ceph-mon[78593]: pgmap v10691: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:12:28.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:12:28 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:12:28.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:12:28 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 20680 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T05:12:28.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:12:28 smithi047 ceph-mon[77147]: pgmap v10691: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:12:28.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:12:28 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:12:28.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:12:28 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 20680 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T05:12:29.570 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:12:29 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:12:29.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:12:29 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:12:29.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:12:29 smithi115 conmon[90393]: 2022-05-14T05:12:29.562+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:12:30.578 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:12:30 smithi115 ceph-mon[78593]: pgmap v10692: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T05:12:30.578 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:12:30 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:12:30.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:12:30 smithi047 ceph-mon[77147]: pgmap v10692: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T05:12:30.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:12:30 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:12:30.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:12:30 smithi115 conmon[90393]: 2022-05-14T05:12:30.575+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:12:31.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:12:31 smithi115 conmon[90393]: 2022-05-14T05:12:31.548+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:12:31.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:12:31 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T05:12:31.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:12:31 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T05:12:31.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:12:31 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:12:31.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:12:31 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T05:12:31.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:12:31 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T05:12:31.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:12:31 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:12:32.575 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:12:32 smithi047 ceph-mon[77147]: pgmap v10693: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:12:32.576 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:12:32 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:12:32.579 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:12:32 smithi115 conmon[90393]: 2022-05-14T05:12:32.516+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:12:32.579 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:12:32 smithi115 ceph-mon[78593]: pgmap v10693: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:12:32.579 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:12:32 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:12:33.610 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:12:33 smithi115 conmon[90393]: 2022-05-14T05:12:33.501+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:12:33.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:12:33 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:12:33.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:12:33 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T05:12:33.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:12:33 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T05:12:33.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:12:33 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T05:12:33.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:12:33 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 20685 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T05:12:33.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:12:33 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:12:33.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:12:33 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T05:12:33.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:12:33 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T05:12:33.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:12:33 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T05:12:33.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:12:33 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 20685 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T05:12:34.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:12:34 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:12:34.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:12:34 smithi115 ceph-mon[78593]: pgmap v10694: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:12:34.610 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:12:34 smithi115 conmon[90393]: 2022-05-14T05:12:34.475+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:12:34.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:12:34 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:12:34.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:12:34 smithi047 ceph-mon[77147]: pgmap v10694: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:12:35.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:12:35 smithi115 conmon[90393]: 2022-05-14T05:12:35.451+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:12:35.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:12:35 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T05:12:35.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:12:35 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T05:12:35.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:12:35 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:12:35.634 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:12:35 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T05:12:35.635 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:12:35 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T05:12:35.635 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:12:35 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:12:36.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:12:35 smithi047 conmon[78521]: 2022-05-14T05:12:35.632+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T05:12:36.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:12:35 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T05:12:36.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:12:35 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T05:12:36.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:12:35 smithi047 conmon[78521]: yield 2022-05-14T05:12:36.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:12:35 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T05:12:36.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:12:35 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T05:12:36.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:12:35 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T05:12:36.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:12:35 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T05:12:36.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:12:35 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T05:12:36.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:12:35 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T05:12:36.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:12:35 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T05:12:36.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:12:35 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T05:12:36.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:12:35 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T05:12:36.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:12:35 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T05:12:36.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:12:35 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T05:12:36.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:12:35 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T05:12:36.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:12:35 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T05:12:36.048 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:12:35 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T05:12:36.048 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:12:35 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T05:12:36.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:12:36 smithi115 conmon[90393]: 2022-05-14T05:12:36.489+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:12:36.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:12:36 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:12:36.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:12:36 smithi115 ceph-mon[78593]: pgmap v10695: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T05:12:36.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:12:36 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T05:12:36.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T05:12:36.610 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T05:12:36.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T05:12:36.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T05:12:36.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T05:12:36.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T05:12:36.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T05:12:36.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T05:12:36.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T05:12:36.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T05:12:36.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T05:12:36.612 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T05:12:36.613 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T05:12:36.613 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T05:12:36.613 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T05:12:36.613 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T05:12:36.614 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T05:12:36.614 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T05:12:36.614 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:12:36 smithi115 ceph-mon[78593]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T05:12:36.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:12:36 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:12:36.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:12:36 smithi047 ceph-mon[77147]: pgmap v10695: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T05:12:36.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:12:36 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T05:12:36.793 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T05:12:36.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T05:12:36.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T05:12:36.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T05:12:36.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T05:12:36.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T05:12:36.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T05:12:36.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T05:12:36.797 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T05:12:36.797 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T05:12:36.797 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T05:12:36.797 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T05:12:36.798 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T05:12:36.798 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T05:12:36.798 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T05:12:36.798 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T05:12:36.799 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T05:12:36.799 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T05:12:36.799 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:12:36 smithi047 ceph-mon[77147]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T05:12:37.610 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:12:37 smithi115 conmon[90393]: 2022-05-14T05:12:37.520+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:12:37.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:12:37 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:12:37.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:12:37 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:12:38.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:12:38 smithi115 conmon[90393]: 2022-05-14T05:12:38.540+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:12:38.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:12:38 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T05:12:38.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:12:38 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T05:12:38.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:12:38 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:12:38.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:12:38 smithi115 ceph-mon[78593]: pgmap v10696: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:12:38.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:12:38 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 20690 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T05:12:38.706 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:12:38 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T05:12:38.707 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:12:38 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T05:12:38.707 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:12:38 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:12:38.707 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:12:38 smithi047 ceph-mon[77147]: pgmap v10696: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:12:38.707 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:12:38 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 20690 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T05:12:39.042 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:12:38 smithi047 conmon[78521]: 2022-05-14T05:12:38.704+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T05:12:39.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:12:38 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T05:12:39.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:12:38 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T05:12:39.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:12:38 smithi047 conmon[78521]: yield 2022-05-14T05:12:39.043 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:12:38 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T05:12:39.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:12:38 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T05:12:39.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:12:38 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T05:12:39.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:12:38 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T05:12:39.044 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:12:38 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T05:12:39.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:12:38 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T05:12:39.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:12:38 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T05:12:39.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:12:38 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T05:12:39.045 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:12:38 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T05:12:39.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:12:38 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T05:12:39.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:12:38 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T05:12:39.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:12:38 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T05:12:39.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:12:38 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T05:12:39.046 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:12:38 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T05:12:39.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:12:38 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T05:12:39.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:12:38 smithi047 conmon[78521]: 2022-05-14T05:12:38.705+0000 7f5340779700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T05:12:39.047 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:12:38 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T05:12:39.048 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:12:38 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T05:12:39.048 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:12:38 smithi047 conmon[78521]: yield 2022-05-14T05:12:39.049 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:12:38 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T05:12:39.049 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:12:38 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T05:12:39.049 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:12:38 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T05:12:39.049 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:12:38 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T05:12:39.050 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:12:38 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T05:12:39.050 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:12:38 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T05:12:39.050 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:12:38 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T05:12:39.051 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:12:38 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T05:12:39.051 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:12:38 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T05:12:39.051 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:12:38 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T05:12:39.052 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:12:38 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T05:12:39.052 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:12:38 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T05:12:39.052 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:12:38 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T05:12:39.052 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:12:38 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T05:12:39.053 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:12:38 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T05:12:39.053 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:12:38 smithi047 conmon[78521]: 2022-05-14T05:12:39.053 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:12:38 smithi047 conmon[78521]: During handling of the above exception, another exception occurred: 2022-05-14T05:12:39.054 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:12:38 smithi047 conmon[78521]: 2022-05-14T05:12:39.054 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:12:38 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T05:12:39.054 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:12:38 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T05:12:39.054 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:12:38 smithi047 conmon[78521]: return f(*arg) 2022-05-14T05:12:39.055 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:12:38 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T05:12:39.055 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:12:38 smithi047 conmon[78521]: self._write_client_files(client_files, host) 2022-05-14T05:12:39.055 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:12:38 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T05:12:39.056 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:12:38 smithi047 conmon[78521]: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T05:12:39.056 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:12:38 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T05:12:39.056 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:12:38 smithi047 conmon[78521]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T05:12:39.056 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:12:38 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T05:12:39.057 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:12:38 smithi047 conmon[78521]: return self.event_loop.get_result(coro) 2022-05-14T05:12:39.057 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:12:38 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T05:12:39.057 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:12:38 smithi047 conmon[78521]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T05:12:39.057 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:12:38 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T05:12:39.058 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:12:38 smithi047 conmon[78521]: return self.__get_result() 2022-05-14T05:12:39.058 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:12:38 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T05:12:39.058 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:12:38 smithi047 conmon[78521]: raise self._exception 2022-05-14T05:12:39.059 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:12:38 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T05:12:39.059 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:12:38 smithi047 conmon[78521]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T05:12:39.059 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:12:38 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T05:12:39.059 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:12:38 smithi047 conmon[78521]: conn = await self._remote_connection(host, addr) 2022-05-14T05:12:39.059 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:12:38 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T05:12:39.060 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:12:38 smithi047 conmon[78521]: raise 2022-05-14T05:12:39.060 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:12:38 smithi047 conmon[78521]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T05:12:39.060 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:12:38 smithi047 conmon[78521]: self.gen.throw(type, value, traceback) 2022-05-14T05:12:39.060 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:12:38 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T05:12:39.061 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:12:38 smithi047 conmon[78521]: raise OrchestratorError(msg) 2022-05-14T05:12:39.061 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:12:38 smithi047 conmon[78521]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T05:12:39.610 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:12:39 smithi115 conmon[90393]: 2022-05-14T05:12:39.507+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:12:39.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:12:39 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:12:39.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:12:39 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T05:12:39.611 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T05:12:39.613 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T05:12:39.613 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T05:12:39.613 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T05:12:39.613 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T05:12:39.614 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T05:12:39.614 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T05:12:39.614 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T05:12:39.614 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T05:12:39.615 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T05:12:39.615 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T05:12:39.615 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T05:12:39.615 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T05:12:39.616 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T05:12:39.616 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T05:12:39.616 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T05:12:39.616 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T05:12:39.617 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T05:12:39.617 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:12:39 smithi115 ceph-mon[78593]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T05:12:39.617 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T05:12:39.618 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T05:12:39.618 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T05:12:39.618 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T05:12:39.618 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T05:12:39.618 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T05:12:39.619 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T05:12:39.619 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T05:12:39.619 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T05:12:39.619 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T05:12:39.620 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T05:12:39.620 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T05:12:39.620 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T05:12:39.620 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T05:12:39.621 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T05:12:39.621 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T05:12:39.623 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T05:12:39.623 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T05:12:39.624 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T05:12:39.624 INFO:journalctl@ceph.mon.c.smithi115.stdout: During handling of the above exception, another exception occurred: 2022-05-14T05:12:39.624 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T05:12:39.625 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T05:12:39.625 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T05:12:39.625 INFO:journalctl@ceph.mon.c.smithi115.stdout: return f(*arg) 2022-05-14T05:12:39.625 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T05:12:39.626 INFO:journalctl@ceph.mon.c.smithi115.stdout: self._write_client_files(client_files, host) 2022-05-14T05:12:39.626 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T05:12:39.626 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T05:12:39.626 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T05:12:39.626 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T05:12:39.627 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T05:12:39.627 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.event_loop.get_result(coro) 2022-05-14T05:12:39.627 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T05:12:39.627 INFO:journalctl@ceph.mon.c.smithi115.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T05:12:39.628 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T05:12:39.628 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.__get_result() 2022-05-14T05:12:39.628 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T05:12:39.629 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise self._exception 2022-05-14T05:12:39.629 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T05:12:39.629 INFO:journalctl@ceph.mon.c.smithi115.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T05:12:39.629 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T05:12:39.630 INFO:journalctl@ceph.mon.c.smithi115.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T05:12:39.630 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T05:12:39.630 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise 2022-05-14T05:12:39.630 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T05:12:39.630 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.gen.throw(type, value, traceback) 2022-05-14T05:12:39.631 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T05:12:39.631 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OrchestratorError(msg) 2022-05-14T05:12:39.631 INFO:journalctl@ceph.mon.c.smithi115.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T05:12:39.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:12:39 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:12:39.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:12:39 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T05:12:39.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T05:12:39.794 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T05:12:39.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T05:12:39.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T05:12:39.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T05:12:39.795 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T05:12:39.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T05:12:39.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T05:12:39.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T05:12:39.796 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T05:12:39.797 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T05:12:39.797 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T05:12:39.798 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T05:12:39.798 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T05:12:39.799 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T05:12:39.799 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T05:12:39.800 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T05:12:39.800 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T05:12:39.801 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:12:39 smithi047 ceph-mon[77147]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T05:12:39.801 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T05:12:39.802 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T05:12:39.803 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T05:12:39.804 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T05:12:39.804 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T05:12:39.805 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T05:12:39.805 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T05:12:39.806 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T05:12:39.806 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T05:12:39.807 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T05:12:39.807 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T05:12:39.807 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T05:12:39.808 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T05:12:39.808 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T05:12:39.809 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T05:12:39.809 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T05:12:39.809 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T05:12:39.810 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T05:12:39.810 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T05:12:39.811 INFO:journalctl@ceph.mon.b.smithi047.stdout: During handling of the above exception, another exception occurred: 2022-05-14T05:12:39.811 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T05:12:39.811 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T05:12:39.812 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T05:12:39.812 INFO:journalctl@ceph.mon.b.smithi047.stdout: return f(*arg) 2022-05-14T05:12:39.813 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T05:12:39.813 INFO:journalctl@ceph.mon.b.smithi047.stdout: self._write_client_files(client_files, host) 2022-05-14T05:12:39.814 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T05:12:39.814 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T05:12:39.815 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T05:12:39.815 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T05:12:39.816 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T05:12:39.816 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.event_loop.get_result(coro) 2022-05-14T05:12:39.816 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T05:12:39.816 INFO:journalctl@ceph.mon.b.smithi047.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T05:12:39.817 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T05:12:39.817 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.__get_result() 2022-05-14T05:12:39.818 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T05:12:39.818 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise self._exception 2022-05-14T05:12:39.818 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T05:12:39.819 INFO:journalctl@ceph.mon.b.smithi047.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T05:12:39.819 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T05:12:39.819 INFO:journalctl@ceph.mon.b.smithi047.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T05:12:39.820 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T05:12:39.820 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise 2022-05-14T05:12:39.820 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T05:12:39.821 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.gen.throw(type, value, traceback) 2022-05-14T05:12:39.821 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T05:12:39.821 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OrchestratorError(msg) 2022-05-14T05:12:39.821 INFO:journalctl@ceph.mon.b.smithi047.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T05:12:40.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:12:40 smithi115 conmon[90393]: 2022-05-14T05:12:40.485+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:12:40.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:12:40 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:12:40.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:12:40 smithi115 ceph-mon[78593]: pgmap v10697: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T05:12:40.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:12:40 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:12:40.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:12:40 smithi047 ceph-mon[77147]: pgmap v10697: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T05:12:41.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:12:41 smithi115 conmon[90393]: 2022-05-14T05:12:41.466+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:12:41.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:12:41 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:12:41.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:12:41 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:12:42.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:12:42 smithi115 conmon[90393]: 2022-05-14T05:12:42.471+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:12:42.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:12:42 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T05:12:42.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:12:42 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T05:12:42.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:12:42 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:12:42.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:12:42 smithi115 ceph-mon[78593]: pgmap v10698: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:12:42.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:12:42 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T05:12:42.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:12:42 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T05:12:42.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:12:42 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:12:42.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:12:42 smithi047 ceph-mon[77147]: pgmap v10698: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:12:43.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:12:43 smithi115 conmon[90393]: 2022-05-14T05:12:43.422+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:12:43.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:12:43 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:12:43.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:12:43 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 20695 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T05:12:43.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:12:43 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:12:43.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:12:43 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 20695 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T05:12:44.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:12:44 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:12:44.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:12:44 smithi047 ceph-mon[77147]: pgmap v10699: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:12:44.860 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:12:44 smithi115 conmon[90393]: 2022-05-14T05:12:44.396+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:12:44.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:12:44 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:12:44.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:12:44 smithi115 ceph-mon[78593]: pgmap v10699: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:12:45.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:12:45 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:12:45.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:12:45 smithi115 conmon[90393]: 2022-05-14T05:12:45.371+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:12:45.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:12:45 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:12:46.610 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:12:46 smithi115 conmon[90393]: 2022-05-14T05:12:46.337+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:12:46.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:12:46 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:12:46.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:12:46 smithi115 ceph-mon[78593]: pgmap v10700: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T05:12:46.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:12:46 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:12:46.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:12:46 smithi047 ceph-mon[77147]: pgmap v10700: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T05:12:47.610 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:12:47 smithi115 conmon[90393]: 2022-05-14T05:12:47.345+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:12:47.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:12:47 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:12:47.800 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:12:47 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:12:48.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:12:48 smithi115 conmon[90393]: 2022-05-14T05:12:48.349+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:12:48.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:12:48 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:12:48.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:12:48 smithi115 ceph-mon[78593]: pgmap v10701: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:12:48.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:12:48 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 20700 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T05:12:48.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:12:48 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:12:48.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:12:48 smithi047 ceph-mon[77147]: pgmap v10701: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:12:48.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:12:48 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 20700 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T05:12:49.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:12:49 smithi115 conmon[90393]: 2022-05-14T05:12:49.342+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:12:49.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:12:49 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:12:49.810 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:12:49 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:12:50.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:12:50 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:12:50.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:12:50 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T05:12:50.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:12:50 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T05:12:50.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:12:50 smithi047 ceph-mon[77147]: pgmap v10702: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T05:12:50.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:12:50 smithi115 conmon[90393]: 2022-05-14T05:12:50.362+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:12:50.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:12:50 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:12:50.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:12:50 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T05:12:50.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:12:50 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T05:12:50.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:12:50 smithi115 ceph-mon[78593]: pgmap v10702: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T05:12:51.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:12:51 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:12:51.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:12:51 smithi115 conmon[90393]: 2022-05-14T05:12:51.369+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:12:51.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:12:51 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:12:52.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:12:52 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:12:52.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:12:52 smithi047 ceph-mon[77147]: pgmap v10703: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:12:52.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:12:52 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:12:52.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:12:52 smithi115 ceph-mon[78593]: pgmap v10703: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:12:52.860 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:12:52 smithi115 conmon[90393]: 2022-05-14T05:12:52.367+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:12:53.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:12:53 smithi115 conmon[90393]: 2022-05-14T05:12:53.341+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:12:53.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:12:53 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:12:53.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:12:53 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 20705 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T05:12:53.795 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:12:53 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:12:53.795 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:12:53 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 20705 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T05:12:54.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:12:54 smithi115 conmon[90393]: 2022-05-14T05:12:54.342+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:12:54.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:12:54 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:12:54.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:12:54 smithi115 ceph-mon[78593]: pgmap v10704: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:12:54.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:12:54 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T05:12:54.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:12:54 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T05:12:54.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:12:54 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:12:54.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:12:54 smithi047 ceph-mon[77147]: pgmap v10704: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:12:54.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:12:54 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T05:12:54.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:12:54 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T05:12:55.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:12:55 smithi115 conmon[90393]: 2022-05-14T05:12:55.311+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:12:55.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:12:55 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:12:55.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:12:55 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:12:56.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:12:56 smithi115 conmon[90393]: 2022-05-14T05:12:56.305+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:12:56.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:12:56 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:12:56.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:12:56 smithi115 ceph-mon[78593]: pgmap v10705: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T05:12:56.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:12:56 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:12:56.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:12:56 smithi047 ceph-mon[77147]: pgmap v10705: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T05:12:57.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:12:57 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:12:57.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:12:57 smithi115 conmon[90393]: 2022-05-14T05:12:57.345+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:12:57.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:12:57 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:12:58.610 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:12:58 smithi115 conmon[90393]: 2022-05-14T05:12:58.310+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:12:58.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:12:58 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:12:58.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:12:58 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T05:12:58.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:12:58 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T05:12:58.611 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:12:58 smithi115 ceph-mon[78593]: pgmap v10706: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:12:58.612 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:12:58 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 20710 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T05:12:58.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:12:58 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:12:58.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:12:58 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T05:12:58.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:12:58 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T05:12:58.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:12:58 smithi047 ceph-mon[77147]: pgmap v10706: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:12:58.795 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:12:58 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 20710 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T05:12:59.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:12:59 smithi115 conmon[90393]: 2022-05-14T05:12:59.352+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:12:59.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:12:59 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:12:59.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:12:59 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:13:00.610 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:13:00 smithi115 conmon[90393]: 2022-05-14T05:13:00.355+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:13:00.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:13:00 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:13:00.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:13:00 smithi115 ceph-mon[78593]: pgmap v10707: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T05:13:00.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:13:00 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:13:00.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:13:00 smithi047 ceph-mon[77147]: pgmap v10707: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T05:13:01.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:13:01 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:13:01.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:13:01 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:13:01.860 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:13:01 smithi115 conmon[90393]: 2022-05-14T05:13:01.375+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:13:02.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:13:02 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:13:02.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:13:02 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T05:13:02.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:13:02 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T05:13:02.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:13:02 smithi047 ceph-mon[77147]: pgmap v10708: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:13:02.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:13:02 smithi115 conmon[90393]: 2022-05-14T05:13:02.397+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:13:02.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:13:02 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:13:02.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:13:02 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T05:13:02.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:13:02 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T05:13:02.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:13:02 smithi115 ceph-mon[78593]: pgmap v10708: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:13:03.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:13:03 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:13:03.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:13:03 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 20715 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T05:13:03.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:13:03 smithi115 conmon[90393]: 2022-05-14T05:13:03.404+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:13:03.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:13:03 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:13:03.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:13:03 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 20715 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T05:13:04.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:13:04 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:13:04.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:13:04 smithi047 ceph-mon[77147]: pgmap v10709: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:13:04.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:13:04 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:13:04.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:13:04 smithi115 ceph-mon[78593]: pgmap v10709: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:13:04.862 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:13:04 smithi115 conmon[90393]: 2022-05-14T05:13:04.438+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:13:05.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:13:05 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:13:05.862 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:13:05 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:13:05.862 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:13:05 smithi115 conmon[90393]: 2022-05-14T05:13:05.405+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:13:06.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:13:06 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:13:06.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:13:06 smithi047 ceph-mon[77147]: pgmap v10710: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T05:13:06.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:13:06 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:13:06.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:13:06 smithi115 ceph-mon[78593]: pgmap v10710: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T05:13:06.860 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:13:06 smithi115 conmon[90393]: 2022-05-14T05:13:06.405+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:13:07.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:13:07 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:13:07.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:13:07 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:13:07.860 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:13:07 smithi115 conmon[90393]: 2022-05-14T05:13:07.409+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:13:08.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:13:08 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:13:08.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:13:08 smithi047 ceph-mon[77147]: pgmap v10711: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:13:08.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:13:08 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 20720 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T05:13:08.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:13:08 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:13:08.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:13:08 smithi115 ceph-mon[78593]: pgmap v10711: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:13:08.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:13:08 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 20720 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T05:13:08.860 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:13:08 smithi115 conmon[90393]: 2022-05-14T05:13:08.392+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:13:09.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:13:09 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:13:09.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:13:09 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:13:09.862 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:13:09 smithi115 conmon[90393]: 2022-05-14T05:13:09.358+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:13:10.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:13:10 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:13:10.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:13:10 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T05:13:10.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:13:10 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T05:13:10.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:13:10 smithi047 ceph-mon[77147]: pgmap v10712: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T05:13:10.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:13:10 smithi115 conmon[90393]: 2022-05-14T05:13:10.370+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:13:10.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:13:10 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:13:10.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:13:10 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T05:13:10.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:13:10 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T05:13:10.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:13:10 smithi115 ceph-mon[78593]: pgmap v10712: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T05:13:11.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:13:11 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:13:11.869 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:13:11 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:13:11.869 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:13:11 smithi115 conmon[90393]: 2022-05-14T05:13:11.377+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:13:12.613 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:13:12 smithi115 conmon[90393]: 2022-05-14T05:13:12.349+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:13:12.613 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:13:12 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:13:12.613 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:13:12 smithi115 ceph-mon[78593]: pgmap v10713: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:13:12.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:13:12 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:13:12.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:13:12 smithi047 ceph-mon[77147]: pgmap v10713: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:13:13.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:13:13 smithi115 conmon[90393]: 2022-05-14T05:13:13.351+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:13:13.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:13:13 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:13:13.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:13:13 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 20725 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T05:13:13.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:13:13 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:13:13.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:13:13 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 20725 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T05:13:14.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:13:14 smithi115 conmon[90393]: 2022-05-14T05:13:14.346+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:13:14.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:13:14 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:13:14.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:13:14 smithi115 ceph-mon[78593]: pgmap v10714: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:13:14.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:13:14 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T05:13:14.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:13:14 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T05:13:14.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:13:14 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:13:14.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:13:14 smithi047 ceph-mon[77147]: pgmap v10714: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:13:14.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:13:14 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T05:13:14.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:13:14 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T05:13:15.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:13:15 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:13:15.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:13:15 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T05:13:15.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:13:15 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T05:13:15.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:13:15 smithi115 conmon[90393]: 2022-05-14T05:13:15.360+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:13:15.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:13:15 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:13:15.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:13:15 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T05:13:15.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:13:15 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T05:13:16.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:13:16 smithi115 conmon[90393]: 2022-05-14T05:13:16.313+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:13:16.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:13:16 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:13:16.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:13:16 smithi115 ceph-mon[78593]: pgmap v10715: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T05:13:16.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:13:16 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:13:16.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:13:16 smithi047 ceph-mon[77147]: pgmap v10715: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T05:13:17.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:13:17 smithi115 conmon[90393]: 2022-05-14T05:13:17.346+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:13:17.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:13:17 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:13:17.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:13:17 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:13:18.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:13:18 smithi115 conmon[90393]: 2022-05-14T05:13:18.355+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:13:18.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:13:18 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:13:18.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:13:18 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T05:13:18.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:13:18 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T05:13:18.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:13:18 smithi115 ceph-mon[78593]: pgmap v10716: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:13:18.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:13:18 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 20730 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T05:13:18.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:13:18 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:13:18.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:13:18 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T05:13:18.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:13:18 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T05:13:18.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:13:18 smithi047 ceph-mon[77147]: pgmap v10716: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:13:18.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:13:18 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 20730 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T05:13:19.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:13:19 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:13:19.609 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:13:19 smithi115 conmon[90393]: 2022-05-14T05:13:19.350+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:13:19.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:13:19 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:13:20.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:13:20 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:13:20.794 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:13:20 smithi047 ceph-mon[77147]: pgmap v10717: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T05:13:20.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:13:20 smithi115 conmon[90393]: 2022-05-14T05:13:20.370+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:13:20.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:13:20 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:13:20.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:13:20 smithi115 ceph-mon[78593]: pgmap v10717: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T05:13:21.802 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:13:21 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:13:21.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:13:21 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:13:21.860 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:13:21 smithi115 conmon[90393]: 2022-05-14T05:13:21.405+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:13:22.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:13:22 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:13:22.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:13:22 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T05:13:22.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:13:22 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T05:13:22.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:13:22 smithi047 ceph-mon[77147]: pgmap v10718: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:13:22.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:13:22 smithi115 conmon[90393]: 2022-05-14T05:13:22.441+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:13:22.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:13:22 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:13:22.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:13:22 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T05:13:22.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:13:22 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T05:13:22.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:13:22 smithi115 ceph-mon[78593]: pgmap v10718: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:13:23.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:13:23 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:13:23.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:13:23 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 20735 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T05:13:23.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:13:23 smithi115 conmon[90393]: 2022-05-14T05:13:23.487+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:13:23.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:13:23 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:13:23.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:13:23 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 20735 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T05:13:24.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:13:24 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:13:24.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:13:24 smithi047 ceph-mon[77147]: pgmap v10719: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:13:24.860 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:13:24 smithi115 conmon[90393]: 2022-05-14T05:13:24.521+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:13:24.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:13:24 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:13:24.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:13:24 smithi115 ceph-mon[78593]: pgmap v10719: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:13:25.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:13:25 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:13:25.860 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:13:25 smithi115 conmon[90393]: 2022-05-14T05:13:25.488+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:13:25.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:13:25 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:13:26.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:13:26 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:13:26.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:13:26 smithi047 ceph-mon[77147]: pgmap v10720: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T05:13:26.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:13:26 smithi115 conmon[90393]: 2022-05-14T05:13:26.480+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:13:26.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:13:26 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:13:26.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:13:26 smithi115 ceph-mon[78593]: pgmap v10720: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T05:13:27.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:13:27 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:13:27.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:13:27 smithi115 conmon[90393]: 2022-05-14T05:13:27.529+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:13:27.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:13:27 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:13:28.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:13:28 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:13:28.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:13:28 smithi047 ceph-mon[77147]: pgmap v10721: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:13:28.793 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:13:28 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 20740 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T05:13:28.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:13:28 smithi115 conmon[90393]: 2022-05-14T05:13:28.486+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:13:28.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:13:28 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:13:28.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:13:28 smithi115 ceph-mon[78593]: pgmap v10721: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:13:28.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:13:28 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 20740 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T05:13:29.792 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:13:29 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:13:29.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:13:29 smithi115 conmon[90393]: 2022-05-14T05:13:29.484+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:13:29.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:13:29 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:13:30.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:13:30 smithi115 conmon[90393]: 2022-05-14T05:13:30.466+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:13:30.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:13:30 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:13:30.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:13:30 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T05:13:30.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:13:30 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T05:13:30.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:13:30 smithi115 ceph-mon[78593]: pgmap v10722: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T05:13:31.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:13:30 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:13:31.046 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:13:30 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T05:13:31.046 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:13:30 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T05:13:31.046 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:13:30 smithi047 ceph-mon[77147]: pgmap v10722: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T05:13:31.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:13:31 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:13:31.860 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:13:31 smithi115 conmon[90393]: 2022-05-14T05:13:31.482+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:13:32.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:13:31 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:13:32.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:13:32 smithi115 conmon[90393]: 2022-05-14T05:13:32.440+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:13:32.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:13:32 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:13:32.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:13:32 smithi115 ceph-mon[78593]: pgmap v10723: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:13:33.046 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:13:32 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:13:33.046 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:13:32 smithi047 ceph-mon[77147]: pgmap v10723: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:13:33.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:13:33 smithi115 conmon[90393]: 2022-05-14T05:13:33.453+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:13:33.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:13:33 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:13:33.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:13:33 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 20745 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T05:13:34.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:13:33 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:13:34.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:13:33 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 20745 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T05:13:34.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:13:34 smithi115 conmon[90393]: 2022-05-14T05:13:34.477+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:13:34.865 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:13:34 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:13:34.866 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:13:34 smithi115 ceph-mon[78593]: pgmap v10724: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:13:34.866 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:13:34 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T05:13:34.866 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:13:34 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T05:13:35.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:13:34 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:13:35.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:13:34 smithi047 ceph-mon[77147]: pgmap v10724: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:13:35.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:13:34 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T05:13:35.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:13:34 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T05:13:35.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:13:35 smithi115 conmon[90393]: 2022-05-14T05:13:35.434+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:13:35.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:13:35 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:13:36.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:13:35 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:13:36.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:13:36 smithi115 conmon[90393]: 2022-05-14T05:13:36.413+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:13:36.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:13:36 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:13:36.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:13:36 smithi115 ceph-mon[78593]: pgmap v10725: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T05:13:37.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:13:36 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:13:37.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:13:36 smithi047 ceph-mon[77147]: pgmap v10725: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T05:13:37.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:13:37 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:13:37.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:13:37 smithi115 conmon[90393]: 2022-05-14T05:13:37.400+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:13:38.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:13:37 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:13:38.761 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:13:38 smithi115 conmon[90393]: 2022-05-14T05:13:38.433+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:13:38.762 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:13:38 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:13:38.762 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:13:38 smithi115 ceph-mon[78593]: pgmap v10726: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:13:38.763 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:13:38 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T05:13:38.763 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:13:38 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T05:13:38.764 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:13:38 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 20750 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T05:13:38.844 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:13:38 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:13:38.844 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:13:38 smithi047 ceph-mon[77147]: pgmap v10726: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:13:38.844 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:13:38 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T05:13:38.845 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:13:38 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T05:13:38.845 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:13:38 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 20750 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T05:13:39.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:13:39 smithi047 conmon[78521]: 2022-05-14T05:13:39.183+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T05:13:39.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:13:39 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T05:13:39.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:13:39 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T05:13:39.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:13:39 smithi047 conmon[78521]: yield 2022-05-14T05:13:39.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:13:39 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T05:13:39.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:13:39 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T05:13:39.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:13:39 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T05:13:39.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:13:39 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T05:13:39.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:13:39 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T05:13:39.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:13:39 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T05:13:39.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:13:39 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T05:13:39.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:13:39 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T05:13:39.547 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:13:39 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T05:13:39.547 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:13:39 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T05:13:39.548 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:13:39 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T05:13:39.549 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:13:39 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T05:13:39.549 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:13:39 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T05:13:39.549 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:13:39 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T05:13:39.549 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:13:39 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T05:13:39.576 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:13:39 smithi115 conmon[90393]: 2022-05-14T05:13:39.424+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:13:39.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:13:39 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:13:39.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:13:39 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T05:13:39.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:13:39 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T05:13:39.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:13:39 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T05:13:40.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:13:39 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:13:40.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:13:39 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T05:13:40.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:13:39 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T05:13:40.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:13:39 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T05:13:40.862 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:13:40 smithi115 conmon[90393]: 2022-05-14T05:13:40.407+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:13:40.863 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:13:40 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T05:13:40.863 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T05:13:40.864 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T05:13:40.864 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T05:13:40.865 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T05:13:40.865 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T05:13:40.866 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T05:13:40.866 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T05:13:40.867 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T05:13:40.867 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T05:13:40.867 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T05:13:40.868 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T05:13:40.868 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T05:13:40.869 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T05:13:40.869 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T05:13:40.870 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T05:13:40.871 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T05:13:40.871 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T05:13:40.872 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T05:13:40.872 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:13:40 smithi115 ceph-mon[78593]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T05:13:40.872 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:13:40 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:13:40.873 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:13:40 smithi115 ceph-mon[78593]: pgmap v10727: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T05:13:41.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:13:40 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T05:13:41.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T05:13:41.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T05:13:41.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T05:13:41.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T05:13:41.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T05:13:41.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T05:13:41.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T05:13:41.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T05:13:41.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T05:13:41.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T05:13:41.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T05:13:41.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T05:13:41.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T05:13:41.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T05:13:41.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T05:13:41.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T05:13:41.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T05:13:41.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T05:13:41.047 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:13:40 smithi047 ceph-mon[77147]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T05:13:41.048 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:13:40 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:13:41.049 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:13:40 smithi047 ceph-mon[77147]: pgmap v10727: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T05:13:41.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:13:41 smithi115 conmon[90393]: 2022-05-14T05:13:41.390+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:13:41.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:13:41 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:13:42.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:13:41 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:13:42.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:13:42 smithi047 conmon[78521]: 2022-05-14T05:13:42.255+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T05:13:42.543 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:13:42 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T05:13:42.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:13:42 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T05:13:42.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:13:42 smithi047 conmon[78521]: yield 2022-05-14T05:13:42.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:13:42 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T05:13:42.544 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:13:42 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T05:13:42.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:13:42 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T05:13:42.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:13:42 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T05:13:42.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:13:42 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T05:13:42.545 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:13:42 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T05:13:42.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:13:42 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T05:13:42.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:13:42 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T05:13:42.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:13:42 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T05:13:42.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:13:42 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T05:13:42.546 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:13:42 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T05:13:42.547 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:13:42 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T05:13:42.547 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:13:42 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T05:13:42.547 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:13:42 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T05:13:42.547 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:13:42 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T05:13:42.548 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:13:42 smithi047 conmon[78521]: 2022-05-14T05:13:42.256+0000 7f533ff78700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T05:13:42.548 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:13:42 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T05:13:42.548 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:13:42 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T05:13:42.549 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:13:42 smithi047 conmon[78521]: yield 2022-05-14T05:13:42.549 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:13:42 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T05:13:42.552 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:13:42 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T05:13:42.553 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:13:42 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T05:13:42.553 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:13:42 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T05:13:42.553 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:13:42 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T05:13:42.553 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:13:42 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T05:13:42.554 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:13:42 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T05:13:42.554 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:13:42 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T05:13:42.554 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:13:42 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T05:13:42.555 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:13:42 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T05:13:42.555 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:13:42 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T05:13:42.555 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:13:42 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T05:13:42.555 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:13:42 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T05:13:42.556 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:13:42 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T05:13:42.556 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:13:42 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T05:13:42.556 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:13:42 smithi047 conmon[78521]: 2022-05-14T05:13:42.556 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:13:42 smithi047 conmon[78521]: During handling of the above exception, another exception occurred: 2022-05-14T05:13:42.557 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:13:42 smithi047 conmon[78521]: 2022-05-14T05:13:42.557 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:13:42 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T05:13:42.557 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:13:42 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T05:13:42.557 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:13:42 smithi047 conmon[78521]: return f(*arg) 2022-05-14T05:13:42.557 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:13:42 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T05:13:42.558 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:13:42 smithi047 conmon[78521]: self._write_client_files(client_files, host) 2022-05-14T05:13:42.558 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:13:42 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T05:13:42.559 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:13:42 smithi047 conmon[78521]: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T05:13:42.559 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:13:42 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T05:13:42.559 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:13:42 smithi047 conmon[78521]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T05:13:42.559 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:13:42 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T05:13:42.560 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:13:42 smithi047 conmon[78521]: return self.event_loop.get_result(coro) 2022-05-14T05:13:42.560 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:13:42 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T05:13:42.560 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:13:42 smithi047 conmon[78521]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T05:13:42.560 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:13:42 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T05:13:42.561 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:13:42 smithi047 conmon[78521]: return self.__get_result() 2022-05-14T05:13:42.561 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:13:42 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T05:13:42.561 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:13:42 smithi047 conmon[78521]: raise self._exception 2022-05-14T05:13:42.561 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:13:42 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T05:13:42.562 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:13:42 smithi047 conmon[78521]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T05:13:42.562 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:13:42 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T05:13:42.562 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:13:42 smithi047 conmon[78521]: conn = await self._remote_connection(host, addr) 2022-05-14T05:13:42.563 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:13:42 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T05:13:42.563 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:13:42 smithi047 conmon[78521]: raise 2022-05-14T05:13:42.563 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:13:42 smithi047 conmon[78521]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T05:13:42.563 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:13:42 smithi047 conmon[78521]: self.gen.throw(type, value, traceback) 2022-05-14T05:13:42.563 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:13:42 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T05:13:42.564 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:13:42 smithi047 conmon[78521]: raise OrchestratorError(msg) 2022-05-14T05:13:42.564 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:13:42 smithi047 conmon[78521]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T05:13:42.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:13:42 smithi115 conmon[90393]: 2022-05-14T05:13:42.379+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:13:42.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:13:42 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:13:42.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:13:42 smithi115 ceph-mon[78593]: pgmap v10728: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:13:42.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:13:42 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T05:13:42.862 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:13:42 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T05:13:43.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:13:42 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:13:43.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:13:42 smithi047 ceph-mon[77147]: pgmap v10728: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:13:43.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:13:42 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T05:13:43.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:13:42 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T05:13:43.861 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:13:43 smithi115 conmon[90393]: 2022-05-14T05:13:43.418+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:13:43.862 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:13:43 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T05:13:43.862 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T05:13:43.863 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T05:13:43.863 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T05:13:43.864 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T05:13:43.865 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T05:13:43.865 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T05:13:43.865 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T05:13:43.866 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T05:13:43.866 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T05:13:43.866 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T05:13:43.866 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T05:13:43.867 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T05:13:43.867 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T05:13:43.867 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T05:13:43.867 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T05:13:43.867 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T05:13:43.868 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T05:13:43.868 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T05:13:43.868 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:13:43 smithi115 ceph-mon[78593]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T05:13:43.868 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T05:13:43.869 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T05:13:43.869 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T05:13:43.869 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T05:13:43.869 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T05:13:43.870 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T05:13:43.870 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T05:13:43.870 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T05:13:43.870 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T05:13:43.871 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T05:13:43.871 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T05:13:43.871 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T05:13:43.872 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T05:13:43.872 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T05:13:43.872 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T05:13:43.873 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T05:13:43.873 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T05:13:43.874 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T05:13:43.874 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T05:13:43.874 INFO:journalctl@ceph.mon.c.smithi115.stdout: During handling of the above exception, another exception occurred: 2022-05-14T05:13:43.874 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T05:13:43.874 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T05:13:43.875 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T05:13:43.875 INFO:journalctl@ceph.mon.c.smithi115.stdout: return f(*arg) 2022-05-14T05:13:43.875 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T05:13:43.876 INFO:journalctl@ceph.mon.c.smithi115.stdout: self._write_client_files(client_files, host) 2022-05-14T05:13:43.876 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T05:13:43.876 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T05:13:43.876 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T05:13:43.877 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T05:13:43.877 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T05:13:43.877 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.event_loop.get_result(coro) 2022-05-14T05:13:43.877 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T05:13:43.878 INFO:journalctl@ceph.mon.c.smithi115.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T05:13:43.878 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T05:13:43.884 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.__get_result() 2022-05-14T05:13:43.885 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T05:13:43.885 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise self._exception 2022-05-14T05:13:43.885 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T05:13:43.886 INFO:journalctl@ceph.mon.c.smithi115.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T05:13:43.886 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T05:13:43.887 INFO:journalctl@ceph.mon.c.smithi115.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T05:13:43.887 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T05:13:43.888 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise 2022-05-14T05:13:43.888 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T05:13:43.888 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.gen.throw(type, value, traceback) 2022-05-14T05:13:43.889 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T05:13:43.891 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OrchestratorError(msg) 2022-05-14T05:13:43.891 INFO:journalctl@ceph.mon.c.smithi115.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T05:13:43.892 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:13:43 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:13:43.892 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:13:43 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 20755 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T05:13:44.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:13:43 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T05:13:44.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T05:13:44.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T05:13:44.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T05:13:44.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T05:13:44.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T05:13:44.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T05:13:44.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T05:13:44.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T05:13:44.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T05:13:44.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T05:13:44.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T05:13:44.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T05:13:44.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T05:13:44.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T05:13:44.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T05:13:44.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T05:13:44.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T05:13:44.049 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T05:13:44.049 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:13:43 smithi047 ceph-mon[77147]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T05:13:44.049 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T05:13:44.050 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T05:13:44.050 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T05:13:44.050 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T05:13:44.051 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T05:13:44.051 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T05:13:44.051 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T05:13:44.052 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T05:13:44.052 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T05:13:44.053 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T05:13:44.053 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T05:13:44.053 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T05:13:44.054 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T05:13:44.054 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T05:13:44.055 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T05:13:44.055 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T05:13:44.055 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T05:13:44.056 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T05:13:44.056 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T05:13:44.057 INFO:journalctl@ceph.mon.b.smithi047.stdout: During handling of the above exception, another exception occurred: 2022-05-14T05:13:44.057 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T05:13:44.057 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T05:13:44.058 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T05:13:44.058 INFO:journalctl@ceph.mon.b.smithi047.stdout: return f(*arg) 2022-05-14T05:13:44.059 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T05:13:44.059 INFO:journalctl@ceph.mon.b.smithi047.stdout: self._write_client_files(client_files, host) 2022-05-14T05:13:44.059 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T05:13:44.060 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T05:13:44.060 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T05:13:44.060 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T05:13:44.061 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T05:13:44.061 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.event_loop.get_result(coro) 2022-05-14T05:13:44.062 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T05:13:44.071 INFO:journalctl@ceph.mon.b.smithi047.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T05:13:44.071 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T05:13:44.072 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.__get_result() 2022-05-14T05:13:44.072 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T05:13:44.073 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise self._exception 2022-05-14T05:13:44.073 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T05:13:44.074 INFO:journalctl@ceph.mon.b.smithi047.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T05:13:44.077 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T05:13:44.078 INFO:journalctl@ceph.mon.b.smithi047.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T05:13:44.078 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T05:13:44.078 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise 2022-05-14T05:13:44.079 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T05:13:44.079 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.gen.throw(type, value, traceback) 2022-05-14T05:13:44.080 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T05:13:44.080 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OrchestratorError(msg) 2022-05-14T05:13:44.080 INFO:journalctl@ceph.mon.b.smithi047.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T05:13:44.081 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:13:43 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:13:44.081 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:13:43 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 20755 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T05:13:44.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:13:44 smithi115 conmon[90393]: 2022-05-14T05:13:44.429+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:13:44.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:13:44 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:13:44.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:13:44 smithi115 ceph-mon[78593]: pgmap v10729: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:13:45.046 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:13:44 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:13:45.046 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:13:44 smithi047 ceph-mon[77147]: pgmap v10729: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:13:45.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:13:45 smithi115 conmon[90393]: 2022-05-14T05:13:45.392+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:13:45.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:13:45 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:13:46.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:13:45 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:13:46.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:13:46 smithi115 conmon[90393]: 2022-05-14T05:13:46.429+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:13:46.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:13:46 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:13:46.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:13:46 smithi115 ceph-mon[78593]: pgmap v10730: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T05:13:47.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:13:46 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:13:47.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:13:46 smithi047 ceph-mon[77147]: pgmap v10730: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T05:13:47.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:13:47 smithi115 conmon[90393]: 2022-05-14T05:13:47.468+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:13:47.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:13:47 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:13:48.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:13:47 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:13:48.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:13:48 smithi115 conmon[90393]: 2022-05-14T05:13:48.444+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:13:48.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:13:48 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:13:48.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:13:48 smithi115 ceph-mon[78593]: pgmap v10731: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:13:48.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:13:48 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 20760 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T05:13:49.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:13:48 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:13:49.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:13:48 smithi047 ceph-mon[77147]: pgmap v10731: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:13:49.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:13:48 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 20760 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T05:13:49.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:13:49 smithi115 conmon[90393]: 2022-05-14T05:13:49.466+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:13:49.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:13:49 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:13:50.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:13:49 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:13:50.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:13:50 smithi115 conmon[90393]: 2022-05-14T05:13:50.474+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:13:50.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:13:50 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:13:50.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:13:50 smithi115 ceph-mon[78593]: pgmap v10732: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T05:13:50.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:13:50 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T05:13:50.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:13:50 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T05:13:51.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:13:50 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:13:51.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:13:50 smithi047 ceph-mon[77147]: pgmap v10732: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T05:13:51.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:13:50 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T05:13:51.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:13:50 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T05:13:51.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:13:51 smithi115 conmon[90393]: 2022-05-14T05:13:51.481+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:13:51.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:13:51 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:13:52.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:13:51 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:13:52.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:13:52 smithi115 conmon[90393]: 2022-05-14T05:13:52.512+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:13:52.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:13:52 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:13:52.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:13:52 smithi115 ceph-mon[78593]: pgmap v10733: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:13:53.047 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:13:52 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:13:53.047 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:13:52 smithi047 ceph-mon[77147]: pgmap v10733: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:13:53.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:13:53 smithi115 conmon[90393]: 2022-05-14T05:13:53.484+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:13:53.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:13:53 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:13:53.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:13:53 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 20765 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T05:13:54.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:13:53 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:13:54.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:13:53 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 20765 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T05:13:54.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:13:54 smithi115 conmon[90393]: 2022-05-14T05:13:54.515+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:13:54.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:13:54 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:13:54.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:13:54 smithi115 ceph-mon[78593]: pgmap v10734: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:13:54.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:13:54 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T05:13:54.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:13:54 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T05:13:55.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:13:54 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:13:55.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:13:54 smithi047 ceph-mon[77147]: pgmap v10734: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:13:55.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:13:54 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T05:13:55.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:13:54 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T05:13:55.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:13:55 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:13:55.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:13:55 smithi115 conmon[90393]: 2022-05-14T05:13:55.474+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:13:56.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:13:55 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:13:56.861 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:13:56 smithi115 conmon[90393]: 2022-05-14T05:13:56.439+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:13:56.862 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:13:56 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:13:56.862 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:13:56 smithi115 ceph-mon[78593]: pgmap v10735: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T05:13:57.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:13:56 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:13:57.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:13:56 smithi047 ceph-mon[77147]: pgmap v10735: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T05:13:57.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:13:57 smithi115 conmon[90393]: 2022-05-14T05:13:57.476+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:13:57.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:13:57 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:13:58.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:13:57 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:13:58.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:13:58 smithi115 conmon[90393]: 2022-05-14T05:13:58.513+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:13:58.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:13:58 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:13:58.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:13:58 smithi115 ceph-mon[78593]: pgmap v10736: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:13:58.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:13:58 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T05:13:58.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:13:58 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T05:13:58.861 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:13:58 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 20770 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T05:13:59.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:13:58 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:13:59.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:13:58 smithi047 ceph-mon[77147]: pgmap v10736: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:13:59.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:13:58 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T05:13:59.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:13:58 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T05:13:59.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:13:58 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 20770 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T05:13:59.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:13:59 smithi115 conmon[90393]: 2022-05-14T05:13:59.493+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:13:59.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:13:59 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:14:00.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:13:59 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:14:00.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:14:00 smithi115 conmon[90393]: 2022-05-14T05:14:00.518+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:14:00.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:14:00 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:14:00.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:14:00 smithi115 ceph-mon[78593]: pgmap v10737: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T05:14:01.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:14:00 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:14:01.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:14:00 smithi047 ceph-mon[77147]: pgmap v10737: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T05:14:01.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:14:01 smithi115 conmon[90393]: 2022-05-14T05:14:01.526+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:14:01.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:14:01 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:14:02.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:14:01 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:14:02.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:14:02 smithi115 conmon[90393]: 2022-05-14T05:14:02.576+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:14:02.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:14:02 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:14:02.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:14:02 smithi115 ceph-mon[78593]: pgmap v10738: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:14:02.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:14:02 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T05:14:02.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:14:02 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T05:14:03.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:14:02 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:14:03.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:14:02 smithi047 ceph-mon[77147]: pgmap v10738: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:14:03.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:14:02 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T05:14:03.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:14:02 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T05:14:03.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:14:03 smithi115 conmon[90393]: 2022-05-14T05:14:03.604+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:14:03.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:14:03 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:14:03.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:14:03 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 20775 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T05:14:04.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:14:03 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:14:04.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:14:03 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 20775 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T05:14:05.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:14:04 smithi047 ceph-mon[77147]: pgmap v10739: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:14:05.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:14:04 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:14:05.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:14:04 smithi115 conmon[90393]: 2022-05-14T05:14:04.619+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:14:05.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:14:04 smithi115 ceph-mon[78593]: pgmap v10739: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:14:05.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:14:04 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:14:05.862 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:14:05 smithi115 conmon[90393]: 2022-05-14T05:14:05.585+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:14:05.862 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:14:05 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:14:06.046 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:14:05 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:14:06.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:14:06 smithi115 conmon[90393]: 2022-05-14T05:14:06.560+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:14:06.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:14:06 smithi115 ceph-mon[78593]: pgmap v10740: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T05:14:06.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:14:06 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:14:07.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:14:06 smithi047 ceph-mon[77147]: pgmap v10740: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T05:14:07.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:14:06 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:14:07.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:14:07 smithi115 conmon[90393]: 2022-05-14T05:14:07.599+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:14:07.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:14:07 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:14:08.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:14:07 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:14:09.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:14:08 smithi047 ceph-mon[77147]: pgmap v10741: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:14:09.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:14:08 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:14:09.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:14:08 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 20780 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T05:14:09.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:14:08 smithi115 conmon[90393]: 2022-05-14T05:14:08.612+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:14:09.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:14:08 smithi115 ceph-mon[78593]: pgmap v10741: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:14:09.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:14:08 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:14:09.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:14:08 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 20780 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T05:14:09.863 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:14:09 smithi115 conmon[90393]: 2022-05-14T05:14:09.568+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:14:09.863 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:14:09 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:14:10.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:14:09 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:14:10.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:14:10 smithi115 conmon[90393]: 2022-05-14T05:14:10.598+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:14:10.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:14:10 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:14:10.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:14:10 smithi115 ceph-mon[78593]: pgmap v10742: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T05:14:10.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:14:10 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T05:14:10.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:14:10 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T05:14:11.048 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:14:10 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:14:11.049 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:14:10 smithi047 ceph-mon[77147]: pgmap v10742: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T05:14:11.049 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:14:10 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T05:14:11.049 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:14:10 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T05:14:12.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:14:11 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:14:12.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:14:11 smithi115 conmon[90393]: 2022-05-14T05:14:11.629+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:14:12.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:14:11 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:14:12.861 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:14:12 smithi115 conmon[90393]: 2022-05-14T05:14:12.590+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:14:12.862 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:14:12 smithi115 ceph-mon[78593]: pgmap v10743: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:14:12.862 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:14:12 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:14:13.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:14:12 smithi047 ceph-mon[77147]: pgmap v10743: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:14:13.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:14:12 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:14:13.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:14:13 smithi115 conmon[90393]: 2022-05-14T05:14:13.600+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:14:13.859 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:14:13 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:14:13.860 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:14:13 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 20785 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T05:14:14.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:14:13 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:14:14.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:14:13 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 20785 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T05:14:15.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:14:14 smithi047 ceph-mon[77147]: pgmap v10744: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:14:15.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:14:14 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:14:15.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:14:14 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T05:14:15.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:14:14 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T05:14:15.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:14:14 smithi115 conmon[90393]: 2022-05-14T05:14:14.615+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:14:15.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:14:14 smithi115 ceph-mon[78593]: pgmap v10744: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:14:15.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:14:14 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:14:15.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:14:14 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T05:14:15.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:14:14 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T05:14:16.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:14:15 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:14:16.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:14:15 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T05:14:16.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:14:15 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T05:14:16.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:14:15 smithi115 conmon[90393]: 2022-05-14T05:14:15.613+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:14:16.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:14:15 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:14:16.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:14:15 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T05:14:16.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:14:15 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T05:14:17.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:14:16 smithi047 ceph-mon[77147]: pgmap v10745: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T05:14:17.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:14:16 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:14:17.110 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:14:16 smithi115 conmon[90393]: 2022-05-14T05:14:16.637+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:14:17.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:14:16 smithi115 ceph-mon[78593]: pgmap v10745: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T05:14:17.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:14:16 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:14:18.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:14:17 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:14:18.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:14:17 smithi115 conmon[90393]: 2022-05-14T05:14:17.666+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:14:18.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:14:17 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:14:19.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:14:18 smithi047 ceph-mon[77147]: pgmap v10746: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:14:19.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:14:18 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:14:19.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:14:18 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T05:14:19.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:14:18 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T05:14:19.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:14:18 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 20790 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T05:14:19.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:14:18 smithi115 conmon[90393]: 2022-05-14T05:14:18.681+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:14:19.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:14:18 smithi115 ceph-mon[78593]: pgmap v10746: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:14:19.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:14:18 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:14:19.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:14:18 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T05:14:19.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:14:18 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T05:14:19.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:14:18 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 20790 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T05:14:20.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:14:19 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:14:20.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:14:19 smithi115 conmon[90393]: 2022-05-14T05:14:19.715+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:14:20.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:14:19 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:14:21.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:14:20 smithi047 ceph-mon[77147]: pgmap v10747: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T05:14:21.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:14:20 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:14:21.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:14:20 smithi115 conmon[90393]: 2022-05-14T05:14:20.753+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:14:21.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:14:20 smithi115 ceph-mon[78593]: pgmap v10747: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T05:14:21.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:14:20 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:14:22.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:14:21 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:14:22.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:14:21 smithi115 conmon[90393]: 2022-05-14T05:14:21.724+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:14:22.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:14:21 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:14:23.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:14:22 smithi047 ceph-mon[77147]: pgmap v10748: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:14:23.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:14:22 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T05:14:23.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:14:22 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T05:14:23.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:14:22 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:14:23.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:14:22 smithi115 conmon[90393]: 2022-05-14T05:14:22.747+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:14:23.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:14:22 smithi115 ceph-mon[78593]: pgmap v10748: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:14:23.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:14:22 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T05:14:23.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:14:22 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T05:14:23.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:14:22 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:14:24.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:14:23 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:14:24.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:14:23 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 20795 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T05:14:24.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:14:23 smithi115 conmon[90393]: 2022-05-14T05:14:23.790+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:14:24.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:14:23 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:14:24.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:14:23 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 20795 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T05:14:25.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:14:24 smithi047 ceph-mon[77147]: pgmap v10749: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:14:25.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:14:24 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:14:25.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:14:24 smithi115 ceph-mon[78593]: pgmap v10749: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:14:25.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:14:24 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:14:25.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:14:24 smithi115 conmon[90393]: 2022-05-14T05:14:24.754+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:14:26.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:14:25 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:14:26.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:14:25 smithi115 conmon[90393]: 2022-05-14T05:14:25.729+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:14:26.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:14:25 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:14:27.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:14:26 smithi047 ceph-mon[77147]: pgmap v10750: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T05:14:27.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:14:26 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:14:27.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:14:26 smithi115 ceph-mon[78593]: pgmap v10750: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T05:14:27.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:14:26 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:14:27.110 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:14:26 smithi115 conmon[90393]: 2022-05-14T05:14:26.774+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:14:28.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:14:27 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:14:28.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:14:27 smithi115 conmon[90393]: 2022-05-14T05:14:27.754+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:14:28.146 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:14:27 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:14:29.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:14:28 smithi047 ceph-mon[77147]: pgmap v10751: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:14:29.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:14:28 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:14:29.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:14:28 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 20800 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T05:14:29.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:14:28 smithi115 conmon[90393]: 2022-05-14T05:14:28.785+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:14:29.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:14:28 smithi115 ceph-mon[78593]: pgmap v10751: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:14:29.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:14:28 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:14:29.112 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:14:28 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 20800 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T05:14:30.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:14:29 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:14:30.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:14:29 smithi115 conmon[90393]: 2022-05-14T05:14:29.784+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:14:30.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:14:29 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:14:31.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:14:30 smithi047 ceph-mon[77147]: pgmap v10752: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T05:14:31.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:14:30 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T05:14:31.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:14:30 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T05:14:31.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:14:30 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:14:31.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:14:30 smithi115 conmon[90393]: 2022-05-14T05:14:30.736+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:14:31.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:14:30 smithi115 ceph-mon[78593]: pgmap v10752: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T05:14:31.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:14:30 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T05:14:31.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:14:30 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T05:14:31.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:14:30 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:14:32.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:14:31 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:14:32.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:14:31 smithi115 conmon[90393]: 2022-05-14T05:14:31.707+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:14:32.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:14:31 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:14:33.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:14:32 smithi047 ceph-mon[77147]: pgmap v10753: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:14:33.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:14:32 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:14:33.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:14:32 smithi115 conmon[90393]: 2022-05-14T05:14:32.660+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:14:33.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:14:32 smithi115 ceph-mon[78593]: pgmap v10753: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:14:33.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:14:32 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:14:34.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:14:33 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:14:34.045 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:14:33 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 20805 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T05:14:34.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:14:33 smithi115 conmon[90393]: 2022-05-14T05:14:33.698+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:14:34.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:14:33 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:14:34.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:14:33 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 20805 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T05:14:35.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:14:34 smithi047 ceph-mon[77147]: pgmap v10754: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:14:35.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:14:34 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:14:35.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:14:34 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T05:14:35.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:14:34 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T05:14:35.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:14:34 smithi115 conmon[90393]: 2022-05-14T05:14:34.709+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:14:35.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:14:34 smithi115 ceph-mon[78593]: pgmap v10754: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:14:35.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:14:34 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:14:35.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:14:34 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T05:14:35.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:14:34 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T05:14:36.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:14:35 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:14:36.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:14:35 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:14:36.111 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:14:35 smithi115 conmon[90393]: 2022-05-14T05:14:35.707+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:14:37.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:14:36 smithi047 ceph-mon[77147]: pgmap v10755: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T05:14:37.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:14:36 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:14:37.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:14:36 smithi115 conmon[90393]: 2022-05-14T05:14:36.745+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:14:37.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:14:36 smithi115 ceph-mon[78593]: pgmap v10755: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T05:14:37.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:14:36 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:14:38.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:14:37 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:14:38.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:14:37 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:14:38.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:14:37 smithi115 conmon[90393]: 2022-05-14T05:14:37.746+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:14:39.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:14:38 smithi047 ceph-mon[77147]: pgmap v10756: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:14:39.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:14:38 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T05:14:39.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:14:38 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T05:14:39.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:14:38 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:14:39.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:14:38 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 20810 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T05:14:39.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:14:38 smithi115 conmon[90393]: 2022-05-14T05:14:38.701+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:14:39.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:14:38 smithi115 ceph-mon[78593]: pgmap v10756: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:14:39.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:14:38 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T05:14:39.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:14:38 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T05:14:39.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:14:38 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:14:39.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:14:38 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 20810 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T05:14:40.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:14:39 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:14:40.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:14:39 smithi115 conmon[90393]: 2022-05-14T05:14:39.712+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:14:40.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:14:39 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:14:41.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:14:40 smithi047 ceph-mon[77147]: pgmap v10757: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T05:14:41.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:14:40 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:14:41.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:14:40 smithi115 ceph-mon[78593]: pgmap v10757: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T05:14:41.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:14:40 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:14:41.110 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:14:40 smithi115 conmon[90393]: 2022-05-14T05:14:40.760+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:14:42.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:14:41 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:14:42.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:14:41 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:14:42.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:14:41 smithi115 conmon[90393]: 2022-05-14T05:14:41.715+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:14:43.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:14:42 smithi047 ceph-mon[77147]: pgmap v10758: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:14:43.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:14:42 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:14:43.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:14:42 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T05:14:43.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:14:42 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T05:14:43.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:14:42 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T05:14:43.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:14:42 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T05:14:43.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:14:42 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T05:14:43.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:14:42 smithi115 conmon[90393]: 2022-05-14T05:14:42.721+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:14:43.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:14:42 smithi115 ceph-mon[78593]: pgmap v10758: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:14:43.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:14:42 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:14:43.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:14:42 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T05:14:43.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:14:42 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T05:14:43.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:14:42 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T05:14:43.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:14:42 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T05:14:43.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:14:42 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T05:14:44.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:14:43 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:14:44.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:14:43 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 20815 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T05:14:44.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:14:43 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:14:44.113 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:14:43 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 20815 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T05:14:44.113 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:14:43 smithi115 conmon[90393]: 2022-05-14T05:14:43.758+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:14:45.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:14:44 smithi047 ceph-mon[77147]: pgmap v10759: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:14:45.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:14:44 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:14:45.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:14:44 smithi115 ceph-mon[78593]: pgmap v10759: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:14:45.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:14:44 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:14:45.110 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:14:44 smithi115 conmon[90393]: 2022-05-14T05:14:44.780+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:14:45.764 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:14:45 smithi047 conmon[78521]: 2022-05-14T05:14:45.359+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T05:14:45.765 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:14:45 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T05:14:45.765 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:14:45 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T05:14:45.765 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:14:45 smithi047 conmon[78521]: yield 2022-05-14T05:14:45.766 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:14:45 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T05:14:45.766 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:14:45 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T05:14:45.766 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:14:45 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T05:14:45.766 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:14:45 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T05:14:45.767 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:14:45 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T05:14:45.767 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:14:45 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T05:14:45.767 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:14:45 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T05:14:45.767 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:14:45 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T05:14:45.767 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:14:45 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T05:14:45.768 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:14:45 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T05:14:45.768 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:14:45 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T05:14:45.768 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:14:45 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T05:14:45.768 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:14:45 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T05:14:45.768 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:14:45 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T05:14:45.769 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:14:45 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T05:14:46.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:14:45 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:14:46.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:14:45 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:14:46.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:14:45 smithi115 conmon[90393]: 2022-05-14T05:14:45.754+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:14:47.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:14:46 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T05:14:47.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T05:14:47.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T05:14:47.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T05:14:47.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T05:14:47.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T05:14:47.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T05:14:47.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T05:14:47.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T05:14:47.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T05:14:47.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T05:14:47.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T05:14:47.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T05:14:47.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T05:14:47.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T05:14:47.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T05:14:47.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T05:14:47.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T05:14:47.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T05:14:47.047 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:14:46 smithi047 ceph-mon[77147]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T05:14:47.047 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:14:46 smithi047 ceph-mon[77147]: pgmap v10760: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T05:14:47.047 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:14:46 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:14:47.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:14:46 smithi115 conmon[90393]: 2022-05-14T05:14:46.796+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:14:47.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:14:46 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T05:14:47.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T05:14:47.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T05:14:47.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T05:14:47.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T05:14:47.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T05:14:47.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T05:14:47.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T05:14:47.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T05:14:47.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T05:14:47.113 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T05:14:47.113 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T05:14:47.113 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T05:14:47.113 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T05:14:47.114 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T05:14:47.114 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T05:14:47.114 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T05:14:47.114 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T05:14:47.114 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T05:14:47.115 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:14:46 smithi115 ceph-mon[78593]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T05:14:47.115 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:14:46 smithi115 ceph-mon[78593]: pgmap v10760: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T05:14:47.115 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:14:46 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:14:48.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:14:47 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:14:48.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:14:47 smithi115 conmon[90393]: 2022-05-14T05:14:47.846+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:14:48.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:14:47 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:14:48.774 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:14:48 smithi047 conmon[78521]: 2022-05-14T05:14:48.431+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T05:14:48.774 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:14:48 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T05:14:48.774 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:14:48 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T05:14:48.775 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:14:48 smithi047 conmon[78521]: yield 2022-05-14T05:14:48.775 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:14:48 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T05:14:48.775 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:14:48 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T05:14:48.775 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:14:48 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T05:14:48.776 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:14:48 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T05:14:48.776 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:14:48 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T05:14:48.776 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:14:48 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T05:14:48.776 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:14:48 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T05:14:48.777 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:14:48 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T05:14:48.777 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:14:48 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T05:14:48.777 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:14:48 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T05:14:48.777 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:14:48 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T05:14:48.778 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:14:48 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T05:14:48.778 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:14:48 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T05:14:48.778 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:14:48 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T05:14:48.778 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:14:48 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T05:14:48.778 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:14:48 smithi047 conmon[78521]: 2022-05-14T05:14:48.432+0000 7f533ef76700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T05:14:48.779 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:14:48 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T05:14:48.779 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:14:48 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T05:14:48.782 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:14:48 smithi047 conmon[78521]: yield 2022-05-14T05:14:48.782 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:14:48 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T05:14:48.782 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:14:48 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T05:14:48.783 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:14:48 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T05:14:48.783 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:14:48 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T05:14:48.783 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:14:48 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T05:14:48.784 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:14:48 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T05:14:48.784 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:14:48 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T05:14:48.784 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:14:48 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T05:14:48.784 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:14:48 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T05:14:48.785 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:14:48 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T05:14:48.785 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:14:48 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T05:14:48.785 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:14:48 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T05:14:48.786 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:14:48 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T05:14:48.786 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:14:48 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T05:14:48.786 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:14:48 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T05:14:48.787 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:14:48 smithi047 conmon[78521]: 2022-05-14T05:14:48.787 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:14:48 smithi047 conmon[78521]: During handling of the above exception, another exception occurred: 2022-05-14T05:14:48.788 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:14:48 smithi047 conmon[78521]: 2022-05-14T05:14:48.789 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:14:48 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T05:14:48.790 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:14:48 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T05:14:48.791 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:14:48 smithi047 conmon[78521]: return f(*arg) 2022-05-14T05:14:48.792 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:14:48 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T05:14:48.792 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:14:48 smithi047 conmon[78521]: self._write_client_files(client_files, host) 2022-05-14T05:14:48.792 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:14:48 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T05:14:48.793 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:14:48 smithi047 conmon[78521]: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T05:14:48.793 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:14:48 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T05:14:48.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:14:48 smithi047 conmon[78521]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T05:14:48.794 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:14:48 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T05:14:48.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:14:48 smithi047 conmon[78521]: return self.event_loop.get_result(coro) 2022-05-14T05:14:48.795 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:14:48 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T05:14:48.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:14:48 smithi047 conmon[78521]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T05:14:48.796 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:14:48 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T05:14:48.797 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:14:48 smithi047 conmon[78521]: return self.__get_result() 2022-05-14T05:14:48.797 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:14:48 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T05:14:48.798 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:14:48 smithi047 conmon[78521]: raise self._exception 2022-05-14T05:14:48.798 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:14:48 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T05:14:48.798 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:14:48 smithi047 conmon[78521]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T05:14:48.799 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:14:48 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T05:14:48.799 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:14:48 smithi047 conmon[78521]: conn = await self._remote_connection(host, addr) 2022-05-14T05:14:48.799 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:14:48 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T05:14:48.800 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:14:48 smithi047 conmon[78521]: raise 2022-05-14T05:14:48.800 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:14:48 smithi047 conmon[78521]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T05:14:48.802 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:14:48 smithi047 conmon[78521]: self.gen.throw(type, value, traceback) 2022-05-14T05:14:48.802 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:14:48 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T05:14:48.802 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:14:48 smithi047 conmon[78521]: raise OrchestratorError(msg) 2022-05-14T05:14:48.802 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:14:48 smithi047 conmon[78521]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T05:14:49.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:14:48 smithi047 ceph-mon[77147]: pgmap v10761: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:14:49.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:14:48 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:14:49.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:14:48 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 20820 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T05:14:49.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:14:48 smithi115 conmon[90393]: 2022-05-14T05:14:48.873+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:14:49.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:14:48 smithi115 ceph-mon[78593]: pgmap v10761: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:14:49.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:14:48 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:14:49.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:14:48 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 20820 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T05:14:50.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:14:49 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T05:14:50.042 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T05:14:50.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T05:14:50.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T05:14:50.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T05:14:50.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T05:14:50.043 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T05:14:50.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T05:14:50.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T05:14:50.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T05:14:50.044 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T05:14:50.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T05:14:50.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T05:14:50.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T05:14:50.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T05:14:50.045 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T05:14:50.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T05:14:50.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T05:14:50.046 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T05:14:50.046 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:14:49 smithi047 ceph-mon[77147]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T05:14:50.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T05:14:50.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T05:14:50.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T05:14:50.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T05:14:50.047 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T05:14:50.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T05:14:50.048 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T05:14:50.049 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T05:14:50.049 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T05:14:50.049 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T05:14:50.049 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T05:14:50.050 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T05:14:50.050 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T05:14:50.050 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T05:14:50.050 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T05:14:50.050 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T05:14:50.051 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T05:14:50.051 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T05:14:50.052 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T05:14:50.052 INFO:journalctl@ceph.mon.b.smithi047.stdout: During handling of the above exception, another exception occurred: 2022-05-14T05:14:50.052 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T05:14:50.052 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T05:14:50.052 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T05:14:50.053 INFO:journalctl@ceph.mon.b.smithi047.stdout: return f(*arg) 2022-05-14T05:14:50.053 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T05:14:50.053 INFO:journalctl@ceph.mon.b.smithi047.stdout: self._write_client_files(client_files, host) 2022-05-14T05:14:50.053 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T05:14:50.053 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T05:14:50.054 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T05:14:50.054 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T05:14:50.055 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T05:14:50.055 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.event_loop.get_result(coro) 2022-05-14T05:14:50.055 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T05:14:50.055 INFO:journalctl@ceph.mon.b.smithi047.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T05:14:50.055 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T05:14:50.056 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.__get_result() 2022-05-14T05:14:50.056 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T05:14:50.056 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise self._exception 2022-05-14T05:14:50.056 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T05:14:50.057 INFO:journalctl@ceph.mon.b.smithi047.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T05:14:50.057 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T05:14:50.057 INFO:journalctl@ceph.mon.b.smithi047.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T05:14:50.057 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T05:14:50.058 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise 2022-05-14T05:14:50.058 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T05:14:50.058 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.gen.throw(type, value, traceback) 2022-05-14T05:14:50.058 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T05:14:50.058 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OrchestratorError(msg) 2022-05-14T05:14:50.059 INFO:journalctl@ceph.mon.b.smithi047.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T05:14:50.059 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:14:49 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:14:50.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:14:49 smithi115 conmon[90393]: 2022-05-14T05:14:49.869+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:14:50.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:14:49 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T05:14:50.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T05:14:50.110 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T05:14:50.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T05:14:50.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T05:14:50.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T05:14:50.111 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T05:14:50.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T05:14:50.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T05:14:50.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T05:14:50.112 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T05:14:50.113 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T05:14:50.113 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T05:14:50.114 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T05:14:50.114 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T05:14:50.114 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T05:14:50.115 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T05:14:50.115 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T05:14:50.115 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T05:14:50.116 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:14:49 smithi115 ceph-mon[78593]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T05:14:50.116 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T05:14:50.116 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T05:14:50.116 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T05:14:50.117 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T05:14:50.117 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T05:14:50.117 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T05:14:50.118 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T05:14:50.118 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T05:14:50.118 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T05:14:50.118 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T05:14:50.119 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T05:14:50.119 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T05:14:50.119 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T05:14:50.119 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T05:14:50.119 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T05:14:50.120 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T05:14:50.120 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T05:14:50.120 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T05:14:50.120 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T05:14:50.121 INFO:journalctl@ceph.mon.c.smithi115.stdout: During handling of the above exception, another exception occurred: 2022-05-14T05:14:50.121 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T05:14:50.121 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T05:14:50.122 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T05:14:50.122 INFO:journalctl@ceph.mon.c.smithi115.stdout: return f(*arg) 2022-05-14T05:14:50.124 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T05:14:50.124 INFO:journalctl@ceph.mon.c.smithi115.stdout: self._write_client_files(client_files, host) 2022-05-14T05:14:50.124 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T05:14:50.125 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T05:14:50.126 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T05:14:50.126 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T05:14:50.126 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T05:14:50.127 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.event_loop.get_result(coro) 2022-05-14T05:14:50.127 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T05:14:50.127 INFO:journalctl@ceph.mon.c.smithi115.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T05:14:50.127 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T05:14:50.128 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.__get_result() 2022-05-14T05:14:50.128 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T05:14:50.128 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise self._exception 2022-05-14T05:14:50.128 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T05:14:50.129 INFO:journalctl@ceph.mon.c.smithi115.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T05:14:50.129 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T05:14:50.129 INFO:journalctl@ceph.mon.c.smithi115.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T05:14:50.130 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T05:14:50.130 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise 2022-05-14T05:14:50.130 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T05:14:50.130 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.gen.throw(type, value, traceback) 2022-05-14T05:14:50.131 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T05:14:50.131 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OrchestratorError(msg) 2022-05-14T05:14:50.131 INFO:journalctl@ceph.mon.c.smithi115.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T05:14:50.131 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:14:49 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:14:51.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:14:50 smithi047 ceph-mon[77147]: pgmap v10762: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T05:14:51.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:14:50 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T05:14:51.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:14:50 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T05:14:51.045 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:14:50 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:14:51.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:14:50 smithi115 conmon[90393]: 2022-05-14T05:14:50.826+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:14:51.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:14:50 smithi115 ceph-mon[78593]: pgmap v10762: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T05:14:51.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:14:50 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T05:14:51.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:14:50 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T05:14:51.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:14:50 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:14:52.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:14:51 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:14:52.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:14:51 smithi115 conmon[90393]: 2022-05-14T05:14:51.843+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:14:52.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:14:51 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:14:53.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:14:52 smithi047 ceph-mon[77147]: pgmap v10763: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:14:53.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:14:52 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:14:53.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:14:52 smithi115 conmon[90393]: 2022-05-14T05:14:52.875+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:14:53.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:14:52 smithi115 ceph-mon[78593]: pgmap v10763: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:14:53.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:14:52 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:14:54.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:14:53 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:14:54.043 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:14:53 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 20825 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T05:14:54.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:14:53 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:14:54.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:14:53 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 20825 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T05:14:54.111 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:14:53 smithi115 conmon[90393]: 2022-05-14T05:14:53.830+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:14:55.042 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:14:54 smithi047 ceph-mon[77147]: pgmap v10764: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:14:55.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:14:54 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:14:55.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:14:54 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T05:14:55.044 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:14:54 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T05:14:55.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:14:54 smithi115 conmon[90393]: 2022-05-14T05:14:54.818+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:14:55.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:14:54 smithi115 ceph-mon[78593]: pgmap v10764: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:14:55.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:14:54 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:14:55.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:14:54 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T05:14:55.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:14:54 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T05:14:56.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:14:55 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:14:56.110 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:14:55 smithi115 conmon[90393]: 2022-05-14T05:14:55.836+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:14:56.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:14:55 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:14:57.110 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:14:56 smithi115 conmon[90393]: 2022-05-14T05:14:56.800+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:14:57.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:14:56 smithi115 ceph-mon[78593]: pgmap v10765: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T05:14:57.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:14:56 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:14:57.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:14:56 smithi047 ceph-mon[77147]: pgmap v10765: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T05:14:57.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:14:56 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:14:58.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:14:57 smithi115 conmon[90393]: 2022-05-14T05:14:57.752+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:14:58.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:14:57 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:14:58.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:14:57 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:14:59.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:14:58 smithi115 conmon[90393]: 2022-05-14T05:14:58.779+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:14:59.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:14:58 smithi115 ceph-mon[78593]: pgmap v10766: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:14:59.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:14:58 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:14:59.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:14:58 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T05:14:59.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:14:58 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T05:14:59.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:14:58 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 20830 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T05:14:59.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:14:58 smithi047 ceph-mon[77147]: pgmap v10766: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:14:59.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:14:58 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:14:59.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:14:58 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T05:14:59.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:14:58 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T05:14:59.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:14:58 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 20830 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T05:15:00.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:14:59 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:15:00.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:14:59 smithi115 conmon[90393]: 2022-05-14T05:14:59.807+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:15:00.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:14:59 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:15:01.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:15:00 smithi115 conmon[90393]: 2022-05-14T05:15:00.821+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:15:01.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:15:00 smithi115 ceph-mon[78593]: pgmap v10767: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T05:15:01.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:15:00 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:15:01.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:15:00 smithi047 ceph-mon[77147]: pgmap v10767: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T05:15:01.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:15:00 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:15:02.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:15:01 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:15:02.111 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:15:01 smithi115 conmon[90393]: 2022-05-14T05:15:01.800+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:15:02.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:15:01 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:15:03.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:15:02 smithi115 conmon[90393]: 2022-05-14T05:15:02.769+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:15:03.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:15:02 smithi115 ceph-mon[78593]: pgmap v10768: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:15:03.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:15:02 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:15:03.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:15:02 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T05:15:03.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:15:02 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T05:15:03.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:15:02 smithi047 ceph-mon[77147]: pgmap v10768: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:15:03.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:15:02 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:15:03.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:15:02 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T05:15:03.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:15:02 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T05:15:04.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:15:03 smithi115 conmon[90393]: 2022-05-14T05:15:03.803+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:15:04.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:15:03 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:15:04.112 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:15:03 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 20835 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T05:15:04.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:15:03 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:15:04.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:15:03 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 20835 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T05:15:05.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:15:04 smithi115 conmon[90393]: 2022-05-14T05:15:04.772+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:15:05.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:15:04 smithi115 ceph-mon[78593]: pgmap v10769: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:15:05.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:15:04 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:15:05.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:15:04 smithi047 ceph-mon[77147]: pgmap v10769: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:15:05.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:15:04 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:15:06.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:15:05 smithi115 conmon[90393]: 2022-05-14T05:15:05.820+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:15:06.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:15:05 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:15:06.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:15:05 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:15:07.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:15:06 smithi115 conmon[90393]: 2022-05-14T05:15:06.824+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:15:07.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:15:06 smithi115 ceph-mon[78593]: pgmap v10770: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T05:15:07.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:15:06 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:15:07.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:15:06 smithi047 ceph-mon[77147]: pgmap v10770: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T05:15:07.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:15:06 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:15:08.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:15:07 smithi115 conmon[90393]: 2022-05-14T05:15:07.784+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:15:08.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:15:07 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:15:08.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:15:07 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:15:09.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:15:08 smithi115 conmon[90393]: 2022-05-14T05:15:08.758+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:15:09.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:15:08 smithi115 ceph-mon[78593]: pgmap v10771: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:15:09.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:15:08 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:15:09.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:15:08 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 20840 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T05:15:09.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:15:08 smithi047 ceph-mon[77147]: pgmap v10771: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:15:09.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:15:08 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:15:09.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:15:08 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 20840 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T05:15:10.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:15:09 smithi115 conmon[90393]: 2022-05-14T05:15:09.764+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:15:10.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:15:09 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:15:10.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:15:09 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:15:11.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:15:10 smithi115 conmon[90393]: 2022-05-14T05:15:10.789+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:15:11.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:15:10 smithi115 ceph-mon[78593]: pgmap v10772: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T05:15:11.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:15:10 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:15:11.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:15:10 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T05:15:11.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:15:10 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T05:15:11.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:15:10 smithi047 ceph-mon[77147]: pgmap v10772: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T05:15:11.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:15:10 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:15:11.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:15:10 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T05:15:11.295 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:15:10 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T05:15:12.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:15:11 smithi115 conmon[90393]: 2022-05-14T05:15:11.762+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:15:12.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:15:11 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:15:12.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:15:11 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:15:13.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:15:12 smithi115 conmon[90393]: 2022-05-14T05:15:12.720+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:15:13.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:15:12 smithi115 ceph-mon[78593]: pgmap v10773: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:15:13.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:15:12 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:15:13.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:15:12 smithi047 ceph-mon[77147]: pgmap v10773: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:15:13.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:15:12 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:15:14.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:15:13 smithi115 conmon[90393]: 2022-05-14T05:15:13.727+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:15:14.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:15:13 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:15:14.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:15:13 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 20845 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T05:15:14.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:15:13 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:15:14.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:15:13 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 20845 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T05:15:15.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:15:14 smithi115 conmon[90393]: 2022-05-14T05:15:14.773+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:15:15.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:15:14 smithi115 ceph-mon[78593]: pgmap v10774: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:15:15.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:15:14 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:15:15.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:15:14 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T05:15:15.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:15:14 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T05:15:15.113 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:15:14 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T05:15:15.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:15:14 smithi047 ceph-mon[77147]: pgmap v10774: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:15:15.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:15:14 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:15:15.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:15:14 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T05:15:15.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:15:14 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T05:15:15.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:15:14 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T05:15:16.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:15:15 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:15:16.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:15:15 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T05:15:16.110 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:15:15 smithi115 conmon[90393]: 2022-05-14T05:15:15.781+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:15:16.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:15:15 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:15:16.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:15:15 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T05:15:17.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:15:16 smithi115 ceph-mon[78593]: pgmap v10775: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T05:15:17.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:15:16 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:15:17.110 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:15:16 smithi115 conmon[90393]: 2022-05-14T05:15:16.771+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:15:17.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:15:16 smithi047 ceph-mon[77147]: pgmap v10775: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T05:15:17.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:15:16 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:15:18.118 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:15:17 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:15:18.119 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:15:17 smithi115 conmon[90393]: 2022-05-14T05:15:17.750+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:15:18.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:15:17 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:15:19.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:15:18 smithi115 conmon[90393]: 2022-05-14T05:15:18.740+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:15:19.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:15:18 smithi115 ceph-mon[78593]: pgmap v10776: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:15:19.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:15:18 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:15:19.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:15:18 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T05:15:19.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:15:18 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T05:15:19.112 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:15:18 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 20850 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T05:15:19.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:15:18 smithi047 ceph-mon[77147]: pgmap v10776: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:15:19.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:15:18 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:15:19.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:15:18 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T05:15:19.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:15:18 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T05:15:19.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:15:18 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 20850 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T05:15:20.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:15:19 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:15:20.110 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:15:19 smithi115 conmon[90393]: 2022-05-14T05:15:19.714+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:15:20.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:15:19 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:15:21.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:15:20 smithi115 ceph-mon[78593]: pgmap v10777: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T05:15:21.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:15:20 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:15:21.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:15:20 smithi115 conmon[90393]: 2022-05-14T05:15:20.719+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:15:21.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:15:20 smithi047 ceph-mon[77147]: pgmap v10777: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T05:15:21.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:15:20 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:15:22.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:15:21 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:15:22.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:15:21 smithi115 conmon[90393]: 2022-05-14T05:15:21.754+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:15:22.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:15:21 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:15:23.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:15:22 smithi115 conmon[90393]: 2022-05-14T05:15:22.750+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:15:23.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:15:22 smithi115 ceph-mon[78593]: pgmap v10778: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:15:23.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:15:22 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:15:23.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:15:22 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T05:15:23.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:15:22 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T05:15:23.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:15:22 smithi047 ceph-mon[77147]: pgmap v10778: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:15:23.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:15:22 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:15:23.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:15:22 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T05:15:23.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:15:22 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T05:15:24.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:15:23 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:15:24.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:15:23 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 20855 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T05:15:24.110 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:15:23 smithi115 conmon[90393]: 2022-05-14T05:15:23.745+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:15:24.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:15:23 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:15:24.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:15:23 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 20855 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T05:15:25.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:15:24 smithi115 conmon[90393]: 2022-05-14T05:15:24.732+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:15:25.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:15:24 smithi115 ceph-mon[78593]: pgmap v10779: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:15:25.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:15:24 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:15:25.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:15:24 smithi047 ceph-mon[77147]: pgmap v10779: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:15:25.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:15:24 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:15:26.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:15:25 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:15:26.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:15:25 smithi115 conmon[90393]: 2022-05-14T05:15:25.739+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:15:26.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:15:25 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:15:27.112 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:15:26 smithi115 conmon[90393]: 2022-05-14T05:15:26.707+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:15:27.112 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:15:26 smithi115 ceph-mon[78593]: pgmap v10780: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T05:15:27.113 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:15:26 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:15:27.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:15:26 smithi047 ceph-mon[77147]: pgmap v10780: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T05:15:27.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:15:26 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:15:28.108 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:15:27 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:15:28.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:15:27 smithi115 conmon[90393]: 2022-05-14T05:15:27.683+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:15:28.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:15:27 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:15:29.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:15:28 smithi115 conmon[90393]: 2022-05-14T05:15:28.731+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:15:29.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:15:28 smithi115 ceph-mon[78593]: pgmap v10781: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:15:29.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:15:28 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:15:29.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:15:28 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 20860 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T05:15:29.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:15:28 smithi047 ceph-mon[77147]: pgmap v10781: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:15:29.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:15:28 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:15:29.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:15:28 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 20860 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T05:15:30.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:15:29 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:15:30.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:15:29 smithi115 conmon[90393]: 2022-05-14T05:15:29.779+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:15:30.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:15:29 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:15:31.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:15:30 smithi115 conmon[90393]: 2022-05-14T05:15:30.751+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:15:31.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:15:30 smithi115 ceph-mon[78593]: pgmap v10782: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T05:15:31.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:15:30 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:15:31.112 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:15:30 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T05:15:31.112 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:15:30 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T05:15:31.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:15:30 smithi047 ceph-mon[77147]: pgmap v10782: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T05:15:31.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:15:30 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:15:31.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:15:30 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T05:15:31.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:15:30 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T05:15:32.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:15:31 smithi115 conmon[90393]: 2022-05-14T05:15:31.797+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:15:32.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:15:31 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:15:32.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:15:31 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:15:33.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:15:32 smithi115 conmon[90393]: 2022-05-14T05:15:32.758+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:15:33.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:15:32 smithi115 ceph-mon[78593]: pgmap v10783: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:15:33.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:15:32 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:15:33.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:15:32 smithi047 ceph-mon[77147]: pgmap v10783: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:15:33.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:15:32 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:15:34.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:15:33 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:15:34.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:15:33 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 20865 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T05:15:34.110 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:15:33 smithi115 conmon[90393]: 2022-05-14T05:15:33.734+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:15:34.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:15:33 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:15:34.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:15:33 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 20865 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T05:15:35.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:15:34 smithi115 conmon[90393]: 2022-05-14T05:15:34.685+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:15:35.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:15:34 smithi115 ceph-mon[78593]: pgmap v10784: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:15:35.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:15:34 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:15:35.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:15:34 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T05:15:35.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:15:34 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T05:15:35.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:15:34 smithi047 ceph-mon[77147]: pgmap v10784: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:15:35.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:15:34 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:15:35.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:15:34 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T05:15:35.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:15:34 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T05:15:36.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:15:35 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:15:36.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:15:35 smithi115 conmon[90393]: 2022-05-14T05:15:35.663+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:15:36.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:15:35 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:15:36.910 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:15:36 smithi115 conmon[90393]: 2022-05-14T05:15:36.640+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:15:37.295 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:15:36 smithi047 ceph-mon[77147]: pgmap v10785: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T05:15:37.296 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:15:36 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:15:37.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:15:36 smithi115 ceph-mon[78593]: pgmap v10785: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T05:15:37.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:15:36 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:15:37.916 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:15:37 smithi115 conmon[90393]: 2022-05-14T05:15:37.641+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:15:38.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:15:37 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:15:38.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:15:37 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:15:38.917 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:15:38 smithi115 conmon[90393]: 2022-05-14T05:15:38.621+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:15:39.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:15:38 smithi047 ceph-mon[77147]: pgmap v10786: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:15:39.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:15:38 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:15:39.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:15:38 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T05:15:39.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:15:38 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T05:15:39.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:15:38 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 20870 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T05:15:39.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:15:38 smithi115 ceph-mon[78593]: pgmap v10786: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:15:39.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:15:38 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:15:39.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:15:38 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T05:15:39.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:15:38 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T05:15:39.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:15:38 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 20870 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T05:15:39.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:15:39 smithi115 conmon[90393]: 2022-05-14T05:15:39.591+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:15:40.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:15:39 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:15:40.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:15:39 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:15:40.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:15:40 smithi115 conmon[90393]: 2022-05-14T05:15:40.555+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:15:41.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:15:40 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:15:41.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:15:40 smithi047 ceph-mon[77147]: pgmap v10787: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T05:15:41.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:15:40 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:15:41.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:15:40 smithi115 ceph-mon[78593]: pgmap v10787: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T05:15:41.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:15:41 smithi115 conmon[90393]: 2022-05-14T05:15:41.541+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:15:42.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:15:41 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:15:42.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:15:41 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:15:42.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:15:42 smithi115 conmon[90393]: 2022-05-14T05:15:42.503+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:15:43.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:15:42 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:15:43.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:15:42 smithi047 ceph-mon[77147]: pgmap v10788: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:15:43.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:15:42 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T05:15:43.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:15:42 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T05:15:43.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:15:42 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:15:43.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:15:42 smithi115 ceph-mon[78593]: pgmap v10788: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:15:43.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:15:42 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T05:15:43.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:15:42 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T05:15:43.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:15:43 smithi115 conmon[90393]: 2022-05-14T05:15:43.526+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:15:44.301 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:15:43 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:15:44.302 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:15:43 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 20875 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T05:15:44.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:15:43 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:15:44.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:15:43 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 20875 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T05:15:44.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:15:44 smithi115 conmon[90393]: 2022-05-14T05:15:44.518+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:15:45.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:15:44 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:15:45.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:15:44 smithi047 ceph-mon[77147]: pgmap v10789: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:15:45.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:15:44 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:15:45.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:15:44 smithi115 ceph-mon[78593]: pgmap v10789: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:15:45.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:15:45 smithi115 conmon[90393]: 2022-05-14T05:15:45.534+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:15:46.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:15:45 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:15:46.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:15:45 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:15:46.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:15:46 smithi115 conmon[90393]: 2022-05-14T05:15:46.551+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:15:47.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:15:46 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:15:47.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:15:46 smithi047 ceph-mon[77147]: pgmap v10790: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T05:15:47.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:15:46 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:15:47.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:15:46 smithi115 ceph-mon[78593]: pgmap v10790: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T05:15:47.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:15:47 smithi115 conmon[90393]: 2022-05-14T05:15:47.574+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:15:48.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:15:47 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:15:48.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:15:47 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:15:48.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:15:48 smithi115 conmon[90393]: 2022-05-14T05:15:48.616+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:15:49.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:15:48 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:15:49.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:15:48 smithi047 ceph-mon[77147]: pgmap v10791: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:15:49.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:15:48 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 20880 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T05:15:49.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:15:48 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T05:15:49.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:15:48 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T05:15:49.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:15:48 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T05:15:49.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:15:48 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' 2022-05-14T05:15:49.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:15:48 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' 2022-05-14T05:15:49.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:15:48 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:15:49.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:15:48 smithi115 ceph-mon[78593]: pgmap v10791: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:15:49.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:15:48 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 20880 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T05:15:49.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:15:48 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-05-14T05:15:49.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:15:48 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-05-14T05:15:49.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:15:48 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-05-14T05:15:49.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:15:48 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' 2022-05-14T05:15:49.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:15:48 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' 2022-05-14T05:15:49.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:15:49 smithi115 conmon[90393]: 2022-05-14T05:15:49.603+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:15:50.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:15:49 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:15:50.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:15:49 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:15:50.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:15:50 smithi115 conmon[90393]: 2022-05-14T05:15:50.600+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:15:51.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:15:50 smithi047 ceph-mon[77147]: pgmap v10792: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T05:15:51.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:15:50 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:15:51.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:15:50 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T05:15:51.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:15:50 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T05:15:51.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:15:50 smithi115 ceph-mon[78593]: pgmap v10792: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T05:15:51.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:15:50 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:15:51.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:15:50 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T05:15:51.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:15:50 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T05:15:51.792 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:15:51 smithi047 conmon[78521]: 2022-05-14T05:15:51.536+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T05:15:51.793 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:15:51 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T05:15:51.793 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:15:51 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T05:15:51.793 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:15:51 smithi047 conmon[78521]: yield 2022-05-14T05:15:51.799 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:15:51 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T05:15:51.799 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:15:51 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T05:15:51.799 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:15:51 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T05:15:51.800 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:15:51 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T05:15:51.800 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:15:51 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T05:15:51.800 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:15:51 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T05:15:51.801 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:15:51 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T05:15:51.801 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:15:51 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T05:15:51.801 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:15:51 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T05:15:51.801 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:15:51 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T05:15:51.802 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:15:51 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T05:15:51.802 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:15:51 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T05:15:51.802 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:15:51 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T05:15:51.802 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:15:51 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T05:15:51.803 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:15:51 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T05:15:51.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:15:51 smithi115 conmon[90393]: 2022-05-14T05:15:51.608+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:15:52.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:15:51 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:15:52.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:15:51 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' 2022-05-14T05:15:52.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:15:51 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' 2022-05-14T05:15:52.363 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:15:51 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:15:52.364 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:15:51 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' 2022-05-14T05:15:52.364 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:15:51 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' 2022-05-14T05:15:52.956 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:15:52 smithi115 conmon[90393]: 2022-05-14T05:15:52.615+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:15:53.301 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:15:52 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T05:15:53.304 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T05:15:53.305 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T05:15:53.306 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T05:15:53.306 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T05:15:53.306 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T05:15:53.307 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T05:15:53.307 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T05:15:53.307 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T05:15:53.308 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T05:15:53.308 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T05:15:53.308 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T05:15:53.309 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T05:15:53.309 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T05:15:53.309 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T05:15:53.310 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T05:15:53.311 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T05:15:53.312 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T05:15:53.312 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T05:15:53.313 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:15:52 smithi047 ceph-mon[77147]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T05:15:53.314 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:15:52 smithi047 ceph-mon[77147]: pgmap v10793: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:15:53.314 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:15:52 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:15:53.314 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:15:52 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-05-14T05:15:53.315 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:15:52 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-05-14T05:15:53.315 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:15:52 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-05-14T05:15:53.315 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:15:52 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-05-14T05:15:53.316 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:15:52 smithi047 ceph-mon[77147]: Adjusting osd_memory_target on smithi115 to 197.0M 2022-05-14T05:15:53.316 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:15:52 smithi047 ceph-mon[77147]: Unable to set osd_memory_target on smithi115 to 206629785: error parsing value: Value '206629785' is below minimum 939524096 2022-05-14T05:15:53.317 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:15:52 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-05-14T05:15:53.317 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:15:52 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-05-14T05:15:53.318 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:15:52 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-05-14T05:15:53.318 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:15:52 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-05-14T05:15:53.318 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:15:52 smithi047 ceph-mon[77147]: Adjusting osd_memory_target on smithi047 to 198.1M 2022-05-14T05:15:53.319 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:15:52 smithi047 ceph-mon[77147]: Unable to set osd_memory_target on smithi047 to 207755878: error parsing value: Value '207755878' is below minimum 939524096 2022-05-14T05:15:53.364 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:15:52 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T05:15:53.365 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T05:15:53.366 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T05:15:53.366 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T05:15:53.367 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T05:15:53.367 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T05:15:53.368 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T05:15:53.368 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T05:15:53.369 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T05:15:53.369 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T05:15:53.369 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T05:15:53.370 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T05:15:53.371 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T05:15:53.371 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T05:15:53.372 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T05:15:53.373 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T05:15:53.373 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T05:15:53.374 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T05:15:53.374 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T05:15:53.375 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:15:52 smithi115 ceph-mon[78593]: Removing smithi002:/etc/ceph/ceph.conf 2022-05-14T05:15:53.375 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:15:52 smithi115 ceph-mon[78593]: pgmap v10793: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:15:53.376 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:15:52 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:15:53.376 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:15:52 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-05-14T05:15:53.377 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:15:52 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-05-14T05:15:53.378 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:15:52 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-05-14T05:15:53.378 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:15:52 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-05-14T05:15:53.378 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:15:52 smithi115 ceph-mon[78593]: Adjusting osd_memory_target on smithi115 to 197.0M 2022-05-14T05:15:53.379 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:15:52 smithi115 ceph-mon[78593]: Unable to set osd_memory_target on smithi115 to 206629785: error parsing value: Value '206629785' is below minimum 939524096 2022-05-14T05:15:53.379 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:15:52 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-05-14T05:15:53.380 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:15:52 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-05-14T05:15:53.380 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:15:52 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-05-14T05:15:53.381 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:15:52 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-05-14T05:15:53.381 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:15:52 smithi115 ceph-mon[78593]: Adjusting osd_memory_target on smithi047 to 198.1M 2022-05-14T05:15:53.382 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:15:52 smithi115 ceph-mon[78593]: Unable to set osd_memory_target on smithi047 to 207755878: error parsing value: Value '207755878' is below minimum 939524096 2022-05-14T05:15:53.956 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:15:53 smithi115 conmon[90393]: 2022-05-14T05:15:53.635+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:15:54.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:15:53 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:15:54.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:15:53 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 20885 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T05:15:54.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:15:53 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:15:54.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:15:53 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 20885 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T05:15:54.960 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:15:54 smithi115 conmon[90393]: 2022-05-14T05:15:54.669+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:15:54.963 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:15:54 smithi047 conmon[78521]: 2022-05-14T05:15:54.608+0000 7f5338f6a700 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T05:15:54.963 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:15:54 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T05:15:54.964 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:15:54 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T05:15:54.964 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:15:54 smithi047 conmon[78521]: yield 2022-05-14T05:15:54.964 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:15:54 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T05:15:54.967 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:15:54 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T05:15:54.967 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:15:54 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T05:15:54.967 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:15:54 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T05:15:54.967 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:15:54 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T05:15:54.968 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:15:54 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T05:15:54.968 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:15:54 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T05:15:54.968 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:15:54 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T05:15:54.968 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:15:54 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T05:15:54.968 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:15:54 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T05:15:54.969 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:15:54 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T05:15:54.969 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:15:54 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T05:15:54.969 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:15:54 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T05:15:54.970 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:15:54 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T05:15:54.970 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:15:54 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T05:15:54.970 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:15:54 smithi047 conmon[78521]: 2022-05-14T05:15:54.609+0000 7f5344781700 -1 log_channel(cephadm) log [ERR] : executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T05:15:54.970 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:15:54 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T05:15:54.970 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:15:54 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T05:15:54.971 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:15:54 smithi047 conmon[78521]: yield 2022-05-14T05:15:54.971 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:15:54 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T05:15:54.971 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:15:54 smithi047 conmon[78521]: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T05:15:54.971 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:15:54 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T05:15:54.972 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:15:54 smithi047 conmon[78521]: 'Opening SSH connection to') 2022-05-14T05:15:54.972 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:15:54 smithi047 conmon[78521]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T05:15:54.972 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:15:54 smithi047 conmon[78521]: local_addr=local_addr) 2022-05-14T05:15:54.972 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:15:54 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T05:15:54.972 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:15:54 smithi047 conmon[78521]: raise exceptions[0] 2022-05-14T05:15:54.973 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:15:54 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T05:15:54.973 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:15:54 smithi047 conmon[78521]: yield from self.sock_connect(sock, address) 2022-05-14T05:15:54.973 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:15:54 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T05:15:54.973 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:15:54 smithi047 conmon[78521]: return (yield from fut) 2022-05-14T05:15:54.973 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:15:54 smithi047 conmon[78521]: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T05:15:54.974 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:15:54 smithi047 conmon[78521]: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T05:15:54.974 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:15:54 smithi047 conmon[78521]: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T05:15:54.974 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:15:54 smithi047 conmon[78521]: 2022-05-14T05:15:54.974 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:15:54 smithi047 conmon[78521]: During handling of the above exception, another exception occurred: 2022-05-14T05:15:54.975 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:15:54 smithi047 conmon[78521]: 2022-05-14T05:15:54.975 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:15:54 smithi047 conmon[78521]: Traceback (most recent call last): 2022-05-14T05:15:54.975 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:15:54 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T05:15:54.975 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:15:54 smithi047 conmon[78521]: return f(*arg) 2022-05-14T05:15:54.975 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:15:54 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T05:15:54.976 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:15:54 smithi047 conmon[78521]: self._write_client_files(client_files, host) 2022-05-14T05:15:54.976 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:15:54 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T05:15:54.976 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:15:54 smithi047 conmon[78521]: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T05:15:54.976 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:15:54 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T05:15:54.976 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:15:54 smithi047 conmon[78521]: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T05:15:54.977 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:15:54 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T05:15:54.977 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:15:54 smithi047 conmon[78521]: return self.event_loop.get_result(coro) 2022-05-14T05:15:54.977 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:15:54 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T05:15:54.977 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:15:54 smithi047 conmon[78521]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T05:15:54.977 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:15:54 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T05:15:54.978 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:15:54 smithi047 conmon[78521]: return self.__get_result() 2022-05-14T05:15:54.978 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:15:54 smithi047 conmon[78521]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T05:15:54.978 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:15:54 smithi047 conmon[78521]: raise self._exception 2022-05-14T05:15:54.978 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:15:54 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T05:15:54.979 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:15:54 smithi047 conmon[78521]: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T05:15:54.979 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:15:54 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T05:15:54.979 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:15:54 smithi047 conmon[78521]: conn = await self._remote_connection(host, addr) 2022-05-14T05:15:54.979 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:15:54 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T05:15:54.979 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:15:54 smithi047 conmon[78521]: raise 2022-05-14T05:15:54.980 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:15:54 smithi047 conmon[78521]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T05:15:54.980 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:15:54 smithi047 conmon[78521]: self.gen.throw(type, value, traceback) 2022-05-14T05:15:54.980 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:15:54 smithi047 conmon[78521]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T05:15:54.980 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:15:54 smithi047 conmon[78521]: raise OrchestratorError(msg) 2022-05-14T05:15:54.980 INFO:journalctl@ceph.mgr.y.smithi047.stdout:May 14 05:15:54 smithi047 conmon[78521]: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T05:15:55.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:15:54 smithi047 ceph-mon[77147]: pgmap v10794: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:15:55.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:15:54 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:15:55.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:15:54 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T05:15:55.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:15:54 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T05:15:55.362 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:15:54 smithi115 ceph-mon[78593]: pgmap v10794: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:15:55.363 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:15:54 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:15:55.363 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:15:54 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T05:15:55.364 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:15:54 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T05:15:55.966 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:15:55 smithi115 conmon[90393]: 2022-05-14T05:15:55.634+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:15:56.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:15:55 smithi047 ceph-mon[77147]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T05:15:56.292 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T05:15:56.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T05:15:56.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T05:15:56.293 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T05:15:56.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T05:15:56.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T05:15:56.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T05:15:56.295 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T05:15:56.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T05:15:56.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T05:15:56.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T05:15:56.296 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T05:15:56.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T05:15:56.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T05:15:56.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T05:15:56.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T05:15:56.297 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T05:15:56.298 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T05:15:56.298 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:15:55 smithi047 ceph-mon[77147]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T05:15:56.298 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T05:15:56.298 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T05:15:56.298 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield 2022-05-14T05:15:56.299 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T05:15:56.299 INFO:journalctl@ceph.mon.b.smithi047.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T05:15:56.299 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T05:15:56.299 INFO:journalctl@ceph.mon.b.smithi047.stdout: 'Opening SSH connection to') 2022-05-14T05:15:56.300 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T05:15:56.300 INFO:journalctl@ceph.mon.b.smithi047.stdout: local_addr=local_addr) 2022-05-14T05:15:56.300 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T05:15:56.300 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise exceptions[0] 2022-05-14T05:15:56.300 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T05:15:56.301 INFO:journalctl@ceph.mon.b.smithi047.stdout: yield from self.sock_connect(sock, address) 2022-05-14T05:15:56.301 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T05:15:56.301 INFO:journalctl@ceph.mon.b.smithi047.stdout: return (yield from fut) 2022-05-14T05:15:56.301 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T05:15:56.302 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T05:15:56.302 INFO:journalctl@ceph.mon.b.smithi047.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T05:15:56.302 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T05:15:56.302 INFO:journalctl@ceph.mon.b.smithi047.stdout: During handling of the above exception, another exception occurred: 2022-05-14T05:15:56.302 INFO:journalctl@ceph.mon.b.smithi047.stdout: 2022-05-14T05:15:56.304 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2022-05-14T05:15:56.305 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T05:15:56.305 INFO:journalctl@ceph.mon.b.smithi047.stdout: return f(*arg) 2022-05-14T05:15:56.305 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T05:15:56.305 INFO:journalctl@ceph.mon.b.smithi047.stdout: self._write_client_files(client_files, host) 2022-05-14T05:15:56.306 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T05:15:56.306 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T05:15:56.306 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T05:15:56.306 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T05:15:56.307 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T05:15:56.307 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.event_loop.get_result(coro) 2022-05-14T05:15:56.307 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T05:15:56.307 INFO:journalctl@ceph.mon.b.smithi047.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T05:15:56.307 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T05:15:56.308 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.__get_result() 2022-05-14T05:15:56.308 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T05:15:56.308 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise self._exception 2022-05-14T05:15:56.308 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T05:15:56.308 INFO:journalctl@ceph.mon.b.smithi047.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T05:15:56.309 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T05:15:56.309 INFO:journalctl@ceph.mon.b.smithi047.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T05:15:56.309 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T05:15:56.309 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise 2022-05-14T05:15:56.310 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T05:15:56.310 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.gen.throw(type, value, traceback) 2022-05-14T05:15:56.310 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T05:15:56.310 INFO:journalctl@ceph.mon.b.smithi047.stdout: raise OrchestratorError(msg) 2022-05-14T05:15:56.311 INFO:journalctl@ceph.mon.b.smithi047.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T05:15:56.311 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:15:55 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:15:56.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:15:55 smithi115 ceph-mon[78593]: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T05:15:56.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T05:15:56.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T05:15:56.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T05:15:56.361 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T05:15:56.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T05:15:56.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T05:15:56.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T05:15:56.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T05:15:56.362 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T05:15:56.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T05:15:56.363 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T05:15:56.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T05:15:56.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T05:15:56.364 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T05:15:56.365 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T05:15:56.365 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T05:15:56.365 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T05:15:56.365 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T05:15:56.366 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:15:55 smithi115 ceph-mon[78593]: executing refresh((['smithi002', 'smithi047', 'smithi115'],)) failed. 2022-05-14T05:15:56.366 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T05:15:56.366 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 103, in redirect_log 2022-05-14T05:15:56.367 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield 2022-05-14T05:15:56.367 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 82, in _remote_connection 2022-05-14T05:15:56.367 INFO:journalctl@ceph.mon.c.smithi115.stdout: preferred_auth=['publickey'], options=ssh_options) 2022-05-14T05:15:56.367 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect 2022-05-14T05:15:56.367 INFO:journalctl@ceph.mon.c.smithi115.stdout: 'Opening SSH connection to') 2022-05-14T05:15:56.368 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 299, in _connect 2022-05-14T05:15:56.368 INFO:journalctl@ceph.mon.c.smithi115.stdout: local_addr=local_addr) 2022-05-14T05:15:56.368 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 794, in create_connection 2022-05-14T05:15:56.368 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise exceptions[0] 2022-05-14T05:15:56.369 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/base_events.py", line 781, in create_connection 2022-05-14T05:15:56.369 INFO:journalctl@ceph.mon.c.smithi115.stdout: yield from self.sock_connect(sock, address) 2022-05-14T05:15:56.369 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect 2022-05-14T05:15:56.369 INFO:journalctl@ceph.mon.c.smithi115.stdout: return (yield from fut) 2022-05-14T05:15:56.370 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb 2022-05-14T05:15:56.370 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OSError(err, 'Connect call failed %s' % (address,)) 2022-05-14T05:15:56.371 INFO:journalctl@ceph.mon.c.smithi115.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T05:15:56.371 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T05:15:56.371 INFO:journalctl@ceph.mon.c.smithi115.stdout: During handling of the above exception, another exception occurred: 2022-05-14T05:15:56.371 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2022-05-14T05:15:56.372 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2022-05-14T05:15:56.372 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/utils.py", line 78, in do_work 2022-05-14T05:15:56.372 INFO:journalctl@ceph.mon.c.smithi115.stdout: return f(*arg) 2022-05-14T05:15:56.372 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh 2022-05-14T05:15:56.372 INFO:journalctl@ceph.mon.c.smithi115.stdout: self._write_client_files(client_files, host) 2022-05-14T05:15:56.373 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1060, in _write_client_files 2022-05-14T05:15:56.373 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.mgr.ssh.check_execute_command(host, cmd) 2022-05-14T05:15:56.373 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in check_execute_command 2022-05-14T05:15:56.373 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.mgr.wait_async(self._check_execute_command(host, cmd, stdin, addr)) 2022-05-14T05:15:56.374 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 589, in wait_async 2022-05-14T05:15:56.374 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.event_loop.get_result(coro) 2022-05-14T05:15:56.374 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result 2022-05-14T05:15:56.374 INFO:journalctl@ceph.mon.c.smithi115.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() 2022-05-14T05:15:56.374 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2022-05-14T05:15:56.375 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.__get_result() 2022-05-14T05:15:56.375 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2022-05-14T05:15:56.375 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise self._exception 2022-05-14T05:15:56.375 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _check_execute_command 2022-05-14T05:15:56.375 INFO:journalctl@ceph.mon.c.smithi115.stdout: out, err, code = await self._execute_command(host, cmd, stdin, addr) 2022-05-14T05:15:56.376 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 138, in _execute_command 2022-05-14T05:15:56.376 INFO:journalctl@ceph.mon.c.smithi115.stdout: conn = await self._remote_connection(host, addr) 2022-05-14T05:15:56.376 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 88, in _remote_connection 2022-05-14T05:15:56.376 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise 2022-05-14T05:15:56.377 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ 2022-05-14T05:15:56.377 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.gen.throw(type, value, traceback) 2022-05-14T05:15:56.377 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 109, in redirect_log 2022-05-14T05:15:56.377 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OrchestratorError(msg) 2022-05-14T05:15:56.377 INFO:journalctl@ceph.mon.c.smithi115.stdout: orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.21.15.2`, possibly because python3 is not installed there. [Errno 113] Connect call failed ('172.21.15.2', 22) 2022-05-14T05:15:56.378 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:15:55 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:15:56.968 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:15:56 smithi115 conmon[90393]: 2022-05-14T05:15:56.644+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:15:57.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:15:56 smithi047 ceph-mon[77147]: pgmap v10795: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T05:15:57.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:15:56 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:15:57.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:15:56 smithi115 ceph-mon[78593]: pgmap v10795: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T05:15:57.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:15:56 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:15:57.978 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:15:57 smithi115 conmon[90393]: 2022-05-14T05:15:57.649+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:15:58.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:15:57 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:15:58.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:15:57 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:15:58.972 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:15:58 smithi115 conmon[90393]: 2022-05-14T05:15:58.610+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:15:59.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:15:58 smithi047 ceph-mon[77147]: pgmap v10796: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:15:59.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:15:58 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:15:59.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:15:58 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T05:15:59.295 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:15:58 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T05:15:59.295 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:15:58 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 20890 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T05:15:59.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:15:58 smithi115 ceph-mon[78593]: pgmap v10796: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:15:59.362 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:15:58 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:15:59.362 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:15:58 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T05:15:59.362 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:15:58 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T05:15:59.362 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:15:58 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 20890 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T05:15:59.977 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:15:59 smithi115 conmon[90393]: 2022-05-14T05:15:59.611+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:16:00.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:15:59 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:16:00.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:15:59 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:16:00.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:16:00 smithi115 conmon[90393]: 2022-05-14T05:16:00.564+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:16:01.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:16:00 smithi047 ceph-mon[77147]: pgmap v10797: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T05:16:01.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:16:00 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:16:01.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:16:00 smithi115 ceph-mon[78593]: pgmap v10797: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T05:16:01.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:16:00 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:16:01.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:16:01 smithi115 conmon[90393]: 2022-05-14T05:16:01.538+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:16:02.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:16:01 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:16:02.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:16:01 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:16:02.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:16:02 smithi115 conmon[90393]: 2022-05-14T05:16:02.577+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:16:03.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:16:02 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:16:03.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:16:02 smithi047 ceph-mon[77147]: pgmap v10798: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:16:03.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:16:02 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T05:16:03.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:16:02 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T05:16:03.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:16:02 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:16:03.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:16:02 smithi115 ceph-mon[78593]: pgmap v10798: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:16:03.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:16:02 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T05:16:03.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:16:02 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T05:16:03.987 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:16:03 smithi115 conmon[90393]: 2022-05-14T05:16:03.619+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:16:04.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:16:03 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:16:04.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:16:03 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 20895 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T05:16:04.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:16:03 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:16:04.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:16:03 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 20895 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T05:16:04.991 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:16:04 smithi115 conmon[90393]: 2022-05-14T05:16:04.621+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:16:05.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:16:04 smithi047 ceph-mon[77147]: pgmap v10799: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:16:05.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:16:04 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:16:05.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:16:04 smithi115 ceph-mon[78593]: pgmap v10799: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:16:05.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:16:04 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:16:05.993 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:16:05 smithi115 conmon[90393]: 2022-05-14T05:16:05.641+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:16:06.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:16:05 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:16:06.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:16:05 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:16:06.994 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:16:06 smithi115 conmon[90393]: 2022-05-14T05:16:06.678+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:16:07.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:16:06 smithi047 ceph-mon[77147]: pgmap v10800: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T05:16:07.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:16:06 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:16:07.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:16:06 smithi115 ceph-mon[78593]: pgmap v10800: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T05:16:07.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:16:06 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:16:07.999 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:16:07 smithi115 conmon[90393]: 2022-05-14T05:16:07.671+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:16:08.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:16:07 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:16:08.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:16:07 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:16:09.000 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:16:08 smithi115 conmon[90393]: 2022-05-14T05:16:08.712+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:16:09.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:16:08 smithi047 ceph-mon[77147]: pgmap v10801: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:16:09.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:16:08 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:16:09.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:16:08 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 20900 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T05:16:09.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:16:08 smithi115 ceph-mon[78593]: pgmap v10801: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:16:09.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:16:08 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:16:09.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:16:08 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 20900 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T05:16:10.006 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:16:09 smithi115 conmon[90393]: 2022-05-14T05:16:09.690+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:16:10.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:16:10 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:16:10.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:16:10 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:16:11.007 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:16:10 smithi115 conmon[90393]: 2022-05-14T05:16:10.735+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:16:11.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:16:11 smithi047 ceph-mon[77147]: pgmap v10802: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T05:16:11.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:16:11 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:16:11.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:16:11 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T05:16:11.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:16:11 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T05:16:11.362 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:16:11 smithi115 ceph-mon[78593]: pgmap v10802: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T05:16:11.362 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:16:11 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:16:11.362 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:16:11 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T05:16:11.362 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:16:11 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T05:16:12.012 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:16:11 smithi115 conmon[90393]: 2022-05-14T05:16:11.758+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:16:12.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:16:12 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:16:12.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:16:12 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:16:13.014 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:16:12 smithi115 conmon[90393]: 2022-05-14T05:16:12.731+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:16:13.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:16:13 smithi047 ceph-mon[77147]: pgmap v10803: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:16:13.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:16:13 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:16:13.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:16:13 smithi115 ceph-mon[78593]: pgmap v10803: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:16:13.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:16:13 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:16:14.016 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:16:13 smithi115 conmon[90393]: 2022-05-14T05:16:13.749+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:16:14.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:16:14 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:16:14.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:16:14 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 20905 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T05:16:14.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:16:14 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:16:14.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:16:14 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 20905 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T05:16:15.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:16:14 smithi115 conmon[90393]: 2022-05-14T05:16:14.772+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:16:15.109 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:16:15 smithi115 ceph-mon[78593]: pgmap v10804: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:16:15.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:16:15 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:16:15.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:16:15 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T05:16:15.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:16:15 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T05:16:15.110 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:16:15 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T05:16:15.111 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:16:15 smithi115 ceph-mon[78593]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T05:16:15.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:16:15 smithi047 ceph-mon[77147]: pgmap v10804: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:16:15.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:16:15 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:16:15.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:16:15 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T05:16:15.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:16:15 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T05:16:15.294 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:16:15 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2022-05-14T05:16:15.295 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:16:15 smithi047 ceph-mon[77147]: from='mgr.24115 172.21.15.47:0/1130007079' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2022-05-14T05:16:16.025 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:16:15 smithi115 conmon[90393]: 2022-05-14T05:16:15.758+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:16:16.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:16:16 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:16:16.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:16:16 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:16:17.024 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:16:16 smithi115 conmon[90393]: 2022-05-14T05:16:16.725+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:16:17.025 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:16:17 smithi115 ceph-mon[78593]: pgmap v10805: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T05:16:17.025 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:16:17 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:16:17.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:16:17 smithi047 ceph-mon[77147]: pgmap v10805: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T05:16:17.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:16:17 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:16:18.032 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:16:17 smithi115 conmon[90393]: 2022-05-14T05:16:17.719+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:16:18.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:16:18 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:16:18.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:16:18 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:16:19.034 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:16:18 smithi115 conmon[90393]: 2022-05-14T05:16:18.694+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:16:19.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:16:19 smithi047 ceph-mon[77147]: pgmap v10806: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:16:19.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:16:19 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:16:19.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:16:19 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T05:16:19.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:16:19 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T05:16:19.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:16:19 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 20910 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T05:16:19.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:16:19 smithi115 ceph-mon[78593]: pgmap v10806: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:16:19.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:16:19 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:16:19.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:16:19 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T05:16:19.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:16:19 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T05:16:19.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:16:19 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 20910 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T05:16:20.037 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:16:19 smithi115 conmon[90393]: 2022-05-14T05:16:19.703+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:16:20.292 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:16:20 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:16:20.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:16:20 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:16:21.038 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:16:20 smithi115 conmon[90393]: 2022-05-14T05:16:20.691+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:16:21.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:16:21 smithi047 ceph-mon[77147]: pgmap v10807: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T05:16:21.293 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:16:21 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:16:21.358 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:16:21 smithi115 ceph-mon[78593]: pgmap v10807: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T05:16:21.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:16:21 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:16:22.043 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:16:21 smithi115 conmon[90393]: 2022-05-14T05:16:21.720+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:16:22.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:16:22 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:16:22.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:16:22 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:16:23.044 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:16:22 smithi115 conmon[90393]: 2022-05-14T05:16:22.676+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:16:23.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:16:23 smithi115 ceph-mon[78593]: pgmap v10808: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:16:23.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:16:23 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:16:23.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:16:23 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T05:16:23.360 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:16:23 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T05:16:23.361 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:16:23 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:16:23.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:16:23 smithi047 ceph-mon[77147]: pgmap v10808: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:16:23.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:16:23 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:16:23.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:16:23 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T05:16:23.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:16:23 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T05:16:23.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:16:23 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:16:24.046 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:16:23 smithi115 conmon[90393]: 2022-05-14T05:16:23.700+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:16:24.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:16:24 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 20915 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T05:16:24.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:16:24 smithi115 ceph-mon[78593]: pgmap v10809: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:16:24.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:16:24 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:16:24.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:16:24 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 20915 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T05:16:24.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:16:24 smithi047 ceph-mon[77147]: pgmap v10809: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:16:24.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:16:24 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:16:25.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:16:24 smithi115 conmon[90393]: 2022-05-14T05:16:24.690+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:16:25.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:16:25 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:16:25.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:16:25 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:16:26.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:16:25 smithi115 conmon[90393]: 2022-05-14T05:16:25.649+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:16:26.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:16:26 smithi047 ceph-mon[77147]: pgmap v10810: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T05:16:26.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:16:26 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:16:26.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:16:26 smithi115 ceph-mon[78593]: pgmap v10810: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T05:16:26.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:16:26 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:16:27.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:16:26 smithi115 conmon[90393]: 2022-05-14T05:16:26.672+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:16:27.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:16:27 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:16:27.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:16:27 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:16:28.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:16:27 smithi115 conmon[90393]: 2022-05-14T05:16:27.685+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:16:28.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:16:28 smithi047 ceph-mon[77147]: pgmap v10811: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:16:28.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:16:28 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:16:28.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:16:28 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 20920 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T05:16:28.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:16:28 smithi115 ceph-mon[78593]: pgmap v10811: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:16:28.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:16:28 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:16:28.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:16:28 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 20920 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T05:16:29.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:16:28 smithi115 conmon[90393]: 2022-05-14T05:16:28.640+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:16:29.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:16:29 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:16:29.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:16:29 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:16:29.858 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:16:29 smithi115 conmon[90393]: 2022-05-14T05:16:29.606+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:16:30.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:16:30 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:16:30.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:16:30 smithi047 ceph-mon[77147]: pgmap v10812: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T05:16:30.605 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:16:30 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:16:30.605 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:16:30 smithi115 ceph-mon[78593]: pgmap v10812: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T05:16:30.859 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:16:30 smithi115 conmon[90393]: 2022-05-14T05:16:30.603+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:16:31.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:16:31 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T05:16:31.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:16:31 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T05:16:31.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:16:31 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:16:31.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:16:31 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T05:16:31.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:16:31 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T05:16:31.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:16:31 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:16:32.112 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:16:31 smithi115 conmon[90393]: 2022-05-14T05:16:31.621+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:16:32.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:16:32 smithi047 ceph-mon[77147]: pgmap v10813: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:16:32.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:16:32 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:16:32.608 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:16:32 smithi115 ceph-mon[78593]: pgmap v10813: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:16:32.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:16:32 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:16:33.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:16:32 smithi115 conmon[90393]: 2022-05-14T05:16:32.613+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:16:33.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:16:33 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:16:33.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:16:33 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 20925 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T05:16:33.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:16:33 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:16:33.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:16:33 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 20925 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T05:16:34.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:16:33 smithi115 conmon[90393]: 2022-05-14T05:16:33.615+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:16:34.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:16:34 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:16:34.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:16:34 smithi047 ceph-mon[77147]: pgmap v10814: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:16:34.593 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:16:34 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:16:34.594 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:16:34 smithi115 ceph-mon[78593]: pgmap v10814: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:16:34.861 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:16:34 smithi115 conmon[90393]: 2022-05-14T05:16:34.591+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:16:35.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:16:35 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T05:16:35.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:16:35 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T05:16:35.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:16:35 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:16:35.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:16:35 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T05:16:35.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:16:35 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T05:16:35.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:16:35 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:16:36.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:16:35 smithi115 conmon[90393]: 2022-05-14T05:16:35.632+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:16:36.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:16:36 smithi047 ceph-mon[77147]: pgmap v10815: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T05:16:36.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:16:36 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:16:36.614 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:16:36 smithi115 ceph-mon[78593]: pgmap v10815: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2022-05-14T05:16:36.614 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:16:36 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:16:37.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:16:36 smithi115 conmon[90393]: 2022-05-14T05:16:36.654+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:16:37.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:16:37 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:16:37.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:16:37 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:16:38.109 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:16:37 smithi115 conmon[90393]: 2022-05-14T05:16:37.686+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:16:38.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:16:38 smithi047 ceph-mon[77147]: pgmap v10816: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:16:38.542 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:16:38 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:16:38.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:16:38 smithi047 ceph-mon[77147]: Health check update: 9 slow ops, oldest one blocked for 20930 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T05:16:38.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:16:38 smithi115 ceph-mon[78593]: pgmap v10816: 97 pgs: 96 active+clean, 1 unknown; 146 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2022-05-14T05:16:38.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:16:38 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:16:38.610 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:16:38 smithi115 ceph-mon[78593]: Health check update: 9 slow ops, oldest one blocked for 20930 sec, osd.5 has slow ops (SLOW_OPS) 2022-05-14T05:16:39.108 INFO:journalctl@ceph.osd.5.smithi115.stdout:May 14 05:16:38 smithi115 conmon[90393]: 2022-05-14T05:16:38.702+0000 7fa085a08700 -1 osd.5 152 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.2.20:53 2.1 2.96a7dda1 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) 2022-05-14T05:16:39.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:16:39 smithi047 ceph-mon[77147]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T05:16:39.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:16:39 smithi047 ceph-mon[77147]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T05:16:39.543 INFO:journalctl@ceph.mon.b.smithi047.stdout:May 14 05:16:39 smithi047 ceph-mon[77147]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:16:39.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:16:39 smithi115 ceph-mon[78593]: mds.? [v2:172.21.15.115:6838/4149542923,v1:172.21.15.115:6839/4149542923] up:active 2022-05-14T05:16:39.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:16:39 smithi115 ceph-mon[78593]: fsmap cephfs:4/5 {0=l=up:active,1=i=up:active,3=c=up:active,4=k=up:active} 3 up:standby-replay 1 up:standby, 1 damaged 2022-05-14T05:16:39.609 INFO:journalctl@ceph.mon.c.smithi115.stdout:May 14 05:16:39 smithi115 ceph-mon[78593]: 9 slow requests (by type [ 'queued for pg' : 9 ] most affected pool [ 'cephfs_metadata' : 9 ]) 2022-05-14T05:16:39.659 DEBUG:teuthology.exit:Got signal 15; running 1 handler... 2022-05-14T05:16:39.663 DEBUG:teuthology.task.console_log:Killing console logger for smithi002 2022-05-14T05:16:39.669 DEBUG:teuthology.task.console_log:Killing console logger for smithi047 2022-05-14T05:16:39.669 DEBUG:teuthology.task.console_log:Killing console logger for smithi115 2022-05-14T05:16:39.670 DEBUG:teuthology.exit:Finished running handlers